2.3 Creating test suites
Robot Framework test cases are created in test case files, which can be organized into directories. These files and directories create a hierarchical test suite structure.
- 2.3.1 Test case files
- 2.3.2 Test suite directories
- 2.3.3 Test suite name and documentation
- 2.3.4 Free test suite metadata
- 2.3.5 Suite setup and teardown
2.3.1 Test case files
Robot Framework test cases are created using test case tables in test case files. Such a file automatically creates a test suite from all the test cases it contains. There is no upper limit for how many test cases there can be, but it is recommended to have less than ten, unless the data-driven approach is used, where one test case consists of only one high-level keyword.
The following settings in the Setting table can be used to customize the test suite:
Documentation
Used for specifying a test suite documentation
Metadata
Used for setting free test suite metadata as name-value pairs.
Suite Setup, Suite Teardown
Specify suite setup and teardown. Have also synonyms Suite Precondition and Suite Postcondition, respectively.
Note
All setting names can optionally include a colon at the end, for example Documentation:. This can make reading the settings easier especially when using the plain text format. This is a new feature in Robot Framework 2.5.5.
2.3.2 Test suite directories
Test case files can be organized into directories, and these directories create higher-level test suites. A test suite created from a directory cannot have any test cases directly, but it contains other test suites with test cases, instead. These directories can then be placed into other directories creating an even higher-level suite. There are no limits for the structure, so test cases can be organized as needed.
When a test directory is executed, the files and directories it contains are processed recursively as follows:
- Files and directories with names starting with a dot (.) or an underscore (_) are ignored.
- Directories with the name CVS are ignored (case-sensitive).
- Files not having one of the recognized extensions (.html, .xhtml, .htm, .tsv, .txt, .rst, or .rest) are ignored (case-insensitive).
- Other files and directories are processed.
If a file or directory that is processed does not contain any test cases, it is silently ignored (a message is written to the syslog) and the processing continues.
Warning on invalid files
Normally files that do not have a valid test case table are silently ignored with a message written to the syslog. As of Robot Framework 2.5.5 it is possible to use a command line option --warnonskippedfiles, which turns the message into a warning shown in test execution errors.
Initialization files
A test suite created from a directory can have similar settings as a suite created from a test case file. Because a directory alone cannot have that kind of information, it must be placed into a special test suite initialization file. An initialization file name must always be of the format _init.ext, where the extension must be one of the supported file formats (for example, _init.robot or _init.html_). The name format is borrowed from Python, where files named in this manner denote that a directory is a module.
Initialization files have the same structure and syntax as test case files, except that they cannot have test case tables and not all settings are supported. Variables and keywords created or imported in initialization files are not available in the lower level test suites. If you need to share variables or keywords, you can put them into resource files that can be imported both by initialization and test case files.
The main usage for initialization files is specifying test suite related settings similarly as in test case files, but setting some test case related settings is also possible. How to use different settings in the initialization files is explained below.
Documentation, Metadata, Suite Setup, Suite Teardown
These test suite specific settings work the same way as in test case files.
Force Tags
Specified tags are unconditionally set to all test cases in all test case files this directory contains directly or recursively.
Test Setup, Test Teardown, Test Timeout
Set the default value for test setup/teardown or test timeout to all test cases this directory contains. Can be overridden on lower level. Support for defining test timeout in initialization files was added in Robot Framework 2.7.
Default Tags, Test Template
Not supported in initialization files.
| An example test suite initialization file | ||
|---|---|---|
| Setting | Value | Value |
| Documentation | Example suite | |
| Suite Setup | Do Something | ${MESSAGE} |
| Force Tags | example | |
| Library | SomeLibrary |
| Variable | Value | Value |
|---|---|---|
| ${MESSAGE} | Hello, world! |
| Keyword | Action | Argument | Argument |
|---|---|---|---|
| Do Something | [Arguments] | ${arg} | |
| Some Keyword | ${arg} | ||
| Another Keyword |
2.3.3 Test suite name and documentation
The test suite name is constructed from the file or directory name. The name is created so that the extension is ignored, possible underscores are replaced with spaces, and names fully in lower case are title cased. For example, some_tests.html becomes Some Tests and My_test_directory_becomes _My test directory.
Note
The rules for creating test suite names changed slightly in Robot Framework 2.5.
The file or directory name can contain a prefix to control the execution order of the suites. The prefix is separated from the base name by two underscores and, when constructing the actual test suite name, both the prefix and underscores are removed. For example files 01__some_tests.txt and02__more_tests.txt create test suites Some Tests and More Tests, respectively, and the former is executed before the latter.
The documentation for a test suite is set using the Documentation setting in the Setting table. It can be used in test case files or, with higher-level suites, in test suite initialization files. Test suite documentation has exactly the same characteristics regarding to where it is shown and how it can be created as test case documentation.
| Test suite documentation example | |||
|---|---|---|---|
| Setting | Value | Value | Value |
| Documentation | An example test suite | documentation with | some formatting. |
| ... | See test documentation | for more documentation | examples. |
Both the name and documentation of the top-level test suite can be overridden in test execution. This can be done with the command line options --name and --doc, respectively, as explained in section Setting metadata.
2.3.4 Free test suite metadata
Test suites can also have other metadata than the documentation. This metadata is defined in the Setting table using the Metadata setting. Metadata set in this manner is shown in test reports and logs.
The name and value for the metadata are located in the columns following Metadata. The value is handled similarly as documentation, which means that it can be split into several cells (joined together with spaces) or into several rows (joined together with newlines), simple HTML formattingworks and even variables can be used.
| Metadata examples | ||||
|---|---|---|---|---|
| Setting | Value | Value | Value | Value |
| Metadata | Version | 2.0 | ||
| Metadata | More Info | For more information | about Robot Framework | see http://robotframework.org |
| Metadata | Executed At | ${HOST} |
For top-level test suites, it is possible to set metadata also with the --metadata command line option. This is discussed in more detail in section Setting metadata.
Prior to Robot Framework 2.5 the free metadata was specified with syntax like Meta: <name>, where <name> was the metadata name and the value was defined in subsequent column. Robot Framework 2.5 still supports this old format but it will be deprecated in the future.
2.3.5 Suite setup and teardown
Not only test cases but also test suites can have a setup and a teardown. A suite setup is executed before running any of the suite's test cases or child test suites, and a test teardown is executed after them. All test suites can have a setup and a teardown; with suites created from a directory they must be specified in a test suite initialization file.
Similarly as with test cases, a suite setup and teardown are keywords that may take arguments. They are defined in the Setting table with Suite Setup and Suite Teardown settings, respectively. They also have similar synonyms, Suite Precondition and Suite Postcondition, as a test case setup and teardown have. Keyword names and possible arguments are located in the columns after the setting name.
If a suite setup fails, all test cases in it and its child test suites are immediately assigned a fail status and they are not actually executed. This makes suite setups ideal for checking preconditions that must be met before running test cases is possible.
A suite teardown is normally used for cleaning up after all the test cases have been executed. It is executed even if the setup of the same suite fails. If the suite teardown fails, all test cases in the suite are marked failed, regardless of their original execution status. Starting from Robot Framework 2.5, all the keywords in suite teardowns are executed even if one of them fails.
The name of the keyword to be executed as a setup or a teardown can be a variable. This facilitates having different setups or teardowns in different environments by giving the keyword name as a variable from the command line.