2.2 Creating test cases

This section describes the overall test case syntax. Organizing test cases into test suites using test case files and test suite directories is discussed in the next section.

• 2.2.1 Test case syntax
  • Basic syntax
  • Settings in the Test Case table
  • Test case related settings in the Setting table
• 2.2.2 Using arguments
  • Mandatory arguments
  • Default values
  • Variable number of arguments
  • Named arguments
  • Free keyword arguments
  • Arguments embedded to keyword names
• 2.2.3 Failures
  • When test case fails
  • Error messages
• 2.2.4 Test case name and documentation
• 2.2.5 Tagging test cases
• 2.2.6 Test setup and teardown
• 2.2.7 Test templates
  • Basic usage
  • Templates with embedded arguments
  • Templates with for loops
• 2.2.8 Different test case styles
  • Keyword-driven style
  • Data-driven style
  • Behavior-driven style

2.2.1 Test case syntax

Basic syntax

Test cases are constructed in test case tables from the available keywords. Keywords can be imported from test libraries or resource files, or created in the keyword table of the test case file itself.

The first column in the test case table contains test case names. A test case starts from the row with something in this column and continues to the next test case name or to the end of the table. It is an error to have something between the table headers and the first test.

The second column normally has keyword names. An exception to this rule is setting variables from keyword return values, when the second and possibly also the subsequent columns contain variable names and a keyword name is located after them. In either case, columns after the keyword name contain possible arguments to the specified keyword.

Settings in the Test Case table

Test cases can also have their own settings. Setting names are always in the second column, where keywords normally are, and their values are in the subsequent columns. Setting names have square brackets around them to distinguish them from keywords. The available settings are listed below and explained later in this section.

[Documentation]

Used for specifying a test case documentation.

[Tags]

Used for tagging test cases.

[Setup], [Teardown]

Specify test setup and teardown. Have also synonyms [Precondition] and [Postcondition], respectively.

[Template]

Specifies the template keyword to use. The test itself will contain only data to use as arguments to that keyword.

[Timeout]

Used for setting a test case timeout. Timeouts are discussed in their own section.

Test case related settings in the Setting table

The Setting table can have the following test case related settings. These settings are mainly default values for the test case specific settings listed earlier.

Force Tags, Default Tags

The forced and default values for tags.

Test Setup, Test Teardown

The default values for test setup and teardown. Have also synonyms Test Precondition and Test Postcondition, respectively.

Test Template

The default template keyword to use.

Test Timeout

The default value for test case timeout. Timeouts are discussed in their own section.

2.2.2 Using arguments

The earlier examples have already demonstrated keywords taking different arguments, and this section discusses this important functionality more thoroughly. How to actually implement user keywords and library keywords with different arguments is discussed in separate sections.

Keywords can accept zero or more arguments, and some arguments may have default values. What arguments a keyword accepts depends on its implementation, and typically the best place to search this information is keyword's documentation. In the examples in this section the documentation is expected to be generated using the libdoc tool, but the same information is available on documentation generated by generic documentation tools such as javadoc.

Mandatory arguments

Most keywords have a certain number of arguments that must always be given. In the keyword documentation this is denoted by specifying the argument names separated with a comma like first, second, third. The argument names actually do not matter in this case, except that they should explain what the argument does, but it is important to have exactly the same number of arguments as specified in the documentation. Using too few or too many arguments will result in an error.

The test below uses keywords Create Directory and Copy File from the OperatingSystem library. Their arguments are specified as path and source, destination, which means that they take one and two arguments, respectively. The last keyword, No Operation from BuiltIn, takes no arguments.

Default values

Arguments often have default values which can either be given or not. In the documentation the default value is typically separated from the argument name with an equal sign like name=default value, but with keywords implemented using Java there may be multiple implementations of the same keyword with different arguments instead. It is possible that all the arguments have default values, but there cannot be any positional arguments after arguments with default values.

Using default values is illustrated by the example below that uses Create File keyword which has arguments path, content=, encoding=UTF-8. Trying to use it without any arguments or more than three arguments would not work.

Variable number of arguments

It is also possible to create keywords that accept any number of arguments. These arguments can be combined with mandatory arguments and arguments with default values, but the so called varargs are always the last ones. In the documentation they typically have an asterisk before the argument name like *varargs , but there are again differences with Java libraries.

Remove Files and Join Paths keywords used in the example below have arguments paths and base, parts, respectively. The former can be used with any number of arguments, but the latter requires at least one argument.

Named arguments

The named argument syntax makes using arguments with default values more flexible, and allows explicitly labeling what a certain argument value means. Technically named arguments work exactly like keyword arguments in Python.

Basic syntax

It is possible to name an argument given to a keyword by prefixing the value with the name of the argument like arg=value. This is especially useful when multiple arguments have default values, as it is possible to name only some the arguments and let others use their defaults. For example, if a keyword accepts arguments arg1=a, arg2=b, arg3=c, and it is called with one argument arg3=override, arguments arg1 and arg2 get their default values, but arg3 gets value override. If this sounds complicated, the named arguments example below hopefully makes it more clear.

The named argument syntax is both case and space sensitive. The former means that if you have an argument arg, you must use it like arg=value, and neither Arg=value nor ARG=value works. The latter means that spaces are not allowed before the = sign, and possible spaces after it are considered part of the given value.

When the named argument syntax is used with user keywords, the argument names must be given without the ${} decoration. For example, user keyword with arguments ${arg1}=first, ${arg2}=second must be used like arg2=override.

Using normal positional arguments after named arguments like, for example, | Keyword | arg=value | positional |, does not work. Starting from Robot Framework 2.8 this causes an explicit error. The relative order of the named arguments does not matter.

Note

Prior to Robot Framework 2.8 it was not possible to name arguments that did not have a default value.

Named arguments with variables

It is possible to use variables in both named argument names and values. If the value is a single scalar variable, it is passed to the keyword as-is. This allows using any objects, not only strings, as values also when using the named argument syntax. For example, calling a keyword likearg=${object} will pass the variable ${object} to the keyword without converting it to a string.

If variables are used in named argument names, variables are resolved before matching them against argument names. This is a new feature in Robot Framework 2.8.6.

The named argument syntax requires the equal sign to be written literally in the keyword call. This means that if a variable has value like foo=bar, it can never trigger the named argument syntax. This is important to remember especially when wrapping keywords into other keywords. If, for example, a keyword takes a variable number of arguments like @{args} and passes all of them to another keyword using the same @{args} syntax, the values are not recognized as named. See the example below:

Escaping named arguments syntax

The named argument syntax is used only when the part of the argument before the equal sign matches one of the keyword's arguments. It is possible that there is a positional argument with a literal value like foo=quux, and also an unrelated argument with name foo. In this case the argument fooeither incorrectly gets the value quux or, more likely, there is a syntax error.

In these rare cases where there are accidental matches, it is possible to use the backslash character to escape the syntax like foo\=quux. Now the argument will get a literal value foo=quux. Note that escaping is not needed if there are no arguments with name foo, but because it makes the situation more explicit, it may nevertheless be a good idea.

Where named arguments are supported

As already explained, the named argument syntax works with keywords. In addition to that, it also works when taking test libraries into use.

Naming arguments is supported by user keywords and by most test libraries. The only exception are Java based libraries that use the static library API. Library documentation generated with Libdoc has a note does the library support named arguments or not.

Note

Prior to Robot Framework 2.8 named argument syntax did not work with test libraries using the dynamic library API.

Named arguments example

The following example demonstrates using the named arguments syntax with library keywords, user keywords, and when importing the Telnet test library.

Free keyword arguments

Robot Framework 2.8 added support for Python style free keyword arguments (**kwargs). What this means is that keywords can receive all arguments at the end of the keyword call that use the name=value syntax, and do not match any other arguments, as kwargs.

Free keyword arguments support variables similarly as named arguments. In practice that means that variables can be used both in names and values, but the escape sign must always be visible literally. For example, both foo=${bar} and ${foo}=${bar} are valid, as long as the variables that are used exist. An extra limitation is that free keyword argument names must always be strings. Support for variables in names is a new feature in Robot Framework 2.8.6, prior to that possible variables were left un-resolved.

Initially free keyword arguments only worked with Python based libraries, but Robot Framework 2.8.2 extended the support to the dynamic library API and Robot Framework 2.8.3 extended it further to Java based libraries and to the remote library interface. In other words, all libraries nowadays support kwargs. Unfortunately user keywords no not support them yet, but that support is planned for Robot Framework 2.9.

For a real life example of using kwargs, let's take a look at Run Process keyword in the Process library. It has a signature command, arguments, *configuration, which means that it takes the command to execute, its arguments as variable number of arguments, and finally optional configuration parameters as free keyword arguments configuration.

As the above example illustrates, using variables with free keyword arguments works exactly like when using the named argument syntax.

See Free keyword arguments (**kwargs) section under Creating test libraries for more information about using the kwargs syntax in your custom test libraries.

Arguments embedded to keyword names

A totally different approach to specify arguments is embedding them into keyword names. This syntax is, at least currently, only supported by user keywords.

2.2.3 Failures

When test case fails

A test case fails if any of the keyword it uses fails. Normally this means that execution of that test case is stopped, possible test teardown is executed, and then execution continues from the next test case. It is also possible to use special continuable failures if stopping test execution is not desired.

Error messages

The error message assigned to a failed test case is got directly from the failed keyword. Often the error message is created by the keyword itself, but some keywords allow configuring them.

In some circumstances, for example when continuable failures are used, a test case can fail multiple times. In that case the final error message is got by combining the individual errors. Very long error messages are automatically cut from the middle to keep reports easier to read. Full error messages are always visible in log file as a message of the failed keyword.

By default error messages are normal text, but starting from Robot Framework 2.8 they can contain HTML formatting. This is enabled by starting the error message with marker string HTML. This marker will be removed from the final error message shown in reports and logs. Using HTML in a custom message is shown in the second example below.

2.2.4 Test case name and documentation

The test case name comes directly from the Test Case table: it is exactly what is entered into the test case column. Test cases in one test suite should have unique names. Pertaining to this, you can also use the automatic variable ${TEST_NAME} within the test itself to refer to the test name. It is available whenever a test is being executed, including all user keywords, as well as the test setup and the test teardown.

The [Documentation] setting allows you to set a free documentation for a test case. That text is shown in the command line output, as well as the resulting test logs and test reports.

If the documentation is long, it can be split into several cells that are catenated together with spaces. It is possible to use simple HTML formatting and variables can be used to make the documentation dynamic. Starting from Robot Framework 2.7, if documentation is split in multiple lines, the lines themselves are catenated using newlines. Newlines are not added if the line already ends with a newline or it ends with an escaping backslash.

It is important that test cases have clear and descriptive names, and in that case they normally do not need any documentation. If the logic of the test case needs documenting, it is often a sign that keywords in the test case need better names and they are to be enhanced, instead of adding extra documentation. Finally, metadata, such as the environment and user information in the last example above, is often better specified using tags.

2.2.5 Tagging test cases

$ pybot --outputdir /var/www/html --include runCase -e notrun test_telnet.robot

Using tags in Robot Framework is a simple, yet powerful mechanism for classifying test cases. Tags are free text and they can be used at least for the following purposes:

• Tags are shown in test reports, logs and, of course, in the test data, so they provide metadata to test cases.
• Statistics about test cases (total, passed, failed are automatically collected based on tags).
• With tags, you can include or exclude test cases to be executed.
• With tags, you can specify which test cases are considered critical.

In this section it is only explained how to set tags for test cases, and different ways to do it are listed below. These approaches can naturally be used together.

Force Tags in the Setting table

All test cases in a test case file with this setting always get specified tags. If it is used in the test suite initialization file, all test cases in sub test suites get these tags.

Default Tags in the Setting table

Test cases that do not have a [Tags] setting of their own get these tags. Starting from Robot Framework version 2.5 default tags are no longer supported in test suite initialization files.

[Tags] in the Test Case table

A test case always gets these tags. Additionally, it does not get the possible tags specified with Default Tags, so it is possible to override the Default Tags by using empty value. Starting from Robot Framework 2.5.6, is also possible to use value NONE to override default tags.

--settag command line option

All executed test cases get tags set with this option in addition to tags they got elsewhere.

Set Tags, Remove Tags, Fail and Pass Execution keywords

These BuiltIn keywords can be used to manipulate tags dynamically during the test execution.

Tags are free text, but they are normalized so that they are converted to lowercase and all spaces are removed. If a test case gets the same tag several times, other occurrences than the first one are removed. Tags can be created using variables, assuming that those variables exist.

2.2.6 Test setup and teardown

Robot Framework has similar test setup and teardown functionality as many other test automation frameworks. In short, a test setup is something that is executed before a test case, and a test teardown is executed after a test case. In Robot Framework setups and teardowns are just normal keywords with possible arguments.

Setup and teardown are always a single keyword. If they need to take care of multiple separate tasks, it is possible to create higher-level user keywords for that purpose. An alternative solution is executing multiple keywords using the BuiltIn keyword Run Keywords that was added in Robot Framework 2.5.

The test teardown is special in two ways. First of all, it is executed also when a test case fails, so it can be used for clean-up activities that must be done regardless of the test case status. Starting from Robot Framework 2.5, all the keywords in the teardown are also executed even if one of them fails. This continue on failure functionality can be used also with normal keywords, but inside teardowns it is on by default.

The easiest way to specify a setup or a teardown for test cases in a test case file is using the Test Setup and Test Teardown settings in the Setting table. Individual test cases can also have their own setup or teardown. They are defined with the [Setup] or [Teardown] settings in the test case table and they override possible Test Setup and Test Teardown settings. Having no keyword after a [Setup] or [Teardown] setting means having no setup or teardown. Starting from Robot Framework 2.5.6, it is also possible to use value NONE to indicate that a test has no setup/teardown.

Often when creating use-case-like test cases, the terms precondition and postcondition are preferred over the terms setup and teardown. Robot Framework supports also this terminology, so that a precondition is a synonym to a setup and a postcondition to a teardown.

The name of the keyword to be executed as a setup or a teardown can be a variable. This facilitates having different setups or teardowns in different environments by giving the keyword name as a variable from the command line.

Note

Test suites can have a setup and teardown of their own. A suite setup is executed before any test cases or sub test suites in that test suite, and similarly a suite teardown is executed after them.

2.2.7 Test templates

Test templates convert normal keyword-driven test cases into data-driven tests. Whereas the body of a keyword-driven test case is constructed from keywords and their possible arguments, test cases with template contain only the arguments for the template keyword. Instead of repeating the same keyword multiple times per test and/or with all tests in a file, it is possible to use it only per test or just once per file.

Template keywords can accept both normal positional and named arguments, as well as arguments embedded to the keyword name. Unlike with other settings, it is not possible to define a template using a variable.

在这里我们介绍robotFramework一个很有用的功能——测试模板。测试用例中的关键字，例如：执行时，只会执行一次。可有时候我们需要连续执行这个关键字。如果我们连续写这个关键字，用例会显得很臃肿。robotFramework提供了一个解决方案，在关键字前面加上一个[Template]，表示这个用例是一个“测试模板”。例如：执行以上用例：这个用例里，“log”关键字被连续执行了三次。与普通的关键字写法不同，带有[Template]的关键字的参数不能和关键字同行。每带有一行参数，关键字就需要执行一次。熟悉QTP的朋友是不是觉得似曾相识啊？ 需要特别注意的是，一个用例中只能使用一个[Template]，不能多次使用。也不能带[Template]的关键字和普通的关键字混合使用。 利用“[Template]”，robotFramework可以轻松的实现数据驱动。

Basic usage

How a keyword accepting normal positional arguments can be used as a template is illustrated by the following example test cases. These two tests are functionally fully identical.

* Test Cases **

Normal test case

Example keyword first argument second argument

Templated test case

[Template] Example keyword

first argument second argument

As the example illustrates, it is possible to specify the template for an individual test case using the [Template] setting. An alternative approach is using the Test Template setting in the Setting table, in which case the template is applied for all test cases in that test case file. The [Template]_setting overrides the possible template set in the Setting table, and an empty value for [Template] means that the test has no template even when _Test Template is used. Starting from Robot Framework 2.5.6, it is also possible to use value NONE to indicate that a test has no template.

If a templated test case has multiple data rows in its body, the template is applied for all the rows one by one. This means that the same keyword is executed multiple times, once with data on each row. Templated tests are also special so that all the rounds are executed even if one or more of them fails. It is possible to use this kind of continue on failure mode with normal tests too, but with the templated tests the mode is on automatically.

* Settings *

Test Template Example keyword

* Test Cases *

Templated test case

first round 1 first round 2

second round 1 second round 2

third round 1 third round 2

Using arguments with default values or varargs, as well as using named arguments and free keyword arguments, work with templates exactly like they work otherwise. Using variables in arguments is also supported normally.

Templates with embedded arguments

Starting from Robot Framework 2.8.2, templates support a variation of the embedded argument syntax. With templates this syntax works so that if the template keyword has variables in its name, they are considered placeholders for arguments and replaced with the actual arguments used with the template. The resulting keyword is then used without positional arguments. This is best illustrated with an example:

* Test Case *

Normal test case with embedded arguments

The result of 1 + 1 should be 2

The result of 1 + 2 should be 3

Template with embedded arguments

[Template] The result of ${calculation} should be ${expected}

1 + 1 2

1 + 2 3

* Keywords *

The result of ${calculation} should be ${expected}

${result} = Calculate ${calculation}

Should Be Equal ${result} ${expected}

When embedded arguments are used with templates, the number of arguments in the template keyword name must match the number of arguments it is used with. The argument names do not need to match the arguments of the original keyword, though, and it is also possible to use different arguments altogether:

* Test Case *

Different argument names

[Template] The result of ${foo} should be ${bar}

1 + 1 2

1 + 2 3

Only some arguments

[Template] The result of ${calculation} should be 3

1 + 2

4 - 1

New arguments

[Template] The ${meaning} of ${life} should be 42

result 21 * 2

The main benefit of using embedded arguments with templates is that argument names are specified explicitly. When using normal arguments, the same effect can be achieved by naming the columns that contain arguments. This is illustrated by the data-driven style example in the next section.

Templates with for loops

If templates are used with for loops, the template is applied for all the steps inside the loop. The continue on failure mode is in use also in this case, which means that all the steps are executed with all the looped elements even if there are failures.

2.2.8 Different test case styles

There are several different ways in which test cases may be written. Test cases that describe some kind of workflow may be written either in keyword-driven or behavior-driven style. Data-driven style can be used to test the same workflow with varying input data.

Keyword-driven style

Workflow tests, such as the Valid Login test described earlier, are constructed from several keywords and their possible arguments. Their normal structure is that first the system is taken into the initial state (Open Login Page in the Valid Login example), then something is done to the system (Input Name, Input Password, Submit Credentials), and finally it is verified that the system behaved as expected (Welcome Page Should Be Open).

Data-driven style

Another style to write test cases is the data-driven approach where test cases use only one higher-level keyword, normally created as a user keyword, that hides the actual test workflow. These tests are very useful when there is a need to test the same scenario with different input and/or output data. It would be possible to repeat the same keyword with every test, but the test template functionality allows specifying the keyword to use only once.

The above example has six separate tests, one for each invalid user/password combination, and the example below illustrates how to have only one test with all the combinations. When using test templates, all the rounds in a test are executed even if there are failures, so there is no real functional difference between these two styles. In the above example separate combinations are named so it is easier to see what they test, but having potentially large number of these tests may mess-up statistics. Which style to use depends on the context and personal preferences.

Tip

In both of the above examples, column headers have been changed to match the data. This is possible because on the first row other cells except the first one are ignored.

Behavior-driven style

It is also possible to write test cases as requirements that also non-technical project stakeholders must understand. These executable requirements are a corner stone of a process commonly called Acceptance Test Driven Development (ATDD) or Specification by Example.

One way to write these requirements/tests is Given-When-Then style popularized by Behavior Driven Development (BDD). When writing test cases in this style, the initial state is usually expressed with a keyword starting with word Given, the actions are described with keyword starting withWhen and the expectations with a keyword starting with Then. Keyword starting with And or But may be used if a step has more than one action.

Ignoring Given/When/Then/And/But prefixes

Test Cases Valid Login [Tags] run2 Given username is correct When password is correct And credential are submitted Then Welcome to Login

Prefixes Given, When, Then, And and But are dropped when matching keywords are searched, if no match with the full name is found. This works for both user keywords and library keywords. For example, Given login page is open in the above example can be implemented as user keyword either with or without the word Given. Ignoring prefixes also allows using the same keyword with different prefixes. For example Welcome page should be open could also used as And welcome page should be open.

Note

Ignoring But prefix is new in Robot Framework 2.8.7.

Embedding data to keywords

When writing concrete examples it is useful to be able pass actual data to keyword implementations. User keywords support this by allowing embedding arguments into keyword name.