8000 UG: Enhance --test and --suite documentation. · robotframework/robotframework@adf1a86 · GitHub
[go: up one dir, main page]
Skip to content

Commit adf1a86

Browse files
pekkaklarckpekkaklarck
committed
UG: Enhance --test and --suite documentation.
- Explain their usage and how they work together better. - Explain that new Name is not compatible with --suite (#4583) - Explain how --suite is going to be changed (incl. fixing of the above issue) in RF 7.0 (#4720, #4721, #4688)
1 parent 1ad6c52 commit adf1a86

File tree

3 files changed

+91
-37
lines changed

3 files changed

+91
-37
lines changed

‎doc/userguide/src/Appendices/CommandLineOptions.rst

Lines changed: 2 additions & 3 deletions
Original file line numberDiff line numberDiff line change
@@ -148,15 +148,14 @@ Command line options for post-processing outputs
148148
-h, --help Prints `usage instructions`_.
149149
--version Prints the `version information`_.
150150

151-
152151
.. _generic automation: `Task execution`_
153152
.. _Parse only these files: `Selecting files to parse`_
154153
.. _Sets the name: `Setting the name`_
155154
.. _Sets the documentation: `Setting the documentation`_
156155
.. _Sets free metadata: `Setting free metadata`_
157156
.. _Sets the tag(s): `Setting tags`_
158-
.. _Selects the test cases by name: `By test suite and test case names`_
159-
.. _Selects the test suites: `Selects the test cases by name`_
157+
.. _Selects the test cases by name: `By test names`_
158+
.. _Selects the test suites: `By suite names`_
160159
.. _Selects failed test suites: `Re-executing failed test suites`_
161160
.. _Selects failed tests: `Re-executing failed test cases`_
162161
.. _Selects the test cases: `By tag names`_

‎doc/userguide/src/CreatingTestData/CreatingTestSuites.rst

Lines changed: 6 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -156,6 +156,12 @@ to a suite by using the :setting:`Name` setting in the Setting section:
156156
*** Settings ***
157157
Name Custom suite name
158158

159+
.. note:: The :setting:`Name` setting is not compatible with the :option:`--suite`
160+
option that can be used to select tests `by suite names`_. This `will
161+
fixed`__ in Robot Framework 7.0.
162+
163+
__ https://github.com/robotframework/robotframework/issues/4688
164+
159165
Suite documentation
160166
-------------------
161167

‎doc/userguide/src/ExecutingTestCases/ConfiguringExecution.rst

Lines changed: 83 additions & 34 deletions
Original file line numberDiff line numberDiff line change
@@ -54,49 +54,98 @@ Robot Framework offers several command line options for selecting
5454
which test cases to execute. The same options work also when `executing
5555
tasks`_ and when post-processing outputs with Rebot_.
5656

57-
By test suite and test case names
58-
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
59-
60-
Test suites and test cases can be selected by their names with the command
61-
line options :option:`--suite (-s)` and :option:`--test (-t)`,
62-
respectively. Both of these options can be used several times to
63-
select several test suites or cases. Arguments to these options are
64-
case- and space-insensitive, and there can also be `simple
65-
patterns`_ matching multiple names. If both the :option:`--suite` and
66-
:option:`--test` options are used, only test cases in matching suites
67-
with matching names are selected.
57+
By test names
58+
~~~~~~~~~~~~~
6859

69-
::
60+
The easiest way to select only some tests to be run is using the
61+
:option:`--test (-t)` option. As the name implies, it can be used for
62+
selecting tests by their names. Given names are case, space and underscore
63+
insensitive and they also support `simple patterns`_. The option can be
64+
used multiple times to match multiple tests::
7065

71-
--test Example
72-
--test mytest --test yourtest
73-
--test example*
74-
--test mysuite.mytest
75-
--test *.suite.mytest
76-
--suite example-??
77-
--suite mysuite --test mytest --test your*
78-
79-
Using the :option:`--suite` option is more or less the same as executing only
80-
the appropriate test case file or directory. One major benefit is the
81-
possibility to select the suite based on its parent suite. The syntax
82-
for this is specifying both the parent and child suite names separated
83-
with a dot. In this case, the possible setup and teardown of the parent
84-
suite are executed.
66+
--test Example # Match only tests with name 'Example'.
67+
--test example* # Match tests starting with 'example'.
68+
--test first --test second # Match tests with name 'first' or 'second'.
8569

86-
::
70+
To pinpoint a test more precisely, it is possible to prefix the test name
71+
with a suite name::
72+
73+
--test mysuite.mytest # Match test 'mytest' in suite 'mysuite'.
74+
--test root.sub.test # Match test 'test' in suite 'sub' in suite 'root'.
75+
--test *.sub.test # Match test 'test' in suite 'sub' anywhere.
8776

88-
--suite parent.child
89-
--suite myhouse.myhousemusic --test jack*
77+
Notice that when the given name includes a suite name, it must match the whole
78+
suite name starting from the root suite. Using a wildcard as in the last example
79+
above allows matching suites anywhere.
9080

91-
Selecting individual test cases with the :option:`--test` option is very
92-
practical when creating test cases, but quite limited when running tests
93-
automatically. The :option:`--suite` option can be useful in that
94-
case, but in general, selecting test cases by tag names is more
95-
flexible.
81+
Using the :option:`--test` option is convenient when only a few tests needs
82+
to be selected. A common use case is running just the test that is currently
83+
being worked on. If a bigger number of tests needs to be selected,
84+
it is typically easier to select them `by suite names`_ or `by tag names`_.
9685

9786
When `executing tasks`_, it is possible to use the :option:`--task` option
9887
as an alias for :option:`--test`.
9988

89+
By suite names
90+
~~~~~~~~~~~~~~
91+
92+
Tests can be selected also by suite names with the :option:`--suite (-s)`
93+
option that selects all tests in matching suites. Similarly
94+
as with :option:`--test`, given names are case, space and underscore
95+
insensitive and support `simple patterns`_. To pinpoint a suite
96+
more precisely, it is possible to prefix the name with the parent suite
97+
name::
98+
99+
--suite Example # Match only suites with name 'Example'.
100+
--suite example* # Match suites starting with 'example'.
101+
--suite first --suite second # Match suites with name 'first' or 'second'.
102+
--suite parent.child # Match suite 'child' in suite 'parent'.
103+
104+
Unlike with :option:`--test`, the name does not need to match the whole
105+
suite name, starting from the root suite, when the name contains a parent
106+
suite name. This behavior `will be changed`__ in the future and should not be relied
107+
upon. It is recommended to use the full name like `--suite root.parent.child`
108+
or `--suite *.parent.child`.
109+
110+
If both :option:`--suite` and :option:`--test` options are used, only the
111+
specified tests in specified suites are selected::
112+
113+
--suite mysuite --test mytest # Match test 'mytest' if its inside suite 'mysuite'.
114+
115+
Also this behavior `is likely to change`__ in the future and the above changed to mean
116+
selecting all tests in suite `mysuite` in addition to all tests with name `mytest`.
117+
A more reliable way to select a test in a suite is using `--test *.mysuite.mytest`
118+
or `--test *.mysuite.*.mytest` depending on should the test be directly inside
119+
the suite or not.
120+
121+
Using the :option:`--suite` option is more or less the same as executing
122+
the appropriate suite file or directory directly. The main difference is
123+
that if a file or directory is run directly, possible suite setups and teardowns
124+
on higher level are not executed::
125+
126+
# Root suite is 'Tests' and its possible setup and teardown are run.
127+
robot --suite example path/to/tests
128+
129+
# Root suite is 'Example' and possible higher level setups and teardowns are ignored.
130+
robot path/to/tests/example.robot
131+
132+
When using the :option:`--suite` option, Robot Framework does not parse
133+
files that do not match the given suite name. For example, when using
134+
`--suite example`, only files that have a name :file:`example.robot` or are in
135+
a directory :file:`example` are parsed. This is done for performance reasons
136+
to avoid the parsing overhead with larger directory structures. Unfortunately
137+
this approach does not work well with the new :setting:`Name` setting that can
138+
be used for setting a custom `suite name`_. In practice the new setting and
139+
the :option:`--suite` option are incompatible. This will be changed in Robot
140+
Framework 7.0 so that `files are not excluded`__ when using the :option:`--suite`
141+
option. The plan is to add an explicit option for `selecting files to parse`__
142+
before that.
143+
144+
__ https://github.com/robotframework/robotframework/issues/4720
145+
__ https://github.com/robotframework/robotframework/issues/4721
146+
__ https://github.com/robotframework/robotframework/issues/4688
147+
__ https://github.com/robotframework/robotframework/issues/4687
148+
100149
By tag names
101150
~~~~~~~~~~~~
102151

0 commit comments

Comments
 (0)
0