8000 Change behavior of `--test` and `--include` so that they are cumulative · Issue #4721 · robotframework/robotframework · GitHub
[go: up one dir, main page]
Skip to content

Change behavior of --test and --include so that they are cumulative #4721

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom
Closed
pekkaklarck opened this issue Apr 5, 2023 · 3 comments
Closed

Change behavior of --test and --include so that they are cumulative #4721

pekkaklarck opened 8000 this issue Apr 5, 2023 · 3 comments
Assignees
@pekkaklarck
Labels
alpha 1 backwards incompatible effort: small enhancement priority: medium
Milestone
v7.0

Comments

@pekkaklarck
Copy link
Member
pekkaklarck commented Apr 5, 2023 •
edited
Loading

Currently when you use --suite x --test y --include z, a test is selected only if it is in suite x, has name y and contains a tag z. This is rarely if ever useful. It would typically be enough to just use --test y to select that test.

On the other hand, it would more useful to be able to use --suite x --test y so that all tests in suite x are selected in addition to test y. Similarly, --include x --test y should mean selecting all tests containing tag x in addition to test y.

This change is backwards incompatible, but because the current behavior is pretty strange it's unlikely there are lot of problems. Changing the behavior in Robot Framework 7.0 ought to thus be fine. The plan is to change the behavior with --suite also otherwise and making all these changes in the same release would be good.

Although these options should have a cumulative effect, we probably should interpret --exclude so that tests containing the specified tag aren't selected. That would preserve its behavior when used with other options, most importantly with --include.

UPDATE: We decided to keep the old --suite behavior. See the comment below for more details. Notice also that --suite not affecting which files are parsed was implemented already in RF 6.1 (#4688).

UPDATE: This change will be reverted in RF 7.0.1 due to problems it caused with Pabot and --rerunfailed. See comments and other notes below for more information.
@pekkaklarck pekkaklarck added this to the v7.0 milestone Apr 5, 2023
pekkaklarck added a commit that referenced this issue Apr 5, 2023
@pekkaklarck
UG: Enhance --test and --suite documentation.
adf1a86 
- Explain their usage and how they work together better.
- Explain that new Name is not compatible with --suite (#4583)
- Explain how --suite is going to be changed (incl. fixing of the
  above issue) in RF 7.0 (#4720, #4721, #4688)
@pekkaklarck pekkaklarck self-assigned this Sep 11, 2023
@pekkaklarck pekkaklarck changed the title Change behavior of using --suite, --test and --include together so that they are cumulative Change behavior of --test and --include so that they are cumulative Sep 12, 2023
@pekkaklarck
Copy link < 8000 div aria-live="polite" aria-atomic="true" class="sr-only" data-clipboard-copy-feedback>
Member Author

After discussion on Slack, we decided to keep the old --suite behavior. The main motivation is that robot --suite x root is often used instead of robot root/x.robot to get suite setup and teardown on the root level executed. In such usage robot --suite x --include y root is handy to further filter tests to be run under the x suite. Changing this behavior would be pretty badly backwards incompatible and, even more importantly, would remove a useful functionality.

We will change --test and --include so that they are cumulative, though. In practice --test x --include y will in the future select tests that have name x or contain a tag y. This is useful if you want to run one or few extra named tests in addition to ones selected based on tags.

Earlier --test x --include y meant that only tests with name x and a tag y were selected. This behavior didn't make too much sense, because typically just --test x is enough. If you have multiple tests with name x, it's easier to use the full name like --test root.intermediate.x to pinpoint which one to select instead of using tags. Also this is a backwards incompatible change, but the benefits are bigger than possible problems.

@roland-ruedenauer roland-ruedenauer mentioned this issue Jan 27, 2024
Closed
@pekkaklarck pekkaklarck mentioned this issue Feb 4, 2024
Closed
@pekkaklarck
Copy link
Member Author

This change caused annoying problems with Pabot and with --rerunfailed. Instead of trying to workaround these problems, it's better to revert this change. There weren't that much benefits with it in the first place, the change was done mainly for consistency.

@pekkaklarck
Copy link
Member Author

The aforementioned change will happen already in RF 7.0.1. See #5023 for more information.

pekkaklarck added a commit that referenced this issue May 29, 2024
@pekkaklarck
Make --test and --include non-cumulative again.
6c6476d 
These options were made cumulative in RF 7.0 (#4721), but that
behavior caused backwards incompatibility problems with Pabot and with
--rerunfailed. The benefits of the change weren't that big and the
behavior was changed back to original in RF 7.0.1.

Fixes #5023.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees

@pekkaklarck pekkaklarck

Labels
alpha 1 backwards incompatible effort: small enhancement priority: medium
Projects
None yet
Milestone
v7.0
Development

No branches or pull requests
1 participant
@pekkaklarck
0