8000 feat: Add filter support for TestAdapter by filzrev · Pull Request #2788 · dotnet/BenchmarkDotNet · GitHub
[go: up one dir, main page]
Skip to content
8000

feat: Add filter support for TestAdapter #2788

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom
Draft
wants to merge 2 commits into
base: master
Choose a base branch
from
Draft

feat: Add filter support for TestAdapter #2788

wants to merge 2 commits into from

Conversation

filzrev
Copy link
Contributor
@filzrev filzrev commented Jun 20, 2025 •
edited
Loading

This PR add --filter parameter support for dotnet test.
It intended to resolve #2767 issue (and some part of issue #2662)

What's changed in this PR.

1. Add benchmark filter logics
Add following files to support benchmark filtering.

  • TestCaseFilter.cs
  • LoggerHelper.cs

Note: These files are based on xUnit v2 TestAdapter Code (License: Apache 2.0)

2. Modify VSTestAdapter.cs
Add benchmark filter logics to DiscoverTests and RunTests

3. BenchmarkEnumerator.cs
Remove code for Debug configuration.
(By #2774 changes. It can load DLLs that are build with Debug configurations)

4. Disable TestTfmsInParallel MSBuild property
It's enabled by default on .NET 9 or later.
But it's not expected multiple benchmarks are executed in parallel.

5. Temporary comment out IntroVisualStudioProfiler benchmark
Because it cause errors when benchmarks are enumerated/evaluated twice. (#2758)

Remaining tasks

  • Benchmark progress is not shown when running benchmark with dotnet test by default
    Currently, it need to specify --logger:"console;verbosity=normal" explicitly
  • Need to revert IntroVisualStudioProfiler code when issue is resolved. ([Bug] InvalidOperationException thrown on some conditions when using BenchmarkDotNetDiagnosers #2758)
  • DisplayName filter is same as FullyQualifiedName (Currently there is no attribute for DisplayName)
  • Same log is outputted twice on console/OutputWindow. (By ConsoleLogger/VSTestLogger)
  • Enforce TestTfmsInParallel:false setting on BenchmarkDotNet.TestAdapter package's props.

What's Tested

Currently it's hard to write unit tests that use TestAdapter.
So it need to run tests manually.

TestExplorer

  1. All benchmarks(net462/net80) are displayed when using Release configuration.
  2. It can run selected test and multiple selected tests.
  3. When running multiple TFMs benchmarks. It's sequencially executed. (Not in parallel)

dotnet test

  1. Verify default behavior is not changed when run command without --filter parameter

    dotnet test -c Release -tl:off --no-build --framework net8.0 --logger:"console;verbosity=detailed" --list-tests
    dotnet test -c Release -tl:off --no-build --framework net8.0 --logger:"console;verbosity=detailed"

  2. Filter by partial query works as expected.

    dotnet test -c Release --filter IntroBasic -tl:off --no-build --framework net8.0 --logger:"console;verbosity=detailed" --list-tests
    dotnet test -c Release --filter IntroBasic -tl:off --no-build --framework net8.0 --logger:"console;verbosity=detailed"

  3. Filter by property value works as expected.

    dotnet test -c Release --filter FullyQualifiedName=BenchmarkDotNet.Samples.IntroBasic.Sleep -tl:off --no-build --framework net8.0 --logger:"console;verbosity=detailed" --list-tests
    dotnet test -c Release --filter DisplayName=BenchmarkDotNet.Samples.IntroBasic.Sleep -tl:off --no-build --framework net8.0 --logger:"console;verbosity=detailed" --list-tests

    dotnet test -c Release --filter FullyQualifiedName=BenchmarkDotNet.Samples.IntroBasic.Sleep -tl:off --no-build --framework net8.0 --logger:"console;verbosity=detailed"
    dotnet test -c Release --filter DisplayName=BenchmarkDotNet.Samples.IntroBasic.Sleep -tl:off --no-build --framework net8.0 --logger:"console;verbosity=detailed"

  4. Filter by Category works as expected

    dotnet test -c Release --filter Category=Fast -tl:off --no-build --framework net8.0 --logger:"console;verbosity=normal" --list-tests
    dotnet test -c Release --filter Category=Fast -tl:off --no-build --framework net8.0 --logger:"console;verbosity=normal"

  5. Verify it can run .NET462 benchmarks also

    dotnet test -c Release --filter IntroBasic -tl:off --no-build --framework net462 --logger:"console;verbosity=detailed" --list-tests
    dotnet test -c Release --filter IntroBasic -tl:off --no-build --framework net462 --logger:"console;verbosity=detailed"

@timcassell
Copy link
Collaborator
  1. All benchmarks(net462/net8.0) are displayed on Debug/Release configuration.

In the original PR, not including Debug without in-process was on purpose. See comment chain #2438 (comment). We decided to leave it as a future improvement to include debugger attachment. I think that should be done in a separate PR.

@filzrev
Copy link
Contributor Author
filzrev commented Jun 20, 2025

In the original PR, not including Debug without in-process was on purpose.

Thanks for pointed out it's expected behavior that display only InProcess jobs on Debug configuration.
I've add commit to revert related changes.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers
No reviews
Assignees
No one assigned
Labels
None yet
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.

[Feature Request] Support filtering feature on BenchmarkDotNet.TestAdapter
2 participants
@filzrev @timcassell
0