Description
The following outlines a proposal for how to handle testing args.
Submitted issues related to this problem:
- Add separate
python.testing.pytestArgssettings for regular, debug and discovery runs #21255 - Argument filtering for pytest fails to handle custom arguments #11736
- pytest arg parsing bug #21176
- Support different options for discovering and running tests with pytest #15148
- Read test debug configuration from settings not just launch.json vscode-python-debugger#532
- Support setting environmental variables for the test runner #17171
- design of UI as someone selects args: Test discovery flow is unusual/weird #17094
- launch.json "args" don't get passed to pytest unittests #12997
- Add "Run Tests" to launch.json #11932
- launch.json "args" don't get passed to pytest unittests #12997
- setup testing configs will be changed: Unittest rewrite - configure tests quick pick #21116
- people can use plugins on only a single on of the actions: Test Discovery fails on large codebases #21922?
- fixing how setup works to be less confusing: Configure Tests command overrides project testpaths #23714
- As part of this overhaul we will stop parsing args:
- consider coverage as well: Provide separate configurations for running tests with coverage #24865
Current design:
Currently, to pass any command line args for test run/discovery via the extension there is a single setting per test type (one for unittest, one for pytest). The two settings are python.testing.pytestArgs: [] and python.testing.unittestArgs: [] which take a string array of command line args. The args are then processed by the extension to remove some not applicable to the current action (removing run args during discovery and discovery args during run) but this is buggy and hard to maintain. Additionally to debug a debug config can be created in the launch.json.
Issues with current design:
- With only a single place to put args for each test type, a user cannot distinguish between which command line args they want to include during test run, discovery, coverage, or debug.
- Since pytest does not allow some args to be included during with the arg
--collect-only(used to distinguish run from discovery by the extension) these args are parsed out by the extension before sending the request. This means custom args are removed from test args and generally it creates a buggy experience that is hard to maintain with the breadth of args users would like to pass into their tests. - There is no straightforward solution to providing environment variables while running and discovering tests.
Proposed new design:
Create new custom launch configs that can be setup for test run and discovery. Also, increase compatibility with debug launch configs to make sure these work in tandem. The launch config would be set up something like this:
{
"name": "Python: pytest run",
"type": "python-pytest",
"request": "launch",
"args": ["", ""],
"env": {
"PYTHONPATH": "${workspaceFolder}"
}
},
{
"name": "Python: pytest discover",
"type": "python-pytest",
"request": "launch",
"args": ["", ""],
"env": {
"VAR": "true"
}
}
The two particular fields "args" and "env" would provide functionality useful to users. The rest would be default values.
The new launch config types that would be added would be:
Python: pytest runPython: pytest discoverPython: unittest runPython: unittest discover
The names of each one are up for discussion, please provide feedback below!
Next a way to set up these launch configs would be introduced to the extension. The initial steps would be:
add configuration -> python -> python config menu
Two possible options for how to display the testing config options would be:
A: to have another submenu called something like "testing" which, when opened, lists the 4 config options above
B: have the 4 config options sit in the current python submenu.
(I think option A is better for clarification even though it requires one more menu clickthrough).
additionally the button "configure tests" found on the test explorer would be changed so it routed the user to a menu of the launch config options for testing (similar to option A).
Concerns with proposed design:
- The use of launch configs may be harder for new / beginner users.
- The discoverability of launch configs is harder.
- The transition from a setting to using a config could be complicated/confusing for current users.
Possible ways to address above concerns
- help beginner users
a. Increase documentation and provide specific examples of launch configs.
b. provide default values for all fields in the launch config (this is already planned so the launch config should be runnable upon creation without edits meaning no args or env vars need to be provided) - discoverability
a. There is already a larger discussion happening on how we can improve this, switching to configs for testing provides even more reason to tackle this problem
b. switching the button "configure tests" to the new config flow is very discoverable since that is already the workflow - transition
a. We can have a deprecation notification regarding the setting and show it to all those still using the setting for test args
b. It should be simple to take someone's test args in their settings and turn that into a launch config so we could offer a button that handles this for the user to switch them over (drawback is we would be setting discovery and run to the same args but the user could see this then go change it)
Notes on Debugging:
Less investigation has been done into test debugging as we already offer a similar launch config. Further analysis should be done to see what is missing in this flow and how to improve it to align with test configs once a design is decided on.
Action Items:
We are looking for feedback on this proposal so please comment below any thoughts. Thanks!