8000 Proposal: Enable separate configs per run/discovery/debug/coverage with testing · Issue #21845 · microsoft/vscode-python · GitHub
[go: up one dir, main page]

Skip to content
Proposal: Enable separate configs per run/discovery/debug/coverage with testing #21845
Open
@eleanorjboyd

Description

@eleanorjboyd

The following outlines a proposal for how to handle testing args.

Submitted issues related to this problem:

Current design:
Currently, to pass any command line args for test run/discovery via the extension there is a single setting per test type (one for unittest, one for pytest). The two settings are python.testing.pytestArgs: [] and python.testing.unittestArgs: [] which take a string array of command line args. The args are then processed by the extension to remove some not applicable to the current action (removing run args during discovery and discovery args during run) but this is buggy and hard to maintain. Additionally to debug a debug config can be created in the launch.json.

Issues with current design:

  1. With only a single place to put args for each test type, a user cannot distinguish between which command line args they want to include during test run, discovery, coverage, or debug.
  2. Since pytest does not allow some args to be included during with the arg --collect-only (used to distinguish run from discovery by the extension) these args are parsed out by the extension before sending the request. This means custom args are removed from test args and generally it creates a buggy experience that is hard to maintain with the breadth of args users would like to pass into their tests.
  3. There is no straightforward solution to providing environment variables while running and discovering tests.

Proposed new design:
Create new custom launch configs that can be setup for test run and discovery. Also, increase compatibility with debug launch configs to make sure these work in tandem. The launch config would be set up something like this:

    {
      "name": "Python: pytest run",
      "type": "python-pytest",
      "request": "launch",
      "args": ["", ""],
      "env": {
        "PYTHONPATH": "${workspaceFolder}"
      }
    },
    {
      "name": "Python: pytest discover",
      "type": "python-pytest",
      "request": "launch",
      "args": ["", ""],
      "env": {
        "VAR": "true"
      }
    }

The two particular fields "args" and "env" would provide functionality useful to users. The rest would be default values.

The new launch config types that would be added would be:

  1. Python: pytest run
  2. Python: pytest discover
  3. Python: unittest run
  4. Python: unittest discover
    The names of each one are up for discussion, please provide feedback below!

Next a way to set up these launch configs would be introduced to the extension. The initial steps would be:
image
add configuration -> python -> python config menu
Two possible options for how to display the testing config options would be:
A: to have another submenu called something like "testing" which, when opened, lists the 4 config options above
B: have the 4 config options sit in the current python submenu.

(I think option A is better for clarification even though it requires one more menu clickthrough).

additionally the button "configure tests" found on the test explorer would be changed so it routed the user to a menu of the launch config options for testing (similar to option A).
image

Concerns with proposed design:

  1. The use of launch configs may be harder for new / beginner users.
  2. The discoverability of launch configs is harder.
  3. The transition from a setting to using a config could be complicated/confusing for current users.

Possible ways to address above concerns

  1. help beginner users
    a. Increase documentation and provide specific examples of launch configs.
    b. provide default values for all fields in the launch config (this is already planned so the launch config should be runnable upon creation without edits meaning no args or env vars need to be provided)
  2. discoverability
    a. There is already a larger discussion happening on how we can improve this, switching to configs for testing provides even more reason to tackle this problem
    b. switching the button "configure tests" to the new config flow is very discoverable since that is already the workflow
  3. transition
    a. We can have a deprecation notification regarding the setting and show it to all those still using the setting for test args
    b. It should be simple to take someone's test args in their settings and turn that into a launch config so we could offer a button that handles this for the user to switch them over (drawback is we would be setting discovery and run to the same args but the user could see this then go change it)

Notes on Debugging:
Less investigation has been done into test debugging as we already offer a similar launch config. Further analysis should be done to see what is missing in this flow and how to improve it to align with test configs once a design is decided on.

Action Items:
We are looking for feedback on this proposal so please comment below any thoughts. Thanks!

Metadata

Metadata

Assignees

Labels

Type

No type

Projects

No projects

Milestone

No milestone

Relationships

None yet

Development

No branches or pull requests

Issue actions

    0