10000 gh-108828: Support selecting tests by labels by serhiy-storchaka · Pull Request #108829 · python/cpython · GitHub
[go: up one dir, main page]

Skip to content

gh-108828: Support selecting tests by labels #108829

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Open
wants to merge 7 commits into
base: main
Choose a base branch
from
Open
Show file tree
Hide file tree
Changes from 1 commit
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
Prev Previous commit
Next Next commit
Add documentation.
  • Loading branch information
serhiy-storchaka committed Sep 11, 2023
commit fe7fcd04ba1cbbff601c95faf341998a2bfddafb
37 changes: 35 additions & 2 deletions Doc/library/test.rst
Original file line number Diff line number Diff line change
Expand Up @@ -482,6 +482,8 @@ The :mod:`test.support` module defines the following functions:
``True`` if called by a function whose ``__name__`` is ``'__main__'``.
Used when tests are executed by :mod:`test.regrtest`.

If called at the top level, sets label "requires_\ *resource*" on the module.


.. function:: sortdict(dict)

Expand All @@ -498,16 +500,34 @@ The :mod:`test.support` module defines the following functions:
rather than looking directly in the path directories.


.. function:: mark(label, *, globals=None)
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I would prefer to always call them "marks". The main reason is that pytest uses the same idea and pytest's marks are well-known.

Suggested change
.. function:: mark(label, *, globals=None)
.. function:: mark(name, *, globals=None)

Right now mark function adds a "label". It does not sound right.

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Or we can always call it "label".

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Yes, it is similar to pytest's markers, but there are enough differences in applying them and filtering by them. I do not have preference, "label", "marker" and "tag" are all look like synonyms in this context to me.


Add a label to tests.
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Suggested change
Add a label to tests.
Add a mark to tests.

The ``@mark('label')`` de 10000 corator adds a label to method or class.
``test.support.mark('label', globals=globals())`` adds a label to the whole
module.

Many :mod:`test.support` decorators like :func:`requires_resource`,
:func:`~test.support.cpython_only` or :func:`bigmemtest` add labels
automatically.


.. function:: match_test(test)

Determine whether *test* matches the patterns set in :func:`set_match_tests`.
Determine whether *test* matches the patterns set in :func:`set_match_tests`
and labels set in :func:`set_match_tests2`.


.. function:: set_match_tests(accept_patterns=None, ignore_patterns=None)

Define match patterns on test filenames and test method names for filtering tests.


.. function:: set_match_tests2(accept_labels=None, ignore_labels=None)

Define labels on tests for filtering.


.. function:: run_unittest(*classes)

Execute :class:`unittest.TestCase` subclasses passed to the function. The
Expand Down Expand Up @@ -774,26 +794,31 @@ The :mod:`test.support` module defines the following functions:
.. decorator:: requires_zlib

Decorator for skipping tests if :mod:`zlib` doesn't exist.
Adds label "requires_zlib".


.. decorator:: requires_gzip

Decorator for skipping tests if :mod:`gzip` doesn't exist.
Adds label "requires_gzip".


.. decorator:: requires_bz2

Decorator for skipping tests if :mod:`bz2` doesn't exist.
Adds label "requires_bz2".


.. decorator:: requires_lzma

Decorator for skipping tests if :mod:`lzma` doesn't exist.
Adds label "requires_lzma".


.. decorator:: requires_resource(resource)

Decorator for skipping tests if *resource* is not available.
Adds label "requires_\ *resource*".


.. decorator:: requires_docstrings
Expand All @@ -810,13 +835,16 @@ The :mod:`test.support` module defines the following functions:
.. decorator:: cpython_only

Decorator for tests only applicable to CPython.
Adds label "impl_detail_cpython".


.. decorator:: impl_detail(msg=None, **guards)

Decorator for invoking :func:`check_impl_detail` on *guards*. If that
returns ``False``, then uses *msg* as the reason for skipping the test.

For every keyword argument *implname* adds a label
"impl_detail_\ *implname*" if its value is true or
"impl_detail_no_\ *implname*" otherwise.

.. decorator:: no_tracing

Expand Down Expand Up @@ -845,10 +873,13 @@ The :mod:`test.support` module defines the following functions:
method may be less than the requested value. If *dry_run* is ``False``, it
means the test doesn't support dummy runs when ``-M`` is not specified.

Adds label "bigmemtest".


.. decorator:: bigaddrspacetest

Decorator for tests that fill the address space.
Adds label "bigaddrspacetest".


.. function:: check_syntax_error(testcase, statement, errtext='', *, lineno=None, offset=None)
Expand Down Expand Up @@ -1630,6 +1661,8 @@ The :mod:`test.support.import_helper` module provides support for import tests.
optional for others, set *required_on* to an iterable of platform prefixes
which will be compared against :data:`sys.platform`.

If called at the top level, sets label "requires_\ *name*" on the module.

.. versionadded:: 3.1


Expand Down
3 changes: 2 additions & 1 deletion Lib/test/support/__init__.py
Original file line number Diff line number Diff line change
Expand Up @@ -527,7 +527,8 @@ def requires_debug_ranges(reason='requires co_positions / debug_ranges'):
has_fork_support = hasattr(os, "fork") and not is_emscripten and not is_wasi

def requires_fork():
return skipUnless(has_fork_support, "requires working os.fork()", label='requires_fork')
return skipUnless(has_fork_support, "requires working os.fork()",
label='requires_fork')

has_subprocess_support = not is_emscripten and not is_wasi

Expand Down
Original file line number Diff line number Diff line change
@@ -0,0 +1,7 @@
Add support of labels in tests. The ``@test.support.mark('label')``
decorator adds a label to method or class. ``test.support.mark('label',
globals=globals())`` adds a label to the whole module. Many
:mod:`test.support` decorators like :func:`~test.support.requires_resource`,
:func:`~test.support.cpython_only` or :func:`~test.support.bigmemtest` add
labels automatically. Tests which have or have not the specified label can
be filtered by options ``--label`` and ``--no-label``.
0