8000 gh-108828: Support selecting tests by labels by serhiy-storchaka · Pull Request #108829 · python/cpython · GitHub
[go: up one dir, main page]

Skip to content

gh-108828: Support selecting tests by labels #108829

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Open
wants to merge 7 commits into
base: main
Choose a base branch
from
Open
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
29 changes: 28 additions & 1 deletion Doc/library/test.rst
Original file line number Diff line number Diff line change
Expand Up @@ -482,6 +482,8 @@ The :mod:`test.support` module defines the following functions:
``True`` if called by a function whose ``__name__`` is ``'__main__'``.
Used when tests are executed by :mod:`test.regrtest`.

If called at the top level, sets label "requires\_\ *resource*" on the module.


.. function:: sortdict(dict)

Expand All @@ -498,6 +500,18 @@ The :mod:`test.support` module defines the following functions:
rather than looking directly in the path directories.


.. function:: mark(label, *, globals=None)
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I would prefer to always call them "marks". The main reason is that pytest uses the same idea and pytest's marks are well-known.

Suggested change
.. function:: mark(label, *, globals=None)
.. function:: mark(name, *, globals=None)

Right now mark function adds a "label". It does not sound right.

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Or we can always call it "label".

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Yes, it is similar to pytest's markers, but there are enough differences in applying them and filtering by them. I do not have preference, "label", "marker" and "tag" are all look like synonyms in this context to me.


Add a label to tests.
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Suggested change
Add a label to tests.
Add a mark to tests.

The ``@mark('label')`` decorator adds a label to method or class.
``test.support.mark('label', globals=globals())`` adds a label to the whole
module.

Many :mod:`test.support` decorators like :func:`requires_resource`,
:func:`~test.support.cpython_only` or :func:`bigmemtest` add labels
automatically.


.. function:: get_pagesize()

Get size of a page in bytes.
Expand Down Expand Up @@ -736,26 +750,31 @@ The :mod:`test.support` module defines the following functions:
.. decorator:: requires_zlib

Decorator for skipping tests if :mod:`zlib` doesn't exist.
Adds label "requires_zlib".


.. decorator:: requires_gzip

Decorator for skipping tests if :mod:`gzip` doesn't exist.
Adds label "requires_gzip".


.. decorator:: requires_bz2

Decorator for skipping tests if :mod:`bz2` doesn't exist.
Adds label "requires_bz2".


.. decorator:: requires_lzma

Decorator for skipping tests if :mod:`lzma` doesn't exist.
Adds label "requires_lzma".


.. decorator:: requires_resource(resource)

Decorator for skipping tests if *resource* is not available.
Adds label "requires\_\ *resource*".


.. decorator:: requires_docstrings
Expand All @@ -772,13 +791,16 @@ The :mod:`test.support` module defines the following functions:
.. decorator:: cpython_only

Decorator for tests only applicable to CPython.
Adds label "impl_detail_cpython".


.. decorator:: impl_detail(msg=None, **guards)

Decorator for invoking :func:`check_impl_detail` on *guards*. If that
returns ``False``, then uses *msg* as the reason for skipping the test.

For every keyword argument *name* adds a label
"impl_detail\_\ *name*" if its value is true or
"impl_detail_no\_\ *name*" otherwise.

.. decorator:: no_tracing

Expand Down Expand Up @@ -807,10 +829,13 @@ The :mod:`test.support` module defines the following functions:
method may be less than the requested value. If *dry_run* is ``False``, it
means the test doesn't support dummy runs when ``-M`` is not specified.

Adds label "bigmemtest".


.. decorator:: bigaddrspacetest

Decorator for tests that fill the address space.
Adds label "bigaddrspacetest".


.. function:: check_syntax_error(testcase, statement, errtext='', *, lineno=None, offset=None)
Expand Down Expand Up @@ -1592,6 +1617,8 @@ The :mod:`test.support.import_helper` module provides support for import tests.
optional for others, set *required_on* to an iterable of platform prefixes
which will be compared against :data:`sys.platform`.

If called at the top level, sets label "requires\_\ *name*" on the module.

.. versionadded:: 3.1


Expand Down
7 changes: 7 additions & 0 deletions Lib/test/libregrtest/cmdline.py
Original file line number Diff line number Diff line change
Expand Up @@ -162,6 +162,7 @@ def __init__(self, **kwargs) -> None:
self.header = False
self.failfast = False
self.match_tests = []
self.match_labels = []
self.pgo = False
self.pgo_extended = False
self.worker_json = None
Expand Down Expand Up @@ -270,6 +271,12 @@ def _create_parser():
group.add_argument('-i', '--ignore', metavar='PAT',
dest='match_tests', action=FilterAction, const=False,
help='ignore test cases and methods with glob pattern PAT')
group.add_argument('--label', metavar='NAME',
dest='match_labels', action=FilterAction, const=True,
help='match test cases and methods with label NAME')
group.add_argument('--no-label', metavar='NAME',
dest='match_labels', action=FilterAction, const=False,
help='ignore test cases and methods with label NAME')
group.add_argument('--matchfile', metavar='FILENAME',
dest='match_tests',
action=FromFileFilterAction, const=True,
Expand Down
31 changes: 30 additions & 1 deletion Lib/test/libregrtest/filter.py
Original file line number Diff line number Diff line change
@@ -1,21 +1,50 @@
import itertools
import operator
import re
import sys


# By default, don't filter tests
_test_matchers = ()
_test_patterns = ()
_match_labels = ()


def match_test(test):
# Function used by support.run_unittest() and regrtest --list-cases
return match_test_id(test) and match_test_label(test)

def match_test_id(test):
result = False
for matcher, result in reversed(_test_matchers):
if matcher(test.id()):
return result
return not result

def match_test_label(test):
result = False
for label, result in reversed(_match_labels):
if _has_label(test, label):
return result
return not result

def _has_label(test, label):
attrname = f'_label_{label}'
if hasattr(test, attrname):
return True
testMethod = getattr(test, test._testMethodName)
while testMethod is not None:
if hasattr(testMethod, attrname):
return True
testMethod = getattr(testMethod, '__wrapped__', None)
try:
module = sys.modules[test.__class__.__module__]
if hasattr(module, attrname):
return True
except KeyError:
pass
return False


def _is_full_match_test(pattern):
# If a pattern contains at least one dot, it's considered
Expand All @@ -27,7 +56,7 @@ def _is_full_match_test(pattern):
return ('.' in pattern) and (not re.search(r'[?*\[\]]', pattern))


def set_match_tests(patterns):
def set_match_tests(patterns=None, match_labels=None):
global _test_matchers, _test_patterns

if not patterns:
Expand Down
3 changes: 2 additions & 1 deletion Lib/test/libregrtest/findtests.py
Original file line number Diff line number Diff line change
Expand Up @@ -85,9 +85,10 @@ def _list_cases(suite):

def list_cases(tests: TestTuple, *,
match_tests: TestFilter | None = None,
match_labels: TestFilter | None = None,
test_dir: StrPath | None = None):
support.verbose = False
set_match_tests(match_tests)
set_match_tests(match_tests, match_labels)

skipped = []
for test_name in tests:
Expand Down
3 changes: 3 additions & 0 deletions Lib/test/libregrtest/main.py
Original file line number Diff line number Diff line change
Expand Up @@ -79,6 +79,7 @@ def __init__(self, ns: Namespace, _add_python_opts: bool = False):

# Select tests
self.match_tests: TestFilter = ns.match_tests
self.match_labels: TestFilter = ns.match_labels
self.exclude: bool = ns.exclude
self.fromfile: StrPath | None = ns.fromfile
self.starting_test: TestName | None = ns.start
Expand Down Expand Up @@ -400,6 +401,7 @@ def create_run_tests(self, tests: TestTuple):
fail_fast=self.fail_fast,
fail_env_changed=self.fail_env_changed,
match_tests=self.match_tests,
match_labels=self.match_labels,
match_tests_dict=None,
rerun=False,
forever=self.forever,
Expand Down Expand Up @@ -652,6 +654,7 @@ def main(self, tests: TestList | None = None):
elif self.want_list_cases:
list_cases(selected,
match_tests=self.match_tests,
match_labels=self.match_labels,
test_dir=self.test_dir)
else:
exitcode = self.run_tests(selected, tests)
Expand Down
1 change: 1 addition & 0 deletions Lib/test/libregrtest/runtests.py
Original file line number Diff line number Diff line change
Expand Up @@ -73,6 +73,7 @@ class RunTests:
fail_fast: bool
fail_env_changed: bool
match_tests: TestFilter
match_labels: TestFilter
match_tests_dict: FilterDict | None
rerun: bool
forever: bool
Expand Down
2 changes: 1 addition & 1 deletion Lib/test/libregrtest/setup.py
Original file line number Diff line number Diff line change
Expand Up @@ -93,7 +93,7 @@ def setup_tests(runtests: RunTests):
support.PGO = runtests.pgo
support.PGO_EXTENDED = runtests.pgo_extended

set_match_tests(runtests.match_tests)
set_match_tests(runtests.match_tests, runtests.match_labels)

if runtests.use_junit:
support.junit_xml_list = []
Expand Down
Loading
0