8000 gh-80958: unittest: discovery support for namespace packages as start… · python/cpython@c75ff2e · GitHub
[go: up one dir, main page]
Skip to content

Commit c75ff2e

Browse files
jacobtylerwallsjacobtylerwalls
authored
gh-80958: unittest: discovery support for namespace packages as start directory (#123820)
1 parent 34653bb commit c75ff2e

File tree

12 files changed

+145
-37
lines changed
  • Makefile.pre.in
  • Misc/NEWS.d/next/Library

    • 12 files changed

    +145
    -37
    lines changed

    ‎Doc/library/unittest.rst

    Lines changed: 14 additions & 21 deletions
    Original file line numberDiff line numberDiff line change
    @@ -340,28 +340,21 @@ Test modules and packages can customize test loading and discovery by through
    340340
    the `load_tests protocol`_.
    341341

    342342
    .. versionchanged:: 3.4
    343-
    Test discovery supports :term:`namespace packages <namespace package>`
    344-
    for the start directory. Note that you need to specify the top level
    345-
    directory too (e.g.
    346-
    ``python -m unittest discover -s root/namespace -t root``).
    343+
    Test discovery supports :term:`namespace packages <namespace package>`.
    347344

    348345
    .. versionchanged:: 3.11
    349-
    :mod:`unittest` dropped the :term:`namespace packages <namespace package>`
    350-
    support in Python 3.11. It has been broken since Python 3.7. Start directory and
    351-
    subdirectories containing tests must be regular package that have
    352-
    ``__init__.py`` file.
    346+
    Test discovery dropped the :term:`namespace packages <namespace package>`
    347+
    support. It has been broken since Python 3.7.
    348+
    Start directory and its subdirectories containing tests must be regular
    349+
    package that have ``__init__.py`` file.
    353350

    354-
    Directories containing start directory still can be a namespace package.
    355-
    In this case, you need to specify start directory as dotted package name,
    356-
    and target directory explicitly. For example::
    351+
    If the start directory is the dotted name of the package, the ancestor packages
    352+
    can be namespace packages.
    357353

    358-
    # proj/ <-- current directory
    359-
    # namespace/
    360-
    # mypkg/
    361-
    # __init__.py
    362-
    # test_mypkg.py
    363-
    364-
    python -m unittest discover -s namespace.mypkg -t .
    354+
    .. versionchanged:: 3.14
    355+
    Test discovery supports :term:`namespace package` as start directory again.
    356+
    To avoid scanning directories unrelated to Python,
    357+
    tests are not searched in subdirectories that do not contain ``__init__.py``.
    365358

    366359

    367360
    .. _organizing-tests:
    @@ -1915,10 +1908,8 @@ Loading and running tests
    19151908
    Modules that raise :exc:`SkipTest` on import are recorded as skips,
    19161909
    not errors.
    19171910

    1918-
    .. versionchanged:: 3.4
    19191911
    *start_dir* can be a :term:`namespace packages <namespace package>`.
    19201912

    1921-
    .. versionchanged:: 3.4
    19221913
    Paths are sorted before being imported so that execution order is the
    19231914
    same even if the underlying file system's ordering is not dependent
    19241915
    on file name.
    @@ -1930,11 +1921,13 @@ Loading and running tests
    19301921

    19311922
    .. versionchanged:: 3.11
    19321923
    *start_dir* can not be a :term:`namespace packages <namespace package>`.
    1933-
    It has been broken since Python 3.7 and Python 3.11 officially remove it.
    1924+
    It has been broken since Python 3.7, and Python 3.11 officially removes it.
    19341925

    19351926
    .. versionchanged:: 3.13
    19361927
    *top_level_dir* is only stored for the duration of *discover* call.
    19371928

    1929+
    .. versionchanged:: 3.14
    1930+
    *start_dir* can once again be a :term:`namespace package`.
    19381931

    19391932
    The following attributes of a :class:`TestLoader` can be configured either by
    19401933
    subclassing or assignment on an instance:

    ‎Doc/whatsnew/3.14.rst

    Lines changed: 9 additions & 0 deletions
    Original file line numberDiff line numberDiff line change
    @@ -421,6 +421,15 @@ unicodedata
    421421

    422422
    * The Unicode database has been updated to Unicode 16.0.0.
    423423

    424+
    425+
    unittest
    426+
    --------
    427+
    428+
    * unittest discovery supports :term:`namespace package` as start
    429+
    directory again. It was removed in Python 3.11.
    430+
    (Contributed by Jacob Walls in :gh:`80958`.)
    431+
    432+
    424433
    .. Add improved modules above alphabetically, not here at the end.
    425434
    426435
    Optimizations

    ‎Lib/test/test_unittest/namespace_test_pkg/bar/__init__.py

    Whitespace-only changes.

    ‎Lib/test/test_unittest/namespace_test_pkg/bar/test_bar.py

    Lines changed: 5 additions & 0 deletions
    Original file line numberDiff line numberDiff line change
    @@ -0,0 +1,5 @@
    1+
    import unittest
    2+
    3+
    class PassingTest(unittest.TestCase):
    4+
    def test_true(self):
    5+
    self.assertTrue(True)

    ‎Lib/test/test_unittest/namespace_test_pkg/noop/no2/__init__.py

    Whitespace-only changes.

    ‎Lib/test/test_unittest/namespace_test_pkg/noop/no2/test_no2.py

    Lines changed: 5 additions & 0 deletions
    Original file line numberDiff line numberDiff line change
    @@ -0,0 +1,5 @@
    1+
    import unittest
    2+
    3+
    class PassingTest(unittest.TestCase):
    4+
    def test_true(self):
    5+
    self.assertTrue(True)

    ‎Lib/test/test_unittest/namespace_test_pkg/noop/test_noop.py

    Lines changed: 5 additions & 0 deletions
    Original file line numberDiff line numberDiff line change
    @@ -0,0 +1,5 @@
    1+
    import unittest
    2+
    3+
    class PassingTest(unittest.TestCase):
    4+
    def test_true(self):
    5+
    self.assertTrue(True)

    ‎Lib/test/test_unittest/namespace_test_pkg/test_foo.py

    Lines changed: 5 additions & 0 deletions
    Original file line numberDiff line numberDiff line change
    @@ -0,0 +1,5 @@
    1+
    import unittest
    2+
    3+
    class PassingTest(unittest.TestCase):
    4+
    def test_true(self):
    5+
    self.assertTrue(True)

    ‎Lib/test/test_unittest/test_discovery.py

    Lines changed: 52 additions & 2 deletions
    Original file line numberDiff line numberDiff line change
    @@ -4,12 +4,14 @@
    44
    import sys
    55
    import types
    66
    import pickle
    7+
    from importlib._bootstrap_external import NamespaceLoader
    78
    from test import support
    89
    from test.support import import_helper
    910

    1011
    import unittest
    1112
    import unittest.mock
    1213
    import test.test_unittest
    14+
    from test.test_importlib import util as test_util
    1315

    1416

    1517
    class TestableTestProgram(unittest.TestProgram):
    @@ -395,7 +397,7 @@ def restore_isdir():
    395397
    self.addCleanup(restore_isdir)
    396398

    397399
    _find_tests_args = []
    398-
    def _find_tests(start_dir, pattern):
    400+
    def _find_tests(start_dir, pattern, namespace=None):
    399401
    _find_tests_args.append((start_dir, pattern))
    400402
    return ['tests']
    401403
    loader._find_tests = _find_tests
    @@ -815,7 +817,7 @@ def test_discovery_from_dotted_path(self):
    815817
    expectedPath = os.path.abspath(os.path.dirname(test.test_unittest.__file__))
    816818

    817819
    self.wasRun = False
    818-
    def _find_tests(start_dir, pattern):
    820+
    def _find_tests(start_dir, pattern, namespace=None):
    819821
    self.wasRun = True
    820822
    self.assertEqual(start_dir, expectedPath)
    821823
    return tests
    @@ -848,6 +850,54 @@ def restore():
    848850
    'Can not use builtin modules '
    849851
    'as dotted module names')
    850852

    853+
    def test_discovery_from_dotted_namespace_packages(self):
    854+
    loader = unittest.TestLoader()
    855+
    856+
    package = types.ModuleType('package')
    857+
    package.__name__ = "tests"
    858+
    package.__path__ = ['/a', '/b']
    859+
    package.__file__ = None
    860+
    package.__spec__ = types.SimpleNamespace(
    861+
    name=package.__name__,
    862+
    loader=NamespaceLoader(package.__name__, package.__path__, None),
    863+
    submodule_search_locations=['/a', '/b']
    864+
    )
    865+
    866+
    def _import(packagename, *args, **kwargs):
    867+
    sys.modules[packagename] = package
    868+
    return package
    869+
    870+
    _find_tests_args = []
    871+
    def _find_tests(start_dir, pattern, namespace=None):
    872+
    _find_tests_args.append((start_dir, pattern))
    873+
    return ['%s/tests' % start_dir]
    874+
    875+
    loader._find_tests = _find_tests
    876+
    loader.suiteClass = list
    877+
    878+
    with unittest.mock.patch('builtins.__import__', _import):
    879+
    # Since loader.discover() can modify sys.path, restore it when done.
    880+
    with import_helper.DirsOnSysPath():
    881+
    # Make sure to remove 'package' from sys.modules when done.
    882+
    with test_util.uncache('package'):
    883+
    suite = loader.discover('package')
    884+
    885+
    self.assertEqual(suite, ['/a/tests', '/b/tests'])
    886+
    887+
    def test_discovery_start_dir_is_namespace(self):
    888+
    """Subdirectory discovery not affected if start_dir is a namespace pkg."""
    889+
    loader = unittest.TestLoader()
    890+
    with (
    891+
    import_helper.DirsOnSysPath(os.path.join(os.path.dirname(__file__))),
    892+
    test_util.uncache('namespace_test_pkg')
    893+
    ):
    894+
    suite = loader.discover('namespace_test_pkg')
    895+
    self.assertEqual(
    896+
    {list(suite)[0]._tests[0].__module__ for suite in suite._tests if list(suite)},
    897+
    # files under namespace_test_pkg.noop not discovered.
    898+
    {'namespace_test_pkg.test_foo', 'namespace_test_pkg.bar.test_bar'},
    899+
    )
    900+
    851901
    def test_discovery_failed_discovery(self):
    852902
    from test.test_importlib import util
    853903

    ‎Lib/unittest/loader.py

    Lines changed: 45 additions & 14 deletions
    Original file line numberDiff line numberDiff line change
    @@ -274,6 +274,8 @@ def discover(self, start_dir, pattern='test*.py', top_level_dir=None):
    274274
    self._top_level_dir = top_level_dir
    275275

    276276
    is_not_importable = False
    277+
    is_namespace = False
    278+
    tests = []
    277279
    if os.path.isdir(os.path.abspath(start_dir)):
    278280
    start_dir = os.path.abspath(start_dir)
    279281
    if start_dir != top_level_dir:
    @@ -286,12 +288,25 @@ def discover(self, start_dir, pattern='test*.py', top_level_dir=None):
    286288
    is_not_importable = True
    287289
    else:
    288290
    the_module = sys.modules[start_dir]
    289-
    top_part = start_dir.split('.')[0]
    290-
    try:
    291-
    start_dir = os.path.abspath(
    292-
    os.path.dirname((the_module.__file__)))
    293-
    except AttributeError:
    294-
    if the_module.__name__ in sys.builtin_module_names:
    291+
    if not hasattr(the_module, "__file__") or the_module.__file__ is None:
    292+
    # look for namespace packages
    293+
    try:
    294+
    spec = the_module.__spec__
    295+
    except AttributeError:
    296+
    spec = None
    297+
    298+
    if spec and spec.submodule_search_locations is not None:
    299+
    is_namespace = True
    300+
    301+
    for path in the_module.__path__:
    302+
    if (not set_implicit_top and
    303+
    not path.startswith(top_level_dir)):
    304+
    continue
    305+
    self._top_level_dir = \
    306+
    (path.split(the_module.__name__
    307+
    .replace(".", os.path.sep))[0])
    308+
    tests.extend(self._find_tests(path, pattern, namespace=True))
    309+
    elif the_module.__name__ in sys.builtin_module_names:
    295310
    # builtin module
    296311
    raise TypeError('Can not use builtin modules '
    297312
    'as dotted module names') from None
    @@ -300,14 +315,27 @@ def discover(self, start_dir, pattern='test*.py', top_level_dir=None):
    300315
    f"don't know how to discover from {the_module!r}"
    301316
    ) from None
    302317

    318+
    else:
    319+
    top_part = start_dir.split('.')[0]
    320+
    start_dir = os.path.abspath(os.path.dirname((the_module.__file__)))
    321+
    303322
    if set_implicit_top:
    304-
    self._top_level_dir = self._get_directory_containing_module(top_part)
    323+
    if not is_namespace:
    324+
    if sys.modules[top_part].__file__ is None:
    325+
    self._top_level_dir = os.path.dirname(the_module.__file__)
    326+
    if self._top_level_dir not in sys.path:
    327+
    sys.path.insert(0, self._top_level_dir)
    328+
    else:
    329+
    self._top_level_dir = \
    330+
    self._get_directory_containing_module(top_part)
    305331
    sys.path.remove(top_level_dir)
    306332

    307333
    if is_not_importable:
    308334
    raise ImportError('Start directory is not importable: %r' % start_dir)
    309335

    310-
    tests = list(self._find_tests(start_dir, pattern))
    336+
    if not is_namespace:
    337+
    tests = list(self._find_tests(start_dir, pattern))
    338+
    311339
    self._top_level_dir = original_top_level_dir
    312340
    return self.suiteClass(tests)
    313341

    @@ -343,7 +371,7 @@ def _match_path(self, path, full_path, pattern):
    343371
    # override this method to use alternative matching strategy
    344372
    return fnmatch(path, pattern)
    345373

    346-
    def _find_tests(self, start_dir, pattern):
    374+
    def _find_tests(self, start_dir, pattern, namespace=False):
    347375
    """Used by discovery. Yields test suites it loads."""
    348376
    # Handle the __init__ in this package
    349377
    name = self._get_name_from_path(start_dir)
    @@ -352,7 +380,8 @@ def _find_tests(self, start_dir, pattern):
    352380
    if name != '.' and name not in self._loading_packages:
    353381
    # name is in self._loading_packages while we have called into
    354382
    # loadTestsFromModule with name.
    355-
    tests, should_recurse = self._find_test_path(start_dir, pattern)
    383+
    tests, should_recurse = self._find_test_path(
    384+
    start_dir, pattern, namespace)
    356385
    if tests is not None:
    357386
    yield tests
    358387
    if not should_recurse:
    @@ -363,19 +392,20 @@ def _find_tests(self, start_dir, pattern):
    363392
    paths = sorted(os.listdir(start_dir))
    364393
    for path in paths:
    365394
    full_path = os.path.join(start_dir, path)
    366-
    tests, should_recurse = self._find_test_path(full_path, pattern)
    395+
    tests, should_recurse = self._find_test_path(
    396+
    full_path, pattern, False)
    367397
    if tests is not None:
    368398
    yield tests
    369399
    if should_recurse:
    370400
    # we found a package that didn't use load_tests.
    371401
    name = self._get_name_from_path(full_path)
    372402
    self._loading_packages.add(name)
    373403
    try:
    374-
    yield from self._find_tests(full_path, pattern)
    404+
    yield from self._find_tests(full_path, pattern, False)
    375405
    finally:
    376406
    self._loading_packages.discard(name)
    377407

    378-
    def _find_test_path(self, full_path, pattern):
    408+
    def _find_test_path(self, full_path, pattern, namespace=False):
    379409
    """Used by discovery.
    380410
    381411
    Loads tests from a single file, or a directories' __init__.py when
    @@ -419,7 +449,8 @@ def _find_test_path(self, full_path, pattern):
    419449
    msg % (mod_name, module_dir, expected_dir))
    420450
    return self.loadTestsFromModule(module, pattern=pattern), False
    421451
    elif os.path.isdir(full_path):
    422-
    if not os.path.isfile(os.path.join(full_path, '__init__.py')):
    452+
    if (not namespace and
    453+
    not os.path.isfile(os.path.join(full_path, '__init__.py'))):
    423454
    return None, False
    424455

    425456
    load_tests = None

    ‎Makefile.pre.in

    Lines changed: 4 additions & 0 deletions
    Original file line numberDiff line numberDiff line change
    @@ -2534,6 +2534,10 @@ TESTSUBDIRS= idlelib/idle_test \
    25342534
    test/test_tools \
    25352535
    test/test_ttk \
    25362536
    test/test_unittest \
    2537+
    test/test_unittest/namespace_test_pkg \
    2538+
    test/test_unittest/namespace_test_pkg/bar \
    2539+
    test/test_unittest/namespace_test_pkg/noop \
    2540+
    test/test_unittest/namespace_test_pkg/noop/no2 \
    25372541
    test/test_unittest/testmock \
    25382542
    test/test_warnings \
    25392543
    test/test_warnings/data \

    ‎Misc/NEWS.d/next/Library/2024-09-07-13-57-49.gh-issue-80958.fVYnqV.rst

    Lines changed: 1 addition & 0 deletions
    Original file line numberDiff line numberDiff line change
    @@ -0,0 +1 @@
    1+
    unittest discovery supports PEP 420 namespace packages as start directory again.

    0 commit comments

    Comments
     (0)
    0