Importing types from namespace packages emits false positives without `from ... import ...` #5758

chrisphilip322 · 2018-10-09T14:36:21Z

Importing type hints from namespace packages only works when using the from ... import ... form.

Using the --namespace-packages flag works properly when the typedpkg_namespace-alpha is not installed but mypy is executed from the directory containing the package contents (as expected) but only for imports using the from ... import ... style. import ... does not work as expected. Likewise the same is true without the --namespace-packages flag if typedpkg_namespace-alpha is installed.

Specifically, the type hints are correctly imported but a false positive error message is still emitted of Cannot find module named 'typedpkg_namespace'.

Repro steps

Virtual env state

$ pip list
Package         Version
--------------- --------------------------------------------------
mypy            0.640+dev.e12be3ba9e6be3e9242e3a81ffaa9b3a136876fe
mypy-extensions 0.4.1
pip             18.1
setuptools      40.4.3
typed-ast       1.1.0

Repro file structure

$ find . \( -path ./.mypy_cache -o -path ./venv \) -prune -o -print
.
./typedpkg_namespace
./typedpkg_namespace/alpha
./typedpkg_namespace/alpha/__init__.py
./typedpkg_namespace/alpha/py.typed
./typedpkg_namespace/alpha/alpha_module.py
./not_from_imp.py
./from_imp.py
./setup.py

Repro files

setup.py

setup(
    name='typedpkg_namespace.alpha',
    version='1.0.0',
    packages=find_namespace_packages(),
    namespace_packages=['typedpkg_namespace'],
    zip_safe=False,
    package_data={'typedpkg_namespace.alpha': ['py.typed']}
)

typedpkg_namespace/alpha/alpha_module.py

def alpha_func(a: bool) -> bool:
    return not a

from_imp.py

from typedpkg_namespace.alpha.alpha_module import alpha_func
alpha_func(False)
alpha_func(2)

not_from_imp.py

import typedpkg_namespace.alpha.alpha_module as am; alpha_func = am.alpha_func
alpha_func(False)
alpha_func(2)

typedpkg_namespace/alpha/__init__.py and typedpkg_namespace/alpha/py.typed are empty

Repro Example

using --namespace-packages

$ mypy from_imp.py  --namespace-packages
from_imp.py:3: error: Argument 1 to "alpha_func" has incompatible type "int"; expected "bool"

$ mypy not_from_imp.py --namespace-packages
not_from_imp.py:1: error: Cannot find module named 'typedpkg_namespace'
not_from_imp.py:1: note: (Perhaps setting MYPYPATH or using the "--ignore-missing-imports" flag would help)
not_from_imp.py:3: error: Argument 1 has incompatible type "int"; expected "bool"

using installed package

$ pip install .
Looking in indexes: http://artifactory.factset.com/artifactory/api/pypi/python/simple/
Processing /home/user/cphilip/side_projects/test_namespace/typedpkg_namespace-alpha
Installing collected packages: typedpkg-namespace.alpha
  Running setup.py install for typedpkg-namespace.alpha ... done
Successfully installed typedpkg-namespace.alpha-1.0.0

$ pip list
Package                  Version
------------------------ --------------------------------------------------
mypy                     0.640+dev.e12be3ba9e6be3e9242e3a81ffaa9b3a136876fe
mypy-extensions          0.4.1
pip                      18.1
setuptools               40.4.3
typed-ast                1.1.0
typedpkg-namespace.alpha 1.0.0

$ mypy from_imp.py
from_imp.py:3: error: Argument 1 to "alpha_func" has incompatible type "int"; expected "bool"

$ mypy not_from_imp.py
not_from_imp.py:1: error: Cannot find module named 'typedpkg_namespace'
not_from_imp.py:1: note: (Perhaps setting MYPYPATH or using the "--ignore-missing-imports" flag would help)
not_from_imp.py:3: error: Argument 1 has incompatible type "int"; expected "bool"

The text was updated successfully, but these errors were encountered:

gvanrossum · 2018-10-09T22:58:57Z

Yup, I can repro this. There must be a corner case that I don't handle correctly yet.

gvanrossum · 2018-10-09T23:46:01Z

Can you try with #5762?

chrisphilip322 · 2018-10-10T00:54:25Z

I tested #5762 locally at it looks like it fixes the first part of this issue, using --namespace-packages when the typedpkg_namespace source directory is in the current working directory.

When the package is installed, and the current working directory does not contain the package source directory, I get the same false positive error with or without --namespace-packages.

Too be clear, I get the same error when the package is installed with a file structure like this

$ find . \( -path *.mypy_cache -o -path ./typedpkg_namespace-alpha/venv \) -prune -o -print
.
./not_from_imp.py
./typedpkg_namespace-alpha
./typedpkg_namespace-alpha/typedpkg_namespace
./typedpkg_namespace-alpha/typedpkg_namespace/alpha
./typedpkg_namespace-alpha/typedpkg_namespace/alpha/__init__.py
./typedpkg_namespace-alpha/typedpkg_namespace/alpha/py.typed
./typedpkg_namespace-alpha/typedpkg_namespace/alpha/alpha_module.py
./typedpkg_namespace-alpha/not_from_imp.py
./typedpkg_namespace-alpha/from_imp.py
./typedpkg_namespace-alpha/setup.py
./from_imp.py

gvanrossum · 2018-10-10T01:05:06Z

So this seems specific to when it's installed. Could you make those files available as a throwaway GitHub repo so I can test with it? (I'm too lazy to try and copy/paste the entire setup from your initial message.)

chrisphilip322 · 2018-10-10T01:35:25Z

I made https://github.com/macbeth322/mypy-namespace-bug-repro for reproducing the issue. I also noticed that if you do import .. as .. it works but with false positive output, but without the as .. it actually doesn't find the type hints.

I was looking through the code and I believe it may be related to https://github.com/python/mypy/blob/master/mypy/build.py#L577 and the fact that when you do import foo.bar.baz, foo, foo.bar and foo.bar.baz all become available in the current scope. In the case where you have foo/bar/py.typed the foo package is never going to have type hints. In most cases foo will be a namespace package so there shouldn't be any code there to have type hints so there might be a workaround to check if a import name is missing type hints and it is a namespace package.

gvanrossum · 2018-10-10T04:04:10Z

Thanks for making the repro! I'm still banging my head against the wall but you're definitely right that it's got something to do with the ancestors. Unfortunately I don't think it's as simple as dropping that code (or even dropping it when the ancestor is a toplevel namespace), because later references to e.g. foo.bar.baz are going to expect foo to be a module object, and then do a getattr() on that to get baz, and so on. I'll try to continue the research tomorrow, but if you have time to experiment with some careful surgery of the code don't let me stop you!

Fixes #5758. (Sadly there's no test for the second half of that issue; I've opened #5767 to track adding one.)

finswimmer · 2021-12-17T07:16:53Z

Hey,

sorry for commenting on this quite old issue. It looks like I hit the same issue but with mypy 0.920.

Having an import like from namespacepackage import subpackage works and mypy detects possible errors. But having an import like import namespacepackage.subpackage mypy cannot find the type annotations and is telling me error: Skipping analyzing "namespacepackage": module is installed, but missing library stubs or py.typed marker.

Should I open a new issue for this?

fin swimmer

This patch keeps a scoped ignore of the extension itself since MyPy doesn't like namespace packages somehow and it needs to be investigated further [[1]]. [1]: python/mypy#5758

webknjaz · 2023-09-27T00:12:45Z

Looks like there's another corner case: namespace packages in projects with src layout. I've got src/sphinxcontrib/towncrier/ where the package is sphinxcontrib.towncrier and when I'm running mypy with --namespace-packages against tests/, it says that sphinxcontrib.towncrier doesn't expose py.typed.

chrisphilip322 changed the title ~~Importing types from namespace packages doesn't work with out from ... import ...~~ Importing types from namespace packages emits false positives without from ... import ... Oct 9, 2018

gvanrossum mentioned this issue Oct 9, 2018

Add docs for --namespace-packages #5749

Merged

gvanrossum self-assigned this Oct 9, 2018

gvanrossum added bug mypy got something wrong priority-0-high labels Oct 9, 2018

gvanrossum pushed a commit to gvanrossum/mypy that referenced this issue Oct 9, 2018

Fix python#5758 -- plain import didn't work with namespace package

0fa27de

gvanrossum pushed a commit to gvanrossum/mypy that referenced this issue Oct 10, 2018

Possibly a fix for part two of python#5758

8090838

gvanrossum mentioned this issue Oct 10, 2018

Fix #5758 -- plain import didn't work with namespace package #5762

Merged

gvanrossum closed this as completed in 725247b Oct 11, 2018

gvanrossum added a commit that referenced this issue Oct 11, 2018

Fix some edge cases of namespace package imports (#5762)

9d62ac8

Fixes #5758. (Sadly there's no test for the second half of that issue; I've opened #5767 to track adding one.)

chrisphilip322 added a commit to chrisphilip322/mypy that referenced this issue Oct 12, 2018

add testcase which closes python#5758

94c854d

chrisphilip322 mentioned this issue Oct 12, 2018

Refactor testpep561 and add test to address #5767 #5782

Merged

gvanrossum mentioned this issue Oct 13, 2018

Avoid "No such file or directory" if module ancestor doesn't exist #5785

Merged

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Importing types from namespace packages emits false positives without `from ... import ...` #5758

Importing types from namespace packages emits false positives without `from ... import ...` #5758

Importing types from namespace packages emits false positives without from ... import ... #5758

Importing types from namespace packages emits false positives without from ... import ... #5758

Comments

Repro steps

Repro Example

Importing types from namespace packages emits false positives without `from ... import ...` #5758

Importing types from namespace packages emits false positives without `from ... import ...` #5758