8000 Cannot run tests after `python setup.py build_ext -i` · Issue #6569 · numpy/numpy · GitHub
[go: up one dir, main page]

Skip to content

Cannot run tests after python setup.py build_ext -i #6569

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Closed
wackywendell opened this issue Oct 26, 2015 · 24 comments
Closed

Cannot run tests after python setup.py build_ext -i #6569

wackywendell opened this issue Oct 26, 2015 · 24 comments

Comments

@wackywendell
Copy link
Contributor

Here is what I mean:

$ git clean -xdf
$ python setup.py build_ext -i
Running from numpy source directory.
Cythonizing sources
[... lots of building, completes successfully ...]
$ $ ./runtests.py
Building, see build.log...
Running from numpy source directory.
Traceback (most recent call last):
  File "setup.py", line 263, in <module>
    setup_package()
  File "setup.py", line 247, in setup_package
    from numpy.distutils.core import setup
  File "/home/wendell/code/numpy/numpy/distutils/__init__.py", line 21, in <module>
    from numpy.testing import Tester
  File "/home/wendell/code/numpy/numpy/testing/__init__.py", line 14, in <module>
    from .utils import *
  File "/home/wendell/code/numpy/numpy/testing/utils.py", line 17, in <module>
    from numpy.core import float32, empty, arange, array_repr, ndarray
  File "/home/wendell/code/numpy/numpy/core/__init__.py", line 59, in <module>
    test = Tester().test
  File "/home/wendell/code/numpy/numpy/testing/nosetester.py", line 180, in __init__
    if raise_warnings is None and '.dev0' in np.__version__:
AttributeError: module 'numpy' has no attribute '__version__'

Build failed!

The issue is pretty clear:

When numpy/__init__.py is loaded, under some conditions it will not set the __version__ variable; these conditions occur when you do not use a virtualenv or python setup.py install, and just try and run runtests.py. After 7b438fa (PR), nosetester.py tries to access this __version__ variable, which doesn't exist and so it crashes.

CC @rgommers @charris?

@charris
Copy link
Member
charris commented Oct 27, 2015

Apart from the error, why are you building inplace?

@njsmith
Copy link
Member
njsmith commented Oct 27, 2015

just doing "git clean -xdf; ./runtests.py" should work (the runtests.py
script does both building and testing). It would be nice if build_ext -i
followed by runtests.py worked too, no reason we'd want it to fail in
particular and if someone feels like fixing it then that'd be cool. But
that's probably why you're having a problem, and if you just want to
successfully build and run tests from a git checkout in the quickest way
possible, then you might want to try that...

On Mon, Oct 26, 2015 at 5:36 PM, Charles Harris notifications@github.com
wrote:

Apart from the error, why are you building inplace?


Reply to this email directly or view it on GitHub
#6569 (comment).

Nathaniel J. Smith -- http://vorpus.org

@wackywendell
Copy link
Contributor Author

@charris:

The main reason I used python setup.py build_ext -i is because it is suggested in the Development Environment documentation.

Secondly: I could use a virtualenv, but that adds complexity, which --inplace avoids. With --inplace, I can go to a different directory, and just by cd I automatically "reenable" the default numpy.

@njsmith:

Thanks, that does work nicely. I had gotten around it by editing the __init__ function in a hacky way, but using runtests.py to build is a better solution!

@charris
Copy link
Member
charris commented Oct 27, 2015

@wackywendell Looks like the documentation could use an update (hint). I suppose inplace builds do have their uses, but generally I've preferred --user. Of course, I never use system numpy anyway ;) But for running tests 'python runtests.py` is quite lovely. Note that you can use different python versions to run the tests and the builds are cached if you just want to fiddle with a test.

What was your hacky fix?

@njsmith
Copy link
Member
njsmith commented Oct 27, 2015

I think that "Building in-place" section should probably be revised or removed or something... certainly the suggestion to do build_ext -i and then add the directory to PYTHONPATH is dangerous without further elaboration (it leads to weird brokenness because it doesn't generate any package metadata, so you end up with a numpy in your environment that pip thinks doesn't exist -> eventually something confusing happens). I don't use build_ext -i or develop at all, so for me it would work just to remove the section or replace it with an admonition not to use those, but I hesitate to do that because I get the impression that other people do find it more useful... @rgommers, any thoughts?

@charris
Copy link
Member
charris commented Oct 27, 2015

I think @rkern, among others, used to be a proponent. That was in the early paleolithic age, maybe 7 years ago, and odd artifacts from that primitive time are occasionally discovered in ancient strata.

@rkern
Copy link
Member
rkern commented Oct 27, 2015

I use develop all the time on things that I actually work on. I haven't worked on numpy or scipy in earnest in a long time, so I don't know how well it still works for these packages. It used to work just fine.

@njsmith If you are using build_ext -i and not develop, all you need to do is also run egg_info to get the package metadata. The instructions probably predate egg_info as a standard distutils command rather than a setuptools extension.

@rgommers
Copy link
Member

I use in-place builds every day, it's essential functionality for development imho.

Note that using runtests.py is explicitly the recommended way to go: http://docs.scipy.org/doc/numpy-dev/dev/development_environment.html#recommended-development-setup

I don't see the point in removing info on build_ext -i or PYTHONPATH. But some extra note/warning with details can of course be added to the docs.

As an example of why it's useful: imagine that you have the latest numpy release (now 1.10.1) installed, and want to test if your scipy built against 1.10.1 also works when run against master. What's the easiest way: build numpy in-place and add the path to PYTHONPATH. Maybe it's possible with runtests.py (is it?), but I'd expect not and I certainly wouldn't know how.

(and before anyone suggests it: no, virtualenvs are not a valid alternative)

@pv
Copy link
Member
pv commented Oct 27, 2015

python runtests.py --shell
(not arguing for or against in-place builds, though)

@rgommers
Copy link
Member

python runtests.py --shell

That might be of use, thanks. Not sure it helps 100% with my example (because then you're inside an interpreter so cannot run Scipy's runtests.py I think), but it or even beter --ipython are handy.

@njsmith
Copy link
Member
njsmith commented Oct 27, 2015

I'm not saying the section should be deleted, I'm saying it should be
updated to include crucial information about when build_ext -i or whatever
is useful and what the pitfalls are to watch out for, and that I'm not
really in a position to do this given that I don't use it :-)
.
(If you want to test scipy against it, then why not just do a regular
installation of numpy into a venv? Honest question, it seems more
straightforward to me.)
On Oct 27, 2015 3:23 PM, "Ralf Gommers" notifications@github.com wrote:

python runtests.py --shell

That might be of use, thanks. Not sure it helps 100% with my example
(because then you're inside an interpreter so cannot run Scipy's
runtests.py I think), but it or even beter --ipython are handy.


Reply to this email directly or view it on GitHub
#6569 (comment).

@pv
Copy link
Member
pv commented Oct 27, 2015

@rgommers: it spawns an unix shell, not python. (getting off-topic for this discussion..)

@rgommers
Copy link
Member

The question about build_ext distracted me from reading the initial bug report. Which is pretty serious. I'll fix it up tomorrow night (unless someone does it sooner).

@charris charris added this to the 1.10.2 release milestone Oct 28, 2015
@rgommers
Copy link
Member

Wow, this is pretty obscure actually. The code does essentially:

if __NUMPY_SETUP__:
    import sys as _sys
    _sys.stderr.write('Running from numpy source directory.\n')
    del _sys
else:
    from .testing import Tester
    test = Tester().test

And the output of runtests.py and setupegg.py is:

Running from numpy source directory.
...
  File "/home/rgommers/Code/numpy/numpy/testing/nosetester.py", line 180, in __init__
    if raise_warnings is None and '.dev0' in np.__version__:
AttributeError: 'module' object has no attribute '__version__'

It's due to using exec. The exact details of why I see both the if and else clauses of an if-else executed here I don't really want to know - I'm sure it'll be headache-inducing.

@rgommers
Copy link
Member

Fixing this is less urgent than I thought, because the failure only happens for $ python setup.py build_ext -i followed by one of:

$ python runtests.py
$ python setupegg.py install
$ make dist  # in doc/, uses setupegg.py

We're planning to delete setupegg.py soon, so the only issue is runtests.py. It's a bit too clever for its own good (or for me). I'm not immediately inclined to change what looks like perfectly valid code to work around an issue where one mixes 2 build command that shouldn't really be mixed.

Opinions?

@wackywendell
Copy link
Contributor Author

Maybe you've already figured this out, but that '.dev0' in np.__version__ is from the commit that I think started this (7b438fa); setting either np.__version__ here or adding a not hasattr(np, '__version__') here relieves the issue.

Where is exec coming in? AFAICT, its only taking the first branch of that if-else in __init__.py, and crashing when it gets to the if statement in nosetester.py. In __init__.py, the variable np.__version__ is never set, so then the expression '.dev0' in np.__version__ causes an AttributeError.

I don't understand the details of why setup.py is doing all the things its doing, but why can't we either set np.__version__ for both branches of that if statement here, or add an additional test for not hasattr(np, '__version__') here?

@wackywendell
Copy link
Contributor Author

Also, I believe its triggered by calling python setup.py build_ext -i followed by just about anything; even just calling python setup.py build_ext -i a second time triggers it:

$ git clean -xdf
$ python setup.py build_ext -i
[... normal build, I think ...]
$ python setup.py build_ext -i
Running from numpy source directory.
Traceback (most recent call last):
  File "setup.py", line 263, in <module>
    setup_package()
  File "setup.py", line 247, in setup_package
    from numpy.distutils.core import setup
  File "/home/wendell/code/numpy/numpy/distutils/__init__.py", line 21, in <module>
    from numpy.testing import Tester
  File "/home/wendell/code/numpy/numpy/testing/__init__.py", line 14, in <module>
    from .utils import *
  File "/home/wendell/code/numpy/numpy/testing/utils.py", line 17, in <module>
    from numpy.core import float32, empty, arange, array_repr, ndarray
  File "/home/wendell/code/numpy/numpy/core/__init__.py", line 59, in <module>
    test = Tester().test
  File "/home/wendell/code/numpy/numpy/testing/nosetester.py", line 180, in __init__
    if raise_warnings is None and '.dev0' in np.__version__:
AttributeError: module 'numpy' has no attribute '__version__'

@rgommers
Copy link
Member

set np.__version__ for both branches of that if statement here,

this isn't possible, because version.py may not exist yet (we're in the building/installing numpy phase)

or add an additional test for not hasattr(np, 'version') here?

that seems to work

its only taking the first branch

yeah I misread the traceback; there's identical test = Tester().test lines in a couple of files.

@rgommers
Copy link
Member

@wackywendell do you want to submit a PR or should I?

@wackywendell
Copy link
Contributor Author

Either way! You said you'd take it, so I think its yours. I didn't mean to try and take it from you, I just spent a while when I made the report trying to find a way around it.

If you'd prefer, I can take it, although I'm less familiar with the process or the inner workings of numpy.

@rgommers
Copy link
Member

I didn't mean to try and take it from you

I know, you were only being very helpful:) It'll only take me a couple of minutes, but since you did the hard work I thought that you might like to submit the fix.

@wackywendell
Copy link
Contributor Author

Thanks, actually, I wouldn't mind; it would mean a little to be a bit of a numpy contributor (especially now that I'm on the job market), and I guess another PR to your name would just be a drop in the bucket. I'll submit a PR.

@rgommers
Copy link
Member

great!

wackywendell added a commit to wackywendell/numpy that referenced this issue Oct 28, 2015
rgommers added a commit that referenced this issue Oct 29, 2015
BUG: Fix for #6569, allowing build_ext --inplace
rgommers pushed a commit to rgommers/numpy that referenced this issue Oct 29, 2015
@charris
Copy link
Member
charris commented Oct 30, 2015

Fixed by a76c275.

@charris charris closed this as completed Oct 30, 2015
tpn added a commit to pyparallel/numpy that referenced this issue Oct 30, 2015
* 'master' of https://github.com/numpy/numpy: (384 commits)
  BUG: fix MANIFEST.in for removal of a file in numpygh-8047.
  DOC: Release notes for Numpy 1.10.2.
  MAINT: remove useless files with outdated info from repo root and doc/.
  MAINT: fix mistake in doc upload rule
  TST: attempt to make test_load_refcount deterministic
  BUG: Fix for numpy#6569, allowing build_ext --inplace
  TST: Added regression test empty percentile, in ref to numpy#6530 and numpy#6553
  TST: Added tests for empty partition and argpartition
  BUG: revert view safety checks
  TST: Remove tests of view safety checks (see next commit)
  BUG: Revert some import * fixes in f2py.
  BUG: Fixed partition errors on empty input. Closes numpy#6530
  DOC: import "numpy for matlab users" from the wiki
  DOC: reorganize user guide a bit + import "tentative numpy tutorial" from wiki
  DOC: remove placeholders and incompleteness warnings
  MAINT: minor update to "make upload" doc build command.
  BUG: error in broadcast_arrays with as_strided array
  BUG: fix inner() by copying if needed to enforce contiguity
  DOC: clarify usage of 'argparse' return value.
  BUG: Make median work for empty arrays (issue numpy#6462)
  ...
jaimefrio pushed a commit to jaimefrio/numpy that referenced this issue Mar 22, 2016
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Development

No branches or pull requests

6 participants
0