Cannot run tests after `python setup.py build_ext -i` #6569

wackywendell · 2015-10-26T23:48:10Z

Here is what I mean:

$ git clean -xdf
$ python setup.py build_ext -i
Running from numpy source directory.
Cythonizing sources
[... lots of building, completes successfully ...]
$ $ ./runtests.py
Building, see build.log...
Running from numpy source directory.
Traceback (most recent call last):
  File "setup.py", line 263, in <module>
    setup_package()
  File "setup.py", line 247, in setup_package
    from numpy.distutils.core import setup
  File "/home/wendell/code/numpy/numpy/distutils/__init__.py", line 21, in <module>
    from numpy.testing import Tester
  File "/home/wendell/code/numpy/numpy/testing/__init__.py", line 14, in <module>
    from .utils import *
  File "/home/wendell/code/numpy/numpy/testing/utils.py", line 17, in <module>
    from numpy.core import float32, empty, arange, array_repr, ndarray
  File "/home/wendell/code/numpy/numpy/core/__init__.py", line 59, in <module>
    test = Tester().test
  File "/home/wendell/code/numpy/numpy/testing/nosetester.py", line 180, in __init__
    if raise_warnings is None and '.dev0' in np.__version__:
AttributeError: module 'numpy' has no attribute '__version__'

Build failed!

The issue is pretty clear:

When numpy/__init__.py is loaded, under some conditions it will not set the __version__ variable; these conditions occur when you do not use a virtualenv or python setup.py install, and just try and run runtests.py. After 7b438fa (PR), nosetester.py tries to access this __version__ variable, which doesn't exist and so it crashes.

CC @rgommers @charris?

The text was updated successfully, but these errors were encountered:

charris · 2015-10-27T00:36:47Z

Apart from the error, why are you building inplace?

njsmith · 2015-10-27T01:15:00Z

just doing "git clean -xdf; ./runtests.py" should work (the runtests.py
script does both building and testing). It would be nice if build_ext -i
followed by runtests.py worked too, no reason we'd want it to fail in
particular and if someone feels like fixing it then that'd be cool. But
that's probably why you're having a problem, and if you just want to
successfully build and run tests from a git checkout in the quickest way
possible, then you might want to try that...

On Mon, Oct 26, 2015 at 5:36 PM, Charles Harris notifications@github.com
wrote:

Apart from the error, why are you building inplace?

—
Reply to this email directly or view it on GitHub
#6569 (comment).

Nathaniel J. Smith -- http://vorpus.org

wackywendell · 2015-10-27T02:00:56Z

@charris:

The main reason I used python setup.py build_ext -i is because it is suggested in the Development Environment documentation.

Secondly: I could use a virtualenv, but that adds complexity, which --inplace avoids. With --inplace, I can go to a different directory, and just by cd I automatically "reenable" the default numpy.

@njsmith:

Thanks, that does work nicely. I had gotten around it by editing the __init__ function in a hacky way, but using runtests.py to build is a better solution!

charris · 2015-10-27T02:17:25Z

@wackywendell Looks like the documentation could use an update (hint). I suppose inplace builds do have their uses, but generally I've preferred --user. Of course, I never use system numpy anyway ;) But for running tests 'python runtests.py` is quite lovely. Note that you can use different python versions to run the tests and the builds are cached if you just want to fiddle with a test.

What was your hacky fix?

njsmith · 2015-10-27T02:17:36Z

I think that "Building in-place" section should probably be revised or removed or something... certainly the suggestion to do build_ext -i and then add the directory to PYTHONPATH is dangerous without further elaboration (it leads to weird brokenness because it doesn't generate any package metadata, so you end up with a numpy in your environment that pip thinks doesn't exist -> eventually something confusing happens). I don't use build_ext -i or develop at all, so for me it would work just to remove the section or replace it with an admonition not to use those, but I hesitate to do that because I get the impression that other people do find it more useful... @rgommers, any thoughts?

charris · 2015-10-27T02:33:14Z

I think @rkern, among others, used to be a proponent. That was in the early paleolithic age, maybe 7 years ago, and odd artifacts from that primitive time are occasionally discovered in ancient strata.

rkern · 2015-10-27T10:43:11Z

I use develop all the time on things that I actually work on. I haven't worked on numpy or scipy in earnest in a long time, so I don't know how well it still works for these packages. It used to work just fine.

@njsmith If you are using build_ext -i and not develop, all you need to do is also run egg_info to get the package metadata. The instructions probably predate egg_info as a standard distutils command rather than a setuptools extension.

rgommers · 2015-10-27T21:17:13Z

I use in-place builds every day, it's essential functionality for development imho.

Note that using runtests.py is explicitly the recommended way to go: http://docs.scipy.org/doc/numpy-dev/dev/development_environment.html#recommended-development-setup

I don't see the point in removing info on build_ext -i or PYTHONPATH. But some extra note/warning with details can of course be added to the docs.

As an example of why it's useful: imagine that you have the latest numpy release (now 1.10.1) installed, and want to test if your scipy built against 1.10.1 also works when run against master. What's the easiest way: build numpy in-place and add the path to PYTHONPATH. Maybe it's possible with runtests.py (is it?), but I'd expect not and I certainly wouldn't know how.

(and before anyone suggests it: no, virtualenvs are not a valid alternative)

pv · 2015-10-27T22:06:47Z

python runtests.py --shell
(not arguing for or against in-place builds, though)

rgommers · 2015-10-27T22:23:26Z

python runtests.py --shell

That might be of use, thanks. Not sure it helps 100% with my example (because then you're inside an interpreter so cannot run Scipy's runtests.py I think), but it or even beter --ipython are handy.

njsmith · 2015-10-27T22:33:30Z

I'm not saying the section should be deleted, I'm saying it should be
updated to include crucial information about when build_ext -i or whatever
is useful and what the pitfalls are to watch out for, and that I'm not
really in a position to do this given that I don't use it :-)
.
(If you want to test scipy against it, then why not just do a regular
installation of numpy into a venv? Honest question, it seems more
straightforward to me.)
On Oct 27, 2015 3:23 PM, "Ralf Gommers" notifications@github.com wrote:

python runtests.py --shell

That might be of use, thanks. Not sure it helps 100% with my example
(because then you're inside an interpreter so cannot run Scipy's
runtests.py I think), but it or even beter --ipython are handy.

—
Reply to this email directly or view it on GitHub
#6569 (comment).

pv · 2015-10-27T22:43:54Z

@rgommers: it spawns an unix shell, not python. (getting off-topic for this discussion..)

rgommers · 2015-10-27T23:16:36Z

The question about build_ext distracted me from reading the initial bug report. Which is pretty serious. I'll fix it up tomorrow night (unless someone does it sooner).

rgommers · 2015-10-28T21:22:16Z

Wow, this is pretty obscure actually. The code does essentially:

if __NUMPY_SETUP__:
    import sys as _sys
    _sys.stderr.write('Running from numpy source directory.\n')
    del _sys
else:
    from .testing import Tester
    test = Tester().test

And the output of runtests.py and setupegg.py is:

Running from numpy source directory.
...
  File "/home/rgommers/Code/numpy/numpy/testing/nosetester.py", line 180, in __init__
    if raise_warnings is None and '.dev0' in np.__version__:
AttributeError: 'module' object has no attribute '__version__'

It's due to using exec. The exact details of why I see both the if and else clauses of an if-else executed here I don't really want to know - I'm sure it'll be headache-inducing.

rgommers · 2015-10-28T21:38:05Z

Fixing this is less urgent than I thought, because the failure only happens for $ python setup.py build_ext -i followed by one of:

$ python runtests.py
$ python setupegg.py install
$ make dist  # in doc/, uses setupegg.py

We're planning to delete setupegg.py soon, so the only issue is runtests.py. It's a bit too clever for its own good (or for me). I'm not immediately inclined to change what looks like perfectly valid code to work around an issue where one mixes 2 build command that shouldn't really be mixed.

Opinions?

wackywendell · 2015-10-28T21:50:31Z

Maybe you've already figured this out, but that '.dev0' in np.__version__ is from the commit that I think started this (7b438fa); setting either np.__version__ here or adding a not hasattr(np, '__version__') here relieves the issue.

Where is exec coming in? AFAICT, its only taking the first branch of that if-else in __init__.py, and crashing when it gets to the if statement in nosetester.py. In __init__.py, the variable np.__version__ is never set, so then the expression '.dev0' in np.__version__ causes an AttributeError.

I don't understand the details of why setup.py is doing all the things its doing, but why can't we either set np.__version__ for both branches of that if statement here, or add an additional test for not hasattr(np, '__version__') here?

wackywendell · 2015-10-28T22:01:43Z

Also, I believe its triggered by calling python setup.py build_ext -i followed by just about anything; even just calling python setup.py build_ext -i a second time triggers it:

$ git clean -xdf
$ python setup.py build_ext -i
[... normal build, I think ...]
$ python setup.py build_ext -i
Running from numpy source directory.
Traceback (most recent call last):
  File "setup.py", line 263, in <module>
    setup_package()
  File "setup.py", line 247, in setup_package
    from numpy.distutils.core import setup
  File "/home/wendell/code/numpy/numpy/distutils/__init__.py", line 21, in <module>
    from numpy.testing import Tester
  File "/home/wendell/code/numpy/numpy/testing/__init__.py", line 14, in <module>
    from .utils import *
  File "/home/wendell/code/numpy/numpy/testing/utils.py", line 17, in <module>
    from numpy.core import float32, empty, arange, array_repr, ndarray
  File "/home/wendell/code/numpy/numpy/core/__init__.py", line 59, in <module>
    test = Tester().test
  File "/home/wendell/code/numpy/numpy/testing/nosetester.py", line 180, in __init__
    if raise_warnings is None and '.dev0' in np.__version__:
AttributeError: module 'numpy' has no attribute '__version__'

rgommers · 2015-10-28T22:03:32Z

set np.__version__ for both branches of that if statement here,

this isn't possible, because version.py may not exist yet (we're in the building/installing numpy phase)

or add an additional test for not hasattr(np, 'version') here?

that seems to work

its only taking the first branch

yeah I misread the traceback; there's identical test = Tester().test lines in a couple of files.

rgommers · 2015-10-28T22:05:53Z

@wackywendell do you want to submit a PR or should I?

wackywendell · 2015-10-28T22:10:32Z

Either way! You said you'd take it, so I think its yours. I didn't mean to try and take it from you, I just spent a while when I made the report trying to find a way around it.

If you'd prefer, I can take it, although I'm less familiar with the process or the inner workings of numpy.

rgommers · 2015-10-28T22:19:35Z

I didn't mean to try and take it from you

I know, you were only being very helpful:) It'll only take me a couple of minutes, but since you did the hard work I thought that you might like to submit the fix.

wackywendell · 2015-10-28T22:23:13Z

Thanks, actually, I wouldn't mind; it would mean a little to be a bit of a numpy contributor (especially now that I'm on the job market), and I guess another PR to your name would just be a drop in the bucket. I'll submit a PR.

rgommers · 2015-10-28T22:24:00Z

great!

BUG: Fix for #6569, allowing build_ext --inplace

(cherry picked from commit 02b42d4)

charris · 2015-10-30T00:36:22Z

Fixed by a76c275.

* 'master' of https://github.com/numpy/numpy: (384 commits) BUG: fix MANIFEST.in for removal of a file in numpygh-8047. DOC: Release notes for Numpy 1.10.2. MAINT: remove useless files with outdated info from repo root and doc/. MAINT: fix mistake in doc upload rule TST: attempt to make test_load_refcount deterministic BUG: Fix for numpy#6569, allowing build_ext --inplace TST: Added regression test empty percentile, in ref to numpy#6530 and numpy#6553 TST: Added tests for empty partition and argpartition BUG: revert view safety checks TST: Remove tests of view safety checks (see next commit) BUG: Revert some import * fixes in f2py. BUG: Fixed partition errors on empty input. Closes numpy#6530 DOC: import "numpy for matlab users" from the wiki DOC: reorganize user guide a bit + import "tentative numpy tutorial" from wiki DOC: remove placeholders and incompleteness warnings MAINT: minor update to "make upload" doc build command. BUG: error in broadcast_arrays with as_strided array BUG: fix inner() by copying if needed to enforce contiguity DOC: clarify usage of 'argparse' return value. BUG: Make median work for empty arrays (issue numpy#6462) ...

rgommers added component: build 00 - Bug labels Oct 27, 2015

charris added this to the 1.10.2 release milestone Oct 28, 2015

wackywendell added a commit to wackywendell/numpy that referenced this issue Oct 28, 2015

BUG: Fix for numpy#6569, allowing build_ext --inplace

02b42d4

wackywendell mentioned this issue Oct 28, 2015

BUG: Fix for #6569, allowing build_ext --inplace #6577

Merged

rgommers added a commit that referenced this issue Oct 29, 2015

Merge pull request #6577 from wackywendell/buildfix

c7456fe

BUG: Fix for #6569, allowing build_ext --inplace

rgommers pushed a commit to rgommers/numpy that referenced this issue Oct 29, 2015

BUG: Fix for numpy#6569, allowing build_ext --inplace

a76c275

(cherry picked from commit 02b42d4)

charris closed this as completed Oct 30, 2015

jaimefrio pushed a commit to jaimefrio/numpy that referenced this issue Mar 22, 2016

BUG: Fix for numpy#6569, allowing build_ext --inplace

8c03e29

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Uh oh!

Uh oh!

Cannot run tests after `python setup.py build_ext -i` #6569

Cannot run tests after `python setup.py build_ext -i` #6569

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Cannot run tests after python setup.py build_ext -i #6569

Cannot run tests after python setup.py build_ext -i #6569

Comments

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Cannot run tests after `python setup.py build_ext -i` #6569

Cannot run tests after `python setup.py build_ext -i` #6569