8000 parse_flags fails with whitespaces in directories · Issue #4382 · numpy/numpy · GitHub
[go: up one dir, main page]
Skip to content

parse_flags fails with whitespaces in directories #4382

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom
Closed
MichaelAdello opened this issue Feb 27, 2014 · 9 comments · Fixed by #4421
Closed

parse_flags fails with whitespaces in directories #4382

MichaelAdello opened this issue Feb 27, 2014 · 9 comments · Fixed by #4421
Labels
00 - Bug component: numpy.distutils

Comments

@MichaelAdello
Copy link

Scipy failed to build today when using a self-compiled version of numpy, being in a folder that had some whitespaces in it. I could track the issue back to numpy's parse_flags method, which fails if there are whitespaces in the directory.

The following 5 lines of python code will allow to reproduce the issue:

from numpy.distutils.npy_pkg_config import parse_flags
d = parse_flags("-L/my_folder/without_whitespaces/in_it")
print d
d = parse_flags("-L/my_folder/with whitespaces/in_it")
print d
@rgommers rgommers added Defect and removed Defect labels Feb 27, 2014
@ddasilva
Copy link
Contributor
ddasilva commented Mar 2, 2014

I investigated this a little bit.

The parse_flags(line) function uses shlex.shlex from the python stdlib. By default shlex.shlex is rather stupid. We can pass an option to shlex.shlex that will enable POSIX mode, that will allow for smart interpreting of quotes and escaping (as you would expect from a shell). Then you could do any of these:

>>> from numpy.distutils.npy_pkg_config import parse_flags
>>> d = parse_flags('"-L/my_folder/with whitespaces/in_it"')
>>> d = parse_flags('-L/my_folder/"with whitespaces"/in_it')
>>> d = parse_flags('-L/my_folder/with\\ whitespaces/in_it')

The only issue that I see would be that this is not backwards compatible. It would break any code relying on literal interpretation of quotes and backslashes. It would be unusual for someone to be relying on this, so it may not be an issue.

The full details of the parsing rules for POSIX mode and the default mode are documented. If a core developer gives me a go-ahead, I will submit a PR to enable POSIX mode.

@rgommers
Copy link
Member
rgommers commented Mar 2, 2014

I don't think that that will give problems. Please go ahead with the PR I'd say. Besides the above, checking that raw strings work as expected would also be good.

@ddasilva ddasilva mentioned this issue Mar 2, 2014
Merged
@MichaelAdello
Copy link
Author

First of all, great that someone had a look on that!

However, the original problem, namely building scipy with numpy being in a folder with spaces still exists. The reason is, that get_pkg_info from distutils.misc_utils that calls parse_flags calls it with strings like e.g.: "-L/my_folder/with space/in_it -lnpymath -lm" This does however still not work.

So either this needs to return the config properly escaped such that the fixed parse_flags can parse it easily or parse_flags needs to be fixed such that it can deal also with not properly escaped input.

@ddasilva
Copy link
Contributor
ddasilva commented Mar 3, 2014

@MichaelAdello Can you supply the command you are using to build SciPy as well as the directory setup so I can reproduce your problem?

@MichaelAdello
Copy link
Author
  1. Download latest numpy and scipy
  2. Create a folder e.g. $HOME/TEST WITH SPACE/ and copy tarballs in there and unzip them
  3. Create a build subfolder in $HOME/TEST WITH SPACE/
  4. Go to the unzipped numpy folder and run:
    python setup.py install --home="$HOME/TEST WITH SPACE/build"
  5. Now set PYTHONPATH to use the newly build numpy $HOME/TEST WITH SPACE/build/lib/python. In my case this is the only element in the pythonpath.
  6. Now go to the unzipped scipy folder and again run
    python setup.py install --home="$HOME/TEST WITH SPACE/build

The build will fail for two issues:
One problem in parse_flags, that can be seen in the log when it searches for the special functions code location, and later the build will crash because calls to the fortran compiler do not properly escape the input source file location.

@ddasilva
Copy link
Contributor
ddasilva commented Mar 5, 2014

Thanks! I believe I fixed the issue. Can you try the code in PR #4421 and tell me if it works for you?

ddasilva pushed a commit to ddasilva/numpy that referenced this issue Mar 5, 2014
BUG: Fix support for builds in dirs with whitespace
50bf6df 
Add escaping and quoting of dirs and enabled POSIX support in lexer.

Closes numpy#4382.
@MichaelAdello
Copy link
Author

Doesn't crash during the installation process any longer and source files seem to be escaped properly, however I still spotted that error message in the log:

non-existing path in 'scipy/special': "'''/home/mprim/python/folder with space/build/lib/python/numpy/core''/lib' -lnpymath -lm"
non-existing path in 'scipy/special': "'''/home/mprim/python/folder with space/build/lib/python/numpy/core''/include'"

It is around line 100 in the building log, so you should see it also at the beginning. As that's the argument that is passed to parse_flags there might be still some small issue.

@charris charris reopened this Mar 6, 2014
@ddasilva ddasilva mentioned this issue Mar 7, 2014
Closed
@seberg
Copy link
Member
seberg commented Jan 5, 2019

Can't check, but the linked pull request was merged and claims to fix this. Plus it is an old build issue. gh-5765 may also be related.

@seberg seberg closed this as completed Jan 5, 2019
@ddasilva
Copy link
Contributor
ddasilva commented Jan 5, 2019

Closing a issue logged almost 5 years ago! Truly a success story for issue tracking tools ;)

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
00 - Bug component: numpy.distutils
Projects
None yet
Milestone
No milestone
Development

Successfully merging a pull request may close this issue.

BUG: Fix support for builds with directories that contain whitespace ddasilva/numpy
5 participants
@seberg @charris @rgommers @ddasilva @MichaelAdello
0