Add path used by pip's build isolation procedure to DLL search #131340
Conversation
Without this, trying to `import torch` in a downstream `setup.py` file would result in ``` The specified module could not be found. Error loading "C:\...\pip-build-env-himl3xh3\normal\Lib\site-packages\torch\lib\shm.dll" or one of its dependencies." ``` This seems to be because pip does not use a full virtualenv for build isolation, instead creating directories and manually adding them to `sys.path`
🔗 Helpful Links🧪 See artifacts and rendered test results at hud.pytorch.org/pr/131340
Note: Links to docs will display an error until the docs builds have been completed. ❌ 1 New Failure, 1 Cancelled JobAs of commit f2b49a9 with merge base 0246b28 ( NEW FAILURE - The following job has failed:
This comment was automatically generated by Dr. CI and updates every 15 minutes. |
|
For completeness, we are using |
| # torch & mkl in a custom path that's included in sys.path but does not match | ||
| # any of the other DLLs path above, so we need to manually add it here. | ||
| mkl_dll_path = os.path.join( | ||
| os.path.dirname(__file__), "..", "..", "..", "Library", "bin" |
There was a problem hiding this comment.
This feels wrong, I would prefer it to be expressed as sysconfig.get_config_var("XYZ") + something else, because during the normal installation this will add a search path to some random folder, that could be used to mess with some existing installs
There was a problem hiding this comment.
I had a look through what's in sysconfig.get_config_vars(), but this was not here. I think this is really a specific of how pip does build isolation. Another solution might be to explicitly search for mkl distribution and use this path, but I could not find a way to find it other than walking through sys.path and looking for mkl.dist-info (import mkl fails since the package does not install any python files)
|
@malfet anything I can do to help merge this? |
|
Looks like this PR hasn't been updated in a while so we're going to go ahead and mark this as |
|
@malfet could you please remove the stale label & let me know how I can help to get this through? |
Without this, trying to
import torchin a downstreamsetup.pyfile would result inThis seems to be because pip does not use a full virtualenv for build isolation, instead creating directories and manually adding them to
sys.path. The same issue does not seem to apply when usingpython -m build.To reproduce, you can create a directory with two files:
Then, trying to build a wheel with
pip install .will give some output similar to:Torch is properly installed in
C:\Users\runneradmin\AppData\Local\Temp\pip-build-env-himl3xh3\normal\Lib\site-packages\torch\and all the mkl libraries are inC:\Users\runneradmin\AppData\Local\Temp\pip-build-env-himl3xh3\normal\Library\bin, but this directory is not covered by existing DLL paths.This is similar to #125109, and the fix is similar to #125684. Ping @atalman and @malfet since you fixed & reviewed the previous similar fix.
cc @malfet @seemethere @peterjc123 @mszhanyi @skyline75489 @nbcsm @vladimir-aubrecht @iremyux @Blackhex @cristianPanaite