Description
What's the problem this feature will solve?
Windows Dev Drive is a new OS feature that allows users to create a high performance disk that's designed for development activities. The performance improvement is based on new optimisations to the file system driver, and separating it out from the default OS drive reduces other overheads.
It's perfect for storing temporary files, however, it was decided that it's not safe for Windows to redirect all TEMP accesses to a Dev Drive by default. But our testing showed the improvements get significantly better when you do it, and we believe apps that can switch ought to switch. Pip's temporary files and package cache featured heavily in our testing.
I was part of the design team for this feature, so happy to add as much more context as you'd like and as I'm able to. Obviously these are public statements, so I have to be careful about things that might be interpreted as promises and not merely dreams, hopes and ambitions 😉
Describe the solution you'd like
At a high level, I'd like to see pip installs targeting an environment on a Dev Drive also use that drive for temporary files, including the build environment.
I'm in the process of adding an os.path.isdevdrive function to CPython for 3.12, so would love to hear whether this is something you'd consider using, and whether a simple test like this suits your needs.1
This is the kind of logic I'd expect to see (around here):
root = None
# sys.prefix feels wrong here, but you're already using site.getsitepackages() in this
# code path, so I guess either we know we're in the target runtime or there's other
# code cleanup to do?
if hasattr(os.path, 'isdevdrive'):
root = os.path.join(''.join(os.path.splitroot(sys.prefix)[:2]), '.pip')
if not os.path.isdevdrive(root): # negative check to clear the overridden root
root = None
path = os.path.realpath(tempfile.mkdtemp(prefix=f"pip-{kind}-", dir=root))
isdevdrive will work on the full path, but I strip back to the drive name first because it could change which drive is actually used (e.g. the project might be in a mounted directory). There's not really an efficient way to handle this case, and on balance it makes the most sense to just ignore the optimisation anyway (access through a mounted directory is typically slow).
We'd also love to see the Dev Drive used for the various caches, and the perf benefits are solid. However, that seems a bit more complex, and might be better served by suggesting users set a global environment variable. There is already an os.listdrives() API that could be used to find a Dev Drive, but in the presence of multiple drives it's not really decidable which one they intend for caching (we think some people will create separate drives for each project, while others will have one big one).
Obviously this can only light up in 3.12 and not earlier (unless you want to port the code into ctypes, which I'll understand if you don't bother). Dev Drive doesn't become widely available until the end of the year anyway, so 3.12 will also be available. Just another reason for people to upgrade! 😄
Alternative Solutions
An alternative would be to detect and suggest to users that since their code is on a Dev Drive, they should also manually override settings to store temporary files on it as well. I'm not sure this would be really valuable for pip users, but it's
Another alternative would be to merely document it. I'd be disappointed by this, because we really want the fact that a Dev Drive is being used to be the signal that dev-related files should be stored there, and we don't want to encourage users to set global environment variables for something like this. It also seems unlikely to be discovered by existing users, who probably aren't going back to read this documentation regularly.
Additional context
If you prefer video demonstrations: https://build.microsoft.com/en-US/sessions/7ed9bb72-b4f4-4490-9b26-911d1ac263d1?source=/home
The docs again: https://learn.microsoft.com/en-us/windows/dev-drive/
Right now it's available on the Dev Channel of Windows Insider, which means it is public and people are trying it out. pip's test suite was one of the key scenarios we were testing during development, and the reason it isn't mentioned in the promotional material is because the perf improvement was so big that it looked like a contrived scenario, so we decided to cut it 😆
Code of Conduct
- I agree to follow the PSF Code of Conduct.
Footnotes
-
Obviously we're past the first beta, but I got an allowance to add it before beta 2 (next Tuesday!) because of NDAs surrounding the feature. ↩