Cannot create venvs on small-xattr filesystems when CPython is installed somewhere with large xattrs (Regression from 3.12) #127638
Labels
3.13
bugs and security fixes
3.14
bugs and security fixes
topic-venv
Related to the venv module
type-bug
An unexpected behavior, bug, or error
Comments
|
There's a CVMFS feature that is so far undocumented called |
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Bug report
Description
When setting up a virtual environment, the
venvpackage copies its scripts to the createdvenv/bindirectory. In Python 3.12, the source file is read and the contents are written to the destination file. However, in Python 3.13, the script also attempts to copy the source file's extended file attributes to the destination file. This behaviour can be seen observed usingstrace.When CPython is installed on a filesystem which contains large extended attributes (such as CVMFS, more below), and the venv is created on a filesystem which only supports
xattrvalues one block big (usually 4KiB, such as ext3/4 or btrfs), an error arises when Python fails to write the extended attributes:This error message is misleading, as the filesystem behind the quoted path has lots of free space. The error message actually refers to the extended attribute value being longer than the maximum allowed. It is clear in
stracethat this error is returned by thesetxattrsystem call:Cause
This change in behaviour is due to the changes in this commit, from line 461 onwards in
Lib/venv/__init__.py. The procedure for writing the destination files previously wrote the file directly and copied the file mode usingshutil.copymode, which does not copy extended attributes. After the commit, if no replacements were made in the source file,shutil.copy2is called, which in turn callsshutil.copystatwhich does copy extended attributes.This commit was merged as part of #113129.
Impact on CVMFS users
CernVM-FS (CVMFS) is a
fuse-based filesystem used to distribute software in the scientific computing community and HPC environments. It uses extended attributes to provide information about the filesystem. Its extended attributes apply to every file - they do not belong to the file but are generated by the filesystem when requested. For example, theuser.pubkeysattribute contains the public keys used to sign the filesystem contents, which can easily be tens of kilobytes - way more than can fit in anext4extended attribute. There is no need for these attributes to be copied to the venv.This bug means that Python 3.13
venvs are broken for the those using CVMFS to distribute their software and usingext/btrfslocally, which I expect will cause problems adopting Python 3.13 for the scientific computing and HPC community.Proposed fix
It is not necessary for the extended attributes to be copied from Python's installation directory to the
venvdirectory - Python does not and should not rely on them to function properly. Therefore, my proposed fix is simply to eliminate this if statement in thevenvpackage. The code currently in theelsebranch should always run, as was the case in Python 3.12.This change can be hacked into effect using
sed -i 's/new_data == data:/False and new_data == data:/' /venv/__init__.py, and with this patch the extended attributes are no longer copied and thevenvis created successfully.CPython versions tested on:
3.13
Operating systems tested on:
Linux
(cc @vsajip as author of stdlib
venv)Linked PRs
The text was updated successfully, but these errors were encountered: