8000 gh-117587: Add C implementation of `os.path.abspath` by nineteendo · Pull Request #117855 · python/cpython · GitHub
[go: up one dir, main page]

Skip to content

gh-117587: Add C implementation of os.path.abspath #117855

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Closed
wants to merge 87 commits into from
Closed
Changes from 1 commit
Commits
Show all changes
87 commits
Select commit Hold shift + click to select a range
2475af9
Speedup `posixpath.abspath()` for relative paths
nineteendo Apr 12, 2024
807b494
Fix cwd length for byte paths & bus error
nineteendo Apr 12, 2024
aa3ed0c
Merge branch 'python:main' into speedup-posixpath.abspath
nineteendo Apr 12, 2024
fa50a4c
📜🤖 Added by blurb_it.
blurb-it[bot] Apr 13, 2024
f502dd1
Merge branch 'main' into speedup-posixpath.abspath
nineteendo Apr 13, 2024
716f594
Fix `_Py_normpath_and_size` when `size=-1`
nineteendo Apr 13, 2024
cc3a59c
Update Modules/posixmodule.c
nineteendo Apr 13, 2024
f5f2c7b
Update Python/fileutils.c
nineteendo Apr 13, 2024
a1bc125
fastcall & unicode type
nineteendo Apr 13, 2024
ae2c040
Revert changes to `os__path_normpath_impl`
nineteendo Apr 14, 2024
95773d9
simply `os.fsdecode(os.getcwdb())`
nineteendo Apr 14, 2024
b77a15d
1st attempt at full C implementation
nineteendo Apr 14, 2024
f39ab83
Merge branch 'speedup-posixpath.abspath' of https://github.com/ninete…
nineteendo Apr 14, 2024
c63518b
Update Modules/posixmodule.c
nineteendo Apr 14, 2024
94b12c2
2nd attempt at full C implementation
nineteendo Apr 14, 2024
c6c62d2
Merge branch 'speedup-posixpath.abspath' of https://github.com/ninete…
nineteendo Apr 14, 2024
51c9541
3rd attempt at full C implementation
nineteendo Apr 14, 2024
ea0b4d1
Fix tests
nineteendo Apr 14, 2024
8d435d6
Fix tests attempt 2
nineteendo Apr 14, 2024
7057f3b
special case for '' & '.'
nineteendo Apr 15, 2024
fff561e
follow pep 7
nineteendo Apr 15, 2024
15c8f0d
check type with argument clinic
nineteendo Apr 15, 2024
34858a8
Fix memory leak & variable names
nineteendo Apr 16, 2024
21cd24a
Use `PyMem_RawFree`
nineteendo Apr 16, 2024
1113ec3
Update Modules/posixmodule.c
nineteendo Apr 16, 2024
7cca80e
Update Modules/posixmodule.c
nineteendo Apr 16, 2024
0f1f28b
Merge branch 'main' into speedup-posixpath.abspath
eryksun Apr 16, 2024
1313ceb
Merge branch 'main' into speedup-posixpath.abspath
nineteendo Apr 16, 2024
7b6727c
Update Modules/posixmodule.c
nineteendo Apr 17, 2024
97516f6
narrow scoping
nineteendo Apr 17, 2024
7e87ccd
Free `cwd_buf` for all exits
nineteendo Apr 17, 2024
60c99be
Merge branch 'main' into speedup-posixpath.abspath
nineteendo Apr 17, 2024
c06d34f
rename `start` to `prefix_len`
nineteendo Apr 17, 2024
ba14b7d
Merge branch 'main' into speedup-posixpath.abspath
nineteendo Apr 18, 2024
3d2836e
test different cache key
nineteendo Apr 21, 2024
9fcedb5
Merge branch 'main' into speedup-posixpath.abspath
hugovk Apr 23, 2024
111df43
Add C implementation of `ntpath.abspath()`
nineteendo Apr 23, 2024
a538218
Replace with `nt._path_abspath`
nineteendo Apr 23, 2024
d36379d
Update Lib/posixpath.py
nineteendo Apr 23, 2024
6e645f1
Remove fallback
nineteendo Apr 24, 2024
8fd2984
Update Modules/posixmodule.c
nineteendo Apr 24, 2024
4d8eb1d
Follow pep 7
nineteendo Apr 24, 2024
c470123
Check for embedded null characters
nineteendo Apr 25, 2024
3a9af3f
Stricter check for embedded null characters
nineteendo Apr 25, 2024
329692b
Merge branch 'main' into speedup-os.path
nineteendo Apr 25, 2024
9fbc62e
Support qualified referencing
nineteendo Apr 25, 2024
d83a580
Add additional test
nineteendo Apr 25, 2024
08b9674
null in cwd?
nineteendo Apr 25, 2024
6eb1442
Add Windows 11 check
nineteendo Apr 25, 2024
b7f9df6
Add additional test
nineteendo Apr 25, 2024
b127f05
Merge branch 'main' into speedup-posixpath.abspath
nineteendo Apr 26, 2024
3ad0d7f
Fix drive relative paths
nineteendo Apr 26, 2024
549fa4e
Handle all qualified referencing
nineteendo Apr 27, 2024
a768039
Fix ambiguous operation
nineteendo Apr 27, 2024
3339c19
Special case for '.'
nineteendo Apr 27, 2024
0a40599
Handle drive relative paths
nineteendo Apr 28, 2024
2315bbb
Add tests
nineteendo Apr 28, 2024
bbbba83
Merge branch 'main' into speedup-posixpath.abspath
nineteendo Apr 28, 2024
6bac80e
Merge branch 'main' into speedup-posixpath.abspath
nineteendo Apr 30, 2024
b3d04ac
Direct C call for `nt.isdevdrive()`
nineteendo May 4, 2024
415f831
Merge branch 'main' into speedup-posixpath.abspath
nineteendo May 4, 2024
9c55537
Revert "Direct C call for `nt.isdevdrive()`"
nineteendo May 4, 2024
2bf2cd3
Revert "Revert "Direct C call for `nt.isdevdrive()`""
nineteendo May 4, 2024
824f0d2
Restore lost code
nineteendo May 4, 2024
5850c3b
Zero-valued flag enum has no name
nineteendo May 9, 2024
eb26e22
Revert "Zero-valued flag enum has no name"
nineteendo May 9, 2024
bfb235c
Merge branch 'main' into speedup-posixpath.abspath
nineteendo May 27, 2024
d89e875
Direct C call for `posixpath.abspath()`
nineteendo May 28, 2024
66c057f
Optimisation
nineteendo May 28, 2024
8c1e11c
Use root directory for other drives
nineteendo May 28, 2024
50f5882
Fix drive relative paths
nineteendo May 28, 2024
74d7431
Simplify implementation
nineteendo May 28, 2024
807f83f
Update Lib/ntpath.py
nineteendo May 28, 2024
b19bbeb
Improve readability
nineteendo May 28, 2024
f934977
Check embedded null manually
nineteendo May 28, 2024
cf01b90
Revert fallback code on POSIX
nineteendo May 29, 2024
29f9cfc
Update Lib/ntpath.py
nineteendo May 29, 2024
374a39e
Update Lib/test/test_ntpath.py
nineteendo May 29, 2024
adc708d
Apply suggestions from code review
nineteendo May 29, 2024
4d0289a
Narrow scope
nineteendo May 29, 2024
fcffc53
Improve comments
nineteendo May 29, 2024
1ebe5f4
Just check if path is absolute
nineteendo May 29, 2024
e047f03
Narrow scope of `cwd_buf`
nineteendo May 30, 2024
625fc79
Update Python/fileutils.c
nineteendo May 30, 2024
fc3f227
Free memory before calling the `PyErr` APIs
nineteendo May 30, 2024
54c7021
Apply suggestions from code review
nineteendo May 30, 2024
dc66ef9
Use explicit return values
nineteendo May 30, 2024
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
Prev Previous commit
Next Next commit
Merge branch 'main' into speedup-posixpath.abspath
  • Loading branch information
nineteendo committed May 4, 2024
commit 415f831c60e0cf802b162df6d2adc1ef92f6f519
143 changes: 143 additions & 0 deletions Modules/posixmodule.c
Original file line number Diff line number Diff line change
Expand Up @@ -37,6 +37,9 @@
# endif
# include <winioctl.h>
# include <lmcons.h> // UNLEN
# include "osdefs.h" // SEP
# include <aclapi.h> // SetEntriesInAcl
# include <sddl.h> // SDDL_REVISION_1
# if defined(MS_WINDOWS_DESKTOP) || defined(MS_WINDOWS_SYSTEM)
# define HAVE_SYMLINK
# endif /* MS_WINDOWS_DESKTOP | MS_WINDOWS_SYSTEM */
Expand Down Expand Up @@ -4740,7 +4743,7 @@
return NULL;
}

PyObject *abs_obj = posix_abspath((wchar_t *)_path.wide, _path.length, 0);

Check warning on line 4746 in Modules/posixmodule.c

View workflow job for this annotation

GitHub Actions / Windows / build and test (x64)

'posix_abspath' undefined; assuming extern returning int [D:\a\cpython\cpython\PCbuild\_freeze_module.vcxproj]

Check warning on line 4746 in Modules/posixmodule.c

View workflow job for this annotation

GitHub Actions / Windows / build and test (x64)

'initializing': 'PyObject *' differs in levels of indirection from 'int' [D:\a\cpython\cpython\PCbuild\_freeze_module.vcxproj]

Check warning on line 4746 in Modules/posixmodule.c

View workflow job for this annotation

GitHub Actions / Windows / build (arm64)

'posix_abspath' undefined; assuming extern returning int [D:\a\cpython\cpython\PCbuild\_freeze_module.vcxproj]

Check warning on line 4746 in Modules/posixmodule.c

View workflow job for this annotation

GitHub Actions / Windows / build (arm64)

'initializing': 'PyObject *' differs in levels of indirection from 'int' [D:\a\cpython\cpython\PCbuild\_freeze_module.vcxproj]

Check warning on line 4746 in Modules/posixmodule.c

View workflow job for this annotation

GitHub Actions / Windows (free-threading) / build and test (x64)

'posix_abspath' undefined; assuming extern returning int [D:\a\cpython\cpython\PCbuild\_freeze_module.vcxproj]

Check warning on line 4746 in Modules/posixmodule.c

View workflow job for this annotation

GitHub Actions / Windows (free-threading) / build and test (x64)

'initializing': 'PyObject *' differs in levels of indirection from 'int' [D:\a\cpython\cpython\PCbuild\_freeze_module.vcxproj]

Check warning on line 4746 in Modules/posixmodule.c

View workflow job for this annotation

GitHub Actions / Windows (free-threading) / build (arm64)

'posix_abspath' undefined; assuming extern returning int [D:\a\cpython\cpython\PCbuild\_freeze_module.vcxproj]

Check warning on line 4746 in Modules/posixmodule.c

View workflow job for this annotation

GitHub Actions / Windows (free-threading) / build (arm64)

'initializing': 'PyObject *' differs in levels of indirection from 'int' [D:\a\cpython\cpython\PCbuild\_freeze_module.vcxproj]
if (!abs_obj) {
path_cleanup(&_path);
return NULL;
Expand Down Expand Up @@ -5529,8 +5532,148 @@
return result;
}

#ifdef MS_WINDOWS

/* We centralise SECURITY_ATTRIBUTE initialization based around
templates that will probably mostly match common POSIX mode settings.
The _Py_SECURITY_ATTRIBUTE_DATA structure contains temporary data, as
a constructed SECURITY_ATTRIBUTE structure typically refers to memory
that has to be alive while it's being used.

Typical use will look like:
SECURITY_ATTRIBUTES *pSecAttr = NULL;
struct _Py_SECURITY_ATTRIBUTE_DATA secAttrData;
int error, error2;

Py_BEGIN_ALLOW_THREADS
switch (mode) {
case 0x1C0: // 0o700
error = initializeMkdir700SecurityAttributes(&pSecAttr, &secAttrData);
break;
...
default:
error = initializeDefaultSecurityAttributes(&pSecAttr, &secAttrData);
break;
}

if (!error) {
// do operation, passing pSecAttr
}

// Unconditionally clear secAttrData.
error2 = clearSecurityAttributes(&pSecAttr, &secAttrData);
if (!error) {
error = error2;
}
Py_END_ALLOW_THREADS

if (error) {
PyErr_SetFromWindowsErr(error);
return NULL;
}
*/

struct _Py_SECURITY_ATTRIBUTE_DATA {
SECURITY_ATTRIBUTES securityAttributes;
PACL acl;
SECURITY_DESCRIPTOR sd;
EXPLICIT_ACCESS_W ea[4];
char sid[64];
};

static int
initializeDefaultSecurityAttributes(
PSECURITY_ATTRIBUTES *securityAttributes,
struct _Py_SECURITY_ATTRIBUTE_DATA *data
) {
assert(securityAttributes);
assert(data);
*securityAttributes = NULL;
memset(data, 0, sizeof(*data));
return 0;
}

static int
initializeMkdir700SecurityAttributes(
PSECURITY_ATTRIBUTES *securityAttributes,
struct _Py_SECURITY_ATTRIBUTE_DATA *data
) {
assert(securityAttributes);
assert(data);
*securityAttributes = NULL;
memset(data, 0, sizeof(*data));

if (!InitializeSecurityDescriptor(&data->sd, SECURITY_DESCRIPTOR_REVISION)
|| !SetSecurityDescriptorGroup(&data->sd, NULL, TRUE)) {
return GetLastError();
}

int use_alias = 0;
DWORD cbSid = sizeof(data->sid);
if (!CreateWellKnownSid(WinCreatorOwnerRightsSid, NULL, (PSID)data->sid, &cbSid)) {
use_alias = 1;
}

data->securityAttributes.nLength = sizeof(SECURITY_ATTRIBUTES);
data->ea[0].grfAccessPermissions = GENERIC_ALL;
data->ea[0].grfAccessMode = SET_ACCESS;
data->ea[0].grfInheritance = SUB_CONTAINERS_AND_OBJECTS_INHERIT;
if (use_alias) {
data->ea[0].Trustee.TrusteeForm = TRUSTEE_IS_NAME;
data->ea[0].Trustee.TrusteeType = TRUSTEE_IS_ALIAS;
data->ea[0].Trustee.ptstrName = L"CURRENT_USER";
} else {
data->ea[0].Trustee.TrusteeForm = TRUSTEE_IS_SID;
data->ea[0].Trustee.TrusteeType = TRUSTEE_IS_WELL_KNOWN_GROUP;
data->ea[0].Trustee.ptstrName = (LPWCH)(SID*)data->sid;
}

data->ea[1].grfAccessPermissions = GENERIC_ALL;
data->ea[1].grfAccessMode = SET_ACCESS;
data->ea[1].grfInheritance = SUB_CONTAINERS_AND_OBJECTS_INHERIT;
data->ea[1].Trustee.TrusteeForm = TRUSTEE_IS_NAME;
data->ea[1].Trustee.TrusteeType = TRUSTEE_IS_ALIAS;
data->ea[1].Trustee.ptstrName = L"SYSTEM";

data->ea[2].grfAccessPermissions = GENERIC_ALL;
data->ea[2].grfAccessMode = SET_ACCESS;
data->ea[2].grfInheritance = SUB_CONTAINERS_AND_OBJECTS_INHERIT;
data->ea[2].Trustee.TrusteeForm = TRUSTEE_IS_NAME;
data->ea[2].Trustee.TrusteeType = TRUSTEE_IS_ALIAS;
data->ea[2].Trustee.ptstrName = L"ADMINISTRATORS";

int r = SetEntriesInAclW(3, data->ea, NULL, &data->acl);
if (r) {
return r;
}
if (!SetSecurityDescriptorDacl(&data->sd, TRUE, data->acl, FALSE)) {
return GetLastError();
}
data->securityAttributes.lpSecurityDescriptor = &data->sd;
*securityAttributes = &data->securityAttributes;
return 0;
}

static int
clearSecurityAttributes(
PSECURITY_ATTRIBUTES *securityAttributes,
struct _Py_SECURITY_ATTRIBUTE_DATA *data
) {
assert(securityAttributes);
assert(data);
*securityAttributes = NULL;
if (data->acl) {
if (LocalFree((void *)data->acl)) {
return GetLastError();
}
}
return 0;
}

#endif

static PyObject *
posix_abspath(wchar_t *path_buf, Py_ssize_t path_len, int use_bytes)

Check failure on line 5676 in Modules/posixmodule.c

View workflow job for this annotation

GitHub Actions / Windows / build and test (x64)

'posix_abspath': 'PyObject *(wchar_t *,Py_ssize_t,int)' differs in levels of indirection from 'int ()' [D:\a\cpython\cpython\PCbuild\_freeze_module.vcxproj]

Check failure on line 5676 in Modules/posixmodule.c

View workflow job for this annotation

GitHub Actions / Windows / build (arm64)

'posix_abspath': 'PyObject *(wchar_t *,Py_ssize_t,int)' differs in levels of indirection from 'int ()' [D:\a\cpython\cpython\PCbuild\_freeze_module.vcxproj]

Check failure on line 5676 in Modules/posixmodule.c

View workflow job for this annotation

GitHub Actions / Windows (free-threading) / build and test (x64)

'posix_abspath': 'PyObject *(wchar_t *,Py_ssize_t,int)' differs in levels of indirection from 'int ()' [D:\a\cpython\cpython\PCbuild\_freeze_module.vcxproj]

Check failure on line 5676 in Modules/posixmodule.c

View workflow job for this annotation

GitHub Actions / Windows (free-threading) / build (arm64)

'posix_abspath': 'PyObject *(wchar_t *,Py_ssize_t,int)' differs in levels of indirection from 'int ()' [D:\a\cpython\cpython\PCbuild\_freeze_module.vcxproj]
{
Py_ssize_t abs_len;
wchar_t *abs, *abs_buf = NULL, *cwd_buf = NULL;
Expand All @@ -5544,7 +5687,7 @@
goto exit;
}
if (_PyOS_getfullpathname(abs, &abs_buf) < 0) {
result = win32_error_object("GetFullPathNameW", path);

Check failure on line 5690 in Modules/posixmodule.c

View workflow job for this annotation

GitHub Actions / Windows / build and test (x64)

'path': undeclared identifier [D:\a\cpython\cpython\PCbuild\_freeze_module.vcxproj]

Check warning on line 5690 in Modules/posixmodule.c

View workflow job for this annotation

GitHub Actions / Windows / build and test (x64)

'function': 'PyObject *' differs in levels of indirection from 'int' [D:\a\cpython\cpython\PCbuild\_freeze_module.vcxproj]

Check warning on line 5690 in Modules/posixmodule.c

View workflow job for this annotation

GitHub Actions / Windows / build and test (x64)

'win32_error_object': different types for formal and actual parameter 2 [D:\a\cpython\cpython\PCbuild\_freeze_module.vcxproj]

Check failure on line 5690 in Modules/posixmodule.c

View workflow job for this annotation

GitHub Actions / Windows / build (arm64)

'path': undeclared identifier [D:\a\cpython\cpython\PCbuild\_freeze_module.vcxproj]

Check warning on line 5690 in Modules/posixmodule.c

View workflow job for this annotation

GitHub Actions / Windows / build (arm64)

'function': 'PyObject *' differs in levels of indirection from 'int' [D:\a\cpython\cpython\PCbuild\_freeze_module.vcxproj]

Check warning on line 5690 in Modules/posixmodule.c

View workflow job for this annotation

GitHub Actions / Windows / build (arm64)

'win32_error_object': different types for formal and actual parameter 2 [D:\a\cpython\cpython\PCbuild\_freeze_module.vcxproj]

Check failure on line 5690 in Modules/posixmodule.c

View workflow job for this annotation

GitHub Actions / Windows (free-threading) / build and test (x64)

'path': undeclared identifier [D:\a\cpython\cpython\PCbuild\_freeze_module.vcxproj]

Check warning on line 5690 in Modules/posixmodule.c

View workflow job for this annotation

GitHub Actions / Windows (free-threading) / build and test (x64)

'function': 'PyObject *' differs in levels of indirection from 'int' [D:\a\cpython\cpython\PCbuild\_freeze_module.vcxproj]

Check warning on line 5690 in Modules/posixmodule.c

View workflow job for this annotation

GitHub Actions / Windows (free-threading) / build and test (x64)

'win32_error_object': different types for formal and actual parameter 2 [D:\a\cpython\cpython\PCbuild\_freeze_module.vcxproj]

Check failure on line 5690 in Modules/posixmodule.c

View workflow job for this annotation

GitHub Actions / Windows (free-threading) / build (arm64)

'path': undeclared identifier [D:\a\cpython\cpython\PCbuild\_freeze_module.vcxproj]

Check warning on line 5690 in Modules/posixmodule.c

View workflow job for this annotation

GitHub Actions / Windows (free-threading) / build (arm64)

'function': 'PyObject *' differs in levels of indirection from 'int' [D:\a\cpython\cpython\PCbuild\_freeze_module.vcxproj]

Check warning on line 5690 in Modules/posixmodule.c

View workflow job for this annotation

GitHub Actions / Windows (free-threading) / build (arm64)

'win32_error_object': different types for formal and actual parameter 2 [D:\a\cpython\cpython\PCbuild\_freeze_module.vcxproj]
goto exit;
}
abs = abs_buf;
Expand Down Expand Up @@ -5638,7 +5781,7 @@
}
#endif

result = posix_abspath(path_buf, path_len, 0);

Check warning on line 5784 in Modules/posixmodule.c

View workflow job for this annotation

GitHub Actions / Windows / build and test (x64)

'=': 'PyObject *' differs in levels of indirection from 'int' [D:\a\cpython\cpython\PCbuild\_freeze_module.vcxproj]

Check warning on line 5784 in Modules/posixmodule.c

View workflow job for this annotation

GitHub Actions / Windows / build (arm64)

'=': 'PyObject *' differs in levels of indirection from 'int' [D:\a\cpython\cpython\PCbuild\_freeze_module.vcxproj]

Check warning on line 5784 in Modules/posixmodule.c

View workflow job for this annotation

GitHub Actions / Windows (free-threading) / build and test (x64)

'=': 'PyObject *' differs in levels of indirection from 'int' [D:\a\cpython\cpython\PCbuild\_freeze_module.vcxproj]

Check warning on line 5784 in Modules/posixmodule.c

View workflow job for this annotation

GitHub Actions / Windows (free-threading) / build (arm64)

'=': 'PyObject *' differs in levels of indirection from 'int' [D:\a\cpython\cpython\PCbuild\_freeze_module.vcxproj]

exit:
PyMem_Free(path_buf);
Expand Down
Loading
You are viewing a condensed version of this merge commit. You can view the full changes here.
0