8000 gh-108512: Add and use new replacements for PySys_GetObject() by serhiy-storchaka · Pull Request #111035 · python/cpython · GitHub
[go: up one dir, main page]

Skip to content

gh-108512: Add and use new replacements for PySys_GetObject() #111035

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
merged 24 commits into from
May 28, 2025
Merged
Show file tree
Hide file tree
Changes from 10 commits
Commits
Show all changes
24 commits
Select commit Hold shift + click to select a range
4d0f508
gh-108512: Add and use new replacements for PySys_GetObject()
serhiy-storchaka Sep 20, 2023
eb42b39
Update Misc/stable_abi.toml
serhiy-storchaka Oct 18, 2023
2d4588d
Merge branch 'main' into capi-PySys_GetAttr
serhiy-storchaka Oct 18, 2023
2eb9533
Add tests.
serhiy-storchaka Oct 19, 2023
9fc2f3d
Apply suggestions from code review
serhiy-storchaka Oct 19, 2023
65713ce
Address review comments.
serhiy-storchaka Oct 19, 2023
e6ecf11
Check that the name is a string.
serhiy-storchaka Oct 19, 2023
8a0f5f2
Merge remote-tracking branch 'refs/remotes/origin/capi-PySys_GetAttr'…
serhiy-storchaka Oct 19, 2023
516829e
Make the new C API not public.
serhiy-storchaka Oct 19, 2023
9503aaf
Remove from Misc/stable_abi.toml.
serhiy-storchaka Oct 19, 2023
e2857ef
Merge branch 'main' into capi-PySys_GetAttr
serhiy-storchaka Jan 28, 2025
dc26ec2
Add to the limited C API.
serhiy-storchaka Jan 28, 2025
104dcc2
Replace few new occurrences of PySys_GetObject().
serhiy-storchaka Jan 28, 2025
cf75fc3
Update the documentation.
serhiy-storchaka Jan 28, 2025 < 8000 /span>
9434659
Clean up.
serhiy-storchaka Jan 28, 2025
56639d8
Fix a typo.
serhiy-storchaka Jan 28, 2025
b40a665
Merge branch 'main' into capi-PySys_GetAttr
serhiy-storchaka Feb 6, 2025
03b9c0a
Merge branch 'main' into capi-PySys_GetAttr
serhiy-storchaka Feb 6, 2025
5d793c5
Merge remote-tracking branch 'refs/remotes/origin/capi-PySys_GetAttr'…
serhiy-storchaka Feb 6, 2025
09869ed
Merge branch 'main' into capi-PySys_GetAttr
serhiy-storchaka Feb 25, 2025
439bc3c
Merge branch 'main' into capi-PySys_GetAttr
serhiy-storchaka May 21, 2025
93ab31b
Move to 3.15.
serhiy-storchaka May 21, 2025
154a82a
Address review comments.
serhiy-storchaka May 22, 2025
81c7605
Improve tests.
serhiy-storchaka May 22, 2025
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
2 changes: 1 addition & 1 deletion Doc/c-api/init_config.rst
Original file line number Diff line number Diff line change
Expand Up @@ -1417,7 +1417,7 @@ initialization::

/* Specify sys.path explicitly */
/* If you want to modify the default set of paths, finish
initialization first and then use PySys_GetObject("path") */
initialization first and then use PySys_GetAttr("path") */
config.module_search_paths_set = 1;
status = PyWideStringList_Append(&config.module_search_paths,
L"/path/to/stdlib");
Expand Down
49 changes: 47 additions & 2 deletions Doc/c-api/sys.rst
Original file line number Diff line number Diff line change
Expand Up @@ -222,10 +222,55 @@ These are utility functions that make functionality from the :mod:`sys` module
accessible to C code. They all work with the current interpreter thread's
:mod:`sys` module's dict, which is contained in the internal thread state structure.

.. c:function:: PyObject *PySys_GetAttr(PyObject *name)

Get the attribute *name* of the :mod:`sys` module. Return a :term:`strong reference`.
Raise :exc:`RuntimeError` and return ``NULL`` if it does not exist.
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Suggested change
Raise :exc:`RuntimeError` and return ``NULL`` if it does not exist.
Raise :exc:`RuntimeError` and return ``NULL`` if it does not exist or if the :mod:`sys` module cannot be found.


If the non-existing object should not be treated as a failure, you can use
:c:func:`PySys_GetOptionalAttr` instead.

.. versionadded:: 3.13

.. c:function:: PyObject *PySys_GetAttrString(const char *name)

This is the same as :c:func:`PySys_GetAttr`, but *name* is
specified as a :c:expr:`const char*` UTF-8 encoded bytes string,
rather than a :c:expr:`PyObject*`.

If the non-existing object should not be treated as a failure, you can use
:c:func:`PySys_GetOptionalAttrString` instead.

.. versionadded:: 3.13

.. c:function:: int PySys_GetOptionalAttr(PyObject *name, PyObject **result);

Variant of :c:func:`PySys_GetAttr` which doesn't raise
exception if the object does not exist.

If the object exists, set *\*result* to a new :term:`strong reference`
to the object and return ``1``.
If the object does not exist, set *\*result* to ``NULL`` and return ``0``,
without setting an exception.
If other error occurred, set an exception, set *\*result* to ``NULL`` and
return ``-1``.
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I suggest to use a list and start with the return value, so it's easier to see the 3 cases:

Suggested change
If the object exists, set *\*result* to a new :term:`strong reference`
to the object and return ``1``.
If the object does not exist, set *\*result* to ``NULL`` and return ``0``,
without setting an exception.
If other error occurred, set an exception, set *\*result* to ``NULL`` and
return ``-1``.
* Return ``1`` and set *\*result* to a new :term:`strong reference`
to the object if the attribute exists.
* Return ``0`` without setting an exception and set *\*result* to ``NULL``
if the attribute does not exist.
* Return ``-1``, set an exception and set *\*result* to ``NULL``
if an error occurred.

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

"Return and then set exception and variable" looks like a wrong sequence to me. It cannot do anything after returning.

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Just say it in the opposite order in this case:

Set an exception, set *\*result* to ``NULL``, and return ``-1``, if an error occurred.


.. versionadded:: 3.13

.. c:function:: int PySys_GetOptionalAttrString(const char *name, PyObject **result);

This is the same as :c:func:`PySys_GetOptionalAttr`, but *name* is
specified as a :c:expr:`const char*` UTF-8 encoded bytes string,
rather than a :c:expr:`PyObject*`.

.. versionadded:: 3.13

.. c:function:: PyObject *PySys_GetObject(const char *name)

Return the object *name* from the :mod:`sys` module or ``NULL`` if it does
not exist, without setting an exception.
Similar to :c:func:`PySys_GetAttrString`, but return a :term:`borrowed
reference` and return ``NULL`` *without* setting exception on failure.

Preserves exception that was set before the call.

.. c:function:: int PySys_SetObject(const char *name, PyObject *v)

Expand Down
5 changes: 5 additions & 0 deletions Include/cpython/sysmodule.h
Original file line number Diff line number Diff line change
Expand Up @@ -10,6 +10,11 @@ PyAPI_FUNC(int) PySys_Audit(
...);
PyAPI_FUNC(int) PySys_AddAuditHook(Py_AuditHookFunction, void*);

PyAPI_FUNC(PyObject *) PySys_GetAttr(PyObject *);
PyAPI_FUNC(PyObject *) PySys_GetAttrString(const char *);
PyAPI_FUNC(int) PySys_GetOptionalAttr(PyObject *, PyObject **);
PyAPI_FUNC(int) PySys_GetOptionalAttrString(const char *, PyObject **);

typedef struct {
FILE* perf_map;
PyThread_type_lock map_lock;
Expand Down
3 changes: 0 additions & 3 deletions Include/internal/pycore_sysmodule.h
Original file line number Diff line number Diff line change
Expand Up @@ -8,9 +8,6 @@ extern "C" {
# error "this header requires Py_BUILD_CORE define"
#endif

// Export for '_pickle' shared extension
PyAPI_FUNC(PyObject*) _PySys_GetAttr(PyThreadState *tstate, PyObject *name);

// Export for '_pickle' shared extension
PyAPI_FUNC(size_t) _PySys_GetSizeOf(PyObject *);

Expand Down
48 changes: 48 additions & 0 deletions Lib/test/test_capi/test_misc.py
Original file line number Diff line number Diff line change
Expand Up @@ -1098,6 +1098,54 @@ class Data(_testcapi.ObjExtraData):
del d.extra
self.assertIsNone(d.extra)

def test_sys_getattr(self):
sys_getattr = _testcapi.sys_getattr

self.assertIs(sys_getattr('stdout'), sys.stdout)
with support.swap_attr(sys, '\U0001f40d', 42):
self.assertEqual(sys_getattr('\U0001f40d'), 42)

with self.assertRaisesRegex(RuntimeError, r'lost sys\.nonexisting'):
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This error message is surprising. sys has no attribute "nonexisting". It's not "lost", it simply doesn't exist.

I would prefer to always raise AttributeError with a message like "module 'sys' has no attribute 'x'", similar than in Python:

>>> import sys
>>> sys.x
AttributeError: module 'sys' has no attribute 'x'

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

It leaves no use cases for PySys_GetAttr(). They all can be replaced with PySys_GetOptionalAttr() followed by PyErr_SetString(PyExc_RuntimeError,).

If you leave PySys_GetAttr(), you will always use it with PyErr_ExceptionMatches(PyExc_AttributeError) followed by PyErr_SetString(PyExc_RuntimeError,).

sys_getattr('nonexisting')
self.assertRaises(TypeError, sys_getattr, 1)
self.assertRaises(TypeError, sys_getattr, [])
# CRASHES sys_getattr(NULL)

def test_sys_getattrstring(self):
getattrstring = _testcapi.sys_getattrstring

self.assertIs(getattrstring(b'stdout'), sys.stdout)
with support.swap_attr(sys, '\U0001f40d', 42):
self.assertEqual(getattrstring('\U0001f40d'.encode()), 42)

with self.assertRaisesRegex(RuntimeError, r'lost sys\.nonexisting'):
getattrstring(b'nonexisting')
self.assertRaises(UnicodeDecodeError, getattrstring, b'\xff')
# CRASHES getattrstring(NULL)

def test_sys_getoptionalattr(self):
getoptionalattr = _testcapi.sys_getoptionalattr

self.assertIs(getoptionalattr('stdout'), sys.stdout)
with support.swap_attr(sys, '\U0001f40d', 42):
self.assertEqual(getoptionalattr('\U0001f40d'), 42)

self.assertIs(getoptionalattr('nonexisting'), AttributeError)
self.assertRaises(TypeError, getoptionalattr, 1)
self.assertRaises(TypeError, getoptionalattr, [])
# CRASHES getoptionalattr(NULL)

def test_sys_getoptionalattrstring(self):
getoptionalattrstring = _testcapi.sys_getoptionalattrstring

self.assertIs(getoptionalattrstring(b'stdout'), sys.stdout)
with support.swap_attr(sys, '\U0001f40d', 42):
self.assertEqual(getoptionalattrstring('\U0001f40d'.encode()), 42)

self.assertIs(getoptionalattrstring(b'nonexisting'), AttributeError)
self.assertRaises(UnicodeDecodeError, getoptionalattrstring, b'\xff')
# CRASHES getoptionalattrstring(NULL)

def test_sys_getobject(self):
getobject = _testcapi.sys_getobject

Expand Down
Original file line number Diff line number Diff line change
@@ -0,0 +1,2 @@
Add private functions :c:func:`PySys_GetAttr`, :c:func:`PySys_GetAttrString`,
:c:func:`PySys_GetOptionalAttr` and :c:func:`PySys_GetOptionalAttrString`.
26 changes: 5 additions & 21 deletions Modules/_ctypes/callbacks.c
Original file line number Diff line number Diff line change
Expand Up @@ -82,22 +82,6 @@ PyType_Spec cthunk_spec = {

/**************************************************************/

static void
PrintError(const char *msg, ...)
{
char buf[512];
PyObject *f = PySys_GetObject("stderr");
va_list marker;

va_start(marker, msg);
PyOS_vsnprintf(buf, sizeof(buf), msg, marker);
va_end(marker);
if (f != NULL && f != Py_None)
PyFile_WriteString(buf, f);
PyErr_Print();
}


#ifdef MS_WIN32
/*
* We must call AddRef() on non-NULL COM pointers we receive as arguments
Expand All @@ -116,7 +100,7 @@ TryAddRef(StgDictObject *dict, CDataObject *obj)
int r = PyDict_Contains((PyObject *)dict, &_Py_ID(_needs_com_addref_));
if (r <= 0) {
if (r < 0) {
PrintError("getting _needs_com_addref_");
PySys_WriteStderr("getting _needs_com_addref_");
}
return;
}
Expand Down Expand Up @@ -160,7 +144,7 @@ static void _CallPythonObject(void *mem,
if (dict && dict->getfunc && !_ctypes_simple_instance(cnv)) {
PyObject *v = dict->getfunc(*pArgs, dict->size);
if (!v) {
PrintError("create argument %zd:\n", i);
PySys_WriteStderr("create argument %zd:\n", i);
goto Done;
}
args[i] = v;
Expand All @@ -173,12 +157,12 @@ static void _CallPythonObject(void *mem,
/* Hm, shouldn't we use PyCData_AtAddress() or something like that instead? */
CDataObject *obj = (CDataObject *)_PyObject_CallNoArgs(cnv);
if (!obj) {
PrintError("create argument %zd:\n", i);
PySys_WriteStderr("create argument %zd:\n", i);
goto Done;
}
if (!CDataObject_Check(obj)) {
Py_DECREF(obj);
PrintError("unexpected result of create argument %zd:\n", i);
PySys_WriteStderr("unexpected result of create argument %zd:\n", i);
goto Done;
}
memcpy(obj->b_ptr, *pArgs, dict->size);
Expand All @@ -189,7 +173,7 @@ static void _CallPythonObject(void *mem,
} else {
PyErr_SetString(PyExc_TypeError,
"cannot build parameter");
PrintError("Parsing argument %zd\n", i);
PySys_WriteStderr("Parsing argument %zd\n", i);
goto Done;
}
/* XXX error handling! */
Expand Down
14 changes: 8 additions & 6 deletions Modules/_cursesmodule.c
Original file line number Diff line number Diff line change
Expand Up @@ -3373,17 +3373,19 @@ _curses_setupterm_impl(PyObject *module, const char *term, int fd)
if (fd == -1) {
PyObject* sys_stdout;

sys_stdout = PySys_GetObject("stdout");
sys_stdout = PySys_GetAttrString("stdout");
if (sys_stdout == NULL) {
return NULL;
}

if (sys_stdout == NULL || sys_stdout == Py_None) {
PyErr_SetString(
PyCursesError,
"lost sys.stdout");
if (sys_stdout == Py_None) {
PyErr_SetString(PyCursesError, "lost sys.stdout");
Py_DECREF(sys_stdout);
return NULL;
}

fd = PyObject_AsFileDescriptor(sys_stdout);

Py_DECREF(sys_stdout);
if (fd == -1) {
return NULL;
}
Expand Down
6 changes: 3 additions & 3 deletions Modules/_lsprof.c
Original file line number Diff line number Diff line change
Expand Up @@ -716,7 +716,7 @@ profiler_enable(ProfilerObject *self, PyObject *args, PyObject *kwds)
return NULL;
}

PyObject* monitoring = _PyImport_GetModuleAttrString("sys", "monitoring");
PyObject* monitoring = PySys_GetAttrString("monitoring");
if (!monitoring) {
return NULL;
}
Expand Down Expand Up @@ -778,7 +778,7 @@ profiler_disable(ProfilerObject *self, PyObject* noarg)
{
if (self->flags & POF_ENABLED) {
PyObject* result = NULL;
PyObject* monitoring = _PyImport_GetModuleAttrString("sys", "monitoring");
PyObject* monitoring = PySys_GetAttrString("monitoring");

if (!monitoring) {
return NULL;
Expand Down Expand Up @@ -880,7 +880,7 @@ profiler_init(ProfilerObject *pObj, PyObject *args, PyObject *kw)
Py_XSETREF(pObj->externalTimer, Py_XNewRef(timer));
pObj->tool_id = PY_MONITORING_PROFILER_ID;

PyObject* monitoring = _PyImport_GetModuleAttrString("sys", "monitoring");
PyObject* monitoring = PySys_GetAttrString("monitoring");
if (!monitoring) {
return -1;
}
Expand Down
13 changes: 9 additions & 4 deletions Modules/_pickle.c
Original file line number Diff line number Diff line change
Expand Up @@ -17,7 +17,7 @@
#include "pycore_pystate.h" // _PyThreadState_GET()
#include "pycore_runtime.h" // _Py_ID()
#include "pycore_setobject.h" // _PySet_NextEntry()
#include "pycore_sysmodule.h" // _PySys_GetAttr()
#include "pycore_sysmodule.h" // _PySys_GetSizeOf()

#include <stdlib.h> // strtol()

Expand Down Expand Up @@ -1986,49 +1986,54 @@ whichmodule(PyObject *global, PyObject *dotted_path)
assert(module_name == NULL);

/* Fallback on walking sys.modules */
PyThreadState *tstate = _PyThreadState_GET();
modules = _PySys_GetAttr(tstate, &_Py_ID(modules));
modules = PySys_GetAttr(&_Py_ID(modules));
if (modules == NULL) {
PyErr_SetString(PyExc_RuntimeError, "unable to get sys.modules");
return NULL;
}
if (PyDict_CheckExact(modules)) {
i = 0;
while (PyDict_Next(modules, &i, &module_name, &module)) {
if (_checkmodule(module_name, module, global, dotted_path) == 0) {
Py_DECREF(modules);
return Py_NewRef(module_name);
}
if (PyErr_Occurred()) {
Py_DECREF(modules);
return NULL;
}
}
}
else {
PyObject *iterator = PyObject_GetIter(modules);
if (iterator == NULL) {
Py_DECREF(modules);
return NULL;
}
while ((module_name = PyIter_Next(iterator))) {
module = PyObject_GetItem(modules, module_name);
if (module == NULL) {
Py_DECREF(module_name);
Py_DECREF(iterator);
Py_DECREF(modules);
return NULL;
}
if (_checkmodule(module_name, module, global, dotted_path) == 0) {
Py_DECREF(module);
Py_DECREF(iterator);
Py_DECREF(modules);
return module_name;
}
Py_DECREF(module);
Py_DECREF(module_name);
if (PyErr_Occurred()) {
Py_DECREF(iterator);
Py_DECREF(modules);
return NULL;
}
}
Py_DECREF(iterator);
}
Py_DECREF(modules);

/* If no module is found, use __main__. */
return &_Py_ID(__main__);
Expand Down
Loading
0