10BC0 Stackless issue #139: enable pickling of asynchronous generators · akruis/cpython@80b3f97 · GitHub
[go: up one dir, main page]

Skip to content

Commit 80b3f97

Browse files
committed
Stackless issue python#139: enable pickling of asynchronous generators
Stackless now pickles asynchronous generators. Read the documentation about Stackless pickling for details.
1 parent 16328bc commit 80b3f97

File tree

12 files changed

+453
-14
lines changed

12 files changed

+453
-14
lines changed

Doc/library/stackless/pickling.rst

Lines changed: 40 additions & 5 deletions
Original file line numberDiff line numberDiff line change
@@ -4,7 +4,7 @@
44
Pickling --- Serialisation of running tasklets
55
**********************************************
66

7-
One of the most impressive features of Stackless, is the ability to pickle
7+
One of the most impressive features of |SLP|, is the ability to pickle
88
tasklets. This allows you to take a tasklet mid-execution, serialise it to
99
a chunk of data and then unserialise that data at a later point, creating a
1010
new tasklet from it that resumes where the last left off.
@@ -98,7 +98,7 @@ different address than *t1*, which was displayed earlier.
9898
It should be possible to pickle any tasklets that you might want to.
9999
However, not all tasklets can be unpickled. One of the cases in which
100100
this is true, is where not all the functions called by the code within
101-
the tasklet are |PY| functions. The Stackless pickling mechanism
101+
the tasklet are |PY| functions. The |SLP| pickling mechanism
102102
has no ability to deal with C functions that may have been called.
103103

104104
.. note::
@@ -112,11 +112,12 @@ different address than *t1*, which was displayed earlier.
112112
Pickling other objects
113113
======================
114114

115-
In order to be able to pickle tasklets Stackless needs to be able to pickle
115+
In order to be able to pickle tasklets |SLP| needs to be able to pickle
116116
several other objects, which can't be pickled by |CPY|. |SLP|
117117
uses :func:`copyreg.pickle` to register “reduction” functions for the following
118118
types:
119119
:data:`~types.FunctionType`,
120+
:data:`~types.AsyncGeneratorType`,
120121
:data:`~types.CodeType`,
121122
:data:`~types.CoroutineType`,
122123
:data:`~types.GeneratorType`,
@@ -130,7 +131,41 @@ Frames
130131

131132
|SLP| can pickle frames, but only as part of a
132133
tasklet, a traceback-object, a generator, a coroutine or an asynchronous
133-
generator. Stackless does not register a "reduction" function for
134-
:data:`~types.FrameType`. This way Stackless stays compatible with application
134+
generator. |SLP| does not register a "reduction" function for
135+
:data:`~types.FrameType`. This way |SLP| stays compatible with application
135136
code that registers its own "reduction" function for :data:`~types.FrameType`.
136137

138+
.. _slp_pickling_asyncgen:
139+
140+
Asynchronous Generators
141+
=======================
142+
143+
.. versionadded:: 3.7
144+
145+
At C-level asynchronous generators have an attribute ``ag_finalizer`` and a flag,
146+
if ag_finalizer has been initialised. The value of ``ag_finalizer`` is a callable
147+
|PY|-object, which has been set by :func:`sys.set_asyncgen_hooks`.
148+
You can use :func:`stackless.pickle_flags` to control how |SLP| pickles and
149+
unpickles an asynchronous generator.
150+
151+
Pickling
152+
--------
153+
154+
By default (no flags set) |SLP| does not pickle ``ag_finalizer`` but a
155+
marker, if a ``ag_finalizer`` has been set.
156+
If :const:`~stackless.PICKLEFLAGS_PRESERVE_AG_FINALIZER` has been set,
157+
|SLP| pickles ``ag_finalizer`` by value.
158+
Otherwise, if :const:`~stackless.PICKLEFLAGS_RESET_AG_FINALIZER` has
159+
been set, |SLP| pickles ``ag_finalizer`` as uninitialised.
160+
161+
Unpickling
162+
----------
163+
164+
By default |SLP| initialises the generator upon unpickling using the
165+
``firstiter`` and ``finalizer`` values set by :func:`sys.set_asyncgen_hooks`,
166+
if ``ag_finalizer`` of the original asynchronous generator was initialised.
167+
If :const:`~stackless.PICKLEFLAGS_PRESERVE_AG_FINALIZER` has been set and if
168+
``ag_finalizer`` has been pickled by value, |SLP| unpickles
169+
``ag_finalizer`` by value.
170+
Otherwise, if :const:`~stackless.PICKLEFLAGS_RESET_AG_FINALIZER` has
171+
been set, |SLP| unpickles ``ag_finalizer`` as uninitialised.

Doc/library/stackless/stackless.rst

Lines changed: 17 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -37,6 +37,20 @@ Constants
3737
.. versionadded:: 3.7
3838

3939

40+
.. data:: PICKLEFLAGS_PRESERVE_AG_FINALIZER
41+
PICKLEFLAGS_RESET_AG_FINALIZER
42+
43+
These two constants define the option flags for the function
44+
:func:`pickle_flags`.
45+
46+
.. seealso:: :ref:`Pickling of Asynchronous Generators <slp_pickling_asyncgen>`
47+
48+
.. versionadded:: 3.7
49+
50+
.. note::
51+
These constants have been added on a provisional basis (see :pep:`411`
52+
for DB68 details.)
53+
4054
---------
4155
Functions
4256
---------
@@ -302,7 +316,9 @@ Pickling related functions:
302316

303317
Currently the following pickle option flags are defined:
304318

305-
- bit 0: :const:`PICKLEFLAGS_PRESERVE_TRACING_STATE`.
319+
- bit 0: :const:`PICKLEFLAGS_PRESERVE_TRACING_STATE`;
320+
- bit 1: :const:`PICKLEFLAGS_PRESERVE_AG_FINALIZER`;
321+
- bit 2: :const:`PICKLEFLAGS_RESET_AG_FINALIZER`.
306322

307323
All other bits must be set to 0.
308324

Lib/stackless.py

Lines changed: 2 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -22,6 +22,8 @@ def __reduce_ex__(*args):
2222
return "stackless"
2323

2424
PICKLEFLAGS_PRESERVE_TRACING_STATE = 1
25+
PICKLEFLAGS_PRESERVE_AG_FINALIZER = 2
26+
PICKLEFLAGS_RESET_AG_FINALIZER = 4
2527

2628
# Backwards support for unpickling older pickles, even from 2.7
2729
from _stackless import _wrap

Objects/genobject.c

Lines changed: 9 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -1465,6 +1465,15 @@ async_gen_init_hooks(PyAsyncGenObject *o)
14651465
}
14661466

14671467

1468+
#ifdef STACKLESS
1469+
int
1470+
slp_async_gen_init_hooks(PyAsyncGenObject *o)
1471+
{
1472+
return async_gen_init_hooks(o);
1473+
}
1474+
#endif
1475+
1476+
14681477
static PyObject *
14691478
async_gen_anext(PyAsyncGenObject *o)
14701479
{

Stackless/changelog.txt

Lines changed: 1 addition & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -16,6 +16,7 @@ What's New in Stackless 3.X.X?
1616
- https://github.com/stackless-dev/stackless/issues/139
1717
New functions stackless.pickle_flags() and stackless.pickle_flags_default().
1818
They can be used to control pickling of certain objects.
19+
Stackless can now pickle asynchronous generators.
1920

2021
- https://github.com/stackless-dev/stackless/issues/175
2122
Cleanup of the Stackless C-API: Including stackless_api.h in an extension

Stackless/core/stackless_impl.h

Lines changed: 1 addition & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -802,6 +802,7 @@ int slp_safe_pickling(int(*save)(PyObject *, PyObject *, int),
802802
int pers_save);
803803

804804
PyObject * PyStackless_Pickle_ModuleDict(PyObject *pickler, PyObject *self);
805+
int slp_async_gen_init_hooks(PyAsyncGenObject *o);
805806

806807
/* debugging/monitoring */
807808

Stackless/module/clinic/stacklessmodule.c.h

Lines changed: 4 additions & 3 deletions
Original file line numberDiff line numberDiff line change
@@ -65,8 +65,9 @@ PyDoc_STRVAR(_stackless_pickle_flags__doc__,
6565
"pickle-flags without changing them, omit the arguments.\n"
6666
"\n"
6767
"Currently the following bits are defined:\n"
68-
" - bit 0, value 1: pickle the tracing/profiling state of a tasklet.\n"
69-
"\n"
68+
" - bit 0, value 1: pickle the tracing/profiling state of a tasklet;\n"
69+
" - bit 1, value 2: preserve the finalizer of an asynchronous generator;\n"
70+
" - bit 2, value 4: reset the finalizer of an asynchronous generator.\n"
7071
"All other bits must be set to 0.");
7172

7273
#define _STACKLESS_PICKLE_FLAGS_METHODDEF \
@@ -103,4 +104,4 @@ _stackless_pickle_flags(PyObject *module, PyObject *const *args, Py_ssize_t narg
103104
#ifndef _STACKLESS_PICKLE_FLAGS_METHODDEF
104105
#define _STACKLESS_PICKLE_FLAGS_METHODDEF
105106
#endif /* !defined(_STACKLESS_PICKLE_FLAGS_METHODDEF) */
106-
/*[clinic end generated code: output=6426d5e99d494bc0 input=a9049054013a1b77]*/
107+
/*[clinic end generated code: output=4ad6a9f914110629 input=a9049054013a1b77]*/

Stackless/module/stacklessmodule.c

Lines changed: 4 additions & 3 deletions
Original file line numberDiff line numberDiff line change
@@ -198,14 +198,15 @@ The function returns the previous pickle-flags. To inquire the
198198
pickle-flags without changing them, omit the arguments.
199199
200200
Currently the following bits are defined:
201-
- bit 0, value 1: pickle the tracing/profiling state of a tasklet.
202-
201+
- bit 0, value 1: pickle the tracing/profiling state of a tasklet;
202+
- bit 1, value 2: preserve the finalizer of an asynchronous generator;
203+
- bit 2, value 4: reset the finalizer of an asynchronous generator.
203204
All other bits must be set to 0.
204205
[clinic start generated code]*/
205206

206207
static PyObject *
207208
_stackless_pickle_flags_impl(PyObject *module, long new_flags, long mask)
208-
/*[clinic end generated code: output=fd773a5b06ab0c48 input=4a6920285b2c9019]*/
209+
/*[clinic end generated code: output=fd773a5b06ab0c48 input=36479a8e5486c711]*/
209210
{
210211
PyThreadState *ts = PyThreadState_GET();
211212
long old_flags;

Stackless/pickling/prickelpit.c

Lines changed: 152 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -1521,6 +1521,7 @@ static int init_dictitemsviewtype(PyObject * mod)
15211521

15221522
static PyTypeObject wrap_PyGen_Type;
15231523
static PyTypeObject wrap_PyCoro_Type;
1524+
static PyTypeObject wrap_PyAsyncGen_Type;
15241525

15251526
/* Used to initialize a generator created by gen_new. */
15261527
static PyFrameObject *gen_exhausted_frame = NULL;
@@ -1958,6 +1959,157 @@ static int init_coroutinetype(PyObject * mod)
19581959
#undef initchain
19591960
#define initchain init_coroutinetype
19601961

1962+
static PyObject *
1963+
async_gen_reduce(PyAsyncGenObject *async_gen)
1964+
{
1965+
PyThreadState *ts = PyThreadState_GET();
1966+
PyObject *tup;
1967+
gen_obj_head_ty goh;
1968+
PyObject *finalizer;
1969+
int hooks_inited;
1970+
1971+
if (reduce_to_gen_obj_head(&goh, async_gen->ag_frame, &async_gen->ag_exc_state))
1972+
return NULL;
1973+
1974+
if (ts->st.pickleflags & SLP_PICKLEFLAGS_PRESERVE_AG_FINALIZER) {
1975+
hooks_inited = async_gen->ag_hooks_inited;
1976+
finalizer = async_gen->ag_finalizer;
1977+
assert(finalizer != Py_None);
1978+
if (finalizer == NULL) {
1979+
/* encode NULL as Py_None */
1980+
finalizer = Py_None; /* borrowed ref */
1981+
}
1982+
}
1983+
else if (ts->st.pickleflags & SLP_PICKLEFLAGS_RESET_AG_FINALIZER) {
1984+
hooks_inited = 0;
1985+
finalizer = Py_None;
1986+
}
1987+
else {
1988+
hooks_inited = async_gen->ag_hooks_inited;
1989+
finalizer = async_gen->ag_finalizer != NULL ? Py_True : Py_None;
1990+
}
1991+
Py_INCREF(finalizer);
1992+
1993+
tup = Py_BuildValue("(O()(ObOOOOOOOii))",
1994+
&wrap_PyAsyncGen_Type,
1995+
goh.frame,
1996+
async_gen->ag_running,
1997+
async_gen->ag_name,
1998+
async_gen->ag_qualname,
1999+
goh.exc_type,
2000+
goh.exc_value,
2001+
goh.exc_traceback,
2002+
goh.exc_info_obj,
2003+
finalizer,
2004+
hooks_inited,
2005+
async_gen->ag_closed
2006+
);
2007+
2008+
Py_DECREF(finalizer);
2009+
gen_obj_head_clear(&goh);
2010+
return tup;
2011+
}
2012+
2013+
static PyObject *
2014+
async_gen_new(PyTypeObject *type, PyObject *args, PyObject *kwds)
2015+
{
2016+
PyAsyncGenObject *async_gen;
2017+
if (is_wrong_type(type)) return NULL;
2018+
2019+
/* A reference to frame is stolen by PyGen_New. */
2020+
assert(gen_exhausted_frame != NULL);
2021+
assert(PyFrame_Check(gen_exhausted_frame));
2022+
assert(type == &wrap_PyAsyncGen_Type);
2023+
async_gen = (PyAsyncGenObject *)PyAsyncGen_New(slp_ensure_new_frame(gen_exhausted_frame), NULL, NULL);
2024+
if (async_gen == NULL)
2025+
return NULL;
2026+
Py_TYPE(async_gen) = type;
2027+
return (PyObject *)async_gen;
2028+
}
2029+
2030+
static PyObject *
2031+
async_gen_setstate(PyObject *self, PyObject *args)
2032+
{
2033+
PyThreadState *ts = PyThreadState_GET();
2034+
PyAsyncGenObject *async_gen = (PyAsyncGenObject *)self;
2035+
gen_obj_head_ty goh;
2036+
PyObject *finalizer;
2037+
int closed;
2038+
int hooks_inited;
2039+
2040+
if (is_wrong_type(Py_TYPE(self))) return NULL;
2041+
2042+
if ((args = unwrap_frame_arg(args)) == NULL) /* now args is a counted ref! */
2043+
return NULL;
2044+
2045+
if (!PyArg_ParseTuple(args, "ObOOOOOOOii:async_gen_setstate",
2046+
&goh.frame,
2047+
&goh.running,
2048+
&goh.name,
2049+
&goh.qualname,
2050+
&goh.exc_type,
2051+
&goh.exc_value,
2052+
&goh.exc_traceback,
2053+
&goh.exc_info_obj,
2054+
&finalizer,
2055+
&hooks_inited,
2056+
&closed)) {
2057+
Py_DECREF(args);
2058+
return NULL;
2059+
}
2060+
2061+
if (setstate_from_gen_obj_head(&goh,
2062+
&async_gen->ag_frame, &async_gen->ag_exc_state, &async_gen->ag_code,
2063+
&async_gen->ag_name, &async_gen->ag_qualname, &async_gen->ag_running)) {
2064+
Py_DECREF(args);
2065+
return NULL;
2066+
}
2067+
2068+
async_gen->ag_closed = closed;
2069+
Py_TYPE(async_gen) = Py_TYPE(async_gen)->tp_base;
2070+
2071+
if (ts->st.pickleflags & SLP_PICKLEFLAGS_PRESERVE_AG_FINALIZER &&
2072+
finalizer != Py_True) {
2073+
if (finalizer == Py_None) {
2074+
/* NULL is pickled as Py_None */
2075+
async_gen->ag_hooks_inited = hooks_inited;
2076+
Py_CLEAR(async_gen->ag_finalizer);
2077+
}
2078+
else {
2079+
async_gen->ag_hooks_inited = hooks_inited;
2080+
Py_INCREF(finalizer);
2081+
Py_XSETREF(async_gen->ag_finalizer, finalizer);
2082+
}
2083+
}
2084+
else if (ts->st.pickleflags & SLP_PICKLEFLAGS_RESET_AG_FINALIZER ||
2085+
!hooks_inited) {
2086+
async_gen->ag_hooks_inited = 0;
2087+
Py_CLEAR(async_gen->ag_finalizer);
2088+
} else {
2089+
assert(hooks_inited);
2090+
async_gen->ag_hooks_inited = 0;
2091+
if (slp_async_gen_init_hooks(async_gen)) {
2092+
async_gen = NULL;
2093+
goto error;
2094+
}
2095+
}
2096+
2097+
Py_INCREF(async_gen);
2098+
error:
2099+
Py_DECREF(args); /* holds the frame and name refs */
2100+
return (PyObject *)async_gen;
2101+
}
2102+
2103+
MAKE_WRAPPERTYPE(PyAsyncGen_Type, async_gen, "async_generator", async_gen_reduce,
2104+
async_gen_new, async_gen_setstate)
2105+
2106+
static int
2107+
init_async_gentype(PyObject * mod)
2108+
{
2109+
return init_type(&wrap_PyAsyncGen_Type, initchain, mod);
2110+
}
2111+
#undef initchain
2112+
#define initchain init_async_gentype
19612113

19622114
/******************************************************
19632115

Stackless/pickling/prickelpit.h

Lines changed: 3 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -46,7 +46,9 @@ Py_ssize_t slp_from_tuple_with_nulls(PyObject **start, PyObject *tup);
4646

4747
/* flags */
4848
#define SLP_PICKLEFLAGS_PRESERVE_TRACING_STATE (1U)
49-
#define SLP_PICKLEFLAGS__MAX_VALUE ((1<<1)-1) /* must be a signed value */
49+
#define SLP_PICKLEFLAGS_PRESERVE_AG_FINALIZER (1U<<1)
50+
#define SLP_PICKLEFLAGS_RESET_AG_FINALIZER (1U<<2)
51+
#define SLP_PICKLEFLAGS__MAX_VALUE ((1<<3)-1) /* must be a signed value */
5052

5153
/* helper functions for module dicts */
5254

0 commit comments

Comments
 (0)
0