Description
Describe the bug
When using async/await, arbitrary objects can leak since 3.1.0.
Their refcount is 1 more than it should be.
It seems to leak return values of async functions that have been awaited inside Cython modules.
It seems that the async function must have at least one await.
Code to reproduce the behaviour:
hello.py:
import asyncio
async def helper(func):
await asyncio.sleep(1)
return func()
async def leak_return_value_of(func):
await helper(func)main.py:
import asyncio
import gc
import hello
class Leaked:
pass
async def main():
while True:
x = await hello.leak_return_value_of(Leaked)
del x
gc.collect()
print('num leaked:', sum(1 for x in gc.get_objects() if type(x) is Leaked))
try:
asyncio.run(main())
except KeyboardInterrupt:
passExpected behaviour
$ python ./main.py
num leaked: 0
num leaked: 0
num leaked: 0
...$ cythonize --inplace ./hello.py
$ python ./main.py
num leaked: 1
num leaked: 2
num leaked: 3
...Objects of the class Leaked are being leaked, which can be made arbitrarily big if we add more stuff to that class.
(I noticed the bug because the OS killed my program when it used 100 GB in 1 minute 😅, it leaked huge bytes objects in my case)
OS
Linux & macOS, probably Windows too but haven't checked
Python version
3.12.10
Cython version
≥ 3.1.0, including alpha versions. Everything since the commit 007982e (9 months ago)
Additional context
I debugged the INCREFs and DECREFs and I know there's exactly one extra ref that was supposed to be decreased.
I'm pretty sure the problem is that __Pyx_Coroutine_FinishDelegation is somehow not called on the value being awaited, which is the missing DECREF.
Since the problem seems to be in coroutines it can probably be triggered even with just yield without any async.