8000 Crash after error in PickleBuffer · Issue #122306 · python/cpython · GitHub
[go: up one dir, main page]
Skip to content

Crash after error in PickleBuffer #122306

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom
Closed
serhiy-storchaka opened this issue Jul 26, 2024 · 5 comments
Closed

Crash after error in PickleBuffer #122306

serhiy-storchaka opened this issue Jul 26, 2024 · 5 comments
Labels
3.12 only security fixes 3.13 bugs and security fixes 3.14 bugs and security fixes type-crash A hard crash of the interpreter, possibly with a core dump

Comments

@serhiy-storchaka
Copy link
Member
serhiy-storchaka commented Jul 26, 2024 •
edited by bedevere-app bot
Loading

Crash report

The following script crashes at finalization stage:

import unittest
import pickle

class Test(unittest.TestCase):
    def test_non_continuous_buffer(self):
        pb = pickle.PickleBuffer(memoryview(b"foobar")[::2])
        pb.raw()

unittest.main()

Output:

E
======================================================================
ERROR: test_non_continuous_buffer (__main__.Test.test_non_continuous_buffer)
----------------------------------------------------------------------
Traceback (most recent call last):
  File "/home/serhiy/py/cpython/test_non_continuous_buffer.py", line 7, in test_non_continuous_buffer
    pb.raw()
    ~~~~~~^^
BufferError: cannot extract raw buffer from non-contiguous buffer

----------------------------------------------------------------------
Ran 1 test in 0.001s

FAILED (errors=1)
python: Objects/memoryobject.c:125: mbuf_dealloc: Assertion `self->exports == 0' failed.
Aborted (core dumped)

I cannot yet reproduce the crash without unittest, perhaps it is related to some reference loop.

The crash is only happens in 3.12 an newer. In 3.8-3.11 it just raises an exception (PickleBuffer was added in 3.8).

cc @pitrou

Linked PRs
@serhiy-storchaka serhiy-storchaka added type-crash A hard crash of the interpreter, possibly with a core dump 3.12 only security fixes 3.13 bugs and security fixes 3.14 bugs and security fixes labels Jul 26, 2024
@devdanzin
Copy link
Contributor

Try:

import pickle
import atexit

m = memoryview(b"foobar")[::2]
pb = pickle.PickleBuffer(m)
def p(): pb.raw()
atexit.register(m.bit_count)

@serhiy-storchaka serhiy-storchaka mentioned this issue Jul 26, 2024
Closed
@devdanzin
Copy link
Contributor

Better:

import pickle

m = memoryview(b"foobar")[::2]
pb = pickle.PickleBuffer(m)
def p(): m.bit_length()
m.invalid

@marko1616
Copy link
marko1616 commented Aug 27, 2024 •
edited
Loading

The following analysis report will explain the bug by inserting comments into the log.

Main Causes

  1. Creating a circular reference through a constructed exception, preventing garbage collection of PickleBuffer and memory objects
  2. Calling gc.collect()
  3. During the collection of the memory object, PyMemoryViewObject->mbuf is cleared for the first time (mbuf is set to a null pointer)
  4. When collecting PickleBuffer, memory_dealloc is called again through picklebuf_clear, at which point an exception occurs because mbuf was set to a null pointer in step 3

Next

I'm trying to find a method fix this this days.

Bug reproduction(tested on mainline)

import gc, pickle

mv = memoryview(b"test")
error = None

def func():
    try:
        pb = pickle.PickleBuffer(mv)
        raise Exception("test")
    except Exception as e:
        # Ensure `error` references the exception
        error = e
    finally:
        print(f"pb:{pb}\nmv:{mv}")

# Cyclic References Explanation:
# An exception creates a traceback object referencing the frame.
# The frame references local variables (`error`, `pb`).
# The exception object references the traceback.
# The `error` variable references the exception.

func()
del mv
gc.collect() # Crash here

GDB Debugging Details

╭─xxx at xxx in ~/path/to/repos/cpython on main✔
╰─± gdb ./python 
GNU gdb (Debian 13.1-3) 13.1
Copyright (C) 2023 Free Software Foundation, Inc.
License GPLv3+: GNU GPL version 3 or later <http://gnu.org/licenses/gpl.html>
This is free software: you are free to change and redistribute it.
There is NO WARRANTY, to the extent permitted by law.
Type "show copying" and "show warranty" for details.
This GDB was configured as "x86_64-linux-gnu".
Type "show configuration" for configuration details.
For bug reporting instructions, please see:
<https://www.gnu.org/software/gdb/bugs/>.
Find the GDB manual and other documentation resources online at:
    <http://www.gnu.org/software/gdb/documentation/>.

For help, type "help".
Type "apropos word" to search for commands related to "word"...
Reading symbols from ./python...
warning: File "/home/marko1616/Documents/repos/cpython/python-gdb.py" auto-loading has been declined by your `auto-load safe-path' set to "$debugdir:$datadir/auto-load".
To enable execution of this file add
        add-auto-load-safe-path /home/marko1616/Documents/repos/cpython/python-gdb.py
line to your configuration file "/home/marko1616/.config/gdb/gdbinit".
To completely disable this security protection add
        set auto-load 
8000
safe-path /
line to your configuration file "/home/marko1616/.config/gdb/gdbinit".
For more information about this security protection see the
"Auto-loading safe path" section in the GDB manual.  E.g., run from the shell:
        info "(gdb)Auto-loading safe path"
(gdb) b _memory_release
Breakpoint 1 at 0x14a97e: _memory_release. (3 locations)
(gdb) disable 1
(gdb) run
Starting program: /home/marko1616/Documents/repos/cpython/python 
[Thread debugging using libthread_db enabled]
Using host libthread_db library "/lib/x86_64-linux-gnu/libthread_db.so.1".
Python 3.14.0a0 (heads/main:54a05a46002, Aug 27 2024, 11:19:04) [GCC 12.2.0] on linux
Type "help", "copyright", "credits" or "license" for more information.
[Detaching after vfork from child process 302825]
>>> import gc, pickle
... 
... mv = memoryview(b"test")
... error = None
... 
... def func():
...     try:
...         pb = pickle.PickleBuffer(mv)
...         raise Exception("test")
...     except Exception as e:
...         # Ensure `error` references the exception
...         error = e
...     finally:
...         print(f"pb:{pb}\nmv:{mv}")
... 
... # Cyclic References Explanation:
... # An exception creates a traceback object referencing the frame.
... # The frame references local variables (`error`, `pb`).
... # The exception object references the traceback.
... # The `error` variable references the exception.
... 
... func()
... input(f"del mv {mv}")
... del mv
... input("before gc")
... gc.collect() # Crash here
... 
pb:<pickle.PickleBuffer object at 0x7ffff744f740>
mv:<memory at 0x7ffff76cbf40>
del mv <memory at 0x7ffff76cbf40>
before gc
         Program received signal SIGINT, Interrupt.
                                                   0x00007ffff7de419d in __GI___libc_read (fd=fd@entry=3, buf=buf@entry=0x555555de78e0, nbytes=nbytes@entry=10000) at ../sysdeps/unix/sysv/linux/read.c:26
26      ../sysdeps/unix/sysv/linux/read.c: No such file or directory.
(gdb) watch (*(PyMemoryViewObject *)0x7ffff76cbf40)->mbuf
Hardware watchpoint 2: (*(PyMemoryViewObject *)0x7ffff76cbf40)->mbuf
(gdb) c
Continuing.


Hardware watchpoint 2: (*(PyMemoryViewObject *)0x7ffff76cbf40)->mbuf

Old value = (_PyManagedBufferObject *) 0x7ffff744e3c0
New value = (_PyManagedBufferObject *) 0x0
memory_clear (_self=0x7ffff76cbf40) at Objects/memoryobject.c:1168
1168        Py_CLEAR(self->mbuf);
(gdb) bt
#0  memory_clear (_self=0x7ffff76cbf40) at Objects/memoryobject.c:1168
#1  0x00005555557c5ec4 in delete_garbage (old=0x555555ae2930 <_PyRuntime+183280>, collectable=0x7fffffffd740, gcstate=0x555555ae28e8 <_PyRuntime+183208>, 
    tstate=0x555555b17858 <_PyRuntime+400152>) at Python/gc.c:1119
...
--Type <RET> for more, q to quit, c to continue without paging--q
Quit
(gdb) c
Continuing.
Exception ignored in tp_clear of memoryview:
Traceback (most recent call last):
  File "<python-input-0>", line 26, in <module>
BufferError: memoryview has 1 exported buffer

Program received signal SIGSEGV, Segmentation fault.
0x000055555569e9ad in _memory_release (self=0x7ffff76cbf40) at Objects/memoryobject.c:1111
1111            if (--self->mbuf->exports == 0)
(gdb) bt
#0  0x000055555569e9ad in _memory_release (self=0x7ffff76cbf40) at Objects/memoryobject.c:1111
#1  memory_dealloc (_self=0x7ffff76cbf40) at Objects/memoryobject.c:1148
#2  0x00005555556c999d in picklebuf_clear (self=<optimized out>) at Objects/picklebufobject.c:103
#3  0x00005555557c5ec4 in delete_garbage (old=0x555555ae2930 <_PyRuntime+183280>, collectable=0x7fffffffd740, gcstate=0x555555ae28e8 <_PyRuntime+183208>, 
...
(gdb)

@marko1616 marko1616 mentioned this issue Aug 27, 2024
Closed
@marko1616
Copy link

cc @serhiy-storchaka

@serhiy-storchaka
Copy link
Member Author

This looks like an example of the hypothetical issue discussed in #77894. The solution was proposed there, but we did not have any reliable way to test it. Now we can, using pickle.PickleBuffer. And other way -- using memoryview.__getbuffer__(). Both ways were not available when the original issue was opened.

I'm closing this issue as a duplicate of #77894.

@serhiy-storchaka serhiy-storchaka closed this as not planned Won't fix, can't repro, duplicate, stale Sep 10, 2024
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
3.12 only security fixes 3.13 bugs and security fixes 3.14 bugs and security fixes type-crash A hard crash of the interpreter, possibly with a core dump
Projects
Pickle and copy issues 🥒
Status: Done
Milestone
No milestone
Development

No branches or pull requests
3 participants
@serhiy-storchaka @marko1616 @devdanzin
0