8000 test__xxsubinterpreters: test_already_running() crash randomly on Python built with TraceRefs: invalid object chain · Issue #107080 · python/cpython · GitHub
[go: up one dir, main page]
Skip to content

test__xxsubinterpreters: test_already_running() crash randomly on Python built with TraceRefs: invalid object chain #107080

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom
Closed
vstinner opened this issue Jul 22, 2023 · 6 comments · Fixed by #107751
Closed

test__xxsubinterpreters: test_already_running() crash randomly on Python built with TraceRefs: invalid object chain #107080

vstinner opened this issue Jul 22, 2023 · 6 comments · Fixed by #107751
Assignees
@ericsnowcurrently
Labels
3.12 only security fixes 3.13 bugs and security fixes interpreter-core (Objects, Python, Grammar, and Parser dirs) topic-subinterpreters type-bug An unexpected behavior, bug, or error

Comments

@vstinner
Copy link
Member
vstinner commented Jul 22, 2023 •
edited by bedevere-bot
Loading

AMD64 Arch Linux TraceRefs 3.x: https://buildbot.python.org/all/#/builders/484/builds/3721

Assertion error: Objects/object.c:2235: _Py_ForgetReference: Assertion failed: invalid object chain.

$ ./python -m test -j2 test__xxsubinterpreters test__xxsubinterpreters -m test_already_running -v
(...)
== Python build: debug TraceRefs
(...)
== CPU count: 12
(...)
0:00:00 load avg: 1.19 Run tests in parallel using 2 child processes
0:00:00 load avg: 1.19 [1/2] test__xxsubinterpreters passed
test_already_running (test.test__xxsubinterpreters.RunStringTests.test_already_running) ... ok
(...)

0:00:00 load avg: 1.19 [2/2/1] test__xxsubinterpreters crashed (Exit code -6)
test_already_running (test.test__xxsubinterpreters.RunStringTests.test_already_running) ...

Objects/object.c:2235: _Py_ForgetReference: Assertion failed: invalid object chain
Enable tracemalloc to get the memory block allocation traceback

object address  : 0x7f01d0be95a0
object refcount : 0
object type     : 0xa1dc00
object type name: str
object repr     : <refcnt 0 at 0x7f01d0be95a0>

Fatal Python error: _PyObject_AssertFailed: _PyObject_AssertFailed
Python runtime state: initialized

Current thread 0x00007f01df4a1740 (most recent call first):
  <no Python frame>
(...)

Python built with:

git clean -fdx
./configure --with-pydebug --with-trace-refs CFLAGS="-O0"
make -j14

Linked PRs
@vstinner vstinner added the type-bug An unexpected behavior, bug, or error label Jul 22, 2023
@vstinner vstinner changed the title test__xxsubinterpreters: test_already_running() crash randomly on Python built with TraceRefs test__xxsubinterpreters: test_already_running() crash randomly on Python built with TraceRefs: invalid object chain Jul 22, 2023
@vstinner
8000 Copy link
Member Author

The object chain is invalid:

(gdb) frame
#13 0x000000000055e731 in _Py_ForgetReference (op=(None, None, None)) at Objects/object.c:2235
2235	        _PyObject_ASSERT_FAILED_MSG(op, "invalid object chain");

(gdb) l
2230	    }
2231	
2232	    if (op == &refchain ||
2233	        op->_ob_prev->_ob_next != op || op->_ob_next->_ob_prev != op)
2234	    {
2235	        _PyObject_ASSERT_FAILED_MSG(op, "invalid object chain");
2236	    }
2237	
2238	#ifdef SLOW_UNREF_CHECK
2239	    PyObject *p;

(gdb) p op->_ob_prev->_ob_next != op
$14 = 1

(gdb) p op->_ob_next->_ob_prev != op
$15 = 1

@vstinner
Copy link
Member Author

Similar error in test_interpreters.test_create_many_threaded():

$ ./python -m test test_interpreters -m test_create_many_threaded -v
(...)
test_create_many_threaded (test.test_interpreters.StressTests.test_create_many_threaded) ...

Objects/object.c:2235: _Py_ForgetReference: Assertion failed: invalid object chain
(...)

@vstinner
Copy link
Member Author

cc @ericsnowcurrently

@ericsnowcurrently ericsnowcurrently self-assigned this Aug 2, 2023
@ericsnowcurrently
Copy link
Member

--with-trace-refs builds use a C global variable to store the linked list of objects. The problem is that the variable is used by all interpreters. The solution is to make that variable per-interpreter.

@ericsnowcurrently ericsnowcurrently added 3.12 only security fixes 3.13 bugs and security fixes interpreter-core (Objects, Python, Grammar, and Parser dirs) topic-subinterpreters labels Aug 2, 2023
@bedevere-bot bedevere-bot mentioned this issue Aug 2, 2023
Merged
This was referenced Aug 3, 2023
Merged
Closed
ericsnowcurrently added a commit that referenced this issue Aug 3, 2023
@ericsnowcurrently
gh-107080: Fix Py_TRACE_REFS Crashes Under Isolated Subinterpreters (g…
58ef741 
…h-107567)

The linked list of objects was a global variable, which broke isolation between interpreters, causing crashes. To solve this, we've moved the linked list to each interpreter.
@bedevere-bot bedevere-bot mentioned this issue Aug 3, 2023
Merged
miss-islington pushed a commit to miss-islington/cpython that referenced this issue Aug 3, 2023
@ericsnowcurrently @miss-islington
pythongh-107080: Fix Py_TRACE_REFS Crashes Under Isolated Subinterpre…
b55de7a 
…ters (pythongh-107567)

The linked list of objects was a global variable, which broke isolation between interpreters, causing crashes. To solve this, we've moved the linked list to each interpreter.
(cherry picked from commit 58ef741)

Co-authored-by: Eric Snow <ericsnowcurrently@gmail.com>
Yhg1s pushed a commit that referenced this issue Aug 3, 2023
@miss-islington @ericsnowcurrently
[3.12] gh-107080: Fix Py_TRACE_REFS Crashes Under Isolated Subinterpr…
58af229 
…eters (gh-107567) (#107599)

gh-107080: Fix Py_TRACE_REFS Crashes Under Isolated Subinterpreters (gh-107567)

The linked list of objects was a global variable, which broke isolation between interpreters, causing crashes. To solve this, we've moved the linked list to each interpreter.
(cherry picked from commit 58ef741)

Co-authored-by: Eric Snow <ericsnowcurrently@gmail.com>
@github-project-automation github-project-automation bot moved this from Todo to Done in Subinterpreters Aug 3, 2023
ericsnowcurrently added a commit to ericsnowcurrently/cpython that referenced this issue Aug 4, 2023
@ericsnowcurrently
Revert "[3.12] pythongh-107080: Fix Py_TRACE_REFS Crashes Under Isola…
b380392 
…ted Subinterpreters (pythongh-107567) (python#107599)"

This reverts commit 58af229.
@bedevere-bot bedevere-bot mentioned this issue Aug 4, 2023
Merged
Yhg1s pushed a commit that referenced this issue Aug 5, 2023
@ericsnowcurrently
[3.12] gh-107630: Revert "[3.12] gh-107080: Fix Py_TRACE_REFS Crashes…
6e4eec7 
… Under Isolated Subinterpreters (gh-107567) (#107599)" (#107648)

Revert "[3.12] gh-107080: Fix Py_TRACE_REFS Crashes Under Isolated Subinterpreters (gh-107567) (#107599)"

This reverts commit 58af229.
ericsnowcurrently added a commit to ericsnowcurrently/cpython that referenced this issue Aug 7, 2023
@ericsnowcurrently
Unrevert "[3.12] pythongh-107080: Fix Py_TRACE_REFS Crashes Under Iso…
6f8b84f 
…lated Subinterpreters (pythongh-107567) (python#107599)".

This reverts commit 6e4eec7 (pythongh-107648).
This was referenced Aug 7, 2023
Closed
Merged
@serhiy-storchaka serhiy-storchaka linked a pull request Aug 16, 2023 that will close this issue
[3.12] gh-107080: Fix Py_TRACE_REFS Crashes Under Isolated Subinterpreters #107751
Merged
Yhg1s pushed a commit that referenced this issue Aug 16, 2023
@ericsnowcurrently
[3.12] gh-107080: Fix Py_TRACE_REFS Crashes Under Isolated Subinterpr…
aa9707d 
…eters (#107751)

* Unrevert "[3.12] gh-107080: Fix Py_TRACE_REFS Crashes Under Isolated Subinterpreters (gh-107567) (#107599)".

This reverts commit 6e4eec7 (gh-107648).

* Initialize each interpreter's refchain properly.

* Skip test_basic_multiple_interpreters_deleted_no_reset on tracerefs builds.
@vstinner
Copy link
Member Author

serhiy-storchaka reopened this Aug 16, 2023

@serhiy-storchaka: Hum, is this issue fixed or not?

@serhiy-storchaka
Copy link
Member

I reopened it only because the backport to 3.12 was not merged yet. I linked the backport PR to this issue and thought that merging it will automatically close the issue, but it did not happen.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees

@ericsnowcurrently ericsnowcurrently

Labels
3.12 only security fixes 3.13 bugs and security fixes interpreter-core (Objects, Python, Grammar, and Parser dirs) topic-subinterpreters type-bug An unexpected behavior, bug, or error
Projects
Subinterpreters
Status: Done
Milestone
No milestone
Development

Successfully merging a pull request may close this issue.

[3.12] gh-107080: Fix Py_TRACE_REFS Crashes Under Isolated Subinterpreters ericsnowcurrently/cpython
3 participants
@vstinner @ericsnowcurrently @serhiy-storchaka
0