gh-91054: make code watcher tests resilient to other watchers #107821
Conversation
|
Skipping news since this is purely a fix to CPython's own test suite. |
|
@Yhg1s how would you feel about backporting this test-suite-only fix to 3.12, to reduce spurious failures in 3.12 if anyone runs the 3.12 tests with a code watcher active, and reduce conflicts on future backports in these tests? |
f32672c to
b40b696
Compare
|
in what scenario is this test run with other code watchers registered? |
In practice today, probably it isn't. I found this bug when backporting code watchers to Cinder 3.10 so that the Cinder JIT can use them; in that case the Cinder JIT installs a code watcher, and the test suite should pass with the JIT active. In future, in principle, any Python optimizer or debug/monitoring tool that installs a code watcher may want to validate that the Python test suite passes with it active, and I think the tests should work in such cases. (The Cinder JIT for Python 3.12 will certainly be one such case.) There is the possible edge case in which someone tries to run these tests with more than six code watchers active, in which case they'd fail because there aren't enough code watcher slots left. I doubt that case will ever be encountered unless artificially constructed, but a possible safeguard could be to skip the tests if too many code watchers are already installed. |
|
This makes sense to me to backport to 3.12. I think it's worth realising that the Python test suite does get run in situations different from the default one. (At work we run the whole testsuite with an embedded interpreter and custom importers, for example, and I could totally see us adding something that uses watchers for something or other.) |
|
Thanks @carljm for the PR 🌮🎉.. I'm working now to backport this PR to: 3.12. |
…ythonGH-107821) (cherry picked from commit 2ec16fe) Co-authored-by: Carl Meyer <carl@oddbird.net>
|
GH-107835 is a backport of this pull request to the 3.12 branch. |
Summary: This reverts the revert commit 94d0dbf6891cff0324fdb2d75e9d0fcc9529675f (D47448305), re-landing D47201503, but with a key change: in the first version I'd missed the line that actually initializes the code watcher on JIT startup, so the code watcher never took effect. This meant that in a JIT-list scenario, we didn't ever remove code objects from the to-be-compiled registry when they were destroyed, so we ended up with dangling pointers on the registry. This situation should be better tested, and I think it would have been likely caught by the multithreaded-compile test, but that test has been broken and disabled for quite a while. I created T158334137 to track fixing and re-enabling it. This diff also adds a small improvement to the debug message on a JIT_CHECK. I also had to fix the code watchers test suite; it didn't work correctly when another code watcher was already enabled. I upstreamed this fix in python/cpython#107821 Reviewed By: alexmalyshev Differential Revision: D47459869 fbshipit-source-id: c53c9847a300892d1e0b01a6a75f524ed730fd48
The code watcher tests implicitly assume that the watchers they will register will be numbered
0and1. But this assumption may not hold true if the tests are run with another watcher already active.This PR fixes the tests to record the actual watcher ID for each "test watcher index" and use that to ensure we clear the correct recorded data when clearing a watcher.