8000 MAINT: fix thread-unsafe cache initialization in PyUFunc_TrueDivisionTypeResolver by ngoldbaum · Pull Request #26497 · numpy/numpy · GitHub
[go: up one dir, main page]

Skip to content

MAINT: fix thread-unsafe cache initialization in PyUFunc_TrueDivisionTypeResolver #26497

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Closed
wants to merge 1 commit into from

Conversation

ngoldbaum
Copy link
Member

This moves initialization for a static tuple used in the division type resolver from inside the resolver body to the initialization for the umath module.

This is only really needed for the free-threaded build of Python. Initializing globals like this during module initialization is thread safe.

No test because I can't think of a way to cause breakage from the data race. Still fixing it since I'm globally auditing usages of static globals in the codebase.

@ngoldbaum ngoldbaum added 03 - Maintenance 39 - free-threading PRs and issues related to support for free-threading CPython (a.k.a. no-GIL, PEP 703) labels May 21, 2024
@charris
Copy link
Member
charris commented May 22, 2024

LGTM, but will wait on @seberg.

@seberg
Copy link
Member
seberg commented May 22, 2024

This LGTM, but I wanted to discuss it briefly tonight. We have a lot of similar initializations, so I think it would be much nicer to see if we cannot find a local pattern instead.

@ngoldbaum
Copy link
Member Author

FWIW, at least for npy_cache_import, I'm pretty sure it's safe to do nothing. It's implemented using the CPython C API so any thread safety issues in the result are CPython bugs and the result of initializing the static reference multiple times simultaneously should be the same as initializing it once. That's the majority of the C static variable uses I'm seeing.

@seberg
Copy link
Member
seberg commented May 22, 2024

Sorry, but I am not following. What is actually different about this one? Arguably, all it seems to do is one INCREF of an immortal object (fine) and then call PyTuple_Pack (i.e. CPython C API).

@ngoldbaum
Copy link
Member Author

The difference here is this one causes a memory leak, whereas npy_cache_import will give you a reference to the already-imported module if it's been imported. It's not a big problem since the memory leak can only be triggered if threads call this function simultaneously. Still, seemed straightforward to fix.

@seberg
Copy link
Member
seberg commented May 23, 2024

Aha, right so one leaks only reference to things that are practically immortal while this one is a bit worse as it leaks a fresh tuple.

We had discussed yesterday a bit and landed a bit on trying to consolidate this "static data" into a global struct (which makes it a step towards a module level struct for multi-phase init/sub-interpreters).

Sounds like that might be worthwhile, but will also be a somewhat larger refactor. FWIW, I am OK with this if it seems easier or necessary right now, I still think it would be nice to use this delayed-init pattern slightly more rather than less so having a solution for it would be nice. (And I think the global struct will do that).

@ngoldbaum
Copy link
Member Author

Closing in favor of #26607

@ngoldbaum ngoldbaum closed this Jun 3, 2024
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
03 - Maintenance 39 - free-threading PRs and issues related to support for free-threading CPython (a.k.a. no-GIL, PEP 703)
Projects
None yet
Development

Successfully merging this pull request may close these issues.

3 participants
0