8000 Vendor threadpoolctl · Issue #14979 · scikit-learn/scikit-learn · GitHub
[go: up one dir, main page]
Skip to content

Vendor threadpoolctl #14979

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom
Closed
jeremiedbb opened this issue Sep 13, 2019 · 2 comments
Closed

Vendor threadpoolctl #14979

jeremiedbb opened this issue Sep 13, 2019 · 2 comments
Labels
Build / CI

Comments

@jeremiedbb
Copy link
Member
jeremiedbb commented Sep 13, 2019 •
edited
Loading

As discussed in #11950 and #14265, we need a way to control at runtime the number of threads OpenMP or BLAS are allowed to use. This is to prevent oversubscription in nested parallelism of the form

joblib(backend = 'threading')
    BLAS or OpenMP

or

OpenMP
    BLAS

(Note the backend='loky' case should/will be handled by loky itself)

With @ogrisel and @tomMoral, we've been working on threadpoolctl to solve this issue. There might be a plan to include it in numpy so at some point we'll be able to import it from numpy. In the meantime I propose to vendor it in scikit-learn (I openend #14980).

The first place where it's needed is in #11950, but there might be other places as well.
@jeremiedbb jeremiedbb mentioned this issue Sep 13, 2019
Closed
@rth
Copy link
Member
rth commented Sep 13, 2019

Could you remind me what was the argument for vendoring it in scikit-learn as opposed to joblib? Given that it's very parallelism dependent.

@rth rth mentioned this issue Sep 24, 2019
Closed
@thomasjpfan thomasjpfan mentioned this issue Oct 2, 2019
Merged
@ogrisel
Copy link
Member
ogrisel commented Oct 15, 2019

Could you remind me what was the argument for vendoring it in scikit-learn as opposed to joblib?

At the moment we do not use threadpoolctl in joblib so it's not included in joblib itself.

@TomAugspurger TomAugspurger mentioned this issue Oct 15, 2019
Open
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
Build / CI
Projects
None yet
Milestone
No milestone
Development

Successfully merging a pull request may close this issue.

[MRG] Vendor threadpoolctl jeremiedbb/scikit-learn
4 participants
@ogrisel @rth @thomasjpfan @jeremiedbb
0