8000 cross_val_score threshold adjustment · Issue #15019 · scikit-learn/scikit-learn · GitHub
[go: up one dir, main page]
Skip to content

cross_val_score threshold adjustment #15019

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom
Closed
tjdurant opened this issue Sep 19, 2019 · 4 comments
Closed

cross_val_score threshold adjustment #15019

tjdurant opened this issue Sep 19, 2019 · 4 comments
Labels
module:model_selection New Feature

Comments

@tjdurant
Copy link

Apologies if this has been brought up before but a search through current and closed issues didn't return anything to indicate it had.

I have a use case wherein I have picked an optimal decision threshold based on the intersection of sensitivity and PPV when plotted as a function of the decision threshold. The point of intersection corresponds with a decision threshold of 0.3. I would like to run cross_val_score with a LogisticRegression model that uses a threshold of 0.3 to calculate a distribution of sensitivity and FPR (separate issue, here). However, it doesn't seem as though the threshold can be varied in the context of cross_val_score or at the estimator level. Might be nice to have that functionality if possible.

Thanks to all the contributors who work on this project : )

-Tommy
@amueller
Copy link
Member
amueller commented Nov 5, 2019

That's not currently possible out-of-the-box.
You can write a meta-estimator to achieve that yourself, though (i.e. something that wraps the LogisticRegression and changes the threshold).
There is a more general solution in the works here: #10117

@cmarmo
Copy link
Contributor
cmarmo commented Aug 20, 2020

Cross referencing here #16525 , as it is meant to supersede #10117. @glemaitre do you think your PR will also close this issue? Thanks.

@glemaitre
Copy link
Member

This is a use-case mentioned in #16525 (still not implemented thought) by @lorentzenchr. It might make sense to add the option and it would close this issue as well.

@glemaitre
Copy link
Member

This can be done with the TunedThresholdClassifierCV with additional tuning. #28944 intends to show such example but at least now, we should be able to implement this pipeline.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
module:model_selectio 4129 n New Feature
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
5 participants
@amueller @cmarmo @thomasjpfan @glemaitre @tjdurant
0