8000 Allow for multiple scoring metrics in `RFECV` · Issue #28937 · scikit-learn/scikit-learn · GitHub
[go: up one dir, main page]
Skip to content

Allow for multiple scoring metrics in RFECV #28937

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom
Open
ArturoSbr opened this issue May 2, 2024 · 4 comments · May be fixed by Agent-Benchmarking/scikit-learn#18 or Agent-Benchmarking/scikit-learn#23
Open

Allow for multiple scoring metrics in RFECV #28937

ArturoSbr opened this issue May 2, 2024 · 4 comments · May be fixed by Agent-Benchmarking/scikit-learn#18 or Agent-Benchmarking/scikit-learn#23
Labels
New Feature

Comments

@ArturoSbr
Copy link
Contributor
ArturoSbr commented May 2, 2024 •
edited
Loading

Workflow

In its current state, RFECV only allows for a single scoring metric. In my opinion, calculating multiple scores on each model using k <= K features would be extremely valuable.

For example, if I wanted to study how the precision and recall metrics of a binary classifier evolve as I feed less and less features to a model, I would have to run RFECV twice: one with scoring='precision' and another with scoring='recall'.
This is inefficient, as it implies running RFECV twice instead of once.

The cv_results_ attribute of GridSearchCV returns one rank per metric used to evaluate each combination of hyperparameters. Replicating this behavior in RFECV would be extremely helpful.

Proposed solution

Notation

  • K is the number of folds used for cross-validation.
  • P is the total number of features available.
  • p is the number of features tried at each step. That is, an integer such that min_features_to_select <= p <= P.
  • m is one of M performance metrics passed by the user (e.g., 'precision').

Solution

User can pass a list of strings representing M predefined scoring metrics and at each step, the algorithm stores the performance metric of the k models trained with p <= P features.

The cv_results_ attribute of the resulting RFECV would now include the following keys for each metric m and fold k:

  • 'split{k}_test_{m}'
  • mean_test_{m}
  • 'std_test_{m}'
  • 'rank_test_{m}'

Example

rfecv = RFECV(
    estimator=clf,  # Some classifier instance
    step=1,
    min_features_to_select=1,
    cv=10,
    scoring=['precision', 'recall', 'f1', 'roc_auc', 'accuracy']
)
Considerations

It is likely that rank_test_{m1} will differ from rank_test_{m2} for any pair of performance metrics m1 and m2. Hence, adding this feature will no longer allow RFECV to automatically pick the best number of features, as the rankings can differ from one metric to another. This part of the workflow would be up to the user.

Describe alternatives you've considered, if relevant

Running RFECV as many times as there are metrics.

Additional context

I asked this question on StackOverflow and the community agrees that the most viable way to do this is to run one RFECV per performance metric I need to evaluate.
@ArturoSbr ArturoSbr added Needs Triage Issue requires triage New Feature labels May 2, 2024
@bmreiniger
Copy link
Contributor

For your "Considerations" section, I wonder if it makes sense to also adopt the refit approach from the hyperparameter tuners: pick one metric to choose the best k, or even define a custom strategy callable.

(Maybe this should keep in mind #21291 too?)

@ogrisel ogrisel removed the Needs Triage Issue requires triage label May 3, 2024
@ogrisel
Copy link
Member
ogrisel commented May 3, 2024

This sounds like a legitimate need that cannot be efficiently be addressed with the current API as far as I understand.

So we could indeed consider expanding RFECV to accept multiple metrics + passing a custom callable as done for GridSearchCV, see:

https://scikit-learn.org/stable/auto_examples/model_selection/plot_grid_search_digits.html

@glemaitre
Copy link
Member

Did we improve a bit the situation by merging: #28360

@adrinjalali
Copy link
Member

With #28360 the situation is much improved, but we need to be able to understand multiple metrics the same way as *SeachCV here. For now we assume a single value as metric value.

This is related to #28944 for me too, since we're choosing sth based on certain metrics values while understanding multiple metrics.

@sokolat sokolat mentioned this issue May 6, 2024
Closed
@dokterbob dokterbob linked a pull request Mar 25, 2025 that will close this issue
Open
@Nusnus Nusnus linked a pull request Mar 28, 2025 that will close this issue
Open
5 tasks
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
New Feature
Projects
None yet
Milestone
No milestone
5 participants
@ogrisel @adrinjalali @glemaitre @bmreiniger @ArturoSbr
0