SLEP006: globally setting request values #26050
Description
One of the issues raised in #25776 and #23928 is about a code such as following to work under SLEP6:
est = AdaBoostClassifier(LogisticRegression())
est.fit(X, y, sample_weight=sw)or
GridSearchCV(
LogisticRegression(),
scorer=a_scorer_supporting_sample_weight, ...
).fit(..., sample_weight=sw)without having to change the code, or w/o having to write too much boilerplate code, or w/o having to explicitly set request values for sample_weight for these usual usecases.
In order to allow the above pattern, the proposal is to allow users to set request values globally. These values can either be set as a configuration, or via dedicated methods. The user code at the end would look like:
sklearn.set_fit_request(sample_weight=True)
sklearn.set_score_request(sample_weight=True)
GridSearchCV(
LogisticRegression(),
scorer=a_scorer_supporting_sample_weight, ...
).fit(..., sample_weight=sw)What the above code does, is to set he default value to REQUESTED instead of ERROR_IF_PASSED for sample_weight, on fit and scoremethods / scorers everywhere.
API
A generic way to achieve this would be to set global default request values, on any method, for any metadata, which can generalize to groups and other metadata present in third party libraries.
Alternatively, we could only allow this for sample_weight (I rather not do this), and do it with something like:
sklearn.set_config(fit_requests_sample_weight=True)or if we don't want to distinguish between methods:
sklearn.set_config(sample_weight_request=True)My proposal would be:
sklearn.set_{method}_request(metadata=value)which follows the API on the estimator level, except that here we're setting it on the library scope.
Context Manager
An addition to the above API, we can provide a context manager for the same purpose. If we expose this through a config, it's automatically done via config_context. If we follow another API, we can also provide the corresponding context managers.
Effect on SLEP6's examples
Here we look at how the examples in the SLEP would change.
Nested Grouped Cross Validation
weighted_acc = make_scorer(accuracy_score).set_score_request(sample_weight=True)
log_reg = LogisticRegressionCV(cv=GroupKFold(), scoring=weighted_acc).set_fit_request(
sample_weight=True
)
cv_results = cross_validate(
log_reg,
X,
y,
cv=GroupKFold(),
metadata={"sample_weight": my_weights, "groups": my_groups},
scoring=weighted_acc,
)becomes:
sklearn.set_fit_request(sample_weight=True)
sklearn.set_score_request(sample_weight=True)
weighted_acc = make_scorer(accuracy_score)
log_reg = LogisticRegressionCV(cv=GroupKFold(), scoring=weighted_acc)
cv_results = cross_validate(
log_reg,
X,
y,
cv=GroupKFold(),
metadata={"sample_weight": my_weights, "groups": my_groups},
scoring=weighted_acc,
)And users can still override the default global value at the estimator level, for example, the next example:
weighted_acc = make_scorer(accuracy_score).set_score_request(sample_weight=True)
log_reg = LogisticRegressionCV(cv=group_cv, scoring=weighted_acc).fit_request(
sample_weight=False
)
cross_validate(
log_reg,
X,
y,
cv=GroupKFold(),
metadata={"sample_weight": weights, "groups": groups},
scoring=weighted_acc,
)becomes
sklearn.set_fit_request(sample_weight=True)
sklearn.set_score_request(sample_weight=True)
weighted_acc = make_scorer(accuracy_score)
log_reg = LogisticRegressionCV(cv=group_cv, scoring=weighted_acc).fit_request(
sample_weight=False
)
cross_validate(
log_reg,
X,
y,
cv=GroupKFold(),
metadata={"sample_weight": weights, "groups": groups},
scoring=weighted_acc,
)Unweighted Feature selection
Here there's no request value for SelectKBest set since it doesn't support sample_weight:
weighted_acc = make_scorer(accuracy_score).set_score_request(sample_weight=True)
log_reg = LogisticRegressionCV(cv=GroupKFold(), scoring=weighted_acc).set_fit_request(
sample_weight=True
)
sel = SelectKBest(k=2)
pipe = make_pipeline(sel, log_reg)
pipe.fit(X, y, sample_weight=weights, groups=groups)it becomes:
sklearn.set_fit_request(sample_weight=True)
sklearn.set_score_request(sample_weight=True)
weighted_acc = make_scorer(accuracy_score)
log_reg = LogisticRegressionCV(cv=GroupKFold(), scoring=weighted_acc)
sel = SelectKBest(k=2)
pipe = make_pipeline(sel, log_reg)
pipe.fit(X, y, sample_weight=weights, groups=groups)More advanced use cases with aliased routing
Aliased routing is not supported at the moment, and if the user wants to pass different sample weights to different objects, they'll need to use the verbose API.
Open Questions
- How should the API exactly look like? Do we like the proposal here?
- Do we want to also implement the context manager?
sample_weight="auto"
We also talked about good default request values for sample_weight specifically (#25776 (comment)), to allow users to follow our recommendations. This would be enabled by doing something like:
sklearn.set_config(auto_sample_weight_routing=True)or more explicitly
sklearn.set_{method}_request(sample_weight='auto')This would enable recommended routing, whatever we decide that to be. This is NOT a blocker for releasing SLEP6 and can be implemented after it's released, and recommendations are subject to change, and we tell users that they can change.
There shall be a separate issue to talk about the default values and how the end user API should look like. It is only mentioned here to give context on what we plan to do.
cc @scikit-learn/core-devs