8000 Revert "ENH Added penalty='none' to LogisticRegression (#12860)" · xhluca/scikit-learn@0391f5f · GitHub
[go: up one dir, main page]
Skip to content

Commit 0391f5f

Browse files
author
Xing
authored
Revert "ENH Added penalty='none' to LogisticRegression (scikit-learn#12860)"
This reverts commit 8adbddb.
1 parent 439295b commit 0391f5f

File tree

4 files changed

+23
-94
lines changed

4 files changed

+23
-94
lines changed

‎doc/modules/linear_model.rst‎

Lines changed: 6 additions & 8 deletions
Original file line numberDiff line numberDiff line change
@@ -731,7 +731,7 @@ or the log-linear classifier. In this model, the probabilities describing the po
731731
The implementation of logistic regression in scikit-learn can be accessed from
732732
class :class:`LogisticRegression`. This implementation can fit binary, One-vs-
733733
Rest, or multinomial logistic regression with optional L2, L1 or Elastic-Net
734-
regularization. Note that regularization is applied by default.
734+
regularization.
735735

736736
As an optimization problem, binary class L2 penalized logistic regression
737737
minimizes the following cost function:
@@ -771,11 +771,11 @@ classifiers. For L1 penalization :func:`sklearn.svm.l1_min_c` allows to
771771
calculate the lower bound for C in order to get a non "null" (all feature
772772
weights to zero) model.
773773

774-
The "lbfgs", "sag" and "newton-cg" solvers only support L2 penalization or no
775-
regularization, and are found to converge faster for some high dimensional
776-
data. Setting `multi_class` to "multinomial" with these solvers learns a true
777-
multinomial logistic regression model [5]_, which means that its probability
778-
estimates should be better calibrated than the default "one-vs-rest" setting.
774+
The "lbfgs", "sag" and "newton-cg" solvers only support L2 penalization and
775+
are found to converge faster for some high dimensional data. Setting
776+
`multi_class` to "multinomial" with these solvers learns a true multinomial
777+
logistic regression model [5]_, which means that its probability estimates
778+
should be better calibrated than the default "one-vs-rest" setting.
779779

780780
The "sag" solver uses a Stochastic Average Gradient descent [6]_. It is faster
781781
than other solvers for large datasets, when both the number of samples and the
@@ -808,8 +808,6 @@ The following table summarizes the penalties supported by each solver:
808808
+------------------------------+-----------------+-------------+-----------------+-----------+------------+
809809
| Elastic-Net | no | no | no | no | yes |
810810
+------------------------------+-----------------+-------------+-----------------+-----------+------------+
811-
| No penalty ('none') | no | yes | yes | yes | yes |
812-
+------------------------------+-----------------+-------------+-----------------+-----------+------------+
813811
| **Behaviors** | |
814812
+------------------------------+-----------------+-------------+-----------------+-----------+------------+
815813
| Penalize the intercept (bad) | yes | no | no | no | no |

‎doc/whats_new/v0.21.rst‎

Lines changed: 0 additions & 6 deletions
Original file line numberDiff line numberDiff line change
@@ -101,12 +101,6 @@ Support for Python 3.4 and below has been officially dropped.
101101
:class:`linear_model.LogisticRegressionCV` now support Elastic-Net penalty,
102102
with the 'saga' solver. :issue:`11646` by :user:`Nicolas Hug <NicolasHug>`.
103103

104-
- |Enhancement| :class:`linear_model.LogisticRegression` now supports an
105-
unregularized objective by setting ``penalty`` to ``'none'``. This is
106-
equivalent to setting ``C=np.inf`` with l2 regularization. Not supported
107-
by the liblinear solver. :issue:`12860` by :user:`Nicolas Hug
108-
<NicolasHug>`.
109-
110104
- |Fix| Fixed a bug in :class:`linear_model.LogisticRegression` and
111105
:class:`linear_model.LogisticRegressionCV` with 'saga' solver, where the
112106
weights would not be correctly updated in some cases.

‎sklearn/linear_model/logistic.py‎

Lines changed: 16 additions & 45 deletions
Original file line numberDiff line numberDiff line change
@@ -437,13 +437,13 @@ def _check_solver(solver, penalty, dual):
437437
raise ValueError("Logistic Regression supports only solvers in %s, got"
438438
" %s." % (all_solvers, solver))
439439

440-
all_penalties = ['l1', 'l2', 'elasticnet', 'none']
440+
all_penalties = ['l1', 'l2', 'elasticnet']
441441
if penalty not in all_penalties:
442442
raise ValueError("Logistic Regression supports only penalties in %s,"
443443
" got %s." % (all_penalties, penalty))
444444

445-
if solver not in ['liblinear', 'saga'] and penalty not in ('l2', 'none'):
446-
raise ValueError("Solver %s supports only 'l2' or 'none' penalties, "
445+
if solver not in ['liblinear', 'saga'] and penalty != 'l2':
446+
raise ValueError("Solver %s supports only l2 penalties, "
447447
"got %s penalty." % (solver, penalty))
448448
if solver != 'liblinear' and dual:
449449
raise ValueError("Solver %s supports only "
@@ -452,12 +452,6 @@ def _check_solver(solver, penalty, dual):
452452
if penalty == 'elasticnet' and solver != 'saga':
453453
raise ValueError("Only 'saga' solver supports elasticnet penalty,"
454454
" got solver={}.".format(solver))
455-
456-
if solver == 'liblinear' and penalty == 'none':
457-
raise ValueError(
458-
"penalty='none' is not supported for the liblinear solver"
459-
)
460-
461455
return solver
462456

463457

@@ -1211,27 +1205,24 @@ class LogisticRegression(BaseEstimator, LinearClassifierMixin,
12111205
'sag', 'saga' and 'newton-cg' solvers.)
12121206
12131207
This class implements regularized logistic regression using the
1214-
'liblinear' library, 'newton-cg', 'sag', 'saga' and 'lbfgs' solvers. **Note
1215-
that regularization is applied by default**. It can handle both dense
1216-
and sparse input. Use C-ordered arrays or CSR matrices containing 64-bit
1217-
floats for optimal performance; any other input format will be converted
1218-
(and copied).
1208+
'liblinear' library, 'newton-cg', 'sag', 'saga' and 'lbfgs' solvers. It can
1209+
handle both dense and sparse input. Use C-ordered arrays or CSR matrices
1210+
containing 64-bit floats for optimal performance; any other input format
1211+
will be converted (and copied).
12191212
12201213
The 'newton-cg', 'sag', and 'lbfgs' solvers support only L2 regularization
1221-
with primal formulation, or no regularization. The 'liblinear' solver
1222-
supports both L1 and L2 regularization, with a dual formulation only for
1223-
the L2 penalty. The Elastic-Net regularization is only supported by the
1224-
'saga' solver.
1214+
with primal formulation. The 'liblinear' solver supports both L1 and L2
1215+
regularization, with a dual formulation only for the L2 penalty. The
1216+
Elastic-Net regularization is only supported by the 'saga' solver.
12251217
12261218
Read more in the :ref:`User Guide <logistic_regression>`.
12271219
12281220
Parameters
12291221
----------
1230-
penalty : str, 'l1', 'l2', 'elasticnet' or 'none', optional (default='l2')
1222+
penalty : str, 'l1', 'l2', or 'elasticnet', optional (default='l2')
12311223
Used to specify the norm used in the penalization. The 'newton-cg',
12321224
'sag' and 'lbfgs' solvers support only l2 penalties. 'elasticnet' is
1233-
only supported by the 'saga' solver. If 'none' (not supported by the
1234-
liblinear solver), no regularization is applied.
1225+
only supported by the 'saga' solver.
12351226
12361227
.. versionadded:: 0.19
12371228
l1 penalty with SAGA solver (allowing 'multinomial' + L1)
@@ -1298,10 +1289,8 @@ class LogisticRegression(BaseEstimator, LinearClassifierMixin,
12981289
- For multiclass problems, only 'newton-cg', 'sag', 'saga' and 'lbfgs'
12991290
handle multinomial loss; 'liblinear' is limited to one-versus-rest
13001291
schemes.
1301-
- 'newton-cg', 'lbfgs', 'sag' and 'saga' handle L2 or no penalty
1302-
- 'liblinear' and 'saga' also handle L1 penalty
1303-
- 'saga' also supports 'elasticnet' penalty
1304-
- 'liblinear' does not handle no penalty
1292+
- 'newton-cg', 'lbfgs' and 'sag' only handle L2 penalty, whereas
1293+
'liblinear' and 'saga' handle L1 penalty.
13051294
13061295
Note that 'sag' and 'saga' fast convergence is only guaranteed on
13071296
features with approximately the same scale. You can
@@ -1502,18 +1491,6 @@ def fit(self, X, y, sample_weight=None):
15021491
warnings.warn("l1_ratio parameter is only used when penalty is "
15031492
"'elasticnet'. Got "
15041493
"(penalty={})".format(self.penalty))
1505-
if self.penalty == 'none':
1506-
if self.C != 1.0: # default values
1507-
warnings.warn(
1508-
"Setting penalty='none' will ignore the C and l1_ratio "
1509-
"parameters"
1510-
)
1511-
# Note that check for l1_ratio is done right above
1512-
C_ = np.inf
1513-
penalty = 'l2'
1514-
else:
1515-
C_ = self.C
1516-
penalty = self.penalty
15171494
if not isinstance(self.max_iter, numbers.Number) or self.max_iter < 0:
15181495
raise ValueError("Maximum number of iteration must be positive;"
15191496
" got (max_iter=%r)" % self.max_iter)
@@ -1593,13 +1570,13 @@ def fit(self, X, y, sample_weight=None):
15931570
prefer = 'processes'
15941571
fold_coefs_ = Parallel(n_jobs=self.n_jobs, verbose=self.verbose,
15951572
**_joblib_parallel_args(prefer=prefer))(
1596-
path_func(X, y, pos_class=class_, Cs=[C_],
1573+
path_func(X, y, pos_class=class_, Cs=[self.C],
15971574
l1_ratio=self.l1_ratio, fit_intercept=self.fit_intercept,
15981575
tol=self.tol, verbose=self.verbose, solver=solver,
15991576
multi_class=multi_class, max_iter=self.max_iter,
16001577
class_weight=self.class_weight, check_input=False,
16011578
random_state=self.random_state, coef=warm_start_coef_,
1602-
penalty=penalty, max_squared_sum=max_squared_sum,
1579+
penalty=self.penalty, max_squared_sum=max_squared_sum,
16031580
sample_weight=sample_weight)
16041581
for class_, warm_start_coef_ in zip(classes_, warm_start_coef))
16051582

@@ -1991,12 +1968,6 @@ def fit(self, X, y, sample_weight=None):
19911968

19921969
l1_ratios_ = [None]
19931970

1994-
if self.penalty == 'none':
1995-
raise ValueError(
1996-
"penalty='none' is not useful and not supported by "
1997-
"LogisticRegressionCV."
1998-
)
1999-
20001971
X, y = check_X_y(X, y, accept_sparse='csr', dtype=np.float64,
20011972
order="C",
20021973
accept_large_sparse=solver != 'liblinear')

‎sklearn/linear_model/tests/test_logistic.py‎

Lines changed: 1 addition & 35 deletions
Original file line numberDiff line numberDiff line change
@@ -234,7 +234,7 @@ def test_check_solver_option(LR):
234234

235235
# all solvers except 'liblinear' and 'saga'
236236
for solver in ['newton-cg', 'lbfgs', 'sag']:
237-
msg = ("Solver %s supports only 'l2' or 'none' penalties," %
237+
msg = ("Solver %s supports only l2 penalties, got l1 penalty." %
238238
solver)
239239
lr = LR(solver=solver, penalty='l1', multi_class='ovr')
240240
assert_raise_message(ValueError, msg, lr.fit, X, y)
@@ -253,11 +253,6 @@ def test_check_solver_option(LR):
253253
lr = LR(solver=solver, penalty='elasticnet')
254254
assert_raise_message(ValueError, msg, lr.fit, X, y)
255255

256-
# liblinear does not support penalty='none'
257-
msg = "penalty='none' is not supported for the liblinear solver"
258-
lr = LR(penalty='none', solver='liblinear')
259-
assert_raise_message(ValueError, msg, lr.fit, X, y)
260-
261256

262257
@pytest.mark.parametrize('model, params, warn_solver',
263258
[(LogisticRegression, {}, True),
@@ -1759,32 +1754,3 @@ def test_logistic_regression_path_deprecation():
17591754
assert_warns_message(DeprecationWarning,
17601755
"logistic_regression_path was deprecated",
17611756
logistic_regression_path, X, Y1)
1762-
1763-
1764-
@pytest.mark.parametrize('solver', ('lbfgs', 'newton-cg', 'sag', 'saga'))
1765-
def test_penalty_none(solver):
1766-
# - Make sure warning is raised if penalty='none' and C is set to a
1767-
# non-default value.
1768-
# - Make sure setting penalty='none' is equivalent to setting C=np.inf with
1769-
# l2 penalty.
1770-
X, y = make_classification(n_samples=1000, random_state=0)
1771-
1772-
msg = "Setting penalty='none' will ignore the C"
1773-
lr = LogisticRegression(penalty='none', solver=solver, C=4)
1774-
assert_warns_message(UserWarning, msg, lr.fit, X, y)
1775-
1776-
lr_none = LogisticRegression(penalty='none', solver=solver,
1777-
random_state=0)
1778-
lr_l2_C_inf = LogisticRegression(penalty='l2', C=np.inf, solver=solver,
1779-
random_state=0)
1780-
pred_none = lr_none.fit(X, y).predict(X)
1781-
pred_l2_C_inf = lr_l2_C_inf.fit(X, y).predict(X)
1782-
assert_array_equal(pred_none, pred_l2_C_inf)
1783-
1784-
lr = LogisticRegressionCV(penalty='none')
1785-
assert_raise_message(
1786-
ValueError,
1787-
"penalty='none' is not useful and not supported by "
1788-
"LogisticRegressionCV",
1789-
lr.fit, X, y
1790-
)

0 commit comments

Comments
 (0)
0