You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
Classes included in this module are :class:`PLSRegression`
39
+
Apart from CCA, the PLS estimators are particularly suited when the matrix of
40
+
predictors has more variables than observations, and when there is
41
+
multicollinearity among the features. By contrast, standard linear regression
42
+
would fail in these cases unless it is regularized.
43
+
44
+
Classes included in this module are :class:`PLSRegression`,
32
45
:class:`PLSCanonical`, :class:`CCA` and :class:`PLSSVD`
33
46
47
+
PLSCanonical
48
+
------------
49
+
50
+
We here describe the algorithm used in :class:`PLSCanonical`. The other
51
+
estimators use variants of this algorithm, and are detailed below.
52
+
We recommend section [1]_ for more details and comparisons between these
53
+
algorithms. In [1]_, :class:`PLSCanonical` corresponds to "PLSW2A".
54
+
55
+
Given two centered matrices :math:`X \in\mathbb{R}^{n \times d}` and
56
+
:math:`Y \in\mathbb{R}^{n \times t}`, and a number of components :math:`K`,
57
+
:class:`PLSCanonical` proceeds as follows:
58
+
59
+
Set :math:`X_1` to :math:`X` and :math:`Y_1` to :math:`Y`. Then, for each
60
+
:math:`k \in [1, K]`:
61
+
62
+
- a) compute :math:`u_k \in\mathbb{R}^d` and :math:`v_k \in\mathbb{R}^t`,
63
+
the first left and right singular vectors of the cross-covariance matrix
64
+
:math:`C = X_k^T Y_k`.
65
+
:math:`u_k` and :math:`v_k` are called the *weights*.
66
+
By definition, :math:`u_k` and :math:`v_k` are
67
+
choosen so that they maximize the covariance between the projected
68
+
:math:`X_k` and the projected target, that is :math:`\text{Cov}(X_k u_k,
69
+
Y_k v_k)`.
70
+
- b) Project :math:`X_k` and :math:`Y_k` on the singular vectors to obtain
71
+
*scores*: :math:`\xi_k = X_k u_k` and :math:`\omega_k = Y_k v_k`
72
+
- c) Regress :math:`X_k` on :math:`\xi_k`, i.e. find a vector :math:`\gamma_k
73
+
\in\mathbb{R}^d` such that the rank-1 matrix :math:`\xi_k \gamma_k^T`
74
+
is as close as possible to :math:`X_k`. Do the same on :math:`Y_k` with
75
+
:math:`\omega_k` to obtain :math:`\delta_k`. The vectors
76
+
:math:`\gamma_k` and :math:`\delta_k` are called the *loadings*.
77
+
- d) *deflate* :math:`X_k` and :math:`Y_k`, i.e. subtract the rank-1
78
+
approximations: :math:`X_{k+1} = X_k - \xi_k \gamma_k^T`, and
79
+
:math:`Y_{k + 1} = Y_k - \omega_k \delta_k^T`.
80
+
81
+
At the end, we have approximated :math:`X` as a sum of rank-1 matrices:
82
+
:math:`X = \Xi\Gamma^T` where :math:`\Xi\in\mathbb{R}^{n \times K}`
83
+
contains the scores in its columns, and :math:`\Gamma^T \in\mathbb{R}^{K
84
+
\times d}` contains the loadings in its rows. Similarly for :math:`Y`, we
85
+
have :math:`Y = \Omega\Delta^T`.
86
+
87
+
Note that the scores matrices :math:`\Xi` and :math:`\Omega` correspond to
88
+
the projections of the training data :math:`X` and :math:`Y`, respectively.
89
+
90
+
Step *a)* may be performed in two ways: either by computing the whole SVD of
91
+
:math:`C` and only retain the singular vectors with the biggest singular
92
+
values, or by directly computing the singular vectors using the power method (cf section 11.3 in [1]_),
93
+
which corresponds to the `'nipals'` option of the `algorithm` parameter.
94
+
95
+
96
+
Transforming data
97
+
^^^^^^^^^^^^^^^^^
98
+
99
+
To transform :math:`X` into :math:`\bar{X}`, we need to find a projection
100
+
matrix :math:`P` such that :math:`\bar{X} = XP`. We know that for the
101
+
training data, :math:`\Xi = XP`, and :math:`X = \Xi\Gamma^T`. Setting
102
+
:math:`P = U(\Gamma^T U)^{-1}` where :math:`U` is the matrix with the
103
+
:math:`u_k` in the columns, we have :math:`XP = X U(\Gamma^T U)^{-1} = \Xi
104
+
(\Gamma^T U) (\Gamma^T U)^{-1} = \Xi` as desired. The rotation matrix
105
+
:math:`P` can be accessed from the `x_rotations_` attribute.
106
+
107
+
Similarly, :math:`Y` can be transformed using the rotation matrix
108
+
:math:`V(\Delta^T V)^{-1}`, accessed via the `y_rotations_` attribute.
109
+
110
+
Predicting the targets Y
111
+
^^^^^^^^^^^^^^^^^^^^^^^^
112
+
113
+
To predict the targets of some data :math:`X`, we are looking for a
114
+
coefficient matrix :math:`\beta\in R^{d \times t}` such that :math:`Y =
115
+
X\beta`.
116
+
117
+
The idea is to try to predict the transformed targets :math:`\Omega` as a
118
+
function of the transformed samples :math:`\Xi`, by computing :math:`\alpha
119
+
\in\mathbb{R}` such that :math:`\Omega = \alpha\Xi`.
120
+
121
+
Then, we have :math:`Y = \Omega\Delta^T = \alpha\Xi\Delta^T`, and since
122
+
:math:`\Xi` is the transformed training data we have that :math:`Y = X \alpha
123
+
P \Delta^T`, and as a result the coefficient matrix :math:`\beta = \alpha P
124
+
\Delta^T`.
125
+
126
+
:math:`\beta` can be accessed through the `coef_` attribute.
127
+
128
+
PLSSVD
129
+
------
130
+
131
+
:class:`PLSSVD` is a simplified version of :class:`PLSCanonical`
132
+
described earlier: instead of iteratively deflating the matrices :math:`X_k`
133
+
and :math:`Y_k`, :class:`PLSSVD` computes the SVD of :math:`C = X^TY`
134
+
only *once*, and stores the `n_components` singular vectors corresponding to
135
+
the biggest singular values in the matrices `U` and `V`, corresponding to the
136
+
`x_weights_` and `y_weights_` attributes. Here, the transformed data is
137
+
simply `transformed(X) = XU` and `transformed(Y) = YV`.
138
+
139
+
If `n_components == 1`, :class:`PLSSVD` and :class:`PLSCanonical` are
140
+
strictly equivalent.
141
+
142
+
PLSRegression
143
+
-------------
144
+
145
+
The :class:`PLSRegression` estimator is similar to
146
+
:class:`PLSCanonical` with `algorithm='nipals'`, with 2 significant
147
+
differences:
148
+
149
+
- at step a) in the power method to compute :math:`u_k` and :math:`v_k`,
150
+
:math:`v_k` is never normalized.
151
+
- at step c), the targets :math:`Y_k` are approximated using the projection
152
+
of :math:`X_k` (i.e. :math:`\xi_k`) instead of the projection of
153
+
:math:`Y_k` (i.e. :math:`\omega_k`). In other words, the loadings
154
+
computation is different. As a result, the deflation in step d) will also
155
+
be affected.
156
+
157
+
These two modifications affect the output of `predict` and `transform`,
158
+
which are not the same as for :class:`PLSCanonical`. Also, while the number
159
+
of components is limited by `min(n_samples, n_features, n_targets)` in
160
+
:class:`PLSCanonical`, here the limit is the rank of :math:`X^TX`, i.e.
161
+
`min(n_samples, n_features)`.
162
+
163
+
:class:`PLSRegression` is also known as PLS1 (single targets) and PLS2
164
+
(multiple targets). Much like :class:`~sklearn.linear_model.Lasso`,
165
+
:class:`PLSRegression` is a form of regularized linear regression where the
166
+
number of components controls the strength of the regularization.
167
+
168
+
Canonical Correlation Analysis
169
+
------------------------------
170
+
171
+
Canonical Correlation Analysis was developed prior and independently to PLS.
172
+
But it turns out that :class:`CCA` is a special case of PLS, and corresponds
173
+
to PLS in "Mode B" in the literature.
174
+
175
+
:class:`CCA` differs from :class:`PLSCanonical` in the way the weights
176
+
:math:`u_k` and :math:`v_k` are computed in the power method of step a).
177
+
Details can be found in section 10 of [1]_.
178
+
179
+
Since :class:`CCA` involves the inversion of :math:`X_k^TX_k` and
180
+
:math:`Y_k^TY_k`, this estimator can be unstable if the number of features or
181
+
targets is greater than the number of samples.
182
+
34
183
35
184
.. topic:: Reference:
36
185
37
-
* JA Wegelin
38
-
`A survey of Partial Least Squares (PLS) methods, with emphasis on the two-block case <https://www.stat.washington.edu/research/reports/2000/tr371.pdf>`_
186
+
.. [1] `A survey of Partial Least Squares (PLS) methods, with emphasis on
0 commit comments