FEA Implement classical MDS #31322

dkobak · 2025-05-06T13:50:42Z

Fixes #15272. Supersedes #22330.

This PR implements classical MDS, also known as principal coordinates analysis (PCoA) or Torgerson's scaling, see https://en.wikipedia.org/wiki/Multidimensional_scaling#Classical_multidimensional_scaling. As discussed in #22330, it is implemented as a new class ClassicalMDS.

Simple demonstration:

import pylab as plt
import numpy as np

from sklearn.datasets import load_iris
from sklearn.manifold import ClassicalMDS
from sklearn.decomposition import PCA

X, y = load_iris(return_X_y=True)

Z1 = PCA(n_components=2).fit_transform(X)
Z2 = ClassicalMDS(n_components=2, metric="euclidean").fit_transform(X)
Z3 = ClassicalMDS(n_components=2, metric="cosine"   ).fit_transform(X)
Z4 = ClassicalMDS(n_components=2, metric="manhattan").fit_transform(X)

fig, axs = plt.subplots(nrows=2, ncols=2, figsize=(6, 6), layout="constrained")

axs.flat[0].scatter(Z1[:,0], Z1[:,1], c=y)
axs.flat[0].set_title("PCA")

axs.flat[1].scatter(Z2[:,0], Z2[:,1], c=y)
axs.flat[1].set_title("Classical MDS, Euclidean dist.")

axs.flat[2].scatter(-Z3[:,0], Z3[:,1], c=y)
axs.flat[2].set_title("Classical MDS, cosine dist.")

axs.flat[3].scatter(Z4[:,0], Z4[:,1], c=y)
axs.flat[3].set_title("Classical MDS, Manhattan dist.")

~~Classical MDS is also set as default initialization for metric/non-metric MDS in the MDS() class.~~

~~For consistency, this PR also adds support for non-Euclidean metrics to the MDS class.~~

github-actions · 2025-05-06T13:51:41Z

✔️ Linting Passed

All linting checks passed. Your pull request is in excellent shape! ☀️

_{Generated for commit: 1d53e85. Link to the linter CI: here}

dkobak · 2025-05-08T09:25:34Z

Pinging @antoinebaker for a review :-) I have been doing some updates to this PR but am now finished with it.

antoinebaker · 2025-05-12T08:32:18Z

Thanks for the PR @dkobak !

Could you maybe use this PR to implement ClassicalMDS only, and do the enhancements of MDS in a separate / follow up PR ? It will ease the reviewing / merging process I think.

dkobak · 2025-05-12T08:50:02Z

Could you maybe use this PR to implement ClassicalMDS only, and do the enhancements of MDS in a separate / follow up PR ? It will ease the reviewing / merging process I think.

All right. I took all changes to the MDS class out of this PR now. I will wait until this PR is merged and then do a follow-up PR to change the MDS class.

antoinebaker · 2025-05-28T13:02:05Z

Could you please add ClassicalMDS to the common test suite for estimators ? It can be done by adding an entry to the INIT_PARAMS dict:

scikit-learn/sklearn/utils/_test_common/instance_generator.py

Lines 185 to 187 in 4493f86

    
           INIT_PARAMS = { 
        
               AdaBoostClassifier: dict(n_estimators=5), 
        
               AdaBoostRegressor: dict(n_estimators=5),

Then it will be tested for common checks on estimators in sklearn/tests/test_common.py::test_estimators

dkobak · 2025-05-28T13:14:32Z

Hi @antoinebaker. I think the INIT_PARAMS dictionary is only needed to provide non-default params (like a small number of iterations), which for ClassicalMDS is not needed. An estimator does not have to be in INIT_PARAMS in order to be tested, it will simply be tested with default params in that case. Let me know if I am wrong.

antoinebaker

Here a first round of reviews

sklearn/manifold/_classical_mds.py

antoinebaker · 2025-05-28T14:07:16Z

Hi @antoinebaker. I think the INIT_PARAMS dictionary is only needed to provide non-default params (like a small number of iterations), which for ClassicalMDS is not needed. An estimator does not have to be in INIT_PARAMS in order to be tested, it will simply be tested with default params in that case. Let me know if I am wrong.

Ah my bad, I think you're right. If ClassicalMDS appears in the sklearn/tests/test_common.py::test_estimators suite that's all good.

doc/whats_new/upcoming_changes/sklearn.manifold/31322.feature.rst

adrinjalali · 2025-06-03T13:27:34Z

I'm not sure if we discussed this, but I favor this comment (#22330 (comment)) (except the default value) as an overall API, instead of introducing a new class. Any blockers for doing so?

dkobak · 2025-06-03T13:38:41Z

@adrinjalali: Yes, we did discuss it. Just below the comment you linked to, @antoinebaker gave detailed reasons for why he prefers a separate class, please see here: #22330 (comment). He convinced me, and you wrote "I'm happy with the suggestions here" (#22330 (comment)), which is why I implemented a separate class...

adrinjalali · 2025-06-03T13:58:42Z

Hmm. Yeah fair enough. Just to avoid future surprises, maybe @lorentzenchr @GaelVaroquaux could also give their opinion?

antoinebaker · 2025-06-04T08:35:14Z

The TLDR of #22330 (comment) is that ClassicalMDS (Principal Coordinates Analysis) implemented in this PR and the current MDS (SMACOF) implemented in sklearn don't have much in common except their names (different algorithms, objectives, arguments, attributes).

EDIT: removed outdated comment.

Co-authored-by: Adrin Jalali <adrin.jalali@gmail.com>

dkobak · 2025-07-03T09:38:01Z

Regarding the user guide, it is already there.

Regarding examples, good point, I have now added classical MDS (and also non-metric MDS) to plot_compare_methods.py, plot_lle_digits.py, plot_manifold_sphere.py, plot_mds.py.

(Edit: checked the rendered docs and examples are fine.)

dkobak · 2025-07-16T09:50:51Z

@adrinjalali See above -- I have added examples. Let me know if there is anything else I can do on my side.

antoinebaker

I read the examples, and overall this LGTM !

The nonmetric MDS seems to perform badly on S-curve and sphere (plot_compare_methods and plot_manifold_sphere). Is this an expected limitation of this method on such dataset ? Or is there something wrong in term of hyperparameters (eg convergence controlled by eps) ? Note that it's not directly related to this PR (which is on ClassicalMDS which seems to perform okay), so I still approve the changes. But it could be nice to fix this nonmetric mds example if possible.

examples/manifold/plot_manifold_sphere.py

Co-authored-by: antoinebaker <antoinebaker@users.noreply.github.com>

dkobak · 2025-07-30T12:08:43Z

The nonmetric MDS seems to perform badly on S-curve and sphere (plot_compare_methods and plot_manifold_sphere). Is this an expected limitation of this method on such dataset ? Or is there something wrong in term of hyperparameters (eg convergence controlled by eps) ? Note that it's not directly related to this PR (which is on ClassicalMDS which seems to perform okay), so I still approve the changes. But it could be nice to fix this nonmetric mds example if possible.

Yes, I also noticed that. It is not expected and should be fixed. Optimization of non-metric MDS is finicky and I am not sure what exactly is going wrong there. But I would like to switch both metric and non-metric MDS to use classical MDS as default initialization (rather than random initialization), and I think this should take care of this issue.

Once this PR is merged, I will prepare a follow-up PR that

adds other metrics to the MDS class (metric="cosine" etc.)
changes the current metric=True argument to the MDS class to metric_mds=True
sets classical MDS as default initialization for MDS.

dkobak · 2025-08-04T08:59:08Z

Hi @adrinjalali, just a gentle ping to take another look here. But if you are on holidays then enjoy your holidays :-)

adrinjalali · 2025-08-04T12:30:16Z

Thanks for the work so far. I'm happy with what we have, except the docstring / userguide for this class. In none of the two places the formulation is explained well enough I feel. Could you please improve the small section in the user guide and also the docstring please?

…nto classical-mds

dkobak · 2025-08-04T14:49:29Z

@adrinjalali Fair enough. I added some details, take a look if this is sufficient now.

doc/modules/manifold.rst

sklearn/manifold/_classical_mds.py

Co-authored-by: Adrin Jalali <adrin.jalali@gmail.com>

adrinjalali

LGTM. Thanks. Since it's a new estimator, I'd like another pair of eyes on this before merging. @adam2392 or @OmarManzoor could have a look maybe?

dkobak · 2025-08-25T09:59:46Z

Dear @adam2392, hope you are having a nice summer! Just a gentle reminder that it would be amazing if you could look at this PR :-) Cheers.

OmarManzoor

LGTM. Thanks @dkobak

Implement Classical MDS

33b9600

github-actions bot added the module:manifold label May 6, 2025

Rename what's new file

5184552

dkobak mentioned this pull request May 6, 2025

ENH Add eigh as a solver in MDS #22330

Open

dkobak added 2 commits May 6, 2025 16:22

validate input data

c215d0a

describe attributes

d6773c7

dkobak mentioned this pull request May 6, 2025

sklearn MDS vs skbio PCoA #15272

Closed

dkobak added 4 commits May 6, 2025 22:06

Merge branch 'main' into classical-mds

b4f49d5

Make classical MDS default init for MDS

f018564

Fix tests

99854db

Merge branch 'main' into classical-mds

8134b9f

Take all changes to the MDS class out of this PR

bafac04

dkobak changed the title ~~Implement classical MDS~~ FEA Implement classical MDS May 14, 2025

antoinebaker reviewed May 28, 2025

View reviewed changes

sklearn/manifold/_classical_mds.py Outdated Show resolved Hide resolved

sklearn/manifold/_classical_mds.py Outdated Show resolved Hide resolved

sklearn/manifold/_classical_mds.py Outdated Show resolved Hide resolved

sklearn/manifold/_classical_mds.py Outdated Show resolved Hide resolved

antoinebaker reviewed May 28, 2025

View reviewed changes

doc/whats_new/upcoming_changes/sklearn.manifold/31322.feature.rst Outdated Show resolved Hide resolved

dkobak added 3 commits May 28, 2025 17:00

Add metric_params

c408bd3

Fix kwargs

77732f6

Rename kwargs

804c524

Rename dissimilarity into metric

0999113

dkobak and others added 3 commits July 3, 2025 11:02

Update doc/api_reference.py

a088ac4

Co-authored-by: Adrin Jalali <adrin.jalali@gmail.com>

Rename feature into major-feature

5b17eb8

Add Classical MDS to examples

f3fdf51

dkobak added 3 commits July 3, 2025 12:00

Fix one example

ab479c6

Fix examples

5e6c3fe

More example edits

fe76707

antoinebaker approved these changes Jul 30, 2025

View reviewed changes

examples/manifold/plot_manifold_sphere.py Outdated Show resolved Hide resolved

dkobak and others added 2 commits July 30, 2025 14:01

Update examples/manifold/plot_manifold_sphere.py

b4713d6

Co-authored-by: antoinebaker <antoinebaker@users.noreply.github.com>

Merge branch 'main' into classical-mds

75c9b84

dkobak added 2 commits July 30, 2025 14:12

Use absolute import paths

bf4a01c

Sort imports

deb2e2f

dkobak added 4 commits August 4, 2025 15:27

Add details to the user guide and docstring

de62dcd

Merge branch 'main' of https://github.com/scikit-learn/scikit-learn i…

f8f1d6b

…nto classical-mds

Format docstring correctly

3112cbe

Fix renumbered images in Digits

e295e6a

adrinjalali reviewed Aug 5, 2025

View reviewed changes

doc/modules/manifold.rst Outdated Show resolved Hide resolved

sklearn/manifold/_classical_mds.py Show resolved Hide resolved

Update doc/modules/manifold.rst

e9b2e76

Co-authored-by: Adrin Jalali <adrin.jalali@gmail.com>

adrinjalali approved these changes Aug 5, 2025

View reviewed changes

Add NMDS picture to the user guide

1c1a333

adam2392 self-requested a review August 5, 2025 14:51

Merge branch 'main' into classical-mds

1d53e85

OmarManzoor approved these changes Sep 8, 2025

View reviewed changes

OmarManzoor merged commit 730d651 into scikit-learn:main Sep 8, 2025
36 checks passed

Uh oh!

FEA Implement classical MDS #31322

FEA Implement classical MDS #31322

Uh oh!

Conversation

Uh oh!

Uh oh!

Uh oh!

✔️ Linting Passed

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

antoinebaker left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!