10000 when i use the isomap on 27000 X 315 size matrix ,it is giving a memory error on shortest path calculation ? · Issue #8352 · scikit-learn/scikit-learn · GitHub
[go: up one dir, main page]
Skip to content

when i use the isomap on 27000 X 315 size matrix ,it is giving a memory error on shortest path calculation ? #8352

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom
Closed
hackerman008 opened this issue Feb 13, 2017 · 9 comments
Closed

when i use the isomap on 27000 X 315 size matrix ,it is giving a memory error on shortest path calculation ? #8352

hackerman008 opened this issue Feb 13, 2017 · 9 comments

Comments

@hackerman008
Copy link

Description

Steps/Code to Reproduce

Expected Results

Actual Results

Versions
@glemaitre
Copy link
Member
glemaitre commented Feb 13, 2017 •
edited
Loading

You have to fill up in between the markdown sections to give use more info.
Anyway, I would think that the graph required to compute the shortest path will be of size 27000 x 27000. Does such graph fit in your memory computer?

@hackerman008
Copy link
Author
hackerman008 commented Feb 14, 2017 •
edited
Loading

I hav a 4gb ram 32 bit computer.I am using spider as the environment to run the code.As i run the code using the fit_transform() method it throws an error on shortest path calculation here:
self.dist_matrix_ = graph_shortest_path(kng, method=self.path_method, directed=False)
I think the matrix doesnt fit into memory.Is there any other way around this problem. ?
I am using ISOMAP for non-linear dimensionality reduction.Is their any other algorithm which does not require high memory for finding non-linear embedding.?

@glemaitre
Copy link
Member

You could check if there is some of the methods which use sparse matrix for the adjacency/affinity matrix.

@hackerman008
Copy link
Author

the isomap should work for sparse martrix for shortest path finding..?

@glemaitre
Copy link
Member
glemaitre commented Feb 14, 2017 •
edited
Loading

It seems that you need to create dense matrix at some point.

@hackerman008
Copy link
Author
hackerman008 commented Feb 15, 2017 •
edited
Loading

the distance matrix is calcuated by the algorithm and that is passed to this function:
def graph_shortest_path(dist_matrix, directed=True, method='auto'):
which calculates the shortest path distance for each point to every other using the 'FW' or 'Dijkstra' method
so the dist matrix calculated would stored as sparse or dense. ?
and since sparse matrix is compressed how will that cause memory errors.?

@GaelVaroquaux
Copy link
Member
GaelVaroquaux commented Feb 15, 2017 via email

@hackerman008
Copy link
Author

but before finding the shortest path it converts it into a sparse matrix for memory consideration.
It shows an error while computing the shortest path.I wanted to is there any way around it? @GaelVaroquaux

@TomDLT TomDLT mentioned this issue Aug 8, 2019
Closed
@TomDLT TomDLT mentioned this issue Jul 14, 2021
Merged
@TomDLT
Copy link
Member
TomDLT commented Aug 3, 2021

FIxed in #20531, the shortest path can now be computed on sparse matrices.

@TomDLT TomDLT closed this as completed Aug 3, 2021
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
None yet
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
4 participants
@GaelVaroquaux @glemaitre @TomDLT @hackerman008
0