8000 switch CUDA svd and qr to using cuSolver · Issue #4689 · pytorch/pytorch · GitHub
[go: up one dir, main page]
Skip to content

switch CUDA svd and qr to using cuSolver #4689

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom
Closed
soumith opened this issue Jan 16, 2018 · 5 comments
Closed

switch CUDA svd and qr to using cuSolver #4689

soumith opened this issue Jan 16, 2018 · 5 comments
Labels
module: cuda Related to torch.cuda, and CUDA support in general module: linear algebra Issues related to specialized linear algebra operations in PyTorch; includes matrix multiply matmul triaged This issue has been looked at a team member, and triaged and prioritized into an appropriate module

Comments

@soumith
Copy link
Member
soumith commented Jan 16, 2018 •
edited by pytorch-probot bot
Loading

Currently we use MAGMA for these solvers, which uses a mix of CPU and GPU. Wonder if cuSolver will be faster, and we can avoid the CPU path entirely.

cc: @ngimel just want to confirm, cuSolver (dense) wont use the CPU right?

cc @ngimel @vincentqb @vishwakftw @ssnl @jianyuh
@ngimel
Copy link
Collaborator
ngimel commented Jan 16, 2018

Checked with cuSolver team, qr is GPU only, for svd they have CPU path (gesvd) and GPU-only path (gesvdj). They don't benchmark against magma, so YMMV.

@PiotrSokol
Copy link

@soumith
I'm sorry to go a bit off topic but I can't find reference about using gpu supported svd/qr. Is it possible to force MAGMA to run the factorization on the gpu?

@ssnl
Copy link
Collaborator
ssnl commented Jan 19, 2018

@PiotrSokol MAGMA's implementation of qr and svd has CPU calls. There are no alternative in MAGMA for these calls afaik. So no.

@PiotrSokol
Copy link

@ssnl Thanks!

@soumith Concerning the benchmarking, cupy uses cusolver as a backend and I compared it, though not very dilligently, to pytorch/MAGMA. MAGMA is ~2.5 faster on my setup, which is a 46 core Xeon CPU E5-2690 v4 @ 2.60GHz and a Tesla P100 GPU.

@jjbouza jjbouza mentioned this issue Nov 19, 2018
Closed
@gchanan gchanan added module: cuda Related to torch.cuda, and CUDA support in general module: linear algebra Issues related to specialized linear algebra operations in PyTorch; includes matrix multiply matmul triaged This issue has been looked at a team member, and triaged and prioritized into an appropriate module labels Jul 20, 2020
@mruberry mruberry mentioned this issue Oct 5, 2020
Closed
4 tasks
@IvanYashchuk IvanYashchuk mentioned this issue Feb 4, 2021
Open
@lezcano lezcano mentioned this issue Sep 6, 2021
Closed
@IvanYashchuk
Copy link
Collaborator

cuSOLVER paths for QR and SVD for CUDA 10.2+ builds were added with #56256 and #48436.

facebook-github-bot pushed a commit that referenced this issue Oct 26, 2021
@xwang233 @facebook-github-bot
[CUDA][Linalg] Add gesvd as SVD fallback; optimize SVD gesvdj perform…
204ffd3 
…ance (#64533)

Summary:
Fix #64237
Fix #28293
Fix #4689

See also #47953

cc ngimel jianyuh nikitaved pearu mruberry walterddr IvanYashchuk xwang233 Lezcano

Pull Request resolved: #64533

Reviewed By: albanD

Differential Revision: D31915794

Pulled By: ngimel

fbshipit-source-id: 29ea48696531ced8a48474e891a9e2d5f11e9d7a
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
module: cuda Related to torch.cuda, and CUDA support in general module: linear algebra Issues related to specialized linear algebra operations in PyTorch; includes matrix multiply matmul triaged This issue has been looked at a team member, and triaged and prioritized into an appropriate module
Projects
None yet
Milestone
No milestone
Development

Successfully merging a pull request may close this issue.

[CUDA][Linalg] Add gesvd as SVD fallback; optimize SVD gesvdj performance xwang233/pytorch
6 participants
@soumith @gchanan @ssnl @PiotrSokol @ngimel @IvanYashchuk
0