8000 [feature request] Global GPU Flag · Issue #7535 · pytorch/pytorch · GitHub
[go: up one dir, main page]
Skip to content

[feature request] Global GPU Flag #7535

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom
Open
svaisakh opened this issue May 8000 13, 2018 · 12 comments
Open

[feature request] Global GPU Flag #7535

svaisakh opened this issue May 13, 2018 · 12 comments
Labels
feature A request for a proper, new feature. module: cuda Related to torch.cuda, and CUDA support in general triaged This issue has been looked at a team member, and triaged and prioritized into an appropriate module

Comments

@svaisakh
Copy link

The PyTorch 0.4 Migration Guide, simplifies writing device-agnostic code as follows:

# at beginning of the script
device = torch.device("cuda:0" if torch.cuda.is_available() else "cpu")

...

# then whenever you get a new Tensor or Module
# this won't copy if they are already on the desired device
input = data.to(device)
model = MyModule(...).to(device)

However, this is still not clean.

Ideally, we would like PyTorch to move everything over to the GPU, if it's available...
much like TensorFlow.

I tried setting the global tensor type to a cuda tensor using the torch.set_default_tensor_type() method.

However, there are some fundamental problems with setting the default tensor type.

  • Dataloaders give normal (non-cuda) tensors by default. They have to be manually cast using the Tensor.to() method.

  • Many methods are simply not implemented for torch.cuda.*Tensor. Thus, setting the global tensor type to cuda fails.

  • Conversions to numpy using the numpy() method aren’t available for cuda tensors. One has to go x.cpu().numpy().
    Although this chain is agnostic, it defeats the purpose.

I find that I use methods like .to(device) and .cpu() far too often in my projects.

In my view, it makes the code more verbose than it needs to be and makes it just a little harder to read.

I think that there is room for a global use_gpu flag that can enable developers to run the entire subsequent code in the GPU, where required.

Specifically, my request is the following:

1. Abolish need for the .to(device) suffix:

Circumvent it by letting the developer set the device using a global method like torch.set_default_device() or a convinience method/flag like use_gpu

Then, whenever, an error is encountered because a CUDA tensor is expected in place of a regular tensor or vice-versa, automatically cast the tensor to the expected device.

Additionally,
a. Move nn.Module automatically to the default device.
b. Move the yield of DataLoaders to the default device.

Prevent the need to manually cast to the default device.

2. Add the numpy() method to cuda tensors:

The existing way is to move the tensor to cpu first.
Thus, we have x.cpu().numpy(), which is agnostic but redundant.

3. Use GPU by default if available:

PyTorch is built from the ground up with the Deep Learning community in mind.

With most Deep Learning done on GPUs, they be considered as the default device automatically.

Let PyTorch give first preference to the GPU.
@ssnl
Copy link
Collaborator
ssnl commented May 14, 2018

I am not really a fan of being so clever and managing devices automatically because it makes it less clear which tensors are on which device, and multi-device code harder to read.

That said, I definitely understand your needs and where you come from. I think, at least, we should have options to create Module directly on a given device.

@svaisakh
Copy link
Author

Thanks.

I understand that the kind of control PyTorch gives is unparalleled in this regard.

I think a global setting will be equally useful for quick prototyping.
Multi-GPU scenarios were certainly not my target with this.
unbornchikken reacted with thumbs up emoji

@svaisakh svaisakh mentioned this issue Sep 14, 2018
Closed
@arnaghizadeh
Copy link

Hi, this feature is an absolute requirement, please make it happen.

@juancamilog
Copy link

Actually, I'm in a situation where pytorch 1.0 seems to be using the GPU automatically for all operations (even if I explicitly call .cpu() on the model and data), but I don't want it to use it (because it is actually slower). How can i make sure that everything inside a torch.nn.Module is run on CPU?

@soumith
Copy link
Member
soumith commented Dec 12, 2018

@juancamilog that's not a situation that's possible. There must be a bug in your code, and you are overlooking it.

Sorry, something went wrong.

@juancamilog
Copy link

@soumith You are definitely right, it was a bug in my code.

@parthagar
Copy link
Contributor

Looking forward to it being implemented.

@zou3519 zou3519 added triaged This issue has been looked at a team member, and triaged and prioritized into an appropriate module module: cuda Related to torch.cuda, and CUDA support in general triage review labels Jul 16, 2019
@jeffreyksmithjr jeffreyksmithjr added the feature A request for a proper, new feature. label Jul 18, 2019
@ezyang ezyang mentioned this issue Aug 9, 2019
Closed
@ezyang
Copy link
Contributor
ezyang commented Aug 9, 2019

cc @ali-mahdavifar

@ali-mahdavifar
Copy link

Well... automatically moving all of the Modules into GPU sounds a bit unpleasant to me, but automatically moving tensors to prevent runtime error seems a very intuitive thing to do.
And im not a backend developer, but it sounds simpler than many of the already implemented features, so... has there been any specific problems?

@ezyang ezyang mentioned this issue Aug 13, 2019
Open
@ezyang ezyang added needs research We need to decide whether or not this merits inclusion, based on research world and removed triage review labels Aug 26, 2019
@yaroslavvb
Copy link
Contributor

Could this be solved by having a global default_device setting which places most tensors on GPU, and have the library make the decision intelligently? The issue #24261 mentions that internal library should place some tensors on CPU regardless of setting. Perhaps such Tensors should remain in CPU when constructed by user (scalars maybe?)

This would be analogous to handling of default_dtype which makes an exception for integer valued tensors. For device setting, the users would be aware that non-default value may still keep some of their tensors on CPU, and this special handling will be documented.

@yaroslavvb yaroslavvb mentioned this issue Sep 20, 2019
Closed
@ssnl ssnl mentioned this issue Oct 14, 2019
Closed
@gchanan
Copy link
Contributor
gchanan commented Mar 10, 2020

The current accepted proposal for this is #27878.

@gchanan gchanan removed the needs research We need to decide whether or not this merits inclusion, based on research world label Mar 10, 2020
@vadimkantorov
Copy link
Contributor

I think, this was solved in in #27878 and in https://docs.pytorch.org/docs/stable/generated/torch.set_default_device.html

Should this issue be closed?

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
feature A request for a proper, new feature. module: cuda Related to torch.cuda, and CUDA support in general triaged This issue has been looked at a team member, and triaged and prioritized into an appropriate module
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
15 participants
@ezyang @yaroslavvb @vadimkantorov @soumith @cpuhrsch @juancamilog @jeffreyksmithjr @gchanan @arnaghizadeh @zou3519 @ssnl @ali-mahdavifar @svaisakh @parthagar and others
0