8000 Functions in torch._C._nn and torch._C._onnx are not pickleable · Issue #38137 · pytorch/pytorch · GitHub
[go: up one dir, main page]
Skip to content

Functions in torch._C._nn and torch._C._onnx are not pickleable #38137

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom
Closed
ezyang opened this issue May 8, 2020 · 4 comments
Closed

Functions in torch._C._nn and torch._C._onnx are not pickleable #38137

ezyang opened this issue May 8, 2020 · 4 comments
Assignees
@mattip
Labels
high priority module: bootcamp We plan to do a full writeup on the issue, and then get someone to do it for onboarding module: pickle Problems related to pickling of PyTorch objects triaged This issue has been looked at a team member, and triaged and prioritized into an appropriate module

Comments

@ezyang
Copy link
Contributor
ezyang commented May 8, 2020 •
edited by pytorch-probot bot
Loading

This is due to the same bug fixed in #38136 We need to apply the same fix in the other situations too

cc @ezyang @gchanan @zou3519 @bdhirsh
@ailzhang ailzhang added module: pickle Problems related to pickling of PyTorch objects triaged This issue has been looked at a team member, and triaged and prioritized into an appropriate module labels May 12, 2020
@albertcthomas
Copy link

gpytorch models cannot be pickled because of this, see related issue cornellius-gp/gpytorch#907. A temporary fix is to use dill or redefine needed functions if possible, see this comment. However maybe this can be fixed in pytorch as you suggest @ezyang?

@albertcthomas albertcthomas mentioned this issue Nov 8, 2020
Open
@albertcthomas
Copy link

By the way I might be missing something but with pytorch 1.7 if I do a import torch._C._VariableFunctions I get a ModuleNotFoundError: No module named 'torch._C._VariableFunctions'; 'torch._C' is not a package

@ezyang ezyang added high priority module: bootcamp We plan to do a full writeup on the issue, and then get someone to do it for onboarding labels Nov 9, 2020
@ezyang
Copy link
Contributor Author
ezyang commented Nov 9, 2020

Yes it should be fixable by PyTorch. _VariableFunctions is not a module, I think it's a faux class (but I could be wrong).

@mattip mattip self-assigned this Mar 1, 2021
@mattip
Copy link
Collaborator
mattip commented Mar 2, 2021

It seems this is a shortcoming in python3. The meaning of torch._C._onnx is that there really is a directory _C under the directory torch, and inside that directory there is a module _onnx.

_onnx is created via a call to pybind11's def_submodule, and a discussion of use of def_submodule points out the shortcomings. Some of the other submodules, like _nn, are created with PyModule_Create. Refactoring _C to break out these submodules would be one solution. The easier thing to do is to push all these submodules into sys.modules as was done in gh-38136:

for x in dir(torch._C):
    c = getattr(torch._C, x)
    if type(c) is type(torch):
        sys.modules[f'torch._C.{x}'] = c

I also opened an issue for CPython

This was referenced Mar 2, 2021
Closed
Closed
aocsa pushed a commit to Quansight/pytorch that referenced this issue Mar 15, 2021
@mattip @aocsa
add submodules to sys.modules so their attributes can be pickled (pyt…
c7e013a 
…orch#53107)

Summary:
Fixes pytorch#38137

As mentioned in the issue, this is a workaround for [python issue 43367](https://bugs.python.org/issue43367). There are a number of other places where `sys.modules` is modified, if something changes in python perhaps those should be reviewed as well.

Pull Request resolved: pytorch#53107

Reviewed By: zou3519

Differential Revision: D26753571

Pulled By: ezyang

fbshipit-source-id: 2bda03bab39ff9ca58ce4bc13befe021da91b9c4
xsacha pushed a commit to xsacha/pytorch that referenced this issue Mar 31, 2021
@mattip
add submodules to sys.modules so their attributes can be pickled (pyt…
ba0d15d 
…orch#53107)

Summary:
Fixes pytorch#38137

As mentioned in the issue, this is a workaround for [python issue 43367](https://bugs.python.org/issue43367). There are a number of other places where `sys.modules` is modified, if something changes in python perhaps those should be reviewed as well.

Pull Request resolved: pytorch#53107

Reviewed By: zou3519

Differential Revision: D26753571

Pulled By: ezyang

fbshipit-source-id: 2bda03bab39ff9ca58ce4bc13befe021da91b9c4
This was referenced May 16, 2024
Open
Open
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees

@mattip mattip

Labels
high priority module: bootcamp We plan to do a full writeup on the issue, and then get someone to do it for onboarding module: pickle Problems related to pickling of PyTorch objects triaged This issue has been looked at a team member, and triaged and prioritized into an appropriate module
Projects
None yet
Milestone
No milestone
Development

Successfully merging a pull request may close this issue.

add submodules to sys.modules so their attributes can be pickled mattip/pytorch
5 participants
@ezyang @mattip @ailzhang @albertcthomas @bdhirsh
0