8000 [dynamo] Module tracker test errors in dynamo at runtime · Issue #127112 · pytorch/pytorch · GitHub
[go: up one dir, main page]
Skip to content

[dynamo] Module tracker test errors in dynamo at runtime #127112

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom
Open
williamwen42 opened this issue May 24, 2024 · 1 comment
Open

[dynamo] Module tracker test errors in dynamo at runtime #127112

williamwen42 opened this issue May 24, 2024 · 1 comment
Labels
module: dynamo oncall: pt2 triaged This issue has been looked at a team member, and triaged and prioritized into an appropriate module

Comments

@williamwen42
Copy link
Member
williamwen42 commented May 24, 2024 •
edited by pytorch-bot bot
Loading

Discovered with #126341.

Repro command: PYTORCH_TEST_WITH_DYNAMO=1 pytest test/test_module_tracker.py::TestModuleTracker::test_module_hierarchy

Logs:

Traceback (most recent call last):
  File "/data/users/williamwen/pytorch2/torch/_dynamo/backends/debugging.py", line 33, in inner
    return gm(*args)
  File "/data/users/williamwen/pytorch2/torch/nn/modules/module.py", line 1532, in _wrapped_call_impl
    return self._call_impl(*args, **kwargs)
  File "/data/users/williamwen/pytorch2/torch/nn/modules/module.py", line 1595, in _call_impl
    hook_result = hook(self, args, result)
  File "/data/users/williamwen/pytorch2/torch/utils/module_tracker.py", line 125, in _fw_post_hook
    self._get_pop_fn(name, False)()
  File "/data/users/williamwen/pytorch2/torch/utils/module_tracker.py", line 108, in fn
    raise RuntimeError(
RuntimeError: The Module hierarchy tracking is wrong. Report a bug to PyTorch

The above exception was the direct cause of the following exception:

Traceback (most recent call last):
  File "/data/users/williamwen/py310-env/lib/python3.10/unittest/case.py", line 59, in testPartExecutor
    yield
  File "/data/users/williamwen/py310-env/lib/python3.10/unittest/case.py", line 591, in run
    self._callTestMethod(testMethod)
  File "/data/users/williamwen/py310-env/lib/python3.10/unittest/case.py", line 549, in _callTestMethod
    method()
  File "/data/users/williamwen/pytorch2/torch/testing/_internal/common_utils.py", line 2756, in wrapper
    method(*args, **kwargs)
  File "/data/users/williamwen/pytorch2/test/test_module_tracker.py", line 38, in test_module_hierarchy
    mod({"a": torch.randn(10, 10, requires_grad=True).clone()})[
  File "/data/users/williamwen/pytorch2/torch/nn/modules/module.py", line 1532, in _wrapped_call_impl
    return self._call_impl(*args, **kwargs)
  File "/data/users/williamwen/pytorch2/torch/nn/modules/module.py", line 1582, in _call_impl
    result = forward_call(*args, **kwargs)
  File "/data/users/williamwen/pytorch2/test/test_module_tracker.py", line 32, in forward
    x = self.c[0](x)
  File "/data/users/williamwen/pytorch2/torch/nn/modules/module.py", line 1532, in _wrapped_call_impl
    return self._call_impl(*args, **kwargs)
  File "/data/users/williamwen/pytorch2/torch/nn/modules/module.py", line 1582, in _call_impl
    result = forward_call(*args, **kwargs)
  File "/data/users/williamwen/pytorch2/test/test_module_tracker.py", line 16, in forward
    def forward(self, x):
  File "/data/users/williamwen/pytorch2/torch/_dynamo/eval_frame.py", line 548, in _fn
    return fn(*args, **kwargs)
  File "/data/users/williamwen/pytorch2/torch/_dynamo/backends/debugging.py", line 36, in inner
    raise torch._dynamo.exc.TorchDynamoException(
torch._dynamo.exc.TorchDynamoException: Unexpected exception when running generated GraphModule

cc @ezyang @msaroufim @bdhirsh @anijain2305 @chauhang @voznesenskym @penguinwu @EikanWang @jgong5 @Guobing-Chen @XiaobingSuper @zhuhaozhe @blzheng @wenzhe-nrv @jiayisunx @chenyang78 @kadeng @albanD
@williamwen42 williamwen42 added triaged This issue has been looked at a team member, and triaged and prioritized into an appropriate module oncall: pt2 module: dynamo labels May 24, 2024
@albanD
Copy link
Collaborator
albanD commented May 24, 2024 •
edited
Loading

I guess the hooks being used there get dropped in Dynamo?
Leading to the tracking to fail

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
module: dynamo oncall: pt2 triaged This issue has been looked at a team member, and triaged and prioritized into an appropriate module
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
2 participants
@albanD @williamwen42
0