[wip][ca][ddp] traceable C++ reducer #153501

xmfan · 2025-05-13T23:27:55Z

Stack from ghstack (oldest at bottom):

cc @H-Huang @awgu @wanchaol @fegin @fduwjj @wz337 @wconstab @d4l3k @voznesenskym @penguinwu @EikanWang @jgong5 @Guobing-Chen @XiaobingSuper @zhuhaozhe @blzheng @wenzhe-nrv @jiayisunx @chenyang78 @kadeng @chauhang @amjames

[ghstack-poisoned]

pytorch-bot · 2025-05-13T23:27:58Z

🔗 Helpful Links

🧪 See artifacts and rendered test results at hud.pytorch.org/pr/153501

Python docs built from this PR

📄 Preview C++ docs built from this PR
❓ Need help or want to give feedback on the CI? Visit the bot commands wiki or our office hours

Note: Links to docs will display an error until the docs builds have been completed.

❌ 23 New Failures, 1 Unrelated Failure

As of commit b4f550f with merge base 20ba8fe ():

NEW FAILURES - The following jobs have failed:

inductor / unit-test / cuda12.6-py3.10-gcc9-sm86 / test (inductor_distributed, 1, 1, linux.g5.12xlarge.nvidia.gpu) (gh)
distributed/tensor/test_dtensor_compile.py::TestDTensorCompileE2E::test_2d_fsdp_tp_ac_compile_use_ca_True
inductor-rocm / rocm-py3.10-inductor / test (inductor, 1, 2, linux.rocm.gpu.2) (gh)
distributed/tensor/test_dtensor_compile.py::TestDTensorCompileE2E::test_2d_fsdp_tp_ac_compile_use_ca_True
Lint / lintrunner-clang / linux-job (gh)
>>> Lint for torch/csrc/dynamo/python_compiled_autograd.cpp:
pull / linux-focal-cuda11.8-py3.10-gcc9 / test (distributed, 1, 3, lf.linux.g4dn.12xlarge.nvidia.gpu) (gh)
distributed/test_c10d_nccl.py::DistributedDataParallelTest::test_accumulate_gradients_module
pull / linux-focal-cuda11.8-py3.10-gcc9 / test (distributed, 2, 3, lf.linux.g4dn.12xlarge.nvidia.gpu) (gh)
distributed/test_c10d_nccl.py::DistributedDataParallelTest::test_ddp_checkpointing_dynamic_module
pull / linux-focal-cuda11.8-py3.10-gcc9 / test (distributed, 3, 3, lf.linux.g4dn.12xlarge.nvidia.gpu) (gh)
distributed/test_c10d_nccl.py::DistributedDataParallelTest::test_dataclass_output_unused_param
pull / linux-focal-cuda12.6-py3.10-gcc11 / test (default, 2, 5, lf.linux.4xlarge.nvidia.gpu) (gh)
'test/inductor/test_compiled_autograd.py::TestCompiledAutograd::test_disk_offloading'
pull / linux-focal-cuda12.6-py3.10-gcc11 / test (default, 3, 5, lf.linux.4xlarge.nvidia.gpu) (gh)
'test/inductor/test_compiled_autograd.py::TestCompiledAutograd::test_checkpointing_sac'
pull / linux-focal-cuda12.6-py3.10-gcc11 / test (default, 4, 5, lf.linux.4xlarge.nvidia.gpu) (gh)
'test/inductor/test_compiled_autograd.py::TestCompiledAutograd::test_backward_hook_relative_ordering_partial'
pull / linux-focal-cuda12.6-py3.10-gcc11-sm89 / test (default, 2, 5, lf.linux.g6.4xlarge.experimental.nvidia.gpu) (gh)
'test/inductor/test_compiled_autograd.py::TestCompiledAutograd::test_backward_hook_relative_ordering_partial'
pull / linux-focal-cuda12.6-py3.10-gcc11-sm89 / test (default, 3, 5, lf.linux.g6.4xlarge.experimental.nvidia.gpu) (gh)
'test/inductor/test_compiled_autograd.py::TestCompiledAutograd::test_reorder_all_bwd_hooks'
pull / linux-focal-py3.13-clang10 / test (default, 2, 5, lf.linux.4xlarge) (gh)
'test/inductor/test_compiled_autograd.py::TestCompiledAutograd::test_backward_hook_relative_ordering_partial'
pull / linux-focal-py3.13-clang10 / test (default, 3, 5, lf.linux.4xlarge) (gh)
'test/inductor/test_compiled_autograd.py::TestCompiledAutograd::test_reorder_all_bwd_hooks'
pull / linux-focal-py3.9-clang10 / test (default, 2, 5, lf.linux.4xlarge) (gh)
'test/inductor/test_compiled_autograd.py::TestCompiledAutograd::test_backward_hook_relative_ordering_partial'
pull / linux-focal-py3.9-clang10 / test (default, 3, 5, lf.linux.4xlarge) (gh)
'test/inductor/test_compiled_autograd.py::TestCompiledAutograd::test_reorder_all_bwd_hooks'
pull / linux-jammy-py3.10-clang15-asan / test (default, 2, 6, lf.linux.4xlarge) (gh)
'test/inductor/test_compiled_autograd.py::TestCompiledAutograd::test_ddp_cpp_reducer_error'
pull / linux-jammy-py3.10-clang15-asan / test (default, 3, 6, lf.linux.4xlarge) (gh)
'test/inductor/test_compiled_autograd.py::TestCompiledAutograd::test_reorder_all_bwd_hooks'
pull / linux-jammy-py3.10-clang15-asan / test (default, 4, 6, lf.linux.4xlarge) (gh)
'test/inductor/test_compiled_autograd.py::TestCompiledAutograd::test_backward_hook_relative_ordering_partial'
pull / linux-jammy-py3.10-clang15-asan / test (default, 5, 6, lf.linux.4xlarge) (gh)
'test/inductor/test_compiled_autograd.py::TestCompiledAutograd::test_reorder_post_hook1'
pull / linux-jammy-py3.9-gcc11 / test (default, 2, 5, lf.linux.2xlarge) (gh)
'test/inductor/test_compiled_autograd.py::TestCompiledAutograd::test_disk_offloading'
pull / linux-jammy-py3.9-gcc11 / test (default, 3, 5, lf.linux.2xlarge) (gh)
'test/inductor/test_compiled_autograd.py::TestCompiledAutograd::test_checkpointing_sac'
pull / linux-jammy-py3.9-gcc11 / test (default, 4, 5, lf.linux.2xlarge) (gh)
'test/inductor/test_compiled_autograd.py::TestCompiledAutograd::test_backward_hook_relative_ordering_partial'
pull / win-vs2022-cpu-py3 / build (gh)
C:\actions-runner\_work\pytorch\pytorch\torch\csrc\distributed\c10d\reducer.cpp(226): error C2065: 'rpc_context_': undeclared identifier

FLAKY - The following job failed but was likely due to flakiness present on trunk:

inductor / unit-test / cuda12.6-py3.10-gcc9-sm86 / test (inductor_cpp_wrapper, 1, 2, linux.g5.4xlarge.nvidia.gpu) (gh) (similar failure)
[ FAILED ] AotInductorTest.FreeInactiveConstantBufferCuda

This comment was automatically generated by Dr. CI and updates every 15 minutes.

ghstack-source-id: c4086d2 Pull Request resolved: #153501

[ghstack-poisoned]

ghstack-source-id: 249ce8d Pull Request resolved: #153501

[ghstack-poisoned]

ghstack-source-id: 1d3c565 Pull Request resolved: #153501

[ghstack-poisoned]

ghstack-source-id: d9b4ff7 Pull Request resolved: #153501

Update

e378aaa

[ghstack-poisoned]

This was referenced May 13, 2025

[ca][dynamo] always run eager checkpoint region's recomputation in eager #153300

Closed

[ca][dtensor] run real PG dtensor tests under CA #152689

Closed

xmfan mentioned this pull request May 13, 2025

[ddp] propagate use_python_reducer to C++ reducer #152735

Closed

pytorch-bot bot added ciflow/inductor module: compiled autograd compiled_autograd module: dynamo oncall: distributed Add this issue/PR to distributed oncall triage queue release notes: distributed (c10d) release notes category labels May 13, 2025

xmfan added a commit that referenced this pull request May 13, 2025

[wip][ca][ddp] traceable C++ reducer

5b669f2

ghstack-source-id: c4086d2 Pull Request resolved: #153501

Update

e75b046

[ghstack-poisoned]

xmfan added a commit that referenced this pull request May 14, 2025

[wip][ca][ddp] traceable C++ reducer

156d7de

ghstack-source-id: 249ce8d Pull Request resolved: #153501

Update

196ea58

[ghstack-poisoned]

xmfan added a commit that referenced this pull request May 14, 2025

[wip][ca][ddp] traceable C++ reducer

4bf7e54

ghstack-source-id: 1d3c565 Pull Request resolved: #153501

Update

b4f550f

[ghstack-poisoned]

xmfan added a commit that referenced this pull request May 15, 2025

[wip][ca][ddp] traceable C++ reducer

fc93022

ghstack-source-id: d9b4ff7 Pull Request resolved: #153501

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Uh oh!

[wip][ca][ddp] traceable C++ reducer #153501

[wip][ca][ddp] traceable C++ reducer #153501

Uh oh!

Uh oh!

Uh oh!

[wip][ca][ddp] traceable C++ reducer #153501

Are you sure you want to change the base?

[wip][ca][ddp] traceable C++ reducer #153501

Conversation

Uh oh!

Uh oh!

Uh oh!

🔗 Helpful Links

🧪 See artifacts and rendered test results at hud.pytorch.org/pr/153501

❌ 23 New Failures, 1 Unrelated Failure

Uh oh!

Uh oh!