8000 [c10d] PGNCCL refactor part 2: Simplify ProcessGroupNCCL into single-device style by kwen2501 · Pull Request #119421 · pytorch/pytorch · GitHub
[go: up one dir, main page]
Skip to content

[c10d] PGNCCL refactor part 2: Simplify ProcessGroupNCCL into single-device style #119421

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Closed
wants to merge 14 commits into from
Closed

[c10d] PGNCCL refactor part 2: Simplify ProcessGroupNCCL into single-device style #119421

Changes from 1 commit
Commits
Show all changes
14 commits
Select commit Hold shift + click to select a range
79ec54c
Simplify ProcessGroupNCCL into single-device style
kwen2501 Jan 30, 2024
6c4e05c
Refactor overriden functions in tests
kwen2501 Jan 30, 2024
c16ec89
Fix segv
kwen2501 Feb 1, 2024
8bbaf7c
Correctly print device
kwen2501 Feb 2, 2024
36ce680
Fix NcclErrorHandlingTest
kwen2501 Feb 2, 2024
c25468d
Add logging in barrier wait
kwen2501 Feb 2, 2024
98fb9b6
Fix AG and RS recording
kwen2501 Feb 2, 2024
9884725
Make flight recorder happy
kwen2501 Feb 2, 2024
edf9cbb
Bump seq number once only for batch p2p
kwen2501 Feb 2, 2024
8b249cd
Lint
kwen2501 Feb 2, 2024
242b2a5
Re-enable flatten path of AG and RS
kwen2501 Feb 5, 2024
4de6892
Lint
kwen2501 Feb 6, 2024
45f0b15
Swap check_gpu_tensors_different_devices with check_gpu_single_tensor
kwen2501 Feb 8, 2024
9b72df7
Generalize error message to make both CI and devGPU happy
kwen2501 Feb 12, 2024
File filter

Filter by extension

Filter by extension 8000
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
Prev Previous commit
Next Next commit
Lint
  • Loading branch information
@kwen2501
kwen2501 committed Feb 8, 2024
commit 4de6892651737a8450c07026c237d8dfc072ad0b
4 changes: 2 additions & 2 deletions torch/csrc/distributed/c10d/ProcessGroupNCCL.cpp
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -3254,8 +3254,8 @@ c10::intrusive_ptr<Work> ProcessGroupNCCL::reduce_scatter(
output.storage().data_ptr(), stream);
}
const auto ncclDataType = getNcclDataType(input.scalar_type());
const auto ncclReduceOp = getNcclReduceOp(
opts.reduceOp, input, ncclDataType, comm);
const auto ncclReduceOp =
getNcclReduceOp(opts.reduceOp, input, ncclDataType, comm);
return ncclReduceScatter(
input.data_ptr(),
output.data_ptr(),
Expand Down
0