8000 [c10d] Simplify ProcessGroupNCCL into single-device style by kwen2501 · Pull Request #118674 · pytorch/pytorch · GitHub
[go: up one dir, main page]
Skip to content

[c10d] Simplify ProcessGroupNCCL into single-device style #118674

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Closed
wants to merge 16 commits into from
Closed

[c10d] Simplify ProcessGroupNCCL into single-device style #118674

Changes from 1 commit
Commits
Show all changes
16 commits
Select commit Hold shift + click to select a range
e8bbc7b
Simplify ProcessGroupNCCL into single-device style
kwen2501 Jan 30, 2024
8338feb
Refactor overriden functions in tests
kwen2501 Jan 30, 2024
a923440
Fix segv
kwen2501 Feb 1, 2024
18c1f5e
Fix test_gather_checks
kwen2501 Feb 1, 2024
87b14fb
Correctly print device
kwen2501 Feb 2, 2024
204253c
Refactor ProcessGroupNCCLTest from multi-device to multi-thread style
kwen2501 Feb 2, 2024
f13b03b
Refactor ProcessGroupNCCLErrorsTest
kwen2501 Feb 2, 2024
efe2ace
Fix NcclErrorHandlingTest
kwen2501 Feb 2, 2024
a4fb6fd
Add logging in barrier wait
kwen2501 Feb 2, 2024
dff9b73
Fix AG and RS recording
kwen2501 Feb 2, 2024
87816df
Make flight recorder happy
kwen2501 Feb 2, 2024
bb435ee
Bump seq number once only for batch p2p
kwen2501 Feb 2, 2024
bd4a7f0
Lint
kwen2501 Feb 2, 2024
32704d4
Improve error message
kwen2501 Feb 2, 2024
c144c1b
Fix unused-variable warning
kwen2501 Feb 2, 2024
db92422
Re-enable flatten path of AG and RS
kwen2501 Feb 5, 2024
8000
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
Prev Previous commit
Next Next commit
Fix test_gather_checks
  • Loading branch information
@kwen2501
kwen2501 committed Feb 1, 2024
commit 18c1f5e0bedfeaea8283ffb4e1cd8799e137e4e2
35 changes: 8 additions & 27 deletions test/distributed/test_c10d_nccl.py
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -718,55 +718,36 @@ def gather(output_t, input_t, rootRank):
def test_gather_checks(self):
store = c10d.FileStore(self.file_name, self.world_size)
pg = self._create_process_group_nccl(store, self.opts())
local_device_ids = self.rank_to_GPU[self.rank]
num_gpus = len(local_device_ids)
device_id = self.rank_to_GPU[self.rank][0]

# init input
tensors = []
for device_id in local_device_ids:
tensors.append(torch.tensor([self.rank]).cuda(device_id))
tensor = torch.tensor([self.rank]).cuda(device_id)

# init output
output_ts = []
for idx in range(num_gpus):
gpu_idx = local_device_ids[idx]
output_ts.append([])
for rank in range(self.world_size):
output_ts[idx].append(torch.tensor([-1]).cuda(gpu_idx))
for rank in range(self.world_size):
output_ts.append(torch.tensor([-1]).cuda(device_id))

with self.assertRaisesRegex(ValueError, "invalid root rank"):
opts = c10d.GatherOptions()
opts.rootRank = -1
pg.gather(output_ts, tensors, opts)
pg.gather([output_ts], [tensor], opts)

with self.assertRaisesRegex(TypeError, "incompatible function arguments"):
pg.gather(output_ts, tensors, 0)
pg.gather([output_ts], [tensor], 0)

with self.assertRaisesRegex(ValueError, "invalid root rank"):
opts = c10d.GatherOptions()
opts.rootRank = self.world_size
pg.gather(output_ts, tensors, opts)
pg.gather([output_ts], [tensor], opts)

with self.assertRaisesRegex(
# throws error message from dispatcher
RuntimeError, "There were no tensor arguments to this function"
):
opts = c10d.GatherOptions()
opts.rootRank = 0
pg.gather(output_ts, [], opts)

with self.assertRaisesRegex(
ValueError, "Tensors must be on distinct GPU devices"
):
# init input
tensors2 = []
for device_id in local_device_ids:
tensors2.append(torch.tensor([self.rank]).cuda(device_id))
tensors2.append(torch.tensor([self.rank]).cuda(device_id))

opts = c10d.GatherOptions()
opts.rootRank = 0
pg.gather(output_ts, tensors2, opts)
pg.gather([output_ts], [], opts)

@requires_nccl()
@skip_but_pass_in_sandcastle_if(not TEST_MULTIGPU, "NCCL test requires 2+ GPUs")
Expand Down
0