8000 [not for land] small compile-on-one-rank example by bdhirsh · Pull Request #153743 · pytorch/pytorch · GitHub
[go: up one dir, main page]
Skip to content

[not for land] small compile-on-one-rank example #153743

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom
Open
wants to merge 1 commit into
base: gh/bdhirsh/663/base
Choose a base branch
from
Open

[not for land] small compile-on-one-rank example #153743

wants to merge 1 commit into from

Conversation

bdhirsh
Copy link
Contributor
@bdhirsh bdhirsh commented May 16, 2025 •
edited
Loading

This is just for stashing changes (it is full of hacks and is missing features). See precompile_on_one_symbolic_rank_example.py for the example code, which splits the job into two steps (pre-compile on a single rank, run distributed job with that shared artifact)

Linking the generated output code in a bit, where the main feature you can see is that:
(1) rank is an input to the compiled artifact
(2) the triton kernels hardcode torch.distributed.get_rank() instead of the rank they were compiled with (to be fair this is not actually what we want, since it implies that each rank needs to compile the triton kernels separately. but at the very least the same compiled artifact python file can be shared by every rank)

Stack from ghstack (oldest at bottom):

cc @H-Huang @awgu @wanchaol @fegin @fduwjj @wz337 @wconstab @d4l3k @ezyang @SherlockNoMad @EikanWang @jgong5 @wenzhe-nrv @voznesenskym @penguinwu @Guobing-Chen @XiaobingSuper @zhuhaozhe @blzheng @jiayisunx @ipiszy @chenyang78 @kadeng @muchulee8 @amjames @chauhang @aakhundov

@pytorch-bot PyTorch Bot
Copy link
pytorch-bot bot commented May 16, 2025 •
edited
Loading

🔗 Helpful Links

🧪 See artifacts and rendered test results at hud.pytorch.org/pr/153743

Note: Links to docs will display an error until the docs builds have been completed.

❗ 1 Active SEVs

There are 1 currently active SEVs. If your PR is affected, please view them below:

❌ 70 New Failures, 4 Cancelled Jobs, 2 Unrelated Failures

As of commit eec15c9 with merge base b32b002 (image):

NEW FAILURES - The following jobs have failed:

CANCELLED JOBS - The following jobs were cancelled. Please retry:

UNSTABLE - The following jobs are marked as unstable, possibly due to flakiness on trunk:

This comment was automatically generated by Dr. CI and updates every 15 minutes.

bdhirsh added a commit that referenced this pull request May 16, 2025
@bdhirsh
[not for land] small compile-on-one-rank example
c3bbd95 
ghstack-source-id: 5e21251
Pull Request resolved: #153743
@pytorch-bot pytorch-bot bot added ciflow/inductor module: dynamo module: inductor oncall: distributed Add this issue/PR to distributed oncall triage queue release notes: distributed (c10d) release notes category labels May 16, 2025
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers
No reviews
Assignees
No one assigned
Labels
ciflow/inductor fx module: dynamo module: inductor oncall: distributed Add this issue/PR to distributed oncall triage queue release notes: distributed (c10d) release notes category
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.
2 participants
@bdhirsh @facebook-github-bot
0