8000 Bf16 fused adam(W) by janeyx99 · Pull Request #147653 · pytorch/pytorch · GitHub
[go: up one dir, main page]
Skip to content

Bf16 fused adam(W) #147653

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom
Draft
wants to merge 9 commits into
base: gh/janeyx99/225/base
Choose a base branch
from
Draft

Bf16 fused adam(W) #147653

wants to merge 9 commits into from

Conversation

janeyx99
Copy link
Contributor
@janeyx99 janeyx99 commented Feb 22, 2025 •
edited
Loading

Many things do work!

Some things do not:
amsgrad does not work
some checks are removed (not critical imo, but def less safe) to let multidevice work
adam + adamw should work
tensor lr would work if it's on cpu
the float, float, bf16, bf16 pattern is specialized, other mixed precision will not work

Stack from ghstack (oldest at bottom):

@janeyx99 janeyx99 mentioned this pull request Feb 22, 2025
Draft
@pytorch-bot PyTorch Bot
Copy link
pytorch-bot bot commented Feb 22, 2025 •
edited
Loading

🔗 Helpful Links

🧪 See artifacts and rendered test results at hud.pytorch.org/pr/147653

Note: Links to docs will display an error until the docs builds have been completed.

❌ 12 New Failures, 1 Cancelled Job

As of commit 6a74269 with merge base 83bb921 (image):

NEW FAILURES - The following jobs have failed:

CANCELLED JOB - The following job was cancelled. Please retry:

This comment was automatically generated by Dr. CI and updates every 15 minutes.

@pytorch-bot pytorch-bot bot added the release notes: foreach_frontend release notes category label Feb 22, 2025
janeyx99 added a commit that referenced this pull request Feb 22, 2025
@janeyx99
[skip ci] Attempt a mixed precision fused adam
fcf21a8 
ghstack-source-id: eab9384
Pull Request resolved: #147653
janeyx99 added a commit that referenced this pull request Feb 22, 2025
@janeyx99
[skip ci] Attempt a mixed precision fused adam
0d094cb 
ghstack-source-id: 9adfe54
Pull Request resolved: #147653
janeyx99 added a commit that referenced this pull request Feb 22, 2025
@janeyx99
[skip ci] Attempt a mixed precision fused adam
0301945 
ghstack-source-id: 18ed96b
Pull Request resolved: #147653
janeyx99 added a commit that referenced this pull request Feb 24, 2025
@janeyx99
[skip ci] Attempt a mixed precision fused adam
bce08b7 
ghstack-source-id: 5b6560e
Pull Request resolved: #147653
janeyx99 added a commit that referenced this pull request Feb 25, 2025
@janeyx99
[skip ci] Attempt a mixed precision fused adam
2899fe1 
ghstack-source-id: 5e557ae
Pull Request resolved: #147653
@janeyx99 janeyx99 changed the title [skip ci] Attempt a mixed precision fused adam Attempt a mixed precision fused adam Feb 25, 2025
janeyx99 added a commit that referenced this pull request Feb 25, 2025
@janeyx99
Attempt a mixed precision fused adam
da5b584 
ghstack-source-id: 63ec4a7
Pull Request resolved: #147653
janeyx99 added a commit that referenced this pull request Feb 27, 2025
@janeyx99
Attempt a mixed precision fused adam
45daa94 
ghstack-source-id: 71c4ddb
Pull Request resolved: #147653
janeyx99 added a commit that referenced this pull request Feb 27, 2025
@janeyx99
Attempt a mixed precision fused adam
fa90587 
ghstack-source-id: 7a14380
Pull Request resolved: #147653
janeyx99 added a commit that referenced this pull request Feb 27, 2025
@janeyx99
Attempt a mixed precision fused adam 8000
1e467d1 
ghstack-source-id: 94c6a34
Pull Request resolved: #147653
@janeyx99 janeyx99 changed the title Attempt a mixed precision fused adam Bf16 fused adam(W) Mar 12, 2025
This was referenced Apr 28, 2025
Open
Draft
@github-actions GitHub Actions
Copy link
Contributor

Looks like this PR hasn't been updated in a while so we're going to go ahead and mark this as Stale.
Feel free to remove the Stale label if you feel this was a mistake.
If you are unable to remove the Stale label please contact a maintainer in order to do so.
If you want the bot to never mark this PR stale again, add the no-stale label.
Stale pull requests will automatically be closed after 30 days of inactivity.

@github-actions github-actions bot added the Stale label May 11, 2025
@janeyx99 janeyx99 removed the Stale label May 12, 2025
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers

@eqy eqy Awaiting requested review from eqy eqy will be requested when the pull request is marked ready for review eqy is a code owner

@syed-ahmed syed-ahmed Awaiting requested review from syed-ahmed syed-ahmed will be requested when the pull request is marked ready for review syed-ahmed is a code owner

@albanD albanD Awaiting requested review from albanD albanD will be requested when the pull request is marked ready for review albanD is a code owner

Assignees
No one assigned
Labels
no-stale release notes: foreach_frontend release notes category
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.
1 participant
@janeyx99
0