8000 [inductor] enable bf32 for mkldnn linear pointwise/binary in inductor by zhuhaozhe · Pull Request #127294 · pytorch/pytorch · GitHub
[go: up one dir, main page]
Skip to content

[inductor] enable bf32 for mkldnn linear pointwise/binary in inductor #127294

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom
Closed
wants to merge 72 commits into from
Closed

[inductor] enable bf32 for mkldnn linear pointwise/binary in inductor #127294

wants to merge 72 commits into from

Conversation

@zhuhaozhe
Copy link
Collaborator
@zhuhaozhe zhuhaozhe commented May 28, 2024 •
edited by yanbing-j
Loading

When torch.backends.mkldnn.matmul.fp32_precision=='bf16', we also enabled mkldnn linear in inductor path and allow to run with bf16 computation data type.

Testplan:

python test/inductor/test_mkldnn_pattern_matcher.py -k test_linear_unary
python test/inductor/test_mkldnn_pattern_matcher.py -k test_linear_fp32
python test/inductor/test_mkldnn_pattern_matcher.py -k test_multi_linear_share_same_input

Stack from ghstack (oldest at bottom):

cc @jgong5 @mingfeima @XiaobingSuper @sanchitintel @ashokei @jingxu10 @jerryzh168 @voznesenskym @penguinwu @EikanWang @Guobing-Chen @blzheng @wenzhe-nrv @jiayisunx @ipiszy @chenyang78 @kadeng @muchulee8 @amjames @chauhang @aakhundov @peterbell10 @yf225 @ColinPeppler @desertfire

@pytorch-bot
Copy link
pytorch-bot bot commented May 28, 2024 •
edited
Loading

🔗 Helpful Links

🧪 See artifacts and rendered test results at hud.pytorch.org/pr/127294

Note: Links to docs will display an error until the docs builds have been completed.

✅ You can merge normally! (7 Unrelated Failures)

As of commit 63915d3 with merge base 2022588 (image):

FLAKY - The following job failed but was likely due to flakiness present on trunk:

BROKEN TRUNK - The following jobs failed but was present on the merge base:

👉 Rebase onto the `viable/strict` branch to avoid these failures

This comment was automatically generated by Dr. CI and updates every 15 minutes.

@zhuhaozhe zhuhaozhe mentioned this pull request May 28, 2024
Closed
@pytorch-bot pytorch-bot bot added ciflow/inductor module: cpu CPU specific problem (e.g., perf, algorithm) module: inductor labels May 28, 2024
This was referenced May 28, 2024
Closed
Closed
Closed
zhuhaozhe added a commit that referenced this pull request May 28, 2024
@zhuhaozhe
[inductor] enable bf32 for mkldnn linear pointwise/binary in inductor
ab9b956 
ghstack-source-id: 3582e56
Pull Request resolved: #127294
@zhuhaozhe
Update
e6d2625 
[ghstack-poisoned]
@zhuhaozhe zhuhaozhe marked this pull request as draft May 28, 2024 14:01
@zhuhaozhe zhuhaozhe requested a review from jgong5 May 28, 2024 14:15
jgong5
jgong5 reviewed May 29, 2024
View reviewed changes
torch/_inductor/fx_passes/mkldnn_fusion.py Outdated
Comment on lines 786 to 790
if not hasattr(add_node.args[1], "meta"):
# May add an "int"
# We meet this issue while enabling bf32 for test_linear_unary
# the hardsimoid case will add "3" here and cannot check meta
# TODO: Further investigate on this issue
Copy link
Collaborator
@jgong5 jgong5 May 29, 2024

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Is it for the bias node or something else? Can you double check? Why didn't we get the problem in the past?

Copy link
Collaborator Author
@zhuhaozhe zhuhaozhe May 31, 2024

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

It should be another input node for "add".
We did not meet this in the past because we only enabled and tested lp for mkldnn fusion. For lp case, there will be a "to_dtype" inserted by autocast and will not match this pattern.

I am trying to find a case to satisfy this pattern by raising a runtime error if this function will return True.
#127597

Copy link
Collaborator Author
@zhuhaozhe zhuhaozhe Jun 5, 2024

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Fixed in #127597

zhuhaozhe added a commit that referenced this pull request May 30, 2024
@zhuhaozhe
[inductor] enable bf32 for mkldnn linear pointwise/binary in inductor
db42984 
ghstack-source-id: 371129b
Pull Request resolved: #127294
@zhuhaozhe
Update
a197dd0 
[ghstack-poisoned]
@zhuhaozhe
Update
7e0bd49 
[ghstack-poisoned]
zhuhaozhe added a commit that referenced this pull request Jun 4, 2024
@zhuhaozhe
[inductor] enable bf32 for mkldnn linear pointwise/binary in inductor
bc94d01 
ghstack-source-id: 5deb9a0
Pull Request resolved: #127294
@zhuhaozhe
Update
e01772b 
[ghstack-poisoned]
zhuhaozhe added a commit that referenced this pull request Jun 5, 2024
@zhuhaozhe
[inductor] enable bf32 for mkldnn linear pointwise/binary in inductor
b58e116 
ghstack-source-id: 02c3c91
Pull Request resolved: #127294
@zhuhaozhe
Update
1dde237 
[ghstack-poisoned]
@zhuhaozhe zhuhaozhe requested a review from jgong5 June 5, 2024 08:33
zhuhaozhe added a commit that referenced this pull request Jun 5, 2024
@zhuhaozhe
[inductor] enable bf32 for mkldnn linear pointwise/binary in inductor
4d90ff0 
ghstack-source-id: 02c3c91
Pull Request resolved: #127294
@zhuhaozhe
Update
ed0c68f 
[ghstack-poisoned]
zhuhaozhe added a commit that referenced this pull request Jun 7, 2024
@zhuhaozhe
[inductor] enable bf32 for mkldnn linear pointwise/binary in inductor
ad9bcb8 
ghstack-source-id: 02c3c91
Pull Request resolved: #127294
@zhuhaozhe
Update
ab13767 
[ghstack-poisoned]
@yanbing-j yanbing-j mentioned this pull request Jun 25, 2025
Closed
yanbing-j added 3 commits June 25, 2025 07:33
@yanbing-j
Update
79e8dfd 
[ghstack-poisoned]
@yanbing-j
Update
3776c31 
[ghstack-poisoned]
@yanbing-j
Update
5761f5d 
[ghstack-poisoned]
yanbing-j pushed a commit to yanbing-j/pytorch that referenced this pull request Jun 30, 2025
@zhuhaozhe @yanbing-j
[inductor] enable bf32 for mkldnn linear pointwise/binary in inductor
e389464 
ghstack-source-id: 8ea2aaa
Pull Request resolved: pytorch#127294
yanbing-j added 4 commits June 30, 2025 08:56
@yanbing-j
Update
2192f93 
[ghstack-poisoned]
@yanbing-j
Update
682b6c5 
[ghstack-poisoned]
@yanbing-j
Update
1d12fbb 
[ghstack-poisoned]
8000 @yanbing-j
Update
c7e8536 
[ghstack-poisoned]
@yanbing-j yanbing-j mentioned this pull request Jul 3, 2025
Closed
yanbing-j added 2 commits July 3, 2025 01:36
@yanbing-j
Update
2d2956b 
[ghstack-poisoned]
@yanbing-j
Update
fc2e94e 
[ghstack-poisoned]
@yanbing-j yanbing-j marked this pull request as ready for review July 3, 2025 08:41
@yanbing-j
Copy link
Collaborator
yanbing-j commented Jul 3, 2025

@pytorchbot merge

@pytorchmergebot
Copy link
Collaborator
pytorchmergebot commented Jul 3, 2025

Merge failed

Reason: Approvers from one of the following sets are needed:

  • superuser (pytorch/metamates)
  • Core Reviewers (mruberry, lezcano, Skylion007, ngimel, peterbell10, ...)
  • Core Maintainers (soumith, gchanan, ezyang, dzhulgakov, malfet, ...)
Details for Dev Infra team Raised by workflow job

Failing merge rule: Core Maintainers

@yanbing-j yanbing-j requested a review from jansel July 3, 2025 09:49
@yanbing-j
Copy link
Collaborator
yanbing-j commented Jul 4, 2025

Hi @jansel , could you please help review this PR as well? Thanks!

This PR is the follow-up of #125888, which is to refine fp32 precision API and has been merged. This PR will allow BF32 for mkldnn linear, which is use BF16 as FP32 internal precision for calculation.

@yanbing-j
Update
63915d3 
[ghstack-poisoned]
@yanbing-j
Copy link
Collaborator
yanbing-j commented Jul 7, 2025

@pytorchbot merge

@pytorchmergebot
Copy link
Collaborator
pytorchmergebot commented Jul 7, 2025

Merge started

Your change will be merged once all checks pass (ETA 0-4 Hours).

Learn more about merging in the wiki.

Questions? Feedback? Please reach out to the PyTorch DevX Team

Advanced Debugging
Check the merge workflow status
here

@github-actions github-actions bot deleted the gh/zhuhaozhe/33/head branch August 7, 2025 02:21
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

@jansel jansel jansel approved these changes

@jgong5 jgong5 jgong5 approved these changes

Assignees

No one assigned

Labels

ci-no-td Do not run TD on this PR ciflow/inductor ciflow/rocm Trigger "default" config CI on ROCm ciflow/rocm-mi300 Trigger "default" config CI on ROCm MI300 ciflow/trunk Trigger trunk jobs on your pull request ciflow/xpu Run XPU CI tasks Merged module: cpu CPU specific problem (e.g., perf, algorithm) module: inductor open source topic: not user facing topic category

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

7 participants

@zhuhaozhe @pytorchmergebot @yanbing-j @jansel @jgong5 @pytorchbot
0