8000 Decomp for aten.dropout by SherlockNoMad · Pull Request #106274 · pytorch/pytorch · GitHub
[go: up one dir, main page]
Skip to content

Decomp for aten.dropout #106274

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom
Closed
wants to merge 1 commit into from
Closed

Decomp for aten.dropout #106274

wants to merge 1 commit into from

Conversation

SherlockNoMad
Copy link
Contributor
@SherlockNoMad SherlockNoMad commented Jul 30, 2023 •
edited
Loading

When exporting dropout with cpu tensor, we get following graph module

    class GraphModule(torch.nn.Module):
        def forward(self, arg0_1: f32[512, 10]):
            empty_memory_format: f32[512, 10] = torch.ops.aten.empty.memory_format([512, 10], dtype = torch.float32, layout = torch.strided, device = device(type='cpu'), pin_memory = False, memory_format = torch.contiguous_format)
            bernoulli_p: f32[512, 10] = torch.ops.aten.bernoulli.p(empty_memory_format, 0.9);  empty_memory_format = None
            div_scalar: f32[512, 10] = torch.ops.aten.div.Scalar(bernoulli_p, 0.9);  bernoulli_p = None
            mul_tensor: f32[512, 10] = torch.ops.aten.mul.Tensor(arg0_1, div_scalar);  arg0_1 = div_scalar = None
            return (mul_tensor,)

In addition, if we export with eval() mode, we will have an empty graph.

However, when exporting with cuda tensor, we got

    class GraphModule(torch.nn.Module):
        def forward(self, arg0_1: f32[512, 10]):
            native_dropout_default = torch.ops.aten.native_dropout.default(arg0_1, 0.1, True);  arg0_1 = None
            getitem: f32[512, 10] = native_dropout_default[0];  native_dropout_default = None
            return (getitem,)

and exporting under eval() mode will still have a dropout node in graph.

This PR make exporting with CPU tensor also produce aten.native_dropout.

cc @voznesenskym @penguinwu @EikanWang @jgong5 @Guobing-Chen @XiaobingSuper @zhuhaozhe @blzheng @Xia-Weiwen @wenzhe-nrv @jiayisunx @peterbell10 @ipiszy @ngimel @yf225 @chenyang78 @kadeng @muchulee8 @aakhundov

@pytorch-bot
Copy link
pytorch-bot bot commented Jul 30, 2023 •
edited
Loading

🔗 Helpful Links

🧪 See artifacts and rendered test results at hud.pytorch.org/pr/106274

Note: Links to docs will display an error until the docs builds have been completed.

✅ 2 Unrelated Failures

As of commit 1bbb40a with merge base de8a91f (image):

BROKEN TRUNK - The following job failed but were present on the merge base:

👉 Rebase onto the `viable/strict` branch to avoid these failures

UNSTABLE - The following job failed but was likely due to flakiness present on trunk and has been marked as unstable:

This comment was automatically generated by Dr. CI and updates every 15 minutes.

peterbell10
peterbell10 reviewed Jul 31, 2023
View reviewed changes
@ezyang
Copy link
Contributor
ezyang commented Jul 31, 2023

I'm down, if you can get it to pass tests...

@SherlockNoMad SherlockNoMad added the topic: not user facing topic category label Jul 31, 2023
@andrewor14 andrewor14 mentioned this pull request Aug 22, 2023
Closed
@github-actions github-actions bot added the release notes: quantization release notes category label Aug 22, 2023
ezyang
ezyang reviewed Aug 22, 2023
View reviewed changes
torch/_decomp/decompositions.py Outdated
if train and p != 0:
return aten.native_dropout(input, p, train)[0]
else:
return input
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I hope this is what the real op does (return the input directly), because this can cause some gnarly bugs if it isn't (especially because you are registering a regular decomp for this op)

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@bdhirsh I remember you mentioned that native_dropout returns a clone in both train and eval branches. Is this OK?

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

both native_dropout_cpu and native_droput_cuda returns a

    output = input.clone();

IIUC, this is different from returning the input direction. This is a difference instance of the tensor with different storage.

I need to update this to return input.clone()

@SherlockNoMad SherlockNoMad force-pushed the bahuang/dropout branch 2 times, most recently from 1f9bf22 to 442d74c Compare August 22, 2023 22:44
@SherlockNoMad
Copy link
Contributor Author

@pytorchbot merge

@pytorch-bot pytorch-bot bot added the ciflow/trunk Trigger trunk jobs on your pull request label Aug 22, 2023
@pytorchmergebot
8000 Copy link
Collaborator

Merge started

Your change will be merged once all checks pass (ETA 0-4 Hours).

Learn more about merging in the wiki.

Questions? Feedback? Please reach out to the PyTorch DevX Team

Advanced Debugging
Check the merge workflow status
here

@pytorchmergebot
Copy link
Collaborator

Merge failed

Reason: 1 mandatory check(s) failed. The first few are:

Dig deeper by viewing the failures on hud

Details for Dev Infra team Raised by workflow job

Failing merge rule: Core Maintainers

@SherlockNoMad
Copy link
Contributor Author

@pytorchbot merge

@pytorchmergebot
Copy link
Collaborator

Merge started

Your change will be merged once all checks pass (ETA 0-4 Hours).

Learn more about merging in the wiki.

Questions? Feedback? Please reach out to the PyTorch DevX Team

Advanced Debugging
Check the merge workflow status
here

@Valentine233 Valentine233 mentioned this pull request Aug 29, 2023
Closed
pytorchmergebot pushed a commit that referenced this pull request Aug 30, 2023
@Valentine233 @pytorchmergebot
[Inductor] Add fused_attention pattern matcher with additional clone (#…
3a79621 
…108141)

A previous PR #106274 decomposes `aten.dropout` and would create a `clone()` when `eval()` or `p=0`. This makes many SDPA-related models fail to match fused_attention pattern matchers.

This PR adds new fused_attention pattern matchers with an additional clone to re-enable the SDPA op matching.

Pull Request resolved: #108141
Approved by: https://github.com/jgong5, https://github.com/eellison
Valentine233 added a commit that referenced this pull request Aug 31, 2023
@Valentine233
[Inductor] Add fused_attention pattern matcher with additional clone (#…
8000
be8a422 
…108141)

A previous PR #106274 decomposes `aten.dropout` and would create a `clone()` when `eval()` or `p=0`. This makes many SDPA-related models fail to match fused_attention pattern matchers.

This PR adds new fused_attention pattern matchers with an additional clone to re-enable the SDPA op matching.

Pull Request resolved: #108141
Approved by: https://github.com/jgong5, https://github.com/eellison
@Valentine233 Valentine233 mentioned this pull request Aug 31, 2023
Closed
CaoE pushed a commit to CaoE/pytorch that referenced this pull request Sep 1, 2023
@Valentine233 @CaoE
[Inductor] Add fused_attention pattern matcher with additional clone (p…
a3ec13e 
…ytorch#108141)

A previous PR pytorch#106274 decomposes `aten.dropout` and would create a `clone()` when `eval()` or `p=0`. This makes many SDPA-related models fail to match fused_attention pattern matchers.

This PR adds new fused_attention pattern matchers with an additional clone to re-enable the SDPA op matching.

Pull Request resolved: pytorch#108141
Approved by: https://github.com/jgong5, https://github.com/eellison
atalman pushed a commit that referenced this pull request Sep 5, 2023
@Valentine233
[Inductor] Add fused_attention pattern matcher with additional clone (#…
03e7f0b 
…108141) (#108327)

A previous PR #106274 decomposes `aten.dropout` and would create a `clone()` when `eval()` or `p=0`. This makes many SDPA-related models fail to match fused_attention pattern matchers.

This PR adds new fused_attention pattern matchers with an additional clone to re-enable the SDPA op matching.

Pull Request resolved: #108141
Approved by: https://github.com/jgong5, https://github.com/eellison
@andrewor14 andrewor14 mentioned this pull request Sep 22, 2023
Closed
andrewor14 added a commit that referenced this pull request Sep 22, 2023
@andrewor14
Decompose to native_dropout in eval mode as well
a38e200 
Summary: #106274 added a
decomp for dropout such that for train mode we get
`aten.native_dropout`, while for eval mode we get `aten.clone`.
This commit makes it such that we get `aten.native_dropout` for
both train and eval modes. For eval mode, we let the cloning
happen in the aten op itself.

The main motivation behind this change is QAT, which needs to
swap between `aten.native_dropout(train=True)` and
`aten.native_dropout(train=False)` in the graph. This was
previously difficult to do since there was no dropout op to
match and replace in eval mode.

Test Plan: python test/test_ops.py

Reviewers: SherlockNoMad, bdhirsh

Subscribers: SherlockNoMad, bdhirsh, supriyar

[ghstack-poisoned]
andrewor14 added a commit that referenced this pull request Sep 22, 2023
@andrewor14
Decompose to native_dropout in eval mode as well
ea769f4 
Summary: #106274 added a
decomp for dropout such that for train mode we get
`aten.native_dropout`, while for eval mode we get `aten.clone`.
This commit makes it such that we get `aten.native_dropout` for
both train and eval modes. For eval mode, we let the cloning
happen in the aten op itself.

The main motivation behind this change is QAT, which needs to
swap between `aten.native_dropout(train=True)` and
`aten.native_dropout(train=False)` in the graph. This was
previously difficult to do since there was no dropout op to
match and replace in eval mode.

Test Plan: python test/test_ops.py

Reviewers: SherlockNoMad, bdhirsh

Subscribers: SherlockNoMad, bdhirsh, supriyar

ghstack-source-id: c899961
Pull Request resolved: #109914
andrewor14 added a commit that referenced this pull request Nov 7, 2023
@andrewor14
Update base for Update on "Decompose to native_dropout in eval mode a…
6198cfc 
…s well"

Summary: #106274 added a
decomp for dropout such that for train mode we get
`aten.native_dropout`, while for eval mode we get `aten.clone`.
This commit makes it such that we get `aten.native_dropout` for
both train and eval modes. For eval mode, we let the cloning
happen in the aten op itself.

The main motivation behind this change is QAT, which needs to
swap between `aten.native_dropout(train=True)` and
`aten.native_dropout(train=False)` in the graph. This was
previously difficult to do since there was no dropout op to
match and replace in eval mode.

Test Plan: python test/test_ops.py

Reviewers: SherlockNoMad, bdhirsh

Subscribers: SherlockNoMad, bdhirsh, supriyar

[ghstack-poisoned]
andrewor14 added a commit that referenced this pull request Nov 7, 2023
@andrewor14
Update on "Decompose to native_dropout in eval mode as well"
992456d 
Summary: #106274 added a
decomp for dropout such that for train mode we get
`aten.native_dropout`, while for eval mode we get `aten.clone`.
This commit makes it such that we get `aten.native_dropout` for
both train and eval modes. For eval mode, we let the cloning
happen in the aten op itself.

The main motivation behind this change is QAT, which needs to
swap between `aten.native_dropout(train=True)` and
`aten.native_dropout(train=False)` in the graph. This was
previously difficult to do since there was no dropout op to
match and replace in eval mode.

Test Plan: python test/test_ops.py

Reviewers: SherlockNoMad, bdhirsh

Subscribers: SherlockNoMad, bdhirsh, supriyar

[ghstack-poisoned]
andrewor14 added a commit that referenced this pull request Nov 7, 2023
@andrewor14
Update base for Update on "Decompose to native_dropout in eval mode a…
fc02cf7 
…s well"

Summary: #106274 added a
decomp for dropout such that for train mode we get
`aten.native_dropout`, while for eval mode we get `aten.clone`.
This commit makes it such that we get `aten.native_dropout` for
both train and eval modes. For eval mode, we let the cloning
happen in the aten op itself.

The main motivation behind this change is QAT, which needs to
swap between `aten.native_dropout(train=True)` and
`aten.native_dropout(train=False)` in the graph. This was
previously difficult to do since there was no dropout op to
match and replace in eval mode.

Test Plan: python test/test_ops.py

Reviewers: SherlockNoMad, bdhirsh

Subscribers: SherlockNoMad, bdhirsh, supriyar

[ghstack-poisoned]
andrewor14 added a commit that referenced this pull request Nov 7, 2023
@andrewor14
Update on "Decompose to native_dropout in eval mode as well"
0dd2675 
Summary: #106274 added a
decomp for dropout such that for train mode we get
`aten.native_dropout`, while for eval mode we get `aten.clone`.
This commit makes it such that we get `aten.native_dropout` for
both train and eval modes. For eval mode, we let the cloning
happen in the aten op itself.

The main motivation behind this change is QAT, which needs to
swap between `aten.native_dropout(train=True)` and
`aten.native_dropout(train=False)` in the graph. This was
previously difficult to do since there was no dropout op to
match and replace in eval mode.

Test Plan: python test/test_ops.py

Reviewers: SherlockNoMad, bdhirsh

Subscribers: SherlockNoMad, bdhirsh, supriyar

[ghstack-poisoned]
andrewor14 added a commit that referenced this pull request Nov 7, 2023
@andrewor14
Decompose to native_dropout in eval mode as well
10ee0c2 
Summary: #106274 added a
decomp for dropout such that for train mode we get
`aten.native_dropout`, while for eval mode we get `aten.clone`.
This commit makes it such that we get `aten.native_dropout` for
both train and eval modes. For eval mode, we let the cloning
happen in the aten op itself.

The main motivation behind this change is QAT, which needs to
swap between `aten.native_dropout(train=True)` and
`aten.native_dropout(train=False)` in the graph. This was
previously difficult to do since there was no dropout op to
match and replace in eval mode.

Test Plan: python test/test_ops.py

Reviewers: SherlockNoMad, bdhirsh

Subscribers: SherlockNoMad, bdhirsh, supriyar

ghstack-source-id: c7635cf
Pull Request resolved: #109914
andrewor14 added a commit that referenced this pull request Nov 10, 2023
@andrewor14
Decompose to native_dropout in eval mode as well
eda7635 
Summary: #106274 added a
decomp for dropout such that for train mode we get
`aten.native_dropout`, while for eval mode we get `aten.clone`.
This commit makes it such that we get `aten.native_dropout` for
both train and eval modes. For eval mode, we let the cloning
happen in the aten op itself.

The main motivation behind this change is QAT, which needs to
swap between `aten.native_dropout(train=True)` and
`aten.native_dropout(train=False)` in the graph. This was
previously difficult to do since there was no dropout op to
match and replace in eval mode.

Test Plan: python test/test_ops.py

Reviewers: SherlockNoMad, bdhirsh

Subscribers: SherlockNoMad, bdhirsh, supriyar

ghstack-source-id: c7635cf
Pull Request resolved: #109914
andrewor14 added a commit that referenced this pull request Nov 10, 2023
@andrewor14
Update base for Update on "Decompose to native_dropout in eval mode a…
3b4d65d 
…s well"

Summary: #106274 added a
decomp for dropout such that for train mode we get
`aten.native_dropout`, while for eval mode we get `aten.clone`.
This commit makes it such that we get `aten.native_dropout` for
both train and eval modes. For eval mode, we let the cloning
happen in the aten op itself.

The main motivation behind this change is QAT, which needs to
swap between `aten.native_dropout(train=True)` and
`aten.native_dropout(train=False)` in the graph. This was
previously difficult to do since there was no dropout op to
match and replace in eval mode.

Test Plan: python test/test_ops.py

Reviewers: SherlockNoMad, bdhirsh

Subscribers: SherlockNoMad, bdhirsh, supriyar

cc avikchaudhuri gmagogsfm zhxchen17 tugsbayasgalan angelayi suo ydwu4

[ghstack-poisoned]
andrewor14 added a commit that referenced this pull request Nov 10, 2023
@andrewor14
Update on "Decompose to native_dropout in eval mode as well"
19783c4 
Summary: #106274 added a
decomp for dropout such that for train mode we get
`aten.native_dropout`, while for eval mode we get `aten.clone`.
This commit makes it such that we get `aten.native_dropout` for
both train and eval modes. For eval mode, we let the cloning
happen in the aten op itself.

The main motivation behind this change is QAT, which needs to
swap between `aten.native_dropout(train=True)` and
`aten.native_dropout(train=False)` in the graph. This was
previously difficult to do since there was no dropout op to
match and replace in eval mode.

Test Plan: python test/test_ops.py

Reviewers: SherlockNoMad, bdhirsh

Subscribers: SherlockNoMad, bdhirsh, supriyar

cc avikchaudhuri gmagogsfm zhxchen17 tugsbayasgalan angelayi suo ydwu4

[ghstack-poisoned]
andrewor14 added a commit that referenced this pull request Nov 13, 2023
@andrewor14
Update base for Update on "Decompose to native_dropout in eval mode a…
94db621 
…s well"

Summary: #106274 added a
decomp for dropout such that for train mode we get
`aten.native_dropout`, while for eval mode we get `aten.clone`.
This commit makes it such that we get `aten.native_dropout` for
both train and eval modes. For eval mode, we let the cloning
happen in the aten op itself.

The main motivation behind this change is QAT, which needs to
swap between `aten.native_dropout(train=True)` and
`aten.native_dropout(train=False)` in the graph. This was
previously difficult to do since there was no dropout op to
match and replace in eval mode.

Test Plan: python test/test_ops.py

Reviewers: SherlockNoMad, bdhirsh

Subscribers: SherlockNoMad, bdhirsh, supriyar

cc avikchaudhuri gmagogsfm zhxchen17 tugsbayasgalan angelayi suo ydwu4

[ghstack-poisoned]
andrewor14 added a commit that referenced this pull request Nov 13, 2023
@andrewor14
Update on "Decompose to native_dropout in eval mode as well"
a0ee609 
Summary: #106274 added a
decomp for dropout such that for train mode we get
`aten.native_dropout`, while for eval mode we get `aten.clone`.
This commit makes it such that we get `aten.native_dropout` for
both train and eval modes. For eval mode, we let the cloning
happen in the aten op itself.

The main motivation behind this change is QAT, which needs to
swap between `aten.native_dropout(train=True)` and
`aten.native_dropout(train=False)` in the graph. This was
previously difficult to do since there was no dropout op to
match and replace in eval mode.

Test Plan: python test/test_ops.py

Reviewers: SherlockNoMad, bdhirsh

Subscribers: SherlockNoMad, bdhirsh, supriyar

cc avikchaudhuri gmagogsfm zhxchen17 tugsbayasgalan angelayi suo ydwu4

[ghstack-poisoned]
andrewor14 added a commit that referenced this pull request Nov 14, 2023
@andrewor14
Update base for Update on "Decompose to native_dropout in eval mode a…
1eff314 
…s well"

Summary: #106274 added a
decomp for dropout such that for train mode we get
`aten.native_dropout`, while for eval mode we get `aten.clone`.
This commit makes it such that we get `aten.native_dropout` for
both train and eval modes. For eval mode, we let the cloning
happen in the aten op itself.

The main motivation behind this change is QAT, which needs to
swap between `aten.native_dropout(train=True)` and
`aten.native_dropout(train=False)` in the graph. This was
previously difficult to do since there was no dropout op to
match and replace in eval mode.

Test Plan: python test/test_ops.py

Reviewers: SherlockNoMad, bdhirsh

Subscribers: SherlockNoMad, bdhirsh, supriyar

cc avikchaudhuri gmagogsfm zhxchen17 tugsbayasgalan angelayi suo ydwu4

[ghstack-poisoned]
andrewor14 added a commit that referenced this pull request Nov 14, 2023
@andrewor14
Update on "Decompose to native_dropout in eval mode as well"
61ad335 
Summary: #106274 added a
decomp for dropout such that for train mode we get
`aten.native_dropout`, while for eval mode we get `aten.clone`.
This commit makes it such that we get `aten.native_dropout` for
both train and eval modes. For eval mode, we let the cloning
happen in the aten op itself.

The main motivation behind this change is QAT, which needs to
swap between `aten.native_dropout(train=True)` and
`aten.native_dropout(train=False)` in the graph. This was
previously difficult to do since there was no dropout op to
match and replace in eval mode.

Test Plan: python test/test_ops.py

Reviewers: SherlockNoMad, bdhirsh

Subscribers: SherlockNoMad, bdhirsh, supriyar

cc avikchaudhuri gmagogsfm zhxchen17 tugsbayasgalan angelayi suo ydwu4

[ghstack-poisoned]
@andrewor14 andrewor14 mentioned this pull request Dec 12, 2023
Open
@github-actions github-actions bot deleted the bahuang/dropout branch February 22, 2025 02:02
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers

@andrewor14 andrewor14 andrewor14 left review comments

@peterbell10 peterbell10 peterbell10 left review comments

@ezyang ezyang ezyang approved these changes

@zhxchen17 zhxchen17 Awaiting requested review from zhxchen17

@mruberry mruberry Awaiting requested review from mruberry mruberry is a code owner

@ngimel ngimel Awaiting requested review from ngimel

Assignees
No one assigned
Labels
ciflow/inductor ciflow/trunk Trigger trunk jobs on your pull request Merged module: inductor release notes: quantization release notes category topic: not user facing topic category
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.
5 participants
@SherlockNoMad @ezyang @pytorchmergebot @andrewor14 @peterbell10
0