[inductor] weight prepack for _convolution_transpose_pointwise #90266

chunyuan-w · 2022-12-06T07:34:11Z

Stack from ghstack (oldest at bottom):

This PR implements weight prepack for _convolution_transpose_pointwise, similar to #88988.

In addition, we add a kernel for Meta tensor input to reduce the compilation time.

cc @VitalyFedyunin @jgong5 @mingfeima @XiaobingSuper @sanchitintel @ashokei @jingxu10 @mlazos @soumith @voznesenskym @yanboliang @penguinwu @anijain2305 @EikanWang @Guobing-Chen @zhuhaozhe @blzheng @Xia-Weiwen @wenzhe-nrv @jiayisunx @peterbell10 @desertfire

pytorch-bot · 2022-12-06T07:34:14Z

🔗 Helpful Links

🧪 See artifacts and rendered test results at hud.pytorch.org/pr/90266

📄 Preview Python docs built from this PR
📄 Preview C++ docs built from this PR
❓ Need help or want to give feedback on the CI? Visit the bot commands wiki or our office hours

Note: Links to docs will display an error until the docs builds have been completed.

❌ 4 Failures

As of commit 7ccfae3:

NEW FAILURES - The following jobs have failed:

This comment was automatically generated by Dr. CI and updates every 15 minutes.

ghstack-source-id: e570efe Pull Request resolved: pytorch#90266

[ghstack-poisoned]

…wise" cc VitalyFedyunin jgong5 mingfeima XiaobingSuper sanchitintel ashokei jingxu10 mlazos soumith voznesenskym yanboliang penguinwu anijain2305 EikanWang Guobing-Chen zhuhaozhe blzheng Xia-Weiwen wenzhe-nrv jiayisunx peterbell10 desertfire [ghstack-poisoned]

jgong5 · 2022-12-09T03:38:41Z

aten/src/ATen/native/mkldnn/Conv.cpp

+    // aten tensor with the shape of the prepacked tensor
+    at::Tensor origin_weight_t;
+    if (groups > 1) {
+      origin_weight_t = weight.transpose(0, 1).reshape(weight_IOHW_sizes);
+    } else {
+      origin_weight_t = weight.transpose(0, 1);
+    }
+    w = itensor_from_tensor(origin_weight_t);
+    w.transpose_(0, 1);


Do we have a case that we want to handle this "aten tensor with the shape of the prepacked tensor", or we always model prepacked weight with "MKLDNNTensor"?

When compiling the graph, each node is run with FakeTensor inputs and in this case, the weight tensor is an aten tensor with the shape of the prepacked tensor.

For Meta tensor: added an implementation mkldnn_convolution_transpose_pointwise_meta.

For MKLDNN and CPU tensor: supported the computation for both of them in the original implementation: mkldnn_convolution_transpose_pointwise.

…wise" cc VitalyFedyunin jgong5 mingfeima XiaobingSuper sanchitintel ashokei jingxu10 mlazos soumith voznesenskym yanboliang penguinwu anijain2305 EikanWang Guobing-Chen zhuhaozhe blzheng Xia-Weiwen wenzhe-nrv jiayisunx peterbell10 desertfire [ghstack-poisoned]

…wise" This PR implements weight prepack for `_convolution_transpose_pointwise`, similar to #88988. Different from Conv2d, once ConvTranspose weight has been prepacked, the size changes: - Original weight size: `[i, o, ...]` - Prepacked size: - Groups > 1: `[g*o, i/g, ...]` - Groups == 1: `[o, i, ...]` The `_convolution_transpose_pointwise` kernel handles the below two situations: - During compilation, when running the node of the FX graph, the kernel gets a public weight tensor with prepacked size. The kernel will convert the weight back to the original weight size for computation. - During execution, it gets a mkldnn tensor and will directly use it for computation. cc VitalyFedyunin jgong5 mingfeima XiaobingSuper sanchitintel ashokei jingxu10 mlazos soumith voznesenskym yanboliang penguinwu anijain2305 EikanWang Guobing-Chen zhuhaozhe blzheng Xia-Weiwen wenzhe-nrv jiayisunx peterbell10 desertfire [ghstack-poisoned]

…wise" This PR implements weight prepack for `_convolution_transpose_pointwise`, similar to #88988. Different from Conv2d, once ConvTranspose weight has been prepacked, the size changes: - Original weight size: `[i, o, ...]` - Prepacked size: - Groups > 1: `[g*o, i/g, ...]` - Groups == 1: `[o, i, ...]` The `_convolution_transpose_pointwise` kernel handles the below two situations: - During compilation, when running the node of the FX graph, the kernel gets a public 10000 weight tensor with prepacked size. The kernel will convert the weight back to the original weight size for computation. - During execution, it gets a mkldnn tensor and will directly use it for computation. cc VitalyFedyunin jgong5 mingfeima XiaobingSuper sanchitintel ashokei jingxu10 mlazos soumith voznesenskym yanboliang penguinwu anijain2305 EikanWang Guobing-Chen zhuhaozhe blzheng Xia-Weiwen wenzhe-nrv jiayisunx peterbell10 desertfire [ghstack-poisoned]

…wise" This PR implements weight prepack for `_convolution_transpose_pointwise`, similar to #88988. Different from Conv2d, once ConvTranspose weight has been prepacked, the size changes: - Original weight size: `[i, o, ...]` - Prepacked size: - Groups > 1: `[g*o, i/g, ...]` - Groups == 1: `[o, i, ...]` The `_convolution_transpose_pointwise` kernel handles the below two situations: - During compilation, when running the node of the FX graph, the kernel gets a public weight tensor with prepacked size. The kernel will convert the weight back to the original weight size for computation. - During execution, it gets a mkldnn tensor and will directly use it for computation. cc VitalyFedyunin jgong5 mingfeima XiaobingSuper sanchitintel ashokei jingxu10 mlazos soumith voznesenskym yanboliang penguinwu anijain2305 EikanWang Guobing-Chen zhuhaozhe blzheng Xia-Weiwen wenzhe-nrv jiayisunx peterbell10 desertfire [ghstack-poisoned]

jgong5 · 2022-12-13T01:20:39Z

aten/src/ATen/native/mkldnn/Conv.cpp

+
+  const ideep::tensor x = itensor_from_tensor(input);
+
+  ideep::tensor w = itensor_from_tensor(weight);;


Suggested change

ideep::tensor w = itensor_from_tensor(weight);;

ideep::tensor w = itensor_from_tensor(weight);

Fixed the typo.

jgong5 · 2022-12-13T01:25:30Z

aten/src/ATen/native/mkldnn/Conv.cpp

@@ -709,6 +725,8 @@ Tensor mkldnn_convolution_transpose_pointwise(
    torch::List<c10::optional<at::Scalar>> scalars,
    c10::optional<c10::string_view> algorithm) {
  c10::impl::ExcludeDispatchKeyGuard edkg(c10::autograd_dispatch_keyset);
+  bool use_channels_last =
+      weight_t.is_mkldnn() || mkldnn_conv_use_channels_last(input_t, weight_t);


The logic is ok. Not related to this PR, but does it make more sense that mkldnn_conv_use_channels_last should prefer channels last if input is a strided tensor and weight is mkldnn? @XiaobingSuper

jgong5 · 2022-12-13T01:31:04Z

torch/_inductor/ir.py

+def _conv_input_size(
+    output_size, weight_size, padding, output_padding, stride, dilation, groups
+):


How about we keep this function local to _prepare_convolution_fusion_create first? It is only used by this function.

Made this function local to _prepare_convolution_fusion_create.

jgong5 · 2022-12-13T01:31:12Z

torch/_inductor/ir.py

+def _original_deconv_weight_size(
+    prepacked_weight,
+    groups,
+):


ghstack-source-id: 3239214 Pull Request resolved: pytorch#90266

…wise" This PR implements weight prepack for `_convolution_transpose_pointwise`, similar to #88988. In addition, we add a kernel for Meta tensor input to reduce the compilation time. cc VitalyFedyunin jgong5 mingfeima XiaobingSuper sanchitintel ashokei jingxu10 mlazos soumith voznesenskym yanboliang penguinwu anijain2305 EikanWang Guobing-Chen zhuhaozhe blzheng Xia-Weiwen wenzhe-nrv jiayisunx peterbell10 desertfire [ghstack-poisoned]

ghstack-source-id: e204dd7 Pull Request resolved: pytorch#90266

chunyuan-w · 2023-01-11T07:09:48Z

Re-opened in #91955

…for _convolution_transpose_pointwise" Re-open #90266 since earlier pr on that stack got reverted. Depend on internal ideep upgrade. cc jgong5 mingfeima XiaobingSuper sanchitintel ashokei jingxu10 mlazos soumith voznesenskym yanboliang penguinwu anijain2305 EikanWang Guobing-Chen zhuhaozhe blzheng Xia-Weiwen wenzhe-nrv jiayisunx peterbell10 desertfire [ghstack-poisoned]

…_transpose_pointwise" Re-open #90266 since earlier pr on that stack got reverted. Depend on internal ideep upgrade. cc jgong5 mingfeima XiaobingSuper sanchitintel ashokei jingxu10 mlazos soumith voznesenskym yanboliang penguinwu anijain2305 EikanWang Guobing-Chen zhuhaozhe blzheng Xia-Weiwen wenzhe-nrv jiayisunx peterbell10 desertfire [ghstack-poisoned]

…for _convolution_transpose_pointwise" Re-open #90266 since earlier pr on that stack got reverted. Depend on internal ideep upgrade. [Update]: internal ideep upgrade issue is resolved in #92239. cc jgong5 mingfeima XiaobingSuper sanchitintel ashokei jingxu10 mlazos soumith voznesenskym yanboliang penguinwu anijain2305 EikanWang Guobing-Chen zhuhaozhe blzheng Xia-Weiwen wenzhe-nrv jiayisunx peterbell10 desertfire [ghstack-poisoned]

…_transpose_pointwise" Re-open #90266 since earlier pr on that stack got reverted. Depend on internal ideep upgrade. [Update]: internal ideep upgrade issue is resolved in #92239. cc jgong5 mingfeima XiaobingSuper sanchitintel ashokei jingxu10 mlazos soumith voznesenskym yanboliang penguinwu anijain2305 EikanWang Guobing-Chen zhuhaozhe blzheng Xia-Weiwen wenzhe-nrv jiayisunx peterbell10 desertfire [ghstack-poisoned]

…pointwise (#91955) Re-open #90266 since earlier pr on that stack got reverted. Depend on internal ideep upgrade. [Update]: internal ideep upgrade issue is resolved in #92239. Pull Request resolved: #91955 Approved by: https://github.com/jgong5, https://github.com/desertfire

github-actions bot added ciflow/inductor module: cpu CPU specific problem (e.g., perf, algorithm) module: inductor labels Dec 6, 2022

This was referenced Dec 6, 2022

add conv_transpose2d pointwise(unary) fusion kernel #90264

Closed

[inductor] add conv_transpose2d unary fusion for cpu in inference mode #90265

Closed

[inductor] weight prepack for single conv_transpose2d #90267

Closed

chunyuan-w marked this pull request as draft December 6, 2022 07:35

chunyuan-w added the topic: not user facing topic category label Dec 6, 2022

pytorchbot added the open source label Dec 6, 2022

chunyuan-w added a commit to chunyuan-w/pytorch that referenced this pull request Dec 6, 2022

[inductor] weight prepack for _convolution_transpose_pointwise

16aa150

ghstack-source-id: e570efe Pull Request resolved: pytorch#90266

chunyuan-w added 6 commits December 6, 2022 14:51

[inductor] weight prepack for _convolution_transpose_pointwise

3a3fed6

[ghstack-poisoned]

chunyuan-w requested review from jgong5, EikanWang and XiaobingSuper December 9, 2022 03:29

jgong5 reviewed Dec 9, 2022

View reviewed changes

chunyuan-w added 4 commits December 9, 2022 09:38

jgong5 requested changes Dec 13, 2022

View reviewed changes

chunyuan-w requested a review from jgong5 December 13, 2022 02:03

chunyuan-w added a commit to chunyuan-w/pytorch that referenced this pull request Dec 13, 2022

[inductor] weight prepack for _convolution_transpose_pointwise

bb052c7

ghstack-source-id: 3239214 Pull Request resolved: pytorch#90266

jgong5 approved these changes Dec 13, 2022

View reviewed changes

chunyuan-w requested review from jansel, Chillee and desertfire December 13, 2022 10:04

chunyuan-w marked this pull request as ready for review December 15, 2022 01:33

chunyuan-w added 3 commits December 15, 2022 09:55

chunyuan-w added the ciflow/trunk Trigger trunk jobs on your pull request label Dec 15, 2022

chunyuan-w added 2 commits December 15, 2022 14:56

jansel approved these changes Dec 16, 2022

View reviewed changes

chunyuan-w added 2 commits December 19, 2022 08:53

chunyuan-w added a commit to chunyuan-w/pytorch that referenced this pull request Jan 10, 2023

[inductor] weight prepack for _convolution_transpose_pointwise

e4d665e

ghstack-source-id: e204dd7 Pull Request resolved: pytorch#90266

chunyuan-w mentioned this pull request Jan 10, 2023

[Re-open 90266] [inductor] weight prepack for _convolution_transpose_pointwise #91955

Closed

chunyuan-w closed this Jan 11, 2023

EikanWang mentioned this pull request Jan 13, 2023

[PT2.0 Feature Proposal] TorchInductor CPU FP32 Inference Optimization #92135

Closed

facebook-github-bot deleted the gh/chunyuan-w/17/head branch June 8, 2023 15:56

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Uh oh!

[inductor] weight prepack for _convolution_transpose_pointwise #90266

[inductor] weight prepack for _convolution_transpose_pointwise #90266

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!


		const ideep::tensor x = itensor_from_tensor(input);

		ideep::tensor w = itensor_from_tensor(weight);;

[inductor] weight prepack for _convolution_transpose_pointwise #90266

[inductor] weight prepack for _convolution_transpose_pointwise #90266

Uh oh!

Conversation

Uh oh!

Uh oh!

Uh oh!

🔗 Helpful Links

🧪 See artifacts and rendered test results at hud.pytorch.org/pr/90266

❌ 4 Failures

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!