8000 Enable qint8 and quint8 add for AArch64 using ACL directly by davsva01 · Pull Request #146620 · pytorch/pytorch · GitHub
[go: up one dir, main page]
Skip to content

Enable qint8 and quint8 add for AArch64 using ACL directly #146620

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom
Closed
wants to merge 4 commits into from
Closed

Enable qint8 and quint8 add for AArch64 using ACL directly #146620

wants to merge 4 commits into from

Conversation

davsva01
Copy link
Contributor
@davsva01 davsva01 commented Feb 6, 2025 •
edited by pytorch-bot bot
Loading

This enables qint8 and quint8 add for AArch64 through Arm Compute Library (ACL) directly.
It’s based on changes in PR #145942 which enables the use of ACL directly in ATen.
Relative performance improvement using OMP_NUM_THREADS=1 is ~15x, using OMP_NUM_THREADS=32 it’s ~5.4x.

Script to benchmark quantised add performance:

import torch
import torch.profiler as profiler

a_f32 = torch.rand((400, 3456),dtype=torch.float)
b_f32 = torch.rand((400, 3456),dtype=torch.float)
a_q = torch.quantize_per_tensor(a_f32, 1.2, 0, torch.qint8)
b_q = torch.quantize_per_tensor(b_f32, 1.7, 5, torch.qint8)

with profiler.profile(with_stack=True, profile_memory=False, record_shapes=True) as prof:
    for i in range(1000):     
        _ = torch.ops.quantized.add(a_q, b_q, 1.3, 2)
print(prof.key_averages(group_by_input_shape=True).table(sort_by='self_cpu_time_total', row_limit=50))

cc @jgong5 @mingfeima @XiaobingSuper @sanchitintel @ashokei @jingxu10

@pytorch-bot pytorch-bot bot added module: cpu CPU specific problem (e.g., perf, algorithm) release notes: quantization release notes category release notes: releng release notes category labels Feb 6, 2025
@linux-foundation-easycla Linux Foundation: EasyCLA
Copy link
linux-foundation-easycla bot commented Feb 6, 2025 •
edited
Loading

CLA Signed

The committers listed above are authorized under a signed CLA.

@pytorch-bot PyTorch Bot
Copy link
pytorch-bot bot commented Feb 6, 2025 •
edited
Loading

🔗 Helpful Links

🧪 See artifacts and rendered test results at hud.pytorch.org/pr/146620

Note: Links to docs will display an error until the docs builds have been completed.

✅ No Failures

As of commit a0d2046 with merge base 6c3492b (image):
💚 Looks good so far! There are no failures yet. 💚

This comment was automatically generated by Dr. CI and updates every 15 minutes.

malfet
malfet reviewed Feb 6, 2025
View reviewed changes
aten/src/ATen/native/quantized/cpu/ACLUtils.h Outdated
#pragma once

#include <ATen/Config.h>
#if defined(__aarch64__) && AT_MKLDNN_ACL_ENABLED()
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Why restrict it to aarch64?

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks for the quick review!
Yes this is redundant, I have rebased on top of latest changes in #145942 addressing this.

malfet
malfet reviewed
View reviewed changes
Copy link
Contributor
@malfet malfet left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Same generic feedback is to previous PR: why restrict it to aarch64? Shouldn't IS_ACL_ENABLE be sufficient?

@mikaylagawarecki mikaylagawarecki added the triaged This issue has been looked at a team member, and triaged and prioritized into an appropriate module label Feb 7, 2025
@davsva01 davsva01 force-pushed the acl_qadd branch 2 times, most recently from 4a1385a to bb887be Compare February 10, 2025 16:40
fadara01 added a commit to fadara01/pytorch that referenced this pull request Feb 19, 2025
@fadara01
Update Arm Compute Library (ACL) to v25.02
1555bb5 
Among many things, this version of ACL fixes the redundant declaration  warning that we're blocked on in (pytorch#145942, pytorch#146620, 147337) and introduces better scheduling heuristics for GEMMs
@fadara01 fadara01 mentioned this pull request Feb 19, 2025
Closed
pytorchmergebot pushed a commit that referenced this pull request Feb 19, 2025
@fadara01 @pytorchmergebot
Update Arm Compute Library (ACL) to v25.02 (#147454)
5a3a50c 
Among many things, this version of ACL fixes the redundant declaration  warning that we're blocked on in (#145942, #146620, #147337) and introduces better scheduling heuristics for GEMMs

Fixes #ISSUE_NUMBER

Pull Request resolved: #147454
Approved by: https://github.com/malfet
@fadara01
Copy link
Collaborator

@pytorchbot label "arm priority"

@fadara01 fadara01 requested a review from malfet February 20, 2025 10:35
Raymo111 pushed a commit that referenced this pull request Feb 20, 2025
@fadara01 @Raymo111
Update Arm Compute Library (ACL) to v25.02 (#147454)
9e0e193 
Among many things, this version of ACL fixes the redundant declaration  warning that we're blocked on in (#145942, #146620, #147337) and introduces better scheduling heuristics for GEMMs

Fixes #ISSUE_NUMBER

Pull Request resolved: #147454
Approved by: https://github.com/malfet
pytorch-bot bot pushed a commit that referenced this pull request Feb 24, 2025
@fadara01 @Ryo-not-rio
Update Arm Compute Library (ACL) to v25.02 (#147454)
cd7d40f 
Among many things, this version of ACL fixes the redundant declaration  warning that we're blocked on in (#145942, #146620, #147337) and introduces better scheduling heuristics for GEMMs

Fixes #ISSUE_NUMBER

Pull Request resolved: #147454
Approved by: https://github.com/malfet
majing921201 pushed a commit to majing921201/pytorch that referenced this pull request Mar 4, 2025
@fadara01 @majing921201
Update Arm Compute Library (ACL) to v25.02 (pytorch#147454)
00d71f5 
Among many things, this version of ACL fixes the redundant declaration  warning that we're blocked on in (pytorch#145942, pytorch#146620, pytorch#147337) and introduces better scheduling heuristics for GEMMs

Fixes #ISSUE_NUMBER

Pull Request resolved: pytorch#147454
Approved by: https://github.com/malfet
@fadara01
Copy link
Collaborator
fadara01 commented Mar 4, 2025

@malfet, @digantdesai - Could you please give this another look?

@fadara01
Enable Direct Use of Arm Compute Library (ACL) in ATen
99e5d35 
ACL is already built with PyTorch as a shared library when USE_MKLDNN_ACL is set.
Currently, it is only used indirectly in ATen via oneDNN for AArch64 targets. However there are cases where it makes sense to utilize ACL directly without  oneDNN as an intermediary - e.g. quantization. See pytorch#145942, pytorch#147337, pytorch#146620.
This patch enables such use cases by exposing ACL to ATen
@fadara01 fadara01 mentioned this pull request Mar 5, 2025
Closed
fadara01 and others added 3 commits March 5, 2025 11:27
@fadara01
Enable fast qlinear_dynamic path for AArch64 through ACL directly
ac5618b 
This enables a fast path for eager mode dynamic quantization for AArch64 through Arm Compute Library (ACL) directly.

Context: PR pytorch#126687 enabled an optimized implementation for qlinear_dynamic for aarch64 through ideep → oneDNN → ACL which improved performance by ~10x compared to the previous implementation.
However, the current qlinear_dynamic path (ideep → oneDNN → ACL) suffers from high overhead due to the API friction between the stateless oneDNN API and the stateful ACL low-precision GEMM (lowp_gemm) API - for example, ACL's lowp_gemm objects cache information like weights reduction or weights in optimized memory format which oneDNN does not allow due to its stateless nature.
Hence, ACL currently runs a (redundant) sum of columns and pre-transposition (to the gemm kerne's optimal format) for each GEMM operation.
This PR addresses the sub-optimalities above by integrating ACL directly with qlinear_dynamic. This approach yields an average speedup (averaged over context_lengths of 2^3 up to 2^9) of ~ 50% for bert-base-uncased, bert-large-uncased, roberta-base, distilbert-base-uncased with 16 threads on a Neoverse-V1 (with transformers==4.48).
To achieve this we introduce PackedLinearWeightsACL (as a subclasses of PackedLinearWeightsOnednn ) with an implementation of qlinear_dynamic that uses ACL directly, while qlinear still follows the oneDNN path.
@fadara01
Enable a fast path for (static) qlinear for AArch64 through ACL direc…
0b06ae1 
…tly.

This enables a fast path for eager mode static quantization for AArch64 through Arm Compute Library (ACL) directly.

PR pytorch#145942 addressed the high overhead in qlinear_dynamic on AArch64 (due to redundant weight pretranspositions and reductions) by enabling a path that calls ACL directly.
This does the same thing but for (static) qlinear.
@davsva01
Enable qint8 and quint8 add for AArch64 using ACL directly
a0d2046 
This enables qint8 and quint8 add for AArch64 through Arm Compute Library (ACL) directly.
It's based on changes in PR pytorch#145942 which enables the use of ACL directly in ATen.
Relative performance improvement using OMP_NUM_THREADS=1 is ~15x, using OMP_NUM_THREADS=32 it’s ~5.4x.
@fadara01 fadara01 mentioned this pull request Mar 5, 2025
Closed
fadara01 added a commit that referenced this pull request Mar 5, 2025
@fadara01
Enable Direct Use of Arm Compute Library (ACL) in ATen
d7108f2 
ACL is already built with PyTorch as a shared library when USE_MKLDNN_ACL is set.
Currently, it is only used indirectly in ATen via oneDNN for AArch64 targets. However there are cases where it makes sense to utilize ACL directly without  oneDNN as an intermediary - e.g. quantization. See #145942, #147337, #146620.
This patch enables such use cases by exposing ACL to ATen

ghstack-source-id: 266c621
Pull Request resolved: #148581
This was referenced Mar 5, 2025
Closed
Closed
pytorchmergebot pushed a commit that referenced this pull request Mar 10, 2025
@fadara01 @pytorchmergebot
Enable Direct Use of Arm Compute Library (ACL) in ATen (#148584)
d1f21d8 
ACL is already built with PyTorch as a shared library when USE_MKLDNN_ACL is set.
Currently, it is only used indirectly in ATen via oneDNN for AArch64 targets. However there are cases where it makes sense to utilize ACL directly without  oneDNN as an intermediary - e.g. quantization. See #145942, #147337, #146620.
This patch enables such use cases by exposing ACL to ATen

Pull Request resolved: #148584
Approved by: https://github.com/malfet
@fadara01
Copy link
Collaborator

Closing in favor of ghstack PR: #148653 which has all comments addressed

@fadara01 fadara01 closed this Mar 10, 2025
@fadara01 fadara01 mentioned this pull request Mar 10, 2025
Closed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers

@jeffdaily jeffdaily Awaiting requested review from jeffdaily

@jerryzh168 jerryzh168 Awaiting requested review from jerryzh168 jerryzh168 is a code owner

@salilsdesai salilsdesai Awaiting requested review from salilsdesai salilsdesai is a code owner

@kimishpatel kimishpatel Awaiting requested review from kimishpatel kimishpatel is a code owner

@digantdesai digantdesai Awaiting requested review from digantdesai digantdesai is a code owner

@jianyuh jianyuh Awaiting requested review from jianyuh jianyuh is a code owner

@malfet malfet Awaiting requested review from malfet

Assignees
No one assigned
Labels
arm priority module: cpu CPU specific problem (e.g., perf, algorithm) open source release notes: quantization release notes category release notes: releng release notes category triaged This issue has been looked at a team member, and triaged and prioritized into an appropriate module
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.
5 participants
@davsva01 @fadara01 @malfet @pytorchbot @mikaylagawarecki
0