8000 HADOOP-19472: [ABFS] Improve write workload performance for ABFS by efficient concurrency utilization by anmolanmol1234 · Pull Request #7669 · apache/hadoop · GitHub
[go: up one dir, main page]
Skip to content

HADOOP-19472: [ABFS] Improve write workload performance for ABFS by efficient concurrency utilization #7669

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom
Draft
wants to merge 14 commits into
base: trunk
Choose a base branch
from
Draft

HADOOP-19472: [ABFS] Improve write workload performance for ABFS by efficient concurrency utilization #7669

wants to merge 14 commits into from
+598 −14

Conversation

anmolanmol1234
Copy link
Contributor

Enhance the performance of ABFS Driver for write-heavy workloads by improving concurrency within writes.

{05B55BCA-EF1F-496D-B1ED-17DCD394DDA1}

The proposed design advocates for a centralized WriteThreadPoolSizeManager class to handle the collective thread allocation required for all write operations across the system, replacing the current CachedThreadPool in AzureBlobFileSystemStore. This centralized approach ensures that the initial thread pool size is set at 4 * number of available processors and dynamically adjusts the pool size based on the system's current CPU utilization. This adaptive scaling and descaling mechanism optimizes resource usage and responsiveness. Moreover, this shared thread pool is accessible and utilized by all output streams, streamlining resource management and promoting efficient concurrency across write operations.

@anmolanmol1234
Copy link
Contributor Author

============================================================
HNS-OAuth-DFS

[WARNING] Tests run: 174, Failures: 0, Errors: 0, Skipped: 4
[WARNING] Tests run: 798, Failures: 0, Errors: 0, Skipped: 164
[WARNING] Tests run: 155, Failures: 0, Errors: 0, Skipped: 5
[WARNING] Tests run: 272, Failures: 0, Errors: 0, Skipped: 23

============================================================
HNS-SharedKey-DFS

[WARNING] Tests run: 174, Failures: 0, Errors: 0, Skipped: 5
[WARNING] Tests run: 801, Failures: 0, Errors: 0, Skipped: 117
[ERROR] Tests run: 146, Failures: 0, Errors: 0, Skipped: 5
[WARNING] Tests run: 272, Failures: 0, Errors: 0, Skipped: 10

============================================================
NonHNS-SharedKey-DFS

[WARNING] Tests run: 174, Failures: 0, Errors: 0, Skipped: 11
[WARNING] Tests run: 640, Failures: 0, Errors: 0, Skipped: 215
[WARNING] Tests run: 155, Failures: 0, Errors: 0, Skipped: 6
[WARNING] Tests run: 272, Failures: 0, Errors: 0, Skipped: 11

============================================================
AppendBlob-HNS-OAuth-DFS

[WARNING] Tests run: 174, Failures: 0, Errors: 0, Skipped: 4
[WARNING] Tests run: 798, Failures: 0, Errors: 0, Skipped: 171
[WARNING] Tests run: 132, Failures: 0, Errors: 0, Skipped: 6
[WARNING] Tests run: 272, Failures: 0, Errors: 0, Skipped: 23

============================================================
NonHNS-SharedKey-Blob

[WARNING] Tests run: 174, Failures: 0, Errors: 0, Skipped: 11
[WARNING] Tests run: 643, Failures: 0, Errors: 0, Skipped: 144
[WARNING] Tests run: 155, Failures: 0, Errors: 0, Skipped: 3
[WARNING] Tests run: 272, Failures: 0, Errors: 0, Skipped: 11

============================================================
NonHNS-OAuth-DFS

[WARNING] Tests run: 174, Failures: 0, Errors: 0, Skipped: 11
[WARNING] Tests run: 637, Failures: 0, Errors: 0, Skipped: 217
[WARNING] Tests run: 155, Failures: 0, Errors: 0, Skipped: 6
[WARNING] Tests run: 272, Failures: 0, Errors: 0, Skipped: 24

============================================================
NonHNS-OAuth-Blob

[WARNING] Tests run: 174, Failures: 0, Errors: 0, Skipped: 11
[ERROR] Tests run: 640, Failures: 0, Errors: 0, Skipped: 146
[WARNING] Tests run: 155, Failures: 0, Errors: 0, Skipped: 3
[WARNING] Tests run: 272, Failures: 0, Errors: 0, Skipped: 24

============================================================
AppendBlob-NonHNS-OAuth-Blob

[WARNING] Tests run: 174, Failures: 0, Errors: 0, Skipped: 11
[ERROR] Tests run: 638, Failures: 0, Errors: 0, Skipped: 164
[WARNING] Tests run: 132, Failures: 0, Errors: 0, Skipped: 4
[WARNING] Tests run: 272, Failures: 0, Errors: 0, Skipped: 24

============================================================
HNS-Oauth-DFS-IngressBlob

[WARNING] Tests run: 174, Failures: 0, Errors: 0, Skipped: 4
[ERROR] Tests run: 672, Failures: 0, Errors: 0, Skipped: 167
[WARNING] Tests run: 155, Failures: 0, Errors: 0, Skipped: 5
[WARNING] Tests run: 272, Failures: 0, Errors: 0, Skipped: 23

============================================================
NonHNS-OAuth-DFS-IngressBlob

[WARNING] Tests run: 174, Failures: 0, Errors: 0, Skipped: 11
[WARNING] Tests run: 637, Failures: 0, Errors: 0, Skipped: 215
[WARNING] Tests run: 155, Failures: 0, Errors: 0, Skipped: 6
[WARNING] Tests run: 272, Failures: 0, Errors: 0, Skipped: 24

@hadoop-yetus
Copy link

💔 -1 overall

Vote Subsystem Runtime Logfile Comment
+0 🆗 reexec 0m 21s Docker mode activated.
_ Prechecks _
+1 💚 dupname 0m 0s No case conflicting files found.
+0 🆗 codespell 0m 0s codespell was not available.
+0 🆗 detsecrets 0m 0s detect-secrets was not available.
+1 💚 @author 0m 0s The patch does not contain any @author tags.
+1 💚 test4tests 0m 0s The patch appears to include 1 new or modified test files.
_ trunk Compile Tests _
+1 💚 mvninstall 26m 3s trunk passed
+1 💚 compile 0m 23s trunk passed with JDK Ubuntu-11.0.26+4-post-Ubuntu-1ubuntu120.04
+1 💚 compile 0m 20s trunk passed with JDK Private Build-1.8.0_442-8u442-b06us1-0ubuntu120.04-b06
+1 💚 checkstyle 0m 21s trunk passed
+1 💚 mvnsite 0m 29s trunk passed
+1 💚 javadoc 0m 29s trunk passed with JDK Ubuntu-11.0.26+4-post-Ubuntu-1ubuntu120.04
+1 💚 javadoc 0m 22s trunk passed with JDK Private Build-1.8.0_442-8u442-b06us1-0ubuntu120.04-b06
+1 💚 spotbugs 0m 45s trunk passed
+1 💚 shadedclient 21m 19s branch has no errors when building and testing our client artifacts.
-0 ⚠️ patch 21m 31s Used diff version of patch file. Binary files and potentially other changes not applied. Please rebase and squash commits if necessary.
_ Patch Compile Tests _
+1 💚 mvninstall 0m 20s the patch passed
+1 💚 compile 0m 19s the patch passed with JDK Ubuntu-11.0.26+4-post-Ubuntu-1ubuntu120.04
+1 💚 javac 0m 19s the patch passed
+1 💚 compile 0m 16s the patch passed with JDK Private Build-1.8.0_442-8u442-b06us1-0ubuntu120.04-b06
+1 💚 javac 0m 16s the patch passed
+1 💚 blanks 0m 0s The patch has no blanks issues.
-0 ⚠️ checkstyle 0m 13s /results-checkstyle-hadoop-tools_hadoop-azure.txt hadoop-tools/hadoop-azure: The patch generated 14 new + 2 unchanged - 0 fixed = 16 total (was 2)
+1 💚 mvnsite 0m 20s the patch passed
+1 💚 javadoc 0m 18s the patch passed with JDK Ubuntu-11.0.26+4-post-Ubuntu-1ubuntu120.04
+1 💚 javadoc 0m 17s the patch passed with JDK Private Build-1.8.0_442-8u442-b06us1-0ubuntu120.04-b06
-1 ❌ spotbugs 0m 47s /new-spotbugs-hadoop-tools_hadoop-azure.html hadoop-tools/hadoop-azure generated 1 new + 0 unchanged - 0 fixed = 1 total (was 0)
+1 💚 shadedclient 20m 18s patch has no errors when building and testing our client artifacts.
_ Other Tests _
+1 💚 unit 2m 19s hadoop-azure in the patch passed.
+1 💚 asflicense 0m 23s The patch does not generate ASF License warnings.
77m 36s
Reason Tests
SpotBugs module:hadoop-tools/hadoop-azure
org.apache.hadoop.fs.azurebfs.WriteThreadPoolSizeManager.adjustThreadPoolSizeBasedOnCPU(double) does not release lock on all exception paths At WriteThreadPoolSizeManager.java:on all exception paths At WriteThreadPoolSizeManager.java:[line 268]
Subsystem Report/Notes
Docker ClientAPI=1.49 ServerAPI=1.49 base: https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-7669/1/artifact/out/Dockerfile
GITHUB PR #7669
JIRA Issue HADOOP-19472
Optional Tests dupname asflicense compile javac javadoc mvninstall mvnsite unit shadedclient spotbugs checkstyle codespell detsecrets
uname Linux a1000f66baec 5.15.0-136-generic #147-Ubuntu SMP Sat Mar 15 15:53:30 UTC 2025 x86_64 x86_64 x86_64 GNU/Linux
Build tool maven
Personality dev-support/bin/hadoop.sh
git revision trunk / 1ba12f6
Default Java Private Build-1.8.0_442-8u442-b06us1-0ubuntu120.04-b06
Multi-JDK versions /usr/lib/jvm/java-11-openjdk-amd64:Ubuntu-11.0.26+4-post-Ubuntu-1ubuntu120.04 /usr/lib/jvm/java-8-openjdk-amd64:Private Build-1.8.0_442-8u442-b06us1-0ubuntu120.04-b06
Test Results https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-7669/1/testReport/
Max. process+thread count 545 (vs. ulimit of 5500)
modules C: hadoop-tools/hadoop-azure U: hadoop-tools/hadoop-azure
Console output https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-7669/1/console
versions git=2.25.1 maven=3.6.3 spotbugs=4.2.2
Powered by Apache Yetus 0.14.0 https://yetus.apache.org

This message was automatically generated.

@anmolanmol1234 anmolanmol1234 changed the title HADOOP-19472: [ABFS][Improve write workload performance for ABFS] HADOOP-19472: [ABFS] Improve write workload performance for ABFS by efficient concurrency utilization May 7, 2025
@hadoop-yetus
Copy link

💔 -1 overall

Vote Subsystem Runtime Logfile Comment
+0 🆗 reexec 0m 39s Docker mode activated.
_ Prechecks _
+1 💚 dupname 0m 0s No case conflicting files found.
+0 🆗 codespell 0m 1s codespell was not available.
+0 🆗 detsecrets 0m 1s detect-secrets was not available.
+1 💚 @author 0m 0s The patch does not contain any @author tags.
+1 💚 test4tests 0m 0s The patch appears to include 1 new or modified test files.
_ trunk Compile Tests _
+1 💚 mvninstall 24m 39s trunk passed
+1 💚 compile 0m 23s trunk passed with JDK Ubuntu-11.0.27+6-post-Ubuntu-0ubuntu120.04
+1 💚 compile 0m 20s trunk passed with JDK Private Build-1.8.0_452-8u452-gaus1-0ubuntu120.04-b09
+1 💚 checkstyle 0m 18s trunk passed
+1 💚 mvnsite 0m 22s trunk passed
+1 💚 javadoc 0m 24s trunk passed with JDK Ubuntu-11.0.27+6-post-Ubuntu-0ubuntu120.04
+1 💚 javadoc 0m 20s trunk passed with JDK Private Build-1.8.0_452-8u452-gaus1-0ubuntu120.04-b09
+1 💚 spotbugs 0m 41s trunk passed
+1 💚 shadedclient 22m 18s branch has no errors when building and testing our client artifacts.
-0 ⚠️ patch 22m 30s Used diff version of patch file. Binary files and potentially other changes not applied. Please rebase and squash commits if necessary.
_ Patch Compile Tests _
+1 💚 mvninstall 0m 17s the patch passed
+1 💚 compile 0m 18s the patch passed with JDK Ubuntu-11.0.27+6-post-Ubuntu-0ubuntu120.04
+1 💚 javac 0m 18s the patch passed
+1 💚 compile 0m 16s the patch passed with JDK Private Build-1.8.0_452-8u452-gaus1-0ubuntu120.04-b09
+1 💚 javac 0m 16s the patch passed
+1 💚 blanks 0m 0s The patch has no blanks issues.
+1 💚 checkstyle 0m 11s the patch passed
+1 💚 mvnsite 0m 19s the patch passed
+1 💚 javadoc 0m 16s the patch passed with JDK Ubuntu-11.0.27+6-post-Ubuntu-0ubuntu120.04
+1 💚 javadoc 0m 15s the patch passed with JDK Private Build-1.8.0_452-8u452-gaus1-0ubuntu120.04-b09
-1 ❌ spotbugs 0m 40s /new-spotbugs-hadoop-tools_hadoop-azure.html hadoop-tools/hadoop-azure generated 1 new + 0 unchanged - 0 fixed = 1 total (was 0)
+1 💚 shadedclient 22m 6s patch has no errors when building and testing our client artifacts.
_ Other Tests _
+1 💚 unit 2m 20s hadoop-azure in the patch passed.
+1 💚 asflicense 0m 25s The patch does not generate ASF License warnings.
78m 44s
Reason Tests
SpotBugs module:hadoop-tools/hadoop-azure
org.apache.hadoop.fs.azurebfs.WriteThreadPoolSizeManager.adjustThreadPoolSizeBasedOnCPU(double) does not release lock on all exception paths At WriteThreadPoolSizeManager.java:on all exception paths At WriteThreadPoolSizeManager.java:[line 263]
Subsystem Report/Notes
Docker ClientAPI=1.49 ServerAPI=1.49 base: https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-7669/2/artifact/out/Dockerfile
GITHUB PR #7669
JIRA Issue HADOOP-19472
Optional Tests dupname asflicense compile javac javadoc mvninstall mvnsite unit shadedclient spotbugs checkstyle codespell detsecrets
uname Linux 3b526ffc404b 5.15.0-136-generic #147-Ubuntu SMP Sat Mar 15 15:53:30 UTC 2025 x86_64 x86_64 x86_64 GNU/Linux
Build tool maven
Personality dev-support/bin/hadoop.sh
git revision trunk / 8fe0842
Default Java Private Build-1.8.0_452-8u452-gaus1-0ubuntu120.04-b09
Multi-JDK versions /usr/lib/jvm/java-11-openjdk-amd64:Ubuntu-11.0.27+6-post-Ubuntu-0ubuntu120.04 /usr/lib/jvm/java-8-openjdk-amd64:Private Build-1.8.0_452-8u452-gaus1-0ubuntu120.04-b09
Test Results https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-7669/2/testReport/
Max. process+thread count 558 (vs. ulimit of 5500)
modules C: hadoop-tools/hadoop-azure U: hadoop-tools/hadoop-azure
Console output https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-7669/2/console
versions git=2.25.1 maven=3.6.3 spotbugs=4.2.2
Powered by Apache Yetus 0.14.0 https://yetus.apache.org

This message was automatically generated.

@hadoop-yetus
Copy link

🎊 +1 overall

Vote Subsystem Runtime Logfile Comment
+0 🆗 reexec 0m 21s Docker mode activated.
_ Prechecks _
+1 💚 dupname 0m 0s No case conflicting files found.
+0 🆗 codespell 0m 1s codespell was not available.
+0 🆗 detsecrets 0m 1s detect-secrets was not available.
+1 💚 @author 0m 0s The patch does not contain any @author tags.
+1 💚 test4tests 0m 0s The patch appears to include 1 new or modified test files.
_ trunk Compile Tests _
+1 💚 mvninstall 23m 7s trunk passed
+1 💚 compile 0m 24s trunk passed with JDK Ubuntu-11.0.27+6-post-Ubuntu-0ubuntu120.04
+1 💚 compile 0m 19s trunk passed with JDK Private Build-1.8.0_452-8u452-gaus1-0ubuntu120.04-b09
+1 💚 checkstyle 0m 23s trunk passed
+1 💚 mvnsite 0m 27s trunk passed
+1 💚 javadoc 0m 28s trunk passed with JDK Ubuntu-11.0.27+6-post-Ubuntu-0ubuntu120.04
+1 💚 javadoc 0m 22s trunk passed with JDK Private Build-1.8.0_452-8u452-gaus1-0ubuntu120.04-b09
+1 💚 spotbugs 0m 46s trunk passed
+1 💚 shadedclient 20m 50s branch has no errors when building and testing our client artifacts.
-0 ⚠️ patch 21m 2s Used diff version of patch file. Binary files and potentially other changes not applied. Please rebase and squash commits if necessary.
_ Patch Compile Tests _
+1 💚 mvninstall 0m 20s the patch passed
+1 💚 compile 0m 18s the patch passed with JDK Ubuntu-11.0.27+6-post-Ubuntu-0ubuntu120.04
+1 💚 javac 0m 18s the patch passed
+1 💚 compile 0m 19s the patch passed with JDK Private Build-1.8.0_452-8u452-gaus1-0ubuntu120.04-b09
+1 💚 javac 0m 19s the patch passed
+1 💚 blanks 0m 0s The patch has no blanks issues.
+1 💚 checkstyle 0m 13s the patch passed
+1 💚 mvnsite 0m 22s the patch passed
+1 💚 javadoc 0m 19s the patch passed with JDK Ubuntu-11.0.27+6-post-Ubuntu-0ubuntu120.04
+1 💚 javadoc 0m 19s the patch passed with JDK Private Build-1.8.0_452-8u452-gaus1-0ubuntu120.04-b09
+1 💚 spotbugs 0m 44s the patch passed
+1 💚 shadedclient 20m 29s patch has no errors when building and testing our client artifacts.
_ Other Tests _
+1 💚 unit 2m 17s hadoop-azure in the patch passed.
+1 💚 asflicense 0m 25s The patch does not generate ASF License warnings.
74m 18s
Subsystem Report/Notes
Docker ClientAPI=1.49 ServerAPI=1.49 base: https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-7669/3/artifact/out/Dockerfile
GITHUB PR #7669
JIRA Issue HADOOP-19472
Optional Tests dupname asflicense compile javac javadoc mvninstall mvnsite unit shadedclient spotbugs checkstyle codespell detsecrets
uname Linux 3ee980188ead 5.15.0-136-generic #147-Ubuntu SMP Sat Mar 15 15:53:30 UTC 2025 x86_64 x86_64 x86_64 GNU/Linux
Build tool maven
Personality dev-support/bin/hadoop.sh
git revision trunk / c56bdcb
Default Java Private Build-1.8.0_452-8u452-gaus1-0ubuntu120.04-b09
Multi-JDK versions /usr/lib/jvm/java-11-openjdk-amd64:Ubuntu-11.0.27+6-post-Ubuntu-0ubuntu120.04 /usr/lib/jvm/java-8-openjdk-amd64:Private Build-1.8.0_452-8u452-gaus1-0ubuntu120.04-b09
Test Results https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-7669/3/testReport/
Max. process+thread count 555 (vs. ulimit of 5500)
modules C: hadoop-tools/hadoop-azure U: hadoop-tools/hadoop-azure
Console output https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-7669/3/console
versions git=2.25.1 maven=3.6.3 spotbugs=4.2.2
Powered by Apache Yetus 0.14.0 https://yetus.apache.org

This message was automatically generated.

@anmolanmol1234 anmolanmol1234 marked this pull request as draft May 8, 2025 07:48
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers
No reviews
Assignees
No one assigned
Labels
ABFS TOOLS trunk
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.
2 participants
@anmolanmol1234 @hadoop-yetus
0