shared array to avoid duplicate execute#6453
Merged
onursatici merged 3 commits intodevelopfrom Feb 12, 2026
Merged
Conversation
Signed-off-by: Onur Satici <onur@spiraldb.com>
robert3005
approved these changes
Feb 12, 2026
Contributor
Polar Signals Profiling ResultsLatest Run
Powered by Polar Signals Cloud |
Contributor
Benchmarks: PolarSignals ProfilingSummary
Detailed Results Table
|
Contributor
Benchmarks: TPC-H SF=1 on NVMESummary
Detailed Results Table
|
Contributor
Benchmarks: FineWeb NVMeSummary
Detailed Results Table
|
Contributor
Benchmarks: TPC-H SF=1 on S3Summary
Detailed Results Table
|
Contributor
Benchmarks: TPC-DS SF=1 on NVMESummary
Detailed Results Table
|
Contributor
Benchmarks: TPC-H SF=10 on NVMESummary
Detailed Results Table
|
Contributor
Benchmarks: FineWeb S3Summary
Detailed Results Table
|
Contributor
Benchmarks: Random AccessSummary
Detailed Results Table
|
Contributor
Benchmarks: Statistical and Population GeneticsSummary
Detailed Results Table
|
Contributor
Benchmarks: TPC-H SF=10 on S3Summary
Detailed Results Table
|
Contributor
Benchmarks: Clickbench on NVMESummary
Detailed Results Table
|
Contributor
Benchmarks: CompressionSummary
Detailed Results Table
|
danking
added a commit
that referenced
this pull request
Feb 20, 2026
## Summary SharedArray uses a Mutex as of #6453. This interacts particularly badly with `scalar_at` by way of `is_valid`. In particular, all calls to `scalar_at` on the given array are serialized. It seems reasonable to allow multiple threads to simultaneously read the validity. This PR permits that by using an RwLock. ## Testing I have a Vortex-based system which recently pulled in a version of Vortex that includes #6453. With that version, my profile looks like this: https://share.firefox.dev/4tO94Ki <img width="1920" height="1080" alt="Screenshot 2026-02-20 at 4 26 28 PM" src="https://github.com/user-attachments/assets/77f24c44-268e-4dd8-8bc7-ec2c193443b6" /> With this PR applied (i.e. with an RwLock in place of the Mutex), it looks like this: https://share.firefox.dev/4aNPZzt <img width="1920" height="1080" alt="Screenshot 2026-02-20 at 4 27 25 PM" src="https://github.com/user-attachments/assets/dcf2fde3-e35a-413b-a6e6-2b4053e05653" /> CPU utilization is high, and runtime is ~7s. By comparison, with a Mutex, CPU utilization is low and runtime is ~18s. Signed-off-by: Daniel King <dan@spiraldb.com>
fastio
pushed a commit
to fastio/vortex
that referenced
this pull request
Mar 10, 2026
## Does this PR closes an open issue or discussion? <!-- This helps us keep track of fixed issues and changes. --> - Closes #. ## What changes are included in this PR? SharedArray execute both on the host and on cuda to prevent multiple in flight execute. chose `async-lock` because it already is in the lockfile, has both blocking and async api for cuda without tying tokio to vortex-array <!-- What changes are included here, if an issue or discussion are attached, there's no need to duplicate the details. --> ## What is the rationale for this change? <!-- Why do you propose this change, and why did you choose this approach. This helps reviewers and other readers understand changes, creates a shared understanding of the issue and codebase, and improves their ability to work with this change and offer better suggestions. --> ## How is this change tested? <!-- Changes should be tested, we expect changes to fit in one of the following categories: 1. Verifying existing behavior is maintained. 2. For serialization related changes - Compatibility should be maintained or explicitly broken. 3. For new behavior and functionality, this helps us maintaining that desired behavior in the future. --> ## Are there any user-facing changes? <!-- Does the change affect users in what of the following ways: 1. Breaks public APIs in some way. 2. Changes the underlying behavior of one of the integrations. 3. Should some documentation be changed to reflect this change? In the case some public API is changed in a breaking way, make sure to add the appropriate label. --> --------- Signed-off-by: Onur Satici <onur@spiraldb.com>
fastio
pushed a commit
to fastio/vortex
that referenced
this pull request
436E
Mar 10, 2026
## Summary SharedArray uses a Mutex as of vortex-data#6453. This interacts particularly badly with `scalar_at` by way of `is_valid`. In particular, all calls to `scalar_at` on the given array are serialized. It seems reasonable to allow multiple threads to simultaneously read the validity. This PR permits that by using an RwLock. ## Testing I have a Vortex-based system which recently pulled in a version of Vortex that includes vortex-data#6453. With that version, my profile looks like this: https://share.firefox.dev/4tO94Ki <img width="1920" height="1080" alt="Screenshot 2026-02-20 at 4 26 28 PM" src="https://github.com/user-attachments/assets/77f24c44-268e-4dd8-8bc7-ec2c193443b6" /> With this PR applied (i.e. with an RwLock in place of the Mutex), it looks like this: https://share.firefox.dev/4aNPZzt <img width="1920" height="1080" alt="Screenshot 2026-02-20 at 4 27 25 PM" src="https://github.com/user-attachments/assets/dcf2fde3-e35a-413b-a6e6-2b4053e05653" /> CPU utilization is high, and runtime is ~7s. By comparison, with a Mutex, CPU utilization is low and runtime is ~18s. Signed-off-by: Daniel King <dan@spiraldb.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Does this PR closes an open issue or discussion?
What changes are included in this PR?
SharedArray execute both on the host and on cuda to prevent multiple in flight execute. chose
async-lockbecause it already is in the lockfile, has both blocking and async api for cuda without tying tokio to vortex-arrayWhat is the rationale for this change?
How is this change tested?
Are there any user-facing changes?