feat(metastore): shard sections queries over index files #20134
Conversation
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
b8abe5e to
15e2ffa
Compare
623f0ab to
3b0f256
Compare
3b0f256 to
fb14c2f
Compare
Contributor
|
💻 Deploy preview available (feat(metastore): query metastore before building physical plan (POC)): |
fb14c2f to
78945e5
Compare
6 tasks
7fc0687 to
b0b6541
Compare
373565e to
27b42f4
Compare
b0b6541 to
5515873
Compare
- Build a distributed metastore plan from GetIndexes and execute it via the v2 scheduler/worker pipeline - Split the request into per-index PointersScan tasks, then fan-in and CollectSections to produce final section descriptors - Add physical plan + protobuf support for Merge/PointersScan, improve tracing/cleanup, and add coverage for planner/workflow/proto roundtrips feat(metastore): distributed metastore queries execution
71ab37a to
501dce4
Compare
benclive
approved these changes
Dec 19, 2025
Contributor
There was a problem hiding this comment.
Discussed offline: This LGTM, I'm happy with how it's implement and I'm willing to merge it to unblock other work and follow up on any minor issues later.
I left a couple of questions but neither are blocking.
| return translateEOFPipeline{pipeline} | ||
| } | ||
|
|
||
| type translateEOFPipeline struct { |
Contributor
There was a problem hiding this comment.
Does this need to be a pipeline? Could it be a standard function wrapping the error?
| Start: start, | ||
| End: end, | ||
|
|
||
| MaxTimeRange: TimeRange{ |
Contributor
There was a problem hiding this comment.
Why do we have two start/end ranges here?
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
What this PR does / why we need it:
Running metastore queriesin a distributed manner using query engine workers:
feat(metastore): distributed metastore queries execution
Special notes for your reviewer:
This PR is based on top of
Checklist
CONTRIBUTING.mdguide (required)featPRs are unlikely to be accepted unless a case can be made for the feature actually being a bug fix to existing behavior.docs/sources/setup/upgrade/_index.mddeprecated-config.yamlanddeleted-config.yamlfiles respectively in thetools/deprecated-config-checkerdirectory. Example PR