Conversation
Signed-off-by: Nicholas Gates <nick@nickgates.com>
Signed-off-by: Nicholas Gates <nick@nickgates.com>
Signed-off-by: Nicholas Gates <nick@nickgates.com>
Contributor
Polar Signals Profiling ResultsLatest Run
Previous Runs (4)
Powered by Polar Signals Cloud |
Merging this PR will not alter performance
Comparing 
Footnotes
|
This comment was marked as off-topic.
This comment was marked as off-topic.
Contributor
Benchmarks: TPC-H SF=1 on NVMESummary
Detailed Results Table
|
Contributor
Benchmarks: FineWeb NVMeSummary
Detailed Results Table
|
Contributor
Benchmarks: TPC-H SF=1 on S3Summary
Detailed Results Table
|
Contributor
Benchmarks: TPC-DS SF=1 on NVMESummary
Detailed Results Table
|
Contributor
Benchmarks: TPC-H SF=10 on NVMESummary
Detailed Results Table
|
Contributor
Benchmarks: FineWeb S3Summary
Detailed Results Table
|
Contributor
Benchmarks: Statistical and Population GeneticsSummary
Detailed Results Table
|
Contributor
Benchmarks: TPC-H SF=10 on S3Summary
Detailed Results Table
|
Contributor
Benchmarks: Clickbench on NVMESummary
Detailed Results Table
|
Signed-off-by: Nicholas Gates <nick@nickgates.com>
Signed-off-by: Nicholas Gates <nick@nickgates.com>
Signed-off-by: Nicholas Gates <nick@nickgates.com>
Signed-off-by: Nicholas Gates <nick@nickgates.com>
Signed-off-by: Nicholas Gates <nick@nickgates.com>
lwwmanning
reviewed
Feb 24, 2026
2D77
| // This is used by duckdb whenever there is no projection id in a logical_get node. | ||
| // For some reason we cannot return an empty DataChunk and duckdb will look for the virtual column | ||
| // with this index and create a data chunk with a single vector of that type. | ||
| static EMPTY_COLUMN_IDX: u64 = 18446744073709551614; |
Contributor
There was a problem hiding this comment.
fun fact: if you cast this to i64, it's -1
lwwmanning
approved these changes
Feb 25, 2026
Contributor
|
@gatesn why did we remove the footer cache? |
Contributor
Author
|
We want a multi-file version of a Vortex scan that is entirely implemented in Vortex, doesn't rely on query engine features. This is so we can provide good support to e.g. PyVortex where users might just write a scan directly. So the MultiFileDataSource handles this internally now. |
fastio
pushed a commit
to fastio/vortex
that referenced
this pull request
Mar 10, 2026
This PR migrates the DuckDB implementation extension over to only using the Scan API. We create a generic DuckDB table function that operates over any DataSource, and then a specific implementation that takes a path glob and runs a multi-file scan as is the existing behavior. In the future we can easily add alternate table functions such as creating one from a DataSourceProvider URI, or by passing in a C pointer to a DataSource implementation. --------- Signed-off-by: Nicholas Gates <nick@nickgates.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
3 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
This PR migrates the DuckDB implementation extension over to only using the Scan API.
We create a generic DuckDB table function that operates over any DataSource, and then a specific implementation that takes a path glob and runs a multi-file scan as is the existing behavior.
In the future we can easily add alternate table functions such as creating one from a DataSourceProvider URI, or by passing in a C pointer to a DataSource implementation.