Releases: vortex-data/vortex
Releases · vortex-data/vortex
0.63.0
What's Changed
- Aggregates: accumulate columnar by @gatesn in #6839
- Add a UUID extension type by @connortsui20 in #6832
- move stuff around in
vortex-tensorby @connortsui20 in #6857 - chore[ci]: enable cuda build+lint by @joseph-isaacs in #6861
- Configure benchmark boxes to try and reduce noise by @AdamGS in #6745
- feat[gpu]: slice support for dyn dispatch by @0ax1 in #6862
- Extension Array Cleanup by @connortsui20 in #6869
- Reducing lock contention around expression partition caches by @AdamGS in #6871
- perf[runend]: run_end_bool decompress by @joseph-isaacs in #6229
- fix[ci]: sccache longer timout by @joseph-isaacs in #6875
- perf[buffer]: faster buffer iterator by @joseph-isaacs in #6876
- Fix fuzzer incorrect mask application by @myrrc in #6877
- chore[ci]: re-enable cuda tests by @joseph-isaacs in #6878
- Move fuzzer logging to debug by @myrrc in #6881
- fix[ci]: use correct protoc by @joseph-isaacs in #6884
- Use unaligned bit buffer iterators where possible by @robert3005 in #6880
- Change benchmark display order by @AdamGS in #6883
- chore: compile with C++ 20 by @0ax1 in #6889
- Remove ExtDTypeInner by @gatesn in #6874
- Remove ScalarFnInner by @gatesn in #6897
- chore: trim micro-bench input sizes by @0ax1 in #6890
Full Changelog: 0.62.0...0.63.0
Contributors
robert3005, 0ax1, and 5 other contributors
Assets 3
0.62.0
Immutable
release. Only release title and notes can be modified.
Changes
⚠️ Breaks
- feat: Iterative Execution (#6824) @joseph-isaacs
- Public array modules (#6834) @gatesn
- Layout readers don't need a mutable context (#6808) @robert3005
- Rename Array -> DynArray (#6760) @gatesn
- Add a
VortexSessionfor scalar value deserialization (#6772) @connortsui20
✨ Features
- Duckdb 1.5-variegata (#6627) @myrrc
- Aggregates: ChunkedArray kernel (#6837) @gatesn
- Vortex Fixed-Shape Tensor (#6812) @connortsui20
- feat[vortex-runend]: handle canonicalizing Ree (#6788) @asubiotto
- Aggregate Fns (#6721) @gatesn
- feat[vortex-array]: support converting arrow REE dtype to vortex (#6784) @asubiotto
- feat[expr]: N-ary CASE WHEN expression (#6786) @lukekim
- gpu scan cli convert subcommand (#6704) @onursatici
- Vortex TUI web (#6774) @gatesn
🚀 Performance
- gpu: transpose patches faster (#6829) @a10y
- perf: bypass async I/O pipeline for in-memory buffer reads (#6813) @0ax1
- Canonical::empty creates arrays directly instead of going through builders (#6805) @robert3005
- perf: faster sequence array decompress (#6809) @joseph-isaacs
- perf: ensure that
into_arrayis used for owned specific arrays (#6795) @joseph-isaacs - fix[ci]: run slow test in parallel (#6776) @joseph-isaacs
- data parallel patching for FastLanes (#6708) @a10y
- Fast path append_to_builder ConstantArray (#6767) @palaska
🐛 Bug Fixes
- duckdb: read from local path, correct err on invalid path (#6851) @myrrc
- use full column set on empty projection list in duckdb (#6831) @myrrc
- fix: don't commit to case when binary format (#6806) @joseph-isaacs
- fix[layout]: add required ALLOWED_ENCODINGS to
WriteStrategyBuilder(#6792) @joseph-isaacs - fix[layout]: canonical validity in compressor (#6797) @joseph-isaacs
- fix[ci]: duckdb download retry logic (#6791) @joseph-isaacs
- fix: cuda scan requires tokio (#6779) @joseph-isaacs
- fix: listview size & offset ptype mismatch (#6775) @0ax1
📖 Documentation
- add brief docs to the layouts (#6814) @connortsui20
🧰 Maintenance
24 changes
- Fix projection pushdown SLT test (#6850) @AdamGS
- Add TPC-H to sqllogictests (#6830) @AdamGS
- chore(deps): update datafusion monorepo to v52.2.0 (#6847) @renovate[bot]
- chore(deps): update taiki-e/cache-cargo-install-action digest to 414f3dc (#6845) @renovate[bot]
- chore(deps): update codspeedhq/action digest to 281164b (#6844) @renovate[bot]
- Use execute where possible (#6835) @gatesn
- AggregateFn naming (#6828) @gatesn
- chore[ci]: disable CUDA test (temp) codspeed new runner (#6826) @joseph-isaacs
- Engine specific SLT (#6822) @AdamGS
- Log which FS is used in DuckDB as debug (#6821) @AdamGS
- chore: switch over codspeed runner (#6819) @joseph-isaacs
- chore[ci]: use optional runs-on and also use runner by name (#6818) @joseph-isaacs
- ignore CUDA tests when cuda not available (#6796) @AdamGS
- Remove reuse lint check (#6810) @connortsui20
- perf: faster sequence array decompress iter (#6816) @joseph-isaacs
- chore[benchmark-website]: point to new benchmark bucket (#6801) @joseph-isaacs
- chore[ci]: smaller instances (#6807) @joseph-isaacs
- chore[ci]: smaller instances (#6803) @joseph-isaacs
- Branchless decimal sum (#6793) @AdamGS
- chore[ci]: Enable SPOT for cuda jobs (#6802) @joseph-isaacs
- chore[fuzz]: run issue closer every night (#6790) @joseph-isaacs
- chore: ci bump machine types (#6794) @joseph-isaacs
- chore: simplify bit-unpack CUDA generator (#6781) @0ax1
- chore: cast remove to_canonical (#6768) @joseph-isaacs
Contributors
lukekim, a10y, and 10 other contributors
Assets 3
0.61.0
Immutable
release. Only release title and notes can be modified.
Changes
⚠️ Breaks
- break: zip arg (mask, true, false) (#6766) @joseph-isaacs
- break: remove all compute public functions (#6752) @joseph-isaacs
- clean up numeric compute (#6593) @joseph-isaacs
- Remove deprecated to_arrow compute (#6730) @gatesn
- Remove &dyn Array call sites (#6725) @gatesn
- Make ExecutionArgs a dyn trait (#6715) @gatesn
- Inline visitor vtable to ArrayVTable (#6700) @gatesn
- Inline BaseArrayVTable (#6697) @gatesn
✨ Features
- rwlock instead of mutex around encoding ids (#6720) @danking
- Make the constant metadata a
Scalar(#6751) @connortsui20 - feat: remove to canonical p2 (#6742) @joseph-isaacs
- Improve benchmark output formatting (#6744) @AdamGS
- vortex read at for pooled cuda reads (#6675) @onursatici
🚀 Performance
- unsafe Scalar::struct_unchecked in scalar_at vtable (#6741) @palaska
- Implement BetweenReduceRule for ConstantArray (#6695) @robert3005
- Remove unnecessary to_canonical in DictArray::take (#6686) @palaska
- Eliminate redundant decompressions in VarBinArray canonicalization (#6692) @palaska
- Remove unnecessary to_canonical in ChunkedArray filter_indices (#6688) @palaska
- separate in_memory vs file coalescing config (#6690) @onursatici
- Remove to_canonical from MaskedArray::take (#6689) @palaska
🐛 Bug Fixes
- Make sequence metadata in-memory + clean up (#6759) @connortsui20
- fix: CUDA bitpacking bench build (#6750) @0ax1
- fix[util]: add balanced reduce iterator (#6737) @joseph-isaacs
- Fix S3 globbing for duckdbfs with httplib (#6709) @myrrc
- Fix old extension scalar logic (#6701) @connortsui20
📖 Documentation
🧰 Maintenance
22 changes
- chore: replace Buffer::copy_from with Buffer::from_iter for improved performance (#6749) @SYaoJun
- Remove unused BitView (#6763) @gatesn
- Remove sparse metadata prost wrapper (#6761) @connortsui20
- Parallelize slow CUDA tests (#6746) @joseph-isaacs
- Add explain feature to DF benchmarks (#6683) @AdamGS
- chore: build, test and lint all CUDA crates on CI (#6743) @0ax1
- test: Enhance tests for RLE and PCO casting: add assertions for dtype… (#6726) @SYaoJun
- chore: remove to_canonical types p1 (#6738) @joseph-isaacs
- chore(deps): update dependency ray to v2.54.0 [security] (#6633) @renovate[bot]
- fix(deps): update junit-framework monorepo to v6.0.3 (#6733) @renovate[bot]
- chore(deps): update crate-ci/typos action to v1.44.0 (#6734) @renovate[bot]
- fix(deps): update dependency ch.qos.logback:logback-classic to v1.5.32 (#6732) @renovate[bot]
- chore(deps): update rust crate moka to v0.12.14 (#6731) @renovate[bot]
- chore(deps): update all patch updates (#6728) @renovate[bot]
- chore(deps): update rust crate wasm-bindgen-futures to v0.4.64 (#6729) @renovate[bot]
- Some adjustments to methods on
ExtDTypeRef(#6716) @connortsui20 - chore: Update .clang-tidy to check all C++ source files (#6711) @SYaoJun
- chore: vortex-cuda cleanups (#6703) @0ax1
- Extension Scalar Cleanup (#6699) @connortsui20
- Update public-api (#6698) @gatesn
- Simplify ScalarFnPlugin (#6687) @gatesn
- Remove code for old benchmarks website (#6684) @AdamGS
Contributors
danking, robert3005, and 9 other contributors
Assets 3
0.60.0
Immutable
release. Only release title and notes can be modified.
0.60.0 largely involves unifying our vtable concepts as per the internal docs page in order to lay the groundwork for future extension points. Most notably: full Extension DType support, as well as AggregateFns.
Changes
⚠️ Breaks
- ScalarFn VTable (#6679) @gatesn
- Extension Scalar VTable (#6680) @connortsui20
- Rename spawn_blocking -> spawn_blocking_io (#6678) @gatesn
- Unify ExtVTable per vtable design doc (#6662) @gatesn
- Add an
extensionmodule tovortex(#6624) @connortsui20 - break: clean up compare (#6591) @joseph-isaacs
- Merge
vortex-dtypeintovortex-array(#6582) @connortsui20 - break: clean up compute (#6572) @joseph-isaacs
- Bump Rust MSRV to 1.90 (#6597) @robert3005
- break[array]: lazy list_contains (#6571) @joseph-isaacs
✨ Features
- Support building duckdb from a hash, including CI (#6658) @myrrc
- DuckDB to use the Scan API (#6652) @gatesn
- Use Scan API (#6391) @gatesn
- feat[gpu]: multi-stage dynamic dispatch with plan builder (#6636) @0ax1
- DuckDB lifetimes (#6620) @joseph-isaacs
- Expose table scan progress to DuckDB (#6596) @myrrc
- expose Executor, AbortHandle, AbortHandleRef, and Handle (#6595) @danking
🚀 Performance
- Avoid redundant chunk execution in pack_struct_chunks (#6682) @palaska
- read future to skip polled and dropped events for completed requests (#6667) @onursatici
- perf[gpu]: speed up dyn dispatch run-end kernel (#6665) @0ax1
- Route default append_to_builder through full execution loop (#6655) @palaska
- Add take-based rebuild strategy for ListView with small lists (#6492) @palaska
- oncelock shared array for zero cost access after materialisation (#6653) @onursatici
- oncelock shared array future on cuda flat layout (#6585) @onursatici
- cow buffers on array decode (#6590) @onursatici
- Extract to buffer once in list_contains (#6642) @palaska
- perf(layout): add empty-range fast paths in chunked reader (#6513) @caicancai
- chore[btrblocks]: more precise
FoRexpected_compression_ratio(#6301) @joseph-isaacs - perf: remove double execute in exprs (#6608) @joseph-isaacs
- Use approx footer size for caching instead of walking the layout tree (#6601) @robert3005
- do not double validate the flatbuffer on the cuda layout (#6583) @onursatici
- increase default nvme coalescing (#6586) @onursatici
- decoding stats should lock once (#6587) @onursatici
- perf[array]: chunked take with bounded compute graph (#6589) @joseph-isaacs
🐛 Bug Fixes
14 changes
- cuda ctx bind to thread before unsafe (#6681) @onursatici
- keep DropOnSync alive until after dispatching for cuda slices (#6673) @onursatici
- Convert &mut to & for DuckDB GlobalData (#6634) @gatesn
- resize FSST compressor buffer to be large enough for largest string (#6676) @a10y
- Revert "Support building duckdb from a hash, including CI" (#6670) @myrrc
- Close DuckDB database before re-opening (#6654) @gatesn
- More minimal listview rebuild benches, force canonicalize (#6629) @palaska
- only a read lock for current_array_ref (#6631) @danking
- fix: faster codspeed bench for
listview_rebuild(#6628) @joseph-isaacs - Balance expression trees during binary stat falsification (#6622) @AdamGS
- Fix compression benchmarks build (#6614) @AdamGS
- Fix scalar deserialisation where value was unsigned/signed for signed/unsigned type (#6604) @robert3005
- fix: chunked array take with nulls (#6603) @joseph-isaacs
- fix: fuzzer repeat issue tracking (#6577) @joseph-isaacs
📖 Documentation
4 changes
🧰 Maintenance
24 changes
- DuckDB: test curl impl, don't auto-load extensions (#6677) @myrrc
- chore: exclude cuda gen files from auto-formatting (#6671) @0ax1
- use match_each_canonical macro to reduce some canonical impl boilerplate (#6668) @AdamGS
- chore: trim unused crate deps with
cargo-machete(#6663) @0ax1 - cudarc use 12.5 instead of 12.8 (#6660) @a10y
- docs: resurrect lost ddb build.rs comment (#6659) @0ax1
- chore[gpu]: alp + for + bp CUDA benchmark (#6644) @0ax1
- chore:
clang-format- braces afteriforfor(#6643) @0ax1 - chore(deps): update codspeedhq/action digest to 2ac5728 (#6638) @renovate[bot]
- duckdb filesystem: don't use wrapper type (#6641) @myrrc
- use PublicBI fork with R2 hosted dataset (#6640) @a10y
- chore(deps): update all patch updates (#6639) @renovate[bot]
- Remove bad paths (#6626) @connortsui20
- Make the tui an default optional dependecy of vortex-python (#6625) @AdamGS
- refactor: simplify Buffer creation from Vec (#6621) @SYaoJun
- chore[ci]: explicitly use stable or night for all cargo commands (#6553) @joseph-isaacs
- Some nice to have things before extension types work (#6619) @connortsui20
- Use cache-cargo-install action instead of installing manually (#6616) @robert3005
- chore: delete commented out metadata tests (#6615) @vignesh-sankaran
- Add publish dry-run CI task (#6594) @robert3005
- More detailed backcompat comments (#6609) @connortsui20
- chore: rename device allocator to device read file (#6605) @0ax1
- chore: vortex-cuda only depend on vortex (#6602) @0ax1
- lower your expectations (#6579) @a10y
Contributors
danking, a10y, and 12 other contributors
Assets 3
0.59.4
Immutable
release. Only release title and notes can be modified.
Contributors
robert3005
Assets 3
C388
0.59.3
Immutable
release. Only release title and notes can be modified.
Changes
🐛 Bug Fixes
- minor c++ duckdb integration changes (#6566) @myrrc
- Vortex crate doesn't unconditionally depend on vortex-cuda-macros (#6580) @robert3005
Contributors
robert3005 and myrrc
Assets 3
0.59.2
Immutable
release. Only release title and notes can be modified.
Contributors
robert3005
Assets 3
Uh oh!
There was an error while loading. < 4838 a class="Link--inTextBlock" data-turbo="false" href="" aria-label="Please reload this page">Please reload this page.
0.59.1
Immutable
release. Only release title and notes can be modified.
0.59.0
Immutable
release. Only release title and notes can be modified.
Changes
- docs: tidy and unify Python quickstart examples (#5972) @caicancai
- Better
vx-benchdocs in README (#5900) @AdamGS
⚠️ Breaks
- Remove
vortex-compute(#6561) @connortsui20 - Introduce
VortexError::Otherand simplifyDisplayimplementation (#6410) @AdamGS - Move
vortex-scalarintovortex-array(#6523) @connortsui20 - break[array]: lazy between (#6528) @joseph-isaacs
- break[array]: mask compute to lazy mask execution (#6485) @joseph-isaacs
- break[array]: lazy compare function (#6495) @joseph-isaacs
- Remove
vortex-vector(#6529) @connortsui20 - break[array]: move like compute to lazy compute (#6497) @joseph-isaacs
- Pass buffers to
VTable::deserialize(#6473) @connortsui20 - break[array]: move cast traits to exprs (#6487) @joseph-isaacs
- Revert #6363 (#6476) @connortsui20
- Use
Scalardirectly for constant array metadata inlining (#6439) @connortsui20 - Fix
Scalarde/serialization with proto and outline nulls fromScalarValue(#6309) @connortsui20 - chore[array]: inline take compute function (#6376) @joseph-isaacs
- Add Array::deserialize arguments (#6340) @gatesn
- Update FSST to store components of compressed codes (#6325) @gatesn
- Array Execution (#6307) @gatesn
- Execute parent to return ArrayRef (#6257) @gatesn
- Extension DType vtables (#6081) @gatesn
- break[expr]: remove Expression.evaluate use Array.apply(expr) instead (#6167) @joseph-isaacs
- Make metrics more local to prevent unbounded memory usage (#6156) @AdamGS
- fallible scalar_at and validity (#6102) @joseph-isaacs
- Remove the python feature from vortex-error (#6064) @AdamGS
- break[*]: make array methods fallible (#6086) @joseph-isaacs
- chore[array]: replace to_canonical with to execute. (#6061) @joseph-isaacs
- Array VTable (#6000) @gatesn
- Remove duplicate ValidityVTable logic (#5975) @gatesn
- chore[array]: remove CanonicalVTable and move append_to_builder to main VTable (#6045) @joseph-isaacs
- Remove EncodeVTable (#6036) @gatesn
- chore: fallible canonical (#6006) @joseph-isaacs
- feat[array]: use execute to return
Canonicalarray notVector(#5895) @joseph-isaacs
✨ Features
- Add FileSystem trait and make DuckDB configurable (#6564) @gatesn
- Add scan concurrency config for datafusion source (#5945) @AdamGS
- feat: implement DuckDB filesystem integration for Vortex file handling (#6198) @iceboundrock
- vx convert to support cuda compatible vortex files (#6522) @onursatici
- cuda flat layout (#6521) @onursatici
- feat[xtask]: faster public-api (#6498) @joseph-isaacs
- Improve substrait support to include more types and expressions (#6484) @AdamGS
- feat[array]: refactor
zipcompute kernel to lazyzipvia ScalarFn Array (#6461) @joseph-isaacs - feat[gpu]: CUDA dynamic dispatch kernel (#6483) @0ax1
- feat[array]: lazy numeric (#6469) @joseph-isaacs
- feat[array]: move over to a expr based cast (#6404) @joseph-isaacs
- refactor[array]: move boolean ops to lazy binary expr (#6447) @joseph-isaacs
- Public api lockfiles (#6348) @gatesn
- feat[array]: refactor invert compute kernel to lazy
not(#6446) @joseph-isaacs - sqllogictest runner for DF and DuckDB on Vortex (#6273) @AdamGS
- feat[array]: move over fill_null to scalar_fn array making it lazy (#6406) @joseph-isaacs
- Upgrade Pco to 1.0.1 (#6354) @mwlon
- Add execution target that recursively canonicalises as well as canonical with canonical validity (#6380) @robert3005
- dict of strings kernel to support outlined views (#6401) @onursatici
- zstd buffers cuda kernel (#6394) @onursatici
- Make
VortexSource::newpublic (#6392) @AdamGS - Make projection pushdown in DF an opt-in feature (#6373) @AdamGS
- feat[expr]: new fill null expr (#6385) @joseph-isaacs
- zstd buffers array (#6362) @onursatici
- gpu compatible write strategy, move compact strategy to use btrblocks with zstd and pco (#6322) @onursatici
- cuda dictionary of vbv decompression (#6361) @0ax1
- CLI flag to enable tracing for benchmarks (#6350) @gatesn
- chore[array]: move take kernel to execute/reduce rules (#6310) @joseph-isaacs
- chore[cuda]: add date32 exporter test (#6345) @a10y
- feat: display tree with no metadata (#5898) @danking
- feat[cuda]: device exports for varbinview (#6338) @a10y
- vortex-metrics v2 (#6187) @AdamGS
- feat[array]: dict array allows signed codes (#6336) @joseph-isaacs
- feat[cuda]: export arrays to ArrowDeviceArray (#6253) @a10y
- feat[gpu]: CUDA
date_time_partskernel (#6303) @0ax1 - feat:
alternateforStructFields(#6276) @blaginin - Add a canonical matcher (#6292) @gatesn
- Move over to array matchers (#6291) @gatesn
- feat[gpu]: cuda runend decoding (#6275) @0ax1
- feat[btrblocks]: add a dynamic btr blocks compressor (#6207) @joseph-isaacs
- feat[cuda]: constant array for numeric types (#6248) @joseph-isaacs
- feat[array]: validate array encodings on write (#6241) @joseph-isaacs
- feat[cuda]: SequenceArray (#6232) @a10y
- feat[cuda]: patches kernel (#6231) @a10y
- Pass limit through Python Vortex (#6239) @gatesn
- Ship Vortex CLI as part of Python package (#6233) @gatesn
- Feature: Add more
Filter<_>implementations (#6173) @connortsui20 - Benchmarks website revamped (#6215) @palaska
- Dataset API (#5949) @gatesn
- feat[cuda]: filter for Decimal/VarBinView (#6196) @a10y
- feat[cuda]: add a cuda stream pool (#6205) @joseph-isaacs
- feat[python]: support full read/write from object storage (#6022) @a10y
- feat[scan]: gpu scan (#6199) @joseph-isaacs
- feat[gpu]: nvidia cub filter kernel (#6188) @0ax1
- feature: Add fastlanes bit unpacking cuda kernels (#6145) @robert3005
- Use built-in DF cache for file metadata and fix metrics wiring (#6164) @AdamGS
- feat: reduce size of segments containing fixed-size-list arrays (#6180) @danking
- Feature: Use
array.filter()instead offilter(array)compute (#6146) @connortsui20 - Rewrite benchmark website (#6160) @palaska
- feat: stats pruning for LIKE filters (#6049) @a10y
- feat: Zstd CUDA nvCOMP decompression (#6136) @0ax1
- feat[cuda]: canonical
to_host(#6130) @joseph-isaacs - feat[cuda]: dict decimal take (#6129) @joseph-isaacs
- feat[gpu]: dict take values prim (#6101) @joseph-isaacs
- chore[cuda]: decimal_byte_parts (#6118) @joseph-isaacs
- feat[gpu]: scalar encodings (#6109) @joseph-isaacs
- Add VortexScanReplacement to support reading from '*.vortex' (#6111) @robert3005
- feat:
nvcompwrapper crate (#6108) @0ax1 - Allow configuring different readers for DF integration (#6027) @AdamGS
- feat: introduce
CudaDeviceBuffer(#6060) @0ax1 - Array VTable (#6000) @gatesn
- feat[arary]: add a slice array to replace the slice op. (#5958) @joseph-isaacs
- Attempting to simplify the IO read traits (#5935) @AdamGS
- feat: CUDA compilation infra & FoR kernel (#5966) @0ax1
- feat[array]:
execute::<PrimitiveArray>()(#5954) @joseph-isaacs - Spark 4 (#5944) @a10y
- feat: cuda gpu skeleton (#5936) @0ax1
- Add flag to run benchmarks under Samply (#5940) @gatesn
- Datafusion 52 upgrade (#5863) @AdamGS
- Support precision casting for
DScalar(#5937) @AdamGS - feat[array]: use execute in canonical (#5925) @joseph-isaacs
- feat(vx): add JSON output, segments command, and SQL query support (#5642) @palaska
- feat[array]: use execute to return
Canonicalarray notVector(#5895) @joseph-isaacs - feature: TableStrategy (#5640) @a10y
🚀 Performance
- Faster primitive
take(#6559) @connortsui20 - perf[vortex-btrblocks]: remove listview expansion ratio (#6530) @asubiotto
- VarBinBuilder: eagerly set IsSorted stat on offsets (#6452) @dimitarvdimitrov
- Implement ReduceNode for Array to not require creating new arc when returning children (#6443) @robert3005
- chore: clean cancelled read requests earlier in driver state transitions (#6382) @caicancai
- shared array to avoid duplicate execute (#6453) @onursatici
- vortex-btrblocks: write listviews when elements expand > 50% (#6393) @asubiotto
- Implement ReduceNode for ScalarFnArray to avoid boxing (#6435) @robert3005
- Improve visitor performance by reducing
Vecand String allocations (#6337) @AdamGS - lazy dict layout values eval (#6423) @onursatici
- cuda strategy to use zstd buffers for strings (#6403) @onursatici
- bitunpacking cuda kernels store output into shared memory before copying to main memory (#6384) @robert3005
- chore:optimize next_coalesced() by tracking removed ids in a HashSet (#6353) @caicancai
- constant array metadata for small scalars (#6363) @onursatici
- chore[array]: move over filtering to
{reduce/execute}_parent(#6312) @joseph-isaacs - perf[buf]: inline
collect_bool(#6313) @joseph-isaacs - perf[vortex-array]: avoid expensive list view to list in arrow execution (#6304) @asubiotto
- chore: list contains over scalars (#6242) @blaginin
- perf[arrays]: move over all filter kernels to exec/reduce rules (#6298) @joseph-isaacs
- performance: Use push_unchecked when building indices for element take in fixedsizedlist take (#6222) @robert3005
- perf[vortex-array]: don't materialize const arrays in to_arrow_dictionary (#6265) @asubiotto
- Check read-only cache first (#6249) @gatesn
- Perf: Stream conversion from Parquet to Vortex to prevent OOM (#6228) @connortsui20
- Perf: Use optimized filter implementations is
FilterArray::execute(#6189) @connortsui20 - coalesced request align all segments (#6158) @onursatici
- perf: CUDA async to host copy (#6085) @0ax1
- feat: make host to device copy async (#6075) @0ax1
- Remove duplicate ValidityVTable logic (#5975) @gatesn
- performance[vortex-array]: avoid is_valid call in varbinview builder (#6033) @asubiotto
- Only check for ConstantVTable (#6002) @gatesn
- do not validate is_sorted on runend (#6025) @onursatici
- perf: CUDA FoR loop unrolling (#6017) @0ax1
- feat: optimize repeatedScan for limit == 0 (#6015) @caicancai
- Remove unnecessary work from
ListArray::validate(#6004) @AdamGS - scan: make LazyS...
Contributors
danking, iceboundrock, and 20 other contributors
Assets 3
0.59.0-rc.0
Immutable
release. Only release title and notes can be modified.
Changes
- docs: tidy and unify Python quickstart examples (#5972) @caicancai
- Better
vx-benchdocs in README (#5900) @AdamGS
⚠️ Breaks
- chore[array]: inline take compute function (#6376) @joseph-isaacs
- Add Array::deserialize arguments (#6340) @gatesn
- Update FSST to store components of compressed codes (#6325) @gatesn
- Array Execution (#6307) @gatesn
- Execute parent to return ArrayRef (#6257) @gatesn
- Extension DType vtables (#6081) @gatesn
- break[expr]: remove Expression.evaluate use Array.apply(expr) instead (#6167) @joseph-isaacs
- Make metrics more local to prevent unbounded memory usage (#6156) @AdamGS
- fallible scalar_at and validity (#6102) @joseph-isaacs
- Remove the python feature from vortex-error (#6064) @AdamGS
- break[*]: make array methods fallible (#6086) @joseph-isaacs
- chore[array]: replace to_canonical with to execute. (#6061) @joseph-isaacs
- Array VTable (#6000) @gatesn
- Remove duplicate ValidityVTable logic (#5975) @gatesn
- chore[array]: remove CanonicalVTable and move append_to_builder to main VTable (#6045) @joseph-isaacs
- Remove EncodeVTable (#6036) @gatesn
- chore: fallible canonical (#6006) @joseph-isaacs
- feat[array]: use execute to return
Canonicalarray notVector(#5895) @joseph-isaacs
✨ Features
- Make
VortexSource::newpublic (#6392) @AdamGS - Make projection pushdown in DF an opt-in feature (#6373) @AdamGS
- feat[expr]: new fill null expr (#6385) @joseph-isaacs
- zstd buffers array (#6362) @onursatici
- gpu compatible write strategy, move compact strategy to use btrblocks with zstd and pco (#6322) @onursatici
- cuda dictionary of vbv decompression (#6361) @0ax1
- CLI flag to enable tracing for benchmarks (#6350) @gatesn
- chore[array]: move take kernel to execute/reduce rules (#6310) @joseph-isaacs
- chore[cuda]: add date32 exporter test (#6345) @a10y
- feat: display tree with no metadata (#5898) @danking
- feat[cuda]: device exports for varbinview (#6338) @a10y
- vortex-metrics v2 (#6187) @AdamGS
- feat[array]: dict array allows signed codes (#6336) @joseph-isaacs
- feat[cuda]: export arrays to ArrowDeviceArray (#6253) @a10y
- feat[gpu]: CUDA
date_time_partskernel (#6303) @0ax1 - feat:
alternateforStructFields(#6276) @blaginin - Add a canonical matcher (#6292) @gatesn
- Move over to array matchers (#6291) @gatesn
- feat[gpu]: cuda runend decoding (#6275) @0ax1
- feat[btrblocks]: add a dynamic btr blocks compressor (#6207) @joseph-isaacs
- feat[cuda]: constant array for numeric types (#6248) @joseph-isaacs
- feat[array]: validate array encodings on write (#6241) @joseph-isaacs
- feat[cuda]: SequenceArray (#6232) @a10y
- feat[cuda]: patches kernel (#6231) @a10y
- Pass limit through Python Vortex (#6239) @gatesn
- Ship Vortex CLI as part of Python package (#6233) @gatesn
- Feature: Add more
Filter<_>implementations (#6173) @connortsui20 - Benchmarks website revamped (#6215) @palaska
- Dataset API (#5949) @gatesn
- feat[cuda]: filter for Decimal/VarBinView (#6196) @a10y
- feat[cuda]: add a cuda stream pool (#6205) @joseph-isaacs
- feat[python]: support full read/write from object storage (#6022) @a10y
- feat[scan]: gpu scan (#6199) @joseph-isaacs
- feat[gpu]: nvidia cub filter kernel (#6188) @0ax1
- feature: Add fastlanes bit unpacking cuda kernels (#6145) @robert3005
- Use built-in DF cache for file metadata and fix metrics wiring (#6164) @AdamGS
- feat: reduce size of segments containing fixed-size-list arrays (#6180) @danking
- Feature: Use
array.filter()instead offilter(array)compute (#6146) @connortsui20 - Rewrite benchmark website (#6160) @palaska
- feat: stats pruning for LIKE filters (#6049) @a10y
- feat: Zstd CUDA nvCOMP decompression (#6136) @0ax1
- feat[cuda]: canonical
to_host(#6130) @joseph-isaacs - feat[cuda]: dict decimal take (#6129) @joseph-isaacs
- feat[gpu]: dict take values prim (#6101) @joseph-isaacs
- chore[cuda]: decimal_byte_parts (#6118) @joseph-isaacs
- feat[gpu]: scalar encodings (#6109) @joseph-isaacs
- Add VortexScanReplacement to support reading from '*.vortex' (#6111) @robert3005
- feat:
nvcompwrapper crate (#6108) @0ax1 - Allow configuring different readers for DF integration (#6027) @AdamGS
- feat: introduce
CudaDeviceBuffer(#6060) @0ax1 - Array VTable (#6000) @gatesn
- feat[arary]: add a slice array to replace the slice op. (#5958) @joseph-isaacs
- Attempting to simplify the IO read traits (#5935) @AdamGS
- feat: CUDA compilation infra & FoR kernel (#5966) @0ax1
- feat[array]:
execute::<PrimitiveArray>()(#5954) @joseph-isaacs - Spark 4 (#5944) @a10y
- feat: cuda gpu skeleton (#5936) @0ax1
- Add flag to run benchmarks under Samply (#5940) @gatesn
- Datafusion 52 upgrade (#5863) @AdamGS
- Support precision casting for
DScalar(#5937) @AdamGS - feat[array]: use execute in canonical (#5925) @joseph-isaacs
- feat(vx): add JSON output, segments command, and SQL query support (#5642) @palaska
- feat[array]: use execute to return
Canonicalarray notVector(#5895) @joseph-isaacs - feature: TableStrategy (#5640) @a10y
🚀 Performance
- chore:optimize next_coalesced() by tracking removed ids in a HashSet (#6353) @caicancai
- constant array metadata for small scalars (#6363) @onursatici
- chore[array]: move over filtering to
{reduce/execute}_parent(#6312) @joseph-isaacs - perf[buf]: inline
collect_bool(#6313) @joseph-isaacs - perf[vortex-array]: avoid expensive list view to list in arrow execution (#6304) @asubiotto
- chore: list contains over scalars (#6242) @blaginin
- perf[arrays]: move over all filter kernels to exec/reduce rules (#6298) @joseph-isaacs
- performance: Use push_unchecked when building indices for element take in fixedsizedlist take (#6222) @robert3005
- perf[vortex-array]: don't materialize const arrays in to_arrow_dictionary (#6265) @asubiotto
- Check read-only cache first (#6249) @gatesn
- Perf: Stream conversion from Parquet to Vortex to prevent OOM (#6228) @connortsui20
- Perf: Use optimized filter implementations is
FilterArray::execute(#6189) @connortsui20 - coalesced request align all segments (#6158) @onursatici
- perf: CUDA async to host copy (#6085) @0ax1
- feat: make host to device copy async (#6075) @0ax1
- Remove duplicate ValidityVTable logic (#5975) @gatesn
- performance[vortex-array]: avoid is_valid call in varbinview builder (#6033) @asubiotto
- Only check for ConstantVTable (#6002) @gatesn
- do not validate is_sorted on runend (#6025) @onursatici
- perf: CUDA FoR loop unrolling (#6017) @0ax1
- feat: optimize repeatedScan for limit == 0 (#6015) @caicancai
- Remove unnecessary work from
ListArray::validate(#6004) @AdamGS - scan: make LazyScanStream initialization non-blocking (#5906) @godnight10061
- perf[sequence]: array constant compare execution kernel (#5795) @joseph-isaacs
- perf: reduce time_elapsed_opening overhead (#5879) @godnight10061
🐛 Bug Fixes
89 changes
- Measure output bytes in cuda bitunpack benchmarks (#6395) @robert3005
- fix[array]: decimal fill_null larger than buffer (#6388) @joseph-isaacs
- fix[array]: dict value nullable values (#6387) @joseph-isaacs
- fix[fuzz]: coverage report (#6375) @joseph-isaacs
- join ops before in format (#6370) @blaginin
- Include config.js in benchmark website docker image (#6379) @palaska
- Exporting struct array to arrow respects vortex nullabilty (#6365) @robert3005
- Fix null take on structs (#6364) @blaginin
- varbinview on device build skips validation (#6358) @onursatici
- fix: vortex-layout doc tests require zstd (#6179) @danking
- DF: Only calculate the size of the metadata cache entry once (#6342) @AdamGS
- bump time version to clear CI (#6335) @onursatici
- All-invalid compressor (#6330) @gatesn
- fix: Empty select (#6314) @gatesn
- Avoid some potential stack overflow cases with deep expression trees (#6321) @AdamGS
- shared array in dict layout (#6305) @onursatici
- skip sparse array validation for device resident arrays (#6315) @onursatici
- varbin to be constructed from buffer handle directly (#6317) @onursatici
- Do not use display_tree in error message (#6251) @gatesn
- Minor memory-related improvements in vortex-array and vortex-duckdb (#6299) @AdamGS
- use is_host to recursively check residency to skip validation (#6297) @onursatici
- Add slice kernel for BitPackedArray execute_parent (#6283) @onursatici
selectpreserve validity of fields (#6260) @connortsui20- fsst array: AllTrue and AllFalse are handled by filter fn (#6272) @blaginin
- fix: separate ptx build profiles (#6252) @0ax1
- fix[cuda]: Fix gpu_scan test (#6208) @a10y
- [benchmark-website] Fix mobile charts, better colors, better tooltip (#6226) @palaska
- fix: exclude flaky benchmarks from codspeed (#6220) @0ax1
- fix: skip buffer validation if on device (#6219) @0ax1
- fix: simd build (#6217) @0ax1
- chore[cuda]: fix windows (#6206) @joseph-isaacs
- Fix: dont do lazy filters on write (#6194) @connortsui20
- fix: evaluate OR for validity (#6184) @blaginin
- fix: cuda ptx folder (#6170) @joseph-isaacs
- Fix: Comment on a failed benchmark (#6181) @connortsui20
- fix[alp]: cast didn't cast patches (#6174) @joseph-isaacs
- fix[fuzz]: fix gpu fuzzer cfg flags (#6165) @joseph-isaacs
- fix[gpu]: synchronous copy to host (#6157) @0ax1
- fix[fuzz]: single sesssion for the fuzzer (#6155) @joseph-isaacs
- fix: sample multiples of 1024 (#6151) @danking
- fix: only link nvcomp for linux (#6144) @0ax1
- fix: fastlanes canonicalize benchmarks (#6141) @0ax1
- fix[fsst]: execute_parent empty validity non empty filter (#6133) @joseph-isaacs
- fix[vortex-array]: fix filter of sliced list (#6113) @asubiotto
- fix[list]: filter list execute parent (#6119) @blaginin
- fix[dtype]: fix typo in conversion from
dtypetodecimaltype(#6121) @joseph-isaacs - fix[bitpacking]: filter (#6120) @joseph-isaacs
- fix[vortex-...
Contributors
danking, a10y, and 14 other contributors