Chunk-local PatchedArray

[mhk197]

Summary

Moves the experimental Patched encoding onto the chunk-local layout from #9611 and gives it a new wire ID, vortex.patched_v2. This is the first step of the plan in #7818 to replace the interior Patches helper in BitPacked, ALP, ALP-RD and Sparse with one patch array.

One in-memory type, two wire IDs. Patched keeps its name and in-memory ID. The chunk-local layout is written under vortex.patched_v2; the lane-transposed layout of RFC 0027 stays readable under vortex.patched and is re-sorted into chunk order on load, so no other code path sees lanes. Per docs/specs/editions.md, a form an old reader does not understand needs a new ID, and CI wrote lane arrays under VORTEX_EXPERIMENTAL_PATCHED_ARRAY=1 before edition enforcement landed in #9660.

Layout

Metadata is n_patches and offset (grid position of row zero, below 1024). Children:

slot	child	dtype	length
0	inner	array dtype	len
1	patch_indices	u16 non-nullable, strictly increasing within a chunk	n_patches
2	patch_values	array dtype	n_patches
3	chunk_offsets	u32 non-nullable prefix counts with a terminator	(offset + len).div_ceil(1024) + 1

Chunk c owns patch_indices[chunk_offsets[c]..chunk_offsets[c + 1]]. Lookups select the chunk in constant time and binary search at most 1024 indices.

Slicing is zero-copy: the inner is sliced exactly, chunk_offsets is sliced by chunk, and the patch children are shared. Patches of the first and last chunk that fall outside the slice stay in the children and are skipped on read. compact() rebases when a self-contained array is wanted. This is what removes v1's offset_within_chunk bookkeeping: nothing is sliced at element granularity.

Changes

• arrays/patched/layout.rs: PatchedView over canonical slices (search, live, for_each), layout validation, and global-to-chunk-local conversion. Joe's PatchesV2 container reshaped as the array's internals.
• array.rs: new slots and metadata, Patched::try_new with full validation on canonical children, from_array_and_patches accepting any unsigned index width, slice_range, compact.
• Kernels ported without lane loops: execute, append_to_builder, scalar_at, slice and filter reduce rules, compare against a constant, take (binary search instead of a hash map).
• plugin.rs: PatchedPlugin writes vortex.patched_v2 and reads both IDs. vortex-file and the btrblocks trace tests register it instead of the bare vtable.
• testdata/lane_*.bin: three lane-layout arrays serialized by the pre-change code on develop, including one whose lane order reverses row order and one sliced mid-chunk with a dead patch. Regenerate from a pre-change checkout if they ever need to change.
• Tests: existing suite adapted, a model-based suite over 48 seeds (execute, scalar_at, nested slices, compaction), layout validation rejections, serde round trips including a slice with dead patches.

Nothing outside vortex-array/src/arrays/patched/ referenced the lane internals. Behaviour under VORTEX_EXPERIMENTAL_PATCHED_ARRAY is unchanged, including the existing problem that enforced editions drop the BitPacking and ALP schemes when it is set; the patched edition family and the compressor gating are the next PR.

Checks run

• cargo nextest run -p vortex-array (3458 passed)
• cargo nextest run -p vortex-fastlanes -p vortex-alp -E 'test(/plugin|patched/)' (8 passed)
• cargo nextest run -p vortex-btrblocks -E 'test(/trace/)' (6 passed)
• cargo check -p vortex-compat -p vortex-file --all-targets
• cargo clippy -p vortex-array -p vortex-file -p vortex-btrblocks --all-targets
• cargo test --doc -p vortex-array, RUSTDOCFLAGS=-D warnings cargo doc -p vortex-array --no-deps
• cargo +nightly fmt --all

Not run: workspace-wide build, GPU crates.

[codspeed-hq]

Merging this PR will improve performance by 14.49%

⚠️ Unknown Walltime execution environment detected

Using the Walltime instrument on standard Hosted Runners will lead to inconsistent data.

For the most accurate results, we recommend using CodSpeed Macro Runners: bare-metal machines fine-tuned for performance measurement consistency.

⚠️ Different runtime environments detected

Some benchmarks with significant performance changes were compared across different runtime environments,
which may affect the accuracy of the results.

Open the report in CodSpeed to investigate

⚡ 7 improved benchmarks
✅ 2174 untouched benchmarks
⏩ 206 skipped benchmarks¹

Performance Changes

	Mode	Benchmark	BASE	HEAD	Efficiency
⚡	WallTime	arrow_checked_add_u32_avx2[16384]	21.4 µs	17.6 µs	+21.49%
⚡	Simulation	decompress[u64, (4000, 1024)]	85.7 µs	70.7 µs	+21.27%
⚡	Simulation	allocate_drop_arrow[0]	456.9 ns	402.7 ns	+13.45%
⚡	Simulation	compact_sliced[(1024, 90)]	1.5 µs	1.4 µs	+12.03%
⚡	Simulation	allocate_drop_bytes[0]	520.2 ns	466 ns	+11.62%
⚡	Simulation	compact_sliced[(2048, 90)]	1.6 µs	1.4 µs	+11.53%
⚡	WallTime	mul_u32_nonnull_avx512	6.3 µs	5.7 µs	+10.63%

Tip

Curious why performance improved? Comment @codspeedbot explain why performance improved on this PR, or directly use the CodSpeed MCP with your agent.

---

_{Comparing mk/patched-chunk-local (66f93b0) with develop (dab1684)}

Footnotes

1. 206 benchmarks were skipped, so the baseline results were used instead. If they were deleted from the codebase, click here and archive them to remove them from the performance reports. ↩