[Claptar]

Closes #97. Part of #87. For 0.17.6.

Reads and writes now line up with the source's chunks and Zarr v3 shards. Outputs keep the source's chunks and shards (clamped to the output) and are written a whole shard or chunk at a time through a new storage.RowWriter. A writer holds at most 8 MiB of rows per output, and 128 MiB across the outputs split writes at once. Reads take whole source chunks; subset, split and convert read column tiles when one chunk height across the matrix is over the read budget. Sparse subset/split blocks follow --chunk. Shard indexes are read once, under one 64 MiB budget for the process.

Findings

• F031 (dense subset/split of anndata 0.13's sharded Zarr v3 rewrote a shard per block): outputs keep the shard and are written a whole shard at a time. RowWriter fills one preallocated unit in place and writes whole units straight from the block. A unit is cut along its rows to 8 MiB, or to the output's share of 128 MiB in split.

• F032 (sparse blocks were 4,096 majors whatever --chunk said; a CSC block grew with n_obs): a block is about as many entries as --chunk rows hold on average (sparse_block_entries), in either layout. A major holding more, such as a gene in every cell, is read in parts (_sparse_blocks).

• F033 (outputs dropped the shard and kept the ~1 KB inner chunk; HDF5 outputs took it as their chunk): fit_chunks keeps shards, clamped with the chunk. _growable_like keeps the source's shard, clamped to the most entries the output can hold. An HDF5 target of a sharded source lets HDF5 choose its chunks. A v2 target takes the shard as its chunk.

• F036 (unverified; reproduced: recast --dtype 15.4×, --indices-dtype 9.7×, a transpose's output 7–13×, concat 15×): every streamed writer of an output that keeps a shard goes through RowWriter:

  • recast's cast_dense, cast_sparse, transpose pass 3, sparsify and densify;
  • concat's dense, sparse, CSC, varm and varp writers;
  • CsrAppender, copy_dataset and _copy_records.

  A sparse output's data and indices share one budget. A Zarr output whose source has no chunks takes zarr's own chunks, made explicit (backend_chunks) and cut like any other.

• F060 (split's outputs got --chunk/G rows a block into chunks many times taller): each output buffers rows to whole chunk heights, within its share of 256 MiB.

• F061 (row blocks unaligned to the source's chunk height): read_blocks rounds --chunk up to whole chunk heights, within max(32 MiB, --chunk rows). subset and split read column tiles of whole chunks past that, each output column band with its own writer. concat, recast (and its cast check) and export array/sparse read through row_bands, the same rounding with --chunk rows past the budget.

• F062 (shard index re-read on every block): Zarr stores opened for reading are wrapped in a WrapperStore that keeps suffix reads (shard indexes). They share one LRU budget of 64 MiB for the process, and a store's entries are dropped when it is collected.

• F079 (split RSS grew with group count): measured first. B6a's 1 MiB cache did not change it, and neither does any chunk-cache or metadata-cache setting: an open HDF5 file costs ~0.55 MB of HDF5's own per-file structures. split now writes at most 128 outputs per pass, in any format, so each output's share of the write budget stays at 1 MiB or more. HDF5 outputs open with no chunk cache, since RowWriter writes whole chunks. The cost: past 128 groups the matrices are read once per 128.

• F081 (copy_dataset read a whole chunk height across the full width, i.e. the whole matrix for column-chunked arrays): column tiles of whole chunks within TARGET_READ_BYTES. A shard is made to fit the tile, and whole rows are written through RowWriter.

Measurements (B6a's harness: bytes, never a clock)

"Encoded ×" is bytes of whole chunks or shards encoded per byte written to X; "decoded ×" is the same for reads. The zarr31 fixtures are anndata 0.13's layout as zarr 3.1 wrote it (inner (16, 8) or 245 elements, shards of the whole array; perf_fixtures.zarr31_layout). zarr 3.4.1 gives ~1 MB inner chunks, so a store written today hides these costs.

Ref	Scenario	main	this PR
F031	subset, dense zarr31 8192×64, --chunk 256	encoded 15.3× (33.7 MB for 2.2 MB); store fetched 4.8 MB	1.03×; 0.47 MB
F031	split into 8, same	encoded 121× (269 MB)	1.03×
F036	recast --dtype float64, dense zarr31	15.4× (67 MB for 4.4 MB)	1.0×
F036	recast --indices-dtype int64, CSR zarr31	9.7×	1.0×
F036	concat of two dense zarr31	15.4×	1.03×
F031/F036	peak allocation writing 1 MiB units: dense (32 MB shard) / CSR (8 MB of entries)	19.3 MB / 29.8 MB	under 10 MB (guard)
F031/F036	peak_rss subset dense zarr3 16384×2048 to Zarr, -C 64	1,727 MB	68 MB
F031/F036	peak_rss subset CSR zarr31 8M entries to Zarr	50 MB	58 MB
F036	peak_rss recast --indices-dtype int64, same CSR	122 MB	60 MB
F033	subset, CSR zarr31 20000×200: X/data	no shard, chunks of 245, 1,626 files	shard kept, 2 files
F033	convert zarr31 → h5ad: X/data chunks	245 (file 2.52 MB)	6,250, as anndata's own (2.05 MB)
F032	CSC subset, --chunk 64, 4,096 → 32,768 cells: largest read	26,214 → 209,715 entries	448 → 448
F032	memory grid, subset CSR/CSC × n_obs/n_var	strict xfail (B6a measured 0.53–5.2 of the input)	pass (≤ 0.5); GRID_GAPS now empty
F060	split gzip dense h5ad 4096×1024, chunks (512, 8), 8 groups	encoded 16× (268 MB for 16.8 MB)	1.0×
F061	subset of gzip chunks (n_obs, 8): 16,384 / 65,536 rows	decoded 15× / 60×	1.0× / 1.0×
F061	subset of zarr2's own chunks (8192, 16) / (32768, 8)	decoded 7.4× / 29×	1.0× / 1.0×
F061	concat / recast --dtype / recast --layout csr / export array, gzip chunks (8192, 8)	decoded 7.7× / 8.0× / 8.0× / 8.0×	≤ 1.25× (guard)
F062	subset, CSR zarr31 20k / 80k rows: fetched ÷ X on disk	1.22× / 1.71×	1.13× / 1.12×
F062	subset dense / concat CSR zarr31: fetched ÷ store	3.2× / 2.9×	≤ 1.3×
F079	split CSR 40000×500, peak RSS added, 8 / 128 / 512 groups	32 / 161 / 581 MB	23 / 143 / 143 MB
F081	convert (n, 8)-chunked h5ad, read budget 256 KiB: largest read	1 MB (the whole matrix)	256 KiB

Every new guard fails on main and passes here. I checked this by running the same test file with main's src on PYTHONPATH: 23 of 23 guards in test_performance.py fail on main, each by its assertion. The reviewer also reverted each fix in turn on this branch, and each guard failed (below).

Tests

• tests/test_performance.py, new section "blocks and shards":

  • each sharded output encoded once: subset, split, concat, recast --dtype / --indices-dtype / --layout csc (X's arrays, not the transpose's scratch), convert;
  • each source chunk decoded once: subset (tall gzip h5ad, zarr2 default), concat, recast --dtype, recast --layout csr, export array;
  • peak allocation a few units, not the shard: dense, CSR;
  • split writes each output chunk once;
  • each shard index fetched once: subset dense, concat CSR;
  • sparse blocks set by --chunk: CSR, CSC, a gene in every cell;
  • convert reads tiles, not a column-chunked matrix whole;
  • split RSS flat past 128 groups (slow, peak_rss; the runner now raises the open-file limit, so macOS's 256 does not cap the outputs first).

  The four F032 grid xfails are removed. perf_counters gains written_by_name / encoded_by_name; perf_fixtures gains zarr31_layout and matrix_store(groups=, compression=).

• tests/test_block_layout.py (new), with anndata as the oracle:

  • subset at --chunk 2 (CSC majors read in parts) on h5ad, zarr2, zarr3 and zarr31, in every layout;
  • split at a 512-byte write budget (units cut to a row or two);
  • tile mode with a 64-byte read budget, for subset, split and convert.

  Explicit layout checks (F033): chunks and shards equal the source's (clamped) after subset, split, concat and recast; a zarr31 → h5ad convert lets HDF5 choose its chunks and stores X in what anndata's own h5ad takes; v3 → v2 takes the shard as its chunk. Unit tests cover RowWriter (every backend, sharded, grow, cols, budget), read_blocks, a property test of _sparse_blocks (every kept major's entries exactly once, each block within budget), and the shard-index cache (one index fetch per array; none kept at a 0-byte budget; one budget across stores, dropped with them).

• Updated:

  • test_storage.py: the old test asserted that a clamped chunk drops its shard, which is F033.
  • test_subset_fan_out.py: chunk_major → chunk_rows, at 1 and 5.
  • B6a's F029 cast-check guard: the largest read is --chunk rows or one source chunk height, whichever is taller.
  • test_recast_streaming.py's F013 overflow test: its source is chunked 16 rows tall, so the refusal after the first block is still shown.

CI cost: the new tests take about 45 s on this machine: test_block_layout.py 27 s, the new guards 18 s (the RSS guard 5 s of that).

Local results

At head 8ddfb29:

• COLUMNS=80 uv run pytest -m "not integration" --cov=adata --cov-fail-under=90: 3,648 passed, 9 skipped, coverage 96.03%
• uv run pytest -m perf: 4 passed
• ADATA_REQUIRE_VERSION_FIXTURES=1 uv run pytest tests/test_anndata_versions.py -m integration: 268 passed, 12 skipped
• At the zarr floor (3.2.0, anndata 0.13.4): test_block_layout.py and the new guards pass (89 passed, 2 skipped).

Docs

• docs/reference/commands.md: under "Writing outputs", what outputs keep (shards, inner chunks) and when a unit is cut (8 MiB, about four times that while zarr encodes it; 128 MiB shared across split's outputs). Also how reads are sized (whole chunks up to 32 MiB, then column tiles in subset, split and convert), how sparse blocks follow --chunk, and that shard indexes are cached. In split, one pass per 128 outputs. subset's and split's --chunk help now reads "Rows per block, of a dense or a sparse matrix".
• docs/guides/pipelines.md (memory), docs/troubleshooting.md (memory, and "It is slow on a Zarr store anndata 0.13 wrote"), docs/reference/testing.md (harness fields, the zarr31 fixture, the new guards).
• AGENTS.md and docs/developers.md: shards are kept by fit_chunks; write streamed outputs through RowWriter and flush it.

Files outside the batch

• src/adata/core/recast.py: cast_dense, cast_sparse, transpose pass 3, sparsify and densify write through RowWriter (F036's repro is recast); the cast check, cast_dense and sparsify read through row_bands (F061).
• src/adata/core/concat.py: the dense, sparse, CSC, varm and varp writers write through RowWriter, the dense readers through row_bands; _layout_like passes itemsize; the growable writers are flushed.
• src/adata/core/csr_build.py: CsrAppender.finish flushes its growable writers (2 lines).
• src/adata/formats/array.py, src/adata/formats/mtx.py: export's dense reads go through row_bands (F061; a few lines each).
• src/adata/cli.py: the help text of subset's and split's --chunk.
• tests/test_recast_streaming.py: the F013 overflow test's source chunks (above).

These are mechanical. Each writer's slice assignment becomes writer.write(block), followed by a flush(). Each reader's range(0, n, chunk_rows) becomes row_bands(array, chunk_rows). _growable_like now returns a RowWriter, so every caller of it flushes.

Follow-ups (not in this PR)

• A transpose into a Zarr output keeps its bucket scratch inside the output, appended to unbuffered (csr_build.scratch_group), so recast --layout csc to Zarr still encodes ~135× its scratch writes. The output arrays themselves are written once now. An HDF5 scratch beside the output, as concat does, would fix it.
• copy_dataset writes HDF5 variable-length strings uncompressed: obs _index/values from a zarr → h5ad convert is 320 KB against anndata's 37 KB.
• concat, recast and export read --chunk rows, not column tiles, where one chunk height across the matrix is over the read budget (chunks of (n_obs, 8) at scale).

Changelog entry

Fixed

• Subset, split, concat, recast and convert write each Zarr v3 shard once (#97). anndata 0.13 writes Zarr v3 sharded, a whole array in one shard. Commands wrote outputs a --chunk block at a time, and zarr rewrites a whole shard for a write of any part of it: subset and concat wrote 15 times the bytes they meant to, split 121 times, and a dense subset held 1.7 GB. Outputs are written a whole shard (or chunk) at a time now. A shard is cut along its rows to at most 8 MiB, 128 MiB shared across the outputs split writes at once, and about four times that is held while zarr encodes it.
• Outputs keep a sharded source's shards and inner chunks (#97). A sparse matrix's data and indices dropped the shard and kept the inner chunk, 245 elements a file under anndata 0.13 with zarr 3.1: thousands of files per array. An h5ad output took the inner chunk as its HDF5 chunk, making a file twice anndata's own. Zarr v3 outputs keep the shard. HDF5 outputs let HDF5 choose their chunks, and --zarr-format 2 takes the shard as the chunk.
• Sparse subset and split follow --chunk (#97). Blocks were 4,096 rows or columns whatever --chunk said, so a CSC block (4,096 genes across every cell) grew with the number of cells. A block holds about as many entries as --chunk rows do now, and a gene expressed in every cell is read in parts.
• Reads follow the source's chunks (#97). subset, split, concat, recast, convert and export read a dense matrix in whole chunks of the source, so a chunk taller than --chunk is no longer decoded once for every block: 7 to 60 times the matrix for zarr's own chunks or chunks of (n_obs, 8). Past 32 MiB, subset, split and convert read a band of columns, so convert no longer reads a column-chunked matrix whole. A Zarr v3 shard's index is read once per command, not once per block.
• split memory no longer grows with the number of groups (#97). An open HDF5 file costs about 0.6 MB whatever is in it, and split held up to 1,024 open at once: 581 MB of RSS at 512 groups, against 143 MB now. It writes at most 128 outputs per pass over the matrices, so past 128 groups the matrices are read once per 128. Each output is written a whole chunk at a time, so a chunk 512 rows tall is no longer decoded and encoded once for every block.

Independent review

A fresh subagent with none of my context reviewed the diff against the audit entries and AGENTS.md. It reverted each fix in turn, compared outputs with anndata, and measured with the harness.

Verdict at 52d063d: not ready.

• Every output matched anndata: subset, split, concat, recast and convert, on h5ad (gzip, chunked), zarr2, zarr3 and the zarr31 layout, in every layout, at --chunk 1 / 7 / 1024. The same held with the unit, buffer and read budgets set to a few KB.
• Edge cases passed too: one row, one column, all-empty CSR, a gene in every cell, zero-width obsm, string obsm and uns, raw, obsp, varp, --inplace, and --zarr-format 2 from sharded v3.
• Every guard failed with its fix reverted.

What it found, and what changed (head now 8ddfb29):

1. Medium. Writing a unit cost 4–5× WRITE_UNIT_BYTES; sparse subset to Zarr regressed (335 MB vs main's 50 MB at 64 MiB units). Fixed.
   • RowWriter fills one preallocated unit buffer in place and writes whole units from the block itself.
   • The budget is 8 MiB (128 MiB across split's outputs), and data and indices share it everywhere.
   • The docs state the real multiplier (about four times a unit while zarr encodes it).
   • New guard test_writing_whole_units_holds_a_few_units_not_the_shard. The sparse case is now 58 MB vs main's 50 MB.
2. Medium. The shard-index cache was 64 MiB per store, and concat holds up to 64 inputs open. Fixed: one process-wide budget, entries keyed per store and dropped when it is collected. The duplicate-insert double count (its finding 10) is fixed too. New test test_the_shard_index_cache_is_one_budget_across_stores.
3. Medium. F061 was fixed only in subset/split/convert. Fixed: row_bands in concat, recast (cast check, cast_dense, sparsify) and export array / sparse. The F061 guard now covers concat, both recasts and export array; all fail on main at 7.7–8.0×. Column tiles stay subset/split/convert only, which is documented and listed as a follow-up.
4. Medium/low. recast --layout dense wrote --chunk rows straight in (13× on Zarr). Fixed: it writes through RowWriter, with zarr's own chunks made explicit and cut.
5. Low/medium. Tile mode cut the output's chunks against the full width. Fixed: no cut when one band covers every row (chunks of (n_obs, 8) keep their height).
6. Low/medium. With many outputs every unit was cut to a sliver (1 row at 1,000 outputs). Fixed: at most 128 outputs per pass in any format, so a share is at least 1 MiB.
7. Low. Growable outputs kept the source's shard length. Fixed: _growable_like(size=) clamps it to the most entries the output can hold, known from indptr.
8. Low. The F079 guard passed under macOS's 256-file limit with the fix reverted. Fixed: peak_rss's runner raises the soft limit. The guard now fails on main under ulimit -n 256 (536 vs 138 MB).
9. Low. Split past 128 groups reads the matrices once per 128. Documented, and now stated in the changelog entry.
10. Nit. Cache double count on a duplicate insert. Fixed (with 2).
11. Low, pre-existing. Zarr's own default chunks for a wide matrix were written several times. Fixed by item 4's explicit chunks, in copy_dataset and the dense fan-out too.

Not acted on: it noted that STATUS.md does not list the follow-ups. The coordinating session owns STATUS.md, per the plan's rules.

🤖 Generated with Claude Code

[chatgpt-codex-connector]

Codex Review Summary

This comment shows the latest Codex review activity on this pull request.

Review	Status	Commit	Review trigger
📝 Code Review	✅ Completed 2026-11-02T17:14:36.015988Z	52d063d	PR opened

ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

• Open a pull request for review
• Mark a draft as ready
• Comment "@codex review" or "@codex security review".

Codex reacts with 👀 while any review is running, comments if it has suggestions, and reacts with 👍 once all reviews finish with no findings.

[Copilot]

Copilot was unable to review this pull request because the user who requested the review has insufficient remaining quota. Try requesting a review at a lower effort level.

An organization owner can choose the repository organization as the billing source for reviews requested by members with a paid Copilot license. Learn about Copilot code review billing.

[github-actions]

Test Results (py3.13)

3 569 tests   - 103   3 557 ✅  - 101   10m 43s ⏱️ - 4m 24s
    1 suites ±  0      12 💤  -   2 
    1 files   ±  0       0 ❌ ±  0 

Results for commit 72c66c5. ± Comparison against base commit 417aa4a.

This pull request removes 197 and adds 94 tests. Note that renamed tests count towards both.

tests.test_cistopic ‑ test_writing_an_imputed_matrix_holds_a_block_not_every_feature_of_4096_cells[csr]
tests.test_cistopic ‑ test_writing_an_imputed_matrix_holds_a_block_not_every_feature_of_4096_cells[dense]
tests.test_performance ‑ test_10x_export_reads_grow_linearly
tests.test_performance ‑ test_10x_import_reads_grow_linearly[extra0]
tests.test_performance ‑ test_10x_import_reads_grow_linearly[extra1]
tests.test_performance ‑ test_10x_mtx_export_reads_each_block_once
tests.test_performance ‑ test_10x_mtx_import_work_grows_linearly[cell-extra0]
tests.test_performance ‑ test_10x_mtx_import_work_grows_linearly[cell-extra2]
tests.test_performance ‑ test_10x_mtx_import_work_grows_linearly[shuffled-extra1]
tests.test_performance ‑ test_10x_mtx_import_writes_in_blocks_not_per_barcode
…

tests.test_block_layout ‑ test_a_matrix_read_in_tiles_matches_anndata
tests.test_block_layout ‑ test_a_row_writer_does_not_hold_a_unit_over_its_budget[h5ad]
tests.test_block_layout ‑ test_a_row_writer_does_not_hold_a_unit_over_its_budget[zarr2]
tests.test_block_layout ‑ test_a_row_writer_does_not_hold_a_unit_over_its_budget[zarr3]
tests.test_block_layout ‑ test_a_row_writer_grows_an_array_and_writes_columns[h5ad]
tests.test_block_layout ‑ test_a_row_writer_grows_an_array_and_writes_columns[zarr2]
tests.test_block_layout ‑ test_a_row_writer_grows_an_array_and_writes_columns[zarr3]
tests.test_block_layout ‑ test_a_row_writer_writes_each_unit_once_whatever_the_blocks[h5ad-False]
tests.test_block_layout ‑ test_a_row_writer_writes_each_unit_once_whatever_the_blocks[h5ad-True]
tests.test_block_layout ‑ test_a_row_writer_writes_each_unit_once_whatever_the_blocks[zarr2-False]
…

This pull request removes 4 skipped tests and adds 2 skipped tests. Note that renamed tests count towards both.

tests.test_performance ‑ test_peak_memory_is_a_small_share_of_the_input[subset-csc-n_obs]
tests.test_performance ‑ test_peak_memory_is_a_small_share_of_the_input[subset-csc-n_var]
tests.test_performance ‑ test_peak_memory_is_a_small_share_of_the_input[subset-csr-n_obs]
tests.test_performance ‑ test_peak_memory_is_a_small_share_of_the_input[subset-csr-n_var]

tests.test_block_layout ‑ test_a_row_writer_writes_each_unit_once_whatever_the_blocks[h5ad-True]
tests.test_block_layout ‑ test_a_row_writer_writes_each_unit_once_whatever_the_blocks[zarr2-True]

♻️ This comment has been updated with latest results.

[github-actions]

Test Results (py3.12)

3 569 tests   - 103   3 557 ✅  - 101   8m 0s ⏱️ - 1m 17s
    1 suites ±  0      12 💤  -   2 
    1 files   ±  0       0 ❌ ±  0 

Results for commit 72c66c5. ± Comparison against base commit 417aa4a.

This pull request removes 197 and adds 94 tests. Note that renamed tests count towards both.

tests.test_cistopic ‑ test_writing_an_imputed_matrix_holds_a_block_not_every_feature_of_4096_cells[csr]
tests.test_cistopic ‑ test_writing_an_imputed_matrix_holds_a_block_not_every_feature_of_4096_cells[dense]
tests.test_performance ‑ test_10x_export_reads_grow_linearly
tests.test_performance ‑ test_10x_import_reads_grow_linearly[extra0]
tests.test_performance ‑ test_10x_import_reads_grow_linearly[extra1]
tests.test_performance ‑ test_10x_mtx_export_reads_each_block_once
tests.test_performance ‑ test_10x_mtx_import_work_grows_linearly[cell-extra0]
tests.test_performance ‑ test_10x_mtx_import_work_grows_linearly[cell-extra2]
tests.test_performance ‑ test_10x_mtx_import_work_grows_linearly[shuffled-extra1]
tests.test_performance ‑ test_10x_mtx_import_writes_in_blocks_not_per_barcode
…

tests.test_block_layout ‑ test_a_matrix_read_in_tiles_matches_anndata
tests.test_block_layout ‑ test_a_row_writer_does_not_hold_a_unit_over_its_budget[h5ad]
tests.test_block_layout ‑ test_a_row_writer_does_not_hold_a_unit_over_its_budget[zarr2]
tests.test_block_layout ‑ test_a_row_writer_does_not_hold_a_unit_over_its_budget[zarr3]
tests.test_block_layout ‑ test_a_row_writer_grows_an_array_and_writes_columns[h5ad]
tests.test_block_layout ‑ test_a_row_writer_grows_an_array_and_writes_columns[zarr2]
tests.test_block_layout ‑ test_a_row_writer_grows_an_array_and_writes_columns[zarr3]
tests.test_block_layout ‑ test_a_row_writer_writes_each_unit_once_whatever_the_blocks[h5ad-False]
tests.test_block_layout ‑ test_a_row_writer_writes_each_unit_once_whatever_the_blocks[h5ad-True]
tests.test_block_layout ‑ test_a_row_writer_writes_each_unit_once_whatever_the_blocks[zarr2-False]
…

This pull request removes 4 skipped tests and adds 2 skipped tests. Note that renamed tests count towards both.

tests.test_performance ‑ test_peak_memory_is_a_small_share_of_the_input[subset-csc-n_obs]
tests.test_performance ‑ test_peak_memory_is_a_small_share_of_the_input[subset-csc-n_var]
tests.test_performance ‑ test_peak_memory_is_a_small_share_of_the_input[subset-csr-n_obs]
tests.test_performance ‑ test_peak_memory_is_a_small_share_of_the_input[subset-csr-n_var]

tests.test_block_layout ‑ test_a_row_writer_writes_each_unit_once_whatever_the_blocks[h5ad-True]
tests.test_block_layout ‑ test_a_row_writer_writes_each_unit_once_whatever_the_blocks[zarr2-True]

♻️ This comment has been updated with latest results.