[Claptar]

Closes #96. Part of #87. For 0.17.6: merge after 0.17.5 is tagged.

The perf suite now sees memory that --chunk does not bound: it counts bytes (returned, decoded in whole chunks, re-encoded in whole chunks or shards), measures resident memory in a subprocess, builds stores in every layout and format (including anndata 0.13's default sharded Zarr v3), and grows n_var as well as n_obs. Recast was then brought within it. Each guard was run against main's src/ first and failed there.

Findings

Ref	What changed	How it is tested
F091	test_peak_memory_is_a_small_share_of_the_input: recast (every dtype and layout change), subset and concat over CSR, CSC and dense, growing n_obs and then n_var, asserting peak memory added per byte of input added (assert_memory_is_a_fraction_of_the_input, at most half: bounded paths reach 0.21, holding the matrix 1 or more). --chunk is 256 rows growing n_obs (a block of columns over every row shows plainly) and 32 growing n_var (a row block widens with the input). Each store is built once and shared by the commands. Sparse subset is a strict xfail for F032 (B6b); concat passes on main and on B3b's branch.	On the pre-fix code, fails for recast's dense --dtype (both axes), CSC densify and dense to CSC.
F146	count_io counts read_bytes, largest_read (overall and per dataset), decoded_bytes, written_bytes, encoded_bytes (whole shards for sharded Zarr) and store bytes; hooks oindex, Zarr writes and create_array(data=). The chunk model follows HDF5's per-dataset cache (whole chunks into the cache; in place only for an unfiltered chunk too big for it) and Zarr's lack of one. peak_rss(argv) runs a command in a fresh interpreter and reads its own high-water mark: VmHWM on Linux, where ru_maxrss starts at the parent's peak (the kernel folds the replaced memory map into it at exec), and ru_maxrss on macOS. tests/perf_fixtures.py builds h5ad, Zarr v2 and sharded Zarr v3 stores with anndata, any layout and shape.	test_the_byte_counters_see_whole_chunks_decoded_and_rewritten (numbers worked by hand), test_peak_rss_sees_a_command_that_holds_the_matrix (--in-memory must show).
F029	check_cast reads --chunk rows of a dense matrix (rows=), not chunk rows, which was the whole matrix.	Largest read independent of n_obs on h5ad and sharded Zarr v3; peak RSS in a subprocess as a share of a 8-64 MB input; the grid. All fail on main.
F030	CSC densify goes through the streaming transpose to a CSR scratch and then dense row blocks; dense-to-CSC writes a plain CSR scratch in row blocks and transposes it, the output taking the dense source's codecs. Free space is checked again once dense-to-CSC knows its nnz. Neither holds n_obs x --chunk any more, and neither reads or writes narrow column slabs (each of which decoded every chunk of a row-chunked source, or rewrote every chunk of the dense output). Same route the loom exporter already takes.	Largest read, and memory added per row against a block of --chunk columns per row, on h5ad (fail on main); a gzip source's scratch is not rewritten per block and the output keeps gzip; every layout to every other in blocks of 3 rows / 7 entries against scipy and anndata, on new_store's three backends.
F077	The transpose's pass 2 steps through the nonzeros chunk entries at a time (_majors_of), splitting a major across blocks. Pass 3 gives any output major larger than a bucket a bucket of its own and streams it in chunks without sorting (pass 2 adds entries in input order, so it is already in minor order).	Skewed matrices (8 adjacent genes in every cell), CSC to CSR on h5ad and sharded Zarr v3, CSR to CSC on h5ad: no read above two buckets of entries (fail on main: the dense genes were read whole). Plus empty majors at both ends and majors larger than the chunk against scipy.
F078	sparsify builds data/indices with _growable_from: the dense template's codecs, never its 2-D chunks, so a 4-row chunk height no longer becomes a 4-element chunk. Covers the 10x-h5 export, which calls sparsify.	Output chunks at least 4,096 elements from a source chunked 4 rows tall, on all backends (fails on main).
F013	The index range check always runs for a sparse output; --force no longer waives it. Coordinates are checked against the output's minor axis. A dense source has its coordinates checked before writing and its nonzero count while sparsifying, refused as soon as it passes the range, CSC scratch included (check_index_range). The checks that need no read run before the cast check. --indices-dtype is parsed against INDEX_DTYPES.	int16 standing in for int32 (2^31 nonzeros cannot be built in a test): with --force, refused, nothing written; dense coordinates refused up front; dense nnz overflow refused while writing, before the whole matrix is read, and staging leaves nothing; a CSR with more rows than int16 holds keeps int16 column indices; offsets that fit are written narrow. On main, --force wrote int16 indptr ending at -5,536 and dense-to-CSR wrapped coordinates to -32,768.
F058	float16 is refused for a sparse output (CSR, CSC, --all over sparse layers), with "use --dtype float32, or --layout dense".	CLI refusals on every backend; every allowed dtype x layout read back by anndata and compared with scipy.
F121	sparsify casts each block before finding its nonzeros, so a lossy cast from a dense matrix drops the entries it zeroes, as scipy does. A sparse source keeps its stored entries, as scipy's astype does.	Dense float to int32 CSR/CSC: no explicit zeros, nnz equal to scipy's.
F043	_write descends to any depth (_write_members), so uns/neighbors/connectivities and uns/a/b/c are converted, not copied.	Nested paths at two and three levels plus obsm/X_pca, siblings untouched, read back by anndata (fails on main for the nested ones).

Also, because the harness showed it: h5py 3.15.x leaks 32 to 160 bytes on every dataset read, write and .shape (48 MB of resident memory after 300,000 writes; 3.13, 3.14 and 3.16.0 do not). Every streaming loop grew with its block count, the transpose (whose bucket writes are quadratic in buckets) fastest. The floor is now h5py>=3.16.0 (pyproject and uv.lock), guarded by test_h5py_calls_keep_no_memory_once_they_return. h5py 3.16 brings HDF5 2.0, whose chunk cache is 8 MiB a dataset rather than 1 MiB, so open_store now opens HDF5 with rdcc_nbytes of 1 MiB (HDF5_CHUNK_CACHE): split into 8 groups of a 200,000 x 2,000 CSR took 530 MB of RSS without it, 431 MB with it, and 429 MB on main with h5py 3.15.1. Its Linux wheels are manylinux_2_28 (glibc 2.28 or later). The free-space estimate for recast also counts the transpose's scratch (_scratch).

Not reproducing: none. All ten reproduced on main.

Files outside the batch

• pyproject.toml, uv.lock: the h5py floor. B5 (B5: Element fidelity, legacy stores and HDF5 safety #117) also changes both (zarr); on merge, take main's lock and re-run uv lock --upgrade-package h5py.
• docs/reference/testing.md, docs/reference/commands.md: the harness and recast's refusals.
• src/adata/storage/__init__.py: HDF5_CHUNK_CACHE and rdcc_nbytes on the h5py.File(path, mode) line of open_store, for HDF5 2.0's cache (above). B5 (B5: Element fidelity, legacy stores and HDF5 safety #117) adds lines next to it.
• Nothing in core/subset.py, core/csr_build.py or conversion/.

Follow-ups (not in this batch)

• The streaming transpose, and everything on BucketSorter, writes (nnz/chunk)^2 x 3 bucket appends, since every input block touches every bucket: 126k appends and 36 s at 838k nnz and a 4,096-entry chunk. A two-level scatter would make it about (nnz/chunk)^1.5.
• Recast on anndata's default sharded Zarr v3 rewrites a whole shard for every block it writes (537 MB encoded for 4.4 MB written, recast --dtype of 8192 x 64 at --chunk 64). That is F036's class, B6b's; the new encoded_bytes counter measures it.
• BucketSorter keeps three growable datasets a bucket (about 57 KB of memory and one 65,536-entry chunk of disk each, B3b measured). Through the CLI recast's sparse chunk is 1M entries, about 0.06 bytes an entry; at the API's 4,096-entry floor about 14. The transpose knows its row sizes, so fixed regions with cursors, as concat's _ColumnBuckets does, would remove it.
• concat's _csc_bounds/_csc_runs (B3b) and recast's _transpose_bounds/_bucket_runs isolate and stream oversized majors the same way; they could share one helper in csr_build.
• Zarr appends at a tiny --chunk rewrite their 65,536-element chunk each time (zarr2 dense 20,000 x 64 to CSR at --chunk 16: 108 MB written for a 9.8 MB store). Buffering appends to whole chunks would fix it; F036's class.

Merge notes

• Merged with main after wave 2 (B3b, B4, B5, B8a, B8b, B9). Conflicts: sparsify keeps this branch's _sparsify_rows, which already has B3b's numpy per-row counts (B3b's sparsify guard passes: 10 bytes a row against 30); open_store takes B5's content check and this branch's 1 MiB chunk cache; the lock keeps B5's zarr 3.4.1 with h5py 3.16.0. The F071 bool-readback xfail is gone, since B5 fixed it.
• B6b drops the four subset entries of GRID_GAPS when F032 is fixed.

Tests

• COLUMNS=80 uv run pytest -m "not integration" --cov=adata --cov-fail-under=90 on the merge with wave 2: 3,545 passed, 7 skipped, 4 xfailed (the strict F032 subset cases); coverage 96.1%.
• uv run pytest -m perf: 4 passed.
• CI time. The unit job's own duration varies a lot between runs (main at 610c5b1 took 12 and 16 minutes on its two legs), so this is measured from the JUnit results instead:
  • At 0a47406, this PR's tests took 340 s of test time on CI (py3.12).
  • At a3a3f7d, they took 169 s. That run's job took 14 to 15 minutes, on a main without wave 2.
  • Since then, the CSC per-row guard and the skewed CSR case run on h5ad only. Their Zarr v3 cases took 47 s on CI and measured the algorithm, not the store. That brings this PR to about 120 s on CI, on top of main's 899 s at 610c5b1.
  • What remains: the grid (35 s), the dtype x layout readback over three backends (25 s), and the CSC per-row guard (about 13 s). Each size of that guard is measured twice, because a one-time 4 MB growth once landed in a single measurement.
• CI at 0a47406 failed on Linux in test_peak_rss_sees_a_command_that_holds_the_matrix. On Linux ru_maxrss starts at the parent pytest's peak, so every command measured 0 MB and the RSS guards were vacuous. The fix reads VmHWM, and CI at a3a3f7d passed all 11 checks with it.

Independent review

A fresh subagent with the diff, the audit entries and AGENTS.md (no other context) reverted each fix and ran the guards, checked recast against scipy over 153 conversions and 384 transposes of non-canonical input on all three backends, and read the new paths for memory that --chunk does not bound. Every reverted fix failed its guards (F029: 9 guards; F030 densify: 4; F030 sparsify: 3; F077 pass 2: 3; F077 pass 3: 3; F078: 6; F013 12, F058 21, F121 6, F043 6 tests). It found no blocker in recast. What it raised, and what was done:

#	Finding	Outcome
1	h5py 3.16 brings HDF5 2.0, whose chunk cache is 8 MiB a dataset: split memory grew with outputs	Fixed: open_store keeps 1 MiB (530 MB -> 431 MB, main 429 MB), guarded
2	Dense to CSC copied a gzip source's codec onto a scratch with no cache: 30-100x slower at small --chunk	Fixed: plain scratch, the output takes the source's codecs; guarded on encoded bytes and on the output's gzip
3	The free-space check did not count dense-to-CSC's scratch; --in-memory CSC densify's CSR scratch neither	Fixed: checked again once nnz is known; _scratch counts the in-memory case
4	The grid's sparse n_var cases passed with about 0.1 MB of slack	Fixed: chunk 32,768 entries and density 0.1 (half the per-bucket share), each size measured twice
5	Zarr appends at a tiny --chunk rewrite their 65,536-element chunk	Follow-up (above)
6	Docs and changelog over-stated F121 for sparse sources	Fixed: dense sources only, said so
7	The chunk model read unfiltered chunks in place; stale text	Fixed: whole chunks into the cache unless too big for it; self-test updated
8	The full-read cast check ran before cheap refusals	Fixed: float16, index range and growth first
9	Dense to CSC refused an nnz overflow only after the whole scratch	Fixed: refused as soon as the count passes the output's range; guarded on bytes read
10	Largest-read guards had no lower bound	Fixed
11	Coordinates were checked against the larger dimension	Fixed: the output's minor axis; tested both ways
12	h5py 3.16's wheels need glibc 2.28; .shape leaks 32 B, not 100+	Changelog and wording updated
13	STATUS.md not updated	Not done: this batch's brief says not to edit STATUS.md
14	cast_dense to bool fails on h5ad	F071, B5's (#117); not this PR

Each new test was run against the previous head and failed there.

Changelog entry

Fixed

• recast --dtype checks a dense matrix a block at a time (#96). The round-trip check read the whole matrix at once, and two more copies of it, whatever --chunk said. It reads --chunk rows now, as the conversion does.
• Densifying a CSC matrix, and writing a dense one as CSC, hold --chunk rows (#96). Each block was --chunk columns of every row, so memory followed the number of cells: 8 GB a block at 2M cells. Both now go through a CSR in scratch and the streaming transpose.
• The streaming transpose reads about a chunk at a time on skewed matrices (#96). It stepped by the average row, so a run of genes expressed in every cell was read many chunks at once, and a gene expressed in every cell was sorted whole. It steps by entries now, and streams such a gene from a bucket of its own.
• A sparse matrix made from a dense one keeps element-sized chunks (#96). recast --layout csr/csc and the 10x-h5 export gave data and indices the dense source's chunk height as their chunk, down to 16 elements: hundreds of thousands of chunks and a file three times the size.
• --force no longer skips the index range check (#96). Given to accept a lossy cast, it also let indptr overflow --indices-dtype and wrap negative, a corrupt matrix with exit 0. A dense matrix made sparse is range-checked too.
• float16 is refused for a sparse matrix (#96). scipy has no float16 sparse matrix, so the store recast wrote could not be read by anndata or scanpy. Use --dtype float32, or --layout dense.
• A dense matrix made sparse by a lossy cast drops the zeros the cast makes (#96). Values that became 0 were stored as explicit zeros, three times scipy's nnz in one case, which misleads nnz-based QC.
• Matrices nested two or more groups deep are converted (#96). recast of uns/neighbors/connectivities reported converting it and copied it unchanged.
• h5py 3.16.0 or later is required (#96). h5py 3.15 kept 32 to 160 bytes of memory for good on every dataset read and write, so long streaming runs grew with their number of blocks. h5py 3.16's HDF5 2.0 gives each dataset an 8 MiB chunk cache; stores are opened with 1 MiB, as before. Its Linux wheels need glibc 2.28 or later.

🤖 Generated with Claude Code

[chatgpt-codex-connector]

Codex Review Summary

This comment shows the latest Codex review activity on this pull request.

Review	Status	Commit	Review trigger
📝 Code Review	✅ Completed 2026-11-01T23:42:55.156582Z	db33a98	PR opened

ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

• Open a pull request for review
• Mark a draft as ready
• Comment "@codex review" or "@codex security review".

Codex reacts with 👀 while any review is running, comments if it has suggestions, and reacts with 👍 once all reviews finish with no findings.

[Copilot]

Copilot was unable to review this pull request because the user who requested the review has insufficient remaining quota. Try requesting a review at a lower effort level.

An organization owner can choose the repository organization as the billing source for reviews requested by members with a paid Copilot license. Learn about Copilot code review billing.

[chatgpt-codex-connector]

💡 Codex Review

Here are some automated review suggestions for this pull request.

Reviewed commit: db33a98781

ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

• Open a pull request for review
• Mark a draft as ready
• Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".

[chatgpt-codex-connector]

Include intermediate CSR files in the space estimate

For every dense-to-CSC conversion, and for CSC-to-dense with --in-memory, the core converter still writes an intermediate CSR into scratch_group, but this early return estimates no scratch at all. With an HDF5 output on a disk sized near the preflight estimate, that uncounted file can exhaust the filesystem during an HDF5 operation—the exact failure the preflight is intended to prevent—so count the intermediate CSR independently of whether the subsequent transpose uses buckets or memory.

AGENTS.md reference: AGENTS.md:L178-L183

Useful? React with 👍 / 👎.

[github-actions]

Test Results (py3.12)

3 553 tests  +268   3 539 ✅ +258   16m 48s ⏱️ + 1m 23s
    1 suites ±  0      14 💤 + 10 
    1 files   ±  0       0 ❌ ±  0 

Results for commit 1bec201. ± Comparison against base commit 610c5b1.

♻️ This comment has been updated with latest results.

[github-actions]

Test Results (py3.13)

3 553 tests  +268   3 539 ✅ +258   12m 27s ⏱️ + 1m 0s
    1 suites ±  0      14 💤 + 10 
    1 files   ±  0       0 ❌ ±  0 

Results for commit 1bec201. ± Comparison against base commit 610c5b1.

♻️ This comment has been updated with latest results.