B6b: Block sizing and Zarr v3 shards

.github/workflows/tests.yml

@@ -40,6 +40,11 @@ jobs:
       # before the one setup-python puts on PATH, so up to 0.17.4 the py3.13
       # leg ran the suite on 3.12. UV_PYTHON outranks the file.
       UV_PYTHON: ${{ matrix.python-version }}
+      # Line coverage through sys.monitoring (3.12+) rather than a trace
+      # function: the same numbers, without the trace function's cost on
+      # every line, which had doubled this job's time and taken it past
+      # its timeout.
+      COVERAGE_CORE: sysmon
 
     steps:
       - uses: actions/checkout@11d5960a326750d5838078e36cf38b85af677262 # v4.4.0
@@ -72,9 +77,10 @@ jobs:
             exit 1
           fi
 
+      # The cost guards (`perf`) run in the job below, at the same time.
       - name: Run tests with coverage
         run: |
-          uv run pytest -v -W default tests/ -m "not integration" \
+          uv run pytest -v -W default tests/ -m "not integration and not perf" \
             --cov=adata \
             --cov-report=term-missing \
             --cov-report=xml \
@@ -107,6 +113,67 @@ jobs:
           files: coverage.xml
           fail_ci_if_error: false
 
+  # The cost guards: what an operation reads, writes and holds, counted
+  # rather than timed (tests/test_performance.py, and the few `perf` tests
+  # beside the code they guard). A job of their own, beside the unit suite,
+  # because together they ran past the unit job's timeout; publish.yml runs
+  # this workflow whole, so a release still needs both.
+  perf:
+    runs-on: ubuntu-latest
+    timeout-minutes: 20
+
+    permissions:
+      contents: read
+
+    strategy:
+      fail-fast: false
+      matrix:
+        python-version: ["3.12", "3.13"]
+
+    name: cost guards (py${{ matrix.python-version }})
+
+    env:
+      UV_PYTHON: ${{ matrix.python-version }}
+
+    steps:
+      - uses: actions/checkout@11d5960a326750d5838078e36cf38b85af677262 # v4.4.0
+        with:
+          persist-credentials: false
+
+      - uses: actions/setup-python@a26af69be951a213d495a4c3e4e4022e16d87065 # v5.6.0
+        with:
+          python-version: ${{ matrix.python-version }}
+
+      - name: Set up uv
+        uses: astral-sh/setup-uv@caf0cab7a618c569241d31dcd442f54681755d39 # v3.2.4
+        with:
+          version: "0.13.1"
+          enable-cache: false
+
+      - name: Install dependencies (frozen)
+        run: uv sync --extra dev --frozen
+
+      - name: The guards run on the Python this leg names
+        run: |
+          actual=$(uv run --frozen python -c 'import sys; print("%d.%d" % sys.version_info[:2])')
+          if [ "$actual" != "$UV_PYTHON" ]; then
+            echo "::error::the py$UV_PYTHON leg would run the guards on Python $actual"
+            exit 1
+          fi
+
+      - name: Run the cost guards
+        run: |
+          uv run pytest -v -W default tests/ -m "perf and not integration" \
+            --junitxml="perf-results-py$UV_PYTHON.xml"
+
+      - name: Upload the guards' results
+        uses: actions/upload-artifact@ea165f8d65b6e75b540449e92b4886f43607fa02 # v4.6.2
+        if: always()
+        with:
+          name: perf-py${{ matrix.python-version }}
+          path: perf-results-py${{ matrix.python-version }}.xml
+          retention-days: 30
+
   # What `pip install pyadata-cli` gets today, not what uv.lock pins. 0.7.0
   # to 0.17.2 imported click, which typer had stopped installing in 0.26, so
   # every `adata convert` on a fresh install crashed while the locked suite

AGENTS.md

@@ -35,8 +35,8 @@ uv sync --extra dev                                   # set up
 uv run adata --help                                   # run the CLI from source
 uv run pytest -m "not integration"                    # unit suite, ~2.5 min; what CI gates on
 uv run pytest tests/test_recast.py::test_inplace_replaces_the_source   # one test
-uv run pytest -m "not integration" --cov=adata --cov-fail-under=90     # CI's coverage floor
-uv run pytest -m perf                                 # tracing complexity guards only
+uv run pytest -m "not integration and not perf" --cov=adata --cov-fail-under=90  # CI's coverage job
+uv run pytest -m perf                                 # the cost guards, CI's own job
 ADATA_REQUIRE_VERSION_FIXTURES=1 uv run pytest tests/test_anndata_versions.py -m integration
 uv run python -m benchmarks.run --tier smoke --case concat-inner --out results.json
 uv run python -m benchmarks.report results.json
@@ -46,7 +46,9 @@ Test markers:
 - `integration` tests build uv environments with real anndata releases (0.7.8
   to 0.13) and the oracles (scanpy, PyTables and others), all pinned exactly
   in `tests/reference_stores.py`, so they need the network on the first run.
-- `perf` tests are slow, but CI runs them in the main job.
+- `perf` marks the cost guards (all of `tests/test_performance.py`, and a
+  few beside the code they guard). CI runs them as a job of their own,
+  beside the unit suite's coverage job, and a release needs both.
 
 CI also installs the package from a fresh resolution, without the lock,
 imports every module on the base dependencies alone, and runs `convert`.
@@ -191,7 +193,14 @@ These are easy to break without any test noticing straight away:
   - Commands read the source's version from `Store.zarr_format`.
   - `concat` uses the version its Zarr inputs share.
 - **Outputs keep the source's index dtypes, chunking and compression.**
-  Changing them is `recast`'s job, or that of an explicit flag.
+  Changing them is `recast`'s job, or that of an explicit flag. Zarr v3
+  shards are kept too: `fit_chunks` clamps them with the chunk, and given
+  `itemsize` cuts a unit too large to hold along its rows.
+- **Write a streamed output through `storage.RowWriter`**, and `flush()` it
+  after the last block. A write to part of a Zarr shard rewrites the whole
+  shard, and to part of a chunk Zarr keeps no cache, so writing a `--chunk`
+  block at a time straight into the array costs a unit per block.
+  `_growable_like` returns one for a sparse matrix's `data` and `indices`.
 
 ## Tests
 

docs/developers.md

@@ -136,7 +136,13 @@ before changing anything that writes:
   `open_store`; without it zarr-python writes v3, upgrading a v2 input.
   `concat` uses the version its Zarr inputs share.
 - **Outputs keep the source's index dtypes, chunking and compression.**
-  Changing them is `recast`'s job, or an explicit flag's.
+  Changing them is `recast`'s job, or an explicit flag's. Zarr v3 shards are
+  kept too: `fit_chunks` clamps them with the chunk, and given `itemsize`
+  cuts a unit too large to hold along its rows.
+- **A streamed output is written through `storage.RowWriter`**, flushed after
+  the last block. A write to part of a Zarr shard rewrites the whole shard,
+  and Zarr keeps no chunk cache, so a `--chunk` block written straight into
+  the array costs a whole unit a block.
 - **Memory is set by `--chunk`.** Loading a whole matrix or column is a
   regression unless it is behind an explicit `--in-memory`. `--chunk` is
   defined once in `cli.py` (`_chunk_option`, with `min=1`); reuse it.
@@ -164,8 +170,8 @@ before changing anything that writes:
   page has the front matter the sidebar needs.
 
 ```bash
-uv run pytest -m "not integration" --cov=adata --cov-fail-under=90   # CI's coverage floor
-uv run pytest -m perf                                                # the complexity guards
+uv run pytest -m "not integration and not perf" --cov=adata --cov-fail-under=90  # CI's coverage job
+uv run pytest -m perf                                                # the cost guards, CI's own job
 ADATA_REQUIRE_VERSION_FIXTURES=1 uv run pytest tests/test_anndata_versions.py -m integration
 ```
 

docs/guides/pipelines.md

@@ -123,8 +123,21 @@ most data:
 | `import array` | 1,048,576 | elements |
 
 A dense block holds `--chunk` × n_var values, so lower it for very wide dense
-matrices. It must be at least 1; zero or a negative value is a usage error. Sparse matrices stream in blocks too. obs and var columns are read
+matrices. It must be at least 1; zero or a negative value is a usage error. Sparse matrices stream in blocks too: in `subset` and `split`, about as
+many entries a block as `--chunk` rows hold. obs and var columns are read
 whole, which is what grows with the number of cells.
+
+Three things are set by the store's layout rather than by `--chunk`, each
+bounded:
+
+- A read takes whole chunks of the source, up to 32 MiB.
+- An output is written a whole shard or chunk at a time, so its rows are
+  held until one is full: at most 8 MiB, and 128 MiB across the outputs
+  `split` writes at once. Encoding a shard holds about four times its
+  size again, one shard at a time.
+- `split` writes up to 128 outputs a pass, an HDF5 one about 0.6 MB open.
+
+See [Writing outputs](../reference/commands.md#writing-outputs).
 [Benchmarks](../reference/benchmarks.md) gives peak memory per command on a
 50,000 × 20,000 store, measured on every release: between 70 and 330 MB in
 0.16.1. A request of 1 to 2 GB leaves room for larger stores.

docs/reference/commands.md

@@ -87,7 +87,7 @@ adata subset data.h5ad --inplace --obs barcodes.txt
 | `--inplace` | Replace the source (written to a temporary path first) |
 | `--obs` / `--var` | File of names to keep, one per line |
 | `--obs-query`, `-q` / `--var-query` | Keep rows matching an [expression](query.md) |
-| `--chunk`, `-C` | Row chunk size for dense matrices (at least 1) |
+| `--chunk`, `-C` | Rows per block, of a dense or a sparse matrix (at least 1) |
 | `--zarr-format` | Zarr version to write; defaults to the source's |
 | `--no-space-check` | Write even if the disk looks too small; see [Writing outputs](#writing-outputs) |
 
@@ -285,7 +285,7 @@ adata split data.h5ad --by cluster -o out/ --min-size 50
 | `--min-size` | Skip groups smaller than this |
 | `--dry-run` | Print the plan without writing |
 | `--manifest / --no-manifest` | Write a CSV listing the outputs (default on) |
-| `--chunk`, `-C` | Row chunk size for dense matrices (at least 1) |
+| `--chunk`, `-C` | Rows per block, of a dense or a sparse matrix (at least 1) |
 | `--zarr-format` | Zarr version to write; defaults to the source's |
 | `--no-space-check` | Write even if the disk looks too small; see [Writing outputs](#writing-outputs) |
 
@@ -304,6 +304,11 @@ all of them are complete, and the manifest is written last, so a split that
 fails leaves no partial stores. An output that would land on the input is
 refused, on a case-insensitive filesystem too (a label `A` beside `a.h5ad`).
 
+Each matrix is read once for up to 128 outputs: an open HDF5 file costs
+about 0.6 MB however little it holds, and each output holds rows to write
+its shards and chunks whole. With more groups than that, the matrices are
+read once per 128.
+
 ## `concat`
 
 Concatenate along the obs axis, streaming each input in turn.
@@ -621,6 +626,39 @@ skips it.
 **`--chunk` is at least 1** wherever it appears. A zero or negative value is
 a usage error (exit 2), where it used to write an all-zero dense matrix.
 
+**Outputs keep the source's chunks and shards**, and are read and written a
+whole one at a time:
+
+- A Zarr v3 output keeps the source's shards and the inner chunks inside
+  them, each clamped to the output. Until 0.17.6, `subset`, `split` and
+  `concat` dropped the shard of a sparse matrix and kept its inner chunk:
+  under anndata 0.13 that is 245 elements, a file each.
+- Each shard, or each chunk of a Zarr or chunked HDF5 output, is written
+  once, whole. A shard is rewritten in full for a write of any part of it,
+  so writing one a `--chunk` block at a time took hundreds of times longer.
+  To write a shard whole, the command holds its rows: a shard, or chunk,
+  whose rows across the matrix come to more than 8 MiB is cut along its
+  rows to fit (anndata aims at 1 GB shards). While zarr encodes a unit it
+  holds about four times that again, so this costs up to about 40 MB.
+  `split` shares 128 MiB among the outputs it writes at once, so with many
+  groups their shards and chunks are cut further. A Zarr output of a store
+  with no chunks of its own (a contiguous h5ad X) takes zarr's own chunks,
+  cut the same way.
+- Zarr v2 has no shards: written from a sharded store, a shard becomes the
+  chunk. An HDF5 output of a sharded store lets HDF5 choose its chunks,
+  rather than taking the shard's inner chunk.
+- A dense matrix is read in whole chunks of the source: `--chunk` rows
+  rounded up to whole chunk heights, up to 32 MiB a read (more if
+  `--chunk` asks for more), in `subset`, `split`, `concat`, `recast`,
+  `convert` and `export array` alike. Where one chunk height across every
+  column is over that, as for chunks of (n_obs, 8), `subset`, `split` and
+  `convert` read a band of columns, whole chunks at a time; `concat`,
+  `recast` and `export` read `--chunk` rows, as before.
+- A sparse matrix in `subset` and `split` is read about as many entries a
+  block as `--chunk` rows hold on average, in either layout; a column of a
+  CSC matrix holding more, a gene expressed in every cell, is read in parts.
+- A Zarr v3 shard's index is read once per command, not once a read.
+
 ## Zarr format selection
 
 New Zarr stores follow the source store's version, so a v2 input is not

docs/reference/testing.md

@@ -12,13 +12,13 @@ redirect_from:
 uv sync --extra dev
 uv run pytest                      # everything
 uv run pytest -m "not integration" # no environment building, ~2.5 min
-uv run pytest -m perf              # just the tracing complexity guards
+uv run pytest -m perf              # just the cost guards
 uv run pytest -m integration       # anndata releases, scanpy and PyTables
 ```
 
 CI runs the unit suite on Python 3.12 and 3.13 with a **90% coverage floor**,
-and the anndata compatibility suite and the 10x external-reader suite as
-separate jobs. A fresh-install job installs the package alone from a fresh
+and the cost guards (`-m perf`), the anndata compatibility suite and the 10x
+external-reader suite as separate jobs. A fresh-install job installs the package alone from a fresh
 resolution, without `uv.lock`, as `pip install pyadata-cli` would. It
 imports every module on the base dependencies alone, runs `adata convert`
 with and without click beside typer, and runs the docs-accuracy suite with no
@@ -169,6 +169,7 @@ shard rewritten on every block, so `IOCounts` also counts bytes:
 | `decoded_bytes` | bytes of every whole chunk the reads touched | reads that decode a chunk to use a row of it |
 | `written_bytes`, `largest_write` | bytes handed to writes | |
 | `encoded_bytes` | bytes of every whole chunk the writes touched, and every whole shard of a sharded Zarr array | writes smaller than the unit, each rewriting all of it |
+| `written_by_name`, `encoded_by_name` | the two above, per array | a writer's own output apart from its scratch |
 | `store_get_bytes`, `store_set_bytes` | bytes Zarr's store fetched and stored | |
 
 Decoded and encoded bytes are a model worked out from the selection, the chunk
@@ -276,10 +277,13 @@ path handles well. A CSC densify holding n_obs x `--chunk` values, a dense cast
 check reading the whole matrix and a transpose stepping by the average major
 all passed them. `tests/perf_fixtures.py` builds stores the way users have
 them -- every layout, written by anndata as h5ad, Zarr v2 and anndata 0.13's
-default sharded Zarr v3 (inner chunks of a few hundred bytes in shards of the
-whole array) -- and `test_peak_memory_is_a_small_share_of_the_input` runs
+default sharded Zarr v3 -- and `test_peak_memory_is_a_small_share_of_the_input` runs
 `recast` (each dtype and layout change), `subset` and `concat` over each
-layout, growing n_obs and then n_var.
+layout, growing n_obs and then n_var. zarr 3.4 gives anndata's shards inner
+chunks of about a megabyte; `zarr31_layout` rewrites a store's X as zarr 3.1
+laid it out, inner chunks of a kilobyte in shards of the whole array, which
+is what users' stores hold and where a read or write that splits a unit
+costs most.
 
 A growth ratio cannot be the claim along n_var: widen a matrix and a block of
 `--chunk` rows widens with it, as fast as the input, so a ratio cannot tell it
@@ -290,8 +294,8 @@ bounded paths came to at most 0.21. Growing n_obs, `--chunk` is 256 rows, so a
 block of `--chunk` columns over every row (CSC densify before F030) is plain.
 Each fixture is built once and shared by the commands.
 
-The known gap is marked `xfail` with its owner: sparse `subset` (F032, B6b),
-strict, so the fix has to flip it.
+Sparse `subset` was the known gap, read 4,096 majors a block whatever
+`--chunk` said (F032); 0.17.6 closed it, and `GRID_GAPS` is empty.
 
 The recast guards beside it pin each block in bytes, on h5ad and sharded Zarr
 v3 (Zarr v2 is covered by the correctness tests): the cast
@@ -368,10 +372,19 @@ should scale with, scale only that, and hold everything else fixed.
 | `recast` (every dtype and layout change), `subset`, `concat` | n_obs, n_var; CSR, CSC, dense | **peak memory** a small share of the input |
 | `recast --dtype` (dense), a skewed transpose | n_obs; h5ad, sharded Zarr v3 | the **largest read** set by the chunk |
 | CSC `--layout dense`, dense `--layout csc` | n_obs; h5ad | the **largest read** set by the chunk, and memory per row |
+| `subset`, `split`, `concat`, `recast`, `convert` into anndata 0.13's shards | Zarr v3 as zarr 3.1 wrote it | each shard **encoded once** |
+| dense `subset`, `concat`, `recast`, `export array` | tall gzip chunks, zarr's own chunks | each chunk **decoded once** |
+| `subset` into a sharded output | 1 MiB units; dense, CSR | **peak allocation** a few units, not the shard |
+| `split` into chunked HDF5 | eight groups | each output chunk **encoded once** |
+| `subset`, `concat` of a sharded store | Zarr v3 as zarr 3.1 wrote it | each shard index **fetched once** |
+| sparse `subset` | n_obs; CSR, CSC, a gene in every cell | the **largest read** about `--chunk` rows of entries |
+| `convert` of a matrix chunked (n_obs, 8) | n_obs | the **largest read** a tile within the budget |
+| `split` | 128 against 512 groups | **resident memory** flat past 128 outputs a pass |
 
 The `slow` marker is on the guards that build a 65,536-row store or run `peak_rss`;
-they still gate merges, and `-m "not slow"` skips them locally. `perf` is on the
-tracing guards.
+they still gate merges, and `-m "not slow"` skips them locally. `perf` is on
+every guard in the file, and on the few beside the code they guard; CI runs
+them as a job of their own, beside the unit suite's coverage job.
 
 h5py 3.15 kept 32 to 160 bytes of every dataset read, write and `.shape` for good, so
 any long streaming loop grew with its block count;
@@ -547,7 +560,7 @@ ADATA_SKIP_VERSION_FIXTURES=1 uv run pytest
 ## Coverage
 
 ```bash
-uv run pytest -m "not integration" --cov=adata --cov-report=term-missing
+uv run pytest -m "not integration and not perf" --cov=adata --cov-report=term-missing
 ```
 
 The floor is 90%. What remains uncovered is mostly defensive `except` branches

docs/troubleshooting.md

@@ -338,9 +338,23 @@ store it wrote; for a store another tool changed, run
 
 Lower `--chunk`: a dense block holds `--chunk` × n_var values, 1,024 rows by
 default. Two things are held whole by design: obs and var columns, and a
-pycisTopic pickle, which cannot be read in part. See
+pycisTopic pickle, which cannot be read in part. Writing an output a whole
+shard or chunk at a time holds at most 8 MiB more (128 MiB across
+`split`'s outputs), and about four times a shard while it is encoded. See
 [Pipelines and containers](guides/pipelines.md#memory-and---chunk).
 
+### It is slow on a Zarr store anndata 0.13 wrote
+
+anndata 0.13 writes Zarr v3 sharded: a whole array in one shard file,
+inner chunks of a kilobyte (with zarr 3.1) inside. Until 0.17.6, `subset`,
+`split`, `concat`, `recast` and `convert` rewrote a whole shard for every
+block they wrote and read its index again for every block they read,
+hundreds of times slower than the same data as `.h5ad`. They write each
+shard once now, keep each index once read, and keep the source's shards on
+output. To cap what writing a shard whole holds, a shard whose rows come to
+more than 8 MiB is cut along its rows. See
+[Writing outputs](reference/commands.md#writing-outputs).
+
 ### `A loom holds its matrices dense: 54.0 GB to compress and write. It is written, but may take a while`
 
 A loom has no sparse matrices, so a sparse store's zeros are all written.