B10: CLI output and documentation sweep

AGENTS.md

@@ -24,9 +24,10 @@ Naming:
 - The import package and the command are `adata`.
 - The repo directory is still called `h5ad-cli`.
 - `h5ad` is a deprecated alias of the command (`main_deprecated` in `cli.py`).
+- `info` is a hidden, deprecated alias of `view`.
 - `convert` given recast's old flags forwards to `recast` with a warning.
 
-Both of these deprecated paths stay until 1.0.0.
+These deprecated paths stay until 1.0.0.
 
 ## Commands
 
@@ -50,19 +51,31 @@ Test markers:
   few beside the code they guard). CI runs them as a job of their own,
   beside the unit suite's coverage job, and a release needs both.
 
-CI also installs the package from a fresh resolution, without the lock,
-imports every module on the base dependencies alone, and runs `convert`.
-That catches a module-level import of something undeclared; one inside a
-function is caught only on the paths it runs. Don't import click: typer
-0.26 and later no longer install it. Compare the enums typer hands back
-(such as `ParameterSource`) by `.name`.
+CI (`.github/workflows/tests.yml`) runs on every pull request:
+- the unit suite on Python 3.12 and 3.13, with the coverage floor;
+- the cost guards (`-m perf`) on both interpreters, at the same time;
+- a fresh install: the package from a fresh resolution, without the lock,
+  every module imported on the base dependencies alone, `convert` run with
+  and without click beside typer, and the docs-accuracy suite run with no
+  click. That catches a module-level import of something undeclared; one
+  inside a function is caught only on the paths it runs;
+- the unit suite at each direct dependency's floor;
+- the `integration` tests of `test_anndata_versions.py` and
+  `test_external_readers.py`, each its own job;
+- `workflow lint` (actionlint and zizmor) and `container image`, which
+  builds the Docker image and smoke-tests it.
 
-Neither CI nor the repo runs a linter or formatter. Match the style of the
-surrounding code.
+Don't import click: typer 0.26 and later no longer install it. Compare the
+enums typer hands back (such as `ParameterSource`) by `.name`.
+
+No linter or formatter runs on the code; CI lints only the workflows. Match
+the style of the surrounding code.
 
 ## Architecture
 
-Each layer calls only the layers below it:
+Each layer calls only the layers below it, with one exception: `storage/`
+uses `elements/` (the spec, `strings`, and the readers and writers that
+`copy_tree` and `create_dataset` need to copy an element faithfully).
 
 - **`cli.py`**: the whole Typer app, including the `export` and `import` sub-apps.
   - Commands parse options, validate cheap things such as `--zarr-format`, call
@@ -71,7 +84,13 @@ Each layer calls only the layers below it:
     `--chunk` its `min=1`, and `_space_check_option` is `--no-space-check`.
     A new command that takes either uses these, not its own `typer.Option`.
   - `console` writes to **stderr**. Only `out_console` and data go to stdout,
-    so output pipes cleanly.
+    so output pipes cleanly. Rich asks each stream whether it is a terminal:
+    off one, as in a log file or a pipe, it writes no colour, cursor or
+    spinner codes, and `console` keeps each message on one line.
+  - `raise _fail(e)` prints an exception as the one `Error:` line on stderr
+    and exits 1. `main(argv)` runs the arguments it is given and raises
+    `SystemExit` for its caller; run as the command (`argv` None), a failed
+    run ends the process at once (`_leave`), past h5py's teardown.
 - **`commands/`**: one module per command. Each orchestrates its command:
   - planning and refusals;
   - progress output;
@@ -128,7 +147,11 @@ Each layer calls only the layers below it:
   in the matrix's order; it also reads the `analysis/` (clusters, UMAP, PCA)
   that Xenium and Space Ranger (spots, bins and cells) share.
 - **`formats/`**: the per-file-type halves of `export` and `import`: npy, mtx,
-  csv, json and images. `mtx.py` is a streaming Matrix Market codec (two
+  csv, json and images, with `validate.py` checking an element's kind and
+  dimensions before an import writes it. Beside them are what converters
+  read: `parquet.py` (Visium HD's positions, the only pyarrow import),
+  `pickled.py` (pycisTopic's pickles, through an allowlist) and
+  `pandas_frame.py` (pandas frames written as anndata writes them). `mtx.py` is a streaming Matrix Market codec (two
   passes in, block writes out) shared by `import`/`export sparse` and the
   `10x-mtx` and `parse` converters; `textio.py` holds its gzip-aware block
   I/O, and `table.py` a streaming CSV reader that types columns as pandas
@@ -137,7 +160,8 @@ Each layer calls only the layers below it:
   (categorical when values repeat); the loom import uses it. `geojson.py`
   walks a GeoJSON FeatureCollection a feature at a time and measures each
   polygon's centroid and area (Space Ranger 4's cell outlines).
-- **`util/`**: `staging.py` and `path.py` (below), plus `names.py`.
+- **`util/`**: `staging.py` and `path.py` (below), plus `names.py` and
+  `markup.py` (`literal`, below).
   - `path.check_output(output, inputs)` is the one output check, and
     `path.output_target(source, output, inplace)` wraps it for commands with
     `--inplace`.
@@ -186,7 +210,20 @@ These are easy to break without any test noticing straight away:
   no cleanup runs. Honour `--no-space-check`.
 - **Validate before the output exists.** Resolve selections, check casts and
   build plans first, so that a refusal writes nothing. An empty selection is
-  a refusal, never an empty store.
+  a refusal, never an empty store. What only the data can show (an index out
+  of range) refuses during the write; staging keeps the output from
+  appearing.
+- **Escape what the store or the user wrote before Rich prints it.** Rich
+  reads `[word]` as a style tag, so `pct_counts[mt]` printed as it is shows
+  `pct_counts`, and `[/x]` crashes the command. Member names, labels, paths
+  and attribute values go through `util/markup.literal`, in messages, status
+  lines, progress descriptions and tree labels alike. It escapes as
+  `rich.markup.escape` does, but a trailing backslash prints once whatever
+  follows. Never write brackets around a value in a template
+  (`obs\\[{literal(column)}]`): a value with an unclosed `[` closes into a
+  tag there; escape the whole `obs[column]` instead. Help text is markup
+  too: a literal `[` there is `\\[` in the source, which the docstring holds
+  as `\[`.
 - **Zarr outputs keep the source's version.**
   - Pass `zarr_format=` to `open_store`. Without it zarr-python writes v3, and a
     v2 input gets silently upgraded.
@@ -208,20 +245,30 @@ These are easy to break without any test noticing straight away:
   `zarr3`. Write element-level tests against it, so one test body covers all
   three backends.
 - CLI tests use a module-level `CliRunner`. Strip ANSI codes and collapse
-  whitespace before matching messages, because Rich wraps them. The CLI test
-  files define an `_out(result)` helper for this.
+  whitespace before matching messages: `FORCE_COLOR` brings the codes back,
+  and help text and stdout's trees wrap at Rich's width. The CLI test files
+  define an `_out(result)` helper for this.
 - Correctness is checked against an oracle: anndata reads back what we write,
   and scipy checks `recast`. Self-consistency alone is not accepted.
 - `tests/test_performance.py` guards complexity by **counting** elements and
   bytes read through hooks on h5py and zarr (`tests/perf_counters.py`). It never
   uses a clock. A change that adds reads per row or per group will fail it.
+  - The same hooks count bytes: the largest single read (the block a loop
+    holds), and the whole chunks and shards each read decodes and each write
+    encodes.
+  - `perf_counters.peak_rss(argv)` runs `adata` in a fresh interpreter and
+    returns the peak memory it added. The memory guards grow the input
+    along n_obs and n_var and assert the peak is a small share of it.
+  - `tests/perf_fixtures.py` builds stores as anndata writes them: every
+    layout, h5ad, Zarr v2 and anndata 0.13's sharded Zarr v3.
 - `tests/test_docs_are_accurate.py` parses every `adata …` line in the README,
   `docs/` and the `--help` examples, and checks it against the real Click tree.
   A new or renamed option has to be documented correctly in
   `docs/reference/commands.md`. A new command also needs a section there, an
   entry in the table of contents and a mention in the app help. The same file
-  checks that relative links name a page and an anchor that exist, and that
-  every page has `title` front matter and a `parent` that exists.
+  checks that relative links name a page and an anchor that exist, that
+  every page has `title` front matter and a `parent` that exists, and that
+  every test file has a row in `docs/reference/testing.md`'s map.
 - House style:
   - test names read as sentences
     (`test_a_failed_selection_keeps_an_existing_output`);

README.md

@@ -4,9 +4,9 @@ Explore and edit huge AnnData stores (`.h5ad` and `.zarr`) from the command line
 without loading them into memory and without a Python session. `adata` inspects,
 subsets, splits, concatenates, recasts and converts stores, and moves elements in
 and out as CSV, JSON, `.npy`, `.mtx` or images. It implements the AnnData on-disk
-spec directly against h5py and zarr, so it streams stores larger than RAM, and
-converts to and from 10x, STARsolo, Visium, Xenium, CellBender, Parse, loom, MuData
-and more.
+spec directly against h5py and zarr, so it streams stores larger than RAM. It
+reads 10x, STARsolo, Visium, Xenium, CellBender, Parse, loom, MuData and more,
+and writes 10x, loom and MuData.
 
 **Documentation: <https://cellgeni.github.io/adata-cli/>**
 

STATUS.md

@@ -54,9 +54,48 @@ Steps:
 - [x] Wave 1, then 0.17.4
 - [x] Wave 2, then 0.17.5
 - [ ] Waves 3 and 4, then 0.17.6
+  - [x] B6a: the memory harness, then recast streaming (#118)
+  - [x] B8c: pickles, Xenium and images (#120)
+  - [ ] B6b: block sizing and Zarr v3 shards (#121)
+  - [ ] B10: CLI output and the documentation sweep
+  - [ ] Release 0.17.6
 
 ## Follow-ups noticed
 
+- From waves 3 and 4 of the audit fixes:
+  - The streaming transpose and other users of `csr_build.BucketSorter`
+    append every input block to every bucket, (nnz/chunk)² appends, and
+    each bucket holds three growable datasets (1.8 GB at 30,000 buckets).
+    The transpose knows its row sizes, so fixed regions as in concat's
+    `_ColumnBuckets` would do; concat's `_csc_runs` and recast's
+    `_bucket_runs` could then move into `csr_build` (B3b, B6a).
+  - The transpose's scratch lives inside a Zarr output and is appended to
+    unbuffered, about 135 times its writes; an HDF5 scratch beside the
+    output, as concat has, would fix it. It is also missing from recast's
+    free-space estimate (B3b, B6a, B6b).
+  - recast's sparse reads are a fixed million entries, not `--chunk`; and
+    `concat`, `recast` and `export` read `--chunk` rows where one chunk
+    height is over the read budget, with column tiles only in `subset`,
+    `split` and `convert` (B6a, B6b).
+  - `copy_dataset` writes HDF5 text uncompressed: a Zarr to h5ad
+    conversion's obs index is about nine times anndata's (B6b).
+  - `convert` has no `--chunk`: the cisTopic and MuData converters use a
+    fixed byte budget (B8b, B8c).
+  - A pickle holding pandas 3's Arrow strings is refused; pycisTopic's
+    pandas 1.5 never writes one (B8c).
+  - Python warnings reach stderr with their source line; printing them as
+    `Warning: ...` lines would match the rest of the output (B5).
+  - CI's `external-readers` job is named "10x h5 against scanpy and
+    PyTables" and its `compatibility` job "anndata 0.8-0.13", though the
+    first checks every converter's reader and the second reads 0.7.8 too.
+    Renaming a job changes its required-check name, so it waits for the
+    repository settings (B10). `test_concat_versions.py` is an
+    `integration` test no CI job runs (B10).
+  - `test_write_safety.py::test_export_writes_to_devices_and_pipes_as_they_are`
+    failed once in a dozen local runs on macOS: its FIFO reader stayed
+    blocked with no writer left. The reader thread is not a daemon, so it
+    also kept pytest from exiting; a daemon thread would at least let the
+    run end (B10).
 - From wave 2 of the audit fixes:
   - CI: a Python 3.14 leg; Dependabot for the pinned actions, uv and base
     image (the pinned actions run on Node 20, which GitHub is retiring); the
@@ -67,6 +106,14 @@ Steps:
     that fix's scope (B5).
   - Flex's `target_sets` group is not carried, and `--make-unique` names
     export with their suffixes (B8a).
+  - `formats/textio.py` does not unquote a field that starts with `"`, as
+    pandas does; if it did, MTX export could quote such names rather than
+    refuse them (B8a).
+  - A fixed-width text dataset that HDF5 compressed can inflate far beyond
+    its stored size when read whole, as the loom import reads var columns;
+    only loom's global attributes are bounded (B8b).
+  - Under an outer join, an input lacking some genes counts as lacking var
+    columns, varm and varp entirely; anndata pads them (B3b).
   - A numeric `in` list with thousands of values is slow (B4).
 - From wave 1 of the audit fixes:
   - `core/csr_build.py`'s on-disk sort makes many small appends, slow for
@@ -86,17 +133,6 @@ Steps:
 - Dropped: spatial bundles whose matrix is a MEX folder only. 10x always
   writes the `.h5` beside it, and the refusal points to `--from 10x-mtx`.
 
-- Fixed in 0.16.2 (#77): `export image` without `-o` crashed (`'NoneType'
-  object has no attribute 'parent'`); it now writes PNG to stdout, as every
-  other export writes there. Found by the docs review.
-- Fixed in 0.16.1 (#74): `subset`, `split` and the MuData export kept
-  every category of a categorical column, where anndata's subsetting keeps
-  only those used (and narrows `uns["<col>_colors"]`). The ~50% "slack"
-  noted here before was mostly a measuring error -- `get_storage_size()`
-  leaves out the variable-length string heap -- but the unpruned categories
-  were real: the pbmc3k Multiome `.h5mu` drops from 84.7 to 74.2 MB, and a
-  subset's obs and var now equal anndata's own subset.
-
 ## Next candidates (#31)
 
 - alevin-fry USA mode, kallisto | bustools

docs/changelog.md

@@ -645,9 +645,9 @@ Parquet positions.
 
 ### Changed
 
-- **Images are written a strip of rows at a time**, as they are read, rather
-  than loaded before the write starts: Visium HD's hires image is 6,000
-  pixels, some 100 MB.
+- **Images are copied into the store a strip of rows at a time**, rather
+  than as a second whole copy: Visium HD's hires image is 6,000 pixels, some
+  100 MB. The image itself is still decoded whole, as a PNG can only be.
 - **Barcodes are matched as bytes**, a quarter of the memory of numpy's
   unicode, for Visium, Visium HD and Xenium alike.