diff --git a/STATUS.md b/STATUS.md index a3ba3d6..127a475 100644 --- a/STATUS.md +++ b/STATUS.md @@ -7,6 +7,7 @@ moves; the plan of record for the current item is in the section below. | Version | What | Issues | |---|---|---| +| 0.17.6 | Audit fixes, waves 3 and 4 (streaming): reads and writes follow the source's chunks and Zarr v3 shards, each shard written once; `recast` checks and converts in `--chunk` blocks; `split`'s memory no longer grows with its groups; pickle arrays checked before use; 16-bit images export as they are; plain-line logs and names printed as stored; h5py >= 3.16 | #96, #97, #101, #103 | | 0.17.5 | Audit fixes, wave 2 (fidelity): scanpy's marker genes and anndata 0.7 files survive every copy, data held outside an h5ad refused; subset/split keep every named row and subset every element as anndata; concat takes any number of inputs in anndata's order; 10x, MTX, Parse, CellBender, loom, SCope and MuData fixes; release pipeline hardened, image non-root; zarr >= 3.2 | #93, #94, #95, #99, #100, #102 | | 0.17.4 | Audit fixes, wave 1 (data safety): every writer staged and checked, close errors fail the command, `--chunk` at least 1, a free-space check; text/MTX/dense imports refuse what they cannot read; concat refuses repeated var names and keeps missing values and dtypes; import/export type values as pandas does; Pillow 13 | #89, #90, #91, #92, #98 | | 0.17.3 | Space Ranger's analysis (clusters, UMAP, PCA) for Visium spots and HD bins; `convert` works on fresh installs (broken since 0.7.0) | #85, #88 | @@ -25,40 +26,10 @@ The conversion epic's sub-issues (#21–#30) are all done and ticked in #20. ## In progress -The fixes from the October 2026 audit, epic #87: 164 findings in 16 -batches, one PR and one sub-issue each. Batches in a wave own disjoint files -and run in parallel; a batch that shares files with another starts once that -one has merged. - -| Wave | Batches | Release | -|---|---|---| -| 0 | B0 #88 | 0.17.3 | -| 1 | B1 #89, B2a #90, B2b #91, B3a #92, B7 #98 | 0.17.4: data safety | -| 2 | B3b #93, B4 #94, B5 #95, B8a #99, B8b #100, B9 #102 | 0.17.5: fidelity | -| 3 | B6a #96, B6b #97, B8c #101 | 0.17.6: streaming | -| 4 | B10 #103 | 0.17.6 | - -Decisions: -- Every batch ships as a 0.17.x patch. -- `concat` orders an outer join's var and merged categories as anndata does - (sorted), and the changelog says so (B3b). -- Pickle reading is hardened, and the docs say pickles are trusted input - (B8c). -- A full disk aborts inside HDF5, beyond the tool's reach, so writers check - free space against the output's estimated size first (B1). -- Each PR carries its changelog entry in its description, and the release PR - collects them, so parallel batches do not conflict in `docs/changelog.md`. - -Steps: -- [x] B0: `convert` on fresh installs (#88), in 0.17.3 -- [x] Wave 1, then 0.17.4 -- [x] Wave 2, then 0.17.5 -- [ ] Waves 3 and 4, then 0.17.6 - - [x] B6a: the memory harness, then recast streaming (#118) - - [x] B8c: pickles, Xenium and images (#120) - - [ ] B6b: block sizing and Zarr v3 shards (#121) - - [ ] B10: CLI output and the documentation sweep - - [ ] Release 0.17.6 +Nothing. The fixes from the October 2026 audit (epic #87, 164 findings in +16 batches) finished with 0.17.6; the sub-issues (#88–#103) are all done +and ticked in #87. The batch names in the follow-ups below (B1 to B10) are +theirs. ## Follow-ups noticed diff --git a/docs/changelog.md b/docs/changelog.md index db07345..a7d5330 100644 --- a/docs/changelog.md +++ b/docs/changelog.md @@ -8,6 +8,167 @@ nav_order: 7 Notable changes to `adata-cli`. Versions are `MAJOR.MINOR.PATCH`; tags carry no `v` prefix. +## 0.17.6 + +The last wave of fixes from the October 2026 audit +([#87](https://github.com/cellgeni/adata-cli/issues/87)), on streaming, which +closes it. Reads and writes follow the source's chunks and shards, so +anndata 0.13's sharded Zarr v3 stores stream at the cost `--chunk` implies +and each shard is written once; `recast` checks and converts in `--chunk` +blocks; `split`'s memory no longer grows with its groups. Pickle reading is +hardened, 16-bit images export as they are, and the CLI's output is plain +lines in logs and pipes, with names printed as the store spells them. +h5py 3.16 or later is now required; its Linux wheels need glibc 2.28. + +### Changed + +- **h5py 3.16.0 or later is required** + ([#96](https://github.com/cellgeni/adata-cli/issues/96)). h5py 3.15 kept 32 + to 160 bytes of memory for good on every dataset read and write, so long + streaming runs grew with their number of blocks. h5py 3.16's HDF5 2.0 gives + each dataset an 8 MiB chunk cache; stores are opened with 1 MiB, as before. + Its Linux wheels need glibc 2.28 or later. + +### Fixed + +- **Subset, split, concat, recast and convert write each Zarr v3 shard once** + ([#97](https://github.com/cellgeni/adata-cli/issues/97)). anndata 0.13 writes + Zarr v3 sharded, a whole array in one shard. Commands wrote outputs a + `--chunk` block at a time, and zarr rewrites a whole shard for a write of any + part of it: subset and concat wrote 15 times the bytes they meant to, split + 121 times, and a dense subset held 1.7 GB. Outputs are written a whole shard + (or chunk) at a time now. A shard is cut along its rows to at most 8 MiB, 128 + MiB shared across the outputs `split` writes at once, and about four times + that is held while zarr encodes it. +- **Outputs keep a sharded source's shards and inner chunks** + ([#97](https://github.com/cellgeni/adata-cli/issues/97)). A sparse matrix's + `data` and `indices` dropped the shard and kept the inner chunk, 245 elements + a file under anndata 0.13 with zarr 3.1: thousands of files per array. An + h5ad output took the inner chunk as its HDF5 chunk, making a file twice + anndata's own. Zarr v3 outputs keep the shard. HDF5 outputs let HDF5 choose + their chunks, and `--zarr-format 2` takes the shard as the chunk. +- **Sparse `subset` and `split` follow `--chunk`** + ([#97](https://github.com/cellgeni/adata-cli/issues/97)). Blocks were 4,096 + rows or columns whatever `--chunk` said, so a CSC block (4,096 genes across + every cell) grew with the number of cells. A block holds about as many + entries as `--chunk` rows do now, and a gene expressed in every cell is read + in parts. +- **Reads follow the source's chunks** + ([#97](https://github.com/cellgeni/adata-cli/issues/97)). `subset`, `split`, + `concat`, `recast`, `convert` and `export` read a dense matrix in whole + chunks of the source, so a chunk taller than `--chunk` is no longer decoded + once for every block: 7 to 60 times the matrix for zarr's own chunks or + chunks of (n_obs, 8). Past 32 MiB, `subset`, `split` and `convert` read a + band of columns, so `convert` no longer reads a column-chunked matrix whole. + A Zarr v3 shard's index is read once per command, not once per block. +- **`split` memory no longer grows with the number of groups** + ([#97](https://github.com/cellgeni/adata-cli/issues/97)). An open HDF5 file + costs about 0.6 MB whatever is in it, and split held up to 1,024 open at + once: 581 MB of RSS at 512 groups, against 143 MB now. It writes at most 128 + outputs per pass over the matrices, so past 128 groups the matrices are read + once per 128. Each output is written a whole chunk at a time, so a chunk 512 + rows tall is no longer decoded and encoded once for every block. +- **`recast --dtype` checks a dense matrix a block at a time** + ([#96](https://github.com/cellgeni/adata-cli/issues/96)). The round-trip + check read the whole matrix at once, and two more copies of it, whatever + `--chunk` said. It reads `--chunk` rows now, as the conversion does. +- **Densifying a CSC matrix, and writing a dense one as CSC, hold `--chunk` + rows** ([#96](https://github.com/cellgeni/adata-cli/issues/96)). Each block + was `--chunk` columns of every row, so memory followed the number of cells: 8 + GB a block at 2M cells. Both now go through a CSR in scratch and the + streaming transpose. +- **The streaming transpose reads about a chunk at a time on skewed matrices** + ([#96](https://github.com/cellgeni/adata-cli/issues/96)). It stepped by the + average row, so a run of genes expressed in every cell was read many chunks + at once, and a gene expressed in every cell was sorted whole. It steps by + entries now, and streams such a gene from a bucket of its own. +- **A sparse matrix made from a dense one keeps element-sized chunks** + ([#96](https://github.com/cellgeni/adata-cli/issues/96)). `recast --layout + csr`/`csc` and the 10x-h5 export gave `data` and `indices` the dense source's + chunk height as their chunk, down to 16 elements: hundreds of thousands of + chunks and a file three times the size. +- **`--force` no longer skips the index range check** + ([#96](https://github.com/cellgeni/adata-cli/issues/96)). Given to accept a + lossy cast, it also let `indptr` overflow `--indices-dtype` and wrap + negative, a corrupt matrix with exit 0. A dense matrix made sparse is + range-checked too. +- **float16 is refused for a sparse matrix** + ([#96](https://github.com/cellgeni/adata-cli/issues/96)). scipy has no + float16 sparse matrix, so the store `recast` wrote could not be read by + anndata or scanpy. Use `--dtype float32`, or `--layout dense`. +- **A dense matrix made sparse by a lossy cast drops the zeros the cast makes** + ([#96](https://github.com/cellgeni/adata-cli/issues/96)). Values that became + 0 were stored as explicit zeros, three times scipy's nnz in one case, which + misleads nnz-based QC. +- **Matrices nested two or more groups deep are converted** + ([#96](https://github.com/cellgeni/adata-cli/issues/96)). `recast` of + `uns/neighbors/connectivities` reported converting it and copied it + unchanged. +- **Pickle arrays are checked before use** + ([#101](https://github.com/cellgeni/adata-cli/issues/101)). Reading a + pycisTopic pickle checks each array it rebuilds (dtype, shape, and a sparse + matrix's indices) before numpy, scipy or pandas uses it, and refuses a pickle + that fails before anything is written, so a crafted pickle can no longer make + scipy write out of bounds or have numpy read memory as another type. Dates + and times are no longer read from pickles, which also stops a crafted + timezone from opening a local file; pycisTopic's objects hold none. Pickles + are still trusted input: read them only from a source you would trust with + the data. Pickles that pandas 1.5 writes with missing values in a nullable + text column now read. +- **Imputed pycisTopic matrices convert whatever their width** + ([#101](https://github.com/cellgeni/adata-cli/issues/101)). `--dense` + chunked X as 4096 cells by every feature, which HDF5 refused past 262,144 + float32 features, and CSR output held 4096 cells' coordinates at once, + about 50 GB at 500,000 regions. Both now work about a million entries at + a time, and a dense X is chunked the same way. +- **`--dense` is honoured for gene activity** + ([#101](https://github.com/cellgeni/adata-cli/issues/101)). An imputed + object holding a sparse matrix was written as CSR although `--dense` was + given and the summary said dense. +- **A Xenium bundle converts into a new folder** + ([#101](https://github.com/cellgeni/adata-cli/issues/101)). With only + `analysis.tar.gz`, converting to an output whose folder did not exist yet + failed with a bare "No such file or directory" naming a temporary path. +- **`export image` keeps 16-bit images** + ([#101](https://github.com/cellgeni/adata-cli/issues/101)). Every value + from 255 up was clipped to 255, silently. One channel of 16 bits is now + written as a 16-bit PNG or TIFF, and values an image cannot keep are + refused, naming their range; the new `--scale` stretches them onto 0-255. +- **Names with square brackets print as the store spells them** + ([#103](https://github.com/cellgeni/adata-cli/issues/103)). Rich read the + `[mt]` of scanpy's `pct_counts[mt]` as a style tag. `view`, `ls`, `split` and + messages showed `pct_counts`, a name the next command could not find, and a + name, label or attribute holding `[/x]` stopped the command. Every name, + label, path and value from a store or the command line is escaped now. +- **`view FILE ENTRY` on a missing entry fails** + ([#103](https://github.com/cellgeni/adata-cli/issues/103)). It printed its + `Error:` line on stdout and exited 0, so a script took the message for the + entry's description. It goes to stderr with exit code 1 now, as `ls` does. +- **Logs hold plain lines** + ([#103](https://github.com/cellgeni/adata-cli/issues/103)). Written to a file + or a pipe, stderr held colour and cursor codes and wrapped messages at 80 + columns, so `grep '^Error:' adata.log` found nothing. Off a terminal there + are no codes now and an error is one line. On a terminal nothing changes. +- **`convert --help` names the extras to install** + ([#103](https://github.com/cellgeni/adata-cli/issues/103)): + `pyadata-cli[spatial]` and `pyadata-cli[pickle]` showed as `pyadata-cli`. +- **Help says what the commands do** + ([#103](https://github.com/cellgeni/adata-cli/issues/103)): + - `--to` lists only the formats `convert` can write; + - `--inplace` says the source is replaced by a store written beside it, not + modified directly; + - `convert --help` says which formats are only read. +- **`adata.cli.main(argv)` runs `argv`** + ([#103](https://github.com/cellgeni/adata-cli/issues/103)). It parsed the + process's own command line instead. +- **The documentation matches the code** + ([#103](https://github.com/cellgeni/adata-cli/issues/103)): + - the command reference has every option; + - troubleshooting and the pipelines guide list everything held whole in + memory, images among them; + - the testing page maps every test file and CI job; + - the 0.17.1 entry below no longer says images are not loaded whole. + ## 0.17.5 The second wave of fixes from the October 2026 audit @@ -325,7 +486,6 @@ user, so a Docker run that writes to a mounted folder needs `--user`. longer runs unpinned packages with a token that can push to main. Actions and uv are pinned, and tags reach scripts through `env:`. - ## 0.17.4 The first wave of fixes from the October 2026 audit @@ -530,7 +690,6 @@ values as pandas does. Pillow moves to 13.0.0 for its decoder advisories. advisories in its decoders. The floor is now `pillow>=13.0.0`, and the lock also takes pygments 2.20.0 and pytest 9.0.3 (dev) for theirs. - ## 0.17.3 Standard Visium spots and Visium HD's 8 and 16 µm bins now bring in Space diff --git a/pyproject.toml b/pyproject.toml index c20d738..d20a6e1 100644 --- a/pyproject.toml +++ b/pyproject.toml @@ -3,7 +3,7 @@ # so the distribution is published as `pyadata-cli`. The import package # and the command are both still `adata`. name = "pyadata-cli" -version = "0.17.5" +version = "0.17.6" description = "Streaming CLI for exploring and editing large AnnData .h5ad and .zarr stores" readme = "README.md" requires-python = ">=3.12" diff --git a/uv.lock b/uv.lock index 364fc51..c1392da 100644 --- a/uv.lock +++ b/uv.lock @@ -570,7 +570,7 @@ wheels = [ [[package]] name = "pyadata-cli" -version = "0.17.5" +version = "0.17.6" source = { editable = "." } dependencies = [ { name = "h5py" },