B3b: Concat fidelity and scale

docs/reference/commands.md

@@ -305,7 +305,7 @@ adata concat a.h5ad b.h5ad -o m.h5ad --keys a,b --index-unique - --uns-merge sam
 | `--keys` | Names for the inputs; defaults to their filenames |
 | `--index-unique` | Delimiter for suffixing obs names with their key |
 | `--merge` / `--uns-merge` | `drop` (default), `same`, `unique`, `first`, `only` |
-| `--fill-value` | Value for dense cells introduced by an outer join |
+| `--fill-value` | Value for dense cells introduced by an outer join; an integer matrix that cannot hold it becomes float64 |
 | `--chunk`, `-C` | Row chunk size while streaming (at least 1) |
 | `--zarr-format` | Zarr version to write; defaults to the inputs' when they agree, otherwise v3 |
 | `--no-space-check` | Write even if the disk looks too small; see [Writing outputs](#writing-outputs) |
@@ -317,11 +317,21 @@ them; make them unique first (anndata's `var_names_make_unique()`).
 are refused under `--join inner`; inputs with no variables at all
 concatenate to n_obs × 0.
 
+var is in the order `anndata.concat` gives it. Under `--join inner` that is
+the first input's order. Under `--join outer` it is the inputs' own order
+when every input has the same variables in the same order, and the union
+sorted otherwise; an input with no variables has no say.
+
 obs columns keep their dtypes: categoricals union their category sets
 (numeric categories stay numeric, and anndata 0.7's categoricals are read by
 label), and nullable columns keep their masks. Under `--join outer` a column
-missing from one input is padded rather than dropped. Missing values stay
-missing:
+missing from one input is padded rather than dropped. Merged categories are
+ordered as anndata orders them: the first input's order, if its categories
+already include every input's; otherwise the union in natural order (`t2`
+before `t10`, numbers by value). An ordered categorical stays ordered only
+if every input has the same categories in the same order; otherwise it is
+written unordered, with a warning, as anndata keeps no order either.
+Missing values stay missing:
 
 - Text with missing values (from a categorical, a nullable column, or an
   input without the column) is written as a categorical whose missing rows
@@ -342,15 +352,30 @@ obsm entries present in every input are stacked: arrays as they are, sparse
 matrices as X is, and dataframes (a SCope import's `RegulonsAUC`) as obs is,
 their columns following `--join`. Like X, an entry must be stored the same
 way in every input (CSR, CSC or dense); concat refuses a mix before writing,
-and `adata recast` can convert one to match. var's columns and varm (a SCope
-import's `Regulons`, pycisTopic's `topic_region`) follow `--merge`, compared
-on the output's genes: an input lacking some of them, as in an outer join,
-counts as not having the entry. Without `--merge`, varm is dropped with a
-note.
-
-`obsp`, `varp` and `raw` are **not** carried over — a pairwise value has no
-meaning between cells from different inputs. anndata's own `concat_on_disk`
-refuses these too.
+and `adata recast` can convert one to match. var's columns, varm (a SCope
+import's `Regulons`, pycisTopic's `topic_region`) and varp follow `--merge`,
+compared on the output's genes: an input lacking some of them, as in an
+outer join, counts as not having the entry. Sparse varp entries are compared
+by value, whatever their format (CSR or CSC) and dtype, and stored zeros
+count as absent, as in anndata; a sparse entry never equals a dense one,
+where anndata's answer depends on which input comes first. Without
+`--merge`, varm and varp are dropped with a note.
+
+`obsp` and `raw` are **not** carried over — a value between two cells has no
+meaning when they come from different inputs. anndata's own
+`concat_on_disk` refuses these too.
+
+Dense matrices and obsm arrays keep the first input's chunking and
+compression, as sparse ones do. A dense integer matrix stays integer unless
+an outer join fills cells with a `--fill-value` it cannot hold, such as 0.5
+or NaN; then it becomes float64.
+
+concat opens its inputs as it needs them, at most 64 at once (or a quarter
+of the process's file descriptor limit, `ulimit -n`, if that is fewer), so
+it takes any number of inputs. CSC matrices are read once each, a block of entries at a
+time, and their columns put together through a scratch file beside the
+output, of about 16 bytes plus the value per stored entry of the largest
+one. The free-space check counts it, for a Zarr output too.
 
 ### Merge strategies
 

src/adata/cli.py

@@ -444,7 +444,9 @@ def concat(
         "(default: drop)",
     ),
     fill_value: float = typer.Option(
-        0.0, "--fill-value", help="Value for dense cells introduced by an outer join"
+        0.0, "--fill-value",
+        help="Value for dense cells introduced by an outer join; an integer "
+        "matrix that cannot hold it becomes float64",
     ),
     chunk_rows: int = _chunk_option(1024, "Row chunk size while streaming"),
     zarr_format: Optional[int] = typer.Option(
@@ -458,7 +460,8 @@ def concat(
     Concatenate stores along the obs axis.
 
     Streams each input in turn, so memory use is set by --chunk rather than by
-    the size of the inputs. obsp, varp and raw are not carried over.
+    the size of the inputs. obsp and raw are not carried over; var's columns,
+    varm and varp follow --merge.
 
     Examples:
         adata concat a.h5ad b.h5ad -o merged.h5ad

src/adata/commands/concat.py

@@ -9,7 +9,6 @@
 
 from adata.core.concat import MERGE_CHOICES, MERGE_STRATEGIES, concat_on_disk
 from adata.util.path import check_output
-from adata.util.staging import check_free_space, disk_size
 
 __all__ = ["MERGE_CHOICES", "MERGE_STRATEGIES", "concat_stores"]
 
@@ -25,9 +24,6 @@ def concat_stores(
         # concat never replaces an output (it refuses one that exists), but
         # a new one may still name a path inside a Zarr input.
         check_output(output, files)
-    # About the sum of the inputs: an inner join keeps less, an outer join's
-    # dense fill can add some.
-    check_free_space(
-        output, lambda: sum(disk_size(f) for f in files), check=space_check
-    )
-    concat_on_disk(files, output, console, **kwargs)  # type: ignore[arg-type]
+    # The free space is checked by concat_on_disk, once it knows whether a
+    # CSC matrix needs scratch space beside the output too.
+    concat_on_disk(files, output, console, space_check=space_check, **kwargs)  # type: ignore[arg-type]