emit: Crop-Independent Emissions bands, from which the existing bands derive

.dockerignore

@@ -0,0 +1,23 @@
+# Claude worktrees and other local tooling state.
+.claude/
+
+# A host venv, at any depth, must not shadow the one built inside the image.
+**/.venv/
+
+# Exact-patch pin (e.g. 3.14.5) a base image's uv may not fetch; requires-python (>=3.14,<3.15)
+# is satisfied by any 3.14.x, so let uv use the base image's own.
+.python-version
+
+# Secrets: supply .env at runtime, never bake it into the image. `*.env` matches any env file
+# but not the `*.env.example` templates, which stay.
+.env
+**/*.env
+
+# Caches / build noise.
+**/__pycache__/
+*.pyc
+.mypy_cache/
+.pytest_cache/
+.ruff_cache/
+.cache/
+.DS_Store

.env.example

@@ -1,3 +1,7 @@
+# EXPORT_ROOT is where terminal deliverables land (the emissions-factor table): not a cache
+# the pipeline reads back. Any fsspec URI, like the roots below.
+# EXPORT_ROOT = '/absolute/path/to/export'
+EXPORT_ROOT = 'gs://cornerstone-emissions-us-central1'
 HARVARD_DATAVERSE_GUESTBOOK_JSON = '{"email":Replace with your email address,"institution":Replace with your institution}'
 # INGEST_ROOT and SCRATCH_ROOT take any fsspec URI. A local root must be an absolute path
 # that already exists; a gs:// root needs application-default credentials.

.pre-commit-config.yaml

@@ -18,7 +18,7 @@ repos:
       - id: mypy
         name: mypy
         language: system
-        entry: uv run mypy jdluc validation
+        entry: uv run mypy jdluc orchestration validation
         pass_filenames: false
   - repo: https://github.com/astral-sh/uv-pre-commit
     rev: 0.12.7

Dockerfile

@@ -0,0 +1,25 @@
+# syntax=docker/dockerfile:1.19
+# Build from a full clone, not a git worktree: a worktree's .git is a file pointing outside the
+# build context, and the image needs the repository for git provenance (utils.get_git_version).
+
+FROM ghcr.io/astral-sh/uv:python3.14-bookworm-slim
+
+WORKDIR /app
+
+# The rasterio/GDAL wheels vendor GDAL but still link libexpat, which the slim image omits.
+# safe.directory lets git read /app when the container runs as a user other than its owner.
+RUN apt-get update \
+    && apt-get install -y --no-install-recommends git libexpat1 \
+    && rm -rf /var/lib/apt/lists/* \
+    && git config --system --add safe.directory /app
+
+COPY pyproject.toml uv.lock ./
+RUN uv sync --frozen --no-install-project --no-dev
+
+COPY --exclude=.git . .
+RUN uv sync --frozen --no-dev
+
+# .git in its own last layer, so a commit that changes no source rebuilds only this layer.
+COPY .git .git
+
+ENV PATH="/app/.venv/bin:${PATH}"

README.md

@@ -45,6 +45,7 @@ Once you're ready to look under the hood:
 - `docs/data.md` — how to get the data products, grids, and full schemas for the harmonized inputs, per-pixel emissions, and emissions-factor table.
 - `docs/coverage.md` — the full list of countries and crops the pipeline produces emissions factors for, by ISO 3166-1 alpha-3 code.
 - `docs/validation.md` — how the pipeline is measured against external datasets: how the (country, crop) targets are chosen, which relationships are held as regression controls, and what such a comparison can and cannot conclude. The tooling lives in `validation/` and reports against the gaps in `docs/further_research.md`.
+- `docs/orchestration.md` — an optional way to run the pipeline on Kubernetes with Argo Workflows, one tile per pod, for country runs and global backfills in hours rather than days.
 
 ## Running and contributing
 

docs/crop_independent_emissions.md

@@ -0,0 +1,70 @@
+# Crop-Independent Emissions (CIE)
+
+`emit` writes a second set of bands, prefixed `cie-`, beside its destination-gated output (see [`data.md`](data.md#2-per-pixel-emissions-zarr)). They describe the same land conversions and carbon pools, but stop **before** anything that depends on what the land became:
+
+- **No destination gate.** A pixel is charged as soon as it lost a source class (forest, natural grassland or pasture) inside the 20-year window, whether or not a destination dataset claims it in the assessment year.
+- **No destination-specific factor.** Mineral soil is the full stock at risk. The cropland loss fraction is not applied.
+- **No discount.** Each pool is a single undiscounted value. The span split and the 20-year linear discount are left to the consumer.
+
+This makes the `cie-` bands a common input from which any destination- or crop-specific accounting can be derived: a consumer that wants a different soil factor per crop, a different discount, or the emissions a conversion would carry under a destination other than the one observed, can compute it from these bands without rerunning `emit`. The existing bands are one such derivation (see [Deriving the existing bands](#deriving-the-existing-bands-from-cie) below). The code is `emit.get_crop_independent_emissions`.
+
+## Side by side
+
+| quantity                | existing band(s)                                                                                                                                                                                                      | CIE band                                                                                                                                                                                | difference                                                                             |
+| ----------------------- | --------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | --------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | -------------------------------------------------------------------------------------- |
+| what happened           | `conversion`: source × destination, 0–5 (`emit.Conversion`)                                                                                                                                                           | `cie-conversion-source`: 0 none · 1 forest · 2 natural grassland · 3 pasture (`emit.ConversionSource`)                                                                                  | source only; set whether or not a destination resolved                                 |
+| when                    | `conversion-year`                                                                                                                                                                                                     | `cie-conversion-year`                                                                                                                                                                   | copy: the year is already 0 wherever there is no source                                |
+| vegetation              | `vegetation-emissions:tco2e-per-ha:{span}` ×4                                                                                                                                                                         | `cie-vegetation-emissions-undiscounted:tco2e-per-ha`                                                                                                                                    | one band rather than one per span; not gated on destination                            |
+| soil                    | `soil-emissions:tco2e-per-ha:{span}` ×4, one band holding either term: on peat, the 621 tCO2e/ha pulse; off peat, SOC × 44/12 × the cropland loss fraction for a cropland destination, or 0 for a pasture destination | two bands: `cie-mineral-soil-carbon-at-risk:tco2e-per-ha` (SOC × 44/12 off peat, 0 on peat) and `cie-peat-transformation-emissions-undiscounted:tco2e-per-ha` (621 on peat, 0 off peat) | mineral and peat kept apart; no loss fraction on mineral; neither gated on destination |
+| peat occupation         | `cropland-peatland-occupation:tco2e-per-ha`, `pastureland-peatland-occupation:tco2e-per-ha`: 37.3 on peat under that destination                                                                                      | `cie-peat-occupation-emissions:tco2e-per-ha-per-year`: 37.3 on all peat                                                                                                                 | the potential, whatever the destination and whether or not the pixel was converted     |
+| discounted total        | `emissions-per-hectare:tco2e-per-ha`                                                                                                                                                                                  | —                                                                                                                                                                                       | the discount is applied downstream                                                     |
+| unclaimed source carbon | `dropped-emissions:tco2e-per-ha`                                                                                                                                                                                      | —                                                                                                                                                                                       | these pixels are carried in the CIE pools themselves                                   |
+| destination evidence    | `destination-dataset`: bitmask, 1 Descals oil palm · 2 GACED30 cropland · 4 GPW cultivated grassland                                                                                                                  | `cie-destination-dataset`                                                                                                                                                               | copy                                                                                   |
+| climate zone            | — (input only)                                                                                                                                                                                                        | `cie-climate-zone`: `ipcc_climate_zones.Zone`, 1–10                                                                                                                                     | new: needed to look up destination-specific soil factors                               |
+| pixel area              | `hectares-per-pixel:ha`                                                                                                                                                                                               | `cie-hectares-per-pixel:ha`                                                                                                                                                             | copy                                                                                   |
+
+`{span}` is one of `2000-2005`, `2005-2010`, `2010-2015` and `2015-2020`. A conversion falls in span `(a, b)` when `a < year ≤ b`. The copied bands are duplicated so that the `cie-` set is complete on its own.
+
+## Deriving the existing bands from CIE
+
+Every existing band can be computed from the `cie-` bands. `emit.derive_from_cie` does so, though `jurisdictional_direct` and `statistical` still read the existing bands directly. The only extra inputs are two lookup tables already in `emit`: the soil loss fraction and the discount weights. Below, `ConversionSource` and `Conversion` refer to the band values in the table above.
+
+**Destination.** Cropland outranks pasture:
+
+- `to_cropland = cie-destination-dataset & (1 | 2)`
+- `to_pasture = ¬to_cropland ∧ cie-destination-dataset & 4`
+
+**Conversion.** It fired where `cie-conversion-source ≠ NONE ∧ (to_cropland ∨ to_pasture)`, except pasture to pasture, which is not a conversion. The `conversion` value is the pair of the two: forest to cropland, forest to pasture, natural grassland to cropland, natural grassland to pasture, and pasture to cropland. A pasture-to-pasture pixel appears in neither the per-span pools nor `dropped-emissions`.
+
+**Per-span pools.** Each pool is set only where a conversion fired and `cie-conversion-year` is in the span:
+
+- `vegetation-emissions:{span}` = `cie-vegetation-emissions-undiscounted`
+- `soil-emissions:{span}` = `cie-peat-transformation-emissions-undiscounted` + `f(cie-climate-zone)` × `cie-mineral-soil-carbon-at-risk` for a cropland destination
+- `soil-emissions:{span}` = `cie-peat-transformation-emissions-undiscounted` for a pasture destination
+- the per-span total, `e(span)`, is vegetation + soil (not itself a band)
+
+Here `f` is the cropland soil loss fraction, 1 − retention (IPCC 2019, Vol 4, Table 5.5; `emit.CLIMATE_ZONE_TO_SOC_RETENTION_FRACTION`).
+
+**Occupation.** Each band is gated on its destination only:
+
+- `cropland-peatland-occupation` = `cie-peat-occupation-emissions` where `to_cropland`
+- `pastureland-peatland-occupation` = `cie-peat-occupation-emissions` where `to_pasture`
+
+**Discounted values.** Both use the span weights `w` = 0.0125, 0.0375, 0.0625 and 0.0875, which sum to 0.2 (`emit.SPAN_TO_LINEAR_DISCOUNT_WEIGHT`):
+
+- `emissions-per-hectare` = Σ<sub>span</sub> `w` × `e(span)` + both occupation bands
+- `dropped-emissions` = Σ<sub>span</sub> `w` × `cie-vegetation-emissions-undiscounted`, where there is a source, the year is in the span, and neither `to_cropland` nor `to_pasture` holds
+
+The vegetation, soil and year identities are unit-tested in `jdluc/__tests__/emit_test.py` (`test_cie_*`), and `test_derive_from_cie_reproduces_every_band_it_replaces` checks every existing band against `derive_from_cie` on a grid covering each event, destination and soil case. `tools/compare-emit-layers.py` checks the vegetation and year identities on a real tile, and the largest difference between each existing band and its derived counterpart. It also totals the old and new pools by source, and by whether a conversion fired, which shows how much carbon the CIE bands carry that the existing bands leave uncharged.
+
+## Encoding
+
+- Every band is float32 with NaN as no-data, including the enum and bitmask bands (`geo.unify_dtype_and_no_data`).
+- Bands share the tile's 30 m grid. The units follow the band name after the first `:`.
+- Adding the `cie-` bands takes each tile from 16 bands to 25. The `emit` cache version is 5.
+
+## Not yet covered
+
+- **Assessment year.** It is fixed at 2020 (`emit.ASSESSMENT_YEAR`), which sets the lookback window and the destination predicates.
+- **Continent.** Some destination-specific factors vary by continent, which no ingested dataset provides yet.
+- **Land cover at the start of the window.** There is no band for it. A forest bit would need a year-2000 tree-cover extent, which is not currently ingested.

docs/data.md

@@ -10,7 +10,7 @@ Of the three artifacts below, one is published and two are not.
 
 **The emissions-factor table is published as an archived dataset.** Each data version is deposited as `emissions-factors.parquet` in a public data archive that mints a DOI per version, and the matching git tag names the commit behind it. The archive is still being set up; cite a version by its DOI, or follow the fallback in `ATTRIBUTION.md`.
 
-**The two raster artifacts are pipeline outputs.** They are zarr stores far too large to deposit, so they exist only where a run writes them: under the `SCRATCH_ROOT` of whoever ran the pipeline, keyed by the cache hash `storage.py` derives (see [`architecture.md`](architecture.md#caching-versioning-and-provenance)). Sharing the underlying maps in a form that can be read remotely is open work — the schemas below are documented so a run's own output is legible in the meantime.
+**The two raster artifacts are pipeline outputs.** They are zarr stores far too large to deposit, so they exist only where a run writes them: under the `SCRATCH_ROOT` of whoever ran the pipeline, keyed by the cache hash `storage.py` derives (see [`architecture.md`](architecture.md#caching-versioning-and-provenance)). Sharing the underlying maps in a form that can be read remotely is open work — the schemas below are documented so a run's own output is legible in the meantime. A run can also write a step's per-tile output as a COG or a zarr under `EXPORT_ROOT` with `export.write_tile`; that copies the maps out of the cache, but does not by itself publish them.
 
 ## Conventions
 
@@ -72,6 +72,20 @@ The crop-agnostic, per-pixel, per-span LUC emissions produced by the `emit` stag
 | `emissions-per-hectare:tco2e-per-ha`                 | float32 | t CO₂e/ha       | 20-year linearly-discounted LUC emissions + peatland occupation — the headline per-hectare layer         |
 | `hectares-per-pixel:ha`                              | float32 | ha              | Pixel area (varies with latitude)                                                                        |
 
+Beside these, the `cie-` bands carry Crop-Independent Emissions: the same conversions and pools, stopped before anything that depends on what the land became (no destination gate, no destination-specific soil factor, no discount). Every band above can be derived from them; see [`crop_independent_emissions.md`](crop_independent_emissions.md).
+
+| Variable                                                      | Type    | Units           | Description                                                                                                      |
+| ------------------------------------------------------------- | ------- | --------------- | ---------------------------------------------------------------------------------------------------------------- |
+| `cie-conversion-source`                                       | float32 | Source code     | The class the pixel left inside the window, whatever it became: 0 none, 1 forest, 2 natural grassland, 3 pasture |
+| `cie-conversion-year`                                         | float32 | year            | As `conversion-year`                                                                                             |
+| `cie-vegetation-emissions-undiscounted:tco2e-per-ha`          | float32 | t CO₂e/ha       | Vegetation carbon lost with the source class, undiscounted and not gated on a destination                        |
+| `cie-mineral-soil-carbon-at-risk:tco2e-per-ha`                | float32 | t CO₂e/ha       | The full mineral soil-carbon stock under a converted pixel off peat, before any land-use factor                  |
+| `cie-peat-transformation-emissions-undiscounted:tco2e-per-ha` | float32 | t CO₂e/ha       | The peatland drainage pulse under a converted pixel on peat, undiscounted                                        |
+| `cie-peat-occupation-emissions:tco2e-per-ha-per-year`         | float32 | t CO₂e/ha/yr    | Annual peatland-occupation emissions on all peat, whatever occupies it                                           |
+| `cie-climate-zone`                                            | float32 | Zone code       | IPCC climate zone, for looking up destination-specific soil factors                                              |
+| `cie-destination-dataset`                                     | float32 | Dataset bitmask | As `destination-dataset`                                                                                         |
+| `cie-hectares-per-pixel:ha`                                   | float32 | ha              | As `hectares-per-pixel:ha`                                                                                       |
+
 ```python
 import xarray