Repository navigation
Conversation
For zarr files, set up a .complete marker file to check when data is fully written. This replaces relying on zarr.json -- that doesn't work because zarr.json is the first to be written. This might happen from a failed or aborted calculation, or because multiple processes are trying to write the same file (e.g. running 2 countries which share a tile) A partially-written zarr might be not cause any warning signs later, since it might be techncially valid in spite of missing data. Now an empty .complete marker is written beside the zarr after its last chunk, and the entry counts as cached only once the marker exists. A zarr found without its marker is a write that was killed part-way or is still running: serialize raises FileExistsError and leaves it for someone to resolve by hand, usually by deleting the zarr, rather than writing over it and risking a race with a live writer. Entries written before this change have no marker, so each raises once until deleted.
An autouse fixture in jdluc/__tests__/conftest.py stubs Config.from_dot_env for every test not marked integration. Each field gets its type's empty value, with two dask workers, so a unit test that reaches for a storage root fails instead of touching a bucket. Integration tests still read the real .env. This replaces the fixture storage_test carried for itself, so the next unit test that writes a zarr needs no copy of it, and a new Config field needs no edit to the tests.
An optional `orchestration` package with one subcommand per workflow task (plan, ingest-tile, compute, trace), and an example Argo WorkflowTemplate that runs each tile's ingest -> compute as a DAG of its own, so a failed tile stops only itself. jdluc runs unmodified in the pods; every cache-key argument passes straight through, and an import-linter contract keeps jdluc from importing orchestration. trace writes the emissions-factor table to a new EXPORT_ROOT, through a small `export.write_table` utility. The package is built to be wrapped rather than edited: `build_parser` and `run` let a wrapper add subcommands, `plan.get_task_to_items` returns the lists for it to add to before `write_task_to_items`, and the template's tasks can be called by templateRef. See docs/orchestration.md. Also adds the Dockerfile the workflow's image is built from.
get_name no longer mentions mosaics, get_task_to_items describes the flat lists without a flat workflow to point at, and trace's --help is a whole sentence.
A Python API only: nothing calls it, a workflow opts in. Writes the chosen
variables of a lazy dataset to {EXPORT_ROOT}/{name}/{tile_id}.{tif|zarr},
provenance from get_metadata.
- COG: every variable as float32 2-D bands, NaN no-data; extra dims are
flattened to one band per combination of values, named
variable:value[:value...]. Staged locally and uploaded whole, through
storage.write_dask_dataset_to_cog (moved into storage with stack_bands).
- zarr: variables as they are, dtypes and extra dims kept. An export is
re-runnable, so it writes over an existing zarr (write_dask_dataset_to_zarr
gains mode=, default "w-" so the cache is unchanged), dropping the
.complete marker first and touching it once the write finishes.
…on}/
write_tile and write_table take version=, such as a run id or a release
label. Blank, the default, writes directly under EXPORT_ROOT as before; set,
the whole export goes under {EXPORT_ROOT}/{version}/ and the version is
stamped as cornerstone-export-version. A version must be one path segment.
Also from review:
- write_tile rejects a bare str as variables, rather than reading its letters
- the COG compute runs under COMPUTE_LOCK, as the zarr write does
- iter_bands notes how datetime and float coords name their bands
- to_zarr_dataset says why only coords become variable-length strings
A detached HEAD, as CI checks out a pull request, has no active branch, so get_git_version raised and with it every export that stamps provenance. It now returns the short commit SHA. No .git still fails, as before.
… derive emit writes nine `cie-` bands beside its output: the conversion source and year, and each carbon pool the conversion puts at risk, before any destination gate, destination-specific soil factor or discount. `derive_from_cie` rebuilds every existing band from them alone, and a test checks it does on a grid covering each event, destination and soil case. Additive: the existing bands, and every consumer of them, are unchanged. `CIE_BANDS` names the bands for `export.write_tile`. docs/crop_independent_emissions.md sets the two side by side; tools/compare-emit-layers.py checks the identities on a real tile.
For every pixel and each crop class (annual, perennial, paddy rice, pasture), the emissions if that class occupied it: F_LU by crop class and climate zone on mineral soil, a peat occupation factor by crop class and peat climate regime, and a choice of discounting over the 20-year window. Nothing is stored; a product names the fields, crop classes and discountings to write, and `export.write_tile` writes them per tile from `emit`'s cached output. Annual crops take `emit`'s cropland retention fractions, so with five-year-span discounting the layer reproduces `emit`'s cropland and pasture bands wherever the destination matches, which the tests check.
This was referenced Oct 19, 2026
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
jdluc/derive.pyadds a crop-class layer. For every pixel and each crop class (annual, perennial, paddy rice, pasture), it computes the emissions if that class occupied the pixel. It reads onlyemit'scie-bands, and adds the three things CIE leaves out:cie-mineral-soil-carbon-at-risk × (1 − F_LU), with F_LU by crop class and climate zone, capped at 1.linear,equal, andfive-year-spans(the weightsemituses).Fields carry a
crop_classdim where they depend on a factor, and adiscountingdim where they are discounted. Nothing is stored. AProductnames the fields, crop classes and discountings to write, andexport.write_tilewrites them per tile fromemit's cached output, computing only what the product names. There are three products:crop-class-emissions(the default),perennial-debug(zarr, every field) andperennial-discounting-comparison.CLI:
python -m jdluc.derive HND [--product=…] [--format=cog|zarr] [--version=…] [--modulus/--residues]. The flags followemit's.Notes
write_tile. A product's selection applies to the whole dataset, and COG bands are named<variable>:<crop class>:<discounting>.emit.CLIMATE_ZONE_TO_SOC_RETENTION_FRACTIONitself. So derive andemitagree on every climate zone, and the regression test againstderive_from_ciecovers them all.jdluc.derivesits between JD/statistical andemit, and is forbidden to the utilities.docs/crop_independent_emissions.mdgains a section on the layer.Open questions
Testing
uv run pytest jdluc orchestration: 1476 passed.uv run pre-commit run --all-files: passes, including import-linter and mypy.python -m jdluc.derive --help.