Repository navigation
Conversation
An optional `orchestration` package with one subcommand per workflow task (plan, ingest-tile, compute, trace), and an example Argo WorkflowTemplate that runs each tile's ingest -> compute as a DAG of its own, so a failed tile stops only itself. jdluc runs unmodified in the pods; every cache-key argument passes straight through, and an import-linter contract keeps jdluc from importing orchestration. trace writes the emissions-factor table to a new EXPORT_ROOT, through a small `export.write_table` utility. The package is built to be wrapped rather than edited: `build_parser` and `run` let a wrapper add subcommands, `plan.get_task_to_items` returns the lists for it to add to before `write_task_to_items`, and the template's tasks can be called by templateRef. See docs/orchestration.md. Also adds the Dockerfile the workflow's image is built from.
get_name no longer mentions mosaics, get_task_to_items describes the flat lists without a flat workflow to point at, and trace's --help is a whole sentence.
338c953 to
c6e3b0a
Compare
| # EXPORT_ROOT is where terminal deliverables land (the emissions-factor table): not a cache | ||
| # the pipeline reads back. Any fsspec URI, like the roots below. | ||
| # EXPORT_ROOT = '/absolute/path/to/export' | ||
| EXPORT_ROOT = 'gs://cornerstone-emissions-us-central1' |
There was a problem hiding this comment.
EXPORT_ROOT is the active line here and it points at the real bucket. Anyone copying this file and running trace would write straight into it. Could we make the placeholder the active line and keep the bucket commented out?
| resources: { requests: { storage: 150Gi } } | ||
|
|
||
| # harmonize -> emit -> attribute for one tile, from the warm ingest cache, without keys | ||
| - name: compute |
There was a problem hiding this comment.
No retryStrategy on any template, so a reclaimed spot node means a manual argo retry. Maybe retryStrategy: {limit: 2, retryPolicy: OnError} on ingest-tile and compute? The cache makes retries cheap.
| RUN uv sync --frozen --no-dev | ||
|
|
||
| # .git in its own last layer, so a commit that changes no source rebuilds only this layer. | ||
| COPY .git .git |
There was a problem hiding this comment.
This copies the whole .git into the image. If it's built from a clone with a token in the remote URL, the token ends up in the image (get_git_remote_url reads that URL). Could we pass the SHA and remote as build args instead, or strip the credentials? Also, the container runs as root, and the safe.directory line above suggests a non-root user was intended.
deliverable_name becomes product_name in write_table and get_table_output_uri, matching the cornerstone-product-name tag that get_metadata already writes from it.
orchestration: run the pipeline on Argo, one tile per pod (cornerstone-data#19): write_table names its product # Conflicts: # jdluc/export.py
Brings in cornerstone-data#19's write_table product_name rename.
|
@danohu, One more thing building on @tankoyarik comment about .git/credentials in the Dockerfile: export.get_metadata() stamps utils.get_git_remote_url() into cornerstone-remote-url on every exported table's attrs, and the new test confirms attrs survives the to_parquet/read_parquet round-trip. So if the image is ever built from a clone whose remote URL carries a token, that credential doesn't just sit in the image. It propagates into the metadata of every emissions-factors parquet handed downstream. Worth stripping credentials from the URL before it's used as provenance, not just before it's baked into the image. |
Stacked on #16 (zarr
.completemarker): the docs here rely on a killed pod leaving no zarr that passes for a result. Review #16 first; this PR's diff is only the orchestration work.What
An optional layer for running the pipeline on Kubernetes with Argo Workflows, one 10° tile per pod, so that a country or a global backfill takes hours rather than days.
jdlucruns unmodified inside the pods; Argo takes over only sequencing, fan-out, retries and visibility.docs/orchestration.mdhas the design and setup.orchestrationpackage, one subcommand per workflow task:plan,ingest-tile,compute,tracejdluc/export.py+trace.export_workflow: write the emissions-factor table, with provenance, to a newEXPORT_ROOTorchestration/workflows/cornerstone.example.yaml: an ArgoWorkflowTemplate, one DAG per tile, with cloud-neutral placeholdersDockerfile,.dockerignorejdlucnever importsorchestration), and mypy overorchestrationDesign notes
orchestrationmay not change a result. Every cache-key argument goes straight through tojdluc, withjdluc's own defaults.skip_ingest=Truefrom a warmINGEST_ROOT. That needs the list of datasets a run reads, whichorchestration/plan.pykeeps by hand.traceruns only once every tile has succeeded.argo retryresumes from the cache.__main__.build_parser()returns the parser and its subparsers,plan.get_task_to_items()returns the plan lists without writing them, and the template's tasks can be called withtemplateRef.Heads-up
EXPORT_ROOTis a new.envkey, andConfig.from_dot_env()requires every key, so an existing.envneeds it added (see.env.example).Verification
pytestpasses, including 18 new tests inorchestrationpre-commit run --all-filespasses (ruff, import-linter, mypy, mdformat)argo lint --offlineon the example templatedocker build