feat: ingest and store CWL processes, flat per transformation by ryuwd · Pull Request #1097 · DIRACGrid/diracx

ryuwd · 2026-10-19T14:42:22Z

Refs #1080 (CWL Processes ingestion), following the discussion there. Not marked as closing it: the storage layout differs from the issue's original plan (see below) and should be agreed first.

How to review

About 1,500 of the ~5,750 added lines are source, in 8 files. The rest is tests (1,700), pixi.lock (1,150), CWL fixtures (700), the benchmark (500) and docs. Reading the source in this order should be enough:

docs/dev/explanations/cwl-process-store.md (~140 lines): the design in prose, worth reading first.
diracx-core/src/diracx/core/cwl/canonical.py (650): the canonical form and identity. The module docstring numbers the rules, and the code follows them in order. This is the file where a mistake would change identities, so it deserves the closest look.
diracx-core/src/diracx/core/cwl/policy.py (85): what is accepted beyond valid CWL.
diracx-logic/src/diracx/logic/transformation/processes.py (260) and validation.py (85): storing and loading, the operator-only replacement, and how cwltool is confined (no fetching, no node).
diracx-db/src/diracx/db/sql/transformation/db.py (160): six small query methods, with no schema change.
diracx-cli/src/diracx/cli/cwl.py (200): the client-side YAML loader. It never runs in a service.

The first nine commits follow the same order (dependencies, core, CLI, DB, logic, replacement, CI, docs, benchmark), and each passes its own tests, so reviewing commit by commit works too. A tenth accepts DockerRequirement.

What can be skimmed or skipped:

Tests: their names describe the behaviour, so the names are a good checklist. Reading their bodies is optional.
Fixtures: diracx-cli/tests/fixtures/mc-simulation/ is the MCSimulation workgraph from [TransformationSystem] CWL Processes ingestion #1080 as YAML files, and diracx-logic/tests/transformation/fixtures/mc-simulation.json is the same document resolved to JSON, as the endpoint will receive it.
pixi.lock: generated, for the cwltool and ruamel.yaml dependencies.
benchmarks/process_store/: the measurements behind the flat-storage decision, which are summarised below. Not run by CI or shipped.
.github/workflows/main.yml: one new job that runs the diracx-db transformation tests on MySQL and MariaDB service containers.

What this adds

diracx.core.cwl (shared by client and services)
- canonical: the canonical form and SHA-256 identity of a CWL process, defined over the parsed document rather than any library's output (approach B in [TransformationSystem] CWL Processes ingestion #1080). cwl_utils.parser.save() was not usable: it is not deterministic across loads and leaks absolute paths. Child run bodies are replaced by their identity, so a workflow's identity covers its children's (a Merkle tree). pack/unpack convert to and from a self-contained $graph whose components are named by their identity.
- policy: an allow-list of requirements every backend can honour, derived from a year of real documents, and a size cap per process. DockerRequirement is on it, to run with Apptainer, for an image to pull (dockerPull, dockerImageId); dockerFile, dockerLoad and dockerImport are refused, in hints too, since they build or fetch an image on the worker node.
diracx.cli.cwl: the loader. It reads CWL as YAML 1.2 (as runners do) and inlines each step's run path, $import, $include and #fragments, refusing cycles, deep nesting, oversized files and remote URLs. YAML stays in the CLI: services only ever receive the resolved JSON. They refuse any $import, $include or other directive left in it, and cwltool is given a fetcher that reads only the packed document.
TransformationDB: insert_processes (INSERT IGNORE) and get_processes, on the DX-ADR-004 Processes table unchanged.
diracx.logic.transformation: store_process canonicalises, checks the policy, returns early if the process is already stored, otherwise validates the packed payload with cwltool (schema and static checker) and stores it. load_process is a single read of what a parcel runs.
Docs: docs/dev/explanations/cwl-process-store.md.
Benchmarks: benchmarks/process_store/ measures flat against tree storage on a corpus of submissions.

Design decision: flat per transformation

Each transformation's whole subgraph is one Processes row (a packed $graph), not one row per CWL process with ProcessReferences edges as DX-ADR-004/007 describe. The identity is the Merkle root either way, and the packed document names every component by its own hash, so per-component rows or a lineage index can be derived later without changing any identity.

Measured on a year of LHCb submissions (84,406 workgraphs, 182,071 transformations, every AnalysisProductions tag and mc-requests merge of the year, converted with LbAPCommon), on MySQL 8.4:

	flat	tree
`Processes` rows	89,779	169,872
on disk	623 MiB	504 MiB + 32 MiB edges
compressed (zstd)	121 MB	124 MB
lookup p50 / p99, 1 client	2.1 / 6.6 ms	2.3 / 9.5 ms
lookups/s, 8 clients	2,787	2,169

The tree deduplicates little, because the variation between jobs sits in the leaf tools. Flat is a single read at dispatch and is what a stock runner executes. Validation runs only on processes not already stored: 16–19 ms each, about 25 CPU-minutes for the year.

Operator-only: replacing a transformation's process

For a hotfix (say, a software version change), an operator can make a paused transformation run a different process: replace_transformation_process in diracx.logic.transformation.processes. End users cannot: it requires the ProductionManagement or Operator property, and the submission API offers no such change. A stored process itself never changes, because its key is its hash; what changes is which process the transformation points at.

It refuses unless:

the transformation is Paused (its row is locked);
a reason is given;
the replacement passes the same policy check and cwltool validation as any submission;
the replacement keeps the input and output names and types, because the feeder, the packer, the declared outputs and downstream edges depend on them.

Then, in the caller's transaction, it repoints ComputeTransformations.ProcessID and the transformation's Unassigned parcels, and records ProcessReplaced in TransformationLog (old and new identities, parcels repointed, operator, reason). Parcels from Reserved on keep the process they were created with.

This is why ComputeParcels.ProcessID must stay a column of every compute parcel, rather than being derived from the transformation as I suggested in the DX-ADR-004 scalability notes. Once a transformation's process can change, a parcel's own copy is the only record of what it ran.

It departs from DX-ADR-004 ("ProcessID is frozen once any parcel of the transformation exists") and DX-ADR-005 (a wrong configuration is cancelled and resubmitted). Both still hold for owners; this is the operator's exception.

Deferred

Lineage/provenance (ProcessReferences, or a component index), references to arbitrary components by hash, ProcessRegistry, garbage collection, VO visibility.
Recording CANONICAL_VERSION per row (needs a schema change).
Open questions from [TransformationSystem] CWL Processes ingestion #1080: pinning mutable references (container tags: a dockerPull tag is accepted as written, so one identity can name different software if the tag moves), whether doc/label count towards identity, renaming step ids.
Workgraph decomposition (StrippedSpec, hints into Transformations columns, Actions) and the submission route.
For the workgraph decomposition, which decides what each transformation's process contains:
- Inherited requirements: CWL passes a workflow's requirements and hints, and a step's own, down to the step's process. A transformation stored on its own loses them. For example, with EnvVarRequirement: {APP_VERSION: v57r1} on the workgraph, cwltool running the whole workgraph sees v57r1, but the transformation run alone, as a parcel would run it, sees nothing. When the workgraph is split, they must be copied into each transformation's process before it is hashed.
- Workgraph wiring: each transformation is validated on its own, so the types of the edges between them are not checked. Submission should validate the whole workgraph once as well.
- Validation and the transaction: cwltool takes about 20 ms per new transformation. Validation should happen before the submission transaction and off the event loop, with only the insert inside it, so that a large submission does not hold Processes locks.

Notes for reviewers

New dependencies: cwltool (diracx-logic) and ruamel.yaml (diracx-cli). ruamel.yaml is still in the services image as a dependency of cwltool, but no service code uses it; the JSON cwltool reads is always JSON we wrote.
A process may only use named types (SchemaDefRequirement) that it defines itself, so its identity covers their definitions. Inheriting them from the parent workflow is refused for now; allowing it later is safe.
The [TransformationSystem] CWL Processes ingestion #1080 MCSimulation example lacks SubworkflowFeatureRequirement, so stock cwltool rejects it; the test fixture adds it.
The diracx-db transformation tests, including the JSON round trip (test_round_trip.py) and the methods a replacement uses, also run on every server given by DIRACX_TEST_MYSQL_URL / DIRACX_TEST_MARIADB_URL. A new CI job runs them on MySQL 8.4 and MariaDB 11.4 service containers, where MySQL visibly re-sorts keys and reformats numbers.

Fixes from the Copilot review

Each fix has a regression test and is folded into the commit it belongs to:

the service refuses unresolved $include/$import directives, which cwltool would otherwise have fetched on the server;
the loader treats run as a reference only on workflow steps;
dirac: requirements are refused rather than silently dropped;
names a user chose that are called doc or label (such as envDef keys) stay in the identity;
namespace prefixes are expanded to full URIs before $namespaces is dropped;
the scope is stripped only from fully scoped identifiers, and step ports are shortened against the step;
named types stay valid through packing and are compared by definition when a process is replaced;
unpack checks each shared component once and refuses a component that runs itself;
the benchmark applies the same acceptance rules to both layouts.

Since the review

runs in steps that come from an $import are resolved, relative to the imported file, found while filling coverage gaps;
cwltool no longer starts node to lint JavaScript expressions, nor pulls a node container when there is none;
type shorthands are expanded in the canonical form (File[] is {type: array, items: File}, File? is [null, File]), so the identity no longer depends on which tool turned a document into JSON: our loader and cwl_utils.pack now agree. This changes identities, including the golden one; CANONICAL_VERSION stays 1, as no identity has been stored outside tests;
the loader moves from diracx-core to diracx-cli with ruamel.yaml, its tests and the YAML MCSimulation fixture; diracx-logic now tests with the resolved JSON fixture, as the endpoint will receive it;
DockerRequirement is accepted for images that are pulled, so that workflows such as CTAO's run; images built or fetched on the worker node are refused;
the policy refuses abstract Operations, which are valid CWL but which no runner can execute, asking for a CommandLineTool, ExpressionTool or Workflow with the same inputs and outputs instead;
tests for the remaining paths, with the diracx-db transformation tests (JSON round trip, row locking, repointing) also run on real servers, and a CI job that runs them on MySQL 8.4 and MariaDB 11.4.

Testing

diracx-core: 193 passed; diracx-db: 175 passed; diracx-logic: 53 passed; diracx-cli: 42 passed
diracx-db transformation tests on MySQL 8.4 and MariaDB 11.4: 27 passed, locally and in the new CI job
A sample of 3,000 of the year's workgraphs still canonicalises, passes the policy and survives pack/unpack
pre-commit clean

read-the-docs-community · 2026-10-19T14:44:03Z

Documentation build overview

📚 diracx | 🛠️ Build #35330122 | 📁 Comparing 2013a8a against latest (a59148a)

🔍 Preview build

2 files changed

+ dev/explanations/cwl-process-store/index.html
± dev/how-to/contribute/index.html

Copilot

🟡 Changes recommended

Unresolved issues affect executable-content preservation, service-boundary validation, and replacement interface compatibility.

10 open issues

Preserve namespace bindings during canonicalization · New
Shorten step-local ports against the step scope · New
Retain mandatory requirements for policy checks · New
Preserve process prefixes in relative references · New
Reject unresolved directives before hashing or validation · New
Compare resolved named type definitions · New
Apply validation and policy checks to tree storage · New
Cache shared components during graph traversal · New
Preserve executable data keys during metadata filtering · New
Resolve run only on workflow steps · New

What changed in this PR

Adds content-addressed CWL ingestion to DiracX’s transformation system, storing each transformation’s complete process graph in one row for single-read retrieval. It also introduces operator-only process replacement for paused transformations.

Changes:

Adds shared CWL loading, canonical identities, packing, and policy checks.
Adds validated process storage and audited replacement of transformation and unassigned-parcel process references.
Adds regression tests, simulation fixtures, documentation, and flat-versus-tree benchmarks.

File	Description
`pyproject.toml`	Adds benchmark-specific lint exceptions.
`mkdocs.yml`	Links the process-store documentation.
`docs/dev/explanations/cwl-process-store.md`	Explains storage, identity, ingestion, and replacement.
`diracx-logic/tests/transformation/test_validation.py`	Tests schema and workflow wiring validation.
`diracx-logic/tests/transformation/test_round_trip.py`	Tests identity preservation through MySQL and MariaDB.
`diracx-logic/tests/transformation/test_replace_process.py`	Tests replacement permissions, compatibility, and parcel updates.
`diracx-logic/tests/transformation/test_processes.py`	Tests storage, deduplication, and stored references.
`diracx-logic/tests/transformation/test_mc_simulation.py`	Tests simulation ingestion and shared-process identities.
`diracx-logic/tests/transformation/fixtures/mc-simulation/transformations/merge.cwl`	Adds the merge workflow fixture.
`diracx-logic/tests/transformation/fixtures/mc-simulation/transformations/gauss.cwl`	Adds the simulation workflow fixture.
`diracx-logic/tests/transformation/fixtures/mc-simulation/transformations/digi-reco.cwl`	Adds nested digitisation and reconstruction steps.
`diracx-logic/tests/transformation/fixtures/mc-simulation/transformations/boole-moore.cwl`	Wraps the reconstruction subworkflow.
`diracx-logic/tests/transformation/fixtures/mc-simulation/tools/lb-run-app.cwl`	Adds the shared application tool fixture.
`diracx-logic/tests/transformation/fixtures/mc-simulation/mc-simulation.cwl`	Adds the complete simulation workgraph fixture.
`diracx-logic/src/diracx/logic/transformation/validation.py`	Validates packed processes with cwltool.
`diracx-logic/src/diracx/logic/transformation/processes.py`	Implements storage, retrieval, and operator replacement.
`diracx-logic/src/diracx/logic/transformation/__init__.py`	Introduces the transformation logic package.
`diracx-logic/pyproject.toml`	Adds the cwltool dependency.
`diracx-db/tests/transformation/test_processes.py`	Tests idempotent insertion and process retrieval.
`diracx-db/src/diracx/db/sql/transformation/db.py`	Adds process persistence, locking, repointing, and logging.
`diracx-core/tests/test_cwl_policy.py`	Tests requirement restrictions and size limits.
`diracx-core/tests/test_cwl_loader.py`	Tests file resolution and loader limits.
`diracx-core/tests/test_cwl_canonical.py`	Tests identities, normalization, and packing integrity.
`diracx-core/src/diracx/core/cwl/policy.py`	Defines supported requirements and process size limits.
`diracx-core/src/diracx/core/cwl/loader.py`	Loads YAML and resolves local CWL references.
`diracx-core/src/diracx/core/cwl/canonical.py`	Implements canonical identities and graph packing.
`diracx-core/src/diracx/core/cwl/__init__.py`	Exposes shared CWL utilities.
`diracx-core/pyproject.toml`	Adds YAML 1.2 parsing support.
`benchmarks/process_store/README.md`	Documents benchmark usage and results.
`benchmarks/process_store/corpus.py`	Summarises submission corpora and storage sizes.
`benchmarks/process_store/bench.py`	Compares flat and tree storage performance.

_{Review effort: Balanced}

The identity rules, the loader step, the policy and the replacement check changed in the fixes for the review of DIRACGrid#1097. Refs DIRACGrid#1080

cwltool validates submitted CWL processes in the services. ruamel.yaml reads CWL as YAML 1.2, as runners do; it is only needed by the CLI, which resolves documents to JSON before submitting them, so YAML never reaches a service. Refs DIRACGrid#1080

A process is identified by the SHA-256 of a canonical form defined over the parsed JSON document, not over any CWL library's output, so that it is deterministic, stable across library upgrades and reproducible by any tool (approach B in DIRACGrid#1080). Ids, doc and label where they are metadata, $namespaces and dirac: hints are dropped; prefixes are expanded to full URIs; lists become maps; fully scoped identifiers are shortened against their scope; type shorthands are expanded and named types written by short name. Each child run body is replaced by its own identity, so a workflow's identity covers its children's (a Merkle tree). pack and unpack convert to and from a self-contained $graph whose components are named by their identity. Documents must be resolved before canonicalisation: $import, $include and any other schema-salad directive is refused, since the identity would not cover what it names and a runner would fetch it. check_policy refuses requirements outside an allow-list every backend can honour (dirac: requirements included), abstract Operations, which no runner can execute, and processes larger than 256 kB. Refs DIRACGrid#1080

CWL is written in YAML, but services only ever receive JSON: the CLI resolves a document here and submits the result. load_document reads YAML 1.2, as runners do, so a value such as yes stays the string a user tested, and inlines every step's run path, $import, $include and #fragment into a $graph. It refuses reference cycles, deep nesting, oversized files and anything but local files, and keeps references to stored processes (proc:sha256-<hex>). The MCSimulation workgraph of DIRACGrid#1080 is a fixture; resolving the same files from another directory gives the same identities, and an edit to the shared tool changes every transformation using it. Refs DIRACGrid#1080

…onDB insert_processes stores documents under their content hash with INSERT IGNORE, so concurrent submissions of a shared process are safe, and get_processes fetches them, on the DX-ADR-004 Processes table unchanged. MySQL's JSON type re-sorts keys and reformats numbers, so a stored row is not the bytes that were hashed: what must hold is that the value read back re-hashes to its identity. The transformation tests run on SQLite and on every MySQL or MariaDB server given by DIRACX_TEST_MYSQL_URL or DIRACX_TEST_MARIADB_URL. Refs DIRACGrid#1080

store_process canonicalises a submitted process, checks the policy, and returns at once if it is already stored. Otherwise it inlines any stored process it references, packs the tree into one self-contained $graph, validates that with cwltool and stores it: one Processes row per transformation, keyed by the Merkle root, so a parcel's payload is a single read (load_process) and exactly what a stock runner executes. Storing flat rather than one row per node was measured on a year of LHCb submissions (see the benchmark) and the identity is the same either way. cwltool is confined: it reads nothing but the packed document, so a reference to another file or URL is refused rather than fetched on the server, and its JavaScript checks, which would start node or pull a node container, are off. The MCSimulation fixture is the JSON the CLI resolves the DIRACGrid#1080 workgraph to, as the submission endpoint will receive it. Refs DIRACGrid#1080

For a hotfix, such as a new software version, an operator can make a paused transformation run another process. End users cannot: it needs the ProductionManagement or Operator property, and no submission API offers it. A stored process never changes, since its key is its hash; what changes is which process the transformation points at. It refuses unless the transformation is Paused (its row is locked for the change), a reason is given, the replacement passes the same policy and cwltool validation as any submission, and it keeps the interface of the process it replaces (input and output names and types, named types compared by definition), since the feeder, the packer, the declared outputs and downstream edges depend on it. In the caller's transaction it then repoints the transformation and its Unassigned parcels, and records ProcessReplaced in TransformationLog. From Reserved on, a parcel keeps the process it was created with, which is why ComputeParcels.ProcessID must stay a column of every compute parcel: once a transformation's process can change, a parcel's own copy is the only record of what it ran. DX-ADR-004 freezes a transformation's ProcessID once it has parcels, and DX-ADR-005 cancels and resubmits a wrong configuration. Both still hold for owners; this is the operator's exception. Refs DIRACGrid#1080

Unit tests run on SQLite only, so the JSON round trip and the row locking a process replacement relies on would never be checked in CI. A job now starts MySQL 8.4 and MariaDB 11.4 as service containers, pinned by digest, and runs the diracx-db transformation tests against both. Refs DIRACGrid#1080

How a process's identity is computed, how a submission becomes a stored row (loading in the CLI, canonicalisation, policy, validation, storage), what a parcel reads back, the operator-only replacement, and what is not covered yet. Refs DIRACGrid#1080

Ingests a corpus of real submissions into two schemas, flat (one row per transformation, as stored) and tree (one row per node with ProcessReferences edges, as DX-ADR-004 first described), applying the same acceptance rules to both, then measures what a dispatcher pays to materialise a payload and the size on disk. On a year of LHCb submissions (84,406 workgraphs, 182,071 transformations) on MySQL 8.4, the tree saves nothing compressed and costs more rows and slower lookups. Not run by CI. Refs DIRACGrid#1080

Grid sites and compute backends typically provide Apptainer, and workflows such as CTAO's need a container, so DockerRequirement joins the allow-list. An image must be one to pull (dockerPull or dockerImageId): dockerFile, dockerLoad and dockerImport are refused, in hints too, because they make the worker node build an image, which Apptainer cannot do there, or fetch one from a URL at run time. A dockerPull tag is accepted as written; pinning it to a digest stays an open question. Refs DIRACGrid#1080

chrisburr · 2026-10-23T12:57:55Z

      - name: Upload coverage report
        uses: codecov/codecov-action@303a32d7a59b442fa8d48b6a1cc6825c09c847a5 # v7.1.1

+  pytest-sql-servers:


I think this is a perfectly reasonable idea but nothing to do with the issue at hand

chrisburr · 2026-10-23T12:58:26Z

@@ -0,0 +1,253 @@
+"""Benchmark flat vs tree storage of CWL processes against a real database.


This doesn't belong in the repo

chrisburr · 2026-10-23T13:06:20Z

I'm going to close this as it needs to be broken down into smaller parts and a decent chunk of this is out of scope for the GitHub issue.

ryuwd force-pushed the feat/cwl-process-store branch from aa52368 to c232a46 Compare October 19, 2026 14:43

mmascher requested a balanced review from Copilot October 22, 2026 09:07

Copilot started reviewing on behalf of mmascher October 22, 2026 09:07

View session

Copilot AI reviewed Oct 22, 2026

View reviewed changes

ryuwd force-pushed the feat/cwl-process-store branch from dc780a6 to f55fbff Compare October 22, 2026 14:55

aldbr linked an issue Oct 22, 2026 that may be closed by this pull request

[TransformationSystem] CWL Processes ingestion #1080

Open

ryuwd added 9 commits October 22, 2026 17:39

docs: explain the CWL process store

db9c011

How a process's identity is computed, how a submission becomes a stored row (loading in the CLI, canonicalisation, policy, validation, storage), what a parcel reads back, the operator-only replacement, and what is not covered yet. Refs DIRACGrid#1080

ryuwd force-pushed the feat/cwl-process-store branch from 1eec3a6 to 6cf3936 Compare October 22, 2026 15:42

ryuwd marked this pull request as ready for review October 22, 2026 15:47

chrisburr marked this pull request as draft October 23, 2026 12:54

chrisburr reviewed Oct 23, 2026

View reviewed changes

chrisburr closed this Oct 23, 2026

Navigation Menu

Repository navigation

feat: ingest and store CWL processes, flat per transformation - #1097

feat: ingest and store CWL processes, flat per transformation#1097
ryuwd wants to merge 10 commits into
DIRACGrid:mainfrom
ryuwd:feat/cwl-process-store

ryuwd commented Oct 19, 2026 •

edited

Loading

Uh oh!

read-the-docs-community Bot commented Oct 19, 2026 •

edited

Loading

Uh oh!

Copilot AI left a comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

chrisburr Oct 23, 2026

Uh oh!

chrisburr Oct 23, 2026

Uh oh!

chrisburr commented Oct 23, 2026

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

		@@ -0,0 +1,253 @@
		"""Benchmark flat vs tree storage of CWL processes against a real database.

Conversation

ryuwd commented Oct 19, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

How to review

What this adds

Design decision: flat per transformation

Operator-only: replacing a transformation's process

Deferred

Notes for reviewers

Fixes from the Copilot review

Since the review

Testing

Uh oh!

read-the-docs-community Bot commented Oct 19, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Documentation build overview

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

🟡 Changes recommended

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

chrisburr Oct 23, 2026

Choose a reason for hiding this comment

Uh oh!

chrisburr Oct 23, 2026

Choose a reason for hiding this comment

Uh oh!

chrisburr commented Oct 23, 2026

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

ryuwd commented Oct 19, 2026 •

edited

Loading

read-the-docs-community Bot commented Oct 19, 2026 •

edited

Loading