refactor[next]!: make Backend abstract with a CompiledBackend kind and an internal Toolchain by egparedes · Pull Request #2741 · GridTools/gt4py

egparedes · 2026-07-30T10:49:26Z

Stack: #2741 → #2956 → #2742 → #2743.

Reworked after review. The first version renamed the user-facing Backend to Toolchain. Users think of "backend" as the whole execution choice, frontend included, and a complete rename would have had to reach every backend=. This version keeps Backend as the public concept and moves the split one level down: what users pick is a backend (compiler + allocator), and the frontend, which is the same for every backend, is no longer part of it. ADR 0029 records the design.

What changes

Backend is an abstract base with name and allocator; instantiating it directly raises TypeError. gtx.typing.Backend, backend=, with_backend and Program.backend keep their names.
CompiledBackend(Backend) adds compiler (formerly executor) and loader (formerly the overridable load_artifact()). Embedded execution is still backend=None here; EmbeddedBackend objects come in feat[next]!: represent embedded execution by EmbeddedBackend objects #2956.
The frontend is global. Every backend already used the default Transforms, now DEFAULT_FRONTEND; the one exception, roundtrip.gtir, built an identical copy and now uses it too (a test pins that both lower identically). An internal, unexported Toolchain composes the frontend with a CompiledBackend where compilation happens: in the compiled-programs pool, FieldOperatorFromFoast and the iterator runtime.
One dispatch helper in decorator.py for programs, field operators and scan operators: CompiledBackend compiles, None runs embedded, other Backends raise TypeError. Non-backends are rejected when the program or operator is created.
No overridable compile. The opaque "customized compile" path of the process-pool runner (ADR 0024) and CompilationTask.no_offload_reason are removed; test doubles wrap compiler instead. An unpicklable compiler still compiles in the calling thread, with a warning.
Backend.name is a documented label for metrics and logs: expected to be unique, not enforced, never used as identity.
DaCe: the DaCeBackend subclass is replaced by a DaCeLoader (backend.loader.external_workspace). __sdfg__ recognizes DaCe backends by type (is_dace_backend), not by name.
Fix: __sdfg__ no longer mutates the cached frontend result. It overwrote the cached lowering in place, so a later compilation of the same program on another backend failed. The bug exists on main for the backends sharing the default frontend; this PR would have extended it to roundtrip.gtir. Found in an adversarial review; covered by a regression test.
Builders and config follow the backend vocabulary: make_gtfn_backend, make_dace_backend, CompiledBackendConfig. The flat-keyword make_dace_backend(gpu=...), deprecated in the last release, is removed and its name reused; old-style calls fail with a TypeError explaining the new signature.
ADR 0029 (renamed "Backend Structure and Pipeline Simplification") presents the final design first and the transition after it. Notes are added to ADRs 0011, 0017, 0024, 0027 and 0028.

Also fixed: the "no recompilation" guards in test_compiled_program.py swapped program.backend after the pool existed, so they guarded nothing. They now replace the pool's toolchain, and a test proves the guard fires.

Breaking (no aliases)

`main` (v1.2.3)	After this PR
`Backend(name, executor, allocator, transforms)`	`CompiledBackend(name=, allocator=, compiler=, loader=)`; `Backend` is abstract
`backend.executor` / `backend.transforms`	`backend.compiler` / removed (global frontend)
overriding `load_artifact` / `compile`	`dataclasses.replace(backend, loader=... / compiler=...)`
`DaCeBackend`, `backend.external_workspace`	`DaCeLoader`, `backend.loader.external_workspace`
`make_gtfn_toolchain`, `make_dace_toolchain`, `ToolchainConfig`	`make_gtfn_backend`, `make_dace_backend`, `CompiledBackendConfig`
`make_dace_backend(gpu=..., auto_optimize=..., use_metrics=...)`	`make_dace_backend(DaCeConfig(gpu=..., auto_optimize=...), translation=functools.partial(make_dace_translator, use_metrics=...))`
`ConcreteArtifact` / `.data`, `CompilableProgramDef`	`ProgramWithArgs` / `.definition`, `CompilableProgram`
`backend.DEFAULT_TRANSFORMS`	`backend.DEFAULT_FRONTEND`
duck-typed backend objects	rejected when the program is created
`compile_variant_hook(..., backend: Backend)`	`backend: CompiledBackend`

Code that only passes backend instances around (gtx.gtfn_cpu, run_dace_cpu, ...) is unaffected. icon4py needs two changes in model_backends.py (the builder calls).

Cache keys

The ProgramWithArgs rename rotates the persistent translation-cache keys (gtfn, dace): a cold rebuild, never a stale hit. The backend restructuring rotates nothing: Program.backend is excluded from fingerprints.

Merge order

#2953 overlaps this PR in 7 files and reshapes CompilationTask; we should agree which lands first.

Testing

Full suites run in CI. The rework passed the internal and dace CPU nox sessions, pre-commit -a, typing exports and examples locally; later commits were checked with targeted tests.

…loading Adopt the vocabulary ADR 0017 already defines: the root object that carries a program from DSL source to a runnable callable is the *toolchain*, not the *backend* (which is one of its two halves). Renamed: - `next.backend.Backend` -> `next.backend.Toolchain` - `Toolchain.transforms` -> `Toolchain.frontend` - `Toolchain.executor` -> `Toolchain.backend` - `otf.workflow.ConcreteArtifact` -> `otf.workflow.ProgramWithArgs` - `ProgramWithArgs.data` -> `ProgramWithArgs.definition` - `otf.stages.CompilableProgramDef` -> `otf.stages.CompilableProgram` - `gtx.typing.Backend` -> `gtx.typing.Toolchain` - `ffront.foast_to_past.ItirShim.definition` -> `ItirShim.operator_def`, freeing the name for the renamed `ProgramWithArgs` field it now wraps Beyond the renames, this restructures how artifacts are loaded. `Backend.load_artifact()` was an overridable method, so a toolchain needing backend-specific runtime data in the loaded program subclassed the root object (ADR 0027's dace external-workspace path did exactly that). It becomes `Toolchain.loading`, a plain step field defaulting to `artifacts.load_artifact`, so customization is composition-time like the other three fields. The `DaCeBackend` subclass is therefore deleted and `run_dace_*` are plain `Toolchain` instances carrying a `DaCeLoadingStep`; the public path to the workspace moves from `backend.external_workspace` to `backend.loading.external_workspace`. This also keeps subclassing off the load path, which matters because per ADR 0024 a toolchain that customizes `compile` is opaque to the process-pool runner. These are hard renames: no deprecation aliases are kept, so downstream has to move in this release rather than drift on a compatibility layer. An alias would have been partial anyway -- constructing with the old keyword names cannot be aliased on a dataclass without hand-writing `__init__` -- and a half-working compatibility layer is harder to act on than one loud break. Also resolves the `TODO(tehrengruber)` in `backend.py` asking for exactly this rename, and records the decision in a new ADR 0029, which partially supersedes ADR 0011 and refines ADR 0017. Programs compile and run exactly as before. Fingerprints embed qualified class and field names, so the `ProgramWithArgs` rename rotates the two persistent translation-cache keys (gtfn, dace); this is the second such rotation in the stack, after the module move that created `otf.workflow.ConcreteArtifact`. Each is a cold rebuild rather than a stale hit because the key itself changes -- entries written under the old name are simply never looked up. On top of that, `BUILD_CACHE_VERSION_ID` (the gt4py version) already rotates these keys at every release, so the cost is nothing extra for anyone crossing a release boundary. The two rotations collapse into one if the stack lands in a single release.

havogt

I stopped here, because if we want to do this change, then many renames are actually missing. I think we should discuss if we really want to rename the user facing "Backend" to "Toolchain". It's a quite an established concept and from the user perspective both the frontend transformation and the backend transforms are actually the backend.

havogt · 2026-10-27T15:00:45Z

@@ -129,11 +129,11 @@ def compile(self, program, compile_time_args):
            return self._wrapped.compile(program, compile_time_args=compile_time_args)

    backend = _WrapperBackend(


shouldn't backend be renamed to toolchain here?

havogt · 2026-10-27T15:01:16Z



-class NoBackend(next_backend.Backend):
+class NoBackend(next_backend.Toolchain):


Suggested change

class NoBackend(next_backend.Toolchain):

class NoToolchain(next_backend.Toolchain):

havogt · 2026-10-27T15:01:32Z

        cls,
        mesh_descriptor: MeshDescriptor,
-        backend: Optional[next_backend.Backend] = None,
+        backend: Optional[next_backend.Toolchain] = None,


Suggested change

backend: Optional[next_backend.Toolchain] = None,

toolchain: Optional[next_backend.Toolchain] = None,

etc

havogt · 2026-10-28T10:30:46Z

+    backend: workflow.Workflow[stages.CompilableProgram, artifacts.CompilationArtifact]
    allocator: next_allocators.FieldBufferAllocatorProtocol[core_defs.DeviceTypeT]
-    transforms: workflow.Workflow[stages.ConcreteProgramDef, stages.CompilableProgramDef]
+    frontend: workflow.Workflow[stages.ConcreteProgramDef, stages.CompilableProgram]


We agree, what we call "Backend" should:

not contain frontend transformations

should contain the allocator

… an internal Toolchain Rework the root object after review: the user-facing object keeps the name `Backend` (`gtx.typing.Backend` again), but it no longer carries the frontend transforms. - `backend.Backend` is an abstract base with `name` and `allocator` (instantiating it raises `TypeError`). - `backend.CompiledBackend(Backend)` adds `compiler` (formerly `Toolchain.backend` / `Backend.executor`) and `loader` (formerly `Toolchain.loading`), and provides `__gt_allocator__`. - `backend.Toolchain` becomes internal: the global frontend (`DEFAULT_TRANSFORMS`) composed with a `CompiledBackend`. The compiled-programs pool holds one, `FieldOperatorFromFoast` and the iterator runtime build one for their one-shot compiles. - Programs and operators dispatch on the backend kind in one helper; objects that are not backends are rejected when they are created. - The frontend is global, so `Program`/`FieldOperator` always lower with `DEFAULT_TRANSFORMS`; `roundtrip.gtir` drops its own, identical `Transforms`. - `compile` is no longer a backend method, so the opaque "customized compile" path of `make_compilation_task` is deleted. Tests: the compilation guards in `test_compiled_program.py` swapped `program.backend` after the pool was created and therefore guarded nothing; they now replace the pool's toolchain (`_forbid_compilation`), with a test proving the guard raises. Test doubles wrap `compiler` instead of overriding `compile`.

… frontend roundtrip.gtir now uses the global default frontend; its former private Transforms used the same factories with the same caching flags and only owned separate in-memory caches. The test compares both lowerings of a program and of a field operator.

Its only producer was the opaque 'customized compile' path of make_compilation_task, deleted with the backend restructuring. Tasks still fall back to the calling thread when their executor cannot be pickled; a new test covers that for a backend with a closure compiler.

…y DaCe backends by type - DaCeLoadingStep -> DaCeLoader (the external workspace stays reachable as backend.loader.external_workspace); both are exported from runners.dace. - New is_dace_backend(backend): a CompiledBackend whose compiler is the standard compile pipeline with a DaCeTranslator translation step (looking through CachedStep). Program.__sdfg__ uses it instead of checking that the backend name contains 'dace', so renamed DaCe backends convert too; the orchestration and vertical-advection tests use it as well.

The builders return a `CompiledBackend` and `Toolchain` is now internal, so the public names follow the backend vocabulary: - `gtfn.make_gtfn_toolchain` -> `gtfn.make_gtfn_backend` - `dace.make_dace_toolchain` -> `dace.make_dace_backend` - `backend.ToolchainConfig` -> `backend.CompiledBackendConfig` The flat-keyword `make_dace_backend(gpu, auto_optimize=..., ...)`, deprecated in the previous release, is removed; its name now belongs to the config-based builder. Calls in the removed shape (`make_dace_backend(gpu=...)` or `make_dace_backend(True)`) fail at the call with a `TypeError` that explains the new signature and where each old argument goes. The `name_postfix` docstrings now say what the generated name encodes instead of claiming the name "must stay unique".

…fication ADR 0029 (introduced by this PR, not yet on main) is rewritten for the reworked design: the user-facing Backend stays, as an abstract base with a CompiledBackend kind; the frontend is global and composed with a compiled backend by an internal Toolchain; dispatch on the backend kind; no overridable compile; Backend.name is a label; builder/config renames. The 'Pipeline, not combinators', 'Stage observability' and 'Phasing' sections of the later stack PRs are kept in the new vocabulary. The file is renamed accordingly. Also: notes in ADRs 0011, 0017, 0024, 0027 and 0028; HackTheToolchain.md says the frontend is not a user customization point; WorkflowPatterns.md and the next AGENTS.md module map use the new names.

egparedes · 2026-10-29T06:44:34Z

Thanks for stopping here; you were right on both counts. A complete rename would have had to reach every backend=, with_backend and Program.backend, and the half-done version produced code like self.backend.backend. Users also think of "backend" as the whole thing, frontend included. So I kept Backend as the public concept and moved the split one level down instead: what users pick is the backend (compiler + allocator), and the frontend, which is the same for every backend, is no longer part of it.

The reworked PR (same branch, new commits; the description has the details):

Backend stays the public name and becomes an abstract base with only name and allocator (gtx.typing.Backend is unchanged).
The concrete kind is CompiledBackend(Backend), which adds compiler (formerly executor) and loader (formerly the overridable load_artifact). The DaCeBackend subclass is gone; DaCe passes its workspace through a DaCeLoader.
The frontend is no longer on any backend. It is the global DEFAULT_TRANSFORMS. Every backend already used it; the one exception, roundtrip.gtir, built an identical copy. Toolchain survives only as an internal class (not exported) that composes the frontend with a CompiledBackend where a compilation happens.
Programs and operators dispatch on the backend kind in one helper. A CompiledBackend is compiled, None still means embedded, and anything else raises. compile is no longer a backend method, so the opaque process-pool fallback for customized compile is gone; test doubles wrap compiler instead.
Embedded backends as real objects come in a follow-up PR (ADR 0030). It adds EmbeddedBackend(Backend) instances for NumPy/CuPy/JAX, makes the NumPy one the default, turns backend=None into a loud error, and makes allocator=program work for embedded programs.
Builders and config follow the backend vocabulary:
- make_gtfn_toolchain → make_gtfn_backend;
- make_dace_toolchain → make_dace_backend;
- ToolchainConfig → CompiledBackendConfig.
The flat-keyword make_dace_backend(gpu=...), deprecated since the last release, is removed and its name reused. A call in the old shape fails with a TypeError that spells out the new call and where each old argument goes. This means a two-line change in icon4py's model_backends.py.

Your inline comments:

tests/next_tests/unit_tests/otf_tests/test_runners.py:131: these tests now build an internal Toolchain(CompiledBackend(...), frontend=...), named toolchain. The test of the "customized compile" path is deleted along with that path, and a test for a backend whose compiler cannot be pickled replaces it.
tests/next_tests/integration_tests/cases_utils.py:62: I kept the name NoBackend, since it is a Backend again. It is now a direct Backend subclass with just a name and an allocator. Programs can be created with it, but calling them raises TypeError. The TODO about needing the transforms for linting is gone, because the frontend no longer comes from the backend.
tests/next_tests/integration_tests/cases.py:751: kept as backend: Optional[next_backend.Backend]. It holds a Backend again, so the name fits.

One coordination point: your #2953 overlaps this PR in 7 files (compilation_tasks.py, runners.py, dace/program.py, the unit test_compiled_program.py, test_runners.py, ADRs 0019 and 0024), and it reshapes CompilationTask. Shall we agree on which lands first? I'm happy to merge yours into this branch and adapt if it is ready earlier. If this one goes first, your test backends would move from Backend(executor=..., transforms=...) to CompiledBackend(compiler=...).

Bring the reworked stack (#2741 backend structure, #2956 embedded backends, #2742 observability on the new design) under the pipeline PR and adapt it: - `CompiledBackend.compiler` / `.loader` and `Toolchain.frontend` are typed as `workflow.Step`; `Toolchain.translate` and `is_dace_backend` narrow to `CompilePipeline`; the dace SDFG conversion replaces the translation step inside `CompiledBackend.compiler`. - The allocator-vs-compiler device-agreement check moves from `Toolchain.__post_init__` to `CompiledBackend.__post_init__` (after the abstract-base check); its tests and the pre-built-backend checks use `backend.compiler`. - `roundtrip.gtir` keeps the default frontend (no private `Transforms`); the `Transforms.foast_to_itir` comment no longer names `roundtrip`. - ADR 0029 edits of this PR (CachedStep without `.replace`/`.chain`, static-checking scope, "Cross-step agreement is checked on construction", phasing) are carried over into the rewritten, renamed ADR; the old file stays deleted. HackTheToolchain.md / WorkflowPatterns.md / AGENTS.md use the backend vocabulary.

…_sdfg__ `__sdfg__` lowers through the shared frontend and then overwrote the cached `past_to_itir` result in place (transformed definition, offset provider types instead of tables). Every later compilation of the same program reused the corrupted stage, so running it on another backend failed (an assertion in domain inference, or `'NeighborConnectivityType' object has no attribute 'asnumpy'` when offloading the compilation). Build a new stage instead. The bug exists on main for the backends sharing the default frontend; the previous commits extended it to `roundtrip.gtir`, which now shares it too. Found in an adversarial review of #2741.

Present the backend structure first on its own terms (concepts, the route from definition to callable, customization, pipelines, observability) and move the transition from the old structure (name mapping, removed mechanisms, no-alias policy, cache-key rotation, phasing, earlier ADRs) into a later section. No decision changes. Also corrects claims that did not match the code: compiling in the calling thread is not only the unpicklable-compiler fallback (serial mode), the cache version salt does not change on every source commit, `__sdfg__` no longer mutates the cached stage, and DaCe names encode whether auto-optimization is on.

egparedes

I have several questions and comments

egparedes · 2026-10-29T09:39:32Z

+def load_artifact(artifact: CompilationArtifact) -> ExecutableProgram:
+    """
+    Load a compilation artifact into a directly-callable program.
+
+    This is the default `CompiledBackend.loader`. Backends that need to inject
+    backend-specific runtime data into the loaded program supply their own
+    loader instead of overriding a method.
+
+    Args:
+        artifact: The artifact produced by the backend's compiler.
+
+    Returns:
+        The loaded, directly-callable program.
+    """
+    return artifact.load()
+


Why is this needed?

It's the default value of CompiledBackend.loader: a dataclass default has to be a callable, and a named module-level function stays picklable, readable in reprs and documented. DaCeLoader also calls it after injecting the workspace. The alternative, an optional loader with a fallback to artifact.load(), would put a None check in every caller.

egparedes · 2026-10-29T09:42:57Z


 CompiledProgramsKey: TypeAlias = Annotated[compiled_program.CompiledProgramsKey, _ONLY_FOR_TYPING]

 Backend: TypeAlias = Annotated[backend.Backend, _ONLY_FOR_TYPING]


Is this alias still needed or can we just use the actual Backend, since it's an abstract class?

I'd keep it: every name in gtx.typing is an Annotated[..., "only for typing"] alias, including concrete classes like Program and FieldOperator. The alias marks the name as meant for annotations, which doesn't change with Backend being abstract.

egparedes · 2026-10-29T09:44:54Z

-        """
-        return artifact.load()
+@dataclasses.dataclass(frozen=True)
+class CompiledBackend(Backend):


Shouldn't it be Generic on the Device Type?

Yes, done in e9f4113: CompiledBackend and Toolchain are Generic[DeviceTypeT] again, as Backend was on main, so the allocator keeps its device type. The abstract Backend stays non-generic because an embedded backend's allocator (#2956) is an array namespace, which has no device type parameter.

egparedes · 2026-10-29T09:45:41Z


 @dataclasses.dataclass(frozen=True)
-class ToolchainConfig:
+class Toolchain:


Shouldn't it be Generic on the Device Type?

Yes, done in e9f4113 (see the comment on CompiledBackend).

egparedes · 2026-10-29T09:46:07Z

+
+    backend: CompiledBackend
+    frontend: workflow.Workflow[stages.ConcreteProgramDef, stages.CompilableProgram] = (
+        DEFAULT_TRANSFORMS


Maybe we should rename DEFAULT_TRANSFORMS to something more meaningful

Done in e9f4113: it's now DEFAULT_FRONTEND, since it is the single global frontend, not a backend's default transforms. ADR 0029's name table lists the rename.

egparedes · 2026-10-29T09:48:52Z

+def _compiled_backend_or_none(
+    backend: next_backend.Backend | None,
+) -> next_backend.CompiledBackend | None:
+    """
+    Dispatch on the kind of `backend`.
+
+    Returns:
+        The compiled backend to run on, or `None` for embedded execution.
+
+    Raises:
+        TypeError: If `backend` is a `Backend` of a kind that cannot run programs.
+    """
+    match backend:
+        case next_backend.CompiledBackend():
+            return backend
+        case None:
+            return None
+        case _:
+            raise TypeError(
+                f"Backend '{backend.name}' of type '{type(backend).__name__}' is not supported:"
+                " expected a 'CompiledBackend', or None for embedded execution."
+            )


Wouldn't it be simpler to just call isinstance(... , CompiledBackend)?

The helper also rejects unsupported kinds. With a bare isinstance(..., CompiledBackend), everything else would silently run embedded, including a Backend subclass that is neither kind (e.g. the NoBackend test double, which must fail loudly). #2956 extends it to three outcomes (compiled, embedded, error), and keeping that in one place avoids repeating it in Program, FieldOperator and the scan operator.

egparedes · 2026-10-29T09:49:20Z

@@ -263,8 +297,9 @@ def __gt_type__(self) -> ts_ffront.ProgramType:

    # TODO(ricoh): linting should become optional, up to the backend.


Is this comment still accurate and relevant?

Not anymore: "up to the backend" no longer fits, since backends don't own the frontend. Reworded to "linting should become optional" in e9f4113.

egparedes · 2026-10-29T09:50:26Z

    @property
    def _frontend_transforms(self) -> next_backend.Transforms:
-        if self.backend is None:
-            return next_backend.DEFAULT_TRANSFORMS
-        # TODO(tehrengruber): This class relies heavily on `self.backend.transforms` being
-        #  a `next_backend.Transforms`, but the backend type annotation does not reflect that.
-        assert isinstance(self.backend.transforms, next_backend.Transforms)
-        return self.backend.transforms
+        # The frontend is global: it is not part of any backend (ADR 0029).
+        return next_backend.DEFAULT_TRANSFORMS


Is this property still really needed or useful? Who's using it?

No, it only returned the global frontend. Removed in e9f4113; Program, FieldOperator and the DaCe __sdfg__ use DEFAULT_FRONTEND directly.

egparedes · 2026-10-29T09:51:12Z

    @property
    def _frontend_transforms(self) -> next_backend.Transforms:
-        if self.backend is None:
-            return next_backend.DEFAULT_TRANSFORMS
-        # TODO(tehrengruber): This class relies heavily on `self.backend.transforms` being
-        #  a `next_backend.Transforms`, but the backend type annotation does not reflect that.
-        assert isinstance(self.backend.transforms, next_backend.Transforms)
-        return self.backend.transforms
+        # The frontend is global: it is not part of any backend (ADR 0029).
+        return next_backend.DEFAULT_TRANSFORMS


Same question as in Program: is this still needed? why?

Same as in Program: removed in e9f4113.

- `CompiledBackend` and `Toolchain` are generic on the device type again, as `Backend` was on main, so the allocator keeps its device type. - Rename `DEFAULT_TRANSFORMS` to `DEFAULT_FRONTEND`: it is the global frontend, not a backend's default transforms. - Drop the `_frontend_transforms` properties of `Program` and `FieldOperator`; they only returned the global frontend. - Reword the linting TODO: backends no longer own the frontend.

havogt · 2026-10-29T08:47:42Z

-    executor: workflow.Workflow[stages.CompilableProgramDef, artifacts.CompilationArtifact]
-    allocator: next_allocators.FieldBufferAllocatorProtocol[core_defs.DeviceTypeT]
-    transforms: workflow.Workflow[stages.ConcreteProgramDef, stages.CompilableProgramDef]
+    allocator: constructors.Allocator


Why not __gt_allocator__ in the abstract interface?

havogt · 2026-10-29T09:00:00Z

+> [!NOTE]
+> The frontend (`gtx.backend.Transforms`, the definition transformations shown in the next two
+> sections) is global and not a user customization point: a backend does not carry a frontend
+> (ADR 0029). A modified `Transforms` can only be used through the internal
+> `gtx.backend.Toolchain`, e.g. in gt4py's own tests. What a backend does is customized through
+> its `compiler` and `loader`, see "Alternative Workflow" below.
+


Suggested change

> [!NOTE]

> The frontend (`gtx.backend.Transforms`, the definition transformations shown in the next two

> sections) is global and not a user customization point: a backend does not carry a frontend

> (ADR 0029). A modified `Transforms` can only be used through the internal

> `gtx.backend.Toolchain`, e.g. in gt4py's own tests. What a backend does is customized through

> its `compiler` and `loader`, see "Alternative Workflow" below.

havogt · 2026-10-29T09:02:49Z

    @property
    def _frontend_transforms(self) -> next_backend.Transforms:
-        if self.backend is None:
-            return next_backend.DEFAULT_TRANSFORMS
-        # TODO(tehrengruber): This class relies heavily on `self.backend.transforms` being
-        #  a `next_backend.Transforms`, but the backend type annotation does not reflect that.
-        assert isinstance(self.backend.transforms, next_backend.Transforms)
-        return self.backend.transforms
+        # The frontend is global: it is not part of any backend (ADR 0029).
+        return next_backend.DEFAULT_TRANSFORMS


I'd make it a private field of the dataclass which is default initialized to DEFAULT_TRANSFORMS.

havogt · 2026-10-29T09:03:57Z

-    # TODO(tehrengruber): We can not use transforms from `self.backend` since this can be
-    #  a different backend than the one of the program that calls this field operator. Just use
-    #  the hard-coded lowering until this is cleaned up.
+    # The lowering does not depend on the backend: the frontend is global (ADR 0029).


Suggested change

# The lowering does not depend on the backend: the frontend is global (ADR 0029).

havogt · 2026-10-29T09:06:11Z

-    def load_artifact(self, artifact: artifacts.CompilationArtifact) -> artifacts.ExecutableProgram:
-        program = super().load_artifact(artifact)
+    def __call__(self, artifact: artifacts.CompilationArtifact) -> artifacts.ExecutableProgram:
+        program = artifacts.load_artifact(artifact)


Suggested change

program = artifacts.load_artifact(artifact)

program = artifact.load()

havogt · 2026-10-29T10:09:51Z

+    compiler: workflow.Workflow[stages.CompilableProgram, artifacts.CompilationArtifact]
+    # A plain `Callable`, not `workflow.Workflow`: the protocol names its
+    # parameter `inp`, which would force every loader to use that name.
+    loader: Callable[[artifacts.CompilationArtifact], artifacts.ExecutableProgram] = (


The custom loader was introduced with the dace external workspace in #2719. It's to my knowledge the only stateful aspect of any backend, wondering if we should provide the workspace at execution time instead of binding it to the program via backend. That approach would also have downsides, because the workspace is clearly coupled to a dace backend (unless we generalize it).

egparedes mentioned this pull request Jul 30, 2026

refactor[next]: add stage observability and Toolchain.translate, fix dace __sdfg__ reach-in #2742

Draft

egparedes marked this pull request as ready for review July 30, 2026 17:58

egparedes force-pushed the otf-split-2-toolchain-naming branch from 40c134b to 63713c3 Compare July 30, 2026 17:58

egparedes force-pushed the otf-split-2-toolchain-naming branch from 63713c3 to 5dfb78f Compare July 31, 2026 16:15

egparedes force-pushed the otf-split-2-toolchain-naming branch from 5dfb78f to ef9785b Compare August 17, 2026 15:07

egparedes force-pushed the otf-split-2-toolchain-naming branch from ef9785b to 70b9425 Compare August 20, 2026 16:17

egparedes mentioned this pull request Aug 20, 2026

refactor[next]: replace factory-boy factories with plain builders #2808

Merged

egparedes changed the base branch from otf-split-1-stages-artifacts to otf-split-1b-plain-builders August 20, 2026 16:21

egparedes force-pushed the otf-split-2-toolchain-naming branch from 70b9425 to ee68592 Compare August 20, 2026 16:52

egparedes force-pushed the otf-split-2-toolchain-naming branch 2 times, most recently from 49edc24 to a78e837 Compare August 26, 2026 11:05

egparedes force-pushed the otf-split-2-toolchain-naming branch from a78e837 to f743ebd Compare August 28, 2026 13:06

egparedes force-pushed the otf-split-2-toolchain-naming branch from f743ebd to 4fced1e Compare August 28, 2026 13:09

egparedes changed the title ~~refactor[next]: rename Backend to Toolchain and align field names~~ refactor[next]: rename Backend to Toolchain and restructure artifact loading Aug 28, 2026

egparedes force-pushed the otf-split-2-toolchain-naming branch from 4fced1e to 41da37c Compare September 1, 2026 18:30

egparedes force-pushed the otf-split-2-toolchain-naming branch from 41da37c to 247de77 Compare September 9, 2026 16:34

egparedes force-pushed the otf-split-2-toolchain-naming branch from 247de77 to 0e18073 Compare September 9, 2026 16:51

egparedes force-pushed the otf-split-2-toolchain-naming branch 2 times, most recently from c19045d to 7f33e8a Compare September 30, 2026 06:14

egparedes mentioned this pull request Sep 30, 2026

refactor[next]: [experiment] replace factory-boy factories with TypedDict option builders #2929

Closed

egparedes force-pushed the otf-split-2-toolchain-naming branch 2 times, most recently from aaeaf90 to 007dac0 Compare October 1, 2026 14:56

Base automatically changed from otf-split-1b-plain-builders to main October 1, 2026 15:30

egparedes force-pushed the otf-split-2-toolchain-naming branch from 007dac0 to 3cd88c1 Compare October 1, 2026 15:31

egparedes force-pushed the otf-split-2-toolchain-naming branch from 3cd88c1 to 9cf30a3 Compare October 19, 2026 15:16

havogt requested changes Oct 27, 2026

View reviewed changes

havogt reviewed Oct 28, 2026

View reviewed changes

egparedes added 6 commits October 28, 2026 18:28

egparedes changed the title ~~refactor[next]: rename Backend to Toolchain and restructure artifact loading~~ refactor[next]!: make Backend abstract with a CompiledBackend kind and an internal Toolchain Oct 29, 2026

egparedes mentioned this pull request Oct 29, 2026

feat[next]!: represent embedded execution by EmbeddedBackend objects #2956

Open

egparedes removed this pull request from stack #2850 October 29, 2026 07:48

egparedes mentioned this pull request Oct 29, 2026

refactor[next]: replace workflow combinators with explicit typed pipelines #2743

Draft

egparedes added this pull request to stack #2957 October 29, 2026 07:48

egparedes added 2 commits October 29, 2026 09:11

egparedes commented Oct 29, 2026

View reviewed changes

havogt reviewed Oct 29, 2026

View reviewed changes

		@@ -129,11 +129,11 @@ def compile(self, program, compile_time_args):
		return self._wrapped.compile(program, compile_time_args=compile_time_args)

		backend = _WrapperBackend(



		class NoBackend(next_backend.Backend):
		class NoBackend(next_backend.Toolchain):

	class NoBackend(next_backend.Toolchain):
	class NoToolchain(next_backend.Toolchain):

	backend: Optional[next_backend.Toolchain] = None,
	toolchain: Optional[next_backend.Toolchain] = None,


		CompiledProgramsKey: TypeAlias = Annotated[compiled_program.CompiledProgramsKey, _ONLY_FOR_TYPING]

		Backend: TypeAlias = Annotated[backend.Backend, _ONLY_FOR_TYPING]

		@@ -263,8 +297,9 @@ def __gt_type__(self) -> ts_ffront.ProgramType:

		# TODO(ricoh): linting should become optional, up to the backend.

	program = artifacts.load_artifact(artifact)
	program = artifact.load()

Conversation

egparedes commented Jul 30, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

What changes

Breaking (no aliases)

Cache keys

Merge order

Testing

Uh oh!

havogt left a comment

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

egparedes commented Oct 29, 2026

Uh oh!

egparedes left a comment

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

This branch has not been deployed

Reviewers

Uh oh!

Assignees

Labels

Projects

egparedes commented Jul 30, 2026 •

edited

Loading