feat(ci): add a reusable workflow that gates a kind's tree and suites by CameronBrooks11 · Pull Request #24 · kindspec/kindkit

CameronBrooks11 · 2026-10-07T21:01:49Z

What

.github/workflows/kind.yml is a reusable workflow (on: workflow_call). Its inputs, all required: fixture-root, suite, second-implementation and mutants. It runs as one job named conformance, and its four gates are steps with if: always(). Each gate runs even when an earlier gate is red. The setup steps before them are not like that: if one fails, the gates still run and fail too, for want of kindkit's tools.

step	runs	exit 0 only if
the case tree follows the convention	`tools/validate_case_tree.py <root>`	every manifest is valid; exit 2 on an empty or missing tree
the reference implementation passes every case in the tree	`tools/conform.py <root> <suite>`	the runner's report names `root` as the fixture root, it ran exactly the case directories under it, and none failed
the second implementation passes every case in the tree	`tools/conform.py --differs-from <suite> <root> <second-implementation>`	the same as above, and the command is not the suite's command with only whitespace changed
the suite must be able to fail	`tools/check_gate.py <root> <mutants>`	`kindkit.gate`'s own report exists, killed ≥1 mutant, has no SURVIVED, STALE, BOGUS or BROKEN mutant, and its baseline ran exactly the tree under that root

How a verdict is tied to the tree the workflow names:

kindkit.cli writes the absolute root it read into its report. A report whose root is not absolute is refused as malformed (round 2).
kindkit.cli takes the report path from KINDKIT_REPORT_JSON when --report-json is not given, and refuses a file already at that path.
Verdict carries the root that probe_command reads from the report.
kindkit.gate writes its verdicts, the baseline's case ids and its root to KINDKIT_GATE_REPORT when that is set. It creates the file exclusively, so a second gate in the same command raises.

conform.py and check_gate.py run the caller's command as written, with those variables set. Nothing is appended to the command, so a trailing newline (suite: |), a cd, or a trailing # comment cannot separate the runner from its report path.

Why one job. A ruleset matches a required check by name. A called job reports as <calling job's name, else its id> / conformance, so a consumer has exactly one name to keep stable.

How it finds its own tools. Inside a called workflow, github.job_workflow_sha is empty (measured). The workflow reads its own commit from the run's referenced_workflows and fetches kindkit's tools/ there. That read needs actions: read, which the workflow requests, so the calling job must grant contents: read and actions: read. kindkit's check.yml kind job does.

Exercised in kindkit's CI by the kind job, which calls the workflow on the toy kv tree with:

tests/run_kv.py as the runner;
kvkind.Alt as the second implementation, which shares no source fragment with Good;
tests/kv_mutants.py as the gate: 3 mutants of Good, run through kindkit.gate and probe_command.

The README has a new section, "Using the workflow". It covers what each step requires, the check name, the permissions, and how to adopt the workflow where conformance is already a required check (§3.2): which steps stay in the adopter's own job, when to add the new context and when to swap it in, and to edit the ruleset just before merging, not in the change.

Testing

Local, on the current head, with main (#21) merged in:

$ just check
All checks passed!
$ just test
201 passed
$ just cases
0 invalid case(s) across 6 case(s) in 1 tree(s)
$ just mutants        # throwaway detached worktree of 3e426b7
136 mutations, 136 caught, 0 survived, 0 broken

There are 30 mutations on tools/conform.py, tools/check_gate.py, and the new root, environment-path and gate-report code in kindkit/. For each one, I applied it with the gate's hash-checked apply and ran every test it lists on its own, in a separate pytest process. Every listed test exited 1, and each file restored to its original hash.

Review findings, each shown red first

F1, the root was never checked. These tests were red on the previous head:
- test_a_runner_that_falls_back_to_its_default_tree_is_no_verdict: the reviewer's case. conform.py <broken copy> "…run_kv.py … Good" returned 0.
- test_a_runner_that_read_a_same_named_tree_elsewhere_is_no_verdict: also returned 0.
- A control, test_the_broken_copy_is_a_failure_when_the_runner_reads_it, shows the copy really is broken: a run that reads it exits 1.
All of these now exit 2 as expected. A report with no root is also exit 2.
Commands with a trailing newline or comment were red on the previous head: NO VERDICT: the suite exited 127 and NO VERDICT: the suite wrote no --report-json file. Both are now 0.
F2, the mutation step accepted any exit 0. check_gate.py has 21 tests. true, "", blank, and a command that exits 0 without a gate are all exit 2. A report with a hole or with nothing killed is exit 1. A report with no root, another root, fewer cases or an unknown format is exit 2. The real kv gate over a same-named copy is exit 2.
F3, a second implementation run by the suite's command. conform(..., differs_from=command) raised TypeError before this change; now it is exit 2, also for a respaced copy of the command.
The "wrote no report" mutation. It is reworked so that its mutant returns exit 0, a false pass the test catches by assertion, not a FileNotFoundError.
The tree check was hidden behind the root check. The full gate reported an empty or missing tree is left to the suite to judge as SURVIVED once the root check existed: the tests were being refused by the root check first. The tests now hand a report naming the same root, so only the tree check can refuse it. check_gate.py got the same test shape and its own mutation.

On CI

Green, PR run on 3e426b7: kind / conformance passed. kindkit was fetched at the PR's commit, both suites ran all 6 cases under tests/fixtures/kv, and the line the gate killed 3 mutant(s) over all 6 case(s) appears in the log.

Red, earlier head: throwaway branches tmp/kind-red-broken-case (run 37685697020) and tmp/kind-red-inputs (run 37685723625). A broken kv case, a tree with no cases, a missing tree, a suite run on a subtree, a failing second implementation, a suite with no report, blank mutants, and a failing gate each went red at exactly the expected steps.

This head: throwaway branch tmp/kind-red-review, runs 37690440366 and 37690440235.

job	result	reason in the log
runner reads its default tree (`fixture-root: tests/fixtures/kv-copy`, one case broken)	red: suite, second, mutants	`the suite read '…/tests/fixtures/kv', not '…/tests/fixtures/kv-copy'` (and the same for the gate)
`mutants: "true"`	red: mutants	`the command exited 0 and no gate wrote $KINDKIT_GATE_REPORT`
second implementation = suite command, respaced	red: second	`the second implementation's command is the suite's command`
commands as YAML blocks and with a trailing `# comment`; caller job with `name: a named caller`	green, reported as `a named caller / conformance`	all 6 cases, 3 killed
caller granting only `contents: read` (separate workflow file)	`startup_failure`: the run never starts	GitHub reports a workflow-file issue; the API does not expose the message text

All throwaway branches are deleted (git ls-remote --heads origin | grep -c tmp/ gives 0).

Against rowspec, without changing rowspec

In a scratch clone of rowspec main (2a89617), I installed this branch's kindkit into the clone's venv and ran this branch's tools with rowspec's own commands:

conform.py conformance/cases "cd conformance && …/python run_cases.py"                    0 failure(s) across all 410 case(s)   exit 0
conform.py --differs-from <that> conformance/cases "… run_cases.py rowspec_alt.table"     0 failure(s) across all 410 case(s)   exit 0
check_gate.py conformance/cases "cd conformance && …/python mutants.py"                    the gate killed 74 mutant(s) over all 410 case(s)   exit 0

On the earlier head, a broken rowspec fixture gave the same red verdict through conform.py as rowspec's runner run directly: 1 failure(s) in each.

rowspec's current pin, a26f8f8, has neither the root key nor KINDKIT_REPORT_JSON. git show a26f8f8:kindkit/runner.py | grep -c '"root"' gives 0, and the same for cli.py. So adopting the workflow in rowspec includes bumping that pin.

Limits, stated in the README

Different command text that reaches the same second implementation cannot be told apart from outside.
The gate has to probe through probe_command, or set Verdict(root=...) itself, and one command may run only one gate() call.
The referenced_workflows read has not been tried on a private repository.
A matrix caller's check-name suffix is not measured here.

Not done here, deliberately

rowspec does not call this yet. Adopting it is a separate change in rowspec: bump the kindkit pin, then follow the README's §3.2 steps.
kind / conformance is not a required check on kindkit. Adding it is a ruleset change for the owner.

After merging main with #19 (merge commit 3d6f94b): just check all checks passed; just test 202 passed; just mutants (throwaway detached worktree) 137 mutations, 137 caught, 0 survived, 0 broken. CI green.

Round 2 of review (head `9951ee2`, with `main` merged in, which brought #20, #22 and #23)

M1, a relative root was accepted. Before this change, _parse_report checked only that root is a string, and conform.py resolved it against its own working directory. The reviewer's runner did cd good && runner kvcopy and reported "root": "kvcopy". That passed against a broken ./kvcopy. New test test_a_report_whose_root_is_relative_is_no_verdict rebuilds that shape and was red on 3d6f94b: assert 0 == 2. _parse_report now refuses a non-absolute root with 'root' 'kvcopy' is not an absolute path; unit test test_a_report_whose_root_is_relative_is_refused. The README contract now says the root is absolute. New gate mutation "a relative root is accepted, to be resolved by whoever reads it"; both listed tests fail on their own under it.
L1, the every-case-ran check was masked. The subtree test was also refused by the root check, so emptying the list of unrun cases survived the suite. New test test_a_report_from_the_right_root_that_ran_only_some_cases_is_no_verdict hands conform.py a report naming the same root with 2 of the 6 cases. It is now listed by "a suite that ran a different set of cases than the tree passes", and by a new mutation "a case of the tree that did not run goes unnoticed" (sorted(tree - ran) → []). Run on its own, each mutation fails that test.
L2, which file was mutated. This is documented as a limit: a gate whose only kill breaks the adapter or the second implementation still passes. The gate report now records source, the file mutated, and check_gate.py prints it. On CI: the gate killed 3 mutant(s) of '/home/runner/work/kindkit/kindkit/tests/kvkind.py' over all 6 case(s).
The gate report's keys are now documented in the README: kindkit_gate_report, ok, killed, survived, equivalent, stale, bogus, broken, ran, root and source.

Gates at 9951ee2:

$ just check
All checks passed!
$ just test
220 passed
$ just mutants        # throwaway detached worktree
148 mutations, 148 caught, 0 survived, 0 broken

CI on the PR: check and kind / conformance pass; mergeable state clean.

Refs #4. Every bullet is met on kindkit's side. "rowspec's existing workflows can call it and produce the same verdicts" is shown above against rowspec, but not yet by rowspec's own workflows.
Refs #3. "Validate the schema against all 410 cases in CI" becomes true when rowspec calls this workflow. Until then it holds for kindkit's toy tree in CI, and for rowspec's 410 cases only locally (above).

`.github/workflows/kind.yml` (`on: workflow_call`) takes a kind's fixture root and three commands -- the suite, the second implementation and the mutation gate -- and runs, as steps of one job named `conformance`: - the case-tree validator over the fixture root; - `tools/conform.py` for each implementation. It appends `--report-json` and passes only when the cases the report says ran are exactly the case directories under the root and none failed. An empty or missing tree, a runner pointed elsewhere or at a subtree, a missing report, or a report its exit code contradicts is exit 2; - the mutation gate, which must exit 0. A blank command is refused. One job, so a consumer's required check is the single, stable `<job id> / conformance`. The workflow fetches kindkit's tools at the commit the caller pinned, read from the run's `referenced_workflows`, because `github.job_workflow_sha` is empty in a called workflow. kindkit's check.yml calls it on the toy kv tree, with a runner (`tests/run_kv.py`), a second kv implementation (`kvkind.Alt`) and a kv mutation gate (`tests/kv_mutants.py`). `tools/conform.py` is under the mutation gate.

The suite steps compared case ids only. A runner that fell back to its own default tree, or read a same-named copy elsewhere, passed over a fixture root it never opened. And the mutation step accepted any command that exits 0. - kindkit.cli writes the absolute `root` it read into the report, and takes the report path from KINDKIT_REPORT_JSON when no --report-json is given, refusing a file already there. - tools/conform.py runs the command as written with that variable set, so a trailing newline or comment no longer separates the runner from its report path. It requires the report's root to be the fixture root, and with --differs-from refuses a second implementation run by the suite's own command. - kindkit.gate writes its verdicts, the baseline's case ids and its root to KINDKIT_GATE_REPORT when set; Verdict carries the root probe_command reads. tools/check_gate.py requires that report: a kill, no hole, and a baseline that ran exactly the named tree. - kind.yml requests actions: read for the referenced_workflows lookup, so the calling job must grant it; check.yml's kind job does. - README: the report's root and environment path, what each step requires, the check name's prefix, and how to adopt the workflow where `conformance` is already a required check. The "wrote no report" mutation now yields a false pass the test catches by assertion, instead of a crash.

…rkflow # Conflicts: # tools/mutation_gate.py

With the report's root now compared, the empty- and missing-tree tests were refused by the root check first, and the gate mutation that skips the tree check survived. Those tests now hand a report naming the same root, so only the tree check can refuse it; check_gate.py gets the same test shape and a mutation of its own. A cli mutation whose pattern the environment-path change had made stale is updated.

…rkflow # Conflicts: # tools/mutation_gate.py

…rkflow

A report's `root` was accepted as any string, and conform.py resolved it against its own working directory. A runner that changed directory and reported "kvcopy" passed against a different, broken ./kvcopy. `_parse_report` now refuses a root that is not absolute. The every-case-ran check was reachable only behind the root check, so a mutation emptying its list of unrun cases survived the suite. A test now hands conform.py a report from the right root with fewer cases. The gate report also records `source`, the file the gate mutated, and check_gate.py prints it. The README documents the gate report's keys, the absolute root, and that the mutation step cannot tell which file was mutated.

## What Moves rowspec's kindkit dev-dependency pin from `bf716e3` (the commit `v0.1.0` names) to `8e21b558c85d5612009f3a1d69a3a68bc0191637`. That is kindkit's `main` after its reusable workflow landed (kindspec/kindkit#24). kindkit has no tag newer than `v0.1.0`, so the pin is a commit, as before. This is the first of two PRs. The second makes rowspec's CI call `kind.yml`, and that workflow reads three things `bf716e3` does not have: - the `root` key in the runner's `--report-json`; - `KINDKIT_REPORT_JSON`, the runner's report path taken from the environment; - `KINDKIT_GATE_REPORT`, where `gate()` writes its verdicts. Changed files: `pyproject.toml` (pin and its comment), `uv.lock`, and `CHANGELOG.md` (Unreleased). ## No rowspec code changes `conformance/mutants.py` already probes through `kindkit.probe_command`. At the new pin that call carries the report's `root` into `Verdict.root`, and `gate()` writes the gate report itself when `KINDKIT_GATE_REPORT` is set. So the probe needs no edit to satisfy the workflow's contract. Shown below by running kindkit's own workflow tools against this branch. ## Measured Each recipe ran on `origin/main` (old pin) and on this branch, from the same worktree, with output captured to files and diffed: ``` recipe old pin new pin diff check exit 0 exit 0 identical conform exit 0 exit 0 identical 0 failure(s) across 410 case(s) conform-alt exit 0 exit 0 identical 0 failure(s) across 410 case(s) mutants exit 0 exit 0 identical 73 killed, 0 survived, 2 equivalent, 0 stale, 0 broken test exit 0 exit 0 wall time only ``` The only differing line, from `diff before/test.log after/test.log`: ``` < 72 passed, 1 skipped in 133.35s (0:02:13) > 72 passed, 1 skipped in 135.44s (0:02:15) ``` `mutants` is identical line for line, killing cases included. The kit's new per-write mtime stamp and the stricter `gitmerge` changed no verdict. ### The workflow's contract, run locally These are kindkit's `tools/` at `8e21b55`, run from this branch's root, as `kind.yml` runs them: ``` $ python3 $K/tools/validate_case_tree.py conformance/cases 0 invalid case(s) across 410 case(s) in 1 tree(s) exit 0 $ python3 $K/tools/conform.py conformance/cases "cd conformance && uv run python run_cases.py" 0 failure(s) across all 410 case(s) under 'conformance/cases' $ python3 $K/tools/conform.py --differs-from "cd conformance && uv run python run_cases.py" \ conformance/cases "cd conformance && uv run python run_cases.py rowspec_alt.table" 0 failure(s) across all 410 case(s) under 'conformance/cases' $ python3 $K/tools/check_gate.py conformance/cases "cd conformance && uv run python mutants.py" 73 killed, 0 survived, 2 equivalent, 0 stale, 0 broken the gate killed 73 mutant(s) of '<worktree>/reference/rowspec/table.py' over all 410 case(s) exit 0 ``` `check_gate.py` passing means three things. The gate wrote a format-1 report. Its baseline `root` resolved to `conformance/cases`. Its `ran` set was exactly the 410 case directories. The same `conform.py` run against `origin/main` (old pin) has no verdict. The old runner ignores `KINDKIT_REPORT_JSON` and writes no report: ``` NO VERDICT: the suite wrote no report at $KINDKIT_REPORT_JSON exit 2 ``` ## Not in this PR There is no CI change, so the required checks are the same four and no ruleset edit is needed to merge this. Refs kindspec/kindkit#4 Refs kindspec/kindkit#3

CameronBrooks11 added 7 commits October 7, 2026 16:52

Merge remote-tracking branch 'origin/main' into feat/reusable-kind-wo…

970cfc0

…rkflow # Conflicts: # tools/mutation_gate.py

Merge remote-tracking branch 'origin/main' into feat/reusable-kind-wo…

3d6f94b

…rkflow # Conflicts: # tools/mutation_gate.py

Merge remote-tracking branch 'origin/main' into feat/reusable-kind-wo…

e2bb2cd

…rkflow

CameronBrooks11 merged commit 8e21b55 into main Oct 7, 2026
7 checks passed

CameronBrooks11 deleted the feat/reusable-kind-workflow branch October 7, 2026 22:08

CameronBrooks11 mentioned this pull request Oct 8, 2026

build(deps): move the kindkit pin to 8e21b55 kindspec/rowspec#77

Merged

CameronBrooks11 mentioned this pull request Oct 8, 2026

A reusable CI workflow that runs the suite, the gate, and the second implementation #4

Closed

Navigation Menu

Repository navigation

feat(ci): add a reusable workflow that gates a kind's tree and suites - #24

feat(ci): add a reusable workflow that gates a kind's tree and suites#24
CameronBrooks11 merged 7 commits into
mainfrom
feat/reusable-kind-workflow

CameronBrooks11 commented Oct 7, 2026 •

edited

Loading

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

1 participant

Conversation

CameronBrooks11 commented Oct 7, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

What

Testing

Review findings, each shown red first

On CI

Against rowspec, without changing rowspec

Limits, stated in the README

Not done here, deliberately

Round 2 of review (head 9951ee2, with main merged in, which brought #20, #22 and #23)

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

1 participant

CameronBrooks11 commented Oct 7, 2026 •

edited

Loading

Round 2 of review (head `9951ee2`, with `main` merged in, which brought #20, #22 and #23)