Skip to content

fix(gate): refuse a byte splice that breaks the JSON it mutates - #22

Merged
CameronBrooks11 merged 5 commits into
mainfrom
fix/byte-splice-refuses-a-syntax-break
Oct 7, 2026
Merged

CameronBrooks11 merged 5 commits into
mainfrom
fix/byte-splice-refuses-a-syntax-break

Conversation

@CameronBrooks11

@CameronBrooks11 CameronBrooks11 commented Oct 7, 2026 •

Copy link
Copy Markdown
Member

What

tools/mutation_gate.py, _splice: after a byte splice into a .json target, the result is passed to json.loads. If it does not parse, _splice raises MutantError("mutated JSON does not parse: ..."). apply already hands a MutantError back to main, which scores the mutation BROKEN and fails the run. The docstring now says this check exists, and that any other extension is spliced unchecked.

tests/test_mutation_gate.py:

  • test_a_byte_splice_that_breaks_the_json_is_refused_not_scored splices "required": ["kind",, into the schema and requires MutantError.
  • test_every_json_mutation_in_the_gate_still_parses, parametrised over every .json mutation in MUTATIONS, covers the other direction: the new refusal must not reject any real mutation.

One new mutation, "a byte splice that breaks the JSON is scored instead of refused", disables the check.

Why

A token splice is compiled; a byte splice was parsed by nothing. A mutation that only broke the file's syntax made pytest fall over on a parse error, and the gate scored that as a kill of a change nothing had detected.

Testing

Red first, on main (bf716e3). I put the issue's mutation through the real gate (MUTATIONS replaced by that one entry, main() called) in a throwaway worktree, because the gate writes in place:

  caught    syntax break in the schema  [9dc95205726c -> 93b44a35270c]
1 mutations, 1 caught, 0 survived, 0 broken
gate exit 0

The new unit test on main:

$ uv run pytest -q tests/test_mutation_gate.py
E       Failed: DID NOT RAISE MutantError
1 failed, 6 passed

Same one-entry gate run, on this branch:

  BROKEN    syntax break in the schema  [mutated JSON does not parse: Expecting value: line 8 column 23 (char 626)]
1 mutations, 0 caught, 0 survived, 1 broken
gate exit 1

The valid direction: all four real schema mutations still splice to parseable JSON and are still caught.

$ just check
All checks passed!
$ just test
150 passed
$ just mutants        # in a throwaway detached worktree of this commit
  caught    'kind' stops being required
  caught    'kind' may be any JSON scalar
  caught    'kind' stops having to be lower-kebab
  caught    case-body keys stop having to be snake_case
  caught    a byte splice that breaks the JSON is scored instead of refused
100 mutations, 100 caught, 0 survived, 0 broken

The count is main's 99 plus the one new mutation, and the 99 are all still caught. The issue's "73" was the count when it was filed. I also applied the new mutation with the gate's hash-checked apply and ran its listed test on its own: exit 1, and the file restored to its original hash.

The issue's last bullet asks for _splice's docstring to be corrected. That correction is already on main: since #18 the docstring says a layout-sensitive pattern is "loud rather than silent" and scored BROKEN. This PR only adds the paragraph about checking the splice.

Interaction with #19: #19 adds a byte-path mutation of .pre-commit-config.yaml. Under this change, YAML is spliced unchecked, as the docstring now says, because the stdlib has no YAML parser. That mutation keeps the file valid YAML and its test reads the file with a regex, so it is unaffected.

Follow-up (review): the parse check could skip itself

test_every_json_mutation_in_the_gate_still_parses is parametrised over MUTATIONS filtered to .json paths. With an empty filter, pytest skips the test and exits 0. Shown in a throwaway worktree, with the filter changed to .nothing:

SKIPPED [1] tests/test_mutation_gate.py:128: got empty parameter set for (mutation)
1 skipped, 3 deselected
exit=0
  • The filtered list is now a module constant, JSON_MUTATIONS. A plain, non-parametrised test_there_are_json_mutations_to_check requires at least 4 entries, the schema mutations the gate carries today. With the same emptied filter it fails: FAILED ...test_there_are_json_mutations_to_check, 1 failed, 3 passed, 1 skipped.
  • New mutation "the JSON mutations the parse check covers filter down to none". It empties the filter, and its listed test exits 1 on its own (gate apply, hash-checked, file restored).

With main (#21) merged in, at 5ee2d6f:

$ just check
All checks passed!
$ just test
158 passed
$ just mutants        # throwaway detached worktree
  caught    a byte splice that breaks the JSON is scored instead of refused
  caught    the JSON mutations the parse check covers filter down to none
109 mutations, 109 caught, 0 survived, 0 broken

After merging main with #19 (merge commit 3404ef9): just check all checks passed; just test 159 passed; just mutants (throwaway detached worktree) 110 mutations, 110 caught, 0 survived, 0 broken. CI green.

Closes #10

`_splice` compiles a token splice into Python but accepted any byte
splice unparsed. Splicing `"required": ["kind",,` into the case-tree
schema was written, pytest fell over on the invalid JSON, and the gate
scored the mutation caught: a kill credited to a parse error rather
than to the change the mutation names.

Parse the result when the target is `.json` and raise MutantError if it
does not parse, so the gate reports BROKEN. A test requires every JSON
mutation in the gate to still parse, so the refusal cannot quietly cost
a real one. The gate disables the new check and requires its test to
notice.
`test_every_json_mutation_in_the_gate_still_parses` is parametrised
over the gate's .json mutations. An empty filter makes pytest SKIP it
with "got empty parameter set" and exit 0, a pass over nothing.

A plain test now requires at least four, the schema mutations the gate
carries today, and a gate mutation empties the filter to watch it fail.
…es-a-syntax-break

# Conflicts:
#	tools/mutation_gate.py
@CameronBrooks11
CameronBrooks11 merged commit 273a323 into main Oct 7, 2026
5 checks passed
@CameronBrooks11
CameronBrooks11 deleted the fix/byte-splice-refuses-a-syntax-break branch October 7, 2026 21:51
CameronBrooks11 added a commit that referenced this pull request Oct 7, 2026
…#24)

## What

**`.github/workflows/kind.yml`** is a reusable workflow (`on:
workflow_call`). Its inputs, all required: `fixture-root`, `suite`,
`second-implementation` and `mutants`. It runs as **one job named
`conformance`**, and its four gates are steps with `if: always()`. Each
gate runs even when an earlier gate is red. The setup steps before them
are not like that: if one fails, the gates still run and fail too, for
want of kindkit's tools.

| step | runs | exit 0 only if |
|---|---|---|
| the case tree follows the convention | `tools/validate_case_tree.py
<root>` | every manifest is valid; exit 2 on an empty or missing tree |
| the reference implementation passes every case in the tree |
`tools/conform.py <root> <suite>` | the runner's report names `root` as
the fixture root, it ran exactly the case directories under it, and none
failed |
| the second implementation passes every case in the tree |
`tools/conform.py --differs-from <suite> <root> <second-implementation>`
| the same as above, and the command is not the suite's command with
only whitespace changed |
| the suite must be able to fail | `tools/check_gate.py <root>
<mutants>` | `kindkit.gate`'s own report exists, killed ≥1 mutant, has
no SURVIVED, STALE, BOGUS or BROKEN mutant, and its baseline ran exactly
the tree under that root |

**How a verdict is tied to the tree the workflow names:**

- `kindkit.cli` writes the absolute `root` it read into its report. A
report whose `root` is not absolute is refused as malformed (round 2).
- `kindkit.cli` takes the report path from `KINDKIT_REPORT_JSON` when
`--report-json` is not given, and refuses a file already at that path.
- `Verdict` carries the `root` that `probe_command` reads from the
report.
- `kindkit.gate` writes its verdicts, the baseline's case ids and its
root to `KINDKIT_GATE_REPORT` when that is set. It creates the file
exclusively, so a second gate in the same command raises.

`conform.py` and `check_gate.py` run the caller's command **as
written**, with those variables set. Nothing is appended to the command,
so a trailing newline (`suite: |`), a `cd`, or a trailing `# comment`
cannot separate the runner from its report path.

**Why one job.** A ruleset matches a required check by name. A called
job reports as `<calling job's name, else its id> / conformance`, so a
consumer has exactly one name to keep stable.

**How it finds its own tools.** Inside a called workflow,
`github.job_workflow_sha` is empty (measured). The workflow reads its
own commit from the run's `referenced_workflows` and fetches kindkit's
`tools/` there. That read needs `actions: read`, which the workflow
requests, so **the calling job must grant `contents: read` and `actions:
read`**. kindkit's `check.yml` `kind` job does.

**Exercised in kindkit's CI** by the `kind` job, which calls the
workflow on the toy kv tree with:

- `tests/run_kv.py` as the runner;
- `kvkind.Alt` as the second implementation, which shares no source
fragment with `Good`;
- `tests/kv_mutants.py` as the gate: 3 mutants of `Good`, run through
`kindkit.gate` and `probe_command`.

The README has a new section, "Using the workflow". It covers what each
step requires, the check name, the permissions, and **how to adopt the
workflow where `conformance` is already a required check** (§3.2): which
steps stay in the adopter's own job, when to add the new context and
when to swap it in, and to edit the ruleset just before merging, not in
the change.

## Testing

Local, on the current head, with `main` (#21) merged in:

```
$ just check
All checks passed!
$ just test
201 passed
$ just cases
0 invalid case(s) across 6 case(s) in 1 tree(s)
$ just mutants        # throwaway detached worktree of 3e426b7
136 mutations, 136 caught, 0 survived, 0 broken
```

There are 30 mutations on `tools/conform.py`, `tools/check_gate.py`, and
the new `root`, environment-path and gate-report code in `kindkit/`. For
each one, I applied it with the gate's hash-checked `apply` and ran
every test it lists on its own, in a separate pytest process. Every
listed test exited 1, and each file restored to its original hash.

### Review findings, each shown red first

- **F1, the root was never checked.** These tests were red on the
previous head:
- `test_a_runner_that_falls_back_to_its_default_tree_is_no_verdict`: the
reviewer's case. `conform.py <broken copy> "…run_kv.py … Good"` returned
`0`.
- `test_a_runner_that_read_a_same_named_tree_elsewhere_is_no_verdict`:
also returned `0`.
- A control,
`test_the_broken_copy_is_a_failure_when_the_runner_reads_it`, shows the
copy really is broken: a run that reads it exits 1.

All of these now exit 2 as expected. A report with no `root` is also
exit 2.
- **Commands with a trailing newline or comment** were red on the
previous head: `NO VERDICT: the suite exited 127` and `NO VERDICT: the
suite wrote no --report-json file`. Both are now `0`.
- **F2, the mutation step accepted any exit 0.** `check_gate.py` has 21
tests. `true`, `""`, blank, and a command that exits 0 without a gate
are all exit 2. A report with a hole or with nothing killed is exit 1. A
report with no root, another root, fewer cases or an unknown format is
exit 2. The real kv gate over a same-named copy is exit 2.
- **F3, a second implementation run by the suite's command.**
`conform(..., differs_from=command)` raised `TypeError` before this
change; now it is exit 2, also for a respaced copy of the command.
- **The "wrote no report" mutation.** It is reworked so that its mutant
returns exit 0, a false pass the test catches by assertion, not a
`FileNotFoundError`.
- **The tree check was hidden behind the root check.** The full gate
reported `an empty or missing tree is left to the suite to judge` as
**SURVIVED** once the root check existed: the tests were being refused
by the root check first. The tests now hand a report naming the same
root, so only the tree check can refuse it. `check_gate.py` got the same
test shape and its own mutation.

### On CI

**Green**, PR run on `3e426b7`: `kind / conformance` passed. kindkit was
fetched at the PR's commit, both suites ran all 6 cases under
`tests/fixtures/kv`, and the line `the gate killed 3 mutant(s) over all
6 case(s)` appears in the log.

**Red, earlier head:** throwaway branches `tmp/kind-red-broken-case`
(run 37685697020) and `tmp/kind-red-inputs` (run 37685723625). A broken
kv case, a tree with no cases, a missing tree, a suite run on a subtree,
a failing second implementation, a suite with no report, blank
`mutants`, and a failing gate each went red at exactly the expected
steps.

**This head:** throwaway branch `tmp/kind-red-review`, runs 37690440366
and 37690440235.

| job | result | reason in the log |
|---|---|---|
| runner reads its default tree (`fixture-root: tests/fixtures/kv-copy`,
one case broken) | red: suite, second, mutants | `the suite read
'…/tests/fixtures/kv', not '…/tests/fixtures/kv-copy'` (and the same for
the gate) |
| `mutants: "true"` | red: mutants | `the command exited 0 and no gate
wrote $KINDKIT_GATE_REPORT` |
| second implementation = suite command, respaced | red: second | `the
second implementation's command is the suite's command` |
| commands as YAML blocks and with a trailing `# comment`; caller job
with `name: a named caller` | **green**, reported as `a named caller /
conformance` | all 6 cases, 3 killed |
| caller granting only `contents: read` (separate workflow file) |
`startup_failure`: the run never starts | GitHub reports a workflow-file
issue; the API does not expose the message text |

All throwaway branches are deleted (`git ls-remote --heads origin | grep
-c tmp/` gives `0`).

### Against rowspec, without changing rowspec

In a scratch clone of rowspec `main` (`2a89617`), I installed this
branch's kindkit into the clone's venv and ran this branch's tools with
rowspec's own commands:

```
conform.py conformance/cases "cd conformance && …/python run_cases.py"                    0 failure(s) across all 410 case(s)   exit 0
conform.py --differs-from <that> conformance/cases "… run_cases.py rowspec_alt.table"     0 failure(s) across all 410 case(s)   exit 0
check_gate.py conformance/cases "cd conformance && …/python mutants.py"                    the gate killed 74 mutant(s) over all 410 case(s)   exit 0
```

On the earlier head, a broken rowspec fixture gave the same red verdict
through `conform.py` as rowspec's runner run directly: `1 failure(s)` in
each.

rowspec's current pin, `a26f8f8`, has neither the `root` key nor
`KINDKIT_REPORT_JSON`. `git show a26f8f8:kindkit/runner.py | grep -c
'"root"'` gives `0`, and the same for `cli.py`. So adopting the workflow
in rowspec includes bumping that pin.

## Limits, stated in the README

- Different command text that reaches the same second implementation
cannot be told apart from outside.
- The gate has to probe through `probe_command`, or set
`Verdict(root=...)` itself, and one command may run only one `gate()`
call.
- The `referenced_workflows` read has not been tried on a private
repository.
- A matrix caller's check-name suffix is not measured here.

## Not done here, deliberately

- **rowspec does not call this yet.** Adopting it is a separate change
in rowspec: bump the kindkit pin, then follow the README's §3.2 steps.
- **`kind / conformance` is not a required check on kindkit.** Adding it
is a ruleset change for the owner.

**After merging `main` with #19** (merge commit `3d6f94b`): `just check`
all checks passed; `just test` 202 passed; `just mutants` (throwaway
detached worktree) 137 mutations, 137 caught, 0 survived, 0 broken. CI
green.

## Round 2 of review (head `9951ee2`, with `main` merged in, which
brought #20, #22 and #23)

- **M1, a relative `root` was accepted.** Before this change,
`_parse_report` checked only that `root` is a string, and `conform.py`
resolved it against its own working directory. The reviewer's runner did
`cd good && runner kvcopy` and reported `"root": "kvcopy"`. That passed
against a **broken** `./kvcopy`. New test
`test_a_report_whose_root_is_relative_is_no_verdict` rebuilds that shape
and was red on `3d6f94b`: `assert 0 == 2`. `_parse_report` now refuses a
non-absolute `root` with `'root' 'kvcopy' is not an absolute path`; unit
test `test_a_report_whose_root_is_relative_is_refused`. The README
contract now says the root is **absolute**. New gate mutation "a
relative root is accepted, to be resolved by whoever reads it"; both
listed tests fail on their own under it.
- **L1, the every-case-ran check was masked.** The subtree test was also
refused by the root check, so emptying the list of unrun cases survived
the suite. New test
`test_a_report_from_the_right_root_that_ran_only_some_cases_is_no_verdict`
hands `conform.py` a report naming the same root with 2 of the 6 cases.
It is now listed by "a suite that ran a different set of cases than the
tree passes", and by a new mutation "a case of the tree that did not run
goes unnoticed" (`sorted(tree - ran)` → `[]`). Run on its own, each
mutation fails that test.
- **L2, which file was mutated.** This is documented as a limit: a gate
whose only kill breaks the adapter or the second implementation still
passes. The gate report now records `source`, the file mutated, and
`check_gate.py` prints it. On CI: `the gate killed 3 mutant(s) of
'/home/runner/work/kindkit/kindkit/tests/kvkind.py' over all 6 case(s)`.
- **The gate report's keys** are now documented in the README:
`kindkit_gate_report`, `ok`, `killed`, `survived`, `equivalent`,
`stale`, `bogus`, `broken`, `ran`, `root` and `source`.

Gates at `9951ee2`:

```
$ just check
All checks passed!
$ just test
220 passed
$ just mutants        # throwaway detached worktree
148 mutations, 148 caught, 0 survived, 0 broken
```

CI on the PR: `check` and `kind / conformance` pass; mergeable state
`clean`.

Refs #4. Every bullet is met on kindkit's side. "rowspec's existing
workflows can call it and produce the same verdicts" is shown above
against rowspec, but not yet by rowspec's own workflows.
Refs #3. "Validate the schema against all 410 cases in CI" becomes true
when rowspec calls this workflow. Until then it holds for kindkit's toy
tree in CI, and for rowspec's 410 cases only locally (above).
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

The byte-splice path scores a syntax break as a kill

1 participant