Repository navigation
fix(gate): refuse a byte splice that breaks the JSON it mutates - #22
Merged
Merged
Conversation
`_splice` compiles a token splice into Python but accepted any byte splice unparsed. Splicing `"required": ["kind",,` into the case-tree schema was written, pytest fell over on the invalid JSON, and the gate scored the mutation caught: a kill credited to a parse error rather than to the change the mutation names. Parse the result when the target is `.json` and raise MutantError if it does not parse, so the gate reports BROKEN. A test requires every JSON mutation in the gate to still parse, so the refusal cannot quietly cost a real one. The gate disables the new check and requires its test to notice.
`test_every_json_mutation_in_the_gate_still_parses` is parametrised over the gate's .json mutations. An empty filter makes pytest SKIP it with "got empty parameter set" and exit 0, a pass over nothing. A plain test now requires at least four, the schema mutations the gate carries today, and a gate mutation empties the filter to watch it fail.
…es-a-syntax-break
…es-a-syntax-break # Conflicts: # tools/mutation_gate.py
CameronBrooks11
added a commit
that referenced
this pull request
Oct 7, 2026
…#24) ## What **`.github/workflows/kind.yml`** is a reusable workflow (`on: workflow_call`). Its inputs, all required: `fixture-root`, `suite`, `second-implementation` and `mutants`. It runs as **one job named `conformance`**, and its four gates are steps with `if: always()`. Each gate runs even when an earlier gate is red. The setup steps before them are not like that: if one fails, the gates still run and fail too, for want of kindkit's tools. | step | runs | exit 0 only if | |---|---|---| | the case tree follows the convention | `tools/validate_case_tree.py <root>` | every manifest is valid; exit 2 on an empty or missing tree | | the reference implementation passes every case in the tree | `tools/conform.py <root> <suite>` | the runner's report names `root` as the fixture root, it ran exactly the case directories under it, and none failed | | the second implementation passes every case in the tree | `tools/conform.py --differs-from <suite> <root> <second-implementation>` | the same as above, and the command is not the suite's command with only whitespace changed | | the suite must be able to fail | `tools/check_gate.py <root> <mutants>` | `kindkit.gate`'s own report exists, killed ≥1 mutant, has no SURVIVED, STALE, BOGUS or BROKEN mutant, and its baseline ran exactly the tree under that root | **How a verdict is tied to the tree the workflow names:** - `kindkit.cli` writes the absolute `root` it read into its report. A report whose `root` is not absolute is refused as malformed (round 2). - `kindkit.cli` takes the report path from `KINDKIT_REPORT_JSON` when `--report-json` is not given, and refuses a file already at that path. - `Verdict` carries the `root` that `probe_command` reads from the report. - `kindkit.gate` writes its verdicts, the baseline's case ids and its root to `KINDKIT_GATE_REPORT` when that is set. It creates the file exclusively, so a second gate in the same command raises. `conform.py` and `check_gate.py` run the caller's command **as written**, with those variables set. Nothing is appended to the command, so a trailing newline (`suite: |`), a `cd`, or a trailing `# comment` cannot separate the runner from its report path. **Why one job.** A ruleset matches a required check by name. A called job reports as `<calling job's name, else its id> / conformance`, so a consumer has exactly one name to keep stable. **How it finds its own tools.** Inside a called workflow, `github.job_workflow_sha` is empty (measured). The workflow reads its own commit from the run's `referenced_workflows` and fetches kindkit's `tools/` there. That read needs `actions: read`, which the workflow requests, so **the calling job must grant `contents: read` and `actions: read`**. kindkit's `check.yml` `kind` job does. **Exercised in kindkit's CI** by the `kind` job, which calls the workflow on the toy kv tree with: - `tests/run_kv.py` as the runner; - `kvkind.Alt` as the second implementation, which shares no source fragment with `Good`; - `tests/kv_mutants.py` as the gate: 3 mutants of `Good`, run through `kindkit.gate` and `probe_command`. The README has a new section, "Using the workflow". It covers what each step requires, the check name, the permissions, and **how to adopt the workflow where `conformance` is already a required check** (§3.2): which steps stay in the adopter's own job, when to add the new context and when to swap it in, and to edit the ruleset just before merging, not in the change. ## Testing Local, on the current head, with `main` (#21) merged in: ``` $ just check All checks passed! $ just test 201 passed $ just cases 0 invalid case(s) across 6 case(s) in 1 tree(s) $ just mutants # throwaway detached worktree of 3e426b7 136 mutations, 136 caught, 0 survived, 0 broken ``` There are 30 mutations on `tools/conform.py`, `tools/check_gate.py`, and the new `root`, environment-path and gate-report code in `kindkit/`. For each one, I applied it with the gate's hash-checked `apply` and ran every test it lists on its own, in a separate pytest process. Every listed test exited 1, and each file restored to its original hash. ### Review findings, each shown red first - **F1, the root was never checked.** These tests were red on the previous head: - `test_a_runner_that_falls_back_to_its_default_tree_is_no_verdict`: the reviewer's case. `conform.py <broken copy> "…run_kv.py … Good"` returned `0`. - `test_a_runner_that_read_a_same_named_tree_elsewhere_is_no_verdict`: also returned `0`. - A control, `test_the_broken_copy_is_a_failure_when_the_runner_reads_it`, shows the copy really is broken: a run that reads it exits 1. All of these now exit 2 as expected. A report with no `root` is also exit 2. - **Commands with a trailing newline or comment** were red on the previous head: `NO VERDICT: the suite exited 127` and `NO VERDICT: the suite wrote no --report-json file`. Both are now `0`. - **F2, the mutation step accepted any exit 0.** `check_gate.py` has 21 tests. `true`, `""`, blank, and a command that exits 0 without a gate are all exit 2. A report with a hole or with nothing killed is exit 1. A report with no root, another root, fewer cases or an unknown format is exit 2. The real kv gate over a same-named copy is exit 2. - **F3, a second implementation run by the suite's command.** `conform(..., differs_from=command)` raised `TypeError` before this change; now it is exit 2, also for a respaced copy of the command. - **The "wrote no report" mutation.** It is reworked so that its mutant returns exit 0, a false pass the test catches by assertion, not a `FileNotFoundError`. - **The tree check was hidden behind the root check.** The full gate reported `an empty or missing tree is left to the suite to judge` as **SURVIVED** once the root check existed: the tests were being refused by the root check first. The tests now hand a report naming the same root, so only the tree check can refuse it. `check_gate.py` got the same test shape and its own mutation. ### On CI **Green**, PR run on `3e426b7`: `kind / conformance` passed. kindkit was fetched at the PR's commit, both suites ran all 6 cases under `tests/fixtures/kv`, and the line `the gate killed 3 mutant(s) over all 6 case(s)` appears in the log. **Red, earlier head:** throwaway branches `tmp/kind-red-broken-case` (run 37685697020) and `tmp/kind-red-inputs` (run 37685723625). A broken kv case, a tree with no cases, a missing tree, a suite run on a subtree, a failing second implementation, a suite with no report, blank `mutants`, and a failing gate each went red at exactly the expected steps. **This head:** throwaway branch `tmp/kind-red-review`, runs 37690440366 and 37690440235. | job | result | reason in the log | |---|---|---| | runner reads its default tree (`fixture-root: tests/fixtures/kv-copy`, one case broken) | red: suite, second, mutants | `the suite read '…/tests/fixtures/kv', not '…/tests/fixtures/kv-copy'` (and the same for the gate) | | `mutants: "true"` | red: mutants | `the command exited 0 and no gate wrote $KINDKIT_GATE_REPORT` | | second implementation = suite command, respaced | red: second | `the second implementation's command is the suite's command` | | commands as YAML blocks and with a trailing `# comment`; caller job with `name: a named caller` | **green**, reported as `a named caller / conformance` | all 6 cases, 3 killed | | caller granting only `contents: read` (separate workflow file) | `startup_failure`: the run never starts | GitHub reports a workflow-file issue; the API does not expose the message text | All throwaway branches are deleted (`git ls-remote --heads origin | grep -c tmp/` gives `0`). ### Against rowspec, without changing rowspec In a scratch clone of rowspec `main` (`2a89617`), I installed this branch's kindkit into the clone's venv and ran this branch's tools with rowspec's own commands: ``` conform.py conformance/cases "cd conformance && …/python run_cases.py" 0 failure(s) across all 410 case(s) exit 0 conform.py --differs-from <that> conformance/cases "… run_cases.py rowspec_alt.table" 0 failure(s) across all 410 case(s) exit 0 check_gate.py conformance/cases "cd conformance && …/python mutants.py" the gate killed 74 mutant(s) over all 410 case(s) exit 0 ``` On the earlier head, a broken rowspec fixture gave the same red verdict through `conform.py` as rowspec's runner run directly: `1 failure(s)` in each. rowspec's current pin, `a26f8f8`, has neither the `root` key nor `KINDKIT_REPORT_JSON`. `git show a26f8f8:kindkit/runner.py | grep -c '"root"'` gives `0`, and the same for `cli.py`. So adopting the workflow in rowspec includes bumping that pin. ## Limits, stated in the README - Different command text that reaches the same second implementation cannot be told apart from outside. - The gate has to probe through `probe_command`, or set `Verdict(root=...)` itself, and one command may run only one `gate()` call. - The `referenced_workflows` read has not been tried on a private repository. - A matrix caller's check-name suffix is not measured here. ## Not done here, deliberately - **rowspec does not call this yet.** Adopting it is a separate change in rowspec: bump the kindkit pin, then follow the README's §3.2 steps. - **`kind / conformance` is not a required check on kindkit.** Adding it is a ruleset change for the owner. **After merging `main` with #19** (merge commit `3d6f94b`): `just check` all checks passed; `just test` 202 passed; `just mutants` (throwaway detached worktree) 137 mutations, 137 caught, 0 survived, 0 broken. CI green. ## Round 2 of review (head `9951ee2`, with `main` merged in, which brought #20, #22 and #23) - **M1, a relative `root` was accepted.** Before this change, `_parse_report` checked only that `root` is a string, and `conform.py` resolved it against its own working directory. The reviewer's runner did `cd good && runner kvcopy` and reported `"root": "kvcopy"`. That passed against a **broken** `./kvcopy`. New test `test_a_report_whose_root_is_relative_is_no_verdict` rebuilds that shape and was red on `3d6f94b`: `assert 0 == 2`. `_parse_report` now refuses a non-absolute `root` with `'root' 'kvcopy' is not an absolute path`; unit test `test_a_report_whose_root_is_relative_is_refused`. The README contract now says the root is **absolute**. New gate mutation "a relative root is accepted, to be resolved by whoever reads it"; both listed tests fail on their own under it. - **L1, the every-case-ran check was masked.** The subtree test was also refused by the root check, so emptying the list of unrun cases survived the suite. New test `test_a_report_from_the_right_root_that_ran_only_some_cases_is_no_verdict` hands `conform.py` a report naming the same root with 2 of the 6 cases. It is now listed by "a suite that ran a different set of cases than the tree passes", and by a new mutation "a case of the tree that did not run goes unnoticed" (`sorted(tree - ran)` → `[]`). Run on its own, each mutation fails that test. - **L2, which file was mutated.** This is documented as a limit: a gate whose only kill breaks the adapter or the second implementation still passes. The gate report now records `source`, the file mutated, and `check_gate.py` prints it. On CI: `the gate killed 3 mutant(s) of '/home/runner/work/kindkit/kindkit/tests/kvkind.py' over all 6 case(s)`. - **The gate report's keys** are now documented in the README: `kindkit_gate_report`, `ok`, `killed`, `survived`, `equivalent`, `stale`, `bogus`, `broken`, `ran`, `root` and `source`. Gates at `9951ee2`: ``` $ just check All checks passed! $ just test 220 passed $ just mutants # throwaway detached worktree 148 mutations, 148 caught, 0 survived, 0 broken ``` CI on the PR: `check` and `kind / conformance` pass; mergeable state `clean`. Refs #4. Every bullet is met on kindkit's side. "rowspec's existing workflows can call it and produce the same verdicts" is shown above against rowspec, but not yet by rowspec's own workflows. Refs #3. "Validate the schema against all 410 cases in CI" becomes true when rowspec calls this workflow. Until then it holds for kindkit's toy tree in CI, and for rowspec's 410 cases only locally (above).
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
What
tools/mutation_gate.py,_splice: after a byte splice into a.jsontarget, the result is passed tojson.loads. If it does not parse,_spliceraisesMutantError("mutated JSON does not parse: ...").applyalready hands aMutantErrorback tomain, which scores the mutation BROKEN and fails the run. The docstring now says this check exists, and that any other extension is spliced unchecked.tests/test_mutation_gate.py:test_a_byte_splice_that_breaks_the_json_is_refused_not_scoredsplices"required": ["kind",,into the schema and requiresMutantError.test_every_json_mutation_in_the_gate_still_parses, parametrised over every.jsonmutation inMUTATIONS, covers the other direction: the new refusal must not reject any real mutation.One new mutation, "a byte splice that breaks the JSON is scored instead of refused", disables the check.
Why
A token splice is compiled; a byte splice was parsed by nothing. A mutation that only broke the file's syntax made pytest fall over on a parse error, and the gate scored that as a kill of a change nothing had detected.
Testing
Red first, on
main(bf716e3). I put the issue's mutation through the real gate (MUTATIONSreplaced by that one entry,main()called) in a throwaway worktree, because the gate writes in place:The new unit test on
main:Same one-entry gate run, on this branch:
The valid direction: all four real schema mutations still splice to parseable JSON and are still caught.
The count is
main's 99 plus the one new mutation, and the 99 are all still caught. The issue's "73" was the count when it was filed. I also applied the new mutation with the gate's hash-checkedapplyand ran its listed test on its own: exit 1, and the file restored to its original hash.The issue's last bullet asks for
_splice's docstring to be corrected. That correction is already onmain: since #18 the docstring says a layout-sensitive pattern is "loud rather than silent" and scored BROKEN. This PR only adds the paragraph about checking the splice.Interaction with #19: #19 adds a byte-path mutation of
.pre-commit-config.yaml. Under this change, YAML is spliced unchecked, as the docstring now says, because the stdlib has no YAML parser. That mutation keeps the file valid YAML and its test reads the file with a regex, so it is unaffected.Follow-up (review): the parse check could skip itself
test_every_json_mutation_in_the_gate_still_parsesis parametrised overMUTATIONSfiltered to.jsonpaths. With an empty filter, pytest skips the test and exits 0. Shown in a throwaway worktree, with the filter changed to.nothing:JSON_MUTATIONS. A plain, non-parametrisedtest_there_are_json_mutations_to_checkrequires at least 4 entries, the schema mutations the gate carries today. With the same emptied filter it fails:FAILED ...test_there_are_json_mutations_to_check,1 failed, 3 passed, 1 skipped.apply, hash-checked, file restored).With
main(#21) merged in, at5ee2d6f:After merging
mainwith #19 (merge commit3404ef9):just checkall checks passed;just test159 passed;just mutants(throwaway detached worktree) 110 mutations, 110 caught, 0 survived, 0 broken. CI green.Closes #10