Repository navigation
fix(mutation): date each scratch write with its own mtime second - #23
Merged
Merged
Conversation
A `.pyc` is validated on the source's (mtime second, size). The purge clears the two caches the gate can see, but a probe whose interpreter caches under a prefix of its own is invisible to it. Fast probes then land several same-size writes in one wall-clock second, and each is served the previous write's bytecode. Driven through `gate` with no mtime pinned anywhere, 0 or 1 of 10 detectable mutants were killed. `gate` now stamps every scratch write with a distinct whole-second mtime: a random second between 2000 and 2020, plus one per write. A `.pyc` from any earlier write in the run then fails validation wherever it lives. The purge stays for probes that reset the mtime themselves. The README and docstrings say what the probe still owns. Two gate mutations drop the stamp and repeat it; the new test catches both.
FAT and exFAT store mtimes in 2-second steps, where stamps one second apart can be stored equal and collide again. Stamps now step by two. The start was drawn with `random.randrange`, so a caller that seeded `random` got the same start on every run, and the gate consumed the caller's random sequence. It is now drawn with `random.SystemRandom`. The README and docstrings claimed a stale `.pyc` fails validation wherever it lives. That holds only for bytecode compiled from the scratch file itself: a probe that copies or rewrites the source before importing it gives the copy a wall-clock mtime. The README now lists what the probe still owns: setting the mtime, copying the source, a filesystem coarser than 2 seconds, unchecked hash-based `.pyc` files, and, across runs, a later run's first write landing on the stamp an earlier run left. Three tests pin the step, the untouched generator and the unseeded start; two gate mutations break the step and the generator.
This was referenced Oct 7, 2026
CameronBrooks11
added a commit
that referenced
this pull request
Oct 7, 2026
…#24) ## What **`.github/workflows/kind.yml`** is a reusable workflow (`on: workflow_call`). Its inputs, all required: `fixture-root`, `suite`, `second-implementation` and `mutants`. It runs as **one job named `conformance`**, and its four gates are steps with `if: always()`. Each gate runs even when an earlier gate is red. The setup steps before them are not like that: if one fails, the gates still run and fail too, for want of kindkit's tools. | step | runs | exit 0 only if | |---|---|---| | the case tree follows the convention | `tools/validate_case_tree.py <root>` | every manifest is valid; exit 2 on an empty or missing tree | | the reference implementation passes every case in the tree | `tools/conform.py <root> <suite>` | the runner's report names `root` as the fixture root, it ran exactly the case directories under it, and none failed | | the second implementation passes every case in the tree | `tools/conform.py --differs-from <suite> <root> <second-implementation>` | the same as above, and the command is not the suite's command with only whitespace changed | | the suite must be able to fail | `tools/check_gate.py <root> <mutants>` | `kindkit.gate`'s own report exists, killed ≥1 mutant, has no SURVIVED, STALE, BOGUS or BROKEN mutant, and its baseline ran exactly the tree under that root | **How a verdict is tied to the tree the workflow names:** - `kindkit.cli` writes the absolute `root` it read into its report. A report whose `root` is not absolute is refused as malformed (round 2). - `kindkit.cli` takes the report path from `KINDKIT_REPORT_JSON` when `--report-json` is not given, and refuses a file already at that path. - `Verdict` carries the `root` that `probe_command` reads from the report. - `kindkit.gate` writes its verdicts, the baseline's case ids and its root to `KINDKIT_GATE_REPORT` when that is set. It creates the file exclusively, so a second gate in the same command raises. `conform.py` and `check_gate.py` run the caller's command **as written**, with those variables set. Nothing is appended to the command, so a trailing newline (`suite: |`), a `cd`, or a trailing `# comment` cannot separate the runner from its report path. **Why one job.** A ruleset matches a required check by name. A called job reports as `<calling job's name, else its id> / conformance`, so a consumer has exactly one name to keep stable. **How it finds its own tools.** Inside a called workflow, `github.job_workflow_sha` is empty (measured). The workflow reads its own commit from the run's `referenced_workflows` and fetches kindkit's `tools/` there. That read needs `actions: read`, which the workflow requests, so **the calling job must grant `contents: read` and `actions: read`**. kindkit's `check.yml` `kind` job does. **Exercised in kindkit's CI** by the `kind` job, which calls the workflow on the toy kv tree with: - `tests/run_kv.py` as the runner; - `kvkind.Alt` as the second implementation, which shares no source fragment with `Good`; - `tests/kv_mutants.py` as the gate: 3 mutants of `Good`, run through `kindkit.gate` and `probe_command`. The README has a new section, "Using the workflow". It covers what each step requires, the check name, the permissions, and **how to adopt the workflow where `conformance` is already a required check** (§3.2): which steps stay in the adopter's own job, when to add the new context and when to swap it in, and to edit the ruleset just before merging, not in the change. ## Testing Local, on the current head, with `main` (#21) merged in: ``` $ just check All checks passed! $ just test 201 passed $ just cases 0 invalid case(s) across 6 case(s) in 1 tree(s) $ just mutants # throwaway detached worktree of 3e426b7 136 mutations, 136 caught, 0 survived, 0 broken ``` There are 30 mutations on `tools/conform.py`, `tools/check_gate.py`, and the new `root`, environment-path and gate-report code in `kindkit/`. For each one, I applied it with the gate's hash-checked `apply` and ran every test it lists on its own, in a separate pytest process. Every listed test exited 1, and each file restored to its original hash. ### Review findings, each shown red first - **F1, the root was never checked.** These tests were red on the previous head: - `test_a_runner_that_falls_back_to_its_default_tree_is_no_verdict`: the reviewer's case. `conform.py <broken copy> "…run_kv.py … Good"` returned `0`. - `test_a_runner_that_read_a_same_named_tree_elsewhere_is_no_verdict`: also returned `0`. - A control, `test_the_broken_copy_is_a_failure_when_the_runner_reads_it`, shows the copy really is broken: a run that reads it exits 1. All of these now exit 2 as expected. A report with no `root` is also exit 2. - **Commands with a trailing newline or comment** were red on the previous head: `NO VERDICT: the suite exited 127` and `NO VERDICT: the suite wrote no --report-json file`. Both are now `0`. - **F2, the mutation step accepted any exit 0.** `check_gate.py` has 21 tests. `true`, `""`, blank, and a command that exits 0 without a gate are all exit 2. A report with a hole or with nothing killed is exit 1. A report with no root, another root, fewer cases or an unknown format is exit 2. The real kv gate over a same-named copy is exit 2. - **F3, a second implementation run by the suite's command.** `conform(..., differs_from=command)` raised `TypeError` before this change; now it is exit 2, also for a respaced copy of the command. - **The "wrote no report" mutation.** It is reworked so that its mutant returns exit 0, a false pass the test catches by assertion, not a `FileNotFoundError`. - **The tree check was hidden behind the root check.** The full gate reported `an empty or missing tree is left to the suite to judge` as **SURVIVED** once the root check existed: the tests were being refused by the root check first. The tests now hand a report naming the same root, so only the tree check can refuse it. `check_gate.py` got the same test shape and its own mutation. ### On CI **Green**, PR run on `3e426b7`: `kind / conformance` passed. kindkit was fetched at the PR's commit, both suites ran all 6 cases under `tests/fixtures/kv`, and the line `the gate killed 3 mutant(s) over all 6 case(s)` appears in the log. **Red, earlier head:** throwaway branches `tmp/kind-red-broken-case` (run 37685697020) and `tmp/kind-red-inputs` (run 37685723625). A broken kv case, a tree with no cases, a missing tree, a suite run on a subtree, a failing second implementation, a suite with no report, blank `mutants`, and a failing gate each went red at exactly the expected steps. **This head:** throwaway branch `tmp/kind-red-review`, runs 37690440366 and 37690440235. | job | result | reason in the log | |---|---|---| | runner reads its default tree (`fixture-root: tests/fixtures/kv-copy`, one case broken) | red: suite, second, mutants | `the suite read '…/tests/fixtures/kv', not '…/tests/fixtures/kv-copy'` (and the same for the gate) | | `mutants: "true"` | red: mutants | `the command exited 0 and no gate wrote $KINDKIT_GATE_REPORT` | | second implementation = suite command, respaced | red: second | `the second implementation's command is the suite's command` | | commands as YAML blocks and with a trailing `# comment`; caller job with `name: a named caller` | **green**, reported as `a named caller / conformance` | all 6 cases, 3 killed | | caller granting only `contents: read` (separate workflow file) | `startup_failure`: the run never starts | GitHub reports a workflow-file issue; the API does not expose the message text | All throwaway branches are deleted (`git ls-remote --heads origin | grep -c tmp/` gives `0`). ### Against rowspec, without changing rowspec In a scratch clone of rowspec `main` (`2a89617`), I installed this branch's kindkit into the clone's venv and ran this branch's tools with rowspec's own commands: ``` conform.py conformance/cases "cd conformance && …/python run_cases.py" 0 failure(s) across all 410 case(s) exit 0 conform.py --differs-from <that> conformance/cases "… run_cases.py rowspec_alt.table" 0 failure(s) across all 410 case(s) exit 0 check_gate.py conformance/cases "cd conformance && …/python mutants.py" the gate killed 74 mutant(s) over all 410 case(s) exit 0 ``` On the earlier head, a broken rowspec fixture gave the same red verdict through `conform.py` as rowspec's runner run directly: `1 failure(s)` in each. rowspec's current pin, `a26f8f8`, has neither the `root` key nor `KINDKIT_REPORT_JSON`. `git show a26f8f8:kindkit/runner.py | grep -c '"root"'` gives `0`, and the same for `cli.py`. So adopting the workflow in rowspec includes bumping that pin. ## Limits, stated in the README - Different command text that reaches the same second implementation cannot be told apart from outside. - The gate has to probe through `probe_command`, or set `Verdict(root=...)` itself, and one command may run only one `gate()` call. - The `referenced_workflows` read has not been tried on a private repository. - A matrix caller's check-name suffix is not measured here. ## Not done here, deliberately - **rowspec does not call this yet.** Adopting it is a separate change in rowspec: bump the kindkit pin, then follow the README's §3.2 steps. - **`kind / conformance` is not a required check on kindkit.** Adding it is a ruleset change for the owner. **After merging `main` with #19** (merge commit `3d6f94b`): `just check` all checks passed; `just test` 202 passed; `just mutants` (throwaway detached worktree) 137 mutations, 137 caught, 0 survived, 0 broken. CI green. ## Round 2 of review (head `9951ee2`, with `main` merged in, which brought #20, #22 and #23) - **M1, a relative `root` was accepted.** Before this change, `_parse_report` checked only that `root` is a string, and `conform.py` resolved it against its own working directory. The reviewer's runner did `cd good && runner kvcopy` and reported `"root": "kvcopy"`. That passed against a **broken** `./kvcopy`. New test `test_a_report_whose_root_is_relative_is_no_verdict` rebuilds that shape and was red on `3d6f94b`: `assert 0 == 2`. `_parse_report` now refuses a non-absolute `root` with `'root' 'kvcopy' is not an absolute path`; unit test `test_a_report_whose_root_is_relative_is_refused`. The README contract now says the root is **absolute**. New gate mutation "a relative root is accepted, to be resolved by whoever reads it"; both listed tests fail on their own under it. - **L1, the every-case-ran check was masked.** The subtree test was also refused by the root check, so emptying the list of unrun cases survived the suite. New test `test_a_report_from_the_right_root_that_ran_only_some_cases_is_no_verdict` hands `conform.py` a report naming the same root with 2 of the 6 cases. It is now listed by "a suite that ran a different set of cases than the tree passes", and by a new mutation "a case of the tree that did not run goes unnoticed" (`sorted(tree - ran)` → `[]`). Run on its own, each mutation fails that test. - **L2, which file was mutated.** This is documented as a limit: a gate whose only kill breaks the adapter or the second implementation still passes. The gate report now records `source`, the file mutated, and `check_gate.py` prints it. On CI: `the gate killed 3 mutant(s) of '/home/runner/work/kindkit/kindkit/tests/kvkind.py' over all 6 case(s)`. - **The gate report's keys** are now documented in the README: `kindkit_gate_report`, `ok`, `killed`, `survived`, `equivalent`, `stale`, `bogus`, `broken`, `ran`, `root` and `source`. Gates at `9951ee2`: ``` $ just check All checks passed! $ just test 220 passed $ just mutants # throwaway detached worktree 148 mutations, 148 caught, 0 survived, 0 broken ``` CI on the PR: `check` and `kind / conformance` pass; mergeable state `clean`. Refs #4. Every bullet is met on kindkit's side. "rowspec's existing workflows can call it and produce the same verdicts" is shown above against rowspec, but not yet by rowspec's own workflows. Refs #3. "Validate the schema against all 410 cases in CI" becomes true when rowspec calls this workflow. Until then it holds for kindkit's toy tree in CI, and for rowspec's 410 cases only locally (above).
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
What
kindkit/mutation.py:gategives every write of the scratch file a distinct whole-second mtime, applied withos.utimein_proberight after the write. A run starts at a random second between 2000-01-01 and 2020-01-01 UTC (_STAMP_RANGE, drawn withrandom.SystemRandom) and adds two per write (_STAMP_STEP). (Step and generator changed in the follow-up below.) The purge from #14 is unchanged and still runs.A
.pycrecords the (mtime second, size) of the source it was compiled from. So a.pyccompiled from an earlier write of the scratch file fails validation wherever it is cached, including under aPYTHONPYCACHEPREFIXthat the probe chose for itself and the purge cannot see. A copy of the source is a different file with its own mtime; see the follow-up.tests/test_mutation.py: newtest_a_mutant_is_never_served_bytecode_the_purge_cannot_see. It drives the realgatewith a probe that loads the scratch file in a child process with its own fixedPYTHONPYCACHEPREFIX. Nothing in the probe sets an mtime. There are 20 same-size mutants, alternating between detected (b..) and claimed equivalent (a..), so any write served the previous write's bytecode gets the other class's verdict. Two preconditions are asserted so the test cannot pass vacuously: bytecode exists in the probe's prefix, and the run was fast enough for writes to share a second.tools/mutation_gate.py: two new mutations. One drops theos.utimecall; the other swapsitertools.countforitertools.repeat, giving every write the same stamp. One existing pattern ("the gate mutates the implementation in place") is updated for_probe's new argument. The gate reported it STALE until I updated it.README.md("What a probe has to get right") and thegate/_purge_bytecodedocstrings: what the kit now does, and what the probe still owns.Why
This is the candidate fix recorded in #11's comments. A fixed private prefix is the case #14 left open, and this PR shows that case produces wrong verdicts without the probe pinning anything.
Testing
Red first, on
main(bf716e3), with only the test added. Three consecutive runs:So on
main, the gate killed 0 or 1 of 10 mutants that the suite detects. The same shape through a standalone script onmain(10 pairs) gave20 mutants in 0.31s: ok=False killed=0 survived=10 equiv=10 bogus=0.After the fix:
For each new mutation, and for the updated one, I applied it with the gate's hash-checked
applyand ran its listed test on its own. Each exited 1, and each file restored to its original hash.Follow-up (review)
Overclaim corrected. The README and docstrings said a stale
.pycfails validation "wherever it was cached ... or copied elsewhere". That holds only for bytecode compiled from the scratch file itself. A probe that copies or rewrites the source before importing it gives the copy a wall-clock mtime, so the stamp does nothing for it. The reviewer reproduced this with a copy to a fixed path plus a child with its own fixed prefix. "Bytecode or source copied anywhere else" is back in the README's list of what the probe owns.2-second filesystems. FAT and exFAT store mtimes in 2-second steps, so stamps
2kand2k+1can be stored equal. The stamps now step by 2 (_STAMP_STEP = 2). New testtest_consecutive_scratch_writes_are_two_seconds_apartrecords the scratch file's mtime at each probe. It was red before the change:assert [1.0, 1.0, 1.0] == [2, 2, 2]. Gate mutation:_STAMP_STEP = 1.Figure corrected. The range is
1_577_836_800 - 946_684_800= 631,152,000 seconds (python3 -c "print(1_577_836_800 - 946_684_800)"). The previous body said 630,720,000, which was wrong.random.SystemRandom()for the start. Before the change:test_a_callers_random_generator_is_not_drawn_fromfailed with "the gate drew from the caller's random generator";test_a_callers_random_seed_does_not_fix_the_stampsfailed withassert 1419917831.0 != 1419917831.0, so two runs seeded the same way started on the same stamp.Gate mutation:
SystemRandom().randrangereverted torandom.randrange. Both listed tests exit 1 under it.What the probe still owns (now listed in the README)
.pycfiles;.pyc. So a later run can inherit only the one.pycan earlier run left for that path, and only if the later run's first write lands on that.pyc's exact stamp at the exact size. The start is drawn uniformly from 631,152,000 seconds; that odds figure is computed from the range, not measured.#11's first acceptance bullet asks for a purge that removes nothing to be reported. This change makes such a purge harmless instead, which is why this PR says
Refs.Gates after the follow-up, with
main(#21) merged inFor each new mutation, and for the updated
repeatmutation, I applied it with the gate's hash-checkedapplyand ran each listed test on its own. All exited 1, and each file restored to its original hash.After merging
mainwith #19 (merge commit21ebf67):just checkall checks passed;just test157 passed;just mutants(throwaway detached worktree) 112 mutations, 112 caught, 0 survived, 0 broken. CI green.Refs #11