Skip to content

fix(mutation): date each scratch write with its own mtime second - #23

Merged
CameronBrooks11 merged 5 commits into
mainfrom
fix/stamp-each-scratch-write
Oct 7, 2026
Merged

CameronBrooks11 merged 5 commits into
mainfrom
fix/stamp-each-scratch-write

Conversation

@CameronBrooks11

@CameronBrooks11 CameronBrooks11 commented Oct 7, 2026 •

Copy link
Copy Markdown
Member

What

kindkit/mutation.py: gate gives every write of the scratch file a distinct whole-second mtime, applied with os.utime in _probe right after the write. A run starts at a random second between 2000-01-01 and 2020-01-01 UTC (_STAMP_RANGE, drawn with random.SystemRandom) and adds two per write (_STAMP_STEP). (Step and generator changed in the follow-up below.) The purge from #14 is unchanged and still runs.

A .pyc records the (mtime second, size) of the source it was compiled from. So a .pyc compiled from an earlier write of the scratch file fails validation wherever it is cached, including under a PYTHONPYCACHEPREFIX that the probe chose for itself and the purge cannot see. A copy of the source is a different file with its own mtime; see the follow-up.

  • tests/test_mutation.py: new test_a_mutant_is_never_served_bytecode_the_purge_cannot_see. It drives the real gate with a probe that loads the scratch file in a child process with its own fixed PYTHONPYCACHEPREFIX. Nothing in the probe sets an mtime. There are 20 same-size mutants, alternating between detected (b..) and claimed equivalent (a..), so any write served the previous write's bytecode gets the other class's verdict. Two preconditions are asserted so the test cannot pass vacuously: bytecode exists in the probe's prefix, and the run was fast enough for writes to share a second.
  • tools/mutation_gate.py: two new mutations. One drops the os.utime call; the other swaps itertools.count for itertools.repeat, giving every write the same stamp. One existing pattern ("the gate mutates the implementation in place") is updated for _probe's new argument. The gate reported it STALE until I updated it.
  • README.md ("What a probe has to get right") and the gate / _purge_bytecode docstrings: what the kit now does, and what the probe still owns.

Why

This is the candidate fix recorded in #11's comments. A fixed private prefix is the case #14 left open, and this PR shows that case produces wrong verdicts without the probe pinning anything.

Testing

Red first, on main (bf716e3), with only the test added. Three consecutive runs:

$ uv run pytest -q tests/test_mutation.py -k bytecode_the_purge
E       AssertionError: assert () == ('b00', 'b01'...', 'b05', ...)
E       AssertionError: assert ('b09',) == ('b00', 'b01'...', 'b05', ...)
E       AssertionError: assert ('b09',) == ('b00', 'b01'...', 'b05', ...)

So on main, the gate killed 0 or 1 of 10 mutants that the suite detects. The same shape through a standalone script on main (10 pairs) gave 20 mutants in 0.31s: ok=False killed=0 survived=10 equiv=10 bogus=0.

After the fix:

$ for i in $(seq 20); do pytest -q tests/test_mutation.py -k bytecode_the_purge; done
20 x "1 passed"
$ just check
All checks passed!
$ just test
146 passed
$ just mutants        # in a throwaway detached worktree of this commit
  caught    the previous probe's bytecode is served to the next mutant
  caught    bytecode cached beside the scratch file is no longer purged
  caught    bytecode cached under the interpreter's prefix is no longer purged
  caught    a scratch write keeps the wall-clock mtime, so writes share a second
  caught    every scratch write in a run gets the same mtime
101 mutations, 101 caught, 0 survived, 0 broken

For each new mutation, and for the updated one, I applied it with the gate's hash-checked apply and ran its listed test on its own. Each exited 1, and each file restored to its original hash.

Follow-up (review)

  1. Overclaim corrected. The README and docstrings said a stale .pyc fails validation "wherever it was cached ... or copied elsewhere". That holds only for bytecode compiled from the scratch file itself. A probe that copies or rewrites the source before importing it gives the copy a wall-clock mtime, so the stamp does nothing for it. The reviewer reproduced this with a copy to a fixed path plus a child with its own fixed prefix. "Bytecode or source copied anywhere else" is back in the README's list of what the probe owns.

  2. 2-second filesystems. FAT and exFAT store mtimes in 2-second steps, so stamps 2k and 2k+1 can be stored equal. The stamps now step by 2 (_STAMP_STEP = 2). New test test_consecutive_scratch_writes_are_two_seconds_apart records the scratch file's mtime at each probe. It was red before the change: assert [1.0, 1.0, 1.0] == [2, 2, 2]. Gate mutation: _STAMP_STEP = 1.

  3. Figure corrected. The range is 1_577_836_800 - 946_684_800 = 631,152,000 seconds (python3 -c "print(1_577_836_800 - 946_684_800)"). The previous body said 630,720,000, which was wrong.

  4. random.SystemRandom() for the start. Before the change:

    • test_a_callers_random_generator_is_not_drawn_from failed with "the gate drew from the caller's random generator";
    • test_a_callers_random_seed_does_not_fix_the_stamps failed with assert 1419917831.0 != 1419917831.0, so two runs seeded the same way started on the same stamp.

    Gate mutation: SystemRandom().randrange reverted to random.randrange. Both listed tests exit 1 under it.

What the probe still owns (now listed in the README)

  • a probe that sets the scratch file's mtime;
  • a probe that copies or rewrites the source before importing it, and bytecode copied anywhere else;
  • a filesystem whose mtimes are coarser than 2 seconds;
  • an interpreter told to trust unchecked hash-based .pyc files;
  • across runs: within a run each import recompiles over the previous .pyc. So a later run can inherit only the one .pyc an earlier run left for that path, and only if the later run's first write lands on that .pyc's exact stamp at the exact size. The start is drawn uniformly from 631,152,000 seconds; that odds figure is computed from the range, not measured.

#11's first acceptance bullet asks for a purge that removes nothing to be reported. This change makes such a purge harmless instead, which is why this PR says Refs.

Gates after the follow-up, with main (#21) merged in

$ just check
All checks passed!
$ just test
156 passed
$ just mutants        # throwaway detached worktree
  caught    a scratch write keeps the wall-clock mtime, so writes share a second
  caught    every scratch write in a run gets the same mtime
  caught    scratch writes are stamped one second apart, which FAT stores as equal
  caught    the first stamp comes from the caller's seedable random generator
111 mutations, 111 caught, 0 survived, 0 broken

For each new mutation, and for the updated repeat mutation, I applied it with the gate's hash-checked apply and ran each listed test on its own. All exited 1, and each file restored to its original hash.

After merging main with #19 (merge commit 21ebf67): just check all checks passed; just test 157 passed; just mutants (throwaway detached worktree) 112 mutations, 112 caught, 0 survived, 0 broken. CI green.

Refs #11

A `.pyc` is validated on the source's (mtime second, size). The purge
clears the two caches the gate can see, but a probe whose interpreter
caches under a prefix of its own is invisible to it. Fast probes then
land several same-size writes in one wall-clock second, and each is
served the previous write's bytecode. Driven through `gate` with no
mtime pinned anywhere, 0 or 1 of 10 detectable mutants were killed.

`gate` now stamps every scratch write with a distinct whole-second
mtime: a random second between 2000 and 2020, plus one per write. A
`.pyc` from any earlier write in the run then fails validation wherever
it lives. The purge stays for probes that reset the mtime themselves.

The README and docstrings say what the probe still owns. Two gate
mutations drop the stamp and repeat it; the new test catches both.
FAT and exFAT store mtimes in 2-second steps, where stamps one second
apart can be stored equal and collide again. Stamps now step by two.

The start was drawn with `random.randrange`, so a caller that seeded
`random` got the same start on every run, and the gate consumed the
caller's random sequence. It is now drawn with `random.SystemRandom`.

The README and docstrings claimed a stale `.pyc` fails validation
wherever it lives. That holds only for bytecode compiled from the
scratch file itself: a probe that copies or rewrites the source before
importing it gives the copy a wall-clock mtime. The README now lists
what the probe still owns: setting the mtime, copying the source, a
filesystem coarser than 2 seconds, unchecked hash-based `.pyc` files,
and, across runs, a later run's first write landing on the stamp an
earlier run left.

Three tests pin the step, the untouched generator and the unseeded
start; two gate mutations break the step and the generator.
@CameronBrooks11
CameronBrooks11 merged commit e6d4efc into main Oct 7, 2026
5 checks passed
@CameronBrooks11
CameronBrooks11 deleted the fix/stamp-each-scratch-write branch October 7, 2026 21:53
CameronBrooks11 added a commit that referenced this pull request Oct 7, 2026
…#24)

## What

**`.github/workflows/kind.yml`** is a reusable workflow (`on:
workflow_call`). Its inputs, all required: `fixture-root`, `suite`,
`second-implementation` and `mutants`. It runs as **one job named
`conformance`**, and its four gates are steps with `if: always()`. Each
gate runs even when an earlier gate is red. The setup steps before them
are not like that: if one fails, the gates still run and fail too, for
want of kindkit's tools.

| step | runs | exit 0 only if |
|---|---|---|
| the case tree follows the convention | `tools/validate_case_tree.py
<root>` | every manifest is valid; exit 2 on an empty or missing tree |
| the reference implementation passes every case in the tree |
`tools/conform.py <root> <suite>` | the runner's report names `root` as
the fixture root, it ran exactly the case directories under it, and none
failed |
| the second implementation passes every case in the tree |
`tools/conform.py --differs-from <suite> <root> <second-implementation>`
| the same as above, and the command is not the suite's command with
only whitespace changed |
| the suite must be able to fail | `tools/check_gate.py <root>
<mutants>` | `kindkit.gate`'s own report exists, killed ≥1 mutant, has
no SURVIVED, STALE, BOGUS or BROKEN mutant, and its baseline ran exactly
the tree under that root |

**How a verdict is tied to the tree the workflow names:**

- `kindkit.cli` writes the absolute `root` it read into its report. A
report whose `root` is not absolute is refused as malformed (round 2).
- `kindkit.cli` takes the report path from `KINDKIT_REPORT_JSON` when
`--report-json` is not given, and refuses a file already at that path.
- `Verdict` carries the `root` that `probe_command` reads from the
report.
- `kindkit.gate` writes its verdicts, the baseline's case ids and its
root to `KINDKIT_GATE_REPORT` when that is set. It creates the file
exclusively, so a second gate in the same command raises.

`conform.py` and `check_gate.py` run the caller's command **as
written**, with those variables set. Nothing is appended to the command,
so a trailing newline (`suite: |`), a `cd`, or a trailing `# comment`
cannot separate the runner from its report path.

**Why one job.** A ruleset matches a required check by name. A called
job reports as `<calling job's name, else its id> / conformance`, so a
consumer has exactly one name to keep stable.

**How it finds its own tools.** Inside a called workflow,
`github.job_workflow_sha` is empty (measured). The workflow reads its
own commit from the run's `referenced_workflows` and fetches kindkit's
`tools/` there. That read needs `actions: read`, which the workflow
requests, so **the calling job must grant `contents: read` and `actions:
read`**. kindkit's `check.yml` `kind` job does.

**Exercised in kindkit's CI** by the `kind` job, which calls the
workflow on the toy kv tree with:

- `tests/run_kv.py` as the runner;
- `kvkind.Alt` as the second implementation, which shares no source
fragment with `Good`;
- `tests/kv_mutants.py` as the gate: 3 mutants of `Good`, run through
`kindkit.gate` and `probe_command`.

The README has a new section, "Using the workflow". It covers what each
step requires, the check name, the permissions, and **how to adopt the
workflow where `conformance` is already a required check** (§3.2): which
steps stay in the adopter's own job, when to add the new context and
when to swap it in, and to edit the ruleset just before merging, not in
the change.

## Testing

Local, on the current head, with `main` (#21) merged in:

```
$ just check
All checks passed!
$ just test
201 passed
$ just cases
0 invalid case(s) across 6 case(s) in 1 tree(s)
$ just mutants        # throwaway detached worktree of 3e426b7
136 mutations, 136 caught, 0 survived, 0 broken
```

There are 30 mutations on `tools/conform.py`, `tools/check_gate.py`, and
the new `root`, environment-path and gate-report code in `kindkit/`. For
each one, I applied it with the gate's hash-checked `apply` and ran
every test it lists on its own, in a separate pytest process. Every
listed test exited 1, and each file restored to its original hash.

### Review findings, each shown red first

- **F1, the root was never checked.** These tests were red on the
previous head:
- `test_a_runner_that_falls_back_to_its_default_tree_is_no_verdict`: the
reviewer's case. `conform.py <broken copy> "…run_kv.py … Good"` returned
`0`.
- `test_a_runner_that_read_a_same_named_tree_elsewhere_is_no_verdict`:
also returned `0`.
- A control,
`test_the_broken_copy_is_a_failure_when_the_runner_reads_it`, shows the
copy really is broken: a run that reads it exits 1.

All of these now exit 2 as expected. A report with no `root` is also
exit 2.
- **Commands with a trailing newline or comment** were red on the
previous head: `NO VERDICT: the suite exited 127` and `NO VERDICT: the
suite wrote no --report-json file`. Both are now `0`.
- **F2, the mutation step accepted any exit 0.** `check_gate.py` has 21
tests. `true`, `""`, blank, and a command that exits 0 without a gate
are all exit 2. A report with a hole or with nothing killed is exit 1. A
report with no root, another root, fewer cases or an unknown format is
exit 2. The real kv gate over a same-named copy is exit 2.
- **F3, a second implementation run by the suite's command.**
`conform(..., differs_from=command)` raised `TypeError` before this
change; now it is exit 2, also for a respaced copy of the command.
- **The "wrote no report" mutation.** It is reworked so that its mutant
returns exit 0, a false pass the test catches by assertion, not a
`FileNotFoundError`.
- **The tree check was hidden behind the root check.** The full gate
reported `an empty or missing tree is left to the suite to judge` as
**SURVIVED** once the root check existed: the tests were being refused
by the root check first. The tests now hand a report naming the same
root, so only the tree check can refuse it. `check_gate.py` got the same
test shape and its own mutation.

### On CI

**Green**, PR run on `3e426b7`: `kind / conformance` passed. kindkit was
fetched at the PR's commit, both suites ran all 6 cases under
`tests/fixtures/kv`, and the line `the gate killed 3 mutant(s) over all
6 case(s)` appears in the log.

**Red, earlier head:** throwaway branches `tmp/kind-red-broken-case`
(run 37685697020) and `tmp/kind-red-inputs` (run 37685723625). A broken
kv case, a tree with no cases, a missing tree, a suite run on a subtree,
a failing second implementation, a suite with no report, blank
`mutants`, and a failing gate each went red at exactly the expected
steps.

**This head:** throwaway branch `tmp/kind-red-review`, runs 37690440366
and 37690440235.

| job | result | reason in the log |
|---|---|---|
| runner reads its default tree (`fixture-root: tests/fixtures/kv-copy`,
one case broken) | red: suite, second, mutants | `the suite read
'…/tests/fixtures/kv', not '…/tests/fixtures/kv-copy'` (and the same for
the gate) |
| `mutants: "true"` | red: mutants | `the command exited 0 and no gate
wrote $KINDKIT_GATE_REPORT` |
| second implementation = suite command, respaced | red: second | `the
second implementation's command is the suite's command` |
| commands as YAML blocks and with a trailing `# comment`; caller job
with `name: a named caller` | **green**, reported as `a named caller /
conformance` | all 6 cases, 3 killed |
| caller granting only `contents: read` (separate workflow file) |
`startup_failure`: the run never starts | GitHub reports a workflow-file
issue; the API does not expose the message text |

All throwaway branches are deleted (`git ls-remote --heads origin | grep
-c tmp/` gives `0`).

### Against rowspec, without changing rowspec

In a scratch clone of rowspec `main` (`2a89617`), I installed this
branch's kindkit into the clone's venv and ran this branch's tools with
rowspec's own commands:

```
conform.py conformance/cases "cd conformance && …/python run_cases.py"                    0 failure(s) across all 410 case(s)   exit 0
conform.py --differs-from <that> conformance/cases "… run_cases.py rowspec_alt.table"     0 failure(s) across all 410 case(s)   exit 0
check_gate.py conformance/cases "cd conformance && …/python mutants.py"                    the gate killed 74 mutant(s) over all 410 case(s)   exit 0
```

On the earlier head, a broken rowspec fixture gave the same red verdict
through `conform.py` as rowspec's runner run directly: `1 failure(s)` in
each.

rowspec's current pin, `a26f8f8`, has neither the `root` key nor
`KINDKIT_REPORT_JSON`. `git show a26f8f8:kindkit/runner.py | grep -c
'"root"'` gives `0`, and the same for `cli.py`. So adopting the workflow
in rowspec includes bumping that pin.

## Limits, stated in the README

- Different command text that reaches the same second implementation
cannot be told apart from outside.
- The gate has to probe through `probe_command`, or set
`Verdict(root=...)` itself, and one command may run only one `gate()`
call.
- The `referenced_workflows` read has not been tried on a private
repository.
- A matrix caller's check-name suffix is not measured here.

## Not done here, deliberately

- **rowspec does not call this yet.** Adopting it is a separate change
in rowspec: bump the kindkit pin, then follow the README's §3.2 steps.
- **`kind / conformance` is not a required check on kindkit.** Adding it
is a ruleset change for the owner.

**After merging `main` with #19** (merge commit `3d6f94b`): `just check`
all checks passed; `just test` 202 passed; `just mutants` (throwaway
detached worktree) 137 mutations, 137 caught, 0 survived, 0 broken. CI
green.

## Round 2 of review (head `9951ee2`, with `main` merged in, which
brought #20, #22 and #23)

- **M1, a relative `root` was accepted.** Before this change,
`_parse_report` checked only that `root` is a string, and `conform.py`
resolved it against its own working directory. The reviewer's runner did
`cd good && runner kvcopy` and reported `"root": "kvcopy"`. That passed
against a **broken** `./kvcopy`. New test
`test_a_report_whose_root_is_relative_is_no_verdict` rebuilds that shape
and was red on `3d6f94b`: `assert 0 == 2`. `_parse_report` now refuses a
non-absolute `root` with `'root' 'kvcopy' is not an absolute path`; unit
test `test_a_report_whose_root_is_relative_is_refused`. The README
contract now says the root is **absolute**. New gate mutation "a
relative root is accepted, to be resolved by whoever reads it"; both
listed tests fail on their own under it.
- **L1, the every-case-ran check was masked.** The subtree test was also
refused by the root check, so emptying the list of unrun cases survived
the suite. New test
`test_a_report_from_the_right_root_that_ran_only_some_cases_is_no_verdict`
hands `conform.py` a report naming the same root with 2 of the 6 cases.
It is now listed by "a suite that ran a different set of cases than the
tree passes", and by a new mutation "a case of the tree that did not run
goes unnoticed" (`sorted(tree - ran)` → `[]`). Run on its own, each
mutation fails that test.
- **L2, which file was mutated.** This is documented as a limit: a gate
whose only kill breaks the adapter or the second implementation still
passes. The gate report now records `source`, the file mutated, and
`check_gate.py` prints it. On CI: `the gate killed 3 mutant(s) of
'/home/runner/work/kindkit/kindkit/tests/kvkind.py' over all 6 case(s)`.
- **The gate report's keys** are now documented in the README:
`kindkit_gate_report`, `ok`, `killed`, `survived`, `equivalent`,
`stale`, `bogus`, `broken`, `ran`, `root` and `source`.

Gates at `9951ee2`:

```
$ just check
All checks passed!
$ just test
220 passed
$ just mutants        # throwaway detached worktree
148 mutations, 148 caught, 0 survived, 0 broken
```

CI on the PR: `check` and `kind / conformance` pass; mergeable state
`clean`.

Refs #4. Every bullet is met on kindkit's side. "rowspec's existing
workflows can call it and produce the same verdicts" is shown above
against rowspec, but not yet by rowspec's own workflows.
Refs #3. "Validate the schema against all 410 cases in CI" becomes true
when rowspec calls this workflow. Until then it holds for kindkit's toy
tree in CI, and for rowspec's 410 cases only locally (above).
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant