Repository navigation
feat: flag a required check that another workflow produces on push - #19
Merged
Merged
Conversation
The DUPLICATE rule read only a workflow's own push trigger. A job of the same name in a different workflow that runs on push to a pull request's branch gives the required context a second suite on that head, and the audit passed it. The audit now collects, across every active workflow, the contexts produced on push to some branch other than the default, using the same filter evaluation and refusals as the own-push rule, and expanding called workflows into their '<caller> / <called>' contexts. A required context produced on pull_request by one workflow and on push by another is a DUPLICATE that names both. A push job whose context cannot be resolved, or an undecided filter on a job of the same context, is refused with exit 2. The mutation sweep gains a per-mutant timeout and prints the target's digest before and after. Closes #18 Signed-off-by: Cameron Brooks <cambrooks3393@gmail.com>
CameronBrooks11
added a commit
that referenced
this pull request
Oct 8, 2026
## What changes The DUPLICATE rule (#17, #19) finds a second producer only on `push`. A required context can also get one from the pull request itself: - workflow A runs `check` on `pull_request`; - workflow B runs a job also named `check` on `pull_request`, or on `pull_request_target`. The head then carries `check` from two suites, and the gate takes whichever run it finds (kindspec/rowspec#43). The audit passed this. The PR also corrects how `pull_request_target` is treated, and adds TARGET-ONLY for a check that only it produces. ### DUPLICATE: a second producer on the pull request `push_producers()` is now `producers(data, branch, admits)`. - With `push_duplicates` it does what it did before. - With the new `pr_admits` it collects every **active** workflow whose `pull_request` or `pull_request_target` runs for a PR into the default branch, and the context of each job. That context is `name:` or else the id, or `<caller> / <called>` for a call. The same `expand()` and refusals apply. Three shapes are reported: - **Two workflows.** For each required context, every pair of workflows that produce it on a pull request is a DUPLICATE, reported once per pair. The pairs come straight from the producer list, so `ALLOW` on either job does not hide them. The message names both workflows and their events: DUPLICATE kindkit: required check 'check' is produced on a pull request into 'main' by .github/workflows/check.yml: job 'check' (on pull_request) and by .github/workflows/second.yml: job 'check' (on pull_request_target), so a pull request head can carry it from two suites and the merge takes either - **One workflow on both** `pull_request` and `pull_request_target` is also a DUPLICATE. - **Several producers.** These findings do not overlap with #17/#19's push rules; each one names a different pair of suites. Three workflows give three findings. `push` and `pull_request` in another workflow are two distinct suites from that workflow, so they give two findings, plus that workflow's own #17 finding. Cases pin each count. **Refused (exit 2), as #18 does:** - a job in another active pull request workflow whose context cannot be resolved (a matrix, an expression or block name, or a call not read), since it could be any context; - a branch filter on that workflow that cannot be read, when a job there has the same context. ### `pull_request_target` reports on the PR head Until now, a required check produced only by `pull_request_target` was ORPHANED, on the grounds that "its check runs attach to the base commit". That is wrong. I checked it against the issue's cpython run and two more independent repositories: | run | event | job's `head_sha` | PR head | listed in `commits/<sha>/check-runs` | in the PR's `statusCheckRollup` | |---|---|---|---|---|---| | python/cpython 25066296099 | pull_request_target | `77e73c2` | PR 149109 (fork `warsaw/cpython`) | per the issue | not checked | | electron/electron 37748122900 | pull_request_target | `a965781` | PR 54550 head `a965781` (same-repo branch, `compare main...sha` ahead 10, behind 0) | yes, check run 113214421021 | yes, `event: pull_request_target` | | react/react 37730121633 | pull_request_target | `7af445b` | PR 37788 head `7af445b` (fork `abdulmajidcse/react`, ahead 1, behind 0) | yes, check run 113157110710 | yes, `event: pull_request_target` | The commands were: - `gh api repos/<o>/<r>/actions/runs/<id>/jobs` - `gh api repos/<o>/<r>/pulls/<n>` - `gh api repos/<o>/<r>/commits/<sha>/check-runs` - a GraphQL query of the last commit's `statusCheckRollup.contexts` with `checkSuite.workflowRun.event` react's head also carries `check_access` from two different `pull_request_target` workflows, which is this issue's hazard in a real repository. GitHub's documentation says such checks count. The source is github/docs `content/pull-requests/how-tos/merge-and-close-pull-requests/troubleshooting-required-status-checks.md`, lines 55–64 at `60f9f34`: > For checks created by workflow jobs to be evaluated for a pull request, the workflow run must be triggered by one of these events: > `push`, `pull_request`, `pull_request_review`, `pull_request_target`, `deployment`, `deployment_status` > > For example, if a workflow is triggered by `workflow_dispatch` on a pull request's head branch, checks reported by its jobs do not appear in the pull request's checks section. Even if the checks pass for the head commit, they do not satisfy a required status check in a branch ruleset. That a ruleset accepts a `pull_request_target` check is the documentation's statement; I have not watched a kindspec ruleset do it. The same page answers the issue's "unverified routes": `workflow_dispatch` on a PR branch does not count, and `create` is not in the list. So `pull_request_target` is now a pull request event: - **ORPHANED.** A required check that `pull_request_target` alone produces is no longer ORPHANED; it gets TARGET-ONLY instead (below). - **UNGATED.** A `pull_request_target` job that is not required is UNGATED, like a `pull_request` job. A job meant to gate nothing, such as a labeller, goes in `ALLOW` with that reason. - **CONDITIONAL, TRIGGER, PATH-FILTER** apply to it as well. ### TARGET-ONLY: a check that may be unable to fail A `pull_request_target` job runs the workflow file from the default branch of the base repository, with `GITHUB_SHA` that branch's last commit. Unless it checks out the pull request, it runs on the default branch's code, so it may be unable to fail because of the pull request. That is the §2.2 pattern. The audit cannot tell whether a job checks out the pull request, so: - A required check whose **only** pull request producers are `pull_request_target` jobs is TARGET-ONLY, exit 1. - **`TARGET_OK`**, a new set of `(repo, context): reason`, accepts one, and only that finding. The job's other findings (CONDITIONAL, TRIGGER, PATH-FILTER, DUPLICATE) still fire. - **`ALLOW` silences only UNGATED**, for a job that gates nothing. A required job in `ALLOW` is checked like any other. An allowlisted job also still counts as producing its contexts; before, a required context from one was left ORPHANED. An allowlisted job whose context cannot be resolved, such as a matrix labeller, is still not refused, as on main. If it produced a required check, that check would be ORPHANED, so this is not a silent pass. - A check also produced on `pull_request` gets no TARGET-ONLY; the DUPLICATE rule covers it instead. - **A `push` producer clears TARGET-ONLY.** If a `push` run on a pull request's branch also produces the check, in the same workflow or another, TARGET-ONLY does not fire. That run tests the pull request's code, so the check can fail because of the pull request, and the shape is already a DUPLICATE. I chose this over rewording the message, because a message saying the check "may be unable to fail" would be false there. The message now says the check is "reported on a pull request only by pull_request_target". ### PATH-FILTER and TRIGGER, per event Each event that reaches the PR is blocked by a path filter, or by `types:` that omit opened or synchronize. - The blocks are findings only when the events that are not path-filtered together miss opened or synchronize; otherwise every head is reported. - Each finding names its event: `its pull_request_target trigger is path-filtered`, or `pull_request_target types ['labeled'] omit …`. - So `paths` on one event with `types: [labeled]` on the other gives both findings, and `types: [opened]` with `types: [synchronize]` gives neither. - One-event workflows behave as before, with one exception: `types: []` now gives TRIGGER (types `[]`). Before, it gave nothing, because the empty list read as false. GitHub does not document `types: []`, so the audit reads it fail-closed, and the docstring says so. ## Measured Live, with this branch: $ python3 -I scripts/check_required_checks.py --public-only --save snap.json 1 private repositories skipped (--public-only) 6 repositories, 7 workflows, 7 pull request jobs, 7 required checks: 0 finding(s), 0 not evaluated exit 0 No TARGET-ONLY in the org. The summary now says "pull request jobs", because the count includes `pull_request_target` jobs. **The issue's reproduction.** I added `.github/workflows/second.yml` to kindkit in that snapshot, with job `check` (`runs-on: ubuntu-latest`, one `run: 'true'` step) and `workflow_state` `active`: == pull_request on branch DUPLICATE kindkit: required check 'check' is produced on a pull request into 'main' by .github/workflows/check.yml: job 'check' (on pull_request) and by .github/workflows/second.yml: job 'check' (on pull_request), so a pull request head can carry it from two suites and the merge takes either 6 repositories, 8 workflows, 8 pull request jobs, 7 required checks: 1 finding(s), 0 not evaluated exit 1 == pull_request on main 6 repositories, 8 workflows, 8 pull_request jobs, 7 required checks: 0 finding(s), 0 not evaluated exit 0 == pull_request_target on branch DUPLICATE kindkit: required check 'check' is produced on a pull request into 'main' by .github/workflows/check.yml: job 'check' (on pull_request) and by .github/workflows/second.yml: job 'check' (on pull_request_target), so a pull request head can carry it from two suites and the merge takes either 6 repositories, 8 workflows, 8 pull request jobs, 7 required checks: 1 finding(s), 0 not evaluated exit 1 == pull_request_target on main 6 repositories, 8 workflows, 7 pull_request jobs, 7 required checks: 0 finding(s), 0 not evaluated exit 0 Main's figures match the issue's. **pull_request_target only.** On the same snapshot I changed kindkit's `check.yml` from `pull_request` to `pull_request_target`: == target-only on branch TARGET-ONLY kindkit: required check 'check' is reported on a pull request only by pull_request_target, which runs the default branch's workflow on the default branch's code unless the job checks out the pull request, so it may be unable to fail because of the pull request TARGET-ONLY kindkit: required check 'kind / conformance' is reported on a pull request only by pull_request_target, which runs the default branch's workflow on the default branch's code unless the job checks out the pull request, so it may be unable to fail because of the pull request 6 repositories, 7 workflows, 7 pull request jobs, 7 required checks: 2 finding(s), 0 not evaluated exit 1 == target-only on main ORPHANED kindkit: required check 'check' is produced by no job that reports on a pull request into 'main' (only pull_request_target, push) ORPHANED kindkit: required check 'kind / conformance' is produced by no job that reports on a pull request into 'main' 6 repositories, 7 workflows, 5 pull_request jobs, 7 required checks: 2 finding(s), 0 not evaluated exit 1 `ruff check` and `ruff format --check` (kindkit's ruff 0.16.6, `--line-length 100`) are clean on `scripts/`. ## Shown to fail **Red first.** I grafted this branch's self-test onto main's logic, as #15, #17 and #19 did: main's module up to `# --- self-test`, then this branch's self-test section. Main gives `selftest: 185 of 225 as expected`, exit 1. The three `TARGET_OK` cases crash there, because main's `evaluate` takes no such argument: FAIL pull_request_target reports on the PR head, but only base code: got ['ORPHANED'] exit 1, want ['TARGET-ONLY'] exit 1 FAIL two pull_request workflows: got [] exit 0, want ['DUPLICATE'] exit 1 FAIL pull_request and pull_request_target workflows: got [] exit 0, want ['DUPLICATE'] exit 1 FAIL pull_request_target first by path, pull_request second: got [] exit 0, want ['DUPLICATE'] exit 1 FAIL pull_request_target job named by name:: got [] exit 0, want ['DUPLICATE'] exit 1 FAIL one workflow on pull_request and pull_request_target: got [] exit 0, want ['DUPLICATE'] exit 1 FAIL three workflows give one finding per pair: got [] exit 0, want ['DUPLICATE', 'DUPLICATE', 'DUPLICATE'] exit 1 FAIL push and pull_request in another workflow are two producers: got ['DUPLICATE', 'DUPLICATE'] exit 1, want ['DUPLICATE', 'DUPLICATE', 'DUPLICATE'] exit 1 FAIL called workflow under pull_request in another workflow: got [] exit 0, want ['DUPLICATE'] exit 1 FAIL unrequired pull_request_target job: got [] exit 0, want ['UNGATED'] exit 1 FAIL pull_request_target with an if:: got ['ORPHANED'] exit 1, want ['CONDITIONAL', 'TARGET-ONLY'] exit 1 FAIL pull_request_target types: [labeled]: got ['ORPHANED'] exit 1, want ['TRIGGER', 'TARGET-ONLY'] exit 1 FAIL pull_request_target path-filtered: got ['ORPHANED'] exit 1, want ['PATH-FILTER', 'TARGET-ONLY'] exit 1 FAIL pull_request path-filtered, pull_request_target not: got ['PATH-FILTER'] exit 1, want ['DUPLICATE'] exit 1 FAIL both events path-filtered: got ['PATH-FILTER'] exit 1, want ['DUPLICATE', 'PATH-FILTER', 'PATH-FILTER'] exit 1 FAIL pull_request types: [labeled], pull_request_target default: got ['TRIGGER'] exit 1, want ['DUPLICATE'] exit 1 FAIL both events types: [labeled]: got ['TRIGGER'] exit 1, want ['DUPLICATE', 'TRIGGER', 'TRIGGER'] exit 1 FAIL pull_request path-filtered, pull_request_target types: [labeled]: got ['PATH-FILTER'] exit 1, want ['DUPLICATE', 'PATH-FILTER', 'TRIGGER'] exit 1 FAIL pull_request types: [labeled], pull_request_target path-filtered: got ['TRIGGER'] exit 1, want ['DUPLICATE', 'TRIGGER', 'PATH-FILTER'] exit 1 FAIL types: [opened] and types: [synchronize] together cover every head: got ['TRIGGER'] exit 1, want ['DUPLICATE'] exit 1 FAIL scalar types: opened and synchronize cover every head: got ['TRIGGER'] exit 1, want ['DUPLICATE'] exit 1 FAIL types: []: got [] exit 0, want ['TRIGGER'] exit 1 FAIL two pull_request_target workflows: got ['ORPHANED'] exit 1, want ['DUPLICATE', 'TARGET-ONLY'] exit 1 FAIL other pull_request_target job with a different name: got ['ORPHANED'] exit 1, want ['TARGET-ONLY'] exit 1 FAIL other pull_request matrix job is refused: got ['INCOMPLETE'] exit 2, want ['INCOMPLETE', 'INCOMPLETE'] exit 2 FAIL other pull_request filter it cannot read is refused: got ['INCOMPLETE'] exit 2, want ['INCOMPLETE', 'INCOMPLETE'] exit 2 FAIL allowlisted first producer still pairs: got [] exit 0, want ['DUPLICATE'] exit 1 FAIL allowlisted first pull_request_target producer still pairs: got ['STALE-ALLOW'] exit 1, want ['DUPLICATE'] exit 1 FAIL three producers, first allowlisted: got [] exit 0, want ['DUPLICATE', 'DUPLICATE', 'DUPLICATE'] exit 1 FAIL every producer allowlisted still pairs: got ['ORPHANED'] exit 1, want ['DUPLICATE'] exit 1 FAIL ALLOW does not accept target-only: got ['ORPHANED', 'STALE-ALLOW'] exit 1, want ['TARGET-ONLY'] exit 1 FAIL ALLOW on a required job keeps its own push DUPLICATE: got ['ORPHANED', 'STALE-ALLOW'] exit 1, want ['DUPLICATE'] exit 1 FAIL ALLOW on a required job keeps its CONDITIONAL: got ['ORPHANED', 'STALE-ALLOW'] exit 1, want ['CONDITIONAL', 'TARGET-ONLY'] exit 1 FAIL ALLOW on a required pull_request job: target-only needs every producer: got ['ORPHANED'] exit 1, want ['DUPLICATE'] exit 1 FAIL TARGET_OK accepts target-only: got ['CRASH TypeError'] exit -1, want [] exit 0 FAIL TARGET_OK keeps CONDITIONAL: got ['CRASH TypeError'] exit -1, want ['CONDITIONAL'] exit 1 FAIL TARGET_OK for another context accepts nothing: got ['CRASH TypeError'] exit -1, want ['TARGET-ONLY'] exit 1 FAIL pull_request_target and push in one workflow: got ['ORPHANED'] exit 1, want ['DUPLICATE'] exit 1 FAIL pull_request_target, and push in another workflow: got ['ORPHANED'] exit 1, want ['DUPLICATE'] exit 1 FAIL pull_request_target, push to the default branch only elsewhere: got ['ORPHANED'] exit 1, want ['TARGET-ONLY'] exit 1 selftest: 185 of 225 as expected Some new cases pass on main, and the sweep pins them: - the negative controls: - another `pull_request` filtered to `develop`; - another `pull_request_target` with `branches-ignore: [main]`; - a `pull_request` job with a different name; - a `merge_group` job of the same name; - a `disabled_manually` workflow; - two `disabled_inactivity` workflows, one `pull_request_target` and one `push`; - `pull_request_target` into another branch, which is still ORPHANED; - a `disabled_inactivity` workflow that produces nothing, which is ORPHANED; - a path-filtered job whose types include opened and synchronize, which gives PATH-FILTER only; - the undecided filter with a different name; - an allowlisted matrix job that gates nothing, which is not refused. At this head: `selftest: 225 of 225 as expected`. **The `disabled_inactivity` gap the issue names.** I mutated main's `if state != "active"` to `if state == "disabled_manually"`. Main's self-test then gives `selftest: 173 of 173 as expected`, exit 0, so the mutant survives there. On this branch, that mutant and the same one in `producers()` are `caught`. **Mutation sweep.** Against main, 128 mutants become 158: 33 added and 3 removed, all three replaced by mutants on the rewritten code. 8 keep their label but have new patterns, because the code they target was rewritten or renamed: the four push-doubt mutants, "push producers not collected", "path filter ignored", "restrictive types ignored" and "types check inverted". The lines for the new and rewritten mutants on the pull request rules: target sha256 before: 7e638faf8fc1126d963195568b71f7311a76702ff33f47cadf49c407ca4dde04 ... caught merge_group counted as a pr event caught pull_request_target not a pr event caught pr producers not collected caught two-event workflow not flagged caught pr branch filter ignored elsewhere caught own undecided pr refused again caught undecided pr refused for any context caught unknown-name pr job refused only by name caught pr doubts never refused caught only disabled_manually excluded caught only disabled_manually excluded from producers caught path filter ignored caught restrictive types ignored caught types check inverted caught filtered event counts as covering caught covering ignored caught default types do not cover caught blocks ignored for called jobs caught every block named a path filter caught types with opened and synchronize counted as restricted caught scalar types not wrapped caught pr pairs never reported caught pr pairs only between the first two caught allowlisted workflow dropped from pr pairs caught pr events not recorded caught target-only never reported caught target-only with another event too caught allowlisted job skips every check caught allow does not silence ungated caught allowlisted unresolved job refused caught target_ok ignored caught target_ok matched by repo only caught push producer does not clear target-only caught types: [] reads as unrestricted caught coverage needs only one default type caught pairs skipped when every producer is allowlisted target sha256 after: 7e638faf8fc1126d963195568b71f7311a76702ff33f47cadf49c407ca4dde04 158 mutants: 157 caught, 0 crashed the self-test, 1 hung past 30s, 0 survived, 0 broken The one `hung` is #19's `unclosed [ accepted`, which loops forever by construction and is counted red. Every other line is `caught`. The hash is that of the committed file. No CI runs the self-test or the sweep; the daily audit workflow runs the audit only. ## Docs - **AGENTS.md §3.2** now has five items. - The two-producer bullet covers a second producer on `pull_request` or `pull_request_target`, and one workflow on both, whether or not `ALLOW` names either job. - A new TARGET-ONLY bullet says why such a check may be unable to fail. It cites the docs source, says that the default branch of the base repository's workflow and code are what run, and says that `TARGET_OK` accepts one and `ALLOW` does not. - **README** names the new producer, TARGET-ONLY and `TARGET_OK`. - **STATE.md**: - the "still does not see" paragraph is replaced by what the audit now sees, with the evidence above, the doc source, and TARGET-ONLY; - the live figure says "pull request jobs"; - §6's .github#10 entry no longer lists `pull_request_target` among events that never report on the PR, and says why; - §6 gains this gap; - a new "still does not see" paragraph records what this leaves out. The audit does not read `pull_request_review`, `deployment` or `deployment_status` as producers, although the same page lists them as eligible. Nor does it flag two jobs of one workflow with the same context. `evaluate` returns no finding for either. These are left to a separate issue. ## Review An independent review of the first head found one HIGH and two MEDIUMs, plus some LOWs. This head addresses each one. - **H1, fixed.** `ALLOW` on the first-by-path producer hid the pull request DUPLICATE, and the audit exited 0. Pairs are now computed from the producer list, independently of `ALLOW`. The reviewer's A4, A4c and A5b are now cases, and all three are red on main. - **M1, fixed.** Paths and types were decided separately across events. They are now decided per event, a covered head clears the blocks, and each finding names its event. A3, A3b and A3c are now cases. - **M2, fixed.** TARGET-ONLY is added; see the second round for how it is accepted. - **LOW, fixed.** - The summary label now says "pull request jobs". - The docstring and AGENTS.md say "the default branch of the base repository". - Ruleset acceptance is marked as the documentation's statement, with the source path and lines cited. - This body cites the source path rather than the redirect URL. The reviewer's harness against this head: - **`attack.py`** gives the following: | cases | result | |---|---| | A3, A3b | DUPLICATE, plus PATH-FILTER and TRIGGER, each naming its event | | A3c | DUPLICATE only | | A4, A4b, A4c, A5, A5b | every pair, exit 1 | | A2b, A8 | TARGET-ONLY | | A9 | PATH-FILTER and TARGET-ONLY | - **`mymut.py`**: - R4, R6, R7, R8, R11, R12, R14 and R15 are caught. - R1 survives. It drops the "(… excludes 'main')" note from an ORPHANED message. The self-test checks finding kinds and exit codes, not wording, and pinning wording would keep a wrong message alive rather than catch it, so this survivor is accepted. - R2, R3, R5, R9, R10 and R13 are BROKEN, because the code they targeted was rewritten. I ported them to the new code and ran them under the same hashing. R3', R5', R9', R10' and R13' are caught. R3' and R10' were caught only after adding the two cases above; their sweep mutants are "types with opened and synchronize counted as restricted" and "scalar types not wrapped". R2' survives for the same reason as R1: it changes only the types list printed in a message. **Second round**, at `a85060d`. H1 and M1 were confirmed fixed, and the reviewer re-derived every figure. One new HIGH, one MEDIUM and one LOW came up; this head addresses each one. - **HIGH, fixed.** Accepting TARGET-ONLY through `ALLOW` also silenced every other finding on that required job, because allowlisted jobs skipped the whole loop. `ALLOW` now silences UNGATED only, and the new `TARGET_OK` accepts TARGET-ONLY only. The reviewer's B3b and B4 are now cases, as is "TARGET_OK keeps CONDITIONAL". `attack2.py` at this head: every one of B1 to B9 exits 1. - B3b, B3c, B4, B5, B6 and B7 now report the job's DUPLICATE, CONDITIONAL, TRIGGER or PATH-FILTER. - B8 gives DUPLICATE without TARGET-ONLY. - B3d exits 1, not 2: its `ma?n` push filter is settled, not undecided. `?` makes the `a` optional, so the filter admits `mn`, a witness branch, and the result is a DUPLICATE. Any finding gives exit 1 in any case. - **MEDIUM, fixed.** - `types: []` and a lone `types: [opened]` are now cases that give TRIGGER. "Every producer allowlisted still pairs" is a case too. - S11 and S12 are in the sweep as "types: [] reads as unrestricted" and "coverage needs only one default type", and S14 as "pairs skipped when every producer is allowlisted". All are caught. - The reviewer's literal S14 (`len(pr_events_of.get(context, ())) == 0`) is now an equivalent mutant: allowlisted jobs record their events, so the condition never holds for a context that has a producer. - **LOW, fixed.** A `push` producer clears TARGET-ONLY, as above, and the message is reworded. `mymut2.py` at this head: - Caught: S1, S2, S7–S12. - BROKEN, because their code was rewritten: S4, S5, S6. Their analogues in the sweep are caught: "target-only with another event too", "target-only never reported", and "allowlisted job skips every check". - Survive: S3 and S13, which change only message text, and S14, which is equivalent as explained above. Not reviewed again since the second round. Closes #20 --------- Signed-off-by: Cameron Brooks <cambrooks3393@gmail.com>
This was referenced Oct 8, 2026
CameronBrooks11
added a commit
that referenced
this pull request
Oct 15, 2026
## What changes `STATE.md` brought to 2026-10-15. Every figure was re-measured today, in fresh clones of each public repository's `main`, through `just`, with a private uv cache (`UV_CACHE_DIR=<scratch> UV_LINK_MODE=copy`). Sections that no longer describe anything are removed rather than annotated. - **blockspec**: the binding spike verdict is pre-registration 3's run 2, FOUND, under the owner's ruling of 2026-10-14 (`spike/LOG.md` §34–§35), with the rules it overrides and the noise figures beside it. Pre-registration 2 and run 1 stay INCONCLUSIVE. Adjudications 0001 (CommonMark 0.31.2 block-level profile) and 0002 (the block model), with their addenda. The case tree (blockspec#36, #38), the reference implementation (blockspec#37), and the required checks `conformance` and `reference`. The "trending toward do not build" paragraph and the blockspec#19 timing paragraph are gone. - **rowspec**: refusal identifiers, the CSV profile and its own tree and gate, and xlsx formulas. The fixes and spec rulings of rowspec#104 to #139, including that #136 was closed on #128 and #137 by #139. The 22 open issues, including #142–#145 filed today, and the one open pull request, #141, which addresses rowspec#112, #113 and #132–#135. The GitHub Release object that now exists for `v0.3.0`. - **nodespec**: the mermaid spike ran and is closed; identity is parked. - **kindkit**: blockspec is a second consumer, at a different pin from rowspec's. - **.github**: the audit now counts 9 pull request jobs and 9 required checks, with 0 findings. Daily runs listed. - **What comes next**, owner decisions, setup, traps and the decided list updated to match. blockspec's own `README.md` and `spike/README.md` still say there is no verdict on its `main`; blockspec#39 (open) updates them, and `STATE.md` points at it. - **The spike's 432** is described as what it is: each FOUND cell's distinct count summed over the 35 cells, not a count of distinct units. The noise figures of pre-registration 3 §2.6 are quoted in full beside the verdict. - **Citations corrected**: - the three recorded rule-suite rejections by id, in place of a month query that cannot tell old rejections from none; - blockspec's prereg 2 pull requests as #15–#19 and #26–#28; - the `ambiguous-table` schema change as landing through blockspec#38 (`ebfc86a`); - the information-barrier decision in one place, no longer attributing blind judgement to `AGENTS.md` §2.1; - the merge-count search as `merged:2026-10-14..2026-10-15`. - **A setup note**: kindkit's figures need bytecode writing enabled. - **`AGENTS.md` and `profile/README.md`** (second commit): §1's table and the org page said blockspec was "spike, no verdict"; they now give draft 0 (decisions, cases, a reference) and the spike's FOUND, with nodespec's spike closed. §3.1's required-checks table adds blockspec's `reference`, which `gh api repos/kindspec/blockspec/rules/branches/main` lists beside `conformance`. - §6 gains one entry: 98 of rowspec's 116 refusing `parse` cases had asserted `refusal_contains: ""` (rowspec#61, per `CHANGELOG.md`). ## Measured All on 2026-10-15. The audit, here at `cb04c2c`: $ python3 -I scripts/check_required_checks.py --public-only 1 private repositories skipped (--public-only) 6 repositories, 8 workflows, 9 pull request jobs, 9 required checks: 0 finding(s), 0 not evaluated rowspec at `40e5647` (rowspec#139's merge): find conformance/cases -name expect.json | wc -l 483 find conformance/cases-csv -name expect.json | wc -l 85 just conform 0 failure(s) across 483 case(s) in the fixture tree just conform-alt 0 failure(s) across 483 case(s) in the fixture tree just conform-csv 0 failure(s) across 85 case(s) in the fixture tree just mutants 86 killed, 0 survived, 2 equivalent, 0 stale, 0 broken just mutants-csv 33 killed, 0 survived, 2 equivalent, 0 stale, 0 broken just test 418 passed, 2 skipped just test-xlsx 20 passed just test-libreoffice 1 passed just check 450 files already formatted / All checks passed! gh api repos/kindspec/rowspec/compare/v0.3.0...main --jq .ahead_by 28 gh issue list -R kindspec/rowspec --state open --limit 200 --json number --jq length 22 gh pr list -R kindspec/rowspec --state merged --search 'merged:2026-10-14..2026-10-15' --limit 200 --json number --jq length 27 grep -rl '"refusal":' conformance/cases --include=expect.json | wc -l 130 grep -rl '"refusal_contains"' conformance/cases-csv --include=expect.json | wc -l 43 blockspec at `d5355ab`: find conformance/cases -name expect.json | wc -l 237 just tree 237 cases: 160 accept, 77 refuse / 0 problem(s) just conform 0 failure(s) across 237 case(s) in the fixture tree just mutants 91 killed, 0 survived, 0 equivalent, 0 stale, 0 broken just test 8 passed just check 10 files already formatted / All checks passed! gh api repos/kindspec/blockspec/rules/branches/main 22723622 strict, conformance, reference python3 -I -c "import json,collections,sys; v=json.load(open(sys.argv[1])); print(v['overall'], dict(collections.Counter(c['verdict'] for c in v['cells'].values())))" <file> spike/results/prereg2/verdict.json INCONCLUSIVE {'NO VERDICT': 52, 'NOT FOUND': 4} spike/results/prereg3/verdict.json INCONCLUSIVE {'NO VERDICT': 52, 'NOT FOUND': 4} spike/results/prereg3/run2/verdict.json FOUND {'NO VERDICT': 16, 'FOUND': 35, 'NOT FOUND': 5} kindkit at `6ebd80d` (throwaway clone, node on PATH): just mutants 148 mutations, 148 caught, 0 survived, 0 broken just test 220 passed just check 28 files already formatted just cases 6 case(s), 0 invalid gh api repos/kindspec/kindkit/compare/v0.1.0...main --jq .ahead_by 8 with PYTHONDONTWRITEBYTECODE=1: uv run pytest -q 3 failed, 217 passed just mutants 148 mutations, 147 caught, 1 survived, 0 broken Rule suites (`gh api repos/kindspec/<repo>/rulesets/rule-suites/<id> --jq '.result+" "+.pushed_at'`): .github 4014197203 fail 2026-09-10T00:19:14-04:00 rowspec 4014189520 fail 2026-09-10T00:18:20-04:00 nodespec 4014318589 fail 2026-09-10T00:34:02-04:00 git show --stat --format= ebfc86a -- conformance/cases/case-body.schema.json (in blockspec) conformance/cases/case-body.schema.json | 3 ++- Issue and pull request lists: `gh issue list` and `gh pr list` with `--state all --limit 200` for rowspec, blockspec, nodespec, kindkit, research and .github. Every other figure in the file names its command beside it. ## Shown to fail n/a: a documentation change; no check is added or altered. ## Review None yet. --------- Signed-off-by: Cameron Brooks <cambrooks3393@gmail.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
The DUPLICATE rule (#17) reads only a workflow's own
pushtrigger. Suppose workflow A runscheckonpull_request, and workflow B runs a job also namedcheckonpushwith no branch filter. A pull request head then carriescheckfrom two suites, and the merge gate takes whichever run it finds (kindspec/rowspec#43). The audit passed that: the issue's reproduction snapshot exits 0 on main.The rule
push_producers()walks every active workflow in a repository. It keeps the ones whosepushadmits some branch other than the default, decided by #17'spush_duplicates(). For each of their jobs it records the context the job produces:name:if present, else the job id;<caller> / <called>context per called job, through the sameexpand()thepull_requestside uses.A job's
if:is ignored here: a skipped job still reports its context, which is why CONDITIONAL exists. A case pins this.For every required context that a
pull_requestjob produces in workflow W, each push producer in a workflow other than W is a finding.Same kind, DUPLICATE. The hazard, its effect on the merge and the fix (restrict
pushto the default branch or to tags) are the same as #17's. AGENTS.md §3.2 treats them as one hazard, so a second finding name would split one hazard across two labels. The message names both workflows:No double reporting. W's own
pushis #17's finding, so the new rule skips producers in W. The existing caseon: [push, pull_request]still expects exactly one DUPLICATE. The mutant that drops the self-exclusion is caught there. If W has its ownpushand another workflow also pushescheck, that is two producers and gives two findings. A case pins this too.Refusals (exit 2), only where they could change the answer:
pushfilter cannot be settled (for example['*', '!*'], or any of feat: flag a required check that push also produces on a pr branch #17's refusals, such as!inbranches-ignoreorbranches: []), and a job in B has the same context. If the job name differs, nothing is refused.pushadmits another branch or cannot be settled, and one of its jobs has a context that cannot be resolved: a matrix, an expression or block-scalar name, or a call that cannot be read. That could be any context, so every requiredpull_requestcontext in another workflow is refused.A workflow B that cannot be parsed is already refused by the main pass. An inactive B, or one whose
pushadmits only the default branch or only tags, produces nothing.Red first
The cases are in a new
crosslist. Five are positive: the same job id, a matchingname:, a job withif:,branches-ignore: [main], and a called workflow underpush. There is also the own-push-plus-elsewhere pair, and an unrequired context, which stays UNGATED only. The negative controls are:pushfiltered to the default branch only;push;name:;push.The refusals are an expression name, a matrix job, an undecided filter with the same name, and a call that is not in the snapshot.
These were grafted onto main's logic, as #15 and #17 were: main's module up to
# --- self-test, then this branch's self-test section.The other 9 new cases pass on main. They are the negative controls and the UNGATED case, which main gets right because it has no rule. The sweep pins them. At this head:
Mutation sweep
Two changes to
scripts/sweep_check_required_checks.py:hung, which is red but counted apart fromcaughtandcrashed. The self-test takes about 0.05 s.main()with one of each plus one real mutant. It printedBROKEN,BROKEN,caught, andexit 1.#17 left out the
end < 0mutant on the unclosed-[check because it loops forever. It is in now, and it comes outhung. The sweep counts it red. No CI here runs the self-test or the sweep; if that mutant shipped, the daily audit would hang onma[inand fail on its 10-minute job timeout. Thema[incase covers the refusal itself. 13 mutants were added for the new rule:All other lines are
caught. Nothing survived.ruff format --checkandruff check(kindkit's ruff 0.16.6, line length 100) are clean onscripts/.Live
The issue's reproduction. I took a
--savesnapshot from that run and added.github/workflows/push.ymlto kindkit, withon: push, a jobcheckwith onerun: "true"step, andworkflow_stateactive:Main's logic on the same snapshot gives
8 workflows, 7 pull_request jobs, 7 required checks: 0 finding(s), 0 not evaluatedand exits 0, which matches the issue.A called workflow, on the same snapshot. Here
push.ymlis insteadon: pushwithkind: uses: ./.github/workflows/kind.yml:Docs
pushjob elsewhere, and that apushjob whose name the audit cannot resolve is refused.Not covered
Two
pull_requestproducers. Two workflows that both run a job namedcheckonpull_requestalso give a required context two suites. The audit reports nothing for that. I checked by callingevaluateon a two-workflow snapshot, which returned([], [], ...). That is a separate rule with no issue yet. STATE.md now says so.Refs #16, #17
Closes #18