Repository navigation
feat: expand reusable workflow calls into their contexts - #15
Merged
Merged
Conversation
A job that calls a reusable workflow was refused, so kindkit's and rowspec's `kind` jobs, and their required `kind / conformance` contexts, were never evaluated and the audit exited 2 on the real org. A caller job now produces one context per called job, named '<caller job> / <called job>' from each one's name or id. The caller's triggers, if: and needs: gate every called job, and each called job's own if: and needs: gate it. A local call is read from the default-branch snapshot; a call into another repository is fetched at exactly its pinned ref and recorded under "called" in the snapshot, so --load reproduces it offline. Still not evaluated: a call that cannot be fetched or that the snapshot lacks, any other uses: form, a called workflow with no workflow_call trigger, and a called job that itself calls a workflow or has a matrix, expression or block-scalar name. Refs #3 Signed-off-by: Cameron Brooks <cambrooks3393@gmail.com>
…rkflows Signed-off-by: Cameron Brooks <cambrooks3393@gmail.com>
A job carrying uses: together with runs-on: or steps: is a workflow GitHub rejects. The expansion let it through as a clean caller; it is now refused, as main refused it. A skipped caller job leaves its '<caller> / <called>' contexts unreported, so a required one holds the merge at Expected rather than reporting success. The CONDITIONAL line for that case now says so. New self-test cases pin what was correct but unchecked: each call is read at its own ref, a caller's path filter and pull_request types apply to its called jobs, and the summary counts called jobs. The sweep gains a mutant for each, and for the new refusal. Refs #3 Signed-off-by: Cameron Brooks <cambrooks3393@gmail.com>
CameronBrooks11
added a commit
that referenced
this pull request
Oct 8, 2026
The audit exits 0 on the public repositories now that it expands reusable workflow calls (#15), so the "teach it reusable workflows" item leaves the next-work list. Record kindkit's move to one producer per required check (kindkit#26), and name the second-producer case the audit still does not see. Refs #16 Signed-off-by: Cameron Brooks <cambrooks3393@gmail.com>
CameronBrooks11
added a commit
that referenced
this pull request
Oct 8, 2026
Runs `scripts/check_required_checks.py` on a schedule, the half of #3 its comment lists as still open. ## What it adds `.github/workflows/required-checks-audit.yml`: - `on: schedule` (daily, 06:17 UTC) and `workflow_dispatch`; - `permissions: contents: read`; - one job, `audit`, running `python3 -I scripts/check_required_checks.py --public-only` with `GH_TOKEN` set to the job's own token. `gh` needs a token in the environment even for endpoints that answer unauthenticated. **No `pull_request` trigger.** With one, `audit` would run on pull requests here and gate nothing, which is the UNGATED finding the script exists to report. Checked offline: adding `pull_request:` to this file in a saved snapshot makes the script print `UNGATED .github: .github/workflows/required-checks-audit.yml: job 'audit' runs on pull requests; 'audit' is not a required check` and exit 1. Without it, the snapshot evaluates to the same result as live, so the file parses under the script's YAML subset and contributes no pull_request job. `AGENTS.md` §3.2 and `README.md` now point at it, and the §3.1 row for `.github` says its one workflow never runs on a pull request. ## Observed in Actions This branch cannot be dispatched before it merges, so a throwaway branch `tmp/audit-probe` (commit `9ed1335`, now deleted) carried this file with an added `push:` trigger for that branch, plus a second workflow running the same script on two `--load` snapshot fixtures. | run | job | result | what it printed | |---|---|---|---| | 37716690043 | `audit` (this workflow, live, job token) | **red, exit 2** | `6 repositories, 6 workflows, 7 pull_request jobs, 7 required checks: 0 finding(s), 4 not evaluated`, and `an unknown number of private repositories skipped (--public-only)` | | 37716690040 | `clean` (fixture: one PR job, required) | **green, exit 0** | `1 repositories, ... 0 finding(s), 0 not evaluated` | | 37716690040 | `finding` (fixture: one PR job, not required) | **red, exit 1** | `UNGATED demo: .github/workflows/ci.yml: job 'check' runs on pull requests; 'check' is not a required check` | So the job token reads all six public repositories, the same totals as a local run with a personal token (`python3 -I scripts/check_required_checks.py --public-only`). The only difference is that the job token cannot see the org's private count, so it says "an unknown number" where a local run says `1`. The green and the exit-1 red come from fixtures, not from the live org. The `audit` row above ran before .github#15 merged, so it was red (exit 2) for the reason in the next section. ## The live audit is now green (since .github#15) .github#15 (merged) expands a job that calls a reusable workflow into `<caller> / <called job>` contexts, so the four INCOMPLETE lines from the first version of this PR are gone. This branch has `main` merged in (`f7b35c7`). On this branch at `bee8a3f`, with a personal token: $ python3 -I scripts/check_required_checks.py --public-only 1 private repositories skipped (--public-only) 6 repositories, 6 workflows, 7 pull_request jobs, 7 required checks: 0 finding(s), 0 not evaluated exit 0 $ python3 -I scripts/check_required_checks.py --selftest selftest: 91 of 91 as expected The live run reads `main`, which does not have this workflow yet. Adding `required-checks-audit.yml` to `.github` in the saved snapshot (`--save`, then `--load` the edited copy) gives `6 repositories, 7 workflows, 7 pull_request jobs, 7 required checks: 0 finding(s), 0 not evaluated`, exit 0. So after merge the workflow parses, contributes no pull_request job, and does not report itself. `STATE.md` is updated in this PR: - the audit exits 0 on the org (§1, `.github`); - kindkit#26 merged, so each required check has one producer. Merge commit `6ebd80d` shows `check` and `kind / conformance` from one `push` suite, both `success`; - .github#16, which is still open, is the gap the audit does not see: a required context also produced by `push` runs on other branches; - "teach it reusable workflows" leaves the next-work list. ## Issue 3 stays open until the first scheduled run is green #3's acceptance, bullet by bullet: 1. *A check exists that fails when a workflow job is not in the ruleset, or a required context matches no job.* **Met** by #10 and extended by #15. 2. *Its red state is demonstrated in both directions.* **Met.** #10's body records UNGATED and ORPHANED on edited snapshots of the real org, and #15's body records both again for the expanded `kind / conformance` contexts. 3. *`blockspec`, `research` and `.github` are picked up automatically when they gain their first workflow.* **Not yet shown.** The script lists every repository in the org on each run, so a new workflow is read without configuration. "Automatically" is the schedule, though, and a schedule only exists on the default branch, so nothing before merge can show it firing. **Shown before merge:** - the job runs in Actions with its own token and reads all six public repositories (run 37716690043, on a throwaway branch); - the script exits 0 on the live org, and 1 or 2 on fixtures; - this workflow does not report itself. **Checked after merge:** - a `workflow_dispatch` run on `main` goes green with the `0 finding(s), 0 not evaluated` line; - then the first `schedule` run (06:17 UTC) does the same, which shows the trigger fires. A closing keyword would close the issue at merge, before either run. This PR therefore uses `Refs`, and #3 should be closed by hand, citing the first green scheduled run. One caveat for later, which does not change the above. GitHub disables a public repository's scheduled workflows after 60 days with no repository activity (GitHub's documented behaviour; not observed here). If `.github` goes quiet for that long, the audit stops running. Refs #3. Refs #16. --------- Signed-off-by: Cameron Brooks <cambrooks3393@gmail.com>
CameronBrooks11
added a commit
that referenced
this pull request
Oct 8, 2026
The DUPLICATE rule (#17) reads only a workflow's own `push` trigger. Suppose workflow A runs `check` on `pull_request`, and workflow B runs a job also named `check` on `push` with no branch filter. A pull request head then carries `check` from two suites, and the merge gate takes whichever run it finds (kindspec/rowspec#43). The audit passed that: the issue's reproduction snapshot exits 0 on main. ## The rule `push_producers()` walks every **active** workflow in a repository. It keeps the ones whose `push` admits some branch other than the default, decided by #17's `push_duplicates()`. For each of their jobs it records the context the job produces: - `name:` if present, else the job id; - for a job that calls a reusable workflow, one `<caller> / <called>` context per called job, through the same `expand()` the `pull_request` side uses. A job's `if:` is ignored here: a skipped job still reports its context, which is why CONDITIONAL exists. A case pins this. For every **required** context that a `pull_request` job produces in workflow W, each push producer in a workflow **other than W** is a finding. **Same kind, DUPLICATE.** The hazard, its effect on the merge and the fix (restrict `push` to the default branch or to tags) are the same as #17's. AGENTS.md §3.2 treats them as one hazard, so a second finding name would split one hazard across two labels. The message names both workflows: DUPLICATE kindkit: .github/workflows/check.yml: job 'check' is required, and .github/workflows/push.yml: job 'check' also produces 'check' on push to other branches (push has no branch filter), so a pull request head can carry it from two suites and the merge takes either **No double reporting.** W's own `push` is #17's finding, so the new rule skips producers in W. The existing case `on: [push, pull_request]` still expects exactly one DUPLICATE. The mutant that drops the self-exclusion is caught there. If W has its own `push` and another workflow also pushes `check`, that is two producers and gives two findings. A case pins this too. **Refusals (exit 2), only where they could change the answer:** - B's `push` filter cannot be settled (for example `['*', '!*']`, or any of #17's refusals, such as `!` in `branches-ignore` or `branches: []`), and a job in B has the **same** context. If the job name differs, nothing is refused. - B's `push` admits another branch or cannot be settled, and one of its jobs has a context that cannot be resolved: a matrix, an expression or block-scalar name, or a call that cannot be read. That could be any context, so every required `pull_request` context in another workflow is refused. A workflow B that cannot be parsed is already refused by the main pass. An inactive B, or one whose `push` admits only the default branch or only tags, produces nothing. ## Red first The cases are in a new `cross` list. Five are positive: the same job id, a matching `name:`, a job with `if:`, `branches-ignore: [main]`, and a called workflow under `push`. There is also the own-push-plus-elsewhere pair, and an unrequired context, which stays UNGATED only. The negative controls are: - `push` filtered to the default branch only; - a tags-only `push`; - a different job id; - the same id with a different `name:`; - a disabled workflow; - a called workflow with other job names; - an undecided filter with a different name; - an expression name under a default-only `push`. The refusals are an expression name, a matrix job, an undecided filter with the same name, and a call that is not in the snapshot. These were grafted onto main's logic, as #15 and #17 were: main's module up to `# --- self-test`, then this branch's self-test section. FAIL push job of the same id in another workflow: got [] exit 0, want ['DUPLICATE'] exit 1 FAIL push job named by name: in another workflow: got [] exit 0, want ['DUPLICATE'] exit 1 FAIL push job with an if: still reports: got [] exit 0, want ['DUPLICATE'] exit 1 FAIL push with branches-ignore in another workflow: got [] exit 0, want ['DUPLICATE'] exit 1 FAIL called workflow under push in another workflow: got [] exit 0, want ['DUPLICATE'] exit 1 FAIL own push and another workflow's push are two findings: got ['DUPLICATE'] exit 1, want ['DUPLICATE', 'DUPLICATE'] exit 1 FAIL push job with an expression name is refused: got [] exit 0, want ['INCOMPLETE'] exit 2 FAIL push matrix job is refused: got [] exit 0, want ['INCOMPLETE'] exit 2 FAIL undecided push filter, same name, is refused: got [] exit 0, want ['INCOMPLETE'] exit 2 FAIL unreadable called workflow under push is refused: got [] exit 0, want ['INCOMPLETE'] exit 2 selftest: 163 of 173 as expected The other 9 new cases pass on main. They are the negative controls and the UNGATED case, which main gets right because it has no rule. The sweep pins them. At this head: selftest: 173 of 173 as expected ## Mutation sweep Two changes to `scripts/sweep_check_required_checks.py`: - **A 30 s timeout per mutant.** A mutant that runs past it is counted as `hung`, which is red but counted apart from `caught` and `crashed`. The self-test takes about 0.05 s. - **The target's sha256 is printed before and after.** A pattern that matches nothing, or a replacement that leaves the text unchanged, is still BROKEN. I checked this by driving `main()` with one of each plus one real mutant. It printed `BROKEN`, `BROKEN`, `caught`, and `exit 1`. #17 left out the `end < 0` mutant on the unclosed-`[` check because it loops forever. It is in now, and it comes out `hung`. The sweep counts it red. No CI here runs the self-test or the sweep; if that mutant shipped, the daily audit would hang on `ma[in` and fail on its 10-minute job timeout. The `ma[in` case covers the refusal itself. 13 mutants were added for the new rule: target sha256 before: 7db249f493b73faf8bcb004c9a60b675a5c267803de6db92cc5245a9eb30ab71 ... hung unclosed [ accepted caught cross-workflow duplicate never reported caught own push counted again as another workflow caught own undecided push refused again caught undecided push refused for any context caught unknown-name push job refused only by name caught push doubts never refused caught inactive workflow pushes caught push filter ignored elsewhere caught undecided push elsewhere passed caught undecided push elsewhere is a finding caught push calls not expanded caught unknown push job accepted caught push producers not collected target sha256 after: 7db249f493b73faf8bcb004c9a60b675a5c267803de6db92cc5245a9eb30ab71 128 mutants: 127 caught, 0 crashed the self-test, 1 hung past 30s, 0 survived, 0 broken All other lines are `caught`. Nothing survived. `ruff format --check` and `ruff check` (kindkit's ruff 0.16.6, line length 100) are clean on `scripts/`. ## Live $ python3 -I scripts/check_required_checks.py --public-only 1 private repositories skipped (--public-only) 6 repositories, 7 workflows, 7 pull_request jobs, 7 required checks: 0 finding(s), 0 not evaluated exit 0 **The issue's reproduction.** I took a `--save` snapshot from that run and added `.github/workflows/push.yml` to kindkit, with `on: push`, a job `check` with one `run: "true"` step, and `workflow_state` `active`: $ python3 -I scripts/check_required_checks.py --load snap-push.json DUPLICATE kindkit: .github/workflows/check.yml: job 'check' is required, and .github/workflows/push.yml: job 'check' also produces 'check' on push to other branches (push has no branch filter), so a pull request head can carry it from two suites and the merge takes either 6 repositories, 8 workflows, 7 pull_request jobs, 7 required checks: 1 finding(s), 0 not evaluated exit 1 Main's logic on the same snapshot gives `8 workflows, 7 pull_request jobs, 7 required checks: 0 finding(s), 0 not evaluated` and exits 0, which matches the issue. **A called workflow, on the same snapshot.** Here `push.yml` is instead `on: push` with `kind: uses: ./.github/workflows/kind.yml`: DUPLICATE kindkit: .github/workflows/check.yml: job 'kind' (called job 'conformance') is required, and .github/workflows/push.yml: job 'kind' (called job 'conformance') also produces 'kind / conformance' on push to other branches (push has no branch filter), so a pull request head can carry it from two suites and the merge takes either 6 repositories, 8 workflows, 7 pull_request jobs, 7 required checks: 1 finding(s), 0 not evaluated exit 1 ## Docs - **AGENTS.md §3.2.** The audit bullet no longer excepts a second producer in a different workflow. The two-producers bullet now says the second producer can be a `push` job elsewhere, and that a `push` job whose name the audit cannot resolve is refused. - **README** adds "in the same workflow or another" to the audit's description. - **STATE.md:** - the "still does not see" paragraph is replaced with what the audit now sees; - the live figure's heading now names both rules; - §6 gains this gap. ## Not covered **Two `pull_request` producers.** Two workflows that both run a job named `check` on `pull_request` also give a required context two suites. The audit reports nothing for that. I checked by calling `evaluate` on a two-workflow snapshot, which returned `([], [], ...)`. That is a separate rule with no issue yet. STATE.md now says so. Refs #16, #17 Closes #18 Signed-off-by: Cameron Brooks <cambrooks3393@gmail.com>
CameronBrooks11
added a commit
that referenced
this pull request
Oct 8, 2026
## What changes The DUPLICATE rule (#17, #19) finds a second producer only on `push`. A required context can also get one from the pull request itself: - workflow A runs `check` on `pull_request`; - workflow B runs a job also named `check` on `pull_request`, or on `pull_request_target`. The head then carries `check` from two suites, and the gate takes whichever run it finds (kindspec/rowspec#43). The audit passed this. The PR also corrects how `pull_request_target` is treated, and adds TARGET-ONLY for a check that only it produces. ### DUPLICATE: a second producer on the pull request `push_producers()` is now `producers(data, branch, admits)`. - With `push_duplicates` it does what it did before. - With the new `pr_admits` it collects every **active** workflow whose `pull_request` or `pull_request_target` runs for a PR into the default branch, and the context of each job. That context is `name:` or else the id, or `<caller> / <called>` for a call. The same `expand()` and refusals apply. Three shapes are reported: - **Two workflows.** For each required context, every pair of workflows that produce it on a pull request is a DUPLICATE, reported once per pair. The pairs come straight from the producer list, so `ALLOW` on either job does not hide them. The message names both workflows and their events: DUPLICATE kindkit: required check 'check' is produced on a pull request into 'main' by .github/workflows/check.yml: job 'check' (on pull_request) and by .github/workflows/second.yml: job 'check' (on pull_request_target), so a pull request head can carry it from two suites and the merge takes either - **One workflow on both** `pull_request` and `pull_request_target` is also a DUPLICATE. - **Several producers.** These findings do not overlap with #17/#19's push rules; each one names a different pair of suites. Three workflows give three findings. `push` and `pull_request` in another workflow are two distinct suites from that workflow, so they give two findings, plus that workflow's own #17 finding. Cases pin each count. **Refused (exit 2), as #18 does:** - a job in another active pull request workflow whose context cannot be resolved (a matrix, an expression or block name, or a call not read), since it could be any context; - a branch filter on that workflow that cannot be read, when a job there has the same context. ### `pull_request_target` reports on the PR head Until now, a required check produced only by `pull_request_target` was ORPHANED, on the grounds that "its check runs attach to the base commit". That is wrong. I checked it against the issue's cpython run and two more independent repositories: | run | event | job's `head_sha` | PR head | listed in `commits/<sha>/check-runs` | in the PR's `statusCheckRollup` | |---|---|---|---|---|---| | python/cpython 25066296099 | pull_request_target | `77e73c2` | PR 149109 (fork `warsaw/cpython`) | per the issue | not checked | | electron/electron 37748122900 | pull_request_target | `a965781` | PR 54550 head `a965781` (same-repo branch, `compare main...sha` ahead 10, behind 0) | yes, check run 113214421021 | yes, `event: pull_request_target` | | react/react 37730121633 | pull_request_target | `7af445b` | PR 37788 head `7af445b` (fork `abdulmajidcse/react`, ahead 1, behind 0) | yes, check run 113157110710 | yes, `event: pull_request_target` | The commands were: - `gh api repos/<o>/<r>/actions/runs/<id>/jobs` - `gh api repos/<o>/<r>/pulls/<n>` - `gh api repos/<o>/<r>/commits/<sha>/check-runs` - a GraphQL query of the last commit's `statusCheckRollup.contexts` with `checkSuite.workflowRun.event` react's head also carries `check_access` from two different `pull_request_target` workflows, which is this issue's hazard in a real repository. GitHub's documentation says such checks count. The source is github/docs `content/pull-requests/how-tos/merge-and-close-pull-requests/troubleshooting-required-status-checks.md`, lines 55–64 at `60f9f34`: > For checks created by workflow jobs to be evaluated for a pull request, the workflow run must be triggered by one of these events: > `push`, `pull_request`, `pull_request_review`, `pull_request_target`, `deployment`, `deployment_status` > > For example, if a workflow is triggered by `workflow_dispatch` on a pull request's head branch, checks reported by its jobs do not appear in the pull request's checks section. Even if the checks pass for the head commit, they do not satisfy a required status check in a branch ruleset. That a ruleset accepts a `pull_request_target` check is the documentation's statement; I have not watched a kindspec ruleset do it. The same page answers the issue's "unverified routes": `workflow_dispatch` on a PR branch does not count, and `create` is not in the list. So `pull_request_target` is now a pull request event: - **ORPHANED.** A required check that `pull_request_target` alone produces is no longer ORPHANED; it gets TARGET-ONLY instead (below). - **UNGATED.** A `pull_request_target` job that is not required is UNGATED, like a `pull_request` job. A job meant to gate nothing, such as a labeller, goes in `ALLOW` with that reason. - **CONDITIONAL, TRIGGER, PATH-FILTER** apply to it as well. ### TARGET-ONLY: a check that may be unable to fail A `pull_request_target` job runs the workflow file from the default branch of the base repository, with `GITHUB_SHA` that branch's last commit. Unless it checks out the pull request, it runs on the default branch's code, so it may be unable to fail because of the pull request. That is the §2.2 pattern. The audit cannot tell whether a job checks out the pull request, so: - A required check whose **only** pull request producers are `pull_request_target` jobs is TARGET-ONLY, exit 1. - **`TARGET_OK`**, a new set of `(repo, context): reason`, accepts one, and only that finding. The job's other findings (CONDITIONAL, TRIGGER, PATH-FILTER, DUPLICATE) still fire. - **`ALLOW` silences only UNGATED**, for a job that gates nothing. A required job in `ALLOW` is checked like any other. An allowlisted job also still counts as producing its contexts; before, a required context from one was left ORPHANED. An allowlisted job whose context cannot be resolved, such as a matrix labeller, is still not refused, as on main. If it produced a required check, that check would be ORPHANED, so this is not a silent pass. - A check also produced on `pull_request` gets no TARGET-ONLY; the DUPLICATE rule covers it instead. - **A `push` producer clears TARGET-ONLY.** If a `push` run on a pull request's branch also produces the check, in the same workflow or another, TARGET-ONLY does not fire. That run tests the pull request's code, so the check can fail because of the pull request, and the shape is already a DUPLICATE. I chose this over rewording the message, because a message saying the check "may be unable to fail" would be false there. The message now says the check is "reported on a pull request only by pull_request_target". ### PATH-FILTER and TRIGGER, per event Each event that reaches the PR is blocked by a path filter, or by `types:` that omit opened or synchronize. - The blocks are findings only when the events that are not path-filtered together miss opened or synchronize; otherwise every head is reported. - Each finding names its event: `its pull_request_target trigger is path-filtered`, or `pull_request_target types ['labeled'] omit …`. - So `paths` on one event with `types: [labeled]` on the other gives both findings, and `types: [opened]` with `types: [synchronize]` gives neither. - One-event workflows behave as before, with one exception: `types: []` now gives TRIGGER (types `[]`). Before, it gave nothing, because the empty list read as false. GitHub does not document `types: []`, so the audit reads it fail-closed, and the docstring says so. ## Measured Live, with this branch: $ python3 -I scripts/check_required_checks.py --public-only --save snap.json 1 private repositories skipped (--public-only) 6 repositories, 7 workflows, 7 pull request jobs, 7 required checks: 0 finding(s), 0 not evaluated exit 0 No TARGET-ONLY in the org. The summary now says "pull request jobs", because the count includes `pull_request_target` jobs. **The issue's reproduction.** I added `.github/workflows/second.yml` to kindkit in that snapshot, with job `check` (`runs-on: ubuntu-latest`, one `run: 'true'` step) and `workflow_state` `active`: == pull_request on branch DUPLICATE kindkit: required check 'check' is produced on a pull request into 'main' by .github/workflows/check.yml: job 'check' (on pull_request) and by .github/workflows/second.yml: job 'check' (on pull_request), so a pull request head can carry it from two suites and the merge takes either 6 repositories, 8 workflows, 8 pull request jobs, 7 required checks: 1 finding(s), 0 not evaluated exit 1 == pull_request on main 6 repositories, 8 workflows, 8 pull_request jobs, 7 required checks: 0 finding(s), 0 not evaluated exit 0 == pull_request_target on branch DUPLICATE kindkit: required check 'check' is produced on a pull request into 'main' by .github/workflows/check.yml: job 'check' (on pull_request) and by .github/workflows/second.yml: job 'check' (on pull_request_target), so a pull request head can carry it from two suites and the merge takes either 6 repositories, 8 workflows, 8 pull request jobs, 7 required checks: 1 finding(s), 0 not evaluated exit 1 == pull_request_target on main 6 repositories, 8 workflows, 7 pull_request jobs, 7 required checks: 0 finding(s), 0 not evaluated exit 0 Main's figures match the issue's. **pull_request_target only.** On the same snapshot I changed kindkit's `check.yml` from `pull_request` to `pull_request_target`: == target-only on branch TARGET-ONLY kindkit: required check 'check' is reported on a pull request only by pull_request_target, which runs the default branch's workflow on the default branch's code unless the job checks out the pull request, so it may be unable to fail because of the pull request TARGET-ONLY kindkit: required check 'kind / conformance' is reported on a pull request only by pull_request_target, which runs the default branch's workflow on the default branch's code unless the job checks out the pull request, so it may be unable to fail because of the pull request 6 repositories, 7 workflows, 7 pull request jobs, 7 required checks: 2 finding(s), 0 not evaluated exit 1 == target-only on main ORPHANED kindkit: required check 'check' is produced by no job that reports on a pull request into 'main' (only pull_request_target, push) ORPHANED kindkit: required check 'kind / conformance' is produced by no job that reports on a pull request into 'main' 6 repositories, 7 workflows, 5 pull_request jobs, 7 required checks: 2 finding(s), 0 not evaluated exit 1 `ruff check` and `ruff format --check` (kindkit's ruff 0.16.6, `--line-length 100`) are clean on `scripts/`. ## Shown to fail **Red first.** I grafted this branch's self-test onto main's logic, as #15, #17 and #19 did: main's module up to `# --- self-test`, then this branch's self-test section. Main gives `selftest: 185 of 225 as expected`, exit 1. The three `TARGET_OK` cases crash there, because main's `evaluate` takes no such argument: FAIL pull_request_target reports on the PR head, but only base code: got ['ORPHANED'] exit 1, want ['TARGET-ONLY'] exit 1 FAIL two pull_request workflows: got [] exit 0, want ['DUPLICATE'] exit 1 FAIL pull_request and pull_request_target workflows: got [] exit 0, want ['DUPLICATE'] exit 1 FAIL pull_request_target first by path, pull_request second: got [] exit 0, want ['DUPLICATE'] exit 1 FAIL pull_request_target job named by name:: got [] exit 0, want ['DUPLICATE'] exit 1 FAIL one workflow on pull_request and pull_request_target: got [] exit 0, want ['DUPLICATE'] exit 1 FAIL three workflows give one finding per pair: got [] exit 0, want ['DUPLICATE', 'DUPLICATE', 'DUPLICATE'] exit 1 FAIL push and pull_request in another workflow are two producers: got ['DUPLICATE', 'DUPLICATE'] exit 1, want ['DUPLICATE', 'DUPLICATE', 'DUPLICATE'] exit 1 FAIL called workflow under pull_request in another workflow: got [] exit 0, want ['DUPLICATE'] exit 1 FAIL unrequired pull_request_target job: got [] exit 0, want ['UNGATED'] exit 1 FAIL pull_request_target with an if:: got ['ORPHANED'] exit 1, want ['CONDITIONAL', 'TARGET-ONLY'] exit 1 FAIL pull_request_target types: [labeled]: got ['ORPHANED'] exit 1, want ['TRIGGER', 'TARGET-ONLY'] exit 1 FAIL pull_request_target path-filtered: got ['ORPHANED'] exit 1, want ['PATH-FILTER', 'TARGET-ONLY'] exit 1 FAIL pull_request path-filtered, pull_request_target not: got ['PATH-FILTER'] exit 1, want ['DUPLICATE'] exit 1 FAIL both events path-filtered: got ['PATH-FILTER'] exit 1, want ['DUPLICATE', 'PATH-FILTER', 'PATH-FILTER'] exit 1 FAIL pull_request types: [labeled], pull_request_target default: got ['TRIGGER'] exit 1, want ['DUPLICATE'] exit 1 FAIL both events types: [labeled]: got ['TRIGGER'] exit 1, want ['DUPLICATE', 'TRIGGER', 'TRIGGER'] exit 1 FAIL pull_request path-filtered, pull_request_target types: [labeled]: got ['PATH-FILTER'] exit 1, want ['DUPLICATE', 'PATH-FILTER', 'TRIGGER'] exit 1 FAIL pull_request types: [labeled], pull_request_target path-filtered: got ['TRIGGER'] exit 1, want ['DUPLICATE', 'TRIGGER', 'PATH-FILTER'] exit 1 FAIL types: [opened] and types: [synchronize] together cover every head: got ['TRIGGER'] exit 1, want ['DUPLICATE'] exit 1 FAIL scalar types: opened and synchronize cover every head: got ['TRIGGER'] exit 1, want ['DUPLICATE'] exit 1 FAIL types: []: got [] exit 0, want ['TRIGGER'] exit 1 FAIL two pull_request_target workflows: got ['ORPHANED'] exit 1, want ['DUPLICATE', 'TARGET-ONLY'] exit 1 FAIL other pull_request_target job with a different name: got ['ORPHANED'] exit 1, want ['TARGET-ONLY'] exit 1 FAIL other pull_request matrix job is refused: got ['INCOMPLETE'] exit 2, want ['INCOMPLETE', 'INCOMPLETE'] exit 2 FAIL other pull_request filter it cannot read is refused: got ['INCOMPLETE'] exit 2, want ['INCOMPLETE', 'INCOMPLETE'] exit 2 FAIL allowlisted first producer still pairs: got [] exit 0, want ['DUPLICATE'] exit 1 FAIL allowlisted first pull_request_target producer still pairs: got ['STALE-ALLOW'] exit 1, want ['DUPLICATE'] exit 1 FAIL three producers, first allowlisted: got [] exit 0, want ['DUPLICATE', 'DUPLICATE', 'DUPLICATE'] exit 1 FAIL every producer allowlisted still pairs: got ['ORPHANED'] exit 1, want ['DUPLICATE'] exit 1 FAIL ALLOW does not accept target-only: got ['ORPHANED', 'STALE-ALLOW'] exit 1, want ['TARGET-ONLY'] exit 1 FAIL ALLOW on a required job keeps its own push DUPLICATE: got ['ORPHANED', 'STALE-ALLOW'] exit 1, want ['DUPLICATE'] exit 1 FAIL ALLOW on a required job keeps its CONDITIONAL: got ['ORPHANED', 'STALE-ALLOW'] exit 1, want ['CONDITIONAL', 'TARGET-ONLY'] exit 1 FAIL ALLOW on a required pull_request job: target-only needs every producer: got ['ORPHANED'] exit 1, want ['DUPLICATE'] exit 1 FAIL TARGET_OK accepts target-only: got ['CRASH TypeError'] exit -1, want [] exit 0 FAIL TARGET_OK keeps CONDITIONAL: got ['CRASH TypeError'] exit -1, want ['CONDITIONAL'] exit 1 FAIL TARGET_OK for another context accepts nothing: got ['CRASH TypeError'] exit -1, want ['TARGET-ONLY'] exit 1 FAIL pull_request_target and push in one workflow: got ['ORPHANED'] exit 1, want ['DUPLICATE'] exit 1 FAIL pull_request_target, and push in another workflow: got ['ORPHANED'] exit 1, want ['DUPLICATE'] exit 1 FAIL pull_request_target, push to the default branch only elsewhere: got ['ORPHANED'] exit 1, want ['TARGET-ONLY'] exit 1 selftest: 185 of 225 as expected Some new cases pass on main, and the sweep pins them: - the negative controls: - another `pull_request` filtered to `develop`; - another `pull_request_target` with `branches-ignore: [main]`; - a `pull_request` job with a different name; - a `merge_group` job of the same name; - a `disabled_manually` workflow; - two `disabled_inactivity` workflows, one `pull_request_target` and one `push`; - `pull_request_target` into another branch, which is still ORPHANED; - a `disabled_inactivity` workflow that produces nothing, which is ORPHANED; - a path-filtered job whose types include opened and synchronize, which gives PATH-FILTER only; - the undecided filter with a different name; - an allowlisted matrix job that gates nothing, which is not refused. At this head: `selftest: 225 of 225 as expected`. **The `disabled_inactivity` gap the issue names.** I mutated main's `if state != "active"` to `if state == "disabled_manually"`. Main's self-test then gives `selftest: 173 of 173 as expected`, exit 0, so the mutant survives there. On this branch, that mutant and the same one in `producers()` are `caught`. **Mutation sweep.** Against main, 128 mutants become 158: 33 added and 3 removed, all three replaced by mutants on the rewritten code. 8 keep their label but have new patterns, because the code they target was rewritten or renamed: the four push-doubt mutants, "push producers not collected", "path filter ignored", "restrictive types ignored" and "types check inverted". The lines for the new and rewritten mutants on the pull request rules: target sha256 before: 7e638faf8fc1126d963195568b71f7311a76702ff33f47cadf49c407ca4dde04 ... caught merge_group counted as a pr event caught pull_request_target not a pr event caught pr producers not collected caught two-event workflow not flagged caught pr branch filter ignored elsewhere caught own undecided pr refused again caught undecided pr refused for any context caught unknown-name pr job refused only by name caught pr doubts never refused caught only disabled_manually excluded caught only disabled_manually excluded from producers caught path filter ignored caught restrictive types ignored caught types check inverted caught filtered event counts as covering caught covering ignored caught default types do not cover caught blocks ignored for called jobs caught every block named a path filter caught types with opened and synchronize counted as restricted caught scalar types not wrapped caught pr pairs never reported caught pr pairs only between the first two caught allowlisted workflow dropped from pr pairs caught pr events not recorded caught target-only never reported caught target-only with another event too caught allowlisted job skips every check caught allow does not silence ungated caught allowlisted unresolved job refused caught target_ok ignored caught target_ok matched by repo only caught push producer does not clear target-only caught types: [] reads as unrestricted caught coverage needs only one default type caught pairs skipped when every producer is allowlisted target sha256 after: 7e638faf8fc1126d963195568b71f7311a76702ff33f47cadf49c407ca4dde04 158 mutants: 157 caught, 0 crashed the self-test, 1 hung past 30s, 0 survived, 0 broken The one `hung` is #19's `unclosed [ accepted`, which loops forever by construction and is counted red. Every other line is `caught`. The hash is that of the committed file. No CI runs the self-test or the sweep; the daily audit workflow runs the audit only. ## Docs - **AGENTS.md §3.2** now has five items. - The two-producer bullet covers a second producer on `pull_request` or `pull_request_target`, and one workflow on both, whether or not `ALLOW` names either job. - A new TARGET-ONLY bullet says why such a check may be unable to fail. It cites the docs source, says that the default branch of the base repository's workflow and code are what run, and says that `TARGET_OK` accepts one and `ALLOW` does not. - **README** names the new producer, TARGET-ONLY and `TARGET_OK`. - **STATE.md**: - the "still does not see" paragraph is replaced by what the audit now sees, with the evidence above, the doc source, and TARGET-ONLY; - the live figure says "pull request jobs"; - §6's .github#10 entry no longer lists `pull_request_target` among events that never report on the PR, and says why; - §6 gains this gap; - a new "still does not see" paragraph records what this leaves out. The audit does not read `pull_request_review`, `deployment` or `deployment_status` as producers, although the same page lists them as eligible. Nor does it flag two jobs of one workflow with the same context. `evaluate` returns no finding for either. These are left to a separate issue. ## Review An independent review of the first head found one HIGH and two MEDIUMs, plus some LOWs. This head addresses each one. - **H1, fixed.** `ALLOW` on the first-by-path producer hid the pull request DUPLICATE, and the audit exited 0. Pairs are now computed from the producer list, independently of `ALLOW`. The reviewer's A4, A4c and A5b are now cases, and all three are red on main. - **M1, fixed.** Paths and types were decided separately across events. They are now decided per event, a covered head clears the blocks, and each finding names its event. A3, A3b and A3c are now cases. - **M2, fixed.** TARGET-ONLY is added; see the second round for how it is accepted. - **LOW, fixed.** - The summary label now says "pull request jobs". - The docstring and AGENTS.md say "the default branch of the base repository". - Ruleset acceptance is marked as the documentation's statement, with the source path and lines cited. - This body cites the source path rather than the redirect URL. The reviewer's harness against this head: - **`attack.py`** gives the following: | cases | result | |---|---| | A3, A3b | DUPLICATE, plus PATH-FILTER and TRIGGER, each naming its event | | A3c | DUPLICATE only | | A4, A4b, A4c, A5, A5b | every pair, exit 1 | | A2b, A8 | TARGET-ONLY | | A9 | PATH-FILTER and TARGET-ONLY | - **`mymut.py`**: - R4, R6, R7, R8, R11, R12, R14 and R15 are caught. - R1 survives. It drops the "(… excludes 'main')" note from an ORPHANED message. The self-test checks finding kinds and exit codes, not wording, and pinning wording would keep a wrong message alive rather than catch it, so this survivor is accepted. - R2, R3, R5, R9, R10 and R13 are BROKEN, because the code they targeted was rewritten. I ported them to the new code and ran them under the same hashing. R3', R5', R9', R10' and R13' are caught. R3' and R10' were caught only after adding the two cases above; their sweep mutants are "types with opened and synchronize counted as restricted" and "scalar types not wrapped". R2' survives for the same reason as R1: it changes only the types list printed in a message. **Second round**, at `a85060d`. H1 and M1 were confirmed fixed, and the reviewer re-derived every figure. One new HIGH, one MEDIUM and one LOW came up; this head addresses each one. - **HIGH, fixed.** Accepting TARGET-ONLY through `ALLOW` also silenced every other finding on that required job, because allowlisted jobs skipped the whole loop. `ALLOW` now silences UNGATED only, and the new `TARGET_OK` accepts TARGET-ONLY only. The reviewer's B3b and B4 are now cases, as is "TARGET_OK keeps CONDITIONAL". `attack2.py` at this head: every one of B1 to B9 exits 1. - B3b, B3c, B4, B5, B6 and B7 now report the job's DUPLICATE, CONDITIONAL, TRIGGER or PATH-FILTER. - B8 gives DUPLICATE without TARGET-ONLY. - B3d exits 1, not 2: its `ma?n` push filter is settled, not undecided. `?` makes the `a` optional, so the filter admits `mn`, a witness branch, and the result is a DUPLICATE. Any finding gives exit 1 in any case. - **MEDIUM, fixed.** - `types: []` and a lone `types: [opened]` are now cases that give TRIGGER. "Every producer allowlisted still pairs" is a case too. - S11 and S12 are in the sweep as "types: [] reads as unrestricted" and "coverage needs only one default type", and S14 as "pairs skipped when every producer is allowlisted". All are caught. - The reviewer's literal S14 (`len(pr_events_of.get(context, ())) == 0`) is now an equivalent mutant: allowlisted jobs record their events, so the condition never holds for a context that has a producer. - **LOW, fixed.** A `push` producer clears TARGET-ONLY, as above, and the message is reworded. `mymut2.py` at this head: - Caught: S1, S2, S7–S12. - BROKEN, because their code was rewritten: S4, S5, S6. Their analogues in the sweep are caught: "target-only with another event too", "target-only never reported", and "allowlisted job skips every check". - Survive: S3 and S13, which change only message text, and S14, which is equivalent as explained above. Not reviewed again since the second round. Closes #20 --------- Signed-off-by: Cameron Brooks <cambrooks3393@gmail.com>
This was referenced Oct 8, 2026
CameronBrooks11
added a commit
that referenced
this pull request
Oct 15, 2026
## What changes `STATE.md` brought to 2026-10-15. Every figure was re-measured today, in fresh clones of each public repository's `main`, through `just`, with a private uv cache (`UV_CACHE_DIR=<scratch> UV_LINK_MODE=copy`). Sections that no longer describe anything are removed rather than annotated. - **blockspec**: the binding spike verdict is pre-registration 3's run 2, FOUND, under the owner's ruling of 2026-10-14 (`spike/LOG.md` §34–§35), with the rules it overrides and the noise figures beside it. Pre-registration 2 and run 1 stay INCONCLUSIVE. Adjudications 0001 (CommonMark 0.31.2 block-level profile) and 0002 (the block model), with their addenda. The case tree (blockspec#36, #38), the reference implementation (blockspec#37), and the required checks `conformance` and `reference`. The "trending toward do not build" paragraph and the blockspec#19 timing paragraph are gone. - **rowspec**: refusal identifiers, the CSV profile and its own tree and gate, and xlsx formulas. The fixes and spec rulings of rowspec#104 to #139, including that #136 was closed on #128 and #137 by #139. The 22 open issues, including #142–#145 filed today, and the one open pull request, #141, which addresses rowspec#112, #113 and #132–#135. The GitHub Release object that now exists for `v0.3.0`. - **nodespec**: the mermaid spike ran and is closed; identity is parked. - **kindkit**: blockspec is a second consumer, at a different pin from rowspec's. - **.github**: the audit now counts 9 pull request jobs and 9 required checks, with 0 findings. Daily runs listed. - **What comes next**, owner decisions, setup, traps and the decided list updated to match. blockspec's own `README.md` and `spike/README.md` still say there is no verdict on its `main`; blockspec#39 (open) updates them, and `STATE.md` points at it. - **The spike's 432** is described as what it is: each FOUND cell's distinct count summed over the 35 cells, not a count of distinct units. The noise figures of pre-registration 3 §2.6 are quoted in full beside the verdict. - **Citations corrected**: - the three recorded rule-suite rejections by id, in place of a month query that cannot tell old rejections from none; - blockspec's prereg 2 pull requests as #15–#19 and #26–#28; - the `ambiguous-table` schema change as landing through blockspec#38 (`ebfc86a`); - the information-barrier decision in one place, no longer attributing blind judgement to `AGENTS.md` §2.1; - the merge-count search as `merged:2026-10-14..2026-10-15`. - **A setup note**: kindkit's figures need bytecode writing enabled. - **`AGENTS.md` and `profile/README.md`** (second commit): §1's table and the org page said blockspec was "spike, no verdict"; they now give draft 0 (decisions, cases, a reference) and the spike's FOUND, with nodespec's spike closed. §3.1's required-checks table adds blockspec's `reference`, which `gh api repos/kindspec/blockspec/rules/branches/main` lists beside `conformance`. - §6 gains one entry: 98 of rowspec's 116 refusing `parse` cases had asserted `refusal_contains: ""` (rowspec#61, per `CHANGELOG.md`). ## Measured All on 2026-10-15. The audit, here at `cb04c2c`: $ python3 -I scripts/check_required_checks.py --public-only 1 private repositories skipped (--public-only) 6 repositories, 8 workflows, 9 pull request jobs, 9 required checks: 0 finding(s), 0 not evaluated rowspec at `40e5647` (rowspec#139's merge): find conformance/cases -name expect.json | wc -l 483 find conformance/cases-csv -name expect.json | wc -l 85 just conform 0 failure(s) across 483 case(s) in the fixture tree just conform-alt 0 failure(s) across 483 case(s) in the fixture tree just conform-csv 0 failure(s) across 85 case(s) in the fixture tree just mutants 86 killed, 0 survived, 2 equivalent, 0 stale, 0 broken just mutants-csv 33 killed, 0 survived, 2 equivalent, 0 stale, 0 broken just test 418 passed, 2 skipped just test-xlsx 20 passed just test-libreoffice 1 passed just check 450 files already formatted / All checks passed! gh api repos/kindspec/rowspec/compare/v0.3.0...main --jq .ahead_by 28 gh issue list -R kindspec/rowspec --state open --limit 200 --json number --jq length 22 gh pr list -R kindspec/rowspec --state merged --search 'merged:2026-10-14..2026-10-15' --limit 200 --json number --jq length 27 grep -rl '"refusal":' conformance/cases --include=expect.json | wc -l 130 grep -rl '"refusal_contains"' conformance/cases-csv --include=expect.json | wc -l 43 blockspec at `d5355ab`: find conformance/cases -name expect.json | wc -l 237 just tree 237 cases: 160 accept, 77 refuse / 0 problem(s) just conform 0 failure(s) across 237 case(s) in the fixture tree just mutants 91 killed, 0 survived, 0 equivalent, 0 stale, 0 broken just test 8 passed just check 10 files already formatted / All checks passed! gh api repos/kindspec/blockspec/rules/branches/main 22723622 strict, conformance, reference python3 -I -c "import json,collections,sys; v=json.load(open(sys.argv[1])); print(v['overall'], dict(collections.Counter(c['verdict'] for c in v['cells'].values())))" <file> spike/results/prereg2/verdict.json INCONCLUSIVE {'NO VERDICT': 52, 'NOT FOUND': 4} spike/results/prereg3/verdict.json INCONCLUSIVE {'NO VERDICT': 52, 'NOT FOUND': 4} spike/results/prereg3/run2/verdict.json FOUND {'NO VERDICT': 16, 'FOUND': 35, 'NOT FOUND': 5} kindkit at `6ebd80d` (throwaway clone, node on PATH): just mutants 148 mutations, 148 caught, 0 survived, 0 broken just test 220 passed just check 28 files already formatted just cases 6 case(s), 0 invalid gh api repos/kindspec/kindkit/compare/v0.1.0...main --jq .ahead_by 8 with PYTHONDONTWRITEBYTECODE=1: uv run pytest -q 3 failed, 217 passed just mutants 148 mutations, 147 caught, 1 survived, 0 broken Rule suites (`gh api repos/kindspec/<repo>/rulesets/rule-suites/<id> --jq '.result+" "+.pushed_at'`): .github 4014197203 fail 2026-09-10T00:19:14-04:00 rowspec 4014189520 fail 2026-09-10T00:18:20-04:00 nodespec 4014318589 fail 2026-09-10T00:34:02-04:00 git show --stat --format= ebfc86a -- conformance/cases/case-body.schema.json (in blockspec) conformance/cases/case-body.schema.json | 3 ++- Issue and pull request lists: `gh issue list` and `gh pr list` with `--state all --limit 200` for rowspec, blockspec, nodespec, kindkit, research and .github. Every other figure in the file names its command beside it. ## Shown to fail n/a: a documentation change; no check is added or altered. ## Review None yet. --------- Signed-off-by: Cameron Brooks <cambrooks3393@gmail.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
scripts/check_required_checks.pyrefused every job that calls a reusable workflow. kindkit's and rowspec'skindjobs call kindkit'skind.yml, so the audit exited 2 on the real org with four INCOMPLETE lines, and the daily run in #12 would be red on every run. This teaches the script to expand those calls.What it does now
A job with
uses: ./.github/workflows/F.ymloruses: OWNER/REPO/.github/workflows/F.yml@REFproduces one context per called job,<caller job> / <called job>, each named by itsname:or else its id. That is the naming the existing required contexts already use (kind / conformance).pull_requesttrigger, branch filter, path filter andtypes:apply to every called job. The caller'sif:andneeds:chain make every called job CONDITIONAL; a skipped caller is reported under its own name only, so its<caller> / <called>contexts never appear and a required one holds the merge at Expected (the finding says so, rather than "reports success"). Each called job's ownif:andneeds:(within the called workflow) also make it CONDITIONAL, and a skipped called job does report as success.needs:a caller job is skippable unless every called context is required and none of the called jobs is itself skippable.REF(?ref=, URL-encoded) and recorded in the snapshot under a new per-repository"called"key, as text or as{"unreadable": why}.--loadtherefore reproduces the run offline.uses:form; a job withuses:together withruns-on:orsteps:, which GitHub rejects; a called workflow with noworkflow_calltrigger or that the parser refuses; and a called job that itself calls a workflow (no recursion), or is matrix-, expression- or block-scalar-named. A caller job with a matrix or an expression name stays refused, as before.pull_request jobsin the summary line now counts each called job, not the caller.Live, on the real org
The saved snapshot (
--save) holds one called workflow,kindspec/kindkit/.github/workflows/kind.yml@8e21b558c85d5612009f3a1d69a3a68bc0191637underrowspec; kindkit's own call is local.--loadof it gives the same line and exit 0.The expanded contexts are evaluated, not waved through. Edited copies of that snapshot:
Red first (first head, c5c303a)
26 self-test cases were added (23 snapshot, 3 collection through the fake API) before the implementation.
live()now round-trips each collected snapshot through JSON, so a collection case also checks what--savewrites is what--loadreads. The fake API answers a key containing?only for that exact path, which is how the collection case checks that the ref was used.Against main's logic with the new cases:
selftest: 73 of 85 as expected. The 12 failures are every case that expects a called workflow to be read: the local and remote expansions, the caller/called naming, UNGATED, ORPHANED and CONDITIONAL (callerif:, calledif:, calledneeds:), the threeneeds:a caller cases, and the fetch at the pinned ref. The 14 new refusal cases already pass on main, because main refuses every call; what pins them is the sweep below.At this head:
selftest: 85 of 85 as expected.Mutation sweep (first head)
python3 -I scripts/sweep_check_required_checks.py(it hashes the target, prints BROKEN for a pattern that does not match exactly once or a no-op mutation, and asserts the target's sha256 unchanged afterwards). 18 mutants were added for the new code, and one (reusable treated as plain) was retargeted. Three existing mutants first came out BROKEN because their patterns moved (conditional never reported,unrequired need ignored,jobs not a mapping accepted, the last now matching twice); they were retargeted to the same checks. Output at this head:(The other 48 lines are
caught.) sha256 ofscripts/check_required_checks.pybefore and after the sweep:2c4e9a42c2a0395b5b5ff00387570f2ed7a52198f5017af75b45b2ba88334cbdboth times. The claim is exactly that each named mutation makes--selftestreport a FAIL; the refusal for a missing local called file has a case but no mutant of its own.ruff format --checkandruff check(line length 100, the ruff kindkit locks) are clean onscripts/.After review (3bdbcfc, on top of a merge of main at 319a944)
Refused again:
uses:withruns-on:orsteps:. The first head let it through as a clean caller (main refused it). The two new cases, against the first head's logic:selftest: 89 of 91 as expected, both failing withgot [] exit 0.Pinned what was right but unchecked. New cases: two callers of one file at different refs, the second ref renaming the called job (UNGATED + ORPHANED); a path-filtered caller (PATH-FILTER); a caller on
types: [labeled](TRIGGER); and the summary's pull_request job count (3 for a plain job plus a caller of two jobs). The review's mutants for these all survived on the first head's self-test (its sha2562c4e9a42…unchanged before and after):and are caught at this head. They are now in the committed sweep as
call looked up without its ref,path filter ignored for called jobs,types ignored for called jobsandcalled jobs not counted, withuses with runs-on or steps acceptedfor the new refusal.called job conditions ignoredwas retargeted after its line changed (it first came out BROKEN).CONDITIONAL wording for a skipped caller, as described above. The finding kind is unchanged, and no case pins the wording.
At this head:
selftest: 91 of 91 as expected. Sweep, with the target's sha256fe662a80eb1b6ebac5e9b753f5ecc571ba04f2f1c4c8d96e48039eba35c5df09before and after:Live, at this head:
6 repositories, 6 workflows, 7 pull_request jobs, 7 required checks: 0 finding(s), 0 not evaluated, exit 0.Three of the review's mutants still survive and are not in the sweep: not URL-encoding the ref (no test ref contains a reserved character), accepting a fetched non-file (the fake API never returns a directory listing for a file path), and reading an unreadable call as empty text (an empty workflow is refused anyway, for its missing
on).Not covered: two producers for one required check
kindkit's
check.ymlruns on a barepushas well aspull_request, so on a PR head bothcheckandkind / conformancecome from two suites (kindspec/kindkit#26 fixes that, as kindspec/rowspec#67 did for rowspec). This audit does not flag it, before or after this change: only thepull_requesttrigger is read, and apushtrigger is ignored. With this change the expanded contexts would be visible to such a rule, but no rule exists. It would be a new finding class (a workflow producing a required context onpull_requestwhosepushtrigger also admits non-default branches), and is not added here.Refs #3. Refs #12, which goes green on its live run once this merges.