Repository navigation
ci: run the required-checks audit daily - #12
Merged
Merged
Conversation
Adds .github/workflows/required-checks-audit.yml: a daily schedule plus workflow_dispatch, contents: read, running `python3 -I scripts/check_required_checks.py --public-only` with the job's own token. No pull_request trigger, so it never runs as an ungated pull request job here. AGENTS.md and README.md point at it. Refs #3 (not closed here: the audit currently exits 2 on kindkit and rowspec, whose `kind` jobs call a reusable workflow the script does not expand).
CameronBrooks11
added a commit
that referenced
this pull request
Oct 8, 2026
`scripts/check_required_checks.py` refused every job that calls a reusable workflow. kindkit's and rowspec's `kind` jobs call kindkit's `kind.yml`, so the audit exited 2 on the real org with four INCOMPLETE lines, and the daily run in #12 would be red on every run. This teaches the script to expand those calls. ## What it does now A job with `uses: ./.github/workflows/F.yml` or `uses: OWNER/REPO/.github/workflows/F.yml@REF` produces one context per called job, `<caller job> / <called job>`, each named by its `name:` or else its id. That is the naming the existing required contexts already use (`kind / conformance`). - **Gating comes from the caller.** The caller workflow's `pull_request` trigger, branch filter, path filter and `types:` apply to every called job. The caller's `if:` and `needs:` chain make every called job CONDITIONAL; a skipped caller is reported under its own name only, so its `<caller> / <called>` contexts never appear and a required one holds the merge at *Expected* (the finding says so, rather than "reports success"). Each called job's own `if:` and `needs:` (within the called workflow) also make it CONDITIONAL, and a skipped called job does report as success. - **A job that `needs:` a caller job** is skippable unless every called context is required and none of the called jobs is itself skippable. - **Where the called file comes from.** A local call is read from the same default-branch snapshot as its caller. A call into another repository is fetched with the contents API at exactly `REF` (`?ref=`, URL-encoded) and recorded in the snapshot under a new per-repository `"called"` key, as text or as `{"unreadable": why}`. `--load` therefore reproduces the run offline. - **Still refused (exit 2):** a call into another repository that cannot be fetched at its ref, or that a snapshot does not hold (which includes every snapshot saved before this change; a local call still expands from an old snapshot, because it reads the workflow files the snapshot already holds); a local call whose file is not among the default branch's workflows; any other `uses:` form; a job with `uses:` together with `runs-on:` or `steps:`, which GitHub rejects; a called workflow with no `workflow_call` trigger or that the parser refuses; and a called job that itself calls a workflow (no recursion), or is matrix-, expression- or block-scalar-named. A caller job with a matrix or an expression name stays refused, as before. - `pull_request jobs` in the summary line now counts each called job, not the caller. ## Live, on the real org $ python3 -I scripts/check_required_checks.py --public-only # this branch 1 private repositories skipped (--public-only) 6 repositories, 6 workflows, 7 pull_request jobs, 7 required checks: 0 finding(s), 0 not evaluated exit 0 $ python3 -I scripts/check_required_checks.py --public-only # main, cb976f4 INCOMPLETE kindkit: .github/workflows/check.yml: job 'kind': not evaluated: calls a reusable workflow; contexts are '<caller> / <called job>' INCOMPLETE kindkit: required 'kind / conformance' not matched, and a job above was not read INCOMPLETE rowspec: .github/workflows/check.yml: job 'kind': not evaluated: calls a reusable workflow; contexts are '<caller> / <called job>' INCOMPLETE rowspec: required 'kind / conformance' not matched, and a job above was not read 1 private repositories skipped (--public-only) 6 repositories, 6 workflows, 7 pull_request jobs, 7 required checks: 0 finding(s), 4 not evaluated exit 2 The saved snapshot (`--save`) holds one called workflow, `kindspec/kindkit/.github/workflows/kind.yml@8e21b558c85d5612009f3a1d69a3a68bc0191637` under `rowspec`; kindkit's own call is local. `--load` of it gives the same line and exit 0. **The expanded contexts are evaluated, not waved through.** Edited copies of that snapshot: kindkit's 'kind / conformance' removed from required: UNGATED kindkit: .github/workflows/check.yml: job 'kind' (called job 'conformance') runs on pull requests; 'kind / conformance' is not a required check exit 1 rowspec's called kind.yml with the job renamed to `conform`: UNGATED rowspec: .github/workflows/check.yml: job 'kind' (called job 'conformance') runs on pull requests; 'kind / conform' is not a required check ORPHANED rowspec: required check 'kind / conformance' is produced by no job that reports on a pull request into 'main' exit 1 rowspec's "called" emptied: INCOMPLETE rowspec: .github/workflows/check.yml: job 'kind': not evaluated: calls kindspec/kindkit@8e21b55..., which the snapshot does not hold INCOMPLETE rowspec: required 'kind / conformance' not matched, and a job above was not read exit 2 ## Red first (first head, c5c303a) 26 self-test cases were added (23 snapshot, 3 collection through the fake API) before the implementation. `live()` now round-trips each collected snapshot through JSON, so a collection case also checks what `--save` writes is what `--load` reads. The fake API answers a key containing `?` only for that exact path, which is how the collection case checks that the ref was used. Against main's logic with the new cases: `selftest: 73 of 85 as expected`. The 12 failures are every case that expects a called workflow to be read: the local and remote expansions, the caller/called naming, UNGATED, ORPHANED and CONDITIONAL (caller `if:`, called `if:`, called `needs:`), the three `needs:` a caller cases, and the fetch at the pinned ref. The 14 new refusal cases already pass on main, because main refuses every call; what pins them is the sweep below. At this head: `selftest: 85 of 85 as expected`. ## Mutation sweep (first head) `python3 -I scripts/sweep_check_required_checks.py` (it hashes the target, prints BROKEN for a pattern that does not match exactly once or a no-op mutation, and asserts the target's sha256 unchanged afterwards). 18 mutants were added for the new code, and one (`reusable treated as plain`) was retargeted. Three existing mutants first came out **BROKEN** because their patterns moved (`conditional never reported`, `unrequired need ignored`, `jobs not a mapping accepted`, the last now matching twice); they were retargeted to the same checks. Output at this head: caught reusable treated as plain caught local call not read caught caller name not prefixed caught called job if ignored caught called job needs ignored caught caller conditions ignored for called jobs caught called job conditions ignored caught nested call accepted caught unread called job accepted caught no workflow_call accepted caught unreadable call accepted caught call missing from snapshot accepted caught unrecognised uses accepted caught unparseable call accepted caught call fetched without its ref caught calls not collected caught needed caller's called contexts ignored caught needed caller's called conditions ignored 66 mutants: 66 caught, 0 crashed the self-test, 0 survived, 0 broken (The other 48 lines are `caught`.) sha256 of `scripts/check_required_checks.py` before and after the sweep: `2c4e9a42c2a0395b5b5ff00387570f2ed7a52198f5017af75b45b2ba88334cbd` both times. The claim is exactly that each named mutation makes `--selftest` report a FAIL; the refusal for a missing local called file has a case but no mutant of its own. `ruff format --check` and `ruff check` (line length 100, the ruff kindkit locks) are clean on `scripts/`. ## After review (3bdbcfc, on top of a merge of main at 319a944) - **Refused again: `uses:` with `runs-on:` or `steps:`.** The first head let it through as a clean caller (main refused it). The two new cases, against the first head's logic: `selftest: 89 of 91 as expected`, both failing with `got [] exit 0`. - **Pinned what was right but unchecked.** New cases: two callers of one file at different refs, the second ref renaming the called job (UNGATED + ORPHANED); a path-filtered caller (PATH-FILTER); a caller on `types: [labeled]` (TRIGGER); and the summary's pull_request job count (3 for a plain job plus a caller of two jobs). The review's mutants for these all **survived** on the first head's self-test (its sha256 `2c4e9a42…` unchanged before and after): SURVIVED R1 called lookup ignores the ref SURVIVED R2 path filter not applied to called jobs SURVIVED R3 types not applied to called jobs SURVIVED R12 summary counts callers, not called jobs and are **caught** at this head. They are now in the committed sweep as `call looked up without its ref`, `path filter ignored for called jobs`, `types ignored for called jobs` and `called jobs not counted`, with `uses with runs-on or steps accepted` for the new refusal. `called job conditions ignored` was retargeted after its line changed (it first came out BROKEN). - **CONDITIONAL wording** for a skipped caller, as described above. The finding kind is unchanged, and no case pins the wording. At this head: `selftest: 91 of 91 as expected`. Sweep, with the target's sha256 `fe662a80eb1b6ebac5e9b753f5ecc571ba04f2f1c4c8d96e48039eba35c5df09` before and after: 71 mutants: 71 caught, 0 crashed the self-test, 0 survived, 0 broken Live, at this head: `6 repositories, 6 workflows, 7 pull_request jobs, 7 required checks: 0 finding(s), 0 not evaluated`, exit 0. Three of the review's mutants still survive and are not in the sweep: not URL-encoding the ref (no test ref contains a reserved character), accepting a fetched non-file (the fake API never returns a directory listing for a file path), and reading an unreadable call as empty text (an empty workflow is refused anyway, for its missing `on`). ## Not covered: two producers for one required check kindkit's `check.yml` runs on a bare `push` as well as `pull_request`, so on a PR head both `check` and `kind / conformance` come from two suites (kindspec/kindkit#26 fixes that, as kindspec/rowspec#67 did for rowspec). This audit does not flag it, before or after this change: only the `pull_request` trigger is read, and a `push` trigger is ignored. With this change the expanded contexts would be visible to such a rule, but no rule exists. It would be a new finding class (a workflow producing a required context on `pull_request` whose `push` trigger also admits non-default branches), and is not added here. Refs #3. Refs #12, which goes green on its live run once this merges. --------- Signed-off-by: Cameron Brooks <cambrooks3393@gmail.com>
…dit-followup Signed-off-by: Cameron Brooks <cambrooks3393@gmail.com>
The audit exits 0 on the public repositories now that it expands reusable workflow calls (#15), so the "teach it reusable workflows" item leaves the next-work list. Record kindkit's move to one producer per required check (kindkit#26), and name the second-producer case the audit still does not see. Refs #16 Signed-off-by: Cameron Brooks <cambrooks3393@gmail.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Runs
scripts/check_required_checks.pyon a schedule, the half of #3 its comment lists as still open.What it adds
.github/workflows/required-checks-audit.yml:on: schedule(daily, 06:17 UTC) andworkflow_dispatch;permissions: contents: read;audit, runningpython3 -I scripts/check_required_checks.py --public-onlywithGH_TOKENset to the job's own token.ghneeds a token in the environment even for endpoints that answer unauthenticated.No
pull_requesttrigger. With one,auditwould run on pull requests here and gate nothing, which is the UNGATED finding the script exists to report. Checked offline: addingpull_request:to this file in a saved snapshot makes the script printUNGATED .github: .github/workflows/required-checks-audit.yml: job 'audit' runs on pull requests; 'audit' is not a required checkand exit 1. Without it, the snapshot evaluates to the same result as live, so the file parses under the script's YAML subset and contributes no pull_request job.AGENTS.md§3.2 andREADME.mdnow point at it, and the §3.1 row for.githubsays its one workflow never runs on a pull request.Observed in Actions
This branch cannot be dispatched before it merges, so a throwaway branch
tmp/audit-probe(commit9ed1335, now deleted) carried this file with an addedpush:trigger for that branch, plus a second workflow running the same script on two--loadsnapshot fixtures.audit(this workflow, live, job token)6 repositories, 6 workflows, 7 pull_request jobs, 7 required checks: 0 finding(s), 4 not evaluated, andan unknown number of private repositories skipped (--public-only)clean(fixture: one PR job, required)1 repositories, ... 0 finding(s), 0 not evaluatedfinding(fixture: one PR job, not required)UNGATED demo: .github/workflows/ci.yml: job 'check' runs on pull requests; 'check' is not a required checkSo the job token reads all six public repositories, the same totals as a local run with a personal token (
python3 -I scripts/check_required_checks.py --public-only). The only difference is that the job token cannot see the org's private count, so it says "an unknown number" where a local run says1. The green and the exit-1 red come from fixtures, not from the live org.The
auditrow above ran before .github#15 merged, so it was red (exit 2) for the reason in the next section.The live audit is now green (since .github#15)
.github#15 (merged) expands a job that calls a reusable workflow into
<caller> / <called job>contexts, so the four INCOMPLETE lines from the first version of this PR are gone. This branch hasmainmerged in (f7b35c7). On this branch atbee8a3f, with a personal token:The live run reads
main, which does not have this workflow yet. Addingrequired-checks-audit.ymlto.githubin the saved snapshot (--save, then--loadthe edited copy) gives6 repositories, 7 workflows, 7 pull_request jobs, 7 required checks: 0 finding(s), 0 not evaluated, exit 0. So after merge the workflow parses, contributes no pull_request job, and does not report itself.STATE.mdis updated in this PR:.github);6ebd80dshowscheckandkind / conformancefrom onepushsuite, bothsuccess;pushruns on other branches;Issue 3 stays open until the first scheduled run is green
#3's acceptance, bullet by bullet:
kind / conformancecontexts.blockspec,researchand.githubare picked up automatically when they gain their first workflow. Not yet shown. The script lists every repository in the org on each run, so a new workflow is read without configuration. "Automatically" is the schedule, though, and a schedule only exists on the default branch, so nothing before merge can show it firing.Shown before merge:
Checked after merge:
workflow_dispatchrun onmaingoes green with the0 finding(s), 0 not evaluatedline;schedulerun (06:17 UTC) does the same, which shows the trigger fires.A closing keyword would close the issue at merge, before either run. This PR therefore uses
Refs, and #3 should be closed by hand, citing the first green scheduled run.One caveat for later, which does not change the above. GitHub disables a public repository's scheduled workflows after 60 days with no repository activity (GitHub's documented behaviour; not observed here). If
.githubgoes quiet for that long, the audit stops running.Refs #3. Refs #16.