Skip to content

ci: run the required-checks audit daily - #12

Merged
CameronBrooks11 merged 3 commits into
mainfrom
ci/required-checks-audit
Oct 8, 2026
Merged

CameronBrooks11 merged 3 commits into
mainfrom
ci/required-checks-audit

Conversation

@CameronBrooks11

@CameronBrooks11 CameronBrooks11 commented Oct 8, 2026 •

Copy link
Copy Markdown
Member

Runs scripts/check_required_checks.py on a schedule, the half of #3 its comment lists as still open.

What it adds

.github/workflows/required-checks-audit.yml:

  • on: schedule (daily, 06:17 UTC) and workflow_dispatch;
  • permissions: contents: read;
  • one job, audit, running python3 -I scripts/check_required_checks.py --public-only with GH_TOKEN set to the job's own token. gh needs a token in the environment even for endpoints that answer unauthenticated.

No pull_request trigger. With one, audit would run on pull requests here and gate nothing, which is the UNGATED finding the script exists to report. Checked offline: adding pull_request: to this file in a saved snapshot makes the script print UNGATED .github: .github/workflows/required-checks-audit.yml: job 'audit' runs on pull requests; 'audit' is not a required check and exit 1. Without it, the snapshot evaluates to the same result as live, so the file parses under the script's YAML subset and contributes no pull_request job.

AGENTS.md §3.2 and README.md now point at it, and the §3.1 row for .github says its one workflow never runs on a pull request.

Observed in Actions

This branch cannot be dispatched before it merges, so a throwaway branch tmp/audit-probe (commit 9ed1335, now deleted) carried this file with an added push: trigger for that branch, plus a second workflow running the same script on two --load snapshot fixtures.

run job result what it printed
37716690043 audit (this workflow, live, job token) red, exit 2 6 repositories, 6 workflows, 7 pull_request jobs, 7 required checks: 0 finding(s), 4 not evaluated, and an unknown number of private repositories skipped (--public-only)
37716690040 clean (fixture: one PR job, required) green, exit 0 1 repositories, ... 0 finding(s), 0 not evaluated
37716690040 finding (fixture: one PR job, not required) red, exit 1 UNGATED demo: .github/workflows/ci.yml: job 'check' runs on pull requests; 'check' is not a required check

So the job token reads all six public repositories, the same totals as a local run with a personal token (python3 -I scripts/check_required_checks.py --public-only). The only difference is that the job token cannot see the org's private count, so it says "an unknown number" where a local run says 1. The green and the exit-1 red come from fixtures, not from the live org.

The audit row above ran before .github#15 merged, so it was red (exit 2) for the reason in the next section.

The live audit is now green (since .github#15)

.github#15 (merged) expands a job that calls a reusable workflow into <caller> / <called job> contexts, so the four INCOMPLETE lines from the first version of this PR are gone. This branch has main merged in (f7b35c7). On this branch at bee8a3f, with a personal token:

$ python3 -I scripts/check_required_checks.py --public-only
1 private repositories skipped (--public-only)
6 repositories, 6 workflows, 7 pull_request jobs, 7 required checks: 0 finding(s), 0 not evaluated
exit 0

$ python3 -I scripts/check_required_checks.py --selftest
selftest: 91 of 91 as expected

The live run reads main, which does not have this workflow yet. Adding required-checks-audit.yml to .github in the saved snapshot (--save, then --load the edited copy) gives 6 repositories, 7 workflows, 7 pull_request jobs, 7 required checks: 0 finding(s), 0 not evaluated, exit 0. So after merge the workflow parses, contributes no pull_request job, and does not report itself.

STATE.md is updated in this PR:

  • the audit exits 0 on the org (§1, .github);
  • kindkit#26 merged, so each required check has one producer. Merge commit 6ebd80d shows check and kind / conformance from one push suite, both success;
  • .github#16, which is still open, is the gap the audit does not see: a required context also produced by push runs on other branches;
  • "teach it reusable workflows" leaves the next-work list.

Issue 3 stays open until the first scheduled run is green

#3's acceptance, bullet by bullet:

  1. A check exists that fails when a workflow job is not in the ruleset, or a required context matches no job. Met by feat: add a check that every pull_request job is a required check #10 and extended by feat: expand reusable workflow calls into their contexts #15.
  2. Its red state is demonstrated in both directions. Met. feat: add a check that every pull_request job is a required check #10's body records UNGATED and ORPHANED on edited snapshots of the real org, and feat: expand reusable workflow calls into their contexts #15's body records both again for the expanded kind / conformance contexts.
  3. blockspec, research and .github are picked up automatically when they gain their first workflow. Not yet shown. The script lists every repository in the org on each run, so a new workflow is read without configuration. "Automatically" is the schedule, though, and a schedule only exists on the default branch, so nothing before merge can show it firing.

Shown before merge:

  • the job runs in Actions with its own token and reads all six public repositories (run 37716690043, on a throwaway branch);
  • the script exits 0 on the live org, and 1 or 2 on fixtures;
  • this workflow does not report itself.

Checked after merge:

  • a workflow_dispatch run on main goes green with the 0 finding(s), 0 not evaluated line;
  • then the first schedule run (06:17 UTC) does the same, which shows the trigger fires.

A closing keyword would close the issue at merge, before either run. This PR therefore uses Refs, and #3 should be closed by hand, citing the first green scheduled run.

One caveat for later, which does not change the above. GitHub disables a public repository's scheduled workflows after 60 days with no repository activity (GitHub's documented behaviour; not observed here). If .github goes quiet for that long, the audit stops running.

Refs #3. Refs #16.

Adds .github/workflows/required-checks-audit.yml: a daily schedule plus
workflow_dispatch, contents: read, running
`python3 -I scripts/check_required_checks.py --public-only` with the
job's own token. No pull_request trigger, so it never runs as an ungated
pull request job here. AGENTS.md and README.md point at it.

Refs #3 (not closed here: the audit currently exits 2 on kindkit and
rowspec, whose `kind` jobs call a reusable workflow the script does not
expand).
CameronBrooks11 added a commit that referenced this pull request Oct 8, 2026
`scripts/check_required_checks.py` refused every job that calls a
reusable workflow. kindkit's and rowspec's `kind` jobs call kindkit's
`kind.yml`, so the audit exited 2 on the real org with four INCOMPLETE
lines, and the daily run in #12 would be red on every run. This teaches
the script to expand those calls.

## What it does now

A job with `uses: ./.github/workflows/F.yml` or `uses:
OWNER/REPO/.github/workflows/F.yml@REF` produces one context per called
job, `<caller job> / <called job>`, each named by its `name:` or else
its id. That is the naming the existing required contexts already use
(`kind / conformance`).

- **Gating comes from the caller.** The caller workflow's `pull_request`
trigger, branch filter, path filter and `types:` apply to every called
job. The caller's `if:` and `needs:` chain make every called job
CONDITIONAL; a skipped caller is reported under its own name only, so
its `<caller> / <called>` contexts never appear and a required one holds
the merge at *Expected* (the finding says so, rather than "reports
success"). Each called job's own `if:` and `needs:` (within the called
workflow) also make it CONDITIONAL, and a skipped called job does report
as success.
- **A job that `needs:` a caller job** is skippable unless every called
context is required and none of the called jobs is itself skippable.
- **Where the called file comes from.** A local call is read from the
same default-branch snapshot as its caller. A call into another
repository is fetched with the contents API at exactly `REF` (`?ref=`,
URL-encoded) and recorded in the snapshot under a new per-repository
`"called"` key, as text or as `{"unreadable": why}`. `--load` therefore
reproduces the run offline.
- **Still refused (exit 2):** a call into another repository that cannot
be fetched at its ref, or that a snapshot does not hold (which includes
every snapshot saved before this change; a local call still expands from
an old snapshot, because it reads the workflow files the snapshot
already holds); a local call whose file is not among the default
branch's workflows; any other `uses:` form; a job with `uses:` together
with `runs-on:` or `steps:`, which GitHub rejects; a called workflow
with no `workflow_call` trigger or that the parser refuses; and a called
job that itself calls a workflow (no recursion), or is matrix-,
expression- or block-scalar-named. A caller job with a matrix or an
expression name stays refused, as before.
- `pull_request jobs` in the summary line now counts each called job,
not the caller.

## Live, on the real org

$ python3 -I scripts/check_required_checks.py --public-only # this
branch
    1 private repositories skipped (--public-only)
6 repositories, 6 workflows, 7 pull_request jobs, 7 required checks: 0
finding(s), 0 not evaluated
    exit 0

$ python3 -I scripts/check_required_checks.py --public-only # main,
cb976f4
INCOMPLETE kindkit: .github/workflows/check.yml: job 'kind': not
evaluated: calls a reusable workflow; contexts are '<caller> / <called
job>'
INCOMPLETE kindkit: required 'kind / conformance' not matched, and a job
above was not read
INCOMPLETE rowspec: .github/workflows/check.yml: job 'kind': not
evaluated: calls a reusable workflow; contexts are '<caller> / <called
job>'
INCOMPLETE rowspec: required 'kind / conformance' not matched, and a job
above was not read
    1 private repositories skipped (--public-only)
6 repositories, 6 workflows, 7 pull_request jobs, 7 required checks: 0
finding(s), 4 not evaluated
    exit 2

The saved snapshot (`--save`) holds one called workflow,
`kindspec/kindkit/.github/workflows/kind.yml@8e21b558c85d5612009f3a1d69a3a68bc0191637`
under `rowspec`; kindkit's own call is local. `--load` of it gives the
same line and exit 0.

**The expanded contexts are evaluated, not waved through.** Edited
copies of that snapshot:

    kindkit's 'kind / conformance' removed from required:
UNGATED kindkit: .github/workflows/check.yml: job 'kind' (called job
'conformance') runs on pull requests; 'kind / conformance' is not a
required check
    exit 1

    rowspec's called kind.yml with the job renamed to `conform`:
UNGATED rowspec: .github/workflows/check.yml: job 'kind' (called job
'conformance') runs on pull requests; 'kind / conform' is not a required
check
ORPHANED rowspec: required check 'kind / conformance' is produced by no
job that reports on a pull request into 'main'
    exit 1

    rowspec's "called" emptied:
INCOMPLETE rowspec: .github/workflows/check.yml: job 'kind': not
evaluated: calls kindspec/kindkit@8e21b55...,
which the snapshot does not hold
INCOMPLETE rowspec: required 'kind / conformance' not matched, and a job
above was not read
    exit 2

## Red first (first head, c5c303a)

26 self-test cases were added (23 snapshot, 3 collection through the
fake API) before the implementation. `live()` now round-trips each
collected snapshot through JSON, so a collection case also checks what
`--save` writes is what `--load` reads. The fake API answers a key
containing `?` only for that exact path, which is how the collection
case checks that the ref was used.

Against main's logic with the new cases: `selftest: 73 of 85 as
expected`. The 12 failures are every case that expects a called workflow
to be read: the local and remote expansions, the caller/called naming,
UNGATED, ORPHANED and CONDITIONAL (caller `if:`, called `if:`, called
`needs:`), the three `needs:` a caller cases, and the fetch at the
pinned ref. The 14 new refusal cases already pass on main, because main
refuses every call; what pins them is the sweep below.

At this head: `selftest: 85 of 85 as expected`.

## Mutation sweep (first head)

`python3 -I scripts/sweep_check_required_checks.py` (it hashes the
target, prints BROKEN for a pattern that does not match exactly once or
a no-op mutation, and asserts the target's sha256 unchanged afterwards).
18 mutants were added for the new code, and one (`reusable treated as
plain`) was retargeted. Three existing mutants first came out **BROKEN**
because their patterns moved (`conditional never reported`, `unrequired
need ignored`, `jobs not a mapping accepted`, the last now matching
twice); they were retargeted to the same checks. Output at this head:

    caught   reusable treated as plain
    caught   local call not read
    caught   caller name not prefixed
    caught   called job if ignored
    caught   called job needs ignored
    caught   caller conditions ignored for called jobs
    caught   called job conditions ignored
    caught   nested call accepted
    caught   unread called job accepted
    caught   no workflow_call accepted
    caught   unreadable call accepted
    caught   call missing from snapshot accepted
    caught   unrecognised uses accepted
    caught   unparseable call accepted
    caught   call fetched without its ref
    caught   calls not collected
    caught   needed caller's called contexts ignored
    caught   needed caller's called conditions ignored
    66 mutants: 66 caught, 0 crashed the self-test, 0 survived, 0 broken

(The other 48 lines are `caught`.) sha256 of
`scripts/check_required_checks.py` before and after the sweep:
`2c4e9a42c2a0395b5b5ff00387570f2ed7a52198f5017af75b45b2ba88334cbd` both
times. The claim is exactly that each named mutation makes `--selftest`
report a FAIL; the refusal for a missing local called file has a case
but no mutant of its own.

`ruff format --check` and `ruff check` (line length 100, the ruff
kindkit locks) are clean on `scripts/`.

## After review (3bdbcfc, on top of a merge of main at 319a944)

- **Refused again: `uses:` with `runs-on:` or `steps:`.** The first head
let it through as a clean caller (main refused it). The two new cases,
against the first head's logic: `selftest: 89 of 91 as expected`, both
failing with `got [] exit 0`.
- **Pinned what was right but unchecked.** New cases: two callers of one
file at different refs, the second ref renaming the called job (UNGATED
+ ORPHANED); a path-filtered caller (PATH-FILTER); a caller on `types:
[labeled]` (TRIGGER); and the summary's pull_request job count (3 for a
plain job plus a caller of two jobs). The review's mutants for these all
**survived** on the first head's self-test (its sha256 `2c4e9a42…`
unchanged before and after):

      SURVIVED   R1 called lookup ignores the ref
      SURVIVED   R2 path filter not applied to called jobs
      SURVIVED   R3 types not applied to called jobs
      SURVIVED   R12 summary counts callers, not called jobs

and are **caught** at this head. They are now in the committed sweep as
`call looked up without its ref`, `path filter ignored for called jobs`,
`types ignored for called jobs` and `called jobs not counted`, with
`uses with runs-on or steps accepted` for the new refusal. `called job
conditions ignored` was retargeted after its line changed (it first came
out BROKEN).
- **CONDITIONAL wording** for a skipped caller, as described above. The
finding kind is unchanged, and no case pins the wording.

At this head: `selftest: 91 of 91 as expected`. Sweep, with the target's
sha256
`fe662a80eb1b6ebac5e9b753f5ecc571ba04f2f1c4c8d96e48039eba35c5df09`
before and after:

    71 mutants: 71 caught, 0 crashed the self-test, 0 survived, 0 broken

Live, at this head: `6 repositories, 6 workflows, 7 pull_request jobs, 7
required checks: 0 finding(s), 0 not evaluated`, exit 0.

Three of the review's mutants still survive and are not in the sweep:
not URL-encoding the ref (no test ref contains a reserved character),
accepting a fetched non-file (the fake API never returns a directory
listing for a file path), and reading an unreadable call as empty text
(an empty workflow is refused anyway, for its missing `on`).

## Not covered: two producers for one required check

kindkit's `check.yml` runs on a bare `push` as well as `pull_request`,
so on a PR head both `check` and `kind / conformance` come from two
suites (kindspec/kindkit#26 fixes that, as kindspec/rowspec#67 did for
rowspec). This audit does not flag it, before or after this change: only
the `pull_request` trigger is read, and a `push` trigger is ignored.
With this change the expanded contexts would be visible to such a rule,
but no rule exists. It would be a new finding class (a workflow
producing a required context on `pull_request` whose `push` trigger also
admits non-default branches), and is not added here.

Refs #3. Refs #12, which goes green on its live run once this merges.

---------

Signed-off-by: Cameron Brooks <cambrooks3393@gmail.com>
…dit-followup

Signed-off-by: Cameron Brooks <cambrooks3393@gmail.com>
The audit exits 0 on the public repositories now that it expands
reusable workflow calls (#15), so the "teach it reusable workflows"
item leaves the next-work list. Record kindkit's move to one producer
per required check (kindkit#26), and name the second-producer case the
audit still does not see.

Refs #16

Signed-off-by: Cameron Brooks <cambrooks3393@gmail.com>
@CameronBrooks11
CameronBrooks11 merged commit 215b8c9 into main Oct 8, 2026
2 checks passed
@CameronBrooks11
CameronBrooks11 deleted the ci/required-checks-audit branch October 8, 2026 02:54
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant