Skip to content

feat: expand reusable workflow calls into their contexts - #15

Merged
CameronBrooks11 merged 3 commits into
mainfrom
feat/expand-called-workflows
Oct 8, 2026
Merged

CameronBrooks11 merged 3 commits into
mainfrom
feat/expand-called-workflows

Conversation

@CameronBrooks11

@CameronBrooks11 CameronBrooks11 commented Oct 8, 2026 •

Copy link
Copy Markdown
Member

scripts/check_required_checks.py refused every job that calls a reusable workflow. kindkit's and rowspec's kind jobs call kindkit's kind.yml, so the audit exited 2 on the real org with four INCOMPLETE lines, and the daily run in #12 would be red on every run. This teaches the script to expand those calls.

What it does now

A job with uses: ./.github/workflows/F.yml or uses: OWNER/REPO/.github/workflows/F.yml@REF produces one context per called job, <caller job> / <called job>, each named by its name: or else its id. That is the naming the existing required contexts already use (kind / conformance).

  • Gating comes from the caller. The caller workflow's pull_request trigger, branch filter, path filter and types: apply to every called job. The caller's if: and needs: chain make every called job CONDITIONAL; a skipped caller is reported under its own name only, so its <caller> / <called> contexts never appear and a required one holds the merge at Expected (the finding says so, rather than "reports success"). Each called job's own if: and needs: (within the called workflow) also make it CONDITIONAL, and a skipped called job does report as success.
  • A job that needs: a caller job is skippable unless every called context is required and none of the called jobs is itself skippable.
  • Where the called file comes from. A local call is read from the same default-branch snapshot as its caller. A call into another repository is fetched with the contents API at exactly REF (?ref=, URL-encoded) and recorded in the snapshot under a new per-repository "called" key, as text or as {"unreadable": why}. --load therefore reproduces the run offline.
  • Still refused (exit 2): a call into another repository that cannot be fetched at its ref, or that a snapshot does not hold (which includes every snapshot saved before this change; a local call still expands from an old snapshot, because it reads the workflow files the snapshot already holds); a local call whose file is not among the default branch's workflows; any other uses: form; a job with uses: together with runs-on: or steps:, which GitHub rejects; a called workflow with no workflow_call trigger or that the parser refuses; and a called job that itself calls a workflow (no recursion), or is matrix-, expression- or block-scalar-named. A caller job with a matrix or an expression name stays refused, as before.
  • pull_request jobs in the summary line now counts each called job, not the caller.

Live, on the real org

$ python3 -I scripts/check_required_checks.py --public-only        # this branch
1 private repositories skipped (--public-only)
6 repositories, 6 workflows, 7 pull_request jobs, 7 required checks: 0 finding(s), 0 not evaluated
exit 0

$ python3 -I scripts/check_required_checks.py --public-only        # main, cb976f4
INCOMPLETE   kindkit: .github/workflows/check.yml: job 'kind': not evaluated: calls a reusable workflow; contexts are '<caller> / <called job>'
INCOMPLETE   kindkit: required 'kind / conformance' not matched, and a job above was not read
INCOMPLETE   rowspec: .github/workflows/check.yml: job 'kind': not evaluated: calls a reusable workflow; contexts are '<caller> / <called job>'
INCOMPLETE   rowspec: required 'kind / conformance' not matched, and a job above was not read
1 private repositories skipped (--public-only)
6 repositories, 6 workflows, 7 pull_request jobs, 7 required checks: 0 finding(s), 4 not evaluated
exit 2

The saved snapshot (--save) holds one called workflow, kindspec/kindkit/.github/workflows/kind.yml@8e21b558c85d5612009f3a1d69a3a68bc0191637 under rowspec; kindkit's own call is local. --load of it gives the same line and exit 0.

The expanded contexts are evaluated, not waved through. Edited copies of that snapshot:

kindkit's 'kind / conformance' removed from required:
UNGATED      kindkit: .github/workflows/check.yml: job 'kind' (called job 'conformance') runs on pull requests; 'kind / conformance' is not a required check
exit 1

rowspec's called kind.yml with the job renamed to `conform`:
UNGATED      rowspec: .github/workflows/check.yml: job 'kind' (called job 'conformance') runs on pull requests; 'kind / conform' is not a required check
ORPHANED     rowspec: required check 'kind / conformance' is produced by no job that reports on a pull request into 'main'
exit 1

rowspec's "called" emptied:
INCOMPLETE   rowspec: .github/workflows/check.yml: job 'kind': not evaluated: calls kindspec/kindkit/.github/workflows/kind.yml@8e21b55..., which the snapshot does not hold
INCOMPLETE   rowspec: required 'kind / conformance' not matched, and a job above was not read
exit 2

Red first (first head, c5c303a)

26 self-test cases were added (23 snapshot, 3 collection through the fake API) before the implementation. live() now round-trips each collected snapshot through JSON, so a collection case also checks what --save writes is what --load reads. The fake API answers a key containing ? only for that exact path, which is how the collection case checks that the ref was used.

Against main's logic with the new cases: selftest: 73 of 85 as expected. The 12 failures are every case that expects a called workflow to be read: the local and remote expansions, the caller/called naming, UNGATED, ORPHANED and CONDITIONAL (caller if:, called if:, called needs:), the three needs: a caller cases, and the fetch at the pinned ref. The 14 new refusal cases already pass on main, because main refuses every call; what pins them is the sweep below.

At this head: selftest: 85 of 85 as expected.

Mutation sweep (first head)

python3 -I scripts/sweep_check_required_checks.py (it hashes the target, prints BROKEN for a pattern that does not match exactly once or a no-op mutation, and asserts the target's sha256 unchanged afterwards). 18 mutants were added for the new code, and one (reusable treated as plain) was retargeted. Three existing mutants first came out BROKEN because their patterns moved (conditional never reported, unrequired need ignored, jobs not a mapping accepted, the last now matching twice); they were retargeted to the same checks. Output at this head:

caught   reusable treated as plain
caught   local call not read
caught   caller name not prefixed
caught   called job if ignored
caught   called job needs ignored
caught   caller conditions ignored for called jobs
caught   called job conditions ignored
caught   nested call accepted
caught   unread called job accepted
caught   no workflow_call accepted
caught   unreadable call accepted
caught   call missing from snapshot accepted
caught   unrecognised uses accepted
caught   unparseable call accepted
caught   call fetched without its ref
caught   calls not collected
caught   needed caller's called contexts ignored
caught   needed caller's called conditions ignored
66 mutants: 66 caught, 0 crashed the self-test, 0 survived, 0 broken

(The other 48 lines are caught.) sha256 of scripts/check_required_checks.py before and after the sweep: 2c4e9a42c2a0395b5b5ff00387570f2ed7a52198f5017af75b45b2ba88334cbd both times. The claim is exactly that each named mutation makes --selftest report a FAIL; the refusal for a missing local called file has a case but no mutant of its own.

ruff format --check and ruff check (line length 100, the ruff kindkit locks) are clean on scripts/.

After review (3bdbcfc, on top of a merge of main at 319a944)

  • Refused again: uses: with runs-on: or steps:. The first head let it through as a clean caller (main refused it). The two new cases, against the first head's logic: selftest: 89 of 91 as expected, both failing with got [] exit 0.

  • Pinned what was right but unchecked. New cases: two callers of one file at different refs, the second ref renaming the called job (UNGATED + ORPHANED); a path-filtered caller (PATH-FILTER); a caller on types: [labeled] (TRIGGER); and the summary's pull_request job count (3 for a plain job plus a caller of two jobs). The review's mutants for these all survived on the first head's self-test (its sha256 2c4e9a42… unchanged before and after):

    SURVIVED   R1 called lookup ignores the ref
    SURVIVED   R2 path filter not applied to called jobs
    SURVIVED   R3 types not applied to called jobs
    SURVIVED   R12 summary counts callers, not called jobs
    

    and are caught at this head. They are now in the committed sweep as call looked up without its ref, path filter ignored for called jobs, types ignored for called jobs and called jobs not counted, with uses with runs-on or steps accepted for the new refusal. called job conditions ignored was retargeted after its line changed (it first came out BROKEN).

  • CONDITIONAL wording for a skipped caller, as described above. The finding kind is unchanged, and no case pins the wording.

At this head: selftest: 91 of 91 as expected. Sweep, with the target's sha256 fe662a80eb1b6ebac5e9b753f5ecc571ba04f2f1c4c8d96e48039eba35c5df09 before and after:

71 mutants: 71 caught, 0 crashed the self-test, 0 survived, 0 broken

Live, at this head: 6 repositories, 6 workflows, 7 pull_request jobs, 7 required checks: 0 finding(s), 0 not evaluated, exit 0.

Three of the review's mutants still survive and are not in the sweep: not URL-encoding the ref (no test ref contains a reserved character), accepting a fetched non-file (the fake API never returns a directory listing for a file path), and reading an unreadable call as empty text (an empty workflow is refused anyway, for its missing on).

Not covered: two producers for one required check

kindkit's check.yml runs on a bare push as well as pull_request, so on a PR head both check and kind / conformance come from two suites (kindspec/kindkit#26 fixes that, as kindspec/rowspec#67 did for rowspec). This audit does not flag it, before or after this change: only the pull_request trigger is read, and a push trigger is ignored. With this change the expanded contexts would be visible to such a rule, but no rule exists. It would be a new finding class (a workflow producing a required context on pull_request whose push trigger also admits non-default branches), and is not added here.

Refs #3. Refs #12, which goes green on its live run once this merges.

A job that calls a reusable workflow was refused, so kindkit's and
rowspec's `kind` jobs, and their required `kind / conformance`
contexts, were never evaluated and the audit exited 2 on the real org.

A caller job now produces one context per called job, named
'<caller job> / <called job>' from each one's name or id. The
caller's triggers, if: and needs: gate every called job, and each
called job's own if: and needs: gate it. A local call is read from
the default-branch snapshot; a call into another repository is
fetched at exactly its pinned ref and recorded under "called" in the
snapshot, so --load reproduces it offline.

Still not evaluated: a call that cannot be fetched or that the
snapshot lacks, any other uses: form, a called workflow with no
workflow_call trigger, and a called job that itself calls a workflow
or has a matrix, expression or block-scalar name.

Refs #3

Signed-off-by: Cameron Brooks <cambrooks3393@gmail.com>
…rkflows

Signed-off-by: Cameron Brooks <cambrooks3393@gmail.com>
A job carrying uses: together with runs-on: or steps: is a workflow
GitHub rejects. The expansion let it through as a clean caller; it is
now refused, as main refused it.

A skipped caller job leaves its '<caller> / <called>' contexts
unreported, so a required one holds the merge at Expected rather than
reporting success. The CONDITIONAL line for that case now says so.

New self-test cases pin what was correct but unchecked: each call is
read at its own ref, a caller's path filter and pull_request types
apply to its called jobs, and the summary counts called jobs. The
sweep gains a mutant for each, and for the new refusal.

Refs #3

Signed-off-by: Cameron Brooks <cambrooks3393@gmail.com>
@CameronBrooks11
CameronBrooks11 merged commit 6774187 into main Oct 8, 2026
2 checks passed
@CameronBrooks11
CameronBrooks11 deleted the feat/expand-called-workflows branch October 8, 2026 02:46
CameronBrooks11 added a commit that referenced this pull request Oct 8, 2026
The audit exits 0 on the public repositories now that it expands
reusable workflow calls (#15), so the "teach it reusable workflows"
item leaves the next-work list. Record kindkit's move to one producer
per required check (kindkit#26), and name the second-producer case the
audit still does not see.

Refs #16

Signed-off-by: Cameron Brooks <cambrooks3393@gmail.com>
CameronBrooks11 added a commit that referenced this pull request Oct 8, 2026
Runs `scripts/check_required_checks.py` on a schedule, the half of #3
its comment lists as still open.

## What it adds

`.github/workflows/required-checks-audit.yml`:

- `on: schedule` (daily, 06:17 UTC) and `workflow_dispatch`;
- `permissions: contents: read`;
- one job, `audit`, running `python3 -I scripts/check_required_checks.py
--public-only` with `GH_TOKEN` set to the job's own token. `gh` needs a
token in the environment even for endpoints that answer unauthenticated.

**No `pull_request` trigger.** With one, `audit` would run on pull
requests here and gate nothing, which is the UNGATED finding the script
exists to report. Checked offline: adding `pull_request:` to this file
in a saved snapshot makes the script print `UNGATED .github:
.github/workflows/required-checks-audit.yml: job 'audit' runs on pull
requests; 'audit' is not a required check` and exit 1. Without it, the
snapshot evaluates to the same result as live, so the file parses under
the script's YAML subset and contributes no pull_request job.

`AGENTS.md` §3.2 and `README.md` now point at it, and the §3.1 row for
`.github` says its one workflow never runs on a pull request.

## Observed in Actions

This branch cannot be dispatched before it merges, so a throwaway branch
`tmp/audit-probe` (commit `9ed1335`, now deleted) carried this file with
an added `push:` trigger for that branch, plus a second workflow running
the same script on two `--load` snapshot fixtures.

| run | job | result | what it printed |
|---|---|---|---|
| 37716690043 | `audit` (this workflow, live, job token) | **red, exit
2** | `6 repositories, 6 workflows, 7 pull_request jobs, 7 required
checks: 0 finding(s), 4 not evaluated`, and `an unknown number of
private repositories skipped (--public-only)` |
| 37716690040 | `clean` (fixture: one PR job, required) | **green, exit
0** | `1 repositories, ... 0 finding(s), 0 not evaluated` |
| 37716690040 | `finding` (fixture: one PR job, not required) | **red,
exit 1** | `UNGATED demo: .github/workflows/ci.yml: job 'check' runs on
pull requests; 'check' is not a required check` |

So the job token reads all six public repositories, the same totals as a
local run with a personal token (`python3 -I
scripts/check_required_checks.py --public-only`). The only difference is
that the job token cannot see the org's private count, so it says "an
unknown number" where a local run says `1`. The green and the exit-1 red
come from fixtures, not from the live org.

The `audit` row above ran before .github#15 merged, so it was red (exit
2) for the reason in the next section.

## The live audit is now green (since .github#15)

.github#15 (merged) expands a job that calls a reusable workflow into
`<caller> / <called job>` contexts, so the four INCOMPLETE lines from
the first version of this PR are gone. This branch has `main` merged in
(`f7b35c7`). On this branch at `bee8a3f`, with a personal token:

    $ python3 -I scripts/check_required_checks.py --public-only
    1 private repositories skipped (--public-only)
6 repositories, 6 workflows, 7 pull_request jobs, 7 required checks: 0
finding(s), 0 not evaluated
    exit 0

    $ python3 -I scripts/check_required_checks.py --selftest
    selftest: 91 of 91 as expected

The live run reads `main`, which does not have this workflow yet. Adding
`required-checks-audit.yml` to `.github` in the saved snapshot
(`--save`, then `--load` the edited copy) gives `6 repositories, 7
workflows, 7 pull_request jobs, 7 required checks: 0 finding(s), 0 not
evaluated`, exit 0. So after merge the workflow parses, contributes no
pull_request job, and does not report itself.

`STATE.md` is updated in this PR:
- the audit exits 0 on the org (§1, `.github`);
- kindkit#26 merged, so each required check has one producer. Merge
commit `6ebd80d` shows `check` and `kind / conformance` from one `push`
suite, both `success`;
- .github#16, which is still open, is the gap the audit does not see: a
required context also produced by `push` runs on other branches;
- "teach it reusable workflows" leaves the next-work list.

## Issue 3 stays open until the first scheduled run is green

#3's acceptance, bullet by bullet:

1. *A check exists that fails when a workflow job is not in the ruleset,
or a required context matches no job.* **Met** by #10 and extended by
#15.
2. *Its red state is demonstrated in both directions.* **Met.** #10's
body records UNGATED and ORPHANED on edited snapshots of the real org,
and #15's body records both again for the expanded `kind / conformance`
contexts.
3. *`blockspec`, `research` and `.github` are picked up automatically
when they gain their first workflow.* **Not yet shown.** The script
lists every repository in the org on each run, so a new workflow is read
without configuration. "Automatically" is the schedule, though, and a
schedule only exists on the default branch, so nothing before merge can
show it firing.

**Shown before merge:**
- the job runs in Actions with its own token and reads all six public
repositories (run 37716690043, on a throwaway branch);
- the script exits 0 on the live org, and 1 or 2 on fixtures;
- this workflow does not report itself.

**Checked after merge:**
- a `workflow_dispatch` run on `main` goes green with the `0 finding(s),
0 not evaluated` line;
- then the first `schedule` run (06:17 UTC) does the same, which shows
the trigger fires.

A closing keyword would close the issue at merge, before either run.
This PR therefore uses `Refs`, and #3 should be closed by hand, citing
the first green scheduled run.

One caveat for later, which does not change the above. GitHub disables a
public repository's scheduled workflows after 60 days with no repository
activity (GitHub's documented behaviour; not observed here). If
`.github` goes quiet for that long, the audit stops running.

Refs #3. Refs #16.

---------

Signed-off-by: Cameron Brooks <cambrooks3393@gmail.com>
CameronBrooks11 added a commit that referenced this pull request Oct 8, 2026
The DUPLICATE rule (#17) reads only a workflow's own `push` trigger.
Suppose workflow A runs `check` on `pull_request`, and workflow B runs a
job also named `check` on `push` with no branch filter. A pull request
head then carries `check` from two suites, and the merge gate takes
whichever run it finds (kindspec/rowspec#43). The audit passed that: the
issue's reproduction snapshot exits 0 on main.

## The rule

`push_producers()` walks every **active** workflow in a repository. It
keeps the ones whose `push` admits some branch other than the default,
decided by #17's `push_duplicates()`. For each of their jobs it records
the context the job produces:

- `name:` if present, else the job id;
- for a job that calls a reusable workflow, one `<caller> / <called>`
context per called job, through the same `expand()` the `pull_request`
side uses.

A job's `if:` is ignored here: a skipped job still reports its context,
which is why CONDITIONAL exists. A case pins this.

For every **required** context that a `pull_request` job produces in
workflow W, each push producer in a workflow **other than W** is a
finding.

**Same kind, DUPLICATE.** The hazard, its effect on the merge and the
fix (restrict `push` to the default branch or to tags) are the same as
#17's. AGENTS.md §3.2 treats them as one hazard, so a second finding
name would split one hazard across two labels. The message names both
workflows:

DUPLICATE kindkit: .github/workflows/check.yml: job 'check' is required,
and .github/workflows/push.yml: job 'check' also produces 'check' on
push to other branches (push has no branch filter), so a pull request
head can carry it from two suites and the merge takes either

**No double reporting.** W's own `push` is #17's finding, so the new
rule skips producers in W. The existing case `on: [push, pull_request]`
still expects exactly one DUPLICATE. The mutant that drops the
self-exclusion is caught there. If W has its own `push` and another
workflow also pushes `check`, that is two producers and gives two
findings. A case pins this too.

**Refusals (exit 2), only where they could change the answer:**

- B's `push` filter cannot be settled (for example `['*', '!*']`, or any
of #17's refusals, such as `!` in `branches-ignore` or `branches: []`),
and a job in B has the **same** context. If the job name differs,
nothing is refused.
- B's `push` admits another branch or cannot be settled, and one of its
jobs has a context that cannot be resolved: a matrix, an expression or
block-scalar name, or a call that cannot be read. That could be any
context, so every required `pull_request` context in another workflow is
refused.

A workflow B that cannot be parsed is already refused by the main pass.
An inactive B, or one whose `push` admits only the default branch or
only tags, produces nothing.

## Red first

The cases are in a new `cross` list. Five are positive: the same job id,
a matching `name:`, a job with `if:`, `branches-ignore: [main]`, and a
called workflow under `push`. There is also the own-push-plus-elsewhere
pair, and an unrequired context, which stays UNGATED only. The negative
controls are:

- `push` filtered to the default branch only;
- a tags-only `push`;
- a different job id;
- the same id with a different `name:`;
- a disabled workflow;
- a called workflow with other job names;
- an undecided filter with a different name;
- an expression name under a default-only `push`.

The refusals are an expression name, a matrix job, an undecided filter
with the same name, and a call that is not in the snapshot.

These were grafted onto main's logic, as #15 and #17 were: main's module
up to `# --- self-test`, then this branch's self-test section.

FAIL push job of the same id in another workflow: got [] exit 0, want
['DUPLICATE'] exit 1
FAIL push job named by name: in another workflow: got [] exit 0, want
['DUPLICATE'] exit 1
FAIL push job with an if: still reports: got [] exit 0, want
['DUPLICATE'] exit 1
FAIL push with branches-ignore in another workflow: got [] exit 0, want
['DUPLICATE'] exit 1
FAIL called workflow under push in another workflow: got [] exit 0, want
['DUPLICATE'] exit 1
FAIL own push and another workflow's push are two findings: got
['DUPLICATE'] exit 1, want ['DUPLICATE', 'DUPLICATE'] exit 1
FAIL push job with an expression name is refused: got [] exit 0, want
['INCOMPLETE'] exit 2
FAIL push matrix job is refused: got [] exit 0, want ['INCOMPLETE'] exit
2
FAIL undecided push filter, same name, is refused: got [] exit 0, want
['INCOMPLETE'] exit 2
FAIL unreadable called workflow under push is refused: got [] exit 0,
want ['INCOMPLETE'] exit 2
    selftest: 163 of 173 as expected

The other 9 new cases pass on main. They are the negative controls and
the UNGATED case, which main gets right because it has no rule. The
sweep pins them. At this head:

    selftest: 173 of 173 as expected

## Mutation sweep

Two changes to `scripts/sweep_check_required_checks.py`:

- **A 30 s timeout per mutant.** A mutant that runs past it is counted
as `hung`, which is red but counted apart from `caught` and `crashed`.
The self-test takes about 0.05 s.
- **The target's sha256 is printed before and after.** A pattern that
matches nothing, or a replacement that leaves the text unchanged, is
still BROKEN. I checked this by driving `main()` with one of each plus
one real mutant. It printed `BROKEN`, `BROKEN`, `caught`, and `exit 1`.

#17 left out the `end < 0` mutant on the unclosed-`[` check because it
loops forever. It is in now, and it comes out `hung`. The sweep counts
it red. No CI here runs the self-test or the sweep; if that mutant
shipped, the daily audit would hang on `ma[in` and fail on its 10-minute
job timeout. The `ma[in` case covers the refusal itself. 13 mutants were
added for the new rule:

target sha256 before:
7db249f493b73faf8bcb004c9a60b675a5c267803de6db92cc5245a9eb30ab71
    ...
    hung     unclosed [ accepted
    caught   cross-workflow duplicate never reported
    caught   own push counted again as another workflow
    caught   own undecided push refused again
    caught   undecided push refused for any context
    caught   unknown-name push job refused only by name
    caught   push doubts never refused
    caught   inactive workflow pushes
    caught   push filter ignored elsewhere
    caught   undecided push elsewhere passed
    caught   undecided push elsewhere is a finding
    caught   push calls not expanded
    caught   unknown push job accepted
    caught   push producers not collected
target sha256 after:
7db249f493b73faf8bcb004c9a60b675a5c267803de6db92cc5245a9eb30ab71
128 mutants: 127 caught, 0 crashed the self-test, 1 hung past 30s, 0
survived, 0 broken

All other lines are `caught`. Nothing survived.

`ruff format --check` and `ruff check` (kindkit's ruff 0.16.6, line
length 100) are clean on `scripts/`.

## Live

    $ python3 -I scripts/check_required_checks.py --public-only
    1 private repositories skipped (--public-only)
6 repositories, 7 workflows, 7 pull_request jobs, 7 required checks: 0
finding(s), 0 not evaluated
    exit 0

**The issue's reproduction.** I took a `--save` snapshot from that run
and added `.github/workflows/push.yml` to kindkit, with `on: push`, a
job `check` with one `run: "true"` step, and `workflow_state` `active`:

    $ python3 -I scripts/check_required_checks.py --load snap-push.json
DUPLICATE kindkit: .github/workflows/check.yml: job 'check' is required,
and .github/workflows/push.yml: job 'check' also produces 'check' on
push to other branches (push has no branch filter), so a pull request
head can carry it from two suites and the merge takes either
6 repositories, 8 workflows, 7 pull_request jobs, 7 required checks: 1
finding(s), 0 not evaluated
    exit 1

Main's logic on the same snapshot gives `8 workflows, 7 pull_request
jobs, 7 required checks: 0 finding(s), 0 not evaluated` and exits 0,
which matches the issue.

**A called workflow, on the same snapshot.** Here `push.yml` is instead
`on: push` with `kind: uses: ./.github/workflows/kind.yml`:

DUPLICATE kindkit: .github/workflows/check.yml: job 'kind' (called job
'conformance') is required, and .github/workflows/push.yml: job 'kind'
(called job 'conformance') also produces 'kind / conformance' on push to
other branches (push has no branch filter), so a pull request head can
carry it from two suites and the merge takes either
6 repositories, 8 workflows, 7 pull_request jobs, 7 required checks: 1
finding(s), 0 not evaluated
    exit 1

## Docs

- **AGENTS.md §3.2.** The audit bullet no longer excepts a second
producer in a different workflow. The two-producers bullet now says the
second producer can be a `push` job elsewhere, and that a `push` job
whose name the audit cannot resolve is refused.
- **README** adds "in the same workflow or another" to the audit's
description.
- **STATE.md:**
- the "still does not see" paragraph is replaced with what the audit now
sees;
  - the live figure's heading now names both rules;
  - §6 gains this gap.

## Not covered

**Two `pull_request` producers.** Two workflows that both run a job
named `check` on `pull_request` also give a required context two suites.
The audit reports nothing for that. I checked by calling `evaluate` on a
two-workflow snapshot, which returned `([], [], ...)`. That is a
separate rule with no issue yet. STATE.md now says so.

Refs #16, #17

Closes #18

Signed-off-by: Cameron Brooks <cambrooks3393@gmail.com>
CameronBrooks11 added a commit that referenced this pull request Oct 8, 2026
## What changes

The DUPLICATE rule (#17, #19) finds a second producer only on `push`. A
required context can also get one from the pull request itself:

- workflow A runs `check` on `pull_request`;
- workflow B runs a job also named `check` on `pull_request`, or on
`pull_request_target`.

The head then carries `check` from two suites, and the gate takes
whichever run it finds (kindspec/rowspec#43). The audit passed this. The
PR also corrects how `pull_request_target` is treated, and adds
TARGET-ONLY for a check that only it produces.

### DUPLICATE: a second producer on the pull request

`push_producers()` is now `producers(data, branch, admits)`.

- With `push_duplicates` it does what it did before.
- With the new `pr_admits` it collects every **active** workflow whose
`pull_request` or `pull_request_target` runs for a PR into the default
branch, and the context of each job. That context is `name:` or else the
id, or `<caller> / <called>` for a call. The same `expand()` and
refusals apply.

Three shapes are reported:

- **Two workflows.** For each required context, every pair of workflows
that produce it on a pull request is a DUPLICATE, reported once per
pair. The pairs come straight from the producer list, so `ALLOW` on
either job does not hide them. The message names both workflows and
their events:

DUPLICATE kindkit: required check 'check' is produced on a pull request
into 'main' by .github/workflows/check.yml: job 'check' (on
pull_request) and by .github/workflows/second.yml: job 'check' (on
pull_request_target), so a pull request head can carry it from two
suites and the merge takes either

- **One workflow on both** `pull_request` and `pull_request_target` is
also a DUPLICATE.
- **Several producers.** These findings do not overlap with #17/#19's
push rules; each one names a different pair of suites. Three workflows
give three findings. `push` and `pull_request` in another workflow are
two distinct suites from that workflow, so they give two findings, plus
that workflow's own #17 finding. Cases pin each count.

**Refused (exit 2), as #18 does:**

- a job in another active pull request workflow whose context cannot be
resolved (a matrix, an expression or block name, or a call not read),
since it could be any context;
- a branch filter on that workflow that cannot be read, when a job there
has the same context.

### `pull_request_target` reports on the PR head

Until now, a required check produced only by `pull_request_target` was
ORPHANED, on the grounds that "its check runs attach to the base
commit". That is wrong. I checked it against the issue's cpython run and
two more independent repositories:

| run | event | job's `head_sha` | PR head | listed in
`commits/<sha>/check-runs` | in the PR's `statusCheckRollup` |
|---|---|---|---|---|---|
| python/cpython 25066296099 | pull_request_target | `77e73c2` | PR
149109 (fork `warsaw/cpython`) | per the issue | not checked |
| electron/electron 37748122900 | pull_request_target | `a965781` | PR
54550 head `a965781` (same-repo branch, `compare main...sha` ahead 10,
behind 0) | yes, check run 113214421021 | yes, `event:
pull_request_target` |
| react/react 37730121633 | pull_request_target | `7af445b` | PR 37788
head `7af445b` (fork `abdulmajidcse/react`, ahead 1, behind 0) | yes,
check run 113157110710 | yes, `event: pull_request_target` |

The commands were:

- `gh api repos/<o>/<r>/actions/runs/<id>/jobs`
- `gh api repos/<o>/<r>/pulls/<n>`
- `gh api repos/<o>/<r>/commits/<sha>/check-runs`
- a GraphQL query of the last commit's `statusCheckRollup.contexts` with
`checkSuite.workflowRun.event`

react's head also carries `check_access` from two different
`pull_request_target` workflows, which is this issue's hazard in a real
repository.

GitHub's documentation says such checks count. The source is github/docs
`content/pull-requests/how-tos/merge-and-close-pull-requests/troubleshooting-required-status-checks.md`,
lines 55–64 at `60f9f34`:

> For checks created by workflow jobs to be evaluated for a pull
request, the workflow run must be triggered by one of these events:
> `push`, `pull_request`, `pull_request_review`, `pull_request_target`,
`deployment`, `deployment_status`
>
> For example, if a workflow is triggered by `workflow_dispatch` on a
pull request's head branch, checks reported by its jobs do not appear in
the pull request's checks section. Even if the checks pass for the head
commit, they do not satisfy a required status check in a branch ruleset.

That a ruleset accepts a `pull_request_target` check is the
documentation's statement; I have not watched a kindspec ruleset do it.
The same page answers the issue's "unverified routes":
`workflow_dispatch` on a PR branch does not count, and `create` is not
in the list.

So `pull_request_target` is now a pull request event:

- **ORPHANED.** A required check that `pull_request_target` alone
produces is no longer ORPHANED; it gets TARGET-ONLY instead (below).
- **UNGATED.** A `pull_request_target` job that is not required is
UNGATED, like a `pull_request` job. A job meant to gate nothing, such as
a labeller, goes in `ALLOW` with that reason.
- **CONDITIONAL, TRIGGER, PATH-FILTER** apply to it as well.

### TARGET-ONLY: a check that may be unable to fail

A `pull_request_target` job runs the workflow file from the default
branch of the base repository, with `GITHUB_SHA` that branch's last
commit. Unless it checks out the pull request, it runs on the default
branch's code, so it may be unable to fail because of the pull request.
That is the §2.2 pattern. The audit cannot tell whether a job checks out
the pull request, so:

- A required check whose **only** pull request producers are
`pull_request_target` jobs is TARGET-ONLY, exit 1.
- **`TARGET_OK`**, a new set of `(repo, context): reason`, accepts one,
and only that finding. The job's other findings (CONDITIONAL, TRIGGER,
PATH-FILTER, DUPLICATE) still fire.
- **`ALLOW` silences only UNGATED**, for a job that gates nothing. A
required job in `ALLOW` is checked like any other. An allowlisted job
also still counts as producing its contexts; before, a required context
from one was left ORPHANED. An allowlisted job whose context cannot be
resolved, such as a matrix labeller, is still not refused, as on main.
If it produced a required check, that check would be ORPHANED, so this
is not a silent pass.
- A check also produced on `pull_request` gets no TARGET-ONLY; the
DUPLICATE rule covers it instead.
- **A `push` producer clears TARGET-ONLY.** If a `push` run on a pull
request's branch also produces the check, in the same workflow or
another, TARGET-ONLY does not fire. That run tests the pull request's
code, so the check can fail because of the pull request, and the shape
is already a DUPLICATE. I chose this over rewording the message, because
a message saying the check "may be unable to fail" would be false there.
The message now says the check is "reported on a pull request only by
pull_request_target".

### PATH-FILTER and TRIGGER, per event

Each event that reaches the PR is blocked by a path filter, or by
`types:` that omit opened or synchronize.

- The blocks are findings only when the events that are not
path-filtered together miss opened or synchronize; otherwise every head
is reported.
- Each finding names its event: `its pull_request_target trigger is
path-filtered`, or `pull_request_target types ['labeled'] omit …`.
- So `paths` on one event with `types: [labeled]` on the other gives
both findings, and `types: [opened]` with `types: [synchronize]` gives
neither.
- One-event workflows behave as before, with one exception: `types: []`
now gives TRIGGER (types `[]`). Before, it gave nothing, because the
empty list read as false. GitHub does not document `types: []`, so the
audit reads it fail-closed, and the docstring says so.

## Measured

Live, with this branch:

$ python3 -I scripts/check_required_checks.py --public-only --save
snap.json
    1 private repositories skipped (--public-only)
6 repositories, 7 workflows, 7 pull request jobs, 7 required checks: 0
finding(s), 0 not evaluated
    exit 0

No TARGET-ONLY in the org. The summary now says "pull request jobs",
because the count includes `pull_request_target` jobs.

**The issue's reproduction.** I added `.github/workflows/second.yml` to
kindkit in that snapshot, with job `check` (`runs-on: ubuntu-latest`,
one `run: 'true'` step) and `workflow_state` `active`:

    == pull_request on branch
DUPLICATE kindkit: required check 'check' is produced on a pull request
into 'main' by .github/workflows/check.yml: job 'check' (on
pull_request) and by .github/workflows/second.yml: job 'check' (on
pull_request), so a pull request head can carry it from two suites and
the merge takes either
6 repositories, 8 workflows, 8 pull request jobs, 7 required checks: 1
finding(s), 0 not evaluated
    exit 1
    == pull_request on main
6 repositories, 8 workflows, 8 pull_request jobs, 7 required checks: 0
finding(s), 0 not evaluated
    exit 0
    == pull_request_target on branch
DUPLICATE kindkit: required check 'check' is produced on a pull request
into 'main' by .github/workflows/check.yml: job 'check' (on
pull_request) and by .github/workflows/second.yml: job 'check' (on
pull_request_target), so a pull request head can carry it from two
suites and the merge takes either
6 repositories, 8 workflows, 8 pull request jobs, 7 required checks: 1
finding(s), 0 not evaluated
    exit 1
    == pull_request_target on main
6 repositories, 8 workflows, 7 pull_request jobs, 7 required checks: 0
finding(s), 0 not evaluated
    exit 0

Main's figures match the issue's.

**pull_request_target only.** On the same snapshot I changed kindkit's
`check.yml` from `pull_request` to `pull_request_target`:

    == target-only on branch
TARGET-ONLY kindkit: required check 'check' is reported on a pull
request only by pull_request_target, which runs the default branch's
workflow on the default branch's code unless the job checks out the pull
request, so it may be unable to fail because of the pull request
TARGET-ONLY kindkit: required check 'kind / conformance' is reported on
a pull request only by pull_request_target, which runs the default
branch's workflow on the default branch's code unless the job checks out
the pull request, so it may be unable to fail because of the pull
request
6 repositories, 7 workflows, 7 pull request jobs, 7 required checks: 2
finding(s), 0 not evaluated
    exit 1
    == target-only on main
ORPHANED kindkit: required check 'check' is produced by no job that
reports on a pull request into 'main' (only pull_request_target, push)
ORPHANED kindkit: required check 'kind / conformance' is produced by no
job that reports on a pull request into 'main'
6 repositories, 7 workflows, 5 pull_request jobs, 7 required checks: 2
finding(s), 0 not evaluated
    exit 1

`ruff check` and `ruff format --check` (kindkit's ruff 0.16.6,
`--line-length 100`) are clean on `scripts/`.

## Shown to fail

**Red first.** I grafted this branch's self-test onto main's logic, as
#15, #17 and #19 did: main's module up to `# --- self-test`, then this
branch's self-test section. Main gives `selftest: 185 of 225 as
expected`, exit 1. The three `TARGET_OK` cases crash there, because
main's `evaluate` takes no such argument:

FAIL pull_request_target reports on the PR head, but only base code: got
['ORPHANED'] exit 1, want ['TARGET-ONLY'] exit 1
FAIL two pull_request workflows: got [] exit 0, want ['DUPLICATE'] exit
1
FAIL pull_request and pull_request_target workflows: got [] exit 0, want
['DUPLICATE'] exit 1
FAIL pull_request_target first by path, pull_request second: got [] exit
0, want ['DUPLICATE'] exit 1
FAIL pull_request_target job named by name:: got [] exit 0, want
['DUPLICATE'] exit 1
FAIL one workflow on pull_request and pull_request_target: got [] exit
0, want ['DUPLICATE'] exit 1
FAIL three workflows give one finding per pair: got [] exit 0, want
['DUPLICATE', 'DUPLICATE', 'DUPLICATE'] exit 1
FAIL push and pull_request in another workflow are two producers: got
['DUPLICATE', 'DUPLICATE'] exit 1, want ['DUPLICATE', 'DUPLICATE',
'DUPLICATE'] exit 1
FAIL called workflow under pull_request in another workflow: got [] exit
0, want ['DUPLICATE'] exit 1
FAIL unrequired pull_request_target job: got [] exit 0, want ['UNGATED']
exit 1
FAIL pull_request_target with an if:: got ['ORPHANED'] exit 1, want
['CONDITIONAL', 'TARGET-ONLY'] exit 1
FAIL pull_request_target types: [labeled]: got ['ORPHANED'] exit 1, want
['TRIGGER', 'TARGET-ONLY'] exit 1
FAIL pull_request_target path-filtered: got ['ORPHANED'] exit 1, want
['PATH-FILTER', 'TARGET-ONLY'] exit 1
FAIL pull_request path-filtered, pull_request_target not: got
['PATH-FILTER'] exit 1, want ['DUPLICATE'] exit 1
FAIL both events path-filtered: got ['PATH-FILTER'] exit 1, want
['DUPLICATE', 'PATH-FILTER', 'PATH-FILTER'] exit 1
FAIL pull_request types: [labeled], pull_request_target default: got
['TRIGGER'] exit 1, want ['DUPLICATE'] exit 1
FAIL both events types: [labeled]: got ['TRIGGER'] exit 1, want
['DUPLICATE', 'TRIGGER', 'TRIGGER'] exit 1
FAIL pull_request path-filtered, pull_request_target types: [labeled]:
got ['PATH-FILTER'] exit 1, want ['DUPLICATE', 'PATH-FILTER', 'TRIGGER']
exit 1
FAIL pull_request types: [labeled], pull_request_target path-filtered:
got ['TRIGGER'] exit 1, want ['DUPLICATE', 'TRIGGER', 'PATH-FILTER']
exit 1
FAIL types: [opened] and types: [synchronize] together cover every head:
got ['TRIGGER'] exit 1, want ['DUPLICATE'] exit 1
FAIL scalar types: opened and synchronize cover every head: got
['TRIGGER'] exit 1, want ['DUPLICATE'] exit 1
    FAIL types: []: got [] exit 0, want ['TRIGGER'] exit 1
FAIL two pull_request_target workflows: got ['ORPHANED'] exit 1, want
['DUPLICATE', 'TARGET-ONLY'] exit 1
FAIL other pull_request_target job with a different name: got
['ORPHANED'] exit 1, want ['TARGET-ONLY'] exit 1
FAIL other pull_request matrix job is refused: got ['INCOMPLETE'] exit
2, want ['INCOMPLETE', 'INCOMPLETE'] exit 2
FAIL other pull_request filter it cannot read is refused: got
['INCOMPLETE'] exit 2, want ['INCOMPLETE', 'INCOMPLETE'] exit 2
FAIL allowlisted first producer still pairs: got [] exit 0, want
['DUPLICATE'] exit 1
FAIL allowlisted first pull_request_target producer still pairs: got
['STALE-ALLOW'] exit 1, want ['DUPLICATE'] exit 1
FAIL three producers, first allowlisted: got [] exit 0, want
['DUPLICATE', 'DUPLICATE', 'DUPLICATE'] exit 1
FAIL every producer allowlisted still pairs: got ['ORPHANED'] exit 1,
want ['DUPLICATE'] exit 1
FAIL ALLOW does not accept target-only: got ['ORPHANED', 'STALE-ALLOW']
exit 1, want ['TARGET-ONLY'] exit 1
FAIL ALLOW on a required job keeps its own push DUPLICATE: got
['ORPHANED', 'STALE-ALLOW'] exit 1, want ['DUPLICATE'] exit 1
FAIL ALLOW on a required job keeps its CONDITIONAL: got ['ORPHANED',
'STALE-ALLOW'] exit 1, want ['CONDITIONAL', 'TARGET-ONLY'] exit 1
FAIL ALLOW on a required pull_request job: target-only needs every
producer: got ['ORPHANED'] exit 1, want ['DUPLICATE'] exit 1
FAIL TARGET_OK accepts target-only: got ['CRASH TypeError'] exit -1,
want [] exit 0
FAIL TARGET_OK keeps CONDITIONAL: got ['CRASH TypeError'] exit -1, want
['CONDITIONAL'] exit 1
FAIL TARGET_OK for another context accepts nothing: got ['CRASH
TypeError'] exit -1, want ['TARGET-ONLY'] exit 1
FAIL pull_request_target and push in one workflow: got ['ORPHANED'] exit
1, want ['DUPLICATE'] exit 1
FAIL pull_request_target, and push in another workflow: got ['ORPHANED']
exit 1, want ['DUPLICATE'] exit 1
FAIL pull_request_target, push to the default branch only elsewhere: got
['ORPHANED'] exit 1, want ['TARGET-ONLY'] exit 1
    selftest: 185 of 225 as expected

Some new cases pass on main, and the sweep pins them:

- the negative controls:
  - another `pull_request` filtered to `develop`;
  - another `pull_request_target` with `branches-ignore: [main]`;
  - a `pull_request` job with a different name;
  - a `merge_group` job of the same name;
  - a `disabled_manually` workflow;
- two `disabled_inactivity` workflows, one `pull_request_target` and one
`push`;
- `pull_request_target` into another branch, which is still ORPHANED;
- a `disabled_inactivity` workflow that produces nothing, which is
ORPHANED;
- a path-filtered job whose types include opened and synchronize, which
gives PATH-FILTER only;
- the undecided filter with a different name;
- an allowlisted matrix job that gates nothing, which is not refused.

At this head: `selftest: 225 of 225 as expected`.

**The `disabled_inactivity` gap the issue names.** I mutated main's `if
state != "active"` to `if state == "disabled_manually"`. Main's
self-test then gives `selftest: 173 of 173 as expected`, exit 0, so the
mutant survives there. On this branch, that mutant and the same one in
`producers()` are `caught`.

**Mutation sweep.** Against main, 128 mutants become 158: 33 added and 3
removed, all three replaced by mutants on the rewritten code. 8 keep
their label but have new patterns, because the code they target was
rewritten or renamed: the four push-doubt mutants, "push producers not
collected", "path filter ignored", "restrictive types ignored" and
"types check inverted". The lines for the new and rewritten mutants on
the pull request rules:

target sha256 before:
7e638faf8fc1126d963195568b71f7311a76702ff33f47cadf49c407ca4dde04
    ...
    caught   merge_group counted as a pr event
    caught   pull_request_target not a pr event
    caught   pr producers not collected
    caught   two-event workflow not flagged
    caught   pr branch filter ignored elsewhere
    caught   own undecided pr refused again
    caught   undecided pr refused for any context
    caught   unknown-name pr job refused only by name
    caught   pr doubts never refused
    caught   only disabled_manually excluded
    caught   only disabled_manually excluded from producers
    caught   path filter ignored
    caught   restrictive types ignored
    caught   types check inverted
    caught   filtered event counts as covering
    caught   covering ignored
    caught   default types do not cover
    caught   blocks ignored for called jobs
    caught   every block named a path filter
    caught   types with opened and synchronize counted as restricted
    caught   scalar types not wrapped
    caught   pr pairs never reported
    caught   pr pairs only between the first two
    caught   allowlisted workflow dropped from pr pairs
    caught   pr events not recorded
    caught   target-only never reported
    caught   target-only with another event too
    caught   allowlisted job skips every check
    caught   allow does not silence ungated
    caught   allowlisted unresolved job refused
    caught   target_ok ignored
    caught   target_ok matched by repo only
    caught   push producer does not clear target-only
    caught   types: [] reads as unrestricted
    caught   coverage needs only one default type
    caught   pairs skipped when every producer is allowlisted
target sha256 after:
7e638faf8fc1126d963195568b71f7311a76702ff33f47cadf49c407ca4dde04
158 mutants: 157 caught, 0 crashed the self-test, 1 hung past 30s, 0
survived, 0 broken

The one `hung` is #19's `unclosed [ accepted`, which loops forever by
construction and is counted red. Every other line is `caught`. The hash
is that of the committed file.

No CI runs the self-test or the sweep; the daily audit workflow runs the
audit only.

## Docs

- **AGENTS.md §3.2** now has five items.
- The two-producer bullet covers a second producer on `pull_request` or
`pull_request_target`, and one workflow on both, whether or not `ALLOW`
names either job.
- A new TARGET-ONLY bullet says why such a check may be unable to fail.
It cites the docs source, says that the default branch of the base
repository's workflow and code are what run, and says that `TARGET_OK`
accepts one and `ALLOW` does not.
- **README** names the new producer, TARGET-ONLY and `TARGET_OK`.
- **STATE.md**:
- the "still does not see" paragraph is replaced by what the audit now
sees, with the evidence above, the doc source, and TARGET-ONLY;
  - the live figure says "pull request jobs";
- §6's .github#10 entry no longer lists `pull_request_target` among
events that never report on the PR, and says why;
  - §6 gains this gap;
- a new "still does not see" paragraph records what this leaves out. The
audit does not read `pull_request_review`, `deployment` or
`deployment_status` as producers, although the same page lists them as
eligible. Nor does it flag two jobs of one workflow with the same
context. `evaluate` returns no finding for either. These are left to a
separate issue.

## Review

An independent review of the first head found one HIGH and two MEDIUMs,
plus some LOWs. This head addresses each one.

- **H1, fixed.** `ALLOW` on the first-by-path producer hid the pull
request DUPLICATE, and the audit exited 0. Pairs are now computed from
the producer list, independently of `ALLOW`. The reviewer's A4, A4c and
A5b are now cases, and all three are red on main.
- **M1, fixed.** Paths and types were decided separately across events.
They are now decided per event, a covered head clears the blocks, and
each finding names its event. A3, A3b and A3c are now cases.
- **M2, fixed.** TARGET-ONLY is added; see the second round for how it
is accepted.
- **LOW, fixed.**
  - The summary label now says "pull request jobs".
- The docstring and AGENTS.md say "the default branch of the base
repository".
- Ruleset acceptance is marked as the documentation's statement, with
the source path and lines cited.
  - This body cites the source path rather than the redirect URL.

The reviewer's harness against this head:

- **`attack.py`** gives the following:

  | cases | result |
  |---|---|
| A3, A3b | DUPLICATE, plus PATH-FILTER and TRIGGER, each naming its
event |
  | A3c | DUPLICATE only |
  | A4, A4b, A4c, A5, A5b | every pair, exit 1 |
  | A2b, A8 | TARGET-ONLY |
  | A9 | PATH-FILTER and TARGET-ONLY |

- **`mymut.py`**:
  - R4, R6, R7, R8, R11, R12, R14 and R15 are caught.
- R1 survives. It drops the "(… excludes 'main')" note from an ORPHANED
message. The self-test checks finding kinds and exit codes, not wording,
and pinning wording would keep a wrong message alive rather than catch
it, so this survivor is accepted.
- R2, R3, R5, R9, R10 and R13 are BROKEN, because the code they targeted
was rewritten. I ported them to the new code and ran them under the same
hashing. R3', R5', R9', R10' and R13' are caught. R3' and R10' were
caught only after adding the two cases above; their sweep mutants are
"types with opened and synchronize counted as restricted" and "scalar
types not wrapped". R2' survives for the same reason as R1: it changes
only the types list printed in a message.

**Second round**, at `a85060d`. H1 and M1 were confirmed fixed, and the
reviewer re-derived every figure. One new HIGH, one MEDIUM and one LOW
came up; this head addresses each one.

- **HIGH, fixed.** Accepting TARGET-ONLY through `ALLOW` also silenced
every other finding on that required job, because allowlisted jobs
skipped the whole loop. `ALLOW` now silences UNGATED only, and the new
`TARGET_OK` accepts TARGET-ONLY only. The reviewer's B3b and B4 are now
cases, as is "TARGET_OK keeps CONDITIONAL". `attack2.py` at this head:
every one of B1 to B9 exits 1.
- B3b, B3c, B4, B5, B6 and B7 now report the job's DUPLICATE,
CONDITIONAL, TRIGGER or PATH-FILTER.
  - B8 gives DUPLICATE without TARGET-ONLY.
- B3d exits 1, not 2: its `ma?n` push filter is settled, not undecided.
`?` makes the `a` optional, so the filter admits `mn`, a witness branch,
and the result is a DUPLICATE. Any finding gives exit 1 in any case.
- **MEDIUM, fixed.**
- `types: []` and a lone `types: [opened]` are now cases that give
TRIGGER. "Every producer allowlisted still pairs" is a case too.
- S11 and S12 are in the sweep as "types: [] reads as unrestricted" and
"coverage needs only one default type", and S14 as "pairs skipped when
every producer is allowlisted". All are caught.
- The reviewer's literal S14 (`len(pr_events_of.get(context, ())) == 0`)
is now an equivalent mutant: allowlisted jobs record their events, so
the condition never holds for a context that has a producer.
- **LOW, fixed.** A `push` producer clears TARGET-ONLY, as above, and
the message is reworded.

`mymut2.py` at this head:

- Caught: S1, S2, S7–S12.
- BROKEN, because their code was rewritten: S4, S5, S6. Their analogues
in the sweep are caught: "target-only with another event too",
"target-only never reported", and "allowlisted job skips every check".
- Survive: S3 and S13, which change only message text, and S14, which is
equivalent as explained above.

Not reviewed again since the second round.

Closes #20

---------

Signed-off-by: Cameron Brooks <cambrooks3393@gmail.com>
CameronBrooks11 added a commit that referenced this pull request Oct 15, 2026
## What changes

`STATE.md` brought to 2026-10-15. Every figure was re-measured today, in
fresh clones of each public repository's `main`, through `just`, with a
private uv cache (`UV_CACHE_DIR=<scratch> UV_LINK_MODE=copy`). Sections
that no longer describe anything are removed rather than annotated.

- **blockspec**: the binding spike verdict is pre-registration 3's run
2, FOUND, under the owner's ruling of 2026-10-14 (`spike/LOG.md`
§34–§35), with the rules it overrides and the noise figures beside it.
Pre-registration 2 and run 1 stay INCONCLUSIVE. Adjudications 0001
(CommonMark 0.31.2 block-level profile) and 0002 (the block model), with
their addenda. The case tree (blockspec#36, #38), the reference
implementation (blockspec#37), and the required checks `conformance` and
`reference`. The "trending toward do not build" paragraph and the
blockspec#19 timing paragraph are gone.
- **rowspec**: refusal identifiers, the CSV profile and its own tree and
gate, and xlsx formulas. The fixes and spec rulings of rowspec#104 to
#139, including that #136 was closed on #128 and #137 by #139. The 22
open issues, including #142–#145 filed today, and the one open pull
request, #141, which addresses rowspec#112, #113 and #132–#135. The
GitHub Release object that now exists for `v0.3.0`.
- **nodespec**: the mermaid spike ran and is closed; identity is parked.
- **kindkit**: blockspec is a second consumer, at a different pin from
rowspec's.
- **.github**: the audit now counts 9 pull request jobs and 9 required
checks, with 0 findings. Daily runs listed.
- **What comes next**, owner decisions, setup, traps and the decided
list updated to match. blockspec's own `README.md` and `spike/README.md`
still say there is no verdict on its `main`; blockspec#39 (open) updates
them, and `STATE.md` points at it.
- **The spike's 432** is described as what it is: each FOUND cell's
distinct count summed over the 35 cells, not a count of distinct units.
The noise figures of pre-registration 3 §2.6 are quoted in full beside
the verdict.
- **Citations corrected**:
- the three recorded rule-suite rejections by id, in place of a month
query that cannot tell old rejections from none;
  - blockspec's prereg 2 pull requests as #15–#19 and #26–#28;
- the `ambiguous-table` schema change as landing through blockspec#38
(`ebfc86a`);
- the information-barrier decision in one place, no longer attributing
blind judgement to `AGENTS.md` §2.1;
  - the merge-count search as `merged:2026-10-14..2026-10-15`.
- **A setup note**: kindkit's figures need bytecode writing enabled.
- **`AGENTS.md` and `profile/README.md`** (second commit): §1's table
and the org page said blockspec was "spike, no verdict"; they now give
draft 0 (decisions, cases, a reference) and the spike's FOUND, with
nodespec's spike closed. §3.1's required-checks table adds blockspec's
`reference`, which `gh api repos/kindspec/blockspec/rules/branches/main`
lists beside `conformance`.
- §6 gains one entry: 98 of rowspec's 116 refusing `parse` cases had
asserted `refusal_contains: ""` (rowspec#61, per `CHANGELOG.md`).

## Measured

All on 2026-10-15.

The audit, here at `cb04c2c`:

    $ python3 -I scripts/check_required_checks.py --public-only
    1 private repositories skipped (--public-only)
6 repositories, 8 workflows, 9 pull request jobs, 9 required checks: 0
finding(s), 0 not evaluated

rowspec at `40e5647` (rowspec#139's merge):

    find conformance/cases -name expect.json | wc -l       483
    find conformance/cases-csv -name expect.json | wc -l   85
just conform 0 failure(s) across 483 case(s) in the fixture tree
just conform-alt 0 failure(s) across 483 case(s) in the fixture tree
just conform-csv 0 failure(s) across 85 case(s) in the fixture tree
just mutants 86 killed, 0 survived, 2 equivalent, 0 stale, 0 broken
just mutants-csv 33 killed, 0 survived, 2 equivalent, 0 stale, 0 broken
    just test           418 passed, 2 skipped
    just test-xlsx      20 passed
    just test-libreoffice   1 passed
    just check          450 files already formatted / All checks passed!
gh api repos/kindspec/rowspec/compare/v0.3.0...main --jq .ahead_by 28
gh issue list -R kindspec/rowspec --state open --limit 200 --json number
--jq length 22
gh pr list -R kindspec/rowspec --state merged --search
'merged:2026-10-14..2026-10-15' --limit 200 --json number --jq length 27
grep -rl '"refusal":' conformance/cases --include=expect.json | wc -l
130
grep -rl '"refusal_contains"' conformance/cases-csv
--include=expect.json | wc -l 43

blockspec at `d5355ab`:

    find conformance/cases -name expect.json | wc -l   237
    just tree      237 cases: 160 accept, 77 refuse / 0 problem(s)
    just conform   0 failure(s) across 237 case(s) in the fixture tree
just mutants 91 killed, 0 survived, 0 equivalent, 0 stale, 0 broken
    just test      8 passed
    just check     10 files already formatted / All checks passed!
gh api repos/kindspec/blockspec/rules/branches/main 22723622 strict,
conformance, reference

python3 -I -c "import json,collections,sys;
v=json.load(open(sys.argv[1])); print(v['overall'],
dict(collections.Counter(c['verdict'] for c in v['cells'].values())))"
<file>
spike/results/prereg2/verdict.json INCONCLUSIVE {'NO VERDICT': 52, 'NOT
FOUND': 4}
spike/results/prereg3/verdict.json INCONCLUSIVE {'NO VERDICT': 52, 'NOT
FOUND': 4}
spike/results/prereg3/run2/verdict.json FOUND {'NO VERDICT': 16,
'FOUND': 35, 'NOT FOUND': 5}

kindkit at `6ebd80d` (throwaway clone, node on PATH):

    just mutants   148 mutations, 148 caught, 0 survived, 0 broken
    just test      220 passed
    just check     28 files already formatted
    just cases     6 case(s), 0 invalid
gh api repos/kindspec/kindkit/compare/v0.1.0...main --jq .ahead_by 8

    with PYTHONDONTWRITEBYTECODE=1:
    uv run pytest -q    3 failed, 217 passed
    just mutants        148 mutations, 147 caught, 1 survived, 0 broken

Rule suites (`gh api repos/kindspec/<repo>/rulesets/rule-suites/<id>
--jq '.result+" "+.pushed_at'`):

    .github  4014197203   fail 2026-09-10T00:19:14-04:00
    rowspec  4014189520   fail 2026-09-10T00:18:20-04:00
    nodespec 4014318589   fail 2026-09-10T00:34:02-04:00

git show --stat --format= ebfc86a --
conformance/cases/case-body.schema.json (in blockspec)
     conformance/cases/case-body.schema.json | 3 ++-

Issue and pull request lists: `gh issue list` and `gh pr list` with
`--state all --limit 200` for rowspec, blockspec, nodespec, kindkit,
research and .github. Every other figure in the file names its command
beside it.

## Shown to fail

n/a: a documentation change; no check is added or altered.

## Review

None yet.

---------

Signed-off-by: Cameron Brooks <cambrooks3393@gmail.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant