Skip to content

feat: flag a required check that another workflow produces on push - #19

Merged
CameronBrooks11 merged 1 commit into
mainfrom
feat/audit-cross-workflow-duplicate
Oct 8, 2026
Merged

CameronBrooks11 merged 1 commit into
mainfrom
feat/audit-cross-workflow-duplicate

Conversation

@CameronBrooks11

@CameronBrooks11 CameronBrooks11 commented Oct 8, 2026 •

Copy link
Copy Markdown
Member

The DUPLICATE rule (#17) reads only a workflow's own push trigger. Suppose workflow A runs check on pull_request, and workflow B runs a job also named check on push with no branch filter. A pull request head then carries check from two suites, and the merge gate takes whichever run it finds (kindspec/rowspec#43). The audit passed that: the issue's reproduction snapshot exits 0 on main.

The rule

push_producers() walks every active workflow in a repository. It keeps the ones whose push admits some branch other than the default, decided by #17's push_duplicates(). For each of their jobs it records the context the job produces:

  • name: if present, else the job id;
  • for a job that calls a reusable workflow, one <caller> / <called> context per called job, through the same expand() the pull_request side uses.

A job's if: is ignored here: a skipped job still reports its context, which is why CONDITIONAL exists. A case pins this.

For every required context that a pull_request job produces in workflow W, each push producer in a workflow other than W is a finding.

Same kind, DUPLICATE. The hazard, its effect on the merge and the fix (restrict push to the default branch or to tags) are the same as #17's. AGENTS.md §3.2 treats them as one hazard, so a second finding name would split one hazard across two labels. The message names both workflows:

DUPLICATE    kindkit: .github/workflows/check.yml: job 'check' is required, and .github/workflows/push.yml: job 'check' also produces 'check' on push to other branches (push has no branch filter), so a pull request head can carry it from two suites and the merge takes either

No double reporting. W's own push is #17's finding, so the new rule skips producers in W. The existing case on: [push, pull_request] still expects exactly one DUPLICATE. The mutant that drops the self-exclusion is caught there. If W has its own push and another workflow also pushes check, that is two producers and gives two findings. A case pins this too.

Refusals (exit 2), only where they could change the answer:

  • B's push filter cannot be settled (for example ['*', '!*'], or any of feat: flag a required check that push also produces on a pr branch #17's refusals, such as ! in branches-ignore or branches: []), and a job in B has the same context. If the job name differs, nothing is refused.
  • B's push admits another branch or cannot be settled, and one of its jobs has a context that cannot be resolved: a matrix, an expression or block-scalar name, or a call that cannot be read. That could be any context, so every required pull_request context in another workflow is refused.

A workflow B that cannot be parsed is already refused by the main pass. An inactive B, or one whose push admits only the default branch or only tags, produces nothing.

Red first

The cases are in a new cross list. Five are positive: the same job id, a matching name:, a job with if:, branches-ignore: [main], and a called workflow under push. There is also the own-push-plus-elsewhere pair, and an unrequired context, which stays UNGATED only. The negative controls are:

  • push filtered to the default branch only;
  • a tags-only push;
  • a different job id;
  • the same id with a different name:;
  • a disabled workflow;
  • a called workflow with other job names;
  • an undecided filter with a different name;
  • an expression name under a default-only push.

The refusals are an expression name, a matrix job, an undecided filter with the same name, and a call that is not in the snapshot.

These were grafted onto main's logic, as #15 and #17 were: main's module up to # --- self-test, then this branch's self-test section.

FAIL push job of the same id in another workflow: got [] exit 0, want ['DUPLICATE'] exit 1
FAIL push job named by name: in another workflow: got [] exit 0, want ['DUPLICATE'] exit 1
FAIL push job with an if: still reports: got [] exit 0, want ['DUPLICATE'] exit 1
FAIL push with branches-ignore in another workflow: got [] exit 0, want ['DUPLICATE'] exit 1
FAIL called workflow under push in another workflow: got [] exit 0, want ['DUPLICATE'] exit 1
FAIL own push and another workflow's push are two findings: got ['DUPLICATE'] exit 1, want ['DUPLICATE', 'DUPLICATE'] exit 1
FAIL push job with an expression name is refused: got [] exit 0, want ['INCOMPLETE'] exit 2
FAIL push matrix job is refused: got [] exit 0, want ['INCOMPLETE'] exit 2
FAIL undecided push filter, same name, is refused: got [] exit 0, want ['INCOMPLETE'] exit 2
FAIL unreadable called workflow under push is refused: got [] exit 0, want ['INCOMPLETE'] exit 2
selftest: 163 of 173 as expected

The other 9 new cases pass on main. They are the negative controls and the UNGATED case, which main gets right because it has no rule. The sweep pins them. At this head:

selftest: 173 of 173 as expected

Mutation sweep

Two changes to scripts/sweep_check_required_checks.py:

  • A 30 s timeout per mutant. A mutant that runs past it is counted as hung, which is red but counted apart from caught and crashed. The self-test takes about 0.05 s.
  • The target's sha256 is printed before and after. A pattern that matches nothing, or a replacement that leaves the text unchanged, is still BROKEN. I checked this by driving main() with one of each plus one real mutant. It printed BROKEN, BROKEN, caught, and exit 1.

#17 left out the end < 0 mutant on the unclosed-[ check because it loops forever. It is in now, and it comes out hung. The sweep counts it red. No CI here runs the self-test or the sweep; if that mutant shipped, the daily audit would hang on ma[in and fail on its 10-minute job timeout. The ma[in case covers the refusal itself. 13 mutants were added for the new rule:

target sha256 before: 7db249f493b73faf8bcb004c9a60b675a5c267803de6db92cc5245a9eb30ab71
...
hung     unclosed [ accepted
caught   cross-workflow duplicate never reported
caught   own push counted again as another workflow
caught   own undecided push refused again
caught   undecided push refused for any context
caught   unknown-name push job refused only by name
caught   push doubts never refused
caught   inactive workflow pushes
caught   push filter ignored elsewhere
caught   undecided push elsewhere passed
caught   undecided push elsewhere is a finding
caught   push calls not expanded
caught   unknown push job accepted
caught   push producers not collected
target sha256 after:  7db249f493b73faf8bcb004c9a60b675a5c267803de6db92cc5245a9eb30ab71
128 mutants: 127 caught, 0 crashed the self-test, 1 hung past 30s, 0 survived, 0 broken

All other lines are caught. Nothing survived.

ruff format --check and ruff check (kindkit's ruff 0.16.6, line length 100) are clean on scripts/.

Live

$ python3 -I scripts/check_required_checks.py --public-only
1 private repositories skipped (--public-only)
6 repositories, 7 workflows, 7 pull_request jobs, 7 required checks: 0 finding(s), 0 not evaluated
exit 0

The issue's reproduction. I took a --save snapshot from that run and added .github/workflows/push.yml to kindkit, with on: push, a job check with one run: "true" step, and workflow_state active:

$ python3 -I scripts/check_required_checks.py --load snap-push.json
DUPLICATE    kindkit: .github/workflows/check.yml: job 'check' is required, and .github/workflows/push.yml: job 'check' also produces 'check' on push to other branches (push has no branch filter), so a pull request head can carry it from two suites and the merge takes either
6 repositories, 8 workflows, 7 pull_request jobs, 7 required checks: 1 finding(s), 0 not evaluated
exit 1

Main's logic on the same snapshot gives 8 workflows, 7 pull_request jobs, 7 required checks: 0 finding(s), 0 not evaluated and exits 0, which matches the issue.

A called workflow, on the same snapshot. Here push.yml is instead on: push with kind: uses: ./.github/workflows/kind.yml:

DUPLICATE    kindkit: .github/workflows/check.yml: job 'kind' (called job 'conformance') is required, and .github/workflows/push.yml: job 'kind' (called job 'conformance') also produces 'kind / conformance' on push to other branches (push has no branch filter), so a pull request head can carry it from two suites and the merge takes either
6 repositories, 8 workflows, 7 pull_request jobs, 7 required checks: 1 finding(s), 0 not evaluated
exit 1

Docs

  • AGENTS.md §3.2. The audit bullet no longer excepts a second producer in a different workflow. The two-producers bullet now says the second producer can be a push job elsewhere, and that a push job whose name the audit cannot resolve is refused.
  • README adds "in the same workflow or another" to the audit's description.
  • STATE.md:
    • the "still does not see" paragraph is replaced with what the audit now sees;
    • the live figure's heading now names both rules;
    • §6 gains this gap.

Not covered

Two pull_request producers. Two workflows that both run a job named check on pull_request also give a required context two suites. The audit reports nothing for that. I checked by calling evaluate on a two-workflow snapshot, which returned ([], [], ...). That is a separate rule with no issue yet. STATE.md now says so.

Refs #16, #17

Closes #18

The DUPLICATE rule read only a workflow's own push trigger. A job of
the same name in a different workflow that runs on push to a pull
request's branch gives the required context a second suite on that
head, and the audit passed it.

The audit now collects, across every active workflow, the contexts
produced on push to some branch other than the default, using the
same filter evaluation and refusals as the own-push rule, and
expanding called workflows into their '<caller> / <called>' contexts.
A required context produced on pull_request by one workflow and on
push by another is a DUPLICATE that names both. A push job whose
context cannot be resolved, or an undecided filter on a job of the
same context, is refused with exit 2.

The mutation sweep gains a per-mutant timeout and prints the target's
digest before and after.

Closes #18

Signed-off-by: Cameron Brooks <cambrooks3393@gmail.com>
@CameronBrooks11
CameronBrooks11 merged commit 2d43f6f into main Oct 8, 2026
3 checks passed
@CameronBrooks11
CameronBrooks11 deleted the feat/audit-cross-workflow-duplicate branch October 8, 2026 08:16
CameronBrooks11 added a commit that referenced this pull request Oct 8, 2026
## What changes

The DUPLICATE rule (#17, #19) finds a second producer only on `push`. A
required context can also get one from the pull request itself:

- workflow A runs `check` on `pull_request`;
- workflow B runs a job also named `check` on `pull_request`, or on
`pull_request_target`.

The head then carries `check` from two suites, and the gate takes
whichever run it finds (kindspec/rowspec#43). The audit passed this. The
PR also corrects how `pull_request_target` is treated, and adds
TARGET-ONLY for a check that only it produces.

### DUPLICATE: a second producer on the pull request

`push_producers()` is now `producers(data, branch, admits)`.

- With `push_duplicates` it does what it did before.
- With the new `pr_admits` it collects every **active** workflow whose
`pull_request` or `pull_request_target` runs for a PR into the default
branch, and the context of each job. That context is `name:` or else the
id, or `<caller> / <called>` for a call. The same `expand()` and
refusals apply.

Three shapes are reported:

- **Two workflows.** For each required context, every pair of workflows
that produce it on a pull request is a DUPLICATE, reported once per
pair. The pairs come straight from the producer list, so `ALLOW` on
either job does not hide them. The message names both workflows and
their events:

DUPLICATE kindkit: required check 'check' is produced on a pull request
into 'main' by .github/workflows/check.yml: job 'check' (on
pull_request) and by .github/workflows/second.yml: job 'check' (on
pull_request_target), so a pull request head can carry it from two
suites and the merge takes either

- **One workflow on both** `pull_request` and `pull_request_target` is
also a DUPLICATE.
- **Several producers.** These findings do not overlap with #17/#19's
push rules; each one names a different pair of suites. Three workflows
give three findings. `push` and `pull_request` in another workflow are
two distinct suites from that workflow, so they give two findings, plus
that workflow's own #17 finding. Cases pin each count.

**Refused (exit 2), as #18 does:**

- a job in another active pull request workflow whose context cannot be
resolved (a matrix, an expression or block name, or a call not read),
since it could be any context;
- a branch filter on that workflow that cannot be read, when a job there
has the same context.

### `pull_request_target` reports on the PR head

Until now, a required check produced only by `pull_request_target` was
ORPHANED, on the grounds that "its check runs attach to the base
commit". That is wrong. I checked it against the issue's cpython run and
two more independent repositories:

| run | event | job's `head_sha` | PR head | listed in
`commits/<sha>/check-runs` | in the PR's `statusCheckRollup` |
|---|---|---|---|---|---|
| python/cpython 25066296099 | pull_request_target | `77e73c2` | PR
149109 (fork `warsaw/cpython`) | per the issue | not checked |
| electron/electron 37748122900 | pull_request_target | `a965781` | PR
54550 head `a965781` (same-repo branch, `compare main...sha` ahead 10,
behind 0) | yes, check run 113214421021 | yes, `event:
pull_request_target` |
| react/react 37730121633 | pull_request_target | `7af445b` | PR 37788
head `7af445b` (fork `abdulmajidcse/react`, ahead 1, behind 0) | yes,
check run 113157110710 | yes, `event: pull_request_target` |

The commands were:

- `gh api repos/<o>/<r>/actions/runs/<id>/jobs`
- `gh api repos/<o>/<r>/pulls/<n>`
- `gh api repos/<o>/<r>/commits/<sha>/check-runs`
- a GraphQL query of the last commit's `statusCheckRollup.contexts` with
`checkSuite.workflowRun.event`

react's head also carries `check_access` from two different
`pull_request_target` workflows, which is this issue's hazard in a real
repository.

GitHub's documentation says such checks count. The source is github/docs
`content/pull-requests/how-tos/merge-and-close-pull-requests/troubleshooting-required-status-checks.md`,
lines 55–64 at `60f9f34`:

> For checks created by workflow jobs to be evaluated for a pull
request, the workflow run must be triggered by one of these events:
> `push`, `pull_request`, `pull_request_review`, `pull_request_target`,
`deployment`, `deployment_status`
>
> For example, if a workflow is triggered by `workflow_dispatch` on a
pull request's head branch, checks reported by its jobs do not appear in
the pull request's checks section. Even if the checks pass for the head
commit, they do not satisfy a required status check in a branch ruleset.

That a ruleset accepts a `pull_request_target` check is the
documentation's statement; I have not watched a kindspec ruleset do it.
The same page answers the issue's "unverified routes":
`workflow_dispatch` on a PR branch does not count, and `create` is not
in the list.

So `pull_request_target` is now a pull request event:

- **ORPHANED.** A required check that `pull_request_target` alone
produces is no longer ORPHANED; it gets TARGET-ONLY instead (below).
- **UNGATED.** A `pull_request_target` job that is not required is
UNGATED, like a `pull_request` job. A job meant to gate nothing, such as
a labeller, goes in `ALLOW` with that reason.
- **CONDITIONAL, TRIGGER, PATH-FILTER** apply to it as well.

### TARGET-ONLY: a check that may be unable to fail

A `pull_request_target` job runs the workflow file from the default
branch of the base repository, with `GITHUB_SHA` that branch's last
commit. Unless it checks out the pull request, it runs on the default
branch's code, so it may be unable to fail because of the pull request.
That is the §2.2 pattern. The audit cannot tell whether a job checks out
the pull request, so:

- A required check whose **only** pull request producers are
`pull_request_target` jobs is TARGET-ONLY, exit 1.
- **`TARGET_OK`**, a new set of `(repo, context): reason`, accepts one,
and only that finding. The job's other findings (CONDITIONAL, TRIGGER,
PATH-FILTER, DUPLICATE) still fire.
- **`ALLOW` silences only UNGATED**, for a job that gates nothing. A
required job in `ALLOW` is checked like any other. An allowlisted job
also still counts as producing its contexts; before, a required context
from one was left ORPHANED. An allowlisted job whose context cannot be
resolved, such as a matrix labeller, is still not refused, as on main.
If it produced a required check, that check would be ORPHANED, so this
is not a silent pass.
- A check also produced on `pull_request` gets no TARGET-ONLY; the
DUPLICATE rule covers it instead.
- **A `push` producer clears TARGET-ONLY.** If a `push` run on a pull
request's branch also produces the check, in the same workflow or
another, TARGET-ONLY does not fire. That run tests the pull request's
code, so the check can fail because of the pull request, and the shape
is already a DUPLICATE. I chose this over rewording the message, because
a message saying the check "may be unable to fail" would be false there.
The message now says the check is "reported on a pull request only by
pull_request_target".

### PATH-FILTER and TRIGGER, per event

Each event that reaches the PR is blocked by a path filter, or by
`types:` that omit opened or synchronize.

- The blocks are findings only when the events that are not
path-filtered together miss opened or synchronize; otherwise every head
is reported.
- Each finding names its event: `its pull_request_target trigger is
path-filtered`, or `pull_request_target types ['labeled'] omit …`.
- So `paths` on one event with `types: [labeled]` on the other gives
both findings, and `types: [opened]` with `types: [synchronize]` gives
neither.
- One-event workflows behave as before, with one exception: `types: []`
now gives TRIGGER (types `[]`). Before, it gave nothing, because the
empty list read as false. GitHub does not document `types: []`, so the
audit reads it fail-closed, and the docstring says so.

## Measured

Live, with this branch:

$ python3 -I scripts/check_required_checks.py --public-only --save
snap.json
    1 private repositories skipped (--public-only)
6 repositories, 7 workflows, 7 pull request jobs, 7 required checks: 0
finding(s), 0 not evaluated
    exit 0

No TARGET-ONLY in the org. The summary now says "pull request jobs",
because the count includes `pull_request_target` jobs.

**The issue's reproduction.** I added `.github/workflows/second.yml` to
kindkit in that snapshot, with job `check` (`runs-on: ubuntu-latest`,
one `run: 'true'` step) and `workflow_state` `active`:

    == pull_request on branch
DUPLICATE kindkit: required check 'check' is produced on a pull request
into 'main' by .github/workflows/check.yml: job 'check' (on
pull_request) and by .github/workflows/second.yml: job 'check' (on
pull_request), so a pull request head can carry it from two suites and
the merge takes either
6 repositories, 8 workflows, 8 pull request jobs, 7 required checks: 1
finding(s), 0 not evaluated
    exit 1
    == pull_request on main
6 repositories, 8 workflows, 8 pull_request jobs, 7 required checks: 0
finding(s), 0 not evaluated
    exit 0
    == pull_request_target on branch
DUPLICATE kindkit: required check 'check' is produced on a pull request
into 'main' by .github/workflows/check.yml: job 'check' (on
pull_request) and by .github/workflows/second.yml: job 'check' (on
pull_request_target), so a pull request head can carry it from two
suites and the merge takes either
6 repositories, 8 workflows, 8 pull request jobs, 7 required checks: 1
finding(s), 0 not evaluated
    exit 1
    == pull_request_target on main
6 repositories, 8 workflows, 7 pull_request jobs, 7 required checks: 0
finding(s), 0 not evaluated
    exit 0

Main's figures match the issue's.

**pull_request_target only.** On the same snapshot I changed kindkit's
`check.yml` from `pull_request` to `pull_request_target`:

    == target-only on branch
TARGET-ONLY kindkit: required check 'check' is reported on a pull
request only by pull_request_target, which runs the default branch's
workflow on the default branch's code unless the job checks out the pull
request, so it may be unable to fail because of the pull request
TARGET-ONLY kindkit: required check 'kind / conformance' is reported on
a pull request only by pull_request_target, which runs the default
branch's workflow on the default branch's code unless the job checks out
the pull request, so it may be unable to fail because of the pull
request
6 repositories, 7 workflows, 7 pull request jobs, 7 required checks: 2
finding(s), 0 not evaluated
    exit 1
    == target-only on main
ORPHANED kindkit: required check 'check' is produced by no job that
reports on a pull request into 'main' (only pull_request_target, push)
ORPHANED kindkit: required check 'kind / conformance' is produced by no
job that reports on a pull request into 'main'
6 repositories, 7 workflows, 5 pull_request jobs, 7 required checks: 2
finding(s), 0 not evaluated
    exit 1

`ruff check` and `ruff format --check` (kindkit's ruff 0.16.6,
`--line-length 100`) are clean on `scripts/`.

## Shown to fail

**Red first.** I grafted this branch's self-test onto main's logic, as
#15, #17 and #19 did: main's module up to `# --- self-test`, then this
branch's self-test section. Main gives `selftest: 185 of 225 as
expected`, exit 1. The three `TARGET_OK` cases crash there, because
main's `evaluate` takes no such argument:

FAIL pull_request_target reports on the PR head, but only base code: got
['ORPHANED'] exit 1, want ['TARGET-ONLY'] exit 1
FAIL two pull_request workflows: got [] exit 0, want ['DUPLICATE'] exit
1
FAIL pull_request and pull_request_target workflows: got [] exit 0, want
['DUPLICATE'] exit 1
FAIL pull_request_target first by path, pull_request second: got [] exit
0, want ['DUPLICATE'] exit 1
FAIL pull_request_target job named by name:: got [] exit 0, want
['DUPLICATE'] exit 1
FAIL one workflow on pull_request and pull_request_target: got [] exit
0, want ['DUPLICATE'] exit 1
FAIL three workflows give one finding per pair: got [] exit 0, want
['DUPLICATE', 'DUPLICATE', 'DUPLICATE'] exit 1
FAIL push and pull_request in another workflow are two producers: got
['DUPLICATE', 'DUPLICATE'] exit 1, want ['DUPLICATE', 'DUPLICATE',
'DUPLICATE'] exit 1
FAIL called workflow under pull_request in another workflow: got [] exit
0, want ['DUPLICATE'] exit 1
FAIL unrequired pull_request_target job: got [] exit 0, want ['UNGATED']
exit 1
FAIL pull_request_target with an if:: got ['ORPHANED'] exit 1, want
['CONDITIONAL', 'TARGET-ONLY'] exit 1
FAIL pull_request_target types: [labeled]: got ['ORPHANED'] exit 1, want
['TRIGGER', 'TARGET-ONLY'] exit 1
FAIL pull_request_target path-filtered: got ['ORPHANED'] exit 1, want
['PATH-FILTER', 'TARGET-ONLY'] exit 1
FAIL pull_request path-filtered, pull_request_target not: got
['PATH-FILTER'] exit 1, want ['DUPLICATE'] exit 1
FAIL both events path-filtered: got ['PATH-FILTER'] exit 1, want
['DUPLICATE', 'PATH-FILTER', 'PATH-FILTER'] exit 1
FAIL pull_request types: [labeled], pull_request_target default: got
['TRIGGER'] exit 1, want ['DUPLICATE'] exit 1
FAIL both events types: [labeled]: got ['TRIGGER'] exit 1, want
['DUPLICATE', 'TRIGGER', 'TRIGGER'] exit 1
FAIL pull_request path-filtered, pull_request_target types: [labeled]:
got ['PATH-FILTER'] exit 1, want ['DUPLICATE', 'PATH-FILTER', 'TRIGGER']
exit 1
FAIL pull_request types: [labeled], pull_request_target path-filtered:
got ['TRIGGER'] exit 1, want ['DUPLICATE', 'TRIGGER', 'PATH-FILTER']
exit 1
FAIL types: [opened] and types: [synchronize] together cover every head:
got ['TRIGGER'] exit 1, want ['DUPLICATE'] exit 1
FAIL scalar types: opened and synchronize cover every head: got
['TRIGGER'] exit 1, want ['DUPLICATE'] exit 1
    FAIL types: []: got [] exit 0, want ['TRIGGER'] exit 1
FAIL two pull_request_target workflows: got ['ORPHANED'] exit 1, want
['DUPLICATE', 'TARGET-ONLY'] exit 1
FAIL other pull_request_target job with a different name: got
['ORPHANED'] exit 1, want ['TARGET-ONLY'] exit 1
FAIL other pull_request matrix job is refused: got ['INCOMPLETE'] exit
2, want ['INCOMPLETE', 'INCOMPLETE'] exit 2
FAIL other pull_request filter it cannot read is refused: got
['INCOMPLETE'] exit 2, want ['INCOMPLETE', 'INCOMPLETE'] exit 2
FAIL allowlisted first producer still pairs: got [] exit 0, want
['DUPLICATE'] exit 1
FAIL allowlisted first pull_request_target producer still pairs: got
['STALE-ALLOW'] exit 1, want ['DUPLICATE'] exit 1
FAIL three producers, first allowlisted: got [] exit 0, want
['DUPLICATE', 'DUPLICATE', 'DUPLICATE'] exit 1
FAIL every producer allowlisted still pairs: got ['ORPHANED'] exit 1,
want ['DUPLICATE'] exit 1
FAIL ALLOW does not accept target-only: got ['ORPHANED', 'STALE-ALLOW']
exit 1, want ['TARGET-ONLY'] exit 1
FAIL ALLOW on a required job keeps its own push DUPLICATE: got
['ORPHANED', 'STALE-ALLOW'] exit 1, want ['DUPLICATE'] exit 1
FAIL ALLOW on a required job keeps its CONDITIONAL: got ['ORPHANED',
'STALE-ALLOW'] exit 1, want ['CONDITIONAL', 'TARGET-ONLY'] exit 1
FAIL ALLOW on a required pull_request job: target-only needs every
producer: got ['ORPHANED'] exit 1, want ['DUPLICATE'] exit 1
FAIL TARGET_OK accepts target-only: got ['CRASH TypeError'] exit -1,
want [] exit 0
FAIL TARGET_OK keeps CONDITIONAL: got ['CRASH TypeError'] exit -1, want
['CONDITIONAL'] exit 1
FAIL TARGET_OK for another context accepts nothing: got ['CRASH
TypeError'] exit -1, want ['TARGET-ONLY'] exit 1
FAIL pull_request_target and push in one workflow: got ['ORPHANED'] exit
1, want ['DUPLICATE'] exit 1
FAIL pull_request_target, and push in another workflow: got ['ORPHANED']
exit 1, want ['DUPLICATE'] exit 1
FAIL pull_request_target, push to the default branch only elsewhere: got
['ORPHANED'] exit 1, want ['TARGET-ONLY'] exit 1
    selftest: 185 of 225 as expected

Some new cases pass on main, and the sweep pins them:

- the negative controls:
  - another `pull_request` filtered to `develop`;
  - another `pull_request_target` with `branches-ignore: [main]`;
  - a `pull_request` job with a different name;
  - a `merge_group` job of the same name;
  - a `disabled_manually` workflow;
- two `disabled_inactivity` workflows, one `pull_request_target` and one
`push`;
- `pull_request_target` into another branch, which is still ORPHANED;
- a `disabled_inactivity` workflow that produces nothing, which is
ORPHANED;
- a path-filtered job whose types include opened and synchronize, which
gives PATH-FILTER only;
- the undecided filter with a different name;
- an allowlisted matrix job that gates nothing, which is not refused.

At this head: `selftest: 225 of 225 as expected`.

**The `disabled_inactivity` gap the issue names.** I mutated main's `if
state != "active"` to `if state == "disabled_manually"`. Main's
self-test then gives `selftest: 173 of 173 as expected`, exit 0, so the
mutant survives there. On this branch, that mutant and the same one in
`producers()` are `caught`.

**Mutation sweep.** Against main, 128 mutants become 158: 33 added and 3
removed, all three replaced by mutants on the rewritten code. 8 keep
their label but have new patterns, because the code they target was
rewritten or renamed: the four push-doubt mutants, "push producers not
collected", "path filter ignored", "restrictive types ignored" and
"types check inverted". The lines for the new and rewritten mutants on
the pull request rules:

target sha256 before:
7e638faf8fc1126d963195568b71f7311a76702ff33f47cadf49c407ca4dde04
    ...
    caught   merge_group counted as a pr event
    caught   pull_request_target not a pr event
    caught   pr producers not collected
    caught   two-event workflow not flagged
    caught   pr branch filter ignored elsewhere
    caught   own undecided pr refused again
    caught   undecided pr refused for any context
    caught   unknown-name pr job refused only by name
    caught   pr doubts never refused
    caught   only disabled_manually excluded
    caught   only disabled_manually excluded from producers
    caught   path filter ignored
    caught   restrictive types ignored
    caught   types check inverted
    caught   filtered event counts as covering
    caught   covering ignored
    caught   default types do not cover
    caught   blocks ignored for called jobs
    caught   every block named a path filter
    caught   types with opened and synchronize counted as restricted
    caught   scalar types not wrapped
    caught   pr pairs never reported
    caught   pr pairs only between the first two
    caught   allowlisted workflow dropped from pr pairs
    caught   pr events not recorded
    caught   target-only never reported
    caught   target-only with another event too
    caught   allowlisted job skips every check
    caught   allow does not silence ungated
    caught   allowlisted unresolved job refused
    caught   target_ok ignored
    caught   target_ok matched by repo only
    caught   push producer does not clear target-only
    caught   types: [] reads as unrestricted
    caught   coverage needs only one default type
    caught   pairs skipped when every producer is allowlisted
target sha256 after:
7e638faf8fc1126d963195568b71f7311a76702ff33f47cadf49c407ca4dde04
158 mutants: 157 caught, 0 crashed the self-test, 1 hung past 30s, 0
survived, 0 broken

The one `hung` is #19's `unclosed [ accepted`, which loops forever by
construction and is counted red. Every other line is `caught`. The hash
is that of the committed file.

No CI runs the self-test or the sweep; the daily audit workflow runs the
audit only.

## Docs

- **AGENTS.md §3.2** now has five items.
- The two-producer bullet covers a second producer on `pull_request` or
`pull_request_target`, and one workflow on both, whether or not `ALLOW`
names either job.
- A new TARGET-ONLY bullet says why such a check may be unable to fail.
It cites the docs source, says that the default branch of the base
repository's workflow and code are what run, and says that `TARGET_OK`
accepts one and `ALLOW` does not.
- **README** names the new producer, TARGET-ONLY and `TARGET_OK`.
- **STATE.md**:
- the "still does not see" paragraph is replaced by what the audit now
sees, with the evidence above, the doc source, and TARGET-ONLY;
  - the live figure says "pull request jobs";
- §6's .github#10 entry no longer lists `pull_request_target` among
events that never report on the PR, and says why;
  - §6 gains this gap;
- a new "still does not see" paragraph records what this leaves out. The
audit does not read `pull_request_review`, `deployment` or
`deployment_status` as producers, although the same page lists them as
eligible. Nor does it flag two jobs of one workflow with the same
context. `evaluate` returns no finding for either. These are left to a
separate issue.

## Review

An independent review of the first head found one HIGH and two MEDIUMs,
plus some LOWs. This head addresses each one.

- **H1, fixed.** `ALLOW` on the first-by-path producer hid the pull
request DUPLICATE, and the audit exited 0. Pairs are now computed from
the producer list, independently of `ALLOW`. The reviewer's A4, A4c and
A5b are now cases, and all three are red on main.
- **M1, fixed.** Paths and types were decided separately across events.
They are now decided per event, a covered head clears the blocks, and
each finding names its event. A3, A3b and A3c are now cases.
- **M2, fixed.** TARGET-ONLY is added; see the second round for how it
is accepted.
- **LOW, fixed.**
  - The summary label now says "pull request jobs".
- The docstring and AGENTS.md say "the default branch of the base
repository".
- Ruleset acceptance is marked as the documentation's statement, with
the source path and lines cited.
  - This body cites the source path rather than the redirect URL.

The reviewer's harness against this head:

- **`attack.py`** gives the following:

  | cases | result |
  |---|---|
| A3, A3b | DUPLICATE, plus PATH-FILTER and TRIGGER, each naming its
event |
  | A3c | DUPLICATE only |
  | A4, A4b, A4c, A5, A5b | every pair, exit 1 |
  | A2b, A8 | TARGET-ONLY |
  | A9 | PATH-FILTER and TARGET-ONLY |

- **`mymut.py`**:
  - R4, R6, R7, R8, R11, R12, R14 and R15 are caught.
- R1 survives. It drops the "(… excludes 'main')" note from an ORPHANED
message. The self-test checks finding kinds and exit codes, not wording,
and pinning wording would keep a wrong message alive rather than catch
it, so this survivor is accepted.
- R2, R3, R5, R9, R10 and R13 are BROKEN, because the code they targeted
was rewritten. I ported them to the new code and ran them under the same
hashing. R3', R5', R9', R10' and R13' are caught. R3' and R10' were
caught only after adding the two cases above; their sweep mutants are
"types with opened and synchronize counted as restricted" and "scalar
types not wrapped". R2' survives for the same reason as R1: it changes
only the types list printed in a message.

**Second round**, at `a85060d`. H1 and M1 were confirmed fixed, and the
reviewer re-derived every figure. One new HIGH, one MEDIUM and one LOW
came up; this head addresses each one.

- **HIGH, fixed.** Accepting TARGET-ONLY through `ALLOW` also silenced
every other finding on that required job, because allowlisted jobs
skipped the whole loop. `ALLOW` now silences UNGATED only, and the new
`TARGET_OK` accepts TARGET-ONLY only. The reviewer's B3b and B4 are now
cases, as is "TARGET_OK keeps CONDITIONAL". `attack2.py` at this head:
every one of B1 to B9 exits 1.
- B3b, B3c, B4, B5, B6 and B7 now report the job's DUPLICATE,
CONDITIONAL, TRIGGER or PATH-FILTER.
  - B8 gives DUPLICATE without TARGET-ONLY.
- B3d exits 1, not 2: its `ma?n` push filter is settled, not undecided.
`?` makes the `a` optional, so the filter admits `mn`, a witness branch,
and the result is a DUPLICATE. Any finding gives exit 1 in any case.
- **MEDIUM, fixed.**
- `types: []` and a lone `types: [opened]` are now cases that give
TRIGGER. "Every producer allowlisted still pairs" is a case too.
- S11 and S12 are in the sweep as "types: [] reads as unrestricted" and
"coverage needs only one default type", and S14 as "pairs skipped when
every producer is allowlisted". All are caught.
- The reviewer's literal S14 (`len(pr_events_of.get(context, ())) == 0`)
is now an equivalent mutant: allowlisted jobs record their events, so
the condition never holds for a context that has a producer.
- **LOW, fixed.** A `push` producer clears TARGET-ONLY, as above, and
the message is reworded.

`mymut2.py` at this head:

- Caught: S1, S2, S7–S12.
- BROKEN, because their code was rewritten: S4, S5, S6. Their analogues
in the sweep are caught: "target-only with another event too",
"target-only never reported", and "allowlisted job skips every check".
- Survive: S3 and S13, which change only message text, and S14, which is
equivalent as explained above.

Not reviewed again since the second round.

Closes #20

---------

Signed-off-by: Cameron Brooks <cambrooks3393@gmail.com>
CameronBrooks11 added a commit that referenced this pull request Oct 15, 2026
## What changes

`STATE.md` brought to 2026-10-15. Every figure was re-measured today, in
fresh clones of each public repository's `main`, through `just`, with a
private uv cache (`UV_CACHE_DIR=<scratch> UV_LINK_MODE=copy`). Sections
that no longer describe anything are removed rather than annotated.

- **blockspec**: the binding spike verdict is pre-registration 3's run
2, FOUND, under the owner's ruling of 2026-10-14 (`spike/LOG.md`
§34–§35), with the rules it overrides and the noise figures beside it.
Pre-registration 2 and run 1 stay INCONCLUSIVE. Adjudications 0001
(CommonMark 0.31.2 block-level profile) and 0002 (the block model), with
their addenda. The case tree (blockspec#36, #38), the reference
implementation (blockspec#37), and the required checks `conformance` and
`reference`. The "trending toward do not build" paragraph and the
blockspec#19 timing paragraph are gone.
- **rowspec**: refusal identifiers, the CSV profile and its own tree and
gate, and xlsx formulas. The fixes and spec rulings of rowspec#104 to
#139, including that #136 was closed on #128 and #137 by #139. The 22
open issues, including #142–#145 filed today, and the one open pull
request, #141, which addresses rowspec#112, #113 and #132–#135. The
GitHub Release object that now exists for `v0.3.0`.
- **nodespec**: the mermaid spike ran and is closed; identity is parked.
- **kindkit**: blockspec is a second consumer, at a different pin from
rowspec's.
- **.github**: the audit now counts 9 pull request jobs and 9 required
checks, with 0 findings. Daily runs listed.
- **What comes next**, owner decisions, setup, traps and the decided
list updated to match. blockspec's own `README.md` and `spike/README.md`
still say there is no verdict on its `main`; blockspec#39 (open) updates
them, and `STATE.md` points at it.
- **The spike's 432** is described as what it is: each FOUND cell's
distinct count summed over the 35 cells, not a count of distinct units.
The noise figures of pre-registration 3 §2.6 are quoted in full beside
the verdict.
- **Citations corrected**:
- the three recorded rule-suite rejections by id, in place of a month
query that cannot tell old rejections from none;
  - blockspec's prereg 2 pull requests as #15–#19 and #26–#28;
- the `ambiguous-table` schema change as landing through blockspec#38
(`ebfc86a`);
- the information-barrier decision in one place, no longer attributing
blind judgement to `AGENTS.md` §2.1;
  - the merge-count search as `merged:2026-10-14..2026-10-15`.
- **A setup note**: kindkit's figures need bytecode writing enabled.
- **`AGENTS.md` and `profile/README.md`** (second commit): §1's table
and the org page said blockspec was "spike, no verdict"; they now give
draft 0 (decisions, cases, a reference) and the spike's FOUND, with
nodespec's spike closed. §3.1's required-checks table adds blockspec's
`reference`, which `gh api repos/kindspec/blockspec/rules/branches/main`
lists beside `conformance`.
- §6 gains one entry: 98 of rowspec's 116 refusing `parse` cases had
asserted `refusal_contains: ""` (rowspec#61, per `CHANGELOG.md`).

## Measured

All on 2026-10-15.

The audit, here at `cb04c2c`:

    $ python3 -I scripts/check_required_checks.py --public-only
    1 private repositories skipped (--public-only)
6 repositories, 8 workflows, 9 pull request jobs, 9 required checks: 0
finding(s), 0 not evaluated

rowspec at `40e5647` (rowspec#139's merge):

    find conformance/cases -name expect.json | wc -l       483
    find conformance/cases-csv -name expect.json | wc -l   85
just conform 0 failure(s) across 483 case(s) in the fixture tree
just conform-alt 0 failure(s) across 483 case(s) in the fixture tree
just conform-csv 0 failure(s) across 85 case(s) in the fixture tree
just mutants 86 killed, 0 survived, 2 equivalent, 0 stale, 0 broken
just mutants-csv 33 killed, 0 survived, 2 equivalent, 0 stale, 0 broken
    just test           418 passed, 2 skipped
    just test-xlsx      20 passed
    just test-libreoffice   1 passed
    just check          450 files already formatted / All checks passed!
gh api repos/kindspec/rowspec/compare/v0.3.0...main --jq .ahead_by 28
gh issue list -R kindspec/rowspec --state open --limit 200 --json number
--jq length 22
gh pr list -R kindspec/rowspec --state merged --search
'merged:2026-10-14..2026-10-15' --limit 200 --json number --jq length 27
grep -rl '"refusal":' conformance/cases --include=expect.json | wc -l
130
grep -rl '"refusal_contains"' conformance/cases-csv
--include=expect.json | wc -l 43

blockspec at `d5355ab`:

    find conformance/cases -name expect.json | wc -l   237
    just tree      237 cases: 160 accept, 77 refuse / 0 problem(s)
    just conform   0 failure(s) across 237 case(s) in the fixture tree
just mutants 91 killed, 0 survived, 0 equivalent, 0 stale, 0 broken
    just test      8 passed
    just check     10 files already formatted / All checks passed!
gh api repos/kindspec/blockspec/rules/branches/main 22723622 strict,
conformance, reference

python3 -I -c "import json,collections,sys;
v=json.load(open(sys.argv[1])); print(v['overall'],
dict(collections.Counter(c['verdict'] for c in v['cells'].values())))"
<file>
spike/results/prereg2/verdict.json INCONCLUSIVE {'NO VERDICT': 52, 'NOT
FOUND': 4}
spike/results/prereg3/verdict.json INCONCLUSIVE {'NO VERDICT': 52, 'NOT
FOUND': 4}
spike/results/prereg3/run2/verdict.json FOUND {'NO VERDICT': 16,
'FOUND': 35, 'NOT FOUND': 5}

kindkit at `6ebd80d` (throwaway clone, node on PATH):

    just mutants   148 mutations, 148 caught, 0 survived, 0 broken
    just test      220 passed
    just check     28 files already formatted
    just cases     6 case(s), 0 invalid
gh api repos/kindspec/kindkit/compare/v0.1.0...main --jq .ahead_by 8

    with PYTHONDONTWRITEBYTECODE=1:
    uv run pytest -q    3 failed, 217 passed
    just mutants        148 mutations, 147 caught, 1 survived, 0 broken

Rule suites (`gh api repos/kindspec/<repo>/rulesets/rule-suites/<id>
--jq '.result+" "+.pushed_at'`):

    .github  4014197203   fail 2026-09-10T00:19:14-04:00
    rowspec  4014189520   fail 2026-09-10T00:18:20-04:00
    nodespec 4014318589   fail 2026-09-10T00:34:02-04:00

git show --stat --format= ebfc86a --
conformance/cases/case-body.schema.json (in blockspec)
     conformance/cases/case-body.schema.json | 3 ++-

Issue and pull request lists: `gh issue list` and `gh pr list` with
`--state all --limit 200` for rowspec, blockspec, nodespec, kindkit,
research and .github. Every other figure in the file names its command
beside it.

## Shown to fail

n/a: a documentation change; no check is added or altered.

## Review

None yet.

---------

Signed-off-by: Cameron Brooks <cambrooks3393@gmail.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

audit: flag a required context produced on push by a different workflow

1 participant