feat: flag a required check with a second pull request producer

[CameronBrooks11]

What changes

The DUPLICATE rule (#17, #19) finds a second producer only on push. A required context can also get one from the pull request itself:

• workflow A runs check on pull_request;
• workflow B runs a job also named check on pull_request, or on pull_request_target.

The head then carries check from two suites, and the gate takes whichever run it finds (kindspec/rowspec#43). The audit passed this. The PR also corrects how pull_request_target is treated, and adds TARGET-ONLY for a check that only it produces.

DUPLICATE: a second producer on the pull request

push_producers() is now producers(data, branch, admits).

• With push_duplicates it does what it did before.
• With the new pr_admits it collects every active workflow whose pull_request or pull_request_target runs for a PR into the default branch, and the context of each job. That context is name: or else the id, or <caller> / <called> for a call. The same expand() and refusals apply.

Three shapes are reported:

• Two workflows. For each required context, every pair of workflows that produce it on a pull request is a DUPLICATE, reported once per pair. The pairs come straight from the producer list, so ALLOW on either job does not hide them. The message names both workflows and their events:

  DUPLICATE    kindkit: required check 'check' is produced on a pull request into 'main' by .github/workflows/check.yml: job 'check' (on pull_request) and by .github/workflows/second.yml: job 'check' (on pull_request_target), so a pull request head can carry it from two suites and the merge takes either
  

• One workflow on both pull_request and pull_request_target is also a DUPLICATE.

• Several producers. These findings do not overlap with feat: flag a required check that push also produces on a pr branch #17/feat: flag a required check that another workflow produces on push #19's push rules; each one names a different pair of suites. Three workflows give three findings. push and pull_request in another workflow are two distinct suites from that workflow, so they give two findings, plus that workflow's own feat: flag a required check that push also produces on a pr branch #17 finding. Cases pin each count.

Refused (exit 2), as #18 does:

• a job in another active pull request workflow whose context cannot be resolved (a matrix, an expression or block name, or a call not read), since it could be any context;
• a branch filter on that workflow that cannot be read, when a job there has the same context.

pull_request_target reports on the PR head

Until now, a required check produced only by pull_request_target was ORPHANED, on the grounds that "its check runs attach to the base commit". That is wrong. I checked it against the issue's cpython run and two more independent repositories:

run	event	job's head_sha	PR head	listed in commits/<sha>/check-runs	in the PR's statusCheckRollup
python/cpython 25066296099	pull_request_target	77e73c2	PR 149109 (fork warsaw/cpython)	per the issue	not checked
electron/electron 37748122900	pull_request_target	a965781	PR 54550 head a965781 (same-repo branch, compare main...sha ahead 10, behind 0)	yes, check run 113214421021	yes, event: pull_request_target
react/react 37730121633	pull_request_target	7af445b	PR 37788 head 7af445b (fork abdulmajidcse/react, ahead 1, behind 0)	yes, check run 113157110710	yes, event: pull_request_target

The commands were:

• gh api repos/<o>/<r>/actions/runs/<id>/jobs
• gh api repos/<o>/<r>/pulls/<n>
• gh api repos/<o>/<r>/commits/<sha>/check-runs
• a GraphQL query of the last commit's statusCheckRollup.contexts with checkSuite.workflowRun.event

react's head also carries check_access from two different pull_request_target workflows, which is this issue's hazard in a real repository.

GitHub's documentation says such checks count. The source is github/docs content/pull-requests/how-tos/merge-and-close-pull-requests/troubleshooting-required-status-checks.md, lines 55–64 at 60f9f34:

For checks created by workflow jobs to be evaluated for a pull request, the workflow run must be triggered by one of these events:
push, pull_request, pull_request_review, pull_request_target, deployment, deployment_status

For example, if a workflow is triggered by workflow_dispatch on a pull request's head branch, checks reported by its jobs do not appear in the pull request's checks section. Even if the checks pass for the head commit, they do not satisfy a required status check in a branch ruleset.

That a ruleset accepts a pull_request_target check is the documentation's statement; I have not watched a kindspec ruleset do it. The same page answers the issue's "unverified routes": workflow_dispatch on a PR branch does not count, and create is not in the list.

So pull_request_target is now a pull request event:

• ORPHANED. A required check that pull_request_target alone produces is no longer ORPHANED; it gets TARGET-ONLY instead (below).
• UNGATED. A pull_request_target job that is not required is UNGATED, like a pull_request job. A job meant to gate nothing, such as a labeller, goes in ALLOW with that reason.
• CONDITIONAL, TRIGGER, PATH-FILTER apply to it as well.

TARGET-ONLY: a check that may be unable to fail

A pull_request_target job runs the workflow file from the default branch of the base repository, with GITHUB_SHA that branch's last commit. Unless it checks out the pull request, it runs on the default branch's code, so it may be unable to fail because of the pull request. That is the §2.2 pattern. The audit cannot tell whether a job checks out the pull request, so:

• A required check whose only pull request producers are pull_request_target jobs is TARGET-ONLY, exit 1.
• TARGET_OK, a new set of (repo, context): reason, accepts one, and only that finding. The job's other findings (CONDITIONAL, TRIGGER, PATH-FILTER, DUPLICATE) still fire.
• ALLOW silences only UNGATED, for a job that gates nothing. A required job in ALLOW is checked like any other. An allowlisted job also still counts as producing its contexts; before, a required context from one was left ORPHANED. An allowlisted job whose context cannot be resolved, such as a matrix labeller, is still not refused, as on main. If it produced a required check, that check would be ORPHANED, so this is not a silent pass.
• A check also produced on pull_request gets no TARGET-ONLY; the DUPLICATE rule covers it instead.
• A push producer clears TARGET-ONLY. If a push run on a pull request's branch also produces the check, in the same workflow or another, TARGET-ONLY does not fire. That run tests the pull request's code, so the check can fail because of the pull request, and the shape is already a DUPLICATE. I chose this over rewording the message, because a message saying the check "may be unable to fail" would be false there. The message now says the check is "reported on a pull request only by pull_request_target".

PATH-FILTER and TRIGGER, per event

Each event that reaches the PR is blocked by a path filter, or by types: that omit opened or synchronize.

• The blocks are findings only when the events that are not path-filtered together miss opened or synchronize; otherwise every head is reported.
• Each finding names its event: its pull_request_target trigger is path-filtered, or pull_request_target types ['labeled'] omit ….
• So paths on one event with types: [labeled] on the other gives both findings, and types: [opened] with types: [synchronize] gives neither.
• One-event workflows behave as before, with one exception: types: [] now gives TRIGGER (types []). Before, it gave nothing, because the empty list read as false. GitHub does not document types: [], so the audit reads it fail-closed, and the docstring says so.

Measured

Live, with this branch:

$ python3 -I scripts/check_required_checks.py --public-only --save snap.json
1 private repositories skipped (--public-only)
6 repositories, 7 workflows, 7 pull request jobs, 7 required checks: 0 finding(s), 0 not evaluated
exit 0

No TARGET-ONLY in the org. The summary now says "pull request jobs", because the count includes pull_request_target jobs.

The issue's reproduction. I added .github/workflows/second.yml to kindkit in that snapshot, with job check (runs-on: ubuntu-latest, one run: 'true' step) and workflow_state active:

== pull_request on branch
DUPLICATE    kindkit: required check 'check' is produced on a pull request into 'main' by .github/workflows/check.yml: job 'check' (on pull_request) and by .github/workflows/second.yml: job 'check' (on pull_request), so a pull request head can carry it from two suites and the merge takes either
6 repositories, 8 workflows, 8 pull request jobs, 7 required checks: 1 finding(s), 0 not evaluated
exit 1
== pull_request on main
6 repositories, 8 workflows, 8 pull_request jobs, 7 required checks: 0 finding(s), 0 not evaluated
exit 0
== pull_request_target on branch
DUPLICATE    kindkit: required check 'check' is produced on a pull request into 'main' by .github/workflows/check.yml: job 'check' (on pull_request) and by .github/workflows/second.yml: job 'check' (on pull_request_target), so a pull request head can carry it from two suites and the merge takes either
6 repositories, 8 workflows, 8 pull request jobs, 7 required checks: 1 finding(s), 0 not evaluated
exit 1
== pull_request_target on main
6 repositories, 8 workflows, 7 pull_request jobs, 7 required checks: 0 finding(s), 0 not evaluated
exit 0

Main's figures match the issue's.

pull_request_target only. On the same snapshot I changed kindkit's check.yml from pull_request to pull_request_target:

== target-only on branch
TARGET-ONLY  kindkit: required check 'check' is reported on a pull request only by pull_request_target, which runs the default branch's workflow on the default branch's code unless the job checks out the pull request, so it may be unable to fail because of the pull request
TARGET-ONLY  kindkit: required check 'kind / conformance' is reported on a pull request only by pull_request_target, which runs the default branch's workflow on the default branch's code unless the job checks out the pull request, so it may be unable to fail because of the pull request
6 repositories, 7 workflows, 7 pull request jobs, 7 required checks: 2 finding(s), 0 not evaluated
exit 1
== target-only on main
ORPHANED     kindkit: required check 'check' is produced by no job that reports on a pull request into 'main' (only pull_request_target, push)
ORPHANED     kindkit: required check 'kind / conformance' is produced by no job that reports on a pull request into 'main'
6 repositories, 7 workflows, 5 pull_request jobs, 7 required checks: 2 finding(s), 0 not evaluated
exit 1

ruff check and ruff format --check (kindkit's ruff 0.16.6, --line-length 100) are clean on scripts/.

Shown to fail

Red first. I grafted this branch's self-test onto main's logic, as #15, #17 and #19 did: main's module up to # --- self-test, then this branch's self-test section. Main gives selftest: 185 of 225 as expected, exit 1. The three TARGET_OK cases crash there, because main's evaluate takes no such argument:

FAIL pull_request_target reports on the PR head, but only base code: got ['ORPHANED'] exit 1, want ['TARGET-ONLY'] exit 1
FAIL two pull_request workflows: got [] exit 0, want ['DUPLICATE'] exit 1
FAIL pull_request and pull_request_target workflows: got [] exit 0, want ['DUPLICATE'] exit 1
FAIL pull_request_target first by path, pull_request second: got [] exit 0, want ['DUPLICATE'] exit 1
FAIL pull_request_target job named by name:: got [] exit 0, want ['DUPLICATE'] exit 1
FAIL one workflow on pull_request and pull_request_target: got [] exit 0, want ['DUPLICATE'] exit 1
FAIL three workflows give one finding per pair: got [] exit 0, want ['DUPLICATE', 'DUPLICATE', 'DUPLICATE'] exit 1
FAIL push and pull_request in another workflow are two producers: got ['DUPLICATE', 'DUPLICATE'] exit 1, want ['DUPLICATE', 'DUPLICATE', 'DUPLICATE'] exit 1
FAIL called workflow under pull_request in another workflow: got [] exit 0, want ['DUPLICATE'] exit 1
FAIL unrequired pull_request_target job: got [] exit 0, want ['UNGATED'] exit 1
FAIL pull_request_target with an if:: got ['ORPHANED'] exit 1, want ['CONDITIONAL', 'TARGET-ONLY'] exit 1
FAIL pull_request_target types: [labeled]: got ['ORPHANED'] exit 1, want ['TRIGGER', 'TARGET-ONLY'] exit 1
FAIL pull_request_target path-filtered: got ['ORPHANED'] exit 1, want ['PATH-FILTER', 'TARGET-ONLY'] exit 1
FAIL pull_request path-filtered, pull_request_target not: got ['PATH-FILTER'] exit 1, want ['DUPLICATE'] exit 1
FAIL both events path-filtered: got ['PATH-FILTER'] exit 1, want ['DUPLICATE', 'PATH-FILTER', 'PATH-FILTER'] exit 1
FAIL pull_request types: [labeled], pull_request_target default: got ['TRIGGER'] exit 1, want ['DUPLICATE'] exit 1
FAIL both events types: [labeled]: got ['TRIGGER'] exit 1, want ['DUPLICATE', 'TRIGGER', 'TRIGGER'] exit 1
FAIL pull_request path-filtered, pull_request_target types: [labeled]: got ['PATH-FILTER'] exit 1, want ['DUPLICATE', 'PATH-FILTER', 'TRIGGER'] exit 1
FAIL pull_request types: [labeled], pull_request_target path-filtered: got ['TRIGGER'] exit 1, want ['DUPLICATE', 'TRIGGER', 'PATH-FILTER'] exit 1
FAIL types: [opened] and types: [synchronize] together cover every head: got ['TRIGGER'] exit 1, want ['DUPLICATE'] exit 1
FAIL scalar types: opened and synchronize cover every head: got ['TRIGGER'] exit 1, want ['DUPLICATE'] exit 1
FAIL types: []: got [] exit 0, want ['TRIGGER'] exit 1
FAIL two pull_request_target workflows: got ['ORPHANED'] exit 1, want ['DUPLICATE', 'TARGET-ONLY'] exit 1
FAIL other pull_request_target job with a different name: got ['ORPHANED'] exit 1, want ['TARGET-ONLY'] exit 1
FAIL other pull_request matrix job is refused: got ['INCOMPLETE'] exit 2, want ['INCOMPLETE', 'INCOMPLETE'] exit 2
FAIL other pull_request filter it cannot read is refused: got ['INCOMPLETE'] exit 2, want ['INCOMPLETE', 'INCOMPLETE'] exit 2
FAIL allowlisted first producer still pairs: got [] exit 0, want ['DUPLICATE'] exit 1
FAIL allowlisted first pull_request_target producer still pairs: got ['STALE-ALLOW'] exit 1, want ['DUPLICATE'] exit 1
FAIL three producers, first allowlisted: got [] exit 0, want ['DUPLICATE', 'DUPLICATE', 'DUPLICATE'] exit 1
FAIL every producer allowlisted still pairs: got ['ORPHANED'] exit 1, want ['DUPLICATE'] exit 1
FAIL ALLOW does not accept target-only: got ['ORPHANED', 'STALE-ALLOW'] exit 1, want ['TARGET-ONLY'] exit 1
FAIL ALLOW on a required job keeps its own push DUPLICATE: got ['ORPHANED', 'STALE-ALLOW'] exit 1, want ['DUPLICATE'] exit 1
FAIL ALLOW on a required job keeps its CONDITIONAL: got ['ORPHANED', 'STALE-ALLOW'] exit 1, want ['CONDITIONAL', 'TARGET-ONLY'] exit 1
FAIL ALLOW on a required pull_request job: target-only needs every producer: got ['ORPHANED'] exit 1, want ['DUPLICATE'] exit 1
FAIL TARGET_OK accepts target-only: got ['CRASH TypeError'] exit -1, want [] exit 0
FAIL TARGET_OK keeps CONDITIONAL: got ['CRASH TypeError'] exit -1, want ['CONDITIONAL'] exit 1
FAIL TARGET_OK for another context accepts nothing: got ['CRASH TypeError'] exit -1, want ['TARGET-ONLY'] exit 1
FAIL pull_request_target and push in one workflow: got ['ORPHANED'] exit 1, want ['DUPLICATE'] exit 1
FAIL pull_request_target, and push in another workflow: got ['ORPHANED'] exit 1, want ['DUPLICATE'] exit 1
FAIL pull_request_target, push to the default branch only elsewhere: got ['ORPHANED'] exit 1, want ['TARGET-ONLY'] exit 1
selftest: 185 of 225 as expected

Some new cases pass on main, and the sweep pins them:

• the negative controls:
  • another pull_request filtered to develop;
  • another pull_request_target with branches-ignore: [main];
  • a pull_request job with a different name;
  • a merge_group job of the same name;
  • a disabled_manually workflow;
  • two disabled_inactivity workflows, one pull_request_target and one push;
• pull_request_target into another branch, which is still ORPHANED;
• a disabled_inactivity workflow that produces nothing, which is ORPHANED;
• a path-filtered job whose types include opened and synchronize, which gives PATH-FILTER only;
• the undecided filter with a different name;
• an allowlisted matrix job that gates nothing, which is not refused.

At this head: selftest: 225 of 225 as expected.

The disabled_inactivity gap the issue names. I mutated main's if state != "active" to if state == "disabled_manually". Main's self-test then gives selftest: 173 of 173 as expected, exit 0, so the mutant survives there. On this branch, that mutant and the same one in producers() are caught.

Mutation sweep. Against main, 128 mutants become 158: 33 added and 3 removed, all three replaced by mutants on the rewritten code. 8 keep their label but have new patterns, because the code they target was rewritten or renamed: the four push-doubt mutants, "push producers not collected", "path filter ignored", "restrictive types ignored" and "types check inverted". The lines for the new and rewritten mutants on the pull request rules:

target sha256 before: 7e638faf8fc1126d963195568b71f7311a76702ff33f47cadf49c407ca4dde04
...
caught   merge_group counted as a pr event
caught   pull_request_target not a pr event
caught   pr producers not collected
caught   two-event workflow not flagged
caught   pr branch filter ignored elsewhere
caught   own undecided pr refused again
caught   undecided pr refused for any context
caught   unknown-name pr job refused only by name
caught   pr doubts never refused
caught   only disabled_manually excluded
caught   only disabled_manually excluded from producers
caught   path filter ignored
caught   restrictive types ignored
caught   types check inverted
caught   filtered event counts as covering
caught   covering ignored
caught   default types do not cover
caught   blocks ignored for called jobs
caught   every block named a path filter
caught   types with opened and synchronize counted as restricted
caught   scalar types not wrapped
caught   pr pairs never reported
caught   pr pairs only between the first two
caught   allowlisted workflow dropped from pr pairs
caught   pr events not recorded
caught   target-only never reported
caught   target-only with another event too
caught   allowlisted job skips every check
caught   allow does not silence ungated
caught   allowlisted unresolved job refused
caught   target_ok ignored
caught   target_ok matched by repo only
caught   push producer does not clear target-only
caught   types: [] reads as unrestricted
caught   coverage needs only one default type
caught   pairs skipped when every producer is allowlisted
target sha256 after:  7e638faf8fc1126d963195568b71f7311a76702ff33f47cadf49c407ca4dde04
158 mutants: 157 caught, 0 crashed the self-test, 1 hung past 30s, 0 survived, 0 broken

The one hung is #19's unclosed [ accepted, which loops forever by construction and is counted red. Every other line is caught. The hash is that of the committed file.

No CI runs the self-test or the sweep; the daily audit workflow runs the audit only.

Docs

• AGENTS.md §3.2 now has five items.
  • The two-producer bullet covers a second producer on pull_request or pull_request_target, and one workflow on both, whether or not ALLOW names either job.
  • A new TARGET-ONLY bullet says why such a check may be unable to fail. It cites the docs source, says that the default branch of the base repository's workflow and code are what run, and says that TARGET_OK accepts one and ALLOW does not.
• README names the new producer, TARGET-ONLY and TARGET_OK.
• STATE.md:
  • the "still does not see" paragraph is replaced by what the audit now sees, with the evidence above, the doc source, and TARGET-ONLY;
  • the live figure says "pull request jobs";
  • §6's .github#10 entry no longer lists pull_request_target among events that never report on the PR, and says why;
  • §6 gains this gap;
  • a new "still does not see" paragraph records what this leaves out. The audit does not read pull_request_review, deployment or deployment_status as producers, although the same page lists them as eligible. Nor does it flag two jobs of one workflow with the same context. evaluate returns no finding for either. These are left to a separate issue.

Review

An independent review of the first head found one HIGH and two MEDIUMs, plus some LOWs. This head addresses each one.

• H1, fixed. ALLOW on the first-by-path producer hid the pull request DUPLICATE, and the audit exited 0. Pairs are now computed from the producer list, independently of ALLOW. The reviewer's A4, A4c and A5b are now cases, and all three are red on main.
• M1, fixed. Paths and types were decided separately across events. They are now decided per event, a covered head clears the blocks, and each finding names its event. A3, A3b and A3c are now cases.
• M2, fixed. TARGET-ONLY is added; see the second round for how it is accepted.
• LOW, fixed.
  • The summary label now says "pull request jobs".
  • The docstring and AGENTS.md say "the default branch of the base repository".
  • Ruleset acceptance is marked as the documentation's statement, with the source path and lines cited.
  • This body cites the source path rather than the redirect URL.

The reviewer's harness against this head:

• attack.py gives the following:

  cases	result
  A3, A3b	DUPLICATE, plus PATH-FILTER and TRIGGER, each naming its event
  A3c	DUPLICATE only
  A4, A4b, A4c, A5, A5b	every pair, exit 1
  A2b, A8	TARGET-ONLY
  A9	PATH-FILTER and TARGET-ONLY

• mymut.py:

  • R4, R6, R7, R8, R11, R12, R14 and R15 are caught.
  • R1 survives. It drops the "(… excludes 'main')" note from an ORPHANED message. The self-test checks finding kinds and exit codes, not wording, and pinning wording would keep a wrong message alive rather than catch it, so this survivor is accepted.
  • R2, R3, R5, R9, R10 and R13 are BROKEN, because the code they targeted was rewritten. I ported them to the new code and ran them under the same hashing. R3', R5', R9', R10' and R13' are caught. R3' and R10' were caught only after adding the two cases above; their sweep mutants are "types with opened and synchronize counted as restricted" and "scalar types not wrapped". R2' survives for the same reason as R1: it changes only the types list printed in a message.

Second round, at a85060d. H1 and M1 were confirmed fixed, and the reviewer re-derived every figure. One new HIGH, one MEDIUM and one LOW came up; this head addresses each one.

• HIGH, fixed. Accepting TARGET-ONLY through ALLOW also silenced every other finding on that required job, because allowlisted jobs skipped the whole loop. ALLOW now silences UNGATED only, and the new TARGET_OK accepts TARGET-ONLY only. The reviewer's B3b and B4 are now cases, as is "TARGET_OK keeps CONDITIONAL". attack2.py at this head: every one of B1 to B9 exits 1.
  • B3b, B3c, B4, B5, B6 and B7 now report the job's DUPLICATE, CONDITIONAL, TRIGGER or PATH-FILTER.
  • B8 gives DUPLICATE without TARGET-ONLY.
  • B3d exits 1, not 2: its ma?n push filter is settled, not undecided. ? makes the a optional, so the filter admits mn, a witness branch, and the result is a DUPLICATE. Any finding gives exit 1 in any case.
• MEDIUM, fixed.
  • types: [] and a lone types: [opened] are now cases that give TRIGGER. "Every producer allowlisted still pairs" is a case too.
  • S11 and S12 are in the sweep as "types: [] reads as unrestricted" and "coverage needs only one default type", and S14 as "pairs skipped when every producer is allowlisted". All are caught.
  • The reviewer's literal S14 (len(pr_events_of.get(context, ())) == 0) is now an equivalent mutant: allowlisted jobs record their events, so the condition never holds for a context that has a producer.
• LOW, fixed. A push producer clears TARGET-ONLY, as above, and the message is reworded.

mymut2.py at this head:

• Caught: S1, S2, S7–S12.
• BROKEN, because their code was rewritten: S4, S5, S6. Their analogues in the sweep are caught: "target-only with another event too", "target-only never reported", and "allowlisted job skips every check".
• Survive: S3 and S13, which change only message text, and S14, which is equivalent as explained above.

Not reviewed again since the second round.

Closes #20