fix(scheduler): stop provisioning runners for jobs that no longer exist

[luthermonson]

Problem

On 2026-10-05 the Windows node (2 slots) ran ~10 real jobs in 24h while booting ~23 runners for jobs GitHub had already cancelled or finished, some three days earlier. Each dead-job runner idled out the 90-minute orphan grace, so ~34 of the node's 48 slot-hours went to nothing, and live jobs queued for hours behind them: the ephpm Nightly's Windows jobs waited 4h with the node otherwise idle. At the time of diagnosis, 26 dispatches were parked on the node's slots, most of them for dead jobs; 7 sampled IDs were all completed at GitHub, the oldest from 10-02.

The macOS pool's week-long outage was this same loop on a 1-slot pool (alongside the stale-runner glob, which is a separate fix).

Root cause

Three defects compound:

1. admitDispatch trusts a record that expires. It decides "already ran" from the in-memory started map, pruned after ~JobTimeout, but a slot wait had no bound. Dispatches 24h and 28h into their wait were admitted for jobs that had been finished for days.
2. reprovisionIfStranded re-queues any job not in started. Each idle runner's exit put the dead job straight back in line.
3. The zombie cap never fires. cleanSeen reset attempts[key] whenever seen[key] aged past seenTTL (10m), including for jobs still pending on a slot or running. Every strand cycle is longer than 10 minutes, so the counter was zero at each re-provision.

Fix

• Ask the platform. New optional providers.JobStateReporter. GitHub implements it with GET /repos/{o}/{r}/actions/jobs/{id}: in_progress, completed, or 404 means gone; queued, waiting, pending, or requested means still needs a runner.
  • admitDispatch asks when the dispatch waited for its slot. A dispatch that found a free slot at once skips the call, since its webhook is seconds old.
  • reprovisionIfStranded asks before every re-queue.
  • A gone verdict abandons the dispatch, releases the slot, and is recorded in started.
  • Fails open: an API error provisions exactly as before. A stranded live job costs more than one wasted runner.
• Bound the wait. acquireSlot gives up after maxSlotWait (6h). Nothing is lost: a still-queued job is re-dispatched by the reconcile poll or its next queued event.
• Keep the zombie counter while the job is held. cleanSeen keeps attempts and jobLabels while the key is in pending or running.

Tests

pkg/scheduler/stale_dispatch_test.go:

• Waited dispatch × {gone, still queued, API error, poll mode}: checks claim count, slot release, and that the gone verdict is recorded.
• The fast path never calls the API.
• reprovisionIfStranded × {gone, queued, API error}.
• The production loop end to end: runners exit unassigned immediately and the platform says gone, so there is exactly one claim. Before this change the same loop ran to the cap.
• Wait deadline: the job is dropped with no claim and no slot leak, its dedup is cleared, and it re-dispatches once there is room.
• cleanSeen keeps attempts for pending and running jobs, and drops them for a job that has left.

pkg/github/jobawaiting_test.go: the status mapping, 404 meaning gone, and 5xx returning an error.

The new scheduler tests pass 20/20 runs locally; pkg/scheduler, pkg/github and pkg/providers/... all pass; vet is clean on windows and linux. The local golangci-lint build is broken, so lint relies on CI.

docs/arch/webhook-job-lifecycle.md has a new Stale dispatch decisions section.

Relief applied in the meantime

The Windows node was restarted onto v0.2.17 with --no-drain, which cleared the in-memory backlog. Without this fix, the backlog builds up again within a day or two.

[ephpm]

ePHPm Preview — removed

Preview deployment has been torn down.

[ephpm]

ePHPm Preview — deployed (health check pending)

URL	https://ephpm-ephemerd-pr-210.preview.ephpm.dev
Framework	PHP
PHP	8.5
Deployed in	61.6s

Preview updates automatically on each push to this PR.