[CodexVeritas]

Summary

Open-weight AIB bots are pinned to a single OpenRouter host with fallbacks disabled, so when a host stops serving a model (or OpenRouter delists it) every call from that bot 404s. The live pin test catches this, but it only ran by hand. This PR runs it on a schedule so an outage fails a workflow run within hours.

• New workflow check-openrouter-pins.yaml runs code_tests/integration_tests/test_run_bots_live.py every 6 hours (27 */6 * * *) and on workflow_dispatch. It uses the existing repo-level OPENROUTER_API_KEY secret, the same one the bot jobs use.
• A 429 from the pinned host now skips that pin instead of failing it. The host is still routed, so a rate limit is not an outage. Skips are listed in the run summary (-rfEs).
• Tracebacks use --tb=short. Long tracebacks print each frame's arguments, which include the request's Authorization header. GitHub masks the secret, but this keeps it out of the logs in the first place.

What fails a run

Any non-429 error on the one-word test call to a pinned LLM of an active bot:

• Host dropped the model: OpenRouter 404 No endpoints found ... Filter by Fallback removed <host>
• Model delisted: OpenRouter 404 <model> is no longer available on OpenRouter
• Served by an unpinned provider: RuntimeError from the pinned-provider check
• Auth or other API errors

A failed scheduled run sends GitHub's standard failed-workflow email to the user who last changed the workflow's cron line, provided their notification settings include failed Actions runs.

What it does not catch

• A host that returns 429 on every call for hours shows up as skips in a green run, not as a failure.
• Problems that only appear under bot load, such as bursts of ~25 concurrent calls or long prompts. The check sends one short prompt per pin.

Cost

Measured at $0.0049 per run across 24 working pins. Calls that 404 cost nothing. At 4 runs a day that is about $0.60/month on the bots' OpenRouter key.

Verification

• Ran the test locally from a clean copy of the repo (no .env, only an OpenRouter key set, as in CI): 24 passed and 3 failed. The 3 failures are pins that are broken on main right now: thinkingmachines/inkling@deepinfra/fp8 (host dropped the model) and deepseek/deepseek-r1 / deepseek/deepseek-r1:online (delisted). Deprecate DeepSeek R1 bots and re-pin Inkling to together #379 fixes all three, so this check will fail on every run until Deprecate DeepSeek R1 bots and re-pin Inkling to together #379 merges.
• Confirmed the short-traceback output has no auth header or key-shaped strings.
• Confirmed offline that a litellm.RateLimitError makes the test skip.
• workflow_dispatch can't run before merge: GitHub only dispatches workflows that exist on the default branch, so the first CI run has to happen after merge.

🤖 Generated with Claude Code