Skip to content

model quota: refuse an over-quota request with no usable fallback - #150

Merged
CMGS merged 1 commit into
mainfrom
feat/model-quota-deny
Oct 7, 2026
Merged

CMGS merged 1 commit into
mainfrom
feat/model-quota-deny

Conversation

@CMGS

@CMGS CMGS commented Oct 7, 2026

Copy link
Copy Markdown
Collaborator

Problem

A per-model daily quota (tenants[].model_quotas / access_keys[].model_quotas) only ever triggered the tenant fallback_model. In these cases there was no usable fallback, the request was served on the requested model, and the quota never bound:

  • no fallback is configured;
  • the fallback's protocol cannot serve a typed surface, e.g. embeddings with a chat fallback;
  • the request already names the fallback;
  • the request engages or replays reasoning and so stays on its model.

Change

  • ModelQuotaGate: over quota, the request continues only when the tenant fallback is actually swapped in. Every other case answers 400 service_quota_exceeded_exception with "model quota exhausted for <model>", the same refusal realtime admission already gives.
  • No per-model counter is armed on a refusal.
  • A fallback_models chain hop over its quota ends the chain with that refusal, because the chain only continues on upstream faults. In a batch the refusal is a failed item; the offline retry loop retries throttles only.
  • Moderation degrade is unchanged.
  • Docs: governance, configuration and api describe the refusal.

Hot-path cost

Zero under quota: the path is unchanged, with the same single quota_check and no new allocation. Prod −11 lines.

Evidence

  • New test model_quota_without_a_usable_fallback_denies (no fallback, and a request that already names the fallback).
  • The thinking-sticky test is rewritten: under quota the continuation stays on its variant; over quota it is refused, not degraded.
  • Both tests fail on main's gate.
  • Live, real OpenAI and Anthropic, main vs this branch (second call over a 1-token quota):
case main branch
no fallback, /v1/chat/completions and /v1/messages 200 400 service_quota_exceeded_exception
fallback configured 200, degraded to gpt-4o-mini, echoes the requested name same
embeddings with a chat fallback 200 400
thinking request 200 400
  • Gates: mac and Linux arm64 (rust 1.98.1), fmt and clippy clean; 34 suites, 738 passed / 0 failed / 4 ignored, including the PG16 and Redis7 suites.

A per-model daily quota used to only trigger the tenant fallback_model: with
none configured, one whose protocol cannot serve the surface, a request that
already names the fallback, or a reasoning request pinned to its model, the
request was served on the requested model and the quota never bound. Those
cases now answer 400 service_quota_exceeded_exception, the refusal the
realtime admission already gives; a fallback_models chain hop over its quota
ends the chain with it.
@CMGS
CMGS merged commit 0365b6f into main Oct 7, 2026
2 checks passed
@CMGS
CMGS deleted the feat/model-quota-deny branch October 7, 2026 05:25
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant