Skip to content

Narrow guest chat keys so a request fits a guest's credit - #18

Open
Marketen wants to merge 1 commit into
mainfrom
feat/guest-chat
Open

Marketen wants to merge 1 commit into
mainfrom
feat/guest-chat

Conversation

@Marketen

@Marketen Marketen commented Oct 29, 2026 •

Copy link
Copy Markdown
Contributor

Part of guest chat: a landing-page visitor chats without signing up. Companion PRs: dappnode/dappnode-nexus (control plane, metering, UI) and dappnode/infrastructure (settings). The design is in dappnode-nexus docs/guest-chat.md.

What this changes

Guest accounts (Metering now returns is_guest) hold a 1-cent credit. Metering refuses any request whose worst-case cost the credit can't cover, and by default that worst case is the model's whole context window: 1M tokens for Flash, far beyond any guest credit. services.GuestPolicy makes a guest request's worst case small and honest:

  • Models: only GUEST_MODELS (default deepseek/deepseek-v4.1-flash); never a router.
  • Request size: at most GUEST_MAX_REQUEST_BYTES (16 KiB).
  • Prompt reservation: reserved for the bytes actually sent upstream plus 256 tokens of template overhead, sent as the new max_input_tokens. Tokenizers emit at most one token per byte.
  • Output: capped at GUEST_MAX_OUTPUT_TOKENS (1,024) and written into the request, so the provider enforces it too.
  • Content: plain chat fields and text only. Tools, images, reasoning_effort and provider switches such as thinking get 403 guest_not_allowed.

Other behavior:

  • A guest key reaching a gateway without a policy is refused.
  • Signed-in accounts are untouched.
  • Invalid settings stop the gateway at startup.

Deploy order

Deploy metering (dappnode-nexus) before or with this. Both directions are safe:

  • Old metering, new gateway: is_guest is absent, so guest accounts are treated as normal accounts. They then reserve the full context window, which their 1-cent credit can't cover, so they're refused.
  • New metering, old gateway: max_input_tokens isn't sent, so guest reservations again use the full window and fail closed.

Tests

  • Unit tests cover:
    • GuestPolicy: every refusal, the output cap rewrite and the prompt bound
    • the request flow: the reservation bounds, a refused request never reaching metering or a provider, a guest without a policy being refused, other accounts untouched
    • the metering client: sends max_input_tokens, reads is_guest
    • the env parsing
  • go test ./..., go vet ./..., GOOS=linux go vet -tags nitro_enclave ./... and gitleaks are clean.
  • Live, against real DeepSeek (local Postgres + metering + this gateway + the control plane):
    • first token 1.07 s
    • a short message reserved 0.13 cents
    • a request with max_tokens: 100000 produced exactly 1,024 tokens
    • 12 parallel requests on one key: 7 answered, 5 refused with 402; 0.42 of 1 cent charged, nothing uncharged
    • every refusal above returned the expected code, including /v1/systemone

CodeQL

The two go/allocation-size-overflow warnings are in code this PR doesn't change: GatewayError.WithMeta (pkg/domain/error.go:66) and the Presidio adapter. This PR only adds new error constructors to error.go.

🤖 Generated with Claude Code

Guest chat accounts (is_guest, from Metering) hold a fraction of a cent.
Metering refuses any request whose worst case the credit cannot cover, and
by default that worst case is the model's whole context window. GuestPolicy
keeps it small and honest:

- only GUEST_MODELS (default deepseek/deepseek-v4.1-flash), never a router;
- requests up to GUEST_MAX_REQUEST_BYTES (16 KiB), and the reservation's
  prompt bound is the bytes sent upstream plus template overhead;
- output capped at GUEST_MAX_OUTPUT_TOKENS (1,024), written into the
  request so the provider enforces it too;
- plain chat fields and text content only: tools, images, reasoning options
  and provider switches such as `thinking` are refused.

A guest key without a configured policy is refused. Signed-in accounts are
unaffected. Invalid settings stop the gateway at startup. See the Nexus
repo's docs/guest-chat.md.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant