Skip to content

refactor(llm): keep a single transport-level retry policy - #130

Merged
wolfy-j merged 2 commits into
masterfrom
refactor/llm-single-retry-layer
Sep 22, 2026
Merged

wolfy-j merged 2 commits into
masterfrom
refactor/llm-single-retry-layer

Conversation

@wolfy-j

@wolfy-j wolfy-j commented Sep 22, 2026

Copy link
Copy Markdown
Contributor

Cause

LLM retry had two implementations for one concern:

  • Runtime layer: providers.open moved retry ({max_attempts, initial_delay}) out of the driver context into contract call options, so the runtime re-ran the whole driver call on a retryable error, including a streaming call that had already delivered chunks.
  • Transport layer: the shared provider transport ({attempts, backoff_ms}), used only by the OpenAI and OpenAI-compatible clients. Claude, Bedrock and Google never received a retry policy, so a configured retry was silently ignored for them.

Change

  • Only the transport layer remains. providers.open keeps retry in the driver context; llm.lua no longer copies provider retry into contract arguments (it reaches the driver through its context); a per-call retry is still hoisted and replaces the provider policy.
  • transport.request_retry resolves the policy per request (request value replaces context, false sends once).
  • Claude, Bedrock and Google clients and handlers apply it like OpenAI, OpenAI-compatible and TypeSafe. Retry happens before any response body is read, so streams are never replayed.
  • Status handlers send a single probe.
  • README documents the one retry option.
  • Separate commit: the Bedrock embed handler read the hoisted timeout from the wrong table, so a per-call embed timeout never reached the client.

Tests

Converted: provider tests assert retry stays in the driver context and with_options is never used; the retry integration app asserts the driver receives retry through its context and runs exactly once per call. Added: client retry tests (503 then 200, no retry, per-request override) for Claude, Bedrock and Google; handler forwarding tests; single-probe status tests for all six drivers; Bedrock embed timeout test. Each failed before its implementation.

  • make run-tests, make run-lint, make check-manifests: pass
  • Live run with OpenAI, Anthropic and Gemini keys: 1410 passed, 0 failed

LLM retry had two implementations: providers.open moved retry into
contract call options for runtime replay of the whole driver call, while
the OpenAI clients retried in the shared transport. Only the transport
layer remains. retry travels in the driver context from provider
options, a per-call retry replaces it, and every driver (OpenAI,
OpenAI-compatible, TypeSafe, Claude, Bedrock, Google) applies it before
any response body is read, so streams are never replayed. Health probes
always send a single request.
llm.lua hoists a per-call timeout to contract_args.timeout; the Bedrock
embed handler read it from contract_args.options, so the timeout never
reached the client.
@wolfy-j
wolfy-j merged commit 7e73730 into master Sep 22, 2026
21 checks passed
@wolfy-j
wolfy-j deleted the refactor/llm-single-retry-layer branch September 22, 2026 18:34
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant