continuation_gateway

English | 简体中文

Important

The gateway supports SGLang and vLLM as the inference engine, selected with CONTINUATION_ENGINE. Within this project, each engine + model combination has its own continuation strategy. The combinations verified so far:

Status: ✅ verified, ⚠️ supported with issues (continuation works, but with the problems described in the notes), ❌ not supported.

SGLang (NVIDIA GPU, tested on v0.5.19+)

Model	Status	Notes
Kimi-K3	✅ Verified
Qwen3.6-35B-A3B	✅ Verified
GLM-5.3	✅ Verified

vLLM (NVIDIA GPU, tested on v0.30.0+)

Model	Status	Notes
Qwen3.6-35B-A3B	✅ Verified
GLM-5.3	✅ Verified
DeepSeek-V4-Flash-0731-nvfp4	✅ Verified
Kimi-K3	✅ Verified	vLLM must be started with --enable-scale-out, otherwise continuation does not work. Multimodal requests are not continued

vLLM (Ascend 910B4, tested on vllm-ascend 0.30.0+)

Model	Status	Notes
Qwen3.6-35B-A3B	✅ Verified	Tested with the W8A8 weights
Qwen3.6-27B	✅ Verified	Tested with the W8A8 weights
Qwen3-30B-A3B	✅ Verified	Tested with the W8A8 weights
Qwen3-14B	✅ Verified
Qwen3-8B	✅ Verified
GLM-5.3-Flash	✅ Verified	Tested with the W8A8 weights
Gemma-4-31B-it	✅ Verified	The model itself is occasionally unstable: it may emit its control tokens as plain text in content (for example </think>, <|tool_call>) so that the tool_call is not parsed and is lost. This comes from the model and its parser, not from how the continuation prefix is built
Gemma-4-26B-A4B-it	✅ Verified	Same as Gemma-4-31B-it. Its reasoning may also occasionally end up in content
DeepSeek-V4.1-Flash-1017	⚠️ Supported with issues	Tested with the W4A8 weights. Requests are continued and tool_calls are produced, but the model's internal special tokens may leak into the continued output as plain text (for example <ds_safety>…</ds_safety>, <DS_STREAMING>, @@INLINE_ICON@@), in which case the answer may be cut short or end up in reasoning instead of content. A continuation from a short prefix may also drift off topic or switch language. The seam can occasionally lose or gain a character
Qwen3-Omni-30B-A3B-Instruct	❌ Not supported

The model names in the tables are exactly the strings CONTINUATION_MODEL should be set to (case-insensitive, exact match). A quantized or fine-tuned variant of a listed model, such as GLM-5.3-FP8, should use the listed name GLM-5.3; a different size or version (Qwen3-32B, GLM-5.4) is a different model and is not matched.

Other models and engines have not been verified; they may work, but please verify them yourself before relying on them. We are continuing to adapt more engine + model combinations.

A crash-continuation gateway for LLM inference services. It sits between clients and an OpenAI-compatible inference service (SGLang or vLLM — see the note above).

An inference instance can crash in the middle of a streaming generation, have its connection reset, or stall (the process is alive but stops emitting tokens). The client is then left with a truncated reply that never received a finish_reason, and usually has no choice but to retry the whole request — the content already generated and the tokens already spent are wasted. When this gateway detects that a streaming response has stalled (idle timeout) or has been disconnected, it automatically issues one continuation request: the reasoning/content already sent to the client is assembled into an assistant prefix, and the model is asked to carry on from there. To the client, it is still a single, continuous, complete streaming response.

The way the prefix is assembled is determined by CONTINUATION_MODEL (the model this deployment actually serves; the model field of the request is not consulted), see continuation_gateway/reconstruct.py. Two formats are built in: Kimi-K3 (XTML) and a generic <think>...</think> format. Supporting a model with another format only takes adding a builder there. A model that matches no verified model is handled with the generic format, and the gateway logs a warning at startup. On vLLM, some models (such as Qwen3.6-35B-A3B) additionally split the content-done partial message into separate reasoning / content fields.

                               leg1: original request
  client ──▶ continuation_gateway ──────────────────▶ downstream router ──▶ inference instance
     ▲         │  forwards chunk by chunk, parses SSE   (SGLang / vLLM)
     │         │  accumulates reasoning / content
     │         │  detects stall / disconnect (no finish_reason seen)
     │         ▼
     │      builds the prefix, issues leg2: continuation request ──▶ downstream router
     │         │                                    (can be set separately via CONTINUATION_URL)
     └─────────┘  leg2's output continues right after what leg1 already forwarded;
                  id / reasoning field name / usage are rewritten before forwarding

Design principles

• Orchestration at the gateway layer; the inference engine is untouched. Continuation requires parsing SSE semantics, remembering what has already been forwarded, issuing an extra request on its own initiative, and rewriting usage in place. So within a single request the gateway is stateful rather than a transparent byte-forwarder.
• Conservative first: when in doubt, pass through. A request that is out of scope, or does not meet the conditions for continuation, behaves exactly as if this gateway were not there: it does not hang up the connection, swallow exceptions, or rewrite the response. The gateway steps in only when a meaningful continuation can actually be made.
• One continuation, at most two legs. A single client request involves at most leg1 + leg2. If leg2 itself fails, no third leg is issued and the request ends as it stands — no worse than having no gateway at all. This is also why the usage correction formula can assume "exactly two legs".
• The gateway must not become a point of failure itself. Any unexpected error during the continuation phase is only logged, never propagated; content already forwarded to the client is unaffected.
• Transparent to the client. The id and the reasoning field name stay consistent across the whole stream. The recovered portion is counted in this request's usage, and the continuation request's max_tokens is reduced by what has already been generated, so the original budget is never exceeded.
• Billing is addition-only and does not aim for absolute precision. The recovered content is converted to a token count by character-based estimation and added to the usage reported by leg2. A small deviation is accepted in preference to the risk of a completely wrong measurement, and nothing depends on the downstream's /v1/tokenize.
• No cross-request state. All state lives in the coroutine of a single request. Replicas share nothing, so the gateway scales horizontally and needs no sticky sessions.

Out of scope for this gateway: general request retry (any problem before the first chunk arrives is propagated as-is), choosing or avoiding failed inference instances (left to the downstream router), and admission control and rate limiting (left to the upstream ingress).

Scope

Only free-text scenarios are handled, where no tool_call chunk had appeared at the time of the crash and the request is not constrained by response_format to json_object / json_schema. The following cases are passed through without continuation; see the comments on should_intervene() in continuation_gateway/server.py for details:

• A crash after a tool_call chunk has already appeared: under the standard continue_final_message semantics there is no "tool call still being generated" state, so supporting it would require changing the SGLang source. BUFFER_TOOL_CALLS=true (see below) works around this limitation: tool_call chunks are held in gateway memory and released to the client only once complete; on a crash the whole held batch is discarded and the continuation starts from the state before the tool_call appeared. The client never received a tool_call, so there is no "cannot be continued" state.
• Structured output with response_format set to json_object / json_schema: the constrained JSON content is mixed into ordinary content chunks, and there is no separate signal to recognize it.
• Requests that already carry continue_final_message from the client: continuation is not stacked on top.
• With CONTINUATION_ENGINE=vllm, requests whose messages carry non-text content (images, audio, video).
• Request bodies larger than MAX_CONTINUATION_BODY_MB (by default, large multimodal bodies).

Building the image

The repository root includes a Dockerfile (the only dependency is aiohttp). The base image and the PyPI index are build args:

build arg	default	description
BUILDER_IMAGE	python:3.12-slim	Base image used to build and run
PIP_INDEX_URL	empty (direct PyPI)	Optional PyPI mirror

docker build -t continuation-gateway:latest .

# If Docker Hub / PyPI are not reachable, use your own registry and PyPI mirror:
docker build -t continuation-gateway:latest \
    --build-arg BUILDER_IMAGE=<registry>/python:3.12-slim \
    --build-arg PIP_INDEX_URL=https://pypi.tuna.tsinghua.edu.cn/simple .

Run the container (environment variables are described in the next section):

docker run --rm -p 8000:8000 \
    -e DOWNSTREAM_URL=http://<router>:<port> \
    -e CONTINUATION_MODEL=<actual model> \
    -e CONTINUATION_ENGINE=<sglang|vllm> \
    continuation-gateway:latest

Running

Without a container, run it directly with Python 3.12 (pip install -r requirements.txt).

The downstream is expected to be a datacenter/cluster router (not a specific SGLang instance directly). The original request and the continuation request go to the same URL by default:

DOWNSTREAM_URL=http://<router>:<port> CONTINUATION_MODEL=<actual model> \
    CONTINUATION_ENGINE=<sglang|vllm> python -m continuation_gateway.server

CONTINUATION_MODEL is the model the downstream of this deployment actually serves (for example kimi-k3, case-insensitive). It is not the model field of the client's request, nor the inference engine's --served-model-name (which can be set to anything and need not match the real model). The gateway assumes a deployment's downstream serves a single model; this value decides how the continuation prefix is built and applies uniformly to every request it receives — requests are no longer filtered by their model field. When it is not set, the gateway triggers no continuation at all and passes every request through unchanged.

CONTINUATION_ENGINE is the inference engine behind the downstream, sglang (default) or vllm. See the compatibility table at the top for which engine + model combinations are verified.

Optional environment variables (defaults are in continuation_gateway/config.py): PORT, CONTINUATION_URL (send the continuation request to a different address; if unset it is the same as DOWNSTREAM_URL), CONTINUATION_ENABLED (default true; when set to false the gateway still does all of leg1's monitoring, TRIGGERED detection and logging, but does not actually issue the second leg — useful for verifying the monitoring path itself without adding real continuation load to the downstream), STALL_IDLE_TIMEOUT_SECONDS, CONNECT_TIMEOUT_SECONDS, MAX_CONTINUATION_BODY_MB, MAX_REQUEST_BODY_MB, CJK_CHARS_PER_TOKEN, OTHER_CHARS_PER_TOKEN.

DERENDER_BATCH_CHUNKS (default 8, minimum 1) only applies to the combination whose continuation leg uses render-generate (vLLM + Kimi-K3): that leg runs render, tokenize, /inference/v1/generate and derender against CONTINUATION_URL, so the address must expose all four endpoints (the vLLM server needs --enable-scale-out). Each derender call replays all previously generated tokens, so the generated chunks are batched, DERENDER_BATCH_CHUNKS of them per call; the remainder is flushed when the final chunk (the one carrying finish_reason) arrives. A larger value means fewer derender calls but a more bursty stream.

BUFFER_TOOL_CALLS (default false; behavior is unchanged when off; it only takes effect when CONTINUATION_ENABLED=true, see below): once a chunk carrying tool_calls appears on leg1, that chunk and every line after it are held in gateway memory until finish_reason (or [DONE]) arrives, and are then forwarded all at once. If the stream stalls or disconnects while holding, the held content is discarded as a whole, and the continuation starts from the thinking-partial / content-done state before the first tool_call chunk (the continuation leg regenerates, and may produce a tool_call again). When there is no recoverable content, the held content is released to the client unchanged. Cost: tool_call arguments no longer arrive chunk by chunk but all at once after the tool_call completes.

CONTINUATION_ENABLED=true is a prerequisite for BUFFER_TOOL_CALLS to take effect: the only purpose of holding is to keep continuation feasible. Once the continuation itself is turned off, holding would buy no recovery and would only make the tool_call arguments reach the client a round later for nothing, so with this combination the gateway skips holding and treats the request as a plain pass-through, the same as when BUFFER_TOOL_CALLS is not enabled.

Tests

tests/smoke_test.py: does not connect to a real cluster. It starts a local fake downstream (simulating SGLang's streaming responses) plus the real continuation_gateway service, and uses real HTTP requests to verify the gateway's own forwarding and orchestration logic (continuation triggered by stall/disconnect, scenarios such as tool_call / response_format / oversized body being left alone, usage rewriting, and prefix reconstruction dispatched by CONTINUATION_MODEL). It does not verify whether the continued content is semantically coherent — that has to be verified on a real cluster.

python tests/smoke_test.py

License

This project is open-sourced under the Apache License 2.0.