Efficient Cross-Model KV Transfer for Multi-Model LLM Serving
Paper · Quick Start · Pipeline · Fitting · Serving · Extend CacheBridge · Agent Guide
CacheBridge is a serving system that accelerates heterogeneous multi-model LLM workloads through cross-model KV transfer. When routers, cascades or agent workflows switch models, it reconstructs the target model's history KV from the source cache to avoid re-prefilling the shared history. It co-designs a compact, attention-calibrated KV mapper with an optimized online handoff path to reduce latency and memory overhead while preserving task quality.
The paper presents four components:
- Compact head assignment. Reconstruct each target KV head from one source KV head per selected source layer using affine ridge maps, with head assignment co-designed for source-side reconstruction.
- Attention-readout calibration. Calibrate value maps against the target's attention readouts from re-prefill, keeping key maps and head assignment fixed and folding the corrections into the existing map coefficients.
- Systematic optimization. Pipeline reconstruction, transfer and direct installation into target paged KV storage with double buffering and bounded temporary memory.
- Online serving policy. Select transfer or re-prefill for each request and coordinate transfer concurrency and buffer allocation with serving demand and available GPU memory.
Offline fitting combines compact head assignment and initial ridge fitting with layer-ordered attention-readout calibration. Online serving combines source-side reconstruction, concurrent pipelined transfer and an online serving policy. Source history stays protected during reconstruction, and the target prefix is published only after every block and rank is ready. The model router chooses the next model, and CacheBridge chooses how to prepare its history state.
See the pipeline-to-code guide for stage boundaries and the architecture guide for resource lifetimes.
Requirements: Python 3.11+ and Git. The following setup runs the CPU examples with synthetic tensors and a toy agent environment.
git clone https://github.com/WHU-XingyuQu/CacheBridge.git
cd CacheBridge
python -m venv .venv
source .venv/bin/activate
python -m pip install --upgrade pip
python -m pip install torch==2.7.1 --index-url https://download.pytorch.org/whl/cpu
python -m pip install -e .
python examples/fit_and_map.py
python examples/sample_trajectory.py
python examples/inspect_architecture.pyThe examples cover fitting and applying a small mapper, sampling a routed trajectory, and inspecting model structure. They run without model weights. For model-backed fitting and serving, create a separate CUDA environment using the serving setup.
The complete fitting and serving recipe uses Qwen3-14B → Qwen3-32B, with Source TP4 + Target TP4 on one eight-GPU host. The reference environment uses PyTorch 2.7.1, Transformers 4.54.1, CUDA 12.6 and a pinned SGLang 0.4.10.post2.
| Model family | Available architecture operations |
|---|---|
| Qwen3 | Model description and layerwise query execution; complete 14B → 32B recipe |
| Llama | Model description and full-attention layer execution |
| Ministral 3 | Model description and full-attention layer execution |
| Gemma 3 | Local/global KV layout, window materialization and cache installation |
| Qwen3.5 | Attention layout, recurrent-state transformation and cache installation |
Architecture operations are building blocks for new pairs. A complete pair also requires a fitting recipe and a serving integration. See model adapters for the capability and dependency boundaries; structured-cache helpers use a separate Transformers 5.15 environment.
The trajectory recipe consumes native TRAIN handoffs, paired KV snapshots and an existing initializer that supplies the selected source layers and initial V coefficients. Model weights, initializers and fitted mapper weights are external inputs; see the artifact format.
In the fitting environment:
cachebridge recipes
cachebridge fit prepare \
--recipe qwen3_14b_32b_trajectory \
--inputs /path/to/native-bank/inputs.jsonl \
--target-model /path/to/Qwen3-32B \
--initializer /path/to/initializer/weights \
--output /path/to/new-fit
CUDA_VISIBLE_DEVICES=0,1,2,3 cachebridge fit run \
--directory /path/to/new-fit \
--mapper-id my-qwen3-mapper
cachebridge verify /path/to/new-fit/candidate.jsonUse --freeze-keys during preparation to retain K coefficients while calibrating
value maps on a new trajectory bank. The fitting guide covers
native sampling, continuation collection, input contracts and the calibration
objective.
SGLang is a Git submodule. CacheBridge's adapters and compatibility patch are part of the Python package; preparation applies the patch to a separate engine directory and keeps the upstream checkout clean.
git submodule update --init third_party/sglang
cachebridge backend prepareFollow the CUDA dependency setup, then copy the runtime configuration:
cp recipes/runtime.example.json recipes/runtime.local.jsonSet source_model, target_model, mapper_descriptor and engine_path to your
local paths. Relative paths resolve against the configuration file. Then check
the configured backend and run the workload's functional gate:
cachebridge serve --config recipes/runtime.local.json --check
cachebridge serve --config recipes/runtime.local.json -- \
python benchmarks/tau_bench/run.py evaluate \
--tasks recipes/tau_test.json \
--functional-only \
--output /path/to/functional-checkThe prepared engine defaults to .cache/cachebridge/sglang. For a source archive
or wheel installation, use cachebridge backend prepare --fetch to explicitly
obtain the same upstream commit. See external modules.
Implement the protocols in cachebridge.agents to connect another workload:
| Interface | Responsibility |
|---|---|
AgentEnvironment |
Task observations, tool execution and termination |
ModelRouter |
Select the next model from the current context |
AgentBackend |
Inference, token identities, cache handles and transfer |
sample_trajectory |
Record turns, route decisions and executed cache behavior |
Start with the runnable sampling example, then follow the trajectory API guide. The included τ-bench adapter supports native TRAIN collection and matched task evaluation:
cachebridge serve --config recipes/runtime.local.json -- \
python benchmarks/tau_bench/run.py sample \
--tasks recipes/tau_train.example.json \
--output /path/to/native-bank
cachebridge serve --config recipes/runtime.local.json -- \
python benchmarks/tau_bench/run.py evaluate \
--tasks recipes/tau_test.json \
--output /path/to/mapped-evaluationAdd --native to evaluate for target re-prefill under the same router.
Use benchmarks/tau_bench/compare.py for paired evaluation with official tool
and reward replay. See the agent development guide.
src/cachebridge/
├── architectures/ # Model adapters and complete model-pair contracts
├── offline/ # Head assignment, ridge fitting, readout calibration, export
├── online/ # KV reconstruction, pipelined transfer and handoff coordination
├── backends/sglang/ # Engine adapter, runtime and compatibility patch
├── agents/ # Router/backend protocols and trajectory sampling
├── artifacts.py # Mapper format and integrity checks
└── trajectory.py # Exact-token trajectory format
third_party/sglang/ # Pinned external SGLang submodule
benchmarks/tau_bench/ # Workload adapter and official environment/scorer
recipes/ # Runtime configuration and task rosters
requirements/ # Separate dependency environments
examples/ # Small runnable examples
tests/ # Numerical, lifecycle and interface checks
docs/ # Design, usage and extension guides
| Guide | Contents |
|---|---|
| Pipeline and code | Offline/online stages and their implementations |
| Architecture | Dependency boundaries and resource ownership |
| Fitting | Trace collection, initial ridge fitting and attention-readout calibration |
| Serving | CUDA environment, SGLang setup and runtime configuration |
| Model adapters | Family capabilities and cache-state operations |
| Trajectory API | Agent integration and exact-token records |
| Mapper artifacts | Descriptor, tensor geometry and integrity checks |
| Extension guide | Add a model pair, agent workload or backend |
| Agent development guide | Trajectory sampling, fitting and evaluation |
python -m pip install -e '.[dev,fitting]'
python -m pytestCPU CI covers numerical and lifecycle contracts, model adapters and installed package entry points. Backend functional checks run separately in the CUDA environment. See CONTRIBUTING.md for module ownership and validation guidance.
The cachebridge-fit, cachebridge-verify and integrations/sglang/run.py
entry points remain available for compatibility.
CacheBridge integrates SGLang and the τ-bench environments and scorer, and builds on PyTorch, Transformers and safetensors.
The project license is pending. Third-party components retain their upstream licenses; see license status and third-party notices.
If you use CacheBridge in your research, please cite our paper:
@misc{qu2026cachebridge,
title = {{CacheBridge}: Efficient Cross-Model {KV} Transfer for Multi-Model {LLM} Serving},
author = {Qu, Xingyu and Lu, Siyuan and Chen, Zhiyu and Wang, Sheng and Lin, Tao},
year = {2026},
eprint = {2609.00891},
archivePrefix = {arXiv},
primaryClass = {cs.AI},
doi = {10.48550/arXiv.2609.00891},
url = {https://arxiv.org/abs/2609.00891}
}