Agent Loop Visualizer

This project is for learning: it shows how an agent loop works underneath, in a way more intuitive and visual than reading about it. Behind an agent's chat box runs a loop: step after step, the model proposes an action and the harness carries it out, until it stops. A chat app shows only the question and the answer; this page draws every step as it happens, down to each token the model writes, so the whole mechanism is laid out in front of you. Each part of the page matches a section of the Unit 01 reading of Agent Atlas (in Traditional Chinese), the course this project belongs to, and Experiments to try turns those ideas into things you can try yourself.

Models: pick google/gemma-4-E2B-it or Qwen/Qwen3.5-0.8B in the sidebar; they run on your machine with Hugging Face transformers, or on vLLM: pick vLLM (Docker) under What runs the model in the sidebar and the page starts a vLLM server in Docker for you, on the GPU if there is an NVIDIA one and on the CPU otherwise, or set VLLM_URL to use a server of your own, on this machine or on a Linux machine with a GPU (Running the model on a vLLM server)
The harness: the loop is this project's own (agent/harnesses/simple_harness.py), or, picked under What runs the loop in the sidebar, that of pi, a coding agent. pi asks either model above, on transformers or vLLM, which the page serves it on 127.0.0.1, and then every token shows as with the built-in harness; or Claude, GPT or another model pi has credentials for, and then the page draws pi's steps from the events pi reports, without tokens (pi as the harness)
MTP speculative decoding: one forward pass produces several tokens. With transformers you can switch it on in the sidebar for Gemma 4 E2B (it is off by default), with google/gemma-4-E2B-it-assistant as the drafter; on vLLM with a GPU both models decode speculatively, Gemma 4 E2B with that drafter and Qwen3.5 with the MTP layer in its own checkpoint, and vLLM in Docker does so by default
Files: attach images and videos (both models) and sound (Gemma 4 E2B, which also takes a recording from the microphone) with the + in the input box, or pick one of the sample files above it (a receipt, a recording, a video, sales data) and ask your own question about it; each becomes a run of tokens in the context, and the page shows them
Writing code (optional): Let the model write code in the sidebar gives the model a run_python tool, which runs its Python in a Docker sandbox, reads the data files you attach (CSV, TSV, JSON, text, Markdown, Excel) and draws charts with matplotlib, which appear in the chat. Docker is needed only for this and for vLLM (Docker)
Languages: the page and the How it works guide are in English and Traditional Chinese. They start in your browser's language, the switch at the top right changes it, and the language you pick is remembered for your next visit
License: Apache License 2.0

On the left is a chat like ChatGPT's; on the right the agent loop is drawn step by step: the model reads the context and proposes an action, the harness carries it out, and the result goes back into the context, until it stops. The model runs on your own computer, or on a vLLM server that sends back each token's log probabilities, so every token's probability, the candidate tokens, the special tokens, the prefill and the generation speed are all in view (on vLLM, as much of it as the server's flags let it send). The models run the same harness, so switching models changes only their text formats, which makes them easy to compare.

How it works maps each step of one turn, from sending a question to the reply, to the code that does it, with file names, line numbers and excerpts, each a click away from those lines on GitHub. Where the two models, transformers and vLLM, or the built-in harness and pi do a step differently, the step tells each in a tab of its own, and picking one switches every tab of that kind on the page. It lives on GitHub Pages; How it works at the top of the app opens it in a new tab, in the language the app is in.

What the page shows

On the page	What you see	Unit 01 reading (in Chinese)
Loop diagram	Which box you are in (context, model, harness, environment, stop) and the arrow that is flowing	§4 An agent is a model in a loop
Context	The whole string the model reads in each step (with pi on a provider's model, the request pi sent instead, as JSON): the system prompt, the tool schemas, the conversation and the tool result added after the last step. Hovering over a dark special token says what it is for. An attached image, clip or video is the run of tokens it expands into, drawn as one chip with its count (`<\|image\|>` ×266), and the step card says how many tokens each kind took	§1 The context has a limit, §3 Tool definitions are tokens too
Model	Every generated token, darker the less likely the chosen token was. Hovering over a token shows its probability, its rank, the top 5 candidates and which ones top-k / top-p cut; a special token also says what it is for. With MTP on, tokens the drafter guessed right are underlined, the popover says what the drafter guessed and whether it was kept, and the step card shows how many were kept and how many forward passes the model saved	§1 An LLM samples from a conditional probability distribution; decoding settings
Harness	The tool calls parsed from the model's own format (Gemma 4 or Qwen3.5), the schema check, warnings about repeated calls, and your Allow or Reject	§3 The model never touches the environment
Environment	What the tool really returned and how long it took; for `run_python`, the code is written out above and what it printed, or its traceback, below	§2 The environment and observable state
Stop	Why it stopped and at which token. Gemma stops at `<turn\|>` after an answer (like end_turn) and at `<\|tool_response>` when it wants a tool (like tool_use); Qwen stops at `<\|im_end\|>` either way. Then there are the step limit, the token limit and the user pressing stop	§5 When to stop
Model tab	The parameter breakdown, the number and kinds of layers, attention heads, the context limit, what the model reads (text, images, sound, video), the MTP drafter, what each decoding setting does, each step's prefill time and tok/s (with MTP, also how many guesses were kept and how many decoding forward passes ran), the id and purpose of every special token of the tokenizer (24 for Gemma, 33 for Qwen), and every file in the model's folder in `models/`: its size, what it is for, what it holds for that model and the values of its key fields	§1, §7 An agent is a system
How it works	The 18 stages of one turn (including how an image becomes tokens, how MTP guesses and checks, and what can run the model), with tabs where the built-in harness and pi differ, each with the code that does it and its line numbers, which open those lines on GitHub, in English and in Chinese. It is on GitHub Pages; the line numbers are looked up in the source when the page is built, the links go to that commit, and it is rebuilt on every push to main, so it follows the code	All of it

Quick start

You need:

macOS (Apple Silicon, using MPS), Linux with CUDA, or a CPU alone (very slow)
Disk space for the weights: about 10 GB for Gemma 4 E2B, 1.8 GB for Qwen3.5 0.8B, and about 190 MB more for Gemma's drafter if you want MTP
Memory: with MPS or CUDA, 16 GB or more for Gemma (the bfloat16 weights take about 10 GB); on a CPU it loads in float32, about 20.5 GB, so 24 GB or more. Qwen3.5 0.8B is much smaller and runs on an 8 GB machine. Only one model is loaded at a time; switching models lets the old one go
Linux + CUDA: the pinned torch is built for CUDA 13, so the CUDA Version at the top right of nvidia-smi must be 13.0 or newer. With an older driver torch does not see the GPU and runs on the CPU; the top of the sidebar shows the device in use
uv 0.13.4 or newer (it installs Python 3.12 for you). Older versions still trust eMudhra's compromised emSign certificates (the announcement), so uv sync refuses to run with them; uv self update (or brew upgrade uv) updates it
A network connection: to download the models the first time, and for the Wikipedia tools
Optional: Docker (Docker Desktop, OrbStack or Colima), only to let the model write code or to run it on vLLM (Docker)
Optional: pi and Node.js 22.19 or newer, only to hand the loop to pi (pi as the harness)
Optional: just (brew install just), only to run this README's commands by name (below)

uv sync
uv run hf download google/gemma-4-E2B-it --revision 3e22461f65e89153144f8adb70e3b8c2cc9845a7 --cache-dir models   # download the weights first (about 10 GB); otherwise they download the first time the model is picked
uv run hf download Qwen/Qwen3.5-0.8B --revision 2fc06364715b967f1860aea9cf38778875588b17 --cache-dir models       # about 1.8 GB
uv run hf download google/gemma-4-E2B-it-assistant --revision 2d874ef7d29f9a30599a1e4b3c1cbc9595f005df --cache-dir models   # the MTP drafter, about 190 MB; otherwise it downloads the first time MTP is switched on
uv run streamlit run streamlit_app.py              # loads Gemma first; MODEL_ID=Qwen/Qwen3.5-0.8B loads Qwen first
docker build -t agent-loop-sandbox agent/sandbox    # optional: the sandbox for Let the model write code (about 400 MB); again whenever agent/sandbox/Dockerfile changes

Your browser opens http://localhost:8501, in its own language (English unless the browser is set to Chinese); the 中文 | English switch at the top right changes it, and the next visit starts in the one you picked. Loading Gemma and warming it up takes about 10–20 seconds the first time, Qwen a few seconds; reloading the page does not load it again. The Model menu at the top of the sidebar switches models; switching starts a new conversation, resets the decoding settings to the new model's recommendations and turns MTP off. MTP speculative decoding in the sidebar loads the drafter the first time it is switched on (in under a second), and it stays in memory after that.

Let the model write code in the sidebar is off by default, and while it is off (and vLLM (Docker) is not picked) the page never looks for Docker. Switched on, it checks for Docker and the agent-loop-sandbox image, and if either is missing, or the image was built from an older agent/sandbox/Dockerfile, it says in the sidebar how to install Docker or build the image; the model gets run_python only once both are there, and the examples that need Python, which appear once it is on, can be pressed only then. Files you attach are saved in a folder per conversation under ~/.cache/agent-loop-visualizer/uploads/ (New chat removes it), which the sandbox sees, read-only, as /data.

The models download into models/ in the project folder, which git ignores, not into the ~/.cache/huggingface/hub other projects share; deleting the folder frees the space. Each downloads at a fixed revision, the commit in REVISIONS in agent/model_families/__init__.py (the --revision above), so whenever you download it you get the files this page was written and tested against, the same on every operating system. To keep them somewhere else, set AGENT_LOOP_MODELS to that folder (and give the downloads above the same --cache-dir); AGENT_LOOP_MODELS=$HOME/.cache/huggingface/hub uses the models already downloaded there. HF_HUB_CACHE does not move them, since it is often set once for every project on a machine.

To run the model on vLLM instead, pick vLLM (Docker) under What runs the model in the sidebar, and the page starts a vLLM server in Docker for it (vLLM in Docker, started by the page); or point the page at a server of your own (Running the model on a vLLM server).

A 401 while downloading, though the model is public? Most likely the Hugging Face token saved on your machine has expired. Log in again (uv run hf auth login), or add HF_HUB_DISABLE_IMPLICIT_TOKEN=1 to send no token.

Cloned before agent/ was called harness/? Git ignores the __pycache__ folders Python wrote there, so a pull leaves them behind, in what looks like an empty harness/. Nothing imports it; git clean -fdX harness removes it, and only what git ignores in it.

The commands by name

The justfile gives the commands in this README names, for just: just lists them, and just <name> runs one from the project's folder, wherever you are in it. Each runs the command this README gives, so none of them needs just. The environment reaches them as it reaches those commands, and what follows a name goes to the command:

just sync                            # uv sync
just download                        # every model and drafter in REVISIONS at its pinned commit, into models/ (or AGENT_LOOP_MODELS): about 12 GB
just download Qwen/Qwen3.5-0.8B      # only the ones named
just run                             # the app; MODEL_ID=Qwen/Qwen3.5-0.8B just run loads Qwen first, VLLM_URL=http://localhost:8000 just run asks that server
just sandbox                         # the sandbox image for Let the model write code
just check                           # what checks.yml runs on a pull request: ruff check, mypy, pytest, and How it works with pygments alone
just test -k render                  # pytest; just lint --fix, just types, and the tests that need models, vLLM or pi under Tests below
just vllm-up                         # vLLM in Docker by hand, on the CPU (just vllm-up gpu on NVIDIA GPUs); just vllm-down stops it

just download reads the commits from REVISIONS in agent/model_families/__init__.py, so it downloads what the hf download lines above do, and keeps doing so when a model moves to a newer commit.

Experiments to try

Just a chatbot: switch off Use tools in the sidebar and ask "What time is it in Taipei right now?". The model sees no tools, so it can only guess or admit it does not know. Switch tools back on, ask again, and compare the two in the Context tab.
Sampling: ask the same question with temperature 0 and with 1.5, and watch how the token colors and the candidates left in the popover change.
Stop conditions: set Max steps to 1 and ask something that needs a tool; the stop reason becomes "step limit reached".
Where the human sits: switch on Ask me before running a tool, then press Reject. It is the harness that blocks the call; the model only sees an error observation in the next step's context. Compare the mail deletion incident in §5 of the reading: a "confirm first" written in the prompt can get compressed away, one written into the harness cannot.
Reasoning is tokens too: switch on Thinking mode, and italic thinking tokens appear before the tool call; in the Context tab, the system turn starts with an extra <|think|>.
The loop goes in circles: ask "Who proposed the ReAct paper? Look it up on Wikipedia." E2B often repeats the same search, and the harness flags "exactly the same call as before"; compare the error analysis of ReAct in §4 of the reading.
Switch models, keep the harness: ask Gemma and Qwen the same question and compare them in the Context tab: the turn markers (<|turn> against <|im_start|>), where the tool schemas go, how a tool call is written (call:name{…} against XML) and where the tool result goes back all differ, yet the harness, environment and stop sections of the step cards are exactly the same.
MTP is fast only when it guesses right: with Gemma, switch on MTP speculative decoding, press "What time is it in Taipei right now?", type "Write a Python function to test for primes." (with Let the model write code off), then ask "用三段話介紹台北。" (introduce Taipei in three paragraphs, in Chinese). Compare each step's tok/s and underlines: the drafter guesses almost every token of a tool call, about half of English code and only about a tenth of long Chinese text. Set temperature to 0 and run with MTP on and off: the output is exactly the same, because the drafter only decides how many forward passes run, not what gets written.
An image is tokens too: press the receipt example. In the step card's context section and in the Context tab, the image is a run of a few hundred <|image|> (Gemma) or <|image_pad|> (Qwen): what it costs in the context. The items add up to 360, but the receipt says 370; see whether the model adds them up itself or calls a tool.
The context decides whether the model listens: with Gemma, press the recording example on the Chinese page, and it writes the English sentence down and translates it. Ask "Transcribe this recording." on the English page with Use tools on, and it often says it cannot hear audio, sometimes after writing the very words into a tool call; switch Use tools off and ask again, and it transcribes it. The model and the audio tokens are the same; only the context differs.
Code in a sandbox: switch on Let the model write code (Docker needed) and press the sales data example. The note at the end of your message gives the file's path and first line; the model's code is written out in the step card, and what the sandbox printed, or its traceback, comes back as the observation, so a failing program can be fixed in the next step. Try Qwen too: it usually gets as far as the totals per month, then breaks its own code.
A chart is for you, not the model: with Let the model write code on, pick sales.csv among the sample files above the input box and ask for a chart of the revenue per month. The chart appears in the chat and under the step that drew it, but the observation the model reads is only {"figures": ["/data/figure-1.png"]}: it describes a chart it has never seen.

Architecture

The project has two parts. agent/ runs a turn: the harness that runs the agent loop, the tools, the model's family and what runs the model. app/ is what you see: the page, the HTML it draws from the agent's events, and How it works. They depend one way: app/ builds on agent/, and nothing in agent/ imports app/ or Streamlit (test_layout.py checks), so the loop runs, and is tested, without a page.

streamlit_app.py       where Streamlit starts: it runs app/page.py's main() on every rerun
justfile               this README's commands by name, for just
app/                   what a person sees, built on the agent
  page.py              layout, session state, model menu, language switch, background thread; redraws every 0.5 s while a run is live
  render.py            turns the event list into HTML: the loop diagram, step cards, token heat and popovers, the context view
  guide.py             the How it works page: its content in both languages, line lookups, and the static HTML for GitHub Pages
  assets/              the page's styles and the guide's script; logo.svg; examples/, the files the examples attach and the sample files above the input box
agent/                 what runs a turn: the harness, the model, what runs the model, the tools; it never imports Streamlit
  harnesses/           what runs the loop
    harness.py         the Harness the page runs a turn through: run, stop, decide, and the events every harness sends
    simple_harness.py  this project's loop: build the prompt → generate → parse → validate → (ask you) → run the tool → back into the context → stop?
    pi_harness.py      pi as the harness: pi in RPC mode, its JSON records turned into the page's events
    pi_extension.ts    the pi extension the page loads: it reports each request pi sends, registers the page's model, and asks you before pi runs a tool
    model_endpoint.py  the page's model as an OpenAI-compatible endpoint on 127.0.0.1, for pi: render, encode, generate, and the tokens back to pi
  model_backends/      what runs the model
    backend.py         the Backend that the harness and the page run the model through: render, encode, generate, and the fields of a token event
    simple_backend.py  local inference with a hand-written decoding loop; MTP's drafting and checking are here too
    vllm_backend.py    the model on a vLLM server (VLLM_URL): the prompt rendered and tokenized here, the token ids to the server, what it can tell about each token back; media, sampling masks and MTP's guesses on a server started with the flags for them
    vllm_docker.py     vLLM in Docker for the sidebar's vLLM (Docker): docker compose up with the picked model, waiting until it answers; docker compose down
  model_families/      everything that goes with the model: how each model family writes turns, thinking and tool calls, which tokens stop it, what each means
    __init__.py        the Family definition, the model menu MODELS, the MTP drafters DRAFTERS, finding a family by config.json's model_type
    gemma4.py          Gemma 4: the output parser and notes on its 24 special tokens
    qwen3_5.py         Qwen3.5: the output parser and notes on its 33 special tokens
  tools.py             tools: the schema (for the model) plus the Python function (for the harness to run); run_python's Docker sandbox
  media.py             attachments: saving them, writing them into the user's message, decoding images, sound and video for the processor
  i18n.py              the two languages: pick() chooses the text for one, from_locale() picks the default from the browser's language; the errors the model reads use it too
  sandbox/             run_python's sandbox: the Dockerfile (Python 3.12 with numpy, pandas, openpyxl, tabulate and matplotlib), and the matplotlib backend and settings that send the charts back
docker-compose/        the vLLM server the page starts in Docker
  compose.yaml         vLLM on the CPU (any machine, a Mac's included), on 127.0.0.1 only, models/ read-only
  compose.gpu.yaml     on top of compose.yaml: vLLM's CUDA image on NVIDIA GPUs
  vllm-cpu/            vLLM 0.32.0's CPU image with the two fixes it lacks (patch_vllm.py)
tests/                 pytest tests; see Tests below
models/                the downloaded models (git ignores it; AGENT_LOOP_MODELS moves it)

The agent runs in a background thread and appends an event at every stage; the UI only draws the event list. So when you press Stop or Allow and Streamlit reruns the script, the loop in the background is not interrupted. The page reaches the loop only through a Harness (harnesses/harness.py): run(), stop(), decide() and the events it sends. SimpleHarness is this project's own, and PiHarness hands the loop to pi (pi as the harness).

How it works is a static page: uv run python -m app.guide _site (just guide) writes _site/en/index.html and _site/zh/index.html (styles and a small script inlined, so they open straight in a browser), and _site/index.html, which sends a reader to the language they last picked on the page's switch, or else their browser's. On a push to main, .github/workflows/pages.yml runs tests/test_guide.py, then does the same and publishes it to GitHub Pages; if the tests fail, the page already up stays as it was.

The events of one run, in order: run_start, then for each step step_start → context → prefill → token ×N → model_end → parsed, with a tool call also tool_call → (approval_request → approval) → (tool_start, when the tool runs) → tool_result, and finally run_end, after an error if something failed: its message is the exception's, and a backend's error (a vLLM server's, say) also comes in both languages (text), so the page shows the one it is in. context carries the prompt as the model reads it, with each attachment expanded into its tokens, and how many tokens each kind of media took (media_tokens). token carries the token and, when the backend knows them, its probability, its rank, the top 5 candidates and what MTP's drafter guessed (backend.TOKEN_DETAILS): simple_backend.py knows them all, and the page leaves out whatever a backend does not report rather than showing a number nobody measured. A backend that learns which tokens were MTP's guesses only when the step ends (a vLLM server) says so in model_end's MTP counts (kept_at, turned_down_at), and the page marks those tokens then. tool_result carries the observation the model reads and, apart from it, the images the tool made for you (images: the paths of run_python's charts), which the model never reads. run_start says which harness ran the turn (harness: simple or pi). A step pi runs begins with request, the request pi sent the model, as JSON. When pi asks the page's model, the model endpoint sends that step's context, prefill, token and model_end as the built-in harness does; when pi asks a provider's model, there is no family and none of those four but delta, a piece of the model's output and its channel (thinking, reply, tool call) as it streams, and a model_end with the token counts the provider reported (usage).

Two languages

Every piece of text on the page is written in both languages side by side in the code: pick(lang, "中文", "English") in code, and {"zh": …, "en": …} in data (the special token notes, the stop reasons, the paragraphs of How it works). Every function in render.py that writes words takes a lang, and the events themselves have no language, so switching the language redraws the same events in the other one, finished turns included. The page starts in the language last picked, kept in a cookie, or else in the browser's navigator.language (anything starting with zh is Chinese, the rest English); picking one writes the cookie with a small script, because Streamlit can read cookies but not write them.

What the model reads from the harness (wrong arguments, a failed tool, your refusal) is worded in the language the page was in when the turn started, because it becomes the observation word for word; the system prompt is the same in both, so the model's behavior does not depend on the page's language.

Why a hand-written decoding loop

model.generate() does not hand out the probability distribution at each step, so simple_backend.py runs the loop itself: one forward pass per token (with a KV cache), then temperature, top-k and top-p in turn, then a sample. So every token can report its probability and rank in the model's own distribution, its probability when sampled, how many candidates were left after top-k / top-p, and the top 5 candidates. In greedy mode the output matches generate(), as measured; of two equally likely tokens (in bfloat16, logits tie more often than one would think) it takes the lower id, as generate()'s argmax does.

MTP speculative decoding

Decoding runs a full forward pass for every token, and the time goes mostly into reading the weights from memory; computing a few more positions in the same pass is hardly slower (Gemma 4 E2B on an Apple M5: about 56 ms for 1 position, about 70 ms for 7). So if the next few tokens are guessed cheaply first, the model can check them all in one forward pass. Google trained such a drafter for Gemma 4 (MTP, multi-token prediction), and transformers' generate(assistant_model=…) uses it, but that again hides the distribution at each step, so simple_backend.py wires it into its own decoding loop:

Guess (_draft): the drafter gemma-4-E2B-it-assistant has about 78M parameters and 4 layers. It has no KV cache of its own and reads the model's keys and values directly (the last 20 layers of Gemma 4 E2B already share those two earlier layers computed); its input is the last token's embedding next to the model's last-layer hidden state. It makes 6 greedy guesses at a time (the checkpoint's num_assistant_tokens).
Check (_drafted): the model runs one forward pass over the token just settled plus the 6 guesses, gets back the distributions at 7 positions and hands them to _sample() in order. A kept guess needs no forward pass of its own; the first guess turned down is replaced by the model's own pick at that position, everything after it is dropped, and the KV cache is cut back with crop(); when all 6 hold, the last position gives one more token.
Same output: in greedy decoding a guess is kept only when it is the top token, so the output is the same as without MTP (tests/test_real_model.py compares them with the real model). When sampling, a guess is kept with its probability after temperature, top-k and top-p, and otherwise the token is drawn from the other candidates, so every token comes out exactly as often as without MTP (speculative sampling; tests/test_simple_backend.py checks it over twenty thousand samples).

How many guesses are kept depends a lot on the content. Measured in the page with Gemma's default decoding settings (Apple M5):

What the step writes	Without MTP	With MTP	Guesses kept
A tool call (step 1 of "台北現在幾點？紐約呢？", "What time is it in Taipei now? And New York?")	12.6 tok/s	40.9 tok/s	34 / 35
The answer from the tool result (step 2 of the same question)	13.0 tok/s	22.6 tok/s	42 / 106
English code, about 750 tokens	8.8 tok/s	11.6 tok/s (`simple_backend.py` alone: 8.1 → 29.0)	51%
Long Chinese text ("用三段話介紹台北。")	10.3 tok/s	10.3 tok/s	7%

Images, sound and video

Which of them a model reads is found out from the checkpoint when it loads: an encoder in its config and a part of its processor to feed it. Gemma 4 E2B reads images, sound and video; Qwen3.5 0.8B reads images and video. The input box takes only what the model reads:

Kind	Formats	Gemma 4 E2B	Qwen3.5 0.8B
Image	png, jpg, jpeg, webp, gif, bmp	up to 280 tokens	depends on the size (about 300 for 640×480)
Sound	wav, mp3, m4a, flac, ogg	one token per 40 ms, the first 30 seconds	— (with Let the model write code, a file for `run_python`)
Video	mp4, mov, webm, mkv	32 frames (all of a shorter video), each after its time as `mm:ss`, up to 70 tokens each	2 frames a second, in pairs, each pair after its time
Data	csv, tsv, json, txt, md, xlsx	with Let the model write code, for `run_python`	the same

The chat template writes one placeholder where each file goes (<|image|> for Gemma, <|vision_start|><|image_pad|><|vision_end|> for Qwen), and SimpleBackend.encode() hands the processor every file in the conversation, which expands each placeholder into the file's tokens; the Context tab shows that expanded string, since it is what the model reads. media.py decodes the files first: images to RGB, turned upright by their EXIF; sound with PyAV to mono at the feature extractor's 16 kHz; video to at most 64 evenly spaced frames, 768 pixels on the longer side (transformers' own video loading needs torchcodec). Each file is decoded once. In the prefill, images and videos go through the vision encoder first, as generate() does it (handed to the forward pass as they are, videos lose their frame counts in transformers 5.20's Gemma 4), and Qwen3.5's position offset for images is cleared before every prefill. After the prefill decoding is the same as for text, and tests/test_real_model.py checks that greedy output with an image, a video and a recording matches generate().

Writing code in a Docker sandbox

With Let the model write code on and Docker ready, the model gets one more tool, run_python. Each call starts a new container from the image agent/sandbox/Dockerfile builds: no network, a read-only file system except a small /tmp, 512 MB of memory, one CPU, at most 64 processes, no Linux capabilities and your own user, never root (on Linux the mounted folder keeps its permissions, which might not let another user in), with the conversation's folder read-only at /data; the code goes in on stdin, and the container is deleted afterwards. Code that runs past 30 seconds is stopped; a nonzero exit, using too much memory or printing nothing (and drawing nothing) comes back to the model as an error, in the turn's language, with the traceback, so it can fix the code in the next step.

matplotlib's charts come back too, although the container cannot write anywhere the page can read: the image sets matplotlib's backend to agent/sandbox/agent_loop_figures.py, which turns each figure into a PNG at plt.show() or when the program ends and, as it exits, writes them after what the code printed, behind a marker line that run_python passes in. run_python cuts them off what the model reads (at most four), saves them in the conversation's folder as figure-1.png, figure-2.png… and returns them apart from the observation: the model reads only {"stdout": …, "figures": ["/data/figure-1.png"]}, and the page shows the images in the chat and under the step. A label in Chinese is drawn with WenQuanYi Micro Hei, the font the image installs for it. Importing matplotlib makes a call about half a second slower (0.8 s against 0.2 s for plain Python).

The image installs pip 26.2.2 first, from its wheel by hash, and everything else with it: an older pip also trusts its own copy of certifi, which still has the compromised emSign certificates. The page reads the agent-loop-sandbox.version label of the image, and when it is older than agent/sandbox/Dockerfile the sidebar asks for a rebuild instead of offering run_python.

The model cannot see into a data file, so the harness adds a note to the user's message: where run_python finds each file, and the first line of a text file (for a CSV, its column names). Without that line a small model guesses columns in its code, a Date that is not there, and fails; with a few rows of data as well, Gemma copied the rows into its code and answered from them alone. tabulate is in the image because DataFrame.to_markdown(), which small models like to print tables with, needs it.

Running the model on a vLLM server

The model can also run on a vLLM server: one the page starts for you in Docker (see below), or any server, on this machine or on a Linux machine with a GPU, while the page stays on yours. For a server of your own, set VLLM_URL to where it listens:

# Where vLLM runs (vLLM 0.32 or newer, on a Linux machine with a GPU). --revision is the commit in REVISIONS, so the
# server and the page read the same tokenizer and chat template. The other flags let the page see more (below).
vllm serve Qwen/Qwen3.5-0.8B --revision 2fc06364715b967f1860aea9cf38778875588b17 --enable-scale-out
# Or Gemma 4 E2B with its MTP drafter, saying which tokens were guesses:
vllm serve google/gemma-4-E2B-it --revision 3e22461f65e89153144f8adb70e3b8c2cc9845a7 --enable-scale-out \
  --speculative-config '{"method": "mtp", "model": "google/gemma-4-E2B-it-assistant", "revision": "2d874ef7d29f9a30599a1e4b3c1cbc9595f005df", "num_speculative_tokens": 6}' \
  --per-request-spec-decode-metrics detailed
# Or Qwen3.5 with the MTP layer in its own checkpoint, which transformers cannot use:
vllm serve Qwen/Qwen3.5-0.8B --revision 2fc06364715b967f1860aea9cf38778875588b17 --enable-scale-out \
  --speculative-config '{"method": "mtp", "num_speculative_tokens": 2}' --per-request-spec-decode-metrics detailed
# Or with sampling masks, which show how many candidates were left in place of the model's own probabilities (below):
vllm serve Qwen/Qwen3.5-0.8B --revision 2fc06364715b967f1860aea9cf38778875588b17 --enable-scale-out \
  --return-sampling-mask --logprobs-mode processed_logprobs

# Where the page runs:
VLLM_URL=http://localhost:8000 uv run streamlit run streamlit_app.py

The Model menu then lists what the server serves. A model of a family the page does not support yet is marked ⚠, which the page tells from its config.json alone, and a new tab starts on the first model it does support. Picking a marked model says why it does not run and offers a GitHub issue asking for it, with the title and description already filled in: the model and its model_type, never the server's address. If the server was started with --api-key, give the page the same key in VLLM_API_KEY (the variable vllm serve reads it from). If the server cannot hold the model's whole context in its KV cache, lower it with --max-model-len; the page's context limit follows the server's.

vllm_backend.py is a second Backend (backend.py) next to simple_backend.py; the harness, the tools and the page do not change. The prompt stays exact: the page downloads the model's config, tokenizer and chat template into models/ (not the weights), renders and tokenizes the prompt itself, and sends the server the token ids, so the server reads exactly what the Context tab shows. Each token comes back as its id with its log probability and those of the most likely tokens at its position. With vLLM's default --logprobs-mode raw_logprobs those are the model's own distribution, as in simple_backend.py, and give the popover's model probability and top 5 candidates. How much more the page sees depends on the server's flags; at load, the page asks the server, with one-token requests, which of them it was started with:

Server flags	Where the steps go	What the page shows
none	`/v1/completions`	the probability, the top 5, the rank only when the token is one of them; text only
`--enable-scale-out`	`/inference/v1/generate`	also every token's rank, and the images, sound and video the server takes
`--return-sampling-mask --logprobs-mode processed_logprobs` too	the same	the candidates left and the sampling probability, in place of the model probability, the heat and the top 5: vLLM sends sampling masks only with processed log probabilities, and then refuses temperature 0 and top-k 0, so the sidebar starts at 0.05 and 1
`--per-request-spec-decode-metrics summary` / `detailed`	either	MTP's counts in each step; with `detailed`, the underlines on the guesses too, once the step is done

What the page leaves out it does not make up (backend.TOKEN_DETAILS). In detail:

--logprobs-mode decides what the numbers are, and no route says which one the server runs, so at load the page asks for one token at temperature 1 with top-k 2 and the 5 most likely tokens: in a processed mode the 3 that top-k cut come back as -9999, and log probabilities, unlike logits, add up to at most 1. With processed_logprobs (which --return-sampling-mask needs; vLLM refuses to start without it, or without its Model Runner V2, which it picks by default where it can), a token's log probability is its probability in the distribution it was drawn from, after temperature, top-k and top-p, so the popover shows it as the sampling probability, with the rank, and shows no model probability, heat or top 5. How many candidates were left comes from the sampling mask, or, without one, shows when top-k / top-p cut some of the most likely tokens listed. With raw_logits or processed_logits, the numbers are not probabilities, and the page shows none of them as one. Which of the top 5 top-k / top-p cut (gray in simple_backend.py's popover) shows in no mode outside greedy decoding: the model's own probabilities and what the sampler did never come back together. The Model tab says which mode the server runs.
The rank counts the tokens more likely than the chosen one, as in simple_backend.py, so a token tied for first (common with bfloat16 logits) is #1. /inference/v1/generate's rank counts the tokens tied with it as well (vLLM 0.32 compares with >=), and with speculative decoding it compares the logits, while the log probabilities it lists can tie where the logits do not. So while a listed token is no likelier than the chosen one, every likelier one is listed too, and the page counts the rank from the log probabilities it shows; a token below all of them keeps the server's rank, which may still count ties the server did not list.
Greedy decoding (temperature 0) needs no server to say that one candidate was left and drawn with probability 1, so the popover shows that on any server that takes it. A server started with --return-sampling-mask does not: vLLM refuses temperature 0 and top-k 0 there, on every route, so the sidebar's sliders stop at 0.05 and 1: a recommended temperature below 0.05 moves up to 0.05, and an unlimited top-k (0) to 128, the slider's top. Without --enable-scale-out no response shows the masks, so at load the page also asks such a server (one with processed log probabilities) for a greedy token, and a refusal says the same.
Images, sound and video go next to the placeholders the chat template wrote, as the server's content_parts: an image as PNG, turned upright by its EXIF; sound as WAV, decoded here; a video as the file itself, whose frames the server picks. The server expands each placeholder into the file's tokens. So that the Context tab shows the prompt as the server reads it, the page asks it for that expansion first, with a request that generates one token; if the server expands differently when generating, the step is an error. At load the page sends the server a tiny image, clip and video to see which kinds it takes; a kind it refuses (or every kind, without --enable-scale-out) cannot be attached, the line above the input box says so, and the Model tab lists why.
MTP is the server's (--speculative-config), so the sidebar's switch is off and disabled. With --per-request-spec-decode-metrics summary (which vLLM takes only with --speculative-config), each step says how many guesses were kept and how many decoding forward passes ran. vLLM counts a verify step's kept guesses before it cuts the step at a stop token, so when the stop token is one of them, the guesses after it count too. With detailed it also says how many guesses each verify step kept, from which the page marks the guesses: the first token comes from the prefill, then each verify step gives the guesses it kept and one token of the model's own. vLLM does not count a step without guesses, so the page marks the tokens only when the counts account for every one of them, and only once the step is done, counting as kept only the guesses up to the stop token, as simple_backend.py does; what a turned-down guess was, the server does not say. Both models decode speculatively on a vLLM server with a GPU, and every real-server test passes with it: on a DGX Spark's GB10, Qwen3.5 0.8B went from about 113 to 164 tok/s with 2 guesses, and Gemma 4 E2B from about 41 to 108 tok/s with 4. vLLM 0.32's CPU image runs neither: Qwen3.5's first request takes the engine down in a Triton kernel the CPU build lacks (the one that keeps its Gated DeltaNet state in step with the guesses), and Gemma 4's drafter sets up CUDA graphs as it loads.
Prefill is the time to the first token, the network included. vLLM keeps the KV cache of what a prompt shares with the last one (prefix caching), so from step 2 on it reads only the new part, where simple_backend.py reads the whole prompt every step. With media, the expansion request before it has already read most of the prompt. The page times prefill and decoding as the tokens reach it, so Streamlit's full garbage collection after every script run is off (.streamlit/config.toml): it held the GIL for up to a quarter of a second, as a question started and on every redraw, and the step's tokens then reached the page all at once, at thousands of tok/s.
The Model tab names the server and its version; the specs come from config.json and the context limit from the server. The device, the precision and the parameter count are the server's and are not shown, and the folder card lists the weights as on the server only.
Not identical to generate(): vLLM batches requests and has kernels of its own, so greedy output need not match transformers' token for token, as simple_backend.py's does.

How it works tells the two backends apart in tabs at every step where they differ, from what each reads in models/ to MTP, and has a section on the Backend interface they share, all with their code.

vLLM in Docker, started by the page

Without VLLM_URL, the sidebar's What runs the model switches between transformers and vLLM (Docker). Picking vLLM lets go of the model loaded here first (release_model() clears the cache and hands the memory back, on CUDA cuBLAS's workspace too, which would otherwise keep the whole model's memory; a switch made while the model loads takes effect once it has loaded, since Streamlit's fast reruns are off), then agent/model_backends/vllm_docker.py downloads the model into models/ if it is not there yet and starts the server in docker-compose/ with docker compose up -d, showing what Docker and vLLM print meanwhile. Picking transformers again runs docker compose down, and so does quitting the app (Ctrl+C), so only one of the two holds the model. What runs the model is one choice for every tab, as the model in memory is: a tab open on transformers follows another that picks vLLM, and back, and a new tab or a reload joins the server with the model it runs, rather than loading the model here beside it. Switching models on vLLM goes through docker compose up again, which replaces the container; a rerun with the server already up starts nothing. A start that failed (no Docker, Docker not running, out of memory) says why in the page's language and is tried again only when you press Try again; the switch back to transformers stays in the sidebar.

# The same server by hand (VLLM_MODEL and VLLM_REVISION default to Qwen3.5 0.8B at its pinned commit); the weights must be in models/ first:
docker compose -f docker-compose/compose.yaml up -d                                         # on the CPU, any machine
docker compose -f docker-compose/compose.yaml -f docker-compose/compose.gpu.yaml up -d      # on NVIDIA GPUs (Linux, NVIDIA Container Toolkit)
VLLM_URL=http://localhost:8000 uv run streamlit run streamlit_app.py

With just, just vllm-up and just vllm-up gpu run the two docker compose lines, and just vllm-down stops the server.

Where it runs. Where Docker has the nvidia runtime, compose.gpu.yaml goes on top and vLLM's own CUDA image (vllm/vllm-openai:v0.32.0) runs on the GPUs; elsewhere, a Mac included, compose.yaml runs vLLM's CPU image inside Docker's Linux VM. AGENT_LOOP_VLLM_DEVICE=cpu or gpu decides instead, and VLLM_PORT moves the port (8000). On the GPU the KV cache is 2 GiB, as on the CPU (--kv-cache-memory-bytes, VLLM_KV_CACHE_MEMORY_BYTES), rather than 92% of the GPU's memory, and vLLM wants only 30% of that memory free to start (--gpu-memory-utilization, VLLM_GPU_MEMORY_UTILIZATION): at its default 0.92 it refuses on a small card beside the page's own CUDA context, and on a GPU that shares the machine's memory, such as a DGX Spark's.
MTP. On a GPU the server decodes speculatively by default: Gemma 4 with its drafter, Qwen3.5 with the MTP layer in its own checkpoint (model_families.VLLM_SPECULATIVE, handed to Compose as VLLM_SPECULATIVE_CONFIG), and each step underlines the guesses it kept. The sidebar's MTP switch turns it off or back on, which starts the server again the other way, and every tab follows. vLLM 0.32's CPU image can decode neither model speculatively, so on a CPU the switch is disabled and says why. Started by hand, the GPU file runs without it.
What the server reads. It never downloads: models/ is mounted read-only with HF_HUB_OFFLINE=1, so it loads exactly the commit in REVISIONS, as the page does. It listens on 127.0.0.1 only, like the page. It is started with --enable-scale-out, so the page sees every token's rank and the server takes images, sound and video, and keeps vLLM's default raw_logprobs, so the popover shows the model's own probabilities and the top 5. vLLM's compile cache is kept in a Docker volume, so only the first start of a model compiles.
The CPU image (docker-compose/vllm-cpu/) is vllm/vllm-openai-cpu:v0.32.0 with two fixes (patch_vllm.py): vLLM's own #61992, which came a day after 0.32.0 and without which every model's first request fails on a CPU, and one of ours, which makes the input of a CPU linear layer contiguous, without which the first clip Gemma 4 hears takes the engine down. On the CPU, compose.yaml sets the KV cache to 2 GiB (VLLM_CPU_KVCACHE_SPACE; otherwise vLLM wants 92% of Docker's memory), skips measuring the image encoder at startup (over 10 minutes on a CPU), and lets one step take up to 30 minutes rather than vLLM's 5 before it gives up on the engine.
On a DGX Spark (NVIDIA GB10, whose GPU shares the machine's 128 GB): on the GPU, Qwen3.5 0.8B starts in about 2 minutes and decodes about 113 tok/s, Gemma 4 E2B starts in about 3 minutes and decodes about 41 tok/s, and the real-server tests pass for both, every kind of media Gemma reads included. At vLLM's default gpu_memory_utilization (0.92) the server refuses to start there: vLLM takes the machine's available memory for the GPU's free memory, about 108 of 122 GiB. On the same machine's CPU, Qwen3.5 0.8B starts in about 1.3 minutes and decodes about 23 tok/s.
On a Mac (Apple M5, 10 cores given to Docker): Qwen3.5 0.8B starts in about 1.5 minutes, takes about 9 GB of Docker's memory and decodes about 10 tok/s. Gemma 4 E2B needs more memory than Docker Desktop's default: with 16 GB it is killed while loading (the page says so and how to give Docker more), with 24 GB it starts in 3 to 5 minutes and decodes about 4.6 tok/s, but checking at load which media the server takes runs its image, audio and video encoders on the CPU and takes about 10 minutes the first time.

pi as the harness

The page can also hand the loop to pi, a coding agent with a loop, a system prompt and tools of its own (read, bash, edit, write, grep, find, ls). Pick pi under What runs the loop at the top of the sidebar, then under The model pi asks which model it asks, and the page draws what pi does in the same diagram, step cards and event list. Each step's context section shows the request pi sent the model (its system prompt, its tools and the conversation, as JSON), with what the step added marked.

This page's model (the default): Gemma 4 E2B or Qwen3.5 0.8B, on transformers or vLLM, picked with the same model menu and What runs the model as for the built-in harness. pi asks models over HTTP alone, so agent/harnesses/model_endpoint.py serves the model in memory as an OpenAI-compatible chat completions endpoint on 127.0.0.1, behind a token, and pi's extension registers it as the provider agent-loop. Each request is rendered with the model's chat template, encoded and generated as a step of the built-in harness is, and its events go into pi's step: the Context tab shows the string the model read (and, folded under it, the request pi sent), and every token shows with its probability and candidates. The reply streams back to pi, the thinking as reasoning_content and the reply as content, with the tool calls the family parses out of the model's own format as tool_calls at the end. The decoding settings are the sidebar's; Thinking mode reaches the model as pi's thinking level, which the endpoint turns into the chat template's thinking switch.
pi's own models: one pi has credentials for (Claude, GPT, Gemini, or any compatible endpoint in its models.json); no model loads here. An API sends back text, not tokens, so there are no token probabilities, candidates or special tokens, and how the API writes the request out for the model nobody says: a step shows pi's request, the text as it streams, the token counts the provider reported (with how many came from the cache) and the cost at pi's prices.

nvm install                                      # with nvm: installs the newest Node 22 (.nvmrc) and switches to it; nvm use could pick an older 22 already installed, too old for pi. Install pi under it, since nvm keeps global packages per version
npm install -g @earendil-works/pi-coding-agent   # pi 1.4 needs Node.js 22.19 or newer
pi                                               # then /login, once; or set an API key such as ANTHROPIC_API_KEY before starting the app
PI_COMMAND=/path/to/pi uv run streamlit run streamlit_app.py   # if pi is not on the PATH (any command works, npx included)

How the page runs it. agent/harnesses/pi_harness.py starts pi for each question in RPC mode (pi --mode rpc): commands go to its stdin and records come from its stdout, one JSON object per line. It picks the model and the thinking level, turns pi's own retries off (a retry would be another step, so a failed request is shown as an error instead) and sends the question, each command only once pi has answered the last; one of pi's turns (one request to the model, and the tools its reply calls) is one step. pi keeps the conversation in a session file in the conversation's folder, so the next question continues it; New chat removes it.
What it starts pi without. The user's own extensions, skills, prompt templates and context files (AGENTS.md, CLAUDE.md) stay out (--no-extensions and the like), so the page draws pi's own loop, the same on every machine; the models and the credentials are still the user's, in ~/.pi/agent.
Tools. The sidebar picks which of pi's tools the model sees (read, bash, edit and write by default, as pi offers). They run on this computer, in the conversation's folder, without a sandbox: unlike run_python, bash reaches whatever you can. A file you attach is saved in that folder and named in the question; an image also goes with the question to a model that reads images.
The extension. Every pi the page starts loads agent/harnesses/pi_extension.ts. Before each request to the model it sends the request as a notification, which in RPC mode is an extension_ui_request on stdout and becomes the step's request; it registers the page's model when pi asks it; and it is what asks before a tool call (below).
Ask me before running a tool. pi does not ask by itself; with this on, the extension opens a confirm dialog before every tool call. In RPC mode the dialog comes out on stdout, the page shows it as Allow and Reject, and the answer goes back on stdin; a rejected call does not run, and the model reads the refusal in the turn's language.
Stopping. Stop sends pi abort. pi has no step limit of its own, so the page keeps Max steps: when pi starts a turn past it, the page aborts pi. An unanswered confirmation stops pi after 5 minutes, as with the built-in harness.
Tested with pi 1.4.0, through a stand-in that speaks its protocol from a script (tests/fake_pi.py, which asks the model endpoint over HTTP as pi does) and, with RUN_PI_TESTS=1, the real pi, pointed at a scripted model on 127.0.0.1 and at the model endpoint with the fake model behind it.

Supporting more models

The harness (harnesses/simple_harness.py), the decoding loop (simple_backend.py) and the page (render.py) are written once, without a single "if Gemma" branch. Whatever differs between models is gathered in one Family object in model_families/:

Reading the output: parse (splits the output into thinking, tool calls and the reply), thought_prefix, channel_switch (which tokens switch to thinking, a tool call or the reply), stop_tokens
Writing the conversation back: reasoning_key (the field the thinking is kept in, so the chat template can read it), text_with_calls (whether text in front of a tool call is kept)
The page: user_turn, byte_level (whether the vocabulary spells bytes as characters, as Qwen's does, so a token popover shows 如果 instead of å¦Ĥæŀľ), a note on each special token, file_notes and drafter_file_notes (what each file of a checkpoint's folder holds, and the fields whose values the Model tab shows), the help text for the sidebar and the Model tab, and the recommended decoding settings when there is no generation_config.json; each in English and in Chinese
Media: processor_kwargs, settings for the processor (Qwen caps the pixels of each video frame); which kinds a model reads comes from the checkpoint, not the family

To add a model:

Another checkpoint of a family that is already here (a bigger Gemma 4 or Qwen3.5, say): add a line to MODELS in agent/model_families/__init__.py, and one to REVISIONS with the commit to download it at. Other Gemma 4 checkpoints have MTP drafters too (google/gemma-4-*-it-assistant); add a line to DRAFTERS (and the drafter's commit to REVISIONS) and the sidebar's MTP switch works for them.
A new family: write a module like agent/model_families/qwen3_5.py (a parser for the output plus notes on the special tokens, in both languages) and add its Family to FAMILIES. Loading a model finds it automatically by the model_type in the checkpoint's config.json. tests/test_chat_template.py checks a new family against the real tokenizer and chat template: every special token has a note, tool calls the template writes into the history parse back, and what the step card marks as new is exactly the tool result.

Or ask for it: on a vLLM server, picking a model of a family the page does not know offers a GitHub issue already filled in, and an issue for any other model you would like to watch is welcome too, at https://github.com/cyyeh/agent-loop-visualizer/issues.

The two models' formats

Gemma 4 wraps strings in the special token <|"|> and leaves keys unquoted, so it is not JSON, and agent/model_families/gemma4.py has its own small parser:

<|tool_call>call:get_current_time{timezone:<|"|>Asia/Taipei<|"|>}<tool_call|><|tool_response>

After writing a tool call, Gemma stops at <|tool_response> to wait for the harness to add the result; after an answer it stops at <turn|>. These two stop tokens line up with stop_reason: tool_use and end_turn in the Anthropic Messages API.

Qwen3.5 uses XML, and every value is plain text, typed by the tool's schema (as vLLM's qwen3_coder parser does):

<tool_call>
<function=get_current_time>
<parameter=timezone>
Asia/Taipei
</parameter>
</function>
</tool_call><|im_end|>

Qwen stops at <|im_end|> both after an answer and for a tool, so the harness decides whether to end by whether parsing found a tool call, not by the stop token. The tool result goes into a new user message (<tool_response>…</tool_response>) instead of the same model turn as with Gemma.

Tests

uv run pytest                                          # about 40 s; loads no weights and touches no network
uv run ruff check                                      # lint: likely bugs, unused imports and the order of imports (--fix sorts them)
uv run mypy                                            # types: agent/ and app/ inside every function, tests/ where annotated
RUN_MODEL_TESTS=1 uv run pytest tests/test_real_model.py   # loads the real Gemma (with its MTP drafter) and Qwen, about 4 minutes; add -k Qwen for one
VLLM_TEST_URL=http://localhost:8000 uv run pytest tests/test_vllm_backend.py -k real_server   # asks a running vLLM server
RUN_VLLM_DOCKER_TESTS=1 uv run pytest tests/test_vllm_docker.py -k real                        # starts vLLM in Docker with Qwen3.5 and asks it
RUN_PI_TESTS=1 uv run pytest tests/test_pi_harness.py -k real                                  # starts the real pi (on the PATH, or PI_COMMAND) against a scripted model on 127.0.0.1, and against the model endpoint

With just, these are just test, just lint, just types, just test-models, just test-vllm (another server's address after it, if not localhost:8000), just test-vllm-docker and just test-pi; what follows the name goes to the command (just test-models -k Qwen). just check runs the first three and then How it works' tests and build with pygments alone, as checks.yml does.

Most tests use the fake model in tests/fakes.py: it replays scripted output one token at a time and is a Backend (backend.py) like SimpleBackend, so the harness and the page run from start to finish. In pi's place they start tests/fake_pi.py, which speaks pi 1.4's RPC protocol from a script, so they need neither Node nor a model.

Every file format the page accepts has a test: tests/samples.py makes a small file of each one from the examples' files (the receipt in every image format, the recording in every audio format, the rolling ball in every video container, the sales data in every data format). test_media.py fails if media.py accepts a format without a sample, and test_real_model.py asks both models about each one. The tests that need Docker run only when Docker and the sandbox image are there.

On every pull request and every push to main, .github/workflows/checks.yml runs ruff check, mypy and uv run pytest on Ubuntu; the tests that need the models, the sandbox image, a vLLM server or pi skip there. Beside them it runs tests/test_guide.py and builds How it works with pygments alone, as pages.yml does on main, so a pull request that has the page's code import torch or the rest of the app fails before it is merged, not when the page is published.

How it works quotes code, and tests/guide_excerpts.txt keeps everything it quotes, without line numbers. When a change touches quoted code, test_guide.py fails and names the sections that quote it: reread them, fix what they say if it is no longer true, then rewrite the file with UPDATE_GUIDE_SNAPSHOT=1 uv run pytest tests/test_guide.py (just guide-snapshot).

File	What it tests
`test_gemma4.py`	Gemma 4's parser: splitting thinking, tool calls and the reply; `<\|"\|>` strings, nested arguments, all kinds of format errors, in both languages
`test_qwen3_5.py`	Qwen3.5's parser: XML tool calls, types from the schema, thinking, all kinds of format errors; a whole harness run in Qwen's format
`test_model_families.py`	Every family explains the tokens it uses, in both languages and for the same tokens; `model_type` finds the family; a token shows as the text it stands for, and a byte that is only part of a character as `<0xF0>`; every family explains the files of its checkpoints in both languages, naming the same fields and tokens; every model and drafter downloads at a pinned commit; vLLM drafts for a listed model with the MTP layer in its checkpoint or the drafter transformers uses
`test_tools.py`	Schema validation and defaults, the calculator's AST allowlist (rejecting `__import__` and the like), timezones, Wikipedia (with fake HTTP); `run_python`'s locked-down `docker run`, its errors (a traceback, no output, too much memory, too long, no image) and what is missing when Docker is not ready or the image is out of date, in both languages; the charts after the marker saved without overwriting a file, only PNGs, at most four, none from code that failed; the image's version label and its pip installed by hash; with Docker, the real sandbox reads `/data` but cannot change it or reach the network, reads every data format, sends back its charts with Chinese labels drawn, and has a pip (with its certifi) past the emSign fix
`test_simple_backend.py`	`SimpleBackend` has every method and attribute of a `Backend`, and its tokens carry every field and detail; the hand-written decoding loop: KV cache, stop tokens, the token limit, interruption, channel switches, Chinese characters split across tokens; `encode()` handing the processor each kind of media in order with the family's settings, and a short video all its frames; images and videos through the vision encoder before the prefill, and no position offset left from the last prompt; greedy taking the lower id of two tied tokens, as `generate()` does; the candidates and probabilities of temperature / top-k / top-p; MTP with a scripted fake drafter: kept and rejected guesses, the extra token when all hold, how far the KV cache is cut back, the drafter never seeing dropped guesses, and a check that guesses do not change the distribution when sampling; models downloading at their pinned revision into `models/`, which `AGENT_LOOP_MODELS` moves (a relative one from where the app started) and `HF_HUB_CACHE` does not, letting a model go on CUDA freeing cuBLAS's workspaces before emptying the cache, and a model folder's files, their sizes (through the cache's links) and the fields its notes name
`test_simple_harness.py`	The fake model is a `Backend` whose tokens carry what `SimpleBackend`'s do; the harness: event order, tool results back in the context, validation errors, repeated calls, truncated results, every stop reason, Allow / Reject / Stop while waiting; the model reads errors in the turn's language, and the system prompt is the same in both; attachments as parts of the question and a note for `run_python`, the media of every turn going to the processor, the expanded context still marking only the tool result as new, `run_python` getting the conversation's folder, and a turn without `run_python`; the model reading where the charts are while the page gets the images
`test_pi_harness.py`	pi as the harness, with `tests/fake_pi.py` in pi's place: both harnesses are a `Harness`, and pi sends only the events it names; a turn becomes the page's events in the loop's order (the thinking, text and tool-call arguments streamed by channel, the provider's token counts with the cache, the calls and their results); pi started in the conversation's folder on its session, with the sidebar's tools and without the user's extensions, the model set and its retries off before the question goes, no tools at all, attachments named for the model and images sent inline, a repeated call marked, a tool that fails, the token limit, the next question on the same session; asking first through the extension: Allow, Reject with the refusal in the turn's language, Stop while a call waits, a timeout; Stop mid-reply and before pi has started, and the step limit drawing no step past it; a failed request, a refused question, a model pi does not have, pi dying as it starts and pi not installed, said in both languages; which models pi has; with `RUN_PI_TESTS=1`, the real pi against a scripted model on 127.0.0.1, with and without asking first; each step's request with what is new marked and no image bytes; pi asking the page's model through the endpoint (Gemma and Qwen): every token, the context with the tool result as new, the endpoint's own `model_end`, no `delta`, pi's thinking level as the template's thinking, an image reaching the model, Stop stopping the model before its next token, asking first (Allow and Reject, the model reading what came of it), the step limit turning the request past it away at once and leaving the endpoint free; with `RUN_PI_TESTS=1`, the real pi asking the page's model through the endpoint, thinking on and off, and once asking first and stopping at the step limit
`test_model_endpoint.py`	The page's model served to pi, over HTTP on 127.0.0.1 with the fake model behind it: only on this machine, only with its token, nothing while no turn is served, one request at a time; a reply streamed as `content` with its usage, a tool call in Gemma's or Qwen's own format sent back as `tool_calls` and not as the reply, thinking as `reasoning_content` when the request turns it on, a call the family cannot parse not sent, the sidebar's token limit as `length`, a failing backend in the finish reason; the request drawn as the step's context (what is new being the tool result), prefill, tokens and end, one outside every step answered undrawn, a stopped turn stopping the model before its next token; the conversation as `SimpleHarness` keeps it (one system message, the arguments as a dict, the thinking under the family's key, a tool result named by its call) and an image saved once under its hash as a part
`test_render.py`	Every prefix of a run's events renders in both languages (the page redraws every 0.5 s mid-run), HTML tags pair up, the loop diagram is valid SVG, model output is always escaped, special token popovers and lists; MTP's underlines, popovers, counts and drafter card; a run of one special token as one chip with its count, a cut snippet never splitting a token, how many tokens the media took, code and output over several lines written out, tools left out of a turn, the charts of a run and of each step; the model folder card: the files in order with their sizes and notes, a file the family notes but the checkpoint lacks, the drafter's folder, escaping; tokens from a backend that reports no probabilities, or only log probabilities, drawn with what they carry and no more; a run pi ran: its request as the context section with what each step added marked, the text it streamed by channel and no token numbers, escaped, joined without changing the events, the diagram, the stop card and the step table saying pi runs the loop, and pi's cards in both languages; a run of pi on the page's model drawn with its tokens and context; no Chinese left on an English page
`test_app.py`	Drives the whole page with Streamlit's `AppTest`: Streamlit not collecting garbage after every script run, and a rerun waiting for the run in progress; sending a question, the example buttons, sidebar settings reaching the model, multi-turn chat, the Context and Model tabs, Allow / Reject / Stop, a new chat, resetting the settings, two tabs never running at once, switching models, How it works linking to GitHub Pages in the page's language; MTP off by default, the drafter loaded only when it is switched on, and switched off and disabled for Qwen, which has none; an English browser gets English, the last pick beats the browser's language, switching languages keeps the conversation and the settings, and the model's errors follow the language; each page shows only its own language's examples, an example attaches its file and shows only when the model reads that kind, a sample file picked above the input box goes with the next question typed and is offered only when the model can use it, Let the model write code looks for Docker only once it is on and says what is missing, the sales example runs Python on the attached file, a chart shows in the chat and in the step that drew it, and New chat removes the conversation's files; with `VLLM_URL`, the menu lists what the server serves, the sidebar and the Model tab name the server, MTP is the server's, a server with sampling masks gets no greedy decoding or unlimited top-k from the sidebar, the model reads the media the server takes and the page says which it refused and why, the menu leaves out the LoRA adapters the server serves, marks the models of a family the page does not support and starts on the first it does, and picking one of those says why and offers a GitHub issue filled in in the page's language, naming the model (a folder by its name alone) and never the server's address, an error the server sends mid-run is shown in the page's language, and an unreachable server, at load or later, is said in the page's language; without `VLLM_URL`, What runs the model starts on transformers and leaves Docker alone, picking vLLM (Docker) lets go of the local model (nothing keeps it) and starts the server with it, a rerun starts nothing, another model starts the server with that one, picking transformers again stops the container, a new tab or a reload while vLLM runs joins its server with its model and a tab on transformers follows another that picks vLLM and back (the model loaded here once), on a GPU vLLM starts with MTP on and the switch starts it again without (and back, with the reset button), on a CPU it runs without MTP and the switch says why, a tab follows the MTP another tab picked, and a start that failed is said in the page's language, tried again only on Try again, with the switch back to transformers still there; with pi picked under What runs the loop, the intro saying what shows of the model pi asks, pi starting on the page's model with its model menu and decoding settings, a question to it drawing every token, its context and pi's request in the Context tab, and the model's card in the Model tab; on pi's own models, its models and settings in the sidebar and no decoding settings, a question drawn as pi's loop, run in the conversation's folder, the sidebar's settings reaching pi, Allow through the page, Stop, the Context tab showing pi's request and saying what it cannot show, and the Model tab showing pi's model and tools, pi without a model or not installed said with a Try again, Docker never looked for, the built-in harness's model and settings kept across a switch, a switch starting a new conversation, and an English page with no Chinese, on either kind of model
`test_chat_template.py`	With each model's real tokenizer and chat template (only when downloaded into `models/`): every special token is a single token and has a note, every token's popover shows the text it stands for, tool calls the template writes into the history parse back; a harness run with the real template checks that what the step card marks as new is exactly the tool result (two arguments, thinking mode, text before a tool call), and with the real processor, with an image and a video in the question too; every file in each model's folder (and its drafter's) has a note, and every field a note names is in the file
`test_vllm_backend.py`	The vLLM backend against a fake server that speaks the streams of vLLM 0.32: `/v1/completions` gets the rendered prompt's token ids, every sidebar setting with every other sampling setting neutral, the stop tokens and a `max_tokens` that fits the server's context; each token's probability and the top 5, the rank only within them; with `--enable-scale-out`, `/inference/v1/generate` and every token's rank, the top 5 by rank though the chosen token comes first; on either route a token tied for first ranked #1, though vLLM's rank counts the tie; with sampling masks, which vLLM sends only with processed log probabilities, the candidates left and the sampling probability but not the model's own probabilities, and temperature 0 and top-k 0 refused on every route, as vLLM refuses them, which the page finds out at load on a server with masks but without `--enable-scale-out` too, while a server that is not ready is no such refusal; the server's `--logprobs-mode` worked out at load with one token, processed log probabilities without masks showing how many were left when some of the top were cut, logits showing only the rank; greedy as one candidate drawn for sure; which media the server takes, each sent as a file with the placeholders and the Context tab showing the server's expansion, a different expansion at generation time an error, a refused kind a plain file; MTP's counts on either route, the per-step ones marking the guesses only when they add up and counting as kept only those up to the stop token; the stop token, the token limit, Stop closing the stream; an unreachable server, a model it does not serve, one started from a folder of its own or with a model of a family the page (or its transformers) does not know or without a config.json, a missing or failing `/inference/v1/generate`, a rank the server took from the logits staying true to the listed probabilities where they tie, its refusals and errors, in both languages, and an error mid-run shown in the page's language; which family a served model is of, from its config.json alone (a folder on this machine read where it is, a repo without one of no family, and with no way to the Hub or from a folder only on the server not told), and the model id and `model_type` an unsupported model's error carries for the page's GitHub issue; warming up, and a step that ends before its first token; the model card's specs, the server, what its numbers are in each logprobs mode, its sampling masks and the media it refused; a whole turn with a tool call and one with an image, every moment of each rendered; with the real tokenizer and chat template (when in `models/`), the server gets exactly the prompt the Context tab shows; with `VLLM_TEST_URL`, a real server, each feature it was started with
`test_vllm_docker.py`	Starting vLLM in Docker, with Docker faked at the level of `subprocess`: the weights downloaded at their pinned commit before `docker compose up` runs `docker-compose/compose.yaml` with the model, its revision and `models/`, and `compose.gpu.yaml` on top where Docker has the nvidia runtime (or `AGENT_LOOP_VLLM_DEVICE` says so); a server already up for the model not started again, another model replacing it; no Docker, Docker not running, no Compose, `docker compose up` failing, a container killed for memory, one that exits (the exceptions in its log, cause first) and one that never answers, each said in both languages; a rerun while Compose is busy stopping it too; `docker compose down`; the model the app asked for kept through a failed start until it is stopped, and quitting the app stopping the server it started (and leaving Docker alone without one); on the GPU, MTP with Gemma's drafter at its pinned commit (downloaded first) or Qwen's own MTP layer, never on the CPU or unasked, and switching it starting the server again. The compose files: on 127.0.0.1 only, `models/` read-only and offline, Qwen3.5 at its pinned commit by default, the GPU file the same vLLM with the same flags but the CPU-only one, plus a KV cache of the CPU's size, a low `--gpu-memory-utilization`, and speculative decoding off by hand, its counts in detail only with a config (vLLM refuses them without one), Docker Compose handing vLLM the page's config as it is, and Docker Compose reading both (when installed) and mounting the folder a relative `AGENT_LOOP_MODELS` names; with `RUN_VLLM_DOCKER_TESTS=1`, the real server in Docker answering through `VLLMBackend`, with MTP and then without where Docker has a GPU
`test_media.py`	Every format the page accepts has a sample and is read as what it is (sound is a file for Qwen) and decodes; names saved distinct and plain; the question's parts and the note naming a text file's columns; photos turned upright, sound at the model's rate, a video's frames evenly spaced at their own rate; the type each clip plays back as, and the frames shown for an MKV video
`test_guide.py`	Every excerpt and line number on How it works resolves and links to its lines on GitHub at the page's commit, the event table points at the line that really sends each event, HTML tags pair up; the code each section quotes matches `tests/guide_excerpts.txt`, which an edit to that code changes and a line merely moving does not; each paragraph points at the same code and sections in both languages, with no Chinese left in the English one; every set of tabs is radio buttons of known backends or models, one panel each, that switch without a script, and an excerpt of `compose.yaml` ends where it says; the built pages have light and dark palettes and loop diagrams and name their commit, and the root page sends readers to the right language
`test_layout.py`	No file in `agent/` imports `app` or Streamlit, at the top or inside a function, read from each file's syntax tree: the agent never knows the app
`test_real_model.py`	Skipped by default. For each model: the loaded model is a `Backend`; greedy decoding matches `model.generate()`, with text and with an image, a video and (Gemma) a recording, and a text prompt after them too; the real model calls the calculator and answers; it reads the receipt's total in every image format, transcribes the recording in every audio format (Gemma; Qwen gets it as a file), tells which way the ball rolls in every video container, opens every data format with `run_python`, and writes code that runs in the sandbox (with Docker); for Gemma with MTP, greedy output is exactly the same as without it, with an image too, and the drafter's guesses really are kept; on CUDA, a warmed-up model let go hands all its GPU memory back; with pi installed, the real pi asks the model through the model endpoint, calls bash and answers, every token drawn

Known limitations

Speed: on an Apple M5 (32 GB), Gemma runs at about 7–13 tok/s and Qwen3.5 0.8B at about 20–25 tok/s. In thinking mode a step often has two or three hundred thinking tokens, about 30 seconds of waiting. With MTP on, easily guessed steps such as tool calls are about 3× faster, and long Chinese text barely changes (see the table under MTP speculative decoding).
With long replies in the page, MTP speeds things up much less than in simple_backend.py alone (English code: 8 → 29 tok/s alone, 9 → 12 tok/s in the page), for reasons not yet pinned down. Perhaps the page redrawing every 0.5 s and the browser competing with the model for the same machine slow MTP down more than plain decoding, since each MTP round has several small drafter forward passes.
With transformers only Gemma 4 E2B can use MTP: transformers supports a drafter reading the model's KV cache this way only for Gemma 4, and Qwen3.5 has no matching drafter. On a vLLM server with a GPU, Qwen3.5 can too, with the MTP layer in its own checkpoint (see Running the model on a vLLM server).
With MTP on, the model computes 7 positions at once, which gives very slightly different numbers from computing 1; when two tokens are almost equally likely, greedy decoding may now and then pick the other one.
18 of Qwen3.5's 24 layers are Gated DeltaNet. transformers has two kernels to speed them up, flash-linear-attention (Triton) and causal-conv1d (CUDA). Both need a CUDA GPU, so neither is a dependency: the PyTorch reference implementation runs instead, with the same results, only slower, and simple_backend.py silences transformers' notice about that fallback on every device. With a GPU the difference is small for a 0.8B model. On an NVIDIA GB10 (DGX Spark), flash-linear-attention made prefill about 4× faster (3,571 tokens in 93 ms instead of 368 ms) but decoding barely changed (56 → 60 tok/s), and both paths stayed equally close to float32. causal-conv1d has no wheel for that machine and would have to be compiled. Don't install flash-linear-attention on a Linux machine without a GPU: transformers still picks it, and the first forward pass fails.
On MPS, transformers hangs loading Qwen3.5's weights with several threads, so on MPS simple_backend.py loads with one thread (HF_DEACTIVATE_ASYNC_LOAD=1), which does not make either model load slower.
Qwen3.5 0.8B has thinking off by default; on, it easily keeps thinking without stopping, and lowering Max tokens per step helps. It ships no generation_config.json, so its recommended decoding settings are in agent/model_families/qwen3_5.py (the presence penalty it recommends is not implemented here).
Only one model is kept in memory. If two tabs pick different models, using them in turn keeps reloading; it is best for every tab to use the same model.
There is one copy of the model, so only one agent runs at a time and other tabs wait. Reloading or closing the tab during a run does not stop the agent; other tabs (including the reloaded one) show it is still running and offer a Stop that agent button.
With Ask me before running a tool on, if no one answers within 5 minutes the harness takes it as a rejection and stops the loop, with "confirmation timed out" as the stop reason.
Token popovers appear only once the step finishes generating; while it generates there are only the heat colors (special tokens still explain themselves), so each redraw does not send a lot of HTML. The popovers are pure CSS, and one for a token too near the bottom of the timeline may be cut off.
Special token popovers are placed next to the token with CSS anchor positioning, so a scrolling box does not cut them off; browsers without it open them right under the token, where they may be cut off near the bottom of the box.
Both models are small: they may answer in Simplified Chinese, repeat the same search, answer half the question or get their sums wrong while thinking. Asked in English about two cities' times at once, Gemma calls the tool twice and then usually answers in Chinese, or stops right after the tool results without a reply; asked about one city, it answers in English. All of that is material to look at.
Only Gemma 4 E2B hears, and only the first 30 seconds of a clip. Neither model hears a video's sound: a video is frames. Each frame costs tokens: Gemma takes 32 frames of any video, about 2,000 tokens of prefill; Qwen two a second, so its count grows with the video's length (640 tokens for the 8-second example).
In English with Use tools on, Gemma 4 E2B often says it cannot hear an attached recording, sometimes after writing its words into a tool call; asked in Chinese, or with Use tools off, it transcribes it. So the English recording example asks about what is said rather than for a transcription, which works about two times in three.
Qwen3.5 0.8B opens every data format with run_python but rarely gets multi-step pandas code right; Gemma 4 E2B answers the sales example.
The sandbox has no network, so the model cannot pip install; it has numpy, pandas, openpyxl, tabulate and matplotlib. Each call is a new container, so nothing (variables, files in /tmp) carries over to the next call; only its charts, saved in the conversation's folder.
The model never sees the charts it draws: they reach the page, not the context, so what it says about one comes from its own code and numbers. A chart saved with plt.savefig() and then closed is not sent; one shown or left open is.
In the chat, a clip plays back in the browser's own player; Safari cannot play an MKV video, so for one the chat shows four of the frames the model reads instead.
Attachments are up to 50 MB each. They stay under ~/.cache/agent-loop-visualizer/uploads/ until New chat or a model switch; a closed tab leaves its folder behind.
The Wikipedia tools need the network.
On a vLLM server (VLLM_URL) greedy output may differ from generate()'s, and what the page shows depends on the server's flags (see Running the model on a vLLM server): without --enable-scale-out the model reads text only, and a token's popover has either the model's own probabilities and the top 5 (the default) or, with --logprobs-mode processed_logprobs (which --return-sampling-mask needs), the sampling probability and how many candidates were left, never both; which of the top 5 were cut never shows outside greedy decoding, and a server with sampling masks takes no greedy decoding or unlimited top-k. MTP's guesses are underlined only once the step is done, only when the server's counts account for every token, and without what a turned-down guess was. With media, each step asks the server for the expansion first, a second request that reads the prompt. The page cannot see which revision the server loaded, so start it with the --revision there.
vllm_backend.py follows the protocols of vLLM 0.32, read from its source, and is tested against a fake server that speaks them, with the real tokenizers and in the real page; /inference/v1/generate and its sampling masks and per-step counts are marked experimental in vLLM and may change. Its tests against a real server (VLLM_TEST_URL) have been run on one GPU only, a DGX Spark's GB10.
vLLM in Docker runs on the CPU on a Mac, and slowly: Gemma 4 E2B decodes about 4.6 tok/s, needs Docker to have about 20 GB of memory or more, and its first load spends about 10 minutes running its media encoders to see which kinds the server takes. compose.gpu.yaml has been run on one NVIDIA GPU only, a DGX Spark's GB10. The CPU image patches vLLM 0.32.0 (docker-compose/vllm-cpu/patch_vllm.py); a newer vLLM may need neither fix, and the build stops if the code it patches is not the one it was written for.
The app stops vLLM's container when it quits with Ctrl+C or a SIGTERM. Stopped any other way (its terminal closed, kill -9), it leaves the container running and holding the model; docker compose -p agent-loop-vllm down (just vllm-down) stops it.
With pi on its own models, the page sees only what pi reports: no tokens, probabilities or special tokens, and not the string the model reads, only the request pi sends. The token counts are the provider's, tok/s is measured as the reply arrives (the network included), and the cost is at the prices in pi's catalog, an estimate. pi's own models are a provider's: they cost money, and the conversation goes to that provider.
With pi on the page's model, a tool call the family cannot parse is not sent to pi, and pi's turn ends there: unlike the built-in harness, nothing tells the model what it wrote wrong. pi's system prompt and tool schemas are written for large models; Gemma 4 E2B and Qwen3.5 0.8B follow them less well than the built-in harness's prompt. pi sends images alone of the media, so audio and video attached for pi are plain files in its folder. A request that comes outside every step (pi compacting a long conversation, say) waits up to 10 seconds for one, then is answered without being drawn.
pi's tools run on this computer, without a sandbox; with bash on, the model can do whatever you can. Ask me before running a tool shows each call first.
pi_harness.py follows pi 1.4's RPC protocol, from its documentation and the records pi 1.4.0 sends; the real-pi tests have been run with pi 1.4.0 only. pi 1.4 needs Node.js 22.19 or newer: under an older Node it does not start, and the page shows what it said. The page asks pi which models it has at most once a minute; after logging in to a provider in pi, Try again lists them.
The MODEL_ID environment variable picks the model selected at first; it may also name a checkpoint the menu does not list, as long as it belongs to a supported family (Gemma 4 or Qwen3.5); another family needs one added first, as in Supporting more models.

License

This project's code is licensed under the Apache License 2.0. The Gemma 4 weights and its MTP drafter are released by Google under Apache 2.0 (see the model card and the drafter's card), and the Qwen3.5 weights by Alibaba's Qwen team under Apache 2.0 (see the model card); none of them are included in this project. app/assets/examples/speech.flac is utterance 1272-128104-0000 of the LibriSpeech ASR corpus (Panayotov et al., 2015), read from LibriVox, under CC BY 4.0; the other example files (receipt.png, ball.mp4, sales.csv) and the logo were made for this project.

About the use of AI: the code was written with the help of Claude Code, run on this machine with gemma-4-E2B-it (and its MTP drafter gemma-4-E2B-it-assistant) and Qwen3.5-0.8B, and checked in a browser.

Navigation Menu

Repository navigation

About

Repository files navigation

Agent Loop Visualizer

What the page shows

Quick start

The commands by name

Experiments to try

Architecture

Two languages

Why a hand-written decoding loop

MTP speculative decoding

Images, sound and video

Writing code in a Docker sandbox

Running the model on a vLLM server

vLLM in Docker, started by the page

pi as the harness

Supporting more models

The two models' formats

Tests

Known limitations

License

About

Releases

Packages

Contributors

Languages

Name		Name	Last commit message	Last commit date
.github/workflows		.github/workflows
.streamlit		.streamlit
agent		agent
app		app
docker-compose		docker-compose
docs		docs
tests		tests
.gitignore		.gitignore
.nvmrc		.nvmrc
.python-version		.python-version
AGENTS.md		AGENTS.md
CLAUDE.md		CLAUDE.md
LICENSE		LICENSE
README.md		README.md
README.zh-TW.md		README.zh-TW.md
justfile		justfile
pyproject.toml		pyproject.toml
streamlit_app.py		streamlit_app.py
uv.lock		uv.lock

Latest commit

History

Folders and files

Repository files navigation

Agent Loop Visualizer

What the page shows

Quick start

The commands by name

Experiments to try

Architecture

Two languages

Why a hand-written decoding loop

MTP speculative decoding

Images, sound and video

Writing code in a Docker sandbox

Running the model on a vLLM server

vLLM in Docker, started by the page

pi as the harness

Supporting more models

The two models' formats

Tests

Known limitations

License

About

Topics

Resources

Stars

Watchers

Forks

Releases

Packages

Contributors

Languages