[ajensenwaud]

What this changes

Muse Spark Open's chat template has the model reply as a series of messages, each addressed to a recipient:
to=self<|message|>…<|eom|><|start|>assistant to=user<|message|>…<|eot|>. Tool calls are messages addressed to the
tool, and their bodies are <atem:function_calls> blocks. The server on main does not recognise this format, so the
whole reply, markers included, ends up in content. This PR adds zig/src/server/recipient_reply.zig (270 lines):

• Detection (detect). The format is used when <|start|>, <|message|>, <|eom|> and <|eot|> are each a
  single token and the chat template contains to=self. This is set once in Server.init as srv.recipients,
  alongside the existing Harmony and Gemma channel-marker check. There is no hook per model family.
• Split (split):
  • self goes to reasoning_content.
  • user, commentary and a missing recipient go to content.
  • Any other recipient is a tool call. Its <atem:invoke>/<atem:parameter> block is rewritten as the
    Qwen3-Coder <tool_call><function=…><parameter=…> markup that tool_parse and tool_stream already read, so
    the OpenAI tool_calls arguments get the tools' schema types (an integer parameter comes out as 3, not "3").
  • Attribute escapes in names (esc_attr) are undone. Parameter values are kept byte for byte, as the template
    writes them.
• Streaming. Three things are held back until they finish: an unfinished header, a call message that has not
  closed yet, and a trailing partial marker. As a result, each streamed split extends the previous one, and the
  finished split extends them all. Leading whitespace is stripped from each piece in both modes, so this prefix
  property holds.
• A chat's reading (Reading.of). Each chat reads its reply with the request's tools and thinking switch, and
  with the message its prompt left open (a prompt that opened the self message keeps the reasoning that follows).
  A streamed read goes on from the last call's bytes, so streaming costs no rescans; it equals a fresh read at every
  length.
• Ends (End): still streaming, the model's end token, a stop string, or max_tokens. A call that max_tokens
  cuts is the reply's cut tool call, whole and streamed, as GLM's cut call is; the one-call policy hides it.
• Thinking off stays off: self messages stay out of the reply, and the template, which has no switch, hears its
  lowest effort (Server.off_effort, minimal coerced to the template's levels) instead of its default.
• chat.zig (596 lines merged with main 15a0b2a9):
  • A chat on a recipient server, other than a raw prompt, gets a Reading; the streamed and finished splits use it.
    Generation.thinking is unchanged, so logprobs and reasoning counts follow the request's switch.
  • When the reply finishes, the call streamer is fed once more for recipient replies. Without this, a last call
    ended by <|eot|> (the end token, which is stripped) would never stream, even though earlier calls ended by
    <|eom|> did.
  • The sampling-field flag reader moves unchanged to fields.zig, keeping chat.zig under 600 lines beside
    main's loop-guard stop.

gpt-oss's Harmony tokenizer also uses <|start|> and <|message|>, but it has no <|eom|> or <|eot|>, and its
template never writes to=self. So srv.recipients stays off and Harmony replies still go through parseHarmony.

The old gated engine's 0023 chat-rule hook is not carried over: detection uses the markers and the template instead.
This replaces the recipient half of the old patch 0047. The generation_config.json EOS half is already on main.

Receipt

• Environment: this branch merged with up/main 15a0b2a9, Zig 0.17.0, x86_64 Linux host (no GPU needed). Served on the
  muse_spark family (Muse Spark Open on CUDA: EXL3, DFlash2, two GPUs #1295) at TP=2 on two GB10s: one chat and one tool call against the real checkpoint parse into
  reasoning_content, content and tool_calls (get_weather with typed arguments).
• Exactness: n/a. No engine, sampling or token path changes. Only server-side text splitting and tool parsing change.
• Decode speed before and after: n/a, as above.
• Prompt speed before and after: n/a. The change runs only after tokens are decoded and cannot reach prompt
  processing.
• Tests added (zig/src/server/recipient_reply_test.zig, 431 lines, 10 tests, and recipient_cut_test.zig, 69 lines,
  3 tests, both registered in server/root.zig):
  • Detection is on for the Muse markers plus a to=self template. It is off for a Harmony tokenizer and template.
  • A finished split of a reply with reasoning, commentary and two calls gives the reasoning and the answer with
    both <tool_call> blocks. Every streamed prefix only grows, never shows <| or atem, holds the open call
    back, and is a prefix of the finished split. A reply with no markup is returned plain.
  • calls + tool_parse.parse gives {"city":"Rio de Janeiro","days":4} against a schema with an integer
    days. A name escape (a&b) is undone.
  • End to end through openai.run with a byte tokenizer whose markers are single tokens:
    • A plain chat, whole and streamed at strides 1, 7 and 4096, gives the same reasoning_content and content.
    • A two-call reply gives the same reasoning, content: "Checking both.", two calls and identical
      schema-typed arguments, whole and streamed. The streamed case checks that the last call, ended by the end
      token, is streamed.
  • A Harmony reply (<|channel|>analysis / final), whole and streamed, still gives content: "Blue sky." and
    reasoning_content: "Think.".
  • A prompt that opened the self message keeps the reasoning that follows it; a call to a tool not offered stays
    as the model wrote it; thinking off keeps self out of the reply and the template at its lowest effort; a
    streamed read equals a fresh read at every length; a reply with no markup streams as it comes.
  • recipient_cut_test.zig: a call max_tokens cuts is the reply's tool call, cut, whole and streamed at every
    stride; a call cut before its invoke names the tool is the header's call; the one-call policy hides it.
  • The Muse tests fail before this change: the markup lands in content and there are no calls. The Harmony test
    passes before and after, as a regression guard.
• Lean check: python3 tools/zig/lean_check.py reports 428 problems on up/main 15a0b2a9 and 428 merged with this
  branch, so there are 0 new findings.
• git diff --check: clean.
• Tests run, merged with 15a0b2a9: zig build test-server-cpu 881/881 passed (the 13 new tests included), and
  test-golden passes. zig fmt --check is clean.
• Served: the six Muse cases (chat and a tool call, thinking on and off, and the call cut by max_tokens, each whole
  and streamed) on the muse_spark family (Muse Spark Open on CUDA: EXL3, DFlash2, two GPUs #1295) at TP=2 on two GB10s, on be4286a2 (the last commit only moves
  flag).
• Not run: Metal (server-only change; no backend code).

Checklist

• Every token still goes through the lane rounds. No serial path, and nothing that needs drafts off.
• Drafted output equals "draft": false, a resumed prompt equals a fresh one, and concurrent equals solo. (No
  token path change.)
• No precision traded for speed. No bits change.
• A new or changed GPU kernel is registered with the kernel contract. (None.)
• Prompt processing is no slower than the last release. (Not reached.)
• The description says which platforms I ran, Metal M1 to M5 and CUDA, and which I could not. (x86_64 host tests; served on CUDA GB10 via Muse Spark Open on CUDA: EXL3, DFlash2, two GPUs #1295.)
• New tests fail before the change, pass after it, and skip cleanly without their dependency.
• Comments and docstrings are one line. No measurements or history in the source.
• No personal data, machine names, internal hosts or local paths. No AI attribution lines.

[ashhart]

Thanks @ajensenwaud for splitting this out. Rebased onto current main, it conflicts with #1299's cut tool calls in chat.zig; we kept main's token_ids and length cut condition and added or srv.recipients in both places (please confirm). Other families look unaffected: everything is gated on srv.recipients, which no template on main enables, and the Harmony test and goldens pass. Before landing:

1. A call cut by max_tokens before <atem:invoke name="…"> completes falls back to the raw body, so content shows <atem:function_… markup, whole and streamed (recipient_reply.zig around line 109). That's the class #1298/#1299 fixed for GLM. An unclosed call shouldn't fall back to markup; please add cut tests mirroring #1299's.
2. recipient_reply.zig (around line 87) assumes the prompt ends at <|start|>assistant. If the generation prompt already opens to=self<|message|> (as the test's own fake template does), the reasoning is silently dropped. Please handle both, or show the real template only produces the first.
3. Without tools in the request, call messages are rewritten as <tool_call> XML in content, which is markup the model never wrote.
4. chat.zig (around line 242) turns thinking on for recipient models even when the request asks for thinking off.
5. No family on main produces this format yet, so this would land with (or just before) the Muse Spark family.

[ajensenwaud]

Fixing this.

[ajensenwaud]

Done, fixes are in a new commit (your rebase resolution is right):

1. A call cut by max_tokens now comes out as the reply's cut call, like server: a call max_tokens cuts is the whole reply's tool call too (#1238) #1299, with cut tests mirroring yours.
2. A prompt that already opened to=self keeps its reasoning. The real template always ends at <|start|>assistant.
3. Only calls to offered tools become <tool_call>; anything else stays as the model wrote it.
4. Thinking off is respected, and the template gets its lowest effort rather than its default.
5. Agreed, it lands with or just before the family.

[ashhart]

Review against main 03c2820. This looks right: detection needs the four markers as single tokens plus a to=self template, so every other family's path is unchanged, and the gpt-oss/Harmony case is covered by a test. The second commit's cut-call handling, offered-tools-only calls and thinking-off behaviour are good additions. Host checks here: test-server-cpu and test-golden pass with the 11 new tests (no unexpected golden differences), fmt and lean clean, and with detection disabled 7 of the new end-to-end tests fail, so they test what they claim.

The one thing holding it: no family on main emits this format until Muse Spark lands (#1295), so there's nothing to device-check yet. We'll land it with #1295 (or with whichever first family uses it) after a served chat and tool call, whole and streamed, including a call cut by max_tokens and a thinking-off run. Two small notes for then: the streamed split re-reads the whole reply on every token, and a reply with no markup shows nothing until it ends.

[ajensenwaud]

Both of your notes are fixed in be4286a. A streamed reply now reads only the text since the last token instead of the whole reply again, and a reply with no markup streams as it comes. While streaming, a message's trailing whitespace is held back, so the streamed text ends up the same as the whole reply. The new tests check that a streamed read matches a fresh read at every length.

Served check on Muse (built from #1295's head, which carries this), temperature 0, each case whole and streamed, 6/6 pass:

• chat, thinking on and off
• a tool call, thinking on and off (get_weather with JSON arguments, finish tool_calls)
• the same call cut by max_tokens at 48 and 92 of 96 tokens (finish length, the cut call reported as a call, no raw markup)

In every case the streamed content, reasoning and calls match the whole reply.

[ashhart]

Thanks @ajensenwaud. Both notes are fixed, and the tests that compare a streamed read with a fresh read at every
length are the right check. Rebased onto current main, it applies cleanly. fmt, lean, zig build test,
test-server-cpu and test-golden pass here, and the Linux CUDA server builds. Every change stays behind
srv.recipients, so other families are unchanged.

Nothing else holds it. We'll land it with #1295 once that's ready, after repeating your six served Muse cases on our
side: chat and a tool call, thinking on and off, cut by max_tokens, each whole and streamed.

[ajensenwaud]

One commit on top, 2ff98f8b: main's loop-guard stop (#1463) brought chat.zig to 600 lines, and with this PR's recipient hook it reaches 603 once merged, past lean's limit. The sampling-field flag reader (Python's bool(fields.get(key))) moves to fields.zig unchanged, and chat.zig calls fields_mod.flag. Merged with main 41abcf40: test-server-cpu, test-golden and compile-config-server pass, zig fmt --check is clean, and lean finds 428 problems, as main.

[ashhart]

Thanks @ajensenwaud. Moving flag into fields.zig unchanged is the right way to keep chat.zig under 600; it is 596 lines on the merge. We merged 2ff98f8 onto main a1e6d34 (clean). fmt and lean are clean, test-server-cpu and test-golden pass with no unexpected golden differences, and the Linux CUDA server compiles. The server files are byte-for-byte the ones #1295 carries, and the loop-guard stop and the recipient split sit together correctly.

One small thing: the description still gives the first push's numbers (recipient_reply.zig at 116 lines, chat.zig at 599, 300 test lines, lean at 435, 743 tests). Please refresh it so it matches the branch.

Nothing else holds this PR. It lands with #1295, after we repeat your six served Muse cases on our GB10 pair: chat and a tool call, with thinking on and off, and the call cut by max_tokens, each whole and streamed.

[ajensenwaud]

Thanks @ashhart. The description is refreshed to match the branch: recipient_reply.zig 270 lines, chat.zig 596 on the merge, the two test files (431 + 69 lines, 13 tests), lean 428 as main, and test-server-cpu 881/881 merged with current main (db3fd5ab). It also now describes the later commits: a chat's reading of its prompt's open message, the cut call, thinking off staying off, and the flag move.