feat(gguf): serve GLM-5.3-Flash GGUF checkpoints on glm5-next

[void-mckenzie]

Part of #671 (Model Support: glm5-next).

Follows the per-family GGUF adapter pattern established for qwen4exp in #690 (config from KV metadata through the family's own parse_config, checkpoint-leaf to_gguf map, fused() state-dict emission, GGUF expert banks, text-only TP=1).

What

Serves llama.cpp glm5-next GGUFs — unsloth GLM-5.3-Flash-GGUF — on the GGUF quant kind from #690, reusing the existing glm5_next model classes:

• config: parse_gguf_config builds the text config from the GGUF KV metadata. The new llama.cpp writer carries attention.head_count_kv as a per-block list (0 = KDA, 1 = MLA), which encodes the 34/11 hybrid layout directly; unsloth's earlier glm5next spelling predates it and falls back to a tensor-name census. The trailing MTP block (nextn_predict_layers) is dropped exactly like the HF loader drops the MTP layer. MLA head widths come from the new *_length_mla keys with a q_b/v_b-rows fallback. The served-model facts the KV cannot express (NoPE, interleaved indexer rope) are pinned to the HF release values.
• weights: iter_gguf_weights maps every tensor into the glm5_next state dict and undoes the converter transforms: A_log is stored pre-transformed as -exp(A_log) and inverted at load; the MLA kv_b_proj is split per-head as attn_k_b/attn_v_b (k transposed) and re-fused into the dense [H*(nope+v), lora] layout the module's absorption reads; the q|k|v conv kernels concatenate on the channel axis. The mixed-type fused KDA in_proj (Q8_0 q|k|v + BF16 b|f_a|g_a, as shipped) rides the feat(gguf): run experts whose gate_up and down mix float and quantized types #702 segments; kv_b, the indexer projections and the router stay dense where the file stores floats.
• experts: the routed experts load as GGUF expert banks at their shipped per-layer types — unsloth dynamic quants (IQ3_S gate/up + IQ4_XS down with Q8_0 outlier layers) pass through natively. load_gguf_expert_banks gains the dense-MLP prefix offset (first_k_dense_replace): glm5-next's first MoE layer sits mid-stack, and the banks walked blk.0 before.
• registration: Glm5NextGGUFForCausalLM registered and claimed in the AOT table; also claims the two qwen3.5 GGUF archs feat(gguf): serve Qwen3.5 dense and MoE GGUF checkpoints #701 left unclaimed (that omission breaks test_every_registry_architecture_is_claimed_by_an_aot_entry on current main). embed_tokens takes quant_config so the Q8_0 token_embd serves packed.

Verified

• Against unsloth GLM-5.3-Flash-IQ4_XS (5 split shards, 159 GB): the full 45-layer engine (RTX 5090 32GB, 40k context, 6 GiB expert slot cache) loads 43 layers of IQ3_S/IQ4_XS experts as host banks and serves coherent, correct output over /v1/chat/completions; the tokenizer, eos set and chat template come from the GGUF via the shared loader.
• Measured, bs=1 on that setup: prefill ≈ 1.6k tok/s steady-state on 8k-token chunks (29k-token prompt in 4 chunks; first-ever chunk pays a one-time Triton autotune), decode ≈ 14–16 tok/s — expert-fetch bound (top-8 of 288 × 43 layers streaming from pinned host banks over PCIe).
• Config parse field-for-field matches the zai-org/GLM-5.3-Flash config.json through glm5_next.config.parse_config; the state dict matches the constructed model 1042/1042 tensors (names + shapes).
• Unit tests build a synthetic miniature glm5-next GGUF (both arch spellings, head_count_kv present and stripped, trailing MTP block) and cover the config parse, the full state-dict mapping and each converter transform.

Text-only, TP=1 — same status as the other GGUF families. Vision (mmproj sidecar) is a natural follow-up.

[void-mckenzie]

Test stats & setup

Workload results (bs=1, single RTX 5090):

Test	Result
Prompt processing, 29,207-token prompt (4× 8,192 chunks)	steady-state ~1.6–1.7k tok/s per chunk (log input throughput: 1599.6 / 1678.1 on the middle chunks); first-ever chunk pays a one-time Triton autotune at the first big prefill shape; whole request 40.6 s incl. tokenizing 103 KB
Decode, coding task (write a SQLite-backed LRU cache module), model-default sampling, 600 tokens	14.3–15.6 tok/s steady-state (log gen throughput, 40-step interval); 44.2 s end-to-end — expert-fetch bound: top-8 of 288 experts × 43 MoE layers streaming from pinned host banks over PCIe into the 6 GiB slot cache
Functional	coherent, factually correct chat + code generation via /v1/chat/completions

Model: unsloth GLM-5.3-Flash-IQ4_XS (5 split shards, ~159 GB), arch glm5-next. Geometry: 45 decoder layers (34 KDA linear + 11 DSA NoPE-MLA with kpool-4 indexer) + 1 MTP block (dropped at load), hidden 4096, 288 experts top-8 + 1 shared, mHC×4, vocab 154,880. Quant mix as shipped: experts gate/up IQ3_S + down IQ4_XS with one all-Q8_0 outlier layer, embedding & lm_head Q8_0, kv_b / indexer projections BF16, norms / routers / mHC F32.

Launch: ft serve --gpu 0 --max-seq-len-override 40000 --moe-cache-mem 6GiB --memory-ratio 0.82 → MoE offload (552 expert slots × 11.12 MiB unified cache), radix cache, bf16, TP=1. One serving note: at the default --memory-ratio 0.9 with a 40k window the KV pool leaves the GPU with ~50 MiB free and the first-use Triton autotune (float-expert grouped GEMM) OOMs the worker — 0.82 leaves ~5.5 GiB headroom and is what the numbers above ran with.

Host: Ryzen 9 7950X3D (32 threads), 187 GiB RAM (~130 GiB pinned expert banks), RTX 5090 32 GB (sm_120), driver 580.95.05, CUDA 13.0, Linux 6.8.0-136-generic, shards on a Crucial T500 2 TB NVMe. Stack: Python 3.10.19, torch 2.14.1, triton 3.8, transformers 5.18.0.

@jason-fxz Upstreaming my private branch that I was using for GLM 5.3 Flash GGUF. Rebased to current main to align this branch with the gguf direction chosen by the Repo. I also have a branch up for deepseek v4 privately, if you'd like me to help you finish these gguf ports, let me know :)

[jason-fxz]

Could you first rebase onto the current main? The branch conflicts with it, and these GGUF changes landed since its base:

• feat(moe): serve experts of several formats through per-format slot pools #700: experts of several formats through per-format slot pools (touches moe/expert_banks.py and glm5_next/weight.py)
• feat(gguf): serve Llama and Mistral llama-arch GGUF checkpoints #722 / feat(gguf): serve Muse-Glimmer GGUF checkpoints #724 / feat(gguf): serve MiniMax-M2 and MiniMax-M3 GGUF checkpoints #725: Llama, Muse-Glimmer and MiniMax GGUF loaders, the pattern a new family follows; feat(gguf): serve MiniMax-M2 and MiniMax-M3 GGUF checkpoints #725 adds load_gguf_expert_banks(first_layer=...)
• feat(cpu-moe): serve GGUF experts on the cpu and hybrid backends through ggml-cpu #704: GGUF experts on the cpu and hybrid MoE backends through ggml-cpu
• fix(gguf): set ignore_merges for llama 3 gguf tokenizers #728 / feat(gguf): serve GLM-4.5, GLM-4.6 and GLM-4.7 GGUF checkpoints #731: ignore_merges for Llama 3 tokenizers, and the glm4 pre-tokenizer split, which GLM-5.3 uses too
• feat(gguf): apply the weight scales of nvfp4 gguf tensors #732: NVFP4 weight scales (check_scales)

Then, before this goes further, could you run the GGUF load check on the GLM-5.3-Flash files? It is on the feat/gguf-check-tool branch (no PR yet), under scripts/gguf_check/, and runs against your checkout:

git fetch https://github.com/FlashML-org/FreeToken feat/gguf-check-tool
git checkout FETCH_HEAD -- scripts/gguf_check

# header-only stubs of every GLM-5.3-Flash GGUF on HF (about 15 MB each)
PYTHONPATH=python python scripts/gguf_check/fetch_headers.py ~/gguf-stubs --search GLM-5.3-Flash > fetch.jsonl
# kernel table, once per GPU architecture
PYTHONPATH=python python scripts/gguf_check/probe_kernels.py kernels.json
# CPU only, no weights read
PYTHONPATH=python python scripts/gguf_check/check.py --kernels kernels.json --out glm53.jsonl ~/gguf-stubs

For each file it checks that the config parses, that the model builds and the reader yields exactly its state dict, that the expert banks build with the right shapes, that every ggml type has its kernels, and that the tokenizer and chat template load. README.md has the details.

The bar: the common quants from unsloth, ggml-org, bartowski and others should pass. Every failure should be a clear refusal from the loader or a broken file. Please post the check.py --report glm53.jsonl summary here.

These failures are things we do not serve today, so you can ignore them:

• IQ1_M: no MMQ kernel, so it fails above 8 tokens per batch
• TQ1_0 / TQ2_0: no ggml-cuda kernel
• N is not a valid GGMLQuantizationType: ik_llama.cpp types (IQ2_KT, IQ4_KSS, ...) and the removed Q4_0_4_4 / Q4_0_4_8 / Q4_0_8_8
• NVFP4 token_embd
• experts whose gate and up use different ggml types
• MTP-only draft files
• tokenizer: no chat template: base models
• runs past the end of the file: truncated uploads
• architectures listed under "not checked" (draft models such as dflash, eagle3)

For NVFP4 load: the .scale / .input_scale tensors are handled since #732. Please call check_scales from models/gguf/tensors.py the way the llama, qwen3 and gemma4 loaders do, instead of refusing these files.