--- type: runbook title: frank-primary — GLM-5.3-Flash W4A16 int4 (tcclaviger, 2x R9700) description: VERIFIED 2026-09-30 by boot probe, bench and live expert-cache telemetry. frank-primary cut over from Qwen3.8-Flash-Next-Uncensored-FP8 + MTP4 to Intel/GLM-5.3-Flash-W4A16-AutoRound, the first payload this fleet serves through the fork's W4A16 expert-offload path (INC scheme -> R4dW4a16MoEMethod, libr4d grouped W4A16 GEMM). Measured decode 9.9-10.0 tok/s and prefill 918 tok/s at 131,072 ctx. THE 35 tok/s TARGET IS NOT MET and this page shows why it is out of reach: the step is bounded by the fork's expert-gather read path at ~8.2 GB/s effective against a PCIe link that delivers 22.74 GB/s, with the cache at its measured hit-rate ceiling (0.53-0.66 per layer) from an arena that cannot grow past ~15 GiB under a 73.64 GiB/rank expert set. tags: - dankserver - r9700 - gfx1201 - glm5.3 - w4a16 - int4 - expert-offload - expert-cache - autoround - vllm timestamp: 2026-09-30 --- # frank-primary — GLM-5.3-Flash W4A16 int4 (tcclaviger, 2x R9700) > **Question this page answers.** Can the fleet serve the 320B-class > `GLM-5.3-Flash` at int4 on two R9700s with expert offload, and how fast is it? > **Answer.** Yes — it serves, with vision, tool calls and 131,072 ctx. It runs > at **9.9-10.0 tok/s decode / 918 tok/s prefill**, and **35 tok/s is not > reachable** with this payload on this hardware. The bound is not the model, > the quant, or the link: it is the fork's expert-gather read path, and it is > measured, not inferred. Related: [[dankserver-qwen38-fp8-expert-offload-tuning]] (the same fork, the same hardware, the same gather-path wall, found first on qwen), [[dankserver-frontier-moe-feasibility]] (the earlier fit analysis), [[VLLM Index]], [[LLM/index|LLM fleet]]. ## The deployed shape | term | value | | --- | --- | | payload | `Intel/GLM-5.3-Flash-W4A16-AutoRound` — AutoRound **symmetric int4, group 128**, packing `auto_round:auto_gptq`, 34 shards + 2 extra safetensors, 113,074 tensors, ~170 GiB | | runtime | `docker.io/tcclaviger/vllm@sha256:da4029666…` (same digest-pinned image as the qwen cutover) | | engine | TP=2, `--enable-expert-offload --expert-precise-memory`, max-model-len 131,072 x 2 seqs, bf16 KV, util 0.98, NBATCH 2048, prefix caching | | MoE kernel | `R4dW4a16MoEMethod` → libr4d grouped W4A16 GEMM, 6.234 MiB per expert-rank slot | | expert cache | 14.51 GiB = **2,383 slots** per rank; 12,096 host-backed experts, **73.64 GiB pinned per rank** | | unit | `scripts/frank-primary.service` (root unit, `LimitMEMLOCK=infinity` for the pin) | | measured | **decode 9.9-10.0 tok/s**, **prefill 918 tok/s** (warm, 2.6k-token prompt), KV cache 138,561 tokens | ```mermaid graph LR A[Intel AutoRound W4A16
sym int4 g128, auto_gptq] -->|quant_method auto-round| B[inc.py override to inc] B --> C[inc_wna16_scheme
_resolve_gptq_moe] C -->|sym int4 g128 + gfx1201 + libr4d| D[R4dW4a16MoEMethod
grouped W4A16 GEMM] D --> E[ExpertLRUCache
2,383 VRAM slots / rank] D --> F[73.64 GiB pinned host rows / rank] E --> G[decode step] F -->|8.2 GB/s effective| G ``` ## Why this payload and not another The W4A16 fast path on this fork is reachable **only** through the INC (AutoRound) quant method. `compressed_tensors` checkpoints of the same model — for example `dealignai/GLM-5.3-Flash-CYBERSECURITY-W4A16`, 182 GiB — resolve their MoE through the WNA16 oracle to `TritonWNA16Experts`, which **has no offload recipe**, and the engine refuses to serve at all: 170 GiB of experts cannot fit 64 GiB of VRAM, so offload is mandatory, not optional. The chain that makes Intel's checkpoint work, with the exact evidence: ``` [expert_plan.py] quantization=inc ← inc.py:364 overrides "auto-round" [inc_wna16_scheme.py:249] INC GPTQ MoE on ROCm: R4dW4a16MoEExperts (libr4d grouped W4A16 GEMM) [r4d_w4a16_moe.py:682] Using MoEPrepareAndFinalizeNoDPEPModular / R4dW4a16MoEExperts [expert.py:450] Expert offload: 42 MoE layers staged; 2,383 cache slots in 14.51 GiB planned per rank [expert.py:413] 42 layers split: 0 VRAM-only experts, 2,383 swap slots, arena 14.51 GiB, host 73.64 GiB pinned, host rows prepacked on the GPU ``` `R4dW4a16MoEExperts` is what the offload machinery wraps: the offload plan is format-agnostic but requires the experts class to expose the hot/cold split. ## The numbers, and the wall Measured on the deployed engine, 2026-09-30 (`bench_glm.py`, real prose, greedy): | depth | prefill | decode | | --- | --- | --- | | 2,621 tok prompt, run 1 (cold cache) | 259 tok/s | 9.88 | | 2,621 tok prompt, runs 2-3 (warm) | 918 / 885 tok/s | 9.91 / 9.41 | | 493 tok prompt (telemetry window) | 124 tok/s | 8.90 | Decode is flat across context and across every arena change — the signature of a per-step cost that is not KV- or residency-driven at the margin. ### The arithmetic that bounds it Per decode step, per rank: 42 MoE layers x 8 routed experts x 6.234 MiB = **2.045 GiB of expert weights**, which must either be in VRAM or cross PCIe. | hit rate | miss traffic | step @ 8.2 GB/s | decode | | --- | --- | --- | --- | | 0.60 (measured) | 0.82 GiB | ~105 ms | **9.5-10.0 tok/s** | | 0.70 | 0.61 GiB | ~78 ms | 12.8 tok/s | | 0.89 (what 35 tok/s needs) | 0.23 GiB | ~29 ms | 35 tok/s | Two independent measurements pin the missed term: - **Live expert-cache telemetry** (`R4D_LRU_TELEMETRY=1`, profiler window) gives the hit rate directly — the LRU is *working*: `layers.34: slots 56 (pinned 0, tier 0) hit 0.6279 routed 2002 installs 543 read-through 202` … `layers.44: hit 0.5335`. Per-layer range **0.53-0.66**. - **The link delivers 22.74 GB/s** at the engine's own slab size (H2D probe, pinned, 2.345 MiB). The step moves 0.82 GiB in ~100 ms = **8.2 GB/s — 36% of the wire**, matching the fork's own measured 6.5-9.5 GB/s. So the step is bounded by the fork's gather read path, and the only other lever is the hit rate — which is capped by the arena. A 14.51 GiB arena holds 20% of a 73.64 GiB expert set; the measured hit rate beats random by ~3x thanks to routing locality, but it saturates: growing the arena 13.08 → 14.51 GiB (+9%) changed decode by **nothing** (10.03 → 9.88 tok/s). Reaching 0.89 would need roughly 4.5x the per-layer slot count — an arena of well over 100 GiB. ## Measured arms (all 2026-09-30, one boot each) | arm | decode | prefill | verdict | | --- | --- | --- | --- | | baseline int4, MAXLEN 131,072, util 0.92, NBATCH 2048 | 10.03 / 10.08 / 9.41 | 877 / 878 / 885 | qualified incumbent | | + `--expert-precise-memory` (arena 13.08 → 14.51 GiB) | 9.88 / 9.91 | 918 | **neutral** — proves residency is no longer the binding term | | + MTP4 spec decode (draft loads, acceptance excellent) | 9.64-11.16 | **255** | **rejected** — see below | | + MTP4 + explicit `--kv-cache-memory-bytes` (patched cap) | boots, 151,767-token KV | — | enabler for the MTP arm only | ### Why MTP is rejected here (it is not that it fails) MTP **works** on GLM in this fork: the head loads, the architecture resolves to `Glm5NextMTPModel`, and acceptance is excellent — ``` SpecDecoding metrics: Mean acceptance length: 3.83-4.90, Per-position acceptance rate: 0.917/0.833/0.583/0.500 … 0.962/0.846/0.692/0.615, Avg Draft acceptance rate: 69-97% ``` — and it buys **nothing**: decode 9.6-11.2 tok/s against 10.0 without it, while prefill **collapses 918 → 255 tok/s**. The reason is structural to offload: a verification forward gathers the **union** of the batch's experts, so spec decode amortizes compute but not a residency-bound gather. It also costs 2.6 GiB of arena. On a compute-bound (fully-resident) shape this would invert; here it does not. ## Served surface (verified 2026-09-30 against the live engine) | capability | state | evidence | | --- | --- | --- | | `/v1/models` | `glm53-flash`, max_model_len 131,072 | model list probe | | completion | yes | `bench_glm.py` runs above | | **tool calls** | **yes** — needs `--enable-auto-tool-choice --tool-call-parser glm47` | a `get_weather` prompt returns `tool_calls: [{"id": "chatcmpl-tool-…", "type": "function", "function": {"name": "get_weather", "arguments": "{\"city\": \"Seattle\"}"}}]`, `finish_reason: tool_calls` | | **vision** | **yes** | a 64x64 solid-red PNG is answered "The image appears to be a solid red color…" | | reasoning channel | **yes** — needs `--reasoning-parser glm47` | without it the thinking leaks into `content` (measured: content began with the answer, then "Let me verify: … = 391", then a bare ` 2** (`is_aiter_found_and_supported()` → `get_cdna_version() > 2`). With fp8 KV the MTP draft step demands an op that cannot exist here: `RuntimeError: Worker failed with error ''_OpNamespace' 'vllm' object has no attribute 'rocm_aiter_mla_decode_fwd''`. KV dtype therefore stays auto/bf16. (`VLLM_ROCM_USE_AITER_MLA=1` does **not** help — the umbrella gate is the CDNA check, not the env var.) 2. **The plan's KV reserve under-models this model, and the cap enforces the mistake.** The plan charges 13,600 B/token + a small mamba term, but GLM carries a per-sequence KDA state (34 linear-attention layers) and, under MTP, the draft's KV. `gpu_worker` then *caps* KV at that reserve, so a legal `--max-model-len` becomes unbootable: `ValueError: To serve at least one request with the model's max seq len (131072), (2.07 GiB KV cache is needed, which is larger than the available KV cache memory (1.74 GiB))`. Raising `max_model_len` only helps past ~440k tokens (the cap's slope is ~13,845 B/token vs the need's ~12,700 + 0.52 GiB fixed) — where the reserve would eat a third of the arena. `scripts/patches/gpu_worker-kv-cap.patch` lets an explicit operator KV request raise the cap; **it is not part of the deployed shape**, which boots on the plan's own reserve. 3. **A window is forced, not chosen.** The production `max_model_len 131072` is what the reserve model requires for this payload (below ~92k the boot refuses); it is not a capacity decision. 4. **Expert prefetch remains poison** — unchanged from the qwen campaign (36% precision, +40-51% decode from deleting it). No GLM arm re-opened it. ## Operations ```bash # rollback to the pre-cutover qwen FP8 + MTP4 shape (payload and R9V state # dirs are untouched on disk): git show 2339fae -- scripts/ | git apply # or: git checkout 2339fae -- scripts/ ./scripts/deploy.sh dankserver --service frank-primary # payload inventory (34 shards + 2 extras, weight-map index): bash scripts/download-frank-primary.sh --verify-only # live telemetry (needs the profiler window; with_stack=false is mandatory — # with stack the stop path hangs and the trace buffer grows to tens of GB): podman logs frank-primary-glm 2>&1 | grep "expert_lru.py" # boot: ~7 min warm page cache, longer cold; health returns 503 while loading curl -s -o /dev/null -w "%{http_code}\n" http://127.0.0.1:7770/health ``` Boot gate: the engine refuses a legal-looking shape in two measured ways — the KV cap above, and the profile-run activation peak (`TritonMLA` indexer scores: 64 heads x 2,048 topk x NBATCH tokens x 2 B = 2.06 GiB at NBATCH 8192), which is what killed the first two boots at util 0.95. ## Rollback targets | state | where | | --- | --- | | qwen FP8 + MTP4 (2026-09-29 → 09-30) | commit `2339fae`, payload `/home/dank/models/Qwen3.8-Flash-Next-Uncensored-FP8` | | R9V radiance Qwen3.8-Flash-Next Q8_0 | [[frank-primary-r9v-radiance-qwen38-q8]], state dirs `/home/dank/r9v-state/{q8xl,uncensored}` |