serverAppConfigs/serverAppConfigs-docs/LLM/frank-primary-glm53-flash-w4a16.md

---

type: runbook title: frank-primary — GLM-5.3-Flash W4A16 int4 (tcclaviger, 2x R9700) description: VERIFIED 2026-09-30 by boot probe, bench and live expert-cache telemetry. frank-primary cut over from Qwen3.8-Flash-Next-Uncensored-FP8 + MTP4 to Intel/GLM-5.3-Flash-W4A16-AutoRound, the first payload this fleet serves through the fork's W4A16 expert-offload path (INC scheme -> R4dW4a16MoEMethod, libr4d grouped W4A16 GEMM). Measured decode 9.9-10.0 tok/s and prefill 918 tok/s at 131,072 ctx. THE 35 tok/s TARGET IS NOT MET and this page shows why it is out of reach: the step is bounded by the fork's expert-gather read path at ~8.2 GB/s effective against a PCIe link that delivers 22.74 GB/s, with the cache at its measured hit-rate ceiling (0.53-0.66 per layer) from an arena that cannot grow past ~15 GiB under a 73.64 GiB/rank expert set. tags:

• dankserver
• r9700
• gfx1201
• glm5.3
• w4a16
• int4
• expert-offload
• expert-cache
• autoround
• vllm timestamp: 2026-09-30

---

frank-primary — GLM-5.3-Flash W4A16 int4 (tcclaviger, 2x R9700)

Question this page answers. Can the fleet serve the 320B-class GLM-5.3-Flash at int4 on two R9700s with expert offload, and how fast is it? Answer. Yes — it serves, with vision, tool calls and 131,072 ctx. It runs at 9.9-10.0 tok/s decode / 918 tok/s prefill, and 35 tok/s is not reachable with this payload on this hardware. The bound is not the model, the quant, or the link: it is the fork's expert-gather read path, and it is measured, not inferred.

Related: dankserver-qwen38-fp8-expert-offload-tuning (the same fork, the same hardware, the same gather-path wall, found first on qwen), dankserver-frontier-moe-feasibility (the earlier fit analysis), VLLM Index, LLM/index.

The deployed shape

term	value
payload	Intel/GLM-5.3-Flash-W4A16-AutoRound — AutoRound symmetric int4, group 128, packing auto_round:auto_gptq, 34 shards + 2 extra safetensors, 113,074 tensors, ~170 GiB
runtime	docker.io/tcclaviger/vllm@sha256:da4029666… (same digest-pinned image as the qwen cutover)
engine	TP=2, --enable-expert-offload --expert-precise-memory, max-model-len 131,072 x 2 seqs, bf16 KV, util 0.98, NBATCH 2048, prefix caching
MoE kernel	R4dW4a16MoEMethod → libr4d grouped W4A16 GEMM, 6.234 MiB per expert-rank slot
expert cache	14.51 GiB = 2,383 slots per rank; 12,096 host-backed experts, 73.64 GiB pinned per rank
unit	scripts/frank-primary.service (root unit, LimitMEMLOCK=infinity for the pin)
measured	decode 9.9-10.0 tok/s, prefill 918 tok/s (warm, 2.6k-token prompt), KV cache 138,561 tokens

graph LR
    A[Intel AutoRound W4A16<br/>sym int4 g128, auto_gptq] -->|quant_method auto-round| B[inc.py override to inc]
    B --> C[inc_wna16_scheme<br/>_resolve_gptq_moe]
    C -->|sym int4 g128 + gfx1201 + libr4d| D[R4dW4a16MoEMethod<br/>grouped W4A16 GEMM]
    D --> E[ExpertLRUCache<br/>2,383 VRAM slots / rank]
    D --> F[73.64 GiB pinned host rows / rank]
    E --> G[decode step]
    F -->|8.2 GB/s effective| G

Why this payload and not another

The W4A16 fast path on this fork is reachable only through the INC (AutoRound) quant method. compressed_tensors checkpoints of the same model — for example dealignai/GLM-5.3-Flash-CYBERSECURITY-W4A16, 182 GiB — resolve their MoE through the WNA16 oracle to TritonWNA16Experts, which has no offload recipe, and the engine refuses to serve at all: 170 GiB of experts cannot fit 64 GiB of VRAM, so offload is mandatory, not optional.

The chain that makes Intel's checkpoint work, with the exact evidence:

[expert_plan.py]  quantization=inc            ← inc.py:364 overrides "auto-round"
[inc_wna16_scheme.py:249]  INC GPTQ MoE on ROCm: R4dW4a16MoEExperts (libr4d grouped W4A16 GEMM)
[r4d_w4a16_moe.py:682]     Using MoEPrepareAndFinalizeNoDPEPModular / R4dW4a16MoEExperts
[expert.py:450]            Expert offload: 42 MoE layers staged; 2,383 cache slots in 14.51 GiB planned per rank
[expert.py:413]            42 layers split: 0 VRAM-only experts, 2,383 swap slots, arena 14.51 GiB,
                           host 73.64 GiB pinned, host rows prepacked on the GPU

R4dW4a16MoEExperts is what the offload machinery wraps: the offload plan is format-agnostic but requires the experts class to expose the hot/cold split.

The numbers, and the wall

Measured on the deployed engine, 2026-09-30 (bench_glm.py, real prose, greedy):

depth	prefill	decode
2,621 tok prompt, run 1 (cold cache)	259 tok/s	9.88
2,621 tok prompt, runs 2-3 (warm)	918 / 885 tok/s	9.91 / 9.41
493 tok prompt (telemetry window)	124 tok/s	8.90

Decode is flat across context and across every arena change — the signature of a per-step cost that is not KV- or residency-driven at the margin.

The arithmetic that bounds it

Per decode step, per rank: 42 MoE layers x 8 routed experts x 6.234 MiB = 2.045 GiB of expert weights, which must either be in VRAM or cross PCIe.

hit rate	miss traffic	step @ 8.2 GB/s	decode
0.60 (measured)	0.82 GiB	~105 ms	9.5-10.0 tok/s
0.70	0.61 GiB	~78 ms	12.8 tok/s
0.89 (what 35 tok/s needs)	0.23 GiB	~29 ms	35 tok/s

Two independent measurements pin the missed term:

• Live expert-cache telemetry (R4D_LRU_TELEMETRY=1, profiler window) gives the hit rate directly — the LRU is working: layers.34: slots 56 (pinned 0, tier 0) hit 0.6279 routed 2002 installs 543 read-through 202 … layers.44: hit 0.5335. Per-layer range 0.53-0.66.
• The link delivers 22.74 GB/s at the engine's own slab size (H2D probe, pinned, 2.345 MiB). The step moves 0.82 GiB in ~100 ms = 8.2 GB/s — 36% of the wire, matching the fork's own measured 6.5-9.5 GB/s.

So the step is bounded by the fork's gather read path, and the only other lever is the hit rate — which is capped by the arena. A 14.51 GiB arena holds 20% of a 73.64 GiB expert set; the measured hit rate beats random by ~3x thanks to routing locality, but it saturates: growing the arena 13.08 → 14.51 GiB (+9%) changed decode by nothing (10.03 → 9.88 tok/s). Reaching 0.89 would need roughly 4.5x the per-layer slot count — an arena of well over 100 GiB.

Measured arms (all 2026-09-30, one boot each)

arm	decode	prefill	verdict
baseline int4, MAXLEN 131,072, util 0.92, NBATCH 2048	10.03 / 10.08 / 9.41	877 / 878 / 885	qualified incumbent
+ --expert-precise-memory (arena 13.08 → 14.51 GiB)	9.88 / 9.91	918	neutral — proves residency is no longer the binding term
+ MTP4 spec decode (draft loads, acceptance excellent)	9.64-11.16	255	rejected — see below
+ MTP4 + explicit --kv-cache-memory-bytes (patched cap)	boots, 151,767-token KV	—	enabler for the MTP arm only

Why MTP is rejected here (it is not that it fails)

MTP works on GLM in this fork: the head loads, the architecture resolves to Glm5NextMTPModel, and acceptance is excellent —

SpecDecoding metrics: Mean acceptance length: 3.83-4.90,
Per-position acceptance rate: 0.917/0.833/0.583/0.500 … 0.962/0.846/0.692/0.615,
Avg Draft acceptance rate: 69-97%

— and it buys nothing: decode 9.6-11.2 tok/s against 10.0 without it, while prefill collapses 918 → 255 tok/s. The reason is structural to offload: a verification forward gathers the union of the batch's experts, so spec decode amortizes compute but not a residency-bound gather. It also costs 2.6 GiB of arena. On a compute-bound (fully-resident) shape this would invert; here it does not.

Served surface (verified 2026-09-30 against the live engine)

capability	state	evidence
/v1/models	glm53-flash, max_model_len 131,072	model list probe
completion	yes	bench_glm.py runs above
tool calls	yes — needs --enable-auto-tool-choice --tool-call-parser glm47	a get_weather prompt returns tool_calls: [{"id": "chatcmpl-tool-…", "type": "function", "function": {"name": "get_weather", "arguments": "{\"city\": \"Seattle\"}"}}], finish_reason: tool_calls
vision	yes	a 64x64 solid-red PNG is answered "The image appears to be a solid red color…"
reasoning channel	yes — needs --reasoning-parser glm47	without it the thinking leaks into content (measured: content began with the answer, then "Let me verify: … = 391", then a bare </think) with an empty reasoning field; with it, a train-time prompt answers in content ("Step 1: Add the hours. 14:20 + 3 hours = 17:20 …") and the chain of thought lands in reasoning — the vLLM field name the fleet documents (llama.cpp endpoints use reasoning_content)

Without the tool flags the engine answers HTTP 400 — "auto" tool choice requires --enable-auto-tool-choice and --tool-call-parser to be set — which breaks every tool-using agent on the endpoint. Both parsers are glm47, the newest GLM dialect the fork ships (glm45/glm47 exist for both).

Fork traps found (for the engine authors)

1. --kv-cache-dtype fp8 is illegal on gfx12 with this model. The ROCM_AITER_MLA_SPARSE backend's _use_rocm_sparse_triton() requires a non-fp8 KV cache, and that rope-free Triton path is the only MLA decode available on gfx12: aiter's MLA decode op is registered under @if_aiter_supported, which is gated on CDNA > 2 (is_aiter_found_and_supported() → get_cdna_version() > 2). With fp8 KV the MTP draft step demands an op that cannot exist here: RuntimeError: Worker failed with error ''_OpNamespace' 'vllm' object has no attribute 'rocm_aiter_mla_decode_fwd''. KV dtype therefore stays auto/bf16. (VLLM_ROCM_USE_AITER_MLA=1 does not help — the umbrella gate is the CDNA check, not the env var.)
2. The plan's KV reserve under-models this model, and the cap enforces the mistake. The plan charges 13,600 B/token + a small mamba term, but GLM carries a per-sequence KDA state (34 linear-attention layers) and, under MTP, the draft's KV. gpu_worker then caps KV at that reserve, so a legal --max-model-len becomes unbootable: ValueError: To serve at least one request with the model's max seq len (131072), (2.07 GiB KV cache is needed, which is larger than the available KV cache memory (1.74 GiB)). Raising max_model_len only helps past ~440k tokens (the cap's slope is ~13,845 B/token vs the need's ~12,700 + 0.52 GiB fixed) — where the reserve would eat a third of the arena. scripts/patches/gpu_worker-kv-cap.patch lets an explicit operator KV request raise the cap; it is not part of the deployed shape, which boots on the plan's own reserve.
3. A window is forced, not chosen. The production max_model_len 131072 is what the reserve model requires for this payload (below ~92k the boot refuses); it is not a capacity decision.
4. Expert prefetch remains poison — unchanged from the qwen campaign (36% precision, +40-51% decode from deleting it). No GLM arm re-opened it.

Operations

# rollback to the pre-cutover qwen FP8 + MTP4 shape (payload and R9V state
# dirs are untouched on disk):
git show 2339fae -- scripts/ | git apply   # or: git checkout 2339fae -- scripts/
./scripts/deploy.sh dankserver --service frank-primary

# payload inventory (34 shards + 2 extras, weight-map index):
bash scripts/download-frank-primary.sh --verify-only

# live telemetry (needs the profiler window; with_stack=false is mandatory —
# with stack the stop path hangs and the trace buffer grows to tens of GB):
podman logs frank-primary-glm 2>&1 | grep "expert_lru.py"

# boot: ~7 min warm page cache, longer cold; health returns 503 while loading
curl -s -o /dev/null -w "%{http_code}\n" http://127.0.0.1:7770/health

Boot gate: the engine refuses a legal-looking shape in two measured ways — the KV cap above, and the profile-run activation peak (TritonMLA indexer scores: 64 heads x 2,048 topk x NBATCH tokens x 2 B = 2.06 GiB at NBATCH 8192), which is what killed the first two boots at util 0.95.

PYTORCH_CUDA_ALLOC_CONF=expandable_segments:True is part of the qualified shape, not a tuning: the plan fills VRAM to its modelled headroom by design, so the default allocator's fragmentation fails the profile/graph-capture phase (measured on the deployed unit: "Tried to allocate 662.00 MiB. GPU 0 … 240.00 MiB is free").

The deploy.sh e2e gate also drives the admin panel's open-links spec, which walks every guarded vhost — including http://192.168.0.112:8989/ on FrankBook-Pro. When that Mac mini is asleep the spec fails on a page.goto timeout and the deploy reports DEPLOY INCOMPLETE even though the engine is healthy; wake the host (magic packet to de:6b:6d:c1:52:23) or re-run tests/e2e/run.sh once it is up. Seen 2026-09-30: 32 passed, this one failed, and the WoL packet did not wake it (macOS "Wake for network access" needs toggling on the mini itself).

Rollback targets

state	where
qwen FP8 + MTP4 (2026-09-29 → 09-30)	commit 2339fae, payload /home/dank/models/Qwen3.8-Flash-Next-Uncensored-FP8
R9V radiance Qwen3.8-Flash-Next Q8_0	frank-primary-r9v-radiance-qwen38-q8, state dirs /home/dank/r9v-state/{q8xl,uncensored}