serverAppConfigs/serverAppConfigs-docs/LLM/frank-primary-glm53-flash-w4a16.md

---

type: runbook title: frank-primary — GLM-5.3-Flash W4A16 int4 (tcclaviger, 2x R9700) description: VERIFIED 2026-09-30 by boot probe, bench, live expert-cache telemetry AND a torch-profiler kernel-time window. frank-primary cut over from Qwen3.8-Flash-Next-Uncensored-FP8 + MTP4 to Intel/GLM-5.3-Flash-W4A16-AutoRound, the first payload this fleet serves through the fork's W4A16 expert-offload path (INC scheme -> R4dW4a16MoEMethod, libr4d grouped W4A16 GEMM). Measured decode 9.9-10.0 tok/s and prefill 918 tok/s at 131,072 ctx. THE 35 tok/s TARGET IS NOT MET, and the profiler window FALSIFIED the first explanation: the expert gather is 1.05 % of device time, not the bound. 44 % of the 93 ms/token budget is pointwise glue (1,428 elementwise launches per token) because the fork's GLM trunk is NOT torch.compiled while its own Qwen trunk is; the second-largest lever, the fused router, is binary-blocked at 288 experts. The compile changelist is staged but incomplete (three patches, one blocker left). tags:

• dankserver
• r9700
• gfx1201
• glm5.3
• w4a16
• int4
• expert-offload
• expert-cache
• autoround
• vllm timestamp: 2026-09-30

---

frank-primary — GLM-5.3-Flash W4A16 int4 (tcclaviger, 2x R9700)

Question this page answers. Can the fleet serve the 320B-class GLM-5.3-Flash at int4 on two R9700s with expert offload, and how fast is it? Answer. Yes — it serves, with vision, tool calls and 131,072 ctx. It runs at 9.9-10.0 tok/s decode / 918 tok/s prefill, and 35 tok/s is not reachable on this payload as the fork ships it. The bound is not the model, the quant, the link, or the expert gather (measured: 1.05 % of device time). It is the unfused pointwise glue of an uncompiled trunk — 44 % of a 93 ms per-token budget spread over ~1,428 elementwise launches per token — because the fork compiles its own Qwen trunk and not its GLM one. See The measured profile.

Related: dankserver-qwen38-fp8-expert-offload-tuning (the same fork, the same hardware — its conclusion was drawn from throughput arithmetic and is superseded by this page's kernel-time measurement), dankserver-frontier-moe-feasibility (the earlier fit analysis), VLLM Index, LLM/index.

The deployed shape

term	value
payload	Intel/GLM-5.3-Flash-W4A16-AutoRound — AutoRound symmetric int4, group 128, packing auto_round:auto_gptq, 34 shards + 2 extra safetensors, 113,074 tensors, ~170 GiB
runtime	docker.io/tcclaviger/vllm@sha256:da4029666… (same digest-pinned image as the qwen cutover)
engine	TP=2, --enable-expert-offload --expert-precise-memory, max-model-len 131,072 x 2 seqs, bf16 KV, util 0.98, NBATCH 2048, prefix caching
MoE kernel	R4dW4a16MoEMethod → libr4d grouped W4A16 GEMM, 6.234 MiB per expert-rank slot
expert cache	14.51 GiB = 2,383 slots per rank; 12,096 host-backed experts, 73.64 GiB pinned per rank
unit	scripts/frank-primary.service (root unit, LimitMEMLOCK=infinity for the pin)
measured	decode 9.9-10.0 tok/s, prefill 918 tok/s (warm, 2.6k-token prompt), KV cache 138,561 tokens

graph LR
    A[Intel AutoRound W4A16<br/>sym int4 g128, auto_gptq] -->|quant_method auto-round| B[inc.py override to inc]
    B --> C[inc_wna16_scheme<br/>_resolve_gptq_moe]
    C -->|sym int4 g128 + gfx1201 + libr4d| D[R4dW4a16MoEMethod<br/>grouped W4A16 GEMM]
    D --> E[ExpertLRUCache<br/>2,383 VRAM slots / rank]
    D --> F[73.64 GiB pinned host rows / rank]
    E --> G[decode step]
    F -->|host misses<br/>1.05% of device time| G

Why this payload and not another

The W4A16 fast path on this fork is reachable only through the INC (AutoRound) quant method. compressed_tensors checkpoints of the same model — for example dealignai/GLM-5.3-Flash-CYBERSECURITY-W4A16, 182 GiB — resolve their MoE through the WNA16 oracle to TritonWNA16Experts, which has no offload recipe, and the engine refuses to serve at all: 170 GiB of experts cannot fit 64 GiB of VRAM, so offload is mandatory, not optional.

The chain that makes Intel's checkpoint work, with the exact evidence:

[expert_plan.py]  quantization=inc            ← inc.py:364 overrides "auto-round"
[inc_wna16_scheme.py:249]  INC GPTQ MoE on ROCm: R4dW4a16MoEExperts (libr4d grouped W4A16 GEMM)
[r4d_w4a16_moe.py:682]     Using MoEPrepareAndFinalizeNoDPEPModular / R4dW4a16MoEExperts
[expert.py:450]            Expert offload: 42 MoE layers staged; 2,383 cache slots in 14.51 GiB planned per rank
[expert.py:413]            42 layers split: 0 VRAM-only experts, 2,383 swap slots, arena 14.51 GiB,
                           host 73.64 GiB pinned, host rows prepacked on the GPU

R4dW4a16MoEExperts is what the offload machinery wraps: the offload plan is format-agnostic but requires the experts class to expose the hot/cold split.

The numbers, and the wall

Measured on the deployed engine, 2026-09-30 (bench_glm.py, real prose, greedy):

depth	prefill	decode
2,621 tok prompt, run 1 (cold cache)	259 tok/s	9.88
2,621 tok prompt, runs 2-3 (warm)	918 / 885 tok/s	9.91 / 9.41
493 tok prompt (telemetry window)	124 tok/s	8.90

Decode is flat across context and across every arena change — the signature of a per-step cost that is not KV- or residency-driven at the margin.

The measured profile, and the real wall

This section replaces an earlier explanation, which the measurement falsified. The first reading of this page bounded the step with throughput arithmetic: 42 MoE layers x 8 experts x 6.234 MiB = 2.045 GiB per step per rank, 0.82 GiB of it missed at a measured 0.53-0.66 hit rate, giving "0.82 GiB in ~100 ms = 8.2 GB/s = the gather is the wall". That arithmetic was circular — it assumed the gather consumed the whole step — and it is wrong. The engine, asked directly, disagrees:

A torch-profiler window (--profiler-config, POST /start_profile / stop_profile, 496 decode tokens at the deployed shape, rank0 trace parsed for device-activity time) gives the true split. Total device kernel time 46.18 s / 496 tokens = ~93 ms per token against a measured ~104 ms step (9.6 tok/s with the profiler attached):

ms	%	launches	kernel
16,074	34.8	708,500	vectorized_elementwise_kernel (22.7 us each)
4,602	10.0	42,000	r4d_gemm_moe_w4a16_nt_b16_m64 — the MoE GEMM
3,108	6.7	174,708	elementwise_kernel_manual_unroll
2,943	6.4	138,252	wvSplitK_hf_sml_ — split-K GEMM (projections / head)
2,829	6.1	83,160	router: sbtopk::gatherTopK + warpMergeSortTopK + bitonicSortKVInPlace (1,208 + 1,051 + 559 ms; the 1,051 row is not in the list below)
3,015	6.5	133,920	r4d_mhc_pre_finish/pre_partial/post — hyper-connections
2,782	6.0	127,916	inductor triton_poi_* / triton_per_* (the fusions that DO happen)
487	1.05	20,664	r4d_moe_lru_gather_kernel — the accused
250	0.54	20,832	_moe_lru_tele_kernel — the telemetry that measured it (TELEM was on)

So the gather is 1.05 % of device time, and r4d_moe_lru_fused_kernel (the LRU manager, not the copy) is a further 1,599 ms / 3.5 %. Residency work and the PCIe link are not the story: 44 % of the budget is pointwise glue — bf16<->fp32 conversions, adds, pow/rsqrt from fp32 RMSNorm, launch_clamp_scale quant passes, reduce_kernel means — spread over ~1,428 elementwise launches per token (3,734 kernel launches per token in total). The GPU is busy 86 % of the wall clock, so this is real device work, not launch-gap overhead.

Two consequences that kill three whole families of planned work:

• Spec decode cannot help this shape, and the MTP measurement was not an accident. Pointwise cost scales with the token count, so a draft batch that verifies 4 tokens pays ~4x the glue for the same per-token output — which is exactly why MTP4 measured "acceptance 3.76-4.90, decode unchanged". Same reason the gather was never the bound.
• Arena size and hit rate are not levers. Confirmed twice: --expert-precise-memory (arena 13.08 -> 14.51 GiB) moved decode by nothing, and the gather is 1 %.

Fork work: the compile changelist (WIP, does not boot yet)

The fork compiles its own flagship trunk — vllm/models/qwen4_exp/amd/model.py carries @support_torch_compile — and not its GLM trunk. The engine says so on every boot:

Cudagraph mode FULL_AND_PIECEWISE requires piecewise capture, but the loaded model
provides neither a compiled submodule nor breakable CUDA graphs.
Overriding to FULL_DECODE_ONLY.

Inductor fusion is therefore the only lever that touches the 44 % pointwise mass. Three patches are staged in scripts/patches/glm53-compile-0*.patch, each carrying its own evidence header, base/patched sha256 and the next blocker:

patch	target	fixes
-01-model-decorator	models/glm5next/common/model.py	decorates Glm5NextModel (env R4D_GLM_COMPILE, default on)
-02-trace-safe-probes	model_executor/kernels/r4d_lib.py	freezes 10 capability probes that do os.path.exists/.so lookups inside the traced forward
-03-fullgraph-optout	compilation/wrapper.py	makes fullgraph=True opt-out (env VLLM_COMPILE_FULLGRAPH=0)

What the boots established, in order (each one ~7 min):

1. 01 alone dies on the first cold probe call inside the traced region: Dynamo does not know how to trace the builtin 'posix.stat' — r4d_mhc_supported -> r4d_lib.r4d_mhc_enabled -> available -> os.path.exists. functools.lru_cache does not protect these probes: dynamo re-traces the wrapped body to build its guard even when the cache is warm (it warns about exactly this), verified in-image — a warm lru_cache-calling function still fails fullgraph=True, while a plain module-global read compiles.
2. 01+02 get past it (libr4d capability probes frozen: 10 warm [...] in all four processes) and die on RuntimeError: Graph breaks are not supported with aot compile, because this build runs AOT compile by default and AOT refuses breaks.
3. 01+02+03 with VLLM_COMPILE_FULLGRAPH=0 VLLM_USE_AOT_COMPILE=0 traces the trunk successfully (Dynamo bytecode transform time: 1.08 s) and then dies in profile_run with KeyError: 'in_proj_qkvbfg_a' at glm5next/common/model.py:456 (the KDA layer's merged projection, kda.py:217) — a dict/registry lookup keyed by attribute name that the eager path never reaches, i.e. a latent bug the compile exposes, not a dynamo limit.

That last row is the state of the work: the remaining blocker is a real bug in the fork's mHC/KDA wiring, and it must be fixed (plus whatever follows it) before the changelist can boot. Two verified facts make the payoff concrete: suppress_errors does not cover graph breaks (only fullgraph=False does), and the GLM trunk can never satisfy fullgraph because its indexer metadata syncs to host (glm5next/amd/sparse_indexer.py:263,419: int(...max().item())).

Two structural constraints on any further fork work

• No new HIP kernels are possible. The image ships r4d.so as a binary with no sources. The obvious second lever — the 9 % router — is gated num_experts in {128, 256, 512} in router/r4d_route_router.py and the binary contains only r4d_moe_route_kernel<128|256|512> instantiations. GLM has 288 experts, so it falls back to the generic sort-based top-k and cannot be moved to the fused router without rebuilding libr4d. Only Python/config/inductor changes (and runtime-JIT Triton) are available.
• --kv-cache-dtype fp8 is a dead end for capacity too. The plan's KV reserve is dtype-blind (1.76 GiB at 131,072 either way), so fp8 buys no arena and no window; it only removes the only MLA decode path gfx12 has (trap 1 above).

DFlash2 for GLM — what the drafter actually is

incoai/GLM-5.3-Flash-DFlash2 is not a GLM-shaped drafter and needs no MLA/DSA port: its config.json declares architectures: ["DFlash2DraftModel"] and model_type: "qwen3" — 5 dense layers, hidden 4096, 32 heads / 8 kv heads, head_dim 128, sliding attention (window 2048), is_causal: false, use_cache: false, dflash_config = block 8, conv group 16, selector rank 256 / top-k 16, and target_layer_ids: [5, 14, 24, 33, 42] against num_target_layers: 45.

The fork already maps that architecture: registry.py:636 "DFlash2DraftModel": ("qwen3_dflash2", "DFlash2Qwen3ForCausalLM"), and the spec method is "dflash" (config/speculative.py:67). So the port is target-side only: the GLM trunk must expose auxiliary hidden states at those five layer indices, the same interface deepseek_v2.py (also MLA+MoE) implements — set_aux_hidden_state_layers() / get_eagle3_aux_hidden_state_layers() plus capture at idx in self.aux_hidden_state_layers in the layer loop. glm5next implements neither.

Expected benefit is not assumed: DFlash2 amortizes a draft block (8 tokens) over one verification pass, and per the profile above a verification pass costs pointwise work proportional to tokens — the same reason MTP4 was flat. Measure it; do not budget on it.

Note also that the KV-cap patch's predecessor arithmetic in dankserver-qwen38-fp8-expert-offload-tuning should be read the same way: its "gather-bound" conclusion came from the same throughput arithmetic that this page's profiler window has now falsified.

Measured arms (all 2026-09-30, one boot each)

arm	decode	prefill	verdict
baseline int4, MAXLEN 131,072, util 0.92, NBATCH 2048	10.03 / 10.08 / 9.41	877 / 878 / 885	qualified incumbent
+ --expert-precise-memory (arena 13.08 → 14.51 GiB)	9.88 / 9.91	918	neutral — proves residency is no longer the binding term
+ MTP4 spec decode (draft loads, acceptance excellent)	9.64-11.16	255	rejected — see below
+ MTP4 + explicit --kv-cache-memory-bytes (patched cap)	boots, 151,767-token KV	—	enabler for the MTP arm only

Why MTP is rejected here (it is not that it fails)

MTP works on GLM in this fork: the head loads, the architecture resolves to Glm5NextMTPModel, and acceptance is excellent —

SpecDecoding metrics: Mean acceptance length: 3.83-4.90,
Per-position acceptance rate: 0.917/0.833/0.583/0.500 … 0.962/0.846/0.692/0.615,
Avg Draft acceptance rate: 69-97%

— and it buys nothing: decode 9.6-11.2 tok/s against 10.0 without it, while prefill collapses 918 → 255 tok/s. The reason is structural to offload: a verification forward gathers the union of the batch's experts, so spec decode amortizes compute but not a residency-bound gather. It also costs 2.6 GiB of arena. On a compute-bound (fully-resident) shape this would invert; here it does not.

Served surface (verified 2026-09-30 against the live engine)

capability	state	evidence
/v1/models	glm53-flash, max_model_len 131,072	model list probe
completion	yes	bench_glm.py runs above
tool calls	yes — needs --enable-auto-tool-choice --tool-call-parser glm47	a get_weather prompt returns tool_calls: [{"id": "chatcmpl-tool-…", "type": "function", "function": {"name": "get_weather", "arguments": "{\"city\": \"Seattle\"}"}}], finish_reason: tool_calls
vision	yes	a 64x64 solid-red PNG is answered "The image appears to be a solid red color…"
reasoning channel	yes — needs --reasoning-parser glm47	without it the thinking leaks into content (measured: content began with the answer, then "Let me verify: … = 391", then a bare </think) with an empty reasoning field; with it, a train-time prompt answers in content ("Step 1: Add the hours. 14:20 + 3 hours = 17:20 …") and the chain of thought lands in reasoning — the vLLM field name the fleet documents (llama.cpp endpoints use reasoning_content)

Without the tool flags the engine answers HTTP 400 — "auto" tool choice requires --enable-auto-tool-choice and --tool-call-parser to be set — which breaks every tool-using agent on the endpoint. Both parsers are glm47, the newest GLM dialect the fork ships (glm45/glm47 exist for both).

Fork traps found (for the engine authors)

1. --kv-cache-dtype fp8 is illegal on gfx12 with this model. The ROCM_AITER_MLA_SPARSE backend's _use_rocm_sparse_triton() requires a non-fp8 KV cache, and that rope-free Triton path is the only MLA decode available on gfx12: aiter's MLA decode op is registered under @if_aiter_supported, which is gated on CDNA > 2 (is_aiter_found_and_supported() → get_cdna_version() > 2). With fp8 KV the MTP draft step demands an op that cannot exist here: RuntimeError: Worker failed with error ''_OpNamespace' 'vllm' object has no attribute 'rocm_aiter_mla_decode_fwd''. KV dtype therefore stays auto/bf16. (VLLM_ROCM_USE_AITER_MLA=1 does not help — the umbrella gate is the CDNA check, not the env var.)
2. The plan's KV reserve under-models this model, and the cap enforces the mistake. The plan charges 13,600 B/token + a small mamba term, but GLM carries a per-sequence KDA state (34 linear-attention layers) and, under MTP, the draft's KV. gpu_worker then caps KV at that reserve, so a legal --max-model-len becomes unbootable: ValueError: To serve at least one request with the model's max seq len (131072), (2.07 GiB KV cache is needed, which is larger than the available KV cache memory (1.74 GiB)). Raising max_model_len only helps past ~440k tokens (the cap's slope is ~13,845 B/token vs the need's ~12,700 + 0.52 GiB fixed) — where the reserve would eat a third of the arena. scripts/patches/gpu_worker-kv-cap.patch lets an explicit operator KV request raise the cap; it is not part of the deployed shape, which boots on the plan's own reserve.
3. A window is forced, not chosen. The production max_model_len 131072 is what the reserve model requires for this payload (below ~92k the boot refuses); it is not a capacity decision.
4. Expert prefetch remains poison — unchanged from the qwen campaign (36% precision, +40-51% decode from deleting it). No GLM arm re-opened it.

Operations

# rollback to the pre-cutover qwen FP8 + MTP4 shape (payload and R9V state
# dirs are untouched on disk):
git show 2339fae -- scripts/ | git apply   # or: git checkout 2339fae -- scripts/
./scripts/deploy.sh dankserver --service frank-primary

# payload inventory (34 shards + 2 extras, weight-map index):
bash scripts/download-frank-primary.sh --verify-only

# live telemetry (needs the profiler window; with_stack=false is mandatory —
# with stack the stop path hangs and the trace buffer grows to tens of GB):
podman logs frank-primary-glm 2>&1 | grep "expert_lru.py"

# boot: ~7 min warm page cache, longer cold; health returns 503 while loading
curl -s -o /dev/null -w "%{http_code}\n" http://127.0.0.1:7770/health

Boot gate: the engine refuses a legal-looking shape in two measured ways — the KV cap above, and the profile-run activation peak (TritonMLA indexer scores: 64 heads x 2,048 topk x NBATCH tokens x 2 B = 2.06 GiB at NBATCH 8192), which is what killed the first two boots at util 0.95.

PYTORCH_CUDA_ALLOC_CONF=expandable_segments:True is part of the qualified shape, not a tuning: the plan fills VRAM to its modelled headroom by design, so the default allocator's fragmentation fails the profile/graph-capture phase (measured on the deployed unit: "Tried to allocate 662.00 MiB. GPU 0 … 240.00 MiB is free").

The deploy.sh e2e gate also drives the admin panel's open-links spec, which walks every guarded vhost — including http://192.168.0.112:8989/ on FrankBook-Pro. When that Mac mini is asleep the spec fails on a page.goto timeout and the deploy reports DEPLOY INCOMPLETE even though the engine is healthy; wake the host (magic packet to de:6b:6d:c1:52:23) or re-run tests/e2e/run.sh once it is up. Seen 2026-09-30: 32 passed, this one failed, and the WoL packet did not wake it (macOS "Wake for network access" needs toggling on the mini itself).

Rollback targets

state	where
qwen FP8 + MTP4 (2026-09-29 → 09-30)	commit 2339fae, payload /home/dank/models/Qwen3.8-Flash-Next-Uncensored-FP8
R9V radiance Qwen3.8-Flash-Next Q8_0	frank-primary-r9v-radiance-qwen38-q8, state dirs /home/dank/r9v-state/{q8xl,uncensored}