--- type: runbook title: frank-primary — GLM-5.3-Flash W4A16 int4 (tcclaviger, 2x R9700) description: VERIFIED 2026-09-30 by boot probe, bench, live expert-cache telemetry AND a torch-profiler kernel-time window. frank-primary cut over from Qwen3.8-Flash-Next-Uncensored-FP8 + MTP4 to Intel/GLM-5.3-Flash-W4A16-AutoRound, the first payload this fleet serves through the fork's W4A16 expert-offload path (INC scheme -> R4dW4a16MoEMethod, libr4d grouped W4A16 GEMM). Measured decode 9.9-10.0 tok/s and prefill 918 tok/s at 131,072 ctx. THE 35 tok/s TARGET IS NOT MET, and the profiler window FALSIFIED the first explanation: the expert gather is 1.05 % of device time, not the bound. 44 % of the 93 ms/token budget is pointwise glue (1,428 elementwise launches per token) because the fork's GLM trunk is NOT torch.compiled while its own Qwen trunk is; the second-largest lever, the fused router, is binary-blocked at 288 experts. The compile changelist is staged but incomplete (three patches, one blocker left). tags: - dankserver - r9700 - gfx1201 - glm5.3 - w4a16 - int4 - expert-offload - expert-cache - autoround - vllm timestamp: 2026-09-30 --- # frank-primary — GLM-5.3-Flash W4A16 int4 (tcclaviger, 2x R9700) > **Question this page answers.** Can the fleet serve the 320B-class > `GLM-5.3-Flash` at int4 on two R9700s with expert offload, and how fast is it? > **Answer.** Yes — it serves, with vision, tool calls and 131,072 ctx. It runs > at **9.9-10.0 tok/s decode / 918 tok/s prefill**, and **35 tok/s is not > reachable** on this payload as the fork ships it. The bound is **not** the > model, the quant, the link, or the expert gather (measured: 1.05 % of device > time). It is the **unfused pointwise glue of an uncompiled trunk** — 44 % of a > 93 ms per-token budget spread over ~1,428 elementwise launches per token — > because the fork compiles its own Qwen trunk and not its GLM one. See > [The measured profile](#the-measured-profile-and-the-real-wall). Related: [[dankserver-qwen38-fp8-expert-offload-tuning]] (the same fork, the same hardware — its conclusion was drawn from throughput arithmetic and is superseded by this page's kernel-time measurement), [[dankserver-frontier-moe-feasibility]] (the earlier fit analysis), [[VLLM Index]], [[LLM/index|LLM fleet]]. ## The deployed shape | term | value | | --- | --- | | payload | `Intel/GLM-5.3-Flash-W4A16-AutoRound` — AutoRound **symmetric int4, group 128**, packing `auto_round:auto_gptq`, 34 shards + 2 extra safetensors, 113,074 tensors, ~170 GiB | | runtime | `docker.io/tcclaviger/vllm@sha256:da4029666…` (same digest-pinned image as the qwen cutover) | | engine | TP=2, `--enable-expert-offload --expert-precise-memory`, max-model-len 131,072 x 2 seqs, bf16 KV, util 0.98, NBATCH 2048, prefix caching | | MoE kernel | `R4dW4a16MoEMethod` → libr4d grouped W4A16 GEMM, 6.234 MiB per expert-rank slot | | expert cache | 14.51 GiB = **2,383 slots** per rank; 12,096 host-backed experts, **73.64 GiB pinned per rank** | | unit | `scripts/frank-primary.service` (root unit, `LimitMEMLOCK=infinity` for the pin) | | measured | **decode 9.9-10.0 tok/s**, **prefill 918 tok/s** (warm, 2.6k-token prompt), KV cache 138,561 tokens | ```mermaid graph LR A[Intel AutoRound W4A16
sym int4 g128, auto_gptq] -->|quant_method auto-round| B[inc.py override to inc] B --> C[inc_wna16_scheme
_resolve_gptq_moe] C -->|sym int4 g128 + gfx1201 + libr4d| D[R4dW4a16MoEMethod
grouped W4A16 GEMM] D --> E[ExpertLRUCache
2,383 VRAM slots / rank] D --> F[73.64 GiB pinned host rows / rank] E --> G[decode step] F -->|host misses
1.05% of device time| G ``` ## Why this payload and not another The W4A16 fast path on this fork is reachable **only** through the INC (AutoRound) quant method. `compressed_tensors` checkpoints of the same model — for example `dealignai/GLM-5.3-Flash-CYBERSECURITY-W4A16`, 182 GiB — resolve their MoE through the WNA16 oracle to `TritonWNA16Experts`, which **has no offload recipe**, and the engine refuses to serve at all: 170 GiB of experts cannot fit 64 GiB of VRAM, so offload is mandatory, not optional. The chain that makes Intel's checkpoint work, with the exact evidence: ``` [expert_plan.py] quantization=inc ← inc.py:364 overrides "auto-round" [inc_wna16_scheme.py:249] INC GPTQ MoE on ROCm: R4dW4a16MoEExperts (libr4d grouped W4A16 GEMM) [r4d_w4a16_moe.py:682] Using MoEPrepareAndFinalizeNoDPEPModular / R4dW4a16MoEExperts [expert.py:450] Expert offload: 42 MoE layers staged; 2,383 cache slots in 14.51 GiB planned per rank [expert.py:413] 42 layers split: 0 VRAM-only experts, 2,383 swap slots, arena 14.51 GiB, host 73.64 GiB pinned, host rows prepacked on the GPU ``` `R4dW4a16MoEExperts` is what the offload machinery wraps: the offload plan is format-agnostic but requires the experts class to expose the hot/cold split. ## The numbers, and the wall Measured on the deployed engine, 2026-09-30 (`bench_glm.py`, real prose, greedy): | depth | prefill | decode | | --- | --- | --- | | 2,621 tok prompt, run 1 (cold cache) | 259 tok/s | 9.88 | | 2,621 tok prompt, runs 2-3 (warm) | 918 / 885 tok/s | 9.91 / 9.41 | | 493 tok prompt (telemetry window) | 124 tok/s | 8.90 | Decode is flat across context and across every arena change — the signature of a per-step cost that is not KV- or residency-driven at the margin. ### The measured profile, and the real wall **This section replaces an earlier explanation, which the measurement falsified.** The first reading of this page bounded the step with throughput arithmetic: 42 MoE layers x 8 experts x 6.234 MiB = 2.045 GiB per step per rank, 0.82 GiB of it missed at a measured 0.53-0.66 hit rate, giving "0.82 GiB in ~100 ms = 8.2 GB/s = the gather is the wall". That arithmetic was circular — it *assumed* the gather consumed the whole step — and it is wrong. The engine, asked directly, disagrees: A torch-profiler window (`--profiler-config`, `POST /start_profile` / `stop_profile`, 496 decode tokens at the deployed shape, rank0 trace parsed for device-activity time) gives the true split. Total device kernel time **46.18 s / 496 tokens = ~93 ms per token** against a measured ~104 ms step (9.6 tok/s with the profiler attached): | ms | % | launches | kernel | | --- | --- | --- | --- | | 16,074 | **34.8** | 708,500 | `vectorized_elementwise_kernel` (22.7 us each) | | 4,602 | 10.0 | 42,000 | `r4d_gemm_moe_w4a16_nt_b16_m64` — the MoE GEMM | | 3,108 | 6.7 | 174,708 | `elementwise_kernel_manual_unroll` | | 2,943 | 6.4 | 138,252 | `wvSplitK_hf_sml_` — split-K GEMM (projections / head) | | 2,829 | 6.1 | 83,160 | router: `sbtopk::gatherTopK` + `warpMergeSortTopK` + `bitonicSortKVInPlace` (1,208 + 1,051 + 559 ms; the 1,051 row is not in the list below) | | 3,015 | 6.5 | 133,920 | `r4d_mhc_pre_finish/pre_partial/post` — hyper-connections | | 2,782 | 6.0 | 127,916 | inductor `triton_poi_*` / `triton_per_*` (the fusions that DO happen) | | **487** | **1.05** | 20,664 | **`r4d_moe_lru_gather_kernel`** — the accused | | 250 | 0.54 | 20,832 | `_moe_lru_tele_kernel` — the telemetry that measured it (TELEM was on) | So the gather is **1.05 %** of device time, and `r4d_moe_lru_fused_kernel` (the LRU *manager*, not the copy) is a further 1,599 ms / 3.5 %. Residency work and the PCIe link are not the story: 44 % of the budget is **pointwise glue** — bf16<->fp32 conversions, adds, `pow`/`rsqrt` from fp32 RMSNorm, `launch_clamp_scale` quant passes, `reduce_kernel` means — spread over **~1,428 elementwise launches per token** (3,734 kernel launches per token in total). The GPU is busy 86 % of the wall clock, so this is real device work, not launch-gap overhead. Two consequences that kill three whole families of planned work: - **Spec decode cannot help this shape, and the MTP measurement was not an accident.** Pointwise cost scales with the token count, so a draft batch that verifies 4 tokens pays ~4x the glue for the same per-token output — which is exactly why MTP4 measured "acceptance 3.76-4.90, decode unchanged". Same reason the gather was never the bound. - **Arena size and hit rate are not levers.** Confirmed twice: `--expert-precise-memory` (arena 13.08 -> 14.51 GiB) moved decode by nothing, and the gather is 1 %. ## Fork work: the compile changelist (WIP, does not boot yet) The fork compiles its own flagship trunk — `vllm/models/qwen4_exp/amd/model.py` carries `@support_torch_compile` — and **not** its GLM trunk. The engine says so on every boot: ``` Cudagraph mode FULL_AND_PIECEWISE requires piecewise capture, but the loaded model provides neither a compiled submodule nor breakable CUDA graphs. Overriding to FULL_DECODE_ONLY. ``` Inductor fusion is therefore the only lever that touches the 44 % pointwise mass. Three patches are staged in `scripts/patches/glm53-compile-0*.patch`, each carrying its own evidence header, base/patched sha256 and the next blocker: | patch | target | fixes | | --- | --- | --- | | `-01-model-decorator` | `models/glm5next/common/model.py` | decorates `Glm5NextModel` (env `R4D_GLM_COMPILE`, default on) | | `-02-trace-safe-probes` | `model_executor/kernels/r4d_lib.py` | freezes 10 capability probes that do `os.path.exists`/`.so` lookups **inside the traced forward** | | `-03-fullgraph-optout` | `compilation/wrapper.py` | makes `fullgraph=True` opt-out (env `VLLM_COMPILE_FULLGRAPH=0`) | What the boots established, in order (each one ~7 min): 1. **01 alone dies** on the first cold probe call inside the traced region: `Dynamo does not know how to trace the builtin 'posix.stat'` — `r4d_mhc_supported -> r4d_lib.r4d_mhc_enabled -> available -> os.path.exists`. **`functools.lru_cache` does not protect these probes**: dynamo re-traces the wrapped body to build its guard even when the cache is warm (it warns about exactly this), verified in-image — a *warm* lru_cache-calling function still fails `fullgraph=True`, while a plain module-global read compiles. 2. **01+02 get past it** (`libr4d capability probes frozen: 10 warm [...]` in all four processes) and die on `RuntimeError: Graph breaks are not supported with aot compile`, because this build runs AOT compile by default and AOT refuses breaks. 3. **01+02+03 with `VLLM_COMPILE_FULLGRAPH=0 VLLM_USE_AOT_COMPILE=0` traces the trunk successfully** (`Dynamo bytecode transform time: 1.08 s`) and then dies on the decode forward with `KeyError: 'in_proj_qkvbfg_a'` (`Glm5NextModel.forward` line 748 calling the decoder layer at line 456). `in_proj_qkvbfg_a` is the KDA layer's merged column-parallel projection (`kda.py:217`, called at :407) — a real submodule attribute name, also present in the `load_weights` shard table (`model.py:780-785`). Reading the full traceback (the reported line is inside dynamo-rewritten code) traces the throw to dynamo's own name→module map: `torch/_dynamo/output_graph.py` registers into `nn_modules` at :1259/:1384, looks up with `self.nn_modules[node.target]` at :3000 (a bare-name `KeyError` when absent), and documents stripping `_buffers[..]/_parameters[..]/_modules[..]` from those names at :1225. So the compiler is failing to **resolve a submodule by name** while tracing the GLM decoder layer; whether the root cause is the fork's module tree or dynamo's registration is not yet settled. The next step is the one torch itself suggests in that error — `TORCHDYNAMO_VERBOSE=1` for the internal stack trace (and note the launcher resets `ENVARGS`, so it must be added inside `serve_glm_autoround.sh`'s COMPILE_PATCH branch rather than in the outer environment). 4. **04 removes the fused-norm call from the traced region** (`R4D_GLM_FUSE_QKV_RMSNORM=0`: the GLM MLA block stops calling the fork's fused-norm JIT kernel, whose kernel object in `models/common/ops/fused_qk_rmsnorm.py` is bound *after* the function that uses it — `_FUSED_Q_KV_RMSNORM_KERNEL` at line 223 vs its use at line 211). **It removes a real break** (under `fullgraph=True` the fused call was the first break dynamo reported; after this patch the report moves on to the workspace acquisition) — but it is not by itself what blocks a boot, because with breaks genuinely tolerated the blocker is the name-map `KeyError`. See the run history below. 5. **A launcher bug that made four boots misleading — worth knowing because the failure kept moving.** The scratch launcher's compile branches were restructured on 2026-09-30 and the `-e VLLM_COMPILE_FULLGRAPH=0` line was dropped, so four consecutive boots mounted the wrapper patch while `_FULLGRAPH` still defaulted to True. With `fullgraph=True` **every** break is an error, so those boots reported whichever break came first: | config | outcome | | --- | --- | | `fullgraph=False`, AOT off | breaks tolerated → dies later on the dynamo name-map `KeyError: 'in_proj_qkvbfg_a'` | | `fullgraph=True`, AOT off | dies on the first break (cell → workspace → compiler-disable) | So the reported breaks are real *as breaks*, but they are not what blocks a break-tolerant compile — and an earlier version of this page blamed `TORCH_LOGS=graph_breaks` for the difference. **That was wrong**; the flag only logs. The launcher sets the env var again. **How to see the breaks (and its trap).** `TORCH_LOGS=graph_breaks` names each break with a user stack, and `VLLM_USE_AOT_COMPILE=0` is required because AOT compile hard-refuses breaks — but a fullgraph=True run reports the first break as fatal, so check which configuration you are actually in (`podman exec printenv VLLM_COMPILE_FULLGRAPH`) before concluding a break blocks the boot. For the name-map KeyError there is a ready-made probe: put `/home/dank/dynamo-probe` on `PYTHONPATH` with `VLLM_DYNAMO_DUMP_ATTRS=1` (a `sitecustomize.py` — a `.pth` on `PYTHONPATH` is NOT executed, and the image's own `sitecustomize` is chained so its apport hook survives) and it prints, at the miss, the node target, the near-miss keys, and whether another live `OutputGraph` holds the key. **State of the changelist (2026-09-30, latest).** The trunk **traces** (`Dynamo bytecode transform time: 1.08 s`) and the boot dies on the dynamo name-map `KeyError: 'in_proj_qkvbfg_a'`. Established about it, in order: - It is raised **inside the traced `Glm5NextDecoderLayer.forward`** — the frame chain is `profile_run → _dummy_run → execute_model → glm4_1v.py:2358 → the compile wrapper → eval_frame.compile_wrapper → Glm5NextModel.forward:748 (layer(...)) → module _call_impl → Glm5NextDecoderLayer.forward:456 → KeyError`. `TORCHDYNAMO_VERBOSE=1` adds **no** internal stack trace, i.e. it is a plain Python KeyError in the traced frame, raised in a callee dynamo **inlined** (the reported line is the `def`, which is how inlined frames surface). - It is **not** raised from any of dynamo's four `nn_modules` lookup sites: `/home/dank/dynamo-probe` (a `sitecustomize.py` on `PYTHONPATH`, `VLLM_DYNAMO_DUMP_ATTRS=1`) wraps three of them and stayed silent across two runs with the probe confirmed installed in both the API-server and worker processes. - It is **not** a string-keyed lookup of a literal: `in_proj_qkvbfg_a` appears in the whole tree only as the KDA layer's attribute (`kda.py:217`, used at `:407`) and in the `load_weights` shard table (`model.py:780-785`). The four `proj_name = prefix.split(".")[-1]` sites in the quantization utils do compute exactly this bare key, but every one of them is guarded (`if proj_name not in fused_mapping: return`), so they cannot raise it. The pluggable-layer registry is keyed by class name, not module path. - It reproduces on **every** run of this configuration (5 of 5), and it is independent of patches 04/05: with breaks genuinely tolerated, those remove breaks that were never fatal in the first place — the KeyError is the only thing standing between this changelist and a boot. **Next instrument, precisely:** the inlined caller has to be named. Two cheap ways: `TORCH_LOGS=+dynamo` on a boot and grep the log for the key (dynamo prints inlined frames with their source), or wrap the inlining candidates in that forward — the mHC CustomOps (`hc_pre` / `hc_post` / `hc_fused_post_pre`, `layers/mhc.py`) and the indexer op — with a probe that reports on `KeyError`. Until that names the caller, this is a dynamo-side name-resolution failure and not a fixable fork bug. Two verified facts bound the rest of the shape: `suppress_errors` does **not** cover graph breaks (only `fullgraph=False` does), and the GLM trunk can never satisfy `fullgraph` because its indexer metadata syncs to host (`glm5next/amd/sparse_indexer.py:263,419`: `int(...max().item())`) — both are dead fallbacks on the decode path that the fork author already replaced, so they are not the blockers they appear to be at first glance. ### Two structural constraints on any further fork work - **No new HIP kernels are possible.** The image ships `r4d.so` as a binary with no sources. The obvious second lever — the 9 % router — is gated `num_experts in {128, 256, 512}` in `router/r4d_route_router.py` and the binary contains only `r4d_moe_route_kernel<128|256|512>` instantiations. GLM has **288** experts, so it falls back to the generic sort-based top-k and **cannot be moved to the fused router without rebuilding libr4d**. Only Python/config/inductor changes (and runtime-JIT Triton) are available. - **`--kv-cache-dtype fp8` is a dead end for capacity too.** The plan's KV reserve is dtype-blind (1.76 GiB at 131,072 either way), so fp8 buys no arena and no window; it only removes the only MLA decode path gfx12 has (trap 1 above). ### DFlash2 for GLM — what the drafter actually is `incoai/GLM-5.3-Flash-DFlash2` is **not** a GLM-shaped drafter and needs no MLA/DSA port: its `config.json` declares `architectures: ["DFlash2DraftModel"]` and `model_type: "qwen3"` — 5 dense layers, hidden 4096, 32 heads / 8 kv heads, head_dim 128, sliding attention (window 2048), `is_causal: false`, `use_cache: false`, `dflash_config` = block 8, conv group 16, selector rank 256 / top-k 16, and **`target_layer_ids: [5, 14, 24, 33, 42]` against `num_target_layers: 45`**. The fork already maps that architecture: `registry.py:636 "DFlash2DraftModel": ("qwen3_dflash2", "DFlash2Qwen3ForCausalLM")`, and the spec method is `"dflash"` (`config/speculative.py:67`). So the port is **target-side only**: the GLM trunk must expose auxiliary hidden states at those five layer indices, the same interface `deepseek_v2.py` (also MLA+MoE) implements — `set_aux_hidden_state_layers()` / `get_eagle3_aux_hidden_state_layers()` plus capture at `idx in self.aux_hidden_state_layers` in the layer loop. `glm5next` implements neither. **The load was tried, and the gap is exactly one interface.** Payload fetched to `/home/dank/models/GLM-5.3-Flash-DFlash2` and hash-verified (`model.safetensors` 2,342,169,800 B, sha256 `b038e1d9…` = the Hub's LFS oid). Booting the engine with `--speculative-config '{"method":"dflash","model":"/draft","num_speculative_tokens":8}'` accepts the config and the drafter and dies at worker init with: ``` RuntimeError: Model does not support EAGLE3 interface ``` So nothing about the drafter, the method name, or the payload is wrong — the GLM trunk simply does not implement the aux-hidden-state contract the dflash runner requires. The reference is in this same image, twice: `deepseek_v2.py` (`set_aux_hidden_state_layers` / `get_eagle3_aux_hidden_state_layers` at :1948-1951, capture at :1554 and :1579) and the fork's own `qwen4_exp` trunk. The contract is small: - `self.aux_hidden_state_layers: tuple[int, ...]` on the trunk, default `()`, set by the wrapper's `set_aux_hidden_state_layers()`; - in the layer loop, `if idx in self.aux_hidden_state_layers: aux_hidden_states.append()` (DeepSeek appends `hidden_states + residual`, with a TP all-gather fixup when the rank holds fewer tokens than `positions`); - `Glm5NextModel.forward` returns `(hidden_states, aux_hidden_states)` when the list is non-empty; - `get_eagle3_aux_hidden_state_layers()` supplies a default triple when the drafter config omits `target_layer_ids` (this drafter provides `[5, 14, 24, 33, 42]`, so the runner derives trunk indices `[6, 15, 25, 34, 43]`). **The GLM-specific unknown** is what "the single-stream state entering layer k" means for a trunk whose layers carry mHC multi-stream state: DeepSeek's `(hidden_states, residual)` pair has no counterpart here — GLM's decoder layer returns `(hidden_states, residual, post, comb)` with `hc_expand`/`hc_pre` mixing `hc_mult` streams, so the plausible answer is the pre-mHC single-stream input `x` (4096-wide, matching the drafter's `hidden_size`), not a post-layer value. That is a semantic choice to validate by acceptance rate, not by inspection. **Budget no speedup.** Per the profile above, a verification pass pays pointwise cost proportional to its token count — the same reason MTP4 measured flat — and MTP4's acceptance was already excellent (3.76-4.90). The port is worth doing as *enabling work* (it is the prerequisite for any spec decode on this endpoint, and it is small), not as a performance fix, and the honest expectation on this shape is flat decode with a prefill regression like MTP's until the pointwise mass is fused. Note also that the KV-cap patch's predecessor arithmetic in [[dankserver-qwen38-fp8-expert-offload-tuning]] should be read the same way: its "gather-bound" conclusion came from the same throughput arithmetic that this page's profiler window has now falsified. ## Measured arms (all 2026-09-30, one boot each) | arm | decode | prefill | verdict | | --- | --- | --- | --- | | baseline int4, MAXLEN 131,072, util 0.92, NBATCH 2048 | 10.03 / 10.08 / 9.41 | 877 / 878 / 885 | qualified incumbent | | + `--expert-precise-memory` (arena 13.08 → 14.51 GiB) | 9.88 / 9.91 | 918 | **neutral** — proves residency is no longer the binding term | | + MTP4 spec decode (draft loads, acceptance excellent) | 9.64-11.16 | **255** | **rejected** — see below | | + MTP4 + explicit `--kv-cache-memory-bytes` (patched cap) | boots, 151,767-token KV | — | enabler for the MTP arm only | ### Why MTP is rejected here (it is not that it fails) MTP **works** on GLM in this fork: the head loads, the architecture resolves to `Glm5NextMTPModel`, and acceptance is excellent — ``` SpecDecoding metrics: Mean acceptance length: 3.83-4.90, Per-position acceptance rate: 0.917/0.833/0.583/0.500 … 0.962/0.846/0.692/0.615, Avg Draft acceptance rate: 69-97% ``` — and it buys **nothing**: decode 9.6-11.2 tok/s against 10.0 without it, while prefill **collapses 918 → 255 tok/s**. The reason is structural to offload: a verification forward gathers the **union** of the batch's experts, so spec decode amortizes compute but not a residency-bound gather. It also costs 2.6 GiB of arena. On a compute-bound (fully-resident) shape this would invert; here it does not. ## Served surface (verified 2026-09-30 against the live engine) | capability | state | evidence | | --- | --- | --- | | `/v1/models` | `glm53-flash`, max_model_len 131,072 | model list probe | | completion | yes | `bench_glm.py` runs above | | **tool calls** | **yes** — needs `--enable-auto-tool-choice --tool-call-parser glm47` | a `get_weather` prompt returns `tool_calls: [{"id": "chatcmpl-tool-…", "type": "function", "function": {"name": "get_weather", "arguments": "{\"city\": \"Seattle\"}"}}]`, `finish_reason: tool_calls` | | **vision** | **yes** | a 64x64 solid-red PNG is answered "The image appears to be a solid red color…" | | reasoning channel | **yes** — needs `--reasoning-parser glm47` | without it the thinking leaks into `content` (measured: content began with the answer, then "Let me verify: … = 391", then a bare ` 2** (`is_aiter_found_and_supported()` → `get_cdna_version() > 2`). With fp8 KV the MTP draft step demands an op that cannot exist here: `RuntimeError: Worker failed with error ''_OpNamespace' 'vllm' object has no attribute 'rocm_aiter_mla_decode_fwd''`. KV dtype therefore stays auto/bf16. (`VLLM_ROCM_USE_AITER_MLA=1` does **not** help — the umbrella gate is the CDNA check, not the env var.) - **The capacity payoff is measured at zero.** The point of fp8 KV is fitting more tokens in the same reserve, and this fork's reserve model is **dtype-blind**: the fp8 boot printed the *identical* plan line as bf16 — `KV reserve 1.76 GiB (13600 B/token x 131072 + mamba pages)`. The reserve is dominated by the per-sequence KDA state (34 linear-attention layers) and the DSA indexer's pools, not by the MLA KV bytes those 13,600 B/token barely cover — so halving the KV dtype cannot move it. - **What a fix would take, if it were ever wanted**: a new fp8 path in the rope-free Triton kernels (`rocm_sparse_attn_prefill` / `rocm_sparse_attn_decode` in `v1/attention/ops/rocm_aiter_mla_sparse.py`) that dequantizes pages in-kernel, plus loosening `_use_rocm_sparse_triton()`. The fork's `_r4d_fp8_paged_mqa_logits` shows the author already has an fp8 MQA-logits path, but the sparse-MLA backend is not wired to it. Verdict 2026-09-30: **do not spend effort here** — the payoff is capacity, and the reserve is dtype-blind. 2. **The plan's KV reserve under-models this model, and the cap enforces the mistake.** The plan charges 13,600 B/token + a small mamba term, but GLM carries a per-sequence KDA state (34 linear-attention layers) and, under MTP, the draft's KV. `gpu_worker` then *caps* KV at that reserve, so a legal `--max-model-len` becomes unbootable: `ValueError: To serve at least one request with the model's max seq len (131072), (2.07 GiB KV cache is needed, which is larger than the available KV cache memory (1.74 GiB))`. Raising `max_model_len` only helps past ~440k tokens (the cap's slope is ~13,845 B/token vs the need's ~12,700 + 0.52 GiB fixed) — where the reserve would eat a third of the arena. `scripts/patches/gpu_worker-kv-cap.patch` lets an explicit operator KV request raise the cap; **it is not part of the deployed shape**, which boots on the plan's own reserve. 3. **A window is forced, not chosen.** The production `max_model_len 131072` is what the reserve model requires for this payload (below ~92k the boot refuses); it is not a capacity decision. 4. **Expert prefetch remains poison** — unchanged from the qwen campaign (36% precision, +40-51% decode from deleting it). No GLM arm re-opened it. ## Operations ```bash # rollback to the pre-cutover qwen FP8 + MTP4 shape (payload and R9V state # dirs are untouched on disk): git show 2339fae -- scripts/ | git apply # or: git checkout 2339fae -- scripts/ ./scripts/deploy.sh dankserver --service frank-primary # payload inventory (34 shards + 2 extras, weight-map index): bash scripts/download-frank-primary.sh --verify-only # live telemetry (needs the profiler window; with_stack=false is mandatory — # with stack the stop path hangs and the trace buffer grows to tens of GB): podman logs frank-primary-glm 2>&1 | grep "expert_lru.py" ``` **Experiment-window discipline (learned the hard way, 2026-09-30).** This engine **ignores SIGTERM**: after `systemctl stop` its container sits in `podman ps` state `Stopping` for 20+ minutes *while still answering `/health` with 200*. That makes a liveness probe a lying oracle for a restore — twice a restore script printed `PROD HEALTHY` while `systemctl is-active` said `failed` and the runtime mask was still in place. The restore oracle must be a **conjunction**: `is-active` == `active` **and** `is-enabled` == `enabled` **and** no `/run/systemd/system/` mask symlink **and** the container is owned by the unit and not `Stopping`. The repair recipe, all four steps needed: ```bash sudo mv /run/systemd/system/frank-primary.service /tmp/stale-mask-N.service # runtime mask SURVIVES stop and blocks start sudo systemctl daemon-reload sudo systemctl reset-failed frank-primary.service # a failed unit refuses to start podman rm -f frank-primary-glm # SIGKILL; `podman stop` hangs on the same SIGTERM sudo systemctl start frank-primary.service # then ~7 min to health 200 ``` Also: **never let two experiment jobs overlap.** A later job's `podman rm -f glm-probe` cleanup ran while an earlier job was still mid-flight and deleted its container, so one whole round produced no evidence at all (`no container with name glm-probe found`) — check `proc://` for running jobs before opening a window. ```bash # boot: ~7 min warm page cache, longer cold; health returns 503 while loading curl -s -o /dev/null -w "%{http_code}\n" http://127.0.0.1:7770/health # experiment windows (production stopped + `systemctl mask --runtime`, which # self-clears at reboot) ran from the host-side scratch launcher # /home/dank/serve_glm_autoround.sh: env knobs MAXLEN SEQS NBATCH UTIL KVBYTES # KVDTYPE SPEC SPECJSON_FILE TOOLPARSER REASONINGPARSER TELEM GCHUNKS GLANES # PROFDIR GLM_PATCH_MOUNT COMPILE_PATCH GATHER_PATCH DRAFT_MOUNT ENVARGS. # DRAFT_MOUNT=1 mounts /home/dank/models/GLM-5.3-Flash-DFlash2 at /draft; # SPECJSON_FILE passes a whole --speculative-config JSON (JSON with no spaces, # word-split by the launcher's unquoted $EXTRA). bash /home/dank/serve_glm_autoround.sh ``` Boot gate: the engine refuses a legal-looking shape in two measured ways — the KV cap above, and the profile-run activation peak (`TritonMLA` indexer scores: 64 heads x 2,048 topk x NBATCH tokens x 2 B = 2.06 GiB at NBATCH 8192), which is what killed the first two boots at util 0.95. `PYTORCH_CUDA_ALLOC_CONF=expandable_segments:True` is part of the qualified shape, not a tuning: the plan fills VRAM to its modelled headroom by design, so the default allocator's fragmentation fails the profile/graph-capture phase (measured on the deployed unit: "Tried to allocate 662.00 MiB. GPU 0 … 240.00 MiB is free"). The `deploy.sh` e2e gate also drives the admin panel's open-links spec, which walks every guarded vhost — including `http://192.168.0.112:8989/` on FrankBook-Pro. When that Mac mini is asleep the spec fails on a `page.goto` timeout and the deploy reports `DEPLOY INCOMPLETE` even though the engine is healthy; wake the host (magic packet to `de:6b:6d:c1:52:23`) or re-run `tests/e2e/run.sh` once it is up. Seen 2026-09-30: 32 passed, this one failed, and the WoL packet did not wake it (macOS "Wake for network access" needs toggling on the mini itself). ## Rollback targets | state | where | | --- | --- | | qwen FP8 + MTP4 (2026-09-29 → 09-30) | commit `2339fae`, payload `/home/dank/models/Qwen3.8-Flash-Next-Uncensored-FP8` | | R9V radiance Qwen3.8-Flash-Next Q8_0 | [[frank-primary-r9v-radiance-qwen38-q8]], state dirs `/home/dank/r9v-state/{q8xl,uncensored}` |