serverAppConfigs/serverAppConfigs-docs/LLM/frank-archon-glm53-flash-w4a16.md

---

type: runbook title: frank-archon — GLM-5.3-Flash W4A16 int4 (tcclaviger, 2x R9700) description: VERIFIED 2026-09-30 by boot probe, bench, live expert-cache telemetry AND a torch-profiler kernel-time window. frank-archon cut over from Qwen3.8-Flash-Next-Uncensored-FP8 + MTP4 to Intel/GLM-5.3-Flash-W4A16-AutoRound, the first payload this fleet serves through the fork's W4A16 expert-offload path (INC scheme -> R4dW4a16MoEMethod, libr4d grouped W4A16 GEMM). Measured decode 9.9-10.0 tok/s and prefill 918 tok/s at 131,072 ctx. THE 35 tok/s TARGET IS NOT MET, and the profiler window FALSIFIED the first explanation: the expert gather is 1.05 % of device time, not the bound. 44 % of the 93 ms/token budget is pointwise glue (1,428 elementwise launches per token) because the fork's GLM trunk is NOT torch.compiled while its own Qwen trunk is; the second-largest lever, the fused router, is binary-blocked at 288 experts. The compile changelist is staged but incomplete (three patches, one blocker left). tags:

• dankserver
• r9700
• gfx1201
• glm5.3
• w4a16
• int4
• expert-offload
• expert-cache
• autoround
• vllm timestamp: 2026-09-30

---

frank-archon — GLM-5.3-Flash W4A16 int4 (tcclaviger, 2x R9700)

RETIRED 2026-10-01. Cutover reversed to Qwen3.8-Flash-Next-Uncensored-FP8

• MTP4 (commit 2339fae trio). The DFlash2 drafter experiment is a failure: after five patches the drafter served and drafted (12.8 tok/s proposed, 65,536 ctx, both TP ranks capturing FULL dflash2 cuda graphs) but acceptance pinned at 1.00-1.38 on every text class including verbatim repetition — the signature of a mis-wired aux-hidden-state capture, not a weak drafter. At ~1.2 acceptance the 8-token verification tax makes decode 5x SLOWER than the 9.9-10.0 no-draft baseline (1.6-1.8 tok/s measured). The untested alternative capture semantics: post-layer output state (DeepSeek's hidden_states + residual) or the mHC comb stream. Payload (170 GiB) + drafter (2.2 GiB) staged to /home/dank/.AGENTTRASH/20261001-glm-purge/ pending purge approval. KV-cache work that survives: patches 02/03/04 in scripts/patches/ are generic unifier/tailpin/grouping fixes, apply-verified against the image.

Question this page answers. Can the fleet serve the 320B-class GLM-5.3-Flash at int4 on two R9700s with expert offload, and how fast is it? Answer. Yes — it serves, with vision, tool calls and 131,072 ctx. It runs at 9.9-10.0 tok/s decode / 918 tok/s prefill, and 35 tok/s is not reachable on this payload as the fork ships it. The bound is not the model, the quant, the link, or the expert gather (measured: 1.05 % of device time). It is the unfused pointwise glue of an uncompiled trunk — 44 % of a 93 ms per-token budget spread over ~1,428 elementwise launches per token — because the fork compiles its own Qwen trunk and not its GLM one. See The measured profile.

DFlash2 target-side port — the four laws and the KV ladder

Distilled 2026-10-05 from the probe windows (skills glm53-dflash-boot-ladder / glm53-dflash-eagle3-port); each law was proven by a real failed boot.

The four EAGLE3-port laws (make the GLM trunk satisfy SupportsEagle3):

1. Patch the instance the runner calls. model_runner.py:425 calls set_eagle3_aux_hidden_state_layers(self.model) on the loaded model — GLM loads as Glm5NextForConditionalGeneration (multimodal wrapper; self.language_model = Glm5NextForCausalLM). Members placed only on the CausalLM fail the check on the conditional — same error text, wrong file. Patch BOTH wrappers.
2. Presence-checked member set. SupportsEagle3 (interfaces.py:1659) is @runtime_checkable; isinstance walks ALL __protocol_attrs__ (including base-protocol members) with getattr_static, breaking on the first miss. Both wrappers need, as class attributes: supports_eagle3: ClassVar[Literal[True]] = True, has_own_lm_head: bool = False, has_own_embed_tokens: bool = False. Verify in-image with stock typing — no boot needed.
3. MRO forbids inheriting the Protocol. GLM wrappers carry four/five Protocol-derived bases; adding SupportsEagle3 anywhere dies with "Cannot create a consistent method resolution order" (sim-proven: the ordering rejects it in every base position). The attribute route avoids the MRO, so the two methods must be written by hand: the CausalLM unwraps to self.model (the EagleModelMixin trunk), the conditional delegates to self.language_model (pixtral.py:498 pattern); get_eagle3_default_aux_hidden_state_layers returns (2, L//2, L-3), with the drafter config's target_layer_ids winning when present.
4. Trunk capture point: append aux_hidden_states BEFORE the layer call — GLM's loop carries the single-stream state entering the layer.

The KV blocker ladder (patches scripts/patches/glm53-dflash-0*):

1. EAGLE3 gate → presence-checked attrs on both wrappers (above).
2. KV page-size unifier: MLA layers cannot be padded (sparse MLA reads whole token rows) but CAN grow block_size when the max page is an exact multiple — bump max_page_size in unify_kv_cache_spec_page_size to the lcm of all MLA-family pages first (patch 02).
3. kpool tail assert tail_kv_cache.shape[2] == pool_size: the unifier's divisible branch grew KpoolTailSpec's block; its kernel pins block == index_kpool. Add a KpoolTailSpec branch that pads page_size_padded instead of growing (patch 03).
4. KV capacity: the glm5 layout ALIASES mamba states INTO MLA pages (why GLM fits 32 GiB cards); the generic path (taken when draft/hidden specs exist) cannot alias → ~30 GiB demanded vs a 0.92 GiB reserve. Extend _get_kv_cache_groups_glm5_next to partition HiddenStateCacheSpec + drafter attention specs; the drafter forms its own uniform group (per-group pages, NO unification → the MLA block stays at the engine-auto-set 2,432 from interface.py:918, kpool guard block_size % 512 == 0 unstressed); layout returns a 10-tuple (+draft_group, +hidden_groups); the builder sizes ONE shared backing allocation (allocate_kv_cache asserts all KVCacheTensor sizes equal) = aliased pool + appended draft/hidden tensors (patch 04). After 04 the capacity requirement collapsed 31.76 → 1.88 GiB at 65,536 max len and the engine SERVED (READY ~500 s, dflash2 fused prep ON, drafter FULL graphs on both TP ranks).
5. Reserve registry: _glm53_kv_bytes_per_token must add 4,096 (drafter KV) + 5×hidden×2 (aux hidden cache) keyed on speculative_config.method in {dflash, dspark, extract_hidden_states}. NEVER key on use_aux_hidden_state_outputs — it lives on the gpu_model_runner instance (gpu_model_runner.py:623/678), unset at plan time, so the check silently fails and the reserve stays 13,600 (patch 04b). Proven reserve math at 65,536: 13,600 + 4,096 + 5×4096×2 ≈ 58,656 B/token → ~4.15 GiB reserve, arena ~10.3 GiB.

Verify cheaply before booting: smoke-test the pure KV logic on CPU torch with the image's real spec classes — stub vllm.logger/vllm.utils.*, register the spec classes in KVCacheSpecRegistry with a dummy manager, add from __future__ import annotations, and run synthetic GLM+draft+hidden spec sets through the patched grouping/layout/builder functions. A 7-minute boot per hypothesis is the alternative.

Window discipline: stop + mask frank-archon FIRST and gate on MemAvailable ≥ 230 GiB before booting the probe (the expert-offload planner reads MemAvailable; beside-prod boots die at plan time with ExpertPlanError: host holds 5113 of 12096 swappable experts). Save podman logs glm-probe to a file BEFORE teardown — a podman logs | grep | head pipeline died once under pipefail via SIGPIPE before log-save, losing the capture. The abort path must restore prod (unmask + daemon-reload + reset-failed + start), never bare-exit after stop+mask. Launcher knobs: DFLASH_PATCH=1 (mounts the patched model.py), DRAFT_MOUNT=1, SPECJSON_FILE=… ({"method": "dflash","model":"/draft","num_speculative_tokens":8}) — note the launcher's echo prints patch=$PATCH_MOUNT (the COMPILE knob), not DFLASH_PATCH. Boot timeline: EAGLE3 gate fires ~4 min in at worker init; drafter torch.compile is the frontier (~380 s).

Open telemetry discrepancy: the contaminated window read 1.5–2.1 tok/s with acceptance 1.00–1.25; Frank live-measured ~18 tok/s decode / ~370 tok/s prefill. Reconcile with a clean single-request measurement before trusting either.

Related: dankserver-qwen38-fp8-expert-offload-tuning (the same fork, the same hardware — its conclusion was drawn from throughput arithmetic and is superseded by this page's kernel-time measurement), dankserver-frontier-moe-feasibility (the earlier fit analysis), VLLM Index, LLM/index.

The deployed shape

term	value
payload	Intel/GLM-5.3-Flash-W4A16-AutoRound — AutoRound symmetric int4, group 128, packing auto_round:auto_gptq, 34 shards + 2 extra safetensors, 113,074 tensors, ~170 GiB
runtime	docker.io/tcclaviger/vllm@sha256:da4029666… (same digest-pinned image as the qwen cutover)
engine	TP=2, --enable-expert-offload --expert-precise-memory, max-model-len 131,072 x 2 seqs, bf16 KV, util 0.98, NBATCH 2048, prefix caching
MoE kernel	R4dW4a16MoEMethod → libr4d grouped W4A16 GEMM, 6.234 MiB per expert-rank slot
expert cache	14.51 GiB = 2,383 slots per rank; 12,096 host-backed experts, 73.64 GiB pinned per rank
unit	scripts/frank-archon.service (root unit, LimitMEMLOCK=infinity for the pin)
measured	decode 9.9-10.0 tok/s, prefill 918 tok/s (warm, 2.6k-token prompt), KV cache 138,561 tokens

graph LR
    A[Intel AutoRound W4A16<br/>sym int4 g128, auto_gptq] -->|quant_method auto-round| B[inc.py override to inc]
    B --> C[inc_wna16_scheme<br/>_resolve_gptq_moe]
    C -->|sym int4 g128 + gfx1201 + libr4d| D[R4dW4a16MoEMethod<br/>grouped W4A16 GEMM]
    D --> E[ExpertLRUCache<br/>2,383 VRAM slots / rank]
    D --> F[73.64 GiB pinned host rows / rank]
    E --> G[decode step]
    F -->|host misses<br/>1.05% of device time| G

Why this payload and not another

The W4A16 fast path on this fork is reachable only through the INC (AutoRound) quant method. compressed_tensors checkpoints of the same model — for example dealignai/GLM-5.3-Flash-CYBERSECURITY-W4A16, 182 GiB — resolve their MoE through the WNA16 oracle to TritonWNA16Experts, which has no offload recipe, and the engine refuses to serve at all: 170 GiB of experts cannot fit 64 GiB of VRAM, so offload is mandatory, not optional.

The chain that makes Intel's checkpoint work, with the exact evidence:

[expert_plan.py]  quantization=inc            ← inc.py:364 overrides "auto-round"
[inc_wna16_scheme.py:249]  INC GPTQ MoE on ROCm: R4dW4a16MoEExperts (libr4d grouped W4A16 GEMM)
[r4d_w4a16_moe.py:682]     Using MoEPrepareAndFinalizeNoDPEPModular / R4dW4a16MoEExperts
[expert.py:450]            Expert offload: 42 MoE layers staged; 2,383 cache slots in 14.51 GiB planned per rank
[expert.py:413]            42 layers split: 0 VRAM-only experts, 2,383 swap slots, arena 14.51 GiB,
                           host 73.64 GiB pinned, host rows prepacked on the GPU

R4dW4a16MoEExperts is what the offload machinery wraps: the offload plan is format-agnostic but requires the experts class to expose the hot/cold split.

The numbers, and the wall

Measured on the deployed engine, 2026-09-30 (bench_glm.py, real prose, greedy):

depth	prefill	decode
2,621 tok prompt, run 1 (cold cache)	259 tok/s	9.88
2,621 tok prompt, runs 2-3 (warm)	918 / 885 tok/s	9.91 / 9.41
493 tok prompt (telemetry window)	124 tok/s	8.90

Decode is flat across context and across every arena change — the signature of a per-step cost that is not KV- or residency-driven at the margin.

The measured profile, and the real wall

This section replaces an earlier explanation, which the measurement falsified. The first reading of this page bounded the step with throughput arithmetic: 42 MoE layers x 8 experts x 6.234 MiB = 2.045 GiB per step per rank, 0.82 GiB of it missed at a measured 0.53-0.66 hit rate, giving "0.82 GiB in ~100 ms = 8.2 GB/s = the gather is the wall". That arithmetic was circular — it assumed the gather consumed the whole step — and it is wrong. The engine, asked directly, disagrees:

A torch-profiler window (--profiler-config, POST /start_profile / stop_profile, 496 decode tokens at the deployed shape, rank0 trace parsed for device-activity time) gives the true split. Total device kernel time 46.18 s / 496 tokens = ~93 ms per token against a measured ~104 ms step (9.6 tok/s with the profiler attached):

ms	%	launches	kernel
16,074	34.8	708,500	vectorized_elementwise_kernel (22.7 us each)
4,602	10.0	42,000	r4d_gemm_moe_w4a16_nt_b16_m64 — the MoE GEMM
3,108	6.7	174,708	elementwise_kernel_manual_unroll
2,943	6.4	138,252	wvSplitK_hf_sml_ — split-K GEMM (projections / head)
2,829	6.1	83,160	router: sbtopk::gatherTopK + warpMergeSortTopK + bitonicSortKVInPlace (1,208 + 1,051 + 559 ms; the 1,051 row is not in the list below)
3,015	6.5	133,920	r4d_mhc_pre_finish/pre_partial/post — hyper-connections
2,782	6.0	127,916	inductor triton_poi_* / triton_per_* (the fusions that DO happen)
487	1.05	20,664	r4d_moe_lru_gather_kernel — the accused
250	0.54	20,832	_moe_lru_tele_kernel — the telemetry that measured it (TELEM was on)

So the gather is 1.05 % of device time, and r4d_moe_lru_fused_kernel (the LRU manager, not the copy) is a further 1,599 ms / 3.5 %. Residency work and the PCIe link are not the story: 44 % of the budget is pointwise glue — bf16<->fp32 conversions, adds, pow/rsqrt from fp32 RMSNorm, launch_clamp_scale quant passes, reduce_kernel means — spread over ~1,428 elementwise launches per token (3,734 kernel launches per token in total). The GPU is busy 86 % of the wall clock, so this is real device work, not launch-gap overhead.

Two consequences that kill three whole families of planned work:

• Spec decode cannot help this shape, and the MTP measurement was not an accident. Pointwise cost scales with the token count, so a draft batch that verifies 4 tokens pays ~4x the glue for the same per-token output — which is exactly why MTP4 measured "acceptance 3.76-4.90, decode unchanged". Same reason the gather was never the bound.
• Arena size and hit rate are not levers. Confirmed twice: --expert-precise-memory (arena 13.08 -> 14.51 GiB) moved decode by nothing, and the gather is 1 %.

Fork work: the compile changelist (WIP, does not boot yet)

The fork compiles its own flagship trunk — vllm/models/qwen4_exp/amd/model.py carries @support_torch_compile — and not its GLM trunk. The engine says so on every boot:

Cudagraph mode FULL_AND_PIECEWISE requires piecewise capture, but the loaded model
provides neither a compiled submodule nor breakable CUDA graphs.
Overriding to FULL_DECODE_ONLY.

Inductor fusion is therefore the only lever that touches the 44 % pointwise mass. Three patches are staged in scripts/patches/glm53-compile-0*.patch, each carrying its own evidence header, base/patched sha256 and the next blocker:

patch	target	fixes
-01-model-decorator	models/glm5next/common/model.py	decorates Glm5NextModel (env R4D_GLM_COMPILE, default on)
-02-trace-safe-probes	model_executor/kernels/r4d_lib.py	freezes 10 capability probes that do os.path.exists/.so lookups inside the traced forward
-03-fullgraph-optout	compilation/wrapper.py	makes fullgraph=True opt-out (env VLLM_COMPILE_FULLGRAPH=0)

What the boots established, in order (each one ~7 min):

1. 01 alone dies on the first cold probe call inside the traced region: Dynamo does not know how to trace the builtin 'posix.stat' — r4d_mhc_supported -> r4d_lib.r4d_mhc_enabled -> available -> os.path.exists. functools.lru_cache does not protect these probes: dynamo re-traces the wrapped body to build its guard even when the cache is warm (it warns about exactly this), verified in-image — a warm lru_cache-calling function still fails fullgraph=True, while a plain module-global read compiles.

2. 01+02 get past it (libr4d capability probes frozen: 10 warm [...] in all four processes) and die on RuntimeError: Graph breaks are not supported with aot compile, because this build runs AOT compile by default and AOT refuses breaks.

3. 01+02+03 with VLLM_COMPILE_FULLGRAPH=0 VLLM_USE_AOT_COMPILE=0 traces the trunk successfully (Dynamo bytecode transform time: 1.08 s) and then dies on the decode forward with KeyError: 'in_proj_qkvbfg_a' (Glm5NextModel.forward line 748 calling the decoder layer at line 456). in_proj_qkvbfg_a is the KDA layer's merged column-parallel projection (kda.py:217, called at :407) — a real submodule attribute name, also present in the load_weights shard table (model.py:780-785). Reading the full traceback (the reported line is inside dynamo-rewritten code) traces the throw to dynamo's own name→module map: torch/_dynamo/output_graph.py registers into nn_modules at :1259/:1384, looks up with self.nn_modules[node.target] at :3000 (a bare-name KeyError when absent), and documents stripping _buffers[..]/_parameters[..]/_modules[..] from those names at :1225. So the compiler is failing to resolve a submodule by name while tracing the GLM decoder layer; whether the root cause is the fork's module tree or dynamo's registration is not yet settled. The next step is the one torch itself suggests in that error — TORCHDYNAMO_VERBOSE=1 for the internal stack trace (and note the launcher resets ENVARGS, so it must be added inside serve_glm_autoround.sh's COMPILE_PATCH branch rather than in the outer environment).

4. 04 removes the fused-norm call from the traced region (R4D_GLM_FUSE_QKV_RMSNORM=0: the GLM MLA block stops calling the fork's fused-norm JIT kernel, whose kernel object in models/common/ops/fused_qk_rmsnorm.py is bound after the function that uses it — _FUSED_Q_KV_RMSNORM_KERNEL at line 223 vs its use at line 211). It removes a real break (under fullgraph=True the fused call was the first break dynamo reported; after this patch the report moves on to the workspace acquisition) — but it is not by itself what blocks a boot, because with breaks genuinely tolerated the blocker is the name-map KeyError. See the run history below.

5. A launcher bug that made four boots misleading — worth knowing because the failure kept moving. The scratch launcher's compile branches were restructured on 2026-09-30 and the -e VLLM_COMPILE_FULLGRAPH=0 line was dropped, so four consecutive boots mounted the wrapper patch while _FULLGRAPH still defaulted to True. With fullgraph=True every break is an error, so those boots reported whichever break came first:

   config	outcome
   fullgraph=False, AOT off	breaks tolerated → dies later on the dynamo name-map KeyError: 'in_proj_qkvbfg_a'
   fullgraph=True, AOT off	dies on the first break (cell → workspace → compiler-disable)

   So the reported breaks are real as breaks, but they are not what blocks a break-tolerant compile — and an earlier version of this page blamed TORCH_LOGS=graph_breaks for the difference. That was wrong; the flag only logs. The launcher sets the env var again.

How to see the breaks (and its trap). TORCH_LOGS=graph_breaks names each break with a user stack, and VLLM_USE_AOT_COMPILE=0 is required because AOT compile hard-refuses breaks — but a fullgraph=True run reports the first break as fatal, so check which configuration you are actually in (podman exec <ctr> printenv VLLM_COMPILE_FULLGRAPH) before concluding a break blocks the boot. For the name-map KeyError there is a ready-made probe: put /home/dank/dynamo-probe on PYTHONPATH with VLLM_DYNAMO_DUMP_ATTRS=1 (a sitecustomize.py — a .pth on PYTHONPATH is NOT executed, and the image's own sitecustomize is chained so its apport hook survives) and it prints, at the miss, the node target, the near-miss keys, and whether another live OutputGraph holds the key.

State of the changelist (2026-09-30, latest). The trunk traces (Dynamo bytecode transform time: 1.08 s) and the boot dies on the dynamo name-map KeyError: 'in_proj_qkvbfg_a'. Established about it, in order:

• It is raised inside the traced Glm5NextDecoderLayer.forward — the frame chain is profile_run → _dummy_run → execute_model → glm4_1v.py:2358 → the compile wrapper → eval_frame.compile_wrapper → Glm5NextModel.forward:748 (layer(...)) → module _call_impl → Glm5NextDecoderLayer.forward:456 → KeyError. TORCHDYNAMO_VERBOSE=1 adds no internal stack trace, i.e. it is a plain Python KeyError in the traced frame, raised in a callee dynamo inlined (the reported line is the def, which is how inlined frames surface).
• It is not raised from any of dynamo's four nn_modules lookup sites: /home/dank/dynamo-probe (a sitecustomize.py on PYTHONPATH, VLLM_DYNAMO_DUMP_ATTRS=1) wraps three of them and stayed silent across two runs with the probe confirmed installed in both the API-server and worker processes.
• It is not a string-keyed lookup of a literal: in_proj_qkvbfg_a appears in the whole tree only as the KDA layer's attribute (kda.py:217, used at :407) and in the load_weights shard table (model.py:780-785). The four proj_name = prefix.split(".")[-1] sites in the quantization utils do compute exactly this bare key, but every one of them is guarded (if proj_name not in fused_mapping: return), so they cannot raise it. The pluggable-layer registry is keyed by class name, not module path.
• It reproduces on every run of this configuration (5 of 5), and it is independent of patches 04/05: with breaks genuinely tolerated, those remove breaks that were never fatal in the first place — the KeyError is the only thing standing between this changelist and a boot.

Next instrument, precisely: the inlined caller has to be named. Two cheap ways: TORCH_LOGS=+dynamo on a boot and grep the log for the key (dynamo prints inlined frames with their source), or wrap the inlining candidates in that forward — the mHC CustomOps (hc_pre / hc_post / hc_fused_post_pre, layers/mhc.py) and the indexer op — with a probe that reports on KeyError. Until that names the caller, this is a dynamo-side name-resolution failure and not a fixable fork bug.

Two verified facts bound the rest of the shape: suppress_errors does not cover graph breaks (only fullgraph=False does), and the GLM trunk can never satisfy fullgraph because its indexer metadata syncs to host (glm5next/amd/sparse_indexer.py:263,419: int(...max().item())) — both are dead fallbacks on the decode path that the fork author already replaced, so they are not the blockers they appear to be at first glance.

Two structural constraints on any further fork work

• No new HIP kernels are possible. The image ships r4d.so as a binary with no sources. The obvious second lever — the 9 % router — is gated num_experts in {128, 256, 512} in router/r4d_route_router.py and the binary contains only r4d_moe_route_kernel<128|256|512> instantiations. GLM has 288 experts, so it falls back to the generic sort-based top-k and cannot be moved to the fused router without rebuilding libr4d. Only Python/config/inductor changes (and runtime-JIT Triton) are available.
• --kv-cache-dtype fp8 is a dead end for capacity too. The plan's KV reserve is dtype-blind (1.76 GiB at 131,072 either way), so fp8 buys no arena and no window; it only removes the only MLA decode path gfx12 has (trap 1 above).

DFlash2 for GLM — what the drafter actually is

incoai/GLM-5.3-Flash-DFlash2 is not a GLM-shaped drafter and needs no MLA/DSA port: its config.json declares architectures: ["DFlash2DraftModel"] and model_type: "qwen3" — 5 dense layers, hidden 4096, 32 heads / 8 kv heads, head_dim 128, sliding attention (window 2048), is_causal: false, use_cache: false, dflash_config = block 8, conv group 16, selector rank 256 / top-k 16, and target_layer_ids: [5, 14, 24, 33, 42] against num_target_layers: 45.

The fork already maps that architecture: registry.py:636 "DFlash2DraftModel": ("qwen3_dflash2", "DFlash2Qwen3ForCausalLM"), and the spec method is "dflash" (config/speculative.py:67). So the port is target-side only: the GLM trunk must expose auxiliary hidden states at those five layer indices, the same interface deepseek_v2.py (also MLA+MoE) implements — set_aux_hidden_state_layers() / get_eagle3_default_aux_hidden_state_layers() plus capture at idx in self.aux_hidden_state_layers in the layer loop. glm5next implements neither.

The load was tried, and the gap is exactly one interface. Payload fetched to /home/dank/models/GLM-5.3-Flash-DFlash2 and hash-verified (model.safetensors 2,342,169,800 B, sha256 b038e1d9… = the Hub's LFS oid). Booting the engine with --speculative-config '{"method":"dflash","model":"/draft","num_speculative_tokens":8}' accepts the config and the drafter and dies at worker init with:

RuntimeError: Model does not support EAGLE3 interface

So nothing about the drafter, the method name, or the payload is wrong — the GLM trunk simply does not implement the aux-hidden-state contract the dflash runner requires. The reference is in this same image, twice: deepseek_v2.py (set_aux_hidden_state_layers / get_eagle3_default_aux_hidden_state_layers at :1948-1951, capture at :1554 and :1579) and the fork's own qwen4_exp trunk. The contract is small:

• self.aux_hidden_state_layers: tuple[int, ...] on the trunk, default (), set by the wrapper's set_aux_hidden_state_layers();
• in the layer loop, if idx in self.aux_hidden_state_layers: aux_hidden_states.append(<single-stream state entering layer idx>) (DeepSeek appends hidden_states + residual, with a TP all-gather fixup when the rank holds fewer tokens than positions);
• Glm5NextModel.forward returns (hidden_states, aux_hidden_states) when the list is non-empty;
• get_eagle3_default_aux_hidden_state_layers() supplies a default triple when the drafter config omits target_layer_ids (this drafter provides [5, 14, 24, 33, 42], so the runner derives trunk indices [6, 15, 25, 34, 43]).

The GLM-specific unknown is what "the single-stream state entering layer k" means for a trunk whose layers carry mHC multi-stream state: DeepSeek's (hidden_states, residual) pair has no counterpart here — GLM's decoder layer returns (hidden_states, residual, post, comb) with hc_expand/hc_pre mixing hc_mult streams, so the plausible answer is the pre-mHC single-stream input x (4096-wide, matching the drafter's hidden_size), not a post-layer value. That is a semantic choice to validate by acceptance rate, not by inspection.

Budget no speedup. Per the profile above, a verification pass pays pointwise cost proportional to its token count — the same reason MTP4 measured flat — and MTP4's acceptance was already excellent (3.76-4.90). The port is worth doing as enabling work (it is the prerequisite for any spec decode on this endpoint, and it is small), not as a performance fix, and the honest expectation on this shape is flat decode with a prefill regression like MTP's until the pointwise mass is fused.

The port exists now: scripts/patches/glm53-dflash-01-aux-hidden-states.patch (WIP, independent of the compile changelist — it patches the same file as compile-01, so compile-01 goes first if both are applied; base 008eeaa2… → patched bc9913a1…). It implements exactly the contract read out of this image: EagleModelMixin on Glm5NextModel, capture of the single-stream hidden_states ENTERING each requested layer in the loop, the conditional (hidden_states, aux_hidden_states) tuple return, and the three SupportsEagle3 member attributes plus both interface methods on the wrappers (capture shape from deepseek_v2.py:1551-1585; tuple consumption at v1/worker/gpu/cudagraph_utils.py:621).

The patch went through five versions, and the decisive discovery was where the instance actually lives. The traceback's set_eagle3_aux_hidden_state_layers(self.model) is called on the model the runner LOADED — and for GLM-5.3-Flash that is Glm5NextForConditionalGeneration (the multimodal wrapper; the APIServer log names it), which holds the text model as self.language_model = Glm5NextForCausalLM (glm5next/common/model.py:1113). Patching only Glm5NextForCausalLM (v4) still failed with the identical error because the isinstance ran against the wrapper. v5 fixes both wrappers and is the first version to clear the gate (both TP workers log Using Eagle3 auxiliary layers from config: (6, 15, 25, 34, …)).

The full evolution, each an instructive trap:

1. v1 hand-wrote the two methods the traceback names → identical error. The methods are not the whole contract.
2. v2 inherited SupportsEagle3 as a base (the image's own idiom — laguna.py:717, mimo_v2.py:834, …) → dies at class creation with Cannot create a consistent method resolution order (MRO) for bases …. GLM's wrappers already carry four/five Protocol-derived bases, and simulation confirmed the ForCausalLM's five-base ordering cannot take SupportsEagle3 in any position.
3. v3 declared the members as class attributes (MRO-safe) — but only on Glm5NextForCausalLM → identical error (the wrong wrapper, see above).
4. v4 added all three members (supports_eagle3, has_own_lm_head, has_own_embed_tokens — a runtime-checkable Protocol's isinstance walks ALL __protocol_attrs__ including base-Protocol members via getattr_static, verified experimentally in-image with stock typing) on Glm5NextForCausalLM → identical error.
5. v5 adds the three attributes to BOTH wrappers + the two methods on both (Glm5NextForCausalLM unwraps to self.model — the EagleModelMixin trunk; Glm5NextForConditionalGeneration delegates to self.language_model, the pixtral.py:498 pattern for exactly this wrapper shape) → gate passes.

The lesson generalizes twice over: when a runtime error names an interface, read the interface's declaration first — and establish WHICH INSTANCE the check runs against before patching anything: multimodal wrappers (…ForConditionalGeneration) are loaded models, not pass-throughs, and the text model behind self.language_model is invisible to a check run on the wrapper.

Retired diagnostic (do not resurrect as-is): the /probe startup hook. An op_probe.py + sitecustomize.py pair was mounted to wrap vLLM module forwards and torch's name-map lookups. It produced no evidence and caused a fork bomb (see the hazard note under experiment-window discipline). Use TORCH_LOGS for dynamo questions instead — it needs no startup hook.

Note also that the KV-cap patch's predecessor arithmetic in dankserver-qwen38-fp8-expert-offload-tuning should be read the same way: its "gather-bound" conclusion came from the same throughput arithmetic that this page's profiler window has now falsified.

The KV-cache grouping journey: clearing the unifier, then the kpool tail

With patch 01 v5 the gate passes and the boot moves to KV-cache layout construction — where the next two blockers live, both in v1/core/kv_cache_utils.py:

1. The page-size unifier refuses GLM's MLA family. With a draft model the spec dict gains non-MLA drafter attention specs and HiddenStateCacheSpec entries, so _get_kv_cache_groups_glm5_next (which requires ALL attention specs to be MLAAttentionSpec) bails to the generic grouping path — whose unify_kv_cache_spec_page_size raises NotImplementedError: Layer language_model.model.layers.3.self_attn.indexer.k_cache: page size is not divisible… cannot be padded. Padding is only supported for non-MLA attention layers. MLA layers cannot be padded (sparse MLA indexes whole token rows) — but they CAN grow block_size when the max page is an exact multiple. The indexer's page is just tokens_per_state x the MLA page, so glm53-dflash-02-kv-unify-mla-multiple.patch bumps the unified max page to a multiple of every MLAAttentionSpec page before the loop: all MLA-family layers then take the legal divisible block-growth branch, drafter + mamba layers pad. The 2026-09-30 window proved it: unify passes, memory profiling passes, torch.compile passes, and the boot reaches cudagraph capture — the NotImplementedError is gone.

2. The kpool tail kernel pins its block. Cudagraph capture then dies in kpool_ops.kpool_decode_update_and_maybe_write_cache_batched: assert tail_kv_cache.shape[2] == pool_size. The tail cache must view as [num_blocks, 2, index_kpool, head_dim] (KpoolTailSpec(block_size=index_kpool), Glm5NextTailCache in glm5next/common/attention.py:159), but the tail's page (8192 B) divided the raised max page, so the unifier's divisible branch grew its block_size by the ratio and the view became [num_blocks, 2, index_kpool x ratio, head_dim]. Verified not the hybrid promotion: the log carries no "Hybrid KV cache manager is disabled" warning, so disable_hybrid_kv_cache_manager is unset and unify_hybrid_kv_cache_specs never ran. glm53-dflash-03-kv-unify-tailpin.patch (= 02 + a KpoolTailSpec branch that pads page_size_padded instead of ever growing its block) verified 2026-09-30: the assert is gone and the boot captures the DFlash2 drafter's FULL cuda graphs on both TP workers (Capturing model for DFlash2 speculator...). Supersedes 02 in the repo.

3. The capacity wall: the generic path cannot alias the mamba states. Past capture, the boot dies at _check_enough_kv_cache_memory: Available KV cache memory: 0.92 GiB against 31.76 GiB needed at max len 65536 (32.91 GiB at 131,072 — nearly flat, a fixed intercept, not per-token KV). Two coupled causes, read out of the image:

   • The glm5 tensor layout aliases the mamba states INTO the MLA pages (get_kv_cache_config_from_groups places every mamba layer at the MLA offsets — that is why GLM-5.3 fits 32 GiB cards at all, and why the expert plan's KV registry is calibrated at 13,600 B/token). On the generic path the mamba groups claim their OWN blocks, so ~30 GiB of unaliased state pages appears.
   • The plan's KV reserve (expert_activation_registry.py:_glm53_kv_bytes_per_token, a flat 13,600 B/token) knows nothing about the drafter. The engine caps KV at the plan's reserve, so even a correct generic-path estimate would be starved.

   Learned en route: the engine AUTO-SETS the attention block size to 2,432 tokens (interface.py:918: Setting attention block size to 2432 tokens to ensure that attention page size is >= mamba page size), so every attention group's block counts scale from that floor — a 16-token drafter block would need 4x the shared num_blocks; with the global 2,432 all groups align.

   glm53-dflash-04-kv-groups-draft.patch (+ -04b-kv-registry-dflash.patch for the registry) keeps the aliased glm5 pool and extends it: the grouping partitions the hidden-state specs (HiddenStateCacheSpec is an MLAAttentionSpec subclass — excluded before the exact-type type(spec) is MLAAttentionSpec partition) and the drafter attention specs; the drafter layers form their own uniform group (per-group pages, NO unification — the MLA block size stays at the engine-set 2,432 and the kpool guard block_size % 512 == 0 is never stressed); the layout returns a 10-tuple with draft_group/hidden_groups; the builder sizes ONE shared backing allocation (allocate_kv_cache asserts all tensor sizes equal) as aliased pool + appended draft/hidden tensors with their own strides; the memory estimator adds the extras. Patch 04b extends the registry: 13,600 + 4,096 (drafter 8 kv heads x 128 x K+V x bf16) + 5 x hidden_size x 2 B (the aux hidden-state cache) per token when speculative_config.use_aux_hidden_state_outputs is set.

   The pure logic is smoke-tested locally (CPU torch + the image's real spec classes against synthetic GLM+draft+hidden spec sets): grouping shape, layout recognition, builder block math and memory accounting all pass. Real-fit arithmetic: per-token ≈ 58,656 B → reserve ≈ 4.15 GiB at max len 65,536 (arena shrinks ~13.5 → ~10.3 GiB, ~1,650 slots), num_blocks ≈ 29 x 2,432 tokens ≈ 70,500 ≥ 65,536.

All of these mount over site-packages via the scratch launcher's KVUNIFY_PATCH=1 knob (host /home/dank/dflash-patch2/new/…, patched shas 5521e50b… for kv_cache_utils.py and 1a623d5c… for expert_activation_registry.py; apply-verified against the image's pristine files). The window script's decision pipeline was also fixed for a SIGPIPE death (head -80 on a podman logs grep pipeline killed one run before log-save and restore; the log is now captured to a file FIRST, then grepped).

Measured arms (all 2026-09-30, one boot each)

arm	decode	prefill	verdict
baseline int4, MAXLEN 131,072, util 0.92, NBATCH 2048	10.03 / 10.08 / 9.41	877 / 878 / 885	qualified incumbent
+ --expert-precise-memory (arena 13.08 → 14.51 GiB)	9.88 / 9.91	918	neutral — proves residency is no longer the binding term
+ MTP4 spec decode (draft loads, acceptance excellent)	9.64-11.16	255	rejected — see below
+ MTP4 + explicit --kv-cache-memory-bytes (patched cap)	boots, 151,767-token KV	—	enabler for the MTP arm only

Why MTP is rejected here (it is not that it fails)

MTP works on GLM in this fork: the head loads, the architecture resolves to Glm5NextMTPModel, and acceptance is excellent —

SpecDecoding metrics: Mean acceptance length: 3.83-4.90,
Per-position acceptance rate: 0.917/0.833/0.583/0.500 … 0.962/0.846/0.692/0.615,
Avg Draft acceptance rate: 69-97%

— and it buys nothing: decode 9.6-11.2 tok/s against 10.0 without it, while prefill collapses 918 → 255 tok/s. The reason is structural to offload: a verification forward gathers the union of the batch's experts, so spec decode amortizes compute but not a residency-bound gather. It also costs 2.6 GiB of arena. On a compute-bound (fully-resident) shape this would invert; here it does not.

Served surface (verified 2026-09-30 against the live engine)

capability	state	evidence
/v1/models	glm53-flash, max_model_len 131,072	model list probe
completion	yes	bench_glm.py runs above
tool calls	yes — needs --enable-auto-tool-choice --tool-call-parser glm47	a get_weather prompt returns tool_calls: [{"id": "chatcmpl-tool-…", "type": "function", "function": {"name": "get_weather", "arguments": "{\"city\": \"Seattle\"}"}}], finish_reason: tool_calls
vision	yes	a 64x64 solid-red PNG is answered "The image appears to be a solid red color…"
reasoning channel	yes — needs --reasoning-parser glm47	without it the thinking leaks into content (measured: content began with the answer, then "Let me verify: … = 391", then a bare </think) with an empty reasoning field; with it, a train-time prompt answers in content ("Step 1: Add the hours. 14:20 + 3 hours = 17:20 …") and the chain of thought lands in reasoning — the vLLM field name the fleet documents (llama.cpp endpoints use reasoning_content)

Without the tool flags the engine answers HTTP 400 — "auto" tool choice requires --enable-auto-tool-choice and --tool-call-parser to be set — which breaks every tool-using agent on the endpoint. Both parsers are glm47, the newest GLM dialect the fork ships (glm45/glm47 exist for both).

Fork traps found (for the engine authors)

1. --kv-cache-dtype fp8 is illegal on gfx12 with this model — and it buys nothing even if fixed. The ROCM_AITER_MLA_SPARSE backend's _use_rocm_sparse_triton() requires a non-fp8 KV cache, and that rope-free Triton path is the only MLA decode available on gfx12: aiter's MLA decode op is registered under @if_aiter_supported, which is gated on CDNA > 2 (is_aiter_found_and_supported() → get_cdna_version() > 2). With fp8 KV the MTP draft step demands an op that cannot exist here: RuntimeError: Worker failed with error ''_OpNamespace' 'vllm' object has no attribute 'rocm_aiter_mla_decode_fwd''. KV dtype therefore stays auto/bf16. (VLLM_ROCM_USE_AITER_MLA=1 does not help — the umbrella gate is the CDNA check, not the env var.)
   • The capacity payoff is measured at zero. The point of fp8 KV is fitting more tokens in the same reserve, and this fork's reserve model is dtype-blind: the fp8 boot printed the identical plan line as bf16 — KV reserve 1.76 GiB (13600 B/token x 131072 + mamba pages). The reserve is dominated by the per-sequence KDA state (34 linear-attention layers) and the DSA indexer's pools, not by the MLA KV bytes those 13,600 B/token barely cover — so halving the KV dtype cannot move it.
   • What a fix would take, if it were ever wanted: a new fp8 path in the rope-free Triton kernels (rocm_sparse_attn_prefill / rocm_sparse_attn_decode in v1/attention/ops/rocm_aiter_mla_sparse.py) that dequantizes pages in-kernel, plus loosening _use_rocm_sparse_triton(). The fork's _r4d_fp8_paged_mqa_logits shows the author already has an fp8 MQA-logits path, but the sparse-MLA backend is not wired to it. Verdict 2026-09-30: do not spend effort here — the payoff is capacity, and the reserve is dtype-blind.
2. The plan's KV reserve under-models this model, and the cap enforces the mistake. The plan charges 13,600 B/token + a small mamba term, but GLM carries a per-sequence KDA state (34 linear-attention layers) and, under MTP, the draft's KV. gpu_worker then caps KV at that reserve, so a legal --max-model-len becomes unbootable: ValueError: To serve at least one request with the model's max seq len (131072), (2.07 GiB KV cache is needed, which is larger than the available KV cache memory (1.74 GiB)). Raising max_model_len only helps past ~440k tokens (the cap's slope is ~13,845 B/token vs the need's ~12,700 + 0.52 GiB fixed) — where the reserve would eat a third of the arena. scripts/patches/gpu_worker-kv-cap.patch lets an explicit operator KV request raise the cap; it is not part of the deployed shape, which boots on the plan's own reserve.
3. A window is forced, not chosen. The production max_model_len 131072 is what the reserve model requires for this payload (below ~92k the boot refuses); it is not a capacity decision.
4. Expert prefetch remains poison — unchanged from the qwen campaign (36% precision, +40-51% decode from deleting it). No GLM arm re-opened it.

Operations

# rollback to the pre-cutover qwen FP8 + MTP4 shape (payload and R9V state
# dirs are untouched on disk):
git show 2339fae -- scripts/ | git apply   # or: git checkout 2339fae -- scripts/
./scripts/deploy.sh dankserver --service frank-archon

# payload inventory (34 shards + 2 extras, weight-map index):
bash scripts/download-frank-archon.sh --verify-only

# live telemetry (needs the profiler window; with_stack=false is mandatory —
# with stack the stop path hangs and the trace buffer grows to tens of GB):
podman logs frank-archon-glm 2>&1 | grep "expert_lru.py"

Experiment-window discipline (learned the hard way, 2026-09-30). This engine ignores SIGTERM: after systemctl stop its container sits in podman ps state Stopping for 20+ minutes while still answering /health with 200. That makes a liveness probe a lying oracle for a restore — twice a restore script printed PROD HEALTHY while systemctl is-active said failed and the runtime mask was still in place. The restore oracle must be a conjunction: is-active == active and is-enabled == enabled and no /run/systemd/system/<unit> mask symlink and the container is owned by the unit and not Stopping. The repair recipe, all four steps needed:

sudo mv /run/systemd/system/frank-archon.service /tmp/stale-mask-N.service  # runtime mask SURVIVES stop and blocks start
sudo systemctl daemon-reload
sudo systemctl reset-failed frank-archon.service                            # a failed unit refuses to start
podman rm -f frank-archon-glm                                               # SIGKILL; `podman stop` hangs on the same SIGTERM
sudo systemctl start frank-archon.service                                   # then ~7 min to health 200

Also: never let two experiment jobs overlap. A later job's podman rm -f glm-probe cleanup ran while an earlier job was still mid-flight and deleted its container, so one whole round produced no evidence at all (no container with name glm-probe found) — check proc:// for running jobs before opening a window.

Never put a diagnostic hook on PYTHONPATH without a marker guard — it is a fork bomb. On 2026-09-30 a sitecustomize.py mounted at /probe (to wrap dynamo's name-map lookups and vLLM module forwards) produced a self-replicating chain of python3 processes 19+ levels deep inside the probe container: sitecustomize is imported by every interpreter, the hook imported torch/vLLM at startup, something in that import chain spawned a helper python, and the child inherited PYTHONPATH (hence the hook) and spawned again. It was still alive 18 minutes later with the container still showing Up. Symptoms: ps -eo pid,ppid,comm --forest shows a deep ladder of identical interpreters, each child's PPID being the previous one. Guard (now in the hook): arm once per environment via a marker in os.environ, which children inherit —

if os.environ.get("VLLM_PROBE_ARMED") == "1":
    pass                       # an ancestor already armed it; stay inert
else:
    os.environ["VLLM_PROBE_ARMED"] = "1"
    <install probes>

Kill recipe: podman rm -f <experiment-container> (the container cgroup owns every generation), then verify with a precise count over /proc/<pid>/environ for the marker variable — never pattern-kill an interpreter name, because the media stack on this host (romm, jellyfin, plex, glances) runs python in production.

# boot: ~7 min warm page cache, longer cold; health returns 503 while loading
curl -s -o /dev/null -w "%{http_code}\n" http://127.0.0.1:7770/health

# experiment windows (production stopped + `systemctl mask --runtime`, which
# self-clears at reboot) ran from the host-side scratch launcher
# /home/dank/serve_glm_autoround.sh: env knobs MAXLEN SEQS NBATCH UTIL KVBYTES
# KVDTYPE SPEC SPECJSON_FILE TOOLPARSER REASONINGPARSER TELEM GCHUNKS GLANES
# PROFDIR GLM_PATCH_MOUNT COMPILE_PATCH GATHER_PATCH DRAFT_MOUNT ENVARGS.
# DRAFT_MOUNT=1 mounts /home/dank/models/GLM-5.3-Flash-DFlash2 at /draft;
# SPECJSON_FILE passes a whole --speculative-config JSON (JSON with no spaces,
# word-split by the launcher's unquoted $EXTRA).
bash /home/dank/serve_glm_autoround.sh

Boot gate: the engine refuses a legal-looking shape in two measured ways — the KV cap above, and the profile-run activation peak (TritonMLA indexer scores: 64 heads x 2,048 topk x NBATCH tokens x 2 B = 2.06 GiB at NBATCH 8192), which is what killed the first two boots at util 0.95.

PYTORCH_CUDA_ALLOC_CONF=expandable_segments:True is part of the qualified shape, not a tuning: the plan fills VRAM to its modelled headroom by design, so the default allocator's fragmentation fails the profile/graph-capture phase (measured on the deployed unit: "Tried to allocate 662.00 MiB. GPU 0 … 240.00 MiB is free").

The deploy.sh e2e gate also drives the admin panel's open-links spec, which walks every guarded vhost — including http://192.168.0.112:8989/ on FrankBook-Pro. When that Mac mini is asleep the spec fails on a page.goto timeout and the deploy reports DEPLOY INCOMPLETE even though the engine is healthy; wake the host (magic packet to de:6b:6d:c1:52:23) or re-run tests/e2e/run.sh once it is up. Seen 2026-09-30: 32 passed, this one failed, and the WoL packet did not wake it (macOS "Wake for network access" needs toggling on the mini itself).

Rollback targets

state	where
qwen FP8 + MTP4 (2026-09-29 → 09-30)	commit 2339fae, payload /home/dank/models/Qwen3.8-Flash-Next-Uncensored-FP8
R9V radiance Qwen3.8-Flash-Next Q8_0	frank-archon-r9v-radiance-qwen38-q8, state dirs /home/dank/r9v-state/{q8xl,uncensored}