diff --git a/AGENTS.md b/AGENTS.md
index 581679d..155a2cb 100644
--- a/AGENTS.md
+++ b/AGENTS.md
@@ -24,7 +24,7 @@ merge absorbed the former `vllmScripts` repo, history preserved).
| Target | Service | Runtime | Model | Port |
|--------|---------|---------|-------|------|
-| Dankserver | `frank-primary` | tcclaviger RDNA4 vLLM fork (`docker.io/tcclaviger/vllm@sha256:da4029666…`, digest-pinned, plain `podman run`, root systemd unit) | `Intel/GLM-5.3-Flash-W4A16-AutoRound` — AutoRound symmetric int4 g128 (170 GiB, 34 shards, 113,074 tensors), TP=2, expert offload to 73.64 GiB/rank of pinned host rows behind a 14.51 GiB / 2,383-slot VRAM LRU arena, 131072 ctx × 2 seqs, bf16 KV, no MTP, no prefetch; measured 9.9-10.0 tok/s decode / 918 tok/s prefill | 7770 |
+| Dankserver | `frank-primary` | tcclaviger RDNA4 vLLM fork (`docker.io/tcclaviger/vllm@sha256:da4029666…`, digest-pinned, plain `podman run`, root systemd unit) | `Intel/GLM-5.3-Flash-W4A16-AutoRound` — AutoRound symmetric int4 g128 (170 GiB, 34 shards, 113,074 tensors), TP=2, expert offload to 73.64 GiB/rank of pinned host rows behind a 14.51 GiB / 2,383-slot VRAM LRU arena, 131072 ctx × 2 seqs, bf16 KV, no MTP, no prefetch; measured 9.9-10.0 tok/s decode / 918 tok/s prefill, bounded by unfused pointwise glue (the expert gather is 1.05 % of device time; the trunk is uncompiled) | 7770 |
| Dankclaw | `frank-large` | gufo `gufo-org/gufo` v0.1.1, the Strix Halo engine, BUILT FROM SOURCE (root systemd unit; links the host ROCm 7.2.4) | Swift-1.5-Qwen3.8-27B-Uncensored UD-Q8_K_XL (27B dense, ABLITERATED/uncensored, 29.3 GiB, SWIFT OPEN LICENSE 1.0) with the z-lab DFlash2 Q4_K_M drafter (adaptive block policy, cap 7) on the gfx1151 iGPU (Strix Halo LPDDR, 114 GiB GTT), 262144 ctx single session, greedy, gufo's compiled-in Qwen renderer (NO Qwen-Sharp), text only | 7770 |
| Dankclaw | `frank-small` | mainline llama.cpp master 37b53fd (root systemd unit) | HauhauCS Gemma4-12B-QAT-Uncensored-Balanced Q4_K_M (BF16 mmproj + MTP assistant sidecar, `draft-mtp,ngram-mod`) on the RTX 3090, 1048576 pool / 4 slots of 262,144, q8_0 KV, the fleet's fastest vision endpoint | 7771 |
@@ -445,14 +445,51 @@ serve. Built shape: 14.51 GiB arena = **2,383 slots × 6.234 MiB per rank** over
12,096 host-backed experts = **73.64 GiB pinned host rows per rank**.
**Measured 2026-09-30: decode 9.9–10.0 tok/s, prefill 918 tok/s** (warm, 2.6k
-prompt), KV cache 138,561 tokens. **The 35 tok/s target is NOT met, and this is
-a measured wall rather than a tuning gap:** each decode step must move 2.045 GiB
-of routed experts per rank (42 layers × 8 experts × 6.234 MiB), live telemetry
-(`R4D_LRU_TELEMETRY=1`) puts the expert-cache hit rate at **0.53–0.66**, and the
-fork's gather read path delivers **~8.2 GB/s against a link measured at
-22.74 GB/s** — the same structural gap the qwen campaign found. Growing the
-arena 13.08 → 14.51 GiB changed decode by *nothing* (10.03 → 9.88 tok/s), so
-residency stopped being the binding term; 0.89 hit rate would be needed for 35.
+prompt), KV cache 138,561 tokens. **The 35 tok/s target is NOT met. The first
+explanation for that — "the expert gather is the bound at ~8.2 GB/s" — was
+FALSIFIED by a torch-profiler kernel-time window** (496 decode tokens, rank0
+trace parsed for device-activity time): 46.18 s of kernel time = ~93 ms per
+token, of which **44 % is pointwise glue** — `vectorized_elementwise_kernel`
+alone is 16.07 s over **708,500 launches (1,428 per token)**, plus 3.11 s of
+`elementwise_kernel_manual_unroll` — against **1.05 % for
+`r4d_moe_lru_gather_kernel`**. The MoE GEMM is 10 %, the split-K GEMM 6 %, the
+router/top-k 9 %, the mHC kernels 6.5 %. The real wall is that **the fork
+compiles its own Qwen trunk (`@support_torch_compile` in
+`models/qwen4_exp/amd/model.py`) and not its GLM trunk**, so every pointwise op
+is a separate eager kernel. Two consequences: spec decode **cannot** help this
+shape (pointwise cost scales with the token count, which is exactly why MTP4
+measured flat), and neither arena size nor hit rate is a lever (the gather is
+1 %).
+
+**Fork work is staged, not deployed.** `scripts/patches/glm53-compile-0{1,2,3}*.patch`
+are the compile-enablement changelist (decorate the trunk; freeze the ten libr4d
+capability probes that do `os.path.exists`/`.so` lookups *inside* the traced
+forward; make `fullgraph=True` opt-out). Boots established that
+`functools.lru_cache` does **not** protect those probes (dynamo re-traces the
+wrapped body even warm), that `suppress_errors` does **not** cover graph breaks
+(only `fullgraph=False` does), and that the GLM trunk can never satisfy
+`fullgraph` (its indexer metadata syncs to host at
+`glm5next/amd/sparse_indexer.py:263,419`). With all three the trunk **traces**
+(`Dynamo bytecode transform time: 1.08 s`) and then dies on
+`KeyError: 'in_proj_qkvbfg_a'` in the mHC/KDA path — a latent bug the compile
+exposes, and the remaining blocker.
+
+**No new HIP kernels are possible** — the image ships `r4d.so` as a binary with
+no sources. The 9 % router is therefore **binary-blocked**: its fused path is
+gated `num_experts in {128, 256, 512}` (`router/r4d_route_router.py`) and only
+`r4d_moe_route_kernel<128|256|512>` instantiations exist, while GLM has **288**
+experts. Only Python/config/inductor changes (or runtime-JIT Triton) are
+available on this fork.
+
+**DFlash2 for GLM is a target-side port, not a drafter port.**
+`incoai/GLM-5.3-Flash-DFlash2` declares `architectures: ["DFlash2DraftModel"]`
+and `model_type: "qwen3"` (5 dense layers, sliding attention, block 8,
+`target_layer_ids [5, 14, 24, 33, 42]` of 45), and the fork **already maps it**
+(`registry.py:636` → `qwen3_dflash2.DFlash2Qwen3ForCausalLM`; spec method
+`"dflash"`). What is missing is the target side: `glm5next` implements neither
+`set_aux_hidden_state_layers()` nor the per-layer aux capture that
+`deepseek_v2.py` (also MLA+MoE) demonstrates. Budget no speedup: per the profile
+above a verification pass pays pointwise cost proportional to its token count.
**MTP works here and is still rejected.** The head loads
(`Resolved architecture: Glm5NextMTPModel`) and accepts well (mean accepted
diff --git a/scripts/patches/glm53-compile-01-model-decorator.patch b/scripts/patches/glm53-compile-01-model-decorator.patch
new file mode 100644
index 0000000..675190f
--- /dev/null
+++ b/scripts/patches/glm53-compile-01-model-decorator.patch
@@ -0,0 +1,67 @@
+# glm53-compile-01-model-decorator.patch — put the GLM-5.3-Flash trunk under torch.compile.
+#
+# STATUS: WIP — first of three patches in the "compile the GLM trunk" changelist. The
+# set does NOT yet boot the engine (see 03 for the next blocker), so it is NOT part of
+# the deployed frank-primary shape. Nothing in scripts/ mounts these files.
+#
+# WHY THIS IS THE RIGHT LEVER (measured 2026-09-30, not inferred):
+#
+# The fork's own flagship trunk — vllm/models/qwen4_exp/amd/model.py — carries
+# @support_torch_compile. vllm/models/glm5next/common/model.py does not, and the engine
+# says so on every boot:
+#
+# Cudagraph mode FULL_AND_PIECEWISE requires piecewise capture, but the loaded model
+# provides neither a compiled submodule nor breakable CUDA graphs.
+# Overriding to FULL_DECODE_ONLY.
+#
+# A torch-profiler window on the live endpoint (496 decode tokens, rank0, 3 x 160-token
+# generations at the deployed shape) shows what the missing compilation costs. Total
+# device kernel time 46.18 s / 496 tokens = ~93 ms per token, against a measured step
+# time of ~104 ms (9.6 tok/s with the profiler attached):
+#
+# 16.07 s 34.8 % 708,500 launches vectorized_elementwise_kernel (22.7 us each)
+# 4.60 s 10.0 % 42,000 r4d_gemm_moe_w4a16_nt_b16_m64 (the MoE GEMM)
+# 3.11 s 6.7 % 174,708 elementwise_kernel_manual_unroll
+# 2.94 s 6.4 % 138,252 wvSplitK_hf_sml_ (split-K GEMM)
+# 4.09 s 8.9 % 83,160 topk/router (sbtopk gatherTopK + warpMergeSortTopK
+# + bitonicSortKVInPlace)
+# 3.01 s 6.5 % 133,920 r4d_mhc_* (hyper-connection pre/post/finish)
+# 2.78 s 6.0 % 127,916 inductor triton_poi_* / triton_per_* (the only
+# fusions that DO happen)
+# 0.49 s 1.05% 20,664 r4d_moe_lru_gather_kernel <-- the suspect
+#
+# That last row retired the fleet's working hypothesis. Decode on this shape is NOT
+# expert-gather bound: the gather is 1 % of device time, the same order as the telemetry
+# kernel that was switched on to measure it. 44 % of the budget is pointwise glue —
+# bf16<->fp32 conversions, adds, pows/rsqrts from fp32 RMSNorm, clamp/scale quant
+# passes — spread over ~1,428 launches per token. Inductor fusion is the only lever
+# that touches that mass; the router is second at 9 % but is binary-blocked (the
+# libr4d route kernel exists only as r4d_moe_route_kernel<128|256|512> instantiations
+# and GLM has 288 experts, with no HIP sources shipped to rebuild).
+#
+# TARGET: vllm/models/glm5next/common/model.py (inside the pinned image)
+# base sha256 008eeaa2a07e091465319f86900b5a321d921238723ccbe22e6b9e971e1d2b28
+# patched sha256 6e9cb81b84a9a88bd41403ca404aa04c3b791899dd77d661c3f25ae686cecd40
+#
+# APPLY: `patch -p1` from the vllm package root, or bind-mount new/.../model.py over
+# site-packages. Then R4D_GLM_COMPILE=1 (default) enables it; =0 makes the decorator a
+# no-op, so the file can stay mounted for an A/B without a second artifact.
+# Requires 02 (trace-safe probes) — without it the worker dies during profile_run on
+# the first probe call inside the traced region.
+--- a/vllm/models/glm5next/common/model.py
++++ b/vllm/models/glm5next/common/model.py
+@@ Glm5NextModel: add @support_torch_compile (env): R4D_GLM_COMPILE=0 disables @@
++_GLM_COMPILE = os.environ.get("R4D_GLM_COMPILE", "1") == "1"
++
++
++@support_torch_compile(
++ dynamic_arg_dims={
++ "input_ids": 0,
++ "positions": -1,
++ "intermediate_tensors": 0,
++ "inputs_embeds": 0,
++ },
++ enable_if=lambda _cfg: _GLM_COMPILE,
++)
+-class Glm5NextModel(nn.Module):
++class Glm5NextModel(nn.Module): # decorated above
diff --git a/scripts/patches/glm53-compile-02-trace-safe-probes.patch b/scripts/patches/glm53-compile-02-trace-safe-probes.patch
new file mode 100644
index 0000000..fdaffb3
--- /dev/null
+++ b/scripts/patches/glm53-compile-02-trace-safe-probes.patch
@@ -0,0 +1,114 @@
+# glm53-compile-02-trace-safe-probes.patch — libr4d capability probes must not do I/O
+# inside a torch.compile region.
+#
+# STATUS: WIP — second of three patches in the "compile the GLM trunk" changelist.
+# Inert (but harmless) without 01. Not part of the deployed shape.
+#
+# THE BREAK (measured: with 01 mounted and R4D_GLM_COMPILE=1, the worker dies during
+# profile_run, i.e. the first — cold — forward):
+#
+# Explanation: Dynamo does not know how to trace the builtin `posix.stat`
+# ... r4d_mhc_supported -> r4d_lib.r4d_mhc_enabled -> available -> os.path.exists
+#
+# THE NON-OBVIOUS PART: functools.lru_cache does NOT protect these probes. Dynamo
+# re-traces the wrapped body to build its guard even when the cache is warm — it warns
+# about exactly this ("Dynamo detected a call to a `functools.lru_cache`-wrapped
+# function ... Dynamo ignores the cache wrapper and directly traces the wrapped
+# function"). Verified in-image, torch 2.11.0+rocm10.0, torch.compile(backend="eager"):
+#
+# cold lru_cache call -> RAISE Unsupported
+# WARM lru_cache call -> RAISE Unsupported <-- warming is not enough
+# plain module-global read -> compiles
+# uncached os.path.exists -> RAISE Unsupported
+#
+# So each probe must resolve ONCE into a plain value and be read as a global. The set
+# below is every no-argument probe / symbol lookup reachable from a forward pass,
+# grep-verified call sites:
+# model_executor/layers/fused_moe/experts/r4d_w4a16_moe.py:79 available()
+# model_executor/layers/fused_moe/experts/r4d_w4a16_moe.py:81 has_w4a16_moe()
+# model_executor/kernels/mhc/r4d_dsv4.py:16 r4d_mhc_enabled()
+# v1/attention/ops/rocm_aiter_mla_sparse.py:694 r4d_indexer_enabled(), mqa_max_heads()
+# v1/attention/ops/rocm_aiter_mla_sparse.py:1124 r4d_indexer_enabled()
+# model_executor/layers/fused_moe/experts/r4d_mxfp4_moe.py:683 r4d_act_quant_enabled()
+# model_executor/layers/fused_moe/experts/r4d_w4a16_moe.py _entry() moe_w4a16()/moe_w4a16_mt1() per MoE call
+# (lib() is the transitive dependency of all of them.)
+#
+# TARGET: vllm/model_executor/kernels/r4d_lib.py (inside the pinned image)
+# base sha256 b8dfeefb4acc72b67b0579a4ab0653ef5d48c40cc8649187d757c80ce6b9617a
+# patched sha256 0751852e1ca813ad2bdb1da7bf10c605130e63e04ef9ec57e70b5f675a29c08e
+#
+# APPLY: append the block below to the module. R4D_FREEZE_PROBES=0 restores stock
+# behaviour. Cost when nothing is compiled: ten one-time probes at import (measured:
+# logged once per process, e.g. "libr4d capability probes frozen: 10 warm [...]").
+#
+# OBSERVED EFFECT: with 01+02 the worker gets past this break and reaches the next one
+# (see 03) — dynamo reports "Dynamo bytecode transform time: 1.08 s" for the trunk.
+--- a/vllm/model_executor/kernels/r4d_lib.py
++++ b/vllm/model_executor/kernels/r4d_lib.py
+@@ append: trace-safe capability probes (env): R4D_FREEZE_PROBES=0 disables @@
++# FORK-LOCAL (patch: trace-safe capability probes). torch.compile traces the
++# BODY of a functools.lru_cache call even when the cache is warm, so a capability
++# probe called from a forward pass is a dynamo graph break. Measured 2026-09-30
++# on the GLM-5.3-Flash trunk under @support_torch_compile:
++# Dynamo does not know how to trace the builtin `posix.stat`
++# r4d_mhc_supported -> r4d_mhc_enabled -> available -> os.path.exists
++# Each probe below is wrapped to resolve ONCE into a memo and is warmed at import
++# (outside any tracing region), so a traced call is a dict read. The probe set is
++# every no-argument capability probe / symbol lookup reachable from a forward
++# pass: the W4A16 + mxfp4 MoE dispatch, the mHC op, the MLA sparse indexer gate
++# and the draft/lm-head guards.
++_PROBE_NAMES = (
++ "available",
++ "lib",
++ "has_w4a16_moe",
++ "moe_w4a16",
++ "moe_w4a16_mt1",
++ "has_mhc_dsv4",
++ "r4d_mhc_enabled",
++ "r4d_indexer_enabled",
++ "mqa_max_heads",
++ "r4d_act_quant_enabled",
++)
++
++
++def _freeze_probe(fn):
++ """Return fn bound to its first result. Result must be a plain value."""
++
++ memo = []
++
++ def probe(*args, **kwargs):
++ if not memo:
++ memo.append(fn(*args, **kwargs))
++ return memo[0]
++
++ return probe
++
++
++def _freeze_probes() -> list[str]:
++ frozen = []
++ for name in _PROBE_NAMES:
++ fn = globals().get(name)
++ if fn is None:
++ continue
++ # Only the fast branch of a bound probe is traced; an empty memo means the
++ # first call still does I/O, so populate it here as well.
++ globals()[name] = _freeze_probe(fn)
++ frozen.append(name)
++ return frozen
++
++
++from vllm.logger import init_logger as _init_logger
++
++_probe_log = _init_logger(__name__)
++
++if os.environ.get("R4D_FREEZE_PROBES", "1") == "1":
++ _warmed, _failed = [], []
++ for _n in _freeze_probes():
++ try:
++ globals()[_n]()
++ _warmed.append(_n)
++ except Exception: # noqa: BLE001 -- probe unavailable here; recompute lazily
++ _failed.append(_n)
++ _probe_log.info("libr4d capability probes frozen: %d warm %s%s",
++ len(_warmed), _warmed,
++ f" (deferred: {_failed})" if _failed else "")
diff --git a/scripts/patches/glm53-compile-03-fullgraph-optout.patch b/scripts/patches/glm53-compile-03-fullgraph-optout.patch
new file mode 100644
index 0000000..a0140f2
--- /dev/null
+++ b/scripts/patches/glm53-compile-03-fullgraph-optout.patch
@@ -0,0 +1,64 @@
+# glm53-compile-03-fullgraph-optout.patch — let torch.compile split the graph instead
+# of failing on a break.
+#
+# STATUS: WIP — third of three patches in the "compile the GLM trunk" changelist.
+# With 01+02+03 the model TRACES, then hits the NEXT blocker (below). The set does not
+# yet boot the engine; nothing in scripts/ mounts it.
+#
+# THE BREAK: with 01+02 mounted the worker dies with
+#
+# RuntimeError: Graph breaks are not supported with aot compile.
+# Please use torch.compile(fullgraph=True).
+#
+# because vllm/compilation/wrapper.py calls torch.compile(..., fullgraph=True, ...) and
+# this build has AOT compile on by default (VLLM_USE_AOT_COMPILE). The GLM trunk cannot
+# satisfy fullgraph: its indexer metadata path syncs to the host —
+# models/glm5next/amd/sparse_indexer.py:263 int(positions[...].max().item()) + 1
+# models/glm5next/amd/sparse_indexer.py:419 lmax = int(decode_metadata.decode_lens.max().item())
+# — and a .item() inside a captured graph is not expressible.
+#
+# WHAT WAS VERIFIED (in-image, torch 2.11.0+rocm10.0, torch.compile(backend="eager")):
+#
+# fullgraph=True suppress_errors=False -> RAISE Unsupported
+# fullgraph=True suppress_errors=True -> RAISE Unsupported <-- suppress_errors
+# does NOT cover breaks
+# fullgraph=False suppress_errors=False -> ok (traceable segments compiled by
+# inductor, the rest runs eagerly;
+# numerics unchanged)
+#
+# So breaks must be TOLERATED, not suppressed: the traceable mass still fuses. This
+# patch makes the fullgraph requirement opt-out via VLLM_COMPILE_FULLGRAPH=0 (default 1,
+# so every other model keeps stock behaviour), and the run also needs
+# VLLM_USE_AOT_COMPILE=0 because AOT compile refuses breaks outright.
+#
+# TARGET: vllm/compilation/wrapper.py (inside the pinned image)
+# base sha256 969961dca41c78385212e985cfc0c57ef067f4a2d7490c73c658c600a1b95cac
+# patched sha256 732363bb1a1367772f13c60a655c678e7e2e9e56186677b14238f5e84cac31b3
+# (the file already imports os; the patch does not add a second import)
+#
+# NEXT BLOCKER — NOT FIXED HERE. With 01+02+03 and
+# R4D_GLM_COMPILE=1 VLLM_COMPILE_FULLGRAPH=0 VLLM_USE_AOT_COMPILE=0 the trunk traces
+# ("Dynamo bytecode transform time: 1.08 s") and then the worker dies during profile_run:
+#
+# KeyError: 'in_proj_qkvbfg_a'
+# vllm/models/glm5next/common/model.py:456 in Glm5NextDecoderLayer.forward
+#
+# `in_proj_qkvbfg_a` is the KDA layer's merged column-parallel projection
+# (models/glm5next/common/kda.py:217, called at :407). The key is looked up in a
+# dict/mapping that the EAGER path never reaches, so compiling the trunk exposes a
+# latent bug in the mHC/KDA wiring rather than a dynamo limitation. Fix that and
+# whatever follows before this changelist can boot — then re-measure: the prize is the
+# 44 % pointwise mass (see 01), and the frozen-probe log line is the signal that 02 is
+# live.
+--- a/vllm/compilation/wrapper.py
++++ b/vllm/compilation/wrapper.py
+@@ fullgraph opt-out (env VLLM_COMPILE_FULLGRAPH=0) @@
++# FORK-LOCAL (patch: fullgraph opt-out). torch.compile(fullgraph=True) turns any
++# dynamo graph break into a hard failure; suppress_errors does NOT cover breaks
++# (verified in-image, torch 2.11+rocm10.0). With VLLM_COMPILE_FULLGRAPH=0 the
++# traceable segments are still compiled by inductor and the untraceable ones run
++# eagerly, which is what a model with device-sync metadata (the GLM-5.3-Flash
++# indexer) needs. Default 1 keeps stock behaviour for every other model.
++_FULLGRAPH = os.environ.get("VLLM_COMPILE_FULLGRAPH", "1") == "1"
+- fullgraph=True,
++ fullgraph=_FULLGRAPH,
diff --git a/serverAppConfigs-docs/LLM/frank-primary-glm53-flash-w4a16.md b/serverAppConfigs-docs/LLM/frank-primary-glm53-flash-w4a16.md
index f6e3b68..b60e578 100644
--- a/serverAppConfigs-docs/LLM/frank-primary-glm53-flash-w4a16.md
+++ b/serverAppConfigs-docs/LLM/frank-primary-glm53-flash-w4a16.md
@@ -1,7 +1,7 @@
---
type: runbook
title: frank-primary — GLM-5.3-Flash W4A16 int4 (tcclaviger, 2x R9700)
-description: VERIFIED 2026-09-30 by boot probe, bench and live expert-cache telemetry. frank-primary cut over from Qwen3.8-Flash-Next-Uncensored-FP8 + MTP4 to Intel/GLM-5.3-Flash-W4A16-AutoRound, the first payload this fleet serves through the fork's W4A16 expert-offload path (INC scheme -> R4dW4a16MoEMethod, libr4d grouped W4A16 GEMM). Measured decode 9.9-10.0 tok/s and prefill 918 tok/s at 131,072 ctx. THE 35 tok/s TARGET IS NOT MET and this page shows why it is out of reach: the step is bounded by the fork's expert-gather read path at ~8.2 GB/s effective against a PCIe link that delivers 22.74 GB/s, with the cache at its measured hit-rate ceiling (0.53-0.66 per layer) from an arena that cannot grow past ~15 GiB under a 73.64 GiB/rank expert set.
+description: VERIFIED 2026-09-30 by boot probe, bench, live expert-cache telemetry AND a torch-profiler kernel-time window. frank-primary cut over from Qwen3.8-Flash-Next-Uncensored-FP8 + MTP4 to Intel/GLM-5.3-Flash-W4A16-AutoRound, the first payload this fleet serves through the fork's W4A16 expert-offload path (INC scheme -> R4dW4a16MoEMethod, libr4d grouped W4A16 GEMM). Measured decode 9.9-10.0 tok/s and prefill 918 tok/s at 131,072 ctx. THE 35 tok/s TARGET IS NOT MET, and the profiler window FALSIFIED the first explanation: the expert gather is 1.05 % of device time, not the bound. 44 % of the 93 ms/token budget is pointwise glue (1,428 elementwise launches per token) because the fork's GLM trunk is NOT torch.compiled while its own Qwen trunk is; the second-largest lever, the fused router, is binary-blocked at 288 experts. The compile changelist is staged but incomplete (three patches, one blocker left).
tags:
- dankserver
- r9700
@@ -22,12 +22,16 @@ timestamp: 2026-09-30
> `GLM-5.3-Flash` at int4 on two R9700s with expert offload, and how fast is it?
> **Answer.** Yes — it serves, with vision, tool calls and 131,072 ctx. It runs
> at **9.9-10.0 tok/s decode / 918 tok/s prefill**, and **35 tok/s is not
-> reachable** with this payload on this hardware. The bound is not the model,
-> the quant, or the link: it is the fork's expert-gather read path, and it is
-> measured, not inferred.
+> reachable** on this payload as the fork ships it. The bound is **not** the
+> model, the quant, the link, or the expert gather (measured: 1.05 % of device
+> time). It is the **unfused pointwise glue of an uncompiled trunk** — 44 % of a
+> 93 ms per-token budget spread over ~1,428 elementwise launches per token —
+> because the fork compiles its own Qwen trunk and not its GLM one. See
+> [The measured profile](#the-measured-profile-and-the-real-wall).
-Related: [[dankserver-qwen38-fp8-expert-offload-tuning]] (the same fork, the
-same hardware, the same gather-path wall, found first on qwen),
+Related: [[dankserver-qwen38-fp8-expert-offload-tuning]] (the same fork, the same
+hardware — its conclusion was drawn from throughput arithmetic and is superseded
+by this page's kernel-time measurement),
[[dankserver-frontier-moe-feasibility]] (the earlier fit analysis),
[[VLLM Index]], [[LLM/index|LLM fleet]].
@@ -51,7 +55,7 @@ graph LR
D --> E[ExpertLRUCache
2,383 VRAM slots / rank]
D --> F[73.64 GiB pinned host rows / rank]
E --> G[decode step]
- F -->|8.2 GB/s effective| G
+ F -->|host misses
1.05% of device time| G
```
## Why this payload and not another
@@ -90,33 +94,142 @@ Measured on the deployed engine, 2026-09-30 (`bench_glm.py`, real prose, greedy)
Decode is flat across context and across every arena change — the signature of a
per-step cost that is not KV- or residency-driven at the margin.
-### The arithmetic that bounds it
+### The measured profile, and the real wall
-Per decode step, per rank: 42 MoE layers x 8 routed experts x 6.234 MiB =
-**2.045 GiB of expert weights**, which must either be in VRAM or cross PCIe.
+**This section replaces an earlier explanation, which the measurement falsified.**
+The first reading of this page bounded the step with throughput arithmetic: 42 MoE
+layers x 8 experts x 6.234 MiB = 2.045 GiB per step per rank, 0.82 GiB of it
+missed at a measured 0.53-0.66 hit rate, giving "0.82 GiB in ~100 ms = 8.2 GB/s =
+the gather is the wall". That arithmetic was circular — it *assumed* the gather
+consumed the whole step — and it is wrong. The engine, asked directly, disagrees:
-| hit rate | miss traffic | step @ 8.2 GB/s | decode |
+A torch-profiler window (`--profiler-config`, `POST /start_profile` /
+`stop_profile`, 496 decode tokens at the deployed shape, rank0 trace parsed for
+device-activity time) gives the true split. Total device kernel time **46.18 s /
+496 tokens = ~93 ms per token** against a measured ~104 ms step (9.6 tok/s with
+the profiler attached):
+
+| ms | % | launches | kernel |
| --- | --- | --- | --- |
-| 0.60 (measured) | 0.82 GiB | ~105 ms | **9.5-10.0 tok/s** |
-| 0.70 | 0.61 GiB | ~78 ms | 12.8 tok/s |
-| 0.89 (what 35 tok/s needs) | 0.23 GiB | ~29 ms | 35 tok/s |
+| 16,074 | **34.8** | 708,500 | `vectorized_elementwise_kernel` (22.7 us each) |
+| 4,602 | 10.0 | 42,000 | `r4d_gemm_moe_w4a16_nt_b16_m64` — the MoE GEMM |
+| 3,108 | 6.7 | 174,708 | `elementwise_kernel_manual_unroll` |
+| 2,943 | 6.4 | 138,252 | `wvSplitK_hf_sml_` — split-K GEMM (projections / head) |
+| 2,829 | 6.1 | 83,160 | router: `sbtopk::gatherTopK` + `warpMergeSortTopK` + `bitonicSortKVInPlace` (1,208 + 1,051 + 559 ms; the 1,051 row is not in the list below) |
+| 3,015 | 6.5 | 133,920 | `r4d_mhc_pre_finish/pre_partial/post` — hyper-connections |
+| 2,782 | 6.0 | 127,916 | inductor `triton_poi_*` / `triton_per_*` (the fusions that DO happen) |
+| **487** | **1.05** | 20,664 | **`r4d_moe_lru_gather_kernel`** — the accused |
+| 250 | 0.54 | 20,832 | `_moe_lru_tele_kernel` — the telemetry that measured it (TELEM was on) |
-Two independent measurements pin the missed term:
+So the gather is **1.05 %** of device time, and `r4d_moe_lru_fused_kernel` (the
+LRU *manager*, not the copy) is a further 1,599 ms / 3.5 %. Residency work and the
+PCIe link are not the story: 44 % of the budget is **pointwise glue** — bf16<->fp32
+conversions, adds, `pow`/`rsqrt` from fp32 RMSNorm, `launch_clamp_scale` quant
+passes, `reduce_kernel` means — spread over **~1,428 elementwise launches per
+token** (3,734 kernel launches per token in total). The GPU is busy 86 % of the
+wall clock, so this is real device work, not launch-gap overhead.
-- **Live expert-cache telemetry** (`R4D_LRU_TELEMETRY=1`, profiler window)
- gives the hit rate directly — the LRU is *working*:
- `layers.34: slots 56 (pinned 0, tier 0) hit 0.6279 routed 2002 installs 543
- read-through 202` … `layers.44: hit 0.5335`. Per-layer range **0.53-0.66**.
-- **The link delivers 22.74 GB/s** at the engine's own slab size (H2D probe,
- pinned, 2.345 MiB). The step moves 0.82 GiB in ~100 ms = **8.2 GB/s —
- 36% of the wire**, matching the fork's own measured 6.5-9.5 GB/s.
+Two consequences that kill three whole families of planned work:
-So the step is bounded by the fork's gather read path, and the only other lever
-is the hit rate — which is capped by the arena. A 14.51 GiB arena holds 20% of a
-73.64 GiB expert set; the measured hit rate beats random by ~3x thanks to
-routing locality, but it saturates: growing the arena 13.08 → 14.51 GiB
-(+9%) changed decode by **nothing** (10.03 → 9.88 tok/s). Reaching 0.89 would
-need roughly 4.5x the per-layer slot count — an arena of well over 100 GiB.
+- **Spec decode cannot help this shape, and the MTP measurement was not an
+ accident.** Pointwise cost scales with the token count, so a draft batch that
+ verifies 4 tokens pays ~4x the glue for the same per-token output — which is
+ exactly why MTP4 measured "acceptance 3.76-4.90, decode unchanged". Same reason
+ the gather was never the bound.
+- **Arena size and hit rate are not levers.** Confirmed twice: `--expert-precise-memory`
+ (arena 13.08 -> 14.51 GiB) moved decode by nothing, and the gather is 1 %.
+
+## Fork work: the compile changelist (WIP, does not boot yet)
+
+The fork compiles its own flagship trunk — `vllm/models/qwen4_exp/amd/model.py`
+carries `@support_torch_compile` — and **not** its GLM trunk. The engine says so on
+every boot:
+
+```
+Cudagraph mode FULL_AND_PIECEWISE requires piecewise capture, but the loaded model
+provides neither a compiled submodule nor breakable CUDA graphs.
+Overriding to FULL_DECODE_ONLY.
+```
+
+Inductor fusion is therefore the only lever that touches the 44 % pointwise mass.
+Three patches are staged in `scripts/patches/glm53-compile-0*.patch`, each carrying
+its own evidence header, base/patched sha256 and the next blocker:
+
+| patch | target | fixes |
+| --- | --- | --- |
+| `-01-model-decorator` | `models/glm5next/common/model.py` | decorates `Glm5NextModel` (env `R4D_GLM_COMPILE`, default on) |
+| `-02-trace-safe-probes` | `model_executor/kernels/r4d_lib.py` | freezes 10 capability probes that do `os.path.exists`/`.so` lookups **inside the traced forward** |
+| `-03-fullgraph-optout` | `compilation/wrapper.py` | makes `fullgraph=True` opt-out (env `VLLM_COMPILE_FULLGRAPH=0`) |
+
+What the boots established, in order (each one ~7 min):
+
+1. **01 alone dies** on the first cold probe call inside the traced region:
+ `Dynamo does not know how to trace the builtin 'posix.stat'` —
+ `r4d_mhc_supported -> r4d_lib.r4d_mhc_enabled -> available -> os.path.exists`.
+ **`functools.lru_cache` does not protect these probes**: dynamo re-traces the
+ wrapped body to build its guard even when the cache is warm (it warns about
+ exactly this), verified in-image — a *warm* lru_cache-calling function still
+ fails `fullgraph=True`, while a plain module-global read compiles.
+2. **01+02 get past it** (`libr4d capability probes frozen: 10 warm [...]` in all
+ four processes) and die on
+ `RuntimeError: Graph breaks are not supported with aot compile`, because this
+ build runs AOT compile by default and AOT refuses breaks.
+3. **01+02+03 with `VLLM_COMPILE_FULLGRAPH=0 VLLM_USE_AOT_COMPILE=0` traces the
+ trunk successfully** (`Dynamo bytecode transform time: 1.08 s`) and then dies in
+ `profile_run` with `KeyError: 'in_proj_qkvbfg_a'` at
+ `glm5next/common/model.py:456` (the KDA layer's merged projection,
+ `kda.py:217`) — a dict/registry lookup keyed by attribute name that the eager
+ path never reaches, i.e. **a latent bug the compile exposes, not a dynamo
+ limit**.
+
+That last row is the state of the work: the remaining blocker is a real bug in the
+fork's mHC/KDA wiring, and it must be fixed (plus whatever follows it) before the
+changelist can boot. Two verified facts make the payoff concrete: `suppress_errors`
+does **not** cover graph breaks (only `fullgraph=False` does), and the GLM trunk can
+never satisfy `fullgraph` because its indexer metadata syncs to host
+(`glm5next/amd/sparse_indexer.py:263,419`: `int(...max().item())`).
+
+### Two structural constraints on any further fork work
+
+- **No new HIP kernels are possible.** The image ships `r4d.so` as a binary with no
+ sources. The obvious second lever — the 9 % router — is gated
+ `num_experts in {128, 256, 512}` in `router/r4d_route_router.py` and the binary
+ contains only `r4d_moe_route_kernel<128|256|512>` instantiations. GLM has **288**
+ experts, so it falls back to the generic sort-based top-k and **cannot be moved to
+ the fused router without rebuilding libr4d**. Only Python/config/inductor changes
+ (and runtime-JIT Triton) are available.
+- **`--kv-cache-dtype fp8` is a dead end for capacity too.** The plan's KV reserve
+ is dtype-blind (1.76 GiB at 131,072 either way), so fp8 buys no arena and no
+ window; it only removes the only MLA decode path gfx12 has (trap 1 above).
+
+### DFlash2 for GLM — what the drafter actually is
+
+`incoai/GLM-5.3-Flash-DFlash2` is **not** a GLM-shaped drafter and needs no MLA/DSA
+port: its `config.json` declares `architectures: ["DFlash2DraftModel"]` and
+`model_type: "qwen3"` — 5 dense layers, hidden 4096, 32 heads / 8 kv heads,
+head_dim 128, sliding attention (window 2048), `is_causal: false`,
+`use_cache: false`, `dflash_config` = block 8, conv group 16, selector rank 256 /
+top-k 16, and **`target_layer_ids: [5, 14, 24, 33, 42]` against
+`num_target_layers: 45`**.
+
+The fork already maps that architecture:
+`registry.py:636 "DFlash2DraftModel": ("qwen3_dflash2", "DFlash2Qwen3ForCausalLM")`,
+and the spec method is `"dflash"` (`config/speculative.py:67`). So the port is
+**target-side only**: the GLM trunk must expose auxiliary hidden states at those five
+layer indices, the same interface `deepseek_v2.py` (also MLA+MoE) implements —
+`set_aux_hidden_state_layers()` / `get_eagle3_aux_hidden_state_layers()` plus
+capture at `idx in self.aux_hidden_state_layers` in the layer loop. `glm5next`
+implements neither.
+
+Expected benefit is **not** assumed: DFlash2 amortizes a draft block (8 tokens) over
+one verification pass, and per the profile above a verification pass costs pointwise
+work proportional to tokens — the same reason MTP4 was flat. Measure it; do not
+budget on it.
+
+Note also that the KV-cap patch's predecessor arithmetic in
+[[dankserver-qwen38-fp8-expert-offload-tuning]] should be read the same way: its
+"gather-bound" conclusion came from the same throughput arithmetic that this page's
+profiler window has now falsified.
## Measured arms (all 2026-09-30, one boot each)
diff --git a/serverAppConfigs-docs/log.md b/serverAppConfigs-docs/log.md
index 20773c7..b6f9d7b 100644
--- a/serverAppConfigs-docs/log.md
+++ b/serverAppConfigs-docs/log.md
@@ -20,17 +20,58 @@
- Measured on the deployed engine: **decode 9.9-10.0 tok/s, prefill 918 tok/s**
(warm, 2.6k prompt), KV 138,561 tokens; arena 14.51 GiB = 2,383 slots per
rank over 12,096 host-backed experts (73.64 GiB pinned per rank).
- - **The wall:** live expert-cache telemetry (`R4D_LRU_TELEMETRY=1` + a
- `with_stack=false` profiler window) puts the cache hit rate at 0.53-0.66 per
- layer; a step moves 0.82 GiB of misses in ~100 ms = **8.2 GB/s effective
- against a link measured at 22.74 GB/s** — the same structural gather-read-path
- gap the 2026-09-29 qwen campaign found. Growing the arena 13.08 -> 14.51 GiB
- (+9%) changed decode by *nothing* (10.03 -> 9.88 tok/s), so residency stopped
- being the binding term; 35 tok/s would need a 0.89 hit rate.
+ - **The wall — and a corrected one.** The first account of the shortfall
+ bounded the step with throughput arithmetic (a step moves 0.82 GiB of misses
+ in ~100 ms = "8.2 GB/s effective against a 22.74 GB/s link", needing a 0.89
+ hit rate for 35 tok/s). **A torch-profiler kernel-time window falsified it**
+ (496 decode tokens, rank0 trace parsed for device-activity time): 46.18 s of
+ kernel time = ~93 ms/token, of which **44 % is pointwise glue** —
+ `vectorized_elementwise_kernel` alone 16.07 s over **708,500 launches (1,428
+ per token)**, plus 3.11 s of `elementwise_kernel_manual_unroll` — against
+ **1.05 % for `r4d_moe_lru_gather_kernel`**. The MoE GEMM is 10 %, the split-K
+ GEMM 6.4 %, the router/top-k 8.9 %, the mHC kernels 6.5 %, and the inductor
+ fusions that DO happen 6 %. The real cause is that the fork compiles its own
+ Qwen trunk (`@support_torch_compile` in `models/qwen4_exp/amd/model.py`) and
+ **not** its GLM trunk, so every pointwise op is a separate eager kernel; the
+ engine says so at boot ("...provides neither a compiled submodule nor
+ breakable CUDA graphs. Overriding to FULL_DECODE_ONLY."). Growing the arena
+ 13.08 -> 14.51 GiB still changed nothing, consistent with a 1 % gather.
+ - **Fork work staged (compile changelist, WIP — does not boot yet).**
+ `scripts/patches/glm53-compile-01-model-decorator.patch`,
+ `-02-trace-safe-probes.patch`, `-03-fullgraph-optout.patch`, each with its own
+ evidence header and base/patched sha256. Boots established, in order: (1)
+ `functools.lru_cache` does NOT make a probe trace-safe — dynamo re-traces the
+ wrapped body even when warm, so `r4d_mhc_supported -> r4d_mhc_enabled ->
+ available -> os.path.exists` breaks the graph (a plain module-global read
+ compiles; verified in-image); (2) `suppress_errors` does NOT cover graph
+ breaks, only `fullgraph=False` does; (3) the GLM trunk can never satisfy
+ `fullgraph` because its indexer metadata syncs to host
+ (`glm5next/amd/sparse_indexer.py:263,419`), and AOT compile refuses breaks
+ outright. With all three the trunk **traces** (`Dynamo bytecode transform
+ time: 1.08 s`) and then dies on `KeyError: 'in_proj_qkvbfg_a'` in the
+ mHC/KDA path — a latent bug the compile exposes. That is the remaining
+ blocker.
+ - **No new HIP kernels are possible** (the image ships `r4d.so` with no
+ sources), which binary-blocks the second-largest lever: the fused router is
+ gated `num_experts in {128, 256, 512}` and only
+ `r4d_moe_route_kernel<128|256|512>` instantiations exist, while GLM has 288
+ experts. `--kv-cache-dtype fp8` buys no capacity either (the plan's KV
+ reserve is dtype-blind), only removing gfx12's only MLA decode path.
+ - **DFlash2 for GLM is a target-side port, not a drafter port.**
+ `incoai/GLM-5.3-Flash-DFlash2` declares `architectures: ["DFlash2DraftModel"]`
+ and `model_type: "qwen3"` (5 dense layers, sliding attention, block 8,
+ `target_layer_ids [5, 14, 24, 33, 42]` of 45) and the fork **already maps it**
+ (`registry.py:636` -> `qwen3_dflash2.DFlash2Qwen3ForCausalLM`; spec method
+ `"dflash"`). What `glm5next` lacks is the target side: aux-hidden-state
+ capture (`set_aux_hidden_state_layers` / per-layer capture, as `deepseek_v2.py`
+ demonstrates). No speedup is budgeted: a verification pass pays pointwise cost
+ proportional to its token count.
- Measured and rejected: **MTP4** (the head loads, arch resolves to
`Glm5NextMTPModel`, acceptance is excellent at mean length 3.76-4.90 — but
- decode was unchanged (9.6-11.2 vs 10.0) while prefill COLLAPSED 918 -> 255,
- because a verification forward gathers the *union* of the batch's experts);
+ decode was unchanged (9.6-11.2 vs 10.0) while prefill COLLAPSED 918 -> 255.
+ The profiler window above explains it: the per-token cost is dominated by
+ pointwise glue that scales with the token count, so verifying a draft batch
+ pays ~N x that glue for the same N tokens of output);
**fp8 KV** (disables the sparse MLA backend's rope-free Triton path, and
aiter's MLA decode op is gated on CDNA>2, so the MTP draft step demands an op
that cannot exist on gfx12); **NBATCH 8192** (boots, identical prefill).