frank-primary: profile the GLM decode step and stage the compile changelist

A torch-profiler kernel-time window (496 decode tokens, rank0 trace parsed for
device-activity time) on the deployed GLM-5.3-Flash shape falsifies this page's
own "the expert gather is the bound" explanation:

  46.18 s of kernel time / 496 tokens = ~93 ms per token (actual step ~104 ms)
   34.8 %  vectorized_elementwise_kernel      708,500 launches (1,428/token)
   10.0 %  r4d_gemm_moe_w4a16_nt_b16_m64      (the MoE GEMM)
    6.7 %  elementwise_kernel_manual_unroll
    6.4 %  wvSplitK_hf_sml_                    (split-K GEMM)
    8.9 %  router: sbtopk + warpMergeSortTopK + bitonicSortKVInPlace
    6.5 %  r4d_mhc_*                           (hyper-connections)
    1.05 % r4d_moe_lru_gather_kernel           <-- the retracted suspect

The earlier arithmetic was circular (it assumed the gather consumed the whole
step). The real bound is unfused pointwise glue: the fork compiles its own Qwen
trunk (support_torch_compile in models/qwen4_exp/amd/model.py) and not its GLM
trunk, so the engine boots with "provides neither a compiled submodule nor
breakable CUDA graphs. Overriding to FULL_DECODE_ONLY." That also explains why
MTP4 measured flat — pointwise cost scales with the token count, so spec decode
amortizes nothing here — and it retires arena/hit-rate tuning (the gather is 1 %).

Staged as documented patch artifacts (not wired into any serve script; the
engine does not yet boot with them) — scripts/patches/glm53-compile-0{1,2,3}*.patch:

  -01 models/glm5next/common/model.py    decorate Glm5NextModel (R4D_GLM_COMPILE)
  -02 model_executor/kernels/r4d_lib.py  freeze 10 capability probes that do
      os.path.exists/.so lookups inside the traced forward. functools.lru_cache
      does NOT protect them: dynamo re-traces the wrapped body even when warm
      (verified in-image); a plain module-global read compiles.
  -03 compilation/wrapper.py             make fullgraph=True opt-out, because
      suppress_errors does NOT cover graph breaks and this trunk can never
      satisfy fullgraph (indexer metadata syncs to host — .item() at
      glm5next/amd/sparse_indexer.py:263,419).

With all three the trunk traces (Dynamo bytecode transform time 1.08 s) and dies
on KeyError: in_proj_qkvbfg_a in the mHC/KDA path — a latent bug the compile
exposes, and the remaining blocker.

Also records two structural constraints: no new HIP kernels are possible (r4d.so
ships without sources), which binary-blocks the fused router at GLM's 288
experts; and DFlash2 for GLM is a TARGET-side port, not a drafter port
(incoai/GLM-5.3-Flash-DFlash2 is architectures ["DFlash2DraftModel"] +
model_type qwen3, already mapped at registry.py:636; glm5next lacks only the
aux-hidden-state plumbing deepseek_v2.py demonstrates).

Runbook frontmatter, intro, mermaid and wall section, AGENTS.md and the work log
corrected to match the measurement. tests/run.sh: 362 seams pass.

AGENTS.md

@@ -24,7 +24,7 @@ merge absorbed the former `vllmScripts` repo, history preserved).
 
 | Target | Service | Runtime | Model | Port |
 |--------|---------|---------|-------|------|
-| Dankserver | `frank-primary` | tcclaviger RDNA4 vLLM fork (`docker.io/tcclaviger/vllm@sha256:da4029666…`, digest-pinned, plain `podman run`, root systemd unit) | `Intel/GLM-5.3-Flash-W4A16-AutoRound` — AutoRound symmetric int4 g128 (170 GiB, 34 shards, 113,074 tensors), TP=2, expert offload to 73.64 GiB/rank of pinned host rows behind a 14.51 GiB / 2,383-slot VRAM LRU arena, 131072 ctx × 2 seqs, bf16 KV, no MTP, no prefetch; measured 9.9-10.0 tok/s decode / 918 tok/s prefill | 7770 |
+| Dankserver | `frank-primary` | tcclaviger RDNA4 vLLM fork (`docker.io/tcclaviger/vllm@sha256:da4029666…`, digest-pinned, plain `podman run`, root systemd unit) | `Intel/GLM-5.3-Flash-W4A16-AutoRound` — AutoRound symmetric int4 g128 (170 GiB, 34 shards, 113,074 tensors), TP=2, expert offload to 73.64 GiB/rank of pinned host rows behind a 14.51 GiB / 2,383-slot VRAM LRU arena, 131072 ctx × 2 seqs, bf16 KV, no MTP, no prefetch; measured 9.9-10.0 tok/s decode / 918 tok/s prefill, bounded by unfused pointwise glue (the expert gather is 1.05 % of device time; the trunk is uncompiled) | 7770 |
 | Dankclaw | `frank-large` | gufo `gufo-org/gufo` v0.1.1, the Strix Halo engine, BUILT FROM SOURCE (root systemd unit; links the host ROCm 7.2.4) | Swift-1.5-Qwen3.8-27B-Uncensored UD-Q8_K_XL (27B dense, ABLITERATED/uncensored, 29.3 GiB, SWIFT OPEN LICENSE 1.0) with the z-lab DFlash2 Q4_K_M drafter (adaptive block policy, cap 7) on the gfx1151 iGPU (Strix Halo LPDDR, 114 GiB GTT), 262144 ctx single session, greedy, gufo's compiled-in Qwen renderer (NO Qwen-Sharp), text only | 7770 |
 | Dankclaw | `frank-small` | mainline llama.cpp master 37b53fd (root systemd unit) | HauhauCS Gemma4-12B-QAT-Uncensored-Balanced Q4_K_M (BF16 mmproj + MTP assistant sidecar, `draft-mtp,ngram-mod`) on the RTX 3090, 1048576 pool / 4 slots of 262,144, q8_0 KV, the fleet's fastest vision endpoint | 7771 |
 
@@ -445,14 +445,51 @@ serve. Built shape: 14.51 GiB arena = **2,383 slots × 6.234 MiB per rank** over
 12,096 host-backed experts = **73.64 GiB pinned host rows per rank**.
 
 **Measured 2026-09-30: decode 9.9–10.0 tok/s, prefill 918 tok/s** (warm, 2.6k
-prompt), KV cache 138,561 tokens. **The 35 tok/s target is NOT met, and this is
-a measured wall rather than a tuning gap:** each decode step must move 2.045 GiB
-of routed experts per rank (42 layers × 8 experts × 6.234 MiB), live telemetry
-(`R4D_LRU_TELEMETRY=1`) puts the expert-cache hit rate at **0.53–0.66**, and the
-fork's gather read path delivers **~8.2 GB/s against a link measured at
-22.74 GB/s** — the same structural gap the qwen campaign found. Growing the
-arena 13.08 → 14.51 GiB changed decode by *nothing* (10.03 → 9.88 tok/s), so
-residency stopped being the binding term; 0.89 hit rate would be needed for 35.
+prompt), KV cache 138,561 tokens. **The 35 tok/s target is NOT met. The first
+explanation for that — "the expert gather is the bound at ~8.2 GB/s" — was
+FALSIFIED by a torch-profiler kernel-time window** (496 decode tokens, rank0
+trace parsed for device-activity time): 46.18 s of kernel time = ~93 ms per
+token, of which **44 % is pointwise glue** — `vectorized_elementwise_kernel`
+alone is 16.07 s over **708,500 launches (1,428 per token)**, plus 3.11 s of
+`elementwise_kernel_manual_unroll` — against **1.05 % for
+`r4d_moe_lru_gather_kernel`**. The MoE GEMM is 10 %, the split-K GEMM 6 %, the
+router/top-k 9 %, the mHC kernels 6.5 %. The real wall is that **the fork
+compiles its own Qwen trunk (`@support_torch_compile` in
+`models/qwen4_exp/amd/model.py`) and not its GLM trunk**, so every pointwise op
+is a separate eager kernel. Two consequences: spec decode **cannot** help this
+shape (pointwise cost scales with the token count, which is exactly why MTP4
+measured flat), and neither arena size nor hit rate is a lever (the gather is
+1 %).
+
+**Fork work is staged, not deployed.** `scripts/patches/glm53-compile-0{1,2,3}*.patch`
+are the compile-enablement changelist (decorate the trunk; freeze the ten libr4d
+capability probes that do `os.path.exists`/`.so` lookups *inside* the traced
+forward; make `fullgraph=True` opt-out). Boots established that
+`functools.lru_cache` does **not** protect those probes (dynamo re-traces the
+wrapped body even warm), that `suppress_errors` does **not** cover graph breaks
+(only `fullgraph=False` does), and that the GLM trunk can never satisfy
+`fullgraph` (its indexer metadata syncs to host at
+`glm5next/amd/sparse_indexer.py:263,419`). With all three the trunk **traces**
+(`Dynamo bytecode transform time: 1.08 s`) and then dies on
+`KeyError: 'in_proj_qkvbfg_a'` in the mHC/KDA path — a latent bug the compile
+exposes, and the remaining blocker.
+
+**No new HIP kernels are possible** — the image ships `r4d.so` as a binary with
+no sources. The 9 % router is therefore **binary-blocked**: its fused path is
+gated `num_experts in {128, 256, 512}` (`router/r4d_route_router.py`) and only
+`r4d_moe_route_kernel<128|256|512>` instantiations exist, while GLM has **288**
+experts. Only Python/config/inductor changes (or runtime-JIT Triton) are
+available on this fork.
+
+**DFlash2 for GLM is a target-side port, not a drafter port.**
+`incoai/GLM-5.3-Flash-DFlash2` declares `architectures: ["DFlash2DraftModel"]`
+and `model_type: "qwen3"` (5 dense layers, sliding attention, block 8,
+`target_layer_ids [5, 14, 24, 33, 42]` of 45), and the fork **already maps it**
+(`registry.py:636` → `qwen3_dflash2.DFlash2Qwen3ForCausalLM`; spec method
+`"dflash"`). What is missing is the target side: `glm5next` implements neither
+`set_aux_hidden_state_layers()` nor the per-layer aux capture that
+`deepseek_v2.py` (also MLA+MoE) demonstrates. Budget no speedup: per the profile
+above a verification pass pays pointwise cost proportional to its token count.
 
 **MTP works here and is still rejected.** The head loads
 (`Resolved architecture: Glm5NextMTPModel`) and accepts well (mean accepted

scripts/patches/glm53-compile-01-model-decorator.patch

@@ -0,0 +1,67 @@
+# glm53-compile-01-model-decorator.patch — put the GLM-5.3-Flash trunk under torch.compile.
+#
+# STATUS: WIP — first of three patches in the "compile the GLM trunk" changelist. The
+# set does NOT yet boot the engine (see 03 for the next blocker), so it is NOT part of
+# the deployed frank-primary shape. Nothing in scripts/ mounts these files.
+#
+# WHY THIS IS THE RIGHT LEVER (measured 2026-09-30, not inferred):
+#
+# The fork's own flagship trunk — vllm/models/qwen4_exp/amd/model.py — carries
+# @support_torch_compile. vllm/models/glm5next/common/model.py does not, and the engine
+# says so on every boot:
+#
+#   Cudagraph mode FULL_AND_PIECEWISE requires piecewise capture, but the loaded model
+#   provides neither a compiled submodule nor breakable CUDA graphs.
+#   Overriding to FULL_DECODE_ONLY.
+#
+# A torch-profiler window on the live endpoint (496 decode tokens, rank0, 3 x 160-token
+# generations at the deployed shape) shows what the missing compilation costs. Total
+# device kernel time 46.18 s / 496 tokens = ~93 ms per token, against a measured step
+# time of ~104 ms (9.6 tok/s with the profiler attached):
+#
+#   16.07 s  34.8 %  708,500 launches  vectorized_elementwise_kernel      (22.7 us each)
+#    4.60 s  10.0 %    42,000           r4d_gemm_moe_w4a16_nt_b16_m64     (the MoE GEMM)
+#    3.11 s   6.7 %   174,708           elementwise_kernel_manual_unroll
+#    2.94 s   6.4 %   138,252           wvSplitK_hf_sml_                  (split-K GEMM)
+#    4.09 s   8.9 %    83,160           topk/router (sbtopk gatherTopK + warpMergeSortTopK
+#                                       + bitonicSortKVInPlace)
+#    3.01 s   6.5 %   133,920           r4d_mhc_* (hyper-connection pre/post/finish)
+#    2.78 s   6.0 %   127,916           inductor triton_poi_* / triton_per_* (the only
+#                                       fusions that DO happen)
+#    0.49 s   1.05%    20,664           r4d_moe_lru_gather_kernel         <-- the suspect
+#
+# That last row retired the fleet's working hypothesis. Decode on this shape is NOT
+# expert-gather bound: the gather is 1 % of device time, the same order as the telemetry
+# kernel that was switched on to measure it. 44 % of the budget is pointwise glue —
+# bf16<->fp32 conversions, adds, pows/rsqrts from fp32 RMSNorm, clamp/scale quant
+# passes — spread over ~1,428 launches per token. Inductor fusion is the only lever
+# that touches that mass; the router is second at 9 % but is binary-blocked (the
+# libr4d route kernel exists only as r4d_moe_route_kernel<128|256|512> instantiations
+# and GLM has 288 experts, with no HIP sources shipped to rebuild).
+#
+# TARGET: vllm/models/glm5next/common/model.py   (inside the pinned image)
+#   base    sha256 008eeaa2a07e091465319f86900b5a321d921238723ccbe22e6b9e971e1d2b28
+#   patched sha256 6e9cb81b84a9a88bd41403ca404aa04c3b791899dd77d661c3f25ae686cecd40
+#
+# APPLY: `patch -p1` from the vllm package root, or bind-mount new/.../model.py over
+# site-packages. Then R4D_GLM_COMPILE=1 (default) enables it; =0 makes the decorator a
+# no-op, so the file can stay mounted for an A/B without a second artifact.
+# Requires 02 (trace-safe probes) — without it the worker dies during profile_run on
+# the first probe call inside the traced region.
+--- a/vllm/models/glm5next/common/model.py
++++ b/vllm/models/glm5next/common/model.py
+@@ Glm5NextModel: add @support_torch_compile (env): R4D_GLM_COMPILE=0 disables @@
++_GLM_COMPILE = os.environ.get("R4D_GLM_COMPILE", "1") == "1"
++
++
++@support_torch_compile(
++    dynamic_arg_dims={
++        "input_ids": 0,
++        "positions": -1,
++        "intermediate_tensors": 0,
++        "inputs_embeds": 0,
++    },
++    enable_if=lambda _cfg: _GLM_COMPILE,
++)
+-class Glm5NextModel(nn.Module):
++class Glm5NextModel(nn.Module):   # decorated above

scripts/patches/glm53-compile-02-trace-safe-probes.patch

@@ -0,0 +1,114 @@
+# glm53-compile-02-trace-safe-probes.patch — libr4d capability probes must not do I/O
+# inside a torch.compile region.
+#
+# STATUS: WIP — second of three patches in the "compile the GLM trunk" changelist.
+# Inert (but harmless) without 01. Not part of the deployed shape.
+#
+# THE BREAK (measured: with 01 mounted and R4D_GLM_COMPILE=1, the worker dies during
+# profile_run, i.e. the first — cold — forward):
+#
+#   Explanation: Dynamo does not know how to trace the builtin `posix.stat`
+#   ... r4d_mhc_supported -> r4d_lib.r4d_mhc_enabled -> available -> os.path.exists
+#
+# THE NON-OBVIOUS PART: functools.lru_cache does NOT protect these probes. Dynamo
+# re-traces the wrapped body to build its guard even when the cache is warm — it warns
+# about exactly this ("Dynamo detected a call to a `functools.lru_cache`-wrapped
+# function ... Dynamo ignores the cache wrapper and directly traces the wrapped
+# function"). Verified in-image, torch 2.11.0+rocm10.0, torch.compile(backend="eager"):
+#
+#   cold lru_cache call       -> RAISE Unsupported
+#   WARM lru_cache call       -> RAISE Unsupported     <-- warming is not enough
+#   plain module-global read  -> compiles
+#   uncached os.path.exists   -> RAISE Unsupported
+#
+# So each probe must resolve ONCE into a plain value and be read as a global. The set
+# below is every no-argument probe / symbol lookup reachable from a forward pass,
+# grep-verified call sites:
+#   model_executor/layers/fused_moe/experts/r4d_w4a16_moe.py:79  available()
+#   model_executor/layers/fused_moe/experts/r4d_w4a16_moe.py:81  has_w4a16_moe()
+#   model_executor/kernels/mhc/r4d_dsv4.py:16                    r4d_mhc_enabled()
+#   v1/attention/ops/rocm_aiter_mla_sparse.py:694                r4d_indexer_enabled(), mqa_max_heads()
+#   v1/attention/ops/rocm_aiter_mla_sparse.py:1124               r4d_indexer_enabled()
+#   model_executor/layers/fused_moe/experts/r4d_mxfp4_moe.py:683 r4d_act_quant_enabled()
+#   model_executor/layers/fused_moe/experts/r4d_w4a16_moe.py _entry()  moe_w4a16()/moe_w4a16_mt1() per MoE call
+# (lib() is the transitive dependency of all of them.)
+#
+# TARGET: vllm/model_executor/kernels/r4d_lib.py   (inside the pinned image)
+#   base    sha256 b8dfeefb4acc72b67b0579a4ab0653ef5d48c40cc8649187d757c80ce6b9617a
+#   patched sha256 0751852e1ca813ad2bdb1da7bf10c605130e63e04ef9ec57e70b5f675a29c08e
+#
+# APPLY: append the block below to the module. R4D_FREEZE_PROBES=0 restores stock
+# behaviour. Cost when nothing is compiled: ten one-time probes at import (measured:
+# logged once per process, e.g. "libr4d capability probes frozen: 10 warm [...]").
+#
+# OBSERVED EFFECT: with 01+02 the worker gets past this break and reaches the next one
+# (see 03) — dynamo reports "Dynamo bytecode transform time: 1.08 s" for the trunk.
+--- a/vllm/model_executor/kernels/r4d_lib.py
++++ b/vllm/model_executor/kernels/r4d_lib.py
+@@ append: trace-safe capability probes (env): R4D_FREEZE_PROBES=0 disables @@
++# FORK-LOCAL (patch: trace-safe capability probes).  torch.compile traces the
++# BODY of a functools.lru_cache call even when the cache is warm, so a capability
++# probe called from a forward pass is a dynamo graph break.  Measured 2026-09-30
++# on the GLM-5.3-Flash trunk under @support_torch_compile:
++#   Dynamo does not know how to trace the builtin `posix.stat`
++#   r4d_mhc_supported -> r4d_mhc_enabled -> available -> os.path.exists
++# Each probe below is wrapped to resolve ONCE into a memo and is warmed at import
++# (outside any tracing region), so a traced call is a dict read.  The probe set is
++# every no-argument capability probe / symbol lookup reachable from a forward
++# pass: the W4A16 + mxfp4 MoE dispatch, the mHC op, the MLA sparse indexer gate
++# and the draft/lm-head guards.
++_PROBE_NAMES = (
++    "available",
++    "lib",
++    "has_w4a16_moe",
++    "moe_w4a16",
++    "moe_w4a16_mt1",
++    "has_mhc_dsv4",
++    "r4d_mhc_enabled",
++    "r4d_indexer_enabled",
++    "mqa_max_heads",
++    "r4d_act_quant_enabled",
++)
++
++
++def _freeze_probe(fn):
++    """Return fn bound to its first result.  Result must be a plain value."""
++
++    memo = []
++
++    def probe(*args, **kwargs):
++        if not memo:
++            memo.append(fn(*args, **kwargs))
++        return memo[0]
++
++    return probe
++
++
++def _freeze_probes() -> list[str]:
++    frozen = []
++    for name in _PROBE_NAMES:
++        fn = globals().get(name)
++        if fn is None:
++            continue
++        # Only the fast branch of a bound probe is traced; an empty memo means the
++        # first call still does I/O, so populate it here as well.
++        globals()[name] = _freeze_probe(fn)
++        frozen.append(name)
++    return frozen
++
++
++from vllm.logger import init_logger as _init_logger
++
++_probe_log = _init_logger(__name__)
++
++if os.environ.get("R4D_FREEZE_PROBES", "1") == "1":
++    _warmed, _failed = [], []
++    for _n in _freeze_probes():
++        try:
++            globals()[_n]()
++            _warmed.append(_n)
++        except Exception:  # noqa: BLE001 -- probe unavailable here; recompute lazily
++            _failed.append(_n)
++    _probe_log.info("libr4d capability probes frozen: %d warm %s%s",
++                    len(_warmed), _warmed,
++                    f" (deferred: {_failed})" if _failed else "")

scripts/patches/glm53-compile-03-fullgraph-optout.patch

@@ -0,0 +1,64 @@
+# glm53-compile-03-fullgraph-optout.patch — let torch.compile split the graph instead
+# of failing on a break.
+#
+# STATUS: WIP — third of three patches in the "compile the GLM trunk" changelist.
+# With 01+02+03 the model TRACES, then hits the NEXT blocker (below). The set does not
+# yet boot the engine; nothing in scripts/ mounts it.
+#
+# THE BREAK: with 01+02 mounted the worker dies with
+#
+#   RuntimeError: Graph breaks are not supported with aot compile.
+#   Please use torch.compile(fullgraph=True).
+#
+# because vllm/compilation/wrapper.py calls torch.compile(..., fullgraph=True, ...) and
+# this build has AOT compile on by default (VLLM_USE_AOT_COMPILE). The GLM trunk cannot
+# satisfy fullgraph: its indexer metadata path syncs to the host —
+#   models/glm5next/amd/sparse_indexer.py:263  int(positions[...].max().item()) + 1
+#   models/glm5next/amd/sparse_indexer.py:419  lmax = int(decode_metadata.decode_lens.max().item())
+# — and a .item() inside a captured graph is not expressible.
+#
+# WHAT WAS VERIFIED (in-image, torch 2.11.0+rocm10.0, torch.compile(backend="eager")):
+#
+#   fullgraph=True   suppress_errors=False  -> RAISE Unsupported
+#   fullgraph=True   suppress_errors=True   -> RAISE Unsupported   <-- suppress_errors
+#                                                                     does NOT cover breaks
+#   fullgraph=False  suppress_errors=False  -> ok   (traceable segments compiled by
+#                                                    inductor, the rest runs eagerly;
+#                                                    numerics unchanged)
+#
+# So breaks must be TOLERATED, not suppressed: the traceable mass still fuses. This
+# patch makes the fullgraph requirement opt-out via VLLM_COMPILE_FULLGRAPH=0 (default 1,
+# so every other model keeps stock behaviour), and the run also needs
+# VLLM_USE_AOT_COMPILE=0 because AOT compile refuses breaks outright.
+#
+# TARGET: vllm/compilation/wrapper.py   (inside the pinned image)
+#   base    sha256 969961dca41c78385212e985cfc0c57ef067f4a2d7490c73c658c600a1b95cac
+#   patched sha256 732363bb1a1367772f13c60a655c678e7e2e9e56186677b14238f5e84cac31b3
+#   (the file already imports os; the patch does not add a second import)
+#
+# NEXT BLOCKER — NOT FIXED HERE. With 01+02+03 and
+# R4D_GLM_COMPILE=1 VLLM_COMPILE_FULLGRAPH=0 VLLM_USE_AOT_COMPILE=0 the trunk traces
+# ("Dynamo bytecode transform time: 1.08 s") and then the worker dies during profile_run:
+#
+#   KeyError: 'in_proj_qkvbfg_a'
+#     vllm/models/glm5next/common/model.py:456  in Glm5NextDecoderLayer.forward
+#
+# `in_proj_qkvbfg_a` is the KDA layer's merged column-parallel projection
+# (models/glm5next/common/kda.py:217, called at :407). The key is looked up in a
+# dict/mapping that the EAGER path never reaches, so compiling the trunk exposes a
+# latent bug in the mHC/KDA wiring rather than a dynamo limitation. Fix that and
+# whatever follows before this changelist can boot — then re-measure: the prize is the
+# 44 % pointwise mass (see 01), and the frozen-probe log line is the signal that 02 is
+# live.
+--- a/vllm/compilation/wrapper.py
++++ b/vllm/compilation/wrapper.py
+@@ fullgraph opt-out (env VLLM_COMPILE_FULLGRAPH=0) @@
++# FORK-LOCAL (patch: fullgraph opt-out).  torch.compile(fullgraph=True) turns any
++# dynamo graph break into a hard failure; suppress_errors does NOT cover breaks
++# (verified in-image, torch 2.11+rocm10.0).  With VLLM_COMPILE_FULLGRAPH=0 the
++# traceable segments are still compiled by inductor and the untraceable ones run
++# eagerly, which is what a model with device-sync metadata (the GLM-5.3-Flash
++# indexer) needs.  Default 1 keeps stock behaviour for every other model.
++_FULLGRAPH = os.environ.get("VLLM_COMPILE_FULLGRAPH", "1") == "1"
+-                fullgraph=True,
++                fullgraph=_FULLGRAPH,

serverAppConfigs-docs/LLM/frank-primary-glm53-flash-w4a16.md

@@ -1,7 +1,7 @@
 ---
 type: runbook
 title: frank-primary — GLM-5.3-Flash W4A16 int4 (tcclaviger, 2x R9700)
-description: VERIFIED 2026-09-30 by boot probe, bench and live expert-cache telemetry. frank-primary cut over from Qwen3.8-Flash-Next-Uncensored-FP8 + MTP4 to Intel/GLM-5.3-Flash-W4A16-AutoRound, the first payload this fleet serves through the fork's W4A16 expert-offload path (INC scheme -> R4dW4a16MoEMethod, libr4d grouped W4A16 GEMM). Measured decode 9.9-10.0 tok/s and prefill 918 tok/s at 131,072 ctx. THE 35 tok/s TARGET IS NOT MET and this page shows why it is out of reach: the step is bounded by the fork's expert-gather read path at ~8.2 GB/s effective against a PCIe link that delivers 22.74 GB/s, with the cache at its measured hit-rate ceiling (0.53-0.66 per layer) from an arena that cannot grow past ~15 GiB under a 73.64 GiB/rank expert set.
+description: VERIFIED 2026-09-30 by boot probe, bench, live expert-cache telemetry AND a torch-profiler kernel-time window. frank-primary cut over from Qwen3.8-Flash-Next-Uncensored-FP8 + MTP4 to Intel/GLM-5.3-Flash-W4A16-AutoRound, the first payload this fleet serves through the fork's W4A16 expert-offload path (INC scheme -> R4dW4a16MoEMethod, libr4d grouped W4A16 GEMM). Measured decode 9.9-10.0 tok/s and prefill 918 tok/s at 131,072 ctx. THE 35 tok/s TARGET IS NOT MET, and the profiler window FALSIFIED the first explanation: the expert gather is 1.05 % of device time, not the bound. 44 % of the 93 ms/token budget is pointwise glue (1,428 elementwise launches per token) because the fork's GLM trunk is NOT torch.compiled while its own Qwen trunk is; the second-largest lever, the fused router, is binary-blocked at 288 experts. The compile changelist is staged but incomplete (three patches, one blocker left).
 tags:
   - dankserver
   - r9700
@@ -22,12 +22,16 @@ timestamp: 2026-09-30
 > `GLM-5.3-Flash` at int4 on two R9700s with expert offload, and how fast is it?
 > **Answer.** Yes — it serves, with vision, tool calls and 131,072 ctx. It runs
 > at **9.9-10.0 tok/s decode / 918 tok/s prefill**, and **35 tok/s is not
-> reachable** with this payload on this hardware. The bound is not the model,
-> the quant, or the link: it is the fork's expert-gather read path, and it is
-> measured, not inferred.
+> reachable** on this payload as the fork ships it. The bound is **not** the
+> model, the quant, the link, or the expert gather (measured: 1.05 % of device
+> time). It is the **unfused pointwise glue of an uncompiled trunk** — 44 % of a
+> 93 ms per-token budget spread over ~1,428 elementwise launches per token —
+> because the fork compiles its own Qwen trunk and not its GLM one. See
+> [The measured profile](#the-measured-profile-and-the-real-wall).
 
-Related: [[dankserver-qwen38-fp8-expert-offload-tuning]] (the same fork, the
-same hardware, the same gather-path wall, found first on qwen),
+Related: [[dankserver-qwen38-fp8-expert-offload-tuning]] (the same fork, the same
+hardware — its conclusion was drawn from throughput arithmetic and is superseded
+by this page's kernel-time measurement),
 [[dankserver-frontier-moe-feasibility]] (the earlier fit analysis),
 [[VLLM Index]], [[LLM/index|LLM fleet]].
 
@@ -51,7 +55,7 @@ graph LR
     D --> E[ExpertLRUCache<br/>2,383 VRAM slots / rank]
     D --> F[73.64 GiB pinned host rows / rank]
     E --> G[decode step]
-    F -->|8.2 GB/s effective| G
+    F -->|host misses<br/>1.05% of device time| G
 ```
 
 ## Why this payload and not another
@@ -90,33 +94,142 @@ Measured on the deployed engine, 2026-09-30 (`bench_glm.py`, real prose, greedy)
 Decode is flat across context and across every arena change — the signature of a
 per-step cost that is not KV- or residency-driven at the margin.
 
-### The arithmetic that bounds it
+### The measured profile, and the real wall
 
-Per decode step, per rank: 42 MoE layers x 8 routed experts x 6.234 MiB =
-**2.045 GiB of expert weights**, which must either be in VRAM or cross PCIe.
+**This section replaces an earlier explanation, which the measurement falsified.**
+The first reading of this page bounded the step with throughput arithmetic: 42 MoE
+layers x 8 experts x 6.234 MiB = 2.045 GiB per step per rank, 0.82 GiB of it
+missed at a measured 0.53-0.66 hit rate, giving "0.82 GiB in ~100 ms = 8.2 GB/s =
+the gather is the wall". That arithmetic was circular — it *assumed* the gather
+consumed the whole step — and it is wrong. The engine, asked directly, disagrees:
 
-| hit rate | miss traffic | step @ 8.2 GB/s | decode |
+A torch-profiler window (`--profiler-config`, `POST /start_profile` /
+`stop_profile`, 496 decode tokens at the deployed shape, rank0 trace parsed for
+device-activity time) gives the true split. Total device kernel time **46.18 s /
+496 tokens = ~93 ms per token** against a measured ~104 ms step (9.6 tok/s with
+the profiler attached):
+
+| ms | % | launches | kernel |
 | --- | --- | --- | --- |
-| 0.60 (measured) | 0.82 GiB | ~105 ms | **9.5-10.0 tok/s** |
-| 0.70 | 0.61 GiB | ~78 ms | 12.8 tok/s |
-| 0.89 (what 35 tok/s needs) | 0.23 GiB | ~29 ms | 35 tok/s |
+| 16,074 | **34.8** | 708,500 | `vectorized_elementwise_kernel` (22.7 us each) |
+| 4,602 | 10.0 | 42,000 | `r4d_gemm_moe_w4a16_nt_b16_m64` — the MoE GEMM |
+| 3,108 | 6.7 | 174,708 | `elementwise_kernel_manual_unroll` |
+| 2,943 | 6.4 | 138,252 | `wvSplitK_hf_sml_` — split-K GEMM (projections / head) |
+| 2,829 | 6.1 | 83,160 | router: `sbtopk::gatherTopK` + `warpMergeSortTopK` + `bitonicSortKVInPlace` (1,208 + 1,051 + 559 ms; the 1,051 row is not in the list below) |
+| 3,015 | 6.5 | 133,920 | `r4d_mhc_pre_finish/pre_partial/post` — hyper-connections |
+| 2,782 | 6.0 | 127,916 | inductor `triton_poi_*` / `triton_per_*` (the fusions that DO happen) |
+| **487** | **1.05** | 20,664 | **`r4d_moe_lru_gather_kernel`** — the accused |
+| 250 | 0.54 | 20,832 | `_moe_lru_tele_kernel` — the telemetry that measured it (TELEM was on) |
 
-Two independent measurements pin the missed term:
+So the gather is **1.05 %** of device time, and `r4d_moe_lru_fused_kernel` (the
+LRU *manager*, not the copy) is a further 1,599 ms / 3.5 %. Residency work and the
+PCIe link are not the story: 44 % of the budget is **pointwise glue** — bf16<->fp32
+conversions, adds, `pow`/`rsqrt` from fp32 RMSNorm, `launch_clamp_scale` quant
+passes, `reduce_kernel` means — spread over **~1,428 elementwise launches per
+token** (3,734 kernel launches per token in total). The GPU is busy 86 % of the
+wall clock, so this is real device work, not launch-gap overhead.
 
-- **Live expert-cache telemetry** (`R4D_LRU_TELEMETRY=1`, profiler window)
-  gives the hit rate directly — the LRU is *working*:
-  `layers.34: slots 56 (pinned 0, tier 0) hit 0.6279 routed 2002 installs 543
-  read-through 202` … `layers.44: hit 0.5335`. Per-layer range **0.53-0.66**.
-- **The link delivers 22.74 GB/s** at the engine's own slab size (H2D probe,
-  pinned, 2.345 MiB). The step moves 0.82 GiB in ~100 ms = **8.2 GB/s —
-  36% of the wire**, matching the fork's own measured 6.5-9.5 GB/s.
+Two consequences that kill three whole families of planned work:
 
-So the step is bounded by the fork's gather read path, and the only other lever
-is the hit rate — which is capped by the arena. A 14.51 GiB arena holds 20% of a
-73.64 GiB expert set; the measured hit rate beats random by ~3x thanks to
-routing locality, but it saturates: growing the arena 13.08 → 14.51 GiB
-(+9%) changed decode by **nothing** (10.03 → 9.88 tok/s). Reaching 0.89 would
-need roughly 4.5x the per-layer slot count — an arena of well over 100 GiB.
+- **Spec decode cannot help this shape, and the MTP measurement was not an
+  accident.** Pointwise cost scales with the token count, so a draft batch that
+  verifies 4 tokens pays ~4x the glue for the same per-token output — which is
+  exactly why MTP4 measured "acceptance 3.76-4.90, decode unchanged". Same reason
+  the gather was never the bound.
+- **Arena size and hit rate are not levers.** Confirmed twice: `--expert-precise-memory`
+  (arena 13.08 -> 14.51 GiB) moved decode by nothing, and the gather is 1 %.
+
+## Fork work: the compile changelist (WIP, does not boot yet)
+
+The fork compiles its own flagship trunk — `vllm/models/qwen4_exp/amd/model.py`
+carries `@support_torch_compile` — and **not** its GLM trunk. The engine says so on
+every boot:
+
+```
+Cudagraph mode FULL_AND_PIECEWISE requires piecewise capture, but the loaded model
+provides neither a compiled submodule nor breakable CUDA graphs.
+Overriding to FULL_DECODE_ONLY.
+```
+
+Inductor fusion is therefore the only lever that touches the 44 % pointwise mass.
+Three patches are staged in `scripts/patches/glm53-compile-0*.patch`, each carrying
+its own evidence header, base/patched sha256 and the next blocker:
+
+| patch | target | fixes |
+| --- | --- | --- |
+| `-01-model-decorator` | `models/glm5next/common/model.py` | decorates `Glm5NextModel` (env `R4D_GLM_COMPILE`, default on) |
+| `-02-trace-safe-probes` | `model_executor/kernels/r4d_lib.py` | freezes 10 capability probes that do `os.path.exists`/`.so` lookups **inside the traced forward** |
+| `-03-fullgraph-optout` | `compilation/wrapper.py` | makes `fullgraph=True` opt-out (env `VLLM_COMPILE_FULLGRAPH=0`) |
+
+What the boots established, in order (each one ~7 min):
+
+1. **01 alone dies** on the first cold probe call inside the traced region:
+   `Dynamo does not know how to trace the builtin 'posix.stat'` —
+   `r4d_mhc_supported -> r4d_lib.r4d_mhc_enabled -> available -> os.path.exists`.
+   **`functools.lru_cache` does not protect these probes**: dynamo re-traces the
+   wrapped body to build its guard even when the cache is warm (it warns about
+   exactly this), verified in-image — a *warm* lru_cache-calling function still
+   fails `fullgraph=True`, while a plain module-global read compiles.
+2. **01+02 get past it** (`libr4d capability probes frozen: 10 warm [...]` in all
+   four processes) and die on
+   `RuntimeError: Graph breaks are not supported with aot compile`, because this
+   build runs AOT compile by default and AOT refuses breaks.
+3. **01+02+03 with `VLLM_COMPILE_FULLGRAPH=0 VLLM_USE_AOT_COMPILE=0` traces the
+   trunk successfully** (`Dynamo bytecode transform time: 1.08 s`) and then dies in
+   `profile_run` with `KeyError: 'in_proj_qkvbfg_a'` at
+   `glm5next/common/model.py:456` (the KDA layer's merged projection,
+   `kda.py:217`) — a dict/registry lookup keyed by attribute name that the eager
+   path never reaches, i.e. **a latent bug the compile exposes, not a dynamo
+   limit**.
+
+That last row is the state of the work: the remaining blocker is a real bug in the
+fork's mHC/KDA wiring, and it must be fixed (plus whatever follows it) before the
+changelist can boot. Two verified facts make the payoff concrete: `suppress_errors`
+does **not** cover graph breaks (only `fullgraph=False` does), and the GLM trunk can
+never satisfy `fullgraph` because its indexer metadata syncs to host
+(`glm5next/amd/sparse_indexer.py:263,419`: `int(...max().item())`).
+
+### Two structural constraints on any further fork work
+
+- **No new HIP kernels are possible.** The image ships `r4d.so` as a binary with no
+  sources. The obvious second lever — the 9 % router — is gated
+  `num_experts in {128, 256, 512}` in `router/r4d_route_router.py` and the binary
+  contains only `r4d_moe_route_kernel<128|256|512>` instantiations. GLM has **288**
+  experts, so it falls back to the generic sort-based top-k and **cannot be moved to
+  the fused router without rebuilding libr4d**. Only Python/config/inductor changes
+  (and runtime-JIT Triton) are available.
+- **`--kv-cache-dtype fp8` is a dead end for capacity too.** The plan's KV reserve
+  is dtype-blind (1.76 GiB at 131,072 either way), so fp8 buys no arena and no
+  window; it only removes the only MLA decode path gfx12 has (trap 1 above).
+
+### DFlash2 for GLM — what the drafter actually is
+
+`incoai/GLM-5.3-Flash-DFlash2` is **not** a GLM-shaped drafter and needs no MLA/DSA
+port: its `config.json` declares `architectures: ["DFlash2DraftModel"]` and
+`model_type: "qwen3"` — 5 dense layers, hidden 4096, 32 heads / 8 kv heads,
+head_dim 128, sliding attention (window 2048), `is_causal: false`,
+`use_cache: false`, `dflash_config` = block 8, conv group 16, selector rank 256 /
+top-k 16, and **`target_layer_ids: [5, 14, 24, 33, 42]` against
+`num_target_layers: 45`**.
+
+The fork already maps that architecture:
+`registry.py:636 "DFlash2DraftModel": ("qwen3_dflash2", "DFlash2Qwen3ForCausalLM")`,
+and the spec method is `"dflash"` (`config/speculative.py:67`). So the port is
+**target-side only**: the GLM trunk must expose auxiliary hidden states at those five
+layer indices, the same interface `deepseek_v2.py` (also MLA+MoE) implements —
+`set_aux_hidden_state_layers()` / `get_eagle3_aux_hidden_state_layers()` plus
+capture at `idx in self.aux_hidden_state_layers` in the layer loop. `glm5next`
+implements neither.
+
+Expected benefit is **not** assumed: DFlash2 amortizes a draft block (8 tokens) over
+one verification pass, and per the profile above a verification pass costs pointwise
+work proportional to tokens — the same reason MTP4 was flat. Measure it; do not
+budget on it.
+
+Note also that the KV-cap patch's predecessor arithmetic in
+[[dankserver-qwen38-fp8-expert-offload-tuning]] should be read the same way: its
+"gather-bound" conclusion came from the same throughput arithmetic that this page's
+profiler window has now falsified.
 
 ## Measured arms (all 2026-09-30, one boot each)
 

serverAppConfigs-docs/log.md

@@ -20,17 +20,58 @@
   - Measured on the deployed engine: **decode 9.9-10.0 tok/s, prefill 918 tok/s**
     (warm, 2.6k prompt), KV 138,561 tokens; arena 14.51 GiB = 2,383 slots per
     rank over 12,096 host-backed experts (73.64 GiB pinned per rank).
-  - **The wall:** live expert-cache telemetry (`R4D_LRU_TELEMETRY=1` + a
-    `with_stack=false` profiler window) puts the cache hit rate at 0.53-0.66 per
-    layer; a step moves 0.82 GiB of misses in ~100 ms = **8.2 GB/s effective
-    against a link measured at 22.74 GB/s** — the same structural gather-read-path
-    gap the 2026-09-29 qwen campaign found. Growing the arena 13.08 -> 14.51 GiB
-    (+9%) changed decode by *nothing* (10.03 -> 9.88 tok/s), so residency stopped
-    being the binding term; 35 tok/s would need a 0.89 hit rate.
+  - **The wall — and a corrected one.** The first account of the shortfall
+    bounded the step with throughput arithmetic (a step moves 0.82 GiB of misses
+    in ~100 ms = "8.2 GB/s effective against a 22.74 GB/s link", needing a 0.89
+    hit rate for 35 tok/s). **A torch-profiler kernel-time window falsified it**
+    (496 decode tokens, rank0 trace parsed for device-activity time): 46.18 s of
+    kernel time = ~93 ms/token, of which **44 % is pointwise glue** —
+    `vectorized_elementwise_kernel` alone 16.07 s over **708,500 launches (1,428
+    per token)**, plus 3.11 s of `elementwise_kernel_manual_unroll` — against
+    **1.05 % for `r4d_moe_lru_gather_kernel`**. The MoE GEMM is 10 %, the split-K
+    GEMM 6.4 %, the router/top-k 8.9 %, the mHC kernels 6.5 %, and the inductor
+    fusions that DO happen 6 %. The real cause is that the fork compiles its own
+    Qwen trunk (`@support_torch_compile` in `models/qwen4_exp/amd/model.py`) and
+    **not** its GLM trunk, so every pointwise op is a separate eager kernel; the
+    engine says so at boot ("...provides neither a compiled submodule nor
+    breakable CUDA graphs. Overriding to FULL_DECODE_ONLY."). Growing the arena
+    13.08 -> 14.51 GiB still changed nothing, consistent with a 1 % gather.
+  - **Fork work staged (compile changelist, WIP — does not boot yet).**
+    `scripts/patches/glm53-compile-01-model-decorator.patch`,
+    `-02-trace-safe-probes.patch`, `-03-fullgraph-optout.patch`, each with its own
+    evidence header and base/patched sha256. Boots established, in order: (1)
+    `functools.lru_cache` does NOT make a probe trace-safe — dynamo re-traces the
+    wrapped body even when warm, so `r4d_mhc_supported -> r4d_mhc_enabled ->
+    available -> os.path.exists` breaks the graph (a plain module-global read
+    compiles; verified in-image); (2) `suppress_errors` does NOT cover graph
+    breaks, only `fullgraph=False` does; (3) the GLM trunk can never satisfy
+    `fullgraph` because its indexer metadata syncs to host
+    (`glm5next/amd/sparse_indexer.py:263,419`), and AOT compile refuses breaks
+    outright. With all three the trunk **traces** (`Dynamo bytecode transform
+    time: 1.08 s`) and then dies on `KeyError: 'in_proj_qkvbfg_a'` in the
+    mHC/KDA path — a latent bug the compile exposes. That is the remaining
+    blocker.
+  - **No new HIP kernels are possible** (the image ships `r4d.so` with no
+    sources), which binary-blocks the second-largest lever: the fused router is
+    gated `num_experts in {128, 256, 512}` and only
+    `r4d_moe_route_kernel<128|256|512>` instantiations exist, while GLM has 288
+    experts. `--kv-cache-dtype fp8` buys no capacity either (the plan's KV
+    reserve is dtype-blind), only removing gfx12's only MLA decode path.
+  - **DFlash2 for GLM is a target-side port, not a drafter port.**
+    `incoai/GLM-5.3-Flash-DFlash2` declares `architectures: ["DFlash2DraftModel"]`
+    and `model_type: "qwen3"` (5 dense layers, sliding attention, block 8,
+    `target_layer_ids [5, 14, 24, 33, 42]` of 45) and the fork **already maps it**
+    (`registry.py:636` -> `qwen3_dflash2.DFlash2Qwen3ForCausalLM`; spec method
+    `"dflash"`). What `glm5next` lacks is the target side: aux-hidden-state
+    capture (`set_aux_hidden_state_layers` / per-layer capture, as `deepseek_v2.py`
+    demonstrates). No speedup is budgeted: a verification pass pays pointwise cost
+    proportional to its token count.
   - Measured and rejected: **MTP4** (the head loads, arch resolves to
     `Glm5NextMTPModel`, acceptance is excellent at mean length 3.76-4.90 — but
-    decode was unchanged (9.6-11.2 vs 10.0) while prefill COLLAPSED 918 -> 255,
-    because a verification forward gathers the *union* of the batch's experts);
+    decode was unchanged (9.6-11.2 vs 10.0) while prefill COLLAPSED 918 -> 255.
+    The profiler window above explains it: the per-token cost is dominated by
+    pointwise glue that scales with the token count, so verifying a draft batch
+    pays ~N x that glue for the same N tokens of output);
     **fp8 KV** (disables the sparse MLA backend's rope-free Triton path, and
     aiter's MLA decode op is gated on CDNA>2, so the MTP draft step demands an op
     that cannot exist on gfx12); **NBATCH 8192** (boots, identical prefill).