A torch-profiler kernel-time window (496 decode tokens, rank0 trace parsed for
device-activity time) on the deployed GLM-5.3-Flash shape falsifies this page's
own "the expert gather is the bound" explanation:
46.18 s of kernel time / 496 tokens = ~93 ms per token (actual step ~104 ms)
34.8 % vectorized_elementwise_kernel 708,500 launches (1,428/token)
10.0 % r4d_gemm_moe_w4a16_nt_b16_m64 (the MoE GEMM)
6.7 % elementwise_kernel_manual_unroll
6.4 % wvSplitK_hf_sml_ (split-K GEMM)
8.9 % router: sbtopk + warpMergeSortTopK + bitonicSortKVInPlace
6.5 % r4d_mhc_* (hyper-connections)
1.05 % r4d_moe_lru_gather_kernel <-- the retracted suspect
The earlier arithmetic was circular (it assumed the gather consumed the whole
step). The real bound is unfused pointwise glue: the fork compiles its own Qwen
trunk (support_torch_compile in models/qwen4_exp/amd/model.py) and not its GLM
trunk, so the engine boots with "provides neither a compiled submodule nor
breakable CUDA graphs. Overriding to FULL_DECODE_ONLY." That also explains why
MTP4 measured flat — pointwise cost scales with the token count, so spec decode
amortizes nothing here — and it retires arena/hit-rate tuning (the gather is 1 %).
Staged as documented patch artifacts (not wired into any serve script; the
engine does not yet boot with them) — scripts/patches/glm53-compile-0{1,2,3}*.patch:
-01 models/glm5next/common/model.py decorate Glm5NextModel (R4D_GLM_COMPILE)
-02 model_executor/kernels/r4d_lib.py freeze 10 capability probes that do
os.path.exists/.so lookups inside the traced forward. functools.lru_cache
does NOT protect them: dynamo re-traces the wrapped body even when warm
(verified in-image); a plain module-global read compiles.
-03 compilation/wrapper.py make fullgraph=True opt-out, because
suppress_errors does NOT cover graph breaks and this trunk can never
satisfy fullgraph (indexer metadata syncs to host — .item() at
glm5next/amd/sparse_indexer.py:263,419).
With all three the trunk traces (Dynamo bytecode transform time 1.08 s) and dies
on KeyError: in_proj_qkvbfg_a in the mHC/KDA path — a latent bug the compile
exposes, and the remaining blocker.
Also records two structural constraints: no new HIP kernels are possible (r4d.so
ships without sources), which binary-blocks the fused router at GLM's 288
experts; and DFlash2 for GLM is a TARGET-side port, not a drafter port
(incoai/GLM-5.3-Flash-DFlash2 is architectures ["DFlash2DraftModel"] +
model_type qwen3, already mapped at registry.py:636; glm5next lacks only the
aux-hidden-state plumbing deepseek_v2.py demonstrates).
Runbook frontmatter, intro, mermaid and wall section, AGENTS.md and the work log
corrected to match the measurement. tests/run.sh: 362 seams pass.
Payload: Intel/GLM-5.3-Flash-W4A16-AutoRound (AutoRound symmetric int4 group
128, auto_round:auto_gptq, 34 shards + 2 extras, 113,074 tensors, ~170 GiB).
It is the only W4A16 checkpoint this fork can serve with expert offload: its
quant_method "auto-round" is overridden to "inc", which routes the MoE to
libr4d's grouped W4A16 GEMM (R4dW4a16MoEMethod). A compressed-tensors W4A16
build of the same model resolves to TritonWNA16Experts, which has no offload
recipe, and refuses to serve at all (170 GiB of experts cannot fit 64 GiB of
VRAM, so offload is mandatory).
Shape: tcclaviger RDNA4 vLLM fork (same digest-pinned image as the 2026-09-29
qwen cutover), TP=2, 131,072 ctx x 2 seqs, bf16 KV (fp8 is ILLEGAL here — it
disables the sparse MLA backend's rope-free Triton path, the only MLA decode
gfx12 has), util 0.98, NBATCH 2048, --enable-expert-offload
--expert-precise-memory, prefix caching, glm47 tool + reasoning parsers,
MTP and expert prefetch absent by measurement.
Measured: decode 9.9-10.0 tok/s, prefill 918 tok/s (warm), arena 14.51 GiB =
2,383 slots/rank over 12,096 host-backed experts (73.64 GiB pinned/rank).
THE 35 tok/s TARGET IS NOT MET and the shortfall is measured: live telemetry
(R4D_LRU_TELEMETRY=1) gives a 0.53-0.66 per-layer hit rate, a step moves
0.82 GiB of misses in ~100 ms = 8.2 GB/s effective against a link measured at
22.74 GB/s — the same structural gather-path gap the qwen campaign found.
Growing the arena 13.08 -> 14.51 GiB changed decode by nothing.
Rejected by measurement: MTP4 (head loads, acceptance 3.76-4.90, but decode
unchanged and prefill collapses 918 -> 255 — a verification forward gathers the
union of the batch's experts), NBATCH 8192 (identical prefill).
Also: scripts/patches/gpu_worker-kv-cap.patch (fork-local fix for a KV-reserve
model that under-counts GLM's per-sequence KDA state; not needed by this shape),
runbook, AGENTS.md row + section, both indexes, work log.
Rollback: 2339fae (qwen FP8 + MTP4).
fa on aborted graph build: build_attn_qsa called ggml_fill on the
F16-converted kq mask (GGML_ASSERT F32 at ggml.c:5371). Patch mirrors the
MLA sparse path in llm_graph_context::build_attn (F32 base mask for the
top-k surgery, F16 cast only at the attention call, gated on
cparams.flash_attn). Vendored at scripts/patches/qwen4exp-qsa-f32-mask.patch
(sha256 f2f4a486...); the build-qwen38 binary on Dankclaw is rebuilt with
it and validated live at CTX_SIZE=32768 (health 200, real completion).
Comrade Chief Designer
Frank