dank/serverAppConfigs
1
0
Fork 0
Code Issues 6 Pull Requests Actions Packages Projects Releases Wiki Activity

frank-primary: profile the GLM decode step and stage the compile changelist

Browse Source 
A torch-profiler kernel-time window (496 decode tokens, rank0 trace parsed for
device-activity time) on the deployed GLM-5.3-Flash shape falsifies this page's
own "the expert gather is the bound" explanation:

  46.18 s of kernel time / 496 tokens = ~93 ms per token (actual step ~104 ms)
   34.8 %  vectorized_elementwise_kernel      708,500 launches (1,428/token)
   10.0 %  r4d_gemm_moe_w4a16_nt_b16_m64      (the MoE GEMM)
    6.7 %  elementwise_kernel_manual_unroll
    6.4 %  wvSplitK_hf_sml_                    (split-K GEMM)
    8.9 %  router: sbtopk + warpMergeSortTopK + bitonicSortKVInPlace
    6.5 %  r4d_mhc_*                           (hyper-connections)
    1.05 % r4d_moe_lru_gather_kernel           <-- the retracted suspect

The earlier arithmetic was circular (it assumed the gather consumed the whole
step). The real bound is unfused pointwise glue: the fork compiles its own Qwen
trunk (support_torch_compile in models/qwen4_exp/amd/model.py) and not its GLM
trunk, so the engine boots with "provides neither a compiled submodule nor
breakable CUDA graphs. Overriding to FULL_DECODE_ONLY." That also explains why
MTP4 measured flat — pointwise cost scales with the token count, so spec decode
amortizes nothing here — and it retires arena/hit-rate tuning (the gather is 1 %).

Staged as documented patch artifacts (not wired into any serve script; the
engine does not yet boot with them) — scripts/patches/glm53-compile-0{1,2,3}*.patch:

  -01 models/glm5next/common/model.py    decorate Glm5NextModel (R4D_GLM_COMPILE)
  -02 model_executor/kernels/r4d_lib.py  freeze 10 capability probes that do
      os.path.exists/.so lookups inside the traced forward. functools.lru_cache
      does NOT protect them: dynamo re-traces the wrapped body even when warm
      (verified in-image); a plain module-global read compiles.
  -03 compilation/wrapper.py             make fullgraph=True opt-out, because
      suppress_errors does NOT cover graph breaks and this trunk can never
      satisfy fullgraph (indexer metadata syncs to host — .item() at
      glm5next/amd/sparse_indexer.py:263,419).

With all three the trunk traces (Dynamo bytecode transform time 1.08 s) and dies
on KeyError: in_proj_qkvbfg_a in the mHC/KDA path — a latent bug the compile
exposes, and the remaining blocker.

Also records two structural constraints: no new HIP kernels are possible (r4d.so
ships without sources), which binary-blocks the fused router at GLM's 288
experts; and DFlash2 for GLM is a TARGET-side port, not a drafter port
(incoai/GLM-5.3-Flash-DFlash2 is architectures ["DFlash2DraftModel"] +
model_type qwen3, already mapped at registry.py:636; glm5next lacks only the
aux-hidden-state plumbing deepseek_v2.py demonstrates).

Runbook frontmatter, intro, mermaid and wall section, AGENTS.md and the work log
corrected to match the measurement. tests/run.sh: 362 seams pass.
This commit is contained in:
Comrade Chief Designer
2026-09-30 09:36:48 -07:00
parent 979a4308f4
commit bd8c72ff47
6 changed files with 482 additions and 46 deletions
Download Patch File Download Diff File Expand all files Collapse all files
serverAppConfigs-docs/LLM/frank-primary-glm53-flash-w4a16.md
+141 -28
View File
@@ -1,7 +1,7 @@
---
type: runbook
title: frank-primary — GLM-5.3-Flash W4A16 int4 (tcclaviger, 2x R9700)
description: VERIFIED 2026-09-30 by boot probe, bench and live expert-cache telemetry. frank-primary cut over from Qwen3.8-Flash-Next-Uncensored-FP8 + MTP4 to Intel/GLM-5.3-Flash-W4A16-AutoRound, the first payload this fleet serves through the fork's W4A16 expert-offload path (INC scheme -> R4dW4a16MoEMethod, libr4d grouped W4A16 GEMM). Measured decode 9.9-10.0 tok/s and prefill 918 tok/s at 131,072 ctx. THE 35 tok/s TARGET IS NOT MET and this page shows why it is out of reach: the step is bounded by the fork's expert-gather read path at ~8.2 GB/s effective against a PCIe link that delivers 22.74 GB/s, with the cache at its measured hit-rate ceiling (0.53-0.66 per layer) from an arena that cannot grow past ~15 GiB under a 73.64 GiB/rank expert set.
description: VERIFIED 2026-09-30 by boot probe, bench, live expert-cache telemetry AND a torch-profiler kernel-time window. frank-primary cut over from Qwen3.8-Flash-Next-Uncensored-FP8 + MTP4 to Intel/GLM-5.3-Flash-W4A16-AutoRound, the first payload this fleet serves through the fork's W4A16 expert-offload path (INC scheme -> R4dW4a16MoEMethod, libr4d grouped W4A16 GEMM). Measured decode 9.9-10.0 tok/s and prefill 918 tok/s at 131,072 ctx. THE 35 tok/s TARGET IS NOT MET, and the profiler window FALSIFIED the first explanation: the expert gather is 1.05 % of device time, not the bound. 44 % of the 93 ms/token budget is pointwise glue (1,428 elementwise launches per token) because the fork's GLM trunk is NOT torch.compiled while its own Qwen trunk is; the second-largest lever, the fused router, is binary-blocked at 288 experts. The compile changelist is staged but incomplete (three patches, one blocker left).
tags:
- dankserver
- r9700
@@ -22,12 +22,16 @@ timestamp: 2026-09-30
> `GLM-5.3-Flash` at int4 on two R9700s with expert offload, and how fast is it?
> **Answer.** Yes — it serves, with vision, tool calls and 131,072 ctx. It runs
> at **9.9-10.0 tok/s decode / 918 tok/s prefill**, and **35 tok/s is not
> reachable** with this payload on this hardware. The bound is not the model,
> the quant, or the link: it is the fork's expert-gather read path, and it is
> measured, not inferred.
> reachable** on this payload as the fork ships it. The bound is **not** the
> model, the quant, the link, or the expert gather (measured: 1.05 % of device
> time). It is the **unfused pointwise glue of an uncompiled trunk** — 44 % of a
> 93 ms per-token budget spread over ~1,428 elementwise launches per token —
> because the fork compiles its own Qwen trunk and not its GLM one. See
> [The measured profile](#the-measured-profile-and-the-real-wall).
Related: [[dankserver-qwen38-fp8-expert-offload-tuning]] (the same fork, the
same hardware, the same gather-path wall, found first on qwen),
Related: [[dankserver-qwen38-fp8-expert-offload-tuning]] (the same fork, the same
hardware — its conclusion was drawn from throughput arithmetic and is superseded
by this page's kernel-time measurement),
[[dankserver-frontier-moe-feasibility]] (the earlier fit analysis),
[[VLLM Index]], [[LLM/index|LLM fleet]].
@@ -51,7 +55,7 @@ graph LR
D --> E[ExpertLRUCache<br/>2,383 VRAM slots / rank]
D --> F[73.64 GiB pinned host rows / rank]
E --> G[decode step]
F -->|8.2 GB/s effective| G
F -->|host misses<br/>1.05% of device time| G
```
## Why this payload and not another
@@ -90,33 +94,142 @@ Measured on the deployed engine, 2026-09-30 (`bench_glm.py`, real prose, greedy)
Decode is flat across context and across every arena change — the signature of a
per-step cost that is not KV- or residency-driven at the margin.
### The arithmetic that bounds it
### The measured profile, and the real wall
Per decode step, per rank: 42 MoE layers x 8 routed experts x 6.234 MiB =
**2.045 GiB of expert weights**, which must either be in VRAM or cross PCIe.
**This section replaces an earlier explanation, which the measurement falsified.**
The first reading of this page bounded the step with throughput arithmetic: 42 MoE
layers x 8 experts x 6.234 MiB = 2.045 GiB per step per rank, 0.82 GiB of it
missed at a measured 0.53-0.66 hit rate, giving "0.82 GiB in ~100 ms = 8.2 GB/s =
the gather is the wall". That arithmetic was circular — it *assumed* the gather
consumed the whole step — and it is wrong. The engine, asked directly, disagrees:
| hit rate | miss traffic | step @ 8.2 GB/s | decode |
A torch-profiler window (`--profiler-config`, `POST /start_profile` /
`stop_profile`, 496 decode tokens at the deployed shape, rank0 trace parsed for
device-activity time) gives the true split. Total device kernel time **46.18 s /
496 tokens = ~93 ms per token** against a measured ~104 ms step (9.6 tok/s with
the profiler attached):
| ms | % | launches | kernel |
| --- | --- | --- | --- |
| 0.60 (measured) | 0.82 GiB | ~105 ms | **9.5-10.0 tok/s** |
| 0.70 | 0.61 GiB | ~78 ms | 12.8 tok/s |
| 0.89 (what 35 tok/s needs) | 0.23 GiB | ~29 ms | 35 tok/s |
| 16,074 | **34.8** | 708,500 | `vectorized_elementwise_kernel` (22.7 us each) |
| 4,602 | 10.0 | 42,000 | `r4d_gemm_moe_w4a16_nt_b16_m64` — the MoE GEMM |
| 3,108 | 6.7 | 174,708 | `elementwise_kernel_manual_unroll` |
| 2,943 | 6.4 | 138,252 | `wvSplitK_hf_sml_` — split-K GEMM (projections / head) |
| 2,829 | 6.1 | 83,160 | router: `sbtopk::gatherTopK` + `warpMergeSortTopK` + `bitonicSortKVInPlace` (1,208 + 1,051 + 559 ms; the 1,051 row is not in the list below) |
| 3,015 | 6.5 | 133,920 | `r4d_mhc_pre_finish/pre_partial/post` — hyper-connections |
| 2,782 | 6.0 | 127,916 | inductor `triton_poi_*` / `triton_per_*` (the fusions that DO happen) |
| **487** | **1.05** | 20,664 | **`r4d_moe_lru_gather_kernel`** — the accused |
| 250 | 0.54 | 20,832 | `_moe_lru_tele_kernel` — the telemetry that measured it (TELEM was on) |
Two independent measurements pin the missed term:
So the gather is **1.05 %** of device time, and `r4d_moe_lru_fused_kernel` (the
LRU *manager*, not the copy) is a further 1,599 ms / 3.5 %. Residency work and the
PCIe link are not the story: 44 % of the budget is **pointwise glue** — bf16<->fp32
conversions, adds, `pow`/`rsqrt` from fp32 RMSNorm, `launch_clamp_scale` quant
passes, `reduce_kernel` means — spread over **~1,428 elementwise launches per
token** (3,734 kernel launches per token in total). The GPU is busy 86 % of the
wall clock, so this is real device work, not launch-gap overhead.
- **Live expert-cache telemetry** (`R4D_LRU_TELEMETRY=1`, profiler window)
gives the hit rate directly — the LRU is *working*:
`layers.34: slots 56 (pinned 0, tier 0) hit 0.6279 routed 2002 installs 543
read-through 202` … `layers.44: hit 0.5335`. Per-layer range **0.53-0.66**.
- **The link delivers 22.74 GB/s** at the engine's own slab size (H2D probe,
pinned, 2.345 MiB). The step moves 0.82 GiB in ~100 ms = **8.2 GB/s —
36% of the wire**, matching the fork's own measured 6.5-9.5 GB/s.
Two consequences that kill three whole families of planned work:
So the step is bounded by the fork's gather read path, and the only other lever
is the hit rate — which is capped by the arena. A 14.51 GiB arena holds 20% of a
73.64 GiB expert set; the measured hit rate beats random by ~3x thanks to
routing locality, but it saturates: growing the arena 13.08 → 14.51 GiB
(+9%) changed decode by **nothing** (10.03 → 9.88 tok/s). Reaching 0.89 would
need roughly 4.5x the per-layer slot count — an arena of well over 100 GiB.
- **Spec decode cannot help this shape, and the MTP measurement was not an
accident.** Pointwise cost scales with the token count, so a draft batch that
verifies 4 tokens pays ~4x the glue for the same per-token output — which is
exactly why MTP4 measured "acceptance 3.76-4.90, decode unchanged". Same reason
the gather was never the bound.
- **Arena size and hit rate are not levers.** Confirmed twice: `--expert-precise-memory`
(arena 13.08 -> 14.51 GiB) moved decode by nothing, and the gather is 1 %.
## Fork work: the compile changelist (WIP, does not boot yet)
The fork compiles its own flagship trunk — `vllm/models/qwen4_exp/amd/model.py`
carries `@support_torch_compile` — and **not** its GLM trunk. The engine says so on
every boot:
```
Cudagraph mode FULL_AND_PIECEWISE requires piecewise capture, but the loaded model
provides neither a compiled submodule nor breakable CUDA graphs.
Overriding to FULL_DECODE_ONLY.
```
Inductor fusion is therefore the only lever that touches the 44 % pointwise mass.
Three patches are staged in `scripts/patches/glm53-compile-0*.patch`, each carrying
its own evidence header, base/patched sha256 and the next blocker:
| patch | target | fixes |
| --- | --- | --- |
| `-01-model-decorator` | `models/glm5next/common/model.py` | decorates `Glm5NextModel` (env `R4D_GLM_COMPILE`, default on) |
| `-02-trace-safe-probes` | `model_executor/kernels/r4d_lib.py` | freezes 10 capability probes that do `os.path.exists`/`.so` lookups **inside the traced forward** |
| `-03-fullgraph-optout` | `compilation/wrapper.py` | makes `fullgraph=True` opt-out (env `VLLM_COMPILE_FULLGRAPH=0`) |
What the boots established, in order (each one ~7 min):
1. **01 alone dies** on the first cold probe call inside the traced region:
`Dynamo does not know how to trace the builtin 'posix.stat'` —
`r4d_mhc_supported -> r4d_lib.r4d_mhc_enabled -> available -> os.path.exists`.
**`functools.lru_cache` does not protect these probes**: dynamo re-traces the
wrapped body to build its guard even when the cache is warm (it warns about
exactly this), verified in-image — a *warm* lru_cache-calling function still
fails `fullgraph=True`, while a plain module-global read compiles.
2. **01+02 get past it** (`libr4d capability probes frozen: 10 warm [...]` in all
four processes) and die on
`RuntimeError: Graph breaks are not supported with aot compile`, because this
build runs AOT compile by default and AOT refuses breaks.
3. **01+02+03 with `VLLM_COMPILE_FULLGRAPH=0 VLLM_USE_AOT_COMPILE=0` traces the
trunk successfully** (`Dynamo bytecode transform time: 1.08 s`) and then dies in
`profile_run` with `KeyError: 'in_proj_qkvbfg_a'` at
`glm5next/common/model.py:456` (the KDA layer's merged projection,
`kda.py:217`) — a dict/registry lookup keyed by attribute name that the eager
path never reaches, i.e. **a latent bug the compile exposes, not a dynamo
limit**.
That last row is the state of the work: the remaining blocker is a real bug in the
fork's mHC/KDA wiring, and it must be fixed (plus whatever follows it) before the
changelist can boot. Two verified facts make the payoff concrete: `suppress_errors`
does **not** cover graph breaks (only `fullgraph=False` does), and the GLM trunk can
never satisfy `fullgraph` because its indexer metadata syncs to host
(`glm5next/amd/sparse_indexer.py:263,419`: `int(...max().item())`).
### Two structural constraints on any further fork work
- **No new HIP kernels are possible.** The image ships `r4d.so` as a binary with no
sources. The obvious second lever — the 9 % router — is gated
`num_experts in {128, 256, 512}` in `router/r4d_route_router.py` and the binary
contains only `r4d_moe_route_kernel<128|256|512>` instantiations. GLM has **288**
experts, so it falls back to the generic sort-based top-k and **cannot be moved to
the fused router without rebuilding libr4d**. Only Python/config/inductor changes
(and runtime-JIT Triton) are available.
- **`--kv-cache-dtype fp8` is a dead end for capacity too.** The plan's KV reserve
is dtype-blind (1.76 GiB at 131,072 either way), so fp8 buys no arena and no
window; it only removes the only MLA decode path gfx12 has (trap 1 above).
### DFlash2 for GLM — what the drafter actually is
`incoai/GLM-5.3-Flash-DFlash2` is **not** a GLM-shaped drafter and needs no MLA/DSA
port: its `config.json` declares `architectures: ["DFlash2DraftModel"]` and
`model_type: "qwen3"` — 5 dense layers, hidden 4096, 32 heads / 8 kv heads,
head_dim 128, sliding attention (window 2048), `is_causal: false`,
`use_cache: false`, `dflash_config` = block 8, conv group 16, selector rank 256 /
top-k 16, and **`target_layer_ids: [5, 14, 24, 33, 42]` against
`num_target_layers: 45`**.
The fork already maps that architecture:
`registry.py:636 "DFlash2DraftModel": ("qwen3_dflash2", "DFlash2Qwen3ForCausalLM")`,
and the spec method is `"dflash"` (`config/speculative.py:67`). So the port is
**target-side only**: the GLM trunk must expose auxiliary hidden states at those five
layer indices, the same interface `deepseek_v2.py` (also MLA+MoE) implements —
`set_aux_hidden_state_layers()` / `get_eagle3_aux_hidden_state_layers()` plus
capture at `idx in self.aux_hidden_state_layers` in the layer loop. `glm5next`
implements neither.
Expected benefit is **not** assumed: DFlash2 amortizes a draft block (8 tokens) over
one verification pass, and per the profile above a verification pass costs pointwise
work proportional to tokens — the same reason MTP4 was flat. Measure it; do not
budget on it.
Note also that the KV-cap patch's predecessor arithmetic in
[[dankserver-qwen38-fp8-expert-offload-tuning]] should be read the same way: its
"gather-bound" conclusion came from the same throughput arithmetic that this page's
profiler window has now falsified.
## Measured arms (all 2026-09-30, one boot each)
serverAppConfigs-docs/log.md
+50 -9
View File
@@ -20,17 +20,58 @@
- Measured on the deployed engine: **decode 9.9-10.0 tok/s, prefill 918 tok/s**
(warm, 2.6k prompt), KV 138,561 tokens; arena 14.51 GiB = 2,383 slots per
rank over 12,096 host-backed experts (73.64 GiB pinned per rank).
- **The wall:** live expert-cache telemetry (`R4D_LRU_TELEMETRY=1` + a
`with_stack=false` profiler window) puts the cache hit rate at 0.53-0.66 per
layer; a step moves 0.82 GiB of misses in ~100 ms = **8.2 GB/s effective
against a link measured at 22.74 GB/s** — the same structural gather-read-path
gap the 2026-09-29 qwen campaign found. Growing the arena 13.08 -> 14.51 GiB
(+9%) changed decode by *nothing* (10.03 -> 9.88 tok/s), so residency stopped
being the binding term; 35 tok/s would need a 0.89 hit rate.
- **The wall — and a corrected one.** The first account of the shortfall
bounded the step with throughput arithmetic (a step moves 0.82 GiB of misses
in ~100 ms = "8.2 GB/s effective against a 22.74 GB/s link", needing a 0.89
hit rate for 35 tok/s). **A torch-profiler kernel-time window falsified it**
(496 decode tokens, rank0 trace parsed for device-activity time): 46.18 s of
kernel time = ~93 ms/token, of which **44 % is pointwise glue** —
`vectorized_elementwise_kernel` alone 16.07 s over **708,500 launches (1,428
per token)**, plus 3.11 s of `elementwise_kernel_manual_unroll` — against
**1.05 % for `r4d_moe_lru_gather_kernel`**. The MoE GEMM is 10 %, the split-K
GEMM 6.4 %, the router/top-k 8.9 %, the mHC kernels 6.5 %, and the inductor
fusions that DO happen 6 %. The real cause is that the fork compiles its own
Qwen trunk (`@support_torch_compile` in `models/qwen4_exp/amd/model.py`) and
**not** its GLM trunk, so every pointwise op is a separate eager kernel; the
engine says so at boot ("...provides neither a compiled submodule nor
breakable CUDA graphs. Overriding to FULL_DECODE_ONLY."). Growing the arena
13.08 -> 14.51 GiB still changed nothing, consistent with a 1 % gather.
- **Fork work staged (compile changelist, WIP — does not boot yet).**
`scripts/patches/glm53-compile-01-model-decorator.patch`,
`-02-trace-safe-probes.patch`, `-03-fullgraph-optout.patch`, each with its own
evidence header and base/patched sha256. Boots established, in order: (1)
`functools.lru_cache` does NOT make a probe trace-safe — dynamo re-traces the
wrapped body even when warm, so `r4d_mhc_supported -> r4d_mhc_enabled ->
available -> os.path.exists` breaks the graph (a plain module-global read
compiles; verified in-image); (2) `suppress_errors` does NOT cover graph
breaks, only `fullgraph=False` does; (3) the GLM trunk can never satisfy
`fullgraph` because its indexer metadata syncs to host
(`glm5next/amd/sparse_indexer.py:263,419`), and AOT compile refuses breaks
outright. With all three the trunk **traces** (`Dynamo bytecode transform
time: 1.08 s`) and then dies on `KeyError: 'in_proj_qkvbfg_a'` in the
mHC/KDA path — a latent bug the compile exposes. That is the remaining
blocker.
- **No new HIP kernels are possible** (the image ships `r4d.so` with no
sources), which binary-blocks the second-largest lever: the fused router is
gated `num_experts in {128, 256, 512}` and only
`r4d_moe_route_kernel<128|256|512>` instantiations exist, while GLM has 288
experts. `--kv-cache-dtype fp8` buys no capacity either (the plan's KV
reserve is dtype-blind), only removing gfx12's only MLA decode path.
- **DFlash2 for GLM is a target-side port, not a drafter port.**
`incoai/GLM-5.3-Flash-DFlash2` declares `architectures: ["DFlash2DraftModel"]`
and `model_type: "qwen3"` (5 dense layers, sliding attention, block 8,
`target_layer_ids [5, 14, 24, 33, 42]` of 45) and the fork **already maps it**
(`registry.py:636` -> `qwen3_dflash2.DFlash2Qwen3ForCausalLM`; spec method
`"dflash"`). What `glm5next` lacks is the target side: aux-hidden-state
capture (`set_aux_hidden_state_layers` / per-layer capture, as `deepseek_v2.py`
demonstrates). No speedup is budgeted: a verification pass pays pointwise cost
proportional to its token count.
- Measured and rejected: **MTP4** (the head loads, arch resolves to
`Glm5NextMTPModel`, acceptance is excellent at mean length 3.76-4.90 — but
decode was unchanged (9.6-11.2 vs 10.0) while prefill COLLAPSED 918 -> 255,
because a verification forward gathers the *union* of the batch's experts);
decode was unchanged (9.6-11.2 vs 10.0) while prefill COLLAPSED 918 -> 255.
The profiler window above explains it: the per-token cost is dominated by
pointwise glue that scales with the token count, so verifying a draft batch
pays ~N x that glue for the same N tokens of output);
**fp8 KV** (disables the sparse MLA backend's rope-free Triton path, and
aiter's MLA decode op is gated on CDNA>2, so the MTP draft step demands an op
that cannot exist on gfx12); **NBATCH 8192** (boots, identical prefill).