diff --git a/AGENTS.md b/AGENTS.md index 581679d..155a2cb 100644 --- a/AGENTS.md +++ b/AGENTS.md @@ -24,7 +24,7 @@ merge absorbed the former `vllmScripts` repo, history preserved). | Target | Service | Runtime | Model | Port | |--------|---------|---------|-------|------| -| Dankserver | `frank-primary` | tcclaviger RDNA4 vLLM fork (`docker.io/tcclaviger/vllm@sha256:da4029666…`, digest-pinned, plain `podman run`, root systemd unit) | `Intel/GLM-5.3-Flash-W4A16-AutoRound` — AutoRound symmetric int4 g128 (170 GiB, 34 shards, 113,074 tensors), TP=2, expert offload to 73.64 GiB/rank of pinned host rows behind a 14.51 GiB / 2,383-slot VRAM LRU arena, 131072 ctx × 2 seqs, bf16 KV, no MTP, no prefetch; measured 9.9-10.0 tok/s decode / 918 tok/s prefill | 7770 | +| Dankserver | `frank-primary` | tcclaviger RDNA4 vLLM fork (`docker.io/tcclaviger/vllm@sha256:da4029666…`, digest-pinned, plain `podman run`, root systemd unit) | `Intel/GLM-5.3-Flash-W4A16-AutoRound` — AutoRound symmetric int4 g128 (170 GiB, 34 shards, 113,074 tensors), TP=2, expert offload to 73.64 GiB/rank of pinned host rows behind a 14.51 GiB / 2,383-slot VRAM LRU arena, 131072 ctx × 2 seqs, bf16 KV, no MTP, no prefetch; measured 9.9-10.0 tok/s decode / 918 tok/s prefill, bounded by unfused pointwise glue (the expert gather is 1.05 % of device time; the trunk is uncompiled) | 7770 | | Dankclaw | `frank-large` | gufo `gufo-org/gufo` v0.1.1, the Strix Halo engine, BUILT FROM SOURCE (root systemd unit; links the host ROCm 7.2.4) | Swift-1.5-Qwen3.8-27B-Uncensored UD-Q8_K_XL (27B dense, ABLITERATED/uncensored, 29.3 GiB, SWIFT OPEN LICENSE 1.0) with the z-lab DFlash2 Q4_K_M drafter (adaptive block policy, cap 7) on the gfx1151 iGPU (Strix Halo LPDDR, 114 GiB GTT), 262144 ctx single session, greedy, gufo's compiled-in Qwen renderer (NO Qwen-Sharp), text only | 7770 | | Dankclaw | `frank-small` | mainline llama.cpp master 37b53fd (root systemd unit) | HauhauCS Gemma4-12B-QAT-Uncensored-Balanced Q4_K_M (BF16 mmproj + MTP assistant sidecar, `draft-mtp,ngram-mod`) on the RTX 3090, 1048576 pool / 4 slots of 262,144, q8_0 KV, the fleet's fastest vision endpoint | 7771 | @@ -445,14 +445,51 @@ serve. Built shape: 14.51 GiB arena = **2,383 slots × 6.234 MiB per rank** over 12,096 host-backed experts = **73.64 GiB pinned host rows per rank**. **Measured 2026-09-30: decode 9.9–10.0 tok/s, prefill 918 tok/s** (warm, 2.6k -prompt), KV cache 138,561 tokens. **The 35 tok/s target is NOT met, and this is -a measured wall rather than a tuning gap:** each decode step must move 2.045 GiB -of routed experts per rank (42 layers × 8 experts × 6.234 MiB), live telemetry -(`R4D_LRU_TELEMETRY=1`) puts the expert-cache hit rate at **0.53–0.66**, and the -fork's gather read path delivers **~8.2 GB/s against a link measured at -22.74 GB/s** — the same structural gap the qwen campaign found. Growing the -arena 13.08 → 14.51 GiB changed decode by *nothing* (10.03 → 9.88 tok/s), so -residency stopped being the binding term; 0.89 hit rate would be needed for 35. +prompt), KV cache 138,561 tokens. **The 35 tok/s target is NOT met. The first +explanation for that — "the expert gather is the bound at ~8.2 GB/s" — was +FALSIFIED by a torch-profiler kernel-time window** (496 decode tokens, rank0 +trace parsed for device-activity time): 46.18 s of kernel time = ~93 ms per +token, of which **44 % is pointwise glue** — `vectorized_elementwise_kernel` +alone is 16.07 s over **708,500 launches (1,428 per token)**, plus 3.11 s of +`elementwise_kernel_manual_unroll` — against **1.05 % for +`r4d_moe_lru_gather_kernel`**. The MoE GEMM is 10 %, the split-K GEMM 6 %, the +router/top-k 9 %, the mHC kernels 6.5 %. The real wall is that **the fork +compiles its own Qwen trunk (`@support_torch_compile` in +`models/qwen4_exp/amd/model.py`) and not its GLM trunk**, so every pointwise op +is a separate eager kernel. Two consequences: spec decode **cannot** help this +shape (pointwise cost scales with the token count, which is exactly why MTP4 +measured flat), and neither arena size nor hit rate is a lever (the gather is +1 %). + +**Fork work is staged, not deployed.** `scripts/patches/glm53-compile-0{1,2,3}*.patch` +are the compile-enablement changelist (decorate the trunk; freeze the ten libr4d +capability probes that do `os.path.exists`/`.so` lookups *inside* the traced +forward; make `fullgraph=True` opt-out). Boots established that +`functools.lru_cache` does **not** protect those probes (dynamo re-traces the +wrapped body even warm), that `suppress_errors` does **not** cover graph breaks +(only `fullgraph=False` does), and that the GLM trunk can never satisfy +`fullgraph` (its indexer metadata syncs to host at +`glm5next/amd/sparse_indexer.py:263,419`). With all three the trunk **traces** +(`Dynamo bytecode transform time: 1.08 s`) and then dies on +`KeyError: 'in_proj_qkvbfg_a'` in the mHC/KDA path — a latent bug the compile +exposes, and the remaining blocker. + +**No new HIP kernels are possible** — the image ships `r4d.so` as a binary with +no sources. The 9 % router is therefore **binary-blocked**: its fused path is +gated `num_experts in {128, 256, 512}` (`router/r4d_route_router.py`) and only +`r4d_moe_route_kernel<128|256|512>` instantiations exist, while GLM has **288** +experts. Only Python/config/inductor changes (or runtime-JIT Triton) are +available on this fork. + +**DFlash2 for GLM is a target-side port, not a drafter port.** +`incoai/GLM-5.3-Flash-DFlash2` declares `architectures: ["DFlash2DraftModel"]` +and `model_type: "qwen3"` (5 dense layers, sliding attention, block 8, +`target_layer_ids [5, 14, 24, 33, 42]` of 45), and the fork **already maps it** +(`registry.py:636` → `qwen3_dflash2.DFlash2Qwen3ForCausalLM`; spec method +`"dflash"`). What is missing is the target side: `glm5next` implements neither +`set_aux_hidden_state_layers()` nor the per-layer aux capture that +`deepseek_v2.py` (also MLA+MoE) demonstrates. Budget no speedup: per the profile +above a verification pass pays pointwise cost proportional to its token count. **MTP works here and is still rejected.** The head loads (`Resolved architecture: Glm5NextMTPModel`) and accepts well (mean accepted diff --git a/scripts/patches/glm53-compile-01-model-decorator.patch b/scripts/patches/glm53-compile-01-model-decorator.patch new file mode 100644 index 0000000..675190f --- /dev/null +++ b/scripts/patches/glm53-compile-01-model-decorator.patch @@ -0,0 +1,67 @@ +# glm53-compile-01-model-decorator.patch — put the GLM-5.3-Flash trunk under torch.compile. +# +# STATUS: WIP — first of three patches in the "compile the GLM trunk" changelist. The +# set does NOT yet boot the engine (see 03 for the next blocker), so it is NOT part of +# the deployed frank-primary shape. Nothing in scripts/ mounts these files. +# +# WHY THIS IS THE RIGHT LEVER (measured 2026-09-30, not inferred): +# +# The fork's own flagship trunk — vllm/models/qwen4_exp/amd/model.py — carries +# @support_torch_compile. vllm/models/glm5next/common/model.py does not, and the engine +# says so on every boot: +# +# Cudagraph mode FULL_AND_PIECEWISE requires piecewise capture, but the loaded model +# provides neither a compiled submodule nor breakable CUDA graphs. +# Overriding to FULL_DECODE_ONLY. +# +# A torch-profiler window on the live endpoint (496 decode tokens, rank0, 3 x 160-token +# generations at the deployed shape) shows what the missing compilation costs. Total +# device kernel time 46.18 s / 496 tokens = ~93 ms per token, against a measured step +# time of ~104 ms (9.6 tok/s with the profiler attached): +# +# 16.07 s 34.8 % 708,500 launches vectorized_elementwise_kernel (22.7 us each) +# 4.60 s 10.0 % 42,000 r4d_gemm_moe_w4a16_nt_b16_m64 (the MoE GEMM) +# 3.11 s 6.7 % 174,708 elementwise_kernel_manual_unroll +# 2.94 s 6.4 % 138,252 wvSplitK_hf_sml_ (split-K GEMM) +# 4.09 s 8.9 % 83,160 topk/router (sbtopk gatherTopK + warpMergeSortTopK +# + bitonicSortKVInPlace) +# 3.01 s 6.5 % 133,920 r4d_mhc_* (hyper-connection pre/post/finish) +# 2.78 s 6.0 % 127,916 inductor triton_poi_* / triton_per_* (the only +# fusions that DO happen) +# 0.49 s 1.05% 20,664 r4d_moe_lru_gather_kernel <-- the suspect +# +# That last row retired the fleet's working hypothesis. Decode on this shape is NOT +# expert-gather bound: the gather is 1 % of device time, the same order as the telemetry +# kernel that was switched on to measure it. 44 % of the budget is pointwise glue — +# bf16<->fp32 conversions, adds, pows/rsqrts from fp32 RMSNorm, clamp/scale quant +# passes — spread over ~1,428 launches per token. Inductor fusion is the only lever +# that touches that mass; the router is second at 9 % but is binary-blocked (the +# libr4d route kernel exists only as r4d_moe_route_kernel<128|256|512> instantiations +# and GLM has 288 experts, with no HIP sources shipped to rebuild). +# +# TARGET: vllm/models/glm5next/common/model.py (inside the pinned image) +# base sha256 008eeaa2a07e091465319f86900b5a321d921238723ccbe22e6b9e971e1d2b28 +# patched sha256 6e9cb81b84a9a88bd41403ca404aa04c3b791899dd77d661c3f25ae686cecd40 +# +# APPLY: `patch -p1` from the vllm package root, or bind-mount new/.../model.py over +# site-packages. Then R4D_GLM_COMPILE=1 (default) enables it; =0 makes the decorator a +# no-op, so the file can stay mounted for an A/B without a second artifact. +# Requires 02 (trace-safe probes) — without it the worker dies during profile_run on +# the first probe call inside the traced region. +--- a/vllm/models/glm5next/common/model.py ++++ b/vllm/models/glm5next/common/model.py +@@ Glm5NextModel: add @support_torch_compile (env): R4D_GLM_COMPILE=0 disables @@ ++_GLM_COMPILE = os.environ.get("R4D_GLM_COMPILE", "1") == "1" ++ ++ ++@support_torch_compile( ++ dynamic_arg_dims={ ++ "input_ids": 0, ++ "positions": -1, ++ "intermediate_tensors": 0, ++ "inputs_embeds": 0, ++ }, ++ enable_if=lambda _cfg: _GLM_COMPILE, ++) +-class Glm5NextModel(nn.Module): ++class Glm5NextModel(nn.Module): # decorated above diff --git a/scripts/patches/glm53-compile-02-trace-safe-probes.patch b/scripts/patches/glm53-compile-02-trace-safe-probes.patch new file mode 100644 index 0000000..fdaffb3 --- /dev/null +++ b/scripts/patches/glm53-compile-02-trace-safe-probes.patch @@ -0,0 +1,114 @@ +# glm53-compile-02-trace-safe-probes.patch — libr4d capability probes must not do I/O +# inside a torch.compile region. +# +# STATUS: WIP — second of three patches in the "compile the GLM trunk" changelist. +# Inert (but harmless) without 01. Not part of the deployed shape. +# +# THE BREAK (measured: with 01 mounted and R4D_GLM_COMPILE=1, the worker dies during +# profile_run, i.e. the first — cold — forward): +# +# Explanation: Dynamo does not know how to trace the builtin `posix.stat` +# ... r4d_mhc_supported -> r4d_lib.r4d_mhc_enabled -> available -> os.path.exists +# +# THE NON-OBVIOUS PART: functools.lru_cache does NOT protect these probes. Dynamo +# re-traces the wrapped body to build its guard even when the cache is warm — it warns +# about exactly this ("Dynamo detected a call to a `functools.lru_cache`-wrapped +# function ... Dynamo ignores the cache wrapper and directly traces the wrapped +# function"). Verified in-image, torch 2.11.0+rocm10.0, torch.compile(backend="eager"): +# +# cold lru_cache call -> RAISE Unsupported +# WARM lru_cache call -> RAISE Unsupported <-- warming is not enough +# plain module-global read -> compiles +# uncached os.path.exists -> RAISE Unsupported +# +# So each probe must resolve ONCE into a plain value and be read as a global. The set +# below is every no-argument probe / symbol lookup reachable from a forward pass, +# grep-verified call sites: +# model_executor/layers/fused_moe/experts/r4d_w4a16_moe.py:79 available() +# model_executor/layers/fused_moe/experts/r4d_w4a16_moe.py:81 has_w4a16_moe() +# model_executor/kernels/mhc/r4d_dsv4.py:16 r4d_mhc_enabled() +# v1/attention/ops/rocm_aiter_mla_sparse.py:694 r4d_indexer_enabled(), mqa_max_heads() +# v1/attention/ops/rocm_aiter_mla_sparse.py:1124 r4d_indexer_enabled() +# model_executor/layers/fused_moe/experts/r4d_mxfp4_moe.py:683 r4d_act_quant_enabled() +# model_executor/layers/fused_moe/experts/r4d_w4a16_moe.py _entry() moe_w4a16()/moe_w4a16_mt1() per MoE call +# (lib() is the transitive dependency of all of them.) +# +# TARGET: vllm/model_executor/kernels/r4d_lib.py (inside the pinned image) +# base sha256 b8dfeefb4acc72b67b0579a4ab0653ef5d48c40cc8649187d757c80ce6b9617a +# patched sha256 0751852e1ca813ad2bdb1da7bf10c605130e63e04ef9ec57e70b5f675a29c08e +# +# APPLY: append the block below to the module. R4D_FREEZE_PROBES=0 restores stock +# behaviour. Cost when nothing is compiled: ten one-time probes at import (measured: +# logged once per process, e.g. "libr4d capability probes frozen: 10 warm [...]"). +# +# OBSERVED EFFECT: with 01+02 the worker gets past this break and reaches the next one +# (see 03) — dynamo reports "Dynamo bytecode transform time: 1.08 s" for the trunk. +--- a/vllm/model_executor/kernels/r4d_lib.py ++++ b/vllm/model_executor/kernels/r4d_lib.py +@@ append: trace-safe capability probes (env): R4D_FREEZE_PROBES=0 disables @@ ++# FORK-LOCAL (patch: trace-safe capability probes). torch.compile traces the ++# BODY of a functools.lru_cache call even when the cache is warm, so a capability ++# probe called from a forward pass is a dynamo graph break. Measured 2026-09-30 ++# on the GLM-5.3-Flash trunk under @support_torch_compile: ++# Dynamo does not know how to trace the builtin `posix.stat` ++# r4d_mhc_supported -> r4d_mhc_enabled -> available -> os.path.exists ++# Each probe below is wrapped to resolve ONCE into a memo and is warmed at import ++# (outside any tracing region), so a traced call is a dict read. The probe set is ++# every no-argument capability probe / symbol lookup reachable from a forward ++# pass: the W4A16 + mxfp4 MoE dispatch, the mHC op, the MLA sparse indexer gate ++# and the draft/lm-head guards. ++_PROBE_NAMES = ( ++ "available", ++ "lib", ++ "has_w4a16_moe", ++ "moe_w4a16", ++ "moe_w4a16_mt1", ++ "has_mhc_dsv4", ++ "r4d_mhc_enabled", ++ "r4d_indexer_enabled", ++ "mqa_max_heads", ++ "r4d_act_quant_enabled", ++) ++ ++ ++def _freeze_probe(fn): ++ """Return fn bound to its first result. Result must be a plain value.""" ++ ++ memo = [] ++ ++ def probe(*args, **kwargs): ++ if not memo: ++ memo.append(fn(*args, **kwargs)) ++ return memo[0] ++ ++ return probe ++ ++ ++def _freeze_probes() -> list[str]: ++ frozen = [] ++ for name in _PROBE_NAMES: ++ fn = globals().get(name) ++ if fn is None: ++ continue ++ # Only the fast branch of a bound probe is traced; an empty memo means the ++ # first call still does I/O, so populate it here as well. ++ globals()[name] = _freeze_probe(fn) ++ frozen.append(name) ++ return frozen ++ ++ ++from vllm.logger import init_logger as _init_logger ++ ++_probe_log = _init_logger(__name__) ++ ++if os.environ.get("R4D_FREEZE_PROBES", "1") == "1": ++ _warmed, _failed = [], [] ++ for _n in _freeze_probes(): ++ try: ++ globals()[_n]() ++ _warmed.append(_n) ++ except Exception: # noqa: BLE001 -- probe unavailable here; recompute lazily ++ _failed.append(_n) ++ _probe_log.info("libr4d capability probes frozen: %d warm %s%s", ++ len(_warmed), _warmed, ++ f" (deferred: {_failed})" if _failed else "") diff --git a/scripts/patches/glm53-compile-03-fullgraph-optout.patch b/scripts/patches/glm53-compile-03-fullgraph-optout.patch new file mode 100644 index 0000000..a0140f2 --- /dev/null +++ b/scripts/patches/glm53-compile-03-fullgraph-optout.patch @@ -0,0 +1,64 @@ +# glm53-compile-03-fullgraph-optout.patch — let torch.compile split the graph instead +# of failing on a break. +# +# STATUS: WIP — third of three patches in the "compile the GLM trunk" changelist. +# With 01+02+03 the model TRACES, then hits the NEXT blocker (below). The set does not +# yet boot the engine; nothing in scripts/ mounts it. +# +# THE BREAK: with 01+02 mounted the worker dies with +# +# RuntimeError: Graph breaks are not supported with aot compile. +# Please use torch.compile(fullgraph=True). +# +# because vllm/compilation/wrapper.py calls torch.compile(..., fullgraph=True, ...) and +# this build has AOT compile on by default (VLLM_USE_AOT_COMPILE). The GLM trunk cannot +# satisfy fullgraph: its indexer metadata path syncs to the host — +# models/glm5next/amd/sparse_indexer.py:263 int(positions[...].max().item()) + 1 +# models/glm5next/amd/sparse_indexer.py:419 lmax = int(decode_metadata.decode_lens.max().item()) +# — and a .item() inside a captured graph is not expressible. +# +# WHAT WAS VERIFIED (in-image, torch 2.11.0+rocm10.0, torch.compile(backend="eager")): +# +# fullgraph=True suppress_errors=False -> RAISE Unsupported +# fullgraph=True suppress_errors=True -> RAISE Unsupported <-- suppress_errors +# does NOT cover breaks +# fullgraph=False suppress_errors=False -> ok (traceable segments compiled by +# inductor, the rest runs eagerly; +# numerics unchanged) +# +# So breaks must be TOLERATED, not suppressed: the traceable mass still fuses. This +# patch makes the fullgraph requirement opt-out via VLLM_COMPILE_FULLGRAPH=0 (default 1, +# so every other model keeps stock behaviour), and the run also needs +# VLLM_USE_AOT_COMPILE=0 because AOT compile refuses breaks outright. +# +# TARGET: vllm/compilation/wrapper.py (inside the pinned image) +# base sha256 969961dca41c78385212e985cfc0c57ef067f4a2d7490c73c658c600a1b95cac +# patched sha256 732363bb1a1367772f13c60a655c678e7e2e9e56186677b14238f5e84cac31b3 +# (the file already imports os; the patch does not add a second import) +# +# NEXT BLOCKER — NOT FIXED HERE. With 01+02+03 and +# R4D_GLM_COMPILE=1 VLLM_COMPILE_FULLGRAPH=0 VLLM_USE_AOT_COMPILE=0 the trunk traces +# ("Dynamo bytecode transform time: 1.08 s") and then the worker dies during profile_run: +# +# KeyError: 'in_proj_qkvbfg_a' +# vllm/models/glm5next/common/model.py:456 in Glm5NextDecoderLayer.forward +# +# `in_proj_qkvbfg_a` is the KDA layer's merged column-parallel projection +# (models/glm5next/common/kda.py:217, called at :407). The key is looked up in a +# dict/mapping that the EAGER path never reaches, so compiling the trunk exposes a +# latent bug in the mHC/KDA wiring rather than a dynamo limitation. Fix that and +# whatever follows before this changelist can boot — then re-measure: the prize is the +# 44 % pointwise mass (see 01), and the frozen-probe log line is the signal that 02 is +# live. +--- a/vllm/compilation/wrapper.py ++++ b/vllm/compilation/wrapper.py +@@ fullgraph opt-out (env VLLM_COMPILE_FULLGRAPH=0) @@ ++# FORK-LOCAL (patch: fullgraph opt-out). torch.compile(fullgraph=True) turns any ++# dynamo graph break into a hard failure; suppress_errors does NOT cover breaks ++# (verified in-image, torch 2.11+rocm10.0). With VLLM_COMPILE_FULLGRAPH=0 the ++# traceable segments are still compiled by inductor and the untraceable ones run ++# eagerly, which is what a model with device-sync metadata (the GLM-5.3-Flash ++# indexer) needs. Default 1 keeps stock behaviour for every other model. ++_FULLGRAPH = os.environ.get("VLLM_COMPILE_FULLGRAPH", "1") == "1" +- fullgraph=True, ++ fullgraph=_FULLGRAPH, diff --git a/serverAppConfigs-docs/LLM/frank-primary-glm53-flash-w4a16.md b/serverAppConfigs-docs/LLM/frank-primary-glm53-flash-w4a16.md index f6e3b68..b60e578 100644 --- a/serverAppConfigs-docs/LLM/frank-primary-glm53-flash-w4a16.md +++ b/serverAppConfigs-docs/LLM/frank-primary-glm53-flash-w4a16.md @@ -1,7 +1,7 @@ --- type: runbook title: frank-primary — GLM-5.3-Flash W4A16 int4 (tcclaviger, 2x R9700) -description: VERIFIED 2026-09-30 by boot probe, bench and live expert-cache telemetry. frank-primary cut over from Qwen3.8-Flash-Next-Uncensored-FP8 + MTP4 to Intel/GLM-5.3-Flash-W4A16-AutoRound, the first payload this fleet serves through the fork's W4A16 expert-offload path (INC scheme -> R4dW4a16MoEMethod, libr4d grouped W4A16 GEMM). Measured decode 9.9-10.0 tok/s and prefill 918 tok/s at 131,072 ctx. THE 35 tok/s TARGET IS NOT MET and this page shows why it is out of reach: the step is bounded by the fork's expert-gather read path at ~8.2 GB/s effective against a PCIe link that delivers 22.74 GB/s, with the cache at its measured hit-rate ceiling (0.53-0.66 per layer) from an arena that cannot grow past ~15 GiB under a 73.64 GiB/rank expert set. +description: VERIFIED 2026-09-30 by boot probe, bench, live expert-cache telemetry AND a torch-profiler kernel-time window. frank-primary cut over from Qwen3.8-Flash-Next-Uncensored-FP8 + MTP4 to Intel/GLM-5.3-Flash-W4A16-AutoRound, the first payload this fleet serves through the fork's W4A16 expert-offload path (INC scheme -> R4dW4a16MoEMethod, libr4d grouped W4A16 GEMM). Measured decode 9.9-10.0 tok/s and prefill 918 tok/s at 131,072 ctx. THE 35 tok/s TARGET IS NOT MET, and the profiler window FALSIFIED the first explanation: the expert gather is 1.05 % of device time, not the bound. 44 % of the 93 ms/token budget is pointwise glue (1,428 elementwise launches per token) because the fork's GLM trunk is NOT torch.compiled while its own Qwen trunk is; the second-largest lever, the fused router, is binary-blocked at 288 experts. The compile changelist is staged but incomplete (three patches, one blocker left). tags: - dankserver - r9700 @@ -22,12 +22,16 @@ timestamp: 2026-09-30 > `GLM-5.3-Flash` at int4 on two R9700s with expert offload, and how fast is it? > **Answer.** Yes — it serves, with vision, tool calls and 131,072 ctx. It runs > at **9.9-10.0 tok/s decode / 918 tok/s prefill**, and **35 tok/s is not -> reachable** with this payload on this hardware. The bound is not the model, -> the quant, or the link: it is the fork's expert-gather read path, and it is -> measured, not inferred. +> reachable** on this payload as the fork ships it. The bound is **not** the +> model, the quant, the link, or the expert gather (measured: 1.05 % of device +> time). It is the **unfused pointwise glue of an uncompiled trunk** — 44 % of a +> 93 ms per-token budget spread over ~1,428 elementwise launches per token — +> because the fork compiles its own Qwen trunk and not its GLM one. See +> [The measured profile](#the-measured-profile-and-the-real-wall). -Related: [[dankserver-qwen38-fp8-expert-offload-tuning]] (the same fork, the -same hardware, the same gather-path wall, found first on qwen), +Related: [[dankserver-qwen38-fp8-expert-offload-tuning]] (the same fork, the same +hardware — its conclusion was drawn from throughput arithmetic and is superseded +by this page's kernel-time measurement), [[dankserver-frontier-moe-feasibility]] (the earlier fit analysis), [[VLLM Index]], [[LLM/index|LLM fleet]]. @@ -51,7 +55,7 @@ graph LR D --> E[ExpertLRUCache
2,383 VRAM slots / rank] D --> F[73.64 GiB pinned host rows / rank] E --> G[decode step] - F -->|8.2 GB/s effective| G + F -->|host misses
1.05% of device time| G ``` ## Why this payload and not another @@ -90,33 +94,142 @@ Measured on the deployed engine, 2026-09-30 (`bench_glm.py`, real prose, greedy) Decode is flat across context and across every arena change — the signature of a per-step cost that is not KV- or residency-driven at the margin. -### The arithmetic that bounds it +### The measured profile, and the real wall -Per decode step, per rank: 42 MoE layers x 8 routed experts x 6.234 MiB = -**2.045 GiB of expert weights**, which must either be in VRAM or cross PCIe. +**This section replaces an earlier explanation, which the measurement falsified.** +The first reading of this page bounded the step with throughput arithmetic: 42 MoE +layers x 8 experts x 6.234 MiB = 2.045 GiB per step per rank, 0.82 GiB of it +missed at a measured 0.53-0.66 hit rate, giving "0.82 GiB in ~100 ms = 8.2 GB/s = +the gather is the wall". That arithmetic was circular — it *assumed* the gather +consumed the whole step — and it is wrong. The engine, asked directly, disagrees: -| hit rate | miss traffic | step @ 8.2 GB/s | decode | +A torch-profiler window (`--profiler-config`, `POST /start_profile` / +`stop_profile`, 496 decode tokens at the deployed shape, rank0 trace parsed for +device-activity time) gives the true split. Total device kernel time **46.18 s / +496 tokens = ~93 ms per token** against a measured ~104 ms step (9.6 tok/s with +the profiler attached): + +| ms | % | launches | kernel | | --- | --- | --- | --- | -| 0.60 (measured) | 0.82 GiB | ~105 ms | **9.5-10.0 tok/s** | -| 0.70 | 0.61 GiB | ~78 ms | 12.8 tok/s | -| 0.89 (what 35 tok/s needs) | 0.23 GiB | ~29 ms | 35 tok/s | +| 16,074 | **34.8** | 708,500 | `vectorized_elementwise_kernel` (22.7 us each) | +| 4,602 | 10.0 | 42,000 | `r4d_gemm_moe_w4a16_nt_b16_m64` — the MoE GEMM | +| 3,108 | 6.7 | 174,708 | `elementwise_kernel_manual_unroll` | +| 2,943 | 6.4 | 138,252 | `wvSplitK_hf_sml_` — split-K GEMM (projections / head) | +| 2,829 | 6.1 | 83,160 | router: `sbtopk::gatherTopK` + `warpMergeSortTopK` + `bitonicSortKVInPlace` (1,208 + 1,051 + 559 ms; the 1,051 row is not in the list below) | +| 3,015 | 6.5 | 133,920 | `r4d_mhc_pre_finish/pre_partial/post` — hyper-connections | +| 2,782 | 6.0 | 127,916 | inductor `triton_poi_*` / `triton_per_*` (the fusions that DO happen) | +| **487** | **1.05** | 20,664 | **`r4d_moe_lru_gather_kernel`** — the accused | +| 250 | 0.54 | 20,832 | `_moe_lru_tele_kernel` — the telemetry that measured it (TELEM was on) | -Two independent measurements pin the missed term: +So the gather is **1.05 %** of device time, and `r4d_moe_lru_fused_kernel` (the +LRU *manager*, not the copy) is a further 1,599 ms / 3.5 %. Residency work and the +PCIe link are not the story: 44 % of the budget is **pointwise glue** — bf16<->fp32 +conversions, adds, `pow`/`rsqrt` from fp32 RMSNorm, `launch_clamp_scale` quant +passes, `reduce_kernel` means — spread over **~1,428 elementwise launches per +token** (3,734 kernel launches per token in total). The GPU is busy 86 % of the +wall clock, so this is real device work, not launch-gap overhead. -- **Live expert-cache telemetry** (`R4D_LRU_TELEMETRY=1`, profiler window) - gives the hit rate directly — the LRU is *working*: - `layers.34: slots 56 (pinned 0, tier 0) hit 0.6279 routed 2002 installs 543 - read-through 202` … `layers.44: hit 0.5335`. Per-layer range **0.53-0.66**. -- **The link delivers 22.74 GB/s** at the engine's own slab size (H2D probe, - pinned, 2.345 MiB). The step moves 0.82 GiB in ~100 ms = **8.2 GB/s — - 36% of the wire**, matching the fork's own measured 6.5-9.5 GB/s. +Two consequences that kill three whole families of planned work: -So the step is bounded by the fork's gather read path, and the only other lever -is the hit rate — which is capped by the arena. A 14.51 GiB arena holds 20% of a -73.64 GiB expert set; the measured hit rate beats random by ~3x thanks to -routing locality, but it saturates: growing the arena 13.08 → 14.51 GiB -(+9%) changed decode by **nothing** (10.03 → 9.88 tok/s). Reaching 0.89 would -need roughly 4.5x the per-layer slot count — an arena of well over 100 GiB. +- **Spec decode cannot help this shape, and the MTP measurement was not an + accident.** Pointwise cost scales with the token count, so a draft batch that + verifies 4 tokens pays ~4x the glue for the same per-token output — which is + exactly why MTP4 measured "acceptance 3.76-4.90, decode unchanged". Same reason + the gather was never the bound. +- **Arena size and hit rate are not levers.** Confirmed twice: `--expert-precise-memory` + (arena 13.08 -> 14.51 GiB) moved decode by nothing, and the gather is 1 %. + +## Fork work: the compile changelist (WIP, does not boot yet) + +The fork compiles its own flagship trunk — `vllm/models/qwen4_exp/amd/model.py` +carries `@support_torch_compile` — and **not** its GLM trunk. The engine says so on +every boot: + +``` +Cudagraph mode FULL_AND_PIECEWISE requires piecewise capture, but the loaded model +provides neither a compiled submodule nor breakable CUDA graphs. +Overriding to FULL_DECODE_ONLY. +``` + +Inductor fusion is therefore the only lever that touches the 44 % pointwise mass. +Three patches are staged in `scripts/patches/glm53-compile-0*.patch`, each carrying +its own evidence header, base/patched sha256 and the next blocker: + +| patch | target | fixes | +| --- | --- | --- | +| `-01-model-decorator` | `models/glm5next/common/model.py` | decorates `Glm5NextModel` (env `R4D_GLM_COMPILE`, default on) | +| `-02-trace-safe-probes` | `model_executor/kernels/r4d_lib.py` | freezes 10 capability probes that do `os.path.exists`/`.so` lookups **inside the traced forward** | +| `-03-fullgraph-optout` | `compilation/wrapper.py` | makes `fullgraph=True` opt-out (env `VLLM_COMPILE_FULLGRAPH=0`) | + +What the boots established, in order (each one ~7 min): + +1. **01 alone dies** on the first cold probe call inside the traced region: + `Dynamo does not know how to trace the builtin 'posix.stat'` — + `r4d_mhc_supported -> r4d_lib.r4d_mhc_enabled -> available -> os.path.exists`. + **`functools.lru_cache` does not protect these probes**: dynamo re-traces the + wrapped body to build its guard even when the cache is warm (it warns about + exactly this), verified in-image — a *warm* lru_cache-calling function still + fails `fullgraph=True`, while a plain module-global read compiles. +2. **01+02 get past it** (`libr4d capability probes frozen: 10 warm [...]` in all + four processes) and die on + `RuntimeError: Graph breaks are not supported with aot compile`, because this + build runs AOT compile by default and AOT refuses breaks. +3. **01+02+03 with `VLLM_COMPILE_FULLGRAPH=0 VLLM_USE_AOT_COMPILE=0` traces the + trunk successfully** (`Dynamo bytecode transform time: 1.08 s`) and then dies in + `profile_run` with `KeyError: 'in_proj_qkvbfg_a'` at + `glm5next/common/model.py:456` (the KDA layer's merged projection, + `kda.py:217`) — a dict/registry lookup keyed by attribute name that the eager + path never reaches, i.e. **a latent bug the compile exposes, not a dynamo + limit**. + +That last row is the state of the work: the remaining blocker is a real bug in the +fork's mHC/KDA wiring, and it must be fixed (plus whatever follows it) before the +changelist can boot. Two verified facts make the payoff concrete: `suppress_errors` +does **not** cover graph breaks (only `fullgraph=False` does), and the GLM trunk can +never satisfy `fullgraph` because its indexer metadata syncs to host +(`glm5next/amd/sparse_indexer.py:263,419`: `int(...max().item())`). + +### Two structural constraints on any further fork work + +- **No new HIP kernels are possible.** The image ships `r4d.so` as a binary with no + sources. The obvious second lever — the 9 % router — is gated + `num_experts in {128, 256, 512}` in `router/r4d_route_router.py` and the binary + contains only `r4d_moe_route_kernel<128|256|512>` instantiations. GLM has **288** + experts, so it falls back to the generic sort-based top-k and **cannot be moved to + the fused router without rebuilding libr4d**. Only Python/config/inductor changes + (and runtime-JIT Triton) are available. +- **`--kv-cache-dtype fp8` is a dead end for capacity too.** The plan's KV reserve + is dtype-blind (1.76 GiB at 131,072 either way), so fp8 buys no arena and no + window; it only removes the only MLA decode path gfx12 has (trap 1 above). + +### DFlash2 for GLM — what the drafter actually is + +`incoai/GLM-5.3-Flash-DFlash2` is **not** a GLM-shaped drafter and needs no MLA/DSA +port: its `config.json` declares `architectures: ["DFlash2DraftModel"]` and +`model_type: "qwen3"` — 5 dense layers, hidden 4096, 32 heads / 8 kv heads, +head_dim 128, sliding attention (window 2048), `is_causal: false`, +`use_cache: false`, `dflash_config` = block 8, conv group 16, selector rank 256 / +top-k 16, and **`target_layer_ids: [5, 14, 24, 33, 42]` against +`num_target_layers: 45`**. + +The fork already maps that architecture: +`registry.py:636 "DFlash2DraftModel": ("qwen3_dflash2", "DFlash2Qwen3ForCausalLM")`, +and the spec method is `"dflash"` (`config/speculative.py:67`). So the port is +**target-side only**: the GLM trunk must expose auxiliary hidden states at those five +layer indices, the same interface `deepseek_v2.py` (also MLA+MoE) implements — +`set_aux_hidden_state_layers()` / `get_eagle3_aux_hidden_state_layers()` plus +capture at `idx in self.aux_hidden_state_layers` in the layer loop. `glm5next` +implements neither. + +Expected benefit is **not** assumed: DFlash2 amortizes a draft block (8 tokens) over +one verification pass, and per the profile above a verification pass costs pointwise +work proportional to tokens — the same reason MTP4 was flat. Measure it; do not +budget on it. + +Note also that the KV-cap patch's predecessor arithmetic in +[[dankserver-qwen38-fp8-expert-offload-tuning]] should be read the same way: its +"gather-bound" conclusion came from the same throughput arithmetic that this page's +profiler window has now falsified. ## Measured arms (all 2026-09-30, one boot each) diff --git a/serverAppConfigs-docs/log.md b/serverAppConfigs-docs/log.md index 20773c7..b6f9d7b 100644 --- a/serverAppConfigs-docs/log.md +++ b/serverAppConfigs-docs/log.md @@ -20,17 +20,58 @@ - Measured on the deployed engine: **decode 9.9-10.0 tok/s, prefill 918 tok/s** (warm, 2.6k prompt), KV 138,561 tokens; arena 14.51 GiB = 2,383 slots per rank over 12,096 host-backed experts (73.64 GiB pinned per rank). - - **The wall:** live expert-cache telemetry (`R4D_LRU_TELEMETRY=1` + a - `with_stack=false` profiler window) puts the cache hit rate at 0.53-0.66 per - layer; a step moves 0.82 GiB of misses in ~100 ms = **8.2 GB/s effective - against a link measured at 22.74 GB/s** — the same structural gather-read-path - gap the 2026-09-29 qwen campaign found. Growing the arena 13.08 -> 14.51 GiB - (+9%) changed decode by *nothing* (10.03 -> 9.88 tok/s), so residency stopped - being the binding term; 35 tok/s would need a 0.89 hit rate. + - **The wall — and a corrected one.** The first account of the shortfall + bounded the step with throughput arithmetic (a step moves 0.82 GiB of misses + in ~100 ms = "8.2 GB/s effective against a 22.74 GB/s link", needing a 0.89 + hit rate for 35 tok/s). **A torch-profiler kernel-time window falsified it** + (496 decode tokens, rank0 trace parsed for device-activity time): 46.18 s of + kernel time = ~93 ms/token, of which **44 % is pointwise glue** — + `vectorized_elementwise_kernel` alone 16.07 s over **708,500 launches (1,428 + per token)**, plus 3.11 s of `elementwise_kernel_manual_unroll` — against + **1.05 % for `r4d_moe_lru_gather_kernel`**. The MoE GEMM is 10 %, the split-K + GEMM 6.4 %, the router/top-k 8.9 %, the mHC kernels 6.5 %, and the inductor + fusions that DO happen 6 %. The real cause is that the fork compiles its own + Qwen trunk (`@support_torch_compile` in `models/qwen4_exp/amd/model.py`) and + **not** its GLM trunk, so every pointwise op is a separate eager kernel; the + engine says so at boot ("...provides neither a compiled submodule nor + breakable CUDA graphs. Overriding to FULL_DECODE_ONLY."). Growing the arena + 13.08 -> 14.51 GiB still changed nothing, consistent with a 1 % gather. + - **Fork work staged (compile changelist, WIP — does not boot yet).** + `scripts/patches/glm53-compile-01-model-decorator.patch`, + `-02-trace-safe-probes.patch`, `-03-fullgraph-optout.patch`, each with its own + evidence header and base/patched sha256. Boots established, in order: (1) + `functools.lru_cache` does NOT make a probe trace-safe — dynamo re-traces the + wrapped body even when warm, so `r4d_mhc_supported -> r4d_mhc_enabled -> + available -> os.path.exists` breaks the graph (a plain module-global read + compiles; verified in-image); (2) `suppress_errors` does NOT cover graph + breaks, only `fullgraph=False` does; (3) the GLM trunk can never satisfy + `fullgraph` because its indexer metadata syncs to host + (`glm5next/amd/sparse_indexer.py:263,419`), and AOT compile refuses breaks + outright. With all three the trunk **traces** (`Dynamo bytecode transform + time: 1.08 s`) and then dies on `KeyError: 'in_proj_qkvbfg_a'` in the + mHC/KDA path — a latent bug the compile exposes. That is the remaining + blocker. + - **No new HIP kernels are possible** (the image ships `r4d.so` with no + sources), which binary-blocks the second-largest lever: the fused router is + gated `num_experts in {128, 256, 512}` and only + `r4d_moe_route_kernel<128|256|512>` instantiations exist, while GLM has 288 + experts. `--kv-cache-dtype fp8` buys no capacity either (the plan's KV + reserve is dtype-blind), only removing gfx12's only MLA decode path. + - **DFlash2 for GLM is a target-side port, not a drafter port.** + `incoai/GLM-5.3-Flash-DFlash2` declares `architectures: ["DFlash2DraftModel"]` + and `model_type: "qwen3"` (5 dense layers, sliding attention, block 8, + `target_layer_ids [5, 14, 24, 33, 42]` of 45) and the fork **already maps it** + (`registry.py:636` -> `qwen3_dflash2.DFlash2Qwen3ForCausalLM`; spec method + `"dflash"`). What `glm5next` lacks is the target side: aux-hidden-state + capture (`set_aux_hidden_state_layers` / per-layer capture, as `deepseek_v2.py` + demonstrates). No speedup is budgeted: a verification pass pays pointwise cost + proportional to its token count. - Measured and rejected: **MTP4** (the head loads, arch resolves to `Glm5NextMTPModel`, acceptance is excellent at mean length 3.76-4.90 — but - decode was unchanged (9.6-11.2 vs 10.0) while prefill COLLAPSED 918 -> 255, - because a verification forward gathers the *union* of the batch's experts); + decode was unchanged (9.6-11.2 vs 10.0) while prefill COLLAPSED 918 -> 255. + The profiler window above explains it: the per-token cost is dominated by + pointwise glue that scales with the token count, so verifying a draft batch + pays ~N x that glue for the same N tokens of output); **fp8 KV** (disables the sparse MLA backend's rope-free Triton path, and aiter's MLA decode op is gated on CDNA>2, so the MTP draft step demands an op that cannot exist on gfx12); **NBATCH 8192** (boots, identical prefill).