frank-primary: profile the GLM decode step and stage the compile changelist
A torch-profiler kernel-time window (496 decode tokens, rank0 trace parsed for
device-activity time) on the deployed GLM-5.3-Flash shape falsifies this page's
own "the expert gather is the bound" explanation:
46.18 s of kernel time / 496 tokens = ~93 ms per token (actual step ~104 ms)
34.8 % vectorized_elementwise_kernel 708,500 launches (1,428/token)
10.0 % r4d_gemm_moe_w4a16_nt_b16_m64 (the MoE GEMM)
6.7 % elementwise_kernel_manual_unroll
6.4 % wvSplitK_hf_sml_ (split-K GEMM)
8.9 % router: sbtopk + warpMergeSortTopK + bitonicSortKVInPlace
6.5 % r4d_mhc_* (hyper-connections)
1.05 % r4d_moe_lru_gather_kernel <-- the retracted suspect
The earlier arithmetic was circular (it assumed the gather consumed the whole
step). The real bound is unfused pointwise glue: the fork compiles its own Qwen
trunk (support_torch_compile in models/qwen4_exp/amd/model.py) and not its GLM
trunk, so the engine boots with "provides neither a compiled submodule nor
breakable CUDA graphs. Overriding to FULL_DECODE_ONLY." That also explains why
MTP4 measured flat — pointwise cost scales with the token count, so spec decode
amortizes nothing here — and it retires arena/hit-rate tuning (the gather is 1 %).
Staged as documented patch artifacts (not wired into any serve script; the
engine does not yet boot with them) — scripts/patches/glm53-compile-0{1,2,3}*.patch:
-01 models/glm5next/common/model.py decorate Glm5NextModel (R4D_GLM_COMPILE)
-02 model_executor/kernels/r4d_lib.py freeze 10 capability probes that do
os.path.exists/.so lookups inside the traced forward. functools.lru_cache
does NOT protect them: dynamo re-traces the wrapped body even when warm
(verified in-image); a plain module-global read compiles.
-03 compilation/wrapper.py make fullgraph=True opt-out, because
suppress_errors does NOT cover graph breaks and this trunk can never
satisfy fullgraph (indexer metadata syncs to host — .item() at
glm5next/amd/sparse_indexer.py:263,419).
With all three the trunk traces (Dynamo bytecode transform time 1.08 s) and dies
on KeyError: in_proj_qkvbfg_a in the mHC/KDA path — a latent bug the compile
exposes, and the remaining blocker.
Also records two structural constraints: no new HIP kernels are possible (r4d.so
ships without sources), which binary-blocks the fused router at GLM's 288
experts; and DFlash2 for GLM is a TARGET-side port, not a drafter port
(incoai/GLM-5.3-Flash-DFlash2 is architectures ["DFlash2DraftModel"] +
model_type qwen3, already mapped at registry.py:636; glm5next lacks only the
aux-hidden-state plumbing deepseek_v2.py demonstrates).
Runbook frontmatter, intro, mermaid and wall section, AGENTS.md and the work log
corrected to match the measurement. tests/run.sh: 362 seams pass.
This commit is contained in:
Comrade Chief Designer
@@ -0,0 +1,67 @@
|
||||
# glm53-compile-01-model-decorator.patch — put the GLM-5.3-Flash trunk under torch.compile.
|
||||
#
|
||||
# STATUS: WIP — first of three patches in the "compile the GLM trunk" changelist. The
|
||||
# set does NOT yet boot the engine (see 03 for the next blocker), so it is NOT part of
|
||||
# the deployed frank-primary shape. Nothing in scripts/ mounts these files.
|
||||
#
|
||||
# WHY THIS IS THE RIGHT LEVER (measured 2026-09-30, not inferred):
|
||||
#
|
||||
# The fork's own flagship trunk — vllm/models/qwen4_exp/amd/model.py — carries
|
||||
# @support_torch_compile. vllm/models/glm5next/common/model.py does not, and the engine
|
||||
# says so on every boot:
|
||||
#
|
||||
# Cudagraph mode FULL_AND_PIECEWISE requires piecewise capture, but the loaded model
|
||||
# provides neither a compiled submodule nor breakable CUDA graphs.
|
||||
# Overriding to FULL_DECODE_ONLY.
|
||||
#
|
||||
# A torch-profiler window on the live endpoint (496 decode tokens, rank0, 3 x 160-token
|
||||
# generations at the deployed shape) shows what the missing compilation costs. Total
|
||||
# device kernel time 46.18 s / 496 tokens = ~93 ms per token, against a measured step
|
||||
# time of ~104 ms (9.6 tok/s with the profiler attached):
|
||||
#
|
||||
# 16.07 s 34.8 % 708,500 launches vectorized_elementwise_kernel (22.7 us each)
|
||||
# 4.60 s 10.0 % 42,000 r4d_gemm_moe_w4a16_nt_b16_m64 (the MoE GEMM)
|
||||
# 3.11 s 6.7 % 174,708 elementwise_kernel_manual_unroll
|
||||
# 2.94 s 6.4 % 138,252 wvSplitK_hf_sml_ (split-K GEMM)
|
||||
# 4.09 s 8.9 % 83,160 topk/router (sbtopk gatherTopK + warpMergeSortTopK
|
||||
# + bitonicSortKVInPlace)
|
||||
# 3.01 s 6.5 % 133,920 r4d_mhc_* (hyper-connection pre/post/finish)
|
||||
# 2.78 s 6.0 % 127,916 inductor triton_poi_* / triton_per_* (the only
|
||||
# fusions that DO happen)
|
||||
# 0.49 s 1.05% 20,664 r4d_moe_lru_gather_kernel <-- the suspect
|
||||
#
|
||||
# That last row retired the fleet's working hypothesis. Decode on this shape is NOT
|
||||
# expert-gather bound: the gather is 1 % of device time, the same order as the telemetry
|
||||
# kernel that was switched on to measure it. 44 % of the budget is pointwise glue —
|
||||
# bf16<->fp32 conversions, adds, pows/rsqrts from fp32 RMSNorm, clamp/scale quant
|
||||
# passes — spread over ~1,428 launches per token. Inductor fusion is the only lever
|
||||
# that touches that mass; the router is second at 9 % but is binary-blocked (the
|
||||
# libr4d route kernel exists only as r4d_moe_route_kernel<128|256|512> instantiations
|
||||
# and GLM has 288 experts, with no HIP sources shipped to rebuild).
|
||||
#
|
||||
# TARGET: vllm/models/glm5next/common/model.py (inside the pinned image)
|
||||
# base sha256 008eeaa2a07e091465319f86900b5a321d921238723ccbe22e6b9e971e1d2b28
|
||||
# patched sha256 6e9cb81b84a9a88bd41403ca404aa04c3b791899dd77d661c3f25ae686cecd40
|
||||
#
|
||||
# APPLY: `patch -p1` from the vllm package root, or bind-mount new/.../model.py over
|
||||
# site-packages. Then R4D_GLM_COMPILE=1 (default) enables it; =0 makes the decorator a
|
||||
# no-op, so the file can stay mounted for an A/B without a second artifact.
|
||||
# Requires 02 (trace-safe probes) — without it the worker dies during profile_run on
|
||||
# the first probe call inside the traced region.
|
||||
--- a/vllm/models/glm5next/common/model.py
|
||||
+++ b/vllm/models/glm5next/common/model.py
|
||||
@@ Glm5NextModel: add @support_torch_compile (env): R4D_GLM_COMPILE=0 disables @@
|
||||
+_GLM_COMPILE = os.environ.get("R4D_GLM_COMPILE", "1") == "1"
|
||||
+
|
||||
+
|
||||
+@support_torch_compile(
|
||||
+ dynamic_arg_dims={
|
||||
+ "input_ids": 0,
|
||||
+ "positions": -1,
|
||||
+ "intermediate_tensors": 0,
|
||||
+ "inputs_embeds": 0,
|
||||
+ },
|
||||
+ enable_if=lambda _cfg: _GLM_COMPILE,
|
||||
+)
|
||||
-class Glm5NextModel(nn.Module):
|
||||
+class Glm5NextModel(nn.Module): # decorated above
|
||||
@@ -0,0 +1,114 @@
|
||||
# glm53-compile-02-trace-safe-probes.patch — libr4d capability probes must not do I/O
|
||||
# inside a torch.compile region.
|
||||
#
|
||||
# STATUS: WIP — second of three patches in the "compile the GLM trunk" changelist.
|
||||
# Inert (but harmless) without 01. Not part of the deployed shape.
|
||||
#
|
||||
# THE BREAK (measured: with 01 mounted and R4D_GLM_COMPILE=1, the worker dies during
|
||||
# profile_run, i.e. the first — cold — forward):
|
||||
#
|
||||
# Explanation: Dynamo does not know how to trace the builtin `posix.stat`
|
||||
# ... r4d_mhc_supported -> r4d_lib.r4d_mhc_enabled -> available -> os.path.exists
|
||||
#
|
||||
# THE NON-OBVIOUS PART: functools.lru_cache does NOT protect these probes. Dynamo
|
||||
# re-traces the wrapped body to build its guard even when the cache is warm — it warns
|
||||
# about exactly this ("Dynamo detected a call to a `functools.lru_cache`-wrapped
|
||||
# function ... Dynamo ignores the cache wrapper and directly traces the wrapped
|
||||
# function"). Verified in-image, torch 2.11.0+rocm10.0, torch.compile(backend="eager"):
|
||||
#
|
||||
# cold lru_cache call -> RAISE Unsupported
|
||||
# WARM lru_cache call -> RAISE Unsupported <-- warming is not enough
|
||||
# plain module-global read -> compiles
|
||||
# uncached os.path.exists -> RAISE Unsupported
|
||||
#
|
||||
# So each probe must resolve ONCE into a plain value and be read as a global. The set
|
||||
# below is every no-argument probe / symbol lookup reachable from a forward pass,
|
||||
# grep-verified call sites:
|
||||
# model_executor/layers/fused_moe/experts/r4d_w4a16_moe.py:79 available()
|
||||
# model_executor/layers/fused_moe/experts/r4d_w4a16_moe.py:81 has_w4a16_moe()
|
||||
# model_executor/kernels/mhc/r4d_dsv4.py:16 r4d_mhc_enabled()
|
||||
# v1/attention/ops/rocm_aiter_mla_sparse.py:694 r4d_indexer_enabled(), mqa_max_heads()
|
||||
# v1/attention/ops/rocm_aiter_mla_sparse.py:1124 r4d_indexer_enabled()
|
||||
# model_executor/layers/fused_moe/experts/r4d_mxfp4_moe.py:683 r4d_act_quant_enabled()
|
||||
# model_executor/layers/fused_moe/experts/r4d_w4a16_moe.py _entry() moe_w4a16()/moe_w4a16_mt1() per MoE call
|
||||
# (lib() is the transitive dependency of all of them.)
|
||||
#
|
||||
# TARGET: vllm/model_executor/kernels/r4d_lib.py (inside the pinned image)
|
||||
# base sha256 b8dfeefb4acc72b67b0579a4ab0653ef5d48c40cc8649187d757c80ce6b9617a
|
||||
# patched sha256 0751852e1ca813ad2bdb1da7bf10c605130e63e04ef9ec57e70b5f675a29c08e
|
||||
#
|
||||
# APPLY: append the block below to the module. R4D_FREEZE_PROBES=0 restores stock
|
||||
# behaviour. Cost when nothing is compiled: ten one-time probes at import (measured:
|
||||
# logged once per process, e.g. "libr4d capability probes frozen: 10 warm [...]").
|
||||
#
|
||||
# OBSERVED EFFECT: with 01+02 the worker gets past this break and reaches the next one
|
||||
# (see 03) — dynamo reports "Dynamo bytecode transform time: 1.08 s" for the trunk.
|
||||
--- a/vllm/model_executor/kernels/r4d_lib.py
|
||||
+++ b/vllm/model_executor/kernels/r4d_lib.py
|
||||
@@ append: trace-safe capability probes (env): R4D_FREEZE_PROBES=0 disables @@
|
||||
+# FORK-LOCAL (patch: trace-safe capability probes). torch.compile traces the
|
||||
+# BODY of a functools.lru_cache call even when the cache is warm, so a capability
|
||||
+# probe called from a forward pass is a dynamo graph break. Measured 2026-09-30
|
||||
+# on the GLM-5.3-Flash trunk under @support_torch_compile:
|
||||
+# Dynamo does not know how to trace the builtin `posix.stat`
|
||||
+# r4d_mhc_supported -> r4d_mhc_enabled -> available -> os.path.exists
|
||||
+# Each probe below is wrapped to resolve ONCE into a memo and is warmed at import
|
||||
+# (outside any tracing region), so a traced call is a dict read. The probe set is
|
||||
+# every no-argument capability probe / symbol lookup reachable from a forward
|
||||
+# pass: the W4A16 + mxfp4 MoE dispatch, the mHC op, the MLA sparse indexer gate
|
||||
+# and the draft/lm-head guards.
|
||||
+_PROBE_NAMES = (
|
||||
+ "available",
|
||||
+ "lib",
|
||||
+ "has_w4a16_moe",
|
||||
+ "moe_w4a16",
|
||||
+ "moe_w4a16_mt1",
|
||||
+ "has_mhc_dsv4",
|
||||
+ "r4d_mhc_enabled",
|
||||
+ "r4d_indexer_enabled",
|
||||
+ "mqa_max_heads",
|
||||
+ "r4d_act_quant_enabled",
|
||||
+)
|
||||
+
|
||||
+
|
||||
+def _freeze_probe(fn):
|
||||
+ """Return fn bound to its first result. Result must be a plain value."""
|
||||
+
|
||||
+ memo = []
|
||||
+
|
||||
+ def probe(*args, **kwargs):
|
||||
+ if not memo:
|
||||
+ memo.append(fn(*args, **kwargs))
|
||||
+ return memo[0]
|
||||
+
|
||||
+ return probe
|
||||
+
|
||||
+
|
||||
+def _freeze_probes() -> list[str]:
|
||||
+ frozen = []
|
||||
+ for name in _PROBE_NAMES:
|
||||
+ fn = globals().get(name)
|
||||
+ if fn is None:
|
||||
+ continue
|
||||
+ # Only the fast branch of a bound probe is traced; an empty memo means the
|
||||
+ # first call still does I/O, so populate it here as well.
|
||||
+ globals()[name] = _freeze_probe(fn)
|
||||
+ frozen.append(name)
|
||||
+ return frozen
|
||||
+
|
||||
+
|
||||
+from vllm.logger import init_logger as _init_logger
|
||||
+
|
||||
+_probe_log = _init_logger(__name__)
|
||||
+
|
||||
+if os.environ.get("R4D_FREEZE_PROBES", "1") == "1":
|
||||
+ _warmed, _failed = [], []
|
||||
+ for _n in _freeze_probes():
|
||||
+ try:
|
||||
+ globals()[_n]()
|
||||
+ _warmed.append(_n)
|
||||
+ except Exception: # noqa: BLE001 -- probe unavailable here; recompute lazily
|
||||
+ _failed.append(_n)
|
||||
+ _probe_log.info("libr4d capability probes frozen: %d warm %s%s",
|
||||
+ len(_warmed), _warmed,
|
||||
+ f" (deferred: {_failed})" if _failed else "")
|
||||
@@ -0,0 +1,64 @@
|
||||
# glm53-compile-03-fullgraph-optout.patch — let torch.compile split the graph instead
|
||||
# of failing on a break.
|
||||
#
|
||||
# STATUS: WIP — third of three patches in the "compile the GLM trunk" changelist.
|
||||
# With 01+02+03 the model TRACES, then hits the NEXT blocker (below). The set does not
|
||||
# yet boot the engine; nothing in scripts/ mounts it.
|
||||
#
|
||||
# THE BREAK: with 01+02 mounted the worker dies with
|
||||
#
|
||||
# RuntimeError: Graph breaks are not supported with aot compile.
|
||||
# Please use torch.compile(fullgraph=True).
|
||||
#
|
||||
# because vllm/compilation/wrapper.py calls torch.compile(..., fullgraph=True, ...) and
|
||||
# this build has AOT compile on by default (VLLM_USE_AOT_COMPILE). The GLM trunk cannot
|
||||
# satisfy fullgraph: its indexer metadata path syncs to the host —
|
||||
# models/glm5next/amd/sparse_indexer.py:263 int(positions[...].max().item()) + 1
|
||||
# models/glm5next/amd/sparse_indexer.py:419 lmax = int(decode_metadata.decode_lens.max().item())
|
||||
# — and a .item() inside a captured graph is not expressible.
|
||||
#
|
||||
# WHAT WAS VERIFIED (in-image, torch 2.11.0+rocm10.0, torch.compile(backend="eager")):
|
||||
#
|
||||
# fullgraph=True suppress_errors=False -> RAISE Unsupported
|
||||
# fullgraph=True suppress_errors=True -> RAISE Unsupported <-- suppress_errors
|
||||
# does NOT cover breaks
|
||||
# fullgraph=False suppress_errors=False -> ok (traceable segments compiled by
|
||||
# inductor, the rest runs eagerly;
|
||||
# numerics unchanged)
|
||||
#
|
||||
# So breaks must be TOLERATED, not suppressed: the traceable mass still fuses. This
|
||||
# patch makes the fullgraph requirement opt-out via VLLM_COMPILE_FULLGRAPH=0 (default 1,
|
||||
# so every other model keeps stock behaviour), and the run also needs
|
||||
# VLLM_USE_AOT_COMPILE=0 because AOT compile refuses breaks outright.
|
||||
#
|
||||
# TARGET: vllm/compilation/wrapper.py (inside the pinned image)
|
||||
# base sha256 969961dca41c78385212e985cfc0c57ef067f4a2d7490c73c658c600a1b95cac
|
||||
# patched sha256 732363bb1a1367772f13c60a655c678e7e2e9e56186677b14238f5e84cac31b3
|
||||
# (the file already imports os; the patch does not add a second import)
|
||||
#
|
||||
# NEXT BLOCKER — NOT FIXED HERE. With 01+02+03 and
|
||||
# R4D_GLM_COMPILE=1 VLLM_COMPILE_FULLGRAPH=0 VLLM_USE_AOT_COMPILE=0 the trunk traces
|
||||
# ("Dynamo bytecode transform time: 1.08 s") and then the worker dies during profile_run:
|
||||
#
|
||||
# KeyError: 'in_proj_qkvbfg_a'
|
||||
# vllm/models/glm5next/common/model.py:456 in Glm5NextDecoderLayer.forward
|
||||
#
|
||||
# `in_proj_qkvbfg_a` is the KDA layer's merged column-parallel projection
|
||||
# (models/glm5next/common/kda.py:217, called at :407). The key is looked up in a
|
||||
# dict/mapping that the EAGER path never reaches, so compiling the trunk exposes a
|
||||
# latent bug in the mHC/KDA wiring rather than a dynamo limitation. Fix that and
|
||||
# whatever follows before this changelist can boot — then re-measure: the prize is the
|
||||
# 44 % pointwise mass (see 01), and the frozen-probe log line is the signal that 02 is
|
||||
# live.
|
||||
--- a/vllm/compilation/wrapper.py
|
||||
+++ b/vllm/compilation/wrapper.py
|
||||
@@ fullgraph opt-out (env VLLM_COMPILE_FULLGRAPH=0) @@
|
||||
+# FORK-LOCAL (patch: fullgraph opt-out). torch.compile(fullgraph=True) turns any
|
||||
+# dynamo graph break into a hard failure; suppress_errors does NOT cover breaks
|
||||
+# (verified in-image, torch 2.11+rocm10.0). With VLLM_COMPILE_FULLGRAPH=0 the
|
||||
+# traceable segments are still compiled by inductor and the untraceable ones run
|
||||
+# eagerly, which is what a model with device-sync metadata (the GLM-5.3-Flash
|
||||
+# indexer) needs. Default 1 keeps stock behaviour for every other model.
|
||||
+_FULLGRAPH = os.environ.get("VLLM_COMPILE_FULLGRAPH", "1") == "1"
|
||||
- fullgraph=True,
|
||||
+ fullgraph=_FULLGRAPH,
|
||||
Reference in New Issue
Block a user