Comrade Chief Designer
bd8c72ff47
A torch-profiler kernel-time window (496 decode tokens, rank0 trace parsed for
device-activity time) on the deployed GLM-5.3-Flash shape falsifies this page's
own "the expert gather is the bound" explanation:
46.18 s of kernel time / 496 tokens = ~93 ms per token (actual step ~104 ms)
34.8 % vectorized_elementwise_kernel 708,500 launches (1,428/token)
10.0 % r4d_gemm_moe_w4a16_nt_b16_m64 (the MoE GEMM)
6.7 % elementwise_kernel_manual_unroll
6.4 % wvSplitK_hf_sml_ (split-K GEMM)
8.9 % router: sbtopk + warpMergeSortTopK + bitonicSortKVInPlace
6.5 % r4d_mhc_* (hyper-connections)
1.05 % r4d_moe_lru_gather_kernel <-- the retracted suspect
The earlier arithmetic was circular (it assumed the gather consumed the whole
step). The real bound is unfused pointwise glue: the fork compiles its own Qwen
trunk (support_torch_compile in models/qwen4_exp/amd/model.py) and not its GLM
trunk, so the engine boots with "provides neither a compiled submodule nor
breakable CUDA graphs. Overriding to FULL_DECODE_ONLY." That also explains why
MTP4 measured flat — pointwise cost scales with the token count, so spec decode
amortizes nothing here — and it retires arena/hit-rate tuning (the gather is 1 %).
Staged as documented patch artifacts (not wired into any serve script; the
engine does not yet boot with them) — scripts/patches/glm53-compile-0{1,2,3}*.patch:
-01 models/glm5next/common/model.py decorate Glm5NextModel (R4D_GLM_COMPILE)
-02 model_executor/kernels/r4d_lib.py freeze 10 capability probes that do
os.path.exists/.so lookups inside the traced forward. functools.lru_cache
does NOT protect them: dynamo re-traces the wrapped body even when warm
(verified in-image); a plain module-global read compiles.
-03 compilation/wrapper.py make fullgraph=True opt-out, because
suppress_errors does NOT cover graph breaks and this trunk can never
satisfy fullgraph (indexer metadata syncs to host — .item() at
glm5next/amd/sparse_indexer.py:263,419).
With all three the trunk traces (Dynamo bytecode transform time 1.08 s) and dies
on KeyError: in_proj_qkvbfg_a in the mHC/KDA path — a latent bug the compile
exposes, and the remaining blocker.
Also records two structural constraints: no new HIP kernels are possible (r4d.so
ships without sources), which binary-blocks the fused router at GLM's 288
experts; and DFlash2 for GLM is a TARGET-side port, not a drafter port
(incoai/GLM-5.3-Flash-DFlash2 is architectures ["DFlash2DraftModel"] +
model_type qwen3, already mapped at registry.py:636; glm5next lacks only the
aux-hidden-state plumbing deepseek_v2.py demonstrates).
Runbook frontmatter, intro, mermaid and wall section, AGENTS.md and the work log
corrected to match the measurement. tests/run.sh: 362 seams pass.
115 lines
5.1 KiB
Diff
115 lines
5.1 KiB
Diff
# glm53-compile-02-trace-safe-probes.patch — libr4d capability probes must not do I/O
|
|
# inside a torch.compile region.
|
|
#
|
|
# STATUS: WIP — second of three patches in the "compile the GLM trunk" changelist.
|
|
# Inert (but harmless) without 01. Not part of the deployed shape.
|
|
#
|
|
# THE BREAK (measured: with 01 mounted and R4D_GLM_COMPILE=1, the worker dies during
|
|
# profile_run, i.e. the first — cold — forward):
|
|
#
|
|
# Explanation: Dynamo does not know how to trace the builtin `posix.stat`
|
|
# ... r4d_mhc_supported -> r4d_lib.r4d_mhc_enabled -> available -> os.path.exists
|
|
#
|
|
# THE NON-OBVIOUS PART: functools.lru_cache does NOT protect these probes. Dynamo
|
|
# re-traces the wrapped body to build its guard even when the cache is warm — it warns
|
|
# about exactly this ("Dynamo detected a call to a `functools.lru_cache`-wrapped
|
|
# function ... Dynamo ignores the cache wrapper and directly traces the wrapped
|
|
# function"). Verified in-image, torch 2.11.0+rocm10.0, torch.compile(backend="eager"):
|
|
#
|
|
# cold lru_cache call -> RAISE Unsupported
|
|
# WARM lru_cache call -> RAISE Unsupported <-- warming is not enough
|
|
# plain module-global read -> compiles
|
|
# uncached os.path.exists -> RAISE Unsupported
|
|
#
|
|
# So each probe must resolve ONCE into a plain value and be read as a global. The set
|
|
# below is every no-argument probe / symbol lookup reachable from a forward pass,
|
|
# grep-verified call sites:
|
|
# model_executor/layers/fused_moe/experts/r4d_w4a16_moe.py:79 available()
|
|
# model_executor/layers/fused_moe/experts/r4d_w4a16_moe.py:81 has_w4a16_moe()
|
|
# model_executor/kernels/mhc/r4d_dsv4.py:16 r4d_mhc_enabled()
|
|
# v1/attention/ops/rocm_aiter_mla_sparse.py:694 r4d_indexer_enabled(), mqa_max_heads()
|
|
# v1/attention/ops/rocm_aiter_mla_sparse.py:1124 r4d_indexer_enabled()
|
|
# model_executor/layers/fused_moe/experts/r4d_mxfp4_moe.py:683 r4d_act_quant_enabled()
|
|
# model_executor/layers/fused_moe/experts/r4d_w4a16_moe.py _entry() moe_w4a16()/moe_w4a16_mt1() per MoE call
|
|
# (lib() is the transitive dependency of all of them.)
|
|
#
|
|
# TARGET: vllm/model_executor/kernels/r4d_lib.py (inside the pinned image)
|
|
# base sha256 b8dfeefb4acc72b67b0579a4ab0653ef5d48c40cc8649187d757c80ce6b9617a
|
|
# patched sha256 0751852e1ca813ad2bdb1da7bf10c605130e63e04ef9ec57e70b5f675a29c08e
|
|
#
|
|
# APPLY: append the block below to the module. R4D_FREEZE_PROBES=0 restores stock
|
|
# behaviour. Cost when nothing is compiled: ten one-time probes at import (measured:
|
|
# logged once per process, e.g. "libr4d capability probes frozen: 10 warm [...]").
|
|
#
|
|
# OBSERVED EFFECT: with 01+02 the worker gets past this break and reaches the next one
|
|
# (see 03) — dynamo reports "Dynamo bytecode transform time: 1.08 s" for the trunk.
|
|
--- a/vllm/model_executor/kernels/r4d_lib.py
|
|
+++ b/vllm/model_executor/kernels/r4d_lib.py
|
|
@@ append: trace-safe capability probes (env): R4D_FREEZE_PROBES=0 disables @@
|
|
+# FORK-LOCAL (patch: trace-safe capability probes). torch.compile traces the
|
|
+# BODY of a functools.lru_cache call even when the cache is warm, so a capability
|
|
+# probe called from a forward pass is a dynamo graph break. Measured 2026-09-30
|
|
+# on the GLM-5.3-Flash trunk under @support_torch_compile:
|
|
+# Dynamo does not know how to trace the builtin `posix.stat`
|
|
+# r4d_mhc_supported -> r4d_mhc_enabled -> available -> os.path.exists
|
|
+# Each probe below is wrapped to resolve ONCE into a memo and is warmed at import
|
|
+# (outside any tracing region), so a traced call is a dict read. The probe set is
|
|
+# every no-argument capability probe / symbol lookup reachable from a forward
|
|
+# pass: the W4A16 + mxfp4 MoE dispatch, the mHC op, the MLA sparse indexer gate
|
|
+# and the draft/lm-head guards.
|
|
+_PROBE_NAMES = (
|
|
+ "available",
|
|
+ "lib",
|
|
+ "has_w4a16_moe",
|
|
+ "moe_w4a16",
|
|
+ "moe_w4a16_mt1",
|
|
+ "has_mhc_dsv4",
|
|
+ "r4d_mhc_enabled",
|
|
+ "r4d_indexer_enabled",
|
|
+ "mqa_max_heads",
|
|
+ "r4d_act_quant_enabled",
|
|
+)
|
|
+
|
|
+
|
|
+def _freeze_probe(fn):
|
|
+ """Return fn bound to its first result. Result must be a plain value."""
|
|
+
|
|
+ memo = []
|
|
+
|
|
+ def probe(*args, **kwargs):
|
|
+ if not memo:
|
|
+ memo.append(fn(*args, **kwargs))
|
|
+ return memo[0]
|
|
+
|
|
+ return probe
|
|
+
|
|
+
|
|
+def _freeze_probes() -> list[str]:
|
|
+ frozen = []
|
|
+ for name in _PROBE_NAMES:
|
|
+ fn = globals().get(name)
|
|
+ if fn is None:
|
|
+ continue
|
|
+ # Only the fast branch of a bound probe is traced; an empty memo means the
|
|
+ # first call still does I/O, so populate it here as well.
|
|
+ globals()[name] = _freeze_probe(fn)
|
|
+ frozen.append(name)
|
|
+ return frozen
|
|
+
|
|
+
|
|
+from vllm.logger import init_logger as _init_logger
|
|
+
|
|
+_probe_log = _init_logger(__name__)
|
|
+
|
|
+if os.environ.get("R4D_FREEZE_PROBES", "1") == "1":
|
|
+ _warmed, _failed = [], []
|
|
+ for _n in _freeze_probes():
|
|
+ try:
|
|
+ globals()[_n]()
|
|
+ _warmed.append(_n)
|
|
+ except Exception: # noqa: BLE001 -- probe unavailable here; recompute lazily
|
|
+ _failed.append(_n)
|
|
+ _probe_log.info("libr4d capability probes frozen: %d warm %s%s",
|
|
+ len(_warmed), _warmed,
|
|
+ f" (deferred: {_failed})" if _failed else "")
|