frank-primary: profile the GLM decode step and stage the compile changelist
A torch-profiler kernel-time window (496 decode tokens, rank0 trace parsed for
device-activity time) on the deployed GLM-5.3-Flash shape falsifies this page's
own "the expert gather is the bound" explanation:
46.18 s of kernel time / 496 tokens = ~93 ms per token (actual step ~104 ms)
34.8 % vectorized_elementwise_kernel 708,500 launches (1,428/token)
10.0 % r4d_gemm_moe_w4a16_nt_b16_m64 (the MoE GEMM)
6.7 % elementwise_kernel_manual_unroll
6.4 % wvSplitK_hf_sml_ (split-K GEMM)
8.9 % router: sbtopk + warpMergeSortTopK + bitonicSortKVInPlace
6.5 % r4d_mhc_* (hyper-connections)
1.05 % r4d_moe_lru_gather_kernel <-- the retracted suspect
The earlier arithmetic was circular (it assumed the gather consumed the whole
step). The real bound is unfused pointwise glue: the fork compiles its own Qwen
trunk (support_torch_compile in models/qwen4_exp/amd/model.py) and not its GLM
trunk, so the engine boots with "provides neither a compiled submodule nor
breakable CUDA graphs. Overriding to FULL_DECODE_ONLY." That also explains why
MTP4 measured flat — pointwise cost scales with the token count, so spec decode
amortizes nothing here — and it retires arena/hit-rate tuning (the gather is 1 %).
Staged as documented patch artifacts (not wired into any serve script; the
engine does not yet boot with them) — scripts/patches/glm53-compile-0{1,2,3}*.patch:
-01 models/glm5next/common/model.py decorate Glm5NextModel (R4D_GLM_COMPILE)
-02 model_executor/kernels/r4d_lib.py freeze 10 capability probes that do
os.path.exists/.so lookups inside the traced forward. functools.lru_cache
does NOT protect them: dynamo re-traces the wrapped body even when warm
(verified in-image); a plain module-global read compiles.
-03 compilation/wrapper.py make fullgraph=True opt-out, because
suppress_errors does NOT cover graph breaks and this trunk can never
satisfy fullgraph (indexer metadata syncs to host — .item() at
glm5next/amd/sparse_indexer.py:263,419).
With all three the trunk traces (Dynamo bytecode transform time 1.08 s) and dies
on KeyError: in_proj_qkvbfg_a in the mHC/KDA path — a latent bug the compile
exposes, and the remaining blocker.
Also records two structural constraints: no new HIP kernels are possible (r4d.so
ships without sources), which binary-blocks the fused router at GLM's 288
experts; and DFlash2 for GLM is a TARGET-side port, not a drafter port
(incoai/GLM-5.3-Flash-DFlash2 is architectures ["DFlash2DraftModel"] +
model_type qwen3, already mapped at registry.py:636; glm5next lacks only the
aux-hidden-state plumbing deepseek_v2.py demonstrates).
Runbook frontmatter, intro, mermaid and wall section, AGENTS.md and the work log
corrected to match the measurement. tests/run.sh: 362 seams pass.
This commit is contained in:
Comrade Chief Designer committed
1 parent
979a4308f4
commit
bd8c72ff47
6 files changed
+482
-46
No files matched your search
@@ -0,0 +1,67 @@
|
||||
# glm53-compile-01-model-decorator.patch — put the GLM-5.3-Flash trunk under torch.compile.
|
||||
#
|
||||
# STATUS: WIP — first of three patches in the "compile the GLM trunk" changelist. The
|
||||
# set does NOT yet boot the engine (see 03 for the next blocker), so it is NOT part of
|
||||
# the deployed frank-primary shape. Nothing in scripts/ mounts these files.
|
||||
#
|
||||
# WHY THIS IS THE RIGHT LEVER (measured 2026-09-30, not inferred):
|
||||
#
|
||||
# The fork's own flagship trunk — vllm/models/qwen4_exp/amd/model.py — carries
|
||||
# @support_torch_compile. vllm/models/glm5next/common/model.py does not, and the engine
|
||||
# says so on every boot:
|
||||
#
|
||||
# Cudagraph mode FULL_AND_PIECEWISE requires piecewise capture, but the loaded model
|
||||
# provides neither a compiled submodule nor breakable CUDA graphs.
|
||||
# Overriding to FULL_DECODE_ONLY.
|
||||
#
|
||||
# A torch-profiler window on the live endpoint (496 decode tokens, rank0, 3 x 160-token
|
||||
# generations at the deployed shape) shows what the missing compilation costs. Total
|
||||
# device kernel time 46.18 s / 496 tokens = ~93 ms per token, against a measured step
|
||||
# time of ~104 ms (9.6 tok/s with the profiler attached):
|
||||
#
|
||||
# 16.07 s 34.8 % 708,500 launches vectorized_elementwise_kernel (22.7 us each)
|
||||
# 4.60 s 10.0 % 42,000 r4d_gemm_moe_w4a16_nt_b16_m64 (the MoE GEMM)
|
||||
# 3.11 s 6.7 % 174,708 elementwise_kernel_manual_unroll
|
||||
# 2.94 s 6.4 % 138,252 wvSplitK_hf_sml_ (split-K GEMM)
|
||||
# 4.09 s 8.9 % 83,160 topk/router (sbtopk gatherTopK + warpMergeSortTopK
|
||||
# + bitonicSortKVInPlace)
|
||||
# 3.01 s 6.5 % 133,920 r4d_mhc_* (hyper-connection pre/post/finish)
|
||||
# 2.78 s 6.0 % 127,916 inductor triton_poi_* / triton_per_* (the only
|
||||
# fusions that DO happen)
|
||||
# 0.49 s 1.05% 20,664 r4d_moe_lru_gather_kernel <-- the suspect
|
||||
#
|
||||
# That last row retired the fleet's working hypothesis. Decode on this shape is NOT
|
||||
# expert-gather bound: the gather is 1 % of device time, the same order as the telemetry
|
||||
# kernel that was switched on to measure it. 44 % of the budget is pointwise glue —
|
||||
# bf16<->fp32 conversions, adds, pows/rsqrts from fp32 RMSNorm, clamp/scale quant
|
||||
# passes — spread over ~1,428 launches per token. Inductor fusion is the only lever
|
||||
# that touches that mass; the router is second at 9 % but is binary-blocked (the
|
||||
# libr4d route kernel exists only as r4d_moe_route_kernel<128|256|512> instantiations
|
||||
# and GLM has 288 experts, with no HIP sources shipped to rebuild).
|
||||
#
|
||||
# TARGET: vllm/models/glm5next/common/model.py (inside the pinned image)
|
||||
# base sha256 008eeaa2a07e091465319f86900b5a321d921238723ccbe22e6b9e971e1d2b28
|
||||
# patched sha256 6e9cb81b84a9a88bd41403ca404aa04c3b791899dd77d661c3f25ae686cecd40
|
||||
#
|
||||
# APPLY: `patch -p1` from the vllm package root, or bind-mount new/.../model.py over
|
||||
# site-packages. Then R4D_GLM_COMPILE=1 (default) enables it; =0 makes the decorator a
|
||||
# no-op, so the file can stay mounted for an A/B without a second artifact.
|
||||
# Requires 02 (trace-safe probes) — without it the worker dies during profile_run on
|
||||
# the first probe call inside the traced region.
|
||||
--- a/vllm/models/glm5next/common/model.py
|
||||
+++ b/vllm/models/glm5next/common/model.py
|
||||
@@ Glm5NextModel: add @support_torch_compile (env): R4D_GLM_COMPILE=0 disables @@
|
||||
+_GLM_COMPILE = os.environ.get("R4D_GLM_COMPILE", "1") == "1"
|
||||
+
|
||||
+
|
||||
+@support_torch_compile(
|
||||
+ dynamic_arg_dims={
|
||||
+ "input_ids": 0,
|
||||
+ "positions": -1,
|
||||
+ "intermediate_tensors": 0,
|
||||
+ "inputs_embeds": 0,
|
||||
+ },
|
||||
+ enable_if=lambda _cfg: _GLM_COMPILE,
|
||||
+)
|
||||
-class Glm5NextModel(nn.Module):
|
||||
+class Glm5NextModel(nn.Module): # decorated above
|
||||
Reference in new issue
Block a user