Comrade Chief Designer
fd864b5a42
Working the dyno name-map blocker one step at a time, with the tools to see it:
- All four patch artifacts are now genuine diff -u patches (they were prose + pseudo
hunks before, useless to patch(1)). Verified against the pinned image: each applies
clean and reproduces its recorded patched sha256 (6e9cb81b… / 0751852e… / 732363bb…
/ 84969993…).
- New: scripts/patches/glm53-compile-04-fused-qkv-rmsnorm-optout.patch — takes GLM's
MLA block off the fused-norm JIT kernel (R4D_GLM_FUSE_QKV_RMSNORM=0; the flag is
already a constructor arg glm5next sets at attention.py:583). Reason: in
models/common/ops/fused_qk_rmsnorm.py the kernel object is bound at line 223, AFTER
its use at line 211 — the launcher family (@kernel_launcher, 39 call sites) is what
breaks dynamo's break analysis.
- New diagnostics, both reusable and recorded: TORCH_LOGS=graph_breaks +
VLLM_USE_AOT_COMPILE=0 to name each break with its user stack, and
/home/dank/dynamo-probe (sitecustomize.py on PYTHONPATH, VLLM_DYNAMO_DUMP_ATTRS=1)
to print the dynamo name-map miss detail. Gotchas: a .pth on PYTHONPATH is NOT
executed, and the scratch launcher reset ENVARGS (now ${ENVARGS:-}), which silently
dropped env vars and wasted a boot.
Measurement discipline correction: I first recorded the 'Read uninitialized cell' and
'Unsupported method call' failures as fatal blockers. They are not established as
such — they appear only when TORCH_LOGS=graph_breaks is set, while a flag-free
break-tolerance run gets past break handling and dies on the name-map KeyError. The
likely mechanism is that graph_breaks logging makes dynamo build a break
*explanation*, and that analysis itself trips on the uninitialized cell. Patch 04's
header, the runbook and the work log now carry the confound explicitly rather than
the overclaim.
tests/run.sh: 362 seams pass.
95 lines
5.0 KiB
Diff
95 lines
5.0 KiB
Diff
# glm53-compile-03-fullgraph-optout.patch — let torch.compile split the graph instead
|
|
# of failing on a break.
|
|
#
|
|
# STATUS: WIP — third of three patches in the "compile the GLM trunk" changelist.
|
|
# With 01+02+03 the model TRACES, then hits the NEXT blocker (below). The set does not
|
|
# yet boot the engine; nothing in scripts/ mounts it.
|
|
#
|
|
# THE BREAK: with 01+02 mounted the worker dies with
|
|
#
|
|
# RuntimeError: Graph breaks are not supported with aot compile.
|
|
# Please use torch.compile(fullgraph=True).
|
|
#
|
|
# because vllm/compilation/wrapper.py calls torch.compile(..., fullgraph=True, ...) and
|
|
# this build has AOT compile on by default (VLLM_USE_AOT_COMPILE). The GLM trunk cannot
|
|
# satisfy fullgraph: its indexer metadata path syncs to the host —
|
|
# models/glm5next/amd/sparse_indexer.py:263 int(positions[...].max().item()) + 1
|
|
# models/glm5next/amd/sparse_indexer.py:419 lmax = int(decode_metadata.decode_lens.max().item())
|
|
# — and a .item() inside a captured graph is not expressible.
|
|
#
|
|
# WHAT WAS VERIFIED (in-image, torch 2.11.0+rocm10.0, torch.compile(backend="eager")):
|
|
#
|
|
# fullgraph=True suppress_errors=False -> RAISE Unsupported
|
|
# fullgraph=True suppress_errors=True -> RAISE Unsupported <-- suppress_errors
|
|
# does NOT cover breaks
|
|
# fullgraph=False suppress_errors=False -> ok (traceable segments compiled by
|
|
# inductor, the rest runs eagerly;
|
|
# numerics unchanged)
|
|
#
|
|
# So breaks must be TOLERATED, not suppressed: the traceable mass still fuses. This
|
|
# patch makes the fullgraph requirement opt-out via VLLM_COMPILE_FULLGRAPH=0 (default 1,
|
|
# so every other model keeps stock behaviour), and the run also needs
|
|
# VLLM_USE_AOT_COMPILE=0 because AOT compile refuses breaks outright.
|
|
#
|
|
# TARGET: vllm/compilation/wrapper.py (inside the pinned image)
|
|
# base sha256 969961dca41c78385212e985cfc0c57ef067f4a2d7490c73c658c600a1b95cac
|
|
# patched sha256 732363bb1a1367772f13c60a655c678e7e2e9e56186677b14238f5e84cac31b3
|
|
# (the file already imports os; the patch does not add a second import)
|
|
#
|
|
# NEXT BLOCKER — NOT FIXED HERE. With 01+02+03 and
|
|
# R4D_GLM_COMPILE=1 VLLM_COMPILE_FULLGRAPH=0 VLLM_USE_AOT_COMPILE=0 the trunk traces
|
|
# ("Dynamo bytecode transform time: 1.08 s") and then the worker dies during profile_run
|
|
# on the decode forward:
|
|
#
|
|
# KeyError: 'in_proj_qkvbfg_a'
|
|
# model.py:748 Glm5NextModel.forward hidden_states, residual, post, comb = layer(...)
|
|
# model.py:456 Glm5NextDecoderLayer.forward
|
|
#
|
|
# `in_proj_qkvbfg_a` is the KDA layer's merged column-parallel projection
|
|
# (models/glm5next/common/kda.py:217, called at :407) — and it is also a name in the
|
|
# load_weights shard table (model.py:780-785), i.e. a real submodule attribute name,
|
|
# not an arbitrary string.
|
|
#
|
|
# WHERE IT IS THROWN (established 2026-09-30 by reading the full traceback, since the
|
|
# report line is inside dynamo-rewritten code): torch/_dynamo keeps a name -> module map
|
|
# and looks it up by name — output_graph.py:1259 / :1384 register into `nn_modules`,
|
|
# :3000 does `self.nn_modules[node.target]` (a bare-name KeyError when absent), and
|
|
# :1225 documents stripping "_buffers[..]/_parameters[..]/_modules[..]" from those
|
|
# names. So the compiler is failing to RESOLVE A SUBMODULE BY NAME while tracing the
|
|
# GLM decoder layer. Whether the root cause is the fork's module tree (something
|
|
# dynamo cannot register statically) or dynamo's own registration is NOT yet settled.
|
|
#
|
|
# NEXT DEBUG STEP: re-run with TORCHDYNAMO_VERBOSE=1 (torch offers it in the error
|
|
# itself: "Set TORCHDYNAMO_VERBOSE=1 for the internal stack trace") to get the internal
|
|
# frame that computed the name. NOTE the launcher gotcha that wasted one boot:
|
|
# /home/dank/serve_glm_autoround.sh RESETS ENVARGS (it is initialized empty near the
|
|
# top and only the COMPILE_PATCH branch appends), so an outer `ENVARGS=... bash ...`
|
|
# is silently dropped — add the var inside the script or to the branch itself.
|
|
--- a/vllm/compilation/wrapper.py
|
|
+++ b/vllm/compilation/wrapper.py
|
|
@@ -19,6 +19,15 @@
|
|
|
|
logger = init_logger(__name__)
|
|
|
|
+
|
|
+# FORK-LOCAL (patch: fullgraph opt-out). torch.compile(fullgraph=True) turns any
|
|
+# dynamo graph break into a hard failure; suppress_errors does NOT cover breaks
|
|
+# (verified in-image, torch 2.11+rocm10.0). With VLLM_COMPILE_FULLGRAPH=0 the
|
|
+# traceable segments are still compiled by inductor and the untraceable ones run
|
|
+# eagerly, which is what a model with device-sync metadata (the GLM-5.3-Flash
|
|
+# indexer) needs. Default 1 keeps stock behaviour for every other model.
|
|
+_FULLGRAPH = os.environ.get("VLLM_COMPILE_FULLGRAPH", "1") == "1"
|
|
+
|
|
R = TypeVar("R")
|
|
P = ParamSpec("P")
|
|
|
|
@@ -146,7 +155,7 @@
|
|
with aot_context:
|
|
self._compiled_callable = torch.compile(
|
|
compiled_ptr,
|
|
- fullgraph=True,
|
|
+ fullgraph=_FULLGRAPH,
|
|
dynamic=False,
|
|
backend=backend,
|
|
options=options,
|