frank-primary: pin down the compile blocker — a name-resolution KeyError in an inlined callee
Five runs of the changelist now reproduce the same single failure with breaks genuinely
tolerated (patch 01+02+03+04, VLLM_COMPILE_FULLGRAPH=0, VLLM_USE_AOT_COMPILE=0,
R4D_GLM_FUSE_QKV_RMSNORM=0): the trunk TRACES ('Dynamo bytecode transform time: 1.08 s')
and the boot dies on KeyError: 'in_proj_qkvbfg_a'.
What it is NOT, each ruled out by measurement rather than argument:
- not one of dynamo's four nn_modules lookups: /home/dank/dynamo-probe wraps three of
them and stayed silent over two runs, with the probe's install line confirmed in both
the API-server and worker processes;
- not a dynamo-internal failure: TORCHDYNAMO_VERBOSE=1 adds no internal stack trace;
- not a literal string lookup: the name appears tree-wide only as the KDA layer's
attribute (kda.py:217, used at :407) and in the load_weights shard table
(model.py:780-785); the four sites in the
quantization utils do compute exactly this bare key but every one is guarded
(), so none can raise it;
- not the pluggable-layer registry, which is keyed by class name.
What it IS: a plain Python KeyError raised inside the traced
Glm5NextDecoderLayer.forward, in a callee dynamo INLINED (frame chain
profile_run -> _dummy_run -> execute_model -> glm4_1v.py:2358 -> the compile wrapper ->
eval_frame.compile_wrapper -> Glm5NextModel.forward:748 -> module._call_impl ->
Glm5NextDecoderLayer.forward:456; the reported line is that method's def line, which is
how inlined frames surface).
Patch 03's header and the runbook now record the remaining instrument (TORCH_LOGS=+dynamo
and grep, or wrap the inlining candidates — the mHC CustomOps and the indexer op — with a
KeyError probe), and the two breaks that are real but non-fatal in this configuration
(patch 04's fused-norm launcher cell, patch 05's workspace acquisition).
tests/run.sh: 362 seams pass.
This commit is contained in:
@@ -59,12 +59,34 @@
|
||||
# GLM decoder layer. Whether the root cause is the fork's module tree (something
|
||||
# dynamo cannot register statically) or dynamo's own registration is NOT yet settled.
|
||||
#
|
||||
# NEXT DEBUG STEP: re-run with TORCHDYNAMO_VERBOSE=1 (torch offers it in the error
|
||||
# itself: "Set TORCHDYNAMO_VERBOSE=1 for the internal stack trace") to get the internal
|
||||
# frame that computed the name. NOTE the launcher gotcha that wasted one boot:
|
||||
# /home/dank/serve_glm_autoround.sh RESETS ENVARGS (it is initialized empty near the
|
||||
# top and only the COMPILE_PATCH branch appends), so an outer `ENVARGS=... bash ...`
|
||||
# is silently dropped — add the var inside the script or to the branch itself.
|
||||
# STATUS OF THAT BLOCKER (updated 2026-09-30, after the diagnostics below):
|
||||
# * TORCHDYNAMO_VERBOSE=1 adds NO internal stack trace -> it is a plain Python
|
||||
# KeyError raised in a callee dynamo INLINED into Glm5NextDecoderLayer.forward
|
||||
# (the reported line 456 is that method's `def` line, which is how inlined frames
|
||||
# surface).
|
||||
# * It is NOT raised from dynamo's four nn_modules lookups: the probe in
|
||||
# /home/dank/dynamo-probe wraps three of them and stayed silent over two runs with
|
||||
# the probe confirmed installed in the API-server and worker processes.
|
||||
# * It is NOT a literal string lookup: the name appears tree-wide only as the KDA
|
||||
# layer's attribute (kda.py:217, used at :407) and in the load_weights shard table
|
||||
# (model.py:780-785). The four `proj_name = prefix.split(".")[-1]` sites in the
|
||||
# quantization utils do compute exactly this bare key, but each is guarded
|
||||
# (`if proj_name not in fused_mapping: return`), so none of them can raise it.
|
||||
# * It reproduces on every run of this configuration (5 of 5).
|
||||
# REMAINING INSTRUMENT: name the inlined caller. Either `TORCH_LOGS=+dynamo` on a boot
|
||||
# and grep the log for the key (dynamo prints inlined frames with source), or wrap the
|
||||
# inlining candidates in that forward — the mHC CustomOps (hc_pre / hc_post /
|
||||
# hc_fused_post_pre in layers/mhc.py) and the indexer op — with a KeyError-reporting
|
||||
# probe. Until that names the caller this is a dynamo-side name-resolution failure.
|
||||
# NOTE the launcher gotcha that wasted a boot earlier: the scratch launcher now
|
||||
# preserves ENVARGS (`ENVARGS="${ENVARGS:-}"`), but it previously reset it, silently
|
||||
# dropping outer env vars.
|
||||
#
|
||||
# SECOND BLOCKER, KNOWN AND NOT FATAL IN THIS CONFIG: the indexer's in-forward
|
||||
# workspace acquisition (patch 05 makes it compiler-opaque) and the fused-norm
|
||||
# launcher cell (patch 04 removes the call) are real breaks, but with
|
||||
# VLLM_COMPILE_FULLGRAPH=0 they are tolerated — they only become fatal under
|
||||
# fullgraph=True, which this trunk can never satisfy anyway.
|
||||
--- a/vllm/compilation/wrapper.py
|
||||
+++ b/vllm/compilation/wrapper.py
|
||||
@@ -19,6 +19,15 @@
|
||||
|
||||
Reference in New Issue
Block a user