frank-primary: pin down the compile blocker — a name-resolution KeyError in an inlined callee

Five runs of the changelist now reproduce the same single failure with breaks genuinely
tolerated (patch 01+02+03+04, VLLM_COMPILE_FULLGRAPH=0, VLLM_USE_AOT_COMPILE=0,
R4D_GLM_FUSE_QKV_RMSNORM=0): the trunk TRACES ('Dynamo bytecode transform time: 1.08 s')
and the boot dies on KeyError: 'in_proj_qkvbfg_a'.

What it is NOT, each ruled out by measurement rather than argument:
 - not one of dynamo's four nn_modules lookups: /home/dank/dynamo-probe wraps three of
   them and stayed silent over two runs, with the probe's install line confirmed in both
   the API-server and worker processes;
 - not a dynamo-internal failure: TORCHDYNAMO_VERBOSE=1 adds no internal stack trace;
 - not a literal string lookup: the name appears tree-wide only as the KDA layer's
   attribute (kda.py:217, used at :407) and in the load_weights shard table
   (model.py:780-785); the four  sites in the
   quantization utils do compute exactly this bare key but every one is guarded
   (), so none can raise it;
 - not the pluggable-layer registry, which is keyed by class name.

What it IS: a plain Python KeyError raised inside the traced
Glm5NextDecoderLayer.forward, in a callee dynamo INLINED (frame chain
profile_run -> _dummy_run -> execute_model -> glm4_1v.py:2358 -> the compile wrapper ->
eval_frame.compile_wrapper -> Glm5NextModel.forward:748 -> module._call_impl ->
Glm5NextDecoderLayer.forward:456; the reported line is that method's def line, which is
how inlined frames surface).

Patch 03's header and the runbook now record the remaining instrument (TORCH_LOGS=+dynamo
and grep, or wrap the inlining candidates — the mHC CustomOps and the indexer op — with a
KeyError probe), and the two breaks that are real but non-fatal in this configuration
(patch 04's fused-norm launcher cell, patch 05's workspace acquisition).

tests/run.sh: 362 seams pass.

scripts/patches/glm53-compile-03-fullgraph-optout.patch

@@ -59,12 +59,34 @@
 # GLM decoder layer. Whether the root cause is the fork's module tree (something
 # dynamo cannot register statically) or dynamo's own registration is NOT yet settled.
 #
-# NEXT DEBUG STEP: re-run with TORCHDYNAMO_VERBOSE=1 (torch offers it in the error
+# STATUS OF THAT BLOCKER (updated 2026-09-30, after the diagnostics below):
-# itself: "Set TORCHDYNAMO_VERBOSE=1 for the internal stack trace") to get the internal
+#   * TORCHDYNAMO_VERBOSE=1 adds NO internal stack trace -> it is a plain Python
-# frame that computed the name. NOTE the launcher gotcha that wasted one boot:
+#     KeyError raised in a callee dynamo INLINED into Glm5NextDecoderLayer.forward
-# /home/dank/serve_glm_autoround.sh RESETS ENVARGS (it is initialized empty near the
+#     (the reported line 456 is that method's `def` line, which is how inlined frames
-# top and only the COMPILE_PATCH branch appends), so an outer `ENVARGS=... bash ...`
+#     surface).
-# is silently dropped — add the var inside the script or to the branch itself.
+#   * It is NOT raised from dynamo's four nn_modules lookups: the probe in
+#     /home/dank/dynamo-probe wraps three of them and stayed silent over two runs with
+#     the probe confirmed installed in the API-server and worker processes.
+#   * It is NOT a literal string lookup: the name appears tree-wide only as the KDA
+#     layer's attribute (kda.py:217, used at :407) and in the load_weights shard table
+#     (model.py:780-785). The four `proj_name = prefix.split(".")[-1]` sites in the
+#     quantization utils do compute exactly this bare key, but each is guarded
+#     (`if proj_name not in fused_mapping: return`), so none of them can raise it.
+#   * It reproduces on every run of this configuration (5 of 5).
+# REMAINING INSTRUMENT: name the inlined caller. Either `TORCH_LOGS=+dynamo` on a boot
+# and grep the log for the key (dynamo prints inlined frames with source), or wrap the
+# inlining candidates in that forward — the mHC CustomOps (hc_pre / hc_post /
+# hc_fused_post_pre in layers/mhc.py) and the indexer op — with a KeyError-reporting
+# probe. Until that names the caller this is a dynamo-side name-resolution failure.
+# NOTE the launcher gotcha that wasted a boot earlier: the scratch launcher now
+# preserves ENVARGS (`ENVARGS="${ENVARGS:-}"`), but it previously reset it, silently
+# dropping outer env vars.
+#
+# SECOND BLOCKER, KNOWN AND NOT FATAL IN THIS CONFIG: the indexer's in-forward
+# workspace acquisition (patch 05 makes it compiler-opaque) and the fused-norm
+# launcher cell (patch 04 removes the call) are real breaks, but with
+# VLLM_COMPILE_FULLGRAPH=0 they are tolerated — they only become fatal under
+# fullgraph=True, which this trunk can never satisfy anyway.
 --- a/vllm/compilation/wrapper.py
 +++ b/vllm/compilation/wrapper.py
 @@ -19,6 +19,15 @@

serverAppConfigs-docs/LLM/frank-primary-glm53-flash-w4a16.md

@@ -230,14 +230,43 @@ NOT executed, and the image's own `sitecustomize` is chained so its apport hook
 survives) and it prints, at the miss, the node target, the near-miss keys, and
 whether another live `OutputGraph` holds the key.
 
-**State of the changelist:** the trunk traces (`Dynamo bytecode transform time:
+**State of the changelist (2026-09-30, latest).** The trunk **traces**
-1.08 s`). With breaks genuinely tolerated the confirmed blocker is the dynamo
+(`Dynamo bytecode transform time: 1.08 s`) and the boot dies on the dynamo name-map
-name-map `KeyError: 'in_proj_qkvbfg_a'`. Two further breaks are known and worth
+`KeyError: 'in_proj_qkvbfg_a'`. Established about it, in order:
-removing regardless: the fused-norm launcher cell (patch 04 removes the call) and
+
-the indexer's in-forward workspace acquisition (patch 05 makes it
+- It is raised **inside the traced `Glm5NextDecoderLayer.forward`** — the frame chain
-compiler-opaque). Two verified facts bound the rest of the shape: `suppress_errors`
+  is `profile_run → _dummy_run → execute_model → glm4_1v.py:2358 → the compile wrapper
-does **not** cover graph breaks (only `fullgraph=False` does), and the GLM trunk can
+  → eval_frame.compile_wrapper → Glm5NextModel.forward:748 (layer(...)) → module
-never satisfy `fullgraph` because its indexer metadata syncs to host
+  _call_impl → Glm5NextDecoderLayer.forward:456 → KeyError`. `TORCHDYNAMO_VERBOSE=1`
+  adds **no** internal stack trace, i.e. it is a plain Python KeyError in the traced
+  frame, raised in a callee dynamo **inlined** (the reported line is the `def`, which
+  is how inlined frames surface).
+- It is **not** raised from any of dynamo's four `nn_modules` lookup sites:
+  `/home/dank/dynamo-probe` (a `sitecustomize.py` on `PYTHONPATH`,
+  `VLLM_DYNAMO_DUMP_ATTRS=1`) wraps three of them and stayed silent across two runs
+  with the probe confirmed installed in both the API-server and worker processes.
+- It is **not** a string-keyed lookup of a literal: `in_proj_qkvbfg_a` appears in the
+  whole tree only as the KDA layer's attribute (`kda.py:217`, used at `:407`) and in
+  the `load_weights` shard table (`model.py:780-785`). The four
+  `proj_name = prefix.split(".")[-1]` sites in the quantization utils do compute
+  exactly this bare key, but every one of them is guarded (`if proj_name not in
+  fused_mapping: return`), so they cannot raise it. The pluggable-layer registry is
+  keyed by class name, not module path.
+- It reproduces on **every** run of this configuration (5 of 5), and it is
+  independent of patches 04/05: with breaks genuinely tolerated, those remove breaks
+  that were never fatal in the first place — the KeyError is the only thing standing
+  between this changelist and a boot.
+
+**Next instrument, precisely:** the inlined caller has to be named. Two cheap ways:
+`TORCH_LOGS=+dynamo` on a boot and grep the log for the key (dynamo prints inlined
+frames with their source), or wrap the inlining candidates in that forward — the mHC
+CustomOps (`hc_pre` / `hc_post` / `hc_fused_post_pre`, `layers/mhc.py`) and the
+indexer op — with a probe that reports on `KeyError`. Until that names the caller, this
+is a dynamo-side name-resolution failure and not a fixable fork bug.
+
+Two verified facts bound the rest of the shape: `suppress_errors` does **not** cover
+graph breaks (only `fullgraph=False` does), and the GLM trunk can never satisfy
+`fullgraph` because its indexer metadata syncs to host
 (`glm5next/amd/sparse_indexer.py:263,419`: `int(...max().item())`) — both are dead
 fallbacks on the decode path that the fork author already replaced, so they are not
 the blockers they appear to be at first glance.