diff --git a/scripts/patches/glm53-compile-03-fullgraph-optout.patch b/scripts/patches/glm53-compile-03-fullgraph-optout.patch index 6350466..ecf263f 100644 --- a/scripts/patches/glm53-compile-03-fullgraph-optout.patch +++ b/scripts/patches/glm53-compile-03-fullgraph-optout.patch @@ -59,12 +59,34 @@ # GLM decoder layer. Whether the root cause is the fork's module tree (something # dynamo cannot register statically) or dynamo's own registration is NOT yet settled. # -# NEXT DEBUG STEP: re-run with TORCHDYNAMO_VERBOSE=1 (torch offers it in the error -# itself: "Set TORCHDYNAMO_VERBOSE=1 for the internal stack trace") to get the internal -# frame that computed the name. NOTE the launcher gotcha that wasted one boot: -# /home/dank/serve_glm_autoround.sh RESETS ENVARGS (it is initialized empty near the -# top and only the COMPILE_PATCH branch appends), so an outer `ENVARGS=... bash ...` -# is silently dropped — add the var inside the script or to the branch itself. +# STATUS OF THAT BLOCKER (updated 2026-09-30, after the diagnostics below): +# * TORCHDYNAMO_VERBOSE=1 adds NO internal stack trace -> it is a plain Python +# KeyError raised in a callee dynamo INLINED into Glm5NextDecoderLayer.forward +# (the reported line 456 is that method's `def` line, which is how inlined frames +# surface). +# * It is NOT raised from dynamo's four nn_modules lookups: the probe in +# /home/dank/dynamo-probe wraps three of them and stayed silent over two runs with +# the probe confirmed installed in the API-server and worker processes. +# * It is NOT a literal string lookup: the name appears tree-wide only as the KDA +# layer's attribute (kda.py:217, used at :407) and in the load_weights shard table +# (model.py:780-785). The four `proj_name = prefix.split(".")[-1]` sites in the +# quantization utils do compute exactly this bare key, but each is guarded +# (`if proj_name not in fused_mapping: return`), so none of them can raise it. +# * It reproduces on every run of this configuration (5 of 5). +# REMAINING INSTRUMENT: name the inlined caller. Either `TORCH_LOGS=+dynamo` on a boot +# and grep the log for the key (dynamo prints inlined frames with source), or wrap the +# inlining candidates in that forward — the mHC CustomOps (hc_pre / hc_post / +# hc_fused_post_pre in layers/mhc.py) and the indexer op — with a KeyError-reporting +# probe. Until that names the caller this is a dynamo-side name-resolution failure. +# NOTE the launcher gotcha that wasted a boot earlier: the scratch launcher now +# preserves ENVARGS (`ENVARGS="${ENVARGS:-}"`), but it previously reset it, silently +# dropping outer env vars. +# +# SECOND BLOCKER, KNOWN AND NOT FATAL IN THIS CONFIG: the indexer's in-forward +# workspace acquisition (patch 05 makes it compiler-opaque) and the fused-norm +# launcher cell (patch 04 removes the call) are real breaks, but with +# VLLM_COMPILE_FULLGRAPH=0 they are tolerated — they only become fatal under +# fullgraph=True, which this trunk can never satisfy anyway. --- a/vllm/compilation/wrapper.py +++ b/vllm/compilation/wrapper.py @@ -19,6 +19,15 @@ diff --git a/serverAppConfigs-docs/LLM/frank-primary-glm53-flash-w4a16.md b/serverAppConfigs-docs/LLM/frank-primary-glm53-flash-w4a16.md index 8810a32..aae6cd7 100644 --- a/serverAppConfigs-docs/LLM/frank-primary-glm53-flash-w4a16.md +++ b/serverAppConfigs-docs/LLM/frank-primary-glm53-flash-w4a16.md @@ -230,14 +230,43 @@ NOT executed, and the image's own `sitecustomize` is chained so its apport hook survives) and it prints, at the miss, the node target, the near-miss keys, and whether another live `OutputGraph` holds the key. -**State of the changelist:** the trunk traces (`Dynamo bytecode transform time: -1.08 s`). With breaks genuinely tolerated the confirmed blocker is the dynamo -name-map `KeyError: 'in_proj_qkvbfg_a'`. Two further breaks are known and worth -removing regardless: the fused-norm launcher cell (patch 04 removes the call) and -the indexer's in-forward workspace acquisition (patch 05 makes it -compiler-opaque). Two verified facts bound the rest of the shape: `suppress_errors` -does **not** cover graph breaks (only `fullgraph=False` does), and the GLM trunk can -never satisfy `fullgraph` because its indexer metadata syncs to host +**State of the changelist (2026-09-30, latest).** The trunk **traces** +(`Dynamo bytecode transform time: 1.08 s`) and the boot dies on the dynamo name-map +`KeyError: 'in_proj_qkvbfg_a'`. Established about it, in order: + +- It is raised **inside the traced `Glm5NextDecoderLayer.forward`** — the frame chain + is `profile_run → _dummy_run → execute_model → glm4_1v.py:2358 → the compile wrapper + → eval_frame.compile_wrapper → Glm5NextModel.forward:748 (layer(...)) → module + _call_impl → Glm5NextDecoderLayer.forward:456 → KeyError`. `TORCHDYNAMO_VERBOSE=1` + adds **no** internal stack trace, i.e. it is a plain Python KeyError in the traced + frame, raised in a callee dynamo **inlined** (the reported line is the `def`, which + is how inlined frames surface). +- It is **not** raised from any of dynamo's four `nn_modules` lookup sites: + `/home/dank/dynamo-probe` (a `sitecustomize.py` on `PYTHONPATH`, + `VLLM_DYNAMO_DUMP_ATTRS=1`) wraps three of them and stayed silent across two runs + with the probe confirmed installed in both the API-server and worker processes. +- It is **not** a string-keyed lookup of a literal: `in_proj_qkvbfg_a` appears in the + whole tree only as the KDA layer's attribute (`kda.py:217`, used at `:407`) and in + the `load_weights` shard table (`model.py:780-785`). The four + `proj_name = prefix.split(".")[-1]` sites in the quantization utils do compute + exactly this bare key, but every one of them is guarded (`if proj_name not in + fused_mapping: return`), so they cannot raise it. The pluggable-layer registry is + keyed by class name, not module path. +- It reproduces on **every** run of this configuration (5 of 5), and it is + independent of patches 04/05: with breaks genuinely tolerated, those remove breaks + that were never fatal in the first place — the KeyError is the only thing standing + between this changelist and a boot. + +**Next instrument, precisely:** the inlined caller has to be named. Two cheap ways: +`TORCH_LOGS=+dynamo` on a boot and grep the log for the key (dynamo prints inlined +frames with their source), or wrap the inlining candidates in that forward — the mHC +CustomOps (`hc_pre` / `hc_post` / `hc_fused_post_pre`, `layers/mhc.py`) and the +indexer op — with a probe that reports on `KeyError`. Until that names the caller, this +is a dynamo-side name-resolution failure and not a fixable fork bug. + +Two verified facts bound the rest of the shape: `suppress_errors` does **not** cover +graph breaks (only `fullgraph=False` does), and the GLM trunk can never satisfy +`fullgraph` because its indexer metadata syncs to host (`glm5next/amd/sparse_indexer.py:263,419`: `int(...max().item())`) — both are dead fallbacks on the decode path that the fork author already replaced, so they are not the blockers they appear to be at first glance.