diff --git a/scripts/patches/glm53-compile-03-fullgraph-optout.patch b/scripts/patches/glm53-compile-03-fullgraph-optout.patch index a0140f2..d4b49d8 100644 --- a/scripts/patches/glm53-compile-03-fullgraph-optout.patch +++ b/scripts/patches/glm53-compile-03-fullgraph-optout.patch @@ -38,18 +38,33 @@ # # NEXT BLOCKER — NOT FIXED HERE. With 01+02+03 and # R4D_GLM_COMPILE=1 VLLM_COMPILE_FULLGRAPH=0 VLLM_USE_AOT_COMPILE=0 the trunk traces -# ("Dynamo bytecode transform time: 1.08 s") and then the worker dies during profile_run: +# ("Dynamo bytecode transform time: 1.08 s") and then the worker dies during profile_run +# on the decode forward: # # KeyError: 'in_proj_qkvbfg_a' -# vllm/models/glm5next/common/model.py:456 in Glm5NextDecoderLayer.forward +# model.py:748 Glm5NextModel.forward hidden_states, residual, post, comb = layer(...) +# model.py:456 Glm5NextDecoderLayer.forward # # `in_proj_qkvbfg_a` is the KDA layer's merged column-parallel projection -# (models/glm5next/common/kda.py:217, called at :407). The key is looked up in a -# dict/mapping that the EAGER path never reaches, so compiling the trunk exposes a -# latent bug in the mHC/KDA wiring rather than a dynamo limitation. Fix that and -# whatever follows before this changelist can boot — then re-measure: the prize is the -# 44 % pointwise mass (see 01), and the frozen-probe log line is the signal that 02 is -# live. +# (models/glm5next/common/kda.py:217, called at :407) — and it is also a name in the +# load_weights shard table (model.py:780-785), i.e. a real submodule attribute name, +# not an arbitrary string. +# +# WHERE IT IS THROWN (established 2026-09-30 by reading the full traceback, since the +# report line is inside dynamo-rewritten code): torch/_dynamo keeps a name -> module map +# and looks it up by name — output_graph.py:1259 / :1384 register into `nn_modules`, +# :3000 does `self.nn_modules[node.target]` (a bare-name KeyError when absent), and +# :1225 documents stripping "_buffers[..]/_parameters[..]/_modules[..]" from those +# names. So the compiler is failing to RESOLVE A SUBMODULE BY NAME while tracing the +# GLM decoder layer. Whether the root cause is the fork's module tree (something +# dynamo cannot register statically) or dynamo's own registration is NOT yet settled. +# +# NEXT DEBUG STEP: re-run with TORCHDYNAMO_VERBOSE=1 (torch offers it in the error +# itself: "Set TORCHDYNAMO_VERBOSE=1 for the internal stack trace") to get the internal +# frame that computed the name. NOTE the launcher gotcha that wasted one boot: +# /home/dank/serve_glm_autoround.sh RESETS ENVARGS (it is initialized empty near the +# top and only the COMPILE_PATCH branch appends), so an outer `ENVARGS=... bash ...` +# is silently dropped — add the var inside the script or to the branch itself. --- a/vllm/compilation/wrapper.py +++ b/vllm/compilation/wrapper.py @@ fullgraph opt-out (env VLLM_COMPILE_FULLGRAPH=0) @@ diff --git a/serverAppConfigs-docs/LLM/frank-primary-glm53-flash-w4a16.md b/serverAppConfigs-docs/LLM/frank-primary-glm53-flash-w4a16.md index b60e578..4f6be6e 100644 --- a/serverAppConfigs-docs/LLM/frank-primary-glm53-flash-w4a16.md +++ b/serverAppConfigs-docs/LLM/frank-primary-glm53-flash-w4a16.md @@ -175,18 +175,30 @@ What the boots established, in order (each one ~7 min): `RuntimeError: Graph breaks are not supported with aot compile`, because this build runs AOT compile by default and AOT refuses breaks. 3. **01+02+03 with `VLLM_COMPILE_FULLGRAPH=0 VLLM_USE_AOT_COMPILE=0` traces the - trunk successfully** (`Dynamo bytecode transform time: 1.08 s`) and then dies in - `profile_run` with `KeyError: 'in_proj_qkvbfg_a'` at - `glm5next/common/model.py:456` (the KDA layer's merged projection, - `kda.py:217`) — a dict/registry lookup keyed by attribute name that the eager - path never reaches, i.e. **a latent bug the compile exposes, not a dynamo - limit**. + trunk successfully** (`Dynamo bytecode transform time: 1.08 s`) and then dies on + the decode forward with `KeyError: 'in_proj_qkvbfg_a'` (`Glm5NextModel.forward` + line 748 calling the decoder layer at line 456). `in_proj_qkvbfg_a` is the KDA + layer's merged column-parallel projection (`kda.py:217`, called at :407) — a real + submodule attribute name, also present in the `load_weights` shard table + (`model.py:780-785`). Reading the full traceback (the reported line is inside + dynamo-rewritten code) traces the throw to dynamo's own name→module map: + `torch/_dynamo/output_graph.py` registers into `nn_modules` at :1259/:1384, + looks up with `self.nn_modules[node.target]` at :3000 (a bare-name `KeyError` + when absent), and documents stripping `_buffers[..]/_parameters[..]/_modules[..]` + from those names at :1225. So the compiler is failing to **resolve a submodule by + name** while tracing the GLM decoder layer; whether the root cause is the fork's + module tree or dynamo's registration is not yet settled. The next step is the one + torch itself suggests in that error — `TORCHDYNAMO_VERBOSE=1` for the internal + stack trace (and note the launcher resets `ENVARGS`, so it must be added inside + `serve_glm_autoround.sh`'s COMPILE_PATCH branch rather than in the outer + environment). -That last row is the state of the work: the remaining blocker is a real bug in the -fork's mHC/KDA wiring, and it must be fixed (plus whatever follows it) before the -changelist can boot. Two verified facts make the payoff concrete: `suppress_errors` -does **not** cover graph breaks (only `fullgraph=False` does), and the GLM trunk can -never satisfy `fullgraph` because its indexer metadata syncs to host +That last row is the state of the work: the changelist is one unresolved +submodule-name lookup away from booting, and it must be cleared (plus whatever +follows it) before the compile payoff can be measured. Two verified facts make the +rest of the shape concrete: `suppress_errors` does **not** cover graph breaks (only +`fullgraph=False` does), and the GLM trunk can never satisfy `fullgraph` because +its indexer metadata syncs to host (`glm5next/amd/sparse_indexer.py:263,419`: `int(...max().item())`). ### Two structural constraints on any further fork work @@ -221,10 +233,48 @@ layer indices, the same interface `deepseek_v2.py` (also MLA+MoE) implements — capture at `idx in self.aux_hidden_state_layers` in the layer loop. `glm5next` implements neither. -Expected benefit is **not** assumed: DFlash2 amortizes a draft block (8 tokens) over -one verification pass, and per the profile above a verification pass costs pointwise -work proportional to tokens — the same reason MTP4 was flat. Measure it; do not -budget on it. +**The load was tried, and the gap is exactly one interface.** Payload fetched to +`/home/dank/models/GLM-5.3-Flash-DFlash2` and hash-verified (`model.safetensors` +2,342,169,800 B, sha256 `b038e1d9…` = the Hub's LFS oid). Booting the engine with +`--speculative-config '{"method":"dflash","model":"/draft","num_speculative_tokens":8}'` +accepts the config and the drafter and dies at worker init with: + +``` +RuntimeError: Model does not support EAGLE3 interface +``` + +So nothing about the drafter, the method name, or the payload is wrong — the GLM +trunk simply does not implement the aux-hidden-state contract the dflash runner +requires. The reference is in this same image, twice: `deepseek_v2.py` +(`set_aux_hidden_state_layers` / `get_eagle3_aux_hidden_state_layers` at :1948-1951, +capture at :1554 and :1579) and the fork's own `qwen4_exp` trunk. The contract is +small: + +- `self.aux_hidden_state_layers: tuple[int, ...]` on the trunk, default `()`, set + by the wrapper's `set_aux_hidden_state_layers()`; +- in the layer loop, `if idx in self.aux_hidden_state_layers: aux_hidden_states.append()` + (DeepSeek appends `hidden_states + residual`, with a TP all-gather fixup when the + rank holds fewer tokens than `positions`); +- `Glm5NextModel.forward` returns `(hidden_states, aux_hidden_states)` when the + list is non-empty; +- `get_eagle3_aux_hidden_state_layers()` supplies a default triple when the drafter + config omits `target_layer_ids` (this drafter provides `[5, 14, 24, 33, 42]`, so + the runner derives trunk indices `[6, 15, 25, 34, 43]`). + +**The GLM-specific unknown** is what "the single-stream state entering layer k" +means for a trunk whose layers carry mHC multi-stream state: DeepSeek's +`(hidden_states, residual)` pair has no counterpart here — GLM's decoder layer +returns `(hidden_states, residual, post, comb)` with `hc_expand`/`hc_pre` mixing +`hc_mult` streams, so the plausible answer is the pre-mHC single-stream input `x` +(4096-wide, matching the drafter's `hidden_size`), not a post-layer value. That is +a semantic choice to validate by acceptance rate, not by inspection. + +**Budget no speedup.** Per the profile above, a verification pass pays pointwise +cost proportional to its token count — the same reason MTP4 measured flat — and +MTP4's acceptance was already excellent (3.76-4.90). The port is worth doing as +*enabling work* (it is the prerequisite for any spec decode on this endpoint, and +it is small), not as a performance fix, and the honest expectation on this shape is +flat decode with a prefill regression like MTP's until the pointwise mass is fused. Note also that the KV-cap patch's predecessor arithmetic in [[dankserver-qwen38-fp8-expert-offload-tuning]] should be read the same way: its @@ -275,17 +325,32 @@ newest GLM dialect the fork ships (`glm45`/`glm47` exist for both). ## Fork traps found (for the engine authors) -1. **`--kv-cache-dtype fp8` is illegal on gfx12 with this model.** The - `ROCM_AITER_MLA_SPARSE` backend's `_use_rocm_sparse_triton()` requires a - non-fp8 KV cache, and that rope-free Triton path is the **only** MLA decode - available on gfx12: aiter's MLA decode op is registered under - `@if_aiter_supported`, which is gated on **CDNA > 2** +1. **`--kv-cache-dtype fp8` is illegal on gfx12 with this model — and it buys + nothing even if fixed.** The `ROCM_AITER_MLA_SPARSE` backend's + `_use_rocm_sparse_triton()` requires a non-fp8 KV cache, and that rope-free + Triton path is the **only** MLA decode available on gfx12: aiter's MLA decode + op is registered under `@if_aiter_supported`, which is gated on **CDNA > 2** (`is_aiter_found_and_supported()` → `get_cdna_version() > 2`). With fp8 KV the MTP draft step demands an op that cannot exist here: `RuntimeError: Worker failed with error ''_OpNamespace' 'vllm' object has no attribute 'rocm_aiter_mla_decode_fwd''`. KV dtype therefore stays auto/bf16. (`VLLM_ROCM_USE_AITER_MLA=1` does **not** help — the umbrella gate is the CDNA check, not the env var.) + - **The capacity payoff is measured at zero.** The point of fp8 KV is fitting + more tokens in the same reserve, and this fork's reserve model is + **dtype-blind**: the fp8 boot printed the *identical* plan line as bf16 — + `KV reserve 1.76 GiB (13600 B/token x 131072 + mamba pages)`. The reserve is + dominated by the per-sequence KDA state (34 linear-attention layers) and the + DSA indexer's pools, not by the MLA KV bytes those 13,600 B/token barely + cover — so halving the KV dtype cannot move it. + - **What a fix would take, if it were ever wanted**: a new fp8 path in the + rope-free Triton kernels (`rocm_sparse_attn_prefill` / `rocm_sparse_attn_decode` + in `v1/attention/ops/rocm_aiter_mla_sparse.py`) that dequantizes pages + in-kernel, plus loosening `_use_rocm_sparse_triton()`. The fork's + `_r4d_fp8_paged_mqa_logits` shows the author already has an fp8 MQA-logits + path, but the sparse-MLA backend is not wired to it. Verdict 2026-09-30: + **do not spend effort here** — the payoff is capacity, and the reserve is + dtype-blind. 2. **The plan's KV reserve under-models this model, and the cap enforces the mistake.** The plan charges 13,600 B/token + a small mamba term, but GLM carries a per-sequence KDA state (34 linear-attention layers) and, under MTP, @@ -322,6 +387,16 @@ podman logs frank-primary-glm 2>&1 | grep "expert_lru.py" # boot: ~7 min warm page cache, longer cold; health returns 503 while loading curl -s -o /dev/null -w "%{http_code}\n" http://127.0.0.1:7770/health + +# experiment windows (production stopped + `systemctl mask --runtime`, which +# self-clears at reboot) ran from the host-side scratch launcher +# /home/dank/serve_glm_autoround.sh: env knobs MAXLEN SEQS NBATCH UTIL KVBYTES +# KVDTYPE SPEC SPECJSON_FILE TOOLPARSER REASONINGPARSER TELEM GCHUNKS GLANES +# PROFDIR GLM_PATCH_MOUNT COMPILE_PATCH GATHER_PATCH DRAFT_MOUNT ENVARGS. +# DRAFT_MOUNT=1 mounts /home/dank/models/GLM-5.3-Flash-DFlash2 at /draft; +# SPECJSON_FILE passes a whole --speculative-config JSON (JSON with no spaces, +# word-split by the launcher's unquoted $EXTRA). +bash /home/dank/serve_glm_autoround.sh ``` Boot gate: the engine refuses a legal-looking shape in two measured ways — the diff --git a/serverAppConfigs-docs/log.md b/serverAppConfigs-docs/log.md index b6f9d7b..a88f694 100644 --- a/serverAppConfigs-docs/log.md +++ b/serverAppConfigs-docs/log.md @@ -66,6 +66,21 @@ capture (`set_aux_hidden_state_layers` / per-layer capture, as `deepseek_v2.py` demonstrates). No speedup is budgeted: a verification pass pays pointwise cost proportional to its token count. + - **DFlash2 tried, and the gap is exactly that one interface.** Payload fetched + and hash-verified (`/home/dank/models/GLM-5.3-Flash-DFlash2`, 2,342,169,800 B, + sha256 `b038e1d9…` = the Hub's LFS oid). Booting with + `--speculative-config '{"method":"dflash","model":"/draft","num_speculative_tokens":8}'` + accepts the config and the drafter and dies at worker init with + `RuntimeError: Model does not support EAGLE3 interface` — nothing about the + drafter, the method or the payload is wrong; the trunk simply does not + implement the aux-hidden-state contract. + - **Compile blocker narrowed to dynamo's name map.** The `KeyError: + 'in_proj_qkvbfg_a'` (see above) is thrown from `torch/_dynamo/output_graph.py`'s + `nn_modules` name lookup (:3000, registered at :1259/:1384, with `_modules[..]` + prefixes stripped at :1225) — i.e. the compiler cannot resolve a submodule by + name while tracing the GLM decoder layer. Next step is `TORCHDYNAMO_VERBOSE=1` + (torch names it in the error), noting the scratch launcher resets `ENVARGS` so + it must be set inside the script's COMPILE_PATCH branch. - Measured and rejected: **MTP4** (the head loads, arch resolves to `Glm5NextMTPModel`, acceptance is excellent at mean length 3.76-4.90 — but decode was unchanged (9.6-11.2 vs 10.0) while prefill COLLAPSED 918 -> 255.