diff --git a/scripts/patches/glm53-compile-04-fused-qkv-rmsnorm-optout.patch b/scripts/patches/glm53-compile-04-fused-qkv-rmsnorm-optout.patch index 0a63597..6c1d35b 100644 --- a/scripts/patches/glm53-compile-04-fused-qkv-rmsnorm-optout.patch +++ b/scripts/patches/glm53-compile-04-fused-qkv-rmsnorm-optout.patch @@ -4,24 +4,30 @@ # STATUS: WIP — fourth of the "compile the GLM trunk" changelist. Needs 01 (the # decorator) to matter; not part of the deployed shape. # -# MEASUREMENT NOTE — READ BEFORE TRUSTING ANY CLAIM BELOW. Two break-tolerance runs -# disagree, and the difference is a logging flag: -# * 01+02+03, AOT off, breaks tolerated, NO TORCH_LOGS -> gets PAST all graph-break -# handling and dies on the dynamo name-map KeyError ('in_proj_qkvbfg_a'). -# * same + TORCH_LOGS=graph_breaks -> dies on the fatal -# `Unsupported: Read uninitialized cell` at this very call site. -# The likely mechanism is that `graph_breaks` logging makes dynamo build a break -# *explanation* (walk the frames, name the reason), and that analysis is what trips on -# the uninitialized cell; without the flag the break is taken silently. So the cell -# error is PROBABLY an artifact of the diagnostic, not necessarily a blocker for a -# break-tolerant compile. Whether this patch is therefore necessary — as opposed to -# removing a call dynamo cannot *explain* — is being decided by a flag-free run of -# 01+02+03+04. Do not repeat the earlier overclaim ("fatal even with fullgraph=False"); -# it was written from the confounded run. +# MEASUREMENT NOTE — read the run history before trusting any claim below, because a +# launcher bug made four consecutive boots misleading. The scratch launcher +# (serve_glm_autoround.sh) was restructured on 2026-09-30 and the +# `-e VLLM_COMPILE_FULLGRAPH=0` line was DROPPED, so those boots mounted the wrapper +# patch while `_FULLGRAPH` still defaulted to True. With fullgraph=True *every* break +# is an error, which is why the reported failure kept moving as breaks were removed: +# +# fullgraph=False, AOT off -> breaks tolerated; dies later on the dynamo name-map +# `KeyError: 'in_proj_qkvbfg_a'` +# fullgraph=True, AOT off -> dies on whichever break comes first +# (cell -> workspace -> compiler-disable) +# +# An earlier version of this note blamed `TORCH_LOGS=graph_breaks` for the difference. +# That was WRONG — the flag only logs; the launcher silently ran fullgraph=True. +# Launcher fixed; the env var is set again. +# +# What it means for THIS patch: under fullgraph=True the fused-norm call was the first +# break dynamo reported, and after this patch the report moved on to the workspace +# acquisition — so it removes a real break. It is not by itself what unblocks a boot: +# with breaks genuinely tolerated the blocker is the name-map KeyError. # # IT IS STILL THE RIGHT PATTERN TO REMOVE, either way: the launcher family -# (@kernel_launcher, 39 call sites) is exactly what breaks dynamo's break analysis, and -# the object it closes over is bound after use (see below). +# (@kernel_launcher, 39 call sites) is exactly the shape that produces breaks, and the +# object this one closes over is bound after use (see below). # # THE BREAK (measured, TORCH_LOGS=graph_breaks, rank0, 2026-09-30). With 01+02+03 the # trunk traces, and with AOT off + fullgraph=True dynamo reports a FATAL break — fatal diff --git a/scripts/patches/glm53-compile-05-workspace-acquisition-opaque.patch b/scripts/patches/glm53-compile-05-workspace-acquisition-opaque.patch new file mode 100644 index 0000000..8a2e7fb --- /dev/null +++ b/scripts/patches/glm53-compile-05-workspace-acquisition-opaque.patch @@ -0,0 +1,91 @@ +# glm53-compile-05-workspace-acquisition-opaque.patch — the indexer's workspace +# acquisition must not be traced. +# +# STATUS: WIP — fifth of the "compile the GLM trunk" changelist. This is the current +# blocker; it was reached only after 04. +# +# THE BREAK — CONFIRMED with AND WITHOUT the logging flag (2026-09-30): +# +# torch._dynamo.exc.Unsupported: Unsupported method call +# user stack +# glm5next/common/model.py:748 Glm5NextModel.forward +# glm5next/common/model.py:526 Glm5NextDecoderLayer.forward +# glm5next/common/attention.py:599 / :401 +# glm5next/amd/sparse_indexer.py:711 forward_native +# glm5next/amd/sparse_indexer.py:132 sparse_attn_indexer_kpool +# v1/worker/workspace.py:135 get_simultaneous +# v1/worker/workspace.py:155 _ensure_workspace_size +# +# Line 132 is the PROFILE-RUN branch — the source comment says what it is: +# +# if not isinstance(attn_metadata, dict): +# # Reserve workspace for indexer during profiling run +# +# so the engine dies exactly where it sizes its memory. `_ensure_workspace_size` +# reads ContextVar and ubatch state (`_workspace_lane.get()`, +# `dbo_current_ubatch_id()`) and may allocate; none of that is traceable. +# +# FIX: the documented escape hatch — `torch.compiler.disable` around a thin wrapper, +# so dynamo stops analysing the call and executes it eagerly from inside the compiled +# region. Chosen over decorating the workspace manager's own methods because that +# would change workspace behaviour for every model, not just this one. The wrapper is +# a no-op when nothing is compiling, so it needs no env gate and is safe in the +# stock (uncompiled) path — which is why this patch can be reasoned about +# independently of whether the changelist ever ships. +# +# CAVEAT, stated plainly: a compiler-disabled region is still a graph break, so this +# needs patch 03 (VLLM_COMPILE_FULLGRAPH=0) to be tolerable at all, and whether dynamo +# can build a resume function around it is exactly what the next boot decides — the +# difference between this failure and 04's `Read uninitialized cell`. +# +# TARGET: vllm/models/glm5next/amd/sparse_indexer.py (inside the pinned image) +# base sha256 f773a99265d3f440c8fad22c8fb5a8b33f926162094cffc3a2ecdaa42763c5b1 +# patched sha256 c154115afe8253a35c652b7198929ef4c1874957d333fdb3340cacf88e612b78 +# +# APPLY: `patch -p1` from the vllm package root, or bind-mount over site-packages. +--- a/vllm/models/glm5next/amd/sparse_indexer.py ++++ b/vllm/models/glm5next/amd/sparse_indexer.py +@@ -35,6 +35,21 @@ + from vllm.v1.attention.ops.common import pack_seq_triton, unpack_seq_triton + from vllm.v1.worker.workspace import current_workspace_manager + ++@torch.compiler.disable ++def _acquire_workspace(*shapes_and_dtypes): ++ """Acquire shared workspace buffers, untraced. ++ ++ Workspace acquisition reads ContextVar/ubatch state and may allocate ++ (v1/worker/workspace.py: ``_ensure_workspace_size`` -> ``_workspace_lane.get()``, ++ ``dbo_current_ubatch_id()``), which dynamo cannot trace -- it raises ++ ``Unsupported: Unsupported method call`` and kills the engine during the profile ++ run. ``torch.compiler.disable`` keeps the call as an opaque, eagerly-executed ++ region inside the compiled graph. No-op when not compiling. ++ """ ++ ++ return current_workspace_manager().get_simultaneous(*shapes_and_dtypes) ++ ++ + logger = init_logger(__name__) + + # kpool write helper: form pools from the current token batch and compress them +@@ -129,7 +144,7 @@ + values_spec, scales_spec = _gather_workspace_shapes( + total_seq_lens, head_dim, fp8_dtype, use_fp4_cache + ) +- current_workspace_manager().get_simultaneous( ++ _acquire_workspace( + values_spec, + scales_spec, + ((RADIX_TOPK_WORKSPACE_SIZE,), torch.uint8), +@@ -275,11 +290,10 @@ + # Layout switches between FP8 (head_dim bytes + 4-byte fp32 scale) and + # MXFP4 (head_dim/2 bytes packed + head_dim/MXFP4_BLOCK_SIZE ue8m0 + # scales) based on use_fp4_cache. +- workspace_manager = current_workspace_manager() + values_spec, scales_spec = _gather_workspace_shapes( + total_seq_lens, head_dim, fp8_dtype, use_fp4_cache + ) +- k_quant_full, k_scale_full = workspace_manager.get_simultaneous( ++ k_quant_full, k_scale_full = _acquire_workspace( + values_spec, + scales_spec, + ) diff --git a/serverAppConfigs-docs/LLM/frank-primary-glm53-flash-w4a16.md b/serverAppConfigs-docs/LLM/frank-primary-glm53-flash-w4a16.md index 3d11759..40b4f57 100644 --- a/serverAppConfigs-docs/LLM/frank-primary-glm53-flash-w4a16.md +++ b/serverAppConfigs-docs/LLM/frank-primary-glm53-flash-w4a16.md @@ -197,42 +197,47 @@ What the boots established, in order (each one ~7 min): the GLM MLA block stops calling the fork's fused-norm JIT kernel, whose kernel object in `models/common/ops/fused_qk_rmsnorm.py` is bound *after* the function that uses it — `_FUSED_Q_KV_RMSNORM_KERNEL` at line 223 vs its use at line 211). - **Its necessity is not yet established** — see the confound below. -5. **A confound worth knowing about, because it cost two boots.** Two break-tolerance - runs of the same config disagree, and the difference is one logging flag: - `01+02+03` with `TORCH_LOGS=graph_breaks` dies on a fatal - `Unsupported: Read uninitialized cell` at `mla.py:208` (the fused-norm call), while - the same config **without** the flag gets past break handling entirely and dies - later on the dynamo name-map `KeyError`. The likely mechanism: `graph_breaks` - logging makes dynamo build an *explanation* for each break (walk frames, name the - reason), and that analysis is what trips on the uninitialized cell — without the - flag, the break is taken silently. So that fatal-looking cell error is probably an - artifact of the diagnostic. Related: with `graph_breaks` on, the next reported - failure is `Unsupported: Unsupported method call` at - `sparse_indexer.py:132 sparse_attn_indexer_kpool → workspace.py:135 - get_simultaneous / :155 _ensure_workspace_size`, i.e. the indexer allocates its - shared workspace *inside* the forward (its own comment: *"will allocate on first - use"*) — also flagged under explanation, also not yet shown to block a plain - break-tolerant compile. + **It removes a real break** (under `fullgraph=True` the fused call was the first + break dynamo reported; after this patch the report moves on to the workspace + acquisition) — but it is not by itself what blocks a boot, because with breaks + genuinely tolerated the blocker is the name-map `KeyError`. See the run history + below. +5. **A launcher bug that made four boots misleading — worth knowing because the + failure kept moving.** The scratch launcher's compile branches were restructured + on 2026-09-30 and the `-e VLLM_COMPILE_FULLGRAPH=0` line was dropped, so four + consecutive boots mounted the wrapper patch while `_FULLGRAPH` still defaulted to + True. With `fullgraph=True` **every** break is an error, so those boots reported + whichever break came first: + + | config | outcome | + | --- | --- | + | `fullgraph=False`, AOT off | breaks tolerated → dies later on the dynamo name-map `KeyError: 'in_proj_qkvbfg_a'` | + | `fullgraph=True`, AOT off | dies on the first break (cell → workspace → compiler-disable) | + + So the reported breaks are real *as breaks*, but they are not what blocks a + break-tolerant compile — and an earlier version of this page blamed + `TORCH_LOGS=graph_breaks` for the difference. **That was wrong**; the flag only + logs. The launcher sets the env var again. **How to see the breaks (and its trap).** `TORCH_LOGS=graph_breaks` names each break with a user stack, and `VLLM_USE_AOT_COMPILE=0` is required because AOT compile -hard-refuses breaks — but per the confound above, treat the flagged fatal errors as -*candidates*, and confirm each one against a flag-free run before believing it blocks -the boot. For the name-map KeyError there is a ready-made probe: put -`/home/dank/dynamo-probe` on `PYTHONPATH` with `VLLM_DYNAMO_DUMP_ATTRS=1` (a -`sitecustomize.py` — a `.pth` on `PYTHONPATH` is NOT executed, and the image's own -`sitecustomize` is chained so its apport hook survives) and it prints, at the miss, -the node target, the near-miss keys, and whether another live `OutputGraph` holds -the key. +hard-refuses breaks — but a fullgraph=True run reports the first break as fatal, so +check which configuration you are actually in (`podman exec printenv +VLLM_COMPILE_FULLGRAPH`) before concluding a break blocks the boot. For the name-map +KeyError there is a ready-made probe: put `/home/dank/dynamo-probe` on `PYTHONPATH` +with `VLLM_DYNAMO_DUMP_ATTRS=1` (a `sitecustomize.py` — a `.pth` on `PYTHONPATH` is +NOT executed, and the image's own `sitecustomize` is chained so its apport hook +survives) and it prints, at the miss, the node target, the near-miss keys, and +whether another live `OutputGraph` holds the key. **State of the changelist:** the trunk traces (`Dynamo bytecode transform time: -1.08 s`), and the one failure confirmed by a flag-free run is the dynamo name-map -`KeyError: 'in_proj_qkvbfg_a'`. Two further candidates are queued behind it -(the fused-norm launcher cell, the indexer's in-forward workspace allocation). Two -verified facts bound the rest of the shape: `suppress_errors` does **not** cover -graph breaks (only `fullgraph=False` does), and the GLM trunk can never satisfy -`fullgraph` because its indexer metadata syncs to host +1.08 s`). With breaks genuinely tolerated the confirmed blocker is the dynamo +name-map `KeyError: 'in_proj_qkvbfg_a'`. Two further breaks are known and worth +removing regardless: the fused-norm launcher cell (patch 04 removes the call) and +the indexer's in-forward workspace acquisition (patch 05 makes it +compiler-opaque). Two verified facts bound the rest of the shape: `suppress_errors` +does **not** cover graph breaks (only `fullgraph=False` does), and the GLM trunk can +never satisfy `fullgraph` because its indexer metadata syncs to host (`glm5next/amd/sparse_indexer.py:263,419`: `int(...max().item())`) — both are dead fallbacks on the decode path that the fork author already replaced, so they are not the blockers they appear to be at first glance. diff --git a/serverAppConfigs-docs/log.md b/serverAppConfigs-docs/log.md index 3d7dc8e..00d055e 100644 --- a/serverAppConfigs-docs/log.md +++ b/serverAppConfigs-docs/log.md @@ -74,23 +74,24 @@ `RuntimeError: Model does not support EAGLE3 interface` — nothing about the drafter, the method or the payload is wrong; the trunk simply does not implement the aux-hidden-state contract. - - **Compile work: one confirmed blocker, plus a measurement confound worth - recording.** Flag-free break-tolerance runs (01+02+03, AOT off, breaks allowed) - get past break handling and die on the dynamo name-map `KeyError: - 'in_proj_qkvbfg_a'`. Adding `TORCH_LOGS=graph_breaks` to the same config changes - where it dies: a fatal-looking `Unsupported: Read uninitialized cell` at - `mla.py:208` calling the fork's fused-norm JIT kernel - (`models/common/ops/fused_qk_rmsnorm.py`: `_FUSED_Q_KV_RMSNORM_KERNEL` is bound at - line 223, *after* its use at line 211), and with that removed, `Unsupported: - Unsupported method call` in the indexer's in-forward workspace allocation + - **Compile work: one confirmed blocker, and a launcher bug that misled four boots.** + With breaks genuinely tolerated (`fullgraph=False`, AOT off) the trunk traces and + dies on the dynamo name-map `KeyError: 'in_proj_qkvbfg_a'`. Four later boots + reported *different* fatal breaks — `Unsupported: Read uninitialized cell` at + `mla.py:208` (the fused-norm JIT kernel), then `Unsupported method call` in the + indexer's in-forward workspace acquisition (`sparse_indexer.py:132 -> workspace.py:135 get_simultaneous / :155 - _ensure_workspace_size`). **The likely mechanism is that `graph_breaks` logging - makes dynamo build a break *explanation*, and that analysis is what trips** — so - those two are candidates, not proven blockers, and I overclaimed them once before - catching it. `scripts/patches/glm53-compile-04-fused-qkv-rmsnorm-optout.patch` takes - GLM off the fused-norm kernel anyway (`R4D_GLM_FUSE_QKV_RMSNORM=0`, a constructor - flag glm5next sets at `attention.py:583`); whether it is *needed* is what the - flag-free 01+02+03+04 run settles. + _ensure_workspace_size`), then `Skip calling torch.compiler.disable()'d function` + — and the cause was **my own scratch launcher**: its compile branches were + restructured and the `-e VLLM_COMPILE_FULLGRAPH=0` line was dropped, so those + boots ran `fullgraph=True`, where *every* break is an error and the first one + wins. I first blamed `TORCH_LOGS=graph_breaks` for the difference; that was wrong + (the flag only logs). The moving reports are still useful — they are the break + list — and patch 04 demonstrably removes the first of them (the fused-norm cell), + patch 05 makes the workspace acquisition compiler-opaque. + `scripts/patches/glm53-compile-04-fused-qkv-rmsnorm-optout.patch` takes GLM off + the fused-norm kernel (`R4D_GLM_FUSE_QKV_RMSNORM=0`, a constructor flag glm5next + sets at `attention.py:583`). - **Diagnostics built for this (reusable).** `TORCH_LOGS=graph_breaks` + `VLLM_USE_AOT_COMPILE=0` names each break with its user stack (AOT compile hard-refuses breaks outright); `/home/dank/dynamo-probe` on `PYTHONPATH` with