frank-primary: patch 05, five verified artifacts, and correct the wrong confound theory

Root-caused the moving failures: they were my own scratch launcher. Its compile
branches were restructured and the '-e VLLM_COMPILE_FULLGRAPH=0' line was dropped, so
four consecutive boots mounted the wrapper patch while _FULLGRAPH still defaulted to
True — and with fullgraph=True EVERY break is an error, so each boot reported whichever
break came first (cell -> workspace -> compiler-disable). I had blamed
TORCH_LOGS=graph_breaks for the difference; that was wrong, the flag only logs. The
launcher sets the env var again, and patch 04's header, the runbook and the work log
now carry the true run history as a table instead of the bad theory.

New artifact: scripts/patches/glm53-compile-05-workspace-acquisition-opaque.patch —
the indexer acquires its shared workspace INSIDE the forward (its profile-run branch
literally says 'Reserve workspace for indexer during profiling run'), and
v1/worker/workspace.py:_ensure_workspace_size reads ContextVar/ubatch state, which
dynamo cannot trace. Wrap both call sites in a torch.compiler.disable'd helper
(narrower than decorating the workspace manager, which every model would go through;
a no-op when not compiling).

All five artifacts verified against the pinned image: applied in sequence with patch
-p1 they reproduce their recorded patched sha256 exactly (6e9cb81b…, 0751852e…,
732363bb…, 84969993…, c154115a…).

tests/run.sh: 362 seams pass.

scripts/patches/glm53-compile-04-fused-qkv-rmsnorm-optout.patch

@@ -4,24 +4,30 @@
 # STATUS: WIP — fourth of the "compile the GLM trunk" changelist. Needs 01 (the
 # decorator) to matter; not part of the deployed shape.
 #
-# MEASUREMENT NOTE — READ BEFORE TRUSTING ANY CLAIM BELOW.  Two break-tolerance runs
+# MEASUREMENT NOTE — read the run history before trusting any claim below, because a
-# disagree, and the difference is a logging flag:
+# launcher bug made four consecutive boots misleading.  The scratch launcher
-#   * 01+02+03, AOT off, breaks tolerated, NO TORCH_LOGS  -> gets PAST all graph-break
+# (serve_glm_autoround.sh) was restructured on 2026-09-30 and the
-#     handling and dies on the dynamo name-map KeyError ('in_proj_qkvbfg_a').
+# `-e VLLM_COMPILE_FULLGRAPH=0` line was DROPPED, so those boots mounted the wrapper
-#   * same + TORCH_LOGS=graph_breaks                       -> dies on the fatal
+# patch while `_FULLGRAPH` still defaulted to True.  With fullgraph=True *every* break
-#     `Unsupported: Read uninitialized cell` at this very call site.
+# is an error, which is why the reported failure kept moving as breaks were removed:
-# The likely mechanism is that `graph_breaks` logging makes dynamo build a break
+#
-# *explanation* (walk the frames, name the reason), and that analysis is what trips on
+#   fullgraph=False, AOT off  -> breaks tolerated; dies later on the dynamo name-map
-# the uninitialized cell; without the flag the break is taken silently.  So the cell
+#                                `KeyError: 'in_proj_qkvbfg_a'`
-# error is PROBABLY an artifact of the diagnostic, not necessarily a blocker for a
+#   fullgraph=True,  AOT off  -> dies on whichever break comes first
-# break-tolerant compile.  Whether this patch is therefore necessary — as opposed to
+#                                (cell -> workspace -> compiler-disable)
-# removing a call dynamo cannot *explain* — is being decided by a flag-free run of
+#
-# 01+02+03+04.  Do not repeat the earlier overclaim ("fatal even with fullgraph=False");
+# An earlier version of this note blamed `TORCH_LOGS=graph_breaks` for the difference.
-# it was written from the confounded run.
+# That was WRONG — the flag only logs; the launcher silently ran fullgraph=True.
+# Launcher fixed; the env var is set again.
+#
+# What it means for THIS patch: under fullgraph=True the fused-norm call was the first
+# break dynamo reported, and after this patch the report moved on to the workspace
+# acquisition — so it removes a real break.  It is not by itself what unblocks a boot:
+# with breaks genuinely tolerated the blocker is the name-map KeyError.
 #
 # IT IS STILL THE RIGHT PATTERN TO REMOVE, either way: the launcher family
-# (@kernel_launcher, 39 call sites) is exactly what breaks dynamo's break analysis, and
+# (@kernel_launcher, 39 call sites) is exactly the shape that produces breaks, and the
-# the object it closes over is bound after use (see below).
+# object this one closes over is bound after use (see below).
 #
 # THE BREAK (measured, TORCH_LOGS=graph_breaks, rank0, 2026-09-30).  With 01+02+03 the
 # trunk traces, and with AOT off + fullgraph=True dynamo reports a FATAL break — fatal

scripts/patches/glm53-compile-05-workspace-acquisition-opaque.patch

@@ -0,0 +1,91 @@
+# glm53-compile-05-workspace-acquisition-opaque.patch — the indexer's workspace
+# acquisition must not be traced.
+#
+# STATUS: WIP — fifth of the "compile the GLM trunk" changelist. This is the current
+# blocker; it was reached only after 04.
+#
+# THE BREAK — CONFIRMED with AND WITHOUT the logging flag (2026-09-30):
+#
+#   torch._dynamo.exc.Unsupported: Unsupported method call
+#   user stack
+#     glm5next/common/model.py:748      Glm5NextModel.forward
+#     glm5next/common/model.py:526      Glm5NextDecoderLayer.forward
+#     glm5next/common/attention.py:599 / :401
+#     glm5next/amd/sparse_indexer.py:711  forward_native
+#     glm5next/amd/sparse_indexer.py:132  sparse_attn_indexer_kpool
+#     v1/worker/workspace.py:135          get_simultaneous
+#     v1/worker/workspace.py:155          _ensure_workspace_size
+#
+# Line 132 is the PROFILE-RUN branch — the source comment says what it is:
+#
+#     if not isinstance(attn_metadata, dict):
+#         # Reserve workspace for indexer during profiling run
+#
+# so the engine dies exactly where it sizes its memory.  `_ensure_workspace_size`
+# reads ContextVar and ubatch state (`_workspace_lane.get()`,
+# `dbo_current_ubatch_id()`) and may allocate; none of that is traceable.
+#
+# FIX: the documented escape hatch — `torch.compiler.disable` around a thin wrapper,
+# so dynamo stops analysing the call and executes it eagerly from inside the compiled
+# region.  Chosen over decorating the workspace manager's own methods because that
+# would change workspace behaviour for every model, not just this one.  The wrapper is
+# a no-op when nothing is compiling, so it needs no env gate and is safe in the
+# stock (uncompiled) path — which is why this patch can be reasoned about
+# independently of whether the changelist ever ships.
+#
+# CAVEAT, stated plainly: a compiler-disabled region is still a graph break, so this
+# needs patch 03 (VLLM_COMPILE_FULLGRAPH=0) to be tolerable at all, and whether dynamo
+# can build a resume function around it is exactly what the next boot decides — the
+# difference between this failure and 04's `Read uninitialized cell`.
+#
+# TARGET: vllm/models/glm5next/amd/sparse_indexer.py   (inside the pinned image)
+#   base    sha256 f773a99265d3f440c8fad22c8fb5a8b33f926162094cffc3a2ecdaa42763c5b1
+#   patched sha256 c154115afe8253a35c652b7198929ef4c1874957d333fdb3340cacf88e612b78
+#
+# APPLY: `patch -p1` from the vllm package root, or bind-mount over site-packages.
+--- a/vllm/models/glm5next/amd/sparse_indexer.py
++++ b/vllm/models/glm5next/amd/sparse_indexer.py
+@@ -35,6 +35,21 @@
+ from vllm.v1.attention.ops.common import pack_seq_triton, unpack_seq_triton
+ from vllm.v1.worker.workspace import current_workspace_manager
+ 
++@torch.compiler.disable
++def _acquire_workspace(*shapes_and_dtypes):
++    """Acquire shared workspace buffers, untraced.
++
++    Workspace acquisition reads ContextVar/ubatch state and may allocate
++    (v1/worker/workspace.py: ``_ensure_workspace_size`` -> ``_workspace_lane.get()``,
++    ``dbo_current_ubatch_id()``), which dynamo cannot trace -- it raises
++    ``Unsupported: Unsupported method call`` and kills the engine during the profile
++    run.  ``torch.compiler.disable`` keeps the call as an opaque, eagerly-executed
++    region inside the compiled graph.  No-op when not compiling.
++    """
++
++    return current_workspace_manager().get_simultaneous(*shapes_and_dtypes)
++
++
+ logger = init_logger(__name__)
+ 
+ # kpool write helper: form pools from the current token batch and compress them
+@@ -129,7 +144,7 @@
+         values_spec, scales_spec = _gather_workspace_shapes(
+             total_seq_lens, head_dim, fp8_dtype, use_fp4_cache
+         )
+-        current_workspace_manager().get_simultaneous(
++        _acquire_workspace(
+             values_spec,
+             scales_spec,
+             ((RADIX_TOPK_WORKSPACE_SIZE,), torch.uint8),
+@@ -275,11 +290,10 @@
+         # Layout switches between FP8 (head_dim bytes + 4-byte fp32 scale) and
+         # MXFP4 (head_dim/2 bytes packed + head_dim/MXFP4_BLOCK_SIZE ue8m0
+         # scales) based on use_fp4_cache.
+-        workspace_manager = current_workspace_manager()
+         values_spec, scales_spec = _gather_workspace_shapes(
+             total_seq_lens, head_dim, fp8_dtype, use_fp4_cache
+         )
+-        k_quant_full, k_scale_full = workspace_manager.get_simultaneous(
++        k_quant_full, k_scale_full = _acquire_workspace(
+             values_spec,
+             scales_spec,
+         )

serverAppConfigs-docs/LLM/frank-primary-glm53-flash-w4a16.md

@@ -197,42 +197,47 @@ What the boots established, in order (each one ~7 min):
    the GLM MLA block stops calling the fork's fused-norm JIT kernel, whose kernel
    object in `models/common/ops/fused_qk_rmsnorm.py` is bound *after* the function
    that uses it — `_FUSED_Q_KV_RMSNORM_KERNEL` at line 223 vs its use at line 211).
-   **Its necessity is not yet established** — see the confound below.
+   **It removes a real break** (under `fullgraph=True` the fused call was the first
-5. **A confound worth knowing about, because it cost two boots.** Two break-tolerance
+   break dynamo reported; after this patch the report moves on to the workspace
-   runs of the same config disagree, and the difference is one logging flag:
+   acquisition) — but it is not by itself what blocks a boot, because with breaks
-   `01+02+03` with `TORCH_LOGS=graph_breaks` dies on a fatal
+   genuinely tolerated the blocker is the name-map `KeyError`. See the run history
-   `Unsupported: Read uninitialized cell` at `mla.py:208` (the fused-norm call), while
+   below.
-   the same config **without** the flag gets past break handling entirely and dies
+5. **A launcher bug that made four boots misleading — worth knowing because the
-   later on the dynamo name-map `KeyError`. The likely mechanism: `graph_breaks`
+   failure kept moving.** The scratch launcher's compile branches were restructured
-   logging makes dynamo build an *explanation* for each break (walk frames, name the
+   on 2026-09-30 and the `-e VLLM_COMPILE_FULLGRAPH=0` line was dropped, so four
-   reason), and that analysis is what trips on the uninitialized cell — without the
+   consecutive boots mounted the wrapper patch while `_FULLGRAPH` still defaulted to
-   flag, the break is taken silently. So that fatal-looking cell error is probably an
+   True. With `fullgraph=True` **every** break is an error, so those boots reported
-   artifact of the diagnostic. Related: with `graph_breaks` on, the next reported
+   whichever break came first:
-   failure is `Unsupported: Unsupported method call` at
+
-   `sparse_indexer.py:132 sparse_attn_indexer_kpool → workspace.py:135
+   | config | outcome |
-   get_simultaneous / :155 _ensure_workspace_size`, i.e. the indexer allocates its
+   | --- | --- |
-   shared workspace *inside* the forward (its own comment: *"will allocate on first
+   | `fullgraph=False`, AOT off | breaks tolerated → dies later on the dynamo name-map `KeyError: 'in_proj_qkvbfg_a'` |
-   use"*) — also flagged under explanation, also not yet shown to block a plain
+   | `fullgraph=True`, AOT off | dies on the first break (cell → workspace → compiler-disable) |
-   break-tolerant compile.
+
+   So the reported breaks are real *as breaks*, but they are not what blocks a
+   break-tolerant compile — and an earlier version of this page blamed
+   `TORCH_LOGS=graph_breaks` for the difference. **That was wrong**; the flag only
+   logs. The launcher sets the env var again.
 
 **How to see the breaks (and its trap).** `TORCH_LOGS=graph_breaks` names each break
 with a user stack, and `VLLM_USE_AOT_COMPILE=0` is required because AOT compile
-hard-refuses breaks — but per the confound above, treat the flagged fatal errors as
+hard-refuses breaks — but a fullgraph=True run reports the first break as fatal, so
-*candidates*, and confirm each one against a flag-free run before believing it blocks
+check which configuration you are actually in (`podman exec <ctr> printenv
-the boot. For the name-map KeyError there is a ready-made probe: put
+VLLM_COMPILE_FULLGRAPH`) before concluding a break blocks the boot. For the name-map
-`/home/dank/dynamo-probe` on `PYTHONPATH` with `VLLM_DYNAMO_DUMP_ATTRS=1` (a
+KeyError there is a ready-made probe: put `/home/dank/dynamo-probe` on `PYTHONPATH`
-`sitecustomize.py` — a `.pth` on `PYTHONPATH` is NOT executed, and the image's own
+with `VLLM_DYNAMO_DUMP_ATTRS=1` (a `sitecustomize.py` — a `.pth` on `PYTHONPATH` is
-`sitecustomize` is chained so its apport hook survives) and it prints, at the miss,
+NOT executed, and the image's own `sitecustomize` is chained so its apport hook
-the node target, the near-miss keys, and whether another live `OutputGraph` holds
+survives) and it prints, at the miss, the node target, the near-miss keys, and
-the key.
+whether another live `OutputGraph` holds the key.
 
 **State of the changelist:** the trunk traces (`Dynamo bytecode transform time:
-1.08 s`), and the one failure confirmed by a flag-free run is the dynamo name-map
+1.08 s`). With breaks genuinely tolerated the confirmed blocker is the dynamo
-`KeyError: 'in_proj_qkvbfg_a'`. Two further candidates are queued behind it
+name-map `KeyError: 'in_proj_qkvbfg_a'`. Two further breaks are known and worth
-(the fused-norm launcher cell, the indexer's in-forward workspace allocation). Two
+removing regardless: the fused-norm launcher cell (patch 04 removes the call) and
-verified facts bound the rest of the shape: `suppress_errors` does **not** cover
+the indexer's in-forward workspace acquisition (patch 05 makes it
-graph breaks (only `fullgraph=False` does), and the GLM trunk can never satisfy
+compiler-opaque). Two verified facts bound the rest of the shape: `suppress_errors`
-`fullgraph` because its indexer metadata syncs to host
+does **not** cover graph breaks (only `fullgraph=False` does), and the GLM trunk can
+never satisfy `fullgraph` because its indexer metadata syncs to host
 (`glm5next/amd/sparse_indexer.py:263,419`: `int(...max().item())`) — both are dead
 fallbacks on the decode path that the fork author already replaced, so they are not
 the blockers they appear to be at first glance.

serverAppConfigs-docs/log.md

@@ -74,23 +74,24 @@
     `RuntimeError: Model does not support EAGLE3 interface` — nothing about the
     drafter, the method or the payload is wrong; the trunk simply does not
     implement the aux-hidden-state contract.
-  - **Compile work: one confirmed blocker, plus a measurement confound worth
+  - **Compile work: one confirmed blocker, and a launcher bug that misled four boots.**
-    recording.** Flag-free break-tolerance runs (01+02+03, AOT off, breaks allowed)
+    With breaks genuinely tolerated (`fullgraph=False`, AOT off) the trunk traces and
-    get past break handling and die on the dynamo name-map `KeyError:
+    dies on the dynamo name-map `KeyError: 'in_proj_qkvbfg_a'`. Four later boots
-    'in_proj_qkvbfg_a'`. Adding `TORCH_LOGS=graph_breaks` to the same config changes
+    reported *different* fatal breaks — `Unsupported: Read uninitialized cell` at
-    where it dies: a fatal-looking `Unsupported: Read uninitialized cell` at
+    `mla.py:208` (the fused-norm JIT kernel), then `Unsupported method call` in the
-    `mla.py:208` calling the fork's fused-norm JIT kernel
+    indexer's in-forward workspace acquisition
-    (`models/common/ops/fused_qk_rmsnorm.py`: `_FUSED_Q_KV_RMSNORM_KERNEL` is bound at
-    line 223, *after* its use at line 211), and with that removed, `Unsupported:
-    Unsupported method call` in the indexer's in-forward workspace allocation
     (`sparse_indexer.py:132 -> workspace.py:135 get_simultaneous / :155
-    _ensure_workspace_size`). **The likely mechanism is that `graph_breaks` logging
+    _ensure_workspace_size`), then `Skip calling torch.compiler.disable()'d function`
-    makes dynamo build a break *explanation*, and that analysis is what trips** — so
+    — and the cause was **my own scratch launcher**: its compile branches were
-    those two are candidates, not proven blockers, and I overclaimed them once before
+    restructured and the `-e VLLM_COMPILE_FULLGRAPH=0` line was dropped, so those
-    catching it. `scripts/patches/glm53-compile-04-fused-qkv-rmsnorm-optout.patch` takes
+    boots ran `fullgraph=True`, where *every* break is an error and the first one
-    GLM off the fused-norm kernel anyway (`R4D_GLM_FUSE_QKV_RMSNORM=0`, a constructor
+    wins. I first blamed `TORCH_LOGS=graph_breaks` for the difference; that was wrong
-    flag glm5next sets at `attention.py:583`); whether it is *needed* is what the
+    (the flag only logs). The moving reports are still useful — they are the break
-    flag-free 01+02+03+04 run settles.
+    list — and patch 04 demonstrably removes the first of them (the fused-norm cell),
+    patch 05 makes the workspace acquisition compiler-opaque.
+    `scripts/patches/glm53-compile-04-fused-qkv-rmsnorm-optout.patch` takes GLM off
+    the fused-norm kernel (`R4D_GLM_FUSE_QKV_RMSNORM=0`, a constructor flag glm5next
+    sets at `attention.py:583`).
   - **Diagnostics built for this (reusable).** `TORCH_LOGS=graph_breaks` +
     `VLLM_USE_AOT_COMPILE=0` names each break with its user stack (AOT compile
     hard-refuses breaks outright); `/home/dank/dynamo-probe` on `PYTHONPATH` with