frank-primary: patch 05, five verified artifacts, and correct the wrong confound theory

Root-caused the moving failures: they were my own scratch launcher. Its compile
branches were restructured and the '-e VLLM_COMPILE_FULLGRAPH=0' line was dropped, so
four consecutive boots mounted the wrapper patch while _FULLGRAPH still defaulted to
True — and with fullgraph=True EVERY break is an error, so each boot reported whichever
break came first (cell -> workspace -> compiler-disable). I had blamed
TORCH_LOGS=graph_breaks for the difference; that was wrong, the flag only logs. The
launcher sets the env var again, and patch 04's header, the runbook and the work log
now carry the true run history as a table instead of the bad theory.

New artifact: scripts/patches/glm53-compile-05-workspace-acquisition-opaque.patch —
the indexer acquires its shared workspace INSIDE the forward (its profile-run branch
literally says 'Reserve workspace for indexer during profiling run'), and
v1/worker/workspace.py:_ensure_workspace_size reads ContextVar/ubatch state, which
dynamo cannot trace. Wrap both call sites in a torch.compiler.disable'd helper
(narrower than decorating the workspace manager, which every model would go through;
a no-op when not compiling).

All five artifacts verified against the pinned image: applied in sequence with patch
-p1 they reproduce their recorded patched sha256 exactly (6e9cb81b…, 0751852e…,
732363bb…, 84969993…, c154115a…).

tests/run.sh: 362 seams pass.

serverAppConfigs-docs/LLM/frank-primary-glm53-flash-w4a16.md

@@ -197,42 +197,47 @@ What the boots established, in order (each one ~7 min):
    the GLM MLA block stops calling the fork's fused-norm JIT kernel, whose kernel
    object in `models/common/ops/fused_qk_rmsnorm.py` is bound *after* the function
    that uses it — `_FUSED_Q_KV_RMSNORM_KERNEL` at line 223 vs its use at line 211).
-   **Its necessity is not yet established** — see the confound below.
-5. **A confound worth knowing about, because it cost two boots.** Two break-tolerance
-   runs of the same config disagree, and the difference is one logging flag:
-   `01+02+03` with `TORCH_LOGS=graph_breaks` dies on a fatal
-   `Unsupported: Read uninitialized cell` at `mla.py:208` (the fused-norm call), while
-   the same config **without** the flag gets past break handling entirely and dies
-   later on the dynamo name-map `KeyError`. The likely mechanism: `graph_breaks`
-   logging makes dynamo build an *explanation* for each break (walk frames, name the
-   reason), and that analysis is what trips on the uninitialized cell — without the
-   flag, the break is taken silently. So that fatal-looking cell error is probably an
-   artifact of the diagnostic. Related: with `graph_breaks` on, the next reported
-   failure is `Unsupported: Unsupported method call` at
-   `sparse_indexer.py:132 sparse_attn_indexer_kpool → workspace.py:135
-   get_simultaneous / :155 _ensure_workspace_size`, i.e. the indexer allocates its
-   shared workspace *inside* the forward (its own comment: *"will allocate on first
-   use"*) — also flagged under explanation, also not yet shown to block a plain
-   break-tolerant compile.
+   **It removes a real break** (under `fullgraph=True` the fused call was the first
+   break dynamo reported; after this patch the report moves on to the workspace
+   acquisition) — but it is not by itself what blocks a boot, because with breaks
+   genuinely tolerated the blocker is the name-map `KeyError`. See the run history
+   below.
+5. **A launcher bug that made four boots misleading — worth knowing because the
+   failure kept moving.** The scratch launcher's compile branches were restructured
+   on 2026-09-30 and the `-e VLLM_COMPILE_FULLGRAPH=0` line was dropped, so four
+   consecutive boots mounted the wrapper patch while `_FULLGRAPH` still defaulted to
+   True. With `fullgraph=True` **every** break is an error, so those boots reported
+   whichever break came first:
+
+   | config | outcome |
+   | --- | --- |
+   | `fullgraph=False`, AOT off | breaks tolerated → dies later on the dynamo name-map `KeyError: 'in_proj_qkvbfg_a'` |
+   | `fullgraph=True`, AOT off | dies on the first break (cell → workspace → compiler-disable) |
+
+   So the reported breaks are real *as breaks*, but they are not what blocks a
+   break-tolerant compile — and an earlier version of this page blamed
+   `TORCH_LOGS=graph_breaks` for the difference. **That was wrong**; the flag only
+   logs. The launcher sets the env var again.
 
 **How to see the breaks (and its trap).** `TORCH_LOGS=graph_breaks` names each break
 with a user stack, and `VLLM_USE_AOT_COMPILE=0` is required because AOT compile
-hard-refuses breaks — but per the confound above, treat the flagged fatal errors as
-*candidates*, and confirm each one against a flag-free run before believing it blocks
-the boot. For the name-map KeyError there is a ready-made probe: put
-`/home/dank/dynamo-probe` on `PYTHONPATH` with `VLLM_DYNAMO_DUMP_ATTRS=1` (a
-`sitecustomize.py` — a `.pth` on `PYTHONPATH` is NOT executed, and the image's own
-`sitecustomize` is chained so its apport hook survives) and it prints, at the miss,
-the node target, the near-miss keys, and whether another live `OutputGraph` holds
-the key.
+hard-refuses breaks — but a fullgraph=True run reports the first break as fatal, so
+check which configuration you are actually in (`podman exec <ctr> printenv
+VLLM_COMPILE_FULLGRAPH`) before concluding a break blocks the boot. For the name-map
+KeyError there is a ready-made probe: put `/home/dank/dynamo-probe` on `PYTHONPATH`
+with `VLLM_DYNAMO_DUMP_ATTRS=1` (a `sitecustomize.py` — a `.pth` on `PYTHONPATH` is
+NOT executed, and the image's own `sitecustomize` is chained so its apport hook
+survives) and it prints, at the miss, the node target, the near-miss keys, and
+whether another live `OutputGraph` holds the key.
 
 **State of the changelist:** the trunk traces (`Dynamo bytecode transform time:
-1.08 s`), and the one failure confirmed by a flag-free run is the dynamo name-map
-`KeyError: 'in_proj_qkvbfg_a'`. Two further candidates are queued behind it
-(the fused-norm launcher cell, the indexer's in-forward workspace allocation). Two
-verified facts bound the rest of the shape: `suppress_errors` does **not** cover
-graph breaks (only `fullgraph=False` does), and the GLM trunk can never satisfy
-`fullgraph` because its indexer metadata syncs to host
+1.08 s`). With breaks genuinely tolerated the confirmed blocker is the dynamo
+name-map `KeyError: 'in_proj_qkvbfg_a'`. Two further breaks are known and worth
+removing regardless: the fused-norm launcher cell (patch 04 removes the call) and
+the indexer's in-forward workspace acquisition (patch 05 makes it
+compiler-opaque). Two verified facts bound the rest of the shape: `suppress_errors`
+does **not** cover graph breaks (only `fullgraph=False` does), and the GLM trunk can
+never satisfy `fullgraph` because its indexer metadata syncs to host
 (`glm5next/amd/sparse_indexer.py:263,419`: `int(...max().item())`) — both are dead
 fallbacks on the decode path that the fork author already replaced, so they are not
 the blockers they appear to be at first glance.

serverAppConfigs-docs/log.md

@@ -74,23 +74,24 @@
     `RuntimeError: Model does not support EAGLE3 interface` — nothing about the
     drafter, the method or the payload is wrong; the trunk simply does not
     implement the aux-hidden-state contract.
-  - **Compile work: one confirmed blocker, plus a measurement confound worth
-    recording.** Flag-free break-tolerance runs (01+02+03, AOT off, breaks allowed)
-    get past break handling and die on the dynamo name-map `KeyError:
-    'in_proj_qkvbfg_a'`. Adding `TORCH_LOGS=graph_breaks` to the same config changes
-    where it dies: a fatal-looking `Unsupported: Read uninitialized cell` at
-    `mla.py:208` calling the fork's fused-norm JIT kernel
-    (`models/common/ops/fused_qk_rmsnorm.py`: `_FUSED_Q_KV_RMSNORM_KERNEL` is bound at
-    line 223, *after* its use at line 211), and with that removed, `Unsupported:
-    Unsupported method call` in the indexer's in-forward workspace allocation
+  - **Compile work: one confirmed blocker, and a launcher bug that misled four boots.**
+    With breaks genuinely tolerated (`fullgraph=False`, AOT off) the trunk traces and
+    dies on the dynamo name-map `KeyError: 'in_proj_qkvbfg_a'`. Four later boots
+    reported *different* fatal breaks — `Unsupported: Read uninitialized cell` at
+    `mla.py:208` (the fused-norm JIT kernel), then `Unsupported method call` in the
+    indexer's in-forward workspace acquisition
     (`sparse_indexer.py:132 -> workspace.py:135 get_simultaneous / :155
-    _ensure_workspace_size`). **The likely mechanism is that `graph_breaks` logging
-    makes dynamo build a break *explanation*, and that analysis is what trips** — so
-    those two are candidates, not proven blockers, and I overclaimed them once before
-    catching it. `scripts/patches/glm53-compile-04-fused-qkv-rmsnorm-optout.patch` takes
-    GLM off the fused-norm kernel anyway (`R4D_GLM_FUSE_QKV_RMSNORM=0`, a constructor
-    flag glm5next sets at `attention.py:583`); whether it is *needed* is what the
-    flag-free 01+02+03+04 run settles.
+    _ensure_workspace_size`), then `Skip calling torch.compiler.disable()'d function`
+    — and the cause was **my own scratch launcher**: its compile branches were
+    restructured and the `-e VLLM_COMPILE_FULLGRAPH=0` line was dropped, so those
+    boots ran `fullgraph=True`, where *every* break is an error and the first one
+    wins. I first blamed `TORCH_LOGS=graph_breaks` for the difference; that was wrong
+    (the flag only logs). The moving reports are still useful — they are the break
+    list — and patch 04 demonstrably removes the first of them (the fused-norm cell),
+    patch 05 makes the workspace acquisition compiler-opaque.
+    `scripts/patches/glm53-compile-04-fused-qkv-rmsnorm-optout.patch` takes GLM off
+    the fused-norm kernel (`R4D_GLM_FUSE_QKV_RMSNORM=0`, a constructor flag glm5next
+    sets at `attention.py:583`).
   - **Diagnostics built for this (reusable).** `TORCH_LOGS=graph_breaks` +
     `VLLM_USE_AOT_COMPILE=0` names each break with its user stack (AOT compile
     hard-refuses breaks outright); `/home/dank/dynamo-probe` on `PYTHONPATH` with