frank-primary: patch 05, five verified artifacts, and correct the wrong confound theory

Root-caused the moving failures: they were my own scratch launcher. Its compile
branches were restructured and the '-e VLLM_COMPILE_FULLGRAPH=0' line was dropped, so
four consecutive boots mounted the wrapper patch while _FULLGRAPH still defaulted to
True — and with fullgraph=True EVERY break is an error, so each boot reported whichever
break came first (cell -> workspace -> compiler-disable). I had blamed
TORCH_LOGS=graph_breaks for the difference; that was wrong, the flag only logs. The
launcher sets the env var again, and patch 04's header, the runbook and the work log
now carry the true run history as a table instead of the bad theory.

New artifact: scripts/patches/glm53-compile-05-workspace-acquisition-opaque.patch —
the indexer acquires its shared workspace INSIDE the forward (its profile-run branch
literally says 'Reserve workspace for indexer during profiling run'), and
v1/worker/workspace.py:_ensure_workspace_size reads ContextVar/ubatch state, which
dynamo cannot trace. Wrap both call sites in a torch.compiler.disable'd helper
(narrower than decorating the workspace manager, which every model would go through;
a no-op when not compiling).

All five artifacts verified against the pinned image: applied in sequence with patch
-p1 they reproduce their recorded patched sha256 exactly (6e9cb81b…, 0751852e…,
732363bb…, 84969993…, c154115a…).

tests/run.sh: 362 seams pass.
This commit is contained in:
Comrade Chief Designer
2026-09-30 14:19:25 -07:00
parent fd864b5a42
commit 71db0debcc
4 changed files with 166 additions and 63 deletions
+17 -16
View File
@@ -74,23 +74,24 @@
`RuntimeError: Model does not support EAGLE3 interface` — nothing about the
drafter, the method or the payload is wrong; the trunk simply does not
implement the aux-hidden-state contract.
- **Compile work: one confirmed blocker, plus a measurement confound worth
recording.** Flag-free break-tolerance runs (01+02+03, AOT off, breaks allowed)
get past break handling and die on the dynamo name-map `KeyError:
'in_proj_qkvbfg_a'`. Adding `TORCH_LOGS=graph_breaks` to the same config changes
where it dies: a fatal-looking `Unsupported: Read uninitialized cell` at
`mla.py:208` calling the fork's fused-norm JIT kernel
(`models/common/ops/fused_qk_rmsnorm.py`: `_FUSED_Q_KV_RMSNORM_KERNEL` is bound at
line 223, *after* its use at line 211), and with that removed, `Unsupported:
Unsupported method call` in the indexer's in-forward workspace allocation
- **Compile work: one confirmed blocker, and a launcher bug that misled four boots.**
With breaks genuinely tolerated (`fullgraph=False`, AOT off) the trunk traces and
dies on the dynamo name-map `KeyError: 'in_proj_qkvbfg_a'`. Four later boots
reported *different* fatal breaks — `Unsupported: Read uninitialized cell` at
`mla.py:208` (the fused-norm JIT kernel), then `Unsupported method call` in the
indexer's in-forward workspace acquisition
(`sparse_indexer.py:132 -> workspace.py:135 get_simultaneous / :155
_ensure_workspace_size`). **The likely mechanism is that `graph_breaks` logging
makes dynamo build a break *explanation*, and that analysis is what trips** — so
those two are candidates, not proven blockers, and I overclaimed them once before
catching it. `scripts/patches/glm53-compile-04-fused-qkv-rmsnorm-optout.patch` takes
GLM off the fused-norm kernel anyway (`R4D_GLM_FUSE_QKV_RMSNORM=0`, a constructor
flag glm5next sets at `attention.py:583`); whether it is *needed* is what the
flag-free 01+02+03+04 run settles.
_ensure_workspace_size`), then `Skip calling torch.compiler.disable()'d function`
— and the cause was **my own scratch launcher**: its compile branches were
restructured and the `-e VLLM_COMPILE_FULLGRAPH=0` line was dropped, so those
boots ran `fullgraph=True`, where *every* break is an error and the first one
wins. I first blamed `TORCH_LOGS=graph_breaks` for the difference; that was wrong
(the flag only logs). The moving reports are still useful — they are the break
list — and patch 04 demonstrably removes the first of them (the fused-norm cell),
patch 05 makes the workspace acquisition compiler-opaque.
`scripts/patches/glm53-compile-04-fused-qkv-rmsnorm-optout.patch` takes GLM off
the fused-norm kernel (`R4D_GLM_FUSE_QKV_RMSNORM=0`, a constructor flag glm5next
sets at `attention.py:583`).
- **Diagnostics built for this (reusable).** `TORCH_LOGS=graph_breaks` +
`VLLM_USE_AOT_COMPILE=0` names each break with its user stack (AOT compile
hard-refuses breaks outright); `/home/dank/dynamo-probe` on `PYTHONPATH` with