frank-primary: patch 05, five verified artifacts, and correct the wrong confound theory
Root-caused the moving failures: they were my own scratch launcher. Its compile branches were restructured and the '-e VLLM_COMPILE_FULLGRAPH=0' line was dropped, so four consecutive boots mounted the wrapper patch while _FULLGRAPH still defaulted to True — and with fullgraph=True EVERY break is an error, so each boot reported whichever break came first (cell -> workspace -> compiler-disable). I had blamed TORCH_LOGS=graph_breaks for the difference; that was wrong, the flag only logs. The launcher sets the env var again, and patch 04's header, the runbook and the work log now carry the true run history as a table instead of the bad theory. New artifact: scripts/patches/glm53-compile-05-workspace-acquisition-opaque.patch — the indexer acquires its shared workspace INSIDE the forward (its profile-run branch literally says 'Reserve workspace for indexer during profiling run'), and v1/worker/workspace.py:_ensure_workspace_size reads ContextVar/ubatch state, which dynamo cannot trace. Wrap both call sites in a torch.compiler.disable'd helper (narrower than decorating the workspace manager, which every model would go through; a no-op when not compiling). All five artifacts verified against the pinned image: applied in sequence with patch -p1 they reproduce their recorded patched sha256 exactly (6e9cb81b…, 0751852e…, 732363bb…, 84969993…, c154115a…). tests/run.sh: 362 seams pass.
This commit is contained in:
@@ -74,23 +74,24 @@
|
||||
`RuntimeError: Model does not support EAGLE3 interface` — nothing about the
|
||||
drafter, the method or the payload is wrong; the trunk simply does not
|
||||
implement the aux-hidden-state contract.
|
||||
- **Compile work: one confirmed blocker, plus a measurement confound worth
|
||||
recording.** Flag-free break-tolerance runs (01+02+03, AOT off, breaks allowed)
|
||||
get past break handling and die on the dynamo name-map `KeyError:
|
||||
'in_proj_qkvbfg_a'`. Adding `TORCH_LOGS=graph_breaks` to the same config changes
|
||||
where it dies: a fatal-looking `Unsupported: Read uninitialized cell` at
|
||||
`mla.py:208` calling the fork's fused-norm JIT kernel
|
||||
(`models/common/ops/fused_qk_rmsnorm.py`: `_FUSED_Q_KV_RMSNORM_KERNEL` is bound at
|
||||
line 223, *after* its use at line 211), and with that removed, `Unsupported:
|
||||
Unsupported method call` in the indexer's in-forward workspace allocation
|
||||
- **Compile work: one confirmed blocker, and a launcher bug that misled four boots.**
|
||||
With breaks genuinely tolerated (`fullgraph=False`, AOT off) the trunk traces and
|
||||
dies on the dynamo name-map `KeyError: 'in_proj_qkvbfg_a'`. Four later boots
|
||||
reported *different* fatal breaks — `Unsupported: Read uninitialized cell` at
|
||||
`mla.py:208` (the fused-norm JIT kernel), then `Unsupported method call` in the
|
||||
indexer's in-forward workspace acquisition
|
||||
(`sparse_indexer.py:132 -> workspace.py:135 get_simultaneous / :155
|
||||
_ensure_workspace_size`). **The likely mechanism is that `graph_breaks` logging
|
||||
makes dynamo build a break *explanation*, and that analysis is what trips** — so
|
||||
those two are candidates, not proven blockers, and I overclaimed them once before
|
||||
catching it. `scripts/patches/glm53-compile-04-fused-qkv-rmsnorm-optout.patch` takes
|
||||
GLM off the fused-norm kernel anyway (`R4D_GLM_FUSE_QKV_RMSNORM=0`, a constructor
|
||||
flag glm5next sets at `attention.py:583`); whether it is *needed* is what the
|
||||
flag-free 01+02+03+04 run settles.
|
||||
_ensure_workspace_size`), then `Skip calling torch.compiler.disable()'d function`
|
||||
— and the cause was **my own scratch launcher**: its compile branches were
|
||||
restructured and the `-e VLLM_COMPILE_FULLGRAPH=0` line was dropped, so those
|
||||
boots ran `fullgraph=True`, where *every* break is an error and the first one
|
||||
wins. I first blamed `TORCH_LOGS=graph_breaks` for the difference; that was wrong
|
||||
(the flag only logs). The moving reports are still useful — they are the break
|
||||
list — and patch 04 demonstrably removes the first of them (the fused-norm cell),
|
||||
patch 05 makes the workspace acquisition compiler-opaque.
|
||||
`scripts/patches/glm53-compile-04-fused-qkv-rmsnorm-optout.patch` takes GLM off
|
||||
the fused-norm kernel (`R4D_GLM_FUSE_QKV_RMSNORM=0`, a constructor flag glm5next
|
||||
sets at `attention.py:583`).
|
||||
- **Diagnostics built for this (reusable).** `TORCH_LOGS=graph_breaks` +
|
||||
`VLLM_USE_AOT_COMPILE=0` names each break with its user stack (AOT compile
|
||||
hard-refuses breaks outright); `/home/dank/dynamo-probe` on `PYTHONPATH` with
|
||||
|
||||
Reference in New Issue
Block a user