frank-primary: patch 05, five verified artifacts, and correct the wrong confound theory

Root-caused the moving failures: they were my own scratch launcher. Its compile
branches were restructured and the '-e VLLM_COMPILE_FULLGRAPH=0' line was dropped, so
four consecutive boots mounted the wrapper patch while _FULLGRAPH still defaulted to
True — and with fullgraph=True EVERY break is an error, so each boot reported whichever
break came first (cell -> workspace -> compiler-disable). I had blamed
TORCH_LOGS=graph_breaks for the difference; that was wrong, the flag only logs. The
launcher sets the env var again, and patch 04's header, the runbook and the work log
now carry the true run history as a table instead of the bad theory.

New artifact: scripts/patches/glm53-compile-05-workspace-acquisition-opaque.patch —
the indexer acquires its shared workspace INSIDE the forward (its profile-run branch
literally says 'Reserve workspace for indexer during profiling run'), and
v1/worker/workspace.py:_ensure_workspace_size reads ContextVar/ubatch state, which
dynamo cannot trace. Wrap both call sites in a torch.compiler.disable'd helper
(narrower than decorating the workspace manager, which every model would go through;
a no-op when not compiling).

All five artifacts verified against the pinned image: applied in sequence with patch
-p1 they reproduce their recorded patched sha256 exactly (6e9cb81b…, 0751852e…,
732363bb…, 84969993…, c154115a…).

tests/run.sh: 362 seams pass.
This commit is contained in:
Comrade Chief Designer
2026-09-30 14:19:25 -07:00
parent fd864b5a42
commit 71db0debcc
4 changed files with 166 additions and 63 deletions
@@ -197,42 +197,47 @@ What the boots established, in order (each one ~7 min):
the GLM MLA block stops calling the fork's fused-norm JIT kernel, whose kernel
object in `models/common/ops/fused_qk_rmsnorm.py` is bound *after* the function
that uses it — `_FUSED_Q_KV_RMSNORM_KERNEL` at line 223 vs its use at line 211).
**Its necessity is not yet established** — see the confound below.
5. **A confound worth knowing about, because it cost two boots.** Two break-tolerance
runs of the same config disagree, and the difference is one logging flag:
`01+02+03` with `TORCH_LOGS=graph_breaks` dies on a fatal
`Unsupported: Read uninitialized cell` at `mla.py:208` (the fused-norm call), while
the same config **without** the flag gets past break handling entirely and dies
later on the dynamo name-map `KeyError`. The likely mechanism: `graph_breaks`
logging makes dynamo build an *explanation* for each break (walk frames, name the
reason), and that analysis is what trips on the uninitialized cell — without the
flag, the break is taken silently. So that fatal-looking cell error is probably an
artifact of the diagnostic. Related: with `graph_breaks` on, the next reported
failure is `Unsupported: Unsupported method call` at
`sparse_indexer.py:132 sparse_attn_indexer_kpool → workspace.py:135
get_simultaneous / :155 _ensure_workspace_size`, i.e. the indexer allocates its
shared workspace *inside* the forward (its own comment: *"will allocate on first
use"*) — also flagged under explanation, also not yet shown to block a plain
break-tolerant compile.
**It removes a real break** (under `fullgraph=True` the fused call was the first
break dynamo reported; after this patch the report moves on to the workspace
acquisition) — but it is not by itself what blocks a boot, because with breaks
genuinely tolerated the blocker is the name-map `KeyError`. See the run history
below.
5. **A launcher bug that made four boots misleading — worth knowing because the
failure kept moving.** The scratch launcher's compile branches were restructured
on 2026-09-30 and the `-e VLLM_COMPILE_FULLGRAPH=0` line was dropped, so four
consecutive boots mounted the wrapper patch while `_FULLGRAPH` still defaulted to
True. With `fullgraph=True` **every** break is an error, so those boots reported
whichever break came first:
| config | outcome |
| --- | --- |
| `fullgraph=False`, AOT off | breaks tolerated → dies later on the dynamo name-map `KeyError: 'in_proj_qkvbfg_a'` |
| `fullgraph=True`, AOT off | dies on the first break (cell → workspace → compiler-disable) |
So the reported breaks are real *as breaks*, but they are not what blocks a
break-tolerant compile — and an earlier version of this page blamed
`TORCH_LOGS=graph_breaks` for the difference. **That was wrong**; the flag only
logs. The launcher sets the env var again.
**How to see the breaks (and its trap).** `TORCH_LOGS=graph_breaks` names each break
with a user stack, and `VLLM_USE_AOT_COMPILE=0` is required because AOT compile
hard-refuses breaks — but per the confound above, treat the flagged fatal errors as
*candidates*, and confirm each one against a flag-free run before believing it blocks
the boot. For the name-map KeyError there is a ready-made probe: put
`/home/dank/dynamo-probe` on `PYTHONPATH` with `VLLM_DYNAMO_DUMP_ATTRS=1` (a
`sitecustomize.py` — a `.pth` on `PYTHONPATH` is NOT executed, and the image's own
`sitecustomize` is chained so its apport hook survives) and it prints, at the miss,
the node target, the near-miss keys, and whether another live `OutputGraph` holds
the key.
hard-refuses breaks — but a fullgraph=True run reports the first break as fatal, so
check which configuration you are actually in (`podman exec <ctr> printenv
VLLM_COMPILE_FULLGRAPH`) before concluding a break blocks the boot. For the name-map
KeyError there is a ready-made probe: put `/home/dank/dynamo-probe` on `PYTHONPATH`
with `VLLM_DYNAMO_DUMP_ATTRS=1` (a `sitecustomize.py` — a `.pth` on `PYTHONPATH` is
NOT executed, and the image's own `sitecustomize` is chained so its apport hook
survives) and it prints, at the miss, the node target, the near-miss keys, and
whether another live `OutputGraph` holds the key.
**State of the changelist:** the trunk traces (`Dynamo bytecode transform time:
1.08 s`), and the one failure confirmed by a flag-free run is the dynamo name-map
`KeyError: 'in_proj_qkvbfg_a'`. Two further candidates are queued behind it
(the fused-norm launcher cell, the indexer's in-forward workspace allocation). Two
verified facts bound the rest of the shape: `suppress_errors` does **not** cover
graph breaks (only `fullgraph=False` does), and the GLM trunk can never satisfy
`fullgraph` because its indexer metadata syncs to host
1.08 s`). With breaks genuinely tolerated the confirmed blocker is the dynamo
name-map `KeyError: 'in_proj_qkvbfg_a'`. Two further breaks are known and worth
removing regardless: the fused-norm launcher cell (patch 04 removes the call) and
the indexer's in-forward workspace acquisition (patch 05 makes it
compiler-opaque). Two verified facts bound the rest of the shape: `suppress_errors`
does **not** cover graph breaks (only `fullgraph=False` does), and the GLM trunk can
never satisfy `fullgraph` because its indexer metadata syncs to host
(`glm5next/amd/sparse_indexer.py:263,419`: `int(...max().item())`) — both are dead
fallbacks on the decode path that the fork author already replaced, so they are not
the blockers they appear to be at first glance.