frank-primary: patch 05, five verified artifacts, and correct the wrong confound theory
Root-caused the moving failures: they were my own scratch launcher. Its compile branches were restructured and the '-e VLLM_COMPILE_FULLGRAPH=0' line was dropped, so four consecutive boots mounted the wrapper patch while _FULLGRAPH still defaulted to True — and with fullgraph=True EVERY break is an error, so each boot reported whichever break came first (cell -> workspace -> compiler-disable). I had blamed TORCH_LOGS=graph_breaks for the difference; that was wrong, the flag only logs. The launcher sets the env var again, and patch 04's header, the runbook and the work log now carry the true run history as a table instead of the bad theory. New artifact: scripts/patches/glm53-compile-05-workspace-acquisition-opaque.patch — the indexer acquires its shared workspace INSIDE the forward (its profile-run branch literally says 'Reserve workspace for indexer during profiling run'), and v1/worker/workspace.py:_ensure_workspace_size reads ContextVar/ubatch state, which dynamo cannot trace. Wrap both call sites in a torch.compiler.disable'd helper (narrower than decorating the workspace manager, which every model would go through; a no-op when not compiling). All five artifacts verified against the pinned image: applied in sequence with patch -p1 they reproduce their recorded patched sha256 exactly (6e9cb81b…, 0751852e…, 732363bb…, 84969993…, c154115a…). tests/run.sh: 362 seams pass.
This commit is contained in:
@@ -197,42 +197,47 @@ What the boots established, in order (each one ~7 min):
|
||||
the GLM MLA block stops calling the fork's fused-norm JIT kernel, whose kernel
|
||||
object in `models/common/ops/fused_qk_rmsnorm.py` is bound *after* the function
|
||||
that uses it — `_FUSED_Q_KV_RMSNORM_KERNEL` at line 223 vs its use at line 211).
|
||||
**Its necessity is not yet established** — see the confound below.
|
||||
5. **A confound worth knowing about, because it cost two boots.** Two break-tolerance
|
||||
runs of the same config disagree, and the difference is one logging flag:
|
||||
`01+02+03` with `TORCH_LOGS=graph_breaks` dies on a fatal
|
||||
`Unsupported: Read uninitialized cell` at `mla.py:208` (the fused-norm call), while
|
||||
the same config **without** the flag gets past break handling entirely and dies
|
||||
later on the dynamo name-map `KeyError`. The likely mechanism: `graph_breaks`
|
||||
logging makes dynamo build an *explanation* for each break (walk frames, name the
|
||||
reason), and that analysis is what trips on the uninitialized cell — without the
|
||||
flag, the break is taken silently. So that fatal-looking cell error is probably an
|
||||
artifact of the diagnostic. Related: with `graph_breaks` on, the next reported
|
||||
failure is `Unsupported: Unsupported method call` at
|
||||
`sparse_indexer.py:132 sparse_attn_indexer_kpool → workspace.py:135
|
||||
get_simultaneous / :155 _ensure_workspace_size`, i.e. the indexer allocates its
|
||||
shared workspace *inside* the forward (its own comment: *"will allocate on first
|
||||
use"*) — also flagged under explanation, also not yet shown to block a plain
|
||||
break-tolerant compile.
|
||||
**It removes a real break** (under `fullgraph=True` the fused call was the first
|
||||
break dynamo reported; after this patch the report moves on to the workspace
|
||||
acquisition) — but it is not by itself what blocks a boot, because with breaks
|
||||
genuinely tolerated the blocker is the name-map `KeyError`. See the run history
|
||||
below.
|
||||
5. **A launcher bug that made four boots misleading — worth knowing because the
|
||||
failure kept moving.** The scratch launcher's compile branches were restructured
|
||||
on 2026-09-30 and the `-e VLLM_COMPILE_FULLGRAPH=0` line was dropped, so four
|
||||
consecutive boots mounted the wrapper patch while `_FULLGRAPH` still defaulted to
|
||||
True. With `fullgraph=True` **every** break is an error, so those boots reported
|
||||
whichever break came first:
|
||||
|
||||
| config | outcome |
|
||||
| --- | --- |
|
||||
| `fullgraph=False`, AOT off | breaks tolerated → dies later on the dynamo name-map `KeyError: 'in_proj_qkvbfg_a'` |
|
||||
| `fullgraph=True`, AOT off | dies on the first break (cell → workspace → compiler-disable) |
|
||||
|
||||
So the reported breaks are real *as breaks*, but they are not what blocks a
|
||||
break-tolerant compile — and an earlier version of this page blamed
|
||||
`TORCH_LOGS=graph_breaks` for the difference. **That was wrong**; the flag only
|
||||
logs. The launcher sets the env var again.
|
||||
|
||||
**How to see the breaks (and its trap).** `TORCH_LOGS=graph_breaks` names each break
|
||||
with a user stack, and `VLLM_USE_AOT_COMPILE=0` is required because AOT compile
|
||||
hard-refuses breaks — but per the confound above, treat the flagged fatal errors as
|
||||
*candidates*, and confirm each one against a flag-free run before believing it blocks
|
||||
the boot. For the name-map KeyError there is a ready-made probe: put
|
||||
`/home/dank/dynamo-probe` on `PYTHONPATH` with `VLLM_DYNAMO_DUMP_ATTRS=1` (a
|
||||
`sitecustomize.py` — a `.pth` on `PYTHONPATH` is NOT executed, and the image's own
|
||||
`sitecustomize` is chained so its apport hook survives) and it prints, at the miss,
|
||||
the node target, the near-miss keys, and whether another live `OutputGraph` holds
|
||||
the key.
|
||||
hard-refuses breaks — but a fullgraph=True run reports the first break as fatal, so
|
||||
check which configuration you are actually in (`podman exec <ctr> printenv
|
||||
VLLM_COMPILE_FULLGRAPH`) before concluding a break blocks the boot. For the name-map
|
||||
KeyError there is a ready-made probe: put `/home/dank/dynamo-probe` on `PYTHONPATH`
|
||||
with `VLLM_DYNAMO_DUMP_ATTRS=1` (a `sitecustomize.py` — a `.pth` on `PYTHONPATH` is
|
||||
NOT executed, and the image's own `sitecustomize` is chained so its apport hook
|
||||
survives) and it prints, at the miss, the node target, the near-miss keys, and
|
||||
whether another live `OutputGraph` holds the key.
|
||||
|
||||
**State of the changelist:** the trunk traces (`Dynamo bytecode transform time:
|
||||
1.08 s`), and the one failure confirmed by a flag-free run is the dynamo name-map
|
||||
`KeyError: 'in_proj_qkvbfg_a'`. Two further candidates are queued behind it
|
||||
(the fused-norm launcher cell, the indexer's in-forward workspace allocation). Two
|
||||
verified facts bound the rest of the shape: `suppress_errors` does **not** cover
|
||||
graph breaks (only `fullgraph=False` does), and the GLM trunk can never satisfy
|
||||
`fullgraph` because its indexer metadata syncs to host
|
||||
1.08 s`). With breaks genuinely tolerated the confirmed blocker is the dynamo
|
||||
name-map `KeyError: 'in_proj_qkvbfg_a'`. Two further breaks are known and worth
|
||||
removing regardless: the fused-norm launcher cell (patch 04 removes the call) and
|
||||
the indexer's in-forward workspace acquisition (patch 05 makes it
|
||||
compiler-opaque). Two verified facts bound the rest of the shape: `suppress_errors`
|
||||
does **not** cover graph breaks (only `fullgraph=False` does), and the GLM trunk can
|
||||
never satisfy `fullgraph` because its indexer metadata syncs to host
|
||||
(`glm5next/amd/sparse_indexer.py:263,419`: `int(...max().item())`) — both are dead
|
||||
fallbacks on the decode path that the fork author already replaced, so they are not
|
||||
the blockers they appear to be at first glance.
|
||||
|
||||
Reference in New Issue
Block a user