frank-primary: compile changelist — real diffs, break diagnostics, patch 04, corrected overclaim
Working the dyno name-map blocker one step at a time, with the tools to see it:
- All four patch artifacts are now genuine diff -u patches (they were prose + pseudo
hunks before, useless to patch(1)). Verified against the pinned image: each applies
clean and reproduces its recorded patched sha256 (6e9cb81b… / 0751852e… / 732363bb…
/ 84969993…).
- New: scripts/patches/glm53-compile-04-fused-qkv-rmsnorm-optout.patch — takes GLM's
MLA block off the fused-norm JIT kernel (R4D_GLM_FUSE_QKV_RMSNORM=0; the flag is
already a constructor arg glm5next sets at attention.py:583). Reason: in
models/common/ops/fused_qk_rmsnorm.py the kernel object is bound at line 223, AFTER
its use at line 211 — the launcher family (@kernel_launcher, 39 call sites) is what
breaks dynamo's break analysis.
- New diagnostics, both reusable and recorded: TORCH_LOGS=graph_breaks +
VLLM_USE_AOT_COMPILE=0 to name each break with its user stack, and
/home/dank/dynamo-probe (sitecustomize.py on PYTHONPATH, VLLM_DYNAMO_DUMP_ATTRS=1)
to print the dynamo name-map miss detail. Gotchas: a .pth on PYTHONPATH is NOT
executed, and the scratch launcher reset ENVARGS (now ${ENVARGS:-}), which silently
dropped env vars and wasted a boot.
Measurement discipline correction: I first recorded the 'Read uninitialized cell' and
'Unsupported method call' failures as fatal blockers. They are not established as
such — they appear only when TORCH_LOGS=graph_breaks is set, while a flag-free
break-tolerance run gets past break handling and dies on the name-map KeyError. The
likely mechanism is that graph_breaks logging makes dynamo build a break
*explanation*, and that analysis itself trips on the uninitialized cell. Patch 04's
header, the runbook and the work log now carry the confound explicitly rather than
the overclaim.
tests/run.sh: 362 seams pass.
This commit is contained in:
@@ -193,13 +193,49 @@ What the boots established, in order (each one ~7 min):
|
||||
`serve_glm_autoround.sh`'s COMPILE_PATCH branch rather than in the outer
|
||||
environment).
|
||||
|
||||
That last row is the state of the work: the changelist is one unresolved
|
||||
submodule-name lookup away from booting, and it must be cleared (plus whatever
|
||||
follows it) before the compile payoff can be measured. Two verified facts make the
|
||||
rest of the shape concrete: `suppress_errors` does **not** cover graph breaks (only
|
||||
`fullgraph=False` does), and the GLM trunk can never satisfy `fullgraph` because
|
||||
its indexer metadata syncs to host
|
||||
(`glm5next/amd/sparse_indexer.py:263,419`: `int(...max().item())`).
|
||||
4. **04 removes the fused-norm call from the traced region** (`R4D_GLM_FUSE_QKV_RMSNORM=0`:
|
||||
the GLM MLA block stops calling the fork's fused-norm JIT kernel, whose kernel
|
||||
object in `models/common/ops/fused_qk_rmsnorm.py` is bound *after* the function
|
||||
that uses it — `_FUSED_Q_KV_RMSNORM_KERNEL` at line 223 vs its use at line 211).
|
||||
**Its necessity is not yet established** — see the confound below.
|
||||
5. **A confound worth knowing about, because it cost two boots.** Two break-tolerance
|
||||
runs of the same config disagree, and the difference is one logging flag:
|
||||
`01+02+03` with `TORCH_LOGS=graph_breaks` dies on a fatal
|
||||
`Unsupported: Read uninitialized cell` at `mla.py:208` (the fused-norm call), while
|
||||
the same config **without** the flag gets past break handling entirely and dies
|
||||
later on the dynamo name-map `KeyError`. The likely mechanism: `graph_breaks`
|
||||
logging makes dynamo build an *explanation* for each break (walk frames, name the
|
||||
reason), and that analysis is what trips on the uninitialized cell — without the
|
||||
flag, the break is taken silently. So that fatal-looking cell error is probably an
|
||||
artifact of the diagnostic. Related: with `graph_breaks` on, the next reported
|
||||
failure is `Unsupported: Unsupported method call` at
|
||||
`sparse_indexer.py:132 sparse_attn_indexer_kpool → workspace.py:135
|
||||
get_simultaneous / :155 _ensure_workspace_size`, i.e. the indexer allocates its
|
||||
shared workspace *inside* the forward (its own comment: *"will allocate on first
|
||||
use"*) — also flagged under explanation, also not yet shown to block a plain
|
||||
break-tolerant compile.
|
||||
|
||||
**How to see the breaks (and its trap).** `TORCH_LOGS=graph_breaks` names each break
|
||||
with a user stack, and `VLLM_USE_AOT_COMPILE=0` is required because AOT compile
|
||||
hard-refuses breaks — but per the confound above, treat the flagged fatal errors as
|
||||
*candidates*, and confirm each one against a flag-free run before believing it blocks
|
||||
the boot. For the name-map KeyError there is a ready-made probe: put
|
||||
`/home/dank/dynamo-probe` on `PYTHONPATH` with `VLLM_DYNAMO_DUMP_ATTRS=1` (a
|
||||
`sitecustomize.py` — a `.pth` on `PYTHONPATH` is NOT executed, and the image's own
|
||||
`sitecustomize` is chained so its apport hook survives) and it prints, at the miss,
|
||||
the node target, the near-miss keys, and whether another live `OutputGraph` holds
|
||||
the key.
|
||||
|
||||
**State of the changelist:** the trunk traces (`Dynamo bytecode transform time:
|
||||
1.08 s`), and the one failure confirmed by a flag-free run is the dynamo name-map
|
||||
`KeyError: 'in_proj_qkvbfg_a'`. Two further candidates are queued behind it
|
||||
(the fused-norm launcher cell, the indexer's in-forward workspace allocation). Two
|
||||
verified facts bound the rest of the shape: `suppress_errors` does **not** cover
|
||||
graph breaks (only `fullgraph=False` does), and the GLM trunk can never satisfy
|
||||
`fullgraph` because its indexer metadata syncs to host
|
||||
(`glm5next/amd/sparse_indexer.py:263,419`: `int(...max().item())`) — both are dead
|
||||
fallbacks on the decode path that the fork author already replaced, so they are not
|
||||
the blockers they appear to be at first glance.
|
||||
|
||||
### Two structural constraints on any further fork work
|
||||
|
||||
|
||||
Reference in New Issue
Block a user