frank-primary: compile changelist — real diffs, break diagnostics, patch 04, corrected overclaim

Working the dyno name-map blocker one step at a time, with the tools to see it:

- All four patch artifacts are now genuine diff -u patches (they were prose + pseudo
  hunks before, useless to patch(1)). Verified against the pinned image: each applies
  clean and reproduces its recorded patched sha256 (6e9cb81b… / 0751852e… / 732363bb…
  / 84969993…).
- New: scripts/patches/glm53-compile-04-fused-qkv-rmsnorm-optout.patch — takes GLM's
  MLA block off the fused-norm JIT kernel (R4D_GLM_FUSE_QKV_RMSNORM=0; the flag is
  already a constructor arg glm5next sets at attention.py:583). Reason: in
  models/common/ops/fused_qk_rmsnorm.py the kernel object is bound at line 223, AFTER
  its use at line 211 — the launcher family (@kernel_launcher, 39 call sites) is what
  breaks dynamo's break analysis.
- New diagnostics, both reusable and recorded: TORCH_LOGS=graph_breaks +
  VLLM_USE_AOT_COMPILE=0 to name each break with its user stack, and
  /home/dank/dynamo-probe (sitecustomize.py on PYTHONPATH, VLLM_DYNAMO_DUMP_ATTRS=1)
  to print the dynamo name-map miss detail. Gotchas: a .pth on PYTHONPATH is NOT
  executed, and the scratch launcher reset ENVARGS (now ${ENVARGS:-}), which silently
  dropped env vars and wasted a boot.

Measurement discipline correction: I first recorded the 'Read uninitialized cell' and
'Unsupported method call' failures as fatal blockers. They are not established as
such — they appear only when TORCH_LOGS=graph_breaks is set, while a flag-free
break-tolerance run gets past break handling and dies on the name-map KeyError. The
likely mechanism is that graph_breaks logging makes dynamo build a break
*explanation*, and that analysis itself trips on the uninitialized cell. Patch 04's
header, the runbook and the work log now carry the confound explicitly rather than
the overclaim.

tests/run.sh: 362 seams pass.
This commit is contained in:
Comrade Chief Designer
2026-09-30 13:59:09 -07:00
parent 1c18b9c808
commit fd864b5a42
6 changed files with 220 additions and 19 deletions
@@ -193,13 +193,49 @@ What the boots established, in order (each one ~7 min):
`serve_glm_autoround.sh`'s COMPILE_PATCH branch rather than in the outer
environment).
That last row is the state of the work: the changelist is one unresolved
submodule-name lookup away from booting, and it must be cleared (plus whatever
follows it) before the compile payoff can be measured. Two verified facts make the
rest of the shape concrete: `suppress_errors` does **not** cover graph breaks (only
`fullgraph=False` does), and the GLM trunk can never satisfy `fullgraph` because
its indexer metadata syncs to host
(`glm5next/amd/sparse_indexer.py:263,419`: `int(...max().item())`).
4. **04 removes the fused-norm call from the traced region** (`R4D_GLM_FUSE_QKV_RMSNORM=0`:
the GLM MLA block stops calling the fork's fused-norm JIT kernel, whose kernel
object in `models/common/ops/fused_qk_rmsnorm.py` is bound *after* the function
that uses it — `_FUSED_Q_KV_RMSNORM_KERNEL` at line 223 vs its use at line 211).
**Its necessity is not yet established** — see the confound below.
5. **A confound worth knowing about, because it cost two boots.** Two break-tolerance
runs of the same config disagree, and the difference is one logging flag:
`01+02+03` with `TORCH_LOGS=graph_breaks` dies on a fatal
`Unsupported: Read uninitialized cell` at `mla.py:208` (the fused-norm call), while
the same config **without** the flag gets past break handling entirely and dies
later on the dynamo name-map `KeyError`. The likely mechanism: `graph_breaks`
logging makes dynamo build an *explanation* for each break (walk frames, name the
reason), and that analysis is what trips on the uninitialized cell — without the
flag, the break is taken silently. So that fatal-looking cell error is probably an
artifact of the diagnostic. Related: with `graph_breaks` on, the next reported
failure is `Unsupported: Unsupported method call` at
`sparse_indexer.py:132 sparse_attn_indexer_kpool → workspace.py:135
get_simultaneous / :155 _ensure_workspace_size`, i.e. the indexer allocates its
shared workspace *inside* the forward (its own comment: *"will allocate on first
use"*) — also flagged under explanation, also not yet shown to block a plain
break-tolerant compile.
**How to see the breaks (and its trap).** `TORCH_LOGS=graph_breaks` names each break
with a user stack, and `VLLM_USE_AOT_COMPILE=0` is required because AOT compile
hard-refuses breaks — but per the confound above, treat the flagged fatal errors as
*candidates*, and confirm each one against a flag-free run before believing it blocks
the boot. For the name-map KeyError there is a ready-made probe: put
`/home/dank/dynamo-probe` on `PYTHONPATH` with `VLLM_DYNAMO_DUMP_ATTRS=1` (a
`sitecustomize.py` — a `.pth` on `PYTHONPATH` is NOT executed, and the image's own
`sitecustomize` is chained so its apport hook survives) and it prints, at the miss,
the node target, the near-miss keys, and whether another live `OutputGraph` holds
the key.
**State of the changelist:** the trunk traces (`Dynamo bytecode transform time:
1.08 s`), and the one failure confirmed by a flag-free run is the dynamo name-map
`KeyError: 'in_proj_qkvbfg_a'`. Two further candidates are queued behind it
(the fused-norm launcher cell, the indexer's in-forward workspace allocation). Two
verified facts bound the rest of the shape: `suppress_errors` does **not** cover
graph breaks (only `fullgraph=False` does), and the GLM trunk can never satisfy
`fullgraph` because its indexer metadata syncs to host
(`glm5next/amd/sparse_indexer.py:263,419`: `int(...max().item())`) — both are dead
fallbacks on the decode path that the fork author already replaced, so they are not
the blockers they appear to be at first glance.
### Two structural constraints on any further fork work