frank-primary: compile changelist — real diffs, break diagnostics, patch 04, corrected overclaim

Working the dyno name-map blocker one step at a time, with the tools to see it:

- All four patch artifacts are now genuine diff -u patches (they were prose + pseudo
  hunks before, useless to patch(1)). Verified against the pinned image: each applies
  clean and reproduces its recorded patched sha256 (6e9cb81b… / 0751852e… / 732363bb…
  / 84969993…).
- New: scripts/patches/glm53-compile-04-fused-qkv-rmsnorm-optout.patch — takes GLM's
  MLA block off the fused-norm JIT kernel (R4D_GLM_FUSE_QKV_RMSNORM=0; the flag is
  already a constructor arg glm5next sets at attention.py:583). Reason: in
  models/common/ops/fused_qk_rmsnorm.py the kernel object is bound at line 223, AFTER
  its use at line 211 — the launcher family (@kernel_launcher, 39 call sites) is what
  breaks dynamo's break analysis.
- New diagnostics, both reusable and recorded: TORCH_LOGS=graph_breaks +
  VLLM_USE_AOT_COMPILE=0 to name each break with its user stack, and
  /home/dank/dynamo-probe (sitecustomize.py on PYTHONPATH, VLLM_DYNAMO_DUMP_ATTRS=1)
  to print the dynamo name-map miss detail. Gotchas: a .pth on PYTHONPATH is NOT
  executed, and the scratch launcher reset ENVARGS (now ${ENVARGS:-}), which silently
  dropped env vars and wasted a boot.

Measurement discipline correction: I first recorded the 'Read uninitialized cell' and
'Unsupported method call' failures as fatal blockers. They are not established as
such — they appear only when TORCH_LOGS=graph_breaks is set, while a flag-free
break-tolerance run gets past break handling and dies on the name-map KeyError. The
likely mechanism is that graph_breaks logging makes dynamo build a break
*explanation*, and that analysis itself trips on the uninitialized cell. Patch 04's
header, the runbook and the work log now carry the confound explicitly rather than
the overclaim.

tests/run.sh: 362 seams pass.

serverAppConfigs-docs/LLM/frank-primary-glm53-flash-w4a16.md

@@ -193,13 +193,49 @@ What the boots established, in order (each one ~7 min):
    `serve_glm_autoround.sh`'s COMPILE_PATCH branch rather than in the outer
    environment).
 
-That last row is the state of the work: the changelist is one unresolved
-submodule-name lookup away from booting, and it must be cleared (plus whatever
-follows it) before the compile payoff can be measured. Two verified facts make the
-rest of the shape concrete: `suppress_errors` does **not** cover graph breaks (only
-`fullgraph=False` does), and the GLM trunk can never satisfy `fullgraph` because
-its indexer metadata syncs to host
-(`glm5next/amd/sparse_indexer.py:263,419`: `int(...max().item())`).
+4. **04 removes the fused-norm call from the traced region** (`R4D_GLM_FUSE_QKV_RMSNORM=0`:
+   the GLM MLA block stops calling the fork's fused-norm JIT kernel, whose kernel
+   object in `models/common/ops/fused_qk_rmsnorm.py` is bound *after* the function
+   that uses it — `_FUSED_Q_KV_RMSNORM_KERNEL` at line 223 vs its use at line 211).
+   **Its necessity is not yet established** — see the confound below.
+5. **A confound worth knowing about, because it cost two boots.** Two break-tolerance
+   runs of the same config disagree, and the difference is one logging flag:
+   `01+02+03` with `TORCH_LOGS=graph_breaks` dies on a fatal
+   `Unsupported: Read uninitialized cell` at `mla.py:208` (the fused-norm call), while
+   the same config **without** the flag gets past break handling entirely and dies
+   later on the dynamo name-map `KeyError`. The likely mechanism: `graph_breaks`
+   logging makes dynamo build an *explanation* for each break (walk frames, name the
+   reason), and that analysis is what trips on the uninitialized cell — without the
+   flag, the break is taken silently. So that fatal-looking cell error is probably an
+   artifact of the diagnostic. Related: with `graph_breaks` on, the next reported
+   failure is `Unsupported: Unsupported method call` at
+   `sparse_indexer.py:132 sparse_attn_indexer_kpool → workspace.py:135
+   get_simultaneous / :155 _ensure_workspace_size`, i.e. the indexer allocates its
+   shared workspace *inside* the forward (its own comment: *"will allocate on first
+   use"*) — also flagged under explanation, also not yet shown to block a plain
+   break-tolerant compile.
+
+**How to see the breaks (and its trap).** `TORCH_LOGS=graph_breaks` names each break
+with a user stack, and `VLLM_USE_AOT_COMPILE=0` is required because AOT compile
+hard-refuses breaks — but per the confound above, treat the flagged fatal errors as
+*candidates*, and confirm each one against a flag-free run before believing it blocks
+the boot. For the name-map KeyError there is a ready-made probe: put
+`/home/dank/dynamo-probe` on `PYTHONPATH` with `VLLM_DYNAMO_DUMP_ATTRS=1` (a
+`sitecustomize.py` — a `.pth` on `PYTHONPATH` is NOT executed, and the image's own
+`sitecustomize` is chained so its apport hook survives) and it prints, at the miss,
+the node target, the near-miss keys, and whether another live `OutputGraph` holds
+the key.
+
+**State of the changelist:** the trunk traces (`Dynamo bytecode transform time:
+1.08 s`), and the one failure confirmed by a flag-free run is the dynamo name-map
+`KeyError: 'in_proj_qkvbfg_a'`. Two further candidates are queued behind it
+(the fused-norm launcher cell, the indexer's in-forward workspace allocation). Two
+verified facts bound the rest of the shape: `suppress_errors` does **not** cover
+graph breaks (only `fullgraph=False` does), and the GLM trunk can never satisfy
+`fullgraph` because its indexer metadata syncs to host
+(`glm5next/amd/sparse_indexer.py:263,419`: `int(...max().item())`) — both are dead
+fallbacks on the decode path that the fork author already replaced, so they are not
+the blockers they appear to be at first glance.
 
 ### Two structural constraints on any further fork work
 

serverAppConfigs-docs/log.md

@@ -74,13 +74,33 @@
     `RuntimeError: Model does not support EAGLE3 interface` — nothing about the
     drafter, the method or the payload is wrong; the trunk simply does not
     implement the aux-hidden-state contract.
-  - **Compile blocker narrowed to dynamo's name map.** The `KeyError:
-    'in_proj_qkvbfg_a'` (see above) is thrown from `torch/_dynamo/output_graph.py`'s
-    `nn_modules` name lookup (:3000, registered at :1259/:1384, with `_modules[..]`
-    prefixes stripped at :1225) — i.e. the compiler cannot resolve a submodule by
-    name while tracing the GLM decoder layer. Next step is `TORCHDYNAMO_VERBOSE=1`
-    (torch names it in the error), noting the scratch launcher resets `ENVARGS` so
-    it must be set inside the script's COMPILE_PATCH branch.
+  - **Compile work: one confirmed blocker, plus a measurement confound worth
+    recording.** Flag-free break-tolerance runs (01+02+03, AOT off, breaks allowed)
+    get past break handling and die on the dynamo name-map `KeyError:
+    'in_proj_qkvbfg_a'`. Adding `TORCH_LOGS=graph_breaks` to the same config changes
+    where it dies: a fatal-looking `Unsupported: Read uninitialized cell` at
+    `mla.py:208` calling the fork's fused-norm JIT kernel
+    (`models/common/ops/fused_qk_rmsnorm.py`: `_FUSED_Q_KV_RMSNORM_KERNEL` is bound at
+    line 223, *after* its use at line 211), and with that removed, `Unsupported:
+    Unsupported method call` in the indexer's in-forward workspace allocation
+    (`sparse_indexer.py:132 -> workspace.py:135 get_simultaneous / :155
+    _ensure_workspace_size`). **The likely mechanism is that `graph_breaks` logging
+    makes dynamo build a break *explanation*, and that analysis is what trips** — so
+    those two are candidates, not proven blockers, and I overclaimed them once before
+    catching it. `scripts/patches/glm53-compile-04-fused-qkv-rmsnorm-optout.patch` takes
+    GLM off the fused-norm kernel anyway (`R4D_GLM_FUSE_QKV_RMSNORM=0`, a constructor
+    flag glm5next sets at `attention.py:583`); whether it is *needed* is what the
+    flag-free 01+02+03+04 run settles.
+  - **Diagnostics built for this (reusable).** `TORCH_LOGS=graph_breaks` +
+    `VLLM_USE_AOT_COMPILE=0` names each break with its user stack (AOT compile
+    hard-refuses breaks outright); `/home/dank/dynamo-probe` on `PYTHONPATH` with
+    `VLLM_DYNAMO_DUMP_ATTRS=1` prints the dynamo name-map miss detail (node target,
+    near-miss keys, whether another live `OutputGraph` holds the key). Two gotchas
+    recorded: a `.pth` on `PYTHONPATH` is NOT executed (use `sitecustomize.py`,
+    chaining the image's apport hook), and the scratch launcher used to reset
+    `ENVARGS` (now `${ENVARGS:-}`), which silently dropped env vars and wasted a boot.
+    All four patch artifacts are now genuine `diff -u` patches, verified to apply to
+    the pinned image and to reproduce their recorded sha256.
   - Measured and rejected: **MTP4** (the head loads, arch resolves to
     `Glm5NextMTPModel`, acceptance is excellent at mean length 3.76-4.90 — but
     decode was unchanged (9.6-11.2 vs 10.0) while prefill COLLAPSED 918 -> 255.