The frank-primary section still carried the retracted framing ('a latent bug the compile
exposes' in the mHC/KDA path) and listed only three compile patches. It now records:
six apply-verified artifacts across two WIP changelists (compile 01-05 + the DFlash2
enabling patch), the measured facts (lru_cache does not protect a probe; suppress_errors
does not cover breaks; the trunk can never satisfy fullgraph because its indexer
metadata syncs to host), the single remaining blocker
(KeyError: 'in_proj_qkvbfg_a', raised in a callee dynamo inlines into
Glm5NextDecoderLayer.forward, not at any of the four nn_modules lookups and not a literal
string lookup -> currently a dynamo-side name-resolution failure, not a demonstrated fork
bug), and the two operational gotchas this session paid for (the launcher silently
dropping VLLM_COMPILE_FULLGRAPH=0, and the PYTHONPATH sitecustomize fork bomb). The
DFlash2 paragraph now says the aux-hidden-state contract is written, and that the
engine's refusal was literally 'Model does not support EAGLE3 interface'.
tests/run.sh: 362 seams pass.
Clarifies scope in the artifact header: torch.compiler.disable is itself a graph break,
so 05 helps only a hypothetical fullgraph=True route (where the workspace acquisition
would be better restructured out of the forward than made opaque). It is kept because
the discovery is real and the wrapper is a no-op when nothing is compiling, but it is
not required for the break-tolerant route whose only confirmed blocker is the dynamo
name-map KeyError.
Also verified in this pass: glm53-dflash-01-aux-hidden-states.patch applies cleanly to
the pinned image and reproduces its recorded patched sha256 (4ff6ff8d...), so all six
artifacts in scripts/patches are now apply-verified.
Recording the incident: a diagnostic sitecustomize.py mounted at /probe on PYTHONPATH
produced a self-replicating chain of python3 processes 19+ levels deep inside the probe
container (sitecustomize runs in EVERY interpreter; the hook imported torch/vLLM at
startup; something in that import chain spawned a helper python which inherited
PYTHONPATH and the hook, and spawned again). Found alive 18 minutes later, 33 probe-tagged
processes. Killed with 'podman rm -f glm-probe' (the container cgroup owns every
generation) and verified by counting /proc/<pid>/environ for the marker variable --
never by pattern-killing an interpreter name, since this host runs python in production
containers (romm, jellyfin, plex, glances). Fixed at the source with an env marker that
children inherit, verified with a two-generation test: gen 1 installs, gen 2 stays inert.
The runbook now carries the hazard, the guard, the kill recipe, the conjunction the
restore must verify, and the no-overlapping-jobs rule. Patch 06
(glm53-dflash-01-aux-hidden-states.patch) is documented in the DFlash2 section, and the
/probe diagnostic is marked retired -- TORCH_LOGS answers dynamo questions with no
startup hook.
tests/run.sh: 362 seams pass.
glm53-dflash-01-aux-hidden-states.patch: implements the interface whose absence makes
the engine refuse the drafter with 'Model does not support EAGLE3 interface'. Adds
EagleModelMixin to Glm5NextModel, captures the single-stream hidden state entering each
requested layer in the loop, returns (hidden_states, aux_hidden_states) as a tuple when
the list is non-empty, and exposes set_aux_hidden_state_layers /
get_eagle3_default_aux_hidden_state_layers on Glm5NextForCausalLM -- the contract read
out of interfaces.py:1694/1712, eagle3_utils.py:26-31, deepseek_v2.py:1551-1585 and
cudagraph_utils.py:621, with kimi_k25/pixtral as the delegation reference.
Independent of the compile changelist (01-05); it patches the same file as 01, so 01
goes first if both are applied. Scope is honest in the header: this makes the interface
present (the prerequisite for driving DFlash2), it does not promise draft quality and is
not a decode-speed fix.
Five runs of the changelist now reproduce the same single failure with breaks genuinely
tolerated (patch 01+02+03+04, VLLM_COMPILE_FULLGRAPH=0, VLLM_USE_AOT_COMPILE=0,
R4D_GLM_FUSE_QKV_RMSNORM=0): the trunk TRACES ('Dynamo bytecode transform time: 1.08 s')
and the boot dies on KeyError: 'in_proj_qkvbfg_a'.
What it is NOT, each ruled out by measurement rather than argument:
- not one of dynamo's four nn_modules lookups: /home/dank/dynamo-probe wraps three of
them and stayed silent over two runs, with the probe's install line confirmed in both
the API-server and worker processes;
- not a dynamo-internal failure: TORCHDYNAMO_VERBOSE=1 adds no internal stack trace;
- not a literal string lookup: the name appears tree-wide only as the KDA layer's
attribute (kda.py:217, used at :407) and in the load_weights shard table
(model.py:780-785); the four sites in the
quantization utils do compute exactly this bare key but every one is guarded
(), so none can raise it;
- not the pluggable-layer registry, which is keyed by class name.
What it IS: a plain Python KeyError raised inside the traced
Glm5NextDecoderLayer.forward, in a callee dynamo INLINED (frame chain
profile_run -> _dummy_run -> execute_model -> glm4_1v.py:2358 -> the compile wrapper ->
eval_frame.compile_wrapper -> Glm5NextModel.forward:748 -> module._call_impl ->
Glm5NextDecoderLayer.forward:456; the reported line is that method's def line, which is
how inlined frames surface).
Patch 03's header and the runbook now record the remaining instrument (TORCH_LOGS=+dynamo
and grep, or wrap the inlining candidates — the mHC CustomOps and the indexer op — with a
KeyError probe), and the two breaks that are real but non-fatal in this configuration
(patch 04's fused-norm launcher cell, patch 05's workspace acquisition).
tests/run.sh: 362 seams pass.
This engine ignores SIGTERM: after systemctl stop its container sits in podman ps
state 'Stopping' for 20+ minutes WHILE STILL ANSWERING /health with 200. So a liveness
probe is a lying oracle for a window restore — twice a restore reported PROD HEALTHY
while systemctl is-active said failed and the runtime mask was still in place, because
the old container was still serving.
Records the conjunction the restore must verify (is-active == active AND is-enabled ==
enabled AND no /run/systemd/system/<unit> mask symlink AND the container is owned by
the unit and not Stopping) and the four-step repair recipe (mv the runtime mask away —
it survives stop and blocks start; daemon-reload; reset-failed — a failed unit refuses
to start; podman rm -f — podman stop hangs on the same SIGTERM).
Also records the overlap hazard: a later job's 'podman rm -f glm-probe' cleanup
deleted an in-flight job's container, so one compile round produced no evidence at all
('no container with name glm-probe found'). Check proc:// for running jobs before
opening a window.
tests/run.sh: 362 seams pass.
Root-caused the moving failures: they were my own scratch launcher. Its compile
branches were restructured and the '-e VLLM_COMPILE_FULLGRAPH=0' line was dropped, so
four consecutive boots mounted the wrapper patch while _FULLGRAPH still defaulted to
True — and with fullgraph=True EVERY break is an error, so each boot reported whichever
break came first (cell -> workspace -> compiler-disable). I had blamed
TORCH_LOGS=graph_breaks for the difference; that was wrong, the flag only logs. The
launcher sets the env var again, and patch 04's header, the runbook and the work log
now carry the true run history as a table instead of the bad theory.
New artifact: scripts/patches/glm53-compile-05-workspace-acquisition-opaque.patch —
the indexer acquires its shared workspace INSIDE the forward (its profile-run branch
literally says 'Reserve workspace for indexer during profiling run'), and
v1/worker/workspace.py:_ensure_workspace_size reads ContextVar/ubatch state, which
dynamo cannot trace. Wrap both call sites in a torch.compiler.disable'd helper
(narrower than decorating the workspace manager, which every model would go through;
a no-op when not compiling).
All five artifacts verified against the pinned image: applied in sequence with patch
-p1 they reproduce their recorded patched sha256 exactly (6e9cb81b…, 0751852e…,
732363bb…, 84969993…, c154115a…).
tests/run.sh: 362 seams pass.
Working the dyno name-map blocker one step at a time, with the tools to see it:
- All four patch artifacts are now genuine diff -u patches (they were prose + pseudo
hunks before, useless to patch(1)). Verified against the pinned image: each applies
clean and reproduces its recorded patched sha256 (6e9cb81b… / 0751852e… / 732363bb…
/ 84969993…).
- New: scripts/patches/glm53-compile-04-fused-qkv-rmsnorm-optout.patch — takes GLM's
MLA block off the fused-norm JIT kernel (R4D_GLM_FUSE_QKV_RMSNORM=0; the flag is
already a constructor arg glm5next sets at attention.py:583). Reason: in
models/common/ops/fused_qk_rmsnorm.py the kernel object is bound at line 223, AFTER
its use at line 211 — the launcher family (@kernel_launcher, 39 call sites) is what
breaks dynamo's break analysis.
- New diagnostics, both reusable and recorded: TORCH_LOGS=graph_breaks +
VLLM_USE_AOT_COMPILE=0 to name each break with its user stack, and
/home/dank/dynamo-probe (sitecustomize.py on PYTHONPATH, VLLM_DYNAMO_DUMP_ATTRS=1)
to print the dynamo name-map miss detail. Gotchas: a .pth on PYTHONPATH is NOT
executed, and the scratch launcher reset ENVARGS (now ${ENVARGS:-}), which silently
dropped env vars and wasted a boot.
Measurement discipline correction: I first recorded the 'Read uninitialized cell' and
'Unsupported method call' failures as fatal blockers. They are not established as
such — they appear only when TORCH_LOGS=graph_breaks is set, while a flag-free
break-tolerance run gets past break handling and dies on the name-map KeyError. The
likely mechanism is that graph_breaks logging makes dynamo build a break
*explanation*, and that analysis itself trips on the uninitialized cell. Patch 04's
header, the runbook and the work log now carry the confound explicitly rather than
the overclaim.
tests/run.sh: 362 seams pass.
DFlash2: fetched incoai/GLM-5.3-Flash-DFlash2 (2,342,169,800 B, sha256 matches the
Hub LFS oid) and tried --speculative-config {"method":"dflash","model":"/draft"}.
Config and drafter are accepted; the worker dies with 'Model does not support
EAGLE3 interface' - so the only gap is the target-side aux-hidden-state contract
(set_aux_hidden_state_layers + per-layer capture + the two-value forward return,
as deepseek_v2.py demonstrates twice in this image). Recorded the exact interface,
the GLM-specific unknown (what the single-stream state entering layer k means for
an mHC multi-stream trunk) and the honest expectation: no speedup on a
token-proportional pointwise bound.
fp8 KV: recorded the deciding measurement - the fork's KV reserve model is
dtype-blind (the fp8 boot printed the identical 1760 MiB / 13600 B-per-token plan),
so fp8 cannot buy capacity even if the gfx12 MLA path were rewritten. Verdict: do
not spend effort there.
Compile blocker narrowed from 'a latent GLM bug' to dynamo's name map: the
KeyError is thrown by torch/_dynamo/output_graph.py's nn_modules lookup (:3000,
registered :1259/:1384, _modules[..] prefixes stripped :1225), i.e. the compiler
cannot resolve a submodule by name while tracing the decoder layer. Next step is
TORCHDYNAMO_VERBOSE=1, with the note that the scratch launcher resets ENVARGS.
tests/run.sh: 362 seams pass.
A torch-profiler kernel-time window (496 decode tokens, rank0 trace parsed for
device-activity time) on the deployed GLM-5.3-Flash shape falsifies this page's
own "the expert gather is the bound" explanation:
46.18 s of kernel time / 496 tokens = ~93 ms per token (actual step ~104 ms)
34.8 % vectorized_elementwise_kernel 708,500 launches (1,428/token)
10.0 % r4d_gemm_moe_w4a16_nt_b16_m64 (the MoE GEMM)
6.7 % elementwise_kernel_manual_unroll
6.4 % wvSplitK_hf_sml_ (split-K GEMM)
8.9 % router: sbtopk + warpMergeSortTopK + bitonicSortKVInPlace
6.5 % r4d_mhc_* (hyper-connections)
1.05 % r4d_moe_lru_gather_kernel <-- the retracted suspect
The earlier arithmetic was circular (it assumed the gather consumed the whole
step). The real bound is unfused pointwise glue: the fork compiles its own Qwen
trunk (support_torch_compile in models/qwen4_exp/amd/model.py) and not its GLM
trunk, so the engine boots with "provides neither a compiled submodule nor
breakable CUDA graphs. Overriding to FULL_DECODE_ONLY." That also explains why
MTP4 measured flat — pointwise cost scales with the token count, so spec decode
amortizes nothing here — and it retires arena/hit-rate tuning (the gather is 1 %).
Staged as documented patch artifacts (not wired into any serve script; the
engine does not yet boot with them) — scripts/patches/glm53-compile-0{1,2,3}*.patch:
-01 models/glm5next/common/model.py decorate Glm5NextModel (R4D_GLM_COMPILE)
-02 model_executor/kernels/r4d_lib.py freeze 10 capability probes that do
os.path.exists/.so lookups inside the traced forward. functools.lru_cache
does NOT protect them: dynamo re-traces the wrapped body even when warm
(verified in-image); a plain module-global read compiles.
-03 compilation/wrapper.py make fullgraph=True opt-out, because
suppress_errors does NOT cover graph breaks and this trunk can never
satisfy fullgraph (indexer metadata syncs to host — .item() at
glm5next/amd/sparse_indexer.py:263,419).
With all three the trunk traces (Dynamo bytecode transform time 1.08 s) and dies
on KeyError: in_proj_qkvbfg_a in the mHC/KDA path — a latent bug the compile
exposes, and the remaining blocker.
Also records two structural constraints: no new HIP kernels are possible (r4d.so
ships without sources), which binary-blocks the fused router at GLM's 288
experts; and DFlash2 for GLM is a TARGET-side port, not a drafter port
(incoai/GLM-5.3-Flash-DFlash2 is architectures ["DFlash2DraftModel"] +
model_type qwen3, already mapped at registry.py:636; glm5next lacks only the
aux-hidden-state plumbing deepseek_v2.py demonstrates).
Runbook frontmatter, intro, mermaid and wall section, AGENTS.md and the work log
corrected to match the measurement. tests/run.sh: 362 seams pass.
Comrade Chief Designer