The 2026-10-09 wire aliases become the service names everywhere:
systemd units, serve scripts, env files and FRANK_* variables,
deploy-plan/deploy-remote service keys, container names, Prometheus
job names + alert rules, Grafana dashboard queries, bats seams,
runbook filenames, homepage/AGENTS lore, and historical prose.
Rollback tags renamed in lockstep. Residue in the tree: zero.
Retired frank-medium/small/npu names are history and stay.
Known casualty: scripts/patches/*.patch context lines were rewritten;
those retired-era vLLM-fork patches no longer apply byte-exact.
DFlash2 experiment declared failed 2026-10-01: the drafter serves and
drafts (12.8 tok/s proposed) but acceptance is pinned at 1.0-1.38 on ALL
text classes including verbatim repetition — the aux-hidden-state capture
semantics are wrong (single-stream entering-state vs post-layer/comb is
the untested alternative). At ~1.2 acceptance the 8-token verify tax makes
decode 5x SLOWER than the 9.9-10.0 baseline (1.6-1.8 tok/s measured).
Restores the verified 37-40 tok/s shape: qwen3.8-flash-next FP8, MTP4,
prefetch off, 200k x 3 seqs, fp8 KV.
use_aux_hidden_state_outputs lives on the gpu_model_runner instance
(gpu_model_runner.py:623/678), not on the spec config, so the first
registry check never fired and the reserve stayed at 13,600 B/token.
Switch to method in {dflash, dspark, extract_hidden_states}.
The dflash boot cleared the unifier and the kpool tail (02/03) and died on
KV capacity: on the generic path the mamba states are NOT aliased into the
MLA pages (the glm5 tensor layout's whole point), so the capacity estimator
demanded ~30.6 GiB fixed against 0.92 GiB available (the plan's KV reserve
registry only knows the target's 13,600 B/token).
Patch 04 keeps the aliased glm5 pool and appends the drafter's uniform
group and per-layer hidden-state groups beside it: grouping partitions
hidden + draft specs, the layout returns a 10-tuple, the builder sizes one
shared backing allocation (pool + extras), and the memory estimator adds
the extras. Patch 04b extends the GLM registry kv_bytes_per_token by the
drafter (4,096 B/token) and hidden-state (5 x hidden x 2 B/token) costs so
the expert plan reserves enough.
Local smoke tests (CPU torch + real spec classes) validate grouping shape,
layout recognition, builder block math and memory accounting against
synthetic GLM+draft+hidden spec sets. Apply-verified: reproduces shas
5521e50b... / 1a623d5c...
Patch 02 (MLA-multiple max page) cleared the KV page-size unifier: the
dflash boot now reaches cudagraph capture. New blocker: the kpool decode
kernel asserts tail_kv_cache.shape[2] == pool_size; KpoolTailSpec pins
block_size == index_kpool, so the unifier's divisible branch must never
grow it. Patch 03 = 02 + a KpoolTailSpec branch that pads page_size_padded
instead. Apply-verified: reproduces patched sha 5093e4b7... Runbook
updated with the full KV grouping journey.
The v4 boot named the actual miss: model_runner.py:425 calls
set_eagle3_aux_hidden_state_layers(self.model) on the instance model_runner
LOADED -- and for GLM-5.3-Flash that is Glm5NextForConditionalGeneration (the
multimodal wrapper; APIServer log: 'Resolved architecture:
Glm5NextForConditionalGeneration'), which wraps the text model as
self.language_model = Glm5NextForCausalLM (glm5next/common/model.py:1113).
v4 put the three presence-checked attributes only on Glm5NextForCausalLM,
so the isinstance check on the conditional still failed with the same
RuntimeError.
v5 adds to BOTH wrappers, MRO-safely (attribute route, no Protocol base):
- Glm5NextForConditionalGeneration: supports_eagle3 / has_own_lm_head /
has_own_embed_tokens + the two methods delegating to self.language_model
(the pixtral.py:498 pattern);
- Glm5NextForCausalLM: the same three attributes + the two methods that
unwrap to self.model (the EagleModelMixin trunk -- v1's correct bodies).
Also proven by MRO simulation: the ForCausalLM's five-base ordering cannot
take SupportsEagle3 as a base in ANY position, so inheritance was never a
viable route for this model.
base 008eeaa2... -> patched bc9913a1..., apply-verified against the pinned
image (patch -p1 reproduces the sha256 and matches the mounted file).
The v4 window boot named the real presence-check rule: isinstance against a
runtime_checkable Protocol walks __protocol_attrs__ (which includes members
inherited from BASE protocols) and getattr_static()s each one off the instance,
breaking on the first missing attribute. SupportsEagleBase declares
has_own_lm_head/has_own_embed_tokens as plain instance-var annotations, so
v3's bare supports_eagle3 flag still failed. Replicated in-image with stock
typing: flag-only class -> False; flag + the two base members -> True.
v4 declares all three on Glm5NextForCausalLM:
supports_eagle3: ClassVar[Literal[True]] = True
has_own_lm_head: bool = False
has_own_embed_tokens: bool = False
(False is correct for the two: the wrapper delegates embed/lm_head to its
trunk. The MRO-safe attribute path is kept from v3.)
base 008eeaa2... -> patched 60f27aad..., apply-verified against the pinned
image (patch -p1 reproduces the sha256 and matches the mounted file).
v2 (inheriting SupportsEagle3 as a base) taught the real lesson: the isinstance
check PASSES and the class then dies at creation with
'Cannot create a consistent method resolution order (MRO) for bases ...'
-- GLM's outer wrapper already inherits vLLM Protocols (SupportsPP,
MixtureOfExperts, HasInnerState, IsHybrid) whose ordering is incompatible with
appending another Protocol base.
v3 avoids the MRO entirely: declare
supports_eagle3: ClassVar[Literal[True]] = True
as a class attribute on Glm5NextForCausalLM. Runtime-checkable Protocols do
presence-based isinstance, so the bare flag satisfies supports_eagle3(model)
without touching the MRO. The aux-layer plumbing stays on the trunk via
EagleModelMixin.
base 008eeaa2... -> patched 4c9a632b..., apply-verified against the pinned
image (patch -p1 reproduces the sha256 and matches the mounted dflash-patch
file byte for byte).
v1 hand-wrote set_aux_hidden_state_layers/get_eagle3_default_aux_hidden_state_layers
and the boot reproduced the identical 'Model does not support EAGLE3 interface'.
Reason: supports_eagle3() is an isinstance check against a runtime-checkable
Protocol whose members include a required ClassVar flag
(interfaces.py:1663 supports_eagle3: ClassVar[Literal[True]] = True), so methods
alone do not satisfy it. Inheriting SupportsEagle3 -- the idiom every model in
this image uses, including the fork's own DFlashLagunaForCausalLM -- supplies the
flag, SupportsEagleBase's has_own_lm_head/has_own_embed_tokens, and working
default implementations of both methods; the hand-written methods are deleted.
base 008eeaa2... -> patched 5eea89c5..., apply-verified against the pinned image.
Runbook: corrected method name, recorded the Protocol trap as the generalizable
lesson for future spec-decode ports.
Clarifies scope in the artifact header: torch.compiler.disable is itself a graph break,
so 05 helps only a hypothetical fullgraph=True route (where the workspace acquisition
would be better restructured out of the forward than made opaque). It is kept because
the discovery is real and the wrapper is a no-op when nothing is compiling, but it is
not required for the break-tolerant route whose only confirmed blocker is the dynamo
name-map KeyError.
Also verified in this pass: glm53-dflash-01-aux-hidden-states.patch applies cleanly to
the pinned image and reproduces its recorded patched sha256 (4ff6ff8d...), so all six
artifacts in scripts/patches are now apply-verified.
glm53-dflash-01-aux-hidden-states.patch: implements the interface whose absence makes
the engine refuse the drafter with 'Model does not support EAGLE3 interface'. Adds
EagleModelMixin to Glm5NextModel, captures the single-stream hidden state entering each
requested layer in the loop, returns (hidden_states, aux_hidden_states) as a tuple when
the list is non-empty, and exposes set_aux_hidden_state_layers /
get_eagle3_default_aux_hidden_state_layers on Glm5NextForCausalLM -- the contract read
out of interfaces.py:1694/1712, eagle3_utils.py:26-31, deepseek_v2.py:1551-1585 and
cudagraph_utils.py:621, with kimi_k25/pixtral as the delegation reference.
Independent of the compile changelist (01-05); it patches the same file as 01, so 01
goes first if both are applied. Scope is honest in the header: this makes the interface
present (the prerequisite for driving DFlash2), it does not promise draft quality and is
not a decode-speed fix.
Five runs of the changelist now reproduce the same single failure with breaks genuinely
tolerated (patch 01+02+03+04, VLLM_COMPILE_FULLGRAPH=0, VLLM_USE_AOT_COMPILE=0,
R4D_GLM_FUSE_QKV_RMSNORM=0): the trunk TRACES ('Dynamo bytecode transform time: 1.08 s')
and the boot dies on KeyError: 'in_proj_qkvbfg_a'.
What it is NOT, each ruled out by measurement rather than argument:
- not one of dynamo's four nn_modules lookups: /home/dank/dynamo-probe wraps three of
them and stayed silent over two runs, with the probe's install line confirmed in both
the API-server and worker processes;
- not a dynamo-internal failure: TORCHDYNAMO_VERBOSE=1 adds no internal stack trace;
- not a literal string lookup: the name appears tree-wide only as the KDA layer's
attribute (kda.py:217, used at :407) and in the load_weights shard table
(model.py:780-785); the four sites in the
quantization utils do compute exactly this bare key but every one is guarded
(), so none can raise it;
- not the pluggable-layer registry, which is keyed by class name.
What it IS: a plain Python KeyError raised inside the traced
Glm5NextDecoderLayer.forward, in a callee dynamo INLINED (frame chain
profile_run -> _dummy_run -> execute_model -> glm4_1v.py:2358 -> the compile wrapper ->
eval_frame.compile_wrapper -> Glm5NextModel.forward:748 -> module._call_impl ->
Glm5NextDecoderLayer.forward:456; the reported line is that method's def line, which is
how inlined frames surface).
Patch 03's header and the runbook now record the remaining instrument (TORCH_LOGS=+dynamo
and grep, or wrap the inlining candidates — the mHC CustomOps and the indexer op — with a
KeyError probe), and the two breaks that are real but non-fatal in this configuration
(patch 04's fused-norm launcher cell, patch 05's workspace acquisition).
tests/run.sh: 362 seams pass.
Root-caused the moving failures: they were my own scratch launcher. Its compile
branches were restructured and the '-e VLLM_COMPILE_FULLGRAPH=0' line was dropped, so
four consecutive boots mounted the wrapper patch while _FULLGRAPH still defaulted to
True — and with fullgraph=True EVERY break is an error, so each boot reported whichever
break came first (cell -> workspace -> compiler-disable). I had blamed
TORCH_LOGS=graph_breaks for the difference; that was wrong, the flag only logs. The
launcher sets the env var again, and patch 04's header, the runbook and the work log
now carry the true run history as a table instead of the bad theory.
New artifact: scripts/patches/glm53-compile-05-workspace-acquisition-opaque.patch —
the indexer acquires its shared workspace INSIDE the forward (its profile-run branch
literally says 'Reserve workspace for indexer during profiling run'), and
v1/worker/workspace.py:_ensure_workspace_size reads ContextVar/ubatch state, which
dynamo cannot trace. Wrap both call sites in a torch.compiler.disable'd helper
(narrower than decorating the workspace manager, which every model would go through;
a no-op when not compiling).
All five artifacts verified against the pinned image: applied in sequence with patch
-p1 they reproduce their recorded patched sha256 exactly (6e9cb81b…, 0751852e…,
732363bb…, 84969993…, c154115a…).
tests/run.sh: 362 seams pass.
Working the dyno name-map blocker one step at a time, with the tools to see it:
- All four patch artifacts are now genuine diff -u patches (they were prose + pseudo
hunks before, useless to patch(1)). Verified against the pinned image: each applies
clean and reproduces its recorded patched sha256 (6e9cb81b… / 0751852e… / 732363bb…
/ 84969993…).
- New: scripts/patches/glm53-compile-04-fused-qkv-rmsnorm-optout.patch — takes GLM's
MLA block off the fused-norm JIT kernel (R4D_GLM_FUSE_QKV_RMSNORM=0; the flag is
already a constructor arg glm5next sets at attention.py:583). Reason: in
models/common/ops/fused_qk_rmsnorm.py the kernel object is bound at line 223, AFTER
its use at line 211 — the launcher family (@kernel_launcher, 39 call sites) is what
breaks dynamo's break analysis.
- New diagnostics, both reusable and recorded: TORCH_LOGS=graph_breaks +
VLLM_USE_AOT_COMPILE=0 to name each break with its user stack, and
/home/dank/dynamo-probe (sitecustomize.py on PYTHONPATH, VLLM_DYNAMO_DUMP_ATTRS=1)
to print the dynamo name-map miss detail. Gotchas: a .pth on PYTHONPATH is NOT
executed, and the scratch launcher reset ENVARGS (now ${ENVARGS:-}), which silently
dropped env vars and wasted a boot.
Measurement discipline correction: I first recorded the 'Read uninitialized cell' and
'Unsupported method call' failures as fatal blockers. They are not established as
such — they appear only when TORCH_LOGS=graph_breaks is set, while a flag-free
break-tolerance run gets past break handling and dies on the name-map KeyError. The
likely mechanism is that graph_breaks logging makes dynamo build a break
*explanation*, and that analysis itself trips on the uninitialized cell. Patch 04's
header, the runbook and the work log now carry the confound explicitly rather than
the overclaim.
tests/run.sh: 362 seams pass.
DFlash2: fetched incoai/GLM-5.3-Flash-DFlash2 (2,342,169,800 B, sha256 matches the
Hub LFS oid) and tried --speculative-config {"method":"dflash","model":"/draft"}.
Config and drafter are accepted; the worker dies with 'Model does not support
EAGLE3 interface' - so the only gap is the target-side aux-hidden-state contract
(set_aux_hidden_state_layers + per-layer capture + the two-value forward return,
as deepseek_v2.py demonstrates twice in this image). Recorded the exact interface,
the GLM-specific unknown (what the single-stream state entering layer k means for
an mHC multi-stream trunk) and the honest expectation: no speedup on a
token-proportional pointwise bound.
fp8 KV: recorded the deciding measurement - the fork's KV reserve model is
dtype-blind (the fp8 boot printed the identical 1760 MiB / 13600 B-per-token plan),
so fp8 cannot buy capacity even if the gfx12 MLA path were rewritten. Verdict: do
not spend effort there.
Compile blocker narrowed from 'a latent GLM bug' to dynamo's name map: the
KeyError is thrown by torch/_dynamo/output_graph.py's nn_modules lookup (:3000,
registered :1259/:1384, _modules[..] prefixes stripped :1225), i.e. the compiler
cannot resolve a submodule by name while tracing the decoder layer. Next step is
TORCHDYNAMO_VERBOSE=1, with the note that the scratch launcher resets ENVARGS.
tests/run.sh: 362 seams pass.
A torch-profiler kernel-time window (496 decode tokens, rank0 trace parsed for
device-activity time) on the deployed GLM-5.3-Flash shape falsifies this page's
own "the expert gather is the bound" explanation:
46.18 s of kernel time / 496 tokens = ~93 ms per token (actual step ~104 ms)
34.8 % vectorized_elementwise_kernel 708,500 launches (1,428/token)
10.0 % r4d_gemm_moe_w4a16_nt_b16_m64 (the MoE GEMM)
6.7 % elementwise_kernel_manual_unroll
6.4 % wvSplitK_hf_sml_ (split-K GEMM)
8.9 % router: sbtopk + warpMergeSortTopK + bitonicSortKVInPlace
6.5 % r4d_mhc_* (hyper-connections)
1.05 % r4d_moe_lru_gather_kernel <-- the retracted suspect
The earlier arithmetic was circular (it assumed the gather consumed the whole
step). The real bound is unfused pointwise glue: the fork compiles its own Qwen
trunk (support_torch_compile in models/qwen4_exp/amd/model.py) and not its GLM
trunk, so the engine boots with "provides neither a compiled submodule nor
breakable CUDA graphs. Overriding to FULL_DECODE_ONLY." That also explains why
MTP4 measured flat — pointwise cost scales with the token count, so spec decode
amortizes nothing here — and it retires arena/hit-rate tuning (the gather is 1 %).
Staged as documented patch artifacts (not wired into any serve script; the
engine does not yet boot with them) — scripts/patches/glm53-compile-0{1,2,3}*.patch:
-01 models/glm5next/common/model.py decorate Glm5NextModel (R4D_GLM_COMPILE)
-02 model_executor/kernels/r4d_lib.py freeze 10 capability probes that do
os.path.exists/.so lookups inside the traced forward. functools.lru_cache
does NOT protect them: dynamo re-traces the wrapped body even when warm
(verified in-image); a plain module-global read compiles.
-03 compilation/wrapper.py make fullgraph=True opt-out, because
suppress_errors does NOT cover graph breaks and this trunk can never
satisfy fullgraph (indexer metadata syncs to host — .item() at
glm5next/amd/sparse_indexer.py:263,419).
With all three the trunk traces (Dynamo bytecode transform time 1.08 s) and dies
on KeyError: in_proj_qkvbfg_a in the mHC/KDA path — a latent bug the compile
exposes, and the remaining blocker.
Also records two structural constraints: no new HIP kernels are possible (r4d.so
ships without sources), which binary-blocks the fused router at GLM's 288
experts; and DFlash2 for GLM is a TARGET-side port, not a drafter port
(incoai/GLM-5.3-Flash-DFlash2 is architectures ["DFlash2DraftModel"] +
model_type qwen3, already mapped at registry.py:636; glm5next lacks only the
aux-hidden-state plumbing deepseek_v2.py demonstrates).
Runbook frontmatter, intro, mermaid and wall section, AGENTS.md and the work log
corrected to match the measurement. tests/run.sh: 362 seams pass.
Payload: Intel/GLM-5.3-Flash-W4A16-AutoRound (AutoRound symmetric int4 group
128, auto_round:auto_gptq, 34 shards + 2 extras, 113,074 tensors, ~170 GiB).
It is the only W4A16 checkpoint this fork can serve with expert offload: its
quant_method "auto-round" is overridden to "inc", which routes the MoE to
libr4d's grouped W4A16 GEMM (R4dW4a16MoEMethod). A compressed-tensors W4A16
build of the same model resolves to TritonWNA16Experts, which has no offload
recipe, and refuses to serve at all (170 GiB of experts cannot fit 64 GiB of
VRAM, so offload is mandatory).
Shape: tcclaviger RDNA4 vLLM fork (same digest-pinned image as the 2026-09-29
qwen cutover), TP=2, 131,072 ctx x 2 seqs, bf16 KV (fp8 is ILLEGAL here — it
disables the sparse MLA backend's rope-free Triton path, the only MLA decode
gfx12 has), util 0.98, NBATCH 2048, --enable-expert-offload
--expert-precise-memory, prefix caching, glm47 tool + reasoning parsers,
MTP and expert prefetch absent by measurement.
Measured: decode 9.9-10.0 tok/s, prefill 918 tok/s (warm), arena 14.51 GiB =
2,383 slots/rank over 12,096 host-backed experts (73.64 GiB pinned/rank).
THE 35 tok/s TARGET IS NOT MET and the shortfall is measured: live telemetry
(R4D_LRU_TELEMETRY=1) gives a 0.53-0.66 per-layer hit rate, a step moves
0.82 GiB of misses in ~100 ms = 8.2 GB/s effective against a link measured at
22.74 GB/s — the same structural gather-path gap the qwen campaign found.
Growing the arena 13.08 -> 14.51 GiB changed decode by nothing.
Rejected by measurement: MTP4 (head loads, acceptance 3.76-4.90, but decode
unchanged and prefill collapses 918 -> 255 — a verification forward gathers the
union of the batch's experts), NBATCH 8192 (identical prefill).
Also: scripts/patches/gpu_worker-kv-cap.patch (fork-local fix for a KV-reserve
model that under-counts GLM's per-sequence KDA state; not needed by this shape),
runbook, AGENTS.md row + section, both indexes, work log.
Rollback: 2339fae (qwen FP8 + MTP4).
fa on aborted graph build: build_attn_qsa called ggml_fill on the
F16-converted kq mask (GGML_ASSERT F32 at ggml.c:5371). Patch mirrors the
MLA sparse path in llm_graph_context::build_attn (F32 base mask for the
top-k surgery, F16 cast only at the attention call, gated on
cparams.flash_attn). Vendored at scripts/patches/qwen4exp-qsa-f32-mask.patch
(sha256 f2f4a486...); the build-qwen38 binary on Dankclaw is rebuilt with
it and validated live at CTX_SIZE=32768 (health 200, real completion).
Frank
Comrade Chief Designer