The first crash-looping boot (51 restarts): serve received the host path
/home/dank/models/... but the models tree is mounted at /models inside the
container, so transformers failed 'Repo id must be in the form
namespace/repo_name' before the engine even built. Same conversion the spec
draft already used. Pinned by bats. Root docs index links the new runbook
(seam required it) and the work log records the crash-loop plus the 52 G
payload reclaim.
The 2026-10-01 first deploy refused /home/dank/models/Qwen3.5-9B because
the handler checked for a single model.safetensors (the dflash layout);
the Qwen3.5-9B MTP head ships 4 shards behind model.safetensors.index.json.
Same dual-layout acceptance as the serve script, now mirrored and gated by
the seam. The gate fired before the stop, so the live gufo unit was never
touched (health 200 throughout).
Engine: digest-pinned docker.io/kyuz0/vllm-therock-gfx1151 (vLLM 0.30,
TheRock ROCm 10, Triton GDN kernels); the serve script gates the image by
REPO DIGEST, the payload (config + index + exact shard count), the spec
draft (BOTH weight layouts: single model.safetensors for dflash, sharded
index for the Qwen3.5-9B MTP head), and whole /dev/dri, sweeps the bench
identity vllm-mimo, then exec podman wait so a container death trips
Restart=always. Shape: 4 x 262144 at util 0.45 (KV grows only on the 8
full-attention layers), MTP 2 draft tokens.
Measured 2026-10-01 window (frank-large stopped): base 13.1 t/s decode
(~94% of the LPDDR5x BF16 roof), DFlash-8 17.4, MTP-2 19.8 with 71%/44%
per-position acceptance - the cross-model head works because vLLM reads the
draft from a separate path while MiMo ships zero mtp.* tensors. Spec arms
do not preserve greedy bf16 text (batched-verify reduction order).
The seam was rewritten around the new laws (18 tests): unit self-healing,
digest gating, engine wiring, the measured spec table, the retired gufo
gates, the deployer MainPID check. frank_small port test follows the
PORT -> FRANK_LARGE_PUBLISH rename. Rollback: tag frank-large-gufo-q4kxl-20261001.
All three verify every file by byte count + Hub LFS sha256, use -f (a bare
curl -o reports success on a 403 error body), support --verify-only, and
never put a token on the command line. Payloads are on Dankclaw and verified
against these manifests.
bf16 at 2 x 262144 fails with and without --expert-precise-memory. The real
error (captured from the journal this time, container fixed via journald
rather than podman logs):
RuntimeError: Worker failed with error 'Constraints violated
(L['query_start_loc'].size()[0])! You marked it as dynamic but your code
specialized it to be a constant (3)'
- a torch.compile/dynamo dynamic-shape violation, NOT an OOM. Ruled out a
stale compile cache: ~/.cache/vllm holds only modelinfos (16K) and each
podman run gets a fresh container filesystem.
The deltas from the last shape measured serving (fp8 3 x 262144 / MNBT 2048 /
792,069 tokens / 3.02x / arena 6,248) are KV dtype (fp8 -> bf16) and SEQS
(3 -> 2). Restoring the known-good shape; bisect bf16 vs SEQS in a window,
with --enforce-eager as the bypass.
NRestarts climbed to 3, each attempt ~6.7 min then 'Deactivated successfully'
(API server drains after an EngineCore fatal error; Restart=always turned the
silent death into a visible loop). The arena is device-resident, so the budget
decides it: weights 6.28 + pool 6.80 (bf16 2 x 262144) + graphs 0.55 + arena
16.65 (precise overhead 5.25) + non-torch 1.45 = 31.73 of 31.86 GiB -> ~0.13
GiB free at boot, and graph capture is what allocates transiently.
Dropping the flag restores the 7.60 GiB default overhead: arena 14.31 GiB,
total ~29.4 GiB, ~2.4 GiB free. bf16 2 x 262144 (Frank's KV change) stays.
The serve script keeps the opt-in flag; the seam now asserts it is ABSENT.
Frank's call. Payload: ajgazin's abliterated Swift-1.5 UD-Q8_K_XL (29.3 GiB,
SWIFT OPEN LICENSE) -> unsloth/Qwen3.8-27B-GGUF UD-Q4_K_XL (16.35 GiB,
sha256 3f227079...b01e, vanilla/Apache-2.0). NOTE: the endpoint is no longer
uncensored — that was one of the reasons for the Swift payload on 2026-09-28.
New fetcher scripts/download-frank-large-q4.sh (no auth header: repo is public
and ungated; size + sha256 verified after transfer). The speculative drafter
and its fetcher are unchanged.
Shape: --sessions 4 --context 163840 with unquantized KV. gufo exposes no KV
quantization lever (execution_policy.hpp pins kv_cache_storage to kFp16), so
f16 is inherent.
KV arithmetic: f16 KV = 64 KiB/token (16 full-attention layers x 4 KV heads x
256 head_dim x 2 x 2B; the other 48 layers are linear-attention with a
per-sequence fp32 state). 4 x 163840 = 655,360 tokens = 40.0 GiB KV, each
session exactly 10 GiB. Budget ~62 GiB of the 114 GiB GTT.
Seam retargeted (FETCH -> the q4 script, TARGET_STUB, context/sessions pins).
The KV pool stays pinned at 606,208 tokens (card is full: 531 MiB free at
the old shape; autofit's 687,388 would leave 0.53 GB and OOM on the first
long request). 2 x 262,144 = 524,288 leaves ~82k tokens of pool slack.
Motivation: harness compaction. frank-primary's contextWindow 262,144
compacts at 70% = 183,501 tokens, and the compaction request carries a
32,768-token summary budget -> 216,269 must fit ONE frank-small request.
The 151,552 slot window could not hold it; the native window does.
No spec-decode interaction: checkpoint ships zero mtp.* tensors (config
lies), seam keeps every --speculative-* flag away from the engine.
Both of Frank's changes at once.
KV is bf16 (--kv-cache-dtype auto) at two full-size windows (2 x 262144).
bf16 measures 13,839 B/token (bf16 3x241664 boot: 11,047,010,304 B -> 798,280
tokens), decomposing as 12,276 attention + 1,563 mamba/side caches vs fp8's
6,138 + 1,563 = 7,701. 524,288 tokens needs 7.256e9 B; pinned at 7.3e9
(~527,500 tokens, 3,206 margin). Pool cost: 5.68 -> 6.76 GiB per rank.
--expert-precise-memory selects the fitted overhead (5.25 GiB) over the default
(7.60 GiB = fitted + compile/graph-capture headroom). It moves only the plan's
term, so the expert arena goes 6,248 -> ~7,278 slots (~1,025 more) at the
cost of boot-time capture slack. Overhead adds 0.22 GiB/1024 nbt above 4096, so
MNBT 2048 pays nothing.
Seam + docs + AGENTS row + log updated.
ExecStop runs 'podman stop -t 120' but the unit never set TimeoutStopSec, so
systemd's 90 s default cut the graceful stop short: journal shows 'Stopping
timed out. Terminating.' then 'Failed with result timeout' on 2026-10-01
13:27:17 — every stop of this unit was logging a failed stop. 180 s covers the
engine's drain with margin, and the seam now guards the RELATIONSHIP
(TimeoutStopSec > the podman grace) rather than a literal.
StartLimitIntervalSec is a [Unit] directive; the copy in [Service] made systemd
log 'Unknown key StartLimitIntervalSec in section Service' on every load. Seam
asserts it appears exactly once and not in [Service].
Frank's call 2026-10-01: MAXLEN 262144 x SEQS 3 = 786,432 tokens of contract.
Measured fp8 KV costs 7,698 B/token (4-slot boot 8,311,013,376 B -> 1,079,582
tokens; 5-slot boot 10,388,766,720 B -> 1,350,182 tokens), so the pool floor
for the contract is 6.054e9 B. Pinned at 6,100,000,000 (~792k tokens, ~5.9k
margin) so the remaining ~2.2 GiB versus the 4-slot shape goes to the expert
ARENA - the fork sizes its decode-critical expert cache from the remainder
after weights, pool and workspace.
MNBT 2048 (Frank): the arena sweep measured 6,499 slots at every rung <= 4096,
so the larger batch costs the arena nothing.
Seam retargeted (3 seqs / 6.1e9 / MNBT 2048); docs + AGENTS row + log updated,
including the closed DCP investigation.
DCP is passed only when FRANK_PRIMARY_DCP is set, so the deployed shape is
unchanged. Verified in this fork: DCP reuses the TP ranks when PCP is
disabled (tp % dcp == 0), so TP=2 + DCP=2 fits the two R9700s with no extra
ranks; the flash_attn backend carries DCP support, the 36 linear-attention
layers of this hybrid do not — which is what the A/B window tests.
Replaces llama.cpp Gemma4-12B-QAT (payload + binary retained as rollback).
SGLang in a self-contained python3.12 venv with an extracted CUDA 13.0
nvcc toolchain for the sgl-kernel/flashinfer JIT.
Shape: fp8 KV, pinned 606,208-token pool / 4 x 151,552 slots, bf16 mamba
states pinned at 32 slots (the engine autofills the ssm pool to a fixed
~5.1 GB regardless of dtype), eager prefill (no headroom for graph
capture), no spec decode (the checkpoint ships no mtp.* tensors despite
mtp_num_hidden_layers=1 — NEXTN drafts with an uninitialised 4.24 GiB
ghost at 0% acceptance).
Verified: 83.9 tok/s decode, 170k single prompt, 4 concurrent 54k
requests, vision (in-checkpoint tower), reasoning_content field.
Five 262144-token contexts passed every boot gate (KV 1,350,182 / 5.15x) but
left 0 bytes free at runtime: a real client request (995-token prompt,
max_tokens=261149, sampling params straight from models.yml) killed Worker_TP1
with torch.OutOfMemoryError 'Tried to allocate 50.00 MiB ... 0 bytes free'.
Boot is not proof of runtime survivability.
Four slots x 262144 fp8 (KV_BYTES 8,311,013,376/rank) returns ~1.93 GiB per
rank for activations/graph capture.
Unit: Restart=always instead of on-failure — a runtime worker fault makes
vLLM's API server exit clean, so ExecStart returned 0 and on-failure never
fired, leaving the endpoint down until a human noticed. Also
StartLimitIntervalSec=0 and OOMScoreAdjust=-500.
Frank's call after the night ladder: five contexts at the model's full native
262,144 window beat six shortened ones. Six full-length contexts do NOT fit at
any KV dtype (6 x 241664 fp8 asked 10.70 GiB against 9.18 GiB free — the
6-seq graph/activation footprint leaves too little for the pool); 5 x 262144
at fp8 PASSED on the first rung: KV_BYTES 10,388,766,720/rank, measured
'GPU KV cache size: 1,350,182 tokens, Maximum concurrency for 262,144 tokens
per request: 5.15x' (39,462 tokens of margin).
Also documented: the SEQS=3 bf16 max (241,664, KV 732,359 / 3.03x) measured
earlier the same night, and the root-cause read that the 7.2 GiB KV ceiling
law was the MTP4 workspace, not VRAM. Seam and harness (contextWindow 262144)
re-pinned.
Night ladder result (Frank's subtract-16k method): 262144 and 245760 both OOM
at CUDA graph capture ('Tried to allocate 76.00 MiB ... 0 bytes free'), 237568
passed (KV 718,606 tokens), and the +4096 bonus step PASSED -> 241664 is max
possible and DEPLOYED at util 0.98 (KV 732,359 tokens / 3.03x concurrency at
241,664-token requests). Util stays 0.98: lowering it shrinks the engine's own
budget instead of buying slack.
Client contract restored (the GLM rollback d91379d dropped all three and the
qwen rollback inherited the gap): --reasoning-parser qwen3 fixes the malformed
thinking block (thinking was landing inline in content with a dangling
</think> and a null reasoning field), --enable-auto-tool-choice with
--tool-call-parser qwen3_xml makes tools work, and
--default-chat-template-kwargs enables enable_thinking + preserve_thinking.
Harness: contextWindow 241664 (compaction gap recomputed), preserve_thinking
moved into chat_template_kwargs.
The 0.98 boot died at KV allocation: torch OOM tried 10.28 GiB (the exact
3x262144 bf16 pool = 11,047,010,304 bytes) with 9.46 GiB free per rank.
The rootless bench probe had booted the same numbers; the root unit's
pinned-expert path holds ~0.8 GiB/rank more. Util 0.90 returns ~2.5 GiB/rank
of workspace with the pool ask unchanged; measured KV capacity law
(798,280 tokens at 11.05e9) unchanged.
Overnight redo campaign verdict + Frank's call: the fp8+MTP4 shape is retired
to git tag frank-primary-fp8-mtp-20261001. The new incumbent: KV cache
unquantized (auto), no speculative decode, KV_BYTES 11,047,010,304/rank
(engine-measured 798,280 tokens, 3.05x concurrency at 262,144-token
requests), MNBT 1024, MoE gather 32x64 (+18.7% decode, grows with depth).
Measured at the deployed shape: 39.38/38.79/36.42 tok/s (166/1087/4064 ctx).
Seam re-pinned: the 7.2 GiB/rank KV ceiling was the MTP4 workspace, not VRAM;
spec decode absent by measurement; bf16 trade documented (clean cache +
262k windows for ~14% decode at 4064 ctx).
- AGENTS.md: fleet table + frank-primary section back to qwen FP8 + MTP4
(37.4-40.6 tok/s, 54% acceptance); GLM runbook moved to the retired index.
- runbook: RETIRED verdict — DFlash2 drafter served and drafted but
acceptance pinned 1.00-1.38 on every text class including verbatim
repetition (mis-wired aux capture, not a weak drafter); 5x slower than
baseline at ~1.2 acceptance.
- log.md: 2026-10-01 rollback entry.
- deploy-remote.sh: LLM_PRIMARY_CONTAINER back to frank-primary-qwen;
glm-probe joins the legacy sweep list.
- frank_primary_unit.bats: re-pinned to the qwen shape — KV qualification
law (7.2 GiB/rank, 3.65x concurrency; 9/16 GiB OOM), fp8 q8/q8 KV
PRESENT, MTP4 PRESENT (3.1-3.4x over the 12.11 no-spec baseline), no tool
dialect, automatic offload. The old 'MTP buys zero' law carried the
prefetch tax and is retired.
DFlash2 experiment declared failed 2026-10-01: the drafter serves and
drafts (12.8 tok/s proposed) but acceptance is pinned at 1.0-1.38 on ALL
text classes including verbatim repetition — the aux-hidden-state capture
semantics are wrong (single-stream entering-state vs post-layer/comb is
the untested alternative). At ~1.2 acceptance the 8-token verify tax makes
decode 5x SLOWER than the 9.9-10.0 baseline (1.6-1.8 tok/s measured).
Restores the verified 37-40 tok/s shape: qwen3.8-flash-next FP8, MTP4,
prefetch off, 200k x 3 seqs, fp8 KV.
use_aux_hidden_state_outputs lives on the gpu_model_runner instance
(gpu_model_runner.py:623/678), not on the spec config, so the first
registry check never fired and the reserve stayed at 13,600 B/token.
Switch to method in {dflash, dspark, extract_hidden_states}.
The dflash boot cleared the unifier and the kpool tail (02/03) and died on
KV capacity: on the generic path the mamba states are NOT aliased into the
MLA pages (the glm5 tensor layout's whole point), so the capacity estimator
demanded ~30.6 GiB fixed against 0.92 GiB available (the plan's KV reserve
registry only knows the target's 13,600 B/token).
Patch 04 keeps the aliased glm5 pool and appends the drafter's uniform
group and per-layer hidden-state groups beside it: grouping partitions
hidden + draft specs, the layout returns a 10-tuple, the builder sizes one
shared backing allocation (pool + extras), and the memory estimator adds
the extras. Patch 04b extends the GLM registry kv_bytes_per_token by the
drafter (4,096 B/token) and hidden-state (5 x hidden x 2 B/token) costs so
the expert plan reserves enough.
Local smoke tests (CPU torch + real spec classes) validate grouping shape,
layout recognition, builder block math and memory accounting against
synthetic GLM+draft+hidden spec sets. Apply-verified: reproduces shas
5521e50b... / 1a623d5c...
Patch 02 (MLA-multiple max page) cleared the KV page-size unifier: the
dflash boot now reaches cudagraph capture. New blocker: the kpool decode
kernel asserts tail_kv_cache.shape[2] == pool_size; KpoolTailSpec pins
block_size == index_kpool, so the unifier's divisible branch must never
grow it. Patch 03 = 02 + a KpoolTailSpec branch that pads page_size_padded
instead. Apply-verified: reproduces patched sha 5093e4b7... Runbook
updated with the full KV grouping journey.
The v4 boot named the actual miss: model_runner.py:425 calls
set_eagle3_aux_hidden_state_layers(self.model) on the instance model_runner
LOADED -- and for GLM-5.3-Flash that is Glm5NextForConditionalGeneration (the
multimodal wrapper; APIServer log: 'Resolved architecture:
Glm5NextForConditionalGeneration'), which wraps the text model as
self.language_model = Glm5NextForCausalLM (glm5next/common/model.py:1113).
v4 put the three presence-checked attributes only on Glm5NextForCausalLM,
so the isinstance check on the conditional still failed with the same
RuntimeError.
v5 adds to BOTH wrappers, MRO-safely (attribute route, no Protocol base):
- Glm5NextForConditionalGeneration: supports_eagle3 / has_own_lm_head /
has_own_embed_tokens + the two methods delegating to self.language_model
(the pixtral.py:498 pattern);
- Glm5NextForCausalLM: the same three attributes + the two methods that
unwrap to self.model (the EagleModelMixin trunk -- v1's correct bodies).
Also proven by MRO simulation: the ForCausalLM's five-base ordering cannot
take SupportsEagle3 as a base in ANY position, so inheritance was never a
viable route for this model.
base 008eeaa2... -> patched bc9913a1..., apply-verified against the pinned
image (patch -p1 reproduces the sha256 and matches the mounted file).
The v4 window boot named the real presence-check rule: isinstance against a
runtime_checkable Protocol walks __protocol_attrs__ (which includes members
inherited from BASE protocols) and getattr_static()s each one off the instance,
breaking on the first missing attribute. SupportsEagleBase declares
has_own_lm_head/has_own_embed_tokens as plain instance-var annotations, so
v3's bare supports_eagle3 flag still failed. Replicated in-image with stock
typing: flag-only class -> False; flag + the two base members -> True.
v4 declares all three on Glm5NextForCausalLM:
supports_eagle3: ClassVar[Literal[True]] = True
has_own_lm_head: bool = False
has_own_embed_tokens: bool = False
(False is correct for the two: the wrapper delegates embed/lm_head to its
trunk. The MRO-safe attribute path is kept from v3.)
base 008eeaa2... -> patched 60f27aad..., apply-verified against the pinned
image (patch -p1 reproduces the sha256 and matches the mounted file).
v2 (inheriting SupportsEagle3 as a base) taught the real lesson: the isinstance
check PASSES and the class then dies at creation with
'Cannot create a consistent method resolution order (MRO) for bases ...'
-- GLM's outer wrapper already inherits vLLM Protocols (SupportsPP,
MixtureOfExperts, HasInnerState, IsHybrid) whose ordering is incompatible with
appending another Protocol base.
v3 avoids the MRO entirely: declare
supports_eagle3: ClassVar[Literal[True]] = True
as a class attribute on Glm5NextForCausalLM. Runtime-checkable Protocols do
presence-based isinstance, so the bare flag satisfies supports_eagle3(model)
without touching the MRO. The aux-layer plumbing stays on the trunk via
EagleModelMixin.
base 008eeaa2... -> patched 4c9a632b..., apply-verified against the pinned
image (patch -p1 reproduces the sha256 and matches the mounted dflash-patch
file byte for byte).
v1 hand-wrote set_aux_hidden_state_layers/get_eagle3_default_aux_hidden_state_layers
and the boot reproduced the identical 'Model does not support EAGLE3 interface'.
Reason: supports_eagle3() is an isinstance check against a runtime-checkable
Protocol whose members include a required ClassVar flag
(interfaces.py:1663 supports_eagle3: ClassVar[Literal[True]] = True), so methods
alone do not satisfy it. Inheriting SupportsEagle3 -- the idiom every model in
this image uses, including the fork's own DFlashLagunaForCausalLM -- supplies the
flag, SupportsEagleBase's has_own_lm_head/has_own_embed_tokens, and working
default implementations of both methods; the hand-written methods are deleted.
base 008eeaa2... -> patched 5eea89c5..., apply-verified against the pinned image.
Runbook: corrected method name, recorded the Protocol trap as the generalizable
lesson for future spec-decode ports.
Clarifies scope in the artifact header: torch.compiler.disable is itself a graph break,
so 05 helps only a hypothetical fullgraph=True route (where the workspace acquisition
would be better restructured out of the forward than made opaque). It is kept because
the discovery is real and the wrapper is a no-op when nothing is compiling, but it is
not required for the break-tolerant route whose only confirmed blocker is the dynamo
name-map KeyError.
Also verified in this pass: glm53-dflash-01-aux-hidden-states.patch applies cleanly to
the pinned image and reproduces its recorded patched sha256 (4ff6ff8d...), so all six
artifacts in scripts/patches are now apply-verified.
glm53-dflash-01-aux-hidden-states.patch: implements the interface whose absence makes
the engine refuse the drafter with 'Model does not support EAGLE3 interface'. Adds
EagleModelMixin to Glm5NextModel, captures the single-stream hidden state entering each
requested layer in the loop, returns (hidden_states, aux_hidden_states) as a tuple when
the list is non-empty, and exposes set_aux_hidden_state_layers /
get_eagle3_default_aux_hidden_state_layers on Glm5NextForCausalLM -- the contract read
out of interfaces.py:1694/1712, eagle3_utils.py:26-31, deepseek_v2.py:1551-1585 and
cudagraph_utils.py:621, with kimi_k25/pixtral as the delegation reference.
Independent of the compile changelist (01-05); it patches the same file as 01, so 01
goes first if both are applied. Scope is honest in the header: this makes the interface
present (the prerequisite for driving DFlash2), it does not promise draft quality and is
not a decode-speed fix.
Five runs of the changelist now reproduce the same single failure with breaks genuinely
tolerated (patch 01+02+03+04, VLLM_COMPILE_FULLGRAPH=0, VLLM_USE_AOT_COMPILE=0,
R4D_GLM_FUSE_QKV_RMSNORM=0): the trunk TRACES ('Dynamo bytecode transform time: 1.08 s')
and the boot dies on KeyError: 'in_proj_qkvbfg_a'.
What it is NOT, each ruled out by measurement rather than argument:
- not one of dynamo's four nn_modules lookups: /home/dank/dynamo-probe wraps three of
them and stayed silent over two runs, with the probe's install line confirmed in both
the API-server and worker processes;
- not a dynamo-internal failure: TORCHDYNAMO_VERBOSE=1 adds no internal stack trace;
- not a literal string lookup: the name appears tree-wide only as the KDA layer's
attribute (kda.py:217, used at :407) and in the load_weights shard table
(model.py:780-785); the four sites in the
quantization utils do compute exactly this bare key but every one is guarded
(), so none can raise it;
- not the pluggable-layer registry, which is keyed by class name.
What it IS: a plain Python KeyError raised inside the traced
Glm5NextDecoderLayer.forward, in a callee dynamo INLINED (frame chain
profile_run -> _dummy_run -> execute_model -> glm4_1v.py:2358 -> the compile wrapper ->
eval_frame.compile_wrapper -> Glm5NextModel.forward:748 -> module._call_impl ->
Glm5NextDecoderLayer.forward:456; the reported line is that method's def line, which is
how inlined frames surface).
Patch 03's header and the runbook now record the remaining instrument (TORCH_LOGS=+dynamo
and grep, or wrap the inlining candidates — the mHC CustomOps and the indexer op — with a
KeyError probe), and the two breaks that are real but non-fatal in this configuration
(patch 04's fused-norm launcher cell, patch 05's workspace acquisition).
tests/run.sh: 362 seams pass.
Root-caused the moving failures: they were my own scratch launcher. Its compile
branches were restructured and the '-e VLLM_COMPILE_FULLGRAPH=0' line was dropped, so
four consecutive boots mounted the wrapper patch while _FULLGRAPH still defaulted to
True — and with fullgraph=True EVERY break is an error, so each boot reported whichever
break came first (cell -> workspace -> compiler-disable). I had blamed
TORCH_LOGS=graph_breaks for the difference; that was wrong, the flag only logs. The
launcher sets the env var again, and patch 04's header, the runbook and the work log
now carry the true run history as a table instead of the bad theory.
New artifact: scripts/patches/glm53-compile-05-workspace-acquisition-opaque.patch —
the indexer acquires its shared workspace INSIDE the forward (its profile-run branch
literally says 'Reserve workspace for indexer during profiling run'), and
v1/worker/workspace.py:_ensure_workspace_size reads ContextVar/ubatch state, which
dynamo cannot trace. Wrap both call sites in a torch.compiler.disable'd helper
(narrower than decorating the workspace manager, which every model would go through;
a no-op when not compiling).
All five artifacts verified against the pinned image: applied in sequence with patch
-p1 they reproduce their recorded patched sha256 exactly (6e9cb81b…, 0751852e…,
732363bb…, 84969993…, c154115a…).
tests/run.sh: 362 seams pass.
Working the dyno name-map blocker one step at a time, with the tools to see it:
- All four patch artifacts are now genuine diff -u patches (they were prose + pseudo
hunks before, useless to patch(1)). Verified against the pinned image: each applies
clean and reproduces its recorded patched sha256 (6e9cb81b… / 0751852e… / 732363bb…
/ 84969993…).
- New: scripts/patches/glm53-compile-04-fused-qkv-rmsnorm-optout.patch — takes GLM's
MLA block off the fused-norm JIT kernel (R4D_GLM_FUSE_QKV_RMSNORM=0; the flag is
already a constructor arg glm5next sets at attention.py:583). Reason: in
models/common/ops/fused_qk_rmsnorm.py the kernel object is bound at line 223, AFTER
its use at line 211 — the launcher family (@kernel_launcher, 39 call sites) is what
breaks dynamo's break analysis.
- New diagnostics, both reusable and recorded: TORCH_LOGS=graph_breaks +
VLLM_USE_AOT_COMPILE=0 to name each break with its user stack, and
/home/dank/dynamo-probe (sitecustomize.py on PYTHONPATH, VLLM_DYNAMO_DUMP_ATTRS=1)
to print the dynamo name-map miss detail. Gotchas: a .pth on PYTHONPATH is NOT
executed, and the scratch launcher reset ENVARGS (now ${ENVARGS:-}), which silently
dropped env vars and wasted a boot.
Measurement discipline correction: I first recorded the 'Read uninitialized cell' and
'Unsupported method call' failures as fatal blockers. They are not established as
such — they appear only when TORCH_LOGS=graph_breaks is set, while a flag-free
break-tolerance run gets past break handling and dies on the name-map KeyError. The
likely mechanism is that graph_breaks logging makes dynamo build a break
*explanation*, and that analysis itself trips on the uninitialized cell. Patch 04's
header, the runbook and the work log now carry the confound explicitly rather than
the overclaim.
tests/run.sh: 362 seams pass.
DFlash2: fetched incoai/GLM-5.3-Flash-DFlash2 (2,342,169,800 B, sha256 matches the
Hub LFS oid) and tried --speculative-config {"method":"dflash","model":"/draft"}.
Config and drafter are accepted; the worker dies with 'Model does not support
EAGLE3 interface' - so the only gap is the target-side aux-hidden-state contract
(set_aux_hidden_state_layers + per-layer capture + the two-value forward return,
as deepseek_v2.py demonstrates twice in this image). Recorded the exact interface,
the GLM-specific unknown (what the single-stream state entering layer k means for
an mHC multi-stream trunk) and the honest expectation: no speedup on a
token-proportional pointwise bound.
fp8 KV: recorded the deciding measurement - the fork's KV reserve model is
dtype-blind (the fp8 boot printed the identical 1760 MiB / 13600 B-per-token plan),
so fp8 cannot buy capacity even if the gfx12 MLA path were rewritten. Verdict: do
not spend effort there.
Compile blocker narrowed from 'a latent GLM bug' to dynamo's name map: the
KeyError is thrown by torch/_dynamo/output_graph.py's nn_modules lookup (:3000,
registered :1259/:1384, _modules[..] prefixes stripped :1225), i.e. the compiler
cannot resolve a submodule by name while tracing the decoder layer. Next step is
TORCHDYNAMO_VERBOSE=1, with the note that the scratch launcher resets ENVARGS.
tests/run.sh: 362 seams pass.
A torch-profiler kernel-time window (496 decode tokens, rank0 trace parsed for
device-activity time) on the deployed GLM-5.3-Flash shape falsifies this page's
own "the expert gather is the bound" explanation:
46.18 s of kernel time / 496 tokens = ~93 ms per token (actual step ~104 ms)
34.8 % vectorized_elementwise_kernel 708,500 launches (1,428/token)
10.0 % r4d_gemm_moe_w4a16_nt_b16_m64 (the MoE GEMM)
6.7 % elementwise_kernel_manual_unroll
6.4 % wvSplitK_hf_sml_ (split-K GEMM)
8.9 % router: sbtopk + warpMergeSortTopK + bitonicSortKVInPlace
6.5 % r4d_mhc_* (hyper-connections)
1.05 % r4d_moe_lru_gather_kernel <-- the retracted suspect
The earlier arithmetic was circular (it assumed the gather consumed the whole
step). The real bound is unfused pointwise glue: the fork compiles its own Qwen
trunk (support_torch_compile in models/qwen4_exp/amd/model.py) and not its GLM
trunk, so the engine boots with "provides neither a compiled submodule nor
breakable CUDA graphs. Overriding to FULL_DECODE_ONLY." That also explains why
MTP4 measured flat — pointwise cost scales with the token count, so spec decode
amortizes nothing here — and it retires arena/hit-rate tuning (the gather is 1 %).
Staged as documented patch artifacts (not wired into any serve script; the
engine does not yet boot with them) — scripts/patches/glm53-compile-0{1,2,3}*.patch:
-01 models/glm5next/common/model.py decorate Glm5NextModel (R4D_GLM_COMPILE)
-02 model_executor/kernels/r4d_lib.py freeze 10 capability probes that do
os.path.exists/.so lookups inside the traced forward. functools.lru_cache
does NOT protect them: dynamo re-traces the wrapped body even when warm
(verified in-image); a plain module-global read compiles.
-03 compilation/wrapper.py make fullgraph=True opt-out, because
suppress_errors does NOT cover graph breaks and this trunk can never
satisfy fullgraph (indexer metadata syncs to host — .item() at
glm5next/amd/sparse_indexer.py:263,419).
With all three the trunk traces (Dynamo bytecode transform time 1.08 s) and dies
on KeyError: in_proj_qkvbfg_a in the mHC/KDA path — a latent bug the compile
exposes, and the remaining blocker.
Also records two structural constraints: no new HIP kernels are possible (r4d.so
ships without sources), which binary-blocks the fused router at GLM's 288
experts; and DFlash2 for GLM is a TARGET-side port, not a drafter port
(incoai/GLM-5.3-Flash-DFlash2 is architectures ["DFlash2DraftModel"] +
model_type qwen3, already mapped at registry.py:636; glm5next lacks only the
aux-hidden-state plumbing deepseek_v2.py demonstrates).
Runbook frontmatter, intro, mermaid and wall section, AGENTS.md and the work log
corrected to match the measurement. tests/run.sh: 362 seams pass.
The deployed unit failed its first boot with the engine core OOMing during
profile/graph capture: "Tried to allocate 662.00 MiB. GPU 0 has a total
capacity of 31.86 GiB of which 240.00 MiB is free ... try setting
PYTORCH_CUDA_ALLOC_CONF=expandable_segments:True". The expert-offload plan
fills VRAM to its modelled headroom by design, so the default allocator's
fragmentation leaves the capture phase nothing — every successful bring-up
probe (both scratch-port and 7770) carried this env, and the repo recipe
dropped it.
Also pins the seam: the setting is part of the qualified shape, not an
optional tuning.
Payload: Intel/GLM-5.3-Flash-W4A16-AutoRound (AutoRound symmetric int4 group
128, auto_round:auto_gptq, 34 shards + 2 extras, 113,074 tensors, ~170 GiB).
It is the only W4A16 checkpoint this fork can serve with expert offload: its
quant_method "auto-round" is overridden to "inc", which routes the MoE to
libr4d's grouped W4A16 GEMM (R4dW4a16MoEMethod). A compressed-tensors W4A16
build of the same model resolves to TritonWNA16Experts, which has no offload
recipe, and refuses to serve at all (170 GiB of experts cannot fit 64 GiB of
VRAM, so offload is mandatory).
Shape: tcclaviger RDNA4 vLLM fork (same digest-pinned image as the 2026-09-29
qwen cutover), TP=2, 131,072 ctx x 2 seqs, bf16 KV (fp8 is ILLEGAL here — it
disables the sparse MLA backend's rope-free Triton path, the only MLA decode
gfx12 has), util 0.98, NBATCH 2048, --enable-expert-offload
--expert-precise-memory, prefix caching, glm47 tool + reasoning parsers,
MTP and expert prefetch absent by measurement.
Measured: decode 9.9-10.0 tok/s, prefill 918 tok/s (warm), arena 14.51 GiB =
2,383 slots/rank over 12,096 host-backed experts (73.64 GiB pinned/rank).
THE 35 tok/s TARGET IS NOT MET and the shortfall is measured: live telemetry
(R4D_LRU_TELEMETRY=1) gives a 0.53-0.66 per-layer hit rate, a step moves
0.82 GiB of misses in ~100 ms = 8.2 GB/s effective against a link measured at
22.74 GB/s — the same structural gather-path gap the qwen campaign found.
Growing the arena 13.08 -> 14.51 GiB changed decode by nothing.
Rejected by measurement: MTP4 (head loads, acceptance 3.76-4.90, but decode
unchanged and prefill collapses 918 -> 255 — a verification forward gathers the
union of the batch's experts), NBATCH 8192 (identical prefill).
Also: scripts/patches/gpu_worker-kv-cap.patch (fork-local fix for a KV-reserve
model that under-counts GLM's per-sequence KDA state; not needed by this shape),
runbook, AGENTS.md row + section, both indexes, work log.
Rollback: 2339fae (qwen FP8 + MTP4).
The classic 'podman images | grep -q' pattern races under set -o pipefail:
grep exits on its first match, podman dies SIGPIPE (141) writing the
remaining RepoDigests lines, and pipefail promotes 141 to the pipeline
status — the gate reads 'no match' and refuses to deploy (2026-09-30,
dankserver). Same bug in the serve script (it would restart-loop the
same way). Fix: capture podman's output into a variable first, grep the
variable. Patched in the R9V September deploys by luck of timing.
Unquoted value + set -a sourcing parses 'qwen-fp8' as a command ->
ExecStart exit 127 -> unit restart loop (caught on the first deploy).
The deployer's wait_for_health was killed and will re-run on the fresh
deploy.
Replace the R9V radiance fork with the tcclaviger RDNA4 vLLM stack (the
digest-pinned image the 2026-09-29 tuning campaign measured), serving
orcarouter/Qwen3.8-Flash-Next-Uncensored-FP8 (131 shards, 172.8 GiB,
already on disk) with MTP4 spec decode.
Qualified by boot probe on 2026-09-29:
- MAXLEN=200000 x SEQS=3 (600k total) at --kv-cache-dtype fp8 (q8/q8 KV):
'GPU KV cache size: 729,545 tokens; Maximum concurrency for 200,000
tokens per request: 3.65x' at KV_BYTES 7.2 GiB PER RANK.
- KV_BYTES is per-rank: global 16G and 9G both OOM (worker tried to
allocate the full size; ~7.5 GiB/rank is the ceiling with the arena).
- prefetch OFF by law (telemetry: 36% precision, ~50% wasted traffic);
TunableOp off (corrupts numerics).
- decode at this shape: 42.94/34.55/36.09 tok/s; arena 6,032 slots.
Trio: frank-primary.env (all FRANK_PRIMARY_* keys), serve-frank-primary.sh
(direct podman run; entrypoint execs vllm serve; waits on the container),
frank-primary.service (root unit, same ExecStop law, new container name
frank-primary-qwen). Deployer gates rewritten (image digest + shard count
+ weight-map index). download-frank-primary.sh is now inventory/verify
(repo is GATED — 401 anonymous, masked LFS hashes; payload already on
disk). Seam rewritten for the new contract. Pre-cutover tag:
frank-primary-r9v-q8-20260929.
The disk was at 93% with 137G free. Nine retired payloads are now moved
to ~/.AGENTTRASH/20260928-retired-models/ with a MANIFEST.md giving a
restore command per item — moved, not deleted, and nothing is freed until
the purge is approved. Every one maps to a runtime already retired in the
runbook index.
Only the .gguf was taken from Qwen3.8-27B-UD-Q8_K_XL/: the live
frank-large DFlash2 drafter lives in that directory and stays put. The
manifest warns against restoring the directory wholesale.
Kept: checkpoints/ (23G) — unreferenced by the repo, but ComfyUI is live
and its workflows were not searched, so it is not confirmed dead. Also
logs/, because x4e-run.log is a SPARSE file (3.3G apparent, 4K real) and
frees nothing.
The reference check caught a live bug: serve-frank-small.sh's fallback
MODEL_DIR still pointed at the retired GSQ-RCO payload frank-small served
on 2026-09-16/17. frank-small.env has always overridden it, so the drift
was invisible — but a missing or unreadable env file would have turned a
config problem into a "missing asset" naming a model that is no longer
there. Repointed at Gemma4-12B-QAT/, which the env file already sets.
Not taken, by request: podman images (279G reclaimable, 95%) and the
uv+pip caches (73G). Together 352G of regenerable garbage, larger than
what was reclaimed here.
The 29.3G payload rollback target went with the rest, so the payload-only
rollback is now a re-fetch first; the runbook and AGENTS.md say so.
The ad-hoc status commands kept producing wrong readings, most seriously
reporting "MemAvailable 0.0 GiB" on a host that actually had 240 GiB free.
Cause: /proc/meminfo separates key from value with a TAB, so
grep MemAvailable /proc/meminfo | tr -s " " | cut -d" " -f2
returns an empty field and awk prints 0.0. A false "host is out of memory"
reading is worse than no reading, because it invites an unnecessary abort.
dankstatus.sh parses on a colon split (tab-agnostic), cross-checks
MemAvailable against MemTotal before printing either, and refuses to print
a zero value. It also pins the rocm-smi CSV column order (device, total,
used) which was previously read swapped, and takes only the first line of
systemctl is-active/is-enabled, which emit more than one line here.
Incidental finding while validating: rocm-smi reports card2 (the 7700X iGPU)
as 0.50 GiB total, not the 123.0 GiB torch.cuda reports. torch was exposing
the GTT aperture. The device is smaller than documented, not larger.
Tests: 346/346 pass (no test surface touched).
Public, ungated repo so no auth header is sent — a token on the curl
command line is world-readable via /proc/<pid>/cmdline. Size and sha256
are the Hub LFS values (31457990272 / 9076f99a…), verified after fetch.
Not wired into frank-large.env: the gufo load test has to pass first.
gufo is a vertical inference engine built for exactly this hardware
(Ryzen AI MAX+ 395 / Radeon 8060S, gfx1151), and Dankclaw's frank-large
has swapped runtimes seven times in search of a better one. This one
ships no prebuilt Linux binary, so the runtime is BUILT FROM SOURCE on
the host and the payload, port, alias and client contract stay exactly
where they are.
Runtime: gufo v0.1.1 (b0f8673) at /home/dank/GitPackages/gufo-v0.1.1,
built by the new scripts/build-gufo.sh. It links the host ROCm 7.2.4
through its RPATH — the opposite of the unsloth prebuilt it replaces,
which carried its own — so two host workarounds are encoded in that
script: Ubuntu's librocwmma-dev installs a truncated rocWMMA header set
that shadows AMD's and cannot be overridden with -I in HIP mode, and the
ROCm clang prefers Ubuntu 26.04's GCC 16 headers over the GCC 15 whose
libstdc++ is actually installed.
Shape is unchanged: one session, full 262144 ctx, the same UD-Q8_K_XL
target and the same DFlash2 drafter, greedy, text-only. Speculation
moves from llama.cpp's fixed depth 4 to gufo's adaptive block policy
capped at 7 proposals.
Client-contract deltas, all documented: the chat template is compiled
into the binary (no --chat-template-file, so the Qwen-Sharp 73k-token
failure mode is gone), reasoning_effort now takes the full OpenAI set
(so models.yml's reasoningEffortMap fold is no longer required — kept
anyway, so this commit does not also move every high role from xhigh to
high effort), and there is no KV-quant or prompt-cache-RAM flag.
The test seam now runs the serve script against a stub engine and
asserts on the argv it actually execs, instead of grepping the script's
text. The deploy handler gates on the runtime binary and returns before
it stops the serving unit.
Measured trade, from gufo's own Q8_K_XL tables against llama.cpp
b11069: prefill +37% to +80%, cold load 4.5x faster, decode +7% at zero
depth — but -14% at 65,536 tokens of depth and -24% at 131,072. On the
8-bit target the crossover is ~32k, and frank-large runs one 262,144
conversation. The Q4_K_XL target shows no such regression.
download-frank-primary-fp8.sh + the fp8 manifest fetched the vLLM 0.30
payload that never deployed; the R9V cutover's payload is the R9V setup
flow's (packages + PLE, state-gated). Scripts move to old/ per policy.
Replace the vLLM 0.30 UVA-offload wiring (never deployed; llama.cpp
b11030 endpoint replaced by Frank's order, downtime window open) with
the R9V radiance-vLLM shape on the same root unit:
- serve-frank-primary.sh: layer the on-host qualified setup state
(104 tuned R9V_* keys, /home/dank/r9v-state/uncensored/setup.json)
under repo production overrides, then exec the fork's own launcher
through the docker->podman shim and block on docker wait so a crashed
engine is a nonzero ExecStart (launch.sh --detach alone would fake
green). Gates: pinned image by OCI manifest digest in RepoDigests,
PLE at pinned byte count (stat -c %s), all GGUF shards + mmproj.
- frank-primary.env: thin production overrides only (LAN bind 0.0.0.0,
:7770, alias frank-primary, container frank-primary-r9v, GPU BDFs);
tuned engine shape stays in the setup state by design.
- frank-primary.service: same root-unit shape (User=dank,
LimitMEMLOCK=infinity for the tiered-expert pin), plus ExecStop
podman stop — docker wait cannot stop the container child.
- deploy-remote.sh deploy_llm_primary: state-driven gates (state ready,
image digest present in rootless storage, PLE bytes, shard files)
replace the fp030 shard-index gate; false-green MainPID check kept.
- tests/frank_primary_unit.bats: seam rewritten for the R9V shape.
Frank
Comrade Chief Designer