bf16 at 2 x 262144 fails with and without --expert-precise-memory. The real
error (captured from the journal this time, container fixed via journald
rather than podman logs):
RuntimeError: Worker failed with error 'Constraints violated
(L['query_start_loc'].size()[0])! You marked it as dynamic but your code
specialized it to be a constant (3)'
- a torch.compile/dynamo dynamic-shape violation, NOT an OOM. Ruled out a
stale compile cache: ~/.cache/vllm holds only modelinfos (16K) and each
podman run gets a fresh container filesystem.
The deltas from the last shape measured serving (fp8 3 x 262144 / MNBT 2048 /
792,069 tokens / 3.02x / arena 6,248) are KV dtype (fp8 -> bf16) and SEQS
(3 -> 2). Restoring the known-good shape; bisect bf16 vs SEQS in a window,
with --enforce-eager as the bypass.
NRestarts climbed to 3, each attempt ~6.7 min then 'Deactivated successfully'
(API server drains after an EngineCore fatal error; Restart=always turned the
silent death into a visible loop). The arena is device-resident, so the budget
decides it: weights 6.28 + pool 6.80 (bf16 2 x 262144) + graphs 0.55 + arena
16.65 (precise overhead 5.25) + non-torch 1.45 = 31.73 of 31.86 GiB -> ~0.13
GiB free at boot, and graph capture is what allocates transiently.
Dropping the flag restores the 7.60 GiB default overhead: arena 14.31 GiB,
total ~29.4 GiB, ~2.4 GiB free. bf16 2 x 262144 (Frank's KV change) stays.
The serve script keeps the opt-in flag; the seam now asserts it is ABSENT.
Frank's call. Payload: ajgazin's abliterated Swift-1.5 UD-Q8_K_XL (29.3 GiB,
SWIFT OPEN LICENSE) -> unsloth/Qwen3.8-27B-GGUF UD-Q4_K_XL (16.35 GiB,
sha256 3f227079...b01e, vanilla/Apache-2.0). NOTE: the endpoint is no longer
uncensored — that was one of the reasons for the Swift payload on 2026-09-28.
New fetcher scripts/download-frank-large-q4.sh (no auth header: repo is public
and ungated; size + sha256 verified after transfer). The speculative drafter
and its fetcher are unchanged.
Shape: --sessions 4 --context 163840 with unquantized KV. gufo exposes no KV
quantization lever (execution_policy.hpp pins kv_cache_storage to kFp16), so
f16 is inherent.
KV arithmetic: f16 KV = 64 KiB/token (16 full-attention layers x 4 KV heads x
256 head_dim x 2 x 2B; the other 48 layers are linear-attention with a
per-sequence fp32 state). 4 x 163840 = 655,360 tokens = 40.0 GiB KV, each
session exactly 10 GiB. Budget ~62 GiB of the 114 GiB GTT.
Seam retargeted (FETCH -> the q4 script, TARGET_STUB, context/sessions pins).
Both of Frank's changes at once.
KV is bf16 (--kv-cache-dtype auto) at two full-size windows (2 x 262144).
bf16 measures 13,839 B/token (bf16 3x241664 boot: 11,047,010,304 B -> 798,280
tokens), decomposing as 12,276 attention + 1,563 mamba/side caches vs fp8's
6,138 + 1,563 = 7,701. 524,288 tokens needs 7.256e9 B; pinned at 7.3e9
(~527,500 tokens, 3,206 margin). Pool cost: 5.68 -> 6.76 GiB per rank.
--expert-precise-memory selects the fitted overhead (5.25 GiB) over the default
(7.60 GiB = fitted + compile/graph-capture headroom). It moves only the plan's
term, so the expert arena goes 6,248 -> ~7,278 slots (~1,025 more) at the
cost of boot-time capture slack. Overhead adds 0.22 GiB/1024 nbt above 4096, so
MNBT 2048 pays nothing.
Seam + docs + AGENTS row + log updated.
The gate failed twice in the same spec for two different reasons and then passed
33/33 on a manual re-run. Future sessions should re-run before treating the
DEPLOY INCOMPLETE banner as a real fleet fault.
Frank's call 2026-10-01: MAXLEN 262144 x SEQS 3 = 786,432 tokens of contract.
Measured fp8 KV costs 7,698 B/token (4-slot boot 8,311,013,376 B -> 1,079,582
tokens; 5-slot boot 10,388,766,720 B -> 1,350,182 tokens), so the pool floor
for the contract is 6.054e9 B. Pinned at 6,100,000,000 (~792k tokens, ~5.9k
margin) so the remaining ~2.2 GiB versus the 4-slot shape goes to the expert
ARENA - the fork sizes its decode-critical expert cache from the remainder
after weights, pool and workspace.
MNBT 2048 (Frank): the arena sweep measured 6,499 slots at every rung <= 4096,
so the larger batch costs the arena nothing.
Seam retargeted (3 seqs / 6.1e9 / MNBT 2048); docs + AGENTS row + log updated,
including the closed DCP investigation.
Five 262144-token contexts passed every boot gate (KV 1,350,182 / 5.15x) but
left 0 bytes free at runtime: a real client request (995-token prompt,
max_tokens=261149, sampling params straight from models.yml) killed Worker_TP1
with torch.OutOfMemoryError 'Tried to allocate 50.00 MiB ... 0 bytes free'.
Boot is not proof of runtime survivability.
Four slots x 262144 fp8 (KV_BYTES 8,311,013,376/rank) returns ~1.93 GiB per
rank for activations/graph capture.
Unit: Restart=always instead of on-failure — a runtime worker fault makes
vLLM's API server exit clean, so ExecStart returned 0 and on-failure never
fired, leaving the endpoint down until a human noticed. Also
StartLimitIntervalSec=0 and OOMScoreAdjust=-500.
Frank's call after the night ladder: five contexts at the model's full native
262,144 window beat six shortened ones. Six full-length contexts do NOT fit at
any KV dtype (6 x 241664 fp8 asked 10.70 GiB against 9.18 GiB free — the
6-seq graph/activation footprint leaves too little for the pool); 5 x 262144
at fp8 PASSED on the first rung: KV_BYTES 10,388,766,720/rank, measured
'GPU KV cache size: 1,350,182 tokens, Maximum concurrency for 262,144 tokens
per request: 5.15x' (39,462 tokens of margin).
Also documented: the SEQS=3 bf16 max (241,664, KV 732,359 / 3.03x) measured
earlier the same night, and the root-cause read that the 7.2 GiB KV ceiling
law was the MTP4 workspace, not VRAM. Seam and harness (contextWindow 262144)
re-pinned.
Night ladder result (Frank's subtract-16k method): 262144 and 245760 both OOM
at CUDA graph capture ('Tried to allocate 76.00 MiB ... 0 bytes free'), 237568
passed (KV 718,606 tokens), and the +4096 bonus step PASSED -> 241664 is max
possible and DEPLOYED at util 0.98 (KV 732,359 tokens / 3.03x concurrency at
241,664-token requests). Util stays 0.98: lowering it shrinks the engine's own
budget instead of buying slack.
Client contract restored (the GLM rollback d91379d dropped all three and the
qwen rollback inherited the gap): --reasoning-parser qwen3 fixes the malformed
thinking block (thinking was landing inline in content with a dangling
</think> and a null reasoning field), --enable-auto-tool-choice with
--tool-call-parser qwen3_xml makes tools work, and
--default-chat-template-kwargs enables enable_thinking + preserve_thinking.
Harness: contextWindow 241664 (compaction gap recomputed), preserve_thinking
moved into chat_template_kwargs.
- AGENTS.md: fleet table + frank-primary section back to qwen FP8 + MTP4
(37.4-40.6 tok/s, 54% acceptance); GLM runbook moved to the retired index.
- runbook: RETIRED verdict — DFlash2 drafter served and drafted but
acceptance pinned 1.00-1.38 on every text class including verbatim
repetition (mis-wired aux capture, not a weak drafter); 5x slower than
baseline at ~1.2 acceptance.
- log.md: 2026-10-01 rollback entry.
- deploy-remote.sh: LLM_PRIMARY_CONTAINER back to frank-primary-qwen;
glm-probe joins the legacy sweep list.
- frank_primary_unit.bats: re-pinned to the qwen shape — KV qualification
law (7.2 GiB/rank, 3.65x concurrency; 9/16 GiB OOM), fp8 q8/q8 KV
PRESENT, MTP4 PRESENT (3.1-3.4x over the 12.11 no-spec baseline), no tool
dialect, automatic offload. The old 'MTP buys zero' law carried the
prefetch tax and is retired.
Runbook: the five-version evolution recorded, each trap named (method
hand-write, Protocol-base MRO kill, wrong-wrapper attrs). The decisive
discovery: the runner checks the LOADED instance, which for GLM-5.3-Flash
is the multimodal Glm5NextForConditionalGeneration holding the text model
as self.language_model — v4's attrs on Glm5NextForCausalLM were invisible
to it. v5 (attrs + methods on both wrappers) passes: both TP workers log
'Using Eagle3 auxiliary layers from config: (6, 15, 25, 34, ...)' and the
DFlash2 drafter resolves + loads. Log: the DFlash bullet updated from
'the trunk simply does not implement' to 'gap closed; serving end state
still unproven (window RAM contention interrupted the boot twice)'.
Root-caused the moving failures: they were my own scratch launcher. Its compile
branches were restructured and the '-e VLLM_COMPILE_FULLGRAPH=0' line was dropped, so
four consecutive boots mounted the wrapper patch while _FULLGRAPH still defaulted to
True — and with fullgraph=True EVERY break is an error, so each boot reported whichever
break came first (cell -> workspace -> compiler-disable). I had blamed
TORCH_LOGS=graph_breaks for the difference; that was wrong, the flag only logs. The
launcher sets the env var again, and patch 04's header, the runbook and the work log
now carry the true run history as a table instead of the bad theory.
New artifact: scripts/patches/glm53-compile-05-workspace-acquisition-opaque.patch —
the indexer acquires its shared workspace INSIDE the forward (its profile-run branch
literally says 'Reserve workspace for indexer during profiling run'), and
v1/worker/workspace.py:_ensure_workspace_size reads ContextVar/ubatch state, which
dynamo cannot trace. Wrap both call sites in a torch.compiler.disable'd helper
(narrower than decorating the workspace manager, which every model would go through;
a no-op when not compiling).
All five artifacts verified against the pinned image: applied in sequence with patch
-p1 they reproduce their recorded patched sha256 exactly (6e9cb81b…, 0751852e…,
732363bb…, 84969993…, c154115a…).
tests/run.sh: 362 seams pass.
Working the dyno name-map blocker one step at a time, with the tools to see it:
- All four patch artifacts are now genuine diff -u patches (they were prose + pseudo
hunks before, useless to patch(1)). Verified against the pinned image: each applies
clean and reproduces its recorded patched sha256 (6e9cb81b… / 0751852e… / 732363bb…
/ 84969993…).
- New: scripts/patches/glm53-compile-04-fused-qkv-rmsnorm-optout.patch — takes GLM's
MLA block off the fused-norm JIT kernel (R4D_GLM_FUSE_QKV_RMSNORM=0; the flag is
already a constructor arg glm5next sets at attention.py:583). Reason: in
models/common/ops/fused_qk_rmsnorm.py the kernel object is bound at line 223, AFTER
its use at line 211 — the launcher family (@kernel_launcher, 39 call sites) is what
breaks dynamo's break analysis.
- New diagnostics, both reusable and recorded: TORCH_LOGS=graph_breaks +
VLLM_USE_AOT_COMPILE=0 to name each break with its user stack, and
/home/dank/dynamo-probe (sitecustomize.py on PYTHONPATH, VLLM_DYNAMO_DUMP_ATTRS=1)
to print the dynamo name-map miss detail. Gotchas: a .pth on PYTHONPATH is NOT
executed, and the scratch launcher reset ENVARGS (now ${ENVARGS:-}), which silently
dropped env vars and wasted a boot.
Measurement discipline correction: I first recorded the 'Read uninitialized cell' and
'Unsupported method call' failures as fatal blockers. They are not established as
such — they appear only when TORCH_LOGS=graph_breaks is set, while a flag-free
break-tolerance run gets past break handling and dies on the name-map KeyError. The
likely mechanism is that graph_breaks logging makes dynamo build a break
*explanation*, and that analysis itself trips on the uninitialized cell. Patch 04's
header, the runbook and the work log now carry the confound explicitly rather than
the overclaim.
tests/run.sh: 362 seams pass.
DFlash2: fetched incoai/GLM-5.3-Flash-DFlash2 (2,342,169,800 B, sha256 matches the
Hub LFS oid) and tried --speculative-config {"method":"dflash","model":"/draft"}.
Config and drafter are accepted; the worker dies with 'Model does not support
EAGLE3 interface' - so the only gap is the target-side aux-hidden-state contract
(set_aux_hidden_state_layers + per-layer capture + the two-value forward return,
as deepseek_v2.py demonstrates twice in this image). Recorded the exact interface,
the GLM-specific unknown (what the single-stream state entering layer k means for
an mHC multi-stream trunk) and the honest expectation: no speedup on a
token-proportional pointwise bound.
fp8 KV: recorded the deciding measurement - the fork's KV reserve model is
dtype-blind (the fp8 boot printed the identical 1760 MiB / 13600 B-per-token plan),
so fp8 cannot buy capacity even if the gfx12 MLA path were rewritten. Verdict: do
not spend effort there.
Compile blocker narrowed from 'a latent GLM bug' to dynamo's name map: the
KeyError is thrown by torch/_dynamo/output_graph.py's nn_modules lookup (:3000,
registered :1259/:1384, _modules[..] prefixes stripped :1225), i.e. the compiler
cannot resolve a submodule by name while tracing the decoder layer. Next step is
TORCHDYNAMO_VERBOSE=1, with the note that the scratch launcher resets ENVARGS.
tests/run.sh: 362 seams pass.
A torch-profiler kernel-time window (496 decode tokens, rank0 trace parsed for
device-activity time) on the deployed GLM-5.3-Flash shape falsifies this page's
own "the expert gather is the bound" explanation:
46.18 s of kernel time / 496 tokens = ~93 ms per token (actual step ~104 ms)
34.8 % vectorized_elementwise_kernel 708,500 launches (1,428/token)
10.0 % r4d_gemm_moe_w4a16_nt_b16_m64 (the MoE GEMM)
6.7 % elementwise_kernel_manual_unroll
6.4 % wvSplitK_hf_sml_ (split-K GEMM)
8.9 % router: sbtopk + warpMergeSortTopK + bitonicSortKVInPlace
6.5 % r4d_mhc_* (hyper-connections)
1.05 % r4d_moe_lru_gather_kernel <-- the retracted suspect
The earlier arithmetic was circular (it assumed the gather consumed the whole
step). The real bound is unfused pointwise glue: the fork compiles its own Qwen
trunk (support_torch_compile in models/qwen4_exp/amd/model.py) and not its GLM
trunk, so the engine boots with "provides neither a compiled submodule nor
breakable CUDA graphs. Overriding to FULL_DECODE_ONLY." That also explains why
MTP4 measured flat — pointwise cost scales with the token count, so spec decode
amortizes nothing here — and it retires arena/hit-rate tuning (the gather is 1 %).
Staged as documented patch artifacts (not wired into any serve script; the
engine does not yet boot with them) — scripts/patches/glm53-compile-0{1,2,3}*.patch:
-01 models/glm5next/common/model.py decorate Glm5NextModel (R4D_GLM_COMPILE)
-02 model_executor/kernels/r4d_lib.py freeze 10 capability probes that do
os.path.exists/.so lookups inside the traced forward. functools.lru_cache
does NOT protect them: dynamo re-traces the wrapped body even when warm
(verified in-image); a plain module-global read compiles.
-03 compilation/wrapper.py make fullgraph=True opt-out, because
suppress_errors does NOT cover graph breaks and this trunk can never
satisfy fullgraph (indexer metadata syncs to host — .item() at
glm5next/amd/sparse_indexer.py:263,419).
With all three the trunk traces (Dynamo bytecode transform time 1.08 s) and dies
on KeyError: in_proj_qkvbfg_a in the mHC/KDA path — a latent bug the compile
exposes, and the remaining blocker.
Also records two structural constraints: no new HIP kernels are possible (r4d.so
ships without sources), which binary-blocks the fused router at GLM's 288
experts; and DFlash2 for GLM is a TARGET-side port, not a drafter port
(incoai/GLM-5.3-Flash-DFlash2 is architectures ["DFlash2DraftModel"] +
model_type qwen3, already mapped at registry.py:636; glm5next lacks only the
aux-hidden-state plumbing deepseek_v2.py demonstrates).
Runbook frontmatter, intro, mermaid and wall section, AGENTS.md and the work log
corrected to match the measurement. tests/run.sh: 362 seams pass.
Payload: Intel/GLM-5.3-Flash-W4A16-AutoRound (AutoRound symmetric int4 group
128, auto_round:auto_gptq, 34 shards + 2 extras, 113,074 tensors, ~170 GiB).
It is the only W4A16 checkpoint this fork can serve with expert offload: its
quant_method "auto-round" is overridden to "inc", which routes the MoE to
libr4d's grouped W4A16 GEMM (R4dW4a16MoEMethod). A compressed-tensors W4A16
build of the same model resolves to TritonWNA16Experts, which has no offload
recipe, and refuses to serve at all (170 GiB of experts cannot fit 64 GiB of
VRAM, so offload is mandatory).
Shape: tcclaviger RDNA4 vLLM fork (same digest-pinned image as the 2026-09-29
qwen cutover), TP=2, 131,072 ctx x 2 seqs, bf16 KV (fp8 is ILLEGAL here — it
disables the sparse MLA backend's rope-free Triton path, the only MLA decode
gfx12 has), util 0.98, NBATCH 2048, --enable-expert-offload
--expert-precise-memory, prefix caching, glm47 tool + reasoning parsers,
MTP and expert prefetch absent by measurement.
Measured: decode 9.9-10.0 tok/s, prefill 918 tok/s (warm), arena 14.51 GiB =
2,383 slots/rank over 12,096 host-backed experts (73.64 GiB pinned/rank).
THE 35 tok/s TARGET IS NOT MET and the shortfall is measured: live telemetry
(R4D_LRU_TELEMETRY=1) gives a 0.53-0.66 per-layer hit rate, a step moves
0.82 GiB of misses in ~100 ms = 8.2 GB/s effective against a link measured at
22.74 GB/s — the same structural gather-path gap the qwen campaign found.
Growing the arena 13.08 -> 14.51 GiB changed decode by nothing.
Rejected by measurement: MTP4 (head loads, acceptance 3.76-4.90, but decode
unchanged and prefill collapses 918 -> 255 — a verification forward gathers the
union of the batch's experts), NBATCH 8192 (identical prefill).
Also: scripts/patches/gpu_worker-kv-cap.patch (fork-local fix for a KV-reserve
model that under-counts GLM's per-sequence KDA state; not needed by this shape),
runbook, AGENTS.md row + section, both indexes, work log.
Rollback: 2339fae (qwen FP8 + MTP4).
PORTMAP=7773:7770 hook; health 200 on 127.0.0.1 and 192.168.0.235,
decode through the published path in band. 7773 chosen over 7770
(frank-primary owns 7770 when it returns). Log bullet recovered from a
misplaced insert into the 2026-09-28 cleanup section.
Repo tag qwen-fp8-tuned-20260929 at f532e03; serve script launches the
tcclaviger image by digest sha256:da402966... (local alias
qwen-fp8-runtime:tuned-20260929) — the floating :latest is no longer
referenced.
Source reading in the qwen-fp8 container: glm5next is registered with an
amd/ backend dir + DSA indexer; r4d.so has native moe_dsfp4/moe_dsfp4_mt1
entries (has_dsfp4()==True), so the old 'MXFP4 = emulation, missing
kernels' verdict is dead for this fork. DSFP4 = MXFP4 weights + DeepSeek
activation law. Offload recipes exist for mxfp4 and W4A16 (R4dW4a16MoEExperts
in offloader/expert.py:114). GLM gate is now the payload format
(compressed-tensors MXFP4/W4A16, none public) + a HIP boot probe.
The KV trim is inert: arena identical (5,648 slots) at 4.5 and 4.03 GiB
(planner buckets; the trim went to margin). Final 256k config
MAXLEN=262144 KV_BYTES=4328521728 prefetch off; full-speed alternative
MAXLEN=184224 at 3 GiB.
Asked for iterative decode/prefill improvement at a 32k working point
without touching the model or the quantization. There isn't one: gufo's
served settings are already at this engine's tuned point, so no config
change was made and none is proposed.
Measured on the repo's own prose and shell sources at 32,762 prompt
tokens, salted per run against a warm prefix cache, 3-5 repetitions:
* --prefill-chunk 512 -> 8192 moves prefill 410.7-412.4 t/s. A 16x change
in chunk size, 0.4% out. The lever is dead at this depth.
* --draft-tokens 5 -> 10 gives 13.79-15.86 t/s decode, all inside the
run-to-run noise. The 5-rep confirmation REVERSED the 3-rep leaders:
dt9+chunk4096 looked best and came last (12.86), the deployed config
came first (14.71).
* --draft-policy fixed vs adaptive: no effect. --min-draft-tokens:
rejected by the engine ("DFlash2 requires ... 1"). MTP: unsupported for
Q8 targets (token_embd.weight is Q8_0).
* --context 262144 -> 40960, tested as a diagnostic: no gain, and it
would break models.yml's declared contextWindow. Not shipped.
What does move the number is speculation itself — 6.65 t/s off vs ~15 t/s
on, 5.3x — and it is already enabled.
The two traps that cost most of the wall time are now in AGENTS.md's
benchmarking policy, because both produced FALSE results:
1. gufo bench cannot rank speculative settings and inverted the ranking.
Its corpus reports acceptance=0.933 +/- 0.000 with an identical trace
sha256 across draft widths — the same highly-predictable text every
run. It reported 29-37 t/s where real traffic gives ~15, and made
dt16 look 6% faster than dt7 when real text made it 14% slower.
2. A live A/B needs >=128 generated tokens. At 8, inter_token is
first-token noise; the first pass produced a nonsense ordering that
the 128-token pass reversed.
Plus the Server-Timing header is `ttft;dur=402.485` — the unit rides in
the NAME half, so splitting on "=" keys the dict "ttft;dur" and every
lookup silently returns nothing. Four configs reported "no Server-Timing"
while the header was present on all of them.
The disk was at 93% with 137G free. Nine retired payloads are now moved
to ~/.AGENTTRASH/20260928-retired-models/ with a MANIFEST.md giving a
restore command per item — moved, not deleted, and nothing is freed until
the purge is approved. Every one maps to a runtime already retired in the
runbook index.
Only the .gguf was taken from Qwen3.8-27B-UD-Q8_K_XL/: the live
frank-large DFlash2 drafter lives in that directory and stays put. The
manifest warns against restoring the directory wholesale.
Kept: checkpoints/ (23G) — unreferenced by the repo, but ComfyUI is live
and its workflows were not searched, so it is not confirmed dead. Also
logs/, because x4e-run.log is a SPARSE file (3.3G apparent, 4K real) and
frees nothing.
The reference check caught a live bug: serve-frank-small.sh's fallback
MODEL_DIR still pointed at the retired GSQ-RCO payload frank-small served
on 2026-09-16/17. frank-small.env has always overridden it, so the drift
was invisible — but a missing or unreadable env file would have turned a
config problem into a "missing asset" naming a model that is no longer
there. Repointed at Gemma4-12B-QAT/, which the env file already sets.
Not taken, by request: podman images (279G reclaimable, 95%) and the
uv+pip caches (73G). Together 352G of regenerable garbage, larger than
what was reclaimed here.
The 29.3G payload rollback target went with the rest, so the payload-only
rollback is now a re-fetch first; the runbook and AGENTS.md say so.
The target moves from unsloth/Qwen3.8-27B-GGUF:UD-Q8_K_XL to
ajgazin/Swift-1.5-Qwen3.8-27B-Uncensored-Dynamic-MTP-GGUF's UD-Q8_K_XL —
an abliterated fine-tune of the same base, under the non-OSI SWIFT OPEN
LICENCE 1.0. Port, alias, context, drafter and client contract are
unchanged.
Compatibility was settled from the GGUF headers before fetching anything:
both files parse to the same architecture (qwen35), the same 866 tensors
with identical names in identical order, zero per-tensor ggml_type or dim
mismatches, and the same MTP head. Two metadata differences that look
alarming are no-ops, confirmed against gufo's source — general.file_type is
deliberately not consulted (the file gufo already served records Q4_K_M
while containing no Q4_K tensor), and Swift's extra
qwen35.attention.recurrent_layers key is never read; layer kinds come from
full_attention_interval, which both files carry as 4 and which agrees.
The load test agreed: 9.2 s to listening, 66.5 GiB of the 116.7 GiB iGPU,
no loader warnings, and a green contract battery including a two-turn tool
loop.
The DFlash2 drafter did not move, so the payload is now two files in two
directories: DRAFTER_DIR is its own tunable. That broke the deployer's
stray-run sweep, which keyed on the target filename and would have stopped
matching — a stray run would then have held 7770 past the port gate. It
now keys on the drafter, which every frank-large command line carries.
The fetch script sends no auth header: the repo is public, and a token on
the curl command line is world-readable via /proc/<pid>/cmdline. The
anonymous transfer ran at 1.8 MiB/s; the host token through a mode-600
curl -K config reached 7 MiB/s.
Test seam: the mutation pass found four real gaps. Two fetch assertions
were satisfied by a header comment rather than the URL assignment, so
repointing a repository left them green, and nothing bound the env's
MODEL_FILE/DRAFTER_FILE to the filenames the fetch scripts download. Both
are now explicit. A harness bug worth recording: a bats --filter that
matches no test reports zero "not ok", indistinguishable from a mutation
that survived — the harness now checks the 1..N count. 16/16 caught.
The ad-hoc status commands kept producing wrong readings, most seriously
reporting "MemAvailable 0.0 GiB" on a host that actually had 240 GiB free.
Cause: /proc/meminfo separates key from value with a TAB, so
grep MemAvailable /proc/meminfo | tr -s " " | cut -d" " -f2
returns an empty field and awk prints 0.0. A false "host is out of memory"
reading is worse than no reading, because it invites an unnecessary abort.
dankstatus.sh parses on a colon split (tab-agnostic), cross-checks
MemAvailable against MemTotal before printing either, and refuses to print
a zero value. It also pins the rocm-smi CSV column order (device, total,
used) which was previously read swapped, and takes only the first line of
systemctl is-active/is-enabled, which emit more than one line here.
Incidental finding while validating: rocm-smi reports card2 (the 7700X iGPU)
as 0.50 GiB total, not the 123.0 GiB torch.cuda reports. torch was exposing
the GTT aperture. The device is smaller than documented, not larger.
Tests: 346/346 pass (no test surface touched).
Live numbers from the 2026-09-28 deploy: 13 s load to listening, RSS
30.6 GiB, ttft 549.8 ms / inter-token 24.2 ms on a short prompt, and a
green contract battery (health, ready, /v1/models at 262,144, content +
reasoning_content, tool_calls, SSE, and a two-turn tool loop).
Also recorded: gufo accepts every request field the harness sends in
extraBody and the full reasoning_effort set, both verified 200 — which
is why models.yml's reasoningEffortMap fold is now optional rather than
required, and why it is being kept anyway.
Answers Frank's question: given the 2x R9700 box, which of DeepSeek-V4.1-Flash,
MiMo-V2.6-Flash-MOPD and GLM-5.3-Flash runs best, on what runtime, and can
Affinity / vllm-radiance / FreeToken-AMD be modified to run them fast.
Sizing, measured from the HF API rather than the model cards:
- DS4.1 payload 475.27 GiB, of which 189.13 GiB is Engram (two 94.56 GiB
shards, the layer 1 and 14 lookup tables). Lucebox never maps them, so
they do not evict the expert page cache.
- 240 routed experts x ~11 MB per token at the 37-of-384-per-layer residency
the hot stack can hold = ~2.4 GB across PCIe per token. At 55 GB/s real on
Gen5 x8 that is a ~23 tok/s ceiling, under the 35 tok/s goal before any
attention or dense work.
- The backend refuses layer-major prefill for V4.1, so the only legal
prefill is token-by-token; 1000 t/s is ~20x away.
Affinity (Codeberg StillDeadcode) builds clean here (hipcc 23, gfx1201, zero
errors) and claims 110 tok/s for the 284B V4-Flash, because the 101 GiB
container leaves the cold expert set in a pinned host pool in RAM. V4.1 is
2.5x too big to repeat that. Port scope, from both checkpoints' real
safetensors indices: expert machinery ports unchanged (identical tensor
naming, 15,360 tensors at 40x384), 26/32 per-layer tensors and 34/35 config
keys present. Five gaps - Engram, the disjoint {0,1,2} vs {0,4,128} ratio
vocabulary, CED, the DSpark draft geometry, vision. num_hash_layers is the
only key V4.1 lacks, so model.h:36's default of 3 silently survives and
would route layers 0-2 through a table that does not exist.
Also corrects: Dankserver RAM 60 -> 246 GiB (stale since the 2026-09-20 board
swap) and storage; the top-level index credited V4.1 with V4's geometry
(43 layers / 256 experts vs the actual 40 / 384).
Each LLM card now carries a details object (role/spec/use) rendered as a
bold pale-green role, a bright-green spec line, and a dim 'Use:' line.
app.js descHTML() falls back to the plain description for non-LLM cards;
new .desc-role/.desc-spec/.desc-use CSS + bare-mode reset.
Roles made neutral (no nicknames): frank-large = deep reasoning,
frank-medium = primary model, frank-small = fast utility model.
The /docs model guide was fixed — it had drifted to retired model info
(frank-large as Qwen3.8-Flash-Next 87.3 GiB/200k, frank-medium as FP8
~31 GiB) — now Qwen3.8-27B UD-Q8_K_XL 31.5 GiB/262k and MXFP4/W4A8
19.4 GiB, reordered large->medium->small. bats 30/30 + live render.
The auth works via the modern /Users/AuthenticateByName endpoint (what the
web UI uses). The 401 was a test artifact from probing the dead legacy route.
No config change needed; LDAP-Auth plugin + authentik 2026.8.3 outpost healthy.
Knowledge-base follow-up to a45f396. The runbook, both index one-liners, and
the AGENTS.md topology row all still declared FP8 active; flipped to
just1moremodel/Qwen3.8-27B-Uncensored-MXFP4-awq with unquantized bf16 KV (pool
486,464 tokens, full 262,144 window), and recorded the causal nuance: fp8 KV
was on both arms, so the FP8-vs-MXFP4 gap was weight precision, not KV - clean
KV is a quality hypothesis for W4A8, not a throughput one. No verdict yet.
The quadlet's "TEXT ONLY ... 400 here" comment was wrong: vision works on the
MXFP4 checkpoint (Qwen3_5ForConditionalGeneration carries the vision tower
through the Gemma4 merge path, unquantized) - verified by naming a solid red
48x48 PNG "Red". Comment corrected; harness models.yml input stays [text,image].
Docs-only; no re-deploy (the endpoint already serves vision).
The first cutover crash-looped the unit with 'invalid split file name':
the fetch script concatenated the three split parts into a single GGUF,
but part 1's header carries ggml.file_count=3, so the byte-correct
concatenation still advertises itself as a split and the loader demands
a -NNNNN-of-NNNNN filename.
- serve-frank-large.sh: -m points at part 1; llama.cpp loads the split
natively (parts 2-3 auto-discovered in the same directory)
- deploy-remote.sh: LLM_LARGE_MODEL gate path follows
- download-frank-large-unsloth.sh: assembly removed; the parts ARE the
canonical payload, verified against live Hub LFS oids
- docs: runbook model section + troubleshooting note, AGENTS.md, log
- the redundant 87 GiB assembled file is staged on dankclaw at
~/.AGENTTRASH/20260921-frank-large-assembled/
New runbook frank-large-unsloth-qwen38-flash-next.md (runtime, payload,
deployed config, the explicit -md, the 114 GiB GTT memory model, client
contract, rollback). frank-large-ciru-orca.md marked RETIRED (rollback
target, payload and runtime intact on disk). AGENTS.md topology row,
frank-large section, harness note, runbook index, and the GPU-budget
lesson (GTT aperture ~114 GiB, not the stale 64 GiB rocm-smi figure —
the CIRU Orca held 92.6 GiB on the same device). LLM index header,
mermaid, active list, and the vault root index updated. Work-log entry.
DFlash2 (100.0 t/s decode, 287.1 t/s burst-4) beats MTP self-speculation
(85.9 t/s, 179.7 t/s) despite a lower accept length (4.24 vs 5.66): the
small separate drafter steps faster than the target's own MTP layer applied
autoregressively. DFlash2 is the deployed shape; both swap-script paths now
default to dflash/7. Docs updated.
- shared/prometheus/rules/llm-fleet.yml: the llamacpp reuse recording rule
divided cached by prompt, but llama.cpp's prompt_tokens_total EXCLUDES
cached tokens (its own HELP line), so the ratio was unbounded — measured
77 on a fully-warm frank-large, meaning LlmPrefixCacheCollapse could never
fire on that side. Now cached / (cached + prompt), with the traffic floor
counting both terms too; the header documents the unit trap. Also fixed the
LlmEndpointDown description: frank-large's CIRU llama.cpp build DOES export
the llamacpp:* family (verified live), so the alert covers all three
endpoints.
- shared/grafana/provisioning/dashboards/llm-fleet.json: re-cut for two
namespaces. Every panel reads both (the endpoint picker selects); the
KV-pool-fill panel retired 2026-09-16 returned with
vllm:kv_cache_usage_perc; the latency p95 panel returned with the vLLM
histograms (TTFT/ITL/e2e); the alerts panel is scoped to the fleet's
Llm.* rules; the stale halogen-era description is gone. All 21 panel
expressions validated against the live endpoints.
- runbook + log + Observability: prefix caching measured, not inferred —
reuse happens at 1,648-token block granularity (attention block size padded
to the mamba page size); an identical 13,021-token prompt went 2.30 s cold
to 0.68 s warm with 9,888 tokens (76 %) reused, while a 2,521-token prompt
registered zero hits. Rule counts corrected to 6 alerts + 3 recording rules.
./tests/run.sh: 325 pass.
The radiance TP=2 cutover changed the metric namespace frank-medium exports
(llamacpp:* -> vllm:*) and left two scrape targets addressing host processes
by an address a container cannot reach.
- shared/prometheus/rules/llm-fleet.yml: two namespaces side by side. Added
LlmRequestsWaiting, LlmKvCacheSaturated (vllm-only gauges llama.cpp never
had), the job:llm_vllm_prefix_cache_reuse:rate5m and
job:llm_spec_accept_length:rate5m recording rules, and re-cut
LlmPrefixCacheCollapse / LlmSpecDecodeOff to OR both families with
per-family traffic floors so an idle endpoint cannot fire them. Every
expression validated against live Prometheus.
- shared/prometheus/prometheus.yml: frank-small -> host.containers.internal:7771
and frank-large scraped at host.containers.internal:7770. Both are HOST
processes on Dankclaw and netavark DNATs published container ports only, so
the LAN-IP targets were red by construction; frank-large's CIRU llama.cpp
build does export the llamacpp:* family, contra the old comment.
- tests/observability_quadlet.bats: the two seams now encode the new contract
(per-endpoint reachability by process type; every named series must be on the
measured list for its engine; engine-independent rules must name both).
- homepage docs.html/services.json: frank-medium sees pictures, frank-large is
CIRU Orca with a single seat (was halogen w4b, two seats, 200k).
- docs: radiance runbook is ACTIVE (deployed flags, three-arm battery, verified
capabilities, rollback, replica-era sections labelled); llama.cpp runbook
carries the SUPERSEDED banner; Dual-R9700-Cooperation states the measured
verdict; Observability records the namespace split and the DNAT lesson;
AGENTS.md topology/frank-medium/Model Config/Benchmarking Policy/runbook
index flipped; both indexes updated; work-log entry added.
Also repaired agent-pass damage: prose prepended above frontmatter openers,
duplicated runbook bodies, a literal tool-call token pasted into a runbook, and
an index destroyed by diff artifacts (restored from HEAD, flips re-applied).
Vault link check: 128 pages, 0 unresolved. ./tests/run.sh: 325 pass.
Reanalysis after the 2026-09-20 swap to an ASUS ProArt B850-Creator WiFi
NEO: both R9700s now sit on CPU GPP root ports (x8/x8 Gen5 bifurcation),
every bridge reads AtomicOpsCap Routing+, and both endpoints read
AtomicOpsCtl ReqEn+ (CAP_EXP+28.w = 0x1440) — the exact precondition
Dual-R9700-Cooperation's test plan named. TP=2 is therefore unblocked,
not proven: no collective has been dispatched since the swap,
GGML_HIP_RCCL is still OFF, and the 272 t/s "tensor" number on record
measured a butterfly fallback.
- Dual-R9700-Cooperation: short answer rewritten, re-measured 2026-09-20
topology table, rebuild-hypothesis door marked moot, probe plan is now
standing work.
- Dankserver System Specs: Motherboard section; GPU BDFs 0b:00.0 ->
07:00.0, iGPU 10:00.0 -> 11:00.0, kernel 7.0.0-31, SMT on, and the new
eno2 (192.168.0.247) second 5GbE port holding the low-metric default
route while eno1 keeps the fleet's static .235.
- frank-medium-llamacpp-dflash2: rationale and hardware row split past
from present; dated work-log sections stay as history.
- AGENTS.md + both indexes: no longer assert a wall that no longer exists.
Docs only; frank-medium stayed active and healthy throughout (verified
/health). ./tests/run.sh 331/331.
Frank
Comrade Chief Designer