Replace the FreeToken PR587 endpoint (single-GPU: its TP=2 dies in the
pynccl.cu ROCm JIT) with the radiance engine on both R9700s, TP=2 via
libr4d's fused one-shot all-reduce. Official Qwen/Qwen3.8-Flash-Next-FP8
payload at native block-FP8 precision:
- scripts/build-frank-primary-rad.sh: staging (quantization_config drop =
plugin SELECTION law) -> checkpoint reshape -> rad-convert with ONE
--quant rule (PLE rtn fixed-scale); plan-only first.
- scripts/stack-qwen38-official-fp8.py: the two value-preserving reshapes
(experts byte-stacked into the plugin's stacked layout; PLE's 128
scaleless fp8 shards decoded to one bf16 shard with the checkpoint's own
scale; the rtn rule re-encodes at that same scale - codes round-trip
exactly and land the one-element scale plane the plugin declares).
- scripts/hfget.sh + verified download (131/131 shards sha256-match).
- froggeric v22.5 chat template vendored (--override-chat-template).
- serve script (pinned ledger: 2x40 GiB pools + 47.7 GiB PLE + prefix host),
TTM preflight fixed for kernels with pages_limit only (no pages_max -
an absent knob is skipped, not read as zero); pages_limit raised to
52,428,800 pages = 200 GiB on the host pre-cutover.
- root unit, relinked; deployer gates image digest + .rad floor 158,000 MiB
+ health; prometheus re-adds the /metrics job.
- rollback: tag frank-primary-freetoken-pr587-20261005 (FreeToken trio +
Dockerfile); payload + image stay on disk until the window closes.
deploy-remote.sh also carries the parallel session's frank-small payload
gates (swift15) - committed here per Frank's direction to commit the tree.
DFlash2 experiment declared failed 2026-10-01: the drafter serves and
drafts (12.8 tok/s proposed) but acceptance is pinned at 1.0-1.38 on ALL
text classes including verbatim repetition — the aux-hidden-state capture
semantics are wrong (single-stream entering-state vs post-layer/comb is
the untested alternative). At ~1.2 acceptance the 8-token verify tax makes
decode 5x SLOWER than the 9.9-10.0 baseline (1.6-1.8 tok/s measured).
Restores the verified 37-40 tok/s shape: qwen3.8-flash-next FP8, MTP4,
prefetch off, 200k x 3 seqs, fp8 KV.
Payload: Intel/GLM-5.3-Flash-W4A16-AutoRound (AutoRound symmetric int4 group
128, auto_round:auto_gptq, 34 shards + 2 extras, 113,074 tensors, ~170 GiB).
It is the only W4A16 checkpoint this fork can serve with expert offload: its
quant_method "auto-round" is overridden to "inc", which routes the MoE to
libr4d's grouped W4A16 GEMM (R4dW4a16MoEMethod). A compressed-tensors W4A16
build of the same model resolves to TritonWNA16Experts, which has no offload
recipe, and refuses to serve at all (170 GiB of experts cannot fit 64 GiB of
VRAM, so offload is mandatory).
Shape: tcclaviger RDNA4 vLLM fork (same digest-pinned image as the 2026-09-29
qwen cutover), TP=2, 131,072 ctx x 2 seqs, bf16 KV (fp8 is ILLEGAL here — it
disables the sparse MLA backend's rope-free Triton path, the only MLA decode
gfx12 has), util 0.98, NBATCH 2048, --enable-expert-offload
--expert-precise-memory, prefix caching, glm47 tool + reasoning parsers,
MTP and expert prefetch absent by measurement.
Measured: decode 9.9-10.0 tok/s, prefill 918 tok/s (warm), arena 14.51 GiB =
2,383 slots/rank over 12,096 host-backed experts (73.64 GiB pinned/rank).
THE 35 tok/s TARGET IS NOT MET and the shortfall is measured: live telemetry
(R4D_LRU_TELEMETRY=1) gives a 0.53-0.66 per-layer hit rate, a step moves
0.82 GiB of misses in ~100 ms = 8.2 GB/s effective against a link measured at
22.74 GB/s — the same structural gather-path gap the qwen campaign found.
Growing the arena 13.08 -> 14.51 GiB changed decode by nothing.
Rejected by measurement: MTP4 (head loads, acceptance 3.76-4.90, but decode
unchanged and prefill collapses 918 -> 255 — a verification forward gathers the
union of the batch's experts), NBATCH 8192 (identical prefill).
Also: scripts/patches/gpu_worker-kv-cap.patch (fork-local fix for a KV-reserve
model that under-counts GLM's per-sequence KDA state; not needed by this shape),
runbook, AGENTS.md row + section, both indexes, work log.
Rollback: 2339fae (qwen FP8 + MTP4).
Replace the R9V radiance fork with the tcclaviger RDNA4 vLLM stack (the
digest-pinned image the 2026-09-29 tuning campaign measured), serving
orcarouter/Qwen3.8-Flash-Next-Uncensored-FP8 (131 shards, 172.8 GiB,
already on disk) with MTP4 spec decode.
Qualified by boot probe on 2026-09-29:
- MAXLEN=200000 x SEQS=3 (600k total) at --kv-cache-dtype fp8 (q8/q8 KV):
'GPU KV cache size: 729,545 tokens; Maximum concurrency for 200,000
tokens per request: 3.65x' at KV_BYTES 7.2 GiB PER RANK.
- KV_BYTES is per-rank: global 16G and 9G both OOM (worker tried to
allocate the full size; ~7.5 GiB/rank is the ceiling with the arena).
- prefetch OFF by law (telemetry: 36% precision, ~50% wasted traffic);
TunableOp off (corrupts numerics).
- decode at this shape: 42.94/34.55/36.09 tok/s; arena 6,032 slots.
Trio: frank-primary.env (all FRANK_PRIMARY_* keys), serve-frank-primary.sh
(direct podman run; entrypoint execs vllm serve; waits on the container),
frank-primary.service (root unit, same ExecStop law, new container name
frank-primary-qwen). Deployer gates rewritten (image digest + shard count
+ weight-map index). download-frank-primary.sh is now inventory/verify
(repo is GATED — 401 anonymous, masked LFS hashes; payload already on
disk). Seam rewritten for the new contract. Pre-cutover tag:
frank-primary-r9v-q8-20260929.
Fetches into /home/dank/models/Qwen3.8-Flash-Next-GGUF/ keeping the Hub
layout: the six Q8_0 split parts (175.30 GiB — never concatenated, part
1's header advertises file_count=6), the shared Q8_0 MTP drafter head
(2.6 GiB — "shared" borrows the target's embeddings; unsloth's MTP guide
names it the recommended head), and the BF16 vision projector. Total
191,919,144,448 bytes.
Sizes and sha256s captured from the resolve endpoint's x-linked-etag/
x-linked-size on 2026-09-26; every artifact is checked for exact byte
count AND sha256, curl -f so an auth error fails instead of writing the
error body, resumable via -C -. The repo is public: no token machinery.
Frank