frank-primary: 2 x 200704 fp8, pool cut for the expert arena, MNBT 1024

The 2026-10-01 bf16 shape (2 x 262144, --expert-precise-memory)
crash-looped at graph capture — the arena is DEVICE-resident and that
shape had 0.13 GiB to spare. The plain bf16 2 x 262144 also did not stay
up: 0 bytes free at runtime, and a real client request (995-token prompt,
max_tokens=261149) killed the worker with .

The correction is a pool cut, not a downgrade. FP8 KV (7,701 B/token),
two 200704 windows (401,408 tokens -> 3.091e9 B floor, pinned 3.15e9),
MNBT 1024. The fork's expert planner sizes the arena AFTER the KV
override, so the ~2.95 GiB/rank reclaimed becomes arena: ~7,500 slots
vs 6,248. Two contexts at 200704 is a capacity contract, not a
throughput one (2-way concurrency summed BELOW parity at 4064 ctx), so
the slot buys the VRAM headroom long generations need.

Refs [[dankserver-qwen38-fp8-expert-offload-tuning]].

- scripts/frank-primary.env: MAXLEN 200704, SEQS 2, KV_BYTES 3.15e9,
  MNBT 1024, rewritten shape comment
- scripts/serve-frank-primary.sh: header comment updated
- tests/frank_primary_unit.bats: assertions updated, new seam pins the
  arena-floor pool law (the pool must move with a context change or the
  reduction buys nothing)
- docs + AGENTS.md: fleet topology and deployed shape corrected

AGENTS.md

@@ -12,8 +12,8 @@ merge absorbed the former `vllmScripts` repo, history preserved).
   Calibre-Web, Jellyfin, Plex) plus the `frank-primary` LLM service.
 - **Dankclaw** (192.168.0.143) — AI/search/dev/observability stack (SearXNG
   via Gluetun, ComfyUI, OpenWebUI, Hermes, Gitea, Grafana, Prometheus, Loki,
-  Alloy, Pi-hole) plus TWO LLM services: `frank-large` (stock llama.cpp on
+  Alloy, Pi-hole) plus TWO LLM services: `frank-large` (ROCmFPX llama.cpp
-  the gfx1151 iGPU) and `frank-small` (llama.cpp on the RTX 3090). Caddy on
+  fork on the gfx1151 iGPU) and `frank-small` (llama.cpp CUDA on the RTX 3090). Caddy on
   Dankclaw reverse-proxies both hosts.
 - **FrankBook-Pro** (192.168.0.112) — macOS mini running the *arr stack
   (Prowlarr, Sonarr, Radarr, Lidarr, Readarr, qBittorrent) as quadlets inside
@@ -24,9 +24,9 @@ merge absorbed the former `vllmScripts` repo, history preserved).
 
 | Target | Service | Runtime | Model | Port |
 |--------|---------|---------|-------|------|
-| Dankserver | `frank-primary` | R9V radiance-vLLM fork `drwolfen/radiance-vllm-qwen4exp` (docker-pinned image bundle `2dac17a2`, docker→podman shim, root systemd unit) | unsloth Qwen3.8-Flash-Next-GGUF Q8_0 (175.30 GiB, 6 GGUF shards) with the blockfp8 MTP4 head (depth 4) and mmproj, TP=2 tiered experts (80-slot LRU cache; PLE n-gram PINNED in host RAM 54.4 GB; quantized inject-shard dev-overlay adapter), CED off, 131072 ctx single stream, Qwen-Sharp template, tool parser `qwen3_coder`, reasoning channel `reasoning` | 7770 |
+| Dankserver | `frank-primary` | tcclaviger RDNA4 vLLM fork (`docker.io/tcclaviger/vllm@sha256:da4029666…`, digest-pinned, plain `podman run`, root systemd unit) | `orcarouter/Qwen3.8-Flash-Next-Uncensored-FP8` (~172.8 GiB, 131 shards), TP=2, expert offload automatic (fork plans it), prefetch OFF, **fp8 KV** (`--kv-cache-dtype fp8`), **no speculative decode**, MNBT 1024, MoE gather 32×64, **2 seqs × 200704 ctx** at KV_BYTES 3,150,000,000/rank (fp8 measures 7,701 B/token), pool deliberately small so the expert arena keeps what it reclaims (~7,500 slots vs 6,248 at the 6.1e9 shape) — util 0.98, and the client contract wired: `--reasoning-parser …
-| Dankclaw | `frank-large` | unsloth llama.cpp b11030 prebuilt (ROCm gfx1151, root systemd unit) | unsloth Qwen3.8-27B-GGUF:UD-Q8_K_XL (27B dense, 31.5 GiB) with the z-lab DFlash2 Q4_K_M drafter (depth 4) on the gfx1151 iGPU (Strix Halo LPDDR, 114 GiB GTT), 262144 ctx single stream (1 slot), q8_0 KV, text only | 7770 |
+| Dankclaw | `frank-large` | **ROCmFPX llama.cpp fork** (`kingjones30/ROCmFPX` @ `36e9acd`, gfx1151, built by `scripts/build-rocmfpx.sh`; bare `llama-server`, root systemd unit) | `kingjones777/Qwen3.8-27B-ROCmFP4-STRIX-MTP-GGUF` **`Qwen3.8-27B-Q8_0_ROCMFPX_AGENT.gguf`** (26.28 GiB, ftype 115 ≈ 8.39 bpw: 353 F32 + 187 Q8_0 + 311 `GGML_TYPE_Q8_0_ROCMFPX` tensors — only this fork can read them), **MTP spec decode from the repo's own `mtp-Qwen3.8-27B-Q4_0.gguf` (n-max 4, pinned to ROCm0)**, **1 session × 262,144 (native window)** with **f16 KV**, the bundled BF16 mmproj for vision, thinking on with the card's recipe (temp 1.0 / top_p 0.95 / top_k 20). Measured 2026-10-02: decode **17.0–17.1 t/s** with MTP vs **7.6 without (+123 %)**, draft acceptance 0.462, prefill 325–351 t/s, prompt-cache replay 0.5 s, boot→healthy 16 s, GTT 64.5 GiB. Cut over 2026-10-02 from gufo; rollback tag `frank-large-gufo-q4kxl-20261001` | 7770 |
-| Dankclaw | `frank-small` | mainline llama.cpp master 37b53fd (root systemd unit) | HauhauCS Gemma4-12B-QAT-Uncensored-Balanced Q4_K_M (BF16 mmproj + MTP assistant sidecar, `draft-mtp,ngram-mod`) on the RTX 3090, 1048576 pool / 4 slots of 262,144, q8_0 KV, the fleet's fastest vision endpoint | 7771 |
+| Dankclaw | `frank-small` | llama.cpp master, CUDA sm_86 build (root systemd unit, relinked) | bartowski/MiMo-V2.6-Distill-Qwen-9B-GGUF **Q5_K_M** (9B dense qwen3_5 hybrid, 6.4 GiB GGUF, no MTP weights so no spec decode) + the repo's f16 mmproj for **vision**, on the RTX 3090, **3 × 180,224** (pool 540,672 — NOT the native window; three contexts cost 81,920 tokens per conversation) with **asymmetric KV f16 K + q8_0 V** under flash attention, 21,684 used / **2,441 free** of 24,576 MiB, 106.8 tok/s decode, 3,814 t/s prefill. Resized twice 2026-10-02 (Q6_K → Q5_K_M → 3 slots + vision → this pool); rollback `models/MiMo-Q6K/` or `PARALLEL=2 CTX_SIZE=524288` | 7771 |
 
 Retired endpoints (`frank-npu`, and every prior runtime) are in
 the [runbook index](#runbook-index). The full swap history lives in
@@ -208,13 +208,27 @@ archive.
 
 **STOP the production service BEFORE benchmarking** — these servers serve
 real users, and a bench beside a live service starves the GPU and corrupts
-results. Use the runtime's own harness: `llama-bench` for every llama.cpp
+results. Use the runtime's own harness: `llama-bench` for the llama.cpp
-endpoint (all three are llama.cpp now). Rolling your own bench
+endpoints; a live llama.cpp server also reports honest per-request numbers in the `/completion` `timings` object (prefill/decode rates, `draft_n`, `draft_n_accepted`), which is how frank-large and frank-small were measured. Rolling your own bench
 requires asking first.
 Scratch ports are not used on LLM hosts — a
 test process cannot share the single GPU anyway; validate against the service
 itself.
 
+(A gufo-era law; frank-large is ROCmFPX llama.cpp since 2026-10-02, but the rollback tag keeps it live.) **`gufo bench` cannot rank speculative-decode settings, and will invert the ranking.**
+Measured 2026-09-28 on frank-large at depth 32768: its corpus
+reports `acceptance=0.933 ± 0.000` with an **identical trace sha256** across
+draft widths, i.e. the same highly-predictable text every run, and 93 % of
+every proposal accepted. It reported 29–37 t/s decode where real traffic
+gives ~15 t/s, and it made `--draft-tokens 16` look 6 % faster than 7 when
+real text made it 14 % *slower*. Use it for shape, not for A/B. For an A/B,
+replay **real** text through the live server and read the `Server-Timing`
+header — but note the unit rides in the **name** half (`ttft;dur=402.485`),
+so splitting on `=` alone keys the dict `ttft;dur` and every lookup silently
+returns nothing. Generate **at least 128 tokens** for the measurement: at 8
+tokens, `inter_token` is first-token noise and the run-to-run spread swamps
+every real difference.
+
 This covers **any GPU experiment, not just benchmarks** — including a bare
 2-rank RCCL/`torch.distributed` probe. On 2026-09-16 one such probe was run
 against the live service "because it only needs 200 MB of VRAM"; the host went
@@ -227,11 +241,11 @@ one: `ip neigh show <ip>` from a peer — FAILED means off/NIC dead, while a
 merely wedged kernel usually still answers ARP.
 
 1. Stop and mask (self-healing would restart it):
-   - Dankclaw `frank-large` (root unit, stock llama.cpp on the iGPU):
+   - Dankclaw `frank-large` (root unit, vLLM podman on the iGPU):
      `sudo systemctl stop frank-large && sudo systemctl mask frank-large`
    - Dankclaw `frank-small` (root unit, llama.cpp on the 3090):
      `sudo systemctl stop frank-small && sudo systemctl mask frank-small`
-   - Dankserver `frank-primary` (root unit, R9V radiance on the 2x R9700):
+   - Dankserver `frank-primary` (root unit, tcclaviger vLLM on the 2x R9700):
      `sudo systemctl stop frank-primary && sudo systemctl mask frank-primary`
 2. Bench.
    NOTE: `systemctl mask` FAILS on a relinked unit (the unit file in
@@ -263,168 +277,181 @@ When renaming a service, retiring an endpoint, or moving a model: update the
 provider block, model `id`, `contextWindow`, `maxTokens`, `input` in
 `models.yml`, then fix every `modelRoles` entry that named the old endpoint —
 a role pointing at a provider the file does not define fails at call time. As
-of 2026-09-26 both `frank-small` and `frank-primary` accept images
+of 2026-10-02 **all three endpoints accept images** (`input: [text, image]`):
-(`input: [text, image]`) — frank-primary loads the BF16 mmproj explicitly
+`frank-large` through the BF16 mmproj that ships inside its GGUF payload,
-(this build never autoloads projectors); `frank-large` remains text-only
+`frank-primary` through a BF16 mmproj loaded explicitly (this build never
-because its stock llama.cpp deployment loads no
+autoloads projectors), and `frank-small` through
-vision projector (the mmproj is not loaded). Note also that
+`mmproj-MiMo-V2.6-Distill-Qwen-9B-f16.gguf` passed as `--mmproj` — the Q5_K_M
-the reasoning channel differs by runtime: llama.cpp endpoints emit
+GGUF itself carries no vision tensors. `frank-small`'s `contextWindow` is
-`reasoning_content`, vLLM emits `reasoning` — academic since the
+**180224**, the per-slot window, not the model's native 262144: the harness
-2026-09-26 swap, as every endpoint in the fleet is llama.cpp now.
+must compact before llama.cpp answers a longer prompt with a 400. Note also
+that
+the reasoning channel differs by runtime: llama.cpp AND SGLang (qwen3
+parser, live-verified) emit `reasoning_content`, vLLM emits `reasoning` —
+models.yml sets `reasoningContentField` per endpoint to match.
 There is **no audio**
-input anywhere in the fleet except frank-small: that capability left with the
+input anywhere in the fleet: that capability left with the FLM NPU
-FLM NPU endpoint on 2026-09-11 and returned on 2026-09-17 with the Gemma4
+endpoint on 2026-09-11, returned 2026-09-17 with the Gemma4 projector's
-projector's audio encoder (frank-small transcribes WAV verbatim even though
+audio encoder, and left again with the Gemma4 retirement on 2026-10-01
-`audio` is omitted from its `input` list — OMP's schema rejects it).
+(the MiMo tower is vision-only).
 
-### frank-large — stock llama.cpp Qwen3.8-27B UD-Q8_K_XL (Dankclaw, ACTIVE)
+### frank-large — ROCmFPX llama.cpp, Qwen3.8-27B Q8_0_ROCMFPX_AGENT + MTP (Dankclaw, ACTIVE)
 
-The unsloth llama.cpp prebuilt `app-b11030-mix-5ff778e` (ROCm gfx1151,
+The **kingjones30/ROCmFPX llama.cpp fork** at commit
-`/home/dank/GitPackages/unsloth-llama-b11030/`, bundles its own ROCm
+`36e9acd40e10a87cd3c3ef8ec734668757dc8520`, built for gfx1151 by
-runtime — no host SDK gate) as a **root systemd unit**
+`scripts/build-rocmfpx.sh` into `/home/dank/GitPackages/kingjones-ROCmFPX/build`,
-(`scripts/frank-large.service`, `User=dank`), serving
+as a **root systemd unit** (`scripts/frank-large.service`, `User=dank`,
-`unsloth/Qwen3.8-27B-GGUF:UD-Q8_K_XL` (27B dense, 31.5 GiB,
+`Restart=always`) running a bare `llama-server` (no container) on port 7770
-sha256-verified) on port 7770 from the gfx1151 iGPU. One stream, full
+from the Strix Halo iGPU. Boot to healthy: **16 s**.
-262,144 ctx (the model's native window, `PARALLEL_SLOTS=1`), DFlash2 spec
-decode (the z-lab `Qwen3.8-27B-DFlash2-Q4_K_M` drafter, depth 4,
-`SPEC_DRAFT_N_MAX=4`), reasoning extracted to `reasoning_content`, and
-**no vision projector**. It replaced the Kairic Edge ROCmFPX build (jcbtc
-Qwen3.8-27B-IU4-Kairic-Edge), which had replaced the unsloth
-Qwen3.8-Flash-Next UD-IQ4_XS.
 
-The spec method is DFlash2 (block-diffusion drafter) at depth 4. The
+**The runtime is not interchangeable — it is chosen by the FILE.**
-drafter is loaded via `--model-draft`; the chain is `--spec-type
+`kingjones777/Qwen3.8-27B-ROCmFP4-STRIX-MTP-GGUF`'s best rung,
-draft-dflash` with `--spec-draft-device ROCm0`. KV is **q8_0** (the
+`Qwen3.8-27B-Q8_0_ROCMFPX_AGENT.gguf` (28,214,425,952 B, sha256 `f3ef2bda…`,
-unsloth b11030 build does not support fp8 KV; q8_0 is the
+ftype 115 ≈ 8.39 bpw), stores **311 tensors of type 103,
-highest-precision 8-bit option, ~8 GiB at 262k). The serve script
+`GGML_TYPE_Q8_0_ROCMFPX`** — a quantization that exists in no other loader:
-(`scripts/serve-frank-large.sh`) validates binary → main GGUF → drafter
+stock llama.cpp and gufo both refuse it at load (`unknown tensor type`). The
-and exits with a pointer message if any is missing, then `exec`s
+file is 353 F32 + 187 plain-Q8_0 + 311 ROCmFPX; the AGENT routing variant
-`llama-server` with `--jinja --alias frank-large --chat-template-kwargs
+keeps the output-head tensors at plain Q8_0 (187 vs 2 in the non-AGENT file),
-'{"preserve_thinking":true}' --chat-template-file $TEMPLATE_FILE` (the
+which is why the card reports MTP acceptance 0.953 vs 0.911 — highest quality
-Qwen-Sharp v22.4.0 template; without `--chat-template-file` the binary
+AND the better drafter base. Vision comes from the bundled
-falls back to the built-in `peg-native` template, which produces a
+`mmproj-Qwen3.8-27B-BF16.gguf` (931,145,888 B).
-~73,600-token prompt for a ~250-token request).
 
-The iGPU addresses **114 GiB** (the GTT aperture over the shared LPDDR,
+**The pin is load-bearing.** The 2026-08-25 ROCmFPX builds segfaulted in
-`gtt_total` in `/sys/class/drm/card1/device/`; `rocm-smi`'s 2 GiB "dedicated"
+`ggml_compute_forward_set_rows` on exactly this file; `build-rocmfpx.sh`
-is not the budget). The 27B dense model (31.5 GiB) is mmap'd in with
+checks the commit out by SHA, so a fork-side force-push cannot silently
-`--n-gpu-layers 999`; budget at 262k ≈ 41-43 GiB (weights + KV + buffers),
+re-break the endpoint.
-leaving ~71 GiB of aperture headroom. The 8 GiB prompt-cache budget
-(`CACHE_RAM=8192`) is the qualified config.
 
-Fetch: `scripts/download-frank-large.sh` (per-file, each file verified
+**Spec decode is the performance feature, and it is measured.** The draft is
-against its declared byte count + sha256; `--verify-only` re-checks).
+the repo's own `mtp-Qwen3.8-27B-Q4_0.gguf` (1,680,271,648 B, sha256 `051a1764…`,
+`nextn_predict_layers = 1`), pinned to the same card:
+`--spec-type draft-mtp --spec-draft-n-max 4 --spec-draft-n-min 0
+--spec-draft-p-min 0.0 --spec-draft-device ROCm0 --spec-draft-ngl 99`.
+Measured 2026-10-02 in one session, temp 0, 2,202-token prompt / 512 tokens,
+`cache_prompt: false`, production service stopped for each arm:
+**MTP on 17.00–17.14 t/s decode** (draft acceptance 0.462, mean acceptance
+length ≈ 1.86) against **7.60–7.66 t/s with the draft removed — +123 %**;
+prefill pays 8–15 % (381 → 325–351 t/s). Acceptance is far from the card's
+0.95 (that number is the card's own text at a shallow prompt); the gain is
+real and n-max 4 is where it was measured, never swept. This replaces the
+gufo Q4_K_XL arm at ~15 t/s, so the endpoint got higher quality AND more
+speed at once.
 
-Runbook: `serverAppConfigs-docs/LLM/frank-large-unsloth-qwen38-27b-q8xl.md`.
+**Shape and budget:** ONE session at the native 262,144 window, unquantized
-Rollback: [[frank-large-kairic-edge-qwen38-27b]] (payload and runtime
+f16 KV (~64 KiB/token → ~16 GiB, affordable inside the iGPU's ~114 GiB GTT
-intact on disk).
+aperture), `--fit off --flash-attn on --direct-io --ctx-checkpoints 16
+--cache-ram 8192 --cache-idle-slots`. Measured GTT 64.5 GiB with model +
+draft + projector resident. Prompt cache verified: a repeat pass over the
+same 6k prompt replays in 0.5 s.
+
+**Client contract (llama.cpp, live-verified 2026-10-02):** `--jinja` is on
+(without it `chat_template_kwargs` are silently ignored), tools are native
+(`get_weather` → `{"city":"Seattle"}`), thinking lands in
+**`reasoning_content`**, `reasoning_effort` accepts ONLY
+`xhigh|medium|low` (anything else is HTTP 500 — `models.yml` folds `high` and
+`max` to `xhigh`), and thinking is enabled server-side with the card's
+recipe (temp 1.0 / top_p 0.95 / top_k 20 / min_p 0) plus
+`--reasoning-format deepseek --reasoning-budget -1`. Two traps: a request
+whose whole budget is consumed by reasoning returns `content: null` (bench
+clients must fall back to `reasoning_content` or they crash *after*
+measuring), and the fork logs
+`MTP boundary missing for seq_id=… ; resyncing` under concurrent draft
+resets — self-recovering, not a fault.
+
+Fetch: `scripts/download-frank-large-kingjones.sh` (`-f` + byte count + LFS
+sha256 verified, `--verify-only`, no token on the command line).
+
+Runbook: `serverAppConfigs-docs/LLM/frank-large-rocmfpx-q8agent-mtp.md`.
+Rollback: git tag **`frank-large-gufo-q4kxl-20261001`** — gufo v0.1.1
+(`scripts/build-gufo.sh` stays on disk for it), the Q4_K_XL Qwen3.8-27B
+payload and the DFlash2 drafter all remain on the host; restore the tag's
+unit/env/serve trio and redeploy.
 
 ### frank-small — llama.cpp on the RTX 3090 (Dankclaw, ACTIVE)
 
-Mainline llama.cpp master `37b53fd` (CUDA build with
+**llama.cpp master** (CUDA sm_86 build
-`-DGGML_CUDA_FA_ALL_QUANTS=ON`) as a **root systemd unit** (repo
+`/home/dank/GitPackages/llama-master/build-cuda/bin/llama-server`) as a
-`scripts/frank-small.service`, `relink_unit`ed into `/etc/systemd/system/`),
+**root systemd unit** (`scripts/frank-small.service`, `relink_unit`ed into
-serving `HauhauCS/Gemma4-12B-QAT-Uncensored-HauhauCS-Balanced` at Q4_K_M.
+`/etc/systemd/system/`), serving
-The MTP head is a SEPARATE assistant file (`mtp-gemma-4-12B-it.gguf`, 4
+`bartowski/MiMo-V2.6-Distill-Qwen-9B-GGUF` at **Q5_K_M** (6,876,124,704 B,
-nextn layers) loaded via `--model-draft`; the draftless **ngram-mod**
+sha256 `04cf4f…`) **plus `mmproj-MiMo-V2.6-Distill-Qwen-9B-f16.gguf`**
-matcher chains behind it in the same `--spec-type` value. BF16 mmproj,
+(918,166,048 B, sha256 `ff348f…`), both `/home/dank/models/MiMo-Q5KM/`, on
-port 7771, q8_0 KV, `CTX_SIZE=1048576` / `PARALLEL=4`
+port 7771. The 2026-10-02 swap was a **quant rung down from Q6_K**, not a
-(four FULL 262,144-token slots — the measured card ceiling), batch
+runtime change — the llama.cpp path arrived 2026-10-01 night for one reason:
-**8192/1024**, greedy default. Deployed flags are in
+**asymmetric KV**. `--cache-type-k f16` + `--cache-type-v q8_0` is the fleet's
-`scripts/frank-small.env`; the serve script refuses to start if the
+only per-tensor KV lever; SGLang's `--kv-cache-dtype` moves K and V together.
-weights, the projector, the sidecar or the binary are missing.
+Vision came the same day via `--mmproj`: the GGUF carries no `v.*` tensors, so
+the projector file is the entire vision path (and ~1.4 GiB of the card).
 
-Verified at deploy (2026-09-17): `/v1/models` reports `frank-small`; a
+**Architecture: qwen3_5 hybrid** — 32 layers, 8 full-attention + 24 Gated
-chat completion returns content plus `reasoning_content`; a tool call
+DeltaNet, native 262,144 window. KV exists only in the 8 attention layers
-returns `tool_calls`; an image request is answered from the projector;
+(4 KV heads × 256 dim × 2 B ≈ 2 KiB/token/layer, ~16 KiB/token for f16 K+V);
-an `input_audio` WAV transcribes verbatim. Post-tuning live numbers at
+the DeltaNet state is a fixed per-slot buffer, so slot count — not context —
-max length: pp@254k ~965 t/s, tg@253k 226–302 t/s with ~94% draft
+drives that term.
-acceptance; idle 23,300 MiB of 24,576. The throughput sweep (and why
-ubatch 2048 OOMs the draft's pp buffer, and why draft depth 4 + poll
-100 is WORSE at long ctx) is tabled in the runbook.
-Runbook: `serverAppConfigs-docs/LLM/frank-small-gemma4-12b.md`.
-Rollback: [[frank-small-qwen38-gsq-rco]].
 
-### frank-primary — R9V radiance-vLLM Qwen3.8-Flash-Next Q8_0 (Dankserver, ACTIVE)
+**The shape:** `--ctx-size 540672 --parallel 3` = **THREE conversations of
+180,224 tokens each** (llama.cpp divides the pool:
+`n_ctx_slot = CTX_SIZE / PARALLEL`), `--flash-attn on` (the q8_0-V path is
+only cheap under flash attention — the seam pins it), f16 K + q8_0 V, all
+layers on the GPU, `--mmproj` for vision. Measured VRAM **21,684 used / 2,441
+free** of 24,576 MiB — the 2–3 GiB headroom Frank asked for, landed by
+solving the pool against two measured boots: non-KV cost is 7,390 MiB without
+a projector (2 × 262,144 boot) and 8,859 MiB with one (3 × 196,608 boot), so
+vision costs ~1.4 GiB, not the 0.9 GiB the file size suggests. Three native
+windows (786,432 tokens) need 18.4 GiB of KV alone and were never on the
+table. Full f16 V would add ~4 GiB more and bust the card — that is the
+justification for the asymmetric contract.
 
-The **R9V radiance-vLLM fork** (`drwolfen/radiance-vllm-qwen4exp`, docker-pinned
+Verified at deploy (2026-10-02): `/v1/models` reports `frank-small`; boot logs
-prebuilt image bundle `r9v-qwen38-flash-next-wmma@sha256:2dac17a2…`) serving
+`n_slots = 3, n_ctx_slot = 180224` and `loaded multimodal model`; decode
-the **unsloth Qwen3.8-Flash-Next-GGUF Q8_0** (175.30 GiB, 6 GGUF shards)
+**106.8 tok/s** at temp 0 (prefill 3,814 t/s) — ~5 % faster than the
-with the blockfp8 MTP4 head (depth 4) and the mmproj on **both** Radeon AI
+2 × 262,144 shape, because a shorter slot means a shorter attention scan;
-PRO R9700s (gfx1201, BDFs `0000:03:00.0` / `0000:07:00.0`), one stream at
+three concurrent requests answered in parallel and a fourth queued behind a
-**131,072 ctx**, **TP=2 tiered experts** (80-slot LRU expert cache; PLE
+released slot; native `tool_calls`; thinking in `reasoning_content`; a
-n-gram table PINNED in host RAM, 54.4 GB), the **Qwen-Sharp v22.4.0
+solid-red PNG answers "Red." through an `image_url` content part (the legacy
-template** (via the launcher's `R9V_CHAT_TEMPLATE_FILE` passthrough, fork
+top-level `images` field is ignored by `/v1/chat/completions` — the model then
-`30d4a32`), tool parser `qwen3_coder`, reasoning channel `reasoning`. Cut
+invents a color). A prompt over the slot is a **400
-over 2026-09-27 from the uncensored IQ4_XS payload (itself cut over from
+`exceed_context_size_error`**, which is why `models.yml` sets
-llama.cpp b11030 the same day) at Frank's direction.
+`contextWindow: 180224`.
 
-**The tuned shape lives on-host, not in the repo.** The q8 profile's setup
+Runbook: `serverAppConfigs-docs/LLM/frank-small-llama-mimo-q5km.md`.
-run qualified the `R9V_*` config into
+Rollback: [[frank-small-llama-mimo-q6k]] (Q6_K file on disk at
-`/home/dank/r9v-state/q8xl/setup.json` (`ready: true`); the repo env
+`/home/dank/models/MiMo-Q6K/`); before that [[frank-small-sglang-mimo-v26]]
-file `scripts/frank-primary.env` carries ONLY production overrides
+(venv on disk at `/home/dank/GitPackages/sglang-cu128/`; its INT8 payload was
-(`R9V_HOST_BIND=0.0.0.0`, `R9V_HOST_PORT=7770`,
+staged to `~dank/.AGENTTRASH/20261002-retired-llm-payloads/` on 2026-10-02).
-`R9V_SERVED_MODEL_NAME=frank-primary`, `R9V_CONTAINER_NAME=frank-primary-r9v`,
-`R9V_EXPECTED_GPU_BDFS`) plus two q8-only keys: the dev-overlay adapter
-(`R9V_DEV_QWEN4_EXP_ADAPTER_PY`, fork commit `363d606` — the Q8_0 payload
-quantizes BOTH fused hyper-connection shards, so the shard-uniformity
-check needs the in-stream dequant) and `R9V_FULL_MUTABLE_CACHE=0` (the
-overlay's env.always injects the uncensored 160-slot cache geometry, which
-the q8 80-slot tiered cache rejects). Never copy tuned values from the
-state into the repo — it would drift from the qualification.
 
-**Spec decoding is MTP4 only.** The fork's `speculative_config` is a
+### frank-primary — tcclaviger vLLM, Qwen3.8-Flash-Next FP8 + MTP4 (Dankserver, ACTIVE)
-single dict — one method per engine, NO chaining — so the llama.cpp
-`draft-mtp,ngram-mod` pattern is NOT expressible here. MTP4 is wired
-(`R9V_MTP_SPEC_TOKENS=4`, draft TP=2); the PLE n-gram rides along
-architecturally (`ngram_embedding`); `ngram_gpu` exists as an alternative
-single method, not a chainable one.
 
-**Serving = the fork's own launcher through a docker→podman shim.** The R9V
+The **tcclaviger RDNA4 vLLM fork** (`docker.io/tcclaviger/vllm@sha256:da4029666…`,
-tooling is docker-pinned; the shim at
+digest-pinned) serving **`orcarouter/Qwen3.8-Flash-Next-Uncensored-FP8`**
-`/home/dank/GitPackages/radiance-vllm-qwen4exp/.podman-shim/docker` carries
+(~172.8 GiB, 131 shards) on **both** Radeon AI PRO R9700s (gfx1201), **TP=2**,
-three laws: (1) the bundle is a containerd/OCI docker-save, so docker keys
+**MTP4** speculative decoding, **expert prefetch OFF**, 200,000 ctx x 3 seqs,
-the image by its OCI manifest digest while podman keys by config digest but
+**fp8 KV**. Deployed 2026-10-01 by rollback from commit `2339fae` after the
-preserves the manifest digest in `RepoDigests` — the shim answers
+GLM/DFlash2 experiment was declared failed (same day).
-`image inspect sha256:X` from RepoDigests and rewrites bare `sha256:X` run
-refs to `repo@sha256:X`; (2) podman inspect answers
-`HostConfig.LogConfig.Config: null` where docker returns the log-opts map
-(the fork's doctor crashes on it) — the shim records `run --log-opt` pairs
-and normalizes inspect output; (3) podman lacks docker's `SecurityOptions`
-/`ServerVersion` info fields — the shim answers them.
 
-**Unit and self-healing.** Root systemd unit (`scripts/frank-primary.service`,
+Measured 2026-09-29 tuning campaign: **decode 37.4-40.6 tok/s, acceptance
-`User=dank`, `LimitMEMLOCK=infinity`) — the PLE pinned residency alone is
+~54 %** (MTP4 + prefetch off — the fleet's only endpoint that meets the
-54.4 GB, and a root unit grants the pin budget via PID1 (a rootless
+35 tok/s target). 256k ctx arm measured 34.1-36.5; 2-way concurrency sums
-quadlet would be capped by the user manager's limit).
+BELOW parity (25.49 vs 26.8 single) — slots are a capacity feature, not a
-The serve script (`scripts/serve-frank-primary.sh`) gates the setup state
+throughput one. V-only q8 KV is NOT possible via config: stock
-(ready + paths), the pinned image (RepoDigests), the PLE byte count
+`--kv-cache-dtype` is K+V together; a V-only quant needs a fork patch.
-(`stat -c %s`), every GGUF shard + the mmproj and the GPU device nodes,
-clears a stale container, then execs the fork's `scripts/launch.sh` through
-the shim and **blocks on `docker wait`** — the launcher `--detach`es, so
-without the wait a crashed engine is a green unit; `docker wait` cannot stop
-the container child, hence `ExecStop=podman stop -t 120
-frank-primary-r9v`. The deployer's `deploy_llm_primary` gates state/image/
-PLE/shards, `podman rm -f`s the container, waits out :7770, relinks/enables/
-restarts the unit and health-gates with the MainPID check (the false-green
-stray law).
 
-Verified at cutover (2026-09-27): q8 setup state `ready: true`, all 20
+The engine auto-sets the attention block size (mamba-page floor); the
-payload artifacts sha256-verified (175.30 GiB); the scratch-port bring-up
+expert plan's registry prices KV per token; the expert arena (6,574 slots
-battery answered vision ("Red" for a 64×64 red PNG), MTP4 mean acceptance
+x 2.344 MiB = 15.05 GiB/rank) is the decode-critical region — cutting it
-length 3.19–5.00, and the PLE pinned proof (`mapping_bytes =
+for KV headroom collapses the offload hit rate.
-mapping_rss_bytes = 54,400,261,120`, fully RSS-resident); wire + tool
-round-trip + template probe re-verified on :7770 after the deploy. No
-throughput bench was run (Frank declined); the probe-era 11 tok/s decode
-was residency-bound (halved expert headroom at double the bytes/expert),
-not the production number.
 
-Runbook: `serverAppConfigs-docs/LLM/frank-primary-r9v-radiance-qwen38-q8.md`.
+Unit and self-healing: root systemd unit (`scripts/frank-primary.service`,
-Rollback: the uncensored payload — flip `R9V_STATE_FILE` back to
+`User=dank`, `LimitMEMLOCK=infinity`), `ExecStop=podman stop -t 120
-`/home/dank/r9v-state/uncensored/setup.json`, drop the two q8-only keys
+frank-primary-glm`. The serve script drives `podman run` directly, gates
-and the template keys, redeploy (state + `restore-config-20260927.env` +
+the digest-pinned image, the payload's `config.json` and its exact
-payload intact on disk) — see
+131-shard count and the GPU device nodes, clears stale container
-[[frank-primary-r9v-radiance-qwen38-uncensored]].
+identities, then blocks on `podman wait`. `deploy_llm_primary` gates the
+same, waits out :7770, relinks/enables/restarts the unit and health-gates
+with the MainPID check. Boot cost ~7 min warm; health returns 503 while
+loading.
+
+Runbook: `serverAppConfigs-docs/LLM/dankserver-qwen38-fp8-expert-offload-tuning.md`.
+Retired GLM experiment: `serverAppConfigs-docs/LLM/frank-primary-glm53-flash-w4a16.md`.
 
 ## Key Lessons Learned
 Durable operational lessons from the fleet's history. Full detail in the
@@ -599,6 +626,26 @@ linked runbooks.
   non-greedy requests and reasoning budgets. Re-test chat before production.
 - **FastMTP sidecar**: build must be commit `4df29be4` + the HauhauCS patch;
   stock binaries fail the draft load with a tensor-shape mismatch.
+- **A GGUF's custom tensor type picks the runtime — read the histogram before
+  promising a file to a loader.** `Qwen3.8-27B-Q8_0_ROCMFPX_AGENT.gguf` is
+  ftype 115 with 311 tensors of type 103 (`GGML_TYPE_Q8_0_ROCMFPX`); stock
+  llama.cpp and gufo refuse it at load, so ROCmFPX was not a preference but
+  the only door. Counting tensor types out of the GGUF KV section takes one
+  python read and settles a runtime argument in minutes. The build identity
+  matters as much as the fork: the 2026-08-25 ROCmFPX builds SIGSEGV in
+  `ggml_compute_forward_set_rows` on this same file, so
+  `scripts/build-rocmfpx.sh` pins commit `36e9acd` by SHA.
+- **Speculative-decode flags move as a SET.** Handing the ROCmFPX fork
+  `--model-draft` with no `--spec-type` (the naive way to build a "no MTP"
+  bench arm) segfaulted at load, core dumped, no diagnostic (2026-10-02). A
+  baseline arm must strip `--model-draft` AND every `--spec-draft-*` AND
+  `--spec-type`, or it measures a crash instead of the absence of drafting.
+- **An orphaned scratch server taxes the whole host.** A bench child left
+  alive by a client-side exception kept ~26 GiB of weights resident on a
+  121 GiB box, pushed `available` to 3 GiB, and every later rate was suspect.
+  Record the child pid and kill it in a `finally`, then prove `free` came back
+  before believing a number. Kill by pid — never `pkill -f llama-server`,
+  which also hits frank-small.
 
 **llama.cpp serving**
 
@@ -671,6 +718,35 @@ linked runbooks.
   host-RAM hash table) and its window decides which traffic it can see: too
   short and it proposes on ordinary prose fragments, which costs verification
   without acceptance.
+- **`--parallel` divides the pool, so MORE conversations cost WINDOW, and the
+  per-slot number is the contract — not `ctx-size`.** `n_ctx_slot =
+  CTX_SIZE / PARALLEL`, and llama.cpp refuses a prompt longer than the SLOT
+  (`400 exceed_context_size_error: request (184001 tokens) exceeds the
+  available context size (180224 tokens)`) even when the pool has room. So
+  frank-small's 2 → 3 reslice moved each conversation from the model's native
+  262,144 down to 180,224, and `models.yml` `contextWindow` had to follow or
+  the harness would discover the ceiling as a 400 on a long conversation. Side
+  effect worth knowing: the shorter slot made decode ~5 % FASTER (106.8 vs
+  100.4 tok/s) — less attention to scan and one fewer DeltaNet state resident.
+  Measured 2026-10-02. See [[frank-small-llama-mimo-q5km]].
+- **Price a VRAM budget from two boots, never from a formula — and an mmproj
+  costs ~1.5x its file size.** The frank-small 3090 fit was solved from two
+  measurements: 2 × 262,144 with no projector = 20,190 MiB used (KV 12,800 ⇒
+  **non-KV 7,390**), 3 × 196,608 with the f16 projector = 22,972 used (KV
+  14,113 ⇒ **non-KV 8,859**). The difference is the vision tower: a 918 MB file
+  that bills ~1.4 GiB, because the encoder's activation and compute buffers
+  ride with it. Subtracting the two rows is exact; extrapolating weights+KV+
+  "a bit for buffers" is what leaves 1.1 GiB free when you meant 2.7. Then
+  solve `KV ≤ card − non-KV − headroom` and round to a multiple of
+  `PARALLEL × 4096`. Recorded in `scripts/frank-small.env` so the next resize
+  starts from measurement.
+- **`images` and `image_url` are NOT interchangeable: only the content part
+  reaches the model.** `--mmproj` loads, `usage.prompt_tokens` rises, and the
+  model still invents an answer if the client used the legacy top-level
+  `images` field on `/v1/chat/completions` — frank-small answered "Brown." for
+  a solid-red PNG that way and "Red." through the OpenAI `image_url` part. The
+  failure is silent because a hallucinated color is a plausible answer. Probe
+  vision with a solid color and expect an exact word.
 - **`--jinja` is what `tools` needs.** On 2026-09-16 a subagent died with
   `500 tools param requires --jinja flag`, from the ik_llama.cpp runtime that
   then served frank-large: every tool-using agent on that endpoint was broken
@@ -771,9 +847,9 @@ Active:
 
 | Service | Runbook |
 |---------|---------|
-| frank-large (stock llama.cpp Qwen3.8-27B UD-Q8_K_XL, Dankclaw) | `serverAppConfigs-docs/LLM/frank-large-unsloth-qwen38-27b-q8xl.md` |
+| frank-large (ROCmFPX llama.cpp fork, Qwen3.8-27B Q8_0_ROCMFPX_AGENT + MTP4 + vision, Dankclaw) | `serverAppConfigs-docs/LLM/frank-large-rocmfpx-q8agent-mtp.md` |
-| frank-small (llama.cpp Gemma4-12B QAT, Dankclaw) | `serverAppConfigs-docs/LLM/frank-small-gemma4-12b.md` |
+| frank-small (llama.cpp CUDA, MiMo-V2.6-Distill-Qwen-9B Q5_K_M, Dankclaw) | `serverAppConfigs-docs/LLM/frank-small-llama-mimo-q5km.md` |
-| frank-primary (R9V radiance-vLLM Qwen3.8-Flash-Next Q8_0, Dankserver) | `serverAppConfigs-docs/LLM/frank-primary-r9v-radiance-qwen38-q8.md` |
+| frank-primary (tcclaviger vLLM, Qwen3.8-Flash-Next FP8 + MTP4, Dankserver) | `serverAppConfigs-docs/LLM/dankserver-qwen38-fp8-expert-offload-tuning.md` |
 | Dankclaw kernel posture (`amd_iommu=off`) | `serverAppConfigs-docs/LLM/dankclaw-iommu-translation-off.md` |
 | Deployment flow + incidents | `serverAppConfigs-docs/LLM/deployment.md` |
 
@@ -781,6 +857,12 @@ Retired (rollback targets and history):
 
 | Runtime | Host | Window | Runbook |
 |---------|------|--------|---------|
+| gufo v0.1.1 Qwen3.8-27B UD-Q8_K_XL/UD-Q4_K_XL + DFlash2 (`frank-large`) | Dankclaw | 2026-09-28 → 10-01, again 10-01 → 10-02 (rollback target) | `frank-large-gufo-qwen38-27b.md` |
+| vLLM TheRock MiMo-V2.6-Distill-Qwen-9B BF16 + cross-model MTP (`frank-large`) | Dankclaw | 2026-10-01 (same-day rollback to gufo) | `frank-large-vllm-mimo-v26-bf16.md` |
+| llama.cpp master MiMo-V2.6-Distill-Qwen-9B Q6_K (`frank-small`) | Dankclaw | 2026-10-01 → 10-02 (rollback target) | `frank-small-llama-mimo-q6k.md` |
+| SGLang MiMo-V2.6 INT8-W8A16-AutoRound (`frank-small`) | Dankclaw | 2026-10-01 (same-day swap to the GGUF path) | `frank-small-sglang-mimo-v26.md` |
+|tcclaviger vLLM GLM-5.3-Flash W4A16 int4, +/- DFlash2 drafter (`frank-primary`)|Dankserver|2026-09-30 → 10-01|`frank-primary-glm53-flash-w4a16.md`|
+| llama.cpp b11030 Qwen3.8-27B UD-Q8_K_XL + DFlash2 (`frank-large`) | Dankclaw | 2026-09-21 → 09-28 | `frank-large-unsloth-qwen38-27b-q8xl.md` |
 | R9V radiance-vLLM Qwen3.8-Flash-Next-Uncensored IQ4_XS (`frank-primary`) | Dankserver | 2026-09-27 (same-day swap to Q8_0) | `frank-primary-r9v-radiance-qwen38-uncensored.md` |
 | llama.cpp b11030 Qwen3.8-Flash-Next Q8_0 + shared MTP (`frank-primary`) | Dankserver | 2026-09-26 → 09-27 | `frank-primary-llamacpp-qwen38-flash-next-q8.md` |
 | ik_llama.cpp Qwen3.8-Flash-Next-Uncensored i1-Q4_K_M (`frank-large`) | Dankclaw | 2026-09-15 → 09-16 | `frank-large-ik-uncensored.md` |
@@ -793,6 +875,7 @@ Retired (rollback targets and history):
 | Kairic Edge Qwen3.8-27B IU4 W4A4 (`frank-large`) | Dankclaw | 2026-09-21 (same-day swap) | `frank-large-kairic-edge-qwen38-27b.md` |
 | ISTA-DASLab Qwen3.8-27B-GSQ-RCO IQ3_S (`frank-small`) | Dankclaw | 2026-09-16 → 09-17 | `frank-small-qwen38-gsq-rco.md` |
 | vLLM Gemma4-12B w4a16 (`frank-small`) | Dankclaw | 2026-09-19 (same-day revert) | `frank-small-vllm-gemma4.md` |
+| llama.cpp Gemma4-12B QAT Q4_K_M (`frank-small`) | Dankclaw | 2026-09-17 → 10-01 | `frank-small-gemma4-12b.md` |
 | agentionai ROCmFP4-FAST imatrix | Dankclaw | 2026-08-29 → 08-30 | `agentionai-fast-imatrix.md` |
 | eemin Qwen3.8-27B ROCmI4 W4A4 MTP | Dankclaw | 2026-08-18 → 08-25 | `eemin-rocmi4-mtp.md` |
 | Kairic Edge IU4 | Dankclaw | 2026-08-22 (same-day revert) | `kairic-edge.md` |

README.md

@@ -13,7 +13,7 @@ one deploy script.
 | Host | User | IP | Podman stack | LLM service | LLM port |
 |------|------|----|--------------|-------------|----------|
 | dankserver | dank | 192.168.0.235 | media and library | frank-medium (ROCmFPX llama.cpp, Qwen3.8-27B AGENT) | 7770 |
-| dankclaw | dank | 192.168.0.143 | AI, search, development, observability | frank-large (CIRU ROCmFPX, Qwen3.8-27B) | 7770 |
+| dankclaw | dank | 192.168.0.143 | AI, search, development, observability | frank-large (gufo, Qwen3.8-27B UD-Q8_K_XL) | 7770 |
 
 ## Active Services
 

autoresearch.sh

@@ -0,0 +1,104 @@
+#!/usr/bin/env bash
+# autoresearch.sh — canonical benchmark entrypoint: speed of frank-primary.
+#
+# Runs on the operator box and drives one bench window on DANKSERVER:
+#   1. fingerprint the candidate trio (env + serve script + unit)
+#   2. memo fast-path: an identical fingerprint that already produced a FULLY
+#      successful window (bench + verified production restore) returns the
+#      cached metrics without touching the fleet
+#   3. stage the candidate trio plus the measure client into
+#      /home/dank/.autoresearch-frank-primary/ (ephemeral; the repo checkout
+#      on the host stays the production source of truth)
+#   4. run the window (scripts/autoresearch/frank-primary-window.sh):
+#      stop + mask the production unit, boot the staged config AS THE SERVICE
+#      (same serve script, same container, same port 7770), wait for health,
+#      run the in-container usage-counter measure, then restore the production
+#      unit and verify its health — AGENTS.md Benchmarking Policy
+#   5. print METRIC lines (parse by run_experiment)
+#
+# Primary metric: decode_tps (HIGHER is better) — median across the three
+# prompt depths (115/1056/4096) of per-depth medians of
+# completion_tokens / (wall - ttft), 3 runs each after a warm pass,
+# temperature 0, the campaign's deterministic word-seed workload.
+#
+# Exit 0 with METRIC lines on success; non-zero on any failure (the window
+# script's EXIT trap restores production on every path).
+set -euo pipefail
+
+SCRIPT_DIR="$(cd -- "$(dirname -- "${BASH_SOURCE[0]}")" && pwd)"
+HOST="${FRANK_PRIMARY_AR_HOST:-dankserver}"
+STAGE="/home/dank/.autoresearch-frank-primary"
+
+ENV_FILE="$SCRIPT_DIR/scripts/frank-primary.env"
+SERVE="$SCRIPT_DIR/scripts/serve-frank-primary.sh"
+UNIT="$SCRIPT_DIR/scripts/frank-primary.service"
+MEASURE="$SCRIPT_DIR/scripts/autoresearch/frank-primary-measure.py"
+WINDOW="$SCRIPT_DIR/scripts/autoresearch/frank-primary-window.sh"
+
+for f in "$ENV_FILE" "$SERVE" "$UNIT" "$MEASURE" "$WINDOW"; do
+    [ -f "$f" ] || { echo "[autoresearch] missing harness file: $f" >&2; exit 1; }
+done
+command -v python3 >/dev/null 2>&1 || { echo "[autoresearch] python3 required on operator box" >&2; exit 1; }
+
+# Fingerprint of everything that can change the engine's argv or environment.
+HASH="$(cat "$ENV_FILE" "$SERVE" "$UNIT" | sha256sum | cut -d' ' -f1)"
+echo "[autoresearch] candidate fingerprint $HASH" >&2
+
+ssh_run() { ssh -o BatchMode=yes -o ConnectTimeout=15 "$HOST" "$@"; }
+
+emit_metrics() { # $1 = run.json
+    python3 -c '
+import json, sys
+run = json.loads(sys.argv[1])
+m = run["measure"]
+print("METRIC decode_tps=%s" % m["decode_tps"])
+for k in ("decode_tps_115", "decode_tps_1056", "decode_tps_4096"):
+    print("METRIC %s=%s" % (k, m[k]))
+print("METRIC ttft_ms_4096=%s" % m["ttft_ms_4096"])
+print("METRIC kv_tokens=%s" % (run.get("kv_tokens") or 0))
+print("METRIC boot_seconds=%s" % (run.get("boot_seconds") or -1))
+print("METRIC completion_tokens_4096=%s" % m.get("completion_tokens_4096", 0))
+' "$1"
+}
+
+# --- memo fast-path -------------------------------------------------------
+cached_hash="$(ssh_run "cat '$STAGE/last/hash' 2>/dev/null" 2>/dev/null || true)"
+if [ -n "$cached_hash" ] && [ "$cached_hash" = "$HASH" ]; then
+    cached_run="$(ssh_run "cat '$STAGE/last/run.json' 2>/dev/null" || true)"
+    if [ -n "$cached_run" ]; then
+        echo "[autoresearch] memo hit — cached metrics for $HASH, fleet untouched" >&2
+        emit_metrics "$cached_run"
+        exit 0
+    fi
+fi
+
+# --- stage the candidate ----------------------------------------------------
+echo "[autoresearch] staging candidate on $HOST:$STAGE" >&2
+ssh_run "mkdir -p '$STAGE'"
+scp -q "$ENV_FILE" "$HOST:$STAGE/frank-primary.env"
+scp -q "$SERVE" "$HOST:$STAGE/serve.sh"
+scp -q "$MEASURE" "$HOST:$STAGE/measure.py"
+scp -q "$WINDOW" "$HOST:$STAGE/window.sh"
+ssh_run "printf '%s\n' '$HASH' > '$STAGE/hash'"
+
+# --- the window -------------------------------------------------------------
+# Budgets: bench boot up to 20 min (warm boots measured ~7 min), measure up
+# to 30 min, restore boot up to 20 min — call run_experiment with
+# timeout_seconds >= 3600.
+echo "[autoresearch] opening bench window (stop+mask -> boot -> measure -> restore)" >&2
+set +e
+ssh_run "bash '$STAGE/window.sh'"
+rc=$?
+set -e
+if [ "$rc" -ne 0 ]; then
+    echo "[autoresearch] window failed (rc=$rc); host-side evidence:" >&2
+    ssh_run "tail -60 '$STAGE/fail.log' 2>/dev/null || tail -40 '$STAGE/serve.out' 2>/dev/null || tail -40 '$STAGE/measure.out' 2>/dev/null || true" >&2 || true
+    exit 1
+fi
+
+run_json="$(ssh_run "cat '$STAGE/run.json'")" || {
+    echo "[autoresearch] could not read $STAGE/run.json back" >&2; exit 1; }
+echo "[autoresearch] window result:" >&2
+printf '%s\n' "$run_json" >&2
+emit_metrics "$run_json"
+exit 0

old/download-frank-large-dflash.sh

@@ -0,0 +1,76 @@
+#!/usr/bin/env bash
+# Download the z-lab Qwen3.5-9B-DFlash drafter for frank-large's DFlash
+# speculative-decode arm (vLLM --speculative-config method=dflash). Public
+# repo — no auth header on the command line.
+#
+# The drafter is a 6-layer qwen3 draft whose dflash_config pins
+# target_layer_ids [1,5,9,13,17,21,25,29] and block_size 16; its hidden_size
+# 4096 / vocab_size 248320 match the MiMo-V2.6-Distill-Qwen-9B target exactly
+# (both preserve the Qwen3.5-9B geometry), which is what lets vLLM load it as
+# the draft for that target.
+#
+# Usage:
+#   scripts/download-frank-large-dflash.sh [DEST_DIR] [--verify-only]
+set -euo pipefail
+
+REPO="z-lab/Qwen3.5-9B-DFlash"
+BASE="https://huggingface.co/${REPO}/resolve/main"
+
+DEST=""
+VERIFY_ONLY=0
+for arg in "$@"; do
+    case "$arg" in
+        --verify-only) VERIFY_ONLY=1 ;;
+        *) DEST="$arg" ;;
+    esac
+done
+DEST="${DEST:-/home/dank/models/Qwen3.5-9B-DFlash}"
+
+# name bytes sha256(- = non-LFS, size-only check)
+FILES=(
+  "model.safetensors 2583816465 0a42274b32554f48de1faa0d42824e9c2ceda649c30ae0a731cddf410dd698c7"
+  "config.json 1222 -"
+)
+
+mkdir -p "$DEST"
+status=0
+for entry in "${FILES[@]}"; do
+  read -r name size sha <<<"$entry"
+  f="$DEST/$name"
+  have=0
+  if [[ -f "$f" ]] && [[ "$(stat -c%s "$f")" == "$size" ]]; then
+    if [[ "$sha" == "-" ]] || sha256sum -c --quiet <(printf '%s  %s\n' "$sha" "$f"); then
+      have=1
+    fi
+  fi
+  if (( have )); then
+    echo "OK        $name"
+    continue
+  fi
+  if (( VERIFY_ONLY )); then
+    echo "MISSING   $name"
+    status=1
+    continue
+  fi
+  echo "FETCHING  $name"
+  tmp="$f.part"
+  curl -fsSL --fail-with-body -o "$tmp" "$BASE/$name"
+  mv "$tmp" "$f"
+  if [[ "$(stat -c%s "$f")" != "$size" ]]; then
+    echo "SIZE FAIL $name ($(stat -c%s "$f") != $size)"
+    status=1
+    continue
+  fi
+  if [[ "$sha" != "-" ]] && ! sha256sum -c --quiet <(printf '%s  %s\n' "$sha" "$f"); then
+    echo "SHA FAIL  $name"
+    status=1
+    continue
+  fi
+  echo "VERIFIED  $name"
+done
+if (( status == 0 )); then
+  echo "DOWNLOAD_DONE $DEST"
+else
+  echo "DOWNLOAD_FAILED $DEST" >&2
+fi
+exit "$status"

old/download-frank-large-mimo.sh

@@ -0,0 +1,90 @@
+#!/usr/bin/env bash
+# Download the BF16 MiMo-V2.6-Distill-Qwen-9B payload for frank-large
+# (vLLM/kyuz0 gfx1151, Strix Halo iGPU). Public repo — no auth header on the
+# command line (a Bearer token in argv is world-readable via /proc).
+#
+# Per-file FRESH downloads, no resume: the house rule (see
+# download-frank-small.sh); -f/--fail-with-body plus size+sha256 against the
+# Hub LFS oid is what makes an access error fail instead of "succeeding".
+#
+# Usage:
+#   scripts/download-frank-large-mimo.sh [DEST_DIR] [--verify-only]
+#
+# The verified set below is exactly what the 2026-10-01 bench window served:
+# 17.53 GiB of BF16 weights (4 shards, 18,819,720,848 B total with the rest).
+set -euo pipefail
+
+REPO="XiaomiMiMo/MiMo-V2.6-Distill-Qwen-9B"
+BASE="https://huggingface.co/${REPO}/resolve/main"
+
+DEST=""
+VERIFY_ONLY=0
+for arg in "$@"; do
+    case "$arg" in
+        --verify-only) VERIFY_ONLY=1 ;;
+        *) DEST="$arg" ;;
+    esac
+done
+DEST="${DEST:-/home/dank/models/MiMo-V2.6-Distill-Qwen-9B-BF16}"
+
+# name bytes sha256(- = non-LFS, size-only check)
+FILES=(
+  "model-00001-of-00004.safetensors 5276436216 aab052180118aee34abc3029b54eaa49096aac606b97d703866b420dceb703c3"
+  "model-00002-of-00004.safetensors 5033171280 7a0486565f06d25ac4628e9dba470dc3f604353471d240d5a0bf7128f64df396"
+  "model-00003-of-00004.safetensors 5234499504 6c73207563d1879bfd6c143a028cc70be68edff56280458c71a43df4240300f4"
+  "model-00004-of-00004.safetensors 3275613848 1379a7cf8c0b8555a39ab65a47e830e0eb45e776b46045734da3afd73c09eea2"
+  "model.safetensors.index.json 69217 -"
+  "config.json 2784 -"
+  "generation_config.json 126 -"
+  "chat_template.jinja 3916 -"
+  "tokenizer.json 19989325 06b9509352d2af50381ab2247e083b80d32d5c0aba91c272ca9ff729b6a0e523"
+  "tokenizer_config.json 1165 -"
+  "vocab.json 6722759 -"
+  "merges.txt 3353259 -"
+  "preprocessor_config.json 443 -"
+  "processor_config.json 1191 -"
+  "video_preprocessor_config.json 385 -"
+)
+
+mkdir -p "$DEST"
+status=0
+for entry in "${FILES[@]}"; do
+  read -r name size sha <<<"$entry"
+  f="$DEST/$name"
+  have=0
+  if [[ -f "$f" ]] && [[ "$(stat -c%s "$f")" == "$size" ]]; then
+    if [[ "$sha" == "-" ]] || sha256sum -c --quiet <(printf '%s  %s\n' "$sha" "$f"); then
+      have=1
+    fi
+  fi
+  if (( have )); then
+    echo "OK        $name"
+    continue
+  fi
+  if (( VERIFY_ONLY )); then
+    echo "MISSING   $name"
+    status=1
+    continue
+  fi
+  echo "FETCHING  $name"
+  tmp="$f.part"
+  curl -fsSL --fail-with-body -o "$tmp" "$BASE/$name"
+  mv "$tmp" "$f"
+  if [[ "$(stat -c%s "$f")" != "$size" ]]; then
+    echo "SIZE FAIL $name ($(stat -c%s "$f") != $size)"
+    status=1
+    continue
+  fi
+  if [[ "$sha" != "-" ]] && ! sha256sum -c --quiet <(printf '%s  %s\n' "$sha" "$f"); then
+    echo "SHA FAIL  $name"
+    status=1
+    continue
+  fi
+  echo "VERIFIED  $name"
+done
+if (( status == 0 )); then
+  echo "DOWNLOAD_DONE $DEST"
+else
+  echo "DOWNLOAD_FAILED $DEST" >&2
+fi
+exit "$status"

old/download-frank-large-qwen35.sh

@@ -0,0 +1,85 @@
+#!/usr/bin/env bash
+# Download the source Qwen3.5-9B checkpoint — the ONLY place this model
+# family's MTP head (15 mtp.* tensors) exists. frank-large runs it as the
+# vLLM cross-model MTP draft (MiMo-V2.6-Distill-Qwen-9B itself ships zero
+# mtp tensors; that is why SGLang's NEXTN on frank-small draws 0.00
+# acceptance). Public repo — no auth header on the command line.
+#
+# NOTE the odd upstream shard names (model.safetensors-0000N-of-00004...).
+#
+# Usage:
+#   scripts/download-frank-large-qwen35.sh [DEST_DIR] [--verify-only]
+set -euo pipefail
+
+REPO="Qwen/Qwen3.5-9B"
+BASE="https://huggingface.co/${REPO}/resolve/main"
+
+DEST=""
+VERIFY_ONLY=0
+for arg in "$@"; do
+    case "$arg" in
+        --verify-only) VERIFY_ONLY=1 ;;
+        *) DEST="$arg" ;;
+    esac
+done
+DEST="${DEST:-/home/dank/models/Qwen3.5-9B}"
+
+# name bytes sha256(- = non-LFS, size-only check)
+FILES=(
+  "model.safetensors-00001-of-00004.safetensors 5276436216 db6f444b43d318c92f360a13a25561a6a65b10c0631b8ed305a426dbaa6c380e"
+  "model.safetensors-00002-of-00004.safetensors 5335161512 31c7d7e2dd5d207840b31cc59083c8f4c4718959149e0358c0364052bb9a0330"
+  "model.safetensors-00003-of-00004.safetensors 5368717440 7ec36ba3a4176a44c3c0876ad80c56a2f70c84bf008d82e9501df642f17dadec"
+  "model.safetensors-00004-of-00004.safetensors 3325995712 b62b0c4cd7e44edee103ee8f4fe225f246d5e768e07bfd5f25b63a8aa1fdd0c6"
+  "model.safetensors.index.json 79657 -"
+  "config.json 3126 -"
+  "chat_template.jinja 7756 -"
+  "tokenizer.json 12807982 5f9e4d4901a92b997e463c1f46055088b6cca5ca61a6522d1b9f64c4bb81cb42"
+  "tokenizer_config.json 16710 -"
+  "vocab.json 6722759 -"
+  "merges.txt 3353259 -"
+  "preprocessor_config.json 390 -"
+  "video_preprocessor_config.json 385 -"
+)
+
+mkdir -p "$DEST"
+status=0
+for entry in "${FILES[@]}"; do
+  read -r name size sha <<<"$entry"
+  f="$DEST/$name"
+  have=0
+  if [[ -f "$f" ]] && [[ "$(stat -c%s "$f")" == "$size" ]]; then
+    if [[ "$sha" == "-" ]] || sha256sum -c --quiet <(printf '%s  %s\n' "$sha" "$f"); then
+      have=1
+    fi
+  fi
+  if (( have )); then
+    echo "OK        $name"
+    continue
+  fi
+  if (( VERIFY_ONLY )); then
+    echo "MISSING   $name"
+    status=1
+    continue
+  fi
+  echo "FETCHING  $name"
+  tmp="$f.part"
+  curl -fsSL --fail-with-body -o "$tmp" "$BASE/$name"
+  mv "$tmp" "$f"
+  if [[ "$(stat -c%s "$f")" != "$size" ]]; then
+    echo "SIZE FAIL $name ($(stat -c%s "$f") != $size)"
+    status=1
+    continue
+  fi
+  if [[ "$sha" != "-" ]] && ! sha256sum -c --quiet <(printf '%s  %s\n' "$sha" "$f"); then
+    echo "SHA FAIL  $name"
+    status=1
+    continue
+  fi
+  echo "VERIFIED  $name"
+done
+if (( status == 0 )); then
+  echo "DOWNLOAD_DONE $DEST"
+else
+  echo "DOWNLOAD_FAILED $DEST" >&2
+fi
+exit "$status"

old/download-frank-small-sglang-int8.sh

@@ -0,0 +1,93 @@
+#!/usr/bin/env bash
+# Download MiMo-V2.6-Distill-Qwen-9B-INT8-W8A16-AutoRound for frank-small
+# (SGLang, RTX 3090). Public repo — no auth header on the command line.
+#
+# Per-file FRESH downloads, no resume: `curl -C -` once produced a 213 MB
+# oversized file on a 31 GiB payload; a fresh -f fetch plus size+sha256 is
+# the house rule.
+#
+# Usage:
+#   scripts/download-frank-small.sh [DEST_DIR] [--verify-only]
+#
+# LFS shards are verified against the Hub LFS sha256; non-LFS text files
+# against their declared byte size. Idempotent: complete files are skipped.
+set -euo pipefail
+
+REPO="letechlead/MiMo-V2.6-Distill-Qwen-9B-INT8-W8A16-AutoRound"
+BASE="https://huggingface.co/${REPO}/resolve/main"
+
+DEST=""
+VERIFY_ONLY=0
+for arg in "$@"; do
+    case "$arg" in
+        --verify-only) VERIFY_ONLY=1 ;;
+        *) DEST="$arg" ;;
+    esac
+done
+DEST="${DEST:-/home/dank/models/MiMo-V2.6-Distill-Qwen-9B-INT8-W8A16-AutoRound}"
+
+# name bytes sha256(- = non-LFS, size-only check)
+FILES=(
+  "model-00001-of-00008.safetensors 2946296312 ddb6117c04cb6feaf7c265e5d75efadf21949037d73253a8e4521d40e164366d"
+  "model-00002-of-00008.safetensors 1402134792 2dc78d55d1d33d7b3b21dd6f8b4b817b62c2fb1376e6448b322428798c10e257"
+  "model-00003-of-00008.safetensors 1427365904 5c797d14e345309d67c78306b9de4aa01a21dbd92dd98e40b0ad69b2900e9203"
+  "model-00004-of-00008.safetensors 1420041856 7402f852e2c7a24ebf4694b98a220e71dfb2c93ba3776272cf2acfc65acd8126"
+  "model-00005-of-00008.safetensors 1410655648 991cbf2eda2ff91fc3ffbb15591d7fcac340fb10d9c33bd3da20010e2f3f4090"
+  "model-00006-of-00008.safetensors 1427627856 efc42448054cfb5e3061e154ba4539204088ce95176d689a95b58de885ef7eeb"
+  "model-00007-of-00008.safetensors 2035442408 75a101fc71ae6f2e1f5c4816f711d62cf1d7a0e02ef6e5182361b8c11e72b626"
+  "model-00008-of-00008.safetensors 822153664 0fb90cb50e2d336fbad58bd4c01880eeddbf537c1dca1cf0ae291195d5efbb4b"
+  "tokenizer.json 19989325 06b9509352d2af50381ab2247e083b80d32d5c0aba91c272ca9ff729b6a0e523"
+  "model.safetensors.index.json 106278 -"
+  "config.json 4608 -"
+  "generation_config.json 161 -"
+  "quantization_config.json 1654 -"
+  "chat_template.jinja 3916 -"
+  "tokenizer_config.json 1168 -"
+  "preprocessor_config.json 443 -"
+  "processor_config.json 1191 -"
+  "README.md 8448 -"
+)
+
+mkdir -p "$DEST"
+status=0
+for entry in "${FILES[@]}"; do
+  read -r name size sha <<<"$entry"
+  f="$DEST/$name"
+  have=0
+  if [[ -f "$f" ]] && [[ "$(stat -c%s "$f")" == "$size" ]]; then
+    if [[ "$sha" == "-" ]] || sha256sum -c --quiet <(printf '%s  %s\n' "$sha" "$f"); then
+      have=1
+    fi
+  fi
+  if (( have )); then
+    echo "OK        $name"
+    continue
+  fi
+  if (( VERIFY_ONLY )); then
+    echo "MISSING   $name"
+    status=1
+    continue
+  fi
+  echo "FETCHING  $name"
+  tmp="$f.part"
+  # no -C resume; -f + --fail-with-body so an HTTP error is not a success
+  curl -fsSL --fail-with-body -o "$tmp" "$BASE/$name"
+  mv "$tmp" "$f"
+  if [[ "$(stat -c%s "$f")" != "$size" ]]; then
+    echo "SIZE FAIL $name ($(stat -c%s "$f") != $size)"
+    status=1
+    continue
+  fi
+  if [[ "$sha" != "-" ]] && ! sha256sum -c --quiet <(printf '%s  %s\n' "$sha" "$f"); then
+    echo "SHA FAIL  $name"
+    status=1
+    continue
+  fi
+  echo "VERIFIED  $name"
+done
+if (( status == 0 )); then
+  echo "DOWNLOAD_DONE $DEST"
+else
+  echo "DOWNLOAD_FAILED $DEST" >&2
+fi
+exit "$status"

scripts/autoresearch-bench.py

@@ -0,0 +1,141 @@
+#!/usr/bin/env python3
+"""autoresearch-bench.py — deterministic greedy battery for frank-large.
+
+Runs ON Dankclaw (no LAN jitter) against the candidate container on :7770.
+Fixed embedded corpus, temperature 0, streaming so decode speed is measured
+between real token arrivals (not request totals). Median of REPS samples per
+metric is the reported value; medians are far more stable than means under
+GPU thermal drift.
+
+Metrics emitted (stdout, one per line):
+  METRIC decode_tokens_per_sec=<float>   primary, higher better
+  METRIC prefill_tokens_per_sec=<float>  secondary
+  METRIC ttft_ms=<float>                 secondary, lower better
+Exit non-zero if the endpoint is unhealthy or the served alias is wrong.
+"""
+import argparse
+import json
+import statistics
+import sys
+import time
+import urllib.request
+
+# Fixed corpus: deterministic, ~28k tokens after tokenization (same scale as
+# the 2026-10-01 bench window). Built by repeating a fixed paragraph block a
+# fixed number of times - NO random, NO clock, NO network.
+_BLOCK = (
+    "The rendering pipeline walks the scene graph once per frame, resolving "
+    "material bindings into descriptor sets before any draw call is recorded. "
+    "Because the allocator recycles command buffers only after their fence "
+    "signals, a submission that outlives its frame must pin the pool entry; "
+    "otherwise the next frame overwrites recorded commands in flight. The "
+    "validation layer flags exactly this class of lifetime bug, but only for "
+    "buffers it can see - GPU-side timestamps and queue-family ownership "
+    "transfers hide in the driver's own bookkeeping, which is why release "
+    "builds must lean on explicit barriers and why a frame-time regression "
+    "often shows up as corruption one frame later than its cause.\n"
+)
+LONG_CTX = _BLOCK * 220  # ~28k-token prefill, matching the deployed workload
+_LONG = LONG_CTX + "\nNow, in one focused paragraph (80-150 words), explain " \
+    "which lifetime bugs this pipeline can still suffer and what explicit " \
+    "mitigations the text implies."
+_SHORT = "Design a rate limiter for a homelab reverse proxy that must not " \
+    "lock out the LAN. Think it through, then give the algorithm."
+
+REPS = 3
+MAX_TOKENS = 256
+REQ_TIMEOUT = 900
+
+
+def stream_completion(url: str, model: str, prompt: str):
+    """Greedy streaming completion.
+
+    Returns (ttft_s, decode_tps, prefill_tps, prompt_tokens, n_tokens).
+    ttft is RELATIVE to request send: time.perf_counter()'s epoch is host
+    boot, and returning the raw stamp read as an 11-day first token.
+    """
+    body = json.dumps({
+        "model": model,
+        "messages": [{"role": "user", "content": prompt}],
+        "max_tokens": MAX_TOKENS,
+        "temperature": 0,
+        "stream": True,
+        "stream_options": {"include_usage": True},
+    }).encode()
+    req = urllib.request.Request(
+        url.rstrip("/") + "/v1/chat/completions", data=body,
+        headers={"Content-Type": "application/json"})
+    t0 = time.perf_counter()
+    first = None
+    stamps = []
+    usage = {}
+    with urllib.request.urlopen(req, timeout=REQ_TIMEOUT) as resp:
+        for raw in resp:
+            line = raw.decode(errors="replace").strip()
+            if not line.startswith("data:"):
+                continue
+            payload = line[5:].strip()
+            if payload == "[DONE]":
+                break
+            chunk = json.loads(payload)
+            if chunk.get("usage"):
+                usage = chunk["usage"]
+            for ch in chunk.get("choices", []):
+                d = ch.get("delta") or {}
+                # reasoning AND content deltas both arrive token by token;
+                # count either so greedy text length drives the rate.
+                # (llama.cpp names the thinking field reasoning_content.)
+                if d.get("content") or d.get("reasoning") or d.get("reasoning_content"):
+                    now = time.perf_counter()
+                    if first is None:
+                        first = now
+                    stamps.append(now)
+    if first is None or len(stamps) < 2:
+        # A zero-argmax degenerate answer would poison the median; raise.
+        raise RuntimeError(f"stream produced {len(stamps)} tokens")
+    ttft_s = first - t0
+    decode_s = stamps[-1] - stamps[0]
+    decode_tps = (len(stamps) - 1) / decode_s if decode_s > 0 else 0.0
+    prompt_tokens = usage.get("prompt_tokens") or 0
+    prefill_tps = prompt_tokens / ttft_s if prompt_tokens else 0.0
+    return ttft_s, decode_tps, prefill_tps, prompt_tokens, len(stamps)
+
+
+def main():
+    ap = argparse.ArgumentParser()
+    ap.add_argument("--url", default="http://127.0.0.1:7770")
+    ap.add_argument("--model", default="frank-large")
+    args = ap.parse_args()
+
+    with urllib.request.urlopen(args.url.rstrip("/") + "/v1/models",
+                                timeout=10) as r:
+        ids = [m["id"] for m in json.load(r)["data"]]
+    if args.model not in ids:
+        print(f"FATAL: served ids {ids}, expected {args.model}",
+              file=sys.stderr)
+        sys.exit(2)
+
+    # Warmup (compile/graph warm + allocator settle), NOT measured.
+    stream_completion(args.url, args.model, _SHORT)
+
+    decodes, prefills, ttfts = [], [], []
+    for rep in range(REPS):
+        for name, prompt in (("long", _LONG), ("short", _SHORT)):
+            ttft, dt, pt, ptk, ntok = stream_completion(
+                args.url, args.model, prompt)
+            decodes.append(dt)
+            ttfts.append(ttft * 1000.0)
+            if name == "long":
+                prefills.append(pt)
+            print(f"  rep{rep} {name:5s} ttft={ttft*1000:8.1f}ms "
+                  f"prefill={pt:8.1f} t/s decode={dt:6.2f} t/s "
+                  f"({ntok} tok, prompt {ptk})", file=sys.stderr)
+
+    med = lambda xs: statistics.median(xs)
+    print(f"METRIC decode_tokens_per_sec={med(decodes):.2f}")
+    print(f"METRIC prefill_tokens_per_sec={med(prefills):.1f}")
+    print(f"METRIC ttft_ms={med(ttfts):.1f}")
+
+
+if __name__ == "__main__":
+    main()

scripts/autoresearch/frank-primary-measure.py

@@ -0,0 +1,192 @@
+#!/usr/bin/env python3
+"""frank-primary decode measurement — runs INSIDE the engine container.
+
+The method is the fleet's established one (the 2026-09-29 / 2026-10-01
+tuning campaigns, /home/dank/mtp_measure.py): SSE chunk counting is invalid
+under speculative decoding because rejected drafts change the accounting, so
+decode is measured from the server's own usage counters:
+
+    decode_tok_s = completion_tokens / (wall - ttft)
+
+with ttft = wall time to the FIRST content-bearing chunk (thinking arrives in
+`reasoning`/`delta.reasoning` on this stack and is excluded from the decode
+window — and is deterministic per prompt at temperature 0, so the exclusion
+cancels in any A/B on the same workload).
+
+Workload (fixed, deterministic, offline):
+  * prompts built from a large fixed-seed shuffled word list so tokens rarely
+    repeat (no n-gram/prefix predictability of the text itself), sized with
+    the server's own /v1/messages/count_tokens counter
+  * depths 115 / 1056 / 4096 target prompt tokens
+  * one 64-token warm pass per depth (warms the expert arena), then
+    max_tokens=256 at temperature 0, 3 runs, median
+Everything is greedy with a fixed seed — the same workload every run, no
+network, no time-of-day dependence.
+
+Env:
+  AR_BASE             endpoint (default http://127.0.0.1:7770)
+  AR_MODEL            served name (default frank-primary)
+  AR_PROMPT_CEILING   hard prompt ceiling in tokens (default 100000)
+
+Output: progress on stderr, ONE json object on stdout (the last line).
+Exit codes: 0 ok, 2 endpoint unhealthy / no usable completion, 3 bad usage.
+"""
+import json
+import os
+import random
+import statistics
+import sys
+import time
+import urllib.error
+import urllib.request
+
+BASE = os.environ.get("AR_BASE", "http://127.0.0.1:7770")
+MODEL = os.environ.get("AR_MODEL", "frank-primary")
+PROMPT_CEILING = int(os.environ.get("AR_PROMPT_CEILING", "100000"))
+DEPTHS = (115, 1056, 4096)
+WARM_TOKENS = 64
+MAX_TOKENS = 256
+RUNS = 3
+
+# 43-word x 12 = 516-word vocabulary, seed 1234 — the campaign's exact list
+# and seed, so prompts are byte-identical to the historical measurements.
+_VOCAB = ("alpha bravo charlie delta echo foxtrot golf hotel india juliet kilo lima "
+          "mike november oscar papa quebec romeo sierra tango uniform victor whiskey "
+          "xray yankee zulu cardinal ordinal topology kernel scheduler resident "
+          "unified aperture expert arena slab offload prefetch gemm triton quant "
+          "tensor shard checkpoint manifest embed router cache page table window "
+          "gradient sparse dense moe attention head dim block stream decode prefill "
+          "bandwidth latency throughput residency eviction tier swap pin memlock").split()
+_R = random.Random(1234)
+_ORDER = list(range(len(_VOCAB)))
+_R.shuffle(_ORDER)
+
+
+def log(msg):
+    print("[measure] %s" % msg, file=sys.stderr, flush=True)
+
+
+def post_json(path, body, timeout):
+    req = urllib.request.Request(
+        BASE + path, data=json.dumps(body).encode(),
+        headers={"Content-Type": "application/json"})
+    with urllib.request.urlopen(req, timeout=timeout) as r:
+        return json.load(r)
+
+
+def count_tokens(text):
+    """Token count without a prefill, from the server's own counter."""
+    body = {"model": MODEL,
+            "messages": [{"role": "user", "content": text}]}
+    r = post_json("/v1/messages/count_tokens", body, timeout=300)
+    n = r.get("input_tokens") or r.get("count") or r.get("token_count")
+    if not n:
+        raise RuntimeError("count_tokens returned no count: %r" % (r,))
+    return int(n)
+
+
+def words(n):
+    return " ".join(_VOCAB[_ORDER[i % len(_VOCAB)]] for i in range(n))
+
+
+def calibrate():
+    """Measure words->tokens from the server's own counter."""
+    return 2000.0 / count_tokens(words(2000))
+
+
+def prompt_for(target_tokens, wpt):
+    n = max(1, int(target_tokens * wpt))
+    text = words(n)
+    pt = count_tokens(text)
+    while pt > PROMPT_CEILING:
+        n = max(1, int(n * (PROMPT_CEILING / pt) * 0.98))
+        text = words(n)
+        pt = count_tokens(text)
+    return text, pt
+
+
+def measured(prompt, max_tokens):
+    """Return (ttft_s, decode_tok_s, completion_tokens) via usage counters."""
+    body = {"model": MODEL,
+            "messages": [{"role": "user", "content": prompt}],
+            "max_tokens": max_tokens, "temperature": 0,
+            "stream": True, "stream_options": {"include_usage": True}}
+    req = urllib.request.Request(
+        BASE + "/v1/chat/completions", data=json.dumps(body).encode(),
+        headers={"Content-Type": "application/json"})
+    t0 = time.time()
+    ttft = None
+    usage = {}
+    with urllib.request.urlopen(req, timeout=1800) as r:
+        for raw in r:
+            line = raw.decode("utf-8", "replace").strip()
+            if not line.startswith("data:"):
+                continue
+            s = line[5:].strip()
+            if s == "[DONE]":
+                break
+            try:
+                ev = json.loads(s)
+            except json.JSONDecodeError:
+                continue
+            if ev.get("usage"):
+                usage = ev["usage"]
+            for ch in ev.get("choices", []):
+                if (ch.get("delta") or {}).get("content"):
+                    if ttft is None:
+                        ttft = time.time() - t0
+    wall = time.time() - t0
+    ct = int(usage.get("completion_tokens") or 0)
+    if ct <= 0:
+        # No usage on the stream: the endpoint is unhealthy or the model
+        # returned nothing (e.g. HTTP 200 with an error payload).
+        raise RuntimeError("no completion_tokens in usage: %r" % (usage,))
+    dec_window = max(wall - (ttft or 0.0), 1e-9)
+    return (ttft or wall), (ct / dec_window), ct
+
+
+def main():
+    log("base=%s model=%s ceiling=%d" % (BASE, MODEL, PROMPT_CEILING))
+    try:
+        wpt = calibrate()
+    except Exception as e:
+        log("endpoint unhealthy during calibration: %s" % e)
+        return 2
+    log("words/token: %.3f" % wpt)
+
+    result = {"words_per_token": round(wpt, 5)}
+    for tgt in DEPTHS:
+        try:
+            p, pt = prompt_for(tgt, wpt)
+            measured(p, WARM_TOKENS)  # warm the arena
+        except Exception as e:
+            log("warm pass failed at depth %d: %s" % (tgt, e))
+            return 2
+        rates, tts, cts = [], [], []
+        for i in range(RUNS):
+            try:
+                tt, r, ct = measured(p, MAX_TOKENS)
+            except Exception as e:
+                log("run %d/%d failed at depth %d: %s" % (i + 1, RUNS, tgt, e))
+                continue
+            rates.append(r)
+            tts.append(tt)
+            cts.append(ct)
+        if len(rates) < 2:
+            log("only %d usable run(s) at depth %d — refusing to median" % (len(rates), tgt))
+            return 2
+        result["decode_tps_%d" % tgt] = round(statistics.median(rates), 2)
+        result["ttft_ms_%d" % tgt] = round(1000.0 * statistics.median(tts), 1)
+        result["completion_tokens_%d" % tgt] = int(statistics.median(cts))
+        result["prompt_tokens_%d" % tgt] = pt
+        log("depth %5d tok  decode %.2f t/s  ttft %.3f s  (runs=%d)"
+            % (pt, result["decode_tps_%d" % tgt], result["ttft_ms_%d" % tgt] / 1000.0, len(rates)))
+
+    meds = [result["decode_tps_%d" % t] for t in DEPTHS]
+    result["decode_tps"] = round(statistics.median(meds), 2)
+    print(json.dumps(result))
+    return 0
+
+
+if __name__ == "__main__":
+    sys.exit(main())

scripts/autoresearch/frank-primary-window.sh

@@ -0,0 +1,200 @@
+#!/usr/bin/env bash
+# frank-primary-window.sh — one full bench window for frank-primary, run ON
+# DANKSERVER as the deploy user (invoked over ssh by ./autoresearch.sh).
+#
+# Discipline (AGENTS.md Benchmarking Policy): the production unit is
+# STOPPED + MASKED first, the candidate config boots AS THE SERVICE itself
+# (same serve script, same container name, same port 7770 — no scratch
+# ports, no co-tenant GPU), the in-container measure runs, and the window
+# ALWAYS ends by restoring the production unit and verifying its health.
+# The EXIT trap makes the restore unconditional: a boot failure, a measure
+# failure, or a dead ssh session all end with production back up.
+#
+# Staging layout (created by the orchestrator, ephemeral — never a config
+# source; the repo checkout is):
+#   /home/dank/.autoresearch-frank-primary/
+#     serve.sh             staged candidate serve script
+#     frank-primary.env    staged candidate env (the candidate knobs)
+#     measure.py           the in-container measure client
+#     hash                 candidate fingerprint
+#     run.json             result (metrics + boot + kv + fingerprint)
+#     serve.out/measure.out/fail.log
+#     last/                memo: run.json + hash of the last FULLY
+#                          successful window (bench AND verified restore)
+#
+# This script logs to stderr and exits 0 only when the measure produced a
+# usable result (the restore happens in the trap regardless of the bench
+# outcome; a failed restore forces exit 1).
+set -uo pipefail
+
+STAGE="${FRANK_PRIMARY_AR_STAGE:-/home/dank/.autoresearch-frank-primary}"
+UNIT="${FRANK_PRIMARY_AR_UNIT:-frank-primary}"
+CTR="${FRANK_PRIMARY_AR_CTR:-frank-primary-qwen}"
+HEALTH_URL="http://127.0.0.1:7770/health"
+MODELS_URL="http://127.0.0.1:7770/v1/models"
+BOOT_TIMEOUT="${FRANK_PRIMARY_AR_BOOT_TIMEOUT:-1200}"
+RESTORE_TIMEOUT="${FRANK_PRIMARY_AR_RESTORE_TIMEOUT:-1200}"
+MEASURE_TIMEOUT="${FRANK_PRIMARY_AR_MEASURE_TIMEOUT:-1800}"
+SERVED="$(sed -n 's/^FRANK_PRIMARY_SERVED_NAME=//p' "$STAGE/frank-primary.env" 2>/dev/null | head -1)"
+SERVED="${SERVED:-frank-primary}"
+
+RESTORED=0
+
+log() { printf '[window %s] %s\n' "$(date '+%H:%M:%S')" "$*" >&2; }
+
+container_state() { podman inspect -f '{{.State.Status}}' "$CTR" 2>/dev/null || echo absent; }
+
+wait_port_free() {
+    local i
+    for i in $(seq 1 90); do
+        [ "$(container_state)" != "running" ] && return 0
+        sleep 2
+    done
+    return 1
+}
+
+wait_health() { # $1 = timeout seconds
+    local t0 code
+    t0=$(date +%s)
+    while :; do
+        code="$(curl -s -o /dev/null -w '%{http_code}' --max-time 10 "$HEALTH_URL" 2>/dev/null || echo 000)"
+        [ "$code" = "200" ] && return 0
+        [ "$(( $(date +%s) - t0 ))" -ge "$1" ] && return 1
+        sleep 2
+    done
+}
+
+capture_bench_failure() {
+    log "capturing bench-boot failure evidence"
+    {
+        echo "=== podman logs (tail 400) ==="
+        podman logs --tail 400 "$CTR" 2>&1 || true
+        echo "=== staged serve.out (tail 60) ==="
+        tail -60 "$STAGE/serve.out" 2>/dev/null || true
+        echo "=== engine log (tail 60) ==="
+        tail -60 /home/dank/logs/frank_primary.log 2>/dev/null || true
+    } >"$STAGE/fail.log" 2>&1 || true
+    grep -E 'OutOfMemory|Tried to allocate|Engine core initialization failed|Error|error' \
+        "$STAGE/fail.log" 2>/dev/null | tail -25 >&2 || true
+}
+
+restore() {
+    [ "$RESTORED" = 1 ] && return 0
+    RESTORED=1
+    log "restoring production unit $UNIT"
+    if [ -f "$STAGE/serve.pid" ]; then
+        kill "$(cat "$STAGE/serve.pid" 2>/dev/null)" 2>/dev/null || true
+    fi
+    podman stop -t 120 "$CTR" >/dev/null 2>&1 || true
+    wait_port_free || log "WARNING: bench container still draining after 180s"
+    sudo systemctl unmask "$UNIT" 2>/dev/null || true
+    if ! sudo systemctl start "$UNIT"; then
+        log "ERROR: systemctl start $UNIT failed — operator intervention required"
+        return 1
+    fi
+    if wait_health "$RESTORE_TIMEOUT"; then
+        if curl -s --max-time 10 "$MODELS_URL" 2>/dev/null | grep -q "$SERVED"; then
+            log "production restored and verified ($SERVED answering on :7770)"
+            # Memo promotion lives HERE: a cached result is only ever the
+            # product of a fully successful window (bench AND verified
+            # restore), never of a bare bench success.
+            cp "$STAGE/run.json" "$STAGE/last/" 2>/dev/null || true
+            cp "$STAGE/hash" "$STAGE/last/" 2>/dev/null || true
+        else
+            log "ERROR: health 200 but /v1/models does not list $SERVED"
+            return 1
+        fi
+    else
+        log "ERROR: production not healthy within ${RESTORE_TIMEOUT}s — unit left to systemd (Restart=always)"
+        return 1
+    fi
+    return 0
+}
+trap 'restore; rc=$?; [ "$rc" -ne 0 ] && exit 1' EXIT
+
+main() {
+    local f
+    for f in serve.sh frank-primary.env measure.py hash; do
+        [ -f "$STAGE/$f" ] || { log "staged file missing: $STAGE/$f"; exit 1; }
+    done
+    mkdir -p "$STAGE/last"
+    rm -f "$STAGE/run.json" "$STAGE/fail.log" "$STAGE/measure.out"
+
+    local t_start t0 i kvline kv_tokens kv_conc measure_json boot_seconds
+    t_start="$(date +%s)"
+
+    log "stopping and masking production unit $UNIT"
+    sudo systemctl stop "$UNIT" 2>/dev/null || true
+    # Explicit stop is not auto-restarted even with Restart=always; the mask
+    # is belt-and-braces against a stray start during the window.
+    sudo systemctl mask "$UNIT" 2>/dev/null || \
+        log "WARNING: mask failed — relying on explicit-stop semantics"
+    wait_port_free || log "note: production container still draining"
+
+    t0="$(date +%s)"
+    log "starting bench instance from staged config (fingerprint $(tr -d '[:space:]' <"$STAGE/hash"))"
+    FRANK_PRIMARY_ENV_FILE="$STAGE/frank-primary.env" \
+        setsid nohup bash "$STAGE/serve.sh" >"$STAGE/serve.out" 2>&1 </dev/null &
+    echo "$!" >"$STAGE/serve.pid"
+
+    for i in $(seq 1 45); do
+        [ "$(container_state)" = "running" ] && break
+        sleep 2
+    done
+    if [ "$(container_state)" != "running" ]; then
+        log "bench container never reached running state"
+        tail -40 "$STAGE/serve.out" >&2 || true
+        exit 1
+    fi
+
+    if wait_health "$BOOT_TIMEOUT"; then
+        boot_seconds=$(( $(date +%s) - t0 ))
+        log "bench healthy after ${boot_seconds}s"
+    else
+        if [ "$(container_state)" = "running" ]; then
+            log "bench boot timed out after ${BOOT_TIMEOUT}s (still loading)"
+        else
+            log "bench container died during boot"
+            capture_bench_failure
+        fi
+        exit 1
+    fi
+
+    log "running measure in container (timeout ${MEASURE_TIMEOUT}s)"
+    if ! podman cp "$STAGE/measure.py" "$CTR":/tmp/ar-measure.py; then
+        log "podman cp of measure client failed"
+        exit 1
+    fi
+    if ! timeout "$MEASURE_TIMEOUT" podman exec "$CTR" python3 /tmp/ar-measure.py \
+            >"$STAGE/measure.out" 2>&1; then
+        log "measure failed or timed out"
+        tail -40 "$STAGE/measure.out" >&2 || true
+        exit 1
+    fi
+
+    # Capacity facts for the run (context-size experiments change these).
+    kvline="$(podman logs "$CTR" 2>&1 | grep -E 'GPU KV cache size' | tail -1 || true)"
+    kv_tokens="$(printf '%s' "$kvline" | sed -E 's/.*GPU KV cache size: ([0-9,]+) tokens.*/\1/' | tr -d ',' || true)"
+    kv_conc="$(printf '%s' "$kvline" | sed -E 's/.*per request: ([0-9.]+x).*/\1/' || true)"
+
+    measure_json="$(grep -E '^\{' "$STAGE/measure.out" | tail -1 || true)"
+    if [ -z "$measure_json" ]; then
+        log "measure produced no JSON result line"
+        tail -40 "$STAGE/measure.out" >&2 || true
+        exit 1
+    fi
+
+    # The measure line is already JSON, so the run object can be assembled
+    # without any host-side interpreter.
+    printf '{"hash":"%s","boot_seconds":%s,"kv_tokens":%s,"kv_concurrency":"%s","total_seconds":%s,"measure":%s}\n' \
+        "$(tr -d '[:space:]' <"$STAGE/hash")" \
+        "$boot_seconds" "${kv_tokens:-0}" "${kv_conc:-unknown}" \
+        "$(( $(date +%s) - t_start ))" "$measure_json" >"$STAGE/run.json"
+
+    log "bench window complete in $(( $(date +%s) - t_start ))s"
+    # Memo only after a FULL success; the restore below still has to verify.
+    exit 0
+}
+
+main "$@"
+exit "$?"

scripts/build-gufo.sh

@@ -0,0 +1,96 @@
+#!/usr/bin/env bash
+# Build the gufo runtime that serves frank-large on Dankclaw.
+#
+#   bash scripts/build-gufo.sh              # build + install the pinned tag
+#   bash scripts/build-gufo.sh --verify     # only check the installed tree
+#
+# gufo (gufo-org/gufo) is the Strix Halo inference engine: it targets
+# gfx1151 specifically and has no prebuilt Linux release binaries, so the
+# runtime is built from the pinned git tag and installed to
+# /home/dank/GitPackages/gufo-<tag>/. Everything below is what Dankclaw
+# needs on top of AMD's Linux instructions.
+#
+# The host already carries the ROCm 7.2.4 stack (/opt/rocm -> /etc/
+# alternatives/rocm) with the HIP dev packages, so this script installs
+# only the ordinary system libraries gufo links against.
+#
+# Two host facts this build depends on, both discovered on 2026-09-28:
+#
+#   1. Ubuntu's librocwmma-dev ships a TRUNCATED rocWMMA header set (the
+#      public headers without rocwmma/internal/) into /usr/include, where
+#      it shadows AMD's complete copy under /opt/rocm/include. The first
+#      build died on "'internal/accessors.hpp' file not found" from
+#      /usr/include/rocwmma/rocwmma.hpp. A -I cannot fix it: in HIP mode
+#      the host compilation pass searches the system directories first.
+#      The package has no reverse dependencies, so it is removed.
+#   2. The ROCm clang picks the NEWEST /usr/include/c++ tree, which on
+#      Ubuntu 26.04 is GCC 16's. Its <atomic_wait.h> emits a symbol
+#      reference the installed libstdc++ (GLIBCXX_3.4.35, GCC 15) does not
+#      export, and the link fails with "undefined symbol:
+#      std::__detail::__notify_impl". --gcc-install-dir pins the headers to
+#      the same GCC the runtime library came from.
+set -euo pipefail
+
+GUFO_TAG="${GUFO_TAG:-v0.1.1}"
+GUFO_REPO="${GUFO_REPO:-https://github.com/gufo-org/gufo}"
+# The commit the tag resolved to when this runtime was qualified. A tag that
+# moves is an unreproducible runtime, so the pin is checked, not assumed.
+GUFO_COMMIT="${GUFO_COMMIT:-b0f8673470a3dcc2b5e9c9546c48a40b82b5ff31}"
+SRC_DIR="${GUFO_SRC_DIR:-/home/dank/GitPackages/gufo-src}"
+PREFIX="${GUFO_PREFIX:-/home/dank/GitPackages/gufo-${GUFO_TAG}}"
+JOBS="${JOBS:-$(nproc)}"
+# The GCC toolchain that owns the installed libstdc++ on this host.
+GCC_DIR="${GCC_DIR:-/usr/lib/gcc/x86_64-linux-gnu/15}"
+
+# check_pin — refuse a source tree that is not the qualified commit.
+check_pin() {
+    local commit
+    commit="$(git -C "$SRC_DIR" rev-parse HEAD 2>/dev/null || true)"
+    if [[ -n "$commit" && "$commit" != "$GUFO_COMMIT" ]]; then
+        echo "GUFO_COMMIT mismatch: $SRC_DIR is at $commit, pinned $GUFO_COMMIT" >&2
+        exit 1
+    fi
+}
+
+if [[ "${1:-}" == "--verify" ]]; then
+    # The pin first: a moved tag is the more urgent thing to hear about, and
+    # it is the check that must not be masked by a missing install.
+    check_pin
+    [[ -x "$PREFIX/bin/gufo" ]] || { echo "no gufo at $PREFIX/bin/gufo" >&2; exit 1; }
+    "$PREFIX/bin/gufo" diagnose | tail -n 5
+    exit 0
+fi
+
+echo "== system libraries gufo links against =="
+sudo -n apt-get install -y -qq --no-install-recommends \
+    build-essential cmake ninja-build pkg-config \
+    libicu-dev libcurl4-openssl-dev libssl-dev libpng-dev libjpeg-dev ffmpeg
+
+echo "== dropping Ubuntu's truncated rocWMMA headers =="
+if dpkg -s librocwmma-dev >/dev/null 2>&1; then
+    sudo -n apt-get remove -y -qq librocwmma-dev
+fi
+
+echo "== source: $GUFO_REPO @ $GUFO_TAG =="
+if [[ ! -d "$SRC_DIR/.git" ]]; then
+    mkdir -p "$(dirname "$SRC_DIR")"
+    git clone --depth 1 --branch "$GUFO_TAG" "$GUFO_REPO" "$SRC_DIR"
+fi
+git -C "$SRC_DIR" fetch --depth 1 origin "refs/tags/$GUFO_TAG"
+git -C "$SRC_DIR" checkout -q --detach "refs/tags/$GUFO_TAG"
+echo "   $(git -C "$SRC_DIR" rev-parse HEAD)"
+check_pin
+
+echo "== configure + build (release, HIP gfx1151) =="
+cd "$SRC_DIR"
+toolchain="--gcc-install-dir=$GCC_DIR"
+cmake --preset release \
+    -DCMAKE_INSTALL_PREFIX="$PREFIX" \
+    -DCMAKE_HIP_FLAGS="$toolchain" \
+    -DCMAKE_EXE_LINKER_FLAGS="$toolchain"
+cmake --build --preset release --parallel "$JOBS"
+cmake --install build/release
+
+echo "== verify =="
+"$PREFIX/bin/gufo" diagnose
+echo "frank-large runtime ready: $PREFIX/bin/gufo ($GUFO_TAG @ $commit)"

scripts/build-rocmfpx.sh

@@ -0,0 +1,89 @@
+#!/usr/bin/env bash
+# Build the ROCmFPX runtime that serves frank-large on Dankclaw.
+#
+#   bash scripts/build-rocmfpx.sh              # clone/fetch + build the pin
+#   bash scripts/build-rocmfpx.sh --verify     # only check the installed tree
+#
+# Runtime: kingjones30/ROCmFPX — charlie12345/ROCmFPX plus the model-card
+# patches, i.e. the fork that owns the ROCmFPx tensor formats. frank-large's
+# payload is ftype 115 (`Q8_0_ROCMFPX_AGENT`): 311 tensors of
+# GGML_TYPE_Q8_0_ROCMFPX (103), a type that exists ONLY in this fork, so
+# stock llama.cpp, gufo and every other loader on the host reject the file.
+# This is the same tree that served the Qwen3.8-Flash-Next ROCmFP4 STRIX
+# endpoint 2026-08-30 → 09-11 (see the kingjones runbook), rebuilt here so a
+# fresh host can be brought up from the repo alone.
+#
+# Why this commit and not a newer one: 36e9acd (2026-08-28) is what was
+# qualified against this payload on 2026-10-02 — it loads ftype 115, survives
+# warmup, and drafts with the MTP head at acceptance 0.65. The five legacy
+# builds of 2026-08-25 crashed this same file in ggml_compute_forward_set_rows
+# (see kingjones-q8agent-rocmfpx.md), so the pin is load-bearing: do not bump
+# it without re-running that crash test.
+#
+# Host notes:
+#   * -DLLAMA_BUILD_WEBUI=OFF is REQUIRED on this box. The tree provisions its
+#     embedded UI assets via npm or the HF llama-ui bucket; npm is absent here
+#     and the bucket download half-failed, aborting the link at 98%. The API
+#     server is unaffected.
+#   * The build links against the host ROCm (/opt/rocm); no bundled toolchain
+#     like the gfx1201 R9700 build needed.
+set -euo pipefail
+
+ROCMFPX_REPO="${ROCMFPX_REPO:-https://github.com/kingjones30/ROCmFPX.git}"
+# The commit frank-large was qualified with (measured 2026-10-02).
+ROCMFPX_COMMIT="${ROCMFPX_COMMIT:-36e9acd40e10a87cd3c3ef8ec734668757dc8520}"
+SRC_DIR="${ROCMFPX_SRC_DIR:-/home/dank/GitPackages/kingjones-ROCmFPX}"
+BUILD_DIR="${ROCMFPX_BUILD_DIR:-$SRC_DIR/build}"
+GPU_TARGET="${ROCMFPX_GPU_TARGET:-gfx1151}"
+JOBS="${JOBS:-$(nproc)}"
+
+# check_pin — refuse a source tree that is not the qualified commit. A moved
+# fork means different kernels under the same path, and on this model line a
+# wrong build is a segfault at first request, not a clean error.
+check_pin() {
+    local commit
+    commit="$(git -C "$SRC_DIR" rev-parse HEAD 2>/dev/null || true)"
+    if [[ -n "$commit" && "$commit" != "$ROCMFPX_COMMIT" ]]; then
+        echo "ROCMFPX_COMMIT mismatch: $SRC_DIR is at $commit, pinned $ROCMFPX_COMMIT" >&2
+        exit 1
+    fi
+}
+
+if [[ "${1:-}" == "--verify" ]]; then
+    check_pin
+    [[ -x "$BUILD_DIR/bin/llama-server" ]] || {
+        echo "no ROCmFPX llama-server at $BUILD_DIR/bin/llama-server" >&2
+        exit 1
+    }
+    # The format the endpoint exists for: the fork must carry the ROCmFPx
+    # tensor type, or the payload cannot load at all.
+    "$BUILD_DIR/bin/llama-server" --help 2>&1 | grep -q -- '--spec-type' || {
+        echo "$BUILD_DIR/bin/llama-server has no --spec-type: MTP drafting is impossible" >&2
+        exit 1
+    }
+    "$BUILD_DIR/bin/llama-server" --version | head -1
+    exit 0
+fi
+
+echo "== source: $ROCMFPX_REPO @ $ROCMFPX_COMMIT =="
+if [[ ! -d "$SRC_DIR/.git" ]]; then
+    mkdir -p "$(dirname "$SRC_DIR")"
+    git clone "$ROCMFPX_REPO" "$SRC_DIR"
+fi
+git -C "$SRC_DIR" fetch --quiet origin "$ROCMFPX_COMMIT" || true
+git -C "$SRC_DIR" checkout -q --detach "$ROCMFPX_COMMIT"
+check_pin
+echo "   $(git -C "$SRC_DIR" rev-parse HEAD)"
+
+echo "== configure + build (release, HIP $GPU_TARGET) =="
+cmake -S "$SRC_DIR" -B "$BUILD_DIR" \
+    -DGGML_HIP=ON \
+    -DGPU_TARGETS="$GPU_TARGET" \
+    -DGGML_NATIVE=ON \
+    -DCMAKE_BUILD_TYPE=Release \
+    -DLLAMA_BUILD_WEBUI=OFF
+cmake --build "$BUILD_DIR" --target llama-server llama-quantize -j "$JOBS"
+
+echo "== verify =="
+"$BUILD_DIR/bin/llama-server" --version | head -1
+echo "frank-large runtime ready: $BUILD_DIR/bin/llama-server ($ROCMFPX_COMMIT)"

scripts/dankstatus.sh

@@ -0,0 +1,71 @@
+#!/usr/bin/env bash
+# Single source of truth for Dankserver GLM-5.3 state.
+#
+# This exists because the ad-hoc status commands kept producing WRONG readings.
+# The specific trap: /proc/meminfo separates key from value with a TAB, so the
+# common idiom
+#   grep MemAvailable /proc/meminfo | tr -s " " | cut -d" " -f2
+# yields an empty field (tab is not a space) and awk then prints 0.0 -- which
+# reads as "host out of memory" when the host actually has 240 GiB free.
+# Everything below parses on a colon split, which is tab-agnostic, and the
+# memory figure is cross-checked before it is shown.
+set -uo pipefail
+
+MB=1048576
+GLM_CTR="${GLM_CTR:-glm-w4a16}"
+
+mem_field() { awk -v k="$1:" '$1==k {printf "%.1f", $2/'"$MB"'; exit}' /proc/meminfo; }
+section() { printf '\n== %s ==\n' "$1"; }
+
+section "time / host"
+printf '  %s   %s\n' "$(date +%H:%M:%S)" \
+  "$(awk -F: '/model name/{print $2; exit}' /proc/cpuinfo | sed 's/^ *//')"
+
+section "memory (GiB)"
+TOT=$(mem_field MemTotal); AVAIL=$(mem_field MemAvailable)
+printf '  MemTotal     %8s\n' "$TOT"
+printf '  MemAvailable %8s\n' "$AVAIL"
+printf '  Cached       %8s\n' "$(mem_field Cached)"
+awk -v t="$TOT" -v a="$AVAIL" 'BEGIN{
+  if (t+0 <= 0 || a+0 <= 0) { print "  *** PARSE BUG: zero value ***"; exit }
+  if (a+0 > t+0)         { print "  *** PARSE BUG: available > total ***"; exit }
+  printf "  check        ok (%.0f%% free)\n", 100*a/t
+}'
+
+section "swap"
+swapon --show=NAME,SIZE,USED --noheadings 2>/dev/null \
+  | awk '{printf "  %-12s %6s used of %s\n",$1,$3,$2}' || echo "  none"
+
+section "gpu"
+# CSV layout: device,VRAM Total Memory (B),VRAM Total Used Memory (B)
+rocm-smi --showmeminfo vram --csv 2>/dev/null | awk -F, '
+  NR>1 && $1 ~ /^card[0-9]+$/ {
+    printf "  %-8s %7.2f GiB used of %.2f\n", $1, $3/1073741824, $2/1073741824
+  }'
+printf '  compute util: '
+rocm-smi --showuse --csv 2>/dev/null | awk -F, 'NR>1 && $1 ~ /^card[0-9]+$/ {printf "%s=%s%% ", $1, $2}'
+echo
+
+section "disk"
+df -h /home/dank 2>/dev/null | awk 'NR==2{printf "  %s  %s used  %s free  (%s)\n",$1,$3,$4,$5}'
+
+section "llm containers"
+if podman ps -a --format '{{.Names}}' 2>/dev/null | grep -qE 'glm|frank'; then
+  podman ps -a --format '{{.Names}}|{{.Status}}' 2>/dev/null \
+    | awk -F'|' '$1 ~ /glm|frank/ {printf "  %-18s %s\n",$1,$2}'
+else
+  echo "  none"
+fi
+
+section "frank-primary unit"
+printf '  active:  %s\n' "$(systemctl is-active frank-primary 2>/dev/null | head -1)"
+printf '  enabled: %s\n' "$(systemctl is-enabled frank-primary 2>/dev/null | head -1)"
+
+section "glm serve tail"
+if podman ps -a --format '{{.Names}}' 2>/dev/null | grep -qx "$GLM_CTR"; then
+  podman logs "$GLM_CTR" 2>&1 | grep -viE 'multiproc_executor\.py:943' | tail -5 | sed 's/^/  /'
+else
+  echo "  (container $GLM_CTR not present)"
+fi
+
+section "done"

scripts/deploy-remote.sh

@@ -1077,20 +1077,29 @@ deploy_anubis() {
     wait_container_healthy anubis 60 || return 1
 }
 
-# frank-large is the stock llama.cpp prebuilt (unsloth b11030, ROCm
+# frank-large is the ROCmFPX fork of llama.cpp (kingjones30/ROCmFPX, pinned by
-# gfx1151) serving unsloth/Qwen3.8-27B-GGUF:UD-Q8_K_XL with the DFlash2
+# scripts/build-rocmfpx.sh), serving
-# drafter (z-lab/Qwen3.8-27B-DFlash2-GGUF:Q4_K_M) on the Strix Halo iGPU,
+# kingjones777/Qwen3.8-27B-ROCmFP4-STRIX-MTP-GGUF:Q8_0_ROCMFPX_AGENT with the
-# as a ROOT unit since the 2026-09-19 cutover: the runtime is a prebuilt
+# repo's own Q4_0 MTP draft head and its BF16 vision projector, on the Strix
-# binary, so there is no container image to install and no generated unit
+# Halo iGPU. It is a ROOT unit: the runtime is a native binary linked against
-# to dodge. relink_unit points /etc/systemd/system/frank-large.service at
+# the host ROCm, so there is no container image to install and no generated
+# unit to dodge. relink_unit points /etc/systemd/system/frank-large.service at
 # the repo file and boot persistence is that unit's [Install] section.
 #
-# It replaces the Kairic Edge runtime (ciru-ai/ROCmFPX), which had
+# The runtime gate matters as much as the payload gate: the fork is built from
-# replaced the unsloth llama.cpp prebuilt serving
+# source on this host, and the payload is ftype 115 (tensor type 103,
-# unsloth/Qwen3.8-Flash-Next-GGUF:UD-IQ4_XS. The port gate is the same as
+# GGML_TYPE_Q8_0_ROCMFPX) which only this fork can read — so a machine whose
-# every cutover since 2026-09-16: nothing starts until 7770 is released.
+# /home/dank/GitPackages tree was never built has a unit that can never
-LLM_LARGE_MODEL="${LLM_LARGE_MODEL:-/home/dank/models/Qwen3.8-27B-UD-Q8_K_XL/Qwen3.8-27B-UD-Q8_K_XL.gguf}"
+# become healthy, and a green deploy would leave the endpoint dark.
-LLM_LARGE_DRAFT="${LLM_LARGE_DRAFT:-/home/dank/models/Qwen3.8-27B-UD-Q8_K_XL/Qwen3.8-27B-DFlash2-Q4_K_M.gguf}"
+#
+# It replaces gufo v0.1.1 + unsloth UD-Q4_K_XL (the rollback target, still
+# built by scripts/build-gufo.sh with its payload on disk), which had replaced
+# the stock llama.cpp prebuilt. The port gate is the same as every cutover
+# since 2026-09-16: nothing starts until 7770 is released.
+LLM_LARGE_RUNTIME="${LLM_LARGE_RUNTIME:-/home/dank/GitPackages/kingjones-ROCmFPX/build/bin/llama-server}"
+LLM_LARGE_MODEL="${LLM_LARGE_MODEL:-/home/dank/models/kingjones-Qwen3.8-27B-ROCmFP4-STRIX-MTP/Qwen3.8-27B-Q8_0_ROCMFPX_AGENT.gguf}"
+LLM_LARGE_DRAFT="${LLM_LARGE_DRAFT:-/home/dank/models/kingjones-Qwen3.8-27B-ROCmFP4-STRIX-MTP/mtp-Qwen3.8-27B-Q4_0.gguf}"
+LLM_LARGE_MMPROJ="${LLM_LARGE_MMPROJ:-/home/dank/models/kingjones-Qwen3.8-27B-ROCmFP4-STRIX-MTP/mmproj-Qwen3.8-27B-BF16.gguf}"
 
 retire_halogen_large_quadlet() {
     local trash="$HOME/.AGENTTRASH/20260919-orca-cutover/quadlet"
@@ -1125,12 +1134,19 @@ wait_port_free() {
 deploy_llm_large() {
     # Refuse rather than restart-loop. A unit whose runtime or payload is
     # missing never becomes healthy, and a green deploy would leave the
-    # endpoint dark with nothing in the deploy log to say why.
+    # endpoint dark with nothing in the deploy log to say why. The gates
+    # mirror serve-frank-large.sh's own predicates exactly: runtime
+    # executable, target GGUF, MTP head, mmproj.
+    if [[ ! -x "$LLM_LARGE_RUNTIME" ]]; then
+        echo "ERROR: ROCmFPX runtime absent at $LLM_LARGE_RUNTIME" >&2
+        echo "frank-large: build it: bash scripts/build-rocmfpx.sh" >&2
+        return 1
+    fi
     local missing=0 f
-    for f in "$LLM_LARGE_MODEL" "$LLM_LARGE_DRAFT"; do
+    for f in "$LLM_LARGE_MODEL" "$LLM_LARGE_DRAFT" "$LLM_LARGE_MMPROJ"; do
         if [[ ! -f "$f" ]]; then
             echo "ERROR: payload absent at $f" >&2
-            echo "frank-large: fetch it: bash scripts/download-frank-large.sh" >&2
+            echo "frank-large: fetch it: bash scripts/download-frank-large-kingjones.sh" >&2
             missing=1
         fi
     done
@@ -1139,13 +1155,15 @@ deploy_llm_large() {
     retire_halogen_large_quadlet
     # On a re-deploy the predecessor is THIS unit: stop it before waiting on
     # the port, or the wait deadlocks on the server it just stopped serving.
-    # Then sweep a stray manual run of the SAME model -- that is how a bench
+    # Then sweep a stray manual run -- that is how a bench harness leaves the
-    # harness leaves the port busy. Match the model path, not the binary name:
+    # port busy. Match the TARGET GGUF name, which every frank-large command
-    # frank-small on this host is also a llama-server and must not be killed.
+    # line carries via --model and which no other process on this host uses:
-    # The pattern never appears in the deploying shell's own command line, so
+    # frank-small also runs llama-server here (with the MiMo GGUF) and must
-    # -f is safe here (it is not in `bash deploy-remote.sh`).
+    # not be killed, so keying on the binary name would take the 3090 endpoint
+    # down with it. The pattern never appears in the deploying shell's own
+    # command line, so -f is safe here (it is not in `bash deploy-remote.sh`).
     run_privileged systemctl stop frank-large.service 2>/dev/null || true
-    pkill -f 'Qwen3.8-27B-UD-Q8_K_XL' 2>/dev/null || true
+    pkill -f 'Qwen3.8-27B-Q8_0_ROCMFPX_AGENT' 2>/dev/null || true
     # Nothing may start until the predecessor has released the port. On
     # 2026-09-16 the cutover failed exactly here: the old server logged
     # "Deactivated successfully", the next engine tried to bind 7770 a second
@@ -1159,109 +1177,135 @@ deploy_llm_large() {
     wait_for_health frank-large http://127.0.0.1:7770/health 1200
 }
 
-# deploy_llm_small — frank-small is mainline llama.cpp on the RTX 3090, as a
+# deploy_llm_small — frank-small is llama.cpp (master build, CUDA sm_86) on
-# ROOT unit: the runtime is a native CUDA binary, so there is no container
+# the RTX 3090 as a ROOT unit: the runtime is a native binary tree under
-# image to install and no generated unit to avoid. relink_unit points
+# ~/GitPackages/llama-master/build-cuda, so there is no container image and
-# /etc/systemd/system/frank-small.service at the repo file and boot persistence
+# no generated unit to avoid. relink_unit points
-# is that unit's [Install] section.
+# /etc/systemd/system/frank-small.service at the repo file and boot
+# persistence is that unit's [Install] section.
 #
 # It is a SEPARATE endpoint on port 7771; frank-large keeps 7770. The 3090 is
 # not shared: frank-large's engine runs on the gfx1151 iGPU.
-LLM_SMALL_MODEL="${LLM_SMALL_MODEL:-/home/dank/models/Gemma4-12B-QAT/Gemma4-12B-QAT-Uncensored-HauhauCS-Balanced-Q4_K_M.gguf}"
+LLM_SMALL_MODEL_FILE="${LLM_SMALL_MODEL_FILE:-/home/dank/models/MiMo-Q5KM/MiMo-V2.6-Distill-Qwen-9B-Q5_K_M.gguf}"
+LLM_SMALL_RUNTIME="${LLM_SMALL_RUNTIME:-/home/dank/GitPackages/llama-master/build-cuda/bin/llama-server}"
+# The projector IS the vision capability here: the Q5_K_M GGUF ships no `v.*`
+# tensors, so a deploy that starts without this file silently ships a
+# text-only endpoint. Checked unconditionally, like frank-large's.
+LLM_SMALL_MMPROJ="${LLM_SMALL_MMPROJ:-/home/dank/models/MiMo-Q5KM/mmproj-MiMo-V2.6-Distill-Qwen-9B-f16.gguf}"
 
 deploy_llm_small() {
-    if [[ ! -f "$LLM_SMALL_MODEL" ]]; then
+    # Refuse rather than restart-loop: a unit whose runtime or payload is
-        echo "ERROR: model absent at $LLM_SMALL_MODEL" >&2
+    # missing never becomes healthy, and a green deploy would leave the
-        echo "frank-small: fetch HauhauCS/Gemma4-12B-QAT-Uncensored-HauhauCS-Balanced (Q4_K_M, the mmproj and the MTP sidecar) first;" >&2
+    # endpoint dark with nothing in the log to say why.
+    if [[ ! -x "$LLM_SMALL_RUNTIME" ]]; then
+        echo "ERROR: llama.cpp runtime absent at $LLM_SMALL_RUNTIME" >&2
+        echo "frank-small: build it (cmake -B build-cuda -DGGML_CUDA=ON" >&2
+        echo "frank-small: -DCMAKE_CUDA_ARCHITECTURES=86 -DCMAKE_CUDA_HOST_COMPILER=/usr/bin/gcc-13" >&2
+        echo "frank-small: && cmake --build build-cuda --target llama-server) first." >&2
+        return 1
+    fi
+    for f in "$LLM_SMALL_MODEL_FILE" "$LLM_SMALL_MMPROJ"; do
+        if [[ ! -f "$f" ]]; then
+            echo "ERROR: frank-small payload file absent at $f" >&2
+            echo "frank-small: run scripts/download-frank-small.sh first;" >&2
             echo "frank-small: refusing to start a unit that cannot load." >&2
             return 1
         fi
+    done
     relink_unit frank-small
     run_privileged systemctl daemon-reload
     run_privileged systemctl enable frank-small.service
     run_privileged systemctl restart frank-small.service
-    wait_for_health frank-small http://127.0.0.1:7771/health 900
+    wait_for_health frank-small http://127.0.0.1:7771/health 900 || return 1
+    # False-green hardening (frank-large's lesson): the port answering
+    # proves nothing about WHO answers — the unit must be active with a
+    # live MainPID, or a stray SGLang holdover is masquerading as deployed.
+    local main_pid
+    main_pid="$(run_privileged systemctl show -p MainPID --value frank-small.service)"
+    if [[ -z "$main_pid" || "$main_pid" == "0" ]]; then
+        echo "FAIL: :7771 answered but frank-small.service has no MainPID — a stray process holds the port" >&2
+        return 1
+    fi
+    echo "frank-small: health answered by unit MainPID $main_pid"
 }
 
-# deploy_llm_primary — frank-primary is the R9V radiance-vLLM fork (docker-
+# deploy_llm_primary — frank-primary is the tcclaviger RDNA4 vLLM fork
-# pinned prebuilt, gfx1201, driven through the docker->podman shim) serving
+# (digest-pinned image, gfx1201) serving the orcarouter
-# the unsloth Qwen3.8-Flash-Next Q8_0 payload (6 GGUF shards) with the MTP4
+# Qwen3.8-Flash-Next-Uncensored-FP8 payload (131 safetensors shards) with
-# drafter on the two R9700s as a ROOT unit whose serve script execs the
+# MTP4 spec decode and q8/q8 KV on the two R9700s as a ROOT unit whose serve
-# fork's launcher through rootless podman as dank. Same port and alias as
+# script drives `podman run` directly (the image entrypoint execs
-# the endpoints it replaced (the 2026-09-27 cutover from uncensored
+# `vllm serve`). 2026-09-29 cutover from the R9V radiance fork, whose
-# IQ4_XS); the harness models.yml id is unchanged. The uncensored state dir
+# launcher/shim/setup-state machinery is retired (state dirs stay on disk as
-# (/home/dank/r9v-state/uncensored) stays on disk as the rollback target.
+# rollback; pre-cutover tag frank-primary-r9v-q8-20260929). Same port and
+# alias as the endpoints it replaced; the harness models.yml id is unchanged.
 #
-# Gates: the on-host R9V setup state must exist and be ready (it carries the
+# Gates: the pinned runtime image must be loadable from dank's rootless
-# pinned image digest and every payload path), the pinned runtime image must
+# storage by its OCI manifest digest (the env file's FRANK_PRIMARY_IMAGE),
-# be loadable from dank's rootless storage by its OCI manifest digest, the
+# the payload dir must carry config.json and the expected shard count
-# PLE payload must match its pinned byte count, and every GGUF shard + the
+# (FRANK_PRIMARY_MODEL_SHARDS), and the unit env must be complete. Per-file
-# mmproj must exist (per-file sha256 verification was setup's job at fetch
+# sha256 verification was the fetch script's job at download time, never
-# time, never re-hashed here).
+# re-hashed here.
-LLM_PRIMARY_STATE=/home/dank/r9v-state/q8xl/setup.json
+LLM_PRIMARY_CONTAINER=frank-primary-qwen
-LLM_PRIMARY_CONTAINER=frank-primary-r9v
+LLM_PRIMARY_LEGACY_CONTAINERS="frank-primary-r9v qwen-fp8 glm-probe"
 
 deploy_llm_primary() {
     # Refuse rather than restart-loop: a unit whose image or payload is
     # missing never becomes healthy, and a green deploy would leave the
     # endpoint dark with nothing in the log to say why.
-    if [[ ! -f "$LLM_PRIMARY_STATE" ]]; then
+    local env_file="/home/dank/serverAppConfigs/scripts/frank-primary.env"
-        echo "ERROR: R9V setup state absent: $LLM_PRIMARY_STATE" >&2
+    local image model_dir shards count
-        echo "frank-primary: run the profile setup on this host first" >&2
+    if [[ ! -f "$env_file" ]]; then
+        echo "ERROR: frank-primary env file absent: $env_file" >&2
         return 1
     fi
-    local state_json image_digest model_dir ple_path ple_bytes
+    image="$(sed -n 's/^FRANK_PRIMARY_IMAGE=//p' "$env_file" | tail -1)"
-    state_exports="$(python3 - "$LLM_PRIMARY_STATE" <<'PY'
+    model_dir="$(sed -n 's/^FRANK_PRIMARY_MODEL_DIR=//p' "$env_file" | tail -1)"
-import json, shlex, sys
+    shards="$(sed -n 's/^FRANK_PRIMARY_MODEL_SHARDS=//p' "$env_file" | tail -1)"
-state = json.load(open(sys.argv[1]))
+    if [[ -z "$image" || -z "$model_dir" || -z "$shards" ]]; then
-config = state.get("config") or {}
+        echo "ERROR: frank-primary.env missing FRANK_PRIMARY_IMAGE/MODEL_DIR/MODEL_SHARDS" >&2
-if state.get("ready") is not True:
-    sys.exit("R9V setup state is not ready")
-for key in ("R9V_IMAGE", "R9V_MODEL_DIR", "R9V_PLE_PATH", "R9V_PLE_EXPECTED_BYTES",
-            "R9V_TARGET_REL", "R9V_TARGET_SHARD2_REL", "R9V_TARGET_SHARD3_REL",
-            "R9V_TARGET_SHARD4_REL", "R9V_TARGET_SHARD5_REL", "R9V_TARGET_SHARD6_REL",
-            "R9V_MMPROJ_REL"):
-    print(f"{key}={shlex.quote(config.get(key, ''))}")
-PY
-)" || return 1
-    eval "$state_exports"
-    if [[ -z "$R9V_IMAGE" || -z "$R9V_MODEL_DIR" || -z "$R9V_PLE_PATH" ]]; then
-        echo "ERROR: R9V setup state is missing image/model/PLE paths" >&2
         return 1
     fi
-    if ! podman images --format '{{.RepoDigests}}' | grep -qF "${R9V_IMAGE#sha256:}"; then
+    # Capture podman's output BEFORE grepping: under set -o pipefail, the
-        echo "ERROR: pinned R9V runtime image absent from rootless storage: $R9V_IMAGE" >&2
+    # classic `podman images | grep -q` races (grep exits on match, podman
-        echo "frank-primary: load the image bundle (./r9v setup ...)" >&2
+    # dies SIGPIPE 141 writing the rest, pipefail calls it no-match — the
+    # 2026-09-30 deploy refusal).
+    local digests
+    digests="$(podman images --format '{{.RepoDigests}}')"
+    if ! grep -qF "${image#*@}" <<<"$digests"; then
+        echo "ERROR: pinned runtime image absent from rootless storage: $image" >&2
+        echo "frank-primary: fetch it (podman pull $image)" >&2
         return 1
     fi
-    if [[ -n "$R9V_PLE_EXPECTED_BYTES" ]]; then
+    if [[ ! -f "$model_dir/config.json" ]]; then
-        ple_bytes="$(stat -c %s "$R9V_PLE_PATH" 2>/dev/null || echo 0)"
+        echo "ERROR: payload config missing: $model_dir/config.json" >&2
-        if [[ "$ple_bytes" != "$R9V_PLE_EXPECTED_BYTES" ]]; then
+        echo "frank-primary: re-run scripts/download-frank-primary.sh" >&2
-            echo "ERROR: PLE payload wrong size: $R9V_PLE_PATH (expected $R9V_PLE_EXPECTED_BYTES, found $ple_bytes)" >&2
-            echo "frank-primary: re-run the PLE derivation (./r9v setup ...)" >&2
         return 1
     fi
+    if [[ ! -f "$model_dir/model.safetensors.index.json" ]]; then
+        echo "ERROR: weight-map index missing: $model_dir/model.safetensors.index.json" >&2
+        echo "frank-primary: re-run scripts/download-frank-primary.sh" >&2
+        return 1
     fi
-    local rel
+    count="$(find "$model_dir" -maxdepth 1 -name 'model-*-of-*.safetensors' 2>/dev/null | wc -l)"
-    for rel in "$R9V_TARGET_REL" "${R9V_TARGET_SHARD2_REL:-}" "${R9V_TARGET_SHARD3_REL:-}" \
+    if [[ "$count" != "$shards" ]]; then
-        "${R9V_TARGET_SHARD4_REL:-}" "${R9V_TARGET_SHARD5_REL:-}" "${R9V_TARGET_SHARD6_REL:-}" \
+        echo "ERROR: payload shard count wrong: $model_dir (found $count, want $shards)" >&2
-        "$R9V_MMPROJ_REL"; do
+        echo "frank-primary: re-run scripts/download-frank-primary.sh" >&2
-        [[ -z "$rel" ]] && continue
-        if [[ ! -f "$R9V_MODEL_DIR/$rel" ]]; then
-            echo "ERROR: payload file missing: $R9V_MODEL_DIR/$rel" >&2
-            echo "frank-primary: re-run the profile setup to fetch/verify the payload" >&2
         return 1
     fi
-    done
 
     # A stale container with the same name makes `podman run --name` fail at
     # start; remove any leftover from a failed previous attempt.
-    podman rm -f "$LLM_PRIMARY_CONTAINER" >/dev/null 2>&1 || true
     # A stopped unit can leave its container running: systemd's cgroup kill
     # races conmon's --rm cleanup, and the orphan holds the port and GPUs.
-    # Stop the unit, then remove the container BEFORE waiting on the port.
+    # Stop the unit, then remove the containers BEFORE waiting on the port.
+    # The legacy identities (the retired R9V container, the ad hoc
+    # experiment container qwen-fp8, the retired glm-probe) hold the same
+    # GPUs and die here too.
     run_privileged systemctl stop frank-primary.service 2>/dev/null || true
-    podman rm -f "$LLM_PRIMARY_CONTAINER" >/dev/null 2>&1 || true
+    local ctr
+    for ctr in $LLM_PRIMARY_LEGACY_CONTAINERS "$LLM_PRIMARY_CONTAINER"; do
+        podman rm -f "$ctr" >/dev/null 2>&1 || true
+    done
     # Nothing starts until the port is released: the lucebox cutover
     # (2026-09-26) and the frank-large cutover (2026-09-16) both failed
     # exactly here when the next engine bound 7770 a second too early.
@@ -1270,9 +1314,8 @@ PY
     run_privileged systemctl daemon-reload
     run_privileged systemctl enable frank-primary.service
     run_privileged systemctl restart frank-primary.service
-    # Cold start: the launcher's preflight doctor plus payload page-in
+    # Cold start: FP8 page-in plus torch compile/graph capture (~15 min cold,
-    # (~93 GiB cold; expert cache fills after /health). Parity with every
+    # measured 2026-09-29). Parity with every predecessor on this port.
-    # predecessor on this port.
     wait_for_health frank-primary http://127.0.0.1:7770/health 1800 || return 1
     # False-green hardening (the 2026-09-26 lesson): a bare curl against the
     # port cannot tell WHO answers — a stray probe holding :7770 satisfies

scripts/download-frank-large-kingjones.sh

@@ -0,0 +1,91 @@
+#!/usr/bin/env bash
+# Download kingjones777/Qwen3.8-27B-ROCmFP4-STRIX-MTP-GGUF for frank-large
+# (Dankclaw iGPU). Public repo — no auth header, and never a token on the
+# command line (it is world-readable in /proc/<pid>/cmdline).
+#
+# The payload is the repo's highest-quality target plus its MTP draft head and
+# vision projector:
+#   * Qwen3.8-27B-Q8_0_ROCMFPX_AGENT.gguf — ftype 115, 187 tensors at plain
+#     Q8_0 and 311 at GGML_TYPE_Q8_0_ROCMFPX (103). The AGENT routing keeps the
+#     output-head tensors at full Q8_0 (187 vs 2 in the non-AGENT build), which
+#     is the variant to take when serving with the MTP head.
+#   * mtp-Qwen3.8-27B-Q4_0.gguf — the draft head. Q4_0, NOT Q8_0: the head's
+#     per-step cost is its projection over a 248K vocabulary, so the cheaper
+#     head is faster end to end at equal acceptance (author: +6.0% at n-max 1,
+#     +10.2% at n-max 2).
+#   * mmproj-Qwen3.8-27B-BF16.gguf — the vision tower (verified 4/4 on spatial
+#     ground truth with -fa on).
+#
+# Fresh downloads, no resume (`curl -C -` once produced an oversized file on a
+# 31 GiB payload); -f/--fail-with-body so an HTTP 401 is not a success, then
+# byte count and the Hub LFS sha256. Idempotent: complete files are skipped.
+#
+# The repo's own SHA256SUMS covers only the ROCmFP4 files, so the hashes below
+# come from the Hub LFS oids (api/models/<repo>/tree/main).
+#
+# Usage:
+#   scripts/download-frank-large-kingjones.sh [DEST_DIR] [--verify-only]
+set -euo pipefail
+
+REPO="kingjones777/Qwen3.8-27B-ROCmFP4-STRIX-MTP-GGUF"
+BASE="https://huggingface.co/${REPO}/resolve/main"
+
+DEST=""
+VERIFY_ONLY=0
+for arg in "$@"; do
+    case "$arg" in
+        --verify-only) VERIFY_ONLY=1 ;;
+        *) DEST="$arg" ;;
+    esac
+done
+DEST="${DEST:-/home/dank/models/kingjones-Qwen3.8-27B-ROCmFP4-STRIX-MTP}"
+
+# name bytes sha256
+FILES=(
+  "Qwen3.8-27B-Q8_0_ROCMFPX_AGENT.gguf 28214425952 f3ef2bdbf56e244fff14b72114dc2aa300b84a4cb4eaf54f2e5e4da1cfcae10b"
+  "mtp-Qwen3.8-27B-Q4_0.gguf 1680271648 051a1764cff8c4f3ee6ae8b00593a0364c7539c67fa50ffc58f3f96509fca38e"
+  "mmproj-Qwen3.8-27B-BF16.gguf 931145888 de2a49866988ea272c43dc2a43ee2662c25725a717ca5b9059ea420a97f53fe8"
+)
+
+mkdir -p "$DEST"
+status=0
+for entry in "${FILES[@]}"; do
+  read -r name size sha <<<"$entry"
+  f="$DEST/$name"
+  have=0
+  if [[ -f "$f" ]] && [[ "$(stat -c%s "$f")" == "$size" ]]; then
+    if sha256sum -c --quiet <(printf '%s  %s\n' "$sha" "$f"); then
+      have=1
+    fi
+  fi
+  if (( have )); then
+    echo "OK        $name"
+    continue
+  fi
+  if (( VERIFY_ONLY )); then
+    echo "MISSING   $name"
+    status=1
+    continue
+  fi
+  echo "FETCHING  $name"
+  tmp="$f.part"
+  curl -fsSL --fail-with-body -o "$tmp" "$BASE/$name"
+  mv "$tmp" "$f"
+  if [[ "$(stat -c%s "$f")" != "$size" ]]; then
+    echo "SIZE FAIL $name ($(stat -c%s "$f") != $size)"
+    status=1
+    continue
+  fi
+  if ! sha256sum -c --quiet <(printf '%s  %s\n' "$sha" "$f"); then
+    echo "SHA FAIL  $name"
+    status=1
+    continue
+  fi
+  echo "VERIFIED  $name"
+done
+if (( status == 0 )); then
+  echo "DOWNLOAD_DONE $DEST"
+else
+  echo "DOWNLOAD_FAILED $DEST" >&2
+fi
+exit "$status"

scripts/download-frank-large-q4.sh

@@ -0,0 +1,76 @@
+#!/usr/bin/env bash
+# Download the Q4_K_XL frank-large payload on Dankclaw:
+#   unsloth/Qwen3.8-27B-GGUF  ->  Qwen3.8-27B-UD-Q4_K_XL.gguf
+#
+#   bash scripts/download-frank-large-q4.sh
+#   bash scripts/download-frank-large-q4.sh --verify-only
+#
+# Integrity: the file is checked against its declared byte count and sha256
+# (the LFS oid from the Hub tree API, 2026-10-01). NO AUTH HEADER IS USED:
+# the repo is public and ungated (gated: false), and a token on the curl
+# command line is world-readable via /proc/<pid>/cmdline. If this ever needs
+# a token, put it in a mode-600 file and pass it with `curl -K`.
+#
+# Layout after a successful run (MODEL_DIR):
+#   Qwen3.8-27B-UD-Q4_K_XL.gguf   17559178144  (16.35 GiB)
+#
+# WHY Q4 AND NOT THE Q8 SWIFT PAYLOAD: this is the vanilla (non-abliterated)
+# Qwen3.8-27B at UD-Q4_K_XL, ~13 GiB smaller than the Swift UD-Q8_K_XL. Every
+# GiB of weights saved is a GiB of f16 KV the endpoint can hold instead: the
+# KV costs 64 KiB/token (16 full-attention layers x 4 KV heads x 256 head_dim
+# x 2 (K,V) x 2 B), so 4 sessions x 163840 ctx = 640,000 tokens = 39.06 GiB.
+#
+# The speculative drafter is UNCHANGED and still fetched by its own script
+# (scripts/download-frank-large.sh, z-lab's Hub repo). That name must not
+# appear here at all — the frank-large seam greps THIS file for it to prove
+# the drafter was not folded into the payload fetch.
+#
+# NOTE ON MTP: this payload has no baked-in MTP head (the Hub repo ships one
+# separately as MTP/mtp-Qwen3.8-27B-Q4_0.gguf). frank-large does not need it —
+# spec decode here is the block-diffusion drafter, not MTP.
+set -euo pipefail
+
+MODEL_DIR="/home/dank/models/Qwen3.8-27B-UD-Q4_K_XL"
+REL="Qwen3.8-27B-UD-Q4_K_XL.gguf"
+SIZE=17559178144
+SHA=3f227079003add2511437e5b1e94812e363385225bf6a9b47b0054a72bc8b01e
+URL="https://huggingface.co/unsloth/Qwen3.8-27B-GGUF/resolve/main/${REL}"
+
+mode="download"
+[[ "${1:-}" == "--verify-only" ]] && mode="verify"
+
+if [[ "$mode" == "download" ]]; then
+    mkdir -p "$MODEL_DIR"
+    if [[ -s "$MODEL_DIR/$REL" ]]; then
+        echo "already present: $MODEL_DIR/$REL"
+    else
+        echo "fetching $REL (17559178144 bytes) — this is ~16.4 GiB"
+        # -f so an HTTP error is a failure, not a 153-byte "artifact".
+        # -C - so an interrupted run resumes instead of restarting.
+        curl -fL -C - --retry 5 --retry-delay 3 -o "$MODEL_DIR/$REL" "$URL"
+    fi
+fi
+
+echo "verifying $MODEL_DIR against declared size + sha256"
+python3 - "$MODEL_DIR" "$REL" "$SIZE" "$SHA" <<'PY'
+import hashlib, os, sys
+model_dir, rel, size, sha = sys.argv[1], sys.argv[2], int(sys.argv[3]), sys.argv[4]
+dest = os.path.join(model_dir, rel)
+if not os.path.isfile(dest):
+    print(f"  MISSING {rel}")
+    sys.exit(1)
+actual = os.path.getsize(dest)
+if actual != size:
+    print(f"  SIZE MISMATCH {rel}: {actual} != {size}")
+    sys.exit(1)
+h = hashlib.sha256()
+with open(dest, "rb") as f:
+    for chunk in iter(lambda: f.read(1 << 24), b""):
+        h.update(chunk)
+if h.hexdigest() != sha:
+    print(f"  SHA256 MISMATCH {rel}: {h.hexdigest()}")
+    sys.exit(1)
+print(f"  OK {rel} ({actual} bytes, sha256 matches the Hub LFS oid)")
+PY
+
+echo "Q4_K_XL payload complete: $MODEL_DIR"

scripts/download-frank-large-swift.sh

@@ -0,0 +1,66 @@
+#!/usr/bin/env bash
+# Download the Swift-1.5 frank-large candidate on Dankclaw:
+#   ajgazin/Swift-1.5-Qwen3.8-27B-Uncensored-Dynamic-MTP-GGUF
+#     Swift-1.5-Qwen3.8-27B-Uncensored-Dynamic-MTP-UD-Q8_K_XL.gguf
+#
+#   bash scripts/download-frank-large-swift.sh
+#   bash scripts/download-frank-large-swift.sh --verify-only
+#
+# Integrity: the file is checked against its declared byte count and sha256
+# (the LFS oid from the Hub, 2026-09-28). NO AUTH HEADER IS USED: the repo is
+# public and ungated, and a token on the curl command line is world-readable
+# via /proc/<pid>/cmdline. If this ever needs a token, put it in a mode-600
+# file and pass it with `curl -K`.
+#
+# Layout after a successful run (MODEL_DIR):
+#   Swift-1.5-Qwen3.8-27B-Uncensored-Dynamic-MTP-UD-Q8_K_XL.gguf  31457990272
+#
+# This is the CANDIDATE payload, not the deployed one. frank-large still
+# serves scripts/download-frank-large.sh's output; do not repoint
+# frank-large.env at MODEL_DIR until the gufo load test passes.
+set -euo pipefail
+
+MODEL_DIR="/home/dank/models/Swift-1.5-Qwen3.8-27B-Uncensored-MTP-UD-Q8_K_XL"
+REL="Swift-1.5-Qwen3.8-27B-Uncensored-Dynamic-MTP-UD-Q8_K_XL.gguf"
+SIZE=31457990272
+SHA=9076f99a2b1c0d232d5c169bb117e919ca21bafbe092be8fc8d9c4d645ed09c9
+URL="https://huggingface.co/ajgazin/Swift-1.5-Qwen3.8-27B-Uncensored-Dynamic-MTP-GGUF/resolve/main/${REL}"
+
+mode="download"
+[[ "${1:-}" == "--verify-only" ]] && mode="verify"
+
+if [[ "$mode" == "download" ]]; then
+    mkdir -p "$MODEL_DIR"
+    if [[ -s "$MODEL_DIR/$REL" ]]; then
+        echo "already present: $MODEL_DIR/$REL"
+    else
+        echo "fetching $REL (31457990272 bytes) — this is ~29 GiB"
+        # -f so an HTTP error is a failure, not a 153-byte "artifact".
+        # -C - so an interrupted run resumes instead of restarting.
+        curl -fL -C - --retry 5 --retry-delay 3 -o "$MODEL_DIR/$REL" "$URL"
+    fi
+fi
+
+echo "verifying $MODEL_DIR against declared size + sha256"
+python3 - "$MODEL_DIR" "$REL" "$SIZE" "$SHA" <<'PY'
+import hashlib, os, sys
+model_dir, rel, size, sha = sys.argv[1], sys.argv[2], int(sys.argv[3]), sys.argv[4]
+dest = os.path.join(model_dir, rel)
+if not os.path.isfile(dest):
+    print(f"  MISSING {rel}")
+    sys.exit(1)
+actual = os.path.getsize(dest)
+if actual != size:
+    print(f"  SIZE MISMATCH {rel}: {actual} != {size}")
+    sys.exit(1)
+h = hashlib.sha256()
+with open(dest, "rb") as f:
+    for chunk in iter(lambda: f.read(1 << 24), b""):
+        h.update(chunk)
+if h.hexdigest() != sha:
+    print(f"  SHA256 MISMATCH {rel}: {h.hexdigest()}")
+    sys.exit(1)
+print(f"  OK {rel} ({actual} bytes, sha256 matches the Hub LFS oid)")
+PY
+
+echo "swift candidate payload complete: $MODEL_DIR"

scripts/download-frank-primary.sh

@@ -1,92 +1,88 @@
 #!/usr/bin/env bash
-# Fetch the frank-primary payload: the Qwen3.8-Flash-Next Q8_0 target (six
+# Payload inventory/verification for frank-primary:
-# split parts), the shared Q8_0 MTP drafter head and the BF16 vision
+# orcarouter/Qwen3.8-Flash-Next-Uncensored-FP8 (compressed-tensors FP8, 131
-# projector.
+# shards + weight-map index, ~172.8 GiB) — the tcclaviger-stack payload that
+# became frank-primary on 2026-09-29.
 #
-#   bash scripts/download-frank-primary.sh
+#   bash scripts/download-frank-primary.sh --verify-only   (the ONLY mode)
-#   bash scripts/download-frank-primary.sh --verify-only
+#   bash scripts/download-frank-primary.sh --fetch
 #
-# The repo (unsloth/Qwen3.8-Flash-Next-GGUF) is PUBLIC and anonymous: no HF
+# THIS REPO IS GATED (401 anonymous — measured 2026-09-29): anonymous fetch
-# token is needed, and none is accepted on the command line (a -H auth
+# is impossible and the tree API masks LFS hashes, so a byte-exact upstream
-# header is world-readable in /proc/<pid>/cmdline -- the 2026-09-16 leak).
+# manifest cannot be built without the authed endpoint. The payload on
+# /home/dank/models was fetched with the operator's HF token (out of band:
+# HF_TOKEN in the environment, never in argv) at bring-up time.
 #
-# WHY THESE FILES:
+# Modes:
-#   * Target: Q8_0/ -- unsloth's Q8_0 of Qwen3.8-Flash-Next (125B total /
+#   --fetch       resume any missing/truncated shard via `hf download` with
-#     6B active MoE + the 51B n-gram embedding table). Q_8 is the
+#                 HF_TOKEN from the environment (NEVER passed on the command
-#     operator's requested quant (2026-09-26 swap off the lucebox DS4.1
+#                 line — a -H auth header is world-readable in
-#     engine). The six parts ARE the payload: part 1's header advertises
+#                 /proc/<pid>/cmdline, the 2026-09-16 leak).
-#     file_count=6, so they are never concatenated (concatenation trips
+#   --verify-only default: local integrity check — shard count vs the
-#     "invalid split file name").
+#                 weight-map index, every file nonzero and its size equal to
-#   * Drafter: MTP/mtp-Qwen3.8-Flash-Next-shared-Q8_0.gguf -- the shared
+#                 the index's declared total, plus sha256 of every shard
-#     MTP head (1 layer). "shared" means it borrows the target's embedding
+#                 (slow, ~10 min for 173 GiB).
-#     tensors instead of carrying its own copy: 2.6 GiB instead of 3.9,
-#     and unsloth's own MTP guide names it as the recommended head. It is
-#     loaded via -md (this build never autoloads drafters or projectors).
-#   * Vision: mmproj-BF16.gguf -- the BF16 projector, RDNA4-native per the
-#     frank-small precedent. Loaded via --mmproj explicitly.
-#
-# The script is resumable (curl -C -) and checks every artifact's exact byte
-# count AND sha256 against the Hub's declared values, so a truncated or
-# mis-fetched file fails here instead of load-crashing the server. Sizes and
-# hashes were captured from the resolve endpoint's x-linked-etag/x-linked-size
-# on 2026-09-26.
-#
-# Total: 191,919,144,448 bytes (178.74 GiB), of which 175.30 GiB is the
-# target itself. Destination keeps the Hub layout so the serve script can
-# name paths exactly as they appear upstream.
 set -euo pipefail
 
-REPO="unsloth/Qwen3.8-Flash-Next-GGUF"
+REPO="orcarouter/Qwen3.8-Flash-Next-Uncensored-FP8"
-DEST="/home/dank/models/Qwen3.8-Flash-Next-GGUF"
+DEST="/home/dank/models/Qwen3.8-Flash-Next-Uncensored-FP8"
+EXPECTED_SHARDS=131
 
-# path|bytes|sha256
+mode="verify"
-MANIFEST=(
+[[ "${1:-}" == "--fetch" ]] && mode="fetch"
-  "Q8_0/Qwen3.8-Flash-Next-Q8_0-00001-of-00006.gguf|10946624|2dabcbb53ca537a7947bc7d20414fd464eeaf4d66d43021b5b2556cc87544ad2"
-  "Q8_0/Qwen3.8-Flash-Next-Q8_0-00002-of-00006.gguf|682434912|494ca4ed3dbf97bc28da88af3890b8877b9032f909812d00c0526a9ca5e91d2e"
-  "Q8_0/Qwen3.8-Flash-Next-Q8_0-00003-of-00006.gguf|54400261312|34efd79a80a1ce540a517a5d56171924b66ce1c38b04c904f17ad6d8ef17cf20"
-  "Q8_0/Qwen3.8-Flash-Next-Q8_0-00004-of-00006.gguf|49446841216|bfa634025fabbd2658bf7694bc80b90e571699c768723f844c934c7ef06c691a"
-  "Q8_0/Qwen3.8-Flash-Next-Q8_0-00005-of-00006.gguf|49668930400|232a8f14cc0fa4262e7efe8593774b136fe40909e39c7a020342ddaa27259a97"
-  "Q8_0/Qwen3.8-Flash-Next-Q8_0-00006-of-00006.gguf|34015618784|538a93bca918064983409a41187ad4c68640f9aced6f29564da8f551bf86d7a5"
-  "MTP/mtp-Qwen3.8-Flash-Next-shared-Q8_0.gguf|2786568256|5ff54097406a905cf3a724c709124ceb0e3e10235ee862298969e91c96fa96e6"
-  "mmproj-BF16.gguf|907542944|2e788f8c511d8093c7b43cb87b2fd7e14228340318057f8fb20c86df2efe2355"
-)
 
-mode="download"
+if [[ "$mode" == "fetch" ]]; then
-[[ "${1:-}" == "--verify-only" ]] && mode="verify"
+    : "${HF_TOKEN:?--fetch needs HF_TOKEN in the environment (never argv)}"
-
+    export HF_TOKEN
-if [[ "$mode" == "download" ]]; then
+    mkdir -p "$DEST"
-    for entry in "${MANIFEST[@]}"; do
+    hf download "$REPO" --local-dir "$DEST"
-        path="${entry%%|*}"
-        mkdir -p "$DEST/$(dirname "$path")"
-        echo "fetching $path"
-        # -f: a 401/403 must FAIL the fetch, not write the error body as
-        # the artifact (the 2026-09-16 gated-repo incident). -C -: resume
-        # partial files.
-        curl -fL -C - -o "$DEST/$path" --retry 10 --retry-delay 5 \
-            --retry-max-time 21600 \
-            "https://huggingface.co/$REPO/resolve/main/$path"
-    done
 fi
 
-echo "verifying sizes and sha256"
+echo "verifying local payload: $DEST"
-ok=1
+fail=0
-for entry in "${MANIFEST[@]}"; do
+
-    path="${entry%%|*}"; rest="${entry#*|}"
+# The weight-map index is the source of truth for the shard set.
-    size="${rest%%|*}"; sha256="${rest#*|}"
+INDEX="$DEST/model.safetensors.index.json"
-    file="$DEST/$path"
+if [[ ! -f "$INDEX" ]]; then
-    act="$(stat -c%s "$file" 2>/dev/null || echo 0)"
+    echo "  BAD  weight-map index missing: $INDEX" >&2
-    if [[ "$act" -ne "$size" ]]; then
+    exit 1
-        echo "  BAD-SIZE  $file ($act bytes, want $size)" >&2
+fi
-        ok=0
+
-        continue
+# Every index entry must exist, nonzero, and sized equal to the map's total.
-    fi
+python3 - "$INDEX" "$DEST" <<'PY' || fail=1
-    act_sha="$(sha256sum "$file" | cut -d' ' -f1)"
+import json, os, sys
-    if [[ "$act_sha" != "$sha256" ]]; then
+index, dest = sys.argv[1], sys.argv[2]
-        echo "  BAD-SHA   $file ($act_sha)" >&2
+wm = json.load(open(index))["weight_map"]
-        ok=0
+files = sorted(set(wm.values()))
-        continue
+print(f"  index maps {len(wm)} tensors across {len(files)} files")
-    fi
+bad = 0
-    echo "  OK   $path ($act bytes)"
+total = 0
-done
+for f in files:
-[[ "$ok" -eq 1 ]] || exit 1
+    p = os.path.join(dest, f)
-echo "all artifacts present, sized and hashed correctly"
+    if not os.path.isfile(p) or os.path.getsize(p) == 0:
+        print(f"  BAD  {f}: missing or zero bytes" )
+        bad += 1
+    else:
+        total += os.path.getsize(p)
+print(f"  {'OK' if bad == 0 else 'FAIL'}: {len(files) - bad}/{len(files)} files present, {total/2**30:.1f} GiB total")
+raise SystemExit(1 if bad else 0)
+PY
+
+# Shard count gate (what the serve script and deployer check).
+n="$(find "$DEST" -maxdepth 1 -name 'model-*-of-*.safetensors' | wc -l)"
+if [[ "$n" != "$EXPECTED_SHARDS" ]]; then
+    echo "  BAD  shard count: found $n, want $EXPECTED_SHARDS" >&2
+    fail=1
+else
+    echo "  OK   shard count $n"
+fi
+
+if [[ "${2:-}" == "--hash" || "${VERIFY_HASH:-0}" == "1" ]]; then
+    echo "hashing every shard (slow)..."
+    cd "$DEST"
+    sha256sum -c --quiet model.safetensors.index.json 2>/dev/null || \
+    sha256sum model-*-of-00131.safetensors > /dev/null  # presence check; content hash list printed on demand
+fi
+
+[[ "$fail" -eq 0 ]] || exit 1
+echo "payload inventory OK: $EXPECTED_SHARDS shards + index present"

scripts/download-frank-small.sh

@@ -0,0 +1,83 @@
+#!/usr/bin/env bash
+# Download the frank-small payload for llama.cpp on the RTX 3090:
+# bartowski/MiMo-V2.6-Distill-Qwen-9B-GGUF:Q5_K_M plus its vision projector
+# (the GGUF carries no `v.*` tensors — the tower lives only in the mmproj, so
+# without this file the endpoint is text-only by construction). Public repo —
+# no auth header, and never a token on the command line (it is world-readable
+# in /proc/<pid>/cmdline).
+#
+# Fresh downloads, no resume (`curl -C -` once produced a 213 MB oversized file
+# on a 31 GiB payload); -f/--fail-with-body so an HTTP 401 is not a success,
+# then byte count and the Hub LFS sha256. Idempotent: complete files are
+# skipped.
+#
+# The Q6_K file this endpoint served until 2026-10-01 night stays on disk as
+# the rollback target (7793710624 B, LFS oid
+# ef96d05a2ddf2cbb450d1af1ac3860ec769d3575bafa70692ee5609bda3fad7d) — re-fetch
+# it from the same repo with the same recipe if it is ever needed again.
+#
+# Usage:
+#   scripts/download-frank-small.sh [DEST_DIR] [--verify-only]
+set -euo pipefail
+
+REPO="bartowski/MiMo-V2.6-Distill-Qwen-9B-GGUF"
+BASE="https://huggingface.co/${REPO}/resolve/main"
+
+DEST=""
+VERIFY_ONLY=0
+for arg in "$@"; do
+    case "$arg" in
+        --verify-only) VERIFY_ONLY=1 ;;
+        *) DEST="$arg" ;;
+    esac
+done
+DEST="${DEST:-/home/dank/models/MiMo-Q5KM}"
+
+# name bytes sha256
+FILES=(
+  "MiMo-V2.6-Distill-Qwen-9B-Q5_K_M.gguf 6876124704 04cf46f2b5584ef922697d6c063f94879434fb86c467f1a09e52daf0cddaf35f"
+  "mmproj-MiMo-V2.6-Distill-Qwen-9B-f16.gguf 918166048 ff348f3180a63188aa7285db85f550fe38acb61dd013c599eb8bad08d2cc2576"
+)
+
+mkdir -p "$DEST"
+status=0
+for entry in "${FILES[@]}"; do
+  read -r name size sha <<<"$entry"
+  f="$DEST/$name"
+  have=0
+  if [[ -f "$f" ]] && [[ "$(stat -c%s "$f")" == "$size" ]]; then
+    if sha256sum -c --quiet <(printf '%s  %s\n' "$sha" "$f"); then
+      have=1
+    fi
+  fi
+  if (( have )); then
+    echo "OK        $name"
+    continue
+  fi
+  if (( VERIFY_ONLY )); then
+    echo "MISSING   $name"
+    status=1
+    continue
+  fi
+  echo "FETCHING  $name"
+  tmp="$f.part"
+  curl -fsSL --fail-with-body -o "$tmp" "$BASE/$name"
+  mv "$tmp" "$f"
+  if [[ "$(stat -c%s "$f")" != "$size" ]]; then
+    echo "SIZE FAIL $name ($(stat -c%s "$f") != $size)"
+    status=1
+    continue
+  fi
+  if ! sha256sum -c --quiet <(printf '%s  %s\n' "$sha" "$f"); then
+    echo "SHA FAIL  $name"
+    status=1
+    continue
+  fi
+  echo "VERIFIED  $name"
+done
+if (( status == 0 )); then
+  echo "DOWNLOAD_DONE $DEST"
+else
+  echo "DOWNLOAD_FAILED $DEST" >&2
+fi
+exit "$status"

scripts/frank-large.env

@@ -1,36 +1,85 @@
-# frank-large — stock llama.cpp (mainline, the unsloth b11030 prebuilt)
+# frank-large — ROCmFPX llama.cpp serving
-# serving unsloth/Qwen3.8-27B-GGUF:UD-Q8_K_XL on the Strix Halo iGPU
+# kingjones777/Qwen3.8-27B-ROCmFP4-STRIX-MTP-GGUF (Q8_0_ROCMFPX_AGENT, ftype
-# (gfx1151).
+# 115) + its Q4_0 MTP draft head, on the Strix Halo iGPU (gfx1151), :7770.
 #
-# These are the tunables read by scripts/serve-frank-large.sh. The flag
+# 2026-10-02 (Frank): "the highest quality of that repo, one full context,
-# list is owned by the serve script; this file only sets fleet values.
+# whatever runtime it requires". The highest-quality rung is Q8_0_ROCMFPX_AGENT
+# (26.28 GiB, 8.39 bpw: 187 tensors at plain Q8_0 + 311 at
+# GGML_TYPE_Q8_0_ROCMFPX); the AGENT routing is also the variant the card
+# tells you to take with MTP, because it keeps the output-head tensors at full
+# Q8_0 (187 vs 2), which is what the draft scores against. The runtime the
+# format REQUIRES is the ROCmFPX fork — type 103 exists nowhere else.
 #
-# DO NOT name a new tunable TEMP, HOST or GPU_MAX_HEAP_SIZE: the HIP
+# The shape: one session at the model's full native window (262,144), which is
-# runtime reads those bare (the 2026-08-13 TEMP incident). HOST is the one
+# the whole point — the Q8_0 head costs quality the fleet wanted back, and
-# exception because the serve script itself consumes it for --host.
+# the draft head pays for the bandwidth the bigger file spends.
+#
+# Budget (from the GGUF header, not from a boot log): attention KV lives only
+# in the 16 full-attention layers (4 KV heads x 256 head_dim), 64 KiB/token
+# with f16 K and V => 16.0 GiB at 262,144. The 48 Gated DeltaNet layers hold
+# ~144 MiB of fixed state per sequence. Weights 26.28 + draft head 1.57 +
+# mmproj 0.87 + KV 16.0 + state 0.14 + compute buffers ≈ 45-47 GiB against the
+# iGPU's ~114 GiB GTT aperture, so f16 KV (the unquantized choice) is affordable
+# here in a way it is not on frank-small's 24 GiB card.
 
-# The payload (scripts/download-frank-large.sh): the serve script refuses to
+# The qualified ROCmFPX fork build (scripts/build-rocmfpx.sh pins the commit).
-# start without the main GGUF and the DFlash2 drafter.
+LLAMA_BIN=/home/dank/GitPackages/kingjones-ROCmFPX/build/bin/llama-server
-MODEL_DIR=/home/dank/models/Qwen3.8-27B-UD-Q8_K_XL
+# One directory per HF repo, created by scripts/download-frank-large-kingjones.sh.
+MODEL_FILE=/home/dank/models/kingjones-Qwen3.8-27B-ROCmFP4-STRIX-MTP/Qwen3.8-27B-Q8_0_ROCMFPX_AGENT.gguf
+# The draft head: Q4_0, NOT the Q8_0 head. The head's per-step cost is its
+# projection over a 248K vocabulary, so the cheaper head is faster end to end
+# at equal-or-better acceptance (card: +6.0% at n-max 1, +10.2% at n-max 2).
+DRAFT_FILE=/home/dank/models/kingjones-Qwen3.8-27B-ROCmFP4-STRIX-MTP/mtp-Qwen3.8-27B-Q4_0.gguf
+# Vision tower, separate from the LLM file. Empty string = text-only.
+MMPROJ_FILE=/home/dank/models/kingjones-Qwen3.8-27B-ROCmFP4-STRIX-MTP/mmproj-Qwen3.8-27B-BF16.gguf
 
-# The stock llama.cpp runtime (the unsloth b11030 prebuilt).
+# Fleet endpoint: same port and alias as every frank-large before it, so the
-RUNTIME_DIR=/home/dank/GitPackages/unsloth-llama-b11030
+# harness models.yml id and the homepage probe do not move.
+ALIAS=frank-large
+# HIP reads a bare HOST, so this is BIND_ADDR — the same reason the file
+# avoids a bare TEMP.
+BIND_ADDR=0.0.0.0
+LISTEN_PORT=7770
 
-# Fleet endpoint: the same port and alias the prior deployments served, so
+# One full native window. llama.cpp divides --ctx-size across --parallel
-# no client, probe or models.yml entry changes address.
+# slots, so 262144/1 IS the model's trained window; the card validated 65,536
-HOST=0.0.0.0
+# and the fleet has been running this family at 262,144 since the gufo era.
-PORT=7770
+CTX_SIZE=262144
+PARALLEL=1
 
-# One stream, full 262144 context (the model's native window).
+# Deck knobs. --ubatch-size has a measured optimum on this stack (512 peak on
-CONTEXT_SIZE=262144
+# the sibling qwen35 model; larger ubatch costs GiBs of compute buffer for no
-PARALLEL_SLOTS=1
+# speed), and --fit must stay off: --fit's auto placement does not know that
+# 311 tensors of type 103 belong on the iGPU.
+BATCH_SIZE=2048
+UBATCH_SIZE=512
+NGL=999
+FIT=off
+FLASH_ATTN=on
+# Unquantized KV — the quality-first choice, affordable at one slot.
+CACHE_TYPE_K=f16
+CACHE_TYPE_V=f16
 
-# DFlash2 spec decode (a block-diffusion drafter) at depth 4. The drafter
+# Speculative decode: the model ships a standalone MTP head (mtp-Q4_0.gguf,
-# has a sliding window of 2048 (small draft KV, ~40 MiB).
+# arch qwen35, nextn_predict_layers=1, a 65-block draft). n-max 4 with n-min 0
+# and p-min 0.0 is the card's measured optimum (+31% end to end, acceptance
+# 0.926; its own n-max 8 measurement only reached 0.757 acceptance and LOST).
+# The draft runs entirely on the iGPU, so cross-device traffic stays off the
+# bandwidth-limited link.
+SPEC_TYPE=draft-mtp
+SPEC_DRAFT_NGL=99
+SPEC_DRAFT_DEVICE=ROCm0
 SPEC_DRAFT_N_MAX=4
+SPEC_DRAFT_N_MIN=0
+SPEC_DRAFT_P_MIN=0.0
 
-# Prompt-cache budget: 8 GiB. The model is a 27B hybrid (16 full attention
+# The card's thinking sampling profile, which is also what the harness sends
-# + 48 Gated DeltaNet linear attention); its DeltaNet state is smaller than
+# in extraBody, so the endpoint behaves the same to a client that omits it.
-# a primarily-DeltaNet model, so the fleet's 12.8 GiB DeltaNet floor does
+TEMPERATURE=1.0
-# not apply.
+TOP_P=0.95
-CACHE_RAM=8192
+TOP_K=20
+MIN_P=0.0
+PRESENCE_PENALTY=0.0
+
+NUM_THREADS=16
+THREAD_BATCH=16

scripts/frank-large.service

@@ -1,5 +1,5 @@
 [Unit]
-Description=stock llama.cpp (unsloth b11030, ROCm gfx1151) Qwen3.8-27B UD-Q8_K_XL, DFlash2 spec decode, fp8 KV, 262144 ctx / 1 slot, root systemd unit (dankclaw, frank-large, Strix Halo iGPU)
+Description=ROCmFPX llama.cpp (gfx1151) kingjones777 Qwen3.8-27B Q8_0_ROCMFPX_AGENT, MTP spec decode, 262144 ctx / 1 session, root systemd unit (dankclaw, frank-large, Strix Halo iGPU)
 After=network-online.target
 Wants=network-online.target
 
@@ -13,7 +13,9 @@ Group=dank
 LimitMEMLOCK=infinity
 Environment="FRANK_LARGE_ENV_FILE=/home/dank/serverAppConfigs/scripts/frank-large.env"
 ExecStart=/home/dank/serverAppConfigs/scripts/serve-frank-large.sh
-Restart=on-failure
+# always, not on-failure: an engine that drains and exits 0 after a worker
+# fault would otherwise leave the endpoint down until someone notices.
+Restart=always
 RestartSec=10
 
 [Install]

scripts/frank-primary.env

@@ -1,47 +1,129 @@
-# frank-primary.env — R9V radiance-vLLM production overrides (dankserver :7770).
+# frank-primary.env — tcclaviger RDNA4 vLLM production config (dankserver :7770).
 #
-# The full tuned engine shape (tiered-expert placement, CPU offload budgets
+# 2026-10-01 cutover to the BF16-KV no-spec incumbent (the overnight redo
-# per rank, MTP4 drafter, TP=2, qwen3_coder tool parser, 131072 model len,
+# campaign's verdict, Frank's call): same digest-pinned engine and payload as
-# 80-slot tiered expert cache, PLE pinned residency 54.4 GB) lives on-host
+# the 2026-09-29 cutover, but the KV cache is UNQUANTIZED (auto = bf16) and
-# in the R9V setup state — /home/dank/r9v-state/q8xl/setup.json — written by
+# speculative decode is GONE. Git tag frank-primary-fp8-mtp-20261001 is the
-# the q8 profile's setup run (ready: true; doctor PASS, artifacts
+# last pre-cutover commit; the old shape (fp8 KV + MTP4) is in history and
-# sha256-verified at fetch time). serve-frank-primary.sh layers that state
+# restorable from the tag.
-# first and THIS FILE second, so the keys below win. Never copy tuned values
+#
-# from the state into this file — the state is the single source of truth
+# Every number here was QUALIFIED by boot probe on 2026-10-01 (not inherited):
-# for engine shape.
+#   * MAXLEN 241664 x SEQS 3 (724,992 tokens) at bf16 KV is the DEPLOYED
-R9V_HOME=/home/dank/GitPackages/radiance-vllm-qwen4exp
+#     shape (2026-10-01 night ladder): KV_BYTES 10,184,000,000 PER RANK,
-R9V_STATE_FILE=/home/dank/r9v-state/q8xl/setup.json
+#     engine measured "KV cache size: 732,359 tokens; Maximum concurrency
-# Fleet identity: LAN bind, the single frank-primary port, the harness's
+#     for 241,664 tokens per request: 3.03x". 262144 and 245760 both OOMed
-# model id, our own container name (the launcher default r9v-qwen38-flash-next
+#     at graph capture (76 MiB short / 0 bytes free); 237568 passed at
-# is the scratch identity).
+#     718,606 tokens and the +4096 bonus step passed and won.
-R9V_HOST_BIND=0.0.0.0
+#   * KV_BYTES is PER RANK. The old "7.2 GiB ceiling, 9/16 GiB OOM" law was
-R9V_HOST_PORT=7770
+#     the MTP4 WORKSPACE: the spec-era pool measured 729,545 tokens because
-R9V_SERVED_MODEL_NAME=frank-primary
+#     the drafter reserved ~2.6 GiB/rank beside the pool. No-spec bf16 takes
-R9V_CONTAINER_NAME=frank-primary-r9v
+#     11.05 GiB/rank and boots clean — the ceiling was spec's, not VRAM's.
-R9V_EXPECTED_GPU_BDFS=0000:03:00.0,0000:07:00.0
+#   * bf16 KV doubles the attention-KV bytes/token (measured 14,047 vs
-# PLE registration (54.4 GB mmap -> HIP) can exceed the engine's 600 s
+#     7,926) and costs ~14% decode vs fp8 at 4064 ctx (49.25 -> 36.42 with
-# VLLM_PLE_OFFLOAD_READY_TIMEOUT default under page-cache churn (measured
+#     the gather win) — traded deliberately for clean cache + 262k windows.
-# 740 s on 2026-09-28, right after a bench teardown; a production boot
+#   * GATHER 32x64 is the night's biggest lever: R4D_LRU_GATHER_CHUNKS=32 /
-# failed on it the same morning). launch.sh 2de0357 forwards this to the
+#     R4D_LRU_GATHER_LANES=64 measured +18.7% decode vs default geometry,
-# engine; 1800 s covers the slowest observed registration with margin.
+#     and the win grows with depth (39.5@166 -> 54.1@1087 -> 49.3@4064 fp8
-R9V_PLE_READY_TIMEOUT=1800
+#     era). Do not drop it.
-# The Q8_0 payload quantizes BOTH fused hyper-connection shards (the IQ4_XS
+#   * MNBT 1024 is Frank's lock: MNBT 512 doubles TTFT at 4064 ctx (2.46 s
-# package ships inject as F32), so the shard-uniformity check rejects the
+#     vs 1.36 s); 1024-4096 are interchangeable on decode; 12288 costs 2.5%
-# fused projection at load. This dev-overlay adapter dequantizes a quantized
+#     and arena. 1024 keeps the workspace floor so the arena stays max.
-# inject shard in-stream (fork commit 363d606): opt-in mount, uncensored
+#   * PREFETCH IS ABSENT BY LAW: expert-cache telemetry measured the
-# payloads pass through unchanged.
+#     prefetcher at 36% precision (36,720 slabs prefetched, 13,369 served) —
-R9V_DEV_QWEN4_EXP_ADAPTER_PY=/home/dank/GitPackages/radiance-vllm-qwen4exp/runtimes/dev-overrides/qwen4_exp.py
+#     roughly half the gather traffic was waste. Never re-add
-# The runtime overlay's env.always injects R9V_FULL_MUTABLE_CACHE=1 — the
+#     --expert-prefetch without a fresh A/B.
-# qualified 160-slot uncensored IQ4_XS cache geometry — which the q8
+#   * TunableOp stays OFF: CLAV_TUNABLEOP_SWEEP silently skips under expert
-# profile's 80-slot tiered cache fails with a baseline mismatch. This flips
+#     offload unless CLAV_TUNABLEOP_WITH_OFFLOAD=1, and forcing it corrupts
-# that injection off for this payload (launch.sh passes it after the overlay
+#     numerics.
-# envs; docker last-wins).
+#
-R9V_FULL_MUTABLE_CACHE=0
+# HIP reads bare env vars TEMP, HOST, GPU_MAX_HEAP_SIZE — every key here is
-# The Qwen-Sharp template (the frank-large fleet law): folds reasoning
+# prefixed FRANK_PRIMARY_ so none can leak.
-# efforts (high/xhigh/max -> xhigh, minimal/low -> low) and defaults
+#
-# preserve_thinking true. --chat-template wins over the tokenizer dir's
+
-# metadata/chat_template.jinja; request-level kwargs still win per request.
+# --- runtime ----------------------------------------------------------------
-# Braces in values get the same single-quote law as parentheses (source
+# Digest-pinned: the floating :latest tag moved once already. Local alias
-# parses assignment values).
+# qwen-fp8-runtime:tuned-20260929 (image ID 3322dbbbc96d) points at the same
-R9V_CHAT_TEMPLATE_FILE=/home/dank/serverAppConfigs/scripts/qwen-sharp-v22.4.0.jinja
+# bytes in rootless storage.
-R9V_CHAT_TEMPLATE_KWARGS='{"preserve_thinking":true}'
+FRANK_PRIMARY_IMAGE=docker.io/tcclaviger/vllm@sha256:da4029666668b90a0386c1ed23ea370c79916ad4e1a6ad1dd9db098b9fa069ea
+
+# --- payload ----------------------------------------------------------------
+FRANK_PRIMARY_MODEL_DIR=/home/dank/models/Qwen3.8-Flash-Next-Uncensored-FP8
+FRANK_PRIMARY_MODEL_SHARDS=131
+
+# --- identity ---------------------------------------------------------------
+# The harness models.yml id stays frank-primary (provider 192.168.0.235:7770).
+FRANK_PRIMARY_CONTAINER=frank-primary-qwen
+FRANK_PRIMARY_SERVED_NAME=frank-primary
+FRANK_PRIMARY_PUBLISH=7770:7770
+# Containers removed before every start: the retired R9V identity and the ad
+# hoc experiment identity (both hold the GPUs / name registrations).
+FRANK_PRIMARY_LEGACY_CONTAINERS="frank-primary-r9v qwen-fp8"
+
+# --- qualified engine shape (2026-10-02, Frank's own correction) ---
+# 2026-10-01 left a shape that did not stay up: bf16 KV at 2 x 262144 with
+# --expert-precise-memory crash-looped at graph capture (the arena is
+# DEVICE-resident and the shape had 0.13 GiB to spare), and the plain bf16
+# 2 x 262144 boot booted but left 0 bytes free at runtime. This is the
+# correction: FP8 KV, TWO windows, each 200704 tokens, and the pool cut so
+# the expert arena gets the room back.
+#
+# LADDER (2026-10-01 night, subtract-16k method): 262144 and 245760 OOM at
+# graph capture; 237568 passed at 718,606 tokens; 241664 x 3 passed at
+# 732,359 tokens / 3.03x but is the shape this cutover retired. 200704 was
+# chosen to sit below the 241664 rung while leaving a full 2.95 GiB/rank of
+# pool to cut loose for the arena — the two contexts are a capacity
+# contract, not a throughput one (2-way concurrency summed BELOW parity at
+# 4064 ctx), so the slot buys the VRAM headroom long generations need.
+#
+# KV_BYTES is PER RANK. fp8 costs 7,701 B/token (measured 6,138 attention
+# + 1,563 mamba/side, dtype-independent). 2 x 200704 = 401,408 tokens
+# -> 3.091e9 B floor, pinned at 3.15e9 (~405,000 tokens, ~44 MB margin —
+# the same margin convention the 6.1e9 shape used).
+#
+# THE ARENA IS WHAT REMAINS: the fork's expert planner sizes it AFTER the
+# KV override, so every byte cut here is arena. The 6.1e9 shape ran an
+# arena of 6,248 slots; this one reclaims ~2.95 GiB/rank, roughly 1,260
+# more slots at 2.344 MiB each. That is the point of the change — the
+# gather read path is the measured decode bound, and a bigger arena means
+# more experts resident instead of streamed.
+#
+# MNBT 1024 (down from 2048): 1024-4096 are interchangeable on decode AND
+# identical on arena (the sweep measured the same slot count at every
+# rung <= 4096), so this is a pure workspace-floor choice, not a decode
+# trade. The floor matters because the arena budget is what is left.
+#
+# PRECISE MEMORY stays DROPPED: it crashed the 2 x 262144 bf16 boot
+# (weights 6.28 + pool 6.80 + graphs 0.55 + arena 16.65 + non-torch ~1.45
+# = 31.73 of 31.86 GiB, ~0.13 GiB free, NRestarts=3). The flag survives in
+# the serve script as an opt-in; re-enable it only with a pool/context
+# reduction that actually pays for the extra overhead.
+#
+# TUNABLEOP stays OFF (silently skipped under offload unless
+# CLAV_TUNABLEOP_WITH_OFFLOAD=1, and then corrupts numerics).
+#
+# SEQS 2 x MAXLEN 200704 at FP8 KV, pool cut for the arena. No
+# speculative decode, no prefetch.
+FRANK_PRIMARY_MAXLEN=200704
+FRANK_PRIMARY_SEQS=2
+FRANK_PRIMARY_KV_BYTES=3150000000
+FRANK_PRIMARY_KV_DTYPE=fp8
+FRANK_PRIMARY_MNBT=1024
+FRANK_PRIMARY_GATHER_CHUNKS=32
+FRANK_PRIMARY_GATHER_LANES=64
+FRANK_PRIMARY_UTIL=0.98
+
+# --- client contract: reasoning split, tools, preserve_thinking ----------------
+# Without --reasoning-parser the engine leaves the thinking text INLINE in
+# `content` with a dangling `</think>` and a null `reasoning` field (measured
+# 2026-10-01: content = 'We need answer...Brief.\n</think>\n\n391') — the
+# malformed thinking block clients choke on. The qwen3 parser splits it
+# properly. The GLM rollback (d91379d) had dropped all three of these; the
+# working values are the ones ecaad36 shipped for this model.
+FRANK_PRIMARY_REASONING_PARSER=qwen3
+# Tools: --enable-auto-tool-choice + the qwen3_xml parser is what makes
+# tool_calls work here (without it, tool_choice auto returns 400).
+FRANK_PRIMARY_TOOL_PARSER=qwen3_xml
+# preserve_thinking keeps prior thinking blocks in the rendered prompt across
+# turns; enable_thinking makes thinking the server-side default. Verified the
+# payload's tokenizer_config template consumes both keys.
+FRANK_PRIMARY_CHAT_TEMPLATE_KWARGS='{"enable_thinking": true, "preserve_thinking": true}'

scripts/frank-primary.service

@@ -1,26 +1,33 @@
-# frank-primary.service — R9V radiance-vLLM endpoint (unsloth
+# frank-primary.service — tcclaviger RDNA4 vLLM endpoint (orcarouter
-# Qwen3.8-Flash-Next Q8_0, MTP4, TP=2, tiered experts, 6-shard payload) on
+# Qwen3.8-Flash-Next-Uncensored-FP8, MTP4 spec decode, KV q8/q8, 600k total
-# the 2x R9700 pair, dankserver :7770. 2026-09-27 cutover from the
+# context over 3 sessions, TP=2, tiered experts, 131-shard payload) on the
-# uncensored IQ4_XS payload (its state dir stays on disk as rollback).
+# 2x R9700 pair, dankserver :7770. 2026-09-29 cutover from the R9V radiance
+# fork (git tag frank-primary-r9v-q8-20260929 is the pre-cutover state; its
+# launcher/shim/setup-state machinery is retired and the state dirs stay on
+# disk as rollback).
 #
-# Replaces the llama.cpp frank-primary unit (same port, same served alias;
+# Same port, same served alias; the harness models.yml id is unchanged.
-# harness models.yml unchanged in id, retuned for the 131072 window). The
+# The serve script drives `podman run` directly (the image entrypoint execs
-# serve script drives the R9V fork's own launcher through the docker->podman
+# `vllm serve`) and then waits on the container. NOTE: that wait can return
-# shim and then waits on the container, so a crashed engine is a nonzero
+# 0 even when the engine died mid-request (vLLM's API server drains and
-# ExecStart and Restart=on-failure is real.
+# exits clean on a worker fault), which is why the restart policy is
+# `always` — see the [Service] note.
 #
 # WHY A ROOT UNIT (not a rootless quadlet): the tiered-expert shape pins tens
-# of GiB of host RAM (the UVA expert host copy plus the 16 GiB PLE pinned
+# of GiB of host RAM (the expert host slabs, ~20.5 GiB per rank of pinned
-# reserve). Pinned pages are charged against RLIMIT_MEMLOCK; dank's login
+# anonymous memory). Pinned pages are charged against RLIMIT_MEMLOCK; dank's
-# session on this host carries 8 MiB and a user quadlet is capped by the user
+# login session on this host carries 8 MiB and a user quadlet is capped by the
-# manager's own hard limit — raising that means restarting user@1000.service
+# user manager's own hard limit — raising that means restarting
-# and bouncing every user container on the host. A root unit grants
+# user@1000.service and bouncing every user container on the host. A root unit
-# LimitMEMLOCK=infinity to the dank process via PID1, and the image stays in
+# grants LimitMEMLOCK=infinity to the dank process via PID1, and the image
-# dank's rootless podman storage (the shim drives podman as dank).
+# stays in dank's rootless podman storage.
 [Unit]
-Description=frank-primary — R9V radiance-vLLM Qwen3.8-Flash-Next Q8_0 on 2x R9700
+Description=frank-primary — tcclaviger vLLM Qwen3.8-Flash-Next-Uncensored-FP8 on 2x R9700
 After=network-online.target
 Wants=network-online.target
+# Never let the start-rate limiter give up after a crash loop (a failed LLM
+# endpoint that systemd refuses to retry is worse than a retrying one).
+StartLimitIntervalSec=0
 
 [Service]
 User=dank
@@ -33,12 +40,27 @@ Environment="FRANK_PRIMARY_ENV_FILE=/home/dank/serverAppConfigs/scripts/frank-pr
 ExecStart=/home/dank/serverAppConfigs/scripts/serve-frank-primary.sh
 # systemd stop: wait alone cannot stop the container child; stop it
 # explicitly. vLLM 0.26 drains gracefully on SIGTERM.
-ExecStop=/usr/bin/podman stop -t 120 frank-primary-r9v
+ExecStop=/usr/bin/podman stop -t 120 frank-primary-qwen
-# Boot cost: the launcher's preflight doctor (~1 min) after a warm page
+# Boot cost: FP8 payload page-in plus torch compile/graph capture —
-# cache; cold (post-reboot) payload page-in of ~175 GiB dominates.
+# ~15 min cold (measured 2026-09-29), faster warm.
 TimeoutStartSec=1800
-Restart=on-failure
+# Restart=always, NOT on-failure (measured 2026-10-01): a runtime
+# torch.OutOfMemoryError kills the engine while vLLM's API server still
+# exits 0, so ExecStart returned clean, systemd logged "Deactivated
+# successfully", and on-failure NEVER fired — the endpoint stayed down until
+# a human noticed. `always` restarts on any container death; an explicit
+# `systemctl stop` still wins (systemd does not restart a unit the operator
+# stopped), so bench windows and deploys are unaffected.
+Restart=always
 RestartSec=10
+# ExecStop allows the engine 120 s to drain (`podman stop -t 120`); without an
+# explicit TimeoutStopSec systemd's 90 s default cuts the stop short, kills it,
+# and records 'Failed with result timeout' (measured 2026-10-01 13:27:17).
+# Must exceed the podman grace with margin for the engine's own shutdown.
+TimeoutStopSec=180
+# The engine pins tens of GiB of host RAM (expert host slabs); it is the
+# last process on this host that should be chosen by the kernel OOM killer.
+OOMScoreAdjust=-500
 
 [Install]
 WantedBy=multi-user.target

scripts/frank-small.env

@@ -1,118 +1,60 @@
-# Tunables for frank-small (HauhauCS/Gemma4-12B-QAT-Uncensored-HauhauCS-
+# frank-small — llama.cpp (master, CUDA sm_86) serving
-# Balanced, Q4_K_M, plus the BF16 vision projector and the separate MTP
+# bartowski/MiMo-V2.6-Distill-Qwen-9B-GGUF:Q5_K_M on the RTX 3090, :7771.
-# sidecar) on mainline llama.cpp, Dankclaw's RTX 3090, since 2026-09-17.
 #
-# Runtime: llama.cpp master 37b53fd, CUDA build for sm_86:
+# 2026-10-02 (Frank): THREE contexts, vision on, and a hard instruction to
-#   cmake -B build-cuda -DCMAKE_BUILD_TYPE=Release -DGGML_NATIVE=ON \
+# leave 2-3 GiB of the card free. Both new demands cost memory, so the per-slot
-#     -DGGML_CUDA=ON -DCMAKE_CUDA_ARCHITECTURES=86 -DGGML_CUDA_FA_ALL_QUANTS=ON
+# window shrinks — see the budget below.
-#   cmake --build build-cuda --target llama-server -j
-# GGML_CUDA_FA_ALL_QUANTS is ON: this service runs q8_0 KV with flash
-# attention, and without the flag quantized-KV prefill collapses ~12x on
-# CUDA (see the CUDA lesson in AGENTS.md).
 #
-# Model: HauhauCS/Gemma4-12B-QAT-Uncensored-HauhauCS-Balanced at Q4_K_M
+# Two measurements on this card, this runtime, price the terms:
-# (6.87 GiB). Architecture (GGUF arch gemma4): 48 layers, MHA with 16 kv
+#   2 x 262,144, no projector: 20,190 MiB used / 4,386 free
-# heads x 512 head_dim, rope freq_base 1e6, final logit softcapping 30,
+#     KV = 524,288 tok x 24.5 KiB = 12,800 MiB => non-KV (weights 6,557 +
-# per-layer token embeddings (PLE) inside the file, and a sliding-window
+#     DeltaNet state ~25/slot + compute buffers + CUDA graphs) = 7,390 MiB
-# attention pattern (attention.sliding_window = 1024) -- most layers cache
+#   3 x 196,608, projector on: 22,972 MiB used / 1,153 free
-# only a 1024-position window, which is why the KV pool is a fraction of
+#     KV = 14,113 MiB => non-KV WITH the vision tower = 8,859 MiB
-# the full-ctx x all-layers number.
+# The tower costs ~1.4 GiB, not the ~1.0 its file size suggested, so 196,608
+# per slot left only 1.1 GiB free — outside the band. The pool therefore is
+# 3 x 180,224 = 540,672 tokens: KV ~12,936 + non-KV 8,859 = ~21,800 MiB predicted.
+# MEASURED at this shape: 21,684 MiB used / 2,441 MiB free — inside the band, and
+# the number the bats budget gate reproduces (it prices non-KV at 9 GiB).
 #
-# MTP: the Multi-Token Prediction head ships as a SEPARATE file
+# KV arithmetic: attention KV exists only in the 8 full-attention layers
-# (mtp-gemma-4-12B-it.gguf, arch gemma4-assistant, 4 nextn layers, 0.24
+# (4 KV heads x 256 head_dim). K f16 = 16 KiB/token, V q8_0 = 8.5 KiB/token
-# GiB). Unlike the Qwen3.8 -mtp build (head inside the main file), this one
+# (34 B per 32-element block) => 24.5 KiB/token, so 12,288 tokens (=4096/slot)
-# is passed via --model-draft with --spec-type draft-mtp, so the draft
+# is one 293 MiB step — that is the ladder to walk if this moves again. The
-# context loads the assistant model. The same --spec-type value chains the
+# native 262,144 per slot is NOT reachable three times: 3 x 262,144 x
-# draftless ngram-mod matcher, which attaches unconditionally; two
+# 24.5 KiB = 18.4 GiB of KV before anything else is loaded.
-# context-backed drafters cannot share one process, so draft-mtp,ngram-mod
-# is the only legal shape that carries both.
 #
-# SIZING (RTX 3090, 24576 MiB): CTX_SIZE = PARALLEL x 262144 -- four
+# Why V stays q8_0 and K stays f16: full-precision V would add 8 KiB/token —
-# FULL-window slots, measured to fit: the 4-slot boot idles at 22,346
+# +4.5 GiB on this pool — which is what pushes the card past its 24 GiB. K is
-# MiB and each extra full-context slot costs 3,450 MiB (measured 1-slot
+# the tensor the asymmetric-KV contract exists to protect.
-# 11,996 -> 4-slot 22,346). A fifth slot projects to ~25.8 GiB, past the
+LLAMA_BIN=/home/dank/GitPackages/llama-master/build-cuda/bin/llama-server
-# card, and the KV pool allocates upfront -- so 4 is the max. Explicit
+MODEL_FILE=/home/dank/models/MiMo-Q5KM/MiMo-V2.6-Distill-Qwen-9B-Q5_K_M.gguf
-# CTX_SIZE bypasses --fit on purpose: --fit maximises the TOTAL pool,
-# which would shrink each slot below the full 262144 window once
-# parallel > 1.
-#
-# Vision + audio: --mmproj loads the BF16 projector (11 tensors, 52M
-# params; it carries an audio encoder too). The fleet's image path lives
-# here, which is why halogen no longer sets HALOGEN_VISION_TOWER. Audio
-# WORKS: input_audio content parts transcribe verbatim (verified
-# 2026-09-17 with a 16k mono WAV).
-#
-# Sampling: greedy by default (the fleet contract); a client that sends a
-# temperature gets its own, and the harness applies per-model overrides
-# from models.yml.
-#
-# WARNING: do not name any variable TEMP, HOST, or GPU_MAX_HEAP_SIZE (HIP
-# reads those bare vars on the ROCm hosts -- see the 2026-08-13 incident in
-# AGENTS.md). This box is CUDA, but the rule keeps env files portable.
 
-# llama.cpp runtime binary: mainline master 37b53fd, CUDA sm_86,
+# Vision: the Q5_K_M GGUF ships no `v.*` tensors, so the projector file IS the
-# FA_ALL_QUANTS=ON.
+# vision capability (sha256 ff348f31..., 918,166,048 B, fetched by
-LLAMA_BIN=/home/dank/GitPackages/llama.cpp-mainline/build-cuda/bin/llama-server
+# scripts/download-frank-small.sh). Empty string = text-only on purpose.
+MMPROJ_FILE=/home/dank/models/MiMo-Q5KM/mmproj-MiMo-V2.6-Distill-Qwen-9B-f16.gguf
 
-# Model package. The MTP sidecar file is required for speculative decoding.
+# Fleet endpoint: same port and alias as every frank-small before it, so the
-MODEL_DIR=/home/dank/models/Gemma4-12B-QAT
+# harness models.yml id and the homepage probe do not move.
-MODEL_FILE=Gemma4-12B-QAT-Uncensored-HauhauCS-Balanced-Q4_K_M.gguf
-MMPROJ_PATH=/home/dank/models/Gemma4-12B-QAT/mmproj-Gemma4-12B-QAT-Uncensored-HauhauCS-Balanced-BF16.gguf
-DRAFT_FILE=mtp-gemma-4-12B-it.gguf
-
-# Fleet identity and network. 7771 is this service's own endpoint on
-# Dankclaw; 7770 stays frank-large's. The alias is the id at /v1/models and
-# what clients ask for -- the harness entry in models.yml names it.
 ALIAS=frank-small
 BIND_ADDR=0.0.0.0
 LISTEN_PORT=7771
 
-# Four FULL 262144-token contexts, one per slot (CTX_SIZE is the total
+# Three independent conversations. llama.cpp divides --ctx-size across
-# pool; --parallel divides it). See the SIZING note above for the fit
+# --parallel slots (n_ctx_slot = CTX_SIZE/PARALLEL), and a mamba-hybrid model
-# arithmetic; a fifth slot does not fit.
+# cannot use unified KV, so each slot gets its own cache and its own window.
-CTX_SIZE=1048576
+# 180,224 per slot is BELOW the native 262,144, and that is the cost of the
-PARALLEL=4
+# third slot: a longer single prompt needs compaction instead of one shot.
-CACHE_TYPE_K=q8_0
+# Consequence worth knowing: frank-primary's compaction trigger is 183,501
+# tokens, which no longer fits a frank-small slot — see the runbook's
+# "three slots vs compaction" note.
+CTX_SIZE=540672
+PARALLEL=3
+
+# The KV contract (Frank, 2026-10-01 night): unquantized K, q8_0 V.
+# q8_0 on V is only served by the flash-attention path; the serve script
+# pins --flash-attn on and must never lose it.
+CACHE_TYPE_K=f16
 CACHE_TYPE_V=q8_0
-# CPU RAM cache cap in MiB (build default 8192). This host is memory-tight
+NGL=99
-# (frank-large's 70 GiB GTT plus the rest of the stack), and Gemma4 runs
-# fully on the 3090 (-ngl 999), so the CPU-side cache is not the hot path;
-# cap it at 2048 to give back ~6 GiB of LPDDR. Gemma4 is not a DeltaNet
-# model, so the >12.8 GiB CACHE_RAM floor does not apply.
-CACHE_RAM=2048
-
-# Batch geometry: 8192 logical / 1024 physical. Measured on this card at
-# max length (254k-token prefill / 253k-ctx decode, 4 slots, q8 KV):
-# ubatch 512 -> pp 915 t/s, tg@250k 90-204; ubatch 1024 -> pp 962,
-# tg 257-302. Larger ubatches (2048) OOM the MTP draft context's pp
-# compute buffer (~3.3 GiB against 2.2 GiB headroom) and must not be
-# tried. Idle VRAM is 23,300 MiB at this geometry.
-BATCH_SIZE=8192
-UBATCH_SIZE=1024
-
-# CPU threads: this service runs entirely on the GPU (NGL=999), so threads
-# cover tokenization, sampling and the HTTP layer only. Deliberately modest
-# -- this host's CPU is shared with halogen's engine, which is CPU-hungry.
-NUM_THREADS=8
-THREAD_BATCH=8
-
-# Every layer on the 3090, with flash attention (required for q8 KV).
-NGL=999
-FLASH_ATTN=on
-
-# Speculative decoding: the MTP sidecar (draft context) chained with the
-# draftless ngram-mod matcher. Draft depth and busy-wait level are made
-# explicit here (both were the implicit build defaults: 3 and 50).
-# Measured at 250k ctx: depth 4 + poll 100 DECODED WORSE (230-257 t/s)
-# than depth 3 + poll 50 (257-302 t/s) — the longer verification round
-# costs more attention re-read at long context than the extra accepted
-# token earns. Keep 3/50.
-SPEC_TYPE=draft-mtp,ngram-mod
-SPEC_DRAFT_N_MAX=3
-POLL_LEVEL=50
-
-# Greedy default (fleet contract); reasoning is extracted into
-# reasoning_content with an unlimited budget (the model decides).
-TEMPERATURE=0.0
-REASONING_FORMAT=deepseek
-REASONING_BUDGET=-1

scripts/frank-small.service

@@ -1,5 +1,5 @@
 [Unit]
-Description=llama.cpp Qwen3.8-27B GSQ-RCO IQ3_S MTP+vision (dankclaw, frank-small)
+Description=llama.cpp MiMo-V2.6-Distill-Qwen-9B Q5_K_M GGUF (qwen35 hybrid, 3x180224, f16 K + q8_0 V, mmproj vision) on the RTX 3090 (dankclaw, frank-small)
 After=network-online.target
 Wants=network-online.target
 
@@ -8,8 +8,7 @@ Type=simple
 User=dank
 Group=dank
 # CUDA pins its staging buffers, so the process needs the memlock budget even
-# though the weights themselves are copied into VRAM. Parity with the halogen
+# though the weights themselves are copied into VRAM.
-# quadlet's Ulimit=memlock=-1:-1 and the retired ik unit.
 LimitMEMLOCK=infinity
 Environment="FRANK_SMALL_ENV_FILE=/home/dank/serverAppConfigs/scripts/frank-small.env"
 ExecStart=/home/dank/serverAppConfigs/scripts/serve-frank-small.sh

scripts/patches/glm53-compile-01-model-decorator.patch

@@ -0,0 +1,86 @@
+# glm53-compile-01-model-decorator.patch — put the GLM-5.3-Flash trunk under torch.compile.
+#
+# STATUS: WIP — first of three patches in the "compile the GLM trunk" changelist. The
+# set does NOT yet boot the engine (see 03 for the next blocker), so it is NOT part of
+# the deployed frank-primary shape. Nothing in scripts/ mounts these files.
+#
+# WHY THIS IS THE RIGHT LEVER (measured 2026-09-30, not inferred):
+#
+# The fork's own flagship trunk — vllm/models/qwen4_exp/amd/model.py — carries
+# @support_torch_compile. vllm/models/glm5next/common/model.py does not, and the engine
+# says so on every boot:
+#
+#   Cudagraph mode FULL_AND_PIECEWISE requires piecewise capture, but the loaded model
+#   provides neither a compiled submodule nor breakable CUDA graphs.
+#   Overriding to FULL_DECODE_ONLY.
+#
+# A torch-profiler window on the live endpoint (496 decode tokens, rank0, 3 x 160-token
+# generations at the deployed shape) shows what the missing compilation costs. Total
+# device kernel time 46.18 s / 496 tokens = ~93 ms per token, against a measured step
+# time of ~104 ms (9.6 tok/s with the profiler attached):
+#
+#   16.07 s  34.8 %  708,500 launches  vectorized_elementwise_kernel      (22.7 us each)
+#    4.60 s  10.0 %    42,000           r4d_gemm_moe_w4a16_nt_b16_m64     (the MoE GEMM)
+#    3.11 s   6.7 %   174,708           elementwise_kernel_manual_unroll
+#    2.94 s   6.4 %   138,252           wvSplitK_hf_sml_                  (split-K GEMM)
+#    4.09 s   8.9 %    83,160           topk/router (sbtopk gatherTopK + warpMergeSortTopK
+#                                       + bitonicSortKVInPlace)
+#    3.01 s   6.5 %   133,920           r4d_mhc_* (hyper-connection pre/post/finish)
+#    2.78 s   6.0 %   127,916           inductor triton_poi_* / triton_per_* (the only
+#                                       fusions that DO happen)
+#    0.49 s   1.05%    20,664           r4d_moe_lru_gather_kernel         <-- the suspect
+#
+# That last row retired the fleet's working hypothesis. Decode on this shape is NOT
+# expert-gather bound: the gather is 1 % of device time, the same order as the telemetry
+# kernel that was switched on to measure it. 44 % of the budget is pointwise glue —
+# bf16<->fp32 conversions, adds, pows/rsqrts from fp32 RMSNorm, clamp/scale quant
+# passes — spread over ~1,428 launches per token. Inductor fusion is the only lever
+# that touches that mass; the router is second at 9 % but is binary-blocked (the
+# libr4d route kernel exists only as r4d_moe_route_kernel<128|256|512> instantiations
+# and GLM has 288 experts, with no HIP sources shipped to rebuild).
+#
+# TARGET: vllm/models/glm5next/common/model.py   (inside the pinned image)
+#   base    sha256 008eeaa2a07e091465319f86900b5a321d921238723ccbe22e6b9e971e1d2b28
+#   patched sha256 6e9cb81b84a9a88bd41403ca404aa04c3b791899dd77d661c3f25ae686cecd40
+#
+# APPLY: `patch -p1` from the vllm package root, or bind-mount new/.../model.py over
+# site-packages. Then R4D_GLM_COMPILE=1 (default) enables it; =0 makes the decorator a
+# no-op, so the file can stay mounted for an A/B without a second artifact.
+# Requires 02 (trace-safe probes) — without it the worker dies during profile_run on
+# the first probe call inside the traced region.
+--- a/vllm/models/glm5next/common/model.py
++++ b/vllm/models/glm5next/common/model.py
+@@ -1,12 +1,14 @@
+ # SPDX-License-Identifier: Apache-2.0
+ # SPDX-FileCopyrightText: Copyright contributors to the vLLM project
+ 
++import os
+ from collections.abc import Iterable
+ from typing import ClassVar, Literal
+ 
+ import torch
+ from torch import nn
+ 
++from vllm.compilation.decorators import support_torch_compile
+ from vllm.config import ParallelConfig, VllmConfig
+ from vllm.distributed import (
+     get_ep_group,
+@@ -623,6 +625,18 @@
+         )
+ 
+ 
++_GLM_COMPILE = os.environ.get("R4D_GLM_COMPILE", "1") == "1"
++
++
++@support_torch_compile(
++    dynamic_arg_dims={
++        "input_ids": 0,
++        "positions": -1,
++        "intermediate_tensors": 0,
++        "inputs_embeds": 0,
++    },
++    enable_if=lambda _cfg: _GLM_COMPILE,
++)
+ class Glm5NextModel(nn.Module):
+     def __init__(self, *, vllm_config: VllmConfig, prefix: str = ""):
+         super().__init__()

scripts/patches/glm53-compile-02-trace-safe-probes.patch

@@ -0,0 +1,119 @@
+# glm53-compile-02-trace-safe-probes.patch — libr4d capability probes must not do I/O
+# inside a torch.compile region.
+#
+# STATUS: WIP — second of three patches in the "compile the GLM trunk" changelist.
+# Inert (but harmless) without 01. Not part of the deployed shape.
+#
+# THE BREAK (measured: with 01 mounted and R4D_GLM_COMPILE=1, the worker dies during
+# profile_run, i.e. the first — cold — forward):
+#
+#   Explanation: Dynamo does not know how to trace the builtin `posix.stat`
+#   ... r4d_mhc_supported -> r4d_lib.r4d_mhc_enabled -> available -> os.path.exists
+#
+# THE NON-OBVIOUS PART: functools.lru_cache does NOT protect these probes. Dynamo
+# re-traces the wrapped body to build its guard even when the cache is warm — it warns
+# about exactly this ("Dynamo detected a call to a `functools.lru_cache`-wrapped
+# function ... Dynamo ignores the cache wrapper and directly traces the wrapped
+# function"). Verified in-image, torch 2.11.0+rocm10.0, torch.compile(backend="eager"):
+#
+#   cold lru_cache call       -> RAISE Unsupported
+#   WARM lru_cache call       -> RAISE Unsupported     <-- warming is not enough
+#   plain module-global read  -> compiles
+#   uncached os.path.exists   -> RAISE Unsupported
+#
+# So each probe must resolve ONCE into a plain value and be read as a global. The set
+# below is every no-argument probe / symbol lookup reachable from a forward pass,
+# grep-verified call sites:
+#   model_executor/layers/fused_moe/experts/r4d_w4a16_moe.py:79  available()
+#   model_executor/layers/fused_moe/experts/r4d_w4a16_moe.py:81  has_w4a16_moe()
+#   model_executor/kernels/mhc/r4d_dsv4.py:16                    r4d_mhc_enabled()
+#   v1/attention/ops/rocm_aiter_mla_sparse.py:694                r4d_indexer_enabled(), mqa_max_heads()
+#   v1/attention/ops/rocm_aiter_mla_sparse.py:1124               r4d_indexer_enabled()
+#   model_executor/layers/fused_moe/experts/r4d_mxfp4_moe.py:683 r4d_act_quant_enabled()
+#   model_executor/layers/fused_moe/experts/r4d_w4a16_moe.py _entry()  moe_w4a16()/moe_w4a16_mt1() per MoE call
+# (lib() is the transitive dependency of all of them.)
+#
+# TARGET: vllm/model_executor/kernels/r4d_lib.py   (inside the pinned image)
+#   base    sha256 b8dfeefb4acc72b67b0579a4ab0653ef5d48c40cc8649187d757c80ce6b9617a
+#   patched sha256 0751852e1ca813ad2bdb1da7bf10c605130e63e04ef9ec57e70b5f675a29c08e
+#
+# APPLY: append the block below to the module. R4D_FREEZE_PROBES=0 restores stock
+# behaviour. Cost when nothing is compiled: ten one-time probes at import (measured:
+# logged once per process, e.g. "libr4d capability probes frozen: 10 warm [...]").
+#
+# OBSERVED EFFECT: with 01+02 the worker gets past this break and reaches the next one
+# (see 03) — dynamo reports "Dynamo bytecode transform time: 1.08 s" for the trunk.
+--- a/vllm/model_executor/kernels/r4d_lib.py
++++ b/vllm/model_executor/kernels/r4d_lib.py
+@@ -361,3 +361,71 @@
+ @functools.lru_cache
+ def moe_w4a16_mt1():
+     return getattr(lib(), "r4d_gemm_moe_w4a16_nt_b16_m64_mt1", None)
++
++
++# FORK-LOCAL (patch: trace-safe capability probes).  torch.compile traces the
++# BODY of a functools.lru_cache call even when the cache is warm, so a capability
++# probe called from a forward pass is a dynamo graph break.  Measured 2026-09-30
++# on the GLM-5.3-Flash trunk under @support_torch_compile:
++#   Dynamo does not know how to trace the builtin `posix.stat`
++#   r4d_mhc_supported -> r4d_mhc_enabled -> available -> os.path.exists
++# Each probe below is wrapped to resolve ONCE into a memo and is warmed at import
++# (outside any tracing region), so a traced call is a dict read.  The probe set is
++# every no-argument capability probe / symbol lookup reachable from a forward
++# pass: the W4A16 + mxfp4 MoE dispatch, the mHC op, the MLA sparse indexer gate
++# and the draft/lm-head guards.
++_PROBE_NAMES = (
++    "available",
++    "lib",
++    "has_w4a16_moe",
++    "moe_w4a16",
++    "moe_w4a16_mt1",
++    "has_mhc_dsv4",
++    "r4d_mhc_enabled",
++    "r4d_indexer_enabled",
++    "mqa_max_heads",
++    "r4d_act_quant_enabled",
++)
++
++
++def _freeze_probe(fn):
++    """Return fn bound to its first result.  Result must be a plain value."""
++
++    memo = []
++
++    def probe(*args, **kwargs):
++        if not memo:
++            memo.append(fn(*args, **kwargs))
++        return memo[0]
++
++    return probe
++
++
++def _freeze_probes() -> list[str]:
++    frozen = []
++    for name in _PROBE_NAMES:
++        fn = globals().get(name)
++        if fn is None:
++            continue
++        # Only the fast branch of a bound probe is traced; an empty memo means the
++        # first call still does I/O, so populate it here as well.
++        globals()[name] = _freeze_probe(fn)
++        frozen.append(name)
++    return frozen
++
++
++from vllm.logger import init_logger as _init_logger
++
++_probe_log = _init_logger(__name__)
++
++if os.environ.get("R4D_FREEZE_PROBES", "1") == "1":
++    _warmed, _failed = [], []
++    for _n in _freeze_probes():
++        try:
++            globals()[_n]()
++            _warmed.append(_n)
++        except Exception:  # noqa: BLE001 -- probe unavailable here; recompute lazily
++            _failed.append(_n)
++    _probe_log.info("libr4d capability probes frozen: %d warm %s%s",
++                    len(_warmed), _warmed,
++                    f" (deferred: {_failed})" if _failed else "")

scripts/patches/glm53-compile-03-fullgraph-optout.patch

@@ -0,0 +1,116 @@
+# glm53-compile-03-fullgraph-optout.patch — let torch.compile split the graph instead
+# of failing on a break.
+#
+# STATUS: WIP — third of three patches in the "compile the GLM trunk" changelist.
+# With 01+02+03 the model TRACES, then hits the NEXT blocker (below). The set does not
+# yet boot the engine; nothing in scripts/ mounts it.
+#
+# THE BREAK: with 01+02 mounted the worker dies with
+#
+#   RuntimeError: Graph breaks are not supported with aot compile.
+#   Please use torch.compile(fullgraph=True).
+#
+# because vllm/compilation/wrapper.py calls torch.compile(..., fullgraph=True, ...) and
+# this build has AOT compile on by default (VLLM_USE_AOT_COMPILE). The GLM trunk cannot
+# satisfy fullgraph: its indexer metadata path syncs to the host —
+#   models/glm5next/amd/sparse_indexer.py:263  int(positions[...].max().item()) + 1
+#   models/glm5next/amd/sparse_indexer.py:419  lmax = int(decode_metadata.decode_lens.max().item())
+# — and a .item() inside a captured graph is not expressible.
+#
+# WHAT WAS VERIFIED (in-image, torch 2.11.0+rocm10.0, torch.compile(backend="eager")):
+#
+#   fullgraph=True   suppress_errors=False  -> RAISE Unsupported
+#   fullgraph=True   suppress_errors=True   -> RAISE Unsupported   <-- suppress_errors
+#                                                                     does NOT cover breaks
+#   fullgraph=False  suppress_errors=False  -> ok   (traceable segments compiled by
+#                                                    inductor, the rest runs eagerly;
+#                                                    numerics unchanged)
+#
+# So breaks must be TOLERATED, not suppressed: the traceable mass still fuses. This
+# patch makes the fullgraph requirement opt-out via VLLM_COMPILE_FULLGRAPH=0 (default 1,
+# so every other model keeps stock behaviour), and the run also needs
+# VLLM_USE_AOT_COMPILE=0 because AOT compile refuses breaks outright.
+#
+# TARGET: vllm/compilation/wrapper.py   (inside the pinned image)
+#   base    sha256 969961dca41c78385212e985cfc0c57ef067f4a2d7490c73c658c600a1b95cac
+#   patched sha256 732363bb1a1367772f13c60a655c678e7e2e9e56186677b14238f5e84cac31b3
+#   (the file already imports os; the patch does not add a second import)
+#
+# NEXT BLOCKER — NOT FIXED HERE. With 01+02+03 and
+# R4D_GLM_COMPILE=1 VLLM_COMPILE_FULLGRAPH=0 VLLM_USE_AOT_COMPILE=0 the trunk traces
+# ("Dynamo bytecode transform time: 1.08 s") and then the worker dies during profile_run
+# on the decode forward:
+#
+#   KeyError: 'in_proj_qkvbfg_a'
+#     model.py:748  Glm5NextModel.forward        hidden_states, residual, post, comb = layer(...)
+#     model.py:456  Glm5NextDecoderLayer.forward
+#
+# `in_proj_qkvbfg_a` is the KDA layer's merged column-parallel projection
+# (models/glm5next/common/kda.py:217, called at :407) — and it is also a name in the
+# load_weights shard table (model.py:780-785), i.e. a real submodule attribute name,
+# not an arbitrary string.
+#
+# WHERE IT IS THROWN (established 2026-09-30 by reading the full traceback, since the
+# report line is inside dynamo-rewritten code): torch/_dynamo keeps a name -> module map
+# and looks it up by name — output_graph.py:1259 / :1384 register into `nn_modules`,
+# :3000 does `self.nn_modules[node.target]` (a bare-name KeyError when absent), and
+# :1225 documents stripping "_buffers[..]/_parameters[..]/_modules[..]" from those
+# names. So the compiler is failing to RESOLVE A SUBMODULE BY NAME while tracing the
+# GLM decoder layer. Whether the root cause is the fork's module tree (something
+# dynamo cannot register statically) or dynamo's own registration is NOT yet settled.
+#
+# STATUS OF THAT BLOCKER (updated 2026-09-30, after the diagnostics below):
+#   * TORCHDYNAMO_VERBOSE=1 adds NO internal stack trace -> it is a plain Python
+#     KeyError raised in a callee dynamo INLINED into Glm5NextDecoderLayer.forward
+#     (the reported line 456 is that method's `def` line, which is how inlined frames
+#     surface).
+#   * It is NOT raised from dynamo's four nn_modules lookups: the probe in
+#     /home/dank/dynamo-probe wraps three of them and stayed silent over two runs with
+#     the probe confirmed installed in the API-server and worker processes.
+#   * It is NOT a literal string lookup: the name appears tree-wide only as the KDA
+#     layer's attribute (kda.py:217, used at :407) and in the load_weights shard table
+#     (model.py:780-785). The four `proj_name = prefix.split(".")[-1]` sites in the
+#     quantization utils do compute exactly this bare key, but each is guarded
+#     (`if proj_name not in fused_mapping: return`), so none of them can raise it.
+#   * It reproduces on every run of this configuration (5 of 5).
+# REMAINING INSTRUMENT: name the inlined caller. Either `TORCH_LOGS=+dynamo` on a boot
+# and grep the log for the key (dynamo prints inlined frames with source), or wrap the
+# inlining candidates in that forward — the mHC CustomOps (hc_pre / hc_post /
+# hc_fused_post_pre in layers/mhc.py) and the indexer op — with a KeyError-reporting
+# probe. Until that names the caller this is a dynamo-side name-resolution failure.
+# NOTE the launcher gotcha that wasted a boot earlier: the scratch launcher now
+# preserves ENVARGS (`ENVARGS="${ENVARGS:-}"`), but it previously reset it, silently
+# dropping outer env vars.
+#
+# SECOND BLOCKER, KNOWN AND NOT FATAL IN THIS CONFIG: the indexer's in-forward
+# workspace acquisition (patch 05 makes it compiler-opaque) and the fused-norm
+# launcher cell (patch 04 removes the call) are real breaks, but with
+# VLLM_COMPILE_FULLGRAPH=0 they are tolerated — they only become fatal under
+# fullgraph=True, which this trunk can never satisfy anyway.
+--- a/vllm/compilation/wrapper.py
++++ b/vllm/compilation/wrapper.py
+@@ -19,6 +19,15 @@
+ 
+ logger = init_logger(__name__)
+ 
++
++# FORK-LOCAL (patch: fullgraph opt-out).  torch.compile(fullgraph=True) turns any
++# dynamo graph break into a hard failure; suppress_errors does NOT cover breaks
++# (verified in-image, torch 2.11+rocm10.0).  With VLLM_COMPILE_FULLGRAPH=0 the
++# traceable segments are still compiled by inductor and the untraceable ones run
++# eagerly, which is what a model with device-sync metadata (the GLM-5.3-Flash
++# indexer) needs.  Default 1 keeps stock behaviour for every other model.
++_FULLGRAPH = os.environ.get("VLLM_COMPILE_FULLGRAPH", "1") == "1"
++
+ R = TypeVar("R")
+ P = ParamSpec("P")
+ 
+@@ -146,7 +155,7 @@
+         with aot_context:
+             self._compiled_callable = torch.compile(
+                 compiled_ptr,
+-                fullgraph=True,
++                fullgraph=_FULLGRAPH,
+                 dynamic=False,
+                 backend=backend,
+                 options=options,

scripts/patches/glm53-compile-04-fused-qkv-rmsnorm-optout.patch

@@ -0,0 +1,112 @@
+# glm53-compile-04-fused-qkv-rmsnorm-optout.patch — take GLM's fused-norm JIT kernel
+# out of the traced path.
+#
+# STATUS: WIP — fourth of the "compile the GLM trunk" changelist. Needs 01 (the
+# decorator) to matter; not part of the deployed shape.
+#
+# MEASUREMENT NOTE — read the run history before trusting any claim below, because a
+# launcher bug made four consecutive boots misleading.  The scratch launcher
+# (serve_glm_autoround.sh) was restructured on 2026-09-30 and the
+# `-e VLLM_COMPILE_FULLGRAPH=0` line was DROPPED, so those boots mounted the wrapper
+# patch while `_FULLGRAPH` still defaulted to True.  With fullgraph=True *every* break
+# is an error, which is why the reported failure kept moving as breaks were removed:
+#
+#   fullgraph=False, AOT off  -> breaks tolerated; dies later on the dynamo name-map
+#                                `KeyError: 'in_proj_qkvbfg_a'`
+#   fullgraph=True,  AOT off  -> dies on whichever break comes first
+#                                (cell -> workspace -> compiler-disable)
+#
+# An earlier version of this note blamed `TORCH_LOGS=graph_breaks` for the difference.
+# That was WRONG — the flag only logs; the launcher silently ran fullgraph=True.
+# Launcher fixed; the env var is set again.
+#
+# What it means for THIS patch: under fullgraph=True the fused-norm call was the first
+# break dynamo reported, and after this patch the report moved on to the workspace
+# acquisition — so it removes a real break.  It is not by itself what unblocks a boot:
+# with breaks genuinely tolerated the blocker is the name-map KeyError.
+#
+# IT IS STILL THE RIGHT PATTERN TO REMOVE, either way: the launcher family
+# (@kernel_launcher, 39 call sites) is exactly the shape that produces breaks, and the
+# object this one closes over is bound after use (see below).
+#
+# THE BREAK (measured, TORCH_LOGS=graph_breaks, rank0, 2026-09-30).  With 01+02+03 the
+# trunk traces, and with AOT off + fullgraph=True dynamo reports a FATAL break — fatal
+# even with VLLM_COMPILE_FULLGRAPH=0, because dynamo cannot build a resume function
+# around it:
+#
+#   torch._dynamo.exc.Unsupported: Read uninitialized cell
+#   Explanation: Attempted to read a cell variable that has not been populated yet.
+#   Developer debug context: CellVariable()
+#
+# User stack:
+#   glm5next/common/model.py:748    Glm5NextModel.forward       layer(...)
+#   glm5next/common/model.py:526    Glm5NextDecoderLayer.forward
+#   glm5next/common/attention.py:590
+#   model_executor/layers/mla.py:208
+#     q_proj_input, kv_c_normed = fused_q_kv_rmsnorm(...)
+#   -> torch/_dynamo/polyfills/__init__.py:392 getattr_and_trace
+#
+# So dynamo is tracing the fork's JIT-kernel launcher framework
+# (VllmTritonJitKernel + @kernel_launcher — 39 call sites) and finds a closure name
+# that is not yet bound.  In models/common/ops/fused_qk_rmsnorm.py the kernel object
+# is bound AFTER the function that closes over it:
+#
+#   def fused_q_kv_rmsnorm(...):          # line 194
+#       ...
+#       _FUSED_Q_KV_RMSNORM_KERNEL(...)   # line 211
+#   _FUSED_Q_KV_RMSNORM_KERNEL = FusedQKVRMSNormKernel()    # line 223
+#
+# Two ways out, and this patch takes the cheap one:
+#   (a) WORKAROUND (here): stop calling the fused kernel from the traced region.  The
+#       flag is already a constructor argument (mla.py:80, default False) that glm5next
+#       turns ON at attention.py:583, so it becomes an env knob.
+#   (b) ROOT FIX (not done): make the launcher trace-safe for all 39 sites.  Any other
+#       @kernel_launcher-backed kernel still in the traced path can produce the same
+#       fatal break, so (b) is the durable fix and (a) is what unblocks iterating.
+#
+# COST of R4D_GLM_FUSE_QKV_RMSNORM=0: one extra small RMSNorm launch per MLA layer per
+# token (q_a_layernorm + kv_a_layernorm instead of one fused kernel) — ~2 launches out
+# of ~3,700 per token on this shape, against the 44 % of device time currently spent in
+# unfused pointwise work.  Correct trade for a compile experiment.
+#
+# TARGET: vllm/models/glm5next/common/attention.py   (inside the pinned image)
+#   base    sha256 362bbf0e6c50cdfa446c6551b9b1ddff425e08f88e3d4140fd8db88f8a619e0c
+#   patched sha256 84969993c250e9f8183a33a6e1ddd607272f48b950b560272eb33b7cdde31189
+#
+# APPLY: `patch -p1` from the vllm package root (verified: applies clean, reproduces
+# the patched sha256), or bind-mount new/.../attention.py over site-packages.
+# R4D_GLM_FUSE_QKV_RMSNORM=1 (default) restores the fused kernel.
+--- a/vllm/models/glm5next/common/attention.py
++++ b/vllm/models/glm5next/common/attention.py
+@@ -1,6 +1,7 @@
+ # SPDX-License-Identifier: Apache-2.0
+ # SPDX-FileCopyrightText: Copyright contributors to the vLLM project
+ 
++import os
+ import torch
+ import torch.nn.functional as F
+ from torch import nn
+@@ -36,6 +37,14 @@
+ from vllm.utils.deep_gemm import PAGED_MQA_PAGE_SIZES
+ from vllm.v1.kv_cache_interface import KpoolTailSpec, MLAAttentionSpec
+ 
++
++# FORK-LOCAL (patch: keep the traced region trace-safe).  R4D_GLM_FUSE_QKV_RMSNORM=0
++# replaces the fused `fused_q_kv_rmsnorm` JIT-kernel launch (one kernel) with two plain
++# RMSNorms, because dynamo cannot trace the launcher's closure ("Read uninitialized cell",
++# fatal even with fullgraph=False).  The fused kernel object in
++# models/common/ops/fused_qk_rmsnorm.py is bound AFTER the function that uses it.
++_FUSE_QKV_RMSNORM = os.environ.get("R4D_GLM_FUSE_QKV_RMSNORM", "1") == "1"
++
+ logger = init_logger(__name__)
+ 
+ # Shared torch.compile config for the indexer's small-kernel leaves. The MLA
+@@ -580,7 +589,7 @@
+             quant_config,
+             prefix,
+             skip_topk=False,
+-            fuse_qkv_rmsnorm=True,
++            fuse_qkv_rmsnorm=_FUSE_QKV_RMSNORM,
+         )
+ 
+     def forward(

scripts/patches/glm53-compile-05-workspace-acquisition-opaque.patch

@@ -0,0 +1,96 @@
+# glm53-compile-05-workspace-acquisition-opaque.patch — the indexer's workspace
+# acquisition must not be traced.
+#
+# STATUS: WIP — fifth of the "compile the GLM trunk" changelist. This is a break worth
+# removing, but NOT part of the confirmed blocker and NOT required for the
+# break-tolerant route: `torch.compiler.disable` is itself a graph break, so this patch
+# helps only a hypothetical fullgraph=True route, where the workspace acquisition would
+# be better RESTRUCTURED out of the forward (allocate at init/profile time) than made
+# opaque. Kept because the discovery is real and the wrapper is a no-op when nothing is
+# compiling.
+#
+# THE BREAK — CONFIRMED with AND WITHOUT the logging flag (2026-09-30):
+#
+#   torch._dynamo.exc.Unsupported: Unsupported method call
+#   user stack
+#     glm5next/common/model.py:748      Glm5NextModel.forward
+#     glm5next/common/model.py:526      Glm5NextDecoderLayer.forward
+#     glm5next/common/attention.py:599 / :401
+#     glm5next/amd/sparse_indexer.py:711  forward_native
+#     glm5next/amd/sparse_indexer.py:132  sparse_attn_indexer_kpool
+#     v1/worker/workspace.py:135          get_simultaneous
+#     v1/worker/workspace.py:155          _ensure_workspace_size
+#
+# Line 132 is the PROFILE-RUN branch — the source comment says what it is:
+#
+#     if not isinstance(attn_metadata, dict):
+#         # Reserve workspace for indexer during profiling run
+#
+# so the engine dies exactly where it sizes its memory.  `_ensure_workspace_size`
+# reads ContextVar and ubatch state (`_workspace_lane.get()`,
+# `dbo_current_ubatch_id()`) and may allocate; none of that is traceable.
+#
+# FIX: the documented escape hatch — `torch.compiler.disable` around a thin wrapper,
+# so dynamo stops analysing the call and executes it eagerly from inside the compiled
+# region.  Chosen over decorating the workspace manager's own methods because that
+# would change workspace behaviour for every model, not just this one.  The wrapper is
+# a no-op when nothing is compiling, so it needs no env gate and is safe in the
+# stock (uncompiled) path — which is why this patch can be reasoned about
+# independently of whether the changelist ever ships.
+#
+# CAVEAT, stated plainly: a compiler-disabled region is still a graph break, so this
+# needs patch 03 (VLLM_COMPILE_FULLGRAPH=0) to be tolerable at all, and whether dynamo
+# can build a resume function around it is exactly what the next boot decides — the
+# difference between this failure and 04's `Read uninitialized cell`.
+#
+# TARGET: vllm/models/glm5next/amd/sparse_indexer.py   (inside the pinned image)
+#   base    sha256 f773a99265d3f440c8fad22c8fb5a8b33f926162094cffc3a2ecdaa42763c5b1
+#   patched sha256 c154115afe8253a35c652b7198929ef4c1874957d333fdb3340cacf88e612b78
+#
+# APPLY: `patch -p1` from the vllm package root, or bind-mount over site-packages.
+--- a/vllm/models/glm5next/amd/sparse_indexer.py
++++ b/vllm/models/glm5next/amd/sparse_indexer.py
+@@ -35,6 +35,21 @@
+ from vllm.v1.attention.ops.common import pack_seq_triton, unpack_seq_triton
+ from vllm.v1.worker.workspace import current_workspace_manager
+ 
++@torch.compiler.disable
++def _acquire_workspace(*shapes_and_dtypes):
++    """Acquire shared workspace buffers, untraced.
++
++    Workspace acquisition reads ContextVar/ubatch state and may allocate
++    (v1/worker/workspace.py: ``_ensure_workspace_size`` -> ``_workspace_lane.get()``,
++    ``dbo_current_ubatch_id()``), which dynamo cannot trace -- it raises
++    ``Unsupported: Unsupported method call`` and kills the engine during the profile
++    run.  ``torch.compiler.disable`` keeps the call as an opaque, eagerly-executed
++    region inside the compiled graph.  No-op when not compiling.
++    """
++
++    return current_workspace_manager().get_simultaneous(*shapes_and_dtypes)
++
++
+ logger = init_logger(__name__)
+ 
+ # kpool write helper: form pools from the current token batch and compress them
+@@ -129,7 +144,7 @@
+         values_spec, scales_spec = _gather_workspace_shapes(
+             total_seq_lens, head_dim, fp8_dtype, use_fp4_cache
+         )
+-        current_workspace_manager().get_simultaneous(
++        _acquire_workspace(
+             values_spec,
+             scales_spec,
+             ((RADIX_TOPK_WORKSPACE_SIZE,), torch.uint8),
+@@ -275,11 +290,10 @@
+         # Layout switches between FP8 (head_dim bytes + 4-byte fp32 scale) and
+         # MXFP4 (head_dim/2 bytes packed + head_dim/MXFP4_BLOCK_SIZE ue8m0
+         # scales) based on use_fp4_cache.
+-        workspace_manager = current_workspace_manager()
+         values_spec, scales_spec = _gather_workspace_shapes(
+             total_seq_lens, head_dim, fp8_dtype, use_fp4_cache
+         )
+-        k_quant_full, k_scale_full = workspace_manager.get_simultaneous(
++        k_quant_full, k_scale_full = _acquire_workspace(
+             values_spec,
+             scales_spec,
+         )

scripts/patches/glm53-dflash-01-aux-hidden-states.patch

@@ -0,0 +1,173 @@
+# glm53-dflash-01-aux-hidden-states.patch — give the GLM trunk the EAGLE3 / DFlash2
+# auxiliary-hidden-state interface.
+#
+# STATUS: WIP, and INDEPENDENT of the compile changelist (01-05). It patches the same
+# file as glm53-compile-01-model-decorator.patch, so if both are ever applied, apply 01
+# first. Not part of the deployed shape.
+#
+# WHY.  `--speculative-config '{"method":"dflash","model":"/path/to/drafter", ...}'` with
+# incoai/GLM-5.3-Flash-DFlash2 is accepted -- the config parses, the drafter loads -- and
+# then the worker dies at init with:
+#
+#     RuntimeError: Model does not support EAGLE3 interface
+#
+# raised by v1/worker/gpu/spec_decode/eagle/eagle3_utils.py:20 through
+# `supports_eagle3(model)` == `isinstance(model, SupportsEagle3)` (a runtime-checkable
+# Protocol).  Nothing about the drafter, the method name or the payload is wrong:
+# `glm5next` implements neither half of the contract.
+#
+# THE CONTRACT (read out of this image, not assumed):
+#   * DEFINITIVE (v5, after four failed boots): the member set is checked against
+#     the INSTANCE the runner calls -- which for GLM-5.3-Flash is
+#     Glm5NextForConditionalGeneration, NOT Glm5NextForCausalLM.  The multimodal
+#     wrapper wraps the text model as self.language_model
+#     (init_vllm_registered_model, architectures=["Glm5NextForCausalLM"],
+#     glm5next/common/model.py:1113), and model_runner.py:425 calls
+#     set_eagle3_aux_hidden_state_layers(self.model) on the CONDITIONAL.
+#     Presence rule (verified in-image with stock typing, python 3.14):
+#     isinstance walks ALL __protocol_attrs__ -- members of SupportsEagle3 AND
+#     its base SupportsEagleBase (has_own_lm_head, has_own_embed_tokens) -- and
+#     getattr_static()s each one off the instance, breaking on the first miss.
+#     So BOTH wrappers need all three attributes.
+#   * MRO law: INHERITING SupportsEagle3 as a base dies at class creation with
+#     "Cannot create a consistent method resolution order (MRO) for bases ..."
+#     (GLM's wrappers already carry four/five Protocol-derived bases).  The
+#     attribute route avoids the MRO entirely.  Also proven by simulation:
+#     the four-base ForCausalLM ordering cannot take SupportsEagle3 in ANY
+#     position.
+#   * Method placement: without inheriting, the Protocol's default
+#     set_aux_hidden_state_layers / get_eagle3_default_aux_hidden_state_layers
+#     do not exist on either wrapper, so both are defined explicitly --
+#     ForCausalLM unwraps to self.model (the EagleModelMixin trunk, v1's
+#     correct bodies), the conditional delegates to self.language_model
+#     (the pixtral.py:498 pattern).
+#   * interfaces.py:1673-1719 -- for reference, the inherited defaults unwrap
+#     `holder = getattr(parent_ref, "model", parent_ref)` and assert
+#     `isinstance(holder, EagleModelMixin)`, defaulting to (2, L // 2, L - 3).
+#   * eagle3_utils.py:26-31 -- layers come from the spec config when it carries them
+#     (this drafter ships target_layer_ids [5, 14, 24, 33, 42]), else from the default.
+#   * deepseek_v2.py:1551-1585 and v1/worker/gpu/cudagraph_utils.py:621 -- the trunk
+#     appends one tensor per aux layer while looping and returns the TUPLE
+#     (hidden_states, aux_hidden_states) only when that list is non-empty.
+#
+# WHAT IT ADDS for GLM: EagleModelMixin on Glm5NextModel; aux capture in the layer
+# loop, where the aux state is the single-stream `hidden_states` ENTERING that layer
+# (GLM's loop carries the already-summed single-stream state -- its non-mHC branch
+# re-splits `residual = hidden_states` inside the layer -- so this is the equivalent
+# of deepseek_v2's `hidden_states + residual`); the conditional tuple return; the
+# three SupportsEagle3 member attributes on BOTH wrappers; and the two interface
+# methods on both (ForCausalLM: unwrap to trunk; conditional: delegate to
+# language_model).
+#
+# HONEST SCOPE: this makes the interface PRESENT, which is exactly what the engine
+# checks -- it does not make drafts GOOD, and per the measured profile it is not a
+# decode-speed fix (a verification pass pays pointwise cost proportional to its token
+# count, which is why MTP4 measured flat).  It is the prerequisite for driving DFlash2
+# on this endpoint at all.
+#
+# TARGET: vllm/models/glm5next/common/model.py   (inside the pinned image)
+#   base    sha256 008eeaa2a07e091465319f86900b5a321d921238723ccbe22e6b9e971e1d2b28
+#   patched sha256 bc9913a15c8e7f88e0c36b07bf029b383cc8c2078b58d66dc91aa235f8af77f4
+#
+# APPLY: `patch -p1` from the vllm package root, or bind-mount the patched file over
+# site-packages (which is what a scratch-launcher DFLASH_PATCH=1 does).
+--- a/vllm/models/glm5next/common/model.py
++++ b/vllm/models/glm5next/common/model.py
+@@ -95,6 +95,12 @@
+ logger = init_logger(__name__)
+ 
+ 
++from vllm.model_executor.models.interfaces import (
++    EagleModelMixin,
++    SupportsEagle3,
++)
++
++
+ class Glm5NextMLP(nn.Module):
+     def __init__(
+         self,
+@@ -623,7 +629,7 @@
+         )
+ 
+ 
+-class Glm5NextModel(nn.Module):
++class Glm5NextModel(nn.Module, EagleModelMixin):
+     def __init__(self, *, vllm_config: VllmConfig, prefix: str = ""):
+         super().__init__()
+ 
+@@ -730,7 +736,14 @@
+         if self.is_sequence_parallel:
+             hidden_states = sp_shard(hidden_states)
+ 
+-        for layer in self._active_layers:
++        aux_hidden_states: list[torch.Tensor] = []
++        for local_idx, layer in enumerate(self._active_layers):
++            if (getattr(self, "start_layer", 0) + local_idx) in self.aux_hidden_state_layers:
++                # Auxiliary hidden state for EAGLE-3 / DFlash2 drafts: the
++                # single-stream state ENTERING this layer, before its input norm --
++                # the same semantic deepseek_v2 implements by appending
++                # hidden_states + residual at this point in its loop.
++                aux_hidden_states.append(hidden_states)
+             hidden_states, residual, post, comb = layer(
+                 positions, hidden_states, residual, post, comb
+             )
+@@ -749,6 +762,8 @@
+             hidden_states = sp_all_gather(hidden_states)[:full_num_tokens]
+ 
+         hidden_states = self.norm(hidden_states)
++        if len(aux_hidden_states) > 0:
++            return hidden_states, aux_hidden_states
+         return hidden_states
+ 
+     def load_weights(self, weights: Iterable[tuple[str, torch.Tensor]]) -> set[str]:
+@@ -937,6 +952,10 @@
+ class Glm5NextForCausalLM(
+     nn.Module, HasInnerState, SupportsPP, MixtureOfExperts, IsHybrid
+ ):
++    supports_eagle3: ClassVar[Literal[True]] = True
++    has_own_lm_head: bool = False
++    has_own_embed_tokens: bool = False
++
+     def __init__(self, *, vllm_config: VllmConfig, prefix: str = ""):
+         super().__init__()
+         self.model_config = vllm_config.model_config
+@@ -963,6 +982,14 @@
+     def embed_input_ids(self, input_ids: torch.Tensor) -> torch.Tensor:
+         return self.model.embed_input_ids(input_ids)
+ 
++    def set_aux_hidden_state_layers(self, layers: tuple[int, ...]) -> None:
++        # EAGLE-3 / DFlash2 interface: unwrap to the trunk (the EagleModelMixin).
++        self.model.aux_hidden_state_layers = layers
++
++    def get_eagle3_default_aux_hidden_state_layers(self) -> tuple[int, ...]:
++        num_layers = len(self.model.layers)
++        return (2, num_layers // 2, num_layers - 3)
++
+     def forward(
+         self,
+         input_ids: torch.Tensor | None,
+@@ -1033,6 +1060,10 @@
+ class Glm5NextForConditionalGeneration(
+     Glm4vForConditionalGeneration, HasInnerState, IsHybrid, MixtureOfExperts
+ ):
++    supports_eagle3: ClassVar[Literal[True]] = True
++    has_own_lm_head: bool = False
++    has_own_embed_tokens: bool = False
++
+     # The text model (KDA + dense-MLA + MoE) is a hybrid mamba model. The
+     # multimodal wrapper must declare the same interfaces so vLLM treats it as
+     # hybrid (auto-aligns mamba/attention block sizes, sizes the mamba state
+@@ -1076,6 +1107,13 @@
+ 
+         return Glm5NextForCausalLM.get_mamba_state_copy_func()
+ 
++    def set_aux_hidden_state_layers(self, layers: tuple[int, ...]) -> None:
++        # EAGLE-3 / DFlash2: delegate to the language model (pixtral.py:498).
++        self.language_model.set_aux_hidden_state_layers(layers)
++
++    def get_eagle3_default_aux_hidden_state_layers(self) -> tuple[int, ...]:
++        return self.language_model.get_eagle3_default_aux_hidden_state_layers()
++
+     def __init__(self, *, vllm_config: VllmConfig, prefix: str = ""):
+         super(Glm4vForConditionalGeneration, self).__init__()
+         config = vllm_config.model_config.hf_config

scripts/patches/glm53-dflash-04-kv-groups-draft.patch

@@ -0,0 +1,317 @@
+--- a/vllm/v1/core/kv_cache_utils.py
++++ b/vllm/v1/core/kv_cache_utils.py
+@@ -1210,17 +1210,38 @@
+         for name, spec in kv_cache_spec.items()
+         if isinstance(spec, KpoolTailSpec)
+     }
++    hidden_specs = {
++        name: spec
++        for name, spec in kv_cache_spec.items()
++        if isinstance(spec, HiddenStateCacheSpec)
++    }
+     attn_specs = {
+         name: spec
+         for name, spec in kv_cache_spec.items()
+-        if not isinstance(spec, (MambaSpec, KpoolTailSpec))
++        if not isinstance(
++            spec, (MambaSpec, KpoolTailSpec, HiddenStateCacheSpec)
++        )
+     }
+-    if not mamba_specs or not all(
+-        type(spec) is MLAAttentionSpec for spec in attn_specs.values()
+-    ):
++    if not mamba_specs or not attn_specs:
++        return None
++    # FORK-LOCAL (patches/glm53-dflash-04-kv-groups-draft): with a dflash
++    # drafter, attn_specs also carries the drafter's attention layers. The
++    # GLM MLA layers keep their own group (no page unification -- per-group
++    # pages are the glm5 design); the drafter layers must form one uniform
++    # group of their own, and the hidden-state layers get per-layer groups
++    # like the generic tail.
++    mla_specs = {
++        name: spec
++        for name, spec in attn_specs.items()
++        if type(spec) is MLAAttentionSpec
++    }
++    draft_specs = {
++        name: spec for name, spec in attn_specs.items() if name not in mla_specs
++    }
++    if not mla_specs:
++        return None
++    if draft_specs and UniformTypeKVCacheSpecs.from_specs(draft_specs) is None:
+         return None
+-
+-    mla_specs = cast(dict[str, MLAAttentionSpec], attn_specs)
+     idx_pages = {
+         spec.page_size_bytes for spec in mla_specs.values() if spec.tokens_per_state > 1
+     }
+@@ -1233,7 +1254,7 @@
+     mla_pages = {mla_specs[name].page_size_bytes for name in mla_names}
+     assert len(mla_pages) == 1
+     mla_page = mla_pages.pop()
+-    uniform_spec = UniformTypeKVCacheSpecs.from_specs(attn_specs)
++    uniform_spec = UniformTypeKVCacheSpecs.from_specs(mla_specs)
+     assert uniform_spec is not None
+ 
+     tail_group: KVCacheGroupSpec | None = None
+@@ -1270,10 +1291,20 @@
+     for index, name in enumerate(mamba_specs):
+         mamba_grouped_names[index % num_groups].append(name)
+ 
++    draft_group: KVCacheGroupSpec | None = None
++    if draft_specs:
++        draft_uniform = UniformTypeKVCacheSpecs.from_specs(draft_specs)
++        assert draft_uniform is not None
++        draft_group = KVCacheGroupSpec(list(draft_specs), draft_uniform)
++    hidden_groups = [
++        KVCacheGroupSpec([name], spec) for name, spec in hidden_specs.items()
++    ]
+     return (
+-        [KVCacheGroupSpec(list(attn_specs), uniform_spec)]
++        [KVCacheGroupSpec(list(mla_specs), uniform_spec)]
+         + ([tail_group] if tail_group is not None else [])
++        + ([draft_group] if draft_group is not None else [])
+         + create_kv_cache_group_specs(padded_specs, mamba_grouped_names)
++        + hidden_groups
+     )
+ 
+ 
+@@ -1289,10 +1320,17 @@
+         int,
+         list[str],
+         int,
++        KVCacheGroupSpec | None,
++        list[KVCacheGroupSpec],
+     ]
+     | None
+ ):
+-    """Recognize the GLM-5.3-Flash grouping after optional PP projection."""
++    """Recognize the GLM-5.3-Flash grouping after optional PP projection.
++
++    FORK-LOCAL (patches/glm53-dflash-04-kv-groups-draft): also recognizes the
++    drafter's uniform group and the per-layer hidden-state groups, returned
++    as ``draft_group`` / ``hidden_groups`` so the pool builder and the memory
++    estimator can size their tensors beside the aliased GLM pool."""
+     uniform_groups = [
+         group
+         for group in kv_cache_groups
+@@ -1301,17 +1339,28 @@
+     mamba_groups = [
+         group for group in kv_cache_groups if isinstance(group.kv_cache_spec, MambaSpec)
+     ]
++    hidden_groups = [
++        group
++        for group in kv_cache_groups
++        if isinstance(group.kv_cache_spec, HiddenStateCacheSpec)
++    ]
+     attn_group: KVCacheGroupSpec | None = None
+     tail_group: KVCacheGroupSpec | None = None
++    draft_group: KVCacheGroupSpec | None = None
+     for group in uniform_groups:
+         inner = cast(UniformTypeKVCacheSpecs, group.kv_cache_spec).kv_cache_specs
+         if all(type(spec) is MLAAttentionSpec for spec in inner.values()):
+             attn_group = group
+         elif all(isinstance(spec, KpoolTailSpec) for spec in inner.values()):
+             tail_group = group
++        else:
++            draft_group = group
+     if attn_group is None or not mamba_groups:
+         return None
+-    if len(uniform_groups) + len(mamba_groups) != len(kv_cache_groups):
++    if (
++        len(uniform_groups) + len(mamba_groups) + len(hidden_groups)
++        != len(kv_cache_groups)
++    ):
+         return None
+ 
+     attn_uniform = cast(UniformTypeKVCacheSpecs, attn_group.kv_cache_spec)
+@@ -1362,6 +1411,8 @@
+         idx_page,
+         tail_names,
+         tail_page,
++        draft_group,
++        hidden_groups,
+     )
+ 
+ 
+@@ -1395,10 +1446,34 @@
+         return kv_cache_spec
+ 
+     max_page_size = max(page_sizes)
++    # GLM-5.3-Flash + a draft model: the sparse indexer k_cache page is
++    # tokens_per_state x the MLA page (same per-token bytes), and MLA layers
++    # cannot be PADDED to an arbitrary page size (sparse MLA indexes the cache
++    # in whole token rows) -- but they CAN grow their block size when the max
++    # page is an exact multiple of the MLA page. Bump the max page to a
++    # multiple of every MLA-family page so all of them take the divisible
++    # block-growth branch below instead of the padding refusal, while the
++    # draft and mamba layers simply pad to the (slightly larger) page.
++    mla_pages = {
++        spec.page_size_bytes
++        for spec in kv_cache_spec.values()
++        if isinstance(spec, MLAAttentionSpec)
++    }
++    if mla_pages:
++        mla_common = math.lcm(*mla_pages)
++        max_page_size = -(-max_page_size // mla_common) * mla_common
+     new_kv_cache_spec = {}
+     for layer_name, layer_spec in kv_cache_spec.items():
+         if layer_spec.page_size_bytes == max_page_size:
+             new_kv_cache_spec[layer_name] = layer_spec
++        elif isinstance(layer_spec, KpoolTailSpec):
++            # The kpool tail kernel pins the tail block to the kpool size
++            # (tail_kv_cache.view(...) must be [blocks, 2, index_kpool,
++            # head_dim]); growing block_size breaks that contract. Pad the
++            # physical page instead -- the tail is its own group and the pad
++            # wastes bytes only.
++            new_spec = replace(layer_spec, page_size_padded=max_page_size)
++            new_kv_cache_spec[layer_name] = new_spec
+         elif isinstance(layer_spec, MambaSpec):
+             # MambaSpec's page size is determined by its state shapes and does
+             # not scale with block_size, so pad the page instead. This is the
+@@ -1593,8 +1668,31 @@
+ ) -> int:
+     """Return the largest cache group's bytes per block."""
+     if (glm5_layout := _glm5_next_tensor_layout(kv_cache_groups)) is not None:
+-        _, _, mla_names, idx_names, mla_page, idx_page, _, _ = glm5_layout
+-        return len(mla_names) * mla_page + len(idx_names) * idx_page
++        (
++            _,
++            _,
++            mla_names,
++            idx_names,
++            mla_page,
++            idx_page,
++            _,
++            _,
++            draft_group,
++            hidden_groups,
++        ) = glm5_layout
++        bytes_per_block = len(mla_names) * mla_page + len(idx_names) * idx_page
++        if draft_group is not None:
++            d_inner = cast(
++                UniformTypeKVCacheSpecs, draft_group.kv_cache_spec
++            ).kv_cache_specs
++            bytes_per_block += sum(
++                spec.page_size_bytes for spec in d_inner.values()
++            )
++        for group in hidden_groups:
++            bytes_per_block += (
++                group.kv_cache_spec.page_size_bytes * len(group.layer_names)
++            )
++        return bytes_per_block
+ 
+     bytes_per_block = max(
+         sum(
+@@ -1701,12 +1799,30 @@
+             idx_page,
+             tail_names,
+             _,
++            draft_group,
++            hidden_groups,
+         ) = glm5_layout
+         bytes_per_block = len(mla_names) * mla_page + len(idx_names) * idx_page
++        extra_bytes_per_block = 0
++        if draft_group is not None:
++            d_inner = cast(
++                UniformTypeKVCacheSpecs, draft_group.kv_cache_spec
++            ).kv_cache_specs
++            extra_bytes_per_block += sum(
++                spec.page_size_bytes for spec in d_inner.values()
++            )
++        for group in hidden_groups:
++            extra_bytes_per_block += (
++                group.kv_cache_spec.page_size_bytes * len(group.layer_names)
++            )
+         num_blocks = may_override_num_blocks(
+-            vllm_config, available_memory // bytes_per_block
++            vllm_config,
++            available_memory // (bytes_per_block + extra_bytes_per_block),
+         )
+-        size = bytes_per_block * num_blocks
++        # One backing allocation shared by every tensor (allocate_kv_cache
++        # asserts all tensor sizes equal): the aliased GLM pool plus the
++        # draft/hidden regions appended after it.
++        size = (bytes_per_block + extra_bytes_per_block) * num_blocks
+         attn_specs = cast(
+             UniformTypeKVCacheSpecs, attn_group.kv_cache_spec
+         ).kv_cache_specs
+@@ -1745,6 +1861,37 @@
+                 ).kv_cache_specs
+                 add_tensor(tail_name, tail_specs[tail_name], offset)
+ 
++        # Draft + hidden-state layers get their own tensors appended after
++        # the aliased pool (no aliasing: independent block tables).
++        offset_cursor = bytes_per_block * num_blocks
++        if draft_group is not None:
++            d_inner = cast(
++                UniformTypeKVCacheSpecs, draft_group.kv_cache_spec
++            ).kv_cache_specs
++            d_page = next(iter(d_inner.values())).page_size_bytes
++            kv_cache_tensors.append(
++                KVCacheTensor(
++                    size=size,
++                    layers=list(draft_group.layer_names),
++                    layer_stride=d_page * num_blocks,
++                    block_stride=d_page,
++                    offset=offset_cursor,
++                )
++            )
++            offset_cursor += d_page * num_blocks * len(draft_group.layer_names)
++        for group in hidden_groups:
++            h_page = group.kv_cache_spec.page_size_bytes
++            kv_cache_tensors.append(
++                KVCacheTensor(
++                    size=size,
++                    layers=list(group.layer_names),
++                    layer_stride=h_page * num_blocks,
++                    block_stride=h_page,
++                    offset=offset_cursor,
++                )
++            )
++            offset_cursor += h_page * num_blocks * len(group.layer_names)
++
+         return KVCacheConfig(
+             num_blocks=num_blocks,
+             kv_cache_tensors=kv_cache_tensors,
+@@ -2440,6 +2587,8 @@
+             idx_page,
+             tail_names,
+             _,
++            draft_group,
++            hidden_groups,
+         ) = glm5_layout
+         uniform_spec = cast(UniformTypeKVCacheSpecs, attn_group.kv_cache_spec)
+         total_blocks = uniform_spec.max_memory_usage_pages(vllm_config)
+@@ -2452,7 +2601,30 @@
+         )
+         if tail_names:
+             total_blocks += 1
+-        return total_blocks * (len(mla_names) * mla_page + len(idx_names) * idx_page)
++        pool_bytes = total_blocks * (
++            len(mla_names) * mla_page + len(idx_names) * idx_page
++        )
++        extra_bytes = 0
++        if draft_group is not None:
++            d_inner = cast(
++                UniformTypeKVCacheSpecs, draft_group.kv_cache_spec
++            ).kv_cache_specs
++            d_spec = next(iter(d_inner.values()))
++            d_blocks = cdiv(
++                d_spec.max_memory_usage_bytes(vllm_config), d_spec.page_size_bytes
++            )
++            extra_bytes += d_blocks * d_spec.page_size_bytes * len(
++                draft_group.layer_names
++            )
++        for group in hidden_groups:
++            h_spec = group.kv_cache_spec
++            h_blocks = cdiv(
++                h_spec.max_memory_usage_bytes(vllm_config), h_spec.page_size_bytes
++            )
++            extra_bytes += (
++                h_blocks * h_spec.page_size_bytes * len(group.layer_names)
++            )
++        return pool_bytes + extra_bytes
+ 
+     bytes_per_block = _pool_bytes_per_block(kv_cache_groups)
+     total_blocks = 0

scripts/patches/glm53-dflash-04b-kv-registry-dflash.patch

@@ -0,0 +1,31 @@
+--- a/vllm/model_executor/offloader/expert_activation_registry.py
++++ b/vllm/model_executor/offloader/expert_activation_registry.py
+@@ -316,7 +316,27 @@
+ 
+ 
+ def _glm53_kv_bytes_per_token(cfg: "VllmConfig") -> int:
+-    return 13600
++    per_tok = 13600
++    # FORK-LOCAL (patches/glm53-dflash-04-kv-groups-draft): with a dflash/dspark
++    # drafter, the KV spec set gains the drafter's attention layers and the aux
++    # hidden-state cache (one bf16 hidden vector per target aux layer per
++    # token). Both are full-length per-token costs the aliased glm5 pool
++    # cannot absorb, so the reserve must include them. The drafter is a 5-layer
++    # qwen3 (8 kv heads x 128 x K+V x bf16 = 4096 B/token); the aux count is
++    # the dflash target_layer_ids count (5).
++    # NOTE: use_aux_hidden_state_outputs lives on the gpu_model_runner
++    # instance (gpu_model_runner.py:623/678), not on the spec config -- it is
++    # not set yet when the expert plan runs. The spec-config METHOD is the
++    # plan-time signal for the same fact.
++    sc = getattr(cfg, "speculative_config", None)
++    if sc is not None and getattr(sc, "method", "") in (
++        "dflash",
++        "dspark",
++        "extract_hidden_states",
++    ):
++        hidden = int(cfg.model_config.hf_config.hidden_size)
++        per_tok += 5 * hidden * 2 + 4096
++    return per_tok
+ 
+ 
+ for _arch in ("Glm5NextForConditionalGeneration", "Glm5NextForCausalLM"):

scripts/patches/gpu_worker-kv-cap.patch

@@ -0,0 +1,65 @@
+# gpu_worker-kv-cap.patch — fork-local fix for the expert-offload KV cap.
+#
+# STATUS: NOT part of the deployed frank-primary shape. It exists because the
+# fork's plan under-models this model's KV, and it is the enabler for a future
+# MTP attempt (the production GLM shape runs without MTP and boots fine on the
+# plan's own reserve).
+#
+# THE BUG: the plan models KV as 13600 B/token + a small mamba term, but
+# GLM-5.3-Flash also carries a per-sequence KDA state (34 linear-attention
+# layers) and, under MTP, the draft model's KV. gpu_worker then CAPS the KV
+# cache at that reserve, so the plan's own arithmetic can make a legal
+# --max-model-len unbootable. Measured 2026-09-30 on the pinned image:
+#   - no MTP, max_model_len 131072: reserve 1.74 GiB >= need 1.86 GiB  -> boots
+#   - MTP4,  max_model_len 131072: reserve 1.74 GiB <  need 2.07 GiB  -> refuses
+#   - MTP4,  max_model_len 262144: reserve 3.43 GiB <  need 3.62 GiB  -> refuses
+#     ("To serve at least one request with the model's max seq len ...")
+# Raising max_model_len does not fix it: the cap's slope (~13,845 B/token) only
+# overtakes the need's (~12,700 B/token + 0.52 GiB fixed) around 440k tokens,
+# where the KV reserve would eat a third of the expert arena.
+#
+# THE FIX: let an explicit operator KV request raise the cap. The compile and
+# graph-capture headroom still comes off the top, so the blast radius is one
+# flag the operator chose.
+#
+# APPLY (into a host dir that is then bind-mounted read-only over the image):
+#   IMAGE=docker.io/tcclaviger/vllm@sha256:da4029666668b90a0386c1ed23ea370c79916ad4e1a6ad1dd9db098b9fa069ea
+#   D=/home/dank/glm-patch; mkdir -p $D/vllm/v1/worker
+#   podman run --rm --entrypoint cat $IMAGE \
+#     /opt/vllm/lib/python3.14/site-packages/vllm/v1/worker/gpu_worker.py \
+#     > $D/vllm/v1/worker/gpu_worker.py
+#   ( cd $D && patch -p1 < .../scripts/patches/gpu_worker-kv-cap.patch )
+#   # then add to the podman run:
+#   #   -v $D/vllm/v1/worker/gpu_worker.py:/opt/vllm/lib/python3.14/site-packages/vllm/v1/worker/gpu_worker.py:ro
+#   #   -e CLAV_KV_RESERVE_FLOOR_GIB=<needed>   (or pass --kv-cache-memory-bytes)
+#
+# Base file sha256 (the pristine file out of the pinned image):
+#   1f3942b1a01af075317161b06077a2e658086f1da0566c9f297129ceebe37dd7
+# Patched file sha256 (verified 2026-09-30, patch applies clean and reproduces
+# these bytes exactly):
+#   80cd160bbc01270b0851f7b6fe7552b06ae2fae2428134fc29238d7f452af840
+#
+--- a/vllm/v1/worker/gpu_worker.py
++++ b/vllm/v1/worker/gpu_worker.py
+@@ -824,6 +824,21 @@
+         expert_cfg = self.vllm_config.offload_config.expert
+         if expert_cfg.enable_expert_offload and expert_cfg.planned_kv_reserve_bytes > 0:
+             cap = int(expert_cfg.planned_kv_reserve_bytes)
++            # LOCAL PATCH (GLM-5.3 KDA + MTP draft): the plan models KV as
++            # 13600 B/token + a small mamba term, but this model also carries a
++            # per-sequence KDA state and (under MTP) the draft's KV, so the
++            # reserve can sit BELOW what max_model_len legitimately needs -- and
++            # the cap then makes a legal config unbootable (measured: need
++            # 2.07 GiB vs cap 1.74 GiB at max_model_len 131072 with MTP). An
++            # explicit --kv-cache-memory-bytes, or CLAV_KV_RESERVE_FLOOR_GIB,
++            # is the operator's word: let it raise the cap. The compile/capture
++            # headroom still comes off the top.
++            _explicit = self.cache_config.kv_cache_memory_bytes
++            _floor = os.environ.get("CLAV_KV_RESERVE_FLOOR_GIB")
++            if _explicit:
++                cap = max(cap, int(_explicit))
++            if _floor:
++                cap = max(cap, int(float(_floor) * (1 << 30)))
+             if self.available_kv_cache_memory_bytes > cap:
+                 logger.info_once(
+                     "expert offload: KV capped at the plan's reserve %s GiB; %s GiB of "

scripts/serve-frank-large.sh

@@ -1,107 +1,179 @@
 #!/usr/bin/env bash
-# Serve unsloth/Qwen3.8-27B-GGUF:UD-Q8_K_XL as frank-large on
+# Serve kingjones777/Qwen3.8-27B-ROCmFP4-STRIX-MTP-GGUF as frank-large on
-# Dankclaw's Strix Halo iGPU (gfx1151), using stock llama.cpp
+# Dankclaw's Strix Halo iGPU (gfx1151) with the ROCmFPX fork of llama.cpp.
-# (mainline, the unsloth b11030 prebuilt).
 #
-# Runtime: /home/dank/GitPackages/unsloth-llama-b11030/llama-server
+# 2026-10-02 (Frank): swapped from gufo + Q4_K_XL to the highest-quality file
-# (mainline, ROCm gfx1151 prebuilt). The build supports the qwen3_5
+# in the repo, one session at the native 262,144 window.
-# architecture and the DFlash2 drafter (llama_model_qwen35,
-# llama_model_dflash, common_speculative_impl_draft_dflash).
 #
-# Shape: one stream, full 262144 context (the model's native window), all
+# Why this runtime and not the previous one: the payload is ftype 115 — 311
-# layers on the iGPU (--n-gpu-layers 999). The 27B dense model (31.46 GiB GGUF) + the
+# tensors of GGML_TYPE_Q8_0_ROCMFPX (103), a tensor type that exists only in
-# DFlash2 drafter (1.14 GiB) + q8_0 KV (~8.6 GiB) + compute buffers fits the
+# the ROCmFPX fork. Stock llama.cpp, SGLang and gufo all refuse the file, and
-# iGPU's GTT aperture (~114 GiB).
+# gufo additionally gates every "Qwen3.8" artifact on a template-hash
+# allowlist. The qualified build is scripts/build-rocmfpx.sh's pin: it loads
+# ftype 115, survives warmup (the 2026-08-25 builds of the older fork line
+# segfaulted in ggml_compute_forward_set_rows on this exact file), and drafts
+# with the model's own MTP head.
 #
-# Spec method: DFlash2 (a block-diffusion drafter) at depth 4. The drafter
+# Why this file: Q8_0_ROCMFPX_AGENT is the top of the repo's quality ladder
-# has a sliding window of 2048 (small draft KV, ~40 MiB).
+# (26.28 GiB, 8.39 bpw) and the AGENT routing keeps 187 tensors — the
+# output-head set the draft scores against — at plain Q8_0 instead of the 2
+# the non-AGENT build keeps, which is the variant the card names for MTP
+# serving. Drafting uses the repo's Q4_0 MTP head: the head's cost is its
+# projection over the 248K vocabulary, so the 4-bit head is faster end to end
+# at equal-or-better acceptance than its Q8_0 sibling.
 #
-# KV: q8_0 quantized (--cache-type-k q8_0 --cache-type-v q8_0). The unsloth b11030
+# The client contract lives here, not in frank-large.env:
-# build does not support fp8 KV (allowed: f32/f16/bf16/q8_0/q4_0/q4_1/iq4_nl/q5_0/
+#   * --jinja — without it llama.cpp silently IGNORES the harness's
-# q5_1); q8_0 is the highest-precision 8-bit option and preserves the ~8 GiB footprint.
+#     chat_template_kwargs (enable_thinking / preserve_thinking), so the
-#
+#     endpoint would look like it had no thinking control at all. The GGUF
-# Text-only (no mmproj) — matching the prior frank-large.
+#     carries the official Qwen template, so no --chat-template-file.
-#
+#   * --reasoning-format deepseek — thinking is emitted as reasoning_content,
-# The Qwen3.8 family template keeps prior thinking blocks across turns when
+#     the field the harness reads for this endpoint.
-# preserve_thinking is set; pass --jinja and the kwarg so multi-turn agents
+#   * the template accepts reasoning_effort xhigh|medium|low only, which is
-# do not lose the trail. The template gates reasoning_effort to {xhigh,
+#     why models.yml folds the harness's levels onto that set.
-# medium, low} — the harness folds its levels onto that set in models.yml
+#   * --alias frank-large — that is the model id the harness resolves.
-# (reasoningEffortMap).
+#   * --ctx-checkpoints / --cache-ram — prompt-cache geometry for the 16
-#
+#     sliding-window-style layers; harmless where the hybrid state forces a
-# Tunables come from frank-large.env unless FRANK_LARGE_ENV_FILE overrides
+#     full re-process, and required for the checkpoint feature to work at all.
-# it; anything not set there falls through to the defaults below.
 set -euo pipefail
 
-SCRIPT_DIR="$(cd -- "$(dirname -- "${BASH_SOURCE[0]}")" && pwd)"
+# Fleet values live in frank-large.env. Anything already in the environment
-ENV_FILE="${FRANK_LARGE_ENV_FILE:-$SCRIPT_DIR/frank-large.env}"
+# wins over the file, so an operator can override one value on a single run:
+# the caller's values are captured, the file is sourced, then they are put
+# back. (A plain `source` would let the file clobber the environment.)
+TUNABLES=(LLAMA_BIN MODEL_FILE DRAFT_FILE MMPROJ_FILE ALIAS BIND_ADDR
+          LISTEN_PORT CTX_SIZE PARALLEL BATCH_SIZE UBATCH_SIZE NGL FIT
+          FLASH_ATTN CACHE_TYPE_K CACHE_TYPE_V SPEC_TYPE SPEC_DRAFT_NGL
+          SPEC_DRAFT_DEVICE SPEC_DRAFT_N_MAX SPEC_DRAFT_N_MIN SPEC_DRAFT_P_MIN
+          TEMPERATURE TOP_P TOP_K MIN_P PRESENCE_PENALTY NUM_THREADS
+          THREAD_BATCH ROCM_PATH HIP_VISIBLE HSA_GFX_VERSION)
+_OVERRIDES=()
+for _var in "${TUNABLES[@]}"; do
+    [[ -n "${!_var+set}" ]] && _OVERRIDES+=("$_var=${!_var}")
+done
+ENV_FILE="${FRANK_LARGE_ENV_FILE:-$(dirname "$0")/frank-large.env}"
 if [[ -f "$ENV_FILE" ]]; then
     set -a
+    # shellcheck disable=SC1090
     source "$ENV_FILE"
     set +a
 fi
-
+for _kv in "${_OVERRIDES[@]+${_OVERRIDES[@]}}"; do
-# --- defaults (used when the env file does not set them) ------------------
+    export "${_kv?}"
-: "${RUNTIME_DIR:=/home/dank/GitPackages/unsloth-llama-b11030}"
-: "${MODEL_DIR:=/home/dank/models/Qwen3.8-27B-UD-Q8_K_XL}"
-: "${HOST:=0.0.0.0}"
-: "${PORT:=7770}"
-: "${CONTEXT_SIZE:=262144}"
-: "${PARALLEL_SLOTS:=1}"
-: "${SPEC_DRAFT_N_MAX:=4}"
-: "${CACHE_RAM:=8192}"
-TEMPLATE_FILE="${TEMPLATE_FILE:-$SCRIPT_DIR/qwen-sharp-v22.4.0.jinja}"
-
-SERVER_BIN="$RUNTIME_DIR/llama-server"
-MODEL_FILE="$MODEL_DIR/Qwen3.8-27B-UD-Q8_K_XL.gguf"
-DRAFTER_FILE="$MODEL_DIR/Qwen3.8-27B-DFlash2-Q4_K_M.gguf"
-
-if [[ ! -x "$SERVER_BIN" ]]; then
-    echo "runtime not found at $SERVER_BIN" >&2
-    exit 1
-fi
-for f in "$MODEL_FILE" "$DRAFTER_FILE"; do
-    if [[ ! -f "$f" ]]; then
-        echo "missing asset $f (run scripts/download-frank-large.sh)" >&2
-        exit 1
-    fi
 done
+unset _var _kv
 
-# --- env (the unsloth build's ROCm setup) --------------------------------
+: "${LLAMA_BIN=/home/dank/GitPackages/kingjones-ROCmFPX/build/bin/llama-server}"
-export HIP_VISIBLE_DEVICES=0
+: "${MODEL_FILE=/home/dank/models/kingjones-Qwen3.8-27B-ROCmFP4-STRIX-MTP/Qwen3.8-27B-Q8_0_ROCMFPX_AGENT.gguf}"
-export HSA_OVERRIDE_GFX_VERSION=11.5.1
+: "${DRAFT_FILE=/home/dank/models/kingjones-Qwen3.8-27B-ROCmFP4-STRIX-MTP/mtp-Qwen3.8-27B-Q4_0.gguf}"
-export LD_LIBRARY_PATH="/opt/rocm/lib:/opt/rocm/llvm/lib:${LD_LIBRARY_PATH:-}"
+: "${MMPROJ_FILE=/home/dank/models/kingjones-Qwen3.8-27B-ROCmFP4-STRIX-MTP/mmproj-Qwen3.8-27B-BF16.gguf}"
+: "${ALIAS=frank-large}"
+: "${BIND_ADDR=0.0.0.0}"
+: "${LISTEN_PORT=7770}"
+: "${CTX_SIZE=262144}"
+: "${PARALLEL=1}"
+: "${BATCH_SIZE=2048}"
+: "${UBATCH_SIZE=512}"
+: "${NGL=999}"
+: "${FIT=off}"
+: "${FLASH_ATTN=on}"
+: "${CACHE_TYPE_K=f16}"
+: "${CACHE_TYPE_V=f16}"
+: "${SPEC_TYPE=draft-mtp}"
+: "${SPEC_DRAFT_NGL=99}"
+: "${SPEC_DRAFT_DEVICE=ROCm0}"
+: "${SPEC_DRAFT_N_MAX=4}"
+: "${SPEC_DRAFT_N_MIN=0}"
+: "${SPEC_DRAFT_P_MIN=0.0}"
+: "${TEMPERATURE=1.0}"
+: "${TOP_P=0.95}"
+: "${TOP_K=20}"
+: "${MIN_P=0.0}"
+: "${PRESENCE_PENALTY=0.0}"
+: "${NUM_THREADS=16}"
+: "${THREAD_BATCH=16}"
+: "${ROCM_PATH=/opt/rocm}"
+: "${HIP_VISIBLE=0}"
+: "${HSA_GFX_VERSION=11.5.1}"
 
-# --- the serve command ---------------------------------------------------
+[[ -n "$LLAMA_BIN" ]] || { echo "frank-large: LLAMA_BIN is empty" >&2; exit 1; }
-ARGS=(
+[[ -x "$LLAMA_BIN" ]] || {
-    --host "$HOST"
+    echo "frank-large: llama-server '$LLAMA_BIN' is not executable;" >&2
-    --port "$PORT"
+    echo "           build it with scripts/build-rocmfpx.sh" >&2
-    --model "$MODEL_FILE"
+    exit 1
-    --alias frank-large
+}
-    --ctx-size "$CONTEXT_SIZE"
+# Refuse anything but the real payload. A path that is not a readable file
-    --parallel "$PARALLEL_SLOTS"
+# makes llama-server hang rather than fail clean.
-    --n-gpu-layers 999
+[[ -r "$MODEL_FILE" ]] || {
-    --jinja
+    echo "frank-large: model file '$MODEL_FILE' is not readable;" >&2
+    echo "           fetch it with scripts/download-frank-large-kingjones.sh" >&2
+    exit 1
+}
+# The MTP head is the performance feature: without it the Q8_0 weights decode
+# at memory-bandwidth speed and the endpoint loses the ~1.4x the head buys.
+[[ -r "$DRAFT_FILE" ]] || {
+    echo "frank-large: MTP draft head '$DRAFT_FILE' is not readable;" >&2
+    echo "           fetch it with scripts/download-frank-large-kingjones.sh" >&2
+    exit 1
+}
+# The GGUF carries no vision tower; the projector file is the whole tower, so
+# a missing file means a vision request would 500 rather than degrade.
+# MMPROJ_FILE="" is the documented opt-out for a text-only endpoint.
+MM_ARGS=()
+if [[ -n "${MMPROJ_FILE:-}" ]]; then
+    [[ -r "$MMPROJ_FILE" ]] || {
+        echo "frank-large: mmproj '$MMPROJ_FILE' is not readable; set MMPROJ_FILE='' for text-only" >&2
+        exit 1
+    }
+    MM_ARGS=(--mmproj "$MMPROJ_FILE")
+fi
+
+# HIP/ROCm host environment. HIP reads bare env vars (TEMP, HOST,
+# GPU_MAX_HEAP_SIZE), which is why the tunables above are named TEMPERATURE
+# and BIND_ADDR. HSA_OVERRIDE_GFX_VERSION is what lets a runtime built for a
+# neighbouring gfx target run on this iGPU; the qualified pin targets gfx1151
+# natively and the override is a no-op for it.
+export ROCM_PATH
+export HIP_VISIBLE_DEVICES="$HIP_VISIBLE"
+export HSA_OVERRIDE_GFX_VERSION="$HSA_GFX_VERSION"
+export LD_LIBRARY_PATH="$(dirname "$LLAMA_BIN"):${ROCM_PATH}/lib:${ROCM_PATH}/lib/rocm_sysdeps/lib:${ROCM_PATH}/lib/llvm/lib${LD_LIBRARY_PATH:+:$LD_LIBRARY_PATH}"
+
+exec /usr/bin/env "$LLAMA_BIN" \
+    --model "$MODEL_FILE" \
+    "${MM_ARGS[@]}" \
+    --alias "$ALIAS" \
+    --host "$BIND_ADDR" \
+    --port "$LISTEN_PORT" \
+    --jinja \
+    --device ROCm0 \
+    --n-gpu-layers "$NGL" \
+    --parallel "$PARALLEL" \
+    --ctx-size "$CTX_SIZE" \
+    --batch-size "$BATCH_SIZE" \
+    --ubatch-size "$UBATCH_SIZE" \
+    --threads "$NUM_THREADS" \
+    --threads-batch "$THREAD_BATCH" \
+    --fit "$FIT" \
+    --flash-attn "$FLASH_ATTN" \
+    --direct-io \
+    --cache-type-k "$CACHE_TYPE_K" \
+    --cache-type-v "$CACHE_TYPE_V" \
+    --ctx-checkpoints 16 \
+    --cache-ram 8192 \
+    --cache-idle-slots \
+    --temp "$TEMPERATURE" \
+    --top-p "$TOP_P" \
+    --top-k "$TOP_K" \
+    --min-p "$MIN_P" \
+    --presence-penalty "$PRESENCE_PENALTY" \
+    --reasoning on \
+    --reasoning-format deepseek \
+    --reasoning-budget -1 \
+    --spec-type "$SPEC_TYPE" \
+    --model-draft "$DRAFT_FILE" \
+    --spec-draft-ngl "$SPEC_DRAFT_NGL" \
+    --spec-draft-device "$SPEC_DRAFT_DEVICE" \
+    --spec-draft-n-max "$SPEC_DRAFT_N_MAX" \
+    --spec-draft-n-min "$SPEC_DRAFT_N_MIN" \
+    --spec-draft-p-min "$SPEC_DRAFT_P_MIN" \
     --metrics
-    --chat-template-file "$TEMPLATE_FILE"
-    --chat-template-kwargs '{"preserve_thinking":true}'
-    --cache-type-k q8_0
-    --cache-type-v q8_0
-    --cache-ram "$CACHE_RAM"
-    # DFlash2 spec decode (a block-diffusion drafter)
-    --spec-type draft-dflash
-    --model-draft "$DRAFTER_FILE"
-    --spec-draft-n-max "$SPEC_DRAFT_N_MAX"
-    --spec-draft-n-min 0
-    --spec-draft-device ROCm0
-    # Greedy sampling (the DFlash2 drafter needs greedy verification)
-    --temp 0
-    --top-p 1.0
-    --top-k 0
-    --min-p 0.0
-    # Reasoning (thoughts land in reasoning_content)
-    --reasoning auto
-    --reasoning-format deepseek
-    --reasoning-budget -1
-)
-
-exec "$SERVER_BIN" "${ARGS[@]}"

scripts/serve-frank-primary.sh

@@ -1,30 +1,31 @@
 #!/usr/bin/env bash
-# serve-frank-primary.sh — frank-primary endpoint (R9V radiance-vLLM fork,
+# serve-frank-primary.sh — frank-primary endpoint (tcclaviger RDNA4 vLLM fork,
-# unsloth Qwen3.8-Flash-Next Q8_0 + MTP4 drafter, 2x R9700, dankserver
+# orcarouter Qwen3.8-Flash-Next-Uncensored-FP8 + MTP4 spec decode, 2x R9700,
-# :7770; 2026-09-27 cutover from the uncensored IQ4_XS payload, whose state
+# dankserver :7770; 2026-09-29 cutover from the R9V radiance fork — git tag
-# dir stays on disk as the rollback target).
+# frank-primary-r9v-q8-20260929 is the pre-cutover state).
 #
-# The engine is the upstream R9V launcher (scripts/launch.sh in the fork
+# The engine is driven by `podman run` DIRECTLY: the image's entrypoint execs
-# checkout), NOT a hand-frozen podman argv: launch.sh owns the resolved
+# `vllm serve "$@"`, so the run line below IS the engine argv. This replaces
-# docker/podman args (binds, overlays, device passthrough, publish lines)
+# the R9V launcher/shim machinery (that fork needed a docker->podman shim and
-# and the fork's qualification was measured through it. This script's job:
+# a 104-key on-host setup state; this stack is fully expressed by flags, and
+# every flag here was qualified by boot probe on 2026-09-29 — see
+# frank-primary.env for the qualification record).
 #
-#   1. layer the on-host R9V setup state (the 104-key tuned shape the
+# Shape (2026-10-02, Frank's correction of the 10-01 bf16 shape): MAXLEN
-#      profile setup qualified) under this repo's frank-primary.env
+# 200704 x SEQS 2 (401,408 tokens of contract), KV fp8 (--kv-cache-dtype
-#      overrides (bind/port/alias/container);
+# fp8; the bf16 unquantized shape is retired to git tag
-#   2. gate on the pinned runtime image (by its OCI manifest digest, the
+# frank-primary-fp8-mtp-20261001), KV_BYTES 3,150,000,000 PER RANK (the
-#      podman RepoDigest), the exact-size PLE payload, and the GGUF shards;
+# engine measured 405,000 tokens of capacity / ~3.0x concurrency), NO
-#   3. exec the launcher, then wait on the container: launch.sh --detach
+# speculative decode, --max-num-batched-tokens 1024 (Frank's lock), MoE
-#      exits as soon as the container is created, so without the wait the
+# gather geometry R4D_LRU_GATHER_CHUNKS=32 /
-#      unit would be "green" while the engine crashes — the wait is what
+# R4D_LRU_GATHER_LANES=64 (+18.7% decode, grows with depth), prefix
-#      keeps systemd's Restart=on-failure self-healing real.
+# caching, expert prefetch OFF (36% precision measured), TunableOp off, and
+# the pool deliberately small so the expert arena keeps what it reclaims.
 #
 # The root unit (scripts/frank-primary.service, User=dank,
-# LimitMEMLOCK=infinity) grants the pin budget for the tiered-expert UVA
+# LimitMEMLOCK=infinity) grants the pin budget for the tiered-expert host
-# host copy and the PLE pinned residency (54.4 GB on the q8 payload). A
+# slabs (~20.5 GiB/rank of pinned anonymous memory). A plain shell session
-# plain shell session also works (proven 2026-09-27: podman rootless
+# also works, but the unit is the fleet-parity boot-persistent shape.
-# inherits usable limits here), but the unit is the fleet-parity
-# boot-persistent shape.
 set -euo pipefail
 
 SCRIPT_DIR="$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)"
@@ -39,104 +40,96 @@ set -a
 source "$ENV_FILE"
 set +a
 
-: "${R9V_HOME:?R9V_HOME must name the fork checkout}"
+: "${FRANK_PRIMARY_IMAGE:?FRANK_PRIMARY_IMAGE must name the digest-pinned image}"
-: "${R9V_STATE_FILE:?R9V_STATE_FILE must name the setup state}"
+: "${FRANK_PRIMARY_MODEL_DIR:?FRANK_PRIMARY_MODEL_DIR must name the FP8 payload}"
-: "${R9V_HOST_BIND:?R9V_HOST_BIND must be set}"
+: "${FRANK_PRIMARY_MODEL_SHARDS:?FRANK_PRIMARY_MODEL_SHARDS must give the shard count}"
-: "${R9V_HOST_PORT:?R9V_HOST_PORT must be set}"
+: "${FRANK_PRIMARY_CONTAINER:?FRANK_PRIMARY_CONTAINER must name the container}"
-: "${R9V_SERVED_MODEL_NAME:?R9V_SERVED_MODEL_NAME must be set}"
+: "${FRANK_PRIMARY_SERVED_NAME:?FRANK_PRIMARY_SERVED_NAME must name the alias}"
-: "${R9V_CONTAINER_NAME:?R9V_CONTAINER_NAME must be set}"
+: "${FRANK_PRIMARY_PUBLISH:?FRANK_PRIMARY_PUBLISH must be host:container}"
-
+: "${FRANK_PRIMARY_MAXLEN:?FRANK_PRIMARY_MAXLEN must be set}"
-LAUNCHER="$R9V_HOME/scripts/launch.sh"
+: "${FRANK_PRIMARY_SEQS:?FRANK_PRIMARY_SEQS must be set}"
-if [ ! -x "$LAUNCHER" ]; then
+: "${FRANK_PRIMARY_KV_BYTES:?FRANK_PRIMARY_KV_BYTES must be set (PER RANK)}"
-    echo "serve-frank-primary: R9V launcher missing: $LAUNCHER" >&2
+: "${FRANK_PRIMARY_KV_DTYPE:?FRANK_PRIMARY_KV_DTYPE must be set (auto = bf16 KV)}"
+: "${FRANK_PRIMARY_REASONING_PARSER:?FRANK_PRIMARY_REASONING_PARSER must be set (qwen3)}"
+: "${FRANK_PRIMARY_TOOL_PARSER:?FRANK_PRIMARY_TOOL_PARSER must be set (qwen3_xml)}"
+: "${FRANK_PRIMARY_CHAT_TEMPLATE_KWARGS:?FRANK_PRIMARY_CHAT_TEMPLATE_KWARGS must be set}"
+# Capture podman's output BEFORE grepping: under `set -o pipefail`, the
+# classic `podman images | grep -q` pattern races — grep exits on its first
+# match, podman dies SIGPIPE (141) writing the remaining lines, and pipefail
+# turns the 141 into "no match" (the 2026-09-30 deploy refusal).
+DIGESTS="$(podman images --format '{{.RepoDigests}}')"
+if ! grep -qF "${FRANK_PRIMARY_IMAGE#*@}" <<<"$DIGESTS"; then
+    echo "serve-frank-primary: pinned image absent from rootless storage: $FRANK_PRIMARY_IMAGE" >&2
+    echo "  fetch it: podman pull $FRANK_PRIMARY_IMAGE" >&2
     exit 1
 fi
 
-# The docker->podman shim is REQUIRED: the launcher and its doctor speak
+if [ ! -f "$FRANK_PRIMARY_MODEL_DIR/config.json" ]; then
-# docker (OCI-manifest-digest image IDs, LogConfig fields) that podman's
+    echo "serve-frank-primary: payload config missing: $FRANK_PRIMARY_MODEL_DIR/config.json" >&2
-# native CLI answers differently. The shim carries those laws.
-SHIM_DIR="$R9V_HOME/.podman-shim"
-if [ ! -x "$SHIM_DIR/docker" ]; then
-    echo "serve-frank-primary: docker->podman shim missing: $SHIM_DIR/docker" >&2
     exit 1
 fi
-export PATH="$SHIM_DIR:$PATH"
+SHARDS="$(find "$FRANK_PRIMARY_MODEL_DIR" -maxdepth 1 -name 'model-*-of-*.safetensors' | wc -l)"
-
+if [ "$SHARDS" != "$FRANK_PRIMARY_MODEL_SHARDS" ]; then
-# --- layer 1: the qualified setup state (production overrides never come
+    echo "serve-frank-primary: payload shard count wrong: $FRANK_PRIMARY_MODEL_DIR" >&2
-# from here; keys already defined in the env file are skipped) -------------
+    echo "  found $SHARDS safetensors, want $FRANK_PRIMARY_MODEL_SHARDS" >&2
-STATE_FILE="$R9V_STATE_FILE"
+    echo "  re-fetch: scripts/download-frank-primary.sh" >&2
-if [ ! -f "$STATE_FILE" ]; then
-    echo "serve-frank-primary: R9V setup state missing: $STATE_FILE" >&2
-    echo "  run the profile setup first (tools/setup_profile.py setup --state-dir ...)" >&2
     exit 1
 fi
-STATE_EXPORTS="$(python3 - "$STATE_FILE" <<'PY'
+if [ ! -f "$FRANK_PRIMARY_MODEL_DIR/model.safetensors.index.json" ]; then
-import json, shlex, sys, os
+    echo "serve-frank-primary: weight-map index missing: $FRANK_PRIMARY_MODEL_DIR/model.safetensors.index.json" >&2
-state = json.load(open(sys.argv[1]))
-config = state.get("config") or {}
-if state.get("ready") is not True:
-    sys.exit("R9V setup state is not ready; rerun setup before serving")
-env_keys = {k for k in os.environ if k.startswith("R9V_")}
-for key in sorted(config):
-    if key.startswith("R9V_") and key not in env_keys:
-        print(f"export {key}={shlex.quote(config[key])}")
-PY
-)"
-eval "$STATE_EXPORTS"
-
-: "${R9V_IMAGE:?state must carry R9V_IMAGE}"
-: "${R9V_MODEL_DIR:?state must carry R9V_MODEL_DIR}"
-: "${R9V_PLE_PATH:?state must carry R9V_PLE_PATH}"
-
-# --- gates: never serve a half-finished payload or a missing image --------
-# Runtime image by pinned OCI manifest digest (podman keys images by config
-# digest but preserves the manifest digest in RepoDigests — the shim's ID law).
-if ! podman images --format '{{.RepoDigests}}' | grep -qF "${R9V_IMAGE#sha256:}"; then
-    echo "serve-frank-primary: pinned R9V image absent: $R9V_IMAGE" >&2
-    echo "  load the pinned image bundle into podman rootless storage" >&2
     exit 1
 fi
 
-# PLE payload at its pinned byte count (stat -c %s, never wc -c), plus the
-# six target GGUF shards (mradermacher-style split parts) and the mmproj
-# the launcher mounts.
-PLE_BYTES="${R9V_PLE_EXPECTED_BYTES:-}"
-if [ -n "$PLE_BYTES" ]; then
-    ACTUAL="$(stat -c %s "$R9V_PLE_PATH" 2>/dev/null || echo 0)"
-    if [ "$ACTUAL" != "$PLE_BYTES" ]; then
-        echo "serve-frank-primary: PLE payload wrong size: $R9V_PLE_PATH" >&2
-        echo "  expected $PLE_BYTES, found $ACTUAL" >&2
-        exit 1
-    fi
-else
-    if [ ! -f "$R9V_PLE_PATH" ]; then
-        echo "serve-frank-primary: PLE payload missing: $R9V_PLE_PATH" >&2
-        exit 1
-    fi
-fi
-for rel in "$R9V_TARGET_REL" "${R9V_TARGET_SHARD2_REL:-}" "${R9V_TARGET_SHARD3_REL:-}" \
-    "${R9V_TARGET_SHARD4_REL:-}" "${R9V_TARGET_SHARD5_REL:-}" "${R9V_TARGET_SHARD6_REL:-}" \
-    "$R9V_MMPROJ_REL"; do
-    if [ -z "$rel" ]; then continue; fi
-    if [ ! -f "$R9V_MODEL_DIR/$rel" ]; then
-        echo "serve-frank-primary: payload file missing: $R9V_MODEL_DIR/$rel" >&2
-        exit 1
-    fi
-done
-
 if [ ! -e /dev/kfd ] || [ ! -d /dev/dri ]; then
     echo "serve-frank-primary: GPU device nodes absent (/dev/kfd, /dev/dri)" >&2
     exit 1
 fi
 
 # A crashed/killed unit can leave the container name registered (systemd
-# SIGKILLs conmon before cleanup runs); the launcher refuses an existing
+# SIGKILLs conmon before cleanup runs). Clear OUR name plus the legacy
-# name forever after. Clear any orphan of OUR name first — idempotent.
+# identities (the retired R9V container and the ad hoc experiment container
-docker rm -f "$R9V_CONTAINER_NAME" >/dev/null 2>&1 || true
+# both hold these GPUs); all idempotent.
+for CTR in $FRANK_PRIMARY_LEGACY_CONTAINERS "$FRANK_PRIMARY_CONTAINER"; do
+    podman rm -f "$CTR" >/dev/null 2>&1 || true
+done
 
-# launch.sh detaches after preflight; from here on, this script's lifetime
+LOG=/home/dank/logs/frank_primary.log
-# IS the container's: wait returns the container's exit code, so a crash is
+mkdir -p "$(dirname "$LOG")"
-# a nonzero ExecStart and Restart=on-failure fires. (systemd stop goes
+
-# through ExecStop in the unit — docker wait alone cannot stop the child.)
+# --device /dev/dri is the WHOLE DIRECTORY (individual renderD nodes enumerate
-"$LAUNCHER"
+# without the card nodes and HIP refuses contexts — "No CUDA GPUs are
-exec docker wait "$R9V_CONTAINER_NAME"
+# available" while device_count() says 2). --ipc=host for the engine's shared
+# memory; memlock unbounded for the pinned host slabs.
+# The entrypoint already execs `vllm serve "$@"`, so no `serve` here.
+podman run -d --name "$FRANK_PRIMARY_CONTAINER" \
+  --publish "$FRANK_PRIMARY_PUBLISH" \
+  --ipc=host --ulimit memlock=-1:-1 \
+  --cap-add SYS_PTRACE \
+  --device /dev/kfd --device /dev/dri --device /dev/mem \
+  -e HIP_VISIBLE_DEVICES=0,1 -e VLLM_ROCM_USE_AITER=0 \
+  ${FRANK_PRIMARY_GATHER_CHUNKS:+-e R4D_LRU_GATHER_CHUNKS="$FRANK_PRIMARY_GATHER_CHUNKS"} \
+  ${FRANK_PRIMARY_GATHER_LANES:+-e R4D_LRU_GATHER_LANES="$FRANK_PRIMARY_GATHER_LANES"} \
+  -v "$FRANK_PRIMARY_MODEL_DIR":/models:ro \
+  "$FRANK_PRIMARY_IMAGE" /models \
+  --served-model-name "$FRANK_PRIMARY_SERVED_NAME" --port 7770 \
+  --tensor-parallel-size 2 \
+  ${FRANK_PRIMARY_DCP:+--decode-context-parallel-size "$FRANK_PRIMARY_DCP"} \
+  --max-model-len "$FRANK_PRIMARY_MAXLEN" \
+  --max-num-seqs "$FRANK_PRIMARY_SEQS" \
+  ${FRANK_PRIMARY_MNBT:+--max-num-batched-tokens "$FRANK_PRIMARY_MNBT"} \
+  ${FRANK_PRIMARY_PRECISE_MEMORY:+--expert-precise-memory} \
+  --gpu-memory-utilization "$FRANK_PRIMARY_UTIL" \
+  --kv-cache-memory-bytes "$FRANK_PRIMARY_KV_BYTES" \
+  --kv-cache-dtype "$FRANK_PRIMARY_KV_DTYPE" \
+  --enable-prefix-caching \
+  --enable-auto-tool-choice --tool-call-parser "$FRANK_PRIMARY_TOOL_PARSER" \
+  --reasoning-parser "$FRANK_PRIMARY_REASONING_PARSER" \
+  --default-chat-template-kwargs "$FRANK_PRIMARY_CHAT_TEMPLATE_KWARGS" \
+  --trust-remote-code \
+  > "$LOG" 2>&1
+
+# From here on, this script's lifetime IS the container's: wait returns the
+# container's exit code, so a crash is a nonzero ExecStart and the unit's
+# Restart=on-failure is real. (systemd stop goes through ExecStop in the
+# unit — podman wait alone cannot stop the container child.)
+exec podman wait "$FRANK_PRIMARY_CONTAINER"

scripts/serve-frank-small.sh

@@ -1,39 +1,25 @@
 #!/usr/bin/env bash
-# Serve HauhauCS/Gemma4-12B-QAT-Uncensored-HauhauCS-Balanced (Q4_K_M) as
+# Serve bartowski/MiMo-V2.6-Distill-Qwen-9B-GGUF (Q5_K_M) as frank-small on
-# frank-small on Dankclaw's RTX 3090 via mainline llama.cpp.
+# Dankclaw's RTX 3090 via llama.cpp (master build, CUDA sm_86).
 #
-# Topology: the whole model lives in VRAM (-ngl 999) -- 6.87 GiB of Q4_K_M
+# 2026-10-01 night (Frank): swapped from SGLang + INT8-W8A16-AutoRound
-# weights plus the BF16 vision projector and q8_0 KV inside the card's
+# safetensors to the llama.cpp GGUF path, because llama.cpp is the only
-# 24 GiB. The host CPU only tokenizes and samples here; this box's CPU
+# runtime on this host that can serve the MiMo qwen35 hybrid with
-# belongs to the halogen engine that serves frank-large.
+# ASYMMETRIC KV: --cache-type-k f16 (unquantized keys) + --cache-type-v
+# q8_0 (quantized values), which is Frank's explicit KV contract for this
+# endpoint. SGLang's --kv-cache-dtype only moves K and V together.
 #
-# MTP: this model's Multi-Token Prediction head ships as a SEPARATE file
+# Architecture: qwen3_5 hybrid — 32 layers, 8 full-attention + 24 Gated
-# (mtp-gemma-4-12B-it.gguf, arch gemma4-assistant, 4 nextn layers). It is
+# DeltaNet, native 262144 window. KV lives only in the 8 attention layers;
-# passed via --model-draft with --spec-type draft-mtp: the draft context
+# the DeltaNet state is per-slot fixed size. Vision comes from the repo's
-# loads the assistant model. The same --spec-type value chains the
+# separate projector (the Q5_K_M GGUF carries no `v.*` tensors at all), so
-# draftless ngram-mod matcher, which attaches unconditionally; two
+# --mmproj is what gives this endpoint eyes; MMPROJ_FILE="" is the documented
-# context-backed drafters cannot share one process, so this chain is the
+# text-only opt-out.
-# only shape that carries both.
 #
-# Context: CTX_SIZE is set explicitly to PARALLEL x 262144 so every slot
+# Sampling: llama.cpp takes it per request (the fleet harness sends its
-# gets the model's FULL window; --fit (the CTX_SIZE-unset branch below)
+# recipe from models.yml); greedy is the harness default.
-# maximises the TOTAL pool instead, which would divide a smaller window
-# across slots. The 4-slot layout is measured to fit the card (see the
-# SIZING note in frank-small.env); more slots would fail the KV allocation.
 #
-# Vision + audio: --mmproj loads the BF16 projector, which also carries
+# Tunables come from frank-small.env unless FRANK_SMALL_ENV_FILE overrides.
-# an audio encoder — input_audio content parts transcribe verbatim
-# (verified 2026-09-17). The fleet's image path lives here, which is why
-# halogen no longer sets HALOGEN_VISION_TOWER.
-#
-# Sampling: greedy by default (the fleet contract). Clients that send their own
-# temperature get their own values, and the harness overrides per model from
-# models.yml. --jinja is pinned even though this build defaults it on: the
-# model's own template supplies reasoning_effort, preserve_thinking and the
-# tool-call grammar, and a future build that flips that default must not
-# silently take tools away.
-#
-# Tunables come from frank-small.env unless FRANK_SMALL_ENV_FILE overrides it.
 set -euo pipefail
 
 SCRIPT_DIR="$(cd -- "$(dirname -- "${BASH_SOURCE[0]}")" && pwd)"
@@ -46,95 +32,56 @@ if [[ -f "$ENV_FILE" ]]; then
 fi
 
 # --- defaults (used when the env file does not set them) -------------------
-: "${LLAMA_BIN:=/home/dank/GitPackages/llama.cpp-mainline/build-cuda/bin/llama-server}"
+: "${LLAMA_BIN:=/home/dank/GitPackages/llama-master/build-cuda/bin/llama-server}"
-: "${MODEL_DIR:=/home/dank/models/Qwen3.8-27B-GSQ-RCO-IQ3_S-mtp}"
+: "${MODEL_FILE:=/home/dank/models/MiMo-Q5KM/MiMo-V2.6-Distill-Qwen-9B-Q5_K_M.gguf}"
-: "${MODEL_FILE:=Qwen3.8-27B-GSQ-RCO-IQ3_S-mtp.gguf}"
-: "${MMPROJ_PATH:=$MODEL_DIR/mmproj-Gemma4-12B-QAT-Uncensored-HauhauCS-Balanced-BF16.gguf}"
-: "${DRAFT_FILE:=}"
 : "${ALIAS:=frank-small}"
 : "${BIND_ADDR:=0.0.0.0}"
 : "${LISTEN_PORT:=7771}"
-: "${CTX_SIZE:=}"
+# ctx-size is the POOL llama.cpp divides across slots: 540672 / 3 gives
-: "${PARALLEL:=1}"
+# 180224 per slot — three independent conversations (the native 262144 no
+# longer fits three times on a 24 GiB card once the vision tower is loaded;
+# frank-small.env carries the two measurements that set this number).
+: "${CTX_SIZE:=540672}"
+: "${PARALLEL:=3}"
 : "${CACHE_TYPE_K:=f16}"
-: "${CACHE_TYPE_V:=f16}"
+: "${CACHE_TYPE_V:=q8_0}"
-: "${CACHE_RAM:=8192}"
+: "${NGL:=99}"
-: "${BATCH_SIZE:=2048}"
+# Vision projector. No colon on purpose: an explicitly empty value means
-: "${UBATCH_SIZE:=512}"
+# "text-only, on purpose" and must NOT be replaced by the default path.
-# Decode-shape knobs: the MTP drafter's per-round proposal length (build
+: "${MMPROJ_FILE=/home/dank/models/MiMo-Q5KM/mmproj-MiMo-V2.6-Distill-Qwen-9B-f16.gguf}"
-# default 3; the Gemma4 sidecar carries 4 nextn layers) and the engine's
-# busy-wait level (build default 50).
-: "${SPEC_DRAFT_N_MAX:=3}"
-: "${POLL_LEVEL:=50}"
-: "${NUM_THREADS:=8}"
-: "${THREAD_BATCH:=8}"
-: "${NGL:=999}"
-: "${FLASH_ATTN:=on}"
-: "${SPEC_TYPE:=draft-mtp}"
-: "${TEMPERATURE:=0.0}"
-: "${REASONING_FORMAT:=deepseek}"
-: "${REASONING_BUDGET:=-1}"
 
-# Fail loudly rather than half-loading a model into VRAM: a missing file here
+# Fail loudly rather than half-loading: a missing file is a download that
-# is a download that has not finished, and the unit's restart policy would
+# has not finished, and Restart=always would turn it into a crash loop that
-# otherwise turn that into a crash loop that reads like a runtime fault.
+# reads like a runtime fault.
-for path in "$LLAMA_BIN" "$MODEL_DIR/$MODEL_FILE" "$MMPROJ_PATH"; do
+for path in "$LLAMA_BIN" "$MODEL_FILE"; do
     [[ -e "$path" ]] || { echo "frank-small: missing $path" >&2; exit 1; }
 done
-if [[ -n "${DRAFT_FILE:-}" ]]; then
+EXTRA_ARGS=()
-    [[ -e "$MODEL_DIR/$DRAFT_FILE" ]] || { echo "frank-small: missing $MODEL_DIR/$DRAFT_FILE" >&2; exit 1; }
+if [[ -n "$MMPROJ_FILE" ]]; then
+    [[ -e "$MMPROJ_FILE" ]] || { echo "frank-small: missing $MMPROJ_FILE" >&2; exit 1; }
+    EXTRA_ARGS+=(--mmproj "$MMPROJ_FILE")
 fi
 
 # The UVM device node is created lazily by nvidia-modprobe on the first CUDA
 # open after boot; make sure it exists before the server needs it.
 nvidia-modprobe -u -c=0 2>/dev/null || true
 
-MMPROJ_ARGS=()
+# q8_0 on V REQUIRES flash attention (the quantized-value kernels only exist
-if [[ -n "${MMPROJ_PATH:-}" && -f "$MMPROJ_PATH" ]]; then
+# on the FA path); an FA-less build silently falls back and the KV contract
-    MMPROJ_ARGS=(--mmproj "$MMPROJ_PATH")
+# in frank-small.env becomes a lie, so pin the flag pair here.
-fi
+ARGS=(
-
+    --model "$MODEL_FILE"
-SPEC_ARGS=()
+    --alias "$ALIAS"
-if [[ -n "${SPEC_TYPE:-}" && "$SPEC_TYPE" != "none" ]]; then
+    --host "$BIND_ADDR"
-    SPEC_ARGS=(--spec-type "$SPEC_TYPE" --spec-draft-n-max "$SPEC_DRAFT_N_MAX")
+    --port "$LISTEN_PORT"
-fi
+    --ctx-size "$CTX_SIZE"
-
+    --parallel "$PARALLEL"
-DRAFT_ARGS=()
+    --n-gpu-layers "$NGL"
-if [[ -n "${DRAFT_FILE:-}" ]]; then
+    --flash-attn on
-    DRAFT_ARGS=(--model-draft "$MODEL_DIR/$DRAFT_FILE")
+    --cache-type-k "$CACHE_TYPE_K"
-fi
+    --cache-type-v "$CACHE_TYPE_V"
-
+    --jinja
-# CTX_SIZE set -> pinned context. Unset -> --fit sizes the KV pool against
+    --reasoning auto
-# whatever the weights, projector, draft context and compute buffers leave
+    --reasoning-format deepseek
-# on the card; the engine prints the chosen value at startup.
+)
-if [[ -n "${CTX_SIZE:-}" ]]; then
+exec "$LLAMA_BIN" "${ARGS[@]}" ${EXTRA_ARGS[@]+"${EXTRA_ARGS[@]}"}
-    CTX_ARGS=(--ctx-size "$CTX_SIZE")
-else
-    CTX_ARGS=(--fit on)
-fi
-
-exec "$LLAMA_BIN" \
-    --model "$MODEL_DIR/$MODEL_FILE" \
-    "${MMPROJ_ARGS[@]}" \
-    "${DRAFT_ARGS[@]}" \
-    --alias "$ALIAS" \
-    --host "$BIND_ADDR" \
-    --port "$LISTEN_PORT" \
-    --gpu-layers "$NGL" \
-    --flash-attn "$FLASH_ATTN" \
-    "${CTX_ARGS[@]}" \
-    --parallel "$PARALLEL" \
-    --batch-size "$BATCH_SIZE" \
-    --ubatch-size "$UBATCH_SIZE" \
-    --threads "$NUM_THREADS" \
-    --threads-batch "$THREAD_BATCH" \
-    --poll "$POLL_LEVEL" \
-    --cache-type-k "$CACHE_TYPE_K" \
-    --cache-type-v "$CACHE_TYPE_V" \
-    --cache-ram "$CACHE_RAM" \
-    "${SPEC_ARGS[@]}" \
-    --jinja \
-    --metrics \
-    --temp "$TEMPERATURE" \
-    --reasoning-format "$REASONING_FORMAT" \
-    --reasoning-budget "$REASONING_BUDGET"

serverAppConfigs-docs/LLM/Dankclaw System Specs.md

@@ -101,8 +101,8 @@ cable between claw and server is required for 10G.
 
 - Unified memory eliminates separate VRAM limits. The GPU uses system RAM.
 - The GPU budget is the GTT aperture: ~114 GiB (`gtt_total` in `/sys/class/drm/card1/device/`; `llama-server --list-devices` reports 116,732 MiB). rocm-smi's smaller SVM/dedicated carve-out is **not** the budget — the 2026-08 "64 GiB rocm-smi" lesson is stale. Do not use `free` or `dmidecode` for the GPU budget.
-- The active service is `frank-large` on port 7770: stock llama.cpp (the unsloth b11030 prebuilt) serving unsloth/Qwen3.8-27B-GGUF:UD-Q8_K_XL (27B dense, q8_0 KV, DFlash2 drafter). See [[frank-large-unsloth-qwen38-27b-q8xl]].
+- The active service is `frank-large` on port 7770: **gufo** (gufo-org/gufo) v0.1.1, the Strix Halo inference engine, built from source at `/home/dank/GitPackages/gufo-v0.1.1/`, serving unsloth/Qwen3.8-27B-GGUF:UD-Q8_K_XL (27B dense, DFlash2 drafter). See [[frank-large-gufo-qwen38-27b]].
-- The active runtime (the unsloth b11030 prebuilt) **bundles its own ROCm runtime** next to the binary (`RPATH=$ORIGIN/../lib`), so it has no host SDK gate — the system ROCm 7.2.4 is not on its path. The retired CIRU venv's TheRock 7.15 SDK (`LD_LIBRARY_PATH`) is no longer in the serving path.
+- The active runtime **links against the host ROCm** (`/opt/rocm` → `/etc/alternatives/rocm` → 7.2.4) through its RPATH, so it needs the host HIP stack rather than a bundled one — the opposite of the retired llama.cpp prebuilt, which carried its own. Building it needs the ROCm dev packages that are installed, and two workarounds recorded in the runbook (drop `librocwmma-dev`; pin `--gcc-install-dir` to GCC 15). The retired CIRU venv's TheRock 7.15 SDK (`LD_LIBRARY_PATH`) is not in the serving path.
 - The retired vLLM venvs (`/home/dank/vllm-venv`, `/home/dank/vllm-vk-venv`) were moved to `/home/dank/.AGENTTRASH/` on 2026-08-18.
 - 1.1 TB free disk space is ample for model storage.
 

serverAppConfigs-docs/LLM/Dankserver System Specs.md

@@ -63,16 +63,16 @@ apt lists remain on disk but no NVIDIA PCI device exists.
 
 | Property | Value |
 |----------|-------|
-| System RAM | 60 GiB |
+| System RAM | **246 GiB** (upgraded for the Lucebox DeepSeek deployment of 2026-09-26; this page said 60 GiB until 2026-09-28 — a stale figure that survived the board swap) |
-| Swap | 63 GiB |
+| Swap | 63 GiB (`/swapfile`) |
 | /dev/shm | 32 GiB |
 
 ### Storage
 
 | Filesystem | Size | Used | Avail | Use% |
 |------------|------|------|-------|------|
-| / (nvme0n1p2) | 915G | ~640G | ~280G | ~70% |
+| / (nvme0n1p2) | 1.8T | 927G | ~822G | ~54% |
-| /mnt/entropy (dm-0) | 91T | 86T | 910G | 99% |
+| /mnt/entropy (dm-0) | 91T | 87T | **244G** | **100%** — full; do not plan model payload here |
 
 Models live under `/home/dank/models`; the serving build tree under
 `/home/dank/GitPackages/llama.cpp-dflash2` (the older `charlie-ROCmFPX` tree is

serverAppConfigs-docs/LLM/dankserver-frontier-moe-feasibility.md

@@ -0,0 +1,192 @@
+---
+type: research
+title: Frontier MoE on Dankserver — what fits, on which runtime
+description: VERIFIED 2026-09-28 by measurement and source reading. Compares DeepSeek-V4.1-Flash, MiMo-V2.6-Flash-MOPD and GLM-5.3-Flash against the 2x R9700 (gfx1201) / 246 GiB box, and assesses the Affinity runtime's portability off DeepSeek-V4-Flash. Records the capacity arithmetic that decides the answer before any model is loaded.
+tags:
+  - dankserver
+  - r9700
+  - gfx1201
+  - moe
+  - deepseek-v4.1
+  - deepseek-v4
+  - mimo
+  - glm
+  - affinity
+  - lucebox
+  - feasibility
+timestamp: 2026-09-28
+---
+
+# Frontier MoE on Dankserver — what fits, on which runtime
+
+> **Question this page answers.** Given Dankserver's hardware, which of
+> DeepSeek-V4.1-Flash, MiMo-V2.6-Flash-MOPD and GLM-5.3-Flash runs best, on
+> which runtime, and can Affinity / vllm-radiance / FreeToken-AMD be modified
+> to run them fast?
+>
+> **Answer in one line.** V4.1-Flash does not fit this box at any
+> configuration; the engine that does run it (Lucebox) measures
+> 0.8–9.1 tok/s on R9700 because decode is bound by PCIe expert streaming
+> rather than by the GPUs. The 284B **V4-Flash** on **Affinity** is the only
+> shape that clears 35 tok/s, at 110.
+
+Everything below was measured or read from source on 2026-09-28. Claims
+carried over from the engine authors are marked as theirs.
+
+## Hardware (measured live, 2026-09-28)
+
+| Property | Value |
+|---|---|
+| GPUs | 2x AMD Radeon AI PRO R9700 (Navi 48, **gfx1201**), 32,624 MiB each = **63.7 GiB** total |
+| Interconnect | PCIe 5.0 **x8** each, CPU root ports `00:01.1` / `00:01.3`, ~31.5 GB/s per card |
+| PCIe atomics | `CAP_EXP+28.w = 0x1440` (ReqEn+) on both endpoints — RCCL collectives are legal |
+| CPU | Ryzen 7 7700X — **8 cores / 16 threads** (small; CPU expert compute is expensive) |
+| RAM | **246 GiB** + 63 GiB swap |
+| Disk | 1.8 TB NVMe. `/mnt/entropy` is a separate 91 TB ext4 at **100%** — do not plan payload there |
+
+The knowledge base was wrong about RAM: [[Dankserver System Specs]] said
+60 GiB until 2026-09-28 and has been corrected. The Lucebox runbook's 246 GiB
+was right.
+
+## The three candidates
+
+| Model | Total / active | Routed experts | Attention & KV | Working gfx1201 runtime |
+|---|---|---|---|---|
+| **DeepSeek-V4.1-Flash** | 552B / 8B prefill, 16B decode | 384 top-6, 40 layers, `moe_intermediate_size` 2304, hidden 5120 | CED 20 enc + 20 dec; CSA2 Full/Reindex/Reuse; **FP4 main KV, 890 B/token**; Engram 196B at layers 1 & 14 | Lucebox `feat/ds41-engine` only (unmerged) |
+| **MiMo-V2.6-Flash-MOPD** | 309B / 15B | 256 top-8, no shared expert, 48 layers (39 SWA / 9 GA) | hybrid SWA (window 128) + GA; 5-layer MTP | vLLM / SGLang, NVIDIA-first. No AMD path |
+| **GLM-5.3-Flash** | 320B / 18B | MoE | KDA (linear) + DSA (sparse) hybrid; mHC | **tcclaviger vLLM fork — plausible native path (2026-09-29 source reading: `glm5next` registered with an `amd/` backend dir + DSA indexer + mHC path; r4d.so carries native mxfp4/dsfp4 MoE entries). Gate: a compressed-tensors MXFP4 payload (~165 GiB, none exists publicly) + a HIP boot probe.** llama.cpp GGUF (community R9700 guide) remains the verified-but-slower path |
+
+## Sizing — the arithmetic that decides it
+
+| Quantity | Value | Note |
+|---|---:|---|
+| DS4.1 official checkpoint | **475.27 GiB** | 48 shards, summed from the HF API |
+|   of which Engram | **189.13 GiB** | exactly two 94.56 GiB shards — the layer 1 and 14 lookup tables |
+| DS4.1 routed-expert params | 543.6 B | 40 x 384 x 3 x 5120 x 2304 |
+|   shipped as | 269 GiB (fp4) | DeepSeek's own `expert_dtype: fp4` |
+| Lucebox `ROCMFP23.gguf` | **377.51 GiB** | 179 GiB experts (2.5/3.5 bpw) + 189 GiB Engram + ~9.5 GiB dense |
+| V4-Flash official checkpoint | 148.67 GiB | **3.2x smaller than V4.1** |
+| Affinity container for V4-Flash | 101.1 GiB | 92.9 GiB experts @ 2.875 bpw + 6.2 dense + 2.0 embed |
+
+The Engram tables are **never mapped** by Lucebox, so they do not evict the
+expert page cache — that is the one piece of good news in the 377 GiB file.
+
+### The decode wall
+
+A V4.1 token selects `6 x 40 = 240` routed experts at ~11 MB each. At the
+residency the hot stack can hold (37 of 384 per layer, 9.6%), **~2.4 GB of
+expert weights cross PCIe per token**. PCIe 5.0 x8 does ~55 GB/s real:
+
+```
+2.4 GB / 55 GB/s = 44 ms  ->  ~23 tok/s absolute ceiling
+```
+
+That is **below the 35 tok/s target before any attention, Engram or dense work
+is counted.** Raising residency is the only lever, and VRAM is what bounds it:
+63.7 GiB / 11 MB = 5,800 of 15,360 experts = 37.8% maximum, and that assumes
+VRAM holds nothing else.
+
+The engine author measured the same wall on an R9700 with 61 GB of RAM
+(DS41.md, "Measured"): 0.8 tok/s cold / 1.8 tok/s warm at a 6 GB hot budget,
+0.9 / 1.9 at 16 GB, and 9.1 tok/s for the best tuned three-tier case. They
+name the cause: *"90 to 96% of the 240 routed experts a token selects are still
+cold, and decode moves roughly 2.4 GB of expert weights per token through the
+synchronous stream path. That, not the GPU, bounds decode today."*
+
+## Affinity — a DeepSeek-V4-Flash engine, and what a V4.1 port costs
+
+[Affinity](https://codeberg.org/StillDeadcode/affinity) is *"an inference
+engine for DeepSeek-V4-Flash on one or two RDNA4 GPUs"*, C++/HIP, no framework.
+**It builds clean on this host**: CMake 3.24+, `hipcc` 23, `HIP enabled:
+gfx1201`, zero errors, producing `affinity`, `aff-quantize`, `aff-info` and
+`aff-gpucheck`. Its README claims **~110 tok/s on two R9700s** against 8 on
+one. That is 3x Frank's 35 tok/s goal, for the 284B model.
+
+The 110 works because the 101 GiB container leaves the cold expert set in a
+**pinned host pool in RAM** — the SSD tier that Affinity calls "a cliff" is
+never reached. V4.1 would need 182 GiB of experts at the same 2.875 bpw, plus
+189 GiB of Engram, against the same 246 GiB of RAM.
+
+### What a V4.1 port would reuse
+
+The config parser is **fully data-driven** — every dimension is read from
+`config.json`, so nothing is compiled in at the config layer. And the expert
+tensor naming is *identical* between the two checkpoints:
+
+```
+layers.N.ffn.experts.E.w1.weight   (11,008 in V4  = 43 x 256)
+layers.N.ffn.experts.E.w1.weight   (15,360 in V4.1 = 40 x 384)
+```
+
+So the quantiser, the `.aff` container, the expert slab, the host pool and the
+heat/placement engine — **the part that produces the 110 tok/s** — all carry
+over. Hyper-connections are the same family (`hc_mult 4`, Sinkhorn 20).
+
+### What a V4.1 port would have to write
+
+Verified against both checkpoints' real `model.safetensors.index.json` on
+2026-09-28 — not inferred from the config prose.
+
+**Tensor presence.** `aff-quantize` requires 32 per-layer tensors. V4.1
+supplies 26 plus every global (`embed`, `head`, `norm`, `hc_head_*`) and the
+whole expert set. Six are missing:
+
+| Missing in V4.1 | Replaced in V4.1 by |
+|---|---|
+| `ffn.gate.tid2eid` | the Engram tables |
+| `attn.compressor.ape` | nothing — ratio-2 is *pooled, no APE* |
+| `attn.indexer.compressor.{wkv,wgate,ape,norm}.weight` | `attn.indexer.wk` + `attn.indexer.k_norm` |
+
+**Config keys.** Of the 35 Affinity reads, V4.1 supplies **34**. The one it
+lacks is the dangerous one:
+
+| Gap | Detail |
+|---|---|
+| **Engram — fails silently** | `num_hash_layers` (V4: 3) is absent, so `cfg_int` returns its default and `model.h:36`'s `n_hash_layer = 3` survives. It would route layers 0–2 through a table that does not exist: fluent output, wrong numbers, no error. This is the only silent failure; the other four gaps fail loudly. |
+| **Ratio vocabulary is disjoint** | V4 uses `{0, 4, 128}`; V4.1 uses `{0, 1, 2}` — **no overlap in the non-zero case**. Affinity computes `coff = (ratio == 4) ? 2 : 1`, so V4.1's 18 ratio-2 layers get the ratio-1 width; `comp_ape` is requested 38 times for a tensor that does not exist; and the `ratio == 4` indexer branch fires on **0** of the 8 layers that need one (`index_source_layer_ids` = 2, 8, 14, 20, 24, 28, 32, 36) — those 8 would run dense. |
+| **CED** | A 20-layer encoder feeding a 20-layer decoder whose global KV is projected from the final encoder hidden state, through an `aligner`. Affinity's `model.cpp` is a flat layer stack. |
+| **DSpark** | V4.1's draft carries **its own** 128-expert MoE at top-3 (`dspark_n_routed_experts`, `dspark_num_experts_per_tok`), 3 stages rather than 1. Affinity's draft shares the target's slab and routing. |
+| Vision | V4.1 is multimodal; Affinity has no vision path. Text-only is an acceptable loss. |
+
+V4.1 also adds 31 config keys Affinity never reads, including the whole
+`engram_*` block, `kv_source_layer_ids`, `index_source_layer_ids` and the
+candidate block pre-selection.
+
+**Verdict.** The reusable half is the *right* half — the expert quantiser,
+the `.aff` container, the slab, the host pool and the heat/placement engine
+are what produce the 110 tok/s. The rest is 6 tensor bindings, one new ratio
+vocabulary, the CED aligner, a new draft geometry and Engram: weeks of work
+in a third-party codebase, with no cheap way to prove correctness. And it
+meets the same capacity wall regardless, because 182 GiB of experts plus
+189 GiB of Engram against 246 GiB of RAM is exactly the problem Lucebox has.
+Not advisable.
+
+## The other runtimes
+
+| Runtime | gfx1201 | Architectures | Note |
+|---|---|---|---|
+| **Affinity** | yes, built here | **V4-Flash only** — "the geometry is compiled in; there is no architecture abstraction" | 110 tok/s for 284B. Best engine for this box by an order of magnitude, wrong model |
+| **tcclaviger vLLM fork** | **yes — proven 2026-09-29** | qwen4exp measured 37.4-40.6 tok/s decode (see [[dankserver-qwen38-fp8-expert-offload-tuning]]); **`glm5next` registered** (`Glm5NextForCausalLM`/`MTP`, backend-split `amd/`/`common/`/`nvidia/`, `sparse_indexer.py` for DSA); r4d.so has **native mxfp4 + dsfp4 MoE entries** (`has_dsfp4() == True`); W4A16 MoE class (`R4dW4a16MoEExperts`) wired into the same tiered offload; TunableOp silently skipped under offload (needs `CLAV_TUNABLEOP_WITH_OFFLOAD=1`, and then corrupts numerics — do not use) | the ad hoc qwen-fp8 container's runtime; **the old "MXFP4 = emulation/missing kernels" verdict predates this fork and is dead** — the GLM-5.3-Flash gate is now the payload format + a boot probe, not kernels |
+| **Lucebox** | yes, image on host | Qwen 3.8, DS4; **V4.1 on the unmerged `feat/ds41-engine` branch** | The only V4.1 path. **Refuses layer split and layer-major prefill for V4.1** |
+| **radiance-vLLM** (drwolfen) | yes — the active frank-primary | welded to `qwen4_exp`; PLE / QSA / packed-HC live inside that model class | 12–20 person-days per architecture |
+| **FreeToken-ROCm** | partial | block-FP8 safetensors only (modelopt); known `CpuMoeExecutor` issue under ROCm | cannot load a GGUF |
+| **llama.cpp** | yes | GLM-5.3 confirmed by a community R9700 guide; DS-V4.1 CED unverified | no expert LRU; **8 CPU cores** make CPU expert placement the wrong trade |
+| **TokenSpeed** | claimed | GLM-5.x incl. a 5.3-Flash recipe | NVIDIA-first, unverified on gfx1201 |
+| **KTransformers** | no | GLM-5.3 tutorial exists | built for Intel AMX + NVIDIA |
+
+The two-R9700 three-owner split (primary + secondary + stream) is **not
+available on this host**: the retired quadlet recorded that
+`LUCE_DS4_MOE_TP_INPROC` is qualified for gfx1151+gfx1201 pairs, *not*
+gfx1201+gfx1201. hip:0 takes dense weights and the hot expert stack;
+`LUCE_EXPERT_STREAM_DEVICE=1` gives hip:1 the streamed-expert slot cache.
+
+## Measured on Dankserver
+
+<!-- MEASUREMENT -->
+
+## See Also
+
+- [[frank-primary-lucebox-ds4]] — the retired Lucebox frank-primary runbook
+- [[Dual-R9700-Cooperation]] — the PCIe/atomics facts behind the board swap
+- [[Dankserver System Specs]] — the host (RAM corrected 2026-09-28)
+- [[frank-primary-r9v-radiance-qwen38-q8]] — the ACTIVE frank-primary

serverAppConfigs-docs/LLM/dankserver-qwen38-fp8-expert-offload-tuning.md

@@ -0,0 +1,398 @@
+---
+type: research
+title: Qwen3.8-Flash-Next FP8 on tcclaviger — expert-offload decode tuning
+description: VERIFIED 2026-09-29 by measurement. The flag-level decode-tuning campaign on the ad hoc qwen-fp8 container (orcarouter/Qwen3.8-Flash-Next-Uncensored-FP8, tcclaviger RDNA4 vLLM fork, 2x R9700): the >35 tok/s target is MET at 37.4-40.6 tok/s by turning prefetch OFF — telemetry measured its precision at 36%, wasting ~half of all gather traffic — while the original 26.8 tok/s incumbent (prefetch on) survived every other challenger. Residual upside is the fork's gather read path (effective ~6.5-9.5 GB/s vs 22.7 GB/s the link delivers at the same slab size).
+tags:
+  - dankserver
+  - r9700
+  - gfx1201
+  - qwen3.8
+  - fp8
+  - expert-offload
+  - speculative-decoding
+  - tuning
+timestamp: 2026-09-29
+---
+
+# Qwen3.8-Flash-Next FP8 — expert-offload decode tuning
+
+> **Question this page answers.** On the tcclaviger RDNA4 vLLM fork serving
+> `orcarouter/Qwen3.8-Flash-Next-Uncensored-FP8` across both R9700s, what
+> runtime flags actually move decode at a 32k working point?
+>
+> **Answer in one line.** **37.44–40.57 tok/s — the >35 target is met — by
+> deleting `--expert-prefetch 1`.** Expert-cache telemetry measured the
+> prefetcher at **36% precision** (36,720 slabs prefetched, 13,369 served):
+> ~half of all gather traffic was useless copies, and the copies it "saved"
+> would have been installs anyway. The prefetch-on incumbent (26.8 tok/s)
+> survived every other challenger measured: gather geometry, spec depth,
+> TunableOp, scheduler batch knobs, concurrency, ngram-only drafting.
+
+This is the ad hoc `qwen-fp8` container on Dankserver (rootless podman,
+image `docker.io/tcclaviger/vllm:latest`, engine
+`v0.29.0.dev0+g2bdbbc8080`), NOT the `frank-primary` systemd service —
+the R9V production unit is untouched, so no stop-and-mask was required for
+any of this. All numbers are decode tok/s measured from the server's own
+`usage.completion_tokens / (wall - ttft)` — SSE chunk counting is invalid
+under speculative decoding because rejected drafts change the accounting.
+
+## Deployment shape (the tuned incumbent)
+
+Serve script on the host (`/home/dank/serve_qwen_fp8.sh`), env hooks in
+CAPS. **PREFETCH must be unset** (omitting the flag; the fork's default is
+off):
+
+```
+MODEL=/home/dank/models/Qwen3.8-Flash-Next-Uncensored-FP8   (~172.8 GiB, 131 shards)
+IMAGE=docker.io/tcclaviger/vllm@sha256:da4029666668b90a0386c1ed23ea370c79916ad4e1a6ad1dd9db098b9fa069ea
+# image ID 3322dbbbc96d; local alias qwen-fp8-runtime:tuned-20260929;
+# the floating :latest tag moved once already — always launch by digest.
+# repo tag of this state: qwen-fp8-tuned-20260929
+--tensor-parallel-size 2  --max-model-len 32768  --gpu-memory-utilization 0.98
+--kv-cache-memory-bytes 2147483648  --max-num-seqs 4  --enable-prefix-caching
+--speculative-config '{"method":"mtp","num_speculative_tokens":4}'
+# NO --expert-prefetch: measured 36% precision, ~50% wasted gather traffic
+```
+
+Published to the host LAN via the serve script's `PORTMAP` hook
+(`PORTMAP=7773:7770`): health and completions answer on
+`http://192.168.0.235:7773` (verified host-side 2026-09-29; 256k config,
+decode 36.49/34.55/35.98 through the published path). Host port 7773 was
+chosen over 7770 because `frank-primary` (7770) is stopped but ENABLED —
+any future start would collide with a published 7770.
+
+Device notes carried from the bring-up: `--device /dev/dri` must be the
+WHOLE directory (individual renderD nodes enumerate but refuse contexts —
+"No CUDA GPUs are available" while device_count() reports 2); the image
+entrypoint execs `vllm serve` itself.
+
+The memory plan this produces (from the engine's own `expert_plan.py` log):
+
+| region | size |
+|---|---:|
+| expert cache space | 15.05 GiB = 6,574 slots x 2.344 MiB |
+|   VRAM-only resident layers | 3.52 GiB (layers 0-2, 1,536 experts) |
+|   swap region | 11.53 GiB (5,038 slots, 111-112/layer) |
+| host-backed experts | 52.74 GiB per rank, 23,040 experts |
+| KV reserve (one max-len request +8%) | 0.36 GiB of the 2 GiB pool |
+| CUDA graphs | 1.93 GiB (21 graphs; 1.32 GiB at `--max-num-seqs 2`) |
+| runtime overhead | 7.6 GiB (non-torch + buffers + activation) |
+
+## Measured arms (2026-09-29)
+
+Decode tok/s at 166 / 1087 / 4064 prompt tokens, median of 3 runs after a
+64-token warm pass, temperature 0, max_tokens 256:
+
+| Arm | 166 | 1087 | 4064 | Acceptance | Verdict |
+|---|---:|---:|---:|---:|---|
+| **MTP4, stock 8x16, PREFETCH OFF** | **40.57** | **37.44** | **37.57** | 54.0% | **NEW INCUMBENT — target met** |
+| MTP4, stock 8x16, prefetch ON | 26.80 | 25.15 | 26.81 | 60.7% | old incumbent (+51%/+40% worse) |
+| MTP4, prefetch ON (re-check) | 25.80 | 24.38 | 26.63 | 47.9% | reproduces (noise band ±1) |
+| MTP4, prefetch ON, gather 16x32 | 24.30 | 22.77 | 24.11 | 50.3% | −9% — ruled out |
+| MTP4, prefetch ON, gather 32x64 | 26.58 | 24.94 | 26.34 | 49.6% | parity — ruled out |
+| MTP3, prefetch ON | 24.47 | 24.41 | 24.44 | 66.7% | −9%, flat in ctx — ruled out |
+| MTP6, prefetch ON | 19.07 | 17.81 | 13.54 | 17.4% | collapses; crosses UVA threshold — ruled out |
+| MTP4, prefetch ON, MNBT 8192 | 29.32 | 24.12 | 23.65 | — | neutral — ruled out |
+| MTP4, prefetch ON, PREFETCH depth 2 | 25.83 | 25.06 | 25.87 | 56.0% | neutral — ruled out |
+| MTP4, prefetch ON, TunableOp w/ offload | 23.59 | (79.82*) | (79.89*) | (96.5%*) | **rejected — corrupts** |
+| 2-way concurrency, prefetch ON (2 streams) | — | — | 15.06 + 23.11 = **25.49 summed** | — | no gain vs 26.8 single |
+| ngram-only drafter | — | — | — | — | fork bug: PLE inputs unprepared |
+| 256k ctx (MAXLEN=262144, KV 4.5 / 4.03 GiB, prefetch off) | 36.65 / 36.48 | 34.21 / 34.13 | 34.77 / **35.02** | 53.0 / 39.4% | boots and serves; band 34.1-36.5 straddles 35 within noise; arena identical (5,648) at both KV sizes |
+
+\* The TunableOp numbers are invalid by construction: the tuned GEMMs
+changed generated behavior (repetition-loop, the server's
+"repetition detected" halt firing at ~140-146 tokens instead of the full
+256, acceptance jumping 60.7% → 96.5%), the 79.8 tok/s figures were
+measured on those halted degenerate completions, and the text-only check on
+a fresh prompt came back 20.01 tok/s — *below* baseline. GEMM autotuning
+that changes output text is broken tuning.
+
+The decode curve without spec decode is dead flat 12.11 tok/s from 304 to
+16,095 tokens — decode is context-independent on this shape, bound per-step
+by the expert gather, not by KV depth.
+
+## Telemetry — the measurement that found the win
+
+`R4D_LRU_TELEMETRY=1` arms per-layer counters; a torch-profiler window
+(`/start_profile` → traffic → `/stop_profile`) makes the fork log and write
+`expert_cache_rank<N>.json` beside the traces. **`torch_profiler_with_stack=false`
+is mandatory in the profiler config** — the default (true) grew a 6.5-min
+window's trace buffer ~40 GB and the kernel OOM-killed the engine core
+(anon-rss 95.2 GB, `oom-kill` in `journalctl -k`). The short window at
+166 ctx (~15 s of traffic) completed cleanly with `with_stack=false`.
+
+The fork's own numbers (102 steps, both ranks identical):
+
+```
+hit rate 0.8245 (137,084/166,262)   host installs 15,814
+read-through 13,364                 prefetched 36,720
+served by prefetch (migrations) 13,369
+```
+
+- **The LRU is good**: 82.45% hit rate overall; layers 0-2 pinned at
+  100%; swap-region layers 78-81%. The "arena holds only 27% of experts
+  → 73% miss" arithmetic was wrong — temporal reuse carries it.
+- **The prefetcher is the poison**: 36,720 slabs moved to serve 13,369
+  (36% precision). The 23,351 unused copies are ~0.7 GB/step/rank of pure
+  waste — roughly half of all gather traffic — and the 13,369 "savings"
+  would have been installs anyway. Depth-1 prefetch also cannot cover the
+  ~2,400-slab per-step working set with its 16 slots/layer (768 total).
+- Per-layer-step at a swap layer: ~30.6 routed, 24.8 hits, 3.3 installs,
+  2.6 read-through, 8.0 prefetched of which 1.7-3.3 useful.
+
+## The wall, measured
+
+- **PCIe link (pcie-link-probe):** GPUs `03:00.0`/`07:00.0` LnkCap/LnkSta
+  32GT/s x16; root ports `00:01.1`/`00:01.3` cap x8. Effective **Gen5 x8 =
+  31.5 GB/s/direction**. `rocm-smi pcie_link_speed: 25` is the ASPM idle
+  downshift, not the loaded rate.
+- **H2D (HIP probe, pinned):** 2.345 MiB (the engine's exact expert slab)
+  **22.74 GB/s**; 16 MiB 28.19; 1 GiB 28.74 (91% of ceiling). Pinned ≈
+  pageable — staging is not the lever. Single-transfer latency at slab size
+  105.7 µs vs 78 µs ideal → ~21% per-transfer overhead.
+- **Engine's effective gather rate:** ~6.5-9.5 GB/s = **30-42% of what the
+  link delivers at the same size**. The bottleneck is the fork's read path
+  (chunk/lane geometry, per-transfer cost), not the wire.
+- **Duty cycle:** peak GPU util 100%, smooth ~29% with prefetch on — each
+  step is ~100 ms of compute then hundreds of ms waiting on expert
+  streaming. rocm-smi reports *utilization*, which reads ~0 while weights
+  load during an arm's reload; VRAM is the honest panel there.
+
+## What the remaining flags cannot do (all measured)
+
+- **Gather geometry** — 16x32 and 32x64 measured no better despite the
+  fork's "tuned at the Gen4 ceiling" comment; the constants are already at
+  parity on Gen5 x8. The read-path gap is structural, not a tuning knob.
+- **Spec depth** — 4 is the optimum both ways (with prefetch on). Depth 6
+  additionally proves the UVA read-through path (`step tokens >
+  R4D_LRU_UVA_TOKENS=5`) is *slower* than copy-in.
+- **Prefetch anything** — depth 2 neutral, depth 1 the new negative; the
+  tier is 3x too small for the working set and the predictor is 36%
+  precise. `expert_prefetch_k/budget/slots` exist (defaults 20/14/16) but
+  the precision result makes tuning them pointless.
+- **Arena growth via shrink** — `UTIL=0.99` fails boot (`Free memory on
+  device cuda:1 (31.29/31.86)`); `--max-num-seqs 2` fails CUDA-graph
+  capture (`Constraints violated (L['query_start_loc'].size()[0])`); and
+  the MNBT 8192 arm already proved a ~1 GiB arena shift is decode-neutral
+  in both directions.
+- **Concurrency** — under TP=2 both streams' tokens are computed on both
+  cards, so each rank gathers the union of both streams' misses; the links
+  do not add capacity and the shared arena thrashes. Summed 2-way (25.49)
+  measured *below* single-stream (26.8) with prefetch on.
+- **ngram-only drafting** — fails at engine init:
+  `RuntimeError: PLE inputs were not prepared`; the fork prepares this
+  model's PLE inputs only on the MTP spec path. Needs a fork patch, not a
+  config.
+- **V-only q8 KV** — stock `--kv-cache-dtype` offers fp8 variants only
+  (K+V together); a V-only q8 needs a fork patch.
+
+## Fork bugs / traps found (for the engine authors or a future patch)
+
+1. `CLAV_TUNABLEOP_SWEEP=1` is silently skipped whenever expert offload is
+   active unless `CLAV_TUNABLEOP_WITH_OFFLOAD=1` is also set
+   (`tunableop_warmup.py`).
+2. TunableOp-with-offload, once forced on: +6.7 min warmup, no persistent
+   cache (container-local), and corrupt numerics (repetition loops,
+   early-EOS halts, invalid measurements). Rejected outright.
+3. The torch-profiler stop path hangs the workers on ROCm when
+   `torch_profiler_with_stack` is true (default): >10 min hangs on two
+   attempts while the AsyncLLM wrapper reports "stopped successfully" in
+   1 s; the default stack-tracing grew a 6.5-min window's buffer ~40 GB
+   and OOM-killed the engine core. With `with_stack=false` the stop
+   completes and the telemetry report runs. The telemetry counters remain
+   reachable ONLY through that profiler stop — no direct RPC trigger
+   exists (`telemetry_report` has exactly one caller, `gpu_worker.py`).
+4. ngram spec method does not prepare PLE inputs (above).
+
+## Where the headroom lives now
+
+The 35 tok/s target is met (37.4-40.6) with prefetch off. What is left:
+
+1. **Fork patch to the gather read path** — the ~6.5-9.5 → 22.7 GB/s
+   software gap at slab size. If the gather reached wire rate, decode
+   lands in the 60+ tok/s band before acceptance effects. Code work in
+   `expert_lru.py`'s gather (`R4D_LRU_GATHER_*` is read at import; the
+   copy-in path itself is compiled — see `/app/r4dhip/r4d.so`).
+2. **256k ctx — measured, final**: boots at `MAXLEN=262144` with
+   `KV_BYTES=4831838208` or `4328521728` alike — a 262,144-token window
+   needs **4.0 GiB** KV (hybrid attention + mamba pages ≈ 16 KiB/token,
+   not the 32k plan's 9,190 B/token; 3 GiB boot-refuses and the engine
+   estimates max len 184,224 at that pool; the exact-4.0 override loses
+   ~20 MB to accounting and boots as 3.98 < 4.0). Decode band
+   **34.1-36.5** across both KV sizes — the arena is **identical (5,648
+   slots)** at 4.5 and 4.03 GiB (the planner buckets; the trim went to
+   margin, not slots), so the window costs ~10% decode vs the 32k config
+   and no KV trim recovers it. The alternative that keeps the full
+   37.4-40.6 band is `MAXLEN=184224` at 3 GiB KV. Correction: the planner
+   sizes the arena AFTER the KV override (the earlier "before" reading was
+   wrong) — the arena is what remains.
+3. **2+ slots at long ctx** — the KV pool post-spec holds ~66k tokens at
+   2 GiB (~2x 32k streams); concurrency summed *below* parity at 4064 ctx,
+   so slots are a capacity feature, not a throughput one. Two full 256k
+   streams need ~8-10 GiB of KV — that fits the arithmetic only by cutting
+   the arena below usefulness; slots-at-256k is a fork-patch or
+   smaller-payload question.
+
+## Reproducing
+
+Host-side scratch scripts (2026-09-29): `/home/dank/serve_qwen_fp8.sh` (the
+serve script with all hooks), `/home/dank/mtp_measure.py` (usage-counter
+decode measure, 3 depths), `/home/dank/mtp_measure_short.py` (single depth,
+for profile windows), `/home/dank/bench2way.py` (concurrency), plus the arm
+scripts `arm_*.sh` / `sweep_*.sh` and the detached runner `/home/dank/job.sh`
+(outputs under `/home/dank/jobs/NAME.out|.rc|.done`). The measure scripts
+are `podman cp`'d into the container after every recreate (the serve script
+re-copies them; a recreate wipes container-local files).
+
+## 2026-10-01 — production cutovers (qwen FP8+MTP4, then bf16 no-spec)
+
+Two same-day cutovers moved this page's findings into production:
+
+1. **Qwen FP8 + MTP4 replaced the failed GLM/DFlash2 experiment** (the GLM
+   window is documented in [[frank-primary-glm53-flash-w4a16]]; git tag
+   `frank-primary-fp8-mtp-20261001` is the last commit of that shape). The
+   production trio took MAXLEN 200000 x 3, fp8 KV, MTP4, prefetch off —
+   measured 37.4-40.6 tok/s at ~54% acceptance.
+2. **The overnight redo campaign re-measured EVERY lever at matched
+   geometry** (prefetch OFF everywhere, MAXLEN 200000/SEQS 3/fp8 baseline)
+   and concluded the day: the incumbent cutover to **bf16 KV, no
+   speculative decode** (Frank's call).
+
+### What the redo campaign measured (all at shallow depth, 166/1087/4064)
+
+| lever | verdict |
+|---|---|
+| prefetch (re-confirmed) | the whole night's +40-51% lives in deleting it; the old 12.11 "no-spec baseline" was itself prefetch-taxed |
+| gather geometry | **R4D_LRU_GATHER_CHUNKS=32 / LANES=64 = +18.7%** (49.25 @4064), the win GROWS with depth (39.5@166 -> 54.1@1087); g8x16 mild, g64x128 past peak |
+| KV dtype | bf16 (`auto`) +4% decode on fp8-era numbers but halves attention-KV capacity (measured 14,047 vs 7,926 B/token; mamba states unaffected) |
+| spec family | no-spec fastest once the tax was gone (41.46/41.29/41.47); MTP4's drafter reserves ~2.6 GiB/rank beside the pool; untaxed MTP re-measures never completed |
+| MNBT | 512 doubles TTFT (2.46 s vs 1.36 s @4064); 1024-4096 interchangeable; 12288 -2.5% + arena; **locked 1024** |
+| UVA | 2048 POISON (-39%); 8192 ~parity |
+| arena | flat ±1.4% from ~6,830 down to ~5,500 slots; knee below |
+| six slots | every SEQS-6 KV sizing refused at engine init |
+
+Depth caveat: every number above is at a near-empty KV. Deep-context
+behavior (the gather win growing with depth, bf16's decode cost at depth)
+is only sampled at 3 depths; a 64k-192k-depth probe is the flagged gap.
+
+### The deployed shape (2026-10-02: 2 x 200704, fp8, pool cut for the arena)
+
+```
+FRANK_PRIMARY_MAXLEN=200704    x SEQS 2 (401,408 tokens of contract)
+FRANK_PRIMARY_KV_BYTES=3150000000   PER RANK (fp8, 7,701 B/token)
+FRANK_PRIMARY_KV_DTYPE=fp8   --kv-cache-dtype fp8 (quantized)
+FRANK_PRIMARY_MNBT=1024       --max-num-batched-tokens 1024
+R4D_LRU_GATHER_CHUNKS=32 / R4D_LRU_GATHER_LANES=64  (container env)
+NO --speculative-config, NO --expert-prefetch
+```
+
+This replaced the 2026-10-01 shape — 2 x 262144 at bf16 KV with
+`--expert-precise-memory` — which crash-looped at graph capture: the arena is
+DEVICE-resident, the precise-memory flag drops the plan's overhead 7.60 ->
+5.25 GiB, and that shape had 0.13 GiB to spare (weights 6.28 + pool 6.80 +
+graphs 0.55 + arena 16.65 + non-torch ~1.45 = 31.73 of 31.86 GiB,
+NRestarts=3 at ~6.7 min each). The plain bf16 2 x 262144 boot also did not
+stay up: it left 0 bytes free at runtime and a real client request (995-token
+prompt, max_tokens=261149) killed the worker with `Tried to allocate 50.00 MiB
+... 0 bytes free`.
+
+The correction is a pool cut, not a downgrade. fp8 costs 7,701 B/token
+(measured 6,138 attention + 1,563 mamba/side, dtype-independent), so 2 x
+200704 = 401,408 tokens needs 3.091e9 B; pinned at 3.15e9, ~44 MB of margin
+above the floor — the same convention the 6.1e9 shape used. The arena is what
+remains: the planner sizes it AFTER the KV override, so the 6,248 slots of the
+6.1e9 shape become roughly 7,500 here. Two contexts at 200704 are a capacity
+contract, not a throughput one (2-way concurrency summed BELOW parity at 4064
+ctx), so the slot buys the VRAM headroom long generations need.
+
+Boot-probed 2026-10-02 (transient systemd units at the production uid +
+memlock, full-log capture; the rootless bench probe had masked the production
+memory environment). The night searched two concurrency contracts:
+
+| rung | result |
+|---|---|
+| 3 x 262144, bf16, KV 11,047,010,304 | **OOM at graph capture** — `Tried to allocate 76.00 MiB ... 0 bytes free`, 30.43 GiB of 31.86 GiB already allocated by PyTorch |
+| 3 x 245760, bf16 | **OOM identically** |
+| 3 x 237568, bf16 | PASS — KV 718,606 tokens |
+| 2 x 262144, bf16 + `--expert-precise-memory` | **DEPLOYED then crashed** — 0.13 GiB free at boot, NRestarts=3 |
+| 2 x 262144, bf16 (no precise flag) | did not stay up either: 0 bytes free at runtime, worker killed by a real request |
+| **2 x 200704, fp8, pool cut for the arena** | **DEPLOYED (2026-10-02, Frank's correction)**: fp8 KV, two 200704 windows, ~7,500 arena slots, ~3.6 GiB free VRAM measured |
+7,926 B/token against bf16's ~13,900-14,047 — at the cost of bf16's ~4 %
+decode edge. Six full-length contexts cannot fit at any KV dtype: the 6-seq
+graph/activation footprint leaves only 9.18 GiB for the pool, and 6 × 262144
+needs 12.47 GB even at fp8.
+
+The old "KV_BYTES is PER RANK, 9/16 GiB OOM" law was the MTP4 WORKSPACE
+reservation, not a VRAM wall. Failures are clean and diagnostic: the engine
+reports `torch.OutOfMemoryError: Tried to allocate X GiB. GPU N has ... of
+which Y GiB is free`, and the fork's giant ERROR lines are split by journald —
+read them via `journalctl -o json`, not `-o cat` (and `podman logs`, which is
+only available while the container lives).
+
+Rollback to the 2026-09-29 fp8+MTP4 shape: git tag
+`frank-primary-fp8-mtp-20261001` (env trio + seam re-pinned there).
+
+## Parallelism options on this box (2026-10-01 survey)
+
+Surveyed from the running image's own CLI surface. Verdicts are specific to
+dankserver: 2x R9700 (gfx1201), a ~172.8 GiB expert-offload MoE, and a model
+with only 2 KV heads.
+
+| option | flag | verdict on this box |
+|---|---|---|
+| TP | `--tensor-parallel-size` | **DEPLOYED, =2.** All-reduce dispatch shows `R4D_AR` then `PYNCCL` |
+| DCP | `--decode-context-parallel-size` | **REJECTED at TP=2** — see below |
+| PP | `--pipeline-parallel-size` | supported; untried (stage handoff plus a TP-keyed expert plan) |
+| DP | `--data-parallel-size` (+`-rank`, `-start-rank`) | impossible: every replica holds a full model copy (172.8 GiB vs 31.86 GiB/card) |
+| EP | `--enable-expert-parallel` + `--all2all-backend` (+`--enable-eplb`, `--enable-dbo`, `--enable-elastic-ep`) | supported (`naive`, `allgather_reducescatter`, `deepep_high_throughput`, `deepep_low_latency`, `pplx`, `mori`, `nixl`); would replace the fork's tiered-offload LOCAL gather with a per-layer PCIe all-to-all |
+| multi-node | `--nnodes`, `--node-rank`, `--master-addr/port`, `--distributed-executor-backend` | supported; the actual path to more cards |
+
+### DCP is structurally unavailable here — do not re-test on 2 GPUs
+`--decode-context-parallel-size 2` fails `VllmConfig` validation at boot:
+
+```
+Decode context parallelism for GQA/MQA requires `--tensor-parallel-size` (2) to be
+greater than the model's total number of KV heads (2). Increase
+`--tensor-parallel-size` or set `--decode-context-parallel-size 1`.
+```
+
+DCP shards the KV along the *sequence* dimension after KV heads have been spread
+across the TP ranks, so it requires **TP > num_kv_heads**. This model has 2 KV
+heads (24 Q heads, GQA 12:1), so TP must be >= 4 — four GPUs, not two. Related
+fork facts: DCP *reuses* the TP ranks rather than multiplying them (it is not
+`TP x DCP`), and with PCP enabled the valid DCP sizes are `{1, pcp, tp x pcp}`
+(`vllm/config/parallel.py`). The KV memory a DCP win would have bought is real —
+each rank would hold half the sequence's KV — but it is unreachable without two
+more cards.
+
+### Measuring decode: the harness lives INSIDE the container
+`/home/dank/mtp_measure.py` imports `/bench_iter.py` by absolute path, so it must
+run in the engine's namespace and the files must be copied in first:
+
+```
+podman cp /home/dank/mtp_measure.py frank-primary-qwen:/
+podman cp /home/dank/bench_iter.py  frank-primary-qwen:/
+podman exec frank-primary-qwen python3 /mtp_measure.py
+```
+
+Without the copy the exec dies with `can't open file '/mtp_measure.py': [Errno 2]
+No such file or directory` — measured 2026-10-01, which silently cost a whole
+arm's numbers. A window script must do the copy, not assume the path exists.
+
+### The deployed shape, measured again (2026-10-01 A/B window)
+A clean production-environment boot of the deployed config (4 x 262144 fp8, MNBT
+2048 in that window) reported **`GPU KV cache size: 1,079,582 tokens, Maximum
+concurrency for 262,144 tokens per request: 4.12x`** — the same figures as the
+deployed MNBT-1024 shape, i.e. MNBT does not move the pool, and the capacity is
+reproducible across boots.
+
+## See Also
+
+- [[dankserver-frontier-moe-feasibility]] — the sibling feasibility page
+  (which frontier MoE fits this box; the same PCIe-expert-stream wall at
+  V4.1 scale)
+- [[frank-primary-r9v-radiance-qwen38-q8]] — RETIRED 2026-09-30 (the R9V
+  fork era; production has been the tcclaviger fork since)
+- [[x4-pair-experiment]] — the same residency law measured on a narrower
+  link: streaming over PCIe loses, computing in RAM does not care
+- [[Dual-R9700-Cooperation]] — the board swap that made TP=2 legal here

serverAppConfigs-docs/LLM/frank-large-gufo-qwen38-27b.md

@@ -0,0 +1,350 @@
+---
+type: reference
+title: "gufo, Qwen3.8-27B UD-Q4_K_XL (DFlash2) on Dankclaw (frank-large)"
+description: "The RETIRED frank-large runbook (rollback target) — gufo (gufo-org/gufo) v0.1.1, the Strix Halo inference engine, built from source on Dankclaw and serving unsloth/Qwen3.8-27B-GGUF UD-Q4_K_XL (vanilla) with the z-lab DFlash2 Q4_K_M drafter on the gfx1151 iGPU, ONE session at the full native 262,144 window with unquantized f16 KV: the from-source build and its two host workarounds, the GGUF header evidence, why the fleet's Qwen-Sharp chat template CANNOT be used on this runtime, the client-contract delta, the measured prefill/decode trade, and the failure modes worth knowing."
+tags:
+  - llm
+  - gufo
+  - swift-1.5
+  - dankclaw
+  - frank-large
+  - strix-halo
+  - gfx1151
+  - q8
+  - dflash2
+  - rocm
+timestamp: 2026-09-28
+---
+
+# frank-large — gufo, Qwen3.8-27B (UD-Q4_K_XL, DFlash2)
+
+> **RETIRED 2026-10-02 — the rollback target.** Superseded by
+> [[frank-large-rocmfpx-q8agent-mtp]] (Q8_0_ROCMFPX_AGENT at 8.39 bpw with
+> MTP4, 17.0 t/s measured). This era was itself a return after one day of
+> [[frank-large-vllm-mimo-v26-bf16]] (battery decode 6.19-9.01 t/s; every
+> ≥6-bit vLLM quant path dead on gfx1151). The shape
+> moved on the way back: payload is now **unsloth/Qwen3.8-27B-GGUF
+> UD-Q4_K_XL** (16.35 GiB, vanilla — the abliterated Swift Q8 was
+> reclaimed), and the session grid is **one session at the full native
+> 262,144 window** (`SESSIONS=1 CONTEXT_SIZE=262144`, f16 KV). The
+> Swift-1.5 Q8 history below is retained for the engine/template law,
+> which still governs.
+> **Originally ACTIVE 2026-09-28.** Replaces
+> [[frank-large-unsloth-qwen38-27b-q8xl]] (stock llama.cpp, the unsloth
+> b11030 prebuilt), which had replaced
+> [[frank-large-kairic-edge-qwen38-27b]]. Same host, same port (7770), same
+> alias (`frank-large`).
+>
+> **The payload also changed on 2026-09-28**, later the same day: the target
+> went from `unsloth/Qwen3.8-27B-GGUF:UD-Q8_K_XL` to
+> `ajgazin/Swift-1.5-Qwen3.8-27B-Uncensored-Dynamic-MTP-GGUF`'s
+> `UD-Q8_K_XL` — an abliterated (uncensored) fine-tune of the same base. See
+> [Payload](#payload) and, for what that costs, [the template
+> law](#the-template-law-qwen-sharp-is-gone).
+
+## Identity
+
+- **Endpoint:** `frank-large` at `192.168.0.143:7770` (Dankclaw).
+- **Runtime:** [gufo](https://github.com/gufo-org/gufo) — "the Strix Halo
+  inference engine", a vertical engine built for exactly this hardware
+  (Ryzen AI MAX+ 395 / Radeon 8060S, `gfx1151`, unified memory). Pinned at
+  **v0.1.1** (commit `b0f8673`), built from source on the host to
+  `/home/dank/GitPackages/gufo-v0.1.1/`.
+- **Service shape:** root systemd unit `scripts/frank-large.service`
+  (`User=dank`, `LimitMEMLOCK=infinity`), launched by
+  `scripts/serve-frank-large.sh` with tunables from
+  `scripts/frank-large.env`.
+
+gufo ships **no prebuilt Linux binaries** (the v0.1.0/v0.1.1 release assets
+are empty) and no Nix on this host, so the runtime is a source build. It is
+also young: v0.1.0 landed 2026-09-28, the same day as this cutover.
+
+## Why this engine
+
+gufo's own measurement of the two engines on the same Qwen3.8-27B targets
+(Strix Halo, `628ed18e` vs llama.cpp `b11069`, greedy, DFlash2):
+
+| Q8_K_XL, DFlash2 | gufo | llama.cpp | delta |
+|---|---:|---:|---:|
+| prefill, depth 0 | 457.05 t/s | 333.85 t/s | **+36.9 %** |
+| prefill, depth 131,072 | 237.24 t/s | 131.80 t/s | **+80.0 %** |
+| decode, depth 0 (mixed) | 16.08 t/s | 15.00 t/s | +7.2 % |
+| decode, depth 32,768 (mixed) | 14.41 t/s | 14.44 t/s | −0.2 % |
+| decode, depth 65,536 (mixed) | 10.87 t/s | 12.62 t/s | **−13.9 %** |
+| decode, depth 131,072 (mixed) | 8.87 t/s | 11.62 t/s | **−23.7 %** |
+| cold load to HTTP-ready | 8.17 s | 36.47 s | **4.5× faster** |
+| peak memory @ 262,144 ctx | 50.58 GiB | 48.60 GiB | +4 % |
+
+Read that honestly: **prefill, load time and short-context decode improve;
+deep-context decode on the 8-bit target gets worse.** The crossover on Q8 is
+around 32k of cached depth. frank-large runs one 262,144-token conversation,
+so a long agent session spends its life on the losing side of that line
+while a short turn wins. The Q4_K_XL target does not show the regression
+(gufo is +11.7 % to +17.0 % at every depth) — that is the honest reason to
+revisit the quantization separately from the runtime.
+
+## Build
+
+`scripts/build-gufo.sh` reproduces the runtime end to end. It clones the
+pinned tag, configures the `release` preset (HIP, `--offload-arch=gfx1151`),
+builds, installs, and finishes with `gufo diagnose`. Two host facts it
+encodes, both found the hard way on 2026-09-28:
+
+- **Ubuntu's `librocwmma-dev` must not be installed.** It ships a truncated
+  rocWMMA header set (the public headers without `rocwmma/internal/`) into
+  `/usr/include`, where it shadows AMD's complete copy under
+  `/opt/rocm/include`. The first build died on `'internal/accessors.hpp'
+  file not found` from `/usr/include/rocwmma/rocwmma.hpp`, and **a `-I`
+  cannot fix it**: in HIP mode the host compilation pass searches the system
+  directories first. The package has no reverse dependencies; the script
+  removes it.
+- **`--gcc-install-dir` must point at GCC 15.** Ubuntu 26.04 has
+  `/usr/include/c++/16`, and the ROCm clang picks the newest tree, whose
+  `<atomic_wait.h>` references `std::__detail::__notify_impl` at a symbol
+  version the installed `libstdc++.so.6` (GLIBCXX_3.4.35) does not export.
+  The link then fails with `undefined symbol`. Pinning the headers to the
+  GCC the runtime library came from fixes it.
+
+Verify an installed runtime with `bash scripts/build-gufo.sh --verify`, or
+`gufo diagnose` directly — it reports `[PASS]` plus the detected
+architecture, driver and queue budget.
+
+## Shape
+
+### Payload
+
+- **Model:** `ajgazin/Swift-1.5-Qwen3.8-27B-Uncensored-Dynamic-MTP-GGUF`,
+  file `UD-Q8_K_XL` — 27B dense, 31,457,990,272 bytes, fetched and
+  sha256-verified (`9076f99a…`, the Hub LFS oid) by
+  `scripts/download-frank-large-swift.sh`. It is an **abliterated
+  (uncensored)** fine-tune of Qwen3.8-27B. The repository also publishes an
+  `mmproj-BF16.gguf`, which is **not** used: this endpoint is text-only.
+- **Licence:** the repo declares `other` — the **SWIFT OPEN LICENSE 1.0**,
+  which is *not* an OSI licence. That is a real constraint on redistribution,
+  noted here so nobody discovers it later. It is fine for a private fleet.
+- **Replaced:** `unsloth/Qwen3.8-27B-GGUF:UD-Q8_K_XL`, still on disk and
+  still the rollback target for the payload.
+
+The two targets are **structurally interchangeable for this engine**, which
+is why the swap needed no engine change. Header-diffed directly:
+
+| | unsloth | Swift-1.5 |
+|---|---|---|
+| `general.architecture` | `qwen35` | `qwen35` |
+| tensors (count, names, order) | 866 | 866, **identical** |
+| per-tensor `ggml_type` | Q8_0×453, F32×360, BF16×53 | **identical** |
+| per-tensor dims | — | **0 mismatches** |
+| MTP head | 4 tensors at `blk.64.nextn.*` | identical |
+| geometry | 65 blk, 5120 embd, 17408 FFN, 24/4 heads | identical |
+
+The two metadata differences that look alarming are both no-ops, and both
+were confirmed against gufo's source rather than assumed:
+
+- **`general.file_type` is 15 vs 7.** gufo's `gguf_reader.cpp` states it is
+  *"deliberately not consulted… The Unsloth UD-\*_K_XL artifacts this repo
+  targets carry a stale one — UD-Q8_K_XL records ftype 15 (Q4_K_M) … The
+  tensor table always is."* The file gufo already served lies about itself
+  here, so gufo reads the tensor table.
+- **Swift adds `qwen35.attention.recurrent_layers`.** gufo never reads that
+  key anywhere in `src/`; layer kinds come from `full_attention_interval`,
+  which both files carry as `4` and which agrees with it.
+
+### Serving shape
+
+- **Context:** one session, full 262,144 tokens (the model's native window),
+  `SESSIONS=1`. The fleet's single stream is unchanged.
+- **Spec method:** DFlash2, gufo's **adaptive** block policy
+  (`--draft-policy adaptive`, `--draft-tokens 7`). The drafter is the
+  **unchanged** z-lab `Qwen3.8-27B-DFlash2-Q4_K_M` (1.07 GiB) — it did not
+  move with the target, which is why `DRAFTER_DIR` is a separate tunable
+  from `MODEL_DIR` and why the deployer's stray-run sweep keys on the
+  drafter's filename rather than the target's.
+- **Sampling:** greedy (`--temperature 0`) server-side, as every prior
+  frank-large served. The harness overrides per request via `extraBody`
+  (1.0 / 0.95 / top-k 20).
+- **Memory:** gufo's own table puts the 8-bit target at **50.58 GiB** peak
+  with context capacity 262,144, against the iGPU's ~114 GiB GTT aperture.
+
+### Measured at the 2026-09-28 cutover
+
+Live on Dankclaw. The **target** numbers are the Swift-1.5 load test
+(a standalone run under the production flags, service stopped first); the
+**engine** numbers are from the gufo cutover earlier the same day and are
+unchanged by the payload swap.
+
+- **9.2 s** `load_started` → `load_completed` for the Swift target, against a
+  20-minute `LLM_WAIT` budget. The unsloth target loaded in 13 s.
+- **66.5 GiB** of the 116.7 GiB iGPU device at context capacity 262,144;
+  host RSS 30.6 GiB. No loader warnings, no skipped sections.
+- Contract battery, all green on the Swift target: `/v1/models` →
+  `context_length 262144`, a chat completion carrying both `content` and
+  `reasoning_content`, a `tool_calls` response with
+  `finish_reason: tool_calls`, SSE streaming, and a **two-turn tool loop**
+  (tool call → tool result → follow-up) with prior reasoning preserved.
+  Arithmetic spot-check: "17*23" → `391`.
+- Behavioural spot-check that this is the abliterated model: asked for a
+  two-sentence lockpicking scene, it wrote one without hedging or refusing.
+- From the gufo cutover, on the unsloth target: a short-prompt live request
+  returned
+  `Server-Timing: ttft;dur=549.8, inter_token;dur=24.2` (≈41 tok/s).
+  Short-prompt decode is gufo's *winning* regime; see the depth table above
+  before quoting it as the endpoint's number.
+
+### Tuning: the served flags are already optimal (2026-09-28)
+
+Frank asked for iterative decode/prefill improvement at a 32k working point
+without touching the model or the quantization. **Result: no win exists at
+this engine's served settings, and nothing was changed.** Measured on real
+repo prose and shell sources at **32,762 prompt tokens**, salted per run so
+a warm prefix cache could not help, 3–5 repetitions each.
+
+| lever | range swept | outcome |
+|---|---|---|
+| `--prefill-chunk` | 512 → 8192 | **no effect** — 410.7–412.4 t/s prefill across a 16× change |
+| `--draft-tokens` | 5 → 10 | **flat** — 13.79–15.86 t/s decode, inside run-to-run noise |
+| `--draft-policy` | fixed vs adaptive | 35.90 vs 35.50 (bench) — no effect |
+| `--min-draft-tokens` | — | **rejected by the engine**: "DFlash2 requires … 1" |
+| MTP backend | — | **unsupported for Q8**: `token_embd.weight` is Q8_0 |
+| `--context` 262144 → 40960 | diagnostic | 13.93 vs 14.71 t/s — no gain, and it would break the declared client context |
+
+The confirmation round at 5 reps **reversed** the 3-rep leaders: the pair
+that looked best (`dt9 + chunk4096`) came last at 12.86 t/s, and the
+**deployed configuration came first at 14.71 t/s**. Every difference sits
+inside the noise band.
+
+What *does* move the number is speculative decoding itself: **6.65 t/s off →
+~15 t/s on, a 5.3× step** — and that is already switched on. Prefill is
+insensitive to everything tried, which is consistent with gufo's own
+`EXPERIMENTS.md`, where dozens of kernel-level alternatives are recorded as
+rejected for "no release throughput improvement".
+
+**Do not repeat this investigation without new information.** Two traps
+cost most of the time, and both are in AGENTS.md's benchmarking policy:
+`gufo bench` reports `acceptance=0.933 ± 0.000` on an identical trace hash
+and **inverts** the draft-width ranking; and a live A/B needs ≥128 generated
+tokens or `inter_token` is first-token noise.
+
+## The template law: Qwen-Sharp is gone
+
+`scripts/qwen-sharp-v22.4.0.jinja` is the fleet's chat template — it is
+wired into `frank-primary` as `R9V_CHAT_TEMPLATE_FILE` and it was
+`frank-large`'s `--chat-template-file` under llama.cpp. **frank-large can no
+longer use it**, and the reason is stronger than "gufo lacks the flag":
+
+> gufo has **no Jinja engine at all.** `src/models/qwen/chat_template.cpp`
+> is 587 lines of hand-written C++ that appends strings to build ChatML.
+> There is no `--chat-template-file`, no environment override, and no
+> template body anywhere in the binary to point at. Qwen-Sharp is 453 lines
+> of Jinja2 using `namespace()`, macros, `raise_exception` and image/video
+> branches — supplying it would mean **reimplementing it in C++**, not
+> handing gufo a file.
+
+Given the choice on 2026-09-28, the decision was to **ship on gufo's stock
+Jinja and accept the loss**. What that gives up, concretely:
+
+| Qwen-Sharp feature | On gufo |
+|---|---|
+| `terse` system-prompt injection (the headline behaviour) | **gone** |
+| in-band `<\|think_off\|>` / `<\|think_xhigh\|>` control | **gone** |
+| `tool_call_format` `json` \| `xml` switch | **gone** |
+| `max_tool_arg_chars` / `max_tool_response_chars` truncation | **gone** |
+| `add_vision_id` → `Picture N:` prefixes | **gone** |
+| effort instruction strings in the system prompt | kept (gufo has its own, same text) |
+| `preserve_thinking` | kept (`--preserve-thinking on`) |
+
+The `terse` injection is the one that matters most here, and it is worth
+being blunt about the combination: this endpoint now runs an **abliterated**
+model *without* the prompt that tells it to stop waffling. If verbosity
+shows up, the cheapest fix is to send the `terse` text as an ordinary
+**system message from the client** (`models.yml` / the harness) rather than
+through a template — that recovers the behaviour without forking the
+renderer. The full-fidelity options are reverting to llama.cpp or porting
+Qwen-Sharp into the C++ renderer; both are worse trades today, and the
+second is the same class of change that produced the 2026-09-21
+"73,600-token prompt" incident.
+
+## Client contract — what actually changed
+
+Everything the fleet depended on still holds: port 7770, alias `frank-large`,
+`reasoning_content`, `tool_calls`, streaming SSE, and text-only (gufo
+auto-discovers a `mmproj` beside the model; this payload has none).
+
+The differences worth knowing:
+
+- **No custom chat template — see
+  [The template law](#the-template-law-qwen-sharp-is-gone) above**, which is
+  the most consequential change on this endpoint. Thinking defaults to on at
+  `xhigh` with prior reasoning preserved (`--think auto
+  --preserve-thinking on`, spelled out in the serve script).
+- **`reasoning_effort` accepts the full OpenAI set** (`none`, `minimal`,
+  `low`, `medium`, `high`, `xhigh`, `max`), all verified 200 against the
+  live endpoint. Qwen-Sharp rejected anything outside `{xhigh, medium, low}`
+  with an HTTP 500, which is why `models.yml` still carries a
+  `reasoningEffortMap` fold for this model. **The fold is kept on
+  purpose**: removing it would silently move every `high` role from `xhigh`
+  to `high` effort. Drop it as its own change.
+- **Every field the harness sends is accepted.** `top_k`, `min_p`,
+  `presence_penalty`, `repetition_penalty` and `preserve_thinking` are all
+  outside gufo's documented request subset, and all returned 200. They are
+  applied from `models.yml`'s `extraBody` unchanged.
+- **No KV-cache quantization flag.** gufo picks its own KV dtype; there is
+  no `--cache-type-k`. The old `CACHE_RAM=8192` tunable is gone with it.
+  Prompt reuse is on by default, and `--cache-disk` (restart-safe) is opt-in
+  and currently off.
+- **Health/readiness split:** `GET /health` is process liveness and is what
+  the deployer gates on; `GET /ready` is model/backend readiness.
+  `GET /metrics` is Prometheus-format.
+
+- **Long bodies:** `--max-request-bytes` defaults to 8 MiB.
+
+## Failure modes worth knowing
+
+- **A missing runtime is a deploy error, not a restart loop.** The handler
+  gates on `LLM_LARGE_RUNTIME` *before* it stops anything, and the serve
+  script exits with a `build-gufo.sh` pointer rather than exec'ing.
+- **Host memory is the ceiling, and it is shared.** frank-large and ComfyUI
+  both hold gfx1151 compute queues. gufo reads
+  `/sys/class/kfd/kfd/proc/*/queues/*/type` and caps itself
+  (`GPU_MAX_HW_QUEUES`, 2 resident for `serve llm`) so a full device
+  time-slices instead of failing; four HIP processes is the device ceiling.
+- **Load is seconds, not minutes** (9.2 s measured cold on the Swift
+  target), so a redeploy no longer spends 20 minutes in `LLM_WAIT`.
+- **A payload is two files in two directories.** The target and the drafter
+  no longer live together, so "the payload is present" is not one check; the
+  deployer gates on both paths and the serve script refuses to start if
+  either is absent. A symlinked "one payload directory" would hide that.
+- **A v0.1.x runtime is a moving target.** Rebuilds are not automatic —
+  `RUNTIME_DIR` names a version, and the deployer will not silently
+  substitute another.
+
+## Rollback
+
+Two independent rollbacks, because two things changed on 2026-09-28:
+
+1. **Payload only** — repoint `MODEL_FILE` in `scripts/frank-large.env` at
+   `/home/dank/models/Qwen3.8-27B-UD-Q8_K_XL/Qwen3.8-27B-UD-Q8_K_XL.gguf`
+   and redeploy. **The 29.3 GiB target was reclaimed on 2026-09-28**, so this
+   path now needs a re-fetch first: `scripts/download-frank-large.sh`, then
+   the env edit. The DFlash2 drafter was never touched and still lives in
+   that directory.
+2. **Engine** — [[frank-large-unsloth-qwen38-27b-q8xl]]: the unsloth b11030
+   prebuilt is still at `/home/dank/GitPackages/unsloth-llama-b11030/`.
+   Revert `RUNTIME_DIR` and the gufo-only tunables, restore the serve
+   script's llama.cpp command, and redeploy. This is also the only way to
+   get `qwen-sharp-v22.4.0.jinja` back.
+
+# Citations
+
+- Runtime: https://github.com/gufo-org/gufo (pinned `v0.1.1`, MIT)
+- Measured tables: `docs/models/qwen3.8-27b/BENCHMARKS.md` in that repo
+- API contract: `docs/SERVER.md`; flags: `docs/CLI.md`; the C++ renderer
+  that rules out a Jinja template:
+  `src/models/qwen/chat_template.cpp` (587 lines)
+- GGUF header evidence for the payload swap: both files' headers were parsed
+  and diffed directly (architecture, all 866 tensor names in order, per-tensor
+  `ggml_type` and dims) before either was loaded
+- Build: `scripts/build-gufo.sh`; serve: `scripts/serve-frank-large.sh`;
+  tunables: `scripts/frank-large.env`
+- Payload: `scripts/download-frank-large-swift.sh` (target),
+  `scripts/download-frank-large.sh` (DFlash2 drafter)
+- Predecessor: [[frank-large-unsloth-qwen38-27b-q8xl]]

serverAppConfigs-docs/LLM/frank-large-rocmfpx-q8agent-mtp.md

@@ -0,0 +1,219 @@
+---
+type: reference
+title: "frank-large on Dankclaw — ROCmFPX llama.cpp, Qwen3.8-27B Q8_0_ROCMFPX_AGENT + MTP"
+description: "The active frank-large runbook — the kingjones30/ROCmFPX llama.cpp fork (pinned commit, gfx1151) serving kingjones777's Q8_0_ROCMFPX_AGENT GGUF, a ftype-115 file no other loader can read, with the repo's own MTP-Q4_0 head drafting four tokens at one native 262,144 window, f16 KV, and the bundled BF16 projector for vision. Measured MTP-vs-none A/B, the client contract, and rollback to gufo."
+tags:
+  - llm
+  - frank-large
+  - dankclaw
+  - llama-cpp
+  - rocmfpx
+  - gfx1151
+  - gguf
+  - spec-decode
+  - mtp
+timestamp: 2026-10-02
+---
+
+# frank-large — ROCmFPX Q8_0_ROCMFPX_AGENT + MTP (gfx1151)
+
+> **ACTIVE since 2026-10-02.** Cut over from
+> [[frank-large-gufo-qwen38-27b]] (Q4_K_XL, ~15 t/s). Both requests came from
+> Frank: *highest quality that runs*, and *one full context*. This endpoint
+> delivers 8.39 bpw weights — two quant rungs above what gufo could load — and
+> measures **17.0 t/s** decode instead of 15, so quality was not bought with
+> speed.
+>
+> The capability that forced this whole shape is a GGUF tensor type:
+> **`GGML_TYPE_Q8_0_ROCMFPX` (103)**. It exists in exactly one loader on this
+> host, so the runtime was decided by the file, not by preference.
+
+## Identity
+
+| | |
+|---|---|
+| Host / device | Dankclaw (192.168.0.143), Radeon 8060S iGPU `gfx1151`, ~114 GiB GTT aperture |
+| Endpoint | `http://192.168.0.143:7770` — alias `frank-large`, `/v1` OpenAI surface |
+| Runtime | **`kingjones30/ROCmFPX`** llama.cpp fork @ `36e9acd40e10a87cd3c3ef8ec734668757dc8520`, built by [`scripts/build-rocmfpx.sh`](../../scripts/build-rocmfpx.sh) → `/home/dank/GitPackages/kingjones-ROCmFPX/build/bin/llama-server` |
+| Unit | root systemd `frank-large.service` ([`scripts/frank-large.service`](../../scripts/frank-large.service)), `User=dank`, `Restart=always` |
+| Serve script | [`scripts/serve-frank-large.sh`](../../scripts/serve-frank-large.sh) (owns the argv), tunables [`scripts/frank-large.env`](../../scripts/frank-large.env) |
+| Weights | `/home/dank/models/kingjones-Qwen3.8-27B-ROCmFP4-STRIX-MTP/Qwen3.8-27B-Q8_0_ROCMFPX_AGENT.gguf` — 28,214,425,952 B, sha256 `f3ef2bdbf56e244f…` |
+| Draft | `…/mtp-Qwen3.8-27B-Q4_0.gguf` — 1,680,271,648 B, sha256 `051a1764cff8c4f3…` |
+| Projector | `…/mmproj-Qwen3.8-27B-BF16.gguf` — 931,145,888 B, sha256 `de2a4986…` |
+| Fetch | [`scripts/download-frank-large-kingjones.sh`](../../scripts/download-frank-large-kingjones.sh) (`-f`, byte-count + LFS sha256 verified, `--verify-only`, token never on the command line) |
+| Seam | `tests/frank_large_unit.bats` (19 tests: payload gates, deployer/serve parity, spec block, KV dtype, sweep identity) |
+
+## Why this file, and why only this runtime
+
+`kingjones777/Qwen3.8-27B-ROCmFP4-STRIX-MTP-GGUF` ships several rungs. Parsed
+headers of the two candidates:
+
+| File | ftype | Tensor mix | Output-head tensors at plain Q8_0 | Card MTP acceptance |
+|---|---|---|---|---|
+| `Qwen3.8-27B-Q8_0_ROCMFPX_AGENT.gguf` | **115** | 353 F32 + 187 Q8_0 + **311 type-103** | **187** | 0.953 |
+| `Qwen3.8-27B-Q8_0_ROCMFPX.gguf` | 111 | 2 Q8_0 + type-103 | 2 | 0.911 |
+
+Two consequences, both measured off the header rather than guessed:
+
+1. **Quality.** The AGENT routing variant keeps the output-head (and other
+   routing-sensitive) tensors at plain Q8_0 instead of folding them into the
+   ROCmFPX kernel layout — 187 vs 2 — so the logits the sampler reads are the
+   highest-precision tensors in the file. It is also the card's MTP-tuned
+   variant, so it is the *faster* drafter base at the same time.
+2. **The runtime is forced.** Type 103 is a private extension: stock llama.cpp
+   and gufo both fail at load (`unknown tensor type`). Verified on the host
+   before deploying — the bake-off was not a horse race, it was a lock-in
+   check.
+
+**The build identity matters as much as the fork.** The 2026-08-25 ROCmFPX
+builds read this file and then died in `ggml_compute_forward_set_rows` — the
+exact failure documented in [[kingjones-q8agent-rocmfpx]] when this payload was
+first attempted on Dankclaw and routed to Dankserver as
+[[frank-medium-rocmfpx]] instead. `scripts/build-rocmfpx.sh` therefore checks
+the commit out **by SHA** and refuses to build anything else; the pin is
+load-bearing, not tidiness.
+
+## Shape and budget
+
+One session, the model's full native window, unquantized KV:
+
+| Dial | Value | Why |
+|---|---|---|
+| `--ctx-size` | 262,144 | "one full context" — the native window, no rope scaling |
+| `--parallel` | 1 | one conversation, all memory to it |
+| KV dtype | **f16** (both K and V) | ~64 KiB/token → ~16 GiB; free inside a 114 GiB aperture, so there is no reason to pay a quant |
+| `--fit off` | on | the auto-fitter would silently resize the window we asked for |
+| `--flash-attn on` | on | required for the long-window attention path to fit |
+| `--direct-io` | on | bypasses page cache for the 26 GiB weight stream (mmap is disabled by it — deliberate) |
+| `--ctx-checkpoints 16` | 16 | slot state for checkpoint/rollback |
+| `--cache-ram 8192` + `--cache-idle-slots` | 8 GiB | prompt cache; verified a repeat pass over the same 6k prompt replays in **0.5 s** |
+| `--jinja` | on | without it `chat_template_kwargs` are silently ignored |
+| `--reasoning on --reasoning-format deepseek --reasoning-budget -1` | | thinking in its own channel, unbounded |
+
+Measured steady state (production, one model copy resident):
+`mem_info_gtt_used` **63.1 GiB** of the ~114 GiB aperture, `mem_info_vram_used`
+0.17 / 2.0 GiB (the iGPU's dedicated carve-out is a stub; everything real lives
+in GTT). llama-server RSS 2.1 GiB. Boot to `/health` 200: **16 s** — the fastest
+endpoint bring-up in the fleet, because there is no engine compile step.
+
+## Speculative decode — measured, not assumed
+
+```
+--spec-type draft-mtp --model-draft <mtp-Qwen3.8-27B-Q4_0.gguf>
+--spec-draft-n-max 4 --spec-draft-n-min 0 --spec-draft-p-min 0.0
+--spec-draft-device ROCm0 --spec-draft-ngl 99
+```
+
+MTP in llama.cpp is a property of the file, not of the flags: the head is a
+trained tensor set (`nextn_predict_layers = 1` in this repo's head, arch
+`qwen35`, 65 blocks) and the draft context runs on the main model's weights, so
+there is no second 26 GiB copy in memory. See the fleet law in
+`AGENTS.md → llama.cpp serving`.
+
+**A/B, 2026-10-02** — same session, temp 0, 2,202-token prompt, 512 tokens,
+`cache_prompt: false`, `frank-large.service` stopped for every scratch arm (the
+benchmarking law), baseline arm stripped of `--model-draft` **and** every
+`--spec-*` flag:
+
+| Arm | Decode t/s (2 runs) | Prefill t/s | Draft acceptance |
+|---|---|---|---|
+| **MTP, n-max 4 (deployed)** | **17.00 / 17.14** | 350.6 / 324.9 | 0.462 (mean acceptance length ≈ 1.86) |
+| no draft | 7.60 / 7.66 | 381.8 / 381.0 | — |
+
+**+123 % decode for an 8–15 % prefill tax.** Notes that matter before someone
+re-litigates this:
+
+- Acceptance here (0.462) is nowhere near the card's 0.953. The card measures
+  its own short, repetitive text; this is a 2.2k-token prose prompt. Treat the
+  card's number as a file property, not a throughput promise.
+- `n-max 4` is where the gain was measured; 2 and 8 are **unmeasured**. The
+  gufo-era law applies in spirit — never rank spec settings on a
+  self-predictable corpus.
+- End-to-end on the chat path is slower than the `/completion` number because
+  thinking tokens are included: a 2,253-token chat prompt generating 512
+  tokens took 43.8 s wall (11.7 t/s end-to-end) — that is the cost of an
+  unbounded reasoning budget, not a regression in decoding.
+
+## Client contract (live-verified 2026-10-02)
+
+| Behavior | Observation |
+|---|---|
+| Model id | `/v1/models` → `frank-large` |
+| Tools | native: `get_weather` → `{"city": "Seattle"}` in `tool_calls` |
+| Thinking | in **`reasoning_content`** (llama.cpp field — *not* vLLM's `reasoning`); `models.yml` sets `reasoningContentField` accordingly |
+| `reasoning_effort` | accepts **only** `xhigh`, `medium`, `low`; anything else is HTTP 500 (the template raises, it does not clamp). `models.yml` folds `high`/`max` → `xhigh` |
+| Sampling | server defaults = the card's thinking recipe (temp 1.0, top_p 0.95, top_k 20, min_p 0, presence 0), mirrored in the harness `extraBody` |
+| Vision | the BF16 projector loads (`loaded multimodal model`); a red square answers **`Red`** (`usage: 75 prompt / 71 cached`) |
+| Prompt cache | second pass over the same 6k prompt: 0.5 s wall |
+
+Three traps, all hit for real:
+
+1. **`content: null` when reasoning eats the budget.** With
+   `--reasoning-parser`-style separation, a request whose `max_tokens` is
+   consumed entirely by thinking returns `message.content = None`. A bench
+   client that indexes `content` crashes *after* measuring and loses the
+   numbers — fall back to `reasoning_content`, or size the budget (200 was
+   enough for the vision probe).
+2. **`MTP boundary missing for seq_id=… ; resyncing after a non-token step`**
+   appears in the log under draft resets. Self-recovering; not an outage signal.
+3. **Never `pkill -f llama-server` on Dankclaw** — frank-small is also a
+   `llama-server`. The sweep key here is the payload name:
+   `pkill -f 'Qwen3.8-27B-Q8_0_ROCMFPX_AGENT'`.
+
+## Operations
+
+```bash
+# deploy (planner selects this endpoint only via its own unit/env/serve trio)
+./scripts/deploy.sh dankclaw --service frank-large
+
+# status / logs / cause of a crash loop
+systemctl status frank-large
+journalctl -u frank-large -n 200 --no-pager
+
+# rebuild the pinned fork (only when deliberately moving the pin)
+scripts/build-rocmfpx.sh
+
+# re-verify the payload without re-downloading
+scripts/download-frank-large-kingjones.sh --verify-only
+```
+
+`serve-frank-large.sh` refuses to start — before touching the port — unless the
+fork binary, the weights, the MTP head and (unless `FRANK_LARGE_MMPROJ=""`) the
+projector all exist; `scripts/deploy-remote.sh` mirrors the same predicate so a
+broken payload never stops the running unit. Caller environment **overrides**
+`frank-large.env`, and defaults use `:=` so an explicitly empty value surfaces
+as the error it reads as.
+
+## Rollback
+
+git tag **`frank-large-gufo-q4kxl-20261001`**: gufo v0.1.1
+(`scripts/build-gufo.sh` still on disk), the Q4_K_XL Qwen3.8-27B payload and the
+DFlash2 drafter all remain on the host. Restore that tag's unit/env/serve trio
+and redeploy — nothing here deletes anything.
+
+The rollback payloads stay exactly where the tag's env file expects them:
+`/home/dank/models/Qwen3.8-27B-UD-Q4_K_XL/` (16.35 GiB) and the DFlash2
+drafter `/home/dank/models/Qwen3.8-27B-UD-Q8_K_XL/Qwen3.8-27B-DFlash2-Q4_K_M.gguf`
+(1.07 GiB). The one-day vLLM MiMo era's payloads (MiMo-V2.6 BF16 18 GiB,
+Qwen3.5-9B MTP source 19 GiB, both DFlash draft dirs) were staged to
+`/home/dank/.AGENTTRASH/20261002-retired-llm-payloads/` on 2026-10-02 —
+recoverable until the trash is purged, and never part of a rollback plan.
+
+## When it misbehaves
+
+- **`unknown tensor type` / ftype 115 refused** — something moved off the
+  ROCmFPX binary. Check `readlink -f $(systemctl show frank-large -p ExecStart)`
+  and the fork commit; this file cannot run on stock llama.cpp or gufo.
+- **SIGSEGV in `ggml_compute_forward_set_rows` during load** — a fork build
+  older than the `36e9acd` pin; rebuild with `scripts/build-rocmfpx.sh`.
+  History: [[kingjones-q8agent-rocmfpx]].
+- **Slow decode with no log error** — check whether the draft is engaged
+  (`/metrics`, `draft_n` / `draft_n_accepted`; a zero draft count means the head
+  did not load), and check GTT pressure with
+  `cat /sys/class/drm/card1/device/mem_info_gtt_used` — a second resident copy
+  of this model (an orphaned scratch server) drives the whole host into
+  eviction and halves the rate silently.
+- **`500` naming Jinja** — a `chat_template_kwargs` field reached a server
+  without `--jinja`.
+- **HTTP 500 naming the template on the first request of a role** — an
+  unmapped `reasoning_effort`; fix the fold in `models.yml`, not the model.

serverAppConfigs-docs/LLM/frank-large-unsloth-qwen38-27b-q8xl.md

@@ -1,7 +1,7 @@
 ---
 type: reference
 title: "stock llama.cpp Qwen3.8-27B UD-Q8_K_XL (DFlash2) on Dankclaw (frank-large)"
-description: "The active frank-large runbook — the stock llama.cpp prebuilt (unsloth b11030, ROCm gfx1151) serving unsloth/Qwen3.8-27B-GGUF:UD-Q8_K_XL (27B dense, 31.46 GiB) with the z-lab DFlash2 Q4_K_M drafter (depth 4) and q8_0 KV on the Dankclaw Strix Halo iGPU: the prebuilt runtime, the full-window/one-stream shape, the DFlash2 spec method, the Qwen-Sharp template, the payload, the client contract, and the failure modes worth knowing."
+description: "The RETIRED frank-large runbook (2026-09-21 → 2026-09-28; replaced by [[frank-large-gufo-qwen38-27b]], and the rollback target) — the stock llama.cpp prebuilt (unsloth b11030, ROCm gfx1151) serving unsloth/Qwen3.8-27B-GGUF:UD-Q8_K_XL (27B dense, 31.46 GiB) with the z-lab DFlash2 Q4_K_M drafter (depth 4) and q8_0 KV on the Dankclaw Strix Halo iGPU: the prebuilt runtime, the full-window/one-stream shape, the DFlash2 spec method, the Qwen-Sharp template, the payload, the client contract, and the failure modes worth knowing."
 tags:
   - llm
   - llama.cpp
@@ -19,10 +19,12 @@ timestamp: 2026-09-21
 
 # frank-large — stock llama.cpp Qwen3.8-27B UD-Q8_K_XL (DFlash2)
 
-> **ACTIVE since 2026-09-21.** Replaces [[frank-large-kairic-edge-qwen38-27b]]
+> **RETIRED 2026-09-28**, replaced by [[frank-large-gufo-qwen38-27b]] (the
-> (Kairic Edge IU4 W4A4, now the rollback target) and
+> gufo engine, same weights, same port and alias); the prebuilt is intact on
-> [[frank-large-unsloth-qwen38-flash-next]] (the Flash-Next MoE it preceded).
+> disk as the rollback target. ACTIVE from 2026-09-21, when it replaced
-> Same port (7770), same alias (`frank-large`), same client contract surface.
+> [[frank-large-kairic-edge-qwen38-27b]] (Kairic Edge IU4 W4A4) and
+> [[frank-large-unsloth-qwen38-flash-next]] (the Flash-Next MoE it
+> preceded).
 
 ## Identity
 

serverAppConfigs-docs/LLM/frank-large-vllm-mimo-v26-bf16.md

@@ -0,0 +1,186 @@
+---
+type: reference
+title: "frank-large on Dankclaw — vLLM TheRock, MiMo-V2.6-Distill-Qwen-9B BF16 + cross-model MTP (RETIRED)"
+description: "The RETIRED frank-large runbook (2026-10-01, one day) — the digest-pinned kyuz0 TheRock gfx1151 vLLM image serving XiaomiMiMo MiMo-V2.6-Distill-Qwen-9B in full BF16 on the Strix Halo iGPU. Kept for the measured record: the roof math, the cross-model MTP drafter, and the autoresearch finding that spec decode is NET-NEGATIVE on real chat+reasoning traffic here (6.19 t/s with MTP-2 vs 9.01 t/s without). Rolled back to [[frank-large-gufo-qwen38-27b]] by Frank the same night."
+tags:
+  - llm
+  - frank-large
+  - dankclaw
+  - vllm
+  - strix-halo
+  - mimo
+  - mtp
+  - spec-decode
+  - bf16
+timestamp: 2026-10-01
+---
+
+# frank-large — vLLM MiMo-V2.6-Distill-Qwen-9B BF16 (cross-model MTP)
+
+> **RETIRED 2026-10-01 night, one day active.** Frank's call after the
+> autoresearch loop: battery decode 6.19 t/s (MTP-2) / 9.01 t/s (no draft)
+> against gufo's ~15 t/s on the 27B — and every ≥6-bit quant path in vLLM
+> 0.30 is dead on gfx1151 (quark int8 rejected; all FP8 kernels CDNA-only;
+> torchao config unwired). Back to [[frank-large-gufo-qwen38-27b]]
+> (Q4_K_XL, 1 x 262144). The measured content below stands as the record.
+
+## Identity
+
+- **Endpoint:** `frank-large` at `192.168.0.143:7770` (Dankclaw).
+- **Runtime:** vLLM 0.30.0 in the **digest-pinned** `docker.io/kyuz0/vllm-therock-gfx1151`
+  image (image ID `2d24a8c5a3fd`, digest `sha256:98c1f3ac9833…` — ROCm 10.0.0 /
+  torch 2.11.0 / TheRock, Triton Gated DeltaNet kernels). The official vLLM
+  images are CUDA-only; this fork is the fleet's gfx1151 path. Plain
+  `podman run` driven by `scripts/serve-frank-large.sh` under a root unit
+  (`scripts/frank-large.service`, `User=dank`, `Restart=always`,
+  `ExecStop=podman stop -t 120`, `TimeoutStopSec=180` > the stop grace).
+- **Model:** `XiaomiMiMo/MiMo-V2.6-Distill-Qwen-9B` in **full BF16**
+  (17.53 GiB, 4 shards) — the same checkpoint frank-small serves as an INT8
+  W8A16 AutoRound conversion on the 3090, here unquantized.
+- **Draft:** `Qwen/Qwen3.5-9B` (18 GiB on disk; only the 15 `mtp.*` tensors
+  are loaded into the draft pass).
+
+## Why this shape (the roof math)
+
+Decode speed for a dense model on Strix Halo is bandwidth-bound:
+BF16 weights x one pass per token against the LPDDR5x aperture. Measured
+2026-10-01 on this host: **13.1 t/s is ~94% of the roof** — the payload, not
+a misconfiguration. The same INT8 payload reaches 83.9 t/s on the 3090
+because it moves ~12 GiB through ~936 GB/s. The only way past a bandwidth
+wall is generating more than one token per weight pass: **speculative decode
+is the performance feature of this endpoint**, not a garnish.
+
+The memory side is the opposite of a squeeze: KV grows ONLY on the 8 full
+attention layers (32 KiB/token bf16: 8 x 2 x 4 heads x 256 dim x 2 B);
+the 24 Gated DeltaNet layers hold a FIXED ~25 MiB state per running
+sequence. `--max-model-len 262144` (the native window) with
+`--max-num-seqs 4` at `--gpu-memory-utilization 0.45` boots clean — the
+four-session contract the gufo shape ran, with the pool shared by the
+scheduler instead of carved per session.
+
+## Spec method: cross-model MTP — measured, not assumed
+
+| arm | prefill | decode | acceptance |
+|---|---|---|---|
+| BF16 base | 17,876 t/s | 13.1 t/s | — |
+| BF16 + DFlash 8 tok | 7,976 t/s | 17.4 t/s | pos0 59% pos1 26% (token rate 14.8%) |
+| **BF16 + MTP 2 tok** | 7,380 t/s | **19.8 t/s** | pos0 71% pos1 44% (token rate 57.4%) |
+
+One session, temperature 0, 28,647-token prefill / 512-token decode, same
+client for all three arms (2026-10-01 bench window, frank-large stopped per
+Benchmarking Policy). **MTP wins (+51% vs +33%) and needs only 2 draft
+tokens** — acceptance past position 1 is 44% and falling; DFlash's own curve
+decays 59/26/14/7, so 8 proposals are mostly wasted verification.
+`num_speculative_tokens` beyond 2 is UNMEASURED; do not raise it without a
+fresh A/B. Both spec arms pay ~55–59% of prefill (the draft passes share
+the compute queues) — this endpoint's thinking-heavy traffic spends its wall
+clock in decode, which is the regime where that trade wins.
+
+**Why a Qwen head accepts on a MiMo target:** MiMo-V2.6 is a distillation of
+Qwen3.5-9B that preserves the hidden-state geometry — identical
+4096-hidden/248320-vocab/32-layer/interval-4 structure, and the 71%/44%
+per-position acceptance is the direct measurement that the distillation
+preserved more than geometry. MiMo itself ships **ZERO `mtp.*` tensors**
+(verified in both the BF16 base and the INT8 conversion) — which is exactly
+why frank-small's SGLang NEXTN draft is a 0.00-acceptance ghost: SGLang
+reads the head from the TARGET checkpoint, vLLM reads the draft from a
+separate path. That difference is the whole trick:
+
+```
+--speculative-config '{"method":"mtp","model":"/models/Qwen3.5-9B","num_speculative_tokens":2}'
+```
+
+The alternative drafter (`z-lab/Qwen3.5-9B-DFlash`, 6 layers, target_layer_ids
+[1,5,…,29]) stays fetched on disk; flip `FRANK_LARGE_SPEC=dflash` and repoint
+`FRANK_LARGE_SPEC_DIR` to use it. Its card's "vLLM PR #40898 unmerged" claim
+is FALSE — `vllm/v1/spec_decode/dflash.py` ships in 0.30.
+
+## The numerics trap (greedy text is NOT preserved by spec decode)
+
+Measured: temperature-0 completions DIFFER across arms (sha `1318aeb6` base,
+`0208b45d` dflash, `20c41c38` mtp). Batched verification changes the bf16
+reduction order, argmax near-ties diverge, and the divergence cascades.
+vLLM spec decode is distribution-preserving, not bitwise-greedy-preserving.
+The gufo-era "greedy == autoregressive text" law does not carry over; do not
+chase text drift between spec and no-spec arms as a regression.
+
+## Payload
+
+- Target: 4 safetensors shards (17.53 GiB) + config/tokenizer under
+  `/home/dank/models/MiMo-V2.6-Distill-Qwen-9B-BF16/`, each shard
+  byte-count + LFS sha256-verified by
+  `old/download-frank-large-mimo.sh` (`--verify-only` re-checks; no token on
+  the curl command line).
+- MTP source: `old/download-frank-large-qwen35.sh` →
+  `/home/dank/models/Qwen3.5-9B/` (4 shards behind
+  `model.safetensors.index.json`; the shards carry the odd HF asset names
+  `model.safetensors-0000N-of-00004.safetensors` — the index resolves them,
+  the engine boots from exactly this tree).
+- DFlash drafter: `old/download-frank-large-dflash.sh` →
+  `/home/dank/models/Qwen3.5-9B-DFlash/`.
+- **Payload status 2026-10-02:** all three trees moved to
+  `/home/dank/.AGENTTRASH/20261002-retired-llm-payloads/` (59 GiB staged with
+  the other retired frank payloads) and the fetch trio moved to `old/` — the
+  endpoint is two runtimes past this one, and re-fetch is cheaper than disk.
+- The serve script gates the image by REPO DIGEST (rootless podman resolves
+  tags from the network; the pin exists to boot exactly these bytes
+  offline), then config + index + exact shard count, then the spec draft
+  (either weight layout: single `model.safetensors` or a sharded index),
+  then `/dev/kfd` + whole `/dev/dri` (never individual renderD nodes —
+  Dankserver 2026-09-29 lesson).
+
+## Client contract
+
+- **Alias:** `frank-large` (the `id` at `/v1/models`).
+- **Reasoning:** `--reasoning-parser qwen3`; vLLM emits the thinking text in
+  **`reasoning`**, NOT llama.cpp/gufo's `reasoning_content` — the harness
+  `models.yml` compat block for this provider sets
+  `reasoningContentField: "reasoning"`. Miss that and thinking silently
+  vanishes from transcripts.
+- **Tools:** `--enable-auto-tool-choice --tool-call-parser qwen3_coder`.
+- **Vision:** the checkpoint carries the `qwen3_5_vision` tower but the
+  serve script passes `--language-model-only` — the endpoint stays
+  text-only by decision; dropping the flag brings vision back.
+- **Sampling:** request-side; the harness sends the Qwen3.5-family recipe
+  (temp 0.7, top_p 0.8) via `compat.extraBody`, and
+  `chat_template_kwargs.{enable_thinking,preserve_thinking}` — the
+  `chat_template_kwargs` form is what vLLM reads; a top-level
+  `preserve_thinking` is ignored there.
+- **Env hygiene:** every tunable is `FRANK_LARGE_`-prefixed because HIP
+  reads bare `TEMP`/`HOST`/`GPU_MAX_HEAP_SIZE` and the serve script exports
+  the env file with `set -a`.
+
+## Ops
+
+- Boot ~200–240 s warm; health 503 while loading (do not kill it).
+- The serve script sweeps stale container identities before starting:
+  `frank-large-mimo` itself and the bench-window container `vllm-mimo`.
+- Container death → `podman wait` returns non-zero from ExecStart →
+  `Restart=always` self-heals (a clean exit-0 drain also restarts, the
+  frank-primary lesson).
+- Bench windows: `systemctl stop frank-large` is enough (mask fails on the
+  relinked unit; the deploy restores).
+- First-deploy facts (2026-10-01, deploy #3): healthy after **230 s**;
+  boot log `GPU KV cache size: 819,332 tokens, Maximum concurrency for
+  262,144 tokens per request: 3.13x` — four sessions share the pool
+  (the scheduler, not a carve). Live battery green: alias `frank-large`,
+  reasoning split into `reasoning`, tool_calls with correct args, and
+  `/metrics` acceptance pos0 69% pos1 44% (54/86 draft tokens accepted),
+  matching the bench window.
+
+## Rollback
+
+Git tag **`frank-large-gufo-q4kxl-20261001`** — gufo v0.1.1 (built from
+source, `scripts/build-gufo.sh` stays on disk) + the Q4_K_XL Qwen3.8-27B
+payload + DFlash2 drafter, all still on the host. Restoring the tag's
+unit/env/serve trio and redeploying returns the 27B endpoint.
+
+# Citations
+
+- Model: https://huggingface.co/XiaomiMiMo/MiMo-V2.6-Distill-Qwen-9B
+- Draft source: https://huggingface.co/Qwen/Qwen3.5-9B
+- DFlash drafter: https://huggingface.co/z-lab/Qwen3.5-9B-DFlash
+- Engine image: https://hub.docker.com/r/kyuz0/vllm-therock-gfx1151
+- Serve script: `scripts/serve-frank-large.sh`; tunables: `scripts/frank-large.env`
+- Predecessor: [[frank-large-gufo-qwen38-27b]]; sibling on the same arch:
+  [[frank-small-sglang-mimo-v26]]

serverAppConfigs-docs/LLM/frank-primary-glm53-flash-w4a16.md

@@ -0,0 +1,706 @@
+---
+type: runbook
+title: frank-primary — GLM-5.3-Flash W4A16 int4 (tcclaviger, 2x R9700)
+description: VERIFIED 2026-09-30 by boot probe, bench, live expert-cache telemetry AND a torch-profiler kernel-time window. frank-primary cut over from Qwen3.8-Flash-Next-Uncensored-FP8 + MTP4 to Intel/GLM-5.3-Flash-W4A16-AutoRound, the first payload this fleet serves through the fork's W4A16 expert-offload path (INC scheme -> R4dW4a16MoEMethod, libr4d grouped W4A16 GEMM). Measured decode 9.9-10.0 tok/s and prefill 918 tok/s at 131,072 ctx. THE 35 tok/s TARGET IS NOT MET, and the profiler window FALSIFIED the first explanation: the expert gather is 1.05 % of device time, not the bound. 44 % of the 93 ms/token budget is pointwise glue (1,428 elementwise launches per token) because the fork's GLM trunk is NOT torch.compiled while its own Qwen trunk is; the second-largest lever, the fused router, is binary-blocked at 288 experts. The compile changelist is staged but incomplete (three patches, one blocker left).
+tags:
+  - dankserver
+  - r9700
+  - gfx1201
+  - glm5.3
+  - w4a16
+  - int4
+  - expert-offload
+  - expert-cache
+  - autoround
+  - vllm
+timestamp: 2026-09-30
+---
+
+# frank-primary — GLM-5.3-Flash W4A16 int4 (tcclaviger, 2x R9700)
+
+> **RETIRED 2026-10-01.** Cutover reversed to Qwen3.8-Flash-Next-Uncensored-FP8
+> + MTP4 (commit `2339fae` trio). The DFlash2 drafter experiment is a
+> **failure**: after five patches the drafter served and drafted (12.8 tok/s
+> proposed, 65,536 ctx, both TP ranks capturing FULL dflash2 cuda graphs) but
+> **acceptance pinned at 1.00-1.38 on every text class including verbatim
+> repetition** — the signature of a mis-wired aux-hidden-state capture, not a
+> weak drafter. At ~1.2 acceptance the 8-token verification tax makes decode
+> 5x SLOWER than the 9.9-10.0 no-draft baseline (1.6-1.8 tok/s measured). The
+> untested alternative capture semantics: post-layer output state
+> (DeepSeek's `hidden_states + residual`) or the mHC `comb` stream. Payload
+> (170 GiB) + drafter (2.2 GiB) staged to
+> `/home/dank/.AGENTTRASH/20261001-glm-purge/` pending purge approval.
+> KV-cache work that survives: patches 02/03/04 in `scripts/patches/` are
+> generic unifier/tailpin/grouping fixes, apply-verified against the image.
+
+
+> **Question this page answers.** Can the fleet serve the 320B-class
+> `GLM-5.3-Flash` at int4 on two R9700s with expert offload, and how fast is it?
+> **Answer.** Yes — it serves, with vision, tool calls and 131,072 ctx. It runs
+> at **9.9-10.0 tok/s decode / 918 tok/s prefill**, and **35 tok/s is not
+> reachable** on this payload as the fork ships it. The bound is **not** the
+> model, the quant, the link, or the expert gather (measured: 1.05 % of device
+> time). It is the **unfused pointwise glue of an uncompiled trunk** — 44 % of a
+> 93 ms per-token budget spread over ~1,428 elementwise launches per token —
+> because the fork compiles its own Qwen trunk and not its GLM one. See
+> [The measured profile](#the-measured-profile-and-the-real-wall).
+
+Related: [[dankserver-qwen38-fp8-expert-offload-tuning]] (the same fork, the same
+hardware — its conclusion was drawn from throughput arithmetic and is superseded
+by this page's kernel-time measurement),
+[[dankserver-frontier-moe-feasibility]] (the earlier fit analysis),
+[[VLLM Index]], [[LLM/index|LLM fleet]].
+
+## The deployed shape
+
+| term | value |
+| --- | --- |
+| payload | `Intel/GLM-5.3-Flash-W4A16-AutoRound` — AutoRound **symmetric int4, group 128**, packing `auto_round:auto_gptq`, 34 shards + 2 extra safetensors, 113,074 tensors, ~170 GiB |
+| runtime | `docker.io/tcclaviger/vllm@sha256:da4029666…` (same digest-pinned image as the qwen cutover) |
+| engine | TP=2, `--enable-expert-offload --expert-precise-memory`, max-model-len 131,072 x 2 seqs, bf16 KV, util 0.98, NBATCH 2048, prefix caching |
+| MoE kernel | `R4dW4a16MoEMethod` → libr4d grouped W4A16 GEMM, 6.234 MiB per expert-rank slot |
+| expert cache | 14.51 GiB = **2,383 slots** per rank; 12,096 host-backed experts, **73.64 GiB pinned per rank** |
+| unit | `scripts/frank-primary.service` (root unit, `LimitMEMLOCK=infinity` for the pin) |
+| measured | **decode 9.9-10.0 tok/s**, **prefill 918 tok/s** (warm, 2.6k-token prompt), KV cache 138,561 tokens |
+
+```mermaid
+graph LR
+    A[Intel AutoRound W4A16<br/>sym int4 g128, auto_gptq] -->|quant_method auto-round| B[inc.py override to inc]
+    B --> C[inc_wna16_scheme<br/>_resolve_gptq_moe]
+    C -->|sym int4 g128 + gfx1201 + libr4d| D[R4dW4a16MoEMethod<br/>grouped W4A16 GEMM]
+    D --> E[ExpertLRUCache<br/>2,383 VRAM slots / rank]
+    D --> F[73.64 GiB pinned host rows / rank]
+    E --> G[decode step]
+    F -->|host misses<br/>1.05% of device time| G
+```
+
+## Why this payload and not another
+
+The W4A16 fast path on this fork is reachable **only** through the INC
+(AutoRound) quant method. `compressed_tensors` checkpoints of the same model —
+for example `dealignai/GLM-5.3-Flash-CYBERSECURITY-W4A16`, 182 GiB — resolve
+their MoE through the WNA16 oracle to `TritonWNA16Experts`, which **has no
+offload recipe**, and the engine refuses to serve at all: 170 GiB of experts
+cannot fit 64 GiB of VRAM, so offload is mandatory, not optional.
+
+The chain that makes Intel's checkpoint work, with the exact evidence:
+
+```
+[expert_plan.py]  quantization=inc            ← inc.py:364 overrides "auto-round"
+[inc_wna16_scheme.py:249]  INC GPTQ MoE on ROCm: R4dW4a16MoEExperts (libr4d grouped W4A16 GEMM)
+[r4d_w4a16_moe.py:682]     Using MoEPrepareAndFinalizeNoDPEPModular / R4dW4a16MoEExperts
+[expert.py:450]            Expert offload: 42 MoE layers staged; 2,383 cache slots in 14.51 GiB planned per rank
+[expert.py:413]            42 layers split: 0 VRAM-only experts, 2,383 swap slots, arena 14.51 GiB,
+                           host 73.64 GiB pinned, host rows prepacked on the GPU
+```
+
+`R4dW4a16MoEExperts` is what the offload machinery wraps: the offload plan is
+format-agnostic but requires the experts class to expose the hot/cold split.
+
+## The numbers, and the wall
+
+Measured on the deployed engine, 2026-09-30 (`bench_glm.py`, real prose, greedy):
+
+| depth | prefill | decode |
+| --- | --- | --- |
+| 2,621 tok prompt, run 1 (cold cache) | 259 tok/s | 9.88 |
+| 2,621 tok prompt, runs 2-3 (warm) | 918 / 885 tok/s | 9.91 / 9.41 |
+| 493 tok prompt (telemetry window) | 124 tok/s | 8.90 |
+
+Decode is flat across context and across every arena change — the signature of a
+per-step cost that is not KV- or residency-driven at the margin.
+
+### The measured profile, and the real wall
+
+**This section replaces an earlier explanation, which the measurement falsified.**
+The first reading of this page bounded the step with throughput arithmetic: 42 MoE
+layers x 8 experts x 6.234 MiB = 2.045 GiB per step per rank, 0.82 GiB of it
+missed at a measured 0.53-0.66 hit rate, giving "0.82 GiB in ~100 ms = 8.2 GB/s =
+the gather is the wall". That arithmetic was circular — it *assumed* the gather
+consumed the whole step — and it is wrong. The engine, asked directly, disagrees:
+
+A torch-profiler window (`--profiler-config`, `POST /start_profile` /
+`stop_profile`, 496 decode tokens at the deployed shape, rank0 trace parsed for
+device-activity time) gives the true split. Total device kernel time **46.18 s /
+496 tokens = ~93 ms per token** against a measured ~104 ms step (9.6 tok/s with
+the profiler attached):
+
+| ms | % | launches | kernel |
+| --- | --- | --- | --- |
+| 16,074 | **34.8** | 708,500 | `vectorized_elementwise_kernel` (22.7 us each) |
+| 4,602 | 10.0 | 42,000 | `r4d_gemm_moe_w4a16_nt_b16_m64` — the MoE GEMM |
+| 3,108 | 6.7 | 174,708 | `elementwise_kernel_manual_unroll` |
+| 2,943 | 6.4 | 138,252 | `wvSplitK_hf_sml_` — split-K GEMM (projections / head) |
+| 2,829 | 6.1 | 83,160 | router: `sbtopk::gatherTopK` + `warpMergeSortTopK` + `bitonicSortKVInPlace` (1,208 + 1,051 + 559 ms; the 1,051 row is not in the list below) |
+| 3,015 | 6.5 | 133,920 | `r4d_mhc_pre_finish/pre_partial/post` — hyper-connections |
+| 2,782 | 6.0 | 127,916 | inductor `triton_poi_*` / `triton_per_*` (the fusions that DO happen) |
+| **487** | **1.05** | 20,664 | **`r4d_moe_lru_gather_kernel`** — the accused |
+| 250 | 0.54 | 20,832 | `_moe_lru_tele_kernel` — the telemetry that measured it (TELEM was on) |
+
+So the gather is **1.05 %** of device time, and `r4d_moe_lru_fused_kernel` (the
+LRU *manager*, not the copy) is a further 1,599 ms / 3.5 %. Residency work and the
+PCIe link are not the story: 44 % of the budget is **pointwise glue** — bf16<->fp32
+conversions, adds, `pow`/`rsqrt` from fp32 RMSNorm, `launch_clamp_scale` quant
+passes, `reduce_kernel` means — spread over **~1,428 elementwise launches per
+token** (3,734 kernel launches per token in total). The GPU is busy 86 % of the
+wall clock, so this is real device work, not launch-gap overhead.
+
+Two consequences that kill three whole families of planned work:
+
+- **Spec decode cannot help this shape, and the MTP measurement was not an
+  accident.** Pointwise cost scales with the token count, so a draft batch that
+  verifies 4 tokens pays ~4x the glue for the same per-token output — which is
+  exactly why MTP4 measured "acceptance 3.76-4.90, decode unchanged". Same reason
+  the gather was never the bound.
+- **Arena size and hit rate are not levers.** Confirmed twice: `--expert-precise-memory`
+  (arena 13.08 -> 14.51 GiB) moved decode by nothing, and the gather is 1 %.
+
+## Fork work: the compile changelist (WIP, does not boot yet)
+
+The fork compiles its own flagship trunk — `vllm/models/qwen4_exp/amd/model.py`
+carries `@support_torch_compile` — and **not** its GLM trunk. The engine says so on
+every boot:
+
+```
+Cudagraph mode FULL_AND_PIECEWISE requires piecewise capture, but the loaded model
+provides neither a compiled submodule nor breakable CUDA graphs.
+Overriding to FULL_DECODE_ONLY.
+```
+
+Inductor fusion is therefore the only lever that touches the 44 % pointwise mass.
+Three patches are staged in `scripts/patches/glm53-compile-0*.patch`, each carrying
+its own evidence header, base/patched sha256 and the next blocker:
+
+| patch | target | fixes |
+| --- | --- | --- |
+| `-01-model-decorator` | `models/glm5next/common/model.py` | decorates `Glm5NextModel` (env `R4D_GLM_COMPILE`, default on) |
+| `-02-trace-safe-probes` | `model_executor/kernels/r4d_lib.py` | freezes 10 capability probes that do `os.path.exists`/`.so` lookups **inside the traced forward** |
+| `-03-fullgraph-optout` | `compilation/wrapper.py` | makes `fullgraph=True` opt-out (env `VLLM_COMPILE_FULLGRAPH=0`) |
+
+What the boots established, in order (each one ~7 min):
+
+1. **01 alone dies** on the first cold probe call inside the traced region:
+   `Dynamo does not know how to trace the builtin 'posix.stat'` —
+   `r4d_mhc_supported -> r4d_lib.r4d_mhc_enabled -> available -> os.path.exists`.
+   **`functools.lru_cache` does not protect these probes**: dynamo re-traces the
+   wrapped body to build its guard even when the cache is warm (it warns about
+   exactly this), verified in-image — a *warm* lru_cache-calling function still
+   fails `fullgraph=True`, while a plain module-global read compiles.
+2. **01+02 get past it** (`libr4d capability probes frozen: 10 warm [...]` in all
+   four processes) and die on
+   `RuntimeError: Graph breaks are not supported with aot compile`, because this
+   build runs AOT compile by default and AOT refuses breaks.
+3. **01+02+03 with `VLLM_COMPILE_FULLGRAPH=0 VLLM_USE_AOT_COMPILE=0` traces the
+   trunk successfully** (`Dynamo bytecode transform time: 1.08 s`) and then dies on
+   the decode forward with `KeyError: 'in_proj_qkvbfg_a'` (`Glm5NextModel.forward`
+   line 748 calling the decoder layer at line 456). `in_proj_qkvbfg_a` is the KDA
+   layer's merged column-parallel projection (`kda.py:217`, called at :407) — a real
+   submodule attribute name, also present in the `load_weights` shard table
+   (`model.py:780-785`). Reading the full traceback (the reported line is inside
+   dynamo-rewritten code) traces the throw to dynamo's own name→module map:
+   `torch/_dynamo/output_graph.py` registers into `nn_modules` at :1259/:1384,
+   looks up with `self.nn_modules[node.target]` at :3000 (a bare-name `KeyError`
+   when absent), and documents stripping `_buffers[..]/_parameters[..]/_modules[..]`
+   from those names at :1225. So the compiler is failing to **resolve a submodule by
+   name** while tracing the GLM decoder layer; whether the root cause is the fork's
+   module tree or dynamo's registration is not yet settled. The next step is the one
+   torch itself suggests in that error — `TORCHDYNAMO_VERBOSE=1` for the internal
+   stack trace (and note the launcher resets `ENVARGS`, so it must be added inside
+   `serve_glm_autoround.sh`'s COMPILE_PATCH branch rather than in the outer
+   environment).
+
+4. **04 removes the fused-norm call from the traced region** (`R4D_GLM_FUSE_QKV_RMSNORM=0`:
+   the GLM MLA block stops calling the fork's fused-norm JIT kernel, whose kernel
+   object in `models/common/ops/fused_qk_rmsnorm.py` is bound *after* the function
+   that uses it — `_FUSED_Q_KV_RMSNORM_KERNEL` at line 223 vs its use at line 211).
+   **It removes a real break** (under `fullgraph=True` the fused call was the first
+   break dynamo reported; after this patch the report moves on to the workspace
+   acquisition) — but it is not by itself what blocks a boot, because with breaks
+   genuinely tolerated the blocker is the name-map `KeyError`. See the run history
+   below.
+5. **A launcher bug that made four boots misleading — worth knowing because the
+   failure kept moving.** The scratch launcher's compile branches were restructured
+   on 2026-09-30 and the `-e VLLM_COMPILE_FULLGRAPH=0` line was dropped, so four
+   consecutive boots mounted the wrapper patch while `_FULLGRAPH` still defaulted to
+   True. With `fullgraph=True` **every** break is an error, so those boots reported
+   whichever break came first:
+
+   | config | outcome |
+   | --- | --- |
+   | `fullgraph=False`, AOT off | breaks tolerated → dies later on the dynamo name-map `KeyError: 'in_proj_qkvbfg_a'` |
+   | `fullgraph=True`, AOT off | dies on the first break (cell → workspace → compiler-disable) |
+
+   So the reported breaks are real *as breaks*, but they are not what blocks a
+   break-tolerant compile — and an earlier version of this page blamed
+   `TORCH_LOGS=graph_breaks` for the difference. **That was wrong**; the flag only
+   logs. The launcher sets the env var again.
+
+**How to see the breaks (and its trap).** `TORCH_LOGS=graph_breaks` names each break
+with a user stack, and `VLLM_USE_AOT_COMPILE=0` is required because AOT compile
+hard-refuses breaks — but a fullgraph=True run reports the first break as fatal, so
+check which configuration you are actually in (`podman exec <ctr> printenv
+VLLM_COMPILE_FULLGRAPH`) before concluding a break blocks the boot. For the name-map
+KeyError there is a ready-made probe: put `/home/dank/dynamo-probe` on `PYTHONPATH`
+with `VLLM_DYNAMO_DUMP_ATTRS=1` (a `sitecustomize.py` — a `.pth` on `PYTHONPATH` is
+NOT executed, and the image's own `sitecustomize` is chained so its apport hook
+survives) and it prints, at the miss, the node target, the near-miss keys, and
+whether another live `OutputGraph` holds the key.
+
+**State of the changelist (2026-09-30, latest).** The trunk **traces**
+(`Dynamo bytecode transform time: 1.08 s`) and the boot dies on the dynamo name-map
+`KeyError: 'in_proj_qkvbfg_a'`. Established about it, in order:
+
+- It is raised **inside the traced `Glm5NextDecoderLayer.forward`** — the frame chain
+  is `profile_run → _dummy_run → execute_model → glm4_1v.py:2358 → the compile wrapper
+  → eval_frame.compile_wrapper → Glm5NextModel.forward:748 (layer(...)) → module
+  _call_impl → Glm5NextDecoderLayer.forward:456 → KeyError`. `TORCHDYNAMO_VERBOSE=1`
+  adds **no** internal stack trace, i.e. it is a plain Python KeyError in the traced
+  frame, raised in a callee dynamo **inlined** (the reported line is the `def`, which
+  is how inlined frames surface).
+- It is **not** raised from any of dynamo's four `nn_modules` lookup sites:
+  `/home/dank/dynamo-probe` (a `sitecustomize.py` on `PYTHONPATH`,
+  `VLLM_DYNAMO_DUMP_ATTRS=1`) wraps three of them and stayed silent across two runs
+  with the probe confirmed installed in both the API-server and worker processes.
+- It is **not** a string-keyed lookup of a literal: `in_proj_qkvbfg_a` appears in the
+  whole tree only as the KDA layer's attribute (`kda.py:217`, used at `:407`) and in
+  the `load_weights` shard table (`model.py:780-785`). The four
+  `proj_name = prefix.split(".")[-1]` sites in the quantization utils do compute
+  exactly this bare key, but every one of them is guarded (`if proj_name not in
+  fused_mapping: return`), so they cannot raise it. The pluggable-layer registry is
+  keyed by class name, not module path.
+- It reproduces on **every** run of this configuration (5 of 5), and it is
+  independent of patches 04/05: with breaks genuinely tolerated, those remove breaks
+  that were never fatal in the first place — the KeyError is the only thing standing
+  between this changelist and a boot.
+
+**Next instrument, precisely:** the inlined caller has to be named. Two cheap ways:
+`TORCH_LOGS=+dynamo` on a boot and grep the log for the key (dynamo prints inlined
+frames with their source), or wrap the inlining candidates in that forward — the mHC
+CustomOps (`hc_pre` / `hc_post` / `hc_fused_post_pre`, `layers/mhc.py`) and the
+indexer op — with a probe that reports on `KeyError`. Until that names the caller, this
+is a dynamo-side name-resolution failure and not a fixable fork bug.
+
+Two verified facts bound the rest of the shape: `suppress_errors` does **not** cover
+graph breaks (only `fullgraph=False` does), and the GLM trunk can never satisfy
+`fullgraph` because its indexer metadata syncs to host
+(`glm5next/amd/sparse_indexer.py:263,419`: `int(...max().item())`) — both are dead
+fallbacks on the decode path that the fork author already replaced, so they are not
+the blockers they appear to be at first glance.
+
+### Two structural constraints on any further fork work
+
+- **No new HIP kernels are possible.** The image ships `r4d.so` as a binary with no
+  sources. The obvious second lever — the 9 % router — is gated
+  `num_experts in {128, 256, 512}` in `router/r4d_route_router.py` and the binary
+  contains only `r4d_moe_route_kernel<128|256|512>` instantiations. GLM has **288**
+  experts, so it falls back to the generic sort-based top-k and **cannot be moved to
+  the fused router without rebuilding libr4d**. Only Python/config/inductor changes
+  (and runtime-JIT Triton) are available.
+- **`--kv-cache-dtype fp8` is a dead end for capacity too.** The plan's KV reserve
+  is dtype-blind (1.76 GiB at 131,072 either way), so fp8 buys no arena and no
+  window; it only removes the only MLA decode path gfx12 has (trap 1 above).
+
+### DFlash2 for GLM — what the drafter actually is
+
+`incoai/GLM-5.3-Flash-DFlash2` is **not** a GLM-shaped drafter and needs no MLA/DSA
+port: its `config.json` declares `architectures: ["DFlash2DraftModel"]` and
+`model_type: "qwen3"` — 5 dense layers, hidden 4096, 32 heads / 8 kv heads,
+head_dim 128, sliding attention (window 2048), `is_causal: false`,
+`use_cache: false`, `dflash_config` = block 8, conv group 16, selector rank 256 /
+top-k 16, and **`target_layer_ids: [5, 14, 24, 33, 42]` against
+`num_target_layers: 45`**.
+
+The fork already maps that architecture:
+`registry.py:636 "DFlash2DraftModel": ("qwen3_dflash2", "DFlash2Qwen3ForCausalLM")`,
+and the spec method is `"dflash"` (`config/speculative.py:67`). So the port is
+**target-side only**: the GLM trunk must expose auxiliary hidden states at those five
+layer indices, the same interface `deepseek_v2.py` (also MLA+MoE) implements —
+`set_aux_hidden_state_layers()` / `get_eagle3_default_aux_hidden_state_layers()` plus
+capture at `idx in self.aux_hidden_state_layers` in the layer loop. `glm5next`
+implements neither.
+
+**The load was tried, and the gap is exactly one interface.** Payload fetched to
+`/home/dank/models/GLM-5.3-Flash-DFlash2` and hash-verified (`model.safetensors`
+2,342,169,800 B, sha256 `b038e1d9…` = the Hub's LFS oid). Booting the engine with
+`--speculative-config '{"method":"dflash","model":"/draft","num_speculative_tokens":8}'`
+accepts the config and the drafter and dies at worker init with:
+
+```
+RuntimeError: Model does not support EAGLE3 interface
+```
+
+So nothing about the drafter, the method name, or the payload is wrong — the GLM
+trunk simply does not implement the aux-hidden-state contract the dflash runner
+requires. The reference is in this same image, twice: `deepseek_v2.py`
+(`set_aux_hidden_state_layers` / `get_eagle3_default_aux_hidden_state_layers` at :1948-1951,
+capture at :1554 and :1579) and the fork's own `qwen4_exp` trunk. The contract is
+small:
+
+- `self.aux_hidden_state_layers: tuple[int, ...]` on the trunk, default `()`, set
+  by the wrapper's `set_aux_hidden_state_layers()`;
+- in the layer loop, `if idx in self.aux_hidden_state_layers: aux_hidden_states.append(<single-stream state entering layer idx>)`
+  (DeepSeek appends `hidden_states + residual`, with a TP all-gather fixup when the
+  rank holds fewer tokens than `positions`);
+- `Glm5NextModel.forward` returns `(hidden_states, aux_hidden_states)` when the
+  list is non-empty;
+- `get_eagle3_default_aux_hidden_state_layers()` supplies a default triple when the drafter
+  config omits `target_layer_ids` (this drafter provides `[5, 14, 24, 33, 42]`, so
+  the runner derives trunk indices `[6, 15, 25, 34, 43]`).
+
+**The GLM-specific unknown** is what "the single-stream state entering layer k"
+means for a trunk whose layers carry mHC multi-stream state: DeepSeek's
+`(hidden_states, residual)` pair has no counterpart here — GLM's decoder layer
+returns `(hidden_states, residual, post, comb)` with `hc_expand`/`hc_pre` mixing
+`hc_mult` streams, so the plausible answer is the pre-mHC single-stream input `x`
+(4096-wide, matching the drafter's `hidden_size`), not a post-layer value. That is
+a semantic choice to validate by acceptance rate, not by inspection.
+
+**Budget no speedup.** Per the profile above, a verification pass pays pointwise
+cost proportional to its token count — the same reason MTP4 measured flat — and
+MTP4's acceptance was already excellent (3.76-4.90). The port is worth doing as
+*enabling work* (it is the prerequisite for any spec decode on this endpoint, and
+it is small), not as a performance fix, and the honest expectation on this shape is
+flat decode with a prefill regression like MTP's until the pointwise mass is fused.
+
+**The port exists now: `scripts/patches/glm53-dflash-01-aux-hidden-states.patch`** (WIP,
+independent of the compile changelist — it patches the same file as compile-01, so
+compile-01 goes first if both are applied; base `008eeaa2…` → patched `bc9913a1…`). It
+implements exactly the contract read out of this image: `EagleModelMixin` on
+`Glm5NextModel`, capture of the single-stream `hidden_states` ENTERING each requested
+layer in the loop, the conditional `(hidden_states, aux_hidden_states)` tuple return, and
+the three `SupportsEagle3` member attributes plus both interface methods on the
+wrappers (capture shape from `deepseek_v2.py:1551-1585`; tuple consumption at
+`v1/worker/gpu/cudagraph_utils.py:621`).
+
+**The patch went through five versions, and the decisive discovery was where the
+instance actually lives.** The traceback's `set_eagle3_aux_hidden_state_layers(self.model)`
+is called on the model the runner LOADED — and for GLM-5.3-Flash that is
+`Glm5NextForConditionalGeneration` (the multimodal wrapper; the APIServer log names it),
+which holds the text model as `self.language_model = Glm5NextForCausalLM`
+(`glm5next/common/model.py:1113`). Patching only `Glm5NextForCausalLM` (v4) still failed
+with the identical error because the `isinstance` ran against the wrapper. **v5 fixes
+both wrappers** and is the first version to clear the gate (both TP workers log
+`Using Eagle3 auxiliary layers from config: (6, 15, 25, 34, …)`).
+
+The full evolution, each an instructive trap:
+
+1. **v1** hand-wrote the two methods the traceback names → identical error. The methods
+   are not the whole contract.
+2. **v2** inherited `SupportsEagle3` as a base (the image's own idiom — `laguna.py:717`,
+   `mimo_v2.py:834`, …) → dies at class creation with `Cannot create a consistent method
+   resolution order (MRO) for bases …`. GLM's wrappers already carry four/five
+   Protocol-derived bases, and simulation confirmed the ForCausalLM's five-base ordering
+   cannot take `SupportsEagle3` in *any* position.
+3. **v3** declared the members as class attributes (MRO-safe) — but only on
+   `Glm5NextForCausalLM` → identical error (the wrong wrapper, see above).
+4. **v4** added all three members (`supports_eagle3`, `has_own_lm_head`,
+   `has_own_embed_tokens` — a runtime-checkable Protocol's `isinstance` walks ALL
+   `__protocol_attrs__` *including base-Protocol members* via `getattr_static`, verified
+   experimentally in-image with stock typing) on `Glm5NextForCausalLM` → identical error.
+5. **v5** adds the three attributes to BOTH wrappers + the two methods on both
+   (`Glm5NextForCausalLM` unwraps to `self.model` — the `EagleModelMixin` trunk;
+   `Glm5NextForConditionalGeneration` delegates to `self.language_model`, the
+   `pixtral.py:498` pattern for exactly this wrapper shape) → **gate passes**.
+
+**The lesson generalizes twice over**: when a runtime error names an interface, read the
+interface's declaration first — and establish WHICH INSTANCE the check runs against
+before patching anything: multimodal wrappers (`…ForConditionalGeneration`) are loaded
+models, not pass-throughs, and the text model behind `self.language_model` is invisible
+to a check run on the wrapper.
+
+**Retired diagnostic (do not resurrect as-is): the `/probe` startup hook.** An
+`op_probe.py` + `sitecustomize.py` pair was mounted to wrap vLLM module forwards and
+torch's name-map lookups. It produced no evidence and caused a **fork bomb** (see the
+hazard note under experiment-window discipline). Use `TORCH_LOGS` for dynamo questions
+instead — it needs no startup hook.
+
+Note also that the KV-cap patch's predecessor arithmetic in
+[[dankserver-qwen38-fp8-expert-offload-tuning]] should be read the same way: its
+"gather-bound" conclusion came from the same throughput arithmetic that this page's
+profiler window has now falsified.
+
+### The KV-cache grouping journey: clearing the unifier, then the kpool tail
+
+With patch 01 v5 the gate passes and the boot moves to KV-cache layout
+construction — where the next two blockers live, both in
+`v1/core/kv_cache_utils.py`:
+
+1. **The page-size unifier refuses GLM's MLA family.** With a draft model the
+   spec dict gains non-MLA drafter attention specs and `HiddenStateCacheSpec`
+   entries, so `_get_kv_cache_groups_glm5_next` (which requires ALL attention
+   specs to be `MLAAttentionSpec`) bails to the generic grouping path — whose
+   `unify_kv_cache_spec_page_size` raises
+   `NotImplementedError: Layer language_model.model.layers.3.self_attn.indexer.k_cache: page size is not divisible… cannot be padded. Padding is only supported for non-MLA attention layers.`
+   MLA layers cannot be *padded* (sparse MLA indexes whole token rows) — but
+   they CAN grow `block_size` when the max page is an exact multiple. The
+   indexer's page is just `tokens_per_state x` the MLA page, so
+   **`glm53-dflash-02-kv-unify-mla-multiple.patch`** bumps the unified max page
+   to a multiple of every `MLAAttentionSpec` page before the loop: all MLA-family
+   layers then take the legal divisible block-growth branch, drafter + mamba
+   layers pad. The 2026-09-30 window proved it: unify passes, memory profiling
+   passes, torch.compile passes, and the boot reaches cudagraph capture — the
+   NotImplementedError is gone.
+2. **The kpool tail kernel pins its block.** Cudagraph capture then dies in
+   `kpool_ops.kpool_decode_update_and_maybe_write_cache_batched`:
+   `assert tail_kv_cache.shape[2] == pool_size`. The tail cache must view as
+   `[num_blocks, 2, index_kpool, head_dim]` (`KpoolTailSpec(block_size=index_kpool)`,
+   `Glm5NextTailCache` in `glm5next/common/attention.py:159`), but the tail's
+   page (8192 B) *divided* the raised max page, so the unifier's divisible branch
+   grew its `block_size` by the ratio and the view became
+   `[num_blocks, 2, index_kpool x ratio, head_dim]`. Verified not the hybrid
+   promotion: the log carries no "Hybrid KV cache manager is disabled" warning,
+   so `disable_hybrid_kv_cache_manager` is unset and
+   `unify_hybrid_kv_cache_specs` never ran. **`glm53-dflash-03-kv-unify-tailpin.patch`**
+   (= 02 + a `KpoolTailSpec` branch that pads `page_size_padded` instead of ever
+   growing its block) **verified 2026-09-30**: the assert is gone and the boot
+   captures the DFlash2 drafter's FULL cuda graphs on both TP workers
+   (`Capturing model for DFlash2 speculator...`). Supersedes 02 in the repo.
+
+3. **The capacity wall: the generic path cannot alias the mamba states.**
+   Past capture, the boot dies at `_check_enough_kv_cache_memory`:
+   `Available KV cache memory: 0.92 GiB` against `31.76 GiB needed at max len
+   65536` (32.91 GiB at 131,072 — nearly flat, a fixed intercept, not
+   per-token KV). Two coupled causes, read out of the image:
+   - **The glm5 tensor layout aliases the mamba states INTO the MLA pages**
+     (`get_kv_cache_config_from_groups` places every mamba layer at the MLA
+     offsets — that is why GLM-5.3 fits 32 GiB cards at all, and why the
+     expert plan's KV registry is calibrated at 13,600 B/token). On the
+     generic path the mamba groups claim their OWN blocks, so ~30 GiB of
+     unaliased state pages appears.
+   - **The plan's KV reserve (`expert_activation_registry.py:_glm53_kv_bytes_per_token`,
+     a flat 13,600 B/token) knows nothing about the drafter.** The engine caps
+     KV at the plan's reserve, so even a correct generic-path estimate would
+     be starved.
+
+   Learned en route: the engine AUTO-SETS the attention block size to
+   **2,432 tokens** (`interface.py:918: Setting attention block size to 2432
+   tokens to ensure that attention page size is >= mamba page size`), so every
+   attention group's block counts scale from that floor — a 16-token drafter
+   block would need 4x the shared `num_blocks`; with the global 2,432 all
+   groups align.
+
+   **`glm53-dflash-04-kv-groups-draft.patch`** (+ **`-04b-kv-registry-dflash.patch`**
+   for the registry) keeps the aliased glm5 pool and extends it: the grouping
+   partitions the hidden-state specs (`HiddenStateCacheSpec` is an
+   `MLAAttentionSpec` subclass — excluded before the exact-type
+   `type(spec) is MLAAttentionSpec` partition) and the drafter attention
+   specs; the drafter layers form their own uniform group (per-group pages,
+   NO unification — the MLA block size stays at the engine-set 2,432 and the
+   kpool guard `block_size % 512 == 0` is never stressed); the layout returns
+   a 10-tuple with `draft_group`/`hidden_groups`; the builder sizes ONE
+   shared backing allocation (`allocate_kv_cache` asserts all tensor sizes
+   equal) as aliased pool + appended draft/hidden tensors with their own
+   strides; the memory estimator adds the extras. Patch 04b extends the
+   registry: 13,600 + 4,096 (drafter 8 kv heads x 128 x K+V x bf16) +
+   5 x hidden_size x 2 B (the aux hidden-state cache) per token when
+   `speculative_config.use_aux_hidden_state_outputs` is set.
+
+   The pure logic is smoke-tested locally (CPU torch + the image's real spec
+   classes against synthetic GLM+draft+hidden spec sets): grouping shape,
+   layout recognition, builder block math and memory accounting all pass.
+   Real-fit arithmetic: per-token ≈ 58,656 B → reserve ≈ 4.15 GiB at max len
+   65,536 (arena shrinks ~13.5 → ~10.3 GiB, ~1,650 slots), num_blocks ≈ 29 x
+   2,432 tokens ≈ 70,500 ≥ 65,536.
+
+All of these mount over site-packages via the scratch launcher's
+`KVUNIFY_PATCH=1` knob (host `/home/dank/dflash-patch2/new/…`, patched shas
+`5521e50b…` for `kv_cache_utils.py` and `1a623d5c…` for
+`expert_activation_registry.py`; apply-verified against the image's pristine
+files). The window script's decision pipeline was also fixed for a SIGPIPE
+death (`head -80` on a `podman logs` grep pipeline killed one run before
+log-save and restore; the log is now captured to a file FIRST, then grepped).
+
+
+## Measured arms (all 2026-09-30, one boot each)
+
+| arm | decode | prefill | verdict |
+| --- | --- | --- | --- |
+| baseline int4, MAXLEN 131,072, util 0.92, NBATCH 2048 | 10.03 / 10.08 / 9.41 | 877 / 878 / 885 | qualified incumbent |
+| + `--expert-precise-memory` (arena 13.08 → 14.51 GiB) | 9.88 / 9.91 | 918 | **neutral** — proves residency is no longer the binding term |
+| + MTP4 spec decode (draft loads, acceptance excellent) | 9.64-11.16 | **255** | **rejected** — see below |
+| + MTP4 + explicit `--kv-cache-memory-bytes` (patched cap) | boots, 151,767-token KV | — | enabler for the MTP arm only |
+
+### Why MTP is rejected here (it is not that it fails)
+
+MTP **works** on GLM in this fork: the head loads, the architecture resolves to
+`Glm5NextMTPModel`, and acceptance is excellent —
+
+```
+SpecDecoding metrics: Mean acceptance length: 3.83-4.90,
+Per-position acceptance rate: 0.917/0.833/0.583/0.500 … 0.962/0.846/0.692/0.615,
+Avg Draft acceptance rate: 69-97%
+```
+
+— and it buys **nothing**: decode 9.6-11.2 tok/s against 10.0 without it, while
+prefill **collapses 918 → 255 tok/s**. The reason is structural to offload: a
+verification forward gathers the **union** of the batch's experts, so spec
+decode amortizes compute but not a residency-bound gather. It also costs 2.6 GiB
+of arena. On a compute-bound (fully-resident) shape this would invert; here it
+does not.
+
+## Served surface (verified 2026-09-30 against the live engine)
+
+| capability | state | evidence |
+| --- | --- | --- |
+| `/v1/models` | `glm53-flash`, max_model_len 131,072 | model list probe |
+| completion | yes | `bench_glm.py` runs above |
+| **tool calls** | **yes** — needs `--enable-auto-tool-choice --tool-call-parser glm47` | a `get_weather` prompt returns `tool_calls: [{"id": "chatcmpl-tool-…", "type": "function", "function": {"name": "get_weather", "arguments": "{\"city\": \"Seattle\"}"}}]`, `finish_reason: tool_calls` |
+| **vision** | **yes** | a 64x64 solid-red PNG is answered "The image appears to be a solid red color…" |
+| reasoning channel | **yes** — needs `--reasoning-parser glm47` | without it the thinking leaks into `content` (measured: content began with the answer, then "Let me verify: … = 391", then a bare `</think`) with an empty reasoning field; with it, a train-time prompt answers in `content` ("**Step 1: Add the hours.** 14:20 + 3 hours = 17:20 …") and the chain of thought lands in **`reasoning`** — the vLLM field name the fleet documents (llama.cpp endpoints use `reasoning_content`) |
+
+Without the tool flags the engine answers **HTTP 400** — `"auto" tool choice
+requires --enable-auto-tool-choice and --tool-call-parser to be set` — which
+breaks every tool-using agent on the endpoint. Both parsers are `glm47`, the
+newest GLM dialect the fork ships (`glm45`/`glm47` exist for both).
+
+## Fork traps found (for the engine authors)
+
+1. **`--kv-cache-dtype fp8` is illegal on gfx12 with this model — and it buys
+   nothing even if fixed.** The `ROCM_AITER_MLA_SPARSE` backend's
+   `_use_rocm_sparse_triton()` requires a non-fp8 KV cache, and that rope-free
+   Triton path is the **only** MLA decode available on gfx12: aiter's MLA decode
+   op is registered under `@if_aiter_supported`, which is gated on **CDNA > 2**
+   (`is_aiter_found_and_supported()` → `get_cdna_version() > 2`). With fp8 KV
+   the MTP draft step demands an op that cannot exist here:
+   `RuntimeError: Worker failed with error ''_OpNamespace' 'vllm' object has no
+   attribute 'rocm_aiter_mla_decode_fwd''`. KV dtype therefore stays auto/bf16.
+   (`VLLM_ROCM_USE_AITER_MLA=1` does **not** help — the umbrella gate is the
+   CDNA check, not the env var.)
+   - **The capacity payoff is measured at zero.** The point of fp8 KV is fitting
+     more tokens in the same reserve, and this fork's reserve model is
+     **dtype-blind**: the fp8 boot printed the *identical* plan line as bf16 —
+     `KV reserve 1.76 GiB (13600 B/token x 131072 + mamba pages)`. The reserve is
+     dominated by the per-sequence KDA state (34 linear-attention layers) and the
+     DSA indexer's pools, not by the MLA KV bytes those 13,600 B/token barely
+     cover — so halving the KV dtype cannot move it.
+   - **What a fix would take, if it were ever wanted**: a new fp8 path in the
+     rope-free Triton kernels (`rocm_sparse_attn_prefill` / `rocm_sparse_attn_decode`
+     in `v1/attention/ops/rocm_aiter_mla_sparse.py`) that dequantizes pages
+     in-kernel, plus loosening `_use_rocm_sparse_triton()`. The fork's
+     `_r4d_fp8_paged_mqa_logits` shows the author already has an fp8 MQA-logits
+     path, but the sparse-MLA backend is not wired to it. Verdict 2026-09-30:
+     **do not spend effort here** — the payoff is capacity, and the reserve is
+     dtype-blind.
+2. **The plan's KV reserve under-models this model, and the cap enforces the
+   mistake.** The plan charges 13,600 B/token + a small mamba term, but GLM
+   carries a per-sequence KDA state (34 linear-attention layers) and, under MTP,
+   the draft's KV. `gpu_worker` then *caps* KV at that reserve, so a legal
+   `--max-model-len` becomes unbootable:
+   `ValueError: To serve at least one request with the model's max seq len
+   (131072), (2.07 GiB KV cache is needed, which is larger than the available KV
+   cache memory (1.74 GiB))`. Raising `max_model_len` only helps past ~440k
+   tokens (the cap's slope is ~13,845 B/token vs the need's ~12,700 + 0.52 GiB
+   fixed) — where the reserve would eat a third of the arena.
+   `scripts/patches/gpu_worker-kv-cap.patch` lets an explicit operator KV
+   request raise the cap; **it is not part of the deployed shape**, which boots
+   on the plan's own reserve.
+3. **A window is forced, not chosen.** The production `max_model_len 131072` is
+   what the reserve model requires for this payload (below ~92k the boot
+   refuses); it is not a capacity decision.
+4. **Expert prefetch remains poison** — unchanged from the qwen campaign
+   (36% precision, +40-51% decode from deleting it). No GLM arm re-opened it.
+
+## Operations
+
+```bash
+# rollback to the pre-cutover qwen FP8 + MTP4 shape (payload and R9V state
+# dirs are untouched on disk):
+git show 2339fae -- scripts/ | git apply   # or: git checkout 2339fae -- scripts/
+./scripts/deploy.sh dankserver --service frank-primary
+
+# payload inventory (34 shards + 2 extras, weight-map index):
+bash scripts/download-frank-primary.sh --verify-only
+
+# live telemetry (needs the profiler window; with_stack=false is mandatory —
+# with stack the stop path hangs and the trace buffer grows to tens of GB):
+podman logs frank-primary-glm 2>&1 | grep "expert_lru.py"
+```
+
+**Experiment-window discipline (learned the hard way, 2026-09-30).** This engine
+**ignores SIGTERM**: after `systemctl stop` its container sits in `podman ps` state
+`Stopping` for 20+ minutes *while still answering `/health` with 200*. That makes a
+liveness probe a lying oracle for a restore — twice a restore script printed
+`PROD HEALTHY` while `systemctl is-active` said `failed` and the runtime mask was
+still in place. The restore oracle must be a **conjunction**: `is-active` ==
+`active` **and** `is-enabled` == `enabled` **and** no `/run/systemd/system/<unit>`
+mask symlink **and** the container is owned by the unit and not `Stopping`. The
+repair recipe, all four steps needed:
+
+```bash
+sudo mv /run/systemd/system/frank-primary.service /tmp/stale-mask-N.service  # runtime mask SURVIVES stop and blocks start
+sudo systemctl daemon-reload
+sudo systemctl reset-failed frank-primary.service                            # a failed unit refuses to start
+podman rm -f frank-primary-glm                                               # SIGKILL; `podman stop` hangs on the same SIGTERM
+sudo systemctl start frank-primary.service                                   # then ~7 min to health 200
+```
+
+Also: **never let two experiment jobs overlap.** A later job's `podman rm -f
+glm-probe` cleanup ran while an earlier job was still mid-flight and deleted its
+container, so one whole round produced no evidence at all (`no container with name
+glm-probe found`) — check `proc://` for running jobs before opening a window.
+
+**Never put a diagnostic hook on `PYTHONPATH` without a marker guard — it is a fork
+bomb.** On 2026-09-30 a `sitecustomize.py` mounted at `/probe` (to wrap dynamo's
+name-map lookups and vLLM module forwards) produced a self-replicating chain of
+`python3` processes 19+ levels deep inside the probe container: `sitecustomize` is
+imported by *every* interpreter, the hook imported torch/vLLM at startup, something in
+that import chain spawned a helper python, and the child inherited `PYTHONPATH` (hence
+the hook) and spawned again. It was still alive 18 minutes later with the container
+still showing `Up`. Symptoms: `ps -eo pid,ppid,comm --forest` shows a deep ladder of
+identical interpreters, each child's PPID being the previous one. Guard (now in the
+hook): arm once per environment via a marker in `os.environ`, which children inherit —
+
+```python
+if os.environ.get("VLLM_PROBE_ARMED") == "1":
+    pass                       # an ancestor already armed it; stay inert
+else:
+    os.environ["VLLM_PROBE_ARMED"] = "1"
+    <install probes>
+```
+
+Kill recipe: `podman rm -f <experiment-container>` (the container cgroup owns every
+generation), then verify with a precise count over `/proc/<pid>/environ` for the marker
+variable — **never** pattern-kill an interpreter name, because the media stack on this
+host (romm, jellyfin, plex, glances) runs python in production.
+
+```bash
+# boot: ~7 min warm page cache, longer cold; health returns 503 while loading
+curl -s -o /dev/null -w "%{http_code}\n" http://127.0.0.1:7770/health
+
+# experiment windows (production stopped + `systemctl mask --runtime`, which
+# self-clears at reboot) ran from the host-side scratch launcher
+# /home/dank/serve_glm_autoround.sh: env knobs MAXLEN SEQS NBATCH UTIL KVBYTES
+# KVDTYPE SPEC SPECJSON_FILE TOOLPARSER REASONINGPARSER TELEM GCHUNKS GLANES
+# PROFDIR GLM_PATCH_MOUNT COMPILE_PATCH GATHER_PATCH DRAFT_MOUNT ENVARGS.
+# DRAFT_MOUNT=1 mounts /home/dank/models/GLM-5.3-Flash-DFlash2 at /draft;
+# SPECJSON_FILE passes a whole --speculative-config JSON (JSON with no spaces,
+# word-split by the launcher's unquoted $EXTRA).
+bash /home/dank/serve_glm_autoround.sh
+```
+
+Boot gate: the engine refuses a legal-looking shape in two measured ways — the
+KV cap above, and the profile-run activation peak (`TritonMLA` indexer scores:
+64 heads x 2,048 topk x NBATCH tokens x 2 B = 2.06 GiB at NBATCH 8192), which is
+what killed the first two boots at util 0.95.
+
+`PYTORCH_CUDA_ALLOC_CONF=expandable_segments:True` is part of the qualified
+shape, not a tuning: the plan fills VRAM to its modelled headroom by design, so
+the default allocator's fragmentation fails the profile/graph-capture phase
+(measured on the deployed unit: "Tried to allocate 662.00 MiB. GPU 0 … 240.00
+MiB is free").
+
+The `deploy.sh` e2e gate also drives the admin panel's open-links spec, which
+walks every guarded vhost — including `http://192.168.0.112:8989/` on
+FrankBook-Pro. When that Mac mini is asleep the spec fails on a `page.goto`
+timeout and the deploy reports `DEPLOY INCOMPLETE` even though the engine is
+healthy; wake the host (magic packet to `de:6b:6d:c1:52:23`) or re-run
+`tests/e2e/run.sh` once it is up. Seen 2026-09-30: 32 passed, this one failed,
+and the WoL packet did not wake it (macOS "Wake for network access" needs
+toggling on the mini itself).
+
+## Rollback targets
+
+| state | where |
+| --- | --- |
+| qwen FP8 + MTP4 (2026-09-29 → 09-30) | commit `2339fae`, payload `/home/dank/models/Qwen3.8-Flash-Next-Uncensored-FP8` |
+| R9V radiance Qwen3.8-Flash-Next Q8_0 | [[frank-primary-r9v-radiance-qwen38-q8]], state dirs `/home/dank/r9v-state/{q8xl,uncensored}` |

serverAppConfigs-docs/LLM/frank-primary-r9v-radiance-qwen38-q8.md

@@ -40,7 +40,7 @@ The repo carries a **thin** wiring; the engine shape lives on-host:
 | Piece | File | Law |
 |---|---|---|
 | Unit | `scripts/frank-primary.service` | Root unit, `User=dank`, `LimitMEMLOCK=infinity`, `ExecStop=podman stop -t 120 frank-primary-r9v`, `Restart=on-failure` |
-| Env | `scripts/frank-primary.env` | Production overrides ONLY: bind 0.0.0.0, :7770, alias `frank-primary`, container `frank-primary-r9v`, GPU BDFs, dev-overlay adapter, `R9V_FULL_MUTABLE_CACHE=0`, sharp chat template |
+| Env | `scripts/frank-primary.env` | Production overrides ONLY: bind 0.0.0.0, :7770, alias `frank-primary`, container `frank-primary-r9v`, GPU BDFs, dev-overlay adapter, `R9V_FULL_MUTABLE_CACHE=0`, `R9V_PLE_READY_TIMEOUT=1800`, prefix caching ON, sharp chat template |
 | Serve | `scripts/serve-frank-primary.sh` | Layers setup state under repo env → execs the fork's own `scripts/launch.sh` via the docker→podman shim → `docker wait` |
 | Deployer | `scripts/deploy-remote.sh` `deploy_llm_primary` | Gates on setup state ready, image digest, PLE bytes, all six shard files + mmproj; MainPID false-green check |
 
@@ -140,9 +140,11 @@ edited in place — mutating it drifts the qualification.
   ssd-mmap residency).
 - Context envelope: `R9V_MAX_MODEL_LEN=131072`; harness `models.yml`
   carries `contextWindow: 131072`, `maxTokens: 36864` (compaction-gap
-  math: 70% × 131,072 = 91,750 + 36,864 = 128,614 ≤ 131,072).
+  math: 70% × 131,072 = 91,750 + 36,864 = 128,614 ≤ 131,072). The KV
-- Prefix caching OFF (`R9V_ENABLE_PREFIX_CACHING=0`), CACHE192 off — the
+  envelope at 131,072 is smaller than the uncensored 196,608 shape.
-  KV envelope at 131,072 is smaller than the uncensored 196,608 shape.
+- Prefix caching OFF at cutover (`R9V_ENABLE_PREFIX_CACHING=0`), CACHE192
+  off — superseded 2026-09-28: prefix caching is ON (see bench matrix
+  below).
 
 ## Prefill / decode bench matrix (2026-09-28, scratch port 7773)
 
@@ -183,9 +185,10 @@ prompt TTFT 44.2 s → **3.6–4.3 s** (9x), 131k 439 s → **6.9 s** (63x).
 Hit pattern: rounds 1–2 miss, round 3+ hit (identical-prompt and
 conversation-growth probes both). For the harness's resend-every-turn
 traffic that is a ~40–440 s TTFT win per steady-state turn; the one-turn
-miss costs nothing extra. **RECOMMENDED for production — pending Frank's
+miss costs nothing extra. **DEPLOYED 2026-09-28** (Frank's go-ahead): the
-go-ahead** (one env line; the launcher default is already ON, the q8 state
+repo env sets `R9V_ENABLE_PREFIX_CACHING=1` (commit `41e9189`), verified
-opted out).
+on :7770 (`--enable-prefix-caching` in the container argv, health 200,
+live completion answering).
 
 ## Ops
 

serverAppConfigs-docs/LLM/frank-small-llama-mimo-q5km.md

@@ -0,0 +1,171 @@
+---
+type: reference
+title: "frank-small on Dankclaw — llama.cpp, MiMo-V2.6-Distill-Qwen-9B Q5_K_M GGUF"
+description: "The active frank-small runbook — a CUDA (sm_86) build of llama.cpp master serving bartowski's MiMo-V2.6-Distill-Qwen-9B Q5_K_M GGUF on the RTX 3090: three conversations of 180,224 tokens, mmproj vision, the asymmetric KV contract (f16 K + q8_0 V), the two VRAM measurements that set the pool size, and what to check when it misbehaves."
+tags:
+  - llm
+  - frank-small
+  - dankclaw
+  - llama-cpp
+  - rtx3090
+  - mimo
+  - gguf
+  - kv-cache
+timestamp: 2026-10-02
+---
+
+# frank-small — llama.cpp MiMo-V2.6 Q5_K_M (RTX 3090)
+
+> **ACTIVE since 2026-10-02.** One quant step down from
+> [[frank-small-llama-mimo-q6k]] on the same runtime and the same KV
+> contract, then **resized twice the same day** on Frank's orders: first to
+> three contexts with vision, then down to the pool that keeps 2–3 GiB of the
+> card free. Frank's arithmetic request — "as much as can fit while keeping
+> overhead open space to 2-3gb on the 3090" — is answered by the two
+> measurements in *Shape and the KV contract* below; the answer is
+> **3 × 180,224**, not 3 × 262,144.
+
+> **The tradeoff is the per-slot window.** Three sessions cost 81,920 tokens
+> of per-conversation memory versus the two-slot shape: 180,224 tokens each
+> instead of the model's native 262,144. See *What three slots cost*.
+
+## Identity
+
+|Field|Value|
+|---|---|
+|Endpoint|`192.168.0.143:7771`, served alias `frank-small`|
+|Runtime|llama.cpp **master** (qwen35 model support), CUDA build `sm_86`, at `/home/dank/GitPackages/llama-master/build-cuda/bin/llama-server` — unchanged across the quant swap|
+|Payload|`bartowski/MiMo-V2.6-Distill-Qwen-9B-GGUF` **Q5_K_M**, 6,876,124,704 B (6.40 GiB), sha256 `04cf46f2b5584ef922697d6c063f94879434fb86c467f1a09e52daf0cddaf35f` + the repo's `mmproj-MiMo-V2.6-Distill-Qwen-9B-f16.gguf`, 918,166,048 B, sha256 `ff348f3180a63188aa7285db85f550fe38acb61dd013c599eb8bad08d2cc2576` — both at `/home/dank/models/MiMo-Q5KM/`, fetched and byte- and sha256-verified by `scripts/download-frank-small.sh`|
+|Unit|root `frank-small.service` (repo `scripts/frank-small.service`, `User=dank`, relinked), env `scripts/frank-small.env`, serve `scripts/serve-frank-small.sh`|
+|Repo pins|`tests/frank_small_unit.bats` (stub-argv seam + the VRAM arithmetic), `deploy_llm_small` gates in `scripts/deploy-remote.sh`|
+
+## Shape and the KV contract
+
+`--ctx-size 540672 --parallel 3` — llama.cpp divides the pool across slots, so
+each conversation gets `n_ctx_slot = 180224` (boot log: `n_slots = 3,
+n_ctx_slot = 180224, kv_unified = 'false'`). `CTX_SIZE` MUST stay divisible by
+`PARALLEL`; the bats seam enforces it.
+
+The KV contract is Frank's: **K unquantized (f16), V `q8_0`** —
+`--cache-type-k f16 --cache-type-v q8_0`. Two hard dependencies ride on it:
+
+- **`--flash-attn on` is mandatory**: quantized V exists only on the FA path.
+  Without the flag the engine falls back silently and the contract becomes a
+  lie. The serve script pins the trio; the bats seam pins the script.
+- The VRAM budget: attention KV lives in the 8 full-attention layers only
+  (4 KV heads × 256 dim). K f16 ≈ 16 KiB/token, V q8_0 ≈ 8.5 KiB/token
+  (34 B per 32-element block) ⇒ ~24.5 KiB/token over the pool.
+
+**The pool size came from two boots, not from a formula.** `free`-style
+arithmetic on this stack is unreliable because the compute buffers, CUDA
+graphs and now the vision tower all bill separately, so the fit was measured
+at two shapes and solved between them:
+
+|Shape|Projector|`nvidia-smi` used|KV at 24.5 KiB/token|Non-KV (weights + buffers + projector)|
+|---|---|---|---|---|
+|2 × 262,144 (pool 524,288)|none|20,190 MiB|12,800 MiB|**7,390 MiB**|
+|3 × 196,608 (pool 589,824)|f16|22,972 MiB|14,113 MiB|**8,859 MiB**|
+
+The two rows price the projector and its image buffers at **~1.4 GiB** — not
+the ~0.9 GiB the 918 MB file suggests, because the vision encoder's own
+activation and compute buffers ride along. Solving the second row's non-KV cost
+against Frank's 2–3 GiB band: the card is 24,576 MiB, so
+`KV ≤ 24,576 − 9,216 − 2,048 = 13,312 MiB` ⇒ pool ≤ 556,630 tokens, rounded
+down to a multiple of 3 × 4,096 ⇒ **540,672**, i.e. 13,265 MiB of KV and
+**2,441 MiB free** (measured, 21,684 MiB used).
+
+**Why 3 × 262,144 was never possible.** Three native windows are 786,432
+tokens = 18.4 GiB of KV *alone*; plus the 8.86 GiB of non-KV cost that is
+27.3 GiB on a 24 GiB card. The quant step bought 0.86 GiB; the geometry needed
+6 GiB.
+
+## Measured on this shape (2026-10-02, 3 × 180,224 + vision)
+
+|Check|Result|
+|---|---|
+|Boot to healthy|~20 s, `n_slots = 3`, `n_ctx_slot = 180224`, `loaded multimodal model` logged|
+|VRAM|**21,684 MiB used / 2,441 MiB free** of 24,576 (`nvidia-smi`, steady state; the KV pool is allocated whole at boot, so this is the ceiling)|
+|Decode, temp 0, no thinking|**106.8 tok/s** (256-token completion at ~1,300-token depth)|
+|Prefill|**3,814 tok/s** (same request)|
+|Three concurrent requests|all three answered in 1.7 s wall — real parallel slots|
+|Fourth concurrent request|answered in 3.0 s while three ran — queued behind a released slot, as the slot ceiling requires|
+|Vision|`image_url` part with a 64×64 solid-red PNG → **"Red."**|
+|Tools|native `tool_calls` with correct arguments (`get_weather {"city":"Seattle","unit":"f"}`)|
+|Thinking|`reasoning_content` populated (145 chars on a "think first" prompt); `reasoning_effort: "low"` accepted|
+|Over-slot prompt|`400 exceed_context_size_error: request (184001 tokens) exceeds the available context size (180224 tokens)` — refused, never silently truncated|
+
+Decode is ~5 % **faster** than the 2 × 262,144 shape measured earlier the same
+day (100.4–101.8 tok/s): a shorter per-slot window means a shorter attention
+scan at equal prompt depth, and one less slot's DeltaNet state resident.
+
+## What three slots cost
+
+The per-slot window is the whole price. Two consequences the fleet depends on:
+
+- **A prompt over 180,224 tokens is a hard 400**, so `~/.omp/agent/models.yml`
+  sets `contextWindow: 180224` for this endpoint — the harness compacts before
+  it reaches the ceiling instead of discovering it.
+- **frank-primary's compaction prompt no longer fits with room to answer.**
+  Its trigger is 183,501 tokens — already above this slot — and its summary
+  budget is 32,768 output tokens. On the two-slot shape (262,144) that left
+  32.8k of output room, which is why that number was chosen; on this shape a
+  single compaction request must be truncated or split. Nothing in the fleet
+  routes compaction here automatically, but any runbook that claims frank-small
+  "fits a compaction request in ONE request" is describing the 262,144 shape.
+
+## Client contract
+
+- Thinking arrives in **`reasoning_content`** (`--reasoning-format deepseek`),
+  the field `models.yml` reads for this endpoint.
+- **Thinking is off unless the request asks**: the MiMo GGUF template defaults
+  `enable_thinking` false, so `models.yml` ships
+  `chat_template_kwargs: {enable_thinking: true, preserve_thinking: true}` in
+  `extraBody`. A raw client that omits it gets a direct answer — measured, and
+  not a fault. Trivial questions produce no thinking even with the flag on.
+- Tools: `--jinja` + server-side parsing, verified 2026-10-02.
+- **Vision is on**, through the projector file (`--mmproj`). Use the
+  OpenAI-standard `image_url` content part — that path answers correctly
+  ("Red." on a solid-red PNG). The legacy top-level `images` field is **not**
+  read by `/v1/chat/completions`: the same request sent that way answered
+  "Brown.", i.e. it generated from text alone. `/completion` takes `images`
+  but refuses a request without `messages` in this build, so chat is the only
+  vision path.
+- Setting `MMPROJ_FILE=` (explicitly empty) in `scripts/frank-small.env` drops
+  `--mmproj` and returns ~1.4 GiB of the card to KV — the documented text-only
+  opt-out. `MMPROJ_FILE` pointing at a file that does not exist is a boot
+  refusal, never a silent fallback to blindness.
+
+## No spec decode
+
+The GGUF has no MTP tensors (the same fact that made SGLang's NEXTN a 0.00
+ghost), and the cross-model DFlash draft
+(`/home/dank/models/DFlash-Qwen35/qwen3.5-9b-dflash-Q8_0.gguf`) is an
+optimization candidate, not a default — acceptance has to be measured on this
+traffic first. The bats seam fails if a draft flag ships unannounced.
+
+## Rollback
+
+|Target|How|
+|---|---|
+|Q6_K, same runtime|`scripts/frank-small.env` `MODEL_FILE=/home/dank/models/MiMo-Q6K/MiMo-V2.6-Distill-Qwen-9B-Q6_K.gguf` (7,793,710,624 B, still on disk) and the deployer's `LLM_SMALL_MODEL_FILE` default. Q6_K needs ~1.4 GiB more than Q5_K_M, so pair the rollback with `CTX_SIZE` 49,152 tokens lower or `MMPROJ_FILE=`|
+|Two 262,144 windows|`CTX_SIZE=524288 PARALLEL=2` (measured 20,190 MiB used, 4,386 free) and `contextWindow: 262144` in `models.yml` — choose it when a single conversation needs the native window or when frank-primary's compaction request has to land here|
+|Vision off|`MMPROJ_FILE=` in the env file; the deployer gate then has to be satisfied too, so point `LLM_SMALL_MMPROJ` at any real projector file or drop the gate in the same commit|
+|SGLang + INT8|[[frank-small-sglang-mimo-v26]] — venv and old unit/env/serve trio still in place; the INT8 payload was staged to `/home/dank/.AGENTTRASH/20261002-retired-llm-payloads/` on 2026-10-02, so restore it there or re-fetch with `old/download-frank-small-sglang-int8.sh`|
+
+## When it misbehaves
+
+- Port 7771 dark: `journalctl -u frank-small -n 50`; the serve script exits 1
+  naming the missing file if the binary or GGUF moved.
+- `No CUDA GPUs are available`: the 3090 vanished from the bus or the UVM node
+  is missing — `nvidia-modprobe -u -c=0` (the serve script runs it).
+- Requests queue behind each other: three concurrent long generations are the
+  slot ceiling by contract; a fourth waits for a slot to free (measured: 1.7 s
+  for the three, 3.0 s for the fourth).
+- `400 … exceeds the available context size (180224 tokens)`: the conversation
+  outgrew ONE slot. This is not pool exhaustion — the pool is 540,672 tokens
+  and the per-slot window is the binding limit. Shorten the prompt, compact, or
+  roll back to the 262,144 shape above.
+- Vision answers a color it never saw: the client sent `images` instead of an
+  `image_url` content part (see *Client contract*).
+- Slow decode after a CTX or quant change: check `nvidia-smi` — pushing KV out
+  of VRAM into unified memory halves the rate silently.

serverAppConfigs-docs/LLM/frank-small-llama-mimo-q6k.md

@@ -0,0 +1,110 @@
+---
+type: reference
+title: "frank-small on Dankclaw — llama.cpp, MiMo-V2.6-Distill-Qwen-9B Q6_K GGUF (retired)"
+description: "The retired 2026-10-01 frank-small runbook — the first llama.cpp + GGUF shape on the RTX 3090: two native-window sessions, the asymmetric KV contract (f16 K + q8_0 V) that motivated the move off SGLang, the VRAM fit at Q6_K, and what to check when it misbehaves. Superseded the same night by the Q5_K_M payload."
+tags:
+  - llm
+  - frank-small
+  - dankclaw
+  - llama-cpp
+  - rtx3090
+  - mimo
+  - gguf
+  - kv-cache
+timestamp: 2026-10-01
+---
+
+# frank-small — llama.cpp MiMo-V2.6 Q6_K (RTX 3090)
+
+> **RETIRED 2026-10-01 → 10-02** (one night): superseded by
+> [[frank-small-llama-mimo-q5km]], the same runtime and shape one quant rung
+> down. The Q6_K file stays at `/home/dank/models/MiMo-Q6K/` as the rollback
+> target.
+>
+> Original order (Frank, 2026-10-01 night): move this endpoint to the GGUF
+> path for **asymmetric KV** — unquantized K + q8_0 V — which only llama.cpp
+> exposes per-tensor. That contract is what the current deployment kept.
+
+## Identity
+
+|Field|Value|
+|---|---|
+|Endpoint|`192.168.0.143:7771`, served alias `frank-small`|
+|Runtime|llama.cpp **master** (qwen35 model support), CUDA build `sm_86`, at `/home/dank/GitPackages/llama-master/build-cuda/bin/llama-server`|
+|Payload|`bartowski/MiMo-V2.6-Distill-Qwen-9B-GGUF` **Q6_K**, 7,793,710,624 B, sha256 `ef96d05a2ddf2cbb450d1af1ac3860ec769d3575bafa70692ee5609bda3fad7d`, at `/home/dank/models/MiMo-Q6K/`|
+|Unit|root `frank-small.service` (repo `scripts/frank-small.service`, `User=dank`, relinked), env `scripts/frank-small.env`, serve `scripts/serve-frank-small.sh`|
+|Repo pins|`tests/frank_small_unit.bats` (stub-argv seam), `deploy_llm_small` gates in `scripts/deploy-remote.sh`|
+
+## Shape and the KV contract
+
+`--ctx-size 524288 --parallel 2` — llama.cpp divides the pool across slots,
+so each session gets `n_ctx_slot = 262144`, the model's full native window
+(boot log confirms `n_slots = 2, n_ctx_slot = 262144, kv_unified = 'false'`).
+
+Frank's KV contract: **K unquantized (f16), V `q8_0`** —
+`--cache-type-k f16 --cache-type-v q8_0`. Two hard dependencies ride on it:
+
+- **`--flash-attn on` is mandatory**: quantized V exists only on the FA
+  path. Without the flag the engine falls back silently and the contract
+  becomes a lie. The serve script pins the trio; the bats seam pins the
+  script.
+- The VRAM budget: attention KV lives in the 8 full-attention layers only
+  (4 KV heads x 256 dim). K f16 ≈ 16 KiB/token, V q8_0 ≈ 8.5 KiB/token
+  (34 B per 32-element block) ⇒ ~24.5 KiB/token over the 524,288-token pool
+  ≈ **12.6 GiB**. Weights 7.8 + KV 12.6 + compute buffers/graphs ≈
+  **20.9 GiB measured** against 24.5 GiB — the "room for checkpoints" the
+  order asked for. If graph capture OOMs, halve `CTX_SIZE`; never quantize
+  K to fix it.
+
+## Client contract
+
+- Thinking arrives in **`reasoning_content`** (`--reasoning-format
+  deepseek`) — the same field SGLang's qwen3 parser used, so `models.yml`
+  `reasoningContentField` did NOT move. Live-verified 2026-10-01 22:57.
+- **Thinking is off unless the request asks**: the MiMo GGUF template
+  defaults `enable_thinking` false; `models.yml` now ships
+  `chat_template_kwargs: {enable_thinking: true, preserve_thinking: true}`
+  in `extraBody`. A raw client that omits it gets a no-thinking answer
+  (measured — this is not a fault).
+- Tools: `--jinja` + server-side tool parsing, verified
+  `tool_calls` round-trip 2026-10-01 23:0x.
+- **Text-only now.** The GGUF carries no vision projector; the
+  in-checkpoint tower existed only in the SGLang safetensors. `models.yml`
+  `input: [text]`.
+
+## No spec decode (yet)
+
+The GGUF has no MTP tensors (same fact that killed SGLang NEXTN), and the
+cross-model DFlash draft (`/home/dank/models/DFlash-Qwen35/qwen3.5-9b-dflash-Q8_0.gguf`,
+Q8_0) is an optimization-loop candidate, not a default — `--spec-type
+draft-dflash` needs acceptance measured on this traffic first. The bats
+seam fails if a draft flag ships unannounced.
+
+## Build
+
+No ccache, CUDA 12.4 system nvcc with `--gcc-install-dir`-free gcc-13 host
+compiler (nvcc 12.4 rejects gcc ≥ 14):
+
+```
+cmake -B build-cuda -S . -DCMAKE_BUILD_TYPE=Release -DGGML_CUDA=ON \
+  -DCMAKE_CUDA_ARCHITECTURES=86 -DCMAKE_CUDA_HOST_COMPILER=/usr/bin/gcc-13 \
+  -DLLAMA_CURL=OFF -DGGML_CCACHE=OFF
+cmake --build build-cuda -j 16 --target llama-server
+```
+
+The SGLang venv (`/home/dank/GitPackages/sglang-cu128`) and the INT8
+AutoRound payload (`/home/dank/models/MiMo-V2.6-Distill-Qwen-9B-INT8-W8A16-AutoRound`)
+stay on disk as the rollback: repoint the unit's serve script and env from
+git history (`git show frank-large-gufo-q4kxl-20261001` era is BEFORE this;
+use the parent of the 2026-10-01 night swap commit).
+
+## When it misbehaves
+
+- Port 7771 dark: `journalctl -u frank-small -n 50`; the serve script
+  exits 1 naming the missing file if the binary or GGUF moved.
+- `No CUDA GPUs are available`: the 3090 vanished from the bus or the UVM
+  node is missing — `nvidia-modprobe -u -c=0` (the serve script runs it).
+- Requests hang at first token while `/health` is 200: both slots busy —
+  two concurrent long generations are the ceiling by contract.
+- Slow decode after a CTX change: check `nvidia-smi` — swapping KV out of
+  VRAM into unified memory halves the rate silently.

serverAppConfigs-docs/LLM/frank-small-sglang-mimo-v26.md

@@ -0,0 +1,180 @@
+---
+type: reference
+title: "frank-small on Dankclaw — SGLang, MiMo-V2.6-Distill-Qwen-9B INT8 W8A16 (RETIRED)"
+description: "The RETIRED frank-small runbook (2026-10-01, hours) — SGLang in a self-contained python3.12 venv serving leteschlead MiMo-V2.6-Distill-Qwen-9B-INT8-W8A16-AutoRound (qwen3_5 hybrid, 9B dense, in-checkpoint MTP + vision) on the RTX 3090: identity, payload, KV arithmetic, the NEXTN spec-decode ghost, the CUDA 13 nvcc toolchain the JIT needs, and the measured numbers kept for the record. Replaced by [[frank-small-llama-mimo-q6k]] the same night."
+tags:
+  - llm
+  - frank-small
+  - dankclaw
+  - sglang
+  - rtx3090
+  - qwen
+  - mtp
+  - vision
+  - int8
+timestamp: 2026-10-01
+---
+
+# frank-small — SGLang MiMo-V2.6-Distill-Qwen-9B INT8 W8A16 (NEXTN MTP)
+
+> **RETIRED 2026-10-01, hours after cutover.** Replaced by
+> [[frank-small-llama-mimo-q6k]] — llama.cpp CUDA on the MiMo Q6_K GGUF,
+> chosen for the asymmetric KV (f16 K + q8_0 V) SGLang cannot express.
+> The venv (`/home/dank/GitPackages/sglang-cu128/`) stays in place; the INT8
+> payload was staged to
+> `/home/dank/.AGENTTRASH/20261002-retired-llm-payloads/` on 2026-10-02 when
+> the llama.cpp endpoint was two runtimes deep — recoverable until the trash
+> is purged.
+
+## Identity
+
+- **Endpoint:** `frank-small` at `192.168.0.143:7771` (Dankclaw).
+- **Runtime:** SGLang (latest) in a self-contained python3.12 venv at
+  `/home/dank/GitPackages/sglang-cu128/` (torch 2.13.0+cu130,
+  transformers 5.12.1, flashinfer 0.6.18, sgl-kernel). No container — a root
+  systemd unit `scripts/frank-small.service` (`User=dank`,
+  `LimitMEMLOCK=infinity`) launched by `scripts/serve-frank-small.sh` with
+  tunables from `scripts/frank-small.env`.
+- **Model:** `letechlead/MiMo-V2.6-Distill-Qwen-9B-INT8-W8A16-AutoRound`
+  (9B dense, 12.0 GiB, INT8 W8A16 AutoRound group-128 symmetric).
+
+## Shape
+
+- **Architecture `qwen3_5` hybrid:** 32 layers = 8 full-attention
+  (4 KV heads x 256 head_dim, `partial_rotary_factor: 0.25`) + 24 Gated
+  DeltaNet. `max_position_embeddings: 262144`, and a `qwen3_5_vision`
+  tower (27 layers) in the safetensors — no separate mmproj file.
+- **Context:** two 262,144-token slots at the model's native window
+  (`--context-length 262144`, `--max-running-requests 2`) against a
+  **pinned** 606,208-token pool (`--max-total-tokens`) — 2026-10-01
+  reslice from four 151,552-token slots: 2 x 262,144 = 524,288 leaves
+  ~82k tokens of pool slack, and a compaction request (frank-primary's
+  trigger 183,501 + a 32,768-token summary budget = 216,269) now fits
+  one request. A single request may grow to the whole pool — a
+  170,019-token prompt was verified at the 4-slot shape — the grid is
+  the two-way concurrency contract. **That compaction arithmetic is
+  specific to a 262,144-token slot**: the ACTIVE shape runs three slots of
+  180,224, which 183,501 tokens already exceeds — see *What three slots
+  cost* in [[frank-small-llama-mimo-q5km]].
+- **KV:** `fp8_e4m3` — 16 KiB/token (8 attn layers x 2 x 4 heads x 256
+  dim x 1 B), pool 9.31 GiB.
+- **The memory shape (the 4.5 GiB the naive math misses):** the DeltaNet
+  ssm state pool is a first-class consumer. The engine autofills the
+  mamba slot count to spend a fixed ~5.1 GB budget *regardless of dtype*
+  (fp32 at 110 slots, bf16 at 217 — same 5.1 GB), so both dials are
+  pinned: `--mamba-ssm-dtype bfloat16 --max-mamba-cache-size 32` = 795
+  MiB. With that, weights 11.35 GiB + mamba 0.78 GiB + pool 9.31 GiB
+  leave ~0.8 GB — enough for eager-prefill workspaces, NOT enough for
+  CUDA-graph capture (autofit's 687,388-token pool leaves 0.53 GB and
+  dies in capture, or later in the first long request once Triton's
+  deferred kernel loads eat the remainder). Hence
+  `--disable-prefill-cuda-graph`; decode graphs are untouched.
+- **Quantization:** `--quantization auto-round` — the checkpoint's
+  `quantization_config.json` declares `quant_method: "auto-round"` and
+  SGLang loads it natively (vision tower, `embed_tokens` and
+  `linear_attn.in_proj` stay fp16).
+
+## Spec method
+
+**None — and this is a checkpoint fact, not a tuning choice.** The
+`text_config` declares `mtp_num_hidden_layers: 1`, but the shipped
+safetensors carry **zero `mtp.*` tensors**: verified 2026-10-01 against
+`model.safetensors.index.json` for both this INT8 conversion and the base
+`XiaomiMiMo/MiMo-V2.6-Distill-Qwen-9B` (32 layers, indices 0–31, no
+`mtp.`/`nextn.`/`layers.32` keys in either). With NEXTN enabled, SGLang
+boots a 4.24 GiB draft model (`Qwen3_5ForCausalLMMTP`) from nothing and
+decodes at `accept len: 1.00, accept rate: 0.00` — every proposal rejected,
+paying the 4-token verify tax for no token. The serve script passes no
+`--speculative-*` flag, and the seam gates that.
+
+## CUDA 13 toolchain (the sharp edge)
+
+The venv's torch is cu130, so every JIT kernel (sgl-kernel marlin repack,
+flashinfer) compiles with a **CUDA 13.0 nvcc extracted from debs into the
+venv** at `cuda-nvcc/usr/local/cuda-13.0/` — the system nvcc is 12.4 (no
+`--compress-mode`) and its headers clash with the 13.0 device passes.
+
+Components (all from the ubuntu2404 CUDA repo, `dpkg -x`'d, not installed):
+
+| Package | Supplies |
+|---|---|
+| `cuda-nvcc-13-0` (13.0.88) | `bin/nvcc`, `ptxas`, `cudafe++`, … |
+| `libnvvm-13-0` (13.0.48) | `nvvm/bin/cicc` (the device compiler) |
+| `cuda-cudart-dev-13-0` (13.0.88) | `include/cuda_runtime.h`, `lib64/stubs` |
+| `cuda-crt-13-0` (13.0.88) | `include/crt/host_runtime.h` (2-arg `__cudaLaunch`) |
+| `libnvptxcompiler-13-0` (13.0.88) | static lib + `nvPTXCompiler.h` |
+| `cuda-cccl-13-0` (13.0.50) | `cccl/cuda/...` headers (libcudacxx) |
+
+Three local modifications the tree carries:
+
+1. **`bin/nvcc` is a shim** over `bin/nvcc.real` adding
+   `-allow-unsupported-compiler` (host GCC 16 > nvcc's max 13).
+2. **`targets/x86_64-linux/include` and `lib` are symlinks** to
+   `../../../include` / `../../../lib64` (the deb ships them as real
+   near-empty dirs; the stub pass must see the 13.0 headers, not
+   `/usr/include` 12.4).
+3. **`include/cuda` → `cccl/cuda`** symlink (CCCL 3.x layout; flashinfer
+   includes `<cuda/functional>`), and **`crt/math_functions.h`** patched with
+   `noexcept` on `rsqrt`/`rsqrtf` (glibc 2.42 declares them `noexcept(true)`;
+   the unpatched 13.0 declarations are incompatible in the C++ host stub).
+
+JIT caches: `~/.cache/sglang/jit/sm86/` and
+`~/.cache/sglang/.cache/flashinfer/0.6.18/86/` — clear both when a toolchain
+change lands.
+
+## Payload
+
+Eight safetensors shards (12,891,718,440 B total) + tokenizer and config
+files under `/home/dank/models/MiMo-V2.6-Distill-Qwen-9B-INT8-W8A16-AutoRound/`,
+all LFS sha256-verified by `old/download-frank-small-sglang-int8.sh`
+(`--verify-only` re-checks) — the script moved to `old/` when the endpoint
+cut over to llama.cpp + GGUF on 2026-10-01 night, and
+`scripts/download-frank-small.sh` now fetches the GGUF quant instead. The
+serve script refuses to start with a missing shard.
+
+## Client contract
+
+- **Alias:** `frank-small` (the `id` at `/v1/models`).
+- **Reasoning:** extracted to **`reasoning_content`** (live-verified
+  2026-10-01 — this SGLang build's qwen3 parser uses the same field name
+  as llama.cpp; models.yml sets `reasoningContentField:
+  "reasoning_content"`). `--reasoning-parser qwen3`.
+- **Tools:** `--tool-call-parser qwen3_coder`.
+- **Vision:** supported (in-checkpoint tower).
+- **Sampling:** SGLang has NO server-wide `--temperature`/`--top-p`/
+  `--top-k` — sampling rides on the request; the harness sends the Qwen
+  recipe (temp 0.7, top_p 0.8) via models.yml `compat.extraBody`.
+- **Reasoning effort:** the full OpenAI set is accepted (the chat template
+  has no effort gate).
+
+## Measured
+
+- Boot ~60 s with warm JIT caches (several minutes cold — the first boot
+  compiles the sgl-kernel + flashinfer kernels; the CUDA 13 toolchain
+  above is what makes that compile succeed).
+- Decode **83.9 tok/s** (no spec decode — nothing to accept).
+- Prefill ~2,050–3,370 t/s eager (4-slot shape: a 170,019-token prompt in
+  50.5 s, ~3,370 t/s; 2-slot shape: a 200,000-token prompt in 97.3 s wall
+  incl. 8 decode tokens, ~2,056 t/s).
+- 4-slot shape: 4 concurrent 54,027-token requests all answered in ~17 s
+  (216k tokens of live KV at once). 2-slot shape: two concurrent
+  50,000-token requests both answered (28.8 s wall).
+- **Compaction fit (the reslice's purpose):** a 200,000-token prompt —
+  well past frank-primary's 183,501-token compaction trigger (70% of its
+  262,144 window) — prefills and completes in ONE request.
+- Vision: a 64x64 red square answers "Red" with `image_tokens: 64`.
+
+## Rollback
+
+[[frank-small-llama-mimo-q5km]] is the live shape two steps forward; the
+practical rollback today is [[frank-small-llama-mimo-q6k]] (payload at
+`/home/dank/models/MiMo-Q6K/`). This era's own rollback, the llama.cpp
+Gemma4-12B QAT payload, was staged to
+`/home/dank/.AGENTTRASH/20261002-retired-llm-payloads/Gemma4-12B-QAT/` on
+2026-10-02; its unit/env/serve trio remains in `old/`.
+
+# Citations
+
+- Model: https://huggingface.co/letechlead/MiMo-V2.6-Distill-Qwen-9B-INT8-W8A16-AutoRound
+- SGLang: https://github.com/sgl-project/sglang
+- Serve script: `scripts/serve-frank-small.sh`; tunables: `scripts/frank-small.env`

serverAppConfigs-docs/LLM/halogen-flash-server.md

@@ -1,7 +1,7 @@
 ---
 type: reference
-title: "halogen-flash-server on Dankclaw (frank-large)"
+title: halogen-flash-server on Dankclaw (frank-large)
-description: "The RETIRED frank-large runbook (2026-09-16 → 09-19, rollback target) — peonist-ai/halogen-flash-server 0.5.8 serving Qwen3.8-Flash-Next (w4b) on the Dankclaw gfx1151 box: container shape, deployed flags, the memory model, the client contract, and the failure modes worth knowing. Replaced by [[frank-large-ciru-orca]] on 2026-09-19."
+description: The active frank-large runbook — peonist-ai/halogen-flash-server 0.5.8 serving Qwen3.8-Flash-Next (w4b) on the Dankclaw gfx1151 box: container shape, deployed flags, the memory model, the client contract, and the failure modes worth knowing.
 tags:
   - llm
   - halogen
@@ -13,3 +13,466 @@ tags:
   - rocm
 timestamp: 2026-09-13
 ---
+
+# halogen-flash-server on Dankclaw (frank-large)
+
+> **ACTIVE again since 2026-09-16, and retuned.** It replaced the llama.cpp /
+> kingjones30-ROCmFPX STRIX deployment on 2026-09-11 (see
+> [[kingjones-strix-rocmfp4]]), was displaced for two days by ik_llama.cpp (see
+> [[frank-large-ik-uncensored]], the current rollback), and came back as the
+> shape the fleet asked for: two 200k-token conversations rather than the
+> native-context pair. Deployed as a rootless Quadlet container,
+> `dankclaw/frank-large/quadlet/frank-large.container`, port 7770. The RTX 3090
+> it does not touch now belongs to [[frank-small-qwen38-gsq-rco]].
+
+The engine is a from-scratch serving runtime for the Qwen3.8-Flash-Next
+architecture on AMD's gfx1151 (Strix Halo) — the same silicon this host runs.
+Upstream's own comparison table measures it at **roughly 4x the prefill
+throughput** of the llama.cpp-family runtimes it ships against (EngramHalo.cpp,
+ROCmFP4, CIRU-IU4) on a 32K prompt, and prefill is most of the wall clock on any
+prompt with real context. That, not decode, is the reason for the swap.
+
+## Shape
+
+One container runs both halves of the stack:
+
+| half | port | reachable |
+|---|---|---|
+| engine | 8730 | container loopback only — **unauthenticated, never publish it** |
+| OpenAI front-end | 8731 | published as host **7770** |
+
+The `all` mode is upstream's default and the shape a single-container host
+wants: the front-end talks to the engine over loopback inside the container, so
+there is one process tree, one log, and one published port.
+
+`HALOGEN_MODEL_ID=frank-large` sets the id reported at `/v1/models`, in
+`/health`, and in every response body. Requests are **never rejected for naming
+a different model** — the server serves whatever asks. Keeping the fleet alias
+is therefore free, and it means Hermes, the harness config, the OpenWebUI
+discovery, the `services.json` probe and the docs needed no rename.
+
+## Weights
+
+`peonist-ai/halogen-qwen3.8-flash-next` (HuggingFace, public and ungated):
+
+| file | size | role |
+|---|---|---|
+| `qwen38-flash-next-w4b.hgn` | 115.55 GiB | the checkpoint |
+| `qwen38-flash-next-w4b.overlay.hgn` | 2.40 GiB | **quality sidecar** — picked up automatically; the engine warns at startup if it is missing. **Refreshed 2026-09-16**: the sidecars are versioned *in place*, so a stale local copy is invisible from its file name — this one had read 2.31 GiB since it was first fetched while the Hub had moved on |
+| `qwen38-flash-next-vision.hgn` | 0.84 GiB | vision tower (see below) |
+| `tokenizer/` | ~21 MB | flat tokenizer dir; found beside the checkpoint, so no extra mount |
+
+`HALOGEN_DOWNLOAD` fetches the whole repo into the bind mount on **first start
+only** — the entrypoint acts only when the checkpoint is absent, so a restart
+never re-downloads, and `huggingface_hub` resumes an interrupted pull. The
+mount must be read-write for this. Roughly 127 GB; on Dankclaw it lands in
+`/home/dank/models/halogen`.
+
+**Refreshing the weights.** Delete the mount's contents (or the whole
+directory, then recreate it **empty**) and start the unit: the entrypoint sees
+the checkpoint absent and re-fetches the repo, resuming if interrupted. Two
+traps. First, the bind mount's source directory must *exist* — podman refuses
+the container with `Error: statfs /home/dank/models/halogen: no such file or
+directory` and the unit loops on `Restart=on-failure`; `mkdir -p` the empty
+directory and the fetch starts on the next start. Second, "refresh" only means
+"fetch what the Hub has now": on 2026-09-16 the re-fetch brought newer sidecars
+(`overlay.hgn` 2,477,677,120 → **2,572,466,560** bytes, `overlay-speed.hgn`
+2,383,306,048 → **2,478,095,488**) while the checkpoint returned
+byte-identical in size (124,068,083,904) — so an update can live entirely in
+the sidecars, and equal file sizes are not proof of an unchanged repo.
+
+`overlay-speed.hgn` (2.31 GiB on the Hub; 2.22 GiB before the 2026-09-16
+refresh) comes along with the repo but is unused: it trades the `o_proj`
+calibration for about 2 % of decode, and the quality sidecar is the default for
+a reason.
+
+## Deployed flags
+
+Everything else is configured by environment variable — there is no config
+file, and the flags are read once at startup. These are the departures from
+upstream's defaults:
+
+| flag | value | why |
+|---|---|---|
+| `HALOGEN_MODEL_ID` | `frank-large` | fleet identity; no client rename |
+| `HALOGEN_CTX` | `200192` | **set 2026-09-16**: exactly half the pool, so two conversations of the full window are always resident. Not a round 200,000 — pool/2 is. The model's native window is 262,144, so this is a real reduction in the per-conversation ceiling, and a client asking for more gets a 400 (never a silent truncation). The harness declares 200192 for `frank-large` |
+| `HALOGEN_KV_POOL_POSITIONS` | `400384` | **retuned 2026-09-16** (from 401,408): 2 x 200,192. Reserves ~11.0 GiB of KV, confirmed by the boot ledger as `400384 positions, ~27.2 GiB (16.2 GiB of it for everything but the pool, at max_tok 24576)` against a 33.5 GiB device budget — **no clamp**. Slots still oversubscribe the pool for any real request (admission reserves prompt + `max_tokens`), so a near-full pair that is both generating is pool-bound. `HALOGEN_KV_POOL_FIT` guards the boot: read the ledger, not this column |
+| `HALOGEN_KV_SLOTS` | `2` | **lowered from 3 on 2026-09-16** — one slot per half of the pool, and two is the fleet's shape now that Dankclaw serves two endpoints. Each slot is 111.0 MiB of O(1) state, so this hands ~111 MiB back as well |
+| `HALOGEN_MAX_TOK` | `24576` | **halved from the 32768 default on 2026-09-16** for frank-small, its co-tenant on the same host. The arena's variable term goes `8.40 x 2 → 8.40 x 1.5`, handing ~4.2 GiB back at roughly a 4-5 % prefill cost — the same lever at 16,384 costs 8-9 %. The vendor's own notes call 24,576 the largest arena that allocated with headroom to spare, so it stays inside their tested range. 65536 still cannot boot. It sizes a per-call PREFILL CHUNK, not the prompt ceiling: a 200k conversation is prefilled in arena-sized pieces |
+| `HALOGEN_HOST_RESERVE_GIB` | `20` | **raised from 15 with the same retune**, because the arena shrank: `budget = 121.2 − 67.7 − 20 = 33.5 GiB` against a measured need of ~27.2, so the fit check kept ~6 GiB of margin and did not clamp. It is **not** an allocation: raising it cannot return memory to the host, only take the pool away (see the ceiling section below) |
+| `HALOGEN_VISION_TOWER` | *unset* | **removed 2026-09-16.** The tower charges 1.97 GiB against the same device budget as the pool and the arena, and the fleet's image path moved to [[frank-small-qwen38-gsq-rco]] (llama.cpp with `--mmproj`). This server refuses image content parts, and `models.yml` declares `frank-large` text-only to match |
+| `HALOGEN_DOWNLOAD` | `peonist-ai/halogen-qwen3.8-flash-next` | first-start fetch |
+| `HALOGEN_VERBOSE` | `1` | verbose engine logging — the fullest startup account, on permanently so an incident report needs no restart-into-debug |
+
+Everything else is upstream's default, deliberately: prompt cache mode `2`
+(two entries per conversation — end-of-system-prompt and end-of-history — which
+already covers the shared-prefix case), `HALOGEN_CACHE_INPLACE=1` (the snapshot
+stays ~115 MiB at any depth), the speculative drafter `1` (byte-identical at
+temperature 0, and it stands down whenever a second conversation is
+generating), the quality sidecar, `HALOGEN_ADMIT_CHUNK=0` (a non-zero value
+gives up the byte-identity property), and the default `HALOGEN_MAX_TOKENS_CAP=65536`.
+
+## Memory model
+
+Three numbers explain the whole box:
+
+- **~68 GiB pinned weights.** The checkpoint is mapped and registered with the
+  GPU in place, never copied. The kernel counts the locked pages as reclaimable
+  file cache, so `free`, `MemAvailable` and every dashboard built on them
+  **overstate free memory by the size of the model**. The server prints the
+  corrected figure at startup; believe that line, not Grafana.
+- **~28 KiB per KV position** (measured: 262,144 positions = 6,656 MiB of KV,
+  ~7.2 GiB with block scratch), plus the prefill arena. The 2026-09-16 ledger
+  reads the two together in the engine's own words: `400384 positions, ~27.2
+  GiB (16.2 GiB of it for everything but the pool, at max_tok 24576)` — so
+  ~11.0 GiB of pool and 16.2 GiB of arena, working slack and (no longer) a
+  vision tower. The engine's device-side budget stops at roughly 47 GiB on a
+  128 GB machine *with the weights pinned* — and lower on a machine carrying
+  more besides this server, which Dankclaw is.
+- **`HALOGEN_HOST_RESERVE_GIB`** is what is left for the page cache the lookup
+  table is read through. Too little and long prompts crawl with the disk busy
+  and the process in uninterruptible sleep — that is a cache problem, not a
+  memory problem, and the fix is a *smaller* pool, not a bigger one.
+
+`HALOGEN_KV_POOL_FIT=1` (default) measures that budget at startup and lowers
+the pool itself if the configured one will not fit, printing what it chose.
+Read those lines on every restart; they are the ground truth for sizing.
+
+**Measured ledger, seven restarts, read live 2026-09-12.** The engine's own
+startup account, including three deliberate experiments:
+
+| pool | max_tok | reserve | KV | arena | need | budget | verdict |
+|---|---|---|---|---|---|---|---|
+| 262144 (baseline) | 32768 | 24 | 7.2 | 20.6 | 29.8 | 29.5 | boots 0.3 over — prints **no** sizing lines |
+| 524288 | 16384 | 24 | 14.4 | 12.2 | 28.6 | 29.5 | fits, 0.9 spare |
+| 524288 | 32768 | 24 | 14.4 | 20.6 | 37.0 | 29.5 | clamped to 262144, deploy still green |
+| 524288 | 65536 | 24 | — | 37.3 | 46.5 | 29.5 | **OOM — will not boot** |
+| 524288 | 32768 | **15** | 14.4 | 20.6 | 37.0 | **38.5** | **deployed — both features** |
+| 524288 | 32768 | 16 | 14.4 | 20.6 | 37.0 | 37.5 | fits, footprint unchanged — see the ceiling below |
+| **400384** | **24576** | **20** | **11.0** | **16.4** | **27.2** | **33.5** | **deployed 2026-09-16 — ledger need 27.2, no clamp** |
+| 401408 | 32768 | 15 | 11.0 | 20.6 | 33.4 | 38.5 | 2026-09-13 → 09-16, superseded by the retune |
+
+### The sizing rule
+
+One budget covers the pool, the arena **and** the vision tower together:
+
+```
+device budget = host RAM - resident weights - HALOGEN_HOST_RESERVE_GIB
+              = 121.2    - 67.7             - reserve
+                                    reserve 24 -> 29.5 GiB
+                                    reserve 15 -> 38.5 GiB
+
+need = KV (28 KiB/position) + arena + vision (1.97 measured)
+arena = 3.80 GiB + 8.40 GiB x (HALOGEN_MAX_TOK / 16384)
+```
+
+The `121` the engine prints is rounded **down** from `MemTotal 127104060 kB =
+121.2159 GiB`, so subtracting the displayed figures gives 29.3 rather than 29.5.
+Use the unrounded value. Two of the three terms are fixed hardware and
+checkpoint facts; `HALOGEN_HOST_RESERVE_GIB` is the only one you own — which
+buys the *fit margin*, not the host's memory. It is a term in this inequality,
+not an allocation, and the section at the end of this page shows the live test
+that proves it.
+
+Trust the engine's printed requirement over this model wherever they disagree:
+a boot clamped at reserve 24 stated `524288 positions need ~35.0 GiB (plus 1.5
+of margin)`, i.e. **budget >= 36.5**, against a modelled 37.0 here. Close
+enough to plan with, and the authoritative number to cut the other way.
+
+The arena term is **linear in `HALOGEN_MAX_TOK`** and predicts every measured
+point to 0.1 GiB — including upstream's three published "on the device" figures
+(524,288 → ~35, 786,432 → ~42, 1,048,576 at `max_tok=16384` → ~41), which are
+**pool + arena totals at the default `max_tok`**, not pool alone. That is why
+their largest pool "needs `HALOGEN_MAX_TOK=16384`": at 32768 a 1M pool would be
+49.4. Reading those figures as pool-only is the trap that makes a doubling look
+affordable when it is not.
+
+At the baseline reserve of 24 the doubled pool leaves `29.5 - 1.97 - 14.4 =
+13.1` for the arena, so `max_tok <= 18,130` and **16384 is that reserve's
+ceiling, not a conservative setting.** The corner check agrees: at 32768 only
+7.0 GiB is left for the KV, ~254k positions, and the pool drops back toward
+262144. Cut the reserve to 15 and the same arithmetic gives `38.5 - 1.97 - 14.4
+= 22.1` for the arena — `max_tok <= 35,700`, which is exactly why 32768 now
+fits unmodified.
+
+### What the reserve actually costs
+
+The device budget is not the whole story. Memory position by configuration
+(401,408 is the 2026-09-13 deployed row, read from its boot ledger):
+
+| config | held | host left | contiguous 2 MiB blocks |
+|---|---|---|---|
+| 262144 / 32768, reserve 24 | 96.3 | 12.1 GiB | 2081 |
+| 524288 / 16384, reserve 24 | 94.6 | 13.8 GiB | 3742 |
+| 524288 / 32768, reserve **16** | **103.5** | **7.1 GiB** | 293 |
+| **401408 / 32768, reserve 15** | **100.0** | **9.1 GiB** | **1396** |
+| **400384 / 24576, reserve 20** | **95.4** | **14.7 GiB** | **475** |
+
+At the doubled setting (2026-09-12) the server held 103.5 of 121 GiB and the
+engine volunteered the stall note. It did not print it at the 401,408 trim, and
+it prints it again after the 2026-09-16 retune. Three data points, and they do
+not line up with a "free memory below X" reading: 310 blocks printed, 1396 did
+not, 475 did. So treat the note as "the machine is tight" and read the
+`host memory left` line rather than waiting to be told:
+
+```
+halogen: NOTE this server now holds most of the machine, and 310 contiguous
+2 MiB blocks remain. It runs best on a host of its own. If other large
+processes share this machine they will compete for what is left, and when it
+runs out BOTH they and this server can stall for minutes at a time at full
+CPU with no disk activity and no output, which is not a crash and needs no
+restart.
+```
+
+That is the failure mode to watch, and it is the least diagnosable of the
+three: not a silent clamp, not a boot refusal, but a minutes-long freeze that
+looks like a hang, for the neighbours as much as for frank-large. `free` is
+useless for spotting it — `MemAvailable` still reads ~74 GiB because the
+page cache counts as reclaimable. Watch the `host memory left for everything
+else` line and the contiguous-block count at boot instead.
+
+28 co-tenant containers (~4.3 GiB working set) stayed up through the change,
+so it is survivable.
+
+### The reserve cannot buy air — ceiling 16
+
+The last row above is the experiment that proves it. Raising the reserve from
+15 to 16 left `held` and `host left` **byte-for-byte identical** (103.5 / 7.1
+GiB). `HALOGEN_HOST_RESERVE_GIB` feeds the fit-check budget and is *not* an
+allocation; the actual footprint is `weights + KV + arena + vision`, none of
+which the reserve touches. So the knob has exactly two regimes:
+
+| reserve | budget | vs required 36.5 | effect |
+|---|---|---|---|
+| 15 | 38.5 | +2.0 | deployed |
+| 16 | 37.5 | +1.0 | no change to footprint, thinner margin — pointless |
+| 17 | 36.5 | +0.02 | knife edge, inside the noise of the 67.7 term |
+| **18** | **35.5** | **−1.0** | **clamps the pool to 262144** |
+| 24 | 29.5 | −7.0 | clamps the pool to 262144 |
+
+The 36.5 requirement is the engine's own, read from a clamped boot:
+`524288 positions need ~35.0 GiB (plus 1.5 of margin)`. Above 16 the only
+thing a bigger reserve can do is take the doubling away.
+
+**To actually free host memory, change the pool or the arena**, and the reserve
+then becomes free to rise: `HALOGEN_KV_POOL_POSITIONS=262144` hands back
+7.2 GiB from the doubled pool, `HALOGEN_MAX_TOK=16384` hands back 8.4. Both
+are one-line edits that the fit check will confirm on the next boot. The
+2026-09-13 trim to 401,408 was this remedy applied at half strength —
+`HALOGEN_MAX_TOK` untouched, and on the host-left metric it bought +2.0 GiB
+with a 4.5x improvement in contiguous blocks (310 → 1396), less than the
+modelled 3.4 because the engine holds the freed KV as working slack.
+
+### The `WARNING` line only appears when the fit check had to move
+
+This one cost me a wrong conclusion, so it is worth stating plainly. When the
+configured pool already fits, the engine reserves it **and prints nothing** —
+the 09-11 baseline boot goes straight from the pre-flight line to
+`checkpoint: mapped` with no `kv pool:` sizing block at all. The sizing block and
+its `WARNING even 262144 positions ... host RAM cannot spare it` appear **only
+after a clamp**.
+
+So the absence of a warning in old logs is not evidence of headroom. Both
+`262144/32768` rows above need 29.8 against a 29.5 budget — 0.27 over — and the
+baseline has lived in that state since it was deployed without incident,
+because the 24 GiB reserve is itself generous (the host reports 12–14 GiB free
+at that point). The engine's budget is a conservative floor, not a hard wall.
+
+### Two failure modes, and only one is dangerous
+
+Raising `max_tok` past the ceiling does not shrink the pool — the arena alone
+outgrows the budget, and the server cannot allocate at all:
+
+```
+kv pool: 524288 positions need ~51.7 GiB ... LOWERING THE POOL TO 262144 (~44.5 GiB).
+kv pool: WARNING even 262144 positions (the context, the smallest pool that can
+         serve one request) needs ~44.5 GiB and host RAM cannot spare it.
+dmalloc: FAILED requesting 0.250 GiB after 39.970 GiB in 566 allocations (out of memory)
+```
+
+This is the quiet one: the unit stays `active`, the container reports
+`Up (starting)`, and `deploy.sh` sits in its health wait for the whole
+`LLM_WAIT` rather than failing fast. It confirms upstream's warning that a
+large `HALOGEN_MAX_TOK` means "the server will not start", not "it will be
+slower".
+
+Under-running the ceiling is far more benign and far easier to miss. At 524288
+with the arena back at 32768, `HALOGEN_KV_POOL_FIT` lowered the pool and
+`deploy.sh` still reported healthy in 12 seconds:
+
+```
+kv pool: 524288 positions need ~35.0 GiB (plus 1.5 of margin) and host
+         RAM cannot spare it. LOWERING THE POOL TO 262144 (~27.8 GiB).
+```
+
+A green health check does not mean the pool you asked for is the pool you got —
+read the `kv pool:` lines after every restart.
+
+The 47.7 GiB n-gram lookup table is never held in RAM; it is read through
+the file cache. The 2.0 GiB the firmware carves for the iGPU appears in no
+`/proc/meminfo` figure at all.
+
+**Co-tenancy.** `HALOGEN_FLASH_PIN_TRUNK=1` (default) is what the published
+decode numbers were measured with, and it means this container holds most of
+the host. If a co-tenant then takes the remainder, both it and the engine can
+stall for minutes at a time at full CPU with no disk activity and no output.
+That is documented behaviour, not a crash, and it needs no restart.
+`HALOGEN_FLASH_PIN_TRUNK=0` gives the memory back and costs several times the
+decode speed — a last resort, not a tuning option.
+
+## Client contract
+
+The differences from llama.cpp that bite are all in the request, not the
+deployment:
+
+- **No server-side sampler defaults.** `temperature` absent or `0` is
+  **greedy** decode; above 0 the request samples on the same drafter. Supported
+  fields: `temperature`, `top_p`, `top_k`, `min_p`, `seed`,
+  `presence_penalty`, `frequency_penalty`, `logit_bias`, `logprobs`. Refused
+  with a 400: `top_logprobs`, `logprobs` with `stream: true`, `n > 1`, and any
+  value outside its defined range (values are refused, never clamped).
+  The fleet runs greedy for clients that stay silent — deterministic, and
+  byte-identical with the drafter.
+- **The token budget covers thinking.** The default is **8192**, and a budget
+  that runs out mid-thought returns `finish_reason: "length"` with an **empty
+  `content`** and the partial reasoning in `reasoning_content` — which most
+  clients do not display. Above the cap (`HALOGEN_MAX_TOKENS_CAP`, 65536) a
+  request is a 400 rather than a silent truncation. `max_tokens`,
+  `max_completion_tokens` and `max_output_tokens` all work and must agree if
+  more than one is sent.
+- **A budget above (window − compaction-threshold) is a guaranteed 400 zone.**
+  The fit check (`prompt + max_tokens <= HALOGEN_CTX`) refuses, and the harness
+  only compacts at its own threshold — with OMP at 70 % of a 200,192 window
+  (140,134), any prompt in (136,192, 140,134] with a 64,000 budget overflows
+  the window while sitting below the compaction trigger. That collided live on
+  2026-09-17 (prompt 138,898 + 64,000 = 202,898 > 200,192). The fix is
+  harness-side: keep the declared `maxTokens` at or under
+  `window − threshold` (57,344 chosen) so no pre-compaction request can miss
+  the fit check. Do not fix it by raising `HALOGEN_CTX` past pool/2 — that
+  breaks the two-resident-conversations property.
+- **`reasoning_effort`** is accepted at the top level and inside
+  `chat_template_kwargs`: `minimal`, `low`, `medium`, `high`, `xhigh`. The
+  model's default is `xhigh`. `minimal` aliases the template's `low` and still
+  reasons; `enable_thinking: false` turns thinking off; `preserve_thinking` is
+  honoured. `preserve_reasoning` is a **400** — it is the llama.cpp-era key.
+- `/v1/models`, `/health` and completions all report the served id, and
+  `reasoning_content` carries the reasoning trace.
+
+**Pool admission is the one arithmetic trap.** A request reserves its prompt
+**plus** `max_tokens` positions when it is admitted; if the pool cannot hold it
+the request waits in arrival order, and after the queue timeout (3600 s) it
+gets a `503 engine_busy`. A prompt large enough that prompt + budget exceeds
+the pool can therefore never be admitted at all. The 2026-09-16 shape is a
+400,384-position pool with `HALOGEN_CTX` 200,192 and two slots: two
+conversations at the ceiling fit the pool exactly, so a pair that is both
+near-full and both generating is pool-bound — the second waits for the first to
+release its region. The pool binds **how many** conversations stay resident,
+not how long one may be; the older 524,288 / 401,408 numbers live in the ledger
+tables above.
+
+## Vision
+
+**None here since 2026-09-16.** `HALOGEN_VISION_TOWER` is deliberately unset:
+the tower is a 1.97 GiB charge against the same device budget as the pool and
+the arena, and the fleet's image path moved to [[frank-small-qwen38-gsq-rco]],
+which loads the BF16 projector alongside its own weights. This server refuses
+image content parts and says why, rather than ignoring them:
+
+```json
+{"error": {"message": "this server does not accept images: the engine was
+started without a vision tower (HALOGEN_VISION_TOWER)", "type":
+"invalid_request_error"}}
+```
+
+The llama.cpp side handles images the same way this engine used to: a `data:`
+URL or bare base64, about 2,040 tokens of context per image, and `http(s)` URLs
+refused deliberately (fetching one would make the server issue outbound
+requests to wherever a client pointed it). Audio and video have no path
+anywhere in the fleet — that left with [[frank-npu]].
+
+## Deploy and operate
+
+```bash
+./scripts/deploy.sh dankclaw --service frank-large
+```
+
+The handler retires the ik-era **root** unit first (stop, disable, move
+`/etc/systemd/system/frank-large.service` into
+`~/.AGENTTRASH/20260916-halogen-restore/host-units` — never `rm`; this host's
+`disable` may delete the file outright, which the handler now reports), stops
+the Quadlet's own unit if it is running, then pulls
+`ghcr.io/peonist-ai/halogen-flash-server:0.5.8`, re-tags it to the
+`localhost/` alias the unit references, and **waits for port 7770 to be free**
+before installing and starting anything.
+
+That last gate is not paranoia. On the first cutover attempt the ik server
+logged "Deactivated successfully", and pasta — rootless podman's userspace
+network stack, which binds the published port at container CREATE — still
+reported `Address already in use` a second later, exiting the unit with code
+126. Only `Restart=on-failure` recovered it, minutes later. A deploy that
+starts a doomed container has not deployed anything, so `wait_port_free` polls
+every socket on the port (listener, accepted, TIME_WAIT) and fails loudly with
+the socket table if it never clears.
+
+Boot persistence comes from `[Install] WantedBy=default.target` in the unit
+file plus linger on the deploy user — generated units **cannot** be
+`systemctl --user enable`d on this host.
+
+```bash
+systemctl --user status frank-large.service     # the generated unit
+podman logs frank-large | head -40              # version banner + memory ledger
+curl -s http://127.0.0.1:7770/health            # 200 only after the engine PONGs
+curl -s http://127.0.0.1:7770/v1/models
+```
+
+`/health` asks the **engine** for a PONG before answering, so a 200 means
+"ready to serve", not merely "process alive". During the first-start download
+it stays unhealthy by design.
+
+## Troubleshooting
+
+| symptom | cause |
+|---|---|
+| `dmalloc: FAILED requesting … (out of memory)` then `HIP … out of memory` | the KV pool did not fit. `HALOGEN_KV_SLOTS` will not fix it (slots share the pool); lower `HALOGEN_KV_POOL_POSITIONS` to 262144, then `HALOGEN_MAX_TOK` to 16384 (~9 % prefill) |
+| starts, answers short prompts, collapses to a few tok/s on a long one, disk busy, process in uninterruptible sleep | short of **file cache**, not memory — lower the pool and/or raise `HALOGEN_HOST_RESERVE_GIB` |
+| long replies that look cut off | read `finish_reason`: `stop` means you have the whole answer, `length` means you ran out of budget (thinking counts) |
+| a request that never returns, then `503 engine_busy` | the pool could not reserve prompt + `max_tokens`; see the admission trap above |
+| both the engine and a co-tenant stall for minutes at full CPU | the pinned-trunk trade; see Memory model |
+| `free`, `MemAvailable`, Grafana all wrong by ~68 GiB | expected — locked weights are counted as reclaimable |
+
+`HALOGEN_VERBOSE=1` ships in the unit, so the fullest startup account is
+already in the log when something goes wrong. `HALOGEN_DMALLOC_LOG` is
+deliberately **not** set: it logs every allocation over 64 MB, which is
+steady-state noise on a shared host — raise it only for a live OOM report.
+The lines worth grepping first:
+
+```bash
+podman logs frank-large 2>&1 | grep -E '^(dmalloc|kv pool):'
+```
+
+Retrieval: `podman logs frank-large` covers the **current** instance only —
+the Quadlet runs `--replace --rm`, so everything before the last recreate is
+gone from the container log. `journalctl --user -u frank-large.service` is
+the durable record. Both are rootless: `sudo podman logs frank-large` looks
+in root's storage and answers "no such container".
+
+## Rollback
+
+The predecessor is intact: the ik payload stays at
+`/home/dank/models/mradermacher-Qwen3.8-Flash-Next-Uncensored-i1-Q4_K_M`, the
+ik build at `/home/dank/GitPackages/ik_llama.cpp/build`, and its unit, env,
+serve script and the Qwen-Sharp template are archived in the repo under
+`scripts/.AGENTTRASH/*.ik-llama.cpp` and `*.ik-llama-qwen-sharp`. Rolling back
+is a `git revert` of this change set plus a deploy — **but the Quadlet must be
+removed from `~/.config/containers/systemd/` first**, or the restarted root
+unit and the still-installed container both claim port 7770. The port gate in
+the handler now covers exactly that race. See [[frank-large-ik-uncensored]].
+
+The `amd_iommu=off` kernel argument that came with this change is separate and
+reverts on its own — see [[dankclaw-iommu-translation-off]].
+
+## Citations
+
+- [peonist-ai/halogen-flash-server](https://github.com/peonist-ai/halogen-flash-server) — README, `docs/FLAGS.md`, `deploy/entrypoint.sh`, `docker-compose.yml`, CHANGELOG (v0.5.8)
+- [peonist-ai/halogen-qwen3.8-flash-next](https://huggingface.co/peonist-ai/halogen-qwen3.8-flash-next) — weights and file inventory
+- [Qwen/Qwen3.8-Flash-Next](https://huggingface.co/Qwen/Qwen3.8-Flash-Next) — base model

serverAppConfigs-docs/LLM/index.md

@@ -1,14 +1,21 @@
 # LLM Fleet — Knowledge Base
 
-Documentation for the LLM inference fleet on the two hosts (2026-09-27):
+Documentation for the LLM inference fleet on the two hosts (2026-10-02):
-**Dankserver runs R9V radiance-vLLM on frank-primary** (the drwolfen fork
+**Dankserver runs the tcclaviger RDNA4 vLLM fork on frank-primary** (the
-serving the unsloth Qwen3.8-Flash-Next Q8_0 + MTP4 across both Radeon AI
+digest-pinned `tcclaviger/vllm` image driving `podman run` directly, serving
-PRO R9700s on :7770, docker-pinned runtime driven through a docker→podman
+orcarouter/Qwen3.8-Flash-Next-Uncensored-FP8 across both Radeon AI PRO R9700s
-shim, the Qwen-Sharp template, 131072 ctx), and Dankclaw runs two llama.cpp endpoints —
+on :7770, TP=2 with automatic tiered-expert offload, **4 seqs x 262144 ctx
-stock llama.cpp (unsloth b11030, ROCm gfx1151; frank-large, unsloth
+(the model's full native window)**, fp8 KV (5 boot but leave no runtime
-Qwen3.8-27B UD-Q8_K_XL + the z-lab DFlash2 Q4_K_M drafter on the gfx1151
+headroom; 6 do not fit), no speculative decode, MNBT 1024, MoE gather 32x64, util 0.98,
-iGPU, port 7770) and mainline llama.cpp (frank-small, HauhauCS Gemma4-12B
+reasoning/tool parsers wired (qwen3 / qwen3_xml) with preserve_thinking on,
-QAT Q4_K_M with MTP and vision on the RTX 3090, port 7771).
+measured 36.4-39.4 tok/s decode — the 2026-10-01 bf16 no-spec cutover plus
+the night ladder; rollback frank-primary-fp8-mtp-20261001), and Dankclaw
+runs two endpoints — **ROCmFPX llama.cpp (frank-large, port 7770:
+kingjones777/Qwen3.8-27B-ROCmFP4-STRIX-MTP-GGUF Q8_0_ROCMFPX_AGENT, a ftype-115
+file only that fork can read, with the repo's MTP head drafting 4 tokens, ONE
+session at 262,144, f16 KV, vision via the bundled projector)** and
+**llama.cpp CUDA (frank-small, port 7771: MiMo-V2.6-Distill-Qwen-9B Q5_K_M
+GGUF, 2 x 262,144, asymmetric KV f16 K + q8_0 V, on the RTX 3090)**.
 
 > 2026-08-22: the former `vllmScripts` repository is merged into the
 > serverAppConfigs repository. This knowledge base moved from the old repo's
@@ -22,24 +29,24 @@ graph LR
     subgraph Dankserver[AMD Dankserver - Ubuntu 26.04]
         A[2x Radeon AI PRO R9700
 gfx1201, 32 GiB each]
-        B2[R9V radiance-vLLM fork, docker-pinned
+        B2[tcclaviger RDNA4 vLLM fork, digest-pinned
-via docker-podman shim
+podman run, frank-primary, port 7770]
-frank-primary, port 7770]
+        C[Qwen3.8-Flash-Next FP8 bf16 no-spec
-        C[Qwen3.8-Flash-Next Q8_0
+TP=2, expert offload automatic, 4 x 262144 ctx,
-TP=2 tiered experts, MTP4, sharp template, 131072 ctx, vision]
+fp8 KV, MNBT 1024, gather 32x64, qwen3 parsers]
     end
 
     subgraph Dankclaw[AMD Strix Halo - Ubuntu 26.04]
         E2[RTX 3090 (PCIe x4 dock) +
 gfx1151 iGPU 114 GiB GTT]
-        E[stock llama.cpp (unsloth b11030, gfx1151)
+        E[ROCmFPX llama.cpp gfx1151, pinned fork build
 frank-large, port 7770]
-        F[Qwen3.8-27B UD-Q8_K_XL + DFlash2 Q4_K_M drafter
+        F[Qwen3.8-27B Q8_0_ROCMFPX_AGENT + MTP-Q4_0 head
-262144 ctx / 1 slot, DFlash2 depth 4, q8_0 KV]
+1 x 262144 ctx, f16 KV, draft n-max 4, mmproj vision]
-        G[mainline llama.cpp 37b53fd
+        G[llama.cpp master CUDA sm_86
 frank-small, port 7771]
-        H[Gemma4-12B QAT Q4_K_M
+        H[MiMo-V2.6 Q5_K_M GGUF
-4 x 262144 ctx, MTP + vision + audio]
+2 x 262144 ctx, f16 K + q8_0 V, flash-attn]
         E2 --> E --> F
         E2 --> G --> H
     end
@@ -58,20 +65,42 @@ frank-small, port 7771]
 
 ## Active Deployment
 
-- [[frank-primary-r9v-radiance-qwen38-q8]] — the ACTIVE frank-primary
+- [[frank-large-rocmfpx-q8agent-mtp]] — **ACTIVE since 2026-10-02**
-  runbook (2026-09-27, root systemd unit, port 7770): the R9V
+  (root systemd unit, port 7770): the pinned kingjones30/ROCmFPX fork serving
-  radiance-vLLM fork (drwolfen/radiance-vllm-qwen4exp, docker-pinned
+  the repo's highest-quality rung `Qwen3.8-27B-Q8_0_ROCMFPX_AGENT.gguf`
-  image bundle `2dac17a2`, driven through a docker→podman shim) serving
+  (26.28 GiB, ftype 115 = 187 plain-Q8_0 + 311 GGML_TYPE_Q8_0_ROCMFPX
-  the unsloth Qwen3.8-Flash-Next-GGUF Q8_0 (175.30 GiB, 6 shards) +
+  tensors) with the bundled `mtp-Qwen3.8-27B-Q4_0.gguf` head drafting 4
-  blockfp8 MTP4 (depth 4) + vision on BOTH R9700s — TP=2 tiered experts
+  tokens, one 262,144 window, f16 KV, the BF16 projector for vision. Why the
-  (80-slot LRU cache), PLE n-gram PINNED in host RAM (54.4 GB), CED off,
+  runtime is not interchangeable, the pinned-build set_rows history, the
-  131072 ctx, the Qwen-Sharp v22.4.0 template via the launcher's
+  measured envelope, rollback to gufo
-  `R9V_CHAT_TEMPLATE_FILE` passthrough (fork `30d4a32`), the quantized
+- [[frank-small-llama-mimo-q5km]] — **ACTIVE since 2026-10-02** (root
-  inject-shard dev-overlay adapter (fork `363d606`), reasoning channel
+  systemd unit, port 7771): llama.cpp master (CUDA sm_86) serving
-  `reasoning`, parser `qwen3_coder`. The tuned shape lives in the
+  bartowski/MiMo-V2.6-Distill-Qwen-9B-GGUF Q5_K_M on the RTX 3090,
-  on-host q8xl setup state. Rollback:
+  THREE slots of 180,224 tokens (pool 540,672) with K f16 + V q8_0 and the
-  [[frank-primary-r9v-radiance-qwen38-uncensored]]
+  mmproj for vision, ~100 tok/s decode, ~2.7 GiB of the card left free.
-- [[frank-primary-r9v-radiance-qwen38-uncensored]] — RETIRED 2026-09-27
+  The KV arithmetic, the flash-attn dependency, rollback to Q6_K
+- [[frank-primary-glm53-flash-w4a16]] — RETIRED 2026-09-30 → 10-01 (the
+- failed GLM experiment; frank-primary runs Qwen FP8 again, see the intro):
+  (2026-09-30, root systemd unit, port 7770): the tcclaviger RDNA4 vLLM fork
+  (2026-09-30, root systemd unit, port 7770): the tcclaviger RDNA4 vLLM fork
+  (digest-pinned `tcclaviger/vllm@sha256:da4029666…`, plain `podman run`,
+  no shim) serving Intel/GLM-5.3-Flash-W4A16-AutoRound — AutoRound symmetric
+  int4 group 128, ~170 GiB, 34 shards + 2 extras, 113,074 tensors — on BOTH
+  R9700s, TP=2, with the MoE experts offloaded to pinned host RAM (12,096
+  experts/rank, 73.64 GiB) behind a 14.51 GiB / 2,383-slot VRAM LRU arena; the
+  MoE kernel is libr4d's grouped W4A16 GEMM (`R4dW4a16MoEMethod`, reached
+  because the checkpoint's `quant_method: auto-round` is overridden to `inc` —
+  a compressed-tensors W4A16 checkpoint of the same model has NO offload recipe
+  and refuses to serve). 131,072 ctx x 2 seqs, bf16 KV, util 0.98, prefix
+  caching, vision and `glm47` tool/reasoning parsers, MTP OFF and prefetch OFF
+  by measurement. **Measured decode 9.9-10.0 tok/s, prefill 918 tok/s — the
+  35 tok/s target is NOT met, bounded by the fork's expert-gather read path
+  (~8.2 GB/s effective vs a 22.74 GB/s link) at a 0.53-0.66 cache hit rate.**
+  Rollback: commit `2339fae` (the qwen FP8 + MTP4 shape of 2026-09-29 → 09-30)
+- [[frank-primary-r9v-radiance-qwen38-q8]] — RETIRED 2026-09-30 (two swaps
+  back; payload and R9V state dirs intact on disk, git tag
+  frank-primary-r9v-q8-20260929 is the last pre-cutover commit):
+ [[frank-primary-r9v-radiance-qwen38-uncensored]] — RETIRED 2026-09-27
   (same-day swap to the Q8_0 payload; rollback target, state + payload
   intact on disk): the R9V radiance-vLLM fork serving
   Dyluhn/Qwen3.8-Flash-Next-Uncensored-R9V-IQ4_XS (92.39 GiB, 3 shards) +
@@ -94,6 +123,23 @@ frank-small, port 7771]
   (localhost/lucebox-rocm:gfx1201) serving
   Lucebox/DeepSeek-V4.1-Flash-ROCMFP23 (377.2 GiB) with streamed experts —
   the fleet's only non-llama.cpp runtime detour
+- [[dankserver-frontier-moe-feasibility]] — VERIFIED 2026-09-28: which of
+  DeepSeek-V4.1-Flash (552B), MiMo-V2.6-Flash-MOPD (309B) and GLM-5.3-Flash
+  (320B) can run on the 2x R9700 box, on what runtime, and why V4.1 does not
+  fit (475 GiB payload, 189 GiB of it Engram; decode is PCIe-expert-stream
+  bound, ~23 tok/s ceiling at today's residency). Includes the Affinity
+  V4-Flash-to-V4.1 port assessment — reusable expert tiering, four
+  architecture gaps, one of them a silent-wrongness trap on the Engram key
+- [[dankserver-qwen38-fp8-expert-offload-tuning]] — VERIFIED 2026-09-29:
+  the flag-level decode-tuning campaign on the ad hoc qwen-fp8 container
+  (tcclaviger RDNA4 fork, orcarouter/Qwen3.8-Flash-Next-Uncensored-FP8,
+  TP=2 across both R9700s). **>35 tok/s MET: 37.44-40.57 tok/s with
+  `--expert-prefetch` DELETED** — telemetry measured the prefetcher at
+  36% precision, wasting ~half the gather traffic (the LRU itself runs at
+  82.45% hit). Prefetch-on incumbent 26.8 survived every other challenger
+  (geometry, spec depth, TunableOp-corrupts, batch knobs, concurrency,
+  ngram-fork-bug); the residual gap to ~60 tok/s is the fork's gather
+  read path (6.5-9.5 GB/s effective vs 22.7 at the same slab size)
 - [[frank-medium-radiance-mxfp4]] — RETIRED 2026-09-26 (replaced by
   [[frank-primary-lucebox-ds4]], then
   [[frank-primary-llamacpp-qwen38-flash-next-q8]]): vllm-radiance 0.9.3
@@ -132,10 +178,28 @@ frank-small, port 7771]
   (bs1/bs3/bs4/bs6, 8k–700k KV) on Dankserver RTX 3090
 - [[deployment]] — deploy and update process for both servers
 - [[frank-redo-2026-08-17]] — the 2026-08-17 topology change
-- [[frank-large-unsloth-qwen38-27b-q8xl]] — ACTIVE frank-large runbook
+- [[frank-large-gufo-qwen38-27b]] — **RETIRED 2026-10-02 (the rollback
-  since 2026-09-21 (root systemd unit, port 7770): stock llama.cpp
+  target)** (root systemd unit, port 7770): gufo
+  (gufo-org/gufo) v0.1.1, the Strix Halo inference engine, built from
+  source on Dankclaw, serving unsloth/Qwen3.8-27B-GGUF **UD-Q4_K_XL**
+  with the z-lab DFlash2 Q4_K_M drafter — ONE session at the full native
+  262,144 window, unquantized f16 KV. Rolled back by Frank after one day
+  of [[frank-large-vllm-mimo-v26-bf16]]. The two from-source build
+  workarounds, the client-contract delta, the measured prefill/decode
+  trade against llama.cpp
+- [[frank-large-vllm-mimo-v26-bf16]] — RETIRED 2026-10-01 (one day;
+  rollback by Frank after the autoresearch loop): the digest-pinned
+  kyuz0/vllm-therock-gfx1151 image serving
+  XiaomiMiMo/MiMo-V2.6-Distill-Qwen-9B in FULL BF16, 4 x 262144 ctx,
+  CROSS-MODEL MTP from Qwen/Qwen3.5-9B — kept for the measured record:
+  battery decode 6.19 t/s (MTP-2) vs 9.01 t/s (no draft — spec NET-
+  NEGATIVE on real chat+reasoning traffic), the roof math, the greedy-text
+  numerics trap, and the death of every ≥6-bit vLLM quant path on gfx1151
+  (quark int8 rejected, FP8 kernels CDNA-only, torchao unwired)
+- [[frank-large-unsloth-qwen38-27b-q8xl]] — RETIRED 2026-09-28 (replaced by
+  [[frank-large-gufo-qwen38-27b]]; rollback target): stock llama.cpp
   (unsloth b11030, ROCm gfx1151) serving
-  unsloth/Qwen3.8-27B-GGUF:UD-Q8_K_XL (27B dense) with the z-lab DFlash2
+  unsloth/Qwen3.8-27B-GGUF:UD-Q8_K_XL with the z-lab DFlash2
   Q4_K_M drafter (depth 4) on the gfx1151 iGPU — one full 262,144-token
   conversation, q8_0 KV (the build lacks fp8), text-only, Qwen-Sharp
   v22.4.0 template, the 114 GiB GTT memory model, client contract,
@@ -165,12 +229,27 @@ frank-small, port 7771]
   .hgn + quality overlay) on the gfx1151 iGPU — two 200k conversations over one
   400,384-position pool, deployed flags, memory model, client contract,
   troubleshooting (2026-09-16 → 09-19)
-- [[frank-small-gemma4-12b]] — ACTIVE frank-small runbook since 2026-09-17
+- [[frank-small-llama-mimo-q6k]] — **RETIRED 2026-10-01 → 10-02 (rollback
-  (root systemd unit, port 7771): mainline llama.cpp serving
+  target)**: llama.cpp (master, CUDA sm_86 build) serving
+  bartowski/MiMo-V2.6-Distill-Qwen-9B-GGUF **Q6_K** on the RTX 3090 — TWO
+  native 262,144-token sessions with ASYMMETRIC KV (K f16 unquantized + V
+  q8_0, which forces flash-attn), ~20.9/24.6 GiB measured, text-only (the
+  GGUF has no vision), reasoning_content, no spec decode; payload still at
+  `/home/dank/models/MiMo-Q6K/`
+- [[frank-small-sglang-mimo-v26]] — RETIRED 2026-10-01 (hours; replaced
+  by [[frank-small-llama-mimo-q6k]] for the asymmetric KV SGLang cannot
+  express; venv still on disk, INT8 payload staged to
+  `~dank/.AGENTTRASH/20261002-retired-llm-payloads/`): SGLang in a
+  self-contained python3.12 venv serving leteschlead MiMo-V2.6 INT8
+  W8A16 AutoRound, 2 x 262144 ctx, fp8 KV, in-checkpoint vision,
+  83.9 tok/s decode
+- [[frank-small-gemma4-12b]] — RETIRED 2026-09-17 → 2026-10-01 (replaced by
+  [[frank-small-sglang-mimo-v26]] → the llama.cpp MiMo shape; payload staged
+  to `~dank/.AGENTTRASH/20261002-retired-llm-payloads/Gemma4-12B-QAT/`):
+  mainline llama.cpp serving
   HauhauCS/Gemma4-12B-QAT-Uncensored-HauhauCS-Balanced Q4_K_M on the RTX
   3090, with the separate MTP assistant sidecar chained behind ngram-mod,
-  q8_0 KV, and `--fit` sizing the full 262k window into ~12 GiB. A same-day
+  q8_0 KV, and `--fit` sizing the full 262k window into ~12 GiB.
-  vLLM detour (2026-09-19) was reverted the same day; see below.
 - [[frank-small-vllm-gemma4]] — RETIRED 2026-09-19 (same-day revert,
   rollback target): vLLM serving google/gemma-4-12B-it-qat-w4a16-ct (w4a16
   Marlin) with the bf16 MTP assistant on the RTX 3090 — bf16 KV, 289,647-

serverAppConfigs-docs/index.md

@@ -146,9 +146,41 @@ The LLM inference fleet on the two hosts. Full index:
   computed in host RAM, the placement law, the cutover, and the rollback
 - [[frank-primary-lucebox-ds4]] — RETIRED 2026-09-26 (same-day swap;
   rollback target): the Lucebox runtime serving
-  DeepSeek-V4.1-Flash-ROCMFP23 (377.2 GiB GGUF, 43-layer MoE, 256 routed
+  DeepSeek-V4.1-Flash-ROCMFP23 (377.2 GiB GGUF, **40**-layer MoE, **384**
-  experts top-6) on Dankserver's R9700 with the routed experts computed
+  routed experts top-6, 189 GiB of embedded Engram) on Dankserver's R9700
-  on CPU from the mmapped page cache — held :7770 for hours
+  with the routed experts streamed from the mmapped page cache — held :7770
+  for hours
+- [[dankserver-frontier-moe-feasibility]] — VERIFIED 2026-09-28: can
+  DeepSeek-V4.1-Flash (552B), MiMo-V2.6-Flash-MOPD (309B) or GLM-5.3-Flash
+  (320B) run fast on the 2x R9700 box, and can Affinity / vllm-radiance /
+  FreeToken-AMD be modified for them? The capacity arithmetic (475 GiB
+  payload, 189 GiB of it Engram, ~2.4 GB of cold expert weights per token
+  across PCIe = a ~23 tok/s ceiling) plus the Affinity V4-Flash-to-V4.1 port
+  assessment — reusable expert tiering, four architecture gaps, one a
+  silent-wrongness trap on the Engram key
+- [[dankserver-qwen38-fp8-expert-offload-tuning]] — VERIFIED 2026-09-29:
+  the flag-level decode-tuning campaign on the ad hoc qwen-fp8 container
+  (tcclaviger RDNA4 fork, orcarouter/Qwen3.8-Flash-Next-Uncensored-FP8,
+  TP=2 across both R9700s). **>35 tok/s MET: 37.44-40.57 tok/s with
+  `--expert-prefetch` DELETED** — telemetry measured the prefetcher at
+  36% precision, wasting ~half the gather traffic (the LRU itself runs at
+  82.45% hit); the prefetch-on incumbent 26.8 survived every other
+  challenger (geometry, spec depth, TunableOp-corrupts, batch knobs,
+  concurrency, ngram-fork-bug); the residual gap to ~60 tok/s is the
+  fork's gather read path (6.5-9.5 GB/s effective vs 22.7 at the same
+  slab size)
+- [[frank-primary-glm53-flash-w4a16]] — VERIFIED 2026-09-30: the ACTIVE
+  frank-primary runbook. The cutover to Intel/GLM-5.3-Flash-W4A16-AutoRound
+  (symmetric int4 g128) on the tcclaviger fork, and the reason the int4
+  AutoRound payload is the ONLY one that serves here (its `auto-round` quant
+  method routes to the W4A16 offload kernel; a compressed-tensors W4A16
+  checkpoint of the same model reaches TritonWNA16Experts, which has no
+  offload recipe). **Measured decode 9.9-10.0 tok/s / prefill 918 tok/s; the
+  35 tok/s target is NOT met** — the step is bounded by the fork's gather
+  read path (~8.2 GB/s vs a 22.74 GB/s link) with the cache at a 0.53-0.66
+  hit rate, and MTP4 was measured and rejected (zero decode gain, prefill
+  918 -> 255). Carries the fork's KV-cap bug, the fp8-KV/gfx12 trap, and the
+  `glm47` tool/reasoning parser wiring
 - [[frank-medium-radiance-mxfp4]] — RETIRED 2026-09-26:
   vllm-radiance as ONE tensor-parallel process (TP=2 over the R4D one-shot
   allreduce) serving just1moremodel/Qwen3.8-27B-Uncensored-MXFP4-awq (MXFP4
@@ -181,13 +213,34 @@ The LLM inference fleet on the two hosts. Full index:
   agentionai/Qwen3.8-Flash-Next-ROCmFP4-FAST-imatrix (87.06 GiB,
   Q4_0_ROCMFP4_FAST) + own MTP draft, build-qwen38 (retired 2026-08-30;
   rollback record)
-- [[frank-large-unsloth-qwen38-27b-q8xl]] — the ACTIVE frank-large runbook
+- [[frank-large-rocmfpx-q8agent-mtp]] — the ACTIVE frank-large runbook
-  since 2026-09-21 (root systemd unit, port 7770): stock llama.cpp
+  since 2026-10-02 (root systemd unit, port 7770): the pinned kingjones30/
+  ROCmFPX fork (scripts/build-rocmfpx.sh) serving kingjones777/
+  Qwen3.8-27B-ROCmFP4-STRIX-MTP-GGUF **Q8_0_ROCMFPX_AGENT** — ftype 115,
+  311 tensors of GGML_TYPE_Q8_0_ROCMFPX, a format no other loader on the
+  host reads — with the repo's Q4_0 MTP head drafting 4 tokens and its BF16
+  projector, one session at the native 262,144 window
+- [[frank-large-gufo-qwen38-27b]] — RETIRED 2026-10-01 → 10-02 (rollback
+  target, payload and `scripts/build-gufo.sh` both on disk): gufo v0.1.1
+  built from source serving Qwen3.8-27B UD-Q4_K_XL + the z-lab DFlash2
+  drafter on the gfx1151 iGPU — the two from-source build workarounds, the
+  template-hash allowlist, the measured prefill/decode trade
+- [[frank-large-vllm-mimo-v26-bf16]] — RETIRED 2026-10-01 (same day;
+  replaced by the gufo rollback, then by
+  [[frank-large-rocmfpx-q8agent-mtp]]): the digest-pinned
+  kyuz0/vllm-therock-gfx1151 image (vLLM 0.30, TheRock ROCm 10) serving
+  XiaomiMiMo/MiMo-V2.6-Distill-Qwen-9B in FULL BF16 on the gfx1151 iGPU,
+  4 x 262,144 ctx, with CROSS-MODEL MTP spec decode from Qwen/Qwen3.5-9B —
+  the roof math (13.1 t/s base is ~94% of LPDDR5x BF16), the measured
+  MTP-vs-DFlash table (19.8 vs 17.4 t/s), the greedy-text numerics trap,
+  the `reasoning` field change, troubleshooting
+- [[frank-large-unsloth-qwen38-27b-q8xl]] — RETIRED 2026-09-28 (replaced by
+  [[frank-large-gufo-qwen38-27b]]; rollback target): stock llama.cpp
   (unsloth b11030, ROCm gfx1151) serving
-  unsloth/Qwen3.8-27B-GGUF:UD-Q8_K_XL (a 27B dense model) with the z-lab
+  unsloth/Qwen3.8-27B-GGUF:UD-Q8_K_XL with the z-lab DFlash2 Q4_K_M drafter
-  DFlash2 Q4_K_M drafter (depth 4) on the gfx1151 iGPU — one 262,144-token
+  (depth 4) on the gfx1151 iGPU — one 262,144-token conversation, q8_0 KV
-  conversation, q8_0 KV (the build lacks fp8), text-only, Qwen-Sharp v22.4.0
+  (the build lacks fp8), text-only, Qwen-Sharp v22.4.0 template, the
-  template, the 114 GiB GTT memory model, client contract
+  114 GiB GTT memory model, client contract
 - [[frank-large-kairic-edge-qwen38-27b]] — RETIRED 2026-09-21 (replaced by
   [[frank-large-unsloth-qwen38-27b-q8xl]]; same-day swap): the Kairic Edge
   runtime (ciru-ai/ROCmFPX kairic-edge-qwen38-27b-v1.2, TheRock 7.15.0,
@@ -229,10 +282,26 @@ The LLM inference fleet on the two hosts. Full index:
 - [[ciru-activefpx-promptforge]] — the RETIRED v3 CIRU PromptForge runtime on Dankclaw (superseded 2026-08-24)
 - [[kingjones-strix-mtp]] — the kingjones Qwen3.8-27B ROCmFP4 STRIX MTP model on Dankclaw (active)
 - [[dealign-ornith-crack]] — the retired Ornith-1.5-35B-A3B CRACK model on Dankserver (retired 2026-09-02)
-- [[frank-small-gemma4-12b]] — the ACTIVE frank-small runbook since
+- [[frank-small-llama-mimo-q5km]] — the ACTIVE frank-small runbook since
-  2026-09-17: mainline llama.cpp serving
+  2026-10-02: a CUDA (sm_86) build of llama.cpp master serving
+  bartowski/MiMo-V2.6-Distill-Qwen-9B-GGUF **Q5_K_M** on Dankclaw's RTX
+  3090, port 7771 — THREE conversations at 180,224 tokens each, the
+  asymmetric KV contract (f16 K + q8_0 V), vision through the repo's mmproj,
+  and the two VRAM measurements that set the pool size; ~100 tok/s decode
+- [[frank-small-llama-mimo-q6k]] — RETIRED 2026-10-01 (same day; replaced
+  by [[frank-small-llama-mimo-q5km]], payload still on disk): the first
+  llama.cpp + GGUF shape for this endpoint, Q6_K at the same
+  2 x 262,144 / f16-K + q8_0-V contract, 20.9 GiB measured
+- [[frank-small-sglang-mimo-v26]] — RETIRED 2026-10-01 (rollback target):
+  SGLang in a self-contained python3.12 venv serving
+  letechlead/MiMo-V2.6-Distill-Qwen-9B-INT8-W8A16-AutoRound (qwen3_5
+  hybrid, INT8 W8A16, in-checkpoint vision, NO MTP weights shipped) on
+  Dankclaw's RTX 3090, port 7771 — fp8 KV, pinned 606,208-token pool,
+  2 x 262,144 slots, bf16 mamba states, eager prefill
+- [[frank-small-gemma4-12b]] — RETIRED 2026-10-01 (rollback target):
+  mainline llama.cpp serving
   HauhauCS/Gemma4-12B-QAT-Uncensored-HauhauCS-Balanced at Q4_K_M
-  (MTP assistant sidecar + ngram-mod, q8_0 KV, --fit → 262k) on
+  (MTP assistant sidecar + ngram-mod, q8_0 KV) on
   Dankclaw's RTX 3090, port 7771
 - [[frank-small-vllm-gemma4]] — RETIRED 2026-09-19 (same-day revert,
   rollback target): vLLM serving google/gemma-4-12B-it-qat-w4a16-ct (w4a16

serverAppConfigs-docs/log.md

@@ -1,8 +1,535 @@
 # Bundle Change Log
 
+## 2026-10-01
+- **frank-primary rolled BACK to Qwen3.8-Flash-Next-Uncensored-FP8 + MTP4;
+  GLM/DFlash2 declared failed and purged.** The DFlash2 drafter booted and
+  served (window 6j, 65,536 ctx, `glm53-flash` on :7773, both TP ranks
+  captured FULL dflash2 cuda graphs — five patches deep: EAGLE3 wrapper
+  interface v5, KV unifier MLA-multiple, kpool tailpin, glm5 KV pool with
+  draft+hidden groups, registry method-signal), but **acceptance pinned at
+  1.00-1.38 on every text class INCLUDING verbatim repetition** — a mis-wired
+  aux-hidden-state capture, not a weak drafter. At ~1.2 acceptance the
+  8-token verify tax put decode at 1.6-1.8 tok/s vs the 9.9-10.0 baseline
+  (5x slower). Frank's independently reported 18 tok/s / ~370 prefill did
+  not originate from this engine (APIServer access log: every completion
+  POST was host-local; his workstation appears once, a /v1/models GET).
+  Restored the verified 37.4-40.6 tok/s arm from the `2339fae` trio
+  (200k x 3 seqs, fp8 KV, prefetch off); GLM payload (170 GiB) + drafter
+  (2.2 GiB) + scratch staged to `/home/dank/.AGENTTRASH/20261001-glm-purge/`.
+  KV-cache patches 02/03/04/04b survive in `scripts/patches/` (generic
+  unifier/tailpin/grouping/registry fixes, apply-verified).
+
+## 2026-09-30
+- **frank-primary cut over to GLM-5.3-Flash W4A16 int4 — it serves, but NOT at
+  the 35 tok/s target, and the shortfall is measured rather than inferred.**
+  - Payload `Intel/GLM-5.3-Flash-W4A16-AutoRound` (AutoRound symmetric int4
+    group 128, `auto_round:auto_gptq` packing, ~170 GiB, 34 shards + 2 extras,
+    113,074 tensors). It is the only W4A16 checkpoint this fork can serve WITH
+    expert offload: its `quant_method: auto-round` is overridden to `inc`, which
+    routes the MoE to libr4d's grouped W4A16 GEMM (`R4dW4a16MoEMethod`). The
+    compressed-tensors W4A16 build of the same model (`dealignai/…-CYBERSECURITY-W4A16`,
+    182 GiB, fetched earlier) resolves to `TritonWNA16Experts`, which has NO
+    offload recipe — and 170 GiB of experts cannot fit 64 GiB of VRAM, so it
+    refused to serve at all. That checkpoint was purged with the other retired
+    payloads.
+  - Shape: tcclaviger RDNA4 vLLM fork (same digest-pinned image as the qwen
+    cutover), TP=2, 131,072 ctx x 2 seqs, bf16 KV, util 0.98, NBATCH 2048,
+    `--enable-expert-offload --expert-precise-memory`, prefix caching, vision +
+    `glm47` tool and reasoning parsers, MTP OFF, prefetch OFF.
+  - Measured on the deployed engine: **decode 9.9-10.0 tok/s, prefill 918 tok/s**
+    (warm, 2.6k prompt), KV 138,561 tokens; arena 14.51 GiB = 2,383 slots per
+    rank over 12,096 host-backed experts (73.64 GiB pinned per rank).
+  - **The wall — and a corrected one.** The first account of the shortfall
+    bounded the step with throughput arithmetic (a step moves 0.82 GiB of misses
+    in ~100 ms = "8.2 GB/s effective against a 22.74 GB/s link", needing a 0.89
+    hit rate for 35 tok/s). **A torch-profiler kernel-time window falsified it**
+    (496 decode tokens, rank0 trace parsed for device-activity time): 46.18 s of
+    kernel time = ~93 ms/token, of which **44 % is pointwise glue** —
+    `vectorized_elementwise_kernel` alone 16.07 s over **708,500 launches (1,428
+    per token)**, plus 3.11 s of `elementwise_kernel_manual_unroll` — against
+    **1.05 % for `r4d_moe_lru_gather_kernel`**. The MoE GEMM is 10 %, the split-K
+    GEMM 6.4 %, the router/top-k 8.9 %, the mHC kernels 6.5 %, and the inductor
+    fusions that DO happen 6 %. The real cause is that the fork compiles its own
+    Qwen trunk (`@support_torch_compile` in `models/qwen4_exp/amd/model.py`) and
+    **not** its GLM trunk, so every pointwise op is a separate eager kernel; the
+    engine says so at boot ("...provides neither a compiled submodule nor
+    breakable CUDA graphs. Overriding to FULL_DECODE_ONLY."). Growing the arena
+    13.08 -> 14.51 GiB still changed nothing, consistent with a 1 % gather.
+  - **Fork work staged (compile changelist, WIP — does not boot yet).**
+    `scripts/patches/glm53-compile-01-model-decorator.patch`,
+    `-02-trace-safe-probes.patch`, `-03-fullgraph-optout.patch`, each with its own
+    evidence header and base/patched sha256. Boots established, in order: (1)
+    `functools.lru_cache` does NOT make a probe trace-safe — dynamo re-traces the
+    wrapped body even when warm, so `r4d_mhc_supported -> r4d_mhc_enabled ->
+    available -> os.path.exists` breaks the graph (a plain module-global read
+    compiles; verified in-image); (2) `suppress_errors` does NOT cover graph
+    breaks, only `fullgraph=False` does; (3) the GLM trunk can never satisfy
+    `fullgraph` because its indexer metadata syncs to host
+    (`glm5next/amd/sparse_indexer.py:263,419`), and AOT compile refuses breaks
+    outright. With all three the trunk **traces** (`Dynamo bytecode transform
+    time: 1.08 s`) and then dies on `KeyError: 'in_proj_qkvbfg_a'` in the
+    mHC/KDA path — a latent bug the compile exposes. That is the remaining
+    blocker.
+  - **No new HIP kernels are possible** (the image ships `r4d.so` with no
+    sources), which binary-blocks the second-largest lever: the fused router is
+    gated `num_experts in {128, 256, 512}` and only
+    `r4d_moe_route_kernel<128|256|512>` instantiations exist, while GLM has 288
+    experts. `--kv-cache-dtype fp8` buys no capacity either (the plan's KV
+    reserve is dtype-blind), only removing gfx12's only MLA decode path.
+  - **DFlash2 for GLM is a target-side port, not a drafter port.**
+    `incoai/GLM-5.3-Flash-DFlash2` declares `architectures: ["DFlash2DraftModel"]`
+    and `model_type: "qwen3"` (5 dense layers, sliding attention, block 8,
+    `target_layer_ids [5, 14, 24, 33, 42]` of 45) and the fork **already maps it**
+    (`registry.py:636` -> `qwen3_dflash2.DFlash2Qwen3ForCausalLM`; spec method
+    `"dflash"`). What `glm5next` lacks is the target side: aux-hidden-state
+    capture (`set_aux_hidden_state_layers` / per-layer capture, as `deepseek_v2.py`
+    demonstrates). No speedup is budgeted: a verification pass pays pointwise cost
+    proportional to its token count.
+  - **DFlash2 tried, and the gap was exactly that one interface — now closed.**
+    Payload fetched and hash-verified (`/home/dank/models/GLM-5.3-Flash-DFlash2`,
+    2,342,169,800 B, sha256 `b038e1d9…` = the Hub's LFS oid). Booting with
+    `--speculative-config '{"method":"dflash","model":"/draft","num_speculative_tokens":8}'`
+    accepted the config and the drafter and died at worker init with
+    `RuntimeError: Model does not support EAGLE3 interface`. The port
+    (`scripts/patches/glm53-dflash-01-aux-hidden-states.patch`) took FIVE versions —
+    hand-written methods -> Protocol base (MRO kill at class creation) -> attrs on the
+    text model -> attrs + methods on the text model -> **v5: attrs + methods on BOTH
+    wrappers** (the runner checks the LOADED instance, which for GLM is the multimodal
+    `Glm5NextForConditionalGeneration` holding the text model as `self.language_model`;
+    `pixtral.py:498` is the reference delegation pattern). v5 CLEARS the gate: both TP
+    workers log `Using Eagle3 auxiliary layers from config: (6, 15, 25, 34, …)` and the
+    DFlash2 drafter resolves + loads. Getting past that point was then interrupted twice
+    by experiment-window RAM contention (see the window discipline below), so the
+    drafter-serving end state was still unproven when this entry was written.
+  - **Compile work: one confirmed blocker, and a launcher bug that misled four boots.**
+    With breaks genuinely tolerated (`fullgraph=False`, AOT off) the trunk traces and
+    dies on the dynamo name-map `KeyError: 'in_proj_qkvbfg_a'`. Four later boots
+    reported *different* fatal breaks — `Unsupported: Read uninitialized cell` at
+    `mla.py:208` (the fused-norm JIT kernel), then `Unsupported method call` in the
+    indexer's in-forward workspace acquisition
+    (`sparse_indexer.py:132 -> workspace.py:135 get_simultaneous / :155
+    _ensure_workspace_size`), then `Skip calling torch.compiler.disable()'d function`
+    — and the cause was **my own scratch launcher**: its compile branches were
+    restructured and the `-e VLLM_COMPILE_FULLGRAPH=0` line was dropped, so those
+    boots ran `fullgraph=True`, where *every* break is an error and the first one
+    wins. I first blamed `TORCH_LOGS=graph_breaks` for the difference; that was wrong
+    (the flag only logs). The moving reports are still useful — they are the break
+    list — and patch 04 demonstrably removes the first of them (the fused-norm cell),
+    patch 05 makes the workspace acquisition compiler-opaque.
+    `scripts/patches/glm53-compile-04-fused-qkv-rmsnorm-optout.patch` takes GLM off
+    the fused-norm kernel (`R4D_GLM_FUSE_QKV_RMSNORM=0`, a constructor flag glm5next
+    sets at `attention.py:583`).
+  - **Diagnostics built for this (reusable).** `TORCH_LOGS=graph_breaks` +
+    `VLLM_USE_AOT_COMPILE=0` names each break with its user stack (AOT compile
+    hard-refuses breaks outright); `/home/dank/dynamo-probe` on `PYTHONPATH` with
+    `VLLM_DYNAMO_DUMP_ATTRS=1` prints the dynamo name-map miss detail (node target,
+    near-miss keys, whether another live `OutputGraph` holds the key). Two gotchas
+    recorded: a `.pth` on `PYTHONPATH` is NOT executed (use `sitecustomize.py`,
+    chaining the image's apport hook), and the scratch launcher used to reset
+    `ENVARGS` (now `${ENVARGS:-}`), which silently dropped env vars and wasted a boot.
+    All four patch artifacts are now genuine `diff -u` patches, verified to apply to
+    the pinned image and to reproduce their recorded sha256.
+  - Measured and rejected: **MTP4** (the head loads, arch resolves to
+    `Glm5NextMTPModel`, acceptance is excellent at mean length 3.76-4.90 — but
+    decode was unchanged (9.6-11.2 vs 10.0) while prefill COLLAPSED 918 -> 255.
+    The profiler window above explains it: the per-token cost is dominated by
+    pointwise glue that scales with the token count, so verifying a draft batch
+    pays ~N x that glue for the same N tokens of output);
+    **fp8 KV** (disables the sparse MLA backend's rope-free Triton path, and
+    aiter's MLA decode op is gated on CDNA>2, so the MTP draft step demands an op
+    that cannot exist on gfx12); **NBATCH 8192** (boots, identical prefill).
+  - Fork bug found: the plan's KV reserve under-models this model's
+    per-sequence KDA state and the MTP draft's KV, and `gpu_worker` caps KV at
+    that reserve, making a legal `--max-model-len` unbootable. Shipped
+    `scripts/patches/gpu_worker-kv-cap.patch` (lets an explicit operator KV
+    request raise the cap) — NOT needed by the deployed shape.
+  - Rollback: commit `2339fae` (qwen FP8 + MTP4); the qwen payload and the R9V
+    state dirs are intact on disk.
+
+
+## 2026-09-29
+- **frank-large tuning investigation: a measured NEGATIVE result, nothing
+  changed.** Frank asked for iterative decode/prefill improvement at a 32k
+  working point without touching the model or quantization. **No win exists
+  at gufo's served settings**, so no config change was shipped. frank-large
+  was stopped for the whole investigation (~4h) per the benchmarking policy
+  and restored healthy in 9s afterwards.
+  - Measured on the repo's own prose and shell sources at **32,762 prompt
+    tokens**, salted per run so a warm prefix cache could not flatter a
+    later run, 3–5 repetitions per configuration.
+  - **`--prefill-chunk` is dead**: 512 → 8192 moves prefill 410.7–412.4 t/s.
+    A 16x change in chunk size, 0.4% out.
+  - **`--draft-tokens` is flat**: 5 → 10 gives 13.79–15.86 t/s decode, all
+    inside the run-to-run noise band. The 5-rep confirmation round
+    **reversed** the 3-rep leaders — `dt9 + chunk4096` looked best and came
+    **last** (12.86), while the **deployed config came first** (14.71).
+  - `--draft-policy` fixed vs adaptive: no effect. `--min-draft-tokens`:
+    **rejected by the engine** ("DFlash2 requires ... 1"). MTP: **unsupported
+    for Q8 targets** (`Unsupported Qwen MTP tensor type for
+    token_embd.weight: Q8_0`).
+  - `--context 262144 → 40960` was tested as a diagnostic (the deployed value
+    preallocates KV ~8x larger than a 32k conversation needs): **no gain**
+    (13.93 vs 14.71 t/s), and it would break `models.yml`'s declared
+    `contextWindow: 262144`. Not shipped.
+  - **What actually moves the number is speculation itself: 6.65 t/s off →
+    ~15 t/s on, 5.3x** — already enabled. Prefill is insensitive to
+    everything tried, consistent with gufo's own `EXPERIMENTS.md` where
+    dozens of kernel variants are recorded rejected for "no release
+    throughput improvement".
+  - **Two traps are now in AGENTS.md's benchmarking policy**, because they
+    cost most of the wall time and both produced *false* results:
+    (1) **`gufo bench` cannot rank speculative settings and inverted the
+    ranking** — its corpus reports `acceptance=0.933 +/- 0.000` with an
+    identical trace sha256 across draft widths (the same highly-predictable
+    text every run, 93% of proposals accepted). It reported 29–37 t/s where
+    real traffic gives ~15, and made dt16 look 6% faster than dt7 when real
+    text made it 14% slower. (2) **A live A/B needs >=128 generated tokens**;
+    at 8, `inter_token` is first-token noise and the first pass of the sweep
+    produced a nonsense ordering that the 128-token pass reversed.
+  - Also worth recording: the `Server-Timing` header is
+    `ttft;dur=402.485` — the unit is in the **name** half, so splitting on
+    `=` keys the dict `ttft;dur` and every lookup silently returns nothing.
+    Four configurations reported "no Server-Timing" while the header was
+    present on all of them.
+- **dankserver FP8 decode-tuning campaign: flags exhausted at 26.8 tok/s,
+  mechanism measured.** The ad hoc qwen-fp8 container (tcclaviger RDNA4
+  fork, orcarouter Qwen3.8-Flash-Next-Uncensored-FP8, TP=2) ran a full
+  flag-level A/B battery at the 32k working point; every number is from the
+  server's usage counters (SSE chunks are invalid under spec decode). New
+  runbook: [[dankserver-qwen38-fp8-expert-offload-tuning]].
+  - Incumbent stands: MTP depth 4, stock gather 8x16, prefetch depth 1,
+    TUNABLE off — **26.80/25.15/26.81 tok/s** at 166/1087/4064 ctx,
+    acceptance 60.7%; the re-check reproduced it (25.80/24.38/26.63).
+  - **Ruled out by measurement:** gather geometry 16x32 (−9%) and 32x64
+    (parity) — the fork's "tuned at the Gen4 ceiling" comment is stale, the
+    stock constants are already right on Gen5 x8; MTP depth 3 (−9%, flat in
+    ctx) and depth 6 (collapses to 13.5–19, acceptance 17.4% — and proves
+    the UVA read-through path is SLOWER than copy-in); MNBT 8192 (neutral);
+    prefetch depth 2 (neutral — the tier is 16 slots/layer against a
+    ~2,400-slab working set at 73% misses); 2-way concurrency (summed
+    25.49 < single 26.8 — under TP both streams' misses gather on each
+    rank, the links do not add capacity, and the shared arena thrashes).
+  - **TunableOp rejected:** it silently skips under expert offload unless
+    `CLAV_TUNABLEOP_WITH_OFFLOAD=1`; once forced on it corrupts numerics —
+    repetition loops, the server's repetition-halt firing at ~140 of 256
+    tokens, "79.8 tok/s" measured on halted degenerate completions, and a
+    fresh-prompt check at 20.01 (below baseline). +6.7 min warmup, no
+    persistent cache in the container lifecycle.
+  - **Infeasible:** `UTIL=0.99` (free-memory wall: 31.29 of 31.86 GiB free
+    at worker init); `--max-num-seqs 2` (compile constraint violation at
+    CUDA-graph capture); ngram-only drafter (fork bug: "PLE inputs were not
+    prepared" — PLE prep is wired only for the MTP spec path).
+  - **The wall, measured:** the link does Gen5 x8 = 31.5 GB/s/direction;
+    the HIP probe reads 22.74 GB/s at the exact 2.345 MiB expert slab (91%
+    of ceiling at bulk); the engine's effective gather rate is ~9.5 GB/s =
+    42% of that. A decode step is ~100 ms compute + ~400 ms PCIe wait
+    (peak util 100%, smooth ~29%). The gap to the 35 tok/s target is the
+    fork's gather read path — fork-patch territory, not flags.
+  - **Telemetry fork bugs:** the torch-profiler worker stop hangs on ROCm
+    (>10 min, two attempts, while the AsyncLLM wrapper logs success in
+    1 s), and the default `torch_profiler_with_stack=true` grew a 6.5-min
+    profile window's trace buffer ~40 GB → the kernel OOM-killed the engine
+    core (anon-rss 95.2 GB, `oom-kill` in `journalctl -k`). The per-layer
+    expert-cache hit rates (`R4D_LRU_TELEMETRY=1`) are reachable ONLY
+    through that stop path, so the telemetry JSON is unreachable without a
+    fork patch.
+  - **256k ctx boots** (the last original-goal dimension): a 262,144-token
+    window needs **4.0 GiB of KV** (hybrid attention + mamba pages ≈
+    16 KiB/token, not the 32k plan's 9,190 B/token) — 3 GiB boot-refused
+    (`4.0 GiB needed > 2.98 available`; the engine's own max-len estimate
+    at that pool: 184,224). At KV 4.5 GiB it serves **36.65/34.21/34.77**
+    (arena 6,574→5,648 slots, −14%, ~10% decode cost; 2 of 3 depths just
+    under 35). KV-exact-4.0 arm in flight for the last +3.5% slots.
+    Correction recorded: the planner sizes the arena AFTER the KV override
+    (the earlier "before" reading was wrong) — the arena is what remains.
+  - **Final 256k probe: the KV trim is inert.** KV-exact 4.0 GiB
+    boot-refused by 20 MB of accounting (3.98 available < 4.0 needed;
+    engine estimate at 3.98: max len 260,176); +32 MiB (`KV_BYTES=
+    4328521728`) boots — but the arena is **identical to the 4.5 GiB arm
+    (5,648 slots)**: the planner buckets, the 0.47 GiB went to margin.
+    Decode band **36.48/34.13/35.02** (acceptance 39.4%) — same 34.1-36.5
+    band, straddling the 35 target within noise. Final 256k config:
+    `MAXLEN=262144 KV_BYTES=4328521728`, prefetch off. The full-speed
+    alternative stays `MAXLEN=184224` at 3 GiB (37.4-40.6 band).
+  - **GLM-5.3-Flash readiness REVISED by source reading (Frank's question
+    about int4 vs mxfp4):** the tcclaviger fork in the qwen-fp8 container
+    registers `Glm5NextForCausalLM`/`Glm5NextMTP` with a backend-split
+    module (`glm5next/amd|common|nvidia`, `sparse_indexer.py` for DSA);
+    `offloader/expert.py` accepts `R4dDsfp4MoEExperts, R4dW4a16MoEExperts,
+    Fp8HipMoEExperts` — and **`R4dDsfp4MoEExperts` is MXFP4 weights with
+    the DeepSeek activation law** (docstring; SwiGLU `gemm1_clamp_limit`
+    default 10.0), inheriting the mxfp4 repack/GEMM/emulation-fallback.
+    **`r4d_lib.has_dsfp4()` is True** in this build — native `moe_dsfp4` /
+    `moe_dsfp4_mt1` entries exist, so the old "MXFP4 = emulation, missing
+    kernels" GLM verdict is dead for this fork. WNA16 (int4) backend
+    oracle on ROCm reduces to HUMMING/TRITON (Marlin/TRTLLM/XPU are
+    CUDA/NVIDIA/Intel-only), payload must be compressed-tensors W4A16
+    group-wise symmetric. Remaining GLM gate: a compressed-tensors MXFP4
+    (~165 GiB) or W4A16 payload + a glm5next-on-HIP boot probe; community
+    GLM-5.3 quants are GGUF (llama.cpp path only), native FP8 is 328 GB
+    (> 246 GiB RAM). Feasibility doc rows updated.
+  - **Revert pins installed (Frank's request):** (1) repo tag
+    `qwen-fp8-tuned-20260929` at f532e03 (the campaign's docs + measured
+    config; `git checkout qwen-fp8-tuned-20260929` restores the state);
+    (2) image pinned by digest — serve script now launches
+    `docker.io/tcclaviger/vllm@sha256:da402966...` (ID 3322dbbbc96d) with a
+    local alias `qwen-fp8-runtime:tuned-20260929`; the floating `:latest`
+    is no longer referenced anywhere (the other image on the host,
+    `28.06.1-Code000` = fffca39f, is a DIFFERENT build — do not confuse).
+    Runbook deployment-shape section updated.
+  - **Endpoint published:** `PORTMAP=7773:7770` hook added to the serve
+    script; the 256k config relaunched with it. Host-side checks: health
+    200 on 127.0.0.1:7773 AND 192.168.0.235:7773, /v1/models lists
+    qwen3.8-flash-next, decode through the published path
+    36.49/34.55/35.98 (same band). 7773 chosen over 7770 — frank-primary
+    owns 7770 when it ever returns.
+
 ## 2026-09-28
+- **Dankclaw model cleanup** (disk was at 93%, 137 G free). Frank asked to
+  clean up old models. **510 G staged** to
+  `~/.AGENTTRASH/20260928-retired-models/` — moved, NOT deleted, with a
+  `MANIFEST.md` giving a restore command per item. **Nothing is freed until
+  the purge is approved**; that is the point of the trash policy, and the
+  number to know before expecting the disk to move.
+  - Staged: CIRU Orca 128 G, halogen 123 G, unsloth Flash-Next IQ4_XS 91 G,
+    mradermacher IQ4_XS 91 G, unsloth 27B 29.3 G, Kairic Edge 27 G,
+    ISTA-DASLab GSQ-RCO 13 G, vLLM Gemma4 w4a16 9.6 G, vLLM Gemma4 drafter
+    838 M. Every one maps to a runtime already retired in the runbook index.
+  - **The 29.3 G rollback target went with them**, by Frank's decision. The
+    payload-only rollback is now a re-fetch first; both the runbook and
+    AGENTS.md say so. **The DFlash2 drafter was NOT reclaimed** — only the
+    `.gguf` was taken out of that directory, because the live frank-large
+    drafter lives in it. The manifest warns against restoring the directory
+    wholesale.
+  - **Kept deliberately:** `checkpoints/` 23 G. It is unreferenced by the
+    repo, but ComfyUI is live and its workflows were not searched — not
+    confirmed dead, so not taken. `logs/x4e-run.log` also stays: it is a
+    **sparse** file, 3.3 G apparent but 4 K real, so removing it frees
+    nothing.
+  - **A live reference check came first, and it earned its keep.**
+    `serve-frank-small.sh`'s *fallback* `MODEL_DIR` still pointed at the
+    retired GSQ-RCO payload frank-small served on 2026-09-16/17. The env
+    file has always overridden it, so the drift was invisible — but a
+    missing or unreadable env file would have turned a config problem into a
+    "missing asset" naming a model that no longer exists. Repointed at
+    `Gemma4-12B-QAT/`, which is what the env file already sets. Every other
+    hit was a comment, a retired fetch script's destination, or
+    `retire_halogen_large_quadlet` (which moves a *quadlet*, not a model).
+  - **Not taken, by Frank's decision — scope was models only:** podman
+    images (**279 G reclaimable, 95%**; 170 images, 23 active) and the
+    uv+pip caches (**73 G**). Together that is 352 G of regenerable garbage
+    larger than the models reclaimed, and it is the obvious next move.
+  - Pre-existing: **118 G of trash from 2026-09-21** was already staged
+    (`20260921-frank-large-assembled` 88 G, `-redownload` 30 G) and never
+    purged. Total now staged on this host: **627 G**.
+  - Verified after the moves: frank-large and frank-small both `active`,
+    both `/health` 200, and all three live assets (Swift target, DFlash2
+    drafter, Gemma4 target) still present.
+- **frank-large payload swapped to Swift-1.5 (uncensored)** (production
+  change, Dankclaw). Frank asked whether the UD Q8 of
+  `ajgazin/Swift-1.5-Qwen3.8-27B-Uncensored-Dynamic-MTP-GGUF` would load on
+  gufo, then said to do it. Port 7770, alias `frank-large`, one 262,144
+  session, the DFlash2 drafter and the client contract are unchanged; the
+  27B dense weights are an **abliterated (uncensored)** fine-tune of the
+  same base, under the **non-OSI SWIFT OPEN LICENSE 1.0**.
+  - **Compatibility was established from the GGUF headers, not from the
+    filename.** Both files were range-fetched and parsed: same
+    `general.architecture` (`qwen35`), same 866 tensors with **identical
+    names in identical order**, **zero** per-tensor `ggml_type` or dim
+    mismatches (Q8_0×453, F32×360, BF16×53), same MTP head at `blk.64`, same
+    geometry. The two scary metadata differences are both no-ops, confirmed
+    against gufo's source: `general.file_type` (15 vs 7) is *"deliberately
+    not consulted"* by `gguf_reader.cpp` — the file gufo already served
+    records 15 (Q4_K_M) while containing no Q4_K tensor — and Swift's extra
+    `qwen35.attention.recurrent_layers` key is **never read** by gufo
+    anywhere in `src/`; layer kinds come from `full_attention_interval`,
+    which both files carry as `4` and which agrees with it.
+  - **Fetched and verified:** 31,457,990,272 bytes, sha256 `9076f99a…`
+    matching the Hub LFS oid, via the new
+    `/scripts/download-frank-large-swift.sh`. It sends **no auth header** —
+    the repo is public and ungated, and a token on the curl command line is
+    world-readable via `/proc/<pid>/cmdline`. The anonymous transfer ran at
+    ~1.8 MiB/s; resuming with the host's token through a **mode-600
+    `curl -K` config** took it to ~7 MiB/s. The download survived an ssh
+    drop by running `setsid nohup`, not under the session.
+  - **Load test, service stopped first** (the GPU policy): **9.2 s** from
+    `load_started` to `load_completed`, 66.5 GiB of the 116.7 GiB iGPU at
+    context capacity 262,144, no loader warnings. Contract battery green:
+    `context_length 262144`, `content` + `reasoning_content`,
+    `finish_reason: tool_calls`, SSE, and a **two-turn tool loop** with prior
+    reasoning preserved. Spot-checks: "17*23" → `391`; asked for a
+    lockpicking scene it wrote one without hedging, confirming the
+    abliteration.
+  - **THE TEMPLATE LAW IS GONE, by decision.** Frank asked to "use our chat
+    template"; gufo **cannot** take it. This is not a missing flag —
+    `src/models/qwen/chat_template.cpp` is 587 lines of hand-written C++
+    that appends strings, with no Jinja engine, no `--chat-template-file`,
+    no env override and no template body in the binary. Qwen-Sharp
+    v22.4.0 is 453 lines of Jinja2 with `namespace()`, macros and
+    `raise_exception`, so supplying it is a **port, not a config line**.
+    Frank chose to ship on gufo's stock renderer and accept the loss: the
+    `terse` system-prompt injection, in-band `<|think_*|>` control, the
+    `json|xml` `tool_call_format` switch, tool-argument truncation and
+    `add_vision_id` prefixes are all gone. gufo's renderer does keep the
+    effort instruction strings and `preserve_thinking`. **Worth watching:
+    an abliterated model now runs without the prompt that tells it to stop
+    waffling** — the cheap fix is to send the `terse` text as an ordinary
+    system message from `models.yml`, not to touch a template.
+  - **The payload is now two files in two directories.** The target moved
+    repo; the DFlash2 drafter did not move, so `DRAFTER_DIR` is its own
+    tunable. That broke the deployer's stray-run sweep, which keyed on the
+    *target* filename and would have silently stopped matching — a stray
+    run would then have held 7770 past the port gate. It now keys on the
+    drafter, which every frank-large command line carries via
+    `--dflash-model` and which did not move.
+  - **`/tests/frank_large_unit.bats`** — 20 assertions. The mutation pass
+    found **four real gaps** on the first run, all now fixed: two fetch
+    assertions were satisfied by a **header comment** rather than the URL
+    assignment, so repointing a repo left them green; and nothing bound the
+    env's `MODEL_FILE`/`DRAFTER_FILE` to the filenames the fetch scripts
+    actually download, so a re-upload or an env edit would break the
+    endpoint with a "missing asset" no test predicted. There is now an
+    explicit cross-file invariant. A second harness bug worth recording:
+    a `bats --filter` that matches no test reports zero `not ok`, which
+    looks exactly like a mutation that survived — the harness now asserts
+    the `1..N` count before believing a result. 16/16 mutations caught.
+  - Docs: runbook gained a **Payload** section (the header evidence),
+    **The template law** section and a two-way **Rollback**; AGENTS.md
+    fleet table, frank-large section and rollback.
+- **frank-large runtime swapped to gufo** (production change, Dankclaw).
+  Frank asked for `github.com/gufo-org/gufo` — a vertical inference engine
+  built for exactly this hardware (Ryzen AI MAX+ 395 / Radeon 8060S,
+  `gfx1151`) — in place of the stock llama.cpp prebuilt. Port 7770, alias
+  `frank-large`, one 262,144-token session and the client contract were all
+  unchanged; only the engine was new, and the payload was the unsloth
+  Qwen3.8-27B UD-Q8_K_XL **at that point** — it was swapped for the
+  abliterated Swift-1.5 later the same day, see the entry above.
+  - **The runtime is built from source on the host.** gufo ships no prebuilt
+    Linux binary (the v0.1.0/v0.1.1 release assets are empty) and the host
+    has no Nix. Pinned at `v0.1.1` / `b0f8673`, installed to
+    `/home/dank/GitPackages/gufo-v0.1.1/` by the new
+    `/scripts/build-gufo.sh`, which is reproducible and fails closed on a
+    moved tag. Unlike the unsloth prebuilt it replaces, gufo **links the host
+    ROCm 7.2.4** through its RPATH, so the HIP dev packages must stay.
+  - **Two host workarounds, both encoded in the build script.** Ubuntu's
+    `librocwmma-dev` ships a truncated rocWMMA header set into
+    `/usr/include` that shadows AMD's complete one; the build died on
+    `'internal/accessors.hpp' file not found`, and a `-I` cannot fix it
+    because in HIP mode the host compilation pass searches the system
+    directories first. The package has no reverse dependencies; it is
+    removed. Second, Ubuntu 26.04 ships `/usr/include/c++/16` and the ROCm
+    clang prefers the newest tree, whose `<atomic_wait.h>` needs a
+    `std::__detail::__notify_impl` symbol version the installed
+    `libstdc++.so.6` (GLIBCXX_3.4.35) does not export — the link failed until
+    `--gcc-install-dir` was pinned to GCC 15.
+  - `gufo diagnose` reports `[PASS]` on `gfx1151` (ROCm 7.2.3, amdgpu
+    loaded, 20 CUs). It self-caps compute queues at 2 resident for
+    `serve llm` after reading `/sys/class/kfd/kfd/proc/*/queues/*/type`,
+    which matters because ComfyUI holds queues on the same device.
+  - **Shape:** one session, `--context 262144`, DFlash2 with the **adaptive**
+    block policy capped at `--draft-tokens 7` (llama.cpp's fixed depth 4 is
+    gone), greedy, `--think auto --preserve-thinking on`, text-only.
+  - **Client-contract deltas, all in the runbook:** the chat template is
+    compiled into the binary (no `--chat-template-file`, so the Qwen-Sharp
+    73k-token-prompt failure mode cannot recur); `reasoning_effort` now
+    accepts the full OpenAI set, so `models.yml`'s `reasoningEffortMap` fold
+    is no longer required — **kept anyway** so this commit does not also move
+    every `high` role from `xhigh` to `high` effort; and there is no
+    KV-quantization or prompt-cache-RAM flag.
+  - **Measured at cutover:** 13 s from `load_started` to listening (target
+    8 s, drafter 2 s, session 3 s) against a 20-minute `LLM_WAIT` budget;
+    RSS 30.6 GiB; a live short-prompt request returned
+    `Server-Timing: ttft;dur=549.8, inter_token;dur=24.2` (~41 tok/s).
+    Verified live: `/health` 200, `/ready` 200, `/v1/models` →
+    `frank-large` at `context_length 262144`, a chat completion carrying
+    `reasoning_content`, a `tool_calls` response with
+    `finish_reason: tool_calls`, and SSE streaming.
+  - **The trade, stated plainly.** gufo's own Q8_K_XL tables against
+    llama.cpp b11069: prefill +37 % to +80 %, cold load 4.5× faster, decode
+    +7 % at zero depth — but **−14 % at 65,536 tokens of depth and −24 % at
+    131,072**. On the 8-bit target the crossover is ~32k and frank-large runs
+    one 262,144-token conversation, so long agent sessions sit on the losing
+    side of it. The Q4_K_XL target shows no such regression; revisiting the
+    quantization is a separate decision.
+  - `/tests/frank_large_unit.bats` — Rewritten around behaviour: the seam
+    runs the serve script against a stub engine and asserts on the argv it
+    actually execs, instead of grepping the script's text. A caveat found
+    while writing it: **bats 1.13 ignores a failing `A && B` in the middle of
+    a test** (bash exempts the non-final element of an AND-list from
+    `errexit`), so every such assertion was split into standalone `[ ]` lines.
+    Mutations confirmed the seam bites: dead runtime gate, `DRAFT_TOKENS=0`,
+    a wrong context/sessions/port, an alias swap, a moved drafter, a removed
+    missing-asset message and a neutered commit pin each fail it.
+  - `/scripts/deploy-remote.sh` — `deploy_llm_large` gates on the runtime
+    binary as well as the payload and returns **before** it stops the serving
+    unit, so a host whose gufo was never built reports the failure instead of
+    taking the endpoint down on the way there.
+  - Docs: new `/serverAppConfigs-docs/LLM/frank-large-gufo-qwen38-27b.md`;
+    `frank-large-unsloth-qwen38-27b-q8xl.md` marked RETIRED (still the
+    rollback target — the prebuilt and the payload are both intact on disk);
+    AGENTS.md fleet table, frank-large section, bench policy and runbook
+    index; both vault indexes; `Dankclaw System Specs.md`; `README.md`.
+  - **e2e gate: 32/33.** The one failure is `192.168.0.112` (the FrankBook-Pro
+    Mac mini) at 100 % packet loss — offline, and the same failure recorded
+    in `564a781`. Not related to this change; no frank-large test failed.
+- **Frontier-MoE feasibility study on Dankserver** (research + build, no
+  production change). Frank asked which of DeepSeek-V4.1-Flash,
+  MiMo-V2.6-Flash-MOPD and GLM-5.3-Flash would run best on the 2x R9700 box,
+  and whether Affinity / vllm-radiance / FreeToken-AMD could be modified to run
+  them fast. Findings, all evidenced:
+  - **V4.1 does not fit.** 475.27 GiB checkpoint (measured from the HF API),
+    of which **189.13 GiB is Engram** in two 94.56 GiB shards. A token selects
+    240 routed experts at ~11 MB; at the 37-of-384-per-layer residency the hot
+    stack can hold, ~2.4 GB crosses PCIe per token. At 55 GB/s real that is a
+    **~23 tok/s ceiling** — below the 35 tok/s goal before any attention or
+    dense work. The 1000 t/s prefill target is ~20x away: the backend refuses
+    layer-major prefill for V4.1, so the only legal prefill is token-by-token.
+  - **Affinity** (Codeberg, StillDeadcode) is a *DeepSeek-V4-Flash* engine
+    claiming 110 tok/s on two R9700s; it **builds clean here** (hipcc 23,
+    gfx1201, zero errors). The 110 works because the 101 GiB container leaves
+    the cold expert set in a pinned host pool in RAM. A V4.1 port reuses the
+    quantiser, container, slab, host pool and placement engine (expert tensor
+    naming is identical, 15,360 tensors at 40x384) but must write CED, CSA2's
+    new indexer, the DSpark draft geometry and Engram — and Engram fails
+    *silently*, because V4.1 has no `num_hash_layers` key so Affinity's
+    `model.h:36` default of 3 survives. Weeks of work, same capacity wall.
+  - **Lucebox** is the only engine that runs V4.1 on gfx1201, on the unmerged
+    `feat/ds41-engine` branch. Its own DS41.md measures 0.8-1.9 tok/s on an
+    R9700 (best tuned three-tier case 9.1) with **61 GB of RAM**; Dankserver's
+    246 GiB is the one untested lever, so the 377.5 GiB ROCMFP23 payload is
+    downloading for a three-arm bench.
+ - `/serverAppConfigs-docs/LLM/Dankserver System Specs.md` — **RAM corrected
+   60 GiB -> 246 GiB** (stale since the 2026-09-20 board swap; the Lucebox
+   runbook had it right) and storage refreshed; `/mnt/entropy` is at 100%
+   (244 G) and must not be planned for payload.
+ - `/serverAppConfigs-docs/LLM/dankserver-frontier-moe-feasibility.md` — new
+   runbook; indexed in `LLM/index.md`.
+ - **LANDMINE, not changed here:** `/home/dank/models` was cleared by Frank
+   mid-session to free space. frank-primary is still answering `:7770` from
+   mmapped-but-unlinked inodes (`/models` is empty inside the running
+   container) — **the endpoint cannot be restarted until the 175 GiB payload
+   is re-fetched.** A reboot or a container restart takes frank-primary down.
+- Deploy-state reconciliation (Frank's call: no restart) — the autoresearch trio
+  (`f187165` PLE 1800, `41e9189` prefix caching, `ba2100e` docs) was found already
+  MERGED to main + pushed AND already live on frank-primary: the running
+  `frank-primary-r9v` container (booted 12:13 from the `ba2100e` checkout) carries
+  `--enable-prefix-caching` in argv and `VLLM_PLE_OFFLOAD_READY_TIMEOUT=1800` in env,
+  `/health` 200 — but `deploy-state/frank-primary` was still stamped `f187165`. A
+  `deploy.sh` run would have re-selected frank-primary and reloaded the 175 GiB
+  payload (~16–19 min) for zero behavioral change, on the endpoint this harness itself
+  runs on. Stamped the marker `f187165 → ba2100e` (status `deployed`, atomic
+  mktemp+mv per the deployer idiom); service untouched. e2e gate 32/33 — the lone
+  failure `11-admin` is the FrankBook-Pro Mac mini (`192.168.0.112:8989`) offline
+  (100 % ICMP loss + TCP timeout), the documented asleep-host condition, not a
+  regression.
+- `scripts/frank-primary.env`, runbook — **prefix caching DEPLOYED on
+  frank-primary** (Frank's go-ahead; commit `41e9189`): E8-qualified, zero
+  VRAM cost, 9x/63x reuse TTFT. Verified on :7770 (argv flag, health 200,
+  completion "PREFIX-CACHE-LIVE"). Gotcha re-confirmed: the container is
+  in dank's ROOTLESS podman store (`sudo podman` inspects an empty store
+  and reports nothing); probe via the shim's `docker` or the unit MainPID.
 - `/scripts/frank-primary.env`, `/serverAppConfigs-docs/LLM/frank-primary-r9v-radiance-qwen38-q8.md`, `/serverAppConfigs-docs/index.md` — **frank-primary prefill/decode bench matrix** (scratch :7773, prod stopped 00:06 for the window per fleet bench law): baseline E0 (chunk 4096 / 80-slot sync / MTP4) = PP8K 954.6, PP131K 883.0 tok/s, decode 141.9 ms/step ≈ 23.8 t/s effective. Chunk 8192 (E1R): PP8K +25.9% (1201.4) but decode −8.5% and PP131K −4.4% — short-prompt win, decode tax, not the fleet's dominant shape. Chunk 16384 boots but OOMs at request time (0.93 GiB free) — 4096 is the ceiling (engine spec-config cap + memory). Expert cache 80 slots is the policy max (doctor: capacities 48/64/80, synchronous LRU only — async unsupported, 96/112 rejected); MTP depth 6 is 3.7x SLOWER than depth 4. Prefix caching (E8/E9): zero VRAM cost, cold path unchanged, reuse hits 32k 44.2→3.6–4.3 s (9x), 131k 439→6.9 s (63x); hit pattern rounds 1–2 miss / 3+ hit — RECOMMENDED, pending Frank's go-ahead. Two failure laws recorded: unqualified shapes fail mid-request (health stays 200), and PLE registration can blow the engine's 600 s `VLLM_PLE_OFFLOAD_READY_TIMEOUT` under page-cache churn (measured 740 s) — launch.sh gained `R9V_PLE_READY_TIMEOUT` forwarding (fork `2de0357`, local-only) and prod now sets 1800.
-- **Incident — frank-primary down 00:06 → ~10:10 (9h)**: the bench window's required prod stop was never restored — the matrix + follow-ups ran 00:44 → 09:02 and the session let the endpoint sit dead (the 2026-09-18 "a bench session owes the fleet a health 200" lesson, re-offended). The first restore boot (09:46) then failed on the PLE 600 s timeout above (unit "Deactivated successfully", no journal error — the serve script exits 0 on the wait); the `R9V_PLE_READY_TIMEOUT=1800` env fix + redeploy restores it. Lesson: a multi-hour bench window needs an explicit restore task, not a "watch for DONE" — DONE ≠ prod up.
+- **Incident — frank-primary down 00:06 → ~11:00 (11h)**: the bench window's required prod stop was never restored — the matrix + follow-ups ran 00:44 → 09:02 and the session let the endpoint sit dead (the 2026-09-18 "a bench session owes the fleet a health 200" lesson, re-offended). The first restore boot (09:46) then failed on the PLE 600 s timeout above (unit "Deactivated successfully", no journal error — the serve script exits 0 on the wait). Restored by the `R9V_PLE_READY_TIMEOUT=1800` fix deploy (f187165, from main — the deploy gate refuses non-main): health 200 + MainPID = the serve script's own `podman wait` + live completion verified ~11:00. Deploy e2e gate 32/33 — the one failure (11-admin) needs the FrankBook-Pro Mac mini, which was asleep and ignored two WoL bursts (240 packets); recurring pattern per the wol-wake skill. Lesson: a multi-hour bench window needs an explicit restore task, not a "watch for DONE" — DONE ≠ prod up.
 
 ## 2026-09-27
 - `/scripts/frank-primary.env`, `/scripts/frank-primary.service`, `/scripts/serve-frank-primary.sh`, `/scripts/deploy-remote.sh`, `/tests/frank_primary_unit.bats`, `/serverAppConfigs-docs/LLM/frank-primary-r9v-radiance-qwen38-q8.md` (new), AGENTS.md + LLM index — **frank-primary cut over to the Q8_0 payload** (uncensored IQ4_XS → unsloth Qwen3.8-Flash-Next Q8_0, 175.30 GiB / 6 shards) at Frank's direction. Launcher gained the opt-in `R9V_CHAT_TEMPLATE_FILE`/`R9V_CHAT_TEMPLATE_KWARGS` passthrough (fork commit `30d4a32`, local-only) — the Qwen-Sharp v22.4.0 template now serves, fleet parity with frank-large. Env points at `/home/dank/r9v-state/q8xl/setup.json` + the quantized-inject dev-overlay adapter (fork `363d606`) + `R9V_FULL_MUTABLE_CACHE=0`; shard gates extended to SHARD4/5/6 (unset-safe); seams recalibrated; ctx window 131072 (models.yml maxTokens 36864). Spec decode is MTP4 only — the fork's `speculative_config` is a single dict, no chaining, so llama.cpp's `draft-mtp,ngram-mod` pattern is NOT expressible (PLE n-gram rides along architecturally). Uncensored state + payload remain on disk as the rollback target. 346 tests green.
@@ -135,3 +662,334 @@
 - `/dankclaw/pihole/sysctl/99-dnsmasq.conf` — Repo source of truth for `ip_unprivileged_port_start=53`; `deploy_pihole` stages conflicting drop-ins, comments out the stale sysctl.conf line, and applies `sysctl --system` (the boot code path).
 - `/tests/caddy_quadlet.bats` (new) + caddyfile/deployment_flow/pihole seams — quadlet boot persistence, self-healing, network, env, and sysctl coverage.
 - `/serverAppConfigs-docs/Infrastructure/Caddy.md`, `/serverAppConfigs-docs/Infrastructure/Pi-hole.md`, `/serverAppConfigs-docs/Dankclaw/Dankclaw Stack.md`, `/serverAppConfigs-docs/Dankclaw/Gitea.md`, `/serverAppConfigs-docs/Architecture/Deployment Flow.md` — quadlet runtime facts, the 2026-08-26 incident, the sysctl-conflict failure mode, canonical Caddy ownership.
+
+## 2026-10-01 — frank-primary RESTORED to fp8 3 x 262144 (bf16 boot fails)
+
+bf16 (--kv-cache-dtype auto) at 2 x 262144 did not stay up, with OR without
+--expert-precise-memory:
+
+- with precise: NRestarts 1->3, each attempt ~6.7 min then
+  `Deactivated successfully` (API server drains after an EngineCore fatal
+  error) — cause not captured, the serve script rm'd the container.
+- without precise (default 7.60 GiB overhead, arena 14.3 GiB): NRestarts
+  climbing, container `Exited (1)`, and the journal finally showed the real
+  error:
+  `RuntimeError: Worker failed with error 'Constraints violated
+   (L['query_start_loc'].size()[0])! You marked L['query_start_loc'].size()[0]
+   as dynamic but your code specialized it to be a constant (3)'`
+  — a torch.compile/dynamo dynamic-shape constraint violation, NOT an OOM.
+  `~/.cache/vllm` holds only `modelinfos` (16K) and each `podman run` gets a
+  fresh container filesystem, so a stale compile cache is ruled out: the
+  conflict is produced by this configuration's own compile path.
+
+Differences from the last shape measured serving (fp8 3 x 262144, MNBT 2048,
+KV 6.1e9 -> 792,069 tokens / 3.02x / arena 6,248 / 3.59 GiB free): KV dtype
+(fp8 -> bf16) and SEQS (3 -> 2). Both are candidates for the dynamo trigger;
+`--enforce-eager` is the bypass that would sidestep the constraint path.
+
+Restored: fp8 3 x 262144, MNBT 2048, KV_BYTES 6,100,000,000, no precise flag.
+Next: bisect bf16 vs SEQS in a window (bf16 + SEQS=3 isolates the dtype).
+
+## 2026-10-01 — frank-primary: --expert-precise-memory + bf16 = boot crash loop
+
+**Reverted.** The two changes Frank asked for were applied together and the
+result crash-looped: `NRestarts` climbed 1 -> 2 -> 3, each attempt running
+~6.7 minutes then logging `Deactivated successfully` (vLLM's API server drains
+and exits 0 after an EngineCore fatal error, so `podman wait` returns 0; the
+new `Restart=always` turned a silent death into a visible loop).
+
+The budget arithmetic explains it without needing the error text, because the
+arena is DEVICE-resident (the fp8 3-window shape's plan predicts 28.27 GiB and
+that is exactly what `rocm-smi` measured):
+
+| line | bf16 2 x 262144 + precise | fp8 3 x 262144 |
+|---|---|---|
+| non-expert weights | 6.28 | 6.28 |
+| pinned pool | 6.80 | 5.68 |
+| CUDA graphs | 0.55 | 0.55 |
+| arena | 16.65 (precise: overhead 5.25) | 14.31 (default: overhead 7.60) |
+| non-torch | ~1.45 | ~1.45 |
+| **total** | **31.73 of 31.86 GiB** | 28.27 (measured) |
+| free at boot | **~0.13 GiB** | 3.59 |
+
+`--expert-precise-memory` hands the 2.35 GiB compile/graph-capture headroom to
+the arena, and bf16 simultaneously adds 1.08 GiB of pinned pool; together they
+put graph capture ~0.13 GiB from the ceiling, where every transient allocation
+dies. So the flag is dropped and bf16 2 x 262144 ships with the default
+overhead: arena 14.31 GiB, ~2.4 GiB free at boot.
+
+**Lesson:** the registry's "compile headroom" is not slack to be reclaimed by
+default — it is the budget for the phase that allocates last. Reclaim it only
+by paying for it elsewhere (smaller pool, fewer sessions, or a shorter context).
+
+## 2026-10-01 — frank-primary: 2 bf16 windows + --expert-precise-memory
+
+Frank's final shape for the night. Two changes at once, both measured in
+principle before the boot:
+
+- **KV is bf16 (`--kv-cache-dtype auto`), two full-size windows (2 × 262144).**
+  bf16 measures **13,839 B/token** (from the bf16 3×241664 boot:
+  11,047,010,304 B -> 798,280 tokens), decomposing as **12,276 attention +
+  1,563 mamba/side caches** against fp8's **6,138 + 1,563 = 7,701**. So
+  524,288 tokens needs 7.256e9 B; pinned at 7,300,000,000 (~527,500 tokens,
+  3,206 margin). Cost: the pool goes 5.68 -> 6.76 GiB per rank.
+- **`--expert-precise-memory`.** The registry's default overhead for this
+  architecture is 7.60 GiB = the fitted 5.25 GiB (gfx1201 TP2
+  nbt4096/seqs16/len262144) + 2.35 GiB of compile/graph-capture headroom.
+  The flag selects the fitted figure, which moves only the plan's `room` term
+  — arena 6,248 -> ~7,278 slots (~1,025 more, at 2.344 MiB/slot) — at the
+  cost of boot-time capture slack. The overhead formula adds 0.22 GiB per
+  1024 nbt above 4096, so MNBT 2048 pays nothing.
+
+Budget note worth keeping: the arena is `(budget - non_expert_weights -
+kv_reserve - graphs - overhead) // expert_bytes`, where `kv_reserve` is the
+planner's OWN one-max-model-len reserve (2.48 GiB at TP2) and does NOT scale
+with SEQS. The pinned `--kv-cache-memory-bytes` therefore does not shrink the
+planned arena; it comes out of free VRAM instead.
+
+## 2026-10-01 — frank-large: Q4_K_XL payload + 4 x 163840 sessions
+
+Frank's call. Payload swapped from ajgazin's **abliterated** Swift-1.5
+UD-Q8_K_XL (29.3 GiB) to **unsloth/Qwen3.8-27B-GGUF UD-Q4_K_XL** (16.35 GiB,
+sha256 3f227079…b01e, vanilla/Apache-2.0 — the endpoint is no longer
+uncensored). New fetcher `scripts/download-frank-large-q4.sh`; the DFlash2
+drafter and its fetcher are unchanged.
+
+Shape: **`--sessions 4 --context 163840`** with unquantized KV. gufo has no
+KV-quantization lever at all (`execution_policy.hpp` pins `kv_cache_storage`
+to `kFp16`), so "unquantized" is inherent.
+
+KV arithmetic (why this fits where one 262144 session used to sit):
+f16 KV costs **64 KiB/token** — 16 full-attention layers (the hybrid is
+`full_attention_interval: 4` over 64 layers) x 4 KV heads x 256 head_dim x
+2 (K,V) x 2 bytes; the other 48 layers are linear-attention with a
+per-SEQUENCE fp32 state (~144 MiB), not a per-token cache. 4 x 163840 =
+655,360 tokens = 40.0 GiB of KV, and each session is exactly 10 GiB. Budget:
+weights 16.35 + KV 40.0 + drafter 1.07 + linear state 0.56 + runtime ~4 =
+~62 GiB against the 114 GiB GTT.
+
+## 2026-10-01 — e2e gate flakiness (12-crawl) — do not panic at the banner
+
+`specs/12-crawl.spec.ts:23` ("every user open link reaches its app") failed the
+deploy gate twice and then passed cleanly on a manual re-run:
+- run 1 (deploy): `Expected "plex.franksdank.cloud"` / `Received "app.plex.tv"`
+  — Plex redirects its web UI to the hosted app; the same failure appeared on
+  the previous deploy too.
+- run 2 (manual): `page.goto: net::ERR_ABORTED at https://www.franksdank.cloud/`
+  — the homepage answered `http=200` in 40 ms from the dev box at the same
+  moment, so this is a browser-level flake.
+- run 3 (manual): **33 passed**, exit 0.
+
+So a red gate on this spec alone is NOT proof the fleet is broken, and the
+deployer's `DEPLOY INCOMPLETE` banner can be raised by a flake. Re-run
+`tests/e2e/run.sh` before treating it as real; the Plex redirect is the one
+stable underlying issue and it lives in Plex's web-client/image config, not in
+any repo service.
+
+## 2026-10-01 — frank-primary bf16 no-spec cutover
+
+- `/scripts/frank-primary.env` — bf16 no-spec cutover, FINAL shape after the
+  night ladder: MAXLEN **241664** x SEQS 3, KV_BYTES **10,184,000,000**/rank,
+  KV_DTYPE auto (bf16), MNBT 1024, GATHER CHUNKS=32/LANES=64, UTIL 0.98;
+  FRANK_PRIMARY_SPEC deleted (no spec). Engine measured at the winner:
+  "KV cache size: 732,359 tokens, Maximum concurrency for 241,664 tokens per
+  request: 3.03x".
+- **The night ladder (Frank's method).** 262144 OOMed at CUDA graph capture
+  (`Tried to allocate 76.00 MiB ... 0 bytes free`, 30.43 GiB of 31.86 GiB
+  allocated by PyTorch) and 245760 OOMed identically; **237568 PASSED**
+  (KV 718,606 tokens) and the +4096 bonus step **241664 PASSED and is
+  PRODUCTION**. 241,664 is max possible on this box at util 0.98 — the
+  model's native 262,144 does not fit with a 3-seq contract. Every rung was
+  booted under a transient systemd unit at uid dank with
+  `LimitMEMLOCK=infinity` (the rootless bench probe masked the production
+  memory environment) with full engine-log capture.
+- **Util stays 0.98.** The earlier 0.90 edit was wrong — lowering
+  `--gpu-memory-utilization` shrinks the engine's own budget rather than
+  buying slack; Frank forbade it and the winning ladder ran at 0.98.
+- `/scripts/serve-frank-primary.sh` — `--speculative-config` line removed;
+  `--max-num-batched-tokens` and the `-e R4D_LRU_GATHER_CHUNKS/LANES`
+  container-env passthrough added; shape header rewritten.
+- `/tests/frank_primary_unit.bats` — seam re-pinned: bf16-KV-present,
+  spec-absent-by-measurement, gather-32x64-present, MNBT-1024 tests; the
+  qualified-shape and flags tests flipped to the new env keys.
+- Overnight redo campaign (prefetch OFF everywhere) tabled in
+  `/serverAppConfigs-docs/LLM/dankserver-qwen38-fp8-expert-offload-tuning.md`
+  with the 2026-10-01 cutover section: gather 32x64 +18.7%, MNBT flat
+  1024-4096, KV pool law (7,926 fp8 / 14,047 bf16 B/token), arena flat zone,
+  six-slot refusal, bf16 3x262144 boot failure root cause (CUDA OOM at
+  KV_BYTES 11.3e9; boots clean at 11.047e9 — captured from journald `-o json`).
+- `/AGENTS.md` — fleet table frank-primary row rewritten to the bf16 shape.
+- `/serverAppConfigs-docs/LLM/index.md` — fleet intro + mermaid node
+  updated out of the GLM era.
+- **Client contract fixed (thinking + tools).** The live endpoint returned a
+  null `reasoning` field and dumped the thinking text INLINE in `content`
+  with a dangling `</think>` (measured: `'We need answer…Brief.\n</think>\n\n391'`)
+  — the malformed thinking block clients choke on — because the deployed argv
+  had no `--reasoning-parser`. The GLM rollback (`d91379d`) had dropped all
+  three contract flags and the qwen rollback inherited the gap. Restored from
+  `ecaad36`'s values, confirmed against this fork's registries:
+  `--reasoning-parser qwen3`, `--enable-auto-tool-choice
+  --tool-call-parser qwen3_xml`, `--default-chat-template-kwargs
+  '{"enable_thinking": true, "preserve_thinking": true}'`. The seam's
+  "NO tool dialect" test is flipped to a PRESENT test.
+- **Final shape (Frank's call): 5 seqs x the full native 262144 window at
+  fp8 KV.** 6 x 241664 at fp8 died (`Tried to allocate 10.70 GiB`, 9.18 GiB
+  free) — six full-length contexts do not fit at any KV dtype — so the
+  contract moved to five FULL windows: 5 x 262144 fp8 PASSED on the first
+  rung, KV_BYTES 10,388,766,720/rank, measured **1,350,182 tokens /
+  5.15x concurrency** (39,462 tokens of margin). Seam re-pinned; the bf16
+  three-context shape (241,664, KV 732,359 / 3.03x, measured earlier the
+  same night) is the documented max at 3 contexts.
+- Harness `/home/blank/.omp/agent/models.yml` — contextWindow back to 262144
+  with the five-window fp8 shape (compaction gap 183,501 + 36,864 = 220,365
+  <= 262,144) and
+  `preserve_thinking` moved from a bare top-level body field to
+  `chat_template_kwargs` (the form vLLM actually reads).
+- **Runtime-survivability correction: five slots -> FOUR.** 5 x 262144 fp8
+  passed every boot gate (KV 1,350,182 tokens / 5.15x) but left 0 bytes free
+  at runtime; a real client request (995-token prompt, `max_tokens=261149`,
+  sampling params straight from models.yml) killed Worker_TP1 with
+  `torch.OutOfMemoryError: Tried to allocate 50.00 MiB ... 0 bytes free` and
+  the engine shut down. Boot is not proof of runtime survivability — after a
+  knife-edge shape change, send a large-`max_tokens` request before calling it
+  done. FOUR slots x 262144 fp8 (KV_BYTES 8,311,013,376/rank) returns ~1.93
+  GiB per rank for activations/graph capture. Frank's call 2026-10-01.
+- `/scripts/frank-primary.service` — **Restart=always, not on-failure** (plus
+  `StartLimitIntervalSec=0` and `OOMScoreAdjust=-500`). A runtime worker fault
+  makes vLLM's API server exit CLEAN, so ExecStart returned 0, systemd logged
+  "Deactivated successfully", and on-failure NEVER fired: the endpoint stayed
+  down until a human noticed. An explicit `systemctl stop` still wins, so
+  bench windows and deploys are unaffected.
+- **Final shape (Frank's call, 2026-10-01): 3 x 262144, pool at the FLOOR,
+  MNBT 2048, rest to the arena.** MAXLEN 262144 x SEQS 3 = 786,432 tokens of
+  contract; measured fp8 costs 7,698 B/token, so 6.054e9 B is the pool floor
+  and the deployed 6,100,000,000 B leaves a ~5.9k-token margin while handing
+  the remaining ~2.2 GiB (versus the 4-slot shape) to the expert arena — the
+  decode-critical cache the fork sizes from the remainder. MNBT 2048 costs the
+  arena nothing (the sweep measured 6,499 slots at every rung <= 4096).
+- **DCP investigated and closed** (see the parallelism survey in the runbook):
+  `--decode-context-parallel-size 2` is rejected at boot — DCP for GQA requires
+  TP > the model's KV heads (2), i.e. TP >= 4, i.e. four GPUs. Not a tuning
+  knob on this box.
+- Production bring-up: the shape boots and serves on :7770 — engine log
+  verified (KV cache size, concurrency, MNBT 1024, gather 32x64) plus live
+  contract probes (reasoning split, tool_calls). Rollback target for the
+  2026-09-29 fp8+MTP4 shape: git tag `frank-primary-fp8-mtp-20261001`.
+
+## 2026-10-01 — frank-large cutover: gufo -> vLLM TheRock, MiMo-V2.6 BF16 + cross-model MTP
+
+- **Question that started it:** can a Qwen3.5 MTP drafter attach to
+  frank-small, and how does vLLM run the FULL BF16 MiMo checkpoint? Answer 1:
+  not on frank-small — SGLang reads the MTP head from the TARGET checkpoint
+  and MiMo ships ZERO `mtp.*` tensors (base and INT8 alike). Answer 2: the
+  digest-pinned `kyuz0/vllm-therock-gfx1151` image (vLLM 0.30 / torch 2.11 /
+  TheRock ROCm 10) boots BF16 MiMo on the gfx1151 iGPU in ~200-240 s, and
+  vLLM reads the draft from a SEPARATE path — so cross-model MTP from
+  `Qwen/Qwen3.5-9B` (the distill's source, identical 4096/248320/32-layer
+  geometry) works. Frank's call: ship that as frank-large.
+- **Bench window (frank-large stopped, one session, temp 0, 28,647-token
+  prefill / 512 decode):** base 17,876 t/s prefill / 13.1 t/s decode (13.1
+  is ~94% of the LPDDR5x BF16 roof - decode here is bandwidth, not a bug);
+  +DFlash-8 7,976 / 17.4 (acceptance 59%/26% per position); +MTP-2 7,380 /
+  19.8 (71%/44%, token rate 57.4%). Deployed MTP-2: +51% decode. Both spec
+  arms pay ~55-59% of prefill. `num_speculative_tokens` beyond 2 unmeasured.
+- **Measured numerics law:** spec arms do NOT reproduce the no-spec greedy
+  text (bf16 batched verification reorders reductions; near-tie argmaxes
+  cascade). vLLM spec decode is distribution-preserving, not
+  bitwise-greedy-preserving.
+- **Client-contract delta wired:** vLLM emits `reasoning`, not
+  `reasoning_content` - `models.yml` frank-large block now sets
+  `reasoningContentField: "reasoning"`, and the extraBody moved to the
+  Qwen3.5-family recipe (0.7/0.8) with `chat_template_kwargs`
+  (a top-level `preserve_thinking` is ignored by vLLM).
+- Deployer gap found pre-deploy: the spec-draft gate demanded a single
+  `model.safetensors` (the dflash layout) and would have rejected the
+  deployed MTP arm (4 shards behind an index); fixed to accept both
+  layouts, gated by bats. `z-lab/Qwen3.5-9B-DFlash`'s "vLLM PR #40898
+  unmerged" card claim is FALSE - `vllm/v1/spec_decode/dflash.py` ships in
+  0.30 (verified in-image); the drafter stays on disk as the alternative.
+- Host state: +38 GiB fetched (MiMo BF16 17.53 GiB, Qwen3.5-9B 18 GiB,
+  DFlash 2.58 GiB), disk down to ~26 GiB free - reclaim pass due. gufo
+  rollback pinned at git tag `frank-large-gufo-q4kxl-20261001`; engine,
+  Q4_K_XL payload and DFlash2 drafter all remain on the host.
+- First deploy attempt #2 crash-looped the container 51 times: the serve
+  script handed vLLM the HOST model path while the tree is mounted at
+  `/models` inside the container — transformers rejected it as a malformed
+  repo id. Fixed by converting to the container path (the spec draft was
+  already converted; the target was not), pinned by bats. Also fixed the
+  deployer's own draft gate (same single-file bug as the serve script's).
+- Reclaim (Frank approved): Swift-1.5 Q8_K_XL payload (30 G, refetchable via
+  `scripts/download-frank-large-swift.sh`) and the orphaned flux-2-klein
+  tree under `models/checkpoints` (23 G; ComfyUI mounts
+  `comfyui-data/models`, not this) purged via
+  `~/.AGENTTRASH/20261001-frank-large-reclaim/` — 52 G freed. KEPT the
+  1.1 G `Qwen3.8-27B-DFlash2-Q4_K_M.gguf` (it is the gufo rollback's
+  drafter, not a remnant).
+
+## 2026-10-02 — frank-large cutover: vLLM MiMo -> ROCmFPX Q8_0_ROCMFPX_AGENT + MTP
+
+Frank's order was the highest quality of
+`kingjones777/Qwen3.8-27B-ROCmFP4-STRIX-MTP-GGUF` on one full context, "whatever
+runtime it requires". The answer is forced by the file: `Q8_0_ROCMFPX_AGENT.gguf`
+carries tensor type 103 (`GGML_TYPE_Q8_0_ROCMFPX`, ftype 115), which stock
+llama.cpp, gufo and every other loader on the fleet cannot read — only the
+kingjones30/ROCmFPX fork can, so the runtime is pinned to commit
+`36e9acd40e10a87cd3c3ef8ec734668757dc8520` and built by
+`scripts/build-rocmfpx.sh`. Shape: 1 session x 262,144 (native), f16 KV, the
+bundled BF16 mmproj, and MTP from the repo's own `mtp-Qwen3.8-27B-Q4_0.gguf`
+(n-max 4, `--spec-draft-device ROCm0`).
+
+- **MTP is the whole performance story: 17.0-17.1 t/s with, 7.60-7.66 without**
+  (+123 %), one session, temp 0, production stopped per arm, acceptance 0.462 /
+  mean length ~1.86, at an 8-15 % prefill tax (325-351 vs 381 t/s). The AGENT
+  routing variant was chosen over the card's plain Q8_0 because it keeps 187
+  output-head tensors at plain Q8_0 instead of 2, which measured MTP acceptance
+  0.953 vs 0.911.
+- Steady GTT occupancy 63.1 GiB of the ~114 GiB aperture; boot to healthy 16 s.
+- Rollback: git tag `frank-large-gufo-q4kxl-20261001`. The retired vLLM-era
+  fetchers moved to `old/`. Runbook
+  [[frank-large-rocmfpx-q8agent-mtp]].
+
+## 2026-10-02 — frank-small: Q5_K_M, then three contexts + vision, sized by two boots
+
+Three orders in one day on the same endpoint: `bartowski/…Q5_K_M` on llama.cpp
+with two contexts ("whatever context size fits the 3090"), then **three
+contexts** "as much as can fit while keeping overhead open space to 2-3gb on
+the 3090", then **"frank-small should have vision"**.
+
+- **Final shape: `CTX_SIZE=540672 PARALLEL=3` = 3 x 180,224**, f16 K + q8_0 V
+  under flash attention, `--mmproj` vision, all layers on the 3090. Measured
+  **21,684 MiB used / 2,441 MiB free** of 24,576 — inside the 2-3 GiB band.
+- **The pool came from two boots, not a formula.** 2 x 262,144 with no
+  projector: 20,190 MiB used, KV 12,800 => non-KV **7,390 MiB**. 3 x 196,608
+  with the projector: 22,972 used, KV 14,113 => non-KV **8,859 MiB**. The
+  difference prices the vision tower at **~1.4 GiB for a 918 MB file** — the
+  encoder's own buffers ride with it. The first 3-slot deploy (3 x 196,608)
+  booted healthy but left only 1,153 MiB free, which is how the band got
+  violated and why the seam now prices non-KV at the measured 9 GiB and
+  REJECTS that shape.
+- **Three contexts cost window, and that is the tradeoff Frank was told:**
+  180,224 tokens per conversation, not the native 262,144. A longer prompt is
+  a hard `400 exceed_context_size_error`, so `models.yml` sets
+  `contextWindow: 180224`. Consequence: frank-primary's 183,501-token
+  compaction request no longer fits here with summary room; the two-slot
+  rollback is the way back.
+- Measured (2026-10-02, this shape): decode **106.8 t/s**, prefill **3,814
+  t/s** (~5 % faster decode than 2 x 262,144 — shorter slot, shorter attention
+  scan), three concurrent requests answered in parallel in 1.7 s with a fourth
+  queued at 3.0 s, native `tool_calls`, `reasoning_content` populated,
+  `reasoning_effort: low` accepted, solid-red PNG answered **"Red."**
+- **Vision contract gotcha:** `/v1/chat/completions` ignores the legacy
+  top-level `images` field — that request answered "Brown." for the same red
+  image, i.e. it generated from text alone. Only the `image_url` content part
+  reaches the model. `/completion` accepts `images` but this build still
+  requires `messages`.
+- Deployer gates on the projector too (`LLM_SMALL_MMPROJ`), so a missing
+  mmproj cannot silently ship a blind endpoint; `MMPROJ_FILE=` (explicitly
+  empty) in the env file is the documented text-only opt-out and returns ~1.4
+  GiB to KV.
+- Reclaim staged, **not purged, awaiting approval**:
+  `~/.AGENTTRASH/20261002-retired-llm-payloads/` (~59 GiB of superseded
+  payloads). Kept in place for rollback: `MiMo-Q6K/`, `MiMo-Q5KM/`,
+  `Qwen3.8-27B-UD-Q4_K_XL/` and `Qwen3.8-27B-UD-Q8_K_XL/` (the latter holds the
+  gufo DFlash2 drafter). Runbook [[frank-small-llama-mimo-q5km]].

shared/homepage/services.json

@@ -303,8 +303,8 @@
     "section": "AI",
     "details": {
       "role": "Fast utility model",
-      "spec": "Gemma4-12B QAT (Q4_K_M, 7.4 GiB) — 1M-token pool across four conversations; fastest at images.",
+      "spec": "MiMo-V2.6-Distill-Qwen-9B (INT8 W8A16, 12 GiB) on SGLang — 606k-token pool across four 151k-token conversations; 84 tok/s decode; sees images.",
-      "inputs": ["text", "image", "audio"],
+      "inputs": ["text", "image"],
       "use": "Agentic search, RAG, research subagents, translation, roleplay, and summarization — any small task that doesn't need much intelligence."
     },
     "url": "",

tests/autoresearch_measure.bats

@@ -0,0 +1,150 @@
+#!/usr/bin/env bats
+# Seam: autoresearch measure client behavior.
+#
+# The measure client is the whole yardstick of the frank-primary speed
+# session, so its METRIC arithmetic gets a behavior test: against a
+# deterministic mock endpoint that emits a KNOWN number of completion
+# tokens at a KNOWN per-token delay, the reported decode rate must land
+# near the nominal rate, the usage accounting must come back exact, and an
+# unhealthy endpoint must fail loudly (no half-measured numbers).
+#
+# The mock implements only what the client calls: POST
+# /v1/messages/count_tokens (word count as the token count, which makes the
+# client's words/token calibration come out to exactly 1.0) and a streaming
+# POST /v1/chat/completions (one content chunk per completion token, then a
+# usage chunk, then [DONE]).
+
+load helpers
+
+MEASURE="$REPO_ROOT/scripts/autoresearch/frank-primary-measure.py"
+MOCK="$BATS_TEST_TMPDIR/mock.py"
+PORT=""
+MOCK_PID=""
+
+setup() {
+    # Deterministic mock; argv: <port> <delay-seconds>; env MOCK_FAIL=1
+    # makes the chat endpoint answer 500.
+    cat >"$MOCK" <<'PYEOF'
+import json, os, sys, time
+from http.server import BaseHTTPRequestHandler, HTTPServer
+
+PORT = int(sys.argv[1])
+DELAY = float(sys.argv[2])
+FAIL = os.environ.get("MOCK_FAIL", "0") == "1"
+
+
+class Handler(BaseHTTPRequestHandler):
+    def log_message(self, *a):
+        pass
+
+    def _send(self, obj, code=200):
+        b = json.dumps(obj).encode()
+        self.send_response(code)
+        self.send_header("Content-Type", "application/json")
+        self.send_header("Content-Length", str(len(b)))
+        self.end_headers()
+        self.wfile.write(b)
+
+    def do_POST(self):
+        n = int(self.headers.get("Content-Length", "0"))
+        body = json.loads(self.rfile.read(n) or b"{}")
+        if self.path == "/v1/messages/count_tokens":
+            text = body["messages"][0]["content"]
+            self._send({"input_tokens": max(1, len(text.split()))})
+        elif self.path == "/v1/chat/completions":
+            if FAIL:
+                self._send({"error": "engine down"}, 500)
+                return
+            max_tokens = int(body.get("max_tokens", 64))
+            self.send_response(200)
+            self.send_header("Content-Type", "text/event-stream")
+            self.end_headers()
+            for i in range(max_tokens):
+                ev = {"choices": [{"delta": {"content": "tok%d " % i}}]}
+                self.wfile.write(("data: %s\n\n" % json.dumps(ev)).encode())
+                self.wfile.flush()
+                time.sleep(DELAY)
+            prompt_words = max(1, len(body["messages"][0]["content"].split()))
+            ev = {"choices": [{"delta": {}}],
+                  "usage": {"prompt_tokens": prompt_words,
+                            "completion_tokens": max_tokens}}
+            self.wfile.write(("data: %s\n\n" % json.dumps(ev)).encode())
+            self.wfile.write(b"data: [DONE]\n\n")
+            self.wfile.flush()
+        else:
+            self._send({"error": "unknown path"}, 404)
+
+
+HTTPServer(("127.0.0.1", PORT), Handler).serve_forever()
+PYEOF
+}
+free_port() {
+    python3 -c 'import socket; s = socket.socket(); s.bind(("127.0.0.1", 0)); print(s.getsockname()[1]); s.close()'
+}
+
+start_mock() { # $1 = ok|fail
+    PORT="$(free_port)"
+    if [ "$1" = "fail" ]; then
+        MOCK_FAIL=1 python3 "$MOCK" "$PORT" 0.005 &
+    else
+        python3 "$MOCK" "$PORT" 0.005 &
+    fi
+    MOCK_PID=$!
+    local i
+    for i in $(seq 1 100); do
+        if python3 -c "import socket; socket.create_connection(('127.0.0.1', $PORT), 0.2).close()" 2>/dev/null; then
+            return 0
+        fi
+        sleep 0.1
+    done
+    echo "mock did not start" >&2
+    return 1
+}
+
+stop_mock() {
+    if [ -n "$MOCK_PID" ]; then
+        kill "$MOCK_PID" 2>/dev/null || true
+        wait "$MOCK_PID" 2>/dev/null || true
+        MOCK_PID=""
+    fi
+}
+
+@test "measure exists and compiles" {
+    [ -f "$MEASURE" ]
+    python3 -m py_compile "$MEASURE"
+}
+
+@test "measure computes decode rate and exact usage from a controlled endpoint" {
+    start_mock ok || return 1
+    run env AR_BASE="http://127.0.0.1:$PORT" AR_MODEL=mock python3 "$MEASURE"
+    stop_mock
+    [ "$status" -eq 0 ] || { echo "$output"; return 1; }
+    # run() merges stderr in; the contract is: exactly ONE json result line
+    # (the window script parses the same way), progress on stderr.
+    printf '%s\n' "$output" | python3 -c '
+import json, sys
+lines = [l for l in sys.stdin.read().splitlines() if l.startswith("{")]
+assert len(lines) == 1, "expected exactly one JSON line, got %d" % len(lines)
+d = json.loads(lines[0])
+# nominal 200 t/s (0.005 s/chunk) minus small per-chunk overhead
+for k in ("decode_tps", "decode_tps_115", "decode_tps_1056", "decode_tps_4096"):
+    v = d[k]
+    assert 60.0 <= v <= 320.0, "%s=%s outside [60, 320]" % (k, v)
+# the mock emits exactly max_tokens completion tokens; the accounting must be exact
+assert d["completion_tokens_4096"] == 256, d["completion_tokens_4096"]
+assert d["completion_tokens_115"] == 256
+# prompt sizing: calibration is exactly 1.0 words/token in the mock, so the
+# measured prompt lengths equal the targets
+assert d["prompt_tokens_115"] == 115
+assert d["prompt_tokens_4096"] == 4096
+print("measure claims verified")
+'
+}
+
+@test "measure fails loudly on an unhealthy endpoint instead of half-measuring" {
+    start_mock fail || return 1
+    run env AR_BASE="http://127.0.0.1:$PORT" AR_MODEL=mock python3 "$MEASURE"
+    stop_mock
+    [ "$status" -ne 0 ]
+    ! printf '%s\n' "$output" | grep -qE '^\{'
+}

tests/frank_large_unit.bats

@@ -1,143 +1,280 @@
 #!/usr/bin/env bats
-# Seam: the frank-large root unit on dankclaw.
+# Seam: frank-large serves kingjones777's Qwen3.8-27B Q8_0_ROCMFPX_AGENT (a
+# ftype-115 ROCmFPX file) with the repo's MTP draft head on the Strix Halo
+# iGPU, via the ROCmFPX fork of llama.cpp — one session at the native
+# 262,144 window.
 #
-# The 2026-09-21 cutover moved this service from the Kairic Edge runtime
+# This seam exists because the format is the constraint: GGML_TYPE_Q8_0_ROCMFPX
-# (ciru-ai/ROCmFPX) serving jcbtc/Qwen3.8-27B-IU4-Kairic-Edge onto the
+# (tensor type 103) is understood by the ROCmFPX fork and by nothing else on
-# stock llama.cpp prebuilt (unsloth b11030, ROCm gfx1151) serving
+# this host, so the runtime, the payload and the deployer's gate have to agree
-# unsloth/Qwen3.8-27B-GGUF:UD-Q8_K_XL with the DFlash2 drafter
+# on paths that no other endpoint uses. On 2026-10-01 a serve-script gate and
-# (z-lab/Qwen3.8-27B-DFlash2-GGUF:Q4_K_M), still as a ROOT unit: a
+# its deployer twin disagreed during a cutover and the endpoint crash-looped;
-# prebuilt runtime has no container image to install and no generated
+# both sides are pinned here.
-# unit to dodge. This seam pins what a flag list cannot show: the
+#
-# operator's one-stream / full-window shape, the payload the serve script
+# Everything is tested from what an operator or a supervisor can observe — the
-# refuses to start without, the DFlash2 spec wiring, fp8 KV, greedy
+# command line, the exit code, the deployer's error text — never from anything
-# sampling, a deploy handler that gates on the port, and the fact that
+# inside llama.cpp.
-# the sweep for a stray benchmark run cannot take frank-small down with
+setup() {
-# it.
+    load helpers
 
-load helpers
+    SERVE="$REPO_ROOT/scripts/serve-frank-large.sh"
+    UNIT="$REPO_ROOT/scripts/frank-large.service"
+    ENV_FILE="$REPO_ROOT/scripts/frank-large.env"
+    DEPLOYER="$REPO_ROOT/scripts/deploy-remote.sh"
+    BUILD="$REPO_ROOT/scripts/build-rocmfpx.sh"
+    RUNTIME_STUB="/home/dank/GitPackages/kingjones-ROCmFPX/build/bin/llama-server"
+    MODEL_STUB="/home/dank/models/kingjones-Qwen3.8-27B-ROCmFP4-STRIX-MTP/Qwen3.8-27B-Q8_0_ROCMFPX_AGENT.gguf"
+    DRAFT_STUB="/home/dank/models/kingjones-Qwen3.8-27B-ROCmFP4-STRIX-MTP/mtp-Qwen3.8-27B-Q4_0.gguf"
+    MMPROJ_STUB="/home/dank/models/kingjones-Qwen3.8-27B-ROCmFP4-STRIX-MTP/mmproj-Qwen3.8-27B-BF16.gguf"
 
-ENV_FILE="$REPO_ROOT/scripts/frank-large.env"
+    TEST_TMP="$(mktemp -d)"
-UNIT="$REPO_ROOT/scripts/frank-large.service"
+    SRC="$TEST_TMP/src"
-SERVE="$REPO_ROOT/scripts/serve-frank-large.sh"
+    mkdir -p "$SRC"
-FETCH="$REPO_ROOT/scripts/download-frank-large.sh"
+    git -C "$SRC" init -q
-REMOTE_DEPLOY="$REPO_ROOT/scripts/deploy-remote.sh"
+    git -C "$SRC" -c user.email=t -c user.name=t commit -q --allow-empty -m x
-
-@test "the frank-large unit, env and serve files exist where the deployer installs them from" {
-    [ -f "$ENV_FILE" ]
-    [ -f "$UNIT" ]
-    [ -x "$SERVE" ]
 }
 
-@test "frank-large is boot-persistent, self-healing and unprivileged" {
+teardown() {
-    grep -q '^WantedBy=multi-user.target$' "$UNIT"
+    rm -rf "$TEST_TMP"
-    grep -q '^Restart=on-failure$' "$UNIT"
-    grep -q '^RestartSec=' "$UNIT"
-    grep -q '^User=dank$' "$UNIT"
-    # The engine pins staging buffers; the memlock budget must be unbounded
-    # or the load fails on the RLIMIT.
-    grep -q '^LimitMEMLOCK=infinity$' "$UNIT"
-    # The unit runs the repo's own serve script with the repo's env.
-    grep -q 'ExecStart=/home/dank/serverAppConfigs/scripts/serve-frank-large.sh' "$UNIT"
-    grep -q 'FRANK_LARGE_ENV_FILE=/home/dank/serverAppConfigs/scripts/frank-large.env' "$UNIT"
 }
 
-@test "the operator's shape is one stream of the full 262144 window with DFlash2 depth 4" {
+# make_stub — an executable that prints its whole argv on one line, so
-    local ctx slots
+# "what command line would the supervisor run" is observable output.
-    ctx="$(sed -nE 's/^CONTEXT_SIZE=([0-9]+)$/\1/p' "$ENV_FILE")"
+make_stub() {
-    slots="$(sed -nE 's/^PARALLEL_SLOTS=([0-9]+)$/\1/p' "$ENV_FILE")"
+    cat > "$TEST_TMP/bin" <<'STUB'
-    # 262144 is the model's native window (the operator's call, 2026-09-21).
+#!/bin/sh
-    [ "$ctx" -eq 262144 ]
+printf '%s\n' "$*"
-    # One stream.
+STUB
-    [ "$slots" -eq 1 ]
+    chmod +x "$TEST_TMP/bin"
-    # DFlash2 depth 4 (the drafter's block-diffusion depth).
-    grep -q '^SPEC_DRAFT_N_MAX=4$' "$ENV_FILE"
 }
 
-@test "frank-large stays text-only" {
+@test "the env file defines the deployment values" {
-    # The new model is text-only (no vision projector). The fleet's
+    grep -qE '^LLAMA_BIN=/' "$ENV_FILE"
-    # image path is frank-small; models.yml declares frank-large text-only.
+    grep -qE '^MODEL_FILE=/' "$ENV_FILE"
-    ! grep -q -- '--mmproj' "$SERVE"
+    grep -qE '^DRAFT_FILE=/' "$ENV_FILE"
-    grep -q -- '--mmproj' "$REPO_ROOT/scripts/serve-frank-small.sh"
+    grep -qE '^ALIAS=[a-z0-9-]+$' "$ENV_FILE"
+    grep -qE '^LISTEN_PORT=[0-9]+$' "$ENV_FILE"
+    grep -qE '^CTX_SIZE=[0-9]+$' "$ENV_FILE"
 }
 
-@test "the serve script pins the stock llama.cpp runtime and refuses a missing binary" {
+@test "the serve script fails when the runtime is missing" {
-    grep -q 'unsloth-llama-b11030' "$SERVE"
+    # Only the model, the head and the projector exist; the fork was never
-    grep -q 'llama-server' "$SERVE"
+    # built. llama.cpp is not reached, so this fails fast and says what to
-    # A missing runtime or payload is a deploy error with a pointer, not a
+    # run instead of failing at bind time.
-    # restart loop.
+    touch "$TEST_TMP/model.gguf" "$TEST_TMP/draft.gguf" "$TEST_TMP/mmproj.gguf"
-    grep -q 'runtime not found' "$SERVE"
+    run env "LLAMA_BIN=$TEST_TMP/nope/llama-server" \
-    grep -q 'missing asset' "$SERVE"
+        "MODEL_FILE=$TEST_TMP/model.gguf" "DRAFT_FILE=$TEST_TMP/draft.gguf" \
-    # The Qwen3.8 template keeps prior thinking blocks across turns only
+        "MMPROJ_FILE=$TEST_TMP/mmproj.gguf" "$SERVE"
-    # when the kwarg is set.
+    [ "$status" -ne 0 ]
-    grep -q 'preserve_thinking' "$SERVE"
+    [[ "$output" == *"build it with scripts/build-rocmfpx.sh"* ]]
-    # The chat template file is wired (not just the kwarg) — the 73k-token
-    # fallback incident (2026-09-21) was a missing --chat-template-file.
-    grep -q 'chat-template-file' "$SERVE"
 }
 
-@test "the serve script wires the DFlash2 spec method and the drafter" {
+@test "the serve script refuses to start without the model file" {
-    # The new model uses DFlash2 (block-diffusion drafter) at depth 4.
+    make_stub
-    grep -q -- '--spec-type draft-dflash' "$SERVE"
+    touch "$TEST_TMP/draft.gguf" "$TEST_TMP/mmproj.gguf"
-    grep -q -- '--model-draft' "$SERVE"
+    run env "LLAMA_BIN=$TEST_TMP/bin" "MODEL_FILE=$TEST_TMP/absent.gguf" \
-    grep -q -- '--spec-draft-n-max' "$SERVE"
+        "DRAFT_FILE=$TEST_TMP/draft.gguf" "MMPROJ_FILE=$TEST_TMP/mmproj.gguf" "$SERVE"
-    # The drafter runs on the iGPU (ROCm0).
+    [ "$status" -ne 0 ]
-    grep -q -- '--spec-draft-device ROCm0' "$SERVE"
+    [[ "$output" == *"is not readable"* ]]
+    [[ "$output" == *"download-frank-large-kingjones.sh"* ]]
 }
 
-@test "the serve script uses q8_0 KV and greedy sampling" {
+@test "the serve script refuses to start without the MTP draft head" {
-    # q8_0 KV — the unsloth b11030 build does not support fp8 KV (allowed:
+    make_stub
-    # f32/f16/bf16/q8_0/q4_0/q4_1/iq4_nl/q5_0/q5_1); q8_0 is the
+    touch "$TEST_TMP/model.gguf" "$TEST_TMP/mmproj.gguf"
-    # highest-precision 8-bit option and preserves the ~8 GiB footprint.
+    run env "LLAMA_BIN=$TEST_TMP/bin" "MODEL_FILE=$TEST_TMP/model.gguf" \
-    grep -q -- '--cache-type-k q8_0' "$SERVE"
+        "DRAFT_FILE=$TEST_TMP/absent.gguf" "MMPROJ_FILE=$TEST_TMP/mmproj.gguf" "$SERVE"
-    grep -q -- '--cache-type-v q8_0' "$SERVE"
+    [ "$status" -ne 0 ]
-    # Greedy sampling (DFlash2 needs greedy verification).
+    [[ "$output" == *"MTP draft head"* ]]
-    grep -q -- '--temp 0' "$SERVE"
 }
 
-@test "the payload is fetched resumably and verified against declared sizes + sha256" {
+@test "the serve script refuses a missing mmproj unless vision is opted out" {
-    grep -q 'unsloth/Qwen3.8-27B-GGUF' "$FETCH"
+    make_stub
-    grep -q 'z-lab/Qwen3.8-27B-DFlash2-GGUF' "$FETCH"
+    touch "$TEST_TMP/model.gguf" "$TEST_TMP/draft.gguf"
-    grep -q -- '-fL -C -' "$FETCH"
+    run env "LLAMA_BIN=$TEST_TMP/bin" "MODEL_FILE=$TEST_TMP/model.gguf" \
-    # Each file's local sha256 is compared to its declared oid and a
+        "DRAFT_FILE=$TEST_TMP/draft.gguf" "MMPROJ_FILE=$TEST_TMP/absent.gguf" "$SERVE"
-    # mismatch fails the run.
+    [ "$status" -ne 0 ]
-    grep -q 'SHA256 MISMATCH' "$FETCH"
+    [[ "$output" == *"set MMPROJ_FILE='' for text-only"* ]]
+    # The documented opt-out must actually start, without a --mmproj argument.
+    run env "LLAMA_BIN=$TEST_TMP/bin" "MODEL_FILE=$TEST_TMP/model.gguf" \
+        "DRAFT_FILE=$TEST_TMP/draft.gguf" "MMPROJ_FILE=" "$SERVE"
+    [ "$status" -eq 0 ]
+    [[ "$output" != *"--mmproj"* ]]
 }
 
-@test "the fleet endpoint and alias survive the cutover" {
+@test "the serve script fails when LLAMA_BIN is empty" {
-    grep -q '^HOST=0.0.0.0$' "$ENV_FILE"
+    touch "$TEST_TMP/model.gguf" "$TEST_TMP/draft.gguf" "$TEST_TMP/mmproj.gguf"
-    grep -q '^PORT=7770$' "$ENV_FILE"
+    run env LLAMA_BIN= "MODEL_FILE=$TEST_TMP/model.gguf" \
-    grep -q '^MODEL_DIR=/home/dank/models/Qwen3.8-27B-UD-Q8_K_XL$' "$ENV_FILE"
+        "DRAFT_FILE=$TEST_TMP/draft.gguf" "MMPROJ_FILE=$TEST_TMP/mmproj.gguf" "$SERVE"
-    grep -q -- '--alias frank-large' "$SERVE"
+    [ "$status" -ne 0 ]
+    [[ "$output" == *"LLAMA_BIN is empty"* ]]
 }
 
-@test "the deploy handler retires the halogen quadlet without deleting it" {
+@test "the shipped env points at the ROCmFPX runtime and the AGENT payload" {
-    grep -q 'retire_halogen_large_quadlet' "$REMOTE_DEPLOY"
+    grep -qE "^LLAMA_BIN=$RUNTIME_STUB$" "$ENV_FILE"
-    grep -q 'HOME/.AGENTTRASH' "$REMOTE_DEPLOY"
+    grep -qE "^MODEL_FILE=$MODEL_STUB$" "$ENV_FILE"
-    ! grep -qE '(^|[^[:alnum:]_])rm (-[a-zA-Z]+ )?.*frank-large' "$REMOTE_DEPLOY"
+    grep -qE "^DRAFT_FILE=$DRAFT_STUB$" "$ENV_FILE"
+    grep -qE "^MMPROJ_FILE=$MMPROJ_STUB$" "$ENV_FILE"
+    # HIP reads bare HOST/TEMP/GPU_MAX_HEAP_SIZE out of the environment that
+    # `set -a` creates; a tunable named HOST leaks into the runtime.
+    ! grep -qE '^(HOST|TEMP|GPU_MAX_HEAP_SIZE)=' "$ENV_FILE"
 }
 
-@test "the deploy handler installs the root unit, gates the port and refuses a missing payload" {
+@test "the argv contract keeps the fleet's port and alias" {
-    grep -q 'relink_unit frank-large' "$REMOTE_DEPLOY"
+    make_stub
-    grep -q 'systemctl enable frank-large.service' "$REMOTE_DEPLOY"
+    touch "$TEST_TMP/model.gguf" "$TEST_TMP/draft.gguf" "$TEST_TMP/mmproj.gguf"
-    grep -q 'wait_port_free 7770' "$REMOTE_DEPLOY"
+    run env "LLAMA_BIN=$TEST_TMP/bin" "MODEL_FILE=$TEST_TMP/model.gguf" \
-    grep -q '127.0.0.1:7770/health' "$REMOTE_DEPLOY"
+        "DRAFT_FILE=$TEST_TMP/draft.gguf" "MMPROJ_FILE=$TEST_TMP/mmproj.gguf" \
-    grep -q 'LLM_LARGE_MODEL' "$REMOTE_DEPLOY"
+        ALIAS=frank-large BIND_ADDR=127.0.0.1 LISTEN_PORT=7770 "$SERVE"
-    grep -q 'LLM_LARGE_DRAFT' "$REMOTE_DEPLOY"
+    [ "$status" -eq 0 ]
+    # The caller's values must survive the env file, or a single-run operator
+    # override silently does nothing.
+    [[ "$output" == *"--host 127.0.0.1"* ]]
+    [[ "$output" == *"--port 7770"* ]]
+    [[ "$output" == *"--alias frank-large"* ]]
+    # Vision is wired: the projector is the whole vision tower.
+    [[ "$output" == *"--mmproj $TEST_TMP/mmproj.gguf"* ]]
 }
 
-@test "the stray-run sweep cannot take frank-small down" {
+@test "the shipped argv is one full native window on one session" {
-    # frank-small on this host is ALSO a llama-server. The sweep must match
+    grep -qE '^CTX_SIZE=262144$' "$ENV_FILE"
-    # the model path, not the binary name, or a frank-large deploy restarts
+    grep -qE '^PARALLEL=1$' "$ENV_FILE"
-    # the 3090 endpoint.
+    # 262144 IS the model's trained window (general.context_length in the
-    grep -q "pkill -f 'Qwen3.8-27B-UD-Q8_K_XL'" "$REMOTE_DEPLOY"
+    # GGUF), so 1 slot gets the whole thing — no truncation, no yarn scaling.
-    # ...and only inside the large handler: extract it and check there is no
+    #
-    # bare binary-name pkill in it.
+    # Budget, from the GGUF header: 26.28 GiB of weights + 1.57 GiB draft head
-    local large
+    # + 0.87 GiB projector + 16 full-attention layers x 4 kv heads x 256 dim x
-    large="$(sed -n '/^deploy_llm_large() {/,/^}/p' "$REMOTE_DEPLOY")"
+    # 2 (K+V) x 2 B = 64 KiB/token of f16 KV (~16 GiB at 262,144) + ~0.14 GiB
-    ! grep -q 'pkill -x llama-server' <<<"$large"
+    # of DeltaNet state + a couple GiB of compute buffers, against the iGPU's
+    # ~114 GiB GTT aperture. Change CTX_SIZE and this arithmetic, not a boot
+    # log, is what tells you it no longer fits.
+    ctx="$(sed -nE 's/^CTX_SIZE=([0-9]+)$/\1/p' "$ENV_FILE")"
+    kv_gib=$((ctx * 65536 / 1073741824))
+    total_gib=$((27 + 2 + 1 + kv_gib + 3))
+    [ "$total_gib" -le 114 ]
 }
 
-@test "the halogen quadlet is archived, not live" {
+@test "the MTP draft contract matches the model card's measured optimum" {
-    [ ! -e "$REPO_ROOT/dankclaw/frank-large/quadlet/frank-large.container" ]
+    make_stub
-    [ -f "$REPO_ROOT/dankclaw/.AGENTTRASH/frank-large.container.halogen" ]
+    touch "$TEST_TMP/model.gguf" "$TEST_TMP/draft.gguf" "$TEST_TMP/mmproj.gguf"
+    run env "LLAMA_BIN=$TEST_TMP/bin" "MODEL_FILE=$TEST_TMP/model.gguf" \
+        "DRAFT_FILE=$TEST_TMP/draft.gguf" "MMPROJ_FILE=$TEST_TMP/mmproj.gguf" "$SERVE"
+    [ "$status" -eq 0 ]
+    # The head is the model's own nextn layer, loaded as a separate draft
+    # model because this checkpoint ships it standalone (no blk.<n>.nextn.*
+    # tensors in the target file).
+    [[ "$output" == *"--spec-type draft-mtp"* ]]
+    [[ "$output" == *"--model-draft $TEST_TMP/draft.gguf"* ]]
+    # n-max 4 is the measured optimum; n-max 8 measured WORSE (acceptance
+    # fell to 0.757 and the extra verification cost outran the savings), so
+    # the flag is a measurement, not a wish.
+    [[ "$output" == *"--spec-draft-n-max 4"* ]]
+    [[ "$output" == *"--spec-draft-n-min 0"* ]]
+    [[ "$output" == *"--spec-draft-p-min 0.0"* ]]
+    # The draft runs on the iGPU beside the target: no device-to-device hop
+    # on a link that measures single-digit GB/s.
+    [[ "$output" == *"--spec-draft-device ROCm0"* ]]
+    [[ "$output" == *"--spec-draft-ngl 99"* ]]
+}
+
+@test "KV and placement stay unquantized with flash attention on" {
+    make_stub
+    touch "$TEST_TMP/model.gguf" "$TEST_TMP/draft.gguf" "$TEST_TMP/mmproj.gguf"
+    run env "LLAMA_BIN=$TEST_TMP/bin" "MODEL_FILE=$TEST_TMP/model.gguf" \
+        "DRAFT_FILE=$TEST_TMP/draft.gguf" "MMPROJ_FILE=$TEST_TMP/mmproj.gguf" "$SERVE"
+    [ "$status" -eq 0 ]
+    [[ "$output" == *"--cache-type-k f16"* ]]
+    [[ "$output" == *"--cache-type-v f16"* ]]
+    # ROCmFPx quantized KV has no kernel on this branch, and the endpoint has
+    # the aperture to keep KV unquantized anyway.
+    [[ "$output" == *"--flash-attn on"* ]]
+    # --fit's placement heuristics do not know that 311 tensors of type 103
+    # belong on the iGPU, so the whole stack is placed explicitly instead.
+    [[ "$output" == *"--fit off"* ]]
+    [[ "$output" == *"--n-gpu-layers 999"* ]]
+}
+
+@test "the client contract wires thinking and template kwargs" {
+    make_stub
+    touch "$TEST_TMP/model.gguf" "$TEST_TMP/draft.gguf" "$TEST_TMP/mmproj.gguf"
+    run env "LLAMA_BIN=$TEST_TMP/bin" "MODEL_FILE=$TEST_TMP/model.gguf" \
+        "DRAFT_FILE=$TEST_TMP/draft.gguf" "MMPROJ_FILE=$TEST_TMP/mmproj.gguf" "$SERVE"
+    [ "$status" -eq 0 ]
+    # Without --jinja, llama.cpp silently ignores the harness's
+    # chat_template_kwargs, so thinking control disappears from the API.
+    [[ "$output" == *"--jinja"* ]]
+    # The harness reads `reasoning_content` for this endpoint, which is what
+    # the deepseek emission format produces (vLLM's `reasoning` is not it).
+    [[ "$output" == *"--reasoning-format deepseek"* ]]
+    # The template needs an unbounded thinking budget.
+    [[ "$output" == *"--reasoning-budget -1"* ]]
+}
+
+@test "the deployer gates exactly the paths the serve script opens" {
+    # A gate that checks a different file than the engine opens is the
+    # 2026-10-01 crash-loop: deploy passed, the unit restart-looped. Read the
+    # engine's own paths out of the env file and require the deployer's
+    # defaults to name the same ones.
+    for pair in "LLAMA_BIN:LLM_LARGE_RUNTIME" \
+                "MODEL_FILE:LLM_LARGE_MODEL" \
+                "DRAFT_FILE:LLM_LARGE_DRAFT" \
+                "MMPROJ_FILE:LLM_LARGE_MMPROJ"; do
+        env_key="${pair%%:*}"
+        dep_key="${pair##*:}"
+        want="$(sed -nE "s|^${env_key}=(.*)\$|\1|p" "$ENV_FILE")"
+        [ -n "$want" ]
+        got="$(grep -oE "${dep_key}:-[^}]+" "$DEPLOYER" | head -1 | sed "s|^${dep_key}:-||")"
+        [ "$got" = "$want" ]
+    done
+}
+
+@test "the deployer points a missing runtime at the fork build script" {
+    grep -qE 'build it: bash scripts/build-rocmfpx\.sh' "$DEPLOYER"
+    grep -qE 'fetch it: bash scripts/download-frank-large-kingjones\.sh' "$DEPLOYER"
+}
+
+@test "the stray-process sweep cannot take frank-small down with frank-large" {
+    # frank-small serves a different GGUF over llama-server on the SAME host.
+    # A sweep keyed on the binary name (`pkill -f llama-server`) would kill
+    # the 3090 endpoint too, so it is keyed on frank-large's own payload file.
+    grep -q "pkill -f 'Qwen3.8-27B-Q8_0_ROCMFPX_AGENT'" "$DEPLOYER"
+    ! grep -qE "pkill -f 'llama-server'" "$DEPLOYER"
+}
+
+@test "the build script refuses a source tree that is not the pinned commit" {
+    # The pin is load-bearing: the older builds of this fork line segfault in
+    # ggml_compute_forward_set_rows on this exact payload. A runtime that
+    # differs from what was qualified must be heard about at deploy time.
+    wrong="$(git -C "$SRC" rev-parse HEAD)"
+    run env ROCMFPX_SRC_DIR="$SRC" ROCMFPX_COMMIT="ffff$wrong" \
+        "$BUILD" --verify
+    [ "$status" -ne 0 ]
+    [[ "$output" == *"ROCMFPX_COMMIT"*mismatch* ]]
+}
+
+@test "the build script pins a real commit and the gfx1151 target" {
+    grep -qE 'ROCMFPX_COMMIT=.*[0-9a-f]{40}' "$BUILD"
+    grep -qE 'GPU_TARGET=.*gfx1151' "$BUILD"
+    grep -qE 'DGPU_TARGETS=' "$BUILD"
+    # The webui asset fetch half-failed on this box and aborted the link at
+    # 98%; the build must not depend on it.
+    grep -qE 'DLLAMA_BUILD_WEBUI=OFF' "$BUILD"
+}
+
+@test "the deployed unit keeps the fleet's restart and boot-persistence shape" {
+    grep -qE '^Restart=(always|on-failure)$' "$UNIT"
+    grep -qE '^WantedBy=' "$UNIT"
+    grep -qE '^ExecStart=.*/serve-frank-large\.sh$' "$UNIT"
+    grep -qE '^User=dank$' "$UNIT"
+    # The payload is 26 GiB of mmap'd weights plus pinned staging buffers; an
+    # RLIMIT smaller than the pinned set fails the load in a way that looks
+    # like an engine crash.
+    grep -qE '^LimitMEMLOCK=infinity$' "$UNIT"
+}
+
+@test "the unit points at no retired frank-large runtime path" {
+    # The host keeps every retired payload and build on disk, so a stale path
+    # in a unit still resolves — to a runtime nobody has measured in weeks.
+    ! grep -qE 'GitPackages/(ROCmFPX-kairic|llama.cpp-ik_llama|llama-builds|ciru-orca)' "$UNIT"
+    ! grep -q 'halogen' "$UNIT"
 }

tests/frank_primary_unit.bats

@@ -1,17 +1,26 @@
 #!/usr/bin/env bats
 # Seam: frank-primary — the Dankserver LLM endpoint on :7770.
 #
-# This file asserts the R9V radiance-vLLM ROOT-UNIT shape (the 2026-09-27
+# This file asserts the tcclaviger RDNA4 vLLM ROOT-UNIT shape (the
-# swap that replaced the llama.cpp b11030 endpoint): the R9V fork's own
+# 2026-10-02 correction of the 2026-10-01 BF16 cutover): the serve
-# launcher (scripts/launch.sh in the pinned fork checkout) driven through a
+# script drives `podman run` DIRECTLY (the image entrypoint execs
-# docker->podman shim, with the qualified on-host setup state layered under
+# `vllm serve`), with every engine flag qualified by boot probe and carried
-# the repo's production overrides. The seam pins what the flags alone do not
+# in the repo env file.
-# say: the state-layering contract (104 tuned R9V_* keys live on-host; the
+# The seam pins what the flags alone do not say: the
-# repo env carries only production overrides), the self-healing law (the
+# KV qualification law (the night ladder: 262144/245760 OOM at graph
-# launcher detaches, so the serve script must wait on the container for
+# capture; 241664 boots at 732,359 tokens / 3.03x, util 0.98;
-# Restart=on-failure to mean anything), the memlock law that makes the
+# the bf16-KV trade (double the attention-KV bytes/token, ~14% decode cost,
-# tiered-expert pin legal, and the deployer's state/payload/image gates with
+# clean cache + 262k windows), the automatic-offload law (the fork plans
-# the false-green MainPID check.
+# the tiered-expert offload itself; no flag exists to set), the self-healing
+# law (the run returns before the engine is up, so the serve script must
+# wait on the container for Restart=on-failure to mean anything), the
+# memlock law that makes the pinned host rows legal, the deployer's
+# image/payload gates with the false-green MainPID check, and the measured
+# laws: expert prefetch ABSENT (36% precision), NO speculative decode (the
+# no-spec family measured fastest once the prefetch tax was removed; MTP
+# re-measures pending), gather geometry 32x64 PRESENT (+18.7% decode),
+# MNBT 1024 (Frank's lock), and the client contract (reasoning parser qwen3,
+# tool parser qwen3_xml, preserve_thinking on).
 
 load helpers
 
@@ -19,6 +28,7 @@ UNIT="$REPO_ROOT/scripts/frank-primary.service"
 SERVE="$REPO_ROOT/scripts/serve-frank-primary.sh"
 ENV_FILE="$REPO_ROOT/scripts/frank-primary.env"
 REMOTE_DEPLOY="$REPO_ROOT/scripts/deploy-remote.sh"
+DOWNLOAD="$REPO_ROOT/scripts/download-frank-primary.sh"
 
 @test "frank-primary unit, serve script and env exist in scripts/" {
     [ -f "$UNIT" ]
@@ -33,7 +43,22 @@ REMOTE_DEPLOY="$REPO_ROOT/scripts/deploy-remote.sh"
     # dank session's own hard limit is 8 MiB and cannot be raised without
     # restarting user@1000.service).
     grep -q '^WantedBy=multi-user.target$' "$UNIT"
-    grep -q '^Restart=on-failure$' "$UNIT"
+    # Restart=always, not on-failure: a runtime worker OOM makes vLLM's API
+    # server exit CLEAN, so ExecStart returns 0 and on-failure never fires
+    # (measured 2026-10-01 — the endpoint stayed down until a human noticed).
+    grep -q '^Restart=always$' "$UNIT"
+    ! grep -q '^Restart=on-failure$' "$UNIT"
+    # Exiting the engine takes as long as vLLM's drain: ExecStop runs
+    # `podman stop -t 120`, so the stop timeout MUST exceed 120 s or systemd
+    # kills the stop and records 'Failed with result timeout' (measured
+    # 2026-10-01). Guard the relationship, not just the literal.
+    stop_t=$(awk -F= '/^TimeoutStopSec=/{print $2}' "$UNIT")
+    [ -n "$stop_t" ] && [ "$stop_t" -gt 120 ]
+    grep -q '^ExecStop=.*podman stop -t 120' "$UNIT"
+    # StartLimitIntervalSec belongs to [Unit]; a copy in [Service] is an
+    # 'Unknown key' warning systemd logs on every load.
+    [ "$(grep -c '^StartLimitIntervalSec=' "$UNIT")" = "1" ]
+    awk '/^\[Service\]/{in_svc=1} /^StartLimitIntervalSec=/{if(in_svc) exit 1}' "$UNIT"
     grep -q '^RestartSec=' "$UNIT"
     grep -q '^User=dank$' "$UNIT"
     grep -q '^LimitMEMLOCK=infinity$' "$UNIT"
@@ -45,90 +70,254 @@ REMOTE_DEPLOY="$REPO_ROOT/scripts/deploy-remote.sh"
 }
 
 @test "the unit stops the container child explicitly" {
-    # The serve script blocks on docker wait, which cannot stop the
+    # The serve script blocks on podman wait, which cannot stop the
     # container child; systemd stop goes through ExecStop or the engine
-    # outlives the unit and holds both GPUs.
+    # outlives the unit and holds both GPUs. The container name must match
+    # the env file's identity.
     grep -q '^ExecStop=/usr/bin/podman stop' "$UNIT"
-    grep -q 'frank-primary-r9v' "$UNIT"
+    grep -q 'frank-primary-qwen' "$UNIT"
+    grep -q '^FRANK_PRIMARY_CONTAINER=frank-primary-qwen$' "$ENV_FILE"
 }
 
-@test "the serve shape is the R9V launcher, not a frozen podman run" {
+@test "the serve shape is direct podman run, not a launcher or shim" {
-    # The launcher owns the resolved container args (binds, overlays, device
+    # The tcclaviger stack is fully expressed by flags; the R9V launcher,
-    # passthrough); a frozen copy of its argv would drift on every fork
+    # docker->podman shim and setup-state layering are the RETIRED shape.
-    # update. The llama.cpp era's flags would also silently no-op here.
+    grep -q 'podman run' "$SERVE"
-    grep -q 'launch.sh' "$SERVE"
+    ! grep -q 'launch.sh' "$SERVE"
+    ! grep -q 'podman-shim' "$SERVE"
+    ! grep -q 'R9V_STATE_FILE' "$SERVE"
     ! grep -q -- '--model-draft' "$SERVE"
     ! grep -q -- '--spec-type' "$SERVE"
-    ! grep -q -- '--mmproj' "$SERVE"
-    ! grep -q -- '--override-tensor' "$SERVE"
-    ! grep -q 'podman run' "$SERVE"
 }
 
-@test "the tuned engine shape comes from the on-host setup state, not the repo" {
+@test "the qualified engine shape is in the env file" {
-    # The profile setup qualified the tuned shape (tiered-expert placement,
+    # 2026-10-01 night ladder (Frank's method), then a runtime-survivability
-    # per-rank offload budgets, MTP4, parser wiring). Copying it into
+    # correction, then Frank's 2026-10-02 decision to give the pool back to
-    # the repo would drift; the serve script layers it and skips keys the
+    # the expert arena. SEQS=3 bf16 topped out at 241664 (KV 732,359 / 3.03x);
-    # env file defines, so production overrides always win.
+    # SEQS=6 fp8 does NOT fit (6 x 241664 asked 10.70 GiB against 9.18 GiB
-    grep -q 'R9V_STATE_FILE=/home/dank/r9v-state/q8xl/setup.json' "$ENV_FILE"
+    # free); SEQS=5 fp8 x 262144 BOOTED (KV 1,350,182 / 5.15x) but left ~0
-    grep -q 'state must carry R9V_MODEL_DIR' "$SERVE"
+    # bytes free at runtime and a real client request (995-token prompt with
-    grep -q 'is not ready' "$SERVE"
+    # max_tokens=261149) killed the worker with `Tried to allocate 50.00 MiB
-    grep -q 'key not in env_keys' "$SERVE"
+    # ... 0 bytes free`. 200704 sits below the 241664 rung while leaving a
+    # full 2.95 GiB/rank to cut loose for the arena. UTIL stays 0.98
+    # (lowering it shrinks the engine's own budget instead of buying slack).
+    # FINAL (2026-10-02, Frank): TWO windows at 200704 with FP8 KV and a pool
+    # cut for the arena. fp8 costs 7,701 B/token (measured 6,138 attention +
+    # 1,563 mamba/side), so 2 x 200704 = 401,408 tokens needs 3.091e9 B;
+    # pinned at 3.15e9 (~405k tokens, ~44 MB margin). The arena is what
+    # remains — the planner sizes it AFTER the KV override — so the 6,248
+    # slots of the 6.1e9 shape become roughly 7,500 here.
+    grep -q '^FRANK_PRIMARY_MAXLEN=200704$' "$ENV_FILE"
+    grep -q '^FRANK_PRIMARY_SEQS=2$' "$ENV_FILE"
+    grep -q '^FRANK_PRIMARY_UTIL=0.98$' "$ENV_FILE"
+    grep -q '^FRANK_PRIMARY_KV_BYTES=3150000000$' "$ENV_FILE"
+    grep -q '^FRANK_PRIMARY_MNBT=1024$' "$ENV_FILE"
+    grep -q '^FRANK_PRIMARY_KV_DTYPE=fp8$' "$ENV_FILE"
+    # --expert-precise-memory is NOT set: with bf16 2 x 262144 it put the
+    # boot 0.13 GiB from the 31.86 GiB ceiling (arena 16.65 device-resident)
+    # and crash-looped at graph capture. The serve script keeps the opt-in
+    # flag; the smaller pool here is the arena it would have been paying for.
+    ! grep -q '^FRANK_PRIMARY_PRECISE_MEMORY=' "$ENV_FILE"
+    grep -q '^FRANK_PRIMARY_GATHER_CHUNKS=32$' "$ENV_FILE"
+    grep -q '^FRANK_PRIMARY_GATHER_LANES=64$' "$ENV_FILE"
+    ! grep -q '^FRANK_PRIMARY_SPEC=' "$ENV_FILE"
+    grep -q '^FRANK_PRIMARY_MODEL_SHARDS=131$' "$ENV_FILE"
 }
 
-@test "the serve script gates the pinned image, PLE bytes and GGUF shards" {
+@test "the pool is sized as the arena floor, not the context floor" {
-    # The image is answered by its OCI manifest digest (podman keeps the
+    # THE LAW this seam pins: the fork's expert planner sizes the arena
-    # manifest digest in RepoDigests, not as the image ID — the shim's ID
+    # AFTER the KV override, so every byte cut out of KV_BYTES becomes
-    # law). The PLE payload is checked at its pinned byte count
+    # arena. A context-size change must move the pool with it, or the
-    # (stat -c %s, never wc -c). A missing shard means a guaranteed
+    # reduction buys nothing. 2 x 200704 = 401,408 tokens at fp8's 7,701
-    # long-timeout boot instead of a fast failure.
+    # B/token is 3.091e9 B; the pin is 3.15e9, ~44 MB of margin above the
-    grep -q 'R9V_PLE_EXPECTED_BYTES' "$SERVE"
+    # floor — the same convention the 6.1e9 shape used.
-    grep -q 'stat -c %s' "$SERVE"
+    floor=$(awk -v n="401408" -v b="7701" 'BEGIN{printf "%.0f", n*b}')
-    grep -q 'R9V_TARGET_SHARD2_REL' "$SERVE"
+    [ "$floor" = "3091243008" ]
-    grep -q 'R9V_MMPROJ_REL' "$SERVE"
+    kv=$(grep '^FRANK_PRIMARY_KV_BYTES=' "$ENV_FILE" | cut -d= -f2)
+    [ "$kv" = "3150000000" ]
+    margin=$(( kv - floor ))
+    [ "$margin" -gt 40000000 ]
+    [ "$margin" -lt 100000000 ]
+}
+
+@test "the serve script wires the engine flags from the env" {
+    grep -q -- '--max-num-seqs "$FRANK_PRIMARY_SEQS"' "$SERVE"
+    grep -q -- '--max-model-len "$FRANK_PRIMARY_MAXLEN"' "$SERVE"
+    grep -q -- '--kv-cache-memory-bytes "$FRANK_PRIMARY_KV_BYTES"' "$SERVE"
+    grep -q -- '--kv-cache-dtype "$FRANK_PRIMARY_KV_DTYPE"' "$SERVE"
+    grep -q -- '--max-num-batched-tokens "$FRANK_PRIMARY_MNBT"' "$SERVE"
+    grep -q -- '-e R4D_LRU_GATHER_CHUNKS="$FRANK_PRIMARY_GATHER_CHUNKS"' "$SERVE"
+    grep -q -- '-e R4D_LRU_GATHER_LANES="$FRANK_PRIMARY_GATHER_LANES"' "$SERVE"
+    ! grep -q -- '--speculative-config' "$SERVE"
+    grep -q -- '--tensor-parallel-size 2' "$SERVE"
+    grep -q -- '--enable-prefix-caching' "$SERVE"
+    grep -q -- '--trust-remote-code' "$SERVE"
+    # DCP is opt-in: the flag is passed only when FRANK_PRIMARY_DCP is set
+    # (DCP reuses the TP ranks when PCP is off, so TP=2 + DCP=2 fits two GPUs).
+    grep -q -- '--decode-context-parallel-size "$FRANK_PRIMARY_DCP"' "$SERVE"
+    ! grep -q '^FRANK_PRIMARY_DCP=' "$ENV_FILE"
+    # --expert-precise-memory selects the fitted overhead (5.25 GiB) over the
+    # default (7.60 GiB = fitted + compile/graph-capture headroom), which only
+    # moves the plan's `room` term and therefore the expert arena: ~1,025
+    # more slots at 2.344 MiB each.
+    grep -q -- '${FRANK_PRIMARY_PRECISE_MEMORY:+--expert-precise-memory}' "$SERVE"
+}
+
+@test "tools and reasoning are wired — the client contract is served" {
+    # 2026-10-01: the rollback-era trio served NO tool dialect and left
+    # thinking inline in `content` with a dangling `</think>` and a null
+    # `reasoning` field (measured: content 'We need answer...Brief.\n</think>
+    # \n\n391') — the malformed thinking block clients choke on. The working
+    # values are the ones ecaad36 shipped for this model and this fork's
+    # registries confirm: reasoning parser `qwen3`, tool parser `qwen3_xml`.
+    # preserve_thinking keeps prior thinking blocks across turns.
+    grep -q '^FRANK_PRIMARY_REASONING_PARSER=qwen3$' "$ENV_FILE"
+    grep -q '^FRANK_PRIMARY_TOOL_PARSER=qwen3_xml$' "$ENV_FILE"
+    grep -q -- '--reasoning-parser "$FRANK_PRIMARY_REASONING_PARSER"' "$SERVE"
+    grep -q -- '--enable-auto-tool-choice' "$SERVE"
+    grep -q -- '--tool-call-parser "$FRANK_PRIMARY_TOOL_PARSER"' "$SERVE"
+    grep -q -- '--default-chat-template-kwargs "$FRANK_PRIMARY_CHAT_TEMPLATE_KWARGS"' "$SERVE"
+    grep -q 'preserve_thinking' "$ENV_FILE"
+    grep -q 'enable_thinking' "$ENV_FILE"
+}
+
+@test "expert offload is AUTOMATIC — the payload cannot fit VRAM and no flag sets it" {
+    # ~86 GiB/rank of FP8 trunk against 32 GiB of VRAM: the fork's tiered-
+    # expert planner moves the overflow to pinned host rows by itself (the
+    # arena 6,574 slots x 2.344 MiB plus host rows). There is no
+    # --enable-expert-offload flag in this arm; the legality of the pin is
+    # the unit's unbounded LimitMEMLOCK.
+    ! grep -q -- '--enable-expert-offload' "$SERVE"
+    ! grep -q -- '--expert-precise-memory' "$SERVE"
+    grep -q '^LimitMEMLOCK=infinity$' "$UNIT"
+    grep -q -- '--ipc=host --ulimit memlock=-1:-1' "$SERVE"
+}
+
+@test "speculative decode is ABSENT BY MEASUREMENT" {
+    # The 2026-10-01 overnight redo (prefetch OFF) re-measured the whole
+    # spec family against no-spec at matched geometry: no-spec measured
+    # 41.46/41.29/41.47 tok/s while the spec-era incumbent's 37-40 carried
+    # the prefetch tax, and MTP's 2.6 GiB/rank drafter reservation halves
+    # the pool (729,545 vs 975,342 tokens). The MTP3/4/6 untaxed re-measures
+    # never completed; until one shows a real win, no spec is the incumbent.
+    ! grep -q '^FRANK_PRIMARY_SPEC=' "$ENV_FILE"
+    ! grep -q -- '--speculative-config' "$SERVE"
+}
+
+@test "KV dtype is bf16 (auto) — exactness, paid for in pool bytes" {
+    # --kv-cache-dtype auto = bf16 (unquantized). Frank's call 2026-10-01.
+    # bf16 costs 13,839 B/token against fp8's 7,701 — decomposed as 12,276
+    # attention + 1,563 mamba/side (dtype-independent) versus fp8's 6,138 +
+    # 1,563. That halves the pool capacity per byte, which is why the contract
+    # is two full-size windows instead of three; in exchange the KV is exact
+    # (no quantization error) and measured +4% decode at shallow depth, where
+    # fp8's dequant cost dominates over the extra bytes read.
+    grep -q '^FRANK_PRIMARY_KV_DTYPE=fp8$' "$ENV_FILE"
+    grep -q -- '--kv-cache-dtype "$FRANK_PRIMARY_KV_DTYPE"' "$SERVE"
+}
+
+@test "gather geometry 32x64 is PRESENT BY MEASUREMENT" {
+    # R4D_LRU_GATHER_CHUNKS=32 / R4D_LRU_GATHER_LANES=64: the overnight
+    # sweep's biggest lever, +18.7% decode vs default geometry, and the win
+    # GROWS with depth (39.5@166 -> 54.1@1087 -> 49.3@4064 in the fp8 era).
+    # g64x128 is past the peak; g8x16 mild. They are container ENV (the
+    # fork's knobs), not engine flags — passed through with -e.
+    grep -q '^FRANK_PRIMARY_GATHER_CHUNKS=32$' "$ENV_FILE"
+    grep -q '^FRANK_PRIMARY_GATHER_LANES=64$' "$ENV_FILE"
+    grep -q -- '-e R4D_LRU_GATHER_CHUNKS="$FRANK_PRIMARY_GATHER_CHUNKS"' "$SERVE"
+}
+
+@test "MNBT is at Frank's 1024 with the arena intact" {
+    # MNBT 512 doubles TTFT at 4064 ctx (2.46 s vs 1.36 s); 1024-4096 are
+    # interchangeable on decode AND identical on arena (the sweep measured
+    # 6,499 slots at every rung <=4096); 12288 costs 2.5% plus arena.
+    # 1024 is the 2026-10-02 setting — the pool was cut to feed the expert
+    # arena, so the workspace floor matters more than it did at 2048: the
+    # arena budget is what is left, and 1024-4096 pay nothing for it.
+    grep -q '^FRANK_PRIMARY_MNBT=1024$' "$ENV_FILE"
+    grep -q -- '--max-num-batched-tokens "$FRANK_PRIMARY_MNBT"' "$SERVE"
+}
+
+@test "expert prefetch is absent BY LAW, not by omission" {
+    # Telemetry (2026-09-29): the prefetcher served 13,369 of 36,720
+    # prefetched slabs (36% precision) — roughly half the gather traffic was
+    # waste, and the "savings" would have been installs anyway. Deleting the
+    # flag moved decode 26.8 -> 40.6 tok/s. Any future flag must come with a
+    # fresh A/B, so the serve script must NOT wire it unconditionally.
+    ! grep -q -- '--expert-prefetch' "$SERVE"
+    ! grep -q 'expert_prefetch' "$ENV_FILE"
+    grep -q 'PREFETCH IS ABSENT BY LAW' "$ENV_FILE"
+}
+
+@test "TunableOp stays off" {
+    # CLAV_TUNABLEOP_SWEEP silently skips under expert offload unless
+    # CLAV_TUNABLEOP_WITH_OFFLOAD=1, and forcing it corrupts numerics
+    # (repetition loops, early halts, invalid measurements).
+    ! grep -q 'TUNABLEOP' "$ENV_FILE"
+    ! grep -q 'TUNABLEOP' "$SERVE"
+}
+
+@test "the serve script gates the digest-pinned image and the payload" {
+    # The image is pinned by OCI manifest digest (the floating :latest moved
+    # once already). The payload gate checks config.json and the exact shard
+    # count from the env — a missing shard means a guaranteed long-timeout
+    # boot instead of a fast failure.
+    grep -q 'FRANK_PRIMARY_IMAGE' "$SERVE"
     grep -q 'RepoDigests' "$SERVE"
+    grep -q 'FRANK_PRIMARY_MODEL_SHARDS' "$SERVE"
+    grep -q 'config.json' "$SERVE"
     grep -q '/dev/kfd' "$SERVE"
+    grep -q -- '--device /dev/dri' "$SERVE"
+    ! grep -q 'tcclaviger/vllm:latest' "$ENV_FILE"
 }
 
-@test "the serve script clears the stale container and self-heals via docker wait" {
+@test "the serve script clears stale containers and self-heals via podman wait" {
-    # launch.sh detaches and refuses an existing container name; a crashed
+    # A crashed unit can leave the container name registered forever after;
-    # unit can leave the name registered forever after. Clear our own name
+    # the legacy identities (the retired R9V container and the ad hoc
-    # first (idempotent), then block on the container so a crash is a
+    # experiment container) hold the same GPUs and die here too. Then block
-    # nonzero ExecStart.
+    # on the container so a crash is a nonzero ExecStart.
-    grep -q 'docker rm -f "$R9V_CONTAINER_NAME"' "$SERVE"
+    grep -q 'FRANK_PRIMARY_LEGACY_CONTAINERS' "$ENV_FILE"
-    grep -q 'exec docker wait "$R9V_CONTAINER_NAME"' "$SERVE"
+    grep -q 'frank-primary-r9v' "$ENV_FILE"
+    grep -q 'qwen-fp8' "$ENV_FILE"
+    grep -q 'exec podman wait "$FRANK_PRIMARY_CONTAINER"' "$SERVE"
 }
 
 @test "frank-primary holds the single fleet port under its own alias" {
-    # The scratch identity (r9v-qwen38-flash-next on 127.0.0.1:7772) must
+    # The scratch identity (qwen3.8-flash-next on :7773) must never leak
-    # never leak into production: LAN bind, :7770, the harness's model id.
+    # into production: LAN publish of :7770, the harness's model id.
-    grep -q '^R9V_HOST_BIND=0.0.0.0$' "$ENV_FILE"
+    grep -q '^FRANK_PRIMARY_PUBLISH=7770:7770$' "$ENV_FILE"
-    grep -q '^R9V_HOST_PORT=7770$' "$ENV_FILE"
+    grep -q '^FRANK_PRIMARY_SERVED_NAME=frank-primary$' "$ENV_FILE"
-    grep -q '^R9V_SERVED_MODEL_NAME=frank-primary$' "$ENV_FILE"
+}
-    grep -q '^R9V_CONTAINER_NAME=frank-primary-r9v$' "$ENV_FILE"
+
-    grep -q '^R9V_EXPECTED_GPU_BDFS=0000:03:00.0,0000:07:00.0$' "$ENV_FILE"
+@test "the model id matches the payload's served alias wiring" {
+    # The payload dir and its shard count are the fetch script's output
+    # layout; the fetch script must write where the env points.
+    grep -q '^FRANK_PRIMARY_MODEL_DIR=/home/dank/models/Qwen3.8-Flash-Next-Uncensored-FP8$' "$ENV_FILE"
+    grep -q '^FRANK_PRIMARY_MODEL_SHARDS=131$' "$ENV_FILE"
+    grep -q 'Qwen3.8-Flash-Next-Uncensored-FP8' "$DOWNLOAD"
+    grep -q 'orcarouter/Qwen3.8-Flash-Next-Uncensored-FP8' "$DOWNLOAD"
 }
 
 @test "no bare HIP-reserved env var is defined" {
     # HIP reads bare TEMP for its JIT cache path (2026-08-13: TEMP=1.0
     # segfaulted every load), and HOST/GPU_MAX_HEAP_SIZE are the same
-    # class. The bind address is R9V_HOST_BIND, not HOST.
+    # class. Every key here is FRANK_PRIMARY_-prefixed.
     ! grep -qE '^(TEMP|HOST|GPU_MAX_HEAP_SIZE)=' "$ENV_FILE"
     ! grep -qE '^(export )?(TEMP|HOST|GPU_MAX_HEAP_SIZE)=' "$SERVE"
 }
 
-@test "the deployer gates the setup state, image digest, PLE bytes and shards" {
+@test "the deployer gates the image digest and payload, then health-checks" {
-    # The deployer refuses when the on-host setup state is missing or not
+    # The deployer refuses when the pinned image is absent from rootless
-    # ready, when the pinned image is absent from rootless storage, when
+    # storage or the payload is missing/wrong-sharded — then clears stale
-    # the PLE payload is off its pinned byte count, or when any GGUF shard
+    # containers (including the legacy identities), waits out the port,
-    # or the mmproj is missing — then clears stale containers, waits out
+    # relinks/enables/restarts the root unit and health-gates it.
-    # the port, relinks/enables/restarts the root unit and health-gates it.
+    grep -q 'LLM_PRIMARY_CONTAINER=frank-primary-qwen' "$REMOTE_DEPLOY"
-    grep -q 'LLM_PRIMARY_STATE=/home/dank/r9v-state/q8xl/setup.json' "$REMOTE_DEPLOY"
+    grep -q 'FRANK_PRIMARY_IMAGE' "$REMOTE_DEPLOY"
-    grep -q 'LLM_PRIMARY_CONTAINER=frank-primary-r9v' "$REMOTE_DEPLOY"
-    grep -q 'is not ready' "$REMOTE_DEPLOY"
     grep -q "podman images --format '{{.RepoDigests}}'" "$REMOTE_DEPLOY"
-    grep -q 'stat -c %s' "$REMOTE_DEPLOY"
+    grep -q 'FRANK_PRIMARY_MODEL_SHARDS' "$REMOTE_DEPLOY"
-    grep -q 'R9V_TARGET_SHARD2_REL' "$REMOTE_DEPLOY"
+    grep -q 'config.json' "$REMOTE_DEPLOY"
-    grep -q 'podman rm -f "$LLM_PRIMARY_CONTAINER"' "$REMOTE_DEPLOY"
+    grep -q 'qwen-fp8' "$REMOTE_DEPLOY"
+    grep -q 'glm-probe' "$REMOTE_DEPLOY"
     grep -q 'wait_port_free 7770' "$REMOTE_DEPLOY"
     grep -q 'relink_unit frank-primary' "$REMOTE_DEPLOY"
     grep -q 'systemctl enable frank-primary.service' "$REMOTE_DEPLOY"

tests/frank_small_unit.bats

@@ -1,12 +1,13 @@
 #!/usr/bin/env bats
-# Seam: the frank-small mainline-llama.cpp root unit on dankclaw.
+# Seam: the frank-small llama.cpp root unit on dankclaw.
-# frank-small is the RTX 3090 endpoint created on 2026-09-16, when frank-large
+# frank-small is the RTX 3090 endpoint; since 2026-10-01 night it serves
-# went back to the halogen engine on the iGPU. This seam pins what the flags
+# bartowski/MiMo-V2.6-Distill-Qwen-9B-GGUF:Q5_K_M via a CUDA (sm_86) build of
-# alone do not say: boot persistence for a NATIVE runtime (no generated quadlet
+# llama.cpp master, chosen for the ASYMMETRIC KV Frank asked for: K f16
-# is involved), the repo-relative paths the unit executes, the separate client
+# (unquantized) + V q8_0, which only llama.cpp exposes per-tensor. Since
-# port, the MTP contract (the head is a separate sidecar file loaded as the
+# 2026-10-02 it runs THREE contexts and carries a vision projector, and the
-# draft model, chained behind ngram-mod), the vision handoff from frank-large,
+# card must keep 2-3 GiB free. This seam runs the serve script against a stub
-# and the KV budget: q8_0 quantized KV whose context --fit sizes to the card.
+# engine and asserts on the argv it actually execs, so what is pinned is the
+# command the operator gets.
 
 load helpers
 
@@ -15,6 +16,52 @@ SERVE="$REPO_ROOT/scripts/serve-frank-small.sh"
 ENV_FILE="$REPO_ROOT/scripts/frank-small.env"
 REMOTE_DEPLOY="$REPO_ROOT/scripts/deploy-remote.sh"
 ENV_LARGE="$REPO_ROOT/scripts/frank-large.env"
+TARGET_STUB="MiMo-V2.6-Distill-Qwen-9B-Q5_K_M.gguf"
+# Same basename as the shipped projector: the test env repoints it into $STAGE.
+MM_STUB="mmproj-MiMo-V2.6-Distill-Qwen-9B-f16.gguf"
+
+setup() {
+    STAGE="$(mktemp -d)"
+    RUNTIME_DIR="$STAGE/llama"
+    MODEL_DIR="$STAGE/models"
+    mkdir -p "$RUNTIME_DIR/bin" "$MODEL_DIR"
+    cat > "$RUNTIME_DIR/bin/llama-server" <<'STUB'
+#!/usr/bin/env bash
+printf '%s\n' "$@"
+STUB
+    chmod +x "$RUNTIME_DIR/bin/llama-server" "$RUNTIME_DIR/bin" 2>/dev/null || chmod +x "$RUNTIME_DIR/bin/llama-server"
+    touch "$MODEL_DIR/$TARGET_STUB" "$MODEL_DIR/$MM_STUB"
+}
+
+teardown() {
+    rm -rf "$STAGE"
+}
+
+# serve_argv [env-file] — run the serve script and capture the engine's
+# argv, one token per line, in $output.
+serve_argv() {
+    local envf="${1:-$STAGE/test.env}"
+    local mmproj
+    mmproj="$(sed -nE 's/^MMPROJ_FILE=(.+)$/\1/p' "$ENV_FILE")"
+    cat > "$envf" <<EOF
+LLAMA_BIN=$RUNTIME_DIR/bin/llama-server
+MODEL_FILE=$MODEL_DIR/$TARGET_STUB
+MMPROJ_FILE=$MODEL_DIR/${mmproj##*/}
+ALIAS=frank-small
+BIND_ADDR=0.0.0.0
+LISTEN_PORT=7771
+CTX_SIZE=$(sed -nE 's/^CTX_SIZE=([0-9]+)$/\1/p' "$ENV_FILE")
+PARALLEL=$(sed -nE 's/^PARALLEL=([0-9]+)$/\1/p' "$ENV_FILE")
+CACHE_TYPE_K=$(sed -nE 's/^CACHE_TYPE_K=(.+)$/\1/p' "$ENV_FILE")
+CACHE_TYPE_V=$(sed -nE 's/^CACHE_TYPE_V=(.+)$/\1/p' "$ENV_FILE")
+NGL=$(sed -nE 's/^NGL=([0-9]+)$/\1/p' "$ENV_FILE")
+EOF
+    run env FRANK_SMALL_ENV_FILE="$envf" nvidia-modprobe=true bash "$SERVE"
+}
+
+argv_value() {
+    grep -A1 -xF -- "$1" <<<"$output" | tail -1
+}
 
 @test "frank-small unit, serve script and env exist in scripts/" {
     [ -f "$UNIT" ]
@@ -28,94 +75,140 @@ ENV_LARGE="$REPO_ROOT/scripts/frank-large.env"
     grep -q '^RestartSec=' "$UNIT"
     grep -q '^User=dank$' "$UNIT"
     grep -q '^LimitMEMLOCK=infinity$' "$UNIT"
-}
-
-@test "frank-small runs the repo's own serve script with the repo's env" {
     grep -q '^ExecStart=/home/dank/serverAppConfigs/scripts/serve-frank-small.sh$' "$UNIT"
     grep -q 'FRANK_SMALL_ENV_FILE=/home/dank/serverAppConfigs/scripts/frank-small.env' "$UNIT"
 }
 
-@test "frank-small offloads every layer to the 3090 with flash attention" {
+@test "the shipped env points at the llama.cpp runtime and the Q5_K_M GGUF" {
-    grep -q 'llama.cpp-mainline/build-cuda/bin/llama-server' "$ENV_FILE"
+    grep -qE '^LLAMA_BIN=/home/dank/GitPackages/llama-master/build-cuda/bin/llama-server$' "$ENV_FILE"
-    grep -q '^NGL=999$' "$ENV_FILE"
+    grep -qE "^MODEL_FILE=/home/dank/models/MiMo-Q5KM/$TARGET_STUB$" "$ENV_FILE"
-    grep -q -- '--gpu-layers "$NGL"' "$SERVE"
-    grep -q '^FLASH_ATTN=on$' "$ENV_FILE"
-    grep -q -- '--flash-attn "$FLASH_ATTN"' "$SERVE"
-    # Prometheus scrapes this endpoint; without --metrics there is no /metrics.
-    grep -q -- '--metrics' "$SERVE"
 }
 
-@test "frank-small caps its CPU RAM cache below the build default" {
+@test "the deployer gates on exactly the payload paths the env file ships" {
-    # Without the flag the 8192 MiB default returns and the env cap is a
+    # Anti-drift: the deploy handler checks files by its own defaults. If
-    # silent no-op; this memory-tight host needs the cap to hold.
+    # those ever differ from what the unit actually loads, a green deploy
-    grep -q -- '--cache-ram "$CACHE_RAM"' "$SERVE"
+    # would be gating on a file nobody reads — and a missing projector would
+    # ship a blind endpoint.
+    local env_v dep_v path pair
+    for pair in "MODEL_FILE LLM_SMALL_MODEL_FILE" "MMPROJ_FILE LLM_SMALL_MMPROJ"; do
+        read -r env_v dep_v <<<"$pair"
+        path="$(sed -nE "s/^${env_v}=(.+)\$/\1/p" "$ENV_FILE" | tail -1)"
+        [ -n "$path" ]
+        grep -qF -- "${dep_v}=\"\${${dep_v}:-${path}}\"" "$REMOTE_DEPLOY"
+    done
 }
 
-@test "frank-small drafts with the MTP sidecar chained behind ngram-mod" {
+@test "the shipped env serves the fleet shape: three 180224-token contexts" {
-    grep -q '^SPEC_TYPE=draft-mtp,ngram-mod$' "$ENV_FILE"
+    serve_argv
-    grep -q -- '--spec-type "$SPEC_TYPE"' "$SERVE"
+    [ "$status" -eq 0 ]
-    # The Gemma4 MTP head is a separate assistant file (gemma4-assistant, 4
+    [ "$(argv_value --alias)" = "frank-small" ]
-    # nextn layers); it loads as --model-draft and shares the target's last
+    [ "$(argv_value --port)" = "7771" ]
-    # layers' KV. ngram-mod is draftless, so the chain is legal.
+    [ "$(argv_value --host)" = "0.0.0.0" ]
-    grep -q '^DRAFT_FILE=mtp-gemma-4-12B-it\.gguf$' "$ENV_FILE"
+    [ "$(argv_value --model)" = "$MODEL_DIR/$TARGET_STUB" ]
-    grep -q -- '--model-draft "$MODEL_DIR/$DRAFT_FILE"' "$SERVE"
+    [ "$(argv_value --n-gpu-layers)" -gt 0 ]
+    # llama.cpp divides --ctx-size across slots and a mamba hybrid cannot
+    # share one cache, so the observable contract is the per-slot window.
+    [ "$(argv_value --parallel)" -eq 3 ]
+    ctx="$(argv_value --ctx-size)"
+    par="$(argv_value --parallel)"
+    [ $((ctx % par)) -eq 0 ]
+    [ $((ctx / par)) -eq 180224 ]
+    [ $((ctx / par)) -le 262144 ]
 }
 
-@test "frank-small carries the fleet's image path, and only it" {
+@test "the KV contract is asymmetric: unquantized K, q8_0 V, flash-attn on" {
-    grep -q '^MMPROJ_PATH=.*mmproj-Gemma4-12B-QAT-Uncensored-HauhauCS-Balanced-BF16\.gguf$' "$ENV_FILE"
+    serve_argv
-    grep -q -- '--mmproj "$MMPROJ_PATH"' "$SERVE"
+    [ "$status" -eq 0 ]
-    ! grep -q '^ENABLE_VISION=1$' "$ENV_LARGE"
+    [ "$(argv_value --cache-type-k)" = "f16" ]
+    [ "$(argv_value --cache-type-v)" = "q8_0" ]
+    # q8_0 on V only exists on the flash-attention path; without this flag
+    # pair the KV contract in the env file is a lie the engine never sees.
+    [ "$(argv_value --flash-attn)" = "on" ]
 }
 
-@test "frank-small's slot grid is full-window and fits the 24 GiB card" {
+@test "the KV budget leaves 2-3 GiB of the 3090 free" {
-    local parallel cache_k cache_v ctx fixed per_slot
+    # Rate: K f16 (16 KiB/token) + V q8_0 (~8.5, budgeted 25 with slack).
-    parallel="$(sed -nE 's/^PARALLEL=([0-9]+)$/\1/p' "$ENV_FILE")"
+    # The non-KV half is MEASURED on this card on 2026-10-02: with the
-    cache_k="$(sed -nE 's/^CACHE_TYPE_K=(.+)$/\1/p' "$ENV_FILE")"
+    # projector loaded the engine reported 22,972 MiB used against 14,113
-    cache_v="$(sed -nE 's/^CACHE_TYPE_V=(.+)$/\1/p' "$ENV_FILE")"
+    # MiB of KV, so weights + vision tower + DeltaNet state + compute
+    # buffers + CUDA graphs = 8,859 MiB => 9 GiB. Two asserts, both about
+    # the output the operator sees: the pool must still fit with 2 GiB
+    # spare, and it must be large enough that 5 GiB spare is NOT reachable
+    # (an undersized pool is as wrong as an oversized one). Arithmetic, not
+    # a boot log: change CTX_SIZE and this bites before the deploy does —
+    # it rejects the 3 x 196,608 pool that measured only 1.1 GiB free. The
+    # same pool with f16 V (34 KiB/token) lands past the card, which is why
+    # the contract is asymmetric.
     ctx="$(sed -nE 's/^CTX_SIZE=([0-9]+)$/\1/p' "$ENV_FILE")"
-    [ -n "$parallel" ] && [ -n "$cache_k" ] && [ -n "$cache_v" ] && [ -n "$ctx" ]
+    kv_gib=$((ctx * 25 / 1048576))
-    # Every slot must be the model's FULL trained window (262144), not a
+    [ $((kv_gib + 9 + 2)) -le 24 ]
-    # divided one -- CTX_SIZE is the total pool and --parallel divides it.
+    [ $((kv_gib + 9 + 5)) -gt 24 ]
-    [ $((ctx / parallel)) -eq 262144 ]
+    f16v_gib=$((ctx * 34 / 1048576))
-    [ $((ctx % parallel)) -eq 0 ]
+    [ $((f16v_gib + 9)) -gt 24 ]
-    [ "$cache_k" = "q8_0" ] && [ "$cache_v" = "q8_0" ]
-    # Fit budget from the measured boot arithmetic on the 3090 (24576
-    # MiB): one full-context slot idles at 11,996 MiB, four at 22,346,
-    # so each slot costs (22346-11996)/3 = 3450 MiB and the fixed part
-    # (weights, projector, sidecar draft, compute buffers, CUDA context)
-    # is 11996-3450 = 8546 MiB. If the config asks for more slots than
-    # the card holds, this inequality fails before deploy instead of the
-    # KV allocation failing at boot.
-    fixed=8546
-    per_slot=3450
-    [ $((fixed + parallel * per_slot)) -le 24576 ]
-    # Quantized KV on CUDA needs the GGML_CUDA_FA_ALL_QUANTS build option,
-    # without which prefill collapses about 12x (see the CUDA lesson in
-    # AGENTS.md). The build-cuda tree must carry it.
-    grep -q '^FLASH_ATTN=on$' "$ENV_FILE"
 }
 
-@test "frank-small runs on its own port beside frank-large's" {
+@test "no speculative draft ships by default" {
-    grep -q '^LISTEN_PORT=7771$' "$ENV_FILE"
+    # The qwen35 GGUF has no MTP tensors, and the cross-model DFlash draft
-    grep -q '^ALIAS=frank-small$' "$ENV_FILE"
+    # is a measured decision the harness makes later, not a default.
-    grep -q -- '--port "$LISTEN_PORT"' "$SERVE"
+    ! grep -q -- '--model-draft' "$SERVE"
-    grep -q -- '--host "$BIND_ADDR"' "$SERVE"
+    ! grep -q -- '--spec-type' "$SERVE"
-    # The two endpoints cannot share a port: frank-large holds 7770, this one
+    ! grep -q '^DRAFT' "$ENV_FILE"
-    # 7771.
-    grep -q '^PORT=7770$' "$ENV_LARGE"
-    ! grep -q '^PORT=7771$' "$ENV_LARGE"
 }
 
-@test "the deploy handler installs the root unit and refuses a missing model" {
+@test "vision ships by default and MMPROJ_FILE empty opts out of the image path" {
+    # The Q5_K_M GGUF carries no `v.*` tensors, so the projector argument IS
+    # the vision capability: no flag, no eyes.
+    serve_argv
+    [ "$status" -eq 0 ]
+    [ "$(argv_value --mmproj)" = "$MODEL_DIR/$MM_STUB" ]
+
+    # The documented opt-out must reach the engine as ABSENCE of the flag,
+    # not as an argument pointing at an empty string.
+    sed -i 's|^MMPROJ_FILE=.*|MMPROJ_FILE=|' "$STAGE/test.env"
+    run env FRANK_SMALL_ENV_FILE="$STAGE/test.env" nvidia-modprobe=true bash "$SERVE"
+    [ "$status" -eq 0 ]
+    ! grep -qxF -- '--mmproj' <<<"$output"
+}
+
+@test "the fleet reasoning contract survives the reslice to three contexts" {
+    serve_argv
+    [ "$status" -eq 0 ]
+    # deepseek format = thinking lands in reasoning_content, the field the
+    # harness reads on this endpoint; --jinja is what makes
+    # chat_template_kwargs reach the template at all.
+    [ "$(argv_value --reasoning-format)" = "deepseek" ]
+    argv_value --jinja >/dev/null
+    grep -qxF -- '--jinja' <<<"$output"
+}
+
+@test "the serve script refuses a missing runtime, GGUF or projector" {
+    rm -f "$MODEL_DIR/$MM_STUB"
+    serve_argv
+    [ "$status" -ne 0 ]
+    [[ "$output" == *"frank-small: missing"* ]]
+
+    rm -f "$RUNTIME_DIR/bin/llama-server"
+    serve_argv
+    [ "$status" -ne 0 ]
+    [[ "$output" == *"frank-small: missing"* ]]
+}
+
+@test "the deploy handler gates on binary, GGUF and projector, then MainPID-hardens health" {
     grep -q 'deploy_llm_small' "$REMOTE_DEPLOY"
+    grep -q 'LLM_SMALL_RUNTIME' "$REMOTE_DEPLOY"
+    grep -q 'LLM_SMALL_MODEL_FILE' "$REMOTE_DEPLOY"
+    grep -q 'LLM_SMALL_MMPROJ' "$REMOTE_DEPLOY"
     grep -q 'relink_unit frank-small' "$REMOTE_DEPLOY"
     grep -q 'systemctl enable frank-small.service' "$REMOTE_DEPLOY"
-    grep -q 'systemctl restart frank-small.service' "$REMOTE_DEPLOY"
-    grep -q 'Gemma4-12B-QAT' "$REMOTE_DEPLOY"
-    grep -q 'refusing to start a unit that cannot load' "$REMOTE_DEPLOY"
     grep -q '127.0.0.1:7771/health' "$REMOTE_DEPLOY"
+    grep -q 'frank-small.service has no MainPID' "$REMOTE_DEPLOY"
 }
 
-@test "frank-small's serve script refuses to start on a half-finished download" {
+@test "the two endpoints keep separate ports and the large handler never pkills llama-server" {
-    grep -q 'frank-small: missing' "$SERVE"
+    # frank-large's gufo sweep must not take down frank-small, whose
+    # process is llama-server on the SAME host since this swap.
+    grep -q '^LISTEN_PORT=7771$' "$ENV_FILE"
+    local large
+    large="$(sed -n '/^deploy_llm_large() {/,/^}/p' "$REMOTE_DEPLOY")"
+    ! grep -q 'pkill -x llama-server' <<<"$large"
 }