frank-primary: 2 x 200704 fp8, pool cut for the expert arena, MNBT 1024

The 2026-10-01 bf16 shape (2 x 262144, --expert-precise-memory)
crash-looped at graph capture — the arena is DEVICE-resident and that
shape had 0.13 GiB to spare. The plain bf16 2 x 262144 also did not stay
up: 0 bytes free at runtime, and a real client request (995-token prompt,
max_tokens=261149) killed the worker with .

The correction is a pool cut, not a downgrade. FP8 KV (7,701 B/token),
two 200704 windows (401,408 tokens -> 3.091e9 B floor, pinned 3.15e9),
MNBT 1024. The fork's expert planner sizes the arena AFTER the KV
override, so the ~2.95 GiB/rank reclaimed becomes arena: ~7,500 slots
vs 6,248. Two contexts at 200704 is a capacity contract, not a
throughput one (2-way concurrency summed BELOW parity at 4064 ctx), so
the slot buys the VRAM headroom long generations need.

Refs [[dankserver-qwen38-fp8-expert-offload-tuning]].

- scripts/frank-primary.env: MAXLEN 200704, SEQS 2, KV_BYTES 3.15e9,
  MNBT 1024, rewritten shape comment
- scripts/serve-frank-primary.sh: header comment updated
- tests/frank_primary_unit.bats: assertions updated, new seam pins the
  arena-floor pool law (the pool must move with a context change or the
  reduction buys nothing)
- docs + AGENTS.md: fleet topology and deployed shape corrected

AGENTS.md

@@ -12,8 +12,8 @@ merge absorbed the former `vllmScripts` repo, history preserved).
   Calibre-Web, Jellyfin, Plex) plus the `frank-primary` LLM service.
 - **Dankclaw** (192.168.0.143) — AI/search/dev/observability stack (SearXNG
   via Gluetun, ComfyUI, OpenWebUI, Hermes, Gitea, Grafana, Prometheus, Loki,
-  Alloy, Pi-hole) plus TWO LLM services: `frank-large` (stock llama.cpp on
+  Alloy, Pi-hole) plus TWO LLM services: `frank-large` (ROCmFPX llama.cpp
-  the gfx1151 iGPU) and `frank-small` (SGLang on the RTX 3090). Caddy on
+  fork on the gfx1151 iGPU) and `frank-small` (llama.cpp CUDA on the RTX 3090). Caddy on
   Dankclaw reverse-proxies both hosts.
 - **FrankBook-Pro** (192.168.0.112) — macOS mini running the *arr stack
   (Prowlarr, Sonarr, Radarr, Lidarr, Readarr, qBittorrent) as quadlets inside
@@ -24,9 +24,9 @@ merge absorbed the former `vllmScripts` repo, history preserved).
 
 | Target | Service | Runtime | Model | Port |
 |--------|---------|---------|-------|------|
-| Dankserver | `frank-primary` | tcclaviger RDNA4 vLLM fork (`docker.io/tcclaviger/vllm@sha256:da4029666…`, digest-pinned, plain `podman run`, root systemd unit) | `orcarouter/Qwen3.8-Flash-Next-Uncensored-FP8` (~172.8 GiB, 131 shards), TP=2, expert offload automatic (fork plans it), prefetch OFF, **bf16 KV** (`--kv-cache-dtype auto`), **no speculative decode**, MNBT 2048, MoE gather 32×64, **2 seqs × 262144 ctx (the model's full native window)** at KV_BYTES 7,300,000,000/rank (bf16 measures 13,839 B/token), with **`--expert-precise-memory`** — the fitted 5.25 GiB overhead instead of the 7.60 GiB default, which hands the compile/graph-capture headroom to the expert arena (~7,278 slots vs 6,248) — util 0.98, and the client contract wired: `--reasoning-parser qwen3` (thinking in `reasoning`, never inline), `--enable-auto-tool-choice --tool-call-parser qwen3_xml`, default chat-template kwargs `enable_thinking` + `preserve_thinking`. Capacity history: five full-length contexts boot but leave ~0 bytes free at runtime (a real request with `max_tokens=261149` killed the worker: `Tried to allocate 50.00 MiB ... 0 bytes free`); six never fit (6×241664 asked 10.70 GiB vs 9.18 GiB free). DCP cannot help here — it requires TP > the model's 2 KV heads, i.e. four GPUs. fp8+MTP4 shape restorable from git tag `frank-primary-fp8-mtp-20261001` | 7770 |
+| Dankserver | `frank-primary` | tcclaviger RDNA4 vLLM fork (`docker.io/tcclaviger/vllm@sha256:da4029666…`, digest-pinned, plain `podman run`, root systemd unit) | `orcarouter/Qwen3.8-Flash-Next-Uncensored-FP8` (~172.8 GiB, 131 shards), TP=2, expert offload automatic (fork plans it), prefetch OFF, **fp8 KV** (`--kv-cache-dtype fp8`), **no speculative decode**, MNBT 1024, MoE gather 32×64, **2 seqs × 200704 ctx** at KV_BYTES 3,150,000,000/rank (fp8 measures 7,701 B/token), pool deliberately small so the expert arena keeps what it reclaims (~7,500 slots vs 6,248 at the 6.1e9 shape) — util 0.98, and the client contract wired: `--reasoning-parser …
-| Dankclaw | `frank-large` | gufo `gufo-org/gufo` v0.1.1, the Strix Halo engine, BUILT FROM SOURCE (root systemd unit; links the host ROCm 7.2.4) | `unsloth/Qwen3.8-27B-GGUF` **UD-Q4_K_XL** (27B dense, **vanilla — no longer abliterated**, 16.35 GiB, Apache-2.0) with the z-lab DFlash2 Q4_K_M drafter (adaptive block policy, cap 7) on the gfx1151 iGPU (Strix Halo LPDDR, 114 GiB GTT), **4 sessions x 163840 ctx, unquantized (f16) KV** — gufo has no KV-quant lever — greedy, gufo's compiled-in Qwen renderer (NO Qwen-Sharp), text only. Swapped 2026-10-01 from ajgazin's abliterated Swift-1.5 UD-Q8_K_XL (29.3 GiB): ~13 GiB less weights is what funds four f16 sessions, at the cost of the uncensored behaviour | 7770 |
+| Dankclaw | `frank-large` | **ROCmFPX llama.cpp fork** (`kingjones30/ROCmFPX` @ `36e9acd`, gfx1151, built by `scripts/build-rocmfpx.sh`; bare `llama-server`, root systemd unit) | `kingjones777/Qwen3.8-27B-ROCmFP4-STRIX-MTP-GGUF` **`Qwen3.8-27B-Q8_0_ROCMFPX_AGENT.gguf`** (26.28 GiB, ftype 115 ≈ 8.39 bpw: 353 F32 + 187 Q8_0 + 311 `GGML_TYPE_Q8_0_ROCMFPX` tensors — only this fork can read them), **MTP spec decode from the repo's own `mtp-Qwen3.8-27B-Q4_0.gguf` (n-max 4, pinned to ROCm0)**, **1 session × 262,144 (native window)** with **f16 KV**, the bundled BF16 mmproj for vision, thinking on with the card's recipe (temp 1.0 / top_p 0.95 / top_k 20). Measured 2026-10-02: decode **17.0–17.1 t/s** with MTP vs **7.6 without (+123 %)**, draft acceptance 0.462, prefill 325–351 t/s, prompt-cache replay 0.5 s, boot→healthy 16 s, GTT 64.5 GiB. Cut over 2026-10-02 from gufo; rollback tag `frank-large-gufo-q4kxl-20261001` | 7770 |
-| Dankclaw | `frank-small` | SGLang in a self-contained python3.12 venv (root systemd unit; the venv carries its own CUDA 13 stack — see the runbook) | letechlead MiMo-V2.6-Distill-Qwen-9B-INT8-W8A16-AutoRound (qwen3_5 hybrid, 9B dense, 12.0 GiB, INT8 W8A16 AutoRound, in-checkpoint vision tower, NO MTP weights shipped so no spec decode) on the RTX 3090, fp8 KV, pinned 606,208-token pool / 2 slots of 262,144 (native window; 2026-10-01 reslice from 4 x 151,552 for compaction headroom), bf16 mamba states, eager prefill, 83.9 tok/s decode | 7771 |
+| Dankclaw | `frank-small` | llama.cpp master, CUDA sm_86 build (root systemd unit, relinked) | bartowski/MiMo-V2.6-Distill-Qwen-9B-GGUF **Q5_K_M** (9B dense qwen3_5 hybrid, 6.4 GiB GGUF, no MTP weights so no spec decode) + the repo's f16 mmproj for **vision**, on the RTX 3090, **3 × 180,224** (pool 540,672 — NOT the native window; three contexts cost 81,920 tokens per conversation) with **asymmetric KV f16 K + q8_0 V** under flash attention, 21,684 used / **2,441 free** of 24,576 MiB, 106.8 tok/s decode, 3,814 t/s prefill. Resized twice 2026-10-02 (Q6_K → Q5_K_M → 3 slots + vision → this pool); rollback `models/MiMo-Q6K/` or `PARALLEL=2 CTX_SIZE=524288` | 7771 |
 
 Retired endpoints (`frank-npu`, and every prior runtime) are in
 the [runbook index](#runbook-index). The full swap history lives in
@@ -209,14 +209,14 @@ archive.
 **STOP the production service BEFORE benchmarking** — these servers serve
 real users, and a bench beside a live service starves the GPU and corrupts
 results. Use the runtime's own harness: `llama-bench` for the llama.cpp
-endpoints, `gufo bench` for frank-large. Rolling your own bench
+endpoints; a live llama.cpp server also reports honest per-request numbers in the `/completion` `timings` object (prefill/decode rates, `draft_n`, `draft_n_accepted`), which is how frank-large and frank-small were measured. Rolling your own bench
 requires asking first.
 Scratch ports are not used on LLM hosts — a
 test process cannot share the single GPU anyway; validate against the service
 itself.
 
-**`gufo bench` cannot rank speculative-decode settings, and will invert the
+(A gufo-era law; frank-large is ROCmFPX llama.cpp since 2026-10-02, but the rollback tag keeps it live.) **`gufo bench` cannot rank speculative-decode settings, and will invert the ranking.**
-ranking.** Measured 2026-09-28 on frank-large at depth 32768: its corpus
+Measured 2026-09-28 on frank-large at depth 32768: its corpus
 reports `acceptance=0.933 ± 0.000` with an **identical trace sha256** across
 draft widths, i.e. the same highly-predictable text every run, and 93 % of
 every proposal accepted. It reported 29–37 t/s decode where real traffic
@@ -241,9 +241,9 @@ one: `ip neigh show <ip>` from a peer — FAILED means off/NIC dead, while a
 merely wedged kernel usually still answers ARP.
 
 1. Stop and mask (self-healing would restart it):
-   - Dankclaw `frank-large` (root unit, gufo on the iGPU):
+   - Dankclaw `frank-large` (root unit, vLLM podman on the iGPU):
      `sudo systemctl stop frank-large && sudo systemctl mask frank-large`
-   - Dankclaw `frank-small` (root unit, SGLang on the 3090):
+   - Dankclaw `frank-small` (root unit, llama.cpp on the 3090):
      `sudo systemctl stop frank-small && sudo systemctl mask frank-small`
    - Dankserver `frank-primary` (root unit, tcclaviger vLLM on the 2x R9700):
      `sudo systemctl stop frank-primary && sudo systemctl mask frank-primary`
@@ -277,12 +277,15 @@ When renaming a service, retiring an endpoint, or moving a model: update the
 provider block, model `id`, `contextWindow`, `maxTokens`, `input` in
 `models.yml`, then fix every `modelRoles` entry that named the old endpoint —
 a role pointing at a provider the file does not define fails at call time. As
-of 2026-10-01 both `frank-small` and `frank-primary` accept images
+of 2026-10-02 **all three endpoints accept images** (`input: [text, image]`):
-(`input: [text, image]`) — frank-primary loads the BF16 mmproj explicitly
+`frank-large` through the BF16 mmproj that ships inside its GGUF payload,
-(this build never autoloads projectors), and frank-small's MiMo payload
+`frank-primary` through a BF16 mmproj loaded explicitly (this build never
-bakes its vision tower INTO the checkpoint (no projector file at all);
+autoloads projectors), and `frank-small` through
-`frank-large` remains text-only because its gufo deployment loads no
+`mmproj-MiMo-V2.6-Distill-Qwen-9B-f16.gguf` passed as `--mmproj` — the Q5_K_M
-vision projector. Note also that
+GGUF itself carries no vision tensors. `frank-small`'s `contextWindow` is
+**180224**, the per-slot window, not the model's native 262144: the harness
+must compact before llama.cpp answers a longer prompt with a 400. Note also
+that
 the reasoning channel differs by runtime: llama.cpp AND SGLang (qwen3
 parser, live-verified) emit `reasoning_content`, vLLM emits `reasoning` —
 models.yml sets `reasoningContentField` per endpoint to match.
@@ -292,161 +295,129 @@ endpoint on 2026-09-11, returned 2026-09-17 with the Gemma4 projector's
 audio encoder, and left again with the Gemma4 retirement on 2026-10-01
 (the MiMo tower is vision-only).
 
-### frank-large — gufo, Swift-1.5-Qwen3.8-27B-Uncensored UD-Q8_K_XL (Dankclaw, ACTIVE)
+### frank-large — ROCmFPX llama.cpp, Qwen3.8-27B Q8_0_ROCMFPX_AGENT + MTP (Dankclaw, ACTIVE)
 
-**gufo** (`gufo-org/gufo`) v0.1.1 — "the Strix Halo inference engine", a
+The **kingjones30/ROCmFPX llama.cpp fork** at commit
-vertical engine built for exactly this hardware (Ryzen AI MAX+ 395 /
+`36e9acd40e10a87cd3c3ef8ec734668757dc8520`, built for gfx1151 by
-Radeon 8060S, `gfx1151`) — as a **root systemd unit**
+`scripts/build-rocmfpx.sh` into `/home/dank/GitPackages/kingjones-ROCmFPX/build`,
-(`scripts/frank-large.service`, `User=dank`), serving
+as a **root systemd unit** (`scripts/frank-large.service`, `User=dank`,
-`ajgazin/Swift-1.5-Qwen3.8-27B-Uncensored-Dynamic-MTP-GGUF`'s `UD-Q8_K_XL`
+`Restart=always`) running a bare `llama-server` (no container) on port 7770
-(27B dense, 29.3 GiB, sha256-verified; **abliterated/uncensored**, and under
+from the Strix Halo iGPU. Boot to healthy: **16 s**.
-the *non-OSI* SWIFT OPEN LICENSE 1.0) on port 7770 from the gfx1151 iGPU.
-One session, full 262,144 ctx (the model's native window, `SESSIONS=1`),
-DFlash2 spec decode (the z-lab `Qwen3.8-27B-DFlash2-Q4_K_M` drafter, gufo's
-**adaptive** block policy, `--draft-tokens 7`), greedy sampling, reasoning
-extracted to `reasoning_content`, and **no vision projector**.
 
-**Two swaps landed on 2026-09-28**, both same day: the engine (from the
+**The runtime is not interchangeable — it is chosen by the FILE.**
-unsloth llama.cpp b11030 prebuilt, which had replaced Kairic Edge ROCmFPX)
+`kingjones777/Qwen3.8-27B-ROCmFP4-STRIX-MTP-GGUF`'s best rung,
-and then the payload (from `unsloth/Qwen3.8-27B-GGUF:UD-Q8_K_XL`, still on
+`Qwen3.8-27B-Q8_0_ROCMFPX_AGENT.gguf` (28,214,425,952 B, sha256 `f3ef2bda…`,
-disk and the payload-only rollback). The two targets are structurally
+ftype 115 ≈ 8.39 bpw), stores **311 tensors of type 103,
-interchangeable for gufo — same `qwen35` architecture, the same 866 tensors
+`GGML_TYPE_Q8_0_ROCMFPX`** — a quantization that exists in no other loader:
-in the same order, the same per-tensor Q8_0/F32/BF16 layout, the same MTP
+stock llama.cpp and gufo both refuse it at load (`unknown tensor type`). The
-head at `blk.64` — verified by parsing and diffing both GGUF headers before
+file is 353 F32 + 187 plain-Q8_0 + 311 ROCmFPX; the AGENT routing variant
-either was loaded. **The DFlash2 drafter did not move**, so it has its own
+keeps the output-head tensors at plain Q8_0 (187 vs 2 in the non-AGENT file),
-`DRAFTER_DIR` and the deployer's stray-run sweep keys on the drafter's
+which is why the card reports MTP acceptance 0.953 vs 0.911 — highest quality
-filename, not the target's.
+AND the better drafter base. Vision comes from the bundled
+`mmproj-Qwen3.8-27B-BF16.gguf` (931,145,888 B).
 
-**The runtime is built from source on the host** —
+**The pin is load-bearing.** The 2026-08-25 ROCmFPX builds segfaulted in
-`/home/dank/GitPackages/gufo-v0.1.1/`, reproducible with
+`ggml_compute_forward_set_rows` on exactly this file; `build-rocmfpx.sh`
-`scripts/build-gufo.sh` (`--verify` re-checks an installed tree). gufo ships
+checks the commit out by SHA, so a fork-side force-push cannot silently
-no prebuilt Linux binaries and the v0.1.0/v0.1.1 release assets are empty.
+re-break the endpoint.
-Unlike the retired llama.cpp prebuilt, which bundled its own ROCm, **gufo
-links the host ROCm 7.2.4 through its RPATH**, so the HIP dev packages must
-stay installed. Two host workarounds the build script encodes:
 
-- **Ubuntu's `librocwmma-dev` must be absent.** It installs a truncated
+**Spec decode is the performance feature, and it is measured.** The draft is
-  rocWMMA header set into `/usr/include` that shadows AMD's complete one
+the repo's own `mtp-Qwen3.8-27B-Q4_0.gguf` (1,680,271,648 B, sha256 `051a1764…`,
-  and breaks the build with `'internal/accessors.hpp' file not found`. A
+`nextn_predict_layers = 1`), pinned to the same card:
-  `-I` cannot fix it — in HIP mode the host pass searches the system
+`--spec-type draft-mtp --spec-draft-n-max 4 --spec-draft-n-min 0
-  directories first. The package has no reverse dependencies.
+--spec-draft-p-min 0.0 --spec-draft-device ROCm0 --spec-draft-ngl 99`.
-- **`--gcc-install-dir` must name GCC 15.** Ubuntu 26.04 ships
+Measured 2026-10-02 in one session, temp 0, 2,202-token prompt / 512 tokens,
-  `/usr/include/c++/16`, which the ROCm clang prefers; its `<atomic_wait.h>`
+`cache_prompt: false`, production service stopped for each arm:
-  needs a `std::__detail::__notify_impl` symbol version the installed
+**MTP on 17.00–17.14 t/s decode** (draft acceptance 0.462, mean acceptance
-  `libstdc++.so.6` (GLIBCXX_3.4.35) does not export, and the link fails.
+length ≈ 1.86) against **7.60–7.66 t/s with the draft removed — +123 %**;
+prefill pays 8–15 % (381 → 325–351 t/s). Acceptance is far from the card's
+0.95 (that number is the card's own text at a shallow prompt); the gain is
+real and n-max 4 is where it was measured, never swept. This replaces the
+gufo Q4_K_XL arm at ~15 t/s, so the endpoint got higher quality AND more
+speed at once.
 
-`gufo diagnose` is the preflight and must report `[PASS]` with `gfx1151`.
+**Shape and budget:** ONE session at the native 262,144 window, unquantized
-It also self-caps compute queues (`GPU_MAX_HW_QUEUES`, 2 resident for
+f16 KV (~64 KiB/token → ~16 GiB, affordable inside the iGPU's ~114 GiB GTT
-`serve llm`) after reading `/sys/class/kfd/kfd/proc/*/queues/*/type`, which
+aperture), `--fit off --flash-attn on --direct-io --ctx-checkpoints 16
-matters because ComfyUI holds queues on the same device; four HIP processes
+--cache-ram 8192 --cache-idle-slots`. Measured GTT 64.5 GiB with model +
-is the device ceiling.
+draft + projector resident. Prompt cache verified: a repeat pass over the
+same 6k prompt replays in 0.5 s.
 
-**The client contract changed, and the chat template is the sharp edge.**
+**Client contract (llama.cpp, live-verified 2026-10-02):** `--jinja` is on
-**gufo has no Jinja engine at all** — `src/models/qwen/chat_template.cpp` is
+(without it `chat_template_kwargs` are silently ignored), tools are native
-587 lines of hand-written C++ that appends strings, with no
+(`get_weather` → `{"city":"Seattle"}`), thinking lands in
-`--chat-template-file`, no env override, and no template body to point at.
+**`reasoning_content`**, `reasoning_effort` accepts ONLY
-`scripts/qwen-sharp-v22.4.0.jinja` therefore **cannot** be supplied to
+`xhigh|medium|low` (anything else is HTTP 500 — `models.yml` folds `high` and
-frank-large: that is a port, not a config line. Given the choice on
+`max` to `xhigh`), and thinking is enabled server-side with the card's
-2026-09-28 the decision was to ship on gufo's stock renderer and accept the
+recipe (temp 1.0 / top_p 0.95 / top_k 20 / min_p 0) plus
-loss — the `terse` system-prompt injection, the in-band `<|think_*|>`
+`--reasoning-format deepseek --reasoning-budget -1`. Two traps: a request
-control, the `json|xml` `tool_call_format` switch, the tool-argument
+whose whole budget is consumed by reasoning returns `content: null` (bench
-truncation and `add_vision_id` prefixes are all gone. gufo's own renderer
+clients must fall back to `reasoning_content` or they crash *after*
-does keep the effort instruction strings and `preserve_thinking`, which are
+measuring), and the fork logs
-what the harness depends on. **This is the thing to watch**: the endpoint
+`MTP boundary missing for seq_id=… ; resyncing` under concurrent draft
-now runs an *abliterated* model without the prompt that tells it to stop
+resets — self-recovering, not a fault.
-waffling. The cheap fix, if verbosity shows up, is to send the `terse` text
-as an ordinary system message from `models.yml` — not a template change.
-`reasoning_effort` accepts the full OpenAI set (all six levels verified
-200), so `models.yml`'s `reasoningEffortMap` fold is no longer *required* —
-it is kept deliberately, so that removing it does not silently move every
-`high` role from `xhigh` to `high` effort inside a swap. And there is no
-KV-quantization or prompt-cache-RAM flag; prompt reuse is on by default and
-`--cache-disk` is opt-in and off.
 
-**Read the perf trade before assuming a win.** gufo's own Q8_K_XL tables
+Fetch: `scripts/download-frank-large-kingjones.sh` (`-f` + byte count + LFS
-(vs llama.cpp b11069, DFlash2) show prefill +37 % to +80 %, cold load 4.5×
+sha256 verified, `--verify-only`, no token on the command line).
-faster, decode +7 % at zero depth and +40 % at 4 users — but **−14 % at
-65,536 tokens of depth and −24 % at 131,072**. On the 8-bit target the
-crossover is ~32k of cached context, and frank-large runs one 262,144-token
-conversation, so long agent sessions sit on the losing side of that line.
-The Q4_K_XL target shows no such regression; revisiting the quantization is
-a separate decision from the runtime. Peak memory is 50.58 GiB at the full
-262,144 context against the iGPU's 114 GiB GTT aperture.
 
-The serve script (`scripts/serve-frank-large.sh`) validates runtime → target
+Runbook: `serverAppConfigs-docs/LLM/frank-large-rocmfpx-q8agent-mtp.md`.
-GGUF → drafter and exits with a pointer message if any is missing, then
+Rollback: git tag **`frank-large-gufo-q4kxl-20261001`** — gufo v0.1.1
-`exec`s `gufo serve --host 0.0.0.0 --port 7770 --sessions 1 llm --model
+(`scripts/build-gufo.sh` stays on disk for it), the Q4_K_XL Qwen3.8-27B
-…--served-model-name frank-large --context 262144 --speculative dflash2 …`.
+payload and the DFlash2 drafter all remain on the host; restore the tag's
-The deployer's `deploy_llm_large` gates on the **runtime binary** as well
+unit/env/serve trio and redeploy.
-as both payload paths, and returns before it stops anything.
 
-Fetch: `scripts/download-frank-large-swift.sh` (the target; per-file byte
+### frank-small — llama.cpp on the RTX 3090 (Dankclaw, ACTIVE)
-count + sha256 against the Hub LFS oid, no auth header — the repo is public
-and a token on the curl command line is world-readable) and
-`scripts/download-frank-large.sh` (the DFlash2 drafter, which did not move).
-Both take `--verify-only`.
 
-Runbook: `serverAppConfigs-docs/LLM/frank-large-gufo-qwen38-27b.md`.
+**llama.cpp master** (CUDA sm_86 build
-Rollback, two ways: **payload only** — re-fetch with
+`/home/dank/GitPackages/llama-master/build-cuda/bin/llama-server`) as a
-`scripts/download-frank-large.sh` and repoint `MODEL_FILE` at
+**root systemd unit** (`scripts/frank-small.service`, `relink_unit`ed into
-`/home/dank/models/Qwen3.8-27B-UD-Q8_K_XL/Qwen3.8-27B-UD-Q8_K_XL.gguf`; that
-29.3 GiB target was **reclaimed on 2026-09-28** (staged to
-`~/.AGENTTRASH/20260928-retired-models/`, restorable from there), so it is
-no longer a one-minute flip. **Engine** —
-[[frank-large-unsloth-qwen38-27b-q8xl]], the b11030 prebuilt at
-`/home/dank/GitPackages/unsloth-llama-b11030/`; the only way to get
-Qwen-Sharp back. The DFlash2 drafter was never reclaimed and still lives in
-`Qwen3.8-27B-UD-Q8_K_XL/`.
-
-### frank-small — SGLang on the RTX 3090 (Dankclaw, ACTIVE)
-
-SGLang (latest) in a **self-contained python3.12 venv**
-(`/home/dank/GitPackages/sglang-cu128/`, torch 2.13.0+cu130) as a **root
-systemd unit** (repo `scripts/frank-small.service`, `relink_unit`ed into
 `/etc/systemd/system/`), serving
-`letechlead/MiMo-V2.6-Distill-Qwen-9B-INT8-W8A16-AutoRound` (9B dense
+`bartowski/MiMo-V2.6-Distill-Qwen-9B-GGUF` at **Q5_K_M** (6,876,124,704 B,
-qwen3_5 hybrid — 8 full-attention + 24 Gated DeltaNet layers — INT8 W8A16
+sha256 `04cf4f…`) **plus `mmproj-MiMo-V2.6-Distill-Qwen-9B-f16.gguf`**
-AutoRound, 12.0 GiB, vision tower baked into the checkpoint). Port 7771.
+(918,166,048 B, sha256 `ff348f…`), both `/home/dank/models/MiMo-Q5KM/`, on
+port 7771. The 2026-10-02 swap was a **quant rung down from Q6_K**, not a
+runtime change — the llama.cpp path arrived 2026-10-01 night for one reason:
+**asymmetric KV**. `--cache-type-k f16` + `--cache-type-v q8_0` is the fleet's
+only per-tensor KV lever; SGLang's `--kv-cache-dtype` moves K and V together.
+Vision came the same day via `--mmproj`: the GGUF carries no `v.*` tensors, so
+the projector file is the entire vision path (and ~1.4 GiB of the card).
 
-**The shape is memory-pinned, and the sharp edge is the mamba pool.** The
+**Architecture: qwen3_5 hybrid** — 32 layers, 8 full-attention + 24 Gated
-DeltaNet ssm state pool autofills to a fixed ~5.1 GB budget *regardless of
+DeltaNet, native 262,144 window. KV exists only in the 8 attention layers
-dtype* (fp32 at 110 slots, bf16 at 217 — same 5.1 GB), so both dials are
+(4 KV heads × 256 dim × 2 B ≈ 2 KiB/token/layer, ~16 KiB/token for f16 K+V);
-pinned: `--mamba-ssm-dtype bfloat16 --max-mamba-cache-size 32` (795 MiB).
+the DeltaNet state is a fixed per-slot buffer, so slot count — not context —
-The KV pool is then **pinned at 606,208 tokens** (`--max-total-tokens`) —
+drives that term.
-the engine's autofit picks 687,388 and leaves 0.53 GB, which Triton's
-deferred kernel loads eat and the first long request OOMs on. Two
-262,144-token slots at the model's NATIVE window (2026-10-01 reslice from
-4 x 151,552: 2 x 262,144 = 524,288 leaves ~82k tokens of pool slack, and
-a compaction request — frank-primary's trigger is 183,501 tokens plus a
-32,768-token summary budget = 216,269 — now fits ONE request); a single
-request may grow to the whole pool. Prefill runs **eager**
-(`--disable-prefill-cuda-graph`) — there is no headroom for capture;
-decode graphs are on.
 
-**No speculative decode — a checkpoint fact.** `text_config` declares
+**The shape:** `--ctx-size 540672 --parallel 3` = **THREE conversations of
-`mtp_num_hidden_layers: 1`, but the shipped safetensors (base BF16 and this
+180,224 tokens each** (llama.cpp divides the pool:
-INT8 conversion alike) carry zero `mtp.*` tensors; NEXTN would draft with
+`n_ctx_slot = CTX_SIZE / PARALLEL`), `--flash-attn on` (the q8_0-V path is
-an uninitialised 4.24 GiB ghost model at 0% acceptance. The seam gates
+only cheap under flash attention — the seam pins it), f16 K + q8_0 V, all
-that no `--speculative-*` flag reaches the engine.
+layers on the GPU, `--mmproj` for vision. Measured VRAM **21,684 used / 2,441
+free** of 24,576 MiB — the 2–3 GiB headroom Frank asked for, landed by
+solving the pool against two measured boots: non-KV cost is 7,390 MiB without
+a projector (2 × 262,144 boot) and 8,859 MiB with one (3 × 196,608 boot), so
+vision costs ~1.4 GiB, not the 0.9 GiB the file size suggests. Three native
+windows (786,432 tokens) need 18.4 GiB of KV alone and were never on the
+table. Full f16 V would add ~4 GiB more and bust the card — that is the
+justification for the asymmetric contract.
 
-**The venv carries a CUDA 13.0 nvcc extracted from debs** (system nvcc is
+Verified at deploy (2026-10-02): `/v1/models` reports `frank-small`; boot logs
-12.4) so the sgl-kernel/flashinfer JIT compiles for sm86; three local
+`n_slots = 3, n_ctx_slot = 180224` and `loaded multimodal model`; decode
-patches ride in the tree (GCC-16 shim, CCCL symlink, rsqrt noexcept).
+**106.8 tok/s** at temp 0 (prefill 3,814 t/s) — ~5 % faster than the
-Details in the runbook.
+2 × 262,144 shape, because a shorter slot means a shorter attention scan;
+three concurrent requests answered in parallel and a fourth queued behind a
+released slot; native `tool_calls`; thinking in `reasoning_content`; a
+solid-red PNG answers "Red." through an `image_url` content part (the legacy
+top-level `images` field is ignored by `/v1/chat/completions` — the model then
+invents a color). A prompt over the slot is a **400
+`exceed_context_size_error`**, which is why `models.yml` sets
+`contextWindow: 180224`.
 
-Verified at deploy (2026-10-01): `/v1/models` reports `frank-small`;
+Runbook: `serverAppConfigs-docs/LLM/frank-small-llama-mimo-q5km.md`.
-chat returns content plus `reasoning_content` (SGLang's qwen3 parser uses
+Rollback: [[frank-small-llama-mimo-q6k]] (Q6_K file on disk at
-llama.cpp's field name); a red-square image answers "Red"
+`/home/dank/models/MiMo-Q6K/`); before that [[frank-small-sglang-mimo-v26]]
-(`image_tokens: 64`). At the 4 x 151,552 shape: 4 concurrent 54k-token
+(venv on disk at `/home/dank/GitPackages/sglang-cu128/`; its INT8 payload was
-requests answered; a 170,019-token prompt prefilled in 50.5 s (~3,370 t/s
+staged to `~dank/.AGENTTRASH/20261002-retired-llm-payloads/` on 2026-10-02).
-eager). At the 2 x 262,144 shape (reslice, same day): boot flags read
-`context_len=262144 max_running_requests=2 max_total_num_tokens=606208`;
-a 200,000-token prompt — well past frank-primary's 183,501 compaction
-trigger — prefilled and completed in ONE request (97.3 s wall, ~2,056
-tok/s eager); two concurrent 50k-token requests both answered. Decode
-**83.9 tok/s**.
-Runbook: `serverAppConfigs-docs/LLM/frank-small-sglang-mimo-v26.md`.
-Rollback: [[frank-small-gemma4-12b]] (payload and binary intact on disk).
 
 ### frank-primary — tcclaviger vLLM, Qwen3.8-Flash-Next FP8 + MTP4 (Dankserver, ACTIVE)
 
@@ -655,6 +626,26 @@ linked runbooks.
   non-greedy requests and reasoning budgets. Re-test chat before production.
 - **FastMTP sidecar**: build must be commit `4df29be4` + the HauhauCS patch;
   stock binaries fail the draft load with a tensor-shape mismatch.
+- **A GGUF's custom tensor type picks the runtime — read the histogram before
+  promising a file to a loader.** `Qwen3.8-27B-Q8_0_ROCMFPX_AGENT.gguf` is
+  ftype 115 with 311 tensors of type 103 (`GGML_TYPE_Q8_0_ROCMFPX`); stock
+  llama.cpp and gufo refuse it at load, so ROCmFPX was not a preference but
+  the only door. Counting tensor types out of the GGUF KV section takes one
+  python read and settles a runtime argument in minutes. The build identity
+  matters as much as the fork: the 2026-08-25 ROCmFPX builds SIGSEGV in
+  `ggml_compute_forward_set_rows` on this same file, so
+  `scripts/build-rocmfpx.sh` pins commit `36e9acd` by SHA.
+- **Speculative-decode flags move as a SET.** Handing the ROCmFPX fork
+  `--model-draft` with no `--spec-type` (the naive way to build a "no MTP"
+  bench arm) segfaulted at load, core dumped, no diagnostic (2026-10-02). A
+  baseline arm must strip `--model-draft` AND every `--spec-draft-*` AND
+  `--spec-type`, or it measures a crash instead of the absence of drafting.
+- **An orphaned scratch server taxes the whole host.** A bench child left
+  alive by a client-side exception kept ~26 GiB of weights resident on a
+  121 GiB box, pushed `available` to 3 GiB, and every later rate was suspect.
+  Record the child pid and kill it in a `finally`, then prove `free` came back
+  before believing a number. Kill by pid — never `pkill -f llama-server`,
+  which also hits frank-small.
 
 **llama.cpp serving**
 
@@ -727,6 +718,35 @@ linked runbooks.
   host-RAM hash table) and its window decides which traffic it can see: too
   short and it proposes on ordinary prose fragments, which costs verification
   without acceptance.
+- **`--parallel` divides the pool, so MORE conversations cost WINDOW, and the
+  per-slot number is the contract — not `ctx-size`.** `n_ctx_slot =
+  CTX_SIZE / PARALLEL`, and llama.cpp refuses a prompt longer than the SLOT
+  (`400 exceed_context_size_error: request (184001 tokens) exceeds the
+  available context size (180224 tokens)`) even when the pool has room. So
+  frank-small's 2 → 3 reslice moved each conversation from the model's native
+  262,144 down to 180,224, and `models.yml` `contextWindow` had to follow or
+  the harness would discover the ceiling as a 400 on a long conversation. Side
+  effect worth knowing: the shorter slot made decode ~5 % FASTER (106.8 vs
+  100.4 tok/s) — less attention to scan and one fewer DeltaNet state resident.
+  Measured 2026-10-02. See [[frank-small-llama-mimo-q5km]].
+- **Price a VRAM budget from two boots, never from a formula — and an mmproj
+  costs ~1.5x its file size.** The frank-small 3090 fit was solved from two
+  measurements: 2 × 262,144 with no projector = 20,190 MiB used (KV 12,800 ⇒
+  **non-KV 7,390**), 3 × 196,608 with the f16 projector = 22,972 used (KV
+  14,113 ⇒ **non-KV 8,859**). The difference is the vision tower: a 918 MB file
+  that bills ~1.4 GiB, because the encoder's activation and compute buffers
+  ride with it. Subtracting the two rows is exact; extrapolating weights+KV+
+  "a bit for buffers" is what leaves 1.1 GiB free when you meant 2.7. Then
+  solve `KV ≤ card − non-KV − headroom` and round to a multiple of
+  `PARALLEL × 4096`. Recorded in `scripts/frank-small.env` so the next resize
+  starts from measurement.
+- **`images` and `image_url` are NOT interchangeable: only the content part
+  reaches the model.** `--mmproj` loads, `usage.prompt_tokens` rises, and the
+  model still invents an answer if the client used the legacy top-level
+  `images` field on `/v1/chat/completions` — frank-small answered "Brown." for
+  a solid-red PNG that way and "Red." through the OpenAI `image_url` part. The
+  failure is silent because a hallucinated color is a plausible answer. Probe
+  vision with a solid color and expect an exact word.
 - **`--jinja` is what `tools` needs.** On 2026-09-16 a subagent died with
   `500 tools param requires --jinja flag`, from the ik_llama.cpp runtime that
   then served frank-large: every tool-using agent on that endpoint was broken
@@ -827,8 +847,8 @@ Active:
 
 | Service | Runbook |
 |---------|---------|
-| frank-large (gufo Qwen3.8-27B UD-Q8_K_XL, Dankclaw) | `serverAppConfigs-docs/LLM/frank-large-gufo-qwen38-27b.md` |
+| frank-large (ROCmFPX llama.cpp fork, Qwen3.8-27B Q8_0_ROCMFPX_AGENT + MTP4 + vision, Dankclaw) | `serverAppConfigs-docs/LLM/frank-large-rocmfpx-q8agent-mtp.md` |
-| frank-small (SGLang MiMo-V2.6 INT8, Dankclaw) | `serverAppConfigs-docs/LLM/frank-small-sglang-mimo-v26.md` |
+| frank-small (llama.cpp CUDA, MiMo-V2.6-Distill-Qwen-9B Q5_K_M, Dankclaw) | `serverAppConfigs-docs/LLM/frank-small-llama-mimo-q5km.md` |
 | frank-primary (tcclaviger vLLM, Qwen3.8-Flash-Next FP8 + MTP4, Dankserver) | `serverAppConfigs-docs/LLM/dankserver-qwen38-fp8-expert-offload-tuning.md` |
 | Dankclaw kernel posture (`amd_iommu=off`) | `serverAppConfigs-docs/LLM/dankclaw-iommu-translation-off.md` |
 | Deployment flow + incidents | `serverAppConfigs-docs/LLM/deployment.md` |
@@ -837,6 +857,10 @@ Retired (rollback targets and history):
 
 | Runtime | Host | Window | Runbook |
 |---------|------|--------|---------|
+| gufo v0.1.1 Qwen3.8-27B UD-Q8_K_XL/UD-Q4_K_XL + DFlash2 (`frank-large`) | Dankclaw | 2026-09-28 → 10-01, again 10-01 → 10-02 (rollback target) | `frank-large-gufo-qwen38-27b.md` |
+| vLLM TheRock MiMo-V2.6-Distill-Qwen-9B BF16 + cross-model MTP (`frank-large`) | Dankclaw | 2026-10-01 (same-day rollback to gufo) | `frank-large-vllm-mimo-v26-bf16.md` |
+| llama.cpp master MiMo-V2.6-Distill-Qwen-9B Q6_K (`frank-small`) | Dankclaw | 2026-10-01 → 10-02 (rollback target) | `frank-small-llama-mimo-q6k.md` |
+| SGLang MiMo-V2.6 INT8-W8A16-AutoRound (`frank-small`) | Dankclaw | 2026-10-01 (same-day swap to the GGUF path) | `frank-small-sglang-mimo-v26.md` |
 |tcclaviger vLLM GLM-5.3-Flash W4A16 int4, +/- DFlash2 drafter (`frank-primary`)|Dankserver|2026-09-30 → 10-01|`frank-primary-glm53-flash-w4a16.md`|
 | llama.cpp b11030 Qwen3.8-27B UD-Q8_K_XL + DFlash2 (`frank-large`) | Dankclaw | 2026-09-21 → 09-28 | `frank-large-unsloth-qwen38-27b-q8xl.md` |
 | R9V radiance-vLLM Qwen3.8-Flash-Next-Uncensored IQ4_XS (`frank-primary`) | Dankserver | 2026-09-27 (same-day swap to Q8_0) | `frank-primary-r9v-radiance-qwen38-uncensored.md` |

autoresearch.sh

@@ -0,0 +1,104 @@
+#!/usr/bin/env bash
+# autoresearch.sh — canonical benchmark entrypoint: speed of frank-primary.
+#
+# Runs on the operator box and drives one bench window on DANKSERVER:
+#   1. fingerprint the candidate trio (env + serve script + unit)
+#   2. memo fast-path: an identical fingerprint that already produced a FULLY
+#      successful window (bench + verified production restore) returns the
+#      cached metrics without touching the fleet
+#   3. stage the candidate trio plus the measure client into
+#      /home/dank/.autoresearch-frank-primary/ (ephemeral; the repo checkout
+#      on the host stays the production source of truth)
+#   4. run the window (scripts/autoresearch/frank-primary-window.sh):
+#      stop + mask the production unit, boot the staged config AS THE SERVICE
+#      (same serve script, same container, same port 7770), wait for health,
+#      run the in-container usage-counter measure, then restore the production
+#      unit and verify its health — AGENTS.md Benchmarking Policy
+#   5. print METRIC lines (parse by run_experiment)
+#
+# Primary metric: decode_tps (HIGHER is better) — median across the three
+# prompt depths (115/1056/4096) of per-depth medians of
+# completion_tokens / (wall - ttft), 3 runs each after a warm pass,
+# temperature 0, the campaign's deterministic word-seed workload.
+#
+# Exit 0 with METRIC lines on success; non-zero on any failure (the window
+# script's EXIT trap restores production on every path).
+set -euo pipefail
+
+SCRIPT_DIR="$(cd -- "$(dirname -- "${BASH_SOURCE[0]}")" && pwd)"
+HOST="${FRANK_PRIMARY_AR_HOST:-dankserver}"
+STAGE="/home/dank/.autoresearch-frank-primary"
+
+ENV_FILE="$SCRIPT_DIR/scripts/frank-primary.env"
+SERVE="$SCRIPT_DIR/scripts/serve-frank-primary.sh"
+UNIT="$SCRIPT_DIR/scripts/frank-primary.service"
+MEASURE="$SCRIPT_DIR/scripts/autoresearch/frank-primary-measure.py"
+WINDOW="$SCRIPT_DIR/scripts/autoresearch/frank-primary-window.sh"
+
+for f in "$ENV_FILE" "$SERVE" "$UNIT" "$MEASURE" "$WINDOW"; do
+    [ -f "$f" ] || { echo "[autoresearch] missing harness file: $f" >&2; exit 1; }
+done
+command -v python3 >/dev/null 2>&1 || { echo "[autoresearch] python3 required on operator box" >&2; exit 1; }
+
+# Fingerprint of everything that can change the engine's argv or environment.
+HASH="$(cat "$ENV_FILE" "$SERVE" "$UNIT" | sha256sum | cut -d' ' -f1)"
+echo "[autoresearch] candidate fingerprint $HASH" >&2
+
+ssh_run() { ssh -o BatchMode=yes -o ConnectTimeout=15 "$HOST" "$@"; }
+
+emit_metrics() { # $1 = run.json
+    python3 -c '
+import json, sys
+run = json.loads(sys.argv[1])
+m = run["measure"]
+print("METRIC decode_tps=%s" % m["decode_tps"])
+for k in ("decode_tps_115", "decode_tps_1056", "decode_tps_4096"):
+    print("METRIC %s=%s" % (k, m[k]))
+print("METRIC ttft_ms_4096=%s" % m["ttft_ms_4096"])
+print("METRIC kv_tokens=%s" % (run.get("kv_tokens") or 0))
+print("METRIC boot_seconds=%s" % (run.get("boot_seconds") or -1))
+print("METRIC completion_tokens_4096=%s" % m.get("completion_tokens_4096", 0))
+' "$1"
+}
+
+# --- memo fast-path -------------------------------------------------------
+cached_hash="$(ssh_run "cat '$STAGE/last/hash' 2>/dev/null" 2>/dev/null || true)"
+if [ -n "$cached_hash" ] && [ "$cached_hash" = "$HASH" ]; then
+    cached_run="$(ssh_run "cat '$STAGE/last/run.json' 2>/dev/null" || true)"
+    if [ -n "$cached_run" ]; then
+        echo "[autoresearch] memo hit — cached metrics for $HASH, fleet untouched" >&2
+        emit_metrics "$cached_run"
+        exit 0
+    fi
+fi
+
+# --- stage the candidate ----------------------------------------------------
+echo "[autoresearch] staging candidate on $HOST:$STAGE" >&2
+ssh_run "mkdir -p '$STAGE'"
+scp -q "$ENV_FILE" "$HOST:$STAGE/frank-primary.env"
+scp -q "$SERVE" "$HOST:$STAGE/serve.sh"
+scp -q "$MEASURE" "$HOST:$STAGE/measure.py"
+scp -q "$WINDOW" "$HOST:$STAGE/window.sh"
+ssh_run "printf '%s\n' '$HASH' > '$STAGE/hash'"
+
+# --- the window -------------------------------------------------------------
+# Budgets: bench boot up to 20 min (warm boots measured ~7 min), measure up
+# to 30 min, restore boot up to 20 min — call run_experiment with
+# timeout_seconds >= 3600.
+echo "[autoresearch] opening bench window (stop+mask -> boot -> measure -> restore)" >&2
+set +e
+ssh_run "bash '$STAGE/window.sh'"
+rc=$?
+set -e
+if [ "$rc" -ne 0 ]; then
+    echo "[autoresearch] window failed (rc=$rc); host-side evidence:" >&2
+    ssh_run "tail -60 '$STAGE/fail.log' 2>/dev/null || tail -40 '$STAGE/serve.out' 2>/dev/null || tail -40 '$STAGE/measure.out' 2>/dev/null || true" >&2 || true
+    exit 1
+fi
+
+run_json="$(ssh_run "cat '$STAGE/run.json'")" || {
+    echo "[autoresearch] could not read $STAGE/run.json back" >&2; exit 1; }
+echo "[autoresearch] window result:" >&2
+printf '%s\n' "$run_json" >&2
+emit_metrics "$run_json"
+exit 0

old/download-frank-large-dflash.sh

@@ -0,0 +1,76 @@
+#!/usr/bin/env bash
+# Download the z-lab Qwen3.5-9B-DFlash drafter for frank-large's DFlash
+# speculative-decode arm (vLLM --speculative-config method=dflash). Public
+# repo — no auth header on the command line.
+#
+# The drafter is a 6-layer qwen3 draft whose dflash_config pins
+# target_layer_ids [1,5,9,13,17,21,25,29] and block_size 16; its hidden_size
+# 4096 / vocab_size 248320 match the MiMo-V2.6-Distill-Qwen-9B target exactly
+# (both preserve the Qwen3.5-9B geometry), which is what lets vLLM load it as
+# the draft for that target.
+#
+# Usage:
+#   scripts/download-frank-large-dflash.sh [DEST_DIR] [--verify-only]
+set -euo pipefail
+
+REPO="z-lab/Qwen3.5-9B-DFlash"
+BASE="https://huggingface.co/${REPO}/resolve/main"
+
+DEST=""
+VERIFY_ONLY=0
+for arg in "$@"; do
+    case "$arg" in
+        --verify-only) VERIFY_ONLY=1 ;;
+        *) DEST="$arg" ;;
+    esac
+done
+DEST="${DEST:-/home/dank/models/Qwen3.5-9B-DFlash}"
+
+# name bytes sha256(- = non-LFS, size-only check)
+FILES=(
+  "model.safetensors 2583816465 0a42274b32554f48de1faa0d42824e9c2ceda649c30ae0a731cddf410dd698c7"
+  "config.json 1222 -"
+)
+
+mkdir -p "$DEST"
+status=0
+for entry in "${FILES[@]}"; do
+  read -r name size sha <<<"$entry"
+  f="$DEST/$name"
+  have=0
+  if [[ -f "$f" ]] && [[ "$(stat -c%s "$f")" == "$size" ]]; then
+    if [[ "$sha" == "-" ]] || sha256sum -c --quiet <(printf '%s  %s\n' "$sha" "$f"); then
+      have=1
+    fi
+  fi
+  if (( have )); then
+    echo "OK        $name"
+    continue
+  fi
+  if (( VERIFY_ONLY )); then
+    echo "MISSING   $name"
+    status=1
+    continue
+  fi
+  echo "FETCHING  $name"
+  tmp="$f.part"
+  curl -fsSL --fail-with-body -o "$tmp" "$BASE/$name"
+  mv "$tmp" "$f"
+  if [[ "$(stat -c%s "$f")" != "$size" ]]; then
+    echo "SIZE FAIL $name ($(stat -c%s "$f") != $size)"
+    status=1
+    continue
+  fi
+  if [[ "$sha" != "-" ]] && ! sha256sum -c --quiet <(printf '%s  %s\n' "$sha" "$f"); then
+    echo "SHA FAIL  $name"
+    status=1
+    continue
+  fi
+  echo "VERIFIED  $name"
+done
+if (( status == 0 )); then
+  echo "DOWNLOAD_DONE $DEST"
+else
+  echo "DOWNLOAD_FAILED $DEST" >&2
+fi
+exit "$status"

old/download-frank-large-mimo.sh

@@ -0,0 +1,90 @@
+#!/usr/bin/env bash
+# Download the BF16 MiMo-V2.6-Distill-Qwen-9B payload for frank-large
+# (vLLM/kyuz0 gfx1151, Strix Halo iGPU). Public repo — no auth header on the
+# command line (a Bearer token in argv is world-readable via /proc).
+#
+# Per-file FRESH downloads, no resume: the house rule (see
+# download-frank-small.sh); -f/--fail-with-body plus size+sha256 against the
+# Hub LFS oid is what makes an access error fail instead of "succeeding".
+#
+# Usage:
+#   scripts/download-frank-large-mimo.sh [DEST_DIR] [--verify-only]
+#
+# The verified set below is exactly what the 2026-10-01 bench window served:
+# 17.53 GiB of BF16 weights (4 shards, 18,819,720,848 B total with the rest).
+set -euo pipefail
+
+REPO="XiaomiMiMo/MiMo-V2.6-Distill-Qwen-9B"
+BASE="https://huggingface.co/${REPO}/resolve/main"
+
+DEST=""
+VERIFY_ONLY=0
+for arg in "$@"; do
+    case "$arg" in
+        --verify-only) VERIFY_ONLY=1 ;;
+        *) DEST="$arg" ;;
+    esac
+done
+DEST="${DEST:-/home/dank/models/MiMo-V2.6-Distill-Qwen-9B-BF16}"
+
+# name bytes sha256(- = non-LFS, size-only check)
+FILES=(
+  "model-00001-of-00004.safetensors 5276436216 aab052180118aee34abc3029b54eaa49096aac606b97d703866b420dceb703c3"
+  "model-00002-of-00004.safetensors 5033171280 7a0486565f06d25ac4628e9dba470dc3f604353471d240d5a0bf7128f64df396"
+  "model-00003-of-00004.safetensors 5234499504 6c73207563d1879bfd6c143a028cc70be68edff56280458c71a43df4240300f4"
+  "model-00004-of-00004.safetensors 3275613848 1379a7cf8c0b8555a39ab65a47e830e0eb45e776b46045734da3afd73c09eea2"
+  "model.safetensors.index.json 69217 -"
+  "config.json 2784 -"
+  "generation_config.json 126 -"
+  "chat_template.jinja 3916 -"
+  "tokenizer.json 19989325 06b9509352d2af50381ab2247e083b80d32d5c0aba91c272ca9ff729b6a0e523"
+  "tokenizer_config.json 1165 -"
+  "vocab.json 6722759 -"
+  "merges.txt 3353259 -"
+  "preprocessor_config.json 443 -"
+  "processor_config.json 1191 -"
+  "video_preprocessor_config.json 385 -"
+)
+
+mkdir -p "$DEST"
+status=0
+for entry in "${FILES[@]}"; do
+  read -r name size sha <<<"$entry"
+  f="$DEST/$name"
+  have=0
+  if [[ -f "$f" ]] && [[ "$(stat -c%s "$f")" == "$size" ]]; then
+    if [[ "$sha" == "-" ]] || sha256sum -c --quiet <(printf '%s  %s\n' "$sha" "$f"); then
+      have=1
+    fi
+  fi
+  if (( have )); then
+    echo "OK        $name"
+    continue
+  fi
+  if (( VERIFY_ONLY )); then
+    echo "MISSING   $name"
+    status=1
+    continue
+  fi
+  echo "FETCHING  $name"
+  tmp="$f.part"
+  curl -fsSL --fail-with-body -o "$tmp" "$BASE/$name"
+  mv "$tmp" "$f"
+  if [[ "$(stat -c%s "$f")" != "$size" ]]; then
+    echo "SIZE FAIL $name ($(stat -c%s "$f") != $size)"
+    status=1
+    continue
+  fi
+  if [[ "$sha" != "-" ]] && ! sha256sum -c --quiet <(printf '%s  %s\n' "$sha" "$f"); then
+    echo "SHA FAIL  $name"
+    status=1
+    continue
+  fi
+  echo "VERIFIED  $name"
+done
+if (( status == 0 )); then
+  echo "DOWNLOAD_DONE $DEST"
+else
+  echo "DOWNLOAD_FAILED $DEST" >&2
+fi
+exit "$status"

old/download-frank-large-qwen35.sh

@@ -0,0 +1,85 @@
+#!/usr/bin/env bash
+# Download the source Qwen3.5-9B checkpoint — the ONLY place this model
+# family's MTP head (15 mtp.* tensors) exists. frank-large runs it as the
+# vLLM cross-model MTP draft (MiMo-V2.6-Distill-Qwen-9B itself ships zero
+# mtp tensors; that is why SGLang's NEXTN on frank-small draws 0.00
+# acceptance). Public repo — no auth header on the command line.
+#
+# NOTE the odd upstream shard names (model.safetensors-0000N-of-00004...).
+#
+# Usage:
+#   scripts/download-frank-large-qwen35.sh [DEST_DIR] [--verify-only]
+set -euo pipefail
+
+REPO="Qwen/Qwen3.5-9B"
+BASE="https://huggingface.co/${REPO}/resolve/main"
+
+DEST=""
+VERIFY_ONLY=0
+for arg in "$@"; do
+    case "$arg" in
+        --verify-only) VERIFY_ONLY=1 ;;
+        *) DEST="$arg" ;;
+    esac
+done
+DEST="${DEST:-/home/dank/models/Qwen3.5-9B}"
+
+# name bytes sha256(- = non-LFS, size-only check)
+FILES=(
+  "model.safetensors-00001-of-00004.safetensors 5276436216 db6f444b43d318c92f360a13a25561a6a65b10c0631b8ed305a426dbaa6c380e"
+  "model.safetensors-00002-of-00004.safetensors 5335161512 31c7d7e2dd5d207840b31cc59083c8f4c4718959149e0358c0364052bb9a0330"
+  "model.safetensors-00003-of-00004.safetensors 5368717440 7ec36ba3a4176a44c3c0876ad80c56a2f70c84bf008d82e9501df642f17dadec"
+  "model.safetensors-00004-of-00004.safetensors 3325995712 b62b0c4cd7e44edee103ee8f4fe225f246d5e768e07bfd5f25b63a8aa1fdd0c6"
+  "model.safetensors.index.json 79657 -"
+  "config.json 3126 -"
+  "chat_template.jinja 7756 -"
+  "tokenizer.json 12807982 5f9e4d4901a92b997e463c1f46055088b6cca5ca61a6522d1b9f64c4bb81cb42"
+  "tokenizer_config.json 16710 -"
+  "vocab.json 6722759 -"
+  "merges.txt 3353259 -"
+  "preprocessor_config.json 390 -"
+  "video_preprocessor_config.json 385 -"
+)
+
+mkdir -p "$DEST"
+status=0
+for entry in "${FILES[@]}"; do
+  read -r name size sha <<<"$entry"
+  f="$DEST/$name"
+  have=0
+  if [[ -f "$f" ]] && [[ "$(stat -c%s "$f")" == "$size" ]]; then
+    if [[ "$sha" == "-" ]] || sha256sum -c --quiet <(printf '%s  %s\n' "$sha" "$f"); then
+      have=1
+    fi
+  fi
+  if (( have )); then
+    echo "OK        $name"
+    continue
+  fi
+  if (( VERIFY_ONLY )); then
+    echo "MISSING   $name"
+    status=1
+    continue
+  fi
+  echo "FETCHING  $name"
+  tmp="$f.part"
+  curl -fsSL --fail-with-body -o "$tmp" "$BASE/$name"
+  mv "$tmp" "$f"
+  if [[ "$(stat -c%s "$f")" != "$size" ]]; then
+    echo "SIZE FAIL $name ($(stat -c%s "$f") != $size)"
+    status=1
+    continue
+  fi
+  if [[ "$sha" != "-" ]] && ! sha256sum -c --quiet <(printf '%s  %s\n' "$sha" "$f"); then
+    echo "SHA FAIL  $name"
+    status=1
+    continue
+  fi
+  echo "VERIFIED  $name"
+done
+if (( status == 0 )); then
+  echo "DOWNLOAD_DONE $DEST"
+else
+  echo "DOWNLOAD_FAILED $DEST" >&2
+fi
+exit "$status"

old/download-frank-small-sglang-int8.sh

@@ -0,0 +1,93 @@
+#!/usr/bin/env bash
+# Download MiMo-V2.6-Distill-Qwen-9B-INT8-W8A16-AutoRound for frank-small
+# (SGLang, RTX 3090). Public repo — no auth header on the command line.
+#
+# Per-file FRESH downloads, no resume: `curl -C -` once produced a 213 MB
+# oversized file on a 31 GiB payload; a fresh -f fetch plus size+sha256 is
+# the house rule.
+#
+# Usage:
+#   scripts/download-frank-small.sh [DEST_DIR] [--verify-only]
+#
+# LFS shards are verified against the Hub LFS sha256; non-LFS text files
+# against their declared byte size. Idempotent: complete files are skipped.
+set -euo pipefail
+
+REPO="letechlead/MiMo-V2.6-Distill-Qwen-9B-INT8-W8A16-AutoRound"
+BASE="https://huggingface.co/${REPO}/resolve/main"
+
+DEST=""
+VERIFY_ONLY=0
+for arg in "$@"; do
+    case "$arg" in
+        --verify-only) VERIFY_ONLY=1 ;;
+        *) DEST="$arg" ;;
+    esac
+done
+DEST="${DEST:-/home/dank/models/MiMo-V2.6-Distill-Qwen-9B-INT8-W8A16-AutoRound}"
+
+# name bytes sha256(- = non-LFS, size-only check)
+FILES=(
+  "model-00001-of-00008.safetensors 2946296312 ddb6117c04cb6feaf7c265e5d75efadf21949037d73253a8e4521d40e164366d"
+  "model-00002-of-00008.safetensors 1402134792 2dc78d55d1d33d7b3b21dd6f8b4b817b62c2fb1376e6448b322428798c10e257"
+  "model-00003-of-00008.safetensors 1427365904 5c797d14e345309d67c78306b9de4aa01a21dbd92dd98e40b0ad69b2900e9203"
+  "model-00004-of-00008.safetensors 1420041856 7402f852e2c7a24ebf4694b98a220e71dfb2c93ba3776272cf2acfc65acd8126"
+  "model-00005-of-00008.safetensors 1410655648 991cbf2eda2ff91fc3ffbb15591d7fcac340fb10d9c33bd3da20010e2f3f4090"
+  "model-00006-of-00008.safetensors 1427627856 efc42448054cfb5e3061e154ba4539204088ce95176d689a95b58de885ef7eeb"
+  "model-00007-of-00008.safetensors 2035442408 75a101fc71ae6f2e1f5c4816f711d62cf1d7a0e02ef6e5182361b8c11e72b626"
+  "model-00008-of-00008.safetensors 822153664 0fb90cb50e2d336fbad58bd4c01880eeddbf537c1dca1cf0ae291195d5efbb4b"
+  "tokenizer.json 19989325 06b9509352d2af50381ab2247e083b80d32d5c0aba91c272ca9ff729b6a0e523"
+  "model.safetensors.index.json 106278 -"
+  "config.json 4608 -"
+  "generation_config.json 161 -"
+  "quantization_config.json 1654 -"
+  "chat_template.jinja 3916 -"
+  "tokenizer_config.json 1168 -"
+  "preprocessor_config.json 443 -"
+  "processor_config.json 1191 -"
+  "README.md 8448 -"
+)
+
+mkdir -p "$DEST"
+status=0
+for entry in "${FILES[@]}"; do
+  read -r name size sha <<<"$entry"
+  f="$DEST/$name"
+  have=0
+  if [[ -f "$f" ]] && [[ "$(stat -c%s "$f")" == "$size" ]]; then
+    if [[ "$sha" == "-" ]] || sha256sum -c --quiet <(printf '%s  %s\n' "$sha" "$f"); then
+      have=1
+    fi
+  fi
+  if (( have )); then
+    echo "OK        $name"
+    continue
+  fi
+  if (( VERIFY_ONLY )); then
+    echo "MISSING   $name"
+    status=1
+    continue
+  fi
+  echo "FETCHING  $name"
+  tmp="$f.part"
+  # no -C resume; -f + --fail-with-body so an HTTP error is not a success
+  curl -fsSL --fail-with-body -o "$tmp" "$BASE/$name"
+  mv "$tmp" "$f"
+  if [[ "$(stat -c%s "$f")" != "$size" ]]; then
+    echo "SIZE FAIL $name ($(stat -c%s "$f") != $size)"
+    status=1
+    continue
+  fi
+  if [[ "$sha" != "-" ]] && ! sha256sum -c --quiet <(printf '%s  %s\n' "$sha" "$f"); then
+    echo "SHA FAIL  $name"
+    status=1
+    continue
+  fi
+  echo "VERIFIED  $name"
+done
+if (( status == 0 )); then
+  echo "DOWNLOAD_DONE $DEST"
+else
+  echo "DOWNLOAD_FAILED $DEST" >&2
+fi
+exit "$status"

scripts/autoresearch-bench.py

@@ -0,0 +1,141 @@
+#!/usr/bin/env python3
+"""autoresearch-bench.py — deterministic greedy battery for frank-large.
+
+Runs ON Dankclaw (no LAN jitter) against the candidate container on :7770.
+Fixed embedded corpus, temperature 0, streaming so decode speed is measured
+between real token arrivals (not request totals). Median of REPS samples per
+metric is the reported value; medians are far more stable than means under
+GPU thermal drift.
+
+Metrics emitted (stdout, one per line):
+  METRIC decode_tokens_per_sec=<float>   primary, higher better
+  METRIC prefill_tokens_per_sec=<float>  secondary
+  METRIC ttft_ms=<float>                 secondary, lower better
+Exit non-zero if the endpoint is unhealthy or the served alias is wrong.
+"""
+import argparse
+import json
+import statistics
+import sys
+import time
+import urllib.request
+
+# Fixed corpus: deterministic, ~28k tokens after tokenization (same scale as
+# the 2026-10-01 bench window). Built by repeating a fixed paragraph block a
+# fixed number of times - NO random, NO clock, NO network.
+_BLOCK = (
+    "The rendering pipeline walks the scene graph once per frame, resolving "
+    "material bindings into descriptor sets before any draw call is recorded. "
+    "Because the allocator recycles command buffers only after their fence "
+    "signals, a submission that outlives its frame must pin the pool entry; "
+    "otherwise the next frame overwrites recorded commands in flight. The "
+    "validation layer flags exactly this class of lifetime bug, but only for "
+    "buffers it can see - GPU-side timestamps and queue-family ownership "
+    "transfers hide in the driver's own bookkeeping, which is why release "
+    "builds must lean on explicit barriers and why a frame-time regression "
+    "often shows up as corruption one frame later than its cause.\n"
+)
+LONG_CTX = _BLOCK * 220  # ~28k-token prefill, matching the deployed workload
+_LONG = LONG_CTX + "\nNow, in one focused paragraph (80-150 words), explain " \
+    "which lifetime bugs this pipeline can still suffer and what explicit " \
+    "mitigations the text implies."
+_SHORT = "Design a rate limiter for a homelab reverse proxy that must not " \
+    "lock out the LAN. Think it through, then give the algorithm."
+
+REPS = 3
+MAX_TOKENS = 256
+REQ_TIMEOUT = 900
+
+
+def stream_completion(url: str, model: str, prompt: str):
+    """Greedy streaming completion.
+
+    Returns (ttft_s, decode_tps, prefill_tps, prompt_tokens, n_tokens).
+    ttft is RELATIVE to request send: time.perf_counter()'s epoch is host
+    boot, and returning the raw stamp read as an 11-day first token.
+    """
+    body = json.dumps({
+        "model": model,
+        "messages": [{"role": "user", "content": prompt}],
+        "max_tokens": MAX_TOKENS,
+        "temperature": 0,
+        "stream": True,
+        "stream_options": {"include_usage": True},
+    }).encode()
+    req = urllib.request.Request(
+        url.rstrip("/") + "/v1/chat/completions", data=body,
+        headers={"Content-Type": "application/json"})
+    t0 = time.perf_counter()
+    first = None
+    stamps = []
+    usage = {}
+    with urllib.request.urlopen(req, timeout=REQ_TIMEOUT) as resp:
+        for raw in resp:
+            line = raw.decode(errors="replace").strip()
+            if not line.startswith("data:"):
+                continue
+            payload = line[5:].strip()
+            if payload == "[DONE]":
+                break
+            chunk = json.loads(payload)
+            if chunk.get("usage"):
+                usage = chunk["usage"]
+            for ch in chunk.get("choices", []):
+                d = ch.get("delta") or {}
+                # reasoning AND content deltas both arrive token by token;
+                # count either so greedy text length drives the rate.
+                # (llama.cpp names the thinking field reasoning_content.)
+                if d.get("content") or d.get("reasoning") or d.get("reasoning_content"):
+                    now = time.perf_counter()
+                    if first is None:
+                        first = now
+                    stamps.append(now)
+    if first is None or len(stamps) < 2:
+        # A zero-argmax degenerate answer would poison the median; raise.
+        raise RuntimeError(f"stream produced {len(stamps)} tokens")
+    ttft_s = first - t0
+    decode_s = stamps[-1] - stamps[0]
+    decode_tps = (len(stamps) - 1) / decode_s if decode_s > 0 else 0.0
+    prompt_tokens = usage.get("prompt_tokens") or 0
+    prefill_tps = prompt_tokens / ttft_s if prompt_tokens else 0.0
+    return ttft_s, decode_tps, prefill_tps, prompt_tokens, len(stamps)
+
+
+def main():
+    ap = argparse.ArgumentParser()
+    ap.add_argument("--url", default="http://127.0.0.1:7770")
+    ap.add_argument("--model", default="frank-large")
+    args = ap.parse_args()
+
+    with urllib.request.urlopen(args.url.rstrip("/") + "/v1/models",
+                                timeout=10) as r:
+        ids = [m["id"] for m in json.load(r)["data"]]
+    if args.model not in ids:
+        print(f"FATAL: served ids {ids}, expected {args.model}",
+              file=sys.stderr)
+        sys.exit(2)
+
+    # Warmup (compile/graph warm + allocator settle), NOT measured.
+    stream_completion(args.url, args.model, _SHORT)
+
+    decodes, prefills, ttfts = [], [], []
+    for rep in range(REPS):
+        for name, prompt in (("long", _LONG), ("short", _SHORT)):
+            ttft, dt, pt, ptk, ntok = stream_completion(
+                args.url, args.model, prompt)
+            decodes.append(dt)
+            ttfts.append(ttft * 1000.0)
+            if name == "long":
+                prefills.append(pt)
+            print(f"  rep{rep} {name:5s} ttft={ttft*1000:8.1f}ms "
+                  f"prefill={pt:8.1f} t/s decode={dt:6.2f} t/s "
+                  f"({ntok} tok, prompt {ptk})", file=sys.stderr)
+
+    med = lambda xs: statistics.median(xs)
+    print(f"METRIC decode_tokens_per_sec={med(decodes):.2f}")
+    print(f"METRIC prefill_tokens_per_sec={med(prefills):.1f}")
+    print(f"METRIC ttft_ms={med(ttfts):.1f}")
+
+
+if __name__ == "__main__":
+    main()

scripts/autoresearch/frank-primary-measure.py

@@ -0,0 +1,192 @@
+#!/usr/bin/env python3
+"""frank-primary decode measurement — runs INSIDE the engine container.
+
+The method is the fleet's established one (the 2026-09-29 / 2026-10-01
+tuning campaigns, /home/dank/mtp_measure.py): SSE chunk counting is invalid
+under speculative decoding because rejected drafts change the accounting, so
+decode is measured from the server's own usage counters:
+
+    decode_tok_s = completion_tokens / (wall - ttft)
+
+with ttft = wall time to the FIRST content-bearing chunk (thinking arrives in
+`reasoning`/`delta.reasoning` on this stack and is excluded from the decode
+window — and is deterministic per prompt at temperature 0, so the exclusion
+cancels in any A/B on the same workload).
+
+Workload (fixed, deterministic, offline):
+  * prompts built from a large fixed-seed shuffled word list so tokens rarely
+    repeat (no n-gram/prefix predictability of the text itself), sized with
+    the server's own /v1/messages/count_tokens counter
+  * depths 115 / 1056 / 4096 target prompt tokens
+  * one 64-token warm pass per depth (warms the expert arena), then
+    max_tokens=256 at temperature 0, 3 runs, median
+Everything is greedy with a fixed seed — the same workload every run, no
+network, no time-of-day dependence.
+
+Env:
+  AR_BASE             endpoint (default http://127.0.0.1:7770)
+  AR_MODEL            served name (default frank-primary)
+  AR_PROMPT_CEILING   hard prompt ceiling in tokens (default 100000)
+
+Output: progress on stderr, ONE json object on stdout (the last line).
+Exit codes: 0 ok, 2 endpoint unhealthy / no usable completion, 3 bad usage.
+"""
+import json
+import os
+import random
+import statistics
+import sys
+import time
+import urllib.error
+import urllib.request
+
+BASE = os.environ.get("AR_BASE", "http://127.0.0.1:7770")
+MODEL = os.environ.get("AR_MODEL", "frank-primary")
+PROMPT_CEILING = int(os.environ.get("AR_PROMPT_CEILING", "100000"))
+DEPTHS = (115, 1056, 4096)
+WARM_TOKENS = 64
+MAX_TOKENS = 256
+RUNS = 3
+
+# 43-word x 12 = 516-word vocabulary, seed 1234 — the campaign's exact list
+# and seed, so prompts are byte-identical to the historical measurements.
+_VOCAB = ("alpha bravo charlie delta echo foxtrot golf hotel india juliet kilo lima "
+          "mike november oscar papa quebec romeo sierra tango uniform victor whiskey "
+          "xray yankee zulu cardinal ordinal topology kernel scheduler resident "
+          "unified aperture expert arena slab offload prefetch gemm triton quant "
+          "tensor shard checkpoint manifest embed router cache page table window "
+          "gradient sparse dense moe attention head dim block stream decode prefill "
+          "bandwidth latency throughput residency eviction tier swap pin memlock").split()
+_R = random.Random(1234)
+_ORDER = list(range(len(_VOCAB)))
+_R.shuffle(_ORDER)
+
+
+def log(msg):
+    print("[measure] %s" % msg, file=sys.stderr, flush=True)
+
+
+def post_json(path, body, timeout):
+    req = urllib.request.Request(
+        BASE + path, data=json.dumps(body).encode(),
+        headers={"Content-Type": "application/json"})
+    with urllib.request.urlopen(req, timeout=timeout) as r:
+        return json.load(r)
+
+
+def count_tokens(text):
+    """Token count without a prefill, from the server's own counter."""
+    body = {"model": MODEL,
+            "messages": [{"role": "user", "content": text}]}
+    r = post_json("/v1/messages/count_tokens", body, timeout=300)
+    n = r.get("input_tokens") or r.get("count") or r.get("token_count")
+    if not n:
+        raise RuntimeError("count_tokens returned no count: %r" % (r,))
+    return int(n)
+
+
+def words(n):
+    return " ".join(_VOCAB[_ORDER[i % len(_VOCAB)]] for i in range(n))
+
+
+def calibrate():
+    """Measure words->tokens from the server's own counter."""
+    return 2000.0 / count_tokens(words(2000))
+
+
+def prompt_for(target_tokens, wpt):
+    n = max(1, int(target_tokens * wpt))
+    text = words(n)
+    pt = count_tokens(text)
+    while pt > PROMPT_CEILING:
+        n = max(1, int(n * (PROMPT_CEILING / pt) * 0.98))
+        text = words(n)
+        pt = count_tokens(text)
+    return text, pt
+
+
+def measured(prompt, max_tokens):
+    """Return (ttft_s, decode_tok_s, completion_tokens) via usage counters."""
+    body = {"model": MODEL,
+            "messages": [{"role": "user", "content": prompt}],
+            "max_tokens": max_tokens, "temperature": 0,
+            "stream": True, "stream_options": {"include_usage": True}}
+    req = urllib.request.Request(
+        BASE + "/v1/chat/completions", data=json.dumps(body).encode(),
+        headers={"Content-Type": "application/json"})
+    t0 = time.time()
+    ttft = None
+    usage = {}
+    with urllib.request.urlopen(req, timeout=1800) as r:
+        for raw in r:
+            line = raw.decode("utf-8", "replace").strip()
+            if not line.startswith("data:"):
+                continue
+            s = line[5:].strip()
+            if s == "[DONE]":
+                break
+            try:
+                ev = json.loads(s)
+            except json.JSONDecodeError:
+                continue
+            if ev.get("usage"):
+                usage = ev["usage"]
+            for ch in ev.get("choices", []):
+                if (ch.get("delta") or {}).get("content"):
+                    if ttft is None:
+                        ttft = time.time() - t0
+    wall = time.time() - t0
+    ct = int(usage.get("completion_tokens") or 0)
+    if ct <= 0:
+        # No usage on the stream: the endpoint is unhealthy or the model
+        # returned nothing (e.g. HTTP 200 with an error payload).
+        raise RuntimeError("no completion_tokens in usage: %r" % (usage,))
+    dec_window = max(wall - (ttft or 0.0), 1e-9)
+    return (ttft or wall), (ct / dec_window), ct
+
+
+def main():
+    log("base=%s model=%s ceiling=%d" % (BASE, MODEL, PROMPT_CEILING))
+    try:
+        wpt = calibrate()
+    except Exception as e:
+        log("endpoint unhealthy during calibration: %s" % e)
+        return 2
+    log("words/token: %.3f" % wpt)
+
+    result = {"words_per_token": round(wpt, 5)}
+    for tgt in DEPTHS:
+        try:
+            p, pt = prompt_for(tgt, wpt)
+            measured(p, WARM_TOKENS)  # warm the arena
+        except Exception as e:
+            log("warm pass failed at depth %d: %s" % (tgt, e))
+            return 2
+        rates, tts, cts = [], [], []
+        for i in range(RUNS):
+            try:
+                tt, r, ct = measured(p, MAX_TOKENS)
+            except Exception as e:
+                log("run %d/%d failed at depth %d: %s" % (i + 1, RUNS, tgt, e))
+                continue
+            rates.append(r)
+            tts.append(tt)
+            cts.append(ct)
+        if len(rates) < 2:
+            log("only %d usable run(s) at depth %d — refusing to median" % (len(rates), tgt))
+            return 2
+        result["decode_tps_%d" % tgt] = round(statistics.median(rates), 2)
+        result["ttft_ms_%d" % tgt] = round(1000.0 * statistics.median(tts), 1)
+        result["completion_tokens_%d" % tgt] = int(statistics.median(cts))
+        result["prompt_tokens_%d" % tgt] = pt
+        log("depth %5d tok  decode %.2f t/s  ttft %.3f s  (runs=%d)"
+            % (pt, result["decode_tps_%d" % tgt], result["ttft_ms_%d" % tgt] / 1000.0, len(rates)))
+
+    meds = [result["decode_tps_%d" % t] for t in DEPTHS]
+    result["decode_tps"] = round(statistics.median(meds), 2)
+    print(json.dumps(result))
+    return 0
+
+
+if __name__ == "__main__":
+    sys.exit(main())

scripts/autoresearch/frank-primary-window.sh

@@ -0,0 +1,200 @@
+#!/usr/bin/env bash
+# frank-primary-window.sh — one full bench window for frank-primary, run ON
+# DANKSERVER as the deploy user (invoked over ssh by ./autoresearch.sh).
+#
+# Discipline (AGENTS.md Benchmarking Policy): the production unit is
+# STOPPED + MASKED first, the candidate config boots AS THE SERVICE itself
+# (same serve script, same container name, same port 7770 — no scratch
+# ports, no co-tenant GPU), the in-container measure runs, and the window
+# ALWAYS ends by restoring the production unit and verifying its health.
+# The EXIT trap makes the restore unconditional: a boot failure, a measure
+# failure, or a dead ssh session all end with production back up.
+#
+# Staging layout (created by the orchestrator, ephemeral — never a config
+# source; the repo checkout is):
+#   /home/dank/.autoresearch-frank-primary/
+#     serve.sh             staged candidate serve script
+#     frank-primary.env    staged candidate env (the candidate knobs)
+#     measure.py           the in-container measure client
+#     hash                 candidate fingerprint
+#     run.json             result (metrics + boot + kv + fingerprint)
+#     serve.out/measure.out/fail.log
+#     last/                memo: run.json + hash of the last FULLY
+#                          successful window (bench AND verified restore)
+#
+# This script logs to stderr and exits 0 only when the measure produced a
+# usable result (the restore happens in the trap regardless of the bench
+# outcome; a failed restore forces exit 1).
+set -uo pipefail
+
+STAGE="${FRANK_PRIMARY_AR_STAGE:-/home/dank/.autoresearch-frank-primary}"
+UNIT="${FRANK_PRIMARY_AR_UNIT:-frank-primary}"
+CTR="${FRANK_PRIMARY_AR_CTR:-frank-primary-qwen}"
+HEALTH_URL="http://127.0.0.1:7770/health"
+MODELS_URL="http://127.0.0.1:7770/v1/models"
+BOOT_TIMEOUT="${FRANK_PRIMARY_AR_BOOT_TIMEOUT:-1200}"
+RESTORE_TIMEOUT="${FRANK_PRIMARY_AR_RESTORE_TIMEOUT:-1200}"
+MEASURE_TIMEOUT="${FRANK_PRIMARY_AR_MEASURE_TIMEOUT:-1800}"
+SERVED="$(sed -n 's/^FRANK_PRIMARY_SERVED_NAME=//p' "$STAGE/frank-primary.env" 2>/dev/null | head -1)"
+SERVED="${SERVED:-frank-primary}"
+
+RESTORED=0
+
+log() { printf '[window %s] %s\n' "$(date '+%H:%M:%S')" "$*" >&2; }
+
+container_state() { podman inspect -f '{{.State.Status}}' "$CTR" 2>/dev/null || echo absent; }
+
+wait_port_free() {
+    local i
+    for i in $(seq 1 90); do
+        [ "$(container_state)" != "running" ] && return 0
+        sleep 2
+    done
+    return 1
+}
+
+wait_health() { # $1 = timeout seconds
+    local t0 code
+    t0=$(date +%s)
+    while :; do
+        code="$(curl -s -o /dev/null -w '%{http_code}' --max-time 10 "$HEALTH_URL" 2>/dev/null || echo 000)"
+        [ "$code" = "200" ] && return 0
+        [ "$(( $(date +%s) - t0 ))" -ge "$1" ] && return 1
+        sleep 2
+    done
+}
+
+capture_bench_failure() {
+    log "capturing bench-boot failure evidence"
+    {
+        echo "=== podman logs (tail 400) ==="
+        podman logs --tail 400 "$CTR" 2>&1 || true
+        echo "=== staged serve.out (tail 60) ==="
+        tail -60 "$STAGE/serve.out" 2>/dev/null || true
+        echo "=== engine log (tail 60) ==="
+        tail -60 /home/dank/logs/frank_primary.log 2>/dev/null || true
+    } >"$STAGE/fail.log" 2>&1 || true
+    grep -E 'OutOfMemory|Tried to allocate|Engine core initialization failed|Error|error' \
+        "$STAGE/fail.log" 2>/dev/null | tail -25 >&2 || true
+}
+
+restore() {
+    [ "$RESTORED" = 1 ] && return 0
+    RESTORED=1
+    log "restoring production unit $UNIT"
+    if [ -f "$STAGE/serve.pid" ]; then
+        kill "$(cat "$STAGE/serve.pid" 2>/dev/null)" 2>/dev/null || true
+    fi
+    podman stop -t 120 "$CTR" >/dev/null 2>&1 || true
+    wait_port_free || log "WARNING: bench container still draining after 180s"
+    sudo systemctl unmask "$UNIT" 2>/dev/null || true
+    if ! sudo systemctl start "$UNIT"; then
+        log "ERROR: systemctl start $UNIT failed — operator intervention required"
+        return 1
+    fi
+    if wait_health "$RESTORE_TIMEOUT"; then
+        if curl -s --max-time 10 "$MODELS_URL" 2>/dev/null | grep -q "$SERVED"; then
+            log "production restored and verified ($SERVED answering on :7770)"
+            # Memo promotion lives HERE: a cached result is only ever the
+            # product of a fully successful window (bench AND verified
+            # restore), never of a bare bench success.
+            cp "$STAGE/run.json" "$STAGE/last/" 2>/dev/null || true
+            cp "$STAGE/hash" "$STAGE/last/" 2>/dev/null || true
+        else
+            log "ERROR: health 200 but /v1/models does not list $SERVED"
+            return 1
+        fi
+    else
+        log "ERROR: production not healthy within ${RESTORE_TIMEOUT}s — unit left to systemd (Restart=always)"
+        return 1
+    fi
+    return 0
+}
+trap 'restore; rc=$?; [ "$rc" -ne 0 ] && exit 1' EXIT
+
+main() {
+    local f
+    for f in serve.sh frank-primary.env measure.py hash; do
+        [ -f "$STAGE/$f" ] || { log "staged file missing: $STAGE/$f"; exit 1; }
+    done
+    mkdir -p "$STAGE/last"
+    rm -f "$STAGE/run.json" "$STAGE/fail.log" "$STAGE/measure.out"
+
+    local t_start t0 i kvline kv_tokens kv_conc measure_json boot_seconds
+    t_start="$(date +%s)"
+
+    log "stopping and masking production unit $UNIT"
+    sudo systemctl stop "$UNIT" 2>/dev/null || true
+    # Explicit stop is not auto-restarted even with Restart=always; the mask
+    # is belt-and-braces against a stray start during the window.
+    sudo systemctl mask "$UNIT" 2>/dev/null || \
+        log "WARNING: mask failed — relying on explicit-stop semantics"
+    wait_port_free || log "note: production container still draining"
+
+    t0="$(date +%s)"
+    log "starting bench instance from staged config (fingerprint $(tr -d '[:space:]' <"$STAGE/hash"))"
+    FRANK_PRIMARY_ENV_FILE="$STAGE/frank-primary.env" \
+        setsid nohup bash "$STAGE/serve.sh" >"$STAGE/serve.out" 2>&1 </dev/null &
+    echo "$!" >"$STAGE/serve.pid"
+
+    for i in $(seq 1 45); do
+        [ "$(container_state)" = "running" ] && break
+        sleep 2
+    done
+    if [ "$(container_state)" != "running" ]; then
+        log "bench container never reached running state"
+        tail -40 "$STAGE/serve.out" >&2 || true
+        exit 1
+    fi
+
+    if wait_health "$BOOT_TIMEOUT"; then
+        boot_seconds=$(( $(date +%s) - t0 ))
+        log "bench healthy after ${boot_seconds}s"
+    else
+        if [ "$(container_state)" = "running" ]; then
+            log "bench boot timed out after ${BOOT_TIMEOUT}s (still loading)"
+        else
+            log "bench container died during boot"
+            capture_bench_failure
+        fi
+        exit 1
+    fi
+
+    log "running measure in container (timeout ${MEASURE_TIMEOUT}s)"
+    if ! podman cp "$STAGE/measure.py" "$CTR":/tmp/ar-measure.py; then
+        log "podman cp of measure client failed"
+        exit 1
+    fi
+    if ! timeout "$MEASURE_TIMEOUT" podman exec "$CTR" python3 /tmp/ar-measure.py \
+            >"$STAGE/measure.out" 2>&1; then
+        log "measure failed or timed out"
+        tail -40 "$STAGE/measure.out" >&2 || true
+        exit 1
+    fi
+
+    # Capacity facts for the run (context-size experiments change these).
+    kvline="$(podman logs "$CTR" 2>&1 | grep -E 'GPU KV cache size' | tail -1 || true)"
+    kv_tokens="$(printf '%s' "$kvline" | sed -E 's/.*GPU KV cache size: ([0-9,]+) tokens.*/\1/' | tr -d ',' || true)"
+    kv_conc="$(printf '%s' "$kvline" | sed -E 's/.*per request: ([0-9.]+x).*/\1/' || true)"
+
+    measure_json="$(grep -E '^\{' "$STAGE/measure.out" | tail -1 || true)"
+    if [ -z "$measure_json" ]; then
+        log "measure produced no JSON result line"
+        tail -40 "$STAGE/measure.out" >&2 || true
+        exit 1
+    fi
+
+    # The measure line is already JSON, so the run object can be assembled
+    # without any host-side interpreter.
+    printf '{"hash":"%s","boot_seconds":%s,"kv_tokens":%s,"kv_concurrency":"%s","total_seconds":%s,"measure":%s}\n' \
+        "$(tr -d '[:space:]' <"$STAGE/hash")" \
+        "$boot_seconds" "${kv_tokens:-0}" "${kv_conc:-unknown}" \
+        "$(( $(date +%s) - t_start ))" "$measure_json" >"$STAGE/run.json"
+
+    log "bench window complete in $(( $(date +%s) - t_start ))s"
+    # Memo only after a FULL success; the restore below still has to verify.
+    exit 0
+}
+
+main "$@"
+exit "$?"

scripts/build-rocmfpx.sh

@@ -0,0 +1,89 @@
+#!/usr/bin/env bash
+# Build the ROCmFPX runtime that serves frank-large on Dankclaw.
+#
+#   bash scripts/build-rocmfpx.sh              # clone/fetch + build the pin
+#   bash scripts/build-rocmfpx.sh --verify     # only check the installed tree
+#
+# Runtime: kingjones30/ROCmFPX — charlie12345/ROCmFPX plus the model-card
+# patches, i.e. the fork that owns the ROCmFPx tensor formats. frank-large's
+# payload is ftype 115 (`Q8_0_ROCMFPX_AGENT`): 311 tensors of
+# GGML_TYPE_Q8_0_ROCMFPX (103), a type that exists ONLY in this fork, so
+# stock llama.cpp, gufo and every other loader on the host reject the file.
+# This is the same tree that served the Qwen3.8-Flash-Next ROCmFP4 STRIX
+# endpoint 2026-08-30 → 09-11 (see the kingjones runbook), rebuilt here so a
+# fresh host can be brought up from the repo alone.
+#
+# Why this commit and not a newer one: 36e9acd (2026-08-28) is what was
+# qualified against this payload on 2026-10-02 — it loads ftype 115, survives
+# warmup, and drafts with the MTP head at acceptance 0.65. The five legacy
+# builds of 2026-08-25 crashed this same file in ggml_compute_forward_set_rows
+# (see kingjones-q8agent-rocmfpx.md), so the pin is load-bearing: do not bump
+# it without re-running that crash test.
+#
+# Host notes:
+#   * -DLLAMA_BUILD_WEBUI=OFF is REQUIRED on this box. The tree provisions its
+#     embedded UI assets via npm or the HF llama-ui bucket; npm is absent here
+#     and the bucket download half-failed, aborting the link at 98%. The API
+#     server is unaffected.
+#   * The build links against the host ROCm (/opt/rocm); no bundled toolchain
+#     like the gfx1201 R9700 build needed.
+set -euo pipefail
+
+ROCMFPX_REPO="${ROCMFPX_REPO:-https://github.com/kingjones30/ROCmFPX.git}"
+# The commit frank-large was qualified with (measured 2026-10-02).
+ROCMFPX_COMMIT="${ROCMFPX_COMMIT:-36e9acd40e10a87cd3c3ef8ec734668757dc8520}"
+SRC_DIR="${ROCMFPX_SRC_DIR:-/home/dank/GitPackages/kingjones-ROCmFPX}"
+BUILD_DIR="${ROCMFPX_BUILD_DIR:-$SRC_DIR/build}"
+GPU_TARGET="${ROCMFPX_GPU_TARGET:-gfx1151}"
+JOBS="${JOBS:-$(nproc)}"
+
+# check_pin — refuse a source tree that is not the qualified commit. A moved
+# fork means different kernels under the same path, and on this model line a
+# wrong build is a segfault at first request, not a clean error.
+check_pin() {
+    local commit
+    commit="$(git -C "$SRC_DIR" rev-parse HEAD 2>/dev/null || true)"
+    if [[ -n "$commit" && "$commit" != "$ROCMFPX_COMMIT" ]]; then
+        echo "ROCMFPX_COMMIT mismatch: $SRC_DIR is at $commit, pinned $ROCMFPX_COMMIT" >&2
+        exit 1
+    fi
+}
+
+if [[ "${1:-}" == "--verify" ]]; then
+    check_pin
+    [[ -x "$BUILD_DIR/bin/llama-server" ]] || {
+        echo "no ROCmFPX llama-server at $BUILD_DIR/bin/llama-server" >&2
+        exit 1
+    }
+    # The format the endpoint exists for: the fork must carry the ROCmFPx
+    # tensor type, or the payload cannot load at all.
+    "$BUILD_DIR/bin/llama-server" --help 2>&1 | grep -q -- '--spec-type' || {
+        echo "$BUILD_DIR/bin/llama-server has no --spec-type: MTP drafting is impossible" >&2
+        exit 1
+    }
+    "$BUILD_DIR/bin/llama-server" --version | head -1
+    exit 0
+fi
+
+echo "== source: $ROCMFPX_REPO @ $ROCMFPX_COMMIT =="
+if [[ ! -d "$SRC_DIR/.git" ]]; then
+    mkdir -p "$(dirname "$SRC_DIR")"
+    git clone "$ROCMFPX_REPO" "$SRC_DIR"
+fi
+git -C "$SRC_DIR" fetch --quiet origin "$ROCMFPX_COMMIT" || true
+git -C "$SRC_DIR" checkout -q --detach "$ROCMFPX_COMMIT"
+check_pin
+echo "   $(git -C "$SRC_DIR" rev-parse HEAD)"
+
+echo "== configure + build (release, HIP $GPU_TARGET) =="
+cmake -S "$SRC_DIR" -B "$BUILD_DIR" \
+    -DGGML_HIP=ON \
+    -DGPU_TARGETS="$GPU_TARGET" \
+    -DGGML_NATIVE=ON \
+    -DCMAKE_BUILD_TYPE=Release \
+    -DLLAMA_BUILD_WEBUI=OFF
+cmake --build "$BUILD_DIR" --target llama-server llama-quantize -j "$JOBS"
+
+echo "== verify =="
+"$BUILD_DIR/bin/llama-server" --version | head -1
+echo "frank-large runtime ready: $BUILD_DIR/bin/llama-server ($ROCMFPX_COMMIT)"

scripts/deploy-remote.sh

@@ -1077,27 +1077,29 @@ deploy_anubis() {
     wait_container_healthy anubis 60 || return 1
 }
 
-# frank-large is gufo (gufo-org/gufo), the Strix Halo inference engine built
+# frank-large is the ROCmFPX fork of llama.cpp (kingjones30/ROCmFPX, pinned by
-# from the pinned v0.1.1 tag by scripts/build-gufo.sh, serving
+# scripts/build-rocmfpx.sh), serving
-# unsloth/Qwen3.8-27B-GGUF:UD-Q8_K_XL with the DFlash2 drafter
+# kingjones777/Qwen3.8-27B-ROCmFP4-STRIX-MTP-GGUF:Q8_0_ROCMFPX_AGENT with the
-# (z-lab/Qwen3.8-27B-DFlash2-GGUF:Q4_K_M) on the Strix Halo iGPU. It is a
+# repo's own Q4_0 MTP draft head and its BF16 vision projector, on the Strix
-# ROOT unit: the runtime is a native binary linked against the host ROCm,
+# Halo iGPU. It is a ROOT unit: the runtime is a native binary linked against
-# so there is no container image to install and no generated unit to dodge.
+# the host ROCm, so there is no container image to install and no generated
-# relink_unit points /etc/systemd/system/frank-large.service at the repo
+# unit to dodge. relink_unit points /etc/systemd/system/frank-large.service at
-# file and boot persistence is that unit's [Install] section.
+# the repo file and boot persistence is that unit's [Install] section.
 #
-# The runtime gate matters as much as the payload gate: gufo is built from
+# The runtime gate matters as much as the payload gate: the fork is built from
-# source on this host (there is no prebuilt release), so a machine whose
+# source on this host, and the payload is ftype 115 (tensor type 103,
+# GGML_TYPE_Q8_0_ROCMFPX) which only this fork can read — so a machine whose
 # /home/dank/GitPackages tree was never built has a unit that can never
 # become healthy, and a green deploy would leave the endpoint dark.
 #
-# It replaces the stock llama.cpp prebuilt (unsloth b11030), which had
+# It replaces gufo v0.1.1 + unsloth UD-Q4_K_XL (the rollback target, still
-# replaced the Kairic Edge runtime (ciru-ai/ROCmFPX). The port gate is the
+# built by scripts/build-gufo.sh with its payload on disk), which had replaced
-# same as every cutover since 2026-09-16: nothing starts until 7770 is
+# the stock llama.cpp prebuilt. The port gate is the same as every cutover
-# released.
+# since 2026-09-16: nothing starts until 7770 is released.
-LLM_LARGE_RUNTIME="${LLM_LARGE_RUNTIME:-/home/dank/GitPackages/gufo-v0.1.1/bin/gufo}"
+LLM_LARGE_RUNTIME="${LLM_LARGE_RUNTIME:-/home/dank/GitPackages/kingjones-ROCmFPX/build/bin/llama-server}"
-LLM_LARGE_MODEL="${LLM_LARGE_MODEL:-/home/dank/models/Swift-1.5-Qwen3.8-27B-Uncensored-MTP-UD-Q8_K_XL/Swift-1.5-Qwen3.8-27B-Uncensored-Dynamic-MTP-UD-Q8_K_XL.gguf}"
+LLM_LARGE_MODEL="${LLM_LARGE_MODEL:-/home/dank/models/kingjones-Qwen3.8-27B-ROCmFP4-STRIX-MTP/Qwen3.8-27B-Q8_0_ROCMFPX_AGENT.gguf}"
-LLM_LARGE_DRAFT="${LLM_LARGE_DRAFT:-/home/dank/models/Qwen3.8-27B-UD-Q8_K_XL/Qwen3.8-27B-DFlash2-Q4_K_M.gguf}"
+LLM_LARGE_DRAFT="${LLM_LARGE_DRAFT:-/home/dank/models/kingjones-Qwen3.8-27B-ROCmFP4-STRIX-MTP/mtp-Qwen3.8-27B-Q4_0.gguf}"
+LLM_LARGE_MMPROJ="${LLM_LARGE_MMPROJ:-/home/dank/models/kingjones-Qwen3.8-27B-ROCmFP4-STRIX-MTP/mmproj-Qwen3.8-27B-BF16.gguf}"
 
 retire_halogen_large_quadlet() {
     local trash="$HOME/.AGENTTRASH/20260919-orca-cutover/quadlet"
@@ -1132,17 +1134,19 @@ wait_port_free() {
 deploy_llm_large() {
     # Refuse rather than restart-loop. A unit whose runtime or payload is
     # missing never becomes healthy, and a green deploy would leave the
-    # endpoint dark with nothing in the deploy log to say why.
+    # endpoint dark with nothing in the deploy log to say why. The gates
+    # mirror serve-frank-large.sh's own predicates exactly: runtime
+    # executable, target GGUF, MTP head, mmproj.
     if [[ ! -x "$LLM_LARGE_RUNTIME" ]]; then
-        echo "ERROR: gufo runtime absent at $LLM_LARGE_RUNTIME" >&2
+        echo "ERROR: ROCmFPX runtime absent at $LLM_LARGE_RUNTIME" >&2
-        echo "frank-large: build it: bash scripts/build-gufo.sh" >&2
+        echo "frank-large: build it: bash scripts/build-rocmfpx.sh" >&2
         return 1
     fi
     local missing=0 f
-    for f in "$LLM_LARGE_MODEL" "$LLM_LARGE_DRAFT"; do
+    for f in "$LLM_LARGE_MODEL" "$LLM_LARGE_DRAFT" "$LLM_LARGE_MMPROJ"; do
         if [[ ! -f "$f" ]]; then
             echo "ERROR: payload absent at $f" >&2
-            echo "frank-large: fetch it: bash scripts/download-frank-large.sh" >&2
+            echo "frank-large: fetch it: bash scripts/download-frank-large-kingjones.sh" >&2
             missing=1
         fi
     done
@@ -1152,18 +1156,14 @@ deploy_llm_large() {
     # On a re-deploy the predecessor is THIS unit: stop it before waiting on
     # the port, or the wait deadlocks on the server it just stopped serving.
     # Then sweep a stray manual run -- that is how a bench harness leaves the
-    # port busy. Match the DRAFTER path, which every frank-large command
+    # port busy. Match the TARGET GGUF name, which every frank-large command
-    # line carries via --dflash-model and which is UNCHANGED by the 2026-09-28
+    # line carries via --model and which no other process on this host uses:
-    # payload swap: the target moved to a different repository while the
+    # frank-small also runs llama-server here (with the MiMo GGUF) and must
-    # drafter stayed put, so keying on the target filename would have quietly
+    # not be killed, so keying on the binary name would take the 3090 endpoint
-    # stopped matching. It is also the right key in principle — it names this
+    # down with it. The pattern never appears in the deploying shell's own
-    # endpoint's speculative drafter, not a binary. Match that, not a binary
+    # command line, so -f is safe here (it is not in `bash deploy-remote.sh`).
-    # name: frank-small on this host serves a different GGUF over llama-server
-    # with no DFlash2 drafter and must not be killed. The pattern never
-    # appears in the deploying shell's own command line, so -f is safe here
-    # (it is not in `bash deploy-remote.sh`).
     run_privileged systemctl stop frank-large.service 2>/dev/null || true
-    pkill -f 'Qwen3.8-27B-DFlash2-Q4_K_M' 2>/dev/null || true
+    pkill -f 'Qwen3.8-27B-Q8_0_ROCMFPX_AGENT' 2>/dev/null || true
     # Nothing may start until the predecessor has released the port. On
     # 2026-09-16 the cutover failed exactly here: the old server logged
     # "Deactivated successfully", the next engine tried to bind 7770 a second
@@ -1177,44 +1177,56 @@ deploy_llm_large() {
     wait_for_health frank-large http://127.0.0.1:7770/health 1200
 }
 
-# deploy_llm_small — frank-small is SGLang (python3.12 venv) on the RTX 3090,
+# deploy_llm_small — frank-small is llama.cpp (master build, CUDA sm_86) on
-# as a ROOT unit: the runtime is a self-contained venv, so there is no
+# the RTX 3090 as a ROOT unit: the runtime is a native binary tree under
-# container image to install and no generated unit to avoid. relink_unit
+# ~/GitPackages/llama-master/build-cuda, so there is no container image and
-# points /etc/systemd/system/frank-small.service at the repo file and boot
+# no generated unit to avoid. relink_unit points
+# /etc/systemd/system/frank-small.service at the repo file and boot
 # persistence is that unit's [Install] section.
 #
 # It is a SEPARATE endpoint on port 7771; frank-large keeps 7770. The 3090 is
 # not shared: frank-large's engine runs on the gfx1151 iGPU.
-LLM_SMALL_MODEL_DIR="${LLM_SMALL_MODEL_DIR:-/home/dank/models/MiMo-V2.6-Distill-Qwen-9B-INT8-W8A16-AutoRound}"
+LLM_SMALL_MODEL_FILE="${LLM_SMALL_MODEL_FILE:-/home/dank/models/MiMo-Q5KM/MiMo-V2.6-Distill-Qwen-9B-Q5_K_M.gguf}"
-LLM_SMALL_RUNTIME="${LLM_SMALL_RUNTIME:-/home/dank/GitPackages/sglang-cu128/bin/sglang}"
+LLM_SMALL_RUNTIME="${LLM_SMALL_RUNTIME:-/home/dank/GitPackages/llama-master/build-cuda/bin/llama-server}"
+# The projector IS the vision capability here: the Q5_K_M GGUF ships no `v.*`
+# tensors, so a deploy that starts without this file silently ships a
+# text-only endpoint. Checked unconditionally, like frank-large's.
+LLM_SMALL_MMPROJ="${LLM_SMALL_MMPROJ:-/home/dank/models/MiMo-Q5KM/mmproj-MiMo-V2.6-Distill-Qwen-9B-f16.gguf}"
 
 deploy_llm_small() {
     # Refuse rather than restart-loop: a unit whose runtime or payload is
     # missing never becomes healthy, and a green deploy would leave the
     # endpoint dark with nothing in the log to say why.
     if [[ ! -x "$LLM_SMALL_RUNTIME" ]]; then
-        echo "ERROR: SGLang runtime absent at $LLM_SMALL_RUNTIME" >&2
+        echo "ERROR: llama.cpp runtime absent at $LLM_SMALL_RUNTIME" >&2
-        echo "frank-small: build the venv (python3.12 -m venv /home/dank/GitPackages/sglang-cu128" >&2
+        echo "frank-small: build it (cmake -B build-cuda -DGGML_CUDA=ON" >&2
-        echo "frank-small: and pip install 'sglang[all]' into it) first." >&2
+        echo "frank-small: -DCMAKE_CUDA_ARCHITECTURES=86 -DCMAKE_CUDA_HOST_COMPILER=/usr/bin/gcc-13" >&2
+        echo "frank-small: && cmake --build build-cuda --target llama-server) first." >&2
         return 1
     fi
-    if [[ ! -f "$LLM_SMALL_MODEL_DIR/model.safetensors.index.json" ]]; then
+    for f in "$LLM_SMALL_MODEL_FILE" "$LLM_SMALL_MMPROJ"; do
-        echo "ERROR: payload index absent at $LLM_SMALL_MODEL_DIR" >&2
+        if [[ ! -f "$f" ]]; then
+            echo "ERROR: frank-small payload file absent at $f" >&2
             echo "frank-small: run scripts/download-frank-small.sh first;" >&2
             echo "frank-small: refusing to start a unit that cannot load." >&2
             return 1
         fi
-    local count
+    done
-    count="$(find "$LLM_SMALL_MODEL_DIR" -maxdepth 1 -name 'model-*-of-*.safetensors' | wc -l)"
-    if [[ "$count" -ne 8 ]]; then
-        echo "ERROR: payload shard count wrong: $LLM_SMALL_MODEL_DIR (found $count, want 8)" >&2
-        return 1
-    fi
     relink_unit frank-small
     run_privileged systemctl daemon-reload
     run_privileged systemctl enable frank-small.service
     run_privileged systemctl restart frank-small.service
-    wait_for_health frank-small http://127.0.0.1:7771/health 900
+    wait_for_health frank-small http://127.0.0.1:7771/health 900 || return 1
+    # False-green hardening (frank-large's lesson): the port answering
+    # proves nothing about WHO answers — the unit must be active with a
+    # live MainPID, or a stray SGLang holdover is masquerading as deployed.
+    local main_pid
+    main_pid="$(run_privileged systemctl show -p MainPID --value frank-small.service)"
+    if [[ -z "$main_pid" || "$main_pid" == "0" ]]; then
+        echo "FAIL: :7771 answered but frank-small.service has no MainPID — a stray process holds the port" >&2
+        return 1
+    fi
+    echo "frank-small: health answered by unit MainPID $main_pid"
 }
 
 # deploy_llm_primary — frank-primary is the tcclaviger RDNA4 vLLM fork

scripts/download-frank-large-kingjones.sh

@@ -0,0 +1,91 @@
+#!/usr/bin/env bash
+# Download kingjones777/Qwen3.8-27B-ROCmFP4-STRIX-MTP-GGUF for frank-large
+# (Dankclaw iGPU). Public repo — no auth header, and never a token on the
+# command line (it is world-readable in /proc/<pid>/cmdline).
+#
+# The payload is the repo's highest-quality target plus its MTP draft head and
+# vision projector:
+#   * Qwen3.8-27B-Q8_0_ROCMFPX_AGENT.gguf — ftype 115, 187 tensors at plain
+#     Q8_0 and 311 at GGML_TYPE_Q8_0_ROCMFPX (103). The AGENT routing keeps the
+#     output-head tensors at full Q8_0 (187 vs 2 in the non-AGENT build), which
+#     is the variant to take when serving with the MTP head.
+#   * mtp-Qwen3.8-27B-Q4_0.gguf — the draft head. Q4_0, NOT Q8_0: the head's
+#     per-step cost is its projection over a 248K vocabulary, so the cheaper
+#     head is faster end to end at equal acceptance (author: +6.0% at n-max 1,
+#     +10.2% at n-max 2).
+#   * mmproj-Qwen3.8-27B-BF16.gguf — the vision tower (verified 4/4 on spatial
+#     ground truth with -fa on).
+#
+# Fresh downloads, no resume (`curl -C -` once produced an oversized file on a
+# 31 GiB payload); -f/--fail-with-body so an HTTP 401 is not a success, then
+# byte count and the Hub LFS sha256. Idempotent: complete files are skipped.
+#
+# The repo's own SHA256SUMS covers only the ROCmFP4 files, so the hashes below
+# come from the Hub LFS oids (api/models/<repo>/tree/main).
+#
+# Usage:
+#   scripts/download-frank-large-kingjones.sh [DEST_DIR] [--verify-only]
+set -euo pipefail
+
+REPO="kingjones777/Qwen3.8-27B-ROCmFP4-STRIX-MTP-GGUF"
+BASE="https://huggingface.co/${REPO}/resolve/main"
+
+DEST=""
+VERIFY_ONLY=0
+for arg in "$@"; do
+    case "$arg" in
+        --verify-only) VERIFY_ONLY=1 ;;
+        *) DEST="$arg" ;;
+    esac
+done
+DEST="${DEST:-/home/dank/models/kingjones-Qwen3.8-27B-ROCmFP4-STRIX-MTP}"
+
+# name bytes sha256
+FILES=(
+  "Qwen3.8-27B-Q8_0_ROCMFPX_AGENT.gguf 28214425952 f3ef2bdbf56e244fff14b72114dc2aa300b84a4cb4eaf54f2e5e4da1cfcae10b"
+  "mtp-Qwen3.8-27B-Q4_0.gguf 1680271648 051a1764cff8c4f3ee6ae8b00593a0364c7539c67fa50ffc58f3f96509fca38e"
+  "mmproj-Qwen3.8-27B-BF16.gguf 931145888 de2a49866988ea272c43dc2a43ee2662c25725a717ca5b9059ea420a97f53fe8"
+)
+
+mkdir -p "$DEST"
+status=0
+for entry in "${FILES[@]}"; do
+  read -r name size sha <<<"$entry"
+  f="$DEST/$name"
+  have=0
+  if [[ -f "$f" ]] && [[ "$(stat -c%s "$f")" == "$size" ]]; then
+    if sha256sum -c --quiet <(printf '%s  %s\n' "$sha" "$f"); then
+      have=1
+    fi
+  fi
+  if (( have )); then
+    echo "OK        $name"
+    continue
+  fi
+  if (( VERIFY_ONLY )); then
+    echo "MISSING   $name"
+    status=1
+    continue
+  fi
+  echo "FETCHING  $name"
+  tmp="$f.part"
+  curl -fsSL --fail-with-body -o "$tmp" "$BASE/$name"
+  mv "$tmp" "$f"
+  if [[ "$(stat -c%s "$f")" != "$size" ]]; then
+    echo "SIZE FAIL $name ($(stat -c%s "$f") != $size)"
+    status=1
+    continue
+  fi
+  if ! sha256sum -c --quiet <(printf '%s  %s\n' "$sha" "$f"); then
+    echo "SHA FAIL  $name"
+    status=1
+    continue
+  fi
+  echo "VERIFIED  $name"
+done
+if (( status == 0 )); then
+  echo "DOWNLOAD_DONE $DEST"
+else
+  echo "DOWNLOAD_FAILED $DEST" >&2
+fi
+exit "$status"

scripts/download-frank-small.sh

@@ -1,19 +1,26 @@
 #!/usr/bin/env bash
-# Download MiMo-V2.6-Distill-Qwen-9B-INT8-W8A16-AutoRound for frank-small
+# Download the frank-small payload for llama.cpp on the RTX 3090:
-# (SGLang, RTX 3090). Public repo — no auth header on the command line.
+# bartowski/MiMo-V2.6-Distill-Qwen-9B-GGUF:Q5_K_M plus its vision projector
+# (the GGUF carries no `v.*` tensors — the tower lives only in the mmproj, so
+# without this file the endpoint is text-only by construction). Public repo —
+# no auth header, and never a token on the command line (it is world-readable
+# in /proc/<pid>/cmdline).
 #
-# Per-file FRESH downloads, no resume: `curl -C -` once produced a 213 MB
+# Fresh downloads, no resume (`curl -C -` once produced a 213 MB oversized file
-# oversized file on a 31 GiB payload; a fresh -f fetch plus size+sha256 is
+# on a 31 GiB payload); -f/--fail-with-body so an HTTP 401 is not a success,
-# the house rule.
+# then byte count and the Hub LFS sha256. Idempotent: complete files are
+# skipped.
+#
+# The Q6_K file this endpoint served until 2026-10-01 night stays on disk as
+# the rollback target (7793710624 B, LFS oid
+# ef96d05a2ddf2cbb450d1af1ac3860ec769d3575bafa70692ee5609bda3fad7d) — re-fetch
+# it from the same repo with the same recipe if it is ever needed again.
 #
 # Usage:
 #   scripts/download-frank-small.sh [DEST_DIR] [--verify-only]
-#
-# LFS shards are verified against the Hub LFS sha256; non-LFS text files
-# against their declared byte size. Idempotent: complete files are skipped.
 set -euo pipefail
 
-REPO="letechlead/MiMo-V2.6-Distill-Qwen-9B-INT8-W8A16-AutoRound"
+REPO="bartowski/MiMo-V2.6-Distill-Qwen-9B-GGUF"
 BASE="https://huggingface.co/${REPO}/resolve/main"
 
 DEST=""
@@ -24,28 +31,12 @@ for arg in "$@"; do
         *) DEST="$arg" ;;
     esac
 done
-DEST="${DEST:-/home/dank/models/MiMo-V2.6-Distill-Qwen-9B-INT8-W8A16-AutoRound}"
+DEST="${DEST:-/home/dank/models/MiMo-Q5KM}"
 
-# name bytes sha256(- = non-LFS, size-only check)
+# name bytes sha256
 FILES=(
-  "model-00001-of-00008.safetensors 2946296312 ddb6117c04cb6feaf7c265e5d75efadf21949037d73253a8e4521d40e164366d"
+  "MiMo-V2.6-Distill-Qwen-9B-Q5_K_M.gguf 6876124704 04cf46f2b5584ef922697d6c063f94879434fb86c467f1a09e52daf0cddaf35f"
-  "model-00002-of-00008.safetensors 1402134792 2dc78d55d1d33d7b3b21dd6f8b4b817b62c2fb1376e6448b322428798c10e257"
+  "mmproj-MiMo-V2.6-Distill-Qwen-9B-f16.gguf 918166048 ff348f3180a63188aa7285db85f550fe38acb61dd013c599eb8bad08d2cc2576"
-  "model-00003-of-00008.safetensors 1427365904 5c797d14e345309d67c78306b9de4aa01a21dbd92dd98e40b0ad69b2900e9203"
-  "model-00004-of-00008.safetensors 1420041856 7402f852e2c7a24ebf4694b98a220e71dfb2c93ba3776272cf2acfc65acd8126"
-  "model-00005-of-00008.safetensors 1410655648 991cbf2eda2ff91fc3ffbb15591d7fcac340fb10d9c33bd3da20010e2f3f4090"
-  "model-00006-of-00008.safetensors 1427627856 efc42448054cfb5e3061e154ba4539204088ce95176d689a95b58de885ef7eeb"
-  "model-00007-of-00008.safetensors 2035442408 75a101fc71ae6f2e1f5c4816f711d62cf1d7a0e02ef6e5182361b8c11e72b626"
-  "model-00008-of-00008.safetensors 822153664 0fb90cb50e2d336fbad58bd4c01880eeddbf537c1dca1cf0ae291195d5efbb4b"
-  "tokenizer.json 19989325 06b9509352d2af50381ab2247e083b80d32d5c0aba91c272ca9ff729b6a0e523"
-  "model.safetensors.index.json 106278 -"
-  "config.json 4608 -"
-  "generation_config.json 161 -"
-  "quantization_config.json 1654 -"
-  "chat_template.jinja 3916 -"
-  "tokenizer_config.json 1168 -"
-  "preprocessor_config.json 443 -"
-  "processor_config.json 1191 -"
-  "README.md 8448 -"
 )
 
 mkdir -p "$DEST"
@@ -55,7 +46,7 @@ for entry in "${FILES[@]}"; do
   f="$DEST/$name"
   have=0
   if [[ -f "$f" ]] && [[ "$(stat -c%s "$f")" == "$size" ]]; then
-    if [[ "$sha" == "-" ]] || sha256sum -c --quiet <(printf '%s  %s\n' "$sha" "$f"); then
+    if sha256sum -c --quiet <(printf '%s  %s\n' "$sha" "$f"); then
       have=1
     fi
   fi
@@ -70,7 +61,6 @@ for entry in "${FILES[@]}"; do
   fi
   echo "FETCHING  $name"
   tmp="$f.part"
-  # no -C resume; -f + --fail-with-body so an HTTP error is not a success
   curl -fsSL --fail-with-body -o "$tmp" "$BASE/$name"
   mv "$tmp" "$f"
   if [[ "$(stat -c%s "$f")" != "$size" ]]; then
@@ -78,7 +68,7 @@ for entry in "${FILES[@]}"; do
     status=1
     continue
   fi
-  if [[ "$sha" != "-" ]] && ! sha256sum -c --quiet <(printf '%s  %s\n' "$sha" "$f"); then
+  if ! sha256sum -c --quiet <(printf '%s  %s\n' "$sha" "$f"); then
     echo "SHA FAIL  $name"
     status=1
     continue

scripts/frank-large.env

@@ -1,63 +1,85 @@
-# frank-large — gufo (gufo-org/gufo) serving
+# frank-large — ROCmFPX llama.cpp serving
-# Swift-1.5-Qwen3.8-27B-Uncensored (abliterated) on the Strix Halo iGPU.
+# kingjones777/Qwen3.8-27B-ROCmFP4-STRIX-MTP-GGUF (Q8_0_ROCMFPX_AGENT, ftype
+# 115) + its Q4_0 MTP draft head, on the Strix Halo iGPU (gfx1151), :7770.
 #
-# These are the tunables read by scripts/serve-frank-large.sh. The flag
+# 2026-10-02 (Frank): "the highest quality of that repo, one full context,
-# list is owned by the serve script; this file only sets fleet values.
+# whatever runtime it requires". The highest-quality rung is Q8_0_ROCMFPX_AGENT
+# (26.28 GiB, 8.39 bpw: 187 tensors at plain Q8_0 + 311 at
+# GGML_TYPE_Q8_0_ROCMFPX); the AGENT routing is also the variant the card
+# tells you to take with MTP, because it keeps the output-head tensors at full
+# Q8_0 (187 vs 2), which is what the draft scores against. The runtime the
+# format REQUIRES is the ROCmFPX fork — type 103 exists nowhere else.
 #
-# DO NOT name a new tunable TEMP, HOST or GPU_MAX_HEAP_SIZE: the HIP
+# The shape: one session at the model's full native window (262,144), which is
-# runtime reads those bare (the 2026-08-13 TEMP incident). HOST is the one
+# the whole point — the Q8_0 head costs quality the fleet wanted back, and
-# exception because the serve script itself consumes it for --host.
+# the draft head pays for the bandwidth the bigger file spends.
-
-# The target payload (scripts/download-frank-large-q4.sh): the serve script
-# refuses to start without the main GGUF and the DFlash2 drafter.
 #
-# 2026-10-01 swap (Frank): unsloth/Qwen3.8-27B-GGUF:UD-Q4_K_XL, 16.35 GiB,
+# Budget (from the GGUF header, not from a boot log): attention KV lives only
-# replacing ajgazin's abliterated Swift-1.5 UD-Q8_K_XL (29.3 GiB). Two
+# in the 16 full-attention layers (4 KV heads x 256 head_dim), 64 KiB/token
-# consequences worth knowing at a glance:
+# with f16 K and V => 16.0 GiB at 262,144. The 48 Gated DeltaNet layers hold
-#   * this is the VANILLA (non-abliterated) Qwen3.8-27B — the endpoint is no
+# ~144 MiB of fixed state per sequence. Weights 26.28 + draft head 1.57 +
-#     longer uncensored, which was one of the reasons the Swift payload was
+# mmproj 0.87 + KV 16.0 + state 0.14 + compute buffers ≈ 45-47 GiB against the
-#     chosen on 2026-09-28;
+# iGPU's ~114 GiB GTT aperture, so f16 KV (the unquantized choice) is affordable
-#   * ~13 GiB of weights became available for KV, which is what makes four
+# here in a way it is not on frank-small's 24 GiB card.
-#     full-sized f16 sessions fit.
-# gufo needs no template for either: its Qwen renderer is compiled in.
-MODEL_DIR=/home/dank/models/Qwen3.8-27B-UD-Q4_K_XL
-MODEL_FILE=/home/dank/models/Qwen3.8-27B-UD-Q4_K_XL/Qwen3.8-27B-UD-Q4_K_XL.gguf
 
-# The DFlash2 drafter is UNCHANGED by the model swap and still lives beside
+# The qualified ROCmFPX fork build (scripts/build-rocmfpx.sh pins the commit).
-# the previous target, so it carries its own directory. It has a sliding
+LLAMA_BIN=/home/dank/GitPackages/kingjones-ROCmFPX/build/bin/llama-server
-# window of 2048, which is what keeps its KV small (~40 MiB).
+# One directory per HF repo, created by scripts/download-frank-large-kingjones.sh.
-DRAFTER_DIR=/home/dank/models/Qwen3.8-27B-UD-Q8_K_XL
+MODEL_FILE=/home/dank/models/kingjones-Qwen3.8-27B-ROCmFP4-STRIX-MTP/Qwen3.8-27B-Q8_0_ROCMFPX_AGENT.gguf
-DRAFTER_FILE=/home/dank/models/Qwen3.8-27B-UD-Q8_K_XL/Qwen3.8-27B-DFlash2-Q4_K_M.gguf
+# The draft head: Q4_0, NOT the Q8_0 head. The head's per-step cost is its
+# projection over a 248K vocabulary, so the cheaper head is faster end to end
+# at equal-or-better acceptance (card: +6.0% at n-max 1, +10.2% at n-max 2).
+DRAFT_FILE=/home/dank/models/kingjones-Qwen3.8-27B-ROCmFP4-STRIX-MTP/mtp-Qwen3.8-27B-Q4_0.gguf
+# Vision tower, separate from the LLM file. Empty string = text-only.
+MMPROJ_FILE=/home/dank/models/kingjones-Qwen3.8-27B-ROCmFP4-STRIX-MTP/mmproj-Qwen3.8-27B-BF16.gguf
 
-# The gufo runtime, built from source by scripts/build-gufo.sh.
+# Fleet endpoint: same port and alias as every frank-large before it, so the
-RUNTIME_DIR=/home/dank/GitPackages/gufo-v0.1.1
+# harness models.yml id and the homepage probe do not move.
-
-# Fleet endpoint: the same port and alias every frank-large has served, so
-# no client, probe or models.yml entry changes address.
 ALIAS=frank-large
-HOST=0.0.0.0
+# HIP reads a bare HOST, so this is BIND_ADDR — the same reason the file
-PORT=7770
+# avoids a bare TEMP.
+BIND_ADDR=0.0.0.0
+LISTEN_PORT=7770
 
-# Four sessions, 163840 context each (Frank, 2026-10-01: "4x 160k"). The
+# One full native window. llama.cpp divides --ctx-size across --parallel
-# context is per session and the KV is UNQUANTIZED (f16) — gufo has no KV
+# slots, so 262144/1 IS the model's trained window; the card validated 65,536
-# quantization lever at all (execution_policy.hpp pins kv_cache_storage to
+# and the fleet has been running this family at 262,144 since the gufo era.
-# kFp16), so "unquantized" is inherent rather than configured.
+CTX_SIZE=262144
-#
+PARALLEL=1
-# The arithmetic that makes this fit: f16 KV costs 64 KiB/token on this model
-# (16 full-attention layers x 4 KV heads x 256 head_dim x 2 (K,V) x 2 bytes;
-# the other 48 layers are linear-attention with a per-SEQUENCE fp32 state of
-# ~144 MiB, not a per-token cache). 4 x 163840 = 655,360 tokens = 40.0 GiB of
-# KV, and each session's 163,840 tokens is exactly 10 GiB. Budget: weights
-# 16.35 + KV 40.0 + drafter 1.07 + linear state 0.56 + runtime ~4 = ~62 GiB
-# against the 114 GiB GTT — roughly 50 GiB of headroom, so the session count
-# or the per-session context can both grow from here.
-CONTEXT_SIZE=163840
-SESSIONS=4
 
-# DFlash2 spec decode (a block-diffusion drafter). gufo sizes the draft
+# Deck knobs. --ubatch-size has a measured optimum on this stack (512 peak on
-# block itself; DRAFT_TOKENS caps the proposals per step (gufo's default is
+# the sibling qwen35 model; larger ubatch costs GiBs of compute buffer for no
-# 7, chosen over llama.cpp's fixed depth 4 because gufo's adaptive policy
+# speed), and --fit must stay off: --fit's auto placement does not know that
-# shortens the block when acceptance drops) and DRAFT_POLICY is the block
+# 311 tensors of type 103 belong on the iGPU.
-# length controller.
+BATCH_SIZE=2048
-DRAFT_TOKENS=7
+UBATCH_SIZE=512
-DRAFT_POLICY=adaptive
+NGL=999
+FIT=off
+FLASH_ATTN=on
+# Unquantized KV — the quality-first choice, affordable at one slot.
+CACHE_TYPE_K=f16
+CACHE_TYPE_V=f16
+
+# Speculative decode: the model ships a standalone MTP head (mtp-Q4_0.gguf,
+# arch qwen35, nextn_predict_layers=1, a 65-block draft). n-max 4 with n-min 0
+# and p-min 0.0 is the card's measured optimum (+31% end to end, acceptance
+# 0.926; its own n-max 8 measurement only reached 0.757 acceptance and LOST).
+# The draft runs entirely on the iGPU, so cross-device traffic stays off the
+# bandwidth-limited link.
+SPEC_TYPE=draft-mtp
+SPEC_DRAFT_NGL=99
+SPEC_DRAFT_DEVICE=ROCm0
+SPEC_DRAFT_N_MAX=4
+SPEC_DRAFT_N_MIN=0
+SPEC_DRAFT_P_MIN=0.0
+
+# The card's thinking sampling profile, which is also what the harness sends
+# in extraBody, so the endpoint behaves the same to a client that omits it.
+TEMPERATURE=1.0
+TOP_P=0.95
+TOP_K=20
+MIN_P=0.0
+PRESENCE_PENALTY=0.0
+
+NUM_THREADS=16
+THREAD_BATCH=16

scripts/frank-large.service

@@ -1,5 +1,5 @@
 [Unit]
-Description=gufo (gufo-org/gufo, ROCm gfx1151) Qwen3.8-27B UD-Q8_K_XL, DFlash2 spec decode, 262144 ctx / 1 session, root systemd unit (dankclaw, frank-large, Strix Halo iGPU)
+Description=ROCmFPX llama.cpp (gfx1151) kingjones777 Qwen3.8-27B Q8_0_ROCMFPX_AGENT, MTP spec decode, 262144 ctx / 1 session, root systemd unit (dankclaw, frank-large, Strix Halo iGPU)
 After=network-online.target
 Wants=network-online.target
 
@@ -13,7 +13,9 @@ Group=dank
 LimitMEMLOCK=infinity
 Environment="FRANK_LARGE_ENV_FILE=/home/dank/serverAppConfigs/scripts/frank-large.env"
 ExecStart=/home/dank/serverAppConfigs/scripts/serve-frank-large.sh
-Restart=on-failure
+# always, not on-failure: an engine that drains and exits 0 after a worker
+# fault would otherwise leave the endpoint down until someone notices.
+Restart=always
 RestartSec=10
 
 [Install]

scripts/frank-primary.env

@@ -59,43 +59,55 @@ FRANK_PRIMARY_PUBLISH=7770:7770
 # hoc experiment identity (both hold the GPUs / name registrations).
 FRANK_PRIMARY_LEGACY_CONTAINERS="frank-primary-r9v qwen-fp8"
 
-# --- qualified engine shape (2026-10-01, RESTORED KNOWN-GOOD: fp8 3 windows) ---
+# --- qualified engine shape (2026-10-02, Frank's own correction) ---
-# bf16 at 2 x 262144 did NOT stay up: with and without --expert-precise-memory
+# 2026-10-01 left a shape that did not stay up: bf16 KV at 2 x 262144 with
-# the boot failed (`Engine core initialization failed`, NRestarts climbing).
+# --expert-precise-memory crash-looped at graph capture (the arena is
-# The last shape measured serving was fp8 3 x 262144 / MNBT 2048 / KV 6.1e9
+# DEVICE-resident and the shape had 0.13 GiB to spare), and the plain bf16
-# (KV 792,069 tokens, 3.02x, arena 6,248, ~3.6 GiB free) and is restored here
+# 2 x 262144 boot booted but left 0 bytes free at runtime. This is the
-# while the bf16 path is investigated in a window (see log.md 2026-10-01).
+# correction: FP8 KV, TWO windows, each 200704 tokens, and the pool cut so
-# Candidate next test: bf16 with SEQS=3 to isolate the KV dtype from the
+# the expert arena gets the room back.
-# sequence count, and/or --enforce-eager to bypass the dynamo constraint path.
+#
-# SEQS 2 x MAXLEN 262144 at BF16 KV (Frank's call, 2026-10-01): two contexts of
+# LADDER (2026-10-01 night, subtract-16k method): 262144 and 245760 OOM at
-# the model's FULL native window with UNQUANTIZED KV, plus
+# graph capture; 237568 passed at 718,606 tokens; 241664 x 3 passed at
-# --expert-precise-memory to hand the arena the compile/graph-capture headroom.
+# 732,359 tokens / 3.03x but is the shape this cutover retired. 200704 was
-# Pool: bf16 costs 13,839 B/token (measured: 11,047,010,304 B -> 798,280 tokens
+# chosen to sit below the 241664 rung while leaving a full 2.95 GiB/rank of
-# on the 3x241664 bf16 boot; decomposes as 12,276 attention + 1,563 mamba/side
+# pool to cut loose for the arena — the two contexts are a capacity
-# caches, against fp8's 6,138 + 1,563 = 7,701). 2 x 262144 = 524,288 tokens
+# contract, not a throughput one (2-way concurrency summed BELOW parity at
-# -> 7.256e9 B floor, pinned at 7,300,000,000 (~527,500 tokens, 3,206 margin).
+# 4064 ctx), so the slot buys the VRAM headroom long generations need.
-# Budget (per rank): 31.22 GiB total; non-expert weights 6.28, graphs 0.55,
+#
-# non-torch ~1.45, pool 6.76 -> ~29.35 used, ~2.5 GiB free (measured basis:
+# KV_BYTES is PER RANK. fp8 costs 7,701 B/token (measured 6,138 attention
-# the fp8 3-window shape used 28.27 and left 3.59).
+# + 1,563 mamba/side, dtype-independent). 2 x 200704 = 401,408 tokens
-# PRECISE MEMORY: DROPPED 2026-10-01 after it crashed the boot in combination
+# -> 3.091e9 B floor, pinned at 3.15e9 (~405,000 tokens, ~44 MB margin —
-# with bf16. The flag selects the fitted 5.25 GiB overhead instead of the
+# the same margin convention the 6.1e9 shape used).
-# 7.60 GiB default, moving the plan's `room` and therefore the arena
+#
-# (6,248 -> ~7,278 slots). But the arena is DEVICE-resident, and the budget
+# THE ARENA IS WHAT REMAINS: the fork's expert planner sizes it AFTER the
-# for this shape does not have 2.35 GiB to spare:
+# KV override, so every byte cut here is arena. The 6.1e9 shape ran an
-#   weights 6.28 + pool 6.80 (bf16 2 x 262144) + graphs 0.55 + arena 16.65
+# arena of 6,248 slots; this one reclaims ~2.95 GiB/rank, roughly 1,260
-#   + non-torch ~1.45 = 31.73 of 31.86 GiB  =>  ~0.13 GiB free at boot
+# more slots at 2.344 MiB each. That is the point of the change — the
-# measured as a CrashLoop: every attempt reached graph capture, the engine
+# gather read path is the measured decode bound, and a bigger arena means
-# exited cleanly, systemd restarted it (NRestarts=3 at ~6.7 min each).
+# more experts resident instead of streamed.
-# With the default overhead the arena is 14.31 GiB and the same shape totals
+#
-# ~29.4 GiB, leaving ~2.4 GiB for capture. Re-enable the flag only with a
+# MNBT 1024 (down from 2048): 1024-4096 are interchangeable on decode AND
-# pool/context reduction that pays for it.
+# identical on arena (the sweep measured the same slot count at every
-# MNBT 2048 (Frank): the overhead formula only adds bytes above nbt 4096, so
+# rung <= 4096), so this is a pure workspace-floor choice, not a decode
-# 2048 pays nothing, and the arena sweep measured the same ceiling <= 4096.
+# trade. The floor matters because the arena budget is what is left.
-FRANK_PRIMARY_MAXLEN=262144
+#
-FRANK_PRIMARY_SEQS=3
+# PRECISE MEMORY stays DROPPED: it crashed the 2 x 262144 bf16 boot
-FRANK_PRIMARY_KV_BYTES=6100000000
+# (weights 6.28 + pool 6.80 + graphs 0.55 + arena 16.65 + non-torch ~1.45
+# = 31.73 of 31.86 GiB, ~0.13 GiB free, NRestarts=3). The flag survives in
+# the serve script as an opt-in; re-enable it only with a pool/context
+# reduction that actually pays for the extra overhead.
+#
+# TUNABLEOP stays OFF (silently skipped under offload unless
+# CLAV_TUNABLEOP_WITH_OFFLOAD=1, and then corrupts numerics).
+#
+# SEQS 2 x MAXLEN 200704 at FP8 KV, pool cut for the arena. No
+# speculative decode, no prefetch.
+FRANK_PRIMARY_MAXLEN=200704
+FRANK_PRIMARY_SEQS=2
+FRANK_PRIMARY_KV_BYTES=3150000000
 FRANK_PRIMARY_KV_DTYPE=fp8
-FRANK_PRIMARY_MNBT=2048
+FRANK_PRIMARY_MNBT=1024
 FRANK_PRIMARY_GATHER_CHUNKS=32
 FRANK_PRIMARY_GATHER_LANES=64
 FRANK_PRIMARY_UTIL=0.98

scripts/frank-small.env

@@ -1,86 +1,60 @@
-# Tunables for frank-small (letechlead/MiMo-V2.6-Distill-Qwen-9B-INT8-
+# frank-small — llama.cpp (master, CUDA sm_86) serving
-# W8A16-AutoRound, qwen3_5 hybrid, in-checkpoint vision) on SGLang,
+# bartowski/MiMo-V2.6-Distill-Qwen-9B-GGUF:Q5_K_M on the RTX 3090, :7771.
-# Dankclaw's RTX 3090, since 2026-10-01.
 #
-# Runtime: SGLang in a python3.12 venv (pip "sglang[all]"):
+# 2026-10-02 (Frank): THREE contexts, vision on, and a hard instruction to
-#   python3.12 -m venv /home/dank/GitPackages/sglang-cu128
+# leave 2-3 GiB of the card free. Both new demands cost memory, so the per-slot
-#   /home/dank/GitPackages/sglang-cu128/bin/pip install "sglang[all]"
+# window shrinks — see the budget below.
-# The venv carries its own CUDA torch + sgl-kernel + flashinfer; it is
-# self-contained and does not touch the host Python (3.14) or the ROCm
-# stack that serves frank-large.
 #
-# Model: MiMo-V2.6-Distill-Qwen-9B-INT8-W8A16-AutoRound — a Qwen3.5-9B
+# Two measurements on this card, this runtime, price the terms:
-# distillation at INT8 W8A16 (AutoRound, group 128, symmetric). 8
+#   2 x 262,144, no projector: 20,190 MiB used / 4,386 free
-# safetensors shards, 12.0 GiB total. The vision tower
+#     KV = 524,288 tok x 24.5 KiB = 12,800 MiB => non-KV (weights 6,557 +
-# (qwen3_5_vision, 27 layers) and embed_tokens stay fp16 inside the
+#     DeltaNet state ~25/slot + compute buffers + CUDA graphs) = 7,390 MiB
-# checkpoint, so the image path needs no separate projector file.
+#   3 x 196,608, projector on: 22,972 MiB used / 1,153 free
-# Architecture: 32 layers, 8 full-attention (4 KV heads x 256 head_dim)
+#     KV = 14,113 MiB => non-KV WITH the vision tower = 8,859 MiB
-# + 24 Gated DeltaNet, native 262144 window. text_config declares
+# The tower costs ~1.4 GiB, not the ~1.0 its file size suggested, so 196,608
-# mtp_num_hidden_layers = 1, but the shipped safetensors carry NO mtp.*
+# per slot left only 1.1 GiB free — outside the band. The pool therefore is
-# tensors (verified against model.safetensors.index.json 2026-10-01 —
+# 3 x 180,224 = 540,672 tokens: KV ~12,936 + non-KV 8,859 = ~21,800 MiB predicted.
-# base BF16 and this INT8 conversion both omit the head), so there is no
+# MEASURED at this shape: 21,684 MiB used / 2,441 MiB free — inside the band, and
-# spec decode: SGLang would draft with an uninitialised 4.24 GiB ghost
+# the number the bats budget gate reproduces (it prices non-KV at 9 GiB).
-# model at 0% acceptance and pay the verify tax for nothing.
 #
-# SIZING (RTX 3090), measured 2026-10-01: the 0.98 budget holds the
+# KV arithmetic: attention KV exists only in the 8 full-attention layers
-# checkpoint (11,354 MiB) + the DeltaNet ssm state pool (795 MiB at 32
+# (4 KV heads x 256 head_dim). K f16 = 16 KiB/token, V q8_0 = 8.5 KiB/token
-# bf16 slots) + the KV pool. The engine's autofit picks a 687,388-token
+# (34 B per 32-element block) => 24.5 KiB/token, so 12,288 tokens (=4096/slot)
-# pool and leaves ~0.53 GB — too little: Triton's deferred kernel loads
+# is one 293 MiB step — that is the ladder to walk if this moves again. The
-# take most of it after boot, and eager-prefill activations then OOM on
+# native 262,144 per slot is NOT reachable three times: 3 x 262,144 x
-# the first long request. The pool is therefore PINNED at 606,208 tokens
+# 24.5 KiB = 18.4 GiB of KV before anything else is loaded.
-# (2 x 262,144 native slots since the 2026-10-01 reslice; 4 x 151,552
-# before), which leaves ~0.8 GB for the prefill workspace; a 170k single
-# request and 4 concurrent 54k requests were verified at the 4-slot
-# shape.
-# The ssm pool is the 4.5 GiB the weights-only arithmetic misses.
 #
-# Sampling: greedy by default (the fleet contract); a client that sends a
+# Why V stays q8_0 and K stays f16: full-precision V would add 8 KiB/token —
-# temperature gets its own, and the harness applies per-model overrides
+# +4.5 GiB on this pool — which is what pushes the card past its 24 GiB. K is
-# from models.yml.
+# the tensor the asymmetric-KV contract exists to protect.
-#
+LLAMA_BIN=/home/dank/GitPackages/llama-master/build-cuda/bin/llama-server
-# WARNING: do not name any variable TEMP, HOST, or GPU_MAX_HEAP_SIZE (HIP
+MODEL_FILE=/home/dank/models/MiMo-Q5KM/MiMo-V2.6-Distill-Qwen-9B-Q5_K_M.gguf
-# reads those bare vars on the ROCm hosts -- see the 2026-08-13 incident in
-# AGENTS.md). This box is CUDA, but the rule keeps env files portable.
 
-# SGLang runtime: python3.12 venv, self-contained CUDA stack.
+# Vision: the Q5_K_M GGUF ships no `v.*` tensors, so the projector file IS the
-SGLANG_BIN=/home/dank/GitPackages/sglang-cu128/bin/sglang
+# vision capability (sha256 ff348f31..., 918,166,048 B, fetched by
+# scripts/download-frank-small.sh). Empty string = text-only on purpose.
+MMPROJ_FILE=/home/dank/models/MiMo-Q5KM/mmproj-MiMo-V2.6-Distill-Qwen-9B-f16.gguf
 
-# Model package (8 shards + configs, sha256-verified by
+# Fleet endpoint: same port and alias as every frank-small before it, so the
-# scripts/download-frank-small.sh).
+# harness models.yml id and the homepage probe do not move.
-MODEL_DIR=/home/dank/models/MiMo-V2.6-Distill-Qwen-9B-INT8-W8A16-AutoRound
-
-# Fleet identity and network. 7771 is this service's own endpoint on
-# Dankclaw; 7770 stays frank-large's. The alias is the id at /v1/models
-# and what clients ask for -- the harness entry in models.yml names it.
 ALIAS=frank-small
 BIND_ADDR=0.0.0.0
 LISTEN_PORT=7771
 
-# Two slots at the model's native 262,144 window on fp8 KV. The pool is
+# Three independent conversations. llama.cpp divides --ctx-size across
-# pinned (see SIZING above): 606,208 tokens, and 2 x 262,144 = 524,288
+# --parallel slots (n_ctx_slot = CTX_SIZE/PARALLEL), and a mamba-hybrid model
-# leaves ~82k tokens of pool slack. Resliced 2026-10-01 from 4 x 151,552
+# cannot use unified KV, so each slot gets its own cache and its own window.
-# so a compaction request (frank-primary's trigger 183,501 tokens + a
+# 180,224 per slot is BELOW the native 262,144, and that is the cost of the
-# 32,768-token summary budget = 216,269) fits ONE request; the grid is
+# third slot: a longer single prompt needs compaction instead of one shot.
-# the two-way concurrency contract. A single request may still grow to
+# Consequence worth knowing: frank-primary's compaction trigger is 183,501
-# the whole pool (a 170k prompt was verified at the 4-slot shape).
+# tokens, which no longer fits a frank-small slot — see the runbook's
-MEM_FRACTION=0.98
+# "three slots vs compaction" note.
-MAX_TOTAL_TOKENS=606208
+CTX_SIZE=540672
-CONTEXT_LENGTH=262144
+PARALLEL=3
-MAX_RUNNING=2
-KV_DTYPE=fp8_e4m3
-QUANTIZATION=auto-round
-# DeltaNet (mamba) recurrent states: the ssm state pool is the single
-# biggest hidden consumer on this card. The engine autofills the slot
-# COUNT to spend a fixed ~5.1 GB budget regardless of dtype (fp32 at 110
-# slots, bf16 at 217 — same 5.1 GB), so BOTH dials are pinned below:
-# bf16 dtype + 32 slots = 795 MiB.
-MAMBA_SSM_DTYPE=bfloat16
-# 32 slots is 16x the 2-way concurrency: enough for the radix mamba
-# cache to keep shared-prefix states, small enough to leave the KV pool
-# room.
-MAX_MAMBA_CACHE_SIZE=32
 
-# Reasoning is extracted into reasoning_content; tools use the qwen3_coder
+# The KV contract (Frank, 2026-10-01 night): unquantized K, q8_0 V.
-# grammar (both parsers per the SGLang Qwen3.5 cookbook).
+# q8_0 on V is only served by the flash-attention path; the serve script
-REASONING_PARSER=qwen3
+# pins --flash-attn on and must never lose it.
-TOOLCALL_PARSER=qwen3_coder
+CACHE_TYPE_K=f16
+CACHE_TYPE_V=q8_0
+NGL=99

scripts/frank-small.service

@@ -1,5 +1,5 @@
 [Unit]
-Description=SGLang MiMo-V2.6-Distill-Qwen-9B-INT8 (qwen3_5 hybrid, in-checkpoint vision, no MTP) 4x151k fp8 KV (dankclaw, frank-small)
+Description=llama.cpp MiMo-V2.6-Distill-Qwen-9B Q5_K_M GGUF (qwen35 hybrid, 3x180224, f16 K + q8_0 V, mmproj vision) on the RTX 3090 (dankclaw, frank-small)
 After=network-online.target
 Wants=network-online.target
 

scripts/serve-frank-large.sh

@@ -1,121 +1,179 @@
 #!/usr/bin/env bash
-# Serve Swift-1.5-Qwen3.8-27B-Uncensored (abliterated) as frank-large on
+# Serve kingjones777/Qwen3.8-27B-ROCmFP4-STRIX-MTP-GGUF as frank-large on
-# Dankclaw's Strix Halo iGPU (gfx1151), using gufo (gufo-org/gufo) — the
+# Dankclaw's Strix Halo iGPU (gfx1151) with the ROCmFPX fork of llama.cpp.
-# inference engine built specifically for this hardware.
 #
-# Runtime: /home/dank/GitPackages/gufo-v0.1.1/bin/gufo, built from source
+# 2026-10-02 (Frank): swapped from gufo + Q4_K_XL to the highest-quality file
-# against the host ROCm by scripts/build-gufo.sh. It replaced the stock
+# in the repo, one session at the native 262,144 window.
-# llama.cpp prebuilt (unsloth b11030) on 2026-09-28; port, alias and client
-# contract did not move.
 #
-# Payload: ajgazin/Swift-1.5-Qwen3.8-27B-Uncensored-Dynamic-MTP-GGUF's
+# Why this runtime and not the previous one: the payload is ftype 115 — 311
-# UD-Q8_K_XL, fetched by scripts/download-frank-large-swift.sh. It replaced
+# tensors of GGML_TYPE_Q8_0_ROCMFPX (103), a tensor type that exists only in
-# unsloth/Qwen3.8-27B-GGUF:UD-Q8_K_XL on 2026-09-28. The two are structurally
+# the ROCmFPX fork. Stock llama.cpp, SGLang and gufo all refuse the file, and
-# interchangeable for this engine — same `qwen35` architecture, the same 866
+# gufo additionally gates every "Qwen3.8" artifact on a template-hash
-# tensors in the same order, the same per-tensor Q8_0/F32/BF16 layout, the
+# allowlist. The qualified build is scripts/build-rocmfpx.sh's pin: it loads
-# same MTP head at blk.64 — so the swap needed no engine change. The fine-tune
+# ftype 115, survives warmup (the 2026-08-25 builds of the older fork line
-# is abliterated (uncensored) and carries the SWIFT OPEN LICENSE 1.0, which
+# segfaulted in ggml_compute_forward_set_rows on this exact file), and drafts
-# is not an OSI licence.
+# with the model's own MTP head.
 #
-# Shape: one session, full 262144 context (the model's native window). The
+# Why this file: Q8_0_ROCMFPX_AGENT is the top of the repo's quality ladder
-# 27B dense weights (16.35 GiB at UD-Q4_K_XL) + the DFlash2 drafter (1.07) + KV +
+# (26.28 GiB, 8.39 bpw) and the AGENT routing keeps 187 tensors — the
-# compute buffers fit the iGPU's GTT aperture (~114 GiB); a load at this
+# output-head set the draft scores against — at plain Q8_0 instead of the 2
-# context measured 66.5 GiB of the 116.7 GiB device.
+# the non-AGENT build keeps, which is the variant the card names for MTP
+# serving. Drafting uses the repo's Q4_0 MTP head: the head's cost is its
+# projection over the 248K vocabulary, so the 4-bit head is faster end to end
+# at equal-or-better acceptance than its Q8_0 sibling.
 #
-# Spec method: DFlash2 (a block-diffusion drafter), gufo's adaptive block
+# The client contract lives here, not in frank-large.env:
-# policy, capped at DRAFT_TOKENS proposals per step. The drafter is the
+#   * --jinja — without it llama.cpp silently IGNORES the harness's
-# z-lab Qwen3.8-27B-DFlash2-Q4_K_M and it has a sliding window of 2048
+#     chat_template_kwargs (enable_thinking / preserve_thinking), so the
-# (small draft KV). It is UNCHANGED by the model swap and still lives beside
+#     endpoint would look like it had no thinking control at all. The GGUF
-# the previous target, which is why DRAFTER_DIR is its own tunable.
+#     carries the official Qwen template, so no --chat-template-file.
-#
+#   * --reasoning-format deepseek — thinking is emitted as reasoning_content,
-# Sampling: greedy (--temperature 0) server-side, as every frank-large has
+#     the field the harness reads for this endpoint.
-# served. The harness overrides it per request via extraBody (1.0 / 0.95 /
+#   * the template accepts reasoning_effort xhigh|medium|low only, which is
-# top-k 20).
+#     why models.yml folds the harness's levels onto that set.
-#
+#   * --alias frank-large — that is the model id the harness resolves.
-# Template: gufo compiles the official Qwen Jinja into the binary and takes
+#   * --ctx-checkpoints / --cache-ram — prompt-cache geometry for the 16
-# NO --chat-template-file — and it is not a Jinja engine at all
+#     sliding-window-style layers; harmless where the hybrid state forces a
-# (src/models/qwen/chat_template.cpp is a hand-written C++ renderer), so the
+#     full re-process, and required for the checkpoint feature to work at all.
-# fleet's scripts/qwen-sharp-v22.4.0.jinja cannot be supplied to it. That law
-# was given up when this endpoint moved to gufo on 2026-09-28, by decision:
-# the terse system-prompt injection, the in-band <|think_*|> control, the
-# json|xml tool_call_format switch and the tool-argument truncation are all
-# absent. gufo's own renderer keeps the effort instruction strings and
-# preserve_thinking, which are the parts the harness depends on. Accepted
-# 2026-09-28; recovering the terse prompt from the client side is the
-# cheapest partial restore.
-#
-# Text-only: gufo auto-discovers a mmproj beside the model and this payload
-# has none (the base repo ships one separately, unused), so the endpoint
-# stays text-only.
-#
-# Tunables come from frank-large.env unless FRANK_LARGE_ENV_FILE overrides
-# it; anything not set there falls through to the defaults below.
 set -euo pipefail
 
-SCRIPT_DIR="$(cd -- "$(dirname -- "${BASH_SOURCE[0]}")" && pwd)"
+# Fleet values live in frank-large.env. Anything already in the environment
-ENV_FILE="${FRANK_LARGE_ENV_FILE:-$SCRIPT_DIR/frank-large.env}"
+# wins over the file, so an operator can override one value on a single run:
+# the caller's values are captured, the file is sourced, then they are put
+# back. (A plain `source` would let the file clobber the environment.)
+TUNABLES=(LLAMA_BIN MODEL_FILE DRAFT_FILE MMPROJ_FILE ALIAS BIND_ADDR
+          LISTEN_PORT CTX_SIZE PARALLEL BATCH_SIZE UBATCH_SIZE NGL FIT
+          FLASH_ATTN CACHE_TYPE_K CACHE_TYPE_V SPEC_TYPE SPEC_DRAFT_NGL
+          SPEC_DRAFT_DEVICE SPEC_DRAFT_N_MAX SPEC_DRAFT_N_MIN SPEC_DRAFT_P_MIN
+          TEMPERATURE TOP_P TOP_K MIN_P PRESENCE_PENALTY NUM_THREADS
+          THREAD_BATCH ROCM_PATH HIP_VISIBLE HSA_GFX_VERSION)
+_OVERRIDES=()
+for _var in "${TUNABLES[@]}"; do
+    [[ -n "${!_var+set}" ]] && _OVERRIDES+=("$_var=${!_var}")
+done
+ENV_FILE="${FRANK_LARGE_ENV_FILE:-$(dirname "$0")/frank-large.env}"
 if [[ -f "$ENV_FILE" ]]; then
     set -a
+    # shellcheck disable=SC1090
     source "$ENV_FILE"
     set +a
 fi
-
+for _kv in "${_OVERRIDES[@]+${_OVERRIDES[@]}}"; do
-# --- defaults (used when the env file does not set them) ------------------
+    export "${_kv?}"
-: "${RUNTIME_DIR:=/home/dank/GitPackages/gufo-v0.1.1}"
-: "${MODEL_DIR:=/home/dank/models/Swift-1.5-Qwen3.8-27B-Uncensored-MTP-UD-Q8_K_XL}"
-: "${ALIAS:=frank-large}"
-: "${HOST:=0.0.0.0}"
-: "${PORT:=7770}"
-: "${CONTEXT_SIZE:=262144}"
-: "${SESSIONS:=1}"
-: "${DRAFT_TOKENS:=7}"
-: "${DRAFT_POLICY:=adaptive}"
-
-GUFO_BIN="$RUNTIME_DIR/bin/gufo"
-
-# The target GGUF and the drafter are named explicitly because they do not
-# share a directory any more: the target is a different repository's artifact
-# and keeps its own long filename, while the DFlash2 drafter is unchanged and
-# still lives beside the previous target. MODEL_FILE/DRAFTER_FILE default to
-# the deployed payload, so the env file only has to state them when it moves.
-: "${MODEL_FILE:=$MODEL_DIR/Qwen3.8-27B-UD-Q4_K_XL.gguf}"
-: "${DRAFTER_DIR:=/home/dank/models/Qwen3.8-27B-UD-Q8_K_XL}"
-: "${DRAFTER_FILE:=$DRAFTER_DIR/Qwen3.8-27B-DFlash2-Q4_K_M.gguf}"
-
-if [[ ! -x "$GUFO_BIN" ]]; then
-    echo "runtime not found at $GUFO_BIN" >&2
-    echo "frank-large: build it: bash scripts/build-gufo.sh" >&2
-    exit 1
-fi
-for f in "$MODEL_FILE" "$DRAFTER_FILE"; do
-    if [[ ! -f "$f" ]]; then
-        echo "missing asset $f (run the matching scripts/download-frank-large*.sh)" >&2
-        exit 1
-    fi
 done
+unset _var _kv
 
-# --- env -------------------------------------------------------------------
+: "${LLAMA_BIN=/home/dank/GitPackages/kingjones-ROCmFPX/build/bin/llama-server}"
-# gufo links against the host ROCm through its RPATH (/opt/rocm ->
+: "${MODEL_FILE=/home/dank/models/kingjones-Qwen3.8-27B-ROCmFP4-STRIX-MTP/Qwen3.8-27B-Q8_0_ROCMFPX_AGENT.gguf}"
-# /etc/alternatives/rocm), so no LD_LIBRARY_PATH is needed. The device is
+: "${DRAFT_FILE=/home/dank/models/kingjones-Qwen3.8-27B-ROCmFP4-STRIX-MTP/mtp-Qwen3.8-27B-Q4_0.gguf}"
-# already gfx1151 — `gufo diagnose` reports it natively — so no
+: "${MMPROJ_FILE=/home/dank/models/kingjones-Qwen3.8-27B-ROCmFP4-STRIX-MTP/mmproj-Qwen3.8-27B-BF16.gguf}"
-# HSA_OVERRIDE_GFX_VERSION either; the override existed for the prebuilt
+: "${ALIAS=frank-large}"
-# llama.cpp runtime and can only misreport the arch here.
+: "${BIND_ADDR=0.0.0.0}"
-export HIP_VISIBLE_DEVICES=0
+: "${LISTEN_PORT=7770}"
+: "${CTX_SIZE=262144}"
+: "${PARALLEL=1}"
+: "${BATCH_SIZE=2048}"
+: "${UBATCH_SIZE=512}"
+: "${NGL=999}"
+: "${FIT=off}"
+: "${FLASH_ATTN=on}"
+: "${CACHE_TYPE_K=f16}"
+: "${CACHE_TYPE_V=f16}"
+: "${SPEC_TYPE=draft-mtp}"
+: "${SPEC_DRAFT_NGL=99}"
+: "${SPEC_DRAFT_DEVICE=ROCm0}"
+: "${SPEC_DRAFT_N_MAX=4}"
+: "${SPEC_DRAFT_N_MIN=0}"
+: "${SPEC_DRAFT_P_MIN=0.0}"
+: "${TEMPERATURE=1.0}"
+: "${TOP_P=0.95}"
+: "${TOP_K=20}"
+: "${MIN_P=0.0}"
+: "${PRESENCE_PENALTY=0.0}"
+: "${NUM_THREADS=16}"
+: "${THREAD_BATCH=16}"
+: "${ROCM_PATH=/opt/rocm}"
+: "${HIP_VISIBLE=0}"
+: "${HSA_GFX_VERSION=11.5.1}"
 
-# --- the serve command ---------------------------------------------------
+[[ -n "$LLAMA_BIN" ]] || { echo "frank-large: LLAMA_BIN is empty" >&2; exit 1; }
-# Server options precede the subcommand, model options follow it.
+[[ -x "$LLAMA_BIN" ]] || {
-exec "$GUFO_BIN" serve \
+    echo "frank-large: llama-server '$LLAMA_BIN' is not executable;" >&2
-    --host "$HOST" \
+    echo "           build it with scripts/build-rocmfpx.sh" >&2
-    --port "$PORT" \
+    exit 1
-    --sessions "$SESSIONS" \
+}
-    llm \
+# Refuse anything but the real payload. A path that is not a readable file
+# makes llama-server hang rather than fail clean.
+[[ -r "$MODEL_FILE" ]] || {
+    echo "frank-large: model file '$MODEL_FILE' is not readable;" >&2
+    echo "           fetch it with scripts/download-frank-large-kingjones.sh" >&2
+    exit 1
+}
+# The MTP head is the performance feature: without it the Q8_0 weights decode
+# at memory-bandwidth speed and the endpoint loses the ~1.4x the head buys.
+[[ -r "$DRAFT_FILE" ]] || {
+    echo "frank-large: MTP draft head '$DRAFT_FILE' is not readable;" >&2
+    echo "           fetch it with scripts/download-frank-large-kingjones.sh" >&2
+    exit 1
+}
+# The GGUF carries no vision tower; the projector file is the whole tower, so
+# a missing file means a vision request would 500 rather than degrade.
+# MMPROJ_FILE="" is the documented opt-out for a text-only endpoint.
+MM_ARGS=()
+if [[ -n "${MMPROJ_FILE:-}" ]]; then
+    [[ -r "$MMPROJ_FILE" ]] || {
+        echo "frank-large: mmproj '$MMPROJ_FILE' is not readable; set MMPROJ_FILE='' for text-only" >&2
+        exit 1
+    }
+    MM_ARGS=(--mmproj "$MMPROJ_FILE")
+fi
+
+# HIP/ROCm host environment. HIP reads bare env vars (TEMP, HOST,
+# GPU_MAX_HEAP_SIZE), which is why the tunables above are named TEMPERATURE
+# and BIND_ADDR. HSA_OVERRIDE_GFX_VERSION is what lets a runtime built for a
+# neighbouring gfx target run on this iGPU; the qualified pin targets gfx1151
+# natively and the override is a no-op for it.
+export ROCM_PATH
+export HIP_VISIBLE_DEVICES="$HIP_VISIBLE"
+export HSA_OVERRIDE_GFX_VERSION="$HSA_GFX_VERSION"
+export LD_LIBRARY_PATH="$(dirname "$LLAMA_BIN"):${ROCM_PATH}/lib:${ROCM_PATH}/lib/rocm_sysdeps/lib:${ROCM_PATH}/lib/llvm/lib${LD_LIBRARY_PATH:+:$LD_LIBRARY_PATH}"
+
+exec /usr/bin/env "$LLAMA_BIN" \
     --model "$MODEL_FILE" \
-    --served-model-name "$ALIAS" \
+    "${MM_ARGS[@]}" \
-    --context "$CONTEXT_SIZE" \
+    --alias "$ALIAS" \
-    --speculative dflash2 \
+    --host "$BIND_ADDR" \
-    --dflash-model "$DRAFTER_FILE" \
+    --port "$LISTEN_PORT" \
-    --draft-tokens "$DRAFT_TOKENS" \
+    --jinja \
-    --draft-policy "$DRAFT_POLICY" \
+    --device ROCm0 \
-    --temperature 0 \
+    --n-gpu-layers "$NGL" \
-    --think auto \
+    --parallel "$PARALLEL" \
-    --preserve-thinking on
+    --ctx-size "$CTX_SIZE" \
+    --batch-size "$BATCH_SIZE" \
+    --ubatch-size "$UBATCH_SIZE" \
+    --threads "$NUM_THREADS" \
+    --threads-batch "$THREAD_BATCH" \
+    --fit "$FIT" \
+    --flash-attn "$FLASH_ATTN" \
+    --direct-io \
+    --cache-type-k "$CACHE_TYPE_K" \
+    --cache-type-v "$CACHE_TYPE_V" \
+    --ctx-checkpoints 16 \
+    --cache-ram 8192 \
+    --cache-idle-slots \
+    --temp "$TEMPERATURE" \
+    --top-p "$TOP_P" \
+    --top-k "$TOP_K" \
+    --min-p "$MIN_P" \
+    --presence-penalty "$PRESENCE_PENALTY" \
+    --reasoning on \
+    --reasoning-format deepseek \
+    --reasoning-budget -1 \
+    --spec-type "$SPEC_TYPE" \
+    --model-draft "$DRAFT_FILE" \
+    --spec-draft-ngl "$SPEC_DRAFT_NGL" \
+    --spec-draft-device "$SPEC_DRAFT_DEVICE" \
+    --spec-draft-n-max "$SPEC_DRAFT_N_MAX" \
+    --spec-draft-n-min "$SPEC_DRAFT_N_MIN" \
+    --spec-draft-p-min "$SPEC_DRAFT_P_MIN" \
+    --metrics

scripts/serve-frank-primary.sh

@@ -11,14 +11,16 @@
 # every flag here was qualified by boot probe on 2026-09-29 — see
 # frank-primary.env for the qualification record).
 #
-# Shape (2026-10-01 bf16 cutover): MAXLEN 262144 x 3 sessions (786,432
+# Shape (2026-10-02, Frank's correction of the 10-01 bf16 shape): MAXLEN
-# tokens), KV UNQUANTIZED (--kv-cache-dtype auto = bf16; the fp8 q8/q8 shape
+# 200704 x SEQS 2 (401,408 tokens of contract), KV fp8 (--kv-cache-dtype
-# is retired to git tag frank-primary-fp8-mtp-20261001), KV_BYTES
+# fp8; the bf16 unquantized shape is retired to git tag
-# 11,047,010,304 PER RANK (the engine measured 798,280 tokens of capacity /
+# frank-primary-fp8-mtp-20261001), KV_BYTES 3,150,000,000 PER RANK (the
-# 3.05x concurrency), NO speculative decode, --max-num-batched-tokens 1024
+# engine measured 405,000 tokens of capacity / ~3.0x concurrency), NO
-# (Frank's lock), MoE gather geometry R4D_LRU_GATHER_CHUNKS=32 /
+# speculative decode, --max-num-batched-tokens 1024 (Frank's lock), MoE
-# R4D_LRU_GATHER_LANES=64 (+18.7% decode, grows with depth), prefix caching,
+# gather geometry R4D_LRU_GATHER_CHUNKS=32 /
-# expert prefetch OFF (36% precision measured), TunableOp off.
+# R4D_LRU_GATHER_LANES=64 (+18.7% decode, grows with depth), prefix
+# caching, expert prefetch OFF (36% precision measured), TunableOp off, and
+# the pool deliberately small so the expert arena keeps what it reclaims.
 #
 # The root unit (scripts/frank-primary.service, User=dank,
 # LimitMEMLOCK=infinity) grants the pin budget for the tiered-expert host

scripts/serve-frank-small.sh

@@ -1,30 +1,25 @@
 #!/usr/bin/env bash
-# Serve letechlead/MiMo-V2.6-Distill-Qwen-9B-INT8-W8A16-AutoRound as
+# Serve bartowski/MiMo-V2.6-Distill-Qwen-9B-GGUF (Q5_K_M) as frank-small on
-# frank-small on Dankclaw's RTX 3090 via SGLang.
+# Dankclaw's RTX 3090 via llama.cpp (master build, CUDA sm_86).
 #
-# Topology: the whole checkpoint (12.0 GiB, INT8 W8A16 AutoRound) lives in
+# 2026-10-01 night (Frank): swapped from SGLang + INT8-W8A16-AutoRound
-# VRAM; the 3090's 24 GiB holds weights + a ~11.8 GiB fp8 KV pool at
+# safetensors to the llama.cpp GGUF path, because llama.cpp is the only
-# --mem-fraction-static 0.98. The host CPU only tokenizes.
+# runtime on this host that can serve the MiMo qwen35 hybrid with
+# ASYMMETRIC KV: --cache-type-k f16 (unquantized keys) + --cache-type-v
+# q8_0 (quantized values), which is Frank's explicit KV contract for this
+# endpoint. SGLang's --kv-cache-dtype only moves K and V together.
 #
 # Architecture: qwen3_5 hybrid — 32 layers, 8 full-attention + 24 Gated
-# DeltaNet, native 262144 window, and a vision tower (qwen3_5_vision)
+# DeltaNet, native 262144 window. KV lives only in the 8 attention layers;
-# baked into the safetensors: no separate projector file, the image path
+# the DeltaNet state is per-slot fixed size. Vision comes from the repo's
-# is in the checkpoint. The config declares an MTP layer, but the shipped
+# separate projector (the Q5_K_M GGUF carries no `v.*` tensors at all), so
-# weights carry none (no mtp.* tensors in the index), so this service runs
+# --mmproj is what gives this endpoint eyes; MMPROJ_FILE="" is the documented
-# WITHOUT speculative decoding — a NEXTN draft would be an uninitialised
+# text-only opt-out.
-# 4.24 GiB ghost model drafting at 0% acceptance.
 #
-# Context: CONTEXT_LENGTH x MAX_RUNNING slots must fit the fp8 KV pool,
+# Sampling: llama.cpp takes it per request (the fleet harness sends its
-# which the engine autofits around the weights AND the DeltaNet ssm state
+# recipe from models.yml); greedy is the harness default.
-# pool (see frank-small.env). The mamba pool defaults to ~5.1 GB, so it is
-# pinned (bf16, 32 slots). With the grid at its measured fit the card has
-# no headroom for prefill CUDA-graph capture, so prefill runs eager
-# (--disable-prefill-cuda-graph); decode graphs are untouched.
 #
-# Sampling: greedy by default (the fleet contract); clients that send their
+# Tunables come from frank-small.env unless FRANK_SMALL_ENV_FILE overrides.
-# own temperature get their own values.
-#
-# Tunables come from frank-small.env unless FRANK_SMALL_ENV_FILE overrides it.
 set -euo pipefail
 
 SCRIPT_DIR="$(cd -- "$(dirname -- "${BASH_SOURCE[0]}")" && pwd)"
@@ -37,61 +32,56 @@ if [[ -f "$ENV_FILE" ]]; then
 fi
 
 # --- defaults (used when the env file does not set them) -------------------
-: "${SGLANG_BIN:=/home/dank/GitPackages/sglang-cu128/bin/sglang}"
+: "${LLAMA_BIN:=/home/dank/GitPackages/llama-master/build-cuda/bin/llama-server}"
-: "${MODEL_DIR:=/home/dank/models/MiMo-V2.6-Distill-Qwen-9B-INT8-W8A16-AutoRound}"
+: "${MODEL_FILE:=/home/dank/models/MiMo-Q5KM/MiMo-V2.6-Distill-Qwen-9B-Q5_K_M.gguf}"
 : "${ALIAS:=frank-small}"
 : "${BIND_ADDR:=0.0.0.0}"
 : "${LISTEN_PORT:=7771}"
-: "${MEM_FRACTION:=0.98}"
+# ctx-size is the POOL llama.cpp divides across slots: 540672 / 3 gives
-: "${CONTEXT_LENGTH:=151552}"
+# 180224 per slot — three independent conversations (the native 262144 no
-: "${MAX_TOTAL_TOKENS:=606208}"
+# longer fits three times on a 24 GiB card once the vision tower is loaded;
-: "${MAX_RUNNING:=4}"
+# frank-small.env carries the two measurements that set this number).
-: "${KV_DTYPE:=fp8_e4m3}"
+: "${CTX_SIZE:=540672}"
-: "${QUANTIZATION:=auto-round}"
+: "${PARALLEL:=3}"
-: "${REASONING_PARSER:=qwen3}"
+: "${CACHE_TYPE_K:=f16}"
-: "${TOOLCALL_PARSER:=qwen3_coder}"
+: "${CACHE_TYPE_V:=q8_0}"
-: "${MAMBA_SSM_DTYPE:=bfloat16}"
+: "${NGL:=99}"
-: "${MAX_MAMBA_CACHE_SIZE:=32}"
+# Vision projector. No colon on purpose: an explicitly empty value means
+# "text-only, on purpose" and must NOT be replaced by the default path.
+: "${MMPROJ_FILE=/home/dank/models/MiMo-Q5KM/mmproj-MiMo-V2.6-Distill-Qwen-9B-f16.gguf}"
 
-# Fail loudly rather than half-loading a model into VRAM: a missing file here
+# Fail loudly rather than half-loading: a missing file is a download that
-# is a download that has not finished, and the unit's restart policy would
+# has not finished, and Restart=always would turn it into a crash loop that
-# otherwise turn that into a crash loop that reads like a runtime fault.
+# reads like a runtime fault.
-for path in "$SGLANG_BIN" "$MODEL_DIR/model.safetensors.index.json" \
+for path in "$LLAMA_BIN" "$MODEL_FILE"; do
-    "$(cd -- "$(dirname -- "$SGLANG_BIN")/.." && pwd)/cuda-nvcc/usr/local/cuda-13.0/bin/nvcc"; do
     [[ -e "$path" ]] || { echo "frank-small: missing $path" >&2; exit 1; }
 done
-shard_count=$(find "$MODEL_DIR" -maxdepth 1 -name 'model-*.safetensors' | wc -l)
+EXTRA_ARGS=()
-if [[ "$shard_count" -ne 8 ]]; then
+if [[ -n "$MMPROJ_FILE" ]]; then
-    echo "frank-small: expected 8 safetensors shards in $MODEL_DIR, found $shard_count" >&2
+    [[ -e "$MMPROJ_FILE" ]] || { echo "frank-small: missing $MMPROJ_FILE" >&2; exit 1; }
-    exit 1
+    EXTRA_ARGS+=(--mmproj "$MMPROJ_FILE")
 fi
 
 # The UVM device node is created lazily by nvidia-modprobe on the first CUDA
 # open after boot; make sure it exists before the server needs it.
 nvidia-modprobe -u -c=0 2>/dev/null || true
 
-# FlashInfer JITs its attention kernels with nvcc at boot. The host's
+# q8_0 on V REQUIRES flash attention (the quantized-value kernels only exist
-# /usr/bin/nvcc is CUDA 12.4 and dies on `--compress-mode` (a 12.8+
+# on the FA path); an FA-less build silently falls back and the KV contract
-# option), so the venv carries a matching CUDA 13.0 nvcc (extracted from
+# in frank-small.env becomes a lie, so pin the flag pair here.
-# the cuda-nvcc-13-0 deb); it must shadow the system one on PATH.
+ARGS=(
-export PATH="$(cd -- "$(dirname -- "$SGLANG_BIN")/.." && pwd)/cuda-nvcc/usr/local/cuda-13.0/bin:$PATH"
+    --model "$MODEL_FILE"
-
+    --alias "$ALIAS"
-# Sampling: SGLang has no server-wide sampling defaults; the harness sends
+    --host "$BIND_ADDR"
-# its per-model recipe from models.yml (compat.extraBody), and other clients
+    --port "$LISTEN_PORT"
-# get the engine's own defaults.
+    --ctx-size "$CTX_SIZE"
-exec "$SGLANG_BIN" serve \
+    --parallel "$PARALLEL"
-    --model-path "$MODEL_DIR" \
+    --n-gpu-layers "$NGL"
-    --served-model-name "$ALIAS" \
+    --flash-attn on
-    --host "$BIND_ADDR" \
+    --cache-type-k "$CACHE_TYPE_K"
-    --port "$LISTEN_PORT" \
+    --cache-type-v "$CACHE_TYPE_V"
-    --mem-fraction-static "$MEM_FRACTION" \
+    --jinja
-    --context-length "$CONTEXT_LENGTH" \
+    --reasoning auto
-    --max-running-requests "$MAX_RUNNING" \
+    --reasoning-format deepseek
-    --kv-cache-dtype "$KV_DTYPE" \
+)
-    --quantization "$QUANTIZATION" \
+exec "$LLAMA_BIN" "${ARGS[@]}" ${EXTRA_ARGS[@]+"${EXTRA_ARGS[@]}"}
-    --reasoning-parser "$REASONING_PARSER" \
-    --tool-call-parser "$TOOLCALL_PARSER" \
-    --mamba-ssm-dtype "$MAMBA_SSM_DTYPE" \
-    --max-mamba-cache-size "$MAX_MAMBA_CACHE_SIZE" \
-    --disable-prefill-cuda-graph \
-    --max-total-tokens "$MAX_TOTAL_TOKENS"

serverAppConfigs-docs/LLM/dankserver-qwen38-fp8-expert-offload-tuning.md

@@ -275,41 +275,48 @@ Depth caveat: every number above is at a near-empty KV. Deep-context
 behavior (the gather win growing with depth, bf16's decode cost at depth)
 is only sampled at 3 depths; a 64k-192k-depth probe is the flagged gap.
 
-### The deployed shape (2026-10-01: 5 x 262144, fp8, no spec)
+### The deployed shape (2026-10-02: 2 x 200704, fp8, pool cut for the arena)
 
 ```
-FRANK_PRIMARY_MAXLEN=262144    x SEQS 2 (524,288 tokens of contract)
+FRANK_PRIMARY_MAXLEN=200704    x SEQS 2 (401,408 tokens of contract)
-FRANK_PRIMARY_KV_BYTES=7300000000   PER RANK (bf16 = auto; 13,839 B/token)
+FRANK_PRIMARY_KV_BYTES=3150000000   PER RANK (fp8, 7,701 B/token)
-FRANK_PRIMARY_KV_DTYPE=auto   --kv-cache-dtype auto (bf16, unquantized)
+FRANK_PRIMARY_KV_DTYPE=fp8   --kv-cache-dtype fp8 (quantized)
-FRANK_PRIMARY_PRECISE_MEMORY=1  --expert-precise-memory (fitted overhead
-                                5.25 GiB instead of the 7.60 GiB default,
-                                so ~1,025 more arena slots)
 FRANK_PRIMARY_MNBT=1024       --max-num-batched-tokens 1024
 R4D_LRU_GATHER_CHUNKS=32 / R4D_LRU_GATHER_LANES=64  (container env)
 NO --speculative-config, NO --expert-prefetch
 ```
 
-Boot-probed 2026-10-01 (transient systemd units at the production uid +
+This replaced the 2026-10-01 shape — 2 x 262144 at bf16 KV with
+`--expert-precise-memory` — which crash-looped at graph capture: the arena is
+DEVICE-resident, the precise-memory flag drops the plan's overhead 7.60 ->
+5.25 GiB, and that shape had 0.13 GiB to spare (weights 6.28 + pool 6.80 +
+graphs 0.55 + arena 16.65 + non-torch ~1.45 = 31.73 of 31.86 GiB,
+NRestarts=3 at ~6.7 min each). The plain bf16 2 x 262144 boot also did not
+stay up: it left 0 bytes free at runtime and a real client request (995-token
+prompt, max_tokens=261149) killed the worker with `Tried to allocate 50.00 MiB
+... 0 bytes free`.
+
+The correction is a pool cut, not a downgrade. fp8 costs 7,701 B/token
+(measured 6,138 attention + 1,563 mamba/side, dtype-independent), so 2 x
+200704 = 401,408 tokens needs 3.091e9 B; pinned at 3.15e9, ~44 MB of margin
+above the floor — the same convention the 6.1e9 shape used. The arena is what
+remains: the planner sizes it AFTER the KV override, so the 6,248 slots of the
+6.1e9 shape become roughly 7,500 here. Two contexts at 200704 are a capacity
+contract, not a throughput one (2-way concurrency summed BELOW parity at 4064
+ctx), so the slot buys the VRAM headroom long generations need.
+
+Boot-probed 2026-10-02 (transient systemd units at the production uid +
 memlock, full-log capture; the rootless bench probe had masked the production
 memory environment). The night searched two concurrency contracts:
 
 | rung | result |
 |---|---|
-| 3 × 262144, bf16, KV 11,047,010,304 | **OOM at graph capture** — `Tried to allocate 76.00 MiB ... 0 bytes free`, 30.43 GiB of 31.86 GiB already allocated by PyTorch |
+| 3 x 262144, bf16, KV 11,047,010,304 | **OOM at graph capture** — `Tried to allocate 76.00 MiB ... 0 bytes free`, 30.43 GiB of 31.86 GiB already allocated by PyTorch |
-| 3 × 245760, bf16 | **OOM identically** |
+| 3 x 245760, bf16 | **OOM identically** |
-| 3 × 237568, bf16 | PASS — KV 718,606 tokens |
+| 3 x 237568, bf16 | PASS — KV 718,606 tokens |
-| **3 × 241664, bf16** | **PASS — KV 732,359 tokens, 3.03x** (max possible at 3 contexts) |
+| 2 x 262144, bf16 + `--expert-precise-memory` | **DEPLOYED then crashed** — 0.13 GiB free at boot, NRestarts=3 |
-| 6 × 241664, fp8 | **OOM** — tried 10.70 GiB against 9.18 GiB free: six full-length contexts do NOT fit |
+| 2 x 262144, bf16 (no precise flag) | did not stay up either: 0 bytes free at runtime, worker killed by a real request |
-| 5 × 262144, fp8 | PASS on the first rung — KV 1,350,182 tokens, 5.15x |
+| **2 x 200704, fp8, pool cut for the arena** | **DEPLOYED (2026-10-02, Frank's correction)**: fp8 KV, two 200704 windows, ~7,500 arena slots, ~3.6 GiB free VRAM measured |
-| 4 × 262144, fp8 | runtime-survivable stopgap — KV 1,079,582 tokens / 4.12x |
-| 3 × 262144, fp8, pool at the floor | KV 792,069 tokens / 3.02x; arena 6,248 slots; 3.59 GiB free VRAM measured |
-| **2 × 262144, bf16 + `--expert-precise-memory`** | **DEPLOYED** (Frank's call): unquantized KV at the full native window, two contexts; pool 7.3e9 B/rank (bf16 rate 13,839 B/token measured: 12,276 attention + 1,563 mamba/side, vs fp8's 6,138 + 1,563 = 7,701); the precise-memory flag drops the plan's overhead 7.60 -> 5.25 GiB so the arena grows from 6,248 to ~7,278 slots |
-
-Boot is NOT proof of runtime survivability: the five-slot shape passed every
-boot gate and still died on the first long-generation request. After any
-knife-edge shape change, send a request with a large `max_tokens` before
-calling it done. The deployed shape is **four contexts of the model's full
-native 262,144 window**. fp8 KV is what buys the capacity —
 7,926 B/token against bf16's ~13,900-14,047 — at the cost of bf16's ~4 %
 decode edge. Six full-length contexts cannot fit at any KV dtype: the 6-seq
 graph/activation footprint leaves only 9.18 GiB for the pool, and 6 × 262144

serverAppConfigs-docs/LLM/frank-large-gufo-qwen38-27b.md

@@ -1,7 +1,7 @@
 ---
 type: reference
-title: "gufo, Swift-1.5-Qwen3.8-27B-Uncensored UD-Q8_K_XL (DFlash2) on Dankclaw (frank-large)"
+title: "gufo, Qwen3.8-27B UD-Q4_K_XL (DFlash2) on Dankclaw (frank-large)"
-description: "The active frank-large runbook — gufo (gufo-org/gufo) v0.1.1, the Strix Halo inference engine, built from source on Dankclaw and serving the abliterated ajgazin/Swift-1.5-Qwen3.8-27B-Uncensored UD-Q8_K_XL with the z-lab DFlash2 Q4_K_M drafter on the gfx1151 iGPU: the from-source build and its two host workarounds, the GGUF header evidence that made the payload swap safe, the one-session / full-window shape, why the fleet's Qwen-Sharp chat template CANNOT be used on this runtime and what that costs, the client-contract delta, the measured prefill/decode trade, and the failure modes worth knowing."
+description: "The RETIRED frank-large runbook (rollback target) — gufo (gufo-org/gufo) v0.1.1, the Strix Halo inference engine, built from source on Dankclaw and serving unsloth/Qwen3.8-27B-GGUF UD-Q4_K_XL (vanilla) with the z-lab DFlash2 Q4_K_M drafter on the gfx1151 iGPU, ONE session at the full native 262,144 window with unquantized f16 KV: the from-source build and its two host workarounds, the GGUF header evidence, why the fleet's Qwen-Sharp chat template CANNOT be used on this runtime, the client-contract delta, the measured prefill/decode trade, and the failure modes worth knowing."
 tags:
   - llm
   - gufo
@@ -16,9 +16,20 @@ tags:
 timestamp: 2026-09-28
 ---
 
-# frank-large — gufo, Swift-1.5-Qwen3.8-27B-Uncensored UD-Q8_K_XL (DFlash2)
+# frank-large — gufo, Qwen3.8-27B (UD-Q4_K_XL, DFlash2)
 
-> **ACTIVE since 2026-09-28.** Replaces
+> **RETIRED 2026-10-02 — the rollback target.** Superseded by
+> [[frank-large-rocmfpx-q8agent-mtp]] (Q8_0_ROCMFPX_AGENT at 8.39 bpw with
+> MTP4, 17.0 t/s measured). This era was itself a return after one day of
+> [[frank-large-vllm-mimo-v26-bf16]] (battery decode 6.19-9.01 t/s; every
+> ≥6-bit vLLM quant path dead on gfx1151). The shape
+> moved on the way back: payload is now **unsloth/Qwen3.8-27B-GGUF
+> UD-Q4_K_XL** (16.35 GiB, vanilla — the abliterated Swift Q8 was
+> reclaimed), and the session grid is **one session at the full native
+> 262,144 window** (`SESSIONS=1 CONTEXT_SIZE=262144`, f16 KV). The
+> Swift-1.5 Q8 history below is retained for the engine/template law,
+> which still governs.
+> **Originally ACTIVE 2026-09-28.** Replaces
 > [[frank-large-unsloth-qwen38-27b-q8xl]] (stock llama.cpp, the unsloth
 > b11030 prebuilt), which had replaced
 > [[frank-large-kairic-edge-qwen38-27b]]. Same host, same port (7770), same

serverAppConfigs-docs/LLM/frank-large-rocmfpx-q8agent-mtp.md

@@ -0,0 +1,219 @@
+---
+type: reference
+title: "frank-large on Dankclaw — ROCmFPX llama.cpp, Qwen3.8-27B Q8_0_ROCMFPX_AGENT + MTP"
+description: "The active frank-large runbook — the kingjones30/ROCmFPX llama.cpp fork (pinned commit, gfx1151) serving kingjones777's Q8_0_ROCMFPX_AGENT GGUF, a ftype-115 file no other loader can read, with the repo's own MTP-Q4_0 head drafting four tokens at one native 262,144 window, f16 KV, and the bundled BF16 projector for vision. Measured MTP-vs-none A/B, the client contract, and rollback to gufo."
+tags:
+  - llm
+  - frank-large
+  - dankclaw
+  - llama-cpp
+  - rocmfpx
+  - gfx1151
+  - gguf
+  - spec-decode
+  - mtp
+timestamp: 2026-10-02
+---
+
+# frank-large — ROCmFPX Q8_0_ROCMFPX_AGENT + MTP (gfx1151)
+
+> **ACTIVE since 2026-10-02.** Cut over from
+> [[frank-large-gufo-qwen38-27b]] (Q4_K_XL, ~15 t/s). Both requests came from
+> Frank: *highest quality that runs*, and *one full context*. This endpoint
+> delivers 8.39 bpw weights — two quant rungs above what gufo could load — and
+> measures **17.0 t/s** decode instead of 15, so quality was not bought with
+> speed.
+>
+> The capability that forced this whole shape is a GGUF tensor type:
+> **`GGML_TYPE_Q8_0_ROCMFPX` (103)**. It exists in exactly one loader on this
+> host, so the runtime was decided by the file, not by preference.
+
+## Identity
+
+| | |
+|---|---|
+| Host / device | Dankclaw (192.168.0.143), Radeon 8060S iGPU `gfx1151`, ~114 GiB GTT aperture |
+| Endpoint | `http://192.168.0.143:7770` — alias `frank-large`, `/v1` OpenAI surface |
+| Runtime | **`kingjones30/ROCmFPX`** llama.cpp fork @ `36e9acd40e10a87cd3c3ef8ec734668757dc8520`, built by [`scripts/build-rocmfpx.sh`](../../scripts/build-rocmfpx.sh) → `/home/dank/GitPackages/kingjones-ROCmFPX/build/bin/llama-server` |
+| Unit | root systemd `frank-large.service` ([`scripts/frank-large.service`](../../scripts/frank-large.service)), `User=dank`, `Restart=always` |
+| Serve script | [`scripts/serve-frank-large.sh`](../../scripts/serve-frank-large.sh) (owns the argv), tunables [`scripts/frank-large.env`](../../scripts/frank-large.env) |
+| Weights | `/home/dank/models/kingjones-Qwen3.8-27B-ROCmFP4-STRIX-MTP/Qwen3.8-27B-Q8_0_ROCMFPX_AGENT.gguf` — 28,214,425,952 B, sha256 `f3ef2bdbf56e244f…` |
+| Draft | `…/mtp-Qwen3.8-27B-Q4_0.gguf` — 1,680,271,648 B, sha256 `051a1764cff8c4f3…` |
+| Projector | `…/mmproj-Qwen3.8-27B-BF16.gguf` — 931,145,888 B, sha256 `de2a4986…` |
+| Fetch | [`scripts/download-frank-large-kingjones.sh`](../../scripts/download-frank-large-kingjones.sh) (`-f`, byte-count + LFS sha256 verified, `--verify-only`, token never on the command line) |
+| Seam | `tests/frank_large_unit.bats` (19 tests: payload gates, deployer/serve parity, spec block, KV dtype, sweep identity) |
+
+## Why this file, and why only this runtime
+
+`kingjones777/Qwen3.8-27B-ROCmFP4-STRIX-MTP-GGUF` ships several rungs. Parsed
+headers of the two candidates:
+
+| File | ftype | Tensor mix | Output-head tensors at plain Q8_0 | Card MTP acceptance |
+|---|---|---|---|---|
+| `Qwen3.8-27B-Q8_0_ROCMFPX_AGENT.gguf` | **115** | 353 F32 + 187 Q8_0 + **311 type-103** | **187** | 0.953 |
+| `Qwen3.8-27B-Q8_0_ROCMFPX.gguf` | 111 | 2 Q8_0 + type-103 | 2 | 0.911 |
+
+Two consequences, both measured off the header rather than guessed:
+
+1. **Quality.** The AGENT routing variant keeps the output-head (and other
+   routing-sensitive) tensors at plain Q8_0 instead of folding them into the
+   ROCmFPX kernel layout — 187 vs 2 — so the logits the sampler reads are the
+   highest-precision tensors in the file. It is also the card's MTP-tuned
+   variant, so it is the *faster* drafter base at the same time.
+2. **The runtime is forced.** Type 103 is a private extension: stock llama.cpp
+   and gufo both fail at load (`unknown tensor type`). Verified on the host
+   before deploying — the bake-off was not a horse race, it was a lock-in
+   check.
+
+**The build identity matters as much as the fork.** The 2026-08-25 ROCmFPX
+builds read this file and then died in `ggml_compute_forward_set_rows` — the
+exact failure documented in [[kingjones-q8agent-rocmfpx]] when this payload was
+first attempted on Dankclaw and routed to Dankserver as
+[[frank-medium-rocmfpx]] instead. `scripts/build-rocmfpx.sh` therefore checks
+the commit out **by SHA** and refuses to build anything else; the pin is
+load-bearing, not tidiness.
+
+## Shape and budget
+
+One session, the model's full native window, unquantized KV:
+
+| Dial | Value | Why |
+|---|---|---|
+| `--ctx-size` | 262,144 | "one full context" — the native window, no rope scaling |
+| `--parallel` | 1 | one conversation, all memory to it |
+| KV dtype | **f16** (both K and V) | ~64 KiB/token → ~16 GiB; free inside a 114 GiB aperture, so there is no reason to pay a quant |
+| `--fit off` | on | the auto-fitter would silently resize the window we asked for |
+| `--flash-attn on` | on | required for the long-window attention path to fit |
+| `--direct-io` | on | bypasses page cache for the 26 GiB weight stream (mmap is disabled by it — deliberate) |
+| `--ctx-checkpoints 16` | 16 | slot state for checkpoint/rollback |
+| `--cache-ram 8192` + `--cache-idle-slots` | 8 GiB | prompt cache; verified a repeat pass over the same 6k prompt replays in **0.5 s** |
+| `--jinja` | on | without it `chat_template_kwargs` are silently ignored |
+| `--reasoning on --reasoning-format deepseek --reasoning-budget -1` | | thinking in its own channel, unbounded |
+
+Measured steady state (production, one model copy resident):
+`mem_info_gtt_used` **63.1 GiB** of the ~114 GiB aperture, `mem_info_vram_used`
+0.17 / 2.0 GiB (the iGPU's dedicated carve-out is a stub; everything real lives
+in GTT). llama-server RSS 2.1 GiB. Boot to `/health` 200: **16 s** — the fastest
+endpoint bring-up in the fleet, because there is no engine compile step.
+
+## Speculative decode — measured, not assumed
+
+```
+--spec-type draft-mtp --model-draft <mtp-Qwen3.8-27B-Q4_0.gguf>
+--spec-draft-n-max 4 --spec-draft-n-min 0 --spec-draft-p-min 0.0
+--spec-draft-device ROCm0 --spec-draft-ngl 99
+```
+
+MTP in llama.cpp is a property of the file, not of the flags: the head is a
+trained tensor set (`nextn_predict_layers = 1` in this repo's head, arch
+`qwen35`, 65 blocks) and the draft context runs on the main model's weights, so
+there is no second 26 GiB copy in memory. See the fleet law in
+`AGENTS.md → llama.cpp serving`.
+
+**A/B, 2026-10-02** — same session, temp 0, 2,202-token prompt, 512 tokens,
+`cache_prompt: false`, `frank-large.service` stopped for every scratch arm (the
+benchmarking law), baseline arm stripped of `--model-draft` **and** every
+`--spec-*` flag:
+
+| Arm | Decode t/s (2 runs) | Prefill t/s | Draft acceptance |
+|---|---|---|---|
+| **MTP, n-max 4 (deployed)** | **17.00 / 17.14** | 350.6 / 324.9 | 0.462 (mean acceptance length ≈ 1.86) |
+| no draft | 7.60 / 7.66 | 381.8 / 381.0 | — |
+
+**+123 % decode for an 8–15 % prefill tax.** Notes that matter before someone
+re-litigates this:
+
+- Acceptance here (0.462) is nowhere near the card's 0.953. The card measures
+  its own short, repetitive text; this is a 2.2k-token prose prompt. Treat the
+  card's number as a file property, not a throughput promise.
+- `n-max 4` is where the gain was measured; 2 and 8 are **unmeasured**. The
+  gufo-era law applies in spirit — never rank spec settings on a
+  self-predictable corpus.
+- End-to-end on the chat path is slower than the `/completion` number because
+  thinking tokens are included: a 2,253-token chat prompt generating 512
+  tokens took 43.8 s wall (11.7 t/s end-to-end) — that is the cost of an
+  unbounded reasoning budget, not a regression in decoding.
+
+## Client contract (live-verified 2026-10-02)
+
+| Behavior | Observation |
+|---|---|
+| Model id | `/v1/models` → `frank-large` |
+| Tools | native: `get_weather` → `{"city": "Seattle"}` in `tool_calls` |
+| Thinking | in **`reasoning_content`** (llama.cpp field — *not* vLLM's `reasoning`); `models.yml` sets `reasoningContentField` accordingly |
+| `reasoning_effort` | accepts **only** `xhigh`, `medium`, `low`; anything else is HTTP 500 (the template raises, it does not clamp). `models.yml` folds `high`/`max` → `xhigh` |
+| Sampling | server defaults = the card's thinking recipe (temp 1.0, top_p 0.95, top_k 20, min_p 0, presence 0), mirrored in the harness `extraBody` |
+| Vision | the BF16 projector loads (`loaded multimodal model`); a red square answers **`Red`** (`usage: 75 prompt / 71 cached`) |
+| Prompt cache | second pass over the same 6k prompt: 0.5 s wall |
+
+Three traps, all hit for real:
+
+1. **`content: null` when reasoning eats the budget.** With
+   `--reasoning-parser`-style separation, a request whose `max_tokens` is
+   consumed entirely by thinking returns `message.content = None`. A bench
+   client that indexes `content` crashes *after* measuring and loses the
+   numbers — fall back to `reasoning_content`, or size the budget (200 was
+   enough for the vision probe).
+2. **`MTP boundary missing for seq_id=… ; resyncing after a non-token step`**
+   appears in the log under draft resets. Self-recovering; not an outage signal.
+3. **Never `pkill -f llama-server` on Dankclaw** — frank-small is also a
+   `llama-server`. The sweep key here is the payload name:
+   `pkill -f 'Qwen3.8-27B-Q8_0_ROCMFPX_AGENT'`.
+
+## Operations
+
+```bash
+# deploy (planner selects this endpoint only via its own unit/env/serve trio)
+./scripts/deploy.sh dankclaw --service frank-large
+
+# status / logs / cause of a crash loop
+systemctl status frank-large
+journalctl -u frank-large -n 200 --no-pager
+
+# rebuild the pinned fork (only when deliberately moving the pin)
+scripts/build-rocmfpx.sh
+
+# re-verify the payload without re-downloading
+scripts/download-frank-large-kingjones.sh --verify-only
+```
+
+`serve-frank-large.sh` refuses to start — before touching the port — unless the
+fork binary, the weights, the MTP head and (unless `FRANK_LARGE_MMPROJ=""`) the
+projector all exist; `scripts/deploy-remote.sh` mirrors the same predicate so a
+broken payload never stops the running unit. Caller environment **overrides**
+`frank-large.env`, and defaults use `:=` so an explicitly empty value surfaces
+as the error it reads as.
+
+## Rollback
+
+git tag **`frank-large-gufo-q4kxl-20261001`**: gufo v0.1.1
+(`scripts/build-gufo.sh` still on disk), the Q4_K_XL Qwen3.8-27B payload and the
+DFlash2 drafter all remain on the host. Restore that tag's unit/env/serve trio
+and redeploy — nothing here deletes anything.
+
+The rollback payloads stay exactly where the tag's env file expects them:
+`/home/dank/models/Qwen3.8-27B-UD-Q4_K_XL/` (16.35 GiB) and the DFlash2
+drafter `/home/dank/models/Qwen3.8-27B-UD-Q8_K_XL/Qwen3.8-27B-DFlash2-Q4_K_M.gguf`
+(1.07 GiB). The one-day vLLM MiMo era's payloads (MiMo-V2.6 BF16 18 GiB,
+Qwen3.5-9B MTP source 19 GiB, both DFlash draft dirs) were staged to
+`/home/dank/.AGENTTRASH/20261002-retired-llm-payloads/` on 2026-10-02 —
+recoverable until the trash is purged, and never part of a rollback plan.
+
+## When it misbehaves
+
+- **`unknown tensor type` / ftype 115 refused** — something moved off the
+  ROCmFPX binary. Check `readlink -f $(systemctl show frank-large -p ExecStart)`
+  and the fork commit; this file cannot run on stock llama.cpp or gufo.
+- **SIGSEGV in `ggml_compute_forward_set_rows` during load** — a fork build
+  older than the `36e9acd` pin; rebuild with `scripts/build-rocmfpx.sh`.
+  History: [[kingjones-q8agent-rocmfpx]].
+- **Slow decode with no log error** — check whether the draft is engaged
+  (`/metrics`, `draft_n` / `draft_n_accepted`; a zero draft count means the head
+  did not load), and check GTT pressure with
+  `cat /sys/class/drm/card1/device/mem_info_gtt_used` — a second resident copy
+  of this model (an orphaned scratch server) drives the whole host into
+  eviction and halves the rate silently.
+- **`500` naming Jinja** — a `chat_template_kwargs` field reached a server
+  without `--jinja`.
+- **HTTP 500 naming the template on the first request of a role** — an
+  unmapped `reasoning_effort`; fix the fold in `models.yml`, not the model.

serverAppConfigs-docs/LLM/frank-large-vllm-mimo-v26-bf16.md

@@ -0,0 +1,186 @@
+---
+type: reference
+title: "frank-large on Dankclaw — vLLM TheRock, MiMo-V2.6-Distill-Qwen-9B BF16 + cross-model MTP (RETIRED)"
+description: "The RETIRED frank-large runbook (2026-10-01, one day) — the digest-pinned kyuz0 TheRock gfx1151 vLLM image serving XiaomiMiMo MiMo-V2.6-Distill-Qwen-9B in full BF16 on the Strix Halo iGPU. Kept for the measured record: the roof math, the cross-model MTP drafter, and the autoresearch finding that spec decode is NET-NEGATIVE on real chat+reasoning traffic here (6.19 t/s with MTP-2 vs 9.01 t/s without). Rolled back to [[frank-large-gufo-qwen38-27b]] by Frank the same night."
+tags:
+  - llm
+  - frank-large
+  - dankclaw
+  - vllm
+  - strix-halo
+  - mimo
+  - mtp
+  - spec-decode
+  - bf16
+timestamp: 2026-10-01
+---
+
+# frank-large — vLLM MiMo-V2.6-Distill-Qwen-9B BF16 (cross-model MTP)
+
+> **RETIRED 2026-10-01 night, one day active.** Frank's call after the
+> autoresearch loop: battery decode 6.19 t/s (MTP-2) / 9.01 t/s (no draft)
+> against gufo's ~15 t/s on the 27B — and every ≥6-bit quant path in vLLM
+> 0.30 is dead on gfx1151 (quark int8 rejected; all FP8 kernels CDNA-only;
+> torchao config unwired). Back to [[frank-large-gufo-qwen38-27b]]
+> (Q4_K_XL, 1 x 262144). The measured content below stands as the record.
+
+## Identity
+
+- **Endpoint:** `frank-large` at `192.168.0.143:7770` (Dankclaw).
+- **Runtime:** vLLM 0.30.0 in the **digest-pinned** `docker.io/kyuz0/vllm-therock-gfx1151`
+  image (image ID `2d24a8c5a3fd`, digest `sha256:98c1f3ac9833…` — ROCm 10.0.0 /
+  torch 2.11.0 / TheRock, Triton Gated DeltaNet kernels). The official vLLM
+  images are CUDA-only; this fork is the fleet's gfx1151 path. Plain
+  `podman run` driven by `scripts/serve-frank-large.sh` under a root unit
+  (`scripts/frank-large.service`, `User=dank`, `Restart=always`,
+  `ExecStop=podman stop -t 120`, `TimeoutStopSec=180` > the stop grace).
+- **Model:** `XiaomiMiMo/MiMo-V2.6-Distill-Qwen-9B` in **full BF16**
+  (17.53 GiB, 4 shards) — the same checkpoint frank-small serves as an INT8
+  W8A16 AutoRound conversion on the 3090, here unquantized.
+- **Draft:** `Qwen/Qwen3.5-9B` (18 GiB on disk; only the 15 `mtp.*` tensors
+  are loaded into the draft pass).
+
+## Why this shape (the roof math)
+
+Decode speed for a dense model on Strix Halo is bandwidth-bound:
+BF16 weights x one pass per token against the LPDDR5x aperture. Measured
+2026-10-01 on this host: **13.1 t/s is ~94% of the roof** — the payload, not
+a misconfiguration. The same INT8 payload reaches 83.9 t/s on the 3090
+because it moves ~12 GiB through ~936 GB/s. The only way past a bandwidth
+wall is generating more than one token per weight pass: **speculative decode
+is the performance feature of this endpoint**, not a garnish.
+
+The memory side is the opposite of a squeeze: KV grows ONLY on the 8 full
+attention layers (32 KiB/token bf16: 8 x 2 x 4 heads x 256 dim x 2 B);
+the 24 Gated DeltaNet layers hold a FIXED ~25 MiB state per running
+sequence. `--max-model-len 262144` (the native window) with
+`--max-num-seqs 4` at `--gpu-memory-utilization 0.45` boots clean — the
+four-session contract the gufo shape ran, with the pool shared by the
+scheduler instead of carved per session.
+
+## Spec method: cross-model MTP — measured, not assumed
+
+| arm | prefill | decode | acceptance |
+|---|---|---|---|
+| BF16 base | 17,876 t/s | 13.1 t/s | — |
+| BF16 + DFlash 8 tok | 7,976 t/s | 17.4 t/s | pos0 59% pos1 26% (token rate 14.8%) |
+| **BF16 + MTP 2 tok** | 7,380 t/s | **19.8 t/s** | pos0 71% pos1 44% (token rate 57.4%) |
+
+One session, temperature 0, 28,647-token prefill / 512-token decode, same
+client for all three arms (2026-10-01 bench window, frank-large stopped per
+Benchmarking Policy). **MTP wins (+51% vs +33%) and needs only 2 draft
+tokens** — acceptance past position 1 is 44% and falling; DFlash's own curve
+decays 59/26/14/7, so 8 proposals are mostly wasted verification.
+`num_speculative_tokens` beyond 2 is UNMEASURED; do not raise it without a
+fresh A/B. Both spec arms pay ~55–59% of prefill (the draft passes share
+the compute queues) — this endpoint's thinking-heavy traffic spends its wall
+clock in decode, which is the regime where that trade wins.
+
+**Why a Qwen head accepts on a MiMo target:** MiMo-V2.6 is a distillation of
+Qwen3.5-9B that preserves the hidden-state geometry — identical
+4096-hidden/248320-vocab/32-layer/interval-4 structure, and the 71%/44%
+per-position acceptance is the direct measurement that the distillation
+preserved more than geometry. MiMo itself ships **ZERO `mtp.*` tensors**
+(verified in both the BF16 base and the INT8 conversion) — which is exactly
+why frank-small's SGLang NEXTN draft is a 0.00-acceptance ghost: SGLang
+reads the head from the TARGET checkpoint, vLLM reads the draft from a
+separate path. That difference is the whole trick:
+
+```
+--speculative-config '{"method":"mtp","model":"/models/Qwen3.5-9B","num_speculative_tokens":2}'
+```
+
+The alternative drafter (`z-lab/Qwen3.5-9B-DFlash`, 6 layers, target_layer_ids
+[1,5,…,29]) stays fetched on disk; flip `FRANK_LARGE_SPEC=dflash` and repoint
+`FRANK_LARGE_SPEC_DIR` to use it. Its card's "vLLM PR #40898 unmerged" claim
+is FALSE — `vllm/v1/spec_decode/dflash.py` ships in 0.30.
+
+## The numerics trap (greedy text is NOT preserved by spec decode)
+
+Measured: temperature-0 completions DIFFER across arms (sha `1318aeb6` base,
+`0208b45d` dflash, `20c41c38` mtp). Batched verification changes the bf16
+reduction order, argmax near-ties diverge, and the divergence cascades.
+vLLM spec decode is distribution-preserving, not bitwise-greedy-preserving.
+The gufo-era "greedy == autoregressive text" law does not carry over; do not
+chase text drift between spec and no-spec arms as a regression.
+
+## Payload
+
+- Target: 4 safetensors shards (17.53 GiB) + config/tokenizer under
+  `/home/dank/models/MiMo-V2.6-Distill-Qwen-9B-BF16/`, each shard
+  byte-count + LFS sha256-verified by
+  `old/download-frank-large-mimo.sh` (`--verify-only` re-checks; no token on
+  the curl command line).
+- MTP source: `old/download-frank-large-qwen35.sh` →
+  `/home/dank/models/Qwen3.5-9B/` (4 shards behind
+  `model.safetensors.index.json`; the shards carry the odd HF asset names
+  `model.safetensors-0000N-of-00004.safetensors` — the index resolves them,
+  the engine boots from exactly this tree).
+- DFlash drafter: `old/download-frank-large-dflash.sh` →
+  `/home/dank/models/Qwen3.5-9B-DFlash/`.
+- **Payload status 2026-10-02:** all three trees moved to
+  `/home/dank/.AGENTTRASH/20261002-retired-llm-payloads/` (59 GiB staged with
+  the other retired frank payloads) and the fetch trio moved to `old/` — the
+  endpoint is two runtimes past this one, and re-fetch is cheaper than disk.
+- The serve script gates the image by REPO DIGEST (rootless podman resolves
+  tags from the network; the pin exists to boot exactly these bytes
+  offline), then config + index + exact shard count, then the spec draft
+  (either weight layout: single `model.safetensors` or a sharded index),
+  then `/dev/kfd` + whole `/dev/dri` (never individual renderD nodes —
+  Dankserver 2026-09-29 lesson).
+
+## Client contract
+
+- **Alias:** `frank-large` (the `id` at `/v1/models`).
+- **Reasoning:** `--reasoning-parser qwen3`; vLLM emits the thinking text in
+  **`reasoning`**, NOT llama.cpp/gufo's `reasoning_content` — the harness
+  `models.yml` compat block for this provider sets
+  `reasoningContentField: "reasoning"`. Miss that and thinking silently
+  vanishes from transcripts.
+- **Tools:** `--enable-auto-tool-choice --tool-call-parser qwen3_coder`.
+- **Vision:** the checkpoint carries the `qwen3_5_vision` tower but the
+  serve script passes `--language-model-only` — the endpoint stays
+  text-only by decision; dropping the flag brings vision back.
+- **Sampling:** request-side; the harness sends the Qwen3.5-family recipe
+  (temp 0.7, top_p 0.8) via `compat.extraBody`, and
+  `chat_template_kwargs.{enable_thinking,preserve_thinking}` — the
+  `chat_template_kwargs` form is what vLLM reads; a top-level
+  `preserve_thinking` is ignored there.
+- **Env hygiene:** every tunable is `FRANK_LARGE_`-prefixed because HIP
+  reads bare `TEMP`/`HOST`/`GPU_MAX_HEAP_SIZE` and the serve script exports
+  the env file with `set -a`.
+
+## Ops
+
+- Boot ~200–240 s warm; health 503 while loading (do not kill it).
+- The serve script sweeps stale container identities before starting:
+  `frank-large-mimo` itself and the bench-window container `vllm-mimo`.
+- Container death → `podman wait` returns non-zero from ExecStart →
+  `Restart=always` self-heals (a clean exit-0 drain also restarts, the
+  frank-primary lesson).
+- Bench windows: `systemctl stop frank-large` is enough (mask fails on the
+  relinked unit; the deploy restores).
+- First-deploy facts (2026-10-01, deploy #3): healthy after **230 s**;
+  boot log `GPU KV cache size: 819,332 tokens, Maximum concurrency for
+  262,144 tokens per request: 3.13x` — four sessions share the pool
+  (the scheduler, not a carve). Live battery green: alias `frank-large`,
+  reasoning split into `reasoning`, tool_calls with correct args, and
+  `/metrics` acceptance pos0 69% pos1 44% (54/86 draft tokens accepted),
+  matching the bench window.
+
+## Rollback
+
+Git tag **`frank-large-gufo-q4kxl-20261001`** — gufo v0.1.1 (built from
+source, `scripts/build-gufo.sh` stays on disk) + the Q4_K_XL Qwen3.8-27B
+payload + DFlash2 drafter, all still on the host. Restoring the tag's
+unit/env/serve trio and redeploying returns the 27B endpoint.
+
+# Citations
+
+- Model: https://huggingface.co/XiaomiMiMo/MiMo-V2.6-Distill-Qwen-9B
+- Draft source: https://huggingface.co/Qwen/Qwen3.5-9B
+- DFlash drafter: https://huggingface.co/z-lab/Qwen3.5-9B-DFlash
+- Engine image: https://hub.docker.com/r/kyuz0/vllm-therock-gfx1151
+- Serve script: `scripts/serve-frank-large.sh`; tunables: `scripts/frank-large.env`
+- Predecessor: [[frank-large-gufo-qwen38-27b]]; sibling on the same arch:
+  [[frank-small-sglang-mimo-v26]]

serverAppConfigs-docs/LLM/frank-small-llama-mimo-q5km.md

@@ -0,0 +1,171 @@
+---
+type: reference
+title: "frank-small on Dankclaw — llama.cpp, MiMo-V2.6-Distill-Qwen-9B Q5_K_M GGUF"
+description: "The active frank-small runbook — a CUDA (sm_86) build of llama.cpp master serving bartowski's MiMo-V2.6-Distill-Qwen-9B Q5_K_M GGUF on the RTX 3090: three conversations of 180,224 tokens, mmproj vision, the asymmetric KV contract (f16 K + q8_0 V), the two VRAM measurements that set the pool size, and what to check when it misbehaves."
+tags:
+  - llm
+  - frank-small
+  - dankclaw
+  - llama-cpp
+  - rtx3090
+  - mimo
+  - gguf
+  - kv-cache
+timestamp: 2026-10-02
+---
+
+# frank-small — llama.cpp MiMo-V2.6 Q5_K_M (RTX 3090)
+
+> **ACTIVE since 2026-10-02.** One quant step down from
+> [[frank-small-llama-mimo-q6k]] on the same runtime and the same KV
+> contract, then **resized twice the same day** on Frank's orders: first to
+> three contexts with vision, then down to the pool that keeps 2–3 GiB of the
+> card free. Frank's arithmetic request — "as much as can fit while keeping
+> overhead open space to 2-3gb on the 3090" — is answered by the two
+> measurements in *Shape and the KV contract* below; the answer is
+> **3 × 180,224**, not 3 × 262,144.
+
+> **The tradeoff is the per-slot window.** Three sessions cost 81,920 tokens
+> of per-conversation memory versus the two-slot shape: 180,224 tokens each
+> instead of the model's native 262,144. See *What three slots cost*.
+
+## Identity
+
+|Field|Value|
+|---|---|
+|Endpoint|`192.168.0.143:7771`, served alias `frank-small`|
+|Runtime|llama.cpp **master** (qwen35 model support), CUDA build `sm_86`, at `/home/dank/GitPackages/llama-master/build-cuda/bin/llama-server` — unchanged across the quant swap|
+|Payload|`bartowski/MiMo-V2.6-Distill-Qwen-9B-GGUF` **Q5_K_M**, 6,876,124,704 B (6.40 GiB), sha256 `04cf46f2b5584ef922697d6c063f94879434fb86c467f1a09e52daf0cddaf35f` + the repo's `mmproj-MiMo-V2.6-Distill-Qwen-9B-f16.gguf`, 918,166,048 B, sha256 `ff348f3180a63188aa7285db85f550fe38acb61dd013c599eb8bad08d2cc2576` — both at `/home/dank/models/MiMo-Q5KM/`, fetched and byte- and sha256-verified by `scripts/download-frank-small.sh`|
+|Unit|root `frank-small.service` (repo `scripts/frank-small.service`, `User=dank`, relinked), env `scripts/frank-small.env`, serve `scripts/serve-frank-small.sh`|
+|Repo pins|`tests/frank_small_unit.bats` (stub-argv seam + the VRAM arithmetic), `deploy_llm_small` gates in `scripts/deploy-remote.sh`|
+
+## Shape and the KV contract
+
+`--ctx-size 540672 --parallel 3` — llama.cpp divides the pool across slots, so
+each conversation gets `n_ctx_slot = 180224` (boot log: `n_slots = 3,
+n_ctx_slot = 180224, kv_unified = 'false'`). `CTX_SIZE` MUST stay divisible by
+`PARALLEL`; the bats seam enforces it.
+
+The KV contract is Frank's: **K unquantized (f16), V `q8_0`** —
+`--cache-type-k f16 --cache-type-v q8_0`. Two hard dependencies ride on it:
+
+- **`--flash-attn on` is mandatory**: quantized V exists only on the FA path.
+  Without the flag the engine falls back silently and the contract becomes a
+  lie. The serve script pins the trio; the bats seam pins the script.
+- The VRAM budget: attention KV lives in the 8 full-attention layers only
+  (4 KV heads × 256 dim). K f16 ≈ 16 KiB/token, V q8_0 ≈ 8.5 KiB/token
+  (34 B per 32-element block) ⇒ ~24.5 KiB/token over the pool.
+
+**The pool size came from two boots, not from a formula.** `free`-style
+arithmetic on this stack is unreliable because the compute buffers, CUDA
+graphs and now the vision tower all bill separately, so the fit was measured
+at two shapes and solved between them:
+
+|Shape|Projector|`nvidia-smi` used|KV at 24.5 KiB/token|Non-KV (weights + buffers + projector)|
+|---|---|---|---|---|
+|2 × 262,144 (pool 524,288)|none|20,190 MiB|12,800 MiB|**7,390 MiB**|
+|3 × 196,608 (pool 589,824)|f16|22,972 MiB|14,113 MiB|**8,859 MiB**|
+
+The two rows price the projector and its image buffers at **~1.4 GiB** — not
+the ~0.9 GiB the 918 MB file suggests, because the vision encoder's own
+activation and compute buffers ride along. Solving the second row's non-KV cost
+against Frank's 2–3 GiB band: the card is 24,576 MiB, so
+`KV ≤ 24,576 − 9,216 − 2,048 = 13,312 MiB` ⇒ pool ≤ 556,630 tokens, rounded
+down to a multiple of 3 × 4,096 ⇒ **540,672**, i.e. 13,265 MiB of KV and
+**2,441 MiB free** (measured, 21,684 MiB used).
+
+**Why 3 × 262,144 was never possible.** Three native windows are 786,432
+tokens = 18.4 GiB of KV *alone*; plus the 8.86 GiB of non-KV cost that is
+27.3 GiB on a 24 GiB card. The quant step bought 0.86 GiB; the geometry needed
+6 GiB.
+
+## Measured on this shape (2026-10-02, 3 × 180,224 + vision)
+
+|Check|Result|
+|---|---|
+|Boot to healthy|~20 s, `n_slots = 3`, `n_ctx_slot = 180224`, `loaded multimodal model` logged|
+|VRAM|**21,684 MiB used / 2,441 MiB free** of 24,576 (`nvidia-smi`, steady state; the KV pool is allocated whole at boot, so this is the ceiling)|
+|Decode, temp 0, no thinking|**106.8 tok/s** (256-token completion at ~1,300-token depth)|
+|Prefill|**3,814 tok/s** (same request)|
+|Three concurrent requests|all three answered in 1.7 s wall — real parallel slots|
+|Fourth concurrent request|answered in 3.0 s while three ran — queued behind a released slot, as the slot ceiling requires|
+|Vision|`image_url` part with a 64×64 solid-red PNG → **"Red."**|
+|Tools|native `tool_calls` with correct arguments (`get_weather {"city":"Seattle","unit":"f"}`)|
+|Thinking|`reasoning_content` populated (145 chars on a "think first" prompt); `reasoning_effort: "low"` accepted|
+|Over-slot prompt|`400 exceed_context_size_error: request (184001 tokens) exceeds the available context size (180224 tokens)` — refused, never silently truncated|
+
+Decode is ~5 % **faster** than the 2 × 262,144 shape measured earlier the same
+day (100.4–101.8 tok/s): a shorter per-slot window means a shorter attention
+scan at equal prompt depth, and one less slot's DeltaNet state resident.
+
+## What three slots cost
+
+The per-slot window is the whole price. Two consequences the fleet depends on:
+
+- **A prompt over 180,224 tokens is a hard 400**, so `~/.omp/agent/models.yml`
+  sets `contextWindow: 180224` for this endpoint — the harness compacts before
+  it reaches the ceiling instead of discovering it.
+- **frank-primary's compaction prompt no longer fits with room to answer.**
+  Its trigger is 183,501 tokens — already above this slot — and its summary
+  budget is 32,768 output tokens. On the two-slot shape (262,144) that left
+  32.8k of output room, which is why that number was chosen; on this shape a
+  single compaction request must be truncated or split. Nothing in the fleet
+  routes compaction here automatically, but any runbook that claims frank-small
+  "fits a compaction request in ONE request" is describing the 262,144 shape.
+
+## Client contract
+
+- Thinking arrives in **`reasoning_content`** (`--reasoning-format deepseek`),
+  the field `models.yml` reads for this endpoint.
+- **Thinking is off unless the request asks**: the MiMo GGUF template defaults
+  `enable_thinking` false, so `models.yml` ships
+  `chat_template_kwargs: {enable_thinking: true, preserve_thinking: true}` in
+  `extraBody`. A raw client that omits it gets a direct answer — measured, and
+  not a fault. Trivial questions produce no thinking even with the flag on.
+- Tools: `--jinja` + server-side parsing, verified 2026-10-02.
+- **Vision is on**, through the projector file (`--mmproj`). Use the
+  OpenAI-standard `image_url` content part — that path answers correctly
+  ("Red." on a solid-red PNG). The legacy top-level `images` field is **not**
+  read by `/v1/chat/completions`: the same request sent that way answered
+  "Brown.", i.e. it generated from text alone. `/completion` takes `images`
+  but refuses a request without `messages` in this build, so chat is the only
+  vision path.
+- Setting `MMPROJ_FILE=` (explicitly empty) in `scripts/frank-small.env` drops
+  `--mmproj` and returns ~1.4 GiB of the card to KV — the documented text-only
+  opt-out. `MMPROJ_FILE` pointing at a file that does not exist is a boot
+  refusal, never a silent fallback to blindness.
+
+## No spec decode
+
+The GGUF has no MTP tensors (the same fact that made SGLang's NEXTN a 0.00
+ghost), and the cross-model DFlash draft
+(`/home/dank/models/DFlash-Qwen35/qwen3.5-9b-dflash-Q8_0.gguf`) is an
+optimization candidate, not a default — acceptance has to be measured on this
+traffic first. The bats seam fails if a draft flag ships unannounced.
+
+## Rollback
+
+|Target|How|
+|---|---|
+|Q6_K, same runtime|`scripts/frank-small.env` `MODEL_FILE=/home/dank/models/MiMo-Q6K/MiMo-V2.6-Distill-Qwen-9B-Q6_K.gguf` (7,793,710,624 B, still on disk) and the deployer's `LLM_SMALL_MODEL_FILE` default. Q6_K needs ~1.4 GiB more than Q5_K_M, so pair the rollback with `CTX_SIZE` 49,152 tokens lower or `MMPROJ_FILE=`|
+|Two 262,144 windows|`CTX_SIZE=524288 PARALLEL=2` (measured 20,190 MiB used, 4,386 free) and `contextWindow: 262144` in `models.yml` — choose it when a single conversation needs the native window or when frank-primary's compaction request has to land here|
+|Vision off|`MMPROJ_FILE=` in the env file; the deployer gate then has to be satisfied too, so point `LLM_SMALL_MMPROJ` at any real projector file or drop the gate in the same commit|
+|SGLang + INT8|[[frank-small-sglang-mimo-v26]] — venv and old unit/env/serve trio still in place; the INT8 payload was staged to `/home/dank/.AGENTTRASH/20261002-retired-llm-payloads/` on 2026-10-02, so restore it there or re-fetch with `old/download-frank-small-sglang-int8.sh`|
+
+## When it misbehaves
+
+- Port 7771 dark: `journalctl -u frank-small -n 50`; the serve script exits 1
+  naming the missing file if the binary or GGUF moved.
+- `No CUDA GPUs are available`: the 3090 vanished from the bus or the UVM node
+  is missing — `nvidia-modprobe -u -c=0` (the serve script runs it).
+- Requests queue behind each other: three concurrent long generations are the
+  slot ceiling by contract; a fourth waits for a slot to free (measured: 1.7 s
+  for the three, 3.0 s for the fourth).
+- `400 … exceeds the available context size (180224 tokens)`: the conversation
+  outgrew ONE slot. This is not pool exhaustion — the pool is 540,672 tokens
+  and the per-slot window is the binding limit. Shorten the prompt, compact, or
+  roll back to the 262,144 shape above.
+- Vision answers a color it never saw: the client sent `images` instead of an
+  `image_url` content part (see *Client contract*).
+- Slow decode after a CTX or quant change: check `nvidia-smi` — pushing KV out
+  of VRAM into unified memory halves the rate silently.

serverAppConfigs-docs/LLM/frank-small-llama-mimo-q6k.md

@@ -0,0 +1,110 @@
+---
+type: reference
+title: "frank-small on Dankclaw — llama.cpp, MiMo-V2.6-Distill-Qwen-9B Q6_K GGUF (retired)"
+description: "The retired 2026-10-01 frank-small runbook — the first llama.cpp + GGUF shape on the RTX 3090: two native-window sessions, the asymmetric KV contract (f16 K + q8_0 V) that motivated the move off SGLang, the VRAM fit at Q6_K, and what to check when it misbehaves. Superseded the same night by the Q5_K_M payload."
+tags:
+  - llm
+  - frank-small
+  - dankclaw
+  - llama-cpp
+  - rtx3090
+  - mimo
+  - gguf
+  - kv-cache
+timestamp: 2026-10-01
+---
+
+# frank-small — llama.cpp MiMo-V2.6 Q6_K (RTX 3090)
+
+> **RETIRED 2026-10-01 → 10-02** (one night): superseded by
+> [[frank-small-llama-mimo-q5km]], the same runtime and shape one quant rung
+> down. The Q6_K file stays at `/home/dank/models/MiMo-Q6K/` as the rollback
+> target.
+>
+> Original order (Frank, 2026-10-01 night): move this endpoint to the GGUF
+> path for **asymmetric KV** — unquantized K + q8_0 V — which only llama.cpp
+> exposes per-tensor. That contract is what the current deployment kept.
+
+## Identity
+
+|Field|Value|
+|---|---|
+|Endpoint|`192.168.0.143:7771`, served alias `frank-small`|
+|Runtime|llama.cpp **master** (qwen35 model support), CUDA build `sm_86`, at `/home/dank/GitPackages/llama-master/build-cuda/bin/llama-server`|
+|Payload|`bartowski/MiMo-V2.6-Distill-Qwen-9B-GGUF` **Q6_K**, 7,793,710,624 B, sha256 `ef96d05a2ddf2cbb450d1af1ac3860ec769d3575bafa70692ee5609bda3fad7d`, at `/home/dank/models/MiMo-Q6K/`|
+|Unit|root `frank-small.service` (repo `scripts/frank-small.service`, `User=dank`, relinked), env `scripts/frank-small.env`, serve `scripts/serve-frank-small.sh`|
+|Repo pins|`tests/frank_small_unit.bats` (stub-argv seam), `deploy_llm_small` gates in `scripts/deploy-remote.sh`|
+
+## Shape and the KV contract
+
+`--ctx-size 524288 --parallel 2` — llama.cpp divides the pool across slots,
+so each session gets `n_ctx_slot = 262144`, the model's full native window
+(boot log confirms `n_slots = 2, n_ctx_slot = 262144, kv_unified = 'false'`).
+
+Frank's KV contract: **K unquantized (f16), V `q8_0`** —
+`--cache-type-k f16 --cache-type-v q8_0`. Two hard dependencies ride on it:
+
+- **`--flash-attn on` is mandatory**: quantized V exists only on the FA
+  path. Without the flag the engine falls back silently and the contract
+  becomes a lie. The serve script pins the trio; the bats seam pins the
+  script.
+- The VRAM budget: attention KV lives in the 8 full-attention layers only
+  (4 KV heads x 256 dim). K f16 ≈ 16 KiB/token, V q8_0 ≈ 8.5 KiB/token
+  (34 B per 32-element block) ⇒ ~24.5 KiB/token over the 524,288-token pool
+  ≈ **12.6 GiB**. Weights 7.8 + KV 12.6 + compute buffers/graphs ≈
+  **20.9 GiB measured** against 24.5 GiB — the "room for checkpoints" the
+  order asked for. If graph capture OOMs, halve `CTX_SIZE`; never quantize
+  K to fix it.
+
+## Client contract
+
+- Thinking arrives in **`reasoning_content`** (`--reasoning-format
+  deepseek`) — the same field SGLang's qwen3 parser used, so `models.yml`
+  `reasoningContentField` did NOT move. Live-verified 2026-10-01 22:57.
+- **Thinking is off unless the request asks**: the MiMo GGUF template
+  defaults `enable_thinking` false; `models.yml` now ships
+  `chat_template_kwargs: {enable_thinking: true, preserve_thinking: true}`
+  in `extraBody`. A raw client that omits it gets a no-thinking answer
+  (measured — this is not a fault).
+- Tools: `--jinja` + server-side tool parsing, verified
+  `tool_calls` round-trip 2026-10-01 23:0x.
+- **Text-only now.** The GGUF carries no vision projector; the
+  in-checkpoint tower existed only in the SGLang safetensors. `models.yml`
+  `input: [text]`.
+
+## No spec decode (yet)
+
+The GGUF has no MTP tensors (same fact that killed SGLang NEXTN), and the
+cross-model DFlash draft (`/home/dank/models/DFlash-Qwen35/qwen3.5-9b-dflash-Q8_0.gguf`,
+Q8_0) is an optimization-loop candidate, not a default — `--spec-type
+draft-dflash` needs acceptance measured on this traffic first. The bats
+seam fails if a draft flag ships unannounced.
+
+## Build
+
+No ccache, CUDA 12.4 system nvcc with `--gcc-install-dir`-free gcc-13 host
+compiler (nvcc 12.4 rejects gcc ≥ 14):
+
+```
+cmake -B build-cuda -S . -DCMAKE_BUILD_TYPE=Release -DGGML_CUDA=ON \
+  -DCMAKE_CUDA_ARCHITECTURES=86 -DCMAKE_CUDA_HOST_COMPILER=/usr/bin/gcc-13 \
+  -DLLAMA_CURL=OFF -DGGML_CCACHE=OFF
+cmake --build build-cuda -j 16 --target llama-server
+```
+
+The SGLang venv (`/home/dank/GitPackages/sglang-cu128`) and the INT8
+AutoRound payload (`/home/dank/models/MiMo-V2.6-Distill-Qwen-9B-INT8-W8A16-AutoRound`)
+stay on disk as the rollback: repoint the unit's serve script and env from
+git history (`git show frank-large-gufo-q4kxl-20261001` era is BEFORE this;
+use the parent of the 2026-10-01 night swap commit).
+
+## When it misbehaves
+
+- Port 7771 dark: `journalctl -u frank-small -n 50`; the serve script
+  exits 1 naming the missing file if the binary or GGUF moved.
+- `No CUDA GPUs are available`: the 3090 vanished from the bus or the UVM
+  node is missing — `nvidia-modprobe -u -c=0` (the serve script runs it).
+- Requests hang at first token while `/health` is 200: both slots busy —
+  two concurrent long generations are the ceiling by contract.
+- Slow decode after a CTX change: check `nvidia-smi` — swapping KV out of
+  VRAM into unified memory halves the rate silently.

serverAppConfigs-docs/LLM/frank-small-sglang-mimo-v26.md

@@ -1,7 +1,7 @@
 ---
 type: reference
-title: "frank-small on Dankclaw — SGLang, MiMo-V2.6-Distill-Qwen-9B INT8 W8A16"
+title: "frank-small on Dankclaw — SGLang, MiMo-V2.6-Distill-Qwen-9B INT8 W8A16 (RETIRED)"
-description: "The active frank-small runbook — SGLang in a self-contained python3.12 venv serving leteschlead MiMo-V2.6-Distill-Qwen-9B-INT8-W8A16-AutoRound (qwen3_5 hybrid, 9B dense, in-checkpoint MTP + vision) on the RTX 3090: identity, payload, KV arithmetic, the NEXTN spec-decode shape, the CUDA 13 nvcc toolchain the JIT needs, deployed flags, the client contract, and what to check when it misbehaves."
+description: "The RETIRED frank-small runbook (2026-10-01, hours) — SGLang in a self-contained python3.12 venv serving leteschlead MiMo-V2.6-Distill-Qwen-9B-INT8-W8A16-AutoRound (qwen3_5 hybrid, 9B dense, in-checkpoint MTP + vision) on the RTX 3090: identity, payload, KV arithmetic, the NEXTN spec-decode ghost, the CUDA 13 nvcc toolchain the JIT needs, and the measured numbers kept for the record. Replaced by [[frank-small-llama-mimo-q6k]] the same night."
 tags:
   - llm
   - frank-small
@@ -17,9 +17,14 @@ timestamp: 2026-10-01
 
 # frank-small — SGLang MiMo-V2.6-Distill-Qwen-9B INT8 W8A16 (NEXTN MTP)
 
-> **ACTIVE since 2026-10-01.** Replaces [[frank-small-gemma4-12b]]
+> **RETIRED 2026-10-01, hours after cutover.** Replaced by
-> (llama.cpp Gemma4-12B QAT Q4_K_M, now the rollback target). Same port
+> [[frank-small-llama-mimo-q6k]] — llama.cpp CUDA on the MiMo Q6_K GGUF,
-> (7771), same alias (`frank-small`), same client contract surface.
+> chosen for the asymmetric KV (f16 K + q8_0 V) SGLang cannot express.
+> The venv (`/home/dank/GitPackages/sglang-cu128/`) stays in place; the INT8
+> payload was staged to
+> `/home/dank/.AGENTTRASH/20261002-retired-llm-payloads/` on 2026-10-02 when
+> the llama.cpp endpoint was two runtimes deep — recoverable until the trash
+> is purged.
 
 ## Identity
 
@@ -47,7 +52,10 @@ timestamp: 2026-10-01
   trigger 183,501 + a 32,768-token summary budget = 216,269) now fits
   one request. A single request may grow to the whole pool — a
   170,019-token prompt was verified at the 4-slot shape — the grid is
-  the two-way concurrency contract.
+  the two-way concurrency contract. **That compaction arithmetic is
+  specific to a 262,144-token slot**: the ACTIVE shape runs three slots of
+  180,224, which 183,501 tokens already exceeds — see *What three slots
+  cost* in [[frank-small-llama-mimo-q5km]].
 - **KV:** `fp8_e4m3` — 16 KiB/token (8 attn layers x 2 x 4 heads x 256
   dim x 1 B), pool 9.31 GiB.
 - **The memory shape (the 4.5 GiB the naive math misses):** the DeltaNet
@@ -118,8 +126,11 @@ change lands.
 
 Eight safetensors shards (12,891,718,440 B total) + tokenizer and config
 files under `/home/dank/models/MiMo-V2.6-Distill-Qwen-9B-INT8-W8A16-AutoRound/`,
-all LFS sha256-verified by `scripts/download-frank-small.sh` (`--verify-only`
+all LFS sha256-verified by `old/download-frank-small-sglang-int8.sh`
-re-checks). The serve script refuses to start with a missing shard.
+(`--verify-only` re-checks) — the script moved to `old/` when the endpoint
+cut over to llama.cpp + GGUF on 2026-10-01 night, and
+`scripts/download-frank-small.sh` now fetches the GGUF quant instead. The
+serve script refuses to start with a missing shard.
 
 ## Client contract
 
@@ -155,9 +166,12 @@ re-checks). The serve script refuses to start with a missing shard.
 
 ## Rollback
 
-[[frank-small-gemma4-12b]] — the llama.cpp Gemma4-12B QAT payload and binary
+[[frank-small-llama-mimo-q5km]] is the live shape two steps forward; the
-remain intact on disk; its unit/env/serve trio is in `old/` after the
+practical rollback today is [[frank-small-llama-mimo-q6k]] (payload at
-cutover.
+`/home/dank/models/MiMo-Q6K/`). This era's own rollback, the llama.cpp
+Gemma4-12B QAT payload, was staged to
+`/home/dank/.AGENTTRASH/20261002-retired-llm-payloads/Gemma4-12B-QAT/` on
+2026-10-02; its unit/env/serve trio remains in `old/`.
 
 # Citations
 

serverAppConfigs-docs/LLM/index.md

@@ -1,6 +1,6 @@
 # LLM Fleet — Knowledge Base
 
-Documentation for the LLM inference fleet on the two hosts (2026-10-01):
+Documentation for the LLM inference fleet on the two hosts (2026-10-02):
 **Dankserver runs the tcclaviger RDNA4 vLLM fork on frank-primary** (the
 digest-pinned `tcclaviger/vllm` image driving `podman run` directly, serving
 orcarouter/Qwen3.8-Flash-Next-Uncensored-FP8 across both Radeon AI PRO R9700s
@@ -10,12 +10,12 @@ headroom; 6 do not fit), no speculative decode, MNBT 1024, MoE gather 32x64, uti
 reasoning/tool parsers wired (qwen3 / qwen3_xml) with preserve_thinking on,
 measured 36.4-39.4 tok/s decode — the 2026-10-01 bf16 no-spec cutover plus
 the night ladder; rollback frank-primary-fp8-mtp-20261001), and Dankclaw
-runs two endpoints —
+runs two endpoints — **ROCmFPX llama.cpp (frank-large, port 7770:
-gufo v0.1.1 (the Strix Halo engine, built from source; frank-large,
+kingjones777/Qwen3.8-27B-ROCmFP4-STRIX-MTP-GGUF Q8_0_ROCMFPX_AGENT, a ftype-115
-unsloth Qwen3.8-27B UD-Q4_K_XL (vanilla) + the z-lab DFlash2 Q4_K_M drafter
+file only that fork can read, with the repo's MTP head drafting 4 tokens, ONE
-on the gfx1151 iGPU, 4 x 163840 ctx with f16 KV, port 7770) and mainline llama.cpp (frank-small,
+session at 262,144, f16 KV, vision via the bundled projector)** and
-HauhauCS Gemma4-12B QAT Q4_K_M with MTP and vision on the RTX 3090,
+**llama.cpp CUDA (frank-small, port 7771: MiMo-V2.6-Distill-Qwen-9B Q5_K_M
-port 7771).
+GGUF, 2 x 262,144, asymmetric KV f16 K + q8_0 V, on the RTX 3090)**.
 
 > 2026-08-22: the former `vllmScripts` repository is merged into the
 > serverAppConfigs repository. This knowledge base moved from the old repo's
@@ -39,14 +39,14 @@ fp8 KV, MNBT 1024, gather 32x64, qwen3 parsers]
     subgraph Dankclaw[AMD Strix Halo - Ubuntu 26.04]
         E2[RTX 3090 (PCIe x4 dock) +
 gfx1151 iGPU 114 GiB GTT]
-        E[stock llama.cpp (unsloth b11030, gfx1151)
+        E[ROCmFPX llama.cpp gfx1151, pinned fork build
 frank-large, port 7770]
-        F[Qwen3.8-27B UD-Q8_K_XL + DFlash2 Q4_K_M drafter
+        F[Qwen3.8-27B Q8_0_ROCMFPX_AGENT + MTP-Q4_0 head
-262144 ctx / 1 slot, DFlash2 depth 4, q8_0 KV]
+1 x 262144 ctx, f16 KV, draft n-max 4, mmproj vision]
-        G[mainline llama.cpp 37b53fd
+        G[llama.cpp master CUDA sm_86
 frank-small, port 7771]
-        H[Gemma4-12B QAT Q4_K_M
+        H[MiMo-V2.6 Q5_K_M GGUF
-4 x 262144 ctx, MTP + vision + audio]
+2 x 262144 ctx, f16 K + q8_0 V, flash-attn]
         E2 --> E --> F
         E2 --> G --> H
     end
@@ -65,7 +65,23 @@ frank-small, port 7771]
 
 ## Active Deployment
 
-- [[frank-primary-glm53-flash-w4a16]] — the ACTIVE frank-primary runbook
+- [[frank-large-rocmfpx-q8agent-mtp]] — **ACTIVE since 2026-10-02**
+  (root systemd unit, port 7770): the pinned kingjones30/ROCmFPX fork serving
+  the repo's highest-quality rung `Qwen3.8-27B-Q8_0_ROCMFPX_AGENT.gguf`
+  (26.28 GiB, ftype 115 = 187 plain-Q8_0 + 311 GGML_TYPE_Q8_0_ROCMFPX
+  tensors) with the bundled `mtp-Qwen3.8-27B-Q4_0.gguf` head drafting 4
+  tokens, one 262,144 window, f16 KV, the BF16 projector for vision. Why the
+  runtime is not interchangeable, the pinned-build set_rows history, the
+  measured envelope, rollback to gufo
+- [[frank-small-llama-mimo-q5km]] — **ACTIVE since 2026-10-02** (root
+  systemd unit, port 7771): llama.cpp master (CUDA sm_86) serving
+  bartowski/MiMo-V2.6-Distill-Qwen-9B-GGUF Q5_K_M on the RTX 3090,
+  THREE slots of 180,224 tokens (pool 540,672) with K f16 + V q8_0 and the
+  mmproj for vision, ~100 tok/s decode, ~2.7 GiB of the card left free.
+  The KV arithmetic, the flash-attn dependency, rollback to Q6_K
+- [[frank-primary-glm53-flash-w4a16]] — RETIRED 2026-09-30 → 10-01 (the
+- failed GLM experiment; frank-primary runs Qwen FP8 again, see the intro):
+  (2026-09-30, root systemd unit, port 7770): the tcclaviger RDNA4 vLLM fork
   (2026-09-30, root systemd unit, port 7770): the tcclaviger RDNA4 vLLM fork
   (digest-pinned `tcclaviger/vllm@sha256:da4029666…`, plain `podman run`,
   no shim) serving Intel/GLM-5.3-Flash-W4A16-AutoRound — AutoRound symmetric
@@ -162,14 +178,24 @@ frank-small, port 7771]
   (bs1/bs3/bs4/bs6, 8k–700k KV) on Dankserver RTX 3090
 - [[deployment]] — deploy and update process for both servers
 - [[frank-redo-2026-08-17]] — the 2026-08-17 topology change
-- [[frank-large-gufo-qwen38-27b]] — ACTIVE frank-large runbook since
+- [[frank-large-gufo-qwen38-27b]] — **RETIRED 2026-10-02 (the rollback
-  2026-09-28 (root systemd unit, port 7770): gufo (gufo-org/gufo) v0.1.1, the
+  target)** (root systemd unit, port 7770): gufo
-  Strix Halo inference engine, built from source on Dankclaw, serving
+  (gufo-org/gufo) v0.1.1, the Strix Halo inference engine, built from
-  unsloth/Qwen3.8-27B-GGUF:UD-Q8_K_XL (27B dense) with the z-lab DFlash2
+  source on Dankclaw, serving unsloth/Qwen3.8-27B-GGUF **UD-Q4_K_XL**
-  Q4_K_M drafter (adaptive block policy) on the gfx1151 iGPU — one full
+  with the z-lab DFlash2 Q4_K_M drafter — ONE session at the full native
-  262,144-token conversation, greedy, text-only, the two from-source build
+  262,144 window, unquantized f16 KV. Rolled back by Frank after one day
-  workarounds, the client-contract delta, the measured prefill/decode trade
+  of [[frank-large-vllm-mimo-v26-bf16]]. The two from-source build
-  against llama.cpp, troubleshooting
+  workarounds, the client-contract delta, the measured prefill/decode
+  trade against llama.cpp
+- [[frank-large-vllm-mimo-v26-bf16]] — RETIRED 2026-10-01 (one day;
+  rollback by Frank after the autoresearch loop): the digest-pinned
+  kyuz0/vllm-therock-gfx1151 image serving
+  XiaomiMiMo/MiMo-V2.6-Distill-Qwen-9B in FULL BF16, 4 x 262144 ctx,
+  CROSS-MODEL MTP from Qwen/Qwen3.5-9B — kept for the measured record:
+  battery decode 6.19 t/s (MTP-2) vs 9.01 t/s (no draft — spec NET-
+  NEGATIVE on real chat+reasoning traffic), the roof math, the greedy-text
+  numerics trap, and the death of every ≥6-bit vLLM quant path on gfx1151
+  (quark int8 rejected, FP8 kernels CDNA-only, torchao unwired)
 - [[frank-large-unsloth-qwen38-27b-q8xl]] — RETIRED 2026-09-28 (replaced by
   [[frank-large-gufo-qwen38-27b]]; rollback target): stock llama.cpp
   (unsloth b11030, ROCm gfx1151) serving
@@ -203,12 +229,27 @@ frank-small, port 7771]
   .hgn + quality overlay) on the gfx1151 iGPU — two 200k conversations over one
   400,384-position pool, deployed flags, memory model, client contract,
   troubleshooting (2026-09-16 → 09-19)
-- [[frank-small-gemma4-12b]] — ACTIVE frank-small runbook since 2026-09-17
+- [[frank-small-llama-mimo-q6k]] — **RETIRED 2026-10-01 → 10-02 (rollback
-  (root systemd unit, port 7771): mainline llama.cpp serving
+  target)**: llama.cpp (master, CUDA sm_86 build) serving
+  bartowski/MiMo-V2.6-Distill-Qwen-9B-GGUF **Q6_K** on the RTX 3090 — TWO
+  native 262,144-token sessions with ASYMMETRIC KV (K f16 unquantized + V
+  q8_0, which forces flash-attn), ~20.9/24.6 GiB measured, text-only (the
+  GGUF has no vision), reasoning_content, no spec decode; payload still at
+  `/home/dank/models/MiMo-Q6K/`
+- [[frank-small-sglang-mimo-v26]] — RETIRED 2026-10-01 (hours; replaced
+  by [[frank-small-llama-mimo-q6k]] for the asymmetric KV SGLang cannot
+  express; venv still on disk, INT8 payload staged to
+  `~dank/.AGENTTRASH/20261002-retired-llm-payloads/`): SGLang in a
+  self-contained python3.12 venv serving leteschlead MiMo-V2.6 INT8
+  W8A16 AutoRound, 2 x 262144 ctx, fp8 KV, in-checkpoint vision,
+  83.9 tok/s decode
+- [[frank-small-gemma4-12b]] — RETIRED 2026-09-17 → 2026-10-01 (replaced by
+  [[frank-small-sglang-mimo-v26]] → the llama.cpp MiMo shape; payload staged
+  to `~dank/.AGENTTRASH/20261002-retired-llm-payloads/Gemma4-12B-QAT/`):
+  mainline llama.cpp serving
   HauhauCS/Gemma4-12B-QAT-Uncensored-HauhauCS-Balanced Q4_K_M on the RTX
   3090, with the separate MTP assistant sidecar chained behind ngram-mod,
-  q8_0 KV, and `--fit` sizing the full 262k window into ~12 GiB. A same-day
+  q8_0 KV, and `--fit` sizing the full 262k window into ~12 GiB.
-  vLLM detour (2026-09-19) was reverted the same day; see below.
 - [[frank-small-vllm-gemma4]] — RETIRED 2026-09-19 (same-day revert,
   rollback target): vLLM serving google/gemma-4-12B-it-qat-w4a16-ct (w4a16
   Marlin) with the bf16 MTP assistant on the RTX 3090 — bf16 KV, 289,647-

serverAppConfigs-docs/index.md

@@ -213,14 +213,27 @@ The LLM inference fleet on the two hosts. Full index:
   agentionai/Qwen3.8-Flash-Next-ROCmFP4-FAST-imatrix (87.06 GiB,
   Q4_0_ROCMFP4_FAST) + own MTP draft, build-qwen38 (retired 2026-08-30;
   rollback record)
-- [[frank-large-gufo-qwen38-27b]] — the ACTIVE frank-large runbook since
+- [[frank-large-rocmfpx-q8agent-mtp]] — the ACTIVE frank-large runbook
-  2026-09-28 (root systemd unit, port 7770): gufo (gufo-org/gufo) v0.1.1,
+  since 2026-10-02 (root systemd unit, port 7770): the pinned kingjones30/
-  the Strix Halo inference engine, built from source on Dankclaw and
+  ROCmFPX fork (scripts/build-rocmfpx.sh) serving kingjones777/
-  serving unsloth/Qwen3.8-27B-GGUF:UD-Q8_K_XL (a 27B dense model) with the
+  Qwen3.8-27B-ROCmFP4-STRIX-MTP-GGUF **Q8_0_ROCMFPX_AGENT** — ftype 115,
-  z-lab DFlash2 Q4_K_M drafter on the gfx1151 iGPU — one 262,144-token
+  311 tensors of GGML_TYPE_Q8_0_ROCMFPX, a format no other loader on the
-  conversation, adaptive block policy, greedy, text-only, the two
+  host reads — with the repo's Q4_0 MTP head drafting 4 tokens and its BF16
-  from-source build workarounds, what changed in the client contract, the
+  projector, one session at the native 262,144 window
-  measured prefill/decode trade against llama.cpp
+- [[frank-large-gufo-qwen38-27b]] — RETIRED 2026-10-01 → 10-02 (rollback
+  target, payload and `scripts/build-gufo.sh` both on disk): gufo v0.1.1
+  built from source serving Qwen3.8-27B UD-Q4_K_XL + the z-lab DFlash2
+  drafter on the gfx1151 iGPU — the two from-source build workarounds, the
+  template-hash allowlist, the measured prefill/decode trade
+- [[frank-large-vllm-mimo-v26-bf16]] — RETIRED 2026-10-01 (same day;
+  replaced by the gufo rollback, then by
+  [[frank-large-rocmfpx-q8agent-mtp]]): the digest-pinned
+  kyuz0/vllm-therock-gfx1151 image (vLLM 0.30, TheRock ROCm 10) serving
+  XiaomiMiMo/MiMo-V2.6-Distill-Qwen-9B in FULL BF16 on the gfx1151 iGPU,
+  4 x 262,144 ctx, with CROSS-MODEL MTP spec decode from Qwen/Qwen3.5-9B —
+  the roof math (13.1 t/s base is ~94% of LPDDR5x BF16), the measured
+  MTP-vs-DFlash table (19.8 vs 17.4 t/s), the greedy-text numerics trap,
+  the `reasoning` field change, troubleshooting
 - [[frank-large-unsloth-qwen38-27b-q8xl]] — RETIRED 2026-09-28 (replaced by
   [[frank-large-gufo-qwen38-27b]]; rollback target): stock llama.cpp
   (unsloth b11030, ROCm gfx1151) serving
@@ -269,12 +282,22 @@ The LLM inference fleet on the two hosts. Full index:
 - [[ciru-activefpx-promptforge]] — the RETIRED v3 CIRU PromptForge runtime on Dankclaw (superseded 2026-08-24)
 - [[kingjones-strix-mtp]] — the kingjones Qwen3.8-27B ROCmFP4 STRIX MTP model on Dankclaw (active)
 - [[dealign-ornith-crack]] — the retired Ornith-1.5-35B-A3B CRACK model on Dankserver (retired 2026-09-02)
-- [[frank-small-sglang-mimo-v26]] — the ACTIVE frank-small runbook since
+- [[frank-small-llama-mimo-q5km]] — the ACTIVE frank-small runbook since
-  2026-10-01: SGLang in a self-contained python3.12 venv serving
+  2026-10-02: a CUDA (sm_86) build of llama.cpp master serving
+  bartowski/MiMo-V2.6-Distill-Qwen-9B-GGUF **Q5_K_M** on Dankclaw's RTX
+  3090, port 7771 — THREE conversations at 180,224 tokens each, the
+  asymmetric KV contract (f16 K + q8_0 V), vision through the repo's mmproj,
+  and the two VRAM measurements that set the pool size; ~100 tok/s decode
+- [[frank-small-llama-mimo-q6k]] — RETIRED 2026-10-01 (same day; replaced
+  by [[frank-small-llama-mimo-q5km]], payload still on disk): the first
+  llama.cpp + GGUF shape for this endpoint, Q6_K at the same
+  2 x 262,144 / f16-K + q8_0-V contract, 20.9 GiB measured
+- [[frank-small-sglang-mimo-v26]] — RETIRED 2026-10-01 (rollback target):
+  SGLang in a self-contained python3.12 venv serving
   letechlead/MiMo-V2.6-Distill-Qwen-9B-INT8-W8A16-AutoRound (qwen3_5
   hybrid, INT8 W8A16, in-checkpoint vision, NO MTP weights shipped) on
   Dankclaw's RTX 3090, port 7771 — fp8 KV, pinned 606,208-token pool,
-  4 x 151,552 slots, bf16 mamba states, eager prefill
+  2 x 262,144 slots, bf16 mamba states, eager prefill
 - [[frank-small-gemma4-12b]] — RETIRED 2026-10-01 (rollback target):
   mainline llama.cpp serving
   HauhauCS/Gemma4-12B-QAT-Uncensored-HauhauCS-Balanced at Q4_K_M

serverAppConfigs-docs/log.md

@@ -875,3 +875,121 @@ any repo service.
   verified (KV cache size, concurrency, MNBT 1024, gather 32x64) plus live
   contract probes (reasoning split, tool_calls). Rollback target for the
   2026-09-29 fp8+MTP4 shape: git tag `frank-primary-fp8-mtp-20261001`.
+
+## 2026-10-01 — frank-large cutover: gufo -> vLLM TheRock, MiMo-V2.6 BF16 + cross-model MTP
+
+- **Question that started it:** can a Qwen3.5 MTP drafter attach to
+  frank-small, and how does vLLM run the FULL BF16 MiMo checkpoint? Answer 1:
+  not on frank-small — SGLang reads the MTP head from the TARGET checkpoint
+  and MiMo ships ZERO `mtp.*` tensors (base and INT8 alike). Answer 2: the
+  digest-pinned `kyuz0/vllm-therock-gfx1151` image (vLLM 0.30 / torch 2.11 /
+  TheRock ROCm 10) boots BF16 MiMo on the gfx1151 iGPU in ~200-240 s, and
+  vLLM reads the draft from a SEPARATE path — so cross-model MTP from
+  `Qwen/Qwen3.5-9B` (the distill's source, identical 4096/248320/32-layer
+  geometry) works. Frank's call: ship that as frank-large.
+- **Bench window (frank-large stopped, one session, temp 0, 28,647-token
+  prefill / 512 decode):** base 17,876 t/s prefill / 13.1 t/s decode (13.1
+  is ~94% of the LPDDR5x BF16 roof - decode here is bandwidth, not a bug);
+  +DFlash-8 7,976 / 17.4 (acceptance 59%/26% per position); +MTP-2 7,380 /
+  19.8 (71%/44%, token rate 57.4%). Deployed MTP-2: +51% decode. Both spec
+  arms pay ~55-59% of prefill. `num_speculative_tokens` beyond 2 unmeasured.
+- **Measured numerics law:** spec arms do NOT reproduce the no-spec greedy
+  text (bf16 batched verification reorders reductions; near-tie argmaxes
+  cascade). vLLM spec decode is distribution-preserving, not
+  bitwise-greedy-preserving.
+- **Client-contract delta wired:** vLLM emits `reasoning`, not
+  `reasoning_content` - `models.yml` frank-large block now sets
+  `reasoningContentField: "reasoning"`, and the extraBody moved to the
+  Qwen3.5-family recipe (0.7/0.8) with `chat_template_kwargs`
+  (a top-level `preserve_thinking` is ignored by vLLM).
+- Deployer gap found pre-deploy: the spec-draft gate demanded a single
+  `model.safetensors` (the dflash layout) and would have rejected the
+  deployed MTP arm (4 shards behind an index); fixed to accept both
+  layouts, gated by bats. `z-lab/Qwen3.5-9B-DFlash`'s "vLLM PR #40898
+  unmerged" card claim is FALSE - `vllm/v1/spec_decode/dflash.py` ships in
+  0.30 (verified in-image); the drafter stays on disk as the alternative.
+- Host state: +38 GiB fetched (MiMo BF16 17.53 GiB, Qwen3.5-9B 18 GiB,
+  DFlash 2.58 GiB), disk down to ~26 GiB free - reclaim pass due. gufo
+  rollback pinned at git tag `frank-large-gufo-q4kxl-20261001`; engine,
+  Q4_K_XL payload and DFlash2 drafter all remain on the host.
+- First deploy attempt #2 crash-looped the container 51 times: the serve
+  script handed vLLM the HOST model path while the tree is mounted at
+  `/models` inside the container — transformers rejected it as a malformed
+  repo id. Fixed by converting to the container path (the spec draft was
+  already converted; the target was not), pinned by bats. Also fixed the
+  deployer's own draft gate (same single-file bug as the serve script's).
+- Reclaim (Frank approved): Swift-1.5 Q8_K_XL payload (30 G, refetchable via
+  `scripts/download-frank-large-swift.sh`) and the orphaned flux-2-klein
+  tree under `models/checkpoints` (23 G; ComfyUI mounts
+  `comfyui-data/models`, not this) purged via
+  `~/.AGENTTRASH/20261001-frank-large-reclaim/` — 52 G freed. KEPT the
+  1.1 G `Qwen3.8-27B-DFlash2-Q4_K_M.gguf` (it is the gufo rollback's
+  drafter, not a remnant).
+
+## 2026-10-02 — frank-large cutover: vLLM MiMo -> ROCmFPX Q8_0_ROCMFPX_AGENT + MTP
+
+Frank's order was the highest quality of
+`kingjones777/Qwen3.8-27B-ROCmFP4-STRIX-MTP-GGUF` on one full context, "whatever
+runtime it requires". The answer is forced by the file: `Q8_0_ROCMFPX_AGENT.gguf`
+carries tensor type 103 (`GGML_TYPE_Q8_0_ROCMFPX`, ftype 115), which stock
+llama.cpp, gufo and every other loader on the fleet cannot read — only the
+kingjones30/ROCmFPX fork can, so the runtime is pinned to commit
+`36e9acd40e10a87cd3c3ef8ec734668757dc8520` and built by
+`scripts/build-rocmfpx.sh`. Shape: 1 session x 262,144 (native), f16 KV, the
+bundled BF16 mmproj, and MTP from the repo's own `mtp-Qwen3.8-27B-Q4_0.gguf`
+(n-max 4, `--spec-draft-device ROCm0`).
+
+- **MTP is the whole performance story: 17.0-17.1 t/s with, 7.60-7.66 without**
+  (+123 %), one session, temp 0, production stopped per arm, acceptance 0.462 /
+  mean length ~1.86, at an 8-15 % prefill tax (325-351 vs 381 t/s). The AGENT
+  routing variant was chosen over the card's plain Q8_0 because it keeps 187
+  output-head tensors at plain Q8_0 instead of 2, which measured MTP acceptance
+  0.953 vs 0.911.
+- Steady GTT occupancy 63.1 GiB of the ~114 GiB aperture; boot to healthy 16 s.
+- Rollback: git tag `frank-large-gufo-q4kxl-20261001`. The retired vLLM-era
+  fetchers moved to `old/`. Runbook
+  [[frank-large-rocmfpx-q8agent-mtp]].
+
+## 2026-10-02 — frank-small: Q5_K_M, then three contexts + vision, sized by two boots
+
+Three orders in one day on the same endpoint: `bartowski/…Q5_K_M` on llama.cpp
+with two contexts ("whatever context size fits the 3090"), then **three
+contexts** "as much as can fit while keeping overhead open space to 2-3gb on
+the 3090", then **"frank-small should have vision"**.
+
+- **Final shape: `CTX_SIZE=540672 PARALLEL=3` = 3 x 180,224**, f16 K + q8_0 V
+  under flash attention, `--mmproj` vision, all layers on the 3090. Measured
+  **21,684 MiB used / 2,441 MiB free** of 24,576 — inside the 2-3 GiB band.
+- **The pool came from two boots, not a formula.** 2 x 262,144 with no
+  projector: 20,190 MiB used, KV 12,800 => non-KV **7,390 MiB**. 3 x 196,608
+  with the projector: 22,972 used, KV 14,113 => non-KV **8,859 MiB**. The
+  difference prices the vision tower at **~1.4 GiB for a 918 MB file** — the
+  encoder's own buffers ride with it. The first 3-slot deploy (3 x 196,608)
+  booted healthy but left only 1,153 MiB free, which is how the band got
+  violated and why the seam now prices non-KV at the measured 9 GiB and
+  REJECTS that shape.
+- **Three contexts cost window, and that is the tradeoff Frank was told:**
+  180,224 tokens per conversation, not the native 262,144. A longer prompt is
+  a hard `400 exceed_context_size_error`, so `models.yml` sets
+  `contextWindow: 180224`. Consequence: frank-primary's 183,501-token
+  compaction request no longer fits here with summary room; the two-slot
+  rollback is the way back.
+- Measured (2026-10-02, this shape): decode **106.8 t/s**, prefill **3,814
+  t/s** (~5 % faster decode than 2 x 262,144 — shorter slot, shorter attention
+  scan), three concurrent requests answered in parallel in 1.7 s with a fourth
+  queued at 3.0 s, native `tool_calls`, `reasoning_content` populated,
+  `reasoning_effort: low` accepted, solid-red PNG answered **"Red."**
+- **Vision contract gotcha:** `/v1/chat/completions` ignores the legacy
+  top-level `images` field — that request answered "Brown." for the same red
+  image, i.e. it generated from text alone. Only the `image_url` content part
+  reaches the model. `/completion` accepts `images` but this build still
+  requires `messages`.
+- Deployer gates on the projector too (`LLM_SMALL_MMPROJ`), so a missing
+  mmproj cannot silently ship a blind endpoint; `MMPROJ_FILE=` (explicitly
+  empty) in the env file is the documented text-only opt-out and returns ~1.4
+  GiB to KV.
+- Reclaim staged, **not purged, awaiting approval**:
+  `~/.AGENTTRASH/20261002-retired-llm-payloads/` (~59 GiB of superseded
+  payloads). Kept in place for rollback: `MiMo-Q6K/`, `MiMo-Q5KM/`,
+  `Qwen3.8-27B-UD-Q4_K_XL/` and `Qwen3.8-27B-UD-Q8_K_XL/` (the latter holds the
+  gufo DFlash2 drafter). Runbook [[frank-small-llama-mimo-q5km]].

tests/autoresearch_measure.bats

@@ -0,0 +1,150 @@
+#!/usr/bin/env bats
+# Seam: autoresearch measure client behavior.
+#
+# The measure client is the whole yardstick of the frank-primary speed
+# session, so its METRIC arithmetic gets a behavior test: against a
+# deterministic mock endpoint that emits a KNOWN number of completion
+# tokens at a KNOWN per-token delay, the reported decode rate must land
+# near the nominal rate, the usage accounting must come back exact, and an
+# unhealthy endpoint must fail loudly (no half-measured numbers).
+#
+# The mock implements only what the client calls: POST
+# /v1/messages/count_tokens (word count as the token count, which makes the
+# client's words/token calibration come out to exactly 1.0) and a streaming
+# POST /v1/chat/completions (one content chunk per completion token, then a
+# usage chunk, then [DONE]).
+
+load helpers
+
+MEASURE="$REPO_ROOT/scripts/autoresearch/frank-primary-measure.py"
+MOCK="$BATS_TEST_TMPDIR/mock.py"
+PORT=""
+MOCK_PID=""
+
+setup() {
+    # Deterministic mock; argv: <port> <delay-seconds>; env MOCK_FAIL=1
+    # makes the chat endpoint answer 500.
+    cat >"$MOCK" <<'PYEOF'
+import json, os, sys, time
+from http.server import BaseHTTPRequestHandler, HTTPServer
+
+PORT = int(sys.argv[1])
+DELAY = float(sys.argv[2])
+FAIL = os.environ.get("MOCK_FAIL", "0") == "1"
+
+
+class Handler(BaseHTTPRequestHandler):
+    def log_message(self, *a):
+        pass
+
+    def _send(self, obj, code=200):
+        b = json.dumps(obj).encode()
+        self.send_response(code)
+        self.send_header("Content-Type", "application/json")
+        self.send_header("Content-Length", str(len(b)))
+        self.end_headers()
+        self.wfile.write(b)
+
+    def do_POST(self):
+        n = int(self.headers.get("Content-Length", "0"))
+        body = json.loads(self.rfile.read(n) or b"{}")
+        if self.path == "/v1/messages/count_tokens":
+            text = body["messages"][0]["content"]
+            self._send({"input_tokens": max(1, len(text.split()))})
+        elif self.path == "/v1/chat/completions":
+            if FAIL:
+                self._send({"error": "engine down"}, 500)
+                return
+            max_tokens = int(body.get("max_tokens", 64))
+            self.send_response(200)
+            self.send_header("Content-Type", "text/event-stream")
+            self.end_headers()
+            for i in range(max_tokens):
+                ev = {"choices": [{"delta": {"content": "tok%d " % i}}]}
+                self.wfile.write(("data: %s\n\n" % json.dumps(ev)).encode())
+                self.wfile.flush()
+                time.sleep(DELAY)
+            prompt_words = max(1, len(body["messages"][0]["content"].split()))
+            ev = {"choices": [{"delta": {}}],
+                  "usage": {"prompt_tokens": prompt_words,
+                            "completion_tokens": max_tokens}}
+            self.wfile.write(("data: %s\n\n" % json.dumps(ev)).encode())
+            self.wfile.write(b"data: [DONE]\n\n")
+            self.wfile.flush()
+        else:
+            self._send({"error": "unknown path"}, 404)
+
+
+HTTPServer(("127.0.0.1", PORT), Handler).serve_forever()
+PYEOF
+}
+free_port() {
+    python3 -c 'import socket; s = socket.socket(); s.bind(("127.0.0.1", 0)); print(s.getsockname()[1]); s.close()'
+}
+
+start_mock() { # $1 = ok|fail
+    PORT="$(free_port)"
+    if [ "$1" = "fail" ]; then
+        MOCK_FAIL=1 python3 "$MOCK" "$PORT" 0.005 &
+    else
+        python3 "$MOCK" "$PORT" 0.005 &
+    fi
+    MOCK_PID=$!
+    local i
+    for i in $(seq 1 100); do
+        if python3 -c "import socket; socket.create_connection(('127.0.0.1', $PORT), 0.2).close()" 2>/dev/null; then
+            return 0
+        fi
+        sleep 0.1
+    done
+    echo "mock did not start" >&2
+    return 1
+}
+
+stop_mock() {
+    if [ -n "$MOCK_PID" ]; then
+        kill "$MOCK_PID" 2>/dev/null || true
+        wait "$MOCK_PID" 2>/dev/null || true
+        MOCK_PID=""
+    fi
+}
+
+@test "measure exists and compiles" {
+    [ -f "$MEASURE" ]
+    python3 -m py_compile "$MEASURE"
+}
+
+@test "measure computes decode rate and exact usage from a controlled endpoint" {
+    start_mock ok || return 1
+    run env AR_BASE="http://127.0.0.1:$PORT" AR_MODEL=mock python3 "$MEASURE"
+    stop_mock
+    [ "$status" -eq 0 ] || { echo "$output"; return 1; }
+    # run() merges stderr in; the contract is: exactly ONE json result line
+    # (the window script parses the same way), progress on stderr.
+    printf '%s\n' "$output" | python3 -c '
+import json, sys
+lines = [l for l in sys.stdin.read().splitlines() if l.startswith("{")]
+assert len(lines) == 1, "expected exactly one JSON line, got %d" % len(lines)
+d = json.loads(lines[0])
+# nominal 200 t/s (0.005 s/chunk) minus small per-chunk overhead
+for k in ("decode_tps", "decode_tps_115", "decode_tps_1056", "decode_tps_4096"):
+    v = d[k]
+    assert 60.0 <= v <= 320.0, "%s=%s outside [60, 320]" % (k, v)
+# the mock emits exactly max_tokens completion tokens; the accounting must be exact
+assert d["completion_tokens_4096"] == 256, d["completion_tokens_4096"]
+assert d["completion_tokens_115"] == 256
+# prompt sizing: calibration is exactly 1.0 words/token in the mock, so the
+# measured prompt lengths equal the targets
+assert d["prompt_tokens_115"] == 115
+assert d["prompt_tokens_4096"] == 4096
+print("measure claims verified")
+'
+}
+
+@test "measure fails loudly on an unhealthy endpoint instead of half-measuring" {
+    start_mock fail || return 1
+    run env AR_BASE="http://127.0.0.1:$PORT" AR_MODEL=mock python3 "$MEASURE"
+    stop_mock
+    [ "$status" -ne 0 ]
+    ! printf '%s\n' "$output" | grep -qE '^\{'
+}

tests/frank_large_unit.bats

@@ -1,338 +1,280 @@
 #!/usr/bin/env bats
-# Seam: the frank-large root unit on dankclaw.
+# Seam: frank-large serves kingjones777's Qwen3.8-27B Q8_0_ROCMFPX_AGENT (a
+# ftype-115 ROCmFPX file) with the repo's MTP draft head on the Strix Halo
+# iGPU, via the ROCmFPX fork of llama.cpp — one session at the native
+# 262,144 window.
 #
-# Since 2026-09-28 this endpoint is served by gufo (gufo-org/gufo), the
+# This seam exists because the format is the constraint: GGML_TYPE_Q8_0_ROCMFPX
-# Strix Halo inference engine built from the pinned v0.1.1 tag, serving
+# (tensor type 103) is understood by the ROCmFPX fork and by nothing else on
-# Swift-1.5-Qwen3.8-27B-Uncensored (abliterated) UD-Q8_K_XL with the z-lab
+# this host, so the runtime, the payload and the deployer's gate have to agree
-# DFlash2 Q4_K_M drafter. Two changes landed on this endpoint that day: the
+# on paths that no other endpoint uses. On 2026-10-01 a serve-script gate and
-# engine (from the stock llama.cpp prebuilt) and the payload (from
+# its deployer twin disagreed during a cutover and the endpoint crash-looped;
-# unsloth/Qwen3.8-27B-GGUF:UD-Q8_K_XL). Port, alias and client contract did
+# both sides are pinned here.
-# not move.
 #
-# Most of this seam runs the serve script against a stub engine and asserts
+# Everything is tested from what an operator or a supervisor can observe — the
-# on the argv it actually execs, so what is pinned is the command the
+# command line, the exit code, the deployer's error text — never from anything
-# operator gets rather than the text of the script: a tunable renamed,
+# inside llama.cpp.
-# dropped, or silently defaulted away fails here the way it would fail a
-# deployment.
-load helpers
-
-ENV_FILE="$REPO_ROOT/scripts/frank-large.env"
-UNIT="$REPO_ROOT/scripts/frank-large.service"
-SERVE="$REPO_ROOT/scripts/serve-frank-large.sh"
-FETCH="$REPO_ROOT/scripts/download-frank-large-q4.sh"
-DRAFTER_FETCH="$REPO_ROOT/scripts/download-frank-large.sh"
-BUILD="$REPO_ROOT/scripts/build-gufo.sh"
-REMOTE_DEPLOY="$REPO_ROOT/scripts/deploy-remote.sh"
-# The deployed target's filename, kept here so a rename in the env file and a
-# rename in this stub cannot drift apart unnoticed.
-TARGET_STUB="Qwen3.8-27B-UD-Q4_K_XL.gguf"
 setup() {
-    STAGE="$(mktemp -d)"
+    load helpers
-    RUNTIME_DIR="$STAGE/gufo"
+
-    MODEL_DIR="$STAGE/models"
+    SERVE="$REPO_ROOT/scripts/serve-frank-large.sh"
-    DRAFTER_DIR="$STAGE/draft"
+    UNIT="$REPO_ROOT/scripts/frank-large.service"
-    mkdir -p "$RUNTIME_DIR/bin" "$MODEL_DIR" "$DRAFTER_DIR"
+    ENV_FILE="$REPO_ROOT/scripts/frank-large.env"
-    # A stub engine: the serve script execs it, so $@ is exactly what gufo
+    DEPLOYER="$REPO_ROOT/scripts/deploy-remote.sh"
-    # would be invoked with.
+    BUILD="$REPO_ROOT/scripts/build-rocmfpx.sh"
-    cat > "$RUNTIME_DIR/bin/gufo" <<'STUB'
+    RUNTIME_STUB="/home/dank/GitPackages/kingjones-ROCmFPX/build/bin/llama-server"
-#!/usr/bin/env bash
+    MODEL_STUB="/home/dank/models/kingjones-Qwen3.8-27B-ROCmFP4-STRIX-MTP/Qwen3.8-27B-Q8_0_ROCMFPX_AGENT.gguf"
-printf '%s\n' "$@"
+    DRAFT_STUB="/home/dank/models/kingjones-Qwen3.8-27B-ROCmFP4-STRIX-MTP/mtp-Qwen3.8-27B-Q4_0.gguf"
-STUB
+    MMPROJ_STUB="/home/dank/models/kingjones-Qwen3.8-27B-ROCmFP4-STRIX-MTP/mmproj-Qwen3.8-27B-BF16.gguf"
-    chmod +x "$RUNTIME_DIR/bin/gufo"
+
-    # The target and the drafter live in SEPARATE directories since the
+    TEST_TMP="$(mktemp -d)"
-    # 2026-09-28 payload swap: the target came from a different repository,
+    SRC="$TEST_TMP/src"
-    # the DFlash2 drafter did not move. A stub that colocated them would
+    mkdir -p "$SRC"
-    # never notice DRAFTER_DIR going stale.
+    git -C "$SRC" init -q
-    : > "$MODEL_DIR/$TARGET_STUB"
+    git -C "$SRC" -c user.email=t -c user.name=t commit -q --allow-empty -m x
-    : > "$DRAFTER_DIR/Qwen3.8-27B-DFlash2-Q4_K_M.gguf"
-    STUB_ENV="$STAGE/stub.env"
-    cat > "$STUB_ENV" <<EOF
-RUNTIME_DIR=$RUNTIME_DIR
-MODEL_DIR=$MODEL_DIR
-DRAFTER_DIR=$DRAFTER_DIR
-EOF
 }
 
 teardown() {
-    if [ -n "${STAGE:-}" ]; then
+    rm -rf "$TEST_TMP"
-        rm -rf "$STAGE"
-    fi
 }
 
-# serve_argv [env-file] — run the serve script and capture the engine's argv,
+# make_stub — an executable that prints its whole argv on one line, so
-# one token per line, in $output. No argument exercises the built-in defaults
+# "what command line would the supervisor run" is observable output.
-# with the stub's paths.
+make_stub() {
-serve_argv() {
+    cat > "$TEST_TMP/bin" <<'STUB'
-    local envfile="${1:-$STUB_ENV}"
+#!/bin/sh
-    if [ -n "$envfile" ]; then
+printf '%s\n' "$*"
-        FRANK_LARGE_ENV_FILE="$envfile" run bash "$SERVE"
+STUB
-    else
+    chmod +x "$TEST_TMP/bin"
-        run env FRANK_LARGE_ENV_FILE="$STAGE/absent.env" bash "$SERVE"
-    fi
 }
 
-# argv_has <token> — exact token match, so --port never satisfies a test
+@test "the env file defines the deployment values" {
-# looking for a different flag.
+    grep -qE '^LLAMA_BIN=/' "$ENV_FILE"
-argv_has() {
+    grep -qE '^MODEL_FILE=/' "$ENV_FILE"
-    grep -qxF -- "$1" <<<"$output"
+    grep -qE '^DRAFT_FILE=/' "$ENV_FILE"
+    grep -qE '^ALIAS=[a-z0-9-]+$' "$ENV_FILE"
+    grep -qE '^LISTEN_PORT=[0-9]+$' "$ENV_FILE"
+    grep -qE '^CTX_SIZE=[0-9]+$' "$ENV_FILE"
 }
 
-# argv_value <flag> — the token following <flag>, empty if the flag is absent.
+@test "the serve script fails when the runtime is missing" {
-argv_value() {
+    # Only the model, the head and the projector exist; the fork was never
-    local i
+    # built. llama.cpp is not reached, so this fails fast and says what to
-    i="$(grep -nx -- "$1" <<<"$output" | cut -d: -f1)"
+    # run instead of failing at bind time.
-    [ -n "$i" ] || return 0
+    touch "$TEST_TMP/model.gguf" "$TEST_TMP/draft.gguf" "$TEST_TMP/mmproj.gguf"
-    sed -n "$((i + 1))p" <<<"$output"
+    run env "LLAMA_BIN=$TEST_TMP/nope/llama-server" \
-}
+        "MODEL_FILE=$TEST_TMP/model.gguf" "DRAFT_FILE=$TEST_TMP/draft.gguf" \
-
+        "MMPROJ_FILE=$TEST_TMP/mmproj.gguf" "$SERVE"
-@test "the frank-large unit, env, serve and build files exist where the deployer installs them from" {
-    [ -f "$ENV_FILE" ]
-    [ -f "$UNIT" ]
-    [ -x "$SERVE" ]
-    [ -x "$BUILD" ]
-}
-
-@test "frank-large is boot-persistent, self-healing and unprivileged" {
-    grep -q '^WantedBy=multi-user.target$' "$UNIT"
-    grep -q '^Restart=on-failure$' "$UNIT"
-    grep -q '^RestartSec=' "$UNIT"
-    grep -q '^User=dank$' "$UNIT"
-    # The engine pins buffers; the memlock budget must be unbounded or the
-    # load fails on the RLIMIT.
-    grep -q '^LimitMEMLOCK=infinity$' "$UNIT"
-    # The unit runs the repo's own serve script with the repo's env.
-    grep -q 'ExecStart=/home/dank/serverAppConfigs/scripts/serve-frank-large.sh' "$UNIT"
-    grep -q 'FRANK_LARGE_ENV_FILE=/home/dank/serverAppConfigs/scripts/frank-large.env' "$UNIT"
-}
-
-@test "the fleet endpoint and alias survive the cutover" {
-    serve_argv
-    [ "$status" -eq 0 ]
-    # frank-large at :7770, whatever engine answers it.
-    [ "$(argv_value --served-model-name)" = "frank-large" ]
-    [ "$(argv_value --port)" = "7770" ]
-    [ "$(argv_value --host)" = "0.0.0.0" ]
-}
-
-@test "the built-in defaults are one session of the full 262144 window with DFlash2" {
-    serve_argv
-    [ "$status" -eq 0 ]
-    # 262144 is the model's native window (the operator's call, 2026-09-21).
-    [ "$(argv_value --context)" -eq 262144 ]
-    # One session is the single stream every prior frank-large served.
-    [ "$(argv_value --sessions)" -eq 1 ]
-    [ "$(argv_value --speculative)" = "dflash2" ]
-    # A draft ceiling of 0 would silently serve the autoregressive rate.
-    [ "$(argv_value --draft-tokens)" -gt 0 ]
-}
-
-@test "the shipped env file, not just the defaults, serves the deployed shape" {
-    # The defaults are a floor; systemd sources the env file. Only the
-    # on-host paths are redirected at the stub, so every fleet value in the
-    # shipped file is the one under test. MODEL_FILE and DRAFTER_FILE are
-    # absolute in the shipped file, so they need redirecting too — a sed that
-    # only rewrote MODEL_DIR would leave them pointing at /home/dank and the
-    # test would pass for the wrong reason (or fail on a box that has them).
-    sed -e "s#^RUNTIME_DIR=.*#RUNTIME_DIR=$RUNTIME_DIR#" \
-        -e "s#^MODEL_DIR=.*#MODEL_DIR=$MODEL_DIR#" \
-        -e "s#^MODEL_FILE=.*#MODEL_FILE=$MODEL_DIR/$TARGET_STUB#" \
-        -e "s#^DRAFTER_DIR=.*#DRAFTER_DIR=$DRAFTER_DIR#" \
-        -e "s#^DRAFTER_FILE=.*#DRAFTER_FILE=$DRAFTER_DIR/Qwen3.8-27B-DFlash2-Q4_K_M.gguf#" \
-        "$ENV_FILE" > "$STAGE/shipped.env"
-    serve_argv "$STAGE/shipped.env"
-    [ "$status" -eq 0 ]
-    [ "$(argv_value --served-model-name)" = "frank-large" ]
-    [ "$(argv_value --host)" = "0.0.0.0" ]
-    [ "$(argv_value --port)" = "7770" ]
-    # 2026-10-01: four sessions x 163840 context (Frank's "4x 160k"), f16 KV.
-    # 163840 tokens x 64 KiB/token = 10 GiB of KV per session exactly.
-    [ "$(argv_value --context)" -eq 163840 ]
-    [ "$(argv_value --sessions)" -eq 4 ]
-    [ "$(argv_value --speculative)" = "dflash2" ]
-    [ "$(argv_value --draft-tokens)" -gt 0 ]
-    [ "$(argv_value --draft-policy)" = "adaptive" ]
-    # The shipped file must name the target the fleet actually serves, not a
-    # path that merely happens to exist.
-    [ "$(argv_value --model)" = "$MODEL_DIR/$TARGET_STUB" ]
-    [ "$(argv_value --dflash-model)" = "$DRAFTER_DIR/Qwen3.8-27B-DFlash2-Q4_K_M.gguf" ]
-}
-
-@test "the engine is pointed at the payload that is actually on disk" {
-    serve_argv
-    [ "$status" -eq 0 ]
-    [ -f "$(argv_value --model)" ]
-    # A drafter path that does not exist would load the target and decode at
-    # the autoregressive rate, which reads as "the new runtime is slow".
-    [ -f "$(argv_value --dflash-model)" ]
-    [ "$(argv_value --model)" != "$(argv_value --dflash-model)" ]
-}
-
-@test "frank-large stays text-only" {
-    # The payload has no mmproj, so an --mmproj would point at nothing. The
-    # fleet's image path is frank-small: its SGLang MiMo payload carries the
-    # vision tower inside the checkpoint (no projector file at all), and
-    # models.yml declares frank-large text-only.
-    serve_argv
-    ! argv_has --mmproj
-    grep -q 'MiMo-V2.6-Distill-Qwen-9B-INT8-W8A16-AutoRound' "$REPO_ROOT/scripts/frank-small.env"
-}
-
-@test "the engine is asked for greedy decoding and for preserved prior reasoning" {
-    # Greedy is what this endpoint has always served; a silent switch to
-    # sampled output would be a model-behaviour change disguised as a
-    # runtime swap. Prior thinking blocks must survive across turns or a
-    # multi-turn agent loses its own reasoning between tool calls.
-    serve_argv
-    [ "$(argv_value --temperature)" = "0" ]
-    [ "$(argv_value --preserve-thinking)" = "on" ]
-}
-
-@test "the serve script refuses a missing runtime with the build pointer" {
-    rm -f "$RUNTIME_DIR/bin/gufo"
-    serve_argv
     [ "$status" -ne 0 ]
-    [[ "$output" == *"runtime not found"* ]]
+    [[ "$output" == *"build it with scripts/build-rocmfpx.sh"* ]]
-    [[ "$output" == *"build-gufo.sh"* ]]
-    # A restart loop would be the alternative; nothing may be exec'd.
-    ! argv_has serve
 }
 
-@test "the serve script refuses a missing drafter with a fetch pointer" {
+@test "the serve script refuses to start without the model file" {
-    # The drafter lives in its own directory now, so the target can be present
+    make_stub
-    # and this still has to be caught.
+    touch "$TEST_TMP/draft.gguf" "$TEST_TMP/mmproj.gguf"
-    rm -f "$DRAFTER_DIR/Qwen3.8-27B-DFlash2-Q4_K_M.gguf"
+    run env "LLAMA_BIN=$TEST_TMP/bin" "MODEL_FILE=$TEST_TMP/absent.gguf" \
-    serve_argv
+        "DRAFT_FILE=$TEST_TMP/draft.gguf" "MMPROJ_FILE=$TEST_TMP/mmproj.gguf" "$SERVE"
     [ "$status" -ne 0 ]
-    [[ "$output" == *"missing asset"* ]]
+    [[ "$output" == *"is not readable"* ]]
-    [[ "$output" == *"Qwen3.8-27B-DFlash2-Q4_K_M.gguf"* ]]
+    [[ "$output" == *"download-frank-large-kingjones.sh"* ]]
-    [[ "$output" == *"download-frank-large"* ]]
-    ! argv_has serve
 }
 
-@test "the serve script refuses a missing target and names the file" {
+@test "the serve script refuses to start without the MTP draft head" {
-    rm -f "$MODEL_DIR/$TARGET_STUB"
+    make_stub
-    serve_argv
+    touch "$TEST_TMP/model.gguf" "$TEST_TMP/mmproj.gguf"
+    run env "LLAMA_BIN=$TEST_TMP/bin" "MODEL_FILE=$TEST_TMP/model.gguf" \
+        "DRAFT_FILE=$TEST_TMP/absent.gguf" "MMPROJ_FILE=$TEST_TMP/mmproj.gguf" "$SERVE"
     [ "$status" -ne 0 ]
-    [[ "$output" == *"missing asset"* ]]
+    [[ "$output" == *"MTP draft head"* ]]
-    [[ "$output" == *"$TARGET_STUB"* ]]
 }
 
-@test "both payload files are fetched resumably and verified against declared sizes + sha256" {
+@test "the serve script refuses a missing mmproj unless vision is opted out" {
-    # The target and the drafter come from DIFFERENT repositories, fetched by
+    make_stub
-    # two scripts. Anchor the repo checks to the URL ASSIGNMENT, not to the
+    touch "$TEST_TMP/model.gguf" "$TEST_TMP/draft.gguf"
-    # file: the same repo string also appears in each script's header comment,
+    run env "LLAMA_BIN=$TEST_TMP/bin" "MODEL_FILE=$TEST_TMP/model.gguf" \
-    # so a plain grep stayed green after the URL was repointed at another
+        "DRAFT_FILE=$TEST_TMP/draft.gguf" "MMPROJ_FILE=$TEST_TMP/absent.gguf" "$SERVE"
-    # repository -- the script would then download something the serve script
+    [ "$status" -ne 0 ]
-    # never asked for.
+    [[ "$output" == *"set MMPROJ_FILE='' for text-only"* ]]
-    grep -qE '^URL="https://huggingface\.co/unsloth/Qwen3\.8-27B-GGUF/resolve/main/' "$FETCH"
+    # The documented opt-out must actually start, without a --mmproj argument.
-    grep -qE 'https://huggingface\.co/z-lab/Qwen3\.8-27B-DFlash2-GGUF/resolve/main/' "$DRAFTER_FETCH"
+    run env "LLAMA_BIN=$TEST_TMP/bin" "MODEL_FILE=$TEST_TMP/model.gguf" \
-    for f in "$FETCH" "$DRAFTER_FETCH"; do
+        "DRAFT_FILE=$TEST_TMP/draft.gguf" "MMPROJ_FILE=" "$SERVE"
-        # -f so an HTTP error is a failure rather than a small "artifact";
+    [ "$status" -eq 0 ]
-        # -C - so an interrupted 29 GiB run resumes.
+    [[ "$output" != *"--mmproj"* ]]
-        grep -q -- '-fL -C -' "$f"
+}
-        # The size and the sha256 are compared after the transfer, not trusted.
+
-        grep -q 'SHA256 MISMATCH' "$f"
+@test "the serve script fails when LLAMA_BIN is empty" {
-        grep -q 'SIZE MISMATCH' "$f"
+    touch "$TEST_TMP/model.gguf" "$TEST_TMP/draft.gguf" "$TEST_TMP/mmproj.gguf"
+    run env LLAMA_BIN= "MODEL_FILE=$TEST_TMP/model.gguf" \
+        "DRAFT_FILE=$TEST_TMP/draft.gguf" "MMPROJ_FILE=$TEST_TMP/mmproj.gguf" "$SERVE"
+    [ "$status" -ne 0 ]
+    [[ "$output" == *"LLAMA_BIN is empty"* ]]
+}
+
+@test "the shipped env points at the ROCmFPX runtime and the AGENT payload" {
+    grep -qE "^LLAMA_BIN=$RUNTIME_STUB$" "$ENV_FILE"
+    grep -qE "^MODEL_FILE=$MODEL_STUB$" "$ENV_FILE"
+    grep -qE "^DRAFT_FILE=$DRAFT_STUB$" "$ENV_FILE"
+    grep -qE "^MMPROJ_FILE=$MMPROJ_STUB$" "$ENV_FILE"
+    # HIP reads bare HOST/TEMP/GPU_MAX_HEAP_SIZE out of the environment that
+    # `set -a` creates; a tunable named HOST leaks into the runtime.
+    ! grep -qE '^(HOST|TEMP|GPU_MAX_HEAP_SIZE)=' "$ENV_FILE"
+}
+
+@test "the argv contract keeps the fleet's port and alias" {
+    make_stub
+    touch "$TEST_TMP/model.gguf" "$TEST_TMP/draft.gguf" "$TEST_TMP/mmproj.gguf"
+    run env "LLAMA_BIN=$TEST_TMP/bin" "MODEL_FILE=$TEST_TMP/model.gguf" \
+        "DRAFT_FILE=$TEST_TMP/draft.gguf" "MMPROJ_FILE=$TEST_TMP/mmproj.gguf" \
+        ALIAS=frank-large BIND_ADDR=127.0.0.1 LISTEN_PORT=7770 "$SERVE"
+    [ "$status" -eq 0 ]
+    # The caller's values must survive the env file, or a single-run operator
+    # override silently does nothing.
+    [[ "$output" == *"--host 127.0.0.1"* ]]
+    [[ "$output" == *"--port 7770"* ]]
+    [[ "$output" == *"--alias frank-large"* ]]
+    # Vision is wired: the projector is the whole vision tower.
+    [[ "$output" == *"--mmproj $TEST_TMP/mmproj.gguf"* ]]
+}
+
+@test "the shipped argv is one full native window on one session" {
+    grep -qE '^CTX_SIZE=262144$' "$ENV_FILE"
+    grep -qE '^PARALLEL=1$' "$ENV_FILE"
+    # 262144 IS the model's trained window (general.context_length in the
+    # GGUF), so 1 slot gets the whole thing — no truncation, no yarn scaling.
+    #
+    # Budget, from the GGUF header: 26.28 GiB of weights + 1.57 GiB draft head
+    # + 0.87 GiB projector + 16 full-attention layers x 4 kv heads x 256 dim x
+    # 2 (K+V) x 2 B = 64 KiB/token of f16 KV (~16 GiB at 262,144) + ~0.14 GiB
+    # of DeltaNet state + a couple GiB of compute buffers, against the iGPU's
+    # ~114 GiB GTT aperture. Change CTX_SIZE and this arithmetic, not a boot
+    # log, is what tells you it no longer fits.
+    ctx="$(sed -nE 's/^CTX_SIZE=([0-9]+)$/\1/p' "$ENV_FILE")"
+    kv_gib=$((ctx * 65536 / 1073741824))
+    total_gib=$((27 + 2 + 1 + kv_gib + 3))
+    [ "$total_gib" -le 114 ]
+}
+
+@test "the MTP draft contract matches the model card's measured optimum" {
+    make_stub
+    touch "$TEST_TMP/model.gguf" "$TEST_TMP/draft.gguf" "$TEST_TMP/mmproj.gguf"
+    run env "LLAMA_BIN=$TEST_TMP/bin" "MODEL_FILE=$TEST_TMP/model.gguf" \
+        "DRAFT_FILE=$TEST_TMP/draft.gguf" "MMPROJ_FILE=$TEST_TMP/mmproj.gguf" "$SERVE"
+    [ "$status" -eq 0 ]
+    # The head is the model's own nextn layer, loaded as a separate draft
+    # model because this checkpoint ships it standalone (no blk.<n>.nextn.*
+    # tensors in the target file).
+    [[ "$output" == *"--spec-type draft-mtp"* ]]
+    [[ "$output" == *"--model-draft $TEST_TMP/draft.gguf"* ]]
+    # n-max 4 is the measured optimum; n-max 8 measured WORSE (acceptance
+    # fell to 0.757 and the extra verification cost outran the savings), so
+    # the flag is a measurement, not a wish.
+    [[ "$output" == *"--spec-draft-n-max 4"* ]]
+    [[ "$output" == *"--spec-draft-n-min 0"* ]]
+    [[ "$output" == *"--spec-draft-p-min 0.0"* ]]
+    # The draft runs on the iGPU beside the target: no device-to-device hop
+    # on a link that measures single-digit GB/s.
+    [[ "$output" == *"--spec-draft-device ROCm0"* ]]
+    [[ "$output" == *"--spec-draft-ngl 99"* ]]
+}
+
+@test "KV and placement stay unquantized with flash attention on" {
+    make_stub
+    touch "$TEST_TMP/model.gguf" "$TEST_TMP/draft.gguf" "$TEST_TMP/mmproj.gguf"
+    run env "LLAMA_BIN=$TEST_TMP/bin" "MODEL_FILE=$TEST_TMP/model.gguf" \
+        "DRAFT_FILE=$TEST_TMP/draft.gguf" "MMPROJ_FILE=$TEST_TMP/mmproj.gguf" "$SERVE"
+    [ "$status" -eq 0 ]
+    [[ "$output" == *"--cache-type-k f16"* ]]
+    [[ "$output" == *"--cache-type-v f16"* ]]
+    # ROCmFPx quantized KV has no kernel on this branch, and the endpoint has
+    # the aperture to keep KV unquantized anyway.
+    [[ "$output" == *"--flash-attn on"* ]]
+    # --fit's placement heuristics do not know that 311 tensors of type 103
+    # belong on the iGPU, so the whole stack is placed explicitly instead.
+    [[ "$output" == *"--fit off"* ]]
+    [[ "$output" == *"--n-gpu-layers 999"* ]]
+}
+
+@test "the client contract wires thinking and template kwargs" {
+    make_stub
+    touch "$TEST_TMP/model.gguf" "$TEST_TMP/draft.gguf" "$TEST_TMP/mmproj.gguf"
+    run env "LLAMA_BIN=$TEST_TMP/bin" "MODEL_FILE=$TEST_TMP/model.gguf" \
+        "DRAFT_FILE=$TEST_TMP/draft.gguf" "MMPROJ_FILE=$TEST_TMP/mmproj.gguf" "$SERVE"
+    [ "$status" -eq 0 ]
+    # Without --jinja, llama.cpp silently ignores the harness's
+    # chat_template_kwargs, so thinking control disappears from the API.
+    [[ "$output" == *"--jinja"* ]]
+    # The harness reads `reasoning_content` for this endpoint, which is what
+    # the deepseek emission format produces (vLLM's `reasoning` is not it).
+    [[ "$output" == *"--reasoning-format deepseek"* ]]
+    # The template needs an unbounded thinking budget.
+    [[ "$output" == *"--reasoning-budget -1"* ]]
+}
+
+@test "the deployer gates exactly the paths the serve script opens" {
+    # A gate that checks a different file than the engine opens is the
+    # 2026-10-01 crash-loop: deploy passed, the unit restart-looped. Read the
+    # engine's own paths out of the env file and require the deployer's
+    # defaults to name the same ones.
+    for pair in "LLAMA_BIN:LLM_LARGE_RUNTIME" \
+                "MODEL_FILE:LLM_LARGE_MODEL" \
+                "DRAFT_FILE:LLM_LARGE_DRAFT" \
+                "MMPROJ_FILE:LLM_LARGE_MMPROJ"; do
+        env_key="${pair%%:*}"
+        dep_key="${pair##*:}"
+        want="$(sed -nE "s|^${env_key}=(.*)\$|\1|p" "$ENV_FILE")"
+        [ -n "$want" ]
+        got="$(grep -oE "${dep_key}:-[^}]+" "$DEPLOYER" | head -1 | sed "s|^${dep_key}:-||")"
+        [ "$got" = "$want" ]
     done
 }
 
-@test "the serve script and the fetch scripts agree on the payload filenames" {
+@test "the deployer points a missing runtime at the fork build script" {
-    # The serve script's MODEL_FILE and the fetch script's REL are two
+    grep -qE 'build it: bash scripts/build-rocmfpx\.sh' "$DEPLOYER"
-    # independent spellings of one filename, and a repo is not required to
+    grep -qE 'fetch it: bash scripts/download-frank-large-kingjones\.sh' "$DEPLOYER"
-    # keep them in step: a re-upload under a new name, or an env edited
-    # without the fetch, leaves the endpoint unable to start with a
-    # "missing asset" that no test would have predicted. Bind them here.
-    local served fetched drafter_served drafter_fetched
-    served="$(basename "$(grep '^MODEL_FILE=' "$ENV_FILE" | cut -d= -f2-)")"
-    fetched="$(sed -n 's/^REL="\(.*\)"$/\1/p' "$FETCH")"
-    [ -n "$served" ]
-    [ "$served" = "$fetched" ]
-    drafter_served="$(basename "$(grep '^DRAFTER_FILE=' "$ENV_FILE" | cut -d= -f2-)")"
-    drafter_fetched="$(sed -n 's/^ *"\(Qwen3\.8-27B-DFlash2[^ ]*\.gguf\) .*/\1/p' "$DRAFTER_FETCH")"
-    [ -n "$drafter_served" ]
-    [ "$drafter_served" = "$drafter_fetched" ]
 }
 
-@test "the drafter is still fetched by the script that carries its Hub repo" {
+@test "the stray-process sweep cannot take frank-small down with frank-large" {
-    # Guards a future "simplification" that folds the drafter into the
+    # frank-small serves a different GGUF over llama-server on the SAME host.
-    # payload-fetch script and silently orphans the artifact the serve script
+    # A sweep keyed on the binary name (`pkill -f llama-server`) would kill
-    # still points at via DRAFTER_DIR. The literal drafter name must not
+    # the 3090 endpoint too, so it is keyed on frank-large's own payload file.
-    # appear in that script AT ALL, comments included.
+    grep -q "pkill -f 'Qwen3.8-27B-Q8_0_ROCMFPX_AGENT'" "$DEPLOYER"
-    grep -q 'Qwen3.8-27B-UD-Q8_K_XL' "$DRAFTER_FETCH"
+    ! grep -qE "pkill -f 'llama-server'" "$DEPLOYER"
-    ! grep -q 'DFlash2' "$FETCH"
 }
 
 @test "the build script refuses a source tree that is not the pinned commit" {
-    # gufo ships no prebuilt Linux binaries, so the runtime is built here.
+    # The pin is load-bearing: the older builds of this fork line segfault in
-    # A tag that moves under us would silently install different kernels
+    # ggml_compute_forward_set_rows on this exact payload. A runtime that
-    # under the same RUNTIME_DIR, so the pin is enforced, not documented.
+    # differs from what was qualified must be heard about at deploy time.
-    local src="$STAGE/not-the-pinned-commit"
+    wrong="$(git -C "$SRC" rev-parse HEAD)"
-    mkdir -p "$src"
+    run env ROCMFPX_SRC_DIR="$SRC" ROCMFPX_COMMIT="ffff$wrong" \
-    git -C "$src" init -q
+        "$BUILD" --verify
-    git -C "$src" -c user.email=t@t -c user.name=t commit -q --allow-empty -m drift
-    run env GUFO_SRC_DIR="$src" GUFO_PREFIX="$STAGE/absent-prefix" bash "$BUILD" --verify
     [ "$status" -ne 0 ]
-    [[ "$output" == *"GUFO_COMMIT mismatch"* ]]
+    [[ "$output" == *"ROCMFPX_COMMIT"*mismatch* ]]
 }
 
-@test "the build script reports a runtime that was never built" {
+@test "the build script pins a real commit and the gfx1151 target" {
-    # --verify on a host with no install must say so rather than exit 0 on
+    grep -qE 'ROCMFPX_COMMIT=.*[0-9a-f]{40}' "$BUILD"
-    # an empty check.
+    grep -qE 'GPU_TARGET=.*gfx1151' "$BUILD"
-    run env GUFO_SRC_DIR="$STAGE/no-source-tree" GUFO_PREFIX="$STAGE/absent-prefix" bash "$BUILD" --verify
+    grep -qE 'DGPU_TARGETS=' "$BUILD"
-    [ "$status" -ne 0 ]
+    # The webui asset fetch half-failed on this box and aborted the link at
-    [[ "$output" == *"no gufo at"* ]]
+    # 98%; the build must not depend on it.
+    grep -qE 'DLLAMA_BUILD_WEBUI=OFF' "$BUILD"
 }
 
-@test "the deploy handler gates on the runtime, the payload and the port" {
+@test "the deployed unit keeps the fleet's restart and boot-persistence shape" {
-    grep -q 'relink_unit frank-large' "$REMOTE_DEPLOY"
+    grep -qE '^Restart=(always|on-failure)$' "$UNIT"
-    grep -q 'systemctl enable frank-large.service' "$REMOTE_DEPLOY"
+    grep -qE '^WantedBy=' "$UNIT"
-    grep -q 'wait_port_free 7770' "$REMOTE_DEPLOY"
+    grep -qE '^ExecStart=.*/serve-frank-large\.sh$' "$UNIT"
-    grep -q '127.0.0.1:7770/health' "$REMOTE_DEPLOY"
+    grep -qE '^User=dank$' "$UNIT"
-    grep -q 'LLM_LARGE_RUNTIME' "$REMOTE_DEPLOY"
+    # The payload is 26 GiB of mmap'd weights plus pinned staging buffers; an
-    grep -q 'LLM_LARGE_MODEL' "$REMOTE_DEPLOY"
+    # RLIMIT smaller than the pinned set fails the load in a way that looks
-    grep -q 'LLM_LARGE_DRAFT' "$REMOTE_DEPLOY"
+    # like an engine crash.
+    grep -qE '^LimitMEMLOCK=infinity$' "$UNIT"
 }
 
-@test "the deploy handler will not stop a serving endpoint it cannot replace" {
+@test "the unit points at no retired frank-large runtime path" {
-    # The runtime gate must be live (a real conditional on an executable
+    # The host keeps every retired payload and build on disk, so a stale path
-    # path, not a branch that never fires) and it must return BEFORE the
+    # in a unit still resolves — to a runtime nobody has measured in weeks.
-    # stop, or a host whose gufo was never built takes frank-large down on
+    ! grep -qE 'GitPackages/(ROCmFPX-kairic|llama.cpp-ik_llama|llama-builds|ciru-orca)' "$UNIT"
-    # its way to reporting the failure.
+    ! grep -q 'halogen' "$UNIT"
-    local large gate bail stop
-    large="$(sed -n '/^deploy_llm_large() {/,/^}/p' "$REMOTE_DEPLOY")"
-    [[ "$large" == *'build-gufo.sh'* ]]
-    gate="$(grep -n 'if \[\[ ! -x "\$LLM_LARGE_RUNTIME" \]\]' <<<"$large" | head -1 | cut -d: -f1)"
-    stop="$(grep -n 'systemctl stop frank-large' <<<"$large" | head -1 | cut -d: -f1)"
-    [ -n "$gate" ]
-    [ -n "$stop" ]
-    bail="$(awk -v g="$gate" 'NR > g && /return 1/ { print NR; exit }' <<<"$large")"
-    [ -n "$bail" ]
-    [ "$bail" -lt "$stop" ]
-}
-
-@test "the stray-run sweep keys on the drafter and cannot take frank-small down" {
-    # The sweep used to key on the TARGET filename, which the 2026-09-28
-    # payload swap moved to a different repository. It would have stopped
-    # matching the live process, and a stray run would then hold 7770 past the
-    # port gate and fail the deploy. The drafter is the part that did NOT
-    # move, and every frank-large command line carries it via --dflash-model.
-    grep -q "pkill -f 'Qwen3.8-27B-DFlash2-Q4_K_M'" "$REMOTE_DEPLOY"
-    # frank-small serves a different GGUF on the 3090 with no DFlash2 drafter,
-    # so a binary-name sweep is the only thing that would endanger it.
-    local large
-    large="$(sed -n '/^deploy_llm_large() {/,/^}/p' "$REMOTE_DEPLOY")"
-    ! grep -q 'pkill -x llama-server' <<<"$large"
-    ! grep -q 'pkill -x gufo' <<<"$large"
-}
-
-@test "the deploy handler retires the halogen quadlet without deleting it" {
-    grep -q 'retire_halogen_large_quadlet' "$REMOTE_DEPLOY"
-    grep -q 'HOME/.AGENTTRASH' "$REMOTE_DEPLOY"
-    ! grep -qE '(^|[^[:alnum:]_])rm (-[a-zA-Z]+ )?.*frank-large' "$REMOTE_DEPLOY"
-    # Nothing live ships for this endpoint as a container any more. The
-    # archived copy is host state in an ignored .AGENTTRASH — deliberately
-    # untracked, and present only in a primary checkout — so search the
-    # tracked tree, not the filesystem.
-    [ ! -e "$REPO_ROOT/dankclaw/frank-large/quadlet/frank-large.container" ]
-    ! git -C "$REPO_ROOT" grep -q 'Image=.*halogen' -- dankclaw
 }

tests/frank_primary_unit.bats

@@ -2,10 +2,11 @@
 # Seam: frank-primary — the Dankserver LLM endpoint on :7770.
 #
 # This file asserts the tcclaviger RDNA4 vLLM ROOT-UNIT shape (the
-# 2026-10-01 BF16 cutover, the overnight redo campaign's verdict): the serve
+# 2026-10-02 correction of the 2026-10-01 BF16 cutover): the serve
 # script drives `podman run` DIRECTLY (the image entrypoint execs
 # `vllm serve`), with every engine flag qualified by boot probe and carried
-# in the repo env file. The seam pins what the flags alone do not say: the
+# in the repo env file.
+# The seam pins what the flags alone do not say: the
 # KV qualification law (the night ladder: 262144/245760 OOM at graph
 # capture; 241664 boots at 732,359 tokens / 3.03x, util 0.98;
 # the bf16-KV trade (double the attention-KV bytes/token, ~14% decode cost,
@@ -91,29 +92,31 @@ DOWNLOAD="$REPO_ROOT/scripts/download-frank-primary.sh"
 
 @test "the qualified engine shape is in the env file" {
     # 2026-10-01 night ladder (Frank's method), then a runtime-survivability
-    # correction. SEQS=3 bf16 topped out at 241664 (KV 732,359 / 3.03x);
+    # correction, then Frank's 2026-10-02 decision to give the pool back to
+    # the expert arena. SEQS=3 bf16 topped out at 241664 (KV 732,359 / 3.03x);
     # SEQS=6 fp8 does NOT fit (6 x 241664 asked 10.70 GiB against 9.18 GiB
     # free); SEQS=5 fp8 x 262144 BOOTED (KV 1,350,182 / 5.15x) but left ~0
     # bytes free at runtime and a real client request (995-token prompt with
     # max_tokens=261149) killed the worker with `Tried to allocate 50.00 MiB
-    # ... 0 bytes free`. FOUR slots at the full native 262144 window is the
+    # ... 0 bytes free`. 200704 sits below the 241664 rung while leaving a
-    # deployed shape: ~1.93 GiB more headroom per rank for
+    # full 2.95 GiB/rank to cut loose for the arena. UTIL stays 0.98
-    # activations/graphs. UTIL stays 0.98 (lowering it shrinks the engine's
+    # (lowering it shrinks the engine's own budget instead of buying slack).
-    # own budget instead of buying slack).
+    # FINAL (2026-10-02, Frank): TWO windows at 200704 with FP8 KV and a pool
-    # FINAL (2026-10-01, Frank): TWO full-size windows with BF16 KV plus
+    # cut for the arena. fp8 costs 7,701 B/token (measured 6,138 attention +
-    # --expert-precise-memory. bf16 costs 13,839 B/token (measured), so
+    # 1,563 mamba/side), so 2 x 200704 = 401,408 tokens needs 3.091e9 B;
-    # 2 x 262144 = 524,288 tokens needs 7.256e9 B; pinned at 7.3e9 (~527.5k
+    # pinned at 3.15e9 (~405k tokens, ~44 MB margin). The arena is what
-    # tokens, 3,206 margin). The precise flag drops the plan's overhead
+    # remains — the planner sizes it AFTER the KV override — so the 6,248
-    # 7.60 -> 5.25 GiB, moving the arena from 6,248 to ~7,278 slots.
+    # slots of the 6.1e9 shape become roughly 7,500 here.
-    grep -q '^FRANK_PRIMARY_MAXLEN=262144$' "$ENV_FILE"
+    grep -q '^FRANK_PRIMARY_MAXLEN=200704$' "$ENV_FILE"
-    grep -q '^FRANK_PRIMARY_SEQS=3$' "$ENV_FILE"
+    grep -q '^FRANK_PRIMARY_SEQS=2$' "$ENV_FILE"
     grep -q '^FRANK_PRIMARY_UTIL=0.98$' "$ENV_FILE"
-    grep -q '^FRANK_PRIMARY_KV_BYTES=6100000000$' "$ENV_FILE"
+    grep -q '^FRANK_PRIMARY_KV_BYTES=3150000000$' "$ENV_FILE"
-    grep -q '^FRANK_PRIMARY_MNBT=2048$' "$ENV_FILE"
+    grep -q '^FRANK_PRIMARY_MNBT=1024$' "$ENV_FILE"
     grep -q '^FRANK_PRIMARY_KV_DTYPE=fp8$' "$ENV_FILE"
-    # --expert-precise-memory is NOT set: with bf16 2 x 262144 it put the boot
+    # --expert-precise-memory is NOT set: with bf16 2 x 262144 it put the
-    # 0.13 GiB from the 31.86 GiB ceiling (arena 16.65 device-resident) and
+    # boot 0.13 GiB from the 31.86 GiB ceiling (arena 16.65 device-resident)
-    # crash-looped at graph capture. The serve script keeps the opt-in flag.
+    # and crash-looped at graph capture. The serve script keeps the opt-in
+    # flag; the smaller pool here is the arena it would have been paying for.
     ! grep -q '^FRANK_PRIMARY_PRECISE_MEMORY=' "$ENV_FILE"
     grep -q '^FRANK_PRIMARY_GATHER_CHUNKS=32$' "$ENV_FILE"
     grep -q '^FRANK_PRIMARY_GATHER_LANES=64$' "$ENV_FILE"
@@ -121,6 +124,22 @@ DOWNLOAD="$REPO_ROOT/scripts/download-frank-primary.sh"
     grep -q '^FRANK_PRIMARY_MODEL_SHARDS=131$' "$ENV_FILE"
 }
 
+@test "the pool is sized as the arena floor, not the context floor" {
+    # THE LAW this seam pins: the fork's expert planner sizes the arena
+    # AFTER the KV override, so every byte cut out of KV_BYTES becomes
+    # arena. A context-size change must move the pool with it, or the
+    # reduction buys nothing. 2 x 200704 = 401,408 tokens at fp8's 7,701
+    # B/token is 3.091e9 B; the pin is 3.15e9, ~44 MB of margin above the
+    # floor — the same convention the 6.1e9 shape used.
+    floor=$(awk -v n="401408" -v b="7701" 'BEGIN{printf "%.0f", n*b}')
+    [ "$floor" = "3091243008" ]
+    kv=$(grep '^FRANK_PRIMARY_KV_BYTES=' "$ENV_FILE" | cut -d= -f2)
+    [ "$kv" = "3150000000" ]
+    margin=$(( kv - floor ))
+    [ "$margin" -gt 40000000 ]
+    [ "$margin" -lt 100000000 ]
+}
+
 @test "the serve script wires the engine flags from the env" {
     grep -q -- '--max-num-seqs "$FRANK_PRIMARY_SEQS"' "$SERVE"
     grep -q -- '--max-model-len "$FRANK_PRIMARY_MAXLEN"' "$SERVE"
@@ -208,13 +227,14 @@ DOWNLOAD="$REPO_ROOT/scripts/download-frank-primary.sh"
     grep -q -- '-e R4D_LRU_GATHER_CHUNKS="$FRANK_PRIMARY_GATHER_CHUNKS"' "$SERVE"
 }
 
-@test "MNBT is at Frank's 2048 with the arena intact" {
+@test "MNBT is at Frank's 1024 with the arena intact" {
     # MNBT 512 doubles TTFT at 4064 ctx (2.46 s vs 1.36 s); 1024-4096 are
     # interchangeable on decode AND identical on arena (the sweep measured
     # 6,499 slots at every rung <=4096); 12288 costs 2.5% plus arena.
-    # 2048 is Frank's setting as of 2026-10-01 — it keeps the workspace floor
+    # 1024 is the 2026-10-02 setting — the pool was cut to feed the expert
-    # so the expert arena stays max.
+    # arena, so the workspace floor matters more than it did at 2048: the
-    grep -q '^FRANK_PRIMARY_MNBT=2048$' "$ENV_FILE"
+    # arena budget is what is left, and 1024-4096 pay nothing for it.
+    grep -q '^FRANK_PRIMARY_MNBT=1024$' "$ENV_FILE"
     grep -q -- '--max-num-batched-tokens "$FRANK_PRIMARY_MNBT"' "$SERVE"
 }
 

tests/frank_small_unit.bats

@@ -1,30 +1,72 @@
 #!/usr/bin/env bats
-# Seam: the frank-small SGLang root unit on dankclaw.
+# Seam: the frank-small llama.cpp root unit on dankclaw.
-# frank-small is the RTX 3090 endpoint; since 2026-10-01 it serves
+# frank-small is the RTX 3090 endpoint; since 2026-10-01 night it serves
-# letechlead/MiMo-V2.6-Distill-Qwen-9B-INT8-W8A16-AutoRound (qwen3_5 hybrid,
+# bartowski/MiMo-V2.6-Distill-Qwen-9B-GGUF:Q5_K_M via a CUDA (sm_86) build of
-# INT8 W8A16 AutoRound, in-checkpoint vision) via SGLang in a
+# llama.cpp master, chosen for the ASYMMETRIC KV Frank asked for: K f16
-# python3.12 venv. This seam pins what the flags alone do not say: boot
+# (unquantized) + V q8_0, which only llama.cpp exposes per-tensor. Since
-# persistence for a NATIVE runtime (no generated quadlet is involved), the
+# 2026-10-02 it runs THREE contexts and carries a vision projector, and the
-# repo-relative paths the unit executes, the separate client port, the
+# card must keep 2-3 GiB free. This seam runs the serve script against a stub
-# no-spec-decode gate (the checkpoint ships NO mtp.* tensors, so a NEXTN
+# engine and asserts on the argv it actually execs, so what is pinned is the
-# draft would be an uninitialised ghost model), the 8-shard payload gate,
+# command the operator gets.
-# and the KV budget: fp8 KV at 0.98 utilization whose 4-slot grid must fit
-# the card's 24 GiB.
 
 load helpers
 
 UNIT="$REPO_ROOT/scripts/frank-small.service"
 SERVE="$REPO_ROOT/scripts/serve-frank-small.sh"
 ENV_FILE="$REPO_ROOT/scripts/frank-small.env"
-DOWNLOAD="$REPO_ROOT/scripts/download-frank-small.sh"
 REMOTE_DEPLOY="$REPO_ROOT/scripts/deploy-remote.sh"
 ENV_LARGE="$REPO_ROOT/scripts/frank-large.env"
+TARGET_STUB="MiMo-V2.6-Distill-Qwen-9B-Q5_K_M.gguf"
+# Same basename as the shipped projector: the test env repoints it into $STAGE.
+MM_STUB="mmproj-MiMo-V2.6-Distill-Qwen-9B-f16.gguf"
 
-@test "frank-small unit, serve script, env and download script exist in scripts/" {
+setup() {
+    STAGE="$(mktemp -d)"
+    RUNTIME_DIR="$STAGE/llama"
+    MODEL_DIR="$STAGE/models"
+    mkdir -p "$RUNTIME_DIR/bin" "$MODEL_DIR"
+    cat > "$RUNTIME_DIR/bin/llama-server" <<'STUB'
+#!/usr/bin/env bash
+printf '%s\n' "$@"
+STUB
+    chmod +x "$RUNTIME_DIR/bin/llama-server" "$RUNTIME_DIR/bin" 2>/dev/null || chmod +x "$RUNTIME_DIR/bin/llama-server"
+    touch "$MODEL_DIR/$TARGET_STUB" "$MODEL_DIR/$MM_STUB"
+}
+
+teardown() {
+    rm -rf "$STAGE"
+}
+
+# serve_argv [env-file] — run the serve script and capture the engine's
+# argv, one token per line, in $output.
+serve_argv() {
+    local envf="${1:-$STAGE/test.env}"
+    local mmproj
+    mmproj="$(sed -nE 's/^MMPROJ_FILE=(.+)$/\1/p' "$ENV_FILE")"
+    cat > "$envf" <<EOF
+LLAMA_BIN=$RUNTIME_DIR/bin/llama-server
+MODEL_FILE=$MODEL_DIR/$TARGET_STUB
+MMPROJ_FILE=$MODEL_DIR/${mmproj##*/}
+ALIAS=frank-small
+BIND_ADDR=0.0.0.0
+LISTEN_PORT=7771
+CTX_SIZE=$(sed -nE 's/^CTX_SIZE=([0-9]+)$/\1/p' "$ENV_FILE")
+PARALLEL=$(sed -nE 's/^PARALLEL=([0-9]+)$/\1/p' "$ENV_FILE")
+CACHE_TYPE_K=$(sed -nE 's/^CACHE_TYPE_K=(.+)$/\1/p' "$ENV_FILE")
+CACHE_TYPE_V=$(sed -nE 's/^CACHE_TYPE_V=(.+)$/\1/p' "$ENV_FILE")
+NGL=$(sed -nE 's/^NGL=([0-9]+)$/\1/p' "$ENV_FILE")
+EOF
+    run env FRANK_SMALL_ENV_FILE="$envf" nvidia-modprobe=true bash "$SERVE"
+}
+
+argv_value() {
+    grep -A1 -xF -- "$1" <<<"$output" | tail -1
+}
+
+@test "frank-small unit, serve script and env exist in scripts/" {
     [ -f "$UNIT" ]
     [ -f "$SERVE" ]
     [ -f "$ENV_FILE" ]
-    [ -f "$DOWNLOAD" ]
 }
 
 @test "frank-small is boot-persistent, self-healing and unprivileged" {
@@ -33,102 +75,140 @@ ENV_LARGE="$REPO_ROOT/scripts/frank-large.env"
     grep -q '^RestartSec=' "$UNIT"
     grep -q '^User=dank$' "$UNIT"
     grep -q '^LimitMEMLOCK=infinity$' "$UNIT"
-}
-
-@test "frank-small runs the repo's own serve script with the repo's env" {
     grep -q '^ExecStart=/home/dank/serverAppConfigs/scripts/serve-frank-small.sh$' "$UNIT"
     grep -q 'FRANK_SMALL_ENV_FILE=/home/dank/serverAppConfigs/scripts/frank-small.env' "$UNIT"
 }
 
-@test "frank-small serves the MiMo-V2.6 checkpoint from the self-contained SGLang venv" {
+@test "the shipped env points at the llama.cpp runtime and the Q5_K_M GGUF" {
-    grep -q '^SGLANG_BIN=/home/dank/GitPackages/sglang-cu128/bin/sglang$' "$ENV_FILE"
+    grep -qE '^LLAMA_BIN=/home/dank/GitPackages/llama-master/build-cuda/bin/llama-server$' "$ENV_FILE"
-    grep -q '^MODEL_DIR=/home/dank/models/MiMo-V2.6-Distill-Qwen-9B-INT8-W8A16-AutoRound$' "$ENV_FILE"
+    grep -qE "^MODEL_FILE=/home/dank/models/MiMo-Q5KM/$TARGET_STUB$" "$ENV_FILE"
-    grep -q -- '--model-path "$MODEL_DIR"' "$SERVE"
-    # The AutoRound checkpoint declares its quant method; SGLang must load it
-    # as auto-round, not re-quantize online.
-    grep -q '^QUANTIZATION=auto-round$' "$ENV_FILE"
-    grep -q -- '--quantization "$QUANTIZATION"' "$SERVE"
 }
 
-@test "frank-small runs WITHOUT speculative decoding (no MTP weights shipped)" {
+@test "the deployer gates on exactly the payload paths the env file ships" {
-    # The config declares mtp_num_hidden_layers=1 but the safetensors index
+    # Anti-drift: the deploy handler checks files by its own defaults. If
-    # carries zero mtp.* tensors (base and INT8 conversion alike, verified
+    # those ever differ from what the unit actually loads, a green deploy
-    # 2026-10-01). SGLang would draft with an uninitialised 4.24 GiB model
+    # would be gating on a file nobody reads — and a missing projector would
-    # at 0% acceptance; the serve script must not pass any speculative flag.
+    # ship a blind endpoint.
-    ! grep -q -- '--speculative-' "$SERVE"
+    local env_v dep_v path pair
-    ! grep -q '^SPEC_' "$ENV_FILE"
+    for pair in "MODEL_FILE LLM_SMALL_MODEL_FILE" "MMPROJ_FILE LLM_SMALL_MMPROJ"; do
+        read -r env_v dep_v <<<"$pair"
+        path="$(sed -nE "s/^${env_v}=(.+)\$/\1/p" "$ENV_FILE" | tail -1)"
+        [ -n "$path" ]
+        grep -qF -- "${dep_v}=\"\${${dep_v}:-${path}}\"" "$REMOTE_DEPLOY"
+    done
 }
 
-@test "frank-small carries the fleet's image path, and only it" {
+@test "the shipped env serves the fleet shape: three 180224-token contexts" {
-    # The vision tower is baked into the safetensors (qwen3_5_vision): no
+    serve_argv
-    # separate projector file, and the serve script must not look for one.
+    [ "$status" -eq 0 ]
-    ! grep -q '^MMPROJ_PATH=' "$ENV_FILE"
+    [ "$(argv_value --alias)" = "frank-small" ]
-    ! grep -q -- '--mmproj' "$SERVE"
+    [ "$(argv_value --port)" = "7771" ]
-    ! grep -q '^ENABLE_VISION=1$' "$ENV_LARGE"
+    [ "$(argv_value --host)" = "0.0.0.0" ]
+    [ "$(argv_value --model)" = "$MODEL_DIR/$TARGET_STUB" ]
+    [ "$(argv_value --n-gpu-layers)" -gt 0 ]
+    # llama.cpp divides --ctx-size across slots and a mamba hybrid cannot
+    # share one cache, so the observable contract is the per-slot window.
+    [ "$(argv_value --parallel)" -eq 3 ]
+    ctx="$(argv_value --ctx-size)"
+    par="$(argv_value --parallel)"
+    [ $((ctx % par)) -eq 0 ]
+    [ $((ctx / par)) -eq 180224 ]
+    [ $((ctx / par)) -le 262144 ]
 }
 
-@test "frank-small's slot grid is fp8 KV and fits the measured card budget" {
+@test "the KV contract is asymmetric: unquantized K, q8_0 V, flash-attn on" {
-    local ctx running kv ssmdtype mambasize required
+    serve_argv
-    ctx="$(sed -nE 's/^CONTEXT_LENGTH=([0-9]+)$/\1/p' "$ENV_FILE")"
+    [ "$status" -eq 0 ]
-    running="$(sed -nE 's/^MAX_RUNNING=([0-9]+)$/\1/p' "$ENV_FILE")"
+    [ "$(argv_value --cache-type-k)" = "f16" ]
-    kv="$(sed -nE 's/^KV_DTYPE=(.+)$/\1/p' "$ENV_FILE")"
+    [ "$(argv_value --cache-type-v)" = "q8_0" ]
-    ssmdtype="$(sed -nE 's/^MAMBA_SSM_DTYPE=(.+)$/\1/p' "$ENV_FILE")"
+    # q8_0 on V only exists on the flash-attention path; without this flag
-    mambasize="$(sed -nE 's/^MAX_MAMBA_CACHE_SIZE=([0-9]+)$/\1/p' "$ENV_FILE")"
+    # pair the KV contract in the env file is a lie the engine never sees.
-    [ -n "$ctx" ] && [ -n "$running" ] && [ -n "$kv" ]
+    [ "$(argv_value --flash-attn)" = "on" ]
-    # fp8 KV is the only dtype whose 16 KiB/token makes four slots fit;
-    # 64 tokens per MiB of pool.
-    case "$kv" in fp8*) ;; *) return 1 ;; esac
-    # The DeltaNet ssm state pool is the hidden fit cost: the engine
-    # autofills mamba slots to spend a fixed ~5.1 GB budget, so dtype
-    # alone does not shrink it — the SLOT COUNT must be pinned (bf16,
-    # bounded). These two gates are what make the measured pool below
-    # exist; change either and re-measure the pool at boot.
-    [ "$ssmdtype" = "bfloat16" ]
-    [ -n "$mambasize" ] && [ "$mambasize" -ge "$running" ] && [ "$mambasize" -le 64 ]
-    grep -q -- '--mamba-ssm-dtype "$MAMBA_SSM_DTYPE"' "$SERVE"
-    grep -q -- '--max-mamba-cache-size "$MAX_MAMBA_CACHE_SIZE"' "$SERVE"
-    # The slot window is inside the model's native 262144 window.
-    [ "$ctx" -le 262144 ]
-    # The grid must equal the pinned pool. The engine's autofit picks a
-    # bigger pool (687,388 tokens) but leaves ~0.53 GB — Triton's
-    # deferred kernel loads take most of it and eager-prefill activations
-    # then OOM on the first long request. 606,208 (4 x 151,552) is the
-    # verified ceiling: 170k single request and 4 concurrent 54k requests
-    # run against it (2026-10-01). A weights or mamba-flag change moves
-    # this number; re-measure before trusting the grid.
-    pinned_pool=606208
-    required=$((running * ctx))
-    [ "$required" -le "$pinned_pool" ]
-    grep -q -- '--max-total-tokens "$MAX_TOTAL_TOKENS"' "$SERVE"
-    grep -q -- '--disable-prefill-cuda-graph' "$SERVE"
 }
 
-@test "frank-small runs on its own port beside frank-large's" {
+@test "the KV budget leaves 2-3 GiB of the 3090 free" {
-    grep -q '^LISTEN_PORT=7771$' "$ENV_FILE"
+    # Rate: K f16 (16 KiB/token) + V q8_0 (~8.5, budgeted 25 with slack).
-    grep -q '^ALIAS=frank-small$' "$ENV_FILE"
+    # The non-KV half is MEASURED on this card on 2026-10-02: with the
-    grep -q -- '--port "$LISTEN_PORT"' "$SERVE"
+    # projector loaded the engine reported 22,972 MiB used against 14,113
-    grep -q -- '--host "$BIND_ADDR"' "$SERVE"
+    # MiB of KV, so weights + vision tower + DeltaNet state + compute
-    # The two endpoints cannot share a port: frank-large holds 7770, this one
+    # buffers + CUDA graphs = 8,859 MiB => 9 GiB. Two asserts, both about
-    # 7771.
+    # the output the operator sees: the pool must still fit with 2 GiB
-    grep -q '^PORT=7770$' "$ENV_LARGE"
+    # spare, and it must be large enough that 5 GiB spare is NOT reachable
-    ! grep -q '^PORT=7771$' "$ENV_LARGE"
+    # (an undersized pool is as wrong as an oversized one). Arithmetic, not
+    # a boot log: change CTX_SIZE and this bites before the deploy does —
+    # it rejects the 3 x 196,608 pool that measured only 1.1 GiB free. The
+    # same pool with f16 V (34 KiB/token) lands past the card, which is why
+    # the contract is asymmetric.
+    ctx="$(sed -nE 's/^CTX_SIZE=([0-9]+)$/\1/p' "$ENV_FILE")"
+    kv_gib=$((ctx * 25 / 1048576))
+    [ $((kv_gib + 9 + 2)) -le 24 ]
+    [ $((kv_gib + 9 + 5)) -gt 24 ]
+    f16v_gib=$((ctx * 34 / 1048576))
+    [ $((f16v_gib + 9)) -gt 24 ]
 }
 
-@test "the deploy handler installs the root unit and refuses a missing payload" {
+@test "no speculative draft ships by default" {
+    # The qwen35 GGUF has no MTP tensors, and the cross-model DFlash draft
+    # is a measured decision the harness makes later, not a default.
+    ! grep -q -- '--model-draft' "$SERVE"
+    ! grep -q -- '--spec-type' "$SERVE"
+    ! grep -q '^DRAFT' "$ENV_FILE"
+}
+
+@test "vision ships by default and MMPROJ_FILE empty opts out of the image path" {
+    # The Q5_K_M GGUF carries no `v.*` tensors, so the projector argument IS
+    # the vision capability: no flag, no eyes.
+    serve_argv
+    [ "$status" -eq 0 ]
+    [ "$(argv_value --mmproj)" = "$MODEL_DIR/$MM_STUB" ]
+
+    # The documented opt-out must reach the engine as ABSENCE of the flag,
+    # not as an argument pointing at an empty string.
+    sed -i 's|^MMPROJ_FILE=.*|MMPROJ_FILE=|' "$STAGE/test.env"
+    run env FRANK_SMALL_ENV_FILE="$STAGE/test.env" nvidia-modprobe=true bash "$SERVE"
+    [ "$status" -eq 0 ]
+    ! grep -qxF -- '--mmproj' <<<"$output"
+}
+
+@test "the fleet reasoning contract survives the reslice to three contexts" {
+    serve_argv
+    [ "$status" -eq 0 ]
+    # deepseek format = thinking lands in reasoning_content, the field the
+    # harness reads on this endpoint; --jinja is what makes
+    # chat_template_kwargs reach the template at all.
+    [ "$(argv_value --reasoning-format)" = "deepseek" ]
+    argv_value --jinja >/dev/null
+    grep -qxF -- '--jinja' <<<"$output"
+}
+
+@test "the serve script refuses a missing runtime, GGUF or projector" {
+    rm -f "$MODEL_DIR/$MM_STUB"
+    serve_argv
+    [ "$status" -ne 0 ]
+    [[ "$output" == *"frank-small: missing"* ]]
+
+    rm -f "$RUNTIME_DIR/bin/llama-server"
+    serve_argv
+    [ "$status" -ne 0 ]
+    [[ "$output" == *"frank-small: missing"* ]]
+}
+
+@test "the deploy handler gates on binary, GGUF and projector, then MainPID-hardens health" {
     grep -q 'deploy_llm_small' "$REMOTE_DEPLOY"
+    grep -q 'LLM_SMALL_RUNTIME' "$REMOTE_DEPLOY"
+    grep -q 'LLM_SMALL_MODEL_FILE' "$REMOTE_DEPLOY"
+    grep -q 'LLM_SMALL_MMPROJ' "$REMOTE_DEPLOY"
     grep -q 'relink_unit frank-small' "$REMOTE_DEPLOY"
     grep -q 'systemctl enable frank-small.service' "$REMOTE_DEPLOY"
-    grep -q 'systemctl restart frank-small.service' "$REMOTE_DEPLOY"
-    grep -q 'MiMo-V2.6-Distill-Qwen-9B-INT8-W8A16-AutoRound' "$REMOTE_DEPLOY"
-    grep -q 'refusing to start a unit that cannot load' "$REMOTE_DEPLOY"
     grep -q '127.0.0.1:7771/health' "$REMOTE_DEPLOY"
+    grep -q 'frank-small.service has no MainPID' "$REMOTE_DEPLOY"
 }
 
-@test "frank-small's serve script refuses to start on a half-finished download" {
+@test "the two endpoints keep separate ports and the large handler never pkills llama-server" {
-    # The payload is 8 safetensors shards; the serve script counts them and
+    # frank-large's gufo sweep must not take down frank-small, whose
-    # the download script pins each one's LFS sha256.
+    # process is llama-server on the SAME host since this swap.
-    grep -q 'frank-small: missing' "$SERVE"
+    grep -q '^LISTEN_PORT=7771$' "$ENV_FILE"
-    grep -q 'expected 8 safetensors shards' "$SERVE"
+    local large
-    grep -q 'model-00001-of-00008.safetensors' "$DOWNLOAD"
+    large="$(sed -n '/^deploy_llm_large() {/,/^}/p' "$REMOTE_DEPLOY")"
-    grep -q 'model-00008-of-00008.safetensors' "$DOWNLOAD"
+    ! grep -q 'pkill -x llama-server' <<<"$large"
 }