frank-primary: digest-pin FreeToken base image, symlink libhiprtc dev name

- FROM pinned to sha256:a223aee1... (the pulled rocm/pytorch rocm7.14 pytorch 2.11.0 manifest); a re-pushed tag cannot change the stack. - The pip-ROCm torch ships libhiprtc.so.7 only; the PR's _cpu_moe links -lhiprtc and ld needs the unversioned symlink in the already-linked _rocm_sdk_core/lib (first build died: cannot find -lhiprtc).
frank-primary: add FreeToken PR587 ROCm serving image recipe
2026-10-05 19:11:15 -07:00 · 2026-10-05 19:00:30 -07:00 · 2026-10-05 18:40:31 -07:00 · 2026-10-05 18:31:50 -07:00 · 2026-10-05 18:23:44 -07:00 · 2026-10-05 18:18:21 -07:00
78 changed files with 11574 additions and 1211 deletions
@@ -12,8 +12,8 @@ merge absorbed the former `vllmScripts` repo, history preserved).
  Calibre-Web, Jellyfin, Plex) plus the `frank-primary` LLM service.
 - **Dankclaw** (192.168.0.143) — AI/search/dev/observability stack (SearXNG
  via Gluetun, ComfyUI, OpenWebUI, Hermes, Gitea, Grafana, Prometheus, Loki,
-  Alloy, Pi-hole) plus TWO LLM services: `frank-large` (stock llama.cpp on
-  the gfx1151 iGPU) and `frank-small` (llama.cpp on the RTX 3090). Caddy on
+  Alloy, Pi-hole) plus TWO LLM services: `frank-large` (halogen container on
+  the gfx1151 iGPU) and `frank-small` (llama.cpp CUDA on the RTX 3090). Caddy on
  Dankclaw reverse-proxies both hosts.
 - **FrankBook-Pro** (192.168.0.112) — macOS mini running the *arr stack
  (Prowlarr, Sonarr, Radarr, Lidarr, Readarr, qBittorrent) as quadlets inside
@@ -24,9 +24,9 @@ merge absorbed the former `vllmScripts` repo, history preserved).

 | Target | Service | Runtime | Model | Port |
 |--------|---------|---------|-------|------|
-| Dankserver | `frank-primary` | R9V radiance-vLLM fork `drwolfen/radiance-vllm-qwen4exp` (docker-pinned image bundle `2dac17a2`, docker→podman shim, root systemd unit) | unsloth Qwen3.8-Flash-Next-GGUF Q8_0 (175.30 GiB, 6 GGUF shards) with the blockfp8 MTP4 head (depth 4) and mmproj, TP=2 tiered experts (80-slot LRU cache; PLE n-gram PINNED in host RAM 54.4 GB; quantized inject-shard dev-overlay adapter), CED off, 131072 ctx single stream, Qwen-Sharp template, tool parser `qwen3_coder`, reasoning channel `reasoning` | 7770 |
-| Dankclaw | `frank-large` | gufo `gufo-org/gufo` v0.1.1, the Strix Halo engine, BUILT FROM SOURCE (root systemd unit; links the host ROCm 7.2.4) | Swift-1.5-Qwen3.8-27B-Uncensored UD-Q8_K_XL (27B dense, ABLITERATED/uncensored, 29.3 GiB, SWIFT OPEN LICENSE 1.0) with the z-lab DFlash2 Q4_K_M drafter (adaptive block policy, cap 7) on the gfx1151 iGPU (Strix Halo LPDDR, 114 GiB GTT), 262144 ctx single session, greedy, gufo's compiled-in Qwen renderer (NO Qwen-Sharp), text only | 7770 |
-| Dankclaw | `frank-small` | mainline llama.cpp master 37b53fd (root systemd unit) | HauhauCS Gemma4-12B-QAT-Uncensored-Balanced Q4_K_M (BF16 mmproj + MTP assistant sidecar, `draft-mtp,ngram-mod`) on the RTX 3090, 1048576 pool / 4 slots of 262,144, q8_0 KV, the fleet's fastest vision endpoint | 7771 |
+| Dankserver | `frank-primary` | tcclaviger RDNA4 vLLM fork (`docker.io/tcclaviger/vllm@sha256:da4029666…`, digest-pinned, plain `podman run`, root systemd unit) | `orcarouter/Qwen3.8-Flash-Next-Uncensored-FP8` (~172.8 GiB, 131 shards), TP=2, expert offload automatic (fork plans it), prefetch OFF, **fp8 KV** (`--kv-cache-dtype fp8`), **no speculative decode**, MNBT 2048 (arena-free: the registry prices overhead flat up to its nbt 4096 anchor), MoE gather 32×64, **`--expert-precise-memory`** (fitted 5.25 GiB overhead, not the 7.60 GiB default), **`VLLM_CUDAGRAPH_MEM_BASE_MB=0` / `_PER_GRAPH_MB=12`** (the plan's CUDA-graph term priced at the measured capture — 0.31 GiB then a NEGATIVE −0.06 GiB delta, so the 256 MiB analytic base was fiction → graph term 0.07 GiB), **`GPU_MAX_HW_QUEUES=2`**, **mounted cache dir** at `/root/.cache` (the image ships no `/root/.cache` at all; unmounted, every boot re-pays `torch.compile took 181.20 s`), **3 slots × 200704 ctx** at KV_BYTES 3,166,500,000/rank (~400,700 KV tokens at the engine's 7,891.8 B/token here) — the **pool, not the slot count, caps resident conversations** (measured live: `GPU KV cache size: 401,408 tokens, Maximum concurrency for 200,704 tokens per request: 2.00×`; the 3.15e9 pool below it measured 398,658/1.99×), and shorter requests use the spare blocks. **`--max-num-seqs` must stay 3: SEQS=2 cannot boot this model** — torch.fx `ConstraintViolationError` on `query_start_loc`, measured at BOTH MNBT 1024 and 2048. **util stays 0.982 BY LAW**: at `le=1` the worker init refuses `budget == total` with `ValueError: Free memory on device cuda:N (31.29/31.86 GiB)` on both ranks before a single weight loads — measured crash loop 2026-10-03. Arena at shape G: **17.77 GiB / 7,762 slots measured**, KV 401,408 tokens / 2.00× concurrency, 0.89 GiB of real free VRAM per card held through a live 4,096-token generation (the ~3.5 GiB of driver-reported free VRAM at shape E was estimator slack — the registry's 9190 B/token KV reserve vs the real 7,892, and the 5.25 GiB overhead vs far less — NOT allocatable), and the client contract wired: `--reasoning-parser …
+| Dankclaw | `frank-large` | **halogen container** (`ghcr.io/peonist-ai/halogen@sha256:dc0a39a0…`, digest-pinned, plain `podman run` via `scripts/serve-frank-large.sh`, root systemd unit) | `peonist-ai/halogen-qwen3.8-27b` **`qwen3.8-27b-p1w4d-d2.hgn`** (35,865,565,184 B; the `.hgn` format loads in NO other engine — llama.cpp, vLLM and transformers all refuse it), **DFlash2 spec decode (`HALOGEN_DRAFTER=2`)**, **1 slot × 200,700 ctx** — `HALOGEN_KV_SLOTS=1` is load-bearing: batching XOR speculation, slots>1 silently disables the drafter (measured: 8 slots at serial ~6 t/s/stream), d2 KV (2× fp8, 64 KiB/token → 12.2 GiB pool, engine sizes it and prints a boot ledger naming any clamp), **TEXT-ONLY** (the engine ignores the vision encoder — vision is frank-small's job), **censored base** (the ROCmFPX AGENT rollback is the refusal escape hatch), **no server-side sampler: absent temperature is GREEDY**, so the card's thinking recipe (temp 1.0 / top_p 0.95) rides the wire in `models.yml`. Publisher card 31.7 t/s mean across eight text workloads vs the retired ROCmFPX endpoint's measured 27.6–28.1. Prompt over the slot AND `max_tokens` over `HALOGEN_MAX_TOKENS_CAP` are both 400s naming the limit, never truncations. Serve passes NO `--ulimit nofile` (rootless runc cannot raise Dankclaw's 524288 hard limit — see Key Lessons). Cutover 2026-10-05|7770|
+| Dankclaw | `frank-small` | llama.cpp master, CUDA sm_86 build (root systemd unit, relinked) | bartowski/MiMo-V2.6-Distill-Qwen-9B-GGUF **Q6_K** (9B dense qwen3_5 hybrid, 7.26 GiB GGUF, no MTP weights so no spec decode) + the repo's f16 mmproj for **vision**, on the RTX 3090, **2 × 200,704** (pool 401,408 — NOT the native window) with **fully unquantized KV (f16 K + f16 V)** under flash attention, **batch 4096 / ubatch 2048** (doubled 2026-10-05: measured +4.6 % prefill at 8,928 tokens for 616 MiB), 21,846 used / **2,279 free** of 24,576 MiB, 94.4 tok/s decode. Resized 2026-10-03 (3 × 180,224 + q8_0 V → 2 × 200,704 + f16 V → + Q6_K rung: the freed slots paid for exact values AND the higher quant; the rung costs a measured −10 % decode — weight-bandwidth-bound); rollback `MODEL_FILE=…/MiMo-Q5KM/…Q5_K_M.gguf` (buys the 10 % back) or `CTX_SIZE=540672 PARALLEL=3 CACHE_TYPE_V=q8_0` (Q5 only) | 7771 |

 Retired endpoints (`frank-npu`, and every prior runtime) are in
 the [runbook index](#runbook-index). The full swap history lives in
@@ -74,6 +74,14 @@ unrelated
 container changes cannot interrupt inference. Set `FRANK_SUDO_PW` for headless
 runs on a host without passwordless sudo; the value never enters the repo.

+**`deploy.sh <target>` with no `--service` is a fleet-wide DRAIN, not a sync.**
+It also walks every per-service revision marker under the deploy-state dir and
+restarts each service whose marker is behind HEAD *and* whose paths changed
+since that marker — so a backlog accumulated by service-scoped deploys fires
+all at once, and one failing health gate aborts the run with every later
+service left pending. Name the service unless a drain is what you want
+(`./scripts/deploy.sh dankserver --service frank-primary`).
+
 ### Deployment completeness (MANDATORY)

 A deploy is **not complete** when `deploy.sh` exits green. It requires:
@@ -82,7 +90,14 @@ A deploy is **not complete** when `deploy.sh` exits green. It requires:
 2. the Playwright e2e suite (`tests/e2e/run.sh`) — runs **automatically** as
   the `[3/3]` gate at the end of `deploy.sh` against the LIVE fleet; a
   behavior-affecting deploy that fails it exits `DEPLOY INCOMPLETE`. Bypass
-   only with `--skip-e2e` in an emergency.
+   only with `--skip-e2e` in an emergency. The gate runs the conditional
+   `frankbook` Playwright project (the `*arr`/qBittorrent walk,
+   `tests/e2e/specs/19-frankbook.spec.ts`) ONLY when a frankbook blast-radius
+   path changed since origin/main — that spec, `frankbook-pro/**`,
+   `shared/homepage/services.json`, or the `dankclaw/Caddyfile` serving the
+   arr vhosts — so a sleeping FrankBook-Pro never fails an unrelated deploy.
+   The default project set excludes it everywhere else (`E2E_INCLUDE_FRANKBOOK=1`
+   opts in; see `serverAppConfigs-docs/Architecture/E2E Suite.md`).

 ```mermaid
 graph LR
@@ -209,14 +224,14 @@ archive.
 **STOP the production service BEFORE benchmarking** — these servers serve
 real users, and a bench beside a live service starves the GPU and corrupts
 results. Use the runtime's own harness: `llama-bench` for the llama.cpp
-endpoints, `gufo bench` for frank-large. Rolling your own bench
+endpoints; a live llama.cpp server also reports honest per-request numbers in the `/completion` `timings` object (prefill/decode rates, `draft_n`, `draft_n_accepted`), which is how frank-large and frank-small were measured. Rolling your own bench
 requires asking first.
 Scratch ports are not used on LLM hosts — a
 test process cannot share the single GPU anyway; validate against the service
 itself.

-**`gufo bench` cannot rank speculative-decode settings, and will invert the
-ranking.** Measured 2026-09-28 on frank-large at depth 32768: its corpus
+(A gufo-era law; frank-large ran ROCmFPX llama.cpp 2026-10-02 → 10-05 and is the halogen container since, but the rollback tags keep it live.) **`gufo bench` cannot rank speculative-decode settings, and will invert the ranking.**
+Measured 2026-09-28 on frank-large at depth 32768: its corpus
 reports `acceptance=0.933 ± 0.000` with an **identical trace sha256** across
 draft widths, i.e. the same highly-predictable text every run, and 93 % of
 every proposal accepted. It reported 29–37 t/s decode where real traffic
@@ -241,11 +256,11 @@ one: `ip neigh show <ip>` from a peer — FAILED means off/NIC dead, while a
 merely wedged kernel usually still answers ARP.

 1. Stop and mask (self-healing would restart it):
-   - Dankclaw `frank-large` (root unit, gufo on the iGPU):
+   - Dankclaw `frank-large` (root unit, halogen podman on the iGPU):
     `sudo systemctl stop frank-large && sudo systemctl mask frank-large`
   - Dankclaw `frank-small` (root unit, llama.cpp on the 3090):
     `sudo systemctl stop frank-small && sudo systemctl mask frank-small`
-   - Dankserver `frank-primary` (root unit, R9V radiance on the 2x R9700):
+   - Dankserver `frank-primary` (root unit, tcclaviger vLLM on the 2x R9700):
     `sudo systemctl stop frank-primary && sudo systemctl mask frank-primary`
 2. Bench.
   NOTE: `systemctl mask` FAILS on a relinked unit (the unit file in
@@ -277,232 +292,274 @@ When renaming a service, retiring an endpoint, or moving a model: update the
 provider block, model `id`, `contextWindow`, `maxTokens`, `input` in
 `models.yml`, then fix every `modelRoles` entry that named the old endpoint —
 a role pointing at a provider the file does not define fails at call time. As
-of 2026-09-26 both `frank-small` and `frank-primary` accept images
-(`input: [text, image]`) — frank-primary loads the BF16 mmproj explicitly
-(this build never autoloads projectors); `frank-large` remains text-only
-because its stock llama.cpp deployment loads no
-vision projector (the mmproj is not loaded). Note also that
-the reasoning channel differs by runtime: llama.cpp endpoints emit
-`reasoning_content`, vLLM emits `reasoning` — academic since the
-2026-09-26 swap, as every endpoint in the fleet is llama.cpp now.
+of 2026-10-05 **two of the three endpoints accept images** (`input: [text,
+image]`) — `frank-large` went TEXT-ONLY with the halogen swap (the engine
+ignores the vision encoder; the BF16-mmproj era ended 2026-10-05) — while
+`frank-primary` takes them through a BF16 mmproj loaded explicitly (this build never
+autoloads projectors), and `frank-small` through
+`mmproj-MiMo-V2.6-Distill-Qwen-9B-f16.gguf` passed as `--mmproj` — the MiMo
+GGUF itself carries no vision tensors. `frank-small`'s `contextWindow` is
+**200704** (resliced from 180224 on 2026-10-03), the per-slot window, not the
+model's native 262144: the harness must compact before llama.cpp answers a
+longer prompt with a 400. Note also
+that
+the reasoning channel differs by runtime: llama.cpp AND SGLang (qwen3
+parser, live-verified) emit `reasoning_content`, vLLM emits `reasoning` —
+models.yml sets `reasoningContentField` per endpoint to match.
 There is **no audio**
-input anywhere in the fleet except frank-small: that capability left with the
-FLM NPU endpoint on 2026-09-11 and returned on 2026-09-17 with the Gemma4
-projector's audio encoder (frank-small transcribes WAV verbatim even though
-`audio` is omitted from its `input` list — OMP's schema rejects it).
+input anywhere in the fleet: that capability left with the FLM NPU
+endpoint on 2026-09-11, returned 2026-09-17 with the Gemma4 projector's
+audio encoder, and left again with the Gemma4 retirement on 2026-10-01
+(the MiMo tower is vision-only).

-### frank-large — gufo, Swift-1.5-Qwen3.8-27B-Uncensored UD-Q8_K_XL (Dankclaw, ACTIVE)
+### frank-large — halogen container, Qwen3.8-27B + DFlash2 (Dankclaw, ACTIVE)

-**gufo** (`gufo-org/gufo`) v0.1.1 — "the Strix Halo inference engine", a
-vertical engine built for exactly this hardware (Ryzen AI MAX+ 395 /
-Radeon 8060S, `gfx1151`) — as a **root systemd unit**
-(`scripts/frank-large.service`, `User=dank`), serving
-`ajgazin/Swift-1.5-Qwen3.8-27B-Uncensored-Dynamic-MTP-GGUF`'s `UD-Q8_K_XL`
-(27B dense, 29.3 GiB, sha256-verified; **abliterated/uncensored**, and under
-the *non-OSI* SWIFT OPEN LICENSE 1.0) on port 7770 from the gfx1151 iGPU.
-One session, full 262,144 ctx (the model's native window, `SESSIONS=1`),
-DFlash2 spec decode (the z-lab `Qwen3.8-27B-DFlash2-Q4_K_M` drafter, gufo's
-**adaptive** block policy, `--draft-tokens 7`), greedy sampling, reasoning
-extracted to `reasoning_content`, and **no vision projector**.
+The **halogen engine** (`ghcr.io/peonist-ai/halogen@sha256:dc0a39a0…`,
+digest-pinned, published tag 0.1.4) as a podman container driven by
+`scripts/serve-frank-large.sh` under a **root systemd unit**
+(`scripts/frank-large.service`, `User=dank`, `Restart=always`, `ExecStop=podman
+stop -t 120 frank-large-halogen`, the script `exec podman wait` so the restart
+policy is real) on port 7770 from the Strix Halo iGPU. Engine stdout goes to
+`/home/dank/logs/frank_large.log` — **the boot ledger (KV pool, drafter state,
+any clamp) lives there, not in the journal**. Serving
+`peonist-ai/halogen-qwen3.8-27b`: `qwen3.8-27b-p1w4d-d2.hgn` (35,865,565,184 B
+≈ 33.4 GiB, sha256 `274c3fc7…`, 1352 tensors; p1 weights / w4 activations /
+d2 KV) + the flat `tokenizer/` directory (5 files). Cutover 2026-10-05 from the ROCmFPX llama.cpp fork
+([[frank-large-rocmfpx-q6agent-uncensored-mtp]], measured 27.6–28.1 t/s prose),
+chosen for the publisher's 31.7 t/s DFlash2 mean.

-**Two swaps landed on 2026-09-28**, both same day: the engine (from the
-unsloth llama.cpp b11030 prebuilt, which had replaced Kairic Edge ROCmFPX)
-and then the payload (from `unsloth/Qwen3.8-27B-GGUF:UD-Q8_K_XL`, still on
-disk and the payload-only rollback). The two targets are structurally
-interchangeable for gufo — same `qwen35` architecture, the same 866 tensors
-in the same order, the same per-tensor Q8_0/F32/BF16 layout, the same MTP
-head at `blk.64` — verified by parsing and diffing both GGUF headers before
-either was loaded. **The DFlash2 drafter did not move**, so it has its own
-`DRAFTER_DIR` and the deployer's stray-run sweep keys on the drafter's
-filename, not the target's.
+**The format is the lock.** `.hgn` loads in nothing else — the runtime is
+chosen by the FILE, as ROCmFPX's tensor types were before it. This is NOT the
+2026-09 `halogen-flash-server` image (different repo, Flash-Next w4b era); the
+halogen-server repo restarted versioning at 0.1.x.

-**The runtime is built from source on the host** —
-`/home/dank/GitPackages/gufo-v0.1.1/`, reproducible with
-`scripts/build-gufo.sh` (`--verify` re-checks an installed tree). gufo ships
-no prebuilt Linux binaries and the v0.1.0/v0.1.1 release assets are empty.
-Unlike the retired llama.cpp prebuilt, which bundled its own ROCm, **gufo
-links the host ROCm 7.2.4 through its RPATH**, so the HIP dev packages must
-stay installed. Two host workarounds the build script encodes:
+**Two product regressions, both accepted by Frank before deploy:** the
+endpoint is **text-only** (README honest limits: the vision encoder is
+ignored — frank-small owns images now) and the checkpoint is a finetune of the
+**censored** `Qwen/Qwen3.8-27B` base (refusals can return; the rollback is one
+commit-revert away with the AGENT payload still on disk).

- **Ubuntu's `librocwmma-dev` must be absent.** It installs a truncated
-  rocWMMA header set into `/usr/include` that shadows AMD's complete one
-  and breaks the build with `'internal/accessors.hpp' file not found`. A
-  `-I` cannot fix it — in HIP mode the host pass searches the system
-  directories first. The package has no reverse dependencies.
- **`--gcc-install-dir` must name GCC 15.** Ubuntu 26.04 ships
-  `/usr/include/c++/16`, which the ROCm clang prefers; its `<atomic_wait.h>`
-  needs a `std::__detail::__notify_impl` symbol version the installed
-  `libstdc++.so.6` (GLIBCXX_3.4.35) does not export, and the link fails.
+**Engine laws that shape the config** (all in `scripts/frank-large.env` with
+evidence): `HALOGEN_KV_SLOTS=1` — batch XOR speculation, slots>1 turns the
+drafter off silently. `HALOGEN_SLOT_CTX=200700` — the per-conversation HARD
+ceiling (400 naming the limit, never truncation); `models.yml`
+`contextWindow: 200700` must match or the harness discovers the wall as a 400.
+`HALOGEN_MAX_TOKENS_CAP=65536` + `HALOGEN_QUEUE_TIMEOUT=7200` are a coupled
+pair; the harness `maxTokens: 57344` stays under the cap (over-cap is a 400
+naming the limit — README, never a 413). Thinking tokens COUNT against the
+budget: exhausted mid-thought is `finish_reason: "length"` with empty
+`content` and the reasoning in `reasoning_content`.
+`HALOGEN_DRAFTER=2` = DFlash2. **No server-side sampler**: an absent
+temperature is greedy, so the thinking recipe exists only in `models.yml`
+`extraBody`. `response_format` is unsupported (no constrained decoding).
+Thinking lands in **`reasoning_content`** (harness field unchanged); the same
+Qwen3.8 template rejects `reasoning_effort` outside `xhigh|medium|low`, so the
+`models.yml` fold stays mandatory.

-`gufo diagnose` is the preflight and must report `[PASS]` with `gfx1151`.
-It also self-caps compute queues (`GPU_MAX_HW_QUEUES`, 2 resident for
-`serve llm`) after reading `/sys/class/kfd/kfd/proc/*/queues/*/type`, which
-matters because ComfyUI holds queues on the same device; four HIP processes
-is the device ceiling.
+Fetch: `scripts/download-frank-large-halogen.sh` into
+`/home/dank/models/halogen-qwen3.8-27b/` (`-f` + byte count + LFS sha256 +
+`--verify-only`, no token on the command line; resume goes against the
+RESOLVED CDN URL because curl drops `Range` across HF's 302). The deployer
+and serve script both refuse a missing tokenizer file or **any symlink** —
+an HF cache snapshot symlinks each entry into `blobs/` and those links
+dangle inside the container mount.

-**The client contract changed, and the chat template is the sharp edge.**
-**gufo has no Jinja engine at all** — `src/models/qwen/chat_template.cpp` is
-587 lines of hand-written C++ that appends strings, with no
-`--chat-template-file`, no env override, and no template body to point at.
-`scripts/qwen-sharp-v22.4.0.jinja` therefore **cannot** be supplied to
-frank-large: that is a port, not a config line. Given the choice on
-2026-09-28 the decision was to ship on gufo's stock renderer and accept the
-loss — the `terse` system-prompt injection, the in-band `<|think_*|>`
-control, the `json|xml` `tool_call_format` switch, the tool-argument
-truncation and `add_vision_id` prefixes are all gone. gufo's own renderer
-does keep the effort instruction strings and `preserve_thinking`, which are
-what the harness depends on. **This is the thing to watch**: the endpoint
-now runs an *abliterated* model without the prompt that tells it to stop
-waffling. The cheap fix, if verbosity shows up, is to send the `terse` text
-as an ordinary system message from `models.yml` — not a template change.
-`reasoning_effort` accepts the full OpenAI set (all six levels verified
-200), so `models.yml`'s `reasoningEffortMap` fold is no longer *required* —
-it is kept deliberately, so that removing it does not silently move every
-`high` role from `xhigh` to `high` effort inside a swap. And there is no
-KV-quantization or prompt-cache-RAM flag; prompt reuse is on by default and
-`--cache-disk` is opt-in and off.
-
-**Read the perf trade before assuming a win.** gufo's own Q8_K_XL tables
-(vs llama.cpp b11069, DFlash2) show prefill +37 % to +80 %, cold load 4.5×
-faster, decode +7 % at zero depth and +40 % at 4 users — but **−14 % at
-65,536 tokens of depth and −24 % at 131,072**. On the 8-bit target the
-crossover is ~32k of cached context, and frank-large runs one 262,144-token
-conversation, so long agent sessions sit on the losing side of that line.
-The Q4_K_XL target shows no such regression; revisiting the quantization is
-a separate decision from the runtime. Peak memory is 50.58 GiB at the full
-262,144 context against the iGPU's 114 GiB GTT aperture.
-
-The serve script (`scripts/serve-frank-large.sh`) validates runtime → target
-GGUF → drafter and exits with a pointer message if any is missing, then
-`exec`s `gufo serve --host 0.0.0.0 --port 7770 --sessions 1 llm --model
-…--served-model-name frank-large --context 262144 --speculative dflash2 …`.
-The deployer's `deploy_llm_large` gates on the **runtime binary** as well
-as both payload paths, and returns before it stops anything.
-
-Fetch: `scripts/download-frank-large-swift.sh` (the target; per-file byte
-count + sha256 against the Hub LFS oid, no auth header — the repo is public
-and a token on the curl command line is world-readable) and
-`scripts/download-frank-large.sh` (the DFlash2 drafter, which did not move).
-Both take `--verify-only`.
-
-Runbook: `serverAppConfigs-docs/LLM/frank-large-gufo-qwen38-27b.md`.
-Rollback, two ways: **payload only** — re-fetch with
-`scripts/download-frank-large.sh` and repoint `MODEL_FILE` at
-`/home/dank/models/Qwen3.8-27B-UD-Q8_K_XL/Qwen3.8-27B-UD-Q8_K_XL.gguf`; that
-29.3 GiB target was **reclaimed on 2026-09-28** (staged to
-`~/.AGENTTRASH/20260928-retired-models/`, restorable from there), so it is
-no longer a one-minute flip. **Engine** —
-[[frank-large-unsloth-qwen38-27b-q8xl]], the b11030 prebuilt at
-`/home/dank/GitPackages/unsloth-llama-b11030/`; the only way to get
-Qwen-Sharp back. The DFlash2 drafter was never reclaimed and still lives in
-`Qwen3.8-27B-UD-Q8_K_XL/`.
+Runbook: `serverAppConfigs-docs/LLM/frank-large-halogen27b-dflash2.md`.
+Rollback: revert the swap commit and redeploy — the ROCmFPX payload
+(`/home/dank/models/kingjones-Qwen3.8-27B-ROCmFP4-STRIX-MTP/`), the fork build
+and `scripts/build-rocmfpx.sh` (pin `36e9acd`) all remain on disk, seam-tested.
+The older gufo tag `frank-large-gufo-q4kxl-20261001` stays a **re-download**
+rollback (payloads removed 2026-10-03; `scripts/build-gufo.sh` stays).

 ### frank-small — llama.cpp on the RTX 3090 (Dankclaw, ACTIVE)

-Mainline llama.cpp master `37b53fd` (CUDA build with
-`-DGGML_CUDA_FA_ALL_QUANTS=ON`) as a **root systemd unit** (repo
-`scripts/frank-small.service`, `relink_unit`ed into `/etc/systemd/system/`),
-serving `HauhauCS/Gemma4-12B-QAT-Uncensored-HauhauCS-Balanced` at Q4_K_M.
-The MTP head is a SEPARATE assistant file (`mtp-gemma-4-12B-it.gguf`, 4
-nextn layers) loaded via `--model-draft`; the draftless **ngram-mod**
-matcher chains behind it in the same `--spec-type` value. BF16 mmproj,
-port 7771, q8_0 KV, `CTX_SIZE=1048576` / `PARALLEL=4`
-(four FULL 262,144-token slots — the measured card ceiling), batch
-**8192/1024**, greedy default. Deployed flags are in
-`scripts/frank-small.env`; the serve script refuses to start if the
-weights, the projector, the sidecar or the binary are missing.
+**llama.cpp master** (CUDA sm_86 build
+`/home/dank/GitPackages/llama-master/build-cuda/bin/llama-server`) as a
+**root systemd unit** (`scripts/frank-small.service`, `relink_unit`ed into
+`/etc/systemd/system/`), serving
+`bartowski/MiMo-V2.6-Distill-Qwen-9B-GGUF` at **Q6_K** (7,793,710,624 B,
+sha256 `ef96d05a…`) **plus `mmproj-MiMo-V2.6-Distill-Qwen-9B-f16.gguf`**
+(918,166,048 B, sha256 `ff348f…`), both `/home/dank/models/MiMo-Q6K/` (the
+projector is a hardlink of the Q5KM copy — the dir is self-contained), on
+port 7771. The rung went Q5_K_M → Q6_K on 2026-10-03 the same morning as the
+reslice, funded by the freed slots. The runtime arrived 2026-10-01 night for
+one reason:
+**asymmetric KV** — `--cache-type-k f16` + `--cache-type-v q8_0` is the fleet's
+only per-tensor KV lever, and SGLang's `--kv-cache-dtype` moves K and V
+together. On 2026-10-03 Frank spent the headroom that lever bought: the reslice
+from three slots to two made room for **fully unquantized V**
+(`CACHE_TYPE_V=f16`); both flags stay explicit in the serve script. Vision came
+2026-10-02 via `--mmproj`: the GGUF carries no `v.*` tensors, so the projector
+file is the entire vision path (and ~1.4 GiB of the card).

-Verified at deploy (2026-09-17): `/v1/models` reports `frank-small`; a
-chat completion returns content plus `reasoning_content`; a tool call
-returns `tool_calls`; an image request is answered from the projector;
-an `input_audio` WAV transcribes verbatim. Post-tuning live numbers at
-max length: pp@254k ~965 t/s, tg@253k 226–302 t/s with ~94% draft
-acceptance; idle 23,300 MiB of 24,576. The throughput sweep (and why
-ubatch 2048 OOMs the draft's pp buffer, and why draft depth 4 + poll
-100 is WORSE at long ctx) is tabled in the runbook.
-Runbook: `serverAppConfigs-docs/LLM/frank-small-gemma4-12b.md`.
-Rollback: [[frank-small-qwen38-gsq-rco]].
+**Architecture: qwen3_5 hybrid** — 32 layers, 8 full-attention + 24 Gated
+DeltaNet, native 262,144 window. KV exists only in the 8 attention layers
+(4 KV heads × 256 dim × 2 B ≈ 2 KiB/token/layer, ~16 KiB/token for f16 K+V);
+the DeltaNet state is a fixed per-slot buffer, so slot count — not context —
+drives that term.

-### frank-primary — R9V radiance-vLLM Qwen3.8-Flash-Next Q8_0 (Dankserver, ACTIVE)
+**The shape (2026-10-03):** `--ctx-size 401408 --parallel 2` = **TWO
+conversations of 200,704 tokens each** (llama.cpp divides the pool:
+`n_ctx_slot = CTX_SIZE / PARALLEL`), `--flash-attn on` (the seam pins it),
+**f16 K + f16 V**, all layers on the GPU, `--mmproj` for vision, and since
+2026-10-05 **batch 4096 / ubatch 2048** (doubled; the boot-pair A/B is in the
+runbook). Measured trade at 8,928 prompt tokens: prefill 4,024 → **4,209 t/s,
+4.6 %**, decode flat (91.9–93.0 vs 93.8–94.3 — noise), and a 279-token prompt,
+which sits below either ubatch, did not move at all. The old "ubatch 2048
+failed the scratch boot" note belongs to the Q5_K_M predecessor and does NOT
+generalize: this shape boots 2048 clean. Measured VRAM **21,846 used / 2,279
+free** of 24,576 MiB against **21,276 / 2,849** for the 1024 arm, reproduced
+byte-identically the same hour — the doubling costs **616 MiB** and lands at
+the BOTTOM of the 2–3 GiB floor Frank asked for (`4096/1536` is the middle rung
+if that margin is ever wanted; non-KV 9,302 MiB, up from 8,732 by exactly the
+compute buffers; f16 K+V is 32 KiB/token across the 8 attention layers;
+quote `nvidia-smi`'s `free` field, never `total − used` — the driver holds
+~450 MiB of reserve the subtraction hides). The 2026-10-02 3 × 180,224 +
+q8_0-V shape stays one env edit away (`CTX_SIZE=540672 PARALLEL=3
+CACHE_TYPE_V=q8_0`, and it must pair with Q5_K_M — Q6_K breaks the floor
+there). Three native windows need 18.4 GiB of KV alone and were never on the
+table.

-The **R9V radiance-vLLM fork** (`drwolfen/radiance-vllm-qwen4exp`, docker-pinned
-prebuilt image bundle `r9v-qwen38-flash-next-wmma@sha256:2dac17a2…`) serving
-the **unsloth Qwen3.8-Flash-Next-GGUF Q8_0** (175.30 GiB, 6 GGUF shards)
-with the blockfp8 MTP4 head (depth 4) and the mmproj on **both** Radeon AI
-PRO R9700s (gfx1201, BDFs `0000:03:00.0` / `0000:07:00.0`), one stream at
-**131,072 ctx**, **TP=2 tiered experts** (80-slot LRU expert cache; PLE
-n-gram table PINNED in host RAM, 54.4 GB), the **Qwen-Sharp v22.4.0
-template** (via the launcher's `R9V_CHAT_TEMPLATE_FILE` passthrough, fork
-`30d4a32`), tool parser `qwen3_coder`, reasoning channel `reasoning`. Cut
-over 2026-09-27 from the uncensored IQ4_XS payload (itself cut over from
-llama.cpp b11030 the same day) at Frank's direction.
+Verified at deploy (2026-10-03): boot logs `n_slots = 2, n_ctx_slot = 200704`
+and `loaded multimodal model`; decode **94.44 tok/s** at temp 0 on the probe
+that measured 104.84 on Q5_K_M — **the Q6_K rung costs a measured −10 %**:
+decode on this card is weight-bandwidth-bound and Q6_K streams ~10 % more
+bytes/token; health 200, e2e 33/33; native `tool_calls`; thinking in
+`reasoning_content`; a **512×512** solid-red PNG answers `Red` through an
+`image_url` content part (two traps: the legacy top-level `images` field is
+ignored, and a TINY probe image undersupplies image tokens — a 10×10 PNG made
+the model claim "it's a video"; probe with ≥256 px). A prompt over the slot
+is a **400 `exceed_context_size_error`**, which is why `models.yml` sets
+`contextWindow: 200704`.

-**The tuned shape lives on-host, not in the repo.** The q8 profile's setup
-run qualified the `R9V_*` config into
-`/home/dank/r9v-state/q8xl/setup.json` (`ready: true`); the repo env
-file `scripts/frank-primary.env` carries ONLY production overrides
-(`R9V_HOST_BIND=0.0.0.0`, `R9V_HOST_PORT=7770`,
-`R9V_SERVED_MODEL_NAME=frank-primary`, `R9V_CONTAINER_NAME=frank-primary-r9v`,
-`R9V_EXPECTED_GPU_BDFS`) plus two q8-only keys: the dev-overlay adapter
-(`R9V_DEV_QWEN4_EXP_ADAPTER_PY`, fork commit `363d606` — the Q8_0 payload
-quantizes BOTH fused hyper-connection shards, so the shard-uniformity
-check needs the in-stream dequant) and `R9V_FULL_MUTABLE_CACHE=0` (the
-overlay's env.always injects the uncensored 160-slot cache geometry, which
-the q8 80-slot tiered cache rejects). Never copy tuned values from the
-state into the repo — it would drift from the qualification.
+Runbook: `serverAppConfigs-docs/LLM/frank-small-llama-mimo-q6k-2x200704.md`
+(the boot-pair table that prices pool changes lives in the retired
+[[frank-small-llama-mimo-q5km]]).
+Rollback: `MODEL_FILE=/home/dank/models/MiMo-Q5KM/MiMo-V2.6-Distill-Qwen-9B-Q5_K_M.gguf`
+buys back 875 MiB and the −10 % decode rate (3,571 free, 104.84 tok/s, both
+measured); the older shape `CTX_SIZE=540672 PARALLEL=3 CACHE_TYPE_V=q8_0`
+pairs with Q5 only; SGLang rollback in
+[[frank-small-sglang-mimo-v26]] (venv on disk at
+`/home/dank/GitPackages/sglang-cu128/`, INT8 payload staged to
+`~dank/.AGENTTRASH/20261002-retired-llm-payloads/`).

-**Spec decoding is MTP4 only.** The fork's `speculative_config` is a
-single dict — one method per engine, NO chaining — so the llama.cpp
-`draft-mtp,ngram-mod` pattern is NOT expressible here. MTP4 is wired
-(`R9V_MTP_SPEC_TOKENS=4`, draft TP=2); the PLE n-gram rides along
-architecturally (`ngram_embedding`); `ngram_gpu` exists as an alternative
-single method, not a chainable one.
+### frank-primary — tcclaviger vLLM, Qwen3.8-Flash-Next FP8 (Dankserver, ACTIVE)

-**Serving = the fork's own launcher through a docker→podman shim.** The R9V
-tooling is docker-pinned; the shim at
-`/home/dank/GitPackages/radiance-vllm-qwen4exp/.podman-shim/docker` carries
-three laws: (1) the bundle is a containerd/OCI docker-save, so docker keys
-the image by its OCI manifest digest while podman keys by config digest but
-preserves the manifest digest in `RepoDigests` — the shim answers
-`image inspect sha256:X` from RepoDigests and rewrites bare `sha256:X` run
-refs to `repo@sha256:X`; (2) podman inspect answers
-`HostConfig.LogConfig.Config: null` where docker returns the log-opts map
-(the fork's doctor crashes on it) — the shim records `run --log-opt` pairs
-and normalizes inspect output; (3) podman lacks docker's `SecurityOptions`
-/`ServerVersion` info fields — the shim answers them.
+The **tcclaviger RDNA4 vLLM fork** (`docker.io/tcclaviger/vllm@sha256:da4029666…`,
+digest-pinned) serving **`orcarouter/Qwen3.8-Flash-Next-Uncensored-FP8`**
+(~172.8 GiB, 131 shards) on **both** Radeon AI PRO R9700s (gfx1201), **TP=2**,
+**fp8 KV**, **no speculative decode**, **expert prefetch OFF**, and since
+2026-10-03 the **shape-G** geometry (shape E + MNBT 2048 + the CUDA-graph plan
+term at the measured floor): `--max-model-len 200704` × 3 slots with
+`--kv-cache-memory-bytes 3,166,500,000`/rank (~400,700 KV tokens — the engine
+prices this pool at 7,891.8 B/token here), util **0.982** (the law — see the
+unbootable-1.0 lesson below), `--expert-precise-memory`,
+`VLLM_CUDAGRAPH_MEM_BASE_MB=0` / `_PER_GRAPH_MB=12`,
+`GPU_MAX_HW_QUEUES=2` and a mounted `/root/.cache`. `models.yml` carries
+`contextWindow: 200704` for this endpoint — the served window, not the model's
+native 262144, so the harness compacts before vLLM answers a longer prompt
+with a 400.

-**Unit and self-healing.** Root systemd unit (`scripts/frank-primary.service`,
-`User=dank`, `LimitMEMLOCK=infinity`) — the PLE pinned residency alone is
-54.4 GB, and a root unit grants the pin budget via PID1 (a rootless
-quadlet would be capped by the user manager's limit).
-The serve script (`scripts/serve-frank-primary.sh`) gates the setup state
-(ready + paths), the pinned image (RepoDigests), the PLE byte count
-(`stat -c %s`), every GGUF shard + the mmproj and the GPU device nodes,
-clears a stale container, then execs the fork's `scripts/launch.sh` through
-the shim and **blocks on `docker wait`** — the launcher `--detach`es, so
-without the wait a crashed engine is a green unit; `docker wait` cannot stop
-the container child, hence `ExecStop=podman stop -t 120
-frank-primary-r9v`. The deployer's `deploy_llm_primary` gates state/image/
-PLE/shards, `podman rm -f`s the container, waits out :7770, relinks/enables/
-restarts the unit and health-gates with the MainPID check (the false-green
-stray law).
+**`--max-num-seqs 2` is impossible on this model.** Measured 2026-10-02 by
+single-variable boot probe against the then-deployed shape: SEQS 2 fails at
+BOTH MNBT 1024 and 2048 with `torch.fx.experimental.symbolic_shapes.Constraint
+ViolationError: Constraints violated (L['query_start_loc'].size()[0]) …
+specialized it to be a constant (3)`; SEQS 3 passes at both. `model_runner.py`
+sizes that input buffer `max_num_reqs + 1` and `_dummy_run` sets
+`num_reqs = min(max_num_batched_tokens, max_num_reqs)`, so the width is
+`SEQS + 1` for any `MNBT ≥ SEQS` — MNBT only moves the compile RANGE
+((1,1024) vs (1,2048)). `--enforce-eager` bypasses the path but drops CUDA
+graphs. The same failure class killed the 2026-10-01 bf16 2×262144 shape.

-Verified at cutover (2026-09-27): q8 setup state `ready: true`, all 20
-payload artifacts sha256-verified (175.30 GiB); the scratch-port bring-up
-battery answered vision ("Red" for a 64×64 red PNG), MTP4 mean acceptance
-length 3.19–5.00, and the PLE pinned proof (`mapping_bytes =
-mapping_rss_bytes = 54,400,261,120`, fully RSS-resident); wire + tool
-round-trip + template probe re-verified on :7770 after the deploy. No
-throughput bench was run (Frank declined); the probe-era 11 tok/s decode
-was residency-bound (halved expert headroom at double the bytes/expert),
-not the production number.
+**The expert arena has THREE reachable levers: `max_model_len`, the overhead
+term, and the CUDA-graph estimate.**
+The fork's planner sizes expert cache as `budget − non-expert − KV reserve −
+graphs − overhead`, and the KV reserve is registry-priced at 9190 B/token ×
+`max_model_len`: 262144 → 2.48 GiB reserve → arena 14.30 GiB / **6,248 slots**;
+200704 → 1.91 GiB → 14.87 GiB / **6,496 slots**. Δreserve 0.57 GiB = Δarena
+0.57 GiB. The 6.1e9 → 3.15e9 pool cut bought **zero** arena; it buys runtime
+headroom (the shape it replaced died with `0 bytes free` on a real 995-token
+request). The second lever is `--expert-precise-memory`, **ON since
+2026-10-02**: it swaps the registry's default overhead (7.60 GiB = fitted +
+compile/graph-capture headroom) for the fitted figure, and the 2.35 GiB
+difference lands in the arena — measured plan lines at 200704 × 3: overhead
+5.25 GiB, expert cache **17.22 GiB / 7,522 slots** (+1,026 / +16 %), swap region
+13.7 GiB / 5,986 slots, boot healthy in 420 s. The pool stayed at 3.15e9
+deliberately: because the pool is not an arena lever, cutting it would cost
+concurrency (a 2.2e9 probe reports 277,686 tokens / **1.38×**) and buy no slots.
+The arena (2.344 MiB per slot) is the decode-critical region — cutting it for KV
+headroom collapses the offload hit rate.

-Runbook: `serverAppConfigs-docs/LLM/frank-primary-r9v-radiance-qwen38-q8.md`.
-Rollback: the uncensored payload — flip `R9V_STATE_FILE` back to
-`/home/dank/r9v-state/uncensored/setup.json`, drop the two q8-only keys
-and the template keys, redeploy (state + `restore-config-20260927.env` +
-payload intact on disk) — see
-[[frank-primary-r9v-radiance-qwen38-uncensored]].
+**Free VRAM on this endpoint is NOT arena headroom (2026-10-03 audit).**
+`rocm-smi` reads ~3.5 GiB free per card after hours of real traffic, but the
+planner charges `budget = total × util` against terms that overstate reality:
+the KV reserve is registry-priced at 9190 B/token (`1.91 GiB` at 200704) while
+the real pool bills 2.93 GiB at 7,892 B/token, and the precise overhead (5.25
+GiB) is far above the idle-measured figure. So the visible free space is the
+estimator gap, not allocatable arena. `gpu_memory_utilization` is a real arena
+lever (each +0.01 = +0.32 GiB) but **0.982 is the law, and 1.00 is UNBOOTABLE
+on this fork — measured 2026-10-03**: the worker init demands
+`free > budget`, and at `le=1` budget == device total, so both ranks raise
+`ValueError: Free memory on device cuda:N (31.29/31.86 GiB ...)` before a
+single weight loads, twice (2-restart crash loop, zero payload progress). The
+CLI field accepting `le=1` is the trap; Frank's 0.982 is where it always
+ships. What IS reachable from flags: `VLLM_CUDAGRAPH_MEM_BASE_MB` /
+`_PER_GRAPH_MB`, read at call time by `estimate_cudagraph_memory()` — the
+analytic default (256 MiB + 52 MiB/graph) charged 0.55 GiB where the engine's
+own post-capture line reported 0.31 GiB, and a second boot (with the plan
+already charging 0.32 GiB) reported **-0.06 GiB**, so the 256 MiB base pool is
+pure over-charge: shape G charges `0 base + 12 MiB/graph` = 0.07 GiB, which
+still prices per graph for a future 18-graph spec-decode shape (0.21 GiB).
+Verify every deploy by comparing the `Estimated CUDA graph memory`
+line with `Graph capturing finished … took X GiB`; if capture exceeds the
+estimate, raise `_PER_GRAPH_MB` back toward 52. MNBT 1024 → 2048 costs ZERO
+arena: the registry overhead is `base + 0.22 GiB × max(0,(nbt−4096)/1024)`
+anchored at nbt 4096, so every rung ≤4096 charges the same 5.25 GiB — and the
+fitted figure was fitted AT 4096.
+
+**A container's compile cache does NOT survive `podman rm -f`, and on this
+endpoint that is 181 s per boot.** Measured 2026-10-02: the image ships **no
+`/root/.cache` at all** (fresh container: `No such file or directory`); after a
+boot it holds `/root/.cache/vllm` at **1,154,714,333 B in 4,665 files** plus
+`/root/.cache/comgr` at 229 MB, all in the ephemeral layer, and the engine
+prints `torch.compile took 181.20 s in total` (173.19 s of it one graph,
+`Compiling a graph for compile range (1, 1024)`). Every deploy, restart and
+reboot re-pays it. The serve script now mounts
+`FRANK_PRIMARY_CACHE_DIR` at `/root/.cache` — the parent, so the vLLM AOT /
+torch_compile tree (`VLLM_CACHE_ROOT`, whose default is
+`/root/.cache/vllm`) and the comgr HIP-kernel JIT cache both persist, and
+nothing in the image is hidden because the image has no such directory. The
+compile RANGE is part of the cache key, so an MNBT change costs one
+recompilation, not a permanent tax. Same class of fix as the recipe's
+`/cache/{triton,vllm,inductor}` volumes. **Proven live 2026-10-02 on
+consecutive boots:** cold `Compiling a graph for compile range (1, 1024) takes
+179.24 s` / `torch.compile took 188.26 s`; warm `Directly load the compiled
+graph(s) for compile range (1, 1024) from the cache, took 2.143 s` /
+`torch.compile took 3.97 s in total` / `init engine (profile, create kv cache,
+warmup model) took 41.13 s (compilation: 3.98 s)`. Time to `/health` is
+dominated by the 172.8 GiB payload load, so the visible win is the ~184 s of
+inductor/AOT build, not a proportional boot cut.
+
+Measured 2026-09-29 tuning campaign (fp8 + MTP4 era, since retired): decode
+**37.4-40.6 tok/s, acceptance ~54 %**; the 256k ctx arm measured 34.1-36.5;
+2-way concurrency sums BELOW parity (25.49 vs 26.8 single) — slots are a
+capacity feature, not a throughput one. V-only q8 KV is NOT possible via
+config: stock `--kv-cache-dtype` is K+V together; a V-only quant needs a fork
+patch.
+
+Unit and self-healing: root systemd unit (`scripts/frank-primary.service`,
+`User=dank`, `LimitMEMLOCK=infinity`), `ExecStop=podman stop -t 120
+frank-primary-qwen`. The serve script drives `podman run` directly, gates
+the digest-pinned image, the payload's `config.json` and its exact
+131-shard count and the GPU device nodes, clears stale container
+identities, then blocks on `podman wait`. `deploy_llm_primary` gates the
+same, waits out :7770, relinks/enables/restarts the unit and health-gates
+with the MainPID check. Boot cost ~7 min warm (the 172.8 GiB payload load; the
+~184 s inductor/AOT build inside it is what the mounted cache now skips — see
+the compile-cache lesson below); health returns 503 while loading.
+
+Runbook: `serverAppConfigs-docs/LLM/dankserver-qwen38-fp8-expert-offload-tuning.md`.
+Retired GLM experiment: `serverAppConfigs-docs/LLM/frank-primary-glm53-flash-w4a16.md`.

 ## Key Lessons Learned
 Durable operational lessons from the fleet's history. Full detail in the
@@ -668,15 +725,56 @@ linked runbooks.
  the KV pool is worse than the OOM it prevents.
 - Host swap state is invisible to `deploy.sh` (no repo file) — check
  `swapon --show` before reasoning about headroom.
+- **Silent RAM corruption outranks every software theory.** frank-primary's
+  ~40k-token output corruption (2026-10-04/05, repetition/garbage in long
+  agent conversations) was marginal DDR5 at 4800 MT/s on four DIMMs, fixed
+  by 4600. Consumer non-ECC memory raises **no MCE** for a silent flip, and
+  a short in-band pattern check (60 s, 8 GiB of 246 GiB) is not a stability
+  proof — combined, those two instruments said "ruled out" while the RAM
+  was the cause. "4800 is the JEDEC floor on 5600-rated DIMMs" is a rating,
+  not a margin guarantee. When the fault is data-shaped (NaN states,
+  garbage tokens) and a software revert doesn't move it, price in the RAM
+  before the runtime — and remember a revert tested under the same marginal
+  hardware is itself confounded. See the runbook's 2026-10-05 section.

 **Runtime-specific guards**

+- **Rootless podman cannot RAISE rlimits, and `--ulimit nofile=1048576`
+  dies at container create, not at runtime.** runc reports
+  `error setting rlimit type 7: operation not permitted` (type 7 =
+  RLIMIT_NOFILE) when the requested hard limit exceeds the host process's
+  hard limit — on Dankclaw that ceiling is 524288. Inherited-equal limits
+  are fine, so `memlock=-1:-1` works iff the unit grants
+  `LimitMEMLOCK=infinity` (systemd is privileged; the value flows through
+  podman to runc without any raise). Upstream halogen's own run lines pass
+  NO `--ulimit` at all — copy vendor run lines before adding flags
+  (measured 2026-10-05 on the frank-large cutover).
 - **Kairic Edge** (if ever re-deployed): the pinned CK commit needs
  `third_party/composable_kernel/include/ck/config.h` generated manually
  (untracked; `git clean` removes it), and its greedy-argmax fastpath rejects
  non-greedy requests and reasoning budgets. Re-test chat before production.
 - **FastMTP sidecar**: build must be commit `4df29be4` + the HauhauCS patch;
  stock binaries fail the draft load with a tensor-shape mismatch.
+- **A GGUF's custom tensor type picks the runtime — read the histogram before
+  promising a file to a loader.** `Qwen3.8-27B-Q8_0_ROCMFPX_AGENT.gguf` is
+  ftype 115 with 311 tensors of type 103 (`GGML_TYPE_Q8_0_ROCMFPX`); stock
+  llama.cpp and gufo refuse it at load, so ROCmFPX was not a preference but
+  the only door. Counting tensor types out of the GGUF KV section takes one
+  python read and settles a runtime argument in minutes. The build identity
+  matters as much as the fork: the 2026-08-25 ROCmFPX builds SIGSEGV in
+  `ggml_compute_forward_set_rows` on this same file, so
+  `scripts/build-rocmfpx.sh` pins commit `36e9acd` by SHA.
+- **Speculative-decode flags move as a SET.** Handing the ROCmFPX fork
+  `--model-draft` with no `--spec-type` (the naive way to build a "no MTP"
+  bench arm) segfaulted at load, core dumped, no diagnostic (2026-10-02). A
+  baseline arm must strip `--model-draft` AND every `--spec-draft-*` AND
+  `--spec-type`, or it measures a crash instead of the absence of drafting.
+- **An orphaned scratch server taxes the whole host.** A bench child left
+  alive by a client-side exception kept ~26 GiB of weights resident on a
+  121 GiB box, pushed `available` to 3 GiB, and every later rate was suspect.
+  Record the child pid and kill it in a `finally`, then prove `free` came back
+  before believing a number. Kill by pid — never `pkill -f llama-server`,
+  which also hits frank-small.

 **llama.cpp serving**

@@ -686,19 +784,26 @@ linked runbooks.
  the MAIN model's weights (`shares_model`), so there is no second copy of a
  12 GiB model in VRAM and no `-md` argument. The plain build has no MTP
  tensors at all, and pointing `--model-draft` at the file would load it twice.
- **What can be chained, and what cannot.** `--spec-type a,b` builds every
-  enabled implementation and chains drafting through them, in a fixed internal
-  order (ngram types first, then the drafters — CLI order is irrelevant). The
-  gate is `ctx_dft`: **draftless** types (the ngram family) are added
-  unconditionally, while `draft-mtp`, `draft-dflash`, `draft-dspark` and
-  `draft-eagle3` all require a draft CONTEXT. So a chain may hold one
-  context-backed drafter plus any number of ngram types — `draft-dflash,ngram-mod`
-  is legal and measured (2.6x the drafter alone on prompt-echo traffic, byte-
-  identical to it on prose at the fleet's 48/64 window) — but
-  `draft-mtp,draft-dflash` dies at load: *"context type MTP requested but model
-  doesn't contain MTP layers"*, because MTP builds its context against whatever
-  model the draft context holds. Two context-backed types cannot share one
-  process, and MTP's head only loads when MTP **is** the draft model.
+- **What can be chained, what cannot, and what a chain costs on a hybrid.**
+  `--spec-type a,b` builds every enabled implementation and chains drafting
+  through them, in a fixed internal order (ngram types first, then the
+  drafters — CLI order is irrelevant). The gate is `ctx_dft`: **draftless**
+  types (the ngram family) are added unconditionally, while `draft-mtp`,
+  `draft-dflash`, `draft-dspark` and `draft-eagle3` all require a draft
+  CONTEXT. So a chain may hold one context-backed drafter plus any number of
+  ngram types — `draft-dflash,ngram-mod` is legal and measured (2.6x the
+  drafter alone on prompt-echo traffic, byte-identical to it on prose at the
+  fleet's 48/64 window) — but `draft-mtp,draft-dflash` dies at load: *"context
+  type MTP requested but model doesn't contain MTP layers"*, because MTP
+  builds its context against whatever model the draft context holds. Two
+  context-backed types cannot share one process, and MTP's head only loads
+  when MTP **is** the draft model. **A LEGAL chain can still lose on a
+  recurrent model:** `draft-mtp,ngram-mod` on frank-large's qwen35 hybrid
+  measured −15 % prose / −7 % block-copy (2026-10-03) — adding the ngram layer
+  logs `recurrent state rollback is not compatible with 'ngram-mod' -
+  disabling rollback support`, and MTP verification depends on that rollback,
+  so acceptance drops on ALL traffic even where the matcher fires. Measure
+  chains per architecture, never by analogy.
 - **A drafter's KV scales with the TARGET's context, and its window policy
  decides how much that costs.** The draft context is built with
  `cparams.n_ctx = llama_n_ctx(ctx_tgt)` (`common/speculative.cpp`) — there is
@@ -715,7 +820,8 @@ linked runbooks.
  trip moved the balance point FOUR layers (~4.3 GiB of headroom, 40:25 ->
  36:29), which is the practical size of the effect. See
  [[frank-medium-llamacpp-dflash2]].
- **`--ubatch-size` has an OPTIMUM on this stack, not a plateau — and its
+- **`--ubatch-size` has an OPTIMUM on the 2x R9700 layer split, not a plateau —
+  and the curve is BACKEND-SPECIFIC, so never carry it across (next bullet). Its
  compute buffers land about TWICE as heavy on the card that holds the output
  layers and the drafter.** Measured on frank-medium 2026-09-16 at the deployed
  shape (262144 over two slots, a 64k prefill measured at 64k depth,
@@ -733,6 +839,19 @@ linked runbooks.
  its 2020 MiB pp buffer here under the DSpark drafter, when ROCm1 had 0.08 GiB
  free. That was a CONTIGUITY failure, not a total-bytes one, and no totals
  model — the sweep's or the seam's — can see it.)
+- **On the CUDA 3090 the same knob has NO interior optimum up to 2048, so
+  frank-small ships 4096/2048.** Measured 2026-10-05 by boot-pair A/B on the
+  live service: at 8,928 prompt tokens ubatch 1024 gives 3,972 / 4,077 t/s and
+  ubatch 2048 gives 4,207 / 4,212 — **+4.6 %** — for **616 MiB** (free 2,849 →
+  2,279). Decode did not move (91.9–93.0 vs 93.8–94.3, i.e. noise), and a
+  279-token prompt, which is below either ubatch, came back identical — that is
+  the mechanical proof the knob was the only variable, since a prompt shorter
+  than the ubatch cannot see it. The logical `--batch-size` was again free. And
+  the predecessor's "ubatch 2048 failed the scratch boot" was a shape
+  observation, not a law: the same file boots 2048 clean at 2 × 200,704. Bench
+  it by restarting the SERVICE through a transient `FRANK_SMALL_ENV_FILE`
+  drop-in, never a scratch port — one card holds one server. See
+  [[frank-small-llama-mimo-q6k-2x200704]].
 - **One `-ts` step is one LAYER, and a hybrid stack's layers are not the same
  size.** qwen35 is 16 x (3 Gated DeltaNet + 1 Gated Attention), so a step is
  worth ~0.43 GiB on a DeltaNet boundary and ~1.43 GiB on a full-attention one
@@ -749,6 +868,41 @@ linked runbooks.
  host-RAM hash table) and its window decides which traffic it can see: too
  short and it proposes on ordinary prose fragments, which costs verification
  without acceptance.
+- **`--parallel` divides the pool, so MORE conversations cost WINDOW, and the
+  per-slot number is the contract — not `ctx-size`.** `n_ctx_slot =
+  CTX_SIZE / PARALLEL`, and llama.cpp refuses a prompt longer than the SLOT
+  (`400 exceed_context_size_error: request (184001 tokens) exceeds the
+  available context size (180224 tokens)`) even when the pool has room. So
+  frank-small's 2 → 3 reslice moved each conversation from the model's native
+  262,144 down to 180,224, and `models.yml` `contextWindow` had to follow or
+  the harness would discover the ceiling as a 400 on a long conversation. Side
+  effect worth knowing: the shorter slot made decode ~5 % FASTER (106.8 vs
+  100.4 tok/s) — less attention to scan and one fewer DeltaNet state resident.
+  The 2026-10-03 reslice went the other way (3 → 2 slots, 180,224 → 200,704,
+  pool 401,408) to fund unquantized f16 V, and `contextWindow` followed again;
+  decode landed 104.8 tok/s — the ~2 % off 106.8 is the exact-V tax. The same
+  day the freed space also bought the Q6_K rung (now 94.4 tok/s — the rung
+  itself is a further measured −10 %, weight-bandwidth, not slot geometry).
+  All moves measured; both slot directions cost or buy window, never pool.
+  See [[frank-small-llama-mimo-q6k-2x200704]].
+- **Price a VRAM budget from two boots, never from a formula — and an mmproj
+  costs ~1.5x its file size.** The frank-small 3090 fit was solved from two
+  measurements: 2 × 262,144 with no projector = 20,190 MiB used (KV 12,800 ⇒
+  **non-KV 7,390**), 3 × 196,608 with the f16 projector = 22,972 used (KV
+  14,113 ⇒ **non-KV 8,859**). The difference is the vision tower: a 918 MB file
+  that bills ~1.4 GiB, because the encoder's activation and compute buffers
+  ride with it. Subtracting the two rows is exact; extrapolating weights+KV+
+  "a bit for buffers" is what leaves 1.1 GiB free when you meant 2.7. Then
+  solve `KV ≤ card − non-KV − headroom` and round to a multiple of
+  `PARALLEL × 4096`. Recorded in `scripts/frank-small.env` so the next resize
+  starts from measurement.
+- **`images` and `image_url` are NOT interchangeable: only the content part
+  reaches the model.** `--mmproj` loads, `usage.prompt_tokens` rises, and the
+  model still invents an answer if the client used the legacy top-level
+  `images` field on `/v1/chat/completions` — frank-small answered "Brown." for
+  a solid-red PNG that way and "Red." through the OpenAI `image_url` part. The
+  failure is silent because a hallucinated color is a plausible answer. Probe
+  vision with a solid color and expect an exact word.
 - **`--jinja` is what `tools` needs.** On 2026-09-16 a subagent died with
  `500 tools param requires --jinja flag`, from the ik_llama.cpp runtime that
  then served frank-large: every tool-using agent on that endpoint was broken
@@ -849,9 +1003,9 @@ Active:

 | Service | Runbook |
 |---------|---------|
-| frank-large (gufo Qwen3.8-27B UD-Q8_K_XL, Dankclaw) | `serverAppConfigs-docs/LLM/frank-large-gufo-qwen38-27b.md` |
-| frank-small (llama.cpp Gemma4-12B QAT, Dankclaw) | `serverAppConfigs-docs/LLM/frank-small-gemma4-12b.md` |
-| frank-primary (R9V radiance-vLLM Qwen3.8-Flash-Next Q8_0, Dankserver) | `serverAppConfigs-docs/LLM/frank-primary-r9v-radiance-qwen38-q8.md` |
+| frank-large (halogen container, Qwen3.8-27B .hgn + DFlash2, 1×200,700, text-only, Dankclaw) | `serverAppConfigs-docs/LLM/frank-large-halogen27b-dflash2.md` |
+| frank-small (llama.cpp CUDA, MiMo-V2.6-Distill-Qwen-9B Q6_K, 2×200,704 f16 KV, Dankclaw) | `serverAppConfigs-docs/LLM/frank-small-llama-mimo-q6k-2x200704.md` |
+| frank-primary (tcclaviger vLLM, Qwen3.8-Flash-Next FP8, Dankserver) | `serverAppConfigs-docs/LLM/dankserver-qwen38-fp8-expert-offload-tuning.md` |
 | Dankclaw kernel posture (`amd_iommu=off`) | `serverAppConfigs-docs/LLM/dankclaw-iommu-translation-off.md` |
 | Deployment flow + incidents | `serverAppConfigs-docs/LLM/deployment.md` |

@@ -859,6 +1013,14 @@ Retired (rollback targets and history):

 | Runtime | Host | Window | Runbook |
 |---------|------|--------|---------|
+| ROCmFPX llama.cpp fork Qwen3.8-27B Uncensored Q6_0_ROCMFPX_AGENT + MTP (`frank-large`) | Dankclaw | 2026-10-03 → 10-05 (rollback target, payload + fork build on disk) | `frank-large-rocmfpx-q6agent-uncensored-mtp.md` |
+| ROCmFPX llama.cpp fork Qwen3.8-27B Q8_0_ROCMFPX_AGENT + MTP (`frank-large`) | Dankclaw | 2026-10-02 → 10-03 (rollback target, payload on disk) | `frank-large-rocmfpx-q8agent-mtp.md` |
+| gufo v0.1.1 Qwen3.8-27B UD-Q8_K_XL/UD-Q4_K_XL + DFlash2 (`frank-large`) | Dankclaw | 2026-09-28 → 10-01, again 10-01 → 10-02 (payloads removed from disk 2026-10-03; rollback = re-download) | `frank-large-gufo-qwen38-27b.md` |
+| vLLM TheRock MiMo-V2.6-Distill-Qwen-9B BF16 + cross-model MTP (`frank-large`) | Dankclaw | 2026-10-01 (same-day rollback to gufo) | `frank-large-vllm-mimo-v26-bf16.md` |
+| llama.cpp master MiMo-V2.6-Distill-Qwen-9B Q6_K (`frank-small`) | Dankclaw | 2026-10-01 → 10-02 (the rung returned ACTIVE 2026-10-03 at the 2×200,704 shape — `frank-small-llama-mimo-q6k-2x200704.md`) | `frank-small-llama-mimo-q6k.md` |
+| llama.cpp master MiMo-V2.6-Distill-Qwen-9B Q5_K_M 2×200,704 f16 KV (`frank-small`) | Dankclaw | 2026-10-02 → 10-03 (rollback target — one `MODEL_FILE` line, buys back 875 MiB + the −10 % rung tax; owns the boot-pair table) | `frank-small-llama-mimo-q5km.md` |
+| SGLang MiMo-V2.6 INT8-W8A16-AutoRound (`frank-small`) | Dankclaw | 2026-10-01 (same-day swap to the GGUF path) | `frank-small-sglang-mimo-v26.md` |
+|tcclaviger vLLM GLM-5.3-Flash W4A16 int4, +/- DFlash2 drafter (`frank-primary`)|Dankserver|2026-09-30 → 10-01|`frank-primary-glm53-flash-w4a16.md`|
 | llama.cpp b11030 Qwen3.8-27B UD-Q8_K_XL + DFlash2 (`frank-large`) | Dankclaw | 2026-09-21 → 09-28 | `frank-large-unsloth-qwen38-27b-q8xl.md` |
 | R9V radiance-vLLM Qwen3.8-Flash-Next-Uncensored IQ4_XS (`frank-primary`) | Dankserver | 2026-09-27 (same-day swap to Q8_0) | `frank-primary-r9v-radiance-qwen38-uncensored.md` |
 | llama.cpp b11030 Qwen3.8-Flash-Next Q8_0 + shared MTP (`frank-primary`) | Dankserver | 2026-09-26 → 09-27 | `frank-primary-llamacpp-qwen38-flash-next-q8.md` |
@@ -872,6 +1034,7 @@ Retired (rollback targets and history):
 | Kairic Edge Qwen3.8-27B IU4 W4A4 (`frank-large`) | Dankclaw | 2026-09-21 (same-day swap) | `frank-large-kairic-edge-qwen38-27b.md` |
 | ISTA-DASLab Qwen3.8-27B-GSQ-RCO IQ3_S (`frank-small`) | Dankclaw | 2026-09-16 → 09-17 | `frank-small-qwen38-gsq-rco.md` |
 | vLLM Gemma4-12B w4a16 (`frank-small`) | Dankclaw | 2026-09-19 (same-day revert) | `frank-small-vllm-gemma4.md` |
+| llama.cpp Gemma4-12B QAT Q4_K_M (`frank-small`) | Dankclaw | 2026-09-17 → 10-01 | `frank-small-gemma4-12b.md` |
 | agentionai ROCmFP4-FAST imatrix | Dankclaw | 2026-08-29 → 08-30 | `agentionai-fast-imatrix.md` |
 | eemin Qwen3.8-27B ROCmI4 W4A4 MTP | Dankclaw | 2026-08-18 → 08-25 | `eemin-rocmi4-mtp.md` |
 | Kairic Edge IU4 | Dankclaw | 2026-08-22 (same-day revert) | `kairic-edge.md` |
@@ -0,0 +1,117 @@
+#!/usr/bin/env bash
+# autoresearch.sh — canonical benchmark entrypoint: speed of frank-primary.
+#
+# Runs on the operator box and drives one bench window on DANKSERVER:
+#   1. fingerprint the candidate trio (env + serve script + unit)
+#   2. memo fast-path: an identical fingerprint that already produced a FULLY
+#      successful window (bench + verified production restore) returns the
+#      cached metrics without touching the fleet
+#   3. stage the candidate trio plus the measure client into
+#      /home/dank/.autoresearch-frank-primary/ (ephemeral; the repo checkout
+#      on the host stays the production source of truth)
+#   4. run the window (scripts/autoresearch/frank-primary-window.sh):
+#      stop + mask the production unit, boot the staged config AS THE SERVICE
+#      (same serve script, same container, same port 7770), wait for health,
+#      run the in-container usage-counter measure, then restore the production
+#      unit and verify its health — AGENTS.md Benchmarking Policy
+#   5. print METRIC lines (parse by run_experiment)
+#
+# Primary metric: decode_tps (HIGHER is better) — median across the three
+# prompt depths (115/1056/4096) of per-depth medians of
+# completion_tokens / (wall - ttft), 3 runs each after a warm pass,
+# temperature 0, the campaign's deterministic word-seed workload.
+#
+# Exit 0 with METRIC lines on success; non-zero on any failure (the window
+# script's EXIT trap restores production on every path).
+set -euo pipefail
+
+SCRIPT_DIR="$(cd -- "$(dirname -- "${BASH_SOURCE[0]}")" && pwd)"
+HOST="${FRANK_PRIMARY_AR_HOST:-dankserver}"
+STAGE="/home/dank/.autoresearch-frank-primary"
+
+# FRANK_PRIMARY_AR_ENV_FILE stages a DIFFERENT env than the repo's — that is how
+# the arms driver A/Bs engine knobs without mutating the checkout. The
+# fingerprint covers whatever file is staged, so the memo cannot cross arms.
+ENV_FILE="${FRANK_PRIMARY_AR_ENV_FILE:-$SCRIPT_DIR/scripts/frank-primary.env}"
+SERVE="$SCRIPT_DIR/scripts/serve-frank-primary.sh"
+UNIT="$SCRIPT_DIR/scripts/frank-primary.service"
+MEASURE="$SCRIPT_DIR/scripts/autoresearch/frank-primary-measure.py"
+WINDOW="$SCRIPT_DIR/scripts/autoresearch/frank-primary-window.sh"
+
+for f in "$ENV_FILE" "$SERVE" "$UNIT" "$MEASURE" "$WINDOW"; do
+    [ -f "$f" ] || { echo "[autoresearch] missing harness file: $f" >&2; exit 1; }
+done
+command -v python3 >/dev/null 2>&1 || { echo "[autoresearch] python3 required on operator box" >&2; exit 1; }
+
+# Fingerprint of everything that can change the engine's argv or environment.
+HASH="$(cat "$ENV_FILE" "$SERVE" "$UNIT" | sha256sum | cut -d' ' -f1)"
+echo "[autoresearch] candidate fingerprint $HASH" >&2
+
+ssh_run() { ssh -o BatchMode=yes -o ConnectTimeout=15 "$HOST" "$@"; }
+
+emit_metrics() { # $1 = run.json
+    python3 -c '
+import json, sys
+run = json.loads(sys.argv[1])
+m = run["measure"]
+print("METRIC decode_tps=%s" % m["decode_tps"])
+for k in ("decode_tps_115", "decode_tps_1056", "decode_tps_4096"):
+    print("METRIC %s=%s" % (k, m[k]))
+print("METRIC ttft_ms_4096=%s" % m["ttft_ms_4096"])
+print("METRIC kv_tokens=%s" % (run.get("kv_tokens") or 0))
+print("METRIC boot_seconds=%s" % (run.get("boot_seconds") or -1))
+print("METRIC completion_tokens_4096=%s" % m.get("completion_tokens_4096", 0))
+# Any extra sampling temperature the measure swept (AR_TEMPS), e.g. the 0.7
+# arm — that is where speculative-decode acceptance shows up, since a greedy
+# draft matches a greedy target exactly and is rejected more under real
+# sampling. Suffixed keys only (_t0.7), so the historical lines are untouched.
+for k in sorted(m):
+    if "_t" in k and (k.startswith("decode_tps") or k.startswith("ttft_ms")):
+        print("METRIC %s=%s" % (k, m[k]))
+' "$1"
+}
+
+# --- memo fast-path -------------------------------------------------------
+cached_hash="$(ssh_run "cat '$STAGE/last/hash' 2>/dev/null" 2>/dev/null || true)"
+if [ -n "$cached_hash" ] && [ "$cached_hash" = "$HASH" ]; then
+    cached_run="$(ssh_run "cat '$STAGE/last/run.json' 2>/dev/null" || true)"
+    if [ -n "$cached_run" ]; then
+        echo "[autoresearch] memo hit — cached metrics for $HASH, fleet untouched" >&2
+        emit_metrics "$cached_run"
+        exit 0
+    fi
+fi
+
+# --- stage the candidate ----------------------------------------------------
+echo "[autoresearch] staging candidate on $HOST:$STAGE" >&2
+ssh_run "mkdir -p '$STAGE'"
+scp -q "$ENV_FILE" "$HOST:$STAGE/frank-primary.env"
+scp -q "$SERVE" "$HOST:$STAGE/serve.sh"
+scp -q "$MEASURE" "$HOST:$STAGE/measure.py"
+scp -q "$WINDOW" "$HOST:$STAGE/window.sh"
+ssh_run "printf '%s\n' '$HASH' > '$STAGE/hash'"
+
+# --- the window -------------------------------------------------------------
+# Budgets: bench boot up to 20 min (warm boots measured ~7 min), measure up
+# to 30 min, restore boot up to 20 min — call run_experiment with
+# timeout_seconds >= 3600.
+echo "[autoresearch] opening bench window (stop+mask -> boot -> measure -> restore)" >&2
+set +e
+ssh_run "bash '$STAGE/window.sh'"
+rc=$?
+set -e
+if [ "$rc" -ne 0 ]; then
+    echo "[autoresearch] window failed (rc=$rc); host-side evidence:" >&2
+    ssh_run "tail -60 '$STAGE/fail.log' 2>/dev/null || tail -40 '$STAGE/serve.out' 2>/dev/null || tail -40 '$STAGE/measure.out' 2>/dev/null || true" >&2 || true
+    exit 1
+fi
+
+run_json="$(ssh_run "cat '$STAGE/run.json'")" || {
+    echo "[autoresearch] could not read $STAGE/run.json back" >&2; exit 1; }
+echo "[autoresearch] window result:" >&2
+printf '%s\n' "$run_json" >&2
+emit_metrics "$run_json"
+
+echo "[autoresearch] engine evidence (plan / compile path / spec counters):" >&2
+ssh_run "cat '$STAGE/evidence.log' 2>/dev/null" >&2 || true
+exit 0
@@ -0,0 +1,76 @@
+#!/usr/bin/env bash
+# Download the z-lab Qwen3.5-9B-DFlash drafter for frank-large's DFlash
+# speculative-decode arm (vLLM --speculative-config method=dflash). Public
+# repo — no auth header on the command line.
+#
+# The drafter is a 6-layer qwen3 draft whose dflash_config pins
+# target_layer_ids [1,5,9,13,17,21,25,29] and block_size 16; its hidden_size
+# 4096 / vocab_size 248320 match the MiMo-V2.6-Distill-Qwen-9B target exactly
+# (both preserve the Qwen3.5-9B geometry), which is what lets vLLM load it as
+# the draft for that target.
+#
+# Usage:
+#   scripts/download-frank-large-dflash.sh [DEST_DIR] [--verify-only]
+set -euo pipefail
+
+REPO="z-lab/Qwen3.5-9B-DFlash"
+BASE="https://huggingface.co/${REPO}/resolve/main"
+
+DEST=""
+VERIFY_ONLY=0
+for arg in "$@"; do
+    case "$arg" in
+        --verify-only) VERIFY_ONLY=1 ;;
+        *) DEST="$arg" ;;
+    esac
+done
+DEST="${DEST:-/home/dank/models/Qwen3.5-9B-DFlash}"
+
+# name bytes sha256(- = non-LFS, size-only check)
+FILES=(
+  "model.safetensors 2583816465 0a42274b32554f48de1faa0d42824e9c2ceda649c30ae0a731cddf410dd698c7"
+  "config.json 1222 -"
+)
+
+mkdir -p "$DEST"
+status=0
+for entry in "${FILES[@]}"; do
+  read -r name size sha <<<"$entry"
+  f="$DEST/$name"
+  have=0
+  if [[ -f "$f" ]] && [[ "$(stat -c%s "$f")" == "$size" ]]; then
+    if [[ "$sha" == "-" ]] || sha256sum -c --quiet <(printf '%s  %s\n' "$sha" "$f"); then
+      have=1
+    fi
+  fi
+  if (( have )); then
+    echo "OK        $name"
+    continue
+  fi
+  if (( VERIFY_ONLY )); then
+    echo "MISSING   $name"
+    status=1
+    continue
+  fi
+  echo "FETCHING  $name"
+  tmp="$f.part"
+  curl -fsSL --fail-with-body -o "$tmp" "$BASE/$name"
+  mv "$tmp" "$f"
+  if [[ "$(stat -c%s "$f")" != "$size" ]]; then
+    echo "SIZE FAIL $name ($(stat -c%s "$f") != $size)"
+    status=1
+    continue
+  fi
+  if [[ "$sha" != "-" ]] && ! sha256sum -c --quiet <(printf '%s  %s\n' "$sha" "$f"); then
+    echo "SHA FAIL  $name"
+    status=1
+    continue
+  fi
+  echo "VERIFIED  $name"
+done
+if (( status == 0 )); then
+  echo "DOWNLOAD_DONE $DEST"
+else
+  echo "DOWNLOAD_FAILED $DEST" >&2
+fi
+exit "$status"
@@ -0,0 +1,90 @@
+#!/usr/bin/env bash
+# Download the BF16 MiMo-V2.6-Distill-Qwen-9B payload for frank-large
+# (vLLM/kyuz0 gfx1151, Strix Halo iGPU). Public repo — no auth header on the
+# command line (a Bearer token in argv is world-readable via /proc).
+#
+# Per-file FRESH downloads, no resume: the house rule (see
+# download-frank-small.sh); -f/--fail-with-body plus size+sha256 against the
+# Hub LFS oid is what makes an access error fail instead of "succeeding".
+#
+# Usage:
+#   scripts/download-frank-large-mimo.sh [DEST_DIR] [--verify-only]
+#
+# The verified set below is exactly what the 2026-10-01 bench window served:
+# 17.53 GiB of BF16 weights (4 shards, 18,819,720,848 B total with the rest).
+set -euo pipefail
+
+REPO="XiaomiMiMo/MiMo-V2.6-Distill-Qwen-9B"
+BASE="https://huggingface.co/${REPO}/resolve/main"
+
+DEST=""
+VERIFY_ONLY=0
+for arg in "$@"; do
+    case "$arg" in
+        --verify-only) VERIFY_ONLY=1 ;;
+        *) DEST="$arg" ;;
+    esac
+done
+DEST="${DEST:-/home/dank/models/MiMo-V2.6-Distill-Qwen-9B-BF16}"
+
+# name bytes sha256(- = non-LFS, size-only check)
+FILES=(
+  "model-00001-of-00004.safetensors 5276436216 aab052180118aee34abc3029b54eaa49096aac606b97d703866b420dceb703c3"
+  "model-00002-of-00004.safetensors 5033171280 7a0486565f06d25ac4628e9dba470dc3f604353471d240d5a0bf7128f64df396"
+  "model-00003-of-00004.safetensors 5234499504 6c73207563d1879bfd6c143a028cc70be68edff56280458c71a43df4240300f4"
+  "model-00004-of-00004.safetensors 3275613848 1379a7cf8c0b8555a39ab65a47e830e0eb45e776b46045734da3afd73c09eea2"
+  "model.safetensors.index.json 69217 -"
+  "config.json 2784 -"
+  "generation_config.json 126 -"
+  "chat_template.jinja 3916 -"
+  "tokenizer.json 19989325 06b9509352d2af50381ab2247e083b80d32d5c0aba91c272ca9ff729b6a0e523"
+  "tokenizer_config.json 1165 -"
+  "vocab.json 6722759 -"
+  "merges.txt 3353259 -"
+  "preprocessor_config.json 443 -"
+  "processor_config.json 1191 -"
+  "video_preprocessor_config.json 385 -"
+)
+
+mkdir -p "$DEST"
+status=0
+for entry in "${FILES[@]}"; do
+  read -r name size sha <<<"$entry"
+  f="$DEST/$name"
+  have=0
+  if [[ -f "$f" ]] && [[ "$(stat -c%s "$f")" == "$size" ]]; then
+    if [[ "$sha" == "-" ]] || sha256sum -c --quiet <(printf '%s  %s\n' "$sha" "$f"); then
+      have=1
+    fi
+  fi
+  if (( have )); then
+    echo "OK        $name"
+    continue
+  fi
+  if (( VERIFY_ONLY )); then
+    echo "MISSING   $name"
+    status=1
+    continue
+  fi
+  echo "FETCHING  $name"
+  tmp="$f.part"
+  curl -fsSL --fail-with-body -o "$tmp" "$BASE/$name"
+  mv "$tmp" "$f"
+  if [[ "$(stat -c%s "$f")" != "$size" ]]; then
+    echo "SIZE FAIL $name ($(stat -c%s "$f") != $size)"
+    status=1
+    continue
+  fi
+  if [[ "$sha" != "-" ]] && ! sha256sum -c --quiet <(printf '%s  %s\n' "$sha" "$f"); then
+    echo "SHA FAIL  $name"
+    status=1
+    continue
+  fi
+  echo "VERIFIED  $name"
+done
+if (( status == 0 )); then
+  echo "DOWNLOAD_DONE $DEST"
+else
+  echo "DOWNLOAD_FAILED $DEST" >&2
+fi
+exit "$status"
@@ -0,0 +1,85 @@
+#!/usr/bin/env bash
+# Download the source Qwen3.5-9B checkpoint — the ONLY place this model
+# family's MTP head (15 mtp.* tensors) exists. frank-large runs it as the
+# vLLM cross-model MTP draft (MiMo-V2.6-Distill-Qwen-9B itself ships zero
+# mtp tensors; that is why SGLang's NEXTN on frank-small draws 0.00
+# acceptance). Public repo — no auth header on the command line.
+#
+# NOTE the odd upstream shard names (model.safetensors-0000N-of-00004...).
+#
+# Usage:
+#   scripts/download-frank-large-qwen35.sh [DEST_DIR] [--verify-only]
+set -euo pipefail
+
+REPO="Qwen/Qwen3.5-9B"
+BASE="https://huggingface.co/${REPO}/resolve/main"
+
+DEST=""
+VERIFY_ONLY=0
+for arg in "$@"; do
+    case "$arg" in
+        --verify-only) VERIFY_ONLY=1 ;;
+        *) DEST="$arg" ;;
+    esac
+done
+DEST="${DEST:-/home/dank/models/Qwen3.5-9B}"
+
+# name bytes sha256(- = non-LFS, size-only check)
+FILES=(
+  "model.safetensors-00001-of-00004.safetensors 5276436216 db6f444b43d318c92f360a13a25561a6a65b10c0631b8ed305a426dbaa6c380e"
+  "model.safetensors-00002-of-00004.safetensors 5335161512 31c7d7e2dd5d207840b31cc59083c8f4c4718959149e0358c0364052bb9a0330"
+  "model.safetensors-00003-of-00004.safetensors 5368717440 7ec36ba3a4176a44c3c0876ad80c56a2f70c84bf008d82e9501df642f17dadec"
+  "model.safetensors-00004-of-00004.safetensors 3325995712 b62b0c4cd7e44edee103ee8f4fe225f246d5e768e07bfd5f25b63a8aa1fdd0c6"
+  "model.safetensors.index.json 79657 -"
+  "config.json 3126 -"
+  "chat_template.jinja 7756 -"
+  "tokenizer.json 12807982 5f9e4d4901a92b997e463c1f46055088b6cca5ca61a6522d1b9f64c4bb81cb42"
+  "tokenizer_config.json 16710 -"
+  "vocab.json 6722759 -"
+  "merges.txt 3353259 -"
+  "preprocessor_config.json 390 -"
+  "video_preprocessor_config.json 385 -"
+)
+
+mkdir -p "$DEST"
+status=0
+for entry in "${FILES[@]}"; do
+  read -r name size sha <<<"$entry"
+  f="$DEST/$name"
+  have=0
+  if [[ -f "$f" ]] && [[ "$(stat -c%s "$f")" == "$size" ]]; then
+    if [[ "$sha" == "-" ]] || sha256sum -c --quiet <(printf '%s  %s\n' "$sha" "$f"); then
+      have=1
+    fi
+  fi
+  if (( have )); then
+    echo "OK        $name"
+    continue
+  fi
+  if (( VERIFY_ONLY )); then
+    echo "MISSING   $name"
+    status=1
+    continue
+  fi
+  echo "FETCHING  $name"
+  tmp="$f.part"
+  curl -fsSL --fail-with-body -o "$tmp" "$BASE/$name"
+  mv "$tmp" "$f"
+  if [[ "$(stat -c%s "$f")" != "$size" ]]; then
+    echo "SIZE FAIL $name ($(stat -c%s "$f") != $size)"
+    status=1
+    continue
+  fi
+  if [[ "$sha" != "-" ]] && ! sha256sum -c --quiet <(printf '%s  %s\n' "$sha" "$f"); then
+    echo "SHA FAIL  $name"
+    status=1
+    continue
+  fi
+  echo "VERIFIED  $name"
+done
+if (( status == 0 )); then
+  echo "DOWNLOAD_DONE $DEST"
+else
+  echo "DOWNLOAD_FAILED $DEST" >&2
+fi
+exit "$status"
@@ -0,0 +1,93 @@
+#!/usr/bin/env bash
+# Download MiMo-V2.6-Distill-Qwen-9B-INT8-W8A16-AutoRound for frank-small
+# (SGLang, RTX 3090). Public repo — no auth header on the command line.
+#
+# Per-file FRESH downloads, no resume: `curl -C -` once produced a 213 MB
+# oversized file on a 31 GiB payload; a fresh -f fetch plus size+sha256 is
+# the house rule.
+#
+# Usage:
+#   scripts/download-frank-small.sh [DEST_DIR] [--verify-only]
+#
+# LFS shards are verified against the Hub LFS sha256; non-LFS text files
+# against their declared byte size. Idempotent: complete files are skipped.
+set -euo pipefail
+
+REPO="letechlead/MiMo-V2.6-Distill-Qwen-9B-INT8-W8A16-AutoRound"
+BASE="https://huggingface.co/${REPO}/resolve/main"
+
+DEST=""
+VERIFY_ONLY=0
+for arg in "$@"; do
+    case "$arg" in
+        --verify-only) VERIFY_ONLY=1 ;;
+        *) DEST="$arg" ;;
+    esac
+done
+DEST="${DEST:-/home/dank/models/MiMo-V2.6-Distill-Qwen-9B-INT8-W8A16-AutoRound}"
+
+# name bytes sha256(- = non-LFS, size-only check)
+FILES=(
+  "model-00001-of-00008.safetensors 2946296312 ddb6117c04cb6feaf7c265e5d75efadf21949037d73253a8e4521d40e164366d"
+  "model-00002-of-00008.safetensors 1402134792 2dc78d55d1d33d7b3b21dd6f8b4b817b62c2fb1376e6448b322428798c10e257"
+  "model-00003-of-00008.safetensors 1427365904 5c797d14e345309d67c78306b9de4aa01a21dbd92dd98e40b0ad69b2900e9203"
+  "model-00004-of-00008.safetensors 1420041856 7402f852e2c7a24ebf4694b98a220e71dfb2c93ba3776272cf2acfc65acd8126"
+  "model-00005-of-00008.safetensors 1410655648 991cbf2eda2ff91fc3ffbb15591d7fcac340fb10d9c33bd3da20010e2f3f4090"
+  "model-00006-of-00008.safetensors 1427627856 efc42448054cfb5e3061e154ba4539204088ce95176d689a95b58de885ef7eeb"
+  "model-00007-of-00008.safetensors 2035442408 75a101fc71ae6f2e1f5c4816f711d62cf1d7a0e02ef6e5182361b8c11e72b626"
+  "model-00008-of-00008.safetensors 822153664 0fb90cb50e2d336fbad58bd4c01880eeddbf537c1dca1cf0ae291195d5efbb4b"
+  "tokenizer.json 19989325 06b9509352d2af50381ab2247e083b80d32d5c0aba91c272ca9ff729b6a0e523"
+  "model.safetensors.index.json 106278 -"
+  "config.json 4608 -"
+  "generation_config.json 161 -"
+  "quantization_config.json 1654 -"
+  "chat_template.jinja 3916 -"
+  "tokenizer_config.json 1168 -"
+  "preprocessor_config.json 443 -"
+  "processor_config.json 1191 -"
+  "README.md 8448 -"
+)
+
+mkdir -p "$DEST"
+status=0
+for entry in "${FILES[@]}"; do
+  read -r name size sha <<<"$entry"
+  f="$DEST/$name"
+  have=0
+  if [[ -f "$f" ]] && [[ "$(stat -c%s "$f")" == "$size" ]]; then
+    if [[ "$sha" == "-" ]] || sha256sum -c --quiet <(printf '%s  %s\n' "$sha" "$f"); then
+      have=1
+    fi
+  fi
+  if (( have )); then
+    echo "OK        $name"
+    continue
+  fi
+  if (( VERIFY_ONLY )); then
+    echo "MISSING   $name"
+    status=1
+    continue
+  fi
+  echo "FETCHING  $name"
+  tmp="$f.part"
+  # no -C resume; -f + --fail-with-body so an HTTP error is not a success
+  curl -fsSL --fail-with-body -o "$tmp" "$BASE/$name"
+  mv "$tmp" "$f"
+  if [[ "$(stat -c%s "$f")" != "$size" ]]; then
+    echo "SIZE FAIL $name ($(stat -c%s "$f") != $size)"
+    status=1
+    continue
+  fi
+  if [[ "$sha" != "-" ]] && ! sha256sum -c --quiet <(printf '%s  %s\n' "$sha" "$f"); then
+    echo "SHA FAIL  $name"
+    status=1
+    continue
+  fi
+  echo "VERIFIED  $name"
+done
+if (( status == 0 )); then
+  echo "DOWNLOAD_DONE $DEST"
+else
+  echo "DOWNLOAD_FAILED $DEST" >&2
+fi
+exit "$status"
@@ -0,0 +1,86 @@
+# Subsurface Aquifer Monitoring Network - System Design Document (Revision 7)
+
+This document describes the deployment of a subsurface aquifer monitoring network for the Cascadia sedimentary basin. It is written for the operations crew and the regional water authority. Every sensor node, relay, aggregation service, calibration procedure, and failure mode below carries a stable alphanumeric identifier that appears in every table and in the field notes.
+
+## Sensor nodes
+Node SN-000-08B9 sits in a discharge lens at 359.0 m below datum and reports a salinity of 2.38 ppt with the fractured interval reading 5.40 degC at dawn and the wellhead porous 1.72 m3/d while the electrode pair returns 3157 uS/cm.
+Node SN-001-0A65 sits in a percolate lens at 25.9 m below datum and steps down to a salinity of 2.13 ppt with the diapir interval reading 8.55 degC at dawn and the wellhead silty 0.67 m3/d while the electrode pair returns 979 uS/cm.
+Node SN-002-0571 sits in a diapir lens at 287.8 m below datum and peaks at a salinity of 7.63 ppt with the karst interval reading 4.26 degC at dawn and the wellhead spooned 11.50 m3/d while the electrode pair returns 2987 uS/cm.
+Node SN-003-0CE9 sits in a confining lens at 169.5 m below datum and logs a salinity of 0.96 ppt with the alluvial interval reading 5.03 degC at dawn and the wellhead salinized 12.08 m3/d while the electrode pair returns 2607 uS/cm.
+Node SN-004-0E2A sits in a freshened lens at 70.4 m below datum and settles near a salinity of 8.68 ppt with the spooned interval reading 4.88 degC at dawn and the wellhead siphon 4.38 m3/d while the electrode pair returns 2707 uS/cm.
+Node SN-005-0DDD sits in a percolate lens at 44.6 m below datum and reports a salinity of 6.25 ppt with the alluvial interval reading 10.85 degC at dawn and the wellhead brackish 13.99 m3/d while the electrode pair returns 3618 uS/cm.
+Node SN-006-0EFC sits in a till lens at 309.5 m below datum and reports a salinity of 3.49 ppt with the lacustrine interval reading 7.38 degC at dawn and the wellhead spooned 3.21 m3/d while the electrode pair returns 1253 uS/cm.
+Node SN-007-0B02 sits in a recharge lens at 262.0 m below datum and steps down to a salinity of 2.38 ppt with the discharge interval reading 8.18 degC at dawn and the wellhead perched 4.05 m3/d while the electrode pair returns 3900 uS/cm.
+Node SN-008-0448 sits in a karst lens at 406.5 m below datum and reports a salinity of 7.32 ppt with the upconing interval reading 6.17 degC at dawn and the wellhead fluvial 0.75 m3/d while the electrode pair returns 1448 uS/cm.
+Node SN-009-043C sits in a alluvial lens at 159.3 m below datum and flags a salinity of 6.20 ppt with the percolate interval reading 7.60 degC at dawn and the wellhead brackish 13.01 m3/d while the electrode pair returns 2025 uS/cm.
+Node SN-010-0FF5 sits in a karst lens at 264.2 m below datum and flags a salinity of 7.05 ppt with the freshened interval reading 7.88 degC at dawn and the wellhead siphon 8.41 m3/d while the electrode pair returns 1635 uS/cm.
+Node SN-011-098A sits in a drain lens at 365.7 m below datum and reports a salinity of 8.11 ppt with the wedge interval reading 5.51 degC at dawn and the wellhead glacial 2.52 m3/d while the electrode pair returns 2916 uS/cm.
+Node SN-012-0AE8 sits in a till lens at 259.6 m below datum and steps down to a salinity of 9.13 ppt with the recharge interval reading 11.60 degC at dawn and the wellhead spooned 0.46 m3/d while the electrode pair returns 3077 uS/cm.
+Node SN-013-0B83 sits in a salinized lens at 171.2 m below datum and peaks at a salinity of 2.83 ppt with the lacustrine interval reading 5.56 degC at dawn and the wellhead discharge 0.35 m3/d while the electrode pair returns 3083 uS/cm.
+Node SN-014-032E sits in a drain lens at 249.6 m below datum and flags a salinity of 1.09 ppt with the salinized interval reading 9.58 degC at dawn and the wellhead perched 12.00 m3/d while the electrode pair returns 2221 uS/cm.
+Node SN-015-004F sits in a aeolian lens at 264.4 m below datum and reports a salinity of 7.34 ppt with the salinized interval reading 8.76 degC at dawn and the wellhead perched 0.31 m3/d while the electrode pair returns 1483 uS/cm.
+Node SN-016-07C6 sits in a aeolian lens at 155.9 m below datum and reports a salinity of 0.64 ppt with the mineralized interval reading 11.75 degC at dawn and the wellhead diluted 13.46 m3/d while the electrode pair returns 524 uS/cm.
+Node SN-017-06C5 sits in a salinized lens at 72.1 m below datum and averages a salinity of 4.52 ppt with the siphon interval reading 8.93 degC at dawn and the wellhead confining 3.98 m3/d while the electrode pair returns 3687 uS/cm.
+Node SN-018-0E64 sits in a siphon lens at 334.9 m below datum and steps down to a salinity of 6.73 ppt with the salinized interval reading 7.63 degC at dawn and the wellhead freshened 9.64 m3/d while the electrode pair returns 1681 uS/cm.
+Node SN-019-0122 sits in a spooned lens at 117.2 m below datum and logs a salinity of 3.24 ppt with the fractured interval reading 9.26 degC at dawn and the wellhead karst 3.50 m3/d while the electrode pair returns 1065 uS/cm.
+Node SN-020-021D sits in a karst lens at 414.3 m below datum and recovers to a salinity of 0.76 ppt with the glacial interval reading 6.25 degC at dawn and the wellhead silty 9.60 m3/d while the electrode pair returns 1041 uS/cm.
+Node SN-021-06C6 sits in a percolate lens at 379.2 m below datum and drifts to a salinity of 7.61 ppt with the wedge interval reading 5.84 degC at dawn and the wellhead karst 1.65 m3/d while the electrode pair returns 1913 uS/cm.
+Node SN-022-0DC9 sits in a spooned lens at 327.1 m below datum and reports a salinity of 9.25 ppt with the freshened interval reading 5.05 degC at dawn and the wellhead silty 5.90 m3/d while the electrode pair returns 1544 uS/cm.
+Node SN-023-0717 sits in a karst lens at 222.0 m below datum and steps down to a salinity of 4.02 ppt with the alluvial interval reading 6.60 degC at dawn and the wellhead siphon 3.77 m3/d while the electrode pair returns 3892 uS/cm.
+Node SN-024-02A9 sits in a fractured lens at 479.7 m below datum and drifts to a salinity of 7.88 ppt with the silty interval reading 12.53 degC at dawn and the wellhead discharge 13.18 m3/d while the electrode pair returns 3592 uS/cm.
+Node SN-025-0C8F sits in a wedge lens at 434.3 m below datum and holds a salinity of 1.63 ppt with the alluvial interval reading 4.22 degC at dawn and the wellhead till 5.73 m3/d while the electrode pair returns 3905 uS/cm.
+Node SN-026-04BF sits in a colluvial lens at 460.0 m below datum and averages a salinity of 9.36 ppt with the diapir interval reading 8.98 degC at dawn and the wellhead brackish 10.29 m3/d while the electrode pair returns 802 uS/cm.
+Node SN-027-0DA4 sits in a silty lens at 265.7 m below datum and steps down to a salinity of 7.06 ppt with the percolate interval reading 4.69 degC at dawn and the wellhead freshened 8.42 m3/d while the electrode pair returns 2201 uS/cm.
+Node SN-028-01EB sits in a perched lens at 49.5 m below datum and recovers to a salinity of 1.83 ppt with the silty interval reading 9.32 degC at dawn and the wellhead drain 9.69 m3/d while the electrode pair returns 1126 uS/cm.
+Node SN-029-042C sits in a karst lens at 30.6 m below datum and averages a salinity of 0.86 ppt with the percolate interval reading 9.85 degC at dawn and the wellhead recharge 8.16 m3/d while the electrode pair returns 1452 uS/cm.
+
+## Relays
+Relay RL-00-02181 reburses the upconing trunk from the brackish margin and peaks at a round-trip time of 131.8 ms with a frame loss of 1.04 percent across the night window.
+Relay RL-01-00262 reburses the spooned trunk from the fluvial margin and holds a round-trip time of 372.2 ms with a frame loss of 3.65 percent across the night window.
+Relay RL-02-03691 reburses the recharge trunk from the percolate margin and steps down to a round-trip time of 398.5 ms with a frame loss of 0.29 percent across the night window.
+Relay RL-03-0E117 reburses the lacustrine trunk from the confining margin and averages a round-trip time of 373.9 ms with a frame loss of 3.44 percent across the night window.
+Relay RL-04-0B41B reburses the aeolian trunk from the colluvial margin and peaks at a round-trip time of 70.7 ms with a frame loss of 3.25 percent across the night window.
+Relay RL-05-0D12D reburses the recharge trunk from the diluted margin and steps down to a round-trip time of 264.7 ms with a frame loss of 0.03 percent across the night window.
+Relay RL-06-043B4 reburses the colluvial trunk from the upconing margin and reports a round-trip time of 49.5 ms with a frame loss of 3.42 percent across the night window.
+Relay RL-07-09AD6 reburses the fractured trunk from the freshened margin and averages a round-trip time of 225.3 ms with a frame loss of 1.06 percent across the night window.
+Relay RL-08-04394 reburses the brackish trunk from the fluvial margin and steps down to a round-trip time of 88.6 ms with a frame loss of 2.47 percent across the night window.
+Relay RL-09-017A0 reburses the wedge trunk from the lacustrine margin and drifts to a round-trip time of 363.0 ms with a frame loss of 3.30 percent across the night window.
+Relay RL-10-0217D reburses the mineralized trunk from the lacustrine margin and peaks at a round-trip time of 26.2 ms with a frame loss of 1.30 percent across the night window.
+Relay RL-11-08F97 reburses the perched trunk from the freshened margin and logs a round-trip time of 181.8 ms with a frame loss of 2.75 percent across the night window.
+Relay RL-12-02628 reburses the fractured trunk from the glacial margin and steps down to a round-trip time of 378.5 ms with a frame loss of 2.69 percent across the night window.
+Relay RL-13-06E06 reburses the silty trunk from the mineralized margin and flags a round-trip time of 153.4 ms with a frame loss of 2.15 percent across the night window.
+
+## Aggregation services
+Aggregator AG-00-3BB logs a queue depth of 5.2 messages and a histogram bucket of 6.82 seconds before the dedupe pass settles the stream for the next rotation.
+Aggregator AG-01-0FF reports a queue depth of 86.4 messages and a histogram bucket of 4.10 seconds before the dedupe pass settles the stream for the next rotation.
+Aggregator AG-02-37F drifts to a queue depth of 34.7 messages and a histogram bucket of 10.25 seconds before the dedupe pass settles the stream for the next rotation.
+Aggregator AG-03-3B3 averages a queue depth of 93.5 messages and a histogram bucket of 13.59 seconds before the dedupe pass settles the stream for the next rotation.
+Aggregator AG-04-33E flags a queue depth of 83.2 messages and a histogram bucket of 17.58 seconds before the dedupe pass settles the stream for the next rotation.
+Aggregator AG-05-2F2 settles near a queue depth of 84.8 messages and a histogram bucket of 0.84 seconds before the dedupe pass settles the stream for the next rotation.
+Aggregator AG-06-2AD averages a queue depth of 75.4 messages and a histogram bucket of 7.65 seconds before the dedupe pass settles the stream for the next rotation.
+Aggregator AG-07-06E drifts to a queue depth of 26.5 messages and a histogram bucket of 14.26 seconds before the dedupe pass settles the stream for the next rotation.
+Aggregator AG-08-0E3 averages a queue depth of 83.9 messages and a histogram bucket of 15.51 seconds before the dedupe pass settles the stream for the next rotation.
+Aggregator AG-09-138 averages a queue depth of 78.6 messages and a histogram bucket of 8.50 seconds before the dedupe pass settles the stream for the next rotation.
+
+## Calibration procedures
+Procedure CP-00-047 steps the probe through a two-point rinse, then applies a trim of 0.2084 at the 66 milli-bar mark and records a residual of 0.02052 before the next field seal is crimped.
+Procedure CP-01-0A4 steps the probe through a two-point rinse, then applies a trim of 0.7793 at the 96 milli-bar mark and records a residual of 0.01468 before the next field seal is crimped.
+Procedure CP-02-054 steps the probe through a two-point rinse, then applies a trim of 0.4634 at the 68 milli-bar mark and records a residual of 0.04234 before the next field seal is crimped.
+Procedure CP-03-02D steps the probe through a two-point rinse, then applies a trim of 0.8459 at the 12 milli-bar mark and records a residual of 0.04855 before the next field seal is crimped.
+Procedure CP-04-098 steps the probe through a two-point rinse, then applies a trim of 0.5267 at the 97 milli-bar mark and records a residual of 0.00287 before the next field seal is crimped.
+Procedure CP-05-09B steps the probe through a two-point rinse, then applies a trim of 0.3968 at the 72 milli-bar mark and records a residual of 0.01637 before the next field seal is crimped.
+Procedure CP-06-030 steps the probe through a two-point rinse, then applies a trim of 0.8860 at the 12 milli-bar mark and records a residual of 0.04508 before the next field seal is crimped.
+Procedure CP-07-0EC steps the probe through a two-point rinse, then applies a trim of 0.2367 at the 70 milli-bar mark and records a residual of 0.00108 before the next field seal is crimped.
+Procedure CP-08-0BC steps the probe through a two-point rinse, then applies a trim of 0.7278 at the 68 milli-bar mark and records a residual of 0.04707 before the next field seal is crimped.
+Procedure CP-09-0AA steps the probe through a two-point rinse, then applies a trim of 0.6069 at the 37 milli-bar mark and records a residual of 0.00443 before the next field seal is crimped.
+
+## Failure modes
+Mode FM-00-27 declares a fluvial excursion when two independent probes disagree beyond the band for three rotations; the fleet reports a mean time between failures of 36902 hours for this class of excursion.
+Mode FM-01-13 declares a drain excursion when two independent probes disagree beyond the band for three rotations; the fleet peaks at a mean time between failures of 29042 hours for this class of excursion.
+Mode FM-02-14 declares a till excursion when two independent probes disagree beyond the band for three rotations; the fleet peaks at a mean time between failures of 26976 hours for this class of excursion.
+Mode FM-03-0C declares a diapir excursion when two independent probes disagree beyond the band for three rotations; the fleet drifts to a mean time between failures of 22578 hours for this class of excursion.
+Mode FM-04-39 declares a perched excursion when two independent probes disagree beyond the band for three rotations; the fleet recovers to a mean time between failures of 27383 hours for this class of excursion.
+Mode FM-05-19 declares a siphon excursion when two independent probes disagree beyond the band for three rotations; the fleet logs a mean time between failures of 23152 hours for this class of excursion.
+
+## End of document
+Summarize the network's dominant risk in one paragraph, then list the three node identifiers whose readings most limit the model this month.
@@ -0,0 +1,141 @@
+#!/usr/bin/env python3
+"""autoresearch-bench.py — deterministic greedy battery for frank-large.
+
+Runs ON Dankclaw (no LAN jitter) against the candidate container on :7770.
+Fixed embedded corpus, temperature 0, streaming so decode speed is measured
+between real token arrivals (not request totals). Median of REPS samples per
+metric is the reported value; medians are far more stable than means under
+GPU thermal drift.
+
+Metrics emitted (stdout, one per line):
+  METRIC decode_tokens_per_sec=<float>   primary, higher better
+  METRIC prefill_tokens_per_sec=<float>  secondary
+  METRIC ttft_ms=<float>                 secondary, lower better
+Exit non-zero if the endpoint is unhealthy or the served alias is wrong.
+"""
+import argparse
+import json
+import statistics
+import sys
+import time
+import urllib.request
+
+# Fixed corpus: deterministic, ~28k tokens after tokenization (same scale as
+# the 2026-10-01 bench window). Built by repeating a fixed paragraph block a
+# fixed number of times - NO random, NO clock, NO network.
+_BLOCK = (
+    "The rendering pipeline walks the scene graph once per frame, resolving "
+    "material bindings into descriptor sets before any draw call is recorded. "
+    "Because the allocator recycles command buffers only after their fence "
+    "signals, a submission that outlives its frame must pin the pool entry; "
+    "otherwise the next frame overwrites recorded commands in flight. The "
+    "validation layer flags exactly this class of lifetime bug, but only for "
+    "buffers it can see - GPU-side timestamps and queue-family ownership "
+    "transfers hide in the driver's own bookkeeping, which is why release "
+    "builds must lean on explicit barriers and why a frame-time regression "
+    "often shows up as corruption one frame later than its cause.\n"
+)
+LONG_CTX = _BLOCK * 220  # ~28k-token prefill, matching the deployed workload
+_LONG = LONG_CTX + "\nNow, in one focused paragraph (80-150 words), explain " \
+    "which lifetime bugs this pipeline can still suffer and what explicit " \
+    "mitigations the text implies."
+_SHORT = "Design a rate limiter for a homelab reverse proxy that must not " \
+    "lock out the LAN. Think it through, then give the algorithm."
+
+REPS = 3
+MAX_TOKENS = 256
+REQ_TIMEOUT = 900
+
+
+def stream_completion(url: str, model: str, prompt: str):
+    """Greedy streaming completion.
+
+    Returns (ttft_s, decode_tps, prefill_tps, prompt_tokens, n_tokens).
+    ttft is RELATIVE to request send: time.perf_counter()'s epoch is host
+    boot, and returning the raw stamp read as an 11-day first token.
+    """
+    body = json.dumps({
+        "model": model,
+        "messages": [{"role": "user", "content": prompt}],
+        "max_tokens": MAX_TOKENS,
+        "temperature": 0,
+        "stream": True,
+        "stream_options": {"include_usage": True},
+    }).encode()
+    req = urllib.request.Request(
+        url.rstrip("/") + "/v1/chat/completions", data=body,
+        headers={"Content-Type": "application/json"})
+    t0 = time.perf_counter()
+    first = None
+    stamps = []
+    usage = {}
+    with urllib.request.urlopen(req, timeout=REQ_TIMEOUT) as resp:
+        for raw in resp:
+            line = raw.decode(errors="replace").strip()
+            if not line.startswith("data:"):
+                continue
+            payload = line[5:].strip()
+            if payload == "[DONE]":
+                break
+            chunk = json.loads(payload)
+            if chunk.get("usage"):
+                usage = chunk["usage"]
+            for ch in chunk.get("choices", []):
+                d = ch.get("delta") or {}
+                # reasoning AND content deltas both arrive token by token;
+                # count either so greedy text length drives the rate.
+                # (llama.cpp names the thinking field reasoning_content.)
+                if d.get("content") or d.get("reasoning") or d.get("reasoning_content"):
+                    now = time.perf_counter()
+                    if first is None:
+                        first = now
+                    stamps.append(now)
+    if first is None or len(stamps) < 2:
+        # A zero-argmax degenerate answer would poison the median; raise.
+        raise RuntimeError(f"stream produced {len(stamps)} tokens")
+    ttft_s = first - t0
+    decode_s = stamps[-1] - stamps[0]
+    decode_tps = (len(stamps) - 1) / decode_s if decode_s > 0 else 0.0
+    prompt_tokens = usage.get("prompt_tokens") or 0
+    prefill_tps = prompt_tokens / ttft_s if prompt_tokens else 0.0
+    return ttft_s, decode_tps, prefill_tps, prompt_tokens, len(stamps)
+
+
+def main():
+    ap = argparse.ArgumentParser()
+    ap.add_argument("--url", default="http://127.0.0.1:7770")
+    ap.add_argument("--model", default="frank-large")
+    args = ap.parse_args()
+
+    with urllib.request.urlopen(args.url.rstrip("/") + "/v1/models",
+                                timeout=10) as r:
+        ids = [m["id"] for m in json.load(r)["data"]]
+    if args.model not in ids:
+        print(f"FATAL: served ids {ids}, expected {args.model}",
+              file=sys.stderr)
+        sys.exit(2)
+
+    # Warmup (compile/graph warm + allocator settle), NOT measured.
+    stream_completion(args.url, args.model, _SHORT)
+
+    decodes, prefills, ttfts = [], [], []
+    for rep in range(REPS):
+        for name, prompt in (("long", _LONG), ("short", _SHORT)):
+            ttft, dt, pt, ptk, ntok = stream_completion(
+                args.url, args.model, prompt)
+            decodes.append(dt)
+            ttfts.append(ttft * 1000.0)
+            if name == "long":
+                prefills.append(pt)
+            print(f"  rep{rep} {name:5s} ttft={ttft*1000:8.1f}ms "
+                  f"prefill={pt:8.1f} t/s decode={dt:6.2f} t/s "
+                  f"({ntok} tok, prompt {ptk})", file=sys.stderr)
+
+    med = lambda xs: statistics.median(xs)
+    print(f"METRIC decode_tokens_per_sec={med(decodes):.2f}")
+    print(f"METRIC prefill_tokens_per_sec={med(prefills):.1f}")
+    print(f"METRIC ttft_ms={med(ttfts):.1f}")
+
+
+if __name__ == "__main__":
+    main()
@@ -0,0 +1,176 @@
+#!/usr/bin/env bash
+# frank-primary-arms.sh — run a SEQUENCE of frank-primary bench arms and keep
+# every arm's evidence. Runs on the operator box; each arm is one full window
+# of ./autoresearch.sh (stop+mask -> boot the arm as the service -> measure ->
+# restore production and verify), so the fleet is only ever down for one arm at
+# a time and the production unit is back up between arms.
+#
+# An arm is the repo's own frank-primary.env plus an OVERLAY of engine knobs
+# appended after it (later assignment wins when the serve script sources it),
+# staged through FRANK_PRIMARY_AR_ENV_FILE so the checkout is never mutated and
+# each arm gets its own fingerprint (the memo can never answer one arm with
+# another arm's numbers).
+#
+# Usage:  scripts/autoresearch/frank-primary-arms.sh [arm ...]
+#         scripts/autoresearch/frank-primary-arms.sh --list
+# Output, under /tmp/frank-primary-arms-<UTC stamp>/ (override with ARMS_DIR):
+#
+#   <arm>.out        full autoresearch output (metrics + engine evidence)
+#   <arm>.env        the exact staged env
+#   <arm>.rc         window exit code
+#   summary.tsv      arm, rc, decode_tps, decode_tps_t0.7, kv_tokens, boot_s
+#
+# Exit 0 when every arm produced metrics; 1 if any arm failed (a failed boot is
+# a RESULT — the window still restored production, and the rc says which arm).
+set -uo pipefail
+
+SCRIPT_DIR="$(cd -- "$(dirname -- "${BASH_SOURCE[0]}")" && pwd)"
+REPO="$(cd -- "$SCRIPT_DIR/../.." && pwd)"
+BASE_ENV="$REPO/scripts/frank-primary.env"
+ARMS_DIR="${ARMS_DIR:-/tmp/frank-primary-arms-$(date -u +%Y%m%dT%H%M%SZ)}"
+
+# --- arms ------------------------------------------------------------------
+# baseline: the shipped shape E, measured with THIS harness so the comparison is
+# matched (the ~29.6 t/s figure in the docs is an older shape, and no shape-E
+# decode number exists yet).
+OVER_baseline=()
+# B: MTP3 with the vendor's probabilistic draft sampling, a capture ladder whose
+# step is num_speculative_tokens + 1 = 4, and the sampling override. Precise
+# memory stays ON — this arm is simultaneously the MTP test and the test of
+# whether MTP's drafter workspace and the fitted 5.25 GiB overhead can coexist.
+SPEC3='{"method":"mtp","num_speculative_tokens":3,"draft_sample_method":"probabilistic"}'
+LADDER3='{"cudagraph_capture_sizes":[4,8,12],"max_cudagraph_capture_size":12,"inductor_compile_config":{"combo_kernels":false,"benchmark_combo_kernel":false}}'
+# The value must arrive in the staged env file SINGLE-QUOTED: sourcing an env
+# file performs quote removal, so an unquoted {"temperature":0.7} would reach
+# the engine as {temperature:0.7}.
+GEN07='{"temperature":0.7}'
+OVER_B=(
+    "FRANK_PRIMARY_SPEC='$SPEC3'"
+    "FRANK_PRIMARY_COMPCONFIG='$LADDER3'"
+    'FRANK_PRIMARY_GEN_CONFIG='"'$GEN07'"
+)
+# B2: identical to B with the default (7.60 GiB) overhead, i.e. precise memory
+# OFF. It is the fallback if B dies at graph capture, and it prices the arena
+# that precise memory buys (7,522 slots -> ~6,496).
+OVER_B2=(
+    "FRANK_PRIMARY_SPEC='$SPEC3'"
+    "FRANK_PRIMARY_COMPCONFIG='$LADDER3'"
+    'FRANK_PRIMARY_GEN_CONFIG='"'$GEN07'"
+    'FRANK_PRIMARY_PRECISE_MEMORY='
+)
+# C: the vendor's concurrency shape — SEQS 16 with the [4..32] ladder and util
+# 0.95. This is also the direct test of the theory that pinning the capture
+# ladder unlocks SEQS > 3 (the SEQS=2 failure was query_start_loc specialised to
+# the constant 3, which is NOT in the default ladder; 4 is).
+LADDER16='{"cudagraph_capture_sizes":[4,8,12,16,20,24,28,32],"max_cudagraph_capture_size":32,"inductor_compile_config":{"combo_kernels":false,"benchmark_combo_kernel":false}}'
+OVER_C=(
+    "FRANK_PRIMARY_SPEC='$SPEC3'"
+    "FRANK_PRIMARY_COMPCONFIG='$LADDER16'"
+    'FRANK_PRIMARY_GEN_CONFIG='"'$GEN07'"
+    'FRANK_PRIMARY_SEQS=16'
+    'FRANK_PRIMARY_MNBT=4096'
+    'FRANK_PRIMARY_UTIL=0.95'
+)
+
+# D/E: the CONTEXT arms. Measured 2026-10-02 (arms run 20261002T233855Z): the
+# KV pool is a hard byte request divided by a PER-TOKEN PRICE, and MTP raises
+# that price because the drafter keeps its own KV per position:
+#   no spec: 3.15e9 / 398,658 tokens =  7,901 B/token  (1.99x of 200,704)
+#   MTP3:    3.15e9 / 313,002 tokens = 10,064 B/token  (1.56x)
+# So arm C lost a fifth of the context to the drafter, not to util. Frank's
+# contract is >=401,400 tokens over >=2 seqs = 2.00x of 200,704, which at
+# 10,064 B/token costs 4.04e9 -> ask 4.05e9 (+0.83 GiB/rank).
+# D is arm B with EXACTLY ONE variable changed (the KV bytes), so its result
+# says whether the 4k-depth collapse (B 23.09 vs baseline 33.99) was KV or the
+# batch shape.
+KV401K='FRANK_PRIMARY_KV_BYTES=4050000000'
+OVER_D=(
+    "FRANK_PRIMARY_SPEC='$SPEC3'"
+    "FRANK_PRIMARY_COMPCONFIG='$LADDER3'"
+    'FRANK_PRIMARY_GEN_CONFIG='"'$GEN07'"
+    "$KV401K"
+)
+# E: D with C's max_num_batched_tokens (4096 — C is the only arm that did NOT
+# collapse at 4k depth: 33.15 where B gave 23.09) and util 0.99, the value
+# Frank asked about. 0.99 is affordable on paper — production at util 0.98
+# measures 30.37/34.21 GiB per card, 3.58 GiB physically unused — but it leaves
+# only 0.32 GiB above the allowance while the CUDA-graph cost is an ANALYTIC
+# estimate (C's was 1.47 GiB over 24 graphs), so it is measured, not assumed.
+OVER_E=(
+    "FRANK_PRIMARY_SPEC='$SPEC3'"
+    "FRANK_PRIMARY_COMPCONFIG='$LADDER3'"
+    'FRANK_PRIMARY_GEN_CONFIG='"'$GEN07'"
+    "$KV401K"
+    'FRANK_PRIMARY_MNBT=4096'
+    'FRANK_PRIMARY_UTIL=0.99'
+)
+
+arm_overlay() {
+    case "$1" in
+        baseline) ;;                       # the shipped env, no overlay
+        B)  printf '%s\n' "${OVER_B[@]}" ;;
+        B2) printf '%s\n' "${OVER_B2[@]}" ;;
+        C)  printf '%s\n' "${OVER_C[@]}" ;;
+        D)  printf '%s\n' "${OVER_D[@]}" ;;
+        E)  printf '%s\n' "${OVER_E[@]}" ;;
+        *) return 1 ;;
+    esac
+}
+if [ "${1:-}" = "--list" ]; then
+    printf 'baseline B B2 C D E\n'
+    exit 0
+fi
+
+[ -f "$BASE_ENV" ] || { echo "[arms] missing $BASE_ENV" >&2; exit 1; }
+if [ "$#" -gt 0 ]; then ARMS=("$@"); else ARMS=(baseline B B2 C); fi
+mkdir -p "$ARMS_DIR"
+echo "[arms] dir $ARMS_DIR  arms: ${ARMS[*]}" >&2
+
+: >"$ARMS_DIR/summary.tsv"
+printf 'arm\trc\tdecode_tps\tdecode_tps_t0.7\tkv_tokens\tboot_s\n' >>"$ARMS_DIR/summary.tsv"
+overall=0
+
+for arm in "${ARMS[@]}"; do
+    if ! arm_overlay "$arm" >"$ARMS_DIR/$arm.overlay"; then
+        echo "[arms] unknown arm: $arm" >&2
+        overall=1
+        continue
+    fi
+    { cat "$BASE_ENV"; echo; echo "# --- arm $arm overlay ($(date -u +%FT%TZ)) ---"
+      arm_overlay "$arm"; } >"$ARMS_DIR/$arm.env"
+    # Gate the staged env BEFORE burning a ~20-minute window: every knob that
+    # holds JSON must still be one valid JSON object after the file is SOURCED
+    # (that is the step that silently eats unquoted braces).
+    if ! bash -c 'set -a; . "$1" >/dev/null; set +a
+        for k in FRANK_PRIMARY_SPEC FRANK_PRIMARY_COMPCONFIG FRANK_PRIMARY_GEN_CONFIG; do
+            v="${!k:-}"
+            [ -n "$v" ] || continue
+            case "$v" in
+                "{"*) python3 -c "import json,sys; json.loads(sys.argv[1])" "$v" \
+                        || { echo "[arms] $k is not valid JSON: $v" >&2; exit 1; } ;;
+                *) echo "[arms] $k is not a JSON object: $v" >&2; exit 1 ;;
+            esac
+        done' _ "$ARMS_DIR/$arm.env"; then
+        echo "[arms] $arm: staged env rejected, window not opened" >&2
+        echo 99 >"$ARMS_DIR/$arm.rc"
+        overall=1
+        continue
+    fi
+
+    echo "[arms] === $arm: opening window $(date -u +%H:%M:%S)Z ===" >&2
+    FRANK_PRIMARY_AR_ENV_FILE="$ARMS_DIR/$arm.env" \
+        "$REPO/autoresearch.sh" >"$ARMS_DIR/$arm.out" 2>&1
+    rc=$?
+    echo "$rc" >"$ARMS_DIR/$arm.rc"
+    echo "[arms] === $arm: rc=$rc $(date -u +%H:%M:%S)Z ===" >&2
+    grep -E '^METRIC ' "$ARMS_DIR/$arm.out" | sed 's/^METRIC //' >"$ARMS_DIR/$arm.metrics" || true
+    metric() { local v; v="$(sed -n "s/^$1=//p" "$ARMS_DIR/$arm.metrics" | tail -1)"; printf '%s' "${v:-$2}"; }
+    printf '%s\t%s\t%s\t%s\t%s\t%s\n' "$arm" "$rc" \
+        "$(metric decode_tps FAIL)" "$(metric decode_tps_t0.7 FAIL)" \
+        "$(metric kv_tokens ?)" "$(metric boot_seconds ?)" >>"$ARMS_DIR/summary.tsv"
+    [ "$rc" -ne 0 ] && overall=1
+done
+
+echo "[arms] summary:" >&2
+cat "$ARMS_DIR/summary.tsv" >&2
+exit "$overall"
@@ -0,0 +1,216 @@
+#!/usr/bin/env python3
+"""frank-primary decode measurement — runs INSIDE the engine container.
+
+The method is the fleet's established one (the 2026-09-29 / 2026-10-01
+tuning campaigns, /home/dank/mtp_measure.py): SSE chunk counting is invalid
+under speculative decoding because rejected drafts change the accounting, so
+decode is measured from the server's own usage counters:
+
+    decode_tok_s = completion_tokens / (wall - ttft)
+
+with ttft = wall time to the FIRST content-bearing chunk (thinking arrives in
+`reasoning`/`delta.reasoning` on this stack and is excluded from the decode
+window — and is deterministic per prompt at temperature 0, so the exclusion
+cancels in any A/B on the same workload).
+
+Workload (fixed, deterministic, offline):
+  * prompts built from a large fixed-seed shuffled word list so tokens rarely
+    repeat (no n-gram/prefix predictability of the text itself), sized with
+    the server's own /v1/messages/count_tokens counter
+  * depths 115 / 1056 / 4096 target prompt tokens
+  * one 64-token warm pass per depth (warms the expert arena), then
+    max_tokens=256, 3 runs, median — repeated for every temperature in AR_TEMPS
+    (default 0 and 0.7). The prompt workload is deterministic (fixed seed, no
+    network, no time-of-day dependence); at temperature > 0 the generated text
+    is not, but the measured window is bounded by max_tokens, so the rate stays
+    comparable and the non-determinism is exactly the sampling that a
+    speculative-decode arm must survive.
+
+Env:
+  AR_BASE             endpoint (default http://127.0.0.1:7770)
+  AR_MODEL            served name (default frank-primary)
+  AR_PROMPT_CEILING   hard prompt ceiling in tokens (default 100000)
+  AR_TEMPS            comma-separated sampling temperatures (default "0,0.7")
+
+Output: progress on stderr, ONE json object on stdout (the last line).
+Exit codes: 0 ok, 2 endpoint unhealthy / no usable completion, 3 bad usage.
+"""
+import json
+import os
+import random
+import statistics
+import sys
+import time
+import urllib.error
+import urllib.request
+
+BASE = os.environ.get("AR_BASE", "http://127.0.0.1:7770")
+MODEL = os.environ.get("AR_MODEL", "frank-primary")
+PROMPT_CEILING = int(os.environ.get("AR_PROMPT_CEILING", "100000"))
+DEPTHS = (115, 1056, 4096)
+WARM_TOKENS = 64
+MAX_TOKENS = 256
+RUNS = 3
+# Temperature sweep. The unsuffixed keys are temperature 0 — byte-comparable to
+# every historical measurement — and AR_TEMPS adds more. The second one matters
+# for speculative decode: draft ACCEPTANCE is sampling-dependent (a greedy draft
+# matches a greedy target exactly and is rejected more under a non-greedy
+# target), so an MTP arm judged only at temp 0 would be judged on half the
+# contract. Production traffic is non-zero temperature (the payload's
+# generation_config says 1.0; an override to 0.7 is under test).
+TEMPS = tuple(float(x) for x in os.environ.get("AR_TEMPS", "0,0.7").split(","))
+
+# 43-word x 12 = 516-word vocabulary, seed 1234 — the campaign's exact list
+# and seed, so prompts are byte-identical to the historical measurements.
+_VOCAB = ("alpha bravo charlie delta echo foxtrot golf hotel india juliet kilo lima "
+          "mike november oscar papa quebec romeo sierra tango uniform victor whiskey "
+          "xray yankee zulu cardinal ordinal topology kernel scheduler resident "
+          "unified aperture expert arena slab offload prefetch gemm triton quant "
+          "tensor shard checkpoint manifest embed router cache page table window "
+          "gradient sparse dense moe attention head dim block stream decode prefill "
+          "bandwidth latency throughput residency eviction tier swap pin memlock").split()
+_R = random.Random(1234)
+_ORDER = list(range(len(_VOCAB)))
+_R.shuffle(_ORDER)
+
+
+def log(msg):
+    print("[measure] %s" % msg, file=sys.stderr, flush=True)
+
+
+def post_json(path, body, timeout):
+    req = urllib.request.Request(
+        BASE + path, data=json.dumps(body).encode(),
+        headers={"Content-Type": "application/json"})
+    with urllib.request.urlopen(req, timeout=timeout) as r:
+        return json.load(r)
+
+
+def count_tokens(text):
+    """Token count without a prefill, from the server's own counter."""
+    body = {"model": MODEL,
+            "messages": [{"role": "user", "content": text}]}
+    r = post_json("/v1/messages/count_tokens", body, timeout=300)
+    n = r.get("input_tokens") or r.get("count") or r.get("token_count")
+    if not n:
+        raise RuntimeError("count_tokens returned no count: %r" % (r,))
+    return int(n)
+
+
+def words(n):
+    return " ".join(_VOCAB[_ORDER[i % len(_VOCAB)]] for i in range(n))
+
+
+def calibrate():
+    """Measure words->tokens from the server's own counter."""
+    return 2000.0 / count_tokens(words(2000))
+
+
+def prompt_for(target_tokens, wpt):
+    n = max(1, int(target_tokens * wpt))
+    text = words(n)
+    pt = count_tokens(text)
+    while pt > PROMPT_CEILING:
+        n = max(1, int(n * (PROMPT_CEILING / pt) * 0.98))
+        text = words(n)
+        pt = count_tokens(text)
+    return text, pt
+
+
+def measured(prompt, max_tokens, temperature=0):
+    """Return (ttft_s, decode_tok_s, completion_tokens) via usage counters."""
+    body = {"model": MODEL,
+            "messages": [{"role": "user", "content": prompt}],
+            "max_tokens": max_tokens, "temperature": temperature,
+            "stream": True, "stream_options": {"include_usage": True}}
+    req = urllib.request.Request(
+        BASE + "/v1/chat/completions", data=json.dumps(body).encode(),
+        headers={"Content-Type": "application/json"})
+    t0 = time.time()
+    ttft = None
+    usage = {}
+    with urllib.request.urlopen(req, timeout=1800) as r:
+        for raw in r:
+            line = raw.decode("utf-8", "replace").strip()
+            if not line.startswith("data:"):
+                continue
+            s = line[5:].strip()
+            if s == "[DONE]":
+                break
+            try:
+                ev = json.loads(s)
+            except json.JSONDecodeError:
+                continue
+            if ev.get("usage"):
+                usage = ev["usage"]
+            for ch in ev.get("choices", []):
+                if (ch.get("delta") or {}).get("content"):
+                    if ttft is None:
+                        ttft = time.time() - t0
+    wall = time.time() - t0
+    ct = int(usage.get("completion_tokens") or 0)
+    if ct <= 0:
+        # No usage on the stream: the endpoint is unhealthy or the model
+        # returned nothing (e.g. HTTP 200 with an error payload).
+        raise RuntimeError("no completion_tokens in usage: %r" % (usage,))
+    dec_window = max(wall - (ttft or 0.0), 1e-9)
+    return (ttft or wall), (ct / dec_window), ct
+
+
+def fmt_temp(t):
+    return ("%.1f" % t).rstrip("0").rstrip(".") if t else "0"
+
+
+def main():
+    log("base=%s model=%s ceiling=%d temps=%s"
+        % (BASE, MODEL, PROMPT_CEILING, ",".join(fmt_temp(t) for t in TEMPS)))
+    try:
+        wpt = calibrate()
+    except Exception as e:
+        log("endpoint unhealthy during calibration: %s" % e)
+        return 2
+    log("words/token: %.3f" % wpt)
+
+    result = {"words_per_token": round(wpt, 5)}
+    for tgt in DEPTHS:
+        try:
+            p, pt = prompt_for(tgt, wpt)
+            measured(p, WARM_TOKENS)  # warm the arena and the prefix cache
+        except Exception as e:
+            log("warm pass failed at depth %d: %s" % (tgt, e))
+            return 2
+        for temp in TEMPS:
+            suffix = "" if temp == 0 else "_t%s" % fmt_temp(temp)
+            rates, tts, cts = [], [], []
+            for i in range(RUNS):
+                try:
+                    tt, r, ct = measured(p, MAX_TOKENS, temp)
+                except Exception as e:
+                    log("run %d/%d failed at depth %d temp %s: %s"
+                        % (i + 1, RUNS, tgt, fmt_temp(temp), e))
+                    continue
+                rates.append(r)
+                tts.append(tt)
+                cts.append(ct)
+            if len(rates) < 2:
+                log("only %d usable run(s) at depth %d temp %s — refusing to median"
+                    % (len(rates), tgt, fmt_temp(temp)))
+                return 2
+            result["decode_tps_%d%s" % (tgt, suffix)] = round(statistics.median(rates), 2)
+            result["ttft_ms_%d%s" % (tgt, suffix)] = round(1000.0 * statistics.median(tts), 1)
+            result["completion_tokens_%d%s" % (tgt, suffix)] = int(statistics.median(cts))
+            result["prompt_tokens_%d%s" % (tgt, suffix)] = pt
+            log("depth %5d tok temp %-4s decode %7.2f t/s  ttft %.3f s  (runs=%d)"
+                % (pt, fmt_temp(temp), result["decode_tps_%d%s" % (tgt, suffix)],
+                   result["ttft_ms_%d%s" % (tgt, suffix)] / 1000.0, len(rates)))
+
+    for temp in TEMPS:
+        suffix = "" if temp == 0 else "_t%s" % fmt_temp(temp)
+        meds = [result["decode_tps_%d%s" % (t, suffix)] for t in DEPTHS]
+        result["decode_tps%s" % suffix] = round(statistics.median(meds), 2)
+    print(json.dumps(result))
+    return 0
+
+
+if __name__ == "__main__":
+    sys.exit(main())
@@ -0,0 +1,218 @@
+#!/usr/bin/env bash
+# frank-primary-window.sh — one full bench window for frank-primary, run ON
+# DANKSERVER as the deploy user (invoked over ssh by ./autoresearch.sh).
+#
+# Discipline (AGENTS.md Benchmarking Policy): the production unit is
+# STOPPED + MASKED first, the candidate config boots AS THE SERVICE itself
+# (same serve script, same container name, same port 7770 — no scratch
+# ports, no co-tenant GPU), the in-container measure runs, and the window
+# ALWAYS ends by restoring the production unit and verifying its health.
+# The EXIT trap makes the restore unconditional: a boot failure, a measure
+# failure, or a dead ssh session all end with production back up.
+#
+# Staging layout (created by the orchestrator, ephemeral — never a config
+# source; the repo checkout is):
+#   /home/dank/.autoresearch-frank-primary/
+#     serve.sh             staged candidate serve script
+#     frank-primary.env    staged candidate env (the candidate knobs)
+#     measure.py           the in-container measure client
+#     hash                 candidate fingerprint
+#     run.json             result (metrics + boot + kv + fingerprint)
+#     serve.out/measure.out/fail.log
+#     last/                memo: run.json + hash of the last FULLY
+#                          successful window (bench AND verified restore)
+#
+# This script logs to stderr and exits 0 only when the measure produced a
+# usable result (the restore happens in the trap regardless of the bench
+# outcome; a failed restore forces exit 1).
+set -uo pipefail
+
+STAGE="${FRANK_PRIMARY_AR_STAGE:-/home/dank/.autoresearch-frank-primary}"
+UNIT="${FRANK_PRIMARY_AR_UNIT:-frank-primary}"
+CTR="${FRANK_PRIMARY_AR_CTR:-frank-primary-qwen}"
+HEALTH_URL="http://127.0.0.1:7770/health"
+MODELS_URL="http://127.0.0.1:7770/v1/models"
+BOOT_TIMEOUT="${FRANK_PRIMARY_AR_BOOT_TIMEOUT:-1200}"
+RESTORE_TIMEOUT="${FRANK_PRIMARY_AR_RESTORE_TIMEOUT:-1200}"
+MEASURE_TIMEOUT="${FRANK_PRIMARY_AR_MEASURE_TIMEOUT:-1800}"
+SERVED="$(sed -n 's/^FRANK_PRIMARY_SERVED_NAME=//p' "$STAGE/frank-primary.env" 2>/dev/null | head -1)"
+SERVED="${SERVED:-frank-primary}"
+
+RESTORED=0
+
+log() { printf '[window %s] %s\n' "$(date '+%H:%M:%S')" "$*" >&2; }
+
+container_state() { podman inspect -f '{{.State.Status}}' "$CTR" 2>/dev/null || echo absent; }
+
+wait_port_free() {
+    local i
+    for i in $(seq 1 90); do
+        [ "$(container_state)" != "running" ] && return 0
+        sleep 2
+    done
+    return 1
+}
+
+wait_health() { # $1 = timeout seconds
+    local t0 code
+    t0=$(date +%s)
+    while :; do
+        code="$(curl -s -o /dev/null -w '%{http_code}' --max-time 10 "$HEALTH_URL" 2>/dev/null || echo 000)"
+        [ "$code" = "200" ] && return 0
+        [ "$(( $(date +%s) - t0 ))" -ge "$1" ] && return 1
+        sleep 2
+    done
+}
+
+capture_bench_failure() {
+    log "capturing bench-boot failure evidence"
+    {
+        echo "=== podman logs (tail 400) ==="
+        podman logs --tail 400 "$CTR" 2>&1 || true
+        echo "=== staged serve.out (tail 60) ==="
+        tail -60 "$STAGE/serve.out" 2>/dev/null || true
+        echo "=== engine log (tail 60) ==="
+        tail -60 /home/dank/logs/frank_primary.log 2>/dev/null || true
+    } >"$STAGE/fail.log" 2>&1 || true
+    grep -E 'OutOfMemory|Tried to allocate|Engine core initialization failed|Error|error' \
+        "$STAGE/fail.log" 2>/dev/null | tail -25 >&2 || true
+}
+
+restore() {
+    [ "$RESTORED" = 1 ] && return 0
+    RESTORED=1
+    log "restoring production unit $UNIT"
+    if [ -f "$STAGE/serve.pid" ]; then
+        kill "$(cat "$STAGE/serve.pid" 2>/dev/null)" 2>/dev/null || true
+    fi
+    podman stop -t 120 "$CTR" >/dev/null 2>&1 || true
+    wait_port_free || log "WARNING: bench container still draining after 180s"
+    sudo systemctl unmask "$UNIT" 2>/dev/null || true
+    if ! sudo systemctl start "$UNIT"; then
+        log "ERROR: systemctl start $UNIT failed — operator intervention required"
+        return 1
+    fi
+    if wait_health "$RESTORE_TIMEOUT"; then
+        if curl -s --max-time 10 "$MODELS_URL" 2>/dev/null | grep -q "$SERVED"; then
+            log "production restored and verified ($SERVED answering on :7770)"
+            # Memo promotion lives HERE: a cached result is only ever the
+            # product of a fully successful window (bench AND verified
+            # restore), never of a bare bench success.
+            cp "$STAGE/run.json" "$STAGE/last/" 2>/dev/null || true
+            cp "$STAGE/hash" "$STAGE/last/" 2>/dev/null || true
+        else
+            log "ERROR: health 200 but /v1/models does not list $SERVED"
+            return 1
+        fi
+    else
+        log "ERROR: production not healthy within ${RESTORE_TIMEOUT}s — unit left to systemd (Restart=always)"
+        return 1
+    fi
+    return 0
+}
+trap 'restore; rc=$?; [ "$rc" -ne 0 ] && exit 1' EXIT
+
+main() {
+    local f
+    for f in serve.sh frank-primary.env measure.py hash; do
+        [ -f "$STAGE/$f" ] || { log "staged file missing: $STAGE/$f"; exit 1; }
+    done
+    mkdir -p "$STAGE/last"
+    rm -f "$STAGE/run.json" "$STAGE/fail.log" "$STAGE/measure.out" "$STAGE/evidence.log"
+
+    local t_start t0 i kvline kv_tokens kv_conc measure_json boot_seconds
+    t_start="$(date +%s)"
+
+    log "stopping and masking production unit $UNIT"
+    sudo systemctl stop "$UNIT" 2>/dev/null || true
+    # Explicit stop is not auto-restarted even with Restart=always; the mask
+    # is belt-and-braces against a stray start during the window.
+    sudo systemctl mask "$UNIT" 2>/dev/null || \
+        log "WARNING: mask failed — relying on explicit-stop semantics"
+    wait_port_free || log "note: production container still draining"
+
+    t0="$(date +%s)"
+    log "starting bench instance from staged config (fingerprint $(tr -d '[:space:]' <"$STAGE/hash"))"
+    FRANK_PRIMARY_ENV_FILE="$STAGE/frank-primary.env" \
+        setsid nohup bash "$STAGE/serve.sh" >"$STAGE/serve.out" 2>&1 </dev/null &
+    echo "$!" >"$STAGE/serve.pid"
+
+    for i in $(seq 1 45); do
+        [ "$(container_state)" = "running" ] && break
+        sleep 2
+    done
+    if [ "$(container_state)" != "running" ]; then
+        log "bench container never reached running state"
+        tail -40 "$STAGE/serve.out" >&2 || true
+        exit 1
+    fi
+
+    if wait_health "$BOOT_TIMEOUT"; then
+        boot_seconds=$(( $(date +%s) - t0 ))
+        log "bench healthy after ${boot_seconds}s"
+    else
+        if [ "$(container_state)" = "running" ]; then
+            log "bench boot timed out after ${BOOT_TIMEOUT}s (still loading)"
+        else
+            log "bench container died during boot"
+            capture_bench_failure
+        fi
+        exit 1
+    fi
+
+    log "running measure in container (timeout ${MEASURE_TIMEOUT}s)"
+    if ! podman cp "$STAGE/measure.py" "$CTR":/tmp/ar-measure.py; then
+        log "podman cp of measure client failed"
+        exit 1
+    fi
+    if ! timeout "$MEASURE_TIMEOUT" podman exec "$CTR" python3 /tmp/ar-measure.py \
+            >"$STAGE/measure.out" 2>&1; then
+        log "measure failed or timed out"
+        tail -40 "$STAGE/measure.out" >&2 || true
+        exit 1
+    fi
+
+    # Capacity facts for the run (context-size experiments change these).
+    kvline="$(podman logs "$CTR" 2>&1 | grep -E 'GPU KV cache size' | tail -1 || true)"
+    kv_tokens="$(printf '%s' "$kvline" | sed -E 's/.*GPU KV cache size: ([0-9,]+) tokens.*/\1/' | tr -d ',' || true)"
+    kv_conc="$(printf '%s' "$kvline" | sed -E 's/.*per request: ([0-9.]+x).*/\1/' || true)"
+
+    # Human-facing evidence, in a sidecar file because it is multi-line prose,
+    # not metrics: the expert-plan lines (the ONLY place the arena / runtime
+    # overhead / KV-reserve numbers exist — they are never in /metrics), the
+    # compile path (cold build vs cache hit), the graph-capture sizes, the
+    # speculative-decode counters, and any OOM signature. Without this an arm
+    # that loses decode speed is unexplainable afterwards.
+    {
+        echo "=== engine plan / capacity / spec evidence ==="
+        podman logs "$CTR" 2>&1 | grep -E \
+'KV cache size|Maximum concurrency|layers split|expert cache|runtime overhead|non-default args|r4d all-reduce|Using \[|cudagraph|CUDA graph|Capturing|graph for compile range|torch.compile took|Directly load the compiled|speculative|Speculative|acceptance|Acceptance|offload|precise|OutOfMemory|Tried to allocate|Engine core initialization failed' \
+            | tail -150 || true
+        echo "=== /metrics counters (spec decode, caches, success) ==="
+        curl -s --max-time 15 http://127.0.0.1:7770/metrics 2>/dev/null \
+            | grep -E '^vllm:(spec_decode|gpu_cache|prefix_cache|request_success|generation_tokens|iteration)' \
+            | head -80 || true
+    } >"$STAGE/evidence.log" 2>&1 || true
+    log "engine evidence -> $STAGE/evidence.log ($(wc -l <"$STAGE/evidence.log") lines)"
+
+    measure_json="$(grep -E '^\{' "$STAGE/measure.out" | tail -1 || true)"
+    if [ -z "$measure_json" ]; then
+        log "measure produced no JSON result line"
+        tail -40 "$STAGE/measure.out" >&2 || true
+        exit 1
+    fi
+
+    # The measure line is already JSON, so the run object can be assembled
+    # without any host-side interpreter.
+    printf '{"hash":"%s","boot_seconds":%s,"kv_tokens":%s,"kv_concurrency":"%s","total_seconds":%s,"measure":%s}\n' \
+        "$(tr -d '[:space:]' <"$STAGE/hash")" \
+        "$boot_seconds" "${kv_tokens:-0}" "${kv_conc:-unknown}" \
+        "$(( $(date +%s) - t_start ))" "$measure_json" >"$STAGE/run.json"
+
+    log "bench window complete in $(( $(date +%s) - t_start ))s"
+    # Memo only after a FULL success; the restore below still has to verify.
+    exit 0
+}
+
+main "$@"
+exit "$?"
@@ -0,0 +1,89 @@
+#!/usr/bin/env bash
+# Build the ROCmFPX runtime that serves frank-large on Dankclaw.
+#
+#   bash scripts/build-rocmfpx.sh              # clone/fetch + build the pin
+#   bash scripts/build-rocmfpx.sh --verify     # only check the installed tree
+#
+# Runtime: kingjones30/ROCmFPX — charlie12345/ROCmFPX plus the model-card
+# patches, i.e. the fork that owns the ROCmFPx tensor formats. frank-large's
+# payload is ftype 115 (`Q8_0_ROCMFPX_AGENT`): 311 tensors of
+# GGML_TYPE_Q8_0_ROCMFPX (103), a type that exists ONLY in this fork, so
+# stock llama.cpp, gufo and every other loader on the host reject the file.
+# This is the same tree that served the Qwen3.8-Flash-Next ROCmFP4 STRIX
+# endpoint 2026-08-30 → 09-11 (see the kingjones runbook), rebuilt here so a
+# fresh host can be brought up from the repo alone.
+#
+# Why this commit and not a newer one: 36e9acd (2026-08-28) is what was
+# qualified against this payload on 2026-10-02 — it loads ftype 115, survives
+# warmup, and drafts with the MTP head at acceptance 0.65. The five legacy
+# builds of 2026-08-25 crashed this same file in ggml_compute_forward_set_rows
+# (see kingjones-q8agent-rocmfpx.md), so the pin is load-bearing: do not bump
+# it without re-running that crash test.
+#
+# Host notes:
+#   * -DLLAMA_BUILD_WEBUI=OFF is REQUIRED on this box. The tree provisions its
+#     embedded UI assets via npm or the HF llama-ui bucket; npm is absent here
+#     and the bucket download half-failed, aborting the link at 98%. The API
+#     server is unaffected.
+#   * The build links against the host ROCm (/opt/rocm); no bundled toolchain
+#     like the gfx1201 R9700 build needed.
+set -euo pipefail
+
+ROCMFPX_REPO="${ROCMFPX_REPO:-https://github.com/kingjones30/ROCmFPX.git}"
+# The commit frank-large was qualified with (measured 2026-10-02).
+ROCMFPX_COMMIT="${ROCMFPX_COMMIT:-36e9acd40e10a87cd3c3ef8ec734668757dc8520}"
+SRC_DIR="${ROCMFPX_SRC_DIR:-/home/dank/GitPackages/kingjones-ROCmFPX}"
+BUILD_DIR="${ROCMFPX_BUILD_DIR:-$SRC_DIR/build}"
+GPU_TARGET="${ROCMFPX_GPU_TARGET:-gfx1151}"
+JOBS="${JOBS:-$(nproc)}"
+
+# check_pin — refuse a source tree that is not the qualified commit. A moved
+# fork means different kernels under the same path, and on this model line a
+# wrong build is a segfault at first request, not a clean error.
+check_pin() {
+    local commit
+    commit="$(git -C "$SRC_DIR" rev-parse HEAD 2>/dev/null || true)"
+    if [[ -n "$commit" && "$commit" != "$ROCMFPX_COMMIT" ]]; then
+        echo "ROCMFPX_COMMIT mismatch: $SRC_DIR is at $commit, pinned $ROCMFPX_COMMIT" >&2
+        exit 1
+    fi
+}
+
+if [[ "${1:-}" == "--verify" ]]; then
+    check_pin
+    [[ -x "$BUILD_DIR/bin/llama-server" ]] || {
+        echo "no ROCmFPX llama-server at $BUILD_DIR/bin/llama-server" >&2
+        exit 1
+    }
+    # The format the endpoint exists for: the fork must carry the ROCmFPx
+    # tensor type, or the payload cannot load at all.
+    "$BUILD_DIR/bin/llama-server" --help 2>&1 | grep -q -- '--spec-type' || {
+        echo "$BUILD_DIR/bin/llama-server has no --spec-type: MTP drafting is impossible" >&2
+        exit 1
+    }
+    "$BUILD_DIR/bin/llama-server" --version | head -1
+    exit 0
+fi
+
+echo "== source: $ROCMFPX_REPO @ $ROCMFPX_COMMIT =="
+if [[ ! -d "$SRC_DIR/.git" ]]; then
+    mkdir -p "$(dirname "$SRC_DIR")"
+    git clone "$ROCMFPX_REPO" "$SRC_DIR"
+fi
+git -C "$SRC_DIR" fetch --quiet origin "$ROCMFPX_COMMIT" || true
+git -C "$SRC_DIR" checkout -q --detach "$ROCMFPX_COMMIT"
+check_pin
+echo "   $(git -C "$SRC_DIR" rev-parse HEAD)"
+
+echo "== configure + build (release, HIP $GPU_TARGET) =="
+cmake -S "$SRC_DIR" -B "$BUILD_DIR" \
+    -DGGML_HIP=ON \
+    -DGPU_TARGETS="$GPU_TARGET" \
+    -DGGML_NATIVE=ON \
+    -DCMAKE_BUILD_TYPE=Release \
+    -DLLAMA_BUILD_WEBUI=OFF
+cmake --build "$BUILD_DIR" --target llama-server llama-quantize -j "$JOBS"
+
+echo "== verify =="
+"$BUILD_DIR/bin/llama-server" --version | head -1
+echo "frank-large runtime ready: $BUILD_DIR/bin/llama-server ($ROCMFPX_COMMIT)"
@@ -935,15 +935,24 @@ deploy_plex() {
    fi
    systemctl --user restart plex.service
    # Readiness gate: /identity answers MediaContainer XML once PMS is up.
+    #
+    # HTTPS, NOT HTTP. PMS drops plain HTTP off-loopback: the probe's
+    # connection arrives on the container's eth0, so an http:// request is
+    # accepted and then answered with nothing (curl 52, Empty reply) against a
+    # perfectly healthy server. Measured 2026-10-02 — the gate burned its full
+    # 600s and failed a plex whose https://127.0.0.1:32400/identity answered
+    # 200 in 44 ms the whole time. The same law is why the Caddyfile dials the
+    # plex upstream over TLS (tests/caddyfile.bats pins it).
+    # -k: PMS presents its own self-signed cert on 32400.
+    #
    # Reality check from the 2026-09 cutover: the FIRST container boot of a
-    # years-old library ran ~10 min (PMS pre-binds 32400 during boot and
-    # empty-replies until the HTTP layer is ready), so the gate allows 600s.
+    # years-old library ran ~10 min, so the gate still allows 600s.
    # Fail the deploy (pending state preserved) rather than mark it deployed blind.
    for ((i = 1; i <= 120; i++)); do
-        curl -fsS --max-time 3 http://127.0.0.1:32400/identity >/dev/null && return 0
+        curl -fsSk --max-time 3 https://127.0.0.1:32400/identity >/dev/null && return 0
        sleep 5
    done
-    echo "FAIL: plex did not answer /identity within 600s" >&2
+    echo "FAIL: plex did not answer /identity over HTTPS within 600s" >&2
    return 1
 }

@@ -1077,27 +1086,33 @@ deploy_anubis() {
    wait_container_healthy anubis 60 || return 1
 }

-# frank-large is gufo (gufo-org/gufo), the Strix Halo inference engine built
-# from the pinned v0.1.1 tag by scripts/build-gufo.sh, serving
-# unsloth/Qwen3.8-27B-GGUF:UD-Q8_K_XL with the DFlash2 drafter
-# (z-lab/Qwen3.8-27B-DFlash2-GGUF:Q4_K_M) on the Strix Halo iGPU. It is a
-# ROOT unit: the runtime is a native binary linked against the host ROCm,
-# so there is no container image to install and no generated unit to dodge.
-# relink_unit points /etc/systemd/system/frank-large.service at the repo
-# file and boot persistence is that unit's [Install] section.
+# frank-large is the halogen CONTAINER (ghcr.io/peonist-ai/halogen,
+# digest-pinned through scripts/frank-large.env) serving
+# peonist-ai/halogen-qwen3.8-27b — qwen3.8-27b-p1w4d-d2.hgn, 33.4 GiB,
+# DFlash2 spec decode, 1 slot x 200700, text-only — on the Strix Halo
+# iGPU. It stays a ROOT unit because the serve script drives `podman run`
+# directly (the frank-primary pattern): relink_unit points
+# /etc/systemd/system/frank-large.service at the repo file, boot
+# persistence is that unit's [Install], and LimitMEMLOCK=infinity is
+# fleet parity. 2026-10-05 cutover from the ROCmFPX llama.cpp shape —
+# rollback = revert the swap commit's trio; that payload, its draft head
+# and the fork build all stay on disk
+# (~/models/kingjones-Qwen3.8-27B-Uncensored-ROCmFP4-STRIX-MTP,
+# ~/GitPackages/kingjones-ROCmFPX).
 #
-# The runtime gate matters as much as the payload gate: gufo is built from
-# source on this host (there is no prebuilt release), so a machine whose
-# /home/dank/GitPackages tree was never built has a unit that can never
-# become healthy, and a green deploy would leave the endpoint dark.
-#
-# It replaces the stock llama.cpp prebuilt (unsloth b11030), which had
-# replaced the Kairic Edge runtime (ciru-ai/ROCmFPX). The port gate is the
+# The gates mirror serve-frank-large.sh's predicates EXACTLY (the
+# 2026-10-01 crash-loop lesson): image digest in dank's podman storage,
+# the .hgn checkpoint at the exact byte count, the tokenizer directory
+# holding every named file as a regular (not symlinked) file — a green
+# deploy against a missing artifact would
+# leave a restart-looping unit and a dark endpoint. The port gate is the
 # same as every cutover since 2026-09-16: nothing starts until 7770 is
 # released.
-LLM_LARGE_RUNTIME="${LLM_LARGE_RUNTIME:-/home/dank/GitPackages/gufo-v0.1.1/bin/gufo}"
-LLM_LARGE_MODEL="${LLM_LARGE_MODEL:-/home/dank/models/Swift-1.5-Qwen3.8-27B-Uncensored-MTP-UD-Q8_K_XL/Swift-1.5-Qwen3.8-27B-Uncensored-Dynamic-MTP-UD-Q8_K_XL.gguf}"
-LLM_LARGE_DRAFT="${LLM_LARGE_DRAFT:-/home/dank/models/Qwen3.8-27B-UD-Q8_K_XL/Qwen3.8-27B-DFlash2-Q4_K_M.gguf}"
+LLM_LARGE_IMAGE="${LLM_LARGE_IMAGE:-ghcr.io/peonist-ai/halogen@sha256:dc0a39a0016d6cfc58a197978febaafdf8d28403f724ded6111d98b5fb7ac0ea}"
+LLM_LARGE_CKPT="${LLM_LARGE_CKPT:-/home/dank/models/halogen-qwen3.8-27b/qwen3.8-27b-p1w4d-d2.hgn}"
+LLM_LARGE_CKPT_BYTES="${LLM_LARGE_CKPT_BYTES:-35865565184}"
+LLM_LARGE_TOK_DIR="${LLM_LARGE_TOK_DIR:-/home/dank/models/halogen-qwen3.8-27b/tokenizer}"
+LLM_LARGE_TOK_FILES="${LLM_LARGE_TOK_FILES:-tokenizer.json chat_template.jinja tokenizer_config.json merges.txt vocab.json}"

 retire_halogen_large_quadlet() {
    local trash="$HOME/.AGENTTRASH/20260919-orca-cutover/quadlet"
@@ -1130,45 +1145,51 @@ wait_port_free() {
 }

 deploy_llm_large() {
-    # Refuse rather than restart-loop. A unit whose runtime or payload is
-    # missing never becomes healthy, and a green deploy would leave the
-    # endpoint dark with nothing in the deploy log to say why.
-    if [[ ! -x "$LLM_LARGE_RUNTIME" ]]; then
-        echo "ERROR: gufo runtime absent at $LLM_LARGE_RUNTIME" >&2
-        echo "frank-large: build it: bash scripts/build-gufo.sh" >&2
+    # Refuse rather than restart-loop; every gate below has a twin in
+    # serve-frank-large.sh and they must agree (2026-10-01 lesson).
+    # Capture podman output before grepping: `podman images | grep -q`
+    # races under pipefail (SIGPIPE 141 becomes "no match", 2026-09-30).
+    if ! grep -qF "${LLM_LARGE_IMAGE#*@}" <<<"$(podman images --format '{{.RepoDigests}}' 2>/dev/null)"; then
+        echo "ERROR: halogen image absent: $LLM_LARGE_IMAGE" >&2
+        echo "frank-large: pull it: podman pull ${LLM_LARGE_IMAGE%@*}" >&2
        return 1
    fi
-    local missing=0 f
-    for f in "$LLM_LARGE_MODEL" "$LLM_LARGE_DRAFT"; do
-        if [[ ! -f "$f" ]]; then
-            echo "ERROR: payload absent at $f" >&2
-            echo "frank-large: fetch it: bash scripts/download-frank-large.sh" >&2
-            missing=1
+    if [[ ! -f "$LLM_LARGE_CKPT" ]]; then
+        echo "ERROR: halogen checkpoint absent at $LLM_LARGE_CKPT" >&2
+        echo "frank-large: fetch it: bash scripts/download-frank-large-halogen.sh" >&2
+        return 1
+    fi
+    if [[ "$(stat -c %s "$LLM_LARGE_CKPT")" != "$LLM_LARGE_CKPT_BYTES" ]]; then
+        echo "ERROR: halogen checkpoint is not $LLM_LARGE_CKPT_BYTES bytes (partial download?)" >&2
+        echo "frank-large: re-check: bash scripts/download-frank-large-halogen.sh --verify-only" >&2
+        return 1
+    fi
+    if [[ -L "$LLM_LARGE_TOK_DIR" || ! -d "$LLM_LARGE_TOK_DIR" ]]; then
+        echo "ERROR: tokenizer dir missing or a symlink at $LLM_LARGE_TOK_DIR" >&2
+        echo "frank-large: fetch it: bash scripts/download-frank-large-halogen.sh" >&2
+        return 1
+    fi
+    for tf in $LLM_LARGE_TOK_FILES; do
+        if [[ ! -f "$LLM_LARGE_TOK_DIR/$tf" || -L "$LLM_LARGE_TOK_DIR/$tf" ]]; then
+            echo "ERROR: tokenizer file missing or a symlink: $LLM_LARGE_TOK_DIR/$tf" >&2
+            echo "frank-large: fetch it: bash scripts/download-frank-large-halogen.sh" >&2
+            return 1
        fi
    done
-    [[ "$missing" -eq 0 ]] || return 1

    retire_halogen_large_quadlet
-    # On a re-deploy the predecessor is THIS unit: stop it before waiting on
-    # the port, or the wait deadlocks on the server it just stopped serving.
-    # Then sweep a stray manual run -- that is how a bench harness leaves the
-    # port busy. Match the DRAFTER path, which every frank-large command
-    # line carries via --dflash-model and which is UNCHANGED by the 2026-09-28
-    # payload swap: the target moved to a different repository while the
-    # drafter stayed put, so keying on the target filename would have quietly
-    # stopped matching. It is also the right key in principle — it names this
-    # endpoint's speculative drafter, not a binary. Match that, not a binary
-    # name: frank-small on this host serves a different GGUF over llama-server
-    # with no DFlash2 drafter and must not be killed. The pattern never
-    # appears in the deploying shell's own command line, so -f is safe here
-    # (it is not in `bash deploy-remote.sh`).
+    # The predecessor on a re-deploy is THIS unit: stop it (ExecStop stops
+    # the container) before waiting on the port. The stray sweep is keyed
+    # on the FULL container name, which appears in the argv of a manual
+    # `podman run --name frank-large-halogen` / `podman wait` but NOT in
+    # this deployer's own command line (`bash deploy-remote.sh
+    # frank-large`) — so it cannot shoot itself in the foot (the fleet
+    # pkill law), and it can never match frank-small's llama-server.
    run_privileged systemctl stop frank-large.service 2>/dev/null || true
-    pkill -f 'Qwen3.8-27B-DFlash2-Q4_K_M' 2>/dev/null || true
-    # Nothing may start until the predecessor has released the port. On
-    # 2026-09-16 the cutover failed exactly here: the old server logged
-    # "Deactivated successfully", the next engine tried to bind 7770 a second
-    # later and reported "Address already in use". Gate on the port, not on
-    # hope.
+    pkill -f 'frank-large-halogen' 2>/dev/null || true
+    podman rm -f frank-large frank-large-halogen >/dev/null 2>&1 || true
+    # Nothing may start until the predecessor has released the port
+    # (2026-09-16: "Deactivated successfully" was not "port free").
    wait_port_free 7770 || return 1
    relink_unit frank-large
    run_privileged systemctl daemon-reload
@@ -1177,28 +1198,56 @@ deploy_llm_large() {
    wait_for_health frank-large http://127.0.0.1:7770/health 1200
 }

-# deploy_llm_small — frank-small is mainline llama.cpp on the RTX 3090, as a
-# ROOT unit: the runtime is a native CUDA binary, so there is no container
-# image to install and no generated unit to avoid. relink_unit points
-# /etc/systemd/system/frank-small.service at the repo file and boot persistence
-# is that unit's [Install] section.
+# deploy_llm_small — frank-small is llama.cpp (master build, CUDA sm_86) on
+# the RTX 3090 as a ROOT unit: the runtime is a native binary tree under
+# ~/GitPackages/llama-master/build-cuda, so there is no container image and
+# no generated unit to avoid. relink_unit points
+# /etc/systemd/system/frank-small.service at the repo file and boot
+# persistence is that unit's [Install] section.
 #
 # It is a SEPARATE endpoint on port 7771; frank-large keeps 7770. The 3090 is
 # not shared: frank-large's engine runs on the gfx1151 iGPU.
-LLM_SMALL_MODEL="${LLM_SMALL_MODEL:-/home/dank/models/Gemma4-12B-QAT/Gemma4-12B-QAT-Uncensored-HauhauCS-Balanced-Q4_K_M.gguf}"
+LLM_SMALL_MODEL_FILE="${LLM_SMALL_MODEL_FILE:-/home/dank/models/Ornith-Q8/Ornith-1.5-9B-Q8_0.gguf}"
+LLM_SMALL_RUNTIME="${LLM_SMALL_RUNTIME:-/home/dank/GitPackages/llama-master/build-cuda/bin/llama-server}"
+# The projector IS the vision capability here: the Ornith GGUF ships no
+# `v.*` tensors, so a deploy that starts without this file silently ships a
+# text-only endpoint. Checked unconditionally, like frank-large's.
+LLM_SMALL_MMPROJ="${LLM_SMALL_MMPROJ:-/home/dank/models/Ornith-Q8/mmproj-Ornith-1.5-9B-BF16.gguf}"

 deploy_llm_small() {
-    if [[ ! -f "$LLM_SMALL_MODEL" ]]; then
-        echo "ERROR: model absent at $LLM_SMALL_MODEL" >&2
-        echo "frank-small: fetch HauhauCS/Gemma4-12B-QAT-Uncensored-HauhauCS-Balanced (Q4_K_M, the mmproj and the MTP sidecar) first;" >&2
-        echo "frank-small: refusing to start a unit that cannot load." >&2
+    # Refuse rather than restart-loop: a unit whose runtime or payload is
+    # missing never becomes healthy, and a green deploy would leave the
+    # endpoint dark with nothing in the log to say why.
+    if [[ ! -x "$LLM_SMALL_RUNTIME" ]]; then
+        echo "ERROR: llama.cpp runtime absent at $LLM_SMALL_RUNTIME" >&2
+        echo "frank-small: build it (cmake -B build-cuda -DGGML_CUDA=ON" >&2
+        echo "frank-small: -DCMAKE_CUDA_ARCHITECTURES=86 -DCMAKE_CUDA_HOST_COMPILER=/usr/bin/gcc-13" >&2
+        echo "frank-small: && cmake --build build-cuda --target llama-server) first." >&2
        return 1
    fi
+    for f in "$LLM_SMALL_MODEL_FILE" "$LLM_SMALL_MMPROJ"; do
+        if [[ ! -f "$f" ]]; then
+            echo "ERROR: frank-small payload file absent at $f" >&2
+            echo "frank-small: run scripts/download-frank-small.sh first;" >&2
+            echo "frank-small: refusing to start a unit that cannot load." >&2
+            return 1
+        fi
+    done
    relink_unit frank-small
    run_privileged systemctl daemon-reload
    run_privileged systemctl enable frank-small.service
    run_privileged systemctl restart frank-small.service
-    wait_for_health frank-small http://127.0.0.1:7771/health 900
+    wait_for_health frank-small http://127.0.0.1:7771/health 900 || return 1
+    # False-green hardening (frank-large's lesson): the port answering
+    # proves nothing about WHO answers — the unit must be active with a
+    # live MainPID, or a stray SGLang holdover is masquerading as deployed.
+    local main_pid
+    main_pid="$(run_privileged systemctl show -p MainPID --value frank-small.service)"
+    if [[ -z "$main_pid" || "$main_pid" == "0" ]]; then
+        echo "FAIL: :7771 answered but frank-small.service has no MainPID — a stray process holds the port" >&2
+        return 1
+    fi
+    echo "frank-small: health answered by unit MainPID $main_pid"
 }

 # deploy_llm_primary — frank-primary is the tcclaviger RDNA4 vLLM fork
@@ -1218,7 +1267,7 @@ deploy_llm_small() {
 # sha256 verification was the fetch script's job at download time, never
 # re-hashed here.
 LLM_PRIMARY_CONTAINER=frank-primary-qwen
-LLM_PRIMARY_LEGACY_CONTAINERS="frank-primary-r9v qwen-fp8"
+LLM_PRIMARY_LEGACY_CONTAINERS="frank-primary-r9v qwen-fp8 glm-probe"

 deploy_llm_primary() {
    # Refuse rather than restart-loop: a unit whose image or payload is
@@ -1237,7 +1286,13 @@ deploy_llm_primary() {
        echo "ERROR: frank-primary.env missing FRANK_PRIMARY_IMAGE/MODEL_DIR/MODEL_SHARDS" >&2
        return 1
    fi
-    if ! podman images --format '{{.RepoDigests}}' | grep -qF "${image#*@}"; then
+    # Capture podman's output BEFORE grepping: under set -o pipefail, the
+    # classic `podman images | grep -q` races (grep exits on match, podman
+    # dies SIGPIPE 141 writing the rest, pipefail calls it no-match — the
+    # 2026-09-30 deploy refusal).
+    local digests
+    digests="$(podman images --format '{{.RepoDigests}}')"
+    if ! grep -qF "${image#*@}" <<<"$digests"; then
        echo "ERROR: pinned runtime image absent from rootless storage: $image" >&2
        echo "frank-primary: fetch it (podman pull $image)" >&2
        return 1
@@ -1264,8 +1319,9 @@ deploy_llm_primary() {
    # A stopped unit can leave its container running: systemd's cgroup kill
    # races conmon's --rm cleanup, and the orphan holds the port and GPUs.
    # Stop the unit, then remove the containers BEFORE waiting on the port.
-    # The legacy identities (the retired R9V container and the ad hoc
-    # experiment container qwen-fp8) hold the same GPUs and die here too.
+    # The legacy identities (the retired R9V container, the ad hoc
+    # experiment container qwen-fp8, the retired glm-probe) hold the same
+    # GPUs and die here too.
    run_privileged systemctl stop frank-primary.service 2>/dev/null || true
    local ctr
    for ctr in $LLM_PRIMARY_LEGACY_CONTAINERS "$LLM_PRIMARY_CONTAINER"; do
@@ -196,6 +196,21 @@ else
    printf '    - %s\n' "${PLAN[@]}"
 fi

+# --- frankbook e2e trigger --------------------------------------------------
+# The *arr/qBittorrent e2e units walk links that terminate on FrankBook-Pro
+# (192.168.0.112), which sleeps when idle. The [3/3] e2e gate therefore runs
+# the frankbook Playwright project ONLY when something in its blast radius
+# changed since origin/main: the frankbook spec itself, the frankbook-pro
+# stack, the homepage registry, or the Caddyfile serving the arr vhosts.
+# Every other deploy skips it (tests/e2e/run.sh excludes the project by
+# default). See tests/e2e/specs/19-frankbook.spec.ts.
+FB_E2E_TRIGGER=0
+while IFS= read -r fb_path; do
+    case "$fb_path" in
+        tests/e2e/specs/19-frankbook.spec.ts|frankbook-pro/*|shared/homepage/services.json|dankclaw/Caddyfile) FB_E2E_TRIGGER=1 ;;
+    esac
+done < <(git diff --name-only "$BASE_REVISION" "$HEAD_REVISION")
+
 REMOTE="${REMOTE_USER}@${REMOTE_HOST}"
 REMOTE_ARGS=("$TARGET" "$HEAD_REVISION" "$REMOTE_DIR" "$REMOTE_SHELL")
 if [[ "${#EXPLICIT_SERVICES[@]}" -eq 0 ]]; then
@@ -209,6 +224,11 @@ if [[ "$DRY_RUN" -eq 1 ]]; then
    printf '[dry-run] Runner arguments:'
    printf ' %q' "${REMOTE_ARGS[@]}"
    printf '\n'
+    if [[ "$FB_E2E_TRIGGER" -eq 1 ]]; then
+        echo "[dry-run] e2e gate: frankbook project INCLUDED (frankbook trigger paths changed)."
+    else
+        echo "[dry-run] e2e gate: frankbook project EXCLUDED (no frankbook trigger paths)."
+    fi
    exit 0
 fi

@@ -250,7 +270,12 @@ if [[ "$SKIP_E2E" -eq 1 ]]; then
    echo "[3/3] e2e gate SKIPPED (--skip-e2e): live fleet NOT verified." >&2
 else
    echo "[3/3] Running Playwright e2e gate against the live fleet..."
-    if bash "$REPO_ROOT/tests/e2e/run.sh"; then
+    E2E_ENV=()
+    if [[ "$FB_E2E_TRIGGER" -eq 1 ]]; then
+        echo "      frankbook trigger paths changed: including the frankbook e2e project."
+        E2E_ENV=(E2E_INCLUDE_FRANKBOOK=1)
+    fi
+    if env "${E2E_ENV[@]}" bash "$REPO_ROOT/tests/e2e/run.sh"; then
        echo "e2e gate PASSED."
    else
        echo "=== DEPLOY INCOMPLETE: e2e gate FAILED on $TARGET ===" >&2
@@ -0,0 +1,62 @@
+# frank-primary-freetoken.Dockerfile — FreeToken (PR #587 ROCm) serving image
+# for the frank-primary endpoint (dankserver :7770, 1x R9700 gfx1201).
+#
+# Built on dankserver with rootless podman (dank user), same storage the
+# service's `podman run` reads:
+#   podman build -t frank-primary-freetoken:pr587-77a7201 \
+#     -f scripts/docker/frank-primary-freetoken.Dockerfile scripts/docker/
+#
+# Base: the PR's own docs/install_amd.md names this exact image as the
+# known-good ROCm 7.14 stack for gfx1201 (torch 2.11.0+rocm7.14, triton-rocm
+# 3.7 — the original pin; FreeToken main+PR587 accepts torch>=2.11,<2.16 and
+# triton>=3.6,<3.9). There is no rocm7.14+torch2.14 image on Docker Hub (the
+# 2.14 images are rocm10.1.0 — an untested pair for this PR), so the PR doc's
+# verbatim recommendation is the deployed stack.
+#
+# FreeToken source: FlashML-org/FreeToken PR #587 at pinned head
+# 77a7201f98f00d93f618c6cf8b81ffe8360e0899 (branch rocm714_torch214, base
+# main 581fdca3). The PR is ROCm-gated end to end; the NVIDIA/CUDA path is
+# byte-identical to main.
+#
+# Build law from docs/install_amd.md:
+#   - `pip install --no-build-isolation --no-deps -e .` — the image's ROCm
+#     torch is preserved and used to compile the C++/HIP extensions.
+#   - NEVER a plain `pip install -e .` on ROCm: the dependency set is
+#     authored for NVIDIA (flashlib pulls numba + nvidia-cutlass-dsl; torch
+#     metadata pulls cuda-bindings) and the resolve can REPLACE the ROCm
+#     torch with a CUDA build.
+#   - Runtime deps installed explicitly, curated by the same doc. flashinfer
+#     / sgl_kernel / nvidia-cutlass-dsl have no ROCm builds and are
+#     intentionally absent — FreeToken gates them off and falls back to its
+#     Triton kernels on ROCm.
+#
+# Base is digest-pinned (2026-10-05): sha256:a223aee1… is the manifest this
+# build was qualified against; a re-pushed tag cannot change the stack.
+FROM rocm/pytorch@sha256:a223aee17aef5d21c3b9f63436dd19d27d1c665ec8b2f40011c9546cabae2a80
+
+# Pin the PR head, not the branch: a future push to rocm714_torch214 would
+# otherwise silently change the deployed engine.
+# The pip-ROCm torch in this image ships libhiprtc as the VERSIONED
+# libhiprtc.so.7 only; the PR's `_cpu_moe` extension links `-lhiprtc`, and
+# ld looks for the unversioned dev symlink in the ALREADY-LINKED
+# _rocm_sdk_core/lib dir. Create it there (first build failure 2026-10-05:
+# `ld: cannot find -lhiprtc`).
+RUN ln -s libhiprtc.so.7 /opt/venv/lib/python3.12/site-packages/_rocm_sdk_core/lib/libhiprtc.so
+RUN git clone https://github.com/FlashML-org/FreeToken.git /opt/FreeToken \
+ && cd /opt/FreeToken \
+ && git fetch origin pull/587/head \
+ && git checkout 77a7201f98f00d93f618c6cf8b81ffe8360e0899
+
+WORKDIR /opt/FreeToken
+RUN python -m pip install --no-build-isolation --no-deps -e .
+
+# Curated runtime dependencies (docs/install_amd.md). transformers is pinned
+# to the pyproject window (>=5.16,<5.17); the rest are the doc's portable set.
+# torch / torchvision / triton-rocm come from the base image, 1:1 aligned.
+RUN python -m pip install \
+      "transformers>=5.16,<5.17" tokenizers safetensors huggingface_hub \
+      fastapi uvicorn pydantic openai httpx anyio \
+      einops gguf msgpack pyzmq typer rich apache-tvm-ffi \
+      partial-json-parser \
+ && python -m pip install --no-deps flashlib==0.3.0 \
+ && python -m pip install ninja
@@ -0,0 +1,76 @@
+#!/usr/bin/env bash
+# download-frank-large-halogen.sh — fetch frank-large's halogen payload:
+# the qwen3.8-27b .hgn checkpoint and the export tokenizer, from the
+# PUBLIC peonist-ai/halogen-qwen3.8-27b repo (no token — and per the fleet
+# secret law, none would ever touch argv).
+#
+# The fetch law, same as every sibling script: curl -f (an access error
+# must not become a "successful" 153-byte artifact), exact byte count from
+# the model card, LFS sha256 verification, --verify-only to re-check an
+# existing payload without downloading.
+set -euo pipefail
+
+DEST="${FRANK_LARGE_MODEL_DIR:-$HOME/models/halogen-qwen3.8-27b}"
+BASE="https://huggingface.co/peonist-ai/halogen-qwen3.8-27b/resolve/main"
+
+FILES=(qwen3.8-27b-p1w4d-d2.hgn
+       tokenizer/tokenizer.json
+       tokenizer/vocab.json
+       tokenizer/merges.txt
+       tokenizer/chat_template.jinja
+       tokenizer/tokenizer_config.json)
+# Byte sizes as published by the Hub on 2026-10-05, plus sha256 (the LFS
+# oid) where the Hub stores an LFS object — plain git blobs carry no oid
+# and are size-checked only. Re-check with:
+# curl -s "https://huggingface.co/api/models/peonist-ai/halogen-qwen3.8-27b/tree/main?recursive=true"
+SIZES=(35865565184 12809320 6722759 3353259 8952 17928)
+SHAS=(274c3fc767fb57faf025dd03c76376b1fc1b32448aef45aaf02d1898fa962089
+      0997f410c57a1f4e53b09e4be8f4a172d90edd9564368fb0847030937229b9f3 - - - -)
+
+VERIFY_ONLY=0
+[ "${1:-}" = "--verify-only" ] && VERIFY_ONLY=1
+
+mkdir -p "$DEST"
+
+for i in "${!FILES[@]}"; do
+    f="${FILES[$i]}"; want="${SIZES[$i]}"; sha="${SHAS[$i]}"
+    path="$DEST/$f"
+    mkdir -p "$(dirname "$path")"
+    if [ "$VERIFY_ONLY" -eq 0 ]; then
+        # A file already at the published size skips the fetch entirely:
+        # resuming it would ask the CDN for a range past EOF (416 -> curl
+        # error under set -e), and re-pulling 33.4 GiB to prove a point is
+        # not what --verify-only is for. sha256 below still decides.
+        if [ -f "$path" ] && [ "$(stat -c %s "$path")" = "$want" ]; then
+            echo "HAVE  $f ($want bytes) — skipping fetch"
+        else
+            echo "FETCH $f -> $path"
+            # HF answers /resolve/ with a 302 to the Xet CDN and curl DROPS
+            # the Range header across redirects — a naive -C - then
+            # "resumes" by pulling the ~1 KB redirect body (measured
+            # 2026-10-05: file frozen 14 MiB short, curl exit 0). Resolve
+            # the effective URL first, then resume against the CDN, which
+            # honors Range.
+            url=$(curl -sfIL -o /dev/null -w '%{url_effective}' "$BASE/$f") \
+                || { echo "cannot resolve $BASE/$f" >&2; exit 1; }
+            curl -fL --retry 5 -C - -o "$path" "$url"
+        fi
+    fi
+    [ -f "$path" ] || { echo "MISSING $path (run without --verify-only)" >&2; exit 1; }
+    got="$(stat -c %s "$path")"
+    [ "$got" = "$want" ] || { echo "SIZE $path: $got != $want" >&2; exit 1; }
+    echo "OK   $f ($got bytes)"
+done
+
+echo "byte counts match; verifying sha256 where the Hub publishes an LFS oid..."
+for i in "${!FILES[@]}"; do
+    f="${FILES[$i]}"; sha="${SHAS[$i]}"
+    if [ "$sha" = "-" ]; then
+        echo "OK   $f (plain git blob, size-checked)"
+        continue
+    fi
+    got="$(sha256sum "$DEST/$f" | cut -d' ' -f1)"
+    [ "$got" = "$sha" ] || { echo "SHA256 $f: $got != $sha" >&2; exit 1; }
+    echo "OK   $f sha256"
+done
+echo "DOWNLOAD_DONE $DEST"
@@ -0,0 +1,116 @@
+#!/usr/bin/env bash
+# Download kingjones777/Qwen3.8-27B-Uncensored-ROCmFP4-STRIX-MTP-GGUF for
+# frank-large (Dankclaw iGPU). Public repo — no auth header, and never a token
+# on the command line (it is world-readable in /proc/<pid>/cmdline).
+#
+# The payload is the repo's Q6 AGENT rung:
+#   * Qwen3.8-27B-Uncensored-Q6_0_ROCMFPX_AGENT.gguf — ftype 114, 23.15 GiB.
+#     Q6_0_ROCMFPX tensor type (102) + the AGENT routing, which keeps the
+#     output-head and attention tensors at plain Q8_0 — the variant the card
+#     names for MTP serving ("Take Q6 AGENT (114) if you want protected
+#     heads"). Like every ROCmFPX file it loads on nothing but the fork.
+#
+# The other two files frank-large needs are NOT re-downloaded:
+#   * mtp-Qwen3.8-27B-Q4_0.gguf — the uncensored repo redistributes the
+#     ALIGNED repo's draft head byte for byte (identical Hub LFS oid
+#     051a1764…), so it is hardlinked out of the aligned payload dir when that
+#     dir holds the verified bytes, and fetched from this repo otherwise.
+#   * mmproj-Qwen3.8-27B-BF16.gguf — this repo ships a Q8_0 projector; the
+#     fleet keeps the BF16 one, which is the projector that was verified 4/4
+#     on spatial ground truth on this host. Hardlinked the same way, fetched
+#     from the ALIGNED repo if it is not on disk.
+#
+# Fresh downloads, no resume (`curl -C -` once produced an oversized file on a
+# 31 GiB payload); -f/--fail-with-body so an HTTP 401 is not a success, then
+# byte count and the Hub LFS sha256. Idempotent: complete files are skipped.
+#
+# Usage:
+#   scripts/download-frank-large-kingjones-uncensored.sh [DEST_DIR] [--verify-only]
+set -euo pipefail
+
+REPO="kingjones777/Qwen3.8-27B-Uncensored-ROCmFP4-STRIX-MTP-GGUF"
+ALIGNED_REPO="kingjones777/Qwen3.8-27B-ROCmFP4-STRIX-MTP-GGUF"
+BASE="https://huggingface.co/${REPO}/resolve/main"
+ALIGNED_BASE="https://huggingface.co/${ALIGNED_REPO}/resolve/main"
+SOURCE_DIR="${SOURCE_DIR:-/home/dank/models/kingjones-Qwen3.8-27B-ROCmFP4-STRIX-MTP}"
+
+DEST=""
+VERIFY_ONLY=0
+for arg in "$@"; do
+    case "$arg" in
+        --verify-only) VERIFY_ONLY=1 ;;
+        *) DEST="$arg" ;;
+    esac
+done
+DEST="${DEST:-/home/dank/models/kingjones-Qwen3.8-27B-Uncensored-ROCmFP4-STRIX-MTP}"
+
+# name bytes sha256 base_url may_link — may_link=1 means the same bytes may
+# already exist in SOURCE_DIR (hardlink them instead of re-fetching).
+FILES=(
+  "Qwen3.8-27B-Uncensored-Q6_0_ROCMFPX_AGENT.gguf 24856607008 e966deb3d1a5a1b9ca7765126292f2d822f3dccdedb3b4872e673dec1cddc199 $BASE 0"
+  # The draft head is redistributed byte-identical by both repos; the BF16
+  # projector only exists in the aligned repo.
+  "mtp-Qwen3.8-27B-Q4_0.gguf 1680271648 051a1764cff8c4f3ee6ae8b00593a0364c7539c67fa50ffc58f3f96509fca38e $BASE 1"
+  "mmproj-Qwen3.8-27B-BF16.gguf 931145888 de2a49866988ea272c43dc2a43ee2662c25725a717ca5b9059ea420a97f53fe8 $ALIGNED_BASE 1"
+)
+
+matches() {  # matches FILE SIZE SHA — byte count AND LFS sha256
+    [[ -f "$1" ]] && [[ "$(stat -c%s "$1")" == "$2" ]] \
+        && sha256sum -c --quiet <(printf '%s  %s\n' "$3" "$1")
+}
+
+mkdir -p "$DEST"
+status=0
+
+fetch() {  # fetch NAME SIZE SHA BASE_URL
+    local name="$1" size="$2" sha="$3" url="$4" f="$DEST/$1"
+    echo "FETCHING  $name"
+    curl -fsSL --fail-with-body -o "$f.part" "$url/$name"
+    mv "$f.part" "$f"
+    if [[ "$(stat -c%s "$f")" != "$size" ]]; then
+        echo "SIZE FAIL $name ($(stat -c%s "$f") != $size)"
+        return 1
+    fi
+    if ! sha256sum -c --quiet <(printf '%s  %s\n' "$sha" "$f"); then
+        echo "SHA FAIL  $name"
+        return 1
+    fi
+    echo "VERIFIED  $name"
+}
+
+for entry in "${FILES[@]}"; do
+    read -r name size sha url may_link <<<"$entry"
+    f="$DEST/$name"
+    if matches "$f" "$size" "$sha"; then
+        echo "OK        $name"
+        continue
+    fi
+    if (( VERIFY_ONLY )); then
+        echo "MISSING   $name"
+        status=1
+        continue
+    fi
+    # Shared bytes: prefer the local copy (same filesystem, so a hardlink costs
+    # nothing) over re-downloading gigabytes that hash identically.
+    if [[ "$may_link" == 1 && -n "${SOURCE_DIR:-}" ]] && matches "$SOURCE_DIR/$name" "$size" "$sha"; then
+        echo "LINKING   $name from $SOURCE_DIR"
+        ln -f "$SOURCE_DIR/$name" "$f" 2>/dev/null || cp -f "$SOURCE_DIR/$name" "$f"
+        if matches "$f" "$size" "$sha"; then
+            echo "VERIFIED  $name"
+            continue
+        fi
+        echo "SHA FAIL  $name (linked copy)"
+        status=1
+        continue
+    fi
+    if ! fetch "$name" "$size" "$sha" "$url"; then
+        status=1
+    fi
+done
+
+if (( status == 0 )); then
+    echo "DOWNLOAD_DONE $DEST"
+else
+    echo "DOWNLOAD_FAILED $DEST" >&2
+fi
+exit "$status"
@@ -0,0 +1,91 @@
+#!/usr/bin/env bash
+# Download kingjones777/Qwen3.8-27B-ROCmFP4-STRIX-MTP-GGUF for frank-large
+# (Dankclaw iGPU). Public repo — no auth header, and never a token on the
+# command line (it is world-readable in /proc/<pid>/cmdline).
+#
+# The payload is the repo's highest-quality target plus its MTP draft head and
+# vision projector:
+#   * Qwen3.8-27B-Q8_0_ROCMFPX_AGENT.gguf — ftype 115, 187 tensors at plain
+#     Q8_0 and 311 at GGML_TYPE_Q8_0_ROCMFPX (103). The AGENT routing keeps the
+#     output-head tensors at full Q8_0 (187 vs 2 in the non-AGENT build), which
+#     is the variant to take when serving with the MTP head.
+#   * mtp-Qwen3.8-27B-Q4_0.gguf — the draft head. Q4_0, NOT Q8_0: the head's
+#     per-step cost is its projection over a 248K vocabulary, so the cheaper
+#     head is faster end to end at equal acceptance (author: +6.0% at n-max 1,
+#     +10.2% at n-max 2).
+#   * mmproj-Qwen3.8-27B-BF16.gguf — the vision tower (verified 4/4 on spatial
+#     ground truth with -fa on).
+#
+# Fresh downloads, no resume (`curl -C -` once produced an oversized file on a
+# 31 GiB payload); -f/--fail-with-body so an HTTP 401 is not a success, then
+# byte count and the Hub LFS sha256. Idempotent: complete files are skipped.
+#
+# The repo's own SHA256SUMS covers only the ROCmFP4 files, so the hashes below
+# come from the Hub LFS oids (api/models/<repo>/tree/main).
+#
+# Usage:
+#   scripts/download-frank-large-kingjones.sh [DEST_DIR] [--verify-only]
+set -euo pipefail
+
+REPO="kingjones777/Qwen3.8-27B-ROCmFP4-STRIX-MTP-GGUF"
+BASE="https://huggingface.co/${REPO}/resolve/main"
+
+DEST=""
+VERIFY_ONLY=0
+for arg in "$@"; do
+    case "$arg" in
+        --verify-only) VERIFY_ONLY=1 ;;
+        *) DEST="$arg" ;;
+    esac
+done
+DEST="${DEST:-/home/dank/models/kingjones-Qwen3.8-27B-ROCmFP4-STRIX-MTP}"
+
+# name bytes sha256
+FILES=(
+  "Qwen3.8-27B-Q8_0_ROCMFPX_AGENT.gguf 28214425952 f3ef2bdbf56e244fff14b72114dc2aa300b84a4cb4eaf54f2e5e4da1cfcae10b"
+  "mtp-Qwen3.8-27B-Q4_0.gguf 1680271648 051a1764cff8c4f3ee6ae8b00593a0364c7539c67fa50ffc58f3f96509fca38e"
+  "mmproj-Qwen3.8-27B-BF16.gguf 931145888 de2a49866988ea272c43dc2a43ee2662c25725a717ca5b9059ea420a97f53fe8"
+)
+
+mkdir -p "$DEST"
+status=0
+for entry in "${FILES[@]}"; do
+  read -r name size sha <<<"$entry"
+  f="$DEST/$name"
+  have=0
+  if [[ -f "$f" ]] && [[ "$(stat -c%s "$f")" == "$size" ]]; then
+    if sha256sum -c --quiet <(printf '%s  %s\n' "$sha" "$f"); then
+      have=1
+    fi
+  fi
+  if (( have )); then
+    echo "OK        $name"
+    continue
+  fi
+  if (( VERIFY_ONLY )); then
+    echo "MISSING   $name"
+    status=1
+    continue
+  fi
+  echo "FETCHING  $name"
+  tmp="$f.part"
+  curl -fsSL --fail-with-body -o "$tmp" "$BASE/$name"
+  mv "$tmp" "$f"
+  if [[ "$(stat -c%s "$f")" != "$size" ]]; then
+    echo "SIZE FAIL $name ($(stat -c%s "$f") != $size)"
+    status=1
+    continue
+  fi
+  if ! sha256sum -c --quiet <(printf '%s  %s\n' "$sha" "$f"); then
+    echo "SHA FAIL  $name"
+    status=1
+    continue
+  fi
+  echo "VERIFIED  $name"
+done
+if (( status == 0 )); then
+  echo "DOWNLOAD_DONE $DEST"
+else
+  echo "DOWNLOAD_FAILED $DEST" >&2
+fi
+exit "$status"
@@ -0,0 +1,76 @@
+#!/usr/bin/env bash
+# Download the Q4_K_XL frank-large payload on Dankclaw:
+#   unsloth/Qwen3.8-27B-GGUF  ->  Qwen3.8-27B-UD-Q4_K_XL.gguf
+#
+#   bash scripts/download-frank-large-q4.sh
+#   bash scripts/download-frank-large-q4.sh --verify-only
+#
+# Integrity: the file is checked against its declared byte count and sha256
+# (the LFS oid from the Hub tree API, 2026-10-01). NO AUTH HEADER IS USED:
+# the repo is public and ungated (gated: false), and a token on the curl
+# command line is world-readable via /proc/<pid>/cmdline. If this ever needs
+# a token, put it in a mode-600 file and pass it with `curl -K`.
+#
+# Layout after a successful run (MODEL_DIR):
+#   Qwen3.8-27B-UD-Q4_K_XL.gguf   17559178144  (16.35 GiB)
+#
+# WHY Q4 AND NOT THE Q8 SWIFT PAYLOAD: this is the vanilla (non-abliterated)
+# Qwen3.8-27B at UD-Q4_K_XL, ~13 GiB smaller than the Swift UD-Q8_K_XL. Every
+# GiB of weights saved is a GiB of f16 KV the endpoint can hold instead: the
+# KV costs 64 KiB/token (16 full-attention layers x 4 KV heads x 256 head_dim
+# x 2 (K,V) x 2 B), so 4 sessions x 163840 ctx = 640,000 tokens = 39.06 GiB.
+#
+# The speculative drafter is UNCHANGED and still fetched by its own script
+# (scripts/download-frank-large.sh, z-lab's Hub repo). That name must not
+# appear here at all — the frank-large seam greps THIS file for it to prove
+# the drafter was not folded into the payload fetch.
+#
+# NOTE ON MTP: this payload has no baked-in MTP head (the Hub repo ships one
+# separately as MTP/mtp-Qwen3.8-27B-Q4_0.gguf). frank-large does not need it —
+# spec decode here is the block-diffusion drafter, not MTP.
+set -euo pipefail
+
+MODEL_DIR="/home/dank/models/Qwen3.8-27B-UD-Q4_K_XL"
+REL="Qwen3.8-27B-UD-Q4_K_XL.gguf"
+SIZE=17559178144
+SHA=3f227079003add2511437e5b1e94812e363385225bf6a9b47b0054a72bc8b01e
+URL="https://huggingface.co/unsloth/Qwen3.8-27B-GGUF/resolve/main/${REL}"
+
+mode="download"
+[[ "${1:-}" == "--verify-only" ]] && mode="verify"
+
+if [[ "$mode" == "download" ]]; then
+    mkdir -p "$MODEL_DIR"
+    if [[ -s "$MODEL_DIR/$REL" ]]; then
+        echo "already present: $MODEL_DIR/$REL"
+    else
+        echo "fetching $REL (17559178144 bytes) — this is ~16.4 GiB"
+        # -f so an HTTP error is a failure, not a 153-byte "artifact".
+        # -C - so an interrupted run resumes instead of restarting.
+        curl -fL -C - --retry 5 --retry-delay 3 -o "$MODEL_DIR/$REL" "$URL"
+    fi
+fi
+
+echo "verifying $MODEL_DIR against declared size + sha256"
+python3 - "$MODEL_DIR" "$REL" "$SIZE" "$SHA" <<'PY'
+import hashlib, os, sys
+model_dir, rel, size, sha = sys.argv[1], sys.argv[2], int(sys.argv[3]), sys.argv[4]
+dest = os.path.join(model_dir, rel)
+if not os.path.isfile(dest):
+    print(f"  MISSING {rel}")
+    sys.exit(1)
+actual = os.path.getsize(dest)
+if actual != size:
+    print(f"  SIZE MISMATCH {rel}: {actual} != {size}")
+    sys.exit(1)
+h = hashlib.sha256()
+with open(dest, "rb") as f:
+    for chunk in iter(lambda: f.read(1 << 24), b""):
+        h.update(chunk)
+if h.hexdigest() != sha:
+    print(f"  SHA256 MISMATCH {rel}: {h.hexdigest()}")
+    sys.exit(1)
+print(f"  OK {rel} ({actual} bytes, sha256 matches the Hub LFS oid)")
+PY
+
+echo "Q4_K_XL payload complete: $MODEL_DIR"
@@ -0,0 +1,87 @@
+#!/usr/bin/env bash
+# Download the frank-small payload for llama.cpp on the RTX 3090:
+# ornith-ai/Ornith-1.5-9B-GGUF:Q8_0 plus its vision projector
+# (the GGUF carries no `v.*` tensors — the tower lives only in the mmproj, so
+# without this file the endpoint is text-only by construction). Public repo —
+# no auth header, and never a token on the command line (it is world-readable
+# in /proc/<pid>/cmdline).
+#
+# Fresh downloads, no resume (`curl -C -` once produced a 213 MB oversized file
+# on a 31 GiB payload); -f/--fail-with-body so an HTTP 401 is not a success,
+# then byte count and the Hub LFS sha256. Idempotent: complete files are
+# skipped.
+#
+# 2026-10-05 (Frank): the rung is Ornith-1.5-9B Q8_0. The payload moved off
+# MiMo-V2.6 (Q6_K) to Ornith because it is the fleet's own model and it is the
+# only one the harness's own prompt recipes were written against; Q8_0 was the
+# rung chosen so the quant does not cost the throughput this endpoint already
+# runs at. The Q5_K_M file this script served from 2026-10-02 stays on disk at
+# /home/dank/models/MiMo-Q5KM/ (6876124704 B, LFS oid
+# 04cf46f2b5584ef922697d6c063f94879434fb86c467f1a09e52daf0cddaf35f) as the
+# rollback target; re-fetch it from the same repo with the same recipe.
+#
+# Usage:
+#   scripts/download-frank-small.sh [DEST_DIR] [--verify-only]
+set -euo pipefail
+
+REPO="ornith-ai/Ornith-1.5-9B-GGUF"
+BASE="https://huggingface.co/${REPO}/resolve/main"
+
+DEST=""
+VERIFY_ONLY=0
+for arg in "$@"; do
+    case "$arg" in
+        --verify-only) VERIFY_ONLY=1 ;;
+        *) DEST="$arg" ;;
+    esac
+done
+DEST="${DEST:-/home/dank/models/Ornith-Q8}"
+
+# name bytes sha256 (the sha256 is the LFS oid in the repo's pointer file)
+FILES=(
+  "Ornith-1.5-9B-Q8_0.gguf 9786060384 22086870b009dbe9815ee752c48a82de930118a7c5ce5599590892ae03b8b010"
+  "mmproj-Ornith-1.5-9B-BF16.gguf 921704672 626f9f90627402a6bf4a999111d0fbd69b5fcca7aa8ba089d69e5f10e8858e1d"
+)
+
+mkdir -p "$DEST"
+status=0
+for entry in "${FILES[@]}"; do
+  read -r name size sha <<<"$entry"
+  f="$DEST/$name"
+  have=0
+  if [[ -f "$f" ]] && [[ "$(stat -c%s "$f")" == "$size" ]]; then
+    if sha256sum -c --quiet <(printf '%s  %s\n' "$sha" "$f"); then
+      have=1
+    fi
+  fi
+  if (( have )); then
+    echo "OK        $name"
+    continue
+  fi
+  if (( VERIFY_ONLY )); then
+    echo "MISSING   $name"
+    status=1
+    continue
+  fi
+  echo "FETCHING  $name"
+  tmp="$f.part"
+  curl -fsSL --fail-with-body -o "$tmp" "$BASE/$name"
+  mv "$tmp" "$f"
+  if [[ "$(stat -c%s "$f")" != "$size" ]]; then
+    echo "SIZE FAIL $name ($(stat -c%s "$f") != $size)"
+    status=1
+    continue
+  fi
+  if ! sha256sum -c --quiet <(printf '%s  %s\n' "$sha" "$f"); then
+    echo "SHA FAIL  $name"
+    status=1
+    continue
+  fi
+  echo "VERIFIED  $name"
+done
+if (( status == 0 )); then
+  echo "DOWNLOAD_DONE $DEST"
+else
+  echo "DOWNLOAD_FAILED $DEST" >&2
+fi
+exit "$status"
@@ -1,45 +1,113 @@
-# frank-large — gufo (gufo-org/gufo) serving
-# Swift-1.5-Qwen3.8-27B-Uncensored (abliterated) on the Strix Halo iGPU.
+# frank-large.env — tunables for the frank-large endpoint on Dankclaw's
+# Strix Halo iGPU (gfx1151), host port 7770.
 #
-# These are the tunables read by scripts/serve-frank-large.sh. The flag
-# list is owned by the serve script; this file only sets fleet values.
+# RUNTIME (2026-10-05, Frank): halogen — the closed-source C++/HIP engine
+# (github.com/peonist-ai/halogen-server, image ghcr.io/peonist-ai/halogen,
+# digest-pinned below) serving peonist-ai/halogen-qwen3.8-27b, a ~6.32 bpw
+# NVFP4-derivative quant of the BASE Qwen/Qwen3.8-27B (35,865,565,184 B,
+# 1352 tensors), as a podman container under a root unit, driven by
+# scripts/serve-frank-large.sh. This CUTS OVER the
+# 2026-10-02..10-03 ROCmFPX llama.cpp shape (uncensored
+# Q6_0_ROCMFPX_AGENT + MTP, 27.6-28.1 t/s): rollback = revert the swap
+# commit's trio; the ROCmFPX payload, draft head and build all stay on
+# disk at /home/dank/models/kingjones-Qwen3.8-27B-Uncensored-ROCmFP4-STRIX-MTP/.
 #
-# DO NOT name a new tunable TEMP, HOST or GPU_MAX_HEAP_SIZE: the HIP
-# runtime reads those bare (the 2026-08-13 TEMP incident). HOST is the one
-# exception because the serve script itself consumes it for --host.
+# WHY THE RUNTIME MOVED WITH THE MODEL: the payload is .hgn — per its own
+# card it "will not load in transformers, vLLM, or llama.cpp". Accepted
+# consequences (Frank, 2026-10-05): TEXT-ONLY (halogen ignores the model's
+# vision encoder; the fleet's vision path is frank-small) and the CENSORED
+# base (this is a finetune of Qwen/Qwen3.8-27B, not the abliterated file —
+# refusals may return). What it buys: vendor-measured decode DFlash2 31.7
+# t/s mean (vs this endpoint's MTP 27.6-28.1 and the old halogen-era MTP
+# 26.9), prefill ~566 t/s at 32k (was ~400 on the sibling engine), 1.4-2.3x
+# wall-clock on prefill-heavy agent traffic, at HIGHER effective precision
+# (~6.32 bpw vs ~6.6 with the old W4A4 promotion — see FLAGS.md).
+#
+# SHAPE — the two knobs that are NOT free:
+#   HALOGEN_KV_SLOTS=1 KEEPS SPECULATION ON. halogen makes batching and
+#   speculation mutually exclusive: with >1 slot the drafter is disabled
+#   and per-stream decode falls to serial speed (the sibling engine
+#   measured 41 t/s -> 14 t/s when slots were raised, and halogen's own
+#   FLAGS.md prints the same law). frank-large was already a one-session
+#   endpoint under ROCmFPX (PARALLEL=1), so slots=1 is the same contract,
+#   not a regression.
+#   HALOGEN_SLOT_CTX=200700 (Frank's window) is the per-conversation hard
+#   ceiling — a prompt over it is a 400 naming the limit, never a silent
+#   truncation. KV pool = slots x slot_ctx x 64 KiB = 12.2 GiB. Budget
+#   against the 121 GiB of unified memory: weights 33.4 GiB (mmap'd — page
+#   cache, not anon), KV 12.2, engine working set + auto prompt cache
+#   ~20-30, host baseline ~10 -> ~75-85 GiB in use. The 2026-09-16 halogen
+#   era ran the same engine class on this box with LESS headroom.
+#   HALOGEN_DRAFTER=2 (DFlash2) is the shipped default and the fastest
+#   exact drafter on this checkpoint; the .hgn BAKES BOTH drafters (MTP +
+#   DFlash2), so unlike the Flash-Next era there is no sidecar to mount
+#   and no --dflash flag — the drafter is an int knob, byte-identical
+#   output to greedy verification either way.
+#
+# CLIENT CONTRACT deltas vs the ROCmFPX shape (the harness side lives in
+# ~/.omp/agent/models.yml):
+#   * NO server-side sampler defaults: a request without temperature
+#     decodes GREEDY. The harness sends temp 1.0 / top_p 0.95 via
+#     extraBody, so that wire recipe is load-bearing now, not cosmetic.
+#   * max_tokens defaults to 8192 PER REQUEST and INCLUDES thinking; a
+#     reasoning reply over a small budget returns empty content with
+#     finish_reason "length". The harness maxTokens is 57344: compaction
+#     fires at 70% of 200700 = 140,490 and 140,490 + 57,344 = 197,834 <=
+#     200,700 keeps prompt + budget inside the pool (the 262144-era 65536
+#     would overflow). Requests over HALOGEN_MAX_TOKENS_CAP are a 400
+#     naming the cap; cap and queue timeout are COUPLED (see FLAGS.md
+#     table) — do not move one without the other.
+#   * response_format json_object/json_schema is a 400 (no constrained
+#     decoding in this build).
+#   * Thinking lands in reasoning_content — same field as the llama.cpp
+#     era, so the harness compat block does NOT change. reasoning_effort
+#     accepts none|low|xhigh on this build (template default xhigh); the
+#     models.yml fold (high->xhigh) still applies.
+#   * The engine's token protocol has no auth and binds loopback INSIDE
+#     the container; only the frontend port is published. Keep it that way.

-# The target payload (scripts/download-frank-large-swift.sh): the serve
-# script refuses to start without the main GGUF and the DFlash2 drafter.
-# Swapped in from unsloth/Qwen3.8-27B-GGUF:UD-Q8_K_XL on 2026-09-28. The two
-# are structurally interchangeable for gufo (same qwen35 architecture, the
-# same 866 tensors in the same order, the same per-tensor Q8_0/F32/BF16
-# layout, the same MTP head), so only this path changed.
-MODEL_DIR=/home/dank/models/Swift-1.5-Qwen3.8-27B-Uncensored-MTP-UD-Q8_K_XL
-MODEL_FILE=/home/dank/models/Swift-1.5-Qwen3.8-27B-Uncensored-MTP-UD-Q8_K_XL/Swift-1.5-Qwen3.8-27B-Uncensored-Dynamic-MTP-UD-Q8_K_XL.gguf
+# Image: digest-pinned (fetch law: tag pulls silently re-point; the digest
+# is what was qualified). 0.1.4 = the version the model card names.
+FRANK_LARGE_IMAGE=ghcr.io/peonist-ai/halogen@sha256:dc0a39a0016d6cfc58a197978febaafdf8d28403f724ded6111d98b5fb7ac0ea
+# Container identity: the halogen container name.
+FRANK_LARGE_CONTAINER=frank-large-halogen
+# Old quadlet-era container identity; the serve script clears stale names.
+FRANK_LARGE_LEGACY_CONTAINERS=frank-large
+# host:container — the frontend listens on 8731 inside (HALOGEN_API_PORT
+# default); the engine port (8730) is never published.
+FRANK_LARGE_PUBLISH=7770:8731

-# The DFlash2 drafter is UNCHANGED by the model swap and still lives beside
-# the previous target, so it carries its own directory. It has a sliding
-# window of 2048, which is what keeps its KV small (~40 MiB).
-DRAFTER_DIR=/home/dank/models/Qwen3.8-27B-UD-Q8_K_XL
-DRAFTER_FILE=/home/dank/models/Qwen3.8-27B-UD-Q8_K_XL/Qwen3.8-27B-DFlash2-Q4_K_M.gguf
+FRANK_LARGE_MODEL_DIR=/home/dank/models/halogen-qwen3.8-27b
+FRANK_LARGE_CHECKPOINT=qwen3.8-27b-p1w4d-d2.hgn
+FRANK_LARGE_CHECKPOINT_BYTES=35865565184
+FRANK_LARGE_CHECKPOINT_SHA256=274c3fc767fb57faf025dd03c76376b1fc1b32448aef45aaf02d1898fa962089
+# The tokenizer is a FLAT DIRECTORY, $FRANK_LARGE_MODEL_DIR/tokenizer, mounted
+# whole at /tokenizer — the model card ships `tokenizer/` and the engine's
+# HALOGEN_TOKENIZER default is that directory (the card's own docker run sets
+# no tokenizer flag). The Hub stores it as 5 files (sizes from the Hub API,
+# 2026-10-05): tokenizer.json 12809320 (LFS sha256
+# 0997f410c57a1f4e53b09e4be8f4a172d90edd9564368fb0847030937229b9f3),
+# vocab.json 6722759, merges.txt 3353259, chat_template.jinja 8952,
+# tokenizer_config.json 17928 (plain git blobs — size-checked, no LFS oid).
+# The serve gate and the deployer refuse a missing file or ANY symlink: an
+# HF cache snapshot symlinks each file into blobs/, and those links dangle
+# inside the container mount.
+FRANK_LARGE_TOKENIZER_FILES="tokenizer.json chat_template.jinja tokenizer_config.json merges.txt vocab.json"

-# The gufo runtime, built from source by scripts/build-gufo.sh.
-RUNTIME_DIR=/home/dank/GitPackages/gufo-v0.1.1
-
-# Fleet endpoint: the same port and alias every frank-large has served, so
-# no client, probe or models.yml entry changes address.
-ALIAS=frank-large
-HOST=0.0.0.0
-PORT=7770
-
-# One session, full 262144 context (the model's native window).
-CONTEXT_SIZE=262144
-SESSIONS=1
-
-# DFlash2 spec decode (a block-diffusion drafter). gufo sizes the draft
-# block itself; DRAFT_TOKENS caps the proposals per step (gufo's default is
-# 7, chosen over llama.cpp's fixed depth 4 because gufo's adaptive policy
-# shortens the block when acceptance drops) and DRAFT_POLICY is the block
-# length controller.
-DRAFT_TOKENS=7
-DRAFT_POLICY=adaptive
+# --- engine flags (docs/FLAGS.md of the engine repo; anything unset stays
+# at the image default, which is what was qualified) ------------------------
+HALOGEN_MODEL_ID=frank-large
+HALOGEN_SLOT_CTX=200700
+HALOGEN_KV_SLOTS=1
+HALOGEN_DRAFTER=2
+# Coupled pair (FLAGS.md table): the cap bounds how long ONE request holds
+# the GPU, the timeout bounds how long the NEXT client waits. 65536 at
+# ~10 t/s serial worst case needs > 6550 s queue timeout; the defaults
+# honor that. Do not lower the timeout alone.
+HALOGEN_MAX_TOKENS_CAP=65536
+HALOGEN_QUEUE_TIMEOUT=7200
+# Engine prompt cache: EMPTY = auto (sized from available memory at
+# startup). Pin an explicit MB only to argue with the auto figure — a
+# full-context entry is ~18 GB at 262K, so a too-small pin yields a cache
+# that reports enabled and never hits (the engine warns at startup).
+#HALOGEN_CACHE_MB=
@@ -1,5 +1,5 @@
 [Unit]
-Description=gufo (gufo-org/gufo, ROCm gfx1151) Qwen3.8-27B UD-Q8_K_XL, DFlash2 spec decode, 262144 ctx / 1 session, root systemd unit (dankclaw, frank-large, Strix Halo iGPU)
+Description=halogen container (peonist-ai/halogen-qwen3.8-27b, DFlash2 spec decode, 1 slot x 200700, text-only), root systemd unit (dankclaw, frank-large, Strix Halo iGPU)
 After=network-online.target
 Wants=network-online.target

@@ -13,8 +13,19 @@ Group=dank
 LimitMEMLOCK=infinity
 Environment="FRANK_LARGE_ENV_FILE=/home/dank/serverAppConfigs/scripts/frank-large.env"
 ExecStart=/home/dank/serverAppConfigs/scripts/serve-frank-large.sh
-Restart=on-failure
+# systemd stop alone cannot stop the container child; stop it explicitly
+# (frank-primary pattern). The engine drains on SIGTERM.
+ExecStop=/usr/bin/podman stop -t 60 frank-large-halogen
+TimeoutStartSec=1200
+TimeoutStopSec=120
+# always, not on-failure: an engine that exits 0 after an internal fault
+# would otherwise leave the endpoint down until someone notices.
+Restart=always
 RestartSec=10
+# The engine mmaps 33.4 GiB of weights and keeps a multi-GiB working set
+# on a box that also hosts frank-small, ComfyUI and the Caddy/observability
+# stack; make it a late OOM candidate, not the first.
+OOMScoreAdjust=-500

 [Install]
 WantedBy=multi-user.target
@@ -1,38 +1,35 @@
 # frank-primary.env — tcclaviger RDNA4 vLLM production config (dankserver :7770).
 #
-# 2026-09-29 cutover from the R9V radiance-vLLM fork (its launcher/shim/state
-# machinery is retired; /home/dank/r9v-state/{q8xl,uncensored} stay on disk as
-# the rollback target, and git tag frank-primary-r9v-q8-20260929 is the last
-# pre-cutover commit). The engine is the tcclaviger RDNA4 vLLM fork — the same
-# digest-pinned image the 2026-09-29 tuning campaign measured — serving
-# orcarouter/Qwen3.8-Flash-Next-Uncensored-FP8 (~172.8 GiB, 131 shards) with
-# the MTP4 drafter on the two R9700s.
-#
-# Every number here was QUALIFIED by boot probe on 2026-09-29 (not inherited):
-#   * MAXLEN 200000 x SEQS 3 (600k total) at KV q8/q8 fits with headroom:
-#     the engine measured "GPU KV cache size: 729,545 tokens; Maximum
-#     concurrency for 200,000 tokens per request: 3.65x" at KV_BYTES 7.2 GiB
-#     per rank. KV_BYTES is PER RANK — sizing it to the global requirement
-#     OOMs the worker (measured twice: 16 GiB and 9 GiB both die with ~7.5 GiB
-#     free; 7.2 GiB boots).
-#   * --kv-cache-dtype fp8 is the fleet's q8/q8 KV ask (e4m3 for BOTH K and V
-#     — the stock flag quantizes them together, which is exactly what was
-#     requested here). Accepted by the attention path; qualified with a
-#     completion probe.
-#   * PREFETCH IS ABSENT BY LAW: expert-cache telemetry measured the
-#     prefetcher at 36% precision (36,720 slabs prefetched, 13,369 served) —
-#     roughly half the gather traffic was waste; prefetch off is the tuned
-#     incumbent and the +40-51% decode came from deleting the flag. Never
-#     re-add --expert-prefetch without a fresh A/B.
-#   * TunableOp stays OFF: CLAV_TUNABLEOP_SWEEP silently skips under expert
-#     offload unless CLAV_TUNABLEOP_WITH_OFFLOAD=1, and forcing it corrupts
-#     numerics (repetition loops, early halts, invalid measurements).
-# Decode at the tuned shape: 40.57/37.44/37.57 tok/s (166/1087/4064 ctx) at
-# the 32k window; 42.94/34.55/36.09 at this 200k shape (arena 6,032 slots).
+# 2026-10-03 shape G (as shipped, after the 1.00 revert): the CUDA-graph plan
+# term dropped from the analytic 0.55 GiB to the measured 0.07 GiB (0 base +
+# 12 MiB/graph), and UTIL at the 0.982 law. MEASURED on the live deploy: arena
+# 17.77 GiB / 7,762 slots, KV 401,408 tokens / 2.00x concurrency (the first
+# shape holding two full-length conversations with real spare), driver used
+# 30.97 GiB of 31.86 — 0.89 GiB of real free VRAM per card, held through a
+# live 4,096-token generation. UTIL stays 0.982 ALWAYS: 1.00 is unbootable on
+# this fork (worker init refuses budget >= device total) and 0.13-GiB-margin
+# boots die under load.
+# `budget - non-expert - KV reserve - graphs - overhead` and the KV reserve is a
+# registry constant, so those two terms are the whole flag surface.
+# 2026-10-03 shape F (absorbed): MNBT 2048 + the graph term pinned. Measured on
+# the live deploy: arena 17.52 GiB / 7,653 slots, budget 31.29, graphs 0.32.
+# 2026-10-02 shape E: fp8 KV, MAXLEN 200704 x SEQS 3, pool 3.15e9/rank, and
+# --expert-precise-memory ON. The window shrink from the native 262144 and the
+# --max-num-seqs 3 lock both come from shape D (SEQS=2 CANNOT boot this model —
+# the measured 2x2 is below); shape E adds the precise-overhead plan, which
+# trades the default 7.60 GiB compile headroom for the fitted 5.25 GiB and puts
+# the 2.35 GiB difference into the expert arena: 6,496 -> 7,522 slots (+16%).
+# The pool is deliberately NOT cut (option A): the pool is not an arena lever
+# (measured), so 3.15e9 keeps shape D's 1.99x concurrency while the arena grows.
+# Same digest-pinned tcclaviger RDNA4 fork. Rollback for the fp8+MTP4 era is git
+# tag frank-primary-fp8-mtp-20261001.
 #
 # HIP reads bare env vars TEMP, HOST, GPU_MAX_HEAP_SIZE — every key here is
-# prefixed FRANK_PRIMARY_ so none can leak.
-
+# prefixed FRANK_PRIMARY_ so none can leak. The one bare name we WANT inside the
+# container (GPU_MAX_HW_QUEUES) is held here under the prefix and passed on the
+# run line as `-e GPU_MAX_HW_QUEUES=…`.
+#
+#
 # --- runtime ----------------------------------------------------------------
 # Digest-pinned: the floating :latest tag moved once already. Local alias
 # qwen-fp8-runtime:tuned-20260929 (image ID 3322dbbbc96d) points at the same
@@ -50,12 +47,316 @@ FRANK_PRIMARY_SERVED_NAME=frank-primary
 FRANK_PRIMARY_PUBLISH=7770:7770
 # Containers removed before every start: the retired R9V identity and the ad
 # hoc experiment identity (both hold the GPUs / name registrations).
-FRANK_PRIMARY_LEGACY_CONTAINERS=frank-primary-r9v qwen-fp8
+FRANK_PRIMARY_LEGACY_CONTAINERS="frank-primary-r9v qwen-fp8"

-# --- qualified engine shape ---------------------------------------------------
-FRANK_PRIMARY_MAXLEN=200000
+# --- qualified engine shape (2026-10-02, shape E) ---------------------------
+# WHAT THE SLOT COUNT DOES AND DOES NOT DO. The endpoint advertises 3 slots
+# (SEQS 3), but the POOL, not the slot count, decides how many full-length
+# conversations are resident. MEASURED LIVE 2026-10-02 at this shape:
+#   GPU KV cache size: 398,658 tokens,
+#   Maximum concurrency for 200,704 tokens per request: 1.99x
+# (3,146,129,408 B reserved — 7,892 B/token). The per-token price is
+# SHAPE-DEPENDENT, not a constant: the 262144 shape priced at 7,701 B/token
+# (6,100,000,000 B -> 792,069 tokens), ~2.5% cheaper, so never extrapolate a
+# concurrency figure across geometries. Shorter requests use the spare blocks,
+# and a third LONG request queues rather than being refused. Verified: three
+# concurrent 67-token requests answered in parallel, ~4.1 s each.
+#
+# WHY NOT --max-num-seqs 2 (measured 2026-10-02, single-variable boot probes
+# against the then-deployed known-good shape, production stopped for each):
+#   SEQS 2 + MNBT 1024 -> Engine core initialization failed
+#   SEQS 2 + MNBT 2048 -> Engine core initialization failed
+#   SEQS 3 + MNBT 1024 -> healthy (792,069 tokens, 3.02x, arena 6,248)
+#   SEQS 3 + MNBT 2048 -> healthy (the deployed incumbent)
+# The failure is torch.fx ConstraintViolationError: query_start_loc.size()[0]
+# "marked dynamic but your code specialized it to be a constant (3)", raised
+# from AOT build_guards(strict_error=True) during profile_run -> _dummy_run.
+# model_runner.py sizes that input buffer max_num_reqs + 1 and _dummy_run sets
+# num_reqs = min(max_num_batched_tokens, max_num_reqs), so the width is
+# SEQS + 1 for ANY MNBT >= SEQS: 3 at SEQS 2, 4 at SEQS 3. MNBT only moves the
+# compile RANGE ((1,1024) vs (1,2048)) and cannot dodge it — the 2x2 above is
+# why MNBT is not a lever here. --enforce-eager bypasses the path but loses
+# CUDA graphs. Same failure class killed the 2026-10-01 bf16 2 x 262144 shape.
+#
+# THE ARENA LEVER IS max_model_len, NOT KV_BYTES (measured 2026-10-02). The
+# planner sizes expert cache as budget - non-expert - KV reserve - graphs -
+# overhead, and the KV reserve is registry-priced at 9190 B/token x max_model_len:
+#   MAXLEN 262144 -> reserve 2.48 GiB -> arena 14.30 GiB / 6,248 slots
+#   MAXLEN 200704 -> reserve 1.91 GiB -> arena 14.87 GiB / 6,496 slots
+# d(reserve) 0.57 GiB = d(arena) 0.57 GiB = +248 slots. The 6.1e9 -> 3.15e9
+# pool cut buys ZERO arena; it buys runtime HEADROOM (the shape it replaced
+# died with 0 bytes free on a real 995-token request). To grow the arena,
+# lower MAXLEN — not KV_BYTES. The OTHER arena lever is the overhead term:
+# --expert-precise-memory (shape E) takes it from 7.60 to 5.25 GiB and lands
+# those 2.35 GiB in the arena — 14.87 -> 17.22 GiB / 6,496 -> 7,522 slots.
+#
+# GATHER 32x64 stays: R4D_LRU_GATHER_CHUNKS=32 / R4D_LRU_GATHER_LANES=64
+# measured +18.7% decode vs default geometry and the win grows with depth.
+# Do not drop it.
+# CHUNKED PREFILL: the fork exposes the standard scheduler flags. --max-num-batched-tokens
+# (MNBT) is the lever; --enable-chunked-prefill is an auto-flag (True in SchedulerConfig)
+# and --prefill-schedule-interval is the gap between iterations. The running shape carries
+# --max-num-batched-tokens 2048 with --max-model-len 200704, so chunked prefill is ON and
+# already active: a 200704-token prompt is prefilled in ~98 iterations of 2048 tokens.
+# MEASURED 2026-09-29 on the ad hoc qwen-fp8 container (same fork, same payload, TP=2):
+# MNBT 8192 vs the default 2048 is NEUTRAL on decode at every depth, and small values
+# re-chunk prefill only — they hurt TTFT and never touch decode, because a decode step is
+# max_num_seqs x (spec+1) tokens, independent of MNBT. There is NO --ubatch-size
+# equivalent in vLLM: that is a llama.cpp compute-buffer concept (the frank-small pair
+# doubled on 2026-10-05 has no counterpart here). MNBT is therefore NOT a lever on this
+# shape — the deployed 2048 is the fork's default — and every change re-enters the
+# compile RANGE path, so FRANK_PRIMARY_CACHE_DIR pays one recompilation.
+# MNBT 2048 (Frank's call, 2026-10-03; the lock was 1024). The measured law
+# still holds: MNBT 512 doubles TTFT at 4064 ctx (2.46 s vs 1.36 s), 1024-4096
+# are interchangeable on decode, 12288 costs 2.5% plus arena. 2048 is ARENA-FREE
+# by construction: the registry's overhead model is
+# base + 0.22 GiB x max(0, (nbt - 4096)/1024) with the anchor at nbt 4096, so
+# every rung up to and including 4096 prices the SAME 5.25 GiB overhead — and
+# the fitted 5.25 was fitted at nbt 4096, i.e. our 2048 sits UNDER the anchor,
+# so the activation term is covered by a figure measured at a bigger batch.
+# SEQS 3 boots at both 1024 and 2048 (measured 2026-10-02; the SEQS=2
+# ConstraintViolationError is a SEQS property, not an MNBT one). Costs exactly
+# one recompilation: the compile RANGE is part of the cache key, so
+# FRANK_PRIMARY_CACHE_DIR pays for whichever rung is chosen once.
+# PREFETCH IS ABSENT BY LAW: expert-cache telemetry measured the prefetcher at
+# 36% precision (36,720 slabs prefetched, 13,369 served) — roughly half the
+# gather traffic was waste. Never re-add --expert-prefetch without a fresh A/B.
+# TunableOp stays OFF: CLAV_TUNABLEOP_SWEEP silently skips under expert offload
+# unless CLAV_TUNABLEOP_WITH_OFFLOAD=1, and forcing it corrupts numerics.
+# --expert-precise-memory is ON (shape E). It selects the fitted overhead
+# (5.25 GiB) over the default (7.60 GiB = fitted + compile/graph-capture
+# headroom); the 2.35 GiB difference lands in the expert arena. MEASURED
+# 2026-10-02 by a boot probe on the shape-D base (production stopped, candidate
+# booted AS the service, plan read off the engine's own log):
+#   runtime overhead  7.60 -> 5.25 GiB   (registry Qwen4Exp…, precise
+#                                         gfx1201 TP2 nbt4096/seqs16/len262144)
+#   expert cache      14.87 -> 17.22 GiB
+#   slots             6,496 -> 7,522     (+1,026 = +16%, 2.344 MiB each)
+#   swap region                 13.7 GiB / 5,986 slots, 133-134 per layer
+#   boot                        healthy in 420 s, no OOM, no knife-edge
+# The flag was called a non-option after it crash-looped the 2 x 262144 bf16
+# boot (0.13 GiB free of 31.86 GiB, NRestarts=3 at ~6.7 min each) — that shape
+# was already at the ceiling, so precise-memory was the last 2.35 GiB over, not
+# the cause. At 200704 x 3 it fits with room; the run line also reports
+# 'expert_precise_memory': True in non-default args, so a silent argparse drop
+# would be visible in the boot log.
+FRANK_PRIMARY_PRECISE_MEMORY=1
+# GPU_MAX_HW_QUEUES caps the HSA hardware queues per GPU (AMD's own tuning
+# guide recommends 2 for this class of workload; the runtime default on Linux
+# is 4). Verified a REAL knob, not a no-op: the string is present in
+# /opt/rocm/core-10.0/lib/libhsa-runtime64.so. NOT A/B MEASURED — Frank's call,
+# shipped as a deliberate CPU-side tune alongside the 32x64 gather geometry.
+FRANK_PRIMARY_GPU_MAX_HW_QUEUES=2
+# Persistent compile cache. MEASURED 2026-10-02: the image ships NO
+# /root/.cache at all (`ls: cannot access '/root/.cache'` in a fresh
+# container), and after our boot it holds /root/.cache/vllm at 1,154,714,333 B
+# in 4,665 files plus /root/.cache/comgr at 229 MB — both inside the
+# container's ephemeral layer, so every `podman rm -f` (every deploy, restart
+# and reboot) throws them away and re-pays the build. The engine's own log
+# names the loss: `torch.compile took 181.20 s in total`, of which 'Compiling a
+# graph for compile range (1, 1024) takes 173.19 s', followed by 'saved AOT
+# compiled function to /root/.cache/vllm/torch_compile_cache/…'. Mounting a
+# host dir at /root/.cache persists the vLLM AOT/torch_compile tree AND the
+# comgr HIP-kernel JIT cache, and hides nothing (the image has no such dir).
+# PROVEN LIVE 2026-10-02, two boots of this exact shape:
+#   cold  `Compiling a graph for compile range (1, 1024) takes 179.24 s`
+#         `torch.compile took 188.26 s in total`
+#   warm  `Directly load the compiled graph(s) for compile range (1, 1024)
+#          from the cache, took 2.143 s`
+#         `torch.compile took 3.97 s in total`
+#         `init engine (profile, create kv cache, warmup model) took 41.13 s
+#          (compilation: 3.98 s)`
+# (Total time to /health is dominated by the 172.8 GiB payload load, not the
+# compile — the saving is the ~184 s of inductor/AOT build.)
+# Safe to grow/prune: entries are keyed by version + config hash, so a stale
+# entry is a miss, never a wrong answer.
+FRANK_PRIMARY_CACHE_DIR=/home/dank/.cache/frank-primary
+FRANK_PRIMARY_MAXLEN=200704
 FRANK_PRIMARY_SEQS=3
-FRANK_PRIMARY_KV_BYTES=7730934784
+# The POOL is deliberately kept at shape D's 3.15e9 (option A): because the pool
+# is not an arena lever, raising it back from a cut buys concurrency only, and
+# 3.15e9 preserves the measured 1.99x while precise-memory grows the arena. A
+# 2.2e9 pool was probed in the same window (277,686 tokens / 1.38x) and rejected
+# as the shipped value; it remains the conservative arm if free VRAM ever binds.
+# KV forced to 400,700 tokens (Frank, 2026-10-03). The engine prices this pool
+# at 7,891.8 B/token at this shape and loses ~3.87 MB to block alignment, so
+# 3,166,500,000 B is the ask that lands at/above 400,700 (verify the reported
+# "GPU KV cache size" line on every deploy; it is not a constant ratio across
+# geometries — 7,701 B/token at 262144).
+FRANK_PRIMARY_KV_BYTES=3166500000
 FRANK_PRIMARY_KV_DTYPE=fp8
-FRANK_PRIMARY_SPEC=4
-FRANK_PRIMARY_UTIL=0.98
+FRANK_PRIMARY_MNBT=2048
+FRANK_PRIMARY_GATHER_CHUNKS=32
+FRANK_PRIMARY_GATHER_LANES=64
+# PREFIX CACHING: ON. An OFF experiment ran 2026-10-05 (below) and FAILED
+# to stop the corruption, so the historical flag is back.
+# MEASURED 2026-10-04, 18:46:07 PDT (5.6 h into the live boot, one running
+# request, prefix hit rate 48.1%): 48 gdn_hip state faults in one 0.6 s
+# burst, then the process-lifetime silence cap:
+#   [gdn_hip] ERROR gdn_decode_r: state out of ssm-cache range: max|S|=0
+#   with non-finite values (slot=42 head=3 limit=3.40282e+38) fault #1-23
+#   [gdn_hip] ERROR gdn_decode_r: ... (slot=42 head=8 ...) fault #24-46
+#   [gdn_hip] ERROR: state fault report limit (32) reached; further faults
+#   are counted silently
+# One physical SSM slot, two heads, BOTH ranks (heads 3 and 8), 23 layers
+# each. The guard's own message in this build: "known Qwen3.8-Flash-Next
+# (QFN) flaw under prefix caching and is being actively worked; output
+# quality for this request may degrade. Mitigation:
+# --no-enable-prefix-caching".
+# MECHANISM: with prefix caching on, this fork puts QFN on
+# mamba_cache_mode 'align' (the default-for-this-arch branch in
+# config/vllm.py; 'all' is deprecated and 'none' is the no-prefix-cache
+# mode, so align is the only sound mode WITH the flag). Align mode
+# checkpoints each sequence's GDN (Gated DeltaNet) recurrent state into a
+# mamba page every block, where block = 3136 tokens (the attention block
+# size is raised to 3136 so the page sizes match exactly — the boot log's
+# "Padding mamba page size by 0.13% ..."), and RE-SEEDS the state from that
+# page whenever a request resumes from a cached prefix. The corrupt state
+# landed at the 13th boundary: 13 x 3136 = 40,768 tokens — the "~40k" mark
+# where the request's output turned repetitive/corrupted. The NaN stayed
+# in the mamba page, so every later turn of that conversation re-seeded the
+# same non-finite state for those two heads (silent: the fault-report cap
+# was already spent), while fresh short sessions looked fine.
+# RAM WAS "RULED OUT" THE SAME DAY — AND THAT WAS WRONG (RESOLVED below).
+# What the evidence actually supported: zero MCE/EDAC records is the
+# EXPECTED silence of consumer non-ECC memory (a silent flip raises no
+# machine check); an 8 GiB pattern check over 60 s covers ~0.5 % of
+# 246 GiB — no verdict on a speed setting; "4800 is the JEDEC floor on
+# 5600-rated DIMMs" is a rating, not a stability proof at four-DIMM
+# loading. Only the two-head NaN signature argument was load-bearing,
+# and the revert test outvoted it.
+# COST ACCEPTED: no prefix reuse, so each turn re-prefills the whole
+# history instead of at most 3135 tail tokens (a full 40k context costs
+# tens of seconds of TTFT per turn at this shape's prefill rate). A silently
+# corrupted conversation is worse than a slower first token.
+# RESULT (2026-10-05, later that day): THE MITIGATION DID NOT WORK. With
+# --no-enable-prefix-caching deployed and mechanically verified (zero
+# gdn_hip faults, a 40,771-token single-request boundary probe coherent),
+# live long conversations STILL degraded at ~40k on the running endpoint
+# (Frank, direct). The container serving that window logged ZERO faults
+# while the corruption reproduced, so the align-mode page re-seed is not
+# the (only) mechanism — see the runbook's RESULT note for the open
+# hypotheses (re-fed poisoned transcript vs a fault-free long-context
+# path such as fp8 KV at depth). Cost bought nothing: caching is back ON.
+# The knob stays commented for a future fork build with a fixed GDN state
+# path: set 1, redeploy, re-verify past 40,768 tokens (one conversation
+# coherent AND zero gdn_hip lines).
+# RESOLVED 2026-10-05 EVENING (Frank): IT WAS THE RAM. 4800 MT/s (in
+# effect since the 2026-10-04 17:17 boot) was marginal; 4600 MT/s (since
+# the 2026-10-05 12:21 boot) and the endpoint is clean — zero gdn_hip
+# ERROR lines, zero MCE, and the live agent session on it past 140k
+# context with no degradation. Both failure eras sit under the RAM: the
+# 2026-10-04 NaN burst (corrupted weight/activation traffic poisoning a
+# GDN state write — the guard-visible path) and the fault-free ~40k
+# degradation (silent flips in KV/expert data, which no guard covers on
+# non-ECC). CONSEQUENCE FOR THE KNOB: the OFF experiment was CONFOUNDED
+# — both arms ran on the marginal speed — so it neither proved nor
+# disproved the fork's align-mode flaw. Prefix caching stays ON (zero
+# gdn_hip lines under heavy re-send traffic since the speed drop); flip
+# the knob only if a fault recurs on stable RAM or the fork ships a fix.
+# FRANK_PRIMARY_NO_PREFIX_CACHING=1
+# UTIL is an ARENA lever: the fork plans `budget = device_total x util`, and at
+# 31.86 GiB/card every +0.01 is +0.32 GiB of expert cache (~+139 slots), and
+# 0.982 is Frank's standing law: NEVER 1.00. The 1.00 attempt (2026-10-03,
+# shape G v1) died in worker init BEFORE loading a byte of weights — the fork
+# refuses budget >= device total with
+#   ValueError: Free memory on device cuda:N (31.29/31.86 GiB ...)
+# on BOTH ranks at 09:53 and 09:54, NRestarts=2 — so 1.0 is not even a runtime
+# knife-edge here, it is unbootable. The two historical ceiling deaths (the
+# 2x262144 bf16 boot at 0.13 GiB free; the 5x262144 boot at "0 bytes free" on a
+# real 995-token request) remain the reason the margin exists. Rollback note
+# for the other direction: if a request ever dies with
+# `Tried to allocate ... 0 bytes free`, the first knob back is THIS number
+# down a notch, not the window.
+FRANK_PRIMARY_UTIL=0.982
+# CUDA-graph plan term, priced at the measured capture. expert_plan.py charges
+# estimate_cudagraph_memory() = BASE_MB + PER_GRAPH_MB x n_graphs (defaults
+# 256/52 = 0.55 GiB for our 6-graph plan), and the estimator's own docstring
+# says to tune these against the post-capture line. MEASURED twice on this
+# shape: shape E reported "Graph capturing finished ... took 0.31 GiB" (TP0) and
+# 0.17 GiB (TP1); shape F, with the 0.32 GiB we then charged, reported
+# "took -0.06 GiB" — a NEGATIVE delta, i.e. capture fits inside memory the
+# profiler had already counted. So the base pool (256 MiB) is the over-charge:
+# charge 0 base + 12 MiB/graph = 0.07 GiB for the shipped no-spec 6-graph plan,
+# which returns 0.25 GiB more to the arena while still paying per graph, so a
+# future spec-decode shape (18 graphs) is charged 0.21 GiB rather than zero.
+# Verify every deploy against the "Estimated CUDA graph memory" vs "Graph
+# capturing finished ... took X GiB" lines; if capture ever goes positive and
+# exceeds the charge, raise FRANK_PRIMARY_CUDAGRAPH_PER_GRAPH_MB back toward 52.
+# A THIRD env var exists and is the only way to state the graph term outright:
+# VLLM_CUDAGRAPH_MEM_ESTIMATE_MB pins the WHOLE term to an exact MiB total in
+# one knob — estimate_cudagraph_memory() checks it FIRST and returns base/
+# per-graph untouched when set (verified in the shipped image,
+# v1/worker/gpu/cudagraph_mem.py). We ship the BASE/PER_GRAPH pair instead so
+# a future spec-decode shape re-prices automatically; switch to ESTIMATE_MB
+# only if the drift check below ever shows the pair cannot track capture.
+FRANK_PRIMARY_CUDAGRAPH_BASE_MB=0
+FRANK_PRIMARY_CUDAGRAPH_PER_GRAPH_MB=12
+
+# HOST-region cap — the last arena-plan term that still autofit per boot. With
+# no cap the planner's host bound is MemAvailable AT BOOT minus PLE 47.68 -
+# 2 x 7.0 process - 5.0 api/engine - 5% margin (measured boot 2026-10-05:
+# MemAvailable 237.97 -> 159.39 GiB usable across 2 ranks; the demand is 105.48
+# = 52.74/rank x 2 ranks, 23,040 swap-target experts). The same host reads 75.4
+# GiB available while running — the bound tracks whatever the co-tenants hold at
+# boot, and a boot where that drops near the demand silently pins next-hottest
+# experts VRAM-only instead (smaller swap region; the pinned_extra warning is
+# the only trace). Pin 120 GiB: measured demand + 14% headroom, on a host with
+# 246.1 GiB total. If a boot ever logs "expert-offload-mem … exceeds
+# MemAvailable at plan time", THAT is the contract being broken — look at the
+# co-tenant RAM before touching this number. Plan semantics verified in the
+# shipped image (offloader/expert_plan.py): the cap binds as the bound, an
+# exceedance only warns.
+FRANK_PRIMARY_EXPERT_OFFLOAD_MEM=120
+
+# --- optional engine knobs (unset == the flag is absent) ---------------------
+# FRANK_PRIMARY_SPEC holds the WHOLE --speculative-config JSON; setting it adds
+# the flag. THE RETIREMENT VERDICT WAS INVALID: c1248d4 (2026-10-02) retired MTP
+# by comparing fp8+MTP4 (37.4-40.6 tok/s, 54% acceptance, 09-29) against the
+# bf16 no-spec boot (36.4-39.4) — and bf16 then proved unbootable, so fp8 came
+# back without MTP and fp8+spec was NEVER compared against fp8+no-spec on the
+# same shape. The no-spec incumbent measures ~30 tok/s single-stream (below the
+# fleet's 35 tok/s target); every vendor flash-next recipe ships MTP3; and our
+# MTP4 run used the fork's default GREEDY draft sampling, while the recipes pass
+# draft_sample_method=probabilistic (greedy drafts are rejected more often under
+# temp>0 — the plausible cause of the 54%). MTP also re-adds drafter workspace:
+# the old note says the 7.2 GiB/rank KV ceiling WAS the MTP4 workspace, and
+# --expert-precise-memory spends that same budget, so an MTP arm must be tested
+# with precise memory BOTH on and off.
+#   MTP3:  '{"method":"mtp","num_speculative_tokens":3,"draft_sample_method":"probabilistic"}'
+FRANK_PRIMARY_SPEC=
+# FRANK_PRIMARY_COMPCONFIG holds the whole --compilation-config JSON. The
+# capture-ladder law: the piecewise cudagraph ladder must contain
+# num_speculative_tokens + 1 (MTP3 -> 4 -> ladder 4,8,12; MTP4 -> 5,10,15;
+# MTP5 -> 6,12,18,24), because every draft step proposes n+1 tokens. The default
+# ladder is multiples of 1+4 up to 512 then powers of two, so an MTP3 shape can
+# miss its own step size. Pinning the ladder is ALSO the candidate fix for the
+# SEQS>3 wall: query_start_loc is specialised to SEQS+1 and that constant must
+# land on a captured graph size (3 is not in the default ladder, 4 is —
+# measured 2026-10-02, and the vendor runs SEQS 16 on this model/TP2).
+# combo_kernels false/true are BOTH in the shipped recipes, so it is not a law.
+#   MTP3 at SEQS 3:  '{"cudagraph_mode":"PIECEWISE","cudagraph_capture_sizes":[1,4,8,12],"max_cudagraph_capture_size":12}'
+FRANK_PRIMARY_COMPCONFIG=
+# FRANK_PRIMARY_GEN_CONFIG holds the whole --override-generation-config JSON.
+# The payload's HF generation_config.json sets temperature 1.0 / top_p 0.95 /
+# top_k 20 and the engine WARNS about it on every boot, so a client that sends
+# no sampling fields gets creativity it never asked for. This is the one knob
+# that changes ANSWERS, not speed.
+#   '{"max_tokens":65536,"temperature":0.7,"top_p":0.95,"top_k":20}'
+FRANK_PRIMARY_GEN_CONFIG=
+
+# --- client contract: reasoning split, tools, preserve_thinking ----------------
+# Without --reasoning-parser the engine leaves the thinking text INLINE in
+# `content` with a dangling `</think>` and a null `reasoning` field (measured
+# 2026-10-01: content = 'We need answer...Brief.\n\n391') — the
+# malformed thinking block clients choke on. The qwen3 parser splits it
+# properly. The GLM rollback (d91379d) had dropped all three of these; the
+# working values are the ones ecaad36 shipped for this model.
+FRANK_PRIMARY_REASONING_PARSER=qwen3
+# Tools: --enable-auto-tool-choice + the qwen3_xml parser is what makes
+# tool_calls work here (without it, tool_choice auto returns 400).
+FRANK_PRIMARY_TOOL_PARSER=qwen3_xml
+# preserve_thinking keeps prior thinking blocks in the rendered prompt across
+# turns; enable_thinking makes thinking the server-side default. Verified the
+# payload's tokenizer_config template consumes both keys.
+FRANK_PRIMARY_CHAT_TEMPLATE_KWARGS='{"enable_thinking": true, "preserve_thinking": true}'
@@ -1,15 +1,18 @@
 # frank-primary.service — tcclaviger RDNA4 vLLM endpoint (orcarouter
-# Qwen3.8-Flash-Next-Uncensored-FP8, MTP4 spec decode, KV q8/q8, 600k total
-# context over 3 sessions, TP=2, tiered experts, 131-shard payload) on the
-# 2x R9700 pair, dankserver :7770. 2026-09-29 cutover from the R9V radiance
-# fork (git tag frank-primary-r9v-q8-20260929 is the pre-cutover state; its
+# Qwen3.8-Flash-Next-Uncensored-FP8, fp8 KV, NO spec decode, 3 slots x 200704
+# context with the pool sized to ~2 full-length conversations, TP=2, tiered
+# experts, 131-shard payload) on the 2x R9700 pair, dankserver :7770.
+# 2026-09-29 cutover from the R9V radiance fork (git tag
+# frank-primary-r9v-q8-20260929 is the pre-cutover state; its
 # launcher/shim/setup-state machinery is retired and the state dirs stay on
 # disk as rollback).
 #
 # Same port, same served alias; the harness models.yml id is unchanged.
 # The serve script drives `podman run` directly (the image entrypoint execs
-# `vllm serve`) and then waits on the container, so a crashed engine is a
-# nonzero ExecStart and Restart=on-failure is real.
+# `vllm serve`) and then waits on the container. NOTE: that wait can return
+# 0 even when the engine died mid-request (vLLM's API server drains and
+# exits clean on a worker fault), which is why the restart policy is
+# `always` — see the [Service] note.
 #
 # WHY A ROOT UNIT (not a rootless quadlet): the tiered-expert shape pins tens
 # of GiB of host RAM (the expert host slabs, ~20.5 GiB per rank of pinned
@@ -23,6 +26,9 @@
 Description=frank-primary — tcclaviger vLLM Qwen3.8-Flash-Next-Uncensored-FP8 on 2x R9700
 After=network-online.target
 Wants=network-online.target
+# Never let the start-rate limiter give up after a crash loop (a failed LLM
+# endpoint that systemd refuses to retry is worse than a retrying one).
+StartLimitIntervalSec=0

 [Service]
 User=dank
@@ -39,8 +45,23 @@ ExecStop=/usr/bin/podman stop -t 120 frank-primary-qwen
 # Boot cost: FP8 payload page-in plus torch compile/graph capture —
 # ~15 min cold (measured 2026-09-29), faster warm.
 TimeoutStartSec=1800
-Restart=on-failure
+# Restart=always, NOT on-failure (measured 2026-10-01): a runtime
+# torch.OutOfMemoryError kills the engine while vLLM's API server still
+# exits 0, so ExecStart returned clean, systemd logged "Deactivated
+# successfully", and on-failure NEVER fired — the endpoint stayed down until
+# a human noticed. `always` restarts on any container death; an explicit
+# `systemctl stop` still wins (systemd does not restart a unit the operator
+# stopped), so bench windows and deploys are unaffected.
+Restart=always
 RestartSec=10
+# ExecStop allows the engine 120 s to drain (`podman stop -t 120`); without an
+# explicit TimeoutStopSec systemd's 90 s default cuts the stop short, kills it,
+# and records 'Failed with result timeout' (measured 2026-10-01 13:27:17).
+# Must exceed the podman grace with margin for the engine's own shutdown.
+TimeoutStopSec=180
+# The engine pins tens of GiB of host RAM (expert host slabs); it is the
+# last process on this host that should be chosen by the kernel OOM killer.
+OOMScoreAdjust=-500

 [Install]
 WantedBy=multi-user.target
@@ -1,118 +1,136 @@
-# Tunables for frank-small (HauhauCS/Gemma4-12B-QAT-Uncensored-HauhauCS-
-# Balanced, Q4_K_M, plus the BF16 vision projector and the separate MTP
-# sidecar) on mainline llama.cpp, Dankclaw's RTX 3090, since 2026-09-17.
+# frank-small — llama.cpp (master, CUDA sm_86) serving
+# ornith-ai/Ornith-1.5-9B-GGUF:Q8_0 on the RTX 3090, :7771.
 #
-# Runtime: llama.cpp master 37b53fd, CUDA build for sm_86:
-#   cmake -B build-cuda -DCMAKE_BUILD_TYPE=Release -DGGML_NATIVE=ON \
-#     -DGGML_CUDA=ON -DCMAKE_CUDA_ARCHITECTURES=86 -DGGML_CUDA_FA_ALL_QUANTS=ON
-#   cmake --build build-cuda --target llama-server -j
-# GGML_CUDA_FA_ALL_QUANTS is ON: this service runs q8_0 KV with flash
-# attention, and without the flag quantized-KV prefill collapses ~12x on
-# CUDA (see the CUDA lesson in AGENTS.md).
+# 2026-10-05 (Frank): the payload moved off bartowski/MiMo-V2.6 (Q6_K) to
+# ornith-ai/Ornith-1.5-9B Q8_0 (9,786,060,384 B, LFS oid 22086870...).
+# Header-verified facts that re-price the shape: qwen35 hybrid like MiMo, but
+# 33 blocks (not 32) with full_attention_interval 4 => NINE full-attention
+# layers (MiMo had 8), 16 Q heads / 4 KV heads, k/v length 256, native window
+# 262144, dense (no expert keys), nextn_predict_layers=1 (an MTP head is IN
+# the file — capability only; no spec decode ships, same law as before).
 #
-# Model: HauhauCS/Gemma4-12B-QAT-Uncensored-HauhauCS-Balanced at Q4_K_M
-# (6.87 GiB). Architecture (GGUF arch gemma4): 48 layers, MHA with 16 kv
-# heads x 512 head_dim, rope freq_base 1e6, final logit softcapping 30,
-# per-layer token embeddings (PLE) inside the file, and a sliding-window
-# attention pattern (attention.sliding_window = 1024) -- most layers cache
-# only a 1024-position window, which is why the KV pool is a fraction of
-# the full-ctx x all-layers number.
+# THE KV CONTRACT STAYS f16 K + f16 V — NO KV QUANTIZATION (Frank, 2026-10-05,
+# overruling a written q8_0-V proposal). The arithmetic that makes this hard:
+#   KV rate 9 attn layers x 4 heads x 256 x 2 B = 18 KiB/token K, same again
+#   for V at f16 => 36 KiB/token (MiMo was ~24.5 measured: one fewer
+#   full-attn layer AND one fewer KV head).
+#   Weights +1,900 MiB (9,334 vs Q6_K's 7,432). Measured free at the Q6_K
+#   4096/2048 shape was 2,279 MiB; Q8_0 at f16/f16 does not fit that shape —
+#   the deficit is made up the way Frank ordered: UBATCH FIRST (2048 -> 1024,
+#   the measured 616 MiB rung), then POOL STEPS, then measure. Guess-and-check
+#   on the live card is the accepted method for llama.cpp sizing here: boot,
+#   read nvidia-smi free, step the pool (12,288-token increments, ~432 MiB
+#   each at the 36 KiB/token rate) until the 2-3 GiB floor holds. Compute
+#   buffers allocate at graph capture during BOOT, so a failure is loud and
+#   fast, not a surprise at the first long prompt.
 #
-# MTP: the Multi-Token Prediction head ships as a SEPARATE file
-# (mtp-gemma-4-12B-it.gguf, arch gemma4-assistant, 4 nextn layers, 0.24
-# GiB). Unlike the Qwen3.8 -mtp build (head inside the main file), this one
-# is passed via --model-draft with --spec-type draft-mtp, so the draft
-# context loads the assistant model. The same --spec-type value chains the
-# draftless ngram-mod matcher, which attaches unconditionally; two
-# context-backed drafters cannot share one process, so draft-mtp,ngram-mod
-# is the only legal shape that carries both.
+# Previous shape history (MiMo era, for the rollback trail): 2026-10-03 the
+# rung went Q5_K_M -> Q6_K with the pool HELD at 401,408; the freed third
+# slot funded exact f16 V first; the ~3.5 GiB still free funded the +875 MiB
+# of weights.
 #
-# SIZING (RTX 3090, 24576 MiB): CTX_SIZE = PARALLEL x 262144 -- four
-# FULL-window slots, measured to fit: the 4-slot boot idles at 22,346
-# MiB and each extra full-context slot costs 3,450 MiB (measured 1-slot
-# 11,996 -> 4-slot 22,346). A fifth slot projects to ~25.8 GiB, past the
-# card, and the KV pool allocates upfront -- so 4 is the max. Explicit
-# CTX_SIZE bypasses --fit on purpose: --fit maximises the TOTAL pool,
-# which would shrink each slot below the full 262144 window once
-# parallel > 1.
+# 2026-10-03 (Frank): TWO streams instead of three, each one WIDE — 200,704
+# tokens per conversation, above the 200,700 the harness compacts at — and the
+# KV cache is now FULLY UNQUANTIZED (f16 K, f16 V). The two moves are the same
+# move: llama.cpp divides --ctx-size across --parallel slots, so the slot given
+# up is exactly the VRAM that pays for unquantized V. No further quantization
+# anywhere on this endpoint.
 #
-# Vision + audio: --mmproj loads the BF16 projector (11 tensors, 52M
-# params; it carries an audio encoder too). The fleet's image path lives
-# here, which is why halogen no longer sets HALOGEN_VISION_TOWER. Audio
-# WORKS: input_audio content parts transcribe verbatim (verified
-# 2026-09-17 with a 16k mono WAV).
+# Three measurements on this card, this runtime, price the terms:
+#   2 x 262,144, no projector: 20,190 MiB used / 4,386 free
+#     KV = 524,288 tok x 24.5 KiB = 12,800 MiB => non-KV (weights 6,557 +
+#     DeltaNet state ~25/slot + compute buffers + CUDA graphs) = 7,390 MiB
+#   3 x 196,608, projector on: 22,972 MiB used / 1,153 free
+#     KV = 14,113 MiB => non-KV WITH the vision tower = 8,859 MiB
+#   3 x 180,224, projector on: 21,684 MiB used / 2,441 free (the 2026-10-02
+#     shape) => non-KV = 8,748 MiB
+# Two slots give back one DeltaNet state (~25 MiB). Priced at f16 K+V
+# (32 KiB/token) this pool is 12,544 MiB, so the prediction is
+# 12,544 + ~8,800 = ~20.9 GiB used, ~3.1 GiB free — the band the 2026-10-02
+# shape hit, bought with precision instead of a slot. The 1024 ubatch
+# re-prices the compute buffers upward, so MEASURE after any change: if the
+# boot lands under 2 GiB free, step the pool down in 12,288-token
+# (384 MiB at this rate) increments.
 #
-# Sampling: greedy by default (the fleet contract); a client that sends a
-# temperature gets its own, and the harness applies per-model overrides
-# from models.yml.
+# KV arithmetic: attention KV exists only in the 8 full-attention layers
+# (4 KV heads x 256 head_dim x 2 B) = 16 KiB/token for K and the same again for
+# V => 32 KiB/token, so 12,288 tokens (= 6144/slot) is one 384 MiB step — that
+# is the ladder to walk if this moves again. Two native windows (2 x 262,144)
+# are NOT reachable with full-precision V: 524,288 x 32 KiB = 16.1 GiB of KV
+# against a card with ~15.7 GiB of KV room once weights, the vision tower, the
+# DeltaNet states and the graphs are loaded. 200,704 per stream is what full
+# precision buys; q8_0 on V would buy 262,144 instead.
 #
-# WARNING: do not name any variable TEMP, HOST, or GPU_MAX_HEAP_SIZE (HIP
-# reads those bare vars on the ROCm hosts -- see the 2026-08-13 incident in
-# AGENTS.md). This box is CUDA, but the rule keeps env files portable.
+# The asymmetric-KV contract this endpoint was created for (Frank, 2026-10-01
+# night: f16 K + q8_0 V) is now spent — it existed to make a third slot fit,
+# and the third slot is gone. The flags stay explicit so the choice is visible
+# in the argv either way.
+LLAMA_BIN=/home/dank/GitPackages/llama-master/build-cuda/bin/llama-server
+MODEL_FILE=/home/dank/models/Ornith-Q8/Ornith-1.5-9B-Q8_0.gguf

-# llama.cpp runtime binary: mainline master 37b53fd, CUDA sm_86,
-# FA_ALL_QUANTS=ON.
-LLAMA_BIN=/home/dank/GitPackages/llama.cpp-mainline/build-cuda/bin/llama-server
+# Vision: the Ornith Q8_0 GGUF ships no `v.*` tensors either (dense-9B byte
+# arithmetic: 9.79 GB ≈ 9B params x ~1.06 B/param leaves no room for a tower),
+# so the projector file IS the vision capability (mmproj-Ornith-1.5-9B-BF16
+# .gguf, 921,704,672 B, LFS oid 626f9f90..., fetched by
+# scripts/download-frank-small.sh). Empty string = text-only on purpose.
+MMPROJ_FILE=/home/dank/models/Ornith-Q8/mmproj-Ornith-1.5-9B-BF16.gguf

-# Model package. The MTP sidecar file is required for speculative decoding.
-MODEL_DIR=/home/dank/models/Gemma4-12B-QAT
-MODEL_FILE=Gemma4-12B-QAT-Uncensored-HauhauCS-Balanced-Q4_K_M.gguf
-MMPROJ_PATH=/home/dank/models/Gemma4-12B-QAT/mmproj-Gemma4-12B-QAT-Uncensored-HauhauCS-Balanced-BF16.gguf
-DRAFT_FILE=mtp-gemma-4-12B-it.gguf
-
-# Fleet identity and network. 7771 is this service's own endpoint on
-# Dankclaw; 7770 stays frank-large's. The alias is the id at /v1/models and
-# what clients ask for -- the harness entry in models.yml names it.
+# Fleet endpoint: same port and alias as every frank-small before it, so the
+# harness models.yml id and the homepage probe do not move.
 ALIAS=frank-small
 BIND_ADDR=0.0.0.0
 LISTEN_PORT=7771

-# Four FULL 262144-token contexts, one per slot (CTX_SIZE is the total
-# pool; --parallel divides it). See the SIZING note above for the fit
-# arithmetic; a fifth slot does not fit.
-CTX_SIZE=1048576
-PARALLEL=4
-CACHE_TYPE_K=q8_0
-CACHE_TYPE_V=q8_0
-# CPU RAM cache cap in MiB (build default 8192). This host is memory-tight
-# (frank-large's 70 GiB GTT plus the rest of the stack), and Gemma4 runs
-# fully on the 3090 (-ngl 999), so the CPU-side cache is not the hot path;
-# cap it at 2048 to give back ~6 GiB of LPDDR. Gemma4 is not a DeltaNet
-# model, so the >12.8 GiB CACHE_RAM floor does not apply.
-CACHE_RAM=2048
+# ONE conversation at the NATIVE 262,144 window (Frank, 2026-10-05). The
+# two-slot shape died under Ornith and the NO-QUANTIZE law, by arithmetic:
+# the KV rate is the ARCHITECTURE's, not the quant's — 9 full-attn layers x
+# 4 KV heads x 256 dim x 2 B = 36 KiB/token at f16 K+V (MiMo measured 24.5),
+# so 2 x 200,704 wants ~14.1 GiB of KV on top of +1,900 MiB of Q8 weights
+# against 2,279 MiB measured free; ubatch 1024 claws back 616, and no
+# pool step fits BOTH the 2-3 GiB floor and the harness compaction
+# invariant (slot >= 191,148). One native slot fits with room; the window
+# per conversation RISES 200,704 -> 262,144. The cost: a second concurrent
+# request queues behind the released slot (subagent concurrency 2 -> 1).
+CTX_SIZE=262144
+PARALLEL=1

-# Batch geometry: 8192 logical / 1024 physical. Measured on this card at
-# max length (254k-token prefill / 253k-ctx decode, 4 slots, q8 KV):
-# ubatch 512 -> pp 915 t/s, tg@250k 90-204; ubatch 1024 -> pp 962,
-# tg 257-302. Larger ubatches (2048) OOM the MTP draft context's pp
-# compute buffer (~3.3 GiB against 2.2 GiB headroom) and must not be
-# tried. Idle VRAM is 23,300 MiB at this geometry.
-BATCH_SIZE=8192
+# Compute-buffer geometry. This endpoint passed neither flag until 2026-10-03,
+# so it ran on llama.cpp's own defaults (2048/512); the pair is now 4096/2048.
+# Both are tunables because the pair is a measurement, not a law — and the
+# measurement is CARD-SPECIFIC. On the 2x R9700 layer-split sibling the ubatch
+# curve peaks at 512 and every step up costs GiB of compute buffer for no
+# speed (see frank-medium-batch-ubatch). On THIS CUDA 3090 the curve climbs
+# monotonically to 2048: 710 -> 1188 t/s prefill from ubatch 1024 to 2048 on
+# the 35B MoE sweep, 1944 -> 2423 t/s on the Ornith 8k sweep, with decode flat
+# to under 1 t/s on every rung. Ubatch is a prefill knob; decode never moves.
+#
+# The logical batch is free — two independent sweeps agree that at a fixed
+# ubatch, 2048/4096/8192 give identical prefill speed AND identical peak VRAM,
+# because the compute buffers are sized by the ubatch alone. Doubling it costs
+# nothing and buys nothing; it is here because it is the value the pair was
+# measured at.
+#
+# THE RISK IS ALL IN THE UBATCH. The 1024 -> 2048 step cost ~1,244 MiB of peak
+# graph VRAM on the 35B MoE (7,330 -> 8,574 MiB); this 9B hybrid has far
+# smaller compute tensors, so the delta should be well under that, but the
+# predecessor Q5_K_M shape recorded "ubatch 2048 failed the scratch boot" at
+# ~3,571 MiB free. The Q8_0 rung ships UBATCH 1024 (Frank, 2026-10-05: lower
+# ubatch FIRST — the measured 616 MiB step is the first VRAM the Q8 weights
+# claw back, before any pool move). Compute buffers are allocated at graph
+# capture during BOOT, so a failure is loud and fast, not a surprise at the
+# first long prompt. MEASURE nvidia-smi free immediately after the deploy:
+# under ~1.5 GiB, or a boot failure, step the POOL down next (12,288-token
+# increments, ~432 MiB each at the Ornith 36 KiB/token f16 KV rate).
+BATCH_SIZE=4096
 UBATCH_SIZE=1024

-# CPU threads: this service runs entirely on the GPU (NGL=999), so threads
-# cover tokenization, sampling and the HTTP layer only. Deliberately modest
-# -- this host's CPU is shared with halogen's engine, which is CPU-hungry.
-NUM_THREADS=8
-THREAD_BATCH=8
-
-# Every layer on the 3090, with flash attention (required for q8 KV).
-NGL=999
-FLASH_ATTN=on
-
-# Speculative decoding: the MTP sidecar (draft context) chained with the
-# draftless ngram-mod matcher. Draft depth and busy-wait level are made
-# explicit here (both were the implicit build defaults: 3 and 50).
-# Measured at 250k ctx: depth 4 + poll 100 DECODED WORSE (230-257 t/s)
-# than depth 3 + poll 50 (257-302 t/s) — the longer verification round
-# costs more attention re-read at long context than the extra accepted
-# token earns. Keep 3/50.
-SPEC_TYPE=draft-mtp,ngram-mod
-SPEC_DRAFT_N_MAX=3
-POLL_LEVEL=50
-
-# Greedy default (fleet contract); reasoning is extracted into
-# reasoning_content with an unlimited budget (the model decides).
-TEMPERATURE=0.0
-REASONING_FORMAT=deepseek
-REASONING_BUDGET=-1
+# The KV contract (2026-10-05, Frank): f16 K + f16 V — NO KV quantization.
+# The q8_0-V arithmetic was written and OVERRULED. The Q8 weights eat the
+# VRAM that funded f16 V on 2026-10-03, so the remedy ladder is UBATCH FIRST
+# (2048 -> 1024, the measured 616 MiB rung), then POOL STEPS — and
+# guess-and-check on the live card is the accepted method: boot, read
+# nvidia-smi, step down until the floor holds. --flash-attn stays pinned on
+# in the serve script: it is the fast path for f16 KV.
+CACHE_TYPE_K=f16
+CACHE_TYPE_V=f16
+NGL=99
@@ -1,5 +1,5 @@
 [Unit]
-Description=llama.cpp Qwen3.8-27B GSQ-RCO IQ3_S MTP+vision (dankclaw, frank-small)
+Description=llama.cpp MiMo-V2.6-Distill-Qwen-9B Q6_K GGUF (qwen35 hybrid, 2x200704, unquantized f16 KV, mmproj vision) on the RTX 3090 (dankclaw, frank-small)
 After=network-online.target
 Wants=network-online.target

@@ -8,8 +8,7 @@ Type=simple
 User=dank
 Group=dank
 # CUDA pins its staging buffers, so the process needs the memlock budget even
-# though the weights themselves are copied into VRAM. Parity with the halogen
-# quadlet's Ulimit=memlock=-1:-1 and the retired ik unit.
+# though the weights themselves are copied into VRAM.
 LimitMEMLOCK=infinity
 Environment="FRANK_SMALL_ENV_FILE=/home/dank/serverAppConfigs/scripts/frank-small.env"
 ExecStart=/home/dank/serverAppConfigs/scripts/serve-frank-small.sh
@@ -0,0 +1,86 @@
+# glm53-compile-01-model-decorator.patch — put the GLM-5.3-Flash trunk under torch.compile.
+#
+# STATUS: WIP — first of three patches in the "compile the GLM trunk" changelist. The
+# set does NOT yet boot the engine (see 03 for the next blocker), so it is NOT part of
+# the deployed frank-primary shape. Nothing in scripts/ mounts these files.
+#
+# WHY THIS IS THE RIGHT LEVER (measured 2026-09-30, not inferred):
+#
+# The fork's own flagship trunk — vllm/models/qwen4_exp/amd/model.py — carries
+# @support_torch_compile. vllm/models/glm5next/common/model.py does not, and the engine
+# says so on every boot:
+#
+#   Cudagraph mode FULL_AND_PIECEWISE requires piecewise capture, but the loaded model
+#   provides neither a compiled submodule nor breakable CUDA graphs.
+#   Overriding to FULL_DECODE_ONLY.
+#
+# A torch-profiler window on the live endpoint (496 decode tokens, rank0, 3 x 160-token
+# generations at the deployed shape) shows what the missing compilation costs. Total
+# device kernel time 46.18 s / 496 tokens = ~93 ms per token, against a measured step
+# time of ~104 ms (9.6 tok/s with the profiler attached):
+#
+#   16.07 s  34.8 %  708,500 launches  vectorized_elementwise_kernel      (22.7 us each)
+#    4.60 s  10.0 %    42,000           r4d_gemm_moe_w4a16_nt_b16_m64     (the MoE GEMM)
+#    3.11 s   6.7 %   174,708           elementwise_kernel_manual_unroll
+#    2.94 s   6.4 %   138,252           wvSplitK_hf_sml_                  (split-K GEMM)
+#    4.09 s   8.9 %    83,160           topk/router (sbtopk gatherTopK + warpMergeSortTopK
+#                                       + bitonicSortKVInPlace)
+#    3.01 s   6.5 %   133,920           r4d_mhc_* (hyper-connection pre/post/finish)
+#    2.78 s   6.0 %   127,916           inductor triton_poi_* / triton_per_* (the only
+#                                       fusions that DO happen)
+#    0.49 s   1.05%    20,664           r4d_moe_lru_gather_kernel         <-- the suspect
+#
+# That last row retired the fleet's working hypothesis. Decode on this shape is NOT
+# expert-gather bound: the gather is 1 % of device time, the same order as the telemetry
+# kernel that was switched on to measure it. 44 % of the budget is pointwise glue —
+# bf16<->fp32 conversions, adds, pows/rsqrts from fp32 RMSNorm, clamp/scale quant
+# passes — spread over ~1,428 launches per token. Inductor fusion is the only lever
+# that touches that mass; the router is second at 9 % but is binary-blocked (the
+# libr4d route kernel exists only as r4d_moe_route_kernel<128|256|512> instantiations
+# and GLM has 288 experts, with no HIP sources shipped to rebuild).
+#
+# TARGET: vllm/models/glm5next/common/model.py   (inside the pinned image)
+#   base    sha256 008eeaa2a07e091465319f86900b5a321d921238723ccbe22e6b9e971e1d2b28
+#   patched sha256 6e9cb81b84a9a88bd41403ca404aa04c3b791899dd77d661c3f25ae686cecd40
+#
+# APPLY: `patch -p1` from the vllm package root, or bind-mount new/.../model.py over
+# site-packages. Then R4D_GLM_COMPILE=1 (default) enables it; =0 makes the decorator a
+# no-op, so the file can stay mounted for an A/B without a second artifact.
+# Requires 02 (trace-safe probes) — without it the worker dies during profile_run on
+# the first probe call inside the traced region.
+--- a/vllm/models/glm5next/common/model.py
+++ b/vllm/models/glm5next/common/model.py
+@@ -1,12 +1,14 @@
+ # SPDX-License-Identifier: Apache-2.0
+ # SPDX-FileCopyrightText: Copyright contributors to the vLLM project
+ 
+import os
+ from collections.abc import Iterable
+ from typing import ClassVar, Literal
+ 
+ import torch
+ from torch import nn
+ 
+from vllm.compilation.decorators import support_torch_compile
+ from vllm.config import ParallelConfig, VllmConfig
+ from vllm.distributed import (
+     get_ep_group,
+@@ -623,6 +625,18 @@
+         )
+ 
+ 
+_GLM_COMPILE = os.environ.get("R4D_GLM_COMPILE", "1") == "1"
+
+
+@support_torch_compile(
+    dynamic_arg_dims={
+        "input_ids": 0,
+        "positions": -1,
+        "intermediate_tensors": 0,
+        "inputs_embeds": 0,
+    },
+    enable_if=lambda _cfg: _GLM_COMPILE,
+)
+ class Glm5NextModel(nn.Module):
+     def __init__(self, *, vllm_config: VllmConfig, prefix: str = ""):
+         super().__init__()
@@ -0,0 +1,119 @@
+# glm53-compile-02-trace-safe-probes.patch — libr4d capability probes must not do I/O
+# inside a torch.compile region.
+#
+# STATUS: WIP — second of three patches in the "compile the GLM trunk" changelist.
+# Inert (but harmless) without 01. Not part of the deployed shape.
+#
+# THE BREAK (measured: with 01 mounted and R4D_GLM_COMPILE=1, the worker dies during
+# profile_run, i.e. the first — cold — forward):
+#
+#   Explanation: Dynamo does not know how to trace the builtin `posix.stat`
+#   ... r4d_mhc_supported -> r4d_lib.r4d_mhc_enabled -> available -> os.path.exists
+#
+# THE NON-OBVIOUS PART: functools.lru_cache does NOT protect these probes. Dynamo
+# re-traces the wrapped body to build its guard even when the cache is warm — it warns
+# about exactly this ("Dynamo detected a call to a `functools.lru_cache`-wrapped
+# function ... Dynamo ignores the cache wrapper and directly traces the wrapped
+# function"). Verified in-image, torch 2.11.0+rocm10.0, torch.compile(backend="eager"):
+#
+#   cold lru_cache call       -> RAISE Unsupported
+#   WARM lru_cache call       -> RAISE Unsupported     <-- warming is not enough
+#   plain module-global read  -> compiles
+#   uncached os.path.exists   -> RAISE Unsupported
+#
+# So each probe must resolve ONCE into a plain value and be read as a global. The set
+# below is every no-argument probe / symbol lookup reachable from a forward pass,
+# grep-verified call sites:
+#   model_executor/layers/fused_moe/experts/r4d_w4a16_moe.py:79  available()
+#   model_executor/layers/fused_moe/experts/r4d_w4a16_moe.py:81  has_w4a16_moe()
+#   model_executor/kernels/mhc/r4d_dsv4.py:16                    r4d_mhc_enabled()
+#   v1/attention/ops/rocm_aiter_mla_sparse.py:694                r4d_indexer_enabled(), mqa_max_heads()
+#   v1/attention/ops/rocm_aiter_mla_sparse.py:1124               r4d_indexer_enabled()
+#   model_executor/layers/fused_moe/experts/r4d_mxfp4_moe.py:683 r4d_act_quant_enabled()
+#   model_executor/layers/fused_moe/experts/r4d_w4a16_moe.py _entry()  moe_w4a16()/moe_w4a16_mt1() per MoE call
+# (lib() is the transitive dependency of all of them.)
+#
+# TARGET: vllm/model_executor/kernels/r4d_lib.py   (inside the pinned image)
+#   base    sha256 b8dfeefb4acc72b67b0579a4ab0653ef5d48c40cc8649187d757c80ce6b9617a
+#   patched sha256 0751852e1ca813ad2bdb1da7bf10c605130e63e04ef9ec57e70b5f675a29c08e
+#
+# APPLY: append the block below to the module. R4D_FREEZE_PROBES=0 restores stock
+# behaviour. Cost when nothing is compiled: ten one-time probes at import (measured:
+# logged once per process, e.g. "libr4d capability probes frozen: 10 warm [...]").
+#
+# OBSERVED EFFECT: with 01+02 the worker gets past this break and reaches the next one
+# (see 03) — dynamo reports "Dynamo bytecode transform time: 1.08 s" for the trunk.
+--- a/vllm/model_executor/kernels/r4d_lib.py
+++ b/vllm/model_executor/kernels/r4d_lib.py
+@@ -361,3 +361,71 @@
+ @functools.lru_cache
+ def moe_w4a16_mt1():
+     return getattr(lib(), "r4d_gemm_moe_w4a16_nt_b16_m64_mt1", None)
+
+
+# FORK-LOCAL (patch: trace-safe capability probes).  torch.compile traces the
+# BODY of a functools.lru_cache call even when the cache is warm, so a capability
+# probe called from a forward pass is a dynamo graph break.  Measured 2026-09-30
+# on the GLM-5.3-Flash trunk under @support_torch_compile:
+#   Dynamo does not know how to trace the builtin `posix.stat`
+#   r4d_mhc_supported -> r4d_mhc_enabled -> available -> os.path.exists
+# Each probe below is wrapped to resolve ONCE into a memo and is warmed at import
+# (outside any tracing region), so a traced call is a dict read.  The probe set is
+# every no-argument capability probe / symbol lookup reachable from a forward
+# pass: the W4A16 + mxfp4 MoE dispatch, the mHC op, the MLA sparse indexer gate
+# and the draft/lm-head guards.
+_PROBE_NAMES = (
+    "available",
+    "lib",
+    "has_w4a16_moe",
+    "moe_w4a16",
+    "moe_w4a16_mt1",
+    "has_mhc_dsv4",
+    "r4d_mhc_enabled",
+    "r4d_indexer_enabled",
+    "mqa_max_heads",
+    "r4d_act_quant_enabled",
+)
+
+
+def _freeze_probe(fn):
+    """Return fn bound to its first result.  Result must be a plain value."""
+
+    memo = []
+
+    def probe(*args, **kwargs):
+        if not memo:
+            memo.append(fn(*args, **kwargs))
+        return memo[0]
+
+    return probe
+
+
+def _freeze_probes() -> list[str]:
+    frozen = []
+    for name in _PROBE_NAMES:
+        fn = globals().get(name)
+        if fn is None:
+            continue
+        # Only the fast branch of a bound probe is traced; an empty memo means the
+        # first call still does I/O, so populate it here as well.
+        globals()[name] = _freeze_probe(fn)
+        frozen.append(name)
+    return frozen
+
+
+from vllm.logger import init_logger as _init_logger
+
+_probe_log = _init_logger(__name__)
+
+if os.environ.get("R4D_FREEZE_PROBES", "1") == "1":
+    _warmed, _failed = [], []
+    for _n in _freeze_probes():
+        try:
+            globals()[_n]()
+            _warmed.append(_n)
+        except Exception:  # noqa: BLE001 -- probe unavailable here; recompute lazily
+            _failed.append(_n)
+    _probe_log.info("libr4d capability probes frozen: %d warm %s%s",
+                    len(_warmed), _warmed,
+                    f" (deferred: {_failed})" if _failed else "")
@@ -0,0 +1,116 @@
+# glm53-compile-03-fullgraph-optout.patch — let torch.compile split the graph instead
+# of failing on a break.
+#
+# STATUS: WIP — third of three patches in the "compile the GLM trunk" changelist.
+# With 01+02+03 the model TRACES, then hits the NEXT blocker (below). The set does not
+# yet boot the engine; nothing in scripts/ mounts it.
+#
+# THE BREAK: with 01+02 mounted the worker dies with
+#
+#   RuntimeError: Graph breaks are not supported with aot compile.
+#   Please use torch.compile(fullgraph=True).
+#
+# because vllm/compilation/wrapper.py calls torch.compile(..., fullgraph=True, ...) and
+# this build has AOT compile on by default (VLLM_USE_AOT_COMPILE). The GLM trunk cannot
+# satisfy fullgraph: its indexer metadata path syncs to the host —
+#   models/glm5next/amd/sparse_indexer.py:263  int(positions[...].max().item()) + 1
+#   models/glm5next/amd/sparse_indexer.py:419  lmax = int(decode_metadata.decode_lens.max().item())
+# — and a .item() inside a captured graph is not expressible.
+#
+# WHAT WAS VERIFIED (in-image, torch 2.11.0+rocm10.0, torch.compile(backend="eager")):
+#
+#   fullgraph=True   suppress_errors=False  -> RAISE Unsupported
+#   fullgraph=True   suppress_errors=True   -> RAISE Unsupported   <-- suppress_errors
+#                                                                     does NOT cover breaks
+#   fullgraph=False  suppress_errors=False  -> ok   (traceable segments compiled by
+#                                                    inductor, the rest runs eagerly;
+#                                                    numerics unchanged)
+#
+# So breaks must be TOLERATED, not suppressed: the traceable mass still fuses. This
+# patch makes the fullgraph requirement opt-out via VLLM_COMPILE_FULLGRAPH=0 (default 1,
+# so every other model keeps stock behaviour), and the run also needs
+# VLLM_USE_AOT_COMPILE=0 because AOT compile refuses breaks outright.
+#
+# TARGET: vllm/compilation/wrapper.py   (inside the pinned image)
+#   base    sha256 969961dca41c78385212e985cfc0c57ef067f4a2d7490c73c658c600a1b95cac
+#   patched sha256 732363bb1a1367772f13c60a655c678e7e2e9e56186677b14238f5e84cac31b3
+#   (the file already imports os; the patch does not add a second import)
+#
+# NEXT BLOCKER — NOT FIXED HERE. With 01+02+03 and
+# R4D_GLM_COMPILE=1 VLLM_COMPILE_FULLGRAPH=0 VLLM_USE_AOT_COMPILE=0 the trunk traces
+# ("Dynamo bytecode transform time: 1.08 s") and then the worker dies during profile_run
+# on the decode forward:
+#
+#   KeyError: 'in_proj_qkvbfg_a'
+#     model.py:748  Glm5NextModel.forward        hidden_states, residual, post, comb = layer(...)
+#     model.py:456  Glm5NextDecoderLayer.forward
+#
+# `in_proj_qkvbfg_a` is the KDA layer's merged column-parallel projection
+# (models/glm5next/common/kda.py:217, called at :407) — and it is also a name in the
+# load_weights shard table (model.py:780-785), i.e. a real submodule attribute name,
+# not an arbitrary string.
+#
+# WHERE IT IS THROWN (established 2026-09-30 by reading the full traceback, since the
+# report line is inside dynamo-rewritten code): torch/_dynamo keeps a name -> module map
+# and looks it up by name — output_graph.py:1259 / :1384 register into `nn_modules`,
+# :3000 does `self.nn_modules[node.target]` (a bare-name KeyError when absent), and
+# :1225 documents stripping "_buffers[..]/_parameters[..]/_modules[..]" from those
+# names. So the compiler is failing to RESOLVE A SUBMODULE BY NAME while tracing the
+# GLM decoder layer. Whether the root cause is the fork's module tree (something
+# dynamo cannot register statically) or dynamo's own registration is NOT yet settled.
+#
+# STATUS OF THAT BLOCKER (updated 2026-09-30, after the diagnostics below):
+#   * TORCHDYNAMO_VERBOSE=1 adds NO internal stack trace -> it is a plain Python
+#     KeyError raised in a callee dynamo INLINED into Glm5NextDecoderLayer.forward
+#     (the reported line 456 is that method's `def` line, which is how inlined frames
+#     surface).
+#   * It is NOT raised from dynamo's four nn_modules lookups: the probe in
+#     /home/dank/dynamo-probe wraps three of them and stayed silent over two runs with
+#     the probe confirmed installed in the API-server and worker processes.
+#   * It is NOT a literal string lookup: the name appears tree-wide only as the KDA
+#     layer's attribute (kda.py:217, used at :407) and in the load_weights shard table
+#     (model.py:780-785). The four `proj_name = prefix.split(".")[-1]` sites in the
+#     quantization utils do compute exactly this bare key, but each is guarded
+#     (`if proj_name not in fused_mapping: return`), so none of them can raise it.
+#   * It reproduces on every run of this configuration (5 of 5).
+# REMAINING INSTRUMENT: name the inlined caller. Either `TORCH_LOGS=+dynamo` on a boot
+# and grep the log for the key (dynamo prints inlined frames with source), or wrap the
+# inlining candidates in that forward — the mHC CustomOps (hc_pre / hc_post /
+# hc_fused_post_pre in layers/mhc.py) and the indexer op — with a KeyError-reporting
+# probe. Until that names the caller this is a dynamo-side name-resolution failure.
+# NOTE the launcher gotcha that wasted a boot earlier: the scratch launcher now
+# preserves ENVARGS (`ENVARGS="${ENVARGS:-}"`), but it previously reset it, silently
+# dropping outer env vars.
+#
+# SECOND BLOCKER, KNOWN AND NOT FATAL IN THIS CONFIG: the indexer's in-forward
+# workspace acquisition (patch 05 makes it compiler-opaque) and the fused-norm
+# launcher cell (patch 04 removes the call) are real breaks, but with
+# VLLM_COMPILE_FULLGRAPH=0 they are tolerated — they only become fatal under
+# fullgraph=True, which this trunk can never satisfy anyway.
+--- a/vllm/compilation/wrapper.py
+++ b/vllm/compilation/wrapper.py
+@@ -19,6 +19,15 @@
+ 
+ logger = init_logger(__name__)
+ 
+
+# FORK-LOCAL (patch: fullgraph opt-out).  torch.compile(fullgraph=True) turns any
+# dynamo graph break into a hard failure; suppress_errors does NOT cover breaks
+# (verified in-image, torch 2.11+rocm10.0).  With VLLM_COMPILE_FULLGRAPH=0 the
+# traceable segments are still compiled by inductor and the untraceable ones run
+# eagerly, which is what a model with device-sync metadata (the GLM-5.3-Flash
+# indexer) needs.  Default 1 keeps stock behaviour for every other model.
+_FULLGRAPH = os.environ.get("VLLM_COMPILE_FULLGRAPH", "1") == "1"
+
+ R = TypeVar("R")
+ P = ParamSpec("P")
+ 
+@@ -146,7 +155,7 @@
+         with aot_context:
+             self._compiled_callable = torch.compile(
+                 compiled_ptr,
+-                fullgraph=True,
+                fullgraph=_FULLGRAPH,
+                 dynamic=False,
+                 backend=backend,
+                 options=options,
@@ -0,0 +1,112 @@
+# glm53-compile-04-fused-qkv-rmsnorm-optout.patch — take GLM's fused-norm JIT kernel
+# out of the traced path.
+#
+# STATUS: WIP — fourth of the "compile the GLM trunk" changelist. Needs 01 (the
+# decorator) to matter; not part of the deployed shape.
+#
+# MEASUREMENT NOTE — read the run history before trusting any claim below, because a
+# launcher bug made four consecutive boots misleading.  The scratch launcher
+# (serve_glm_autoround.sh) was restructured on 2026-09-30 and the
+# `-e VLLM_COMPILE_FULLGRAPH=0` line was DROPPED, so those boots mounted the wrapper
+# patch while `_FULLGRAPH` still defaulted to True.  With fullgraph=True *every* break
+# is an error, which is why the reported failure kept moving as breaks were removed:
+#
+#   fullgraph=False, AOT off  -> breaks tolerated; dies later on the dynamo name-map
+#                                `KeyError: 'in_proj_qkvbfg_a'`
+#   fullgraph=True,  AOT off  -> dies on whichever break comes first
+#                                (cell -> workspace -> compiler-disable)
+#
+# An earlier version of this note blamed `TORCH_LOGS=graph_breaks` for the difference.
+# That was WRONG — the flag only logs; the launcher silently ran fullgraph=True.
+# Launcher fixed; the env var is set again.
+#
+# What it means for THIS patch: under fullgraph=True the fused-norm call was the first
+# break dynamo reported, and after this patch the report moved on to the workspace
+# acquisition — so it removes a real break.  It is not by itself what unblocks a boot:
+# with breaks genuinely tolerated the blocker is the name-map KeyError.
+#
+# IT IS STILL THE RIGHT PATTERN TO REMOVE, either way: the launcher family
+# (@kernel_launcher, 39 call sites) is exactly the shape that produces breaks, and the
+# object this one closes over is bound after use (see below).
+#
+# THE BREAK (measured, TORCH_LOGS=graph_breaks, rank0, 2026-09-30).  With 01+02+03 the
+# trunk traces, and with AOT off + fullgraph=True dynamo reports a FATAL break — fatal
+# even with VLLM_COMPILE_FULLGRAPH=0, because dynamo cannot build a resume function
+# around it:
+#
+#   torch._dynamo.exc.Unsupported: Read uninitialized cell
+#   Explanation: Attempted to read a cell variable that has not been populated yet.
+#   Developer debug context: CellVariable()
+#
+# User stack:
+#   glm5next/common/model.py:748    Glm5NextModel.forward       layer(...)
+#   glm5next/common/model.py:526    Glm5NextDecoderLayer.forward
+#   glm5next/common/attention.py:590
+#   model_executor/layers/mla.py:208
+#     q_proj_input, kv_c_normed = fused_q_kv_rmsnorm(...)
+#   -> torch/_dynamo/polyfills/__init__.py:392 getattr_and_trace
+#
+# So dynamo is tracing the fork's JIT-kernel launcher framework
+# (VllmTritonJitKernel + @kernel_launcher — 39 call sites) and finds a closure name
+# that is not yet bound.  In models/common/ops/fused_qk_rmsnorm.py the kernel object
+# is bound AFTER the function that closes over it:
+#
+#   def fused_q_kv_rmsnorm(...):          # line 194
+#       ...
+#       _FUSED_Q_KV_RMSNORM_KERNEL(...)   # line 211
+#   _FUSED_Q_KV_RMSNORM_KERNEL = FusedQKVRMSNormKernel()    # line 223
+#
+# Two ways out, and this patch takes the cheap one:
+#   (a) WORKAROUND (here): stop calling the fused kernel from the traced region.  The
+#       flag is already a constructor argument (mla.py:80, default False) that glm5next
+#       turns ON at attention.py:583, so it becomes an env knob.
+#   (b) ROOT FIX (not done): make the launcher trace-safe for all 39 sites.  Any other
+#       @kernel_launcher-backed kernel still in the traced path can produce the same
+#       fatal break, so (b) is the durable fix and (a) is what unblocks iterating.
+#
+# COST of R4D_GLM_FUSE_QKV_RMSNORM=0: one extra small RMSNorm launch per MLA layer per
+# token (q_a_layernorm + kv_a_layernorm instead of one fused kernel) — ~2 launches out
+# of ~3,700 per token on this shape, against the 44 % of device time currently spent in
+# unfused pointwise work.  Correct trade for a compile experiment.
+#
+# TARGET: vllm/models/glm5next/common/attention.py   (inside the pinned image)
+#   base    sha256 362bbf0e6c50cdfa446c6551b9b1ddff425e08f88e3d4140fd8db88f8a619e0c
+#   patched sha256 84969993c250e9f8183a33a6e1ddd607272f48b950b560272eb33b7cdde31189
+#
+# APPLY: `patch -p1` from the vllm package root (verified: applies clean, reproduces
+# the patched sha256), or bind-mount new/.../attention.py over site-packages.
+# R4D_GLM_FUSE_QKV_RMSNORM=1 (default) restores the fused kernel.
+--- a/vllm/models/glm5next/common/attention.py
+++ b/vllm/models/glm5next/common/attention.py
+@@ -1,6 +1,7 @@
+ # SPDX-License-Identifier: Apache-2.0
+ # SPDX-FileCopyrightText: Copyright contributors to the vLLM project
+ 
+import os
+ import torch
+ import torch.nn.functional as F
+ from torch import nn
+@@ -36,6 +37,14 @@
+ from vllm.utils.deep_gemm import PAGED_MQA_PAGE_SIZES
+ from vllm.v1.kv_cache_interface import KpoolTailSpec, MLAAttentionSpec
+ 
+
+# FORK-LOCAL (patch: keep the traced region trace-safe).  R4D_GLM_FUSE_QKV_RMSNORM=0
+# replaces the fused `fused_q_kv_rmsnorm` JIT-kernel launch (one kernel) with two plain
+# RMSNorms, because dynamo cannot trace the launcher's closure ("Read uninitialized cell",
+# fatal even with fullgraph=False).  The fused kernel object in
+# models/common/ops/fused_qk_rmsnorm.py is bound AFTER the function that uses it.
+_FUSE_QKV_RMSNORM = os.environ.get("R4D_GLM_FUSE_QKV_RMSNORM", "1") == "1"
+
+ logger = init_logger(__name__)
+ 
+ # Shared torch.compile config for the indexer's small-kernel leaves. The MLA
+@@ -580,7 +589,7 @@
+             quant_config,
+             prefix,
+             skip_topk=False,
+-            fuse_qkv_rmsnorm=True,
+            fuse_qkv_rmsnorm=_FUSE_QKV_RMSNORM,
+         )
+ 
+     def forward(
@@ -0,0 +1,96 @@
+# glm53-compile-05-workspace-acquisition-opaque.patch — the indexer's workspace
+# acquisition must not be traced.
+#
+# STATUS: WIP — fifth of the "compile the GLM trunk" changelist. This is a break worth
+# removing, but NOT part of the confirmed blocker and NOT required for the
+# break-tolerant route: `torch.compiler.disable` is itself a graph break, so this patch
+# helps only a hypothetical fullgraph=True route, where the workspace acquisition would
+# be better RESTRUCTURED out of the forward (allocate at init/profile time) than made
+# opaque. Kept because the discovery is real and the wrapper is a no-op when nothing is
+# compiling.
+#
+# THE BREAK — CONFIRMED with AND WITHOUT the logging flag (2026-09-30):
+#
+#   torch._dynamo.exc.Unsupported: Unsupported method call
+#   user stack
+#     glm5next/common/model.py:748      Glm5NextModel.forward
+#     glm5next/common/model.py:526      Glm5NextDecoderLayer.forward
+#     glm5next/common/attention.py:599 / :401
+#     glm5next/amd/sparse_indexer.py:711  forward_native
+#     glm5next/amd/sparse_indexer.py:132  sparse_attn_indexer_kpool
+#     v1/worker/workspace.py:135          get_simultaneous
+#     v1/worker/workspace.py:155          _ensure_workspace_size
+#
+# Line 132 is the PROFILE-RUN branch — the source comment says what it is:
+#
+#     if not isinstance(attn_metadata, dict):
+#         # Reserve workspace for indexer during profiling run
+#
+# so the engine dies exactly where it sizes its memory.  `_ensure_workspace_size`
+# reads ContextVar and ubatch state (`_workspace_lane.get()`,
+# `dbo_current_ubatch_id()`) and may allocate; none of that is traceable.
+#
+# FIX: the documented escape hatch — `torch.compiler.disable` around a thin wrapper,
+# so dynamo stops analysing the call and executes it eagerly from inside the compiled
+# region.  Chosen over decorating the workspace manager's own methods because that
+# would change workspace behaviour for every model, not just this one.  The wrapper is
+# a no-op when nothing is compiling, so it needs no env gate and is safe in the
+# stock (uncompiled) path — which is why this patch can be reasoned about
+# independently of whether the changelist ever ships.
+#
+# CAVEAT, stated plainly: a compiler-disabled region is still a graph break, so this
+# needs patch 03 (VLLM_COMPILE_FULLGRAPH=0) to be tolerable at all, and whether dynamo
+# can build a resume function around it is exactly what the next boot decides — the
+# difference between this failure and 04's `Read uninitialized cell`.
+#
+# TARGET: vllm/models/glm5next/amd/sparse_indexer.py   (inside the pinned image)
+#   base    sha256 f773a99265d3f440c8fad22c8fb5a8b33f926162094cffc3a2ecdaa42763c5b1
+#   patched sha256 c154115afe8253a35c652b7198929ef4c1874957d333fdb3340cacf88e612b78
+#
+# APPLY: `patch -p1` from the vllm package root, or bind-mount over site-packages.
+--- a/vllm/models/glm5next/amd/sparse_indexer.py
+++ b/vllm/models/glm5next/amd/sparse_indexer.py
+@@ -35,6 +35,21 @@
+ from vllm.v1.attention.ops.common import pack_seq_triton, unpack_seq_triton
+ from vllm.v1.worker.workspace import current_workspace_manager
+ 
+@torch.compiler.disable
+def _acquire_workspace(*shapes_and_dtypes):
+    """Acquire shared workspace buffers, untraced.
+
+    Workspace acquisition reads ContextVar/ubatch state and may allocate
+    (v1/worker/workspace.py: ``_ensure_workspace_size`` -> ``_workspace_lane.get()``,
+    ``dbo_current_ubatch_id()``), which dynamo cannot trace -- it raises
+    ``Unsupported: Unsupported method call`` and kills the engine during the profile
+    run.  ``torch.compiler.disable`` keeps the call as an opaque, eagerly-executed
+    region inside the compiled graph.  No-op when not compiling.
+    """
+
+    return current_workspace_manager().get_simultaneous(*shapes_and_dtypes)
+
+
+ logger = init_logger(__name__)
+ 
+ # kpool write helper: form pools from the current token batch and compress them
+@@ -129,7 +144,7 @@
+         values_spec, scales_spec = _gather_workspace_shapes(
+             total_seq_lens, head_dim, fp8_dtype, use_fp4_cache
+         )
+-        current_workspace_manager().get_simultaneous(
+        _acquire_workspace(
+             values_spec,
+             scales_spec,
+             ((RADIX_TOPK_WORKSPACE_SIZE,), torch.uint8),
+@@ -275,11 +290,10 @@
+         # Layout switches between FP8 (head_dim bytes + 4-byte fp32 scale) and
+         # MXFP4 (head_dim/2 bytes packed + head_dim/MXFP4_BLOCK_SIZE ue8m0
+         # scales) based on use_fp4_cache.
+-        workspace_manager = current_workspace_manager()
+         values_spec, scales_spec = _gather_workspace_shapes(
+             total_seq_lens, head_dim, fp8_dtype, use_fp4_cache
+         )
+-        k_quant_full, k_scale_full = workspace_manager.get_simultaneous(
+        k_quant_full, k_scale_full = _acquire_workspace(
+             values_spec,
+             scales_spec,
+         )
@@ -0,0 +1,173 @@
+# glm53-dflash-01-aux-hidden-states.patch — give the GLM trunk the EAGLE3 / DFlash2
+# auxiliary-hidden-state interface.
+#
+# STATUS: WIP, and INDEPENDENT of the compile changelist (01-05). It patches the same
+# file as glm53-compile-01-model-decorator.patch, so if both are ever applied, apply 01
+# first. Not part of the deployed shape.
+#
+# WHY.  `--speculative-config '{"method":"dflash","model":"/path/to/drafter", ...}'` with
+# incoai/GLM-5.3-Flash-DFlash2 is accepted -- the config parses, the drafter loads -- and
+# then the worker dies at init with:
+#
+#     RuntimeError: Model does not support EAGLE3 interface
+#
+# raised by v1/worker/gpu/spec_decode/eagle/eagle3_utils.py:20 through
+# `supports_eagle3(model)` == `isinstance(model, SupportsEagle3)` (a runtime-checkable
+# Protocol).  Nothing about the drafter, the method name or the payload is wrong:
+# `glm5next` implements neither half of the contract.
+#
+# THE CONTRACT (read out of this image, not assumed):
+#   * DEFINITIVE (v5, after four failed boots): the member set is checked against
+#     the INSTANCE the runner calls -- which for GLM-5.3-Flash is
+#     Glm5NextForConditionalGeneration, NOT Glm5NextForCausalLM.  The multimodal
+#     wrapper wraps the text model as self.language_model
+#     (init_vllm_registered_model, architectures=["Glm5NextForCausalLM"],
+#     glm5next/common/model.py:1113), and model_runner.py:425 calls
+#     set_eagle3_aux_hidden_state_layers(self.model) on the CONDITIONAL.
+#     Presence rule (verified in-image with stock typing, python 3.14):
+#     isinstance walks ALL __protocol_attrs__ -- members of SupportsEagle3 AND
+#     its base SupportsEagleBase (has_own_lm_head, has_own_embed_tokens) -- and
+#     getattr_static()s each one off the instance, breaking on the first miss.
+#     So BOTH wrappers need all three attributes.
+#   * MRO law: INHERITING SupportsEagle3 as a base dies at class creation with
+#     "Cannot create a consistent method resolution order (MRO) for bases ..."
+#     (GLM's wrappers already carry four/five Protocol-derived bases).  The
+#     attribute route avoids the MRO entirely.  Also proven by simulation:
+#     the four-base ForCausalLM ordering cannot take SupportsEagle3 in ANY
+#     position.
+#   * Method placement: without inheriting, the Protocol's default
+#     set_aux_hidden_state_layers / get_eagle3_default_aux_hidden_state_layers
+#     do not exist on either wrapper, so both are defined explicitly --
+#     ForCausalLM unwraps to self.model (the EagleModelMixin trunk, v1's
+#     correct bodies), the conditional delegates to self.language_model
+#     (the pixtral.py:498 pattern).
+#   * interfaces.py:1673-1719 -- for reference, the inherited defaults unwrap
+#     `holder = getattr(parent_ref, "model", parent_ref)` and assert
+#     `isinstance(holder, EagleModelMixin)`, defaulting to (2, L // 2, L - 3).
+#   * eagle3_utils.py:26-31 -- layers come from the spec config when it carries them
+#     (this drafter ships target_layer_ids [5, 14, 24, 33, 42]), else from the default.
+#   * deepseek_v2.py:1551-1585 and v1/worker/gpu/cudagraph_utils.py:621 -- the trunk
+#     appends one tensor per aux layer while looping and returns the TUPLE
+#     (hidden_states, aux_hidden_states) only when that list is non-empty.
+#
+# WHAT IT ADDS for GLM: EagleModelMixin on Glm5NextModel; aux capture in the layer
+# loop, where the aux state is the single-stream `hidden_states` ENTERING that layer
+# (GLM's loop carries the already-summed single-stream state -- its non-mHC branch
+# re-splits `residual = hidden_states` inside the layer -- so this is the equivalent
+# of deepseek_v2's `hidden_states + residual`); the conditional tuple return; the
+# three SupportsEagle3 member attributes on BOTH wrappers; and the two interface
+# methods on both (ForCausalLM: unwrap to trunk; conditional: delegate to
+# language_model).
+#
+# HONEST SCOPE: this makes the interface PRESENT, which is exactly what the engine
+# checks -- it does not make drafts GOOD, and per the measured profile it is not a
+# decode-speed fix (a verification pass pays pointwise cost proportional to its token
+# count, which is why MTP4 measured flat).  It is the prerequisite for driving DFlash2
+# on this endpoint at all.
+#
+# TARGET: vllm/models/glm5next/common/model.py   (inside the pinned image)
+#   base    sha256 008eeaa2a07e091465319f86900b5a321d921238723ccbe22e6b9e971e1d2b28
+#   patched sha256 bc9913a15c8e7f88e0c36b07bf029b383cc8c2078b58d66dc91aa235f8af77f4
+#
+# APPLY: `patch -p1` from the vllm package root, or bind-mount the patched file over
+# site-packages (which is what a scratch-launcher DFLASH_PATCH=1 does).
+--- a/vllm/models/glm5next/common/model.py
+++ b/vllm/models/glm5next/common/model.py
+@@ -95,6 +95,12 @@
+ logger = init_logger(__name__)
+ 
+ 
+from vllm.model_executor.models.interfaces import (
+    EagleModelMixin,
+    SupportsEagle3,
+)
+
+
+ class Glm5NextMLP(nn.Module):
+     def __init__(
+         self,
+@@ -623,7 +629,7 @@
+         )
+ 
+ 
+-class Glm5NextModel(nn.Module):
+class Glm5NextModel(nn.Module, EagleModelMixin):
+     def __init__(self, *, vllm_config: VllmConfig, prefix: str = ""):
+         super().__init__()
+ 
+@@ -730,7 +736,14 @@
+         if self.is_sequence_parallel:
+             hidden_states = sp_shard(hidden_states)
+ 
+-        for layer in self._active_layers:
+        aux_hidden_states: list[torch.Tensor] = []
+        for local_idx, layer in enumerate(self._active_layers):
+            if (getattr(self, "start_layer", 0) + local_idx) in self.aux_hidden_state_layers:
+                # Auxiliary hidden state for EAGLE-3 / DFlash2 drafts: the
+                # single-stream state ENTERING this layer, before its input norm --
+                # the same semantic deepseek_v2 implements by appending
+                # hidden_states + residual at this point in its loop.
+                aux_hidden_states.append(hidden_states)
+             hidden_states, residual, post, comb = layer(
+                 positions, hidden_states, residual, post, comb
+             )
+@@ -749,6 +762,8 @@
+             hidden_states = sp_all_gather(hidden_states)[:full_num_tokens]
+ 
+         hidden_states = self.norm(hidden_states)
+        if len(aux_hidden_states) > 0:
+            return hidden_states, aux_hidden_states
+         return hidden_states
+ 
+     def load_weights(self, weights: Iterable[tuple[str, torch.Tensor]]) -> set[str]:
+@@ -937,6 +952,10 @@
+ class Glm5NextForCausalLM(
+     nn.Module, HasInnerState, SupportsPP, MixtureOfExperts, IsHybrid
+ ):
+    supports_eagle3: ClassVar[Literal[True]] = True
+    has_own_lm_head: bool = False
+    has_own_embed_tokens: bool = False
+
+     def __init__(self, *, vllm_config: VllmConfig, prefix: str = ""):
+         super().__init__()
+         self.model_config = vllm_config.model_config
+@@ -963,6 +982,14 @@
+     def embed_input_ids(self, input_ids: torch.Tensor) -> torch.Tensor:
+         return self.model.embed_input_ids(input_ids)
+ 
+    def set_aux_hidden_state_layers(self, layers: tuple[int, ...]) -> None:
+        # EAGLE-3 / DFlash2 interface: unwrap to the trunk (the EagleModelMixin).
+        self.model.aux_hidden_state_layers = layers
+
+    def get_eagle3_default_aux_hidden_state_layers(self) -> tuple[int, ...]:
+        num_layers = len(self.model.layers)
+        return (2, num_layers // 2, num_layers - 3)
+
+     def forward(
+         self,
+         input_ids: torch.Tensor | None,
+@@ -1033,6 +1060,10 @@
+ class Glm5NextForConditionalGeneration(
+     Glm4vForConditionalGeneration, HasInnerState, IsHybrid, MixtureOfExperts
+ ):
+    supports_eagle3: ClassVar[Literal[True]] = True
+    has_own_lm_head: bool = False
+    has_own_embed_tokens: bool = False
+
+     # The text model (KDA + dense-MLA + MoE) is a hybrid mamba model. The
+     # multimodal wrapper must declare the same interfaces so vLLM treats it as
+     # hybrid (auto-aligns mamba/attention block sizes, sizes the mamba state
+@@ -1076,6 +1107,13 @@
+ 
+         return Glm5NextForCausalLM.get_mamba_state_copy_func()
+ 
+    def set_aux_hidden_state_layers(self, layers: tuple[int, ...]) -> None:
+        # EAGLE-3 / DFlash2: delegate to the language model (pixtral.py:498).
+        self.language_model.set_aux_hidden_state_layers(layers)
+
+    def get_eagle3_default_aux_hidden_state_layers(self) -> tuple[int, ...]:
+        return self.language_model.get_eagle3_default_aux_hidden_state_layers()
+
+     def __init__(self, *, vllm_config: VllmConfig, prefix: str = ""):
+         super(Glm4vForConditionalGeneration, self).__init__()
+         config = vllm_config.model_config.hf_config
@@ -0,0 +1,317 @@
+--- a/vllm/v1/core/kv_cache_utils.py
+++ b/vllm/v1/core/kv_cache_utils.py
+@@ -1210,17 +1210,38 @@
+         for name, spec in kv_cache_spec.items()
+         if isinstance(spec, KpoolTailSpec)
+     }
+    hidden_specs = {
+        name: spec
+        for name, spec in kv_cache_spec.items()
+        if isinstance(spec, HiddenStateCacheSpec)
+    }
+     attn_specs = {
+         name: spec
+         for name, spec in kv_cache_spec.items()
+-        if not isinstance(spec, (MambaSpec, KpoolTailSpec))
+        if not isinstance(
+            spec, (MambaSpec, KpoolTailSpec, HiddenStateCacheSpec)
+        )
+     }
+-    if not mamba_specs or not all(
+-        type(spec) is MLAAttentionSpec for spec in attn_specs.values()
+-    ):
+    if not mamba_specs or not attn_specs:
+        return None
+    # FORK-LOCAL (patches/glm53-dflash-04-kv-groups-draft): with a dflash
+    # drafter, attn_specs also carries the drafter's attention layers. The
+    # GLM MLA layers keep their own group (no page unification -- per-group
+    # pages are the glm5 design); the drafter layers must form one uniform
+    # group of their own, and the hidden-state layers get per-layer groups
+    # like the generic tail.
+    mla_specs = {
+        name: spec
+        for name, spec in attn_specs.items()
+        if type(spec) is MLAAttentionSpec
+    }
+    draft_specs = {
+        name: spec for name, spec in attn_specs.items() if name not in mla_specs
+    }
+    if not mla_specs:
+        return None
+    if draft_specs and UniformTypeKVCacheSpecs.from_specs(draft_specs) is None:
+         return None
+-
+-    mla_specs = cast(dict[str, MLAAttentionSpec], attn_specs)
+     idx_pages = {
+         spec.page_size_bytes for spec in mla_specs.values() if spec.tokens_per_state > 1
+     }
+@@ -1233,7 +1254,7 @@
+     mla_pages = {mla_specs[name].page_size_bytes for name in mla_names}
+     assert len(mla_pages) == 1
+     mla_page = mla_pages.pop()
+-    uniform_spec = UniformTypeKVCacheSpecs.from_specs(attn_specs)
+    uniform_spec = UniformTypeKVCacheSpecs.from_specs(mla_specs)
+     assert uniform_spec is not None
+ 
+     tail_group: KVCacheGroupSpec | None = None
+@@ -1270,10 +1291,20 @@
+     for index, name in enumerate(mamba_specs):
+         mamba_grouped_names[index % num_groups].append(name)
+ 
+    draft_group: KVCacheGroupSpec | None = None
+    if draft_specs:
+        draft_uniform = UniformTypeKVCacheSpecs.from_specs(draft_specs)
+        assert draft_uniform is not None
+        draft_group = KVCacheGroupSpec(list(draft_specs), draft_uniform)
+    hidden_groups = [
+        KVCacheGroupSpec([name], spec) for name, spec in hidden_specs.items()
+    ]
+     return (
+-        [KVCacheGroupSpec(list(attn_specs), uniform_spec)]
+        [KVCacheGroupSpec(list(mla_specs), uniform_spec)]
+         + ([tail_group] if tail_group is not None else [])
+        + ([draft_group] if draft_group is not None else [])
+         + create_kv_cache_group_specs(padded_specs, mamba_grouped_names)
+        + hidden_groups
+     )
+ 
+ 
+@@ -1289,10 +1320,17 @@
+         int,
+         list[str],
+         int,
+        KVCacheGroupSpec | None,
+        list[KVCacheGroupSpec],
+     ]
+     | None
+ ):
+-    """Recognize the GLM-5.3-Flash grouping after optional PP projection."""
+    """Recognize the GLM-5.3-Flash grouping after optional PP projection.
+
+    FORK-LOCAL (patches/glm53-dflash-04-kv-groups-draft): also recognizes the
+    drafter's uniform group and the per-layer hidden-state groups, returned
+    as ``draft_group`` / ``hidden_groups`` so the pool builder and the memory
+    estimator can size their tensors beside the aliased GLM pool."""
+     uniform_groups = [
+         group
+         for group in kv_cache_groups
+@@ -1301,17 +1339,28 @@
+     mamba_groups = [
+         group for group in kv_cache_groups if isinstance(group.kv_cache_spec, MambaSpec)
+     ]
+    hidden_groups = [
+        group
+        for group in kv_cache_groups
+        if isinstance(group.kv_cache_spec, HiddenStateCacheSpec)
+    ]
+     attn_group: KVCacheGroupSpec | None = None
+     tail_group: KVCacheGroupSpec | None = None
+    draft_group: KVCacheGroupSpec | None = None
+     for group in uniform_groups:
+         inner = cast(UniformTypeKVCacheSpecs, group.kv_cache_spec).kv_cache_specs
+         if all(type(spec) is MLAAttentionSpec for spec in inner.values()):
+             attn_group = group
+         elif all(isinstance(spec, KpoolTailSpec) for spec in inner.values()):
+             tail_group = group
+        else:
+            draft_group = group
+     if attn_group is None or not mamba_groups:
+         return None
+-    if len(uniform_groups) + len(mamba_groups) != len(kv_cache_groups):
+    if (
+        len(uniform_groups) + len(mamba_groups) + len(hidden_groups)
+        != len(kv_cache_groups)
+    ):
+         return None
+ 
+     attn_uniform = cast(UniformTypeKVCacheSpecs, attn_group.kv_cache_spec)
+@@ -1362,6 +1411,8 @@
+         idx_page,
+         tail_names,
+         tail_page,
+        draft_group,
+        hidden_groups,
+     )
+ 
+ 
+@@ -1395,10 +1446,34 @@
+         return kv_cache_spec
+ 
+     max_page_size = max(page_sizes)
+    # GLM-5.3-Flash + a draft model: the sparse indexer k_cache page is
+    # tokens_per_state x the MLA page (same per-token bytes), and MLA layers
+    # cannot be PADDED to an arbitrary page size (sparse MLA indexes the cache
+    # in whole token rows) -- but they CAN grow their block size when the max
+    # page is an exact multiple of the MLA page. Bump the max page to a
+    # multiple of every MLA-family page so all of them take the divisible
+    # block-growth branch below instead of the padding refusal, while the
+    # draft and mamba layers simply pad to the (slightly larger) page.
+    mla_pages = {
+        spec.page_size_bytes
+        for spec in kv_cache_spec.values()
+        if isinstance(spec, MLAAttentionSpec)
+    }
+    if mla_pages:
+        mla_common = math.lcm(*mla_pages)
+        max_page_size = -(-max_page_size // mla_common) * mla_common
+     new_kv_cache_spec = {}
+     for layer_name, layer_spec in kv_cache_spec.items():
+         if layer_spec.page_size_bytes == max_page_size:
+             new_kv_cache_spec[layer_name] = layer_spec
+        elif isinstance(layer_spec, KpoolTailSpec):
+            # The kpool tail kernel pins the tail block to the kpool size
+            # (tail_kv_cache.view(...) must be [blocks, 2, index_kpool,
+            # head_dim]); growing block_size breaks that contract. Pad the
+            # physical page instead -- the tail is its own group and the pad
+            # wastes bytes only.
+            new_spec = replace(layer_spec, page_size_padded=max_page_size)
+            new_kv_cache_spec[layer_name] = new_spec
+         elif isinstance(layer_spec, MambaSpec):
+             # MambaSpec's page size is determined by its state shapes and does
+             # not scale with block_size, so pad the page instead. This is the
+@@ -1593,8 +1668,31 @@
+ ) -> int:
+     """Return the largest cache group's bytes per block."""
+     if (glm5_layout := _glm5_next_tensor_layout(kv_cache_groups)) is not None:
+-        _, _, mla_names, idx_names, mla_page, idx_page, _, _ = glm5_layout
+-        return len(mla_names) * mla_page + len(idx_names) * idx_page
+        (
+            _,
+            _,
+            mla_names,
+            idx_names,
+            mla_page,
+            idx_page,
+            _,
+            _,
+            draft_group,
+            hidden_groups,
+        ) = glm5_layout
+        bytes_per_block = len(mla_names) * mla_page + len(idx_names) * idx_page
+        if draft_group is not None:
+            d_inner = cast(
+                UniformTypeKVCacheSpecs, draft_group.kv_cache_spec
+            ).kv_cache_specs
+            bytes_per_block += sum(
+                spec.page_size_bytes for spec in d_inner.values()
+            )
+        for group in hidden_groups:
+            bytes_per_block += (
+                group.kv_cache_spec.page_size_bytes * len(group.layer_names)
+            )
+        return bytes_per_block
+ 
+     bytes_per_block = max(
+         sum(
+@@ -1701,12 +1799,30 @@
+             idx_page,
+             tail_names,
+             _,
+            draft_group,
+            hidden_groups,
+         ) = glm5_layout
+         bytes_per_block = len(mla_names) * mla_page + len(idx_names) * idx_page
+        extra_bytes_per_block = 0
+        if draft_group is not None:
+            d_inner = cast(
+                UniformTypeKVCacheSpecs, draft_group.kv_cache_spec
+            ).kv_cache_specs
+            extra_bytes_per_block += sum(
+                spec.page_size_bytes for spec in d_inner.values()
+            )
+        for group in hidden_groups:
+            extra_bytes_per_block += (
+                group.kv_cache_spec.page_size_bytes * len(group.layer_names)
+            )
+         num_blocks = may_override_num_blocks(
+-            vllm_config, available_memory // bytes_per_block
+            vllm_config,
+            available_memory // (bytes_per_block + extra_bytes_per_block),
+         )
+-        size = bytes_per_block * num_blocks
+        # One backing allocation shared by every tensor (allocate_kv_cache
+        # asserts all tensor sizes equal): the aliased GLM pool plus the
+        # draft/hidden regions appended after it.
+        size = (bytes_per_block + extra_bytes_per_block) * num_blocks
+         attn_specs = cast(
+             UniformTypeKVCacheSpecs, attn_group.kv_cache_spec
+         ).kv_cache_specs
+@@ -1745,6 +1861,37 @@
+                 ).kv_cache_specs
+                 add_tensor(tail_name, tail_specs[tail_name], offset)
+ 
+        # Draft + hidden-state layers get their own tensors appended after
+        # the aliased pool (no aliasing: independent block tables).
+        offset_cursor = bytes_per_block * num_blocks
+        if draft_group is not None:
+            d_inner = cast(
+                UniformTypeKVCacheSpecs, draft_group.kv_cache_spec
+            ).kv_cache_specs
+            d_page = next(iter(d_inner.values())).page_size_bytes
+            kv_cache_tensors.append(
+                KVCacheTensor(
+                    size=size,
+                    layers=list(draft_group.layer_names),
+                    layer_stride=d_page * num_blocks,
+                    block_stride=d_page,
+                    offset=offset_cursor,
+                )
+            )
+            offset_cursor += d_page * num_blocks * len(draft_group.layer_names)
+        for group in hidden_groups:
+            h_page = group.kv_cache_spec.page_size_bytes
+            kv_cache_tensors.append(
+                KVCacheTensor(
+                    size=size,
+                    layers=list(group.layer_names),
+                    layer_stride=h_page * num_blocks,
+                    block_stride=h_page,
+                    offset=offset_cursor,
+                )
+            )
+            offset_cursor += h_page * num_blocks * len(group.layer_names)
+
+         return KVCacheConfig(
+             num_blocks=num_blocks,
+             kv_cache_tensors=kv_cache_tensors,
+@@ -2440,6 +2587,8 @@
+             idx_page,
+             tail_names,
+             _,
+            draft_group,
+            hidden_groups,
+         ) = glm5_layout
+         uniform_spec = cast(UniformTypeKVCacheSpecs, attn_group.kv_cache_spec)
+         total_blocks = uniform_spec.max_memory_usage_pages(vllm_config)
+@@ -2452,7 +2601,30 @@
+         )
+         if tail_names:
+             total_blocks += 1
+-        return total_blocks * (len(mla_names) * mla_page + len(idx_names) * idx_page)
+        pool_bytes = total_blocks * (
+            len(mla_names) * mla_page + len(idx_names) * idx_page
+        )
+        extra_bytes = 0
+        if draft_group is not None:
+            d_inner = cast(
+                UniformTypeKVCacheSpecs, draft_group.kv_cache_spec
+            ).kv_cache_specs
+            d_spec = next(iter(d_inner.values()))
+            d_blocks = cdiv(
+                d_spec.max_memory_usage_bytes(vllm_config), d_spec.page_size_bytes
+            )
+            extra_bytes += d_blocks * d_spec.page_size_bytes * len(
+                draft_group.layer_names
+            )
+        for group in hidden_groups:
+            h_spec = group.kv_cache_spec
+            h_blocks = cdiv(
+                h_spec.max_memory_usage_bytes(vllm_config), h_spec.page_size_bytes
+            )
+            extra_bytes += (
+                h_blocks * h_spec.page_size_bytes * len(group.layer_names)
+            )
+        return pool_bytes + extra_bytes
+ 
+     bytes_per_block = _pool_bytes_per_block(kv_cache_groups)
+     total_blocks = 0
@@ -0,0 +1,31 @@
+--- a/vllm/model_executor/offloader/expert_activation_registry.py
+++ b/vllm/model_executor/offloader/expert_activation_registry.py
+@@ -316,7 +316,27 @@
+ 
+ 
+ def _glm53_kv_bytes_per_token(cfg: "VllmConfig") -> int:
+-    return 13600
+    per_tok = 13600
+    # FORK-LOCAL (patches/glm53-dflash-04-kv-groups-draft): with a dflash/dspark
+    # drafter, the KV spec set gains the drafter's attention layers and the aux
+    # hidden-state cache (one bf16 hidden vector per target aux layer per
+    # token). Both are full-length per-token costs the aliased glm5 pool
+    # cannot absorb, so the reserve must include them. The drafter is a 5-layer
+    # qwen3 (8 kv heads x 128 x K+V x bf16 = 4096 B/token); the aux count is
+    # the dflash target_layer_ids count (5).
+    # NOTE: use_aux_hidden_state_outputs lives on the gpu_model_runner
+    # instance (gpu_model_runner.py:623/678), not on the spec config -- it is
+    # not set yet when the expert plan runs. The spec-config METHOD is the
+    # plan-time signal for the same fact.
+    sc = getattr(cfg, "speculative_config", None)
+    if sc is not None and getattr(sc, "method", "") in (
+        "dflash",
+        "dspark",
+        "extract_hidden_states",
+    ):
+        hidden = int(cfg.model_config.hf_config.hidden_size)
+        per_tok += 5 * hidden * 2 + 4096
+    return per_tok
+ 
+ 
+ for _arch in ("Glm5NextForConditionalGeneration", "Glm5NextForCausalLM"):
@@ -0,0 +1,65 @@
+# gpu_worker-kv-cap.patch — fork-local fix for the expert-offload KV cap.
+#
+# STATUS: NOT part of the deployed frank-primary shape. It exists because the
+# fork's plan under-models this model's KV, and it is the enabler for a future
+# MTP attempt (the production GLM shape runs without MTP and boots fine on the
+# plan's own reserve).
+#
+# THE BUG: the plan models KV as 13600 B/token + a small mamba term, but
+# GLM-5.3-Flash also carries a per-sequence KDA state (34 linear-attention
+# layers) and, under MTP, the draft model's KV. gpu_worker then CAPS the KV
+# cache at that reserve, so the plan's own arithmetic can make a legal
+# --max-model-len unbootable. Measured 2026-09-30 on the pinned image:
+#   - no MTP, max_model_len 131072: reserve 1.74 GiB >= need 1.86 GiB  -> boots
+#   - MTP4,  max_model_len 131072: reserve 1.74 GiB <  need 2.07 GiB  -> refuses
+#   - MTP4,  max_model_len 262144: reserve 3.43 GiB <  need 3.62 GiB  -> refuses
+#     ("To serve at least one request with the model's max seq len ...")
+# Raising max_model_len does not fix it: the cap's slope (~13,845 B/token) only
+# overtakes the need's (~12,700 B/token + 0.52 GiB fixed) around 440k tokens,
+# where the KV reserve would eat a third of the expert arena.
+#
+# THE FIX: let an explicit operator KV request raise the cap. The compile and
+# graph-capture headroom still comes off the top, so the blast radius is one
+# flag the operator chose.
+#
+# APPLY (into a host dir that is then bind-mounted read-only over the image):
+#   IMAGE=docker.io/tcclaviger/vllm@sha256:da4029666668b90a0386c1ed23ea370c79916ad4e1a6ad1dd9db098b9fa069ea
+#   D=/home/dank/glm-patch; mkdir -p $D/vllm/v1/worker
+#   podman run --rm --entrypoint cat $IMAGE \
+#     /opt/vllm/lib/python3.14/site-packages/vllm/v1/worker/gpu_worker.py \
+#     > $D/vllm/v1/worker/gpu_worker.py
+#   ( cd $D && patch -p1 < .../scripts/patches/gpu_worker-kv-cap.patch )
+#   # then add to the podman run:
+#   #   -v $D/vllm/v1/worker/gpu_worker.py:/opt/vllm/lib/python3.14/site-packages/vllm/v1/worker/gpu_worker.py:ro
+#   #   -e CLAV_KV_RESERVE_FLOOR_GIB=<needed>   (or pass --kv-cache-memory-bytes)
+#
+# Base file sha256 (the pristine file out of the pinned image):
+#   1f3942b1a01af075317161b06077a2e658086f1da0566c9f297129ceebe37dd7
+# Patched file sha256 (verified 2026-09-30, patch applies clean and reproduces
+# these bytes exactly):
+#   80cd160bbc01270b0851f7b6fe7552b06ae2fae2428134fc29238d7f452af840
+#
+--- a/vllm/v1/worker/gpu_worker.py
+++ b/vllm/v1/worker/gpu_worker.py
+@@ -824,6 +824,21 @@
+         expert_cfg = self.vllm_config.offload_config.expert
+         if expert_cfg.enable_expert_offload and expert_cfg.planned_kv_reserve_bytes > 0:
+             cap = int(expert_cfg.planned_kv_reserve_bytes)
+            # LOCAL PATCH (GLM-5.3 KDA + MTP draft): the plan models KV as
+            # 13600 B/token + a small mamba term, but this model also carries a
+            # per-sequence KDA state and (under MTP) the draft's KV, so the
+            # reserve can sit BELOW what max_model_len legitimately needs -- and
+            # the cap then makes a legal config unbootable (measured: need
+            # 2.07 GiB vs cap 1.74 GiB at max_model_len 131072 with MTP). An
+            # explicit --kv-cache-memory-bytes, or CLAV_KV_RESERVE_FLOOR_GIB,
+            # is the operator's word: let it raise the cap. The compile/capture
+            # headroom still comes off the top.
+            _explicit = self.cache_config.kv_cache_memory_bytes
+            _floor = os.environ.get("CLAV_KV_RESERVE_FLOOR_GIB")
+            if _explicit:
+                cap = max(cap, int(_explicit))
+            if _floor:
+                cap = max(cap, int(float(_floor) * (1 << 30)))
+             if self.available_kv_cache_memory_bytes > cap:
+                 logger.info_once(
+                     "expert offload: KV capped at the plan's reserve %s GiB; %s GiB of "
@@ -1,121 +1,146 @@
 #!/usr/bin/env bash
-# Serve Swift-1.5-Qwen3.8-27B-Uncensored (abliterated) as frank-large on
-# Dankclaw's Strix Halo iGPU (gfx1151), using gufo (gufo-org/gufo) — the
-# inference engine built specifically for this hardware.
+# serve-frank-large.sh — frank-large endpoint: halogen (closed-source
+# C++/HIP engine, image ghcr.io/peonist-ai/halogen, digest-pinned in
+# frank-large.env) serving peonist-ai/halogen-qwen3.8-27b on Dankclaw's
+# Strix Halo iGPU, host :7770. 2026-10-05 cutover from the ROCmFPX
+# llama.cpp shape — revert the swap commit's trio to roll back (payload
+# and fork build stay on disk).
 #
-# Runtime: /home/dank/GitPackages/gufo-v0.1.1/bin/gufo, built from source
-# against the host ROCm by scripts/build-gufo.sh. It replaced the stock
-# llama.cpp prebuilt (unsloth b11030) on 2026-09-28; port, alias and client
-# contract did not move.
+# The engine is driven by `podman run` DIRECTLY (the fleet's frank-primary
+# pattern): the image entrypoint `halogen-docker` in default "all" mode
+# starts engine + OpenAI frontend in one container. The frontend listens on
+# 8731 inside (HALOGEN_API_PORT default) and is published as host 7770; the
+# engine's own token port (8730, no auth) stays unpublished on the
+# container's loopback. The image ships NO default checkpoint:
+# HALOGEN_CHECKPOINT selects the file inside the /models mount.
 #
-# Payload: ajgazin/Swift-1.5-Qwen3.8-27B-Uncensored-Dynamic-MTP-GGUF's
-# UD-Q8_K_XL, fetched by scripts/download-frank-large-swift.sh. It replaced
-# unsloth/Qwen3.8-27B-GGUF:UD-Q8_K_XL on 2026-09-28. The two are structurally
-# interchangeable for this engine — same `qwen35` architecture, the same 866
-# tensors in the same order, the same per-tensor Q8_0/F32/BF16 layout, the
-# same MTP head at blk.64 — so the swap needed no engine change. The fine-tune
-# is abliterated (uncensored) and carries the SWIFT OPEN LICENSE 1.0, which
-# is not an OSI licence.
+# Gates mirror the deployer EXACTLY (the 2026-10-01 crash-loop lesson —
+# serve gate and deploy gate disagreeing = green deploy, red unit):
+# tokenizer directory present with every named file a FLAT REGULAR FILE —
+# the model card ships `tokenizer/` as a flat directory and the engine
+# defaults HALOGEN_TOKENIZER to /tokenizer, so a HF cache snapshot's
+# symlinks into blobs/ dangle inside the mount; GPU device nodes present.
 #
-# Shape: one session, full 262144 context (the model's native window). The
-# 27B dense weights (29.3 GiB) + the DFlash2 drafter (1.07 GiB) + KV +
-# compute buffers fit the iGPU's GTT aperture (~114 GiB); a load at this
-# context measured 66.5 GiB of the 116.7 GiB device.
-#
-# Spec method: DFlash2 (a block-diffusion drafter), gufo's adaptive block
-# policy, capped at DRAFT_TOKENS proposals per step. The drafter is the
-# z-lab Qwen3.8-27B-DFlash2-Q4_K_M and it has a sliding window of 2048
-# (small draft KV). It is UNCHANGED by the model swap and still lives beside
-# the previous target, which is why DRAFTER_DIR is its own tunable.
-#
-# Sampling: greedy (--temperature 0) server-side, as every frank-large has
-# served. The harness overrides it per request via extraBody (1.0 / 0.95 /
-# top-k 20).
-#
-# Template: gufo compiles the official Qwen Jinja into the binary and takes
-# NO --chat-template-file — and it is not a Jinja engine at all
-# (src/models/qwen/chat_template.cpp is a hand-written C++ renderer), so the
-# fleet's scripts/qwen-sharp-v22.4.0.jinja cannot be supplied to it. That law
-# was given up when this endpoint moved to gufo on 2026-09-28, by decision:
-# the terse system-prompt injection, the in-band <|think_*|> control, the
-# json|xml tool_call_format switch and the tool-argument truncation are all
-# absent. gufo's own renderer keeps the effort instruction strings and
-# preserve_thinking, which are the parts the harness depends on. Accepted
-# 2026-09-28; recovering the terse prompt from the client side is the
-# cheapest partial restore.
-#
-# Text-only: gufo auto-discovers a mmproj beside the model and this payload
-# has none (the base repo ships one separately, unused), so the endpoint
-# stays text-only.
-#
-# Tunables come from frank-large.env unless FRANK_LARGE_ENV_FILE overrides
-# it; anything not set there falls through to the defaults below.
+# Why containers instead of the bare binary (the old frank-large shape):
+# the payload is .hgn and only halogen reads it; halogen ships as a
+# container; the root unit keeps LimitMEMLOCK=infinity for parity and to
+# outlive the login session, exactly like frank-primary's.
 set -euo pipefail

-SCRIPT_DIR="$(cd -- "$(dirname -- "${BASH_SOURCE[0]}")" && pwd)"
+SCRIPT_DIR="$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)"
 ENV_FILE="${FRANK_LARGE_ENV_FILE:-$SCRIPT_DIR/frank-large.env}"
-if [[ -f "$ENV_FILE" ]]; then
-    set -a
-    source "$ENV_FILE"
-    set +a
-fi
-
-# --- defaults (used when the env file does not set them) ------------------
-: "${RUNTIME_DIR:=/home/dank/GitPackages/gufo-v0.1.1}"
-: "${MODEL_DIR:=/home/dank/models/Swift-1.5-Qwen3.8-27B-Uncensored-MTP-UD-Q8_K_XL}"
-: "${ALIAS:=frank-large}"
-: "${HOST:=0.0.0.0}"
-: "${PORT:=7770}"
-: "${CONTEXT_SIZE:=262144}"
-: "${SESSIONS:=1}"
-: "${DRAFT_TOKENS:=7}"
-: "${DRAFT_POLICY:=adaptive}"
-
-GUFO_BIN="$RUNTIME_DIR/bin/gufo"
-
-# The target GGUF and the drafter are named explicitly because they do not
-# share a directory any more: the target is a different repository's artifact
-# and keeps its own long filename, while the DFlash2 drafter is unchanged and
-# still lives beside the previous target. MODEL_FILE/DRAFTER_FILE default to
-# the deployed payload, so the env file only has to state them when it moves.
-: "${MODEL_FILE:=$MODEL_DIR/Swift-1.5-Qwen3.8-27B-Uncensored-Dynamic-MTP-UD-Q8_K_XL.gguf}"
-: "${DRAFTER_DIR:=/home/dank/models/Qwen3.8-27B-UD-Q8_K_XL}"
-: "${DRAFTER_FILE:=$DRAFTER_DIR/Qwen3.8-27B-DFlash2-Q4_K_M.gguf}"
-
-if [[ ! -x "$GUFO_BIN" ]]; then
-    echo "runtime not found at $GUFO_BIN" >&2
-    echo "frank-large: build it: bash scripts/build-gufo.sh" >&2
+if [ ! -f "$ENV_FILE" ]; then
+    echo "frank-large: env file not found: $ENV_FILE" >&2
    exit 1
 fi
-for f in "$MODEL_FILE" "$DRAFTER_FILE"; do
-    if [[ ! -f "$f" ]]; then
-        echo "missing asset $f (run the matching scripts/download-frank-large*.sh)" >&2
+set -a
+# shellcheck disable=SC1090
+source "$ENV_FILE"
+set +a
+
+: "${FRANK_LARGE_IMAGE:?FRANK_LARGE_IMAGE must name the digest-pinned image}"
+: "${FRANK_LARGE_CONTAINER:?FRANK_LARGE_CONTAINER must name the container}"
+: "${FRANK_LARGE_PUBLISH:?FRANK_LARGE_PUBLISH must be host:container}"
+: "${FRANK_LARGE_MODEL_DIR:?FRANK_LARGE_MODEL_DIR must name the payload dir}"
+: "${FRANK_LARGE_CHECKPOINT:?FRANK_LARGE_CHECKPOINT must name the .hgn inside it}"
+: "${FRANK_LARGE_CHECKPOINT_BYTES:?FRANK_LARGE_CHECKPOINT_BYTES must pin the size}"
+: "${FRANK_LARGE_TOKENIZER_FILES:?FRANK_LARGE_TOKENIZER_FILES must list the tokenizer dir contents}"
+: "${HALOGEN_MODEL_ID:?HALOGEN_MODEL_ID must be the served alias (frank-large)}"
+: "${HALOGEN_SLOT_CTX:?HALOGEN_SLOT_CTX must be set}"
+: "${HALOGEN_KV_SLOTS:?HALOGEN_KV_SLOTS must be set (1 keeps the drafter on)}"
+: "${HALOGEN_DRAFTER:?HALOGEN_DRAFTER must be set (2 = DFlash2)}"
+
+# Capture podman's output BEFORE grepping: under `set -o pipefail` the
+# `podman images | grep -q` pattern races (grep exits on first match,
+# podman dies SIGPIPE 141, pipefail turns the 141 into "no match" — the
+# 2026-09-30 deploy refusal).
+DIGESTS="$(podman images --format '{{.RepoDigests}}')"
+if ! grep -qF "${FRANK_LARGE_IMAGE#*@}" <<<"$DIGESTS"; then
+    echo "frank-large: image $FRANK_LARGE_IMAGE not in podman storage;" >&2
+    echo "  pull it: podman pull ${FRANK_LARGE_IMAGE%@*}" >&2
+    exit 1
+fi
+
+# The checkpoint gate is size-exact: a truncated curl exits 0 on some
+# failures and a partial .hgn loads far enough to look alive, so the byte
+# count from the model card is the contract (the fleet fetch law).
+CKPT="$FRANK_LARGE_MODEL_DIR/$FRANK_LARGE_CHECKPOINT"
+if [ ! -f "$CKPT" ]; then
+    echo "frank-large: checkpoint $CKPT missing;" >&2
+    echo "  fetch it: bash scripts/download-frank-large-halogen.sh" >&2
+    exit 1
+fi
+CKPT_SIZE="$(stat -c %s "$CKPT")"
+if [ "$CKPT_SIZE" != "$FRANK_LARGE_CHECKPOINT_BYTES" ]; then
+    echo "frank-large: checkpoint is $CKPT_SIZE bytes, expected $FRANK_LARGE_CHECKPOINT_BYTES (partial download?)" >&2
+    exit 1
+fi
+
+# The payload ships the tokenizer as a FLAT DIRECTORY (model card:
+# "tokenizer/ flat tokenizer directory"). A HuggingFace cache snapshot
+# symlinks each file into a sibling blobs/, and those symlinks dangle
+# inside the container mount — so the directory must be real and every
+# named file must be a regular file, not a link.
+TOK_DIR="$FRANK_LARGE_MODEL_DIR/tokenizer"
+if [ -L "$TOK_DIR" ] || [ ! -d "$TOK_DIR" ]; then
+    echo "frank-large: tokenizer dir $TOK_DIR missing or a symlink (HF cache snapshot?);" >&2
+    echo "  fetch it: bash scripts/download-frank-large-halogen.sh" >&2
+    exit 1
+fi
+for TF in $FRANK_LARGE_TOKENIZER_FILES; do
+    if [ ! -f "$TOK_DIR/$TF" ] || [ -L "$TOK_DIR/$TF" ]; then
+        echo "frank-large: tokenizer file $TOK_DIR/$TF missing or a symlink;" >&2
+        echo "  fetch it: bash scripts/download-frank-large-halogen.sh" >&2
        exit 1
    fi
 done

-# --- env -------------------------------------------------------------------
-# gufo links against the host ROCm through its RPATH (/opt/rocm ->
-# /etc/alternatives/rocm), so no LD_LIBRARY_PATH is needed. The device is
-# already gfx1151 — `gufo diagnose` reports it natively — so no
-# HSA_OVERRIDE_GFX_VERSION either; the override existed for the prebuilt
-# llama.cpp runtime and can only misreport the arch here.
-export HIP_VISIBLE_DEVICES=0
+# GPU access is mandatory but the node paths are variables so the seam can
+# drive the gate on a box without an AMD GPU.
+DEV_KFD="${FRANK_LARGE_DEV_KFD:-/dev/kfd}"
+DEV_DRI="${FRANK_LARGE_DEV_DRI:-/dev/dri}"
+if [ ! -e "$DEV_KFD" ] || [ ! -d "$DEV_DRI" ]; then
+    echo "frank-large: $DEV_KFD or $DEV_DRI missing — no gfx1151 to serve on" >&2
+    exit 1
+fi
+for CTR in ${FRANK_LARGE_LEGACY_CONTAINERS:-} "$FRANK_LARGE_CONTAINER"; do
+    podman rm -f "$CTR" >/dev/null 2>&1 || true
+done

-# --- the serve command ---------------------------------------------------
-# Server options precede the subcommand, model options follow it.
-exec "$GUFO_BIN" serve \
-    --host "$HOST" \
-    --port "$PORT" \
-    --sessions "$SESSIONS" \
-    llm \
-    --model "$MODEL_FILE" \
-    --served-model-name "$ALIAS" \
-    --context "$CONTEXT_SIZE" \
-    --speculative dflash2 \
-    --dflash-model "$DRAFTER_FILE" \
-    --draft-tokens "$DRAFT_TOKENS" \
-    --draft-policy "$DRAFT_POLICY" \
-    --temperature 0 \
-    --think auto \
-    --preserve-thinking on
+LOG="${FRANK_LARGE_LOG:-/home/dank/logs/frank_large.log}"
+mkdir -p "$(dirname "$LOG")"
+# for the udev-granted supplementary group, seccomp unconfined for the ROCm
+# ioctl surface, and unbounded memlock (the unit's LimitMEMLOCK=infinity is
+# inherited, so runc never has to RAISE it). NO --ulimit nofile: Dankclaw's
+# hard limit is 524288, rootless runc cannot raise a hard limit and dies
+# with "error setting rlimit type 7: operation not permitted" (measured
+# 2026-10-05 cutover), and 524288 fds is already ~100x what the engine
+# wants; the model card and both sibling serve scripts pass no nofile
+# either. No comment lines inside the backslash
+# chain — a mid-chain # terminates the continuation (the 2026-10-05
+# exit-127 crash loop on the sibling serve script).
+podman run -d --name "$FRANK_LARGE_CONTAINER" \
+  --publish "$FRANK_LARGE_PUBLISH" \
+  --ipc=host \
+  --ulimit memlock=-1:-1 \
+  --device "$DEV_KFD" --device "$DEV_DRI" \
+  --group-add keep-groups \
+  --security-opt seccomp=unconfined \
+  -e HALOGEN_MODEL_ID="$HALOGEN_MODEL_ID" \
+  -e HALOGEN_CHECKPOINT="/models/$FRANK_LARGE_CHECKPOINT" \
+  -e HALOGEN_SLOT_CTX="$HALOGEN_SLOT_CTX" \
+  -e HALOGEN_KV_SLOTS="${HALOGEN_KV_SLOTS:-1}" \
+  -e HALOGEN_DRAFTER="$HALOGEN_DRAFTER" \
+  -e HALOGEN_MAX_TOKENS_CAP="${HALOGEN_MAX_TOKENS_CAP:-65536}" \
+  -e HALOGEN_QUEUE_TIMEOUT="${HALOGEN_QUEUE_TIMEOUT:-7200}" \
+  ${HALOGEN_CACHE_MB:+-e HALOGEN_CACHE_MB="$HALOGEN_CACHE_MB"} \
+  -v "$FRANK_LARGE_MODEL_DIR:/models:ro" \
+  -v "$FRANK_LARGE_MODEL_DIR/tokenizer:/tokenizer:ro" \
+  "$FRANK_LARGE_IMAGE" \
+  > "$LOG" 2>&1
+
+# This script's lifetime IS the container's: podman wait returns the
+# container's exit code, so a crash is a nonzero ExecStart and the unit's
+# Restart=always is real. systemd stop goes through the unit's ExecStop
+# (podman wait alone cannot stop the child).
+exec podman wait "$FRANK_LARGE_CONTAINER"
@@ -1,6 +1,6 @@
 #!/usr/bin/env bash
 # serve-frank-primary.sh — frank-primary endpoint (tcclaviger RDNA4 vLLM fork,
-# orcarouter Qwen3.8-Flash-Next-Uncensored-FP8 + MTP4 spec decode, 2x R9700,
+# orcarouter Qwen3.8-Flash-Next-Uncensored-FP8, 2x R9700,
 # dankserver :7770; 2026-09-29 cutover from the R9V radiance fork — git tag
 # frank-primary-r9v-q8-20260929 is the pre-cutover state).
 #
@@ -11,12 +11,43 @@
 # every flag here was qualified by boot probe on 2026-09-29 — see
 # frank-primary.env for the qualification record).
 #
-# Shape: MAXLEN 200000 x 3 sessions (600k total), KV q8/q8 (--kv-cache-dtype
-# fp8: e4m3 for both K and V — the stock flag quantizes them together, which
-# is the requested q8/q8), KV_BYTES 7.2 GiB PER RANK (the engine measured
-# 729,545 tokens of capacity / 3.65x concurrency at 200k per request), MTP
-# depth 4, prefix caching, expert prefetch OFF (telemetry measured it at 36%
-# precision — the tuned incumbent deletes it), TunableOp off.
+# Shape (2026-10-03 shape F, on the 2026-10-02 shape E base): MAXLEN 200704 x 3
+# slots, fp8 KV, KV_BYTES
+# 3,150,000,000 PER RANK — the POOL, not the slot count, caps resident
+# conversations at ~2 full-length (measured: 398,658 tokens = 1.99x);
+# shorter requests use the spare blocks. --max-num-seqs stays 3 because SEQS=2
+# CANNOT boot this model
+# (torch.fx ConstraintViolationError on query_start_loc, measured at both
+# MNBT 1024 and 2048 — the 2x2 lives in frank-primary.env). Speculative decode
+# is OFF today by ENV KNOB, not by measurement: FRANK_PRIMARY_SPEC / _COMPCONFIG
+# / _GEN_CONFIG hold the whole JSON value and unset means no flag. The 2026-10-02
+# retirement of MTP compared the wrong pair — see frank-primary.env.
+# --max-num-batched-tokens 2048 (Frank's call 2026-10-03; arena-free because the
+# registry prices overhead flat up to its nbt 4096 anchor), MoE gather geometry
+# R4D_LRU_GATHER_CHUNKS=32 / R4D_LRU_GATHER_LANES=64 (+18.7% decode, grows
+# with depth), prefix caching OFF 2026-10-05 (the fork's QFN align-mode GDN
+# state fault — the evidence block is in frank-primary.env), expert prefetch
+# OFF (36% precision measured),
+# TunableOp off. Shape E adds --expert-precise-memory (fitted 5.25 GiB overhead
+# instead of the 7.60 GiB default -> expert arena 17.22 GiB / 7,522 slots, +16%)
+# and GPU_MAX_HW_QUEUES=2; shape F pins the plan's CUDA-graph term to the
+# measured capture cost (VLLM_CUDAGRAPH_MEM_BASE_MB / _PER_GRAPH_MB, container
+# env — the analytic default charges 0.55 GiB where capture took 0.31 GiB, so
+# the 0.24 GiB difference goes to the arena) and raises util 0.98 -> 0.982,
+# Frank's cap. The native-262144 and fp8+MTP4 shapes are retired to
+# git tag frank-primary-fp8-mtp-20261001.
+#
+# THE COMPILE CACHE IS MOUNTED, and that is a boot-cost fix, not a tuning knob:
+# the image ships no /root/.cache at all, our first boot writes 1.1 GB of vLLM
+# AOT/torch_compile artifacts plus 229 MB of comgr JIT kernels into the
+# container's ephemeral layer, and `podman rm -f` deletes them — so every
+# deploy/restart/reboot re-pays `torch.compile took 181.20 s in total` (173.19 s
+# of it one graph). Mounting FRANK_PRIMARY_CACHE_DIR at /root/.cache makes all
+# later boots load the compiled artifact instead, and hides nothing. PROVEN
+# LIVE 2026-10-02 on consecutive boots of this shape: cold `torch.compile took
+# 188.26 s`, warm `Directly load the compiled graph(s) for compile range
+# (1, 1024) from the cache, took 2.143 s` / `torch.compile took 3.97 s in
+# total` / `init engine … took 41.13 s (compilation: 3.98 s)`.
 #
 # The root unit (scripts/frank-primary.service, User=dank,
 # LimitMEMLOCK=infinity) grants the pin budget for the tiered-expert host
@@ -45,12 +76,16 @@ set +a
 : "${FRANK_PRIMARY_MAXLEN:?FRANK_PRIMARY_MAXLEN must be set}"
 : "${FRANK_PRIMARY_SEQS:?FRANK_PRIMARY_SEQS must be set}"
 : "${FRANK_PRIMARY_KV_BYTES:?FRANK_PRIMARY_KV_BYTES must be set (PER RANK)}"
-: "${FRANK_PRIMARY_KV_DTYPE:?FRANK_PRIMARY_KV_DTYPE must be set (fp8 = q8/q8 KV)}"
-: "${FRANK_PRIMARY_SPEC:?FRANK_PRIMARY_SPEC must be set (0 disables)}"
-: "${FRANK_PRIMARY_UTIL:?FRANK_PRIMARY_UTIL must be set}"
-
-# --- gates: never serve a half-finished payload or a missing image -----------
-if ! podman images --format '{{.RepoDigests}}' | grep -qF "${FRANK_PRIMARY_IMAGE#*@}"; then
+: "${FRANK_PRIMARY_KV_DTYPE:?FRANK_PRIMARY_KV_DTYPE must be set (auto = bf16 KV)}"
+: "${FRANK_PRIMARY_REASONING_PARSER:?FRANK_PRIMARY_REASONING_PARSER must be set (qwen3)}"
+: "${FRANK_PRIMARY_TOOL_PARSER:?FRANK_PRIMARY_TOOL_PARSER must be set (qwen3_xml)}"
+: "${FRANK_PRIMARY_CHAT_TEMPLATE_KWARGS:?FRANK_PRIMARY_CHAT_TEMPLATE_KWARGS must be set}"
+# Capture podman's output BEFORE grepping: under `set -o pipefail`, the
+# classic `podman images | grep -q` pattern races — grep exits on its first
+# match, podman dies SIGPIPE (141) writing the remaining lines, and pipefail
+# turns the 141 into "no match" (the 2026-09-30 deploy refusal).
+DIGESTS="$(podman images --format '{{.RepoDigests}}')"
+if ! grep -qF "${FRANK_PRIMARY_IMAGE#*@}" <<<"$DIGESTS"; then
    echo "serve-frank-primary: pinned image absent from rootless storage: $FRANK_PRIMARY_IMAGE" >&2
    echo "  fetch it: podman pull $FRANK_PRIMARY_IMAGE" >&2
    exit 1
@@ -88,28 +123,69 @@ done
 LOG=/home/dank/logs/frank_primary.log
 mkdir -p "$(dirname "$LOG")"

+# Persistent compile cache (see the header). Create it before the mount so even
+# the first, cold boot has a target; unset -> no mount, exactly the old
+# ephemeral behaviour. It carries /root/.cache/vllm (AOT + torch_compile) and
+# /root/.cache/comgr (HIP kernel JIT).
+CACHE_DIR="${FRANK_PRIMARY_CACHE_DIR:-}"
+if [ -n "$CACHE_DIR" ]; then
+    mkdir -p "$CACHE_DIR"
+fi
+
 # --device /dev/dri is the WHOLE DIRECTORY (individual renderD nodes enumerate
 # without the card nodes and HIP refuses contexts — "No CUDA GPUs are
 # available" while device_count() says 2). --ipc=host for the engine's shared
 # memory; memlock unbounded for the pinned host slabs.
 # The entrypoint already execs `vllm serve "$@"`, so no `serve` here.
+# Prefix caching: ON — the shipped contract. FRANK_PRIMARY_NO_PREFIX_-
+# CACHING=1 flips to the fork-named mitigation (--no-enable-prefix-caching)
+# for the QFN align-mode GDN state fault; it was tried 2026-10-05 and
+# REVERTED the same day (live conversations still degraded at ~40k with
+# caching off and zero faults — see the evidence block in
+# frank-primary.env). NO comment lines inside the backslash chain below —
+# a comment there terminates the continuation and the next line runs as
+# its own command (the 2026-10-05 exit-127 crash loop).
+# The ${VAR:-word} / ${VAR:+word} double-expansion for the two directions is
+# AMBIGUOUS bash (${VAR:--flag} parses as a substring offset when set —
+# measured 2026-10-05: rendered "--no-enable-prefix-caching1"), so the flag
+# is branched explicitly: exactly one of the two reaches the engine.
+if [ -n "${FRANK_PRIMARY_NO_PREFIX_CACHING:-}" ]; then
+    FRANK_PRIMARY_PREFIX_CACHE_FLAG=--no-enable-prefix-caching
+else
+    FRANK_PRIMARY_PREFIX_CACHE_FLAG=--enable-prefix-caching
+fi
 podman run -d --name "$FRANK_PRIMARY_CONTAINER" \
  --publish "$FRANK_PRIMARY_PUBLISH" \
  --ipc=host --ulimit memlock=-1:-1 \
  --cap-add SYS_PTRACE \
  --device /dev/kfd --device /dev/dri --device /dev/mem \
  -e HIP_VISIBLE_DEVICES=0,1 -e VLLM_ROCM_USE_AITER=0 \
+  ${FRANK_PRIMARY_GPU_MAX_HW_QUEUES:+-e GPU_MAX_HW_QUEUES="$FRANK_PRIMARY_GPU_MAX_HW_QUEUES"} \
+  ${FRANK_PRIMARY_GATHER_CHUNKS:+-e R4D_LRU_GATHER_CHUNKS="$FRANK_PRIMARY_GATHER_CHUNKS"} \
+  ${FRANK_PRIMARY_GATHER_LANES:+-e R4D_LRU_GATHER_LANES="$FRANK_PRIMARY_GATHER_LANES"} \
+  ${FRANK_PRIMARY_CUDAGRAPH_BASE_MB:+-e VLLM_CUDAGRAPH_MEM_BASE_MB="$FRANK_PRIMARY_CUDAGRAPH_BASE_MB"} \
+  ${FRANK_PRIMARY_CUDAGRAPH_PER_GRAPH_MB:+-e VLLM_CUDAGRAPH_MEM_PER_GRAPH_MB="$FRANK_PRIMARY_CUDAGRAPH_PER_GRAPH_MB"} \
+  ${CACHE_DIR:+-v "$CACHE_DIR":/root/.cache} \
  -v "$FRANK_PRIMARY_MODEL_DIR":/models:ro \
  "$FRANK_PRIMARY_IMAGE" /models \
  --served-model-name "$FRANK_PRIMARY_SERVED_NAME" --port 7770 \
  --tensor-parallel-size 2 \
+  ${FRANK_PRIMARY_DCP:+--decode-context-parallel-size "$FRANK_PRIMARY_DCP"} \
  --max-model-len "$FRANK_PRIMARY_MAXLEN" \
  --max-num-seqs "$FRANK_PRIMARY_SEQS" \
+  ${FRANK_PRIMARY_MNBT:+--max-num-batched-tokens "$FRANK_PRIMARY_MNBT"} \
+  ${FRANK_PRIMARY_PRECISE_MEMORY:+--expert-precise-memory} \
+  ${FRANK_PRIMARY_EXPERT_OFFLOAD_MEM:+--expert-offload-mem "$FRANK_PRIMARY_EXPERT_OFFLOAD_MEM"} \
  --gpu-memory-utilization "$FRANK_PRIMARY_UTIL" \
  --kv-cache-memory-bytes "$FRANK_PRIMARY_KV_BYTES" \
  --kv-cache-dtype "$FRANK_PRIMARY_KV_DTYPE" \
-  --enable-prefix-caching \
-  --speculative-config "{\"method\":\"mtp\",\"num_speculative_tokens\":$FRANK_PRIMARY_SPEC}" \
+  "$FRANK_PRIMARY_PREFIX_CACHE_FLAG" \
+  --enable-auto-tool-choice --tool-call-parser "$FRANK_PRIMARY_TOOL_PARSER" \
+  --reasoning-parser "$FRANK_PRIMARY_REASONING_PARSER" \
+  --default-chat-template-kwargs "$FRANK_PRIMARY_CHAT_TEMPLATE_KWARGS" \
+  ${FRANK_PRIMARY_SPEC:+--speculative-config "$FRANK_PRIMARY_SPEC"} \
+  ${FRANK_PRIMARY_COMPCONFIG:+--compilation-config "$FRANK_PRIMARY_COMPCONFIG"} \
+  ${FRANK_PRIMARY_GEN_CONFIG:+--override-generation-config "$FRANK_PRIMARY_GEN_CONFIG"} \
  --trust-remote-code \
  > "$LOG" 2>&1

@@ -1,39 +1,25 @@
 #!/usr/bin/env bash
-# Serve HauhauCS/Gemma4-12B-QAT-Uncensored-HauhauCS-Balanced (Q4_K_M) as
-# frank-small on Dankclaw's RTX 3090 via mainline llama.cpp.
+# Serve ornith-ai/Ornith-1.5-9B-GGUF (Q8_0) as frank-small on
+# Dankclaw's RTX 3090 via llama.cpp (master build, CUDA sm_86).
 #
-# Topology: the whole model lives in VRAM (-ngl 999) -- 6.87 GiB of Q4_K_M
-# weights plus the BF16 vision projector and q8_0 KV inside the card's
-# 24 GiB. The host CPU only tokenizes and samples here; this box's CPU
-# belongs to the halogen engine that serves frank-large.
+# 2026-10-01 night (Frank): swapped from SGLang + INT8-W8A16-AutoRound
+# safetensors to the llama.cpp GGUF path, because llama.cpp is the only
+# runtime on this host that moves K and V independently — the asymmetric
+# f16-K + q8_0-V contract that endpoint was built for. 2026-10-03 (Frank):
+# the pair is now f16/f16 — two wide slots freed the VRAM, so the values are
+# unquantized too. The per-tensor flags stay explicit either way.
 #
-# MTP: this model's Multi-Token Prediction head ships as a SEPARATE file
-# (mtp-gemma-4-12B-it.gguf, arch gemma4-assistant, 4 nextn layers). It is
-# passed via --model-draft with --spec-type draft-mtp: the draft context
-# loads the assistant model. The same --spec-type value chains the
-# draftless ngram-mod matcher, which attaches unconditionally; two
-# context-backed drafters cannot share one process, so this chain is the
-# only shape that carries both.
+# Architecture: qwen3_5 hybrid — 32 layers, 8 full-attention + 24 Gated
+# DeltaNet, native 262144 window. KV lives only in the 8 attention layers;
+# the DeltaNet state is per-slot fixed size. Vision comes from the repo's
+# separate projector (the Q6_K GGUF carries no `v.*` tensors at all), so
+# --mmproj is what gives this endpoint eyes; MMPROJ_FILE="" is the documented
+# text-only opt-out.
 #
-# Context: CTX_SIZE is set explicitly to PARALLEL x 262144 so every slot
-# gets the model's FULL window; --fit (the CTX_SIZE-unset branch below)
-# maximises the TOTAL pool instead, which would divide a smaller window
-# across slots. The 4-slot layout is measured to fit the card (see the
-# SIZING note in frank-small.env); more slots would fail the KV allocation.
+# Sampling: llama.cpp takes it per request (the fleet harness sends its
+# recipe from models.yml); greedy is the harness default.
 #
-# Vision + audio: --mmproj loads the BF16 projector, which also carries
-# an audio encoder — input_audio content parts transcribe verbatim
-# (verified 2026-09-17). The fleet's image path lives here, which is why
-# halogen no longer sets HALOGEN_VISION_TOWER.
-#
-# Sampling: greedy by default (the fleet contract). Clients that send their own
-# temperature get their own values, and the harness overrides per model from
-# models.yml. --jinja is pinned even though this build defaults it on: the
-# model's own template supplies reasoning_effort, preserve_thinking and the
-# tool-call grammar, and a future build that flips that default must not
-# silently take tools away.
-#
-# Tunables come from frank-small.env unless FRANK_SMALL_ENV_FILE overrides it.
+# Tunables come from frank-small.env unless FRANK_SMALL_ENV_FILE overrides.
 set -euo pipefail

 SCRIPT_DIR="$(cd -- "$(dirname -- "${BASH_SOURCE[0]}")" && pwd)"
@@ -46,101 +32,68 @@ if [[ -f "$ENV_FILE" ]]; then
 fi

 # --- defaults (used when the env file does not set them) -------------------
-: "${LLAMA_BIN:=/home/dank/GitPackages/llama.cpp-mainline/build-cuda/bin/llama-server}"
-: "${MODEL_DIR:=/home/dank/models/Gemma4-12B-QAT}"
-: "${MODEL_FILE:=Gemma4-12B-QAT-Uncensored-HauhauCS-Balanced-Q4_K_M.gguf}"
-: "${MMPROJ_PATH:=$MODEL_DIR/mmproj-Gemma4-12B-QAT-Uncensored-HauhauCS-Balanced-BF16.gguf}"
-# NOTE: the fallback above was the retired ISTA-DASLab Qwen3.8-27B GSQ-RCO
-# payload, which frank-small served on 2026-09-16/17 and which no longer
-# exists on disk. frank-small.env has always overridden it, so the drift was
-# invisible — but a missing or unreadable env file would have left this
-# script pointing at a payload that is not there, turning a config problem
-# into a "missing asset" that names the wrong model.
-: "${DRAFT_FILE:=}"
+: "${LLAMA_BIN:=/home/dank/GitPackages/llama-master/build-cuda/bin/llama-server}"
+: "${MODEL_FILE:=/home/dank/models/Ornith-Q8/Ornith-1.5-9B-Q8_0.gguf}"
 : "${ALIAS:=frank-small}"
 : "${BIND_ADDR:=0.0.0.0}"
 : "${LISTEN_PORT:=7771}"
-: "${CTX_SIZE:=}"
-: "${PARALLEL:=1}"
+# ctx-size is the POOL llama.cpp divides across slots: 401408 / 2 gives
+# 200,704 per slot — two independent conversations, each one wide enough to
+# hold a whole frank-primary-sized prompt in a single shot (the native 262,144
+# no longer fits twice on a 24 GiB card once the vision tower is loaded;
+# frank-small.env carries the three measurements that set this number).
+: "${CTX_SIZE:=401408}"
+: "${PARALLEL:=2}"
+# Compute-buffer geometry. Before 2026-10-03 this endpoint never passed either
+# flag, so it ran on llama.cpp's own defaults (2048/512). The pair is 4096/2048
+# since 2026-10-05: on this CUDA card the prefill curve climbs all the way to
+# ubatch 2048 (unlike the R9700 layer-split sibling, where 512 is the peak),
+# and the logical batch is VRAM- and throughput-neutral. frank-small.env
+# carries the sweep tables and the VRAM risk note.
+: "${BATCH_SIZE:=4096}"
+: "${UBATCH_SIZE:=2048}"
 : "${CACHE_TYPE_K:=f16}"
 : "${CACHE_TYPE_V:=f16}"
-: "${CACHE_RAM:=8192}"
-: "${BATCH_SIZE:=2048}"
-: "${UBATCH_SIZE:=512}"
-# Decode-shape knobs: the MTP drafter's per-round proposal length (build
-# default 3; the Gemma4 sidecar carries 4 nextn layers) and the engine's
-# busy-wait level (build default 50).
-: "${SPEC_DRAFT_N_MAX:=3}"
-: "${POLL_LEVEL:=50}"
-: "${NUM_THREADS:=8}"
-: "${THREAD_BATCH:=8}"
-: "${NGL:=999}"
-: "${FLASH_ATTN:=on}"
-: "${SPEC_TYPE:=draft-mtp}"
-: "${TEMPERATURE:=0.0}"
-: "${REASONING_FORMAT:=deepseek}"
-: "${REASONING_BUDGET:=-1}"
+: "${NGL:=99}"
+# Vision projector. No colon on purpose: an explicitly empty value means
+# "text-only, on purpose" and must NOT be replaced by the default path.
+: "${MMPROJ_FILE=/home/dank/models/Ornith-Q8/mmproj-Ornith-1.5-9B-BF16.gguf}"

-# Fail loudly rather than half-loading a model into VRAM: a missing file here
-# is a download that has not finished, and the unit's restart policy would
-# otherwise turn that into a crash loop that reads like a runtime fault.
-for path in "$LLAMA_BIN" "$MODEL_DIR/$MODEL_FILE" "$MMPROJ_PATH"; do
+# Fail loudly rather than half-loading: a missing file is a download that
+# has not finished, and Restart=always would turn it into a crash loop that
+# reads like a runtime fault.
+for path in "$LLAMA_BIN" "$MODEL_FILE"; do
    [[ -e "$path" ]] || { echo "frank-small: missing $path" >&2; exit 1; }
 done
-if [[ -n "${DRAFT_FILE:-}" ]]; then
-    [[ -e "$MODEL_DIR/$DRAFT_FILE" ]] || { echo "frank-small: missing $MODEL_DIR/$DRAFT_FILE" >&2; exit 1; }
+EXTRA_ARGS=()
+if [[ -n "$MMPROJ_FILE" ]]; then
+    [[ -e "$MMPROJ_FILE" ]] || { echo "frank-small: missing $MMPROJ_FILE" >&2; exit 1; }
+    EXTRA_ARGS+=(--mmproj "$MMPROJ_FILE")
 fi

 # The UVM device node is created lazily by nvidia-modprobe on the first CUDA
 # open after boot; make sure it exists before the server needs it.
 nvidia-modprobe -u -c=0 2>/dev/null || true

-MMPROJ_ARGS=()
-if [[ -n "${MMPROJ_PATH:-}" && -f "$MMPROJ_PATH" ]]; then
-    MMPROJ_ARGS=(--mmproj "$MMPROJ_PATH")
-fi
-
-SPEC_ARGS=()
-if [[ -n "${SPEC_TYPE:-}" && "$SPEC_TYPE" != "none" ]]; then
-    SPEC_ARGS=(--spec-type "$SPEC_TYPE" --spec-draft-n-max "$SPEC_DRAFT_N_MAX")
-fi
-
-DRAFT_ARGS=()
-if [[ -n "${DRAFT_FILE:-}" ]]; then
-    DRAFT_ARGS=(--model-draft "$MODEL_DIR/$DRAFT_FILE")
-fi
-
-# CTX_SIZE set -> pinned context. Unset -> --fit sizes the KV pool against
-# whatever the weights, projector, draft context and compute buffers leave
-# on the card; the engine prints the chosen value at startup.
-if [[ -n "${CTX_SIZE:-}" ]]; then
-    CTX_ARGS=(--ctx-size "$CTX_SIZE")
-else
-    CTX_ARGS=(--fit on)
-fi
-
-exec "$LLAMA_BIN" \
-    --model "$MODEL_DIR/$MODEL_FILE" \
-    "${MMPROJ_ARGS[@]}" \
-    "${DRAFT_ARGS[@]}" \
-    --alias "$ALIAS" \
-    --host "$BIND_ADDR" \
-    --port "$LISTEN_PORT" \
-    --gpu-layers "$NGL" \
-    --flash-attn "$FLASH_ATTN" \
-    "${CTX_ARGS[@]}" \
-    --parallel "$PARALLEL" \
-    --batch-size "$BATCH_SIZE" \
-    --ubatch-size "$UBATCH_SIZE" \
-    --threads "$NUM_THREADS" \
-    --threads-batch "$THREAD_BATCH" \
-    --poll "$POLL_LEVEL" \
-    --cache-type-k "$CACHE_TYPE_K" \
-    --cache-type-v "$CACHE_TYPE_V" \
-    --cache-ram "$CACHE_RAM" \
-    "${SPEC_ARGS[@]}" \
-    --jinja \
-    --metrics \
-    --temp "$TEMPERATURE" \
-    --reasoning-format "$REASONING_FORMAT" \
-    --reasoning-budget "$REASONING_BUDGET"
+# Flash attention stays pinned on: it is the fast path for f16 KV, and it is
+# the only path that could serve quantized V at all — so if CACHE_TYPE_V is
+# ever set back to q8_0 this flag must not be lost, or the build falls back
+# silently and the KV contract in frank-small.env becomes a lie.
+ARGS=(
+    --model "$MODEL_FILE"
+    --alias "$ALIAS"
+    --host "$BIND_ADDR"
+    --port "$LISTEN_PORT"
+    --ctx-size "$CTX_SIZE"
+    --parallel "$PARALLEL"
+    --batch-size "$BATCH_SIZE"
+    --ubatch-size "$UBATCH_SIZE"
+    --n-gpu-layers "$NGL"
+    --flash-attn on
+    --cache-type-k "$CACHE_TYPE_K"
+    --cache-type-v "$CACHE_TYPE_V"
+    --jinja
+    --reasoning auto
+    --reasoning-format deepseek
+)
+exec "$LLAMA_BIN" "${ARGS[@]}" ${EXTRA_ARGS[@]+"${EXTRA_ARGS[@]}"}
@@ -20,11 +20,12 @@ user sees it; the `bats` seams assert config shape, not behavior.

 | Path | Role |
 |------|------|
-| `playwright.config.ts` | four projects (below), real-browser UA, trace on failure |
+| `playwright.config.ts` | five projects (below), real-browser UA, trace on failure |
 | `specs/00-auth.setup.ts` | setup project: two real authentik logins, saves storage states |
 | `specs/10-homepage.spec.ts` | user panel set + open links derived from `services.json` (registry-derived — no hardcoded service lists) |
 | `specs/11-admin.spec.ts` | `admin` project: admin grid + `/admin/` RBAC (403 for standard user, members-only 200) |
 | `specs/12-crawl.spec.ts` | every user open link reaches its real app through the guard chain |
+| `specs/19-frankbook.spec.ts` | `frankbook` project: walks the six admin links that terminate on FrankBook-Pro (192.168.0.112) — conditional, see below |
 | `specs/13-sso.spec.ts` | direct guarded-vhost visits resolve silently with ONLY the homepage session (single-login enforcement) |
 | `specs/14-guard.spec.ts` | `guest` project, no storage state: unauthenticated traffic lands on the IdP login |
 | `lib/login.ts` | `akLogin` helper (two-stage flow, shadow-DOM locators) |
@@ -38,7 +39,16 @@ user sees it; the `bats` seams assert config shape, not behavior.
  `.auth/qa-e2e{,-admin}.json` storage states.
 - **standard** — 10/12/13 with the user state.
 - **admin** — 11 with the admin state.
- **guest** — 14 with an empty state.
+- **guest** — 14/15 with an empty state.
+- **frankbook** — 19 with the admin state. **Excluded from every default run**: the
+  *arr/qBittorrent links terminate on FrankBook-Pro, which sleeps when idle, and a
+  sleeping Mac mini must never fail an unrelated deploy. The project joins only
+  when `E2E_INCLUDE_FRANKBOOK=1` (set by `scripts/deploy.sh` when a frankbook
+  trigger path changed: `tests/e2e/specs/19-frankbook.spec.ts`, `frankbook-pro/**`,
+  `shared/homepage/services.json`, or the `dankclaw/Caddyfile` serving the arr
+  vhosts) or when an explicit `--project` argument names it. An `11-admin` edit
+  that touches the arr links must therefore be paired with a `19-frankbook` run
+  against an awake host before it is trusted live.

 Credentials live in `tests/e2e/.env.local` (git-ignored; placeholders in
 `.env.local.example`). The scaffold bats seam fails the commit if creds or
@@ -47,8 +57,9 @@ auth states are ever tracked.
 ## Run

 ```bash
-cd tests/e2e && npx playwright test            # all projects
-npx playwright test --project=standard 12-crawl.spec.ts   # one slice
+tests/e2e/run.sh                          # default set — frankbook project excluded
+E2E_INCLUDE_FRANKBOOK=1 tests/e2e/run.sh  # default set + the frankbook walk
+cd tests/e2e && npx playwright test       # ALL projects, frankbook included
 ```

 Requires LAN reachability of `www.franksdank.cloud` (Pi-hole DNS or public).
@@ -95,6 +95,51 @@ llama-server --hf-repo <target-GGUF> \

 The DFlash drafter proposes, then ngram-mod runs as an additional implementation in the chain. Verified against upstream `common/speculative.cpp` (b10133+).

+### Chaining laws (measured on this fleet, 2026-09 → 10)
+
+`--spec-type` takes a comma list, and the gate is `ctx_dft` in
+`common/speculative.cpp` (`common_speculative_init_result`): **draftless**
+types (the ngram family — simple, map-k, map-k4v, mod, cache) are added
+unconditionally, while **context-backed** types (`draft-mtp`, `draft-dflash`,
+`draft-dspark`, `draft-eagle3`) all require a draft CONTEXT. Consequences:
+
+- A chain may hold ONE context-backed drafter plus any number of ngram types.
+  `draft-dflash,ngram-mod` is legal and measured: 2.6x the drafter alone on
+  prompt-echo traffic, byte-identical to it on prose at the fleet's 48/64
+  window.
+- Two context-backed types cannot share one process: `draft-mtp,draft-dflash`
+  dies at load with `context type MTP requested but model doesn't contain MTP
+  layers` — MTP builds its context against whatever the draft context holds,
+  so it met the DRAFTER's file, not a missing head. Pin that in a seam so
+  nobody "fixes the flag" later.
+- Internal order is fixed (ngram types first, then drafters); CLI order is
+  irrelevant.
+- **A legal chain can still lose on a recurrent model.** `draft-mtp,ngram-mod`
+  on the qwen35 hybrid (frank-large era) measured −15 % prose / −7 %
+  block-copy (2026-10-03): adding the ngram layer logs `recurrent state
+  rollback is not compatible with 'ngram-mod' - disabling rollback support`,
+  and MTP verification depends on that rollback, so acceptance drops on ALL
+  traffic even where the matcher fires. Measure chains per architecture,
+  never by analogy.
+- ngram-mod prints NO startup banner — the load log shows only the drafter's
+  `adding speculative implementation` line. Never conclude "chain not
+  engaged" from the load log; identify behaviourally (`draft_n` > 0 only
+  where text repeats) or by the per-task `draft acceptance = ... mean len`
+  line. It costs ~64 MiB of host-RAM hash table.
+
+### Drafter KV: read the window policy from the header
+
+The draft context is built with `cparams.n_ctx = llama_n_ctx(ctx_tgt)`
+(`common/speculative.cpp`) — there is no flag to cap it — so a drafter that
+declares NO sliding window caches the whole target window on whatever device
+`--spec-draft-device` names. DFlash declares `sliding_window = 2048` (~40 MiB
+of draft KV); DSpark declares none: 5 layers x 2 x 8 kv heads x 128 dim x 2 B
+= 20 KiB per position, i.e. **5.0 GiB at 262144** — enough to kill the first
+draft graph capture with `ROCm error: out of memory` AFTER the model loaded
+and `/health` answered (measured 2026-09-16). Read `sliding_window` out of
+the drafter's GGUF header before sizing any spec-decode deployment, and
+prefer a skewed `-ts` over a context cut when the full window matters.
+
 DFlash on llama.cpp:

 ```
@@ -209,6 +254,151 @@ docker run --rm --name vllm-qwen36-dflash \

 History: before `--spec-default`, the box ran DSpark (`draft-dspark`, retired 2026-08-15 after a ROCm illegal-memory-access crash under two concurrent 16k prompts) and before that bundled MTP (`draft-mtp`, retired same day). DFlash was retired on 2026-08-14 (premature-EOS under parallel load). The old DFlash draft GGUF (`Qwen3.6-27B-DFlash-Q8_0.gguf`), the MTP-only drafter (`Qwen3.6-27B-MTP-ONLY-Q8_0.gguf`), and the DSpark draft (`Qwen3.8-27B-DSpark-Q8_0.gguf`) remain in `/home/dank/models/` as reference only.

+## A/B protocol on a live llama.cpp endpoint
+
+Verdicts are **traffic-shaped**: a proposer can only draft text it has seen,
+so a single prompt shape answers the wrong question. Use fixed shapes:
+
+| Shape | Prompt | Who it can favour |
+|---|---|---|
+| prose | 1k filler + "reply with one sentence" | model-backed drafters only |
+| echo | ~1.5k passage of UNIQUE lines + "repeat verbatim" | ngram (output IS the prompt) |
+| structured | repetitive output, no prompt overlap | ngram, if spans really repeat |
+| identity | short deterministic question | controls for lossless verification |
+
+Design the echo passage so no long span repeats INSIDE it — otherwise you
+measure the matcher against itself. For `structured`, give each line a unique
+id: repeated lines make it a negative control, not a repetition test.
+
+Protocol:
+
+1. Stop the unit (`sudo systemctl stop <svc>`), `pkill -x llama-server`; wait
+   for the port (a retired predecessor's published-port proxy leaves the
+   endpoint port in TIME_WAIT — the bind fails with address-in-use long after
+   the unit stopped).
+2. Per config: write an env file with the overrides, start the serve script
+   with `FRANK_*_ENV_FILE`, poll `/health`, measure. Ready-made harness:
+   `scripts/bench-spec-ab.py <env-file> <serve-script>`.
+3. **Restore the unit in a `finally`** and shout if it does not come back
+   healthy. A crashed harness that leaves production down is worse than no
+   harness.
+4. **Prove losslessness**: one fixed prompt at temp 0 to every config, diff
+   the greedy text. Speculation only buys speed; differing texts are a
+   correctness bug. Empty answers (`content: null` when reasoning ate the
+   budget) make the comparison vacuous — bench clients must fall back to
+   `reasoning_content`, and echo/structured prompts should disable thinking
+   per request: `chat_template_kwargs: {"enable_thinking": false}`.
+5. **Sweep the depth, not just the mode** — draft depth is model-specific and
+   single-valued; deeper often loses (measured: at n=7 a third of proposals
+   survived, at n=3 more than half).
+6. **Sweep the ngram window in the same run as the configs**: too short a
+   `--spec-ngram-mod-n-min/-n-max` proposes on ordinary prose fragments —
+   rejected, verification still paid (measured 43.9 → 36.0 tok/s).
+
+Read the right counters — per request `timings.draft_n` / `draft_n_accepted` /
+`predicted_per_second`; cumulative `/metrics`
+`llamacpp:spec_decode_num_drafts_total` / `..._accepted_tokens_total`;
+per-task `draft acceptance = ... (N accepted / M generated)`. `draft_n == 0`
+while tokens flow = the proposer is NOT engaged, whatever the flags say. A
+zero-draft config is a useful negative control: it proves the other config's
+drafts came from the proposer under test.
+
+Harness hygiene: absolute script paths (a relative path plus `cd` to its own
+dir double-counts the prefix); `pkill -x`, never `-f` (the pattern matches
+your own command line); nested f-strings with backslash escapes are a
+SyntaxError on the fleet's Python (<3.12 semantics) — build log lines with
+concatenation. Report `decode tok/s` per shape plus acceptance and mean
+accepted length; a config that wins on echo and loses on prose is a traffic
+decision, not a winner. Wire the winner into the deployed env file with the
+measured table as its justification comment; record losing numbers with "do
+not switch without re-measuring".
+
+The A/B vehicle for ANY single-GPU llama.cpp knob (batch/ubatch, KV type,
+slot split, quant rung — not just spec flags) is the transient systemd
+drop-in procedure in [[frank-large-benchmark]] and [[frank-small-benchmark]].
+
+## Is it worth it? (arithmetic before deployment)
+
+Answer against the INCUMBENT, not against zero.
+
+**Step 0 — check whether the target already speculates.** Many engines ship a
+drafter of their own; then your baseline is `A_incumbent / t_s`, not `1 / t_s`.
+halogen-flash-server: MTP head ON by default — read its own accounting, e.g.
+`serve_api: mtp 63 tok in 1.31s = 47.97 t/s | 33 rounds, commit 1.88/round`.
+llama.cpp: `/metrics` counters and per-request `draft acceptance` lines.
+
+**Step 1 — measure the two step times, never guess.** Target step `t_s` = 1 /
+(client-visible tok/s ÷ commit-per-round); ask for the rate the operator
+actually SEES (engine-internal decode-only numbers run ~1.7–2x optimistic).
+Drafter step `d_s` from llama.cpp's per-task `eval time` lines when it is a
+plain model on the same box.
+
+**Step 2 — arithmetic:**
+
+```
+round_time = K * d_s + 1.1 * t_s        # K drafts proposed, K+1 verified in ONE target pass
+A          = (1 - p^(K+1)) / (1 - p)    # expected accepted tokens, p = per-position acceptance
+rate       = A / round_time             # compare to the incumbent
+break-even : A > rate_incumbent * (K*d_s + 1.1*t_s)
+```
+
+Worked example that decided a real case: `t_s = 67 ms` (25 tok/s, 1.67
+commit/round), `d_s = 12 ms` (27B IQ3_S drafter on the 3090) → break-even
+`p ≈ 0.70` at K=3, `≈0.68` at K=5; a realistic cross-finetune `p = 0.4–0.6`
+gives 17–20 tok/s, i.e. a LOSS. A drafter only ~3x faster per token cannot
+pay for K sequential forwards.
+
+**Step 3 — classify the drafter by per-round cost:**
+
+| drafter | cost per round | needs |
+|---|---|---|
+| plain model (`-md`/`draft-simple`) | **K sequential forwards** | 10–50x faster than the target |
+| n-gram / prompt-lookup (`ngram-*`) | ~0, no model | repeated context (agent edit loops) |
+| trained head (`draft-mtp`, EAGLE/DFlash) | **~1 pass** | the head must exist FOR THAT CHECKPOINT |
+
+Trained heads are target-specific: EAGLE/DFlash consume the TARGET's hidden
+states, so a dense-27B DFlash2 is useless for a 512-expert MoE trunk.
+
+**Step 4 — acceptance reality checks.** Fleet datum: a trained z-lab DFlash2
+at depth 7 accepted ~41 % on its matched target (`draft-dflash 7 → 43.5
+tok/s`); treat 40–60 % as the honest band for a matched pair. Cheapest proxy
+for `p` at temp 0 when both models are greedy and share a tokenizer: greedy
+continuation agreement on identical prompts, two API calls, no weights moved.
+llama.cpp can also synthesize the projection without any draft artifact:
+`--spec-synth-rates P0,P1,...` / `--spec-synth-len L`. Speculation pays IN
+PROPORTION TO HOW SLOW THE TARGET IS — the same 41 % gave 2.3x on an
+18.8 tok/s layer-split target and a loss on a 25 tok/s one; fast targets are
+usually sparse MoE, the worst case for drafting.
+
+**Step 5 — check the attachment surface BEFORE the arithmetic.** Closed
+engines refuse external drafters by design: halogen-flash-server's
+`DRAFTER_CAP` gates `dflash2` on the loaded checkpoint and an unavailable
+name is a deliberate 400 (a silent downgrade would make comparisons lie);
+there is no device routing in its loopback token protocol, so a drafter can
+never sit on another card. Pure llama.cpp DOES route the draft:
+`--spec-draft-device/-devd` (default follows `--device`), `-ngld`,
+`-otd`, `-ctkd/-ctvd`, `--spec-draft-n-max` (default 3), `--spec-draft-p-min`.
+One process sees only the devices its backends were built with — CUDA+Vulkan
+is the combination that shows a CUDA card AND an AMD iGPU (HIP+CUDA cannot
+coexist); a CUDA-only build has no `Vulkan0` to draft with.
+
+**Step 6 — cross-model pairs need a byte-identical tokenizer.** Speculation
+compares token IDs, so draft and target must map id → same string. Verify
+cheaply from the GGUF headers (first ~8 MB over HTTP, no full download):
+parse `tokenizer.ggml.tokens`, print vocab size, merges, `pre`, and sha256
+over the first 2000 tokens —
+
+```
+vocab=248320 merges=247587 model=gpt2 pre=qwen35
+sha256(first 2000 tokens)=8619c68fe4466d3a   # identical => ids are interchangeable
+```
+
+Identity is necessary, NOT sufficient — the drafter still has to be cheap and
+accepted. Report in this order: does the target already draft; break-even
+`p` at K=1,3,7 versus the incumbent; the realistic band and resulting rate;
+whether the drafter fits (VRAM) and the attachment surface exists; the one
+measurement that would settle it.
+
 ## Notes

 - Speculative decoding helps low-QPS, memory-bound, latency-focused workloads. It helps less at high QPS.
@@ -225,4 +415,8 @@ History: before `--spec-default`, the box ran DSpark (`draft-dspark`, retired 20
 - [[Dankclaw System Specs]]
 - [[Dankserver System Specs]]
 - [[VLLM Overview]]
- [[deployment]]
+- [[deployment]]
+- [[GGUF Compatibility]] — header parsing and custom tensor-type lock-in
+- [[Llama Cpp Serving Sizing]] — slot-pool and drafter KV sizing arithmetic
+- [[Llama Cpp Multi-Device Splits]] — where the drafter's KV actually lands
+- [[frank-large-benchmark]] and [[frank-small-benchmark]] — the single-GPU env drop-in A/B vehicle
@@ -0,0 +1,123 @@
+---
+type: reference
+title: GGUF Compatibility and Runtime Lock-in
+description: How to answer "will engine X load this GGUF?" from the file's header alone, why a custom tensor type forces the runtime, and the discipline for pinning a fork build that can actually serve the file.
+tags: [gguf, llama.cpp, forks, quantization, speculative]
+timestamp: 2026-10-05T00:00:00-07:00
+---
+
+# GGUF Compatibility and Runtime Lock-in
+
+**The law: a GGUF's tensor-type histogram and ftype pick the runtime — read the
+header before promising a file to any loader.** Stock llama.cpp, Ollama and LM
+Studio refuse custom types at load (`unknown tensor type`); a fork-private type
+means exactly one engine can serve the file. Measured on this fleet repeatedly:
+`Qwen3.8-27B-Q8_0_ROCMFPX_AGENT.gguf` is ftype 115 with 311 tensors of type 103
+(`GGML_TYPE_Q8_0_ROCMFPX`) — only charlie12345/ROCmFPX reads it. The full type
+tables live in [[Llama Cpp Runtimes]] and [[Llama Cpp Code Diff]]
+(ggml_type 100–108, ftype families 100–106 / 110–113 / 118).
+
+## 1. Read the header, do not guess
+
+Parse the GGUF KV section and histogram the tensor types plus
+`general.architecture.file_type` — one python read over the first few MiB, no
+model load. Counting types out of the KV section settles a runtime argument in
+minutes; repo names and model cards do not.
+
+## 2. Compare routing variants by tensor mix, not by name
+
+Two files of the same quant can differ in how many tensors stay at plain Q8_0
+(measured: 187 vs 2 output-head tensors in an `_AGENT` variant of the same
+size). That difference is what moves spec-decode acceptance (0.953 vs 0.911)
+— the "same size" file is not the same file.
+
+## 3. Pin the build by commit SHA
+
+A fork that can read the file today can still crash mid-load: the 2026-08-25
+ROCmFPX builds read the ftype-115 file and then SIGSEGV in
+`ggml_compute_forward_set_rows`. Record the SHA in the build script
+(`scripts/build-rocmfpx.sh` pins `36e9acd`) and check it out explicitly — never
+a branch.
+
+## 4. Header diff method (candidate vs known-good)
+
+Answer "will engine X load this GGUF?" by diffing the candidate's header
+against a file the engine **already loads** — never by filenames or model cards;
+filenames lie about quantization and about MTP/vision heads.
+
+1. **Get the bytes**: both files, from byte 0, at least 64 MiB. Tokenizer KVs
+   (151k token strings + merges) routinely eat 5–8 MiB, so a small range
+   desynchronises the parser. True size from `curl -sIL` → `x-linked-size`
+   (a bare HEAD `Content-Length` can lie):
+
+   ```sh
+   curl -sL -r 0-67108863 -o cand.bin \
+     "https://huggingface.co/<owner>/<repo>/resolve/main/<file>.gguf"
+   ssh <host> 'head -c 67108864 /path/to/known-good.gguf' > good.bin
+   ```
+
+2. **Parse — and lock the parser against the known-good file first.** Layout
+   (v2/v3): `magic[4]` `version u32` `n_tensors u64` `n_kv u64`, then KVs, then
+   tensor infos. Strings are u64 length + bytes. **The value-type enum is NOT
+   llama.cpp's `gguf.h`:** in the files seen (2026-09-28, unsloth + ajgazin UD
+   GGUFs) type **8 = STRING, 9 = ARRAY**; llama.cpp's header has STRING=2,
+   ARRAY=3 and desynchronises immediately. Working table:
+
+   ```
+   0 u8   1 i8   2 u16  3 i16  4 u32  5 i32  6 f32
+   7 bool 8 STR  9 ARRAY(u32 et, u64 n)  10 u64  11 i64  12 f64
+   ```
+
+   Validate before trusting: the parse is right iff the tensor-info block ends
+   somewhere sane and names look real (`blk.0.attn_q.weight`,
+   `token_embd.weight`, `output_norm.weight`). EOF throw = wrong enum or
+   offset — do not work around it.
+
+3. **Diff what engines actually bind on**: `general.architecture`; tensor
+   count; tensor name list in order; per-tensor `ggml_type` and dims; geometry
+   KVs (`<arch>.block_count`, `.embedding_length`, `.feed_forward_length`,
+   `.attention.head_count[_kv]`, `.rope.*`, `.ssm.*`, `.context_length`,
+   `.nextn_predict_layers`); MTP tensors (`nextn` in the name); vision tensors
+   (`visual`, `v_` prefixes). A same-family fine-tune quantized by the same
+   tool usually comes out byte-identical in every one of these — that is the
+   answer.
+
+4. **Then read the engine's source before calling a delta a blocker.** A
+   metadata difference only matters if the engine reads that key
+   (`grep -rn "file_type\|recurrent_layers\|full_attention_interval" <src>/`).
+   Worked example (gufo v0.1.1): `general.file_type` is deliberately not
+   consulted (UD-*_K_XL ships a stale ftype 15; the tensor table rules), and
+   `qwen35.attention.recurrent_layers` is never read (layer kinds come from
+   `full_attention_interval`) — both deltas were non-issues the header diff
+   alone would have flagged wrongly.
+
+5. **Say what you did not do.** Header analysis is strong evidence, not a
+   load. State byte sizes, check free disk before proposing a fetch, and do
+   not download tens of GB unasked — pointing the engine at the file is the
+   definitive test.
+
+**Report shape:** verdict first (loads / will not / loads-with-caveats), then
+the side-by-side diff table, then only the load-bearing caveats: text-only vs
+vision tensors, chat-template differences (matters for llama.cpp, not for
+engines with compiled-in Jinja), license, disk headroom, and whether the base
+model is abliterated/uncensored.
+
+## 5. A/B a fork's spec decode by stripping the whole flag set
+
+`--model-draft` without `--spec-type` segfaults at load (measured 2026-10-02,
+core dump, no diagnostic). A baseline "no-draft" arm must remove `--model-draft`
+AND every `--spec-draft-*` AND `--spec-type` together — the flags move as a set.
+
+## 6. Before deleting a retired payload, read the rollback tag's env file
+
+A rollback may need a small companion dir: a drafter living in a directory
+named after an old quant looks like trash but is load-bearing. Same class as
+the gufo payload removal of 2026-10-03, which turned that rollback into a
+re-download.
+
+## Related
+
+- [[Llama Cpp Runtimes]] — which fork ships which custom types
+- [[Llama Cpp Code Diff]] — the enum tables and code-level differences
+- [[frank-large-rocmfpx-q8agent-mtp]] — the ftype-115 lock-in in production
+- [[Llama Cpp Serving Sizing]] — pricing what a payload costs once it loads
@@ -0,0 +1,274 @@
+---
+type: reference
+title: Llama Cpp Multi-Device Splits
+description: What llama.cpp can and cannot express across devices, the no-collective layer-split path, split balancing by measurement, drafter KV sizing from GGUF headers, and heterogeneous placement triage.
+tags: [llama.cpp, multi-gpu, layer-split, pcie, sizing, speculative]
+timestamp: 2026-10-05T00:00:00-07:00
+---
+
+# Llama Cpp Multi-Device Splits
+
+## 1. Expressibility law — classify the arrangement before running anything
+
+llama.cpp expresses exactly two things across devices, and they are not
+interchangeable:
+
+| arrangement | how | expressible? |
+|---|---|---|
+| **Layer split** — layers 0..n on dev A, rest on dev B | `-sm layer -ts <a>/<b>` | **yes**, no collectives needed; works cross-vendor (CUDA + Vulkan measured 2026-09-16) |
+| **Per-tensor placement** — attention+KV on A, experts on B | `-ot <regex>=<dev>` | **NO.** `--override-tensor` sets `has_tensor_overrides()`, a REQUIRED term of the scheduler's `pipeline_parallel` flag (only read site: `llama-context.cpp`, in the `pipeline_parallel = …` expression). That flag sets `n_copies = parallel ? MAX : 1` in `ggml_backend_sched_new`; with `n_copies == 1` a pre-allocated weight on a non-primary device has no backend and the graph aborts at `ggml-backend.cpp:941` (`pre-allocated tensor (blk.0.ffn_down_exps.weight) in a buffer (Vulkan1) that cannot run the operation (NONE)` — `(NONE)` is the leaf's own op name). |
+
+Failure modes observed (mainline `37b53fd`): per-tensor `-ot exps=Vulkan1`
+with `-dev CUDA0,Vulkan1` → scheduler abort; layer-scoped `-ot` with a
+cross-backend pair → model load fails with only `failed to load model`;
+`-ot x=CPU` **works** because the CPU backend is last in the list — **the CPU
+is the only legitimate partner in a heterogeneous split.** On a CUDA+AMD host
+a card can be the trunk OR the offload host, never the complement of an
+AMD-resident peer. Pure Vulkan changes nothing here — the gate is the
+override, not the driver mix.
+
+If the plan needs tensors on a specific device: stop, the answer is no for
+any vendor mix. Expressible alternatives, in order: (1) two independent
+endpoints, one model per device; (2) single-vendor layer split; (3) device +
+CPU per-tensor offload (mainline pinning guard: `GGML_CUDA_NO_PINNED`; ik fork
+was `GGML_CUDA_NO_PINNED_WEIGHTS` — without it a CPU-overridden tensor can
+land in a pinned `CUDA_Host` buffer and page-lock ~108 GiB).
+
+Layer-split arithmetic to check before promising: a whole layer carries its
+experts (MoE) — compute GiB/layer, not per param-group; only some layers carry
+KV (every 4th on the hybrid), so a share of layers is only a share of KV; big
+non-repeating tensors (PLE tables) land on the main device and cannot move —
+they cap how small the primary can get.
+
+## 2. No collectives? The atomics wall and the layer-split answer
+
+Decide with one line before designing:
+
+```bash
+sudo lspci -vv -s <card2> | grep AtomicOpsCtl   # ReqEn- ⇒ no collectives, ever
+lspci -tv                                        # a Routing- bridge is a hardware wall
+```
+
+`ReqEn-` kills vLLM entirely — TP needs a per-layer allreduce AND PP needs a
+device-backend `send`/`recv` of the hidden state
+(`vllm/distributed/parallel_state.py`) — i.e. the same RCCL primitive that
+fails on an atomics-less card. llama.cpp's `--split-mode layer` needs no
+collective: layers distribute in order, only the boundary activation crosses
+PCIe once per forward pass. Full probe/repair detail:
+[[PCIe Link and Atomics]].
+
+## 3. Build: pin by feature, bundle the runtime at the exact depth
+
+- Pin the ref by FEATURE list (e.g. DFlash2 + qwen35 MTP on 2026-09-16 →
+  `ggml-org/llama.cpp#27342`); never float. Verify a fork build the same way
+  (`ls <venv>/…/spec_decode/ # dflash/ vs dflash2/`).
+- `-DCMAKE_HIP_ARCHITECTURES=<gfx only>` — a sibling arch links, loads, then
+  segfaults. `-DGGML_HIP_RCCL=OFF` when there is no atomics path (only tensor
+  mode degrades to the butterfly fallback; layer split unaffected).
+- **TRAP — `cp -r SRC DST/` flattens when `DST` does not exist** and drops one
+  directory level. rocBLAS/hipBLASLt look for device code at a FIXED depth
+  (`../lib/rocblas/library`, `../lib/hipblaslt/library/<arch>`); flattened,
+  the engine loads every GiB, answers `/health`, and dies on the FIRST request
+  with `Cannot read .../TensileLibrary.dat: No such file or directory for GPU
+  arch`. `mkdir -p DST && cp -r SRC/. DST/`, then
+  `patchelf --set-rpath '$ORIGIN/../lib' build/bin/llama-*`.
+- Dual-backend build (CUDA + Vulkan is the practical pair; HIP+CUDA do not
+  coexist): `-DGGML_CUDA=ON -DGGML_VULKAN=ON` in one tree; Vulkan needs
+  `glslc` + libvulkan headers. **Device names are not what you expect:**
+  `--list-devices` lists the discrete card under BOTH backends (`CUDA0` AND
+  `Vulkan0`) and the APU LAST (`Vulkan1`) — a wrong `-ot` name makes
+  llama.cpp print the valid set and exit (self-diagnosing).
+- What the existing build can do: `GPU_BACKEND_*=OFF` lines in its
+  CMakeCache.
+
+## 4. Serve flags that matter on a split
+
+```bash
+HIP_VISIBLE_DEVICES=0,1 llama-server \
+  --device ROCm0,ROCm1 --split-mode layer --tensor-split 1,1 \
+  --gpu-layers 999 --fit off --flash-attn on \
+  --ctx-size <n> --parallel 1 --cache-type-k f16 --cache-type-v f16 \
+  --jinja --metrics --cache-prompt \
+  --spec-type draft-dflash --spec-draft-model <dflash2.gguf> \
+  --spec-draft-device ROCm1 --spec-draft-n-max 7
+```
+
+- `--fit off` + explicit `--ctx-size`: auto-fit must not resize a context set
+  on purpose. **`--parallel N` DIVIDES `--ctx-size`** — one slot at large ctx
+  is the concurrency policy, concurrent agent loops queue.
+- Pin the drafter to the card holding the target's LAST layers
+  (`--spec-draft-device`); under tensor mode an unpinned draft joins the
+  meta-device KV allocation and aborts the KV build. `--spec-draft-n-max` must
+  be ≤ the drafter's `block_size - 1` (DFlash2: 8 ⇒ 7).
+- Qwen3.8-family templates RAISE on `reasoning_effort=high` (500) — probe
+  low/medium/high/xhigh and map harness levels before declaring ready.
+
+## 5. Split balancing: measure, never interpolate
+
+**A `-ts` step is ONE LAYER, and hybrid stacks do not have uniform layers.**
+qwen35 is 16 × (3 Gated DeltaNet + 1 Gated Attention): one step ≈ 0.43 GiB on
+a DeltaNet boundary, ≈ 1.43 GiB on a full-attention one (weights + that
+layer's own f16 KV) — measured on frank-medium. The gap can jump 2.9 GiB in a
+single step; the best integer split can sit 0.2 GiB from balance while its
+neighbours are 1+ GiB off. An affine fit reproduces the deployed point and
+overstates gaps in between (`tests/frank_medium_unit.bats` encodes the fit as
+a gate; treat a NEGATIVE fitted constant as a unit bug).
+
+Knobs that move weight and REQUIRE a re-measure: `--ubatch-size` (compute
+buffers are **~2x heavier on the output/drafter card** — 512→1536 added
+~3.3 GiB there vs ~1.7 GiB elsewhere), a drafter swap (weights + draft KV),
+`--ctx-size`/`--parallel` (KV). Total free VRAM is invariant across splits —
+only the distribution moves; use that to sanity-check a suspicious row.
+
+**Procedure (one split per load):**
+
+```bash
+WORK=/tmp/split.env; cp scripts/serve-<svc>.env $WORK
+export FRANK_MEDIUM_ENV_FILE=$WORK
+for ts in 39,26 38,27 37,28 36,29 35,30; do
+  sed -i "s|^TENSOR_SPLIT=.*|TENSOR_SPLIT=$ts|" $WORK
+  nohup bash scripts/serve-<svc>.sh >/tmp/split.log 2>&1 &
+  # poll /health (or process exit), then:
+  rocm-smi --showmeminfo vram | grep 'VRAM Total Used'   # sample ~6 s after health
+  pkill -x llama-server; sleep 6
+done
+```
+
+Load → sample → kill, NO requests sent (allocation only). Run under `nohup`
+and poll from a separate call (a load is 30–90 s). Pick the smallest
+inter-card gap that clears the per-card floor with room to spare; one `-ts`
+step ≈ 1.5 GiB at ~6k batch / 262k ctx on 30 GiB cards — enough to bracket,
+not to solve. Prefer balance: the busy card's headroom bounds the context
+(`ctx_extra ≈ (floor − measured_free) / per-position cost on the busier
+card`). A faulted split (`SDMA0 UTCL2 page fault` → `illegal memory access` →
+`ggml_abort`) is OFF-LIMITS — re-measure, never re-tune; more headroom does
+not prevent a boundary-copy fault.
+
+## 6. Drafter KV: sized from the TARGET, bounded by its window policy
+
+`common/speculative.cpp` builds the draft context with
+`cparams.n_ctx = llama_n_ctx(ctx_tgt)` — **no flag caps it.** So:
+
+```
+draft KV = ctx_tgt × block_count × 2 (K,V) × kv_heads × key_length × dtype
+```
+
+…unless the drafter declares a sliding window. Measured at 262,144 target
+ctx: DFlash2 (`sliding_window = 2048`) ≈ 40 MiB; a same-shape 5-layer drafter
+with NO window = 20 KiB/position = **5.0 GiB** — a four-layer swing in the
+balance point, and swapping them invalidated an already-balanced split.
+
+**Failure signature:** the engine loads the whole model, answers `/health`,
+and dies on the FIRST draft decode with `ROCm error: out of memory` /
+`cudaMalloc failed` inside `common_speculative_impl_draft_dflash::process` —
+a sizing failure, not a broken drafter; under a self-healing unit it presents
+as a crash-loop. Read `sliding_window`, `block_count`, `head_count_kv`,
+`key_length` from the drafter's GGUF header first. A compact reader (the
+three format traps: no KV alignment — v3 alignment applies only to the tensor
+section; arrays are `type(9) → element_type(u32) → count(u64)`, element type
+FIRST; the enum is `7=BOOL 9=ARRAY`, 64-bit = 10/11/12, and 8=STRING/9=ARRAY
+is the v2/v3 file order, NOT gguf.h's 2/3):
+
+```python
+import struct, sys
+def rd(path):
+    f = open(path, 'rb'); assert f.read(4) == b'GGUF'
+    struct.unpack('<I', f.read(4)); struct.unpack('<Q', f.read(8))
+    nk = struct.unpack('<Q', f.read(8))[0]
+    def s():
+        n = struct.unpack('<Q', f.read(8))[0]; return f.read(n).decode('utf-8', 'replace')
+    SCALAR = {0:1,1:1,2:2,3:2,4:4,5:4,6:4,7:1,10:8,11:8,12:8}
+    FMT = {0:'<B',1:'<b',2:'<H',3:'<h',4:'<I',5:'<i',6:'<f',7:'<?',10:'<Q',11:'<q',12:'<d'}
+    keys = {}
+    for _ in range(nk):
+        k = s(); t = struct.unpack('<I', f.read(4))[0]
+        if t == 8: keys[k] = s()
+        elif t in SCALAR: keys[k] = struct.unpack(FMT[t], f.read(SCALAR[t]))[0]
+        else:
+            et = struct.unpack('<I', f.read(4))[0]; n = struct.unpack('<Q', f.read(8))[0]
+            if et == 8: keys[k] = [s() for _ in range(n)]
+            elif et in SCALAR: keys[k] = ('arr', et, n); f.read(n * SCALAR[et])
+            else: raise Exception('unsupported arr elem type %d' % et)
+    return keys
+```
+
+## 7. Heterogeneous placement triage (go/no-go with arithmetic)
+
+- **Link:** `LnkCap` vs `LnkSta` at every hop; expect **idle-downshift** (a
+  Gen4 x4 dock idles at Gen1 x4, ~1 GB/s) — measure under load. Per-hop cost
+  vs hidden state (5120-dim f16 = 10 KiB/hop ≈ 1.3 µs at Gen4 x4): bytes are
+  almost never the constraint; **hop count is** — every boundary is a host
+  sync, and a per-op-split runtime can hit 100 splits/token at ~5 % SM. Read
+  `graph splits` from the loader log.
+- **KV is usually the bottleneck:** `KV/token = kv_bytes_per_position ×
+  context` (attention reads the whole cache every token). Highest-leverage
+  placement: **KV on the fastest-memory device together with its attention
+  compute** — never KV alone over the link (GBs over 7.88 GB/s = hundreds of
+  ms/token). Check fit: `kv_positions × slots × bytes + attention_trunk ≤
+  VRAM`.
+- **What placement never buys:** prefill is compute-bound and expert weights
+  are read once per batch — moving them buys nothing and adds a serial stage.
+  Calibration: Strix Halo + 3090, moving 12.5 % of expert mass to the GPU
+  gave decode +12 % but prefill 192 → 32 tok/s. The harmonic
+  `1/(f_big/BW_slow + f_fast/BW_fast)` reproduces the decode gain within ~2
+  points (assumes sequential stages — true at batch 1).
+- **UMA check before assuming additivity:** `mem_info_vram_total` (BIOS
+  carve-out) vs `mem_info_gtt_total` (aperture onto SYSTEM RAM). A big GTT
+  window means the "GPU memory" IS the same LPDDR: adding a stage there adds
+  a compute engine, not capacity/bandwidth. Only discrete VRAM is additive.
+- **Speculation as the "cheap" split:** bandwidth-free (token ids per round)
+  but needs the exact tokenizer for a plain drafter or a head trained for
+  that target; a plain drafter costs K sequential forwards (break-even needs
+  p ≈ 0.7+ at r≈0.4–0.5, K=3); compare against the INCUMBENT (a target
+  already committing ~1.7 tok/round on its own MTP head is a high bar); and
+  check the runtime accepts external drafters at all (closed engines 400
+  unknown drafter names by design). Details: [[DFlash DSpark Speculative Decoding]].
+- **Hazards:** keep paged/sidecar tables where they are read from; keep mmap
+  ON for huge-embedding models (anonymous memory → cgroup kill); CUDA graph
+  capture can abort on unusual graph shapes; make the FIRST measurement a
+  single-device baseline of the runtime on the new device — a split's
+  arithmetic is worthless if the baseline loses to the incumbent; a split
+  consumes the fast device entirely — check what it was serving.
+
+**Verdict ranking:** (1) two independent endpoints — often the real answer;
+(2) one process, weights-resident, KV co-located with attention on the fast
+pool; (3) hot-expert cache only if `misses × bytes × tok/s` fits the link and
+overlaps — no mainstream runtime implements it; (4) name what does NOT help:
+TP across unequal devices, KV on the far device, loading over the link,
+prefill-motivated placements.
+
+## 8. Harness syntax and verification
+
+`llama-bench` separates a LIST with `/` and SWEEP values with `,`:
+`-dev CUDA0,Vulkan1` is TWO single-device runs, never one two-device run;
+`-ts 0.85,0.15` is two sweep values each becoming `[0.85, 0, …]` — device 0
+takes everything. Write `-dev A/B -ts x/y`. Read the table's own `dev` column;
+any placement-dependent number needs an independent memory check
+(`nvidia-smi` sampling while loaded). (This artifact produced a wrong fleet
+conclusion 2026-09-16 — "CUDA + Vulkan cannot serve one model" — with two
+bogus OOMs that were really all-111-GiB-on-the-24-GiB-card.)
+
+Verification battery: `/health`; both cards' memory ≈ half the weights each
+(busier card also carries the drafter; compare `model_bytes/2 + drafter +
+ctx×KV/2 + buffers`); raise ctx by N and watch BOTH cards rise by
+N·bytes/2 — neither card's total may equal the model file (that arithmetic
+rules out replication); `spec_decode_num_drafts_total` > 0 and a real
+completion; a deep prompt (140k) proves a raised window is REAL, not
+declared; repeat prompt → `timings.prompt_n` collapses (prefix cache live);
+`/slots` shows expected `(id, n_ctx)` pairs; two concurrent requests with
+`journalctl -o cat | grep 'slot print_timing'` proves drafting on BOTH slots
+(default journal output truncates these lines).
+
+Ops traps: a server started over ssh outlives stopping the runner (keeps the
+port — `pkill -x <binary>` first); `systemctl start` of a root unit needs
+sudo (polkit "Access denied" otherwise); old crash `ROCm error` lines live in
+the journal forever — check `ActiveEnterTimestamp`/`NRestarts` before
+treating a grep count as a live fault.
+
+## Related
+
+- [[PCIe Link and Atomics]] — the atomics wall, link probing, repair
+- [[Llama Cpp Serving Sizing]] — slot-pool and KV pricing
+- [[DFlash DSpark Speculative Decoding]] — spec types, chaining, A/B
+- [[frank-medium-llamacpp-dflash2]] — the layer-split era runbook
+- [[Dual-R9700-Cooperation]] / [[x4-pair-experiment]] — the measured history
@@ -0,0 +1,163 @@
+---
+type: reference
+title: Llama Cpp Serving Sizing
+description: Slot-pool sizing against a headroom band, per-device VRAM accounting, prompt-cache/state sizing and the QSA flash-attn fix — the measurement-first procedures for llama.cpp serving shapes.
+tags: [llama.cpp, vram, kv-cache, slots, flash-attn, sizing]
+timestamp: 2026-10-05T00:00:00-07:00
+---
+
+# Llama Cpp Serving Sizing
+
+## 1. Slot-pool sizing against a headroom band
+
+Goal: choose `CTX_SIZE`/`PARALLEL`/KV types so N conversations fit a GPU while
+keeping a requested amount of VRAM free — from measurement, never arithmetic
+guesswork.
+
+1. **Fix the invariants.** Read the payload's KV geometry (attention layers
+   only, KV heads, head_dim, KV dtype bytes) and the per-slot rule
+   `n_ctx_slot = CTX_SIZE / PARALLEL`. A hybrid model with
+   `kv_unified = 'false'` gives each slot its own cache AND a fixed per-slot
+   state buffer — slot count is a memory term, not free.
+2. **Two boots on the real card** (production stopped, per the bench law),
+   each near the target; record `nvidia-smi` used/total plus the boot log's
+   `n_slots`/`n_ctx_slot`: shape A = current/cheap, shape B = new demand.
+   `non-KV = used − (tokens × KiB/token)`; the two rows separate
+   weights/buffers/graphs from the new demand (a 918 MB projector bills
+   ~1.4 GiB).
+3. **Solve the band:** `KV_tokens ≤ (card − non-KV − headroom_low) /
+   KiB/token`, round the pool DOWN to a multiple of `PARALLEL × 4096` so every
+   slot gets an equal integer window. Boot and measure; if outside the band,
+   walk one step — never re-derive from a formula.
+4. **Propagate the per-slot number — it is the client contract:**
+   `models.yml` `contextWindow` = `n_ctx_slot` (NOT the model's native window),
+   or long conversations die with `400 exceed_context_size_error` instead of
+   compacting; compaction/summary budgets sized to the old window get
+   re-checked; endpoint docs and the rollback table move too.
+5. **Prove behaviour:** boot-log slot count, `nvidia-smi` free in band, N
+   concurrent requests answered in parallel with N+1 queued, and — if a
+   projector ships — a solid-color image probe through an **`image_url`
+   content part** expecting the exact color. The legacy top-level `images`
+   field is silently ignored, so a probe that uses it measures a
+   hallucination, not vision.
+
+Gotchas: shorter slots can make decode FASTER (~5 % measured); the KV pool is
+allocated whole at boot so post-boot `memory.used` IS the ceiling; record both
+measured rows in the service `.env` comment block and mirror the non-KV price
+in the bats budget gate so a shape that busts the band fails the suite instead
+of the boot.
+
+## 2. Per-device VRAM accounting ("where did the memory go?")
+
+First decide which kind of memory is in question:
+
+| device | kind | read it with |
+|---|---|---|
+| RTX 3090 (Dankclaw) | discrete GDDR6X | `nvidia-smi --query-gpu=memory.total,memory.used,memory.free` |
+| gfx1151 iGPU (Dankclaw) | **UMA** — "VRAM" is host RAM | `/sys/class/drm/card1/device/mem_info_gtt_{total,used}` (114 GiB) vs `mem_info_vram_*` (2 GiB carve-out) |
+| R9700s (Dankserver) | discrete | `rocm-smi` figure (64 GiB, SVM-backed) |
+
+Per-process truth on NVIDIA:
+`nvidia-smi --query-compute-apps=pid,process_name,used_memory`. Never add the
+iGPU's GTT to host RAM as a separate pool; `free -g` is the arbiter for
+co-tenants.
+
+- **Engine ledger first.** Halogen prints weights pinned (+ remaining 2 MiB
+  contiguous blocks — the honest fragmentation signal: 240 blocks/12.3 GiB
+  after reboot vs 475/14.7 GiB before means the model did not come back into
+  the same hole), device budget, KV pool/working split, "left for everything
+  else". llama.cpp prints at `llama-model.cpp:1827` (per-backend model buffer)
+  and `llama-context.cpp:692` (compute buffer) — but **`llama-bench`
+  suppresses loader INFO and a systemd unit did not put them in the journal**
+  (verified `grep -c` = 0). Recipe: one launch through `llama-cli`/`llama-server`
+  keeping the first ~200 lines.
+- **Otherwise compose from tensor sizes.** Group the GGUF tensor table by
+  role (Flash-Next example: 75.0 GiB experts at 1.5625 GiB/layer, 32.8 GiB PLE
+  table, 3.2 GiB everything else), KV ≈ 2 KiB/position per f16 attention
+  layer, × derived context. Vendor delta is real: same layers+KV cost 12.80
+  GiB under Vulkan vs 14.99 GiB under CUDA (~3.3 vs ~1.1 GiB of
+  graph/compute pool).
+- **Traps:** mmap'd PLE is reclaimable page cache; `-ot` makes it pinned —
+  "move layers to the card" converts reclaimable cache into device memory and
+  frees nothing for co-tenants. A served model's footprint ≠ weight size
+  (frank-small: 22.6 of 24.1 GiB for an 11.29 GiB model). Do the capacity
+  subtraction before designing: 12 attn layers × 1.62 GiB + 200k f16 KV =
+  28.6 GiB > 24.1 GiB usable → KV-in-VRAM on that pair only reaches ~64k.
+- **Context lengths: derive, never remember.** `llama-bench` computes
+  `n_ctx = n_prompt + n_gen + n_depth` (defaults 512/128/0, type_k/v F16,
+  flash_attn AUTO, split_mode LAYER). `-p 512,16384 -n 64` = contexts
+  576/16,448; a plain `tg` row has ~zero depth — `-d` is the only thing that
+  puts KV in the picture.
+- **Label every number** measured / derived / inference, and state the context
+  each figure was taken at — 12.8 GiB at 65,568 tokens is not the same claim
+  as at 200k.
+
+## 3. Prompt cache and state sizing
+
+`--cache-ram` (MiB, default 8192) is HOST RAM for saved prompt states —
+unpinned, on demand, moves no VRAM. `--ctx-checkpoints` (default 32) is a
+different mechanism: per-slot KV snapshots; `0` disables them.
+
+**State law** (compute before trusting any cap): a state is the serialized KV
+of the whole sequence + constant recurrent state + the draft context's KV:
+`per_token = 2 × n_full_attn_layers × n_kv_heads × head_dim × bytes (K and V) +
+draft_per_token`. Measured on frank-medium (qwen3.5 hybrid, f16, MTP draft):
+16×4×256×2×2 = 64 KiB/token + 4 KiB draft = **68 KiB/token**; a full
+196,608-token slot state ≈ 12.8 GiB. Cross-check against the journal's
+`prompt state size` lines — the two must agree.
+
+**The journal is the instrument:** `exceeds cache size limit … skipping` =
+cap binding in production; `making room … removing oldest` / `cache size
+limit reached` = eviction, and the printed size proves the active cap. Do NOT
+claim a raise is proven by `prompt_tokens_cached_total` rising — on exact
+repeats that counter is 0 by construction.
+
+**Hybrid + zero-checkpoint trap:** for a hybrid (DeltaNet/SSM + attention)
+model, `seq_pos_min` sits at the END of the slot's own prompt (a recurrent
+cell stores only the last position), so with `--ctx-checkpoints 0` there is no
+checkpoint to fall back on and the server logs `forcing full prompt
+re-processing … (likely due to SWA or hybrid/recurrent memory)`. Consequences:
+intra-slot reuse only survives when the request echoes the assistant's
+previous reply verbatim (a pure user resend is always a full re-prefill even
+at f_sim 1.000); the RAM prompt cache becomes the ONLY carrier of a
+conversation state ACROSS slots, so a cap that refuses deep states costs
+minutes of re-prefill whenever slots are reassigned; state restoration only
+runs via the LRU path (or LCP with f_keep < 0.5) — a plain prefix hit never
+calls `prompt_load`.
+
+**Verify a change:** live argv carries the flag (not the deploy log); `/health`
+200 + real completion; VRAM unchanged and host RAM mostly free; no new
+`exceeds cache size limit` lines, with a `making room` entry larger than the
+old cap as direct proof. If the ceiling itself was not exercised live, say so —
+crossing it needs a >120k-token conversation and must not run beside a live
+service (bench law).
+
+## 4. QSA flash-attn assert fix (qwen4exp)
+
+`--flash-attn on` aborts graph build for qwen4exp models:
+`GGML_ASSERT(a->type == GGML_TYPE_F32)` at `ggml.c:5371` through
+`build_attn_qsa`. Root cause: `build_attn_qsa` calls `ggml_fill` on the
+F16-cast kq mask returned by `get_kq_mask()` when flash attn is on, but
+`ggml_fill` requires F32.
+
+Fix (mirrors the MLA sparse path in `llm_graph_context::build_attn`): run the
+top-k mask surgery on the F32 base mask (`inp->self_kq_mask`), cast to F16
+only at the attention call when `flash_attn` is enabled
+(`kq_mask_attn = cparams.flash_attn ? ggml_cast(ctx0, kq_mask_top_k,
+GGML_TYPE_F16) : kq_mask_top_k`). The patch lives at
+`scripts/patches/qwen4exp-qsa-f32-mask.patch`.
+
+Only the charlie12345/ROCmFPX build-qwen38 (PR #27742 qwen4exp merged) takes
+it; CIRU, upstream and stock builds do not. Validate live: load with
+`--flash-attn on`, health 200, a real completion finishing `"stop"`, journal
+clean of `GGML_ASSERT`/`cudaMalloc`/`missing tensor`, and flash-attn engaged
+in the timings. Context cap for full-model residency on that shape stayed
+32,768.
+
+## Related
+
+- [[GGUF Compatibility]] — what a payload is before it is priced
+- [[frank-small-llama-mimo-q6k-2x200704]] — the current two-boot priced shape
+- [[frank-medium-llamacpp-dflash2]] — where the 68 KiB/token datum was measured
+- [[DFlash DSpark Speculative Decoding]] — the spec A/B protocol on llama.cpp
+- [[Dual-R9700-Cooperation]] / [[x4-pair-experiment]] — the multi-device side
@@ -0,0 +1,244 @@
+---
+type: reference
+title: PCIe Link and Atomics
+description: Link probing (idle downshift, path ceilings, CUDA/HIP bandwidth probes), the streaming-vs-compute ceiling arithmetic, and the PCIe-atomics wall that decides whether AMD multi-GPU collectives can exist at all.
+tags: [pcie, rccl, atomics, multi-gpu, bandwidth, amd, nvidia]
+timestamp: 2026-10-05T00:00:00-07:00
+---
+
+# PCIe Link and Atomics
+
+## 1. Degraded or just idle?
+
+The trap: an idle GPU reports `LnkSta: Speed 2.5GT/s (downgraded)` — the
+driver downshifts to Gen1 for power. That is not a fault and not the slot's
+fault. Never conclude from an idle reading.
+
+1. **Path ceiling from the kernel (cheapest, most decisive):**
+   `sudo journalctl -k -b --no-pager | grep "available PCIe bandwidth"` →
+   `pci 0000:c4:00.0: 63.012 Gb/s available PCIe bandwidth, limited by 16.0
+   GT/s PCIe x4 link at 0000:00:03.1 (capable of 252.048 Gb/s with ... x16)`.
+   The "limited by" device names the **bottleneck** (usually a root port) and
+   its width — that is the number a swap could move; "capable of x16" is the
+   endpoint's OWN capability, and the gap between them is the story.
+   `dmesg` rotates; `journalctl -k -b` keeps this boot.
+2. **Both sides separately:** `LnkCap`/`LnkSta`/`LnkCtl2` on endpoint AND root
+   port. `LnkCtl2: Target Link Speed: 16GT/s` on both ends = the platform
+   intends Gen4, so a Gen1 current speed is power management, not
+   misconfiguration. A root port whose `LnkCap` width exceeds the trained
+   width means the socket/adapter wires fewer lanes — **M.2 connectors are 4
+   lanes by pinout; an M.2 riser can never deliver more.**
+3. **Prove the link under load (the step people skip):** on NVIDIA sample NVML
+   while the GPU works
+   (`nvidia-smi --query-gpu=pcie.link.gen.current,pcie.link.gen.hostmax,pcie.link.width.current,pcie.link.width.max`):
+   `gen.current` climbing 1 → `hostmax` = idle downshift, healthy; stuck below
+   `hostmax` under load = real degradation; `width.current < width.max` under
+   load = **physical lane limit** — no software touches it. On a busy
+   production GPU this is a usage smoke test, not a benchmark — never stop a
+   live service to read its link.
+4. **Do not plan hardware from `dmidecode -t slot`** — consumer firmware
+   labels Ethernet/SD controllers as "M.2 slots" and advertises slots in PCI
+   domains that do not exist. Trust `lspci -nn | grep "PCI bridge"` and
+   `lspci -tv` (empty root ports still enumerate; an absent port is absent).
+5. **"Would moving the card help?"** Compare the bottleneck device from step 1
+   with the target port's `LnkCap` width; two M.2 sockets = both x4 = a
+   no-op. Clean `DevSta`/`CESta` with a low trained speed means "wired
+   narrow", not "failing". Report ceiling, card capability, what the idle
+   reading meant, and whether any physical change moves the number.
+
+Gen4 usable payload after 128b/130b: **x4 7.877 GB/s, x8 15.75, x16 31.5**.
+
+## 2. The engine shape decides whether the link matters at all
+
+| engine shape | what crosses PCIe per token | link matters? |
+|---|---|---|
+| streams weights (expert LRU, paging) | the missed weights (~GB/token) | **yes — decisive** |
+| computes offloaded weights in RAM (llama.cpp `-ot exps=CPU`) | activations only (~KB) | no; host RAM bandwidth decides |
+| everything resident | nothing | no |
+
+**Ceiling arithmetic:** `link time/token = active expert bytes / measured
+H2D rate`; `ceiling = 1 / link_time × resident-fraction saved (LRU hits)`.
+Compare against the compute-in-RAM path at host RAM bandwidth (256 GB/s
+LPDDR5X on the Strix Halo). **If the link is >10x slower than the memory
+behind it (x4 is 31x here), streaming loses by construction and no tuning
+rescues it; doubling the link only doubles a losing number.**
+
+Worked example (Dankclaw 2026-09-16): Gen4 x4 measured H2D **6.72 / D2H 6.60
+GB/s** (85 % of spec) — pinned H2D ≈ async, pageable within 1 %, i.e.
+link-limited not staging-limited. A 3 MiB slab (the expert-streaming
+granularity) = 474 µs. A3B MoE at Q4_K_M (~1.70 GiB active/token) → 272
+ms/token ⇒ **3.7 tok/s** (4.6 with 20 % resident) vs measured **20.46 tok/s**
+with experts computed on the CPU, 25–26.5 on the iGPU. The streaming design
+loses 3–5x.
+
+## 3. Measure the link: two probes, same mechanics
+
+Stop **all** production services first (fleet bench law) — the 3090's VRAM
+may be nearly full, and the probe contends for the very link it measures.
+
+**CUDA probe** (`pcie-link-test.cu`): sweep {3, 16, 64, 256 MiB, 1, 2 GiB}
+with `cudaMalloc` + `cudaHostAlloc`, event-timed; per size measure sync H2D,
+async H2D, D2H, one single transfer, and pageable H2D. `3 MiB` is the row
+that matters for expert streaming; big sizes give the ceiling.
+
+**HIP probe for RDNA4** (`pcie_bw.hip`) — the ROCm landmines each cost real
+time, so encode them:
+
+- `hipGetDeviceProperties` SEGFAULTS on some builds; use `hipGetDeviceCount`
+  + `hipSetDevice` only. First line: `setvbuf(stdout, NULL, _IONBF, 0);` — a
+  segfaulting process loses block-buffered stdout and hides the crash point.
+- Build with an explicit arch: `HIP_TARGETS=gfx1201 hipcc -O2
+  --offload-arch=gfx1201 -o p p.hip`; compile with `| grep -E "error:"`,
+  never `| tail -N` (truncation hid a missing `#include <cstring>`).
+- `hipMemset`/`hipDeviceSynchronize` are `[nodiscard]` — use or cast.
+- Mount the whole `/dev/dri` into ROCm containers; individual `renderD*`
+  nodes enumerate but fail context creation.
+- `rocm-smi --showmetrics`: `pcie_link_width` is trustworthy, but
+  `pcie_link_speed` reads the ASPM idle value and `pcie_bandwidth_acc/inst`
+  are usually `N/A` on consumer cards — live traffic generally cannot be
+  sampled.
+
+```cuda
+// pcie_bw.hip — sweep sizes, pinned vs pageable
+#include <cstdio>
+#include <cstring>
+#include <cstdlib>
+#include <hip/hip_runtime.h>
+static double one(size_t bytes, bool pinned, int iters) {
+    void *dev = NULL, *host = NULL;
+    if (hipMalloc(&dev, bytes) != hipSuccess) return -1;
+    if (pinned) { if (hipHostMalloc(&host, bytes, hipHostMallocDefault) != hipSuccess) return -1; }
+    else        { host = malloc(bytes); }
+    memset(host, 0x5a, bytes);
+    hipEvent_t a, b; hipEventCreate(&a); hipEventCreate(&b);
+    for (int i = 0; i < 3; i++) hipMemcpy(dev, host, bytes, hipMemcpyHostToDevice);
+    hipDeviceSynchronize();
+    hipEventRecord(a, 0);
+    for (int i = 0; i < iters; i++) hipMemcpy(dev, host, bytes, hipMemcpyHostToDevice);
+    hipEventRecord(b, 0); hipEventSynchronize(b);
+    float ms = 0; hipEventElapsedTime(&ms, a, b);
+    double gb = (double)bytes * iters / (ms / 1000.0) / 1e9;
+    if (!pinned) free(host); else hipHostFree(host);
+    hipFree(dev); hipEventDestroy(a); hipEventDestroy(b);
+    return gb;
+}
+int main(int argc, char **argv) {
+    setvbuf(stdout, NULL, _IONBF, 0);
+    if (hipSetDevice((argc > 1) ? atoi(argv[1]) : 0) != hipSuccess) return 1;
+    size_t sz[] = {2459136UL, 3UL<<20, 16UL<<20, 64UL<<20, 256UL<<20, 1UL<<30};
+    for (int i = 0; i < 6; i++) {
+        size_t b = sz[i];
+        int iters = (b >= (1UL<<30)) ? 10 : (b >= (256UL<<20)) ? 20 : 100;
+        printf("  %10.3f MiB | pinned %6.2f | pageable %6.2f\n",
+               b/1048576.0, one(b,true,iters), one(b,false,iters));
+    }
+    return 0;
+}
+```
+
+**Reading it:**
+
+- **Engine-vs-link is the diagnosis, not the raw number:** compare
+  `engine_effective_GB_s` to `link_GB_s` at the SAME transfer size. Engine at
+  ~40 % of the link at the same granularity = the wire is not the problem and
+  no hardware change will help.
+- Achieved GB/s is itself the proof of the trained link state (6.7 GB/s only
+  exists at Gen4 x4) — no separate under-load `LnkSta` sample needed.
+- **pinned ≈ pageable ⇒ pinning is not a lever.** async ≈ sync ⇒ link-limited
+  (the usual "pageable costs 2x" rule does not apply at this link).
+- Small-transfer penalty (2.345 MiB: ~106 µs vs ~78 µs ideal, ~21 %) is
+  per-transfer overhead — recoverable only by BATCHING transfers, not by a
+  faster link. (Dankserver 2026-09-29: bulk 22.7 GB/s at the 2.344 MiB slab =
+  91 % of Gen5 x8, while the engine's effective read path ran ~9.5 GB/s — the
+  gap is the fork's gather software path, not the wire.)
+- Both GPUs symmetric ⇒ no cross-device rebalance is available; measure both.
+- Don't quote a derived line from inside a probe without hand-checking it —
+  a printf format/argument mismatch in one of these printed a nonsense
+  ceiling. GiB→GB before dividing.
+
+## 4. The PCIe-atomics wall (AMD multi-GPU collectives)
+
+**Symptom:** multi-GPU AMD host fails TP/PP; the failure surfaces as
+`hipLaunchKernel: hipErrorIllegalState` for `ncclDevKernel_*` — nothing in
+vLLM/RCCL/PyTorch names atomics. Get the real message:
+
+```bash
+AMD_LOG_LEVEL=3 <the failing process>
+# → "Pcie atomics not enabled, hostcall not supported" → "AQL dispatch failed!"
+```
+
+**Cause chain:** RCCL kernels carry a hidden hostcall-buffer parameter; ROCr
+refuses to fill it without PCIe atomics, which it reads **once at driver
+probe** (`rocclr/device/rocm/rocdevice.cpp` `checkAtomicSupport()` — no else,
+no override). **No env var, module param, or sysfs toggle exists** —
+`setpci -s <gpu> CAP_EXP+28.w=0440` (DCTL2 bit 6) changes nothing at runtime.
+
+**Check the CHAIN, not the card:**
+
+```bash
+sudo lspci -vv -s <gpu> | grep -E "AtomicOpsCtl|LnkSta|LnkCap"  # want ReqEn+
+lspci -tv                                                        # every hop
+sudo lspci -vv -s <bridge> | grep AtomicOpsCap                   # want Routing+
+```
+
+`Routing-` is a read-only hardware capability. Typical AM5 layout: one x16
+slot on CPU root ports (atomics fine), the rest behind the chipset (usually
+not). Bandwidth for TP is small anyway: ~10 MB per token-step for a 27B at
+batch 4 ≈ 1.4 ms at Gen4 x4 — compute-bound, not interconnect-bound.
+
+**Probe matrix (bare 2-rank `torch.distributed`, one process per GPU, 25 s PG
+timeout) — one probe answers several design questions:**
+
+| Op | CPU-lane card | chipset card | Decides |
+|---|---|---|---|
+| `broadcast`/`all_reduce`/`barrier` | OK | FAIL | TP is dead |
+| `send`/`recv` (P2P), in ISOLATION | hang | FAIL | **PP is dead too** — vLLM's PP handoff IS this primitive |
+
+Test `send`/`recv` alone (a prior failed op leaves the group broken). Run the
+probe from a REAL file (`torch.multiprocessing.spawn` re-executes it; piping
+into `python3 -` fails with `FileNotFoundError: '/<stdin>'`). `world_size=1`
+always succeeds — **one-card success proves nothing.**
+
+**SAFETY — non-negotiable:** never run a GPU experiment beside a serving
+unit: `sudo systemctl stop <svc> && sudo systemctl mask <svc>` first (mask,
+never disable — disable deletes a relinked unit file), or use a host that
+serves nobody. A hung AQL dispatch can wedge the GPU driver; a probe beside a
+live service on 2026-09-16 preceded a hard host down (no ARP, no ICMP from
+two subnets, power cycle required). Telling off from wedged afterwards: `ip
+neigh show <ip>` from a peer — `FAILED`/`INCOMPLETE` = off/NIC dead; a wedged
+kernel usually still answers ARP. Validate your own probe's naming before
+reporting an outage (`gitea` vs `gitea-server`).
+
+**Fixes, ranked:**
+
+1. CPU-attached M.2 → PCIe x4 riser (~$30) — puts the card on the CPU root
+   complex; x4 suffices for TP.
+2. Board whose CPU x16 bifurcates x8/x8 (done 2026-09-20 on dankserver — both
+   R9700s on root ports `00:01.1`/`00:01.3`, every bridge `Routing+`, both
+   endpoints `ReqEn+`).
+3. llama.cpp `--split-mode layer` — no collectives at all ([[Llama Cpp Multi-Device Splits]]); costs the GGUF conversion plus paged attention /
+   continuous batching / spec decode.
+4. Independent TP=1 replicas + routing proxy — cards never share one model.
+
+**Closed software routes (do not re-litigate):** vLLM's IPC custom allreduce
+is gated by a CUDA-sm-keyed table (`{9.0, 10.0, 10.3, 10.7}`; ROCm reports
+12.0 → off on RDNA4 by code limit) — and patching it is insufficient because
+TP init's `broadcast`/`barrier`/logits `all_gather` have no non-RCCL path. A
+fork wrapping `.all_reduce` (e.g. radiance `RADIANCE_USE_R4D_AR` one-shot
+P2P, gloo for IPC-handle exchange) does not intercept
+`PyNcclCommunicator.__init__`'s warmup allreduce. NCCL symmetric memory needs
+`ncclDevComm*` (many builds ship the version string without the device API).
+P/D disaggregation needs a KV connector AND coordination collectives. gloo is
+CPU/TCP — unreachable for per-layer TP.
+
+**Open, unverified:** rebuilding RCCL so kernels stop declaring a hostcall
+buffer (`-mprintf-kind=buffered` is the usual claim) — counter-evidence: no
+live `printf`/`assert` in RCCL device source. Test path: rebuild, dump a
+kernel's declared args (`roc-obj-extract` + `llvm-readelf --notes`), look for
+`hidden_hostcall_buffer`; if gone, retry the 2-rank probe (service stopped).
+
+## Related
+
+- [[Llama Cpp Multi-Device Splits]] — the layer-split path around the wall
+- [[Dual-R9700-Cooperation]] / [[x4-pair-experiment]] — measured history
+- [[dankserver-qwen38-fp8-expert-offload-tuning]] — the live gather economics
@@ -0,0 +1,250 @@
+---
+type: reference
+title: RDNA4 Container Serving
+description: Bringing a model larger than VRAM onto Dankserver's 2x R9700 (gfx1201) with the tcclaviger vLLM stack — quant-class gating, fetch verification, container GPU-init triage, memory budgeting and the expert-offload plan.
+tags: [rdna4, gfx1201, r9700, vllm, expert-offload, podman, rocm]
+timestamp: 2026-10-05T00:00:00-07:00
+---
+
+# RDNA4 Container Serving
+
+Bring-up path for Dankserver (2x Radeon AI PRO R9700, gfx1201, 31.86 GiB each =
+**63.7 GiB real VRAM**, 246 GiB RAM, 7700X) with
+`docker.io/tcclaviger/vllm` (vLLM 0.29.0.dev, GLM/Qwen-specialised RDNA4 build
+with native MoE quant kernels and a first-class expert offload planner). The
+tuning history and current deployed shape live in
+[[dankserver-qwen38-fp8-expert-offload-tuning]]; this page is the bring-up law.
+
+## 0. Know the box — and its phantom third GPU
+
+`torch.cuda.device_count()` returns **3**; summing `total_memory` overstates
+the box as 186.7 GiB. The third device is the 7700X's integrated Vega
+(gfx1036): torch reports ~123 GiB for it, `rocm-smi` reports 0.50 GiB,
+allocating on it moves system `MemAvailable` 1:1 (it is GTT over DDR5, not
+dedicated VRAM), and it dies with `hipErrorInvalidKernelFile` when the image
+ships no code for that arch. **Never sum device `total_memory`; never count
+the iGPU as headroom or a workload.**
+
+## 1. Preflight: is the quant class EXECUTABLE, not merely present?
+
+Expert offload is mandatory when weights exceed VRAM, and it requires the
+checkpoint's experts class to have BOTH a compiled kernel AND an
+`offload_recipe` ("a pinned expert has no host copy, so there is no
+read-through fallback" — startup aborts otherwise). Check BEFORE downloading
+anything:
+
+```bash
+podman run --rm --device /dev/kfd --device /dev/dri --device /dev/mem \
+  -e HIP_VISIBLE_DEVICES=0,1 --entrypoint bash $IMAGE -c '
+  nm -D /app/r4dhip/r4d.so | grep -oE "gemm_moe_[a-z0-9_]+" | sort -u
+  grep -rl offload_recipe /opt/vllm/lib/python*/site-packages/vllm/model_executor/layers/fused_moe/'
+```
+
+| quant | kernel in `r4d.so` | experts class | verdict |
+|---|---|---|---|
+| FP8 | yes | `Fp8HipMoEExperts` | **serves** |
+| MXFP4 / DSFP4 | yes (`gemm_moe_mxfp4a8_*` / `dsfp4a8_*`) | `R4dMxfp4MoEExperts` / `R4dDsfp4MoEExperts` | **serves** |
+| W4A16 | **no** `gemm_moe_w4a16_*` | `R4dW4a16MoEExperts` exists | silently falls back to generic `TritonWNA16Experts`, no recipe → startup halts |
+| RFA | none | no class at all | out |
+
+Also confirm the architecture routes natively with NO overlay
+(`ModelRegistry.get_supported_archs()` for `Qwen4Exp`/`Glm5`). The stock image
+natively carries Qwen3.8/Qwen4Exp and GLM-5.x including MTP heads. **Do not
+apply vendor overlays** against a fast-moving pre-release image — the breakage
+misleads (a missing symbol in the overlay's OWN copy of a module masks the
+real cause, then moves to another symbol each patch;
+`cannot import name 'checkpoint_has_lm_head'` = overlay clobbered
+`transformers_utils/config.py`). The image ships quantizers for
+`rfa|rfi|mxfp4|mxfp4ocp|paro5` under `/app/tools/quantize.py`
+(`--fp8-attn --quant-only --dry-run` first) if a re-quant is needed — note it
+can run MXFP4/DSFP4/FP8 but only PRODUCE rfa/rfi/mxfp4/paro.
+
+## 2. Fetch with verification, always
+
+Gated repos: token never in argv (`pgrep -a curl` prints command lines).
+Build a mode-600 curl config once:
+
+```bash
+{ printf 'header = "Authorization: Bearer '; cat ~/.cache/huggingface/token; printf '"\n'; } \
+  > ~/.cache/huggingface/curl.conf && chmod 600 ~/.cache/huggingface/curl.conf
+curl -K ~/.cache/huggingface/curl.conf -f -sS -L -C - -o "$out" "$url"
+```
+
+`--fail` is MANDATORY — `curl -o FILE URL` exits 0 on HTTP 401/403 and writes
+the error body as a "successful artifact" (a gated repo once yielded three
+153-byte shards with identical sha256s and a cheerful DONE banner). `-C -`
+resumes torn shards; HTTP/2 cancels leave silently truncated files, so
+re-verify EVERY shard's byte count against the Hub manifest (and the LFS
+sha256) before serving. The AGENTS.md Secret Rules carry the token-hygiene
+law; [[frank-medium-model-swap]] has the same detail for GGUFs and
+`.partNofM` fragments.
+
+## 3. Serve — the known-good invocation and its traps
+
+```bash
+sudo sh -c 'sync; echo 3 > /proc/sys/vm/drop_caches'
+podman run -d --name <ctr> --ipc=host --ulimit memlock=-1:-1 \
+  --cap-add SYS_PTRACE \
+  --device /dev/kfd --device /dev/dri --device /dev/mem \
+  -e HIP_VISIBLE_DEVICES=0,1 -e VLLM_ROCM_USE_AITER=0 \
+  -v /home/dank/models/<model>:/models:ro \
+  $IMAGE /models --served-model-name <name> --port 7770 \
+  --tensor-parallel-size 2 --max-model-len 65536 \
+  --kv-cache-memory-bytes 2147483648 \
+  --max-num-seqs 4 --gpu-memory-utilization 0.95 \
+  --enable-prefix-caching --trust-remote-code
+```
+
+- **Entrypoint already execs `vllm serve "$@"`** — pass the model path as the
+  first argument and do NOT pass `serve` (podman then tries to pull `serve`
+  as an image name). `-e VAR=1` must come BEFORE the image name.
+- **Mount the whole `/dev/dri`, never individual render nodes** (§4).
+- `--ipc=host` and `--shm-size` are mutually exclusive in podman.
+- `podman run -d` without `-a` sends container stdout nowhere — read
+  `podman logs <ctr>`; and vLLM writes `Application startup complete` to
+  **stderr**, so a `podman logs X 2>/dev/null | grep -q` readiness gate never
+  fires.
+- Start at 65536, not the native ceiling; prove the plan, then scale.
+- Load takes ~5 min with `/health` refusing the whole time — do not kill the
+  process (deploy.sh waits 20 min).
+
+## 4. Container GPU-init triage (ordered)
+
+1. **Enumerate-vs-context trap (most common):** individual
+   `renderD*` nodes give HIP the render nodes WITHOUT the primary `card*`
+   nodes — it enumerates agents (`device_count()` returns N) and then refuses
+   every context: `RuntimeError: No CUDA GPUs are available` on the first
+   real op. A/B-confirmed 2026-09-19. Rule: `--device /dev/kfd --device
+   /dev/dri --device /dev/mem`; restrict GPUs with `HIP_VISIBLE_DEVICES`,
+   never with device nodes.
+2. **`device_count()` is not evidence.** Prove usability with a real op
+   (`(a@a).sum().item()` on a fresh `torch.randn`), then repeat via
+   `mp.spawn`, `mp.fork`, and `subprocess.run([sys.executable, ...])`, and
+   cross-check `rocm-smi --showmeminfo vram`.
+3. **memlock rlimit → a MISLEADING CUDA OOM.** Host default 8192 KB; a pin
+   larger than the limit fails as
+   `torch.AcceleratorError: CUDA error: out of memory` raised from
+   `pin_memory()` — it names the accelerator but the fault is host pinned
+   memory. It cannot be raised from inside a container or even from root's
+   own shell (`Operation not permitted`); only PAM applies it at login, so
+   fixes only affect NEW sessions:
+   `/etc/security/limits.d/99-memlock.conf` with
+   `* - memlock unlimited` + `root - memlock unlimited`; the container needs
+   `--ulimit memlock=-1:-1` to inherit. Discriminating test after the fix:
+   the container reports `unlimited` AND small pins succeed; if pins still
+   fail, memlock was never the cause.
+4. **`/proc/meminfo` TAB trap:** the key/value separator is a TAB, so
+   `grep MemAvailable /proc/meminfo | tr -s " " | cut -d" " -f2` returns
+   EMPTY and a following awk prints `0.0` — a false "host out of memory" on a
+   host with hundreds of GiB free, which invites aborting something healthy.
+   Tab-safe: `awk -v k='MemAvailable:' '$1==k {printf "%.1f", $2/1048576;
+   exit}' /proc/meminfo`, cross-check `<= MemTotal`, refuse to print a zero.
+   Use `scripts/dankstatus.sh` instead of hand-rolling.
+5. **Silent-parse traps:** `grep -c PATTERN file || echo 0` yields TWO lines
+   (grep prints 0 AND exits 1) — `[: 0\n0: integer expected`; use
+   `| head -1` or `|| true`. `systemctl is-enabled` can emit multiple lines.
+   `rocm-smi --showmeminfo vram --csv` column order is `device, VRAM Total
+   Memory (B), VRAM Total Used Memory (B)` — reading $3 as total/$4 as used
+   yields 0.00; filter to `^card[0-9]+$`.
+6. **Guard the box:** long loads with co-tenants can drive RAM to 0 — abort
+   before the OOM killer (`MemAvailable < 4 GiB → podman rm -f`, exit 1).
+7. **Drop page cache before a large load** (`echo 3 > /proc/sys/vm/drop_caches`)
+   and confirm the drop was actually in effect for the run before treating a
+   failure as a capacity wall — a stale "it doesn't fit" from a contaminated
+   run is expensive to unwind.
+
+## 5. Memory budgeting: the levers are not where they look
+
+The planner builds `budget = total × util`, subtracts a KV reserve, and gives
+the REMAINDER to the expert arena. Consequences, all measured:
+
+- **`--gpu-memory-utilization` grows the ARENA, not the KV pool** (KV stayed
+  0.64 GiB at util 0.95 and 0.98). The arena is the only elastic term.
+- **Lowering `--max-model-len` is the WRONG lever and backfires:** a smaller
+  context shrinks the reserve, handing the space to the arena — measured 65536
+  → 0.64 GiB KV, 32768 → 0.34 GiB KV.
+- **Hybrid-attention undercount:** the planner's per-token KV figure (9190
+  B/token for Qwen4Exp) misses the large fixed per-sequence Mamba state
+  (~2x), so the reserve is always short and the arena silently covers it.
+  Fix by pinning the pool: `--kv-cache-memory-bytes <BYTES>` (2 GiB →
+  113,595 tokens / 3.47x concurrency at 32k). For long context,
+  `--kv-offloading-size` (GiB) spills KV to host.
+- **`--expert-offload-mem` caps the HOST side, not the VRAM arena**
+  (`cap = int(x * GIB)` against `host_memory_available()`). Do not reach for
+  it to fix a VRAM problem; to grow the arena, shrink the KV pin or raise
+  util (see [[dankserver-qwen38-fp8-expert-offload-tuning]] for the
+  util=1.0 unbootability law).
+
+**Read the plan — it is the real acceptance test:**
+
+```
+expert offload: AUTO-ENABLED, weights 62.54 GiB per rank exceed the 30.27 GiB budget (TP2)
+  budget (total x util)      30.27 GiB
+  non-expert weights          6.28 GiB
+  KV reserve                  0.36 GiB   <- sanity-check against the model's own per-token figure
+  CUDA graphs                 0.66 GiB
+  runtime overhead            7.60 GiB
+  expert cache space         15.36 GiB   6712 slots of 2.344 MiB
+  host-backed experts/rank   52.74 GiB   23040 experts
+  host: 105.48 GiB across 2 ranks of 160.89 GiB usable
+```
+
+Host fits when `host_total < usable`. The PLE n-gram table rides memfd and is
+SHARED across TP ranks.
+
+**Loading mechanism** (for reasoning about peak RAM): weights never materialise
+in VRAM — expert Parameters start as CPU staging tensors; each layer's
+`weight_loader` counts shard events and, on layer completion,
+`ExpertTopology.split_layer` moves that layer's experts into the VRAM arena
+and compacts the rest into exact-size pinned host tensors behind UVA. Peak
+host RAM = `host-backed set + layers still loading`, not the whole model. At
+runtime `apply_hot_cold` sends arena-resident experts down a hot GEMM and the
+rest straight over PCIe via UVA views — no staging copy in the critical path.
+
+**Emulation check:** a dequantize-to-compute path doubles real host RAM (118
+GiB offload request → ~238 GiB) and is invisible in the weight count. Read the
+log for `Using 'EMULATION' Mxfp4 MoE backend` before concluding "doesn't fit".
+
+## 6. Performance profile and honest measurement
+
+Decode on this pair is **PCIe-gather-bound, context-independent**: 12.11
+tok/s FLAT from 304 to 16,095 context tokens — KV attention is not the cost.
+With ~29 % of experts arena-resident, ~0.84 GB crosses PCIe per token at the
+engine's effective read rate; cold-vs-warm differing ~10 % confirms the LRU
+works; growth is linear in hit rate, and only speculative decode cuts
+sequential STEPS rather than making each cheaper.
+
+**Benchmark traps that produced fake numbers here:**
+
+- `max_tokens=1` is not TTFT (the server can satisfy it without doing the
+  work — prefill looked flat at 1.26 s for 16k AND 30k prompts). A 64-token
+  non-streamed reply is time-to-64-tokens (decode-bound, flat ~9.6 s at every
+  length). Correct TTFT: stream and time the FIRST content-bearing chunk.
+- A small repeating word loop is prefix-cacheable AND n-gram-predictable — a
+  43-word cycle reported 22,897 tok/s prefill, almost entirely cache hit. Use
+  a large vocabulary in shuffled order and print `cached_tokens`.
+- ~1.2 s fixed per-request overhead swamps prefill below ~16k tokens; compare
+  prefill at 30k+ only.
+- Size prompts with `/v1/messages/count_tokens` (a chat call for counting
+  pollutes the metric); guessing words-per-token silently overran the window
+  (HTTP 400).
+- Always run a canary: plant a unique token at the END of a long prompt and
+  ask the model to quote it — proves prompts reach the model at all.
+- Measure warm AND long (`podman exec` + urllib at 127.0.0.1:7770): a 33-token
+  reply read 14.6 cold / 20.3 warm; a 768-token generation hit 25.7; two
+  concurrent streams scaled to 47 aggregate.
+
+**Levers (all opt-in):** `--expert-prefetch N` takes an INT depth, not a
+boolean (default 0; at depth 1 only ~10 % cold-vs-warm difference — the cache
+already works); `CLAV_TUNABLEOP_SWEEP=1` autotunes GEMM shapes (silently
+skipped under expert offload unless `CLAV_TUNABLEOP_WITH_OFFLOAD=1` too —
+see [[dankserver-qwen38-fp8-expert-offload-tuning]] for why it was rejected);
+`--expert-precise-memory` swaps the fitted overhead figure in.
+
+## Related
+
+- [[dankserver-qwen38-fp8-expert-offload-tuning]] — deployed shape, plan law,
+  measured campaigns
+- [[PCIe Link and Atomics]] — why decode is gather-bound and what cannot fix it
+- [[RDNA4 MoE Expert Offload VRAM Budget]] — arena/plan arithmetic deep-dive
+- [[frank-large-benchmark]] and [[frank-small-benchmark]] — window discipline on live hosts
@@ -0,0 +1,219 @@
+---
+type: reference
+title: RDNA4 MoE Expert Offload VRAM Budget
+description: The expert-offload plan arithmetic on the tcclaviger fork — which terms a deploy can move, the util law, the CUDA-graph charge, what free VRAM really is, knife-edge survivability, parallelism ceilings and the context ladder.
+tags: [vllm, expert-offload, vram, arena, kv-cache, ladder, tcclaviger]
+timestamp: 2026-10-05T00:00:00-07:00
+---
+
+# RDNA4 MoE Expert Offload VRAM Budget
+
+For VRAM-bound vLLM on the tcclaviger RDNA4 fork (frank-primary: Qwen3.8-
+Flash-Next FP8, TP=2, 2x R9700 at 31.86 GiB each, expert offload to host RAM).
+Tuning history and the deployed shape live in
+[[dankserver-qwen38-fp8-expert-offload-tuning]]; boot-failure capture lives in
+[[vLLM Boot Forensics and Config Interrogation]].
+
+## 1. The plan law
+
+```
+arena = budget − non_expert − kv_reserve − cuda_graphs − runtime_overhead
+budget = device_total × gpu_memory_utilization          # expert_plan.py build_plan
+slots  = arena // expert_bytes                          # 2.344 MiB/slot ⇒ 437 slots/GiB
+```
+
+**The arena is DEVICE-resident, so the plan predicts reality:** the fp8
+3-window shape's plan totalled 28.27 GiB and `rocm-smi` measured exactly
+28.27 GiB. `weights + pinned_pool + graphs + arena + non-torch (≈1.45 + 0.57
+driver reserve)` vs 31.86 GiB predicts a boot failure before it happens.
+
+Which terms a deploy can move:
+
+| term | reachable? | how |
+|---|---|---|
+| budget (util) | **yes, law-capped** | `--gpu-memory-utilization`; +0.01 = +0.32 GiB on 31.86 GiB cards |
+| CUDA graphs | **yes** | container env `VLLM_CUDAGRAPH_MEM_BASE_MB` / `_PER_GRAPH_MB` (or `_ESTIMATE_MB` for an explicit total) |
+| non-expert weights | no | checkpoint census |
+| KV reserve | no | registry constant: 9190 B/token × max_model_len × 1.08 + per-seq mamba pages |
+| runtime overhead | partly | `--expert-precise-memory` swaps default 7.60 GiB for the registry-fitted 5.25 GiB |
+
+Anything else is an IMAGE EDIT, not a deploy — say so; do not invent a flag
+(verify with `podman exec <ctr> grep -n getenv …/offloader/expert_activation_registry.py`).
+
+**UTIL LAW: 0.982, NEVER 1.00 (measured 2026-10-03).** The CLI field is
+`Field(gt=0, le=1)`, but the worker init demands `free > budget`: at 1.0
+budget == device total and BOTH ranks raise
+`ValueError: Free memory on device cuda:N (31.29/31.86 GiB …)` BEFORE a
+single weight loads — a crash loop (NRestarts ~2 min apart, zero payload
+progress). Not a knife-edge: **unbootable**. Even bootable near-ceiling
+shapes die under real load (0.13 GiB free bf16 boot; `0 bytes free` on a real
+995-token request).
+
+## 2. The CUDA-graph term: charge the measured capture
+
+Defaults (256 MiB base + 52 MiB/graph) over-charge badly. Measure with the
+engine's own post-capture line `Graph capturing finished … took X GiB` (per
+rank; heavier rank binds). Measured on the no-spec piecewise plan: 0.31/0.17
+GiB, and one boot reported a NEGATIVE −0.06 GiB delta (capture fit inside
+pages the profiler had already counted) — the analytic base pool is pure
+over-charge. Shipped: `BASE_MB=0`, `PER_GRAPH_MB=12` = 0.07 GiB — per-graph,
+so a spec-decode shape (18 graphs) still pays (0.21 GiB). An explicit
+`_ESTIMATE_MB` total does NOT scale with ladder changes; base+per-graph does.
+
+**Verify every deploy:** `Estimated CUDA graph memory` line vs the `Graph
+capturing finished` line; if capture exceeds the charge, raise `_PER_GRAPH_MB`
+back toward 52. Check `VLLM_CUDAGRAPH_*` actually present in
+`podman exec … env`.
+
+## 3. The pool is invisible to the plan; free VRAM is the estimator gap
+
+`--kv-cache-memory-bytes` does NOT appear in the arena arithmetic — the plan
+reserves the registry price, not the actual pool. Raising the pool consumes
+real free VRAM without growing the arena; the registry over-prices the pool
+(9190 B/token planned vs 7,891.8 measured at 200704 fp8 — 7,701 at 262144;
+**never reuse a B/token ratio across geometries**, and subtract ~3.87 MB
+block-alignment loss when converting a token target to bytes). The ask lands
+block-aligned: 3,166,500,000 B → 401,408 tokens; verify with
+`GPU KV cache size: N tokens, Maximum concurrency … X.Xx`.
+
+**Driver-reported free VRAM after hours of traffic is NOT allocatable
+arena.** It is the estimator gap: the registry over-prices KV and charges
+5.25 GiB overhead against ~2.5 measured; torch's allocator does not return
+slabs, so it reads as the high-water mark. The only reachable levers are util
+(within the law) and the graph charge; everything else needs a registry edit
+inside the image — the fork's own log line says as much.
+
+**MNBT is arena-free up to the anchor:** registry overhead =
+`base + 0.22 GiB × max(0, (nbt−4096)/1024)` anchored at nbt 4096, so rungs
+1024/2048/4096 are plan-identical (the fitted 5.25 GiB was fitted AT 4096).
+MNBT changes re-key the compile range (one recompile, ~24 s at (1,2048) with
+the mounted cache). 512 doubles TTFT at 4064 ctx; 12288 costs ~2.5 % plus
+arena. MNBT does not move the KV pool.
+
+## 4. Registry "compile headroom" is not reclaimable slack
+
+`--expert-precise-memory` selects the fitted overhead (5.25 GiB, this arch at
+TP=2) over the default (7.60 GiB = fitted + **2.35 GiB of compile and
+graph-capture headroom** the registry documents as "transient VRAM the
+profiler never sees"). It moves exactly one term — the plan's `room` — so the
+arena grows by that 2.35 GiB (~1,025 slots). **That headroom is the budget
+for the phase that allocates LAST**: reclaiming it requires paying elsewhere
+(smaller pinned pool, fewer sessions, shorter context). Measured 2026-10-01:
+bf16 2×262,144 + precise put the total at 31.73 of 31.86 GiB and the boot
+crash-looped at graph capture; dropping the flag restored ~2.4 GiB and booted.
+
+## 5. Boot ≠ survival (knife-edge shapes)
+
+A shape can pass EVERY boot gate — weights, profile, KV allocation, graph
+capture, `/health` 200 — and die on the first real request. Measured
+2026-10-01: `5 × 262144 @ fp8` booted (KV 1,350,182 tokens / 5.15x) and was
+killed seconds later by a 995-token prompt with `max_tokens=261149`:
+`torch.OutOfMemoryError: Tried to allocate 50.00 MiB … 0 bytes free`. After
+any knife-edge change: send a LARGE-`max_tokens` request before calling it
+done, and read post-boot free VRAM per rank (`rocm-smi --showmeminfo vram`)
+— the survivor had ~3.6 GiB free; the corpse had 0.
+
+**KV accounting — pin the pool BYTES, floor-sized:** set
+`--kv-cache-memory-bytes` PER RANK to the contract floor
+(`seqs × MAXLEN × measured B/token`), not the native window — extra pool bytes
+come out of the arena/headroom and buy nothing. Measured B/token on this
+hybrid: **7,701 fp8 / 13,839 bf16** (decomposed: attention 6,138/12,276 +
+dtype-independent mamba/side 1,563). Measured rates are shape-dependent —
+7,891.8 at the 200704×3 shape. Capacity does NOT scale with `SEQS`: the
+planner's KV reserve is for ONE max-model-len request.
+
+**Parallelism ceilings you cannot flag past:**
+
+- `--decode-context-parallel-size` refused for GQA when `TP ≤ num_kv_heads`:
+  *"requires TP (2) to be greater than the model's total number of KV heads
+  (2)"* — DCP shards KV along the sequence dimension AFTER KV heads spread
+  across TP ranks (it reuses TP ranks, not TP×DCP), so a 2-KV-head model
+  needs TP ≥ 4 = four GPUs.
+- EP (`--enable-expert-parallel` + `--all2all-backend`) exists but REPLACES
+  the fork's local tiered-offload gather with a per-layer PCIe all-to-all.
+- DP requires a full model copy per replica — on a model larger than the
+  card it is impossible, not slow.
+
+## 6. The context ladder (find the max bootable window by measurement)
+
+"Max possible" = the largest value that BOOTS, measured — never the model's
+spec maximum, never hand arithmetic.
+
+1. **Scale the pool with the window.** If KV_BYTES is pinned, it must shrink
+   in lockstep with MAXLEN or the rung tests nothing:
+   `KV_BYTES = round(KV_BYTES_ref × (seqs×MAXLEN_new) / (seqs×MAXLEN_ref))`
+   off a booted reference rung. Verify the rung's capacity still covers
+   `seqs × MAXLEN` from the engine's own line.
+2. **Boot under the PRODUCTION environment, not a shell probe** — a rootless
+   shell lacks `LimitMEMLOCK=infinity` and can pass where production OOMs.
+   Transient unit:
+   `sudo systemd-run --uid=<svcuser> --unit=probe-$ML -p LimitMEMLOCK=infinity -p Restart=no env <ENVFILE_OVERRIDE> <serve-script>`
+   pointing the serve script at a scratch env file via its `*_ENV_FILE` hook.
+3. **Judge on health + unit liveness:** poll `/health` (200 up / 503 loading
+   / 000 not bound); break early when the unit goes inactive — that is the
+   failure signal, not a timeout.
+4. **Capture the engine log per rung BEFORE teardown** (`podman logs` dies
+   with `podman rm`; a `podman logs` issued too soon after `podman run -d`
+   fails with *no container with that name* — record UNKNOWN, never FAIL).
+5. **Ladder down on FAIL, one bonus step UP on PASS.** Measured steps on this
+   box: OOM −16,384 tokens, bonus +4,096. Tiny margins (76 MiB class) mean
+   the next rung down almost certainly passes: `Tried to allocate 76.00 MiB …
+   0 bytes free … 30.43 GiB allocated by PyTorch` died in graph CAPTION;
+   `Tried to allocate 10.28 GiB … 9.46 GiB free` died in KV ALLOCATION.
+6. **Clean between rungs:** stop the transient unit, `podman rm -f`, kill
+   orphans by explicit PID
+   (`ps aux --sort=-rss | awk '/vllm\/bin\/python3/ && $6 > 50000000 {print $2}'`),
+   wait for `MemAvailable` to recover.
+
+**The hazard that cost production:** a leftover ladder script whose teardown
+`podman rm -f`s the production container fires against a freshly DEPLOYED
+engine. Teardown runs ONLY on failure, never after a pass; kill the whole
+scratch process tree and verify none remain BEFORE deploying. On PASS: print
+the measured `GPU KV cache size / Maximum concurrency` line, LEAVE IT
+SERVING, and pin the number into the repo (env + seam + harness
+contextWindow) before deploying.
+
+Cutover order: ladder → winner passes and serves → repo env + seam + harness
+→ tests → commit+push → verify no scratch scripts remain → stash foreign work
+(`git stash -u`) if the tree is dirty → deploy (one restart) → verify health
+ engine flags in the log.
+
+**Measured ladder on this box (200704×3, fp8 KV, precise memory):**
+
+| shape | util | graphs | arena | result |
+|---|---|---|---|---|
+| E | 0.98 | 0.55 | 17.22 GiB / 7,522 slots | boot ok |
+| F | 0.982 | 0.32 | 17.52 GiB / 7,653 slots | boot ok |
+| 1.0 | 1.00 | 0.07 | — | UNBOOTABLE (worker-init ValueError, both ranks) |
+| G (deployed) | 0.982 | 0.07 | 17.77 GiB / 7,762 slots | KV 401,408 tokens / 2.00x; 0.89 GiB real free/card held through a live 4,096-token generation at ~38–39 tok/s |
+
+bf16 ceilings at util 0.98: 3 seqs → 241,664 passes / 245,760 and 262,144 OOM
+at graph capture; 6 seqs fp8 → 10.70 GiB ask with 9.18 GiB free (fails) —
+~207k per context max.
+
+## 7. Arena audit procedure (when asked for "more experts")
+
+1. Read the live plan before proposing anything — `grep -oE` with a
+   digit-anchored tail (plain grep/journalctl truncate these columns to
+   `17.`):
+
+   ```
+   ssh dankserver 'sudo journalctl CONTAINER_NAME=frank-primary-qwen -b -o cat --no-pager \
+     | grep -aoE "(budget \(total x util\) +[0-9.]+|expert cache space +[0-9.]+ GiB +[0-9,]+ slots|Estimated CUDA graph memory.{0,80}|Graph capturing finished.{0,40}|GPU KV cache size.{0,90})"'
+   ```
+
+2. Classify each term (§1 table); take only the two reachable levers.
+3. Verify after deploy: plan lines show new budget/graphs/arena;
+   `GPU KV cache size` matches the ask; graph-estimate vs post-capture;
+   `rocm-smi` high-water per card = the real margin;
+   `systemctl show frank-primary -p NRestarts --value` = 0; a real chat
+   completion from a PEER (health-probe via `192.168.0.235:7770/health`, not
+   `localhost` inside an ssh session on the host — the latter returns `000`
+   and looks like an outage).
+
+## Related
+
+- [[RDNA4 Container Serving]] — the bring-up path and container traps
+- [[vLLM Boot Forensics and Config Interrogation]] — capture and contracts
+- [[dankserver-qwen38-fp8-expert-offload-tuning]] — measured history
+- [[Llama Cpp Serving Sizing]] — the llama.cpp equivalent
@@ -237,6 +237,46 @@ vLLM supports multiple parallelism strategies:
 - Expert Parallelism (EP) distributes expert networks in MoE models
 - Context Parallelism (CP) splits context across GPUs

+
+## Chunked Prefill — the flag surface, and what it actually does
+
+**YES, the runtime exposes chunked-prefill options, and the running shape
+already uses them.** The tcclaviger fork is a vLLM fork, so the standard
+scheduler flags are present in the shipped image (verified by reading the
+`vllm.engine.arg_utils` and `vllm.config.scheduler` source in the live
+container `frank-primary-qwen`, not from a stub):
+
+| Flag | Default | What it controls |
+|---|---|---|
+| `--enable-chunked-prefill` | `True` in `SchedulerConfig`; the CLI arg is an auto-flag (`"default": None`) | whether prefill may be split into chunks |
+| `--max-num-batched-tokens` (MNBT) | 2048 (`DEFAULT_MAX_NUM_BATCHED_TOKENS`) | tokens processed per prefill iteration |
+| `--prefill-schedule-interval` | 0 | gap between prefill iterations |
+
+The running shape's argv carries `--max-num-batched-tokens 2048` alongside
+`--max-model-len 200704`, so chunked prefill is **ON and already active**: a
+200704-token prompt is prefilled in ~98 iterations of 2048 tokens, not in one
+shot.
+
+**It is a TTFT lever, not a throughput lever.** MEASURED 2026-09-29 on the
+ad hoc qwen-fp8 container (the same fork, same payload, same TP=2):
+
+- MNBT 8192 vs the default 2048: **neutral** on decode at every depth.
+- Small values re-chunk prefill only. They hurt TTFT (each chunk is its own
+  iteration, so the first token waits on more iterations) and they never touch
+  decode: a decode step is `max_num_seqs x (spec+1)` tokens, independent of
+  MNBT.
+
+**There is no `--ubatch-size` equivalent in vLLM.** `--ubatch-size` is a
+llama.cpp compute-buffer concept (`--batch-size`/`--ubatch-size`); the tcclaviger
+fork has no ubatch knob at all. The llama.cpp compute-buffer pair that frank-small
+was doubled on (2026-10-05, +4.6 % prefill at 8,928 tokens for 616 MiB) has no
+counterpart here, and the two backends should not be reasoned about by analogy.
+
+**MNBT is not a lever on this shape either.** The deployed 2048 is the fork's
+default, so a change is a deliberate departure from the shipped value, not a
+tune. The one thing MNBT does cost: every change re-enters the compile RANGE
+path, so `FRANK_PRIMARY_CACHE_DIR` pays one recompilation.
+
 ## See Also

 - [[VLLM Index]]
@@ -168,11 +168,46 @@ Not advisable.
 | **Affinity** | yes, built here | **V4-Flash only** — "the geometry is compiled in; there is no architecture abstraction" | 110 tok/s for 284B. Best engine for this box by an order of magnitude, wrong model |
 | **tcclaviger vLLM fork** | **yes — proven 2026-09-29** | qwen4exp measured 37.4-40.6 tok/s decode (see [[dankserver-qwen38-fp8-expert-offload-tuning]]); **`glm5next` registered** (`Glm5NextForCausalLM`/`MTP`, backend-split `amd/`/`common/`/`nvidia/`, `sparse_indexer.py` for DSA); r4d.so has **native mxfp4 + dsfp4 MoE entries** (`has_dsfp4() == True`); W4A16 MoE class (`R4dW4a16MoEExperts`) wired into the same tiered offload; TunableOp silently skipped under offload (needs `CLAV_TUNABLEOP_WITH_OFFLOAD=1`, and then corrupts numerics — do not use) | the ad hoc qwen-fp8 container's runtime; **the old "MXFP4 = emulation/missing kernels" verdict predates this fork and is dead** — the GLM-5.3-Flash gate is now the payload format + a boot probe, not kernels |
 | **Lucebox** | yes, image on host | Qwen 3.8, DS4; **V4.1 on the unmerged `feat/ds41-engine` branch** | The only V4.1 path. **Refuses layer split and layer-major prefill for V4.1** |
-| **radiance-vLLM** (drwolfen) | yes — the active frank-primary | welded to `qwen4_exp`; PLE / QSA / packed-HC live inside that model class | 12–20 person-days per architecture |
+| **radiance-vLLM** (drwolfen) | yes — served frank-primary 2026-09-27 → 09-30 | welded to `qwen4_exp`; PLE / QSA / packed-HC live inside that model class | 12–20 person-days per architecture |
 | **FreeToken-ROCm** | partial | block-FP8 safetensors only (modelopt); known `CpuMoeExecutor` issue under ROCm | cannot load a GGUF |
 | **llama.cpp** | yes | GLM-5.3 confirmed by a community R9700 guide; DS-V4.1 CED unverified | no expert LRU; **8 CPU cores** make CPU expert placement the wrong trade |
 | **TokenSpeed** | claimed | GLM-5.x incl. a 5.3-Flash recipe | NVIDIA-first, unverified on gfx1201 |
 | **KTransformers** | no | GLM-5.3 tutorial exists | built for Intel AMX + NVIDIA |
+| **radiance** (StillDeadcode — the standalone C++/HIP engine, NOT the vLLM fork of the same name) | image `stilldeadcode/radiance` prebuilt for gfx1201 | qwen35 dense/MoE at fp8, qwen4exp quantised-at-convert, llama-shaped | see the probe below: our FP8 payload is REFUSED; tiers, MTP and vision native |
+
+### radiance (standalone) — 2026-10-05 probe
+
+`codeberg.org/StillDeadcode/radiance` — Deadcode's from-scratch engine
+(v1.0.13): same author as the `vllm-radiance` fork we ran in September
+([[frank-medium-radiance-mxfp4]]), an unrelated codebase — C++/HIP, no
+Python, kernels/architectures/quantisers as `.so` plugins, flags written
+for two 32 GB R9700s, native Qwen3.8-Flash-Next with `--placement
+expert_tiered`, MTP and vision.
+
+Probe with the DEPLOYED payload (`rad-convert --plan-only`, which stops
+at declare — no GPU touched, no bytes written):
+
+```
+E no architecture plugin claims 'qwen4exp' at quantisation 'fp8'.
+  'qwen4exp' is served at: (unquantised).
+```
+
+So the 173 G orcarouter FP8 file cannot enter this engine at all: the
+qwen4exp plugin takes bf16/f16/f32/GGUF in and every quantisation happens
+at `rad-convert` time by recipe. Not a lock-in to their published quant —
+recipes are per-weight-class rules (`rtn`/`gptq`/`awq`/`autoround`/
+`paroquant`/`ggml`, grids from fp8 128×128 to i4/w4nl/w5nl codebooks,
+custom `--quant` rules go ahead of the file's), and libr4d lists routed
+experts-as-fp8 among the encodings it reads — so today's quantization is
+reproducible **from a bf16 source**, not from our file. Uncensored-family
+bf16 source candidates: `windowsxp811203/Qwen3.8-Flash-Next-Abliterated-GGUF`
+(ships the bf16 GGUF) and `mradermacher/Qwen3.8-Flash-Next-Uncensored-GGUF`
+(i1 quants only). 622 G free fits source + container. A trial is:
+download → `rad-convert` (CPU-side) → stop+mask frank-primary window to
+serve. Caveats on record: `--tp-wire wht6` is a lossy int8 all-reduce
+wire; the recurrent GDN state is f32 by default (144 MiB/sequence) with
+an opt-in f16 that changes generated text; every pool is budgeted
+explicitly — no profiling pass, no utilisation fraction.

 The two-R9700 three-owner split (primary + secondary + stream) is **not
 available on this host**: the retired quadlet recorded that
@@ -244,13 +244,481 @@ scripts `arm_*.sh` / `sweep_*.sh` and the detached runner `/home/dank/job.sh`
 are `podman cp`'d into the container after every recreate (the serve script
 re-copies them; a recreate wipes container-local files).

+## 2026-10-01 — production cutovers (qwen FP8+MTP4, then bf16 no-spec)
+
+Two same-day cutovers moved this page's findings into production:
+
+1. **Qwen FP8 + MTP4 replaced the failed GLM/DFlash2 experiment** (the GLM
+   window is documented in [[frank-primary-glm53-flash-w4a16]]; git tag
+   `frank-primary-fp8-mtp-20261001` is the last commit of that shape). The
+   production trio took MAXLEN 200000 x 3, fp8 KV, MTP4, prefetch off —
+   measured 37.4-40.6 tok/s at ~54% acceptance.
+2. **The overnight redo campaign re-measured EVERY lever at matched
+   geometry** (prefetch OFF everywhere, MAXLEN 200000/SEQS 3/fp8 baseline)
+   and concluded the day: the incumbent cutover to **bf16 KV, no
+   speculative decode** (Frank's call).
+
+### What the redo campaign measured (all at shallow depth, 166/1087/4064)
+
+| lever | verdict |
+|---|---|
+| prefetch (re-confirmed) | the whole night's +40-51% lives in deleting it; the old 12.11 "no-spec baseline" was itself prefetch-taxed |
+| gather geometry | **R4D_LRU_GATHER_CHUNKS=32 / LANES=64 = +18.7%** (49.25 @4064), the win GROWS with depth (39.5@166 -> 54.1@1087); g8x16 mild, g64x128 past peak |
+| KV dtype | bf16 (`auto`) +4% decode on fp8-era numbers but halves attention-KV capacity (measured 14,047 vs 7,926 B/token; mamba states unaffected) |
+| spec family | no-spec fastest once the tax was gone (41.46/41.29/41.47); MTP4's drafter reserves ~2.6 GiB/rank beside the pool; untaxed MTP re-measures never completed |
+| MNBT | 512 doubles TTFT (2.46 s vs 1.36 s @4064); 1024-4096 interchangeable; 12288 -2.5% + arena; **locked 1024** |
+| UVA | 2048 POISON (-39%); 8192 ~parity |
+| arena | flat ±1.4% from ~6,830 down to ~5,500 slots; knee below |
+| six slots | every SEQS-6 KV sizing refused at engine init |
+
+Depth caveat: every number above is at a near-empty KV. Deep-context
+behavior (the gather win growing with depth, bf16's decode cost at depth)
+is only sampled at 3 depths; a 64k-192k-depth probe is the flagged gap.
+
+### The deployed shape (2026-10-03 shape G: 3 slots x 200704, fp8, precise memory, MNBT 2048, arena at the flag ceiling)
+
+```
+FRANK_PRIMARY_MAXLEN=200704   200704-token window
+FRANK_PRIMARY_SEQS=3          --max-num-seqs 3 (SEQS=2 CANNOT boot - below)
+FRANK_PRIMARY_KV_BYTES=3166500000   PER RANK (fp8, 7,892 B/token -> 400,700 tokens)
+FRANK_PRIMARY_KV_DTYPE=fp8    --kv-cache-dtype fp8 (quantized)
+FRANK_PRIMARY_MNBT=2048       --max-num-batched-tokens 2048 (arena-free, see shape F)
+FRANK_PRIMARY_UTIL=0.982      budget = total x util, the law (1.0 unbootable, see below)
+FRANK_PRIMARY_CUDAGRAPH_BASE_MB=0 / _PER_GRAPH_MB=12  -e VLLM_CUDAGRAPH_MEM_*
+FRANK_PRIMARY_PRECISE_MEMORY=1      --expert-precise-memory (overhead 7.60 -> 5.25 GiB)
+FRANK_PRIMARY_EXPERT_OFFLOAD_MEM=120  --expert-offload-mem (host-region cap, explicit)
+FRANK_PRIMARY_GPU_MAX_HW_QUEUES=2   -e GPU_MAX_HW_QUEUES=2 (HSA queues/GPU)
+FRANK_PRIMARY_CACHE_DIR=/home/dank/.cache/frank-primary   mounted at /root/.cache
+R4D_LRU_GATHER_CHUNKS=32 / R4D_LRU_GATHER_LANES=64  (container env)
+NO --speculative-config, NO --expert-prefetch
+FRANK_PRIMARY_NO_PREFIX_CACHING=1  --no-enable-prefix-caching (2026-10-05, the QFN align-mode GDN state fault — below)
+```
+
+### 2026-10-05 — prefix caching OFF: the fork's QFN align-mode GDN state fault
+
+**Symptom (Frank, 2026-10-04):** output turned repetitive/corrupted at
+~40k tokens into a long conversation; short sessions stayed clean.
+
+**Evidence (live `podman logs frank-primary-qwen`, the 2026-10-04 18:19
+boot):** one 0.6 s burst at 18:46:07 PDT — 5.6 h into service, one
+running request, and the engine's own 18:46:08 log line reading
+"Prefix cache hit rate: 48.1%". 48 `gdn_hip` faults, then the
+process-lifetime silence cap:
+
+```
+[gdn_hip] ERROR gdn_decode_r: state out of ssm-cache range: max|S|=0
+  with non-finite values (slot=42 head=3 limit=3.40282e+38) fault #1-23
+[gdn_hip] ERROR gdn_decode_r: ... (slot=42 head=8 ...) fault #24-46
+[gdn_hip] ERROR: state fault report limit (32) reached; further faults
+  are counted silently
+```
+
+One physical SSM slot (42), two heads (3 and 8 — one per TP rank), 23
+GDN layers each, the state non-finite (NaN). The guard's own text in
+this build: *"known Qwen3.8-Flash-Next (QFN) flaw under prefix caching
+and is being actively worked; output quality for this request may
+degrade. Mitigation: --no-enable-prefix-caching."*
+
+**Mechanism.** With prefix caching on, the fork defaults QFN to
+`mamba_cache_mode=align` (boot log: "Mamba cache mode is set to 'align'
+for Qwen4ExpForConditionalGeneration … when prefix caching is enabled";
+`config/cache.py`: "none" is the no-prefix-cache mode, "all" is
+deprecated — align is the only sound mode WITH the flag). Align mode
+checkpoints each sequence's GDN (Gated DeltaNet) recurrent state into a
+mamba page per block and RE-SEEDS the state from that page when a
+request resumes from a cached prefix. The block size is raised to
+**3136 tokens** for the page-size equality (boot log: "Setting
+attention block size to 3136 tokens to ensure that attention page size
+is >= mamba page size" + "Padding mamba page size by 0.13% … exactly
+equal"). The corrupt state landed at the **13th boundary: 13 × 3136 =
+40,768 tokens** — the reported "~40k" mark. The NaN persisted in the
+mamba page, so every later turn of that conversation re-seeded the same
+non-finite state for those two heads (silently — the fault-report cap
+was already spent) while fresh short sessions looked fine.
+
+**RAM ruled out the same day** (the 2026-10-04 17:17 host reboot is when
+the current memory speed took effect): zero MCE/EDAC kernel records
+since boot; an 8 GiB in-band pattern check stable over 60 s under live
+load; 4800 MT/s is the JEDEC DDR5 floor on 5600 MT/s-rated DIMMs (not an
+overclock); and a two-head, single-slot NaN state is a state-path fault,
+not scattered finite bit corruption.
+
+**Fix.** `FRANK_PRIMARY_NO_PREFIX_CACHING=1` → `--no-enable-prefix-caching`
+(the fork-named mitigation; the flag renders in both directions from the
+knob so a fork default change cannot silently flip the contract). Cost:
+no prefix reuse — every turn re-prefills the whole history instead of at
+most 3135 tail tokens. A silently corrupted conversation is worse than a
+slower first token.
+
+**Verified 2026-10-05 (deploy b820d34):**
+
+- Engine argv carries `--no-enable-prefix-caching`; health 200; warm boot
+  `init engine … took 76.57 s` (the mounted compile cache is intact).
+- Boot plan unchanged, as predicted: expert cache **17.77 GiB / 7,762
+  slots** (the flag touches no plan term), CUDA-graph estimate 0.07 GiB
+  vs 0.08 GiB captured (inside the 12 MiB/graph charge).
+- `GPU KV cache size: 424,678 tokens` — **up** from 401,408 at the same
+  3,166,500,000 B ask: the pool now prices ~7,456 B/token (was 7,891.8),
+  the mamba/side per-token term dropping once the per-block checkpoint
+  pages are gone; the 3136 attention block-size raise nonetheless stays
+  (the page-size invariant is enforced in mode "none" too). Frank's
+  400,700-token floor is cleared with ~24k of headroom.
+- **Zero `gdn_hip` lines** in the boot-and-traffic window (only the two
+  INFO module banners mention the name).
+- Boundary probe (single request, no prefix reuse): prompt 39,280 +
+  completion 1,491 = **40,771 total tokens — across the 40,768 line**;
+  the task (write 1–400, one per line) completed coherently: 399 integers
+  parsed, 0.997 monotone, tail `… 398 399 400`. Pre-fix, this boundary is
+  exactly where the conversation degraded (the 13th 3136-page re-seed).
+- e2e: 32/33 pass; the one failure is the admin spec's
+  192.168.0.112:8989 leg with FrankBook-Pro asleep (environmental,
+  unrelated to this change; re-run `tests/e2e/run.sh` once the mini is up).
+
+**RESULT — REVERTED THE SAME DAY.** With every mechanical check green
+(zero faults, coherent 40,771-token probe), live long conversations
+STILL degraded at ~40k on the running endpoint (Frank, direct
+observation). The container serving that window (`--no-enable-prefix-
+caching`, booted 2026-10-05 12:47 PDT) logged ZERO `gdn_hip` faults
+while the corruption reproduced — so this recurrence is **fault-free**:
+the NaN-state guard did not fire, which the re-seed theory requires.
+Open hypotheses, in order: (1) the poisoned transcript is being re-fed —
+corrupted assistant turns already in the history keep producing
+corrupted continuations even with perfect state, so the same
+conversation cannot distinguish fixed from unfixed; (2) a second
+long-context mechanism with no guard coverage — fp8 KV degradation at
+depth is the next suspect (a bf16-KV boot costs ~2× the pool per token
+and would settle it). Prefix caching is **back ON**
+(`FRANK_PRIMARY_NO_PREFIX_CACHING` commented out — the cost bought no
+fix). The knob and the serve branch stay for a future fork build with a
+fixed GDN path. Next diagnostic: reproduce on a FRESH conversation; if
+it still breaks at ~40k with zero faults, A/B the KV dtype.
+
+**RESOLVED (2026-10-05 evening): it was the RAM.** Frank dropped the
+memory from 4800 to **4600 MT/s** (in effect since the 2026-10-05 12:21
+PDT host boot) and the corruption stopped: zero `gdn_hip` ERROR lines in
+the container since, zero MCE records, and the live agent session on
+this endpoint past 140k context, "comfy cozy" (Frank). Marginal DDR5 on
+four DIMMs is a common cause for BOTH eras: the 2026-10-04 NaN burst
+(corrupted weight/activation traffic poisoning a GDN state write — the
+guard-visible path) and the fault-free ~40k degradation (silent flips in
+KV/expert data, which no guard covers on non-ECC memory). The re-seed
+theory is therefore UNPROVEN, not disproven — the entire caching
+experiment ran on marginal RAM and was confounded on both arms.
+Post-mortem on the rule-out: consumer non-ECC RAM raises no MCE for a
+silent flip; a 60 s / 8 GiB pattern sample covers ~0.5 % of 246 GiB;
+"4800 is the JEDEC floor on 5600-rated DIMMs" is a rating, not a
+stability proof at four-DIMM loading. The revert test was the decisive
+instrument and it pointed at the RAM all along. Prefix caching stays ON
+(zero faults under heavy re-send traffic since the speed drop); the knob
+stays for a future fork fix. The fp8-KV A/B is NOT needed.
+
+### Shape F/G (2026-10-03): where the free VRAM actually is, and what forcing it cost
+
+`rocm-smi` showed **3.57 GiB free per R9700** at the deployed shape E (28.28 GiB
+used of 31.86 GiB, after hours of real traffic — so it is the high-water mark,
+not an idle artifact). Growing the expert arena into it is NOT a matter of
+raising `--gpu-memory-utilization`, because the arena is
+
+```
+arena = budget - non-expert - KV reserve - CUDA graphs - runtime overhead
+budget = device_total x gpu_memory_utilization          # expert_plan.py:567
+```
+
+and every term on the right is either fixed or a lie:
+
+| term | plan charges | measured reality | reachable from our side? |
+|---|---|---|---|
+| `budget` | 31.22 GiB (0.98) | field is `le=1` so 31.86 GiB is the field max, but 1.0 is UNBOOTABLE (below) | **yes** — shipped 0.982 |
+| `non-expert weights` | 6.28 GiB | checkpoint census | no |
+| `KV reserve` | 1.91 GiB (registry 9,190 B/token x 200704 +8%) | real pool bills 2.93 GiB at 7,892 B/token | no (registry constant) |
+| `CUDA graphs` | 0.55 GiB = 256 MiB + 52 MiB x 6 | `Graph capturing finished in 3 secs, took 0.31 GiB` | **yes** — `VLLM_CUDAGRAPH_MEM_BASE_MB` / `_PER_GRAPH_MB`; `VLLM_CUDAGRAPH_MEM_ESTIMATE_MB` pins the whole term outright (checked first, pair ignored when set) |
+| `runtime overhead` | 5.25 GiB (`--expert-precise-memory`) | far lower at idle | no (registry constant) |
+
+Shape F took the graph term (0.55 -> 0.32 GiB) and nudged util to 0.982:
+**arena 17.52 GiB / 7,653 slots measured**, +131 slots. Shape G takes the
+graph term further (0.32 -> 0.07 GiB) at the same util.
+
+**Shape G takes the graph term to the measured floor, util to the law.** The
+graph base pool is the actual over-charge: shape F's own boot reported
+`Graph capturing finished in 4 secs, took -0.06 GiB` — a **negative** capture
+delta, so the 256 MiB base was fiction. Charging `0 base + 12 MiB/graph`
+(0.07 GiB for the shipped 6-graph plan, and still per-graph so an 18-graph
+spec-decode shape pays 0.21 GiB) lands the plan at budget 31.29 / graphs 0.07 /
+**arena 17.77 GiB / 7,762 slots measured**. The KV ask went to
+`3,166,500,000` B/rank, which the engine's own 7,891.8 B/token price lands at
+**401,408 tokens measured / 2.00x concurrency** — the first shape holding two
+full-length conversations with real spare. (The ask/landed delta is block
+alignment: 196 x 2048. The price is shape-dependent, 7,701 B/token at 262144.)
+
+**UTIL 1.00 is not a knife-edge on this fork, it is UNBOOTABLE — measured
+2026-10-03.** One deploy took the field maximum to buy the last 0.57 GiB of
+arena; both workers refused init BEFORE loading a single weight, twice:
+`ValueError: Free memory on device cuda:N (31.29/31.86 GiB ...)` — the worker
+demands free > budget and budget == total, so `le=1` on the CLI field is a
+trap: the engine dies in a 2-restart crash loop with zero payload progress.
+So `0.982` is a LAW, not a preference, and the "why 0.982" answer now has two
+proofs: this ValueError, and the historical ceiling deaths (the 2x262144 bf16
+boot at 0.13 GiB free; the 5x262144 boot at "0 bytes free" on a real
+995-token request). Everything beyond ~7,912 slots needs the registry edited
+inside the image (the fork's log line says exactly that: *"Update the
+registry entry if this drifts"*), not a deploy.
+
+**The host region was the last arena-plan term still autofitting per boot —
+now it is a literal (2026-10-04).** With no `--expert-offload-mem` the planner
+binds the host region to `MemAvailable` AT BOOT minus `PLE 47.68 - 2 x 7.0
+process - 5.0 api/engine - 5% margin`: the 2026-10-05 boot read 237.97 GiB and
+offered 159.39 GiB usable, while the same host reads **75.4 GiB available while
+running** — the bound tracks whatever the co-tenants hold at that moment. The
+demand is 105.48 GiB (52.74 GiB/rank x 2, 23,040 swap-target experts), and a
+boot where the bound drops near it silently converts next-hottest experts to
+VRAM-only (the `pinned_extra` warning is the only trace).
+`FRANK_PRIMARY_EXPERT_OFFLOAD_MEM=120` pins it: measured demand + 14% headroom,
+on a host with 246.1 GiB total. Semantics verified in the shipped image
+(`offloader/expert_plan.py`): the cap binds as the bound, an exceedance only
+warns, and the plan rewrites the field to the cap it met (+1 MiB/rank).
+
+**MNBT 1024 -> 2048 is free on every axis the plan can see.** The registry
+overhead model is `base + 0.22 GiB x max(0, (nbt - 4096)/1024)` anchored at
+nbt 4096, so any rung up to 4096 charges the identical 5.25 GiB; the fitted 5.25
+was itself fitted at nbt 4096, which means 2048 is covered by an activation
+figure measured at a **larger** batch. SEQS 3 boots at both 1024 and 2048
+(measured 2026-10-02 — the `query_start_loc` `ConstraintViolationError` is a
+SEQS property). The costs are a one-time recompile (the compile **range** is
+part of the cache key, so `FRANK_PRIMARY_CACHE_DIR` pays once) and a wider
+prefill chunk, which the 09-29 sweep found decode-interchangeable across
+1024-4096 while 512 doubled TTFT at 4064 ctx (2.46 s vs 1.36 s).
+
+**`--expert-precise-memory` is the arena lever that does not touch the window.**
+It attacks the planner's overhead term (shape F above attacks the graph term —
+the third and last term this repo can reach), and the flag swaps the default
+registry figure for the fitted one. Measured
+2026-10-02 by a boot probe on the shape-D base (production stopped, candidate
+booted AS the service; numbers are the engine's own plan lines):
+
+| plan line | shape D | **shape E** |
+|---|---|---|
+| `runtime overhead (non-torch + buffers + activation)` | 7.60 GiB | **5.25 GiB** (`precise (gfx1201 TP2 nbt4096/seqs16/len262144)`) |
+| `expert cache space` | 14.87 GiB | **17.22 GiB** |
+| slots | 6,496 | **7,522** (+1,026, +16%) |
+| `swap region` | - | 13.7 GiB / 5,986 slots, 133-134 per layer |
+| boot | - | healthy in 420 s, no OOM |
+
+`'expert_precise_memory': True` also appears in `non-default args`, so a silent
+argparse drop is visible in the boot log. The flag lands **in the arena, not the
+pool** - `--kv-cache-memory-bytes` is untouched at 3.15e9/rank (option A), because
+the pool is not an arena lever and cutting it further would only cost
+concurrency: a 2.2e9 pool was probed in the same window and reports 277,686
+tokens / **1.38x**, so it is the conservative fallback if free VRAM ever binds,
+not the shipped value.
+
+**The compile cache is a boot-cost fix, and it was worth 173 s per boot.**
+Measured 2026-10-02: the image ships **no `/root/.cache` at all** (a fresh
+container: `No such file or directory`), and after a boot that directory holds
+`vllm` at **1,154,714,333 B in 4,665 files** plus `comgr` at 229 MB, all inside
+the container's ephemeral layer. Every
+`podman rm -f` - i.e. every deploy, restart and reboot - discards it, and the
+engine re-pays:
+
+```
+Using cache directory: /root/.cache/vllm/torch_compile_cache/b265bd396a/rank_0_0/backbone
+Dynamo bytecode transform time: 4.99 s
+Compiling a graph for compile range (1, 1024) takes 173.19 s
+saved AOT compiled function to /root/.cache/vllm/torch_compile_cache/torch_aot_compile/…
+torch.compile took 181.20 s in total
+init engine (profile, create kv cache, warmup model) took 217.26 s (compilation: 181.20 s)
+```
+
+Mounting a host directory at `/root/.cache` makes every later boot load the AOT
+artifact instead. The mount is deliberately the **parent**, not
+`/root/.cache/vllm`: `VLLM_CACHE_ROOT` defaults to the `vllm` child, and
+`comgr` is the HIP-kernel JIT cache, so one mount persists both — and it hides
+nothing, because the image ships no `/root/.cache`. The cache key
+includes the compile range, so the MNBT choice costs exactly one compilation.
+Entries are version- and config-hashed: a stale entry is a miss, never a wrong
+answer.
+
+**Proven live on the deployed endpoint, 2026-10-02, consecutive boots of this
+exact shape:**
+
+| boot | engine log |
+|---|---|
+| cold (deploy) | `Compiling a graph for compile range (1, 1024) takes 179.24 s` → `torch.compile took 188.26 s in total` |
+| warm (restart) | `Directly load the compiled graph(s) for compile range (1, 1024) from the cache, took 2.143 s` → `torch.compile took 3.97 s in total` → `init engine (profile, create kv cache, warmup model) took 41.13 s (compilation: 3.98 s)` |
+
+The host directory held **1.4 GB in 4,707 files** after the cold boot. Time to
+`/health` is dominated by the 172.8 GiB payload load (the warm boot reached
+health in 490 s), so the win is the ~184 s of inductor/AOT build rather than a
+proportional cut in boot time.
+
+`GPU_MAX_HW_QUEUES=2` caps the HSA hardware queues per GPU. Verified a real knob
+rather than a no-op - the literal is read out of
+`/opt/rocm/core-10.0/lib/libhsa-runtime64.so` - and shipped deliberately
+alongside the 32x64 gather geometry. It is **not** A/B measured.
+
+**The pool, not the slot count, sets how many conversations live at once.**
+Measured live at this shape: `GPU KV cache size: 398,658 tokens, Maximum
+concurrency for 200,704 tokens per request: 1.99x` (3,146,129,408 B reserved).
+The per-token price is **shape-dependent**: the 262144 shape priced at 7,701
+B/token (6,100,000,000 B -> 792,069 tokens), this pool at 7,892 - ~2.5% dearer,
+so a concurrency figure never extrapolates across geometries. The endpoint
+advertises 3 slots: a third LONG request queues rather than being refused, and
+shorter requests use the spare blocks. Verified: three concurrent 67-token
+requests answered in parallel, ~4.1 s each. Two contexts at 200704 are a
+capacity contract, not a throughput one (2-way concurrency summed BELOW parity
+at 4064 ctx).
+
+#### `--max-num-seqs 2` is impossible on this model (measured 2026-10-02)
+
+Four single-variable boot probes against the then-deployed shape, production
+stopped for each:
+
+| MAXLEN | SEQS | MNBT | result |
+|---|---|---|---|
+| 262144 | 2 | 1024 | **Engine core initialization failed** |
+| 262144 | 2 | 2048 | **Engine core initialization failed** |
+| 262144 | 3 | 1024 | healthy - 792,069 tokens, 3.02x, arena 6,248 |
+| 262144 | 3 | 2048 | healthy - the then-deployed incumbent |
+
+The failure is `torch.fx.experimental.symbolic_shapes.ConstraintViolationError:
+Constraints violated (L['query_start_loc'].size()[0]) ... specialized it to be
+a constant (3)`, raised from AOT `build_guards(strict_error=True)` during
+`determine_available_memory -> profile_run -> _dummy_run`. `model_runner.py`
+sizes that input buffer `max_num_reqs + 1` and `_dummy_run` sets
+`num_reqs = min(max_num_batched_tokens, max_num_reqs)`, so the width is
+`SEQS + 1` for ANY `MNBT >= SEQS`: 3 at SEQS 2, 4 at SEQS 3. **MNBT is not a
+lever here** - it only moves the compile RANGE ((1,1024) vs (1,2048)), which
+is why the identical error appears at both rungs. `--enforce-eager` bypasses
+the path at the cost of CUDA graphs. The same failure class killed the
+2026-10-01 bf16 2 x 262144 shape.
+
+#### The arena lever is `max_model_len`, not `--kv-cache-memory-bytes`
+
+The fork's planner sizes expert cache as `budget - non-expert - KV reserve -
+graphs - overhead`, and the KV reserve is registry-priced at **9190 B/token x
+max_model_len** - not from the `--kv-cache-memory-bytes` value:
+
+| MAXLEN | KV reserve | arena |
+|---|---|---|
+| 262144 | 2.48 GiB | 14.30 GiB / **6,248 slots** |
+| 200704 | 1.91 GiB | 14.87 GiB / **6,496 slots** |
+
+D(reserve) 0.57 GiB = D(arena) 0.57 GiB = +248 slots. The 6.1e9 -> 3.15e9
+pool cut (2.75 GiB) bought **zero** arena slots. It buys runtime HEADROOM,
+which is still worth having - the shape it replaced died with `0 bytes free`
+on a real 995-token request. Grow the arena by lowering `max_model_len`, or by
+taking the overhead term with `--expert-precise-memory` (below); the pool is the
+headroom lever only.
+
+#### The other arena lever: `--expert-precise-memory` (overhead 7.60 -> 5.25 GiB)
+
+The flag is now ON. It was refused on 2026-10-01 for a reason that does not
+generalize: the 2 x 262144 bf16 shape was already at the ceiling (weights 6.28 +
+pool 6.80 + graphs 0.55 + arena 16.65 + non-torch ~1.45 = 31.73 of 31.86 GiB,
+NRestarts=3 at ~6.7 min each), so the 2.35 GiB the flag moves into the arena was
+the last stack over the line, not the cause. The plain bf16 2 x 262144 boot also
+did not stay up: 0 bytes free at runtime, and a real client request (995-token
+prompt, `max_tokens=261149`) killed the worker with `Tried to allocate 50.00 MiB
+... 0 bytes free`.
+
+At 200704 x 3 the flag fits with room and is worth **+1,026 slots (+16%)**. What
+made it affordable is the window trim, not a pool cut: the arena is
+DEVICE-resident, the pool is not part of the budget's arithmetic, and the 200704
+window's KV reserve (1.91 GiB) plus a smaller attention footprint is what the
+plan needed. The pool stays at 3.15e9/rank deliberately (option A) - cutting it
+further would cost concurrency (2.2e9 -> 1.38x) and buy no slots.
+
+Boot-probed 2026-10-02 (transient systemd units at the production uid +
+memlock, full-log capture; the rootless bench probe had masked the production
+memory environment). The night searched two concurrency contracts:
+
+| rung | result |
+|---|---|
+| 3 x 262144, bf16, KV 11,047,010,304 | **OOM at graph capture** — `Tried to allocate 76.00 MiB ... 0 bytes free`, 30.43 GiB of 31.86 GiB already allocated by PyTorch |
+| 3 x 245760, bf16 | **OOM identically** |
+| 3 x 237568, bf16 | PASS — KV 718,606 tokens |
+| 2 x 262144, bf16 + `--expert-precise-memory` | **DEPLOYED then crashed** — 0.13 GiB free at boot, NRestarts=3 (the ceiling, not the flag) |
+| 2 x 262144, bf16 (no precise flag) | did not stay up either: 0 bytes free at runtime, worker killed by a real request |
+| 3 x 200704, fp8, pool sized for 2 contexts | **DEPLOYED 2026-10-02 (shape D)**: 398,658 tokens = 1.99x concurrency, arena 14.87 GiB / 6,496 slots, the 2.75 GiB/rank pool cut returned as runtime headroom |
+| **3 x 200704, fp8, + `--expert-precise-memory`** | shape E (2026-10-02): same 1.99x pool, overhead 5.25 GiB, arena 17.22 GiB / 7,522 slots, boot healthy in 420 s |
+| 3 x 200704, fp8, MNBT 2048, graph term 0.32 GiB, util 0.982 | shape F (2026-10-03): arena 17.52 GiB / **7,653 slots** measured, budget 31.29, `Estimated CUDA graph memory: 0.32 GiB`, KV unchanged 398,658 |
+| 3 x 200704, fp8, graph term 0.07 GiB, util **1.00**, KV 3.1665e9 | **DID NOT BOOT (2026-10-03)**: `ValueError: Free memory on device cuda:N (31.29/31.86 GiB)` on both ranks, twice, zero weights loaded — the worker demands free > budget and budget == total. UTIL 1.0 is unbootable, full stop |
+| **3 x 200704, fp8, graph term 0.07 GiB, util 0.982, KV forced to 400,700** | **DEPLOYED 2026-10-03 (shape G)**: arena **17.77 GiB / 7,762 slots** measured (+109 over F, +240 over E), KV **401,408 tokens / 2.00x** concurrency measured, capture 0.31/0.10 GiB vs the 0.07 charge (under-charged, boots and holds), driver used 30.97 GiB -> **0.89 GiB real free per card** held through a live 4,096-token generation (~38-39 tok/s single-request, +13% vs the 33.99 shape-E baseline); rollback is UTIL or KV_BYTES down a notch |
+
+Six full-length contexts cannot fit at any KV dtype: the 6-seq graph/activation
+footprint leaves only 9.18 GiB for the pool, and 6 × 262144 needs 12.47 GB even
+at fp8.
+
+The old "KV_BYTES is PER RANK, 9/16 GiB OOM" law was the MTP4 WORKSPACE
+reservation, not a VRAM wall. Failures are clean and diagnostic: the engine
+reports `torch.OutOfMemoryError: Tried to allocate X GiB. GPU N has ... of
+which Y GiB is free`, and the fork's giant ERROR lines are split by journald —
+read them via `journalctl -o json`, not `-o cat` (and `podman logs`, which is
+only available while the container lives).
+
+Rollback to the 2026-09-29 fp8+MTP4 shape: git tag
+`frank-primary-fp8-mtp-20261001` (env trio + seam re-pinned there).
+
+## Parallelism options on this box (2026-10-01 survey)
+
+Surveyed from the running image's own CLI surface. Verdicts are specific to
+dankserver: 2x R9700 (gfx1201), a ~172.8 GiB expert-offload MoE, and a model
+with only 2 KV heads.
+
+| option | flag | verdict on this box |
+|---|---|---|
+| TP | `--tensor-parallel-size` | **DEPLOYED, =2.** All-reduce dispatch shows `R4D_AR` then `PYNCCL` |
+| DCP | `--decode-context-parallel-size` | **REJECTED at TP=2** — see below |
+| PP | `--pipeline-parallel-size` | supported; untried (stage handoff plus a TP-keyed expert plan) |
+| DP | `--data-parallel-size` (+`-rank`, `-start-rank`) | impossible: every replica holds a full model copy (172.8 GiB vs 31.86 GiB/card) |
+| EP | `--enable-expert-parallel` + `--all2all-backend` (+`--enable-eplb`, `--enable-dbo`, `--enable-elastic-ep`) | supported (`naive`, `allgather_reducescatter`, `deepep_high_throughput`, `deepep_low_latency`, `pplx`, `mori`, `nixl`); would replace the fork's tiered-offload LOCAL gather with a per-layer PCIe all-to-all |
+| multi-node | `--nnodes`, `--node-rank`, `--master-addr/port`, `--distributed-executor-backend` | supported; the actual path to more cards |
+
+### DCP is structurally unavailable here — do not re-test on 2 GPUs
+`--decode-context-parallel-size 2` fails `VllmConfig` validation at boot:
+
+```
+Decode context parallelism for GQA/MQA requires `--tensor-parallel-size` (2) to be
+greater than the model's total number of KV heads (2). Increase
+`--tensor-parallel-size` or set `--decode-context-parallel-size 1`.
+```
+
+DCP shards the KV along the *sequence* dimension after KV heads have been spread
+across the TP ranks, so it requires **TP > num_kv_heads**. This model has 2 KV
+heads (24 Q heads, GQA 12:1), so TP must be >= 4 — four GPUs, not two. Related
+fork facts: DCP *reuses* the TP ranks rather than multiplying them (it is not
+`TP x DCP`), and with PCP enabled the valid DCP sizes are `{1, pcp, tp x pcp}`
+(`vllm/config/parallel.py`). The KV memory a DCP win would have bought is real —
+each rank would hold half the sequence's KV — but it is unreachable without two
+more cards.
+
+### Measuring decode: the harness lives INSIDE the container
+`/home/dank/mtp_measure.py` imports `/bench_iter.py` by absolute path, so it must
+run in the engine's namespace and the files must be copied in first:
+
+```
+podman cp /home/dank/mtp_measure.py frank-primary-qwen:/
+podman cp /home/dank/bench_iter.py  frank-primary-qwen:/
+podman exec frank-primary-qwen python3 /mtp_measure.py
+```
+
+Without the copy the exec dies with `can't open file '/mtp_measure.py': [Errno 2]
+No such file or directory` — measured 2026-10-01, which silently cost a whole
+arm's numbers. A window script must do the copy, not assume the path exists.
+
+### The then-deployed shape, measured again (2026-10-01 A/B window)
+A clean production-environment boot of the config deployed at that time (4 x 262144 fp8, MNBT
+2048 in that window) reported **`GPU KV cache size: 1,079,582 tokens, Maximum
+concurrency for 262,144 tokens per request: 4.12x`** — the same figures as the
+deployed MNBT-1024 shape, i.e. MNBT does not move the pool, and the capacity is
+reproducible across boots.
+
 ## See Also

 - [[dankserver-frontier-moe-feasibility]] — the sibling feasibility page
  (which frontier MoE fits this box; the same PCIe-expert-stream wall at
  V4.1 scale)
- [[frank-primary-r9v-radiance-qwen38-q8]] — the ACTIVE frank-primary
-  (production is the R9V fork; this page's container is the ad hoc detour)
+- [[frank-primary-r9v-radiance-qwen38-q8]] — RETIRED 2026-09-30 (the R9V
+  fork era; production has been the tcclaviger fork since)
 - [[x4-pair-experiment]] — the same residency law measured on a narrower
  link: streaming over PCIe loses, computing in RAM does not care
 - [[Dual-R9700-Cooperation]] — the board swap that made TP=2 legal here
@@ -1,7 +1,7 @@
 ---
 type: reference
-title: "gufo, Swift-1.5-Qwen3.8-27B-Uncensored UD-Q8_K_XL (DFlash2) on Dankclaw (frank-large)"
-description: "The active frank-large runbook — gufo (gufo-org/gufo) v0.1.1, the Strix Halo inference engine, built from source on Dankclaw and serving the abliterated ajgazin/Swift-1.5-Qwen3.8-27B-Uncensored UD-Q8_K_XL with the z-lab DFlash2 Q4_K_M drafter on the gfx1151 iGPU: the from-source build and its two host workarounds, the GGUF header evidence that made the payload swap safe, the one-session / full-window shape, why the fleet's Qwen-Sharp chat template CANNOT be used on this runtime and what that costs, the client-contract delta, the measured prefill/decode trade, and the failure modes worth knowing."
+title: "gufo, Qwen3.8-27B UD-Q4_K_XL (DFlash2) on Dankclaw (frank-large)"
+description: "The RETIRED frank-large runbook (rollback target) — gufo (gufo-org/gufo) v0.1.1, the Strix Halo inference engine, built from source on Dankclaw and serving unsloth/Qwen3.8-27B-GGUF UD-Q4_K_XL (vanilla) with the z-lab DFlash2 Q4_K_M drafter on the gfx1151 iGPU, ONE session at the full native 262,144 window with unquantized f16 KV: the from-source build and its two host workarounds, the GGUF header evidence, why the fleet's Qwen-Sharp chat template CANNOT be used on this runtime, the client-contract delta, the measured prefill/decode trade, and the failure modes worth knowing."
 tags:
  - llm
  - gufo
@@ -16,9 +16,20 @@ tags:
 timestamp: 2026-09-28
 ---

-# frank-large — gufo, Swift-1.5-Qwen3.8-27B-Uncensored UD-Q8_K_XL (DFlash2)
+# frank-large — gufo, Qwen3.8-27B (UD-Q4_K_XL, DFlash2)

-> **ACTIVE since 2026-09-28.** Replaces
+> **RETIRED 2026-10-02 — the rollback target.** Superseded by
+> [[frank-large-rocmfpx-q8agent-mtp]] (Q8_0_ROCMFPX_AGENT at 8.39 bpw with
+> MTP4, 17.0 t/s measured). This era was itself a return after one day of
+> [[frank-large-vllm-mimo-v26-bf16]] (battery decode 6.19-9.01 t/s; every
+> ≥6-bit vLLM quant path dead on gfx1151). The shape
+> moved on the way back: payload is now **unsloth/Qwen3.8-27B-GGUF
+> UD-Q4_K_XL** (16.35 GiB, vanilla — the abliterated Swift Q8 was
+> reclaimed), and the session grid is **one session at the full native
+> 262,144 window** (`SESSIONS=1 CONTEXT_SIZE=262144`, f16 KV). The
+> Swift-1.5 Q8 history below is retained for the engine/template law,
+> which still governs.
+> **Originally ACTIVE 2026-09-28.** Replaces
 > [[frank-large-unsloth-qwen38-27b-q8xl]] (stock llama.cpp, the unsloth
 > b11030 prebuilt), which had replaced
 > [[frank-large-kairic-edge-qwen38-27b]]. Same host, same port (7770), same
@@ -0,0 +1,130 @@
+# frank-large — halogen container, Qwen3.8-27B + DFlash2 (Dankclaw, ACTIVE 2026-10-05)
+
+frank-large is the Strix Halo iGPU endpoint on Dankclaw (`:7770`). Since
+2026-10-05 it runs the **halogen** engine — `ghcr.io/peonist-ai/halogen` at a
+pinned digest, launched by `scripts/serve-frank-large.sh` under a root
+systemd unit as a podman container — serving
+**`peonist-ai/halogen-qwen3.8-27b`** (`qwen3.8-27b-p1w4d-d2.hgn`,
+35,865,565,184 B ≈ 33.4 GiB, sha256 `274c3fc7…`, 1352 tensors) with the
+**DFlash2** drafter at **one slot x 200,700 ctx**.
+
+It replaced the ROCmFPX llama.cpp fork endpoint ([[frank-large-rocmfpx-q6agent-uncensored-mtp]]),
+which remains the rollback target with its payload on disk.
+
+## Why swap
+
+|Metric|ROCmFPX Q6_0_ROCMFPX_AGENT + MTP (retired)|halogen qwen3.8-27b + DFlash2 (active)|
+|---|---|---|
+|Prose decode|27.6–28.1 t/s measured (4.8k depth, acceptance 0.868)|31.7 t/s mean across eight text workloads (publisher card, bf16 reference hardware)|
+|Copy-echo decode|31.4 t/s|publisher reports echo inside the same 31.7 mean|
+|Context|1 x 262,144|1 x 200,700|
+|Format|GGUF (fork-private tensor types 102/103)|`.hgn` — loads in NO other engine|
+
+The swap was accepted with eyes open on two product regressions, both
+confirmed by Frank before deploy:
+
+1. **Text-only.** halogen loads the language tower only; README "honest
+   limits": *"The vision encoder is ignored; image inputs are not supported."*
+   The mmproj-era vision role moves wholly to frank-small.
+2. **Censored base.** The checkpoint is a finetune of the censored
+   `Qwen/Qwen3.8-27B` base, built to make its DFlash2 head strong. The
+   ROCmFPX AGENT file was an *Uncensored* build; refusals can return.
+
+## Shape and why
+
+`scripts/frank-large.env` carries every tunable with its evidence; the load-
+bearing numbers:
+
+- **`HALOGEN_KV_SLOTS=1` + `HALOGEN_DRAFTER=2`** — the engine offers batch OR
+  speculation, never both (FLAGS.md: "Batching and speculation are exclusive:
+  a step batches when more than one conversation is running, so the drafter
+  stays off with more than one slot"). Slots>1 silently loses DFlash2; the
+  sibling halogen engine measured 41 -> 14 t/s when slots went up. frank-large
+  was a one-conversation endpoint before (PARALLEL=1), so the contract holds.
+- **`HALOGEN_SLOT_CTX=200700`** — the per-conversation HARD ceiling: a longer
+  prompt is a 400 naming the limit, never a truncation. The harness side is
+  wired to it: `models.yml` `contextWindow: 200700`, and OMP compacts at 70%
+  = 140,490, leaving 60,210 of the slot for the reply.
+- **`HALOGEN_MAX_TOKENS_CAP=65536` + `HALOGEN_QUEUE_TIMEOUT=7200`** — coupled
+  pair (FLAGS.md table): a request at the cap must not outlive the timeout.
+  65536 > the harness's 57,344 `maxTokens` so an eager request never meets
+  the engine's 400.
+- **KV math:** 1 slot x 200,700 x 64 KiB/token (d2 KV = 2x fp8) = 12.2 GiB
+  pool, inside the ~65 GiB left after 33.4 GiB mmap'd weights + engine working
+  set + host baseline on the 121.2 GiB box. The engine sizes the pool itself
+  and **prints a boot ledger naming any clamp** — the post-deploy check reads
+  that ledger, not a formula.
+- **No server-side sampler:** an absent temperature is GREEDY. The card's
+  thinking recipe (temp 1.0 / top_p 0.95) lives in `models.yml` `extraBody`
+  — on the wire only. The template defaults `reasoning_effort` to **xhigh**
+  when a request sends none, and **thinking tokens consume the
+  `max_tokens` budget**: a budget exhausted mid-thought returns
+  `finish_reason: "length"` with EMPTY `content` and the partial reasoning
+  in `reasoning_content` — clients must read both fields.
+
+## Client contract deltas vs llama.cpp
+
+|Behavior|halogen|
+|---|---|
+|Prompt over slot|400 naming the limit (no truncation)|
+|`max_tokens` over cap|400 naming the limit (never silent truncation)|
+|full-length request outliving `HALOGEN_QUEUE_TIMEOUT`|503s for everyone queued|
+|`timings`|every response carries llama-server-shape `timings` (`draft_n`, `draft_n_accepted`) — the drafter check reads the RESPONSE, and it proves speculation fired|
+|`response_format`|NOT supported (no constrained decoding)|
+|thinking field|`reasoning_content` (same as before)|
+|`reasoning_effort`|same Qwen3.8 template; outside {xhigh,medium,low} rejected — models.yml fold stays; unset defaults to xhigh, `none` disables thinking|
+|tools|server-side tool calls advertised — round-trip MUST be re-probed after deploy|
+|image input|ignored/error — route to frank-small|
+
+## Deploy mechanics
+
+- **Image:** `ghcr.io/peonist-ai/halogen@sha256:dc0a39a0016d6cfc58a197978febaafdf8d28403f724ded6111d98b5fb7ac0ea`
+  (published tag 0.1.4). The deployer gates the exact digest exists in podman
+  storage before dispatch. NOT the 2026-09 `halogen-flash-server` repo — that
+  was the Flash-Next w4b era, different image, retired.
+- **Payload:** `scripts/download-frank-large-halogen.sh` (fetch law: `curl -f`,
+  exact byte count, LFS sha256 where the Hub publishes one, `--verify-only`,
+  token never in argv) into `/home/dank/models/halogen-qwen3.8-27b/`. The
+  tokenizer is a FLAT DIRECTORY (`tokenizer/`, 5 files — tokenizer.json,
+  vocab.json, merges.txt, chat_template.jinja, tokenizer_config.json),
+  mounted whole at `/tokenizer`; the engine's `HALOGEN_TOKENIZER` default is
+  that directory, so the container gets no tokenizer flag (model-card
+  docker-run parity). The deployer and serve script refuse a missing file or
+  ANY symlink: an HF cache snapshot symlinks each entry into `blobs/` and
+  those links dangle inside the container mount.
+- **Unit:** `scripts/frank-large.service` (root, `User=dank`, relinked into
+  `/etc/systemd/system/`), `ExecStart` = serve script, `ExecStop` = `podman
+  stop -t 120 frank-large-halogen`, script `exec podman wait` so
+  `Restart=always` is real. Engine stdout goes to
+  `/home/dank/logs/frank_large.log` (the boot ledger lives there, not in the
+  journal).
+- **Cutover:** `deploy_llm_large` in `scripts/deploy-remote.sh` sweeps the
+  legacy bare `llama-server --model /home/dank/models/kingjones-*` process and
+  the legacy `frank-large` container name by NAME (never `pkill -f
+  llama-server` — that hits frank-small), retires the old quadlet if the 2026-09
+  halogen era's unit lingers, waits port 7770 free, relinks, health-gates.
+- **Bench law:** stop+mask the unit before any GPU experiment on this box
+  (AGENTS.md); `/health` answers 503 while the 33.4 GiB payload loads — that
+  is loading, not failure.
+
+## Verification (post-deploy checklist)
+
+1. `grep -iE 'clamp|kv|pool|drafter|dflash' /home/dank/logs/frank_large.log`
+   — ledger shows the pool at the asked shape, DFlash2 active, NO clamp line.
+2. `curl -s http://192.168.0.143:7770/health` -> 200.
+3. Real completion with `reasoning_effort` + temp/top_p from the wire recipe;
+   `reasoning_content` populated; decode rate from the timings.
+4. Spec decode fires: multi-token step behavior visible in rate; compare
+   against DRAFTER=0 arm if in doubt (never against gufo — bench law).
+5. Harness round-trip: an OMP call through `llamacpp (dankclaw)/frank-large`
+   incl. a tool call (V1TOOLS).
+6. Prompt > 200,700 tokens returns the 400 naming the limit (contract test).
+
+## Rollback
+
+Revert the frank-large swap commit and redeploy `dankclaw --service frank-large`:
+the ROCmFPX payload (`/home/dank/models/kingjones-Qwen3.8-27B-ROCmFP4-STRIX-MTP/`),
+the fork build (`/home/dank/GitPackages/kingjones-ROCmFPX/build`), and its
+`scripts/build-rocmfpx.sh` pin (commit `36e9acd`, seam-tested) all remain on
+disk. The halogen payload can stay: it costs nothing while the unit points
+elsewhere.
@@ -0,0 +1,177 @@
+---
+type: reference
+title: "frank-large on Dankclaw — ROCmFPX llama.cpp, Qwen3.8-27B Uncensored Q6_0_ROCMFPX_AGENT + MTP"
+description: "The active frank-large runbook — the kingjones30/ROCmFPX llama.cpp fork (pinned commit, gfx1151) serving kingjones777's Uncensored Q6_0_ROCMFPX_AGENT GGUF (ftype 114, protected output heads), with the repo's own MTP-Q4_0 head drafting four tokens at one native 262,144 window, f16 KV, the BF16 projector for vision, and the measured rejection of the ngram-mod chain."
+tags:
+  - llm
+  - frank-large
+  - dankclaw
+  - llama-cpp
+  - rocmfpx
+  - gfx1151
+  - gguf
+  - spec-decode
+  - mtp
+timestamp: 2026-10-03
+---
+
+# frank-large — ROCmFPX Uncensored Q6_0_ROCMFPX_AGENT + MTP (gfx1151)
+
+> **RETIRED 2026-10-03 → 2026-10-05 (rollback target; payload, fork build and
+> build script all on disk).** Replaced by [[frank-large-halogen27b-dflash2]]
+> (DFlash2, publisher 31.7 t/s mean vs the 27.6–28.1 measured here). This page
+> remains the contract record for the rollback. Original banner follows.
+> ACTIVE since 2026-10-03. Cut over from
+> [[frank-large-rocmfpx-q8agent-mtp]] (ALIGNED Q8_0_ROCMFPX_AGENT, 17.0 t/s)
+> on Frank's request: the *uncensored* variant of the same protected-heads
+> family, one quant rung down. The swap turned out to be a strict upgrade on
+> the axis that matters here: **decode 27.6–28.1 t/s at a 4,813-token prompt
+> against 17.0 at a 2,202-token prompt** — faster with a deeper context, not a
+> same-depth comparison, but the sign is not in question.
+
+## Identity
+
+| | |
+|---|---|
+| Host / device | Dankclaw (192.168.0.143), Radeon 8060S iGPU `gfx1151`, ~114 GiB GTT aperture |
+| Endpoint | `http://192.168.0.143:7770` — alias `frank-large`, `/v1` OpenAI surface |
+| Runtime | **`kingjones30/ROCmFPX`** llama.cpp fork @ `36e9acd40e10a87cd3c3ef8ec734668757dc8520`, built by [`scripts/build-rocmfpx.sh`](../../scripts/build-rocmfpx.sh) — unchanged from the Q8 era; the fork already knew type 102 |
+| Unit | root systemd `frank-large.service` ([`scripts/frank-large.service`](../../scripts/frank-large.service)), `User=dank`, `Restart=always` |
+| Serve script | [`scripts/serve-frank-large.sh`](../../scripts/serve-frank-large.sh) (owns the argv), tunables [`scripts/frank-large.env`](../../scripts/frank-large.env) |
+| Weights | `/home/dank/models/kingjones-Qwen3.8-27B-Uncensored-ROCmFP4-STRIX-MTP/Qwen3.8-27B-Uncensored-Q6_0_ROCMFPX_AGENT.gguf` — 24,856,607,008 B, sha256 `e966deb3…` |
+| Draft | `…/mtp-Qwen3.8-27B-Q4_0.gguf` — 1,680,271,648 B, sha256 `051a1764…` (byte-identical to the aligned repo's head; hardlinked, not re-downloaded) |
+| Projector | `…/mmproj-Qwen3.8-27B-BF16.gguf` — 931,145,888 B (hardlinked from the aligned payload dir; the same file the Q8 era served) |
+| Fetch | [`scripts/download-frank-large-kingjones-uncensored.sh`](../../scripts/download-frank-large-kingjones-uncensored.sh) — hardlinks the draft/projector from the aligned dir when present, otherwise fetches; `-f`, byte-count + LFS sha256 verified, `--verify-only`, token never on the command line |
+| Seam | `tests/frank_large_unit.bats` (payload gates, deployer/serve parity, spec block, KV dtype) |
+
+## Why this file
+
+`kingjones777/Qwen3.8-27B-Uncensored-ROCmFP4-STRIX-MTP-GGUF` ships the same
+"protected heads" construction as the aligned repo, in the uncensored tuning.
+Parsed header of the deployed file:
+
+| Field | Value |
+|---|---|
+| ftype | **114** — the fork's 64-bit-block family, ≈7.44 bpw |
+| Tensor mix | 353 F32 + **339 type-102 (`GGML_TYPE_Q6_0_ROCMFPX`)** + **159 type-103 (`Q8_0_ROCMFPX`)** |
+| Protected heads | 64 output-head tensors kept at plain precision — the AGENT construction from the aligned repo, at the 6-bit rung |
+| Arch | qwen35, 64 layers, `full_attention_interval 4`, `context_length 262144` |
+| Embedded sampling recipe | temp 1.0 / top_p 0.95 / top_k 20 |
+
+The same lock-in law as the Q8 era applies: types 102 and 103 exist in no
+loader but this fork; stock llama.cpp and gufo refuse the file at load
+(`unknown tensor type`). The loader was not re-litigated — the file picks the
+runtime, and the pin in `scripts/build-rocmfpx.sh` (commit by SHA) stays.
+
+Rejected sibling: `Q6_0_ROCMFPX_LEAN` (6.45 bpw, no protected heads) — the
+protection is what keeps the output-head logits at full precision and the
+drafter accurate, so it stays deployed.
+
+## Shape and budget
+
+Unchanged from the Q8 era — the swap was payload-only:
+
+| Dial | Value | Why |
+|---|---|---|
+| `--ctx-size` | 262,144 | the native window, no rope scaling |
+| `--parallel` | 1 | one conversation, all memory to it |
+| KV dtype | **f16** (both K and V) | ~64 KiB/token; free inside a 114 GiB aperture |
+| `--fit off` | on | the auto-fitter would silently resize the window |
+| `--flash-attn on` | on | required for the long-window attention path |
+| `--direct-io`, `--ctx-checkpoints 16`, `--cache-ram 8192`, `--cache-idle-slots`, `--jinja`, `--reasoning on --reasoning-format deepseek --reasoning-budget -1` | | same as [[frank-large-rocmfpx-q8agent-mtp]] |
+
+Spec decode: `--spec-type draft-mtp`, the repo's `mtp-Qwen3.8-27B-Q4_0.gguf`,
+`--spec-draft-n-max 4`, pinned `--spec-draft-device ROCm0`.
+
+Measured steady state: GTT **~61 GiB** under live load (host baseline 16.7),
+boot to `/health` **~15 s**.
+
+## Measured on this payload (2026-10-03, production service, `/completion` timings, temp 0, `cache_prompt: false`)
+
+| Traffic | Decode t/s | Prefill t/s | Draft acceptance |
+|---|---|---|---|
+| Prose, 4,813-token prompt, 512 tokens | **27.57 / 28.10** (two runs) | ~205–217 | **396/456 = 0.868** → mean acceptance length ≈ 4.4, i.e. drafting at nearly the full n-max 4 |
+| Schema echo (structured repetition) | **31.44** | — | 639/639 = 1.000 |
+| Block copy (text repeated verbatim) | ~28.0 | — | 0.848 |
+
+vs the Q8 era's 0.462 acceptance at a 2,202-token prompt: the MTP head drafts
+materially better on this file. The card's claimed 0.953-class acceptance is
+no longer science fiction on real prose — but keep reading it as a file
+property, not a throughput promise.
+
+## The ngram-mod chain — measured and rejected
+
+Frank's ask: layer `--spec-type draft-mtp,ngram-mod` on top of MTP for an
+extra gain. Measured in three bench windows (production service stopped per
+the benchmarking law; artifacts `/home/dank/fl-ngram-ab/` on Dankclaw),
+comparing `draft-mtp` alone vs the chain on the same traffic:
+
+| Traffic | MTP alone | MTP + ngram-mod | Delta |
+|---|---|---|---|
+| Prose | 27.57 t/s | 23.40 t/s | **−15 %** |
+| Block copy | 28.0 t/s | 26.1 t/s | **−7 %** (with ngram-mod demonstrably firing: 138 proposals/run) |
+
+Mechanism, straight from arm B's boot log:
+
+```
+recurrent state rollback is not compatible with 'ngram-mod' - disabling rollback support
+```
+
+On the qwen35 hybrid (Gated DeltaNet recurrent layers) adding the ngram layer
+disables the rollback path MTP's own verification relies on, so acceptance —
+and throughput — drop on *all* traffic, even where the ngram matcher fires.
+**Deployed decision: `draft-mtp` alone.** Chaining is not free on recurrent
+models; the fleet law in `AGENTS.md → llama.cpp serving` records it.
+
+## Bench-window hardening laws (learned building the A/B)
+
+1. **Health 200 is not proof of ownership.** After boot, verify the pid you
+   started owns the port (`ss -ltnp`); a zombie server from a prior window
+   answers `/health` too and silently serves your arm.
+2. **Kill scratch servers with SIGKILL after a grace period.** llama-server's
+   graceful SIGTERM waits on open keep-alive connections — a harness holding
+   the socket leaves the process (and its ~24 GiB of GTT) alive after the
+   script exits.
+3. **Prove the port is free and GTT dropped between arms**, or the second
+   arm measures a collision.
+
+## Client contract
+
+Unchanged from the Q8 era and re-verified on this payload 2026-10-03: model id
+`frank-large`, native tools, thinking in `reasoning_content`,
+`reasoning_effort` ∈ {xhigh, medium, low} only (anything else HTTP 500),
+card-recipe sampling defaults server-side, vision through the BF16 projector
+(red PNG → **`Red`**, 74 prompt / 165 completion tokens). The traps list —
+`content: null` when reasoning eats the budget, the self-recovering MTP
+boundary log line, `enable_thinking:false` not `reasoning_effort:"none"` — is
+in [[frank-large-rocmfpx-q8agent-mtp]] and still applies.
+
+Sweep key for scratch servers is now the payload name:
+`pkill -f 'Qwen3.8-27B-Uncensored-Q6_0_ROCMFPX_AGENT'` — never `pkill -f
+llama-server` (frank-small is also one).
+
+## Operations
+
+```bash
+./scripts/deploy.sh dankclaw --service frank-large
+systemctl status frank-large
+journalctl -u frank-large -n 200 --no-pager
+scripts/download-frank-large-kingjones-uncensored.sh --verify-only
+```
+
+## Rollback
+
+| Target | How |
+|---|---|
+| **ALIGNED Q8_0_ROCMFPX_AGENT** (previous shape, 17 t/s) | the payload dir `/home/dank/models/kingjones-Qwen3.8-27B-ROCmFP4-STRIX-MTP/` is untouched on disk; revert the swap commit (unit/env/serve trio) and redeploy, or point `FRANK_LARGE_MODEL_FILE` + `LLM_LARGE_MODEL_FILE` at the old paths. Verify with `scripts/download-frank-large-kingjones.sh --verify-only` |
+| gufo Q4_K_XL era | git tag `frank-large-gufo-q4kxl-20261001`, **but the gufo payloads (UD-Q4_K_XL 16.35 GiB, UD-Q8_K_XL + DFlash2 drafter) were removed from disk 2026-10-03 with Frank's approval** — this rollback now requires re-download per [[frank-large-gufo-qwen38-27b]] |
+
+## When it misbehaves
+
+Same decision tree as [[frank-large-rocmfpx-q8agent-mtp]]:
+`unknown tensor type` ⇒ off the pinned fork; SIGSEGV in
+`ggml_compute_forward_set_rows` ⇒ pre-`36e9acd` build; slow decode with no
+error ⇒ draft not engaged (`draft_n` in `/completion` timings) or an orphaned
+second copy eating GTT (`/sys/class/drm/card1/device/mem_info_gtt_used`);
+500 naming Jinja ⇒ `--jinja` lost; 500 naming the template ⇒ unmapped
+`reasoning_effort` in `models.yml`.
@@ -0,0 +1,232 @@
+---
+type: reference
+title: "frank-large on Dankclaw — ROCmFPX llama.cpp, Qwen3.8-27B Q8_0_ROCMFPX_AGENT + MTP (retired)"
+description: "The retired frank-large runbook — the kingjones30/ROCmFPX llama.cpp fork (pinned commit, gfx1151) serving kingjones777's Q8_0_ROCMFPX_AGENT GGUF, a ftype-115 file no other loader can read, with the repo's own MTP-Q4_0 head, f16 KV, and the BF16 projector. Measured MTP-vs-none A/B, the client contract, and the rollback state as of 2026-10-03."
+tags:
+  - llm
+  - frank-large
+  - dankclaw
+  - llama-cpp
+  - rocmfpx
+  - gfx1151
+  - gguf
+  - spec-decode
+  - mtp
+timestamp: 2026-10-02
+---
+
+# frank-large — ROCmFPX Q8_0_ROCMFPX_AGENT + MTP (gfx1151, retired)
+
+> **RETIRED 2026-10-03** — superseded by
+> [[frank-large-rocmfpx-q6agent-uncensored-mtp]]: Frank swapped to the
+> uncensored Q6_0_ROCMFPX_AGENT file (protected heads, one rung down), which
+> measures **27.6–28.1 t/s** at a deeper prompt than this file's 17.0. This
+> runbook stays live as the rollback reference and the trap catalogue; the
+> payload it documents remains on disk at
+> `/home/dank/models/kingjones-Qwen3.8-27B-ROCmFP4-STRIX-MTP/`.
+>
+> Cut over 2026-10-02 from [[frank-large-gufo-qwen38-27b]] (Q4_K_XL, ~15 t/s)
+> on two requests from Frank: *highest quality that runs*, and *one full
+> context*. It delivered 8.39 bpw weights — two quant rungs above what gufo
+> could load — at 17.0 t/s, so quality was not bought with speed.
+>
+> The capability that forced this whole shape is a GGUF tensor type:
+> **`GGML_TYPE_Q8_0_ROCMFPX` (103)**. It exists in exactly one loader on this
+> host, so the runtime was decided by the file, not by preference.
+
+## Identity
+
+| | |
+|---|---|
+| Host / device | Dankclaw (192.168.0.143), Radeon 8060S iGPU `gfx1151`, ~114 GiB GTT aperture |
+| Endpoint | `http://192.168.0.143:7770` — alias `frank-large`, `/v1` OpenAI surface |
+| Runtime | **`kingjones30/ROCmFPX`** llama.cpp fork @ `36e9acd40e10a87cd3c3ef8ec734668757dc8520`, built by [`scripts/build-rocmfpx.sh`](../../scripts/build-rocmfpx.sh) → `/home/dank/GitPackages/kingjones-ROCmFPX/build/bin/llama-server` |
+| Unit | root systemd `frank-large.service` ([`scripts/frank-large.service`](../../scripts/frank-large.service)), `User=dank`, `Restart=always` |
+| Serve script | [`scripts/serve-frank-large.sh`](../../scripts/serve-frank-large.sh) (owns the argv), tunables [`scripts/frank-large.env`](../../scripts/frank-large.env) |
+| Weights | `/home/dank/models/kingjones-Qwen3.8-27B-ROCmFP4-STRIX-MTP/Qwen3.8-27B-Q8_0_ROCMFPX_AGENT.gguf` — 28,214,425,952 B, sha256 `f3ef2bdbf56e244f…` |
+| Draft | `…/mtp-Qwen3.8-27B-Q4_0.gguf` — 1,680,271,648 B, sha256 `051a1764cff8c4f3…` |
+| Projector | `…/mmproj-Qwen3.8-27B-BF16.gguf` — 931,145,888 B, sha256 `de2a4986…` |
+| Fetch | [`scripts/download-frank-large-kingjones.sh`](../../scripts/download-frank-large-kingjones.sh) (`-f`, byte-count + LFS sha256 verified, `--verify-only`, token never on the command line) |
+| Seam | `tests/frank_large_unit.bats` (19 tests: payload gates, deployer/serve parity, spec block, KV dtype, sweep identity) |
+
+## Why this file, and why only this runtime
+
+`kingjones777/Qwen3.8-27B-ROCmFP4-STRIX-MTP-GGUF` ships several rungs. Parsed
+headers of the two candidates:
+
+| File | ftype | Tensor mix | Output-head tensors at plain Q8_0 | Card MTP acceptance |
+|---|---|---|---|---|
+| `Qwen3.8-27B-Q8_0_ROCMFPX_AGENT.gguf` | **115** | 353 F32 + 187 Q8_0 + **311 type-103** | **187** | 0.953 |
+| `Qwen3.8-27B-Q8_0_ROCMFPX.gguf` | 111 | 2 Q8_0 + type-103 | 2 | 0.911 |
+
+Two consequences, both measured off the header rather than guessed:
+
+1. **Quality.** The AGENT routing variant keeps the output-head (and other
+   routing-sensitive) tensors at plain Q8_0 instead of folding them into the
+   ROCmFPX kernel layout — 187 vs 2 — so the logits the sampler reads are the
+   highest-precision tensors in the file. It is also the card's MTP-tuned
+   variant, so it is the *faster* drafter base at the same time.
+2. **The runtime is forced.** Type 103 is a private extension: stock llama.cpp
+   and gufo both fail at load (`unknown tensor type`). Verified on the host
+   before deploying — the bake-off was not a horse race, it was a lock-in
+   check.
+
+**The build identity matters as much as the fork.** The 2026-08-25 ROCmFPX
+builds read this file and then died in `ggml_compute_forward_set_rows` — the
+exact failure documented in [[kingjones-q8agent-rocmfpx]] when this payload was
+first attempted on Dankclaw and routed to Dankserver as
+[[frank-medium-rocmfpx]] instead. `scripts/build-rocmfpx.sh` therefore checks
+the commit out **by SHA** and refuses to build anything else; the pin is
+load-bearing, not tidiness.
+
+## Shape and budget
+
+One session, the model's full native window, unquantized KV:
+
+| Dial | Value | Why |
+|---|---|---|
+| `--ctx-size` | 262,144 | "one full context" — the native window, no rope scaling |
+| `--parallel` | 1 | one conversation, all memory to it |
+| KV dtype | **f16** (both K and V) | ~64 KiB/token → ~16 GiB; free inside a 114 GiB aperture, so there is no reason to pay a quant |
+| `--fit off` | on | the auto-fitter would silently resize the window we asked for |
+| `--flash-attn on` | on | required for the long-window attention path to fit |
+| `--direct-io` | on | bypasses page cache for the 26 GiB weight stream (mmap is disabled by it — deliberate) |
+| `--ctx-checkpoints 16` | 16 | slot state for checkpoint/rollback |
+| `--cache-ram 8192` + `--cache-idle-slots` | 8 GiB | prompt cache; verified a repeat pass over the same 6k prompt replays in **0.5 s** |
+| `--jinja` | on | without it `chat_template_kwargs` are silently ignored |
+| `--reasoning on --reasoning-format deepseek --reasoning-budget -1` | | thinking in its own channel, unbounded |
+
+Measured steady state (production, one model copy resident):
+`mem_info_gtt_used` **63.1 GiB** of the ~114 GiB aperture, `mem_info_vram_used`
+0.17 / 2.0 GiB (the iGPU's dedicated carve-out is a stub; everything real lives
+in GTT). llama-server RSS 2.1 GiB. Boot to `/health` 200: **16 s** — the fastest
+endpoint bring-up in the fleet, because there is no engine compile step.
+
+## Speculative decode — measured, not assumed
+
+```
+--spec-type draft-mtp --model-draft <mtp-Qwen3.8-27B-Q4_0.gguf>
+--spec-draft-n-max 4 --spec-draft-n-min 0 --spec-draft-p-min 0.0
+--spec-draft-device ROCm0 --spec-draft-ngl 99
+```
+
+MTP in llama.cpp is a property of the file, not of the flags: the head is a
+trained tensor set (`nextn_predict_layers = 1` in this repo's head, arch
+`qwen35`, 65 blocks) and the draft context runs on the main model's weights, so
+there is no second 26 GiB copy in memory. See the fleet law in
+`AGENTS.md → llama.cpp serving`.
+
+**A/B, 2026-10-02** — same session, temp 0, 2,202-token prompt, 512 tokens,
+`cache_prompt: false`, `frank-large.service` stopped for every scratch arm (the
+benchmarking law), baseline arm stripped of `--model-draft` **and** every
+`--spec-*` flag:
+
+| Arm | Decode t/s (2 runs) | Prefill t/s | Draft acceptance |
+|---|---|---|---|
+| **MTP, n-max 4 (deployed)** | **17.00 / 17.14** | 350.6 / 324.9 | 0.462 (mean acceptance length ≈ 1.86) |
+| no draft | 7.60 / 7.66 | 381.8 / 381.0 | — |
+
+**+123 % decode for an 8–15 % prefill tax.** Notes that matter before someone
+re-litigates this:
+
+- Acceptance here (0.462) is nowhere near the card's 0.953. The card measures
+  its own short, repetitive text; this is a 2.2k-token prose prompt. Treat the
+  card's number as a file property, not a throughput promise.
+- `n-max 4` is where the gain was measured; 2 and 8 are **unmeasured**. The
+  gufo-era law applies in spirit — never rank spec settings on a
+  self-predictable corpus.
+- End-to-end on the chat path is slower than the `/completion` number because
+  thinking tokens are included: a 2,253-token chat prompt generating 512
+  tokens took 43.8 s wall (11.7 t/s end-to-end) — that is the cost of an
+  unbounded reasoning budget, not a regression in decoding.
+
+## Client contract (live-verified 2026-10-02)
+
+| Behavior | Observation |
+|---|---|
+| Model id | `/v1/models` → `frank-large` |
+| Tools | native: `get_weather` → `{"city": "Seattle"}` in `tool_calls` |
+| Thinking | in **`reasoning_content`** (llama.cpp field — *not* vLLM's `reasoning`); `models.yml` sets `reasoningContentField` accordingly |
+| `reasoning_effort` | accepts **only** `xhigh`, `medium`, `low`; anything else is HTTP 500 (the template raises, it does not clamp). `models.yml` folds `high`/`max` → `xhigh` |
+| Sampling | server defaults = the card's thinking recipe (temp 1.0, top_p 0.95, top_k 20, min_p 0, presence 0), mirrored in the harness `extraBody` |
+| Vision | the BF16 projector loads (`loaded multimodal model`); a red square answers **`Red`** (`usage: 75 prompt / 71 cached`) |
+| Prompt cache | second pass over the same 6k prompt: 0.5 s wall |
+
+Three traps, all hit for real:
+
+1. **`content: null` when reasoning eats the budget.** With
+   `--reasoning-parser`-style separation, a request whose `max_tokens` is
+   consumed entirely by thinking returns `message.content = None`. A bench
+   client that indexes `content` crashes *after* measuring and loses the
+   numbers — fall back to `reasoning_content`, or size the budget (200 was
+   enough for the vision probe).
+2. **`MTP boundary missing for seq_id=… ; resyncing after a non-token step`**
+   appears in the log under draft resets. Self-recovering; not an outage signal.
+3. **Never `pkill -f llama-server` on Dankclaw** — frank-small is also a
+   `llama-server`. The sweep key here is the payload name:
+   `pkill -f 'Qwen3.8-27B-Q8_0_ROCMFPX_AGENT'`.
+
+## Operations
+
+```bash
+# deploy (planner selects this endpoint only via its own unit/env/serve trio)
+./scripts/deploy.sh dankclaw --service frank-large
+
+# status / logs / cause of a crash loop
+systemctl status frank-large
+journalctl -u frank-large -n 200 --no-pager
+
+# rebuild the pinned fork (only when deliberately moving the pin)
+scripts/build-rocmfpx.sh
+
+# re-verify the payload without re-downloading
+scripts/download-frank-large-kingjones.sh --verify-only
+```
+
+`serve-frank-large.sh` refuses to start — before touching the port — unless the
+fork binary, the weights, the MTP head and (unless `FRANK_LARGE_MMPROJ=""`) the
+projector all exist; `scripts/deploy-remote.sh` mirrors the same predicate so a
+broken payload never stops the running unit. Caller environment **overrides**
+`frank-large.env`, and defaults use `:=` so an explicitly empty value surfaces
+as the error it reads as.
+
+## Rollback (as of retirement, 2026-10-03)
+
+**Primary rollback: the ALIGNED Q8 file itself is the previous shape's
+payload and remains on disk** at
+`/home/dank/models/kingjones-Qwen3.8-27B-ROCmFP4-STRIX-MTP/` — revert the
+2026-10-03 swap commit (unit/env/serve trio) or repoint
+`FRANK_LARGE_MODEL_FILE`/`LLM_LARGE_MODEL_FILE`, verify with
+`scripts/download-frank-large-kingjones.sh --verify-only`, redeploy.
+
+git tag **`frank-large-gufo-q4kxl-20261001`** (gufo v0.1.1, Q4_K_XL +
+DFlash2) is now a **re-download** rollback: the gufo payloads
+`/home/dank/models/Qwen3.8-27B-UD-Q4_K_XL/` (16.35 GiB) and
+`/home/dank/models/Qwen3.8-27B-UD-Q8_K_XL/` (incl. the DFlash2 drafter) were
+removed from disk on 2026-10-03 with Frank's explicit approval to free space.
+`scripts/build-gufo.sh` remains on disk; re-fetch per
+[[frank-large-gufo-qwen38-27b]].
+
+The one-day vLLM MiMo era's payloads (MiMo-V2.6 BF16 18 GiB,
+Qwen3.5-9B MTP source 19 GiB, both DFlash draft dirs) were staged to
+`/home/dank/.AGENTTRASH/20261002-retired-llm-payloads/` on 2026-10-02 —
+recoverable until the trash is purged, and never part of a rollback plan.
+
+## When it misbehaves
+
+- **`unknown tensor type` / ftype 115 refused** — something moved off the
+  ROCmFPX binary. Check `readlink -f $(systemctl show frank-large -p ExecStart)`
+  and the fork commit; this file cannot run on stock llama.cpp or gufo.
+- **SIGSEGV in `ggml_compute_forward_set_rows` during load** — a fork build
+  older than the `36e9acd` pin; rebuild with `scripts/build-rocmfpx.sh`.
+  History: [[kingjones-q8agent-rocmfpx]].
+- **Slow decode with no log error** — check whether the draft is engaged
+  (`/metrics`, `draft_n` / `draft_n_accepted`; a zero draft count means the head
+  did not load), and check GTT pressure with
+  `cat /sys/class/drm/card1/device/mem_info_gtt_used` — a second resident copy
+  of this model (an orphaned scratch server) drives the whole host into
+  eviction and halves the rate silently.
+- **`500` naming Jinja** — a `chat_template_kwargs` field reached a server
+  without `--jinja`.
+- **HTTP 500 naming the template on the first request of a role** — an
+  unmapped `reasoning_effort`; fix the fold in `models.yml`, not the model.
@@ -0,0 +1,186 @@
+---
+type: reference
+title: "frank-large on Dankclaw — vLLM TheRock, MiMo-V2.6-Distill-Qwen-9B BF16 + cross-model MTP (RETIRED)"
+description: "The RETIRED frank-large runbook (2026-10-01, one day) — the digest-pinned kyuz0 TheRock gfx1151 vLLM image serving XiaomiMiMo MiMo-V2.6-Distill-Qwen-9B in full BF16 on the Strix Halo iGPU. Kept for the measured record: the roof math, the cross-model MTP drafter, and the autoresearch finding that spec decode is NET-NEGATIVE on real chat+reasoning traffic here (6.19 t/s with MTP-2 vs 9.01 t/s without). Rolled back to [[frank-large-gufo-qwen38-27b]] by Frank the same night."
+tags:
+  - llm
+  - frank-large
+  - dankclaw
+  - vllm
+  - strix-halo
+  - mimo
+  - mtp
+  - spec-decode
+  - bf16
+timestamp: 2026-10-01
+---
+
+# frank-large — vLLM MiMo-V2.6-Distill-Qwen-9B BF16 (cross-model MTP)
+
+> **RETIRED 2026-10-01 night, one day active.** Frank's call after the
+> autoresearch loop: battery decode 6.19 t/s (MTP-2) / 9.01 t/s (no draft)
+> against gufo's ~15 t/s on the 27B — and every ≥6-bit quant path in vLLM
+> 0.30 is dead on gfx1151 (quark int8 rejected; all FP8 kernels CDNA-only;
+> torchao config unwired). Back to [[frank-large-gufo-qwen38-27b]]
+> (Q4_K_XL, 1 x 262144). The measured content below stands as the record.
+
+## Identity
+
+- **Endpoint:** `frank-large` at `192.168.0.143:7770` (Dankclaw).
+- **Runtime:** vLLM 0.30.0 in the **digest-pinned** `docker.io/kyuz0/vllm-therock-gfx1151`
+  image (image ID `2d24a8c5a3fd`, digest `sha256:98c1f3ac9833…` — ROCm 10.0.0 /
+  torch 2.11.0 / TheRock, Triton Gated DeltaNet kernels). The official vLLM
+  images are CUDA-only; this fork is the fleet's gfx1151 path. Plain
+  `podman run` driven by `scripts/serve-frank-large.sh` under a root unit
+  (`scripts/frank-large.service`, `User=dank`, `Restart=always`,
+  `ExecStop=podman stop -t 120`, `TimeoutStopSec=180` > the stop grace).
+- **Model:** `XiaomiMiMo/MiMo-V2.6-Distill-Qwen-9B` in **full BF16**
+  (17.53 GiB, 4 shards) — the same checkpoint frank-small serves as an INT8
+  W8A16 AutoRound conversion on the 3090, here unquantized.
+- **Draft:** `Qwen/Qwen3.5-9B` (18 GiB on disk; only the 15 `mtp.*` tensors
+  are loaded into the draft pass).
+
+## Why this shape (the roof math)
+
+Decode speed for a dense model on Strix Halo is bandwidth-bound:
+BF16 weights x one pass per token against the LPDDR5x aperture. Measured
+2026-10-01 on this host: **13.1 t/s is ~94% of the roof** — the payload, not
+a misconfiguration. The same INT8 payload reaches 83.9 t/s on the 3090
+because it moves ~12 GiB through ~936 GB/s. The only way past a bandwidth
+wall is generating more than one token per weight pass: **speculative decode
+is the performance feature of this endpoint**, not a garnish.
+
+The memory side is the opposite of a squeeze: KV grows ONLY on the 8 full
+attention layers (32 KiB/token bf16: 8 x 2 x 4 heads x 256 dim x 2 B);
+the 24 Gated DeltaNet layers hold a FIXED ~25 MiB state per running
+sequence. `--max-model-len 262144` (the native window) with
+`--max-num-seqs 4` at `--gpu-memory-utilization 0.45` boots clean — the
+four-session contract the gufo shape ran, with the pool shared by the
+scheduler instead of carved per session.
+
+## Spec method: cross-model MTP — measured, not assumed
+
+| arm | prefill | decode | acceptance |
+|---|---|---|---|
+| BF16 base | 17,876 t/s | 13.1 t/s | — |
+| BF16 + DFlash 8 tok | 7,976 t/s | 17.4 t/s | pos0 59% pos1 26% (token rate 14.8%) |
+| **BF16 + MTP 2 tok** | 7,380 t/s | **19.8 t/s** | pos0 71% pos1 44% (token rate 57.4%) |
+
+One session, temperature 0, 28,647-token prefill / 512-token decode, same
+client for all three arms (2026-10-01 bench window, frank-large stopped per
+Benchmarking Policy). **MTP wins (+51% vs +33%) and needs only 2 draft
+tokens** — acceptance past position 1 is 44% and falling; DFlash's own curve
+decays 59/26/14/7, so 8 proposals are mostly wasted verification.
+`num_speculative_tokens` beyond 2 is UNMEASURED; do not raise it without a
+fresh A/B. Both spec arms pay ~55–59% of prefill (the draft passes share
+the compute queues) — this endpoint's thinking-heavy traffic spends its wall
+clock in decode, which is the regime where that trade wins.
+
+**Why a Qwen head accepts on a MiMo target:** MiMo-V2.6 is a distillation of
+Qwen3.5-9B that preserves the hidden-state geometry — identical
+4096-hidden/248320-vocab/32-layer/interval-4 structure, and the 71%/44%
+per-position acceptance is the direct measurement that the distillation
+preserved more than geometry. MiMo itself ships **ZERO `mtp.*` tensors**
+(verified in both the BF16 base and the INT8 conversion) — which is exactly
+why frank-small's SGLang NEXTN draft is a 0.00-acceptance ghost: SGLang
+reads the head from the TARGET checkpoint, vLLM reads the draft from a
+separate path. That difference is the whole trick:
+
+```
+--speculative-config '{"method":"mtp","model":"/models/Qwen3.5-9B","num_speculative_tokens":2}'
+```
+
+The alternative drafter (`z-lab/Qwen3.5-9B-DFlash`, 6 layers, target_layer_ids
+[1,5,…,29]) stays fetched on disk; flip `FRANK_LARGE_SPEC=dflash` and repoint
+`FRANK_LARGE_SPEC_DIR` to use it. Its card's "vLLM PR #40898 unmerged" claim
+is FALSE — `vllm/v1/spec_decode/dflash.py` ships in 0.30.
+
+## The numerics trap (greedy text is NOT preserved by spec decode)
+
+Measured: temperature-0 completions DIFFER across arms (sha `1318aeb6` base,
+`0208b45d` dflash, `20c41c38` mtp). Batched verification changes the bf16
+reduction order, argmax near-ties diverge, and the divergence cascades.
+vLLM spec decode is distribution-preserving, not bitwise-greedy-preserving.
+The gufo-era "greedy == autoregressive text" law does not carry over; do not
+chase text drift between spec and no-spec arms as a regression.
+
+## Payload
+
+- Target: 4 safetensors shards (17.53 GiB) + config/tokenizer under
+  `/home/dank/models/MiMo-V2.6-Distill-Qwen-9B-BF16/`, each shard
+  byte-count + LFS sha256-verified by
+  `old/download-frank-large-mimo.sh` (`--verify-only` re-checks; no token on
+  the curl command line).
+- MTP source: `old/download-frank-large-qwen35.sh` →
+  `/home/dank/models/Qwen3.5-9B/` (4 shards behind
+  `model.safetensors.index.json`; the shards carry the odd HF asset names
+  `model.safetensors-0000N-of-00004.safetensors` — the index resolves them,
+  the engine boots from exactly this tree).
+- DFlash drafter: `old/download-frank-large-dflash.sh` →
+  `/home/dank/models/Qwen3.5-9B-DFlash/`.
+- **Payload status 2026-10-02:** all three trees moved to
+  `/home/dank/.AGENTTRASH/20261002-retired-llm-payloads/` (59 GiB staged with
+  the other retired frank payloads) and the fetch trio moved to `old/` — the
+  endpoint is two runtimes past this one, and re-fetch is cheaper than disk.
+- The serve script gates the image by REPO DIGEST (rootless podman resolves
+  tags from the network; the pin exists to boot exactly these bytes
+  offline), then config + index + exact shard count, then the spec draft
+  (either weight layout: single `model.safetensors` or a sharded index),
+  then `/dev/kfd` + whole `/dev/dri` (never individual renderD nodes —
+  Dankserver 2026-09-29 lesson).
+
+## Client contract
+
+- **Alias:** `frank-large` (the `id` at `/v1/models`).
+- **Reasoning:** `--reasoning-parser qwen3`; vLLM emits the thinking text in
+  **`reasoning`**, NOT llama.cpp/gufo's `reasoning_content` — the harness
+  `models.yml` compat block for this provider sets
+  `reasoningContentField: "reasoning"`. Miss that and thinking silently
+  vanishes from transcripts.
+- **Tools:** `--enable-auto-tool-choice --tool-call-parser qwen3_coder`.
+- **Vision:** the checkpoint carries the `qwen3_5_vision` tower but the
+  serve script passes `--language-model-only` — the endpoint stays
+  text-only by decision; dropping the flag brings vision back.
+- **Sampling:** request-side; the harness sends the Qwen3.5-family recipe
+  (temp 0.7, top_p 0.8) via `compat.extraBody`, and
+  `chat_template_kwargs.{enable_thinking,preserve_thinking}` — the
+  `chat_template_kwargs` form is what vLLM reads; a top-level
+  `preserve_thinking` is ignored there.
+- **Env hygiene:** every tunable is `FRANK_LARGE_`-prefixed because HIP
+  reads bare `TEMP`/`HOST`/`GPU_MAX_HEAP_SIZE` and the serve script exports
+  the env file with `set -a`.
+
+## Ops
+
+- Boot ~200–240 s warm; health 503 while loading (do not kill it).
+- The serve script sweeps stale container identities before starting:
+  `frank-large-mimo` itself and the bench-window container `vllm-mimo`.
+- Container death → `podman wait` returns non-zero from ExecStart →
+  `Restart=always` self-heals (a clean exit-0 drain also restarts, the
+  frank-primary lesson).
+- Bench windows: `systemctl stop frank-large` is enough (mask fails on the
+  relinked unit; the deploy restores).
+- First-deploy facts (2026-10-01, deploy #3): healthy after **230 s**;
+  boot log `GPU KV cache size: 819,332 tokens, Maximum concurrency for
+  262,144 tokens per request: 3.13x` — four sessions share the pool
+  (the scheduler, not a carve). Live battery green: alias `frank-large`,
+  reasoning split into `reasoning`, tool_calls with correct args, and
+  `/metrics` acceptance pos0 69% pos1 44% (54/86 draft tokens accepted),
+  matching the bench window.
+
+## Rollback
+
+Git tag **`frank-large-gufo-q4kxl-20261001`** — gufo v0.1.1 (built from
+source, `scripts/build-gufo.sh` stays on disk) + the Q4_K_XL Qwen3.8-27B
+payload + DFlash2 drafter, all still on the host. Restoring the tag's
+unit/env/serve trio and redeploying returns the 27B endpoint.
+
+# Citations
+
+- Model: https://huggingface.co/XiaomiMiMo/MiMo-V2.6-Distill-Qwen-9B
+- Draft source: https://huggingface.co/Qwen/Qwen3.5-9B
+- DFlash drafter: https://huggingface.co/z-lab/Qwen3.5-9B-DFlash
+- Engine image: https://hub.docker.com/r/kyuz0/vllm-therock-gfx1151
+- Serve script: `scripts/serve-frank-large.sh`; tunables: `scripts/frank-large.env`
+- Predecessor: [[frank-large-gufo-qwen38-27b]]; sibling on the same arch:
+  [[frank-small-sglang-mimo-v26]]
@@ -0,0 +1,706 @@
+---
+type: runbook
+title: frank-primary — GLM-5.3-Flash W4A16 int4 (tcclaviger, 2x R9700)
+description: VERIFIED 2026-09-30 by boot probe, bench, live expert-cache telemetry AND a torch-profiler kernel-time window. frank-primary cut over from Qwen3.8-Flash-Next-Uncensored-FP8 + MTP4 to Intel/GLM-5.3-Flash-W4A16-AutoRound, the first payload this fleet serves through the fork's W4A16 expert-offload path (INC scheme -> R4dW4a16MoEMethod, libr4d grouped W4A16 GEMM). Measured decode 9.9-10.0 tok/s and prefill 918 tok/s at 131,072 ctx. THE 35 tok/s TARGET IS NOT MET, and the profiler window FALSIFIED the first explanation: the expert gather is 1.05 % of device time, not the bound. 44 % of the 93 ms/token budget is pointwise glue (1,428 elementwise launches per token) because the fork's GLM trunk is NOT torch.compiled while its own Qwen trunk is; the second-largest lever, the fused router, is binary-blocked at 288 experts. The compile changelist is staged but incomplete (three patches, one blocker left).
+tags:
+  - dankserver
+  - r9700
+  - gfx1201
+  - glm5.3
+  - w4a16
+  - int4
+  - expert-offload
+  - expert-cache
+  - autoround
+  - vllm
+timestamp: 2026-09-30
+---
+
+# frank-primary — GLM-5.3-Flash W4A16 int4 (tcclaviger, 2x R9700)
+
+> **RETIRED 2026-10-01.** Cutover reversed to Qwen3.8-Flash-Next-Uncensored-FP8
+> + MTP4 (commit `2339fae` trio). The DFlash2 drafter experiment is a
+> **failure**: after five patches the drafter served and drafted (12.8 tok/s
+> proposed, 65,536 ctx, both TP ranks capturing FULL dflash2 cuda graphs) but
+> **acceptance pinned at 1.00-1.38 on every text class including verbatim
+> repetition** — the signature of a mis-wired aux-hidden-state capture, not a
+> weak drafter. At ~1.2 acceptance the 8-token verification tax makes decode
+> 5x SLOWER than the 9.9-10.0 no-draft baseline (1.6-1.8 tok/s measured). The
+> untested alternative capture semantics: post-layer output state
+> (DeepSeek's `hidden_states + residual`) or the mHC `comb` stream. Payload
+> (170 GiB) + drafter (2.2 GiB) staged to
+> `/home/dank/.AGENTTRASH/20261001-glm-purge/` pending purge approval.
+> KV-cache work that survives: patches 02/03/04 in `scripts/patches/` are
+> generic unifier/tailpin/grouping fixes, apply-verified against the image.
+
+
+> **Question this page answers.** Can the fleet serve the 320B-class
+> `GLM-5.3-Flash` at int4 on two R9700s with expert offload, and how fast is it?
+> **Answer.** Yes — it serves, with vision, tool calls and 131,072 ctx. It runs
+> at **9.9-10.0 tok/s decode / 918 tok/s prefill**, and **35 tok/s is not
+> reachable** on this payload as the fork ships it. The bound is **not** the
+> model, the quant, the link, or the expert gather (measured: 1.05 % of device
+> time). It is the **unfused pointwise glue of an uncompiled trunk** — 44 % of a
+> 93 ms per-token budget spread over ~1,428 elementwise launches per token —
+> because the fork compiles its own Qwen trunk and not its GLM one. See
+> [The measured profile](#the-measured-profile-and-the-real-wall).
+
+Related: [[dankserver-qwen38-fp8-expert-offload-tuning]] (the same fork, the same
+hardware — its conclusion was drawn from throughput arithmetic and is superseded
+by this page's kernel-time measurement),
+[[dankserver-frontier-moe-feasibility]] (the earlier fit analysis),
+[[VLLM Index]], [[LLM/index|LLM fleet]].
+
+## The deployed shape
+
+| term | value |
+| --- | --- |
+| payload | `Intel/GLM-5.3-Flash-W4A16-AutoRound` — AutoRound **symmetric int4, group 128**, packing `auto_round:auto_gptq`, 34 shards + 2 extra safetensors, 113,074 tensors, ~170 GiB |
+| runtime | `docker.io/tcclaviger/vllm@sha256:da4029666…` (same digest-pinned image as the qwen cutover) |
+| engine | TP=2, `--enable-expert-offload --expert-precise-memory`, max-model-len 131,072 x 2 seqs, bf16 KV, util 0.98, NBATCH 2048, prefix caching |
+| MoE kernel | `R4dW4a16MoEMethod` → libr4d grouped W4A16 GEMM, 6.234 MiB per expert-rank slot |
+| expert cache | 14.51 GiB = **2,383 slots** per rank; 12,096 host-backed experts, **73.64 GiB pinned per rank** |
+| unit | `scripts/frank-primary.service` (root unit, `LimitMEMLOCK=infinity` for the pin) |
+| measured | **decode 9.9-10.0 tok/s**, **prefill 918 tok/s** (warm, 2.6k-token prompt), KV cache 138,561 tokens |
+
+```mermaid
+graph LR
+    A[Intel AutoRound W4A16<br/>sym int4 g128, auto_gptq] -->|quant_method auto-round| B[inc.py override to inc]
+    B --> C[inc_wna16_scheme<br/>_resolve_gptq_moe]
+    C -->|sym int4 g128 + gfx1201 + libr4d| D[R4dW4a16MoEMethod<br/>grouped W4A16 GEMM]
+    D --> E[ExpertLRUCache<br/>2,383 VRAM slots / rank]
+    D --> F[73.64 GiB pinned host rows / rank]
+    E --> G[decode step]
+    F -->|host misses<br/>1.05% of device time| G
+```
+
+## Why this payload and not another
+
+The W4A16 fast path on this fork is reachable **only** through the INC
+(AutoRound) quant method. `compressed_tensors` checkpoints of the same model —
+for example `dealignai/GLM-5.3-Flash-CYBERSECURITY-W4A16`, 182 GiB — resolve
+their MoE through the WNA16 oracle to `TritonWNA16Experts`, which **has no
+offload recipe**, and the engine refuses to serve at all: 170 GiB of experts
+cannot fit 64 GiB of VRAM, so offload is mandatory, not optional.
+
+The chain that makes Intel's checkpoint work, with the exact evidence:
+
+```
+[expert_plan.py]  quantization=inc            ← inc.py:364 overrides "auto-round"
+[inc_wna16_scheme.py:249]  INC GPTQ MoE on ROCm: R4dW4a16MoEExperts (libr4d grouped W4A16 GEMM)
+[r4d_w4a16_moe.py:682]     Using MoEPrepareAndFinalizeNoDPEPModular / R4dW4a16MoEExperts
+[expert.py:450]            Expert offload: 42 MoE layers staged; 2,383 cache slots in 14.51 GiB planned per rank
+[expert.py:413]            42 layers split: 0 VRAM-only experts, 2,383 swap slots, arena 14.51 GiB,
+                           host 73.64 GiB pinned, host rows prepacked on the GPU
+```
+
+`R4dW4a16MoEExperts` is what the offload machinery wraps: the offload plan is
+format-agnostic but requires the experts class to expose the hot/cold split.
+
+## The numbers, and the wall
+
+Measured on the deployed engine, 2026-09-30 (`bench_glm.py`, real prose, greedy):
+
+| depth | prefill | decode |
+| --- | --- | --- |
+| 2,621 tok prompt, run 1 (cold cache) | 259 tok/s | 9.88 |
+| 2,621 tok prompt, runs 2-3 (warm) | 918 / 885 tok/s | 9.91 / 9.41 |
+| 493 tok prompt (telemetry window) | 124 tok/s | 8.90 |
+
+Decode is flat across context and across every arena change — the signature of a
+per-step cost that is not KV- or residency-driven at the margin.
+
+### The measured profile, and the real wall
+
+**This section replaces an earlier explanation, which the measurement falsified.**
+The first reading of this page bounded the step with throughput arithmetic: 42 MoE
+layers x 8 experts x 6.234 MiB = 2.045 GiB per step per rank, 0.82 GiB of it
+missed at a measured 0.53-0.66 hit rate, giving "0.82 GiB in ~100 ms = 8.2 GB/s =
+the gather is the wall". That arithmetic was circular — it *assumed* the gather
+consumed the whole step — and it is wrong. The engine, asked directly, disagrees:
+
+A torch-profiler window (`--profiler-config`, `POST /start_profile` /
+`stop_profile`, 496 decode tokens at the deployed shape, rank0 trace parsed for
+device-activity time) gives the true split. Total device kernel time **46.18 s /
+496 tokens = ~93 ms per token** against a measured ~104 ms step (9.6 tok/s with
+the profiler attached):
+
+| ms | % | launches | kernel |
+| --- | --- | --- | --- |
+| 16,074 | **34.8** | 708,500 | `vectorized_elementwise_kernel` (22.7 us each) |
+| 4,602 | 10.0 | 42,000 | `r4d_gemm_moe_w4a16_nt_b16_m64` — the MoE GEMM |
+| 3,108 | 6.7 | 174,708 | `elementwise_kernel_manual_unroll` |
+| 2,943 | 6.4 | 138,252 | `wvSplitK_hf_sml_` — split-K GEMM (projections / head) |
+| 2,829 | 6.1 | 83,160 | router: `sbtopk::gatherTopK` + `warpMergeSortTopK` + `bitonicSortKVInPlace` (1,208 + 1,051 + 559 ms; the 1,051 row is not in the list below) |
+| 3,015 | 6.5 | 133,920 | `r4d_mhc_pre_finish/pre_partial/post` — hyper-connections |
+| 2,782 | 6.0 | 127,916 | inductor `triton_poi_*` / `triton_per_*` (the fusions that DO happen) |
+| **487** | **1.05** | 20,664 | **`r4d_moe_lru_gather_kernel`** — the accused |
+| 250 | 0.54 | 20,832 | `_moe_lru_tele_kernel` — the telemetry that measured it (TELEM was on) |
+
+So the gather is **1.05 %** of device time, and `r4d_moe_lru_fused_kernel` (the
+LRU *manager*, not the copy) is a further 1,599 ms / 3.5 %. Residency work and the
+PCIe link are not the story: 44 % of the budget is **pointwise glue** — bf16<->fp32
+conversions, adds, `pow`/`rsqrt` from fp32 RMSNorm, `launch_clamp_scale` quant
+passes, `reduce_kernel` means — spread over **~1,428 elementwise launches per
+token** (3,734 kernel launches per token in total). The GPU is busy 86 % of the
+wall clock, so this is real device work, not launch-gap overhead.
+
+Two consequences that kill three whole families of planned work:
+
+- **Spec decode cannot help this shape, and the MTP measurement was not an
+  accident.** Pointwise cost scales with the token count, so a draft batch that
+  verifies 4 tokens pays ~4x the glue for the same per-token output — which is
+  exactly why MTP4 measured "acceptance 3.76-4.90, decode unchanged". Same reason
+  the gather was never the bound.
+- **Arena size and hit rate are not levers.** Confirmed twice: `--expert-precise-memory`
+  (arena 13.08 -> 14.51 GiB) moved decode by nothing, and the gather is 1 %.
+
+## Fork work: the compile changelist (WIP, does not boot yet)
+
+The fork compiles its own flagship trunk — `vllm/models/qwen4_exp/amd/model.py`
+carries `@support_torch_compile` — and **not** its GLM trunk. The engine says so on
+every boot:
+
+```
+Cudagraph mode FULL_AND_PIECEWISE requires piecewise capture, but the loaded model
+provides neither a compiled submodule nor breakable CUDA graphs.
+Overriding to FULL_DECODE_ONLY.
+```
+
+Inductor fusion is therefore the only lever that touches the 44 % pointwise mass.
+Three patches are staged in `scripts/patches/glm53-compile-0*.patch`, each carrying
+its own evidence header, base/patched sha256 and the next blocker:
+
+| patch | target | fixes |
+| --- | --- | --- |
+| `-01-model-decorator` | `models/glm5next/common/model.py` | decorates `Glm5NextModel` (env `R4D_GLM_COMPILE`, default on) |
+| `-02-trace-safe-probes` | `model_executor/kernels/r4d_lib.py` | freezes 10 capability probes that do `os.path.exists`/`.so` lookups **inside the traced forward** |
+| `-03-fullgraph-optout` | `compilation/wrapper.py` | makes `fullgraph=True` opt-out (env `VLLM_COMPILE_FULLGRAPH=0`) |
+
+What the boots established, in order (each one ~7 min):
+
+1. **01 alone dies** on the first cold probe call inside the traced region:
+   `Dynamo does not know how to trace the builtin 'posix.stat'` —
+   `r4d_mhc_supported -> r4d_lib.r4d_mhc_enabled -> available -> os.path.exists`.
+   **`functools.lru_cache` does not protect these probes**: dynamo re-traces the
+   wrapped body to build its guard even when the cache is warm (it warns about
+   exactly this), verified in-image — a *warm* lru_cache-calling function still
+   fails `fullgraph=True`, while a plain module-global read compiles.
+2. **01+02 get past it** (`libr4d capability probes frozen: 10 warm [...]` in all
+   four processes) and die on
+   `RuntimeError: Graph breaks are not supported with aot compile`, because this
+   build runs AOT compile by default and AOT refuses breaks.
+3. **01+02+03 with `VLLM_COMPILE_FULLGRAPH=0 VLLM_USE_AOT_COMPILE=0` traces the
+   trunk successfully** (`Dynamo bytecode transform time: 1.08 s`) and then dies on
+   the decode forward with `KeyError: 'in_proj_qkvbfg_a'` (`Glm5NextModel.forward`
+   line 748 calling the decoder layer at line 456). `in_proj_qkvbfg_a` is the KDA
+   layer's merged column-parallel projection (`kda.py:217`, called at :407) — a real
+   submodule attribute name, also present in the `load_weights` shard table
+   (`model.py:780-785`). Reading the full traceback (the reported line is inside
+   dynamo-rewritten code) traces the throw to dynamo's own name→module map:
+   `torch/_dynamo/output_graph.py` registers into `nn_modules` at :1259/:1384,
+   looks up with `self.nn_modules[node.target]` at :3000 (a bare-name `KeyError`
+   when absent), and documents stripping `_buffers[..]/_parameters[..]/_modules[..]`
+   from those names at :1225. So the compiler is failing to **resolve a submodule by
+   name** while tracing the GLM decoder layer; whether the root cause is the fork's
+   module tree or dynamo's registration is not yet settled. The next step is the one
+   torch itself suggests in that error — `TORCHDYNAMO_VERBOSE=1` for the internal
+   stack trace (and note the launcher resets `ENVARGS`, so it must be added inside
+   `serve_glm_autoround.sh`'s COMPILE_PATCH branch rather than in the outer
+   environment).
+
+4. **04 removes the fused-norm call from the traced region** (`R4D_GLM_FUSE_QKV_RMSNORM=0`:
+   the GLM MLA block stops calling the fork's fused-norm JIT kernel, whose kernel
+   object in `models/common/ops/fused_qk_rmsnorm.py` is bound *after* the function
+   that uses it — `_FUSED_Q_KV_RMSNORM_KERNEL` at line 223 vs its use at line 211).
+   **It removes a real break** (under `fullgraph=True` the fused call was the first
+   break dynamo reported; after this patch the report moves on to the workspace
+   acquisition) — but it is not by itself what blocks a boot, because with breaks
+   genuinely tolerated the blocker is the name-map `KeyError`. See the run history
+   below.
+5. **A launcher bug that made four boots misleading — worth knowing because the
+   failure kept moving.** The scratch launcher's compile branches were restructured
+   on 2026-09-30 and the `-e VLLM_COMPILE_FULLGRAPH=0` line was dropped, so four
+   consecutive boots mounted the wrapper patch while `_FULLGRAPH` still defaulted to
+   True. With `fullgraph=True` **every** break is an error, so those boots reported
+   whichever break came first:
+
+   | config | outcome |
+   | --- | --- |
+   | `fullgraph=False`, AOT off | breaks tolerated → dies later on the dynamo name-map `KeyError: 'in_proj_qkvbfg_a'` |
+   | `fullgraph=True`, AOT off | dies on the first break (cell → workspace → compiler-disable) |
+
+   So the reported breaks are real *as breaks*, but they are not what blocks a
+   break-tolerant compile — and an earlier version of this page blamed
+   `TORCH_LOGS=graph_breaks` for the difference. **That was wrong**; the flag only
+   logs. The launcher sets the env var again.
+
+**How to see the breaks (and its trap).** `TORCH_LOGS=graph_breaks` names each break
+with a user stack, and `VLLM_USE_AOT_COMPILE=0` is required because AOT compile
+hard-refuses breaks — but a fullgraph=True run reports the first break as fatal, so
+check which configuration you are actually in (`podman exec <ctr> printenv
+VLLM_COMPILE_FULLGRAPH`) before concluding a break blocks the boot. For the name-map
+KeyError there is a ready-made probe: put `/home/dank/dynamo-probe` on `PYTHONPATH`
+with `VLLM_DYNAMO_DUMP_ATTRS=1` (a `sitecustomize.py` — a `.pth` on `PYTHONPATH` is
+NOT executed, and the image's own `sitecustomize` is chained so its apport hook
+survives) and it prints, at the miss, the node target, the near-miss keys, and
+whether another live `OutputGraph` holds the key.
+
+**State of the changelist (2026-09-30, latest).** The trunk **traces**
+(`Dynamo bytecode transform time: 1.08 s`) and the boot dies on the dynamo name-map
+`KeyError: 'in_proj_qkvbfg_a'`. Established about it, in order:
+
+- It is raised **inside the traced `Glm5NextDecoderLayer.forward`** — the frame chain
+  is `profile_run → _dummy_run → execute_model → glm4_1v.py:2358 → the compile wrapper
+  → eval_frame.compile_wrapper → Glm5NextModel.forward:748 (layer(...)) → module
+  _call_impl → Glm5NextDecoderLayer.forward:456 → KeyError`. `TORCHDYNAMO_VERBOSE=1`
+  adds **no** internal stack trace, i.e. it is a plain Python KeyError in the traced
+  frame, raised in a callee dynamo **inlined** (the reported line is the `def`, which
+  is how inlined frames surface).
+- It is **not** raised from any of dynamo's four `nn_modules` lookup sites:
+  `/home/dank/dynamo-probe` (a `sitecustomize.py` on `PYTHONPATH`,
+  `VLLM_DYNAMO_DUMP_ATTRS=1`) wraps three of them and stayed silent across two runs
+  with the probe confirmed installed in both the API-server and worker processes.
+- It is **not** a string-keyed lookup of a literal: `in_proj_qkvbfg_a` appears in the
+  whole tree only as the KDA layer's attribute (`kda.py:217`, used at `:407`) and in
+  the `load_weights` shard table (`model.py:780-785`). The four
+  `proj_name = prefix.split(".")[-1]` sites in the quantization utils do compute
+  exactly this bare key, but every one of them is guarded (`if proj_name not in
+  fused_mapping: return`), so they cannot raise it. The pluggable-layer registry is
+  keyed by class name, not module path.
+- It reproduces on **every** run of this configuration (5 of 5), and it is
+  independent of patches 04/05: with breaks genuinely tolerated, those remove breaks
+  that were never fatal in the first place — the KeyError is the only thing standing
+  between this changelist and a boot.
+
+**Next instrument, precisely:** the inlined caller has to be named. Two cheap ways:
+`TORCH_LOGS=+dynamo` on a boot and grep the log for the key (dynamo prints inlined
+frames with their source), or wrap the inlining candidates in that forward — the mHC
+CustomOps (`hc_pre` / `hc_post` / `hc_fused_post_pre`, `layers/mhc.py`) and the
+indexer op — with a probe that reports on `KeyError`. Until that names the caller, this
+is a dynamo-side name-resolution failure and not a fixable fork bug.
+
+Two verified facts bound the rest of the shape: `suppress_errors` does **not** cover
+graph breaks (only `fullgraph=False` does), and the GLM trunk can never satisfy
+`fullgraph` because its indexer metadata syncs to host
+(`glm5next/amd/sparse_indexer.py:263,419`: `int(...max().item())`) — both are dead
+fallbacks on the decode path that the fork author already replaced, so they are not
+the blockers they appear to be at first glance.
+
+### Two structural constraints on any further fork work
+
+- **No new HIP kernels are possible.** The image ships `r4d.so` as a binary with no
+  sources. The obvious second lever — the 9 % router — is gated
+  `num_experts in {128, 256, 512}` in `router/r4d_route_router.py` and the binary
+  contains only `r4d_moe_route_kernel<128|256|512>` instantiations. GLM has **288**
+  experts, so it falls back to the generic sort-based top-k and **cannot be moved to
+  the fused router without rebuilding libr4d**. Only Python/config/inductor changes
+  (and runtime-JIT Triton) are available.
+- **`--kv-cache-dtype fp8` is a dead end for capacity too.** The plan's KV reserve
+  is dtype-blind (1.76 GiB at 131,072 either way), so fp8 buys no arena and no
+  window; it only removes the only MLA decode path gfx12 has (trap 1 above).
+
+### DFlash2 for GLM — what the drafter actually is
+
+`incoai/GLM-5.3-Flash-DFlash2` is **not** a GLM-shaped drafter and needs no MLA/DSA
+port: its `config.json` declares `architectures: ["DFlash2DraftModel"]` and
+`model_type: "qwen3"` — 5 dense layers, hidden 4096, 32 heads / 8 kv heads,
+head_dim 128, sliding attention (window 2048), `is_causal: false`,
+`use_cache: false`, `dflash_config` = block 8, conv group 16, selector rank 256 /
+top-k 16, and **`target_layer_ids: [5, 14, 24, 33, 42]` against
+`num_target_layers: 45`**.
+
+The fork already maps that architecture:
+`registry.py:636 "DFlash2DraftModel": ("qwen3_dflash2", "DFlash2Qwen3ForCausalLM")`,
+and the spec method is `"dflash"` (`config/speculative.py:67`). So the port is
+**target-side only**: the GLM trunk must expose auxiliary hidden states at those five
+layer indices, the same interface `deepseek_v2.py` (also MLA+MoE) implements —
+`set_aux_hidden_state_layers()` / `get_eagle3_default_aux_hidden_state_layers()` plus
+capture at `idx in self.aux_hidden_state_layers` in the layer loop. `glm5next`
+implements neither.
+
+**The load was tried, and the gap is exactly one interface.** Payload fetched to
+`/home/dank/models/GLM-5.3-Flash-DFlash2` and hash-verified (`model.safetensors`
+2,342,169,800 B, sha256 `b038e1d9…` = the Hub's LFS oid). Booting the engine with
+`--speculative-config '{"method":"dflash","model":"/draft","num_speculative_tokens":8}'`
+accepts the config and the drafter and dies at worker init with:
+
+```
+RuntimeError: Model does not support EAGLE3 interface
+```
+
+So nothing about the drafter, the method name, or the payload is wrong — the GLM
+trunk simply does not implement the aux-hidden-state contract the dflash runner
+requires. The reference is in this same image, twice: `deepseek_v2.py`
+(`set_aux_hidden_state_layers` / `get_eagle3_default_aux_hidden_state_layers` at :1948-1951,
+capture at :1554 and :1579) and the fork's own `qwen4_exp` trunk. The contract is
+small:
+
+- `self.aux_hidden_state_layers: tuple[int, ...]` on the trunk, default `()`, set
+  by the wrapper's `set_aux_hidden_state_layers()`;
+- in the layer loop, `if idx in self.aux_hidden_state_layers: aux_hidden_states.append(<single-stream state entering layer idx>)`
+  (DeepSeek appends `hidden_states + residual`, with a TP all-gather fixup when the
+  rank holds fewer tokens than `positions`);
+- `Glm5NextModel.forward` returns `(hidden_states, aux_hidden_states)` when the
+  list is non-empty;
+- `get_eagle3_default_aux_hidden_state_layers()` supplies a default triple when the drafter
+  config omits `target_layer_ids` (this drafter provides `[5, 14, 24, 33, 42]`, so
+  the runner derives trunk indices `[6, 15, 25, 34, 43]`).
+
+**The GLM-specific unknown** is what "the single-stream state entering layer k"
+means for a trunk whose layers carry mHC multi-stream state: DeepSeek's
+`(hidden_states, residual)` pair has no counterpart here — GLM's decoder layer
+returns `(hidden_states, residual, post, comb)` with `hc_expand`/`hc_pre` mixing
+`hc_mult` streams, so the plausible answer is the pre-mHC single-stream input `x`
+(4096-wide, matching the drafter's `hidden_size`), not a post-layer value. That is
+a semantic choice to validate by acceptance rate, not by inspection.
+
+**Budget no speedup.** Per the profile above, a verification pass pays pointwise
+cost proportional to its token count — the same reason MTP4 measured flat — and
+MTP4's acceptance was already excellent (3.76-4.90). The port is worth doing as
+*enabling work* (it is the prerequisite for any spec decode on this endpoint, and
+it is small), not as a performance fix, and the honest expectation on this shape is
+flat decode with a prefill regression like MTP's until the pointwise mass is fused.
+
+**The port exists now: `scripts/patches/glm53-dflash-01-aux-hidden-states.patch`** (WIP,
+independent of the compile changelist — it patches the same file as compile-01, so
+compile-01 goes first if both are applied; base `008eeaa2…` → patched `bc9913a1…`). It
+implements exactly the contract read out of this image: `EagleModelMixin` on
+`Glm5NextModel`, capture of the single-stream `hidden_states` ENTERING each requested
+layer in the loop, the conditional `(hidden_states, aux_hidden_states)` tuple return, and
+the three `SupportsEagle3` member attributes plus both interface methods on the
+wrappers (capture shape from `deepseek_v2.py:1551-1585`; tuple consumption at
+`v1/worker/gpu/cudagraph_utils.py:621`).
+
+**The patch went through five versions, and the decisive discovery was where the
+instance actually lives.** The traceback's `set_eagle3_aux_hidden_state_layers(self.model)`
+is called on the model the runner LOADED — and for GLM-5.3-Flash that is
+`Glm5NextForConditionalGeneration` (the multimodal wrapper; the APIServer log names it),
+which holds the text model as `self.language_model = Glm5NextForCausalLM`
+(`glm5next/common/model.py:1113`). Patching only `Glm5NextForCausalLM` (v4) still failed
+with the identical error because the `isinstance` ran against the wrapper. **v5 fixes
+both wrappers** and is the first version to clear the gate (both TP workers log
+`Using Eagle3 auxiliary layers from config: (6, 15, 25, 34, …)`).
+
+The full evolution, each an instructive trap:
+
+1. **v1** hand-wrote the two methods the traceback names → identical error. The methods
+   are not the whole contract.
+2. **v2** inherited `SupportsEagle3` as a base (the image's own idiom — `laguna.py:717`,
+   `mimo_v2.py:834`, …) → dies at class creation with `Cannot create a consistent method
+   resolution order (MRO) for bases …`. GLM's wrappers already carry four/five
+   Protocol-derived bases, and simulation confirmed the ForCausalLM's five-base ordering
+   cannot take `SupportsEagle3` in *any* position.
+3. **v3** declared the members as class attributes (MRO-safe) — but only on
+   `Glm5NextForCausalLM` → identical error (the wrong wrapper, see above).
+4. **v4** added all three members (`supports_eagle3`, `has_own_lm_head`,
+   `has_own_embed_tokens` — a runtime-checkable Protocol's `isinstance` walks ALL
+   `__protocol_attrs__` *including base-Protocol members* via `getattr_static`, verified
+   experimentally in-image with stock typing) on `Glm5NextForCausalLM` → identical error.
+5. **v5** adds the three attributes to BOTH wrappers + the two methods on both
+   (`Glm5NextForCausalLM` unwraps to `self.model` — the `EagleModelMixin` trunk;
+   `Glm5NextForConditionalGeneration` delegates to `self.language_model`, the
+   `pixtral.py:498` pattern for exactly this wrapper shape) → **gate passes**.
+
+**The lesson generalizes twice over**: when a runtime error names an interface, read the
+interface's declaration first — and establish WHICH INSTANCE the check runs against
+before patching anything: multimodal wrappers (`…ForConditionalGeneration`) are loaded
+models, not pass-throughs, and the text model behind `self.language_model` is invisible
+to a check run on the wrapper.
+
+**Retired diagnostic (do not resurrect as-is): the `/probe` startup hook.** An
+`op_probe.py` + `sitecustomize.py` pair was mounted to wrap vLLM module forwards and
+torch's name-map lookups. It produced no evidence and caused a **fork bomb** (see the
+hazard note under experiment-window discipline). Use `TORCH_LOGS` for dynamo questions
+instead — it needs no startup hook.
+
+Note also that the KV-cap patch's predecessor arithmetic in
+[[dankserver-qwen38-fp8-expert-offload-tuning]] should be read the same way: its
+"gather-bound" conclusion came from the same throughput arithmetic that this page's
+profiler window has now falsified.
+
+### The KV-cache grouping journey: clearing the unifier, then the kpool tail
+
+With patch 01 v5 the gate passes and the boot moves to KV-cache layout
+construction — where the next two blockers live, both in
+`v1/core/kv_cache_utils.py`:
+
+1. **The page-size unifier refuses GLM's MLA family.** With a draft model the
+   spec dict gains non-MLA drafter attention specs and `HiddenStateCacheSpec`
+   entries, so `_get_kv_cache_groups_glm5_next` (which requires ALL attention
+   specs to be `MLAAttentionSpec`) bails to the generic grouping path — whose
+   `unify_kv_cache_spec_page_size` raises
+   `NotImplementedError: Layer language_model.model.layers.3.self_attn.indexer.k_cache: page size is not divisible… cannot be padded. Padding is only supported for non-MLA attention layers.`
+   MLA layers cannot be *padded* (sparse MLA indexes whole token rows) — but
+   they CAN grow `block_size` when the max page is an exact multiple. The
+   indexer's page is just `tokens_per_state x` the MLA page, so
+   **`glm53-dflash-02-kv-unify-mla-multiple.patch`** bumps the unified max page
+   to a multiple of every `MLAAttentionSpec` page before the loop: all MLA-family
+   layers then take the legal divisible block-growth branch, drafter + mamba
+   layers pad. The 2026-09-30 window proved it: unify passes, memory profiling
+   passes, torch.compile passes, and the boot reaches cudagraph capture — the
+   NotImplementedError is gone.
+2. **The kpool tail kernel pins its block.** Cudagraph capture then dies in
+   `kpool_ops.kpool_decode_update_and_maybe_write_cache_batched`:
+   `assert tail_kv_cache.shape[2] == pool_size`. The tail cache must view as
+   `[num_blocks, 2, index_kpool, head_dim]` (`KpoolTailSpec(block_size=index_kpool)`,
+   `Glm5NextTailCache` in `glm5next/common/attention.py:159`), but the tail's
+   page (8192 B) *divided* the raised max page, so the unifier's divisible branch
+   grew its `block_size` by the ratio and the view became
+   `[num_blocks, 2, index_kpool x ratio, head_dim]`. Verified not the hybrid
+   promotion: the log carries no "Hybrid KV cache manager is disabled" warning,
+   so `disable_hybrid_kv_cache_manager` is unset and
+   `unify_hybrid_kv_cache_specs` never ran. **`glm53-dflash-03-kv-unify-tailpin.patch`**
+   (= 02 + a `KpoolTailSpec` branch that pads `page_size_padded` instead of ever
+   growing its block) **verified 2026-09-30**: the assert is gone and the boot
+   captures the DFlash2 drafter's FULL cuda graphs on both TP workers
+   (`Capturing model for DFlash2 speculator...`). Supersedes 02 in the repo.
+
+3. **The capacity wall: the generic path cannot alias the mamba states.**
+   Past capture, the boot dies at `_check_enough_kv_cache_memory`:
+   `Available KV cache memory: 0.92 GiB` against `31.76 GiB needed at max len
+   65536` (32.91 GiB at 131,072 — nearly flat, a fixed intercept, not
+   per-token KV). Two coupled causes, read out of the image:
+   - **The glm5 tensor layout aliases the mamba states INTO the MLA pages**
+     (`get_kv_cache_config_from_groups` places every mamba layer at the MLA
+     offsets — that is why GLM-5.3 fits 32 GiB cards at all, and why the
+     expert plan's KV registry is calibrated at 13,600 B/token). On the
+     generic path the mamba groups claim their OWN blocks, so ~30 GiB of
+     unaliased state pages appears.
+   - **The plan's KV reserve (`expert_activation_registry.py:_glm53_kv_bytes_per_token`,
+     a flat 13,600 B/token) knows nothing about the drafter.** The engine caps
+     KV at the plan's reserve, so even a correct generic-path estimate would
+     be starved.
+
+   Learned en route: the engine AUTO-SETS the attention block size to
+   **2,432 tokens** (`interface.py:918: Setting attention block size to 2432
+   tokens to ensure that attention page size is >= mamba page size`), so every
+   attention group's block counts scale from that floor — a 16-token drafter
+   block would need 4x the shared `num_blocks`; with the global 2,432 all
+   groups align.
+
+   **`glm53-dflash-04-kv-groups-draft.patch`** (+ **`-04b-kv-registry-dflash.patch`**
+   for the registry) keeps the aliased glm5 pool and extends it: the grouping
+   partitions the hidden-state specs (`HiddenStateCacheSpec` is an
+   `MLAAttentionSpec` subclass — excluded before the exact-type
+   `type(spec) is MLAAttentionSpec` partition) and the drafter attention
+   specs; the drafter layers form their own uniform group (per-group pages,
+   NO unification — the MLA block size stays at the engine-set 2,432 and the
+   kpool guard `block_size % 512 == 0` is never stressed); the layout returns
+   a 10-tuple with `draft_group`/`hidden_groups`; the builder sizes ONE
+   shared backing allocation (`allocate_kv_cache` asserts all tensor sizes
+   equal) as aliased pool + appended draft/hidden tensors with their own
+   strides; the memory estimator adds the extras. Patch 04b extends the
+   registry: 13,600 + 4,096 (drafter 8 kv heads x 128 x K+V x bf16) +
+   5 x hidden_size x 2 B (the aux hidden-state cache) per token when
+   `speculative_config.use_aux_hidden_state_outputs` is set.
+
+   The pure logic is smoke-tested locally (CPU torch + the image's real spec
+   classes against synthetic GLM+draft+hidden spec sets): grouping shape,
+   layout recognition, builder block math and memory accounting all pass.
+   Real-fit arithmetic: per-token ≈ 58,656 B → reserve ≈ 4.15 GiB at max len
+   65,536 (arena shrinks ~13.5 → ~10.3 GiB, ~1,650 slots), num_blocks ≈ 29 x
+   2,432 tokens ≈ 70,500 ≥ 65,536.
+
+All of these mount over site-packages via the scratch launcher's
+`KVUNIFY_PATCH=1` knob (host `/home/dank/dflash-patch2/new/…`, patched shas
+`5521e50b…` for `kv_cache_utils.py` and `1a623d5c…` for
+`expert_activation_registry.py`; apply-verified against the image's pristine
+files). The window script's decision pipeline was also fixed for a SIGPIPE
+death (`head -80` on a `podman logs` grep pipeline killed one run before
+log-save and restore; the log is now captured to a file FIRST, then grepped).
+
+
+## Measured arms (all 2026-09-30, one boot each)
+
+| arm | decode | prefill | verdict |
+| --- | --- | --- | --- |
+| baseline int4, MAXLEN 131,072, util 0.92, NBATCH 2048 | 10.03 / 10.08 / 9.41 | 877 / 878 / 885 | qualified incumbent |
+| + `--expert-precise-memory` (arena 13.08 → 14.51 GiB) | 9.88 / 9.91 | 918 | **neutral** — proves residency is no longer the binding term |
+| + MTP4 spec decode (draft loads, acceptance excellent) | 9.64-11.16 | **255** | **rejected** — see below |
+| + MTP4 + explicit `--kv-cache-memory-bytes` (patched cap) | boots, 151,767-token KV | — | enabler for the MTP arm only |
+
+### Why MTP is rejected here (it is not that it fails)
+
+MTP **works** on GLM in this fork: the head loads, the architecture resolves to
+`Glm5NextMTPModel`, and acceptance is excellent —
+
+```
+SpecDecoding metrics: Mean acceptance length: 3.83-4.90,
+Per-position acceptance rate: 0.917/0.833/0.583/0.500 … 0.962/0.846/0.692/0.615,
+Avg Draft acceptance rate: 69-97%
+```
+
+— and it buys **nothing**: decode 9.6-11.2 tok/s against 10.0 without it, while
+prefill **collapses 918 → 255 tok/s**. The reason is structural to offload: a
+verification forward gathers the **union** of the batch's experts, so spec
+decode amortizes compute but not a residency-bound gather. It also costs 2.6 GiB
+of arena. On a compute-bound (fully-resident) shape this would invert; here it
+does not.
+
+## Served surface (verified 2026-09-30 against the live engine)
+
+| capability | state | evidence |
+| --- | --- | --- |
+| `/v1/models` | `glm53-flash`, max_model_len 131,072 | model list probe |
+| completion | yes | `bench_glm.py` runs above |
+| **tool calls** | **yes** — needs `--enable-auto-tool-choice --tool-call-parser glm47` | a `get_weather` prompt returns `tool_calls: [{"id": "chatcmpl-tool-…", "type": "function", "function": {"name": "get_weather", "arguments": "{\"city\": \"Seattle\"}"}}]`, `finish_reason: tool_calls` |
+| **vision** | **yes** | a 64x64 solid-red PNG is answered "The image appears to be a solid red color…" |
+| reasoning channel | **yes** — needs `--reasoning-parser glm47` | without it the thinking leaks into `content` (measured: content began with the answer, then "Let me verify: … = 391", then a bare `</think`) with an empty reasoning field; with it, a train-time prompt answers in `content` ("**Step 1: Add the hours.** 14:20 + 3 hours = 17:20 …") and the chain of thought lands in **`reasoning`** — the vLLM field name the fleet documents (llama.cpp endpoints use `reasoning_content`) |
+
+Without the tool flags the engine answers **HTTP 400** — `"auto" tool choice
+requires --enable-auto-tool-choice and --tool-call-parser to be set` — which
+breaks every tool-using agent on the endpoint. Both parsers are `glm47`, the
+newest GLM dialect the fork ships (`glm45`/`glm47` exist for both).
+
+## Fork traps found (for the engine authors)
+
+1. **`--kv-cache-dtype fp8` is illegal on gfx12 with this model — and it buys
+   nothing even if fixed.** The `ROCM_AITER_MLA_SPARSE` backend's
+   `_use_rocm_sparse_triton()` requires a non-fp8 KV cache, and that rope-free
+   Triton path is the **only** MLA decode available on gfx12: aiter's MLA decode
+   op is registered under `@if_aiter_supported`, which is gated on **CDNA > 2**
+   (`is_aiter_found_and_supported()` → `get_cdna_version() > 2`). With fp8 KV
+   the MTP draft step demands an op that cannot exist here:
+   `RuntimeError: Worker failed with error ''_OpNamespace' 'vllm' object has no
+   attribute 'rocm_aiter_mla_decode_fwd''`. KV dtype therefore stays auto/bf16.
+   (`VLLM_ROCM_USE_AITER_MLA=1` does **not** help — the umbrella gate is the
+   CDNA check, not the env var.)
+   - **The capacity payoff is measured at zero.** The point of fp8 KV is fitting
+     more tokens in the same reserve, and this fork's reserve model is
+     **dtype-blind**: the fp8 boot printed the *identical* plan line as bf16 —
+     `KV reserve 1.76 GiB (13600 B/token x 131072 + mamba pages)`. The reserve is
+     dominated by the per-sequence KDA state (34 linear-attention layers) and the
+     DSA indexer's pools, not by the MLA KV bytes those 13,600 B/token barely
+     cover — so halving the KV dtype cannot move it.
+   - **What a fix would take, if it were ever wanted**: a new fp8 path in the
+     rope-free Triton kernels (`rocm_sparse_attn_prefill` / `rocm_sparse_attn_decode`
+     in `v1/attention/ops/rocm_aiter_mla_sparse.py`) that dequantizes pages
+     in-kernel, plus loosening `_use_rocm_sparse_triton()`. The fork's
+     `_r4d_fp8_paged_mqa_logits` shows the author already has an fp8 MQA-logits
+     path, but the sparse-MLA backend is not wired to it. Verdict 2026-09-30:
+     **do not spend effort here** — the payoff is capacity, and the reserve is
+     dtype-blind.
+2. **The plan's KV reserve under-models this model, and the cap enforces the
+   mistake.** The plan charges 13,600 B/token + a small mamba term, but GLM
+   carries a per-sequence KDA state (34 linear-attention layers) and, under MTP,
+   the draft's KV. `gpu_worker` then *caps* KV at that reserve, so a legal
+   `--max-model-len` becomes unbootable:
+   `ValueError: To serve at least one request with the model's max seq len
+   (131072), (2.07 GiB KV cache is needed, which is larger than the available KV
+   cache memory (1.74 GiB))`. Raising `max_model_len` only helps past ~440k
+   tokens (the cap's slope is ~13,845 B/token vs the need's ~12,700 + 0.52 GiB
+   fixed) — where the reserve would eat a third of the arena.
+   `scripts/patches/gpu_worker-kv-cap.patch` lets an explicit operator KV
+   request raise the cap; **it is not part of the deployed shape**, which boots
+   on the plan's own reserve.
+3. **A window is forced, not chosen.** The production `max_model_len 131072` is
+   what the reserve model requires for this payload (below ~92k the boot
+   refuses); it is not a capacity decision.
+4. **Expert prefetch remains poison** — unchanged from the qwen campaign
+   (36% precision, +40-51% decode from deleting it). No GLM arm re-opened it.
+
+## Operations
+
+```bash
+# rollback to the pre-cutover qwen FP8 + MTP4 shape (payload and R9V state
+# dirs are untouched on disk):
+git show 2339fae -- scripts/ | git apply   # or: git checkout 2339fae -- scripts/
+./scripts/deploy.sh dankserver --service frank-primary
+
+# payload inventory (34 shards + 2 extras, weight-map index):
+bash scripts/download-frank-primary.sh --verify-only
+
+# live telemetry (needs the profiler window; with_stack=false is mandatory —
+# with stack the stop path hangs and the trace buffer grows to tens of GB):
+podman logs frank-primary-glm 2>&1 | grep "expert_lru.py"
+```
+
+**Experiment-window discipline (learned the hard way, 2026-09-30).** This engine
+**ignores SIGTERM**: after `systemctl stop` its container sits in `podman ps` state
+`Stopping` for 20+ minutes *while still answering `/health` with 200*. That makes a
+liveness probe a lying oracle for a restore — twice a restore script printed
+`PROD HEALTHY` while `systemctl is-active` said `failed` and the runtime mask was
+still in place. The restore oracle must be a **conjunction**: `is-active` ==
+`active` **and** `is-enabled` == `enabled` **and** no `/run/systemd/system/<unit>`
+mask symlink **and** the container is owned by the unit and not `Stopping`. The
+repair recipe, all four steps needed:
+
+```bash
+sudo mv /run/systemd/system/frank-primary.service /tmp/stale-mask-N.service  # runtime mask SURVIVES stop and blocks start
+sudo systemctl daemon-reload
+sudo systemctl reset-failed frank-primary.service                            # a failed unit refuses to start
+podman rm -f frank-primary-glm                                               # SIGKILL; `podman stop` hangs on the same SIGTERM
+sudo systemctl start frank-primary.service                                   # then ~7 min to health 200
+```
+
+Also: **never let two experiment jobs overlap.** A later job's `podman rm -f
+glm-probe` cleanup ran while an earlier job was still mid-flight and deleted its
+container, so one whole round produced no evidence at all (`no container with name
+glm-probe found`) — check `proc://` for running jobs before opening a window.
+
+**Never put a diagnostic hook on `PYTHONPATH` without a marker guard — it is a fork
+bomb.** On 2026-09-30 a `sitecustomize.py` mounted at `/probe` (to wrap dynamo's
+name-map lookups and vLLM module forwards) produced a self-replicating chain of
+`python3` processes 19+ levels deep inside the probe container: `sitecustomize` is
+imported by *every* interpreter, the hook imported torch/vLLM at startup, something in
+that import chain spawned a helper python, and the child inherited `PYTHONPATH` (hence
+the hook) and spawned again. It was still alive 18 minutes later with the container
+still showing `Up`. Symptoms: `ps -eo pid,ppid,comm --forest` shows a deep ladder of
+identical interpreters, each child's PPID being the previous one. Guard (now in the
+hook): arm once per environment via a marker in `os.environ`, which children inherit —
+
+```python
+if os.environ.get("VLLM_PROBE_ARMED") == "1":
+    pass                       # an ancestor already armed it; stay inert
+else:
+    os.environ["VLLM_PROBE_ARMED"] = "1"
+    <install probes>
+```
+
+Kill recipe: `podman rm -f <experiment-container>` (the container cgroup owns every
+generation), then verify with a precise count over `/proc/<pid>/environ` for the marker
+variable — **never** pattern-kill an interpreter name, because the media stack on this
+host (romm, jellyfin, plex, glances) runs python in production.
+
+```bash
+# boot: ~7 min warm page cache, longer cold; health returns 503 while loading
+curl -s -o /dev/null -w "%{http_code}\n" http://127.0.0.1:7770/health
+
+# experiment windows (production stopped + `systemctl mask --runtime`, which
+# self-clears at reboot) ran from the host-side scratch launcher
+# /home/dank/serve_glm_autoround.sh: env knobs MAXLEN SEQS NBATCH UTIL KVBYTES
+# KVDTYPE SPEC SPECJSON_FILE TOOLPARSER REASONINGPARSER TELEM GCHUNKS GLANES
+# PROFDIR GLM_PATCH_MOUNT COMPILE_PATCH GATHER_PATCH DRAFT_MOUNT ENVARGS.
+# DRAFT_MOUNT=1 mounts /home/dank/models/GLM-5.3-Flash-DFlash2 at /draft;
+# SPECJSON_FILE passes a whole --speculative-config JSON (JSON with no spaces,
+# word-split by the launcher's unquoted $EXTRA).
+bash /home/dank/serve_glm_autoround.sh
+```
+
+Boot gate: the engine refuses a legal-looking shape in two measured ways — the
+KV cap above, and the profile-run activation peak (`TritonMLA` indexer scores:
+64 heads x 2,048 topk x NBATCH tokens x 2 B = 2.06 GiB at NBATCH 8192), which is
+what killed the first two boots at util 0.95.
+
+`PYTORCH_CUDA_ALLOC_CONF=expandable_segments:True` is part of the qualified
+shape, not a tuning: the plan fills VRAM to its modelled headroom by design, so
+the default allocator's fragmentation fails the profile/graph-capture phase
+(measured on the deployed unit: "Tried to allocate 662.00 MiB. GPU 0 … 240.00
+MiB is free").
+
+The `deploy.sh` e2e gate also drives the admin panel's open-links spec, which
+walks every guarded vhost — including `http://192.168.0.112:8989/` on
+FrankBook-Pro. When that Mac mini is asleep the spec fails on a `page.goto`
+timeout and the deploy reports `DEPLOY INCOMPLETE` even though the engine is
+healthy; wake the host (magic packet to `de:6b:6d:c1:52:23`) or re-run
+`tests/e2e/run.sh` once it is up. Seen 2026-09-30: 32 passed, this one failed,
+and the WoL packet did not wake it (macOS "Wake for network access" needs
+toggling on the mini itself).
+
+## Rollback targets
+
+| state | where |
+| --- | --- |
+| qwen FP8 + MTP4 (2026-09-29 → 09-30) | commit `2339fae`, payload `/home/dank/models/Qwen3.8-Flash-Next-Uncensored-FP8` |
+| R9V radiance Qwen3.8-Flash-Next Q8_0 | [[frank-primary-r9v-radiance-qwen38-q8]], state dirs `/home/dank/r9v-state/{q8xl,uncensored}` |
@@ -0,0 +1,173 @@
+---
+type: reference
+title: "frank-small on Dankclaw — llama.cpp, MiMo-V2.6-Distill-Qwen-9B Q5_K_M GGUF (retired)"
+description: "The retired 2026-10-02/03 frank-small runbook — the Q5_K_M rung at two wide unquantized-KV slots on the RTX 3090, superseded 2026-10-03 by the Q6_K file. Kept as the rollback reference: it carries the measured boot-pair table that prices every pool shape on this card, the slot-window contract, and the misbehaves tree."
+tags:
+  - llm
+  - frank-small
+  - dankclaw
+  - llama-cpp
+  - rtx3090
+  - mimo
+  - gguf
+  - kv-cache
+timestamp: 2026-10-03
+---
+
+# frank-small — llama.cpp MiMo-V2.6 Q5_K_M (RTX 3090, retired)
+
+> **RETIRED 2026-10-03** — superseded by
+> [[frank-small-llama-mimo-q6k-2x200704]]: the ~3.5 GiB this shape left free
+> bought the **Q6_K quant rung** (same runtime, same 2 × 200,704 pool, same
+> f16 KV), which measures 2,849 MiB free and 94.44 tok/s — 10 % slower
+> decode is the measured price of the higher rung on this bandwidth-bound
+> card. This file remains the rollback reference and the owner of the
+> measured boot-pair table.
+
+> Its own history: one quant step down from [[frank-small-llama-mimo-q6k]]
+> (2026-10-01), resized twice on 2026-10-02 (three contexts with vision, then
+> the 2–3 GiB-band pool), and on 2026-10-03 morning reshaped to **two
+> conversations of 200,704 tokens with the KV VALUES unquantized** —
+> *"2 streams with 200700 tokens context or more… if we have extra space on
+> the 3090, unquantize the kv."* That shape measured 20,530 used / 4,046
+> apparent free (the true `nvidia-smi` free was 3,571 — see *Shape*).
+
+## Identity
+
+|Field|Value|
+|---|---|
+|Endpoint|`192.168.0.143:7771`, served alias `frank-small`|
+|Runtime|llama.cpp **master** (qwen35 model support), CUDA build `sm_86`, at `/home/dank/GitPackages/llama-master/build-cuda/bin/llama-server` — unchanged across the quant swap|
+|Payload|`bartowski/MiMo-V2.6-Distill-Qwen-9B-GGUF` **Q5_K_M**, 6,876,124,704 B (6.40 GiB), sha256 `04cf46f2b5584ef922697d6c063f94879434fb86c467f1a09e52daf0cddaf35f` + the repo's `mmproj-MiMo-V2.6-Distill-Qwen-9B-f16.gguf`, 918,166,048 B, sha256 `ff348f3180a63188aa7285db85f550fe38acb61dd013c599eb8bad08d2cc2576` — both at `/home/dank/models/MiMo-Q5KM/`, fetched and byte- and sha256-verified by `scripts/download-frank-small.sh`|
+|Unit|root `frank-small.service` (repo `scripts/frank-small.service`, `User=dank`, relinked), env `scripts/frank-small.env`, serve `scripts/serve-frank-small.sh`|
+|Repo pins|`tests/frank_small_unit.bats` (stub-argv seam + the VRAM arithmetic), `deploy_llm_small` gates in `scripts/deploy-remote.sh`|
+
+## Shape and the KV contract
+
+`--ctx-size 401408 --parallel 2` — llama.cpp divides the pool across slots,
+so each conversation gets `n_ctx_slot = 200704` (boot log 2026-10-03:
+`n_slots = 2, n_ctx_slot = 200704, kv_unified = 'false'`). `CTX_SIZE` MUST
+stay divisible by `PARALLEL`; the bats seam enforces it.
+
+The KV contract is Frank's, as of 2026-10-03: **both K and V unquantized
+(f16)** — `--cache-type-k f16 --cache-type-v f16`. The asymmetric contract it
+replaced (f16 K + `q8_0` V) had existed to make three slots fit; with two
+slots the exact-value budget opens, and 4 KiB/token × 8 layers × 2 of extra
+V traffic is the measured ~2 % decode tax. The serve script still passes both
+flags explicitly — the asymmetric-era law that a quantized V requires
+`--flash-attn on` stays pinned by the seam.
+
+**Pool sizes on this stack come from boots, not formulas.** The compute
+buffers, CUDA graphs and vision tower all bill separately, so every shape
+change is priced from measured rows:
+
+|Shape|KV dtype (K+V)|Projector|`nvidia-smi` used|Non-KV (weights + buffers + projector)|
+|---|---|---|---|---|
+|2 × 262,144 (pool 524,288)|f16 + q8_0|none|20,190 MiB|**7,390 MiB**|
+|3 × 196,608 (pool 589,824)|f16 + q8_0|f16|22,972 MiB|**8,859 MiB**|
+|3 × 180,224 (pool 540,672)|f16 + q8_0|f16|21,684 MiB|~8,419 MiB|
+|**2 × 200,704 (pool 401,408)**|**f16 + f16**|f16|**20,530 MiB**|**~7,986 MiB**|
+
+The 32 KiB/token full-attention KV price (8 layers × 4 KV heads × 256 dim ×
+2 B × 2 halves) prices the deployed pool at 401,408 × 32 KiB = **12,544 MiB
+of KV**; the card is 24,576 MiB, so the shape leaves **4,046 MiB free**
+(measured steady-state — the KV pool is allocated whole at boot, so this is
+the ceiling). The headroom above the 2–3 GiB floor is deliberate slack, not
+an invitation: another 32,768 tokens of pool (1 GiB) would trade the last
+safety margin for window the harness compacts before reaching anyway.
+
+**Why 2 × 262,144 with f16 V does not fit either.** The native window pair is
+524,288 tokens = 16,384 MiB of KV plus ~7,986 non-KV = 24.4 GiB: under the
+floor by a hair. 200,704 is the largest 4,096-aligned slice that keeps the
+measured band.
+
+## Measured on this shape (2026-10-03, 2 × 200,704 + vision, f16 KV)
+
+|Check|Result|
+|---|---|
+|Boot to healthy|~5 s, `n_slots = 2`, `n_ctx_slot = 200704`, `loaded multimodal model` logged|
+|VRAM|**20,530 MiB used / 4,046 MiB free** of 24,576 (`nvidia-smi`, steady state)|
+|Decode, temp 0, no thinking|**104.84 tok/s** (128-token completion at ~500-token depth; the 3 × 180,224 q8_0-V shape measured 106.8 — ~2 % is the honest price of exact V)|
+|Prefill|1,650 t/s on the same shallow 506-token probe (rates are depth- and prompt-dependent; not comparable across shapes)|
+|Batch geometry|`--batch-size 2048 --ubatch-size 1024` — ubatch 2048 failed the scratch boot; 1024 boots clean|
+|Vision|projector loads at boot (`loaded multimodal model`); red-PNG probe passed on 2026-10-02. Caveat discovered on 2026-10-03: probes must use a >= 256 px image — a 10x10 px PNG undersupplies image tokens and the model hallucinates ("it's a video")|
+|Tools / thinking|unchanged by the reslice: native `tool_calls`, thinking in `reasoning_content`|
+
+## What the slot geometry costs
+
+The per-slot window is the whole price, in both directions:
+
+- **A prompt over 200,704 tokens is a hard 400**, so `~/.omp/agent/models.yml`
+  sets `contextWindow: 200704` for this endpoint — the harness compacts before
+  it reaches the ceiling instead of discovering it.
+- **frank-large's tallest pre-compaction prompt now fits here.** Compaction
+  for frank-large sessions runs on this endpoint (`compactionModel` in
+  `models.yml`); frank-large compacts at 70 % of 262,144 = 183,501 tokens,
+  which did **not** fit the old 180,224 slot and fits 200,704 with 17,163
+  tokens to spare. The 2026-10-02 window trim had restored the fit for
+  frank-primary's smaller 140,492-token trigger; the 2026-10-03 reslice widens
+  the fit to frank-large itself. Nothing routes compaction automatically —
+  the point is the ceiling no longer forbids it.
+- Two simultaneous generations are the slot ceiling; a third request queues
+  behind a released slot (the mechanism measured on the 3-slot shape).
+
+## Client contract
+
+- Thinking arrives in **`reasoning_content`** (`--reasoning-format deepseek`),
+  the field `models.yml` reads for this endpoint.
+- **Thinking is off unless the request asks**: the MiMo GGUF template defaults
+  `enable_thinking` false, so `models.yml` ships
+  `chat_template_kwargs: {enable_thinking: true, preserve_thinking: true}` in
+  `extraBody`. A raw client that omits it gets a direct answer — measured, and
+  not a fault. Trivial questions produce no thinking even with the flag on.
+- Tools: `--jinja` + server-side parsing, verified 2026-10-02.
+- **Vision is on**, through the projector file (`--mmproj`). Use the
+  OpenAI-standard `image_url` content part — that path answers correctly
+  ("Red." on a solid-red PNG). The legacy top-level `images` field is **not**
+  read by `/v1/chat/completions`: the same request sent that way answered
+  "Brown.", i.e. it generated from text alone. `/completion` takes `images`
+  but refuses a request without `messages` in this build, so chat is the only
+  vision path.
+- Setting `MMPROJ_FILE=` (explicitly empty) in `scripts/frank-small.env` drops
+  `--mmproj` and returns ~1.4 GiB of the card to KV — the documented text-only
+  opt-out. `MMPROJ_FILE` pointing at a file that does not exist is a boot
+  refusal, never a silent fallback to blindness.
+
+## No spec decode
+
+The GGUF has no MTP tensors (the same fact that made SGLang's NEXTN a 0.00
+ghost), and the cross-model DFlash draft
+(`/home/dank/models/DFlash-Qwen35/qwen3.5-9b-dflash-Q8_0.gguf`) is an
+optimization candidate, not a default — acceptance has to be measured on this
+traffic first. The bats seam fails if a draft flag ships unannounced.
+
+## Rollback
+
+|Target|How|
+|---|---|
+|Previous shape: 3 × 180,224 + q8_0 V (2026-10-02)|`scripts/frank-small.env` `CTX_SIZE=540672 PARALLEL=3 CACHE_TYPE_V=q8_0` (measured 21,684 MiB used, 2,441 free) and `contextWindow: 180224` in `models.yml` — choose it when three simultaneous conversations matter more than per-slot window or exact V|
+|Q6_K, same runtime|`scripts/frank-small.env` `MODEL_FILE=/home/dank/models/MiMo-Q6K/MiMo-V2.6-Distill-Qwen-9B-Q6_K.gguf` + `MMPROJ_FILE` in that dir — **this is the deployed rung since 2026-10-03** ([[frank-small-llama-mimo-q6k-2x200704]]): at 2 × 200,704 it fits with 2,849 MiB free, no pool cut needed (the older "+1.4 GiB, drop 49,152 tokens" estimate held only for the 3-slot shapes)|
+|Two 262,144 windows|`CTX_SIZE=524288 PARALLEL=2 CACHE_TYPE_V=q8_0` (measured 20,190 MiB used, 4,386 free **with q8_0 V** — with f16 V the same shape is 16,384 MiB of KV + ~7,986 non-KV and does not fit) and `contextWindow: 262144` in `models.yml` — choose it when a single conversation needs the native window|
+|Vision off|`MMPROJ_FILE=` in the env file; the deployer gate then has to be satisfied too, so point `LLM_SMALL_MMPROJ` at any real projector file or drop the gate in the same commit|
+|SGLang + INT8|[[frank-small-sglang-mimo-v26]] — venv and old unit/env/serve trio still in place; the INT8 payload was staged to `/home/dank/.AGENTTRASH/20261002-retired-llm-payloads/` on 2026-10-02, so restore it there or re-fetch with `old/download-frank-small-sglang-int8.sh`|
+
+## When it misbehaves
+
+- Port 7771 dark: `journalctl -u frank-small -n 50`; the serve script exits 1
+  naming the missing file if the binary or GGUF moved.
+- `No CUDA GPUs are available`: the 3090 vanished from the bus or the UVM node
+  is missing — `nvidia-modprobe -u -c=0` (the serve script runs it).
+- Requests queue behind each other: two concurrent long generations are the
+  slot ceiling by contract (2026-10-03 shape); a third waits for a slot to
+  free (the mechanism measured on the 3-slot shape: 1.7 s for three, 3.0 s for
+  the fourth).
+- `400 … exceeds the available context size (200704 tokens)`: the conversation
+  outgrew ONE slot. This is not pool exhaustion — the pool is 401,408 tokens
+  and the per-slot window is the binding limit. Shorten the prompt, compact,
+  or roll back to the 262,144 shape above.
+- Vision answers a color it never saw, or claims the image "is a video":
+  either the client sent `images` instead of an `image_url` content part, or
+  the probe image is too small to supply real image tokens (use >= 256 px;
+  see *Client contract*).
+- Slow decode after a CTX or quant change: check `nvidia-smi` — pushing KV out
+  of VRAM into unified memory halves the rate silently.
@@ -0,0 +1,158 @@
+---
+type: reference
+title: "frank-small on Dankclaw — llama.cpp, MiMo-V2.6 Q6_K, 2 x 200704, f16 KV"
+description: "The active frank-small runbook — llama.cpp master (CUDA sm_86) serving bartowski's MiMo-V2.6-Distill-Qwen-9B Q6_K GGUF on the RTX 3090: two conversations of 200,704 tokens with fully unquantized f16 KV, vision through the bundled projector, and the boot-pair arithmetic that prices every pool change."
+tags:
+  - llm
+  - frank-small
+  - dankclaw
+  - llama-cpp
+  - gguf
+  - rtx3090
+timestamp: 2026-10-03
+---
+
+# frank-small — llama.cpp MiMo Q6_K, 2 × 200,704, f16 KV
+
+> **ACTIVE since 2026-10-03.** The third shape Frank bought in two days on
+> this card: the freed third slot paid for **exact KV values**
+> ([[frank-small-llama-mimo-q5km]] documents that step and remains the
+> rollback), and the ~3.5 GiB still free then paid for the **Q6_K quant rung**
+> — the same file this endpoint served on 2026-10-01, come back as an
+> upgrade rather than a compromise. The pool held at 401,408 (2 × 200,704);
+> only the weights moved up.
+
+## Identity
+
+| | |
+|---|---|
+| Host / device | Dankclaw (192.168.0.143), RTX 3090, 24,576 MiB |
+| Endpoint | `http://192.168.0.143:7771` — alias `frank-small`, `/v1` OpenAI surface |
+| Runtime | llama.cpp master, CUDA sm_86 build `…/llama-master/build-cuda/bin/llama-server` (`GGML_CUDA_FA_ALL_QUANTS=ON`) |
+| Unit | root systemd `frank-small.service` ([`scripts/frank-small.service`](../../scripts/frank-small.service)), `User=dank`, `Restart=on-failure`, `LimitMEMLOCK=infinity` |
+| Serve script | [`scripts/serve-frank-small.sh`](../../scripts/serve-frank-small.sh), tunables [`scripts/frank-small.env`](../../scripts/frank-small.env) |
+| Weights | `/home/dank/models/MiMo-Q6K/MiMo-V2.6-Distill-Qwen-9B-Q6_K.gguf` — bartowski/MiMo-V2.6-Distill-Qwen-9B-GGUF **Q6_K**, 7,793,710,624 B, sha256 `ef96d05a…` (qwen3_5 hybrid: 32 layers, 8 full-attention + 24 Gated DeltaNet, native 262,144, no MTP tensors) |
+| Projector | `…/MiMo-Q6K/mmproj-MiMo-V2.6-Distill-Qwen-9B-f16.gguf` — 918,166,048 B, sha256 `ff348f…` (hardlink of the Q5KM dir's copy, link-count-2 verified; the dir is self-contained) |
+| Fetch | [`scripts/download-frank-small.sh`](../../scripts/download-frank-small.sh) — `-f`, byte count + LFS sha256 verified, `--verify-only`; both files verified against the pinned sha on the host before this deploy |
+| Seam | `tests/frank_small_unit.bats` (argv assertions against a stub engine: payload path, slot geometry, KV pair, batch pair, projector opt-out) |
+
+## Shape and KV contract
+
+`--ctx-size 401408 --parallel 2` — llama.cpp divides the pool across slots,
+so each conversation gets `n_ctx_slot = 200704`. `CTX_SIZE` MUST stay
+divisible by `PARALLEL`; the seam enforces it. Both K and V are unquantized
+(f16) — the flags stay explicit because `--flash-attn` must not be lost if V
+ever goes back to `q8_0` (the serve script pins it, the seam pins the pin).
+
+**The Q6_K rung is a weights-side spend only.** File delta +917,585,920 B
+(+875 MiB) against Q5_K_M, same KV, same slot geometry, same batch pair.
+Priced from the measured Q5 boot at this shape (non-KV 7,986 MiB): the boot
+landed **21,276 MiB used / 2,849 MiB true free** — non-KV measured 8,732 MiB,
+inside the 2–3 GiB floor, so the pool stayed at 401,408 and the
+12,288-token step-down ladder was never climbed. Quote `nvidia-smi`'s `free`
+field, never `total − used`: the driver holds ~450 MiB of reserve the
+subtraction hides (that error once overstated this card's slack by a
+half-GiB).
+
+**Quant ceiling, evaluated and closed 2026-10-03 (Frank: stay here).**
+`Q8_0` is the only rung above this one on the Hub (9,545,979,424 B = 9,104
+MiB, +1,671 over Q6_K): at the live pool it would leave 1,178 MiB free —
+under the floor — and the calibrated bandwidth law (it predicted this file's
+94.4 t/s from Q5_K_M's 104.84 within 1 %) puts decode at ~72 t/s, a further
+−24 % for the near-lossless rung. Fitting it was priced two ways (trim the
+pool to 2 × 182,272 — which also breaks frank-large's 183,501-token
+compaction-prompt fit by 1,236 tokens — or revert V to `q8_0`), and both
+were declined. Re-open only on evidence the Q6_K residual matters.
+
+**Pool prices on this card come from boots, never formulas** — the measured
+boot-pair table lives in [[frank-small-llama-mimo-q5km]] (*Shape and KV
+contract*) and still prices every rollback shape.
+
+## Compute-buffer geometry (2026-10-05)
+
+`BATCH_SIZE=4096 UBATCH_SIZE=2048` — both doubled from 2048/1024, and the trade
+measured rather than assumed, because the predecessor shape recorded **"ubatch
+2048 failed the scratch boot"** and this card has less free space than it did.
+
+**That failure was a different shape, and it did not reproduce here.** The
+service itself is the bench (one card holds one server, so no scratch port):
+the old rung was re-booted through a transient `FRANK_SMALL_ENV_FILE` drop-in
+and reproduced the 2026-10-03 boot **byte-identically** — 21,276 MiB used /
+2,849 MiB free — which makes the ubatch the only variable in the comparison.
+
+|Probe|ubatch 1024|ubatch 2048|Δ|
+|---|---|---|---|
+|Prefill, 8,928 tokens|3,972 / 4,077 t/s|**4,207 / 4,212 t/s**|**+4.6 %**|
+|Decode, 128 tokens at that depth|91.87 / 93.01 t/s|93.81 / 94.27 t/s|noise|
+|Prefill, 279 tokens|2,675.7 t/s|2,658–2,811 t/s|noise|
+|VRAM used / free|21,276 / 2,849 MiB|**21,846 / 2,279 MiB**|**−616 MiB**|
+
+Three readings:
+
+- **The gain only exists above the old ubatch.** A 279-token prompt is shorter
+  than either micro-batch, so it cannot see the knob — and it did not move.
+  That is the control proving the 8.9k delta is geometry, not drift.
+- **Decode is flat**, as every prior sweep on this card found. Ubatch is a
+  prefill knob.
+- **The price is 616 MiB out of a 2.8 GiB margin.** 2,279 MiB free is still
+  inside the 2–3 GiB band, at its bottom edge. `4096/1536` is the middle rung,
+  and the pool's 12,288-token (384 MiB) ladder in
+  [[frank-small-llama-mimo-q5km]] is the other way to buy room back.
+
+**Why 2048 is right HERE while 512 was right on the R9700 sibling: the curve is
+backend-specific.** The 2× R9700 layer split peaks at ubatch 512 and is 10 %
+off that peak by 2048 ([[frank-medium-batch-ubatch]]); on this CUDA card the
+curve climbs monotonically to 2048 (710 → 1,188 t/s on the 35B MoE sweep,
+1,944 → 2,423 t/s on the Ornith 8k sweep, both with decode flat). Never carry a
+ubatch verdict across backends.
+
+The logical `--batch-size` is VRAM- and throughput-neutral — two sweeps and this
+A/B agree — so 4096 costs nothing and buys nothing by itself; it is here because
+that is the pair the curve was measured at.
+
+## Measured on this shape (2026-10-03, Q6_K, 2 × 200,704, f16 KV)
+
+|Check|Result|
+|---|---|
+|Boot to healthy|model + projector resident in ~2.3 s (load log `0.02.27`), `n_slots = 2`, `n_ctx_slot = 200704`, `loaded multimodal model`, health 200; deploy e2e gate 33/33|
+|VRAM|**21,276 MiB used / 2,849 MiB free** of 24,576 (`nvidia-smi` `free`, steady state) — inside the 2–3 GiB floor with exact V AND the higher rung. Superseded 2026-10-05 by the batch doubling: **21,846 / 2,279** — see *Compute-buffer geometry*|
+|Decode, temp 0, no thinking|**94.44 tok/s** on the identical 506-token probe that measured 104.84 on Q5_K_M — **−10 %**. Decode on this card is weight-bandwidth-bound, and Q6_K streams ~10 % more bytes per token; the quant rung buys quality with decode rate, measurably|
+|Prefill|2,229.9 t/s (same shallow probe; rates are depth-dependent, not comparable across probes)|
+|Vision|512×512 solid-red PNG → **`Red`** (277 prompt tokens — the image really reached the tower). See the probe-size trap in *Client contract*|
+|Tools / thinking|native `tool_calls`, thinking in `reasoning_content`, thinking OFF unless `enable_thinking: true` (MiMo template default; `models.yml` ships it in `extraBody`)|
+
+## Client contract
+
+llama.cpp surface, live-verified on this model family: model id `frank-small`,
+native tools via `--jinja` (`tool_calls` in the response), thinking in
+**`reasoning_content`**, vision via `image_url` content parts. Two vision
+traps, both measured here:
+
+1. A top-level `images` field on `/v1/chat/completions` is silently ignored
+   and the model invents a color.
+2. **The probe image must be big enough to supply real image tokens.** A
+   10×10 px solid-red PNG came back "The image is a video, so I can't see its
+   content" — the engine itself warns at boot that Qwen-VL wants ≥ 1024 image
+   tokens, and a postage-stamp image lands one patch below usable signal.
+   512×512 answers `Red` in 19 tokens. Always probe vision with a ≥ 256 px
+   image; a wrong-color answer from a tiny image is the probe's fault, not
+   the endpoint's.
+
+`~/.omp/agent/models.yml` sets `contextWindow: 200704` — the PER-SLOT window,
+so the harness compacts before llama.cpp answers a longer prompt with a 400.
+
+## Rollback
+
+|Target|How|
+|---|---|
+|Q5_K_M at the same shape|`scripts/frank-small.env` `MODEL_FILE=/home/dank/models/MiMo-Q5KM/MiMo-V2.6-Distill-Qwen-9B-Q5_K_M.gguf` (+ `MMPROJ_FILE` in that dir) and the deployer default — buys back 875 MiB (3,571 free measured) **and the 10 % decode rate** (104.84 tok/s measured). Payload sha-verified on disk|
+|3 × 180,224 + q8_0 V|`CTX_SIZE=540672 PARALLEL=3 CACHE_TYPE_V=q8_0` (measured 21,684 used / 2,441 free **with Q5_K_M** — Q6_K does NOT fit that shape inside the floor; pair them explicitly)|
+|SGLang + INT8|[[frank-small-sglang-mimo-v26]] — payload staged to `/home/dank/.AGENTTRASH/20261002-retired-llm-payloads/`|
+
+## When it misbehaves
+
+Same decision tree as [[frank-small-llama-mimo-q5km]] (*When it misbehaves*):
+`No CUDA GPUs are available` ⇒ `nvidia-modprobe -u -c=0`; a 400 naming
+200,704 ⇒ the conversation outgrew one slot, not the pool; slow decode after
+a shape change ⇒ check `nvidia-smi` for KV pushed into unified memory; queue
+behind two long generations ⇒ the slot ceiling is two, by contract.
@@ -0,0 +1,110 @@
+---
+type: reference
+title: "frank-small on Dankclaw — llama.cpp, MiMo-V2.6-Distill-Qwen-9B Q6_K GGUF (retired)"
+description: "The retired 2026-10-01 frank-small runbook — the first llama.cpp + GGUF shape on the RTX 3090: two native-window sessions, the asymmetric KV contract (f16 K + q8_0 V) that motivated the move off SGLang, the VRAM fit at Q6_K, and what to check when it misbehaves. Superseded the same night by the Q5_K_M payload."
+tags:
+  - llm
+  - frank-small
+  - dankclaw
+  - llama-cpp
+  - rtx3090
+  - mimo
+  - gguf
+  - kv-cache
+timestamp: 2026-10-01
+---
+
+# frank-small — llama.cpp MiMo-V2.6 Q6_K (RTX 3090)
+
+> **RETIRED 2026-10-01 → 10-02** (one night): superseded by
+> [[frank-small-llama-mimo-q5km]], the same runtime and shape one quant rung
+> down. The Q6_K file stays at `/home/dank/models/MiMo-Q6K/` as the rollback
+> target.
+>
+> Original order (Frank, 2026-10-01 night): move this endpoint to the GGUF
+> path for **asymmetric KV** — unquantized K + q8_0 V — which only llama.cpp
+> exposes per-tensor. That contract is what the current deployment kept.
+
+## Identity
+
+|Field|Value|
+|---|---|
+|Endpoint|`192.168.0.143:7771`, served alias `frank-small`|
+|Runtime|llama.cpp **master** (qwen35 model support), CUDA build `sm_86`, at `/home/dank/GitPackages/llama-master/build-cuda/bin/llama-server`|
+|Payload|`bartowski/MiMo-V2.6-Distill-Qwen-9B-GGUF` **Q6_K**, 7,793,710,624 B, sha256 `ef96d05a2ddf2cbb450d1af1ac3860ec769d3575bafa70692ee5609bda3fad7d`, at `/home/dank/models/MiMo-Q6K/`|
+|Unit|root `frank-small.service` (repo `scripts/frank-small.service`, `User=dank`, relinked), env `scripts/frank-small.env`, serve `scripts/serve-frank-small.sh`|
+|Repo pins|`tests/frank_small_unit.bats` (stub-argv seam), `deploy_llm_small` gates in `scripts/deploy-remote.sh`|
+
+## Shape and the KV contract
+
+`--ctx-size 524288 --parallel 2` — llama.cpp divides the pool across slots,
+so each session gets `n_ctx_slot = 262144`, the model's full native window
+(boot log confirms `n_slots = 2, n_ctx_slot = 262144, kv_unified = 'false'`).
+
+Frank's KV contract: **K unquantized (f16), V `q8_0`** —
+`--cache-type-k f16 --cache-type-v q8_0`. Two hard dependencies ride on it:
+
+- **`--flash-attn on` is mandatory**: quantized V exists only on the FA
+  path. Without the flag the engine falls back silently and the contract
+  becomes a lie. The serve script pins the trio; the bats seam pins the
+  script.
+- The VRAM budget: attention KV lives in the 8 full-attention layers only
+  (4 KV heads x 256 dim). K f16 ≈ 16 KiB/token, V q8_0 ≈ 8.5 KiB/token
+  (34 B per 32-element block) ⇒ ~24.5 KiB/token over the 524,288-token pool
+  ≈ **12.6 GiB**. Weights 7.8 + KV 12.6 + compute buffers/graphs ≈
+  **20.9 GiB measured** against 24.5 GiB — the "room for checkpoints" the
+  order asked for. If graph capture OOMs, halve `CTX_SIZE`; never quantize
+  K to fix it.
+
+## Client contract
+
+- Thinking arrives in **`reasoning_content`** (`--reasoning-format
+  deepseek`) — the same field SGLang's qwen3 parser used, so `models.yml`
+  `reasoningContentField` did NOT move. Live-verified 2026-10-01 22:57.
+- **Thinking is off unless the request asks**: the MiMo GGUF template
+  defaults `enable_thinking` false; `models.yml` now ships
+  `chat_template_kwargs: {enable_thinking: true, preserve_thinking: true}`
+  in `extraBody`. A raw client that omits it gets a no-thinking answer
+  (measured — this is not a fault).
+- Tools: `--jinja` + server-side tool parsing, verified
+  `tool_calls` round-trip 2026-10-01 23:0x.
+- **Text-only now.** The GGUF carries no vision projector; the
+  in-checkpoint tower existed only in the SGLang safetensors. `models.yml`
+  `input: [text]`.
+
+## No spec decode (yet)
+
+The GGUF has no MTP tensors (same fact that killed SGLang NEXTN), and the
+cross-model DFlash draft (`/home/dank/models/DFlash-Qwen35/qwen3.5-9b-dflash-Q8_0.gguf`,
+Q8_0) is an optimization-loop candidate, not a default — `--spec-type
+draft-dflash` needs acceptance measured on this traffic first. The bats
+seam fails if a draft flag ships unannounced.
+
+## Build
+
+No ccache, CUDA 12.4 system nvcc with `--gcc-install-dir`-free gcc-13 host
+compiler (nvcc 12.4 rejects gcc ≥ 14):
+
+```
+cmake -B build-cuda -S . -DCMAKE_BUILD_TYPE=Release -DGGML_CUDA=ON \
+  -DCMAKE_CUDA_ARCHITECTURES=86 -DCMAKE_CUDA_HOST_COMPILER=/usr/bin/gcc-13 \
+  -DLLAMA_CURL=OFF -DGGML_CCACHE=OFF
+cmake --build build-cuda -j 16 --target llama-server
+```
+
+The SGLang venv (`/home/dank/GitPackages/sglang-cu128`) and the INT8
+AutoRound payload (`/home/dank/models/MiMo-V2.6-Distill-Qwen-9B-INT8-W8A16-AutoRound`)
+stay on disk as the rollback: repoint the unit's serve script and env from
+git history (`git show frank-large-gufo-q4kxl-20261001` era is BEFORE this;
+use the parent of the 2026-10-01 night swap commit).
+
+## When it misbehaves
+
+- Port 7771 dark: `journalctl -u frank-small -n 50`; the serve script
+  exits 1 naming the missing file if the binary or GGUF moved.
+- `No CUDA GPUs are available`: the 3090 vanished from the bus or the UVM
+  node is missing — `nvidia-modprobe -u -c=0` (the serve script runs it).
+- Requests hang at first token while `/health` is 200: both slots busy —
+  two concurrent long generations are the ceiling by contract.
+- Slow decode after a CTX change: check `nvidia-smi` — swapping KV out of
+  VRAM into unified memory halves the rate silently.
@@ -0,0 +1,180 @@
+---
+type: reference
+title: "frank-small on Dankclaw — SGLang, MiMo-V2.6-Distill-Qwen-9B INT8 W8A16 (RETIRED)"
+description: "The RETIRED frank-small runbook (2026-10-01, hours) — SGLang in a self-contained python3.12 venv serving leteschlead MiMo-V2.6-Distill-Qwen-9B-INT8-W8A16-AutoRound (qwen3_5 hybrid, 9B dense, in-checkpoint MTP + vision) on the RTX 3090: identity, payload, KV arithmetic, the NEXTN spec-decode ghost, the CUDA 13 nvcc toolchain the JIT needs, and the measured numbers kept for the record. Replaced by [[frank-small-llama-mimo-q6k]] the same night."
+tags:
+  - llm
+  - frank-small
+  - dankclaw
+  - sglang
+  - rtx3090
+  - qwen
+  - mtp
+  - vision
+  - int8
+timestamp: 2026-10-01
+---
+
+# frank-small — SGLang MiMo-V2.6-Distill-Qwen-9B INT8 W8A16 (NEXTN MTP)
+
+> **RETIRED 2026-10-01, hours after cutover.** Replaced by
+> [[frank-small-llama-mimo-q6k]] — llama.cpp CUDA on the MiMo Q6_K GGUF,
+> chosen for the asymmetric KV (f16 K + q8_0 V) SGLang cannot express.
+> The venv (`/home/dank/GitPackages/sglang-cu128/`) stays in place; the INT8
+> payload was staged to
+> `/home/dank/.AGENTTRASH/20261002-retired-llm-payloads/` on 2026-10-02 when
+> the llama.cpp endpoint was two runtimes deep — recoverable until the trash
+> is purged.
+
+## Identity
+
+- **Endpoint:** `frank-small` at `192.168.0.143:7771` (Dankclaw).
+- **Runtime:** SGLang (latest) in a self-contained python3.12 venv at
+  `/home/dank/GitPackages/sglang-cu128/` (torch 2.13.0+cu130,
+  transformers 5.12.1, flashinfer 0.6.18, sgl-kernel). No container — a root
+  systemd unit `scripts/frank-small.service` (`User=dank`,
+  `LimitMEMLOCK=infinity`) launched by `scripts/serve-frank-small.sh` with
+  tunables from `scripts/frank-small.env`.
+- **Model:** `letechlead/MiMo-V2.6-Distill-Qwen-9B-INT8-W8A16-AutoRound`
+  (9B dense, 12.0 GiB, INT8 W8A16 AutoRound group-128 symmetric).
+
+## Shape
+
+- **Architecture `qwen3_5` hybrid:** 32 layers = 8 full-attention
+  (4 KV heads x 256 head_dim, `partial_rotary_factor: 0.25`) + 24 Gated
+  DeltaNet. `max_position_embeddings: 262144`, and a `qwen3_5_vision`
+  tower (27 layers) in the safetensors — no separate mmproj file.
+- **Context:** two 262,144-token slots at the model's native window
+  (`--context-length 262144`, `--max-running-requests 2`) against a
+  **pinned** 606,208-token pool (`--max-total-tokens`) — 2026-10-01
+  reslice from four 151,552-token slots: 2 x 262,144 = 524,288 leaves
+  ~82k tokens of pool slack, and a compaction request (frank-primary's
+  trigger 183,501 + a 32,768-token summary budget = 216,269) now fits
+  one request. A single request may grow to the whole pool — a
+  170,019-token prompt was verified at the 4-slot shape — the grid is
+  the two-way concurrency contract. **That compaction arithmetic is
+  specific to a 262,144-token slot**: the ACTIVE shape runs three slots of
+  180,224, which 183,501 tokens already exceeds — see *What three slots
+  cost* in [[frank-small-llama-mimo-q5km]].
+- **KV:** `fp8_e4m3` — 16 KiB/token (8 attn layers x 2 x 4 heads x 256
+  dim x 1 B), pool 9.31 GiB.
+- **The memory shape (the 4.5 GiB the naive math misses):** the DeltaNet
+  ssm state pool is a first-class consumer. The engine autofills the
+  mamba slot count to spend a fixed ~5.1 GB budget *regardless of dtype*
+  (fp32 at 110 slots, bf16 at 217 — same 5.1 GB), so both dials are
+  pinned: `--mamba-ssm-dtype bfloat16 --max-mamba-cache-size 32` = 795
+  MiB. With that, weights 11.35 GiB + mamba 0.78 GiB + pool 9.31 GiB
+  leave ~0.8 GB — enough for eager-prefill workspaces, NOT enough for
+  CUDA-graph capture (autofit's 687,388-token pool leaves 0.53 GB and
+  dies in capture, or later in the first long request once Triton's
+  deferred kernel loads eat the remainder). Hence
+  `--disable-prefill-cuda-graph`; decode graphs are untouched.
+- **Quantization:** `--quantization auto-round` — the checkpoint's
+  `quantization_config.json` declares `quant_method: "auto-round"` and
+  SGLang loads it natively (vision tower, `embed_tokens` and
+  `linear_attn.in_proj` stay fp16).
+
+## Spec method
+
+**None — and this is a checkpoint fact, not a tuning choice.** The
+`text_config` declares `mtp_num_hidden_layers: 1`, but the shipped
+safetensors carry **zero `mtp.*` tensors**: verified 2026-10-01 against
+`model.safetensors.index.json` for both this INT8 conversion and the base
+`XiaomiMiMo/MiMo-V2.6-Distill-Qwen-9B` (32 layers, indices 0–31, no
+`mtp.`/`nextn.`/`layers.32` keys in either). With NEXTN enabled, SGLang
+boots a 4.24 GiB draft model (`Qwen3_5ForCausalLMMTP`) from nothing and
+decodes at `accept len: 1.00, accept rate: 0.00` — every proposal rejected,
+paying the 4-token verify tax for no token. The serve script passes no
+`--speculative-*` flag, and the seam gates that.
+
+## CUDA 13 toolchain (the sharp edge)
+
+The venv's torch is cu130, so every JIT kernel (sgl-kernel marlin repack,
+flashinfer) compiles with a **CUDA 13.0 nvcc extracted from debs into the
+venv** at `cuda-nvcc/usr/local/cuda-13.0/` — the system nvcc is 12.4 (no
+`--compress-mode`) and its headers clash with the 13.0 device passes.
+
+Components (all from the ubuntu2404 CUDA repo, `dpkg -x`'d, not installed):
+
+| Package | Supplies |
+|---|---|
+| `cuda-nvcc-13-0` (13.0.88) | `bin/nvcc`, `ptxas`, `cudafe++`, … |
+| `libnvvm-13-0` (13.0.48) | `nvvm/bin/cicc` (the device compiler) |
+| `cuda-cudart-dev-13-0` (13.0.88) | `include/cuda_runtime.h`, `lib64/stubs` |
+| `cuda-crt-13-0` (13.0.88) | `include/crt/host_runtime.h` (2-arg `__cudaLaunch`) |
+| `libnvptxcompiler-13-0` (13.0.88) | static lib + `nvPTXCompiler.h` |
+| `cuda-cccl-13-0` (13.0.50) | `cccl/cuda/...` headers (libcudacxx) |
+
+Three local modifications the tree carries:
+
+1. **`bin/nvcc` is a shim** over `bin/nvcc.real` adding
+   `-allow-unsupported-compiler` (host GCC 16 > nvcc's max 13).
+2. **`targets/x86_64-linux/include` and `lib` are symlinks** to
+   `../../../include` / `../../../lib64` (the deb ships them as real
+   near-empty dirs; the stub pass must see the 13.0 headers, not
+   `/usr/include` 12.4).
+3. **`include/cuda` → `cccl/cuda`** symlink (CCCL 3.x layout; flashinfer
+   includes `<cuda/functional>`), and **`crt/math_functions.h`** patched with
+   `noexcept` on `rsqrt`/`rsqrtf` (glibc 2.42 declares them `noexcept(true)`;
+   the unpatched 13.0 declarations are incompatible in the C++ host stub).
+
+JIT caches: `~/.cache/sglang/jit/sm86/` and
+`~/.cache/sglang/.cache/flashinfer/0.6.18/86/` — clear both when a toolchain
+change lands.
+
+## Payload
+
+Eight safetensors shards (12,891,718,440 B total) + tokenizer and config
+files under `/home/dank/models/MiMo-V2.6-Distill-Qwen-9B-INT8-W8A16-AutoRound/`,
+all LFS sha256-verified by `old/download-frank-small-sglang-int8.sh`
+(`--verify-only` re-checks) — the script moved to `old/` when the endpoint
+cut over to llama.cpp + GGUF on 2026-10-01 night, and
+`scripts/download-frank-small.sh` now fetches the GGUF quant instead. The
+serve script refuses to start with a missing shard.
+
+## Client contract
+
+- **Alias:** `frank-small` (the `id` at `/v1/models`).
+- **Reasoning:** extracted to **`reasoning_content`** (live-verified
+  2026-10-01 — this SGLang build's qwen3 parser uses the same field name
+  as llama.cpp; models.yml sets `reasoningContentField:
+  "reasoning_content"`). `--reasoning-parser qwen3`.
+- **Tools:** `--tool-call-parser qwen3_coder`.
+- **Vision:** supported (in-checkpoint tower).
+- **Sampling:** SGLang has NO server-wide `--temperature`/`--top-p`/
+  `--top-k` — sampling rides on the request; the harness sends the Qwen
+  recipe (temp 0.7, top_p 0.8) via models.yml `compat.extraBody`.
+- **Reasoning effort:** the full OpenAI set is accepted (the chat template
+  has no effort gate).
+
+## Measured
+
+- Boot ~60 s with warm JIT caches (several minutes cold — the first boot
+  compiles the sgl-kernel + flashinfer kernels; the CUDA 13 toolchain
+  above is what makes that compile succeed).
+- Decode **83.9 tok/s** (no spec decode — nothing to accept).
+- Prefill ~2,050–3,370 t/s eager (4-slot shape: a 170,019-token prompt in
+  50.5 s, ~3,370 t/s; 2-slot shape: a 200,000-token prompt in 97.3 s wall
+  incl. 8 decode tokens, ~2,056 t/s).
+- 4-slot shape: 4 concurrent 54,027-token requests all answered in ~17 s
+  (216k tokens of live KV at once). 2-slot shape: two concurrent
+  50,000-token requests both answered (28.8 s wall).
+- **Compaction fit (the reslice's purpose):** a 200,000-token prompt —
+  well past frank-primary's 183,501-token compaction trigger (70% of its
+  262,144 window) — prefills and completes in ONE request.
+- Vision: a 64x64 red square answers "Red" with `image_tokens: 64`.
+
+## Rollback
+
+[[frank-small-llama-mimo-q5km]] is the live shape two steps forward; the
+practical rollback today is [[frank-small-llama-mimo-q6k]] (payload at
+`/home/dank/models/MiMo-Q6K/`). This era's own rollback, the llama.cpp
+Gemma4-12B QAT payload, was staged to
+`/home/dank/.AGENTTRASH/20261002-retired-llm-payloads/Gemma4-12B-QAT/` on
+2026-10-02; its unit/env/serve trio remains in `old/`.
+
+# Citations
+
+- Model: https://huggingface.co/letechlead/MiMo-V2.6-Distill-Qwen-9B-INT8-W8A16-AutoRound
+- SGLang: https://github.com/sgl-project/sglang
+- Serve script: `scripts/serve-frank-small.sh`; tunables: `scripts/frank-small.env`
@@ -1,7 +1,7 @@
 ---
 type: reference
-title: "halogen-flash-server on Dankclaw (frank-large)"
-description: "The RETIRED frank-large runbook (2026-09-16 → 09-19, rollback target) — peonist-ai/halogen-flash-server 0.5.8 serving Qwen3.8-Flash-Next (w4b) on the Dankclaw gfx1151 box: container shape, deployed flags, the memory model, the client contract, and the failure modes worth knowing. Replaced by [[frank-large-ciru-orca]] on 2026-09-19."
+title: halogen-flash-server on Dankclaw (frank-large)
+description: The active frank-large runbook — peonist-ai/halogen-flash-server 0.5.8 serving Qwen3.8-Flash-Next (w4b) on the Dankclaw gfx1151 box: container shape, deployed flags, the memory model, the client contract, and the failure modes worth knowing.
 tags:
  - llm
  - halogen
@@ -13,3 +13,466 @@ tags:
  - rocm
 timestamp: 2026-09-13
 ---
+
+# halogen-flash-server on Dankclaw (frank-large)
+
+> **ACTIVE again since 2026-09-16, and retuned.** It replaced the llama.cpp /
+> kingjones30-ROCmFPX STRIX deployment on 2026-09-11 (see
+> [[kingjones-strix-rocmfp4]]), was displaced for two days by ik_llama.cpp (see
+> [[frank-large-ik-uncensored]], the current rollback), and came back as the
+> shape the fleet asked for: two 200k-token conversations rather than the
+> native-context pair. Deployed as a rootless Quadlet container,
+> `dankclaw/frank-large/quadlet/frank-large.container`, port 7770. The RTX 3090
+> it does not touch now belongs to [[frank-small-qwen38-gsq-rco]].
+
+The engine is a from-scratch serving runtime for the Qwen3.8-Flash-Next
+architecture on AMD's gfx1151 (Strix Halo) — the same silicon this host runs.
+Upstream's own comparison table measures it at **roughly 4x the prefill
+throughput** of the llama.cpp-family runtimes it ships against (EngramHalo.cpp,
+ROCmFP4, CIRU-IU4) on a 32K prompt, and prefill is most of the wall clock on any
+prompt with real context. That, not decode, is the reason for the swap.
+
+## Shape
+
+One container runs both halves of the stack:
+
+| half | port | reachable |
+|---|---|---|
+| engine | 8730 | container loopback only — **unauthenticated, never publish it** |
+| OpenAI front-end | 8731 | published as host **7770** |
+
+The `all` mode is upstream's default and the shape a single-container host
+wants: the front-end talks to the engine over loopback inside the container, so
+there is one process tree, one log, and one published port.
+
+`HALOGEN_MODEL_ID=frank-large` sets the id reported at `/v1/models`, in
+`/health`, and in every response body. Requests are **never rejected for naming
+a different model** — the server serves whatever asks. Keeping the fleet alias
+is therefore free, and it means Hermes, the harness config, the OpenWebUI
+discovery, the `services.json` probe and the docs needed no rename.
+
+## Weights
+
+`peonist-ai/halogen-qwen3.8-flash-next` (HuggingFace, public and ungated):
+
+| file | size | role |
+|---|---|---|
+| `qwen38-flash-next-w4b.hgn` | 115.55 GiB | the checkpoint |
+| `qwen38-flash-next-w4b.overlay.hgn` | 2.40 GiB | **quality sidecar** — picked up automatically; the engine warns at startup if it is missing. **Refreshed 2026-09-16**: the sidecars are versioned *in place*, so a stale local copy is invisible from its file name — this one had read 2.31 GiB since it was first fetched while the Hub had moved on |
+| `qwen38-flash-next-vision.hgn` | 0.84 GiB | vision tower (see below) |
+| `tokenizer/` | ~21 MB | flat tokenizer dir; found beside the checkpoint, so no extra mount |
+
+`HALOGEN_DOWNLOAD` fetches the whole repo into the bind mount on **first start
+only** — the entrypoint acts only when the checkpoint is absent, so a restart
+never re-downloads, and `huggingface_hub` resumes an interrupted pull. The
+mount must be read-write for this. Roughly 127 GB; on Dankclaw it lands in
+`/home/dank/models/halogen`.
+
+**Refreshing the weights.** Delete the mount's contents (or the whole
+directory, then recreate it **empty**) and start the unit: the entrypoint sees
+the checkpoint absent and re-fetches the repo, resuming if interrupted. Two
+traps. First, the bind mount's source directory must *exist* — podman refuses
+the container with `Error: statfs /home/dank/models/halogen: no such file or
+directory` and the unit loops on `Restart=on-failure`; `mkdir -p` the empty
+directory and the fetch starts on the next start. Second, "refresh" only means
+"fetch what the Hub has now": on 2026-09-16 the re-fetch brought newer sidecars
+(`overlay.hgn` 2,477,677,120 → **2,572,466,560** bytes, `overlay-speed.hgn`
+2,383,306,048 → **2,478,095,488**) while the checkpoint returned
+byte-identical in size (124,068,083,904) — so an update can live entirely in
+the sidecars, and equal file sizes are not proof of an unchanged repo.
+
+`overlay-speed.hgn` (2.31 GiB on the Hub; 2.22 GiB before the 2026-09-16
+refresh) comes along with the repo but is unused: it trades the `o_proj`
+calibration for about 2 % of decode, and the quality sidecar is the default for
+a reason.
+
+## Deployed flags
+
+Everything else is configured by environment variable — there is no config
+file, and the flags are read once at startup. These are the departures from
+upstream's defaults:
+
+| flag | value | why |
+|---|---|---|
+| `HALOGEN_MODEL_ID` | `frank-large` | fleet identity; no client rename |
+| `HALOGEN_CTX` | `200192` | **set 2026-09-16**: exactly half the pool, so two conversations of the full window are always resident. Not a round 200,000 — pool/2 is. The model's native window is 262,144, so this is a real reduction in the per-conversation ceiling, and a client asking for more gets a 400 (never a silent truncation). The harness declares 200192 for `frank-large` |
+| `HALOGEN_KV_POOL_POSITIONS` | `400384` | **retuned 2026-09-16** (from 401,408): 2 x 200,192. Reserves ~11.0 GiB of KV, confirmed by the boot ledger as `400384 positions, ~27.2 GiB (16.2 GiB of it for everything but the pool, at max_tok 24576)` against a 33.5 GiB device budget — **no clamp**. Slots still oversubscribe the pool for any real request (admission reserves prompt + `max_tokens`), so a near-full pair that is both generating is pool-bound. `HALOGEN_KV_POOL_FIT` guards the boot: read the ledger, not this column |
+| `HALOGEN_KV_SLOTS` | `2` | **lowered from 3 on 2026-09-16** — one slot per half of the pool, and two is the fleet's shape now that Dankclaw serves two endpoints. Each slot is 111.0 MiB of O(1) state, so this hands ~111 MiB back as well |
+| `HALOGEN_MAX_TOK` | `24576` | **halved from the 32768 default on 2026-09-16** for frank-small, its co-tenant on the same host. The arena's variable term goes `8.40 x 2 → 8.40 x 1.5`, handing ~4.2 GiB back at roughly a 4-5 % prefill cost — the same lever at 16,384 costs 8-9 %. The vendor's own notes call 24,576 the largest arena that allocated with headroom to spare, so it stays inside their tested range. 65536 still cannot boot. It sizes a per-call PREFILL CHUNK, not the prompt ceiling: a 200k conversation is prefilled in arena-sized pieces |
+| `HALOGEN_HOST_RESERVE_GIB` | `20` | **raised from 15 with the same retune**, because the arena shrank: `budget = 121.2 − 67.7 − 20 = 33.5 GiB` against a measured need of ~27.2, so the fit check kept ~6 GiB of margin and did not clamp. It is **not** an allocation: raising it cannot return memory to the host, only take the pool away (see the ceiling section below) |
+| `HALOGEN_VISION_TOWER` | *unset* | **removed 2026-09-16.** The tower charges 1.97 GiB against the same device budget as the pool and the arena, and the fleet's image path moved to [[frank-small-qwen38-gsq-rco]] (llama.cpp with `--mmproj`). This server refuses image content parts, and `models.yml` declares `frank-large` text-only to match |
+| `HALOGEN_DOWNLOAD` | `peonist-ai/halogen-qwen3.8-flash-next` | first-start fetch |
+| `HALOGEN_VERBOSE` | `1` | verbose engine logging — the fullest startup account, on permanently so an incident report needs no restart-into-debug |
+
+Everything else is upstream's default, deliberately: prompt cache mode `2`
+(two entries per conversation — end-of-system-prompt and end-of-history — which
+already covers the shared-prefix case), `HALOGEN_CACHE_INPLACE=1` (the snapshot
+stays ~115 MiB at any depth), the speculative drafter `1` (byte-identical at
+temperature 0, and it stands down whenever a second conversation is
+generating), the quality sidecar, `HALOGEN_ADMIT_CHUNK=0` (a non-zero value
+gives up the byte-identity property), and the default `HALOGEN_MAX_TOKENS_CAP=65536`.
+
+## Memory model
+
+Three numbers explain the whole box:
+
+- **~68 GiB pinned weights.** The checkpoint is mapped and registered with the
+  GPU in place, never copied. The kernel counts the locked pages as reclaimable
+  file cache, so `free`, `MemAvailable` and every dashboard built on them
+  **overstate free memory by the size of the model**. The server prints the
+  corrected figure at startup; believe that line, not Grafana.
+- **~28 KiB per KV position** (measured: 262,144 positions = 6,656 MiB of KV,
+  ~7.2 GiB with block scratch), plus the prefill arena. The 2026-09-16 ledger
+  reads the two together in the engine's own words: `400384 positions, ~27.2
+  GiB (16.2 GiB of it for everything but the pool, at max_tok 24576)` — so
+  ~11.0 GiB of pool and 16.2 GiB of arena, working slack and (no longer) a
+  vision tower. The engine's device-side budget stops at roughly 47 GiB on a
+  128 GB machine *with the weights pinned* — and lower on a machine carrying
+  more besides this server, which Dankclaw is.
+- **`HALOGEN_HOST_RESERVE_GIB`** is what is left for the page cache the lookup
+  table is read through. Too little and long prompts crawl with the disk busy
+  and the process in uninterruptible sleep — that is a cache problem, not a
+  memory problem, and the fix is a *smaller* pool, not a bigger one.
+
+`HALOGEN_KV_POOL_FIT=1` (default) measures that budget at startup and lowers
+the pool itself if the configured one will not fit, printing what it chose.
+Read those lines on every restart; they are the ground truth for sizing.
+
+**Measured ledger, seven restarts, read live 2026-09-12.** The engine's own
+startup account, including three deliberate experiments:
+
+| pool | max_tok | reserve | KV | arena | need | budget | verdict |
+|---|---|---|---|---|---|---|---|
+| 262144 (baseline) | 32768 | 24 | 7.2 | 20.6 | 29.8 | 29.5 | boots 0.3 over — prints **no** sizing lines |
+| 524288 | 16384 | 24 | 14.4 | 12.2 | 28.6 | 29.5 | fits, 0.9 spare |
+| 524288 | 32768 | 24 | 14.4 | 20.6 | 37.0 | 29.5 | clamped to 262144, deploy still green |
+| 524288 | 65536 | 24 | — | 37.3 | 46.5 | 29.5 | **OOM — will not boot** |
+| 524288 | 32768 | **15** | 14.4 | 20.6 | 37.0 | **38.5** | **deployed — both features** |
+| 524288 | 32768 | 16 | 14.4 | 20.6 | 37.0 | 37.5 | fits, footprint unchanged — see the ceiling below |
+| **400384** | **24576** | **20** | **11.0** | **16.4** | **27.2** | **33.5** | **deployed 2026-09-16 — ledger need 27.2, no clamp** |
+| 401408 | 32768 | 15 | 11.0 | 20.6 | 33.4 | 38.5 | 2026-09-13 → 09-16, superseded by the retune |
+
+### The sizing rule
+
+One budget covers the pool, the arena **and** the vision tower together:
+
+```
+device budget = host RAM - resident weights - HALOGEN_HOST_RESERVE_GIB
+              = 121.2    - 67.7             - reserve
+                                    reserve 24 -> 29.5 GiB
+                                    reserve 15 -> 38.5 GiB
+
+need = KV (28 KiB/position) + arena + vision (1.97 measured)
+arena = 3.80 GiB + 8.40 GiB x (HALOGEN_MAX_TOK / 16384)
+```
+
+The `121` the engine prints is rounded **down** from `MemTotal 127104060 kB =
+121.2159 GiB`, so subtracting the displayed figures gives 29.3 rather than 29.5.
+Use the unrounded value. Two of the three terms are fixed hardware and
+checkpoint facts; `HALOGEN_HOST_RESERVE_GIB` is the only one you own — which
+buys the *fit margin*, not the host's memory. It is a term in this inequality,
+not an allocation, and the section at the end of this page shows the live test
+that proves it.
+
+Trust the engine's printed requirement over this model wherever they disagree:
+a boot clamped at reserve 24 stated `524288 positions need ~35.0 GiB (plus 1.5
+of margin)`, i.e. **budget >= 36.5**, against a modelled 37.0 here. Close
+enough to plan with, and the authoritative number to cut the other way.
+
+The arena term is **linear in `HALOGEN_MAX_TOK`** and predicts every measured
+point to 0.1 GiB — including upstream's three published "on the device" figures
+(524,288 → ~35, 786,432 → ~42, 1,048,576 at `max_tok=16384` → ~41), which are
+**pool + arena totals at the default `max_tok`**, not pool alone. That is why
+their largest pool "needs `HALOGEN_MAX_TOK=16384`": at 32768 a 1M pool would be
+49.4. Reading those figures as pool-only is the trap that makes a doubling look
+affordable when it is not.
+
+At the baseline reserve of 24 the doubled pool leaves `29.5 - 1.97 - 14.4 =
+13.1` for the arena, so `max_tok <= 18,130` and **16384 is that reserve's
+ceiling, not a conservative setting.** The corner check agrees: at 32768 only
+7.0 GiB is left for the KV, ~254k positions, and the pool drops back toward
+262144. Cut the reserve to 15 and the same arithmetic gives `38.5 - 1.97 - 14.4
+= 22.1` for the arena — `max_tok <= 35,700`, which is exactly why 32768 now
+fits unmodified.
+
+### What the reserve actually costs
+
+The device budget is not the whole story. Memory position by configuration
+(401,408 is the 2026-09-13 deployed row, read from its boot ledger):
+
+| config | held | host left | contiguous 2 MiB blocks |
+|---|---|---|---|
+| 262144 / 32768, reserve 24 | 96.3 | 12.1 GiB | 2081 |
+| 524288 / 16384, reserve 24 | 94.6 | 13.8 GiB | 3742 |
+| 524288 / 32768, reserve **16** | **103.5** | **7.1 GiB** | 293 |
+| **401408 / 32768, reserve 15** | **100.0** | **9.1 GiB** | **1396** |
+| **400384 / 24576, reserve 20** | **95.4** | **14.7 GiB** | **475** |
+
+At the doubled setting (2026-09-12) the server held 103.5 of 121 GiB and the
+engine volunteered the stall note. It did not print it at the 401,408 trim, and
+it prints it again after the 2026-09-16 retune. Three data points, and they do
+not line up with a "free memory below X" reading: 310 blocks printed, 1396 did
+not, 475 did. So treat the note as "the machine is tight" and read the
+`host memory left` line rather than waiting to be told:
+
+```
+halogen: NOTE this server now holds most of the machine, and 310 contiguous
+2 MiB blocks remain. It runs best on a host of its own. If other large
+processes share this machine they will compete for what is left, and when it
+runs out BOTH they and this server can stall for minutes at a time at full
+CPU with no disk activity and no output, which is not a crash and needs no
+restart.
+```
+
+That is the failure mode to watch, and it is the least diagnosable of the
+three: not a silent clamp, not a boot refusal, but a minutes-long freeze that
+looks like a hang, for the neighbours as much as for frank-large. `free` is
+useless for spotting it — `MemAvailable` still reads ~74 GiB because the
+page cache counts as reclaimable. Watch the `host memory left for everything
+else` line and the contiguous-block count at boot instead.
+
+28 co-tenant containers (~4.3 GiB working set) stayed up through the change,
+so it is survivable.
+
+### The reserve cannot buy air — ceiling 16
+
+The last row above is the experiment that proves it. Raising the reserve from
+15 to 16 left `held` and `host left` **byte-for-byte identical** (103.5 / 7.1
+GiB). `HALOGEN_HOST_RESERVE_GIB` feeds the fit-check budget and is *not* an
+allocation; the actual footprint is `weights + KV + arena + vision`, none of
+which the reserve touches. So the knob has exactly two regimes:
+
+| reserve | budget | vs required 36.5 | effect |
+|---|---|---|---|
+| 15 | 38.5 | +2.0 | deployed |
+| 16 | 37.5 | +1.0 | no change to footprint, thinner margin — pointless |
+| 17 | 36.5 | +0.02 | knife edge, inside the noise of the 67.7 term |
+| **18** | **35.5** | **−1.0** | **clamps the pool to 262144** |
+| 24 | 29.5 | −7.0 | clamps the pool to 262144 |
+
+The 36.5 requirement is the engine's own, read from a clamped boot:
+`524288 positions need ~35.0 GiB (plus 1.5 of margin)`. Above 16 the only
+thing a bigger reserve can do is take the doubling away.
+
+**To actually free host memory, change the pool or the arena**, and the reserve
+then becomes free to rise: `HALOGEN_KV_POOL_POSITIONS=262144` hands back
+7.2 GiB from the doubled pool, `HALOGEN_MAX_TOK=16384` hands back 8.4. Both
+are one-line edits that the fit check will confirm on the next boot. The
+2026-09-13 trim to 401,408 was this remedy applied at half strength —
+`HALOGEN_MAX_TOK` untouched, and on the host-left metric it bought +2.0 GiB
+with a 4.5x improvement in contiguous blocks (310 → 1396), less than the
+modelled 3.4 because the engine holds the freed KV as working slack.
+
+### The `WARNING` line only appears when the fit check had to move
+
+This one cost me a wrong conclusion, so it is worth stating plainly. When the
+configured pool already fits, the engine reserves it **and prints nothing** —
+the 09-11 baseline boot goes straight from the pre-flight line to
+`checkpoint: mapped` with no `kv pool:` sizing block at all. The sizing block and
+its `WARNING even 262144 positions ... host RAM cannot spare it` appear **only
+after a clamp**.
+
+So the absence of a warning in old logs is not evidence of headroom. Both
+`262144/32768` rows above need 29.8 against a 29.5 budget — 0.27 over — and the
+baseline has lived in that state since it was deployed without incident,
+because the 24 GiB reserve is itself generous (the host reports 12–14 GiB free
+at that point). The engine's budget is a conservative floor, not a hard wall.
+
+### Two failure modes, and only one is dangerous
+
+Raising `max_tok` past the ceiling does not shrink the pool — the arena alone
+outgrows the budget, and the server cannot allocate at all:
+
+```
+kv pool: 524288 positions need ~51.7 GiB ... LOWERING THE POOL TO 262144 (~44.5 GiB).
+kv pool: WARNING even 262144 positions (the context, the smallest pool that can
+         serve one request) needs ~44.5 GiB and host RAM cannot spare it.
+dmalloc: FAILED requesting 0.250 GiB after 39.970 GiB in 566 allocations (out of memory)
+```
+
+This is the quiet one: the unit stays `active`, the container reports
+`Up (starting)`, and `deploy.sh` sits in its health wait for the whole
+`LLM_WAIT` rather than failing fast. It confirms upstream's warning that a
+large `HALOGEN_MAX_TOK` means "the server will not start", not "it will be
+slower".
+
+Under-running the ceiling is far more benign and far easier to miss. At 524288
+with the arena back at 32768, `HALOGEN_KV_POOL_FIT` lowered the pool and
+`deploy.sh` still reported healthy in 12 seconds:
+
+```
+kv pool: 524288 positions need ~35.0 GiB (plus 1.5 of margin) and host
+         RAM cannot spare it. LOWERING THE POOL TO 262144 (~27.8 GiB).
+```
+
+A green health check does not mean the pool you asked for is the pool you got —
+read the `kv pool:` lines after every restart.
+
+The 47.7 GiB n-gram lookup table is never held in RAM; it is read through
+the file cache. The 2.0 GiB the firmware carves for the iGPU appears in no
+`/proc/meminfo` figure at all.
+
+**Co-tenancy.** `HALOGEN_FLASH_PIN_TRUNK=1` (default) is what the published
+decode numbers were measured with, and it means this container holds most of
+the host. If a co-tenant then takes the remainder, both it and the engine can
+stall for minutes at a time at full CPU with no disk activity and no output.
+That is documented behaviour, not a crash, and it needs no restart.
+`HALOGEN_FLASH_PIN_TRUNK=0` gives the memory back and costs several times the
+decode speed — a last resort, not a tuning option.
+
+## Client contract
+
+The differences from llama.cpp that bite are all in the request, not the
+deployment:
+
+- **No server-side sampler defaults.** `temperature` absent or `0` is
+  **greedy** decode; above 0 the request samples on the same drafter. Supported
+  fields: `temperature`, `top_p`, `top_k`, `min_p`, `seed`,
+  `presence_penalty`, `frequency_penalty`, `logit_bias`, `logprobs`. Refused
+  with a 400: `top_logprobs`, `logprobs` with `stream: true`, `n > 1`, and any
+  value outside its defined range (values are refused, never clamped).
+  The fleet runs greedy for clients that stay silent — deterministic, and
+  byte-identical with the drafter.
+- **The token budget covers thinking.** The default is **8192**, and a budget
+  that runs out mid-thought returns `finish_reason: "length"` with an **empty
+  `content`** and the partial reasoning in `reasoning_content` — which most
+  clients do not display. Above the cap (`HALOGEN_MAX_TOKENS_CAP`, 65536) a
+  request is a 400 rather than a silent truncation. `max_tokens`,
+  `max_completion_tokens` and `max_output_tokens` all work and must agree if
+  more than one is sent.
+- **A budget above (window − compaction-threshold) is a guaranteed 400 zone.**
+  The fit check (`prompt + max_tokens <= HALOGEN_CTX`) refuses, and the harness
+  only compacts at its own threshold — with OMP at 70 % of a 200,192 window
+  (140,134), any prompt in (136,192, 140,134] with a 64,000 budget overflows
+  the window while sitting below the compaction trigger. That collided live on
+  2026-09-17 (prompt 138,898 + 64,000 = 202,898 > 200,192). The fix is
+  harness-side: keep the declared `maxTokens` at or under
+  `window − threshold` (57,344 chosen) so no pre-compaction request can miss
+  the fit check. Do not fix it by raising `HALOGEN_CTX` past pool/2 — that
+  breaks the two-resident-conversations property.
+- **`reasoning_effort`** is accepted at the top level and inside
+  `chat_template_kwargs`: `minimal`, `low`, `medium`, `high`, `xhigh`. The
+  model's default is `xhigh`. `minimal` aliases the template's `low` and still
+  reasons; `enable_thinking: false` turns thinking off; `preserve_thinking` is
+  honoured. `preserve_reasoning` is a **400** — it is the llama.cpp-era key.
+- `/v1/models`, `/health` and completions all report the served id, and
+  `reasoning_content` carries the reasoning trace.
+
+**Pool admission is the one arithmetic trap.** A request reserves its prompt
+**plus** `max_tokens` positions when it is admitted; if the pool cannot hold it
+the request waits in arrival order, and after the queue timeout (3600 s) it
+gets a `503 engine_busy`. A prompt large enough that prompt + budget exceeds
+the pool can therefore never be admitted at all. The 2026-09-16 shape is a
+400,384-position pool with `HALOGEN_CTX` 200,192 and two slots: two
+conversations at the ceiling fit the pool exactly, so a pair that is both
+near-full and both generating is pool-bound — the second waits for the first to
+release its region. The pool binds **how many** conversations stay resident,
+not how long one may be; the older 524,288 / 401,408 numbers live in the ledger
+tables above.
+
+## Vision
+
+**None here since 2026-09-16.** `HALOGEN_VISION_TOWER` is deliberately unset:
+the tower is a 1.97 GiB charge against the same device budget as the pool and
+the arena, and the fleet's image path moved to [[frank-small-qwen38-gsq-rco]],
+which loads the BF16 projector alongside its own weights. This server refuses
+image content parts and says why, rather than ignoring them:
+
+```json
+{"error": {"message": "this server does not accept images: the engine was
+started without a vision tower (HALOGEN_VISION_TOWER)", "type":
+"invalid_request_error"}}
+```
+
+The llama.cpp side handles images the same way this engine used to: a `data:`
+URL or bare base64, about 2,040 tokens of context per image, and `http(s)` URLs
+refused deliberately (fetching one would make the server issue outbound
+requests to wherever a client pointed it). Audio and video have no path
+anywhere in the fleet — that left with [[frank-npu]].
+
+## Deploy and operate
+
+```bash
+./scripts/deploy.sh dankclaw --service frank-large
+```
+
+The handler retires the ik-era **root** unit first (stop, disable, move
+`/etc/systemd/system/frank-large.service` into
+`~/.AGENTTRASH/20260916-halogen-restore/host-units` — never `rm`; this host's
+`disable` may delete the file outright, which the handler now reports), stops
+the Quadlet's own unit if it is running, then pulls
+`ghcr.io/peonist-ai/halogen-flash-server:0.5.8`, re-tags it to the
+`localhost/` alias the unit references, and **waits for port 7770 to be free**
+before installing and starting anything.
+
+That last gate is not paranoia. On the first cutover attempt the ik server
+logged "Deactivated successfully", and pasta — rootless podman's userspace
+network stack, which binds the published port at container CREATE — still
+reported `Address already in use` a second later, exiting the unit with code
+126. Only `Restart=on-failure` recovered it, minutes later. A deploy that
+starts a doomed container has not deployed anything, so `wait_port_free` polls
+every socket on the port (listener, accepted, TIME_WAIT) and fails loudly with
+the socket table if it never clears.
+
+Boot persistence comes from `[Install] WantedBy=default.target` in the unit
+file plus linger on the deploy user — generated units **cannot** be
+`systemctl --user enable`d on this host.
+
+```bash
+systemctl --user status frank-large.service     # the generated unit
+podman logs frank-large | head -40              # version banner + memory ledger
+curl -s http://127.0.0.1:7770/health            # 200 only after the engine PONGs
+curl -s http://127.0.0.1:7770/v1/models
+```
+
+`/health` asks the **engine** for a PONG before answering, so a 200 means
+"ready to serve", not merely "process alive". During the first-start download
+it stays unhealthy by design.
+
+## Troubleshooting
+
+| symptom | cause |
+|---|---|
+| `dmalloc: FAILED requesting … (out of memory)` then `HIP … out of memory` | the KV pool did not fit. `HALOGEN_KV_SLOTS` will not fix it (slots share the pool); lower `HALOGEN_KV_POOL_POSITIONS` to 262144, then `HALOGEN_MAX_TOK` to 16384 (~9 % prefill) |
+| starts, answers short prompts, collapses to a few tok/s on a long one, disk busy, process in uninterruptible sleep | short of **file cache**, not memory — lower the pool and/or raise `HALOGEN_HOST_RESERVE_GIB` |
+| long replies that look cut off | read `finish_reason`: `stop` means you have the whole answer, `length` means you ran out of budget (thinking counts) |
+| a request that never returns, then `503 engine_busy` | the pool could not reserve prompt + `max_tokens`; see the admission trap above |
+| both the engine and a co-tenant stall for minutes at full CPU | the pinned-trunk trade; see Memory model |
+| `free`, `MemAvailable`, Grafana all wrong by ~68 GiB | expected — locked weights are counted as reclaimable |
+
+`HALOGEN_VERBOSE=1` ships in the unit, so the fullest startup account is
+already in the log when something goes wrong. `HALOGEN_DMALLOC_LOG` is
+deliberately **not** set: it logs every allocation over 64 MB, which is
+steady-state noise on a shared host — raise it only for a live OOM report.
+The lines worth grepping first:
+
+```bash
+podman logs frank-large 2>&1 | grep -E '^(dmalloc|kv pool):'
+```
+
+Retrieval: `podman logs frank-large` covers the **current** instance only —
+the Quadlet runs `--replace --rm`, so everything before the last recreate is
+gone from the container log. `journalctl --user -u frank-large.service` is
+the durable record. Both are rootless: `sudo podman logs frank-large` looks
+in root's storage and answers "no such container".
+
+## Rollback
+
+The predecessor is intact: the ik payload stays at
+`/home/dank/models/mradermacher-Qwen3.8-Flash-Next-Uncensored-i1-Q4_K_M`, the
+ik build at `/home/dank/GitPackages/ik_llama.cpp/build`, and its unit, env,
+serve script and the Qwen-Sharp template are archived in the repo under
+`scripts/.AGENTTRASH/*.ik-llama.cpp` and `*.ik-llama-qwen-sharp`. Rolling back
+is a `git revert` of this change set plus a deploy — **but the Quadlet must be
+removed from `~/.config/containers/systemd/` first**, or the restarted root
+unit and the still-installed container both claim port 7770. The port gate in
+the handler now covers exactly that race. See [[frank-large-ik-uncensored]].
+
+The `amd_iommu=off` kernel argument that came with this change is separate and
+reverts on its own — see [[dankclaw-iommu-translation-off]].
+
+## Citations
+
+- [peonist-ai/halogen-flash-server](https://github.com/peonist-ai/halogen-flash-server) — README, `docs/FLAGS.md`, `deploy/entrypoint.sh`, `docker-compose.yml`, CHANGELOG (v0.5.8)
+- [peonist-ai/halogen-qwen3.8-flash-next](https://huggingface.co/peonist-ai/halogen-qwen3.8-flash-next) — weights and file inventory
+- [Qwen/Qwen3.8-Flash-Next](https://huggingface.co/Qwen/Qwen3.8-Flash-Next) — base model
@@ -1,14 +1,26 @@
 # LLM Fleet — Knowledge Base

-Documentation for the LLM inference fleet on the two hosts (2026-09-27):
-**Dankserver runs R9V radiance-vLLM on frank-primary** (the drwolfen fork
-serving the unsloth Qwen3.8-Flash-Next Q8_0 + MTP4 across both Radeon AI
-PRO R9700s on :7770, docker-pinned runtime driven through a docker→podman
-shim, the Qwen-Sharp template, 131072 ctx), and Dankclaw runs two llama.cpp endpoints —
-stock llama.cpp (unsloth b11030, ROCm gfx1151; frank-large, unsloth
-Qwen3.8-27B UD-Q8_K_XL + the z-lab DFlash2 Q4_K_M drafter on the gfx1151
-iGPU, port 7770) and mainline llama.cpp (frank-small, HauhauCS Gemma4-12B
-QAT Q4_K_M with MTP and vision on the RTX 3090, port 7771).
+Documentation for the LLM inference fleet on the two hosts (2026-10-03):
+**Dankserver runs the tcclaviger RDNA4 vLLM fork on frank-primary** (the
+digest-pinned `tcclaviger/vllm` image driving `podman run` directly, serving
+orcarouter/Qwen3.8-Flash-Next-Uncensored-FP8 across both Radeon AI PRO R9700s
+on :7770, TP=2 with automatic tiered-expert offload, **shape E since
+2026-10-02: 3 slots x 200704 ctx** with the KV pool sized for **~2 full-length
+conversations** — measured 398,658 tokens = 1.99x — fp8 KV, no speculative
+decode, MNBT 1024, MoE gather 32x64, util 0.98, reasoning/tool parsers wired
+(qwen3 / qwen3_xml) with preserve_thinking on, plus **`--expert-precise-memory`
+(the arena grows to 17.22 GiB / 7,522 slots, +16%), `GPU_MAX_HW_QUEUES=2`, and a
+mounted cache dir** (the image ships no `/root/.cache` at all; unmounted, every boot re-pays
+`torch.compile took 181.20 s`). `--max-num-seqs` must stay 3:
+SEQS 2 cannot compile this model. Rollback frank-primary-fp8-mtp-20261001),
+and Dankclaw
+runs two endpoints — **halogen container (frank-large, port 7770:
+ghcr.io/peonist-ai/halogen @ pinned digest serving peonist-ai
+/halogen-qwen3.8-27b, a 33.4 GiB .hgn checkpoint that loads in no other
+engine, DFlash2 spec decode, ONE slot at 200,700 ctx, text-only — vision
+lives on frank-small)** and
+**llama.cpp CUDA (frank-small, port 7771: MiMo-V2.6-Distill-Qwen-9B Q6_K
+GGUF, 2 x 200,704, fully unquantized f16 KV, on the RTX 3090)**.

 > 2026-08-22: the former `vllmScripts` repository is merged into the
 > serverAppConfigs repository. This knowledge base moved from the old repo's
@@ -22,24 +34,25 @@ graph LR
    subgraph Dankserver[AMD Dankserver - Ubuntu 26.04]
        A[2x Radeon AI PRO R9700
 gfx1201, 32 GiB each]
-        B2[R9V radiance-vLLM fork, docker-pinned
-via docker-podman shim
-frank-primary, port 7770]
-        C[Qwen3.8-Flash-Next Q8_0
-TP=2 tiered experts, MTP4, sharp template, 131072 ctx, vision]
+        B2[tcclaviger RDNA4 vLLM fork, digest-pinned
+podman run, frank-primary, port 7770]
+        C[Qwen3.8-Flash-Next FP8, no spec decode
+TP=2, expert offload automatic, 3 x 200704 ctx,
+fp8 KV pool ~2 full-length, MNBT 1024, gather 32x64, qwen3 parsers,
+expert-precise-memory arena 7,522 slots, /root/.cache mounted]
    end

    subgraph Dankclaw[AMD Strix Halo - Ubuntu 26.04]
        E2[RTX 3090 (PCIe x4 dock) +
 gfx1151 iGPU 114 GiB GTT]
-        E[stock llama.cpp (unsloth b11030, gfx1151)
+        E[halogen container gfx1151, digest-pinned image
 frank-large, port 7770]
-        F[Qwen3.8-27B UD-Q8_K_XL + DFlash2 Q4_K_M drafter
-262144 ctx / 1 slot, DFlash2 depth 4, q8_0 KV]
-        G[mainline llama.cpp 37b53fd
+        F[halogen-qwen3.8-27b .hgn p1w4d-d2 + DFlash2 drafter
+1 x 200700 ctx, d2 KV, text-only]
+        G[llama.cpp master CUDA sm_86
 frank-small, port 7771]
-        H[Gemma4-12B QAT Q4_K_M
-4 x 262144 ctx, MTP + vision + audio]
+        H[MiMo-V2.6 Q5_K_M GGUF
+2 x 200704 ctx, f16 K + f16 V, flash-attn, mmproj vision]
        E2 --> E --> F
        E2 --> G --> H
    end
@@ -58,20 +71,61 @@ frank-small, port 7771]

 ## Active Deployment

- [[frank-primary-r9v-radiance-qwen38-q8]] — the ACTIVE frank-primary
-  runbook (2026-09-27, root systemd unit, port 7770): the R9V
-  radiance-vLLM fork (drwolfen/radiance-vllm-qwen4exp, docker-pinned
-  image bundle `2dac17a2`, driven through a docker→podman shim) serving
-  the unsloth Qwen3.8-Flash-Next-GGUF Q8_0 (175.30 GiB, 6 shards) +
-  blockfp8 MTP4 (depth 4) + vision on BOTH R9700s — TP=2 tiered experts
-  (80-slot LRU cache), PLE n-gram PINNED in host RAM (54.4 GB), CED off,
-  131072 ctx, the Qwen-Sharp v22.4.0 template via the launcher's
-  `R9V_CHAT_TEMPLATE_FILE` passthrough (fork `30d4a32`), the quantized
-  inject-shard dev-overlay adapter (fork `363d606`), reasoning channel
-  `reasoning`, parser `qwen3_coder`. The tuned shape lives in the
-  on-host q8xl setup state. Rollback:
-  [[frank-primary-r9v-radiance-qwen38-uncensored]]
- [[frank-primary-r9v-radiance-qwen38-uncensored]] — RETIRED 2026-09-27
+- [[frank-large-halogen27b-dflash2]] — **ACTIVE since 2026-10-05**
+  (root systemd unit + podman, port 7770): the digest-pinned
+  `ghcr.io/peonist-ai/halogen` image serving `peonist-ai/halogen-qwen3.8-27b`
+  (`qwen3.8-27b-p1w4d-d2.hgn`, 33.4 GiB) with the DFlash2 drafter on ONE slot
+  of 200,700 ctx (slots>1 disables speculation — engine law). Publisher card
+  31.7 t/s mean across eight text workloads. Accepted tradeoffs: TEXT-ONLY
+  (the engine ignores the vision encoder; frank-small owns images) and the
+  CENSORED base model (the ROCmFPX AGENT file was Uncensored by construction).
+  No server-side sampler — the thinking recipe rides the wire in models.yml.
+  Rollback: revert the swap commit; the ROCmFPX payload + fork build +
+  `build-rocmfpx.sh` pin all remain on disk
+- [[frank-small-llama-mimo-q6k-2x200704]] — **ACTIVE since 2026-10-03**
+  (root systemd unit, port 7771): llama.cpp master (CUDA sm_86) serving
+  bartowski/MiMo-V2.6-Distill-Qwen-9B-GGUF **Q6_K** on the RTX 3090, TWO
+  slots of 200,704 tokens (pool 401,408) with FULLY UNQUANTIZED KV (f16 K +
+  f16 V) and the mmproj for vision — 94.4 tok/s decode (the rung costs a
+  measured -10 % vs Q5_K_M: weight-bandwidth-bound), batch 4096 / ubatch 2048
+  since 2026-10-05 (+4.6 % prefill, measured), 2,279 MiB of the card
+  left free. Probe-size trap (>=256 px images or the model says "video")
+  documented. Rollback: one MODEL_FILE line back to Q5_K_M
+- [[frank-small-llama-mimo-q5km]] — **RETIRED 2026-10-02 → 10-03 (rollback
+  target; payload on disk; owns the measured boot-pair table)**: the same
+  runtime and the same 2 x 200,704 + f16 KV shape one quant rung lower —
+  104.8 tok/s, 3,571 MiB true free
+- [[dankserver-qwen38-fp8-expert-offload-tuning]] — **ACTIVE since
+  2026-10-02 (shape E)** (root systemd unit, port 7770): the tcclaviger RDNA4
+  vLLM fork serving orcarouter/Qwen3.8-Flash-Next-Uncensored-FP8 across both
+  R9700s, TP=2 with automatic expert offload — `--max-model-len 200704` x 3
+  slots, fp8 KV, pool 3,150,000,000 B/rank (measured 398,658 tokens = 1.99x),
+  MNBT 1024, gather 32x64, no spec decode, prefetch off, plus
+  `--expert-precise-memory` (arena 17.22 GiB / 7,522 slots), GPU_MAX_HW_QUEUES=2
+  and a mounted compile cache. Why `--max-num-seqs`
+  must stay 3 (SEQS 2 cannot compile this model), why the arena's levers are
+  `max_model_len` and the overhead term rather than the KV pool, and the measured envelope
+- [[frank-primary-glm53-flash-w4a16]] — RETIRED 2026-09-30 → 10-01 (the
+  failed GLM experiment; frank-primary runs Qwen FP8 again, see the intro):
+  the tcclaviger RDNA4 vLLM fork
+  (digest-pinned `tcclaviger/vllm@sha256:da4029666…`, plain `podman run`,
+  no shim) serving Intel/GLM-5.3-Flash-W4A16-AutoRound — AutoRound symmetric
+  int4 group 128, ~170 GiB, 34 shards + 2 extras, 113,074 tensors — on BOTH
+  R9700s, TP=2, with the MoE experts offloaded to pinned host RAM (12,096
+  experts/rank, 73.64 GiB) behind a 14.51 GiB / 2,383-slot VRAM LRU arena; the
+  MoE kernel is libr4d's grouped W4A16 GEMM (`R4dW4a16MoEMethod`, reached
+  because the checkpoint's `quant_method: auto-round` is overridden to `inc` —
+  a compressed-tensors W4A16 checkpoint of the same model has NO offload recipe
+  and refuses to serve). 131,072 ctx x 2 seqs, bf16 KV, util 0.98, prefix
+  caching, vision and `glm47` tool/reasoning parsers, MTP OFF and prefetch OFF
+  by measurement. **Measured decode 9.9-10.0 tok/s, prefill 918 tok/s — the
+  35 tok/s target is NOT met, bounded by the fork's expert-gather read path
+  (~8.2 GB/s effective vs a 22.74 GB/s link) at a 0.53-0.66 cache hit rate.**
+  Rollback: commit `2339fae` (the qwen FP8 + MTP4 shape of 2026-09-29 → 09-30)
+- [[frank-primary-r9v-radiance-qwen38-q8]] — RETIRED 2026-09-30 (two swaps
+  back; payload and R9V state dirs intact on disk, git tag
+  frank-primary-r9v-q8-20260929 is the last pre-cutover commit):
+ [[frank-primary-r9v-radiance-qwen38-uncensored]] — RETIRED 2026-09-27
  (same-day swap to the Q8_0 payload; rollback target, state + payload
  intact on disk): the R9V radiance-vLLM fork serving
  Dyluhn/Qwen3.8-Flash-Next-Uncensored-R9V-IQ4_XS (92.39 GiB, 3 shards) +
@@ -149,14 +203,37 @@ frank-small, port 7771]
  (bs1/bs3/bs4/bs6, 8k–700k KV) on Dankserver RTX 3090
 - [[deployment]] — deploy and update process for both servers
 - [[frank-redo-2026-08-17]] — the 2026-08-17 topology change
- [[frank-large-gufo-qwen38-27b]] — ACTIVE frank-large runbook since
-  2026-09-28 (root systemd unit, port 7770): gufo (gufo-org/gufo) v0.1.1, the
-  Strix Halo inference engine, built from source on Dankclaw, serving
-  unsloth/Qwen3.8-27B-GGUF:UD-Q8_K_XL (27B dense) with the z-lab DFlash2
-  Q4_K_M drafter (adaptive block policy) on the gfx1151 iGPU — one full
-  262,144-token conversation, greedy, text-only, the two from-source build
-  workarounds, the client-contract delta, the measured prefill/decode trade
-  against llama.cpp, troubleshooting
+- [[frank-large-rocmfpx-q6agent-uncensored-mtp]] — **RETIRED 2026-10-03 →
+  10-05 (rollback target; payload, fork build and build script on disk)**:
+  the pinned ROCmFPX fork serving the uncensored protected-heads
+  Q6_0_ROCMFPX_AGENT rung with the repo's MTP-Q4_0 head drafting 4 tokens,
+  one 262,144 window, f16 KV; measured 27.6-28.1 t/s prose decode at 86.8 %
+  acceptance. Replaced by the halogen DFlash2 endpoint; the ngram-mod-onto-MTP
+  rejection (-15 %, recurrent-state rollback) stays a live lesson for this
+  architecture.
+- [[frank-large-rocmfpx-q8agent-mtp]] — **RETIRED 2026-10-02 → 10-03 (the
+  rollback target; payload still on disk)**: the same pinned ROCmFPX fork
+  serving the ALIGNED `Q8_0_ROCMFPX_AGENT` (ftype 115, 17.0 t/s, MTP
+  acceptance 0.462) — the MTP-vs-none +123 % A/B, the trap catalogue
+- [[frank-large-gufo-qwen38-27b]] — **RETIRED 2026-10-02 (history; the
+  payload dirs were removed from disk 2026-10-03, so this rollback is a
+  re-download)** (root systemd unit, port 7770): gufo
+  (gufo-org/gufo) v0.1.1, the Strix Halo inference engine, built from
+  source on Dankclaw, serving unsloth/Qwen3.8-27B-GGUF **UD-Q4_K_XL**
+  with the z-lab DFlash2 Q4_K_M drafter — ONE session at the full native
+  262,144 window, unquantized f16 KV. Rolled back by Frank after one day
+  of [[frank-large-vllm-mimo-v26-bf16]]. The two from-source build
+  workarounds, the client-contract delta, the measured prefill/decode
+  trade against llama.cpp
+- [[frank-large-vllm-mimo-v26-bf16]] — RETIRED 2026-10-01 (one day;
+  rollback by Frank after the autoresearch loop): the digest-pinned
+  kyuz0/vllm-therock-gfx1151 image serving
+  XiaomiMiMo/MiMo-V2.6-Distill-Qwen-9B in FULL BF16, 4 x 262144 ctx,
+  CROSS-MODEL MTP from Qwen/Qwen3.5-9B — kept for the measured record:
+  battery decode 6.19 t/s (MTP-2) vs 9.01 t/s (no draft — spec NET-
+  NEGATIVE on real chat+reasoning traffic), the roof math, the greedy-text
+  numerics trap, and the death of every ≥6-bit vLLM quant path on gfx1151
+  (quark int8 rejected, FP8 kernels CDNA-only, torchao unwired)
 - [[frank-large-unsloth-qwen38-27b-q8xl]] — RETIRED 2026-09-28 (replaced by
  [[frank-large-gufo-qwen38-27b]]; rollback target): stock llama.cpp
  (unsloth b11030, ROCm gfx1151) serving
@@ -190,12 +267,29 @@ frank-small, port 7771]
  .hgn + quality overlay) on the gfx1151 iGPU — two 200k conversations over one
  400,384-position pool, deployed flags, memory model, client contract,
  troubleshooting (2026-09-16 → 09-19)
- [[frank-small-gemma4-12b]] — ACTIVE frank-small runbook since 2026-09-17
-  (root systemd unit, port 7771): mainline llama.cpp serving
+- [[frank-small-llama-mimo-q6k]] — **RETIRED 2026-10-01 → 10-02** (the same
+  rung returned ACTIVE 2026-10-03 at the 2 x 200,704 f16-KV shape —
+  [[frank-small-llama-mimo-q6k-2x200704]]): llama.cpp (master, CUDA sm_86
+  build) serving
+  bartowski/MiMo-V2.6-Distill-Qwen-9B-GGUF **Q6_K** on the RTX 3090 — TWO
+  native 262,144-token sessions with ASYMMETRIC KV (K f16 unquantized + V
+  q8_0, which forces flash-attn), ~20.9/24.6 GiB measured, text-only (the
+  GGUF has no vision), reasoning_content, no spec decode; payload still at
+  `/home/dank/models/MiMo-Q6K/`
+- [[frank-small-sglang-mimo-v26]] — RETIRED 2026-10-01 (hours; replaced
+  by [[frank-small-llama-mimo-q6k]] for the asymmetric KV SGLang cannot
+  express; venv still on disk, INT8 payload staged to
+  `~dank/.AGENTTRASH/20261002-retired-llm-payloads/`): SGLang in a
+  self-contained python3.12 venv serving leteschlead MiMo-V2.6 INT8
+  W8A16 AutoRound, 2 x 262144 ctx, fp8 KV, in-checkpoint vision,
+  83.9 tok/s decode
+- [[frank-small-gemma4-12b]] — RETIRED 2026-09-17 → 2026-10-01 (replaced by
+  [[frank-small-sglang-mimo-v26]] → the llama.cpp MiMo shape; payload staged
+  to `~dank/.AGENTTRASH/20261002-retired-llm-payloads/Gemma4-12B-QAT/`):
+  mainline llama.cpp serving
  HauhauCS/Gemma4-12B-QAT-Uncensored-HauhauCS-Balanced Q4_K_M on the RTX
  3090, with the separate MTP assistant sidecar chained behind ngram-mod,
-  q8_0 KV, and `--fit` sizing the full 262k window into ~12 GiB. A same-day
-  vLLM detour (2026-09-19) was reverted the same day; see below.
+  q8_0 KV, and `--fit` sizing the full 262k window into ~12 GiB.
 - [[frank-small-vllm-gemma4]] — RETIRED 2026-09-19 (same-day revert,
  rollback target): vLLM serving google/gemma-4-12B-it-qat-w4a16-ct (w4a16
  Marlin) with the bf16 MTP assistant on the RTX 3090 — bf16 KV, 289,647-
@@ -0,0 +1,207 @@
+---
+type: reference
+title: vLLM Boot Forensics and Config Interrogation
+description: Capturing a vLLM container boot failure's real root cause from journald (the container log is already gone), classifying the failure, and interrogating a live endpoint's true config and client contracts.
+tags: [vllm, forensics, journalctl, podman, config, contracts]
+timestamp: 2026-10-05T00:00:00-07:00
+---
+
+# vLLM Boot Forensics and Config Interrogation
+
+For serve-script vLLM endpoints (frank-primary/qwen-fp8 on dankserver, any
+`podman rm -f` + restart service).
+
+## 1. Capture the true cause — the journal, not `podman logs`
+
+The serve script runs `podman rm -f <container>` at every start, so a dead
+container's log is GONE by the time you look; `no container with name or ID`
+means the journal is the only source. The engine's stdout survives in
+journald under the container tag:
+
+```bash
+journalctl --no-pager --since "20 min ago" | grep -a "<container>\[" | tail -40
+journalctl --no-pager --since "8 min ago" | grep -a "<container>\[" \
+  | grep -aiE "error|exception|assert|raise|invalid|failed" | grep -av APIServer | tail -12
+```
+
+Why naive greps fail:
+
+- Default format truncates lines at the display cut, and the fork's giant
+  single-line ERROR dumps are journald-SPLIT — the decisive sentence never
+  appears in short format.
+- `-o cat` strips the `svc[pid]:` prefix, so grepping the tag against `-o
+  cat` returns nothing — use the default format for the tag, `-o json` for
+  the content.
+- Time filters: relative (`--since "NN min ago"`) only — absolute timestamps
+  are local PDT while the engine logs UTC; never mix.
+
+**The JSON decode procedure** (proven 2026-10-01, caught
+`torch.OutOfMemoryError` exactly):
+
+1. Bound the window: last traceback lines + the `container died` podman
+   event.
+2. `journalctl --no-pager --since … --until … -o json | python3 -c …` on the
+   remote: handle `MESSAGE` possibly arriving as a **byte list**
+   (`bytes(int(x) for x in m).decode("utf-8","replace")`), strip ANSI
+   (`re.sub(r"\x1b\[[0-9;]*m", "", m)`), dedupe by first ~12 chars (one
+   logical traceback can arrive as 20+ entries).
+3. Reassemble the worker traceback: filter skeleton lines
+   (`File |…| raise |^+|~+|…<N> lines|…|re-`), keep content lines. The
+   decisive line is usually the last content line.
+4. **The diagnosis is the gap math: `ask − free = shortfall`.**
+
+**The wrapper trap:** `RuntimeError: Engine core initialization failed. See
+root cause above` is the WRAPPER — the real cause is the EngineCore traceback
+above it, ending in `RuntimeError: Worker failed with error '<cause>'`. Grep
+for `Worker failed with error`.
+
+**For probe boots / runner scripts:** capture live BEFORE teardown —
+`podman logs -f <ctr> > /home/dank/logs/<arm>.engine.log 2>&1 &` right after
+launch. A runner's `podman rm` destroys the only copy; two 2026-10-01
+failures had their root cause eaten this way, and an 8-line bootfail capture
+is insufficient.
+
+## 2. Failure classes
+
+| signature | class |
+|---|---|
+| `CUDA out of memory. Tried to allocate X GiB … Y GiB free` | capacity; the gap math names the next rung ([[RDNA4 MoE Expert Offload VRAM Budget]]) |
+| `Worker failed with error 'Constraints violated (L['query_start_loc'].size()[0])! … specialized it to a constant (N)'` | torch.compile/dynamo shape specialization (N = num_seqs+1-ish) — NOT memory |
+| `Deactivated successfully` with no traceback | container exited CLEAN (vLLM API server exits 0 after an EngineCore fatal); container already removed — re-read the journal, check `NRestarts` |
+| `no container with name or ID` | container rm'd; journal only |
+
+Boot-order reality: the plan computes fine at impossible budgets; the
+failure lands later in worker init, not at plan time. A plan line printing a
+huge arena does NOT mean it will boot.
+
+Bypasses this fork exposes for compile faults: `--enforce-eager` (skips
+torch.compile AND CUDA graphs — drops graphs, a real cost),
+`VLLM_DISABLE_COMPILE_CACHE=1`, `VLLM_USE_AOT_COMPILE=1`. A stale cache is
+usually NOT the cause — verify before "fixing" it (`~/.cache/vllm` may hold
+only `modelinfos`; each `podman run` gets a fresh container FS unless the
+cache dir is mounted).
+
+**Crash-loop signature (shape does not fit):** `Started … ~6.7 min later …
+Deactivated successfully … restart`. Diagnose with `systemctl show -p
+NRestarts` + per-attempt duration. `Restart=always` loops visibly;
+`Restart=on-failure` stays SILENTLY DEAD (`podman wait` returned 0, systemd
+saw a clean exit).
+
+## 3. Observation hygiene during recovery
+
+- A container being STOPPED still answers `podman logs`, so a watcher
+  grepping its log reports PRE-change values and looks like the fix failed.
+  Discriminate by container identity:
+  `podman ps --sort created -a --format '{{.ID}} {{.Status}} {{.CreatedAt}}'`
+  — require the container to post-date the arm start, then read only that ID.
+- A leftover probe script whose tail does `podman rm -f <prod-container>`
+  fires against a freshly DEPLOYED engine. Signature: a previously-healthy
+  unit exits 0 (`Started → Deactivated`, code 0) — an external rm, not a
+  crash. Suspect an external actor before debugging the engine; `pgrep -af`
+  every script you launched (and kill strays of `deploy-remote.sh` by
+  explicit PID — killing the dev-side deploy job does NOT kill its host-side
+  runner).
+
+## 4. Ops laws that bite during recovery
+
+- `Restart=always` does NOT fire after an explicit `systemctl stop` — a
+  deploy's stop leaves the unit down until something starts it. Check
+  `systemctl is-active` yourself.
+- `ExecStop=podman stop -t 120` requires `TimeoutStopSec` > 120; the 90 s
+  default kills the stop and logs `Failed with result 'timeout'`.
+- A `D`-state worker cannot be reaped by ANY signal (SIGTERM/KILL/`podman rm
+  -f` all fail). Check `ps -o pid,stat,etime,comm` before declaring a GPU
+  wedge — measured 2026-10-01: D → R on its own within ~3 min, then the
+  container stopped normally.
+- Never `pgrep -f`/`pkill -f` a pattern that appears in your own command
+  line — kill by explicit PID list.
+- `git stash -u` (not bare `git stash`) for deploy windows — untracked files
+  still trip the deployer's dirty-tree gate.
+- A production engine started by a transient/ladder unit is NOT
+  boot-persistent; pin the winning numbers in the repo and deploy the real
+  unit.
+
+## 5. Logging audit (what more can this container say?)
+
+Answer with evidence from the live image, not upstream docs:
+
+1. Enumerate the fork's own env registry:
+   `podman exec <ctr> python -c "import vllm.envs as e; print(chr(10).join(k for k in dir(e) if k.isupper() and any(s in k for s in ('LOG','DEBUG','TRACE','STATS','PROFILER','VERBOSE'))))"` — then read defaults/clamps in `vllm/envs.py` (e.g. `VLLM_LOG_STATS_INTERVAL` defaults 10.0 s, clamped ≥ 1.0; telemetry emits only while requests are in flight).
+2. **Find the REAL HIP/ROCm library paths before grepping for AMD env vars** —
+   containers often install ROCm into system dirs, not `/opt/rocm`
+   (`podman exec <ctr> sh -c 'find / -name "libamdhip64.so*"'`). A grep of
+   `/opt/rocm/lib` that finds nothing is a FALSE NEGATIVE, not an absence.
+   Then `strings <lib> | grep -E '^AMD_LOG_LEVEL$|^HSA_ENABLE_DEBUG$'`.
+3. Scan vendor kernel `.so`s for switches — and separate tuning vars from
+   logging vars (both share a prefix).
+4. **Measure the DEBUG payoff**: count `logger.debug` sites in the module of
+   interest. 0–10 sites ⇒ a per-module DEBUG bump buys almost nothing; global
+   `VLLM_LOGGING_LEVEL=DEBUG` additionally makes api_server log full
+   request/response bodies (incident window only).
+5. Check the deployed argv for `--disable-log-stats` / `--log-configs` to
+   know what is already suppressed.
+
+Report: lever / effect / cost table, split into permanent-safe (stats
+interval), incident-window (`AMD_LOG_LEVEL`, DEBUG), and does-not-exist
+(absent strings). Wire via env-gated `-e` pass-throughs so default-off
+preserves behavior.
+
+## 6. Interrogate a live endpoint's REAL config
+
+Flags and env diverge from intent constantly — rollbacks drop them silently.
+Read what the container is actually serving, then prove the client contract
+behaviorally:
+
+```bash
+# live argv (the truth)
+ssh host 'podman exec <ctr> bash -c "cat /proc/1/cmdline | tr \"\\0\" \" \"" | tr " " "\n" \
+  | grep -A1 -E "max-model-len|max-num-seqs|kv-cache|reasoning-parser|tool-call-parser|template-kwargs|utilization|speculative"'
+# container env knobs (fork tuning that never appears in argv)
+ssh host 'podman exec <ctr> bash -c "env | grep -E \"R4D_|CLAV_|VLLM_\" | sort"'
+# capacity contract as measured at boot
+ssh host 'podman logs --tail 3000 <ctr> 2>&1 | grep -aE "GPU KV cache size|Maximum concurrency|non-default args"'
+```
+
+`non-default args` lines print what the server overrode — the fast way to
+catch a flag that never landed. `GPU KV cache size: N tokens, Maximum
+concurrency for M tokens per request: X.XXx` is the capacity contract:
+N ≥ requests × M or the last client gets truncated.
+
+**Thinking contract:** a reasoning model on vLLM returns `reasoning` ONLY if
+the server runs a reasoning parser; without it the thinking lands INLINE in
+`content` with a dangling close-tag and a null `reasoning`. Verify with a
+temp-0 completion: pass = `reasoning` non-empty AND no `<\/think>` in
+`content`. Diagnose with `ReasoningParserManager.list_registered()`
+(import `vllm.reasoning`).
+
+**Tool contract:** `tool_choice: "auto"` on a server WITHOUT
+`--enable-auto-tool-choice` returns HTTP 400 — reads like a model fault, is a
+config gap. Probe with a real tool schema and check `tool_calls` +
+`finish_reason`.
+
+**Restoring a dropped contract from history:**
+
+```bash
+git log --all --oneline -S"reasoning-parser"
+git show <commit>:scripts/serve-<svc>.sh | grep -B3 -A3 "reasoning-parser"
+```
+
+Qwen3-family values that work on the tcclaviger fork: `--reasoning-parser
+qwen3`, `--enable-auto-tool-choice --tool-call-parser qwen3_xml`,
+`--default-chat-template-kwargs '{"enable_thinking": true,
+"preserve_thinking": true}'`. `preserve_thinking`/`enable_thinking` are
+chat-template variables, NOT engine flags: they belong in
+`--default-chat-template-kwargs` server-side or `chat_template_kwargs` in the
+request body; a bare top-level body field of that name is ignored. Check the
+payload's `tokenizer_config.json` carries the key before trusting it either
+way.
+
+`/health` 200 says the engine is up, not that the contract is on the flags
+you think — any client report of odd content (inline thinking, 400 on tools,
+clipping at depth) starts with sections 1–3 here before touching anything.
+
+## Related
+
+- [[RDNA4 MoE Expert Offload VRAM Budget]] — the plan arithmetic behind OOMs
+- [[dankserver-qwen38-fp8-expert-offload-tuning]] — deployed shape and history
+- [[frank-large-benchmark]] and [[frank-small-benchmark]] — probe/ladder hygiene on live hosts
@@ -0,0 +1,330 @@
+---
+type: reference
+title: vLLM Porting and Compile Triage
+description: Forward-porting a model architecture to a new vLLM major (additive-only discipline, gap enumeration, shims that survive worker spawn) and the torch.compile/dynamo break triage, including the EAGLE3/DFlash2 aux-hidden-state protocol port rules.
+tags: [vllm, porting, torch-compile, dynamo, eagle3, dflash, glm]
+timestamp: 2026-10-05T00:00:00-07:00
+---
+
+# vLLM Porting and Compile Triage
+
+## Part I — Forward-porting a model architecture
+
+### 1. Check whether a port already exists; size the work honestly
+
+Read what the target image already ships before assuming a port is needed: a
+backport's upstream PR may be merged in the target version, and the target may
+carry a native kernel path the older line lacks (e.g. `rfi_hip` for RDNA4
+native quantized MoE). Porting around a native path that was already there is
+the expensive mistake.
+
+An upstream backport ships two kinds of file: **additive** (the new model
+package, its config, its kernel modules — port these) and **whole-file
+replacements of vLLM core** (`model_runner.py`, `scheduler.py`,
+`kv_cache_utils.py`, platform files — these are version X's copies of files
+version Y restructured; copying them is a DOWNGRADE, not a port). Measure both
+(`find … | xargs wc -l`), and say the additive-only ratio out loud before
+starting — one real case: a 71,724-line overlay reduced to 10,385 additive,
+and the "missing" drafter was already registered in the target.
+
+### 2. Enumerate the gap in ONE pass, not one ImportError at a time
+
+Inventory the target tree first (`model_executor/layers/`,
+`v1/attention/backends/`, quantization backends, spec-decode drafters in
+`registry.py`, `third_party/flash_linear_attention/ops/`). Then walk the
+staged package with `ast`, resolving every `from vllm… import X` against the
+target, split into MODULE-missing vs NAME-missing.
+
+- **The double-prefix bug:** the dotted module already starts with `vllm.`,
+  so joining onto the package dir needs
+  `rel = mod.split(".", 1)[1].replace(".", os.sep)`.
+- **Static scans over-report:** bare annotations, TypeAlias and star
+  re-exports resolve at runtime. Do a runtime check
+  (`importlib.import_module` + `getattr`) before quoting numbers — one scan
+  said 151/163 resolved; runtime said 10/13.
+- **Classify:** redirects dominate (version moved a symbol to a submodule) —
+  a ~60-line rewriter splitting `from X import (a, b, c)` blocks handles
+  them; **delete a block that empties** (a bare `from X import (` with an
+  empty body is a SyntaxError) and gate the rewriter on `ast.parse` of every
+  touched file.
+- **Port absent symbols with AST extraction**, not copy-paste: locate the
+  `ClassDef`, check every base/decorator resolves in the target, append the
+  exact segment, re-`ast.parse` the target, roll back on failure, skip if
+  already defined.
+
+### 3. Silent-default traps and transformers gates
+
+- Check every `cfg_int(cfg, key, DEFAULT)` against the new checkpoint: if the
+  key was renamed, the DEFAULT silently survives and the model computes fluent
+  nonsense with no error. Diff every config key the old parser reads against
+  the new `config.json`. When the old port used a fallback, the CHECKPOINT's
+  value must win (measured: `num_experts_per_token` fallback 7 vs
+  checkpoint `num_experts_per_tok=8`).
+- `model type X but Transformers does not recognize this architecture` =
+  pinned transformers predates the model. Upgrading transformers inside a
+  pinned vLLM image breaks things hard to attribute — prefer registering a
+  genuine `PretrainedConfig` subclass:
+  `CONFIG_MAPPING.register("glm5_next", Glm5NextConfig)`, gated by a real
+  round-trip (`AutoConfig.from_pretrained(<ckpt>)` returns the right class).
+  If the target image's transformers is below vLLM's declared minimum,
+  upgrading inside the image beats a shim.
+- **Hardware-gate flags from the port's own docs**: search the README for the
+  arch name and "do not" — e.g. a native quant MoE backend that is
+  CUDA/CDNA-gated falls back to per-call quantize-dequantize emulation at
+  ~5x slower; pin those flags in the serve script with a comment citing the
+  line.
+- **Backend selection:** default to `auto`, then READ the chosen-backend
+  line. `TRITON_UNFUSED` rejects some activation keys; `TRITON_ATTN` rejects
+  MLA + sparse attention — both surface as an init rejection. **"Using
+  'EMULATION' backend" is a FAILED configuration, not a success** (weights
+  dequantized per call: memory + throughput cost).
+- `Loading model from scratch…` for many minutes usually means the port
+  SUCCEEDED and you are in weight placement — verify all layers constructed,
+  both TP workers alive, zero errors, before chasing memory. And if it does
+  not fit, **prove the band is empty**: sweep the knob and show no setting
+  works; a single OOM is not a conclusion. Keep `MemAvailable` guarded
+  (abort on low RAM; do not let the OOM killer take co-tenants).
+
+### 4. Shims that must survive worker spawn
+
+vLLM v1 workers are spawned processes — a parent-process monkeypatch is gone
+by weight-load time. In order of preference:
+
+1. **Patch the installed library's source file on disk** (reliable). Locate
+   the real install path at runtime:
+   `python -c "import transformers,os;print(os.path.dirname(transformers.__file__))"`.
+2. `.pth` in site-packages — works but runs at interpreter start and can
+   import a package mid-import (circular import through `accelerate`);
+   expect failure.
+3. `sitecustomize.py` — often silently DEAD: Debian ships its own at
+   `/usr/lib/python3.12/sitecustomize.py`. Verify it is actually imported
+   before debugging why it has no effect. (A `.pth` on `PYTHONPATH` is never
+   executed by `site.py`; `sitecustomize.py` is.)
+
+Always confirm with a FRESH process — never with the process that set the
+patch up (`is_moe`, `num_experts_per_token`, `mhc_num_residual_streams`).
+
+**Misleading OOM names on ROCm:** `torch.AcceleratorError: CUDA error: out of
+memory` from `pin_memory()` is usually the HOST `memlock` rlimit, not VRAM.
+Cannot be raised from inside a container or root's own shell; only PAM at
+login: `/etc/security/limits.d/*.conf` + a NEW session. Confirm by pinning
+2/8/16 GiB — if those succeed, the rlimit was a real bug but is not the
+failure you are chasing.
+
+### 5. Ship as an idempotent patch-at-start applier
+
+One `apply.sh` at container start, each step guarded by a grep-for-marker
+(re-entry safe), ending in a verification block that imports every installed
+symbol and prints PASS/FAIL per symbol — ~10 s instead of a multi-minute
+load, and it turns "I think it works" into evidence.
+
+---
+
+## Part II — torch.compile / dynamo bring-up triage
+
+### The five laws (measured on torch 2.11+rocm, vLLM 0.29-dev)
+
+1. **`suppress_errors` does NOT cover graph breaks** — fullgraph=True +
+   suppress_errors=True still raises `Unsupported`. Only `fullgraph=False`
+   tolerates a break.
+2. **A warm `lru_cache` is not trace-safe** — dynamo re-traces the wrapped
+   body to build its guard even on a hit ("Dynamo ignores the cache wrapper
+   and directly traces the wrapped function"). Resolve the value ONCE into a
+   plain module global and read that; a global read folds, a cached call does
+   not. Verified in-image: warm lru_cache-calling function still fails
+   fullgraph=True; a plain global read compiles.
+3. **AOT compile refuses breaks outright** ("Graph breaks are not supported
+   with aot compile") — set `VLLM_USE_AOT_COMPILE=0` for a break-tolerant
+   run.
+4. **Fatal vs tolerated breaks:** tolerated breaks get a resume function
+   (untraceable part runs eagerly, numerics unchanged). Fatal ones cannot:
+   `Unsupported: Read uninitialized cell` (a traced callee closes over a name
+   bound after its definition) and `Skip calling torch.compiler.disable()'d
+   function` under fullgraph=True. `torch.compiler.disable` is the documented
+   escape hatch for host-side/stateful work (ContextVar reads, lazy
+   allocation, workspace grab) and is a NO-OP when nothing compiles — safe to
+   ship in the stock path.
+5. **Trust the frame chain, not the line number** — dynamo INLINES user
+   callees; an inlined failure reports the enclosing function's `def` line.
+   "KeyError in forward at the def line" is really "a callee raised".
+   `TORCHDYNAMO_VERBOSE=1` adds nothing for a plain Python exception (only
+   for dynamo-internal errors).
+
+**The tell** that the trunk lacks compile support at all:
+
+```
+Cudagraph mode FULL_AND_PIECEWISE requires piecewise capture, but the loaded model
+provides neither a compiled submodule nor breakable CUDA graphs. Overriding to
+FULL_DECODE_ONLY.
+```
+
+Diff against the family that DOES compile and copy its
+`@support_torch_compile` decorator including `dynamic_arg_dims` and any
+`enable_if` hook; env-gate every change so the file can stay mounted for an A/B.
+
+### Procedure
+
+0. **Establish what configuration is ACTUALLY in effect**:
+   `podman exec <ctr> printenv VLLM_COMPILE_FULLGRAPH VLLM_USE_AOT_COMPILE …`.
+   A launcher that "sets" an env var in a branch may silently drop it — four
+   consecutive boots ran `fullgraph=True` while the operator believed
+   otherwise, which makes the reported failure MOVE as breaks are removed and
+   invites entirely wrong theories.
+1. **Boot with the break list, not the first error:** `TORCH_LOGS=graph_breaks`
+   + `VLLM_USE_AOT_COMPILE=0` + `VLLM_COMPILE_FULLGRAPH=0`. Re-verify each
+   candidate break under the flag-free config you actually intend — a break
+   that appears only under the logging flag is a diagnostic artifact.
+2. **Fix in priority order:** (a) host syncs in the traced region
+   (`.item()`/`.cpu()`/`.tolist()` — check reachability first, a dead fallback
+   branch is not your blocker); (b) capability probes doing I/O in `forward`
+   (`os.path.exists`, `.so`/`ctypes` lookups — resolve once into globals,
+   warm at import); (c) lazy stateful work (hoist, or `torch.compiler.disable`
+   a thin wrapper); (d) new kernel specializations — check the vendor ships
+   SOURCES (`nm -D`/`strings`): a binary-only templated `kernel<128|256|512>`
+   makes the 288 you need unreachable. Fix at the PATTERN level (one
+   launcher/decorator family breaking all 39 call sites), not site level.
+3. **KeyError on a bare module name** = dynamo's name→object map miss
+   (`OutputGraph.nn_modules`: registered `output_graph.py:1259`/`:1384`,
+   read back `:3000`, `_buffers/_parameters/_modules` stripped at `:1225`).
+   Ship a probe via `sitecustomize.py` that patches
+   `OutputGraph.example_value_from_input_node` and keeps weakrefs of live
+   graphs, printing `node.target`, key count, near-miss keys — separates a
+   cross-graph reference from a name-derivation mismatch.
+4. **Instrumenting a startup hook is a FIREARM — arm it once:**
+
+   ```python
+   if os.environ.get("PROBE_ARMED") == "1":
+       pass                     # an ancestor already armed it -> stay inert
+   else:
+       os.environ["PROBE_ARMED"] = "1"
+       ...install probes...
+   ```
+
+   A `sitecustomize.py` on `PYTHONPATH` imports in EVERY interpreter and
+   children inherit `PYTHONPATH`: an unguarded hook that imports the engine
+   self-replicates (observed: 19+ levels, 33 processes alive 18 min later).
+   Kill the chain by removing the container whose cgroup owns every
+   generation, then verify with a precise `/proc/<pid>/environ` count —
+   NEVER pattern-kill an interpreter name on a host running python in
+   production containers. Also chain the image's own sitecustomize so its
+   hook survives.
+5. **Cheap in-image classification (no boot):**
+   `g = torch.compile(fn, backend="eager", fullgraph=True)` inside the
+   container (with `--device /dev/kfd --device /dev/dri` — torch.compile
+   queries a device even on the eager backend). ~5 s per question instead of
+   ~7 min boots.
+6. **Package as real patch artifacts:** hand-written `@@` headers are not
+   diffs and `patch(1)` rejects them. Generate
+   `diff -u --label a/<path> --label b/<path> orig new > name.patch` with
+   `--- a/` / `+++ b/` paths so `-p1` works; verify by applying to a fresh
+   extraction of the pinned image and comparing sha256 (`sha256sum` must
+   equal the `b/` hash recorded in the header). Header carries: STATUS, the
+   measured evidence, base+patched sha256, how to apply, the NEXT blocker.
+
+### Cost model before investing
+
+**Profile first** — parse the rank0 chrome trace (`POST /start_profile`,
+device-activity rows) for the kernel table. Measured case: the long-suspected
+expert-gather was **1.05 %** of device time while **44 %** was unfused
+pointwise glue (708,500 `vectorized_elementwise_kernel` launches =
+1,428/token) — a months-old "structural" conclusion died in one afternoon.
+Speculative decoding does NOT help a shape whose per-token cost scales with
+the token count, and it hurts prefill. Decide the ceiling up front: if the
+fused target is ~13–15 tok/s, do not spend a night of boots chasing 35.
+
+---
+
+## Part III — EAGLE3 / DFlash2 / DSpark aux-hidden-state protocol port
+
+Goal: make a trunk satisfy `SupportsEagle3` so a DFlash2 drafter
+(`DFlash2DraftModel`, method `dflash`) boots. Built on the GLM-5.3-Flash port
+(tcclaviger fork, 2026-09-30; four failed boots produced the laws below).
+
+### The four laws
+
+1. **Patch the instance the runner calls.**
+   `set_eagle3_aux_hidden_state_layers(self.model)` (model_runner.py, right
+   after `model_loader.load_model`) runs on the LOADED instance — for a
+   multimodal model that is the `…ForConditionalGeneration` WRAPPER (the
+   APIServer log line `Resolved architecture: …` names it), NOT the text
+   model it holds as `self.language_model`. Patching only the inner
+   `…ForCausalLM` reproduces the identical `RuntimeError: Model does not
+   support EAGLE3 interface` forever. Patch BOTH wrappers.
+2. **The check is Protocol isinstance, NOT duck typing.**
+   `supports_eagle3(model)` = `isinstance(model, SupportsEagle3)`
+   (interfaces.py ~1728; caller `v1/worker/gpu/spec_decode/eagle/eagle3_utils.py:20`).
+   `SupportsEagle3` is `@runtime_checkable` over base `SupportsEagleBase`
+   (`:1486`, `:1659`); `typing._ProtocolMeta.__instancecheck__` does
+   `getattr_static` over ALL `__protocol_attrs__` — including BASE-protocol
+   members — and breaks on the FIRST miss. Full member set:
+   `supports_eagle3`, `has_own_lm_head`, `has_own_embed_tokens`,
+   `set_aux_hidden_state_layers`,
+   `get_eagle3_default_aux_hidden_state_layers`. **Class attributes count as
+   presence; a methods-only implementation FAILS** because the three attrs
+   are missing. Verified in-image with stock typing, no boot: flag-only →
+   False; flag + two base members → True.
+3. **MRO forbids inheriting the Protocol.** Appending `SupportsEagle3` to a
+   wrapper already carrying Protocol-derived bases dies at class creation:
+   `Cannot create a consistent method resolution order (MRO) for bases …` —
+   MRO simulation (stub Protocols + `type("C", bases, {})`) confirmed no
+   position works. This failure also PROVES the isinstance-semantics question
+   is moot: class creation precedes any check.
+4. **Write the two methods by hand** — without inheritance the Protocol
+   defaults do not exist. Text model: unwrap to the trunk
+   (`self.model.aux_hidden_state_layers = layers`); multimodal wrapper:
+   delegate to `self.language_model` (the pixtral.py:498 pattern).
+   `get_eagle3_default_aux_hidden_state_layers` returns `(2, L//2, L-3)`; the
+   drafter config's `target_layer_ids` wins when present (GLM DFlash2 ships
+   `[5, 14, 24, 33, 42]` of 45).
+
+The fix shape (on EACH wrapper class):
+
+```python
+supports_eagle3: ClassVar[Literal[True]] = True
+has_own_lm_head: bool = False          # False when embed/lm_head are delegated
+has_own_embed_tokens: bool = False
+def set_aux_hidden_state_layers(self, layers): ...
+def get_eagle3_default_aux_hidden_state_layers(self): ...
+```
+
+### Trunk mechanics
+
+The trunk (`Glm5NextModel`) must be `EagleModelMixin`; add
+`aux_hidden_state_layers: set[int]` (default empty) + `start_layer` handling.
+Capture aux states in the layer loop: the state ENTERING each requested
+layer, BEFORE its input norm (GLM's loop carries the single-stream state;
+deepseek_v2's equivalent is `hidden_states + residual`).
+`use_aux_hidden_state_outputs=True` is set in model_runner.py (~:287) for
+methods `dflash`, `dspark`, `extract_hidden_states`; the forward returns
+`(hidden_states, aux_hidden_states)` conditionally after `self.norm` (tuple
+consumed at `v1/worker/gpu/cudagraph_utils.py:621`).
+
+### Window discipline and verification
+
+- Scratch boots on dankserver MUST stop + runtime-mask production
+  `frank-primary` first — the expert-offload planner reads `MemAvailable` at
+  plan time, and beside-prod boots die at plan with
+  `ExpertPlanError: host holds N of M swappable experts`. After stop+mask,
+  loop until `MemAvailable ≥ 230 GiB` before launching (a stop against a
+  mid-load engine SIGKILLs at ~90 s and the freed ~170 GiB takes >1 min to
+  return — a new window started during restore inherits the false failure).
+- Never launch a window while the previous window's RESTORE is in flight
+  (`systemctl start` returns immediately; the engine loads ~6 min).
+- Save full probe logs BEFORE teardown (`podman logs <probe> > full.log`);
+  decision greps truncate lines at `…` and tracebacks die with the container.
+- Cheap verification, no boot: AST-parse the patched file for class bases +
+  member presence; MRO simulation with stub Protocols; in-image `typing`
+  replication of the isinstance walk (pure stdlib — importing vllm itself
+  crashes on GPU-less inspection). Boot check: the worker line `Using Eagle3
+  auxiliary layers from config: (…)` = gate cleared.
+- Honest expectations: a target-side DFlash2/EAGLE3 port on a
+  pointwise-glue-bound trunk (~44 % of kernel time) buys ~ZERO decode gain —
+  the verification pass costs ∝ its token count. Port it for capability, not
+  speed.
+
+## Related
+
+- [[vLLM Boot Forensics and Config Interrogation]] — capture and contracts
+- [[RDNA4 Container Serving]] — the serving stack being ported onto
+- [[dankserver-qwen38-fp8-expert-offload-tuning]] — the live deployment
+- [[halogen-flash-server]] — the retired w4b-era endpoint and its profiling notes
@@ -169,6 +169,18 @@ The LLM inference fleet on the two hosts. Full index:
  concurrency, ngram-fork-bug); the residual gap to ~60 tok/s is the
  fork's gather read path (6.5-9.5 GB/s effective vs 22.7 at the same
  slab size)
+- [[frank-primary-glm53-flash-w4a16]] — VERIFIED 2026-09-30: the ACTIVE
+  frank-primary runbook. The cutover to Intel/GLM-5.3-Flash-W4A16-AutoRound
+  (symmetric int4 g128) on the tcclaviger fork, and the reason the int4
+  AutoRound payload is the ONLY one that serves here (its `auto-round` quant
+  method routes to the W4A16 offload kernel; a compressed-tensors W4A16
+  checkpoint of the same model reaches TritonWNA16Experts, which has no
+  offload recipe). **Measured decode 9.9-10.0 tok/s / prefill 918 tok/s; the
+  35 tok/s target is NOT met** — the step is bounded by the fork's gather
+  read path (~8.2 GB/s vs a 22.74 GB/s link) with the cache at a 0.53-0.66
+  hit rate, and MTP4 was measured and rejected (zero decode gain, prefill
+  918 -> 255). Carries the fork's KV-cap bug, the fp8-KV/gfx12 trap, and the
+  `glm47` tool/reasoning parser wiring
 - [[frank-medium-radiance-mxfp4]] — RETIRED 2026-09-26:
  vllm-radiance as ONE tensor-parallel process (TP=2 over the R4D one-shot
  allreduce) serving just1moremodel/Qwen3.8-27B-Uncensored-MXFP4-awq (MXFP4
@@ -201,14 +213,41 @@ The LLM inference fleet on the two hosts. Full index:
  agentionai/Qwen3.8-Flash-Next-ROCmFP4-FAST-imatrix (87.06 GiB,
  Q4_0_ROCMFP4_FAST) + own MTP draft, build-qwen38 (retired 2026-08-30;
  rollback record)
- [[frank-large-gufo-qwen38-27b]] — the ACTIVE frank-large runbook since
-  2026-09-28 (root systemd unit, port 7770): gufo (gufo-org/gufo) v0.1.1,
-  the Strix Halo inference engine, built from source on Dankclaw and
-  serving unsloth/Qwen3.8-27B-GGUF:UD-Q8_K_XL (a 27B dense model) with the
-  z-lab DFlash2 Q4_K_M drafter on the gfx1151 iGPU — one 262,144-token
-  conversation, adaptive block policy, greedy, text-only, the two
-  from-source build workarounds, what changed in the client contract, the
-  measured prefill/decode trade against llama.cpp
+- [[frank-large-halogen27b-dflash2]] — the ACTIVE frank-large runbook since
+  2026-10-05 (root systemd unit + podman, port 7770): the digest-pinned
+  ghcr.io/peonist-ai/halogen image serving peonist-ai/halogen-qwen3.8-27b
+  (qwen3.8-27b-p1w4d-d2.hgn, 33.4 GiB — a .hgn that loads in no other
+  engine) with the DFlash2 drafter on ONE slot of 200,700 ctx (slots>1
+  disables speculation — engine law); TEXT-ONLY and censored base, both
+  accepted tradeoffs; no server-side sampler, so the thinking recipe rides
+  the wire
+- [[frank-large-rocmfpx-q6agent-uncensored-mtp]] — RETIRED 2026-10-03 → 10-05
+  (rollback target; payload, fork build and build script on disk): the pinned
+  kingjones30/ROCmFPX fork (scripts/build-rocmfpx.sh) serving kingjones777/
+  Qwen3.8-27B-Uncensored-ROCmFP4-STRIX-MTP-GGUF
+  **Qwen3.8-27B-Uncensored-Q6_0_ROCMFPX_AGENT** — ftype 114, 339 tensors of
+  GGML_TYPE_Q6_0_ROCMFPX + 159 type-103, a format no other loader on the
+  host reads — with the repo's Q4_0 MTP head drafting 4 tokens (27.6-28.1
+  t/s measured, 86.8 % acceptance; the ngram-mod chain measured and
+  rejected) and its BF16 projector, one session at the native 262,144 window
+- [[frank-large-rocmfpx-q8agent-mtp]] — RETIRED 2026-10-02 → 10-03 (rollback
+  target, aligned Q8 payload still on disk): the same fork serving the
+  ALIGNED Q8_0_ROCMFPX_AGENT at 17.0 t/s — the +123 % MTP A/B, the traps
+- [[frank-large-gufo-qwen38-27b]] — RETIRED 2026-10-01 → 10-02 (history;
+  the payload dirs were removed from disk 2026-10-03, `scripts/build-gufo.sh`
+  remains): gufo v0.1.1
+  built from source serving Qwen3.8-27B UD-Q4_K_XL + the z-lab DFlash2
+  drafter on the gfx1151 iGPU — the two from-source build workarounds, the
+  template-hash allowlist, the measured prefill/decode trade
+- [[frank-large-vllm-mimo-v26-bf16]] — RETIRED 2026-10-01 (same day;
+  replaced by the gufo rollback, then by
+  [[frank-large-rocmfpx-q8agent-mtp]]): the digest-pinned
+  kyuz0/vllm-therock-gfx1151 image (vLLM 0.30, TheRock ROCm 10) serving
+  XiaomiMiMo/MiMo-V2.6-Distill-Qwen-9B in FULL BF16 on the gfx1151 iGPU,
+  4 x 262,144 ctx, with CROSS-MODEL MTP spec decode from Qwen/Qwen3.5-9B —
+  the roof math (13.1 t/s base is ~94% of LPDDR5x BF16), the measured
+  MTP-vs-DFlash table (19.8 vs 17.4 t/s), the greedy-text numerics trap,
+  the `reasoning` field change, troubleshooting
 - [[frank-large-unsloth-qwen38-27b-q8xl]] — RETIRED 2026-09-28 (replaced by
  [[frank-large-gufo-qwen38-27b]]; rollback target): stock llama.cpp
  (unsloth b11030, ROCm gfx1151) serving
@@ -257,10 +296,31 @@ The LLM inference fleet on the two hosts. Full index:
 - [[ciru-activefpx-promptforge]] — the RETIRED v3 CIRU PromptForge runtime on Dankclaw (superseded 2026-08-24)
 - [[kingjones-strix-mtp]] — the kingjones Qwen3.8-27B ROCmFP4 STRIX MTP model on Dankclaw (active)
 - [[dealign-ornith-crack]] — the retired Ornith-1.5-35B-A3B CRACK model on Dankserver (retired 2026-09-02)
- [[frank-small-gemma4-12b]] — the ACTIVE frank-small runbook since
-  2026-09-17: mainline llama.cpp serving
+- [[frank-small-llama-mimo-q6k-2x200704]] — the ACTIVE frank-small runbook
+  since 2026-10-03: llama.cpp master (CUDA sm_86) serving
+  bartowski/MiMo-V2.6-Distill-Qwen-9B-GGUF **Q6_K** on Dankclaw's RTX 3090,
+  port 7771 — TWO conversations at 200,704 tokens each with fully
+  unquantized f16 KV and mmproj vision; 94.4 tok/s decode (the Q6_K rung
+  costs a measured −10 % vs Q5_K_M — weight-bandwidth), batch 4096 / ubatch
+  2048 (+4.6 % prefill, measured 2026-10-05), 2,279 MiB free, and
+  the ≥256 px image-probe trap (tiny images answer "video")
+- [[frank-small-llama-mimo-q5km]] — RETIRED 2026-10-02 → 10-03 (rollback
+  target, payload on disk; owns the measured boot-pair table): same runtime
+  and shape one rung lower — 104.8 tok/s, 3,571 MiB true free
+- [[frank-small-llama-mimo-q6k]] — RETIRED 2026-10-01 (same day; that same
+  Q6_K rung returned ACTIVE 2026-10-03 at the new shape, see
+  [[frank-small-llama-mimo-q6k-2x200704]]): the first llama.cpp + GGUF shape
+  for this endpoint, Q6_K at 2 x 262,144 / f16-K + q8_0-V, 20.9 GiB measured
+- [[frank-small-sglang-mimo-v26]] — RETIRED 2026-10-01 (rollback target):
+  SGLang in a self-contained python3.12 venv serving
+  letechlead/MiMo-V2.6-Distill-Qwen-9B-INT8-W8A16-AutoRound (qwen3_5
+  hybrid, INT8 W8A16, in-checkpoint vision, NO MTP weights shipped) on
+  Dankclaw's RTX 3090, port 7771 — fp8 KV, pinned 606,208-token pool,
+  2 x 262,144 slots, bf16 mamba states, eager prefill
+- [[frank-small-gemma4-12b]] — RETIRED 2026-10-01 (rollback target):
+  mainline llama.cpp serving
  HauhauCS/Gemma4-12B-QAT-Uncensored-HauhauCS-Balanced at Q4_K_M
-  (MTP assistant sidecar + ngram-mod, q8_0 KV, --fit → 262k) on
+  (MTP assistant sidecar + ngram-mod, q8_0 KV) on
  Dankclaw's RTX 3090, port 7771
 - [[frank-small-vllm-gemma4]] — RETIRED 2026-09-19 (same-day revert,
  rollback target): vLLM serving google/gemma-4-12B-it-qat-w4a16-ct (w4a16
@@ -319,6 +379,14 @@ The LLM inference fleet on the two hosts. Full index:
 - [[qwen3.8-spec-matrix]] — Qwen3.8-27B spec history (superseded by FastMTP)
 - [[FastFlowLM NPU Inference]] — the retired FastFlowLM NPU endpoint
 - [[DFlash DSpark Speculative Decoding]] — retired DFlash/DSpark drafting
+- [[GGUF Compatibility]] — which loaders accept which quant formats; the tensor-type histogram check
+- [[Llama Cpp Multi-Device Splits]] — layer vs tensor split across unlike devices; the -ot pipeline gate
+- [[Llama Cpp Serving Sizing]] — VRAM arithmetic from two measured boots, never a formula
+- [[PCIe Link and Atomics]] — link caps vs loaded rate; RCCL atomics routing on the old board
+- [[RDNA4 Container Serving]] — the serving stack recipe: mounts, ulimits, udev, verify gates
+- [[RDNA4 MoE Expert Offload VRAM Budget]] — arena sizing law for expert offload: free VRAM is estimator slack, not allocatable headroom
+- [[vLLM Boot Forensics and Config Interrogation]] — journal/endpoint forensics for boot failures; argv recovery from a live engine
+- [[vLLM Porting and Compile Triage]] — forward-porting vLLM majors; compile failure forensics
 - [[ling-3.0-flash-trial]] — the Ling-3.0-Flash trial runbook
 - [[ling-3.0-flash-vllm-plan]] — the Ling-3.0-Flash vLLM plan (retired)

@@ -1,5 +1,244 @@
 # Bundle Change Log

+## 2026-10-05
+- **e2e: the FrankBook-Pro leg scoped to its own trigger-gated Playwright
+  project.** The `11-admin` walk included the six `*arr`/qBittorrent admin
+  links that terminate on 192.168.0.112, so every deploy gate died there when
+  the Mac mini slept (Caddy 502 on the five vhosts, connection timeout on the
+  raw Sonarr link). Split: those links moved to
+  `tests/e2e/specs/19-frankbook.spec.ts` under a `frankbook` project;
+  `tests/e2e/run.sh` excludes it from the default project set, and
+  `scripts/deploy.sh` computes a trigger from the same BASE..HEAD range the
+  plan uses (the spec, `frankbook-pro/**`, `shared/homepage/services.json`,
+  the `dankclaw/Caddyfile` arr vhosts) and sets `E2E_INCLUDE_FRANKBOOK=1`.
+  Verified live with the host asleep: full suite 33/33 in 1.1 min, the admin
+  walk back to 5.4 s; `--project=frankbook --list` proves the selection.
+  Live pass of the frankbook walk itself is deferred to the next deploy with
+  an awake host.
+  - **Catch while verifying: the 17:47 Ornith-1.5-9B swap (`77decb5`) had
+    left `frank_small_unit.bats` and the deployer's payload gate stale** —
+    the seam still expected MiMo-Q6K names and `deploy-remote.sh` gated
+    `LLM_SMALL_MODEL_FILE`/`LLM_SMALL_MMPROJ` on files the unit no longer
+    loads (the anti-drift seam doing its job). Both repointed to the
+    Ornith-Q8 paths; full `./tests/run.sh` back to green (366 tests).
+- **frank-small: compute-buffer pair doubled to 4096/2048, and the prefill win
+  measured rather than assumed.** Frank asked for the batch and ubatch doubled.
+  The blocker was on record: the Q5_K_M predecessor documented **"ubatch 2048
+  failed the scratch boot"**, and this shape carries 616 MiB less free space
+  than that predecessor did.
+  - The A/B used the service itself as the bench (one card, one server — no
+    scratch port): the old rung was re-booted through a transient
+    `FRANK_SMALL_ENV_FILE` systemd drop-in, and it reproduced the 2026-10-03
+    boot **byte-identically** (21,276 used / 2,849 free), which is what makes
+    ubatch the only variable. Repo never saw a fake revert commit; scaffolding
+    staged to `~/.AGENTTRASH/20261005-frank-small-ubatch-ab/`.
+  - **At 8,928 prompt tokens: prefill 3,972/4,077 -> 4,207/4,212 t/s, +4.6 %,
+    for 616 MiB (free 2,849 -> 2,279).** Decode flat (91.9–93.0 vs 93.8–94.3 —
+    noise). The control that proves it is geometry and not drift: a 279-token
+    prompt sits below *both* micro-batches, so it cannot see the knob, and it
+    did not move (2,676 vs 2,658–2,811).
+  - **The predecessor's boot failure did not reproduce** — that was a shape
+    observation, not a law; this shape boots 2048 clean. And the fleet lesson
+    "**`--ubatch-size` has an optimum**" is now scoped to what it actually is:
+    true on the 2x R9700 layer split (peak 512, −10 % by 2048), false on the
+    CUDA 3090 where the curve climbs monotonically to 2048. AGENTS.md corrected
+    — the curve is backend-specific and must never be carried across backends.
+  - The seam stopped hard-coding a rung: `serve_argv` now carries `BATCH_SIZE`
+    and `UBATCH_SIZE` the way it already carried `CTX_SIZE`, and the pair test
+    asserts the argv follows the shipped env plus the `ubatch <= batch`
+    invariant. Proven live both ways — retuning the env to 8192/4096 keeps it
+    green, inverting to 2048/4096 turns it red. (The old assertions only ever
+    exercised the serve-script *defaults*, because the test env never carried
+    these two keys.)
+  - `4096/1536` is the documented middle rung; 2,279 MiB free sits at the bottom
+    edge of the 2–3 GiB floor. Deployed `e5893dc`, e2e gate PASSED (2 mini specs
+    flaky-then-passed once the sleeping FrankBook-Pro woke mid-run — WoL took a
+    second burst and ~90 s). Docs:
+    [[frank-small-llama-mimo-q6k-2x200704]] (*Compute-buffer geometry*).
+
+## 2026-10-03
+- **frank-primary: shape F then shape G — MNBT 2048, the CUDA-graph plan term
+  pinned to the measured capture, KV forced to 400,700 tokens, and UTIL 1.0
+  proven UNBOOTABLE.** Frank asked what the batch was with GPUs showing open
+  space; the audit found the arena is `budget - non-expert - KV reserve -
+  graphs - overhead` where the only deploy-reachable terms are the budget and
+  the graph estimate (registry KV price 9,190 B/token and the 5.25 GiB
+  overhead are image constants; the ~3.5 GiB of driver-free VRAM was
+  estimator slack, not allocatable).
+  - Shape F (c682c63): MNBT 1024 -> 2048 (arena-free — the registry prices
+    overhead flat up to its nbt 4096 anchor and the fitted 5.25 was fitted AT
+    4096), util 0.98 -> 0.982, graph term 0.55 -> 0.32 GiB (256 base +
+    12/graph vs the fork's 256 + 52/graph). Measured: arena 17.52 GiB /
+    7,653 slots, budget 31.29.
+  - Shape G v1 (0a4ff30) tried util 1.0: **both workers refused init before a
+    single weight loaded** — `ValueError: Free memory on device cuda:N
+    (31.29/31.86 GiB ...)`, NRestarts=2 crash loop, zero payload progress.
+    The CLI field is `Field(gt=0, le=1)`, but the worker demands free >
+    budget, so budget == total is a trap, not a knife-edge. Frank: ".982 is
+    the law, never 1.00" — now with a second proof.
+  - Shape G v2 (c67a2ae, deployed green incl. e2e 33/33): util back to
+    0.982, graph base to 0 (the shape F boot reported a NEGATIVE -0.06 GiB
+    capture delta, so the 256 MiB base was fiction; 0 + 12 MiB/graph still
+    prices a future 18-graph spec-decode shape at 0.21 GiB), KV_BYTES
+    3.15e9 -> 3.1665e9 = Frank's 400,700-token ask.
+  - MEASURED live: arena **17.77 GiB / 7,762 slots** (+240 over shape E),
+    KV **401,408 tokens / 2.00x concurrency** — the first shape holding two
+    full-length conversations with real spare; capture measured 0.31/0.10
+    GiB against the 0.07 charge (under-charged, boots and holds); driver
+    used 30.97 of 31.86 GiB -> **0.89 GiB real free per card**, held through
+    a live 4,096-token generation (~38-39 tok/s single-request wall-incl.,
+    +13% vs the 33.99 shape-E baseline; single sample, not a bench).
+  - Verification seam rewritten to assert the shape by its OUTPUTS (util at
+    the 0.982 law, graph charge below the analytic default and per-graph
+    positive, KV pool >= 400,700 tokens at the measured 7,892 B/token).
+  Runbook: `serverAppConfigs-docs/LLM/dankserver-qwen38-fp8-expert-offload-tuning.md`.
+
+## 2026-10-01
+- **frank-primary rolled BACK to Qwen3.8-Flash-Next-Uncensored-FP8 + MTP4;
+  GLM/DFlash2 declared failed and purged.** The DFlash2 drafter booted and
+  served (window 6j, 65,536 ctx, `glm53-flash` on :7773, both TP ranks
+  captured FULL dflash2 cuda graphs — five patches deep: EAGLE3 wrapper
+  interface v5, KV unifier MLA-multiple, kpool tailpin, glm5 KV pool with
+  draft+hidden groups, registry method-signal), but **acceptance pinned at
+  1.00-1.38 on every text class INCLUDING verbatim repetition** — a mis-wired
+  aux-hidden-state capture, not a weak drafter. At ~1.2 acceptance the
+  8-token verify tax put decode at 1.6-1.8 tok/s vs the 9.9-10.0 baseline
+  (5x slower). Frank's independently reported 18 tok/s / ~370 prefill did
+  not originate from this engine (APIServer access log: every completion
+  POST was host-local; his workstation appears once, a /v1/models GET).
+  Restored the verified 37.4-40.6 tok/s arm from the `2339fae` trio
+  (200k x 3 seqs, fp8 KV, prefetch off); GLM payload (170 GiB) + drafter
+  (2.2 GiB) + scratch staged to `/home/dank/.AGENTTRASH/20261001-glm-purge/`.
+  KV-cache patches 02/03/04/04b survive in `scripts/patches/` (generic
+  unifier/tailpin/grouping/registry fixes, apply-verified).
+
+## 2026-09-30
+- **frank-primary cut over to GLM-5.3-Flash W4A16 int4 — it serves, but NOT at
+  the 35 tok/s target, and the shortfall is measured rather than inferred.**
+  - Payload `Intel/GLM-5.3-Flash-W4A16-AutoRound` (AutoRound symmetric int4
+    group 128, `auto_round:auto_gptq` packing, ~170 GiB, 34 shards + 2 extras,
+    113,074 tensors). It is the only W4A16 checkpoint this fork can serve WITH
+    expert offload: its `quant_method: auto-round` is overridden to `inc`, which
+    routes the MoE to libr4d's grouped W4A16 GEMM (`R4dW4a16MoEMethod`). The
+    compressed-tensors W4A16 build of the same model (`dealignai/…-CYBERSECURITY-W4A16`,
+    182 GiB, fetched earlier) resolves to `TritonWNA16Experts`, which has NO
+    offload recipe — and 170 GiB of experts cannot fit 64 GiB of VRAM, so it
+    refused to serve at all. That checkpoint was purged with the other retired
+    payloads.
+  - Shape: tcclaviger RDNA4 vLLM fork (same digest-pinned image as the qwen
+    cutover), TP=2, 131,072 ctx x 2 seqs, bf16 KV, util 0.98, NBATCH 2048,
+    `--enable-expert-offload --expert-precise-memory`, prefix caching, vision +
+    `glm47` tool and reasoning parsers, MTP OFF, prefetch OFF.
+  - Measured on the deployed engine: **decode 9.9-10.0 tok/s, prefill 918 tok/s**
+    (warm, 2.6k prompt), KV 138,561 tokens; arena 14.51 GiB = 2,383 slots per
+    rank over 12,096 host-backed experts (73.64 GiB pinned per rank).
+  - **The wall — and a corrected one.** The first account of the shortfall
+    bounded the step with throughput arithmetic (a step moves 0.82 GiB of misses
+    in ~100 ms = "8.2 GB/s effective against a 22.74 GB/s link", needing a 0.89
+    hit rate for 35 tok/s). **A torch-profiler kernel-time window falsified it**
+    (496 decode tokens, rank0 trace parsed for device-activity time): 46.18 s of
+    kernel time = ~93 ms/token, of which **44 % is pointwise glue** —
+    `vectorized_elementwise_kernel` alone 16.07 s over **708,500 launches (1,428
+    per token)**, plus 3.11 s of `elementwise_kernel_manual_unroll` — against
+    **1.05 % for `r4d_moe_lru_gather_kernel`**. The MoE GEMM is 10 %, the split-K
+    GEMM 6.4 %, the router/top-k 8.9 %, the mHC kernels 6.5 %, and the inductor
+    fusions that DO happen 6 %. The real cause is that the fork compiles its own
+    Qwen trunk (`@support_torch_compile` in `models/qwen4_exp/amd/model.py`) and
+    **not** its GLM trunk, so every pointwise op is a separate eager kernel; the
+    engine says so at boot ("...provides neither a compiled submodule nor
+    breakable CUDA graphs. Overriding to FULL_DECODE_ONLY."). Growing the arena
+    13.08 -> 14.51 GiB still changed nothing, consistent with a 1 % gather.
+  - **Fork work staged (compile changelist, WIP — does not boot yet).**
+    `scripts/patches/glm53-compile-01-model-decorator.patch`,
+    `-02-trace-safe-probes.patch`, `-03-fullgraph-optout.patch`, each with its own
+    evidence header and base/patched sha256. Boots established, in order: (1)
+    `functools.lru_cache` does NOT make a probe trace-safe — dynamo re-traces the
+    wrapped body even when warm, so `r4d_mhc_supported -> r4d_mhc_enabled ->
+    available -> os.path.exists` breaks the graph (a plain module-global read
+    compiles; verified in-image); (2) `suppress_errors` does NOT cover graph
+    breaks, only `fullgraph=False` does; (3) the GLM trunk can never satisfy
+    `fullgraph` because its indexer metadata syncs to host
+    (`glm5next/amd/sparse_indexer.py:263,419`), and AOT compile refuses breaks
+    outright. With all three the trunk **traces** (`Dynamo bytecode transform
+    time: 1.08 s`) and then dies on `KeyError: 'in_proj_qkvbfg_a'` in the
+    mHC/KDA path — a latent bug the compile exposes. That is the remaining
+    blocker.
+  - **No new HIP kernels are possible** (the image ships `r4d.so` with no
+    sources), which binary-blocks the second-largest lever: the fused router is
+    gated `num_experts in {128, 256, 512}` and only
+    `r4d_moe_route_kernel<128|256|512>` instantiations exist, while GLM has 288
+    experts. `--kv-cache-dtype fp8` buys no capacity either (the plan's KV
+    reserve is dtype-blind), only removing gfx12's only MLA decode path.
+  - **DFlash2 for GLM is a target-side port, not a drafter port.**
+    `incoai/GLM-5.3-Flash-DFlash2` declares `architectures: ["DFlash2DraftModel"]`
+    and `model_type: "qwen3"` (5 dense layers, sliding attention, block 8,
+    `target_layer_ids [5, 14, 24, 33, 42]` of 45) and the fork **already maps it**
+    (`registry.py:636` -> `qwen3_dflash2.DFlash2Qwen3ForCausalLM`; spec method
+    `"dflash"`). What `glm5next` lacks is the target side: aux-hidden-state
+    capture (`set_aux_hidden_state_layers` / per-layer capture, as `deepseek_v2.py`
+    demonstrates). No speedup is budgeted: a verification pass pays pointwise cost
+    proportional to its token count.
+  - **DFlash2 tried, and the gap was exactly that one interface — now closed.**
+    Payload fetched and hash-verified (`/home/dank/models/GLM-5.3-Flash-DFlash2`,
+    2,342,169,800 B, sha256 `b038e1d9…` = the Hub's LFS oid). Booting with
+    `--speculative-config '{"method":"dflash","model":"/draft","num_speculative_tokens":8}'`
+    accepted the config and the drafter and died at worker init with
+    `RuntimeError: Model does not support EAGLE3 interface`. The port
+    (`scripts/patches/glm53-dflash-01-aux-hidden-states.patch`) took FIVE versions —
+    hand-written methods -> Protocol base (MRO kill at class creation) -> attrs on the
+    text model -> attrs + methods on the text model -> **v5: attrs + methods on BOTH
+    wrappers** (the runner checks the LOADED instance, which for GLM is the multimodal
+    `Glm5NextForConditionalGeneration` holding the text model as `self.language_model`;
+    `pixtral.py:498` is the reference delegation pattern). v5 CLEARS the gate: both TP
+    workers log `Using Eagle3 auxiliary layers from config: (6, 15, 25, 34, …)` and the
+    DFlash2 drafter resolves + loads. Getting past that point was then interrupted twice
+    by experiment-window RAM contention (see the window discipline below), so the
+    drafter-serving end state was still unproven when this entry was written.
+  - **Compile work: one confirmed blocker, and a launcher bug that misled four boots.**
+    With breaks genuinely tolerated (`fullgraph=False`, AOT off) the trunk traces and
+    dies on the dynamo name-map `KeyError: 'in_proj_qkvbfg_a'`. Four later boots
+    reported *different* fatal breaks — `Unsupported: Read uninitialized cell` at
+    `mla.py:208` (the fused-norm JIT kernel), then `Unsupported method call` in the
+    indexer's in-forward workspace acquisition
+    (`sparse_indexer.py:132 -> workspace.py:135 get_simultaneous / :155
+    _ensure_workspace_size`), then `Skip calling torch.compiler.disable()'d function`
+    — and the cause was **my own scratch launcher**: its compile branches were
+    restructured and the `-e VLLM_COMPILE_FULLGRAPH=0` line was dropped, so those
+    boots ran `fullgraph=True`, where *every* break is an error and the first one
+    wins. I first blamed `TORCH_LOGS=graph_breaks` for the difference; that was wrong
+    (the flag only logs). The moving reports are still useful — they are the break
+    list — and patch 04 demonstrably removes the first of them (the fused-norm cell),
+    patch 05 makes the workspace acquisition compiler-opaque.
+    `scripts/patches/glm53-compile-04-fused-qkv-rmsnorm-optout.patch` takes GLM off
+    the fused-norm kernel (`R4D_GLM_FUSE_QKV_RMSNORM=0`, a constructor flag glm5next
+    sets at `attention.py:583`).
+  - **Diagnostics built for this (reusable).** `TORCH_LOGS=graph_breaks` +
+    `VLLM_USE_AOT_COMPILE=0` names each break with its user stack (AOT compile
+    hard-refuses breaks outright); `/home/dank/dynamo-probe` on `PYTHONPATH` with
+    `VLLM_DYNAMO_DUMP_ATTRS=1` prints the dynamo name-map miss detail (node target,
+    near-miss keys, whether another live `OutputGraph` holds the key). Two gotchas
+    recorded: a `.pth` on `PYTHONPATH` is NOT executed (use `sitecustomize.py`,
+    chaining the image's apport hook), and the scratch launcher used to reset
+    `ENVARGS` (now `${ENVARGS:-}`), which silently dropped env vars and wasted a boot.
+    All four patch artifacts are now genuine `diff -u` patches, verified to apply to
+    the pinned image and to reproduce their recorded sha256.
+  - Measured and rejected: **MTP4** (the head loads, arch resolves to
+    `Glm5NextMTPModel`, acceptance is excellent at mean length 3.76-4.90 — but
+    decode was unchanged (9.6-11.2 vs 10.0) while prefill COLLAPSED 918 -> 255.
+    The profiler window above explains it: the per-token cost is dominated by
+    pointwise glue that scales with the token count, so verifying a draft batch
+    pays ~N x that glue for the same N tokens of output);
+    **fp8 KV** (disables the sparse MLA backend's rope-free Triton path, and
+    aiter's MLA decode op is gated on CDNA>2, so the MTP draft step demands an op
+    that cannot exist on gfx12); **NBATCH 8192** (boots, identical prefill).
+  - Fork bug found: the plan's KV reserve under-models this model's
+    per-sequence KDA state and the MTP draft's KV, and `gpu_worker` caps KV at
+    that reserve, making a legal `--max-model-len` unbootable. Shipped
+    `scripts/patches/gpu_worker-kv-cap.patch` (lets an explicit operator KV
+    request raise the cap) — NOT needed by the deployed shape.
+  - Rollback: commit `2339fae` (qwen FP8 + MTP4); the qwen payload and the R9V
+    state dirs are intact on disk.
+

 ## 2026-09-29
 - **frank-large tuning investigation: a measured NEGATIVE result, nothing
@@ -516,3 +755,553 @@
 - `/dankclaw/pihole/sysctl/99-dnsmasq.conf` — Repo source of truth for `ip_unprivileged_port_start=53`; `deploy_pihole` stages conflicting drop-ins, comments out the stale sysctl.conf line, and applies `sysctl --system` (the boot code path).
 - `/tests/caddy_quadlet.bats` (new) + caddyfile/deployment_flow/pihole seams — quadlet boot persistence, self-healing, network, env, and sysctl coverage.
 - `/serverAppConfigs-docs/Infrastructure/Caddy.md`, `/serverAppConfigs-docs/Infrastructure/Pi-hole.md`, `/serverAppConfigs-docs/Dankclaw/Dankclaw Stack.md`, `/serverAppConfigs-docs/Dankclaw/Gitea.md`, `/serverAppConfigs-docs/Architecture/Deployment Flow.md` — quadlet runtime facts, the 2026-08-26 incident, the sysctl-conflict failure mode, canonical Caddy ownership.
+
+## 2026-10-01 — frank-primary RESTORED to fp8 3 x 262144 (bf16 boot fails)
+
+bf16 (--kv-cache-dtype auto) at 2 x 262144 did not stay up, with OR without
+--expert-precise-memory:
+
+- with precise: NRestarts 1->3, each attempt ~6.7 min then
+  `Deactivated successfully` (API server drains after an EngineCore fatal
+  error) — cause not captured, the serve script rm'd the container.
+- without precise (default 7.60 GiB overhead, arena 14.3 GiB): NRestarts
+  climbing, container `Exited (1)`, and the journal finally showed the real
+  error:
+  `RuntimeError: Worker failed with error 'Constraints violated
+   (L['query_start_loc'].size()[0])! You marked L['query_start_loc'].size()[0]
+   as dynamic but your code specialized it to be a constant (3)'`
+  — a torch.compile/dynamo dynamic-shape constraint violation, NOT an OOM.
+  `~/.cache/vllm` holds only `modelinfos` (16K) and each `podman run` gets a
+  fresh container filesystem, so a stale compile cache is ruled out: the
+  conflict is produced by this configuration's own compile path.
+
+Differences from the last shape measured serving (fp8 3 x 262144, MNBT 2048,
+KV 6.1e9 -> 792,069 tokens / 3.02x / arena 6,248 / 3.59 GiB free): KV dtype
+(fp8 -> bf16) and SEQS (3 -> 2). Both are candidates for the dynamo trigger;
+`--enforce-eager` is the bypass that would sidestep the constraint path.
+
+Restored: fp8 3 x 262144, MNBT 2048, KV_BYTES 6,100,000,000, no precise flag.
+Next: bisect bf16 vs SEQS in a window (bf16 + SEQS=3 isolates the dtype).
+
+## 2026-10-01 — frank-primary: --expert-precise-memory + bf16 = boot crash loop
+
+**Reverted.** The two changes Frank asked for were applied together and the
+result crash-looped: `NRestarts` climbed 1 -> 2 -> 3, each attempt running
+~6.7 minutes then logging `Deactivated successfully` (vLLM's API server drains
+and exits 0 after an EngineCore fatal error, so `podman wait` returns 0; the
+new `Restart=always` turned a silent death into a visible loop).
+
+The budget arithmetic explains it without needing the error text, because the
+arena is DEVICE-resident (the fp8 3-window shape's plan predicts 28.27 GiB and
+that is exactly what `rocm-smi` measured):
+
+| line | bf16 2 x 262144 + precise | fp8 3 x 262144 |
+|---|---|---|
+| non-expert weights | 6.28 | 6.28 |
+| pinned pool | 6.80 | 5.68 |
+| CUDA graphs | 0.55 | 0.55 |
+| arena | 16.65 (precise: overhead 5.25) | 14.31 (default: overhead 7.60) |
+| non-torch | ~1.45 | ~1.45 |
+| **total** | **31.73 of 31.86 GiB** | 28.27 (measured) |
+| free at boot | **~0.13 GiB** | 3.59 |
+
+`--expert-precise-memory` hands the 2.35 GiB compile/graph-capture headroom to
+the arena, and bf16 simultaneously adds 1.08 GiB of pinned pool; together they
+put graph capture ~0.13 GiB from the ceiling, where every transient allocation
+dies. So the flag is dropped and bf16 2 x 262144 ships with the default
+overhead: arena 14.31 GiB, ~2.4 GiB free at boot.
+
+**Lesson:** the registry's "compile headroom" is not slack to be reclaimed by
+default — it is the budget for the phase that allocates last. Reclaim it only
+by paying for it elsewhere (smaller pool, fewer sessions, or a shorter context).
+
+## 2026-10-01 — frank-primary: 2 bf16 windows + --expert-precise-memory
+
+Frank's final shape for the night. Two changes at once, both measured in
+principle before the boot:
+
+- **KV is bf16 (`--kv-cache-dtype auto`), two full-size windows (2 × 262144).**
+  bf16 measures **13,839 B/token** (from the bf16 3×241664 boot:
+  11,047,010,304 B -> 798,280 tokens), decomposing as **12,276 attention +
+  1,563 mamba/side caches** against fp8's **6,138 + 1,563 = 7,701**. So
+  524,288 tokens needs 7.256e9 B; pinned at 7,300,000,000 (~527,500 tokens,
+  3,206 margin). Cost: the pool goes 5.68 -> 6.76 GiB per rank.
+- **`--expert-precise-memory`.** The registry's default overhead for this
+  architecture is 7.60 GiB = the fitted 5.25 GiB (gfx1201 TP2
+  nbt4096/seqs16/len262144) + 2.35 GiB of compile/graph-capture headroom.
+  The flag selects the fitted figure, which moves only the plan's `room` term
+  — arena 6,248 -> ~7,278 slots (~1,025 more, at 2.344 MiB/slot) — at the
+  cost of boot-time capture slack. The overhead formula adds 0.22 GiB per
+  1024 nbt above 4096, so MNBT 2048 pays nothing.
+
+Budget note worth keeping: the arena is `(budget - non_expert_weights -
+kv_reserve - graphs - overhead) // expert_bytes`, where `kv_reserve` is the
+planner's OWN one-max-model-len reserve (2.48 GiB at TP2) and does NOT scale
+with SEQS. The pinned `--kv-cache-memory-bytes` therefore does not shrink the
+planned arena; it comes out of free VRAM instead.
+
+## 2026-10-01 — frank-large: Q4_K_XL payload + 4 x 163840 sessions
+
+Frank's call. Payload swapped from ajgazin's **abliterated** Swift-1.5
+UD-Q8_K_XL (29.3 GiB) to **unsloth/Qwen3.8-27B-GGUF UD-Q4_K_XL** (16.35 GiB,
+sha256 3f227079…b01e, vanilla/Apache-2.0 — the endpoint is no longer
+uncensored). New fetcher `scripts/download-frank-large-q4.sh`; the DFlash2
+drafter and its fetcher are unchanged.
+
+Shape: **`--sessions 4 --context 163840`** with unquantized KV. gufo has no
+KV-quantization lever at all (`execution_policy.hpp` pins `kv_cache_storage`
+to `kFp16`), so "unquantized" is inherent.
+
+KV arithmetic (why this fits where one 262144 session used to sit):
+f16 KV costs **64 KiB/token** — 16 full-attention layers (the hybrid is
+`full_attention_interval: 4` over 64 layers) x 4 KV heads x 256 head_dim x
+2 (K,V) x 2 bytes; the other 48 layers are linear-attention with a
+per-SEQUENCE fp32 state (~144 MiB), not a per-token cache. 4 x 163840 =
+655,360 tokens = 40.0 GiB of KV, and each session is exactly 10 GiB. Budget:
+weights 16.35 + KV 40.0 + drafter 1.07 + linear state 0.56 + runtime ~4 =
+~62 GiB against the 114 GiB GTT.
+
+## 2026-10-01 — e2e gate flakiness (12-crawl) — do not panic at the banner
+
+`specs/12-crawl.spec.ts:23` ("every user open link reaches its app") failed the
+deploy gate twice and then passed cleanly on a manual re-run:
+- run 1 (deploy): `Expected "plex.franksdank.cloud"` / `Received "app.plex.tv"`
+  — Plex redirects its web UI to the hosted app; the same failure appeared on
+  the previous deploy too.
+- run 2 (manual): `page.goto: net::ERR_ABORTED at https://www.franksdank.cloud/`
+  — the homepage answered `http=200` in 40 ms from the dev box at the same
+  moment, so this is a browser-level flake.
+- run 3 (manual): **33 passed**, exit 0.
+
+So a red gate on this spec alone is NOT proof the fleet is broken, and the
+deployer's `DEPLOY INCOMPLETE` banner can be raised by a flake. Re-run
+`tests/e2e/run.sh` before treating it as real; the Plex redirect is the one
+stable underlying issue and it lives in Plex's web-client/image config, not in
+any repo service.
+
+## 2026-10-01 — frank-primary bf16 no-spec cutover
+
+- `/scripts/frank-primary.env` — bf16 no-spec cutover, FINAL shape after the
+  night ladder: MAXLEN **241664** x SEQS 3, KV_BYTES **10,184,000,000**/rank,
+  KV_DTYPE auto (bf16), MNBT 1024, GATHER CHUNKS=32/LANES=64, UTIL 0.98;
+  FRANK_PRIMARY_SPEC deleted (no spec). Engine measured at the winner:
+  "KV cache size: 732,359 tokens, Maximum concurrency for 241,664 tokens per
+  request: 3.03x".
+- **The night ladder (Frank's method).** 262144 OOMed at CUDA graph capture
+  (`Tried to allocate 76.00 MiB ... 0 bytes free`, 30.43 GiB of 31.86 GiB
+  allocated by PyTorch) and 245760 OOMed identically; **237568 PASSED**
+  (KV 718,606 tokens) and the +4096 bonus step **241664 PASSED and is
+  PRODUCTION**. 241,664 is max possible on this box at util 0.98 — the
+  model's native 262,144 does not fit with a 3-seq contract. Every rung was
+  booted under a transient systemd unit at uid dank with
+  `LimitMEMLOCK=infinity` (the rootless bench probe masked the production
+  memory environment) with full engine-log capture.
+- **Util stays 0.98.** The earlier 0.90 edit was wrong — lowering
+  `--gpu-memory-utilization` shrinks the engine's own budget rather than
+  buying slack; Frank forbade it and the winning ladder ran at 0.98.
+- `/scripts/serve-frank-primary.sh` — `--speculative-config` line removed;
+  `--max-num-batched-tokens` and the `-e R4D_LRU_GATHER_CHUNKS/LANES`
+  container-env passthrough added; shape header rewritten.
+- `/tests/frank_primary_unit.bats` — seam re-pinned: bf16-KV-present,
+  spec-absent-by-measurement, gather-32x64-present, MNBT-1024 tests; the
+  qualified-shape and flags tests flipped to the new env keys.
+- Overnight redo campaign (prefetch OFF everywhere) tabled in
+  `/serverAppConfigs-docs/LLM/dankserver-qwen38-fp8-expert-offload-tuning.md`
+  with the 2026-10-01 cutover section: gather 32x64 +18.7%, MNBT flat
+  1024-4096, KV pool law (7,926 fp8 / 14,047 bf16 B/token), arena flat zone,
+  six-slot refusal, bf16 3x262144 boot failure root cause (CUDA OOM at
+  KV_BYTES 11.3e9; boots clean at 11.047e9 — captured from journald `-o json`).
+- `/AGENTS.md` — fleet table frank-primary row rewritten to the bf16 shape.
+- `/serverAppConfigs-docs/LLM/index.md` — fleet intro + mermaid node
+  updated out of the GLM era.
+- **Client contract fixed (thinking + tools).** The live endpoint returned a
+  null `reasoning` field and dumped the thinking text INLINE in `content`
+  with a dangling `</think>` (measured: `'We need answer…Brief.\n</think>\n\n391'`)
+  — the malformed thinking block clients choke on — because the deployed argv
+  had no `--reasoning-parser`. The GLM rollback (`d91379d`) had dropped all
+  three contract flags and the qwen rollback inherited the gap. Restored from
+  `ecaad36`'s values, confirmed against this fork's registries:
+  `--reasoning-parser qwen3`, `--enable-auto-tool-choice
+  --tool-call-parser qwen3_xml`, `--default-chat-template-kwargs
+  '{"enable_thinking": true, "preserve_thinking": true}'`. The seam's
+  "NO tool dialect" test is flipped to a PRESENT test.
+- **Final shape (Frank's call): 5 seqs x the full native 262144 window at
+  fp8 KV.** 6 x 241664 at fp8 died (`Tried to allocate 10.70 GiB`, 9.18 GiB
+  free) — six full-length contexts do not fit at any KV dtype — so the
+  contract moved to five FULL windows: 5 x 262144 fp8 PASSED on the first
+  rung, KV_BYTES 10,388,766,720/rank, measured **1,350,182 tokens /
+  5.15x concurrency** (39,462 tokens of margin). Seam re-pinned; the bf16
+  three-context shape (241,664, KV 732,359 / 3.03x, measured earlier the
+  same night) is the documented max at 3 contexts.
+- Harness `/home/blank/.omp/agent/models.yml` — contextWindow back to 262144
+  with the five-window fp8 shape (compaction gap 183,501 + 36,864 = 220,365
+  <= 262,144) and
+  `preserve_thinking` moved from a bare top-level body field to
+  `chat_template_kwargs` (the form vLLM actually reads).
+- **Runtime-survivability correction: five slots -> FOUR.** 5 x 262144 fp8
+  passed every boot gate (KV 1,350,182 tokens / 5.15x) but left 0 bytes free
+  at runtime; a real client request (995-token prompt, `max_tokens=261149`,
+  sampling params straight from models.yml) killed Worker_TP1 with
+  `torch.OutOfMemoryError: Tried to allocate 50.00 MiB ... 0 bytes free` and
+  the engine shut down. Boot is not proof of runtime survivability — after a
+  knife-edge shape change, send a large-`max_tokens` request before calling it
+  done. FOUR slots x 262144 fp8 (KV_BYTES 8,311,013,376/rank) returns ~1.93
+  GiB per rank for activations/graph capture. Frank's call 2026-10-01.
+- `/scripts/frank-primary.service` — **Restart=always, not on-failure** (plus
+  `StartLimitIntervalSec=0` and `OOMScoreAdjust=-500`). A runtime worker fault
+  makes vLLM's API server exit CLEAN, so ExecStart returned 0, systemd logged
+  "Deactivated successfully", and on-failure NEVER fired: the endpoint stayed
+  down until a human noticed. An explicit `systemctl stop` still wins, so
+  bench windows and deploys are unaffected.
+- **Final shape (Frank's call, 2026-10-01): 3 x 262144, pool at the FLOOR,
+  MNBT 2048, rest to the arena.** MAXLEN 262144 x SEQS 3 = 786,432 tokens of
+  contract; measured fp8 costs 7,698 B/token, so 6.054e9 B is the pool floor
+  and the deployed 6,100,000,000 B leaves a ~5.9k-token margin while handing
+  the remaining ~2.2 GiB (versus the 4-slot shape) to the expert arena — the
+  decode-critical cache the fork sizes from the remainder. MNBT 2048 costs the
+  arena nothing (the sweep measured 6,499 slots at every rung <= 4096).
+- **DCP investigated and closed** (see the parallelism survey in the runbook):
+  `--decode-context-parallel-size 2` is rejected at boot — DCP for GQA requires
+  TP > the model's KV heads (2), i.e. TP >= 4, i.e. four GPUs. Not a tuning
+  knob on this box.
+- Production bring-up: the shape boots and serves on :7770 — engine log
+  verified (KV cache size, concurrency, MNBT 1024, gather 32x64) plus live
+  contract probes (reasoning split, tool_calls). Rollback target for the
+  2026-09-29 fp8+MTP4 shape: git tag `frank-primary-fp8-mtp-20261001`.
+
+## 2026-10-01 — frank-large cutover: gufo -> vLLM TheRock, MiMo-V2.6 BF16 + cross-model MTP
+
+- **Question that started it:** can a Qwen3.5 MTP drafter attach to
+  frank-small, and how does vLLM run the FULL BF16 MiMo checkpoint? Answer 1:
+  not on frank-small — SGLang reads the MTP head from the TARGET checkpoint
+  and MiMo ships ZERO `mtp.*` tensors (base and INT8 alike). Answer 2: the
+  digest-pinned `kyuz0/vllm-therock-gfx1151` image (vLLM 0.30 / torch 2.11 /
+  TheRock ROCm 10) boots BF16 MiMo on the gfx1151 iGPU in ~200-240 s, and
+  vLLM reads the draft from a SEPARATE path — so cross-model MTP from
+  `Qwen/Qwen3.5-9B` (the distill's source, identical 4096/248320/32-layer
+  geometry) works. Frank's call: ship that as frank-large.
+- **Bench window (frank-large stopped, one session, temp 0, 28,647-token
+  prefill / 512 decode):** base 17,876 t/s prefill / 13.1 t/s decode (13.1
+  is ~94% of the LPDDR5x BF16 roof - decode here is bandwidth, not a bug);
+  +DFlash-8 7,976 / 17.4 (acceptance 59%/26% per position); +MTP-2 7,380 /
+  19.8 (71%/44%, token rate 57.4%). Deployed MTP-2: +51% decode. Both spec
+  arms pay ~55-59% of prefill. `num_speculative_tokens` beyond 2 unmeasured.
+- **Measured numerics law:** spec arms do NOT reproduce the no-spec greedy
+  text (bf16 batched verification reorders reductions; near-tie argmaxes
+  cascade). vLLM spec decode is distribution-preserving, not
+  bitwise-greedy-preserving.
+- **Client-contract delta wired:** vLLM emits `reasoning`, not
+  `reasoning_content` - `models.yml` frank-large block now sets
+  `reasoningContentField: "reasoning"`, and the extraBody moved to the
+  Qwen3.5-family recipe (0.7/0.8) with `chat_template_kwargs`
+  (a top-level `preserve_thinking` is ignored by vLLM).
+- Deployer gap found pre-deploy: the spec-draft gate demanded a single
+  `model.safetensors` (the dflash layout) and would have rejected the
+  deployed MTP arm (4 shards behind an index); fixed to accept both
+  layouts, gated by bats. `z-lab/Qwen3.5-9B-DFlash`'s "vLLM PR #40898
+  unmerged" card claim is FALSE - `vllm/v1/spec_decode/dflash.py` ships in
+  0.30 (verified in-image); the drafter stays on disk as the alternative.
+- Host state: +38 GiB fetched (MiMo BF16 17.53 GiB, Qwen3.5-9B 18 GiB,
+  DFlash 2.58 GiB), disk down to ~26 GiB free - reclaim pass due. gufo
+  rollback pinned at git tag `frank-large-gufo-q4kxl-20261001`; engine,
+  Q4_K_XL payload and DFlash2 drafter all remain on the host.
+- First deploy attempt #2 crash-looped the container 51 times: the serve
+  script handed vLLM the HOST model path while the tree is mounted at
+  `/models` inside the container — transformers rejected it as a malformed
+  repo id. Fixed by converting to the container path (the spec draft was
+  already converted; the target was not), pinned by bats. Also fixed the
+  deployer's own draft gate (same single-file bug as the serve script's).
+- Reclaim (Frank approved): Swift-1.5 Q8_K_XL payload (30 G, refetchable via
+  `scripts/download-frank-large-swift.sh`) and the orphaned flux-2-klein
+  tree under `models/checkpoints` (23 G; ComfyUI mounts
+  `comfyui-data/models`, not this) purged via
+  `~/.AGENTTRASH/20261001-frank-large-reclaim/` — 52 G freed. KEPT the
+  1.1 G `Qwen3.8-27B-DFlash2-Q4_K_M.gguf` (it is the gufo rollback's
+  drafter, not a remnant).
+
+## 2026-10-02 — frank-large cutover: vLLM MiMo -> ROCmFPX Q8_0_ROCMFPX_AGENT + MTP
+
+Frank's order was the highest quality of
+`kingjones777/Qwen3.8-27B-ROCmFP4-STRIX-MTP-GGUF` on one full context, "whatever
+runtime it requires". The answer is forced by the file: `Q8_0_ROCMFPX_AGENT.gguf`
+carries tensor type 103 (`GGML_TYPE_Q8_0_ROCMFPX`, ftype 115), which stock
+llama.cpp, gufo and every other loader on the fleet cannot read — only the
+kingjones30/ROCmFPX fork can, so the runtime is pinned to commit
+`36e9acd40e10a87cd3c3ef8ec734668757dc8520` and built by
+`scripts/build-rocmfpx.sh`. Shape: 1 session x 262,144 (native), f16 KV, the
+bundled BF16 mmproj, and MTP from the repo's own `mtp-Qwen3.8-27B-Q4_0.gguf`
+(n-max 4, `--spec-draft-device ROCm0`).
+
+- **MTP is the whole performance story: 17.0-17.1 t/s with, 7.60-7.66 without**
+  (+123 %), one session, temp 0, production stopped per arm, acceptance 0.462 /
+  mean length ~1.86, at an 8-15 % prefill tax (325-351 vs 381 t/s). The AGENT
+  routing variant was chosen over the card's plain Q8_0 because it keeps 187
+  output-head tensors at plain Q8_0 instead of 2, which measured MTP acceptance
+  0.953 vs 0.911.
+- Steady GTT occupancy 63.1 GiB of the ~114 GiB aperture; boot to healthy 16 s.
+- Rollback: git tag `frank-large-gufo-q4kxl-20261001`. The retired vLLM-era
+  fetchers moved to `old/`. Runbook
+  [[frank-large-rocmfpx-q8agent-mtp]].
+
+## 2026-10-02 — frank-small: Q5_K_M, then three contexts + vision, sized by two boots
+
+Three orders in one day on the same endpoint: `bartowski/…Q5_K_M` on llama.cpp
+with two contexts ("whatever context size fits the 3090"), then **three
+contexts** "as much as can fit while keeping overhead open space to 2-3gb on
+the 3090", then **"frank-small should have vision"**.
+
+- **Final shape: `CTX_SIZE=540672 PARALLEL=3` = 3 x 180,224**, f16 K + q8_0 V
+  under flash attention, `--mmproj` vision, all layers on the 3090. Measured
+  **21,684 MiB used / 2,441 MiB free** of 24,576 — inside the 2-3 GiB band.
+- **The pool came from two boots, not a formula.** 2 x 262,144 with no
+  projector: 20,190 MiB used, KV 12,800 => non-KV **7,390 MiB**. 3 x 196,608
+  with the projector: 22,972 used, KV 14,113 => non-KV **8,859 MiB**. The
+  difference prices the vision tower at **~1.4 GiB for a 918 MB file** — the
+  encoder's own buffers ride with it. The first 3-slot deploy (3 x 196,608)
+  booted healthy but left only 1,153 MiB free, which is how the band got
+  violated and why the seam now prices non-KV at the measured 9 GiB and
+  REJECTS that shape.
+- **Three contexts cost window, and that is the tradeoff Frank was told:**
+  180,224 tokens per conversation, not the native 262,144. A longer prompt is
+  a hard `400 exceed_context_size_error`, so `models.yml` sets
+  `contextWindow: 180224`. Consequence at the time: frank-primary's then
+  183,501-token compaction request did not fit here with summary room —
+  resolved later the same day when frank-primary's window moved to 200,704
+  and its trigger fell to 140,492 (see the shape-D entry below).
+- Measured (2026-10-02, this shape): decode **106.8 t/s**, prefill **3,814
+  t/s** (~5 % faster decode than 2 x 262,144 — shorter slot, shorter attention
+  scan), three concurrent requests answered in parallel in 1.7 s with a fourth
+  queued at 3.0 s, native `tool_calls`, `reasoning_content` populated,
+  `reasoning_effort: low` accepted, solid-red PNG answered **"Red."**
+- **Vision contract gotcha:** `/v1/chat/completions` ignores the legacy
+  top-level `images` field — that request answered "Brown." for the same red
+  image, i.e. it generated from text alone. Only the `image_url` content part
+  reaches the model. `/completion` accepts `images` but this build still
+  requires `messages`.
+- Deployer gates on the projector too (`LLM_SMALL_MMPROJ`), so a missing
+  mmproj cannot silently ship a blind endpoint; `MMPROJ_FILE=` (explicitly
+  empty) in the env file is the documented text-only opt-out and returns ~1.4
+  GiB to KV.
+- Reclaim staged, **not purged, awaiting approval**:
+  `~/.AGENTTRASH/20261002-retired-llm-payloads/` (~59 GiB of superseded
+  payloads). Kept in place for rollback: `MiMo-Q6K/`, `MiMo-Q5KM/`,
+  `Qwen3.8-27B-UD-Q4_K_XL/` and `Qwen3.8-27B-UD-Q8_K_XL/` (the latter holds the
+  gufo DFlash2 drafter). Runbook [[frank-small-llama-mimo-q5km]].
+
+## 2026-10-02 — frank-primary shape D: 3 x 200704, fp8, pool sized for 2 contexts
+
+Frank's order: shrink frank-primary to a 200704-token window with about two
+resident full-length contexts, cut the KV pool, and drop MNBT to 1024. The
+first attempt (`30904ba`, 2 x 200704 + SEQS=2) **crash-looped**; the revert
+went live as `e460bbb`, and the shape was then re-applied one law at a time on
+top of the proven-good base, root cause first.
+
+- **ROOT CAUSE: `--max-num-seqs 2` cannot compile this model. It was never OOM,
+  never MNBT, never the KV dtype.** Four single-variable boot probes against
+  the then-deployed shape (production stopped for each, candidate booted AS the
+  service):
+
+  | MAXLEN | SEQS | MNBT | result |
+  |---|---|---|---|
+  | 262144 | 2 | 1024 | **Engine core initialization failed** |
+  | 262144 | 2 | 2048 | **Engine core initialization failed** |
+  | 262144 | 3 | 1024 | healthy - 792,069 tokens, 3.02x, arena 6,248 |
+  | 262144 | 3 | 2048 | healthy - the then-deployed incumbent |
+
+  `torch.fx.experimental.symbolic_shapes.ConstraintViolationError: Constraints
+  violated (L['query_start_loc'].size()[0]) ... specialized it to be a constant
+  (3)`, raised from AOT `build_guards(strict_error=True)` during
+  `determine_available_memory -> profile_run -> _dummy_run`, ~7 min in.
+  `model_runner.py` sizes that input buffer `max_num_reqs + 1` and `_dummy_run`
+  sets `num_reqs = min(max_num_batched_tokens, max_num_reqs)`, so its width is
+  **SEQS + 1** for ANY `MNBT >= SEQS` - 3 at SEQS 2, 4 at SEQS 3, exactly the
+  constant in the message. MNBT only moves the compile RANGE ((1,1024) vs
+  (1,2048)), which is why the identical error appears at both rungs.
+  `--enforce-eager` bypasses the path but drops CUDA graphs. The 2026-10-01
+  bf16 2 x 262144 failure was this same error class; its message named "KV
+  dtype and SEQS" together and nobody bisected, so the dtype was blamed - it
+  was a red herring.
+- **THE ARENA LEVER IS `max_model_len`, NOT `--kv-cache-memory-bytes`.** The
+  planner sizes expert cache as `budget - non-expert - KV reserve - graphs -
+  overhead`, and the KV reserve is registry-priced at **9190 B/token x
+  max_model_len**, independent of the pool override:
+
+  | MAXLEN | KV reserve | arena |
+  |---|---|---|
+  | 262144 | 2.48 GiB | 14.30 GiB / **6,248 slots** |
+  | 200704 | 1.91 GiB | 14.87 GiB / **6,496 slots** |
+
+  d(reserve) 0.57 GiB = d(arena) 0.57 GiB = +248 slots. The 6.1e9 -> 3.15e9
+  pool cut (2.75 GiB/rank) bought **zero** arena slots. The previous session's
+  premise - "the pool cut becomes arena, ~7,500 slots" - was wrong; the pool
+  cut is a runtime-HEADROOM lever (the shape it replaced died with `0 bytes
+  free` on a real 995-token request).
+- **Final shape (option D):** `--max-model-len 200704` x 3 slots, fp8 KV,
+  `--kv-cache-memory-bytes 3,150,000,000`/rank, MNBT 1024, gather 32x64, no
+  speculative decode, prefetch off, no `--expert-precise-memory`. The argv
+  keeps the only slot count that compiles; the POOL caps resident
+  conversations. Deployed `c769ff2`, healthy in 558 s, e2e 33/33.
+- **Measured live, and one prediction corrected.** `expert cache space 14.87
+  GiB / 6,496 slots` - the arena prediction landed exactly. But `GPU KV cache
+  size: 398,658 tokens, Maximum concurrency for 200,704 tokens per request:
+  1.99x` (3,146,129,408 B reserved), not the 2.04x the arithmetic predicted:
+  **the per-token KV price is shape-dependent** - 7,701 B/token at MAXLEN
+  262144 (6,100,000,000 B -> 792,069 tokens) vs **7,892** at 200704, ~2.5 %
+  dearer. Never extrapolate a concurrency figure across geometries.
+- **Smoke-verified on the live endpoint:** three concurrent 67-token requests
+  answered in parallel (~4.1 s each) - the 3 slots are real; native `tool_calls`
+  (`get_weather {"city":"Seattle"}`); thinking in `reasoning` with clean
+  `content`; `reasoning_effort` accepts only {xhigh, medium, low} and answers
+  400 for anything else (re-measured; the fork's message names the set).
+- **Harness contract followed the window:** `models.yml` `contextWindow`
+  262144 -> **200704** (compaction trigger 140,492 + 36,864 = 177,356 <=
+  200,704). Side effect: frank-primary's compaction request now fits inside a
+  frank-small slot again, which it had stopped doing when frank-small went to
+  3 x 180,224.
+- **Comment-only follow-up** `295d578` recorded the measured numbers and was
+  deployed so the per-service revision marker sits at HEAD; e2e 33/33 again.
+- Rollback for the pre-shape-D fp8+MTP4 era stays git tag
+  `frank-primary-fp8-mtp-20261001`. Runbook
+  [[dankserver-qwen38-fp8-expert-offload-tuning]].
+
+## 2026-10-02 - plex: the readiness gate probed HTTP against a TLS-only service
+
+Found by accident, and it had been latent since the secure-connections posture
+was set: `deploy_plex` gated on `curl -fsS http://127.0.0.1:32400/identity`,
+which **cannot** succeed. PMS drops plain HTTP off-loopback - the probe's
+connection arrives on the container's eth0 - so Plex accepts the TCP connection
+and then answers with nothing, and curl reports "Empty reply from server" (52).
+
+- Trigger: a bare `./scripts/deploy.sh dankserver` (no `--service`) drains the
+  whole pending backlog, not just the service you were thinking about. plex's
+  per-service revision had been behind for long enough that this gate had never
+  run against the current posture. It burned its full 600 s and failed the run;
+  every service after plex in the order was left undeployed.
+- **Plex was healthy the entire time.** `https://127.0.0.1:32400/identity`
+  returned 200 with the MediaContainer XML in 44 ms while the gate was still
+  timing out. The in-container `http://127.0.0.1:32400/identity` also answers
+  200 - loopback is the one path PMS serves in the clear, which is exactly why
+  the host-side probe could not see the difference.
+- Evidence chain that settled it: `podman exec plex curl http://127.0.0.1:32400`
+  = 200; `curl http://127.0.0.1:32400` from the host = empty reply;
+  `/proc/net/tcp` inside the container showed `00000000:7E90` LISTEN plus stuck
+  ESTABLISHED pairs on `1703590A:7E90` (10.89.3.23, the container's own eth0
+  address) that never produced a response.
+- **The repo already knew this law, in a different file:**
+  `tests/caddyfile.bats` pins the plex vhosts to a TLS upstream with the comment
+  "PMS drops plain HTTP off-loopback". The deployer's gate never got the same
+  treatment. Fixed to `curl -fsSk --max-time 3 https://127.0.0.1:32400/identity`
+  (`-k`: PMS serves its own self-signed cert on 32400).
+- Re-deployed `39f0e0d`: plex recorded deployed, the *arr services recorded
+  (compose-recorded on this host, no restart attempted), tautulli deployed,
+  e2e 33/33. Verified after: `plex-https=200`, plex container up, tautulli
+  healthy, frank-primary untouched (`health=200`, NRestarts=0).
+- **Operating lesson: `deploy.sh <target>` with no `--service` is a fleet-wide
+  drain.** It restarts every service whose recorded revision is behind HEAD and
+  whose paths changed since. To touch one service, name it.
+
+## 2026-10-02 — frank-primary shape E: `--expert-precise-memory` (+1,026 arena slots), a mounted compile cache, and the P2P answer
+
+Shape D left one question open — the plan's overhead term was reserving 2.35 GiB
+of compile headroom the boot never used — and the answer to Frank's free-VRAM
+question arrived together with a boot-cost bug that had been costing 181 s on
+every restart since the cutover.
+
+- **`--expert-precise-memory` is real and it fits (option A).** Boot-probed on
+  the shape-D base with production stopped, the candidate booted AS the service
+  (`FRANK_PRIMARY_ENV_FILE=/tmp/probe-b.env bash scripts/serve-frank-primary.sh`,
+  trap-guaranteed restore), the numbers read off the engine's own plan lines:
+
+  | plan line | shape D | shape E |
+  |---|---|---|
+  | `runtime overhead (non-torch + buffers + activation)` | 7.60 GiB | **5.25 GiB** (`precise (gfx1201 TP2 nbt4096/seqs16/len262144)`) |
+  | `expert cache space` | 14.87 GiB | **17.22 GiB** |
+  | slots | 6,496 | **7,522** (+1,026, +16 %) |
+  | `swap region` | — | 13.7 GiB / 5,986 slots, 133-134 per layer |
+  | boot | — | **healthy in 420 s**, no OOM |
+
+  The flag also prints `'expert_precise_memory': True` in `non-default args`, so
+  a silent argparse drop cannot hide. It was refused on 2026-10-01 only because
+  the 2 x 262144 bf16 shape was already 0.13 GiB under the ceiling — there the
+  flag was the last stack over the line, not the cause.
+- **The pool was NOT cut (option A) because the pool is not an arena lever.**
+  A 2.2e9 arm was probed in the same window and reports `277,686 tokens,
+  1.38x` — measured, rejected as the shipped value, kept as the conservative
+  fallback. With 3.15e9/rank the endpoint keeps shape D's 1.99x concurrency and
+  gains the arena, which is the shape D follow-up had actually been looking for.
+- **THE COMPILE CACHE WAS BEING THROWN AWAY ON EVERY RESTART — 181 s each.**
+  Measured: a fresh container has **no `/root/.cache` at all** (`No such file or
+  directory`); after our boot it holds `/root/.cache/vllm` at
+  **1,154,714,333 B in 4,665 files** plus `/root/.cache/comgr` at 229 MB, inside
+  the ephemeral layer, and `podman rm -f` (every deploy, restart, reboot)
+  discards them. The engine names the loss itself: `Compiling a graph for compile
+  range (1, 1024) takes 173.19 s` ... `torch.compile took 181.20 s in total` ...
+  `init engine (profile, create kv cache, warmup model) took 217.26 s
+  (compilation: 181.20 s)`. The serve script now mounts
+  `FRANK_PRIMARY_CACHE_DIR` at `/root/.cache` — the **parent**, so the vLLM
+  AOT/torch_compile tree (`VLLM_CACHE_ROOT`, default `/root/.cache/vllm`) and
+  the comgr JIT cache both persist, and nothing is hidden because the image has
+  no such directory. Same class of fix as the recipe's
+  `/cache/{triton,vllm,inductor}` volumes; the compile RANGE is part of the
+  cache key, so an MNBT change costs one recompilation, not a permanent tax.
+- **`GPU_MAX_HW_QUEUES=2`** (HSA hardware queues per GPU) added at Frank's
+  direction. Verified a real knob rather than a no-op — the literal is read from
+  `/opt/rocm/core-10.0/lib/libhsa-runtime64.so` — and shipped deliberately
+  alongside the 32x64 gather geometry. **Not A/B measured.**
+- **P2P: the hardware supports it, and the engine is already using it.** The
+  live log shows `r4d all-reduce INSTALLED (rank=0 max=49152KB finegrained=T…)`
+  on BOTH ranks and `Using ['R4D_AR', 'PYNCCL'] all-reduce backends` — R4D's
+  P2P path, with RCCL as fallback, measured at **45.274 GB/s bidirectional**.
+  Root ports `00:01.1`/`00:01.3` both report ACS `ReqRedir+`, so P2P hairpins at
+  the root complex; there is no XGMI. The old FAIL predates the 2026-09-20 board
+  swap. Kernel support confirmed: `CONFIG_HSA_AMD_P2P=y`, `CONFIG_PCI_P2PDMA=y`,
+  `CONFIG_HSA_AMD=y`, `CONFIG_ZONE_DEVICE=y`, `iommu=pt` on the cmdline.
+- **Recipe diff (Frank's 3 x R9700 compose vs this podman run), ranked:**
+  - *Acted on:* persistent compile caches (above — the one real operational
+    gap), and `GPU_MAX_HW_QUEUES`.
+  - *Retracted:* pinned `--compilation-config cudagraph_capture_sizes` was on my
+    list; measured, it is **not** a gap here — capture is 3 PIECEWISE + 2 FULL
+    graphs, `Graph capturing finished in 3 secs, took 0.31 GiB` (0.11 GiB on the
+    lighter rank). No 15-52-size ladder to cut.
+  - *Not acted on — measured candidates, none shipped (no bench window; these
+    are CPU-side and our decode is gather-bound, so a spin-wait stealing cores
+    from the gather threads is plausible but unproven):* `OMP_NUM_THREADS=8`
+    (defaults to all 16 logical CPUs on this 7700X), `HSA_ENABLE_INTERRUPT=1`,
+    `HSA_ENABLE_MWAITX=1`, `HSA_ENABLE_SDMA`, `HSA_NO_SCRATCH_RECLAIM`,
+    `HSA_ENABLE_IPC_MODE_LEGACY`. All six HSA names verified real — the literals
+    are read out of `/opt/rocm/core-10.0/lib/libhsa-runtime64.so` — so they are
+    knobs, not no-ops; any of them needs its own A/B before shipping.
+  - *Deliberate divergence, keep ours:* no MTP (the recipe has it; measured
+    fastest here), `qwen3_xml` vs `qwen3_coder` (ours verified live), no
+    `--override-generation-config` (server-side sampling defaults), published
+    port + Caddy rather than `network_mode: host`.
+  - *Non-gaps:* `shm_size 32gb` (ours is `--ipc=host`), `memlock -1` (same),
+    `SYS_PTRACE` (same), `/dev/kfd` + `/dev/dri` (same), `--enable-chunked-prefill`
+    (V1 default), `plecache` volume (the flag is not even passed), and
+    `ROCR_VISIBLE_DEVICES=1,2,3` vs our `HIP_VISIBLE_DEVICES=0,1` — verified
+    equivalent here: `rocm-smi --showid` puts the two R9700s at GPU[0]/GPU[1] and
+    the 7700X iGPU at GPU[2], so our indices are the right two cards.
+- **Harness note, because it cost two probe attempts:** a probe that reported an
+  instantaneous candidate death was a logging artifact of the inline SSH script,
+  not a crash — the container and its serve script must be polled by recorded id
+  and their logs dumped on the exit path, or a failure looks like a bad flag.
+  The flag was fine both times.
+- Live: `health=200`, `NRestarts=0`, arena 17.22 GiB / 7,522 slots, KV 398,658
+  tokens / 1.99x. Docs: [[dankserver-qwen38-fp8-expert-offload-tuning]].
+- **Deployed `1ac4d48`, e2e 33/33, and the live plan matched the probe exactly:**
+  `'expert_precise_memory': True`, `runtime overhead … 5.25 GiB`, `expert cache
+  space 17.22 GiB / 7522 slots`, `swap region 13.7 GiB / 5986 slots`, `GPU KV
+  cache size: 398,658 tokens … 1.99x`, pool 3,150,000,000, MNBT 1024. The
+  precise overhead and the arena growth are now production numbers, not a
+  probe's projection.
+- **The compile-cache fix was then proven on the same endpoint, two consecutive
+  boots of the deployed shape:** cold `Compiling a graph for compile range
+  (1, 1024) takes 179.24 s` → `torch.compile took 188.26 s`; warm `Directly load
+  the compiled graph(s) for compile range (1, 1024) from the cache, took
+  2.143 s` → `torch.compile took 3.97 s in total` → `init engine (profile,
+  create kv cache, warmup model) took 41.13 s (compilation: 3.98 s)`. The host
+  dir held 1.4 GB in 4,707 files after the cold boot. **Time to `/health` is
+  dominated by the 172.8 GiB payload load** (the warm boot still needed 490 s),
+  so the honest claim is ~184 s of inductor/AOT build recovered per restart —
+  not a proportional boot cut.
+
@@ -303,8 +303,8 @@
    "section": "AI",
    "details": {
      "role": "Fast utility model",
-      "spec": "Gemma4-12B QAT (Q4_K_M, 7.4 GiB) — 1M-token pool across four conversations; fastest at images.",
-      "inputs": ["text", "image", "audio"],
+      "spec": "MiMo-V2.6-Distill-Qwen-9B (INT8 W8A16, 12 GiB) on SGLang — 606k-token pool across four 151k-token conversations; 84 tok/s decode; sees images.",
+      "inputs": ["text", "image"],
      "use": "Agentic search, RAG, research subagents, translation, roleplay, and summarization — any small task that doesn't need much intelligence."
    },
    "url": "",
@@ -0,0 +1,150 @@
+#!/usr/bin/env bats
+# Seam: autoresearch measure client behavior.
+#
+# The measure client is the whole yardstick of the frank-primary speed
+# session, so its METRIC arithmetic gets a behavior test: against a
+# deterministic mock endpoint that emits a KNOWN number of completion
+# tokens at a KNOWN per-token delay, the reported decode rate must land
+# near the nominal rate, the usage accounting must come back exact, and an
+# unhealthy endpoint must fail loudly (no half-measured numbers).
+#
+# The mock implements only what the client calls: POST
+# /v1/messages/count_tokens (word count as the token count, which makes the
+# client's words/token calibration come out to exactly 1.0) and a streaming
+# POST /v1/chat/completions (one content chunk per completion token, then a
+# usage chunk, then [DONE]).
+
+load helpers
+
+MEASURE="$REPO_ROOT/scripts/autoresearch/frank-primary-measure.py"
+MOCK="$BATS_TEST_TMPDIR/mock.py"
+PORT=""
+MOCK_PID=""
+
+setup() {
+    # Deterministic mock; argv: <port> <delay-seconds>; env MOCK_FAIL=1
+    # makes the chat endpoint answer 500.
+    cat >"$MOCK" <<'PYEOF'
+import json, os, sys, time
+from http.server import BaseHTTPRequestHandler, HTTPServer
+
+PORT = int(sys.argv[1])
+DELAY = float(sys.argv[2])
+FAIL = os.environ.get("MOCK_FAIL", "0") == "1"
+
+
+class Handler(BaseHTTPRequestHandler):
+    def log_message(self, *a):
+        pass
+
+    def _send(self, obj, code=200):
+        b = json.dumps(obj).encode()
+        self.send_response(code)
+        self.send_header("Content-Type", "application/json")
+        self.send_header("Content-Length", str(len(b)))
+        self.end_headers()
+        self.wfile.write(b)
+
+    def do_POST(self):
+        n = int(self.headers.get("Content-Length", "0"))
+        body = json.loads(self.rfile.read(n) or b"{}")
+        if self.path == "/v1/messages/count_tokens":
+            text = body["messages"][0]["content"]
+            self._send({"input_tokens": max(1, len(text.split()))})
+        elif self.path == "/v1/chat/completions":
+            if FAIL:
+                self._send({"error": "engine down"}, 500)
+                return
+            max_tokens = int(body.get("max_tokens", 64))
+            self.send_response(200)
+            self.send_header("Content-Type", "text/event-stream")
+            self.end_headers()
+            for i in range(max_tokens):
+                ev = {"choices": [{"delta": {"content": "tok%d " % i}}]}
+                self.wfile.write(("data: %s\n\n" % json.dumps(ev)).encode())
+                self.wfile.flush()
+                time.sleep(DELAY)
+            prompt_words = max(1, len(body["messages"][0]["content"].split()))
+            ev = {"choices": [{"delta": {}}],
+                  "usage": {"prompt_tokens": prompt_words,
+                            "completion_tokens": max_tokens}}
+            self.wfile.write(("data: %s\n\n" % json.dumps(ev)).encode())
+            self.wfile.write(b"data: [DONE]\n\n")
+            self.wfile.flush()
+        else:
+            self._send({"error": "unknown path"}, 404)
+
+
+HTTPServer(("127.0.0.1", PORT), Handler).serve_forever()
+PYEOF
+}
+free_port() {
+    python3 -c 'import socket; s = socket.socket(); s.bind(("127.0.0.1", 0)); print(s.getsockname()[1]); s.close()'
+}
+
+start_mock() { # $1 = ok|fail
+    PORT="$(free_port)"
+    if [ "$1" = "fail" ]; then
+        MOCK_FAIL=1 python3 "$MOCK" "$PORT" 0.005 &
+    else
+        python3 "$MOCK" "$PORT" 0.005 &
+    fi
+    MOCK_PID=$!
+    local i
+    for i in $(seq 1 100); do
+        if python3 -c "import socket; socket.create_connection(('127.0.0.1', $PORT), 0.2).close()" 2>/dev/null; then
+            return 0
+        fi
+        sleep 0.1
+    done
+    echo "mock did not start" >&2
+    return 1
+}
+
+stop_mock() {
+    if [ -n "$MOCK_PID" ]; then
+        kill "$MOCK_PID" 2>/dev/null || true
+        wait "$MOCK_PID" 2>/dev/null || true
+        MOCK_PID=""
+    fi
+}
+
+@test "measure exists and compiles" {
+    [ -f "$MEASURE" ]
+    python3 -m py_compile "$MEASURE"
+}
+
+@test "measure computes decode rate and exact usage from a controlled endpoint" {
+    start_mock ok || return 1
+    run env AR_BASE="http://127.0.0.1:$PORT" AR_MODEL=mock python3 "$MEASURE"
+    stop_mock
+    [ "$status" -eq 0 ] || { echo "$output"; return 1; }
+    # run() merges stderr in; the contract is: exactly ONE json result line
+    # (the window script parses the same way), progress on stderr.
+    printf '%s\n' "$output" | python3 -c '
+import json, sys
+lines = [l for l in sys.stdin.read().splitlines() if l.startswith("{")]
+assert len(lines) == 1, "expected exactly one JSON line, got %d" % len(lines)
+d = json.loads(lines[0])
+# nominal 200 t/s (0.005 s/chunk) minus small per-chunk overhead
+for k in ("decode_tps", "decode_tps_115", "decode_tps_1056", "decode_tps_4096"):
+    v = d[k]
+    assert 60.0 <= v <= 320.0, "%s=%s outside [60, 320]" % (k, v)
+# the mock emits exactly max_tokens completion tokens; the accounting must be exact
+assert d["completion_tokens_4096"] == 256, d["completion_tokens_4096"]
+assert d["completion_tokens_115"] == 256
+# prompt sizing: calibration is exactly 1.0 words/token in the mock, so the
+# measured prompt lengths equal the targets
+assert d["prompt_tokens_115"] == 115
+assert d["prompt_tokens_4096"] == 4096
+print("measure claims verified")
+'
+}
+
+@test "measure fails loudly on an unhealthy endpoint instead of half-measuring" {
+    start_mock fail || return 1
+    run env AR_BASE="http://127.0.0.1:$PORT" AR_MODEL=mock python3 "$MEASURE"
+    stop_mock
+    [ "$status" -ne 0 ]
+    ! printf '%s\n' "$output" | grep -qE '^\{'
+}
@@ -45,5 +45,16 @@ export default defineConfig({
      testMatch: /(14|15)-.*\.spec\.ts/,
      use: { storageState: { cookies: [], origins: [] } },
    },
+    {
+      name: "frankbook",
+      // The *arr/qBittorrent walk (19-). Excluded from default runs and from
+      // the deploy e2e gate unless a frankbook trigger path changed — the
+      // links terminate on FrankBook-Pro, which sleeps (see tests/e2e/run.sh
+      // and scripts/deploy.sh). Matched by number prefix like every project
+      // here; joined 2026-10-05.
+      testMatch: /19-.*\.spec\.ts/,
+      use: { storageState: ".auth/qa-e2e-admin.json" },
+      dependencies: ["setup"],
+    },
  ],
 });
@@ -4,8 +4,14 @@
 # part of ./tests/run.sh. Credentials live in tests/e2e/.env.local
 # (git-ignored; see .env.local.example). Usage:
 #
-#   tests/e2e/run.sh                  # full suite
+#   tests/e2e/run.sh                  # full suite (frankbook project excluded)
 #   tests/e2e/run.sh --project guest  # any playwright args pass through
+#
+# The frankbook project (19-frankbook.spec.ts — the *arr/qBittorrent walk)
+# is NOT in the default project set: its links terminate on FrankBook-Pro,
+# which sleeps when idle, so it must not gate unrelated deploys. deploy.sh
+# sets E2E_INCLUDE_FRANKBOOK=1 when a frankbook trigger path changed. An
+# explicit --project argument in "$@" bypasses the default set entirely.
 set -euo pipefail
 cd "$(dirname "$0")"

@@ -24,4 +30,24 @@ fi

 npx playwright install chromium

-exec npx playwright test "$@"
+# Default project set; the frankbook project joins only when deploy.sh saw a
+# frankbook trigger path (E2E_INCLUDE_FRANKBOOK=1). Other playwright args
+# (--list, --grep, file filters) pass through alongside the default set; an
+# explicit --project in "$@" takes over untouched.
+DEFAULT_PROJECTS=(setup standard admin guest)
+if [[ "${E2E_INCLUDE_FRANKBOOK:-0}" == "1" ]]; then
+    DEFAULT_PROJECTS+=(frankbook)
+fi
+has_project=0
+for arg in "$@"; do
+    case "$arg" in
+        --project|--project=*) has_project=1 ;;
+    esac
+done
+PROJECT_ARGS=()
+if [[ "$has_project" -eq 0 ]]; then
+    for project in "${DEFAULT_PROJECTS[@]}"; do
+        PROJECT_ARGS+=(--project "$project")
+    done
+fi
+exec npx playwright test "$@" "${PROJECT_ARGS[@]}"
@@ -24,10 +24,28 @@ test("admin panel renders admin-category services", async ({ page }) => {
 test("admin open links reach their apps through SSO", async ({ page }) => {
  test.setTimeout(240_000);
  // Expected link count comes from the live registry (admin showLink=true
-  // entries, now including the *arr vhosts that use their own auth).
-  const registry: { name: string; category: string; showLink: boolean }[] =
-    await (await page.request.get("/services.json")).json();
+  // entries, now including the *arr vhosts that use their own auth). The
+  // frankbook-hosted links (probe on 192.168.0.112) are NOT walked here:
+  // they terminate on FrankBook-Pro, which sleeps when idle, so their walk
+  // lives in 19-frankbook.spec.ts (frankbook project — excluded from the
+  // default e2e run and from the deploy gate unless a frankbook trigger
+  // path changed; see tests/e2e/run.sh and scripts/deploy.sh).
+  const registry: {
+    name: string;
+    category: string;
+    showLink: boolean;
+    url: string;
+    probe: string;
+  }[] = await (await page.request.get("/services.json")).json();
  const expectedAdminLinks = registry.filter((s) => s.category === "admin" && s.showLink).length;
+  const frankbookUrls = registry
+    .filter(
+      (s) =>
+        s.category === "admin" &&
+        s.showLink &&
+        ((s.probe ?? "").startsWith("192.168.0.112") || s.url.includes("192.168.0.112")),
+    )
+    .map((s) => new URL(s.url).origin);
  await page.goto("/admin/");
  await expect(page.locator("#grid a.open")).toHaveCount(expectedAdminLinks);
  const hrefs = await page.locator("#grid a.open").evaluateAll((els) =>
@@ -35,6 +53,7 @@ test("admin open links reach their apps through SSO", async ({ page }) => {
  );

  for (const href of hrefs) {
+    if (frankbookUrls.includes(new URL(href).origin)) continue; // 19-frankbook walks these
    const target = new URL(href);
    await page.goto(href);
    // Every guarded vhost runs its own Anubis proof-of-work on first visit.
@@ -0,0 +1,69 @@
+import { expect, test } from "@playwright/test";
+
+// The *arr / qBittorrent admin links terminate on FrankBook-Pro
+// (192.168.0.112), a Mac mini that SLEEPS when idle: Caddy answers 502 for
+// the vhosts and the raw Sonarr link times out. This walk therefore lives in
+// its own project, excluded from the default e2e run (tests/e2e/run.sh) and
+// from the deploy gate's invocation (scripts/deploy.sh) unless a frankbook
+// trigger path changed: this spec, frankbook-pro/**,
+// shared/homepage/services.json, or the dankclaw Caddyfile that serves the
+// arr vhosts. A sleeping Mac mini must never fail an unrelated deploy.
+//
+// Assertions mirror 11-admin's "admin open links reach their apps through
+// SSO": the guard chain settles (Anubis PoW where present — Sonarr's raw-IP
+// link is internal-only and has none), the hostname holds, no authentik form
+// appears, and the app answers below 500.
+
+const FRANKBOOK_IP = "192.168.0.112";
+const CHALLENGE = "Making sure you're not a bot";
+
+interface RegistryEntry {
+  name: string;
+  category: string;
+  showLink: boolean;
+  url: string;
+  probe: string;
+}
+
+test("frankbook admin open links reach their apps", async ({ page }) => {
+  test.setTimeout(240_000);
+  const registry: RegistryEntry[] = await (await page.request.get("/services.json")).json();
+  const frankbook = registry.filter(
+    (s) =>
+      s.category === "admin" &&
+      s.showLink &&
+      ((s.probe ?? "").startsWith(FRANKBOOK_IP) || s.url.includes(FRANKBOOK_IP)),
+  );
+  const frankbookUrls = frankbook.map((s) => new URL(s.url).origin);
+
+  await page.goto("/admin/");
+  const hrefs = await page
+    .locator("#grid a.open")
+    .evaluateAll((els) => els.map((el) => (el as HTMLAnchorElement).href));
+  const frankbookHrefs = hrefs.filter((href) => frankbookUrls.includes(new URL(href).origin));
+  // The rendered frankbook link set must mirror the registry: a renamed arr
+  // vhost or a dropped probe fails here instead of silently dropping the
+  // walk's coverage.
+  expect(frankbookHrefs, "rendered frankbook links must match the registry").toHaveLength(
+    frankbook.length,
+  );
+
+  for (const href of frankbookHrefs) {
+    const target = new URL(href);
+    await page.goto(href);
+    // Guarded vhosts run their own Anubis proof-of-work on first visit; the
+    // raw-IP Sonarr link has none and settles immediately.
+    await expect
+      .poll(
+        () => page.locator("body").innerText().then((t) => !t.includes(CHALLENGE)),
+        { timeout: 90_000 },
+      )
+      .toBe(true);
+    await expect
+      .poll(() => new URL(page.url()).hostname, { timeout: 60_000 })
+      .toBe(target.hostname);
+    expect(await page.locator('input[name="uidField"]').count()).toBe(0);
+    const resp = await page.request.get(page.url());
+    expect(resp.status(), `${target.hostname} must answer, not error`).toBeLessThan(500);
+  }
+});
@@ -10,7 +10,8 @@ load helpers
    for f in run.sh package.json playwright.config.ts .env.local.example \
             lib/env.ts lib/login.ts \
             specs/00-auth.setup.ts specs/10-homepage.spec.ts specs/11-admin.spec.ts \
-             specs/12-crawl.spec.ts specs/13-sso.spec.ts specs/14-guard.spec.ts; do
+             specs/12-crawl.spec.ts specs/13-sso.spec.ts specs/14-guard.spec.ts \
+             specs/19-frankbook.spec.ts; do
        [ -f "$REPO_ROOT/tests/e2e/$f" ] || {
            echo "missing tests/e2e/$f"
            return 1
@@ -1,334 +1,273 @@
 #!/usr/bin/env bats
-# Seam: the frank-large root unit on dankclaw.
+# Seam: frank-large serves peonist-ai/halogen-qwen3.8-27b (a .hgn halogen
+# checkpoint) on the Strix Halo iGPU through the halogen CONTAINER
+# (ghcr.io/peonist-ai/halogen, digest-pinned), driven by the root unit's
+# serve script — one conversation of 200,700 tokens with the DFlash2
+# drafter ON. 2026-10-05 cutover from the ROCmFPX llama.cpp shape.
 #
-# Since 2026-09-28 this endpoint is served by gufo (gufo-org/gufo), the
-# Strix Halo inference engine built from the pinned v0.1.1 tag, serving
-# Swift-1.5-Qwen3.8-27B-Uncensored (abliterated) UD-Q8_K_XL with the z-lab
-# DFlash2 Q4_K_M drafter. Two changes landed on this endpoint that day: the
-# engine (from the stock llama.cpp prebuilt) and the payload (from
-# unsloth/Qwen3.8-27B-GGUF:UD-Q8_K_XL). Port, alias and client contract did
-# not move.
+# This seam exists because the artifacts are a triangle that must agree on
+# exact strings the engine reads: the image DIGEST, the checkpoint file
+# name AND byte count inside the /models mount, and the tokenizer's
+# flat-file shape (a HF cache snapshot's symlinks dangle inside the
+# container). A serve gate and a deployer gate that disagree is the
+# 2026-10-01 crash-loop; both sides are pinned here, plus the one engine
+# law the fleet must never "optimize" away: HALOGEN_KV_SLOTS=1 is what
+# keeps speculative decoding on.
 #
-# Most of this seam runs the serve script against a stub engine and asserts
-# on the argv it actually execs, so what is pinned is the command the
-# operator gets rather than the text of the script: a tunable renamed,
-# dropped, or silently defaulted away fails here the way it would fail a
-# deployment.
-load helpers
+# Everything is tested from what an operator or supervisor can observe —
+# the command line podman would run, the exit code, the deployer's error
+# text — never from anything inside the engine.

-ENV_FILE="$REPO_ROOT/scripts/frank-large.env"
-UNIT="$REPO_ROOT/scripts/frank-large.service"
-SERVE="$REPO_ROOT/scripts/serve-frank-large.sh"
-FETCH="$REPO_ROOT/scripts/download-frank-large-swift.sh"
-DRAFTER_FETCH="$REPO_ROOT/scripts/download-frank-large.sh"
-BUILD="$REPO_ROOT/scripts/build-gufo.sh"
-REMOTE_DEPLOY="$REPO_ROOT/scripts/deploy-remote.sh"
-# The deployed target's filename, kept here so a rename in the env file and a
-# rename in this stub cannot drift apart unnoticed.
-TARGET_STUB="Swift-1.5-Qwen3.8-27B-Uncensored-Dynamic-MTP-UD-Q8_K_XL.gguf"
 setup() {
-    STAGE="$(mktemp -d)"
-    RUNTIME_DIR="$STAGE/gufo"
-    MODEL_DIR="$STAGE/models"
-    DRAFTER_DIR="$STAGE/draft"
-    mkdir -p "$RUNTIME_DIR/bin" "$MODEL_DIR" "$DRAFTER_DIR"
-    # A stub engine: the serve script execs it, so $@ is exactly what gufo
-    # would be invoked with.
-    cat > "$RUNTIME_DIR/bin/gufo" <<'STUB'
-#!/usr/bin/env bash
-printf '%s\n' "$@"
-STUB
-    chmod +x "$RUNTIME_DIR/bin/gufo"
-    # The target and the drafter live in SEPARATE directories since the
-    # 2026-09-28 payload swap: the target came from a different repository,
-    # the DFlash2 drafter did not move. A stub that colocated them would
-    # never notice DRAFTER_DIR going stale.
-    : > "$MODEL_DIR/$TARGET_STUB"
-    : > "$DRAFTER_DIR/Qwen3.8-27B-DFlash2-Q4_K_M.gguf"
-    STUB_ENV="$STAGE/stub.env"
-    cat > "$STUB_ENV" <<EOF
-RUNTIME_DIR=$RUNTIME_DIR
-MODEL_DIR=$MODEL_DIR
-DRAFTER_DIR=$DRAFTER_DIR
-EOF
+    ROOT="$(cd "$(dirname "$BATS_TEST_FILENAME")/.." && pwd)"
+    SERVE="$ROOT/scripts/serve-frank-large.sh"
+    ENV_FILE="$ROOT/scripts/frank-large.env"
+    UNIT="$ROOT/scripts/frank-large.service"
+    DEPLOYER="$ROOT/scripts/deploy-remote.sh"
+    BUILD="$ROOT/scripts/build-rocmfpx.sh"
+    TEST_TMP="$(mktemp -d)"
+    touch "$TEST_TMP/empty.env"
 }

 teardown() {
-    if [ -n "${STAGE:-}" ]; then
-        rm -rf "$STAGE"
-    fi
+    rm -rf "$TEST_TMP"
 }

-# serve_argv [env-file] — run the serve script and capture the engine's argv,
-# one token per line, in $output. No argument exercises the built-in defaults
-# with the stub's paths.
-serve_argv() {
-    local envfile="${1:-$STUB_ENV}"
-    if [ -n "$envfile" ]; then
-        FRANK_LARGE_ENV_FILE="$envfile" run bash "$SERVE"
-    else
-        run env FRANK_LARGE_ENV_FILE="$STAGE/absent.env" bash "$SERVE"
-    fi
+# podman_stub — a fake podman on PATH that prints the argv of `run` and
+# `wait` and answers `images` with whatever digests the test names, so
+# "what would the supervisor run" is observable output and the digest
+# gate is testable without a 3.5 GB image.
+podman_stub() {
+    mkdir -p "$TEST_TMP/bin"
+    cat > "$TEST_TMP/bin/podman" <<'STUB'
+#!/usr/bin/env bash
+case "$1" in
+    images) printf '%s\n' "${STUB_DIGESTS:-}" ;;
+    rm)     exit 0 ;;
+    run|wait) echo "PODMAN $*"; exit "${STUB_WAIT_RC:-0}" ;;
+    *)      exit 0 ;;
+esac
+STUB
+    chmod +x "$TEST_TMP/bin/podman"
 }

-# argv_has <token> — exact token match, so --port never satisfies a test
-# looking for a different flag.
-argv_has() {
-    grep -qxF -- "$1" <<<"$output"
+# good_env — the minimum env the serve script demands, with the digest
+# stub agrees on and zero-byte artifacts passing the size gate.
+good_env() {
+    : > "$TEST_TMP/empty.env"
+    touch "$TEST_TMP/qwen.hgn"
+    mkdir -p "$TEST_TMP/tokenizer" && touch "$TEST_TMP/tokenizer/tok.json"
+    GOOD_ENV=(
+        "FRANK_LARGE_ENV_FILE=$TEST_TMP/empty.env"
+        "FRANK_LARGE_IMAGE=ghcr.io/peonist-ai/halogen@sha256:cafe"
+        "FRANK_LARGE_CONTAINER=frank-large-halogen"
+        "FRANK_LARGE_LEGACY_CONTAINERS=frank-large"
+        "FRANK_LARGE_DEV_KFD=$TEST_TMP/qwen.hgn"
+        "FRANK_LARGE_DEV_DRI=$TEST_TMP"
+        "FRANK_LARGE_LOG=$TEST_TMP/serve.log"
+        "FRANK_LARGE_PUBLISH=7770:8731"
+        "FRANK_LARGE_MODEL_DIR=$TEST_TMP"
+        "FRANK_LARGE_CHECKPOINT=qwen.hgn"
+        "FRANK_LARGE_CHECKPOINT_BYTES=0"
+        "FRANK_LARGE_TOKENIZER_FILES=tok.json"
+        "HALOGEN_MODEL_ID=frank-large"
+        "HALOGEN_SLOT_CTX=200700"
+        "HALOGEN_KV_SLOTS=1"
+        "HALOGEN_DRAFTER=2"
+        "STUB_DIGESTS=localhost/x ghcr.io/peonist-ai/halogen@sha256:cafe"
+    )
 }

-# argv_value <flag> — the token following <flag>, empty if the flag is absent.
-argv_value() {
-    local i
-    i="$(grep -nx -- "$1" <<<"$output" | cut -d: -f1)"
-    [ -n "$i" ] || return 0
-    sed -n "$((i + 1))p" <<<"$output"
+@test "the env file defines the halogen deployment values" {
+    grep -qE '^FRANK_LARGE_IMAGE=ghcr\.io/peonist-ai/halogen@sha256:[0-9a-f]{64}$' "$ENV_FILE"
+    grep -qE '^FRANK_LARGE_CONTAINER=frank-large-halogen$' "$ENV_FILE"
+    grep -qE '^FRANK_LARGE_PUBLISH=7770:8731$' "$ENV_FILE"
+    grep -qE '^FRANK_LARGE_CHECKPOINT=.*\.hgn$' "$ENV_FILE"
+    grep -qE '^FRANK_LARGE_CHECKPOINT_BYTES=[0-9]+$' "$ENV_FILE"
+    grep -qE '^FRANK_LARGE_CHECKPOINT_SHA256=[0-9a-f]{64}$' "$ENV_FILE"
+    grep -qE '^FRANK_LARGE_TOKENIZER_FILES="tokenizer\.json( [a-z_.]+)+"$' "$ENV_FILE"
+    grep -qE '^HALOGEN_MODEL_ID=frank-large$' "$ENV_FILE"
+    grep -qE '^HALOGEN_SLOT_CTX=[0-9]+$' "$ENV_FILE"
 }

-@test "the frank-large unit, env, serve and build files exist where the deployer installs them from" {
-    [ -f "$ENV_FILE" ]
-    [ -f "$UNIT" ]
-    [ -x "$SERVE" ]
-    [ -x "$BUILD" ]
+@test "the drafter stays on: slots must stay 1" {
+    # halogen makes batching and speculation mutually exclusive — with
+    # more than one slot the drafter is DISABLED and per-stream decode
+    # falls to serial speed (the sibling engine measured 41 -> 14 t/s
+    # when slots went up). frank-large was a one-session endpoint before
+    # this swap too (PARALLEL=1), so slots=1 preserves the contract;
+    # raising it is a product decision, never a default change.
+    grep -qE '^HALOGEN_KV_SLOTS=1$' "$ENV_FILE"
+    grep -qE '^HALOGEN_DRAFTER=2$' "$ENV_FILE"
+    # The cap/timeout pair is coupled (FLAGS.md table): a request at the
+    # cap must not outlast the queue timeout, or everyone queued behind
+    # it eats a 503.
+    grep -qE '^HALOGEN_MAX_TOKENS_CAP=[0-9]+$' "$ENV_FILE"
+    grep -qE '^HALOGEN_QUEUE_TIMEOUT=[0-9]+$' "$ENV_FILE"
 }

-@test "frank-large is boot-persistent, self-healing and unprivileged" {
-    grep -q '^WantedBy=multi-user.target$' "$UNIT"
-    grep -q '^Restart=on-failure$' "$UNIT"
-    grep -q '^RestartSec=' "$UNIT"
-    grep -q '^User=dank$' "$UNIT"
-    # The engine pins buffers; the memlock budget must be unbounded or the
-    # load fails on the RLIMIT.
-    grep -q '^LimitMEMLOCK=infinity$' "$UNIT"
-    # The unit runs the repo's own serve script with the repo's env.
-    grep -q 'ExecStart=/home/dank/serverAppConfigs/scripts/serve-frank-large.sh' "$UNIT"
-    grep -q 'FRANK_LARGE_ENV_FILE=/home/dank/serverAppConfigs/scripts/frank-large.env' "$UNIT"
+@test "the serve script refuses when there is no GPU to serve on" {
+    podman_stub
+    good_env
+    run env PATH="$TEST_TMP/bin:$PATH" "${GOOD_ENV[@]}" \
+        FRANK_LARGE_DEV_KFD="$TEST_TMP/absent-node" "$SERVE"
+    [ "$status" -ne 0 ]
+    [[ "$output" == *"no gfx1151 to serve on"* ]]
 }

-@test "the fleet endpoint and alias survive the cutover" {
-    serve_argv
+@test "the KV budget fits unified memory with margin" {
+    # Pool arithmetic is the engine's documented product: slots x
+    # slot_ctx x 64 KiB. Against it: 33.4 GiB of mapped weights, the
+    # engine working set + auto prompt cache (~25 GiB of headroom
+    # allowed), host baseline and frank-small co-tenancy. The box has
+    # 121 GiB unified; the 2026-09-16 halogen era ran the same engine
+    # class here on a LARGER pool with less co-tenancy slack.
+    ctx="$(sed -nE 's/^HALOGEN_SLOT_CTX=([0-9]+)$/\1/p' "$ENV_FILE")"
+    slots="$(sed -nE 's/^HALOGEN_KV_SLOTS=([0-9]+)$/\1/p' "$ENV_FILE")"
+    pool_gib=$(( slots * ctx * 65536 / 1073741824 ))
+    total_gib=$(( 34 + pool_gib + 25 + 15 ))
+    [ "$total_gib" -le 121 ]
+}
+
+@test "the serve script refuses when the image is not in storage" {
+    podman_stub
+    good_env
+    run env PATH="$TEST_TMP/bin:$PATH" "${GOOD_ENV[@]}" STUB_DIGESTS="" "$SERVE"
+    [ "$status" -ne 0 ]
+    [[ "$output" == *"not in podman storage"* ]]
+    [[ "$output" == *"podman pull"* ]]
+}
+
+@test "the serve script refuses a missing checkpoint and names the fetch script" {
+    podman_stub
+    good_env
+    run env PATH="$TEST_TMP/bin:$PATH" "${GOOD_ENV[@]}" \
+        FRANK_LARGE_CHECKPOINT=absent.hgn "$SERVE"
+    [ "$status" -ne 0 ]
+    [[ "$output" == *"checkpoint"*"missing"* ]]
+    [[ "$output" == *"download-frank-large-halogen.sh"* ]]
+}
+
+@test "the serve script refuses a checkpoint of the wrong byte count" {
+    # A truncated curl exit-0 artifact must not boot; the model card's
+    # byte count is the contract (fleet fetch law).
+    podman_stub
+    good_env
+    run env PATH="$TEST_TMP/bin:$PATH" "${GOOD_ENV[@]}" \
+        FRANK_LARGE_CHECKPOINT_BYTES=42 "$SERVE"
+    [ "$status" -ne 0 ]
+    [[ "$output" == *"expected 42"* ]]
+}
+
+@test "the serve script refuses a symlinked tokenizer file" {
+    # A HuggingFace cache snapshot's entries are symlinks into blobs/ and
+    # dangle inside the container mount — even when the directory itself
+    # is real.
+    podman_stub
+    good_env
+    rm "$TEST_TMP/tokenizer/tok.json"
+    ln -sf /nowhere/x "$TEST_TMP/tokenizer/tok.json"
+    run env PATH="$TEST_TMP/bin:$PATH" "${GOOD_ENV[@]}" "$SERVE"
+    [ "$status" -ne 0 ]
+    [[ "$output" == *"missing or a symlink"* ]]
+}
+
+@test "the render names the container, the port map and the engine shape" {
+    podman_stub
+    good_env
+    run env PATH="$TEST_TMP/bin:$PATH" "${GOOD_ENV[@]}" "$SERVE"
    [ "$status" -eq 0 ]
-    # frank-large at :7770, whatever engine answers it.
-    [ "$(argv_value --served-model-name)" = "frank-large" ]
-    [ "$(argv_value --port)" = "7770" ]
-    [ "$(argv_value --host)" = "0.0.0.0" ]
+    # `podman run` output is redirected to the log by design (the unit
+    # journal must not absorb engine chatter); the stub's printed argv
+    # lands there, so the render assertions read the same log.
+    output="$output
+$(cat "$TEST_TMP/serve.log")"
+    [[ "$output" == *"--name frank-large-halogen"* ]]
+    [[ "$output" == *"--publish 7770:8731"* ]]
+    [[ "$output" == *"-e HALOGEN_MODEL_ID=frank-large"* ]]
+    [[ "$output" == *"-e HALOGEN_SLOT_CTX=200700"* ]]
+    [[ "$output" == *"-e HALOGEN_KV_SLOTS=1"* ]]
+    [[ "$output" == *"-e HALOGEN_DRAFTER=2"* ]]
+    [[ "$output" == *"-e HALOGEN_CHECKPOINT=/models/qwen.hgn"* ]]
+    [[ "$output" == *"--device $TEST_TMP/qwen.hgn"* ]]
+    [[ "$output" == *"$TEST_TMP/tokenizer:/tokenizer:ro"* ]]
+    [[ "$output" == *"ghcr.io/peonist-ai/halogen@sha256:cafe"* ]]
+    # The unit's restart policy is real because the script becomes
+    # `podman wait`: the container's exit code is the service's exit.
+    [[ "$output" == *"PODMAN wait frank-large-halogen"* ]]
 }

-@test "the built-in defaults are one session of the full 262144 window with DFlash2" {
-    serve_argv
-    [ "$status" -eq 0 ]
-    # 262144 is the model's native window (the operator's call, 2026-09-21).
-    [ "$(argv_value --context)" -eq 262144 ]
-    # One session is the single stream every prior frank-large served.
-    [ "$(argv_value --sessions)" -eq 1 ]
-    [ "$(argv_value --speculative)" = "dflash2" ]
-    # A draft ceiling of 0 would silently serve the autoregressive rate.
-    [ "$(argv_value --draft-tokens)" -gt 0 ]
+@test "the shipped env pins the fleet's window and shape" {
+    # 200700 is the per-conversation hard ceiling (a longer prompt is a
+    # 400 naming the limit, never a silent truncation), and the harness
+    # compaction math must fit prompt + budget under it: OMP compacts at
+    # 70% = 140490, and 140490 + 57344 = 197834 <= 200700.
+    grep -qE '^HALOGEN_SLOT_CTX=200700$' "$ENV_FILE"
+    ctx=200700
+    [ $(( ctx * 70 / 100 + 57344 )) -le "$ctx" ]
 }

-@test "the shipped env file, not just the defaults, serves the deployed shape" {
-    # The defaults are a floor; systemd sources the env file. Only the
-    # on-host paths are redirected at the stub, so every fleet value in the
-    # shipped file is the one under test. MODEL_FILE and DRAFTER_FILE are
-    # absolute in the shipped file, so they need redirecting too — a sed that
-    # only rewrote MODEL_DIR would leave them pointing at /home/dank and the
-    # test would pass for the wrong reason (or fail on a box that has them).
-    sed -e "s#^RUNTIME_DIR=.*#RUNTIME_DIR=$RUNTIME_DIR#" \
-        -e "s#^MODEL_DIR=.*#MODEL_DIR=$MODEL_DIR#" \
-        -e "s#^MODEL_FILE=.*#MODEL_FILE=$MODEL_DIR/$TARGET_STUB#" \
-        -e "s#^DRAFTER_DIR=.*#DRAFTER_DIR=$DRAFTER_DIR#" \
-        -e "s#^DRAFTER_FILE=.*#DRAFTER_FILE=$DRAFTER_DIR/Qwen3.8-27B-DFlash2-Q4_K_M.gguf#" \
-        "$ENV_FILE" > "$STAGE/shipped.env"
-    serve_argv "$STAGE/shipped.env"
-    [ "$status" -eq 0 ]
-    [ "$(argv_value --served-model-name)" = "frank-large" ]
-    [ "$(argv_value --host)" = "0.0.0.0" ]
-    [ "$(argv_value --port)" = "7770" ]
-    [ "$(argv_value --context)" -eq 262144 ]
-    [ "$(argv_value --sessions)" -eq 1 ]
-    [ "$(argv_value --speculative)" = "dflash2" ]
-    [ "$(argv_value --draft-tokens)" -gt 0 ]
-    [ "$(argv_value --draft-policy)" = "adaptive" ]
-    # The shipped file must name the target the fleet actually serves, not a
-    # path that merely happens to exist.
-    [ "$(argv_value --model)" = "$MODEL_DIR/$TARGET_STUB" ]
-    [ "$(argv_value --dflash-model)" = "$DRAFTER_DIR/Qwen3.8-27B-DFlash2-Q4_K_M.gguf" ]
+@test "no HIP-sensitive bare names leak through the env file" {
+    # HIP reads bare env vars out of a `set -a` source; a tunable named
+    # HOST or TEMP used to send the JIT cache to path 1.0 and segfault
+    # every load (kept from the llama.cpp era — the container gets only
+    # the explicit -e list, but the host env still reaches podman itself).
+    ! grep -qE '^(HOST|TEMP|GPU_MAX_HEAP_SIZE)=' "$ENV_FILE"
 }

-@test "the engine is pointed at the payload that is actually on disk" {
-    serve_argv
-    [ "$status" -eq 0 ]
-    [ -f "$(argv_value --model)" ]
-    # A drafter path that does not exist would load the target and decode at
-    # the autoregressive rate, which reads as "the new runtime is slow".
-    [ -f "$(argv_value --dflash-model)" ]
-    [ "$(argv_value --model)" != "$(argv_value --dflash-model)" ]
+@test "the deployer gates exactly the artifacts the serve script opens" {
+    # Serve and deploy must name the SAME image digest, checkpoint and
+    # tokenizer: the 2026-10-01 crash-loop was exactly these twins
+    # disagreeing mid-cutover.
+    img="$(sed -nE 's/^FRANK_LARGE_IMAGE=(.*)$/\1/p' "$ENV_FILE")"
+    grep -qF "LLM_LARGE_IMAGE=\"\${LLM_LARGE_IMAGE:-$img}\"" "$DEPLOYER"
+    dir="$(sed -nE 's/^FRANK_LARGE_MODEL_DIR=(.*)$/\1/p' "$ENV_FILE")"
+    ckpt="$(sed -nE 's/^FRANK_LARGE_CHECKPOINT=(.*)$/\1/p' "$ENV_FILE")"
+    tokf="$(sed -nE 's/^FRANK_LARGE_TOKENIZER_FILES="(.*)"$/\1/p' "$ENV_FILE")"
+    grep -qF "LLM_LARGE_TOK_DIR=\"\${LLM_LARGE_TOK_DIR:-$dir/tokenizer}\"" "$DEPLOYER"
+    grep -qF "LLM_LARGE_TOK_FILES=\"\${LLM_LARGE_TOK_FILES:-$tokf}\"" "$DEPLOYER"
+    bytes="$(sed -nE 's/^FRANK_LARGE_CHECKPOINT_BYTES=([0-9]+)$/\1/p' "$ENV_FILE")"
+    grep -qF "LLM_LARGE_CKPT_BYTES=\"\${LLM_LARGE_CKPT_BYTES:-$bytes}\"" "$DEPLOYER"
 }

-@test "frank-large stays text-only" {
-    # The payload has no mmproj, so an --mmproj would point at nothing. The
-    # fleet's image path is frank-small; models.yml declares frank-large
-    # text-only.
-    serve_argv
-    ! argv_has --mmproj
-    grep -q -- '--mmproj' "$REPO_ROOT/scripts/serve-frank-small.sh"
+@test "the deployer points a missing artifact at the halogen fetch script" {
+    grep -qE 'download-frank-large-halogen\.sh' "$DEPLOYER"
+    grep -qE 'podman pull' "$DEPLOYER"
 }

-@test "the engine is asked for greedy decoding and for preserved prior reasoning" {
-    # Greedy is what this endpoint has always served; a silent switch to
-    # sampled output would be a model-behaviour change disguised as a
-    # runtime swap. Prior thinking blocks must survive across turns or a
-    # multi-turn agent loses its own reasoning between tool calls.
-    serve_argv
-    [ "$(argv_value --temperature)" = "0" ]
-    [ "$(argv_value --preserve-thinking)" = "on" ]
+@test "the stray sweep cannot take frank-small or the deployer down" {
+    # frank-small also listens on this host (7771, llama-server) and the
+    # deployer's own command line contains "frank-large"; only the FULL
+    # container name matches a stray manual halogen run and nothing else.
+    grep -q "pkill -f 'frank-large-halogen'" "$DEPLOYER"
+    ! grep -qE "pkill -f 'llama-server'" "$DEPLOYER"
+    ! grep -qE "pkill -f 'frank-large'" "$DEPLOYER"
 }

-@test "the serve script refuses a missing runtime with the build pointer" {
-    rm -f "$RUNTIME_DIR/bin/gufo"
-    serve_argv
-    [ "$status" -ne 0 ]
-    [[ "$output" == *"runtime not found"* ]]
-    [[ "$output" == *"build-gufo.sh"* ]]
-    # A restart loop would be the alternative; nothing may be exec'd.
-    ! argv_has serve
+@test "the deployed unit keeps the fleet's restart and boot-persistence shape" {
+    grep -qE '^Restart=(always|on-failure)$' "$UNIT"
+    grep -qE '^WantedBy=' "$UNIT"
+    grep -qE '^ExecStart=.*/serve-frank-large\.sh$' "$UNIT"
+    grep -qE '^User=dank$' "$UNIT"
+    grep -qE '^LimitMEMLOCK=infinity$' "$UNIT"
+    # The service's lifetime is the container's; stopping the service
+    # must stop the container by its real name.
+    grep -qE '^ExecStop=.*podman stop .* frank-large-halogen$' "$UNIT"
 }

-@test "the serve script refuses a missing drafter with a fetch pointer" {
-    # The drafter lives in its own directory now, so the target can be present
-    # and this still has to be caught.
-    rm -f "$DRAFTER_DIR/Qwen3.8-27B-DFlash2-Q4_K_M.gguf"
-    serve_argv
-    [ "$status" -ne 0 ]
-    [[ "$output" == *"missing asset"* ]]
-    [[ "$output" == *"Qwen3.8-27B-DFlash2-Q4_K_M.gguf"* ]]
-    [[ "$output" == *"download-frank-large"* ]]
-    ! argv_has serve
+@test "the unit runs the halogen runtime and no retired path" {
+    grep -q 'halogen' "$UNIT"
+    # Every retired payload/build stays on the host, so a stale path in
+    # a unit still resolves — to a runtime nobody has measured in weeks.
+    ! grep -qE 'GitPackages/(ROCmFPX-kairic|llama.cpp-ik_llama|llama-builds|ciru-orca)' "$UNIT"
+    ! grep -qE 'kingjones-ROCmFPX' "$UNIT"
 }

-@test "the serve script refuses a missing target and names the file" {
-    rm -f "$MODEL_DIR/$TARGET_STUB"
-    serve_argv
-    [ "$status" -ne 0 ]
-    [[ "$output" == *"missing asset"* ]]
-    [[ "$output" == *"$TARGET_STUB"* ]]
-}
-
-@test "both payload files are fetched resumably and verified against declared sizes + sha256" {
-    # The target and the drafter come from DIFFERENT repositories, fetched by
-    # two scripts. Anchor the repo checks to the URL ASSIGNMENT, not to the
-    # file: the same repo string also appears in each script's header comment,
-    # so a plain grep stayed green after the URL was repointed at another
-    # repository -- the script would then download something the serve script
-    # never asked for.
-    grep -qE '^URL="https://huggingface\.co/ajgazin/Swift-1\.5-Qwen3\.8-27B-Uncensored-Dynamic-MTP-GGUF/resolve/main/' "$FETCH"
-    grep -qE 'https://huggingface\.co/z-lab/Qwen3\.8-27B-DFlash2-GGUF/resolve/main/' "$DRAFTER_FETCH"
-    for f in "$FETCH" "$DRAFTER_FETCH"; do
-        # -f so an HTTP error is a failure rather than a small "artifact";
-        # -C - so an interrupted 29 GiB run resumes.
-        grep -q -- '-fL -C -' "$f"
-        # The size and the sha256 are compared after the transfer, not trusted.
-        grep -q 'SHA256 MISMATCH' "$f"
-        grep -q 'SIZE MISMATCH' "$f"
-    done
-}
-
-@test "the serve script and the fetch scripts agree on the payload filenames" {
-    # The serve script's MODEL_FILE and the fetch script's REL are two
-    # independent spellings of one filename, and a repo is not required to
-    # keep them in step: a re-upload under a new name, or an env edited
-    # without the fetch, leaves the endpoint unable to start with a
-    # "missing asset" that no test would have predicted. Bind them here.
-    local served fetched drafter_served drafter_fetched
-    served="$(basename "$(grep '^MODEL_FILE=' "$ENV_FILE" | cut -d= -f2-)")"
-    fetched="$(sed -n 's/^REL="\(.*\)"$/\1/p' "$FETCH")"
-    [ -n "$served" ]
-    [ "$served" = "$fetched" ]
-    drafter_served="$(basename "$(grep '^DRAFTER_FILE=' "$ENV_FILE" | cut -d= -f2-)")"
-    drafter_fetched="$(sed -n 's/^ *"\(Qwen3\.8-27B-DFlash2[^ ]*\.gguf\) .*/\1/p' "$DRAFTER_FETCH")"
-    [ -n "$drafter_served" ]
-    [ "$drafter_served" = "$drafter_fetched" ]
-}
-
-@test "the drafter is still fetched by the script that carries its Hub repo" {
-    # Guards a future "simplification" that folds the drafter into the Swift
-    # script and silently orphans the artifact the serve script still points
-    # at via DRAFTER_DIR.
-    grep -q 'Qwen3.8-27B-UD-Q8_K_XL' "$DRAFTER_FETCH"
-    ! grep -q 'DFlash2' "$FETCH"
-}
-
-@test "the build script refuses a source tree that is not the pinned commit" {
-    # gufo ships no prebuilt Linux binaries, so the runtime is built here.
-    # A tag that moves under us would silently install different kernels
-    # under the same RUNTIME_DIR, so the pin is enforced, not documented.
-    local src="$STAGE/not-the-pinned-commit"
-    mkdir -p "$src"
-    git -C "$src" init -q
-    git -C "$src" -c user.email=t@t -c user.name=t commit -q --allow-empty -m drift
-    run env GUFO_SRC_DIR="$src" GUFO_PREFIX="$STAGE/absent-prefix" bash "$BUILD" --verify
-    [ "$status" -ne 0 ]
-    [[ "$output" == *"GUFO_COMMIT mismatch"* ]]
-}
-
-@test "the build script reports a runtime that was never built" {
-    # --verify on a host with no install must say so rather than exit 0 on
-    # an empty check.
-    run env GUFO_SRC_DIR="$STAGE/no-source-tree" GUFO_PREFIX="$STAGE/absent-prefix" bash "$BUILD" --verify
-    [ "$status" -ne 0 ]
-    [[ "$output" == *"no gufo at"* ]]
-}
-
-@test "the deploy handler gates on the runtime, the payload and the port" {
-    grep -q 'relink_unit frank-large' "$REMOTE_DEPLOY"
-    grep -q 'systemctl enable frank-large.service' "$REMOTE_DEPLOY"
-    grep -q 'wait_port_free 7770' "$REMOTE_DEPLOY"
-    grep -q '127.0.0.1:7770/health' "$REMOTE_DEPLOY"
-    grep -q 'LLM_LARGE_RUNTIME' "$REMOTE_DEPLOY"
-    grep -q 'LLM_LARGE_MODEL' "$REMOTE_DEPLOY"
-    grep -q 'LLM_LARGE_DRAFT' "$REMOTE_DEPLOY"
-}
-
-@test "the deploy handler will not stop a serving endpoint it cannot replace" {
-    # The runtime gate must be live (a real conditional on an executable
-    # path, not a branch that never fires) and it must return BEFORE the
-    # stop, or a host whose gufo was never built takes frank-large down on
-    # its way to reporting the failure.
-    local large gate bail stop
-    large="$(sed -n '/^deploy_llm_large() {/,/^}/p' "$REMOTE_DEPLOY")"
-    [[ "$large" == *'build-gufo.sh'* ]]
-    gate="$(grep -n 'if \[\[ ! -x "\$LLM_LARGE_RUNTIME" \]\]' <<<"$large" | head -1 | cut -d: -f1)"
-    stop="$(grep -n 'systemctl stop frank-large' <<<"$large" | head -1 | cut -d: -f1)"
-    [ -n "$gate" ]
-    [ -n "$stop" ]
-    bail="$(awk -v g="$gate" 'NR > g && /return 1/ { print NR; exit }' <<<"$large")"
-    [ -n "$bail" ]
-    [ "$bail" -lt "$stop" ]
-}
-
-@test "the stray-run sweep keys on the drafter and cannot take frank-small down" {
-    # The sweep used to key on the TARGET filename, which the 2026-09-28
-    # payload swap moved to a different repository. It would have stopped
-    # matching the live process, and a stray run would then hold 7770 past the
-    # port gate and fail the deploy. The drafter is the part that did NOT
-    # move, and every frank-large command line carries it via --dflash-model.
-    grep -q "pkill -f 'Qwen3.8-27B-DFlash2-Q4_K_M'" "$REMOTE_DEPLOY"
-    # frank-small serves a different GGUF on the 3090 with no DFlash2 drafter,
-    # so a binary-name sweep is the only thing that would endanger it.
-    local large
-    large="$(sed -n '/^deploy_llm_large() {/,/^}/p' "$REMOTE_DEPLOY")"
-    ! grep -q 'pkill -x llama-server' <<<"$large"
-    ! grep -q 'pkill -x gufo' <<<"$large"
-}
-
-@test "the deploy handler retires the halogen quadlet without deleting it" {
-    grep -q 'retire_halogen_large_quadlet' "$REMOTE_DEPLOY"
-    grep -q 'HOME/.AGENTTRASH' "$REMOTE_DEPLOY"
-    ! grep -qE '(^|[^[:alnum:]_])rm (-[a-zA-Z]+ )?.*frank-large' "$REMOTE_DEPLOY"
-    # Nothing live ships for this endpoint as a container any more. The
-    # archived copy is host state in an ignored .AGENTTRASH — deliberately
-    # untracked, and present only in a primary checkout — so search the
-    # tracked tree, not the filesystem.
-    [ ! -e "$REPO_ROOT/dankclaw/frank-large/quadlet/frank-large.container" ]
-    ! git -C "$REPO_ROOT" grep -q 'Image=.*halogen' -- dankclaw
+@test "the rollback door stays qualified: the ROCmFPX build pin survives" {
+    # Rolling back this swap means rebuilding the fork the old payload
+    # needs; the pin (real commit, gfx1151, no webui fetch) is what makes
+    # that door trustworthy months later.
+    grep -qE 'ROCMFPX_COMMIT=.*[0-9a-f]{40}' "$BUILD"
+    grep -qE 'GPU_TARGET=.*gfx1151' "$BUILD"
+    grep -qE 'DLLAMA_BUILD_WEBUI=OFF' "$BUILD"
 }
@@ -1,19 +1,26 @@
 #!/usr/bin/env bats
 # Seam: frank-primary — the Dankserver LLM endpoint on :7770.
 #
-# This file asserts the tcclaviger RDNA4 vLLM ROOT-UNIT shape (the 2026-09-29
-# cutover that replaced the R9V radiance fork): the serve script drives
-# `podman run` DIRECTLY (the image entrypoint execs `vllm serve`), with every
-# engine flag qualified by boot probe on 2026-09-29 and carried in the repo
-# env file. The seam pins what the flags alone do not say: the qualification
-# law (600k ctx / 3 sessions / q8-q8 KV were MEASURED to fit — 729,545 tokens
-# of capacity at 7.2 GiB per rank — and KV_BYTES is PER RANK), the self-healing
-# law (the run returns before the engine is up, so the serve script must wait
-# on the container for Restart=on-failure to mean anything), the memlock law
-# that makes the tiered-expert pin legal, the deployer's image/payload gates
-# with the false-green MainPID check, and the prefetch law (expert prefetch
-# measured at 36% precision — OFF is the incumbent; a regression that
-# reintroduces it silently costs ~40-50% decode).
+# This file asserts the tcclaviger RDNA4 vLLM ROOT-UNIT shape (the
+# 2026-10-01 BF16 cutover, the overnight redo campaign's verdict): the serve
+# script drives `podman run` DIRECTLY (the image entrypoint execs
+# `vllm serve`), with every engine flag qualified by boot probe and carried
+# in the repo env file. The seam pins what the flags alone do not say: the
+# KV qualification law (the night ladder: 262144/245760 OOM at graph
+# capture; 241664 boots at 732,359 tokens / 3.03x, util 0.98;
+# the bf16-KV trade (double the attention-KV bytes/token, ~14% decode cost,
+# clean cache + 262k windows), the automatic-offload law (the fork plans
+# the tiered-expert offload itself; no flag exists to set), the self-healing
+# law (the run returns before the engine is up, so the serve script must
+# wait on the container for Restart=on-failure to mean anything), the
+# memlock law that makes the pinned host rows legal, the deployer's
+# image/payload gates with the false-green MainPID check, and the measured
+# laws: expert prefetch ABSENT (36% precision), NO speculative decode (the
+# no-spec family measured fastest once the prefetch tax was removed; MTP
+# re-measures pending), gather geometry 32x64 PRESENT (+18.7% decode),
+# MNBT 2048, the CUDA-graph plan term pinned to the measured capture cost, and
+# the client contract (reasoning parser qwen3,
+# tool parser qwen3_xml, preserve_thinking on).

 load helpers

@@ -36,7 +43,22 @@ DOWNLOAD="$REPO_ROOT/scripts/download-frank-primary.sh"
    # dank session's own hard limit is 8 MiB and cannot be raised without
    # restarting user@1000.service).
    grep -q '^WantedBy=multi-user.target$' "$UNIT"
-    grep -q '^Restart=on-failure$' "$UNIT"
+    # Restart=always, not on-failure: a runtime worker OOM makes vLLM's API
+    # server exit CLEAN, so ExecStart returns 0 and on-failure never fires
+    # (measured 2026-10-01 — the endpoint stayed down until a human noticed).
+    grep -q '^Restart=always$' "$UNIT"
+    ! grep -q '^Restart=on-failure$' "$UNIT"
+    # Exiting the engine takes as long as vLLM's drain: ExecStop runs
+    # `podman stop -t 120`, so the stop timeout MUST exceed 120 s or systemd
+    # kills the stop and records 'Failed with result timeout' (measured
+    # 2026-10-01). Guard the relationship, not just the literal.
+    stop_t=$(awk -F= '/^TimeoutStopSec=/{print $2}' "$UNIT")
+    [ -n "$stop_t" ] && [ "$stop_t" -gt 120 ]
+    grep -q '^ExecStop=.*podman stop -t 120' "$UNIT"
+    # StartLimitIntervalSec belongs to [Unit]; a copy in [Service] is an
+    # 'Unknown key' warning systemd logs on every load.
+    [ "$(grep -c '^StartLimitIntervalSec=' "$UNIT")" = "1" ]
+    awk '/^\[Service\]/{in_svc=1} /^StartLimitIntervalSec=/{if(in_svc) exit 1}' "$UNIT"
    grep -q '^RestartSec=' "$UNIT"
    grep -q '^User=dank$' "$UNIT"
    grep -q '^LimitMEMLOCK=infinity$' "$UNIT"
@@ -69,30 +91,249 @@ DOWNLOAD="$REPO_ROOT/scripts/download-frank-primary.sh"
 }

@test "the qualified engine shape is in the env file" {
-    # 2026-09-29 boot probe: MAXLEN 200000 x SEQS 3 (600k total) at KV q8/q8
-    # fits — the engine measured "GPU KV cache size: 729,545 tokens,
-    # Maximum concurrency for 200,000 tokens per request: 3.65x" at
-    # KV_BYTES 7.2 GiB per rank. KV_BYTES is PER RANK: sizing it to the
-    # global requirement OOMs the worker (measured twice). fp8 KV is the
-    # requested q8/q8 (e4m3 for both K and V together). MTP4 spec decode,
-    # util 0.98 (0.99 fails the free-memory wall).
-    grep -q '^FRANK_PRIMARY_MAXLEN=200000$' "$ENV_FILE"
+    # 2026-10-02 shape D. History: the 2026-10-01 night ladder put SEQS=3
+    # bf16 at 241664 (KV 732,359 / 3.03x); SEQS=6 fp8 does NOT fit (6 x
+    # 241664 asked 10.70 GiB against 9.18 GiB free); SEQS=5 fp8 x 262144
+    # BOOTED (1,350,182 / 5.15x) but left ~0 bytes free and a real 995-token
+    # request killed the worker with `Tried to allocate 50.00 MiB ... 0 bytes
+    # free`. UTIL is a REAL arena lever (the fork plans budget = total x util)
+    # but a weak one, and Frank caps it at 0.982 — never 1.00 (see
+    # FRANK_PRIMARY_UTIL for the two ceiling deaths that justify the margin).
+    # SEQS=2 is IMPOSSIBLE on this model, measured 2026-10-02 by
+    # single-variable boot probe against the then-deployed shape: SEQS 2 fails
+    # at BOTH MNBT 1024 and 2048 with torch.fx ConstraintViolationError
+    # (query_start_loc.size()[0] marked dynamic but specialized to a constant
+    # (3)); SEQS 3 passes at both. model_runner sizes that buffer
+    # max_num_reqs + 1, so the width is SEQS + 1 and MNBT only moves the
+    # compile RANGE.
+    # So the shape is 3 slots with a pool that admits only ~2 full-length
+    # conversations: measured live, KV_BYTES 3.15e9/rank reports 398,658
+    # tokens = 1.99x for a 200704-token request (7,892 B/token at THIS shape;
+    # the per-token price is shape-dependent, 7,701 at 262144). Shorter
+    # requests use the spare blocks; a third LONG request queues.
+    # The arena is sized by max_model_len, NOT by KV_BYTES: the planner
+    # reserves registry 9190 B/token x max_model_len (2.48 GiB at 262144 ->
+    # arena 6,248 slots; 1.91 GiB at 200704 -> 6,496). The pool cut buys
+    # runtime headroom, not arena.
+    grep -q '^FRANK_PRIMARY_MAXLEN=200704$' "$ENV_FILE"
    grep -q '^FRANK_PRIMARY_SEQS=3$' "$ENV_FILE"
-    grep -q '^FRANK_PRIMARY_KV_BYTES=7730934784$' "$ENV_FILE"
+    grep -q '^FRANK_PRIMARY_UTIL=0.982$' "$ENV_FILE"
+    grep -q '^FRANK_PRIMARY_KV_BYTES=3166500000$' "$ENV_FILE"
+    grep -q '^FRANK_PRIMARY_MNBT=2048$' "$ENV_FILE"
    grep -q '^FRANK_PRIMARY_KV_DTYPE=fp8$' "$ENV_FILE"
-    grep -q '^FRANK_PRIMARY_SPEC=4$' "$ENV_FILE"
-    grep -q '^FRANK_PRIMARY_UTIL=0.98$' "$ENV_FILE"
+    # --expert-precise-memory IS set (shape E, 2026-10-02). It was refused
+    # while the 2 x 262144 bf16 boot sat 0.13 GiB under the 31.86 GiB ceiling
+    # (arena 16.65 device-resident, NRestarts=3 at ~6.7 min each) — there it was
+    # the last 2.35 GiB over the line, not the cause. Probed on the shape-D base
+    # with production stopped: boots healthy in 420 s, plan reports runtime
+    # overhead 5.25 GiB (not 7.60), expert cache 17.22 GiB / 7,522 slots
+    # (+1,026 / +16% over 6,496). The pool is NOT cut with it (option A): the
+    # pool is not an arena lever, so 3.15e9 keeps the measured 1.99x.
+    grep -q '^FRANK_PRIMARY_PRECISE_MEMORY=1$' "$ENV_FILE"
+    # GPU_MAX_HW_QUEUES caps HSA hw queues per GPU; real knob (the literal is in
+    # libhsa-runtime64.so), passed inside the container as a bare name — the env
+    # file keeps only the prefixed form.
+    grep -q '^FRANK_PRIMARY_GPU_MAX_HW_QUEUES=2$' "$ENV_FILE"
+    # The compile cache is a BOOT-COST fix: the image ships no /root/.cache at
+    # all, our boot writes 1.1 GB of vLLM AOT artifacts (4,665 files) plus
+    # 229 MB of comgr JIT kernels into the ephemeral layer, and every
+    # `podman rm -f` throws them away — re-paying `torch.compile took 181.20 s`
+    # (173.19 s of it one graph) on every deploy, restart and reboot.
+    grep -q '^FRANK_PRIMARY_CACHE_DIR=/home/dank/.cache/frank-primary$' "$ENV_FILE"
+    grep -q '^FRANK_PRIMARY_GATHER_CHUNKS=32$' "$ENV_FILE"
+    grep -q '^FRANK_PRIMARY_GATHER_LANES=64$' "$ENV_FILE"
+    # Prefix caching is ON (reverted 2026-10-05 the same day it was
+    # disabled): the fork's QFN align-mode GDN state fault (48 gdn_hip
+    # state faults, slot 42 heads 3/8, at the 13th 3136-token boundary =
+    # 40,768 — the "~40k corruption" report) was mitigated with
+    # --no-enable-prefix-caching, but live long conversations STILL
+    # degraded at ~40k with caching off and zero faults logged — cost
+    # with no fix, so the historical flag returned. The knob stays
+    # commented in the env file for a future fixed fork build; flip it to
+    # 1 only together with a re-verification past 40,768 tokens.
+    grep -qE '^# FRANK_PRIMARY_NO_PREFIX_CACHING=1$' "$ENV_FILE"
+    grep -q '^FRANK_PRIMARY_MODEL_SHARDS=131$' "$ENV_FILE"
 }

@test "the serve script wires the engine flags from the env" {
-    grep -q -- '--kv-cache-dtype "$FRANK_PRIMARY_KV_DTYPE"' "$SERVE"
-    grep -q -- '--kv-cache-memory-bytes "$FRANK_PRIMARY_KV_BYTES"' "$SERVE"
    grep -q -- '--max-num-seqs "$FRANK_PRIMARY_SEQS"' "$SERVE"
    grep -q -- '--max-model-len "$FRANK_PRIMARY_MAXLEN"' "$SERVE"
+    grep -q -- '--kv-cache-memory-bytes "$FRANK_PRIMARY_KV_BYTES"' "$SERVE"
+    grep -q -- '--kv-cache-dtype "$FRANK_PRIMARY_KV_DTYPE"' "$SERVE"
+    grep -q -- '--max-num-batched-tokens "$FRANK_PRIMARY_MNBT"' "$SERVE"
+    grep -q -- '-e R4D_LRU_GATHER_CHUNKS="$FRANK_PRIMARY_GATHER_CHUNKS"' "$SERVE"
+    grep -q -- '-e R4D_LRU_GATHER_LANES="$FRANK_PRIMARY_GATHER_LANES"' "$SERVE"
+    # Speculative decode, the cudagraph capture ladder and the sampling
+    # defaults are all ENV-DRIVEN knobs (unset -> the flag is absent, which is
+    # the shipped shape today). JSON values carry no shell-special characters
+    # that this quoting cannot carry.
+    grep -q -- '${FRANK_PRIMARY_SPEC:+--speculative-config "$FRANK_PRIMARY_SPEC"}' "$SERVE"
+    grep -q -- '${FRANK_PRIMARY_COMPCONFIG:+--compilation-config "$FRANK_PRIMARY_COMPCONFIG"}' "$SERVE"
+    grep -q -- '${FRANK_PRIMARY_GEN_CONFIG:+--override-generation-config "$FRANK_PRIMARY_GEN_CONFIG"}' "$SERVE"
    grep -q -- '--tensor-parallel-size 2' "$SERVE"
-    grep -q 'num_speculative_tokens' "$SERVE"
-    grep -q -- '--enable-prefix-caching' "$SERVE"
+    # Prefix caching renders exactly ONE flag from the knob, branched
+    # explicitly in the serve script (the ${VAR:--flag} double-expansion is
+    # ambiguous bash — when set it parsed as a substring offset, measured
+    # 2026-10-05): set -> --no-enable-prefix-caching (the fork-named
+    # mitigation, tried and REVERTED the same day — live corruption
+    # persisted at ~40k with zero faults), unset -> the shipped
+    # --enable-prefix-caching. Both directions are spelled out so a fork
+    # default change cannot silently flip the contract.
+    grep -qF 'FRANK_PRIMARY_PREFIX_CACHE_FLAG=--no-enable-prefix-caching' "$SERVE"
+    grep -qF 'FRANK_PRIMARY_PREFIX_CACHE_FLAG=--enable-prefix-caching' "$SERVE"
+    grep -qF '"$FRANK_PRIMARY_PREFIX_CACHE_FLAG"' "$SERVE"
+    # The observable contract: the exact flag word the engine receives,
+    # rendered from the shipped env through the serve script's branch (the
+    # same render-assert pattern the spec knobs use above).
+    run bash -c '
+        set -a; . "$1" >/dev/null; set +a
+        if [ -n "${FRANK_PRIMARY_NO_PREFIX_CACHING:-}" ]; then
+            FRANK_PRIMARY_PREFIX_CACHE_FLAG=--no-enable-prefix-caching
+        else
+            FRANK_PRIMARY_PREFIX_CACHE_FLAG=--enable-prefix-caching
+        fi
+        printf %s "$FRANK_PRIMARY_PREFIX_CACHE_FLAG"' _ "$ENV_FILE"
+    [ "$output" = "--enable-prefix-caching" ]
    grep -q -- '--trust-remote-code' "$SERVE"
+    # DCP is opt-in: the flag is passed only when FRANK_PRIMARY_DCP is set
+    # (DCP reuses the TP ranks when PCP is off, so TP=2 + DCP=2 fits two GPUs).
+    grep -q -- '--decode-context-parallel-size "$FRANK_PRIMARY_DCP"' "$SERVE"
+    ! grep -q '^FRANK_PRIMARY_DCP=' "$ENV_FILE"
+    # --expert-precise-memory selects the fitted overhead (5.25 GiB) over the
+    # default (7.60 GiB = fitted + compile/graph-capture headroom), which only
+    # moves the plan's `room` term and therefore the expert arena: ~1,025
+    # more slots at 2.344 MiB each.
+    grep -q -- '${FRANK_PRIMARY_PRECISE_MEMORY:+--expert-precise-memory}' "$SERVE"
+    # Shape E host/CPU wiring: the cache dir is created before it is mounted
+    # (a cold boot must have a target) and the mount lands on /root/.cache,
+    # whose vllm/ child is the path VLLM_CACHE_ROOT defaults to — the engine log
+    # names it as `Using cache directory: /root/.cache/vllm/torch_compile_cache/…`.
+    grep -q -- 'mkdir -p "$CACHE_DIR"' "$SERVE"
+    grep -q -- '${CACHE_DIR:+-v "$CACHE_DIR":/root/.cache}' "$SERVE"
+    grep -q -- '-e GPU_MAX_HW_QUEUES="$FRANK_PRIMARY_GPU_MAX_HW_QUEUES"' "$SERVE"
+    # Shape F/G: the CUDA-graph plan term is carried as CONTAINER env (the fork
+    # reads it at call time inside estimate_cudagraph_memory), and both must be
+    # present — the graph term and UTIL are the ONLY plan terms this repo can
+    # move, and shrinking them is what grows the arena. The base must be shipped
+    # explicitly: unset, the estimator would fall back to its 256 MiB default.
+    grep -q -- '-e VLLM_CUDAGRAPH_MEM_BASE_MB="$FRANK_PRIMARY_CUDAGRAPH_BASE_MB"' "$SERVE"
+    grep -q -- '-e VLLM_CUDAGRAPH_MEM_PER_GRAPH_MB="$FRANK_PRIMARY_CUDAGRAPH_PER_GRAPH_MB"' "$SERVE"
+    grep -q '^FRANK_PRIMARY_CUDAGRAPH_BASE_MB=0$' "$ENV_FILE"
+    grep -q '^FRANK_PRIMARY_CUDAGRAPH_PER_GRAPH_MB=12$' "$ENV_FILE"
+    # The host region is the ONE plan term that autofits per-boot: with no cap
+    # the bound is MemAvailable at that moment (boot 2026-10-05: 237.97 GiB ->
+    # 159.39 GiB usable, while the running host later reads 75.4 GiB available).
+    # An explicit --expert-offload-mem cap makes the plan deterministic whatever
+    # the co-tenant RAM state is at boot; 120 GiB covers the measured 105.48
+    # GiB of swappable experts (52.74/rank x 2) with 14% headroom.
+    grep -q -- '${FRANK_PRIMARY_EXPERT_OFFLOAD_MEM:+--expert-offload-mem "$FRANK_PRIMARY_EXPERT_OFFLOAD_MEM"}' "$SERVE"
+    grep -q '^FRANK_PRIMARY_EXPERT_OFFLOAD_MEM=120$' "$ENV_FILE"
+}
+
+@test "tools and reasoning are wired — the client contract is served" {
+    # 2026-10-01: the rollback-era trio served NO tool dialect and left
+    # thinking inline in `content` with a dangling `</think>` and a null
+    # `reasoning` field (measured: content 'We need answer...Brief.\n</think>
+    # \n\n391') — the malformed thinking block clients choke on. The working
+    # values are the ones ecaad36 shipped for this model and this fork's
+    # registries confirm: reasoning parser `qwen3`, tool parser `qwen3_xml`.
+    # preserve_thinking keeps prior thinking blocks across turns.
+    grep -q '^FRANK_PRIMARY_REASONING_PARSER=qwen3$' "$ENV_FILE"
+    grep -q '^FRANK_PRIMARY_TOOL_PARSER=qwen3_xml$' "$ENV_FILE"
+    grep -q -- '--reasoning-parser "$FRANK_PRIMARY_REASONING_PARSER"' "$SERVE"
+    grep -q -- '--enable-auto-tool-choice' "$SERVE"
+    grep -q -- '--tool-call-parser "$FRANK_PRIMARY_TOOL_PARSER"' "$SERVE"
+    grep -q -- '--default-chat-template-kwargs "$FRANK_PRIMARY_CHAT_TEMPLATE_KWARGS"' "$SERVE"
+    grep -q 'preserve_thinking' "$ENV_FILE"
+    grep -q 'enable_thinking' "$ENV_FILE"
+}
+
+@test "expert offload is AUTOMATIC — the payload cannot fit VRAM and no flag sets it" {
+    # ~86 GiB/rank of FP8 trunk against 32 GiB of VRAM: the fork's tiered-
+    # expert planner moves the overflow to pinned host rows by itself (the
+    # arena 6,574 slots x 2.344 MiB plus host rows). There is no
+    # --enable-expert-offload flag in this arm; the legality of the pin is
+    # the unit's unbounded LimitMEMLOCK.
+    ! grep -q -- '--enable-expert-offload' "$SERVE"
+    grep -q '^LimitMEMLOCK=infinity$' "$UNIT"
+    grep -q -- '--ipc=host --ulimit memlock=-1:-1' "$SERVE"
+}
+
+@test "speculative decode is an ENV KNOB — the old retirement verdict was invalid" {
+    # The 2026-10-02 c1248d4 retirement of MTP compared the WRONG PAIR:
+    # fp8+MTP4 (37.4-40.6 tok/s, 54% acceptance) against the bf16 no-spec boot
+    # (36.4-39.4), then bf16 proved unbootable and fp8 came back WITHOUT MTP —
+    # so fp8+spec was never compared against fp8+no-spec on the same shape. The
+    # no-spec incumbent runs ~30 tok/s, under the fleet's 35 tok/s target, while
+    # EVERY vendor flash-next recipe ships MTP3, and our MTP4 run used the
+    # default GREEDY draft sampling (probabilistic is the other fork option and
+    # is what the recipes use under temp>0). So spec is a knob, not a law: the
+    # shipped value is whatever a matched-pair re-measurement wins.
+    grep -q 'THE RETIREMENT VERDICT WAS INVALID' "$ENV_FILE"
+    # The knobs are empty in the shipped env, so the `${VAR:+…}` wiring renders
+    # NO spec/compilation flag — asserted by rendering, not by grepping the
+    # assignment (shape E today has neither flag in its argv).
+    run bash -c 'set -a; . "$1" >/dev/null; set +a
+        printf %s "${FRANK_PRIMARY_SPEC:+--speculative-config $FRANK_PRIMARY_SPEC}${FRANK_PRIMARY_COMPCONFIG:+--compilation-config $FRANK_PRIMARY_COMPCONFIG}"' _ "$ENV_FILE"
+    [ -z "$output" ]
+    # The capture-ladder law: the piecewise graph ladder must contain
+    # num_speculative_tokens + 1, so an MTP3 shape captures [4, 8, 12] and the
+    # 2026-10-02 SEQS=2 failure (query_start_loc specialized to the constant 3)
+    # is why SEQS stays a multiple-compatible value.
+    grep -q 'num_speculative_tokens + 1' "$ENV_FILE"
+}
+
+@test "KV dtype is fp8 — the incumbent, and the only one that boots" {
+    # fp8 costs ~7,701 B/token at 262144 (7,892 at 200704 — shape-dependent),
+    # decomposed as 6,138 attention + 1,563 mamba/side (dtype-independent);
+    # bf16 costs 13,839 (12,276 + 1,563).
+    # bf16 was abandoned 2026-10-01: its 2 x 262144 boot died with 0 bytes
+    # free on a real request, and the SEQS=2 slot count it was designed
+    # around cannot compile this model at all (see the shape test above).
+    grep -q '^FRANK_PRIMARY_KV_DTYPE=fp8$' "$ENV_FILE"
+    grep -q -- '--kv-cache-dtype "$FRANK_PRIMARY_KV_DTYPE"' "$SERVE"
+}
+
+@test "gather geometry 32x64 is PRESENT BY MEASUREMENT" {
+    # R4D_LRU_GATHER_CHUNKS=32 / R4D_LRU_GATHER_LANES=64: the overnight
+    # sweep's biggest lever, +18.7% decode vs default geometry, and the win
+    # GROWS with depth (39.5@166 -> 54.1@1087 -> 49.3@4064 in the fp8 era).
+    # g64x128 is past the peak; g8x16 mild. They are container ENV (the
+    # fork's knobs), not engine flags — passed through with -e.
+    grep -q '^FRANK_PRIMARY_GATHER_CHUNKS=32$' "$ENV_FILE"
+    grep -q '^FRANK_PRIMARY_GATHER_LANES=64$' "$ENV_FILE"
+    grep -q -- '-e R4D_LRU_GATHER_CHUNKS="$FRANK_PRIMARY_GATHER_CHUNKS"' "$SERVE"
+}
+
+@test "the arena sits just under the ceiling and the KV ask is honoured" {
+    # The planner sizes expert cache as budget - non-expert - KV reserve -
+    # graphs - overhead, and two of those five terms are reachable from a
+    # deploy: budget (= device_total x util) and graphs (VLLM_CUDAGRAPH_MEM_*).
+    # UTIL has a HARD LAW, not a field cap: 0.982, never 1.00. The 1.00 attempt
+    # is UNBOOTABLE on this fork — worker init raises
+    # `ValueError: Free memory on device cuda:N (31.29/31.86 GiB ...)`
+    # on both ranks because the worker demands free > budget and budget ==
+    # total — and even if it booted, 0.13-GiB-margin shapes have died under
+    # real load. So the arena tops out at 0.982 budget (31.29 GiB).
+    # Graphs: charge strictly below the fork's analytic 563 MiB default while
+    # still paying per graph (the measured post-capture delta is -0.06 GiB, i.e.
+    # the base pool is the over-charge, but a future spec-decode shape captures
+    # 18 graphs and must still be charged).
+    # MNBT stays 2048: 512 doubles TTFT at 4064 ctx, 1024-4096 are decode-
+    # interchangeable AND arena-identical (the registry prices overhead flat up
+    # to its nbt 4096 anchor), 12288 costs 2.5% plus arena.
+    # The KV ask: the pool is priced at the engine's measured 7,892 B/token at
+    # this shape, and a served conversation needs 200,704 tokens, so the pool
+    # must clear 400,700 tokens to keep two full-length resident.
+    grep -q '^FRANK_PRIMARY_MNBT=2048$' "$ENV_FILE"
+    grep -q -- '--max-num-batched-tokens "$FRANK_PRIMARY_MNBT"' "$SERVE"
+    grep -q '^FRANK_PRIMARY_UTIL=0.982$' "$ENV_FILE"
+    base=$(awk -F= '/^FRANK_PRIMARY_CUDAGRAPH_BASE_MB=/{print $2}' "$ENV_FILE")
+    per_graph=$(awk -F= '/^FRANK_PRIMARY_CUDAGRAPH_PER_GRAPH_MB=/{print $2}' "$ENV_FILE")
+    [ "$base" -eq 0 ] && [ "$per_graph" -gt 0 ]
+    [ $((base + 6 * per_graph)) -lt 563 ]
+    kv=$(awk -F= '/^FRANK_PRIMARY_KV_BYTES=/{print $2}' "$ENV_FILE")
+    [ $((kv / 7892)) -ge 400700 ]
 }

@test "expert prefetch is absent BY LAW, not by omission" {
@@ -174,6 +415,7 @@ DOWNLOAD="$REPO_ROOT/scripts/download-frank-primary.sh"
    grep -q 'FRANK_PRIMARY_MODEL_SHARDS' "$REMOTE_DEPLOY"
    grep -q 'config.json' "$REMOTE_DEPLOY"
    grep -q 'qwen-fp8' "$REMOTE_DEPLOY"
+    grep -q 'glm-probe' "$REMOTE_DEPLOY"
    grep -q 'wait_port_free 7770' "$REMOTE_DEPLOY"
    grep -q 'relink_unit frank-primary' "$REMOTE_DEPLOY"
    grep -q 'systemctl enable frank-primary.service' "$REMOTE_DEPLOY"
@@ -1,12 +1,13 @@
 #!/usr/bin/env bats
-# Seam: the frank-small mainline-llama.cpp root unit on dankclaw.
-# frank-small is the RTX 3090 endpoint created on 2026-09-16, when frank-large
-# went back to the halogen engine on the iGPU. This seam pins what the flags
-# alone do not say: boot persistence for a NATIVE runtime (no generated quadlet
-# is involved), the repo-relative paths the unit executes, the separate client
-# port, the MTP contract (the head is a separate sidecar file loaded as the
-# draft model, chained behind ngram-mod), the vision handoff from frank-large,
-# and the KV budget: q8_0 quantized KV whose context --fit sizes to the card.
+# Seam: the frank-small llama.cpp root unit on dankclaw.
+# frank-small is the RTX 3090 endpoint; since 2026-10-05 it serves
+# ornith-ai/Ornith-1.5-9B-GGUF:Q8_0 via a CUDA (sm_86) build of
+# llama.cpp master, chosen because it is the only runtime here that moves K and
+# V independently. Since 2026-10-03 it runs TWO unquantized contexts of
+# 200,704 tokens each and carries a vision projector, and the card must keep
+# 2-3 GiB free. This seam runs the serve script against a stub engine and
+# asserts on the argv it actually execs, so what is pinned is the
+# command the operator gets.

 load helpers

@@ -15,6 +16,54 @@ SERVE="$REPO_ROOT/scripts/serve-frank-small.sh"
 ENV_FILE="$REPO_ROOT/scripts/frank-small.env"
 REMOTE_DEPLOY="$REPO_ROOT/scripts/deploy-remote.sh"
 ENV_LARGE="$REPO_ROOT/scripts/frank-large.env"
+TARGET_STUB="Ornith-1.5-9B-Q8_0.gguf"
+# Same basename as the shipped projector: the test env repoints it into $STAGE.
+MM_STUB="mmproj-Ornith-1.5-9B-BF16.gguf"
+
+setup() {
+    STAGE="$(mktemp -d)"
+    RUNTIME_DIR="$STAGE/llama"
+    MODEL_DIR="$STAGE/models"
+    mkdir -p "$RUNTIME_DIR/bin" "$MODEL_DIR"
+    cat > "$RUNTIME_DIR/bin/llama-server" <<'STUB'
+#!/usr/bin/env bash
+printf '%s\n' "$@"
+STUB
+    chmod +x "$RUNTIME_DIR/bin/llama-server" "$RUNTIME_DIR/bin" 2>/dev/null || chmod +x "$RUNTIME_DIR/bin/llama-server"
+    touch "$MODEL_DIR/$TARGET_STUB" "$MODEL_DIR/$MM_STUB"
+}
+
+teardown() {
+    rm -rf "$STAGE"
+}
+
+# serve_argv [env-file] — run the serve script and capture the engine's
+# argv, one token per line, in $output.
+serve_argv() {
+    local envf="${1:-$STAGE/test.env}"
+    local mmproj
+    mmproj="$(sed -nE 's/^MMPROJ_FILE=(.+)$/\1/p' "$ENV_FILE")"
+    cat > "$envf" <<EOF
+LLAMA_BIN=$RUNTIME_DIR/bin/llama-server
+MODEL_FILE=$MODEL_DIR/$TARGET_STUB
+MMPROJ_FILE=$MODEL_DIR/${mmproj##*/}
+ALIAS=frank-small
+BIND_ADDR=0.0.0.0
+LISTEN_PORT=7771
+CTX_SIZE=$(sed -nE 's/^CTX_SIZE=([0-9]+)$/\1/p' "$ENV_FILE")
+PARALLEL=$(sed -nE 's/^PARALLEL=([0-9]+)$/\1/p' "$ENV_FILE")
+BATCH_SIZE=$(sed -nE 's/^BATCH_SIZE=([0-9]+)$/\1/p' "$ENV_FILE")
+UBATCH_SIZE=$(sed -nE 's/^UBATCH_SIZE=([0-9]+)$/\1/p' "$ENV_FILE")
+CACHE_TYPE_K=$(sed -nE 's/^CACHE_TYPE_K=(.+)$/\1/p' "$ENV_FILE")
+CACHE_TYPE_V=$(sed -nE 's/^CACHE_TYPE_V=(.+)$/\1/p' "$ENV_FILE")
+NGL=$(sed -nE 's/^NGL=([0-9]+)$/\1/p' "$ENV_FILE")
+EOF
+    run env FRANK_SMALL_ENV_FILE="$envf" nvidia-modprobe=true bash "$SERVE"
+}
+
+argv_value() {
+    grep -A1 -xF -- "$1" <<<"$output" | tail -1
+}

@test "frank-small unit, serve script and env exist in scripts/" {
    [ -f "$UNIT" ]
@@ -28,94 +77,178 @@ ENV_LARGE="$REPO_ROOT/scripts/frank-large.env"
    grep -q '^RestartSec=' "$UNIT"
    grep -q '^User=dank$' "$UNIT"
    grep -q '^LimitMEMLOCK=infinity$' "$UNIT"
-}
-
-@test "frank-small runs the repo's own serve script with the repo's env" {
    grep -q '^ExecStart=/home/dank/serverAppConfigs/scripts/serve-frank-small.sh$' "$UNIT"
    grep -q 'FRANK_SMALL_ENV_FILE=/home/dank/serverAppConfigs/scripts/frank-small.env' "$UNIT"
 }

-@test "frank-small offloads every layer to the 3090 with flash attention" {
-    grep -q 'llama.cpp-mainline/build-cuda/bin/llama-server' "$ENV_FILE"
-    grep -q '^NGL=999$' "$ENV_FILE"
-    grep -q -- '--gpu-layers "$NGL"' "$SERVE"
-    grep -q '^FLASH_ATTN=on$' "$ENV_FILE"
-    grep -q -- '--flash-attn "$FLASH_ATTN"' "$SERVE"
-    # Prometheus scrapes this endpoint; without --metrics there is no /metrics.
-    grep -q -- '--metrics' "$SERVE"
+@test "the shipped env points at the llama.cpp runtime and the Ornith-1.5-9B Q8_0 GGUF" {
+    grep -qE '^LLAMA_BIN=/home/dank/GitPackages/llama-master/build-cuda/bin/llama-server$' "$ENV_FILE"
+    grep -qE "^MODEL_FILE=/home/dank/models/Ornith-Q8/$TARGET_STUB$" "$ENV_FILE"
 }

-@test "frank-small caps its CPU RAM cache below the build default" {
-    # Without the flag the 8192 MiB default returns and the env cap is a
-    # silent no-op; this memory-tight host needs the cap to hold.
-    grep -q -- '--cache-ram "$CACHE_RAM"' "$SERVE"
+@test "the deployer gates on exactly the payload paths the env file ships" {
+    # Anti-drift: the deploy handler checks files by its own defaults. If
+    # those ever differ from what the unit actually loads, a green deploy
+    # would be gating on a file nobody reads — and a missing projector would
+    # ship a blind endpoint.
+    local env_v dep_v path pair
+    for pair in "MODEL_FILE LLM_SMALL_MODEL_FILE" "MMPROJ_FILE LLM_SMALL_MMPROJ"; do
+        read -r env_v dep_v <<<"$pair"
+        path="$(sed -nE "s/^${env_v}=(.+)\$/\1/p" "$ENV_FILE" | tail -1)"
+        [ -n "$path" ]
+        grep -qF -- "${dep_v}=\"\${${dep_v}:-${path}}\"" "$REMOTE_DEPLOY"
+    done
 }

-@test "frank-small drafts with the MTP sidecar chained behind ngram-mod" {
-    grep -q '^SPEC_TYPE=draft-mtp,ngram-mod$' "$ENV_FILE"
-    grep -q -- '--spec-type "$SPEC_TYPE"' "$SERVE"
-    # The Gemma4 MTP head is a separate assistant file (gemma4-assistant, 4
-    # nextn layers); it loads as --model-draft and shares the target's last
-    # layers' KV. ngram-mod is draftless, so the chain is legal.
-    grep -q '^DRAFT_FILE=mtp-gemma-4-12B-it\.gguf$' "$ENV_FILE"
-    grep -q -- '--model-draft "$MODEL_DIR/$DRAFT_FILE"' "$SERVE"
+@test "the shipped env serves the fleet shape: one native 262144-token context" {
+    serve_argv
+    [ "$status" -eq 0 ]
+    [ "$(argv_value --alias)" = "frank-small" ]
+    [ "$(argv_value --port)" = "7771" ]
+    [ "$(argv_value --host)" = "0.0.0.0" ]
+    [ "$(argv_value --model)" = "$MODEL_DIR/$TARGET_STUB" ]
+    [ "$(argv_value --n-gpu-layers)" -gt 0 ]
+    # llama.cpp divides --ctx-size across slots and a mamba hybrid cannot
+    # share one cache, so the observable contract is the per-slot window.
+    # 2026-10-05 (Frank): NO KV quantization forces ONE slot at the NATIVE
+    # window — Ornith's 9 full-attn layers price f16 KV at 36 KiB/token, and
+    # two 200,704 slots cannot fit the 2-3 GiB floor on 24 GiB with Q8
+    # weights. The window per conversation ROSE 200,704 -> 262,144.
+    [ "$(argv_value --parallel)" -eq 1 ]
+    ctx="$(argv_value --ctx-size)"
+    par="$(argv_value --parallel)"
+    [ $((ctx % par)) -eq 0 ]
+    [ $((ctx / par)) -eq 262144 ]
 }

-@test "frank-small carries the fleet's image path, and only it" {
-    grep -q '^MMPROJ_PATH=.*mmproj-Gemma4-12B-QAT-Uncensored-HauhauCS-Balanced-BF16\.gguf$' "$ENV_FILE"
-    grep -q -- '--mmproj "$MMPROJ_PATH"' "$SERVE"
-    ! grep -q '^ENABLE_VISION=1$' "$ENV_LARGE"
+@test "the KV contract is unquantized both halves, flash-attn on" {
+    serve_argv
+    [ "$status" -eq 0 ]
+    [ "$(argv_value --cache-type-k)" = "f16" ]
+    # Frank, 2026-10-05: NO KV quantization on this endpoint. The Q8_0 rung
+    # funds itself by UBATCH first (2048 -> 1024), then pool steps — a q8_0-V
+    # proposal was written and overruled. The flag pair is what makes this
+    # observable — a build without the FA path silently falls back, so both
+    # must appear in the argv the supervisor runs.
+    [ "$(argv_value --cache-type-v)" = "f16" ]
+    [ "$(argv_value --flash-attn)" = "on" ]
 }

-@test "frank-small's slot grid is full-window and fits the 24 GiB card" {
-    local parallel cache_k cache_v ctx fixed per_slot
-    parallel="$(sed -nE 's/^PARALLEL=([0-9]+)$/\1/p' "$ENV_FILE")"
-    cache_k="$(sed -nE 's/^CACHE_TYPE_K=(.+)$/\1/p' "$ENV_FILE")"
-    cache_v="$(sed -nE 's/^CACHE_TYPE_V=(.+)$/\1/p' "$ENV_FILE")"
+@test "the KV budget leaves 2-3 GiB of the 3090 free" {
+    # Rate, from the Ornith GGUF header (read from the Hub file, 2026-10-05):
+    # NINE full-attention layers (block_count 33, full_attention_interval 4)
+    # x 4 kv heads x 256 dim x 2 B = 18 KiB/token for K and the same for V
+    # => 36 KiB/token (budgeted 37 with slack; MiMo measured 24.5 — one
+    # fewer layer AND one fewer KV head). The non-KV half: Q8_0 weights
+    # 9,334 MiB (9,786,060,384 B) + vision tower ~1.3 GiB (mmproj 879 MiB at
+    # the measured ~1.5x cost law) + DeltaNet state + compute buffers +
+    # CUDA graphs => 11 GiB budgeted. Two asserts, both about the output
+    # the operator sees: the pool must still fit with 2 GiB spare, and it
+    # must be large enough that 5 GiB spare is NOT reachable (an undersized
+    # pool is as wrong as an oversized one).
+    # Arithmetic, not a boot log: change CTX_SIZE and this bites before the
+    # deploy does. At Ornith's rate the two-200704-slot pool (14.1 GiB KV)
+    # FAILS the first assert — which is exactly why the shipped shape is
+    # ONE native slot.
    ctx="$(sed -nE 's/^CTX_SIZE=([0-9]+)$/\1/p' "$ENV_FILE")"
-    [ -n "$parallel" ] && [ -n "$cache_k" ] && [ -n "$cache_v" ] && [ -n "$ctx" ]
-    # Every slot must be the model's FULL trained window (262144), not a
-    # divided one -- CTX_SIZE is the total pool and --parallel divides it.
-    [ $((ctx / parallel)) -eq 262144 ]
-    [ $((ctx % parallel)) -eq 0 ]
-    [ "$cache_k" = "q8_0" ] && [ "$cache_v" = "q8_0" ]
-    # Fit budget from the measured boot arithmetic on the 3090 (24576
-    # MiB): one full-context slot idles at 11,996 MiB, four at 22,346,
-    # so each slot costs (22346-11996)/3 = 3450 MiB and the fixed part
-    # (weights, projector, sidecar draft, compute buffers, CUDA context)
-    # is 11996-3450 = 8546 MiB. If the config asks for more slots than
-    # the card holds, this inequality fails before deploy instead of the
-    # KV allocation failing at boot.
-    fixed=8546
-    per_slot=3450
-    [ $((fixed + parallel * per_slot)) -le 24576 ]
-    # Quantized KV on CUDA needs the GGML_CUDA_FA_ALL_QUANTS build option,
-    # without which prefill collapses about 12x (see the CUDA lesson in
-    # AGENTS.md). The build-cuda tree must carry it.
-    grep -q '^FLASH_ATTN=on$' "$ENV_FILE"
+    kv_gib=$((ctx * 37 / 1048576))
+    [ $((kv_gib + 11 + 2)) -le 24 ]
+    [ $((kv_gib + 11 + 5)) -gt 24 ]
+    # And the ceiling that sets the per-slot window: two NATIVE windows cannot
+    # be paid for in full precision on this card.
+    native=$((2 * 262144))
+    [ $((native * 37 / 1048576 + 11)) -gt 24 ]
 }

-@test "frank-small runs on its own port beside frank-large's" {
-    grep -q '^LISTEN_PORT=7771$' "$ENV_FILE"
-    grep -q '^ALIAS=frank-small$' "$ENV_FILE"
-    grep -q -- '--port "$LISTEN_PORT"' "$SERVE"
-    grep -q -- '--host "$BIND_ADDR"' "$SERVE"
-    # The two endpoints cannot share a port: frank-large holds 7770, this one
-    # 7771.
-    grep -q '^PORT=7770$' "$ENV_LARGE"
-    ! grep -q '^PORT=7771$' "$ENV_LARGE"
+@test "the compute-buffer pair follows the env file and stays self-consistent" {
+    # This endpoint ran on llama.cpp's own defaults until 2026-10-03, so the
+    # contract is: whatever rung the repo carries is the rung the engine is
+    # exec'd with — the number a benchmark attributes a prefill rate to must be
+    # a repo value, never an engine default nobody wrote down. Asserted against
+    # the env rather than a literal so a retune moves one file, not two.
+    local env_batch env_ubatch
+    env_batch="$(sed -nE 's/^BATCH_SIZE=([0-9]+)$/\1/p' "$ENV_FILE")"
+    env_ubatch="$(sed -nE 's/^UBATCH_SIZE=([0-9]+)$/\1/p' "$ENV_FILE")"
+    serve_argv
+    [ "$status" -eq 0 ]
+    [ "$(argv_value --batch-size)" = "$env_batch" ]
+    [ "$(argv_value --ubatch-size)" = "$env_ubatch" ]
+    # llama.cpp refuses ubatch > batch, so a retune that inverts the pair has
+    # to fail here instead of at boot on a live endpoint.
+    [ "$env_batch" -ge "$env_ubatch" ]
+
+    # And an env that names neither flag still ships an explicit pair: the
+    # serve script's fallbacks are written-down values, not blanks the engine
+    # fills in silently.
+    sed -i '/^BATCH_SIZE=/d;/^UBATCH_SIZE=/d' "$STAGE/test.env"
+    serve_argv
+    [ "$status" -eq 0 ]
+    argv_value --batch-size | grep -qE '^[0-9]+$'
+    argv_value --ubatch-size | grep -qE '^[0-9]+$'
 }

-@test "the deploy handler installs the root unit and refuses a missing model" {
+@test "no speculative draft ships by default" {
+    # The qwen35 GGUF has no MTP tensors, and the cross-model DFlash draft
+    # is a measured decision the harness makes later, not a default.
+    ! grep -q -- '--model-draft' "$SERVE"
+    ! grep -q -- '--spec-type' "$SERVE"
+    ! grep -q '^DRAFT' "$ENV_FILE"
+}
+
+@test "vision ships by default and MMPROJ_FILE empty opts out of the image path" {
+    # The Q6_K GGUF carries no `v.*` tensors, so the projector argument IS
+    # the vision capability: no flag, no eyes.
+    serve_argv
+    [ "$status" -eq 0 ]
+    [ "$(argv_value --mmproj)" = "$MODEL_DIR/$MM_STUB" ]
+
+    # The documented opt-out must reach the engine as ABSENCE of the flag,
+    # not as an argument pointing at an empty string.
+    sed -i 's|^MMPROJ_FILE=.*|MMPROJ_FILE=|' "$STAGE/test.env"
+    run env FRANK_SMALL_ENV_FILE="$STAGE/test.env" nvidia-modprobe=true bash "$SERVE"
+    [ "$status" -eq 0 ]
+    ! grep -qxF -- '--mmproj' <<<"$output"
+}
+
+@test "the fleet reasoning contract survives the reslice to two wide contexts" {
+    serve_argv
+    [ "$status" -eq 0 ]
+    # deepseek format = thinking lands in reasoning_content, the field the
+    # harness reads on this endpoint; --jinja is what makes
+    # chat_template_kwargs reach the template at all.
+    [ "$(argv_value --reasoning-format)" = "deepseek" ]
+    argv_value --jinja >/dev/null
+    grep -qxF -- '--jinja' <<<"$output"
+}
+
+@test "the serve script refuses a missing runtime, GGUF or projector" {
+    rm -f "$MODEL_DIR/$MM_STUB"
+    serve_argv
+    [ "$status" -ne 0 ]
+    [[ "$output" == *"frank-small: missing"* ]]
+
+    rm -f "$RUNTIME_DIR/bin/llama-server"
+    serve_argv
+    [ "$status" -ne 0 ]
+    [[ "$output" == *"frank-small: missing"* ]]
+}
+
+@test "the deploy handler gates on binary, GGUF and projector, then MainPID-hardens health" {
    grep -q 'deploy_llm_small' "$REMOTE_DEPLOY"
+    grep -q 'LLM_SMALL_RUNTIME' "$REMOTE_DEPLOY"
+    grep -q 'LLM_SMALL_MODEL_FILE' "$REMOTE_DEPLOY"
+    grep -q 'LLM_SMALL_MMPROJ' "$REMOTE_DEPLOY"
    grep -q 'relink_unit frank-small' "$REMOTE_DEPLOY"
    grep -q 'systemctl enable frank-small.service' "$REMOTE_DEPLOY"
-    grep -q 'systemctl restart frank-small.service' "$REMOTE_DEPLOY"
-    grep -q 'Gemma4-12B-QAT' "$REMOTE_DEPLOY"
-    grep -q 'refusing to start a unit that cannot load' "$REMOTE_DEPLOY"
    grep -q '127.0.0.1:7771/health' "$REMOTE_DEPLOY"
+    grep -q 'frank-small.service has no MainPID' "$REMOTE_DEPLOY"
 }

-@test "frank-small's serve script refuses to start on a half-finished download" {
-    grep -q 'frank-small: missing' "$SERVE"
+@test "the two endpoints keep separate ports and the large handler never pkills llama-server" {
+    # frank-large's gufo sweep must not take down frank-small, whose
+    # process is llama-server on the SAME host since this swap.
+    grep -q '^LISTEN_PORT=7771$' "$ENV_FILE"
+    local large
+    large="$(sed -n '/^deploy_llm_large() {/,/^}/p' "$REMOTE_DEPLOY")"
+    ! grep -q 'pkill -x llama-server' <<<"$large"
 }