docs(openwebui): frank-dragoon enabled in the picker beside frank-zealot

Live DB trio applied on Dankclaw (enable idx 0, model row + global access_grant, restart); verified /api/models as admin and role=user, chat answers pong. Battery rewritten to gate on /api/models — /api/v1/models/list lists only workspace models on 0.6.x.
docs(log): wire-rename wave — three served ids, OpenWebUI + Hermes re-key, bench drop-in retired
2026-10-09 20:43:28 -07:00 · 2026-10-09 17:10:25 -07:00 · 2026-10-09 17:06:44 -07:00 · 2026-10-09 16:47:19 -07:00 · 2026-10-09 11:49:58 -07:00 · 2026-10-08 23:44:13 -07:00
243 changed files with 22180 additions and 2052 deletions
@@ -12,8 +12,9 @@ merge absorbed the former `vllmScripts` repo, history preserved).
  Calibre-Web, Jellyfin, Plex) plus the `frank-primary` LLM service.
 - **Dankclaw** (192.168.0.143) — AI/search/dev/observability stack (SearXNG
  via Gluetun, ComfyUI, OpenWebUI, Hermes, Gitea, Grafana, Prometheus, Loki,
-  Alloy, Pi-hole) plus TWO LLM services: `frank-large` (stock llama.cpp on
-  the gfx1151 iGPU) and `frank-small` (llama.cpp on the RTX 3090). Caddy on
+  Alloy, Pi-hole) plus TWO LLM services: `frank-large` (flash-server
+  container on the gfx1151 iGPU) and `frank-secondary` (llama.cpp CUDA on the
+  RTX 3090). Caddy on
  Dankclaw reverse-proxies both hosts.
 - **FrankBook-Pro** (192.168.0.112) — macOS mini running the *arr stack
  (Prowlarr, Sonarr, Radarr, Lidarr, Readarr, qBittorrent) as quadlets inside
@@ -24,9 +25,9 @@ merge absorbed the former `vllmScripts` repo, history preserved).

 | Target | Service | Runtime | Model | Port |
 |--------|---------|---------|-------|------|
-| Dankserver | `frank-primary` | tcclaviger RDNA4 vLLM fork (`docker.io/tcclaviger/vllm@sha256:da4029666…`, digest-pinned, plain `podman run`, root systemd unit) | `orcarouter/Qwen3.8-Flash-Next-Uncensored-FP8` (~172.8 GiB, 131 shards), TP=2, expert offload behind a 15.05 GiB / 6,574-slot VRAM LRU arena, MTP4, prefetch OFF, 200000 ctx × 3 seqs, fp8 KV; measured 37.4-40.6 tok/s decode / ~54 % acceptance (the fleet's only endpoint meeting the 35 target). Rolled back 2026-10-01 from the failed GLM/DFlash2 experiment (commit `2339fae` trio) | 7770 |
-| Dankclaw | `frank-large` | gufo `gufo-org/gufo` v0.1.1, the Strix Halo engine, BUILT FROM SOURCE (root systemd unit; links the host ROCm 7.2.4) | Swift-1.5-Qwen3.8-27B-Uncensored UD-Q8_K_XL (27B dense, ABLITERATED/uncensored, 29.3 GiB, SWIFT OPEN LICENSE 1.0) with the z-lab DFlash2 Q4_K_M drafter (adaptive block policy, cap 7) on the gfx1151 iGPU (Strix Halo LPDDR, 114 GiB GTT), 262144 ctx single session, greedy, gufo's compiled-in Qwen renderer (NO Qwen-Sharp), text only | 7770 |
-| Dankclaw | `frank-small` | mainline llama.cpp master 37b53fd (root systemd unit) | HauhauCS Gemma4-12B-QAT-Uncensored-Balanced Q4_K_M (BF16 mmproj + MTP assistant sidecar, `draft-mtp,ngram-mod`) on the RTX 3090, 1048576 pool / 4 slots of 262,144, q8_0 KV, the fleet's fastest vision endpoint | 7771 |
+| Dankserver | `frank-primary` | **radiance** (StillDeadcode, **pin 1.0.13** digest-pinned `stilldeadcode/radiance@sha256:4c3f484564…`, plain `podman run`, root systemd unit) | ukisai/Swift1.5-Qwen3.8-Flash-Next BF16 (335.3 GiB, 397/397 sha256-verified) converted by `scripts/convert-swift15-w4dn8.sh` to `qwen4exp-swift15-w4dn8-i8-hc8m.rad` (**116,416 MiB**): **all 49,152 experts GPTQ w4** (10M-token calibration on engine 1.2.0) + **24 whole bf16-protected experts/layer** (energy-share law), int8 trunk/lm_head, fp8 HC, PLE n-gram fp8 fixed-scale, on **BOTH R9700s TP=2** (`--tp-wire wht6`), experts in a **40 GiB/rank pinned host pool** + VRAM cache (`--placement expert_tiered`, explicit `--vram-weights-mib 26071 --vram-kv-mib 3543`), **47.7 GiB n-gram copy pinned to host RAM** (`--ngram-placement ram`; root unit + `LimitMEMLOCK=infinity`), **fp8 KV** (inherited; pool 3.46 GiB), **MTP spec decode depth 2**, `--max-num-seqs 3`, `--max-model-len 200000`, MNBT 2048, Froggeric v22.5 template. KLD-gated (mean KL 0.0131, top-1 97.92 % vs BF16); decode ~190 t/s live (gate bar 30.8). Cutover 2026-10-08 from fp8-native (payload on disk = rollback). | 7770 |
+| Dankclaw | `frank-large` | **halogen-flash-server 0.16.2** (`ghcr.io/peonist-ai/halogen-flash-server@sha256:0c61bf84…`, digest-pinned, plain `podman run` via `scripts/serve-frank-large.sh`, root systemd unit) | Quat3rnion/halogen-swift1.5-qwen3.8-flash-next-v2-abliterated **`qwen38-flash-next-v2-swift15-abliterated.hgn`** (66,687,678,464 B = compacted r2 of the card's 89,775,868,032 B original, ~62 GiB resident; the v2 `.hgn` format loads in NO other engine), **+ REQUIRED `qwen38-flash-next-ngram.hgn`** (51,200,246,144 B, peonist @5cc17cea — read beside the checkpoint) **+ vision tower** (897,916,416 B, @e053f488 — **vision ON**, `image_url` parts only, ≥256 px probes), **MTP spec decode ALWAYS-ON (`HALOGEN_SPEC_ADAPT=0` — the adaptive default disables the drafter per request and HALVES decode)**, MTP head INSIDE the checkpoint (`HALOGEN_MTP_HEAD` unset; no sidecar fetched), **2 slots × 262,144 ctx** (KV pool 524,288 positions ≈ 14.0 GiB at ~28 KiB/position; the 27B slots>1-disables-drafting law does NOT carry over — batching and speculation coexist, the card's quickstart ships 2 with the head), harness window 217,088 = ceiling − 45k estimate headroom, **server-side sampler 1.0/0.95/20** (card recipe; models.yml extraBody matches), coupled cap/timeout 65536/7200, prompt cache ON by default (warm 100k turn ≈ 2 s vs ≈ 88 s cold)|7770|
+| Dankclaw | `frank-secondary` | llama.cpp master (0.5.0-dev, commit `5fc4f3c`), CUDA sm_86 build (root systemd unit, relinked) | HauhauCS/Qwen3.5-9B-Uncensored-HauhauCS-Aggressive **Q6_K** (7,359,259,008 B; dense `qwen35` hybrid: 32 layers, 8 full-attention + 24 Gated DeltaNet, 4 KV heads × 256 dim, vocab 248,320, native 262,144, **no MTP tensors in the file**) + the publisher's own **BF16 mmproj** (921,704,448 B) for **vision**, on the RTX 3090, **TWO 200,704-token slots** (`--ctx-size 401408 --parallel 2`), **exact f16 K + f16 V** under flash attention (32 KiB/token over the 8 full layers = 12.25 GiB pool), batch 4096 / **ubatch 2048** (measured CUDA optimum), `--log-verbosity 4`, **MTP OFF** (forced: no `nextn` tensors exist). Priced from the MiMo Q6_K boots (same geometry, same card, same shape): 21,276/2,849 at ubatch 1024, 21,846/2,279 at 2048. Cutover 2026-10-07 (late) from Gemma4-26B-A4B Q4_K_M 1×200,704. All three endpoints route harness **compaction** here. | 7771 |

 Retired endpoints (`frank-npu`, and every prior runtime) are in
 the [runbook index](#runbook-index). The full swap history lives in
@@ -40,7 +41,11 @@ the [runbook index](#runbook-index). The full swap history lives in
 - `scripts/` — `deploy.sh` / `deploy-plan.py` / `deploy-remote.sh`, LLM serve
  scripts, systemd units, env tunables, bench harnesses
 - `old/` — retired LLM scripts (reference only)
- `serverAppConfigs-docs/` — knowledge base (Obsidian vault, OKF format)
+- `serverAppConfigs-docs/` — knowledge base (Obsidian vault, OKF format).
+  The managed skill library is distilled there one page per skill-cluster:
+  `LLM/index.md` → *Cross-Cutting References*, plus `Operations/index.md`,
+  `Agent/index.md`, and the Infrastructure/Dankclaw/FrankBook-Pro service
+  pages (`serverAppConfigs-docs/log.md` carries the wave plan and results)
 - `tests/` — bats suite + Playwright e2e

 ## Git Workflow
@@ -68,12 +73,20 @@ All edits flow through git. Never edit files on a remote machine directly.
 The planner maps changed paths to services; the remote runner restarts only
 the selected service and records its revision outside the checkout.
 Documentation-only changes are a no-op. LLM services are selected only by
-their own files — every endpoint (`frank-large`/`frank-small`/`frank-primary`)
+their own files — every endpoint (`frank-large`/`frank-secondary`/`frank-primary`)
 by its unit/env/serve trio in `scripts/` — so
 unrelated
 container changes cannot interrupt inference. Set `FRANK_SUDO_PW` for headless
 runs on a host without passwordless sudo; the value never enters the repo.

+**`deploy.sh <target>` with no `--service` is a fleet-wide DRAIN, not a sync.**
+It also walks every per-service revision marker under the deploy-state dir and
+restarts each service whose marker is behind HEAD *and* whose paths changed
+since that marker — so a backlog accumulated by service-scoped deploys fires
+all at once, and one failing health gate aborts the run with every later
+service left pending. Name the service unless a drain is what you want
+(`./scripts/deploy.sh dankserver --service frank-primary`).
+
 ### Deployment completeness (MANDATORY)

 A deploy is **not complete** when `deploy.sh` exits green. It requires:
@@ -82,7 +95,14 @@ A deploy is **not complete** when `deploy.sh` exits green. It requires:
 2. the Playwright e2e suite (`tests/e2e/run.sh`) — runs **automatically** as
   the `[3/3]` gate at the end of `deploy.sh` against the LIVE fleet; a
   behavior-affecting deploy that fails it exits `DEPLOY INCOMPLETE`. Bypass
-   only with `--skip-e2e` in an emergency.
+   only with `--skip-e2e` in an emergency. The gate runs the conditional
+   `frankbook` Playwright project (the `*arr`/qBittorrent walk,
+   `tests/e2e/specs/19-frankbook.spec.ts`) ONLY when a frankbook blast-radius
+   path changed since origin/main — that spec, `frankbook-pro/**`,
+   `shared/homepage/services.json`, or the `dankclaw/Caddyfile` serving the
+   arr vhosts — so a sleeping FrankBook-Pro never fails an unrelated deploy.
+   The default project set excludes it everywhere else (`E2E_INCLUDE_FRANKBOOK=1`
+   opts in; see `serverAppConfigs-docs/Architecture/E2E Suite.md`).

 ```mermaid
 graph LR
@@ -209,14 +229,17 @@ archive.
 **STOP the production service BEFORE benchmarking** — these servers serve
 real users, and a bench beside a live service starves the GPU and corrupts
 results. Use the runtime's own harness: `llama-bench` for the llama.cpp
-endpoints, `gufo bench` for frank-large. Rolling your own bench
+endpoints; a live llama.cpp server also reports honest per-request numbers in the `/completion` `timings` object (prefill/decode rates, `draft_n`, `draft_n_accepted`), which is how frank-large and frank-secondary were measured. Rolling your own bench
 requires asking first.
 Scratch ports are not used on LLM hosts — a
 test process cannot share the single GPU anyway; validate against the service
 itself.

-**`gufo bench` cannot rank speculative-decode settings, and will invert the
-ranking.** Measured 2026-09-28 on frank-large at depth 32768: its corpus
+(A gufo-era law; frank-large ran ROCmFPX llama.cpp 2026-10-02 → 10-05, the
+halogen 27B container 2026-10-05 → 10-06, and is flash-server since, but the
+rollback tags keep it live.) **`gufo bench` cannot rank speculative-decode
+settings, and will invert the ranking.**
+Measured 2026-09-28 on frank-large at depth 32768: its corpus
 reports `acceptance=0.933 ± 0.000` with an **identical trace sha256** across
 draft widths, i.e. the same highly-predictable text every run, and 93 % of
 every proposal accepted. It reported 29–37 t/s decode where real traffic
@@ -241,11 +264,11 @@ one: `ip neigh show <ip>` from a peer — FAILED means off/NIC dead, while a
 merely wedged kernel usually still answers ARP.

 1. Stop and mask (self-healing would restart it):
-   - Dankclaw `frank-large` (root unit, gufo on the iGPU):
+   - Dankclaw `frank-large` (root unit, halogen podman on the iGPU):
     `sudo systemctl stop frank-large && sudo systemctl mask frank-large`
-   - Dankclaw `frank-small` (root unit, llama.cpp on the 3090):
-     `sudo systemctl stop frank-small && sudo systemctl mask frank-small`
-   - Dankserver `frank-primary` (root unit, tcclaviger vLLM on the 2x R9700):
+   - Dankclaw `frank-secondary` (root unit, llama.cpp on the 3090):
+     `sudo systemctl stop frank-secondary && sudo systemctl mask frank-secondary`
+  - Dankserver `frank-primary` (root unit, radiance on both R9700s, TP=2):
     `sudo systemctl stop frank-primary && sudo systemctl mask frank-primary`
 2. Bench.
   NOTE: `systemctl mask` FAILS on a relinked unit (the unit file in
@@ -263,9 +286,17 @@ deletes the unit file from `/etc/systemd/system/`, not just the symlink.
 The harness reaches the fleet through `.omp/agent/models.yml` in the
 operator's home — `/home/blank/.omp/agent/models.yml` on the dev box, which is
 where the harness runs; there is no copy on the servers. Each provider is one
-HOST:PORT; each model `id` is the served alias: `frank-primary` →
-`192.168.0.235:7770`, `frank-large` → `192.168.0.143:7770`, `frank-small` →
-`modelRoles`.
+HOST:PORT; each model `id` is the served alias. **Official model ids are
+the StarCraft unit names (2026-10-08) and since 2026-10-09 they are also the
+ids on the wire:** `frank-archon` → `192.168.0.235:7770`
+(`FRANK_PRIMARY_SERVED_NAME`), `frank-dragoon` → `192.168.0.143:7770`
+(`HALOGEN_MODEL_ID`), `frank-zealot` → `192.168.0.143:7771` (`ALIAS`). The
+SERVICES, units, scripts, env files and runbooks keep the old names
+(`frank-primary`, `frank-large`, `frank-secondary`); only the served alias,
+the `models.yml`/`config.yml` ids, the homepage cards, the OpenWebUI grants
+and the Hermes routing were renamed. Verify a rename on `/v1/models`, never
+on a chat response: radiance echoes the **requested** model name, so a 200
+proves nothing about the served id (llama.cpp and halogen ignore it).

 Resolution rules: the default model resolves as a `defaultProvider` +
 `defaultModel` pair (the provider must serve the model); a subagent override
@@ -277,184 +308,272 @@ When renaming a service, retiring an endpoint, or moving a model: update the
 provider block, model `id`, `contextWindow`, `maxTokens`, `input` in
 `models.yml`, then fix every `modelRoles` entry that named the old endpoint —
 a role pointing at a provider the file does not define fails at call time. As
-of 2026-09-26 both `frank-small` and `frank-primary` accept images
-(`input: [text, image]`) — frank-primary loads the BF16 mmproj explicitly
-(this build never autoloads projectors); `frank-large` remains text-only
-because its stock llama.cpp deployment loads no
-vision projector (the mmproj is not loaded). Note also that
-the reasoning channel differs by runtime: llama.cpp endpoints emit
-`reasoning_content`, vLLM emits `reasoning` — academic since the
-2026-09-26 swap, as every endpoint in the fleet is llama.cpp now.
+of 2026-10-06 **all three endpoints accept images** (`input: [text,
+image]`) — `frank-large` returned to vision with the flash-server v2 swap
+(the v2 vision tower loads beside the checkpoint; the text-only halogen
+27B window closed 2026-10-06) — while
+`frank-primary` takes them natively (radiance runs the checkpoint's
+27-layer Qwen-VL tower — patching, pad-token expansion and 3-part rotary
+are the engine core; `--mm-max-patches` available),
+and `frank-secondary` through
+the publisher's own BF16 mmproj passed as `--mmproj` — the model is natively
+multimodal but the Q6_K GGUF carries no vision tensors (same arrangement the
+Gemma4 era used). `frank-secondary`'s `contextWindow` is
+**200704** — the PER-SLOT window (`--ctx-size 401408 --parallel 2` since the
+2026-10-07 Qwen3.5-9B swap), not the pool and not the model's native 262144:
+the harness must compact before llama.cpp answers a longer prompt with a 400.
+Note also
+that
+the reasoning channel differs by runtime: llama.cpp AND SGLang (qwen3
+parser, live-verified) emit `reasoning_content`, vLLM emits `reasoning` —
+models.yml sets `reasoningContentField` per endpoint to match.
 There is **no audio**
-input anywhere in the fleet except frank-small: that capability left with the
-FLM NPU endpoint on 2026-09-11 and returned on 2026-09-17 with the Gemma4
-projector's audio encoder (frank-small transcribes WAV verbatim even though
-`audio` is omitted from its `input` list — OMP's schema rejects it).
+input anywhere in the fleet: that capability left with the FLM NPU
+endpoint on 2026-09-11, returned 2026-09-17 with the Gemma4 projector's
+audio encoder, and left again with the Gemma4 retirement on 2026-10-01
+(neither the MiMo nor the Qwen3.5 tower carries audio).

-### frank-large — gufo, Swift-1.5-Qwen3.8-27B-Uncensored UD-Q8_K_XL (Dankclaw, ACTIVE)
+### frank-large — halogen-flash-server 0.16.2, Qwen3.8-Flash-Next v2 Swift1.5 ABLITERATED (Dankclaw, ACTIVE)

-**gufo** (`gufo-org/gufo`) v0.1.1 — "the Strix Halo inference engine", a
-vertical engine built for exactly this hardware (Ryzen AI MAX+ 395 /
-Radeon 8060S, `gfx1151`) — as a **root systemd unit**
-(`scripts/frank-large.service`, `User=dank`), serving
-`ajgazin/Swift-1.5-Qwen3.8-27B-Uncensored-Dynamic-MTP-GGUF`'s `UD-Q8_K_XL`
-(27B dense, 29.3 GiB, sha256-verified; **abliterated/uncensored**, and under
-the *non-OSI* SWIFT OPEN LICENSE 1.0) on port 7770 from the gfx1151 iGPU.
-One session, full 262,144 ctx (the model's native window, `SESSIONS=1`),
-DFlash2 spec decode (the z-lab `Qwen3.8-27B-DFlash2-Q4_K_M` drafter, gufo's
-**adaptive** block policy, `--draft-tokens 7`), greedy sampling, reasoning
-extracted to `reasoning_content`, and **no vision projector**.
+The **flash-server engine** (`ghcr.io/peonist-ai/halogen-flash-server@sha256:0c61bf84…`,
+digest-pinned, published tag 0.16.2, 3.6 GB image) as a podman container
+driven by `scripts/serve-frank-large.sh` under a **root systemd unit**
+(`scripts/frank-large.service`, `User=dank`, `Restart=always`,
+`ExecStop=podman stop -t 60 frank-large-halogen`, the script runs `podman
+run` FOREGROUND — the container's exit code is the service's exit so
+`Restart=always` stays real, and the engine ledger lands on disk) on port
+7770 from the Strix Halo iGPU. Engine stdout goes to
+`/home/dank/logs/frank_large.log` — **the boot ledger (weights resident, KV
+pool, MTP head, prompt cache, any clamp) lives there, not in the journal**.
+Cutover 2026-10-06 from the halogen 27B-dense era
+([[frank-large-halogen27b-dflash2]], retired the same window).

-**Two swaps landed on 2026-09-28**, both same day: the engine (from the
-unsloth llama.cpp b11030 prebuilt, which had replaced Kairic Edge ROCmFPX)
-and then the payload (from `unsloth/Qwen3.8-27B-GGUF:UD-Q8_K_XL`, still on
-disk and the payload-only rollback). The two targets are structurally
-interchangeable for gufo — same `qwen35` architecture, the same 866 tensors
-in the same order, the same per-tensor Q8_0/F32/BF16 layout, the same MTP
-head at `blk.64` — verified by parsing and diffing both GGUF headers before
-either was loaded. **The DFlash2 drafter did not move**, so it has its own
-`DRAFTER_DIR` and the deployer's stray-run sweep keys on the drafter's
-filename, not the target's.
+Serving `Quat3rnion/halogen-swift1.5-qwen3.8-flash-next-v2-abliterated`:
+`qwen38-flash-next-v2-swift15-abliterated.hgn` (66,687,678,464 B ≈ 62.1
+GiB, sha256 `9c880b8b…` — a compacted r2 of the card's 89,775,868,032 B
+append-only original; the abliteration workflow appends and the repo's
+tooling drops dead padding, live tensors byte-identical) + the **REQUIRED**
+n-gram lookup table `qwen38-flash-next-ngram.hgn` (51,200,246,144 B,
+peonist @5cc17cea) + the vision tower `qwen38-flash-next-vision.hgn`
+(897,916,416 B, @e053f488) + the flat `tokenizer/` directory (6 files,
+fetched fresh from peonist). Cutover 2026-10-06 from the halogen
+27B-dense shape, chosen by Frank for the ABLITERATED base.

-**The runtime is built from source on the host** —
-`/home/dank/GitPackages/gufo-v0.1.1/`, reproducible with
-`scripts/build-gufo.sh` (`--verify` re-checks an installed tree). gufo ships
-no prebuilt Linux binaries and the v0.1.0/v0.1.1 release assets are empty.
-Unlike the retired llama.cpp prebuilt, which bundled its own ROCm, **gufo
-links the host ROCm 7.2.4 through its RPATH**, so the HIP dev packages must
-stay installed. Two host workarounds the build script encodes:
+**The format is the lock.** The v2 `.hgn` loads ONLY in the
+flash-server line — the 27B-era halogen 0.1.4 container refuses it, so the
+engine moved with the model. The MTP head is INSIDE the checkpoint
+(`HALOGEN_MTP_HEAD` stays unset; the separate `qwen38-flash-next-mtp.hgn`
+sidecar is deliberately not fetched).

- **Ubuntu's `librocwmma-dev` must be absent.** It installs a truncated
-  rocWMMA header set into `/usr/include` that shadows AMD's complete one
-  and breaks the build with `'internal/accessors.hpp' file not found`. A
-  `-I` cannot fix it — in HIP mode the host pass searches the system
-  directories first. The package has no reverse dependencies.
- **`--gcc-install-dir` must name GCC 15.** Ubuntu 26.04 ships
-  `/usr/include/c++/16`, which the ROCm clang prefers; its `<atomic_wait.h>`
-  needs a `std::__detail::__notify_impl` symbol version the installed
-  `libstdc++.so.6` (GLIBCXX_3.4.35) does not export, and the link fails.
+**Product facts vs the 27B era, both regressions FIXED:** the base is
+**abliterated** (the 27B was a censored-base finetune; refusals were an
+accepted trade there) and **vision is ON** (the 27B engine silently
+ignored its encoder — frank-secondary owned images for exactly one day at this
+endpoint).

-`gufo diagnose` is the preflight and must report `[PASS]` with `gfx1151`.
-It also self-caps compute queues (`GPU_MAX_HW_QUEUES`, 2 resident for
-`serve llm`) after reading `/sys/class/kfd/kfd/proc/*/queues/*/type`, which
-matters because ComfyUI holds queues on the same device; four HIP processes
-is the device ceiling.
+**Engine laws that shape the config** (all in `scripts/frank-large.env`
+with evidence): `HALOGEN_SPEC_ADAPT=0` is MANDATORY — the default adaptive
+policy disables the MTP head for whole requests after low acceptance and
+HALVES decode; the head stays on and temp-1.0 traffic pays some acceptance
+for it (accepted). `HALOGEN_CTX=262144` is the PER-SLOT hard ceiling (400
+naming the limit, never truncation) — RAISED from 200192 on 2026-10-07
+under the 400-zone law: the harness clamps max_tokens to
+`min(45056, contextWindow − est − 4096)`, its chars/4 est UNDER-counts
+the rendered prompt by ~3 % (measured: est 152,938 → server 157,388,
+request 43,158 vs room 42,804), so a harness window == engine ceiling
+made every request over ~146k est a guaranteed 400. `models.yml`
+`contextWindow: 217088` leaves the 45k gap that absorbs the estimate
+error (worst-case request ≈ 219k ≤ 262144); compaction fires at 85 % of
+the harness window = 184,524 est ≈ 190k measured (Frank's 180k target).
+`HALOGEN_KV_SLOTS=2` is LEGAL here — the 27B "slots>1 disables the drafter"
+law does NOT carry over; batching and speculation coexist (the card's
+quickstart ships 2 slots with the head), so 2 is concurrent capacity.
+`HALOGEN_KV_POOL_POSITIONS=524288` (explicit 2 × ctx; ~28 KiB/position ⇒
+14.0 GiB). `HALOGEN_MAX_TOK=8192` is only the per-request DEFAULT (the
+harness sends at most its `maxTokens` 45056); `HALOGEN_MAX_TOKENS_CAP=65536` +
+`HALOGEN_QUEUE_TIMEOUT=7200` are a coupled pair (FLAGS.md table).
+**A server-side sampler EXISTS** (`HALOGEN_TEMPERATURE/TOP_P/TOP_K` =
+1.0/0.95/20, the card's thinking recipe) — an absent temperature is no
+longer greedy; `models.yml` extraBody carries the same values, consistent
+not load-bearing. Prompt cache is ON by default (`PROMPT_CACHE=2`): a warm
+100k follow-up costs ~2 s vs ~88 s cold — the 27B engine's manual
+`HALOGEN_CACHE_MB` pin era is over. `response_format` is unsupported.
+Thinking lands in **`reasoning_content`**; the v2 chat template rejects
+`reasoning_effort` outside `xhigh|medium|low`, so the `models.yml` fold
+stays mandatory. Thinking tokens COUNT against the budget (empty content +
+`finish_reason: "length"` when exhausted mid-thought). Vision: OpenAI
+`image_url` content parts only; ≥256 px probe images or the model answers
+"video".

-**The client contract changed, and the chat template is the sharp edge.**
-**gufo has no Jinja engine at all** — `src/models/qwen/chat_template.cpp` is
-587 lines of hand-written C++ that appends strings, with no
-`--chat-template-file`, no env override, and no template body to point at.
-`scripts/qwen-sharp-v22.4.0.jinja` therefore **cannot** be supplied to
-frank-large: that is a port, not a config line. Given the choice on
-2026-09-28 the decision was to ship on gufo's stock renderer and accept the
-loss — the `terse` system-prompt injection, the in-band `<|think_*|>`
-control, the `json|xml` `tool_call_format` switch, the tool-argument
-truncation and `add_vision_id` prefixes are all gone. gufo's own renderer
-does keep the effort instruction strings and `preserve_thinking`, which are
-what the harness depends on. **This is the thing to watch**: the endpoint
-now runs an *abliterated* model without the prompt that tells it to stop
-waffling. The cheap fix, if verbosity shows up, is to send the `terse` text
-as an ordinary system message from `models.yml` — not a template change.
-`reasoning_effort` accepts the full OpenAI set (all six levels verified
-200), so `models.yml`'s `reasoningEffortMap` fold is no longer *required* —
-it is kept deliberately, so that removing it does not silently move every
-`high` role from `xhigh` to `high` effort inside a swap. And there is no
-KV-quantization or prompt-cache-RAM flag; prompt reuse is on by default and
-`--cache-disk` is opt-in and off.
+**Memory ledger** (121.2 GiB unified; the referee is the boot ledger):
+weights ~62 + KV 14.0 + arena ~8 (`3.8 + 8.4 × MAX_TOK/16384`) + vision
+~2 ≈ 86 GiB engine-side + `HALOGEN_HOST_RESERVE_GIB=20` → ~15 GiB
+margin over baseline co-tenancy. The 51.2 GB n-gram table rides as mmap'd
+page cache beside the checkpoint — the card's quickstart arithmetic does
+not bill it as resident, but READ the boot ledger and re-size if the
+engine prints a clamp.

-**Read the perf trade before assuming a win.** gufo's own Q8_K_XL tables
-(vs llama.cpp b11069, DFlash2) show prefill +37 % to +80 %, cold load 4.5×
-faster, decode +7 % at zero depth and +40 % at 4 users — but **−14 % at
-65,536 tokens of depth and −24 % at 131,072**. On the 8-bit target the
-crossover is ~32k of cached context, and frank-large runs one 262,144-token
-conversation, so long agent sessions sit on the losing side of that line.
-The Q4_K_XL target shows no such regression; revisiting the quantization is
-a separate decision from the runtime. Peak memory is 50.58 GiB at the full
-262,144 context against the iGPU's 114 GiB GTT aperture.
+Fetch: `scripts/download-frank-large-flashnext.sh` into
+`/home/dank/models/halogen-flashnext-v2/` (`-f` + exact byte counts + LFS
+sha256 + `--verify-only`, no token on the command line; resume goes
+against the RESOLVED CDN URL because curl drops `Range` across HF's 302).
+The serve gates mirror the deployer EXACTLY (byte-exact checkpoint +
+n-gram + vision tower, flat tokenizer with all six files as regular
+files) — the 2026-10-01 crash-loop lesson.

-The serve script (`scripts/serve-frank-large.sh`) validates runtime → target
-GGUF → drafter and exits with a pointer message if any is missing, then
-`exec`s `gufo serve --host 0.0.0.0 --port 7770 --sessions 1 llm --model
-…--served-model-name frank-large --context 262144 --speculative dflash2 …`.
-The deployer's `deploy_llm_large` gates on the **runtime binary** as well
-as both payload paths, and returns before it stops anything.
+Runbook:
+`serverAppConfigs-docs/LLM/frank-large-flash-server-v2-swift15.md`.
+Rollback: revert the swap commit and redeploy — the 27B payload
+(`/home/dank/models/halogen-qwen3.8-27b/`), its fetch script and the
+halogen 0.1.4 image digest all remain on disk; the ROCmFPX door behind
+that (`scripts/build-rocmfpx.sh`, pin `36e9acd`) is untouched.

-Fetch: `scripts/download-frank-large-swift.sh` (the target; per-file byte
-count + sha256 against the Hub LFS oid, no auth header — the repo is public
-and a token on the curl command line is world-readable) and
-`scripts/download-frank-large.sh` (the DFlash2 drafter, which did not move).
-Both take `--verify-only`.
+### frank-secondary — llama.cpp on the RTX 3090 (Dankclaw, ACTIVE)

-Runbook: `serverAppConfigs-docs/LLM/frank-large-gufo-qwen38-27b.md`.
-Rollback, two ways: **payload only** — re-fetch with
-`scripts/download-frank-large.sh` and repoint `MODEL_FILE` at
-`/home/dank/models/Qwen3.8-27B-UD-Q8_K_XL/Qwen3.8-27B-UD-Q8_K_XL.gguf`; that
-29.3 GiB target was **reclaimed on 2026-09-28** (staged to
-`~/.AGENTTRASH/20260928-retired-models/`, restorable from there), so it is
-no longer a one-minute flip. **Engine** —
-[[frank-large-unsloth-qwen38-27b-q8xl]], the b11030 prebuilt at
-`/home/dank/GitPackages/unsloth-llama-b11030/`; the only way to get
-Qwen-Sharp back. The DFlash2 drafter was never reclaimed and still lives in
-`Qwen3.8-27B-UD-Q8_K_XL/`.
+**llama.cpp master** (0.5.0-dev, commit `5fc4f3c`; CUDA sm_86 build
+`/home/dank/GitPackages/llama-master/build-cuda/bin/llama-server`,
+`GGML_CUDA_FA_ALL_QUANTS=ON`) as a **root systemd unit**
+(`scripts/frank-secondary.service`, `relink_unit`ed into
+`/etc/systemd/system/`), serving
+`HauhauCS/Qwen3.5-9B-Uncensored-HauhauCS-Aggressive` at **Q6_K**
+(7,359,259,008 B, sha256 `c0ba7beb…` computed locally — the Hub carries no
+LFS oid) **plus the publisher's own BF16 mmproj** (921,704,448 B), both
+`/home/dank/models/Qwen35-9B-HauhauCS-Q6K/`, on port 7771. Cutover
+2026-10-07 (late) from the Gemma4-26B-A4B Q4_K_M 1×200,704 shape (same day).

-### frank-small — llama.cpp on the RTX 3090 (Dankclaw, ACTIVE)
+**Architecture: qwen35 dense hybrid** — read from the GGUF's own header, not
+the model card: 32 layers, `full_attention_interval = 4` ⇒ **8 full-attention
+layers (3, 7, 11, …, 31) + 24 Gated DeltaNet linear layers**, 16 attention
+heads / **4 KV heads** × 256 dim, `ssm` conv_kernel 4 / state_size 128 /
+group_count 16 / time_step_rank 32 / inner_size 4096, native 262,144, vocab
+248,320. Tensor types are Q6_K × 250 + F32 × 177 — standard quants only, so
+the stock master build loads it (arch `qwen35` is registered in this build's
+`src/llama-arch.cpp`). **KV accounting:** the pool scales with the 8 full
+layers only — 8 × 4 × 256 = 8,192 elements/token per cache, so f16 K + f16 V
+bills **32 KiB/token** = 12.25 GiB at the 401,408 pool; the 24 DeltaNet
+layers bill a fixed recurrent state per SLOT, not per token, which is exactly
+why two 200k slots fit where the 26B MoE fit one.

-Mainline llama.cpp master `37b53fd` (CUDA build with
-`-DGGML_CUDA_FA_ALL_QUANTS=ON`) as a **root systemd unit** (repo
-`scripts/frank-small.service`, `relink_unit`ed into `/etc/systemd/system/`),
-serving `HauhauCS/Gemma4-12B-QAT-Uncensored-HauhauCS-Balanced` at Q4_K_M.
-The MTP head is a SEPARATE assistant file (`mtp-gemma-4-12B-it.gguf`, 4
-nextn layers) loaded via `--model-draft`; the draftless **ngram-mod**
-matcher chains behind it in the same `--spec-type` value. BF16 mmproj,
-port 7771, q8_0 KV, `CTX_SIZE=1048576` / `PARALLEL=4`
-(four FULL 262,144-token slots — the measured card ceiling), batch
-**8192/1024**, greedy default. Deployed flags are in
-`scripts/frank-small.env`; the serve script refuses to start if the
-weights, the projector, the sidecar or the binary are missing.
+**The shape (2026-10-07):** `--ctx-size 401408 --parallel 2` = TWO
+conversations of 200,704 (Frank's cap; the harness `contextWindow` quotes
+the per-slot number), `--flash-attn on`, **f16 K + f16 V** (exact KV),
+batch 4096 / **ubatch 2048** (the measured CUDA optimum: +4.6 % prefill,
+decode flat, 616 MiB — MiMo-era A/B on this card), `--log-verbosity 4`,
+`--mmproj` for vision, **speculative decoding OFF** (`SPEC_TYPE=none`).

-Verified at deploy (2026-09-17): `/v1/models` reports `frank-small`; a
-chat completion returns content plus `reasoning_content`; a tool call
-returns `tool_calls`; an image request is answered from the projector;
-an `input_audio` WAV transcribes verbatim. Post-tuning live numbers at
-max length: pp@254k ~965 t/s, tg@253k 226–302 t/s with ~94% draft
-acceptance; idle 23,300 MiB of 24,576. The throughput sweep (and why
-ubatch 2048 OOMs the draft's pp buffer, and why draft depth 4 + poll
-100 is WORSE at long ctx) is tabled in the runbook.
-Runbook: `serverAppConfigs-docs/LLM/frank-small-gemma4-12b.md`.
-Rollback: [[frank-small-qwen38-gsq-rco]].
+**Why MTP is off**: it is not a choice — **this Q6_K file carries no
+`nextn`/`mtp` tensors** (verified from the tensor-name list), and MTP is a
+property of the file, not of the flags. The publisher card advertises "MTP
+support" for the base architecture; the quantized export dropped the head.

-### frank-primary — tcclaviger vLLM, Qwen3.8-Flash-Next FP8 + MTP4 (Dankserver, ACTIVE)
+**Measured boots** (nvidia-smi, the referee): the deployed boot is
+**21,436 used / 2,689 free** (ubatch 2048) — the pricing boots were the
+MiMo-V2.6-Distill-Qwen-9B pair at this exact geometry (**21,276 / 2,849** at
+ubatch 1024, **21,846 / 2,279** at 2048), so the 415 MiB lighter weights
+bought back what the projection predicted. The engine ledger confirms the KV
+law to the MiB: `CUDA0 KV buffer size = 12544.00 MiB` (`K (f16): 6272.00`,
+`V (f16): 6272.00`), compute buffers 1,136 MiB on CUDA0 + 848 MiB host, the
+mmproj 879 MiB + 248 MiB. Decode is ubatch-invariant (Swift-era 3090 law). The
+Q6_K rung costs ~10 % decode vs Q5_K_M (weight bandwidth; MiMo-measured 94.4
+vs 104.8 t/s).

-The **tcclaviger RDNA4 vLLM fork** (`docker.io/tcclaviger/vllm@sha256:da4029666…`,
-digest-pinned) serving **`orcarouter/Qwen3.8-Flash-Next-Uncensored-FP8`**
-(~172.8 GiB, 131 shards) on **both** Radeon AI PRO R9700s (gfx1201), **TP=2**,
-**MTP4** speculative decoding, **expert prefetch OFF**, 200,000 ctx x 3 seqs,
-**fp8 KV**. Deployed 2026-10-01 by rollback from commit `2339fae` after the
-GLM/DFlash2 experiment was declared failed (same day).
+**Vision**: the publisher's own BF16 mmproj via `--mmproj` (the model is
+natively multimodal; the Q6_K file carries no vision tensors). Probe traps
+carry over: ≥256 px probe images or the model answers "video", and only the
+OpenAI `image_url` content part reaches the model.

-Measured 2026-09-29 tuning campaign: **decode 37.4-40.6 tok/s, acceptance
-~54 %** (MTP4 + prefetch off — the fleet's only endpoint that meets the
-35 tok/s target). 256k ctx arm measured 34.1-36.5; 2-way concurrency sums
-BELOW parity (25.49 vs 26.8 single) — slots are a capacity feature, not a
-throughput one. V-only q8 KV is NOT possible via config: stock
-`--kv-cache-dtype` is K+V together; a V-only quant needs a fork patch.
+**Compaction hub**: `models.yml` routes `compactionModel` for
+`frank-archon`, `frank-dragoon` and `frank-zealot` (the 3090 endpoint) here — 200,704 is the
+only window in the fleet that fits a worst-case compaction input (~170k
+tokens). With `--parallel 2` a compaction request no longer serializes behind
+a live generation (the Gemma4 era's `--parallel 1` law).

-The engine auto-sets the attention block size (mamba-page floor); the
-expert plan's registry prices KV per token; the expert arena (6,574 slots
-x 2.344 MiB = 15.05 GiB/rank) is the decode-critical region — cutting it
-for KV headroom collapses the offload hit rate.
+Runbook:
+`serverAppConfigs-docs/LLM/frank-secondary-llama-qwen35-9b-q6k-2x200704.md`.
+Rollback: revert the swap commit — the Gemma4 payload
+(`/home/dank/models/Gemma4-26B-A4B/`), the MiMo Q6_K payload
+(`/home/dank/models/MiMo-Q6K/`), the Swift payload
+(`/home/dank/models/Swift15-UncIQ3XXS/`) and the Ornith Q8_0 payload
+(`/home/dank/models/Ornith-Q8/`) all remain on disk.

-Unit and self-healing: root systemd unit (`scripts/frank-primary.service`,
-`User=dank`, `LimitMEMLOCK=infinity`), `ExecStop=podman stop -t 120
-frank-primary-glm`. The serve script drives `podman run` directly, gates
-the digest-pinned image, the payload's `config.json` and its exact
-131-shard count and the GPU device nodes, clears stale container
-identities, then blocks on `podman wait`. `deploy_llm_primary` gates the
-same, waits out :7770, relinks/enables/restarts the unit and health-gates
-with the MainPID check. Boot cost ~7 min warm; health returns 503 while
-loading.

-Runbook: `serverAppConfigs-docs/LLM/dankserver-qwen38-fp8-expert-offload-tuning.md`.
+### frank-primary — Radiance (StillDeadcode), Swift1.5-Qwen3.8-Flash-Next GPTQ w4, TP=2 (Dankserver, ACTIVE)
+
+The **radiance engine** (StillDeadcode, Apache-2.0; image
+`stilldeadcode/radiance@sha256:4c3f484564…` digest-pinned, **pin 1.0.13**
+for serve/convert/KLD; `libr4d` gfx1201 kernels, plain `podman run`, root
+systemd unit, container name = unit name) serving
+**ukisai/Swift1.5-Qwen3.8-Flash-Next (BF16, 335.3 GiB, 131 shards,
+397/397 sha256-verified)** converted by `scripts/convert-swift15-w4dn8.sh`
+with recipe `scripts/recipes/qwen4exp-swift15-w4dn8-i8-hc8m.recipe` to
+`qwen4exp-swift15-w4dn8-i8-hc8m.rad` (~114 GiB): **all 49,152 experts GPTQ
+w4** (act-order, damp 0.05, 10,000,000-token calibration captured on engine
+**1.2.0** via `RADIANCE_CALIB_DIR`/`RADIANCE_CALIB_TOKENS` — the only pin
+with the token knob; the RADGRAM2 grams are format-identical across pins and
+cross the boundary by design), **24 whole experts per layer cast bf16** (the
+PROTECTED set, derived from the calibration grams' energy share),
+**int8 trunk/lm_head**, **fp8 HC mixtures** (`--hc-mix-dtype`), PLE n-gram
+table at fp8 fixed scale (47.7 GiB in the container, from 95.4 GiB bf16).
+The plugin's **one-table-one-encoding law** (measured at rad-convert
+declare): one expert table takes ONE encoding; only whole bf16 experts are
+legal beside it — no w4/fp8 split of the down/gate_up pair (24-value
+`rad_arch_quant_desc` array, one per layer). KLD gate vs the BF16
+reference (recorded on the same 1.0.13 pin) is the precision referee;
+the throughput gate (decode ≥ 30.8 t/s) was passed at 4.5× the bar
+(122–173 t/s at 44.5k depth). Measured decode/prefill in the runbook.
+Rollback: revert the swap commit — the fp8-native `.rad`
+(`qwen4exp-fp8-native.rad`) and every prior payload remain on disk.
+
+**Shape (both R9700s, TP=2)**: `--tp-wire wht6`, `--max-num-seqs 3`,
+`--max-model-len 200000` (the harness contextWindow must quote this),
+`--placement expert_tiered --host-pool-mib 40960` + **explicit VRAM
+partition `--vram-weights-mib 26071 --vram-kv-mib 3543`** (the vram-pair
+replaced `--expert-vs-cache-ratio`; the pair sizes the fp8-KV shape at the
+deployed boot), `--kv-cache-dtype fp8` (inherited from the fp8-native
+env), `--num-speculative-tokens 2`,
+`--max-num-batched-tokens 2048`, prefix cache 4 GiB pinned host + 64 GiB
+disk at `/home/dank/radiance-state`, **`--ngram-placement ram`**, Froggeric
+v22.5 chat template vendored (`scripts/qwen38-froggeric-v22.5.jinja`,
+`--override-chat-template`). Boot ledger: KV pool 3.46 GiB (~480k fp8
+tokens), two expert pools 57.84 GiB, cards settled 1.41 GiB free.
+
+Shape notes carried from the fp8-native era and unchanged: wht6 gains,
+MNDT 2048 law (2560 collapses decode to 2.6–7.7 t/s), the 64 GiB disk
+prefix cache, and the rule that a prompt over `--max-model-len` is
+refused, never truncated. What changed: the fp8-native era's NVMe expert
+streaming (the 3.5 GB/s wall) is gone — the w4 resident set fits the
+40 GiB pool, so decode ran 122–173 t/s in the campaign arms and ~190 t/s
+on the deployed shape (512-token streamed probe). The campaign also
+measured the int8-KV + spec-3 arm (136.0 t/s median at 44.5k depth,
+accept 0.7246); production shipped the fp8-KV + spec-2 pair at cutover —
+the KLD gate scored at the fp8-KV shape, so the gate and the boot agree.
+Disk-tier sizing law: the engine forces ~9.6 GiB of disk-tier weight
+buffers + arena onto the weights side, so `weights + kv` is capped near
+29.6 GiB/card; boot with the pair, never `--kv-cache-size-mib`.
+
+**The pinned-RAM ledger is why this is a root unit**: 2 × 40 GiB expert
+pools + the 47.7 GiB n-gram copy (`--ngram-placement ram` is pinned RAM
+the card reads) + 4 GiB prefix host ≈ **132 GiB** charged against
+`RLIMIT_MEMLOCK` — the TTM floor is set just above it.
+`ttm.pages_limit`/`pages_max` cap GPU-mappable pinned pages; the serve
+script's TTM preflight (`FRANK_PRIMARY_TTM_MIN_MIB=140000`) refuses to
+start until both knobs are raised in the stop window (sysfs write first,
+kernel cmdline + reboot only if read-only).
+
+**Contract facts** (source-verified): the HTTP listener binds AFTER the
+full load, so a bare `curl /health` gate is honest (`/health` == `/ping`,
+200 serving / 503 stopping); sampling params temperature…preserve_thinking
+accepted, **unknown params 400** (models.yml's extraBody set is fully
+inside it); `response_format` supported; `logprobs` REFUSED; reasoning in
+`reasoning_content` and tool calls in the qwen3.8 dialect via the auto
+chat format — no parser flags; vision native (the 27-layer Qwen-VL tower,
+`--mm-max-patches`); `/metrics` carries vLLM-compatible names +
+`radiance:*` families — the Dankclaw Prometheus scrape is re-added.
+
+Runbook: `serverAppConfigs-docs/LLM/frank-primary-radiance-swift15-w4dn8.md`.
+Retired fp8-native era: `serverAppConfigs-docs/LLM/frank-primary-radiance-qwen38-native-fp8.md`.
+Retired FreeToken era: `serverAppConfigs-docs/LLM/frank-primary-freetoken-pr587.md`.
+Retired vLLM era: `serverAppConfigs-docs/LLM/dankserver-qwen38-fp8-expert-offload-tuning.md`.
 Retired GLM experiment: `serverAppConfigs-docs/LLM/frank-primary-glm53-flash-w4a16.md`.

 ## Key Lessons Learned
@@ -621,15 +740,56 @@ linked runbooks.
  the KV pool is worse than the OOM it prevents.
 - Host swap state is invisible to `deploy.sh` (no repo file) — check
  `swapon --show` before reasoning about headroom.
+- **Silent RAM corruption outranks every software theory.** frank-primary's
+  ~40k-token output corruption (2026-10-04/05, repetition/garbage in long
+  agent conversations) was marginal DDR5 at 4800 MT/s on four DIMMs, fixed
+  by 4600. Consumer non-ECC memory raises **no MCE** for a silent flip, and
+  a short in-band pattern check (60 s, 8 GiB of 246 GiB) is not a stability
+  proof — combined, those two instruments said "ruled out" while the RAM
+  was the cause. "4800 is the JEDEC floor on 5600-rated DIMMs" is a rating,
+  not a margin guarantee. When the fault is data-shaped (NaN states,
+  garbage tokens) and a software revert doesn't move it, price in the RAM
+  before the runtime — and remember a revert tested under the same marginal
+  hardware is itself confounded. See the runbook's 2026-10-05 section.

 **Runtime-specific guards**

+- **Rootless podman cannot RAISE rlimits, and `--ulimit nofile=1048576`
+  dies at container create, not at runtime.** runc reports
+  `error setting rlimit type 7: operation not permitted` (type 7 =
+  RLIMIT_NOFILE) when the requested hard limit exceeds the host process's
+  hard limit — on Dankclaw that ceiling is 524288. Inherited-equal limits
+  are fine, so `memlock=-1:-1` works iff the unit grants
+  `LimitMEMLOCK=infinity` (systemd is privileged; the value flows through
+  podman to runc without any raise). Upstream halogen's own run lines pass
+  NO `--ulimit` at all — copy vendor run lines before adding flags
+  (measured 2026-10-05 on the frank-large cutover).
 - **Kairic Edge** (if ever re-deployed): the pinned CK commit needs
  `third_party/composable_kernel/include/ck/config.h` generated manually
  (untracked; `git clean` removes it), and its greedy-argmax fastpath rejects
  non-greedy requests and reasoning budgets. Re-test chat before production.
 - **FastMTP sidecar**: build must be commit `4df29be4` + the HauhauCS patch;
  stock binaries fail the draft load with a tensor-shape mismatch.
+- **A GGUF's custom tensor type picks the runtime — read the histogram before
+  promising a file to a loader.** `Qwen3.8-27B-Q8_0_ROCMFPX_AGENT.gguf` is
+  ftype 115 with 311 tensors of type 103 (`GGML_TYPE_Q8_0_ROCMFPX`); stock
+  llama.cpp and gufo refuse it at load, so ROCmFPX was not a preference but
+  the only door. Counting tensor types out of the GGUF KV section takes one
+  python read and settles a runtime argument in minutes. The build identity
+  matters as much as the fork: the 2026-08-25 ROCmFPX builds SIGSEGV in
+  `ggml_compute_forward_set_rows` on this same file, so
+  `scripts/build-rocmfpx.sh` pins commit `36e9acd` by SHA.
+- **Speculative-decode flags move as a SET.** Handing the ROCmFPX fork
+  `--model-draft` with no `--spec-type` (the naive way to build a "no MTP"
+  bench arm) segfaulted at load, core dumped, no diagnostic (2026-10-02). A
+  baseline arm must strip `--model-draft` AND every `--spec-draft-*` AND
+  `--spec-type`, or it measures a crash instead of the absence of drafting.
+- **An orphaned scratch server taxes the whole host.** A bench child left
+  alive by a client-side exception kept ~26 GiB of weights resident on a
+  121 GiB box, pushed `available` to 3 GiB, and every later rate was suspect.
+  Record the child pid and kill it in a `finally`, then prove `free` came back
+  before believing a number. Kill by pid — never `pkill -f llama-server`,
+  which also hits frank-secondary.

 **llama.cpp serving**

@@ -639,19 +799,26 @@ linked runbooks.
  the MAIN model's weights (`shares_model`), so there is no second copy of a
  12 GiB model in VRAM and no `-md` argument. The plain build has no MTP
  tensors at all, and pointing `--model-draft` at the file would load it twice.
- **What can be chained, and what cannot.** `--spec-type a,b` builds every
-  enabled implementation and chains drafting through them, in a fixed internal
-  order (ngram types first, then the drafters — CLI order is irrelevant). The
-  gate is `ctx_dft`: **draftless** types (the ngram family) are added
-  unconditionally, while `draft-mtp`, `draft-dflash`, `draft-dspark` and
-  `draft-eagle3` all require a draft CONTEXT. So a chain may hold one
-  context-backed drafter plus any number of ngram types — `draft-dflash,ngram-mod`
-  is legal and measured (2.6x the drafter alone on prompt-echo traffic, byte-
-  identical to it on prose at the fleet's 48/64 window) — but
-  `draft-mtp,draft-dflash` dies at load: *"context type MTP requested but model
-  doesn't contain MTP layers"*, because MTP builds its context against whatever
-  model the draft context holds. Two context-backed types cannot share one
-  process, and MTP's head only loads when MTP **is** the draft model.
+- **What can be chained, what cannot, and what a chain costs on a hybrid.**
+  `--spec-type a,b` builds every enabled implementation and chains drafting
+  through them, in a fixed internal order (ngram types first, then the
+  drafters — CLI order is irrelevant). The gate is `ctx_dft`: **draftless**
+  types (the ngram family) are added unconditionally, while `draft-mtp`,
+  `draft-dflash`, `draft-dspark` and `draft-eagle3` all require a draft
+  CONTEXT. So a chain may hold one context-backed drafter plus any number of
+  ngram types — `draft-dflash,ngram-mod` is legal and measured (2.6x the
+  drafter alone on prompt-echo traffic, byte-identical to it on prose at the
+  fleet's 48/64 window) — but `draft-mtp,draft-dflash` dies at load: *"context
+  type MTP requested but model doesn't contain MTP layers"*, because MTP
+  builds its context against whatever model the draft context holds. Two
+  context-backed types cannot share one process, and MTP's head only loads
+  when MTP **is** the draft model. **A LEGAL chain can still lose on a
+  recurrent model:** `draft-mtp,ngram-mod` on frank-large's qwen35 hybrid
+  measured −15 % prose / −7 % block-copy (2026-10-03) — adding the ngram layer
+  logs `recurrent state rollback is not compatible with 'ngram-mod' -
+  disabling rollback support`, and MTP verification depends on that rollback,
+  so acceptance drops on ALL traffic even where the matcher fires. Measure
+  chains per architecture, never by analogy.
 - **A drafter's KV scales with the TARGET's context, and its window policy
  decides how much that costs.** The draft context is built with
  `cparams.n_ctx = llama_n_ctx(ctx_tgt)` (`common/speculative.cpp`) — there is
@@ -668,7 +835,8 @@ linked runbooks.
  trip moved the balance point FOUR layers (~4.3 GiB of headroom, 40:25 ->
  36:29), which is the practical size of the effect. See
  [[frank-medium-llamacpp-dflash2]].
- **`--ubatch-size` has an OPTIMUM on this stack, not a plateau — and its
+- **`--ubatch-size` has an OPTIMUM on the 2x R9700 layer split, not a plateau —
+  and the curve is BACKEND-SPECIFIC, so never carry it across (next bullet). Its
  compute buffers land about TWICE as heavy on the card that holds the output
  layers and the drafter.** Measured on frank-medium 2026-09-16 at the deployed
  shape (262144 over two slots, a 64k prefill measured at 64k depth,
@@ -686,6 +854,19 @@ linked runbooks.
  its 2020 MiB pp buffer here under the DSpark drafter, when ROCm1 had 0.08 GiB
  free. That was a CONTIGUITY failure, not a total-bytes one, and no totals
  model — the sweep's or the seam's — can see it.)
+- **On the CUDA 3090 the same knob has NO interior optimum up to 2048, so
+  frank-secondary ships 4096/2048.** Measured 2026-10-05 by boot-pair A/B on the
+  live service: at 8,928 prompt tokens ubatch 1024 gives 3,972 / 4,077 t/s and
+  ubatch 2048 gives 4,207 / 4,212 — **+4.6 %** — for **616 MiB** (free 2,849 →
+  2,279). Decode did not move (91.9–93.0 vs 93.8–94.3, i.e. noise), and a
+  279-token prompt, which is below either ubatch, came back identical — that is
+  the mechanical proof the knob was the only variable, since a prompt shorter
+  than the ubatch cannot see it. The logical `--batch-size` was again free. And
+  the predecessor's "ubatch 2048 failed the scratch boot" was a shape
+  observation, not a law: the same file boots 2048 clean at 2 × 200,704. Bench
+  it by restarting the SERVICE through a transient `FRANK_SECONDARY_ENV_FILE`
+  drop-in, never a scratch port — one card holds one server. See
+  [[frank-secondary-llama-mimo-q6k-2x200704]].
 - **One `-ts` step is one LAYER, and a hybrid stack's layers are not the same
  size.** qwen35 is 16 x (3 Gated DeltaNet + 1 Gated Attention), so a step is
  worth ~0.43 GiB on a DeltaNet boundary and ~1.43 GiB on a full-attention one
@@ -702,6 +883,41 @@ linked runbooks.
  host-RAM hash table) and its window decides which traffic it can see: too
  short and it proposes on ordinary prose fragments, which costs verification
  without acceptance.
+- **`--parallel` divides the pool, so MORE conversations cost WINDOW, and the
+  per-slot number is the contract — not `ctx-size`.** `n_ctx_slot =
+  CTX_SIZE / PARALLEL`, and llama.cpp refuses a prompt longer than the SLOT
+  (`400 exceed_context_size_error: request (184001 tokens) exceeds the
+  available context size (180224 tokens)`) even when the pool has room. So
+  frank-secondary's 2 → 3 reslice moved each conversation from the model's native
+  262,144 down to 180,224, and `models.yml` `contextWindow` had to follow or
+  the harness would discover the ceiling as a 400 on a long conversation. Side
+  effect worth knowing: the shorter slot made decode ~5 % FASTER (106.8 vs
+  100.4 tok/s) — less attention to scan and one fewer DeltaNet state resident.
+  The 2026-10-03 reslice went the other way (3 → 2 slots, 180,224 → 200,704,
+  pool 401,408) to fund unquantized f16 V, and `contextWindow` followed again;
+  decode landed 104.8 tok/s — the ~2 % off 106.8 is the exact-V tax. The same
+  day the freed space also bought the Q6_K rung (now 94.4 tok/s — the rung
+  itself is a further measured −10 %, weight-bandwidth, not slot geometry).
+  All moves measured; both slot directions cost or buy window, never pool.
+  See [[frank-secondary-llama-mimo-q6k-2x200704]].
+- **Price a VRAM budget from two boots, never from a formula — and an mmproj
+  costs ~1.5x its file size.** The frank-secondary 3090 fit was solved from two
+  measurements: 2 × 262,144 with no projector = 20,190 MiB used (KV 12,800 ⇒
+  **non-KV 7,390**), 3 × 196,608 with the f16 projector = 22,972 used (KV
+  14,113 ⇒ **non-KV 8,859**). The difference is the vision tower: a 918 MB file
+  that bills ~1.4 GiB, because the encoder's activation and compute buffers
+  ride with it. Subtracting the two rows is exact; extrapolating weights+KV+
+  "a bit for buffers" is what leaves 1.1 GiB free when you meant 2.7. Then
+  solve `KV ≤ card − non-KV − headroom` and round to a multiple of
+  `PARALLEL × 4096`. Recorded in `scripts/frank-secondary.env` so the next resize
+  starts from measurement.
+- **`images` and `image_url` are NOT interchangeable: only the content part
+  reaches the model.** `--mmproj` loads, `usage.prompt_tokens` rises, and the
+  model still invents an answer if the client used the legacy top-level
+  `images` field on `/v1/chat/completions` — frank-secondary answered "Brown." for
+  a solid-red PNG that way and "Red." through the OpenAI `image_url` part. The
+  failure is silent because a hallucinated color is a plausible answer. Probe
+  vision with a solid color and expect an exact word.
 - **`--jinja` is what `tools` needs.** On 2026-09-16 a subagent died with
  `500 tools param requires --jinja flag`, from the ik_llama.cpp runtime that
  then served frank-large: every tool-using agent on that endpoint was broken
@@ -710,7 +926,7 @@ linked runbooks.
  returns `tool_calls`. Check tool support before routing a role at an endpoint.
 - **Quantized KV on CUDA needs `GGML_CUDA_FA_ALL_QUANTS=ON`** — the same
  option and the same ~12x prefill collapse described in the ROCm note above.
-  f16 KV needs nothing; frank-small ran f16 for that reason until
+  f16 KV needs nothing; frank-secondary ran f16 for that reason until
  2026-09-17, when `build-cuda` was rebuilt with the option ON so the
  endpoint could take q8_0 KV.
 - **llama-bench separates a LIST with `/` and SWEEP values with `,`.** Both
@@ -728,7 +944,7 @@ linked runbooks.
  raises on anything else — HTTP 500, not a clamp — so the harness's levels
  must be folded onto that set in `models.yml` (`compat.reasoningEffortMap`).
  A role whose level is unmapped fails on its FIRST request, and the failure
-  names Jinja, not the model: every subagent on `frank-small` died with
+  names Jinja, not the model: every subagent on `frank-secondary` died with
  `Unexpected reasoning effort high` until `high: xhigh` was added on
  2026-09-16. Halogen accepts the wider set, so the two entries legitimately
  differ.
@@ -802,9 +1018,9 @@ Active:

 | Service | Runbook |
 |---------|---------|
-| frank-large (gufo Qwen3.8-27B UD-Q8_K_XL, Dankclaw) | `serverAppConfigs-docs/LLM/frank-large-gufo-qwen38-27b.md` |
-| frank-small (llama.cpp Gemma4-12B QAT, Dankclaw) | `serverAppConfigs-docs/LLM/frank-small-gemma4-12b.md` |
-| frank-primary (tcclaviger vLLM, Qwen3.8-Flash-Next FP8 + MTP4, Dankserver) | `serverAppConfigs-docs/LLM/dankserver-qwen38-fp8-expert-offload-tuning.md` |
+| frank-large (flash-server 0.16.2, abliterated Swift1.5 v2 .hgn + MTP always-on, 2×262,144, vision ON, Dankclaw) | `serverAppConfigs-docs/LLM/frank-large-flash-server-v2-swift15.md` |
+| frank-secondary (llama.cpp CUDA, Qwen3.5-9B-Uncensored Q6_K, 2×200,704 exact f16 KV MTP-off, vision ON, Dankclaw) | `serverAppConfigs-docs/LLM/frank-secondary-llama-qwen35-9b-q6k-2x200704.md` |
+| frank-primary (radiance StillDeadcode, Qwen3.8-Flash-Next-FP8 native, 2×R9700 TP=2, Dankserver) | `serverAppConfigs-docs/LLM/frank-primary-radiance-qwen38-native-fp8.md` |
 | Dankclaw kernel posture (`amd_iommu=off`) | `serverAppConfigs-docs/LLM/dankclaw-iommu-translation-off.md` |
 | Deployment flow + incidents | `serverAppConfigs-docs/LLM/deployment.md` |

@@ -812,6 +1028,19 @@ Retired (rollback targets and history):

 | Runtime | Host | Window | Runbook |
 |---------|------|--------|---------|
+| llama.cpp CUDA Gemma4-26B-A4B QAT Q4_K_M, 1×200,704 f16-K/q8_0-V MTP-off (`frank-secondary`) | Dankclaw | 2026-10-07 → 10-07 (same-day swap; rollback target — payload on disk; owns the `attention_k_eq_v` pair-billing KV law and the gemma4-assistant drafter-cost boots) | `frank-secondary-llama-gemma4-q4km-200704.md` |
+| ROCmFPX llama.cpp fork Qwen3.8-27B Uncensored Q6_0_ROCMFPX_AGENT + MTP (`frank-large`) | Dankclaw | 2026-10-03 → 10-05 (rollback target, payload + fork build on disk) | `frank-large-rocmfpx-q6agent-uncensored-mtp.md` |
+| tcclaviger vLLM fork Qwen3.8-Flash-Next-Uncensored FP8, TP=2, shape G (`frank-primary`) | Dankserver | 2026-09-29 → 10-05 (rollback target — tag `frank-primary-vllm-shapeG-20261003`; trio + env lore in the tag, image in rootless storage) | `dankserver-qwen38-fp8-expert-offload-tuning.md` |
+| FreeToken PR #587 ROCm, Qwen3.8-Flash-Next-Uncensored-FP8, ONE R9700 (`frank-primary`) | Dankserver | 2026-10-05 → 10-05 (rollback target — tag `frank-primary-freetoken-pr587-20261005`; trio + Dockerfile in the tag, image `localhost/frank-primary-freetoken:pr587-77a7201` and orcarouter payload on disk) | `frank-primary-freetoken-pr587.md` |
+| llama.cpp master MiMo-V2.6-Distill-Qwen-9B Q6_K 2×200,704 f16 KV (`frank-secondary`) | Dankclaw | 2026-10-03 → 10-05 (rollback target — payload on disk; owns the ubatch-2048 A/B; the Ornith swap of 10-05 was committed but never deployed, so this was the LIVE payload Swift replaced) | `frank-secondary-llama-mimo-q6k-2x200704.md` |
+| llama.cpp Ornith-1.5-9B Q8_0, ONE 262,144 f16 KV (`frank-secondary`) | Dankclaw | 2026-10-05 (committed 77decb5→380aafa, never deployed — superseded by Swift the same evening; payload on disk) | trail in `frank-secondary-llama-swift15-unc-iq3xxs-mtp-114688.md` |
+| llama.cpp CUDA Swift-1.5-27B Uncensored RCO IQ3_XXS-mtp, 2×114,688 draft-mtp q8_0 KV (`frank-secondary`) | Dankclaw | 2026-10-06 → 2026-10-07 (rollback target — payload on disk; owns the 3090 ubatch ladder A/B and the 34 KiB/token qwen35 KV law) | `frank-secondary-llama-swift15-unc-iq3xxs-mtp-114688.md` |
+| ROCmFPX llama.cpp fork Qwen3.8-27B Q8_0_ROCMFPX_AGENT + MTP (`frank-large`) | Dankclaw | 2026-10-02 → 10-03 (rollback target, payload on disk) | `frank-large-rocmfpx-q8agent-mtp.md` |
+| gufo v0.1.1 Qwen3.8-27B UD-Q8_K_XL/UD-Q4_K_XL + DFlash2 (`frank-large`) | Dankclaw | 2026-09-28 → 10-01, again 10-01 → 10-02 (payloads removed from disk 2026-10-03; rollback = re-download) | `frank-large-gufo-qwen38-27b.md` |
+| vLLM TheRock MiMo-V2.6-Distill-Qwen-9B BF16 + cross-model MTP (`frank-large`) | Dankclaw | 2026-10-01 (same-day rollback to gufo) | `frank-large-vllm-mimo-v26-bf16.md` |
+| llama.cpp master MiMo-V2.6-Distill-Qwen-9B Q6_K (`frank-secondary`) | Dankclaw | 2026-10-01 → 10-02 (the rung returned ACTIVE 2026-10-03 at the 2×200,704 shape — `frank-secondary-llama-mimo-q6k-2x200704.md`) | `frank-secondary-llama-mimo-q6k.md` |
+| llama.cpp master MiMo-V2.6-Distill-Qwen-9B Q5_K_M 2×200,704 f16 KV (`frank-secondary`) | Dankclaw | 2026-10-02 → 10-03 (rollback target — one `MODEL_FILE` line, buys back 875 MiB + the −10 % rung tax; owns the boot-pair table) | `frank-secondary-llama-mimo-q5km.md` |
+| SGLang MiMo-V2.6 INT8-W8A16-AutoRound (`frank-secondary`) | Dankclaw | 2026-10-01 (same-day swap to the GGUF path) | `frank-secondary-sglang-mimo-v26.md` |
 |tcclaviger vLLM GLM-5.3-Flash W4A16 int4, +/- DFlash2 drafter (`frank-primary`)|Dankserver|2026-09-30 → 10-01|`frank-primary-glm53-flash-w4a16.md`|
 | llama.cpp b11030 Qwen3.8-27B UD-Q8_K_XL + DFlash2 (`frank-large`) | Dankclaw | 2026-09-21 → 09-28 | `frank-large-unsloth-qwen38-27b-q8xl.md` |
 | R9V radiance-vLLM Qwen3.8-Flash-Next-Uncensored IQ4_XS (`frank-primary`) | Dankserver | 2026-09-27 (same-day swap to Q8_0) | `frank-primary-r9v-radiance-qwen38-uncensored.md` |
@@ -824,15 +1053,16 @@ Retired (rollback targets and history):
 | CIRU v4.4.1 Orca Qwen3.8-Flash (`frank-large`) | Dankclaw | 2026-09-19 → 09-21 | `frank-large-ciru-orca.md` |
 | unsloth llama.cpp b11030 Qwen3.8-Flash-Next UD-IQ4_XS (`frank-large`) | Dankclaw | 2026-09-21 (same-day swap) | `frank-large-unsloth-qwen38-flash-next.md` |
 | Kairic Edge Qwen3.8-27B IU4 W4A4 (`frank-large`) | Dankclaw | 2026-09-21 (same-day swap) | `frank-large-kairic-edge-qwen38-27b.md` |
-| ISTA-DASLab Qwen3.8-27B-GSQ-RCO IQ3_S (`frank-small`) | Dankclaw | 2026-09-16 → 09-17 | `frank-small-qwen38-gsq-rco.md` |
-| vLLM Gemma4-12B w4a16 (`frank-small`) | Dankclaw | 2026-09-19 (same-day revert) | `frank-small-vllm-gemma4.md` |
+| ISTA-DASLab Qwen3.8-27B-GSQ-RCO IQ3_S (`frank-secondary`) | Dankclaw | 2026-09-16 → 09-17 | `frank-secondary-qwen38-gsq-rco.md` |
+| vLLM Gemma4-12B w4a16 (`frank-secondary`) | Dankclaw | 2026-09-19 (same-day revert) | `frank-secondary-vllm-gemma4.md` |
+| llama.cpp Gemma4-12B QAT Q4_K_M (`frank-secondary`) | Dankclaw | 2026-09-17 → 10-01 | `frank-secondary-gemma4-12b.md` |
 | agentionai ROCmFP4-FAST imatrix | Dankclaw | 2026-08-29 → 08-30 | `agentionai-fast-imatrix.md` |
 | eemin Qwen3.8-27B ROCmI4 W4A4 MTP | Dankclaw | 2026-08-18 → 08-25 | `eemin-rocmi4-mtp.md` |
 | Kairic Edge IU4 | Dankclaw | 2026-08-22 (same-day revert) | `kairic-edge.md` |
 | CIRU ActiveFPX v3 / PromptForge | Dankclaw | through 2026-08-22 | `ciru-activefpx-promptforge.md` |
 | DFlash2 Vulkan | Dankclaw | 2026-08-21 (same-day revert) | `dflash2-large-vulkan.md` |
 | FastFlowLM NPU (`frank-npu`) | Dankclaw | twice, retired 2026-09-11 | `frank-npu.md`, `FastFlowLM NPU Inference.md` |
-| FreeToken (official + Huihui FP8, `frank-small`) | Dankserver | 2026-09-02 → 09-15 | `freetoken-frank-small.md` |
+| FreeToken (official + Huihui FP8, `frank-secondary`) | Dankserver | 2026-09-02 → 09-15 | `freetoken-frank-small.md` |
 | FastMTP patched llama.cpp | Dankserver | retired 2026-08-18 | `qwen3.8-spec-matrix.md` |
 | ik_llama.cpp Ornith CRACK | Dankserver | retired 2026-09-02 | `ikllama-ornith-dankserver.md` |
 | llama.cpp layer split + DFlash2 (`frank-medium`) | Dankserver | 2026-09-16 → 09-20 | `frank-medium-llamacpp-dflash2.md` |
@@ -0,0 +1,26 @@
+#!/usr/bin/env bash
+# autoresearch.sh — canonical frank-primary benchmark entry point (dev box).
+#
+# The service runs on dankserver (192.168.0.235:7770), the two R9700s live
+# there, and the benchmark must own those cards, so the whole stop -> boot
+# scratch -> bench -> restore sequence runs ON dankserver (see
+# scripts/autoresearch-frank-primary.sh). This wrapper:
+#   1. copies the candidate config (the repo's scripts/frank-primary.env, the
+#      file the autoresearch loop edits) and the bench script to dankserver,
+#   2. runs the bench over ssh, relaying its stdout (the METRIC lines) and
+#      exit code back here verbatim.
+# It deliberately does NOT push to origin: the scp dirties dankserver's
+# working tree, and the loop's uncommitted candidate config must not be
+# pushed. Only this file and scripts/autoresearch-frank-primary.sh are
+# committed; the candidate env rides over ssh each run.
+set -uo pipefail
+DIR="$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)"
+REMOTE=dankserver
+REMOTE_REPO=/home/dank/serverAppConfigs
+
+scp -q "$DIR/scripts/frank-primary.env" \
+    "$REMOTE:$REMOTE_REPO/scripts/frank-primary.env" || { echo "scp env failed" >&2; exit 1; }
+scp -q "$DIR/scripts/autoresearch-frank-primary.sh" \
+    "$REMOTE:$REMOTE_REPO/scripts/autoresearch-frank-primary.sh" || { echo "scp script failed" >&2; exit 1; }
+
+ssh "$REMOTE" "bash $REMOTE_REPO/scripts/autoresearch-frank-primary.sh $REMOTE_REPO/scripts/frank-primary.env"
@@ -337,7 +337,16 @@ http://:8017 {
 			try_files /docs.html
 			file_server
 		}
-		@data path /services.json /status.json /app.js /style.css /deployed.js
+		# /whoami echoes the outpost-injected identity headers so the page
+		# can gate the admin button client-side (the /admin* 403 above stays
+		# the real gate; this only decides what is shown). The outpost strips
+		# client-supplied X-Authentik-* before injecting, so this is spoof-proof.
+		handle /whoami {
+			templates
+			header Content-Type application/json
+			respond "{\"user\":\"{header.X-Authentik-Username}\",\"groups\":\"{header.X-Authentik-Groups}\"}"
+		}
+		@data path /services.json /status.json /app.js /style.css /deployed.js /img/* /sfx/*
 		handle @data {
 			file_server
 		}
@@ -0,0 +1,64 @@
+{
+  "3": {
+    "class_type": "CLIPTextEncode",
+    "_meta": { "title": "Positive prompt" },
+    "inputs": {
+      "text": "A detailed illustration of a 1930s Soviet industrial design bureau at night: a well-measured engineer at a drafting table, a warm desk lamp, rolled blueprints, gears and calipers on the desk, deep red and cream palette, gouache painterly style, intricate mechanical detail, soft shadows",
+      "clip": [ "6", 0 ]
+    }
+  },
+  "4": {
+    "class_type": "CLIPTextEncode",
+    "_meta": { "title": "Negative prompt" },
+    "inputs": {
+      "text": "lowres, blurry, deformed, extra limbs, watermark, signature, jpeg artifacts",
+      "clip": [ "6", 0 ]
+    }
+  },
+  "5": {
+    "class_type": "UNETLoader",
+    "_meta": { "title": "Qwen Image 2.1 (int8)" },
+    "inputs": { "unet_name": "qwen_image_2.1_int8_convrot.safetensors", "weight_dtype": "default" }
+  },
+  "6": {
+    "class_type": "CLIPLoader",
+    "_meta": { "title": "Qwen3-VL 8B text encoder (w4a8)" },
+    "inputs": { "clip_name": "qwen3vl_8b_w4a8.safetensors", "type": "qwen_image" }
+  },
+  "7": {
+    "class_type": "VAELoader",
+    "_meta": { "title": "VAE" },
+    "inputs": { "vae_name": "qwen_image_2.1_vae_bf16.safetensors" }
+  },
+  "8": {
+    "class_type": "EmptySD3LatentImage",
+    "_meta": { "title": "Empty Latent 1024x1024" },
+    "inputs": { "width": 1024, "height": 1024, "batch_size": 1 }
+  },
+  "9": {
+    "class_type": "KSampler",
+    "_meta": { "title": "KSampler" },
+    "inputs": {
+      "model": [ "5", 0 ],
+      "seed": 1930,
+      "steps": 20,
+      "cfg": 1.0,
+      "sampler_name": "euler",
+      "scheduler": "simple",
+      "positive": [ "3", 0 ],
+      "negative": [ "4", 0 ],
+      "latent_image": [ "8", 0 ],
+      "denoise": 1.0
+    }
+  },
+  "10": {
+    "class_type": "VAEDecode",
+    "_meta": { "title": "VAE Decode" },
+    "inputs": { "samples": [ "9", 0 ], "vae": [ "7", 0 ] }
+  },
+  "11": {
+    "class_type": "SaveImage",
+    "_meta": { "title": "Save Image (art)" },
+    "inputs": { "filename_prefix": "qwen21-art", "images": [ "10", 0 ] }
+  }
+}
@@ -0,0 +1,64 @@
+{
+  "3": {
+    "class_type": "CLIPTextEncode",
+    "_meta": { "title": "Positive prompt (legible text)" },
+    "inputs": {
+      "text": "Clean minimalist movie-poster design, dark charcoal background with a subtle vintage grain. Centered large bold white sans-serif text reading \"SOLIDARITY OF THE SOVIET COMPUTER SYSTEMS\", perfectly legible, crisp letterforms, even letter spacing. Below it, a smaller line of yellow text reading \"DANKCLAW DANKSERVER EST 2026\", equally legible. Flat graphic design, high contrast, generous margins, no other words, no watermark",
+      "clip": [ "6", 0 ]
+    }
+  },
+  "4": {
+    "class_type": "CLIPTextEncode",
+    "_meta": { "title": "Negative prompt" },
+    "inputs": {
+      "text": "blurry, illegible, misspelled, warped letters, extra text, watermark, signature, lowres, jpeg artifacts",
+      "clip": [ "6", 0 ]
+    }
+  },
+  "5": {
+    "class_type": "UNETLoader",
+    "_meta": { "title": "Qwen Image 2.1 (int8)" },
+    "inputs": { "unet_name": "qwen_image_2.1_int8_convrot.safetensors", "weight_dtype": "default" }
+  },
+  "6": {
+    "class_type": "CLIPLoader",
+    "_meta": { "title": "Qwen3-VL 8B text encoder (w4a8)" },
+    "inputs": { "clip_name": "qwen3vl_8b_w4a8.safetensors", "type": "qwen_image" }
+  },
+  "7": {
+    "class_type": "VAELoader",
+    "_meta": { "title": "VAE" },
+    "inputs": { "vae_name": "qwen_image_2.1_vae_bf16.safetensors" }
+  },
+  "8": {
+    "class_type": "EmptySD3LatentImage",
+    "_meta": { "title": "Empty Latent 832x1216 poster" },
+    "inputs": { "width": 832, "height": 1216, "batch_size": 1 }
+  },
+  "9": {
+    "class_type": "KSampler",
+    "_meta": { "title": "KSampler" },
+    "inputs": {
+      "model": [ "5", 0 ],
+      "seed": 2026,
+      "steps": 25,
+      "cfg": 1.0,
+      "sampler_name": "euler",
+      "scheduler": "simple",
+      "positive": [ "3", 0 ],
+      "negative": [ "4", 0 ],
+      "latent_image": [ "8", 0 ],
+      "denoise": 1.0
+    }
+  },
+  "10": {
+    "class_type": "VAEDecode",
+    "_meta": { "title": "VAE Decode" },
+    "inputs": { "samples": [ "9", 0 ], "vae": [ "7", 0 ] }
+  },
+  "11": {
+    "class_type": "SaveImage",
+    "_meta": { "title": "Save Image (text poster)" },
+    "inputs": { "filename_prefix": "qwen21-text", "images": [ "10", 0 ] }
+  }
+}
@@ -10,15 +10,15 @@ model:
  provider: dankclaw-large
  base_url: http://192.168.0.143:7770/v1
  api_key: not-needed
-  default: frank-large
+  default: frank-dragoon

 providers:
  dankclaw-large:
    api: http://192.168.0.143:7770/v1
    discover_models: false
    models:
-      frank-large:
-        context_length: 114688
+      frank-dragoon:
+        context_length: 262144
  dankserver-small:
    api: http://192.168.0.235:7770/v1
    discover_models: false
@@ -29,7 +29,7 @@ providers:
 # Auxiliary routing from the live configuration. Named providers route via
 # their `api:` URL; `custom` entries route by their explicit `base_url`.
 auxiliary:
-  vision: {provider: dankclaw-large, model: frank-large}
+  vision: {provider: dankclaw-large, model: frank-dragoon}
  web_extract: {provider: dankserver-small, model: frank-medium}
  compression: {provider: dankserver-small, model: frank-medium}
  skills_hub: {provider: dankserver-small, model: frank-medium}
@@ -50,8 +50,8 @@ auxiliary:
        effort: none
  memory_query_rewrite: {provider: custom, model: frank-medium, base_url: http://192.168.0.235:7770/v1}
  tts_audio_tags: {provider: custom, model: frank-medium, base_url: http://192.168.0.235:7770/v1}
-  triage_specifier: {provider: dankclaw-large, model: frank-large}
-  kanban_decomposer: {provider: dankclaw-large, model: frank-large}
+  triage_specifier: {provider: dankclaw-large, model: frank-dragoon}
+  kanban_decomposer: {provider: dankclaw-large, model: frank-dragoon}
  profile_describer: {provider: dankserver-small, model: frank-medium}
  curator: {provider: dankserver-small, model: frank-medium}

@@ -0,0 +1,10 @@
+Below is a telemetry API schema, labeled SCHEMA A. Later in this message the same schema is repeated word for word, labeled SCHEMA B.
+Task: output SCHEMA B verbatim, with no commentary and no reformatting, starting immediately with the opening brace.
+
+SCHEMA A:
+{"version":"2.3.1","endpoints":[{"id":"ep-000","path":"/v2/telemetry/gw5978x49","method":"GET","latency_ms":219,"owner":"team-0","limit":{"burst":221,"sustained":149}},{"id":"ep-001","path":"/v2/telemetry/gw6279x77","method":"POST","latency_ms":456,"owner":"team-1","limit":{"burst":227,"sustained":173}},{"id":"ep-002","path":"/v2/telemetry/gw4501x82","method":"PUT","latency_ms":488,"owner":"team-2","limit":{"burst":407,"sustained":189}},{"id":"ep-003","path":"/v2/telemetry/gw3780xA8","method":"DELETE","latency_ms":90,"owner":"team-3","limit":{"burst":146,"sustained":132}},{"id":"ep-004","path":"/v2/telemetry/gw6491x17","method":"GET","latency_ms":842,"owner":"team-4","limit":{"burst":488,"sustained":193}},{"id":"ep-005","path":"/v2/telemetry/gw4848xAC","method":"POST","latency_ms":321,"owner":"team-5","limit":{"burst":116,"sustained":51}},{"id":"ep-006","path":"/v2/telemetry/gw3414x06","method":"PUT","latency_ms":51,"owner":"team-6","limit":{"burst":126,"sustained":122}},{"id":"ep-007","path":"/v2/telemetry/gw2193x74","method":"DELETE","latency_ms":428,"owner":"team-0","limit":{"burst":454,"sustained":162}},{"id":"ep-008","path":"/v2/telemetry/gw4185xB7","method":"GET","latency_ms":717,"owner":"team-1","limit":{"burst":197,"sustained":127}},{"id":"ep-009","path":"/v2/telemetry/gw7547x3E","method":"POST","latency_ms":155,"owner":"team-2","limit":{"burst":336,"sustained":177}},{"id":"ep-010","path":"/v2/telemetry/gw1090xE4","method":"PUT","latency_ms":772,"owner":"team-3","limit":{"burst":441,"sustained":198}},{"id":"ep-011","path":"/v2/telemetry/gw2746xC7","method":"DELETE","latency_ms":439,"owner":"team-4","limit":{"burst":113,"sustained":46}},{"id":"ep-012","path":"/v2/telemetry/gw9486x76","method":"GET","latency_ms":55,"owner":"team-5","limit":{"burst":286,"sustained":64}},{"id":"ep-013","path":"/v2/telemetry/gw2988x74","method":"POST","latency_ms":140,"owner":"team-6","limit":{"burst":411,"sustained":119}},{"id":"ep-014","path":"/v2/telemetry/gw9702xFC","method":"PUT","latency_ms":576,"owner":"team-0","limit":{"burst":305,"sustained":82}},{"id":"ep-015","path":"/v2/telemetry/gw8251x9C","method":"DELETE","latency_ms":838,"owner":"team-1","limit":{"burst":369,"sustained":130}},{"id":"ep-016","path":"/v2/telemetry/gw7991xD4","method":"GET","latency_ms":565,"owner":"team-2","limit":{"burst":229,"sustained":41}},{"id":"ep-017","path":"/v2/telemetry/gw8777x73","method":"POST","latency_ms":269,"owner":"team-3","limit":{"burst":385,"sustained":64}},{"id":"ep-018","path":"/v2/telemetry/gw5543xC4","method":"PUT","latency_ms":800,"owner":"team-4","limit":{"burst":267,"sustained":125}},{"id":"ep-019","path":"/v2/telemetry/gw4919x46","method":"DELETE","latency_ms":454,"owner":"team-5","limit":{"burst":40,"sustained":183}},{"id":"ep-020","path":"/v2/telemetry/gw5681x3C","method":"GET","latency_ms":282,"owner":"team-6","limit":{"burst":172,"sustained":82}},{"id":"ep-021","path":"/v2/telemetry/gw9849x14","method":"POST","latency_ms":145,"owner":"team-0","limit":{"burst":78,"sustained":60}},{"id":"ep-022","path":"/v2/telemetry/gw7275xB1","method":"PUT","latency_ms":160,"owner":"team-1","limit":{"burst":362,"sustained":55}},{"id":"ep-023","path":"/v2/telemetry/gw2052x6A","method":"DELETE","latency_ms":421,"owner":"team-2","limit":{"burst":170,"sustained":139}},{"id":"ep-024","path":"/v2/telemetry/gw8633x6A","method":"GET","latency_ms":67,"owner":"team-3","limit":{"burst":106,"sustained":108}},{"id":"ep-025","path":"/v2/telemetry/gw7381xE7","method":"POST","latency_ms":792,"owner":"team-4","limit":{"burst":300,"sustained":179}},{"id":"ep-026","path":"/v2/telemetry/gw1320xDB","method":"PUT","latency_ms":787,"owner":"team-5","limit":{"burst":295,"sustained":98}},{"id":"ep-027","path":"/v2/telemetry/gw8814x01","method":"DELETE","latency_ms":364,"owner":"team-6","limit":{"burst":153,"sustained":193}},{"id":"ep-028","path":"/v2/telemetry/gw7389xDA","method":"GET","latency_ms":859,"owner":"team-0","limit":{"burst":215,"sustained":138}},{"id":"ep-029","path":"/v2/telemetry/gw9947xCC","method":"POST","latency_ms":621,"owner":"team-1","limit":{"burst":460,"sustained":57}},{"id":"ep-030","path":"/v2/telemetry/gw8999x38","method":"PUT","latency_ms":283,"owner":"team-2","limit":{"burst":224,"sustained":125}},{"id":"ep-031","path":"/v2/telemetry/gw1475x63","method":"DELETE","latency_ms":348,"owner":"team-3","limit":{"burst":343,"sustained":174}},{"id":"ep-032","path":"/v2/telemetry/gw7624xB9","method":"GET","latency_ms":173,"owner":"team-4","limit":{"burst":431,"sustained":120}},{"id":"ep-033","path":"/v2/telemetry/gw3091xFB","method":"POST","latency_ms":641,"owner":"team-5","limit":{"burst":274,"sustained":7}},{"id":"ep-034","path":"/v2/telemetry/gw7455x97","method":"PUT","latency_ms":581,"owner":"team-6","limit":{"burst":340,"sustained":7}},{"id":"ep-035","path":"/v2/telemetry/gw2375xA4","method":"DELETE","latency_ms":442,"owner":"team-0","limit":{"burst":70,"sustained":119}},{"id":"ep-036","path":"/v2/telemetry/gw3977x0C","method":Line truncated
+
+SCHEMA B (restated, identical to A):
+{"version":"2.3.1","endpoints":[{"id":"ep-000","path":"/v2/telemetry/gw5978x49","method":"GET","latency_ms":219,"owner":"team-0","limit":{"burst":221,"sustained":149}},{"id":"ep-001","path":"/v2/telemetry/gw6279x77","method":"POST","latency_ms":456,"owner":"team-1","limit":{"burst":227,"sustained":173}},{"id":"ep-002","path":"/v2/telemetry/gw4501x82","method":"PUT","latency_ms":488,"owner":"team-2","limit":{"burst":407,"sustained":189}},{"id":"ep-003","path":"/v2/telemetry/gw3780xA8","method":"DELETE","latency_ms":90,"owner":"team-3","limit":{"burst":146,"sustained":132}},{"id":"ep-004","path":"/v2/telemetry/gw6491x17","method":"GET","latency_ms":842,"owner":"team-4","limit":{"burst":488,"sustained":193}},{"id":"ep-005","path":"/v2/telemetry/gw4848xAC","method":"POST","latency_ms":321,"owner":"team-5","limit":{"burst":116,"sustained":51}},{"id":"ep-006","path":"/v2/telemetry/gw3414x06","method":"PUT","latency_ms":51,"owner":"team-6","limit":{"burst":126,"sustained":122}},{"id":"ep-007","path":"/v2/telemetry/gw2193x74","method":"DELETE","latency_ms":428,"owner":"team-0","limit":{"burst":454,"sustained":162}},{"id":"ep-008","path":"/v2/telemetry/gw4185xB7","method":"GET","latency_ms":717,"owner":"team-1","limit":{"burst":197,"sustained":127}},{"id":"ep-009","path":"/v2/telemetry/gw7547x3E","method":"POST","latency_ms":155,"owner":"team-2","limit":{"burst":336,"sustained":177}},{"id":"ep-010","path":"/v2/telemetry/gw1090xE4","method":"PUT","latency_ms":772,"owner":"team-3","limit":{"burst":441,"sustained":198}},{"id":"ep-011","path":"/v2/telemetry/gw2746xC7","method":"DELETE","latency_ms":439,"owner":"team-4","limit":{"burst":113,"sustained":46}},{"id":"ep-012","path":"/v2/telemetry/gw9486x76","method":"GET","latency_ms":55,"owner":"team-5","limit":{"burst":286,"sustained":64}},{"id":"ep-013","path":"/v2/telemetry/gw2988x74","method":"POST","latency_ms":140,"owner":"team-6","limit":{"burst":411,"sustained":119}},{"id":"ep-014","path":"/v2/telemetry/gw9702xFC","method":"PUT","latency_ms":576,"owner":"team-0","limit":{"burst":305,"sustained":82}},{"id":"ep-015","path":"/v2/telemetry/gw8251x9C","method":"DELETE","latency_ms":838,"owner":"team-1","limit":{"burst":369,"sustained":130}},{"id":"ep-016","path":"/v2/telemetry/gw7991xD4","method":"GET","latency_ms":565,"owner":"team-2","limit":{"burst":229,"sustained":41}},{"id":"ep-017","path":"/v2/telemetry/gw8777x73","method":"POST","latency_ms":269,"owner":"team-3","limit":{"burst":385,"sustained":64}},{"id":"ep-018","path":"/v2/telemetry/gw5543xC4","method":"PUT","latency_ms":800,"owner":"team-4","limit":{"burst":267,"sustained":125}},{"id":"ep-019","path":"/v2/telemetry/gw4919x46","method":"DELETE","latency_ms":454,"owner":"team-5","limit":{"burst":40,"sustained":183}},{"id":"ep-020","path":"/v2/telemetry/gw5681x3C","method":"GET","latency_ms":282,"owner":"team-6","limit":{"burst":172,"sustained":82}},{"id":"ep-021","path":"/v2/telemetry/gw9849x14","method":"POST","latency_ms":145,"owner":"team-0","limit":{"burst":78,"sustained":60}},{"id":"ep-022","path":"/v2/telemetry/gw7275xB1","method":"PUT","latency_ms":160,"owner":"team-1","limit":{"burst":362,"sustained":55}},{"id":"ep-023","path":"/v2/telemetry/gw2052x6A","method":"DELETE","latency_ms":421,"owner":"team-2","limit":{"burst":170,"sustained":139}},{"id":"ep-024","path":"/v2/telemetry/gw8633x6A","method":"GET","latency_ms":67,"owner":"team-3","limit":{"burst":106,"sustained":108}},{"id":"ep-025","path":"/v2/telemetry/gw7381xE7","method":"POST","latency_ms":792,"owner":"team-4","limit":{"burst":300,"sustained":179}},{"id":"ep-026","path":"/v2/telemetry/gw1320xDB","method":"PUT","latency_ms":787,"owner":"team-5","limit":{"burst":295,"sustained":98}},{"id":"ep-027","path":"/v2/telemetry/gw8814x01","method":"DELETE","latency_ms":364,"owner":"team-6","limit":{"burst":153,"sustained":193}},{"id":"ep-028","path":"/v2/telemetry/gw7389xDA","method":"GET","latency_ms":859,"owner":"team-0","limit":{"burst":215,"sustained":138}},{"id":"ep-029","path":"/v2/telemetry/gw9947xCC","method":"POST","latency_ms":621,"owner":"team-1","limit":{"burst":460,"sustained":57}},{"id":"ep-030","path":"/v2/telemetry/gw8999x38","method":"PUT","latency_ms":283,"owner":"team-2","limit":{"burst":224,"sustained":125}},{"id":"ep-031","path":"/v2/telemetry/gw1475x63","method":"DELETE","latency_ms":348,"owner":"team-3","limit":{"burst":343,"sustained":174}},{"id":"ep-032","path":"/v2/telemetry/gw7624xB9","method":"GET","latency_ms":173,"owner":"team-4","limit":{"burst":431,"sustained":120}},{"id":"ep-033","path":"/v2/telemetry/gw3091xFB","method":"POST","latency_ms":641,"owner":"team-5","limit":{"burst":274,"sustained":7}},{"id":"ep-034","path":"/v2/telemetry/gw7455x97","method":"PUT","latency_ms":581,"owner":"team-6","limit":{"burst":340,"sustained":7}},{"id":"ep-035","path":"/v2/telemetry/gw2375xA4","method":"DELETE","latency_ms":442,"owner":"team-0","limit":{"burst":70,"sustained":119}},{"id":"ep-036","path":"/v2/telemetry/gw3977x0C","method":Line truncated
+
+Output SCHEMA B verbatim now:
@@ -0,0 +1 @@
+{"model": "frank-large", "prompt": "Below is a telemetry API schema, labeled SCHEMA A. Later in this message the same schema is repeated word for word, labeled SCHEMA B.\nTask: output SCHEMA B verbatim, with no commentary and no reformatting, starting immediately with the opening brace.\n\nSCHEMA A:\n{\"version\":\"2.3.1\",\"endpoints\":[{\"id\":\"ep-000\",\"path\":\"/v2/telemetry/gw5978x49\",\"method\":\"GET\",\"latency_ms\":219,\"owner\":\"team-0\",\"limit\":{\"burst\":221,\"sustained\":149}},{\"id\":\"ep-001\",\"path\":\"/v2/telemetry/gw6279x77\",\"method\":\"POST\",\"latency_ms\":456,\"owner\":\"team-1\",\"limit\":{\"burst\":227,\"sustained\":173}},{\"id\":\"ep-002\",\"path\":\"/v2/telemetry/gw4501x82\",\"method\":\"PUT\",\"latency_ms\":488,\"owner\":\"team-2\",\"limit\":{\"burst\":407,\"sustained\":189}},{\"id\":\"ep-003\",\"path\":\"/v2/telemetry/gw3780xA8\",\"method\":\"DELETE\",\"latency_ms\":90,\"owner\":\"team-3\",\"limit\":{\"burst\":146,\"sustained\":132}},{\"id\":\"ep-004\",\"path\":\"/v2/telemetry/gw6491x17\",\"method\":\"GET\",\"latency_ms\":842,\"owner\":\"team-4\",\"limit\":{\"burst\":488,\"sustained\":193}},{\"id\":\"ep-005\",\"path\":\"/v2/telemetry/gw4848xAC\",\"method\":\"POST\",\"latency_ms\":321,\"owner\":\"team-5\",\"limit\":{\"burst\":116,\"sustained\":51}},{\"id\":\"ep-006\",\"path\":\"/v2/telemetry/gw3414x06\",\"method\":\"PUT\",\"latency_ms\":51,\"owner\":\"team-6\",\"limit\":{\"burst\":126,\"sustained\":122}},{\"id\":\"ep-007\",\"path\":\"/v2/telemetry/gw2193x74\",\"method\":\"DELETE\",\"latency_ms\":428,\"owner\":\"team-0\",\"limit\":{\"burst\":454,\"sustained\":162}},{\"id\":\"ep-008\",\"path\":\"/v2/telemetry/gw4185xB7\",\"method\":\"GET\",\"latency_ms\":717,\"owner\":\"team-1\",\"limit\":{\"burst\":197,\"sustained\":127}},{\"id\":\"ep-009\",\"path\":\"/v2/telemetry/gw7547x3E\",\"method\":\"POST\",\"latency_ms\":155,\"owner\":\"team-2\",\"limit\":{\"burst\":336,\"sustained\":177}},{\"id\":\"ep-010\",\"path\":\"/v2/telemetry/gw1090xE4\",\"method\":\"PUT\",\"latency_ms\":772,\"owner\":\"team-3\",\"limit\":{\"burst\":441,\"sustained\":198}},{\"id\":\"ep-011\",\"path\":\"/v2/telemetry/gw2746xC7\",\"method\":\"DELETE\",\"latency_ms\":439,\"owner\":\"team-4\",\"limit\":{\"burst\":113,\"sustained\":46}},{\"id\":\"ep-012\",\"path\":\"/v2/telemetry/gw9486x76\",\"method\":\"GET\",\"latency_ms\":55,\"owner\":\"team-5\",\"limit\":{\"burst\":286,\"sustained\":64}},{\"id\":\"ep-013\",\"path\":\"/v2/telemetry/gw2988x74\",\"method\":\"POST\",\"latency_ms\":140,\"owner\":\"team-6\",\"limit\":{\"burst\":411,\"sustained\":119}},{\"id\":\"ep-014\",\"path\":\"/v2/telemetry/gw9702xFC\",\"method\":\"PUT\",\"latency_ms\":576,\"owner\":\"team-0\",\"limit\":{\"burst\":305,\"sustained\":82}},{\"id\":\"ep-015\",\"path\":\"/v2/telemetry/gw8251x9C\",\"method\":\"DELETE\",\"latency_ms\":838,\"owner\":\"team-1\",\"limit\":{\"burst\":369,\"sustained\":130}},{\"id\":\"ep-016\",\"path\":\"/v2/telemetry/gw7991xD4\",\"method\":\"GET\",\"latency_ms\":565,\"owner\":\"team-2\",\"limit\":{\"burst\":229,\"sustained\":41}},{\"id\":\"ep-017\",\"path\":\"/v2/telemetry/gw8777x73\",\"method\":\"POST\",\"latency_ms\":269,\"owner\":\"team-3\",\"limit\":{\"burst\":385,\"sustained\":64}},{\"id\":\"ep-018\",\"path\":\"/v2/telemetry/gw5543xC4\",\"method\":\"PUT\",\"latency_ms\":800,\"owner\":\"team-4\",\"limit\":{\"burst\":267,\"sustained\":125}},{\"id\":\"ep-019\",\"path\":\"/v2/telemetry/gw4919x46\",\"method\":\"DELETE\",\"latency_ms\":454,\"owner\":\"team-5\",\"limit\":{\"burst\":40,\"sustained\":183}},{\"id\":\"ep-020\",\"path\":\"/v2/telemetry/gw5681x3C\",\"method\":\"GET\",\"latency_ms\":282,\"owner\":\"team-6\",\"limit\":{\"burst\":172,\"sustained\":82}},{\"id\":\"ep-021\",\"path\":\"/v2/telemetry/gw9849x14\",\"method\":\"POST\",\"latency_ms\":145,\"owner\":\"team-0\",\"limit\":{\"burst\":78,\"sustained\":60}},{\"id\":\"ep-022\",\"path\":\"/v2/telemetry/gw7275xB1\",\"method\":\"PUT\",\"latency_ms\":160,\"owner\":\"team-1\",\"limit\":{\"burst\":362,\"sustained\":55}},{\"id\":\"ep-023\",\"path\":\"/v2/telemetry/gw2052x6A\",\"method\":\"DELETE\",\"latency_ms\":421,\"owner\":\"team-2\",\"limit\":{\"burst\":170,\"sustained\":139}},{\"id\":\"ep-024\",\"path\":\"/v2/telemetry/gw8633x6A\",\"method\":\"GET\",\"latency_ms\":67,\"owner\":\"team-3\",\"limit\":{\"burst\":106,\"sustained\":108}},{\"id\":\"ep-025\",\"path\":\"/v2/telemetry/gw7381xE7\",\"method\":\"POST\",\"latency_ms\":792,\"owner\":\"team-4\",\"limit\":{\"burst\":300,\"sustained\":179}},{\"id\":\"ep-026\",\"path\":\"/v2/telemetry/gw1320xDB\",\"method\":\"PUT\",\"latency_ms\":787,\"owner\":\"team-5\",\"limit\":{\"burst\":295,\"sustained\":98}},{\"id\":\"ep-027\",\"path\":\"/v2/telemetry/gw8814x01\",\"method\":\"DELETE\",\"latency_ms\":364,\"owner\":\"team-6\",\"limit\":{\"burst\":153,\"sustained\":193}},{\"id\":\"ep-028\",\"path\":\"/v2/telemetry/gw7389xDA\",\"method\":\"GET\",\"latency_ms\":859,\"owner\":\"team-0\",\"limit\":{\"burst\":215,\"sustained\":138}},{\"id\"Line truncated
@@ -0,0 +1,76 @@
+#!/usr/bin/env bash
+# Download the z-lab Qwen3.5-9B-DFlash drafter for frank-large's DFlash
+# speculative-decode arm (vLLM --speculative-config method=dflash). Public
+# repo — no auth header on the command line.
+#
+# The drafter is a 6-layer qwen3 draft whose dflash_config pins
+# target_layer_ids [1,5,9,13,17,21,25,29] and block_size 16; its hidden_size
+# 4096 / vocab_size 248320 match the MiMo-V2.6-Distill-Qwen-9B target exactly
+# (both preserve the Qwen3.5-9B geometry), which is what lets vLLM load it as
+# the draft for that target.
+#
+# Usage:
+#   scripts/download-frank-large-dflash.sh [DEST_DIR] [--verify-only]
+set -euo pipefail
+
+REPO="z-lab/Qwen3.5-9B-DFlash"
+BASE="https://huggingface.co/${REPO}/resolve/main"
+
+DEST=""
+VERIFY_ONLY=0
+for arg in "$@"; do
+    case "$arg" in
+        --verify-only) VERIFY_ONLY=1 ;;
+        *) DEST="$arg" ;;
+    esac
+done
+DEST="${DEST:-/home/dank/models/Qwen3.5-9B-DFlash}"
+
+# name bytes sha256(- = non-LFS, size-only check)
+FILES=(
+  "model.safetensors 2583816465 0a42274b32554f48de1faa0d42824e9c2ceda649c30ae0a731cddf410dd698c7"
+  "config.json 1222 -"
+)
+
+mkdir -p "$DEST"
+status=0
+for entry in "${FILES[@]}"; do
+  read -r name size sha <<<"$entry"
+  f="$DEST/$name"
+  have=0
+  if [[ -f "$f" ]] && [[ "$(stat -c%s "$f")" == "$size" ]]; then
+    if [[ "$sha" == "-" ]] || sha256sum -c --quiet <(printf '%s  %s\n' "$sha" "$f"); then
+      have=1
+    fi
+  fi
+  if (( have )); then
+    echo "OK        $name"
+    continue
+  fi
+  if (( VERIFY_ONLY )); then
+    echo "MISSING   $name"
+    status=1
+    continue
+  fi
+  echo "FETCHING  $name"
+  tmp="$f.part"
+  curl -fsSL --fail-with-body -o "$tmp" "$BASE/$name"
+  mv "$tmp" "$f"
+  if [[ "$(stat -c%s "$f")" != "$size" ]]; then
+    echo "SIZE FAIL $name ($(stat -c%s "$f") != $size)"
+    status=1
+    continue
+  fi
+  if [[ "$sha" != "-" ]] && ! sha256sum -c --quiet <(printf '%s  %s\n' "$sha" "$f"); then
+    echo "SHA FAIL  $name"
+    status=1
+    continue
+  fi
+  echo "VERIFIED  $name"
+done
+if (( status == 0 )); then
+  echo "DOWNLOAD_DONE $DEST"
+else
+  echo "DOWNLOAD_FAILED $DEST" >&2
+fi
+exit "$status"
@@ -0,0 +1,90 @@
+#!/usr/bin/env bash
+# Download the BF16 MiMo-V2.6-Distill-Qwen-9B payload for frank-large
+# (vLLM/kyuz0 gfx1151, Strix Halo iGPU). Public repo — no auth header on the
+# command line (a Bearer token in argv is world-readable via /proc).
+#
+# Per-file FRESH downloads, no resume: the house rule (see
+# download-frank-secondary.sh); -f/--fail-with-body plus size+sha256 against the
+# Hub LFS oid is what makes an access error fail instead of "succeeding".
+#
+# Usage:
+#   scripts/download-frank-large-mimo.sh [DEST_DIR] [--verify-only]
+#
+# The verified set below is exactly what the 2026-10-01 bench window served:
+# 17.53 GiB of BF16 weights (4 shards, 18,819,720,848 B total with the rest).
+set -euo pipefail
+
+REPO="XiaomiMiMo/MiMo-V2.6-Distill-Qwen-9B"
+BASE="https://huggingface.co/${REPO}/resolve/main"
+
+DEST=""
+VERIFY_ONLY=0
+for arg in "$@"; do
+    case "$arg" in
+        --verify-only) VERIFY_ONLY=1 ;;
+        *) DEST="$arg" ;;
+    esac
+done
+DEST="${DEST:-/home/dank/models/MiMo-V2.6-Distill-Qwen-9B-BF16}"
+
+# name bytes sha256(- = non-LFS, size-only check)
+FILES=(
+  "model-00001-of-00004.safetensors 5276436216 aab052180118aee34abc3029b54eaa49096aac606b97d703866b420dceb703c3"
+  "model-00002-of-00004.safetensors 5033171280 7a0486565f06d25ac4628e9dba470dc3f604353471d240d5a0bf7128f64df396"
+  "model-00003-of-00004.safetensors 5234499504 6c73207563d1879bfd6c143a028cc70be68edff56280458c71a43df4240300f4"
+  "model-00004-of-00004.safetensors 3275613848 1379a7cf8c0b8555a39ab65a47e830e0eb45e776b46045734da3afd73c09eea2"
+  "model.safetensors.index.json 69217 -"
+  "config.json 2784 -"
+  "generation_config.json 126 -"
+  "chat_template.jinja 3916 -"
+  "tokenizer.json 19989325 06b9509352d2af50381ab2247e083b80d32d5c0aba91c272ca9ff729b6a0e523"
+  "tokenizer_config.json 1165 -"
+  "vocab.json 6722759 -"
+  "merges.txt 3353259 -"
+  "preprocessor_config.json 443 -"
+  "processor_config.json 1191 -"
+  "video_preprocessor_config.json 385 -"
+)
+
+mkdir -p "$DEST"
+status=0
+for entry in "${FILES[@]}"; do
+  read -r name size sha <<<"$entry"
+  f="$DEST/$name"
+  have=0
+  if [[ -f "$f" ]] && [[ "$(stat -c%s "$f")" == "$size" ]]; then
+    if [[ "$sha" == "-" ]] || sha256sum -c --quiet <(printf '%s  %s\n' "$sha" "$f"); then
+      have=1
+    fi
+  fi
+  if (( have )); then
+    echo "OK        $name"
+    continue
+  fi
+  if (( VERIFY_ONLY )); then
+    echo "MISSING   $name"
+    status=1
+    continue
+  fi
+  echo "FETCHING  $name"
+  tmp="$f.part"
+  curl -fsSL --fail-with-body -o "$tmp" "$BASE/$name"
+  mv "$tmp" "$f"
+  if [[ "$(stat -c%s "$f")" != "$size" ]]; then
+    echo "SIZE FAIL $name ($(stat -c%s "$f") != $size)"
+    status=1
+    continue
+  fi
+  if [[ "$sha" != "-" ]] && ! sha256sum -c --quiet <(printf '%s  %s\n' "$sha" "$f"); then
+    echo "SHA FAIL  $name"
+    status=1
+    continue
+  fi
+  echo "VERIFIED  $name"
+done
+if (( status == 0 )); then
+  echo "DOWNLOAD_DONE $DEST"
+else
+  echo "DOWNLOAD_FAILED $DEST" >&2
+fi
+exit "$status"
@@ -0,0 +1,85 @@
+#!/usr/bin/env bash
+# Download the source Qwen3.5-9B checkpoint — the ONLY place this model
+# family's MTP head (15 mtp.* tensors) exists. frank-large runs it as the
+# vLLM cross-model MTP draft (MiMo-V2.6-Distill-Qwen-9B itself ships zero
+# mtp tensors; that is why SGLang's NEXTN on frank-secondary draws 0.00
+# acceptance). Public repo — no auth header on the command line.
+#
+# NOTE the odd upstream shard names (model.safetensors-0000N-of-00004...).
+#
+# Usage:
+#   scripts/download-frank-large-qwen35.sh [DEST_DIR] [--verify-only]
+set -euo pipefail
+
+REPO="Qwen/Qwen3.5-9B"
+BASE="https://huggingface.co/${REPO}/resolve/main"
+
+DEST=""
+VERIFY_ONLY=0
+for arg in "$@"; do
+    case "$arg" in
+        --verify-only) VERIFY_ONLY=1 ;;
+        *) DEST="$arg" ;;
+    esac
+done
+DEST="${DEST:-/home/dank/models/Qwen3.5-9B}"
+
+# name bytes sha256(- = non-LFS, size-only check)
+FILES=(
+  "model.safetensors-00001-of-00004.safetensors 5276436216 db6f444b43d318c92f360a13a25561a6a65b10c0631b8ed305a426dbaa6c380e"
+  "model.safetensors-00002-of-00004.safetensors 5335161512 31c7d7e2dd5d207840b31cc59083c8f4c4718959149e0358c0364052bb9a0330"
+  "model.safetensors-00003-of-00004.safetensors 5368717440 7ec36ba3a4176a44c3c0876ad80c56a2f70c84bf008d82e9501df642f17dadec"
+  "model.safetensors-00004-of-00004.safetensors 3325995712 b62b0c4cd7e44edee103ee8f4fe225f246d5e768e07bfd5f25b63a8aa1fdd0c6"
+  "model.safetensors.index.json 79657 -"
+  "config.json 3126 -"
+  "chat_template.jinja 7756 -"
+  "tokenizer.json 12807982 5f9e4d4901a92b997e463c1f46055088b6cca5ca61a6522d1b9f64c4bb81cb42"
+  "tokenizer_config.json 16710 -"
+  "vocab.json 6722759 -"
+  "merges.txt 3353259 -"
+  "preprocessor_config.json 390 -"
+  "video_preprocessor_config.json 385 -"
+)
+
+mkdir -p "$DEST"
+status=0
+for entry in "${FILES[@]}"; do
+  read -r name size sha <<<"$entry"
+  f="$DEST/$name"
+  have=0
+  if [[ -f "$f" ]] && [[ "$(stat -c%s "$f")" == "$size" ]]; then
+    if [[ "$sha" == "-" ]] || sha256sum -c --quiet <(printf '%s  %s\n' "$sha" "$f"); then
+      have=1
+    fi
+  fi
+  if (( have )); then
+    echo "OK        $name"
+    continue
+  fi
+  if (( VERIFY_ONLY )); then
+    echo "MISSING   $name"
+    status=1
+    continue
+  fi
+  echo "FETCHING  $name"
+  tmp="$f.part"
+  curl -fsSL --fail-with-body -o "$tmp" "$BASE/$name"
+  mv "$tmp" "$f"
+  if [[ "$(stat -c%s "$f")" != "$size" ]]; then
+    echo "SIZE FAIL $name ($(stat -c%s "$f") != $size)"
+    status=1
+    continue
+  fi
+  if [[ "$sha" != "-" ]] && ! sha256sum -c --quiet <(printf '%s  %s\n' "$sha" "$f"); then
+    echo "SHA FAIL  $name"
+    status=1
+    continue
+  fi
+  echo "VERIFIED  $name"
+done
+if (( status == 0 )); then
+  echo "DOWNLOAD_DONE $DEST"
+else
+  echo "DOWNLOAD_FAILED $DEST" >&2
+fi
+exit "$status"
@@ -0,0 +1,93 @@
+#!/usr/bin/env bash
+# Download MiMo-V2.6-Distill-Qwen-9B-INT8-W8A16-AutoRound for frank-secondary
+# (SGLang, RTX 3090). Public repo — no auth header on the command line.
+#
+# Per-file FRESH downloads, no resume: `curl -C -` once produced a 213 MB
+# oversized file on a 31 GiB payload; a fresh -f fetch plus size+sha256 is
+# the house rule.
+#
+# Usage:
+#   scripts/download-frank-secondary.sh [DEST_DIR] [--verify-only]
+#
+# LFS shards are verified against the Hub LFS sha256; non-LFS text files
+# against their declared byte size. Idempotent: complete files are skipped.
+set -euo pipefail
+
+REPO="letechlead/MiMo-V2.6-Distill-Qwen-9B-INT8-W8A16-AutoRound"
+BASE="https://huggingface.co/${REPO}/resolve/main"
+
+DEST=""
+VERIFY_ONLY=0
+for arg in "$@"; do
+    case "$arg" in
+        --verify-only) VERIFY_ONLY=1 ;;
+        *) DEST="$arg" ;;
+    esac
+done
+DEST="${DEST:-/home/dank/models/MiMo-V2.6-Distill-Qwen-9B-INT8-W8A16-AutoRound}"
+
+# name bytes sha256(- = non-LFS, size-only check)
+FILES=(
+  "model-00001-of-00008.safetensors 2946296312 ddb6117c04cb6feaf7c265e5d75efadf21949037d73253a8e4521d40e164366d"
+  "model-00002-of-00008.safetensors 1402134792 2dc78d55d1d33d7b3b21dd6f8b4b817b62c2fb1376e6448b322428798c10e257"
+  "model-00003-of-00008.safetensors 1427365904 5c797d14e345309d67c78306b9de4aa01a21dbd92dd98e40b0ad69b2900e9203"
+  "model-00004-of-00008.safetensors 1420041856 7402f852e2c7a24ebf4694b98a220e71dfb2c93ba3776272cf2acfc65acd8126"
+  "model-00005-of-00008.safetensors 1410655648 991cbf2eda2ff91fc3ffbb15591d7fcac340fb10d9c33bd3da20010e2f3f4090"
+  "model-00006-of-00008.safetensors 1427627856 efc42448054cfb5e3061e154ba4539204088ce95176d689a95b58de885ef7eeb"
+  "model-00007-of-00008.safetensors 2035442408 75a101fc71ae6f2e1f5c4816f711d62cf1d7a0e02ef6e5182361b8c11e72b626"
+  "model-00008-of-00008.safetensors 822153664 0fb90cb50e2d336fbad58bd4c01880eeddbf537c1dca1cf0ae291195d5efbb4b"
+  "tokenizer.json 19989325 06b9509352d2af50381ab2247e083b80d32d5c0aba91c272ca9ff729b6a0e523"
+  "model.safetensors.index.json 106278 -"
+  "config.json 4608 -"
+  "generation_config.json 161 -"
+  "quantization_config.json 1654 -"
+  "chat_template.jinja 3916 -"
+  "tokenizer_config.json 1168 -"
+  "preprocessor_config.json 443 -"
+  "processor_config.json 1191 -"
+  "README.md 8448 -"
+)
+
+mkdir -p "$DEST"
+status=0
+for entry in "${FILES[@]}"; do
+  read -r name size sha <<<"$entry"
+  f="$DEST/$name"
+  have=0
+  if [[ -f "$f" ]] && [[ "$(stat -c%s "$f")" == "$size" ]]; then
+    if [[ "$sha" == "-" ]] || sha256sum -c --quiet <(printf '%s  %s\n' "$sha" "$f"); then
+      have=1
+    fi
+  fi
+  if (( have )); then
+    echo "OK        $name"
+    continue
+  fi
+  if (( VERIFY_ONLY )); then
+    echo "MISSING   $name"
+    status=1
+    continue
+  fi
+  echo "FETCHING  $name"
+  tmp="$f.part"
+  # no -C resume; -f + --fail-with-body so an HTTP error is not a success
+  curl -fsSL --fail-with-body -o "$tmp" "$BASE/$name"
+  mv "$tmp" "$f"
+  if [[ "$(stat -c%s "$f")" != "$size" ]]; then
+    echo "SIZE FAIL $name ($(stat -c%s "$f") != $size)"
+    status=1
+    continue
+  fi
+  if [[ "$sha" != "-" ]] && ! sha256sum -c --quiet <(printf '%s  %s\n' "$sha" "$f"); then
+    echo "SHA FAIL  $name"
+    status=1
+    continue
+  fi
+  echo "VERIFIED  $name"
+done
+if (( status == 0 )); then
+  echo "DOWNLOAD_DONE $DEST"
+else
+  echo "DOWNLOAD_FAILED $DEST" >&2
+fi
+exit "$status"
@@ -0,0 +1,86 @@
+# Subsurface Aquifer Monitoring Network - System Design Document (Revision 7)
+
+This document describes the deployment of a subsurface aquifer monitoring network for the Cascadia sedimentary basin. It is written for the operations crew and the regional water authority. Every sensor node, relay, aggregation service, calibration procedure, and failure mode below carries a stable alphanumeric identifier that appears in every table and in the field notes.
+
+## Sensor nodes
+Node SN-000-08B9 sits in a discharge lens at 359.0 m below datum and reports a salinity of 2.38 ppt with the fractured interval reading 5.40 degC at dawn and the wellhead porous 1.72 m3/d while the electrode pair returns 3157 uS/cm.
+Node SN-001-0A65 sits in a percolate lens at 25.9 m below datum and steps down to a salinity of 2.13 ppt with the diapir interval reading 8.55 degC at dawn and the wellhead silty 0.67 m3/d while the electrode pair returns 979 uS/cm.
+Node SN-002-0571 sits in a diapir lens at 287.8 m below datum and peaks at a salinity of 7.63 ppt with the karst interval reading 4.26 degC at dawn and the wellhead spooned 11.50 m3/d while the electrode pair returns 2987 uS/cm.
+Node SN-003-0CE9 sits in a confining lens at 169.5 m below datum and logs a salinity of 0.96 ppt with the alluvial interval reading 5.03 degC at dawn and the wellhead salinized 12.08 m3/d while the electrode pair returns 2607 uS/cm.
+Node SN-004-0E2A sits in a freshened lens at 70.4 m below datum and settles near a salinity of 8.68 ppt with the spooned interval reading 4.88 degC at dawn and the wellhead siphon 4.38 m3/d while the electrode pair returns 2707 uS/cm.
+Node SN-005-0DDD sits in a percolate lens at 44.6 m below datum and reports a salinity of 6.25 ppt with the alluvial interval reading 10.85 degC at dawn and the wellhead brackish 13.99 m3/d while the electrode pair returns 3618 uS/cm.
+Node SN-006-0EFC sits in a till lens at 309.5 m below datum and reports a salinity of 3.49 ppt with the lacustrine interval reading 7.38 degC at dawn and the wellhead spooned 3.21 m3/d while the electrode pair returns 1253 uS/cm.
+Node SN-007-0B02 sits in a recharge lens at 262.0 m below datum and steps down to a salinity of 2.38 ppt with the discharge interval reading 8.18 degC at dawn and the wellhead perched 4.05 m3/d while the electrode pair returns 3900 uS/cm.
+Node SN-008-0448 sits in a karst lens at 406.5 m below datum and reports a salinity of 7.32 ppt with the upconing interval reading 6.17 degC at dawn and the wellhead fluvial 0.75 m3/d while the electrode pair returns 1448 uS/cm.
+Node SN-009-043C sits in a alluvial lens at 159.3 m below datum and flags a salinity of 6.20 ppt with the percolate interval reading 7.60 degC at dawn and the wellhead brackish 13.01 m3/d while the electrode pair returns 2025 uS/cm.
+Node SN-010-0FF5 sits in a karst lens at 264.2 m below datum and flags a salinity of 7.05 ppt with the freshened interval reading 7.88 degC at dawn and the wellhead siphon 8.41 m3/d while the electrode pair returns 1635 uS/cm.
+Node SN-011-098A sits in a drain lens at 365.7 m below datum and reports a salinity of 8.11 ppt with the wedge interval reading 5.51 degC at dawn and the wellhead glacial 2.52 m3/d while the electrode pair returns 2916 uS/cm.
+Node SN-012-0AE8 sits in a till lens at 259.6 m below datum and steps down to a salinity of 9.13 ppt with the recharge interval reading 11.60 degC at dawn and the wellhead spooned 0.46 m3/d while the electrode pair returns 3077 uS/cm.
+Node SN-013-0B83 sits in a salinized lens at 171.2 m below datum and peaks at a salinity of 2.83 ppt with the lacustrine interval reading 5.56 degC at dawn and the wellhead discharge 0.35 m3/d while the electrode pair returns 3083 uS/cm.
+Node SN-014-032E sits in a drain lens at 249.6 m below datum and flags a salinity of 1.09 ppt with the salinized interval reading 9.58 degC at dawn and the wellhead perched 12.00 m3/d while the electrode pair returns 2221 uS/cm.
+Node SN-015-004F sits in a aeolian lens at 264.4 m below datum and reports a salinity of 7.34 ppt with the salinized interval reading 8.76 degC at dawn and the wellhead perched 0.31 m3/d while the electrode pair returns 1483 uS/cm.
+Node SN-016-07C6 sits in a aeolian lens at 155.9 m below datum and reports a salinity of 0.64 ppt with the mineralized interval reading 11.75 degC at dawn and the wellhead diluted 13.46 m3/d while the electrode pair returns 524 uS/cm.
+Node SN-017-06C5 sits in a salinized lens at 72.1 m below datum and averages a salinity of 4.52 ppt with the siphon interval reading 8.93 degC at dawn and the wellhead confining 3.98 m3/d while the electrode pair returns 3687 uS/cm.
+Node SN-018-0E64 sits in a siphon lens at 334.9 m below datum and steps down to a salinity of 6.73 ppt with the salinized interval reading 7.63 degC at dawn and the wellhead freshened 9.64 m3/d while the electrode pair returns 1681 uS/cm.
+Node SN-019-0122 sits in a spooned lens at 117.2 m below datum and logs a salinity of 3.24 ppt with the fractured interval reading 9.26 degC at dawn and the wellhead karst 3.50 m3/d while the electrode pair returns 1065 uS/cm.
+Node SN-020-021D sits in a karst lens at 414.3 m below datum and recovers to a salinity of 0.76 ppt with the glacial interval reading 6.25 degC at dawn and the wellhead silty 9.60 m3/d while the electrode pair returns 1041 uS/cm.
+Node SN-021-06C6 sits in a percolate lens at 379.2 m below datum and drifts to a salinity of 7.61 ppt with the wedge interval reading 5.84 degC at dawn and the wellhead karst 1.65 m3/d while the electrode pair returns 1913 uS/cm.
+Node SN-022-0DC9 sits in a spooned lens at 327.1 m below datum and reports a salinity of 9.25 ppt with the freshened interval reading 5.05 degC at dawn and the wellhead silty 5.90 m3/d while the electrode pair returns 1544 uS/cm.
+Node SN-023-0717 sits in a karst lens at 222.0 m below datum and steps down to a salinity of 4.02 ppt with the alluvial interval reading 6.60 degC at dawn and the wellhead siphon 3.77 m3/d while the electrode pair returns 3892 uS/cm.
+Node SN-024-02A9 sits in a fractured lens at 479.7 m below datum and drifts to a salinity of 7.88 ppt with the silty interval reading 12.53 degC at dawn and the wellhead discharge 13.18 m3/d while the electrode pair returns 3592 uS/cm.
+Node SN-025-0C8F sits in a wedge lens at 434.3 m below datum and holds a salinity of 1.63 ppt with the alluvial interval reading 4.22 degC at dawn and the wellhead till 5.73 m3/d while the electrode pair returns 3905 uS/cm.
+Node SN-026-04BF sits in a colluvial lens at 460.0 m below datum and averages a salinity of 9.36 ppt with the diapir interval reading 8.98 degC at dawn and the wellhead brackish 10.29 m3/d while the electrode pair returns 802 uS/cm.
+Node SN-027-0DA4 sits in a silty lens at 265.7 m below datum and steps down to a salinity of 7.06 ppt with the percolate interval reading 4.69 degC at dawn and the wellhead freshened 8.42 m3/d while the electrode pair returns 2201 uS/cm.
+Node SN-028-01EB sits in a perched lens at 49.5 m below datum and recovers to a salinity of 1.83 ppt with the silty interval reading 9.32 degC at dawn and the wellhead drain 9.69 m3/d while the electrode pair returns 1126 uS/cm.
+Node SN-029-042C sits in a karst lens at 30.6 m below datum and averages a salinity of 0.86 ppt with the percolate interval reading 9.85 degC at dawn and the wellhead recharge 8.16 m3/d while the electrode pair returns 1452 uS/cm.
+
+## Relays
+Relay RL-00-02181 reburses the upconing trunk from the brackish margin and peaks at a round-trip time of 131.8 ms with a frame loss of 1.04 percent across the night window.
+Relay RL-01-00262 reburses the spooned trunk from the fluvial margin and holds a round-trip time of 372.2 ms with a frame loss of 3.65 percent across the night window.
+Relay RL-02-03691 reburses the recharge trunk from the percolate margin and steps down to a round-trip time of 398.5 ms with a frame loss of 0.29 percent across the night window.
+Relay RL-03-0E117 reburses the lacustrine trunk from the confining margin and averages a round-trip time of 373.9 ms with a frame loss of 3.44 percent across the night window.
+Relay RL-04-0B41B reburses the aeolian trunk from the colluvial margin and peaks at a round-trip time of 70.7 ms with a frame loss of 3.25 percent across the night window.
+Relay RL-05-0D12D reburses the recharge trunk from the diluted margin and steps down to a round-trip time of 264.7 ms with a frame loss of 0.03 percent across the night window.
+Relay RL-06-043B4 reburses the colluvial trunk from the upconing margin and reports a round-trip time of 49.5 ms with a frame loss of 3.42 percent across the night window.
+Relay RL-07-09AD6 reburses the fractured trunk from the freshened margin and averages a round-trip time of 225.3 ms with a frame loss of 1.06 percent across the night window.
+Relay RL-08-04394 reburses the brackish trunk from the fluvial margin and steps down to a round-trip time of 88.6 ms with a frame loss of 2.47 percent across the night window.
+Relay RL-09-017A0 reburses the wedge trunk from the lacustrine margin and drifts to a round-trip time of 363.0 ms with a frame loss of 3.30 percent across the night window.
+Relay RL-10-0217D reburses the mineralized trunk from the lacustrine margin and peaks at a round-trip time of 26.2 ms with a frame loss of 1.30 percent across the night window.
+Relay RL-11-08F97 reburses the perched trunk from the freshened margin and logs a round-trip time of 181.8 ms with a frame loss of 2.75 percent across the night window.
+Relay RL-12-02628 reburses the fractured trunk from the glacial margin and steps down to a round-trip time of 378.5 ms with a frame loss of 2.69 percent across the night window.
+Relay RL-13-06E06 reburses the silty trunk from the mineralized margin and flags a round-trip time of 153.4 ms with a frame loss of 2.15 percent across the night window.
+
+## Aggregation services
+Aggregator AG-00-3BB logs a queue depth of 5.2 messages and a histogram bucket of 6.82 seconds before the dedupe pass settles the stream for the next rotation.
+Aggregator AG-01-0FF reports a queue depth of 86.4 messages and a histogram bucket of 4.10 seconds before the dedupe pass settles the stream for the next rotation.
+Aggregator AG-02-37F drifts to a queue depth of 34.7 messages and a histogram bucket of 10.25 seconds before the dedupe pass settles the stream for the next rotation.
+Aggregator AG-03-3B3 averages a queue depth of 93.5 messages and a histogram bucket of 13.59 seconds before the dedupe pass settles the stream for the next rotation.
+Aggregator AG-04-33E flags a queue depth of 83.2 messages and a histogram bucket of 17.58 seconds before the dedupe pass settles the stream for the next rotation.
+Aggregator AG-05-2F2 settles near a queue depth of 84.8 messages and a histogram bucket of 0.84 seconds before the dedupe pass settles the stream for the next rotation.
+Aggregator AG-06-2AD averages a queue depth of 75.4 messages and a histogram bucket of 7.65 seconds before the dedupe pass settles the stream for the next rotation.
+Aggregator AG-07-06E drifts to a queue depth of 26.5 messages and a histogram bucket of 14.26 seconds before the dedupe pass settles the stream for the next rotation.
+Aggregator AG-08-0E3 averages a queue depth of 83.9 messages and a histogram bucket of 15.51 seconds before the dedupe pass settles the stream for the next rotation.
+Aggregator AG-09-138 averages a queue depth of 78.6 messages and a histogram bucket of 8.50 seconds before the dedupe pass settles the stream for the next rotation.
+
+## Calibration procedures
+Procedure CP-00-047 steps the probe through a two-point rinse, then applies a trim of 0.2084 at the 66 milli-bar mark and records a residual of 0.02052 before the next field seal is crimped.
+Procedure CP-01-0A4 steps the probe through a two-point rinse, then applies a trim of 0.7793 at the 96 milli-bar mark and records a residual of 0.01468 before the next field seal is crimped.
+Procedure CP-02-054 steps the probe through a two-point rinse, then applies a trim of 0.4634 at the 68 milli-bar mark and records a residual of 0.04234 before the next field seal is crimped.
+Procedure CP-03-02D steps the probe through a two-point rinse, then applies a trim of 0.8459 at the 12 milli-bar mark and records a residual of 0.04855 before the next field seal is crimped.
+Procedure CP-04-098 steps the probe through a two-point rinse, then applies a trim of 0.5267 at the 97 milli-bar mark and records a residual of 0.00287 before the next field seal is crimped.
+Procedure CP-05-09B steps the probe through a two-point rinse, then applies a trim of 0.3968 at the 72 milli-bar mark and records a residual of 0.01637 before the next field seal is crimped.
+Procedure CP-06-030 steps the probe through a two-point rinse, then applies a trim of 0.8860 at the 12 milli-bar mark and records a residual of 0.04508 before the next field seal is crimped.
+Procedure CP-07-0EC steps the probe through a two-point rinse, then applies a trim of 0.2367 at the 70 milli-bar mark and records a residual of 0.00108 before the next field seal is crimped.
+Procedure CP-08-0BC steps the probe through a two-point rinse, then applies a trim of 0.7278 at the 68 milli-bar mark and records a residual of 0.04707 before the next field seal is crimped.
+Procedure CP-09-0AA steps the probe through a two-point rinse, then applies a trim of 0.6069 at the 37 milli-bar mark and records a residual of 0.00443 before the next field seal is crimped.
+
+## Failure modes
+Mode FM-00-27 declares a fluvial excursion when two independent probes disagree beyond the band for three rotations; the fleet reports a mean time between failures of 36902 hours for this class of excursion.
+Mode FM-01-13 declares a drain excursion when two independent probes disagree beyond the band for three rotations; the fleet peaks at a mean time between failures of 29042 hours for this class of excursion.
+Mode FM-02-14 declares a till excursion when two independent probes disagree beyond the band for three rotations; the fleet peaks at a mean time between failures of 26976 hours for this class of excursion.
+Mode FM-03-0C declares a diapir excursion when two independent probes disagree beyond the band for three rotations; the fleet drifts to a mean time between failures of 22578 hours for this class of excursion.
+Mode FM-04-39 declares a perched excursion when two independent probes disagree beyond the band for three rotations; the fleet recovers to a mean time between failures of 27383 hours for this class of excursion.
+Mode FM-05-19 declares a siphon excursion when two independent probes disagree beyond the band for three rotations; the fleet logs a mean time between failures of 23152 hours for this class of excursion.
+
+## End of document
+Summarize the network's dominant risk in one paragraph, then list the three node identifiers whose readings most limit the model this month.
@@ -0,0 +1 @@
+{"model": "frank-large", "prompt": "# Subsurface Aquifer Monitoring Network - System Design Document (Revision 7)\n\nThis document describes the deployment of a subsurface aquifer monitoring network for the Cascadia sedimentary basin. It is written for the operations crew and the regional water authority. Every sensor node, relay, aggregation service, calibration procedure, and failure mode below carries a stable alphanumeric identifier that appears in every table and in the field notes.\n\n## Sensor nodes\nNode SN-000-08B9 sits in a discharge lens at 359.0 m below datum and reports a salinity of 2.38 ppt with the fractured interval reading 5.40 degC at dawn and the wellhead porous 1.72 m3/d while the electrode pair returns 3157 uS/cm.\nNode SN-001-0A65 sits in a percolate lens at 25.9 m below datum and steps down to a salinity of 2.13 ppt with the diapir interval reading 8.55 degC at dawn and the wellhead silty 0.67 m3/d while the electrode pair returns 979 uS/cm.\nNode SN-002-0571 sits in a diapir lens at 287.8 m below datum and peaks at a salinity of 7.63 ppt with the karst interval reading 4.26 degC at dawn and the wellhead spooned 11.50 m3/d while the electrode pair returns 2987 uS/cm.\nNode SN-003-0CE9 sits in a confining lens at 169.5 m below datum and logs a salinity of 0.96 ppt with the alluvial interval reading 5.03 degC at dawn and the wellhead salinized 12.08 m3/d while the electrode pair returns 2607 uS/cm.\nNode SN-004-0E2A sits in a freshened lens at 70.4 m below datum and settles near a salinity of 8.68 ppt with the spooned interval reading 4.88 degC at dawn and the wellhead siphon 4.38 m3/d while the electrode pair returns 2707 uS/cm.\nNode SN-005-0DDD sits in a percolate lens at 44.6 m below datum and reports a salinity of 6.25 ppt with the alluvial interval reading 10.85 degC at dawn and the wellhead brackish 13.99 m3/d while the electrode pair returns 3618 uS/cm.\nNode SN-006-0EFC sits in a till lens at 309.5 m below datum and reports a salinity of 3.49 ppt with the lacustrine interval reading 7.38 degC at dawn and the wellhead spooned 3.21 m3/d while the electrode pair returns 1253 uS/cm.\nNode SN-007-0B02 sits in a recharge lens at 262.0 m below datum and steps down to a salinity of 2.38 ppt with the discharge interval reading 8.18 degC at dawn and the wellhead perched 4.05 m3/d while the electrode pair returns 3900 uS/cm.\nNode SN-008-0448 sits in a karst lens at 406.5 m below datum and reports a salinity of 7.32 ppt with the upconing interval reading 6.17 degC at dawn and the wellhead fluvial 0.75 m3/d while the electrode pair returns 1448 uS/cm.\nNode SN-009-043C sits in a alluvial lens at 159.3 m below datum and flags a salinity of 6.20 ppt with the percolate interval reading 7.60 degC at dawn and the wellhead brackish 13.01 m3/d while the electrode pair returns 2025 uS/cm.\nNode SN-010-0FF5 sits in a karst lens at 264.2 m below datum and flags a salinity of 7.05 ppt with the freshened interval reading 7.88 degC at dawn and the wellhead siphon 8.41 m3/d while the electrode pair returns 1635 uS/cm.\nNode SN-011-098A sits in a drain lens at 365.7 m below datum and reports a salinity of 8.11 ppt with the wedge interval reading 5.51 degC at dawn and the wellhead glacial 2.52 m3/d while the electrode pair returns 2916 uS/cm.\nNode SN-012-0AE8 sits in a till lens at 259.6 m below datum and steps down to a salinity of 9.13 ppt with the recharge interval reading 11.60 degC at dawn and the wellhead spooned 0.46 m3/d while the electrode pair returns 3077 uS/cm.\nNode SN-013-0B83 sits in a salinized lens at 171.2 m below datum and peaks at a salinity of 2.83 ppt with the lacustrine interval reading 5.56 degC at dawn and the wellhead discharge 0.35 m3/d while the electrode pair returns 3083 uS/cm.\nNode SN-014-032E sits in a drain lens at 249.6 m below datum and flags a salinity of 1.09 ppt with the salinized interval reading 9.58 degC at dawn and the wellhead perched 12.00 m3/d while the electrode pair returns 2221 uS/cm.\nNode SN-015-004F sits in a aeolian lens at 264.4 m below datum and reports a salinity of 7.34 ppt with the salinized interval reading 8.76 degC at dawn and the wellhead perched 0.31 m3/d while the electrode pair returns 1483 uS/cm.\nNode SN-016-07C6 sits in a aeolian lens at 155.9 m below datum and reports a salinity of 0.64 ppt with the mineralized interval reading 11.75 degC at dawn and the wellhead diluted 13.46 m3/d while the electrode pair returns 524 uS/cm.\nNode SN-017-06C5 sits in a salinized lens at 72.1 m below datum and averages a salinity of 4.52 ppt with the siphon interval reading 8.93 degC at dawn and the wellhead confining 3.98 m3/d while the electrode pair returns 3687 uS/cm.\nNode SN-018-0E64 sits in a siphon lens at 334.9 m below datum and steps down to a salinity of 6.73 ppt with the salinized interval reading 7.63 degC at dawn and the wellhead freshened 9.64 m3/d while the electrode pair returns 1681 uS/cm.\nNode SN-019-0122 sits in a spooned lens at 1Line truncated
@@ -0,0 +1,43 @@
+#!/usr/bin/env bash
+# apply-protected-cast.sh — RUNS ON DANKSERVER. Rewrite the recipe's
+# PROTECTED-BEGIN/END section from Swift's OWN calibration grams: run
+# derive-protected-experts.py over the calib dir, splice its cast lines into
+# the marked section of the recipe in place, keep the pre-edit recipe in the
+# campaign trash. Refuses to run twice (marker text says derived).
+set -euo pipefail
+CALIB=${1:-/home/dank/corpora/calib}
+RECIPE=${2:-/home/dank/serverAppConfigs-swift15/scripts/recipes/qwen4exp-swift15-w4dn8-i8-hc8m.recipe}
+PY=/home/dank/venvs/radcorpus/bin/python
+DERIVE=/home/dank/serverAppConfigs-swift15/scripts/derive-protected-experts.py
+TS=$(date +%Y%m%d-%H%M%S)
+
+grep -q "derive-protected-experts.py rewrote" "$RECIPE" && {
+    echo "recipe already carries derived cast lines; refusing to re-derive" >&2; exit 1; }
+grep -q "PROTECTED-BEGIN" "$RECIPE" || { echo "no PROTECTED markers in $RECIPE" >&2; exit 1; }
+
+"$PY" "$DERIVE" "$CALIB" > "/tmp/protected-cast-$TS.txt"
+N=$(grep -c 'cast dtype=bf16' "/tmp/protected-cast-$TS.txt")
+[ "$N" -ge 2 ] || { echo "derive produced $N cast lines; refusing" >&2; exit 1; }
+echo "derived $N protected experts -> /tmp/protected-cast-$TS.txt"
+
+cp -f "$RECIPE" "/home/dank/.AGENTTRASH/$TS-qwen4exp-swift15-w4dn8-i8-hc8m.recipe"
+"$PY" - "$RECIPE" "/tmp/protected-cast-$TS.txt" <<'PYEOF'
+import sys
+recipe, castfile = sys.argv[1], sys.argv[2]
+lines = open(recipe).read().splitlines()
+cast = [l for l in open(castfile).read().splitlines() if 'cast dtype=bf16' in l]
+out, insec = [], False
+for l in lines:
+    if 'PROTECTED-BEGIN' in l:
+        insec = True
+        out.append(l)
+        out.extend(cast)
+    elif 'PROTECTED-END' in l:
+        insec = False
+        out.append(l)
+        out.append('# (derive-protected-experts.py rewrote this section from Swift\'s own grams)')
+    elif not insec:
+        out.append(l)
+open(recipe, 'w').write('\n'.join(out) + '\n')
+print(f'spliced {len(cast)} cast lines into {recipe}')
+PYEOF
@@ -0,0 +1,141 @@
+#!/usr/bin/env python3
+"""autoresearch-bench.py — deterministic greedy battery for frank-large.
+
+Runs ON Dankclaw (no LAN jitter) against the candidate container on :7770.
+Fixed embedded corpus, temperature 0, streaming so decode speed is measured
+between real token arrivals (not request totals). Median of REPS samples per
+metric is the reported value; medians are far more stable than means under
+GPU thermal drift.
+
+Metrics emitted (stdout, one per line):
+  METRIC decode_tokens_per_sec=<float>   primary, higher better
+  METRIC prefill_tokens_per_sec=<float>  secondary
+  METRIC ttft_ms=<float>                 secondary, lower better
+Exit non-zero if the endpoint is unhealthy or the served alias is wrong.
+"""
+import argparse
+import json
+import statistics
+import sys
+import time
+import urllib.request
+
+# Fixed corpus: deterministic, ~28k tokens after tokenization (same scale as
+# the 2026-10-01 bench window). Built by repeating a fixed paragraph block a
+# fixed number of times - NO random, NO clock, NO network.
+_BLOCK = (
+    "The rendering pipeline walks the scene graph once per frame, resolving "
+    "material bindings into descriptor sets before any draw call is recorded. "
+    "Because the allocator recycles command buffers only after their fence "
+    "signals, a submission that outlives its frame must pin the pool entry; "
+    "otherwise the next frame overwrites recorded commands in flight. The "
+    "validation layer flags exactly this class of lifetime bug, but only for "
+    "buffers it can see - GPU-side timestamps and queue-family ownership "
+    "transfers hide in the driver's own bookkeeping, which is why release "
+    "builds must lean on explicit barriers and why a frame-time regression "
+    "often shows up as corruption one frame later than its cause.\n"
+)
+LONG_CTX = _BLOCK * 220  # ~28k-token prefill, matching the deployed workload
+_LONG = LONG_CTX + "\nNow, in one focused paragraph (80-150 words), explain " \
+    "which lifetime bugs this pipeline can still suffer and what explicit " \
+    "mitigations the text implies."
+_SHORT = "Design a rate limiter for a homelab reverse proxy that must not " \
+    "lock out the LAN. Think it through, then give the algorithm."
+
+REPS = 3
+MAX_TOKENS = 256
+REQ_TIMEOUT = 900
+
+
+def stream_completion(url: str, model: str, prompt: str):
+    """Greedy streaming completion.
+
+    Returns (ttft_s, decode_tps, prefill_tps, prompt_tokens, n_tokens).
+    ttft is RELATIVE to request send: time.perf_counter()'s epoch is host
+    boot, and returning the raw stamp read as an 11-day first token.
+    """
+    body = json.dumps({
+        "model": model,
+        "messages": [{"role": "user", "content": prompt}],
+        "max_tokens": MAX_TOKENS,
+        "temperature": 0,
+        "stream": True,
+        "stream_options": {"include_usage": True},
+    }).encode()
+    req = urllib.request.Request(
+        url.rstrip("/") + "/v1/chat/completions", data=body,
+        headers={"Content-Type": "application/json"})
+    t0 = time.perf_counter()
+    first = None
+    stamps = []
+    usage = {}
+    with urllib.request.urlopen(req, timeout=REQ_TIMEOUT) as resp:
+        for raw in resp:
+            line = raw.decode(errors="replace").strip()
+            if not line.startswith("data:"):
+                continue
+            payload = line[5:].strip()
+            if payload == "[DONE]":
+                break
+            chunk = json.loads(payload)
+            if chunk.get("usage"):
+                usage = chunk["usage"]
+            for ch in chunk.get("choices", []):
+                d = ch.get("delta") or {}
+                # reasoning AND content deltas both arrive token by token;
+                # count either so greedy text length drives the rate.
+                # (llama.cpp names the thinking field reasoning_content.)
+                if d.get("content") or d.get("reasoning") or d.get("reasoning_content"):
+                    now = time.perf_counter()
+                    if first is None:
+                        first = now
+                    stamps.append(now)
+    if first is None or len(stamps) < 2:
+        # A zero-argmax degenerate answer would poison the median; raise.
+        raise RuntimeError(f"stream produced {len(stamps)} tokens")
+    ttft_s = first - t0
+    decode_s = stamps[-1] - stamps[0]
+    decode_tps = (len(stamps) - 1) / decode_s if decode_s > 0 else 0.0
+    prompt_tokens = usage.get("prompt_tokens") or 0
+    prefill_tps = prompt_tokens / ttft_s if prompt_tokens else 0.0
+    return ttft_s, decode_tps, prefill_tps, prompt_tokens, len(stamps)
+
+
+def main():
+    ap = argparse.ArgumentParser()
+    ap.add_argument("--url", default="http://127.0.0.1:7770")
+    ap.add_argument("--model", default="frank-dragoon")
+    args = ap.parse_args()
+
+    with urllib.request.urlopen(args.url.rstrip("/") + "/v1/models",
+                                timeout=10) as r:
+        ids = [m["id"] for m in json.load(r)["data"]]
+    if args.model not in ids:
+        print(f"FATAL: served ids {ids}, expected {args.model}",
+              file=sys.stderr)
+        sys.exit(2)
+
+    # Warmup (compile/graph warm + allocator settle), NOT measured.
+    stream_completion(args.url, args.model, _SHORT)
+
+    decodes, prefills, ttfts = [], [], []
+    for rep in range(REPS):
+        for name, prompt in (("long", _LONG), ("short", _SHORT)):
+            ttft, dt, pt, ptk, ntok = stream_completion(
+                args.url, args.model, prompt)
+            decodes.append(dt)
+            ttfts.append(ttft * 1000.0)
+            if name == "long":
+                prefills.append(pt)
+            print(f"  rep{rep} {name:5s} ttft={ttft*1000:8.1f}ms "
+                  f"prefill={pt:8.1f} t/s decode={dt:6.2f} t/s "
+                  f"({ntok} tok, prompt {ptk})", file=sys.stderr)
+
+    med = lambda xs: statistics.median(xs)
+    print(f"METRIC decode_tokens_per_sec={med(decodes):.2f}")
+    print(f"METRIC prefill_tokens_per_sec={med(prefills):.1f}")
+    print(f"METRIC ttft_ms={med(ttfts):.1f}")
+
+
+if __name__ == "__main__":
+    main()
@@ -0,0 +1,187 @@
+#!/usr/bin/env bash
+# autoresearch-frank-primary.sh — RUNS ON DANKSERVER, driven over ssh by the
+# dev-box autoresearch.sh. It is the canonical frank-primary benchmark entry
+# point for the autoresearch loop.
+#
+# Method (mandated by the Benchmarking Policy): the production service serves
+# real users and shares the two R9700s, so a bench beside it starves the GPUs
+# and corrupts results. This script therefore:
+#   1. stops the production frank-primary unit (root, relink symlink: stop,
+#      NOT mask — mask refuses to replace a symlinked unit file),
+#   2. boots a scratch radiance container named `arbench` on the SAME two
+#      cards with the candidate engine argv (the env file the loop edits),
+#   3. polls /health until the listener is live (it binds AFTER the full
+#      load, so a bare curl is an honest gate),
+#   4. runs ONE self-contained completion request locally (127.0.0.1, no LAN
+#      jitter): a fixed ~140k-char real-text prompt (~32k tokens) decoded for
+#      256 tokens, greedy. The engine's `timings` object reports the honest
+#      prefill and decode rates for that single request.
+#   5. prints METRIC lines and tears the scratch down,
+#   6. restores production (systemctl start), via a trap so a failed bench
+#      never leaves frank-primary down (the stale-probe-tail lesson).
+#
+# Determinism: the prompt is built from a fixed, sorted corpus of repo files
+# with a fixed random seed, so prompt_n is stable across runs. Sampling is
+# greedy (temperature 0), so the decode length is fixed; the only residual
+# nondeterminism is the wht6 lossy TP all-reduce, which is a constant across
+# runs and cancels in A/B comparisons.
+#
+# The scratch mirrors the production container contract exactly (serve-frank-
+# primary.sh): --network host, --init, seccomp=unconfined (io_uring for the
+# n-gram table), memlock unbounded (the ~132 GiB pinned ledger), /dev/kfd +
+# /dev/dri, the .rad ro, the state dir rw, the tune cache mount. The ONLY
+# differences: the container name (arbench, so prod's identity is untouched)
+# and a FRESH prefix-cache disk tier (arbench/, wiped per run) so the cold
+# prefill is cold — the prod tier at kvcache/frank-primary is left intact.
+set -uo pipefail
+
+ENV_FILE="${1:-/home/dank/serverAppConfigs/scripts/frank-primary.env}"
+SCRATCH=arbench
+PORT=7770
+HEALTH="http://127.0.0.1:$PORT/health"
+COMPLETE="http://127.0.0.1:$PORT/v1/completions"
+BODY=/tmp/arbench-body.json
+LOG=/home/dank/logs/arbench.log
+
+restore() {
+  podman rm -f "$SCRATCH" >/dev/null 2>&1 || true
+  sudo systemctl start frank-primary >/dev/null 2>&1 || true
+}
+trap restore EXIT
+
+# --- source the candidate config (same file the production unit reads) ------
+if [ ! -f "$ENV_FILE" ]; then
+  echo "no env file at $ENV_FILE" >&2; exit 1
+fi
+# shellcheck disable=SC1090
+set -a; . "$ENV_FILE"; set +a
+
+# --- stop production; the scratch needs both cards to itself ----------------
+sudo systemctl stop frank-primary >/dev/null 2>&1 || true
+podman rm -f "$SCRATCH" >/dev/null 2>&1 || true
+
+# --- fresh prefix-cache tier for a cold prefill (prod's tier untouched) ------
+FRESH_KV="$FRANK_PRIMARY_STATE_DIR/kvcache/$SCRATCH"
+rm -rf "$FRESH_KV"; mkdir -p "$FRESH_KV"
+mkdir -p "$FRANK_PRIMARY_STATE_DIR/tune"
+
+# --- boot the scratch: the production run line, scratch name + fresh KV ------
+podman run -d --name "$SCRATCH" \
+  --network host \
+  --init \
+  --security-opt seccomp=unconfined \
+  --ulimit memlock=-1:-1 \
+  --device /dev/kfd --device /dev/dri \
+  -v "$FRANK_PRIMARY_MODEL_DIR":/models:ro \
+  -v "$FRANK_PRIMARY_STATE_DIR":/data \
+  -v "$FRANK_PRIMARY_STATE_DIR/tune":/opt/radiance/share/radiance/tune \
+  -v "$FRANK_PRIMARY_TEMPLATE":/templates/template.jinja:ro \
+  "$FRANK_PRIMARY_IMAGE" \
+  --model "/models/$FRANK_PRIMARY_MODEL_FILE" \
+  --host :: --port "$PORT" \
+  --served-model-name "$FRANK_PRIMARY_SERVED_NAME" \
+  --override-chat-template /templates/template.jinja \
+  --tp "$FRANK_PRIMARY_TP" --tp-wire "$FRANK_PRIMARY_TP_WIRE" \
+  --max-num-seqs "$FRANK_PRIMARY_MAX_SEQS" \
+  --max-model-len "$FRANK_PRIMARY_MAX_MODEL_LEN" \
+  --placement "$FRANK_PRIMARY_PLACEMENT" \
+  --host-pool-mib "$FRANK_PRIMARY_HOST_POOL_MIB" \
+  --gpu-headroom-mib "$FRANK_PRIMARY_GPU_HEADROOM_MIB" \
+  --vram-weights-mib "$FRANK_PRIMARY_VRAM_WEIGHTS_MIB" \
+  --vram-kv-mib "$FRANK_PRIMARY_VRAM_KV_MIB" \
+  --kv-cache-dtype "$FRANK_PRIMARY_KV_DTYPE" \
+  --prefix-cache-host-mib "$FRANK_PRIMARY_PREFIX_HOST_MIB" \
+  --prefix-cache-dir "/data/kvcache/$SCRATCH" \
+  --prefix-cache-disk-mib "$FRANK_PRIMARY_PREFIX_DISK_MIB" \
+  --ngram-placement "$FRANK_PRIMARY_NGRAM_PLACEMENT" \
+  --num-speculative-tokens "$FRANK_PRIMARY_SPEC_TOKENS" \
+  --max-num-batched-tokens "$FRANK_PRIMARY_MAX_BATCHED_TOKENS" \
+  ${FRANK_PRIMARY_EXTRA_ARGS:+$FRANK_PRIMARY_EXTRA_ARGS} \
+  > "$LOG" 2>&1
+if [ $? -ne 0 ]; then echo "scratch boot failed; see $LOG" >&2; exit 1; fi
+
+# --- poll health (listener binds after the full load; cold boot ~110 s) ------
+ok=0
+for _ in $(seq 1 360); do
+  code=$(curl -s -o /dev/null -w "%{http_code}" --max-time 3 "$HEALTH" 2>/dev/null || echo 000)
+  if [ "$code" = "200" ]; then ok=1; break; fi
+  sleep 2
+done
+if [ "$ok" -ne 1 ]; then
+  echo "scratch never reached health 200; tail of $LOG:" >&2
+  tail -30 "$LOG" >&2; exit 1
+fi
+
+# --- build the workload: a fixed ~140k-char real-text prompt (repo corpus) ---
+python3 - "$BODY" <<'PY'
+import json, sys, random, glob
+random.seed(1234)
+root = "/home/dank/serverAppConfigs"
+files = sorted(
+    glob.glob(f"{root}/serverAppConfigs-docs/**/*.md", recursive=True)
+    + glob.glob(f"{root}/scripts/*.sh")
+    + glob.glob(f"{root}/scripts/*.py")
+    + glob.glob(f"{root}/AGENTS.md")
+)
+corpus = []
+total = 0
+for f in files:
+    try:
+        t = open(f, encoding="utf-8", errors="ignore").read()
+    except Exception:
+        continue
+    if not t.strip():
+        continue
+    corpus.append(t)
+    total += len(t)
+    if total > 240000:
+        break
+corpus = "\n".join(corpus)
+TGT = 140000
+parts, n = [], 0
+while n < TGT:
+    start = random.randint(0, max(0, len(corpus) - 500))
+    seg = corpus[start:start + random.randint(200, 500)]
+    parts.append(seg)
+    n += len(seg) + 1
+prompt = " ".join(parts)
+json.dump(
+    {"model": "frank-archon", "prompt": prompt,
+     "max_tokens": 256, "temperature": 0.0, "stream": False,
+     "stop": ["<|im_end|>"]},
+    open(sys.argv[1], "w"),
+)
+PY
+if [ ! -f "$BODY" ]; then echo "prompt build failed" >&2; exit 1; fi
+
+# --- the bench request (prefill ~32k + decode 256, greedy) -------------------
+RESP=$(curl -s --max-time 600 -X POST "$COMPLETE" \
+  -H 'Content-Type: application/json' -d "@$BODY")
+echo "$RESP" > /tmp/arbench-resp.json
+
+# --- parse the timings into METRIC lines -------------------------------------
+python3 - <<'PY'
+import json, sys
+try:
+    r = json.load(open("/tmp/arbench-resp.json"))
+except Exception as e:
+    sys.stderr.write(f"unparseable response: {e}\n"); sys.exit(1)
+if "timings" not in r:
+    sys.stderr.write(f"no timings in response: {json.dumps(r)[:500]}\n"); sys.exit(1)
+t = r["timings"]
+dtps = t.get("predicted_per_second")
+ptps = t.get("prompt_per_second")
+pms  = t.get("prompt_ms")
+dn   = t.get("draft_n", 0)
+da   = t.get("draft_n_accepted", 0)
+if dtps is None:
+    sys.stderr.write("predicted_per_second null (no decode produced)\n"); sys.exit(1)
+print(f"METRIC decode_tps={dtps:.4f}")
+if ptps is not None:
+    print(f"METRIC prefill_tps={ptps:.4f}")
+if pms is not None:
+    print(f"METRIC ttft_ms={pms:.4f}")
+if dn:
+    print(f"METRIC draft_accept={da/dn:.4f}")
+print(f"INFO prompt_n={t.get('prompt_n')} cache_n={t.get('cache_n')} predicted_n={t.get('predicted_n')}")
+PY
@@ -0,0 +1,176 @@
+#!/usr/bin/env bash
+# frank-primary-arms.sh — run a SEQUENCE of frank-primary bench arms and keep
+# every arm's evidence. Runs on the operator box; each arm is one full window
+# of ./autoresearch.sh (stop+mask -> boot the arm as the service -> measure ->
+# restore production and verify), so the fleet is only ever down for one arm at
+# a time and the production unit is back up between arms.
+#
+# An arm is the repo's own frank-primary.env plus an OVERLAY of engine knobs
+# appended after it (later assignment wins when the serve script sources it),
+# staged through FRANK_PRIMARY_AR_ENV_FILE so the checkout is never mutated and
+# each arm gets its own fingerprint (the memo can never answer one arm with
+# another arm's numbers).
+#
+# Usage:  scripts/autoresearch/frank-primary-arms.sh [arm ...]
+#         scripts/autoresearch/frank-primary-arms.sh --list
+# Output, under /tmp/frank-primary-arms-<UTC stamp>/ (override with ARMS_DIR):
+#
+#   <arm>.out        full autoresearch output (metrics + engine evidence)
+#   <arm>.env        the exact staged env
+#   <arm>.rc         window exit code
+#   summary.tsv      arm, rc, decode_tps, decode_tps_t0.7, kv_tokens, boot_s
+#
+# Exit 0 when every arm produced metrics; 1 if any arm failed (a failed boot is
+# a RESULT — the window still restored production, and the rc says which arm).
+set -uo pipefail
+
+SCRIPT_DIR="$(cd -- "$(dirname -- "${BASH_SOURCE[0]}")" && pwd)"
+REPO="$(cd -- "$SCRIPT_DIR/../.." && pwd)"
+BASE_ENV="$REPO/scripts/frank-primary.env"
+ARMS_DIR="${ARMS_DIR:-/tmp/frank-primary-arms-$(date -u +%Y%m%dT%H%M%SZ)}"
+
+# --- arms ------------------------------------------------------------------
+# baseline: the shipped shape E, measured with THIS harness so the comparison is
+# matched (the ~29.6 t/s figure in the docs is an older shape, and no shape-E
+# decode number exists yet).
+OVER_baseline=()
+# B: MTP3 with the vendor's probabilistic draft sampling, a capture ladder whose
+# step is num_speculative_tokens + 1 = 4, and the sampling override. Precise
+# memory stays ON — this arm is simultaneously the MTP test and the test of
+# whether MTP's drafter workspace and the fitted 5.25 GiB overhead can coexist.
+SPEC3='{"method":"mtp","num_speculative_tokens":3,"draft_sample_method":"probabilistic"}'
+LADDER3='{"cudagraph_capture_sizes":[4,8,12],"max_cudagraph_capture_size":12,"inductor_compile_config":{"combo_kernels":false,"benchmark_combo_kernel":false}}'
+# The value must arrive in the staged env file SINGLE-QUOTED: sourcing an env
+# file performs quote removal, so an unquoted {"temperature":0.7} would reach
+# the engine as {temperature:0.7}.
+GEN07='{"temperature":0.7}'
+OVER_B=(
+    "FRANK_PRIMARY_SPEC='$SPEC3'"
+    "FRANK_PRIMARY_COMPCONFIG='$LADDER3'"
+    'FRANK_PRIMARY_GEN_CONFIG='"'$GEN07'"
+)
+# B2: identical to B with the default (7.60 GiB) overhead, i.e. precise memory
+# OFF. It is the fallback if B dies at graph capture, and it prices the arena
+# that precise memory buys (7,522 slots -> ~6,496).
+OVER_B2=(
+    "FRANK_PRIMARY_SPEC='$SPEC3'"
+    "FRANK_PRIMARY_COMPCONFIG='$LADDER3'"
+    'FRANK_PRIMARY_GEN_CONFIG='"'$GEN07'"
+    'FRANK_PRIMARY_PRECISE_MEMORY='
+)
+# C: the vendor's concurrency shape — SEQS 16 with the [4..32] ladder and util
+# 0.95. This is also the direct test of the theory that pinning the capture
+# ladder unlocks SEQS > 3 (the SEQS=2 failure was query_start_loc specialised to
+# the constant 3, which is NOT in the default ladder; 4 is).
+LADDER16='{"cudagraph_capture_sizes":[4,8,12,16,20,24,28,32],"max_cudagraph_capture_size":32,"inductor_compile_config":{"combo_kernels":false,"benchmark_combo_kernel":false}}'
+OVER_C=(
+    "FRANK_PRIMARY_SPEC='$SPEC3'"
+    "FRANK_PRIMARY_COMPCONFIG='$LADDER16'"
+    'FRANK_PRIMARY_GEN_CONFIG='"'$GEN07'"
+    'FRANK_PRIMARY_SEQS=16'
+    'FRANK_PRIMARY_MNBT=4096'
+    'FRANK_PRIMARY_UTIL=0.95'
+)
+
+# D/E: the CONTEXT arms. Measured 2026-10-02 (arms run 20261002T233855Z): the
+# KV pool is a hard byte request divided by a PER-TOKEN PRICE, and MTP raises
+# that price because the drafter keeps its own KV per position:
+#   no spec: 3.15e9 / 398,658 tokens =  7,901 B/token  (1.99x of 200,704)
+#   MTP3:    3.15e9 / 313,002 tokens = 10,064 B/token  (1.56x)
+# So arm C lost a fifth of the context to the drafter, not to util. Frank's
+# contract is >=401,400 tokens over >=2 seqs = 2.00x of 200,704, which at
+# 10,064 B/token costs 4.04e9 -> ask 4.05e9 (+0.83 GiB/rank).
+# D is arm B with EXACTLY ONE variable changed (the KV bytes), so its result
+# says whether the 4k-depth collapse (B 23.09 vs baseline 33.99) was KV or the
+# batch shape.
+KV401K='FRANK_PRIMARY_KV_BYTES=4050000000'
+OVER_D=(
+    "FRANK_PRIMARY_SPEC='$SPEC3'"
+    "FRANK_PRIMARY_COMPCONFIG='$LADDER3'"
+    'FRANK_PRIMARY_GEN_CONFIG='"'$GEN07'"
+    "$KV401K"
+)
+# E: D with C's max_num_batched_tokens (4096 — C is the only arm that did NOT
+# collapse at 4k depth: 33.15 where B gave 23.09) and util 0.99, the value
+# Frank asked about. 0.99 is affordable on paper — production at util 0.98
+# measures 30.37/34.21 GiB per card, 3.58 GiB physically unused — but it leaves
+# only 0.32 GiB above the allowance while the CUDA-graph cost is an ANALYTIC
+# estimate (C's was 1.47 GiB over 24 graphs), so it is measured, not assumed.
+OVER_E=(
+    "FRANK_PRIMARY_SPEC='$SPEC3'"
+    "FRANK_PRIMARY_COMPCONFIG='$LADDER3'"
+    'FRANK_PRIMARY_GEN_CONFIG='"'$GEN07'"
+    "$KV401K"
+    'FRANK_PRIMARY_MNBT=4096'
+    'FRANK_PRIMARY_UTIL=0.99'
+)
+
+arm_overlay() {
+    case "$1" in
+        baseline) ;;                       # the shipped env, no overlay
+        B)  printf '%s\n' "${OVER_B[@]}" ;;
+        B2) printf '%s\n' "${OVER_B2[@]}" ;;
+        C)  printf '%s\n' "${OVER_C[@]}" ;;
+        D)  printf '%s\n' "${OVER_D[@]}" ;;
+        E)  printf '%s\n' "${OVER_E[@]}" ;;
+        *) return 1 ;;
+    esac
+}
+if [ "${1:-}" = "--list" ]; then
+    printf 'baseline B B2 C D E\n'
+    exit 0
+fi
+
+[ -f "$BASE_ENV" ] || { echo "[arms] missing $BASE_ENV" >&2; exit 1; }
+if [ "$#" -gt 0 ]; then ARMS=("$@"); else ARMS=(baseline B B2 C); fi
+mkdir -p "$ARMS_DIR"
+echo "[arms] dir $ARMS_DIR  arms: ${ARMS[*]}" >&2
+
+: >"$ARMS_DIR/summary.tsv"
+printf 'arm\trc\tdecode_tps\tdecode_tps_t0.7\tkv_tokens\tboot_s\n' >>"$ARMS_DIR/summary.tsv"
+overall=0
+
+for arm in "${ARMS[@]}"; do
+    if ! arm_overlay "$arm" >"$ARMS_DIR/$arm.overlay"; then
+        echo "[arms] unknown arm: $arm" >&2
+        overall=1
+        continue
+    fi
+    { cat "$BASE_ENV"; echo; echo "# --- arm $arm overlay ($(date -u +%FT%TZ)) ---"
+      arm_overlay "$arm"; } >"$ARMS_DIR/$arm.env"
+    # Gate the staged env BEFORE burning a ~20-minute window: every knob that
+    # holds JSON must still be one valid JSON object after the file is SOURCED
+    # (that is the step that silently eats unquoted braces).
+    if ! bash -c 'set -a; . "$1" >/dev/null; set +a
+        for k in FRANK_PRIMARY_SPEC FRANK_PRIMARY_COMPCONFIG FRANK_PRIMARY_GEN_CONFIG; do
+            v="${!k:-}"
+            [ -n "$v" ] || continue
+            case "$v" in
+                "{"*) python3 -c "import json,sys; json.loads(sys.argv[1])" "$v" \
+                        || { echo "[arms] $k is not valid JSON: $v" >&2; exit 1; } ;;
+                *) echo "[arms] $k is not a JSON object: $v" >&2; exit 1 ;;
+            esac
+        done' _ "$ARMS_DIR/$arm.env"; then
+        echo "[arms] $arm: staged env rejected, window not opened" >&2
+        echo 99 >"$ARMS_DIR/$arm.rc"
+        overall=1
+        continue
+    fi
+
+    echo "[arms] === $arm: opening window $(date -u +%H:%M:%S)Z ===" >&2
+    FRANK_PRIMARY_AR_ENV_FILE="$ARMS_DIR/$arm.env" \
+        "$REPO/autoresearch.sh" >"$ARMS_DIR/$arm.out" 2>&1
+    rc=$?
+    echo "$rc" >"$ARMS_DIR/$arm.rc"
+    echo "[arms] === $arm: rc=$rc $(date -u +%H:%M:%S)Z ===" >&2
+    grep -E '^METRIC ' "$ARMS_DIR/$arm.out" | sed 's/^METRIC //' >"$ARMS_DIR/$arm.metrics" || true
+    metric() { local v; v="$(sed -n "s/^$1=//p" "$ARMS_DIR/$arm.metrics" | tail -1)"; printf '%s' "${v:-$2}"; }
+    printf '%s\t%s\t%s\t%s\t%s\t%s\n' "$arm" "$rc" \
+        "$(metric decode_tps FAIL)" "$(metric decode_tps_t0.7 FAIL)" \
+        "$(metric kv_tokens ?)" "$(metric boot_seconds ?)" >>"$ARMS_DIR/summary.tsv"
+    [ "$rc" -ne 0 ] && overall=1
+done
+
+echo "[arms] summary:" >&2
+cat "$ARMS_DIR/summary.tsv" >&2
+exit "$overall"
@@ -0,0 +1,216 @@
+#!/usr/bin/env python3
+"""frank-primary decode measurement — runs INSIDE the engine container.
+
+The method is the fleet's established one (the 2026-09-29 / 2026-10-01
+tuning campaigns, /home/dank/mtp_measure.py): SSE chunk counting is invalid
+under speculative decoding because rejected drafts change the accounting, so
+decode is measured from the server's own usage counters:
+
+    decode_tok_s = completion_tokens / (wall - ttft)
+
+with ttft = wall time to the FIRST content-bearing chunk (thinking arrives in
+`reasoning`/`delta.reasoning` on this stack and is excluded from the decode
+window — and is deterministic per prompt at temperature 0, so the exclusion
+cancels in any A/B on the same workload).
+
+Workload (fixed, deterministic, offline):
+  * prompts built from a large fixed-seed shuffled word list so tokens rarely
+    repeat (no n-gram/prefix predictability of the text itself), sized with
+    the server's own /v1/messages/count_tokens counter
+  * depths 115 / 1056 / 4096 target prompt tokens
+  * one 64-token warm pass per depth (warms the expert arena), then
+    max_tokens=256, 3 runs, median — repeated for every temperature in AR_TEMPS
+    (default 0 and 0.7). The prompt workload is deterministic (fixed seed, no
+    network, no time-of-day dependence); at temperature > 0 the generated text
+    is not, but the measured window is bounded by max_tokens, so the rate stays
+    comparable and the non-determinism is exactly the sampling that a
+    speculative-decode arm must survive.
+
+Env:
+  AR_BASE             endpoint (default http://127.0.0.1:7770)
+  AR_MODEL            served name (default frank-primary)
+  AR_PROMPT_CEILING   hard prompt ceiling in tokens (default 100000)
+  AR_TEMPS            comma-separated sampling temperatures (default "0,0.7")
+
+Output: progress on stderr, ONE json object on stdout (the last line).
+Exit codes: 0 ok, 2 endpoint unhealthy / no usable completion, 3 bad usage.
+"""
+import json
+import os
+import random
+import statistics
+import sys
+import time
+import urllib.error
+import urllib.request
+
+BASE = os.environ.get("AR_BASE", "http://127.0.0.1:7770")
+MODEL = os.environ.get("AR_MODEL", "frank-archon")
+PROMPT_CEILING = int(os.environ.get("AR_PROMPT_CEILING", "100000"))
+DEPTHS = (115, 1056, 4096)
+WARM_TOKENS = 64
+MAX_TOKENS = 256
+RUNS = 3
+# Temperature sweep. The unsuffixed keys are temperature 0 — byte-comparable to
+# every historical measurement — and AR_TEMPS adds more. The second one matters
+# for speculative decode: draft ACCEPTANCE is sampling-dependent (a greedy draft
+# matches a greedy target exactly and is rejected more under a non-greedy
+# target), so an MTP arm judged only at temp 0 would be judged on half the
+# contract. Production traffic is non-zero temperature (the payload's
+# generation_config says 1.0; an override to 0.7 is under test).
+TEMPS = tuple(float(x) for x in os.environ.get("AR_TEMPS", "0,0.7").split(","))
+
+# 43-word x 12 = 516-word vocabulary, seed 1234 — the campaign's exact list
+# and seed, so prompts are byte-identical to the historical measurements.
+_VOCAB = ("alpha bravo charlie delta echo foxtrot golf hotel india juliet kilo lima "
+          "mike november oscar papa quebec romeo sierra tango uniform victor whiskey "
+          "xray yankee zulu cardinal ordinal topology kernel scheduler resident "
+          "unified aperture expert arena slab offload prefetch gemm triton quant "
+          "tensor shard checkpoint manifest embed router cache page table window "
+          "gradient sparse dense moe attention head dim block stream decode prefill "
+          "bandwidth latency throughput residency eviction tier swap pin memlock").split()
+_R = random.Random(1234)
+_ORDER = list(range(len(_VOCAB)))
+_R.shuffle(_ORDER)
+
+
+def log(msg):
+    print("[measure] %s" % msg, file=sys.stderr, flush=True)
+
+
+def post_json(path, body, timeout):
+    req = urllib.request.Request(
+        BASE + path, data=json.dumps(body).encode(),
+        headers={"Content-Type": "application/json"})
+    with urllib.request.urlopen(req, timeout=timeout) as r:
+        return json.load(r)
+
+
+def count_tokens(text):
+    """Token count without a prefill, from the server's own counter."""
+    body = {"model": MODEL,
+            "messages": [{"role": "user", "content": text}]}
+    r = post_json("/v1/messages/count_tokens", body, timeout=300)
+    n = r.get("input_tokens") or r.get("count") or r.get("token_count")
+    if not n:
+        raise RuntimeError("count_tokens returned no count: %r" % (r,))
+    return int(n)
+
+
+def words(n):
+    return " ".join(_VOCAB[_ORDER[i % len(_VOCAB)]] for i in range(n))
+
+
+def calibrate():
+    """Measure words->tokens from the server's own counter."""
+    return 2000.0 / count_tokens(words(2000))
+
+
+def prompt_for(target_tokens, wpt):
+    n = max(1, int(target_tokens * wpt))
+    text = words(n)
+    pt = count_tokens(text)
+    while pt > PROMPT_CEILING:
+        n = max(1, int(n * (PROMPT_CEILING / pt) * 0.98))
+        text = words(n)
+        pt = count_tokens(text)
+    return text, pt
+
+
+def measured(prompt, max_tokens, temperature=0):
+    """Return (ttft_s, decode_tok_s, completion_tokens) via usage counters."""
+    body = {"model": MODEL,
+            "messages": [{"role": "user", "content": prompt}],
+            "max_tokens": max_tokens, "temperature": temperature,
+            "stream": True, "stream_options": {"include_usage": True}}
+    req = urllib.request.Request(
+        BASE + "/v1/chat/completions", data=json.dumps(body).encode(),
+        headers={"Content-Type": "application/json"})
+    t0 = time.time()
+    ttft = None
+    usage = {}
+    with urllib.request.urlopen(req, timeout=1800) as r:
+        for raw in r:
+            line = raw.decode("utf-8", "replace").strip()
+            if not line.startswith("data:"):
+                continue
+            s = line[5:].strip()
+            if s == "[DONE]":
+                break
+            try:
+                ev = json.loads(s)
+            except json.JSONDecodeError:
+                continue
+            if ev.get("usage"):
+                usage = ev["usage"]
+            for ch in ev.get("choices", []):
+                if (ch.get("delta") or {}).get("content"):
+                    if ttft is None:
+                        ttft = time.time() - t0
+    wall = time.time() - t0
+    ct = int(usage.get("completion_tokens") or 0)
+    if ct <= 0:
+        # No usage on the stream: the endpoint is unhealthy or the model
+        # returned nothing (e.g. HTTP 200 with an error payload).
+        raise RuntimeError("no completion_tokens in usage: %r" % (usage,))
+    dec_window = max(wall - (ttft or 0.0), 1e-9)
+    return (ttft or wall), (ct / dec_window), ct
+
+
+def fmt_temp(t):
+    return ("%.1f" % t).rstrip("0").rstrip(".") if t else "0"
+
+
+def main():
+    log("base=%s model=%s ceiling=%d temps=%s"
+        % (BASE, MODEL, PROMPT_CEILING, ",".join(fmt_temp(t) for t in TEMPS)))
+    try:
+        wpt = calibrate()
+    except Exception as e:
+        log("endpoint unhealthy during calibration: %s" % e)
+        return 2
+    log("words/token: %.3f" % wpt)
+
+    result = {"words_per_token": round(wpt, 5)}
+    for tgt in DEPTHS:
+        try:
+            p, pt = prompt_for(tgt, wpt)
+            measured(p, WARM_TOKENS)  # warm the arena and the prefix cache
+        except Exception as e:
+            log("warm pass failed at depth %d: %s" % (tgt, e))
+            return 2
+        for temp in TEMPS:
+            suffix = "" if temp == 0 else "_t%s" % fmt_temp(temp)
+            rates, tts, cts = [], [], []
+            for i in range(RUNS):
+                try:
+                    tt, r, ct = measured(p, MAX_TOKENS, temp)
+                except Exception as e:
+                    log("run %d/%d failed at depth %d temp %s: %s"
+                        % (i + 1, RUNS, tgt, fmt_temp(temp), e))
+                    continue
+                rates.append(r)
+                tts.append(tt)
+                cts.append(ct)
+            if len(rates) < 2:
+                log("only %d usable run(s) at depth %d temp %s — refusing to median"
+                    % (len(rates), tgt, fmt_temp(temp)))
+                return 2
+            result["decode_tps_%d%s" % (tgt, suffix)] = round(statistics.median(rates), 2)
+            result["ttft_ms_%d%s" % (tgt, suffix)] = round(1000.0 * statistics.median(tts), 1)
+            result["completion_tokens_%d%s" % (tgt, suffix)] = int(statistics.median(cts))
+            result["prompt_tokens_%d%s" % (tgt, suffix)] = pt
+            log("depth %5d tok temp %-4s decode %7.2f t/s  ttft %.3f s  (runs=%d)"
+                % (pt, fmt_temp(temp), result["decode_tps_%d%s" % (tgt, suffix)],
+                   result["ttft_ms_%d%s" % (tgt, suffix)] / 1000.0, len(rates)))
+
+    for temp in TEMPS:
+        suffix = "" if temp == 0 else "_t%s" % fmt_temp(temp)
+        meds = [result["decode_tps_%d%s" % (t, suffix)] for t in DEPTHS]
+        result["decode_tps%s" % suffix] = round(statistics.median(meds), 2)
+    print(json.dumps(result))
+    return 0
+
+
+if __name__ == "__main__":
+    sys.exit(main())
@@ -0,0 +1,218 @@
+#!/usr/bin/env bash
+# frank-primary-window.sh — one full bench window for frank-primary, run ON
+# DANKSERVER as the deploy user (invoked over ssh by ./autoresearch.sh).
+#
+# Discipline (AGENTS.md Benchmarking Policy): the production unit is
+# STOPPED + MASKED first, the candidate config boots AS THE SERVICE itself
+# (same serve script, same container name, same port 7770 — no scratch
+# ports, no co-tenant GPU), the in-container measure runs, and the window
+# ALWAYS ends by restoring the production unit and verifying its health.
+# The EXIT trap makes the restore unconditional: a boot failure, a measure
+# failure, or a dead ssh session all end with production back up.
+#
+# Staging layout (created by the orchestrator, ephemeral — never a config
+# source; the repo checkout is):
+#   /home/dank/.autoresearch-frank-primary/
+#     serve.sh             staged candidate serve script
+#     frank-primary.env    staged candidate env (the candidate knobs)
+#     measure.py           the in-container measure client
+#     hash                 candidate fingerprint
+#     run.json             result (metrics + boot + kv + fingerprint)
+#     serve.out/measure.out/fail.log
+#     last/                memo: run.json + hash of the last FULLY
+#                          successful window (bench AND verified restore)
+#
+# This script logs to stderr and exits 0 only when the measure produced a
+# usable result (the restore happens in the trap regardless of the bench
+# outcome; a failed restore forces exit 1).
+set -uo pipefail
+
+STAGE="${FRANK_PRIMARY_AR_STAGE:-/home/dank/.autoresearch-frank-primary}"
+UNIT="${FRANK_PRIMARY_AR_UNIT:-frank-primary}"
+CTR="${FRANK_PRIMARY_AR_CTR:-frank-primary-qwen}"
+HEALTH_URL="http://127.0.0.1:7770/health"
+MODELS_URL="http://127.0.0.1:7770/v1/models"
+BOOT_TIMEOUT="${FRANK_PRIMARY_AR_BOOT_TIMEOUT:-1200}"
+RESTORE_TIMEOUT="${FRANK_PRIMARY_AR_RESTORE_TIMEOUT:-1200}"
+MEASURE_TIMEOUT="${FRANK_PRIMARY_AR_MEASURE_TIMEOUT:-1800}"
+SERVED="$(sed -n 's/^FRANK_PRIMARY_SERVED_NAME=//p' "$STAGE/frank-primary.env" 2>/dev/null | head -1)"
+SERVED="${SERVED:-frank-primary}"
+
+RESTORED=0
+
+log() { printf '[window %s] %s\n' "$(date '+%H:%M:%S')" "$*" >&2; }
+
+container_state() { podman inspect -f '{{.State.Status}}' "$CTR" 2>/dev/null || echo absent; }
+
+wait_port_free() {
+    local i
+    for i in $(seq 1 90); do
+        [ "$(container_state)" != "running" ] && return 0
+        sleep 2
+    done
+    return 1
+}
+
+wait_health() { # $1 = timeout seconds
+    local t0 code
+    t0=$(date +%s)
+    while :; do
+        code="$(curl -s -o /dev/null -w '%{http_code}' --max-time 10 "$HEALTH_URL" 2>/dev/null || echo 000)"
+        [ "$code" = "200" ] && return 0
+        [ "$(( $(date +%s) - t0 ))" -ge "$1" ] && return 1
+        sleep 2
+    done
+}
+
+capture_bench_failure() {
+    log "capturing bench-boot failure evidence"
+    {
+        echo "=== podman logs (tail 400) ==="
+        podman logs --tail 400 "$CTR" 2>&1 || true
+        echo "=== staged serve.out (tail 60) ==="
+        tail -60 "$STAGE/serve.out" 2>/dev/null || true
+        echo "=== engine log (tail 60) ==="
+        tail -60 /home/dank/logs/frank_primary.log 2>/dev/null || true
+    } >"$STAGE/fail.log" 2>&1 || true
+    grep -E 'OutOfMemory|Tried to allocate|Engine core initialization failed|Error|error' \
+        "$STAGE/fail.log" 2>/dev/null | tail -25 >&2 || true
+}
+
+restore() {
+    [ "$RESTORED" = 1 ] && return 0
+    RESTORED=1
+    log "restoring production unit $UNIT"
+    if [ -f "$STAGE/serve.pid" ]; then
+        kill "$(cat "$STAGE/serve.pid" 2>/dev/null)" 2>/dev/null || true
+    fi
+    podman stop -t 120 "$CTR" >/dev/null 2>&1 || true
+    wait_port_free || log "WARNING: bench container still draining after 180s"
+    sudo systemctl unmask "$UNIT" 2>/dev/null || true
+    if ! sudo systemctl start "$UNIT"; then
+        log "ERROR: systemctl start $UNIT failed — operator intervention required"
+        return 1
+    fi
+    if wait_health "$RESTORE_TIMEOUT"; then
+        if curl -s --max-time 10 "$MODELS_URL" 2>/dev/null | grep -q "$SERVED"; then
+            log "production restored and verified ($SERVED answering on :7770)"
+            # Memo promotion lives HERE: a cached result is only ever the
+            # product of a fully successful window (bench AND verified
+            # restore), never of a bare bench success.
+            cp "$STAGE/run.json" "$STAGE/last/" 2>/dev/null || true
+            cp "$STAGE/hash" "$STAGE/last/" 2>/dev/null || true
+        else
+            log "ERROR: health 200 but /v1/models does not list $SERVED"
+            return 1
+        fi
+    else
+        log "ERROR: production not healthy within ${RESTORE_TIMEOUT}s — unit left to systemd (Restart=always)"
+        return 1
+    fi
+    return 0
+}
+trap 'restore; rc=$?; [ "$rc" -ne 0 ] && exit 1' EXIT
+
+main() {
+    local f
+    for f in serve.sh frank-primary.env measure.py hash; do
+        [ -f "$STAGE/$f" ] || { log "staged file missing: $STAGE/$f"; exit 1; }
+    done
+    mkdir -p "$STAGE/last"
+    rm -f "$STAGE/run.json" "$STAGE/fail.log" "$STAGE/measure.out" "$STAGE/evidence.log"
+
+    local t_start t0 i kvline kv_tokens kv_conc measure_json boot_seconds
+    t_start="$(date +%s)"
+
+    log "stopping and masking production unit $UNIT"
+    sudo systemctl stop "$UNIT" 2>/dev/null || true
+    # Explicit stop is not auto-restarted even with Restart=always; the mask
+    # is belt-and-braces against a stray start during the window.
+    sudo systemctl mask "$UNIT" 2>/dev/null || \
+        log "WARNING: mask failed — relying on explicit-stop semantics"
+    wait_port_free || log "note: production container still draining"
+
+    t0="$(date +%s)"
+    log "starting bench instance from staged config (fingerprint $(tr -d '[:space:]' <"$STAGE/hash"))"
+    FRANK_PRIMARY_ENV_FILE="$STAGE/frank-primary.env" \
+        setsid nohup bash "$STAGE/serve.sh" >"$STAGE/serve.out" 2>&1 </dev/null &
+    echo "$!" >"$STAGE/serve.pid"
+
+    for i in $(seq 1 45); do
+        [ "$(container_state)" = "running" ] && break
+        sleep 2
+    done
+    if [ "$(container_state)" != "running" ]; then
+        log "bench container never reached running state"
+        tail -40 "$STAGE/serve.out" >&2 || true
+        exit 1
+    fi
+
+    if wait_health "$BOOT_TIMEOUT"; then
+        boot_seconds=$(( $(date +%s) - t0 ))
+        log "bench healthy after ${boot_seconds}s"
+    else
+        if [ "$(container_state)" = "running" ]; then
+            log "bench boot timed out after ${BOOT_TIMEOUT}s (still loading)"
+        else
+            log "bench container died during boot"
+            capture_bench_failure
+        fi
+        exit 1
+    fi
+
+    log "running measure in container (timeout ${MEASURE_TIMEOUT}s)"
+    if ! podman cp "$STAGE/measure.py" "$CTR":/tmp/ar-measure.py; then
+        log "podman cp of measure client failed"
+        exit 1
+    fi
+    if ! timeout "$MEASURE_TIMEOUT" podman exec "$CTR" python3 /tmp/ar-measure.py \
+            >"$STAGE/measure.out" 2>&1; then
+        log "measure failed or timed out"
+        tail -40 "$STAGE/measure.out" >&2 || true
+        exit 1
+    fi
+
+    # Capacity facts for the run (context-size experiments change these).
+    kvline="$(podman logs "$CTR" 2>&1 | grep -E 'GPU KV cache size' | tail -1 || true)"
+    kv_tokens="$(printf '%s' "$kvline" | sed -E 's/.*GPU KV cache size: ([0-9,]+) tokens.*/\1/' | tr -d ',' || true)"
+    kv_conc="$(printf '%s' "$kvline" | sed -E 's/.*per request: ([0-9.]+x).*/\1/' || true)"
+
+    # Human-facing evidence, in a sidecar file because it is multi-line prose,
+    # not metrics: the expert-plan lines (the ONLY place the arena / runtime
+    # overhead / KV-reserve numbers exist — they are never in /metrics), the
+    # compile path (cold build vs cache hit), the graph-capture sizes, the
+    # speculative-decode counters, and any OOM signature. Without this an arm
+    # that loses decode speed is unexplainable afterwards.
+    {
+        echo "=== engine plan / capacity / spec evidence ==="
+        podman logs "$CTR" 2>&1 | grep -E \
+'KV cache size|Maximum concurrency|layers split|expert cache|runtime overhead|non-default args|r4d all-reduce|Using \[|cudagraph|CUDA graph|Capturing|graph for compile range|torch.compile took|Directly load the compiled|speculative|Speculative|acceptance|Acceptance|offload|precise|OutOfMemory|Tried to allocate|Engine core initialization failed' \
+            | tail -150 || true
+        echo "=== /metrics counters (spec decode, caches, success) ==="
+        curl -s --max-time 15 http://127.0.0.1:7770/metrics 2>/dev/null \
+            | grep -E '^vllm:(spec_decode|gpu_cache|prefix_cache|request_success|generation_tokens|iteration)' \
+            | head -80 || true
+    } >"$STAGE/evidence.log" 2>&1 || true
+    log "engine evidence -> $STAGE/evidence.log ($(wc -l <"$STAGE/evidence.log") lines)"
+
+    measure_json="$(grep -E '^\{' "$STAGE/measure.out" | tail -1 || true)"
+    if [ -z "$measure_json" ]; then
+        log "measure produced no JSON result line"
+        tail -40 "$STAGE/measure.out" >&2 || true
+        exit 1
+    fi
+
+    # The measure line is already JSON, so the run object can be assembled
+    # without any host-side interpreter.
+    printf '{"hash":"%s","boot_seconds":%s,"kv_tokens":%s,"kv_concurrency":"%s","total_seconds":%s,"measure":%s}\n' \
+        "$(tr -d '[:space:]' <"$STAGE/hash")" \
+        "$boot_seconds" "${kv_tokens:-0}" "${kv_conc:-unknown}" \
+        "$(( $(date +%s) - t_start ))" "$measure_json" >"$STAGE/run.json"
+
+    log "bench window complete in $(( $(date +%s) - t_start ))s"
+    # Memo only after a FULL success; the restore below still has to verify.
+    exit 0
+}
+
+main "$@"
+exit "$?"
@@ -57,7 +57,7 @@ run_bench() {
    # Warmup (small request)
    curl -s -m 60 http://127.0.0.1:$PORT/v1/chat/completions \
        -H "Content-Type: application/json" \
-        -d '{"model":"frank-large","messages":[{"role":"user","content":"Hi"}],"max_tokens":16,"stream":false}' > /dev/null 2>&1 || true
+        -d '{"model":"frank-dragoon","messages":[{"role":"user","content":"Hi"}],"max_tokens":16,"stream":false}' > /dev/null 2>&1 || true
    
    # Build 8K prompt
    local prompt
@@ -0,0 +1,48 @@
+#!/usr/bin/env python3
+"""bench-probe.py — RUNS ON DANKSERVER. The campaign's decode probe: one
+/completion request, a prompt assembled from calib.jsonl to ~TARGET tokens,
+temperature 0, max_tokens 512. Reports the engine's own timings: decode rate,
+prefill rate, draft accept. Run 3x; the median is the arm's number."""
+import json, sys, urllib.request
+
+TARGET = 43600  # tokens
+BYTES_PER_TOK = 3.8
+URL = "http://localhost:7770/v1/completions"
+
+def build_prompt():
+    want = int(TARGET * BYTES_PER_TOK)
+    parts = []
+    n = 0
+    with open("/home/dank/corpora/out/calib.jsonl") as f:
+        for line in f:
+            d = json.loads(line)
+            p = d.get("prompt") or d.get("messages", [{}])[0].get("content", "")
+            parts.append(p)
+            n += len(p.encode())
+            if n >= want:
+                break
+    return "".join(parts)
+
+def probe(salt):
+    prompt = f"[probe {salt}]\n" + build_prompt()
+    body = json.dumps({
+        "model": "frank-archon",
+        "prompt": prompt, "temperature": 0.0, "max_tokens": 512,
+    }).encode()
+    req = urllib.request.Request(URL, data=body,
+        headers={"Content-Type": "application/json"})
+    with urllib.request.urlopen(req, timeout=1800) as r:
+        d = json.load(r)
+    t = d.get("timings", {})
+    return {
+        "prompt_n": t.get("prompt_n"),
+        "prefill_tps": round(t.get("prompt_per_second", 0), 1),
+        "pred_n": t.get("predicted_n"),
+        "decode_tps": round(t.get("predicted_per_second", 0), 2),
+        "draft_n": t.get("draft_n"),
+        "accept": (round(t["draft_n_accepted"] / t["draft_n"], 4)
+                   if t.get("draft_n") else None),
+    }
+
+if __name__ == "__main__":
+    print(json.dumps(probe(sys.argv[1] if len(sys.argv) > 1 else "a")))
@@ -39,7 +39,7 @@ def send_prefill_request():
    with open(PROMPT_FILE, "r") as f:
        prompt = f.read()
    body = json.dumps({
-        "model": "frank-large",
+        "model": "frank-dragoon",
        "messages": [{"role": "user", "content": prompt}],
        "max_tokens": 1,
        "stream": False
@@ -0,0 +1,166 @@
+#!/usr/bin/env bash
+# build-frank-primary-rad.sh — build frank-primary's .rad container from the
+# OFFICIAL Qwen/Qwen3.8-Flash-Next-FP8 checkpoint, at NATIVE precision.
+#
+#   bash scripts/build-frank-primary-rad.sh                # convert
+#   bash scripts/build-frank-primary-rad.sh --plan-only    # declare, no I/O
+#   bash scripts/build-frank-primary-rad.sh --restage      # redo staging
+#   bash scripts/build-frank-primary-rad.sh --force        # rebuild existing
+#
+# WHY A STAGING DIR (the one non-obvious step): rad-convert selects its
+# architecture plugin by the checkpoint's quantisation DESCRIPTOR —
+# derived from config.json's quantization_config (core/format/checkpoint.cpp:
+# "fp8" here). No shipped plugin claims (qwen4exp, "fp8"): the shipped
+# qwen4exp plugin claims (qwen4exp, ""), the descriptor recipe-built
+# containers carry (they are built from the bf16 upstream, which has no
+# quantization_config at all). The match is exact with no fallback, so an
+# unmodified convert refuses before declare. The descriptor is only a
+# selection key; per-weight encodings are data — the checkpoint reader
+# derives them from the tensors themselves (X.weight_scale_inv is the
+# scale plane; checkpoint.h). Staging drops the quantization_config key
+# (everything else is a relative symlink into the pristine download) so
+# the convert selects the shipped plugin. EVERY weight is then copied through
+# in its own encoding — block-FP8 codes + bf16 scale planes stay block-FP8 —
+# except TWO value-preserving reshapes that stack-qwen38-official-fp8.py
+# performs on the staging dir first, and ONE recipe rule that finishes the
+# PLE table:
+#   * experts: the plugin maps the STACKED layout (mlp.experts.gate_up_proj
+#     [E,2F,H] with the _scale_inv plane at the same index) while the official
+#     repo ships per-expert unfused tensors — staging byte-stacks them
+#     (gate rows first, no requantisation).
+#   * PLE: the plugin's fp8 branch wants the table codes beside a ONE-ELEMENT
+#     bf16 scale plane; the official repo ships 128 SCALELESS fp8 shards plus
+#     one bf16 scalar. Staging decodes to bf16 with the checkpoint's own
+#     scale, and the convert's rtn rule re-encodes at that same scale —
+#     every original code is restored exactly and the scale plane lands in
+#     the form the plugin declares.
+#
+# WHY A CONTAINER AT ALL: a checkpoint dir has no mapped/file tiers, and a
+# weight only a host kernel reads (the n-gram / PLE table, 51B params)
+# cannot be served from one (GUIDE §3.4). rad-convert also bakes the
+# tokenizer and the checkpoint's chat template; the runtime replaces the
+# template with froggeric's (--override-chat-template).
+#
+# The convert runs rad-convert inside the digest-pinned radiance image
+# (the host has no ROCm and needs none): /home/dank/models is mounted at
+# /models so the staging dir's RELATIVE symlinks resolve inside the
+# container. Output lands beside the raw download, owned by dank
+# (rootless podman maps container-root to the invoking user).
+set -euo pipefail
+
+SRC="/home/dank/models/Qwen/Qwen3.8-Flash-Next-FP8"
+STAGE="/home/dank/models/stage-qwen4exp-fp8"
+OUT_DIR="/home/dank/models/rad"
+OUT="$OUT_DIR/qwen4exp-fp8-native.rad"
+
+SCRIPT_DIR="$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)"
+# shellcheck disable=SC1091
+source "$SCRIPT_DIR/frank-primary.env"
+IMAGE="$FRANK_PRIMARY_IMAGE"
+
+plan_only=0 restage=0 force=0
+for a in "$@"; do
+    case "$a" in
+        --plan-only) plan_only=1 ;;
+        --restage)   restage=1 ;;
+        --force)     force=1 ;;
+        *) echo "unknown flag: $a" >&2; exit 2 ;;
+    esac
+done
+
+if [ ! -f "$SRC/config.json" ]; then
+    echo "ERROR: raw checkpoint missing: $SRC (run scripts/download-frank-primary.sh --fetch)" >&2
+    exit 1
+fi
+echo "verifying raw checkpoint first..."
+bash "$SCRIPT_DIR/download-frank-primary.sh"
+
+if [ -f "$OUT" ] && [ "$plan_only" -eq 0 ] && [ "$force" -eq 0 ]; then
+    echo "container already exists: $OUT"
+    echo "rebuild it with --force (the old file moves to ~/.AGENTTRASH)"
+    exit 0
+fi
+
+# ---- staging ----------------------------------------------------------------
+if [ "$restage" -eq 1 ] && [ -e "$STAGE" ]; then
+    TRASH="$HOME/.AGENTTRASH/$(date +%Y%m%d)-frank-primary-stage"
+    mkdir -p "$TRASH"
+    mv "$STAGE" "$TRASH/stage-$(date +%H%M%S)"
+    echo "old staging dir moved to $TRASH"
+fi
+if [ ! -f "$STAGE/config.json" ]; then
+    mkdir -p "$STAGE"
+    python3 - "$SRC/config.json" "$STAGE/config.json" <<'PY'
+import json, sys
+src, dst = sys.argv[1], sys.argv[2]
+cfg = json.load(open(src))
+dropped = cfg.pop("quantization_config", None)
+json.dump(cfg, open(dst, "w"), indent=2)
+print(f"staging config.json written (quantization_config dropped: {dropped is not None})")
+PY
+    # The index is COPIED like config.json, never symlinked: the reshape step
+    # rewrites the stage's index, and a symlink would write through to the
+    # raw checkpoint (measured 2026-10-05: the raw index was restamped with
+    # the stacked layout and the raw verify failed 143/144). Shards stay
+    # relative symlinks — relative so the targets resolve inside the convert
+    # container (the stage's parent is mounted at /models).
+    for f in "$SRC"/*; do
+        name="$(basename "$f")"
+        case "$name" in config.json|model.safetensors.index.json) continue ;; esac
+        ln -sfn "../Qwen/Qwen3.8-Flash-Next-FP8/$name" "$STAGE/$name"
+    done
+    cp "$SRC/model.safetensors.index.json" "$STAGE/"
+    echo "staging dir ready: $STAGE ($(ls "$STAGE" | wc -l) entries)"
+fi
+
+# ---- checkpoint reshape (stacked experts + bf16 PLE table) -------------------
+# stack-qwen38-official-fp8.py reshapes the staged checkpoint in place
+# (idempotent) and writes the PLE scale to $STAGE/ple_scale.txt — see the
+# script's docstring for why each transform is value-preserving.
+echo "stacking experts + merging the PLE table (value-preserving)..."
+python3 "$SCRIPT_DIR/stack-qwen38-official-fp8.py" --stage "$STAGE"
+PLE_SCALE="$(tr -d '\n' < "$STAGE/ple_scale.txt")"
+echo "PLE fixed scale: $PLE_SCALE"
+
+# ---- convert ----------------------------------------------------------------
+# rad-convert needs no GPU: declare resolves kernels and asks each about the
+# encoding it will read — CPU-side layout questions only. (If a plugin load
+# ever demands a card, add --device /dev/kfd --device /dev/dri to this run.)
+# --plan-only runs the same declare phase and exits before a byte is written.
+echo "rad-convert: $SRC (staged) -> $OUT"
+mkdir -p "$OUT_DIR"
+CONVERT_ARGS=(/models/stage-qwen4exp-fp8 -o /models/rad/qwen4exp-fp8-native.rad
+              --home /opt/radiance/share/radiance)
+[ "$plan_only" -eq 1 ] && CONVERT_ARGS+=(--plan-only -v)
+# THE ONE RECIPE RULE (see header): the PLE table arrives from staging as a
+# bf16 merge of the checkpoint's own fp8 codes decoded with the checkpoint's
+# own scale; rtn re-encodes it at that same scale — the codes come back
+# exactly and the plugin's one-element scale plane is produced.
+CONVERT_ARGS+=(--quant "blk.*.ple_ngram.weight=rtn:codes=fp8_e4m3,block=*x*,scale=bf16,rule=fixed,scale_value=$PLE_SCALE")
+[ "$force" -eq 1 ] && [ -f "$OUT" ] && {
+    TRASH="$HOME/.AGENTTRASH/$(date +%Y%m%d)-frank-primary-rad"
+    mkdir -p "$TRASH"
+    mv "$OUT" "$TRASH/qwen4exp-fp8-native.rad.$(date +%H%M%S)"
+    echo "previous container moved to $TRASH"
+}
+podman run --rm \
+    --entrypoint rad-convert \
+    -v /home/dank/models:/models \
+    "$IMAGE" "${CONVERT_ARGS[@]}"
+
+if [ "$plan_only" -eq 1 ]; then
+    echo "plan-only run complete (no container written)"
+    exit 0
+fi
+
+# ---- record -----------------------------------------------------------------
+echo "rad-info summary:"
+podman run --rm --entrypoint rad-info -v /home/dank/models:/models:ro \
+    "$IMAGE" "/models/rad/qwen4exp-fp8-native.rad" | head -20
+MIB=$(( $(stat -c %s "$OUT") / 1048576 ))
+echo "container: $OUT (${MIB} MiB; env floor $FRANK_PRIMARY_RAD_MIN_MIB MiB)"
+if [ "$MIB" -lt "$FRANK_PRIMARY_RAD_MIN_MIB" ]; then
+    echo "ERROR: container below the env floor — the convert was truncated" >&2
+    exit 1
+fi
+echo "build complete"
@@ -0,0 +1,89 @@
+#!/usr/bin/env bash
+# Build the ROCmFPX runtime that serves frank-large on Dankclaw.
+#
+#   bash scripts/build-rocmfpx.sh              # clone/fetch + build the pin
+#   bash scripts/build-rocmfpx.sh --verify     # only check the installed tree
+#
+# Runtime: kingjones30/ROCmFPX — charlie12345/ROCmFPX plus the model-card
+# patches, i.e. the fork that owns the ROCmFPx tensor formats. frank-large's
+# payload is ftype 115 (`Q8_0_ROCMFPX_AGENT`): 311 tensors of
+# GGML_TYPE_Q8_0_ROCMFPX (103), a type that exists ONLY in this fork, so
+# stock llama.cpp, gufo and every other loader on the host reject the file.
+# This is the same tree that served the Qwen3.8-Flash-Next ROCmFP4 STRIX
+# endpoint 2026-08-30 → 09-11 (see the kingjones runbook), rebuilt here so a
+# fresh host can be brought up from the repo alone.
+#
+# Why this commit and not a newer one: 36e9acd (2026-08-28) is what was
+# qualified against this payload on 2026-10-02 — it loads ftype 115, survives
+# warmup, and drafts with the MTP head at acceptance 0.65. The five legacy
+# builds of 2026-08-25 crashed this same file in ggml_compute_forward_set_rows
+# (see kingjones-q8agent-rocmfpx.md), so the pin is load-bearing: do not bump
+# it without re-running that crash test.
+#
+# Host notes:
+#   * -DLLAMA_BUILD_WEBUI=OFF is REQUIRED on this box. The tree provisions its
+#     embedded UI assets via npm or the HF llama-ui bucket; npm is absent here
+#     and the bucket download half-failed, aborting the link at 98%. The API
+#     server is unaffected.
+#   * The build links against the host ROCm (/opt/rocm); no bundled toolchain
+#     like the gfx1201 R9700 build needed.
+set -euo pipefail
+
+ROCMFPX_REPO="${ROCMFPX_REPO:-https://github.com/kingjones30/ROCmFPX.git}"
+# The commit frank-large was qualified with (measured 2026-10-02).
+ROCMFPX_COMMIT="${ROCMFPX_COMMIT:-36e9acd40e10a87cd3c3ef8ec734668757dc8520}"
+SRC_DIR="${ROCMFPX_SRC_DIR:-/home/dank/GitPackages/kingjones-ROCmFPX}"
+BUILD_DIR="${ROCMFPX_BUILD_DIR:-$SRC_DIR/build}"
+GPU_TARGET="${ROCMFPX_GPU_TARGET:-gfx1151}"
+JOBS="${JOBS:-$(nproc)}"
+
+# check_pin — refuse a source tree that is not the qualified commit. A moved
+# fork means different kernels under the same path, and on this model line a
+# wrong build is a segfault at first request, not a clean error.
+check_pin() {
+    local commit
+    commit="$(git -C "$SRC_DIR" rev-parse HEAD 2>/dev/null || true)"
+    if [[ -n "$commit" && "$commit" != "$ROCMFPX_COMMIT" ]]; then
+        echo "ROCMFPX_COMMIT mismatch: $SRC_DIR is at $commit, pinned $ROCMFPX_COMMIT" >&2
+        exit 1
+    fi
+}
+
+if [[ "${1:-}" == "--verify" ]]; then
+    check_pin
+    [[ -x "$BUILD_DIR/bin/llama-server" ]] || {
+        echo "no ROCmFPX llama-server at $BUILD_DIR/bin/llama-server" >&2
+        exit 1
+    }
+    # The format the endpoint exists for: the fork must carry the ROCmFPx
+    # tensor type, or the payload cannot load at all.
+    "$BUILD_DIR/bin/llama-server" --help 2>&1 | grep -q -- '--spec-type' || {
+        echo "$BUILD_DIR/bin/llama-server has no --spec-type: MTP drafting is impossible" >&2
+        exit 1
+    }
+    "$BUILD_DIR/bin/llama-server" --version | head -1
+    exit 0
+fi
+
+echo "== source: $ROCMFPX_REPO @ $ROCMFPX_COMMIT =="
+if [[ ! -d "$SRC_DIR/.git" ]]; then
+    mkdir -p "$(dirname "$SRC_DIR")"
+    git clone "$ROCMFPX_REPO" "$SRC_DIR"
+fi
+git -C "$SRC_DIR" fetch --quiet origin "$ROCMFPX_COMMIT" || true
+git -C "$SRC_DIR" checkout -q --detach "$ROCMFPX_COMMIT"
+check_pin
+echo "   $(git -C "$SRC_DIR" rev-parse HEAD)"
+
+echo "== configure + build (release, HIP $GPU_TARGET) =="
+cmake -S "$SRC_DIR" -B "$BUILD_DIR" \
+    -DGGML_HIP=ON \
+    -DGPU_TARGETS="$GPU_TARGET" \
+    -DGGML_NATIVE=ON \
+    -DCMAKE_BUILD_TYPE=Release \
+    -DLLAMA_BUILD_WEBUI=OFF
+cmake --build "$BUILD_DIR" --target llama-server llama-quantize -j "$JOBS"
+
+echo "== verify =="
+"$BUILD_DIR/bin/llama-server" --version | head -1
+echo "frank-large runtime ready: $BUILD_DIR/bin/llama-server ($ROCMFPX_COMMIT)"
@@ -0,0 +1,110 @@
+#!/usr/bin/env python3
+"""calib-push.py — drive a calibration corpus through a radiance scratch
+server's prefill so its tap writes layer Hessians (docs/MOE-W4.md).
+
+Runs ON DANKSERVER next to the scratch server (the one autoresearch- /
+serve-style scratch container booted with RADIANCE_CALIB_DIR set).
+Reads the corpus JSONL produced by make-calib-corpus.py
+({"prompt": "..."} a line), POSTs each prompt to /v1/completions with
+max_tokens=1 (prefill-only cost, one token out), at N-way concurrency
+matching the scratch server's --max-num-seqs budget.
+
+Corpus rounds repeat until the engine prints the tap's completion line
+("calibration wrote layer Hessians") in `podman logs <container>` — that
+is the ground truth that RADIANCE_CALIB_TOKENS was reached. After that
+line, further prompts are pure waste, so the driver exits.
+
+Mechanics notes:
+- requests carry no sampling params that matter (max_tokens=1, greedy).
+- a per-request timeout of 30 min: at ~1k tok/s prefill a 12k-token
+  prompt is a few seconds; the timeout only catches a wedged engine.
+- failures print and continue: the tap pools over tokens, a few lost
+  prompts do not bias the Hessians (the corpus is drawn round-robin over
+  source classes, so loss is class-neutral as long as failures are rare;
+  a systematic 4xx is loud in the log and must be fixed, not ignored).
+
+Usage: calib-push.py [--port 7770] [--parallel 64] [--container arbench]
+       [--corpus /home/dank/corpora/out/calib.jsonl] [--max-rounds 4]
+"""
+import argparse
+import concurrent.futures
+import json
+import subprocess
+import sys
+import time
+import urllib.request
+
+DONE_MARK = "calibration wrote"
+
+
+def tap_done(container):
+    """True if the engine logged the tap's completion line."""
+    try:
+        out = subprocess.run(
+            ["podman", "logs", "--tail", "50", container],
+            capture_output=True, text=True, timeout=30,
+        )
+    except Exception:
+        return False
+    return DONE_MARK in (out.stdout + out.stderr)
+
+
+def push_one(url, prompt):
+    body = json.dumps({
+        "model": "frank-archon",
+        "prompt": prompt,
+        "max_tokens": 1,
+        "temperature": 0,
+        "stream": False,
+    }).encode()
+    req = urllib.request.Request(
+        url, data=body, headers={"Content-Type": "application/json"})
+    try:
+        with urllib.request.urlopen(req, timeout=1800) as r:
+            r.read()
+        return True
+    except Exception as e:
+        print(f"FAIL {str(e)[:120]}", flush=True)
+        return False
+
+
+def main():
+    ap = argparse.ArgumentParser()
+    ap.add_argument("--port", type=int, default=7770)
+    ap.add_argument("--parallel", type=int, default=64)
+    ap.add_argument("--container", default="arbench")
+    ap.add_argument("--corpus", default="/home/dank/corpora/out/calib.jsonl")
+    ap.add_argument("--max-rounds", type=int, default=4)
+    a = ap.parse_args()
+    url = f"http://127.0.0.1:{a.port}/v1/completions"
+
+    prompts = []
+    with open(a.corpus) as f:
+        for line in f:
+            line = line.strip()
+            if line:
+                prompts.append(json.loads(line)["prompt"])
+    print(f"corpus: {len(prompts)} prompts", flush=True)
+
+    for rnd in range(1, a.max_rounds + 1):
+        if tap_done(a.container):
+            break
+        t0 = time.time()
+        ok = 0
+        with concurrent.futures.ThreadPoolExecutor(a.parallel) as ex:
+            for good in ex.map(lambda p: push_one(url, p), prompts):
+                ok += 1 if good else 0
+        dt = time.time() - t0
+        print(f"round {rnd}: {ok}/{len(prompts)} ok in {dt:.0f}s "
+              f"({sum(len(p) for p in prompts)/4/dt:.0f} chars/s est)",
+              flush=True)
+        if tap_done(a.container):
+            break
+    else:
+        print("max rounds exhausted, tap not done", flush=True)
+        sys.exit(1)
+    print("TAP_DONE: calibration wrote layer Hessians", flush=True)
+
+
+if __name__ == "__main__":
+    main()
@@ -0,0 +1,73 @@
+#!/usr/bin/env bash
+# convert-swift15-w4dn8.sh — RUNS ON DANKSERVER. The final recipe convert of
+# the Swift 1.5 checkpoint: gate_up experts through GPTQ against the 10M-token
+# calibration Grams, down projections block-fp8, protected experts bf16,
+# trunk int8, PLE fixed-scale fp8 (scripts/recipes/
+# qwen4exp-swift15-w4dn8-i8-hc8m.recipe).
+#
+# The convert engine is 1.0.13 — the SAME pin that will serve the result, so
+# the .rad crosses no version boundary. Only the Gram files cross versions
+# (1.2.0 wrote them; the RADGRAM2 magic, gram.r%d.%s.bin naming and the
+# calib_gram/calib_resolve/calib_stem symbols are byte-identical between the
+# two libquant.so). rad-convert runs on the host backend (no GPU visible in
+# the container) — production can stay up through this script.
+#
+# Prereqs (campaign order): calibration complete (calib-push printed TAP_DONE),
+# the protected-expert cast lines rewritten into the recipe by
+# derive-protected-experts.py, and the 335 GiB bf16 container deleted —
+# the peak-disk arithmetic needs that space (checkpoint 335.3 + output ~132
+# GiB on the volume; --reuse is NOT used, see the recipe header).
+set -euo pipefail
+
+PIN_1013=sha256:4c3f484564bb2872784e46b0a59cbd71becc9c1e746e301a3596ffae253c322e
+IMAGE="stilldeadcode/radiance@$PIN_1013"
+CKPT=/home/dank/models/ukisai/Swift1.5-Qwen3.8-Flash-Next
+RECIPE=/home/dank/serverAppConfigs-swift15/scripts/recipes/qwen4exp-swift15-w4dn8-i8-hc8m.recipe
+CALIB=/home/dank/corpora/calib
+OUT=/home/dank/models/rad/qwen4exp-swift15-w4dn8-i8-hc8m.rad
+LOG=/home/dank/logs/swift15_convert.log
+# gate_up ~37.6 + down ~18.8 + PLE 47.7 + trunk/embeddings/head ~10 GiB (all-w4 expert table;
+# the plugin's one-encoding law dropped the block-fp8 down tier -- see the recipe header).
+MIN_MIB=${FRANK_SWIFT15_RAD_MIN_MIB:-108000}
+
+# Gate 1: inputs. A missing Gram directory converts UNCALIBRATED with a
+# per-tensor stderr note, which is silently worse than failing here.
+[ -d "$CKPT" ] || { echo "no checkpoint at $CKPT" >&2; exit 1; }
+[ -f "$RECIPE" ] || { echo "no recipe at $RECIPE" >&2; exit 1; }
+GRAMS=$(find "$CALIB" -maxdepth 1 -name 'gram.r*.bin' 2>/dev/null | wc -l)
+[ "$GRAMS" -ge 48 ] || { echo "calib dir $CALIB holds $GRAMS gram files (<48)" >&2; exit 1; }
+if [ -e "$OUT" ]; then echo "$OUT already exists; move it to trash first" >&2; exit 1; fi
+if [ -e /home/dank/models/rad/qwen4exp-swift15-bf16.rad ]; then
+    echo "the bf16 container is still on disk; delete it first (peak disk)" >&2; exit 1
+fi
+
+# Gate 2: disk. Output ~132 GiB plus convert staging, 20 GiB of slack.
+FREE_MIB=$(df --output=avail -m /home/dank/models | tail -1 | tr -d ' ')
+[ "$FREE_MIB" -ge $((MIN_MIB + 20480)) ] || {
+    echo "only ${FREE_MIB} MiB free on /home/dank/models" >&2; exit 1; }
+
+# The recipe names $CALIB; rad-convert expands it from the environment
+# (MOE-W4.md:275). The recipe's cast section must name THIS checkpoint's
+# protected experts — derive-protected-experts.py rewrote it.
+echo "convert: $CKPT -> $OUT (gptq calib=$CALIB, $(date -Is))" | tee "$LOG"
+podman run --rm --name swift15-convert \
+    -v /home/dank/models:/models \
+    -v /home/dank/serverAppConfigs-swift15:/campaign:ro \
+    -v "$CALIB:/data/calib:ro" \
+    -e CALIB=/data/calib \
+    --entrypoint /opt/radiance/bin/rad-convert \
+    "$IMAGE" "/models/${CKPT#/home/dank/models/}" \
+    --recipe /campaign/"${RECIPE#/home/dank/serverAppConfigs-swift15/}" \
+    -o /models/rad/"$(basename "$OUT")" 2>&1 | tee -a "$LOG"
+rc=${PIPESTATUS[0]}
+[ "$rc" -eq 0 ] || { echo "rad-convert rc=$rc; log: $LOG" >&2; exit "$rc"; }
+
+# Gate 3: the artifact. rad-info's recipe echo is the receipt that GPTQ ran.
+SIZE_MIB=$(( $(stat -c %s "$OUT") / 1048576 ))
+[ "$SIZE_MIB" -ge "$MIN_MIB" ] || {
+    echo "$OUT is ${SIZE_MIB} MiB < floor ${MIN_MIB}" >&2; exit 1; }
+podman run --rm --entrypoint /opt/radiance/bin/rad-info \
+    -v /home/dank/models:/models:ro "$IMAGE" /models/rad/"$(basename "$OUT")" \
+    2>&1 | tee -a "$LOG" | grep -c 'gptq' >/dev/null \
+    || echo "WARNING: rad-info shows no gptq rule in the recorded recipe" >&2
+echo "CONVERT_DONE ${SIZE_MIB} MiB -> $OUT"
@@ -0,0 +1,55 @@
+#!/usr/bin/env python3
+"""deep-probe.py — one chat completion at ~43.6k prompt tokens, temp 0, 512
+generated, streamed; decode = completion_tokens / (total - ttft)."""
+import json, time, urllib.request
+
+TARGET_BYTES = int(43600 * 3.8)
+
+def build():
+    parts, n = [], 0
+    with open("/home/dank/corpora/out/calib.jsonl") as f:
+        for line in f:
+            p = json.loads(line)["prompt"]
+            parts.append(p)
+            n += len(p.encode())
+            if n >= TARGET_BYTES:
+                break
+    return "".join(parts)
+
+def main(salt):
+    body = json.dumps({
+        "model": "frank-archon",
+        "messages": [{"role": "user", "content": f"[probe {salt}] Continue and summarize this material in English, 400 words.\n\n" + build()}],
+        "temperature": 0.0, "max_tokens": 512,
+        "stream": True, "stream_options": {"include_usage": True},
+    }).encode()
+    req = urllib.request.Request("http://localhost:7770/v1/chat/completions",
+        data=body, headers={"Content-Type": "application/json"})
+    t0 = time.time(); ttft = None; ctok = 0; usage = {}
+    with urllib.request.urlopen(req, timeout=2400) as r:
+        for raw in r:
+            line = raw.decode().strip()
+            if not line.startswith("data:"):
+                continue
+            d = line[5:].strip()
+            if d == "[DONE]":
+                break
+            j = json.loads(d)
+            if j.get("usage"):
+                usage = j["usage"]
+            for ch in j.get("choices", []):
+                delta = (ch.get("delta") or {})
+                piece = delta.get("content") or delta.get("reasoning_content") or ""
+                if piece and ttft is None:
+                    ttft = time.time() - t0
+                ctok += 1
+    total = time.time() - t0
+    gen = usage.get("completion_tokens") or ctok
+    dec = gen / (total - (ttft or 0)) if total > (ttft or 0) else 0
+    print(json.dumps({"salt": salt, "prompt_tokens": usage.get("prompt_tokens"),
+                      "completion_tokens": gen, "ttft_s": round(ttft or 0, 2),
+                      "total_s": round(total, 1), "decode_tps": round(dec, 2)}))
+
+if __name__ == "__main__":
+    import sys
+    main(sys.argv[1] if len(sys.argv) > 1 else "x")
@@ -62,7 +62,7 @@ SERVICE_ORDER = (
    "lidarr",
    "readarr",
    "frank-large",
-    "frank-small",
+    "frank-secondary",
    "frank-primary",
 )

@@ -94,7 +94,7 @@ DANKCLAW_SERVICES = {
    "caddy",
    "homepage",
    "frank-large",
-    "frank-small",
+    "frank-secondary",
 }

 DANKSERVER_SERVICES = {
@@ -400,12 +400,12 @@ def services_for_path(path: str, target: str) -> set[str]:
            # change to it redeploying alertmanager is the only path that
            # reaches it on the host.
            _add(services, "alertmanager")
-        if name in {"frank-small.env", "frank-small.service", "serve-frank-small.sh"}:
-            # llama.cpp frank-small (2026-09-16) keeps its unit, env and serve
-            # files here; each one selects frank-small only. The ik_llama.cpp
+        if name in {"frank-secondary.env", "frank-secondary.service", "serve-frank-secondary.sh"}:
+            # llama.cpp frank-secondary (2026-09-16) keeps its unit, env and serve
+            # files here; each one selects frank-secondary only. The ik_llama.cpp
            # trio that lived here for two days is in scripts/.AGENTTRASH/,
            # which is archive material and plans nothing.
-            _add(services, "frank-small")
+            _add(services, "frank-secondary")
        if name in {"frank-large.env", "frank-large.service", "serve-frank-large.sh"}:
            # CIRU v4.4.1 frank-large (2026-09-19) keeps its unit, env and
            # serve files here the same way: a root unit on a host-built
@@ -90,7 +90,7 @@ service_is_allowed() {
        dankclaw:authentik-postgres|dankclaw:authentik-server|\
        dankclaw:authentik-worker|dankclaw:authentik-outpost|dankclaw:authentik-outpost-ldap|dankclaw:anubis|\
        dankclaw:caddy|dankclaw:homepage|dankclaw:seerr|dankclaw:marreta|\
-        dankclaw:frank-large|dankclaw:frank-small|\
+        dankclaw:frank-large|dankclaw:frank-secondary|\
        dankserver:romm-mariadb|dankserver:romm-redis|dankserver:romm|\
        dankserver:copyparty|dankserver:jellyfin|dankserver:calibre-web|dankserver:calibre-server|dankserver:plex|dankserver:tautulli|\
        dankserver:prowlarr|dankserver:qbittorrent|dankserver:radarr|\
@@ -322,6 +322,12 @@ deploy_homepage() {
    if [[ -d shared/homepage/fonts ]]; then
        run_privileged cp -r shared/homepage/fonts /srv/homepage/
    fi
+    if [[ -d shared/homepage/img ]]; then
+        run_privileged cp -r shared/homepage/img /srv/homepage/
+    fi
+    if [[ -d shared/homepage/sfx ]]; then
+        run_privileged cp -r shared/homepage/sfx /srv/homepage/
+    fi
    local deployed_at
    deployed_at="$(git log -1 --format=%cI)"
    printf 'window.DEPLOYED_AT = "%s";\n' "$deployed_at" > /tmp/deployed.js
@@ -935,15 +941,24 @@ deploy_plex() {
    fi
    systemctl --user restart plex.service
    # Readiness gate: /identity answers MediaContainer XML once PMS is up.
+    #
+    # HTTPS, NOT HTTP. PMS drops plain HTTP off-loopback: the probe's
+    # connection arrives on the container's eth0, so an http:// request is
+    # accepted and then answered with nothing (curl 52, Empty reply) against a
+    # perfectly healthy server. Measured 2026-10-02 — the gate burned its full
+    # 600s and failed a plex whose https://127.0.0.1:32400/identity answered
+    # 200 in 44 ms the whole time. The same law is why the Caddyfile dials the
+    # plex upstream over TLS (tests/caddyfile.bats pins it).
+    # -k: PMS presents its own self-signed cert on 32400.
+    #
    # Reality check from the 2026-09 cutover: the FIRST container boot of a
-    # years-old library ran ~10 min (PMS pre-binds 32400 during boot and
-    # empty-replies until the HTTP layer is ready), so the gate allows 600s.
+    # years-old library ran ~10 min, so the gate still allows 600s.
    # Fail the deploy (pending state preserved) rather than mark it deployed blind.
    for ((i = 1; i <= 120; i++)); do
-        curl -fsS --max-time 3 http://127.0.0.1:32400/identity >/dev/null && return 0
+        curl -fsSk --max-time 3 https://127.0.0.1:32400/identity >/dev/null && return 0
        sleep 5
    done
-    echo "FAIL: plex did not answer /identity within 600s" >&2
+    echo "FAIL: plex did not answer /identity over HTTPS within 600s" >&2
    return 1
 }

@@ -1077,27 +1092,37 @@ deploy_anubis() {
    wait_container_healthy anubis 60 || return 1
 }

-# frank-large is gufo (gufo-org/gufo), the Strix Halo inference engine built
-# from the pinned v0.1.1 tag by scripts/build-gufo.sh, serving
-# unsloth/Qwen3.8-27B-GGUF:UD-Q8_K_XL with the DFlash2 drafter
-# (z-lab/Qwen3.8-27B-DFlash2-GGUF:Q4_K_M) on the Strix Halo iGPU. It is a
-# ROOT unit: the runtime is a native binary linked against the host ROCm,
-# so there is no container image to install and no generated unit to dodge.
-# relink_unit points /etc/systemd/system/frank-large.service at the repo
-# file and boot persistence is that unit's [Install] section.
+# frank-large is the halogen-flash-server CONTAINER (0.16.2, digest-pinned
+# through scripts/frank-large.env) serving the Quat3rnion ABLITERATED
+# Swift1.5 finetune of peonist's Qwen3.8-Flash-Next v2 checkpoint —
+# qwen38-flash-next-v2-swift15-abliterated.hgn (66,687,678,464 B, ~62 GiB
+# resident) + the REQUIRED n-gram lookup table + the vision tower —
+# MTP spec decode (HALOGEN_SPEC_ADAPT=0), 2 slots x 200192, image input —
+# on the Strix Halo iGPU. It stays a ROOT unit because the serve script
+# drives `podman run` directly (the frank-primary pattern): relink_unit
+# points /etc/systemd/system/frank-large.service at the repo file, boot
+# persistence is that unit's [Install], and LimitMEMLOCK=infinity is
+# fleet parity. 2026-10-06 cutover from the halogen 27B-dense shape —
+# rollback = revert the swap commit's trio; the 27B payload stays on
+# disk (~/models/halogen-qwen3.8-27b).
 #
-# The runtime gate matters as much as the payload gate: gufo is built from
-# source on this host (there is no prebuilt release), so a machine whose
-# /home/dank/GitPackages tree was never built has a unit that can never
-# become healthy, and a green deploy would leave the endpoint dark.
-#
-# It replaces the stock llama.cpp prebuilt (unsloth b11030), which had
-# replaced the Kairic Edge runtime (ciru-ai/ROCmFPX). The port gate is the
-# same as every cutover since 2026-09-16: nothing starts until 7770 is
-# released.
-LLM_LARGE_RUNTIME="${LLM_LARGE_RUNTIME:-/home/dank/GitPackages/gufo-v0.1.1/bin/gufo}"
-LLM_LARGE_MODEL="${LLM_LARGE_MODEL:-/home/dank/models/Swift-1.5-Qwen3.8-27B-Uncensored-MTP-UD-Q8_K_XL/Swift-1.5-Qwen3.8-27B-Uncensored-Dynamic-MTP-UD-Q8_K_XL.gguf}"
-LLM_LARGE_DRAFT="${LLM_LARGE_DRAFT:-/home/dank/models/Qwen3.8-27B-UD-Q8_K_XL/Qwen3.8-27B-DFlash2-Q4_K_M.gguf}"
+# The gates mirror serve-frank-large.sh's predicates EXACTLY (the
+# 2026-10-01 crash-loop lesson): image digest in dank's podman storage,
+# each .hgn payload (checkpoint, n-gram table, vision tower) at the exact
+# byte count, the tokenizer directory holding every named file as a
+# regular (not symlinked) file — a green deploy against a missing
+# artifact would leave a restart-looping unit and a dark endpoint. The
+# port gate is the same as every cutover since 2026-09-16: nothing
+# starts until 7770 is released.
+LLM_LARGE_IMAGE="${LLM_LARGE_IMAGE:-ghcr.io/peonist-ai/halogen-flash-server@sha256:0c61bf84ac22308a53f5d1ca6b86806702d7039e5ebc51cae4c66621b92fe04a}"
+LLM_LARGE_CKPT="${LLM_LARGE_CKPT:-/home/dank/models/halogen-flashnext-v2/qwen38-flash-next-v2-swift15-abliterated.hgn}"
+LLM_LARGE_CKPT_BYTES="${LLM_LARGE_CKPT_BYTES:-66687678464}"
+LLM_LARGE_NGRAM="${LLM_LARGE_NGRAM:-/home/dank/models/halogen-flashnext-v2/qwen38-flash-next-ngram.hgn}"
+LLM_LARGE_NGRAM_BYTES="${LLM_LARGE_NGRAM_BYTES:-51200246144}"
+LLM_LARGE_VISION="${LLM_LARGE_VISION:-/home/dank/models/halogen-flashnext-v2/qwen38-flash-next-vision.hgn}"
+LLM_LARGE_VISION_BYTES="${LLM_LARGE_VISION_BYTES:-897916416}"
+LLM_LARGE_TOK_DIR="${LLM_LARGE_TOK_DIR:-/home/dank/models/halogen-flashnext-v2/tokenizer}"
+LLM_LARGE_TOK_FILES="${LLM_LARGE_TOK_FILES:-tokenizer.json chat_template.jinja tokenizer_config.json merges.txt vocab.json generation_config.json}"

 retire_halogen_large_quadlet() {
    local trash="$HOME/.AGENTTRASH/20260919-orca-cutover/quadlet"
@@ -1130,45 +1155,65 @@ wait_port_free() {
 }

 deploy_llm_large() {
-    # Refuse rather than restart-loop. A unit whose runtime or payload is
-    # missing never becomes healthy, and a green deploy would leave the
-    # endpoint dark with nothing in the deploy log to say why.
-    if [[ ! -x "$LLM_LARGE_RUNTIME" ]]; then
-        echo "ERROR: gufo runtime absent at $LLM_LARGE_RUNTIME" >&2
-        echo "frank-large: build it: bash scripts/build-gufo.sh" >&2
+    # Refuse rather than restart-loop; every gate below has a twin in
+    # serve-frank-large.sh and they must agree (2026-10-01 lesson).
+    # Capture podman output before grepping: `podman images | grep -q`
+    # races under pipefail (SIGPIPE 141 becomes "no match", 2026-09-30).
+    if ! grep -qF "${LLM_LARGE_IMAGE#*@}" <<<"$(podman images --format '{{.RepoDigests}}' 2>/dev/null)"; then
+        echo "ERROR: halogen image absent: $LLM_LARGE_IMAGE" >&2
+        echo "frank-large: pull it: podman pull ${LLM_LARGE_IMAGE%@*}" >&2
        return 1
    fi
-    local missing=0 f
-    for f in "$LLM_LARGE_MODEL" "$LLM_LARGE_DRAFT"; do
-        if [[ ! -f "$f" ]]; then
-            echo "ERROR: payload absent at $f" >&2
-            echo "frank-large: fetch it: bash scripts/download-frank-large.sh" >&2
-            missing=1
+    if [[ ! -f "$LLM_LARGE_CKPT" ]]; then
+        echo "ERROR: halogen checkpoint absent at $LLM_LARGE_CKPT" >&2
+        echo "frank-large: fetch it: bash scripts/download-frank-large-flashnext.sh" >&2
+        return 1
+    fi
+    if [[ "$(stat -c %s "$LLM_LARGE_CKPT")" != "$LLM_LARGE_CKPT_BYTES" ]]; then
+        echo "ERROR: halogen checkpoint is not $LLM_LARGE_CKPT_BYTES bytes (partial download?)" >&2
+        echo "frank-large: re-check: bash scripts/download-frank-large-flashnext.sh --verify-only" >&2
+        return 1
+    fi
+    for gate in "$LLM_LARGE_NGRAM:$LLM_LARGE_NGRAM_BYTES:n-gram table" \
+                "$LLM_LARGE_VISION:$LLM_LARGE_VISION_BYTES:vision tower"; do
+        IFS=: read -r gpath gbytes gname <<<"$gate"
+        if [[ ! -f "$gpath" ]]; then
+            echo "ERROR: halogen $gname absent at $gpath" >&2
+            echo "frank-large: fetch it: bash scripts/download-frank-large-flashnext.sh" >&2
+            return 1
+        fi
+        if [[ "$(stat -c %s "$gpath")" != "$gbytes" ]]; then
+            echo "ERROR: halogen $gname is not $gbytes bytes (partial download?)" >&2
+            echo "frank-large: re-check: bash scripts/download-frank-large-flashnext.sh --verify-only" >&2
+            return 1
+        fi
+    done
+    if [[ -L "$LLM_LARGE_TOK_DIR" || ! -d "$LLM_LARGE_TOK_DIR" ]]; then
+        echo "ERROR: tokenizer dir missing or a symlink at $LLM_LARGE_TOK_DIR" >&2
+        echo "frank-large: fetch it: bash scripts/download-frank-large-flashnext.sh" >&2
+        return 1
+    fi
+    for tf in $LLM_LARGE_TOK_FILES; do
+        if [[ ! -f "$LLM_LARGE_TOK_DIR/$tf" || -L "$LLM_LARGE_TOK_DIR/$tf" ]]; then
+            echo "ERROR: tokenizer file missing or a symlink: $LLM_LARGE_TOK_DIR/$tf" >&2
+            echo "frank-large: fetch it: bash scripts/download-frank-large-flashnext.sh" >&2
+            return 1
        fi
    done
-    [[ "$missing" -eq 0 ]] || return 1

    retire_halogen_large_quadlet
-    # On a re-deploy the predecessor is THIS unit: stop it before waiting on
-    # the port, or the wait deadlocks on the server it just stopped serving.
-    # Then sweep a stray manual run -- that is how a bench harness leaves the
-    # port busy. Match the DRAFTER path, which every frank-large command
-    # line carries via --dflash-model and which is UNCHANGED by the 2026-09-28
-    # payload swap: the target moved to a different repository while the
-    # drafter stayed put, so keying on the target filename would have quietly
-    # stopped matching. It is also the right key in principle — it names this
-    # endpoint's speculative drafter, not a binary. Match that, not a binary
-    # name: frank-small on this host serves a different GGUF over llama-server
-    # with no DFlash2 drafter and must not be killed. The pattern never
-    # appears in the deploying shell's own command line, so -f is safe here
-    # (it is not in `bash deploy-remote.sh`).
+    # The predecessor on a re-deploy is THIS unit: stop it (ExecStop stops
+    # the container) before waiting on the port. The stray sweep is keyed
+    # on the FULL container name, which appears in the argv of a manual
+    # `podman run --name frank-large-halogen` / `podman wait` but NOT in
+    # this deployer's own command line (`bash deploy-remote.sh
+    # frank-large`) — so it cannot shoot itself in the foot (the fleet
+    # pkill law), and it can never match frank-secondary's llama-server.
    run_privileged systemctl stop frank-large.service 2>/dev/null || true
-    pkill -f 'Qwen3.8-27B-DFlash2-Q4_K_M' 2>/dev/null || true
-    # Nothing may start until the predecessor has released the port. On
-    # 2026-09-16 the cutover failed exactly here: the old server logged
-    # "Deactivated successfully", the next engine tried to bind 7770 a second
-    # later and reported "Address already in use". Gate on the port, not on
-    # hope.
+    pkill -f 'frank-large-halogen' 2>/dev/null || true
+    podman rm -f frank-large frank-large-halogen >/dev/null 2>&1 || true
+    # Nothing may start until the predecessor has released the port
+    # (2026-09-16: "Deactivated successfully" was not "port free").
    wait_port_free 7770 || return 1
    relink_unit frank-large
    run_privileged systemctl daemon-reload
@@ -1177,91 +1222,116 @@ deploy_llm_large() {
    wait_for_health frank-large http://127.0.0.1:7770/health 1200
 }

-# deploy_llm_small — frank-small is mainline llama.cpp on the RTX 3090, as a
-# ROOT unit: the runtime is a native CUDA binary, so there is no container
-# image to install and no generated unit to avoid. relink_unit points
-# /etc/systemd/system/frank-small.service at the repo file and boot persistence
-# is that unit's [Install] section.
+# deploy_llm_secondary — frank-secondary is llama.cpp (master build, CUDA sm_86) on
+# the RTX 3090 as a ROOT unit: the runtime is a native binary tree under
+# ~/GitPackages/llama-master/build-cuda, so there is no container image and
+# no generated unit to avoid. relink_unit points
+# /etc/systemd/system/frank-secondary.service at the repo file and boot
+# persistence is that unit's [Install] section.
 #
 # It is a SEPARATE endpoint on port 7771; frank-large keeps 7770. The 3090 is
 # not shared: frank-large's engine runs on the gfx1151 iGPU.
-LLM_SMALL_MODEL="${LLM_SMALL_MODEL:-/home/dank/models/Gemma4-12B-QAT/Gemma4-12B-QAT-Uncensored-HauhauCS-Balanced-Q4_K_M.gguf}"
+LLM_SECONDARY_MODEL_FILE="${LLM_SECONDARY_MODEL_FILE:-/home/dank/models/Qwen35-9B-HauhauCS-Q6K/Qwen3.5-9B-Uncensored-HauhauCS-Aggressive-Q6_K.gguf}"
+LLM_SECONDARY_RUNTIME="${LLM_SECONDARY_RUNTIME:-/home/dank/GitPackages/llama-master/build-cuda/bin/llama-server}"
+# The projector IS the vision capability here: the model is natively
+# multimodal and the Q6_K file carries no vision tensors — the publisher ships
+# its own BF16 projector in the same repo, so a deploy that starts without
+# this file silently ships a text-only endpoint. Checked unconditionally, like
+# frank-large's.
+LLM_SECONDARY_MMPROJ="${LLM_SECONDARY_MMPROJ:-/home/dank/models/Qwen35-9B-HauhauCS-Q6K/mmproj-Qwen3.5-9B-Uncensored-HauhauCS-Aggressive-BF16.gguf}"

-deploy_llm_small() {
-    if [[ ! -f "$LLM_SMALL_MODEL" ]]; then
-        echo "ERROR: model absent at $LLM_SMALL_MODEL" >&2
-        echo "frank-small: fetch HauhauCS/Gemma4-12B-QAT-Uncensored-HauhauCS-Balanced (Q4_K_M, the mmproj and the MTP sidecar) first;" >&2
-        echo "frank-small: refusing to start a unit that cannot load." >&2
+deploy_llm_secondary() {
+    # Refuse rather than restart-loop: a unit whose runtime or payload is
+    # missing never becomes healthy, and a green deploy would leave the
+    # endpoint dark with nothing in the log to say why.
+    if [[ ! -x "$LLM_SECONDARY_RUNTIME" ]]; then
+        echo "ERROR: llama.cpp runtime absent at $LLM_SECONDARY_RUNTIME" >&2
+        echo "frank-secondary: build it (cmake -B build-cuda -DGGML_CUDA=ON" >&2
+        echo "frank-secondary: -DCMAKE_CUDA_ARCHITECTURES=86 -DCMAKE_CUDA_HOST_COMPILER=/usr/bin/gcc-13" >&2
+        echo "frank-secondary: && cmake --build build-cuda --target llama-server) first." >&2
        return 1
    fi
-    relink_unit frank-small
+    for f in "$LLM_SECONDARY_MODEL_FILE" "$LLM_SECONDARY_MMPROJ"; do
+        if [[ ! -f "$f" ]]; then
+            echo "ERROR: frank-secondary payload file absent at $f" >&2
+            echo "frank-secondary: run scripts/download-frank-secondary.sh first;" >&2
+            echo "frank-secondary: refusing to start a unit that cannot load." >&2
+            return 1
+        fi
+    done
+    relink_unit frank-secondary
    run_privileged systemctl daemon-reload
-    run_privileged systemctl enable frank-small.service
-    run_privileged systemctl restart frank-small.service
-    wait_for_health frank-small http://127.0.0.1:7771/health 900
+    run_privileged systemctl enable frank-secondary.service
+    run_privileged systemctl restart frank-secondary.service
+    wait_for_health frank-secondary http://127.0.0.1:7771/health 900 || return 1
+    # False-green hardening (frank-large's lesson): the port answering
+    # proves nothing about WHO answers — the unit must be active with a
+    # live MainPID, or a stray SGLang holdover is masquerading as deployed.
+    local main_pid
+    main_pid="$(run_privileged systemctl show -p MainPID --value frank-secondary.service)"
+    if [[ -z "$main_pid" || "$main_pid" == "0" ]]; then
+        echo "FAIL: :7771 answered but frank-secondary.service has no MainPID — a stray process holds the port" >&2
+        return 1
+    fi
+    echo "frank-secondary: health answered by unit MainPID $main_pid"
 }

-# deploy_llm_primary — frank-primary is the tcclaviger RDNA4 vLLM fork
-# (digest-pinned image, gfx1201) serving the orcarouter
-# Qwen3.8-Flash-Next-Uncensored-FP8 payload (131 safetensors shards) with
-# MTP4 spec decode and q8/q8 KV on the two R9700s as a ROOT unit whose serve
-# script drives `podman run` directly (the image entrypoint execs
-# `vllm serve`). 2026-09-29 cutover from the R9V radiance fork, whose
-# launcher/shim/setup-state machinery is retired (state dirs stay on disk as
-# rollback; pre-cutover tag frank-primary-r9v-q8-20260929). Same port and
-# alias as the endpoints it replaced; the harness models.yml id is unchanged.
+# deploy_llm_primary — frank-primary is the radiance engine (StillDeadcode,
+# digest-pinned stilldeadcode/radiance image) serving the OFFICIAL
+# Qwen/Qwen3.8-Flash-Next-FP8 at native block-FP8 precision from the
+# rad-convert container qwen4exp-fp8-native.rad, on BOTH R9700s (TP=2) as a
+# ROOT unit whose serve script drives `podman run` directly (the engine is
+# the image entrypoint; the image lives in dank's rootless podman storage).
+# 2026-10-05 cutover from FreeToken (pre-cutover tag
+# frank-primary-freetoken-pr587-20261005); before that, the tcclaviger vLLM
+# fork (tag frank-primary-vllm-shapeG-20261003). Same port and alias; the
+# harness models.yml id is unchanged.
 #
-# Gates: the pinned runtime image must be loadable from dank's rootless
-# storage by its OCI manifest digest (the env file's FRANK_PRIMARY_IMAGE),
-# the payload dir must carry config.json and the expected shard count
-# (FRANK_PRIMARY_MODEL_SHARDS), and the unit env must be complete. Per-file
-# sha256 verification was the fetch script's job at download time, never
+# Gates: the image must exist under its digest ref in dank's rootless
+# storage (FRANK_PRIMARY_IMAGE), the .rad container file must exist at the
+# env's size floor (FRANK_PRIMARY_MODEL_FILE >= FRANK_PRIMARY_RAD_MIN_MIB —
+# a truncated convert is a guaranteed long-timeout boot), and the unit env
+# must be complete. Per-file sha256 verification is the fetch script's job
+# at download time and rad-convert/rad-info's job at build time, never
 # re-hashed here.
-LLM_PRIMARY_CONTAINER=frank-primary-qwen
-LLM_PRIMARY_LEGACY_CONTAINERS="frank-primary-r9v qwen-fp8 glm-probe"
+LLM_PRIMARY_CONTAINER=frank-primary
+LLM_PRIMARY_LEGACY_CONTAINERS="frank-primary-qwen frank-primary-r9v qwen-fp8 glm-probe"

 deploy_llm_primary() {
    # Refuse rather than restart-loop: a unit whose image or payload is
    # missing never becomes healthy, and a green deploy would leave the
    # endpoint dark with nothing in the log to say why.
    local env_file="/home/dank/serverAppConfigs/scripts/frank-primary.env"
-    local image model_dir shards count
+    local image model_dir model_file rad_min mib
    if [[ ! -f "$env_file" ]]; then
        echo "ERROR: frank-primary env file absent: $env_file" >&2
        return 1
    fi
    image="$(sed -n 's/^FRANK_PRIMARY_IMAGE=//p' "$env_file" | tail -1)"
    model_dir="$(sed -n 's/^FRANK_PRIMARY_MODEL_DIR=//p' "$env_file" | tail -1)"
-    shards="$(sed -n 's/^FRANK_PRIMARY_MODEL_SHARDS=//p' "$env_file" | tail -1)"
-    if [[ -z "$image" || -z "$model_dir" || -z "$shards" ]]; then
-        echo "ERROR: frank-primary.env missing FRANK_PRIMARY_IMAGE/MODEL_DIR/MODEL_SHARDS" >&2
+    model_file="$(sed -n 's/^FRANK_PRIMARY_MODEL_FILE=//p' "$env_file" | tail -1)"
+    rad_min="$(sed -n 's/^FRANK_PRIMARY_RAD_MIN_MIB=//p' "$env_file" | tail -1)"
+    if [[ -z "$image" || -z "$model_dir" || -z "$model_file" || -z "$rad_min" ]]; then
+        echo "ERROR: frank-primary.env missing FRANK_PRIMARY_IMAGE/MODEL_DIR/MODEL_FILE/RAD_MIN_MIB" >&2
        return 1
    fi
-    # Capture podman's output BEFORE grepping: under set -o pipefail, the
-    # classic `podman images | grep -q` races (grep exits on match, podman
-    # dies SIGPIPE 141 writing the rest, pipefail calls it no-match — the
-    # 2026-09-30 deploy refusal).
-    local digests
-    digests="$(podman images --format '{{.RepoDigests}}')"
-    if ! grep -qF "${image#*@}" <<<"$digests"; then
-        echo "ERROR: pinned runtime image absent from rootless storage: $image" >&2
-        echo "frank-primary: fetch it (podman pull $image)" >&2
+    # The image gate is the digest ref itself: a name@sha256 ref exists in
+    # storage only if THAT manifest was pulled. One command — no
+    # pipefail/grep race (the 2026-09-30 lesson).
+    if ! podman image exists "$image"; then
+        echo "ERROR: runtime image absent from storage: $image" >&2
+        echo "frank-primary: podman pull docker://stilldeadcode/radiance:latest" >&2
        return 1
    fi
-    if [[ ! -f "$model_dir/config.json" ]]; then
-        echo "ERROR: payload config missing: $model_dir/config.json" >&2
-        echo "frank-primary: re-run scripts/download-frank-primary.sh" >&2
+    if [[ ! -f "$model_dir/$model_file" ]]; then
+        echo "ERROR: .rad container missing: $model_dir/$model_file" >&2
+        echo "frank-primary: run scripts/build-frank-primary-rad.sh" >&2
        return 1
    fi
-    if [[ ! -f "$model_dir/model.safetensors.index.json" ]]; then
-        echo "ERROR: weight-map index missing: $model_dir/model.safetensors.index.json" >&2
-        echo "frank-primary: re-run scripts/download-frank-primary.sh" >&2
-        return 1
-    fi
-    count="$(find "$model_dir" -maxdepth 1 -name 'model-*-of-*.safetensors' 2>/dev/null | wc -l)"
-    if [[ "$count" != "$shards" ]]; then
-        echo "ERROR: payload shard count wrong: $model_dir (found $count, want $shards)" >&2
-        echo "frank-primary: re-run scripts/download-frank-primary.sh" >&2
+    mib=$(( $(stat -c %s "$model_dir/$model_file") / 1048576 ))
+    if [[ "$mib" -lt "$rad_min" ]]; then
+        echo "ERROR: .rad container too small: $model_dir/$model_file is ${mib} MiB, want >= $rad_min" >&2
+        echo "frank-primary: rebuild it (scripts/build-frank-primary-rad.sh --force)" >&2
        return 1
    fi

@@ -1286,9 +1356,26 @@ deploy_llm_primary() {
    run_privileged systemctl daemon-reload
    run_privileged systemctl enable frank-primary.service
    run_privileged systemctl restart frank-primary.service
-    # Cold start: FP8 page-in plus torch compile/graph capture (~15 min cold,
-    # measured 2026-09-29). Parity with every predecessor on this port.
-    wait_for_health frank-primary http://127.0.0.1:7770/health 1800 || return 1
+    # Cold start: .rad load + PLE pin + expert host-pool upload (the
+    # publisher's compose healthcheck allows a 15-minute start_period).
+    # Parity with every predecessor on this port. The HTTP listener binds
+    # AFTER the full load (main.cpp: engine.start() then engine.serve()),
+    # so during load the port REFUSES connections — a bare curl is honest
+    # here, unlike FreeToken's 200-while-loading body. (The MainPID check
+    # below still guards the stray-port case.)
+    local i ok=0
+    for ((i=1; i<=900; i++)); do
+        if curl -fsS http://127.0.0.1:7770/health 2>/dev/null >/dev/null; then
+            echo "frank-primary engine healthy after approximately $((i * 2))s"
+            ok=1
+            break
+        fi
+        sleep 2
+    done
+    if [[ "$ok" != 1 ]]; then
+        echo "FAIL: frank-primary /health never answered 200 in 1800s" >&2
+        return 1
+    fi
    # False-green hardening (the 2026-09-26 lesson): a bare curl against the
    # port cannot tell WHO answers — a stray probe holding :7770 satisfies
    # it while the unit is stopped. The deploy is only green if the unit
@@ -1487,7 +1574,7 @@ deploy_service() {
        seerr) deploy_seerr ;;
        marreta) deploy_marreta ;;
        frank-large) deploy_llm_large ;;
-        frank-small) deploy_llm_small ;;
+        frank-secondary) deploy_llm_secondary ;;
        frank-primary) deploy_llm_primary ;;
        hermes|hermes-dashboard|hermes-webui) deploy_hermes ;;
        romm|romm-mariadb|romm-redis|mariadb|redis) deploy_romm_stack "$service" ;;
@@ -141,7 +141,7 @@ valid_service() {
        dankclaw:alertmanager|\
        dankclaw:node-exporter|dankclaw:authentik-postgres|dankclaw:authentik-server|\
        dankclaw:authentik-worker|dankclaw:authentik-outpost|dankclaw:authentik-outpost-ldap|dankclaw:anubis|dankclaw:seerr|\
-        dankclaw:frank-large|dankclaw:frank-small|\
+        dankclaw:frank-large|dankclaw:frank-secondary|\
        dankserver:romm-mariadb|dankserver:romm-redis|dankserver:romm|\
        dankserver:copyparty|dankserver:jellyfin|dankserver:calibre-web|dankserver:calibre-server|dankserver:plex|dankserver:tautulli|\
        dankserver:qbittorrent|dankserver:radarr|dankserver:sonarr|\
@@ -196,6 +196,21 @@ else
    printf '    - %s\n' "${PLAN[@]}"
 fi

+# --- frankbook e2e trigger --------------------------------------------------
+# The *arr/qBittorrent e2e units walk links that terminate on FrankBook-Pro
+# (192.168.0.112), which sleeps when idle. The [3/3] e2e gate therefore runs
+# the frankbook Playwright project ONLY when something in its blast radius
+# changed since origin/main: the frankbook spec itself, the frankbook-pro
+# stack, the homepage registry, or the Caddyfile serving the arr vhosts.
+# Every other deploy skips it (tests/e2e/run.sh excludes the project by
+# default). See tests/e2e/specs/19-frankbook.spec.ts.
+FB_E2E_TRIGGER=0
+while IFS= read -r fb_path; do
+    case "$fb_path" in
+        tests/e2e/specs/19-frankbook.spec.ts|frankbook-pro/*|shared/homepage/services.json|dankclaw/Caddyfile) FB_E2E_TRIGGER=1 ;;
+    esac
+done < <(git diff --name-only "$BASE_REVISION" "$HEAD_REVISION")
+
 REMOTE="${REMOTE_USER}@${REMOTE_HOST}"
 REMOTE_ARGS=("$TARGET" "$HEAD_REVISION" "$REMOTE_DIR" "$REMOTE_SHELL")
 if [[ "${#EXPLICIT_SERVICES[@]}" -eq 0 ]]; then
@@ -209,6 +224,11 @@ if [[ "$DRY_RUN" -eq 1 ]]; then
    printf '[dry-run] Runner arguments:'
    printf ' %q' "${REMOTE_ARGS[@]}"
    printf '\n'
+    if [[ "$FB_E2E_TRIGGER" -eq 1 ]]; then
+        echo "[dry-run] e2e gate: frankbook project INCLUDED (frankbook trigger paths changed)."
+    else
+        echo "[dry-run] e2e gate: frankbook project EXCLUDED (no frankbook trigger paths)."
+    fi
    exit 0
 fi

@@ -250,7 +270,12 @@ if [[ "$SKIP_E2E" -eq 1 ]]; then
    echo "[3/3] e2e gate SKIPPED (--skip-e2e): live fleet NOT verified." >&2
 else
    echo "[3/3] Running Playwright e2e gate against the live fleet..."
-    if bash "$REPO_ROOT/tests/e2e/run.sh"; then
+    E2E_ENV=()
+    if [[ "$FB_E2E_TRIGGER" -eq 1 ]]; then
+        echo "      frankbook trigger paths changed: including the frankbook e2e project."
+        E2E_ENV=(E2E_INCLUDE_FRANKBOOK=1)
+    fi
+    if env "${E2E_ENV[@]}" bash "$REPO_ROOT/tests/e2e/run.sh"; then
        echo "e2e gate PASSED."
    else
        echo "=== DEPLOY INCOMPLETE: e2e gate FAILED on $TARGET ===" >&2
@@ -0,0 +1,117 @@
+#!/usr/bin/env python3
+"""derive-protected-experts.py — rewrite the recipe's protected-expert section
+from a calibration recording. RUNS ON DANKSERVER (venv: numpy).
+
+The law (docs/MOE-W4.md "Protected experts", restated in the recipe):
+an expert whose share of its layer's down-input energy is at least 3% AND
+at least three times its share of the routing is kept bf16; the count must
+be EVEN on each rank's half of the layer (the four-bit GEMM takes the rest
+as two equal tables by parity) — an odd half gains its next most energetic
+non-qualifying expert.
+
+Data: the calibration dir's per-expert down Grams,
+gram.r<rank>.blk.L.ffn_down_exps.<e>.bin — 32-byte RADGRAM2 header
+(magic, k, rank, rows, domain) then k*k f32, named with the checkpoint's
+GLOBAL expert numbering (arch/common/rad_block_moe_fp8.h: ge =
+expert_base + e). rows (header) is the expert's routing count; the trace
+of the Gram is its down-input energy. Rank halves: 0-255, 256-511
+(contiguous expert_base split at TP=2).
+
+Output: the cast-rule lines (recipe format) on stdout, a report on stderr.
+
+Usage: derive-protected-experts.py <calib dir> [--min-share 0.03]
+       [--min-route-mult 3.0]
+"""
+import argparse
+import glob
+import os
+import re
+import struct
+import sys
+
+import numpy as np
+
+NAME_RE = re.compile(r"^gram\.r(\d)\.blk\.(\d+)\.ffn_down_exps\.(\d+)\.bin$")
+
+
+def read_gram(path):
+    """(k, rank, rows, trace) from a RADGRAM2 file (full matrix on disk)."""
+    with open(path, "rb") as f:
+        hdr = f.read(32)
+        magic, k, rank, rows, domain = struct.unpack("<8sIIQI", hdr[:28])
+        if magic != b"RADGRAM2":
+            raise SystemExit(f"not a RADGRAM2 file: {path}")
+        n = k * k
+        a = np.frombuffer(f.read(n * 4), dtype=np.float32)
+        if a.size != n:
+            raise SystemExit(f"short gram file: {path}")
+        tr = float(a.reshape(k, k).diagonal().sum())
+    return k, rank, int(rows), tr
+
+
+def main():
+    ap = argparse.ArgumentParser()
+    ap.add_argument("calib_dir")
+    ap.add_argument("--min-share", type=float, default=0.03)
+    ap.add_argument("--min-route-mult", type=float, default=3.0)
+    a = ap.parse_args()
+
+    # layer -> global expert -> (rank, rows, energy)
+    data = {}
+    for path in sorted(glob.glob(os.path.join(
+            a.calib_dir, "gram.r*.ffn_down_exps.*.bin"))):
+        m = NAME_RE.match(os.path.basename(path))
+        if not m:
+            continue
+        rank, layer, expert = int(m.group(1)), int(m.group(2)), int(m.group(3))
+        k, _, rows, tr = read_gram(path)
+        data.setdefault(layer, {})[expert] = (rank, rows, tr)
+
+    if not data:
+        raise SystemExit("no per-expert down grams found")
+
+    lines, report = [], []
+    total = 0
+    for layer in sorted(data):
+        experts = data[layer]
+        tot_e = sum(v[2] for v in experts.values())
+        tot_r = sum(v[1] for v in experts.values())
+        if tot_e <= 0 or tot_r <= 0:
+            report.append(f"blk.{layer}: empty recording, skipped")
+            continue
+        qual, pool = [], []
+        for e, (rank, rows, tr) in experts.items():
+            es, rs = tr / tot_e, rows / max(tot_r, 1)
+            rec = (e, rank, es, rs)
+            if es >= a.min_share and es >= a.min_route_mult * rs:
+                qual.append(rec)
+            elif es > 0:
+                pool.append(rec)
+        chosen = {r[0]: (r[2], r[3]) for r in qual}
+        for half in (0, 1):  # rank halves by the contiguous expert_base split
+            n_half = sum(1 for e in chosen if e // 256 == half)
+            if n_half % 2:
+                pool.sort(key=lambda r: -r[2])
+                for cand in pool:
+                    if cand[1] == half and cand[0] not in chosen:
+                        chosen[cand[0]] = (cand[2], cand[3])
+                        report.append(
+                            f"blk.{layer}: half {half} odd -> added e{cand[0]} "
+                            f"(energy {cand[2]*100:.2f}%)")
+                        break
+        for e in sorted(chosen):
+            es, rs = chosen[e]
+            lines.append(f"blk.{layer}.ffn_*_exps.{e}.weight"
+                         f"         cast dtype=bf16")
+            report.append(f"blk.{layer} e{e:3d} (rank {e // 256}): "
+                          f"energy {es*100:5.2f}%, routing {rs*100:5.3f}%, "
+                          f"ratio {es/max(rs,1e-12):6.1f}")
+        total += len(chosen)
+
+    print("\n".join(lines))
+    print("\n".join(report), file=sys.stderr)
+    print(f"total protected: {total}", file=sys.stderr)
+
+
+if __name__ == "__main__":
+    main()
@@ -0,0 +1,119 @@
+#!/usr/bin/env bash
+# download-frank-large-flashnext.sh — fetch frank-large's flash-server v2
+# payload: the Quat3rnion abliterated Swift1.5 checkpoint of peonist's v2
+# .hgn, the REQUIRED lookup table and the optional vision tower from
+# peonist-ai/halogen-qwen3.8-flash-next (both at the revisions the model
+# card pins), and the flat tokenizer the two repos share. All public, no
+# token — and per the fleet secret law, none would ever touch argv.
+#
+# The fetch law, same as every sibling script: curl -f (an access error
+# must not become a "successful" 153-byte artifact), exact byte count from
+# the Hub tree API, LFS sha256 verification, --verify-only to re-check an
+# existing payload without downloading.
+#
+# File manifest (verified against the Hub tree API on 2026-10-06):
+#   qwen38-flash-next-v2-swift15-abliterated.hgn  66,687,678,464 B  sha256
+#     9c880b8ba48deabfa06172eff1e7572e4fe465e09e97faec6e50c3f6fdcac037
+#     (Quat3rnion main; compacted r2 of the card's 89,775,868,032 B
+#     append-only original — byte-identical live tensors, dropped padding)
+#   qwen38-flash-next-ngram.hgn  51,200,246,144 B  sha256
+#     3450acada94e19aabad88bd49b45eb70304820949c2240fc178067b20ae5dffe
+#     (peonist @5cc17cea; REQUIRED — the engine reads it beside the
+#     checkpoint)
+#   qwen38-flash-next-vision.hgn  897,916,416 B  sha256
+#     d62e0ae553fe88afd3833733d4a4c669f34d20fd8dfce4b9610525bed2134b10
+#     (peonist @e053f488; optional, enables image input)
+#   tokenizer/* @e053f488 — byte-identical to the 27B-era tokenizer
+#     already on disk (tokenizer.json LFS
+#     0997f410c57a1f4e53b09e4be8f4a172d90edd9564368fb0847030937229b9f3);
+#     fetched fresh rather than symlinked, per the no-symlink law.
+set -euo pipefail
+
+DEST="${FRANK_LARGE_MODEL_DIR:-$HOME/models/halogen-flashnext-v2}"
+ABLIT="https://huggingface.co/Quat3rnion/halogen-swift1.5-qwen3.8-flash-next-v2-abliterated/resolve/main"
+PEONIST="https://huggingface.co/peonist-ai/halogen-qwen3.8-flash-next/resolve"
+NGRAM_REV="5cc17cea1a10b502b4a5db41d8d2b1a0f22dba26"
+TOK_REV="e053f488b120b99ed2525e6ac99f68c51b3b6179"
+
+FILES=(qwen38-flash-next-v2-swift15-abliterated.hgn
+       qwen38-flash-next-ngram.hgn
+       qwen38-flash-next-vision.hgn
+       tokenizer/tokenizer.json
+       tokenizer/chat_template.jinja
+       tokenizer/generation_config.json
+       tokenizer/merges.txt
+       tokenizer/vocab.json
+       tokenizer/tokenizer_config.json)
+SIZES=(66687678464
+       51200246144
+       897916416
+       12809320
+       8952
+       202
+       3353259
+       6722759
+       17928)
+SHAS=(9c880b8ba48deabfa06172eff1e7572e4fe465e09e97faec6e50c3f6fdcac037
+      3450acada94e19aabad88bd49b45eb70304820949c2240fc178067b20ae5dffe
+      d62e0ae553fe88afd3833733d4a4c669f34d20fd8dfce4b9610525bed2134b10
+      0997f410c57a1f4e53b09e4be8f4a172d90edd9564368fb0847030937229b9f3
+      - - - - - -)
+
+# Fetch source per file: repo base + revision (empty = ABLIT @main).
+SRCS=("$ABLIT"
+      "$PEONIST/$NGRAM_REV"
+      "$PEONIST/$TOK_REV"
+      "$PEONIST/$TOK_REV"
+      "$PEONIST/$TOK_REV"
+      "$PEONIST/$TOK_REV"
+      "$PEONIST/$TOK_REV"
+      "$PEONIST/$TOK_REV"
+      "$PEONIST/$TOK_REV")
+
+VERIFY_ONLY=0
+[ "${1:-}" = "--verify-only" ] && VERIFY_ONLY=1
+
+mkdir -p "$DEST"
+
+for i in "${!FILES[@]}"; do
+    f="${FILES[$i]}"; want="${SIZES[$i]}"; sha="${SHAS[$i]}"; base="${SRCS[$i]}"
+    path="$DEST/$f"
+    mkdir -p "$(dirname "$path")"
+    if [ "$VERIFY_ONLY" -eq 0 ]; then
+        # A file already at the published size skips the fetch entirely:
+        # resuming it would ask the CDN for a range past EOF (416 -> curl
+        # error under set -e), and re-pulling 62 GiB to prove a point is
+        # not what --verify-only is for. sha256 below still decides.
+        if [ -f "$path" ] && [ "$(stat -c %s "$path")" = "$want" ]; then
+            echo "HAVE  $f ($want bytes) — skipping fetch"
+        else
+            echo "FETCH $f -> $path"
+            # HF answers /resolve/ with a 302 to the Xet CDN and curl DROPS
+            # the Range header across redirects — a naive -C - then
+            # "resumes" by pulling the ~1 KB redirect body (measured
+            # 2026-10-05 on the 27B fetch: file frozen 14 MiB short, curl
+            # exit 0). Resolve the effective URL first, then resume
+            # against the CDN, which honors Range.
+            url=$(curl -sfIL -o /dev/null -w '%{url_effective}' "$base/$f") \
+                || { echo "cannot resolve $base/$f" >&2; exit 1; }
+            curl -fL --retry 5 -C - -o "$path" "$url"
+        fi
+    fi
+    [ -f "$path" ] || { echo "MISSING $path (run without --verify-only)" >&2; exit 1; }
+    got="$(stat -c %s "$path")"
+    [ "$got" = "$want" ] || { echo "SIZE $path: $got != $want" >&2; exit 1; }
+    echo "OK   $f ($got bytes)"
+done
+
+echo "byte counts match; verifying sha256 where the Hub publishes an LFS oid..."
+for i in "${!FILES[@]}"; do
+    f="${FILES[$i]}"; sha="${SHAS[$i]}"
+    if [ "$sha" = "-" ]; then
+        echo "OK   $f (plain git blob, size-checked)"
+        continue
+    fi
+    got="$(sha256sum "$DEST/$f" | cut -d' ' -f1)"
+    [ "$got" = "$sha" ] || { echo "SHA256 $f: $got != $sha" >&2; exit 1; }
+    echo "OK   $f sha256"
+done
+echo "DOWNLOAD_DONE $DEST"
@@ -0,0 +1,76 @@
+#!/usr/bin/env bash
+# download-frank-large-halogen.sh — fetch frank-large's halogen payload:
+# the qwen3.8-27b .hgn checkpoint and the export tokenizer, from the
+# PUBLIC peonist-ai/halogen-qwen3.8-27b repo (no token — and per the fleet
+# secret law, none would ever touch argv).
+#
+# The fetch law, same as every sibling script: curl -f (an access error
+# must not become a "successful" 153-byte artifact), exact byte count from
+# the model card, LFS sha256 verification, --verify-only to re-check an
+# existing payload without downloading.
+set -euo pipefail
+
+DEST="${FRANK_LARGE_MODEL_DIR:-$HOME/models/halogen-qwen3.8-27b}"
+BASE="https://huggingface.co/peonist-ai/halogen-qwen3.8-27b/resolve/main"
+
+FILES=(qwen3.8-27b-p1w4d-d2.hgn
+       tokenizer/tokenizer.json
+       tokenizer/vocab.json
+       tokenizer/merges.txt
+       tokenizer/chat_template.jinja
+       tokenizer/tokenizer_config.json)
+# Byte sizes as published by the Hub on 2026-10-05, plus sha256 (the LFS
+# oid) where the Hub stores an LFS object — plain git blobs carry no oid
+# and are size-checked only. Re-check with:
+# curl -s "https://huggingface.co/api/models/peonist-ai/halogen-qwen3.8-27b/tree/main?recursive=true"
+SIZES=(35865565184 12809320 6722759 3353259 8952 17928)
+SHAS=(274c3fc767fb57faf025dd03c76376b1fc1b32448aef45aaf02d1898fa962089
+      0997f410c57a1f4e53b09e4be8f4a172d90edd9564368fb0847030937229b9f3 - - - -)
+
+VERIFY_ONLY=0
+[ "${1:-}" = "--verify-only" ] && VERIFY_ONLY=1
+
+mkdir -p "$DEST"
+
+for i in "${!FILES[@]}"; do
+    f="${FILES[$i]}"; want="${SIZES[$i]}"; sha="${SHAS[$i]}"
+    path="$DEST/$f"
+    mkdir -p "$(dirname "$path")"
+    if [ "$VERIFY_ONLY" -eq 0 ]; then
+        # A file already at the published size skips the fetch entirely:
+        # resuming it would ask the CDN for a range past EOF (416 -> curl
+        # error under set -e), and re-pulling 33.4 GiB to prove a point is
+        # not what --verify-only is for. sha256 below still decides.
+        if [ -f "$path" ] && [ "$(stat -c %s "$path")" = "$want" ]; then
+            echo "HAVE  $f ($want bytes) — skipping fetch"
+        else
+            echo "FETCH $f -> $path"
+            # HF answers /resolve/ with a 302 to the Xet CDN and curl DROPS
+            # the Range header across redirects — a naive -C - then
+            # "resumes" by pulling the ~1 KB redirect body (measured
+            # 2026-10-05: file frozen 14 MiB short, curl exit 0). Resolve
+            # the effective URL first, then resume against the CDN, which
+            # honors Range.
+            url=$(curl -sfIL -o /dev/null -w '%{url_effective}' "$BASE/$f") \
+                || { echo "cannot resolve $BASE/$f" >&2; exit 1; }
+            curl -fL --retry 5 -C - -o "$path" "$url"
+        fi
+    fi
+    [ -f "$path" ] || { echo "MISSING $path (run without --verify-only)" >&2; exit 1; }
+    got="$(stat -c %s "$path")"
+    [ "$got" = "$want" ] || { echo "SIZE $path: $got != $want" >&2; exit 1; }
+    echo "OK   $f ($got bytes)"
+done
+
+echo "byte counts match; verifying sha256 where the Hub publishes an LFS oid..."
+for i in "${!FILES[@]}"; do
+    f="${FILES[$i]}"; sha="${SHAS[$i]}"
+    if [ "$sha" = "-" ]; then
+        echo "OK   $f (plain git blob, size-checked)"
+        continue
+    fi
+    got="$(sha256sum "$DEST/$f" | cut -d' ' -f1)"
+    [ "$got" = "$sha" ] || { echo "SHA256 $f: $got != $sha" >&2; exit 1; }
+    echo "OK   $f sha256"
+done
+echo "DOWNLOAD_DONE $DEST"
@@ -0,0 +1,116 @@
+#!/usr/bin/env bash
+# Download kingjones777/Qwen3.8-27B-Uncensored-ROCmFP4-STRIX-MTP-GGUF for
+# frank-large (Dankclaw iGPU). Public repo — no auth header, and never a token
+# on the command line (it is world-readable in /proc/<pid>/cmdline).
+#
+# The payload is the repo's Q6 AGENT rung:
+#   * Qwen3.8-27B-Uncensored-Q6_0_ROCMFPX_AGENT.gguf — ftype 114, 23.15 GiB.
+#     Q6_0_ROCMFPX tensor type (102) + the AGENT routing, which keeps the
+#     output-head and attention tensors at plain Q8_0 — the variant the card
+#     names for MTP serving ("Take Q6 AGENT (114) if you want protected
+#     heads"). Like every ROCmFPX file it loads on nothing but the fork.
+#
+# The other two files frank-large needs are NOT re-downloaded:
+#   * mtp-Qwen3.8-27B-Q4_0.gguf — the uncensored repo redistributes the
+#     ALIGNED repo's draft head byte for byte (identical Hub LFS oid
+#     051a1764…), so it is hardlinked out of the aligned payload dir when that
+#     dir holds the verified bytes, and fetched from this repo otherwise.
+#   * mmproj-Qwen3.8-27B-BF16.gguf — this repo ships a Q8_0 projector; the
+#     fleet keeps the BF16 one, which is the projector that was verified 4/4
+#     on spatial ground truth on this host. Hardlinked the same way, fetched
+#     from the ALIGNED repo if it is not on disk.
+#
+# Fresh downloads, no resume (`curl -C -` once produced an oversized file on a
+# 31 GiB payload); -f/--fail-with-body so an HTTP 401 is not a success, then
+# byte count and the Hub LFS sha256. Idempotent: complete files are skipped.
+#
+# Usage:
+#   scripts/download-frank-large-kingjones-uncensored.sh [DEST_DIR] [--verify-only]
+set -euo pipefail
+
+REPO="kingjones777/Qwen3.8-27B-Uncensored-ROCmFP4-STRIX-MTP-GGUF"
+ALIGNED_REPO="kingjones777/Qwen3.8-27B-ROCmFP4-STRIX-MTP-GGUF"
+BASE="https://huggingface.co/${REPO}/resolve/main"
+ALIGNED_BASE="https://huggingface.co/${ALIGNED_REPO}/resolve/main"
+SOURCE_DIR="${SOURCE_DIR:-/home/dank/models/kingjones-Qwen3.8-27B-ROCmFP4-STRIX-MTP}"
+
+DEST=""
+VERIFY_ONLY=0
+for arg in "$@"; do
+    case "$arg" in
+        --verify-only) VERIFY_ONLY=1 ;;
+        *) DEST="$arg" ;;
+    esac
+done
+DEST="${DEST:-/home/dank/models/kingjones-Qwen3.8-27B-Uncensored-ROCmFP4-STRIX-MTP}"
+
+# name bytes sha256 base_url may_link — may_link=1 means the same bytes may
+# already exist in SOURCE_DIR (hardlink them instead of re-fetching).
+FILES=(
+  "Qwen3.8-27B-Uncensored-Q6_0_ROCMFPX_AGENT.gguf 24856607008 e966deb3d1a5a1b9ca7765126292f2d822f3dccdedb3b4872e673dec1cddc199 $BASE 0"
+  # The draft head is redistributed byte-identical by both repos; the BF16
+  # projector only exists in the aligned repo.
+  "mtp-Qwen3.8-27B-Q4_0.gguf 1680271648 051a1764cff8c4f3ee6ae8b00593a0364c7539c67fa50ffc58f3f96509fca38e $BASE 1"
+  "mmproj-Qwen3.8-27B-BF16.gguf 931145888 de2a49866988ea272c43dc2a43ee2662c25725a717ca5b9059ea420a97f53fe8 $ALIGNED_BASE 1"
+)
+
+matches() {  # matches FILE SIZE SHA — byte count AND LFS sha256
+    [[ -f "$1" ]] && [[ "$(stat -c%s "$1")" == "$2" ]] \
+        && sha256sum -c --quiet <(printf '%s  %s\n' "$3" "$1")
+}
+
+mkdir -p "$DEST"
+status=0
+
+fetch() {  # fetch NAME SIZE SHA BASE_URL
+    local name="$1" size="$2" sha="$3" url="$4" f="$DEST/$1"
+    echo "FETCHING  $name"
+    curl -fsSL --fail-with-body -o "$f.part" "$url/$name"
+    mv "$f.part" "$f"
+    if [[ "$(stat -c%s "$f")" != "$size" ]]; then
+        echo "SIZE FAIL $name ($(stat -c%s "$f") != $size)"
+        return 1
+    fi
+    if ! sha256sum -c --quiet <(printf '%s  %s\n' "$sha" "$f"); then
+        echo "SHA FAIL  $name"
+        return 1
+    fi
+    echo "VERIFIED  $name"
+}
+
+for entry in "${FILES[@]}"; do
+    read -r name size sha url may_link <<<"$entry"
+    f="$DEST/$name"
+    if matches "$f" "$size" "$sha"; then
+        echo "OK        $name"
+        continue
+    fi
+    if (( VERIFY_ONLY )); then
+        echo "MISSING   $name"
+        status=1
+        continue
+    fi
+    # Shared bytes: prefer the local copy (same filesystem, so a hardlink costs
+    # nothing) over re-downloading gigabytes that hash identically.
+    if [[ "$may_link" == 1 && -n "${SOURCE_DIR:-}" ]] && matches "$SOURCE_DIR/$name" "$size" "$sha"; then
+        echo "LINKING   $name from $SOURCE_DIR"
+        ln -f "$SOURCE_DIR/$name" "$f" 2>/dev/null || cp -f "$SOURCE_DIR/$name" "$f"
+        if matches "$f" "$size" "$sha"; then
+            echo "VERIFIED  $name"
+            continue
+        fi
+        echo "SHA FAIL  $name (linked copy)"
+        status=1
+        continue
+    fi
+    if ! fetch "$name" "$size" "$sha" "$url"; then
+        status=1
+    fi
+done
+
+if (( status == 0 )); then
+    echo "DOWNLOAD_DONE $DEST"
+else
+    echo "DOWNLOAD_FAILED $DEST" >&2
+fi
+exit "$status"
@@ -0,0 +1,91 @@
+#!/usr/bin/env bash
+# Download kingjones777/Qwen3.8-27B-ROCmFP4-STRIX-MTP-GGUF for frank-large
+# (Dankclaw iGPU). Public repo — no auth header, and never a token on the
+# command line (it is world-readable in /proc/<pid>/cmdline).
+#
+# The payload is the repo's highest-quality target plus its MTP draft head and
+# vision projector:
+#   * Qwen3.8-27B-Q8_0_ROCMFPX_AGENT.gguf — ftype 115, 187 tensors at plain
+#     Q8_0 and 311 at GGML_TYPE_Q8_0_ROCMFPX (103). The AGENT routing keeps the
+#     output-head tensors at full Q8_0 (187 vs 2 in the non-AGENT build), which
+#     is the variant to take when serving with the MTP head.
+#   * mtp-Qwen3.8-27B-Q4_0.gguf — the draft head. Q4_0, NOT Q8_0: the head's
+#     per-step cost is its projection over a 248K vocabulary, so the cheaper
+#     head is faster end to end at equal acceptance (author: +6.0% at n-max 1,
+#     +10.2% at n-max 2).
+#   * mmproj-Qwen3.8-27B-BF16.gguf — the vision tower (verified 4/4 on spatial
+#     ground truth with -fa on).
+#
+# Fresh downloads, no resume (`curl -C -` once produced an oversized file on a
+# 31 GiB payload); -f/--fail-with-body so an HTTP 401 is not a success, then
+# byte count and the Hub LFS sha256. Idempotent: complete files are skipped.
+#
+# The repo's own SHA256SUMS covers only the ROCmFP4 files, so the hashes below
+# come from the Hub LFS oids (api/models/<repo>/tree/main).
+#
+# Usage:
+#   scripts/download-frank-large-kingjones.sh [DEST_DIR] [--verify-only]
+set -euo pipefail
+
+REPO="kingjones777/Qwen3.8-27B-ROCmFP4-STRIX-MTP-GGUF"
+BASE="https://huggingface.co/${REPO}/resolve/main"
+
+DEST=""
+VERIFY_ONLY=0
+for arg in "$@"; do
+    case "$arg" in
+        --verify-only) VERIFY_ONLY=1 ;;
+        *) DEST="$arg" ;;
+    esac
+done
+DEST="${DEST:-/home/dank/models/kingjones-Qwen3.8-27B-ROCmFP4-STRIX-MTP}"
+
+# name bytes sha256
+FILES=(
+  "Qwen3.8-27B-Q8_0_ROCMFPX_AGENT.gguf 28214425952 f3ef2bdbf56e244fff14b72114dc2aa300b84a4cb4eaf54f2e5e4da1cfcae10b"
+  "mtp-Qwen3.8-27B-Q4_0.gguf 1680271648 051a1764cff8c4f3ee6ae8b00593a0364c7539c67fa50ffc58f3f96509fca38e"
+  "mmproj-Qwen3.8-27B-BF16.gguf 931145888 de2a49866988ea272c43dc2a43ee2662c25725a717ca5b9059ea420a97f53fe8"
+)
+
+mkdir -p "$DEST"
+status=0
+for entry in "${FILES[@]}"; do
+  read -r name size sha <<<"$entry"
+  f="$DEST/$name"
+  have=0
+  if [[ -f "$f" ]] && [[ "$(stat -c%s "$f")" == "$size" ]]; then
+    if sha256sum -c --quiet <(printf '%s  %s\n' "$sha" "$f"); then
+      have=1
+    fi
+  fi
+  if (( have )); then
+    echo "OK        $name"
+    continue
+  fi
+  if (( VERIFY_ONLY )); then
+    echo "MISSING   $name"
+    status=1
+    continue
+  fi
+  echo "FETCHING  $name"
+  tmp="$f.part"
+  curl -fsSL --fail-with-body -o "$tmp" "$BASE/$name"
+  mv "$tmp" "$f"
+  if [[ "$(stat -c%s "$f")" != "$size" ]]; then
+    echo "SIZE FAIL $name ($(stat -c%s "$f") != $size)"
+    status=1
+    continue
+  fi
+  if ! sha256sum -c --quiet <(printf '%s  %s\n' "$sha" "$f"); then
+    echo "SHA FAIL  $name"
+    status=1
+    continue
+  fi
+  echo "VERIFIED  $name"
+done
+if (( status == 0 )); then
+  echo "DOWNLOAD_DONE $DEST"
+else
+  echo "DOWNLOAD_FAILED $DEST" >&2
+fi
+exit "$status"
@@ -0,0 +1,76 @@
+#!/usr/bin/env bash
+# Download the Q4_K_XL frank-large payload on Dankclaw:
+#   unsloth/Qwen3.8-27B-GGUF  ->  Qwen3.8-27B-UD-Q4_K_XL.gguf
+#
+#   bash scripts/download-frank-large-q4.sh
+#   bash scripts/download-frank-large-q4.sh --verify-only
+#
+# Integrity: the file is checked against its declared byte count and sha256
+# (the LFS oid from the Hub tree API, 2026-10-01). NO AUTH HEADER IS USED:
+# the repo is public and ungated (gated: false), and a token on the curl
+# command line is world-readable via /proc/<pid>/cmdline. If this ever needs
+# a token, put it in a mode-600 file and pass it with `curl -K`.
+#
+# Layout after a successful run (MODEL_DIR):
+#   Qwen3.8-27B-UD-Q4_K_XL.gguf   17559178144  (16.35 GiB)
+#
+# WHY Q4 AND NOT THE Q8 SWIFT PAYLOAD: this is the vanilla (non-abliterated)
+# Qwen3.8-27B at UD-Q4_K_XL, ~13 GiB smaller than the Swift UD-Q8_K_XL. Every
+# GiB of weights saved is a GiB of f16 KV the endpoint can hold instead: the
+# KV costs 64 KiB/token (16 full-attention layers x 4 KV heads x 256 head_dim
+# x 2 (K,V) x 2 B), so 4 sessions x 163840 ctx = 640,000 tokens = 39.06 GiB.
+#
+# The speculative drafter is UNCHANGED and still fetched by its own script
+# (scripts/download-frank-large.sh, z-lab's Hub repo). That name must not
+# appear here at all — the frank-large seam greps THIS file for it to prove
+# the drafter was not folded into the payload fetch.
+#
+# NOTE ON MTP: this payload has no baked-in MTP head (the Hub repo ships one
+# separately as MTP/mtp-Qwen3.8-27B-Q4_0.gguf). frank-large does not need it —
+# spec decode here is the block-diffusion drafter, not MTP.
+set -euo pipefail
+
+MODEL_DIR="/home/dank/models/Qwen3.8-27B-UD-Q4_K_XL"
+REL="Qwen3.8-27B-UD-Q4_K_XL.gguf"
+SIZE=17559178144
+SHA=3f227079003add2511437e5b1e94812e363385225bf6a9b47b0054a72bc8b01e
+URL="https://huggingface.co/unsloth/Qwen3.8-27B-GGUF/resolve/main/${REL}"
+
+mode="download"
+[[ "${1:-}" == "--verify-only" ]] && mode="verify"
+
+if [[ "$mode" == "download" ]]; then
+    mkdir -p "$MODEL_DIR"
+    if [[ -s "$MODEL_DIR/$REL" ]]; then
+        echo "already present: $MODEL_DIR/$REL"
+    else
+        echo "fetching $REL (17559178144 bytes) — this is ~16.4 GiB"
+        # -f so an HTTP error is a failure, not a 153-byte "artifact".
+        # -C - so an interrupted run resumes instead of restarting.
+        curl -fL -C - --retry 5 --retry-delay 3 -o "$MODEL_DIR/$REL" "$URL"
+    fi
+fi
+
+echo "verifying $MODEL_DIR against declared size + sha256"
+python3 - "$MODEL_DIR" "$REL" "$SIZE" "$SHA" <<'PY'
+import hashlib, os, sys
+model_dir, rel, size, sha = sys.argv[1], sys.argv[2], int(sys.argv[3]), sys.argv[4]
+dest = os.path.join(model_dir, rel)
+if not os.path.isfile(dest):
+    print(f"  MISSING {rel}")
+    sys.exit(1)
+actual = os.path.getsize(dest)
+if actual != size:
+    print(f"  SIZE MISMATCH {rel}: {actual} != {size}")
+    sys.exit(1)
+h = hashlib.sha256()
+with open(dest, "rb") as f:
+    for chunk in iter(lambda: f.read(1 << 24), b""):
+        h.update(chunk)
+if h.hexdigest() != sha:
+    print(f"  SHA256 MISMATCH {rel}: {h.hexdigest()}")
+    sys.exit(1)
+print(f"  OK {rel} ({actual} bytes, sha256 matches the Hub LFS oid)")
+PY
+
+echo "Q4_K_XL payload complete: $MODEL_DIR"
@@ -0,0 +1,113 @@
+#!/usr/bin/env bash
+# download-frank-primary-swift15.sh — payload fetch/verify for the planned
+# frank-primary swap to the Swift 1.5 finetune: ukisai/Swift1.5-Qwen3.8-Flash-Next
+# (BF16 source, 149 files, 360,037,355,038 bytes = 335.3 GiB: 131 safetensors
+# shards + weight-map index + tokenizer + vision preprocessing configs + chat
+# template + licenses). Key set verified identical to the official
+# Qwen/Qwen3.8-Flash-Next base (0 added / 0 missing), so the qwen4exp recipes
+# apply verbatim; the checkpoint ships BF16 with NO quantization_config.
+# Repo is UNGATED — no token needed (never put one on the command line).
+#
+#   bash scripts/download-frank-primary-swift15.sh --fetch         resume/complete
+#   bash scripts/download-frank-primary-swift15.sh                 verify only
+#
+# Fetch uses the vendored scripts/hfget.sh (StillDeadcode's Apache-2.0
+# listing/resume downloader, curl+jq only): per-file size checks against
+# the Hub listing, `-C -` resume, pickle-format duplicates skipped.
+# Verify (default) is the AGENTS.md law in three parts: every listed file
+# present with the listed size, the shard count matches the weight-map
+# index, and every shard's sha256 equals the Hub's LFS oid. Verify needs
+# only the public API.
+set -euo pipefail
+
+REPO="ukisai/Swift1.5-Qwen3.8-Flash-Next"
+DEST="/home/dank/models/ukisai/Swift1.5-Qwen3.8-Flash-Next"
+EXPECTED_SHARDS=131
+
+mode="verify"
+[[ "${1:-}" == "--fetch" ]] && mode="fetch"
+SCRIPT_DIR="$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)"
+
+if [[ "$mode" == "fetch" ]]; then
+    # HFGET_JOBS=4 is hfget's default (a few streams saturate the link).
+    bash "$SCRIPT_DIR/hfget.sh" "$REPO" "$DEST" main
+fi
+
+echo "verifying local payload: $DEST"
+fail=0
+
+INDEX="$DEST/model.safetensors.index.json"
+if [[ ! -f "$INDEX" ]]; then
+    echo "  BAD  weight-map index missing: $INDEX" >&2
+    exit 1
+fi
+
+python3 - "$REPO" "$DEST" <<'PY' || fail=1
+import hashlib, json, os, sys, urllib.request
+
+repo, dest = sys.argv[1], sys.argv[2]
+
+def listing():
+    """The tree API, following the Link cursor exactly like hfget.sh."""
+    out, api, pages = [], (
+        f"https://huggingface.co/api/models/{repo}/tree/main"
+        "?recursive=true&expand=true"), 0
+    while api:
+        with urllib.request.urlopen(api) as r:
+            out += json.load(r)
+            api = None
+            link = r.headers.get("Link", "")
+            for part in link.split(","):
+                if 'rel="next"' in part:
+                    api = part[part.find("<") + 1:part.find(">")]
+        pages += 1
+        if pages > 200:
+            raise SystemExit("listing did not terminate")
+    return out
+
+files = [e for e in listing() if e.get("type") == "file"]
+sizes = {e["path"]: e.get("size", 0) for e in files}
+lfs = {e["path"]: e["lfs"]["oid"] for e in files if e.get("lfs")}
+
+bad = 0
+for path, want in sorted(sizes.items()):
+    p = os.path.join(dest, path)
+    if not os.path.isfile(p):
+        print(f"  BAD  {path}: missing")
+        bad += 1
+    elif os.path.getsize(p) != want:
+        print(f"  BAD  {path}: {os.path.getsize(p)} bytes, listing says {want}")
+        bad += 1
+print(f"  {'OK' if bad == 0 else 'FAIL'}: {len(sizes) - bad}/{len(sizes)} files present at listed size")
+if bad:
+    raise SystemExit(1)
+
+# Shard set against the weight-map index.
+wm = json.load(open(os.path.join(dest, "model.safetensors.index.json")))["weight_map"]
+shards = sorted(set(wm.values()))
+print(f"  index maps {len(wm)} tensors across {len(shards)} shards")
+if len(shards) != 131:
+    print(f"  BAD  shard count {len(shards)}, want 131")
+    raise SystemExit(1)
+
+# Content: every LFS-tracked file hashed against the Hub oid. The safetensors
+# shards dominate the bytes; tokenizer/config files ride the same check when
+# the Hub tracks them in LFS (they are plain blobs here, already size-checked).
+todo = [p for p in shards if p in lfs]
+for i, path in enumerate(todo):
+    h = hashlib.sha256()
+    with open(os.path.join(dest, path), "rb") as f:
+        for chunk in iter(lambda: f.read(1 << 24), b""):
+            h.update(chunk)
+    if h.hexdigest() != lfs[path]:
+        print(f"  BAD  {path}: sha256 mismatch (want {lfs[path]})")
+        bad += 1
+    if (i + 1) % 16 == 0:
+        print(f"  ...hashed {i + 1}/{len(todo)}")
+print(f"  {'OK' if bad == 0 else 'FAIL'}: {len(todo) - bad}/{len(todo)} shards sha256-match the Hub")
+if bad:
+    raise SystemExit(1)
+PY
+
+[[ "$fail" -eq 0 ]] || exit 1
+echo "payload verified: 131 shards + index + tokenizer + configs, sizes and sha256 checked"
@@ -1,88 +1,111 @@
 #!/usr/bin/env bash
-# Payload inventory/verification for frank-primary:
-# orcarouter/Qwen3.8-Flash-Next-Uncensored-FP8 (compressed-tensors FP8, 131
-# shards + weight-map index, ~172.8 GiB) — the tcclaviger-stack payload that
-# became frank-primary on 2026-09-29.
+# download-frank-primary.sh — payload fetch/verify for frank-primary:
+# the OFFICIAL Qwen/Qwen3.8-Flash-Next-FP8 checkpoint (131 safetensors
+# shards + weight-map index + tokenizer + vision preprocessing configs,
+# ~172.8 GB). 2026-10-05: replaces the orcarouter uncensored payload of the
+# FreeToken era (that script lived here; its repo was gated and needed
+# HF_TOKEN — this one is UNGATED and needs no token at all).
 #
-#   bash scripts/download-frank-primary.sh --verify-only   (the ONLY mode)
-#   bash scripts/download-frank-primary.sh --fetch
+#   bash scripts/download-frank-primary.sh --fetch         resume/complete
+#   bash scripts/download-frank-primary.sh                 verify only
 #
-# THIS REPO IS GATED (401 anonymous — measured 2026-09-29): anonymous fetch
-# is impossible and the tree API masks LFS hashes, so a byte-exact upstream
-# manifest cannot be built without the authed endpoint. The payload on
-# /home/dank/models was fetched with the operator's HF token (out of band:
-# HF_TOKEN in the environment, never in argv) at bring-up time.
-#
-# Modes:
-#   --fetch       resume any missing/truncated shard via `hf download` with
-#                 HF_TOKEN from the environment (NEVER passed on the command
-#                 line — a -H auth header is world-readable in
-#                 /proc/<pid>/cmdline, the 2026-09-16 leak).
-#   --verify-only default: local integrity check — shard count vs the
-#                 weight-map index, every file nonzero and its size equal to
-#                 the index's declared total, plus sha256 of every shard
-#                 (slow, ~10 min for 173 GiB).
+# Fetch uses the vendored scripts/hfget.sh (StillDeadcode's Apache-2.0
+# listing/resume downloader, curl+jq only): per-file size checks against
+# the Hub listing, `-C -` resume, pickle-format duplicates skipped.
+# Verify (default) is the AGENTS.md law in three parts: every listed file
+# present with the listed size, the shard count matches the weight-map
+# index, and every shard's sha256 equals the Hub's LFS oid. Verify needs
+# only the public API.
 set -euo pipefail

-REPO="orcarouter/Qwen3.8-Flash-Next-Uncensored-FP8"
-DEST="/home/dank/models/Qwen3.8-Flash-Next-Uncensored-FP8"
+REPO="Qwen/Qwen3.8-Flash-Next-FP8"
+DEST="/home/dank/models/Qwen/Qwen3.8-Flash-Next-FP8"
 EXPECTED_SHARDS=131

 mode="verify"
 [[ "${1:-}" == "--fetch" ]] && mode="fetch"
+SCRIPT_DIR="$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)"

 if [[ "$mode" == "fetch" ]]; then
-    : "${HF_TOKEN:?--fetch needs HF_TOKEN in the environment (never argv)}"
-    export HF_TOKEN
-    mkdir -p "$DEST"
-    hf download "$REPO" --local-dir "$DEST"
+    # HFGET_JOBS=4 is hfget's default (a few streams saturate the link).
+    bash "$SCRIPT_DIR/hfget.sh" "$REPO" "$DEST" main
 fi

 echo "verifying local payload: $DEST"
 fail=0

-# The weight-map index is the source of truth for the shard set.
 INDEX="$DEST/model.safetensors.index.json"
 if [[ ! -f "$INDEX" ]]; then
    echo "  BAD  weight-map index missing: $INDEX" >&2
    exit 1
 fi

-# Every index entry must exist, nonzero, and sized equal to the map's total.
-python3 - "$INDEX" "$DEST" <<'PY' || fail=1
-import json, os, sys
-index, dest = sys.argv[1], sys.argv[2]
-wm = json.load(open(index))["weight_map"]
-files = sorted(set(wm.values()))
-print(f"  index maps {len(wm)} tensors across {len(files)} files")
+python3 - "$REPO" "$DEST" <<'PY' || fail=1
+import hashlib, json, os, sys, urllib.request
+
+repo, dest = sys.argv[1], sys.argv[2]
+
+def listing():
+    """The tree API, following the Link cursor exactly like hfget.sh."""
+    out, api, pages = [], (
+        f"https://huggingface.co/api/models/{repo}/tree/main"
+        "?recursive=true&expand=true"), 0
+    while api:
+        with urllib.request.urlopen(api) as r:
+            out += json.load(r)
+            api = None
+            link = r.headers.get("Link", "")
+            for part in link.split(","):
+                if 'rel="next"' in part:
+                    api = part[part.find("<") + 1:part.find(">")]
+        pages += 1
+        if pages > 200:
+            raise SystemExit("listing did not terminate")
+    return out
+
+files = [e for e in listing() if e.get("type") == "file"]
+sizes = {e["path"]: e.get("size", 0) for e in files}
+lfs = {e["path"]: e["lfs"]["oid"] for e in files if e.get("lfs")}
+
 bad = 0
-total = 0
-for f in files:
-    p = os.path.join(dest, f)
-    if not os.path.isfile(p) or os.path.getsize(p) == 0:
-        print(f"  BAD  {f}: missing or zero bytes" )
+for path, want in sorted(sizes.items()):
+    p = os.path.join(dest, path)
+    if not os.path.isfile(p):
+        print(f"  BAD  {path}: missing")
        bad += 1
-    else:
-        total += os.path.getsize(p)
-print(f"  {'OK' if bad == 0 else 'FAIL'}: {len(files) - bad}/{len(files)} files present, {total/2**30:.1f} GiB total")
-raise SystemExit(1 if bad else 0)
+    elif os.path.getsize(p) != want:
+        print(f"  BAD  {path}: {os.path.getsize(p)} bytes, listing says {want}")
+        bad += 1
+print(f"  {'OK' if bad == 0 else 'FAIL'}: {len(sizes) - bad}/{len(sizes)} files present at listed size")
+if bad:
+    raise SystemExit(1)
+
+# Shard set against the weight-map index.
+wm = json.load(open(os.path.join(dest, "model.safetensors.index.json")))["weight_map"]
+shards = sorted(set(wm.values()))
+print(f"  index maps {len(wm)} tensors across {len(shards)} shards")
+if len(shards) != 131:
+    print(f"  BAD  shard count {len(shards)}, want 131")
+    raise SystemExit(1)
+
+# Content: every LFS-tracked file hashed against the Hub oid. The safetensors
+# shards dominate the bytes; tokenizer/config files ride the same check when
+# the Hub tracks them in LFS (they are plain blobs here, already size-checked).
+todo = [p for p in shards if p in lfs]
+for i, path in enumerate(todo):
+    h = hashlib.sha256()
+    with open(os.path.join(dest, path), "rb") as f:
+        for chunk in iter(lambda: f.read(1 << 24), b""):
+            h.update(chunk)
+    if h.hexdigest() != lfs[path]:
+        print(f"  BAD  {path}: sha256 mismatch (want {lfs[path]})")
+        bad += 1
+    if (i + 1) % 16 == 0:
+        print(f"  ...hashed {i + 1}/{len(todo)}")
+print(f"  {'OK' if bad == 0 else 'FAIL'}: {len(todo) - bad}/{len(todo)} shards sha256-match the Hub")
+if bad:
+    raise SystemExit(1)
 PY

-# Shard count gate (what the serve script and deployer check).
-n="$(find "$DEST" -maxdepth 1 -name 'model-*-of-*.safetensors' | wc -l)"
-if [[ "$n" != "$EXPECTED_SHARDS" ]]; then
-    echo "  BAD  shard count: found $n, want $EXPECTED_SHARDS" >&2
-    fail=1
-else
-    echo "  OK   shard count $n"
-fi
-
-if [[ "${2:-}" == "--hash" || "${VERIFY_HASH:-0}" == "1" ]]; then
-    echo "hashing every shard (slow)..."
-    cd "$DEST"
-    sha256sum -c --quiet model.safetensors.index.json 2>/dev/null || \
-    sha256sum model-*-of-00131.safetensors > /dev/null  # presence check; content hash list printed on demand
-fi
-
 [[ "$fail" -eq 0 ]] || exit 1
-echo "payload inventory OK: $EXPECTED_SHARDS shards + index present"
+echo "payload verified: 131 shards + index + tokenizer + configs, sizes and sha256 checked"
@@ -0,0 +1,90 @@
+#!/usr/bin/env bash
+# Download the frank-secondary payload for llama.cpp on the RTX 3090:
+# HauhauCS/Qwen3.5-9B-Uncensored-HauhauCS-Aggressive at **Q6_K** plus the
+# PUBLISHER's OWN vision projector (the BF16 mmproj — the model is natively
+# multimodal, so this file is the endpoint's eyes). Public repo — no auth
+# header, and never a token on the command line (it is world-readable in
+# /proc/<pid>/cmdline).
+#
+# Fresh downloads, no resume (`curl -C -` once produced a 213 MB oversized
+# file on a 31 GiB payload); -f/--fail-with-body so an HTTP error is not a
+# success, then byte count and sha256. The Hub exposes no LFS oid for this
+# repo (Xet storage), so the sha256 values below were computed locally after
+# the first verified fetch — byte counts match the Hub's declared sizes.
+# A sha of "-" means "first fetch: size-verified, sha printed for pinning".
+# Idempotent: complete files are skipped.
+#
+# 2026-10-07 (Frank): the payload moved to Qwen3.5-9B-Uncensored Q6_K at
+# 2 x 200,704 (qwen35 hybrid: 32 layers, 8 full-attention + 24 Gated
+# DeltaNet, no MTP tensors in this file). Rollback payloads stay on disk:
+# /home/dank/models/Gemma4-26B-A4B/ (Q4_K_M, the 2026-10-07 era being
+# replaced), Swift15-UncIQ3XXS/, MiMo-Q6K/, MiMo-Q5KM/, Ornith-Q8/.
+#
+# Usage:
+#   scripts/download-frank-secondary.sh [DEST_DIR] [--verify-only]
+set -euo pipefail
+
+DEST=""
+VERIFY_ONLY=0
+for arg in "$@"; do
+    case "$arg" in
+        --verify-only) VERIFY_ONLY=1 ;;
+        *) DEST="$arg" ;;
+    esac
+done
+DEST="${DEST:-/home/dank/models/Qwen35-9B-HauhauCS-Q6K}"
+
+# repo | hub-name | dest-name | bytes | sha256 (computed locally; Xet repo)
+FILES=(
+  "HauhauCS/Qwen3.5-9B-Uncensored-HauhauCS-Aggressive|Qwen3.5-9B-Uncensored-HauhauCS-Aggressive-Q6_K.gguf|Qwen3.5-9B-Uncensored-HauhauCS-Aggressive-Q6_K.gguf|7359259008|c0ba7beb68fd3fe47891bd549486d38dcf62d00817296ea314ad37017f5a4986"
+  "HauhauCS/Qwen3.5-9B-Uncensored-HauhauCS-Aggressive|mmproj-Qwen3.5-9B-Uncensored-HauhauCS-Aggressive-BF16.gguf|mmproj-Qwen3.5-9B-Uncensored-HauhauCS-Aggressive-BF16.gguf|921704448|05f662501f8bd45607b079723a3e238a4e888fd085a10a53f4057a0e250f6934"
+)
+
+status=0
+for entry in "${FILES[@]}"; do
+  IFS='|' read -r repo hubname destname size sha <<<"$entry"
+  url="https://huggingface.co/${repo}/resolve/main/${hubname}"
+  f="$DEST/$destname"
+  have=0
+  if [[ -f "$f" ]] && [[ "$(stat -c%s "$f")" == "$size" ]]; then
+    if [[ "$sha" == "-" ]]; then
+      have=1
+    elif sha256sum -c --quiet <(printf '%s  %s\n' "$sha" "$f"); then
+      have=1
+    fi
+  fi
+  if (( have )); then
+    echo "OK        $destname"
+    continue
+  fi
+  if (( VERIFY_ONLY )); then
+    echo "MISSING   $destname"
+    status=1
+    continue
+  fi
+  echo "FETCHING  $destname (from $repo)"
+  tmp="$f.part"
+  curl -fsSL --fail-with-body -o "$tmp" "$url"
+  mv "$tmp" "$f"
+  if [[ "$(stat -c%s "$f")" != "$size" ]]; then
+    echo "SIZE FAIL $destname ($(stat -c%s "$f") != $size)"
+    status=1
+    continue
+  fi
+  if [[ "$sha" == "-" ]]; then
+    echo "VERIFIED  $destname (size only) sha256=$(sha256sum "$f" | cut -d' ' -f1)"
+    continue
+  fi
+  if ! sha256sum -c --quiet <(printf '%s  %s\n' "$sha" "$f"); then
+    echo "SHA FAIL  $destname"
+    status=1
+    continue
+  fi
+  echo "VERIFIED  $destname"
+done
+if (( status == 0 )); then
+  echo "DOWNLOAD_DONE $DEST"
+else
+  echo "DOWNLOAD_FAILED $DEST" >&2
+fi
+exit "$status"
@@ -0,0 +1,56 @@
+#!/usr/bin/env bash
+# fleet-probes-swift15.sh — RUNS ON DANKSERVER. The cutover contract probes.
+set -uo pipefail
+B=http://localhost:7770/v1
+
+echo "== vision (solid red 512px, expect 'Red')"
+python3 - <<'PY'
+import base64, json, urllib.request, struct, zlib
+def png(color):
+    w = h = 512
+    raw = b"".join(b"\x00" + bytes(color) * w for _ in range(h))
+    def chunk(t, d):
+        c = t + d
+        return struct.pack(">I", len(d)) + c + struct.pack(">I", zlib.crc32(c))
+    return (b"\x89PNG\r\n\x1a\n"
+            + chunk(b"IHDR", struct.pack(">IIBBBBB", w, h, 8, 2, 0, 0, 0))
+            + chunk(b"IDAT", zlib.compress(raw)) + chunk(b"IEND", b""))
+b64 = base64.b64encode(png((255, 0, 0))).decode()
+body = json.dumps({"model": "frank-archon", "max_tokens": 256, "temperature": 0,
+    "messages": [{"role": "user", "content": [
+        {"type": "image_url", "image_url": {"url": "data:image/png;base64," + b64}},
+        {"type": "text", "text": "What color is this image? Answer with one word."}]}]}).encode()
+r = urllib.request.Request("http://localhost:7770/v1" + "/chat/completions",
+                           data=body, headers={"Content-Type": "application/json"})
+d = json.load(urllib.request.urlopen(r, timeout=300))
+m = d["choices"][0]["message"]
+ans = (m.get("content") or m.get("reasoning_content") or "")
+print("VISION ANSWER:", ans[:80].replace("\n", " "))
+PY
+
+echo "== tools (expect tool_calls)"
+curl -s -m 120 $B/chat/completions -H 'Content-Type: application/json' -d '{
+ "model":"frank-archon","temperature":0,"max_tokens":128,
+ "messages":[{"role":"user","content":"What is 23 times 47? Use the calculator tool."}],
+ "tools":[{"type":"function","function":{"name":"calculator","description":"evaluate arithmetic",
+ "parameters":{"type":"object","properties":{"expr":{"type":"string"}},"required":["expr"]}}}]}' \
+ | python3 -c 'import json,sys; d=json.load(sys.stdin); m=d["choices"][0]["message"]; print("TOOL_CALLS:", m.get("tool_calls"))'
+
+echo "== reasoning_effort fold (xhigh/medium/low/high all 200 on froggeric v22.5)"
+for lvl in xhigh medium low high; do
+  code=$(curl -s -o /dev/null -w "%{http_code}" -m 120 $B/chat/completions \
+    -H 'Content-Type: application/json' \
+    -d "{\"model\":\"frank-archon\",\"max_tokens\":8,\"reasoning_effort\":\"$lvl\",\"messages\":[{\"role\":\"user\",\"content\":\"hi\"}]}")
+  echo "  reasoning_effort=$lvl -> HTTP $code"
+done
+
+echo "== reasoning_content channel"
+curl -s -m 120 $B/chat/completions -H 'Content-Type: application/json' \
+ -d '{"model":"frank-archon","max_tokens":64,"messages":[{"role":"user","content":"Is 97 prime? Think briefly."}]}' \
+ | python3 -c 'import json,sys; m=json.load(sys.stdin)["choices"][0]["message"]; print("has reasoning_content:", bool(m.get("reasoning_content")), "| content:", (m.get("content") or "")[:60].replace(chr(10)," "))'
+
+echo "== response_format json_object"
+curl -s -m 120 $B/chat/completions -H 'Content-Type: application/json' \
+ -d '{"model":"frank-archon","max_tokens":64,"response_format":{"type":"json_object"},"messages":[{"role":"user","content":"Return JSON {\"ok\": true}"}]}' \
+ | python3 -c 'import json,sys; d=json.load(sys.stdin); print("JSON MODE:", (d["choices"][0]["message"].get("content") or "")[:60])'
+echo "PROBES_DONE"
@@ -1,45 +1,158 @@
-# frank-large — gufo (gufo-org/gufo) serving
-# Swift-1.5-Qwen3.8-27B-Uncensored (abliterated) on the Strix Halo iGPU.
+# frank-large.env — tunables for the frank-large endpoint on Dankclaw's
+# Strix Halo iGPU (gfx1151), host port 7770.
 #
-# These are the tunables read by scripts/serve-frank-large.sh. The flag
-# list is owned by the serve script; this file only sets fleet values.
+# RUNTIME (2026-10-06, Frank): halogen-flash-server 0.16.2 — the
+# closed-source C++/HIP engine (image ghcr.io/peonist-ai/halogen-flash-server,
+# digest-pinned below) serving the Quat3rnion ABLITERATED Swift1.5 finetune
+# of peonist's Qwen3.8-Flash-Next v2 checkpoint, as a podman container under
+# a root unit, driven by scripts/serve-frank-large.sh. This CUTS OVER the
+# 2026-10-05 halogen 27B-dense shape (censored base, DFlash2, 31.7 t/s):
+# rollback = revert the swap commit's trio; the 27B payload stays on disk at
+# /home/dank/models/halogen-qwen3.8-27b/.
 #
-# DO NOT name a new tunable TEMP, HOST or GPU_MAX_HEAP_SIZE: the HIP
-# runtime reads those bare (the 2026-08-13 TEMP incident). HOST is the one
-# exception because the serve script itself consumes it for --host.
+# WHY THE RUNTIME MOVED WITH THE MODEL: the payload is .hgn — per its own
+# card it "will not load in transformers, vLLM, or llama.cpp" — and the v2
+# .hgn loads ONLY in the flash-server line (the old halogen engine is for
+# the 27B p1w4d file; the v2 quickstart pins flash-server). What the move
+# buys (card + live probe evidence): MTP spec decode (53-57 t/s greedy,
+# 3-10x serial), ~1,640 t/s prefill at 34k, ~72% fewer thinking tokens than
+# the censored 27B (905 mean), the ABLITERATED base (refusals should NOT
+# return), IMAGE INPUT (the v2 vision tower loads beside the checkpoint —
+# red-PNG probe verified on the card; the 27B engine ignored its encoder),
+# tool calls verified, and the native 262,144 context.
+#
+# SHAPE — the knobs that are NOT free, with the arithmetic:
+#   HALOGEN_CTX=262144 is the PER-SLOT window and the per-conversation hard
+#   ceiling — a prompt over it is a 400 naming the limit, never a silent
+#   truncation. RAISED from 200192 on 2026-10-07: the harness clamps
+#   max_tokens to contextWindow − est − 4096, its chars/4 est UNDER-counts
+#   the rendered prompt by ~3 % on code/agent traffic, and compaction only
+#   fires at 85 % — so with window == engine ceiling, every request over
+#   ~146k est (65-85 % of the window) was a guaranteed 400. Measured
+#   2026-10-07: est 152,938, server 157,388, request 43,158 vs room 42,804
+#   — over by 354, margin 4,096 < the 4,450 under-count.
+#   Harness contract (models.yml): contextWindow 217088, maxTokens 45056.
+#   Compaction fires at 85 % = 184,524 est ≈ 190k measured (Frank's 180k
+#   target). Worst-case request: prompt + budget <= 217088 − 4096 + 3 %
+#   est-error ≈ 219k <= 262144 — the 45k gap between harness window and
+#   engine ceiling absorbs the estimate error with ~7x margin (robust to a
+#   10 % under-count). Compaction rides on frank-secondary: input ≈
+#   threshold − keepRecent(20k) ≈ 170k measured, inside its 200,704 slot.
+#   HALOGEN_KV_POOL_POSITIONS=524288 = KV_SLOTS(2) x CTX — the explicit
+#   form of the engine's default, so the seam test can pin the product.
+#   KV costs ~28 KiB/position (FLAGS.md) => 524,288 x 28 KiB = 14.0 GiB.
+#   HALOGEN_KV_SLOTS=2 restores the w4b-era shape on this box: the MTP
+#   drafter is NOT slot-locked on this engine (batching and speculation
+#   coexist — DFlash-era slots>1 law does NOT carry over; the card's own
+#   quickstart ships 2 slots WITH the head), so the second slot is
+#   concurrent capacity, not a speculation tax.
+#   Memory ledger against the box's 121.2 GiB unified (the n-gram table is
+#   mmap'd page cache beside the checkpoint, not a pinned charge — the
+#   card's own quickstart arithmetic does not bill it):
+#     weights ~62 GiB resident + KV 14.0 + arena (HALOGEN_MAX_TOK=8192 =>
+#     ~8 GiB per FLAGS.md arena formula 3.8+8.4*(MT/16384)) + vision ~2
+#     = ~86 GiB engine-side; HALOGEN_HOST_RESERVE_GIB=20 keeps the host
+#     honest; baseline co-tenancy (frank-secondary, ComfyUI, Caddy/observ)
+#     ~10-15. ~15 GiB margin. The boot ledger is the referee — read it
+#     after deploy and re-size if the engine prints a clamp.
+#   HALOGEN_SPEC_ADAPT=0 is MANDATORY (card): the default adaptive policy
+#   turns the MTP head off for whole requests after low acceptance and
+#   HALVES decode as a result. Frank accepted the head always-on trade
+#   (temp 1.0 traffic costs acceptance; the card still ships the recipe).
+#
+# CLIENT CONTRACT deltas vs the 27B shape (harness side lives in
+# ~/.omp/agent/models.yml):
+#   * SERVER-SIDE SAMPLER EXISTS NOW: HALOGEN_TEMPERATURE/TOP_P/TOP_K set
+#     the card's thinking recipe (1.0 / 0.95 / 20) as engine defaults, so
+#     an absent temperature is no longer greedy. models.yml extraBody
+#     carries the same values — consistent, not load-bearing.
+#   * HALOGEN_MAX_TOK=8192 is only the DEFAULT for requests that omit
+#     max_tokens; the harness sends at most its maxTokens 45056. Requests
+#     over HALOGEN_MAX_TOKENS_CAP (65536) are a 400 naming the cap. Cap and
+#     queue timeout are COUPLED (FLAGS.md table): 65536 at ~10 t/s serial
+#     worst case needs > 6550 s; 7200 honors it. Do not move one alone.
+#   * max_tokens INCLUDES thinking; a reasoning reply over a small budget
+#     returns empty content with finish_reason "length".
+#   * Prompt cache is ON by default (PROMPT_CACHE=2): a follow-up turn at
+#     100k context re-prefills ~2 s instead of ~88 s cold. No pin needed —
+#     unlike the 27B engine, auto-sizing fits this box's ledger.
+#   * Thinking lands in reasoning_content (same as the 27B engine); the
+#     v2 chat template rejects reasoning_effort outside xhigh|medium|low,
+#     so the models.yml fold (high->xhigh) still applies.
+#   * Vision: OpenAI image_url content parts only (the fleet's standard
+#     contract). >=256 px probe images or the model answers "video".
+#   * response_format json_object/json_schema unsupported in this line
+#     (no constrained decoding) — unchanged from the 27B engine.
+#   * The engine's token protocol has no auth and binds loopback INSIDE
+#     the container; only the frontend port (8731) is published. Same as
+#     before. Kernel needs CONFIG_HSA_AMD_SVM (CachyOS ships it) and
+#     amdgpu.noretry=0 (check at deploy).

-# The target payload (scripts/download-frank-large-swift.sh): the serve
-# script refuses to start without the main GGUF and the DFlash2 drafter.
-# Swapped in from unsloth/Qwen3.8-27B-GGUF:UD-Q8_K_XL on 2026-09-28. The two
-# are structurally interchangeable for gufo (same qwen35 architecture, the
-# same 866 tensors in the same order, the same per-tensor Q8_0/F32/BF16
-# layout, the same MTP head), so only this path changed.
-MODEL_DIR=/home/dank/models/Swift-1.5-Qwen3.8-27B-Uncensored-MTP-UD-Q8_K_XL
-MODEL_FILE=/home/dank/models/Swift-1.5-Qwen3.8-27B-Uncensored-MTP-UD-Q8_K_XL/Swift-1.5-Qwen3.8-27B-Uncensored-Dynamic-MTP-UD-Q8_K_XL.gguf
+# Image: digest-pinned (fetch law: tag pulls silently re-point; the digest
+# is what was qualified). 0.16.2 = the version the model card names as its
+# quickstart pin; RepoDigest captured at pull on 2026-10-06.
+FRANK_LARGE_IMAGE=ghcr.io/peonist-ai/halogen-flash-server@sha256:0c61bf84ac22308a53f5d1ca6b86806702d7039e5ebc51cae4c66621b92fe04a
+# Container identity: kept from the 27B era (unit ExecStop names it).
+FRANK_LARGE_CONTAINER=frank-large-halogen
+# Old quadlet-era container identity; the serve script clears stale names.
+FRANK_LARGE_LEGACY_CONTAINERS=frank-large
+# host:container — the frontend listens on 8731 inside (HALOGEN_API_PORT
+# default); the engine port (8730) is never published.
+FRANK_LARGE_PUBLISH=7770:8731

-# The DFlash2 drafter is UNCHANGED by the model swap and still lives beside
-# the previous target, so it carries its own directory. It has a sliding
-# window of 2048, which is what keeps its KV small (~40 MiB).
-DRAFTER_DIR=/home/dank/models/Qwen3.8-27B-UD-Q8_K_XL
-DRAFTER_FILE=/home/dank/models/Qwen3.8-27B-UD-Q8_K_XL/Qwen3.8-27B-DFlash2-Q4_K_M.gguf
+FRANK_LARGE_MODEL_DIR=/home/dank/models/halogen-flashnext-v2
+# The checkpoint: Quat3rnion/halogen-swift1.5-qwen3.8-flash-next-v2-abliterated
+# qwen38-flash-next-v2-swift15-abliterated.hgn — compacted r2 of the card's
+# append-only original (byte-identical live tensors, padding dropped:
+# 66,687,678,464 B vs 89,775,868,032 B). LFS sha256 verified at fetch.
+FRANK_LARGE_CHECKPOINT=qwen38-flash-next-v2-swift15-abliterated.hgn
+FRANK_LARGE_CHECKPOINT_BYTES=66687678464
+FRANK_LARGE_CHECKPOINT_SHA256=9c880b8ba48deabfa06172eff1e7572e4fe465e09e97faec6e50c3f6fdcac037
+# The n-gram lookup table: REQUIRED (the engine reads it beside the
+# checkpoint; the card's quickstart names it explicitly). From
+# peonist-ai/halogen-qwen3.8-flash-next @ 5cc17cea.
+FRANK_LARGE_NGRAM_TABLE=qwen38-flash-next-ngram.hgn
+FRANK_LARGE_NGRAM_BYTES=51200246144
+FRANK_LARGE_NGRAM_SHA256=3450acada94e19aabad88bd49b45eb70304820949c2240fc178067b20ae5dffe
+# The vision tower: OPTIONAL on the engine, ENABLED here (image input).
+# From peonist-ai/halogen-qwen3.8-flash-next @ e053f488.
+FRANK_LARGE_VISION_TOWER=qwen38-flash-next-vision.hgn
+FRANK_LARGE_VISION_BYTES=897916416
+FRANK_LARGE_VISION_SHA256=d62e0ae553fe88afd3833733d4a4c669f34d20fd8dfce4b9610525bed2134b10
+# The tokenizer is a FLAT DIRECTORY, $FRANK_LARGE_MODEL_DIR/tokenizer,
+# mounted whole at /tokenizer. The two repos ship the same one; fetched
+# fresh from peonist @ e053f488 (tokenizer.json is LFS, byte-identical to
+# the 27B-era copy; the rest are plain git blobs — size-checked only). The
+# serve gate and the deployer refuse a missing file or ANY symlink: an HF
+# cache snapshot symlinks each file into blobs/, and those links dangle
+# inside the container mount.
+FRANK_LARGE_TOKENIZER_FILES="tokenizer.json chat_template.jinja tokenizer_config.json merges.txt vocab.json generation_config.json"

-# The gufo runtime, built from source by scripts/build-gufo.sh.
-RUNTIME_DIR=/home/dank/GitPackages/gufo-v0.1.1
-
-# Fleet endpoint: the same port and alias every frank-large has served, so
-# no client, probe or models.yml entry changes address.
-ALIAS=frank-large
-HOST=0.0.0.0
-PORT=7770
-
-# One session, full 262144 context (the model's native window).
-CONTEXT_SIZE=262144
-SESSIONS=1
-
-# DFlash2 spec decode (a block-diffusion drafter). gufo sizes the draft
-# block itself; DRAFT_TOKENS caps the proposals per step (gufo's default is
-# 7, chosen over llama.cpp's fixed depth 4 because gufo's adaptive policy
-# shortens the block when acceptance drops) and DRAFT_POLICY is the block
-# length controller.
-DRAFT_TOKENS=7
-DRAFT_POLICY=adaptive
+# --- engine flags (docs/FLAGS.md of the flash-server repo; anything unset
+# stays at the image default, which is what was qualified) ------------------
+# Served model id — Protoss name (wire rename 2026-10-09); the service, unit,
+# scripts and this file keep the frank-large name. OpenWebUI api_configs
+# model_ids for the :7770 backend MUST name this value.
+HALOGEN_MODEL_ID=frank-dragoon
+HALOGEN_CTX=262144
+HALOGEN_KV_POOL_POSITIONS=524288
+HALOGEN_KV_SLOTS=2
+HALOGEN_HOST_RESERVE_GIB=20
+# Spec decode: MTP head is INSIDE the checkpoint (do NOT set HALOGEN_MTP_HEAD);
+# depth 2 is the documented "best all-round" default. HALOGEN_DRAFTER /
+# HALOGEN_DFLASH_* from the 27B engine DO NOT EXIST here.
+HALOGEN_SPEC_ADAPT=0
+# Coupled pair (FLAGS.md table): the cap bounds how long ONE request holds
+# the GPU, the timeout bounds how long the NEXT client waits. Do not move
+# one alone (see the client-contract note above).
+HALOGEN_MAX_TOK=8192
+HALOGEN_MAX_TOKENS_CAP=65536
+HALOGEN_QUEUE_TIMEOUT=7200
+# Server-side sampler = the card's thinking recipe (the engine now HAS one;
+# requests may still override per-call). Matches models.yml extraBody.
+HALOGEN_TEMPERATURE=1.0
+HALOGEN_TOP_P=0.95
+HALOGEN_TOP_K=20
+# Card's belt-and-braces: the engine pins weights by default
+# (HALOGEN_FLASH_PIN_TRUNK=1; only the retired w4b build honored 0).
+HALOGEN_WEIGHTS_LOCK=1
@@ -1,5 +1,5 @@
 [Unit]
-Description=gufo (gufo-org/gufo, ROCm gfx1151) Qwen3.8-27B UD-Q8_K_XL, DFlash2 spec decode, 262144 ctx / 1 session, root systemd unit (dankclaw, frank-large, Strix Halo iGPU)
+Description=halogen-flash-server container (Quat3rnion abliterated Swift1.5 of Qwen3.8-Flash-Next v2, MTP spec decode, 2 slots x 200192, vision ON), root systemd unit (dankclaw, frank-large, Strix Halo iGPU)
 After=network-online.target
 Wants=network-online.target

@@ -13,8 +13,19 @@ Group=dank
 LimitMEMLOCK=infinity
 Environment="FRANK_LARGE_ENV_FILE=/home/dank/serverAppConfigs/scripts/frank-large.env"
 ExecStart=/home/dank/serverAppConfigs/scripts/serve-frank-large.sh
-Restart=on-failure
+# systemd stop alone cannot stop the container child; stop it explicitly
+# (frank-primary pattern). The engine drains on SIGTERM.
+ExecStop=/usr/bin/podman stop -t 60 frank-large-halogen
+TimeoutStartSec=1200
+TimeoutStopSec=120
+# always, not on-failure: an engine that exits 0 after an internal fault
+# would otherwise leave the endpoint down until someone notices.
+Restart=always
 RestartSec=10
+# The engine pins ~62 GiB of v2 weights and keeps a multi-GiB working set
+# on a box that also hosts frank-secondary, ComfyUI and the Caddy/observability
+# stack; make it a late OOM candidate, not the first.
+OOMScoreAdjust=-500

 [Install]
 WantedBy=multi-user.target
@@ -1,61 +1,168 @@
-# frank-primary.env — tcclaviger RDNA4 vLLM production config (dankserver :7770).
+# frank-primary.env — radiance (StillDeadcode, Apache-2.0) serving the
+# OFFICIAL Qwen/Qwen3.8-Flash-Next-FP8 checkpoint at NATIVE precision
+# (modelopt block-FP8 throughout, no requantisation) on BOTH R9700s (TP=2),
+# dankserver :7770.
 #
-# 2026-09-29 cutover from the R9V radiance-vLLM fork (its launcher/shim/state
-# machinery is retired; /home/dank/r9v-state/{q8xl,uncensored} stay on disk as
-# the rollback target, and git tag frank-primary-r9v-q8-20260929 is the last
-# pre-cutover commit). The engine is the tcclaviger RDNA4 vLLM fork — the same
-# digest-pinned image the 2026-09-29 tuning campaign measured — serving
-# orcarouter/Qwen3.8-Flash-Next-Uncensored-FP8 (~172.8 GiB, 131 shards) with
-# the MTP4 drafter on the two R9700s.
+# 2026-10-05 cutover from FreeToken (PR #587 ROCm, one card, experts in
+# pinned host RAM; git tag frank-primary-freetoken-pr587-20261005 is the
+# pre-cutover state — its unit/env/serve trio and Dockerfile are all in the
+# tag and rebuildable). Before that: tcclaviger vLLM fork (tag
+# frank-primary-vllm-shapeG-20261003).
 #
-# Every number here was QUALIFIED by boot probe on 2026-09-29 (not inherited):
-#   * MAXLEN 200000 x SEQS 3 (600k total) at KV q8/q8 fits with headroom:
-#     the engine measured "GPU KV cache size: 729,545 tokens; Maximum
-#     concurrency for 200,000 tokens per request: 3.65x" at KV_BYTES 7.2 GiB
-#     per rank. KV_BYTES is PER RANK — sizing it to the global requirement
-#     OOMs the worker (measured twice: 16 GiB and 9 GiB both die with ~7.5 GiB
-#     free; 7.2 GiB boots).
-#   * --kv-cache-dtype fp8 is the fleet's q8/q8 KV ask (e4m3 for BOTH K and V
-#     — the stock flag quantizes them together, which is exactly what was
-#     requested here). Accepted by the attention path; qualified with a
-#     completion probe.
-#   * PREFETCH IS ABSENT BY LAW: expert-cache telemetry measured the
-#     prefetcher at 36% precision (36,720 slabs prefetched, 13,369 served) —
-#     roughly half the gather traffic was waste; prefetch off is the tuned
-#     incumbent and the +40-51% decode came from deleting the flag. Never
-#     re-add --expert-prefetch without a fresh A/B.
-#   * TunableOp stays OFF: CLAV_TUNABLEOP_SWEEP silently skips under expert
-#     offload unless CLAV_TUNABLEOP_WITH_OFFLOAD=1, and forcing it corrupts
-#     numerics (repetition loops, early halts, invalid measurements).
-# Decode at the tuned shape: 40.57/37.44/37.57 tok/s (166/1087/4064 ctx) at
-# the 32k window; 42.94/34.55/36.09 at this 200k shape (arena 6,032 slots).
+# WHY RADIANCE: the engine is C++/HIP with libr4d kernels built for
+# gfx1201, drives `podman run` directly from the digest-pinned
+# stilldeadcode/radiance image (287 MiB; host needs only the amdgpu driver),
+# and its qwen4exp plugin reads the modelopt FP8 encoding NATIVELY
+# (`X.weight_scale_inv` is the scale plane — checkpoint.h). The PLE n-gram
+# table (51B params, 47.7 GiB) is served from pinned host RAM via io_uring
+# reads — Frank's call: "Native FP8. Experts from ram. Pin the PLE to ram."
+# The 4-bit-codebook publisher shape (qwen3.8-next-flash-fp8-iq4r-moe.rad,
+# --host-pool-mib 12288) remains the documented fallback if the native
+# container proves unbootable.
 #
-# HIP reads bare env vars TEMP, HOST, GPU_MAX_HEAP_SIZE — every key here is
-# prefixed FRANK_PRIMARY_ so none can leak.
+# PAYLOAD: /home/dank/models/rad/qwen4exp-fp8-native.rad — a rad-convert
+# NO-RECIPE pure copy of the official 131-shard checkpoint (every weight
+# keeps its block-FP8 encoding; ~161 GiB). The convert runs against a
+# staging dir whose config.json drops `quantization_config`: the descriptor
+# string is only the plugin SELECTION key (spec §2.4 — exact match, no
+# fallback), and no shipped plugin claims (qwen4exp, "fp8"); the shipped
+# qwen4exp plugin claims (qwen4exp, "") which is what recipe-built
+# containers carry. Per-weight encodings are data, not the descriptor —
+# removing the key changes nothing about what the weights hold. Built by
+# scripts/build-frank-primary-rad.sh (staging + convert + rad-info).
+#
+# SERVING FACTS (docs/GUIDE.md, docs/DOCKER.md, compose/flashnext.yaml):
+# - The HTTP listener binds AFTER the full load (main.cpp: engine.start()
+#   then engine.serve()) — during load the port refuses connections, so a
+#   bare curl /health gate is honest. /health == /ping: 200 JSON while
+#   serving, 503 only while stopping (core/server/admin.cpp:385).
+# - Reasoning: --reasoning-format defaults to auto, emitting thinking in
+#   `reasoning_content` (config.cpp). Tool calls: the qwen3.8 dialect, from
+#   the chat format. Both need no parser flags.
+# - Sampling: the OpenAI endpoint accepts temperature, top_p, top_k, min_p,
+#   typical_p, presence_penalty, frequency_penalty, repetition_penalty,
+#   repeat_last_n, chat_template_kwargs, reasoning_effort,
+#   enable_thinking, preserve_thinking (core/server/oai.cpp:194-205) —
+#   models.yml's extraBody set is fully inside it. Unknown params are
+#   refused (400), which is why nothing unverified rides the wire.
+# - MTP: the official checkpoint carries the mtp.* head (3,101 tensors,
+#   mtp_num_hidden_layers=1) and the plugin probes draft depth 3 — the
+#   publisher's throughput peak for this model. Stated explicitly with
+#   --num-speculative-tokens 3.
+# - The fused TP all-reduce is declared over max-num-seqs x
+#   (1 + --num-speculative-tokens) rows; 3 x 3 = 9 rows fits either wire.
+# Engine pin: Codeberg tag v1.0.13 (commit 53282fa), Docker Hub tag 1.0.13,
+# index digest below. REJECTED UPGRADE: 1.2.0 (sha256:665775ed1d90de6dbb
+# 25558e07fc60d20328f1346f96f3c6e00845eb763f01fa, tag 1.2.0, commit
+# e95d397) was benched 2026-10-06 on this exact shape (fp8-native rad,
+# vram pair, spec 2, seqs 3, 43.6k greedy probe): decode 29.21 t/s and
+# draft acceptance 0.7123 vs the 1.0.13 control's 30.08 and 0.7549.
+# Acceptance on the greedy harness is deterministic (the control
+# reproduced the recorded 0.755 exactly), so the drop is a real numerics
+# change — the 1.2.0 MoE weight-rotation change (92b8aaf) is the named
+# suspect; decode loss tracks the acceptance ratio (0.943 vs 0.945).
+# The vram-pair translation itself is exact (1.2.0 boot ledger took
+# vram_weights 25.46 GiB / vram_kv 3.46 GiB per card byte-identical to
+# the 1.0.13 ledger at EVC 0.82). Re-test 1.2.0+ when the rotation change
+# is untangled; 1.2.0 also REMOVED --expert-vs-cache-ratio in favour of
+# the --vram-weights-mib / --vram-kv-mib pair (core/config.cpp) — this
+# file already uses the pair, which BOTH pins accept (rollback symmetry).
+FRANK_PRIMARY_IMAGE=stilldeadcode/radiance@sha256:4c3f484564bb2872784e46b0a59cbd71becc9c1e746e301a3596ffae253c322e
+# The .rad container FILE (mounted under /models) and the state dir for the
+# prefix-cache disk tier (mounted rw at /data). The model dir holds ONLY
+# .rad files; the raw HF download and its staging dir live beside it and
+# are never mounted.
+FRANK_PRIMARY_MODEL_DIR=/home/dank/models/rad
+FRANK_PRIMARY_MODEL_FILE=qwen4exp-swift15-w4dn8-i8-hc8m.rad
+FRANK_PRIMARY_STATE_DIR=/home/dank/radiance-state
+# The state dir's tune/ subdirectory holds rad-tune's kernel-selection
+# cache (the engine's selector reads $RADIANCE_HOME/tune/<machine>.tune
+# at boot; $RADIANCE_HOME defaults to /opt/radiance, in the image, so the
+# cache must ride a host mount — serve script mounts it, see the runbook's
+# "Tuning levers").
+# Container identity is the SERVICE name (Frank's call, 2026-10-05): one
+# name across systemctl and podman ps. The runtime changes; the name does
+# not. The legacy list retires every predecessor identity that could hold
+# the GPUs or the port.
+FRANK_PRIMARY_CONTAINER=frank-primary
+FRANK_PRIMARY_LEGACY_CONTAINERS="frank-primary-qwen frank-primary-r9v qwen-fp8 glm-probe"
+# Served model id — Protoss name (wire rename 2026-10-09). The container,
+# unit, scripts and this file keep the frank-primary name above; only the
+# alias on the wire moved. Radiance validates the requested model name, so
+# every client (harness, bench scripts, probes) MUST send this value.
+FRANK_PRIMARY_SERVED_NAME=frank-archon
+FRANK_PRIMARY_PORT=7770
+# Frank's chat-template call: froggeric's fixed Qwen3.8 template (the
+# repo's root chat_template.jinja, vendored here; the checkpoint's own
+# template is baked into the .rad by rad-convert and this overrides it).
+# Radiance derives the reasoning/tool-call reply format from it.
+FRANK_PRIMARY_TEMPLATE=/home/dank/serverAppConfigs/scripts/qwen38-froggeric-v22.5.jinja

-# --- runtime ----------------------------------------------------------------
-# Digest-pinned: the floating :latest tag moved once already. Local alias
-# qwen-fp8-runtime:tuned-20260929 (image ID 3322dbbbc96d) points at the same
-# bytes in rootless storage.
-FRANK_PRIMARY_IMAGE=docker.io/tcclaviger/vllm@sha256:da4029666668b90a0386c1ed23ea370c79916ad4e1a6ad1dd9db098b9fa069ea
-
-# --- payload ----------------------------------------------------------------
-FRANK_PRIMARY_MODEL_DIR=/home/dank/models/Qwen3.8-Flash-Next-Uncensored-FP8
-FRANK_PRIMARY_MODEL_SHARDS=131
-
-# --- identity ---------------------------------------------------------------
-# The harness models.yml id stays frank-primary (provider 192.168.0.235:7770).
-FRANK_PRIMARY_CONTAINER=frank-primary-qwen
-FRANK_PRIMARY_SERVED_NAME=frank-primary
-FRANK_PRIMARY_PUBLISH=7770:7770
-# Containers removed before every start: the retired R9V identity and the ad
-# hoc experiment identity (both hold the GPUs / name registrations).
-FRANK_PRIMARY_LEGACY_CONTAINERS="frank-primary-r9v qwen-fp8"
-
-# --- qualified engine shape ---------------------------------------------------
-FRANK_PRIMARY_MAXLEN=200000
-FRANK_PRIMARY_SEQS=3
-FRANK_PRIMARY_KV_BYTES=7730934784
+# --- engine shape (first-boot conservative; the boot ledger arbitrates) --
+# Tensor parallel: BOTH R9700s. The board swap (2026-09-20, ProArt B850
+# x8/x8 on root ports) made collectives legal on this host; radiance's TP
+# wire is libr4d's own, not RCCL.
+FRANK_PRIMARY_TP=2
+# Lossy WHT 6-bit wire for large all-reduces (publisher's production choice:
+# +5.6-6.3% prefill, ~-1.3% 8-stream decode at TP2). min-kb stays the
+# engine default.
+FRANK_PRIMARY_TP_WIRE=wht6
+# Decode slots. 2 is Frank's first-boot value ("use parallelism = 2 to
+# start"); raise only after a real-traffic look at slot occupancy and the
+# KV pool's headroom line. The recurrent-state cost scales with it (54
+# MiB/copy/card, 2 copies with speculation).
+FRANK_PRIMARY_MAX_SEQS=3
+# Served window. Native is 262,144; 200,000 is the publisher's production
+# value (compose) and what models.yml's contextWindow must quote — a longer
+# prompt is refused by the engine, never truncated.
+FRANK_PRIMARY_MAX_MODEL_LEN=200000
+FRANK_PRIMARY_PLACEMENT=expert_tiered
+# Pinned host pool for the expert rows that do not fit VRAM. Native FP8
+# experts are 112.8 GiB total (56.4 GiB per rank slice) vs the 4-bit
+# publisher shape's ~29 GiB — this pool is the cost of native precision.
+# The boot ledger prints the pool's coverage ("the pool backs N% of the
+# worst case ... M full-length sessions"); resize from that line, not from
+# arithmetic.
+FRANK_PRIMARY_HOST_POOL_MIB=40960
+# VRAM left unclaimed per card (publisher default 96 MiB).
+FRANK_PRIMARY_GPU_HEADROOM_MIB=96
+# VRAM split of claimed memory per card, MiB (0 = derive from the card).
+# Engine 1.2.0 REMOVED --expert-vs-cache-ratio; these two knobs replace
+# it. The values transcribe the live 1.0.13 boot ledger at ratio 0.82:
+# vram_weights 25.46 GiB, vram_kv 3.46 GiB per card -> 26071 / 3543.
+# Bracket arms scale the KV term linearly in (1-r) from the 0.82 anchor
+# with weights fixed: ratio 0.90 -> KV 1968, ratio 0.95 -> KV 984. The
+# boot ledger arbitrates; re-derive after any payload change.
+FRANK_PRIMARY_VRAM_WEIGHTS_MIB=26071
+FRANK_PRIMARY_VRAM_KV_MIB=3543
+# KV cache in FP8 (7.32 KiB/token at this model's 12 full-attention layers,
+# 2 KV heads, 256 head_dim, K+V).
 FRANK_PRIMARY_KV_DTYPE=fp8
-FRANK_PRIMARY_SPEC=4
-FRANK_PRIMARY_UTIL=0.98
+# Spec decode: the checkpoint's own MTP head at the publisher's depth-3
+# operating point (probe() returns 3 when mtp_num_hidden_layers > 0; the
+# flag states it so the ledger records the intent).
+FRANK_PRIMARY_SPEC_TOKENS=2
+FRANK_PRIMARY_MAX_BATCHED_TOKENS=2048
+# Prefix cache tiers (publisher pattern; disk tier sized for 2 slots).
+FRANK_PRIMARY_PREFIX_HOST_MIB=4096
+FRANK_PRIMARY_PREFIX_DISK_MIB=65536
+# Frank's pin law: the PLE n-gram table (47.7 GiB) serves from pinned host
+# RAM the cards read. `ram` REFUSES the start if the memlock limit is too
+# small — the unit's LimitMEMLOCK=infinity is the other half of this law.
+FRANK_PRIMARY_NGRAM_PLACEMENT=ram
+# TTM preflight: the pinned ledger (2 x 40 GiB pools + 47.7 GiB PLE + 4 GiB
+# prefix host) is ~132 GiB. ttm's pages_limit/pages_max cap the GPU-mappable
+# pinned pages; the current cap (32,252,917 pages = ~123 GiB) is BELOW the
+# plan, so the serve script refuses to start until both are raised. This is
+# the floor it enforces (in MiB), with ~8 GiB of margin.
+FRANK_PRIMARY_TTM_MIN_MIB=140000
+# Escape hatch for boot-ladder experiments. Unset -> no extra argv. Keep
+# multi-flag values as one string; the serve script expands it unquoted on
+# purpose.
+FRANK_PRIMARY_EXTRA_ARGS=
+# The .rad's minimum size in MiB (deployer gate; a truncated convert is a
+# guaranteed long-timeout boot instead of a fast failure). The converted
+# container measured 177,109 MiB (1,857,127,997,440 B = 173.0 GiB; the
+# plan-only estimate 172.76 GiB covers the data planes but not the metadata
+# tail) - floor sits just under the payload.
+FRANK_PRIMARY_RAD_MIN_MIB=110000
@@ -1,46 +1,64 @@
-# frank-primary.service — tcclaviger RDNA4 vLLM endpoint (orcarouter
-# Qwen3.8-Flash-Next-Uncensored-FP8, MTP4 spec decode, KV q8/q8, 600k total
-# context over 3 sessions, TP=2, tiered experts, 131-shard payload) on the
-# 2x R9700 pair, dankserver :7770. 2026-09-29 cutover from the R9V radiance
-# fork (git tag frank-primary-r9v-q8-20260929 is the pre-cutover state; its
-# launcher/shim/setup-state machinery is retired and the state dirs stay on
-# disk as rollback).
+# frank-primary.service — radiance (StillDeadcode, Apache-2.0) endpoint
+# (OFFICIAL Qwen/Qwen3.8-Flash-Next-FP8 at NATIVE block-FP8 precision, the
+# rad-convert container qwen4exp-fp8-native.rad) on BOTH R9700s (TP=2,
+# wht6 wire), dankserver :7770. 2026-10-05 cutover from FreeToken (git tag
+# frank-primary-freetoken-pr587-20261005 is the pre-cutover state; the
+# retired unit/env/serve trio and Dockerfile are in that tag).
 #
-# Same port, same served alias; the harness models.yml id is unchanged.
-# The serve script drives `podman run` directly (the image entrypoint execs
-# `vllm serve`) and then waits on the container, so a crashed engine is a
-# nonzero ExecStart and Restart=on-failure is real.
-#
-# WHY A ROOT UNIT (not a rootless quadlet): the tiered-expert shape pins tens
-# of GiB of host RAM (the expert host slabs, ~20.5 GiB per rank of pinned
-# anonymous memory). Pinned pages are charged against RLIMIT_MEMLOCK; dank's
-# login session on this host carries 8 MiB and a user quadlet is capped by the
+# Same port, same served alias, same container identity as the unit name;
+# the harness models.yml id is unchanged. The serve script drives
+# `podman run` directly: the digest-pinned stilldeadcode/radiance image
+# (287 MiB, kernels prebuilt for gfx1201) has the engine as its
+# entrypoint. The image lives in dank's ROOTLESS podman storage.
+# WHY A ROOT UNIT (not a rootless quadlet): radiance pins ~132 GiB of HOST
+# RAM (2 x 40 GiB expert pools + the 47.7 GiB PLE ngram table + the prefix
+# host tier). Pinned pages are charged against RLIMIT_MEMLOCK; dank's login
+# session on this host carries 8 MiB and a user quadlet is capped by the
 # user manager's own hard limit — raising that means restarting
-# user@1000.service and bouncing every user container on the host. A root unit
-# grants LimitMEMLOCK=infinity to the dank process via PID1, and the image
-# stays in dank's rootless podman storage.
+# user@1000.service and bouncing every user container on the host. A root
+# unit grants LimitMEMLOCK=infinity to the dank process via PID1; the
+# engine still runs out of dank's rootless podman storage.
 [Unit]
-Description=frank-primary — tcclaviger vLLM Qwen3.8-Flash-Next-Uncensored-FP8 on 2x R9700
+Description=frank-primary — radiance Qwen3.8-Flash-Next-FP8 (native) on 2x R9700 TP=2
 After=network-online.target
 Wants=network-online.target
+# Never let the start-rate limiter give up after a crash loop (a failed LLM
+# endpoint that systemd refuses to retry is worse than a retrying one).
+StartLimitIntervalSec=0

 [Service]
 User=dank
 Group=dank
-# The tiered-expert offloader pins host RAM; an RLIMIT_MEMLOCK smaller than
-# the pinned set fails the load mid-start. PID1 grants infinity here —
-# parity with the fleet's other LLM units.
+# The engine pins ~132 GiB of host RAM (expert pools + PLE table); an
+# RLIMIT_MEMLOCK smaller than the pinned set fails the load mid-start.
+# PID1 grants infinity here — parity with the fleet's other LLM units.
 LimitMEMLOCK=infinity
 Environment="FRANK_PRIMARY_ENV_FILE=/home/dank/serverAppConfigs/scripts/frank-primary.env"
 ExecStart=/home/dank/serverAppConfigs/scripts/serve-frank-primary.sh
 # systemd stop: wait alone cannot stop the container child; stop it
-# explicitly. vLLM 0.26 drains gracefully on SIGTERM.
-ExecStop=/usr/bin/podman stop -t 120 frank-primary-qwen
-# Boot cost: FP8 payload page-in plus torch compile/graph capture —
-# ~15 min cold (measured 2026-09-29), faster warm.
-TimeoutStartSec=1800
-Restart=on-failure
+# explicitly. No graceful drain contract is claimed (the OpenAI front end
+# holds no queue to drain — in-flight decodes are lost, which a stop
+# during a bench window accepts; radiance takes SIGTERM on the engine
+# process and podman reaps the container).
+ExecStop=/usr/bin/podman stop -t 120 frank-primary
+# Boot cost: .rad load (weights + PLE upload) + expert host-pool pin. The
+# publisher's compose healthcheck allows a 15-minute start_period; the
+# vLLM-era budget (1800 s) carried forward.
+# Restart=always, NOT on-failure (the 2026-10-01 vLLM lesson carries over:
+# a clean API-server exit after a worker fault reads as success to
+# systemd). `always` restarts on any container death; an explicit
+# `systemctl stop` still wins, so bench windows and deploys are unaffected.
+Restart=always
 RestartSec=10
+# ExecStop allows the engine 120 s to drain (`podman stop -t 120`); without
+# an explicit TimeoutStopSec systemd's 90 s default cuts the stop short,
+# kills it, and records 'Failed with result timeout' (measured 2026-10-01
+# 13:27:17). Must exceed the podman grace with margin.
+TimeoutStopSec=180
+# The engine pins tens of GiB of host RAM (expert rows + PLE table); it is
+# the last process on this host that should be chosen by the kernel OOM
+# killer.
+OOMScoreAdjust=-500

 [Install]
 WantedBy=multi-user.target
@@ -0,0 +1,62 @@
+# frank-secondary — llama.cpp (master, CUDA sm_86) serving
+# HauhauCS/Qwen3.5-9B-Uncensored-HauhauCS-Aggressive at Q6_K on the RTX 3090,
+# :7771.
+#
+# 2026-10-07 (Frank): swap from Gemma4-26B-A4B QAT Q4_K_M (one 200,704 slot)
+# to Qwen3.5-9B Q6_K at TWO 200,704-token conversations, EXACT (f16) KV,
+# MTP disabled (SPEC_TYPE=none), log verbosity 4.
+#
+# Architecture — read from the GGUF's own header (HTTP range read, not the
+# model card): arch qwen35, 32 layers, dense 9B, full_attention_interval 4
+# ⇒ 8 full-attention layers (3,7,11,...,31) + 24 Gated DeltaNet linear layers,
+# 16 attention heads / 4 KV heads × 256 dim, ssm conv_kernel 4 / state_size
+# 128 / group_count 16 / time_step_rank 32 / inner_size 4096, native 262,144,
+# vocab 248,320, file_type 18 (Q6_K). Tensor-type histogram: Q6_K×250 +
+# F32×177 — standard types only, so the stock master build loads it (no
+# custom-quant lock-in like the ROCmFPX era). NO nextn/mtp tensors exist, so
+# in-file MTP is impossible here; SPEC_TYPE=none is forced by the payload.
+#
+# KV rate law: 8 full layers × 4 KV heads × 256 dim = 8,192 elements per
+# token per cache. f16 K + f16 V = 32,768 B/token = 32 KiB/token, so the
+# 401,408-token pool is 12.25 GiB. The 24 DeltaNet layers cost a fixed
+# recurrent state per SLOT (not per token) — that is why two 200k slots fit
+# where the 26B MoE fit only one.
+#
+# Feasibility comes from the predecessor's MEASURED boots, not a formula:
+# MiMo-V2.6-Distill-Qwen-9B (identical qwen35 geometry, 7,793,710,624 B) at
+# 2×200,704 f16 K + f16 V measured 21,276 used / 2,849 free at ubatch 1024
+# and 21,846 / 2,279 at ubatch 2048. This file is 415 MiB lighter in weights,
+# its BF16 mmproj is 3 MiB heavier ⇒ projected ~21,430 used / ~2,700 free.
+# The verbosity-4 boot ledger is the referee.
+#
+# Ubatch 2048 is the measured CUDA optimum (measured on the MiMo predecessor
+# at this exact arch + shape: +4.6 % prefill at 8.9k depth, decode flat, cost
+# 616 MiB). 1024 is the fallback rung if the boot lands under the 2 GiB floor.
+LLAMA_BIN=/home/dank/GitPackages/llama-master/build-cuda/bin/llama-server
+MODEL_FILE=/home/dank/models/Qwen35-9B-HauhauCS-Q6K/Qwen3.5-9B-Uncensored-HauhauCS-Aggressive-Q6_K.gguf
+MMPROJ_FILE=/home/dank/models/Qwen35-9B-HauhauCS-Q6K/mmproj-Qwen3.5-9B-Uncensored-HauhauCS-Aggressive-BF16.gguf
+# Served model id — the harness/family name (Protoss rename completed to the
+# wire 2026-10-09: harness ids + homepage cards 2026-10-08, /v1/models aliases
+# this day). The SERVICE, unit, scripts and this env file keep the
+# frank-secondary name; only the alias on the wire changed. Consumers that
+# key on the served id: OpenWebUI (model row, access_grant, api_configs
+# model_ids) and the harness models.yml.
+ALIAS=frank-zealot
+BIND_ADDR=0.0.0.0
+LISTEN_PORT=7771
+# CTX_SIZE is the POOL llama.cpp divides across slots: 401,408 / 2 = 200,704
+# per conversation. MUST stay divisible by PARALLEL (the seam enforces it) and
+# the harness contextWindow quotes the PER-SLOT number — a longer prompt is
+# answered with a 400, never truncated.
+CTX_SIZE=401408
+PARALLEL=2
+BATCH_SIZE=4096
+UBATCH_SIZE=2048
+CACHE_TYPE_K=f16
+CACHE_TYPE_V=f16
+SPEC_TYPE=none
+NGL=99
+# Log verbosity threshold (llama.cpp -lv). 4 = debug: KV layout, buffer sizes
+# and graph capture land in the journal, which is what the VRAM-fit forensics
+# read. Levels above 4 flood the journal.
+VERBOSITY=4
@@ -0,0 +1,19 @@
+[Unit]
+Description=llama.cpp Qwen3.5-9B-Uncensored-HauhauCS-Aggressive Q6_K GGUF (qwen35 hybrid, 2x200704 slots, f16 K+V, MTP off, mmproj vision) on the RTX 3090 (dankclaw, frank-secondary)
+After=network-online.target
+Wants=network-online.target
+
+[Service]
+Type=simple
+User=dank
+Group=dank
+# CUDA pins its staging buffers, so the process needs the memlock budget even
+# though the weights themselves are copied into VRAM.
+LimitMEMLOCK=infinity
+Environment="FRANK_SECONDARY_ENV_FILE=/home/dank/serverAppConfigs/scripts/frank-secondary.env"
+ExecStart=/home/dank/serverAppConfigs/scripts/serve-frank-secondary.sh
+Restart=on-failure
+RestartSec=10
+
+[Install]
+WantedBy=multi-user.target
@@ -1,118 +0,0 @@
-# Tunables for frank-small (HauhauCS/Gemma4-12B-QAT-Uncensored-HauhauCS-
-# Balanced, Q4_K_M, plus the BF16 vision projector and the separate MTP
-# sidecar) on mainline llama.cpp, Dankclaw's RTX 3090, since 2026-09-17.
-#
-# Runtime: llama.cpp master 37b53fd, CUDA build for sm_86:
-#   cmake -B build-cuda -DCMAKE_BUILD_TYPE=Release -DGGML_NATIVE=ON \
-#     -DGGML_CUDA=ON -DCMAKE_CUDA_ARCHITECTURES=86 -DGGML_CUDA_FA_ALL_QUANTS=ON
-#   cmake --build build-cuda --target llama-server -j
-# GGML_CUDA_FA_ALL_QUANTS is ON: this service runs q8_0 KV with flash
-# attention, and without the flag quantized-KV prefill collapses ~12x on
-# CUDA (see the CUDA lesson in AGENTS.md).
-#
-# Model: HauhauCS/Gemma4-12B-QAT-Uncensored-HauhauCS-Balanced at Q4_K_M
-# (6.87 GiB). Architecture (GGUF arch gemma4): 48 layers, MHA with 16 kv
-# heads x 512 head_dim, rope freq_base 1e6, final logit softcapping 30,
-# per-layer token embeddings (PLE) inside the file, and a sliding-window
-# attention pattern (attention.sliding_window = 1024) -- most layers cache
-# only a 1024-position window, which is why the KV pool is a fraction of
-# the full-ctx x all-layers number.
-#
-# MTP: the Multi-Token Prediction head ships as a SEPARATE file
-# (mtp-gemma-4-12B-it.gguf, arch gemma4-assistant, 4 nextn layers, 0.24
-# GiB). Unlike the Qwen3.8 -mtp build (head inside the main file), this one
-# is passed via --model-draft with --spec-type draft-mtp, so the draft
-# context loads the assistant model. The same --spec-type value chains the
-# draftless ngram-mod matcher, which attaches unconditionally; two
-# context-backed drafters cannot share one process, so draft-mtp,ngram-mod
-# is the only legal shape that carries both.
-#
-# SIZING (RTX 3090, 24576 MiB): CTX_SIZE = PARALLEL x 262144 -- four
-# FULL-window slots, measured to fit: the 4-slot boot idles at 22,346
-# MiB and each extra full-context slot costs 3,450 MiB (measured 1-slot
-# 11,996 -> 4-slot 22,346). A fifth slot projects to ~25.8 GiB, past the
-# card, and the KV pool allocates upfront -- so 4 is the max. Explicit
-# CTX_SIZE bypasses --fit on purpose: --fit maximises the TOTAL pool,
-# which would shrink each slot below the full 262144 window once
-# parallel > 1.
-#
-# Vision + audio: --mmproj loads the BF16 projector (11 tensors, 52M
-# params; it carries an audio encoder too). The fleet's image path lives
-# here, which is why halogen no longer sets HALOGEN_VISION_TOWER. Audio
-# WORKS: input_audio content parts transcribe verbatim (verified
-# 2026-09-17 with a 16k mono WAV).
-#
-# Sampling: greedy by default (the fleet contract); a client that sends a
-# temperature gets its own, and the harness applies per-model overrides
-# from models.yml.
-#
-# WARNING: do not name any variable TEMP, HOST, or GPU_MAX_HEAP_SIZE (HIP
-# reads those bare vars on the ROCm hosts -- see the 2026-08-13 incident in
-# AGENTS.md). This box is CUDA, but the rule keeps env files portable.
-
-# llama.cpp runtime binary: mainline master 37b53fd, CUDA sm_86,
-# FA_ALL_QUANTS=ON.
-LLAMA_BIN=/home/dank/GitPackages/llama.cpp-mainline/build-cuda/bin/llama-server
-
-# Model package. The MTP sidecar file is required for speculative decoding.
-MODEL_DIR=/home/dank/models/Gemma4-12B-QAT
-MODEL_FILE=Gemma4-12B-QAT-Uncensored-HauhauCS-Balanced-Q4_K_M.gguf
-MMPROJ_PATH=/home/dank/models/Gemma4-12B-QAT/mmproj-Gemma4-12B-QAT-Uncensored-HauhauCS-Balanced-BF16.gguf
-DRAFT_FILE=mtp-gemma-4-12B-it.gguf
-
-# Fleet identity and network. 7771 is this service's own endpoint on
-# Dankclaw; 7770 stays frank-large's. The alias is the id at /v1/models and
-# what clients ask for -- the harness entry in models.yml names it.
-ALIAS=frank-small
-BIND_ADDR=0.0.0.0
-LISTEN_PORT=7771
-
-# Four FULL 262144-token contexts, one per slot (CTX_SIZE is the total
-# pool; --parallel divides it). See the SIZING note above for the fit
-# arithmetic; a fifth slot does not fit.
-CTX_SIZE=1048576
-PARALLEL=4
-CACHE_TYPE_K=q8_0
-CACHE_TYPE_V=q8_0
-# CPU RAM cache cap in MiB (build default 8192). This host is memory-tight
-# (frank-large's 70 GiB GTT plus the rest of the stack), and Gemma4 runs
-# fully on the 3090 (-ngl 999), so the CPU-side cache is not the hot path;
-# cap it at 2048 to give back ~6 GiB of LPDDR. Gemma4 is not a DeltaNet
-# model, so the >12.8 GiB CACHE_RAM floor does not apply.
-CACHE_RAM=2048
-
-# Batch geometry: 8192 logical / 1024 physical. Measured on this card at
-# max length (254k-token prefill / 253k-ctx decode, 4 slots, q8 KV):
-# ubatch 512 -> pp 915 t/s, tg@250k 90-204; ubatch 1024 -> pp 962,
-# tg 257-302. Larger ubatches (2048) OOM the MTP draft context's pp
-# compute buffer (~3.3 GiB against 2.2 GiB headroom) and must not be
-# tried. Idle VRAM is 23,300 MiB at this geometry.
-BATCH_SIZE=8192
-UBATCH_SIZE=1024
-
-# CPU threads: this service runs entirely on the GPU (NGL=999), so threads
-# cover tokenization, sampling and the HTTP layer only. Deliberately modest
-# -- this host's CPU is shared with halogen's engine, which is CPU-hungry.
-NUM_THREADS=8
-THREAD_BATCH=8
-
-# Every layer on the 3090, with flash attention (required for q8 KV).
-NGL=999
-FLASH_ATTN=on
-
-# Speculative decoding: the MTP sidecar (draft context) chained with the
-# draftless ngram-mod matcher. Draft depth and busy-wait level are made
-# explicit here (both were the implicit build defaults: 3 and 50).
-# Measured at 250k ctx: depth 4 + poll 100 DECODED WORSE (230-257 t/s)
-# than depth 3 + poll 50 (257-302 t/s) — the longer verification round
-# costs more attention re-read at long context than the extra accepted
-# token earns. Keep 3/50.
-SPEC_TYPE=draft-mtp,ngram-mod
-SPEC_DRAFT_N_MAX=3
-POLL_LEVEL=50
-
-# Greedy default (fleet contract); reasoning is extracted into
-# reasoning_content with an unlimited budget (the model decides).
-TEMPERATURE=0.0
-REASONING_FORMAT=deepseek
-REASONING_BUDGET=-1
@@ -1,20 +0,0 @@
-[Unit]
-Description=llama.cpp Qwen3.8-27B GSQ-RCO IQ3_S MTP+vision (dankclaw, frank-small)
-After=network-online.target
-Wants=network-online.target
-
-[Service]
-Type=simple
-User=dank
-Group=dank
-# CUDA pins its staging buffers, so the process needs the memlock budget even
-# though the weights themselves are copied into VRAM. Parity with the halogen
-# quadlet's Ulimit=memlock=-1:-1 and the retired ik unit.
-LimitMEMLOCK=infinity
-Environment="FRANK_SMALL_ENV_FILE=/home/dank/serverAppConfigs/scripts/frank-small.env"
-ExecStart=/home/dank/serverAppConfigs/scripts/serve-frank-small.sh
-Restart=on-failure
-RestartSec=10
-
-[Install]
-WantedBy=multi-user.target
@@ -0,0 +1,107 @@
+#!/bin/sh
+# hfget.sh -- fetch a Hugging Face repo with curl and jq, and nothing else.
+#
+# The engine and its tooling do not depend on Python, so `huggingface-cli` is not assumed. The API
+# gives a file listing as JSON and every file has a stable resolve/ URL, which is all a downloader
+# needs.
+#
+#   scripts/hfget.sh <repo-id> [dest-dir] [revision]
+#
+# Set HF_TOKEN for a gated or private repo. Re-running resumes: a file whose local size already
+# matches the listing is skipped and a short one is continued with `curl -C -`, so an interrupted
+# transfer of a 300 GB checkpoint costs only what it had not finished.
+#
+# WEIGHT DUPLICATES ARE SKIPPED. Many repos carry the same tensors several times over -- safetensors
+# beside the older pickle .bin, an ONNX export, a GGUF conversion -- and on a large checkpoint each
+# copy is hundreds of gigabytes of the same numbers. When the listing holds any .safetensors, every
+# other weight format is left behind.
+#
+# HFGET_JOBS concurrent transfers, default 4. These are long sequential reads: a few streams
+# saturate the link and more only makes the destination seek.
+set -e
+
+# ---- child mode: one file, named by "<size><tab><path>" in $1, everything else inherited -------
+# sh cannot export a function, so parallelism is one re-entry of this script per file under
+# `xargs -P` rather than a worker pool.
+if [ -n "$HFGET_CHILD" ]; then
+    size=${1%%	*}; path=${1#*	}
+    out=$HFGET_DEST/$path
+    if [ -f "$out" ]; then
+        cur=$(wc -c < "$out")
+        if [ "$cur" = "$size" ]; then echo "  = $path"; exit 0; fi
+        echo "  + $path (resuming at $cur of $size)"
+    else
+        echo "  > $path"
+    fi
+    mkdir -p "$(dirname "$out")"
+    if [ -n "$HF_TOKEN" ]; then
+        curl -sSL --fail --retry 5 --retry-delay 5 --retry-all-errors -C - \
+             -H "Authorization: Bearer $HF_TOKEN" \
+             -o "$out" "https://huggingface.co/$HFGET_REPO/resolve/$HFGET_REV/$path"
+    else
+        curl -sSL --fail --retry 5 --retry-delay 5 --retry-all-errors -C - \
+             -o "$out" "https://huggingface.co/$HFGET_REPO/resolve/$HFGET_REV/$path"
+    fi
+    got=$(wc -c < "$out")
+    [ "$got" = "$size" ] || { echo "hfget: $path is $got bytes, listing says $size" >&2; exit 1; }
+    exit 0
+fi
+
+# ---- parent ------------------------------------------------------------------------------------
+repo=$1
+[ -n "$repo" ] || { echo "usage: hfget.sh <repo-id> [dest-dir] [revision]" >&2; exit 2; }
+dest=${2:-$HOME/models/$repo}
+rev=${3:-main}
+jobs=${HFGET_JOBS:-4}
+command -v jq >/dev/null || { echo "hfget: jq is required" >&2; exit 2; }
+
+echo "hfget: listing $repo@$rev"
+# THE LISTING IS PAGINATED AND THE PAGE SIZE IS SMALL. A large checkpoint runs to a hundred-odd
+# shards and the API returns fifty entries with a `Link: ...; rel="next"` header. Taking only the
+# first page is the worst possible failure here: every file it did fetch is the right length, so
+# the size check passes and the converter is handed a checkpoint missing two thirds of its
+# tensors. Follow the cursor until there is no next link.
+api="https://huggingface.co/api/models/$repo/tree/$rev?recursive=true&expand=true"
+listing='[]'
+page=0
+while [ -n "$api" ]; do
+    hdr=$(mktemp "${TMPDIR:-/tmp}/hfhdr.XXXXXX")
+    if [ -n "$HF_TOKEN" ]; then
+        body=$(curl -sSL --fail-with-body -D "$hdr" -H "Authorization: Bearer $HF_TOKEN" "$api")
+    else
+        body=$(curl -sSL --fail-with-body -D "$hdr" "$api")
+    fi
+    [ "$(printf '%s' "$body" | jq -r 'type')" = "array" ] || {
+        rm -f "$hdr"
+        echo "hfget: $(printf '%s' "$body" | jq -r '.error // .message // .')" >&2
+        echo "hfget: a gated or private repo needs HF_TOKEN in the environment" >&2; exit 1; }
+    listing=$(printf '%s\n%s' "$listing" "$body" | jq -s 'add')
+    api=$(tr -d '\r' < "$hdr" | sed -n 's/^[Ll]ink: *<\([^>]*\)>; *rel="next".*/\1/p' | head -1)
+    rm -f "$hdr"
+    page=$((page + 1))
+    [ "$page" -gt 200 ] && { echo "hfget: listing did not terminate after 200 pages" >&2; exit 1; }
+done
+echo "hfget: $page page(s) of listing"
+have_st=$(printf '%s' "$listing" | jq '[.[] | select(.path | endswith(".safetensors"))] | length')
+sel='.[] | select(.type == "file")'
+[ "$have_st" -gt 0 ] && sel="$sel"' | select((.path | test("\\.(bin|pth|msgpack|h5|onnx|onnx_data|gguf)$")) | not)'
+
+list=$(mktemp "${TMPDIR:-/tmp}/hfget.XXXXXX")
+trap 'rm -f "$list"' EXIT INT TERM
+printf '%s' "$listing" | jq -r "$sel"' | "\(.size)\t\(.path)"' > "$list"
+
+n=$(wc -l < "$list")
+[ "$n" -gt 0 ] || { echo "hfget: the listing has no files" >&2; exit 1; }
+need=$(awk -F'\t' '{s += $1} END {printf "%.1f", s / 1073741824}' "$list")
+mkdir -p "$dest"
+free=$(df -BG --output=avail "$dest" | tail -1 | tr -dc '0-9')
+echo "hfget: $n file(s), $need GiB -> $dest"
+[ "$have_st" -gt 0 ] && echo "hfget: safetensors present, pickle weight formats skipped"
+echo "hfget: $free GiB free on the destination filesystem"
+awk -v n="$need" -v f="$free" 'BEGIN { if (n > f) { print "hfget: THE DOWNLOAD DOES NOT FIT"; exit 1 } }'
+
+export HFGET_CHILD=1 HFGET_DEST="$dest" HFGET_REPO="$repo" HFGET_REV="$rev"
+self=$(cd "$(dirname "$0")" && pwd)/$(basename "$0")
+tr '\n' '\0' < "$list" | xargs -0 -P "$jobs" -I{} sh "$self" "{}" \
+    || { echo "hfget: incomplete -- re-run to resume" >&2; exit 1; }
+echo "hfget: complete -- $(du -sh "$dest" | cut -f1) in $dest"
@@ -0,0 +1,91 @@
+#!/usr/bin/env bash
+# kld-gate-swift15.sh — RUNS ON DANKSERVER. Score the quantized candidate
+# against the BF16 reference and apply the pre-approved gate:
+#   mean KL <= 0.03 AND top-1 agreement >= 95%.
+# Same engine (1.0.13) and same boot shape as the reference RECORD
+# (run-swift15-bf16.sh kld): disk-tier, no speculation, deterministic, fp8 KV.
+# --kld-ref reads its tokens from the reference (no --kld-corpus here — the
+# (run-swift15-bf16.sh kld): 1.0.13, tp 2, disk-tier, fp8 KV, no speculation,
+# deterministic, max-model-len 16384, vram-kv 3072, 4 slots. Two deviations
+# are forced by the w4 file: MNBT 2048 (the w4 kernels stream from the file
+# tier ONLY at MNBT<=2048 — at the recording's 32768 the engine demands a
+# widened form: "DID NOT FIT ... stored form is not the container's"), and
+# pool 40960 (the recording's 12 GiB pool predates the w4 tables; at 12288 the
+# gate_up slab cannot be cached). Chunk size is softmax-invariant, so the
+# MNBT delta is float-order noise riding on the KL floor.
+set -uo pipefail
+PIN=sha256:4c3f484564bb2872784e46b0a59cbd71becc9c1e746e301a3596ffae253c322e
+C=swift15gate
+LOG=/home/dank/logs/swift15_kldgate.log
+KLD_REF_DIR=/home/dank/corpora/kld-ref
+KLD_OUT=/home/dank/radiance-state/kld-gate.json
+
+[ -d "$KLD_REF_DIR" ] || { echo "missing reference: $KLD_REF_DIR"; exit 2; }
+: > "$KLD_OUT"
+cp -f /home/dank/serverAppConfigs/scripts/qwen38-froggeric-v22.5.jinja \
+      /home/dank/models/tpl.jinja 2>/dev/null || true
+sudo systemctl stop frank-primary >/dev/null 2>&1 || true
+podman rm -f "$C" >/dev/null 2>&1 || true
+
+podman run --rm --name "$C" \
+    --network host --init \
+    --security-opt seccomp=unconfined \
+    --ulimit memlock=-1:-1 \
+    --device /dev/kfd --device /dev/dri \
+    -v /home/dank/models:/models:ro \
+    -v /home/dank/radiance-state:/data \
+    -v "$KLD_REF_DIR:/data/kld-ref:ro" \
+    "stilldeadcode/radiance@$PIN" \
+    --model /models/rad/qwen4exp-swift15-w4dn8-i8-hc8m.rad \
+    --tp 2 --placement expert_tiered --weights-disk-tier \
+    --host-pool-mib 40960 --deterministic --gpu-headroom-mib 768 \
+    --num-speculative-tokens 0 \
+    --kv-cache-dtype fp8 --max-model-len 16384 \
+    --ngram-placement ram \
+    --override-chat-template /models/tpl.jinja \
+    --kld-ref /data/kld-ref --kld-out /data/kld-gate.json \
+    --vram-kv-mib 3072 \
+    --max-num-seqs 4 --max-num-batched-tokens 2048 \
+    2>&1 | tee "$LOG"
+rc=${PIPESTATUS[0]}
+[ "$rc" -eq 0 ] || { echo "KLD GATE BOOT FAILED rc=$rc; log: $LOG"; exit "$rc"; }
+
+python3 - "$KLD_OUT" "$LOG" <<'PY'
+import json, sys, re
+out, log = sys.argv[1], sys.argv[2]
+def num(pat):
+    m = re.findall(pat, open(log, errors="replace").read(), re.I)
+    return float(m[-1]) if m else None
+kl = top1 = None
+try:
+    d = json.load(open(out))
+    def find(obj, keys):
+        if isinstance(obj, dict):
+            for k, v in obj.items():
+                if any(s in k.lower() for s in keys) and isinstance(v, (int, float)):
+                    return float(v)
+            for v in obj.values():
+                r = find(v, keys)
+                if r is not None: return r
+        return None
+    kl = find(d, ("mean_kl", "mean kl", "kl_mean", "mean"))
+    top1 = find(d, ("top1", "top_1", "top-1", "agreement"))
+    if top1 is not None and top1 <= 1.0:
+        top1 *= 100.0
+except Exception as e:
+    print("report JSON unusable (%s); falling back to log" % e)
+if kl is None:
+    kl = num(r"mean[_ ]kl[^0-9]*([0-9]*\.?[0-9]+(?:e[-+][0-9]+)?)")
+if top1 is None:
+    top1 = num(r"top[-_ ]?1[^0-9]*([0-9]*\.?[0-9]+)")
+if kl is None or top1 is None:
+    print("KLD GATE PARSE FAILED: mean_kl=%s top1=%s — inspect %s / %s" % (kl, top1, out, log))
+    sys.exit(2)
+print("KLD GATE: mean KL = %.4f (max 0.03) | top-1 = %.2f%% (min 95.0)" % (kl, top1))
+ok = (kl <= 0.03) and (top1 >= 95.0)
+print("KLD GATE " + ("PASS" if ok else "FAIL"))
+sys.exit(0 if ok else 1)
+PY
+gate=$?
+echo "kld-gate complete rc=$gate; log: $LOG report: $KLD_OUT"
+exit "$gate"
@@ -0,0 +1,40 @@
+#!/bin/bash
+# kld-then-calib.sh — RUNS ON DANKSERVER. Campaign chain: wait for the KLD
+# record to finish, verify it, then boot the 1.2.0 calibration server and
+# push the 10M-token corpus. One GPU, so the phases serialize; prod is
+# restored by each run script's trap.
+set -u
+KLOG=/home/dank/logs/swift15-kld-driver.log
+CLOG=/home/dank/logs/swift15-calib-driver.log
+S=/home/dank/serverAppConfigs-swift15/scripts
+# Phase 1: KLD record (SKIP_KLD=1 when the record is already complete —
+# kld.json's rows count is the receipt; a mid-run script swap can kill the
+# wrapper with rc=127 after the engine exited clean).
+if [ "${SKIP_KLD:-0}" != 1 ]; then
+    while ! grep -q "exited rc=" "$KLOG" 2>/dev/null; do sleep 60; done
+    RC=$(grep -o "exited rc=[0-9]*" "$KLOG" | tail -1)
+    if [ "$RC" != "exited rc=0" ]; then echo "KLD FAILED ($RC); calib NOT started"; exit 1; fi
+fi
+N=$(find /home/dank/corpora/kld-ref -type f | wc -l)
+SZ=$(du -sm /home/dank/corpora/kld-ref | cut -f1)
+echo "KLD OK files=$N size_mib=$SZ"
+[ "$N" -ge 5 ] && [ "$SZ" -ge 1000 ] || { echo "kld-ref looks thin; calib NOT started"; exit 1; }
+grep -q '"rows": 50235' /home/dank/corpora/kld-ref/kld.json || {
+    echo "kld.json rows != 50235; calib NOT started"; exit 1; }
+
+# Phase 2: calibration serve (1.2.0) + corpus push.
+nohup bash "$S/run-swift15-bf16.sh" calib > "$CLOG" 2>&1 &
+CP=$!
+for _ in $(seq 1 160); do
+    sleep 15
+    curl -sf localhost:7770/health >/dev/null 2>&1 && break
+done
+if ! curl -sf localhost:7770/health >/dev/null 2>&1; then
+    echo "CALIB BOOT FAILED; see /home/dank/logs/swift15_bf16.log"; kill "$CP" 2>/dev/null; exit 1
+fi
+echo "CALIB UP, pushing corpus"
+python3 "$S/calib-push.py" --port 7770 --parallel 16 --container swift15ref --max-rounds 6
+PUSH=$?
+kill "$CP" 2>/dev/null
+sleep 10
+echo "CALIB_CHAIN_DONE push=$PUSH grams=$(find /home/dank/corpora/calib -name 'gram.r*.bin' 2>/dev/null | wc -l)"
@@ -0,0 +1,317 @@
+#!/usr/bin/env python3
+"""Build the two Swift1.5 campaign corpora (calibration + KLD).
+
+calib.jsonl — 10M tokens of agent-shaped RAW text (45% code, 30% prose,
+25% tool-call JSON), one {"prompt": text} per line, pushed to
+/v1/completions with max_tokens=1 during the calibration run (the Hessian
+tap is of the prompt tokens; docs/MOE-W4.md step 2). Code and prose mix
+the repo checkout with HF dataset parquets (Frank's call 2026-10-06:
+local material is only ~1.6M unique tokens; MOE-W4 wants 10M):
+nampdn-ai/tiny-codes (code) and Salesforce/wikitext wikitext-103-raw
+(prose). Tool calls are synthesized from commands harvested from the
+repo's own scripts plus a fleet-ops set, with argument-shaped variants.
+
+kld.jsonl — 1.8M tokens, one {"prompt", "score_from", "source"} per
+line; each prompt is a multi-turn conversation rendered through the
+Froggeric v22.5 template with a >= --kld-min-prompt (8192) token target,
+and the last 256 tokens (the tail of the assistant answer) are the
+scored span: score_from = n - 256. At 8k-token prompts the 1.8M corpus
+is ~220 prompts = ~56k scored positions ~= 25 GB of record, matching
+the published Flash-Next KLD scale (83k positions, MOE-W4.md). The KLD
+reference run and every candidate run consume the SAME file (TOOLS.md
+--kld-record / --kld-ref). The KLD corpus is LOCAL material only, so the
+file-level disjointness rule (fnv1a64(path) % 5 == 0 -> KLD pool) is
+untouched by the HF additions.
+
+Deps: tokenizers, jinja2, pyarrow (venv /home/dank/venvs/radcorpus).
+Usage: make-calib-corpus.py --repo ROOT --tokenizer-json PATH
+    --template PATH --out-dir DIR [--hf-code-parquet F,F]
+    [--hf-prose-parquet F,F] [--calib-tokens 10000000]
+    [--kld-tokens 1800000]
+"""
+import argparse
+import json
+import os
+import random
+import re
+
+from jinja2 import Environment, FileSystemLoader
+from tokenizers import Tokenizer
+
+CODE_EXT = (".sh", ".py", ".jinja", ".recipe", ".env", ".example",
+            ".yaml", ".yml", ".bats", ".service", ".container", ".toml")
+JSONL_EXTS = (".jsonl", ".json")
+DOC_DIR = "serverAppConfigs-docs"
+
+FLEET_CMDS = [
+    "systemctl status frank-primary --no-pager",
+    "journalctl -u frank-primary -n 100 --no-pager",
+    "podman ps --format '{{.Names}} {{.Status}}'",
+    "podman logs --tail 50 frank-primary",
+    "curl -s http://192.168.0.235:7770/health",
+    "curl -s http://192.168.0.235:7770/metrics | grep radiance_",
+    "df -h /home/dank | tail -1",
+    "free -g",
+    "sudo systemctl restart frank-primary",
+    "cd /home/dank/serverAppConfigs && git pull --ff-only",
+    "./scripts/deploy.sh dankserver --service frank-primary",
+    "./tests/run.sh 2>&1 | tail -5",
+    "jq . /home/dank/models/rad/index.json | head -40",
+    "python3 -c \"import json; print(json.load(open('config.json'))['num_hidden_layers'])\"",
+    "ip neigh show 192.168.0.235",
+    "amdgpu_top --device-info 2>/dev/null | head -20",
+    "grep -E 'MemAvailable' /proc/meminfo",
+    "sha256sum model-00001-of-00131.safetensors",
+    "git log --oneline -10",
+    "awk '{s+=$1} END {print s}' shard_sizes.txt",
+]
+TOOL_NAMES = ["bash", "read", "grep", "web_search", "edit", "write"]
+WORKDIRS = ["/home/dank", "/home/dank/serverAppConfigs", "/home/dank/models",
+            "/etc/systemd/system", "/var/log"]
+DESCS = ["check service state", "tail the engine ledger", "list containers",
+          "query health", "read metrics", "check disk", "check memory",
+          "restart the endpoint", "sync the checkout", "run deployer",
+          "run the test seams", "inspect the index", "probe the peer"]
+
+
+def fnv(s: str) -> int:
+    h = 0xcbf29ce484222325
+    for b in s.encode():
+        h = ((h ^ b) * 0x100000001b3) & 0xFFFFFFFFFFFFFFFF
+    return h
+
+
+def harvest_cmds(repo: str, rng: random.Random) -> list[str]:
+    cmds = list(FLEET_CMDS)
+    pat = re.compile(r"^\s*(?:sudo\s+)?([a-z][\w./-]*(?:\s+\S+){1,6})\s*$")
+    for root, dirs, files in os.walk(repo):
+        dirs[:] = [d for d in dirs if not d.startswith(".")]
+        for f in files:
+            if not f.endswith(".sh"):
+                continue
+            try:
+                for line in open(os.path.join(root, f), encoding="utf-8",
+                                 errors="replace"):
+                    m = pat.match(line.rstrip())
+                    if m and len(m.group(1)) > 12:
+                        cmds.append("ssh dank@192.168.0.235 '"
+                                    + m.group(1)[:90] + "'")
+            except OSError:
+                pass
+    rng.shuffle(cmds)
+    return cmds[:4000]
+
+
+def tool_call(rng: random.Random, cmds: list[str]) -> str:
+    """One agent turn: 2-4 tool calls with argument-shaped JSON."""
+    parts = []
+    for _ in range(rng.randint(2, 4)):
+        args: dict = {"command": rng.choice(cmds)}
+        if rng.random() < 0.4:
+            args["cwd"] = rng.choice(WORKDIRS)
+        if rng.random() < 0.3:
+            args["timeout"] = rng.choice([10, 30, 60, 300, 600])
+        if rng.random() < 0.5:
+            args["description"] = rng.choice(DESCS)
+        parts.append('<tool_call>{"name": "%s", "arguments": %s}<|im_end|>'
+                     % (rng.choice(TOOL_NAMES),
+                        json.dumps(args, ensure_ascii=False)))
+    return "\n".join(parts)
+
+
+def load_parquet_texts(path: str, kind: str, rng: random.Random,
+                       cap: int) -> list[str]:
+    import pyarrow.parquet as pq
+    d = pq.read_table(path).to_pydict()
+    n = len(next(iter(d.values())))
+    idx = range(n) if n <= cap else rng.sample(range(n), cap)
+    out = []
+    for i in idx:
+        if kind == "code":
+            if "content" in d:
+                frag = d["content"][i]
+            else:
+                frag = ("// %s\n### Instruction:\n%s\n### Response:\n%s"
+                        % (d.get("file_name", [""])[i], d["prompt"][i],
+                           d["code"][i] or d["response"][i]))
+        else:
+            frag = d["text"][i]
+        if frag and frag.strip():
+            out.append(frag[:8000])
+    rng.shuffle(out)
+    return out[:cap]
+
+
+def main() -> None:
+    ap = argparse.ArgumentParser(description=__doc__)
+    ap.add_argument("--repo", required=True)
+    ap.add_argument("--tokenizer-json", required=True)
+    ap.add_argument("--template", required=True)
+    ap.add_argument("--out-dir", required=True)
+    ap.add_argument("--calib-tokens", type=int, default=10_000_000)
+    ap.add_argument("--kld-tokens", type=int, default=1_800_000)
+    ap.add_argument("--chunk-tokens", type=int, default=8192)
+    ap.add_argument("--score-tail", type=int, default=256)
+    ap.add_argument("--kld-min-prompt", type=int, default=8192)
+    ap.add_argument("--hf-code-parquet", default="")
+    ap.add_argument("--hf-prose-parquet", default="")
+    ap.add_argument("--seed", type=int, default=20261006)
+    a = ap.parse_args()
+
+    rng = random.Random(a.seed)
+    tokenizer = Tokenizer.from_file(a.tokenizer_json)
+    env = Environment(loader=FileSystemLoader(os.path.dirname(a.template)),
+                      trim_blocks=True, lstrip_blocks=True)
+    env.filters["tojson"] = lambda v: json.dumps(v, ensure_ascii=False)
+    env.filters["raise_exception"] = lambda m: (_ for _ in ()).throw(
+        ValueError(m))
+    tname = os.path.basename(a.template)
+
+    code, prose, jsonl = [], [], []
+    for root, dirs, files in os.walk(a.repo):
+        dirs[:] = [d for d in dirs if not d.startswith(".")]
+        for f in sorted(files):
+            p = os.path.relpath(os.path.join(root, f), a.repo)
+            if DOC_DIR in p and f.endswith(".md"):
+                prose.append(p)
+            elif p.startswith(DOC_DIR):
+                pass
+            elif f.endswith(CODE_EXT):
+                code.append(p)
+            elif f.endswith(JSONL_EXTS) and f != "package-lock.json":
+                jsonl.append(p)
+    # file-level split: bucket 0 -> KLD pool, 1..4 -> calibration pool
+    pools: dict[str, tuple[list, list]] = {}
+    for cls, lst in (("code", code), ("prose", prose), ("tools", jsonl)):
+        pools[cls] = ([p for p in lst if fnv(p) % 5 == 0],
+                      [p for p in lst if fnv(p) % 5 != 0])
+
+    def text_of(p: str) -> str:
+        path = os.path.join(a.repo, p)
+        raw = open(path, encoding="utf-8", errors="replace").read()
+        if p.endswith(JSONL_EXTS):
+            lines = [json.loads(l)["content"] for l in raw.splitlines()
+                     if l.strip() and "content" in l]
+            raw = "\n".join(lines)
+        toks = tokenizer.encode(raw, add_special_tokens=False).ids
+        return tokenizer.decode(toks[:2400], skip_special_tokens=True)
+
+    cmds = harvest_cmds(a.repo, rng)
+    # fragment pools per class, corpus-split first
+    frags: dict[str, dict[str, list[str]]] = {}
+    for cls in pools:
+        kld_files, cal_files = pools[cls]
+        fr = {}
+        src = {"kld": kld_files, "cal": cal_files}
+        if cls == "tools":
+            fr["kld"] = [tool_call(rng, cmds) for _ in range(600)]
+            fr["cal"] = [tool_call(rng, cmds) for _ in range(40000)]
+        else:
+            fr["kld"] = [text_of(p) for p in src["kld"]]
+            fr["cal"] = [text_of(p) for p in src["cal"]]
+        frags[cls] = fr
+    # per-class fragment caps: prose rows are short (~66 tok) so need many;
+    # codeparrot files are long (~2.6k tok) so a small cap suffices
+    hfcap = {"code": 40000, "prose": 60000, "tools": 0}
+    for cls, spec in (("code", a.hf_code_parquet),
+                      ("prose", a.hf_prose_parquet)):
+        for path in [p for p in spec.split(",") if p]:
+            frags[cls]["cal"].extend(
+                load_parquet_texts(path, cls, rng, cap=hfcap[cls]))
+    for cls in frags:
+        rng.shuffle(frags[cls]["cal"])
+
+    def toks(s: str) -> int:
+        return len(tokenizer.encode(s, add_special_tokens=False).ids)
+
+    def emit_calib() -> tuple[int, list[dict]]:
+        targets = {"code": .45, "prose": .30, "tools": .25}
+        rows: list[dict] = []
+        for cls, frac in targets.items():
+            want = int(a.calib_tokens * frac)
+            pool = frags[cls]["cal"]
+            if not pool:
+                continue
+            got, i, buf, nbuf = 0, 0, [], 0
+            while got < want:
+                frag = pool[i % len(pool)]
+                i += 1
+                n = toks(frag)
+                if nbuf + n > a.chunk_tokens and buf:
+                    rows.append({"prompt": "\n".join(buf), "cls": cls})
+                    got += nbuf
+                    buf, nbuf = [], 0
+                    if got >= want:
+                        break
+                buf.append(frag)
+                nbuf += n
+                if i > len(pool) * 40:
+                    break  # pool exhausted with repeats; stop
+            if buf and got < want:
+                rows.append({"prompt": "\n".join(buf), "cls": cls})
+                got += nbuf
+            print(f"calib {cls}: {got} tokens, {i} fragments")
+        rng.shuffle(rows)
+        return len(rows), [{k: v for k, v in r.items() if k == "prompt"}
+                           for r in rows]
+
+    def emit_kld() -> tuple[int, list[dict]]:
+        rows, total = [], 0
+        classes = ["code", "prose", "tools"]
+        i = {c: 0 for c in classes}
+        guard = 0
+        while total < a.kld_tokens and guard < 5000:
+            guard += 1
+            msgs, srcs = [], []
+            for t in range(rng.choice([2, 3, 4])):
+                cls = classes[(len(rows) + t) % 3]
+                pool = frags[cls]["kld"]
+                if not pool:
+                    continue
+                frag = pool[i[cls] % len(pool)]
+                i[cls] += 2
+                msgs.append({"role": "user",
+                             "content": "Summarize and explain this "
+                                        f"engineering material:\n{frag}"})
+                msgs.append({"role": "assistant",
+                             "content": frag[:4000]})
+                srcs.append(cls)
+            rendered = env.get_template(tname).render(
+                messages=msgs, add_generation_prompt=False)
+            n = toks(rendered)
+            if n < a.kld_min_prompt:
+                continue
+            rows.append({"prompt": rendered,
+                         "score_from": n - a.score_tail,
+                         "source": "+".join(sorted(set(srcs)))})
+            total += n
+        return len(rows), rows
+
+    os.makedirs(a.out_dir, exist_ok=True)
+    nc, rows = emit_calib()
+    with open(os.path.join(a.out_dir, "calib.jsonl"), "w") as f:
+        for r in rows:
+            f.write(json.dumps(r) + "\n")
+    nk, krows = emit_kld()
+    with open(os.path.join(a.out_dir, "kld.jsonl"), "w") as f:
+        for r in krows:
+            f.write(json.dumps(r) + "\n")
+    scored = sum(len(tokenizer.encode(r["prompt"], add_special_tokens=False).ids)
+                 - r["score_from"] for r in krows)
+    manifest = {
+        "seed": a.seed, "template": tname,
+        "tokenizer": os.path.basename(a.tokenizer_json),
+        "calib_prompts": nc, "kld_prompts": nk,
+        "kld_scored_positions": scored,
+        "kld_score_tail": a.score_tail,
+        "disjoint_file_pools": "local KLD material only; HF code feeds calib",
+        "calib_mix": {"code": .45, "prose": .30, "tools": .25},
+        "hf_code": a.hf_code_parquet, "hf_prose": a.hf_prose_parquet,
+    }
+    with open(os.path.join(a.out_dir, "manifest.json"), "w") as f:
+        json.dump(manifest, f, indent=1)
+    print(json.dumps(manifest, indent=1))
+
+
+if __name__ == "__main__":
+    main()
@@ -0,0 +1,460 @@
+{%- set template_version = "qwen3.8-froggeric-v22.5" %}
+{%- set _tool_format = tool_call_format if tool_call_format is defined else 'xml' %}
+{%- set image_count = namespace(value=0) %}
+{%- set video_count = namespace(value=0) %}
+{%- set add_vision_id = add_vision_id if add_vision_id is defined else false %}
+{%- set enable_thinking = enable_thinking if enable_thinking is defined else true %}
+{%- set auto_disable_thinking_with_tools = auto_disable_thinking_with_tools if auto_disable_thinking_with_tools is defined else false %}
+{%- if preserve_reasoning is defined and preserve_reasoning is not none %}
+    {%- set _preserve_thinking = preserve_reasoning %}
+{%- elif preserve_thinking is defined and preserve_thinking is not none %}
+    {%- set _preserve_thinking = preserve_thinking %}
+{%- else %}
+    {%- set _preserve_thinking = true %}
+{%- endif %}
+{%- set max_tool_arg_chars = max_tool_arg_chars if max_tool_arg_chars is defined else 0 %}
+{%- set max_tool_response_chars = max_tool_response_chars if max_tool_response_chars is defined else 0 %}
+{%- set _default_reasoning_effort = 'medium' %}
+{%- set _has_tools = (tools is defined and tools and tools is iterable and tools is not mapping) %}
+{%- set _effort_raw = (reasoning_effort | string | lower) if reasoning_effort is defined and reasoning_effort is not none else _default_reasoning_effort %}
+{%- set _initial_thinking = enable_thinking %}
+{%- if _effort_raw in ('none', 'off') %}
+    {%- set _initial_thinking = false %}
+    {%- set _initial_effort = 'medium' %}
+{%- elif _effort_raw in ('minimal', 'low') %}
+    {%- set _initial_effort = 'low' %}
+{%- elif _effort_raw in ('high', 'xhigh', 'max', 'ultracode', 'extreme') %}
+    {%- set _initial_effort = 'xhigh' %}
+{%- else %}
+    {%- set _initial_effort = 'medium' %}
+{%- endif %}
+{%- set ns_state = namespace(thinking=_initial_thinking, effort=_initial_effort) %}
+{%- if auto_disable_thinking_with_tools and _has_tools %}
+    {%- set ns_state.thinking = false %}
+{%- endif %}
+{%- for msg in messages %}
+    {%- if msg.role == 'system' or msg.role == 'developer' or msg.role == 'user' %}
+        {%- if msg.content is string %}
+            {%- if '<|think_off|>' in msg.content %}
+                {%- set ns_state.thinking = false %}
+            {%- elif '<|think_on|>' in msg.content %}
+                {%- set ns_state.thinking = true %}
+            {%- elif '<|think_xhigh|>' in msg.content or '<|think_high|>' in msg.content or '<|think_ultracode|>' in msg.content or '<|think_extreme|>' in msg.content or '<|think_max|>' in msg.content %}
+                {%- set ns_state.thinking = true %}
+                {%- set ns_state.effort = 'xhigh' %}
+            {%- elif '<|think_low|>' in msg.content or '<|think_minimal|>' in msg.content %}
+                {%- set ns_state.thinking = true %}
+                {%- set ns_state.effort = 'low' %}
+            {%- elif '<|think_medium|>' in msg.content %}
+                {%- set ns_state.thinking = true %}
+                {%- set ns_state.effort = 'medium' %}
+            {%- endif %}
+        {%- elif msg.content is iterable and msg.content is not mapping %}
+            {%- for item in msg.content %}
+                {%- if item is string %}
+                    {%- set _item_text = item %}
+                {%- elif item is mapping and 'text' in item and item.text is string %}
+                    {%- set _item_text = item.text %}
+                {%- else %}
+                    {%- set _item_text = '' %}
+                {%- endif %}
+                {%- if _item_text %}
+                    {%- if '<|think_off|>' in _item_text %}
+                        {%- set ns_state.thinking = false %}
+                    {%- elif '<|think_on|>' in _item_text %}
+                        {%- set ns_state.thinking = true %}
+                    {%- elif '<|think_xhigh|>' in _item_text or '<|think_high|>' in _item_text or '<|think_ultracode|>' in _item_text or '<|think_extreme|>' in _item_text or '<|think_max|>' in _item_text %}
+                        {%- set ns_state.thinking = true %}
+                        {%- set ns_state.effort = 'xhigh' %}
+                    {%- elif '<|think_low|>' in _item_text or '<|think_minimal|>' in _item_text %}
+                        {%- set ns_state.thinking = true %}
+                        {%- set ns_state.effort = 'low' %}
+                    {%- elif '<|think_medium|>' in _item_text %}
+                        {%- set ns_state.thinking = true %}
+                        {%- set ns_state.effort = 'medium' %}
+                    {%- endif %}
+                {%- endif %}
+            {%- endfor %}
+        {%- endif %}
+    {%- endif %}
+{%- endfor %}
+{%- set reasoning_instructions = '' %}
+{%- if ns_state.thinking %}
+    {%- if ns_state.effort == 'xhigh' %}
+        {%- set reasoning_instructions = 'Reasoning effort is set to xhigh. Please think carefully through the task, validate key assumptions, consider plausible alternatives, and prioritize correctness, consistency, and clarity in the final answer.' %}
+    {%- elif ns_state.effort == 'low' %}
+        {%- set reasoning_instructions = 'Reasoning effort is set to low. Keep your thinking brief and focused, moving directly to the conclusion without unnecessary elaboration.' %}
+    {%- endif %}
+{%- endif %}
+{%- macro render_content(content, do_vision_count, is_system_content=false) %}
+    {%- if content is string %}
+        {{- content }}
+    {%- elif content is iterable and content is not mapping %}
+        {%- for item in content %}
+            {%- if item is mapping %}
+                {%- if item.type == 'image' or 'image' in item or 'image_url' in item %}
+                    {%- if is_system_content %}
+                        {{- raise_exception('System message cannot contain images.') }}
+                    {%- endif %}
+                    {%- if do_vision_count %}
+                        {%- set image_count.value = image_count.value + 1 %}
+                    {%- endif %}
+                    {%- if add_vision_id %}
+                        {{- 'Picture ' ~ image_count.value ~ ': ' }}
+                    {%- endif %}
+                    {{- '<|vision_start|><|image_pad|><|vision_end|>' }}
+                {%- elif item.type == 'video' or 'video' in item or 'video_url' in item %}
+                    {%- if is_system_content %}
+                        {{- raise_exception('System message cannot contain videos.') }}
+                    {%- endif %}
+                    {%- if do_vision_count %}
+                        {%- set video_count.value = video_count.value + 1 %}
+                    {%- endif %}
+                    {%- if add_vision_id %}
+                        {{- 'Video ' ~ video_count.value ~ ': ' }}
+                    {%- endif %}
+                    {{- '<|vision_start|><|video_pad|><|vision_end|>' }}
+                {%- elif 'text' in item %}
+                    {{- item.text }}
+                {%- else %}
+                    {{- raise_exception('Unexpected item type in content.') }}
+                {%- endif %}
+            {%- else %}
+                {{- item | string }}
+            {%- endif %}
+        {%- endfor %}
+    {%- elif content is none or content is undefined %}
+        {{- '' }}
+    {%- else %}
+        {{- raise_exception('Unexpected content type.') }}
+    {%- endif %}
+{%- endmacro %}
+{%- if not messages %}
+    {{- raise_exception('No messages provided.') }}
+{%- endif %}
+{%- set head = namespace(count=0, seen_non_system=false) %}
+{%- for message in messages %}
+    {%- set _is_sys = (message.role == 'system' or message.role == 'developer') %}
+    {%- if _is_sys and not head.seen_non_system %}
+        {%- set head.count = head.count + 1 %}
+    {%- else %}
+        {%- set head.seen_non_system = true %}
+    {%- endif %}
+{%- endfor %}
+{%- set sys_state = namespace(content='') %}
+{%- for message in messages[:head.count] %}
+    {%- set _part = render_content(message.content, false, true) | trim %}
+    {%- if '<|think_off|>' in _part %}{%- set _part = _part.split('<|think_off|>') | join('') | trim %}{%- endif %}
+    {%- if '<|think_on|>' in _part %}{%- set _part = _part.split('<|think_on|>') | join('') | trim %}{%- endif %}
+    {%- if '<|think_xhigh|>' in _part %}{%- set _part = _part.split('<|think_xhigh|>') | join('') | trim %}{%- endif %}
+    {%- if '<|think_high|>' in _part %}{%- set _part = _part.split('<|think_high|>') | join('') | trim %}{%- endif %}
+    {%- if '<|think_ultracode|>' in _part %}{%- set _part = _part.split('<|think_ultracode|>') | join('') | trim %}{%- endif %}
+    {%- if '<|think_extreme|>' in _part %}{%- set _part = _part.split('<|think_extreme|>') | join('') | trim %}{%- endif %}
+    {%- if '<|think_max|>' in _part %}{%- set _part = _part.split('<|think_max|>') | join('') | trim %}{%- endif %}
+    {%- if '<|think_medium|>' in _part %}{%- set _part = _part.split('<|think_medium|>') | join('') | trim %}{%- endif %}
+    {%- if '<|think_low|>' in _part %}{%- set _part = _part.split('<|think_low|>') | join('') | trim %}{%- endif %}
+    {%- if '<|think_minimal|>' in _part %}{%- set _part = _part.split('<|think_minimal|>') | join('') | trim %}{%- endif %}
+    {%- if _part %}
+        {%- if sys_state.content %}
+            {%- set sys_state.content = sys_state.content ~ '\n\n' ~ _part %}
+        {%- else %}
+            {%- set sys_state.content = _part %}
+        {%- endif %}
+    {%- endif %}
+{%- endfor %}
+{%- set _sc = sys_state.content %}
+{%- set _msgs = messages[head.count:] %}
+{%- if _has_tools %}
+    {{- '<|im_start|>system\n' }}
+    {%- if reasoning_instructions %}
+        {{- reasoning_instructions + '\n\n' }}
+    {%- endif %}
+    {{- '# Tools\n\nYou have access to the following functions:\n\n<tools>' }}
+    {%- for tool in tools %}
+        {{- '\n' }}
+        {{- tool | tojson }}
+    {%- endfor %}
+    {{- '\n</tools>' }}
+    {%- if _tool_format == 'json' %}
+        {{- '\n\nIf you choose to call a function ONLY reply in the following format with NO suffix:\n\n' }}
+        {%- if ns_state.thinking %}
+            {{- '<think>\nBrief explanation of tool call\n</think>\n' }}
+        {%- endif %}
+        {{- '<tool_call>\n{"name": "example_function_name", "arguments": {"example_parameter_1": "value_1", "example_parameter_2": "This is the value for the second parameter"}}\n</tool_call>\n\n<IMPORTANT>\nReminder:\n' }}
+        {%- if ns_state.thinking %}
+            {{- '- You can use the <think></think> block to plan your next tool call OR to synthesize data and formulate your final response to the user.\n- ALL explanation and reasoning MUST be placed strictly inside the <think></think> block.\n' }}
+        {%- endif %}
+        {{- '- Function calls MUST follow the specified format: a single JSON object with "name" and "arguments" keys inside <tool_call></tool_call> XML tags.\n' }}
+        {%- if ns_state.thinking %}
+            {{- '- If you choose to call a tool, you MUST output the <tool_call> block IMMEDIATELY after thinking, with NO conversational text before it.\n' }}
+        {%- else %}
+            {{- '- If you choose to call a tool, you MUST output the <tool_call> block IMMEDIATELY, with NO conversational text before it.\n' }}
+        {%- endif %}
+        {{- '- The <tool_call> tag MUST be at the very beginning of a new line, with NO spaces or indentation before it.\n- To call multiple functions, output a separate, completely closed <tool_call></tool_call> block for EACH function. Do NOT nest <tool_call> blocks.\n- If you have all necessary data, provide your final answer directly to the user without any tool call.\n</IMPORTANT>' }}
+    {%- else %}
+        {{- '\n\nIf you choose to call a function ONLY reply in the following format with NO suffix:\n\n' }}
+        {%- if ns_state.thinking %}
+            {{- '<think>\nBrief explanation of tool call\n</think>\n' }}
+        {%- endif %}
+        {{- '<tool_call>\n<function=example_function_name>\n<parameter=example_parameter_1>\nvalue_1\n</parameter>\n<parameter=example_parameter_2>\nThis is the value for the second parameter\nthat can span\nmultiple lines\n</parameter>\n</function>\n</tool_call>\n\n<IMPORTANT>\nReminder:\n' }}
+        {%- if ns_state.thinking %}
+            {{- '- You can use the <think></think> block to plan your next tool call OR to synthesize data and formulate your final response to the user.\n- ALL explanation and reasoning MUST be placed strictly inside the <think></think> block.\n' }}
+        {%- endif %}
+        {{- '- Function calls MUST follow the specified format: an inner <function=...></function> block must be nested within <tool_call></tool_call> XML tags.\n' }}
+        {%- if ns_state.thinking %}
+            {{- '- If you choose to call a tool, you MUST output the <tool_call> block IMMEDIATELY after thinking, with NO conversational text before it.\n' }}
+        {%- else %}
+            {{- '- If you choose to call a tool, you MUST output the <tool_call> block IMMEDIATELY, with NO conversational text before it.\n' }}
+        {%- endif %}
+        {{- '- The <tool_call> and <function> tags MUST be at the very beginning of a new line, with NO spaces or indentation before them.\n- To call multiple functions, output a separate, completely closed <tool_call></tool_call> block for EACH function. Do NOT nest <tool_call> blocks.\n- If you have all necessary data, provide your final answer directly to the user without any tool call.\n</IMPORTANT>' }}
+    {%- endif %}
+    {%- if _sc %}
+        {{- '\n\n' + _sc }}
+    {%- endif %}
+    {{- '<|im_end|>\n' }}
+{%- else %}
+    {%- if _sc %}
+        {{- '<|im_start|>system\n' + (reasoning_instructions + '\n\n' if reasoning_instructions else '') + _sc + '<|im_end|>\n' }}
+    {%- elif reasoning_instructions %}
+        {{- '<|im_start|>system\n' + reasoning_instructions + '<|im_end|>\n' }}
+    {%- endif %}
+{%- endif %}
+{%- set _last_idx = _msgs | length - 1 %}
+{%- set ns = namespace(multi_step_tool=true, last_query_index=_last_idx) %}
+{%- for message in _msgs[::-1] %}
+    {%- set index = (_msgs | length - 1) - loop.index0 %}
+    {%- if ns.multi_step_tool and message.role == 'user' %}
+        {%- set _rc = render_content(message.content, false) | trim %}
+        {%- if not (_rc.startswith('<tool_response>') and _rc.endswith('</tool_response>')) %}
+            {%- set ns.multi_step_tool = false %}
+            {%- set ns.last_query_index = index %}
+        {%- endif %}
+    {%- endif %}
+{%- endfor %}
+{%- if ns.multi_step_tool %}
+    {%- if _last_idx > 50 %}
+        {%- set ns.last_query_index = _last_idx %}
+    {%- else %}
+        {%- set ns.last_query_index = 0 %}
+    {%- endif %}
+{%- endif %}
+{%- set ns2 = namespace(prev_role='', consecutive_failures=0) %}
+{%- for message in _msgs %}
+    {%- set is_system = (message.role == "system" or message.role == "developer") %}
+    {%- set content = render_content(message.content, true, is_system) | trim %}
+    {%- if is_system or message.role == 'user' %}
+        {%- if '<|think_off|>' in content %}{%- set content = content.split('<|think_off|>') | join('') | trim %}{%- endif %}
+        {%- if '<|think_on|>' in content %}{%- set content = content.split('<|think_on|>') | join('') | trim %}{%- endif %}
+        {%- if '<|think_xhigh|>' in content %}{%- set content = content.split('<|think_xhigh|>') | join('') | trim %}{%- endif %}
+        {%- if '<|think_high|>' in content %}{%- set content = content.split('<|think_high|>') | join('') | trim %}{%- endif %}
+        {%- if '<|think_ultracode|>' in content %}{%- set content = content.split('<|think_ultracode|>') | join('') | trim %}{%- endif %}
+        {%- if '<|think_extreme|>' in content %}{%- set content = content.split('<|think_extreme|>') | join('') | trim %}{%- endif %}
+        {%- if '<|think_max|>' in content %}{%- set content = content.split('<|think_max|>') | join('') | trim %}{%- endif %}
+        {%- if '<|think_medium|>' in content %}{%- set content = content.split('<|think_medium|>') | join('') | trim %}{%- endif %}
+        {%- if '<|think_low|>' in content %}{%- set content = content.split('<|think_low|>') | join('') | trim %}{%- endif %}
+        {%- if '<|think_minimal|>' in content %}{%- set content = content.split('<|think_minimal|>') | join('') | trim %}{%- endif %}
+    {%- endif %}
+    {%- if is_system %}
+        {{- '<|im_start|>system\n' + content + '<|im_end|>\n' }}
+    {%- elif message.role == 'user' %}
+        {%- set ns2.consecutive_failures = 0 %}
+        {{- '<|im_start|>user\n' + content + '<|im_end|>\n' }}
+    {%- elif message.role == 'assistant' %}
+        {%- set reasoning_content = '' %}
+        {%- set _explicit_reasoning = '' %}
+        {%- if message.reasoning_content is defined and message.reasoning_content is not none %}
+            {%- if message.reasoning_content is string %}
+                {%- set _explicit_reasoning = message.reasoning_content %}
+            {%- else %}
+                {%- set _explicit_reasoning = message.reasoning_content | string %}
+            {%- endif %}
+        {%- elif message.thinking is defined and message.thinking is not none %}
+            {%- if message.thinking is string %}
+                {%- set _explicit_reasoning = message.thinking %}
+            {%- else %}
+                {%- set _explicit_reasoning = message.thinking | string %}
+            {%- endif %}
+        {%- elif message.reasoning is defined and message.reasoning is not none %}
+            {%- if message.reasoning is string %}
+                {%- set _explicit_reasoning = message.reasoning %}
+            {%- else %}
+                {%- set _explicit_reasoning = message.reasoning | string %}
+            {%- endif %}
+        {%- endif %}
+        {%- if _explicit_reasoning %}
+            {%- set _lead_end = '' %}
+            {%- if content.startswith('<think>') and '</think>' in content %}
+                {%- set _lead_end = '</think>' %}
+            {%- elif content.startswith('<thinking>') and '</thinking>' in content %}
+                {%- set _lead_end = '</thinking>' %}
+            {%- elif content.startswith('</think>') %}
+                {%- set _lead_end = '</think>' %}
+            {%- elif content.startswith('</thinking>') %}
+                {%- set _lead_end = '</thinking>' %}
+            {%- endif %}
+            {%- if _lead_end %}
+                {%- set content = content.split(_lead_end)[-1].lstrip('\n') %}
+            {%- endif %}
+            {%- set reasoning_content = _explicit_reasoning %}
+        {%- else %}
+            {%- set _think_end = '' %}
+            {%- if content.startswith('</think>') %}
+                {%- set _think_end = '</think>' %}
+            {%- elif content.startswith('</thinking>') %}
+                {%- set _think_end = '</thinking>' %}
+            {%- elif '\n</think>' in content %}
+                {%- set _think_end = '\n</think>' %}
+            {%- elif '\n</thinking>' in content %}
+                {%- set _think_end = '\n</thinking>' %}
+            {%- elif '\n</ think>' in content %}
+                {%- set _think_end = '\n</ think>' %}
+            {%- elif '\n</think >' in content %}
+                {%- set _think_end = '\n</think >' %}
+            {%- elif content.startswith('<think>') and '</think>' in content %}
+                {%- set _think_end = '</think>' %}
+            {%- elif content.startswith('<thinking>') and '</thinking>' in content %}
+                {%- set _think_end = '</thinking>' %}
+            {%- endif %}
+            {%- if _think_end %}
+                {%- if 'thinking' in _think_end %}
+                    {%- set _think_start = '<thinking>' %}
+                {%- else %}
+                    {%- set _think_start = '<think>' %}
+                {%- endif %}
+                {%- set reasoning_content = content.split(_think_end)[0].rstrip('\n') %}
+                {%- if _think_start in reasoning_content %}
+                    {%- set reasoning_content = reasoning_content.split(_think_start)[-1].lstrip('\n') %}
+                {%- endif %}
+                {%- set content = content.split(_think_end)[-1].lstrip('\n') %}
+            {%- endif %}
+        {%- endif %}
+        {%- set reasoning_content = reasoning_content | trim %}
+        {%- if (_preserve_thinking or loop.index0 > ns.last_query_index) %}
+            {{- '<|im_start|>assistant\n<think>\n' + reasoning_content + '\n</think>\n\n' + content }}
+        {%- else %}
+            {{- '<|im_start|>assistant\n' + content }}
+        {%- endif %}
+        {%- if message.tool_calls is defined and message.tool_calls and message.tool_calls is iterable and message.tool_calls is not mapping %}
+            {%- for tool_call in message.tool_calls %}
+                {%- if tool_call.function is defined and tool_call.function is not none %}
+                    {%- set tc = tool_call.function %}
+                {%- else %}
+                    {%- set tc = tool_call %}
+                {%- endif %}
+                {%- set tc_name = tc.name if (tc.name is defined and tc.name is not none) else '' %}
+                {%- if _tool_format == 'json' %}
+                    {%- if loop.first %}
+                        {%- if content | trim %}
+                            {{- '\n\n' }}
+                        {%- endif %}
+                    {%- else %}
+                        {{- '\n' }}
+                    {%- endif %}
+                    {%- set _args = '{}' %}
+                    {%- if tc.arguments is defined and tc.arguments is not none %}
+                        {%- if tc.arguments is mapping %}
+                            {%- set _args = tc.arguments | tojson %}
+                        {%- elif tc.arguments is string %}
+                            {%- if tc.arguments %}
+                                {%- set _args = tc.arguments %}
+                            {%- endif %}
+                        {%- else %}
+                            {%- set _args = tc.arguments | tojson %}
+                        {%- endif %}
+                    {%- endif %}
+                    {{- '<tool_call>\n{"name": ' }}{{- tc_name | tojson }}{{- ', "arguments": ' }}{{- _args }}{{- '}\n</tool_call>' }}
+                {%- else %}
+                    {%- if loop.first %}
+                        {%- if content | trim %}
+                            {{- '\n\n<tool_call>\n<function=' + tc_name + '>\n' }}
+                        {%- else %}
+                            {{- '<tool_call>\n<function=' + tc_name + '>\n' }}
+                        {%- endif %}
+                    {%- else %}
+                        {{- '\n<tool_call>\n<function=' + tc_name + '>\n' }}
+                    {%- endif %}
+                    {%- if tc.arguments is defined and tc.arguments is not none %}
+                        {%- if tc.arguments is mapping %}
+                            {%- for args_name, args_value in tc.arguments.items() %}
+                                {{- '<parameter=' + args_name + '>\n' }}
+                                {%- if args_value is string %}
+                                    {%- set _av = args_value %}
+                                {%- else %}
+                                    {%- set _av = args_value | tojson %}
+                                {%- endif %}
+                                {%- if max_tool_arg_chars > 0 and _av | length > max_tool_arg_chars %}
+                                    {{- _av[:max_tool_arg_chars] + '\n[TRUNCATED - original length ' ~ (_av | length | string) ~ ' chars]' }}
+                                {%- else %}
+                                    {{- _av }}
+                                {%- endif %}
+                                {{- '\n</parameter>\n' }}
+                            {%- endfor %}
+                        {%- else %}
+                            {%- if tc.arguments is string %}
+                                {%- set _raw_args = tc.arguments %}
+                            {%- else %}
+                                {%- set _raw_args = tc.arguments | tojson %}
+                            {%- endif %}
+                            {%- if _raw_args %}
+                                {%- if max_tool_arg_chars > 0 and _raw_args | length > max_tool_arg_chars %}
+                                    {{- _raw_args[:max_tool_arg_chars] + '\n[TRUNCATED - original length ' ~ (_raw_args | length | string) ~ ' chars]' }}
+                                {%- else %}
+                                    {{- _raw_args }}
+                                {%- endif %}
+                            {%- endif %}
+                        {%- endif %}
+                    {%- endif %}
+                    {{- '</function>\n</tool_call>' }}
+                {%- endif %}
+            {%- endfor %}
+        {%- endif %}
+        {{- '<|im_end|>\n' }}
+    {%- elif message.role == 'tool' %}
+        {%- set _content_lower = content | lower %}
+        {%- set _content_head = _content_lower[:120] %}
+        {%- set _is_code_or_grep = ('throw new ' in _content_lower or 'throw error' in _content_lower or 'console.error' in _content_lower or 'logger.error' in _content_lower or 'logging.error' in _content_lower or 'import ' in _content_head or 'def ' in _content_head or 'function ' in _content_head) %}
+        {%- set _exit_code_zero = ('exit code: 0' in _content_head or 'process exited with code 0' in _content_head) %}
+        {%- set _error_field_ok = ('"error": null' in _content_head or '"error":null' in _content_head or '"error": false' in _content_head or '"error":false' in _content_head or '"error": ""' in _content_head or '"error":""' in _content_head) %}
+        {%- set _strong_error = (('"error":' in _content_head and not _error_field_ok) or '"status": "error"' in _content_head or '"status":"error"' in _content_head or 'traceback (most recent call last):' in _content_head or 'command not found' in _content_head or 'invalid syntax' in _content_head or 'fatal:' in _content_head or (('exit code: ' in _content_head or 'process exited with code' in _content_head) and not _exit_code_zero) or _content_head.startswith('exception:') or _content_head.startswith('failed to ')) %}
+        {%- set _weak_error = ('error:' in _content_head or 'err!' in _content_head) %}
+        {%- set _weak_suppressed = ('$ ' in _content_head or 'took ' in _content_head or content | length >= 600) %}
+        {%- if not _is_code_or_grep and (_strong_error or (_weak_error and not _weak_suppressed)) %}
+            {%- set ns2.consecutive_failures = ns2.consecutive_failures + 1 %}
+        {%- else %}
+            {%- set ns2.consecutive_failures = 0 %}
+        {%- endif %}
+        {%- if ns2.prev_role != 'tool' %}
+            {{- '<|im_start|>user' }}
+        {%- endif %}
+        {%- set _is_json_payload = (_tool_format == 'json' and content | trim | length > 0 and (content | trim)[:1] in ('{', '[')) %}
+        {%- if not _is_json_payload and max_tool_response_chars > 0 and content | length > max_tool_response_chars %}
+            {%- set content = content[:max_tool_response_chars] + '\n[TRUNCATED - original length ' ~ (content | length | string) ~ ' chars]' %}
+        {%- endif %}
+        {{- '\n<tool_response>\n' + content }}
+        {%- if ns2.consecutive_failures >= 2 %}
+            {{- '\n\n⚠️ SYSTEM WARNING: ' ~ ns2.consecutive_failures ~ ' consecutive tool errors detected. Your previous approach is incorrect. You MUST use a fundamentally different approach or corrected arguments.' }}
+        {%- elif ns2.consecutive_failures == 1 %}
+            {{- '\n\n⚠️ SYSTEM WARNING: The previous tool call returned an error. Diagnose the failure and retry with completely corrected arguments.' }}
+        {%- endif %}
+        {{- '\n</tool_response>' }}
+        {%- if loop.last %}
+            {{- '<|im_end|>\n' }}
+        {%- else %}
+            {%- set _next_role = _msgs[loop.index0 + 1].role %}
+            {%- if _next_role != 'tool' %}
+                {{- '<|im_end|>\n' }}
+            {%- endif %}
+        {%- endif %}
+    {%- else %}
+        {{- '<|im_start|>user\n[' + message.role + ']: ' + content + '<|im_end|>\n' }}
+    {%- endif %}
+    {%- set ns2.prev_role = message.role %}
+{%- endfor %}
+{%- if add_generation_prompt %}
+    {{- '<|im_start|>assistant\n' }}
+    {%- if not ns_state.thinking %}
+        {{- '<think>\n\n</think>\n\n' }}
+    {%- else %}
+        {{- '<think>\n' }}
+    {%- endif %}
+{%- endif %}
@@ -0,0 +1,105 @@
+# qwen4exp-swift15-w4dn8-i8-hc8m.recipe -- the Swift 1.5 finetune of Qwen3.8-Flash-Next
+# (ukisai/Swift1.5-Qwen3.8-Flash-Next, BF16 source, 335.3 GiB, key set identical to the official
+# base) as a radiance container for frank-primary: EVERY routed expert -- gate_up AND down --
+# through GPTQ into the w4nl64a8h table. The plugin's one-table-one-encoding law
+# ("one table of experts takes one encoding, and only a whole expert kept plain bf16 is served
+# beside it", qwen4exp_fp8 rad_arch_declare) forbids the original split tier (down block-fp8 +
+# gate_up w4): the served kernel is w4nl64a8h for the whole expert set, bf16 experts beside it.
+# The rest is qwen4exp-w5dn64-i8-hc8m.recipe verbatim: int8 trunk linears, block-fp8 hc mixers
+# and MTP fc, the fixed-scale n-gram table, the 2-bit draft head.
+#
+# Why down is the calibrated half: the calibration tap rides the down projection, so its GPTQ
+# rounding propagates against real down-input Hessians; gate_up shares the same w4 table
+# uncalibrated. The published w5dn64 comparison (mean KL 0.0789 -> 0.0717, top-1 92.85% ->
+# 93.34%, decode +4%, prefill -19%) is the 5-bit upper bound; this tier trades part of that
+# quality back for prefill.
+#
+#   CALIB=DIR rad-convert <Swift1.5-Qwen3.8-Flash-Next checkpoint> \
+#       --recipe scripts/recipes/qwen4exp-swift15-w4dn8-i8-hc8m.recipe -o qwen4exp-swift15-w4dn8-i8-hc8m.rad
+#
+# DIR is the calibration directory the bf16 container wrote (RADIANCE_CALIB_DIR, 10M tokens of
+# traffic-shaped corpus). Everything not named here is served as the checkpoint holds it (bf16:
+# embeddings, the vision tower, the QSA indexer); the first rule that matches a weight decides it.
+# The bf16 container is NOT passed with --reuse: the peak-disk arithmetic does not fit (checkpoint +
+# container + container + output > purge-freed space), and --reuse only saves a re-read.
+
+# The protected experts, cast to bf16 ahead of both expert rules. THE NAMES BELOW ARE THE BASE
+# CHECKPOINT'S (from its 1M-token recording, qwen4exp-w5dn64-i8-hc8m.recipe) -- a warm start only.
+# The Swift 1.5 finetune's routing distribution is not assumed equal: the derivation step rewrites
+# this section from Swift's OWN calibration recording before this recipe is ever run, by the base's
+# law -- an expert whose share of its layer's down-input energy is at least 3% and at least three
+# times its share of the routing, an even count on each rank's half of the layer (the four-bit GEMM
+# takes the rest as two equal tables by parity) by adding that half's next most energetic expert.
+# PROTECTED-BEGIN (rewritten from Swift's OWN grams by derive-protected-experts.py)
+blk.1.ffn_*_exps.271.weight         cast dtype=bf16
+blk.1.ffn_*_exps.355.weight         cast dtype=bf16
+blk.2.ffn_*_exps.279.weight         cast dtype=bf16
+blk.2.ffn_*_exps.413.weight         cast dtype=bf16
+blk.34.ffn_*_exps.146.weight         cast dtype=bf16
+blk.34.ffn_*_exps.153.weight         cast dtype=bf16
+blk.34.ffn_*_exps.407.weight         cast dtype=bf16
+blk.34.ffn_*_exps.453.weight         cast dtype=bf16
+blk.37.ffn_*_exps.288.weight         cast dtype=bf16
+blk.37.ffn_*_exps.377.weight         cast dtype=bf16
+blk.38.ffn_*_exps.406.weight         cast dtype=bf16
+blk.38.ffn_*_exps.420.weight         cast dtype=bf16
+blk.42.ffn_*_exps.37.weight         cast dtype=bf16
+blk.42.ffn_*_exps.90.weight         cast dtype=bf16
+blk.42.ffn_*_exps.302.weight         cast dtype=bf16
+blk.42.ffn_*_exps.388.weight         cast dtype=bf16
+blk.44.ffn_*_exps.315.weight         cast dtype=bf16
+blk.44.ffn_*_exps.350.weight         cast dtype=bf16
+blk.46.ffn_*_exps.290.weight         cast dtype=bf16
+blk.46.ffn_*_exps.419.weight         cast dtype=bf16
+blk.47.ffn_*_exps.119.weight         cast dtype=bf16
+blk.47.ffn_*_exps.143.weight         cast dtype=bf16
+blk.47.ffn_*_exps.399.weight         cast dtype=bf16
+blk.47.ffn_*_exps.445.weight         cast dtype=bf16
+# PROTECTED-END
+# (derive-protected-experts.py rewrote this section from Swift's own grams)
+
+# Every other routed expert, trunk and MTP layer alike: u4 codes into the w4nl table, rotated by
+# the 128-wide Hadamard, an E4M3 scale a group of 64 under the fixed 2^-13 searched for least
+# error as stored, chosen by GPTQ against $CALIB and swept three times by coordinate descent.
+# The calibration Grams are down-input Hessians, so down is the calibrated half of the table;
+# gate_up has no recording of its own and quantises uncalibrated (said on stderr, as the base
+# recipe says). The MTP layer has no recording either. Expert mass: gate_up 150.00 GiB -> ~37.6,
+# down 75.00 GiB -> ~18.8: ~56 GiB of experts, ~28 GiB a rank at TP=2 -- well inside the pool.
+blk.*.ffn_*_exps.*.weight            gptq table=w4nl group=64 scale=fp8_e4m3 scale2=f32 block2=*x* scale2_value=0.0001220703125 transform=fwht128 rule=search cd=3 calib=$CALIB
+
+# Every connection's two mixing matrices -- the 96 trunk connections, the final mixer and the MTP
+# head's three -- as E4M3 rows with an f32 scale a group of each row. A 128x128 block cannot cover
+# them: lowrank is 320, 2.5 blocks. Down is [320, 10240] and takes a scale every 128 columns; up is
+# [10240, 320] and takes four scales a row, 80 columns each.
+*_hc_down.weight                     rtn  codes=fp8_e4m3 group=128 scale=f32
+*_hc_up.weight                       rtn  codes=fp8_e4m3 group=80 scale=f32
+
+# The MTP head's fc_hidden and fc_embedding, the same rows. They only ever change the drafts.
+mtp.fc_hidden.weight                 rtn  codes=fp8_e4m3 group=128 scale=f32
+mtp.fc_embedding.weight              rtn  codes=fp8_e4m3 group=128 scale=f32
+
+# The trunk's block linears as int8, a scale per 128 columns of a row searched for least error.
+# The QSA indexer's projection is not one of them: its top-k is a hard choice between blocks, and
+# keeping it as the checkpoint holds it costs 20 MB.
+blk.*.attn_qg.weight                 rtn  codes=i8 group=128 scale=bf16 clamp=sym rule=search
+blk.*.attn_k.weight                  rtn  codes=i8 group=128 scale=bf16 clamp=sym rule=search
+blk.*.attn_v.weight                  rtn  codes=i8 group=128 scale=bf16 clamp=sym rule=search
+blk.*.attn_output.weight             rtn  codes=i8 group=128 scale=bf16 clamp=sym rule=search
+blk.*.ssm_inz.weight                 rtn  codes=i8 group=128 scale=bf16 clamp=sym rule=search
+blk.*.ssm_out.weight                 rtn  codes=i8 group=128 scale=bf16 clamp=sym rule=search
+blk.*.ffn_gate_up_shexp.weight       rtn  codes=i8 group=128 scale=bf16 clamp=sym rule=search
+blk.*.ffn_down_shexp.weight          rtn  codes=i8 group=128 scale=bf16 clamp=sym rule=search
+
+# The lm_head, int8 with a bf16 scale a row per 128 columns (logits_gemm_i8): at E4M3 it costs
+# ~0.6 points of top-1 on its own.
+output.weight                        rtn  codes=i8 group=128 scale=bf16 clamp=sym rule=search
+
+# The n-gram table at one fixed E4M3 scale; qwen4exp-w4.recipe says where the value comes from.
+# The value sits on a three-octave plateau (2^-13..2^-10, rel_l2 0.02662) with 1.7x headroom over
+# the base table's absmax; the prep phase verifies Swift's PLE absmax lands inside the plateau
+# before this rule is trusted.
+blk.*.ple_ngram.weight               rtn  codes=fp8_e4m3 block=*x* scale=bf16 rule=fixed scale_value=1.99317932128906e-4
+
+# The MTP head's 2-bit draft head. INERT for this checkpoint: the HF weight map has no
+# mtp.draft_head.weight (same as the base), kept for shape parity with the family.
+mtp.draft_head.weight                rtn  codes=u2 zero=u8 group=128 scale=f16
@@ -0,0 +1,116 @@
+#!/usr/bin/env bash
+# run-swift15-bf16.sh — RUNS ON DANKSERVER. Boot the no-recipe bf16 Swift1.5
+# container for the two offline campaign phases. The engine owns the GPUs, so
+# the production unit must be stopped first (this script does it and restores
+# on exit, autoresearch pattern: stop, never mask — relinked unit).
+#
+#   kld    --kld-record the reference logits from the bf16 model
+#          (runs the corpus and EXITS; engine writes DIR and leaves)
+#   calib  serve with the calibration tap; a separate driver
+#          (scripts/calib-push.py) pushes the corpus; the engine writes the
+#          Hessians at RADIANCE_CALIB_TOKENS and the tap switches itself off.
+#          Stop this script after calib-push prints TAP_DONE.
+#
+# Shape = the vendor's bf16 calibration lines (GUIDE.md §4.3:528 + MOE-W4.md
+# "calibrate the checkpoint as it ships"): expert_tiered + weights-disk-tier,
+# no speculation, deterministic. Large prefill steps because a disk-tier step
+# costs the same at 1 token as at 32K. The container declared --max-ctx 4096;
+# --max-model-len 16384 exceeds it on purpose (the KLD prompts run to 12,007
+# tokens, calib to 8,242) — the boot ledger arbitrates the QSA buffers; if
+# declare refuses, re-convert with a wider --max-ctx.
+set -uo pipefail
+
+MODE="${1:?usage: run-swift15-bf16.sh kld|calib}"
+PIN_1013=sha256:4c3f484564bb2872784e46b0a59cbd71becc9c1e746e301a3596ffae253c322e
+PIN_120=sha256:665775ed1d90de6dbb25558e07fc60d20328f1346f96f3c6e00845eb763f01fa
+RAD=/home/dank/models/rad/qwen4exp-swift15-bf16.rad
+C=swift15ref
+LOG=/home/dank/logs/swift15_bf16.log
+KLD_REF_DIR=/home/dank/corpora/kld-ref
+CALIB_DIR=/home/dank/corpora/calib
+KLD_CORPUS=/home/dank/corpora/out/kld.jsonl
+
+restore() { sudo systemctl start frank-primary >/dev/null 2>&1 || true; }
+trap restore EXIT
+sudo systemctl stop frank-primary >/dev/null 2>&1 || true
+podman rm -f "$C" >/dev/null 2>&1 || true
+
+ARGS=(--model /models/rad/qwen4exp-swift15-bf16.rad
+      --tp 2 --placement expert_tiered --weights-disk-tier
+      --host-pool-mib 12288 --deterministic --gpu-headroom-mib 768
+      --num-speculative-tokens 0
+      --kv-cache-dtype fp8 --max-model-len 16384
+      --ngram-placement ram
+      --override-chat-template /models/tpl.jinja)
+
+case "$MODE" in
+  kld)
+    # 1.0.13: the engine that will serve the candidate — the reference is
+    # recorded on the same engine that scores it. The weights slab DERIVES as
+    # card - headroom - kv, and the disk tier refuses under 9.57 GiB (4.21 of
+    # layer buffers + 5.36 fixed), so kv is explicit SMALL: recording is
+    # prefill-only, self-batched at MNBT 32768 = two 16k prompts at a time;
+    # 4 slots x 16k fp8 (29 KiB/pos, 1.7 GiB) + headroom fits 3072.
+    IMAGE="stilldeadcode/radiance@$PIN_1013"
+    mkdir -p "$KLD_REF_DIR"
+    ARGS+=(--kld-record /data/kld-ref --kld-corpus /data/kld.jsonl
+           --vram-kv-mib 3072
+           --max-num-seqs 4 --max-num-batched-tokens 32768)
+    ;;
+  calib)
+    # 1.2.0: the calibration tap (RADIANCE_CALIB_DIR + RADIANCE_CALIB_TOKENS)
+    # lives in the 1.2.0 qwen4exp plugin; 1.0.13 has only the DIR half with an
+    # unparameterisable threshold. Grams are bf16 activations — engine-invariant.
+    # Same KV ceiling as kld: the disk-tier weights floor (9.57 GiB) plus
+    # ~18 GiB of fixed per-rank claims cap the pool near 3.5 GiB on a 32 GiB
+    # card — pool minus the 2.17 GiB fixed holds ~237k fp8 tokens a rank, so
+    # the vendor's big-step calibration shape is affordable: MOE-W4.md:244
+    # measures 640 tok/s at a 32k step against 930 at 55k, and the doc's own
+    # run drives MNBT 65536. 8 slots x 16k = 131k resident tokens fits the
+    # pool; the step batches 65k. KLD's observed rate confirmed step size,
+    # not slot count, is the throughput lever.
+    IMAGE="stilldeadcode/radiance@$PIN_120"
+    mkdir -p "$CALIB_DIR"
+    ARGS+=(--port 7770 --vram-kv-mib 3072
+           --max-num-seqs 8 --max-num-batched-tokens 65536
+           --prefix-cache-dir /data/calib-kvcache)
+    ;;
+  probe)
+    IMAGE="stilldeadcode/radiance@$PIN_1013"
+    ARGS+=(--port 7770 --max-num-seqs 8 --max-num-batched-tokens 4096)
+    ;;
+  *) echo "unknown mode: $MODE" >&2; exit 2 ;;
+esac
+
+# The prod template copy, readable from the container: mount the models tree
+# ro plus the state dir rw (KLD record + prefix tier land there).
+cp -f /home/dank/serverAppConfigs/scripts/qwen38-froggeric-v22.5.jinja \
+      /home/dank/models/tpl.jinja 2>/dev/null || true
+
+KLD_MOUNT=()
+if [ "$MODE" = kld ]; then
+    # Nested under the /data mount, so these must come after it (line order
+    # in podman run below): the corpus read-only, the record dir writable.
+    KLD_MOUNT=(-v "$KLD_CORPUS:/data/kld.jsonl:ro"
+               -v "$KLD_REF_DIR:/data/kld-ref")
+fi
+EXTRA_ENV=()
+if [ "$MODE" = calib ]; then
+    EXTRA_ENV=(-v "$CALIB_DIR:/data/calib"
+               -e RADIANCE_CALIB_DIR=/data/calib
+               -e RADIANCE_CALIB_TOKENS=10000000)
+fi
+
+podman run --rm --name "$C" \
+    --network host --init \
+    --security-opt seccomp=unconfined \
+    --ulimit memlock=-1:-1 \
+    --device /dev/kfd --device /dev/dri \
+    -v /home/dank/models:/models:ro \
+    -v /home/dank/radiance-state:/data \
+    "${KLD_MOUNT[@]}" \
+    "${EXTRA_ENV[@]}" \
+    "$IMAGE" "${ARGS[@]}" 2>&1 | tee "$LOG"
+rc=${PIPESTATUS[0]}
+echo "swift15-bf16 $MODE exited rc=$rc; log: $LOG"
+exit "$rc"
@@ -1,121 +1,200 @@
 #!/usr/bin/env bash
-# Serve Swift-1.5-Qwen3.8-27B-Uncensored (abliterated) as frank-large on
-# Dankclaw's Strix Halo iGPU (gfx1151), using gufo (gufo-org/gufo) — the
-# inference engine built specifically for this hardware.
+# serve-frank-large.sh — frank-large endpoint: halogen-flash-server 0.16.2
+# (closed-source engine, image ghcr.io/peonist-ai/halogen-flash-server,
+# digest-pinned in frank-large.env) serving the Quat3rnion abliterated
+# Swift1.5 finetune of peonist's Qwen3.8-Flash-Next v2 checkpoint on
+# Dankclaw's Strix Halo iGPU, host :7770. 2026-10-06 cutover from the
+# halogen 27B-dense shape — revert the swap commit's trio to roll back
+# (the 27B payload stays on disk).
 #
-# Runtime: /home/dank/GitPackages/gufo-v0.1.1/bin/gufo, built from source
-# against the host ROCm by scripts/build-gufo.sh. It replaced the stock
-# llama.cpp prebuilt (unsloth b11030) on 2026-09-28; port, alias and client
-# contract did not move.
+# The engine is driven by `podman run` DIRECTLY (the fleet's frank-primary
+# pattern): the image starts engine + OpenAI frontend in one container.
+# The frontend listens on 8731 inside (HALOGEN_API_PORT default) and is
+# published as host 7770; the engine's own token port (8730, no auth)
+# stays unpublished on the container's loopback. HALOGEN_CHECKPOINT
+# selects the file inside the /models mount; the n-gram lookup table
+# (HALOGEN_NGRAM_TABLE, the card sets it explicitly though the documented
+# fallback is "beside the checkpoint") and the vision tower
+# (HALOGEN_VISION_TOWER) ride in the same directory.
 #
-# Payload: ajgazin/Swift-1.5-Qwen3.8-27B-Uncensored-Dynamic-MTP-GGUF's
-# UD-Q8_K_XL, fetched by scripts/download-frank-large-swift.sh. It replaced
-# unsloth/Qwen3.8-27B-GGUF:UD-Q8_K_XL on 2026-09-28. The two are structurally
-# interchangeable for this engine — same `qwen35` architecture, the same 866
-# tensors in the same order, the same per-tensor Q8_0/F32/BF16 layout, the
-# same MTP head at blk.64 — so the swap needed no engine change. The fine-tune
-# is abliterated (uncensored) and carries the SWIFT OPEN LICENSE 1.0, which
-# is not an OSI licence.
+# Gates mirror the deployer EXACTLY (the 2026-10-01 crash-loop lesson —
+# serve gate and deploy gate disagreeing = green deploy, red unit):
+# checkpoint/n-gram/vision byte-exact, tokenizer directory present with
+# every named file a FLAT REGULAR FILE (a HF cache snapshot's symlinks
+# into blobs/ dangle inside the mount), GPU device nodes present.
 #
-# Shape: one session, full 262144 context (the model's native window). The
-# 27B dense weights (29.3 GiB) + the DFlash2 drafter (1.07 GiB) + KV +
-# compute buffers fit the iGPU's GTT aperture (~114 GiB); a load at this
-# context measured 66.5 GiB of the 116.7 GiB device.
-#
-# Spec method: DFlash2 (a block-diffusion drafter), gufo's adaptive block
-# policy, capped at DRAFT_TOKENS proposals per step. The drafter is the
-# z-lab Qwen3.8-27B-DFlash2-Q4_K_M and it has a sliding window of 2048
-# (small draft KV). It is UNCHANGED by the model swap and still lives beside
-# the previous target, which is why DRAFTER_DIR is its own tunable.
-#
-# Sampling: greedy (--temperature 0) server-side, as every frank-large has
-# served. The harness overrides it per request via extraBody (1.0 / 0.95 /
-# top-k 20).
-#
-# Template: gufo compiles the official Qwen Jinja into the binary and takes
-# NO --chat-template-file — and it is not a Jinja engine at all
-# (src/models/qwen/chat_template.cpp is a hand-written C++ renderer), so the
-# fleet's scripts/qwen-sharp-v22.4.0.jinja cannot be supplied to it. That law
-# was given up when this endpoint moved to gufo on 2026-09-28, by decision:
-# the terse system-prompt injection, the in-band <|think_*|> control, the
-# json|xml tool_call_format switch and the tool-argument truncation are all
-# absent. gufo's own renderer keeps the effort instruction strings and
-# preserve_thinking, which are the parts the harness depends on. Accepted
-# 2026-09-28; recovering the terse prompt from the client side is the
-# cheapest partial restore.
-#
-# Text-only: gufo auto-discovers a mmproj beside the model and this payload
-# has none (the base repo ships one separately, unused), so the endpoint
-# stays text-only.
-#
-# Tunables come from frank-large.env unless FRANK_LARGE_ENV_FILE overrides
-# it; anything not set there falls through to the defaults below.
+# Why containers instead of a bare binary: the payload is .hgn and only
+# the flash-server line reads it; it ships as a container; the root unit
+# keeps LimitMEMLOCK=infinity for parity and to outlive the login session,
+# exactly like frank-primary's.
 set -euo pipefail

-SCRIPT_DIR="$(cd -- "$(dirname -- "${BASH_SOURCE[0]}")" && pwd)"
+SCRIPT_DIR="$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)"
 ENV_FILE="${FRANK_LARGE_ENV_FILE:-$SCRIPT_DIR/frank-large.env}"
-if [[ -f "$ENV_FILE" ]]; then
-    set -a
-    source "$ENV_FILE"
-    set +a
-fi
-
-# --- defaults (used when the env file does not set them) ------------------
-: "${RUNTIME_DIR:=/home/dank/GitPackages/gufo-v0.1.1}"
-: "${MODEL_DIR:=/home/dank/models/Swift-1.5-Qwen3.8-27B-Uncensored-MTP-UD-Q8_K_XL}"
-: "${ALIAS:=frank-large}"
-: "${HOST:=0.0.0.0}"
-: "${PORT:=7770}"
-: "${CONTEXT_SIZE:=262144}"
-: "${SESSIONS:=1}"
-: "${DRAFT_TOKENS:=7}"
-: "${DRAFT_POLICY:=adaptive}"
-
-GUFO_BIN="$RUNTIME_DIR/bin/gufo"
-
-# The target GGUF and the drafter are named explicitly because they do not
-# share a directory any more: the target is a different repository's artifact
-# and keeps its own long filename, while the DFlash2 drafter is unchanged and
-# still lives beside the previous target. MODEL_FILE/DRAFTER_FILE default to
-# the deployed payload, so the env file only has to state them when it moves.
-: "${MODEL_FILE:=$MODEL_DIR/Swift-1.5-Qwen3.8-27B-Uncensored-Dynamic-MTP-UD-Q8_K_XL.gguf}"
-: "${DRAFTER_DIR:=/home/dank/models/Qwen3.8-27B-UD-Q8_K_XL}"
-: "${DRAFTER_FILE:=$DRAFTER_DIR/Qwen3.8-27B-DFlash2-Q4_K_M.gguf}"
-
-if [[ ! -x "$GUFO_BIN" ]]; then
-    echo "runtime not found at $GUFO_BIN" >&2
-    echo "frank-large: build it: bash scripts/build-gufo.sh" >&2
+if [ ! -f "$ENV_FILE" ]; then
+    echo "frank-large: env file not found: $ENV_FILE" >&2
    exit 1
 fi
-for f in "$MODEL_FILE" "$DRAFTER_FILE"; do
-    if [[ ! -f "$f" ]]; then
-        echo "missing asset $f (run the matching scripts/download-frank-large*.sh)" >&2
+set -a
+# shellcheck disable=SC1090
+source "$ENV_FILE"
+set +a
+
+: "${FRANK_LARGE_IMAGE:?FRANK_LARGE_IMAGE must name the digest-pinned image}"
+: "${FRANK_LARGE_CONTAINER:?FRANK_LARGE_CONTAINER must name the container}"
+: "${FRANK_LARGE_PUBLISH:?FRANK_LARGE_PUBLISH must be host:container}"
+: "${FRANK_LARGE_MODEL_DIR:?FRANK_LARGE_MODEL_DIR must name the payload dir}"
+: "${FRANK_LARGE_CHECKPOINT:?FRANK_LARGE_CHECKPOINT must name the .hgn inside it}"
+: "${FRANK_LARGE_CHECKPOINT_BYTES:?FRANK_LARGE_CHECKPOINT_BYTES must pin the size}"
+: "${FRANK_LARGE_NGRAM_TABLE:?FRANK_LARGE_NGRAM_TABLE must name the lookup table}"
+: "${FRANK_LARGE_NGRAM_BYTES:?FRANK_LARGE_NGRAM_BYTES must pin the table size}"
+: "${FRANK_LARGE_VISION_TOWER:?FRANK_LARGE_VISION_TOWER must name the tower}"
+: "${FRANK_LARGE_VISION_BYTES:?FRANK_LARGE_VISION_BYTES must pin the tower size}"
+: "${FRANK_LARGE_TOKENIZER_FILES:?FRANK_LARGE_TOKENIZER_FILES must list the tokenizer dir contents}"
+: "${HALOGEN_MODEL_ID:?HALOGEN_MODEL_ID must be the served alias (frank-dragoon)}"
+: "${HALOGEN_CTX:?HALOGEN_CTX must be set (per-slot window)}"
+: "${HALOGEN_KV_POOL_POSITIONS:?HALOGEN_KV_POOL_POSITIONS must be set}"
+: "${HALOGEN_KV_SLOTS:?HALOGEN_KV_SLOTS must be set}"
+: "${HALOGEN_SPEC_ADAPT:?HALOGEN_SPEC_ADAPT must be set (0 = head always on)}"
+
+# Capture podman's output BEFORE grepping: under `set -o pipefail` the
+# `podman images | grep -q` pattern races (grep exits on first match,
+# podman dies SIGPIPE 141, pipefail turns the 141 into "no match" — the
+# 2026-09-30 deploy refusal).
+DIGESTS="$(podman images --format '{{.RepoDigests}}')"
+if ! grep -qF "${FRANK_LARGE_IMAGE#*@}" <<<"$DIGESTS"; then
+    echo "frank-large: image $FRANK_LARGE_IMAGE not in podman storage;" >&2
+    echo "  pull it: podman pull ${FRANK_LARGE_IMAGE%@*}" >&2
+    exit 1
+fi
+
+# Size-exact gates on all three payload files: a truncated curl exits 0 on
+# some failures and a partial .hgn loads far enough to look alive, so the
+# byte counts from the Hub tree API are the contract (fleet fetch law).
+ck() {
+    # ck <env-name> <path> <expected-bytes>
+    if [ ! -f "$2" ]; then
+        echo "frank-large: $1 $2 missing;" >&2
+        echo "  fetch it: bash scripts/download-frank-large-flashnext.sh" >&2
+        exit 1
+    fi
+    local sz
+    sz="$(stat -c %s "$2")"
+    if [ "$sz" != "$3" ]; then
+        echo "frank-large: $1 is $sz bytes, expected $3 (partial download?)" >&2
+        exit 1
+    fi
+}
+ck checkpoint "$FRANK_LARGE_MODEL_DIR/$FRANK_LARGE_CHECKPOINT" "$FRANK_LARGE_CHECKPOINT_BYTES"
+ck n-gram-table "$FRANK_LARGE_MODEL_DIR/$FRANK_LARGE_NGRAM_TABLE" "$FRANK_LARGE_NGRAM_BYTES"
+ck vision-tower "$FRANK_LARGE_MODEL_DIR/$FRANK_LARGE_VISION_TOWER" "$FRANK_LARGE_VISION_BYTES"
+
+# The payload ships the tokenizer as a FLAT DIRECTORY. A HuggingFace cache
+# snapshot symlinks each file into a sibling blobs/, and those symlinks
+# dangle inside the container mount — so the directory must be real and
+# every named file must be a regular file, not a link.
+TOK_DIR="$FRANK_LARGE_MODEL_DIR/tokenizer"
+if [ -L "$TOK_DIR" ] || [ ! -d "$TOK_DIR" ]; then
+    echo "frank-large: tokenizer dir $TOK_DIR missing or a symlink (HF cache snapshot?);" >&2
+    echo "  fetch it: bash scripts/download-frank-large-flashnext.sh" >&2
+    exit 1
+fi
+for TF in $FRANK_LARGE_TOKENIZER_FILES; do
+    if [ ! -f "$TOK_DIR/$TF" ] || [ -L "$TOK_DIR/$TF" ]; then
+        echo "frank-large: tokenizer file $TOK_DIR/$TF missing or a symlink;" >&2
+        echo "  fetch it: bash scripts/download-frank-large-flashnext.sh" >&2
        exit 1
    fi
 done

-# --- env -------------------------------------------------------------------
-# gufo links against the host ROCm through its RPATH (/opt/rocm ->
-# /etc/alternatives/rocm), so no LD_LIBRARY_PATH is needed. The device is
-# already gfx1151 — `gufo diagnose` reports it natively — so no
-# HSA_OVERRIDE_GFX_VERSION either; the override existed for the prebuilt
-# llama.cpp runtime and can only misreport the arch here.
-export HIP_VISIBLE_DEVICES=0
+# GPU access is mandatory but the node paths are variables so the seam can
+# drive the gate on a box without an AMD GPU.
+DEV_KFD="${FRANK_LARGE_DEV_KFD:-/dev/kfd}"
+DEV_DRI="${FRANK_LARGE_DEV_DRI:-/dev/dri}"
+if [ ! -e "$DEV_KFD" ] || [ ! -d "$DEV_DRI" ]; then
+    echo "frank-large: $DEV_KFD or $DEV_DRI missing — no gfx1151 to serve on" >&2
+    exit 1
+fi
+# Host-RAM preflight — the engine's pin gate, checked BEFORE podman run.
+# The engine refuses to pin when MemAvailable is below the checkpoint plus
+# a 16 GiB floor: pinned pages cannot be swapped, and under the floor the
+# n-gram lookup pages in from disk on every long prompt. A RAM-short host
+# (a co-tenant spike, a sibling engine dying) crash-loops the unit — each
+# boot dies in ~5 s at the pin gate — until the RAM clears. The 2026-10-06
+# cutover hit this: 8 boots, MemAvailable 74.77 GiB vs the 78.11 GiB needed,
+# self-resolved in ~2 min when the dying 27B engine freed RAM. Check here and
+# WAIT for the RAM to clear, so a transient dip boots once instead of
+# crash-looping; if it never clears, refuse loudly.
+MEMINFO="${FRANK_LARGE_MEMINFO:-/proc/meminfo}"
+PIN_KB=$(( FRANK_LARGE_CHECKPOINT_BYTES / 1024 ))
+NEED_KB=$(( PIN_KB + 16 * 1048576 ))
+ram_avail_kb() { awk '/^MemAvailable:/{print $2; exit}' "$MEMINFO"; }
+MEMAVAIL_KB="$(ram_avail_kb)" || MEMAVAIL_KB=""
+if [ -n "$MEMAVAIL_KB" ] && [ "$MEMAVAIL_KB" -lt "$NEED_KB" ]; then
+    WAIT_S="${FRANK_LARGE_RAM_WAIT:-600}"
+    GIB() { awk -v kb="$1" 'BEGIN{printf "%.2f", kb/1048576}'; }
+    echo "frank-large: host RAM below the pin gate — MemAvailable $(GIB "$MEMAVAIL_KB") GiB < needed $(GIB "$NEED_KB") GiB ($(GIB "$PIN_KB") GiB checkpoint + 16.00 GiB floor); waiting up to ${WAIT_S}s for host RAM to clear" >&2
+    deadline=$(( $(date +%s) + WAIT_S ))
+    while [ "$(date +%s)" -lt "$deadline" ]; do
+        sleep 5
+        MEMAVAIL_KB="$(ram_avail_kb)" || MEMAVAIL_KB=""
+        if [ -n "$MEMAVAIL_KB" ] && [ "$MEMAVAIL_KB" -ge "$NEED_KB" ]; then
+            break
+        fi
+    done
+    if [ -z "$MEMAVAIL_KB" ] || [ "$MEMAVAIL_KB" -lt "$NEED_KB" ]; then
+        echo "frank-large: MemAvailable $(GIB "${MEMAVAIL_KB:-0}") GiB after ${WAIT_S}s; refusing to start (the engine would refuse the pin)" >&2
+        exit 1
+    fi
+    echo "frank-large: MemAvailable cleared to $(GIB "$MEMAVAIL_KB") GiB; starting" >&2
+fi
+for CTR in ${FRANK_LARGE_LEGACY_CONTAINERS:-} "$FRANK_LARGE_CONTAINER"; do
+    podman rm -f "$CTR" >/dev/null 2>&1 || true
+done

-# --- the serve command ---------------------------------------------------
-# Server options precede the subcommand, model options follow it.
-exec "$GUFO_BIN" serve \
-    --host "$HOST" \
-    --port "$PORT" \
-    --sessions "$SESSIONS" \
-    llm \
-    --model "$MODEL_FILE" \
-    --served-model-name "$ALIAS" \
-    --context "$CONTEXT_SIZE" \
-    --speculative dflash2 \
-    --dflash-model "$DRAFTER_FILE" \
-    --draft-tokens "$DRAFT_TOKENS" \
-    --draft-policy "$DRAFT_POLICY" \
-    --temperature 0 \
-    --think auto \
-    --preserve-thinking on
+LOG="${FRANK_LARGE_LOG:-/home/dank/logs/frank_large.log}"
+mkdir -p "$(dirname "$LOG")"
+# --ipc=host for the udev-granted supplementary group, seccomp unconfined
+# for the ROCm ioctl surface, and unbounded memlock (the unit's
+# LimitMEMLOCK=infinity is inherited, so runc never has to RAISE it). NO
+# --ulimit nofile: Dankclaw's hard limit is 524288, rootless runc cannot
+# raise a hard limit and dies with "error setting rlimit type 7:
+# operation not permitted" (measured 2026-10-05 cutover); the model card
+# and both sibling serve scripts pass no nofile either. No comment lines
+# inside the backslash chain — a mid-chain # terminates the continuation
+# (the 2026-10-05 exit-127 crash loop on the sibling serve script).
+podman run --name "$FRANK_LARGE_CONTAINER" \
+  --publish "$FRANK_LARGE_PUBLISH" \
+  --ipc=host \
+  --ulimit memlock=-1:-1 \
+  --device "$DEV_KFD" --device "$DEV_DRI" \
+  --group-add keep-groups \
+  --security-opt seccomp=unconfined \
+  -e HALOGEN_MODEL_ID="$HALOGEN_MODEL_ID" \
+  -e HALOGEN_CHECKPOINT="/models/$FRANK_LARGE_CHECKPOINT" \
+  -e HALOGEN_NGRAM_TABLE="/models/$FRANK_LARGE_NGRAM_TABLE" \
+  -e HALOGEN_VISION_TOWER="/models/$FRANK_LARGE_VISION_TOWER" \
+  -e HALOGEN_TOKENIZER=/tokenizer \
+  -e HALOGEN_CTX="$HALOGEN_CTX" \
+  -e HALOGEN_KV_POOL_POSITIONS="$HALOGEN_KV_POOL_POSITIONS" \
+  -e HALOGEN_KV_SLOTS="$HALOGEN_KV_SLOTS" \
+  -e HALOGEN_HOST_RESERVE_GIB="${HALOGEN_HOST_RESERVE_GIB:-20}" \
+  -e HALOGEN_SPEC_ADAPT="$HALOGEN_SPEC_ADAPT" \
+  -e HALOGEN_MAX_TOK="${HALOGEN_MAX_TOK:-8192}" \
+  -e HALOGEN_MAX_TOKENS_CAP="${HALOGEN_MAX_TOKENS_CAP:-65536}" \
+  -e HALOGEN_QUEUE_TIMEOUT="${HALOGEN_QUEUE_TIMEOUT:-7200}" \
+  -e HALOGEN_TEMPERATURE="${HALOGEN_TEMPERATURE:-1.0}" \
+  -e HALOGEN_TOP_P="${HALOGEN_TOP_P:-0.95}" \
+  -e HALOGEN_TOP_K="${HALOGEN_TOP_K:-20}" \
+  ${HALOGEN_WEIGHTS_LOCK:+-e HALOGEN_WEIGHTS_LOCK="$HALOGEN_WEIGHTS_LOCK"} \
+  -v "$FRANK_LARGE_MODEL_DIR:/models:ro" \
+  -v "$FRANK_LARGE_MODEL_DIR/tokenizer:/tokenizer:ro" \
+  "$FRANK_LARGE_IMAGE" \
+  > "$LOG" 2>&1
+
+# FOREGROUND `podman run`: the engine's stdout/stderr are this script's
+# output, so the boot ledger (kv pool, prompt cache, MTP head, any clamp)
+# lands in $LOG as documented. podman run returns the container's exit
+# code, so a crash is a nonzero ExecStart and the unit's Restart=always
+# stays real; systemd stop goes through the unit's ExecStop and the run
+# client forwards SIGTERM to the engine, which drains on it.
@@ -1,34 +1,33 @@
 #!/usr/bin/env bash
-# serve-frank-primary.sh — frank-primary endpoint (tcclaviger RDNA4 vLLM fork,
-# orcarouter Qwen3.8-Flash-Next-Uncensored-FP8 + MTP4 spec decode, 2x R9700,
-# dankserver :7770; 2026-09-29 cutover from the R9V radiance fork — git tag
-# frank-primary-r9v-q8-20260929 is the pre-cutover state).
+# serve-frank-primary.sh — frank-primary endpoint: radiance (StillDeadcode,
+# Apache-2.0) serving the OFFICIAL Qwen/Qwen3.8-Flash-Next-FP8 checkpoint at
+# NATIVE block-FP8 precision from a rad-convert container
+# (/home/dank/models/rad/qwen4exp-fp8-native.rad), on BOTH R9700s (TP=2,
+# wht6 wire), dankserver :7770. 2026-10-05 cutover from FreeToken (PR #587,
+# one card; git tag frank-primary-freetoken-pr587-20261005 is the
+# pre-cutover state).
 #
-# The engine is driven by `podman run` DIRECTLY: the image's entrypoint execs
-# `vllm serve "$@"`, so the run line below IS the engine argv. This replaces
-# the R9V launcher/shim machinery (that fork needed a docker->podman shim and
-# a 104-key on-host setup state; this stack is fully expressed by flags, and
-# every flag here was qualified by boot probe on 2026-09-29 — see
-# frank-primary.env for the qualification record).
-#
-# Shape: MAXLEN 200000 x 3 sessions (600k total), KV q8/q8 (--kv-cache-dtype
-# fp8: e4m3 for both K and V — the stock flag quantizes them together, which
-# is the requested q8/q8), KV_BYTES 7.2 GiB PER RANK (the engine measured
-# 729,545 tokens of capacity / 3.65x concurrency at 200k per request), MTP
-# depth 4, prefix caching, expert prefetch OFF (telemetry measured it at 36%
-# precision — the tuned incumbent deletes it), TunableOp off.
+# The run line below IS the engine argv: the image's entrypoint is the
+# engine and its tools sit beside it on PATH. Unlike every predecessor
+# there is no per-boot VRAM planner to second-guess: the boot ledger in the
+# log states every budget before anything is allocated (weights vs KV
+# split, the expert host pool's coverage of the worst case, the PLE pin,
+# the P2P answer per pair) — read it after every resize. First-boot shape:
+# 2 decode slots (Frank's call), 40 GiB/rank pinned expert pool, fp8 KV,
+# MTP depth 3, prefix cache host 4 GiB + disk 64 GiB, PLE pinned in RAM.
 #
 # The root unit (scripts/frank-primary.service, User=dank,
-# LimitMEMLOCK=infinity) grants the pin budget for the tiered-expert host
-# slabs (~20.5 GiB/rank of pinned anonymous memory). A plain shell session
-# also works, but the unit is the fleet-parity boot-persistent shape.
+# LimitMEMLOCK=infinity) grants the pin budget: 2 x 40 GiB expert pools +
+# the 47.7 GiB PLE table + 4 GiB prefix host tier — ~132 GiB of pinned
+# pages, which is ALSO why this script refuses to start when ttm's
+# pages_limit/pages_max cap is under FRANK_PRIMARY_TTM_MIN_MIB (see below).
 set -euo pipefail

 SCRIPT_DIR="$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)"
 ENV_FILE="${FRANK_PRIMARY_ENV_FILE:-$SCRIPT_DIR/frank-primary.env}"

 if [ ! -f "$ENV_FILE" ]; then
-    echo "serve-frank-primary: env file missing: $ENV_FILE" >&2
+    echo "ERROR: frank-primary env file absent: $ENV_FILE" >&2
    exit 1
 fi
 set -a
@@ -36,52 +35,120 @@ set -a
 source "$ENV_FILE"
 set +a

-: "${FRANK_PRIMARY_IMAGE:?FRANK_PRIMARY_IMAGE must name the digest-pinned image}"
-: "${FRANK_PRIMARY_MODEL_DIR:?FRANK_PRIMARY_MODEL_DIR must name the FP8 payload}"
-: "${FRANK_PRIMARY_MODEL_SHARDS:?FRANK_PRIMARY_MODEL_SHARDS must give the shard count}"
+: "${FRANK_PRIMARY_IMAGE:?FRANK_PRIMARY_IMAGE must name the radiance image}"
+: "${FRANK_PRIMARY_MODEL_DIR:?FRANK_PRIMARY_MODEL_DIR must name the .rad dir}"
+: "${FRANK_PRIMARY_MODEL_FILE:?FRANK_PRIMARY_MODEL_FILE must name the .rad file}"
+: "${FRANK_PRIMARY_STATE_DIR:?FRANK_PRIMARY_STATE_DIR must name the /data state dir}"
 : "${FRANK_PRIMARY_CONTAINER:?FRANK_PRIMARY_CONTAINER must name the container}"
 : "${FRANK_PRIMARY_SERVED_NAME:?FRANK_PRIMARY_SERVED_NAME must name the alias}"
-: "${FRANK_PRIMARY_PUBLISH:?FRANK_PRIMARY_PUBLISH must be host:container}"
-: "${FRANK_PRIMARY_MAXLEN:?FRANK_PRIMARY_MAXLEN must be set}"
-: "${FRANK_PRIMARY_SEQS:?FRANK_PRIMARY_SEQS must be set}"
-: "${FRANK_PRIMARY_KV_BYTES:?FRANK_PRIMARY_KV_BYTES must be set (PER RANK)}"
-: "${FRANK_PRIMARY_KV_DTYPE:?FRANK_PRIMARY_KV_DTYPE must be set (fp8 = q8/q8 KV)}"
-# Capture podman's output BEFORE grepping: under `set -o pipefail`, the
-# classic `podman images | grep -q` pattern races — grep exits on its first
-# match, podman dies SIGPIPE (141) writing the remaining lines, and pipefail
-# turns the 141 into "no match" (the 2026-09-30 deploy refusal).
-DIGESTS="$(podman images --format '{{.RepoDigests}}')"
-if ! grep -qF "${FRANK_PRIMARY_IMAGE#*@}" <<<"$DIGESTS"; then
-    echo "serve-frank-primary: pinned image absent from rootless storage: $FRANK_PRIMARY_IMAGE" >&2
-    echo "  fetch it: podman pull $FRANK_PRIMARY_IMAGE" >&2
+: "${FRANK_PRIMARY_PORT:?FRANK_PRIMARY_PORT must be the port}"
+: "${FRANK_PRIMARY_TEMPLATE:?FRANK_PRIMARY_TEMPLATE must name the chat template}"
+: "${FRANK_PRIMARY_TP:?FRANK_PRIMARY_TP must be set}"
+: "${FRANK_PRIMARY_TP_WIRE:?FRANK_PRIMARY_TP_WIRE must be set}"
+: "${FRANK_PRIMARY_MAX_SEQS:?FRANK_PRIMARY_MAX_SEQS must be set}"
+: "${FRANK_PRIMARY_MAX_MODEL_LEN:?FRANK_PRIMARY_MAX_MODEL_LEN must be set}"
+: "${FRANK_PRIMARY_PLACEMENT:?FRANK_PRIMARY_PLACEMENT must be set}"
+: "${FRANK_PRIMARY_HOST_POOL_MIB:?FRANK_PRIMARY_HOST_POOL_MIB must be set}"
+: "${FRANK_PRIMARY_GPU_HEADROOM_MIB:?FRANK_PRIMARY_GPU_HEADROOM_MIB must be set}"
+: "${FRANK_PRIMARY_VRAM_WEIGHTS_MIB:?FRANK_PRIMARY_VRAM_WEIGHTS_MIB must be set}"
+: "${FRANK_PRIMARY_VRAM_KV_MIB:?FRANK_PRIMARY_VRAM_KV_MIB must be set}"
+: "${FRANK_PRIMARY_KV_DTYPE:?FRANK_PRIMARY_KV_DTYPE must be set}"
+: "${FRANK_PRIMARY_SPEC_TOKENS:?FRANK_PRIMARY_SPEC_TOKENS must be set}"
+: "${FRANK_PRIMARY_MAX_BATCHED_TOKENS:?FRANK_PRIMARY_MAX_BATCHED_TOKENS must be set}"
+: "${FRANK_PRIMARY_PREFIX_HOST_MIB:?FRANK_PRIMARY_PREFIX_HOST_MIB must be set}"
+: "${FRANK_PRIMARY_PREFIX_DISK_MIB:?FRANK_PRIMARY_PREFIX_DISK_MIB must be set}"
+: "${FRANK_PRIMARY_NGRAM_PLACEMENT:?FRANK_PRIMARY_NGRAM_PLACEMENT must be set}"
+: "${FRANK_PRIMARY_TTM_MIN_MIB:?FRANK_PRIMARY_TTM_MIN_MIB must be set}"
+: "${FRANK_PRIMARY_RAD_MIN_MIB:?FRANK_PRIMARY_RAD_MIN_MIB must be set}"
+
+# The image gate is the digest-pinned ref itself: a name@sha256 ref exists
+# in storage only if THAT manifest was pulled — a retagged look-alike or a
+# newer latest does not satisfy it. `podman image exists` takes a
+# name@digest ref in one command — no pipefail/grep race (the 2026-09-30
+# lesson). Runs as dank: the unit's process uses dank's ROOTLESS podman
+# storage (root unit, User=dank — the fleet's frank-primary shape).
+if ! podman image exists "$FRANK_PRIMARY_IMAGE"; then
+    echo "ERROR: radiance image absent: $FRANK_PRIMARY_IMAGE" >&2
+    echo "frank-primary: podman pull docker://stilldeadcode/radiance:latest (as dank)" >&2
    exit 1
 fi

-if [ ! -f "$FRANK_PRIMARY_MODEL_DIR/config.json" ]; then
-    echo "serve-frank-primary: payload config missing: $FRANK_PRIMARY_MODEL_DIR/config.json" >&2
+# The payload gate: the .rad container file, sized. A truncated convert
+# would otherwise buy a guaranteed long-timeout boot instead of this fast
+# failure. The staging dir / raw checkpoint are NOT this script's concern
+# (scripts/build-frank-primary-rad.sh built the file; its rad-info output
+# is the record).
+RAD="$FRANK_PRIMARY_MODEL_DIR/$FRANK_PRIMARY_MODEL_FILE"
+if [ ! -f "$RAD" ]; then
+    echo "ERROR: .rad container missing: $RAD" >&2
+    echo "frank-primary: run scripts/build-frank-primary-rad.sh" >&2
    exit 1
 fi
-SHARDS="$(find "$FRANK_PRIMARY_MODEL_DIR" -maxdepth 1 -name 'model-*-of-*.safetensors' | wc -l)"
-if [ "$SHARDS" != "$FRANK_PRIMARY_MODEL_SHARDS" ]; then
-    echo "serve-frank-primary: payload shard count wrong: $FRANK_PRIMARY_MODEL_DIR" >&2
-    echo "  found $SHARDS safetensors, want $FRANK_PRIMARY_MODEL_SHARDS" >&2
-    echo "  re-fetch: scripts/download-frank-primary.sh" >&2
+RAD_MIB=$(( $(stat -c %s "$RAD") / 1048576 ))
+if [ "$RAD_MIB" -lt "$FRANK_PRIMARY_RAD_MIN_MIB" ]; then
+    echo "ERROR: .rad container too small: $RAD is ${RAD_MIB} MiB, want >= $FRANK_PRIMARY_RAD_MIN_MIB" >&2
+    echo "frank-primary: the convert was truncated — rebuild it" >&2
    exit 1
 fi
-if [ ! -f "$FRANK_PRIMARY_MODEL_DIR/model.safetensors.index.json" ]; then
-    echo "serve-frank-primary: weight-map index missing: $FRANK_PRIMARY_MODEL_DIR/model.safetensors.index.json" >&2
+
+if [ ! -f "$FRANK_PRIMARY_TEMPLATE" ]; then
+    echo "ERROR: chat template missing: $FRANK_PRIMARY_TEMPLATE" >&2
    exit 1
 fi

+mkdir -p "$FRANK_PRIMARY_STATE_DIR/kvcache/frank-primary"
+# The engine's create is NOT recursive: with only the state dir mounted, its
+# "cannot create /data/kvcache/frank-primary: No such file or directory"
+# left the SSD prefix tier off (measured 2026-10-06 boot ledger). Pre-create
+# the full path on the host so the tier opens on first boot.
+
+mkdir -p "$FRANK_PRIMARY_STATE_DIR/tune"
+# rad-tune's kernel-selection cache (see the runbook's "Tuning levers"):
+# the engine's selector reads $RADIANCE_HOME/tune/<machine>.tune at boot;
+# $RADIANCE_HOME defaults to /opt/radiance, which lives in the image, so
+# the cache must ride a host mount or it dies with the --rm container
+# that wrote it. The dir is pre-created for the same reason as kvcache
+# (the engine's create is not recursive).
+
 if [ ! -e /dev/kfd ] || [ ! -d /dev/dri ]; then
-    echo "serve-frank-primary: GPU device nodes absent (/dev/kfd, /dev/dri)" >&2
+    echo "ERROR: GPU device nodes absent (/dev/kfd, /dev/dri)" >&2
+    exit 1
+fi
+
+# TTM preflight: the pinned ledger (~2 x 40 GiB pools + 47.7 GiB PLE + 4 GiB
+# prefix host = ~132 GiB) rides TTM's shared pages, capped by ttm's
+# pages_limit (the current kernel exposes NO pages_max; the cap function
+# takes the min over the knobs that exist). Below FRANK_PRIMARY_TTM_MIN_MIB
+# the load dies mid-pin with an allocator error instead of this named
+# refusal — raise the knob first (runtime sysfs write; kernel cmdline
+# fallback + one reboot only if the file is read-only), see the runbook.
+# RAISED 2026-10-05 pre-cutover to 52,428,800 pages = 200 GiB.
+ttm_cap_mib() {
+    # min over the knobs THIS kernel exposes — the current kernel has
+    # pages_limit only (no pages_max); a knob that is absent is skipped,
+    # not a zero (a zero would refuse a healthy cap).
+    local v cur=""
+    for p in /sys/module/ttm/parameters/pages_limit /sys/module/ttm/parameters/pages_max; do
+        [ -r "$p" ] || continue
+        v="$(cat "$p" 2>/dev/null)" || continue
+        [ -n "$v" ] || continue
+        v=$(( v * 4 / 1024 ))
+        if [ "$v" -lt "${cur:-999999999}" ]; then cur=$v; fi
+    done
+    echo "${cur:-0}"
+}
+TTM_CAP="$(ttm_cap_mib)"
+if [ "$TTM_CAP" -lt "$FRANK_PRIMARY_TTM_MIN_MIB" ]; then
+    echo "ERROR: ttm pages cap is ${TTM_CAP} MiB, the pinned ledger needs >= $FRANK_PRIMARY_TTM_MIN_MIB" >&2
+    echo "frank-primary: raise /sys/module/ttm/parameters/pages_limit (and pages_max if the kernel exposes one) to >= $FRANK_PRIMARY_TTM_MIN_MIB in 4 KiB pages" >&2
    exit 1
 fi

 # A crashed/killed unit can leave the container name registered (systemd
 # SIGKILLs conmon before cleanup runs). Clear OUR name plus the legacy
-# identities (the retired R9V container and the ad hoc experiment container
-# both hold these GPUs); all idempotent.
+# identities (the FreeToken-era frank-primary-qwen, the retired R9V
+# container and the ad hoc experiment containers all hold this GPU); all
+# idempotent.
 for CTR in $FRANK_PRIMARY_LEGACY_CONTAINERS "$FRANK_PRIMARY_CONTAINER"; do
    podman rm -f "$CTR" >/dev/null 2>&1 || true
 done
@@ -89,33 +156,51 @@ done
 LOG=/home/dank/logs/frank_primary.log
 mkdir -p "$(dirname "$LOG")"

-# --device /dev/dri is the WHOLE DIRECTORY (individual renderD nodes enumerate
-# without the card nodes and HIP refuses contexts — "No CUDA GPUs are
-# available" while device_count() says 2). --ipc=host for the engine's shared
-# memory; memlock unbounded for the pinned host slabs.
-# The entrypoint already execs `vllm serve "$@"`, so no `serve` here.
+# Publisher-vetted container contract (docs/DOCKER.md, compose/common.yaml):
+# --network host (the server binds the host's addresses directly, no NAT),
+# --init (the engine releases the cards on SIGTERM), seccomp=unconfined
+# (Docker/podman's default profile refuses io_uring, which the n-gram
+# table's reads go through, and the NUMA binding calls the ROCm runtime
+# makes for host memory), memlock unbounded (the pinned ledger — legal
+# because the root unit grants LimitMEMLOCK=infinity to this dank process),
+# /dev/kfd + /dev/dri (render/video groups are moot for a root container).
+# The template rides a single-file mount; the .rad dir is ro; /data holds
+# the prefix-cache disk tier.
 podman run -d --name "$FRANK_PRIMARY_CONTAINER" \
-  --publish "$FRANK_PRIMARY_PUBLISH" \
-  --ipc=host --ulimit memlock=-1:-1 \
-  --cap-add SYS_PTRACE \
-  --device /dev/kfd --device /dev/dri --device /dev/mem \
-  -e HIP_VISIBLE_DEVICES=0,1 -e VLLM_ROCM_USE_AITER=0 \
+  --network host \
+  --init \
+  --security-opt seccomp=unconfined \
+  --ulimit memlock=-1:-1 \
+  --device /dev/kfd --device /dev/dri \
  -v "$FRANK_PRIMARY_MODEL_DIR":/models:ro \
-  "$FRANK_PRIMARY_IMAGE" /models \
-  --served-model-name "$FRANK_PRIMARY_SERVED_NAME" --port 7770 \
-  --tensor-parallel-size 2 \
-  --max-model-len "$FRANK_PRIMARY_MAXLEN" \
-  --max-num-seqs "$FRANK_PRIMARY_SEQS" \
-  --gpu-memory-utilization "$FRANK_PRIMARY_UTIL" \
-  --kv-cache-memory-bytes "$FRANK_PRIMARY_KV_BYTES" \
+  -v "$FRANK_PRIMARY_STATE_DIR":/data \
+  -v "$FRANK_PRIMARY_STATE_DIR/tune":/opt/radiance/share/radiance/tune \
+  -v "$FRANK_PRIMARY_TEMPLATE":/templates/template.jinja:ro \
+  "$FRANK_PRIMARY_IMAGE" \
+  --model "/models/$FRANK_PRIMARY_MODEL_FILE" \
+  --host :: --port "$FRANK_PRIMARY_PORT" \
+  --served-model-name "$FRANK_PRIMARY_SERVED_NAME" \
+  --override-chat-template /templates/template.jinja \
+  --tp "$FRANK_PRIMARY_TP" --tp-wire "$FRANK_PRIMARY_TP_WIRE" \
+  --max-num-seqs "$FRANK_PRIMARY_MAX_SEQS" \
+  --max-model-len "$FRANK_PRIMARY_MAX_MODEL_LEN" \
+  --placement "$FRANK_PRIMARY_PLACEMENT" \
+  --host-pool-mib "$FRANK_PRIMARY_HOST_POOL_MIB" \
+  --gpu-headroom-mib "$FRANK_PRIMARY_GPU_HEADROOM_MIB" \
+  --vram-weights-mib "$FRANK_PRIMARY_VRAM_WEIGHTS_MIB" \
+  --vram-kv-mib "$FRANK_PRIMARY_VRAM_KV_MIB" \
  --kv-cache-dtype "$FRANK_PRIMARY_KV_DTYPE" \
-  --enable-prefix-caching \
-  --speculative-config "{\"method\":\"mtp\",\"num_speculative_tokens\":$FRANK_PRIMARY_SPEC}" \
-  --trust-remote-code \
+  --prefix-cache-host-mib "$FRANK_PRIMARY_PREFIX_HOST_MIB" \
+  --prefix-cache-dir /data/kvcache/frank-primary \
+  --prefix-cache-disk-mib "$FRANK_PRIMARY_PREFIX_DISK_MIB" \
+  --ngram-placement "$FRANK_PRIMARY_NGRAM_PLACEMENT" \
+  --num-speculative-tokens "$FRANK_PRIMARY_SPEC_TOKENS" \
+  --max-num-batched-tokens "$FRANK_PRIMARY_MAX_BATCHED_TOKENS" \
+  ${FRANK_PRIMARY_EXTRA_ARGS:+$FRANK_PRIMARY_EXTRA_ARGS} \
  > "$LOG" 2>&1

 # From here on, this script's lifetime IS the container's: wait returns the
 # container's exit code, so a crash is a nonzero ExecStart and the unit's
-# Restart=on-failure is real. (systemd stop goes through ExecStop in the
-# unit — podman wait alone cannot stop the container child.)
+# Restart=always is real. (systemd stop goes through ExecStop in the unit —
+# podman wait alone cannot stop the container child.)
 exec podman wait "$FRANK_PRIMARY_CONTAINER"
@@ -0,0 +1,132 @@
+#!/usr/bin/env bash
+# Serve HauhauCS/Qwen3.5-9B-Uncensored-HauhauCS-Aggressive (Q6_K) as
+# frank-secondary on Dankclaw's RTX 3090 via llama.cpp (master build, CUDA sm_86).
+#
+# 2026-10-07 (Frank): Qwen3.5-9B swap at TWO 200,704-token conversations
+# (--ctx-size 401408 --parallel 2) with EXACT f16 KV — the 9B rung frees the
+# room the 26B MoE Gemma4 shape lacked, so quantized V retires. MTP stays off:
+# this GGUF carries no MTP tensors at all (verified from its tensor-name
+# list), so there is no in-file head to draft with. Verbosity 4 ships so the
+# VRAM-fit forensics (KV layout, buffer sizes, graph capture) land in the journal.
+#
+# Architecture: qwen35 — dense 9B, 32 layers with full_attention_interval 4
+# (8 full-attention layers at 3,7,...,31 + 24 Gated DeltaNet linear layers),
+# 16 heads / 4 KV heads × 256 dim, native 262,144, vocab 248,320. The KV pool
+# scales with the 8 full layers: 32 KiB/token at f16 K + f16 V (12.25 GiB at
+# the 401,408 pool); the DeltaNet layers cost a fixed recurrent state per
+# slot. Feasibility is priced from the MiMo-V2.6-Distill-Qwen-9B boots (same
+# geometry, same card, same shape): 21,276/2,849 at ubatch 1024, 21,846/2,279
+# at ubatch 2048 — the boot ledger, not arithmetic, is the referee.
+# Vision comes from the publisher's own BF16 projector (the model is natively
+# multimodal), so --mmproj is what gives this endpoint eyes; MMPROJ_FILE="" is
+# the documented text-only opt-out.
+#
+# Sampling: llama.cpp takes it per request (the fleet harness sends the
+# publisher recipe from models.yml: thinking temp 0.6 / top_p 0.95 / top_k 20).
+#
+# Tunables come from frank-secondary.env unless FRANK_SECONDARY_ENV_FILE overrides.
+set -euo pipefail
+
+SCRIPT_DIR="$(cd -- "$(dirname -- "${BASH_SOURCE[0]}")" && pwd)"
+ENV_FILE="${FRANK_SECONDARY_ENV_FILE:-$SCRIPT_DIR/frank-secondary.env}"
+if [[ -f "$ENV_FILE" ]]; then
+    set -a
+    # shellcheck disable=SC1090
+    source "$ENV_FILE"
+    set +a
+fi
+
+# --- defaults (used when the env file does not set them) -------------------
+: "${LLAMA_BIN:=/home/dank/GitPackages/llama-master/build-cuda/bin/llama-server}"
+: "${MODEL_FILE:=/home/dank/models/Qwen35-9B-HauhauCS-Q6K/Qwen3.5-9B-Uncensored-HauhauCS-Aggressive-Q6_K.gguf}"
+: "${ALIAS:=frank-zealot}"
+: "${BIND_ADDR:=0.0.0.0}"
+: "${LISTEN_PORT:=7771}"
+# ctx-size is the POOL llama.cpp divides across slots: 401,408 / 2 = 200,704
+# per conversation, inside the model's trained 262,144. The harness
+# contextWindow MUST quote the PER-SLOT number — a longer prompt is answered
+# with a 400, never truncated.
+: "${CTX_SIZE:=401408}"
+: "${PARALLEL:=2}"
+# Compute-buffer geometry. Ubatch is a prefill knob (decode never moves).
+# 2048 is the measured CUDA optimum (MiMo era, same arch/shape: +4.6 % prefill
+# at 8.9k depth, decode flat, 616 MiB). The logical batch is VRAM- and
+# throughput-neutral at a fixed ubatch.
+: "${BATCH_SIZE:=4096}"
+: "${UBATCH_SIZE:=2048}"
+# Exact (unquantized) KV: f16 K + f16 V, restoring the MiMo-era contract that
+# the 26B Gemma4 shape could not afford. 32 KiB/token at this geometry
+# (8 full-attention layers × 4 KV heads × 256 dim); 12.25 GiB at the 401,408
+# pool. Explicit either way so the contract is visible in the argv, and
+# --flash-attn stays required for any quantized-V fallback (q8_0 = llama.cpp's
+# FP8 class; no true fp8 KV dtype exists in this build).
+: "${CACHE_TYPE_K:=f16}"
+: "${CACHE_TYPE_V:=f16}"
+# Speculative decoding is OFF: this Q6_K file carries NO nextn/mtp tensors, so
+# there is no in-file head (the law: MTP is a property of the file, not of the
+# flags). SPEC_TYPE=none strips BOTH speculative flags (flags move as a set);
+# naming a drafter requires SPEC_DRAFT_FILE alongside SPEC_TYPE — the Gemma4
+# assistant on disk is arch gemma4-assistant and cannot drafter-match qwen35.
+: "${SPEC_TYPE:=none}"
+: "${SPEC_DRAFT_N_MAX:=3}"
+: "${SPEC_DRAFT_FILE:=}"
+: "${NGL:=99}"
+# Log verbosity threshold (llama.cpp -lv). 3 is the engine default (info);
+# 4 adds debug detail (KV layout, buffer sizes, graph capture) which is what
+# the VRAM-fit forensics need. Levels above 4 flood the journal.
+: "${VERBOSITY:=3}"
+# Vision projector. No colon on purpose: an explicitly empty value means
+# "text-only, on purpose" and must NOT be replaced by the default path.
+: "${MMPROJ_FILE=/home/dank/models/Qwen35-9B-HauhauCS-Q6K/mmproj-Qwen3.5-9B-Uncensored-HauhauCS-Aggressive-BF16.gguf}"
+
+# Fail loudly rather than half-loading: a missing file is a download that
+# has not finished, and Restart=always would turn it into a crash loop that
+# reads like a runtime fault.
+for path in "$LLAMA_BIN" "$MODEL_FILE"; do
+    [[ -e "$path" ]] || { echo "frank-secondary: missing $path" >&2; exit 1; }
+done
+EXTRA_ARGS=()
+if [[ -n "$MMPROJ_FILE" ]]; then
+    [[ -e "$MMPROJ_FILE" ]] || { echo "frank-secondary: missing $MMPROJ_FILE" >&2; exit 1; }
+    EXTRA_ARGS+=(--mmproj "$MMPROJ_FILE")
+fi
+
+# Speculative flags move as a set: they reach the argv ONLY when a drafter
+# is named. SPEC_TYPE=none (the shipped contract) strips both — a no-spec
+# baseline must not carry half a spec pair.
+if [[ -n "$SPEC_TYPE" && "$SPEC_TYPE" != "none" ]]; then
+    EXTRA_ARGS+=(--spec-type "$SPEC_TYPE" --spec-draft-n-max "$SPEC_DRAFT_N_MAX")
+if [[ -n "$SPEC_DRAFT_FILE" ]]; then
+    [[ -e "$SPEC_DRAFT_FILE" ]] || { echo "frank-secondary: missing $SPEC_DRAFT_FILE" >&2; exit 1; }
+    EXTRA_ARGS+=(--spec-draft-model "$SPEC_DRAFT_FILE")
+fi
+fi
+
+# The UVM device node is created lazily by nvidia-modprobe on the first CUDA
+# open after boot; make sure it exists before the server needs it.
+nvidia-modprobe -u -c=0 2>/dev/null || true
+
+# Flash attention stays pinned on: it is REQUIRED for quantized V (q8_0),
+# and the build carries GGML_CUDA_FA_ALL_QUANTS=ON (since 2026-09-17), so
+# the q8_0 KV contract in frank-secondary.env is a real path — without this
+# flag or that build option the engine silently falls back and the
+# contract becomes a lie.
+ARGS=(
+    --model "$MODEL_FILE"
+    --alias "$ALIAS"
+    --host "$BIND_ADDR"
+    --port "$LISTEN_PORT"
+    --ctx-size "$CTX_SIZE"
+    --parallel "$PARALLEL"
+    --batch-size "$BATCH_SIZE"
+    --ubatch-size "$UBATCH_SIZE"
+    --n-gpu-layers "$NGL"
+    --flash-attn on
+    --cache-type-k "$CACHE_TYPE_K"
+    --cache-type-v "$CACHE_TYPE_V"
+    --log-verbosity "$VERBOSITY"
+    --jinja
+    --reasoning auto
+    --reasoning-format deepseek
+)
+exec "$LLAMA_BIN" "${ARGS[@]}" ${EXTRA_ARGS[@]+"${EXTRA_ARGS[@]}"}
@@ -1,146 +0,0 @@
-#!/usr/bin/env bash
-# Serve HauhauCS/Gemma4-12B-QAT-Uncensored-HauhauCS-Balanced (Q4_K_M) as
-# frank-small on Dankclaw's RTX 3090 via mainline llama.cpp.
-#
-# Topology: the whole model lives in VRAM (-ngl 999) -- 6.87 GiB of Q4_K_M
-# weights plus the BF16 vision projector and q8_0 KV inside the card's
-# 24 GiB. The host CPU only tokenizes and samples here; this box's CPU
-# belongs to the halogen engine that serves frank-large.
-#
-# MTP: this model's Multi-Token Prediction head ships as a SEPARATE file
-# (mtp-gemma-4-12B-it.gguf, arch gemma4-assistant, 4 nextn layers). It is
-# passed via --model-draft with --spec-type draft-mtp: the draft context
-# loads the assistant model. The same --spec-type value chains the
-# draftless ngram-mod matcher, which attaches unconditionally; two
-# context-backed drafters cannot share one process, so this chain is the
-# only shape that carries both.
-#
-# Context: CTX_SIZE is set explicitly to PARALLEL x 262144 so every slot
-# gets the model's FULL window; --fit (the CTX_SIZE-unset branch below)
-# maximises the TOTAL pool instead, which would divide a smaller window
-# across slots. The 4-slot layout is measured to fit the card (see the
-# SIZING note in frank-small.env); more slots would fail the KV allocation.
-#
-# Vision + audio: --mmproj loads the BF16 projector, which also carries
-# an audio encoder — input_audio content parts transcribe verbatim
-# (verified 2026-09-17). The fleet's image path lives here, which is why
-# halogen no longer sets HALOGEN_VISION_TOWER.
-#
-# Sampling: greedy by default (the fleet contract). Clients that send their own
-# temperature get their own values, and the harness overrides per model from
-# models.yml. --jinja is pinned even though this build defaults it on: the
-# model's own template supplies reasoning_effort, preserve_thinking and the
-# tool-call grammar, and a future build that flips that default must not
-# silently take tools away.
-#
-# Tunables come from frank-small.env unless FRANK_SMALL_ENV_FILE overrides it.
-set -euo pipefail
-
-SCRIPT_DIR="$(cd -- "$(dirname -- "${BASH_SOURCE[0]}")" && pwd)"
-ENV_FILE="${FRANK_SMALL_ENV_FILE:-$SCRIPT_DIR/frank-small.env}"
-if [[ -f "$ENV_FILE" ]]; then
-    set -a
-    # shellcheck disable=SC1090
-    source "$ENV_FILE"
-    set +a
-fi
-
-# --- defaults (used when the env file does not set them) -------------------
-: "${LLAMA_BIN:=/home/dank/GitPackages/llama.cpp-mainline/build-cuda/bin/llama-server}"
-: "${MODEL_DIR:=/home/dank/models/Gemma4-12B-QAT}"
-: "${MODEL_FILE:=Gemma4-12B-QAT-Uncensored-HauhauCS-Balanced-Q4_K_M.gguf}"
-: "${MMPROJ_PATH:=$MODEL_DIR/mmproj-Gemma4-12B-QAT-Uncensored-HauhauCS-Balanced-BF16.gguf}"
-# NOTE: the fallback above was the retired ISTA-DASLab Qwen3.8-27B GSQ-RCO
-# payload, which frank-small served on 2026-09-16/17 and which no longer
-# exists on disk. frank-small.env has always overridden it, so the drift was
-# invisible — but a missing or unreadable env file would have left this
-# script pointing at a payload that is not there, turning a config problem
-# into a "missing asset" that names the wrong model.
-: "${DRAFT_FILE:=}"
-: "${ALIAS:=frank-small}"
-: "${BIND_ADDR:=0.0.0.0}"
-: "${LISTEN_PORT:=7771}"
-: "${CTX_SIZE:=}"
-: "${PARALLEL:=1}"
-: "${CACHE_TYPE_K:=f16}"
-: "${CACHE_TYPE_V:=f16}"
-: "${CACHE_RAM:=8192}"
-: "${BATCH_SIZE:=2048}"
-: "${UBATCH_SIZE:=512}"
-# Decode-shape knobs: the MTP drafter's per-round proposal length (build
-# default 3; the Gemma4 sidecar carries 4 nextn layers) and the engine's
-# busy-wait level (build default 50).
-: "${SPEC_DRAFT_N_MAX:=3}"
-: "${POLL_LEVEL:=50}"
-: "${NUM_THREADS:=8}"
-: "${THREAD_BATCH:=8}"
-: "${NGL:=999}"
-: "${FLASH_ATTN:=on}"
-: "${SPEC_TYPE:=draft-mtp}"
-: "${TEMPERATURE:=0.0}"
-: "${REASONING_FORMAT:=deepseek}"
-: "${REASONING_BUDGET:=-1}"
-
-# Fail loudly rather than half-loading a model into VRAM: a missing file here
-# is a download that has not finished, and the unit's restart policy would
-# otherwise turn that into a crash loop that reads like a runtime fault.
-for path in "$LLAMA_BIN" "$MODEL_DIR/$MODEL_FILE" "$MMPROJ_PATH"; do
-    [[ -e "$path" ]] || { echo "frank-small: missing $path" >&2; exit 1; }
-done
-if [[ -n "${DRAFT_FILE:-}" ]]; then
-    [[ -e "$MODEL_DIR/$DRAFT_FILE" ]] || { echo "frank-small: missing $MODEL_DIR/$DRAFT_FILE" >&2; exit 1; }
-fi
-
-# The UVM device node is created lazily by nvidia-modprobe on the first CUDA
-# open after boot; make sure it exists before the server needs it.
-nvidia-modprobe -u -c=0 2>/dev/null || true
-
-MMPROJ_ARGS=()
-if [[ -n "${MMPROJ_PATH:-}" && -f "$MMPROJ_PATH" ]]; then
-    MMPROJ_ARGS=(--mmproj "$MMPROJ_PATH")
-fi
-
-SPEC_ARGS=()
-if [[ -n "${SPEC_TYPE:-}" && "$SPEC_TYPE" != "none" ]]; then
-    SPEC_ARGS=(--spec-type "$SPEC_TYPE" --spec-draft-n-max "$SPEC_DRAFT_N_MAX")
-fi
-
-DRAFT_ARGS=()
-if [[ -n "${DRAFT_FILE:-}" ]]; then
-    DRAFT_ARGS=(--model-draft "$MODEL_DIR/$DRAFT_FILE")
-fi
-
-# CTX_SIZE set -> pinned context. Unset -> --fit sizes the KV pool against
-# whatever the weights, projector, draft context and compute buffers leave
-# on the card; the engine prints the chosen value at startup.
-if [[ -n "${CTX_SIZE:-}" ]]; then
-    CTX_ARGS=(--ctx-size "$CTX_SIZE")
-else
-    CTX_ARGS=(--fit on)
-fi
-
-exec "$LLAMA_BIN" \
-    --model "$MODEL_DIR/$MODEL_FILE" \
-    "${MMPROJ_ARGS[@]}" \
-    "${DRAFT_ARGS[@]}" \
-    --alias "$ALIAS" \
-    --host "$BIND_ADDR" \
-    --port "$LISTEN_PORT" \
-    --gpu-layers "$NGL" \
-    --flash-attn "$FLASH_ATTN" \
-    "${CTX_ARGS[@]}" \
-    --parallel "$PARALLEL" \
-    --batch-size "$BATCH_SIZE" \
-    --ubatch-size "$UBATCH_SIZE" \
-    --threads "$NUM_THREADS" \
-    --threads-batch "$THREAD_BATCH" \
-    --poll "$POLL_LEVEL" \
-    --cache-type-k "$CACHE_TYPE_K" \
-    --cache-type-v "$CACHE_TYPE_V" \
-    --cache-ram "$CACHE_RAM" \
-    "${SPEC_ARGS[@]}" \
-    --jinja \
-    --metrics \
-    --temp "$TEMPERATURE" \
-    --reasoning-format "$REASONING_FORMAT" \
-    --reasoning-budget "$REASONING_BUDGET"
@@ -0,0 +1,556 @@
+#!/usr/bin/env python3
+"""stack-qwen38-official-fp8.py — reshape the official Qwen/Qwen3.8-Flash-Next-FP8
+checkpoint into the form the shipped radiance qwen4exp plugin maps.
+
+Run INSIDE the staging dir's tree (python3 + numpy on the host; data never
+leaves the box). Two transforms, both VALUE-PRESERVING:
+
+1. EXPERTS: the shipped plugin's stacked layout names ONE tensor per family
+   (`mlp.experts.gate_up_proj` [E, 2F, H], `mlp.experts.down_proj` [E, H, F])
+   plus a `<family>_scale_inv` plane at the same index (NO `.weight` in the
+   stacked name — scale_name() appends _scale_inv to the tensor name verbatim),
+   while the official FP8 repo ships per-expert unfused tensors
+   (`experts.<e>.{gate,up,down}_proj.weight` + scale). Stacking is pure byte
+   concatenation along dim 0 — gate rows
+   first inside each expert's gate_up slab, which is the order the block
+   wants (rad_block_moe_fp8.h: "a stacked gate_up is ALREADY FUSED in the
+   order this block wants"). No requantisation: bytes are moved, not decoded.
+
+2. PLE TABLE: the plugin's fp8 branch wants the table's codes beside a
+   ONE-ELEMENT bf16 scale plane (`blk.%d.ple_ngram.scale`, view {1}); a
+   per-shard-blocked scale can never fit that view, and `rtn` re-encoding the
+   RAW scaleless codes would overflow (v / 1.99e-4 exceeds e4m3's 448 max).
+   So staging decodes the scaleless shards with the checkpoint's OWN bf16
+   scale into a bf16 table (exact: every e4m3 code value and the scale are
+   exactly representable in bf16 to within the code's own 2^-3 grid), and the
+   convert's rtn rule re-encodes at that same scale — the nearest-code
+   distance is 2^-3 while the bf16 rounding perturbs by 2^-9, so every
+   original code is restored exactly and the container gets the checkpoint's
+   own codes plus the one-element scale plane.
+
+The convert then runs with NO recipe except:
+  blk.*.ple_ngram.weight=rtn:codes=fp8_e4m3,block=*x*,scale=bf16,rule=fixed,scale_value=<s>
+with <s> read from the checkpoint and written to ple_scale.txt.
+
+Idempotent: re-runs skip when the staged index already carries the stacked
+names, the merged bf16 shard_0, and split_ngram_parts == 1. --force redoes.
+"""
+
+import argparse
+import json
+import mmap
+import multiprocessing as mp
+import os
+import re
+import struct
+import sys
+import time
+import numpy as np
+F8, BF16 = "F8_E4M3", "BF16"
+
+
+def log(msg):
+    print(msg, flush=True)
+
+
+# ---------------------------------------------------------------- safetensors
+
+def parse_header(path):
+    """Return (header_len, {name: (dtype, shape, data_offsets)})."""
+    with open(path, "rb") as fh:
+        (hlen,) = struct.unpack("<Q", fh.read(8))
+        hdr = json.loads(fh.read(hlen))
+    return hlen, hdr
+
+
+def write_safetensors(path, tensors, data_iter):
+    """tensors: [(name, dtype, shape, nbytes)]; data_iter yields bytes per tensor."""
+    hdr = {}
+    off = 0
+    for name, dtype, shape, nbytes in tensors:
+        hdr[name] = {"dtype": dtype, "shape": list(shape),
+                     "data_offsets": [off, off + nbytes]}
+        off += nbytes
+    blob = json.dumps(hdr, separators=(",", ":")).encode("utf-8")
+    pad = (-len(blob)) % 8  # keep the data start 8-byte aligned
+    blob += b" " * pad
+    with open(path, "wb") as fh:
+        fh.write(struct.pack("<Q", len(blob)))
+        fh.write(blob)
+        for _, _, _, nbytes in tensors:
+            fh.write(next(data_iter).tobytes())
+        fh.flush()
+        os.fsync(fh.fileno())
+    return 8 + len(blob) + sum(t[3] for t in tensors)
+
+
+# ------------------------------------------------------------------- preflight
+
+def build_meta(stage, index):
+    """name -> (file, global_byte_offset, nbytes, dtype, shape) for every tensor."""
+    files = sorted(set(index["weight_map"].values()))
+    meta = {}
+    for f in files:
+        path = os.path.join(stage, f)
+        hlen, hdr = parse_header(path)
+        for name, e in hdr.items():
+            if name == "__metadata__":
+                continue
+            b0, b1 = e["data_offsets"]
+            meta[name] = (f, 8 + hlen + b0, b1 - b0, e["dtype"], e["shape"])
+    return meta
+
+
+def e4m3_lut():
+    lut = np.zeros(256, np.float32)
+    for b in range(256):
+        sign = -1.0 if b & 0x80 else 1.0
+        e = (b >> 3) & 0xF
+        m = b & 0x7
+        if e == 15 and m == 7:
+            v = np.float32(np.nan)  # OCP e4m3fn: only 0x7F/0xFF are NaN
+        elif e == 0:
+            v = sign * (m / 8.0) * 2.0 ** -6
+        else:
+            v = sign * (1.0 + m / 8.0) * 2.0 ** (e - 7)
+        lut[b] = v
+    assert lut[0x38] == 1.0 and lut[0x7E] == 448.0
+    assert np.isnan(lut[0x7F]) and lut[0x00] == 0.0
+    assert lut[0x08] == 2.0 ** -6 and lut[0x80] == 0.0
+    return lut
+
+
+def f32_to_bf16_rne(v):
+    """float32 array -> uint16 bf16 bits, round-to-nearest-even."""
+    q = np.ascontiguousarray(v, dtype=np.float32)
+    q[~np.isfinite(q)] = 0.0
+    bits = q.view(np.uint32)
+    r = bits + 0x7FFF + ((bits >> 16) & 1)
+    return (r >> 16).astype(np.uint16)
+
+
+# ------------------------------------------------------------------- workers
+
+def stack_prefix(job):
+    """Byte-stack one layer's 512 experts into a 4-tensor safetensors file."""
+    (stage, pre, idx, n_e, shapes, meta) = job
+    (f2, h) = shapes["ffn2_h"]          # 2*ffn, n_embd
+    (f,) = shapes["ffn"]                # ffn
+    gs, ds = shapes["gs"], shapes["ds"] # scale shapes [5,20] and [20,5]
+
+    # byte widths: fp8 is 1 B/element, bf16 2 B; the fused gate_up scale is
+    # the gate and up scale planes concatenated (gate rows first)
+    g_bytes = (f2 // 2) * h
+    gs_bytes = gs[0] * gs[1] * 2          # ONE projection's scale plane
+    gus_bytes = 2 * gs_bytes              # fused scale per expert
+    dn_bytes = h * f
+    ds_bytes = ds[0] * ds[1] * 2
+
+    gu = np.empty(n_e * f2 * h, np.uint8)
+    gus = np.empty(n_e * gus_bytes, np.uint8)
+    dn = np.empty(n_e * dn_bytes, np.uint8)
+    dns = np.empty(n_e * ds_bytes, np.uint8)
+
+    mmaps = {}
+
+    def view(name):
+        f_, off, ln, dt, shp = meta[name]
+        if f_ not in mmaps:
+            fh = open(os.path.join(stage, f_), "rb")
+            mmaps[f_] = mmap.mmap(fh.fileno(), 0, access=mmap.ACCESS_READ)
+        return memoryview(mmaps[f_])[off:off + ln], (dt, shp)
+
+    def need(name, dt, shp):
+        mv, (got_dt, got_shp) = view(name)
+        assert got_dt == dt and list(got_shp) == list(shp), f"{name}: {got_dt}{got_shp}"
+        return mv
+
+    for e in range(n_e):
+        base = f"{pre}.mlp.experts.{e}"
+        slab = gu[e * f2 * h:(e + 1) * f2 * h]
+        slab[:g_bytes] = need(f"{base}.gate_proj.weight", F8, [f2 // 2, h])
+        slab[g_bytes:] = need(f"{base}.up_proj.weight", F8, [f2 // 2, h])
+        sslab = gus[e * gus_bytes:(e + 1) * gus_bytes]
+        sslab[:gs_bytes] = need(f"{base}.gate_proj.weight_scale_inv", BF16, gs)
+        sslab[gs_bytes:] = need(f"{base}.up_proj.weight_scale_inv", BF16, gs)
+        dn[e * dn_bytes:(e + 1) * dn_bytes] = need(f"{base}.down_proj.weight", F8, [h, f])
+        dns[e * ds_bytes:(e + 1) * ds_bytes] = need(f"{base}.down_proj.weight_scale_inv", BF16, ds)
+
+    for mm in mmaps.values():
+        mm.close()
+
+    # byte-identity check on two sampled experts (first and last), all four
+    # tensors — the stack IS the checkpoint bytes, so they must compare equal
+    for e in (0, n_e - 1):
+        base = f"{pre}.mlp.experts.{e}"
+        s = e * f2 * h
+        assert bytes(gu[s:s + g_bytes]) == bytes(stack_src(f"{base}.gate_proj.weight", meta, stage))
+        assert bytes(gu[s + g_bytes:s + f2 * h]) == bytes(stack_src(f"{base}.up_proj.weight", meta, stage))
+        t = e * gus_bytes
+        assert bytes(gus[t:t + gs_bytes]) == bytes(stack_src(f"{base}.gate_proj.weight_scale_inv", meta, stage))
+        assert bytes(gus[t + gs_bytes:t + gus_bytes]) == bytes(stack_src(f"{base}.up_proj.weight_scale_inv", meta, stage))
+        d = e * dn_bytes
+        assert bytes(dn[d:d + dn_bytes]) == bytes(stack_src(f"{base}.down_proj.weight", meta, stage))
+        u = e * ds_bytes
+        assert bytes(dns[u:u + ds_bytes]) == bytes(stack_src(f"{base}.down_proj.weight_scale_inv", meta, stage))
+
+    out = os.path.join(stage, f"stacked.{idx:03d}.safetensors")
+    tensors = [
+        (f"{pre}.mlp.experts.gate_up_proj", F8, [n_e, f2, h], n_e * f2 * h),
+        (f"{pre}.mlp.experts.gate_up_proj_scale_inv", BF16,
+         [n_e, gs[0] * 2, gs[1]], n_e * gus_bytes),
+        (f"{pre}.mlp.experts.down_proj", F8, [n_e, h, f], n_e * h * f),
+        (f"{pre}.mlp.experts.down_proj_scale_inv", BF16,
+         [n_e, ds[0], ds[1]], n_e * ds_bytes),
+    ]
+    bufs = iter([gu, gus, dn, dns])
+    written = write_safetensors(out, tensors, bufs)
+    return (pre, written)
+
+
+def stack_src(name, meta, stage):
+    """Re-open a source tensor for the sampled-expert check."""
+    f_, off, ln, dt, shp = meta[name]
+    fh = open(os.path.join(stage, f_), "rb")
+    mm = mmap.mmap(fh.fileno(), 0, access=mmap.ACCESS_READ)
+    fh.close()  # mmap keeps the mapping alive
+    return mm[off:off + ln]
+
+
+def fill_ple_shard(job):
+    """Decode one fp8 shard with the table scale, write bf16 rows into the mmap."""
+    (src_path, src_off, src_rows, dst_path, dst_off, scale_f32,
+     cols, chunk_rows) = job
+    lut = e4m3_lut()
+    with open(src_path, "rb") as fh:
+        sm = mmap.mmap(fh.fileno(), 0, access=mmap.ACCESS_READ)
+        with open(dst_path, "r+b") as dfh:
+            dm = mmap.mmap(dfh.fileno(), 0, access=mmap.ACCESS_WRITE)
+            for r0 in range(0, src_rows, chunk_rows):
+                r1 = min(r0 + chunk_rows, src_rows)
+                codes = np.frombuffer(sm, dtype=np.uint8,
+                                      count=(r1 - r0) * cols,
+                                      offset=src_off + r0 * cols)
+                q = lut.take(codes)
+                del codes  # release the mmap export before sm.close()
+                q[~np.isfinite(q)] = 0.0
+                q *= np.float32(scale_f32)
+                packed = f32_to_bf16_rne(q)
+                dm[dst_off + r0 * cols * 2: dst_off + r1 * cols * 2] = packed.tobytes()
+            dm.flush()
+            os.fsync(dfh.fileno())
+            dm.close()
+        sm.close()
+
+
+# ---------------------------------------------------------------------- main
+
+def main():
+    ap = argparse.ArgumentParser(description=__doc__.split("\n")[0])
+    ap.add_argument("--src", default=None,
+                    help="raw checkpoint dir (documentation only; unused)")
+    ap.add_argument("--stage", required=True, help="staging dir to reshape in place")
+    ap.add_argument("--force", action="store_true", help="redo even if already staged")
+    ap.add_argument("--workers", type=int, default=min(8, os.cpu_count() or 1))
+    args = ap.parse_args()
+    stage = args.stage
+    t0 = time.time()
+
+    index_path = os.path.join(stage, "model.safetensors.index.json")
+    cfg_path = os.path.join(stage, "config.json")
+    if not (os.path.isfile(index_path) and os.path.isfile(cfg_path)):
+        log(f"ERROR: {stage} is not a staged checkpoint (missing index/config)")
+        return 2
+    with open(index_path) as fh:
+        index = json.load(fh)
+    with open(cfg_path) as fh:
+        cfg = json.load(fh)
+    wm = index["weight_map"]
+
+    meta = build_meta(stage, index)
+
+    # ---- collect prefixes -------------------------------------------------
+    pre_re = re.compile(r"^(?P<pre>.+)\.mlp\.experts\.(?P<e>\d+)\.gate_proj\.weight$")
+    pres, n_e = {}, 0
+    for k in wm:
+        m = pre_re.match(k)
+        if m:
+            pres.setdefault(m.group("pre"), set()).add(int(m.group("e")))
+            n_e = max(n_e, int(m.group("e")) + 1)
+    prefixes = sorted(pres)
+    if not prefixes:
+        log("ERROR: no per-expert tensors found in the staged index")
+        return 2
+    mtp = [p for p in prefixes if p.startswith("mtp.")]
+    trunk = [p for p in prefixes if p.startswith("model.language_model.layers.")]
+    log(f"expert prefixes: {len(prefixes)} (trunk {len(trunk)}, mtp {len(mtp)}), "
+        f"{n_e} experts each")
+
+    # shapes from expert 0 of the first prefix
+    pre0 = prefixes[0]
+    shapes = {}
+    gdt, gsh = meta[f"{pre0}.mlp.experts.0.gate_proj.weight"][3:5]
+    ddt, dsh = meta[f"{pre0}.mlp.experts.0.down_proj.weight"][3:5]
+    gsd, gssh = meta[f"{pre0}.mlp.experts.0.gate_proj.weight_scale_inv"][3:5]
+    dsd, dssh = meta[f"{pre0}.mlp.experts.0.down_proj.weight_scale_inv"][3:5]
+    scale_law = (gdt == F8 and ddt == F8 and gsd == BF16 and dsd == BF16
+                 and list(gssh) == [(gsh[0] + 127) // 128, (gsh[1] + 127) // 128]
+                 and list(dssh) == [(dsh[0] + 127) // 128, (dsh[1] + 127) // 128])
+    if not scale_law:
+        log(f"ERROR: unexpected expert shapes: gate {gdt}{gsh} scale {gsd}{gssh} "
+            f"down {ddt}{dsh} scale {dsd}{dssh} (scale law 128x128)")
+        return 2
+    shapes = {"ffn2_h": (gsh[0] * 2, gsh[1]), "ffn": (dsh[1],),
+              "gs": tuple(gssh), "ds": tuple(dssh)}
+    f2, h = shapes["ffn2_h"]
+    f = shapes["ffn"][0]
+    log(f"shapes: gate_up [{n_e},{f2},{h}] F8, down [{n_e},{h},{f}] F8, "
+        f"scales {gssh}x2 / {dssh} bf16")
+
+    # ---- PLE table --------------------------------------------------------
+    ple_re = re.compile(r"^(?P<pre>.+)\.ple\.ple_embedding\.ngram_embedding\.shard_(?P<i>\d+)\.weight$")
+    ple_pre, shard_ids = None, {}
+    for k in wm:
+        m = ple_re.match(k)
+        if m:
+            ple_pre = m.group("pre")
+            shard_ids[int(m.group("i"))] = k
+    if ple_pre is None:
+        log("ERROR: no PLE table shards found")
+        return 2
+    n_shards = len(shard_ids)
+    if sorted(shard_ids) != list(range(n_shards)):
+        log(f"ERROR: PLE shard ids not contiguous 0..{n_shards - 1}")
+        return 2
+    sc_name = f"{ple_pre}.ple.ple_embedding.ngram_embedding.weight_scale"
+    sdt, ssh, sln = meta[sc_name][3], meta[sc_name][4], meta[sc_name][2]
+    if sdt != BF16 or list(ssh) != [1]:
+        log(f"ERROR: PLE scale is {sdt}{ssh}, expected BF16 [1]")
+        return 2
+    s_bits = struct.unpack("<H", bytes(view_src_tuple(meta, sc_name, stage)))[0]
+    scale_f32 = struct.unpack("<f", struct.pack("<I", s_bits << 16))[0]
+    log(f"PLE: {ple_pre}, {n_shards} shards, scale bf16 = {scale_f32!r}")
+
+    rows = [meta[shard_ids[i]][4][0] for i in range(n_shards)]
+    total_rows = sum(rows)
+    hd = meta[shard_ids[0]][4][1]
+    if any(meta[shard_ids[i]][4][1] != hd for i in range(n_shards)):
+        log("ERROR: PLE shards disagree on HD")
+        return 2
+    if hd % 128 != 0 and hd != 160:
+        log(f"WARNING: PLE HD={hd} is neither 160 nor a multiple of 128")
+
+    # ---- idempotence gate -------------------------------------------------
+    stacked_names = [f"{p}.mlp.experts.gate_up_proj" for p in prefixes]
+    merged_name = f"{ple_pre}.ple.ple_embedding.ngram_embedding.shard_0.weight"
+    already = (all(n in wm for n in stacked_names)
+               and merged_name in wm
+               and find_split(cfg) is not None and read_split(cfg) == 1)
+    if already and not args.force:
+        log("already staged (stacked names, merged shard_0 and split_ngram_parts=1 "
+            "all present) — nothing to do")
+        return 0
+
+    # ---- disk gate --------------------------------------------------------
+    stacked_bytes = len(prefixes) * n_e * (f2 * h + h * f)
+    ple_bytes = total_rows * hd * 2
+    need = stacked_bytes + ple_bytes
+    st = os.statvfs(stage)
+    avail = st.f_bavail * st.f_frsize
+    if avail < need * 1.05:
+        log(f"ERROR: {avail / 2**30:.1f} GiB free < {need / 2**30:.1f} GiB needed (+5%)")
+        return 2
+    log(f"disk: {avail / 2**30:.1f} GiB free, need {need / 2**30:.1f} GiB "
+        f"(experts {stacked_bytes / 2**30:.1f}, ple {ple_bytes / 2**30:.1f})")
+
+    # ---- stage experts ----------------------------------------------------
+    jobs = []
+    for idx, pre in enumerate(prefixes):
+        jmeta = {n: meta[n] for n in meta if n.startswith(f"{pre}.mlp.experts.")}
+        jobs.append((stage, pre, idx, n_e, shapes, jmeta))
+    ctx = mp.get_context("fork")
+    written_total = 0
+    with ctx.Pool(args.workers) as pool:
+        for pre, nbytes in pool.imap_unordered(stack_prefix, jobs):
+            written_total += nbytes
+            log(f"  stacked {pre} ({nbytes / 2**30:.2f} GiB)")
+    log(f"experts stacked: {written_total / 2**30:.2f} GiB in {len(prefixes)} files")
+
+    # ---- stage PLE --------------------------------------------------------
+    # The file name sorts BEFORE the model-000NN shards ON PURPOSE: the reader
+    # opens every index-named file in sorted order and emplaces EVERY tensor
+    # it physically holds, first-wins (checkpoint.cpp add_shard). The raw
+    # shards carry tensors of the SAME name as the merged one
+    # (...ple.ple_embedding.ngram_embedding.shard_0.weight, [2500012,160]
+    # fp8), so ple.merged.safetensors lost the race and the plan read one raw
+    # shard as the codes. "00-" wins the sort and the name.
+    out_name = "00-ple.merged.safetensors"
+    out_path = os.path.join(stage, out_name)
+    data_bytes = total_rows * hd * 2
+    hdr = {merged_name: {"dtype": BF16, "shape": [total_rows, hd],
+                         "data_offsets": [0, data_bytes]}}
+    blob = json.dumps(hdr, separators=(",", ":")).encode("utf-8")
+    pad = (-len(blob)) % 8
+    blob += b" " * pad
+    hlen = len(blob)
+    with open(out_path, "wb") as fh:
+        fh.write(struct.pack("<Q", hlen))
+        fh.write(blob)
+        fh.truncate(8 + hlen + data_bytes)
+    jobs = []
+    row0 = 0
+    for i in range(n_shards):
+        f_, off, ln, _, shp = meta[shard_ids[i]]
+        jobs.append((os.path.join(stage, f_), off, shp[0], out_path,
+                     8 + hlen + row0 * hd * 2, scale_f32, hd, 131072))
+        row0 += shp[0]
+    assert row0 == total_rows
+    with ctx.Pool(args.workers) as pool:
+        pool.map(fill_ple_shard, jobs)
+    log(f"PLE merged: {data_bytes / 2**30:.2f} GiB bf16 "
+        f"({total_rows} rows x {hd} cols)")
+
+    # ---- post-verify ------------------------------------------------------
+    verify(stage, prefixes, n_e, shapes, meta, shard_ids, rows, hd,
+           scale_f32, out_path, hlen)
+
+    # ---- rewrite index + config -------------------------------------------
+    new_wm = {k: v for k, v in wm.items() if k not in set(shard_ids.values())}
+    for idx, pre in enumerate(prefixes):
+        fname = f"stacked.{idx:03d}.safetensors"
+        for t in (f"{pre}.mlp.experts.gate_up_proj",
+                  f"{pre}.mlp.experts.gate_up_proj_scale_inv",
+                  f"{pre}.mlp.experts.down_proj",
+                  f"{pre}.mlp.experts.down_proj_scale_inv"):
+            new_wm[t] = fname
+    new_wm[merged_name] = out_name
+    total = sum(meta_size(meta, k) for k in new_wm
+                if k in meta)  # originals
+    # new tensors' sizes
+    total += written_total + data_bytes
+    index["weight_map"] = new_wm
+    if "metadata" in index and isinstance(index["metadata"], dict):
+        index["metadata"]["total_size"] = total
+    # ATOMIC REPLACE, never open(path, "w"): the stage's index is a symlink
+    # to the RAW checkpoint (staging symlinks everything but config.json) —
+    # open("w") would rewrite the raw file through the link. os.replace
+    # swaps the link itself for a real file; the raw tree stays pristine.
+    for path, payload in ((index_path, json.dumps(index, indent=2)),
+                          (cfg_path, json.dumps(cfg, indent=2))):
+        tmp = path + ".tmp"
+        with open(tmp, "w") as fh:
+            fh.write(payload)
+        os.replace(tmp, path)
+    n = set_split(cfg, 1)
+    log(f"config: split_ngram_parts=1 written at {n} location(s)")
+
+    with open(os.path.join(stage, "ple_scale.txt"), "w") as fh:
+        fh.write(repr(scale_f32) + "\n")
+
+    dt = time.time() - t0
+    log(f"staged in {dt / 60:.1f} min: {written_total / 2**30:.1f} GiB experts + "
+        f"{data_bytes / 2**30:.1f} GiB PLE; scale {scale_f32!r} -> ple_scale.txt")
+    return 0
+
+
+def meta_size(meta, name):
+    _, _, ln, _, _ = meta[name]
+    return ln
+
+
+def view_src_tuple(meta, name, stage):
+    f_, off, ln, _, _ = meta[name]
+    with open(os.path.join(stage, f_), "rb") as fh:
+        fh.seek(off)
+        return fh.read(ln)
+
+
+def find_split(cfg):
+    """Locate split_ngram_parts keys (recursive); return list of paths."""
+    hits = []
+
+    def walk(d, path):
+        if isinstance(d, dict):
+            for k, v in d.items():
+                if k == "split_ngram_parts":
+                    hits.append(path + [k])
+                walk(v, path + [k])
+    walk(cfg, [])
+    return hits
+
+
+def read_split(cfg):
+    hits = find_split(cfg)
+    if not hits:
+        return None
+    d = cfg
+    for k in hits[0][:-1]:
+        d = d[k]
+    return d[hits[0][-1]]
+
+
+def set_split(cfg, value):
+    hits = find_split(cfg)
+    for path in hits:
+        d = cfg
+        for k in path[:-1]:
+            d = d[k]
+        d[path[-1]] = value
+    if not hits:  # plugin's meta_int2 falls back to the top-level key
+        cfg["split_ngram_parts"] = value
+        hits.append(["split_ngram_parts"])
+    return hits
+
+
+def verify(stage, prefixes, n_e, shapes, meta, shard_ids, rows, hd,
+           scale_f32, out_path, hlen):
+    """Cheap strong checks: headers parse, sizes match, sampled bytes identical."""
+    f2, h = shapes["ffn2_h"]
+    f = shapes["ffn"][0]
+    gs, ds = shapes["gs"], shapes["ds"]
+    # headers of every new file
+    for idx, pre in enumerate(prefixes):
+        path = os.path.join(stage, f"stacked.{idx:03d}.safetensors")
+        hlen2, hdr = parse_header(path)
+        want = {
+            f"{pre}.mlp.experts.gate_up_proj": (F8, [n_e, f2, h]),
+            f"{pre}.mlp.experts.gate_up_proj_scale_inv":
+                (BF16, [n_e, gs[0] * 2, gs[1]]),
+            f"{pre}.mlp.experts.down_proj": (F8, [n_e, h, f]),
+            f"{pre}.mlp.experts.down_proj_scale_inv": (BF16, [n_e, ds[0], ds[1]]),
+        }
+        for name, (dt, shp) in want.items():
+            e = hdr[name]
+            assert e["dtype"] == dt and list(e["shape"]) == shp, name
+            assert e["data_offsets"][1] - e["data_offsets"][0] == \
+                int(np.prod(shp)) * (1 if dt == F8 else 2), name
+    _, phdr = parse_header(out_path)
+    e = phdr[next(iter(phdr))]
+    assert e["dtype"] == BF16 and list(e["shape"]) == [sum(rows), hd]
+
+    # sampled PLE rows: recompute and compare bytes exactly
+    lut = e4m3_lut()
+    out_off = 8 + hlen
+    for i in (0, len(shard_ids) // 2, len(shard_ids) - 1):
+        name = shard_ids[i]
+        f_, off, ln, _, shp = meta[name]
+        with open(os.path.join(stage, f_), "rb") as fh:
+            mm = mmap.mmap(fh.fileno(), 0, access=mmap.ACCESS_READ)
+            for r in (0, shp[0] - 1):
+                codes = np.frombuffer(mm, dtype=np.uint8, count=hd,
+                                      offset=off + r * hd)
+                q = lut.take(codes) * np.float32(scale_f32)
+                del codes  # release the mmap export before mm.close()
+                want = f32_to_bf16_rne(q).tobytes()
+                got_row = sum(rows[:i]) + r
+                with open(out_path, "rb") as ofh:
+                    ofh.seek(out_off + got_row * hd * 2)
+                    got = ofh.read(hd * 2)
+                assert got == want, f"PLE row {got_row} mismatch"
+            mm.close()
+    log("verify: new-file headers, sizes and sampled PLE rows all exact; "
+        "expert slabs were byte-checked in-stack")
+
+
+if __name__ == "__main__":
+    sys.exit(main())
@@ -204,12 +204,12 @@ for path in "${CHANGED[@]}"; do
        scripts/qwen-sharp-v22.4.0.jinja)
            add_test tests/frank_large_unit.bats
            ;;
-        # frank-small is llama.cpp on the 3090 as a ROOT unit: a native CUDA
+        # frank-secondary is llama.cpp on the 3090 as a ROOT unit: a native CUDA
        # binary, so no generated quadlet is involved. Its env, unit and serve
        # script all live in scripts/, which is why this case has to precede
        # the generic scripts/*.service and scripts/* fallbacks below.
-        scripts/frank-small.*|scripts/serve-frank-small.sh)
-            add_test tests/frank_small_unit.bats
+        scripts/frank-secondary.*|scripts/serve-frank-secondary.sh)
+            add_test tests/frank_secondary_unit.bats
            add_test tests/llm_units.bats
            add_test tests/deployment_scoping.bats
            add_test tests/scripts_syntax.bats
@@ -0,0 +1,87 @@
+#!/usr/bin/env python3
+"""Streaming absmax audit of the Swift1.5 PLE n-gram table (BF16).
+
+The PLE fp8 encoding uses ONE fixed scale (1.9931793212890625e-04, the
+official checkpoint's plateau value). fp8_e4m3 saturates at 448, so the
+fixed scale is legal iff absmax <= 448 * SCALE. This walks every
+ngram_embedding.shard_*.weight tensor in the checkpoint (mmap, no copy),
+computes the absmax from the BF16 bit pattern (clear the sign bit, max
+the remaining uint16 — BF16 magnitudes are monotone in their bits), and
+reports the headroom. Exit 0 only when headroom >= 1.5x.
+
+Run from any cwd EXCEPT /tmp on dankserver (/tmp/inspect.py shadows the
+stdlib). Usage: verify-swift15-ple.py <ckpt-dir>
+"""
+import json
+import mmap
+import os
+import struct
+import sys
+
+SCALE = 1.9931793212890625e-04  # bf16 0x3951, the official plateau scale
+FP8_E4M3_MAX = 448.0
+
+import numpy as np
+gmax = 0.0
+gfile = ""
+over_cap = 0
+total = 0
+
+ckpt = sys.argv[1]
+index = json.load(open(os.path.join(ckpt, "model.safetensors.index.json")))
+shards: dict[str, list[str]] = {}
+for k, f in index["weight_map"].items():
+    if "ngram_embedding.shard_" in k and k.endswith(".weight"):
+        shards.setdefault(f, []).append(k)
+
+files = len(shards)
+tensors = sum(len(v) for v in shards.values())
+print(f"n-gram shards: {files} files, {tensors} tensors")
+
+import math
+
+cap = FP8_E4M3_MAX * SCALE
+# largest bf16 <= cap: truncate the float32 bits (cap > 0); bf16
+# magnitudes are monotone in their bits, so |w| > cap <=> bits > cap_bits
+cap_bits = int((np.float32(cap).view(np.uint32) >> 16) & 0xFFFF)
+for fname, keys in sorted(shards.items()):
+    path = os.path.join(ckpt, fname)
+    with open(path, "rb") as fh:
+        hlen = struct.unpack("<Q", fh.read(8))[0]
+        header = json.loads(fh.read(hlen))
+        base = 8 + hlen
+        mm = mmap.mmap(fh.fileno(), 0, prot=mmap.PROT_READ)
+        for k in keys:
+            meta = header[k]
+            off, end = meta["data_offsets"]
+            a = np.frombuffer(mm, dtype=np.uint16, count=(end - off) // 2,
+                              offset=base + off)
+            v = a & 0x7FFF
+            m = int(v.max())
+            over_cap += int((v > np.uint16(cap_bits)).sum())
+            total += v.size
+            del a
+            if m > gmax:
+                gmax = m
+                gfile = k
+        mm.close()
+
+# rebuild the BF16 value: the 16 bits are the HIGH half of the float32
+absmax = np.frombuffer(b"\x00\x00" + np.uint16(gmax).tobytes(),
+                        dtype=np.float32)[0]
+
+print(f"absmax        = {absmax!r}  (tensor: {gfile})")
+print(f"2^log2        = 2^{math.log2(absmax):.3f}")
+print(f"fp8 cap       = 448 * {SCALE:.8g} = {cap:.8g}")
+print(f"headroom      = {cap / absmax:.3f}x")
+ok = cap / absmax >= 1.5
+print("VERDICT:", "PASS (>= 1.5x headroom over the fixed scale)" if ok
+      else "FAIL — fixed scale does not cover the Swift PLE")
+print(f"values > cap  = {over_cap} of {total} ({over_cap / total:.3e})")
+nxt_bits = int((np.float32(absmax / FP8_E4M3_MAX).view(np.uint32)
+                >> 16) & 0xFFFF) + 1
+nxt_scale = np.frombuffer(b"\x00\x00" + np.uint16(nxt_bits).tobytes(),
+                          dtype=np.float32)[0]
+print(f"next-bf16 scale covering absmax: {nxt_scale:.8g} "
+      f"(0x{nxt_bits:04x}, headroom {FP8_E4M3_MAX * nxt_scale / absmax:.4f}x)")
+sys.exit(0 if ok else 1)
@@ -0,0 +1,111 @@
+---
+type: reference
+title: Agent Docs and Edits
+description: Two self-inflicted damage classes — structurally broken markdown after parallel agent edits, and botched hashline PUT/CUT repairs — with the audit checks and recovery order for each.
+tags: [agents, docs, markdown, hashline, edit-recovery]
+timestamp: 2026-10-05T00:00:00-07:00
+---
+
+# Agent Docs and Edits
+
+## Doc-structure audit (after parallel agent edits)
+
+Use after one or more subagents have edited markdown docs (OKF/Obsidian
+vault with `---` frontmatter), before committing. Link checkers and
+completeness seams validate *references*, not *structure* — agents reliably
+produce structurally broken markdown that still links cleanly.
+
+Damage catalogue:
+
+1. Banner/prose prepended **above** the opening `---` → the frontmatter is
+   no longer frontmatter (title/tags/description become body text).
+2. A rewrite **appended** instead of merged → duplicate headings and a stray
+   second `---` mid-file.
+3. Literal model artifacts in the body: `<|channel>}<tool_call|>...`,
+   `+- `, `- - `.
+4. Wrapped bullets re-emitted with a `- ` prefix on every physical line.
+5. Line-range replacement that dropped a sentence tail → dangling fragment,
+   often mid-parenthesis.
+
+Checks (fast, run all):
+
+```bash
+head -3 <file>                          # must be `---` if the page has frontmatter
+grep -c '^---$' <file>                  # exactly 2 (opener + closer)
+grep -n '^## ' <file> | sort -t: -k2 | uniq -f1 -d   # duplicate sections
+grep -rnE '^\+- |^- - |<\|' <files>     # artifacts
+python3 scripts/check-vault-links.py <vault-dir>
+./tests/run.sh                          # docs completeness seam etc.
+```
+
+Repair order (matters):
+
+1. If the file was destroyed rather than edited: stash the agent version to
+   `.AGENTTRASH/<YYYYMMDD-topic>/`, then `git checkout HEAD -- <file>` and
+   re-apply only the intended changes by hand.
+2. If it merely duplicated: **salvage unique content first** — cut the
+   one-of-a-kind section, paste it into the copy you keep, THEN delete the
+   duplicate ranges. Never delete first.
+3. Re-add missing sentence tails (insert before the orphaned fragment),
+   keeping list continuation lines indented.
+4. Re-run all checks above.
+
+Gotchas: agents can write AFTER they appear finished (job settles, file
+changes) — re-verify structure after the last background job reports, not
+before. Editing an agent's already-damaged file with fresh line numbers is
+safe, but its snapshot tag changes on every edit — re-read before each
+subsequent edit. `.AGENTTRASH/` is gitignored: trash is local-only, so
+nothing moved there reaches the repo — that is intended (temporary hold, not
+archive).
+
+## Hashline edit recovery
+
+Failure mode: multi-op edits resolve every `PUT N.=M:` / `CUT N.=M:` index
+against the **original** file numbering, and the post-apply diff view
+**elides lines** (it shows an anchor window, not the whole file). Two traps
+follow:
+
+1. A second edit call built from a diff-view snapshot uses line numbers from
+   a *partial* view. Lines outside the window are invisible but still
+   counted.
+2. If the first apply was itself wrong, the file shifted. The diff view
+   tells you the damage, but not the true offsets.
+
+Result: the repair pastes fresh text **beside** stale text instead of over
+it.
+
+Procedure:
+
+1. **Never carry line numbers across calls.** Re-read the target range for a
+   fresh tag before every subsequent edit. A read is cheap; a duplicated
+   paragraph costs a commit and a revert.
+2. **One op per call when the previous apply was imperfect.** Fixed offsets
+   are only trustworthy when every preceding apply landed exactly as
+   specified.
+3. **Re-read after every apply**, and verify the whole affected region — the
+   diff view is a sample, not proof. Check the *boundaries*: the line before
+   the edit and the line after it. Damage shows up as a repeated sentence
+   straddling the seam.
+4. **Repair by CUT, not by re-PUT.** Once you know the exact stale lines
+   from a fresh read, delete them. Do not try to PUT a corrected superset
+   over a range whose extent you are guessing at.
+5. Verify with a content assertion, not a diff review: `grep -c` for the
+   duplicated sentence, plus a syntax/compile check on any touched script.
+
+Same rule for any imperative `edit` tool with an old/new match: the matched
+text must be the **entire** text being replaced. Matching a bare fragment
+inside a paragraph splices the replacement into the middle of the sentence
+and silently corrupts the file. Re-read the file afterwards to confirm.
+
+### Anchored in-page links
+
+Renaming a Markdown heading breaks `[[#Old Heading]]` anchors. When a
+section is retitled, grep for the old heading text and fix every anchor in
+the same pass — doc link-checkers frequently do not validate `#fragment`
+targets, so a green suite will not catch a dangling one.
+
+## Related
+
+- [[OMP Harness]] — the session context these repairs protect
+- [[Command Center Dotfiles]] — the repo these edits commit to
+- [[Testing Policy]] — the seams that catch broken vault structure
@@ -0,0 +1,87 @@
+---
+type: reference
+title: Command Center Dotfiles
+description: The ~/.omp repo is itself the deployment — stow-equivalent packages, full-replica rsync transport, the pre-overwrite audit for non-blank hosts, deploy.sh mechanics, and the fish/sshpass transport gotchas.
+tags: [dotfiles, omp, deploy, rsync, blankpad]
+timestamp: 2026-10-05T00:00:00-07:00
+---
+
+# Command Center Dotfiles
+
+Repo `~/.omp` is itself the deployment: packages are home-relative trees
+under `dotfiles/<pkg>/`; `scripts/deploy.sh` symlinks live files into the
+repo checkout. `git pull` + `deploy.sh` is the whole propagation story.
+
+## Transport decision (agreed with Frank, executed 2026-09-09)
+
+- Ship via **full-replica rsync of `/home/dank/.omp/` including `.git`** to
+  `<user>@<host>:/home/<user>/.omp/`, NOT clone-from-Gitea — `main` can be
+  ahead of `origin/main`; push is a user-only trigger, so clone would ship
+  stale state.
+- Deploy runs on-target as the target user: `bash
+  /home/<user>/.omp/scripts/deploy.sh` (resolves the repo via `git -C
+  $(dirname $(realpath $0)) rev-parse --show-toplevel`, so `.git` must be
+  present at the target path — full-replica transport guarantees this).
+
+## Pre-overwrite audit (MANDATORY before first sync on a non-blank host)
+
+Diff every file that `deploy.sh --check` reports as DRIFT against the repo
+copy BEFORE letting quarantine-and-replace run. Host-specific divergence
+belongs in an override package, not in quarantine:
+
+1. Pull divergent live files from target
+   (`ssh ... 'cat <path>' > local-staged-copy`).
+2. Secret-audit staged copies (grep token/secret/password/key shapes)
+   before anything enters git.
+3. Compose `dotfiles/zz-<hostname>/` on the source machine from those live
+   copies; commit to repo first.
+4. Ship the package dir via rsync, then run deploy sync — override deploys
+   LAST and win over base packages.
+
+## deploy.sh mechanics (verified on blankpad)
+
+- `--check` = pure report (ADOPT/LINK/DRIFT lines), writes nothing. Run it
+  first, always.
+- Default mode = sync: adopt identicals silently, link new,
+  quarantine-and-replace divergents into `.quarantine/<host>/<ts>/`
+  (gitignored; never rm).
+- **Override-target bug (fixed cc2c652):** `zz-<host>` packages nest
+  base-package dirs internally (`zz-blankpad/niri/.config/...`); target
+  derivation must strip the nested package segment, else overrides deploy to
+  phantom `$HOME/niri/.config/...` while real targets keep pointing at
+  canonical copies (visible symptom: check shows 0 actions but
+  `readlink ~/.zshrc` points at base `zsh/.zshrc`).
+- systemd units auto-enable only when ExecStart binary resolves on that
+  host; dconf packages load via `dconf load /`, never symlinks.
+- `core.hooksPath=.githooks` is armed by deploy.sh per machine — set it once
+  post-transfer even before first sync.
+
+## ssh/sshpass transport gotchas (all bit live)
+
+- Resolve the target via mDNS FQDN (`avahi-resolve -4 -n <host>.local`); a
+  bare hostname may not resolve. Drop-not-refuse on port 22 = sshd down or
+  firewall default-drop, not auth failure — needs a local/console fix on
+  target.
+- `SSHPASS='...' sshpass -e ssh ...` inline per invocation; an env passed
+  via the tool's env parameter did not propagate to the sshpass child
+  process.
+- The remote login shell may be **fish**: never send bash syntax
+  (`for/do/done`, `$?`) bare through `ssh host 'cmd'`; wrap as `bash -c
+  "..."` or ship a script. See [[SSH and Remote Access]].
+- File shipping: `ssh 'cat > /remote/path' < localfile` (stdin redirect). A
+  here-doc attached to the ssh command is consumed by `cat` — the intended
+  script body becomes the file content, silently. Verify shipped bytes with
+  an md5sum compare on both ends; do NOT compare `/dev/stdin` after `cat`
+  drained it (false DIVERGED).
+
+## Verification of a completed deploy
+
+Re-run `deploy.sh --check` → expect exactly zero actions/quarantined/skipped
+= idempotent steady state. Spot-check `readlink` on override files points
+into `dotfiles/zz-<host>/`, not base packages.
+
+## Related
+
+- [[SSH and Remote Access]] — the transport mechanics this deploy rides
+- [[Noctalia and Niri]] — the desktop packages this repo ships
+- [[Deploy State Triage]] — the serverAppConfig deploy flow (different repo)
@@ -0,0 +1,274 @@
+---
+type: reference
+title: OMP Harness
+description: The omp agent harness as configured on this fleet — compaction chain arithmetic, per-model sampler routing via compat.extraBody, and the static-preamble/skill-library audits that keep session context lean.
+tags: [omp, harness, compaction, models-yml, context, samplers]
+timestamp: 2026-10-05T00:00:00-07:00
+---
+
+# OMP Harness
+
+The harness reaches the fleet through `~/.omp/agent/models.yml` (model
+registry; single source of truth — there is no copy on the servers) and
+`~/.omp/agent/config.yml` (roles, compaction, global samplers). Nothing in
+this file is repo-managed; edits are harness-side only.
+
+**Model ids are the StarCraft unit names (official 2026-10-08, on the wire
+since 2026-10-09):** `frank-archon` (dankserver :7770, radiance),
+`frank-dragoon` (dankclaw :7770, flash-server), `frank-zealot` (dankclaw
+:7771, the 3090 compaction hub) — formerly `frank-primary`, `frank-large`,
+`frank-secondary`. The SERVICES keep the old names (units, scripts, env
+files, runbook pages); pages written in the old era name the service,
+the harness names the model. The wire now matches: each endpoint's
+`/v1/models` answers with the Protoss name, so `models.yml`, the OpenWebUI
+picker/grants, the homepage docs and the raw API all use one id per model
+(see [[OpenWebUI]] for the grant re-key and the [[frank-primary-radiance-swift15-w4dn8]]
+note on radiance echoing the requested name).
+
+## Compaction chain arithmetic (verify before ANY window change)
+
+When a fleet endpoint's context grows past its harness-declared window, OMP
+compacts by sending the conversation to a designated target model. Before
+changing any `contextWindow`/`maxTokens`, verify the chain arithmetically —
+a target whose window is smaller than the source's trigger point silently
+breaks compaction.
+
+Where the contract lives:
+
+- `~/.omp/agent/config.yml` → `compaction.remoteEndpoint` (target URL),
+  `methodOrder`, `thresholdPercent` (70), idle settings
+- `~/.omp/agent/models.yml` → per-model `contextWindow`, `maxTokens`,
+  `compactionModel` references
+- Implementation: omp plugin `dist/core/compaction/compaction.js`
+
+The arithmetic (the coupled contract every endpoint's maxTokens comment
+encodes):
+
+1. **Source trigger** = `thresholdPercent × source contextWindow`
+   (0.70 × 200,704 = 140,492 for frank-primary on the current shape)
+2. **Output budget** = capped at the target model's `maxTokens`
+   (32,768 for frank-secondary; 36,864 frank-primary; 57,344 frank-large — the
+   largest, because halogen's `HALOGEN_MAX_TOKENS_CAP` must stay ≥ it)
+3. **REQUIREMENT**: prompt (≤ trigger) + budget ≤ the target's per-slot
+   window. Worked pairs from the live registry: 140,492 + 36,864 = 177,356 ≤
+   200,704 (frank-primary); 140,490 + 57,344 = 197,834 ≤ 200,700
+   (frank-large); 140,492 + 32,768 = 173,260 ≤ 200,704 (frank-secondary).
+
+Procedure (recompute from the LIVE files every time — windows rotate with
+every swap; the numbers above are the 2026-10-05 state):
+
+1. Read the `config.yml` compaction block + `models.yml` entries for source
+   and target. Every endpoint's `contextWindow` comment carries its own
+   arithmetic — read them.
+2. Read the target checkpoint's `max_position_embeddings` (native window) —
+   the served window may be a concurrency contract, not the native limit
+   (`--parallel N` DIVIDES the pool; the per-slot number is the contract).
+3. If native ≥ trigger + budget, reslice the pinned pool:
+   `MAX_RUNNING × CONTEXT_LENGTH ≤ MAX_TOTAL_TOKENS` (the pin exists because
+   the runtime's autofit leaves too little for deferred loads + prefill
+   workspace; check `nvidia-smi` headroom — expect ~0.8 GB workspace margin).
+4. Changes: host env (CONTEXT_LENGTH-style env tunables) via git →
+   `./tests/run.sh` → deploy; harness `models.yml` directly (no repo copy).
+5. Verify: `/health` 200, boot log's pool line, then a long-prompt probe at
+   ≥ trigger height (the compaction-fit guarantee), then concurrent requests.
+
+Known reconciliation trap: comments in models.yml have described compaction
+as running "on the dankserver endpoint" while `remoteEndpoint` pointed at
+dankclaw — the comment/config disagreed. The comments now name the real
+target (`compaction.remoteEndpoint` = dankclaw :7771). **The config is
+truth; fix comments in the same change.** `maxTokens` never changes for a
+reslice — it caps output; `contextWindow` caps input. Boot-shape OOM
+laddering is a different problem ([[Llama Cpp Serving Sizing]]).
+
+## Per-model sampler config (`compat.extraBody`)
+
+`models.yml` has **no per-model sampler keys**. Sampling is global
+(`config.yml`: `temperature`, `topP`, `topK`, `minP`, `presencePenalty`,
+`repetitionPenalty`; default `-1` = **not sent**). The per-model hook is
+`compat.extraBody`:
+
+**Why it works:** `applyOpenAIExtraBody` runs **last** in the
+openai-completions builder — after `applyChatCompletionsCompatPolicy` — via
+`Object.assign(params, extraBody)`. So extraBody overrides global settings
+and compat-policy output.
+
+```yaml
+compat:
+  reasoningEffortMap:
+    max: xhigh
+  extraBody:
+    temperature: 1.0
+    top_p: 0.95
+    top_k: 20
+    min_p: 0.0
+    presence_penalty: 0.0
+    repetition_penalty: 1.0
+    preserve_thinking: true
+```
+
+Use **wire names**, not OMP camelCase. Two traps:
+
+1. **`preserve_thinking` is not user-settable as a compat key.**
+   `qwenPreserveThinking` is auto-derived in `buildOpenAICompat` from the
+   *auto-detected* `thinkingFormat` + local baseUrl, **before**
+   `applyCompatOverrides`. Setting `compat.thinkingFormat: qwen` in
+   models.yml does **not** enable it. Custom-named models (`frank-large`)
+   match no Qwen id pattern → use `extraBody: {preserve_thinking: true}`.
+2. **Never put `chat_template_kwargs` in `extraBody`** where a compat policy
+   also sets it — `Object.assign` is shallow and replaces the whole object,
+   silently dropping the `reasoning_effort` kwarg the compat policy set.
+   Top-level `preserve_thinking` is honoured by halogen. (frank-primary's
+   FreeToken entry currently carries a `chat_template_kwargs` extraBody with
+   a comment asking to verify-and-prune — the exact trap to check at that
+   endpoint's next swap.)
+
+**Gotcha: silent greedy decode.** halogen has **no server-side sampler
+defaults** — `temperature` absent or `0` = greedy. If OMP sends nothing, the
+endpoint decodes greedy while looking configured. This is why frank-large's
+entry carries the recipe on the wire and a comment saying "do not drop it".
+
+**Reasoning-effort folding.** Qwen3.8's template accepts only
+`{xhigh (default), medium, low}` for `reasoning_effort` and 400s on anything
+else ("Unexpected reasoning effort high"). Every endpoint's
+`reasoningEffortMap` folds all harness levels onto that set; a role whose
+level is unmapped fails on its FIRST request, and the failure names Jinja,
+not the model.
+
+### Verify the exact wire body (echo-capture)
+
+Don't trust the YAML — capture what OMP actually emits:
+
+1. Echo server on `127.0.0.1:18099` that dumps the JSON body to a file and
+   returns a canned `chat.completion` (include `reasoning_content` so
+   thinking paths exercise).
+2. Copy the **real** models.yml into a throwaway profile with only the
+   baseUrl swapped:
+   ```bash
+   sed 's|http://<host>:7770/v1|http://127.0.0.1:18099/v1|' \
+     ~/.omp/agent/models.yml > ~/.omp/profiles/wiretest/agent/models.yml
+   ```
+3. Run and inspect:
+   ```bash
+   omp --profile wiretest --model "<provider>/<model>" -p "hi" \
+     --no-tools --no-session --no-extensions --no-skills --no-rules
+   jq 'del(.messages, .tools)' last-body.json
+   ```
+   Confirm every intended field is present with the right value.
+
+**Verify a field is *honoured*, not just accepted** — unknown fields often
+return 200 anyway. Prove behavior with a **prompt-token delta**: replay an
+assistant turn carrying a long `reasoning_content`, then compare
+`usage.prompt_tokens` with the flag true vs false. A large delta means the
+field changed template rendering; zero delta means it was ignored.
+
+Cleanup: trash the probe profile and echo server (`profiles/.AGENTTRASH`),
+never `rm`.
+
+## Static-preamble audit ("the context feels bloated")
+
+What loads statically every session (measure each, don't guess):
+
+1. Repo rules file — `wc -c AGENTS.md` (baseline 2026-10-05: 72,179 B, ~18k
+   tokens — historically the largest block).
+2. Skill index — only names+descriptions load; bodies are on-demand. Sum
+   description bytes across all SKILL.md frontmatter:
+   ```bash
+   python3 - <<'EOF'
+   import glob, re, os
+   for root in [os.path.expanduser("~/.omp/agent/managed-skills"),
+                os.path.expanduser("~/.omp/agent/skills"), ".omp/skills"]:
+       n=tot=0
+       for sk in glob.glob(root+"/*/SKILL.md"):
+           fm=re.search(r'^---.*?---', open(sk, encoding="utf-8", errors="replace").read(), re.S)
+           d=re.search(r'description:\s*(.*(?:\n(?:\s{2,}.*)*)?)', fm.group(0) if fm else "")
+           n+=1; tot+=len(d.group(1)) if d else 0
+       print(root, n, tot)
+   EOF
+   ```
+   Baseline 2026-10-05: 171 managed skills / ~38 KB descriptions + 9 user
+   skills.
+3. Memory lessons — `find ~/.omp/agent/memories -name 'MEMORY.md' -o -name
+   'memory_summary.md' | xargs wc -c`. The injected block ≈ the largest
+   MEMORY.md.
+4. Harness/system prompt + tool schemas + CLI persona: fixed cost, ~20 KB;
+   not prunable.
+
+Known rot patterns:
+
+- Near-duplicate auto-captures of the same lesson (three "authentik ORM"
+  entries, the freetoken-build pair, the three job-runner skills) — dedupe
+  via `manage_skill delete`/`update`.
+- Corrupted captures containing stray CJK characters (`一团ly`, `等方法`)
+  mid-sentence — a known defect of the auto-capture path; delete the corrupt
+  copy, keep the clean one.
+- Stale host/service names in consolidated memory (pre-rename
+  `frank-default-large`) — the repo's AGENTS.md topology table is ground
+  truth.
+
+Fix strategy: AGENTS.md prose (measured runbook narratives) → move to
+`serverAppConfigs-docs/LLM/` runbooks, keep laws + tables in AGENTS.md (the
+repo's own docs-tree rule mandates this). Managed skills live in
+`~/.omp/agent/managed-skills` (safe to edit); user-authored skills in
+`~/.omp/agent/skills` are NEVER touched by tooling. Memory store edits:
+stage removed captures to trash first (`~/.AGENTTRASH/YYYYMMDD-topic`).
+
+Baseline to beat (2026-10-05): ~35–40k tokens of preamble — AGENTS.md ~18k,
+skill index ~10k, lessons ~3.5k, harness ~6k. Proposed reductions: −12k
+(AGENTS.md slimming), −3k (skill dedupe).
+
+## Managed-skill library hygiene
+
+Use when auditing, pruning, or consolidating `~/.omp/agent/managed-skills`
+(the auto-captured skill store; the full index is injected into EVERY
+session's context, so dead or duplicate entries are a permanent tax).
+
+The static-context budget (measured 2026-10-05):
+
+| Block | Size | Notes |
+|---|---|---|
+| `AGENTS.md` (repo) | 72,179 B ≈ 18k tok | biggest single item |
+| Skill index (~173 skills) | ~40 KB ≈ 10k tok | names+descriptions only |
+| Learned-lessons block | ~14 KB | can contain corrupted/duplicate captures |
+| Harness system prompt + tool schemas | ~23 KB | fixed |
+
+Audit procedure:
+
+1. Build an inventory: per skill, name, mtime, body word count, description,
+   first 12 headings (one pass, out to a file; read in pages).
+2. Group into: retired-runtime (host/service gone; its runbook already lives
+   in `serverAppConfigs-docs/LLM/`), verbatim-duplicate pairs/triples (same
+   lesson captured twice — often the SAME DAY), same-lesson families
+   (N skills = one procedure), keep.
+3. Read full bodies of every merge candidate before proposing a merge;
+   description alone proves overlap only for exact dupes.
+4. Present A/B/C/D verdicts for user approval. NEVER execute before the
+   discussion concludes — the user gates execution explicitly.
+5. On approval: stage everything to trash first
+   (`~/.AGENTTRASH/YYYYMMDD-<topic>/`), never rm. After merging: recreate
+   the surviving skill via `manage_skill` update with body = superset,
+   delete absorbed ones.
+
+Known rot classes (this library, 2026-10-05):
+
+- Same-day duplicate captures (freetoken-build-dankclaw vs
+  freetoken-dankclaw-build; the three job-runner skills).
+- CUDA-skill vs its self-declared HIP twin (pcie-link-capability-probe vs
+  pcie-link-probe-hip-rdna4) → merge.
+- Era ghosts: skills naming retired runtimes (freetoken, lucebox, r9v/
+  radiance, old halogen-flash-server image, ik_llama, GLM-5.3 experiment).
+- Skills contradicting user standing orders (live-bench-dashboard vs the
+  2026-09-29 delete-dashboards order) — flag for user, don't assume.
+- Memory store: corrupted captures with CJK noise from broken ssh commands;
+  stale host/service renames.
+
+Sizing verdicts that worked: bench family → 2 skills (harness vs
+discipline), not one mega-skill; keep profiler/kernel-time attribution
+standalone; fold go/no-go arithmetic skills into their probe skill
+(heterogeneous-accelerator-split-triage → PCIe probe).
+
+## Related
+
+- [[Agent Docs and Edits]] — editing vault docs and hashline targets safely
+- [[Command Center Dotfiles]] — the `~/.omp` repo deploy this config rides
+- [[Llama Cpp Serving Sizing]] — the engine-side sizing the chain must fit
+- [[LLM Bench Windows and Measurement]] — measuring the endpoints this config routes
@@ -0,0 +1,23 @@
+---
+type: index
+description: Pages about the agent harness itself — omp config, vault editing discipline, and the dotfiles repo that deploys the agent's own home.
+tags: [agent, index]
+timestamp: 2026-10-05T00:00:00-07:00
+---
+
+# Agent
+
+Pages about the agent's own tooling and context — how omp is configured, how
+docs get edited safely, and how the agent's home is deployed.
+
+- [[OMP Harness]] — compaction chain arithmetic, per-model samplers via
+  `compat.extraBody`, preamble and skill-library audits
+- [[Agent Docs and Edits]] — doc-structure audits after parallel edits;
+  hashline edit recovery
+- [[Command Center Dotfiles]] — the `~/.omp` repo-as-deployment, rsync
+  transport, override packages
+
+## Related
+
+- [[Testing Policy]] — the seams these pages' checks ride
+- [LLM](../LLM/index.md) — the fleet these agents operate
@@ -0,0 +1,151 @@
+---
+type: reference
+title: Deploy State Triage
+description: Reading and reconciling the deploy-state markers — stale marker vs live container, pending-service lockout, interrupted/raced deploys — without restarting healthy heavy endpoints.
+tags: [deployment, deploy-state, systemd, podman, triage]
+timestamp: 2026-10-05T00:00:00-07:00
+---
+
+# Deploy State Triage
+
+The deploy-state store is `~/.local/state/serverAppConfigs/deploy-state/<svc>`
+on each target (rootless, user `dank`, format exactly two lines
+`revision=<sha>` / `status=deployed|pending`, written atomically via tmp file
+ `mv`). It is written ONLY by a `deploy.sh` dispatch.
+
+## 1. Stale marker vs live container
+
+**Why it happens:** an out-of-band relaunch — reboot, `systemctl restart`, or
+a restart after the checkout ff-pulled to a newer revision — starts the
+container from the CURRENT `scripts/<svc>.env` but does NOT advance the
+marker. So `marker != HEAD` means "not dispatched via deploy.sh", NOT
+"behavior pending".
+
+**Why it matters:** a plain `deploy.sh` run re-selects and restarts a service
+whose marker lags. On llama.cpp/R9V endpoints that is a ~16–19 min reload of
+a 100+ GiB payload — and frank-primary (dankserver:7770) is also the
+harness's own model backend, so a blind deploy is a self-inflicted outage for
+possibly zero behavioral change.
+
+**Procedure (read-only first):**
+
+1. Committed state: `git log --oneline main..<branch>` empty ⇒ merged; the
+   operational delta is usually one `scripts/<svc>.env`.
+2. Probe the target (ssh as `dank`, passwordless sudo present):
+   - `cat ~/.local/state/serverAppConfigs/deploy-state/<svc>`
+   - `git -C ~/serverAppConfigs rev-parse HEAD`
+   - `systemctl is-active/is-enabled <svc>`;
+     `curl -s -o /dev/null -w '%{http_code}' http://127.0.0.1:<port>/health`
+   - the on-host env file already carries the new keys (it landed via the
+     ff-pull).
+3. **Decisive — read the LIVE container, not the marker.** Use `podman` as
+   `dank` (rootless store; bare `docker` is NOT the shim → "No such object";
+   `sudo podman` inspects an empty store):
+   - `podman inspect <ctr> --format '{{.State.StartedAt}}'` — booted AFTER
+     the pull ⇒ it read the new env
+   - `podman inspect <ctr> --format '{{.Args}}' | tr ' ' '\n' | grep -i <flag>`
+   - engine env: `podman exec <ctr> sh -c 'for f in /proc/[0-9]*/environ; do
+     tr "\0" "\n" <$f 2>/dev/null; done | grep -i <VAR>'`
+   - confirm the launcher maps each tunable to its flag
+     (`grep scripts/launch.sh`), e.g. `R9V_ENABLE_PREFIX_CACHING=1 →
+     --enable-prefix-caching`.
+
+**Decision:** live argv+env == target config AND `/health` 200 ⇒ behavior is
+live — **do NOT restart**; reconcile the marker (below). Only if the live
+container lacks the flag/env (or predates the pull) is a real
+`deploy.sh <target> --service <svc>` warranted.
+
+**Reconcile (no restart)** — mirror the deployer's atomic idiom:
+
+```bash
+S=$HOME/.local/state/serverAppConfigs/deploy-state
+H=$(git -C ~/serverAppConfigs rev-parse HEAD)
+t=$(mktemp "$S/.<svc>.XXXXXX"); printf 'revision=%s\nstatus=%s\n' "$H" deployed > "$t"; mv -f -- "$t" "$S/<svc>"
+```
+
+Then prove the gate: `./tests/run.sh` + `tests/e2e/run.sh` (needs
+`tests/e2e/.env.local`; expect the documented `11-admin` red if the
+FrankBook-Pro Mac mini 192.168.0.112 is ASLEEP — confirm with `ping -c2` +
+a `/dev/tcp` probe: 100 % loss + timeout = host off, not a regression).
+Log to `serverAppConfigs-docs/log.md`; commit + push (docs-only ⇒ deploy
+no-op).
+
+**Anti-patterns:** trusting a commit message ("X deployed") or the marker
+alone — both are bookkeeping; the container's argv/env is ground truth.
+Reading container state with `sudo podman` / bare `docker`. Restarting a
+healthy heavy endpoint to satisfy a stale marker.
+
+## 2. Pending lockout
+
+Symptom: `deploy.sh <target>` fails or silently stops with
+`service <name> is not deployed on <target>` and the `[3/3]` e2e gate never
+appears.
+
+1. **Get the real exit code first:** `./scripts/deploy.sh X 2>&1 | tail`
+   reports TAIL's 0 — use `set -o pipefail` or redirect to a file, then
+   `echo $?`.
+2. Inspect the state store (no sudo):
+   `ssh <target> 'cat ~/.local/state/serverAppConfigs/deploy-state/<name>'`.
+   A `status=pending` entry for a service absent from `service_is_allowed`
+   in `scripts/deploy-remote.sh` = retired-service lockout.
+3. Fixed behavior (since 02a7876, 2026-09-17): `add_pending_services` prunes
+   not-allowed pending entries to
+   `~/.AGENTTRASH/<YYYYMMDD>-retired-deploy-state/<service>` and the run
+   continues. Verify the WARN line names the service and the state file is
+   gone.
+4. On a host without the fix: move the stale entry to
+   `~/.AGENTTRASH/<date>-retired-deploy-state/` manually (never `rm`), re-run.
+5. Verify: deploy reaches `[3/3] Running Playwright e2e gate` and completes;
+   `deploy-state/` no longer holds the retired name.
+
+Related contracts: the superseded seam
+(`deployment_scoping.bats` "remote runner recognizes a pending service at
+the same revision") now asserts prune-and-continue (exit 0), not exit 2;
+`deployment_flow.bats` extracts `add_pending_services` via sed and
+HOME-overrides the trash path. **When retiring a service going forward:**
+delete or reconcile its deploy-state entry as part of the retirement commit —
+the prune is a safety net, not the intended cleanup path.
+
+## 3. Interrupted or raced deploys
+
+The deploy is TWO processes: `deploy.sh` (dev box) and `deploy-remote.sh`
+(runs on the target over ssh). **Killing the dev-side job does NOT kill the
+host-side one** — two `deploy-remote.sh` processes can end up managing the
+same unit, one deploying a revision you already superseded. Check:
+`pgrep -af "deploy-remote"` on the target, read the revision argument on
+each, kill stale ones by explicit PID.
+
+A killed deploy leaves behind:
+
+- the unit **stopped/failed** (`Restart=always` does NOT fire after an
+  explicit stop — start it yourself);
+- the state marker **pending** (pre-deploy or in-flight revision);
+- the e2e gate un-run (the deployer runs it last).
+
+Reconcile the marker ONLY after the service itself is verified healthy — a
+marker that runs ahead of reality is worse than one that lags (the planner
+uses it to decide what is pending). Then close the loop:
+
+1. Verify against MEASURED, config-specific numbers (KV capacity line, arena
+   slots, argv flags, `NRestarts`), not just health 200.
+2. Run the gate the dead deploy skipped (`tests/e2e/run.sh` from a checkout
+   that has the gitignored `tests/e2e/.env.local` — a fresh `git worktree`
+   does NOT have it; copy it in first).
+3. If the killed revision was restoring a known-good shape, confirm the
+   numbers match the last measured-good boot of that shape — equality is the
+   proof, not health alone.
+
+Boot failures must be read from the journal, not `podman logs`: serve scripts
+`podman rm -f` the container on every start (previous attempt lost), and
+`podman logs` on a container still *Stopping* returns the OLD configuration's
+numbers — which looks exactly like your change having no effect. Key on
+container identity (`podman ps -a --format '{{.CreatedAt}}'`) and read the
+cause from `journalctl --since "N min ago" | grep -a "<container-name>\["`.
+See [[vLLM Boot Forensics and Config Interrogation]] for the full capture
+procedure.
+
+## Related
+
+- [[Deployment Flow]] — the deploy pipeline itself
+- [[Quadlet Lifecycle]] — unit ownership and drift diagnosis
+- [[vLLM Boot Forensics and Config Interrogation]] — journal capture
@@ -63,7 +63,7 @@ service on the other host.
 | Dankclaw compose services | `podman-compose -f dankclaw/compose.yaml up -d --no-deps NAME` | Recreates only `NAME`; Caddy is handled separately because it fronts both hosts. |
 | SearXNG, Gluetun, Valkey, OpenWebUI, Pi-hole, Gitea database/server | user Quadlet + `systemctl --user` | Installs the selected unit and restarts only the required dependency-coupled units. Gitea's web proxy is the canonical Caddy compose service. |
 | `frank-large` | systemd + `systemctl` (root unit) | Relinks and restarts only the large inference service; waits up to 20 minutes for `/health`. |
-| `frank-small` | systemd + `systemctl` (root unit) | Relinks and restarts only the small inference service; waits up to 20 minutes for `/health`. |
+| `frank-secondary` | systemd + `systemctl` (root unit) | Relinks and restarts only the small inference service; waits up to 20 minutes for `/health`. |
 | `frank-medium` | rootless Quadlet + `systemctl --user` | Installs `dankserver/frank-medium/quadlet/frank-medium.container`; on first deploy retires the llama.cpp root unit and the vLLM replica quadlets into the home trash, then restarts the generated unit. Waits up to 20 minutes for `/health`. |
 | Dankserver media stack (RomM trio, Jellyfin, Calibre-Web, Copyparty) | user Quadlet + `systemctl --user` | Installs `dankserver/<svc>/quadlet/*.container`, restarts the generated unit. The first cutover per service is idempotent: it retires the compose-era bare container, rescues any anonymous volume into the data root, and merge-moves `/opt/<svc>` to `/home/dank/Containers/<svc>` (mapped-uid roots move without chown). Steady state is restart-only. |

@@ -20,11 +20,12 @@ user sees it; the `bats` seams assert config shape, not behavior.

 | Path | Role |
 |------|------|
-| `playwright.config.ts` | four projects (below), real-browser UA, trace on failure |
+| `playwright.config.ts` | five projects (below), real-browser UA, trace on failure |
 | `specs/00-auth.setup.ts` | setup project: two real authentik logins, saves storage states |
 | `specs/10-homepage.spec.ts` | user panel set + open links derived from `services.json` (registry-derived — no hardcoded service lists) |
 | `specs/11-admin.spec.ts` | `admin` project: admin grid + `/admin/` RBAC (403 for standard user, members-only 200) |
 | `specs/12-crawl.spec.ts` | every user open link reaches its real app through the guard chain |
+| `specs/19-frankbook.spec.ts` | `frankbook` project: walks the six admin links that terminate on FrankBook-Pro (192.168.0.112) — conditional, see below |
 | `specs/13-sso.spec.ts` | direct guarded-vhost visits resolve silently with ONLY the homepage session (single-login enforcement) |
 | `specs/14-guard.spec.ts` | `guest` project, no storage state: unauthenticated traffic lands on the IdP login |
 | `lib/login.ts` | `akLogin` helper (two-stage flow, shadow-DOM locators) |
@@ -38,7 +39,16 @@ user sees it; the `bats` seams assert config shape, not behavior.
  `.auth/qa-e2e{,-admin}.json` storage states.
 - **standard** — 10/12/13 with the user state.
 - **admin** — 11 with the admin state.
- **guest** — 14 with an empty state.
+- **guest** — 14/15 with an empty state.
+- **frankbook** — 19 with the admin state. **Excluded from every default run**: the
+  *arr/qBittorrent links terminate on FrankBook-Pro, which sleeps when idle, and a
+  sleeping Mac mini must never fail an unrelated deploy. The project joins only
+  when `E2E_INCLUDE_FRANKBOOK=1` (set by `scripts/deploy.sh` when a frankbook
+  trigger path changed: `tests/e2e/specs/19-frankbook.spec.ts`, `frankbook-pro/**`,
+  `shared/homepage/services.json`, or the `dankclaw/Caddyfile` serving the arr
+  vhosts) or when an explicit `--project` argument names it. An `11-admin` edit
+  that touches the arr links must therefore be paired with a `19-frankbook` run
+  against an awake host before it is trusted live.

 Credentials live in `tests/e2e/.env.local` (git-ignored; placeholders in
 `.env.local.example`). The scaffold bats seam fails the commit if creds or
@@ -47,8 +57,9 @@ auth states are ever tracked.
 ## Run

 ```bash
-cd tests/e2e && npx playwright test            # all projects
-npx playwright test --project=standard 12-crawl.spec.ts   # one slice
+tests/e2e/run.sh                          # default set — frankbook project excluded
+E2E_INCLUDE_FRANKBOOK=1 tests/e2e/run.sh  # default set + the frankbook walk
+cd tests/e2e && npx playwright test       # ALL projects, frankbook included
 ```

 Requires LAN reachability of `www.franksdank.cloud` (Pi-hole DNS or public).
@@ -0,0 +1,209 @@
+---
+type: reference
+title: Quadlet Lifecycle
+description: The full Quadlet discipline on this fleet — ownership auditing, compose-to-Quadlet cutover, drift/crash-loop diagnosis, the podman 5.x v1 generator quirks, log retrieval, and compose bring-up with ground truth.
+tags: [podman, quadlet, systemd, boot-persistence, compose, drift]
+timestamp: 2026-10-05T00:00:00-07:00
+---
+
+# Quadlet Lifecycle
+
+Fleet rule (AGENTS.md "Boot Persistence & Self-Healing"): every deployed,
+working container MUST have a self-healing, boot-persistent Quadlet unit —
+`.container` file in the repo (`<host>/<svc>/quadlet/`), installed by the
+deployer, `[Install] WantedBy=default.target`, `Restart=always`, linger on.
+A container that runs only from compose dies silently at reboot. This page
+is the operating manual for that lifecycle.
+
+## 1. Ownership audit — find the containers that will not survive a reboot
+
+```bash
+for c in $(podman ps --format '{{.Names}}'); do
+  printf '%-26s %s\n' "$c" "$(podman inspect $c --format '{{index .Config.Labels "PODMAN_SYSTEMD_UNIT"}}')"
+done
+```
+
+- **label present** → the unit owns it; persistence = `[Install]` + `loginctl
+  show-user <user> -p Linger` = yes.
+- **label absent, compose service exists** → the classic gap: dies at reboot.
+  Check the publish address while there — compose forms often bind the
+  wildcard (an unauthenticated UI/API on every interface).
+- **label present AND a compose service with the same name** → two owners; a
+  stray `podman compose up` can create a duplicate that fights the unit for
+  the name and the port. Remove the compose service, leave an explanatory
+  comment (repo convention).
+
+## 2. Cutover: compose service → Quadlet unit
+
+Checklist (precedents to copy:
+`dankclaw/{caddy,gitea,searxng,open-webui,seerr}/quadlet/`,
+`dankclaw/observability/quadlet/`):
+
+1. Unit file at `<host>/<svc>/quadlet/<name>.container`, mirroring compose
+   line for line: `Image=` (`localhost/` prefix for locally built images +
+   `Pull=never`), `ContainerName=` (Caddy and scrape configs address
+   containers by name), `Network=`, `User=0:0` when the image's own user
+   cannot write a rootless-owned bind mount, `Volume=` (`:ro` preserved),
+   `Exec=` for compose `command:`, `PublishPort=`, health keys,
+   `[Service] Restart=always` + `RestartSec=10`, `[Install]
+   WantedBy=default.target`. **Never `systemctl --user enable` the generated
+   unit** — `[Install]` + linger IS the persistence (`is-enabled` = `generated`
+   is healthy; explicit enable fails with "transient or generated").
+2. Exposure: publish on `127.0.0.1` unless something off-host genuinely needs
+   it (Caddy reaches containers by name over the shared network; the wildcard
+   publish is a guard-chain bypass).
+3. Handler in `scripts/deploy-remote.sh`: `install_quadlet <dir> <name>` →
+   `systemctl --user restart <name>.service` → `wait_for_health`; retire any
+   legacy hand-written unit into `~/.AGENTTRASH/` (a leftover
+   `~/.config/systemd/user/<svc>.service` that shells out to
+   `podman compose up` collides BY NAME with the generated unit);
+   `podman rm -f <container>` so the unit's container can take the name.
+4. Planner: map `<host>/<svc>/**` → service in `scripts/deploy-plan.py`
+   AND add a `service_is_allowed` entry; keep `SERVICE_ORDER` dependencies
+   ordered. Add a `test-changed` case above the generic
+   `*/quadlet/*.container` fallback.
+5. Retire the compose entry in the SAME change — two owners is how a stale
+   duplicate comes back.
+6. Seam test `tests/<svc>_quadlet.bats`: boot persistence, restart policy, no
+   wildcard publish, container name matches Caddy/scrape references, health
+   command, deployer installs+restarts+waits, never `rm`, never `enable`.
+   Migrate any older assertion that read the compose file for devices/flags.
+7. Deploy `--service <name>`; prove: `is-active`/`is-enabled` =
+   active/generated, container `healthy`, endpoint answers,
+   `list-dependencies default.target` shows the unit, off-host probes on
+   previously published ports now refuse, and the container no longer carries
+   `io.podman.compose.project`.
+
+**Traps that cost real time:**
+
+- `HealthCmd` = CMD-SHELL (podman runs `/bin/sh -c`); a distroless image has
+  no shell → the check fails forever. `--health-cmd` won't take exec form
+  either. Put `HEALTHCHECK … CMD ["binary","arg"]` in the image and build
+  with `--format docker` — the default OCI format has no HEALTHCHECK field
+  and drops it with only a warning.
+- Never point a unit's `EnvironmentFile` at the repo-root `.env` — it is a
+  gluetun env carrying `HOME=/root`; systemd sets HOME for the service and
+  the unit dies with `stat /root/.config: permission denied`. Use a
+  per-service env under `/home/dank/Containers/<svc>/env`, seeded by the
+  handler with a single-variable filter, validated so a missing secret fails
+  loudly.
+- `Network=host` exporters scraped by containerized Prometheus must bind the
+  host's LAN address (containers cannot reach host loopback) PLUS `127.0.0.1`
+  for the homepage probe — two `--web.listen-address` flags, never the
+  wildcard.
+- alloy 1.x refuses the `ui` config block; the unit must pass
+  `--server.http.listen-addr=0.0.0.0:12345` before its port can be published.
+- GPU services: `AddDevice=` + `GroupAdd=keep-groups` (user-unit spawns carry
+  no supplementary groups).
+- When the last service leaves a compose file, the compose-validity seam
+  fails (requires ≥1 service): keep the file as comments and relax the seam
+  deliberately, or delete the file and fix the planner's compose-diff path —
+  decide with the owner, the deploy tooling reads it.
+
+## 3. Drift / crash-loop diagnosis (5 minutes, in order)
+
+Symptom: crash-looping or perpetually unhealthy container whose DEFINITION
+looks right; or a service quietly down for days because a restart policy kept
+something running. Cause: the running container was created from an OLDER
+definition — a compose/unit edit does not touch an existing container, and
+the restart policy faithfully resurrects it forever.
+
+1. **Who started it:** `podman inspect <name> --format 'unit={{index
+   .Config.Labels "PODMAN_SYSTEMD_UNIT"}} restart={{.HostConfig.RestartPolicy.Name}}
+   created={{.Created}}'`; compare `created` with when the definition changed
+   (`git log -p --follow <config>`). Created-before-the-edit is the signature.
+2. **What it is ACTUALLY running:** `{{.Config.CreateCommand}}` is the truth.
+   `{{.HostConfig.Devices}}` / `{{.HostConfig.GroupAdd}}` print empty/0 even
+   for containers started with `--device`/`--group-add` — a false negative
+   that sends you hunting the wrong root cause.
+3. **What the process says:** `podman logs <name> | grep -iE 'error|cannot|Runtime'`
+   (a capability error like `No HIP GPUs are available` beside a missing
+   device flag confirms drift). Count the damage: `{{.RestartCount}}` — one
+   instance reached 9,146 restart attempts on stale config.
+
+Quickest unblock: `podman compose up -d --force-recreate <svc>`. Durable fix
+= the §2 cutover. **Verify the capability, not the status:** `podman exec
+<name> python3 -c "import torch; print(torch.cuda.is_available())"`;
+`ss -ltn | grep <port>` for the actual bind addresses. ComfyUI incident:
+2026-09-16, stale device-less container, 9,146 restarts. History repeats:
+2026-08-26 Caddy (no unit), 2026-09-11 five observability services.
+
+## 4. Podman 5.x quadlet v1 generator quirks (verified live 2026-08-28)
+
+- **Hard-fail keys** — the whole unit silently fails to generate
+  (`systemctl --user cat <unit>` shows "could not be found"): any
+  dash-prefixed `[Container]` option (`--shm-size=…` → `unsupported key …
+  in group 'Container'`); `Command=` is unsupported, use `Exec=`.
+- **HealthCmd space-splitting:** the v1 generator splits `HealthCmd` on
+  spaces into the healthcheck argv; a quoted probe
+  `/bin/sh -c "pg_isready …"` lands as argv token `"pg_isready` →
+  command-not-found → container reads **unhealthy while the DB is up**.
+  HealthCmd must contain only space-free tokens
+  (`HealthCmd=pg_isready -d authentik -U authentik`). If health reads
+  unhealthy but logs look fine, compare the baked
+  `{{.Config.Healthcheck.Test}}` array against the intended command before
+  believing the status.
+- **Probe a new key before shipping:** drop a throwaway `probe.container`,
+  `daemon-reload`, `journalctl --user --since "1 min ago" -o cat | grep -i
+  "unsupported key"`, `systemctl --user cat probe.service` (shows the
+  generated ExecStart verbatim), then remove + reload.
+
+## 5. Rootless Quadlet log retrieval
+
+Step 0 — establish ownership: `systemctl --user status <name>.service`
+(works ⇒ rootless user unit); `loginctl show-user <user> -p Linger`. If
+`--user` knows the unit, EVERY command runs as that user with NO sudo.
+
+| Need | Command |
+|---|---|
+| Current instance tail/follow | `podman logs --tail 200 / -f <name>` |
+| Across restarts/recreates | `journalctl --user -u <name>.service -n 200 --no-pager` |
+| Time-bounded / crash-only | `--since "14:30"` / `-p warning` |
+| Kernel-side evidence (OOM, SVM) | `sudo journalctl -k --since …` |
+
+Quadlet units commonly run `--replace --rm`: **`podman logs` is destroyed on
+recreation** — the journal is the only durable record. Traps:
+`sudo podman logs` → "no such container" (root's store does not contain
+rootless containers; same for `sudo podman ps`); a system-level
+`journalctl -u <name>` returns nothing for a user unit; during first-start
+weight downloads health reports unhealthy BY DESIGN (check the payload file
+is growing). Verbose vendor switches are a config change — through git and
+deploy.sh, never a live env poke.
+
+## 6. Compose bring-up with ground truth (rootless hosts)
+
+1. **Audit reality before pressing buttons:** `podman ps -a` runs per-user —
+   check every namespace (`sudo podman ps -a`) and native units
+   (`systemctl list-units`) before declaring a service absent. Compare
+   compose mount sources against the real filesystem — compose files name
+   phantom paths (e.g. `/mnt/media/*` vs the real LUKS array at
+   `/mnt/entropy`). Fix compose through git first, deploy second.
+2. **Pre-create bind-mount sources** (`sudo mkdir -p` + `chown 1000:1000`)
+   or creation fails `statfs …: no such file or directory`.
+3. **Staged bring-up:** infra first (`up -d mariadb redis`), poll
+   `{{.State.Health.Status}}`, then the rest. Fresh containers answer
+   000/refused for the first seconds — poll.
+4. **Never guess image tags:** ghcr.io answers 403 on the bearer-token
+   request for nonexistent/private repos ("manifest unknown"/token len 0 =
+   repo or tag does not exist, not auth). Query the registry API
+   (`ghcr.io/token?scope=repository:<ns>/<repo>:pull` →
+   `/v2/<ns>/<repo>/tags/list`).
+5. **Read the interface from inside the image** before guessing flags/envs:
+   `podman exec <ctr> cat /z/cpp.sh` etc. HTTP 500 with a styled error page
+   means RUNNING-but-misconfigured, not down. Copyparty ac ground truth:
+   entrypoint `/bin/ash /z/cpp.sh -c /z/initcfg`; accounts/volumes are ARGV
+   flags (`-a USER:PASS`, `-v SRC:DST:PERMS`), not envs — and with none
+   configured it serves cwd read/write for everyone (an unauthenticated
+   world-writable share).
+6. Remote-shell quoting: zsh eats bare `===`/`?` tokens; a zero-match from a
+   mangled pattern is not absence — re-verify the target exists.
+7. `user: "1000:1000"` SIGSEGV at boot = unmapped uid in the rootless userns
+   (host 1000 == container 0): drop the user line, run as container-root.
+
+## Related
+
+- [[Deployment Flow]] — the deployer that installs these units
+- [[Stale Signal Audit]] — the audit that catches drift's victims
+- [[Container Network Namespaces]] — reachability between containers
+- [[Dankclaw/ComfyUI]] — the 9,146-restart incident
+- [[Infrastructure/Observability]] — same class, twice
@@ -0,0 +1,144 @@
+---
+type: reference
+title: Stale Signal Audit
+description: The audit battery for surfaces that rot without failing — unreachable probes, vacuous assertions, invalid checkers, gates that never run, and prose invalidated by a change. Four passes found real defects every time.
+tags: [testing, observability, audit, signals, gates]
+timestamp: 2026-10-05T00:00:00-07:00
+---
+
+# Stale Signal Audit
+
+Run after any change and before trusting a green gate. The productive question
+is **which signal here cannot possibly be correct?** Every pass on this repo
+(4 of 4) found real defects — including one in the rule file every future
+agent reads first.
+
+## The surfaces, and what each vantage point can reach
+
+| Surface | Producer's vantage point | Can reach |
+|---|---|---|
+| Homepage status cards | the `homepage-status` timer on **Dankclaw** | Dankclaw's loopback, Dankclaw container names, **published** LAN addresses |
+| Prometheus scrape jobs | the `prometheus` container on Dankclaw | container names on `dankclaw_default`, LAN addresses, that host's loopback only |
+| bats / e2e suites | the dev box | the repo, plus the live fleet over HTTPS/LAN |
+
+Anything else — another host's loopback, a container-internal port never
+published — is unreachable no matter how healthy the service is. The failure
+mode is always the same shape: a signal that **cannot** succeed by
+construction, which teaches the operator to ignore the surface it lives on.
+
+## 1. Permanently-red probes (reachability)
+
+For every monitor/probe/alert target ask: *can this checker physically reach
+that endpoint?* Seen classes: a card probing another host's container-internal
+port; a "probe" job scraping `/` (HTML → `unsupported Content-Type`); a
+`/metrics` that is disabled (Gitea 404).
+
+- A scrape cannot express "is this HTTP service answering?" — the right probe
+  is the **blackbox exporter** in probe mode (which also gives guarded vhosts
+  a real uptime signal, since Prometheus cannot solve an Anubis challenge).
+- **Fix in the direction of honesty, never of silence:** unreachable target →
+  mark the homepage card `wip: true` with the reason in `description` (the
+  checker skips `wip`; `wip` is the existing convention for "real service,
+  not probeable"), or repoint the job at something reachable. NEVER open a
+  database port on the LAN to make a badge green, and never delete a scrape
+  job — an absent scrape and a healthy one are indistinguishable.
+- Alert rules must not depend on always-red signals — scope them to services
+  that can actually be healthy (`LlmEndpointDown` is scoped to `frank-*` for
+  exactly that reason).
+- A target's failure STRING can prove a fix: a `/metrics` scrape flipping from
+  `connection refused` to `unsupported Content-Type "text/html"` means the
+  service came up and only the scrape is wrong. Read the error, not just
+  up/down.
+
+## 2. Vacuous assertions (highest severity — the gate is already dead)
+
+A seam that greps a file passes *by absence* once the file is retired:
+`grep` exits 2 on a missing path and `!` inverts it.
+
+```bash
+grep -rn 'grep.*"\$REPO_ROOT/[^"]*"' tests/*.bats | head -50
+[ -f "$path" ] || echo "MISSING: a seam may be grepping this"
+```
+
+Fix shape — one place asserts the invariant, each local site keeps a live
+check:
+
+```bats
+@test "no <host> compose file competes with the quadlet units" {
+    local cf="$REPO_ROOT/<host>/compose.yaml"
+    [ ! -f "$cf" ] || { echo "$cf is back: every container is unit-managed"; return 1; }
+}
+# at the per-service site: still fails if the entry returns
+[ ! -f "$cf" ] || ! grep -qE '^\s+<svc>:' "$cf"
+```
+
+**Prove teeth:** recreate the offending file
+(`printf 'services:\n  <svc>:\n    image: x\n' > <path>`), run the seams,
+confirm RED, then move the file to trash. A re-armed check you have never
+seen fail is still unverified.
+
+## 3. Gates that never run
+
+Playwright specs in `tests/e2e/` are matched by **number prefix** in
+`playwright.config.ts`; an unregistered spec silently never runs and the
+suite count stays the same. Count tests before/after;
+`tests/e2e_scaffold.bats` fails on an unregistered spec by design.
+
+## 4. Validate the checker before acting on it
+
+A checker's output is a hypothesis about another tool's semantics. Modelling
+Obsidian with naive text matching produced 45 "broken links" of which 3 were
+real — the rest were case-sensitivity (a `qBittorrent` link to
+`qbittorrent.md`), path-qualified resolution (a path-qualified
+`folder/Note` link resolves by path SUFFIX, not
+stem), and `%20` URL-encoding in markdown links. Fix the checker first,
+re-derive the report, only then edit documents. The result lives in
+`scripts/check-vault-links.py` with those three rules in its docstring,
+gated by the `no unresolved links in the docs vault` seam. Same class: a
+`find … | head -1` selector that reads the wrong file; a truncated listing
+that makes you "fix" a file that exists.
+
+## 5. Prose invalidated by your own change
+
+After retiring a file or adding a feature, grep for EVERY reference, not just
+code:
+
+```bash
+grep -rn "<retired-file>" --include="*" . | grep -v '^./.git/'
+grep -rn "no <feature>\|none —\|unconfigured" --include="*.md" --include="*.container" --include="*.yml" .
+```
+
+This catches the ones that bite: AGENTS.md said ComfyUI runs CPU-only long
+after it was restored with GPU access; a unit said "there are no alerting
+rules" hours after five were added; a docs page documented three scrape jobs
+as working when they could never report up. **Clean cutover includes prose** —
+docs, unit docstrings, and the repo's rule file (the most expensive place to
+be stale).
+
+## Order and verification
+
+1. Enumerate the surfaces (status board, metrics targets, alert rules, test
+   gates, docs/rule files, dashboard panels).
+2. For each, ask what the signal CAN express, not what it says.
+3. Validate any checker before editing the things it flags.
+4. Fix at the root; never weaken an assertion to make it pass.
+5. Gate the class so it cannot return (a seam), then prove the seam fails on
+   the defect.
+
+Verify: `./scripts/check-vault-links.py` (exit 1 on breaks); query each
+Prometheus panel expression against the API (0 series on a panel that should
+show data is a defect; 0 `ALERTS` when nothing fires is not); read the
+deployed `/srv/homepage/status.json` and triage every `state != up` entry
+against reality; query `/api/v1/targets?state=active` and print
+`health != "up"` with `lastError`; `./tests/run.sh` → `ALL TESTS PASSED`.
+Report per finding: the surface, why the signal could not have been correct,
+the fix, and the red/green evidence — flag any regression you introduced
+yourself in the same breath (one was found this way: a tightened bind killed
+a reverse-proxied vhost while every gate stayed green, because the spec
+covering that hostname did not exist).
+
+## Related
+
+- [[Testing Policy]] — the seam inventory this audit walks
+- [[Observability]] — the scrape/probe surfaces it audits
+- [[Quadlet Lifecycle]] — the ownership labels the probes read
@@ -0,0 +1,121 @@
+---
+type: howto
+description: "Qwen-Image 2.1 workflows on Dankclaw: two proven txt2img workflows (art + legible text), the API recipe, and which models/LoRAs to add next."
+tags:
+  - dankclaw
+  - comfyui
+  - qwen-image
+timestamp: 2026-10-05
+---
+
+# ComfyUI Qwen-Image Workflows
+
+Two ready-to-load text-to-image workflows for the Qwen-Image 2.1 payload
+already on Dankclaw, both proven end-to-end on 2026-10-05 by POSTing to the
+live API on port 8188 and verifying the rendered PNG. The repo copies are the
+source of truth; the host copies under `/home/dank/comfyui-data/workflows/`
+are for convenience.
+
+## What is on disk
+
+| Path (under `/home/dank/comfyui-data/models/`) | Size | Role |
+|---|---|---|
+| `diffusion_models/qwen_image_2.1_int8_convrot.safetensors` | 6.8 GB | **The one the workflows use** — int8-convrot quant, ComfyUI-core native |
+| `diffusion_models/qwen_image_2.1_bf16.safetensors` | 14 GB | Full-precision fallback |
+| `diffusion_models/qwen-image-2.1-Q8_0.gguf` (+ `-arch`) | 7.1 GB | GGUF alternative (needs ComfyUI-GGUF custom node) |
+| `text_encoders/qwen3vl_8b_w4a8.safetensors` | 5.9 GB | **The one the workflows use** — 4-bit-weights/8-bit-act Qwen3-VL 8B encoder |
+| `text_encoders/qwen3vl_8b_int8_convrot.safetensors` | 8.8 GB | Mid quant |
+| `text_encoders/qwen3vl_8b_bf16.safetensors` | 17 GB | Full precision |
+| `vae/qwen_image_2.1_vae_bf16.safetensors` | 645 MB | 2.1 VAE (used) |
+| `vae/qwen_image_vae.safetensors` | 243 MB | Smaller VAE |
+
+No LoRAs are on disk yet — see [Recommended additions](#recommended-additions).
+
+## The workflows
+
+Both are API-format JSON (ComfyUI 0.37 loads them via the UI's **Load** dialog
+too): `dankclaw/comfyui/workflows/` in the repo,
+`/home/dank/comfyui-data/workflows/` on the host. Node chain in both:
+
+```
+UNETLoader (int8_convrot) ──┐
+CLIPLoader (qwen_image, w4a8) ─┤ CLIPTextEncode pos/neg
+VAELoader (2.1 bf16) ────────┤
+EmptySD3LatentImage ─────────┴─ KSampler ─ VAEDecode ─ SaveImage
+```
+
+**Common sampler settings:** `cfg 1.0` (Qwen-Image 2.1 expects CFG-free
+sampling), `euler` + `simple`, seed fixed in the file.
+
+| Workflow | File | Shape | Steps | Made for |
+|---|---|---|---|---|
+| Art | `qwen-image-2.1-t2i-art.json` | 1024×1024 | 20 | General illustration/prose prompts |
+| Legible text | `qwen-image-2.1-t2i-legible-text.json` | 832×1216 | 25 | Posters, captions, any in-image text |
+
+### Legible-text prompt recipe
+
+Qwen-Image 2.1 is one of the best open-weight text renderers, but only if the
+prompt is written for it:
+
+1. **Quote the exact strings**: `large bold white text reading "SOLIDARITY OF
+   THE SOVIET COMPUTER SYSTEMS"` — quoted text is rendered literally.
+2. **Describe layout, not vibes**: say *centered / below it / large vs small*,
+   the typeface family (sans-serif), and the color.
+3. **Say what NOT to draw**: end with `no other words, no watermark`.
+4. Keep the negative side on `misspelled, warped letters, extra text` (already
+   in the workflow).
+5. More steps help text: the text workflow uses 25 vs 20.
+
+Verified render: `SOLIDARITY OF THE SOVIET COMPUTER SYSTEMS` + `DANKCLAW
+DANKSERVER EST 2026` came back crisp and correctly spelled at 832×1216
+(`qwen21-text_00001_.png`, 953 KB). One caveat from the same render: a
+shorter secondary line is where misspellings creep in — proofread the first
+render before shipping.
+
+## Driving it from the API
+
+ComfyUI 0.37 **requires the graph wrapped in a `prompt` key** — a bare
+`{node-id: node}` body gets `{"error": {"type": "no_prompt"}}` (verified
+2026-10-05):
+
+```bash
+curl -s -X POST http://127.0.0.1:8188/prompt -H 'Content-Type: application/json' \
+  -d "$(python3 -c 'import json;print(json.dumps({"prompt": json.load(open("dankclaw/comfyui/workflows/qwen-image-2.1-t2i-art.json"))}))')"
+# -> {"prompt_id": "...", ...}
+curl -s http://127.0.0.1:8188/history/<prompt_id>   # status.status_str == "success"
+```
+
+Outputs land in `/home/dank/comfyui-data/output/`. The queue is FIFO — the
+two workflows above ran back-to-back unattended.
+
+**Note:** the container's UI-saved workflows live in the ephemeral
+`/opt/ComfyUI/user` layer and do NOT survive an image rebuild — hence the
+host-side `workflows/` dir and the repo copy.
+
+## Recommended additions
+
+Quantized preferred, sized for the 8060S iGPU (GTT-backed; the RTX 3090 is
+frank-secondary's). All repos verified 2026-10-05. None are on disk yet.
+
+| # | What | Repo | Grab | Size | Why |
+|---|---|---|---|---|---|
+| 1 | **Lightning 8-step LoRA** (T2I speed) | `lightx2v/Qwen-Image-Lightning` | `qwen_image_lightning_8steps_*.safetensors` (LoRA) | ~1–2 GB | 20 steps → 8 with a `LoraLoader` node; 2–3× faster renders on the iGPU, negligible quality loss |
+| 2 | **Qwen-Image-Edit-2511 GGUF** (image editing) | `unsloth/Qwen-Image-Edit-2511-GGUF` | Q4_K_M (~13.2 GB) or Q4_0 (~11.9 GB) | ~12–13 GB | Unlocks img2img "edit this photo" workflows — a capability class the fleet has zero of; Q4 keeps it iGPU-friendly |
+| 3 | **DiffSynth ControlNets** (structure control) | `Comfy-Org/Qwen-Image-DiffSynth-ControlNets` | `split_files/model_patches/` canny + depth | ~2–3 GB each | Pose/edge/depth-guided layouts — pairs with the legible-text workflow for exact poster composition. The `QwenImageDiffsynthControlnet` node is already in the running ComfyUI |
+| 4 | **Wan 2.2 TI2V-5B GGUF** (first video model) | `QuantStack/Wan2.2-TI2V-5B-GGUF` | Q4/Q5 GGUF (~6–8 GB) | ~6–8 GB | T2V+I2V in one dense 5B model; the gentlest on-ramp into video generation, runs in ComfyUI core (`wan` CLIP type is already in this build's loader enum) |
+
+Placement: #1 → `models/lora/` (create it), #2 → `models/diffusion_models/`,
+#3 → `models/controlnet/` (create it), #4 → `models/diffusion_models/`.
+Fetch with `scripts/download-frank-*.sh` discipline (`-f` + byte-count +
+sha256 checks) or a plain `hf download` with the token in the environment —
+never on the command line.
+
+Skip list: the original `city96/Qwen-Image-gguf` conversions (superseded by
+the 2.1 files already on disk), and any BF16 of #2/#4 (the iGPU does not need
+them at GTT capacity, and they double the disk bill).
+
+## See Also
+
+- [[ComfyUI]] — the service itself (image, unit, GPU access)
+- [[ComfyUI ROCm and Vulkan]] — the live-verified GPU path
+- [[Dankclaw Stack]] — the host
@@ -110,5 +110,6 @@ The web UI responds on `/`. ComfyUI also exposes `/system_stats`. On this APU, `

 - [[Dankclaw Stack]]
 - [[ComfyUI ROCm and Vulkan]] — the live-verified GPU path research
+- [[ComfyUI Qwen-Image Workflows]] — the Qwen-Image 2.1 payload, proven workflows, and recommended model/LoRA additions
 - [[Lemonade and sd.cpp]] — the ggml-family alternative
 - Knowledge base: [[ComfyUI]] (tools)
@@ -81,6 +81,178 @@ The wiring lives in the live `/data/config.yaml` under `mcp_servers.chrome-devto
 The npx package cache lives in the container layer, so a recreation (not just a
 restart) pays one first-call npm download; subsequent calls are cached.

+## Config-home drift ("cannot connect" to models that answer curl)
+
+`podman exec <container> printenv HERMES_HOME` for each hermes container. If
+the webui reads the image default (`/opt/data`) while the gateway reads
+`/data`, the webui runs the image-default template config (`provider: auto` →
+openrouter.ai with no key = connect failure).
+
+The recreation trap is the same one the volumes table shows: image `VOLUME`
+points get a fresh anonymous volume on every container recreation
+(`podman run --replace --rm`) — edits inside such a mount are lost on
+restart/redeploy. Compare volume ids before/after a restart to confirm.
+
+Fix: pin `Environment=HERMES_HOME=/data` in the quadlet unit (shared state
+root), deploy service-scoped. Verify with the app's own code, not guesses:
+
+```python
+from api import config as C; C.resolve_model_provider("<model-id>")
+# -> (model_id, provider, base_url); bare ids map to the owning provider by
+#    scanning the providers: dict models allow-list; @provider:model forces one.
+hermes_cli.providers.resolve_user_provider(name, cfg["providers"])
+# -> ProviderDef with real base_url; accepts api/url/base_url keys.
+```
+
+Named providers (`providers:` dict) are a catalog + connection source;
+`custom:<slug>` connections resolve only via `custom_providers[]` entries. If
+neither holds the provider name, the request has no endpoint and fails to
+connect. Probe the endpoint with the **exact served model id** — the retired
+FLM/NPU endpoints routed only by exact registry tag and answered HTTP 200
+with `{"error":"Failed to load <tag> model!"}` for unknown tags; status-code
+checks miss this class entirely.
+
+## Live config edits
+
+Live file: `/home/dank/Containers/hermes/data/config.yaml` (dir `drwx------`
+uid 109999; all reads/edits need `sudo`). The repo file
+`dankclaw/hermes/config.yaml.example` is a sanitized backup and **can lag
+live — always diff the live file first; never treat the example as truth**.
+
+1. **Backup first:** `sudo cp -a …/config.yaml …/config.yaml.bak-YYYYMMDD`
+   (in-dir `.AGENTTRASH/` also exists for removals).
+2. **Apply** with surgical `sudo sed -i` or full-file write; YAML comments in
+   the live file are tool-generated — preserve them.
+3. **Sweep stale references** across the whole data dir:
+   `sudo grep -rln "<old-id>" /home/dank/Containers/hermes/data
+   --include="*.yaml" --include="*.json"`. Session files/MEMORY.md are
+   historical — leave them.
+4. **Derived caches:** `webui/models_cache.json` is keyed by a config
+   fingerprint (`_source_fingerprint.config_yaml` mtime/size); after edits it
+   is stale. Let it regenerate lazily or trash it — never hand-edit.
+5. **Restart the model-consuming units** (they parse config at startup):
+   `systemctl --user restart hermes-webui.service hermes.service`.
+   `hermes-dashboard.service` is unaffected by routing changes.
+6. **Verify behaviorally**: one tiny completion per endpoint with the exact
+   served id — POST `/v1/chat/completions` and read the body, never just the
+   status code (endpoint names rotate; see the LLM knowledge base).
+7. **Sync the repo backup** (`dankclaw/hermes/config.yaml.example`) to the
+   live shape, redacting secrets (`password_hash`, phone account stay
+   live-only), then `./tests/run.sh` before commit.
+
+`provider: custom` + `base_url` entries route by base_url and send the model
+id verbatim; named providers resolve via their `api:` URL. **Renaming a
+served alias anywhere (serve scripts, env ALIAS) MUST propagate to this
+config in the same change** or Hermes requests 404 on nonexistent models —
+this exact gap caused the 2026-09-05 outage.
+
+## Auxiliary routing against local backends
+
+When a Hermes feature that calls an auxiliary task (e.g. title generation)
+fails against a local endpoint, the usual cause is a **request shape the
+local runtime does not implement**, not a routing mistake.
+
+Diagnose with the real call path — run the actual function inside the
+container with the app's own constants (the probe cannot drift from
+production):
+
+```bash
+base64 -w0 /tmp/probe.py | ssh dank@192.168.0.143 \
+  'base64 -d > /tmp/probe.py && systemctl --user restart hermes.service && sleep 20 && \
+   podman cp /tmp/probe.py systemd-hermes:/tmp/probe.py && \
+   podman exec -i systemd-hermes /opt/hermes/.venv/bin/python /tmp/probe.py'
+```
+
+Copy the file AFTER any restart (`--replace --rm` discards the container
+layer). The CLI at `/opt/hermes/hermes` uses system python and dies
+`No module named 'yaml'`; use `/opt/hermes/.venv/bin/python`.
+
+```python
+import sys; sys.path.insert(0, "/opt/hermes")
+from agent.title_generator import _TITLE_PROMPT_TEMPLATE, _TITLE_RESPONSE_FORMAT, _clean_title, _extract_title_text
+from agent.auxiliary_client import call_llm
+resp = call_llm(task="title_generation", messages=[...], max_tokens=64,
+                temperature=0.3, extra_body={"response_format": _TITLE_RESPONSE_FORMAT})
+```
+
+What actually breaks, and the fix:
+
+- **`response_format` (json_object / json_schema) is a hard 400 on every
+  local backend here** — neither FreeToken nor halogen implements constrained
+  decoding. Config cannot override it: `call_llm` merges the task's
+  `extra_body` first and then `update()`s the caller's, so the hardcoded one
+  wins. Only temperature and max_tokens have reactive retry paths.
+- **Fix: move that one task to the Responses transport**, which never
+  forwards `response_format`:
+
+  ```yaml
+  auxiliary:
+    title_generation:
+      provider: dankserver-small        # names track the live config; diff first
+      model: frank-secondary
+      api_mode: codex_responses        # real per-task key
+      extra_body:
+        reasoning:
+          effort: none                 # mandatory, see below
+  ```
+
+- **Disable thinking for any cap-limited task.** The adapter omits reasoning
+  only when config says `reasoning.enabled: false`. The
+  `reasoning_effort: none` *shorthand* produces exactly that and leaves the
+  server default (thinking ON) — the 64-token title budget gets spent on the
+  think: empty content, `finish_reason: length`. The explicit
+  `extra_body.reasoning.effort: none` instead sends
+  `{"effort":"none","summary":"auto"}`, which the local `/v1/responses`
+  honors.
+- Both local runtimes also answer `/v1/responses` and `/v1/messages` — no
+  proxy needed for the alternate transport.
+
+Verify and land: re-run the probe with **no** `api_mode`/reasoning kwargs —
+it must pass from config alone (that is the production path); keep the live
+file and the repo example in step; `./tests/run.sh`, commit, push — a Hermes
+config change is docs+example only, no service deploy needed.
+
+Gotchas: `auxiliary.<task>.enabled: false` is checked BEFORE
+`apply_instant_title`, so disabling a broken title task also removes the
+derived title — it does not degrade gracefully the way a failure does. A
+local call can also fail at the model level: a 200 with an error-text body,
+or a chatty non-title answer. Always read the raw content, not the status.
+
+## Webui password rotation ("Hermex")
+
+Auth contract (verified live 2026-08-31): `POST /api/auth/login` with JSON
+`{"password":"…"}` → 200 + signed HttpOnly `hermes_session` cookie (24 h TTL)
+on match, 401 on miss. No username field. `/health` answers 200
+unauthenticated and `/` answers 302 — **never verify auth via those**; probe
+login with a deliberately wrong password (expect 401) and the real one
+(expect 200). Details on the webui surface: [[Hermes-WebUI]].
+
+1. Run the whole probe as a heredoc'd bash script
+   (`ssh -o BatchMode=yes dankclaw 'bash -s'`) — the host's zsh mangles
+   inline `${…}` even inside single-quoted ssh args; never interpolate curl
+   `-w %{http_code}` inline.
+2. Update BOTH env files with
+   `sed -i "s/^HERMES_WEBUI_PASSWORD=.*/HERMES_WEBUI_PASSWORD=<new>/"`:
+   - `/home/dank/serverAppConfigs/.env` ([Service] EnvironmentFile)
+   - `/home/dank/serverAppConfigs/dankclaw/.env` — **the effective one**: the
+     quadlet's `[Container] EnvironmentFile=` is what the container reads at
+     create; rotating only the root file leaves the running container serving
+     the old secret.
+3. `systemctl --user restart hermes-webui.service` (unit recreation re-reads
+   env; no deploy needed if the unit file itself didn't change).
+4. Verify wrong-pw → 401, right-pw → 200, from host loopback **and** a LAN
+   client at `http://192.168.0.143:8787` (webui publishes on the LAN
+   interface since commit `4443897`; off-network ingress is SSH tunnel only —
+   user `tunnel`, port **22**, key-only auth, forward remote end
+   localhost:8787).
+
+Gotchas: `sudo podman exec` fails exit 125 here — root's namespace does not
+see rootless containers; run `podman exec hermes-webui printenv
+HERMES_WEBUI_PASSWORD` unprivileged (the ssh user owns the env files, mode
+600). If rotating because a secret leaked into transcripts/logs, rotate in
+place via a script that fetches the live value from the running container
+(`printenv`) instead of transcribing it.
+
 ## See Also

 - [[Hermes-Dashboard]] — the LAN browser management UI
@@ -24,29 +24,48 @@ https://github.com/open-webui/open-webui
 | Upstream latest | 0.11.4 (2026-09-21) — security/access-control advisory; `:main` is the rolling image, not a stable pin. Imagegen endpoint is `POST /api/v1/images/generations` (the `comfyui` engine is still upstream). |
 | Port | 3005 (host) → 8080 (container). Port 3000 is Gitea's HTTP, so it is not usable. |
 | Data volume | `/home/dank/Containers/open-webui/data` → `/app/backend/data` |
-| Backends | Three fleet endpoints: `frank-large` (host.containers.internal:7770), `frank-medium` (192.168.0.235:7770), `frank-small` (host.containers.internal:7771) — see [The LLM backends](#the-llm-backends) |
+| Backends | Picker exposes two models — `frank-zealot` (host.containers.internal:7771) and `frank-dragoon` (host.containers.internal:7770, added 2026-10-09). The `frank-archon` (Dankserver) connection remains configured in the DB, disabled — see [The LLM backends](#the-llm-backends) |
 | Network | Pinned bridge `systemd-open-webui` (`10.89.3.0/24`, gateway `10.89.3.1`) — see [History](#history) |
 | Persistence | Quadlet unit `open-webui.service`; `[Install] WantedBy=default.target` |
 | Deploy | `scripts/deploy-open-webui-quadlet.sh dankclaw` |

 ## The LLM backends

-OpenWebUI serves the three fleet LLM endpoints. The model picker is built from
-the `model` table in `webui.db` (one row per model), NOT from live connection
-discovery — so the picker shows exactly the models seeded there.
+The picker is built from the `model` table in `webui.db` (one row per model)
+plus the `model_ids` of each **enabled** OpenAI connection, NOT from live
+connection discovery — so the picker shows exactly what is seeded there.
+As of 2026-10-09 the picker exposes **two models: `frank-zealot` and
+`frank-dragoon`**.

-| Picker model | Connection URL | Runtime | Host |
-|--------------|----------------|---------|------|
-| `frank-large` | `http://host.containers.internal:7770/v1` | unsloth llama.cpp (Strix Halo iGPU, 200k ctx) | Dankclaw |
-| `frank-medium` | `http://192.168.0.235:7770/v1` | vllm-radiance TP=2 (FP8, DFlash2, 262k ctx) | Dankserver |
-| `frank-small` | `http://host.containers.internal:7771/v1` | llama.cpp (RTX 3090, Gemma4-12B, 262k ctx) | Dankclaw |
+| Connection (`openai.api_configs` idx) | URL | Enabled | `model_ids` | Runtime | Host |
+|---|---|---|---|---|---|
+| 0 | `http://host.containers.internal:7770/v1` | yes (2026-10-09) | `frank-dragoon` | halogen flash-server (Strix Halo iGPU) | Dankclaw |
+| 1 | `http://192.168.0.235:7770/v1` | no (2026-10-07) | `frank-archon` | radiance TP=2 (2× R9700) | Dankserver |
+| 2 | `http://host.containers.internal:7771/v1` | yes | `frank-zealot` | llama.cpp CUDA (RTX 3090, 2×200,704) | Dankclaw |
+
+The `model` table carries one decoration row per picker model:
+`frank-zealot` (context length 200,704 = the per-slot window; description:
+Qwen3.5-9B Q6_K + BF16 vision mmproj) and `frank-dragoon` (context length
+262,144 = the per-slot `HALOGEN_CTX`; description: Swift1.5 v2 abliterated,
+MTP, vision ON).
+
+A third gate sits *after* the table, and it is the one that bites family
+accounts (2026-10-07): a model row is only visible to a non-admin user when
+its **access permission** grants them. A seeded row defaults to admin-only,
+so the picker renders "No results found" for every regular user even though
+the connection is enabled, `model_ids` is correct, and the endpoint answers
+`/v1/models` — and `GET /api/v1/models` returns `[]` for that user while an
+admin token returns the model. Fix: set the row's `access_control` to public
+(the `access` column on the `model` row) via the admin UI (Admin Panel →
+Models → visibility) or the DB. The verification battery below must therefore
+run **as a non-admin user**, not only as an admin.

 Two gotchas make the wiring non-obvious:

 - **llama.cpp `/v1/models` is Ollama-shaped.** llama.cpp returns
  `{"models":[{"name":...}]}`; OpenWebUI's `extract_data()` only reads the
  OpenAI `{"data":[{"id":...}]}` shape, so connection discovery silently misses
-  the llama.cpp endpoints (frank-large, frank-small). The fix is to set
+  the llama.cpp endpoints (frank-large, frank-secondary). The fix is to set
  `model_ids` in each connection's `api_config` — when non-empty, OpenWebUI
  builds the model list directly and never fetches `/v1/models`. frank-medium
  (vLLM) serves the OpenAI shape and would discover fine, but uses `model_ids`
@@ -62,7 +81,72 @@ The Ollama connection is disabled (`ollama.enable=false`).

 > All three endpoints are bare-metal systemd services managed by this
 > repository. Model swaps, context sizing, and spec tuning happen there. See
-> the [LLM knowledge base](../LLM/index.md).
+> the [LLM knowledge base](../LLM/index.md). Disabling a connection here only
+> hides it from this picker — the endpoint itself keeps serving every other
+> client (the harness, Hermes, API consumers).
+
+## Container introspection (DB and backend python)
+
+- Working dir for imports: `/app/backend`; package name `open_webui`
+  (underscore), NOT `openwebui`. `os.environ["WEBUI_SECRET_KEY"]` MUST be set
+  BEFORE importing any `open_webui` module (import-time hard requirement);
+  the secret lives at `/app/backend/.webui_secret_key`.
+- **The `sqlite3` CLI is NOT in the container** — use Python's `sqlite3`
+  module (`podman exec open-webui python3 -c "import sqlite3;
+  conn = sqlite3.connect('/app/backend/data/webui.db'); …"`). For config
+  keys use parameterized LIKE queries (avoids shell quoting).
+- Backend python: `podman exec -w /app/backend open-webui python3 -c "…"` —
+  `Models.get_all_models()` is a coroutine (await it); model objects are
+  pydantic `ModelModel` instances (attribute access, not `.get()`), with NO
+  `status`/`enabled` attribute — use `is_active` from the DB.
+- **API auth for verification:** `create_token` lives in
+  `open_webui.utils.auth` (NOT `open_webui.utils.utils`);
+  `get_verified_user` requires an `id` claim (not just `sub`);
+  `SESSION_SECRET = WEBUI_SECRET_KEY`. Mint an admin token inside the
+  container:
+
+  ```
+  podman exec -i open-webui sh -c 'export WEBUI_SECRET_KEY=$(cat /app/backend/.webui_secret_key); \
+    python -c "from open_webui.utils.auth import create_token; \
+    print(create_token({\"id\":\"<UID>\",\"sub\":\"<UID>\",\"email\":\"…\",\"role\":\"admin\"}))"'
+  ```
+
+  (admin UID from the `user` table).
+
+## Verification battery
+
+1. `GET /api/models` (the picker endpoint; response key `data`) → lists
+   EXACTLY the intended models, **as an admin AND as a `role='user'` token**.
+   Do NOT gate on `GET /api/v1/models/list` (the Admin Panel → Models page):
+   on the 0.6.x image it returns `items: []` for connection-backed models
+   (workspace-list only; measured 2026-10-09 on both picker ids).
+2. `POST /api/chat/completions` per model with a trivial prompt ("Reply
+   with exactly one word: pong") → each returns content `pong`. Reasoning
+   models need a generous budget: frank-secondary (Qwen3.5 template) returns
+   empty content + `finish_reason: "length"` at `max_tokens 200` (the
+   thinking block eats it) and answers `pong` at 1500.
+3. A 500 "Server Connection Error" → `podman logs open-webui` for
+   `ClientConnectorError`: a network-reachability problem (host LAN IP vs
+   `host.containers.internal`), not a model problem.
+
+**Durability:** the model table + connections are DB-only (not git-managed);
+a fresh data volume loses them — including the public access grant above.
+For reproducibility, add an idempotent first-start seed script or document
+the setup in the service doc.
+
+## API access (auth-gated)
+
+Every `/api/*` endpoint requires an authenticated session (SSO login JWT)
+or a per-user API key (`auth.enable_api_keys=true`; keys are minted per
+account under Profile → API Keys). Anonymous web visitors get the `pending`
+role (`ui.default_user_role=pending`), which cannot chat. Verified
+2026-10-07 from loopback `:3005` and over the public vhost:
+`GET /api/v1/models/list`, `GET /api/v1/models` and
+`POST /api/chat/completions` all return **401** with no key (also with a
+bogus bearer token); only `/api/config`, `/api/version` and `/health`
+answer publicly, and none of them expose the model list. An external
+program needs an account + API key — there is no keyless access.
+

 ## Web search

@@ -106,6 +190,18 @@ browser-driven and only touches the two exempt vhosts (`ai.` and `auth.`).
  into the existing local account with the same email — the pre-SSO admin
  (frank8465@protonmail.com) keeps its data. `ENABLE_OAUTH_SIGNUP=true` lets
  other `openwebui_users` create accounts on first login.
+- **First login mints role `pending`.** `ui.default_user_role=pending` (config
+  DB, verified 2026-10-09), so a freshly-provisioned `openwebui_users` account
+  completes SSO and then cannot chat (picker empty) until its role is flipped:
+  `UPDATE "user" SET role='user' WHERE email='<user>@franksdank.cloud';` —
+  run with host `python3` over ssh against
+  `/home/dank/Containers/open-webui/data/webui.db` (no `sqlite3` CLI on
+  Dankclaw; table is `user`, singular; no `active` column). No per-user model
+  grant is needed: both picker models (`frank-zealot`, `frank-dragoon`) carry
+  the global `access_grant` row `(model, <id>, user, *, read)`, so any
+  `role='user'` account sees them. Flip on FIRST-LOGIN SIGHTING — the row does
+  not exist until the account signs in (2026-10-09: five provisioned accounts
+  had no row).
 - **IdP reachability:** the container is on its own bridge (10.89.3.0/24) and
  cannot reach `authentik-server:9000` by name. It reaches the IdP over the
  public URL `https://auth.franksdank.cloud` — Caddy publishes `0.0.0.0:443`
@@ -147,6 +243,23 @@ The quadlet carries a built-in healthcheck (30s interval, 3 retries, 300s start

 `podman ps` shows the container as `unhealthy` if either fails. Probe manually at `http://dankclaw:3005/health`.

+## Image generation (ComfyUI)
+
+Image generation rides the local [[ComfyUI]] integration, not the chat
+picker — ComfyUI's models never appear in the chat model picker (that is
+chat LLMs only), so ComfyUI needs its own image-generation connection
+configured in OpenWebUI. ComfyUI itself runs GPU-enabled on Dankclaw and
+serves `:8188`.
+
+Endpoint: `POST /api/images/generations` (image `0.11.x`: upstream moved the
+route to `POST /api/v1/images/generations`; the `comfyui` engine is still
+upstream) with `Authorization: Bearer <token>` and body
+`{"model":"imagegen","prompt":"…","n":1,"size":"1024x1024"}`; the response
+carries base64 image data or a URL.
+
+Failure modes: connection refused ⇒ OpenWebUI or ComfyUI is down; timeouts
+on large sizes ⇒ check ComfyUI resources; 401 ⇒ the token is bad.
+
 ## Retired instances

 - The compose `open-webui` service (port 3000) was never run: 3000 is Gitea's HTTP port, so it could not bind.
@@ -158,18 +271,64 @@ The quadlet carries a built-in healthcheck (30s interval, 3 retries, 300s start
 - The image tracks upstream `main`; pin a digest in the quadlet unit if reproducible rollback matters more than tracking. (The old "CPU-only host" framing is stale: this APU runs the gfx1151 ROCm build — see [[ComfyUI]].)
 - The default admin account is created on first login; secure it immediately.
 - If the backend is loading (3–8 minute model load), OpenWebUI shows connection errors until the model is ready.
- The on-host LLM services (frank-large `:7770`, frank-small `:7771`) are reached via `host.containers.internal`, NOT the host LAN IP: the container's netavark bridge cannot hairpin to its own host's non-published ports via `192.168.0.143` (connection refused), and the bridge gateway (`10.89.3.1`) lives in the rootless netns while llama-server runs in the host netns, so it refuses too. `host.containers.internal` (169.254.1.2) resolves to the host on the bridge and works — verified end-to-end for both endpoints. The Dankserver backend (frank-medium) is a different host, reached via its LAN IP `192.168.0.235:7770`.
+- The on-host LLM services (Dankclaw `:7770`, `:7771`) are reached via `host.containers.internal`, NOT the host LAN IP: the container's netavark bridge cannot hairpin to its own host's non-published ports via `192.168.0.143` (connection refused), and the bridge gateway (`10.89.3.1`) lives in the rootless netns while llama-server runs in the host netns, so it refuses too. `host.containers.internal` (169.254.1.2) resolves to the host on the bridge and works — verified end-to-end for both endpoints. The Dankserver backend is a different host, reached via its LAN IP `192.168.0.235:7770`.

 ## History

 - **2026-08-22 — slirp4netns → pinned bridge.** The unit ran on `Network=slirp4netns`. The slirp4netns helper process died (no OOM, no kill entry, no crash log found), and the container kept running with only `lo` in its netns: published port 3005 accepted TCP and reset every request, so chat served empty responses for ~2 days. Nothing logged an error. Moving to a kernel bridge removes the per-container helper entirely, and the healthcheck's interface check makes the zombie state visible if it ever returns.

- **2026-09-21 — model picker rebuilt for the 3-model fleet.** The picker carried 39 stale rows (37 dead Ollama models + 2 `frank-default-*`) and was missing `frank-medium`. Rebuilt the `model` table to exactly `frank-large`/`frank-medium`/`frank-small` and set `model_ids` on each connection (llama.cpp's Ollama-shaped `/v1/models` is invisible to OpenWebUI's discovery). Pointed the two on-host endpoints at `host.containers.internal` — the bridge refuses the host LAN IP for non-published ports. All three verified end-to-end via `/openai/chat/completions`.
+- **2026-09-21 — model picker rebuilt for the 3-model fleet.** The picker carried 39 stale rows (37 dead Ollama models + 2 `frank-default-*`) and was missing `frank-medium`. Rebuilt the `model` table to exactly `frank-large`/`frank-medium`/`frank-secondary` and set `model_ids` on each connection (llama.cpp's Ollama-shaped `/v1/models` is invisible to OpenWebUI's discovery). Pointed the two on-host endpoints at `host.containers.internal` — the bridge refuses the host LAN IP for non-published ports. All three verified end-to-end via `/openai/chat/completions`.

 - **2026-09-24 — SSO via authentik.** OIDC authorization-code flow wired per the [SSO (authentik)](#sso-authentik) section: application/provider/entitlements created in authentik, client secret in the host env file, roles from the `roles` scope claim. E2E coverage in `tests/e2e/specs/18-openwebui-sso.spec.ts`.

 - **2026-09-24 — flipped to SSO-only.** The OIDC round-trip verified in a real browser (login view renders no password form, only "Continue with authentik", and the flow lands on a signed-in session), so the local username/password path was retired: `ENABLE_LOGIN_FORM=false` + `ENABLE_PASSWORD_AUTH=false` in the quadlet and `ui.enable_login_form` set to `false` in the config DB (the DB wins over env, so both were changed together). `qa-e2e-admin` was demoted out of the `authentik Admins` group (it keeps `openwebui_users` / `homepage-admins` for app access).

+- **2026-10-07 — picker narrowed to `frank-secondary` only.** Per Frank: the only model accessible in OpenWebUI is `frank-secondary`. Rebuilt the `model` table to a single `frank-secondary` row (the old rows `frank-large`/`frank-primary`/`frank-small` are deleted), set `openai.api_configs` indices 0 (frank-large) and 1 (frank-primary) to `enable: false`, and retargeted index 2's `model_ids` from the stale `frank-small` label to `frank-secondary` — llama.cpp serves by alias, so the old label addressed a model name the server does not know. Pre-change snapshot: `/home/dank/Containers/open-webui/data/webui.db.bak-20261007` (+ `-wal`/`-shm`). Verified: `/api/v1/models/list` returns exactly `["frank-secondary"]`, chat completion answers `pong` (needs `max_tokens≥1500` — the Qwen3.5 thinking block consumes a 200-token budget), and anonymous API probes are all 401 (see [API access](#api-access-auth-gated)).
+
+- **2026-10-09 — re-keyed to the Protoss ids (wire rename).** The endpoints
+  began serving `frank-zealot` / `frank-dragoon` / `frank-archon`, so three
+  id-keyed objects had to move in the same window as the alias flip:
+  `model.id`+`name` (`frank-secondary` → `frank-zealot`, `params`
+  `{"context":{"length":200704}}` and the `meta.description` kept),
+  `access_grant` (`resource_type='model'`, `resource_id`, principal
+  `user`/`*`, permission `read`), and `openai.api_configs` per-backend
+  `model_ids` (index 2 = `:7771` is the enabled family backend; indices 0/1
+  = `:7770` and Dankserver `:7770` stay `enable: false`). Snapshot first:
+  `/home/dank/Containers/open-webui/data/webui.db.bak-20261009`. Apply from
+  inside the container (`podman exec -i open-webui python3 -`, sqlite3; the
+  image has no sqlite3 binary) then `systemctl --user restart open-webui.service`
+  (it is a rootless quadlet) to drop the cached model list. Verify in a real
+  browser as the family QA account: the welcome screen selects
+  `frank-zealot` and answers. **A stale id fails silently** — the picker
+  simply lists nothing, because OpenWebUI intersects the backend's
+  `/v1/models` with `model_ids`.
+
+  Law: **the model id is a foreign key in three tables.** Any future alias
+  change on an endpoint OpenWebUI fronts must ship with this re-key, in the
+  same deploy.
+
+- **2026-10-09 — composer is `contenteditable`, not `<textarea>`.** A
+  Playwright probe that waits for `textarea` after SSO times out on the
+  welcome screen: OpenWebUI 0.6 renders the composer as a contenteditable
+  div. The stable anchor is the model button (`getByRole("button", {
+  name: /selected model/i })`) and the welcome text "How can I help you
+  today?"; that snapshot is also the cheapest proof the picker is populated.
+
+- **2026-10-09 — `frank-dragoon` added to the picker.** Per Frank: expose the
+  halogen endpoint (service `frank-large`, `host.containers.internal:7770`)
+  beside `frank-zealot`. Three id-keyed writes (the trio the 10-07 rebuild
+  proved): `openai.api_configs` idx 0 `enable: true` (its `model_ids` already
+  carried `frank-dragoon` from the wire rename), a `model` row (`id`/`name`
+  `frank-dragoon`, `params.context.length` 262144 = the per-slot
+  `HALOGEN_CTX`, description of the v2 abliterated payload), and the global
+  `access_grant` `(model, frank-dragoon, user, *, read)`. Snapshot
+  `webui.db.bak-20261009b`, apply with host `python3` over ssh, then
+  `systemctl --user restart open-webui.service`. Verified: `/api/models`
+  lists `frank-dragoon` as admin AND as the QA `role='user'` account, and
+  `POST /api/chat/completions` through the app answers `pong` at
+  `max_tokens 1500`. Halogen's `/v1/models` is OpenAI-shaped, so discovery
+  would have worked too — `model_ids` stays authoritative.
+
 ## See Also

 - [[Dankclaw Stack]]
@@ -117,6 +117,18 @@ frank-large `health=200` throughout.
  the model is loaded. Trust the engine's `host memory left for everything
  else` line, not `free`.

+- **Filesystem prerequisite (other hosts).** ext4 and xfs take swapfiles
+  directly. btrfs needs `chattr +C` (NOCOW), no compression, and no
+  snapshots in the swapfile path — check `df -T` before planning, or
+  `mkswap` will reject the file.
+- **Renaming an *active* swapfile works (inode-based) but leaves a stale
+  path in `/proc/swaps`** — swap it off first, then rename, then `swapon`
+  (which is exactly what the step-3 consolidation does). Migration is
+  observable while it runs: in the 2026-09-12 run `swapoff` of 4 GiB took
+  65 s and used-RAM rose only ~3 GiB (pages touched during migration), the
+  rest relocating on disk. If used-RAM climbs toward the whole swapped
+  amount, let it finish rather than aborting.
+
 ## Trash

 The previous 8 GiB `/swap.img` (dated 2026-07-21) is staged, not deleted, at
@@ -0,0 +1,303 @@
+---
+type: reference
+title: Noctalia and Niri
+description: The noctalia v5 desktop shell on niri (workstation + blankpad) — config layers, remote IPC over ssh, custom-palette recolor, offline template rendering, and output rotation/pinning.
+tags: [desktop, noctalia, niri, blankpad, workstation, ghostty]
+timestamp: 2026-10-05T00:00:00-07:00
+---
+
+# Noctalia and Niri
+
+Noctalia (`/usr/bin/noctalia`) is the desktop shell on niri, running on both
+the workstation and blankpad. Since the 2026-09-10 unification both run the
+**bare `noctalia` v5 binary** (niri `spawn-at-startup "noctalia"`); the old
+Quickshell-hosted v4 architecture (`qs -c noctalia-shell`, JSON config) is
+retired on both hosts.
+
+## Config layers
+
+- Hand-edit declarative layer: `~/.config/noctalia/<name>.toml` (e.g.
+  `config.toml`). Main config: `~/.config/noctalia/config.toml`; the niri
+  config includes `noctalia.kdl` for display settings.
+- Runtime state file `~/.local/state/noctalia/settings.toml` loads LAST and
+  **overrides declarative values for the same keys — never hand-edit it**;
+  UI/IPC writes land there.
+- Main reference: `noctalia-reference` facts — IPC `noctalia msg <command>`
+  replaced the `qs -c noctalia-shell ipc call` wrapper; app templates live at
+  `/usr/share/noctalia/assets/templates/` (gtk, ghostty, alacritty, hyprland,
+  sway, wezterm, helix, btop, starship, qt, kde, cava, scroll, mango,
+  umbriel, foot, emacs, kitty); palette generation `noctalia theme [image]`.
+
+## Remote IPC over ssh (the D-Bus trap)
+
+`noctalia msg <command>` resolves the running instance via the D-Bus session
+bus. sshd-spawned non-login shells lack `DBUS_SESSION_BUS_ADDRESS`, so plain
+ssh exec prints `error: noctalia is not running` even when the shell IS
+running — do not conclude "shell dead" from this error. Working remote
+procedure (runs in the systemd user-manager context where the bus address
+exists):
+
+```
+systemd-run --user --pipe --wait noctalia msg <command> [args]
+```
+
+Verified: `msg status` returns live JSON state; unknown panel ids return an
+error listing all valid panel ids (clipboard, control-center, launcher,
+polkit, session, setup-wizard, test, tray-drawer, wallpaper).
+
+Verified v5 msg verb grammar (keybinds-safe): `panel-toggle <id>`,
+`session lock|suspend|lock-and-suspend|logout|reboot|shutdown`,
+`volume-up [step]`, `volume-down`, `volume-mute`, `mic-mute`,
+`media next|previous|playPause` (graceful no-op, rc 0, with no active
+player), `brightness-up [target] [step]`, `brightness-down`,
+`settings-open [context]`, `settings-toggle [context]`, `config-reload`,
+`wallpaper-get [connector]`, `wallpaper-set <path>`, `wallpaper-next`,
+`color-scheme-get|set`, `theme-mode-get`, `templates-apply`,
+`screenshot-fullscreen`.
+
+**Wallpaper directory semantics:** empty/unset `[wallpaper] directory`
+resolves to XDG Pictures via user-dirs.dirs; an explicit directory key
+overrides. Fleet default pattern: point `directory` at the repo-tracked
+wallpapers dir so git-tracked wallpapers serve the shell without copying
+files. After writing config.toml over ssh the file watcher hot-reloads
+automatically; `config-reload` via systemd-run forces it immediately. Note:
+the *current* wallpaper stays the built-in default until the next rotation
+or an explicit `wallpaper-next`; `wallpaper-get` then reporting a path
+inside the configured directory is the confirmation the config took effect.
+
+**Lock chain (unified):** swayidle-lock.service ExecStart on both hosts:
+`/usr/bin/swayidle -w timeout 300 '/usr/bin/noctalia msg session lock'
+before-sleep '…' after-resume '…'`. `msg` resolves via the session bus, so
+no WAYLAND_DISPLAY pinning is needed in unit context.
+
+## Custom-palette recolor (without restarting the session)
+
+Why custom, never the community cache:
+`~/.local/state/noctalia/community-palettes/*.json` is a **sync cache** —
+`.catalog/palettes.json` carries an `md5` per palette and noctalia re-fetches
+from `https://api.noctalia.dev/palettes`, so edits there get reverted.
+Custom palettes live at `~/.config/noctalia/palettes/<Name>.json`
+(`NOCTALIA_CONFIG_HOME` then `XDG_CONFIG_HOME` honored) and are never
+synced. Missing/invalid file → silent fallback to the builtin palette.
+
+Palette JSON shape = the community file's shape: `dark`/`light` maps of
+`mPrimary`, `mOnPrimary`, `mSecondary`, `mTertiary`, `mError`, `mSurface`,
+`mSurfaceVariant`, `mOutline`, `mShadow`, `mHover` plus a `terminal` block
+(`foreground`, `background`, `normal.{black..white}`, `bright.{…}`,
+`cursor`, `cursorText`, `selectionFg`, `selectionBg`). NOTE: `noctalia theme
+-o` emits M3 snake_case tokens (`on_primary_container`, …) — that is the
+*generator* format, not this one. Copy an existing community JSON and mutate
+with `jq`.
+
+Procedure:
+
+1. Read the live scheme first — never assume: `noctalia msg color-scheme-get`
+   (e.g. `community Gruber Darker`); `noctalia msg theme-mode-get`.
+2. Back up settings: `cp -a ~/.local/state/noctalia/settings.toml
+   /tmp/noctalia-settings-backup-$(date +%s).toml`.
+3. Build the palette from the current one (jq keeps structure):
+
+   ```sh
+   mkdir -p ~/.config/noctalia/palettes
+   jq '.dark.mPrimary = "#f43841"
+       | .dark.terminal.normal.yellow = "#f43841"
+       | .dark.terminal.bright.yellow = "#ff4f58"
+       | .dark.terminal.foreground   = "#f43841"' \
+      "$HOME/.local/state/noctalia/community-palettes/Gruber%20Darker.json" \
+      > ~/.config/noctalia/palettes/"Gruber Red.json"
+   ```
+
+   Prefer a hex already present in the palette (its own `mError` red) so the
+   theme stays internally consistent. Set BOTH `normal` and `bright`
+   variants, and both `dark` and `light` if the user toggles modes.
+4. Switch (persists `theme.source = custom` + `theme.custom_palette`):
+   `noctalia msg color-scheme-set custom "Gruber Red"` → `ok`; confirm with
+   `color-scheme-get`.
+5. Re-render every configured template: `noctalia msg templates-apply`.
+   Regenerates `~/.config/ghostty/themes/noctalia`,
+   `~/.config/gtk-{3,4}.0/noctalia.css`, kcolorscheme, foot, wezterm,
+   starship, etc. Symlinked-into-dotfiles paths show up as modified files
+   worth committing.
+6. Reload surfaces: compositor — `niri msg action load-config-file`
+   (`reload-config` is NOT a valid niri subcommand); ghostty — see the trap
+   below.
+
+Verification: `ghostty +show-config | grep -E '^(foreground|palette)'`
+(proves the theme file resolves — note it prints only the delta from
+defaults; cross-check with `ghostty +show-config --default` before
+concluding a key is unset); `noctalia config validate` → `✓ Config is
+valid`; visual via `noctalia msg screenshot-fullscreen` (saves to
+`~/Pictures/screenshot_<ts>.png` and ALSO overwrites the clipboard with the
+PNG — warn the user); `grep -c <old-hex> ~/.config/noctalia/palettes/*.json`
+→ 0 for no leftovers.
+
+Track it in the dotfiles repo (`~/.omp`, the command-center —
+`scripts/deploy.sh` is a stow-equivalent over `dotfiles/<pkg>/` trees):
+mirror the shape for the new palette so `git pull` + `deploy.sh` is the
+whole propagation story — move the JSON into
+`~/.omp/dotfiles/noctalia/.config/noctalia/palettes/` and symlink it back
+into `~/.config/noctalia/palettes/`; re-set the custom palette (re-reads
+through the symlink). Do the link by hand for one file instead of running
+`deploy.sh` — sync mode quarantines and replaces every divergent live file.
+
+Traps:
+
+- **NEVER `kill -USR1` ghostty to reload.** It does not reload config on
+  this fleet — it kills/respawns the terminal (took out Frank's window
+  2026-09-13). Correct: the default keybind `ctrl+shift+,` = `reload_config`
+  (per `ghostty +show-config --default`). Existing windows keep old colors
+  until reloaded; new windows inherit immediately.
+- Palette hex values are NOT a proxy for what the user sees: in
+  `Gruber Darker` the *accent* (`mPrimary`) is yellow `#ffdd33`, so "change
+  the yellow" means the accent too, not just terminal index 3.
+- Saturating terminal text makes ANSI 1/9 indistinguishable from default
+  text (grep highlights flatten) — flag it and offer a lighter red.
+- Contrast sanity: `#f43841` on `#181818` ≈ 4.6:1 (dark-on-color beats
+  white-on-color here, ~3.9:1 — keep `mOnPrimary` dark).
+- Revert = `noctalia msg color-scheme-set community "<original name>"`; the
+  community cache is untouched by the whole flow.
+- Generated theme files symlinked into the dotfiles repo (`gtk-3.0`/`gtk-4.0`
+  `noctalia.css`, ghostty theme) show up as tracked modifications after
+  `templates-apply`; commit them together with the new palette JSON so every
+  machine gets the same recolor.
+
+## Offline template render (when templates-apply never ran)
+
+Failure signature: niri `failed to read included config from
+~/.config/niri/noctalia.kdl: No such file or directory` — the desktop runs
+but with DEFAULT config (niri 26.04 falls back, does not exit): no custom
+keybinds/rules, no noctalia autostart, default output placement. ghostty:
+`theme "noctalia" not found, tried ~/.config/ghostty/themes/noctalia`.
+alacritty: import of `~/.config/alacritty/themes/noctalia.toml` missing.
+
+Root cause pattern: upstream dotfiles commits theme switches (niri include,
+ghostty `theme = noctalia`, alacritty import) that depend on noctalia
+TEMPLATE-GENERATED files. Those files are per-host outputs (rendered from
+the local palette) and are intentionally NOT in the repo; on any host where
+`templates-apply` never ran, they don't exist.
+
+Fix (offline, works while a session runs) — built-in templates live at
+`/usr/share/noctalia/assets/templates/` (declared in `builtin.toml`); render
+each needed one with the live palette:
+
+```bash
+# niri include (plain host-local file; do NOT symlink into repo — regenerated per host)
+noctalia theme --theme-json ~/.config/noctalia/colors.json \
+  --render "/usr/share/noctalia/assets/templates/niri/niri.kdl:$HOME/.config/niri/noctalia.kdl"
+mkdir -p ~/.config/ghostty/themes && \
+noctalia theme --theme-json ~/.config/noctalia/colors.json \
+  --render "/usr/share/noctalia/assets/templates/ghostty/ghostty:$HOME/.config/ghostty/themes/noctalia"
+mkdir -p ~/.config/alacritty/themes && \
+noctalia theme --theme-json ~/.config/noctalia/colors.json \
+  --render "/usr/share/noctalia/assets/templates/alacritty/alacritty.toml:$HOME/.config/alacritty/themes/noctalia.toml"
+```
+
+All template tokens (primary, surface, error, primary_container, …) resolve
+from the flat 16-key colors.json; container tokens are derived, never blank.
+
+Verify: `niri validate` → rc 0; `ghostty +validate-config` → rc 0; next
+login journal shows `loaded config from config.kdl` with NO parse WARN,
+custom output placement (`niri msg outputs` vs display.kdl), noctalia
+autostarted (`pgrep noctalia`).
+
+Traps: **`noctalia msg templates-apply` on 5.0.1 returns "ok" but applies
+NOTHING** (this build's opt-in config schema does not act on it) — do not
+trust its exit code, use the explicit `--render` calls above. Niri config
+errors at boot are NOT fatal — the compositor keeps running on defaults,
+which reads as "some settings silently stopped working", not a crash.
+Regenerate after any palette change; the rendered files are static
+snapshots.
+
+## Niri output rotation and pinning
+
+Verified on blankpad, niri 26.04 (two PM1601 externals + eDP-1 laptop panel).
+
+**Ground truth first:** `niri msg outputs` (connector names, current mode,
+scale, transform, logical pos/size) and `niri --version`. Output **names are
+connector names** (`eDP-1`, `DP-2`, `HDMI-A-1`), not EDID model strings —
+the EDID strings printed before them are decoration. On blankpad the live
+config splits via `include "./cfg/*.kdl"` from `~/.config/niri/config.kdl`,
+and `cfg/display.kdl` is a **symlink into the dotfiles repo** — edit the
+repo path (that is what gets committed); check with `ls -ld`.
+
+**Transform direction is counter-clockwise** (`niri msg output <NAME>
+transform --help`): `90` = content rotated 90° CCW ⇒ the panel's **top
+bezel sits on the LEFT**; `270` = top bezel RIGHT; `180` = upside down;
+`flipped-90`/`flipped-270` = mirrored variants. `90` vs `270` is a
+physical-stand fact no tool can read (portable monitors expose no
+orientation in EDID/MCCS) — pick one, apply, and hand the user the one-liner
+to flip; don't stall on the question.
+
+Two traps when editing the config:
+
+1. **An `output "X" { … }` block resets every property you don't restate to
+   its default.** Pinning position on `eDP-1` without repeating `scale 1.25`
+   silently drops the laptop panel to scale 1.0. Restate `scale` (and
+   `mode`) in every block you touch.
+2. **Niri re-runs auto-layout for outputs with no pinned position.** Pinning
+   only the externals shoved `eDP-1` from `0,0` to `3936,0`. Pin every
+   output's position to keep the left-to-right order. Positions are logical
+   pixels at each output's own scale; x-offsets chain (eDP-1 logical width
+   1536 → DP-2 at 1536 → +1200 portrait width → HDMI-A-1 at 2736):
+
+```kdl
+output "DP-2"     { scale 1 transform "90" position x=1536 y=0 }
+output "HDMI-A-1" { scale 1 transform "90" position x=2736 y=0 }
+output "eDP-1"    { scale 1.25 position x=0 y=0 }   // MUST restate scale
+```
+
+Apply and verify:
+
+```bash
+niri validate -c ~/.config/niri/config.kdl      # exit 0
+niri msg action load-config-file                # live reload; NOT reload-config
+sleep 1; niri msg outputs | grep -E '^Output|Transform|Logical|Scale'
+```
+
+Success looks like `Logical size: 1200x1920` (swapped) plus
+`Transform: 90° counter-clockwise` on each rotated panel, positions matching
+the intended chain. One-off rotation without touching config (temporary):
+`niri msg output DP-2 transform 90`. Commit via the dotfiles repo
+(`~/.omp`, `main`; push to the local Gitea permitted): `git pull --ff-only`
+→ add the display.kdl → commit → push. Pitfalls: `reload-config` is not a
+niri subcommand; never SIGUSR1 a terminal to "reload" (niri config reloads
+do not touch ghostty themes); no man page for niri on this host — `niri
+<sub> --help` and `/usr/share/doc/niri/default-config.kdl` are the docs.
+
+## GPU RGB over I2C (EVGA iCX3)
+
+Turn RGB on/off (or read it back) on an EVGA FTW3 / iCX3-controller GPU
+(the 3090 on dankclaw) using the GPU's own nvidia i2c bus. No display
+engine, no X, no daemon — the mode is EEPROM-saved and survives reboot.
+
+- **Find the controller:** the nvidia i2c driver exposes the card's lighting
+  ports as `/dev/i2c-N`; on the 3090 FTW3 that is ports 1,5,6,7,8 →
+  i2c-1..i2c-5, and only i2c-1 carries a live slave (NVIDIA port 1). Probe
+  each bus. The iCX3 MCU answers at address **0x2D**. `/dev/i2c-*` are
+  `root:i2c 660` — run with `sudo` (dank has passwordless sudo).
+- **Protocol (raw I2C_RDWR, NOT the SMBus ioctls)** — write the register
+  byte first, then the payload:
+  - enable `0xB2` write `04 C6 EB EA 15` (required once before mode writes)
+  - mode `0xC0` write `09 m0 m1 m2 m3 .. .. .. .. SYNC`; m0..m3 are the four
+    zone modes, `MODE_OFF = 0x00`; OFF = `09 00 00 00 00 FF FF FF FF 00`
+  - save `0x90` write `04 9E EB 00 90` (commit to EEPROM — the
+    reboot-persistence step)
+  - fw `0xB1` block read (firmware version)
+  - Read the mode back: write reg `0xC0`, then read 16 bytes; bytes 1..4 are
+    the zone modes (0x00 = OFF), bytes 6..9 the per-zone LED counts.
+- **Verify persistence:** after a reboot, read the mode — all four zone
+  modes `0x00` means the EEPROM save held (confirmed 2026-09-19 post-reboot).
+- **Tool:** a ~90-line I2C_RDWR reader/writer lived at `/tmp/evga_raw.c`
+  (`evga_raw <bus> mode|off|save|enable|fw`) — `/tmp` is wiped by reboot, so
+  recreate it if needed; OpenRGB's `EVGAGPUv3Controller.cpp` carries the
+  same protocol plus the full colour API.
+- Gotchas: raw I2C_RDWR only (the iCX3 answers the raw form, not the SMBus
+  ioctls); the `enable` write must precede the first mode write in a
+  session; without the `save` write the mode is RAM-only and resets on power
+  cycle.
+
+## Related
+
+- [[Command Center Dotfiles]] — the stow-equivalent deploy this page links into
+- [[Hermes]] — the agent that drove the CDP daemon alongside this desktop
+- [[GGUF Compatibility]] — unrelated but shares the "read the file, not the name" lesson
@@ -5,3 +5,115 @@ description: "The *arr automation stack (Prowlarr, Sonarr, Radarr, Lidarr, Reada
 tags: [arr-stack, prowlarr, sonarr, radarr, lidarr, readarr, qbittorrent, frankbook-pro, dankserver]
 timestamp: 2026-09-04T00:00:00Z
 ---
+
+# *arr Stack
+
+The *arr automation stack runs on **FrankBook-Pro** (the Mac mini) as quadlets
+inside its podman machine; all egress rides the gluetun WireGuard netns. The
+per-service pages ([[Prowlarr]], [[Sonarr]], [[Radarr]], [[Lidarr]],
+[[Readarr]], [[qBittorrent]]) carry deployment facts; this page holds the
+cross-cutting API and auth-posture procedures.
+
+## API version split (memorize before any *arr HTTP call)
+
+- Sonarr/Radarr: `/api/v3/…`; Lidarr/Readarr/Prowlarr: `/api/v1/…`.
+- Prowlarr is port **9696** — 8090 is Jackett.
+- **Empty body = endpoint absent**, not a failure: probe endpoints by status
+  code first; v1 has no `/api/v1/profile` endpoint at all.
+- `PUT /downloadclient` and `PUT /indexer/{id}` take a SINGLE resource object
+  (GET it, patch `settings`, PUT the whole object back). An array body fails
+  with `Path: $ | BytePositionInLine: 1`.
+- API keys live in each container's config:
+  `podman exec <svc> grep -o "<ApiKey>[^<]*" /config/config.xml | cut -d">" -f2`.
+  They are local-only secrets — they stop being secret the moment a vhost
+  serves the API unauthenticated.
+
+## Schema-first probing (validators are free documentation)
+
+Send the FULL schema template object with fields filled; v1 validators reject
+partials, and each error names the next required field — peel validators one
+live call at a time instead of guessing payload shapes:
+
+- `'Fields' must not be empty` → send a full schema copy, not `fields:[]`.
+- `'Base Url' must not be empty` when keys were camelCase → **PascalCase is
+  required** (`BaseUrl`, `ApiKey`) — schema keys, not the JSON wire keys.
+- `DefaultMetadataProfileId must be > 0` → fetch real ids from
+  `GET /api/v1/metadataprofile` (rows like `(1,'Standard')`);
+  `GET /api/v1/qualitydefinition` returns the quality CATALOG — ids ≠
+  profiles.
+- `targetType null` → configContract not registered in that build (this
+  Prowlarr build has no Prowlarr-plugin indexer; integration is
+  Torznab-family only).
+- `testingPackageId` in a PUT body downgrades validation failures to
+  warnings.
+- `syncLevel` is enum `ApplicationSyncLevel` — omit it or use the int; a
+  string is rejected.
+
+**Verification rules:** `POST /downloadclient` returns the COLLECTION, not
+the created object — verify via GET. Test endpoints return verdicts in the
+BODY at HTTP 200 (`{}` = pass, a list = fail) — judge from bodies, not
+status codes. Command POSTs (`MoviesSearch`, `ApplicationSync`) run
+server-side and respond only when the work finishes: curl `-m` timeouts
+(HTTP 000) during a search are NOT failures — poll queue/history afterward
+instead of re-firing; warm small commands (`RefreshMonitoredDownloads`)
+answer in ~20 ms, so a hang means live work in progress, not a dead API.
+
+**DB as arbiter:** when endpoint shapes are opaque, stop the container and
+edit `/opt/prowlarr/config/prowlarr.db` with `sudo python3` sqlite directly,
+then start. The indexers table columns are
+Id/Name/Implementation/Settings/ConfigContract/Enable/Priority/Added — the
+enable flag column is `Enable` (not `EnableRss`). Prowlarr's real application
+settings values live in `prowlarr.db` → `Applications.Settings` JSON (the
+`/api/v1/applications` GET responses carry `fields: []`).
+
+## Auth posture probe (ground truth is behavior, not config flags)
+
+`AuthenticationMethod=None` + `AuthenticationRequired=Enabled` in
+`config.xml` is ambiguous on paper; on the wire it means a login screen with
+**zero users** — first-account-wins.
+
+1. **Confirm which host runs the service before probing** (`podman ps` on
+   the target, not the box you ssh'd into first — the arrs live on
+   FrankBook-Pro).
+2. Probe an API endpoint that bypasses the SPA shell, locally and through
+   any vhost: `curl -s -o /dev/null -w "%{http_code}"
+   http://<host>:<port>/api/v3/system/status`. **SPA roots (`/`) always
+   serve a 200 shell — never conclude "open" or "gated" from `/`.**
+   401 = gated; 200 + JSON = API fully open (anyone on the net can drive it
+   with curl).
+3. For config truth, dump the whole file
+   (`podman exec <svc> cat /config/config.xml`), not a piped
+   `grep | sed -n 2p`: **`sed -n Np` exits 0 on zero matches**, so the
+   pipeline hides a miss and you conclude "empty field" when the grep itself
+   failed.
+4. Decision framing: when posture = gated-with-zero-users, present it as a
+   decision menu (None behind the front-door wall vs External/authentikarr),
+   not a silent change — first-account-wins means whoever logs in first owns
+   the instance.
+
+## The `None` + `Enabled` auth-config trap (v1 .NET apps)
+
+`<AuthenticationMethod>None</AuthenticationMethod>` +
+`<AuthenticationRequired>Enabled</AuthenticationRequired>` is
+self-contradictory: the UI Save is permanently disabled, and any save throws
+`DryIoc.ScopedItemException` in `AuthorizationMiddleware` → 500 on every
+page after restart. The valid no-auth representation is **absence of the
+`<AuthenticationRequired>` key**, not the literal string `Disabled`. Fix =
+delete the key from `/config/config.xml`
+(`sed -i "/<AuthenticationRequired>/d"`), restart the container. Tell-tale:
+one v1 app boots clean while siblings 500-sprawl → diff their config.xml
+auth blocks; the healthy one shows the correct shape. (Prowlarr runs
+`Forms` + `Enabled` healthily — the crash combo is `None`+`Enabled` only.)
+
+General rules that saved the session these came from: container
+`State=running` ≠ app serving — read `podman logs --tail 12 <c>` before
+theorizing; when a UI gate blocks saving, go under it (these apps keep
+plaintext XML config on disk — read/edit via `podman exec`).
+
+## See Also
+
+- [[Prowlarr]] — indexer feeds and the v2 sync model
+- [[Sonarr]] · [[Radarr]] · [[Lidarr]] · [[Readarr]] — the managers
+- [[qBittorrent]] — download client, 4.6.7 pin, E2E verification
+- [[FrankBook-Pro]] — the host and its podman machine
+- [[Homepage Status Probes]] — how the e2e gate treats this sleeping host
@@ -111,6 +111,90 @@ Restart=always
  probe one ssh-context out returns the residential IP — a false leak alarm.
 - Ensure umask is set to `022` so downloaded files are readable by arr apps.

+## E2E verification after any pipeline change
+
+Run after ANY change to the torrent pipeline — WireGuard credential rotation,
+qB version change, gluetun reconfig, category re-seeding. Proves the full
+grab path (tunnel egress → qB download → category save path → clean
+removal). All commands from a dev box via `ssh frankbook-pro`; inside the
+Mac shell export `PATH="/usr/local/bin:/opt/homebrew/bin:$PATH"` for
+`podman machine ssh`.
+
+**Access note (2026-09-21):** the Mac's login user is **dank**@192.168.0.112
+(NOT core — `core` is the podman-machine VM user, reached only via
+`podman machine ssh`). blankpad has key auth for `dank@` (bootstrapped
+2026-09-21) and a `frankbook-pro` alias in `~/.ssh/config`. macOS sshd has a
+low MaxAuthTries: when password-bootstrapping use `-o
+PreferredAuthentications=password -o PubkeyAuthentication=no
+-o IdentitiesOnly=yes` or the key offers exhaust the attempts before the
+password is tried.
+
+**1. Egress gate — MUST run inside gluetun:**
+
+```bash
+podman machine ssh 'podman exec gluetun wget -qO- https://ipinfo.io/json'
+```
+
+Require `org` = AS9009 (M247 — ProtonVPN Panama PA-12 config); judge by org,
+never country alone. NEVER run this one ssh-context out (bare `wget` on the
+Mac host): it silently returns the residential Comcast IP and reads as a
+hard-gate leak that does not exist. A residential reading ⇒ re-run inside
+the container before diagnosing.
+
+**2. Login — generic cookie parse.** The QPW fetch is flaky (~1 in N returns
+empty); assert length before use — an empty-password run once produced a
+Forbidden loop that cost hours:
+
+```bash
+QPW=$(podman machine ssh 'cat /home/core/Containers/qbittorrent/env' | grep QBIT_WEBUI_PASSWORD | cut -d= -f2)
+[ ${#QPW} -eq 20 ] || exit 1
+SID=$(curl -si --data-urlencode "username=admin" --data-urlencode "password=$QPW" \
+  http://127.0.0.1:8080/api/v2/auth/login \
+  | grep -oE '[A-Za-z_]*SID[A-Za-z0-9_]*=[^;[:space:]]*' | head -1)
+curl -b "$SID" …   # pass the WHOLE match
+```
+
+WebUI from the LAN: `http://192.168.0.112:8080` (published on the gluetun
+unit). Username `admin`; password = the machine-local `QBIT_WEBUI_PASSWORD`
+(20 chars, generated once at provisioning by the deploy handler — never in
+the repo). Parse-bug 403s never reach qB as failed logins — no ban accrues;
+server-side truth is `/Volumes/COMPLETE/qbittorrent-data/qBittorrent/config/qBittorrent/logs/qbittorrent.log`
+(`WebAPI login success/failure` + source IP). Repeated REAL failures accrue
+toward qB's 5-failures/3600 s IP ban.
+
+**3. Add a legal test torrent.** Blender open-movie torrents (~300 MB, e.g.
+Big Buck Bunny) via `POST /torrents/add` with the raw `.torrent` file
+(`-F "file=@…"`) and `category=Finished Torrents`. Never leave a real-looking
+grab behind.
+
+**4. Poll and verify.** Poll `GET /torrents/info` (~20 s): state
+`downloading` → `moving` (INCOMPLETE→COMPLETE move) → `stalledUP`/`uploading`
+at progress=1. `save_path` must be `/Volumes/COMPLETE/Finished Torrents`
+(qB 4.x auto-creates unknown categories at DefaultSavePath; 5.x did NOT
+inherit category paths — one more reason for the 4.6.7 pin). Confirm on
+disk: `ls "/Volumes/COMPLETE/Finished Torrents/"`. Throughput ~17 MB/s is
+normal for the tunnel.
+
+**5. Cleanup — delete only your artifact:**
+
+```bash
+curl -s -o /dev/null -w "%{http_code}" -b "$SID" \
+  --data-urlencode "hashes=<literal-hash>" \
+  --data-urlencode "deleteFiles=true" \
+  http://127.0.0.1:8080/api/v2/torrents/delete   # expect 200
+```
+
+Target the known hash literally — the user's real grabs share the directory
+and MUST be untouched. Verify: hash absent from `/torrents/info`, your dir
+gone, all other entries intact.
+
+**Version-coupling constraints:** qB is pinned to 4.6.7 (Readarr's EOL
+client rejects qB ≥ 5's `204` login at runtime — see the pin rationale
+above). Do not unpin while Readarr exists. After ANY qB version flip,
+regenerate the `WebUI\Password_PBKDF2` line fresh (same password, 100k
+iterations) — a 5.x-written hash line makes 4.6.7 reject every login.
+Rewrite while qB is stopped.
+
 ## See Also

 - [[ARR-Stack]] — full *arr stack documentation
@@ -131,6 +131,55 @@ ungated), and gitea (git smart-HTTP clients cannot solve JS challenges —
   distinguish challenge from deny at this release; grep the body for
   `Access Denied` / `Making sure you` instead.

+## Guard-chain layer diagnosis (a user reports a guarded page "broken")
+
+Use when a guarded page (e.g. the `www.franksdank.cloud` homepage) reads as
+"broken / links gone / not loading" and the fault must be localized to
+server-side or client-side/session. Chain topology:
+`Browser → Caddy TLS edge → Anubis :8923 → Caddy :8016 → Authentik outpost
+→ Caddy :8017 (@home/@data handlers) → app/static`.
+
+Layer-by-layer checks (run from the host via ssh):
+
+1. **Caddy unit + guard containers**: `systemctl --user is-active
+   caddy.service`; `podman ps --format '{{.Names}}\t{{.Status}}' | grep -iE
+   'anubis|authentik|caddy'` — expect `Up … (healthy)`.
+2. **Anubis challenge page validity**: `curl -sk
+   https://www.franksdank.cloud/ --resolve www.franksdank.cloud:443:192.168.0.143`
+   — a healthy challenge contains `"anubis_challenge"` JSON with
+   `difficulty` and the main JS.
+3. **Outpost errors**: `podman logs --tail 40 authentik-outpost 2>&1 | grep
+   -iE 'error|exception|traceback|500|provider'` — routine dramatiq task
+   logs are normal.
+4. **Post-auth handlers from INSIDE the caddy container** (8016/8017 live in
+   its netns — `curl http://127.0.0.1:8017` from the HOST returns `000`
+   connection refused, which is EXPECTED, not a fault):
+   `podman exec caddy sh -c 'wget -q -S -T5 --header "Host:
+   www.franksdank.cloud" -O- http://127.0.0.1:8017/'` and the same for
+   `/services.json`, `/status.json`. Expect 200 + body. (BusyBox wget in
+   this container has NO `-H` flag — use `--header "Host: …"`.)
+5. **Files on disk**: `ls -la /srv/homepage/` and validate `status.json`
+   parses (`python3 -c "import json;json.load(open('/srv/homepage/status.json'))"`).
+
+Tool/behavior notes: the anubis container is distroless — `podman exec anubis
+sh` fails (`sh: executable not found`); inspect via its logs or the served
+challenge page. Anubis logs `x-real-ip: 10.89.1.22` (the Caddy container's
+podman IP) for ALL clients — rootless-podman published-port SNAT;
+clients are indistinguishable by IP and clearance is cookie-based. Do NOT
+mistake that for the bug. Homepage render: `app.js` does
+`Promise.all([fetch services.json, fetch status.json])`; on ANY fetch
+failure the catch sets `#summary` to `status unavailable: <err>` and the
+grid stays EMPTY — which a user reads as "links gone / not loading." A
+30-service status.json with a 9-service services.json renders fine
+(stateByName lookup).
+
+**Decision:** if every layer is healthy (challenge valid, outpost clean,
+handlers 200, files render), the fault is client-side/session. STOP
+thrashing the live system — repeated failed browser challenges share the
+masked 10.89.1.22 identity. Get the authenticated view: ask for the exact
+symptom (challenge spinning? login loop? empty grid? "status unavailable"
+text?) or the authentik login to drive the page in a real browser.
+
 ## Verify live

 ```
@@ -342,6 +342,178 @@ The silent re-auth path was proven end-to-end: after deleting
 `qa-e2e-admin`'s rows, the e2e admin spec ran against the stale storage
 state and passed with no login prompt (2 tests, 2.9 s).

+## Guard-chain flip recipe (adding a guarded vhost)
+
+Move a public vhost from `reverse_proxy <app>:<port>` onto the chain:
+client → TLS edge → Anubis :8923 → outpost decision → internal
+`http://:8017` listener → Host-matched handler → real upstream.
+
+**Order (never reorder):**
+
+1. **Provider first.** Create the outpost ProxyProvider before touching
+   routes, or authed traffic hits the outpost error page instead of the app.
+   Fleet convention for `external_host` is the exact browser-facing origin
+   (the inventory table carries both bare origins and trailing slashes;
+   routing matches on hostname, verified 2026-09-02).
+   `internal_host = http://caddy:8017`, all three flows set, and an
+   Application attached — the two provisioning rules above.
+2. **Route flip in ONE commit/deploy**: the vhost becomes
+   `reverse_proxy anubis:8923` with `header_up X-Real-Ip {remote_host}`, and
+   inside the `http://:8017` listener add `@<svc> host <vhost>` +
+   `handle @<svc> { reverse_proxy <upstream> }`. Anubis guard set, Caddy TLS
+   edge, `:8017` routing table, and provider list change together.
+3. Deploy via the repo deploy flow (commit first; deploy pushes main itself).
+
+**Verification (all three):**
+
+- Unauthenticated probe from a client: `curl -s https://<vhost>/` → HTTP 200
+  challenge page; grep the body for `challenge|not a bot|anubis` (~8
+  markers on a live challenge). A 502 here instead of the challenge = chain
+  mis-wired.
+- Post-auth routing proof: grep the LIVE config inside the container —
+  `podman exec caddy sh -c "grep -n -A3 <handler> /etc/caddy/Caddyfile"` —
+  after the deploy prints its reload line; confirm the reload actually ran,
+  don't trust exit 0.
+- Outpost reachability: `podman exec authentik-server getent hosts caddy`
+  resolves the compose DNS alias (10.89.1.x) that `internal_host` dials.
+
+**Pitfalls:** probing `http://localhost:8017/` from the host returns HTTP
+000 (connection refused) — the post-auth listener lives in the caddy
+container's netns; probe inside it or grep the config. Non-browser clients
+cannot solve the JS challenge — before flipping, audit service-to-service
+traffic: ingestion/push paths must target internal addresses (`loki:3100`),
+never a public hostname of a guarded service. Exempt-by-construction
+categories stay unguarded: the IdP itself (`auth.`), git smart-HTTP
+endpoints, services with their own complete auth — document the exemption
+in the vhost comment.
+
+## Driving the ORM (`ak shell`)
+
+All authentik object work happens through the server container's ORM shell —
+no repo change, no deploy (these providers are NOT repo-managed; capture new
+provider+application pairs as a blueprint or setup script to avoid drift).
+
+```bash
+ssh dankclaw 'podman exec -i authentik-server ak shell' <<'EOF' | sed -n '/229 objects/,$p'
+from authentik.core.models import User, Group
+...
+EOF
+```
+
+- The heredoc IS the program: `ak shell` reads all of stdin and execs it
+  (heredoc-through-ssh quoting breaks; the raw-stdin pattern is the proven
+  one).
+- Bootstrap prints JSON log lines, then
+  `### authentik shell (2026.8.3)` and `229 objects imported automatically…`
+  — pipe through `sed -n '/229 objects/,$p'` to drop the noise.
+- 229 models are pre-imported. Import paths (each wrong guess costs a failed
+  exec): `User`, `Group`, `Application`, `ApplicationEntitlement`,
+  `PropertyMapping` → `authentik.core.models`; `ProxyProvider` →
+  `authentik.providers.proxy.models` (NOT `authentik.outposts.models`);
+  `LDAPProvider` → `authentik.providers.ldap.models`;
+  `authentik.core.models.application` does NOT exist as an import path; there
+  is no `authentik.policies.group` module in this build.
+- For live DB truth prefer the postgres container
+  (`podman exec authentik-postgres psql -U authentik -d authentik -At -c
+  "select * from …"`) — **query with `select *` and stop guessing column
+  names**: 2026.x renamed columns (`Provider.objects.filter(type=…)` raises
+  FieldError — `type` moved to subclass tables; the outpost `enabled` column
+  may not exist under that name; provider pk is `oauth2provider_ptr_id`).
+
+## User provisioning playbook (user-level account)
+
+There is no `User.create_user` classmethod, and `is_superuser` is NOT a
+constructor kwarg (it is a property derived from group membership) — passing
+it raises `TypeError: User() got unexpected keyword arguments`:
+
+```python
+from authentik.core.models import User, Group
+u = User(username="ridley", email="ridley@franksdank.cloud", name="Ridley", is_active=True)
+u.set_password("...")        # password arrives on stdin only, never argv
+u.save()
+for g in ("Non-Admin Users", "jellyfin_users", "openwebui_users"):
+    u.groups.add(Group.objects.get(name=g))
+```
+
+`path` defaults to `users` (regular). The access map (verified live
+2026-10-04): there are NO per-app policy bindings on the proxy providers
+(all use the implicit-consent "Authorize Application" flow), so guarded-vhost
+access is decided entirely by GROUP MEMBERSHIP consumed downstream —
+
+- `homepage-admins` → Caddy RBAC for homepage `/admin` (`X-Authentik-Groups`
+  regex gate) and the homepage admin panel; regular users get only
+  `category: user` panels from `services.json`.
+- `openwebui_users` / `openwebui_admins` → ApplicationEntitlements
+  "Open WebUI Users"/"Open WebUI Admins" (bound by group), consumed by the
+  "Open WebUI roles" property mapping → `roles` claim. Verify live with
+  `u.app_entitlements(app)`.
+- `jellyfin_users` / `jellyfin_admins` → Jellyfin via the LDAP outpost
+  (filter `(memberOf=cn=jellyfin_users,ou=groups,dc=franksdank,dc=cloud)`).
+- `Non-Admin Users` = the plain-user marker group (precedents: qa-e2e, and
+  every human account).
+- **The human-user template** (ridley, kassie, alex, artem, kirby, ry, nic,
+  debo, freya — kirby/ry/nic/debo/freya (pk 17–21) created 2026-10-09, kassie
+  and alex created 2026-10-08, artem (pk 16) created 2026-10-07): `username` +
+  `<user>@franksdank.cloud` + display name +
+  `is_active=True` + the password set via `set_password`, membership in
+  `Non-Admin Users` + `jellyfin_users` + `openwebui_users`. `alex` was
+  recreated from scratch on 2026-10-08 into this shape (the old row had an
+  empty email and no `openwebui_users`; audit first — the row carried no MFA
+  device, token, entitlement, or proxy session, and events survive a user
+  delete because `Event.user` is a denormalized JSON field, not a FK).
+- EXCLUDE from a user-level account: `homepage-admins`, `jellyfin_admins`,
+  `openwebui_admins`, `authentik Admins`, `authentik Read-only`.
+- Other user-category vhosts (SearXNG, Gitea, Calibre-Web, RomM, Copyparty,
+  Plex, Seerr, Marreta) need no per-user group: any authenticated user
+  passes the outpost.
+
+Verify login behaviorally — `authenticate()` is the exact PasswordStage code
+path:
+
+```python
+from django.contrib.auth import authenticate
+assert authenticate(request=None, username="ridley", password="...")   # -> User
+assert authenticate(request=None, username="ridley", password="nope") is None
+```
+
+Look flows up by slug, not name:
+`Flow.objects.get(slug="default-authentication-flow")` (its `.name` is
+"Welcome to authentik!" — `.get(name=…)` fails). Stage order:
+Identification(10) → Password(20) → AuthenticatorValidate(30; no-op with no
+enrolled device) → UserLogin(100). With no MFA enrolled, nothing else blocks.
+
+More field gotchas (each cost a failed exec): `ApplicationEntitlement.application`
+can be null (guard with `getattr`); `LDAPProvider` has
+`search_mode`/`bind_mode`/`base_dn`, lacks `search_base`/`of_groups`/
+`organization`; `OAuth2Provider` has no `.scopes`; `Outpost.config` is an
+`OutpostConfig` object, not a dict; `Policy` has no uniform `.policy_cls`
+(use `type(p).__name__`); `User` has no `.status`.
+
+Never rotate `AUTHENTIK_SECRET_KEY` to fix claim issues — it invalidates
+every outpost session fleet-wide; purge the stale `proxysession` rows
+instead (section above).
+
+## Probing the outpost from inside the netns
+
+Rootless podman gives each container its own netns; **host `ss -ltn` cannot
+see listeners inside other containers' namespaces** — a missing listener on
+the host is an observation artifact, not a fault. Distroless images have no
+shell (`podman exec authentik-server sh` fails), so probe by joining the
+namespace:
+
+```
+podman run --rm --network container:<container> --entrypoint /bin/busybox \
+  docker.io/library/busybox:latest \
+  wget -q -T 4 -t 1 -O /dev/null -S "<url>" 2>&1 | head -8
+```
+
+- Quote URLs over ssh: the remote shell is zsh and bare `?` in query strings
+  triggers globbing (`no matches found`).
+- Boot transience: immediately after a restart the same probe can return 503
+  (configure task pending); poll until the answer changes before concluding.
+- `podman inspect --format "{{.Config.NetMode}}"` errors on this podman
+  version — the field is `.HostConfig.NetworkMode`.
+
 ## Failure modes

 | Symptom | Cause | Fix |
@@ -0,0 +1,158 @@
+---
+type: reference
+title: Homepage Status Probes
+description: The homepage TUI status page — how probe targets are chosen (host-netns vantage), the three false-down shapes, triage procedure, local render check, deploy, and the e2e gate interactions (asleep Mac mini, raw-IP matrix).
+tags: [homepage, status, probes, e2e, homepage-status]
+timestamp: 2026-10-05T00:00:00-07:00
+---
+
+# Homepage Status Probes
+
+The homepage is a static site —
+`shared/homepage/{index.html,admin.html,app.js,style.css,services.json,docs.html}`
+— served from `/srv/homepage` on **dankclaw** only (`deploy_homepage` in
+`scripts/deploy-remote.sh` refuses other targets). A checker
+(`scripts/homepage-status.py`, run by the `homepage-status.timer` unit) probes
+each card's target and writes `/srv/homepage/status.json` (mode **0644** so
+Caddy can serve it — losing the mode 403s the page).
+
+## The checker's vantage and its three false-down shapes
+
+The checker probes TCP/HTTP **from the host netns**. It only has three states
+— `wip` | `up` | `down`; there is no "unknown", so an unprobeable service can
+never render honestly and must be dropped or re-pointed, not "fixed".
+
+1. **TLS-only app probed with `method:"http"`.** Plex Media Server serves TLS
+   only on `32400`; plain-HTTP GET → TLS alert → curl exit 52, code `000` →
+   card says down while PMS answers `200` over HTTPS. Fix:
+   `"method":"tcp"` (connect is the honest liveness signal; the public URL
+   stays HTTP because Caddy re-keys).
+2. **Retired port.** The Hermes card probed `127.0.0.1:8090` after the
+   compose→quadlet move moved the dashboard to `9119` and the webui to
+   `8787`; nothing listened on 8090, so a healthy stack read `down`.
+3. **Unpublished port / wrong netns.** `gitea-postgres` (`db:5432`, net
+   `systemd-gitea`), `searxng-valkey` (`valkey:6379`, net `systemd-searxng`)
+   and `alloy` (OTLP 12345) publish **no** ports. Container DNS aliases are
+   netns-scoped, so from the host these are unreachable *by construction* —
+   no probe value makes them honest. Drop those cards.
+
+A host-vantage probe can only reach a listener **published to the host**
+(rootlessport/pasta); unpublished ports are invisible from the host netns.
+See [[Container Host Reachability]] for the netns rules.
+
+## Triage procedure
+
+1. Read the card's probe target in `shared/homepage/services.json`; note
+   method (`tcp` vs `http`).
+2. Does that port have a **host-visible listener**?
+   `podman ps -a --format '{{.Names}}|{{.Status}}'`, then per container
+   `podman inspect <c> --format '{{json .NetworkSettings.Networks}}
+   {{.HostConfig.NetworkMode}}'` and `podman port <c>` (empty output =
+   published nothing → structurally unprobeable from the host).
+3. Probe from the **correct vantage**: from a bridge-net container the host
+   is its gateway IP (`/proc/net/route`, e.g. `10.89.1.1` on
+   `dankclaw_default`) — *not* `host.containers.internal` (`169.254.1.2`),
+   which belongs to the default `podman` net (`10.88.0.0/16`). Probe with
+   bash `/dev/tcp` (no `python3`/`ip` in these images); escape loop vars so
+   they expand **inside** `podman exec bash -c`.
+4. Classify: real outage (container exited — read `State.ExitCode`: 137 =
+   SIGKILL, 143 = SIGTERM/manual stop; `OOMKilled=false` rules out the OOM
+   killer) vs false down (running + unreachable target).
+5. Fix: re-point or re-method for HTTP/TLS mismatches; **drop** cards whose
+   only exposure is an internal netns; never invent a probe target.
+6. Verify by re-running the checker path, not by eyeballing: after deploy,
+   re-read `/srv/homepage/status.json` and diff the `down` list
+   before/after.
+
+Removing a card's `probe` key without a code change crashes the checker
+(`svc["probe"]` KeyError) — dropping the whole entry is the only zero-code
+fix. The `description` field is consumed only by `app.js` `descHTML()`, so
+adding a `details` object (role/spec/inputs/use) is schema-safe:
+`tests/homepage_config.bats` checks only name/category/section/probe.
+
+## Local render check (before every deploy)
+
+The page fetches `services.json` + `status.json` at load; `file://` is
+CORS-blocked, so serve over HTTP:
+
+```bash
+cd shared/homepage
+python3 -c "import json; s=json.load(open('services.json')); \
+  json.dump({'services':[{'name':x['name'],'state':'up'} for x in s]}, open('status.json','w'), indent=2)"
+python3 -m http.server 8899 --bind 127.0.0.1
+```
+
+Then in the browser: load `http://127.0.0.1:8899/index.html`, **wait ~1.5 s**
+(panels render after the async fetch; checking immediately returns 0 panels),
+extract via `tab.evaluate` — LLM cards render `.desc-role`/`.desc-spec`/
+`.desc-inputs`/`.desc-use` from the card's `details` object, other cards a
+plain `.desc`; `getComputedStyle(el).color` confirms tones. Screenshot for
+visual proof. Expected local noise: `deployed.js` 404s (baked in at deploy
+time) — harmless.
+
+The scratch `status.json` is untracked and NOT git-ignored — move it to
+`shared/homepage/.AGENTTRASH/` after the check (repo policy: never `rm`).
+
+CSS palette lives in `style.css` `:root` (`--fg` bright green, `--dim` dim,
+`--head` pale, `--link` cyan, `--wip` yellow, `--down` red). Add new tones as
+`--*` vars and include any new `.desc-*` class in the `html[data-bare="1"]`
+reset so `?bare=1` strips it to black.
+
+## Deploy
+
+```
+./scripts/deploy.sh dankclaw --service homepage --skip-e2e
+```
+
+Copies the assets to `/srv/homepage`, regenerates `deployed.js` (git commit
+datetime), restarts the `homepage-status` timer. Verify live:
+`ssh dank@192.168.0.143 'grep … /srv/homepage/…'`.
+
+**`--skip-e2e` is justified for homepage-only changes when the Mac mini is
+asleep**: the e2e admin crawl (`tests/e2e/specs/11-admin.spec.ts:24`) walks
+Prowlarr at `192.168.0.112:8989` on FrankBook-Pro; an asleep mini (ARP STALE,
+no remote wake) times out that test — a pre-existing environmental failure,
+not a regression. All non-Mac-mini e2e tests pass.
+
+## e2e raw-IP *arr failures — probe from the browser's box, not a peer
+
+When `11-admin` fails with `ERR_CONNECTION_TIMED_OUT`, the instinct is "the
+Mac is asleep". Sometimes true; not always. Playwright runs on the dev box,
+so a healthy probe from dankclaw proves nothing about the **browser's** path:
+
+```bash
+# 1. From the box that RUNS the browser:
+curl -s -o /dev/null -w '%{http_code}\n' --max-time 7 http://192.168.0.112:8989/
+# 2. From a LAN peer:
+ssh dankclaw 'for p in 8989 7878 9696; do curl -s -o /dev/null -w "$p %{http_code}\n" \
+  --max-time 6 http://192.168.0.112:$p/; done'
+# 3. Liveness (ICMP, not TCP):
+ping -c 2 -W 3 192.168.0.112
+```
+
+| peer probe | dev-box probe | ping | meaning |
+|---|---|---|---|
+| 200 | 200 | ok | transient — re-run the spec |
+| timeout | timeout | fails | Mac asleep; wake, ports return in seconds |
+| **200** | **rc=28** | **ok** | **path problem, not sleep** — ICMP allowed, TCP blocked. Waking cannot help |
+
+Third row: confirm the Mac is innocent through the dankclaw jump
+(`pgrep -fl socat`; `netstat -an | grep -c 8989`; loopback
+`curl http://127.0.0.1:8989/` → 200; `/usr/libexec/ApplicationFirewall/socketfilterfw
+--getglobalstate` → disabled). Forwarders up + loopback 200 ⇒ the block is
+between the dev box and the Mac.
+
+Don't hold a deploy hostage to a known-environmental red gate: verify the
+deployed service **directly** (its own ledger/health/real completion), state
+the gate was skipped or partially red and why (`--skip-e2e` prints the
+warning, so the skip is recorded), and report the residual gap plainly
+("29/30 e2e pass; 11-admin fails on Sonarr's raw IP from the dev box; peer
+probe returns 200; deploy verified by other means"). Do not weaken or skip
+the spec — it guards a real SSO-plus-RBAC contract.
+
+## Related
+
+- [[Container Host Reachability]] — the netns rules behind the vantage
+- [[Observability Provisioning]] — the monitoring stack reading the same fleet
+- [[Deploy State Triage]] — service-scoped deploys and the e2e gate
+- [[Testing Policy]] — the seam/e2e contract this page's checks live in
@@ -23,7 +23,7 @@ The homepage is live on Dankclaw. `./scripts/deploy.sh dankclaw` copies the stat
 page (`shared/homepage/docs.html`) — access walkthrough, the running-model
 guide, media how-tos, and troubleshooting. Per-service `/docs/<svc>/` pages
 remain a separate thread. The three LLM cards (`frank-large`, `frank-medium`,
-`frank-small`) moved from admin-only to the **user** page with live model
+`frank-secondary`) moved from admin-only to the **user** page with live model
 descriptions and a `docs` link to `/docs#models`.

 **Sections + WIP split (2026-09-21):** both pages now group their panels
@@ -36,6 +36,211 @@ deliberately unprobeable) via the `wip` value (`"open"` / `"closed"`).
 Tautulli (Plex statistics, `192.168.0.235:8181`) joined the admin page's
 Media Library Apps section.

+**Amber CRT theme + API docs (2026-10-07):** the palette moved from green
+phosphor to amber CRT-AMBIOS (`--fg #ffb000` on `--bg #0a0600`), and the font
+stack now leads with **VT323** loaded from the Google Fonts CDN (the deploy
+also ships `shared/homepage/img/` beside the page, so `/img/*` serves the docs
+screenshots — the post-auth listener serves files only through the `@data`
+path allowlist, so new top-level assets must be added there (the `@home`
+catch-all answers anything else with `index.html`, i.e. a PNG arrives as
+`200 text/html` and renders broken; pinned in `homepage_caddy.bats` — this
+exact bug bit the screenshots on 2026-10-08). The docs page gained a full
+(`/docs#api-access`): per-user OpenWebUI API keys (User menu → Settings →
+Account → API keys), the base URL `https://ai.franksdank.cloud/api/v1`, and
+copy-paste configs for **Opencode** (`~/.config/opencode/opencode.json`) and
+**Pi** (`~/.pi/agent/models.json`) pointing at `frank-secondary`. Shipped
+together with the OpenWebUI side: `features.api_keys=true` for regular users
+and the restored `frank-secondary` model row + public `read` access grant
+without which the picker showed "No results found" for every family account
+(see [[OpenWebUI]]).
+
+**Capability equivalence notes (2026-10-08):** each fleet card (and the
+`/docs#models` section) carries a `details.equiv` line — frank-large
+"equivalent to Claude Sonnet 5 / OpenAI-Luna (Summer 2026 frontier
+intelligence)", frank-primary "equivalent to Claude Opus 4.8 / OpenAI-Terra
+(same era)", frank-secondary "equivalent to last year's frontier level
+(2025)". Rendered by `descHTML()` as `.desc-equiv` (head color, italic);
+pinned in `homepage_config.bats`.
+
+**Protoss rename + Model Information component (2026-10-08):** the model
+cards became the StarCraft trio — `frank-large` → **frank-dragoon**
+(iGPU), `frank-primary` → **frank-archon** (Dankserver), `frank-secondary`
+→ **frank-zealot** (3090). The endpoint/API ids did NOT change in this wave; only the
+family-facing names. The three cards fused into one **Model Information**
+component (`app.js` `renderModelInfo()`, `.model-info`/`.mi-*` styles):
+Protoss unit art (`/img/unit-<unit>.jpg`, hex-clipped), per-unit accent
+colours (Dragoon gold, Archon cyan, Zealot blue), smaller type (15px vs
+19px body), and clickable unit-voice lines that play the original
+StarCraft `.ogg` quotations from `/sfx/` (fetched from
+starcraft.fandom.com via its MediaWiki API; the quote→file mapping lives
+in the `model.sfx`/`model.voice.sfx` fields). New `services.json`
+per-model fields: `unit`, `tagline`, `sfx[]`, `voice`, `supervision`
+(dragoon/archon autonomous-safe, zealot tight-supervision), and `warn`
+(uncensored legal note, rendered as a hover tooltip, not wall text).
+A compaction-model recommendation for frank-zealot rides in its `use`
+text. The Caddy `@data` allowlist grew `/sfx/*`; `deploy_homepage` copies
+`sfx/`. A `wip: "down"` badge state ("WIP: DOWN", red) was added; Seerr,
+Calibre-Web, RomM, and Jellyfin are WIP: DOWN with their Open links
+removed. Card notes added: Copyparty holds every other media app's
+content (including the WIP ones); Gitea and Plex need logins separate
+from the dankcloud SSO; SearXNG is obfuscated, not anonymous (upstream
+API keys tie to Frank). The user docs `/docs#models` section carries the
+same renames, api-id mapping, supervision and compaction notes.
+
+**Icons, card polish, admin identity gate (2026-10-08, later wave):** every
+app card now carries hex-clipped art beside its name (`services.json` `art`
+field, files under `img/apps/`): 19 brand marks tinted amber from the
+Simple Icons CDN plus 19 hand-improvised monogram SVGs (amber hex + VT323
+initials) for apps with no mark — Copyparty, Marreta, Hermes (+dashboard),
+node-exporter, Gluetun, OpenWebUI, Calibre-Web, RomM, Seerr, Loki, Alloy,
+Alertmanager, Valkey, Prowlarr, Readarr, Lidarr, Tautulli, ComfyUI. Model
+cards render in the order zealot → dragoon → archon (registry order is the
+contract; `homepage_config.bats` pins it). Dragoon and zealot share one
+common uncensored legal warning (adult content, harmful/hateful/illegal
+descriptions, "sensitive offensive cybersecurity topics", severe legal
+risk, "don't ruin this server for everybody"), rendered as the hover
+label; archon's ask-Frank rule is a bordered red `.mi-perm` callout.
+SearXNG's obfuscation-not-anonymity note moved out of its description into
+a card-level `warn` (`.svc-warn` hover label) and the same hover treatment
+in the docs. Gitea's note now documents self-service signup — no need to
+ask Frank. Footer: docs / admin / account settings are bordered glowing
+`.foot-btn` buttons; **the admin button is identity-gated** — Caddy's
+`/whoami` echoes the outpost-injected `X-Authentik-Username`/`X-Authentik-Groups`
+headers (the outpost strips client-supplied copies, so the echo is
+spoof-proof), and `app.js` shows the button only to `homepage-admins`
+members. The `/admin*` 403 stays the real gate; the e2e specs pin both
+sides (standard user: hidden + 403; admin: visible). Docs were swept for
+naming consistency: Protoss names primary everywhere, bracketed api ids
+at the model headings, and a note that a future switchover renames the
+ids themselves (OpenWebUI picker + raw API + models.yml all move together
+— not done yet).
+
+**Icons v2, clickable tiles, header nav (2026-10-08, third wave):** five
+icons replaced on request — Copyparty gets an improvised amber folder,
+Marreta a crumbling wall (bricks + red crack), Calibre-Web a book,
+OpenWebUI its official logo (`openwebui.com/logo.png`), Seerr the official
+`seerr.dev/os_logo_filled.svg`. Tiles with an open link are now clickable
+whole (`.panel.clickable`, hover lift + glow; clicks on inner links pass
+through). Any panel whose live state is DOWN gets a red gradient wash +
+slow pulsing glow (`.panel.down`, and `.mi-card:has(.badge.down)` for
+model cards). docs/admin/account-settings moved from the footer to the
+header (`nav.top-links`). The zealot compaction advice moved out of the
+Use paragraph into a bordered glowing `model.compaction` callout
+(`.mi-comp`), pinned by the component seam test.
+
+**Protoss theme, brand-color icons, portraits (2026-10-08, fourth wave):**
+the full-page angular treatment: chamfered 14 px-cut panels and model
+cards, hex-lattice + amber-glow backdrop (`body::after`), scanline
+overlay (`body::before`), diamond-glyph section titles with gradient
+underlines, the gradient MODEL INFORMATION gateway, a blinking cursor
+beside the title, amber scrollbars, cyan text selection, cyan hover glow
+on clickable tiles, and a red pulsing glow on down tiles/cards (the
+`data-state="down"` attribute `applyStatus` now sets). Icons recut to
+official brand colors (Simple Icons CDN): SearXNG, Gitea, Plex, Jellyfin,
+Sonarr, Radarr, Lidarr, Readarr, qBittorrent, Prowlarr, Seerr, OpenWebUI,
+Calibre-Web (amber glyph on amber tile). Handcrafted monograms carry
+per-app colors: Copyparty gold, Marreta brick, Grafana orange, Loki
+indigo, Neo4j network, Alloy violet, Promtail pink, Alertmanager red,
+Prometheus orange, Node Exporter teal, cAdvisor sky, Guake yellow,
+Authentik blue, Seer green radar, RomM a purple GameBoy. New per-service
+fields: `art_bg` (art backdrop — OpenWebUI's dark logo needs `#f5f5f5`),
+`warn` on OpenWebUI (searches ride the Seerr→SearXNG path),
+`model.compaction` (the amber "set this as compaction model" callout
+frank-zealot carries). frank-dragoon's card portrait is the SC2 unit
+portrait (fandom wiki `Dragoon_portrait.jpg`, webp→jpeg). The whole
+effect layer is stripped under `data-bare="1"` (the e2e bare page).
+Unit: `tests/homepage_config.bats` (referenced art files exist, model
+blocks well-formed, deployer ships the `img/` tree).
+
+**Docs reorder + model source links (2026-10-08, fifth wave):** the docs
+page section order is now getting-in → troubleshooting → media → chat →
+models → api-access (non-AI first, AI last). The *arr stack (Prowlarr,
+Sonarr, Radarr, Lidarr, Readarr, qBittorrent) is out of the family docs
+entirely — admin tooling; Seerr is presented as the front door for
+requests. Each fleet model gained a Hugging Face source link: card field
+`model.hf` rendered as the cyan `model on huggingface ↗` line under the
+role, and the docs `Model:` bullets link the checkpoint names (zealot →
+HauhauCS/Qwen3.5-9B-Uncensored-HauhauCS-Aggressive, dragoon →
+Quat3rnion/halogen-swift1.5-qwen3.8-flash-next-v2-abliterated, archon →
+Qwen/Qwen3.8-Flash-Next-FP8).
+
+**Navy Protoss console, maximal cornering, font swap (2026-10-08, sixth
+wave):** the amber CRT palette is retired. The page is now deep space
+navy (`#050d16`) with warp-blue/psychic-cyan accents (`--proto-blue
+#3fa9ff`, `--proto-cyan #7fe7ff`) — StarCraft Protoss console colours.
+Cornering went maximal: every panel, model card, badge, button, callout
+and the MODEL INFORMATION gateway are octagon-cut (all four corners
+chamfered, 18–24 px), and panels/model cards gained a second inset
+octagonal frame line (`::before`) — a double-bevel gateway look. Section
+titles are chamfered plates with the diamond glyph inside the plate.
+Fonts: body is **Share Tech Mono** (legible retro terminal), headings
+are **Rajdhani 600** (the StarCraft UI face) — VT323 is retired; all
+three pages load the pair from Google Fonts. Four tiles that had
+reverted to initials got pictograms: Calibre-Web an open book (orange),
+RomM a purple GameBoy, Copyparty a blue folder with upload arrow,
+Marreta a crumbling brick wall with a red crack (`img/apps/app-*.svg`).
+frank-dragoon's card accent is phase-crystal teal (`#5fe0d0`), matching
+the new palette. Bare mode strips the frames, octagon cuts and fonts.
+
+**Simplified console (2026-10-08, seventh wave):** "too busy — clean up
+and simplify/solidify." The sixth-wave effect stack was stripped to
+essentials. Removed: the scanline overlay (`body::before`), the hex
+lattice + twin radial backdrop (one quiet pylon glow above the horizon
+remains), the panel/model-card inner frame lines, all ambient text
+glows (body, title, model names, callouts), the blinking title cursor,
+the gradient section-title plates (headings are plain Rajdhani with a
+hollow diamond glyph + steel underline), the gradient `border-image`
+header rule (solid 2px warp-blue), the pulsing down-state animation
+(down tiles and cards now show a STATIC red border + tint), and the
+button/hover glows. Kept: octagon-cut panels (softer 14 px), unit-color
+model cards, the pictograms, and the fonts. Motion is reserved for
+state — the only animation left is the speaking-voice pulse. The result
+is flat navy panels on crisp 1px borders: same Protoss direction, half
+the pixels.
+
+**Model ids official (2026-10-08):** the card ids `frank-zealot`,
+`frank-dragoon`, `frank-archon` are now the harness's official model ids
+in `models.yml`/`config.yml` (see [[OMP Harness]]); the
+systemd services keep the `frank-primary`/`frank-large`/
+`frank-secondary` names. Homepage, docs page and harness now agree on
+one name per model.
+
+**Api ids renamed on the wire (2026-10-09):** the promise in
+`docs.html` is kept — the served model ids are now `frank-zealot`,
+`frank-dragoon`, `frank-archon`, so the docs page no longer carries
+`api id:` annotations, and the curl/opencode/Pi examples send
+`frank-zealot`. The systemd services keep the old names. `docs.html`
+names the fleet trio once each; `homepage_config.bats` pins the card
+ids, which did not change.
+
+**Console polish (2026-10-08, eighth wave):** five fixes — octagon
+clip-paths replaced with `border-radius` (the clipped corners read as
+gaps in the 1px border); section-title diamonds moved from absolute
+positioning to a fixed-width inline glyph (aligned, wrap-safe); closed
+WIP items grey out (dim type, desaturated art, muted border via
+`:has(.badge.wip-closed)`); the Dragoon tagline is one spliced track
+(`sfx/dragoon-tagline.ogg` — loudnorm-matched, silence-trimmed, 350 ms
+beat; the two SC2 source oggs are trashed); privacy notes became
+**`<details>` disclosure chips** ("⚠ privacy note ▸") pinned to the
+bottom of each box (`margin-top:auto` in a flex-column panel/card) —
+collapsed by default, tap-to-expand, the touch-safe replacement for the
+unreachable hover tooltips. The panel click-to-open handler ignores
+`.svc-warn` taps so the chip never navigates. Same chips on the docs
+page (privacy + uncensored-model warnings). The API-access section was
+reblessed against the live Open WebUI 0.6 UI: steps now match the real
+Show/copy/rotate flow and `img/openwebui-api-keys.png` is a fresh
+screenshot (old shot trashed). The troubleshooting DNS bullet dropped
+its Pi-hole specifics — family members off the LAN cannot use
+house-DNS advice; the bullet now just says the names resolve on the
+home network.
+
+**Archon card runs the fleet's own model (2026-10-08):** the
+`frank-archon` homepage card and docs Model bullet now say the payload
+is our in-house GPTQ-w4 quant of Swift1.5-Qwen3.8-Flash-Next and link
+our HF publication (`frank8465/Swift-1.5-Qwen3.8-Flash-Next-GPTQ-W4A8-Radiance-AMD-R9700`)
+in place of the FP8-era base-model link; capacity line says three
+submissions (`--max-num-seqs 3`).
+
 ## Design decision — 2026-08-22

 The live up/down check runs server-side. A checker on Dankclaw probes each backend and writes a same-origin `status.json`. The page fetches only that file and renders the badges. The browser never calls a backend directly. The checker runs as a systemd timer every 3 minutes.
@@ -111,7 +316,7 @@ flowchart TD
 ## Service classification

 **User services** (root page), in section order:
- **AI** — OpenWebUI, and the three LLM fleet cards (frank-large, frank-medium, frank-small — status + docs link, no Open link).
+- **AI** — OpenWebUI, and the three LLM fleet cards (frank-large, frank-primary, frank-secondary — status + docs link, no Open link).
 - **Web Apps** — SearXNG, Gitea, Copyparty, Marreta.
 - **Media Library Apps** — Calibre-Web, RomM, Plex, Jellyfin, Seerr.

@@ -131,7 +336,10 @@ shared/homepage/
 ├── style.css         # full theme; a toggle strips to bare HTML
 ├── app.js            # loads status.json once, renders the panels
 ├── services.json     # service config (user + admin)
-└── fonts/            # Terminus (woff2) for @font-face
+├── img/              # served at /img: docs screenshots, unit portraits
+│   ├── apps/         #   (unit-*.jpg) and app card icons (apps/*.svg)
+├── sfx/              # StarCraft unit voice .ogg files served at /sfx
+└── fonts/            # optional self-hosted fonts (deployed with the page; VT323 comes from the Google Fonts CDN, this dir is absent)
 ```

 `status.json` is generated at runtime in `/srv/homepage/`. It is not committed.
@@ -158,6 +158,119 @@ The Porkbun zone `franksdank.cloud` carries an A record for every vhost plus the
 | HTTPS works from the host but not the WAN | No port forward on the G36 for that service (manual router step) |
 | Cert renewal fails in Caddy logs | Porkbun API credentials in the Dankclaw `.env`; test with the Porkbun API `/ping` endpoint |

+## Per-client fix: resolver drop-in + browser DoH bypass
+
+Symptom: a LAN machine gets `connection timeout` (curl rc=28) on
+`https://*.franksdank.cloud` while the server is healthy. Root cause: the
+public DNS hands out the WAN IP `71.227.227.172`, which has **no 443
+port-forward** (filtered). Any resolver that is NOT Pi-hole produces the WAN
+answer → SYN to a filtered port → timeout. There are TWO resolver layers to
+check, in order:
+
+**Layer 1 — systemd-resolved (fixes curl/shell).** A route-only drop-in
+sends `*.franksdank.cloud` ONLY to Pi-hole and never to the default-route
+servers:
+
+```bash
+dig +short www.franksdank.cloud @192.168.0.143   # expect 192.168.0.143
+dig +short www.franksdank.cloud @1.1.1.1         # expect 71.227.227.172 (the trap)
+getent hosts www.franksdank.cloud                # what the OS actually resolves
+curl -sk --resolve www.franksdank.cloud:443:192.168.0.143 \
+     https://www.franksdank.cloud/ -o /dev/null -w '%{http_code}\n'   # 200 = server fine
+
+sudo mkdir -p /etc/systemd/resolved.conf.d
+sudo tee /etc/systemd/resolved.conf.d/franksdank-lan.conf >/dev/null <<'EOF'
+[Resolve]
+DNS=192.168.0.143
+Domains=~franksdank.cloud ~franksdank.casa
+EOF
+sudo systemctl restart systemd-resolved && resolvectl flush-caches
+```
+
+The `~` (route-only) prefix is load-bearing: it routes only those domains to
+Pi-hole. Verify: `getent hosts www.franksdank.cloud` → `192.168.0.143`;
+`resolvectl status` shows `DNS Domain: ~franksdank.cloud ~franksdank.casa`.
+
+**Layer 2 — browser DoH (the bypass: "curl works, browser times out").** If
+Layer 1 is verified but the BROWSER still times out, the browser is doing
+its own DNS over HTTPS and never asks systemd-resolved — it queries the TRR
+provider (Cloudflare et al.) directly → WAN answer → timeout. Prove it:
+`curl -sk -m 6 https://71.227.227.172/` → rc=28 (the WAN answer is a dead
+end). Diagnosis details:
+
+- Firefox profile may be at the XDG path `~/.config/mozilla/firefox/`, NOT
+  `~/.mozilla/firefox/` — find it via the running process if the default
+  path is empty.
+- `grep network.trr. <profile>/prefs.js` — **no entries = shipped default,
+  and since Firefox 113 the shipped default is DoH ON.**
+
+Fix (surgical, keeps DoH for everything else): Firefox → Settings → Privacy
+& Security → **DNS over HTTPS** → "enabled for these domains" → **Add…** →
+`franksdank.cloud` (and `franksdank.casa`). The exception list = domains
+that do NOT use DoH; they fall back to the system resolver, which has the
+route-only drop-in. Or set DoH to **Off** entirely (coarser). Chrome:
+Settings → Privacy and security → Security → "Use secure DNS" → off, or
+"Use the same settings as your system". Do NOT hand-edit `prefs.js` while
+the browser is running — it holds prefs in memory and rewrites the file on
+exit, clobbering the edit.
+
+This is a per-client fix: every LAN client that must reach the homelab by
+name needs Pi-hole as its DNS (or the drop-in) AND a browser that does not
+bypass it with DoH. The server side needs no change. Verified on blankpad
+2026-09-21: the drop-in was present since 09-12, curl returned 200, and
+Firefox 155 with default (ON) DoH was the remaining hole.
+
+## Porkbun zone ops (franksdank.cloud)
+
+Operate the public DNS zone via the Porkbun API v3, run **on Dankclaw** so
+the API key/secret sourced from `/home/dank/serverAppConfigs/.env`
+(`PORKBUN_API_KEY`, `PORKBUN_API_SECRET_KEY`) never leaves the host. Pipe
+scripts to the target with `ssh -o BatchMode=yes dankclaw 'bash -s' <
+/tmp/script.sh`.
+
+Endpoints (all POST, JSON body):
+
+- List: `https://api.porkbun.com/api/json/v3/dns/retrieve/franksdank.cloud`
+  — auth via headers `X-API-Key` + `X-Secret-API-Key` (Bearer is wrong
+  here). Response shape is **top-level `.records[]`**, NOT
+  `.content.records[]` as some docs claim — parse with
+  `jq -r '.records[] | [.id,.name,.type,.content,(.ttl|tostring)] | @tsv'`.
+- Delete: `/dns/delete/{domain}/{id}` (body carries apikey+secretapikey).
+- Edit: `/dns/edit/{domain}/{id}` — pass the full record (`id`, `name`,
+  `type`, `content`, `ttl`, `prio`).
+
+Quirks that bite (verified live 2026-08-31):
+
+1. **`name` is a RELATIVE label on edit.** Passing `ai.franksdank.cloud`
+   yields the record `ai.franksdank.cloud.franksdank.cloud`. Pass the bare
+   label (`ai`) and the API appends the zone itself. Verify after every
+   rename with a retrieve before declaring done.
+2. **Rate limit answers as an HTML 404-style page, not JSON.** A batch of
+   rapid mutations makes some calls return `<!DOCTYPE html>...` where jq
+   then errors (`Cannot iterate over null` / `Invalid numeric literal`). Do
+   not assume the call succeeded — sleep ~8 s and retry the failed ids;
+   verify by re-listing the zone at the end.
+3. Deletes/renames are per-record-id: always retrieve first, map name→id,
+   then mutate by id.
+
+Rename procedure (vhost rename = DNS + Caddy together):
+
+1. `retrieve` → collect ids for the renamed names.
+2. `edit` each id with the new **relative** label, preserving
+   type/content/ttl.
+3. Re-retrieve and confirm `<new>.franksdank.cloud A <ip>`.
+4. In-repo lockstep before deploy: rename the Caddyfile vhost block AND the
+   Pi-hole `dankclaw/pihole/addn-hosts` line in the same commit — a test
+   fails the suite if any Caddyfile vhost lacks an addn-hosts A record on
+   192.168.0.143.
+5. Deploy via `scripts/deploy.sh dankclaw` (plans pihole+caddy from changed
+   paths). Capture the suite's true exit with
+   `out=$(./tests/run.sh 2>&1); rc=$?` — piping to `tail` masks the exit
+   code behind `tail`'s own status and has shipped untested commits here.
+6. Verify live: `dig +short <name> @192.168.0.143` then
+   `curl -o /dev/null -w %{http_code} https://<new-name>` (expect 200/302;
+   first hit triggers dns-01 issuance, which works without inbound port 80).
+
 ## See Also

 - [[Pi-hole]] — the LAN DNS server (quadlet, records, web UI)
@@ -0,0 +1,95 @@
+---
+type: reference
+title: Observability Provisioning
+description: How Prometheus config/rules and Grafana datasources/dashboards are provisioned as code in serverAppConfigs — the config triad, deploy copiers, planner mappings, seam tests, entry points for each change type, and API verification.
+tags: [observability, prometheus, grafana, provisioning, deployment]
+timestamp: 2026-10-05T00:00:00-07:00
+---
+
+# Observability Provisioning
+
+The monitoring stack is **provisioned as code**: dashboards, datasources,
+scrape jobs and alert rules live in the repo and ride the normal deploy flow.
+No hand-edits inside containers — they are wiped on recreate and never
+committed.
+
+## The config triad
+
+| Repo path | What it holds |
+|---|---|
+| `shared/prometheus/prometheus.yml` | scrape jobs (global + per-host) |
+| `shared/prometheus/rules/llm-fleet.yml` | alert rules + recording rules |
+| `shared/grafana/provisioning/` | datasources + dashboards (provisioned tree) |
+
+Targets and datasources reference **container names** (`loki:3100`,
+`alloy:12345`), never loopback-published ports; the LLM endpoints are reached
+by LAN IP (`192.168.0.143:7770`, `:7771`, `192.168.0.235:7770`) because they
+are host-netns services, not containers. Prometheus 3 refuses HTML responses
+as metrics — HTML-scraped "probes" are structurally inert and need a blackbox
+exporter (see [[Stale Signal Audit]]).
+
+## The deploy path
+
+`scripts/deploy-remote.sh` carries two copier functions:
+
+- `copy_prometheus_config` → `/opt/prometheus/config/` on the target
+- `copy_grafana_provisioning` → `/opt/grafana/provisioning/`
+
+`scripts/deploy-plan.py` maps changes under `shared/prometheus/**` and
+`shared/grafana/**` to the `prometheus`/`grafana` services, so
+`deploy.sh dankclaw --service grafana` (etc.) restarts only what the change
+touched. Seam tests in `tests/observability_quadlet.bats` pin: unit files
+exist, the copier functions exist, and the provisioned paths match the units'
+mounts.
+
+## Entry points
+
+**Add a scrape target** — edit `shared/prometheus/prometheus.yml`. Decide the
+vantage first: container name if the exporter is a container on the same
+netns; host LAN IP if it is a host process (LLM endpoints); a blackbox module
+if it is an HTTP probe, not a metrics endpoint.
+
+**Add an alert/recording rule** — edit `shared/prometheus/rules/llm-fleet.yml`.
+Current delivered set: 5 alert rules + 1 recording rule for the LLM fleet.
+Rules reload with the prometheus service.
+
+**Add a datasource** — drop a YAML file into
+`shared/grafana/provisioning/datasources/`. Point it at the container name of
+the backend (`http://prometheus:9090`), not loopback.
+
+**Add a dashboard** — drop the JSON into
+`shared/grafana/provisioning/dashboards/`. Provisioned dashboards live in
+unified storage, not (only) the SQLite table — grep the API, not the DB, when
+verifying (see the verify section of [[Fleet Audit and Posture]]).
+
+## Verify through the APIs, not the filesystem
+
+```bash
+# prometheus: rules and targets actually loaded
+curl -s http://127.0.0.1:9090/api/v1/rules | python3 -m json.tool | head
+curl -s "http://127.0.0.1:9090/api/v1/targets?state=active" | \
+  python3 -c "import json,sys; [print(t['labels'].get('job'), t['health'], (t['lastError'] or '')[:60]) for t in json.load(sys.stdin)['data']['activeTargets']]"
+# grafana: datasource + dashboard provisioned
+curl -s -u admin:<pw> http://127.0.0.1:3002/api/datasources | python3 -m json.tool | grep -E '"name"|"url"'
+```
+
+Health answers from a stale process and provisioned resources can hide in
+unified storage — a file existing on disk is not proof the running server
+loaded it. See [[Stale Signal Audit]] for the never-green-target decision
+table (refused = outage, 404/HTML = config defect).
+
+## Current delivered state (2026-09-17)
+
+Prometheus → Alertmanager (`dankclaw/observability/quadlet/alertmanager.container`,
+loopback `:9093`) → host-side Signal relay (`scripts/signal-relay.py`, user
+unit, `192.168.0.143:18093` — the LAN IP, the one bind a container can reach
+via `host.containers.internal`) → signal-cli. Some targets read down for
+structural reasons (HTML-scraped "probes", Gitea metrics off); the remedy is
+the blackbox exporter. See [[Observability]].
+
+## Related
+
+- [[Observability]] — stack topology and the alert delivery chain
+- [[Stale Signal Audit]] — classifying down targets before "fixing" them
+- [[Quadlet Lifecycle]] — the units that host these services
+- [[Deploy State Triage]] — how the planner selects these services
@@ -38,7 +38,7 @@ Authentik; post-auth the Caddy vhost routes straight to `prometheus:9090`) —
 | `node-dankserver` | `192.168.0.235:9100` | host metrics from [[Node-Exporter]] — a different machine, so the LAN address works |
 | `grafana`, `loki` | `grafana:3000`, `loki:3100` | container names on the `dankclaw_default` network; both serve real `/metrics` |
 | `frank-medium` | `192.168.0.235:7770` | the LLM fleet — `vllm:` series; a different machine, so the LAN address works |
-| `frank-large`, `frank-small` | `host.containers.internal:7770`, `:7771` | the LLM fleet — `llamacpp:` series; same-host net=host binds, so via the gateway |
+| `frank-large`, `frank-secondary` | `host.containers.internal:7770`, `:7771` | the LLM fleet — `llamacpp:` series; same-host net=host binds, so via the gateway |

 **Three jobs read down, and they are not outages** (audited 2026-09-16):

@@ -95,6 +95,51 @@ llama-server --hf-repo <target-GGUF> \

 The DFlash drafter proposes, then ngram-mod runs as an additional implementation in the chain. Verified against upstream `common/speculative.cpp` (b10133+).

+### Chaining laws (measured on this fleet, 2026-09 → 10)
+
+`--spec-type` takes a comma list, and the gate is `ctx_dft` in
+`common/speculative.cpp` (`common_speculative_init_result`): **draftless**
+types (the ngram family — simple, map-k, map-k4v, mod, cache) are added
+unconditionally, while **context-backed** types (`draft-mtp`, `draft-dflash`,
+`draft-dspark`, `draft-eagle3`) all require a draft CONTEXT. Consequences:
+
+- A chain may hold ONE context-backed drafter plus any number of ngram types.
+  `draft-dflash,ngram-mod` is legal and measured: 2.6x the drafter alone on
+  prompt-echo traffic, byte-identical to it on prose at the fleet's 48/64
+  window.
+- Two context-backed types cannot share one process: `draft-mtp,draft-dflash`
+  dies at load with `context type MTP requested but model doesn't contain MTP
+  layers` — MTP builds its context against whatever the draft context holds,
+  so it met the DRAFTER's file, not a missing head. Pin that in a seam so
+  nobody "fixes the flag" later.
+- Internal order is fixed (ngram types first, then drafters); CLI order is
+  irrelevant.
+- **A legal chain can still lose on a recurrent model.** `draft-mtp,ngram-mod`
+  on the qwen35 hybrid (frank-large era) measured −15 % prose / −7 %
+  block-copy (2026-10-03): adding the ngram layer logs `recurrent state
+  rollback is not compatible with 'ngram-mod' - disabling rollback support`,
+  and MTP verification depends on that rollback, so acceptance drops on ALL
+  traffic even where the matcher fires. Measure chains per architecture,
+  never by analogy.
+- ngram-mod prints NO startup banner — the load log shows only the drafter's
+  `adding speculative implementation` line. Never conclude "chain not
+  engaged" from the load log; identify behaviourally (`draft_n` > 0 only
+  where text repeats) or by the per-task `draft acceptance = ... mean len`
+  line. It costs ~64 MiB of host-RAM hash table.
+
+### Drafter KV: read the window policy from the header
+
+The draft context is built with `cparams.n_ctx = llama_n_ctx(ctx_tgt)`
+(`common/speculative.cpp`) — there is no flag to cap it — so a drafter that
+declares NO sliding window caches the whole target window on whatever device
+`--spec-draft-device` names. DFlash declares `sliding_window = 2048` (~40 MiB
+of draft KV); DSpark declares none: 5 layers x 2 x 8 kv heads x 128 dim x 2 B
+= 20 KiB per position, i.e. **5.0 GiB at 262144** — enough to kill the first
+draft graph capture with `ROCm error: out of memory` AFTER the model loaded
+and `/health` answered (measured 2026-09-16). Read `sliding_window` out of
+the drafter's GGUF header before sizing any spec-decode deployment, and
+prefer a skewed `-ts` over a context cut when the full window matters.
+
 DFlash on llama.cpp:

 ```
@@ -170,9 +215,9 @@ The DFlash draft is not a standalone model. It must pair with its target model.

 ## Deployment on Dankserver (NVIDIA RTX 3090)

-The Dankserver llama.cpp deployment (`frank-small`, Qwen3.6-35B-A3B) is **active** on Dankserver (port 7770) as of the 2026-08-17 redo (the earlier 2026-08-16/17 retirement of this service was reversed when the PP split was retired). The vLLM material below documents the archived AWQ/DFlash path on the same hardware.
+The Dankserver llama.cpp deployment (`frank-secondary`, Qwen3.6-35B-A3B) is **active** on Dankserver (port 7770) as of the 2026-08-17 redo (the earlier 2026-08-16/17 retirement of this service was reversed when the PP split was retired). The vLLM material below documents the archived AWQ/DFlash path on the same hardware.

-> **DFlash retired from active service.** `DFlash` (draft-dflash) was tested initially but did not deliver the desired speed gains under the `frank-small` workload (see ngram-mod benchmarks below). The `Alittlehammmer/Qwen3.6-35B-A3B-DFlash` draft GGUF remains in `/home/dank/models/` as reference material only.
+> **DFlash retired from active service.** `DFlash` (draft-dflash) was tested initially but did not deliver the desired speed gains under the `frank-secondary` workload (see ngram-mod benchmarks below). The `Alittlehammmer/Qwen3.6-35B-A3B-DFlash` draft GGUF remains in `/home/dank/models/` as reference material only.

 The vLLM material below documents the archived AWQ/DFlash path on the same hardware. The RTX 3090 is Ampere (compute capability 8.6). Use these conditions:

@@ -209,6 +254,151 @@ docker run --rm --name vllm-qwen36-dflash \

 History: before `--spec-default`, the box ran DSpark (`draft-dspark`, retired 2026-08-15 after a ROCm illegal-memory-access crash under two concurrent 16k prompts) and before that bundled MTP (`draft-mtp`, retired same day). DFlash was retired on 2026-08-14 (premature-EOS under parallel load). The old DFlash draft GGUF (`Qwen3.6-27B-DFlash-Q8_0.gguf`), the MTP-only drafter (`Qwen3.6-27B-MTP-ONLY-Q8_0.gguf`), and the DSpark draft (`Qwen3.8-27B-DSpark-Q8_0.gguf`) remain in `/home/dank/models/` as reference only.

+## A/B protocol on a live llama.cpp endpoint
+
+Verdicts are **traffic-shaped**: a proposer can only draft text it has seen,
+so a single prompt shape answers the wrong question. Use fixed shapes:
+
+| Shape | Prompt | Who it can favour |
+|---|---|---|
+| prose | 1k filler + "reply with one sentence" | model-backed drafters only |
+| echo | ~1.5k passage of UNIQUE lines + "repeat verbatim" | ngram (output IS the prompt) |
+| structured | repetitive output, no prompt overlap | ngram, if spans really repeat |
+| identity | short deterministic question | controls for lossless verification |
+
+Design the echo passage so no long span repeats INSIDE it — otherwise you
+measure the matcher against itself. For `structured`, give each line a unique
+id: repeated lines make it a negative control, not a repetition test.
+
+Protocol:
+
+1. Stop the unit (`sudo systemctl stop <svc>`), `pkill -x llama-server`; wait
+   for the port (a retired predecessor's published-port proxy leaves the
+   endpoint port in TIME_WAIT — the bind fails with address-in-use long after
+   the unit stopped).
+2. Per config: write an env file with the overrides, start the serve script
+   with `FRANK_*_ENV_FILE`, poll `/health`, measure. Ready-made harness:
+   `scripts/bench-spec-ab.py <env-file> <serve-script>`.
+3. **Restore the unit in a `finally`** and shout if it does not come back
+   healthy. A crashed harness that leaves production down is worse than no
+   harness.
+4. **Prove losslessness**: one fixed prompt at temp 0 to every config, diff
+   the greedy text. Speculation only buys speed; differing texts are a
+   correctness bug. Empty answers (`content: null` when reasoning ate the
+   budget) make the comparison vacuous — bench clients must fall back to
+   `reasoning_content`, and echo/structured prompts should disable thinking
+   per request: `chat_template_kwargs: {"enable_thinking": false}`.
+5. **Sweep the depth, not just the mode** — draft depth is model-specific and
+   single-valued; deeper often loses (measured: at n=7 a third of proposals
+   survived, at n=3 more than half).
+6. **Sweep the ngram window in the same run as the configs**: too short a
+   `--spec-ngram-mod-n-min/-n-max` proposes on ordinary prose fragments —
+   rejected, verification still paid (measured 43.9 → 36.0 tok/s).
+
+Read the right counters — per request `timings.draft_n` / `draft_n_accepted` /
+`predicted_per_second`; cumulative `/metrics`
+`llamacpp:spec_decode_num_drafts_total` / `..._accepted_tokens_total`;
+per-task `draft acceptance = ... (N accepted / M generated)`. `draft_n == 0`
+while tokens flow = the proposer is NOT engaged, whatever the flags say. A
+zero-draft config is a useful negative control: it proves the other config's
+drafts came from the proposer under test.
+
+Harness hygiene: absolute script paths (a relative path plus `cd` to its own
+dir double-counts the prefix); `pkill -x`, never `-f` (the pattern matches
+your own command line); nested f-strings with backslash escapes are a
+SyntaxError on the fleet's Python (<3.12 semantics) — build log lines with
+concatenation. Report `decode tok/s` per shape plus acceptance and mean
+accepted length; a config that wins on echo and loses on prose is a traffic
+decision, not a winner. Wire the winner into the deployed env file with the
+measured table as its justification comment; record losing numbers with "do
+not switch without re-measuring".
+
+The A/B vehicle for ANY single-GPU llama.cpp knob (batch/ubatch, KV type,
+slot split, quant rung — not just spec flags) is the transient systemd
+drop-in procedure in [[frank-large-benchmark]] and [[frank-secondary-benchmark]].
+
+## Is it worth it? (arithmetic before deployment)
+
+Answer against the INCUMBENT, not against zero.
+
+**Step 0 — check whether the target already speculates.** Many engines ship a
+drafter of their own; then your baseline is `A_incumbent / t_s`, not `1 / t_s`.
+halogen-flash-server: MTP head ON by default — read its own accounting, e.g.
+`serve_api: mtp 63 tok in 1.31s = 47.97 t/s | 33 rounds, commit 1.88/round`.
+llama.cpp: `/metrics` counters and per-request `draft acceptance` lines.
+
+**Step 1 — measure the two step times, never guess.** Target step `t_s` = 1 /
+(client-visible tok/s ÷ commit-per-round); ask for the rate the operator
+actually SEES (engine-internal decode-only numbers run ~1.7–2x optimistic).
+Drafter step `d_s` from llama.cpp's per-task `eval time` lines when it is a
+plain model on the same box.
+
+**Step 2 — arithmetic:**
+
+```
+round_time = K * d_s + 1.1 * t_s        # K drafts proposed, K+1 verified in ONE target pass
+A          = (1 - p^(K+1)) / (1 - p)    # expected accepted tokens, p = per-position acceptance
+rate       = A / round_time             # compare to the incumbent
+break-even : A > rate_incumbent * (K*d_s + 1.1*t_s)
+```
+
+Worked example that decided a real case: `t_s = 67 ms` (25 tok/s, 1.67
+commit/round), `d_s = 12 ms` (27B IQ3_S drafter on the 3090) → break-even
+`p ≈ 0.70` at K=3, `≈0.68` at K=5; a realistic cross-finetune `p = 0.4–0.6`
+gives 17–20 tok/s, i.e. a LOSS. A drafter only ~3x faster per token cannot
+pay for K sequential forwards.
+
+**Step 3 — classify the drafter by per-round cost:**
+
+| drafter | cost per round | needs |
+|---|---|---|
+| plain model (`-md`/`draft-simple`) | **K sequential forwards** | 10–50x faster than the target |
+| n-gram / prompt-lookup (`ngram-*`) | ~0, no model | repeated context (agent edit loops) |
+| trained head (`draft-mtp`, EAGLE/DFlash) | **~1 pass** | the head must exist FOR THAT CHECKPOINT |
+
+Trained heads are target-specific: EAGLE/DFlash consume the TARGET's hidden
+states, so a dense-27B DFlash2 is useless for a 512-expert MoE trunk.
+
+**Step 4 — acceptance reality checks.** Fleet datum: a trained z-lab DFlash2
+at depth 7 accepted ~41 % on its matched target (`draft-dflash 7 → 43.5
+tok/s`); treat 40–60 % as the honest band for a matched pair. Cheapest proxy
+for `p` at temp 0 when both models are greedy and share a tokenizer: greedy
+continuation agreement on identical prompts, two API calls, no weights moved.
+llama.cpp can also synthesize the projection without any draft artifact:
+`--spec-synth-rates P0,P1,...` / `--spec-synth-len L`. Speculation pays IN
+PROPORTION TO HOW SLOW THE TARGET IS — the same 41 % gave 2.3x on an
+18.8 tok/s layer-split target and a loss on a 25 tok/s one; fast targets are
+usually sparse MoE, the worst case for drafting.
+
+**Step 5 — check the attachment surface BEFORE the arithmetic.** Closed
+engines refuse external drafters by design: halogen-flash-server's
+`DRAFTER_CAP` gates `dflash2` on the loaded checkpoint and an unavailable
+name is a deliberate 400 (a silent downgrade would make comparisons lie);
+there is no device routing in its loopback token protocol, so a drafter can
+never sit on another card. Pure llama.cpp DOES route the draft:
+`--spec-draft-device/-devd` (default follows `--device`), `-ngld`,
+`-otd`, `-ctkd/-ctvd`, `--spec-draft-n-max` (default 3), `--spec-draft-p-min`.
+One process sees only the devices its backends were built with — CUDA+Vulkan
+is the combination that shows a CUDA card AND an AMD iGPU (HIP+CUDA cannot
+coexist); a CUDA-only build has no `Vulkan0` to draft with.
+
+**Step 6 — cross-model pairs need a byte-identical tokenizer.** Speculation
+compares token IDs, so draft and target must map id → same string. Verify
+cheaply from the GGUF headers (first ~8 MB over HTTP, no full download):
+parse `tokenizer.ggml.tokens`, print vocab size, merges, `pre`, and sha256
+over the first 2000 tokens —
+
+```
+vocab=248320 merges=247587 model=gpt2 pre=qwen35
+sha256(first 2000 tokens)=8619c68fe4466d3a   # identical => ids are interchangeable
+```
+
+Identity is necessary, NOT sufficient — the drafter still has to be cheap and
+accepted. Report in this order: does the target already draft; break-even
+`p` at K=1,3,7 versus the incumbent; the realistic band and resulting rate;
+whether the drafter fits (VRAM) and the attachment surface exists; the one
+measurement that would settle it.
+
 ## Notes

 - Speculative decoding helps low-QPS, memory-bound, latency-focused workloads. It helps less at high QPS.
@@ -225,4 +415,8 @@ History: before `--spec-default`, the box ran DSpark (`draft-dspark`, retired 20
 - [[Dankclaw System Specs]]
 - [[Dankserver System Specs]]
 - [[VLLM Overview]]
- [[deployment]]
+- [[deployment]]
+- [[GGUF Compatibility]] — header parsing and custom tensor-type lock-in
+- [[Llama Cpp Serving Sizing]] — slot-pool and drafter KV sizing arithmetic
+- [[Llama Cpp Multi-Device Splits]] — where the drafter's KV actually lands
+- [[frank-large-benchmark]] and [[frank-secondary-benchmark]] — the single-GPU env drop-in A/B vehicle
@@ -46,7 +46,7 @@ Dankclaw is an AMD APU server for CIRU ROCmFPX llama.cpp inference. The host run
 | Runtime | FastFlowLM 0.9.45 + XRT (installed 2026-08-16; **retired 2026-09-11** with the `amd_iommu=off` decision) |
 | Status | Not in service. Research only. See [[FastFlowLM NPU Inference]] |

-The NPU is retired (2026-09-11). The `frank-small` name now serves Gemma4-12B QAT on the RTX 3090 via llama.cpp (see [[frank-small-gemma4-12b]]).
+The NPU is retired (2026-09-11). The `frank-secondary` name now serves Gemma4-12B QAT on the RTX 3090 via llama.cpp (see [[frank-secondary-gemma4-12b]]).

 ### ROCm

@@ -17,7 +17,7 @@ timestamp: 2026-09-20

 Dankserver is a media/library server and the `frank-medium` LLM host. The
 host runs Ubuntu 26.04 (Resolute Raccoon). The RTX 3090 that served the
-FreeToken/CUDA `frank-small` era was removed on 2026-09-15 and replaced with
+FreeToken/CUDA `frank-secondary` era was removed on 2026-09-15 and replaced with
 two AMD Radeon AI PRO R9700 cards.

 ## Hardware Specifications
@@ -107,8 +107,8 @@ hijack `libhsa-runtime64` were purged on 2026-09-15 — see
 The active inference service is `frank-medium.service`: mainline llama-server on
 port 7770, layer-split 32:33 across both R9700s with the DFlash2 drafter pinned to
 ROCm1. See [[frank-medium-llamacpp-dflash2]]. The former
-`frank-small.service` (FreeToken/CUDA) was retired with the 2026-09-15 GPU swap;
-its unit is archived in `~/.AGENTTRASH/20260915-frank-small-retire/host-units/`,
+`frank-secondary.service` (FreeToken/CUDA) was retired with the 2026-09-15 GPU swap;
+its unit is archived in `~/.AGENTTRASH/20260915-frank-secondary-retire/host-units/`,
 and the vllm-radiance replica trio in `~/.AGENTTRASH/20260916-vllm-radiance-retire/`.
 See [[deployment]].

@@ -13,12 +13,12 @@ timestamp: 2026-09-03

 > **SUPERSEDED 2026-08-17.** The Frank redo retired this topology.
 > See [[frank-redo-2026-08-17]] and [[deployment]] for the active configuration.
-> The `frank-small` name and port 52625 then belonged to the vLLM
+> The `frank-secondary` name and port 52625 then belonged to the vLLM
 > service on Dankclaw (nameistoken Quark W8A8); this page is archive.

 # FastFlowLM NPU Inference

-FastFlowLM (FLM) runs LLMs entirely on the AMD Ryzen AI NPU. It targets XDNA2 chips: Strix, Strix Halo, Kraken, and Gorgon Point. The project describes itself as "Ollama, but deeply optimized for NPUs." As of 2026-08-16, FLM is installed on Dankclaw, the NPU validates, and `frank-small` (port 52625) serves in production. See Promotion.
+FastFlowLM (FLM) runs LLMs entirely on the AMD Ryzen AI NPU. It targets XDNA2 chips: Strix, Strix Halo, Kraken, and Gorgon Point. The project describes itself as "Ollama, but deeply optimized for NPUs." As of 2026-08-16, FLM is installed on Dankclaw, the NPU validates, and `frank-secondary` (port 52625) serves in production. See Promotion.

 ## Verdict (2026-08-16)

@@ -203,16 +203,16 @@ Notes:

 ## Promotion (2026-08-16)

-`frank-small` now runs the NPU endpoint on Dankclaw (was the Dankserver llama.cpp service). Unit: `scripts/frank-small.service` (symlinked into `/etc/systemd/system`), wrapper `scripts/serve-flm-small.sh`, tunables `scripts/frank-small.env`. `LimitMEMLOCK=infinity` is set in the unit. Default model: `gemma4-it:e4b`. Swap `FLM_MODEL` in the env to switch.
+`frank-secondary` now runs the NPU endpoint on Dankclaw (was the Dankserver llama.cpp service). Unit: `scripts/frank-secondary.service` (symlinked into `/etc/systemd/system`), wrapper `scripts/serve-flm-small.sh`, tunables `scripts/frank-secondary.env`. `LimitMEMLOCK=infinity` is set in the unit. Default model: `gemma4-it:e4b`. Swap `FLM_MODEL` in the env to switch.

 ### Boot policy (2026-08-17)

-The `frank-small` unit is disabled for boot. Only the PP pair (`frank-large` + `frank-pp-worker`) starts at boot. The unit file was removed from `/etc/systemd/system`; the running FLM service keeps serving until stopped or rebooted. To restore the endpoint, run these commands on Dankclaw:
+The `frank-secondary` unit is disabled for boot. Only the PP pair (`frank-large` + `frank-pp-worker`) starts at boot. The unit file was removed from `/etc/systemd/system`; the running FLM service keeps serving until stopped or rebooted. To restore the endpoint, run these commands on Dankclaw:

 ```
-sudo ln -sf /home/dank/vllmScripts/scripts/frank-small.service /etc/systemd/system/frank-small.service
+sudo ln -sf /home/dank/vllmScripts/scripts/frank-secondary.service /etc/systemd/system/frank-secondary.service
 sudo systemctl daemon-reload
-sudo systemctl enable frank-small
+sudo systemctl enable frank-secondary
 ```

 Live config (2026-08-16): `FLM_PMODE=turbo`, `FLM_ASR=1` (loads Whisper-V3-Turbo-NPU2; exposes `/v1/audio/transcriptions`), `FLM_CTX_LEN=131072` (verified max — Gemma4 natively supports 128K and the NPU budget fits it; KV occupancy measured at 0.11% for a 123-token exchange, decode 12.0 t/s, 13 GiB headroom).
@@ -271,12 +271,89 @@ graph TD
 6. Step up in size: 9B, then gpt-oss-20b.
 7. Attempt `qwen3.6-moe:35b-a3b` only after the smaller runs succeed, and only with the production service stopped. Watch `free -h` during the model download and load.

+## Registering packs absent from the catalog (NPU2)
+
+Use when adding a non-catalog NPU2 pack (an extraction or fine-tuned model)
+to the FLM registry. Registry facts: user registry
+`/home/dank/.config/flm/model_list.json`, shape
+`{"model_path": "models", "models": {"<tag>": {"<size>": {name, url,
+file_url, size, flm_min_version, files[], vlm, default_context_length,
+max_prefill_len, details{format:"NPU2", family, …}, label[]}}}}`; serve
+resolves tags through `FLM_CONFIG_PATH` at startup, and `flm list` with the
+env var set lists custom tags (`⏬` = not downloaded).
+
+1. Back up the registry JSON first (`cp -a`).
+2. Insert the entry via a python json edit + atomic `os.replace`. Set
+   `files` to the FULL HF file list — **including `vision_weight.q4nx` even
+   for text-only packs**: the qwen3.5-family loader opens it unconditionally
+   and a missing file makes the first request HANG silently (no error log).
+3. **Do not use `flm pull <tag>` for custom entries** — it throws
+   `json.exception out_of_range key '<tag>:<size>' not found` while building
+   the download list. Place files manually into
+   `/home/dank/.config/flm/models/<PackName>/` with
+   `curl -sfL "https://huggingface.co/<repo>/resolve/main/<file>?download=true"`
+   — plain `raw/main/<file>` returns 135-byte LFS pointer stubs, not
+   weights; verify byte counts against the HF tree API before trusting a
+   download.
+4. Kernel packs: copy from the same-shape family pack (shape-match
+   `config.json` on hidden_size/num_hidden_layers/head_dim first) into
+   `/opt/fastflowlm/share/flm/xclbins/<PackName>/` via `sudo cp -a`
+   (root-owned dir).
+5. Deploy/restart the serve unit (the deploy path, not ad-hoc restarts) so
+   the new process loads with all files present.
+6. Verify with a REAL completion request (`model: "<tag>:<size>"`) —
+   `/v1/models` alone proves registration, not loadability.
+
+Hard constraints: only ONE `flm serve` process can hold the NPU hardware
+context — a second concurrent serve process fails every load with
+`DRM_IOCTL_AMDXDNA_CREATE_HWCTX err=-22` (aie2 mgmt msg opcode 0x2 status
+0x2000003); never run scratch/verify instances while production holds the
+NPU. Tag switching happens INSIDE one serve process; the first request
+after a model switch pays ~6–11 s cold-load cost (visible in
+`usage.load_duration`). Over ssh, never `pkill -f "<pattern>"` with a
+pattern present in your own command line — use bracket-guarded patterns
+(`[q]wen3.5:4b`) or exact PIDs.
+
+## Tag load-failure diagnosis (200 with an error body)
+
+`POST /v1/chat/completions` answers **HTTP 200** with a flat
+`{"error":"Failed to load <tag> model!"}` and no `choices`; `/v1/completions`
+returns a nested OpenAI-shaped `server_error` (code 500) also under HTTP
+200. A status-code-only client reads this as a successful empty completion —
+**always check for the `error` key**. Two distinct causes, distinguished via
+`journalctl -u <flm-service> --full`:
+
+1. **Unknown tag** (not in the registry): immediate error, nothing loaded.
+   Correct behavior, not a bug.
+2. **Registered tag whose kernel pack is missing**: FLM resolves the tag,
+   downloads missing weight files on the fly (first request hangs ~90–110 s),
+   then fails `Failed to load model: No such file
+   '/home/dank/.config/flm/xclbins/<PackName>/<kernel>.xclbin'`; subsequent
+   attempts fail fast (~2 s). Root cause: the serve script exports
+   `FLM_XCLBIN_PATH=/home/dank/.config/flm/xclbins` (user dir), so every
+   kernel lookup resolves under the user dir, which only contains symlinks
+   someone linked. **A tag listed in `/v1/models` is NOT proof it is
+   loadable on this host.** Fix: one symlink per missing pack
+   (`ln -s /opt/fastflowlm/share/flm/xclbins/<PackName>
+   /home/dank/.config/flm/xclbins/<PackName>`) — verified live: after the
+   symlink the first request loaded and answered in 7.4 s wall.
+
+Switch-cost behavior: the `model` field routes per-request (no restart
+needed); a request whose model differs from the resident one triggers a full
+reload (7–11 s wall for an 8-token completion on ~4B-class packs);
+alternating tags pays the swap each time. Single-lane NPU: concurrent
+requests queue FIFO behind the current lock holder (`NPU Locked!` →
+`NPU Lock Released!`). Probe recipe: `/v1/models` for registered tags; grep
+the journal for `Failed to load model` (the quoted xclbin path names the
+missing pack dir exactly); diff `ls ~/.config/flm/xclbins/` (symlinks =
+installed) against `/opt/fastflowlm/share/flm/xclbins/` (available).
+
 ## Incident: qwen3.6-moe unreachable (2026-08-16)

 Symptom: a client on dankware could not reach `qwen3.6-moe:35b-a3b` on port 52625. The journal shows three separate causes in the 14:19-15:03 window.

 1. **EXEC 203 crash loop (14:19-14:20).** The exec bit was lost on `serve-flm-small.sh` on the target. The unit failed `status=203/EXEC: Permission denied` four times, then started after the bit was restored. The git mode is `100755` (commit `3f42210`), and a `git pull` restores the bit. Verify with `git ls-files -s scripts/serve-flm-small.sh`.
-2. **Wrong model ID (14:29-14:52).** The client config still used the model ID `frank-small` (a service alias). FLM rejects that tag with `[ERROR] Model tag 'frank-small' is not supported.` Each such request also triggered the unknown-tag fallback below.
+2. **Wrong model ID (14:29-14:52).** The client config still used the model ID `frank-secondary` (a service alias). FLM rejects that tag with `[ERROR] Model tag 'frank-secondary' is not supported.` Each such request also triggered the unknown-tag fallback below.
 3. **Swap churn (14:48-15:03).** The swap `gemma4-it:e4b` → `qwen3.6-moe:35b-a3b` restarted the service several times. Each load of the 23 GB model takes about 1.5-2.5 minutes at 262144 ctx. The endpoint does not serve during a load.

 Fix: the client model ID must be the FLM registry tag `qwen3.6-moe:35b-a3b`. The working pi agent entry in `~/.pi/agent/models.json` is `baseUrl http://192.168.0.143:52625/v1`, `id qwen3.6-moe:35b-a3b`, `contextWindow 262144`, `reasoning true`, `input [text, image]`. Verified end-to-end from dankware (192.168.0.226) at 15:32 on 2026-08-16.
@@ -313,7 +390,7 @@ pi sessions failed against this endpoint with `Stream ended without finish_reaso
 - **The FLM serve process is not under systemd.** A reboot of Dankclaw kills it. The PP head service is enabled and returns. The FLM process must be restarted by hand. A unit file is a natural next step.
 - **gpt-oss reasoning budget.** The model can spend its full token budget on `reasoning_content` and return empty content. Clients must handle or cap this.
 - **No llama.cpp GGUF path.** FLM uses its own pre-compiled kernels. Our UD-Q8_K_XL and HauhauCS builds are not importable. NPU work means FLM-specific model tags.
- **NPU production role (2026-08-16).** `frank-small` serves `qwen3.6-moe:35b-a3b` on port 52625 (swap churn and the unknown-tag incident are recorded above). The remaining NPU content (benchmarks, runbook steps) is still research until a step lands and is recorded.
+- **NPU production role (2026-08-16).** `frank-secondary` serves `qwen3.6-moe:35b-a3b` on port 52625 (swap churn and the unknown-tag incident are recorded above). The remaining NPU content (benchmarks, runbook steps) is still research until a step lands and is recorded.

 ## Citations

@@ -0,0 +1,97 @@
+---
+type: reference
+title: FreeToken Builds
+description: Building FreeToken (the official franksdank fork of FreeTok Tokenworks' freetoken vLLM-MTP runtime) on Dankclaw — the CUDA vLLM venv build and the ROCm build, with the dependency traps each one hit.
+tags: [freetoken, build, dankclaw, vllm, mtp]
+timestamp: 2026-10-05T00:00:00-07:00
+---
+
+# FreeToken Builds
+
+FreeToken = the official franksdank fork of FreeTok Tokenworks' freetoken
+vLLM-MTP runtime. Two build paths exist, both on **Dankclaw**, with
+different dependency traps. Both were verified end-to-end 2026-09-11/12.
+
+## Common verification (do not skip)
+
+**Verify the build by real completion, not by version string.** A version
+bump alone does not move data through the model — a build can print a new
+version and still not have completed a real token. Before declaring a
+FreeToken build deployable, run an actual completion (any tiny prompt) and
+read the returned content:
+
+```python
+import json, urllib.request
+req = urllib.request.Request("http://<host>:<port>/v1/chat/completions",
+    json.dumps({"model": "frank-secondary", "messages": [{"role": "user",
+    "content": "Hi"}]}).encode(), {"Content-Type": "application/json"})
+print(json.load(urllib.request.urlopen(req, timeout=600)))
+```
+
+A 200 with a real `choices[0].message.content` is the only build green.
+
+## CUDA build (the torch/vLLM dependency trap)
+
+Both paths build on **Dankclaw** with the same preinstall state: Python
+3.12.10 in the system venv and **Torch 2.10.0.dev** (nightly, repo
+download.pytorch.org/whl/nightly/cu130, installed with
+`uv pip install --index-strategy unsafe-best-match "torch==2.10.0.dev<date>+cu130"
+--index-url https://download.pytorch.org/whl/nightly/cu130` — use
+`--index-strategy unsafe-best-match`, `uv` otherwise refuses the dev
+version). Verify the exact pair first; mismatched preinstalled deps are the
+#1 build-failure cause.
+
+Then install the **matching vllm dev wheel**:
+
+```bash
+uv pip install --index-strategy unsafe-best-match \
+  --index-url https://wheels.vllm.ai/<commit-sha>/vllm-0.6.6.dev0+g<shortsha>.cu130-cp312-cp312-linux_x86_64.whl \
+  --extra-index-url https://wheels.vllm.ai/nightly/cu130
+```
+
+The wheel URL pattern is fixed:
+`https://wheels.vllm.ai/<full-commit>/vllm-0.6.6.dev0+g<8-char>.cu130-cp312-cp312-linux_x86_64.whl`
+— read it back from vLLM's CI-run page (`run details →
+build_and_install_package` job, "install the vllm wheel" step). **The trap:
+vLLM refuses versions < its install_requires floor at runtime, so the wheel
+version must be ≥ the runtime's floor.** A wheel built at a lower base
+(0.5.x) downloads but `import vllm` fails with a version constraint error —
+that failure looks like a build failure and is not one; upgrade the wheel,
+not the code.
+
+If `uv pip install -e . --no-deps` refuses (some repos pin the
+`--no-deps` branch off), fall back to `python3 setup.py` inside the venv,
+after `python -m venv --copies .venv && source .venv/bin/activate` and
+`uv pip install --no-build-isolation "torch==…"` — the venv must NOT
+re-resolve torch (a second torch from PyPI d1 breaks the CUDA extension ABI).
+
+## ROCm build (gfx1151 via TheRock nightly)
+
+Same venv discipline, ROCm wheels from TheRock's nightly index:
+
+```bash
+uv pip install --index-strategy unsafe-best-match \
+  --index-url https://rocm.nightlies.amd.com/v2/gfx1151-dcgpu-supports_fbsgens_split_0717-rc1/ \
+  "torch==2.10.0.dev<date>+rocm" "triton==3.6.0" "pytorch-triton==3.6.0"
+```
+
+Then `vllm` itself (`uv pip install -e . --no-build-isolation --no-deps`,
+repo branch `specs-mtp-dev`, commit `2ee440d`), after copying the two
+prebuilt extension wheels into the tree (the repo's `setup.py` skips
+compiling them when found). The ROCm build also needs
+`install-fastrak-gfx1151.sh` from the repo's scripts and
+`FASTRACK_HOME=/home/dank/FastTrack-runtime` in the unit env. FreeToken's
+x86_64-gfx1151 prebuilt extension (`freetoken-ops`) ships on HF hub
+`franksdank/freetoken-ops-gfx1151` — fetch it once and point
+`FREETOKEN_OPS_HOME` at it; never rebuild the extension from source unless
+the extension API changed.
+
+`--index-strategy unsafe-best-match` is load-bearing on both paths: the
+nightly indexes carry version pairs that `uv`'s strict resolver otherwise
+rejects, and the error (`No candidate found for pytorch-triton`) reads like
+a missing package.
+
+## Related
+
+- [[Model Swap and Rollback]] — the swap/deploy story these builds serve
+- [[Llama Cpp Serving Sizing]] — the harness measurement discipline
@@ -0,0 +1,123 @@
+---
+type: reference
+title: GGUF Compatibility and Runtime Lock-in
+description: How to answer "will engine X load this GGUF?" from the file's header alone, why a custom tensor type forces the runtime, and the discipline for pinning a fork build that can actually serve the file.
+tags: [gguf, llama.cpp, forks, quantization, speculative]
+timestamp: 2026-10-05T00:00:00-07:00
+---
+
+# GGUF Compatibility and Runtime Lock-in
+
+**The law: a GGUF's tensor-type histogram and ftype pick the runtime — read the
+header before promising a file to any loader.** Stock llama.cpp, Ollama and LM
+Studio refuse custom types at load (`unknown tensor type`); a fork-private type
+means exactly one engine can serve the file. Measured on this fleet repeatedly:
+`Qwen3.8-27B-Q8_0_ROCMFPX_AGENT.gguf` is ftype 115 with 311 tensors of type 103
+(`GGML_TYPE_Q8_0_ROCMFPX`) — only charlie12345/ROCmFPX reads it. The full type
+tables live in [[Llama Cpp Runtimes]] and [[Llama Cpp Code Diff]]
+(ggml_type 100–108, ftype families 100–106 / 110–113 / 118).
+
+## 1. Read the header, do not guess
+
+Parse the GGUF KV section and histogram the tensor types plus
+`general.architecture.file_type` — one python read over the first few MiB, no
+model load. Counting types out of the KV section settles a runtime argument in
+minutes; repo names and model cards do not.
+
+## 2. Compare routing variants by tensor mix, not by name
+
+Two files of the same quant can differ in how many tensors stay at plain Q8_0
+(measured: 187 vs 2 output-head tensors in an `_AGENT` variant of the same
+size). That difference is what moves spec-decode acceptance (0.953 vs 0.911)
+— the "same size" file is not the same file.
+
+## 3. Pin the build by commit SHA
+
+A fork that can read the file today can still crash mid-load: the 2026-08-25
+ROCmFPX builds read the ftype-115 file and then SIGSEGV in
+`ggml_compute_forward_set_rows`. Record the SHA in the build script
+(`scripts/build-rocmfpx.sh` pins `36e9acd`) and check it out explicitly — never
+a branch.
+
+## 4. Header diff method (candidate vs known-good)
+
+Answer "will engine X load this GGUF?" by diffing the candidate's header
+against a file the engine **already loads** — never by filenames or model cards;
+filenames lie about quantization and about MTP/vision heads.
+
+1. **Get the bytes**: both files, from byte 0, at least 64 MiB. Tokenizer KVs
+   (151k token strings + merges) routinely eat 5–8 MiB, so a small range
+   desynchronises the parser. True size from `curl -sIL` → `x-linked-size`
+   (a bare HEAD `Content-Length` can lie):
+
+   ```sh
+   curl -sL -r 0-67108863 -o cand.bin \
+     "https://huggingface.co/<owner>/<repo>/resolve/main/<file>.gguf"
+   ssh <host> 'head -c 67108864 /path/to/known-good.gguf' > good.bin
+   ```
+
+2. **Parse — and lock the parser against the known-good file first.** Layout
+   (v2/v3): `magic[4]` `version u32` `n_tensors u64` `n_kv u64`, then KVs, then
+   tensor infos. Strings are u64 length + bytes. **The value-type enum is NOT
+   llama.cpp's `gguf.h`:** in the files seen (2026-09-28, unsloth + ajgazin UD
+   GGUFs) type **8 = STRING, 9 = ARRAY**; llama.cpp's header has STRING=2,
+   ARRAY=3 and desynchronises immediately. Working table:
+
+   ```
+   0 u8   1 i8   2 u16  3 i16  4 u32  5 i32  6 f32
+   7 bool 8 STR  9 ARRAY(u32 et, u64 n)  10 u64  11 i64  12 f64
+   ```
+
+   Validate before trusting: the parse is right iff the tensor-info block ends
+   somewhere sane and names look real (`blk.0.attn_q.weight`,
+   `token_embd.weight`, `output_norm.weight`). EOF throw = wrong enum or
+   offset — do not work around it.
+
+3. **Diff what engines actually bind on**: `general.architecture`; tensor
+   count; tensor name list in order; per-tensor `ggml_type` and dims; geometry
+   KVs (`<arch>.block_count`, `.embedding_length`, `.feed_forward_length`,
+   `.attention.head_count[_kv]`, `.rope.*`, `.ssm.*`, `.context_length`,
+   `.nextn_predict_layers`); MTP tensors (`nextn` in the name); vision tensors
+   (`visual`, `v_` prefixes). A same-family fine-tune quantized by the same
+   tool usually comes out byte-identical in every one of these — that is the
+   answer.
+
+4. **Then read the engine's source before calling a delta a blocker.** A
+   metadata difference only matters if the engine reads that key
+   (`grep -rn "file_type\|recurrent_layers\|full_attention_interval" <src>/`).
+   Worked example (gufo v0.1.1): `general.file_type` is deliberately not
+   consulted (UD-*_K_XL ships a stale ftype 15; the tensor table rules), and
+   `qwen35.attention.recurrent_layers` is never read (layer kinds come from
+   `full_attention_interval`) — both deltas were non-issues the header diff
+   alone would have flagged wrongly.
+
+5. **Say what you did not do.** Header analysis is strong evidence, not a
+   load. State byte sizes, check free disk before proposing a fetch, and do
+   not download tens of GB unasked — pointing the engine at the file is the
+   definitive test.
+
+**Report shape:** verdict first (loads / will not / loads-with-caveats), then
+the side-by-side diff table, then only the load-bearing caveats: text-only vs
+vision tensors, chat-template differences (matters for llama.cpp, not for
+engines with compiled-in Jinja), license, disk headroom, and whether the base
+model is abliterated/uncensored.
+
+## 5. A/B a fork's spec decode by stripping the whole flag set
+
+`--model-draft` without `--spec-type` segfaults at load (measured 2026-10-02,
+core dump, no diagnostic). A baseline "no-draft" arm must remove `--model-draft`
+AND every `--spec-draft-*` AND `--spec-type` together — the flags move as a set.
+
+## 6. Before deleting a retired payload, read the rollback tag's env file
+
+A rollback may need a small companion dir: a drafter living in a directory
+named after an old quant looks like trash but is load-bearing. Same class as
+the gufo payload removal of 2026-10-03, which turned that rollback into a
+re-download.
+
+## Related
+
+- [[Llama Cpp Runtimes]] — which fork ships which custom types
+- [[Llama Cpp Code Diff]] — the enum tables and code-level differences
+- [[frank-large-rocmfpx-q8agent-mtp]] — the ftype-115 lock-in in production
+- [[Llama Cpp Serving Sizing]] — pricing what a payload costs once it loads
@@ -0,0 +1,159 @@
+---
+type: reference
+title: Heterogeneous Split Triage
+description: Triage before building when someone asks to split a model across unequal accelerators on one host (24 GB GDDR6X + LPDDR APU, PCIe x4 link) — link characterisation, the real bottleneck, what placement never buys, UMA pools, drafter splits, and the verdict ranking.
+tags: [heterogeneous, multi-device, llama-cpp, triage, pcie]
+timestamp: 2026-10-05T00:00:00-07:00
+---
+
+# Heterogeneous Split Triage
+
+Produced the 2026-09-16 no-go verdict on splitting Qwen3.8-Flash-Next across
+the RTX 3090 + Strix Halo pair. Use it to answer "could we split model X
+across device A and device B" with arithmetic instead of enthusiasm.
+
+## 1. Characterise the link (bytes vs latency)
+
+```bash
+lspci -vv -s <bdf> | grep -E "LnkCap:|LnkSta:"          # capability vs actual
+nvidia-smi --query-gpu=pcie.link.gen.max,pcie.link.gen.current,pcie.link.width.current --format=csv
+```
+
+Expect **idle-downshift**: a Gen4 x4 dock idles at Gen1 x4 (~1 GB/s) and
+upshifts under load. Measure while the device is busy, or you size the design
+against the wrong number. Per-hop cost against the hidden state
+(`n_embd × 2 B` for f16): a 5120-dim model moves 10 KiB/hop ≈ 1.3 µs at
+Gen4 x4.
+
+- **Bytes per token ≈ hops/token × hidden state** — typically ~1 MiB/token
+  for 2 hops/layer × 64 layers ≈ 0.3 % of a 40 ms token. **Bytes are almost
+  never the constraint.**
+- The real budget line is **hop COUNT**: every device boundary is a
+  host-side sync. A runtime that splits per *op* hits ~100 splits/token and
+  pins the GPU at ~5 % SM. Measure `graph splits` in the loader log; never
+  assume.
+- **Design rule: partition by where the weights live, never by where the
+  activations flow.** Resident weights, activations crossing.
+
+## 2. Get the workload's real bottleneck (KV is usually it)
+
+```
+KV/token = kv_bytes_per_position × context
+```
+
+Attention reads the whole cached context every token, so KV/token scales
+with context — compare it against expert/weight traffic per token and total
+token time. At long context on a hybrid arch (few attention layers, many
+positions), **KV dominates**. Consequences:
+
+- The highest-leverage placement is **KV cache on the fastest-memory device,
+  together with its attention compute** — never KV alone: separating them
+  puts KV/token over the link, which is fatal (GBs across 7.88 GB/s =
+  hundreds of ms/token).
+- Check the fit: `kv_positions_per_slot × slots × bytes_per_position +
+  attention_trunk ≤ VRAM`.
+
+## 3. What placement never buys
+
+Prefill is compute-bound and expert weights are read once per batch, so
+moving them to faster memory buys nothing while adding a serial stage.
+Expect a regression, and size it from calibration: on Strix Halo + RTX 3090
+(Qwen3.8-Flash-Next, ik_llama.cpp) moving 12.5 % of the expert mass to the
+GPU gave **decode +12 %, prefill 192 → 32 tok/s**. A
+`1/(f_big/BW_slow + f_fast/BW_fast)` harmonic calculation reproduces that
+decode gain to within ~2 points — usable for decode projections, and it
+assumes sequential stages (true at batch 1; no pipelining).
+
+## 4. Is the second device even a separate pool?
+
+On a UMA APU, check before assuming additivity:
+
+```bash
+cat /sys/class/drm/card*/device/mem_info_vram_total   # BIOS carve-out (small)
+cat /sys/class/drm/card*/device/mem_info_gtt_total    # aperture onto SYSTEM RAM
+rocm-smi --showmeminfo all | grep -i gtt              # used == what the CPU also sees
+```
+
+A large GTT window means the "GPU memory" is the *same* LPDDR under one
+memory controller: adding a stage there adds a compute engine, not capacity
+or bandwidth. Only the discrete card's VRAM is additive. Fencing more
+(`amdgpu.gttsize`, BIOS UMA size) just removes bytes from CPU-visible RAM.
+
+## 5. Speculative decoding as the "cheap" split
+
+A drafter on the *other* device is bandwidth-free (token ids per round, tens
+of bytes), so it is tempting. Before proposing it:
+
+- the drafter needs the target's **exact tokenizer** for a plain draft model
+  (compare GGUF `tokenizer.ggml.tokens`, never trust the family name), or it
+  must be a head **trained for that target** (EAGLE/DFlash/MTP are
+  conditioned on the target's hidden states and are not portable between
+  models);
+- a plain drafter costs **K sequential forwards per round**, so it must be
+  10–50× faster than the target. Break-even with `r = t_draft/t_target`:
+  `rate_ratio = A/(K·r + 1)`, `A = (1−p^(K+1))/(1−p)`; at `r ≈ 0.4–0.5`
+  (a drafter only ~2× faster) break-even needs `p ≈ 0.7+` at K=3;
+- compare against the **incumbent**, not the no-draft baseline: a target
+  already running its own MTP head is a high bar;
+- check the runtime accepts an external drafter at all — closed engines may
+  enumerate a fixed drafter set and 400 anything else (capability gates
+  keyed on the loaded checkpoint). That is a design boundary, not a config
+  gap. In llama.cpp the portable mechanism is `--spec-draft-device/-devd`
+  (draft gets its own device list) plus `-ngld`/`-otd`/`-ctkd -ctvd`.
+
+## 6. Expressing a split (llama.cpp, the usual vehicle)
+
+```bash
+-dev CUDA0,Vulkan0 -sm layer -ts F0,F1        # contiguous layer ranges (simplest)
+-ot "\.attn_.*=CUDA0" -ot "\.exps\.=Vulkan0"   # per-tensor, by ROLE (KV co-location)
+```
+
+- buffer types match by name at load (`ggml_backend_buft_name`), so device
+  names work as `-ot` values;
+- `-cmoe`/`-ncmoe` are **CPU-only**; device-side MoE placement is `-ot`;
+- several backends in one binary is fine (CUDA + Vulkan is the practical
+  pair; HIP + CUDA do not coexist) — `GPU_BACKEND_A=OFF` in the existing
+  build's CMakeCache tells you what needs adding;
+- contiguous ranges are the WRONG split when KV bandwidth dominates: they
+  drag half the attention layers (and their KV) back to the slow pool.
+
+## 7. Hazards checklist
+
+- Paged/sidecar tables (huge n-gram/embedding tables) must stay where they
+  are read from; moving them is pure loss.
+- Keep `mmap` on for models with huge embedding tables (anonymous memory
+  gets the process cgroup-killed).
+- CUDA graph capture can abort on unusual graph shapes (hyper-connections)
+  — check before assuming optimisations apply.
+- A brand-new backend on the second device may not implement the arch's ops
+  at all: make the first measurement a **baseline of the runtime on that
+  device alone** — a split's arithmetic is worthless if the baseline is
+  worse than the incumbent engine.
+- A split consumes the fast device entirely — check what else it was
+  serving before trading it away.
+- Benchmark discipline on production hosts: stop and mask first
+  ([[Fleet Ops Playbook]]).
+
+## 8. Verdict shape
+
+Rank and report; prefer the boring answer when it wins:
+
+1. **Two independent endpoints** (zero shared traffic) — often the real
+   answer to "what's a useful combination", especially over a narrow link.
+2. **One process, weights-resident, KV co-located with attention on the
+   fast pool** — promising and quantifiable (+% decode at long context);
+   needs a dual-backend build and a baseline measurement.
+3. **Hot-expert cache / per-token weight streaming** — affordable only if
+   the miss set per token is small (a few experts; check
+   `misses × bytes × tok/s` against link bandwidth) and overlapped; injects
+   PCIe latency into the critical path; no mainstream runtime implements a
+   dynamic expert cache.
+4. List explicitly what does NOT help: TP/collectives across unequal
+   devices, KV on the far device, model loading over the link,
+   prefill-motivated placements.
+
+## Related
+
+- [[PCIe Link and Atomics]] — the link and collective walls underneath
+- [[Llama Cpp Multi-Device Splits]] — the deployed split shapes
+- [[Llama Cpp Serving Sizing]] — KV and pool arithmetic
@@ -0,0 +1,271 @@
+---
+type: reference
+title: LLM Bench Windows and Measurement
+description: Stopping-and-restoring benchmark windows on the fleet LLM hosts (llama-bench sweeps, HTTP param benches, multi-arm vLLM offload sweeps), honest benchmark methodology, and the usage-counter measurement law under speculative decoding.
+tags: [benchmarking, llama-bench, ab-testing, speculative, vllm, fleet]
+timestamp: 2026-10-05T00:00:00-07:00
+---
+
+# LLM Bench Windows and Measurement
+
+Fleet law first: **STOP the production service before any GPU experiment**
+(AGENTS.md Benchmarking Policy) — a bench beside a live service starves the
+GPU and corrupts results; an unisolated GPU experiment has precedent for
+wedging a host until power cycle. Reference harnesses:
+`scripts/bench-batch-ubatch.sh`, `scripts/autoresearch.sh`,
+`scripts/autoresearch/frank-primary-arms.sh`.
+
+## 1. The window structure (one config per arm, a driver owns the window)
+
+1. **Stop + gate before every arm** — `wait_for_clear`: unit NOT active AND
+   no RUNNING container of that name (`podman ps`, not `-a`; `-a` counts
+   EXITED containers and can never clear — `rm -f` the exited one first) AND
+   port free (`ss -ltn`). The graceful-stop race cost a whole arm 2026-10-01:
+   `ExecStop=podman stop -t 120` drains up to 2 min; a driver that sleeps 3 s
+   then polls `/health` gets **200 from the still-running PRODUCTION engine**
+   — the arm "passes" with production's boot log, then `podman exec` dies
+   with `container state improper`.
+2. **Boot the arm as the service, not a scratch port** (no scratch ports on
+   LLM hosts — one card, one server). The serve script takes its config via
+   env-file override (`FRANK_SECONDARY_ENV_FILE`, `FRANK_PRIMARY_AR_ENV_FILE`)
+   pointing at a staged copy; the checkout is never mutated.
+3. **Fast-fail the health wait**: poll `podman ps | grep -qx <name>` inside
+   the loop — a refused boot dies in one poll instead of 20 min. Capture the
+   serve script's FULL stdout on failure (a `tail -8` loses engine-init root
+   causes; raise happens above the tail).
+4. **Measure** (§4), capture boot-log engine facts (KV pool, arena slots,
+   SpecDecoding lines) — they survive measure crashes.
+5. **Teardown + converge**: kill by explicit PID/pgid (never pattern kills),
+   `podman rm -f` scratch, wait `MemAvailable ≥ ~230 GiB` on dankserver
+   before the next boot; restore production from the EXIT trap and verify the
+   **conjunction oracle**: health 200 + `is-active` + `is-enabled` +
+   mask-symlink clean. A bare `systemctl start` is not a restore oracle.
+   (`systemctl mask` FAILS on a relinked unit — log it, explicit stop is
+   enough: systemd never auto-restarts an explicitly stopped unit.)
+6. **Warm boots cycle ~2 min** (page cache), cold ~8 min — size timeouts for
+   mixed speeds.
+
+**Destructive tails (2026-10-01):** a scratch script whose tail does
+`systemctl stop` + `podman rm -f <PRODUCTION container>` fires later if a
+probe loop survived the driver kill — it can destroy a just-deployed
+production engine (signature: deploy exits green, then seconds-to-minutes
+later `Exited (0)`, `Deactivated successfully`, clean exit = removed, not
+crashed). Before ANY deploy: `pgrep -af` every scratch script of the session
+and kill strays by explicit PID; killing the dev-side job does NOT kill the
+host-side `deploy-remote.sh` (two of them once managed the same unit).
+
+Two failure shapes inside the window that read as "the bench died":
+
+- **Stale reads from a stopping container.** `podman stop` answers while the
+  engine inside is still flushing; a `podman logs` taken in that window
+  returns the boot-prefix only. Poll `podman ps` for `Exited (0)` (or
+  `podman wait`) before reading logs, or you will write "no results" into
+  the table for a run that has them.
+- **A D-state worker wedges the stop.** A bench child that wedged in the
+  kernel ignores SIGTERM, so `podman stop -t N` burns its full timeout and
+  then the container is `Stopping` forever. Kill by container first
+  (`podman kill <ctr>` → SIGKILL path), verify the process is gone from
+  `podman ps` AND `nvidia-smi`, and only then re-serve. Never let the
+  window script proceed to a redeploy with a zombie bench worker holding
+  VRAM.
+
+## 2. llama-bench sweeps (offline batch/ubatch/split/KV/depth)
+
+**What llama-bench cannot express — decide BEFORE designing:** no
+`--ctx-size` (`n_ctx = n_prompt + n_gen + n_depth`, so a bench can NEVER prove
+activation buffers fit beside the DEPLOYED KV — only the real unit answers
+that), no `--parallel` (no slot shape, no concurrency), no `--spec-*`
+(`tg` numbers are the drafterless baseline, useful as a control only), and a
+warmup run is itself a full-size run. `-d N` is a depth PREFIX (paid once per
+cell via a `llama_state_seq_*` save/restore); without it a `tg` run is at
+~zero depth and meaningless for long conversations.
+
+**The separator trap:** `/` separates devices WITHIN one configuration, `,`
+separates sweep values — for both `-dev` and `-ts`. `-ts 36,29` parses as the
+single value 36 → everything on one card. `--tensor-split 36,29` (server) →
+`-ts 36/29` (bench). Ranges exist: `-ub 512-2048+256`.
+
+**Harness shape:** one invocation per cell (an infeasible cell calls
+`exit(1)` and kills every later cell of a cross-product invocation — and
+infeasibility is usually the question); TSV row per cell as it completes +
+`bench.json` (`-o json`) + `bench.log` (`--progress`) + a 2 s `rocm-smi
+--showmeminfo vram` VRAM trace per cell (a cell that allocated less than
+intended looks fast); resume by skipping non-empty cell outputs; stop the
+service in the harness, restore from `trap … EXIT`, verify afterwards. Match
+the deployment's load flags exactly (`-ngl -sm -ts -fa -ctk/-ctv -t`).
+
+**Run under a transient unit, not a shell:**
+
+```bash
+sudo systemd-run --unit=<name> --collect --no-block --property=Type=oneshot \
+     --property=LimitMEMLOCK=infinity --property=User=dank \
+     /home/dank/serverAppConfigs/scripts/<harness>.sh
+```
+
+Deploy units set `LimitMEMLOCK=infinity` for ROCm staging; an SSH shell has
+8 MiB and fails the load for an unrelated reason. This also survives ssh
+disconnects. **Boot rungs under the production environment, not a bench
+shell** — a rootless shell probe booted 3×262144 bf16 that production refused
+(systemd `User=` + memlock change what the pinned-expert path holds):
+`sudo systemd-run --uid=<user> --unit=<name> -p LimitMEMLOCK=infinity -p
+Restart=no env VARFILE=… /path/serve.sh`.
+
+**Verify before burning hours:** dry-run against a STUB binary with
+`SERVICE=<nonexistent-name>` (the trap/stop never reached), tiny grid,
+including one forced-`exit 3` cell — prove the TSV row, failure capture,
+resume logic. Validate the real flag set in ~1 s by pointing `-m` at a
+nonexistent file. **Reading:** bracket the sweep (reference cell first AND
+last); check per-cell `stddev_ts` (a difference smaller than spread is not a
+result); single-sequence decode is usually flat across batch/ubatch (prefill
+knob); batch ≥ ubatch (llama.cpp clamps ubatch to batch, silently duplicating
+smaller-batch cells); report what the grid cannot prove (drafter, slots, real
+context).
+
+## 3. HTTP benches of server-side parameters
+
+`llama-bench` benches the bench binary, not the running server — server-side
+params (`--threads-batch`, `--ubatch-size`, `--spec-type`, `--parallel`) need
+HTTP:
+
+- **Prefill**: large prompt, `max_tokens=1` (server crashes on 0), measure
+  `prompt_tokens / wall_time`.
+- **Decode**: `max_tokens=16384` minimum, NEVER 1 (warmup, cache growth and
+  spec acceptance all stabilize over thousands of tokens); measure
+  `predicted_per_second` from `timings` or `completion_tokens / wall_time`;
+  temperature 0.
+
+Known fleet harnesses: `scripts/bench-threads-batch-http.py`,
+`scripts/bench-decode-threads-batch.py`, `scripts/bench-spec-decode.py`.
+sglang-specific: `max_tokens=1` on `/v1/chat/completions` with the qwen3
+reasoning parser streams zero deltas — use `/v1/completions`.
+
+## 4. The measurement law (under speculative decoding this is absolute)
+
+**Never count SSE chunks.** Rejected drafts break the 1:1 chunk↔token
+mapping; chunk counting reported 8.3 tok/s for a config whose server counters
+said 23–26 — a 3x false regression that nearly abandoned the change that
+worked. Always:
+
+```python
+decode_tok_s = usage.completion_tokens / (wall - ttft)
+# stream=True, stream_options: {include_usage: true}; ttft = first content-bearing chunk
+```
+
+vLLM's 10 s `Accepted throughput` log lines are a rolling window, not a
+measurement (6.2 → 13.1 across adjacent windows on identical settings) — use
+them for acceptance STATISTICS only (`Avg Draft acceptance rate`, `Mean
+acceptance length`, per-position rates). Warm pass + ≥3 runs, median;
+temperature 0; ≥256 tokens. **A decode "jump" with early-stopped completions
+(ct < 256) and acceptance > 90 % is degenerate generation** (the fork's
+repetition-halt fired at ~140 tokens on corrupted numerics) — inspect text
+before believing a win. Run-to-run spread is 5–10 %: repeat arms; one sample
+nearly shipped the wrong setting.
+
+## 5. Honest methodology
+
+- **Match the corpus to the workload.** Token soup under-reports speculative
+  decoding (2.9–3.6 tok/update vs 3.86–5.8 on real code) and mis-ranks draft
+  settings; acceptance, not step time, moves throughput. Build the corpus
+  from the repo's own sources: `{"prompt": "<instruction>\n\n<file body>"}`,
+  `--custom-output-len`.
+- **Salt every run** with a unique per-arm token in the first bytes — it is
+  the prefix-cache key (and a proxy routing key). Without it arm 2 measures
+  arm 1's warm cache.
+- **Two workload shapes need two clients:** distinct prompts (batch
+  behaviour, prefill parallelism) and sequential turns with growing history
+  (TTFT of the first streamed chunk per turn — the only measurement that
+  exposes prefix-cache behaviour). These pull in opposite directions: a cold
+  batch spreads prefill across replicas, a conversation's next turn only pays
+  if it lands where its prefix is cached.
+- **Read the engine's ledger, not the card:** `Available KV cache memory` /
+  `GPU KV cache size: N tokens` / `Maximum concurrency for M tokens per
+  request: X.Xx` is the capacity number. vLLM's `Prefix cache hit rate:` line
+  is a LIFETIME average across every prompt the replica ever saw — send the
+  same prompt twice to an idle engine and compare TTFTs (measured 22.3 s cold
+  → 1.4 s warm for a 50k prefix). `GPU KV cache usage` says the pool is not
+  full; it does NOT say a prefix survived. The KV budget is NOT monotonic in
+  chunk size (5.44 GiB at CHUNK 8192, 4.22 at 16384, 4.16 at 4096) — measure
+  three points before treating a knob as a lever.
+- **Soak before believing:** 15–30 min of concurrent families, count failures
+  AND latency; report hit/miss split and medians with min/max — the median of
+  a bimodal distribution describes neither mode.
+- **Verification that is not the benchmark:** run the client the fleet
+  actually uses; and a deployed unit is not proven until an unattended reboot
+  brings it back.
+- **Record the "no" answers** with numbers and mechanism — placement balance,
+  concurrency admission and chunk-size tweaks all measured flat or worse
+  here; the wall was arithmetic (cached prefixes + live sequences share one
+  pool).
+
+## 6. Sweep hygiene (the arms that produced wrong verdicts)
+
+- **Prefetch uniformity:** every arm in a sweep carries the same
+  expert-prefetch setting. The 2026-09-29 campaign measured losing arms with
+  prefetch ON and incumbents OFF; untaxed, verdicts inverted (no-spec
+  12.11 → 41.47; gather 32×64 parity → +18.7 %). Never re-read a taxed
+  verdict.
+- **Knob-landed preflight:** for every `-e` knob, confirm
+  `podman exec <ctr> printenv <VAR>` — a knob the fork silently ignores
+  produces a "parity" arm that is really the default (v1 of the gather sweep
+  lost all 3 arms this way; serve-script hooks are additive and can drop
+  vars).
+- **Always run an explicit OFF arm:** the 2026-09-29 campaign's only win was
+  deleting a baseline's silent `--expert-prefetch 1` (+40–51 %). Challengers
+  compare against default-off, not "incumbent + one flag".
+- **Pin the capacity bar:** `--kv-cache-memory-bytes` (contract) rather than
+  autofit (remainder) so pool numbers are reproducible; compute need =
+  tokens × measured B/token (hybrid models: KV dtype does NOT double mamba
+  states — measure, don't assume 2x).
+- **Report decode AND prefill (TTFT) per rung:** a decode-only view nearly
+  deployed a 2x-prefill-tax config. Per-depth tables, never a single median —
+  one fork's gather win grows with depth (54 t/s @1k, 39.5 @short).
+- Container recreate (`podman rm -f`) wipes `podman cp`'d harnesses — re-copy
+  `mtp_measure.py`/`bench_iter.py` after EVERY recreate, or mount them.
+
+## 7. Bash laws that ate windows (each cost an arm)
+
+1. `${VAR:+SPEC=$1}` in front of a command expands to a bare word →
+   `SPEC=3: command not found`, nothing runs. Pass literal assignments:
+   `SPEC="${1:-}" bash serve.sh`; the callee's `${SPEC:+--flag}` skips empty.
+2. `${2WAY:-0}` is fatal `bad substitution` (digit-leading name), and inside
+   an if/else it aborts ONLY that compound statement — the script CONTINUES,
+   every arm tears down clean with zero measurements. Detection law: `ls`
+   the per-arm `.measure.txt` files before trusting any summary; arm echoes
+   are not evidence.
+3. `local a=$1 b=$x` expands `$x` before the builtin assigns — under `set -u`
+   it aborts AFTER the harness stopped production. Two statements.
+4. `sed -i` renames the file; already-running bash keeps the OLD inode —
+   mid-flight patches miss arms launched in the preceding minutes. Patch
+   early or run a make-up window.
+5. Never heredoc python through ssh→zsh→bash — write to a file, ship it
+   (`ssh host 'cat > /path && chmod +x /path' < local.sh`); nested quoting
+   fails silently. Same trap in `git commit -m`: backticks become command
+   substitution.
+6. No comments INSIDE a backslash-continued command (the continuation joins
+   the comment); verify multi-line command construction with `bash -n` AND a
+   render test.
+
+## 8. Results location and recovery (frank-primary arms)
+
+| copy | path |
+|---|---|
+| per-arm evidence | `$ARMS_DIR` = `~/.jobs/<run-stamp>/` (`.rc`, `.done` beside it) |
+| summary | `$ARMS_DIR/summary.tsv` = `arm rc decode_tps decode_tps_t0.7 kv_tokens boot_s` + `<arm>.out/.env/.overlay/.rc/.metrics` |
+| server-side last window | `/home/dank/.autoresearch-frank-primary/run.json` |
+| durable memo | `.../last/run.json` + `last/hash` (promoted only after bench AND verified restore) |
+| fallback | `~/.omp/agent/sessions/<ws>/<session>/<N>.{shake,bash}.log` (permanent) |
+
+`/tmp/frank-primary-arms-*` and `/tmp/armstub/` are stub self-tests —
+constants 99.9/88.8 identical across arms by construction; **identical
+numbers across arms == stub or bug, never a measurement.** After compaction,
+re-harvest with `grep -rhoE 'decode_tps[a-z0-9_.]*=[0-9.]+'` over the session
+logs. A window costs ~20 min warm; a detached runner's timeout ≥ 3600 s per
+window.
+
+## Related
+
+- [[RDNA4 MoE Expert Offload VRAM Budget]] — the plan arithmetic being swept
+- [[DFlash DSpark Speculative Decoding]] — spec A/B protocol and shapes
+- [[LLM Performance Diagnosis]] — profiling and attribution methods
+- [[Remote Job Runner]] — detached execution of long sweeps
+- [[frank-medium-llamacpp-dflash2]] — the measured ubatch-optimum history
@@ -0,0 +1,221 @@
+---
+type: reference
+title: LLM Performance Diagnosis
+description: Probing why a served LLM is slow — the flat-depth probe, the expert-offload duty-cycle arithmetic, kernel-time attribution via the profiler window, live telemetry without benchmarking, RAM-bandwidth ceilings, IBS tensor attribution and prefix-cache-aware routing.
+tags: [performance, profiling, moe, pcie, prefix-cache, routing, telemetry]
+timestamp: 2026-10-05T00:00:00-07:00
+---
+
+# LLM Performance Diagnosis
+
+When a served endpoint is slower than wanted and someone names a suspect
+("PCIe is saturated", "the gather is the bound"), throughput arithmetic cannot
+settle it — it is usually circular (it assumes the suspect consumes the step).
+Get data in this order; the profiler window alone has retired multi-day
+optimization plans.
+
+## 1. Is decode context-bound at all? (the flat-depth probe)
+
+Sweep decode across context depth. **Flat** ⇒ attention over history is not
+the cost; the problem is per-token expert movement and KV/attention tuning is
+ruled out entirely. A genuinely context-bound server shows a clear downward
+slope. Record cold vs warm separately — a small gap means the cache is
+working. Measured on the fp8 expert-offload shape: 12.11 tok/s FLAT from 304
+to 16,095 tokens.
+
+## 2. Duty-cycle arithmetic (the "GPU not fully utilized" answer)
+
+Symptom: smooth ~25–35 % utilization with 100 % peaks on an expert-offload
+MoE. This is the EXPECTED signature, not a fault:
+
+```
+per decode step: compute burst ~100 ms (100 %) → expert-gather wait ~400 ms (~0 %)
+resident fraction = arena slots / total experts   (e.g. 6574/24576 ≈ 27 %)
+PCIe bytes/token  = miss_rate × lookups × slab_bytes  (0.73 × 480 × 2.344 MiB ≈ 0.84 GB)
+predicted duty    ≈ compute_ms / (compute_ms + wait_ms)
+```
+
+Validate the terms against the deployed engine's boot plan
+([[RDNA4 MoE Expert Offload VRAM Budget]]). If predicted ≈ observed, the low
+average is structural. Note small-transfer reality: bulk bandwidth (91 % of
+ceiling at 1 GiB) overstates what 2–3 MiB slabs get (~72 % plus ~20 %
+per-transfer latency). Before blaming the link, measure it
+([[PCIe Link and Atomics]]) and compare the engine's effective gather rate to
+the measured slab-size bandwidth — a 40–50 % gap is the software read path,
+not the wire. `rocm-smi --showuse` reads ~0 % during the whole multi-minute
+weight load: show VRAM (`--showmeminfo vram`) as the load-progress signal.
+
+## 3. Kernel-time attribution (the profiler window)
+
+Run BEFORE building any replacement kernel.
+
+1. Stop + mask production; boot the shape under test with ONLY the profiler
+   config added. vLLM argv:
+   `--profiler-config '{"profiler":"torch","torch_profiler_dir":"/prof","torch_profiler_with_stack":false,"torch_profiler_record_shapes":false,"torch_profiler_with_memory":false,"torch_profiler_dump_cuda_time_total":false}'`
+   — `with_stack=false` is load-bearing (with stacks the stop hangs and the
+   trace buffer grows ~40 GB → kernel OOM-kill of the engine core, anon-rss
+   ~95 GB; the worker-side stop can hang 5 min on ROCm/kineto). Bind-mount
+   `/prof`. If a shell substitutes values into that JSON, verify the escapes
+   actually landed.
+2. Window it: warm request → `POST /start_profile` → N generations of ≥128
+   tokens (~15 s of traffic) → `POST /stop_profile` (client may time out at
+   60 s; the trace still lands). `telemetry_report` has ONE caller: the
+   gpu_worker profiler-stop path. Overhead is small enough to keep decode
+   comparable (9.6 vs 9.9–10.0 tok/s measured).
+3. **Parse ON THE SERVER** (traces are 100–150 MB gzipped, GB expanded).
+   Rank `ph=="X"` events with `cat in {kernel, gpu_memcpy, gpu_memset}` by
+   summed `dur` (µs): report % of device time, launch count AND µs/call
+   (22 µs × 708k launches is a different problem than 4 ms × 4k), plus the
+   first-to-last event span (fraction of wall the device was busy). Aggregate
+   by FULL mangled name (template args are the tuning signal) and by stripped
+   symbol. **Trap: `gpu_user_annotation` events masquerade as GPU time** (a
+   naive sum reported 63 % in one `execute_context_0(...)` entry; spans
+   overlap and can exceed the wall clock).
+4. Divide by the window's token count → ms/token; if kernel time ≈ step
+   time, the table IS the optimization plan.
+
+Worked example (GLM-5.3-Flash, 2×GPU, offload, decode 9.9 tok/s):
+
+```
+34.8 %  708,500 launches  vectorized_elementwise_kernel   <- the actual bound
+10.0 %   42,000           MoE W4A16 GEMM
+ 8.9 %   83,160           router/top-k (sort-based)
+ 6.5 %  133,920           hyper-connection pre/post kernels
+ 1.05 %   20,664          <the long-suspected expert gather>
+```
+
+No amount of link arithmetic would have produced those two conclusions: the
+suspect was 1 %, and the mass was pointwise glue from an uncompiled trunk —
+whose own boot log names the cause (`FULL_DECODE_ONLY` override). It also
+predicted a negative result in advance (spec decode cannot help a shape whose
+per-token cost scales with the token count) which matched an unexplained flat
+MTP measurement.
+
+## 4. Telemetry without benchmarking (live hosts)
+
+When benching is forbidden (production serving), mine the runtime's own
+per-request telemetry — it measures the real workload and costs nothing.
+
+- llama.cpp (root unit, `-lv 3`): `sudo journalctl -u <unit> --since "-3h"
+  --no-pager` → `slot print_timing` lines (prompt eval, eval, total, `graphs
+  reused`, `draft acceptance … mean len`, `n_gen = N, tg = X t/s, tg_3s = …`).
+- halogen (rootless quadlet): `journalctl --user -u <unit>` (**never sudo** —
+  per-user storage) → `serve_api: mtp 28 tok in 0.66s = 42.45 t/s | 16
+  rounds, commit 1.75/round | prompt 58, prefill 0.92s | detok 24us/tok`.
+- Read the envelope, never one sample: short-prompt `prompt eval` is
+  overhead-dominated (57 tok @ 168 t/s) — quote the longest available (1106
+  tok @ 1013 t/s); decode and acceptance are content-dependent (0.46–0.76
+  measured on one unchanged build); `graphs reused > 0` / `selected slot by
+  LCP similarity` prove prefix-cache reuse.
+- **Attribute PCIe without touching the model:** background `nvidia-smi dmon
+  -s t -c 16 -o T`, ONE ordinary `max_tokens=200` request, `wait` at the end
+  (without it ssh closes and the client dies — the journal shows `W srv stop:
+  cancel task` and NO print_timing while curl still reports plausible
+  status). Measured: peak ~230 MB/s against a ~7,900 MB/s link = ~3 %, 0 at
+  idle — the sampled-logits stream, not the transport, is decode traffic on
+  that shape.
+- One `max_tokens=200` request is production traffic and fine; anything that
+  pins the card is a bench → stop-and-mask first. Check WHICH instance you
+  are reading (`systemctl status | head -12`; task ids jump across
+  restarts). The engine's own ledger (`KV self size`, `n_slots`,
+  `n_ctx_slot`) outranks your model of it.
+
+## 5. The fork's data path (before blaming the link)
+
+Offload engines pick between DMA copy-in and read-through UVA dereference
+inside the GEMM. Grep the installed package for the LRU/offload module and
+read the BRANCH CONDITION, not just the env names. A token-count threshold
+decides the path, and it interacts with spec decoding: an MTP step is k+1
+tokens, so the step can silently cross the threshold onto the slow path as
+draft depth grows. Check the threshold's premise: a default tuned at
+12.4 MiB/expert does not transfer to 2.344 MiB/expert, where copy-in can be
+several times faster. TunableOp ghost-skip: `CLAV_TUNABLEOP_SWEEP=1` alone
+does NOTHING under expert offload (`CLAV_TUNABLEOP_WITH_OFFLOAD=1` required);
+check for the skip line before concluding autotuning ran. Gather-geometry
+knobs read at import time → each arm is a full reload.
+
+## 6. RAM-bandwidth-bound MoE ceiling (before recommending a quant)
+
+```
+routed experts/token = top_k × n_layers
+bytes/token          = routed_params × bytes_per_param + shared_expert_bytes
+ceiling (tok/s)      = measured_bandwidth / bytes_per_token
+```
+
+Measure the HOST's real bandwidth, never assume DIMM ratings: 4-DIMM AM5
+boards downclock to DDR5-3600 regardless of 5600 sticks (two large DIMMs beat
+four small). Single-threaded reads under-report dual-channel ~4x — use the
+multi-threaded plateau of a checksumming reader (`scripts` pattern:
+`/tmp/bw.c`, 1/2/4/8/16 threads). Worked example (DeepSeek-V4.1-Flash):
+6×40 experts × 35.4M params = 8.5B params/token; at 3 bpw + 0.53 GB shared =
+**3.7 GB/token** → 42 GB/s ⇒ **11.3 tok/s ceiling**. Sanity anchors: H100
+NVL + CPU offload 226 GB ⇒ 5.8–9.0 tok/s; 2×3090 + 768 GB DDR4 ⇒ 12.2.
+CPU compute rarely binds below 4 bpw (~5.3 flops/byte ⇒ ~223 GFLOPS at 42
+GB/s); PCIe expert STREAMING is worse than CPU compute (Gen5 x8 ~22.7 GB/s
+effective < half of RAM); prefill reads the union of a batch's experts
+(prefill ≈ bandwidth / total_expert_bytes × ubatch). A quant that fits only
+by ignoring the Engram table does not fit (Engram = sparse n-gram lookup,
+NVMe-tolerant; the routed experts are not). An iGPU GTT aperture is an ALIAS
+of the same RAM, not extra memory.
+
+## 7. IBS model-access attribution (AMD CPUs)
+
+Real CPU access patterns for mmap'd weights, live, no stop (established on
+frank-primary, 2026-09-26, ~2.7M attributed samples in 75 s). Reference:
+`scripts/frank-primary-expert-activity.py` (self-contained dump + sample +
+analyze; `--analyze-only`).
+
+1. GGUF header-only scan (ms for a 50 GiB part) → tensor extents. **Extents
+   from CONSECUTIVE offsets, never a type-size table** — a wrong per-element
+   size silently under-covers every tensor and dumps tail samples into a junk
+   bucket (first run lost 59 % of expert samples; a uniform gap/exps ratio
+   ~1.4 across parts is the tell). Data start = `(tell + 31) & ~31`.
+2. `/proc/<pid>/maps` VMAs (7-group regex; pgoff = group 4, path = group 7)
+   matched against part files; multi-VMA per part is normal.
+3. `sudo perf record -e ibs_op// -c 50000 --data -o out.perf -p <pid> --
+   sleep <dur>` WHILE a mixed workload runs (idle perf numbers mean nothing).
+   `sudo perf script -i out.perf -F addr` for addresses.
+4. Attribute: bisect VMAs, `foff = pgoff + (addr − vma_start)`, bisect tensor
+   starts; per-expert index = `int(lo / (span / N_EXPERTS))` (expert-major
+   rows).
+5. Verify before believing histograms: gap ≈ 0, every tensor-bearing part
+   nonzero, and MoE blocks should be FLAT by construction (top-k fires every
+   block every token) — a non-flat block histogram means mis-attribution
+   first. Established law: expert routing is per-block flat, per-EXPERT
+   skewed.
+
+## 8. Prefix-cache-aware routing (multi-replica serving)
+
+Measure with a **sequential-turn client**, not only a batch benchmark — a
+batch of distinct prompts parallelises prefill across replicas and hides the
+problem. The two failure modes: round-robin loses every other cache hit
+(90k-token contexts: warm-turn TTFT 2676 ms round-robin vs 1096 ms pinned);
+pinning-everything loses a card (16 requests: 18.6 s spread vs 30.1 s
+pinned). **The rule that gets both: route by request shape** — a request
+carrying more than one message is a continuation (agent loop re-sending its
+context) → pin to the replica that served that prefix; anything else → spread
+with a COUNTER, not a prefix hash. Add failover: demote a replica on
+connection failure and retry on a live one BEFORE writing anything to the
+client (safe pre-stream; never retry after streaming started).
+
+The wall no policy fixes: **a running sequence and its cached prefix draw on
+the SAME pool.** Three concurrent 50–80k families against 122,636 tokens/card
+thrash under every placement and admission policy tried (counter placement,
+load-balanced placement, one-large-request-per-card admission, both chunk
+directions — the last COSTS KV in both directions). When the working set
+exceeds aggregate KV, say so: the levers are capacity, fewer concurrent
+large contexts, or a different platform. Traps: a lifetime hit-rate counter
+is not a per-request verdict (probe directly: same prompt twice to an idle
+replica — 22.3 s cold → 1.4 s warm); salt corpora; record sample sizes;
+`vllm bench serve` needs `--tokenizer <local path>`; verify metric names
+against the live endpoint before dashboards. Test shape: drive the routing
+function directly with synthetic bodies (stdlib-only policy module);
+conversation identity is the FIRST N KiB of prompt — a shorter tested prefix
+hashes differently per turn and pinning silently never happens.
+
+## Related
+
+- [[LLM Bench Windows and Measurement]] — the window discipline and usage-counter law
+- [[PCIe Link and Atomics]] — link probing and the streaming-vs-compute ceiling
+- [[RDNA4 MoE Expert Offload VRAM Budget]] — the plan behind duty-cycle terms
+- [[vLLM Porting and Compile Triage]] — the compile fix for the pointwise-glue bound
@@ -0,0 +1,274 @@
+---
+type: reference
+title: Llama Cpp Multi-Device Splits
+description: What llama.cpp can and cannot express across devices, the no-collective layer-split path, split balancing by measurement, drafter KV sizing from GGUF headers, and heterogeneous placement triage.
+tags: [llama.cpp, multi-gpu, layer-split, pcie, sizing, speculative]
+timestamp: 2026-10-05T00:00:00-07:00
+---
+
+# Llama Cpp Multi-Device Splits
+
+## 1. Expressibility law — classify the arrangement before running anything
+
+llama.cpp expresses exactly two things across devices, and they are not
+interchangeable:
+
+| arrangement | how | expressible? |
+|---|---|---|
+| **Layer split** — layers 0..n on dev A, rest on dev B | `-sm layer -ts <a>/<b>` | **yes**, no collectives needed; works cross-vendor (CUDA + Vulkan measured 2026-09-16) |
+| **Per-tensor placement** — attention+KV on A, experts on B | `-ot <regex>=<dev>` | **NO.** `--override-tensor` sets `has_tensor_overrides()`, a REQUIRED term of the scheduler's `pipeline_parallel` flag (only read site: `llama-context.cpp`, in the `pipeline_parallel = …` expression). That flag sets `n_copies = parallel ? MAX : 1` in `ggml_backend_sched_new`; with `n_copies == 1` a pre-allocated weight on a non-primary device has no backend and the graph aborts at `ggml-backend.cpp:941` (`pre-allocated tensor (blk.0.ffn_down_exps.weight) in a buffer (Vulkan1) that cannot run the operation (NONE)` — `(NONE)` is the leaf's own op name). |
+
+Failure modes observed (mainline `37b53fd`): per-tensor `-ot exps=Vulkan1`
+with `-dev CUDA0,Vulkan1` → scheduler abort; layer-scoped `-ot` with a
+cross-backend pair → model load fails with only `failed to load model`;
+`-ot x=CPU` **works** because the CPU backend is last in the list — **the CPU
+is the only legitimate partner in a heterogeneous split.** On a CUDA+AMD host
+a card can be the trunk OR the offload host, never the complement of an
+AMD-resident peer. Pure Vulkan changes nothing here — the gate is the
+override, not the driver mix.
+
+If the plan needs tensors on a specific device: stop, the answer is no for
+any vendor mix. Expressible alternatives, in order: (1) two independent
+endpoints, one model per device; (2) single-vendor layer split; (3) device +
+CPU per-tensor offload (mainline pinning guard: `GGML_CUDA_NO_PINNED`; ik fork
+was `GGML_CUDA_NO_PINNED_WEIGHTS` — without it a CPU-overridden tensor can
+land in a pinned `CUDA_Host` buffer and page-lock ~108 GiB).
+
+Layer-split arithmetic to check before promising: a whole layer carries its
+experts (MoE) — compute GiB/layer, not per param-group; only some layers carry
+KV (every 4th on the hybrid), so a share of layers is only a share of KV; big
+non-repeating tensors (PLE tables) land on the main device and cannot move —
+they cap how small the primary can get.
+
+## 2. No collectives? The atomics wall and the layer-split answer
+
+Decide with one line before designing:
+
+```bash
+sudo lspci -vv -s <card2> | grep AtomicOpsCtl   # ReqEn- ⇒ no collectives, ever
+lspci -tv                                        # a Routing- bridge is a hardware wall
+```
+
+`ReqEn-` kills vLLM entirely — TP needs a per-layer allreduce AND PP needs a
+device-backend `send`/`recv` of the hidden state
+(`vllm/distributed/parallel_state.py`) — i.e. the same RCCL primitive that
+fails on an atomics-less card. llama.cpp's `--split-mode layer` needs no
+collective: layers distribute in order, only the boundary activation crosses
+PCIe once per forward pass. Full probe/repair detail:
+[[PCIe Link and Atomics]].
+
+## 3. Build: pin by feature, bundle the runtime at the exact depth
+
+- Pin the ref by FEATURE list (e.g. DFlash2 + qwen35 MTP on 2026-09-16 →
+  `ggml-org/llama.cpp#27342`); never float. Verify a fork build the same way
+  (`ls <venv>/…/spec_decode/ # dflash/ vs dflash2/`).
+- `-DCMAKE_HIP_ARCHITECTURES=<gfx only>` — a sibling arch links, loads, then
+  segfaults. `-DGGML_HIP_RCCL=OFF` when there is no atomics path (only tensor
+  mode degrades to the butterfly fallback; layer split unaffected).
+- **TRAP — `cp -r SRC DST/` flattens when `DST` does not exist** and drops one
+  directory level. rocBLAS/hipBLASLt look for device code at a FIXED depth
+  (`../lib/rocblas/library`, `../lib/hipblaslt/library/<arch>`); flattened,
+  the engine loads every GiB, answers `/health`, and dies on the FIRST request
+  with `Cannot read .../TensileLibrary.dat: No such file or directory for GPU
+  arch`. `mkdir -p DST && cp -r SRC/. DST/`, then
+  `patchelf --set-rpath '$ORIGIN/../lib' build/bin/llama-*`.
+- Dual-backend build (CUDA + Vulkan is the practical pair; HIP+CUDA do not
+  coexist): `-DGGML_CUDA=ON -DGGML_VULKAN=ON` in one tree; Vulkan needs
+  `glslc` + libvulkan headers. **Device names are not what you expect:**
+  `--list-devices` lists the discrete card under BOTH backends (`CUDA0` AND
+  `Vulkan0`) and the APU LAST (`Vulkan1`) — a wrong `-ot` name makes
+  llama.cpp print the valid set and exit (self-diagnosing).
+- What the existing build can do: `GPU_BACKEND_*=OFF` lines in its
+  CMakeCache.
+
+## 4. Serve flags that matter on a split
+
+```bash
+HIP_VISIBLE_DEVICES=0,1 llama-server \
+  --device ROCm0,ROCm1 --split-mode layer --tensor-split 1,1 \
+  --gpu-layers 999 --fit off --flash-attn on \
+  --ctx-size <n> --parallel 1 --cache-type-k f16 --cache-type-v f16 \
+  --jinja --metrics --cache-prompt \
+  --spec-type draft-dflash --spec-draft-model <dflash2.gguf> \
+  --spec-draft-device ROCm1 --spec-draft-n-max 7
+```
+
+- `--fit off` + explicit `--ctx-size`: auto-fit must not resize a context set
+  on purpose. **`--parallel N` DIVIDES `--ctx-size`** — one slot at large ctx
+  is the concurrency policy, concurrent agent loops queue.
+- Pin the drafter to the card holding the target's LAST layers
+  (`--spec-draft-device`); under tensor mode an unpinned draft joins the
+  meta-device KV allocation and aborts the KV build. `--spec-draft-n-max` must
+  be ≤ the drafter's `block_size - 1` (DFlash2: 8 ⇒ 7).
+- Qwen3.8-family templates RAISE on `reasoning_effort=high` (500) — probe
+  low/medium/high/xhigh and map harness levels before declaring ready.
+
+## 5. Split balancing: measure, never interpolate
+
+**A `-ts` step is ONE LAYER, and hybrid stacks do not have uniform layers.**
+qwen35 is 16 × (3 Gated DeltaNet + 1 Gated Attention): one step ≈ 0.43 GiB on
+a DeltaNet boundary, ≈ 1.43 GiB on a full-attention one (weights + that
+layer's own f16 KV) — measured on frank-medium. The gap can jump 2.9 GiB in a
+single step; the best integer split can sit 0.2 GiB from balance while its
+neighbours are 1+ GiB off. An affine fit reproduces the deployed point and
+overstates gaps in between (`tests/frank_medium_unit.bats` encodes the fit as
+a gate; treat a NEGATIVE fitted constant as a unit bug).
+
+Knobs that move weight and REQUIRE a re-measure: `--ubatch-size` (compute
+buffers are **~2x heavier on the output/drafter card** — 512→1536 added
+~3.3 GiB there vs ~1.7 GiB elsewhere), a drafter swap (weights + draft KV),
+`--ctx-size`/`--parallel` (KV). Total free VRAM is invariant across splits —
+only the distribution moves; use that to sanity-check a suspicious row.
+
+**Procedure (one split per load):**
+
+```bash
+WORK=/tmp/split.env; cp scripts/serve-<svc>.env $WORK
+export FRANK_MEDIUM_ENV_FILE=$WORK
+for ts in 39,26 38,27 37,28 36,29 35,30; do
+  sed -i "s|^TENSOR_SPLIT=.*|TENSOR_SPLIT=$ts|" $WORK
+  nohup bash scripts/serve-<svc>.sh >/tmp/split.log 2>&1 &
+  # poll /health (or process exit), then:
+  rocm-smi --showmeminfo vram | grep 'VRAM Total Used'   # sample ~6 s after health
+  pkill -x llama-server; sleep 6
+done
+```
+
+Load → sample → kill, NO requests sent (allocation only). Run under `nohup`
+and poll from a separate call (a load is 30–90 s). Pick the smallest
+inter-card gap that clears the per-card floor with room to spare; one `-ts`
+step ≈ 1.5 GiB at ~6k batch / 262k ctx on 30 GiB cards — enough to bracket,
+not to solve. Prefer balance: the busy card's headroom bounds the context
+(`ctx_extra ≈ (floor − measured_free) / per-position cost on the busier
+card`). A faulted split (`SDMA0 UTCL2 page fault` → `illegal memory access` →
+`ggml_abort`) is OFF-LIMITS — re-measure, never re-tune; more headroom does
+not prevent a boundary-copy fault.
+
+## 6. Drafter KV: sized from the TARGET, bounded by its window policy
+
+`common/speculative.cpp` builds the draft context with
+`cparams.n_ctx = llama_n_ctx(ctx_tgt)` — **no flag caps it.** So:
+
+```
+draft KV = ctx_tgt × block_count × 2 (K,V) × kv_heads × key_length × dtype
+```
+
+…unless the drafter declares a sliding window. Measured at 262,144 target
+ctx: DFlash2 (`sliding_window = 2048`) ≈ 40 MiB; a same-shape 5-layer drafter
+with NO window = 20 KiB/position = **5.0 GiB** — a four-layer swing in the
+balance point, and swapping them invalidated an already-balanced split.
+
+**Failure signature:** the engine loads the whole model, answers `/health`,
+and dies on the FIRST draft decode with `ROCm error: out of memory` /
+`cudaMalloc failed` inside `common_speculative_impl_draft_dflash::process` —
+a sizing failure, not a broken drafter; under a self-healing unit it presents
+as a crash-loop. Read `sliding_window`, `block_count`, `head_count_kv`,
+`key_length` from the drafter's GGUF header first. A compact reader (the
+three format traps: no KV alignment — v3 alignment applies only to the tensor
+section; arrays are `type(9) → element_type(u32) → count(u64)`, element type
+FIRST; the enum is `7=BOOL 9=ARRAY`, 64-bit = 10/11/12, and 8=STRING/9=ARRAY
+is the v2/v3 file order, NOT gguf.h's 2/3):
+
+```python
+import struct, sys
+def rd(path):
+    f = open(path, 'rb'); assert f.read(4) == b'GGUF'
+    struct.unpack('<I', f.read(4)); struct.unpack('<Q', f.read(8))
+    nk = struct.unpack('<Q', f.read(8))[0]
+    def s():
+        n = struct.unpack('<Q', f.read(8))[0]; return f.read(n).decode('utf-8', 'replace')
+    SCALAR = {0:1,1:1,2:2,3:2,4:4,5:4,6:4,7:1,10:8,11:8,12:8}
+    FMT = {0:'<B',1:'<b',2:'<H',3:'<h',4:'<I',5:'<i',6:'<f',7:'<?',10:'<Q',11:'<q',12:'<d'}
+    keys = {}
+    for _ in range(nk):
+        k = s(); t = struct.unpack('<I', f.read(4))[0]
+        if t == 8: keys[k] = s()
+        elif t in SCALAR: keys[k] = struct.unpack(FMT[t], f.read(SCALAR[t]))[0]
+        else:
+            et = struct.unpack('<I', f.read(4))[0]; n = struct.unpack('<Q', f.read(8))[0]
+            if et == 8: keys[k] = [s() for _ in range(n)]
+            elif et in SCALAR: keys[k] = ('arr', et, n); f.read(n * SCALAR[et])
+            else: raise Exception('unsupported arr elem type %d' % et)
+    return keys
+```
+
+## 7. Heterogeneous placement triage (go/no-go with arithmetic)
+
+- **Link:** `LnkCap` vs `LnkSta` at every hop; expect **idle-downshift** (a
+  Gen4 x4 dock idles at Gen1 x4, ~1 GB/s) — measure under load. Per-hop cost
+  vs hidden state (5120-dim f16 = 10 KiB/hop ≈ 1.3 µs at Gen4 x4): bytes are
+  almost never the constraint; **hop count is** — every boundary is a host
+  sync, and a per-op-split runtime can hit 100 splits/token at ~5 % SM. Read
+  `graph splits` from the loader log.
+- **KV is usually the bottleneck:** `KV/token = kv_bytes_per_position ×
+  context` (attention reads the whole cache every token). Highest-leverage
+  placement: **KV on the fastest-memory device together with its attention
+  compute** — never KV alone over the link (GBs over 7.88 GB/s = hundreds of
+  ms/token). Check fit: `kv_positions × slots × bytes + attention_trunk ≤
+  VRAM`.
+- **What placement never buys:** prefill is compute-bound and expert weights
+  are read once per batch — moving them buys nothing and adds a serial stage.
+  Calibration: Strix Halo + 3090, moving 12.5 % of expert mass to the GPU
+  gave decode +12 % but prefill 192 → 32 tok/s. The harmonic
+  `1/(f_big/BW_slow + f_fast/BW_fast)` reproduces the decode gain within ~2
+  points (assumes sequential stages — true at batch 1).
+- **UMA check before assuming additivity:** `mem_info_vram_total` (BIOS
+  carve-out) vs `mem_info_gtt_total` (aperture onto SYSTEM RAM). A big GTT
+  window means the "GPU memory" IS the same LPDDR: adding a stage there adds
+  a compute engine, not capacity/bandwidth. Only discrete VRAM is additive.
+- **Speculation as the "cheap" split:** bandwidth-free (token ids per round)
+  but needs the exact tokenizer for a plain drafter or a head trained for
+  that target; a plain drafter costs K sequential forwards (break-even needs
+  p ≈ 0.7+ at r≈0.4–0.5, K=3); compare against the INCUMBENT (a target
+  already committing ~1.7 tok/round on its own MTP head is a high bar); and
+  check the runtime accepts external drafters at all (closed engines 400
+  unknown drafter names by design). Details: [[DFlash DSpark Speculative Decoding]].
+- **Hazards:** keep paged/sidecar tables where they are read from; keep mmap
+  ON for huge-embedding models (anonymous memory → cgroup kill); CUDA graph
+  capture can abort on unusual graph shapes; make the FIRST measurement a
+  single-device baseline of the runtime on the new device — a split's
+  arithmetic is worthless if the baseline loses to the incumbent; a split
+  consumes the fast device entirely — check what it was serving.
+
+**Verdict ranking:** (1) two independent endpoints — often the real answer;
+(2) one process, weights-resident, KV co-located with attention on the fast
+pool; (3) hot-expert cache only if `misses × bytes × tok/s` fits the link and
+overlaps — no mainstream runtime implements it; (4) name what does NOT help:
+TP across unequal devices, KV on the far device, loading over the link,
+prefill-motivated placements.
+
+## 8. Harness syntax and verification
+
+`llama-bench` separates a LIST with `/` and SWEEP values with `,`:
+`-dev CUDA0,Vulkan1` is TWO single-device runs, never one two-device run;
+`-ts 0.85,0.15` is two sweep values each becoming `[0.85, 0, …]` — device 0
+takes everything. Write `-dev A/B -ts x/y`. Read the table's own `dev` column;
+any placement-dependent number needs an independent memory check
+(`nvidia-smi` sampling while loaded). (This artifact produced a wrong fleet
+conclusion 2026-09-16 — "CUDA + Vulkan cannot serve one model" — with two
+bogus OOMs that were really all-111-GiB-on-the-24-GiB-card.)
+
+Verification battery: `/health`; both cards' memory ≈ half the weights each
+(busier card also carries the drafter; compare `model_bytes/2 + drafter +
+ctx×KV/2 + buffers`); raise ctx by N and watch BOTH cards rise by
+N·bytes/2 — neither card's total may equal the model file (that arithmetic
+rules out replication); `spec_decode_num_drafts_total` > 0 and a real
+completion; a deep prompt (140k) proves a raised window is REAL, not
+declared; repeat prompt → `timings.prompt_n` collapses (prefix cache live);
+`/slots` shows expected `(id, n_ctx)` pairs; two concurrent requests with
+`journalctl -o cat | grep 'slot print_timing'` proves drafting on BOTH slots
+(default journal output truncates these lines).
+
+Ops traps: a server started over ssh outlives stopping the runner (keeps the
+port — `pkill -x <binary>` first); `systemctl start` of a root unit needs
+sudo (polkit "Access denied" otherwise); old crash `ROCm error` lines live in
+the journal forever — check `ActiveEnterTimestamp`/`NRestarts` before
+treating a grep count as a live fault.
+
+## Related
+
+- [[PCIe Link and Atomics]] — the atomics wall, link probing, repair
+- [[Llama Cpp Serving Sizing]] — slot-pool and KV pricing
+- [[DFlash DSpark Speculative Decoding]] — spec types, chaining, A/B
+- [[frank-medium-llamacpp-dflash2]] — the layer-split era runbook
+- [[Dual-R9700-Cooperation]] / [[x4-pair-experiment]] — the measured history
@@ -0,0 +1,163 @@
+---
+type: reference
+title: Llama Cpp Serving Sizing
+description: Slot-pool sizing against a headroom band, per-device VRAM accounting, prompt-cache/state sizing and the QSA flash-attn fix — the measurement-first procedures for llama.cpp serving shapes.
+tags: [llama.cpp, vram, kv-cache, slots, flash-attn, sizing]
+timestamp: 2026-10-05T00:00:00-07:00
+---
+
+# Llama Cpp Serving Sizing
+
+## 1. Slot-pool sizing against a headroom band
+
+Goal: choose `CTX_SIZE`/`PARALLEL`/KV types so N conversations fit a GPU while
+keeping a requested amount of VRAM free — from measurement, never arithmetic
+guesswork.
+
+1. **Fix the invariants.** Read the payload's KV geometry (attention layers
+   only, KV heads, head_dim, KV dtype bytes) and the per-slot rule
+   `n_ctx_slot = CTX_SIZE / PARALLEL`. A hybrid model with
+   `kv_unified = 'false'` gives each slot its own cache AND a fixed per-slot
+   state buffer — slot count is a memory term, not free.
+2. **Two boots on the real card** (production stopped, per the bench law),
+   each near the target; record `nvidia-smi` used/total plus the boot log's
+   `n_slots`/`n_ctx_slot`: shape A = current/cheap, shape B = new demand.
+   `non-KV = used − (tokens × KiB/token)`; the two rows separate
+   weights/buffers/graphs from the new demand (a 918 MB projector bills
+   ~1.4 GiB).
+3. **Solve the band:** `KV_tokens ≤ (card − non-KV − headroom_low) /
+   KiB/token`, round the pool DOWN to a multiple of `PARALLEL × 4096` so every
+   slot gets an equal integer window. Boot and measure; if outside the band,
+   walk one step — never re-derive from a formula.
+4. **Propagate the per-slot number — it is the client contract:**
+   `models.yml` `contextWindow` = `n_ctx_slot` (NOT the model's native window),
+   or long conversations die with `400 exceed_context_size_error` instead of
+   compacting; compaction/summary budgets sized to the old window get
+   re-checked; endpoint docs and the rollback table move too.
+5. **Prove behaviour:** boot-log slot count, `nvidia-smi` free in band, N
+   concurrent requests answered in parallel with N+1 queued, and — if a
+   projector ships — a solid-color image probe through an **`image_url`
+   content part** expecting the exact color. The legacy top-level `images`
+   field is silently ignored, so a probe that uses it measures a
+   hallucination, not vision.
+
+Gotchas: shorter slots can make decode FASTER (~5 % measured); the KV pool is
+allocated whole at boot so post-boot `memory.used` IS the ceiling; record both
+measured rows in the service `.env` comment block and mirror the non-KV price
+in the bats budget gate so a shape that busts the band fails the suite instead
+of the boot.
+
+## 2. Per-device VRAM accounting ("where did the memory go?")
+
+First decide which kind of memory is in question:
+
+| device | kind | read it with |
+|---|---|---|
+| RTX 3090 (Dankclaw) | discrete GDDR6X | `nvidia-smi --query-gpu=memory.total,memory.used,memory.free` |
+| gfx1151 iGPU (Dankclaw) | **UMA** — "VRAM" is host RAM | `/sys/class/drm/card1/device/mem_info_gtt_{total,used}` (114 GiB) vs `mem_info_vram_*` (2 GiB carve-out) |
+| R9700s (Dankserver) | discrete | `rocm-smi` figure (64 GiB, SVM-backed) |
+
+Per-process truth on NVIDIA:
+`nvidia-smi --query-compute-apps=pid,process_name,used_memory`. Never add the
+iGPU's GTT to host RAM as a separate pool; `free -g` is the arbiter for
+co-tenants.
+
+- **Engine ledger first.** Halogen prints weights pinned (+ remaining 2 MiB
+  contiguous blocks — the honest fragmentation signal: 240 blocks/12.3 GiB
+  after reboot vs 475/14.7 GiB before means the model did not come back into
+  the same hole), device budget, KV pool/working split, "left for everything
+  else". llama.cpp prints at `llama-model.cpp:1827` (per-backend model buffer)
+  and `llama-context.cpp:692` (compute buffer) — but **`llama-bench`
+  suppresses loader INFO and a systemd unit did not put them in the journal**
+  (verified `grep -c` = 0). Recipe: one launch through `llama-cli`/`llama-server`
+  keeping the first ~200 lines.
+- **Otherwise compose from tensor sizes.** Group the GGUF tensor table by
+  role (Flash-Next example: 75.0 GiB experts at 1.5625 GiB/layer, 32.8 GiB PLE
+  table, 3.2 GiB everything else), KV ≈ 2 KiB/position per f16 attention
+  layer, × derived context. Vendor delta is real: same layers+KV cost 12.80
+  GiB under Vulkan vs 14.99 GiB under CUDA (~3.3 vs ~1.1 GiB of
+  graph/compute pool).
+- **Traps:** mmap'd PLE is reclaimable page cache; `-ot` makes it pinned —
+  "move layers to the card" converts reclaimable cache into device memory and
+  frees nothing for co-tenants. A served model's footprint ≠ weight size
+  (frank-secondary: 22.6 of 24.1 GiB for an 11.29 GiB model). Do the capacity
+  subtraction before designing: 12 attn layers × 1.62 GiB + 200k f16 KV =
+  28.6 GiB > 24.1 GiB usable → KV-in-VRAM on that pair only reaches ~64k.
+- **Context lengths: derive, never remember.** `llama-bench` computes
+  `n_ctx = n_prompt + n_gen + n_depth` (defaults 512/128/0, type_k/v F16,
+  flash_attn AUTO, split_mode LAYER). `-p 512,16384 -n 64` = contexts
+  576/16,448; a plain `tg` row has ~zero depth — `-d` is the only thing that
+  puts KV in the picture.
+- **Label every number** measured / derived / inference, and state the context
+  each figure was taken at — 12.8 GiB at 65,568 tokens is not the same claim
+  as at 200k.
+
+## 3. Prompt cache and state sizing
+
+`--cache-ram` (MiB, default 8192) is HOST RAM for saved prompt states —
+unpinned, on demand, moves no VRAM. `--ctx-checkpoints` (default 32) is a
+different mechanism: per-slot KV snapshots; `0` disables them.
+
+**State law** (compute before trusting any cap): a state is the serialized KV
+of the whole sequence + constant recurrent state + the draft context's KV:
+`per_token = 2 × n_full_attn_layers × n_kv_heads × head_dim × bytes (K and V) +
+draft_per_token`. Measured on frank-medium (qwen3.5 hybrid, f16, MTP draft):
+16×4×256×2×2 = 64 KiB/token + 4 KiB draft = **68 KiB/token**; a full
+196,608-token slot state ≈ 12.8 GiB. Cross-check against the journal's
+`prompt state size` lines — the two must agree.
+
+**The journal is the instrument:** `exceeds cache size limit … skipping` =
+cap binding in production; `making room … removing oldest` / `cache size
+limit reached` = eviction, and the printed size proves the active cap. Do NOT
+claim a raise is proven by `prompt_tokens_cached_total` rising — on exact
+repeats that counter is 0 by construction.
+
+**Hybrid + zero-checkpoint trap:** for a hybrid (DeltaNet/SSM + attention)
+model, `seq_pos_min` sits at the END of the slot's own prompt (a recurrent
+cell stores only the last position), so with `--ctx-checkpoints 0` there is no
+checkpoint to fall back on and the server logs `forcing full prompt
+re-processing … (likely due to SWA or hybrid/recurrent memory)`. Consequences:
+intra-slot reuse only survives when the request echoes the assistant's
+previous reply verbatim (a pure user resend is always a full re-prefill even
+at f_sim 1.000); the RAM prompt cache becomes the ONLY carrier of a
+conversation state ACROSS slots, so a cap that refuses deep states costs
+minutes of re-prefill whenever slots are reassigned; state restoration only
+runs via the LRU path (or LCP with f_keep < 0.5) — a plain prefix hit never
+calls `prompt_load`.
+
+**Verify a change:** live argv carries the flag (not the deploy log); `/health`
+200 + real completion; VRAM unchanged and host RAM mostly free; no new
+`exceeds cache size limit` lines, with a `making room` entry larger than the
+old cap as direct proof. If the ceiling itself was not exercised live, say so —
+crossing it needs a >120k-token conversation and must not run beside a live
+service (bench law).
+
+## 4. QSA flash-attn assert fix (qwen4exp)
+
+`--flash-attn on` aborts graph build for qwen4exp models:
+`GGML_ASSERT(a->type == GGML_TYPE_F32)` at `ggml.c:5371` through
+`build_attn_qsa`. Root cause: `build_attn_qsa` calls `ggml_fill` on the
+F16-cast kq mask returned by `get_kq_mask()` when flash attn is on, but
+`ggml_fill` requires F32.
+
+Fix (mirrors the MLA sparse path in `llm_graph_context::build_attn`): run the
+top-k mask surgery on the F32 base mask (`inp->self_kq_mask`), cast to F16
+only at the attention call when `flash_attn` is enabled
+(`kq_mask_attn = cparams.flash_attn ? ggml_cast(ctx0, kq_mask_top_k,
+GGML_TYPE_F16) : kq_mask_top_k`). The patch lives at
+`scripts/patches/qwen4exp-qsa-f32-mask.patch`.
+
+Only the charlie12345/ROCmFPX build-qwen38 (PR #27742 qwen4exp merged) takes
+it; CIRU, upstream and stock builds do not. Validate live: load with
+`--flash-attn on`, health 200, a real completion finishing `"stop"`, journal
+clean of `GGML_ASSERT`/`cudaMalloc`/`missing tensor`, and flash-attn engaged
+in the timings. Context cap for full-model residency on that shape stayed
+32,768.
+
+## Related
+
+- [[GGUF Compatibility]] — what a payload is before it is priced
+- [[frank-secondary-llama-mimo-q6k-2x200704]] — the current two-boot priced shape
+- [[frank-medium-llamacpp-dflash2]] — where the 68 KiB/token datum was measured
+- [[DFlash DSpark Speculative Decoding]] — the spec A/B protocol on llama.cpp
+- [[Dual-R9700-Cooperation]] / [[x4-pair-experiment]] — the multi-device side
@@ -0,0 +1,281 @@
+---
+type: reference
+title: Model Swap and Rollback
+description: The full model-swap discipline for the LLM fleet — fit preflight, fetch and payload verification, runtime build/flag checks, git-only config, live validation, systemd handoff, close-out, the three-pin revert law, and research-verification discipline.
+tags: [models, swap, gguf, huggingface, rollback, deployment, verification]
+timestamp: 2026-10-05T00:00:00-07:00
+---
+
+# Model Swap and Rollback
+
+Applies to every endpoint swap on the fleet (frank-secondary / frank-large /
+frank-primary). Repo law: all config flows through git; the live models are
+host state under `/home/dank/models/<name>/`.
+
+## 1. Fit preflight — never deploy a swap without confirming it FITS
+
+1. Get the checkpoint's on-disk/weight size AND precision from the HF card
+   (`Precision: BF16` = 2 B/param; FP8/Q8 ≈ 1 B; Q4 ≈ 0.5 B). 35B params:
+   BF16 ≈ 66–70 GB, FP8/Q8 ≈ 33–35 GB, Q4 ≈ 17–20 GB.
+2. Compare against the HOST'S usable RAM (not just VRAM) for runtimes that
+   stream weights from RAM (host-RAM expert offload, llama.cpp CPU offload,
+   layer splits). Budget weights + KV pool + OS + page cache; **if weights
+   alone exceed RAM, ABORT** — pool tuning shrinks KV, it does not shrink
+   weights.
+3. **The failure mode is a self-sustaining thrash, not a crash:** a large
+   swap makes the kernel PAGE instead of OOM-killing → sshd/user space
+   starved indefinitely, no recovery window opens by itself; systemd
+   `Restart=always` turns it into a crash-reload-rethrash loop; `enabled` +
+   `WantedBy=multi-user.target` re-thrashes on every reboot. A power cycle
+   does NOT fix it.
+
+Recovery (may need console / SysRq-to-GRUB):
+
+```bash
+sudo systemctl mask --now <svc>                       # break the reboot trap FIRST
+cd <checkout> && git fetch origin && git reset --hard origin/main
+grep MODEL_DIR <env>                                  # confirm the fixed target
+sudo systemctl unmask <svc> && sudo systemctl restart <svc>
+curl -s <health>                                      # then a REAL completion
+```
+
+Deploy caveat: "healthy after Ns" = systemd ACTIVE, not model LOADED —
+confirm with a real `/v1/chat/completions`. RTX 3090 (Ampere) has no
+FP8/FP4 tensor cores: FP8 weights run via upcast but must still fit; choose
+quant by size fit, then accept the compute path the card supports. Before
+choosing a smaller quant, state WHICH of the two you are buying — headroom,
+or a design change (a smaller quant does not change what the runtime can
+express; per-tensor cross-device placement stays refused under
+`--override-tensor`, and a layer split moves only contiguous layer ranges).
+
+## 2. Ground truth first
+
+- Read the HF model card (runtime pin, hard constraints, recommended args)
+  + the repo trio `scripts/<svc>.env`, `scripts/serve-<svc>.sh`,
+  `scripts/<svc>.service` + the runbook under `serverAppConfigs-docs/LLM/`.
+- Check the host before downloading anything: `ls /home/dank/models/`,
+  `df -h /home`, `du -sh` — bench-era or prior-deployment files often already
+  exist.
+
+## 3. Fetch and verify (HF payloads)
+
+**Inventory by QUANT NAME, never by extension** — large quants ship as
+`.partNofM` fragments (Q4_K_M = 37.00+37.00+36.97 GiB), so an
+`endswith(".gguf")` filter makes Q4_K_S look nonexistent. Check both
+mradermacher repos (`…-i1-GGUF` imatrix, `…-GGUF` plain) and the model's own
+repo; the tree API's `blobs=true` gives `lfs.oid` per file. Gated mirrors
+return 403 even with a valid token until the account accepts the gate —
+prefer the ungated copy.
+
+```bash
+# token out of band: mode-600 curl config; a token passed as -H is world-readable
+printf 'header = "Authorization: Bearer %s"\n' "$(cat ~/.cache/huggingface/token)" > /tmp/hf.curlrc
+chmod 600 /tmp/hf.curlrc
+curl -fL -C - --retry 10 --retry-delay 5 --retry-max-time 21600 -K /tmp/hf.curlrc -o "$DEST/$name" "$BASE/$name"
+```
+
+`-f` is MANDATORY: `curl -o FILE URL` **exits 0 on HTTP 401/403** and writes
+the error body as the artifact (identical 153-byte shards, cheerful DONE,
+zero bytes of model). Never `pgrep -a` a running fetch (token is in
+`/proc/<pid>/cmdline`); watch progress by size:
+`while [ "$(stat -c%s … || echo 0)" -lt "$EXPECTED" ]; do sleep 30; done`.
+
+**Verify, then assemble:** per part, byte count == Hub's declared size AND
+`sha256sum` == `lfs.oid`; concatenate in NUMERIC order; assembled total ==
+sum of parts (compute the sum with python, not by hand) and `head -c 4` ==
+`GGUF`. The assembled file has no published hash — the per-part oids + exact
+total are the chain. Keep the fragments when disk allows (only copies
+re-verifiable against the Hub); failure artifacts and the replaced payload
+go to `~/.AGENTTRASH/YYYYMMDD-topic`, never `rm` — upstreams DELIST quants
+and the local file may be the last copy.
+
+Fetch-script one-liners: dedicated scripts (`scripts/download-*.sh`) already
+do `-f` + byte-count + LFS sha256 + `--verify-only`; ad-hoc fetches tend to
+skip all three. HF cache snapshots SYMLINK every file into `blobs/` and
+those links dangle inside containers — serve scripts and the deployer refuse
+symlinks and missing tokenizer files for exactly this reason.
+
+## 4. Runtime build (when the card pins a fork/build)
+
+- Clone + build with the card's exact cmake line, plus
+  `-DLLAMA_BUILD_WEBUI=OFF` — WebUI asset provisioning (npm / HF llama-ui
+  bucket) aborts llama.cpp link at 98 % on these hosts; the API server is
+  unaffected.
+- Verify every flag the serve script passes exists in the new binary BEFORE
+  touching the service: `llama-server --help | grep -q -- "<flag>"` for each;
+  list MISSes. Read the GGUF's tensor histogram first if the payload uses a
+  custom type — the FORMAT picks the runtime ([[GGUF Compatibility]]).
+
+## 5. Config via git only
+
+Edit `scripts/<svc>.env` (paths, LLAMA_BIN, MODEL_FILE, MMPROJ_PATH) + serve
+script header/defaults locally; `bash -n`; `./tests/run.sh` must pass;
+commit; push; `git pull --ff-only` on the target. Never edit configs on the
+remote. Renaming a model dir (`mv`) is host state and allowed. The env-file
+hook pattern (`FRANK_*_ENV_FILE`) is what makes bench windows and staged
+arms possible — keep it intact.
+
+## 6. Live validation
+
+- No scratch ports — one GPU cannot host two big models. Ensure the old unit
+  is truly gone first: `systemctl is-active` can hang in `stop-sigterm`
+  (llama.cpp never preempts an in-flight generation); poll until the PID
+  exits.
+- Manual serve for smoke: `setsid nohup bash scripts/serve-<svc>.sh >
+  /tmp/x.log 2>&1 &` — plain `ssh 'nohup … &'` dies with its ssh session.
+- Verify: health 200; a REAL `/v1/chat/completions`; reasoning flows
+  (`reasoning_content` non-empty); if the endpoint takes images, a
+  solid-color probe through an `image_url` content part (legacy top-level
+  `images` is silently ignored; probe with ≥256 px).
+- vLLM specifics: first request after boot can take ~18 s (deferred Triton
+  loads) — warm and discard; `--reasoning-parser` / tool parsers are part of
+  the contract ([[vLLM Boot Forensics and Config Interrogation]]).
+
+## 7. systemd handoff — THE TRAP
+
+`./scripts/deploy.sh <host> --service <svc>` gates health, but a leftover
+manual server holding the port makes the health check pass while the UNIT
+crash-loops on bind. Always confirm ownership:
+`systemctl show <svc> -p MainPID` equals the single `pgrep -x llama-server`
+(or the container runtime PID); only then trust "healthy". If a real
+long-context task is queued, do the handoff anyway — client aborts are
+recoverable, drift is not.
+
+## 8. Close-out (mandatory)
+
+1. Docs: active runbook updated (build recipe, profile, validation numbers
+   incl. real-workload prefill/decode); superseded runbook gets a RETIRED
+   banner and stays as the rollback record; `LLM/index.md` mermaid + active
+   list; AGENTS.md fleet table + runtime section + history chain; work-log
+   entry in `serverAppConfigs-docs/log.md`.
+2. `~/.omp/agent/models.yml`: `contextWindow` must equal the REAL per-slot
+   context — llama.cpp logs `n_ctx_slot` per prompt (`--parallel N` divides
+   the pool; e.g. 262144 native / 2 slots = 131072 per slot); vLLM reports
+   `max-model-len` and its concurrency line. Overselling causes silent
+   client-side overflow or a discovered-wall 400. Also `maxTokens` vs the
+   engine's cap, `input` capabilities (text/image), and
+   `reasoningContentField` per runtime (llama.cpp/SGLang emit
+   `reasoning_content`, vLLM emits `reasoning`). Fix every `modelRoles` entry
+   that named the old endpoint.
+3. **Rollback = one env commit** (paths + LLAMA_BIN + flags), prior binary +
+   payload kept on disk, rollback numbers recorded in the runbook.
+
+## 9. Revert-pin an ad hoc experiment (the three pins)
+
+A campaign is "done" only when all three exist — without them a
+`podman pull`, an image update, or a lost scratch script destroys the way
+back:
+
+1. **Git tag (annotated, pushed):**
+   `git tag -a <exp>-tuned-YYYYMMDD -m "<tuned env/flags + measured headline +
+   runbook path + revert command>"` — a future agent must reconstruct the
+   campaign from the tag message alone.
+2. **Image digest in the serve script:** floating tags move. Capture
+   `podman inspect <ctr> --format '{{.ImageName}} {{.Image}}'` + `podman
+   images --digests`, patch `IMAGE=` to `repo@sha256:<repodigest>` with a
+   comment (image ID, date, old ref), verify `podman image exists
+   repo@sha256:<digest>` and `bash -n`. Digest refs survive upstream pushes;
+   `podman pull repo@sha256:<digest>` re-fetches exact bytes. One host can
+   hold MULTIPLE images of one repo — pin by the digest of the RUNNING
+   container, never by tag name. Never edit the pinned line back to a
+   floating tag; new experiments branch from the digest.
+3. **Local alias:** `podman tag <id> <exp>-runtime:tuned-YYYYMMDD`.
+
+Record scratch-script paths + full flag surface in the runbook (hook env
+vars in CAPS); expired images/scripts go to `.AGENTTRASH`, never `rm`.
+Pure measurement sessions that changed no state need only pin 1.
+
+## 10. Research verification discipline (for "can model X run here?")
+
+- Scouted research summaries have shipped fabricated citations on this fleet
+  (invented HF repos, plausible-but-wrong PR numbers, stale engine versions).
+  NEVER copy a scout's citations into a deliverable — re-verify each
+  load-bearing URL directly (GitHub issues/PRs/repos, HF pages) before it
+  enters anything.
+- The in-repo runbooks are FIRST-HAND evidence and outrank any web claim
+  about our own hosts.
+- Verified anchors (2026-09-27): DeepSeek V4.1-Flash (552B/8B-16B active,
+  CED arch, 1M ctx, MIT; preview was 284B/13B with native FP4+FP8 — early
+  GGUF conversions garbage); Lucebox = the proven gfx1201 ROCm 7.2 path
+  (heterogeneous R9700+Strix-Halo V4.1: 86 tok/s decode, 788 t/s prefill@2K).
+  GLM-5.3-Flash (320B/18B, KDA+DSA+mHC, 1M ctx — ALWAYS bind `-c`, the
+  default hard-hung a 128 GB host; native FP8 328.3 GB; `-fa off` MANDATORY,
+  flash-attn breaks MLA correctness; llama.cpp arch `glm5next` lived in
+  unslothai branch → ggml PR #27754, MTP landed d07e71ede 2026-08-30; quant
+  ladder UD-IQ1_S 93.1 GB → UD-Q4_K_XL 199.7 GB, GSQ-RCO 3.5-bit −1.40 pp
+  MMLU-Pro vs Q8_0).
+- GGUF parsing gotcha: expert tensors are
+  `blk.N.ffn_{gate,up,down}_exps.weight` — 4 dot-parts INCLUDING `.weight`;
+  a `len(parts) == 3` filter matches zero layers with no parse error. Filter
+  on `parts[0]=="blk" and parts[3]=="weight" and "exps" in parts[2]`.
+
+## 11. Adding a second LLM endpoint to a host
+
+When a host must serve two models at once (an iGPU runtime plus a second
+card), or one endpoint is moving and a new one takes the freed hardware.
+
+**0. Decide the shape before touching hardware.** One provider per
+HOST:PORT in `~/.omp/agent/models.yml` (on the box where the harness runs)
+— a second endpoint is a new provider block, not a second model in the
+existing one. Read the budget as an inequality: for unified memory the
+engine's own fit check is the arbiter (`budget = host − pinned weights −
+reserve`; `need = KV + arena + vision`); for a discrete card, weight the KV
+by counting only full-attention layers — and remember `--parallel N`
+DIVIDES `--ctx-size`. A co-tenant is a budget line, not a footnote: free
+RAM by changing the POOL or the ARENA (the reserve is a fit-check term,
+never an allocation) and drop optional charges (a vision tower is ~2 GiB)
+rather than hoping.
+
+**1. Repo first, and retire in the same commit.** New service files
+(`scripts/<svc>.env`, `.service`, `serve-<svc>.sh` with hard guards for
+missing artifacts); planner entries (`SERVICE_ORDER` + the target's service
+set + path mappings, inventories in `deploy.sh`/`deploy-remote.sh`, the
+dispatch case); retire the obsoleted pieces into `.AGENTTRASH/` with a
+suffix — and check what pointed at them: bench harnesses, a chat template
+only the old runtime read, and `scripts/test-changed.sh` mappings are the
+usual victims (a mapping to a retired test file fails the pre-commit
+wrapper silently for weeks). One bats file per service asserting
+INVARIANTS (`pool == 2 × ctx`, KV bytes ≤ card budget, port separation),
+not constants; a negative grep must strip comments first
+(`grep -vE '^\s*#' file | grep -q …`) or it matches the prose explaining
+the very thing you removed.
+
+**2. The port-bind race (rootless podman).** pasta binds the published port
+at container CREATE; a just-stopped predecessor still holds it. Order the
+handler: retire the old manager's unit → stop THIS service's own unit
+(no-op on first cutover) → wait for every socket on the port to clear
+([[Container Host Reachability]]) → install → start. Without the second
+step a redeploy deadlocks against its own gate.
+
+**3. Wire the surfaces, refuse impossible signals.** Point the harness
+small/fast + image roles at the new endpoint; move the vision role WITH the
+vision tower (one owner per capability). `shared/homepage/services.json`: a
+card per endpoint, probed from the monitor's own host. Prometheus: only add
+a job if the runtime actually exposes Prometheus text (halogen publishes
+`/health`, `/cache`, `/v1/*` and nothing else — the job was REMOVED with the
+reason written down rather than left as a permanent red).
+
+**4. Cutover, then verify by driving it.** `deploy.sh <target> --service
+<large> --service <small>` (order matters); per endpoint: `/health`,
+`/v1/models` reports the alias, a chat completion returns content AND
+`reasoning_content`, a `tools` request returns `tool_calls`, an image
+request is answered (or refused with a named reason), the runtime's own
+speculation counters move, and the unit is `enabled`/`generated` with linger
+on. Read the engine's boot ledger for a silent clamp. Finally spend one
+real request through the harness per changed role — routing labels lie; the
+servers' own journals do not.
+
+Traps that look like model faults: `500 tools param requires --jinja flag`
+(runtime lacks the Jinja path — pin it); `500 Jinja Exception: Unexpected
+reasoning effort <level>` (fold via `compat.reasoningEffortMap`); the same
+request works by hand but a subagent dies (subagents run at a different
+thinking level — check both); the container dies instantly although the
+predecessor "stopped" (the port race above).
+
+## Related
+
+- [[GGUF Compatibility]] — which runtime can load a payload at all
+- [[Llama Cpp Serving Sizing]] — pricing the shape once it loads
+- [[RDNA4 MoE Expert Offload VRAM Budget]] — vLLM-side fit arithmetic
+- [[LLM Bench Windows and Measurement]] — measuring the new deployment
+- [[dankserver-qwen38-fp8-expert-offload-tuning]] — the current endpoint
@@ -0,0 +1,244 @@
+---
+type: reference
+title: PCIe Link and Atomics
+description: Link probing (idle downshift, path ceilings, CUDA/HIP bandwidth probes), the streaming-vs-compute ceiling arithmetic, and the PCIe-atomics wall that decides whether AMD multi-GPU collectives can exist at all.
+tags: [pcie, rccl, atomics, multi-gpu, bandwidth, amd, nvidia]
+timestamp: 2026-10-05T00:00:00-07:00
+---
+
+# PCIe Link and Atomics
+
+## 1. Degraded or just idle?
+
+The trap: an idle GPU reports `LnkSta: Speed 2.5GT/s (downgraded)` — the
+driver downshifts to Gen1 for power. That is not a fault and not the slot's
+fault. Never conclude from an idle reading.
+
+1. **Path ceiling from the kernel (cheapest, most decisive):**
+   `sudo journalctl -k -b --no-pager | grep "available PCIe bandwidth"` →
+   `pci 0000:c4:00.0: 63.012 Gb/s available PCIe bandwidth, limited by 16.0
+   GT/s PCIe x4 link at 0000:00:03.1 (capable of 252.048 Gb/s with ... x16)`.
+   The "limited by" device names the **bottleneck** (usually a root port) and
+   its width — that is the number a swap could move; "capable of x16" is the
+   endpoint's OWN capability, and the gap between them is the story.
+   `dmesg` rotates; `journalctl -k -b` keeps this boot.
+2. **Both sides separately:** `LnkCap`/`LnkSta`/`LnkCtl2` on endpoint AND root
+   port. `LnkCtl2: Target Link Speed: 16GT/s` on both ends = the platform
+   intends Gen4, so a Gen1 current speed is power management, not
+   misconfiguration. A root port whose `LnkCap` width exceeds the trained
+   width means the socket/adapter wires fewer lanes — **M.2 connectors are 4
+   lanes by pinout; an M.2 riser can never deliver more.**
+3. **Prove the link under load (the step people skip):** on NVIDIA sample NVML
+   while the GPU works
+   (`nvidia-smi --query-gpu=pcie.link.gen.current,pcie.link.gen.hostmax,pcie.link.width.current,pcie.link.width.max`):
+   `gen.current` climbing 1 → `hostmax` = idle downshift, healthy; stuck below
+   `hostmax` under load = real degradation; `width.current < width.max` under
+   load = **physical lane limit** — no software touches it. On a busy
+   production GPU this is a usage smoke test, not a benchmark — never stop a
+   live service to read its link.
+4. **Do not plan hardware from `dmidecode -t slot`** — consumer firmware
+   labels Ethernet/SD controllers as "M.2 slots" and advertises slots in PCI
+   domains that do not exist. Trust `lspci -nn | grep "PCI bridge"` and
+   `lspci -tv` (empty root ports still enumerate; an absent port is absent).
+5. **"Would moving the card help?"** Compare the bottleneck device from step 1
+   with the target port's `LnkCap` width; two M.2 sockets = both x4 = a
+   no-op. Clean `DevSta`/`CESta` with a low trained speed means "wired
+   narrow", not "failing". Report ceiling, card capability, what the idle
+   reading meant, and whether any physical change moves the number.
+
+Gen4 usable payload after 128b/130b: **x4 7.877 GB/s, x8 15.75, x16 31.5**.
+
+## 2. The engine shape decides whether the link matters at all
+
+| engine shape | what crosses PCIe per token | link matters? |
+|---|---|---|
+| streams weights (expert LRU, paging) | the missed weights (~GB/token) | **yes — decisive** |
+| computes offloaded weights in RAM (llama.cpp `-ot exps=CPU`) | activations only (~KB) | no; host RAM bandwidth decides |
+| everything resident | nothing | no |
+
+**Ceiling arithmetic:** `link time/token = active expert bytes / measured
+H2D rate`; `ceiling = 1 / link_time × resident-fraction saved (LRU hits)`.
+Compare against the compute-in-RAM path at host RAM bandwidth (256 GB/s
+LPDDR5X on the Strix Halo). **If the link is >10x slower than the memory
+behind it (x4 is 31x here), streaming loses by construction and no tuning
+rescues it; doubling the link only doubles a losing number.**
+
+Worked example (Dankclaw 2026-09-16): Gen4 x4 measured H2D **6.72 / D2H 6.60
+GB/s** (85 % of spec) — pinned H2D ≈ async, pageable within 1 %, i.e.
+link-limited not staging-limited. A 3 MiB slab (the expert-streaming
+granularity) = 474 µs. A3B MoE at Q4_K_M (~1.70 GiB active/token) → 272
+ms/token ⇒ **3.7 tok/s** (4.6 with 20 % resident) vs measured **20.46 tok/s**
+with experts computed on the CPU, 25–26.5 on the iGPU. The streaming design
+loses 3–5x.
+
+## 3. Measure the link: two probes, same mechanics
+
+Stop **all** production services first (fleet bench law) — the 3090's VRAM
+may be nearly full, and the probe contends for the very link it measures.
+
+**CUDA probe** (`pcie-link-test.cu`): sweep {3, 16, 64, 256 MiB, 1, 2 GiB}
+with `cudaMalloc` + `cudaHostAlloc`, event-timed; per size measure sync H2D,
+async H2D, D2H, one single transfer, and pageable H2D. `3 MiB` is the row
+that matters for expert streaming; big sizes give the ceiling.
+
+**HIP probe for RDNA4** (`pcie_bw.hip`) — the ROCm landmines each cost real
+time, so encode them:
+
+- `hipGetDeviceProperties` SEGFAULTS on some builds; use `hipGetDeviceCount`
+  + `hipSetDevice` only. First line: `setvbuf(stdout, NULL, _IONBF, 0);` — a
+  segfaulting process loses block-buffered stdout and hides the crash point.
+- Build with an explicit arch: `HIP_TARGETS=gfx1201 hipcc -O2
+  --offload-arch=gfx1201 -o p p.hip`; compile with `| grep -E "error:"`,
+  never `| tail -N` (truncation hid a missing `#include <cstring>`).
+- `hipMemset`/`hipDeviceSynchronize` are `[nodiscard]` — use or cast.
+- Mount the whole `/dev/dri` into ROCm containers; individual `renderD*`
+  nodes enumerate but fail context creation.
+- `rocm-smi --showmetrics`: `pcie_link_width` is trustworthy, but
+  `pcie_link_speed` reads the ASPM idle value and `pcie_bandwidth_acc/inst`
+  are usually `N/A` on consumer cards — live traffic generally cannot be
+  sampled.
+
+```cuda
+// pcie_bw.hip — sweep sizes, pinned vs pageable
+#include <cstdio>
+#include <cstring>
+#include <cstdlib>
+#include <hip/hip_runtime.h>
+static double one(size_t bytes, bool pinned, int iters) {
+    void *dev = NULL, *host = NULL;
+    if (hipMalloc(&dev, bytes) != hipSuccess) return -1;
+    if (pinned) { if (hipHostMalloc(&host, bytes, hipHostMallocDefault) != hipSuccess) return -1; }
+    else        { host = malloc(bytes); }
+    memset(host, 0x5a, bytes);
+    hipEvent_t a, b; hipEventCreate(&a); hipEventCreate(&b);
+    for (int i = 0; i < 3; i++) hipMemcpy(dev, host, bytes, hipMemcpyHostToDevice);
+    hipDeviceSynchronize();
+    hipEventRecord(a, 0);
+    for (int i = 0; i < iters; i++) hipMemcpy(dev, host, bytes, hipMemcpyHostToDevice);
+    hipEventRecord(b, 0); hipEventSynchronize(b);
+    float ms = 0; hipEventElapsedTime(&ms, a, b);
+    double gb = (double)bytes * iters / (ms / 1000.0) / 1e9;
+    if (!pinned) free(host); else hipHostFree(host);
+    hipFree(dev); hipEventDestroy(a); hipEventDestroy(b);
+    return gb;
+}
+int main(int argc, char **argv) {
+    setvbuf(stdout, NULL, _IONBF, 0);
+    if (hipSetDevice((argc > 1) ? atoi(argv[1]) : 0) != hipSuccess) return 1;
+    size_t sz[] = {2459136UL, 3UL<<20, 16UL<<20, 64UL<<20, 256UL<<20, 1UL<<30};
+    for (int i = 0; i < 6; i++) {
+        size_t b = sz[i];
+        int iters = (b >= (1UL<<30)) ? 10 : (b >= (256UL<<20)) ? 20 : 100;
+        printf("  %10.3f MiB | pinned %6.2f | pageable %6.2f\n",
+               b/1048576.0, one(b,true,iters), one(b,false,iters));
+    }
+    return 0;
+}
+```
+
+**Reading it:**
+
+- **Engine-vs-link is the diagnosis, not the raw number:** compare
+  `engine_effective_GB_s` to `link_GB_s` at the SAME transfer size. Engine at
+  ~40 % of the link at the same granularity = the wire is not the problem and
+  no hardware change will help.
+- Achieved GB/s is itself the proof of the trained link state (6.7 GB/s only
+  exists at Gen4 x4) — no separate under-load `LnkSta` sample needed.
+- **pinned ≈ pageable ⇒ pinning is not a lever.** async ≈ sync ⇒ link-limited
+  (the usual "pageable costs 2x" rule does not apply at this link).
+- Small-transfer penalty (2.345 MiB: ~106 µs vs ~78 µs ideal, ~21 %) is
+  per-transfer overhead — recoverable only by BATCHING transfers, not by a
+  faster link. (Dankserver 2026-09-29: bulk 22.7 GB/s at the 2.344 MiB slab =
+  91 % of Gen5 x8, while the engine's effective read path ran ~9.5 GB/s — the
+  gap is the fork's gather software path, not the wire.)
+- Both GPUs symmetric ⇒ no cross-device rebalance is available; measure both.
+- Don't quote a derived line from inside a probe without hand-checking it —
+  a printf format/argument mismatch in one of these printed a nonsense
+  ceiling. GiB→GB before dividing.
+
+## 4. The PCIe-atomics wall (AMD multi-GPU collectives)
+
+**Symptom:** multi-GPU AMD host fails TP/PP; the failure surfaces as
+`hipLaunchKernel: hipErrorIllegalState` for `ncclDevKernel_*` — nothing in
+vLLM/RCCL/PyTorch names atomics. Get the real message:
+
+```bash
+AMD_LOG_LEVEL=3 <the failing process>
+# → "Pcie atomics not enabled, hostcall not supported" → "AQL dispatch failed!"
+```
+
+**Cause chain:** RCCL kernels carry a hidden hostcall-buffer parameter; ROCr
+refuses to fill it without PCIe atomics, which it reads **once at driver
+probe** (`rocclr/device/rocm/rocdevice.cpp` `checkAtomicSupport()` — no else,
+no override). **No env var, module param, or sysfs toggle exists** —
+`setpci -s <gpu> CAP_EXP+28.w=0440` (DCTL2 bit 6) changes nothing at runtime.
+
+**Check the CHAIN, not the card:**
+
+```bash
+sudo lspci -vv -s <gpu> | grep -E "AtomicOpsCtl|LnkSta|LnkCap"  # want ReqEn+
+lspci -tv                                                        # every hop
+sudo lspci -vv -s <bridge> | grep AtomicOpsCap                   # want Routing+
+```
+
+`Routing-` is a read-only hardware capability. Typical AM5 layout: one x16
+slot on CPU root ports (atomics fine), the rest behind the chipset (usually
+not). Bandwidth for TP is small anyway: ~10 MB per token-step for a 27B at
+batch 4 ≈ 1.4 ms at Gen4 x4 — compute-bound, not interconnect-bound.
+
+**Probe matrix (bare 2-rank `torch.distributed`, one process per GPU, 25 s PG
+timeout) — one probe answers several design questions:**
+
+| Op | CPU-lane card | chipset card | Decides |
+|---|---|---|---|
+| `broadcast`/`all_reduce`/`barrier` | OK | FAIL | TP is dead |
+| `send`/`recv` (P2P), in ISOLATION | hang | FAIL | **PP is dead too** — vLLM's PP handoff IS this primitive |
+
+Test `send`/`recv` alone (a prior failed op leaves the group broken). Run the
+probe from a REAL file (`torch.multiprocessing.spawn` re-executes it; piping
+into `python3 -` fails with `FileNotFoundError: '/<stdin>'`). `world_size=1`
+always succeeds — **one-card success proves nothing.**
+
+**SAFETY — non-negotiable:** never run a GPU experiment beside a serving
+unit: `sudo systemctl stop <svc> && sudo systemctl mask <svc>` first (mask,
+never disable — disable deletes a relinked unit file), or use a host that
+serves nobody. A hung AQL dispatch can wedge the GPU driver; a probe beside a
+live service on 2026-09-16 preceded a hard host down (no ARP, no ICMP from
+two subnets, power cycle required). Telling off from wedged afterwards: `ip
+neigh show <ip>` from a peer — `FAILED`/`INCOMPLETE` = off/NIC dead; a wedged
+kernel usually still answers ARP. Validate your own probe's naming before
+reporting an outage (`gitea` vs `gitea-server`).
+
+**Fixes, ranked:**
+
+1. CPU-attached M.2 → PCIe x4 riser (~$30) — puts the card on the CPU root
+   complex; x4 suffices for TP.
+2. Board whose CPU x16 bifurcates x8/x8 (done 2026-09-20 on dankserver — both
+   R9700s on root ports `00:01.1`/`00:01.3`, every bridge `Routing+`, both
+   endpoints `ReqEn+`).
+3. llama.cpp `--split-mode layer` — no collectives at all ([[Llama Cpp Multi-Device Splits]]); costs the GGUF conversion plus paged attention /
+   continuous batching / spec decode.
+4. Independent TP=1 replicas + routing proxy — cards never share one model.
+
+**Closed software routes (do not re-litigate):** vLLM's IPC custom allreduce
+is gated by a CUDA-sm-keyed table (`{9.0, 10.0, 10.3, 10.7}`; ROCm reports
+12.0 → off on RDNA4 by code limit) — and patching it is insufficient because
+TP init's `broadcast`/`barrier`/logits `all_gather` have no non-RCCL path. A
+fork wrapping `.all_reduce` (e.g. radiance `RADIANCE_USE_R4D_AR` one-shot
+P2P, gloo for IPC-handle exchange) does not intercept
+`PyNcclCommunicator.__init__`'s warmup allreduce. NCCL symmetric memory needs
+`ncclDevComm*` (many builds ship the version string without the device API).
+P/D disaggregation needs a KV connector AND coordination collectives. gloo is
+CPU/TCP — unreachable for per-layer TP.
+
+**Open, unverified:** rebuilding RCCL so kernels stop declaring a hostcall
+buffer (`-mprintf-kind=buffered` is the usual claim) — counter-evidence: no
+live `printf`/`assert` in RCCL device source. Test path: rebuild, dump a
+kernel's declared args (`roc-obj-extract` + `llvm-readelf --notes`), look for
+`hidden_hostcall_buffer`; if gone, retry the 2-rank probe (service stopped).
+
+## Related
+
+- [[Llama Cpp Multi-Device Splits]] — the layer-split path around the wall
+- [[Dual-R9700-Cooperation]] / [[x4-pair-experiment]] — measured history
+- [[dankserver-qwen38-fp8-expert-offload-tuning]] — the live gather economics
				`@@ -0,0 +1 @@`
				{"model": "frank-large", "prompt": "Below is a telemetry API schema, labeled SCHEMA A. Later in this message the same schema is repeated word for word, labeled SCHEMA B.\nTask: output SCHEMA B verbatim, with no commentary and no reformatting, starting immediately with the opening brace.\n\nSCHEMA A:\n{\"version\":\"2.3.1\",\"endpoints\":[{\"id\":\"ep-000\",\"path\":\"/v2/telemetry/gw5978x49\",\"method\":\"GET\",\"latency_ms\":219,\"owner\":\"team-0\",\"limit\":{\"burst\":221,\"sustained\":149}},{\"id\":\"ep-001\",\"path\":\"/v2/telemetry/gw6279x77\",\"method\":\"POST\",\"latency_ms\":456,\"owner\":\"team-1\",\"limit\":{\"burst\":227,\"sustained\":173}},{\"id\":\"ep-002\",\"path\":\"/v2/telemetry/gw4501x82\",\"method\":\"PUT\",\"latency_ms\":488,\"owner\":\"team-2\",\"limit\":{\"burst\":407,\"sustained\":189}},{\"id\":\"ep-003\",\"path\":\"/v2/telemetry/gw3780xA8\",\"method\":\"DELETE\",\"latency_ms\":90,\"owner\":\"team-3\",\"limit\":{\"burst\":146,\"sustained\":132}},{\"id\":\"ep-004\",\"path\":\"/v2/telemetry/gw6491x17\",\"method\":\"GET\",\"latency_ms\":842,\"owner\":\"team-4\",\"limit\":{\"burst\":488,\"sustained\":193}},{\"id\":\"ep-005\",\"path\":\"/v2/telemetry/gw4848xAC\",\"method\":\"POST\",\"latency_ms\":321,\"owner\":\"team-5\",\"limit\":{\"burst\":116,\"sustained\":51}},{\"id\":\"ep-006\",\"path\":\"/v2/telemetry/gw3414x06\",\"method\":\"PUT\",\"latency_ms\":51,\"owner\":\"team-6\",\"limit\":{\"burst\":126,\"sustained\":122}},{\"id\":\"ep-007\",\"path\":\"/v2/telemetry/gw2193x74\",\"method\":\"DELETE\",\"latency_ms\":428,\"owner\":\"team-0\",\"limit\":{\"burst\":454,\"sustained\":162}},{\"id\":\"ep-008\",\"path\":\"/v2/telemetry/gw4185xB7\",\"method\":\"GET\",\"latency_ms\":717,\"owner\":\"team-1\",\"limit\":{\"burst\":197,\"sustained\":127}},{\"id\":\"ep-009\",\"path\":\"/v2/telemetry/gw7547x3E\",\"method\":\"POST\",\"latency_ms\":155,\"owner\":\"team-2\",\"limit\":{\"burst\":336,\"sustained\":177}},{\"id\":\"ep-010\",\"path\":\"/v2/telemetry/gw1090xE4\",\"method\":\"PUT\",\"latency_ms\":772,\"owner\":\"team-3\",\"limit\":{\"burst\":441,\"sustained\":198}},{\"id\":\"ep-011\",\"path\":\"/v2/telemetry/gw2746xC7\",\"method\":\"DELETE\",\"latency_ms\":439,\"owner\":\"team-4\",\"limit\":{\"burst\":113,\"sustained\":46}},{\"id\":\"ep-012\",\"path\":\"/v2/telemetry/gw9486x76\",\"method\":\"GET\",\"latency_ms\":55,\"owner\":\"team-5\",\"limit\":{\"burst\":286,\"sustained\":64}},{\"id\":\"ep-013\",\"path\":\"/v2/telemetry/gw2988x74\",\"method\":\"POST\",\"latency_ms\":140,\"owner\":\"team-6\",\"limit\":{\"burst\":411,\"sustained\":119}},{\"id\":\"ep-014\",\"path\":\"/v2/telemetry/gw9702xFC\",\"method\":\"PUT\",\"latency_ms\":576,\"owner\":\"team-0\",\"limit\":{\"burst\":305,\"sustained\":82}},{\"id\":\"ep-015\",\"path\":\"/v2/telemetry/gw8251x9C\",\"method\":\"DELETE\",\"latency_ms\":838,\"owner\":\"team-1\",\"limit\":{\"burst\":369,\"sustained\":130}},{\"id\":\"ep-016\",\"path\":\"/v2/telemetry/gw7991xD4\",\"method\":\"GET\",\"latency_ms\":565,\"owner\":\"team-2\",\"limit\":{\"burst\":229,\"sustained\":41}},{\"id\":\"ep-017\",\"path\":\"/v2/telemetry/gw8777x73\",\"method\":\"POST\",\"latency_ms\":269,\"owner\":\"team-3\",\"limit\":{\"burst\":385,\"sustained\":64}},{\"id\":\"ep-018\",\"path\":\"/v2/telemetry/gw5543xC4\",\"method\":\"PUT\",\"latency_ms\":800,\"owner\":\"team-4\",\"limit\":{\"burst\":267,\"sustained\":125}},{\"id\":\"ep-019\",\"path\":\"/v2/telemetry/gw4919x46\",\"method\":\"DELETE\",\"latency_ms\":454,\"owner\":\"team-5\",\"limit\":{\"burst\":40,\"sustained\":183}},{\"id\":\"ep-020\",\"path\":\"/v2/telemetry/gw5681x3C\",\"method\":\"GET\",\"latency_ms\":282,\"owner\":\"team-6\",\"limit\":{\"burst\":172,\"sustained\":82}},{\"id\":\"ep-021\",\"path\":\"/v2/telemetry/gw9849x14\",\"method\":\"POST\",\"latency_ms\":145,\"owner\":\"team-0\",\"limit\":{\"burst\":78,\"sustained\":60}},{\"id\":\"ep-022\",\"path\":\"/v2/telemetry/gw7275xB1\",\"method\":\"PUT\",\"latency_ms\":160,\"owner\":\"team-1\",\"limit\":{\"burst\":362,\"sustained\":55}},{\"id\":\"ep-023\",\"path\":\"/v2/telemetry/gw2052x6A\",\"method\":\"DELETE\",\"latency_ms\":421,\"owner\":\"team-2\",\"limit\":{\"burst\":170,\"sustained\":139}},{\"id\":\"ep-024\",\"path\":\"/v2/telemetry/gw8633x6A\",\"method\":\"GET\",\"latency_ms\":67,\"owner\":\"team-3\",\"limit\":{\"burst\":106,\"sustained\":108}},{\"id\":\"ep-025\",\"path\":\"/v2/telemetry/gw7381xE7\",\"method\":\"POST\",\"latency_ms\":792,\"owner\":\"team-4\",\"limit\":{\"burst\":300,\"sustained\":179}},{\"id\":\"ep-026\",\"path\":\"/v2/telemetry/gw1320xDB\",\"method\":\"PUT\",\"latency_ms\":787,\"owner\":\"team-5\",\"limit\":{\"burst\":295,\"sustained\":98}},{\"id\":\"ep-027\",\"path\":\"/v2/telemetry/gw8814x01\",\"method\":\"DELETE\",\"latency_ms\":364,\"owner\":\"team-6\",\"limit\":{\"burst\":153,\"sustained\":193}},{\"id\":\"ep-028\",\"path\":\"/v2/telemetry/gw7389xDA\",\"method\":\"GET\",\"latency_ms\":859,\"owner\":\"team-0\",\"limit\":{\"burst\":215,\"sustained\":138}},{\"id\"Line truncated
				`@@ -0,0 +1 @@`
				{"model": "frank-large", "prompt": "# Subsurface Aquifer Monitoring Network - System Design Document (Revision 7)\n\nThis document describes the deployment of a subsurface aquifer monitoring network for the Cascadia sedimentary basin. It is written for the operations crew and the regional water authority. Every sensor node, relay, aggregation service, calibration procedure, and failure mode below carries a stable alphanumeric identifier that appears in every table and in the field notes.\n\n## Sensor nodes\nNode SN-000-08B9 sits in a discharge lens at 359.0 m below datum and reports a salinity of 2.38 ppt with the fractured interval reading 5.40 degC at dawn and the wellhead porous 1.72 m3/d while the electrode pair returns 3157 uS/cm.\nNode SN-001-0A65 sits in a percolate lens at 25.9 m below datum and steps down to a salinity of 2.13 ppt with the diapir interval reading 8.55 degC at dawn and the wellhead silty 0.67 m3/d while the electrode pair returns 979 uS/cm.\nNode SN-002-0571 sits in a diapir lens at 287.8 m below datum and peaks at a salinity of 7.63 ppt with the karst interval reading 4.26 degC at dawn and the wellhead spooned 11.50 m3/d while the electrode pair returns 2987 uS/cm.\nNode SN-003-0CE9 sits in a confining lens at 169.5 m below datum and logs a salinity of 0.96 ppt with the alluvial interval reading 5.03 degC at dawn and the wellhead salinized 12.08 m3/d while the electrode pair returns 2607 uS/cm.\nNode SN-004-0E2A sits in a freshened lens at 70.4 m below datum and settles near a salinity of 8.68 ppt with the spooned interval reading 4.88 degC at dawn and the wellhead siphon 4.38 m3/d while the electrode pair returns 2707 uS/cm.\nNode SN-005-0DDD sits in a percolate lens at 44.6 m below datum and reports a salinity of 6.25 ppt with the alluvial interval reading 10.85 degC at dawn and the wellhead brackish 13.99 m3/d while the electrode pair returns 3618 uS/cm.\nNode SN-006-0EFC sits in a till lens at 309.5 m below datum and reports a salinity of 3.49 ppt with the lacustrine interval reading 7.38 degC at dawn and the wellhead spooned 3.21 m3/d while the electrode pair returns 1253 uS/cm.\nNode SN-007-0B02 sits in a recharge lens at 262.0 m below datum and steps down to a salinity of 2.38 ppt with the discharge interval reading 8.18 degC at dawn and the wellhead perched 4.05 m3/d while the electrode pair returns 3900 uS/cm.\nNode SN-008-0448 sits in a karst lens at 406.5 m below datum and reports a salinity of 7.32 ppt with the upconing interval reading 6.17 degC at dawn and the wellhead fluvial 0.75 m3/d while the electrode pair returns 1448 uS/cm.\nNode SN-009-043C sits in a alluvial lens at 159.3 m below datum and flags a salinity of 6.20 ppt with the percolate interval reading 7.60 degC at dawn and the wellhead brackish 13.01 m3/d while the electrode pair returns 2025 uS/cm.\nNode SN-010-0FF5 sits in a karst lens at 264.2 m below datum and flags a salinity of 7.05 ppt with the freshened interval reading 7.88 degC at dawn and the wellhead siphon 8.41 m3/d while the electrode pair returns 1635 uS/cm.\nNode SN-011-098A sits in a drain lens at 365.7 m below datum and reports a salinity of 8.11 ppt with the wedge interval reading 5.51 degC at dawn and the wellhead glacial 2.52 m3/d while the electrode pair returns 2916 uS/cm.\nNode SN-012-0AE8 sits in a till lens at 259.6 m below datum and steps down to a salinity of 9.13 ppt with the recharge interval reading 11.60 degC at dawn and the wellhead spooned 0.46 m3/d while the electrode pair returns 3077 uS/cm.\nNode SN-013-0B83 sits in a salinized lens at 171.2 m below datum and peaks at a salinity of 2.83 ppt with the lacustrine interval reading 5.56 degC at dawn and the wellhead discharge 0.35 m3/d while the electrode pair returns 3083 uS/cm.\nNode SN-014-032E sits in a drain lens at 249.6 m below datum and flags a salinity of 1.09 ppt with the salinized interval reading 9.58 degC at dawn and the wellhead perched 12.00 m3/d while the electrode pair returns 2221 uS/cm.\nNode SN-015-004F sits in a aeolian lens at 264.4 m below datum and reports a salinity of 7.34 ppt with the salinized interval reading 8.76 degC at dawn and the wellhead perched 0.31 m3/d while the electrode pair returns 1483 uS/cm.\nNode SN-016-07C6 sits in a aeolian lens at 155.9 m below datum and reports a salinity of 0.64 ppt with the mineralized interval reading 11.75 degC at dawn and the wellhead diluted 13.46 m3/d while the electrode pair returns 524 uS/cm.\nNode SN-017-06C5 sits in a salinized lens at 72.1 m below datum and averages a salinity of 4.52 ppt with the siphon interval reading 8.93 degC at dawn and the wellhead confining 3.98 m3/d while the electrode pair returns 3687 uS/cm.\nNode SN-018-0E64 sits in a siphon lens at 334.9 m below datum and steps down to a salinity of 6.73 ppt with the salinized interval reading 7.63 degC at dawn and the wellhead freshened 9.64 m3/d while the electrode pair returns 1681 uS/cm.\nNode SN-019-0122 sits in a spooned lens at 1Line truncated