docs: index frank-primary-freetoken-pr587; mark the r9v q8 runbook retired

All ~173 managed skills read in full and merged, one polished reference
page per skill-cluster:

- LLM cross-cutting (17 pages): GGUF Compatibility, Llama Cpp Serving
  Sizing, Llama Cpp Multi-Device Splits, PCIe Link and Atomics, RDNA4
  Container Serving, RDNA4 MoE Expert Offload VRAM Budget, vLLM Boot
  Forensics, vLLM Porting and Compile Triage, LLM Bench Windows, LLM
  Performance Diagnosis, Model Swap and Rollback, Tcclaviger Fork Ops,
  Heterogeneous Split Triage, FreeToken Builds + deltas folded into the
  retired runbooks (GLM DFlash2 port laws, Lucebox dual-R9700, R9V
  qualification pipeline, FLM pack registration, ik hybrid envelope,
  frank-medium split rebalance, llama.cpp GGUF preflight, batch-ubatch
  split section, bench-window D-state/stale-read notes, second-endpoint
  bring-up).
- Architecture (3): Stale Signal Audit, Quadlet Lifecycle, Deploy State
  Triage.
- Operations (6 + index): Remote Job Runner, Payload and Disk Reclaim,
  Container Host Reachability, Fleet Ops Playbook, Fleet Audit and
  Posture, SSH and Remote Access.
- Infrastructure: Observability Provisioning + Homepage Status Probes;
  updates to Authentik, Anubis, LAN-DNS (per-client DoH bypass, Porkbun
  zone ops), Swap Configuration, OpenWebUI, Hermes (pre-existing page
  preserved and extended).
- FrankBook-Pro: ARR-Stack filled (API split, schema-first probing,
  auth-posture, auth-config trap); qBittorrent E2E verification.
- Desktop/Agent/Research: Noctalia and Niri, OMP Harness, Agent Docs and
  Edits, Command Center Dotfiles, PICO-8 Asset Standards.
- Indexes: LLM Cross-Cutting References section, Operations/ and Agent/
  index pages, root-index Skill Distillation section, AGENTS.md pointer.
- log.md carries the wave plan.

check-vault-links.py 0 unresolved (185 pages); ./tests/run.sh 366/366.

AGENTS.md

@@ -12,8 +12,8 @@ merge absorbed the former `vllmScripts` repo, history preserved).
   Calibre-Web, Jellyfin, Plex) plus the `frank-primary` LLM service.
 - **Dankclaw** (192.168.0.143) — AI/search/dev/observability stack (SearXNG
   via Gluetun, ComfyUI, OpenWebUI, Hermes, Gitea, Grafana, Prometheus, Loki,
-  Alloy, Pi-hole) plus TWO LLM services: `frank-large` (ROCmFPX llama.cpp
-  fork on the gfx1151 iGPU) and `frank-small` (llama.cpp CUDA on the RTX 3090). Caddy on
+  Alloy, Pi-hole) plus TWO LLM services: `frank-large` (halogen container on
+  the gfx1151 iGPU) and `frank-small` (llama.cpp CUDA on the RTX 3090). Caddy on
   Dankclaw reverse-proxies both hosts.
 - **FrankBook-Pro** (192.168.0.112) — macOS mini running the *arr stack
   (Prowlarr, Sonarr, Radarr, Lidarr, Readarr, qBittorrent) as quadlets inside
@@ -24,9 +24,9 @@ merge absorbed the former `vllmScripts` repo, history preserved).
 
 | Target | Service | Runtime | Model | Port |
 |--------|---------|---------|-------|------|
-| Dankserver | `frank-primary` | tcclaviger RDNA4 vLLM fork (`docker.io/tcclaviger/vllm@sha256:da4029666…`, digest-pinned, plain `podman run`, root systemd unit) | `orcarouter/Qwen3.8-Flash-Next-Uncensored-FP8` (~172.8 GiB, 131 shards), TP=2, expert offload automatic (fork plans it), prefetch OFF, **fp8 KV** (`--kv-cache-dtype fp8`), **no speculative decode**, MNBT 1024, MoE gather 32×64, **2 seqs × 200704 ctx** at KV_BYTES 3,150,000,000/rank (fp8 measures 7,701 B/token), pool deliberately small so the expert arena keeps what it reclaims (~7,500 slots vs 6,248 at the 6.1e9 shape) — util 0.98, and the client contract wired: `--reasoning-parser …
-| Dankclaw | `frank-large` | **ROCmFPX llama.cpp fork** (`kingjones30/ROCmFPX` @ `36e9acd`, gfx1151, built by `scripts/build-rocmfpx.sh`; bare `llama-server`, root systemd unit) | `kingjones777/Qwen3.8-27B-ROCmFP4-STRIX-MTP-GGUF` **`Qwen3.8-27B-Q8_0_ROCMFPX_AGENT.gguf`** (26.28 GiB, ftype 115 ≈ 8.39 bpw: 353 F32 + 187 Q8_0 + 311 `GGML_TYPE_Q8_0_ROCMFPX` tensors — only this fork can read them), **MTP spec decode from the repo's own `mtp-Qwen3.8-27B-Q4_0.gguf` (n-max 4, pinned to ROCm0)**, **1 session × 262,144 (native window)** with **f16 KV**, the bundled BF16 mmproj for vision, thinking on with the card's recipe (temp 1.0 / top_p 0.95 / top_k 20). Measured 2026-10-02: decode **17.0–17.1 t/s** with MTP vs **7.6 without (+123 %)**, draft acceptance 0.462, prefill 325–351 t/s, prompt-cache replay 0.5 s, boot→healthy 16 s, GTT 64.5 GiB. Cut over 2026-10-02 from gufo; rollback tag `frank-large-gufo-q4kxl-20261001` | 7770 |
-| Dankclaw | `frank-small` | llama.cpp master, CUDA sm_86 build (root systemd unit, relinked) | bartowski/MiMo-V2.6-Distill-Qwen-9B-GGUF **Q5_K_M** (9B dense qwen3_5 hybrid, 6.4 GiB GGUF, no MTP weights so no spec decode) + the repo's f16 mmproj for **vision**, on the RTX 3090, **3 × 180,224** (pool 540,672 — NOT the native window; three contexts cost 81,920 tokens per conversation) with **asymmetric KV f16 K + q8_0 V** under flash attention, 21,684 used / **2,441 free** of 24,576 MiB, 106.8 tok/s decode, 3,814 t/s prefill. Resized twice 2026-10-02 (Q6_K → Q5_K_M → 3 slots + vision → this pool); rollback `models/MiMo-Q6K/` or `PARALLEL=2 CTX_SIZE=524288` | 7771 |
+| Dankserver | `frank-primary` | **FreeToken PR #587** (branch `rocm714_torch214`, head `77a7201`; local image `frank-primary-freetoken:pr587-77a7201` built on the host from `scripts/docker/frank-primary-freetoken.Dockerfile` — digest-pinned rocm/pytorch rocm7.14 torch-2.11.0 base, plain `podman run`, root systemd unit) | `orcarouter/Qwen3.8-Flash-Next-Uncensored-FP8` (~172.8 GiB, 131 shards, modelopt block-FP8, loaded straight from safetensors), **ONE R9700 (second card idle)**, all experts in **pinned host RAM** (~120 GiB) behind a GPU LRU (`--moe-cache-rate 0.10`), PLE table 47.7 GiB pinned (root unit + `LimitMEMLOCK=infinity`), **KV pool `--num-tokens 299968`** (24 KiB/token, multiple-of-64 law of the `qsa_sparse` auto backend), 2 slots, `--max-output-tokens 36864`, **parsers AUTO** (thinking → `reasoning_content`, tools qwen3.8 dialect), vision tower host-streamed. Measured: boot ~2.3 min cold / 92 s warm, decode **17.1 tok/s** single-stream; **TP=2 does not boot on ROCm** (pynccl JIT); `/metrics` 404 (Prometheus scrape reads down). Deploy gate reads the `/health` BODY (`status:"ok"`) — FreeToken 200s while loading. Rollback tag `frank-primary-vllm-shapeG-20261003` |7770|
+| Dankclaw | `frank-large` | **halogen container** (`ghcr.io/peonist-ai/halogen@sha256:dc0a39a0…`, digest-pinned, plain `podman run` via `scripts/serve-frank-large.sh`, root systemd unit) | `peonist-ai/halogen-qwen3.8-27b` **`qwen3.8-27b-p1w4d-d2.hgn`** (35,865,565,184 B; the `.hgn` format loads in NO other engine — llama.cpp, vLLM and transformers all refuse it), **DFlash2 spec decode (`HALOGEN_DRAFTER=2`)**, **1 slot × 200,700 ctx** — `HALOGEN_KV_SLOTS=1` is load-bearing: batching XOR speculation, slots>1 silently disables the drafter (measured: 8 slots at serial ~6 t/s/stream), d2 KV (2× fp8, 64 KiB/token → 12.2 GiB pool, engine sizes it and prints a boot ledger naming any clamp), **TEXT-ONLY** (the engine ignores the vision encoder — vision is frank-small's job), **censored base** (the ROCmFPX AGENT rollback is the refusal escape hatch), **no server-side sampler: absent temperature is GREEDY**, so the card's thinking recipe (temp 1.0 / top_p 0.95) rides the wire in `models.yml`. Publisher card 31.7 t/s mean across eight text workloads vs the retired ROCmFPX endpoint's measured 27.6–28.1. Prompt over the slot AND `max_tokens` over `HALOGEN_MAX_TOKENS_CAP` are both 400s naming the limit, never truncations. Serve passes NO `--ulimit nofile` (rootless runc cannot raise Dankclaw's 524288 hard limit — see Key Lessons). Cutover 2026-10-05|7770|
+| Dankclaw | `frank-small` | llama.cpp master, CUDA sm_86 build (root systemd unit, relinked) | bartowski/MiMo-V2.6-Distill-Qwen-9B-GGUF **Q6_K** (9B dense qwen3_5 hybrid, 7.26 GiB GGUF, no MTP weights so no spec decode) + the repo's f16 mmproj for **vision**, on the RTX 3090, **2 × 200,704** (pool 401,408 — NOT the native window) with **fully unquantized KV (f16 K + f16 V)** under flash attention, **batch 4096 / ubatch 2048** (doubled 2026-10-05: measured +4.6 % prefill at 8,928 tokens for 616 MiB), 21,846 used / **2,279 free** of 24,576 MiB, 94.4 tok/s decode. Resized 2026-10-03 (3 × 180,224 + q8_0 V → 2 × 200,704 + f16 V → + Q6_K rung: the freed slots paid for exact values AND the higher quant; the rung costs a measured −10 % decode — weight-bandwidth-bound); rollback `MODEL_FILE=…/MiMo-Q5KM/…Q5_K_M.gguf` (buys the 10 % back) or `CTX_SIZE=540672 PARALLEL=3 CACHE_TYPE_V=q8_0` (Q5 only) | 7771 |
 
 Retired endpoints (`frank-npu`, and every prior runtime) are in
 the [runbook index](#runbook-index). The full swap history lives in
@@ -40,7 +40,11 @@ the [runbook index](#runbook-index). The full swap history lives in
 - `scripts/` — `deploy.sh` / `deploy-plan.py` / `deploy-remote.sh`, LLM serve
   scripts, systemd units, env tunables, bench harnesses
 - `old/` — retired LLM scripts (reference only)
-- `serverAppConfigs-docs/` — knowledge base (Obsidian vault, OKF format)
+- `serverAppConfigs-docs/` — knowledge base (Obsidian vault, OKF format).
+  The managed skill library is distilled there one page per skill-cluster:
+  `LLM/index.md` → *Cross-Cutting References*, plus `Operations/index.md`,
+  `Agent/index.md`, and the Infrastructure/Dankclaw/FrankBook-Pro service
+  pages (`serverAppConfigs-docs/log.md` carries the wave plan and results)
 - `tests/` — bats suite + Playwright e2e
 
 ## Git Workflow
@@ -74,6 +78,14 @@ unrelated
 container changes cannot interrupt inference. Set `FRANK_SUDO_PW` for headless
 runs on a host without passwordless sudo; the value never enters the repo.
 
+**`deploy.sh <target>` with no `--service` is a fleet-wide DRAIN, not a sync.**
+It also walks every per-service revision marker under the deploy-state dir and
+restarts each service whose marker is behind HEAD *and* whose paths changed
+since that marker — so a backlog accumulated by service-scoped deploys fires
+all at once, and one failing health gate aborts the run with every later
+service left pending. Name the service unless a drain is what you want
+(`./scripts/deploy.sh dankserver --service frank-primary`).
+
 ### Deployment completeness (MANDATORY)
 
 A deploy is **not complete** when `deploy.sh` exits green. It requires:
@@ -82,7 +94,14 @@ A deploy is **not complete** when `deploy.sh` exits green. It requires:
 2. the Playwright e2e suite (`tests/e2e/run.sh`) — runs **automatically** as
    the `[3/3]` gate at the end of `deploy.sh` against the LIVE fleet; a
    behavior-affecting deploy that fails it exits `DEPLOY INCOMPLETE`. Bypass
-   only with `--skip-e2e` in an emergency.
+   only with `--skip-e2e` in an emergency. The gate runs the conditional
+   `frankbook` Playwright project (the `*arr`/qBittorrent walk,
+   `tests/e2e/specs/19-frankbook.spec.ts`) ONLY when a frankbook blast-radius
+   path changed since origin/main — that spec, `frankbook-pro/**`,
+   `shared/homepage/services.json`, or the `dankclaw/Caddyfile` serving the
+   arr vhosts — so a sleeping FrankBook-Pro never fails an unrelated deploy.
+   The default project set excludes it everywhere else (`E2E_INCLUDE_FRANKBOOK=1`
+   opts in; see `serverAppConfigs-docs/Architecture/E2E Suite.md`).
 
 ```mermaid
 graph LR
@@ -215,7 +234,7 @@ Scratch ports are not used on LLM hosts — a
 test process cannot share the single GPU anyway; validate against the service
 itself.
 
-(A gufo-era law; frank-large is ROCmFPX llama.cpp since 2026-10-02, but the rollback tag keeps it live.) **`gufo bench` cannot rank speculative-decode settings, and will invert the ranking.**
+(A gufo-era law; frank-large ran ROCmFPX llama.cpp 2026-10-02 → 10-05 and is the halogen container since, but the rollback tags keep it live.) **`gufo bench` cannot rank speculative-decode settings, and will invert the ranking.**
 Measured 2026-09-28 on frank-large at depth 32768: its corpus
 reports `acceptance=0.933 ± 0.000` with an **identical trace sha256** across
 draft widths, i.e. the same highly-predictable text every run, and 93 % of
@@ -241,11 +260,11 @@ one: `ip neigh show <ip>` from a peer — FAILED means off/NIC dead, while a
 merely wedged kernel usually still answers ARP.
 
 1. Stop and mask (self-healing would restart it):
-   - Dankclaw `frank-large` (root unit, vLLM podman on the iGPU):
+   - Dankclaw `frank-large` (root unit, halogen podman on the iGPU):
      `sudo systemctl stop frank-large && sudo systemctl mask frank-large`
    - Dankclaw `frank-small` (root unit, llama.cpp on the 3090):
      `sudo systemctl stop frank-small && sudo systemctl mask frank-small`
-   - Dankserver `frank-primary` (root unit, tcclaviger vLLM on the 2x R9700):
+   - Dankserver `frank-primary` (root unit, FreeToken on one R9700):
      `sudo systemctl stop frank-primary && sudo systemctl mask frank-primary`
 2. Bench.
    NOTE: `systemctl mask` FAILS on a relinked unit (the unit file in
@@ -277,14 +296,17 @@ When renaming a service, retiring an endpoint, or moving a model: update the
 provider block, model `id`, `contextWindow`, `maxTokens`, `input` in
 `models.yml`, then fix every `modelRoles` entry that named the old endpoint —
 a role pointing at a provider the file does not define fails at call time. As
-of 2026-10-02 **all three endpoints accept images** (`input: [text, image]`):
-`frank-large` through the BF16 mmproj that ships inside its GGUF payload,
-`frank-primary` through a BF16 mmproj loaded explicitly (this build never
-autoloads projectors), and `frank-small` through
-`mmproj-MiMo-V2.6-Distill-Qwen-9B-f16.gguf` passed as `--mmproj` — the Q5_K_M
+of 2026-10-05 **two of the three endpoints accept images** (`input: [text,
+image]`) — `frank-large` went TEXT-ONLY with the halogen swap (the engine
+ignores the vision encoder; the BF16-mmproj era ended 2026-10-05) — while
+`frank-primary` takes them natively (FreeToken builds the qwen3.8 vision
+tower and streams its block weights from host RAM two at a time by default),
+and `frank-small` through
+`mmproj-MiMo-V2.6-Distill-Qwen-9B-f16.gguf` passed as `--mmproj` — the MiMo
 GGUF itself carries no vision tensors. `frank-small`'s `contextWindow` is
-**180224**, the per-slot window, not the model's native 262144: the harness
-must compact before llama.cpp answers a longer prompt with a 400. Note also
+**200704** (resliced from 180224 on 2026-10-03), the per-slot window, not the
+model's native 262144: the harness must compact before llama.cpp answers a
+longer prompt with a 400. Note also
 that
 the reasoning channel differs by runtime: llama.cpp AND SGLang (qwen3
 parser, live-verified) emit `reasoning_content`, vLLM emits `reasoning` —
@@ -295,76 +317,64 @@ endpoint on 2026-09-11, returned 2026-09-17 with the Gemma4 projector's
 audio encoder, and left again with the Gemma4 retirement on 2026-10-01
 (the MiMo tower is vision-only).
 
-### frank-large — ROCmFPX llama.cpp, Qwen3.8-27B Q8_0_ROCMFPX_AGENT + MTP (Dankclaw, ACTIVE)
+### frank-large — halogen container, Qwen3.8-27B + DFlash2 (Dankclaw, ACTIVE)
 
-The **kingjones30/ROCmFPX llama.cpp fork** at commit
-`36e9acd40e10a87cd3c3ef8ec734668757dc8520`, built for gfx1151 by
-`scripts/build-rocmfpx.sh` into `/home/dank/GitPackages/kingjones-ROCmFPX/build`,
-as a **root systemd unit** (`scripts/frank-large.service`, `User=dank`,
-`Restart=always`) running a bare `llama-server` (no container) on port 7770
-from the Strix Halo iGPU. Boot to healthy: **16 s**.
+The **halogen engine** (`ghcr.io/peonist-ai/halogen@sha256:dc0a39a0…`,
+digest-pinned, published tag 0.1.4) as a podman container driven by
+`scripts/serve-frank-large.sh` under a **root systemd unit**
+(`scripts/frank-large.service`, `User=dank`, `Restart=always`, `ExecStop=podman
+stop -t 120 frank-large-halogen`, the script `exec podman wait` so the restart
+policy is real) on port 7770 from the Strix Halo iGPU. Engine stdout goes to
+`/home/dank/logs/frank_large.log` — **the boot ledger (KV pool, drafter state,
+any clamp) lives there, not in the journal**. Serving
+`peonist-ai/halogen-qwen3.8-27b`: `qwen3.8-27b-p1w4d-d2.hgn` (35,865,565,184 B
+≈ 33.4 GiB, sha256 `274c3fc7…`, 1352 tensors; p1 weights / w4 activations /
+d2 KV) + the flat `tokenizer/` directory (5 files). Cutover 2026-10-05 from the ROCmFPX llama.cpp fork
+([[frank-large-rocmfpx-q6agent-uncensored-mtp]], measured 27.6–28.1 t/s prose),
+chosen for the publisher's 31.7 t/s DFlash2 mean.
 
-**The runtime is not interchangeable — it is chosen by the FILE.**
-`kingjones777/Qwen3.8-27B-ROCmFP4-STRIX-MTP-GGUF`'s best rung,
-`Qwen3.8-27B-Q8_0_ROCMFPX_AGENT.gguf` (28,214,425,952 B, sha256 `f3ef2bda…`,
-ftype 115 ≈ 8.39 bpw), stores **311 tensors of type 103,
-`GGML_TYPE_Q8_0_ROCMFPX`** — a quantization that exists in no other loader:
-stock llama.cpp and gufo both refuse it at load (`unknown tensor type`). The
-file is 353 F32 + 187 plain-Q8_0 + 311 ROCmFPX; the AGENT routing variant
-keeps the output-head tensors at plain Q8_0 (187 vs 2 in the non-AGENT file),
-which is why the card reports MTP acceptance 0.953 vs 0.911 — highest quality
-AND the better drafter base. Vision comes from the bundled
-`mmproj-Qwen3.8-27B-BF16.gguf` (931,145,888 B).
+**The format is the lock.** `.hgn` loads in nothing else — the runtime is
+chosen by the FILE, as ROCmFPX's tensor types were before it. This is NOT the
+2026-09 `halogen-flash-server` image (different repo, Flash-Next w4b era); the
+halogen-server repo restarted versioning at 0.1.x.
 
-**The pin is load-bearing.** The 2026-08-25 ROCmFPX builds segfaulted in
-`ggml_compute_forward_set_rows` on exactly this file; `build-rocmfpx.sh`
-checks the commit out by SHA, so a fork-side force-push cannot silently
-re-break the endpoint.
+**Two product regressions, both accepted by Frank before deploy:** the
+endpoint is **text-only** (README honest limits: the vision encoder is
+ignored — frank-small owns images now) and the checkpoint is a finetune of the
+**censored** `Qwen/Qwen3.8-27B` base (refusals can return; the rollback is one
+commit-revert away with the AGENT payload still on disk).
 
-**Spec decode is the performance feature, and it is measured.** The draft is
-the repo's own `mtp-Qwen3.8-27B-Q4_0.gguf` (1,680,271,648 B, sha256 `051a1764…`,
-`nextn_predict_layers = 1`), pinned to the same card:
-`--spec-type draft-mtp --spec-draft-n-max 4 --spec-draft-n-min 0
---spec-draft-p-min 0.0 --spec-draft-device ROCm0 --spec-draft-ngl 99`.
-Measured 2026-10-02 in one session, temp 0, 2,202-token prompt / 512 tokens,
-`cache_prompt: false`, production service stopped for each arm:
-**MTP on 17.00–17.14 t/s decode** (draft acceptance 0.462, mean acceptance
-length ≈ 1.86) against **7.60–7.66 t/s with the draft removed — +123 %**;
-prefill pays 8–15 % (381 → 325–351 t/s). Acceptance is far from the card's
-0.95 (that number is the card's own text at a shallow prompt); the gain is
-real and n-max 4 is where it was measured, never swept. This replaces the
-gufo Q4_K_XL arm at ~15 t/s, so the endpoint got higher quality AND more
-speed at once.
+**Engine laws that shape the config** (all in `scripts/frank-large.env` with
+evidence): `HALOGEN_KV_SLOTS=1` — batch XOR speculation, slots>1 turns the
+drafter off silently. `HALOGEN_SLOT_CTX=200700` — the per-conversation HARD
+ceiling (400 naming the limit, never truncation); `models.yml`
+`contextWindow: 200700` must match or the harness discovers the wall as a 400.
+`HALOGEN_MAX_TOKENS_CAP=65536` + `HALOGEN_QUEUE_TIMEOUT=7200` are a coupled
+pair; the harness `maxTokens: 57344` stays under the cap (over-cap is a 400
+naming the limit — README, never a 413). Thinking tokens COUNT against the
+budget: exhausted mid-thought is `finish_reason: "length"` with empty
+`content` and the reasoning in `reasoning_content`.
+`HALOGEN_DRAFTER=2` = DFlash2. **No server-side sampler**: an absent
+temperature is greedy, so the thinking recipe exists only in `models.yml`
+`extraBody`. `response_format` is unsupported (no constrained decoding).
+Thinking lands in **`reasoning_content`** (harness field unchanged); the same
+Qwen3.8 template rejects `reasoning_effort` outside `xhigh|medium|low`, so the
+`models.yml` fold stays mandatory.
 
-**Shape and budget:** ONE session at the native 262,144 window, unquantized
-f16 KV (~64 KiB/token → ~16 GiB, affordable inside the iGPU's ~114 GiB GTT
-aperture), `--fit off --flash-attn on --direct-io --ctx-checkpoints 16
---cache-ram 8192 --cache-idle-slots`. Measured GTT 64.5 GiB with model +
-draft + projector resident. Prompt cache verified: a repeat pass over the
-same 6k prompt replays in 0.5 s.
+Fetch: `scripts/download-frank-large-halogen.sh` into
+`/home/dank/models/halogen-qwen3.8-27b/` (`-f` + byte count + LFS sha256 +
+`--verify-only`, no token on the command line; resume goes against the
+RESOLVED CDN URL because curl drops `Range` across HF's 302). The deployer
+and serve script both refuse a missing tokenizer file or **any symlink** —
+an HF cache snapshot symlinks each entry into `blobs/` and those links
+dangle inside the container mount.
 
-**Client contract (llama.cpp, live-verified 2026-10-02):** `--jinja` is on
-(without it `chat_template_kwargs` are silently ignored), tools are native
-(`get_weather` → `{"city":"Seattle"}`), thinking lands in
-**`reasoning_content`**, `reasoning_effort` accepts ONLY
-`xhigh|medium|low` (anything else is HTTP 500 — `models.yml` folds `high` and
-`max` to `xhigh`), and thinking is enabled server-side with the card's
-recipe (temp 1.0 / top_p 0.95 / top_k 20 / min_p 0) plus
-`--reasoning-format deepseek --reasoning-budget -1`. Two traps: a request
-whose whole budget is consumed by reasoning returns `content: null` (bench
-clients must fall back to `reasoning_content` or they crash *after*
-measuring), and the fork logs
-`MTP boundary missing for seq_id=… ; resyncing` under concurrent draft
-resets — self-recovering, not a fault.
-
-Fetch: `scripts/download-frank-large-kingjones.sh` (`-f` + byte count + LFS
-sha256 verified, `--verify-only`, no token on the command line).
-
-Runbook: `serverAppConfigs-docs/LLM/frank-large-rocmfpx-q8agent-mtp.md`.
-Rollback: git tag **`frank-large-gufo-q4kxl-20261001`** — gufo v0.1.1
-(`scripts/build-gufo.sh` stays on disk for it), the Q4_K_XL Qwen3.8-27B
-payload and the DFlash2 drafter all remain on the host; restore the tag's
-unit/env/serve trio and redeploy.
+Runbook: `serverAppConfigs-docs/LLM/frank-large-halogen27b-dflash2.md`.
+Rollback: revert the swap commit and redeploy — the ROCmFPX payload
+(`/home/dank/models/kingjones-Qwen3.8-27B-ROCmFP4-STRIX-MTP/`), the fork build
+and `scripts/build-rocmfpx.sh` (pin `36e9acd`) all remain on disk, seam-tested.
+The older gufo tag `frank-large-gufo-q4kxl-20261001` stays a **re-download**
+rollback (payloads removed 2026-10-03; `scripts/build-gufo.sh` stays).
 
 ### frank-small — llama.cpp on the RTX 3090 (Dankclaw, ACTIVE)
 
@@ -372,15 +382,20 @@ unit/env/serve trio and redeploy.
 `/home/dank/GitPackages/llama-master/build-cuda/bin/llama-server`) as a
 **root systemd unit** (`scripts/frank-small.service`, `relink_unit`ed into
 `/etc/systemd/system/`), serving
-`bartowski/MiMo-V2.6-Distill-Qwen-9B-GGUF` at **Q5_K_M** (6,876,124,704 B,
-sha256 `04cf4f…`) **plus `mmproj-MiMo-V2.6-Distill-Qwen-9B-f16.gguf`**
-(918,166,048 B, sha256 `ff348f…`), both `/home/dank/models/MiMo-Q5KM/`, on
-port 7771. The 2026-10-02 swap was a **quant rung down from Q6_K**, not a
-runtime change — the llama.cpp path arrived 2026-10-01 night for one reason:
-**asymmetric KV**. `--cache-type-k f16` + `--cache-type-v q8_0` is the fleet's
-only per-tensor KV lever; SGLang's `--kv-cache-dtype` moves K and V together.
-Vision came the same day via `--mmproj`: the GGUF carries no `v.*` tensors, so
-the projector file is the entire vision path (and ~1.4 GiB of the card).
+`bartowski/MiMo-V2.6-Distill-Qwen-9B-GGUF` at **Q6_K** (7,793,710,624 B,
+sha256 `ef96d05a…`) **plus `mmproj-MiMo-V2.6-Distill-Qwen-9B-f16.gguf`**
+(918,166,048 B, sha256 `ff348f…`), both `/home/dank/models/MiMo-Q6K/` (the
+projector is a hardlink of the Q5KM copy — the dir is self-contained), on
+port 7771. The rung went Q5_K_M → Q6_K on 2026-10-03 the same morning as the
+reslice, funded by the freed slots. The runtime arrived 2026-10-01 night for
+one reason:
+**asymmetric KV** — `--cache-type-k f16` + `--cache-type-v q8_0` is the fleet's
+only per-tensor KV lever, and SGLang's `--kv-cache-dtype` moves K and V
+together. On 2026-10-03 Frank spent the headroom that lever bought: the reslice
+from three slots to two made room for **fully unquantized V**
+(`CACHE_TYPE_V=f16`); both flags stay explicit in the serve script. Vision came
+2026-10-02 via `--mmproj`: the GGUF carries no `v.*` tensors, so the projector
+file is the entire vision path (and ~1.4 GiB of the card).
 
 **Architecture: qwen3_5 hybrid** — 32 layers, 8 full-attention + 24 Gated
 DeltaNet, native 262,144 window. KV exists only in the 8 attention layers
@@ -388,69 +403,108 @@ DeltaNet, native 262,144 window. KV exists only in the 8 attention layers
 the DeltaNet state is a fixed per-slot buffer, so slot count — not context —
 drives that term.
 
-**The shape:** `--ctx-size 540672 --parallel 3` = **THREE conversations of
-180,224 tokens each** (llama.cpp divides the pool:
-`n_ctx_slot = CTX_SIZE / PARALLEL`), `--flash-attn on` (the q8_0-V path is
-only cheap under flash attention — the seam pins it), f16 K + q8_0 V, all
-layers on the GPU, `--mmproj` for vision. Measured VRAM **21,684 used / 2,441
-free** of 24,576 MiB — the 2–3 GiB headroom Frank asked for, landed by
-solving the pool against two measured boots: non-KV cost is 7,390 MiB without
-a projector (2 × 262,144 boot) and 8,859 MiB with one (3 × 196,608 boot), so
-vision costs ~1.4 GiB, not the 0.9 GiB the file size suggests. Three native
-windows (786,432 tokens) need 18.4 GiB of KV alone and were never on the
-table. Full f16 V would add ~4 GiB more and bust the card — that is the
-justification for the asymmetric contract.
+**The shape (2026-10-03):** `--ctx-size 401408 --parallel 2` = **TWO
+conversations of 200,704 tokens each** (llama.cpp divides the pool:
+`n_ctx_slot = CTX_SIZE / PARALLEL`), `--flash-attn on` (the seam pins it),
+**f16 K + f16 V**, all layers on the GPU, `--mmproj` for vision, and since
+2026-10-05 **batch 4096 / ubatch 2048** (doubled; the boot-pair A/B is in the
+runbook). Measured trade at 8,928 prompt tokens: prefill 4,024 → **4,209 t/s,
++4.6 %**, decode flat (91.9–93.0 vs 93.8–94.3 — noise), and a 279-token prompt,
+which sits below either ubatch, did not move at all. The old "ubatch 2048
+failed the scratch boot" note belongs to the Q5_K_M predecessor and does NOT
+generalize: this shape boots 2048 clean. Measured VRAM **21,846 used / 2,279
+free** of 24,576 MiB against **21,276 / 2,849** for the 1024 arm, reproduced
+byte-identically the same hour — the doubling costs **616 MiB** and lands at
+the BOTTOM of the 2–3 GiB floor Frank asked for (`4096/1536` is the middle rung
+if that margin is ever wanted; non-KV 9,302 MiB, up from 8,732 by exactly the
+compute buffers; f16 K+V is 32 KiB/token across the 8 attention layers;
+quote `nvidia-smi`'s `free` field, never `total − used` — the driver holds
+~450 MiB of reserve the subtraction hides). The 2026-10-02 3 × 180,224 +
+q8_0-V shape stays one env edit away (`CTX_SIZE=540672 PARALLEL=3
+CACHE_TYPE_V=q8_0`, and it must pair with Q5_K_M — Q6_K breaks the floor
+there). Three native windows need 18.4 GiB of KV alone and were never on the
+table.
 
-Verified at deploy (2026-10-02): `/v1/models` reports `frank-small`; boot logs
-`n_slots = 3, n_ctx_slot = 180224` and `loaded multimodal model`; decode
-**106.8 tok/s** at temp 0 (prefill 3,814 t/s) — ~5 % faster than the
-2 × 262,144 shape, because a shorter slot means a shorter attention scan;
-three concurrent requests answered in parallel and a fourth queued behind a
-released slot; native `tool_calls`; thinking in `reasoning_content`; a
-solid-red PNG answers "Red." through an `image_url` content part (the legacy
-top-level `images` field is ignored by `/v1/chat/completions` — the model then
-invents a color). A prompt over the slot is a **400
-`exceed_context_size_error`**, which is why `models.yml` sets
-`contextWindow: 180224`.
+Verified at deploy (2026-10-03): boot logs `n_slots = 2, n_ctx_slot = 200704`
+and `loaded multimodal model`; decode **94.44 tok/s** at temp 0 on the probe
+that measured 104.84 on Q5_K_M — **the Q6_K rung costs a measured −10 %**:
+decode on this card is weight-bandwidth-bound and Q6_K streams ~10 % more
+bytes/token; health 200, e2e 33/33; native `tool_calls`; thinking in
+`reasoning_content`; a **512×512** solid-red PNG answers `Red` through an
+`image_url` content part (two traps: the legacy top-level `images` field is
+ignored, and a TINY probe image undersupplies image tokens — a 10×10 PNG made
+the model claim "it's a video"; probe with ≥256 px). A prompt over the slot
+is a **400 `exceed_context_size_error`**, which is why `models.yml` sets
+`contextWindow: 200704`.
 
-Runbook: `serverAppConfigs-docs/LLM/frank-small-llama-mimo-q5km.md`.
-Rollback: [[frank-small-llama-mimo-q6k]] (Q6_K file on disk at
-`/home/dank/models/MiMo-Q6K/`); before that [[frank-small-sglang-mimo-v26]]
-(venv on disk at `/home/dank/GitPackages/sglang-cu128/`; its INT8 payload was
-staged to `~dank/.AGENTTRASH/20261002-retired-llm-payloads/` on 2026-10-02).
+Runbook: `serverAppConfigs-docs/LLM/frank-small-llama-mimo-q6k-2x200704.md`
+(the boot-pair table that prices pool changes lives in the retired
+[[frank-small-llama-mimo-q5km]]).
+Rollback: `MODEL_FILE=/home/dank/models/MiMo-Q5KM/MiMo-V2.6-Distill-Qwen-9B-Q5_K_M.gguf`
+buys back 875 MiB and the −10 % decode rate (3,571 free, 104.84 tok/s, both
+measured); the older shape `CTX_SIZE=540672 PARALLEL=3 CACHE_TYPE_V=q8_0`
+pairs with Q5 only; SGLang rollback in
+[[frank-small-sglang-mimo-v26]] (venv on disk at
+`/home/dank/GitPackages/sglang-cu128/`, INT8 payload staged to
+`~dank/.AGENTTRASH/20261002-retired-llm-payloads/`).
 
-### frank-primary — tcclaviger vLLM, Qwen3.8-Flash-Next FP8 + MTP4 (Dankserver, ACTIVE)
+### frank-primary — FreeToken (PR #587 ROCm), Qwen3.8-Flash-Next FP8 (Dankserver, ACTIVE)
 
-The **tcclaviger RDNA4 vLLM fork** (`docker.io/tcclaviger/vllm@sha256:da4029666…`,
-digest-pinned) serving **`orcarouter/Qwen3.8-Flash-Next-Uncensored-FP8`**
-(~172.8 GiB, 131 shards) on **both** Radeon AI PRO R9700s (gfx1201), **TP=2**,
-**MTP4** speculative decoding, **expert prefetch OFF**, 200,000 ctx x 3 seqs,
-**fp8 KV**. Deployed 2026-10-01 by rollback from commit `2339fae` after the
-GLM/DFlash2 experiment was declared failed (same day).
+The **FreeToken engine at PR #587 head `77a7201`** (branch
+`rocm714_torch214`, built on the host into the local image
+`localhost/frank-primary-freetoken:pr587-77a7201` by
+`scripts/docker/frank-primary-freetoken.Dockerfile` — the PR's own
+docs/install_amd.md stack: digest-pinned
+`rocm/pytorch:rocm7.14_ubuntu24.04_py3.12_pytorch_release_2.11.0` base =
+torch 2.11.0+rocm7.14/triton 3.7, `--no-build-isolation --no-deps -e .`,
+curated portable deps, `flashlib==0.3.0 --no-deps`, and a `libhiprtc.so`
+dev-symlink the pip-ROCm wheel lacks) serving
+**`orcarouter/Qwen3.8-Flash-Next-Uncensored-FP8`** (~172.8 GiB, 131 shards,
+standard modelopt block-FP8 — 75,264 `weight_scale_inv` tensors, no
+`input_scale`, loaded directly from HF safetensors, no FTW conversion) on
+**ONE R9700 (gfx1201)** — the second card sits idle. Plain `podman run`
+through `scripts/serve-frank-primary.sh` under the root systemd unit
+(`frank-primary.service`, `User=dank`, `LimitMEMLOCK=infinity`); the
+container name is the unit name (`frank-primary`; the vLLM-era
+`frank-primary-qwen` is in the legacy cleanup list). Cutover 2026-10-05;
+rollback tag `frank-primary-vllm-shapeG-20261003` (the tcclaviger vLLM trio
+and its 362-line env lore live in that tag; the tcclaviger image stays in
+dank's rootless storage).
 
-Measured 2026-09-29 tuning campaign: **decode 37.4-40.6 tok/s, acceptance
-~54 %** (MTP4 + prefetch off — the fleet's only endpoint that meets the
-35 tok/s target). 256k ctx arm measured 34.1-36.5; 2-way concurrency sums
-BELOW parity (25.49 vs 26.8 single) — slots are a capacity feature, not a
-throughput one. V-only q8 KV is NOT possible via config: stock
-`--kv-cache-dtype` is K+V together; a V-only quant needs a fork patch.
+**Shape**: all expert weights in **pinned host RAM** (~120 GiB) behind a
+GPU LRU (`--moe-cache-rate 0.10`, ~12 GiB — the single-card ceiling given
+the KV contract), the ~47.7 GiB PLE ngram table pinned in host RAM
+(→ root unit + memlock law), **KV pool `--num-tokens 299968`**
+(24 KiB/token f16 across the 12 full-attention layers ≈ 7.0 GiB; **must be
+a multiple of 64** — the auto attention backend is `qsa_sparse`, which
+overrides page size to 64; 300,000 was refused at first boot), 2 decode
+slots, `--max-output-tokens 36864` (the harness maxTokens; requests that
+omit one get this default). **Parsers are AUTO**: thinking lands in
+`reasoning_content` natively, tool calls in the qwen3.8 dialect — models.yml
+flips `reasoningContentField` to `reasoning_content` on the harness side.
+Vision works (models.yml keeps `input: [text, image]`); the vision tower's
+blocks stream from host RAM two at a time by default.
 
-The engine auto-sets the attention block size (mamba-page floor); the
-expert plan's registry prices KV per token; the expert arena (6,574 slots
-x 2.344 MiB = 15.05 GiB/rank) is the decode-critical region — cutting it
-for KV headroom collapses the offload hit rate.
+**Measured at cutover**: boot ~2.3 min cold-ish / 92 s warm → /health
+status ok; decode **17.1 tok/s** single-stream temp 0 (the vLLM incumbent
+measured ~30 tok/s on TWO cards; the KV contract caps the single-card LRU,
+and **TP=2 exists but does not boot on ROCm** — the TP allreduce JIT
+(`pynccl.cu`) can't resolve `cuda_runtime.h` for gfx1201; probed
+2026-10-06, signature in the runbook). `/metrics` is a 404 — the Dankclaw
+Prometheus llm-fleet scrape of :7770 reads down until repointed.
+`ft ctl cache --kv/--moe` resizes pools live (grow only; shrink needs a
+restart); `ft ctl stats` reports occupancy.
 
-Unit and self-healing: root systemd unit (`scripts/frank-primary.service`,
-`User=dank`, `LimitMEMLOCK=infinity`), `ExecStop=podman stop -t 120
-frank-primary-glm`. The serve script drives `podman run` directly, gates
-the digest-pinned image, the payload's `config.json` and its exact
-131-shard count and the GPU device nodes, clears stale container
-identities, then blocks on `podman wait`. `deploy_llm_primary` gates the
-same, waits out :7770, relinks/enables/restarts the unit and health-gates
-with the MainPID check. Boot cost ~7 min warm; health returns 503 while
-loading.
+**Deploy gate law (changed for FreeToken)**: `/health` returns **200 with
+`{"status":"loading"}`** while the engine loads — the FastAPI front end
+binds before the scheduler is up, so a bare `curl` false-greens (measured:
+"healthy after 6 s" mid-page-in). The deployer loops until the body says
+`"status":"ok"`; the MainPID stray-port guard is unchanged. The image gate
+is `podman image exists` (a local build carries no RepoDigests — the old
+digest grep refused exactly this shape; pinning lives in the Dockerfile).
 
-Runbook: `serverAppConfigs-docs/LLM/dankserver-qwen38-fp8-expert-offload-tuning.md`.
+Runbook: `serverAppConfigs-docs/LLM/frank-primary-freetoken-pr587.md`.
+Retired vLLM era: `serverAppConfigs-docs/LLM/dankserver-qwen38-fp8-expert-offload-tuning.md`.
 Retired GLM experiment: `serverAppConfigs-docs/LLM/frank-primary-glm53-flash-w4a16.md`.
 
 ## Key Lessons Learned
@@ -617,9 +671,30 @@ linked runbooks.
   the KV pool is worse than the OOM it prevents.
 - Host swap state is invisible to `deploy.sh` (no repo file) — check
   `swapon --show` before reasoning about headroom.
+- **Silent RAM corruption outranks every software theory.** frank-primary's
+  ~40k-token output corruption (2026-10-04/05, repetition/garbage in long
+  agent conversations) was marginal DDR5 at 4800 MT/s on four DIMMs, fixed
+  by 4600. Consumer non-ECC memory raises **no MCE** for a silent flip, and
+  a short in-band pattern check (60 s, 8 GiB of 246 GiB) is not a stability
+  proof — combined, those two instruments said "ruled out" while the RAM
+  was the cause. "4800 is the JEDEC floor on 5600-rated DIMMs" is a rating,
+  not a margin guarantee. When the fault is data-shaped (NaN states,
+  garbage tokens) and a software revert doesn't move it, price in the RAM
+  before the runtime — and remember a revert tested under the same marginal
+  hardware is itself confounded. See the runbook's 2026-10-05 section.
 
 **Runtime-specific guards**
 
+- **Rootless podman cannot RAISE rlimits, and `--ulimit nofile=1048576`
+  dies at container create, not at runtime.** runc reports
+  `error setting rlimit type 7: operation not permitted` (type 7 =
+  RLIMIT_NOFILE) when the requested hard limit exceeds the host process's
+  hard limit — on Dankclaw that ceiling is 524288. Inherited-equal limits
+  are fine, so `memlock=-1:-1` works iff the unit grants
+  `LimitMEMLOCK=infinity` (systemd is privileged; the value flows through
+  podman to runc without any raise). Upstream halogen's own run lines pass
+  NO `--ulimit` at all — copy vendor run lines before adding flags
+  (measured 2026-10-05 on the frank-large cutover).
 - **Kairic Edge** (if ever re-deployed): the pinned CK commit needs
   `third_party/composable_kernel/include/ck/config.h` generated manually
   (untracked; `git clean` removes it), and its greedy-argmax fastpath rejects
@@ -655,19 +730,26 @@ linked runbooks.
   the MAIN model's weights (`shares_model`), so there is no second copy of a
   12 GiB model in VRAM and no `-md` argument. The plain build has no MTP
   tensors at all, and pointing `--model-draft` at the file would load it twice.
-- **What can be chained, and what cannot.** `--spec-type a,b` builds every
-  enabled implementation and chains drafting through them, in a fixed internal
-  order (ngram types first, then the drafters — CLI order is irrelevant). The
-  gate is `ctx_dft`: **draftless** types (the ngram family) are added
-  unconditionally, while `draft-mtp`, `draft-dflash`, `draft-dspark` and
-  `draft-eagle3` all require a draft CONTEXT. So a chain may hold one
-  context-backed drafter plus any number of ngram types — `draft-dflash,ngram-mod`
-  is legal and measured (2.6x the drafter alone on prompt-echo traffic, byte-
-  identical to it on prose at the fleet's 48/64 window) — but
-  `draft-mtp,draft-dflash` dies at load: *"context type MTP requested but model
-  doesn't contain MTP layers"*, because MTP builds its context against whatever
-  model the draft context holds. Two context-backed types cannot share one
-  process, and MTP's head only loads when MTP **is** the draft model.
+- **What can be chained, what cannot, and what a chain costs on a hybrid.**
+  `--spec-type a,b` builds every enabled implementation and chains drafting
+  through them, in a fixed internal order (ngram types first, then the
+  drafters — CLI order is irrelevant). The gate is `ctx_dft`: **draftless**
+  types (the ngram family) are added unconditionally, while `draft-mtp`,
+  `draft-dflash`, `draft-dspark` and `draft-eagle3` all require a draft
+  CONTEXT. So a chain may hold one context-backed drafter plus any number of
+  ngram types — `draft-dflash,ngram-mod` is legal and measured (2.6x the
+  drafter alone on prompt-echo traffic, byte-identical to it on prose at the
+  fleet's 48/64 window) — but `draft-mtp,draft-dflash` dies at load: *"context
+  type MTP requested but model doesn't contain MTP layers"*, because MTP
+  builds its context against whatever model the draft context holds. Two
+  context-backed types cannot share one process, and MTP's head only loads
+  when MTP **is** the draft model. **A LEGAL chain can still lose on a
+  recurrent model:** `draft-mtp,ngram-mod` on frank-large's qwen35 hybrid
+  measured −15 % prose / −7 % block-copy (2026-10-03) — adding the ngram layer
+  logs `recurrent state rollback is not compatible with 'ngram-mod' -
+  disabling rollback support`, and MTP verification depends on that rollback,
+  so acceptance drops on ALL traffic even where the matcher fires. Measure
+  chains per architecture, never by analogy.
 - **A drafter's KV scales with the TARGET's context, and its window policy
   decides how much that costs.** The draft context is built with
   `cparams.n_ctx = llama_n_ctx(ctx_tgt)` (`common/speculative.cpp`) — there is
@@ -684,7 +766,8 @@ linked runbooks.
   trip moved the balance point FOUR layers (~4.3 GiB of headroom, 40:25 ->
   36:29), which is the practical size of the effect. See
   [[frank-medium-llamacpp-dflash2]].
-- **`--ubatch-size` has an OPTIMUM on this stack, not a plateau — and its
+- **`--ubatch-size` has an OPTIMUM on the 2x R9700 layer split, not a plateau —
+  and the curve is BACKEND-SPECIFIC, so never carry it across (next bullet). Its
   compute buffers land about TWICE as heavy on the card that holds the output
   layers and the drafter.** Measured on frank-medium 2026-09-16 at the deployed
   shape (262144 over two slots, a 64k prefill measured at 64k depth,
@@ -702,6 +785,19 @@ linked runbooks.
   its 2020 MiB pp buffer here under the DSpark drafter, when ROCm1 had 0.08 GiB
   free. That was a CONTIGUITY failure, not a total-bytes one, and no totals
   model — the sweep's or the seam's — can see it.)
+- **On the CUDA 3090 the same knob has NO interior optimum up to 2048, so
+  frank-small ships 4096/2048.** Measured 2026-10-05 by boot-pair A/B on the
+  live service: at 8,928 prompt tokens ubatch 1024 gives 3,972 / 4,077 t/s and
+  ubatch 2048 gives 4,207 / 4,212 — **+4.6 %** — for **616 MiB** (free 2,849 →
+  2,279). Decode did not move (91.9–93.0 vs 93.8–94.3, i.e. noise), and a
+  279-token prompt, which is below either ubatch, came back identical — that is
+  the mechanical proof the knob was the only variable, since a prompt shorter
+  than the ubatch cannot see it. The logical `--batch-size` was again free. And
+  the predecessor's "ubatch 2048 failed the scratch boot" was a shape
+  observation, not a law: the same file boots 2048 clean at 2 × 200,704. Bench
+  it by restarting the SERVICE through a transient `FRANK_SMALL_ENV_FILE`
+  drop-in, never a scratch port — one card holds one server. See
+  [[frank-small-llama-mimo-q6k-2x200704]].
 - **One `-ts` step is one LAYER, and a hybrid stack's layers are not the same
   size.** qwen35 is 16 x (3 Gated DeltaNet + 1 Gated Attention), so a step is
   worth ~0.43 GiB on a DeltaNet boundary and ~1.43 GiB on a full-attention one
@@ -728,7 +824,13 @@ linked runbooks.
   the harness would discover the ceiling as a 400 on a long conversation. Side
   effect worth knowing: the shorter slot made decode ~5 % FASTER (106.8 vs
   100.4 tok/s) — less attention to scan and one fewer DeltaNet state resident.
-  Measured 2026-10-02. See [[frank-small-llama-mimo-q5km]].
+  The 2026-10-03 reslice went the other way (3 → 2 slots, 180,224 → 200,704,
+  pool 401,408) to fund unquantized f16 V, and `contextWindow` followed again;
+  decode landed 104.8 tok/s — the ~2 % off 106.8 is the exact-V tax. The same
+  day the freed space also bought the Q6_K rung (now 94.4 tok/s — the rung
+  itself is a further measured −10 %, weight-bandwidth, not slot geometry).
+  All moves measured; both slot directions cost or buy window, never pool.
+  See [[frank-small-llama-mimo-q6k-2x200704]].
 - **Price a VRAM budget from two boots, never from a formula — and an mmproj
   costs ~1.5x its file size.** The frank-small 3090 fit was solved from two
   measurements: 2 × 262,144 with no projector = 20,190 MiB used (KV 12,800 ⇒
@@ -847,9 +949,9 @@ Active:
 
 | Service | Runbook |
 |---------|---------|
-| frank-large (ROCmFPX llama.cpp fork, Qwen3.8-27B Q8_0_ROCMFPX_AGENT + MTP4 + vision, Dankclaw) | `serverAppConfigs-docs/LLM/frank-large-rocmfpx-q8agent-mtp.md` |
-| frank-small (llama.cpp CUDA, MiMo-V2.6-Distill-Qwen-9B Q5_K_M, Dankclaw) | `serverAppConfigs-docs/LLM/frank-small-llama-mimo-q5km.md` |
-| frank-primary (tcclaviger vLLM, Qwen3.8-Flash-Next FP8 + MTP4, Dankserver) | `serverAppConfigs-docs/LLM/dankserver-qwen38-fp8-expert-offload-tuning.md` |
+| frank-large (halogen container, Qwen3.8-27B .hgn + DFlash2, 1×200,700, text-only, Dankclaw) | `serverAppConfigs-docs/LLM/frank-large-halogen27b-dflash2.md` |
+| frank-small (llama.cpp CUDA, MiMo-V2.6-Distill-Qwen-9B Q6_K, 2×200,704 f16 KV, Dankclaw) | `serverAppConfigs-docs/LLM/frank-small-llama-mimo-q6k-2x200704.md` |
+| frank-primary (FreeToken PR #587 ROCm, Qwen3.8-Flash-Next FP8, 1×R9700, Dankserver) | `serverAppConfigs-docs/LLM/frank-primary-freetoken-pr587.md` |
 | Dankclaw kernel posture (`amd_iommu=off`) | `serverAppConfigs-docs/LLM/dankclaw-iommu-translation-off.md` |
 | Deployment flow + incidents | `serverAppConfigs-docs/LLM/deployment.md` |
 
@@ -857,9 +959,13 @@ Retired (rollback targets and history):
 
 | Runtime | Host | Window | Runbook |
 |---------|------|--------|---------|
-| gufo v0.1.1 Qwen3.8-27B UD-Q8_K_XL/UD-Q4_K_XL + DFlash2 (`frank-large`) | Dankclaw | 2026-09-28 → 10-01, again 10-01 → 10-02 (rollback target) | `frank-large-gufo-qwen38-27b.md` |
+| ROCmFPX llama.cpp fork Qwen3.8-27B Uncensored Q6_0_ROCMFPX_AGENT + MTP (`frank-large`) | Dankclaw | 2026-10-03 → 10-05 (rollback target, payload + fork build on disk) | `frank-large-rocmfpx-q6agent-uncensored-mtp.md` |
+| tcclaviger vLLM fork Qwen3.8-Flash-Next-Uncensored FP8, TP=2, shape G (`frank-primary`) | Dankserver | 2026-09-29 → 10-05 (rollback target — tag `frank-primary-vllm-shapeG-20261003`; trio + env lore in the tag, image in rootless storage) | `dankserver-qwen38-fp8-expert-offload-tuning.md` |
+| ROCmFPX llama.cpp fork Qwen3.8-27B Q8_0_ROCMFPX_AGENT + MTP (`frank-large`) | Dankclaw | 2026-10-02 → 10-03 (rollback target, payload on disk) | `frank-large-rocmfpx-q8agent-mtp.md` |
+| gufo v0.1.1 Qwen3.8-27B UD-Q8_K_XL/UD-Q4_K_XL + DFlash2 (`frank-large`) | Dankclaw | 2026-09-28 → 10-01, again 10-01 → 10-02 (payloads removed from disk 2026-10-03; rollback = re-download) | `frank-large-gufo-qwen38-27b.md` |
 | vLLM TheRock MiMo-V2.6-Distill-Qwen-9B BF16 + cross-model MTP (`frank-large`) | Dankclaw | 2026-10-01 (same-day rollback to gufo) | `frank-large-vllm-mimo-v26-bf16.md` |
-| llama.cpp master MiMo-V2.6-Distill-Qwen-9B Q6_K (`frank-small`) | Dankclaw | 2026-10-01 → 10-02 (rollback target) | `frank-small-llama-mimo-q6k.md` |
+| llama.cpp master MiMo-V2.6-Distill-Qwen-9B Q6_K (`frank-small`) | Dankclaw | 2026-10-01 → 10-02 (the rung returned ACTIVE 2026-10-03 at the 2×200,704 shape — `frank-small-llama-mimo-q6k-2x200704.md`) | `frank-small-llama-mimo-q6k.md` |
+| llama.cpp master MiMo-V2.6-Distill-Qwen-9B Q5_K_M 2×200,704 f16 KV (`frank-small`) | Dankclaw | 2026-10-02 → 10-03 (rollback target — one `MODEL_FILE` line, buys back 875 MiB + the −10 % rung tax; owns the boot-pair table) | `frank-small-llama-mimo-q5km.md` |
 | SGLang MiMo-V2.6 INT8-W8A16-AutoRound (`frank-small`) | Dankclaw | 2026-10-01 (same-day swap to the GGUF path) | `frank-small-sglang-mimo-v26.md` |
 |tcclaviger vLLM GLM-5.3-Flash W4A16 int4, +/- DFlash2 drafter (`frank-primary`)|Dankserver|2026-09-30 → 10-01|`frank-primary-glm53-flash-w4a16.md`|
 | llama.cpp b11030 Qwen3.8-27B UD-Q8_K_XL + DFlash2 (`frank-large`) | Dankclaw | 2026-09-21 → 09-28 | `frank-large-unsloth-qwen38-27b-q8xl.md` |

autoresearch.sh

@@ -29,7 +29,10 @@ SCRIPT_DIR="$(cd -- "$(dirname -- "${BASH_SOURCE[0]}")" && pwd)"
 HOST="${FRANK_PRIMARY_AR_HOST:-dankserver}"
 STAGE="/home/dank/.autoresearch-frank-primary"
 
-ENV_FILE="$SCRIPT_DIR/scripts/frank-primary.env"
+# FRANK_PRIMARY_AR_ENV_FILE stages a DIFFERENT env than the repo's — that is how
+# the arms driver A/Bs engine knobs without mutating the checkout. The
+# fingerprint covers whatever file is staged, so the memo cannot cross arms.
+ENV_FILE="${FRANK_PRIMARY_AR_ENV_FILE:-$SCRIPT_DIR/scripts/frank-primary.env}"
 SERVE="$SCRIPT_DIR/scripts/serve-frank-primary.sh"
 UNIT="$SCRIPT_DIR/scripts/frank-primary.service"
 MEASURE="$SCRIPT_DIR/scripts/autoresearch/frank-primary-measure.py"
@@ -58,6 +61,13 @@ print("METRIC ttft_ms_4096=%s" % m["ttft_ms_4096"])
 print("METRIC kv_tokens=%s" % (run.get("kv_tokens") or 0))
 print("METRIC boot_seconds=%s" % (run.get("boot_seconds") or -1))
 print("METRIC completion_tokens_4096=%s" % m.get("completion_tokens_4096", 0))
+# Any extra sampling temperature the measure swept (AR_TEMPS), e.g. the 0.7
+# arm — that is where speculative-decode acceptance shows up, since a greedy
+# draft matches a greedy target exactly and is rejected more under real
+# sampling. Suffixed keys only (_t0.7), so the historical lines are untouched.
+for k in sorted(m):
+    if "_t" in k and (k.startswith("decode_tps") or k.startswith("ttft_ms")):
+        print("METRIC %s=%s" % (k, m[k]))
 ' "$1"
 }
 
@@ -101,4 +111,7 @@ run_json="$(ssh_run "cat '$STAGE/run.json'")" || {
 echo "[autoresearch] window result:" >&2
 printf '%s\n' "$run_json" >&2
 emit_metrics "$run_json"
+
+echo "[autoresearch] engine evidence (plan / compile path / spec counters):" >&2
+ssh_run "cat '$STAGE/evidence.log' 2>/dev/null" >&2 || true
 exit 0

prose_prompt.txt

@@ -0,0 +1,86 @@
+# Subsurface Aquifer Monitoring Network - System Design Document (Revision 7)
+
+This document describes the deployment of a subsurface aquifer monitoring network for the Cascadia sedimentary basin. It is written for the operations crew and the regional water authority. Every sensor node, relay, aggregation service, calibration procedure, and failure mode below carries a stable alphanumeric identifier that appears in every table and in the field notes.
+
+## Sensor nodes
+Node SN-000-08B9 sits in a discharge lens at 359.0 m below datum and reports a salinity of 2.38 ppt with the fractured interval reading 5.40 degC at dawn and the wellhead porous 1.72 m3/d while the electrode pair returns 3157 uS/cm.
+Node SN-001-0A65 sits in a percolate lens at 25.9 m below datum and steps down to a salinity of 2.13 ppt with the diapir interval reading 8.55 degC at dawn and the wellhead silty 0.67 m3/d while the electrode pair returns 979 uS/cm.
+Node SN-002-0571 sits in a diapir lens at 287.8 m below datum and peaks at a salinity of 7.63 ppt with the karst interval reading 4.26 degC at dawn and the wellhead spooned 11.50 m3/d while the electrode pair returns 2987 uS/cm.
+Node SN-003-0CE9 sits in a confining lens at 169.5 m below datum and logs a salinity of 0.96 ppt with the alluvial interval reading 5.03 degC at dawn and the wellhead salinized 12.08 m3/d while the electrode pair returns 2607 uS/cm.
+Node SN-004-0E2A sits in a freshened lens at 70.4 m below datum and settles near a salinity of 8.68 ppt with the spooned interval reading 4.88 degC at dawn and the wellhead siphon 4.38 m3/d while the electrode pair returns 2707 uS/cm.
+Node SN-005-0DDD sits in a percolate lens at 44.6 m below datum and reports a salinity of 6.25 ppt with the alluvial interval reading 10.85 degC at dawn and the wellhead brackish 13.99 m3/d while the electrode pair returns 3618 uS/cm.
+Node SN-006-0EFC sits in a till lens at 309.5 m below datum and reports a salinity of 3.49 ppt with the lacustrine interval reading 7.38 degC at dawn and the wellhead spooned 3.21 m3/d while the electrode pair returns 1253 uS/cm.
+Node SN-007-0B02 sits in a recharge lens at 262.0 m below datum and steps down to a salinity of 2.38 ppt with the discharge interval reading 8.18 degC at dawn and the wellhead perched 4.05 m3/d while the electrode pair returns 3900 uS/cm.
+Node SN-008-0448 sits in a karst lens at 406.5 m below datum and reports a salinity of 7.32 ppt with the upconing interval reading 6.17 degC at dawn and the wellhead fluvial 0.75 m3/d while the electrode pair returns 1448 uS/cm.
+Node SN-009-043C sits in a alluvial lens at 159.3 m below datum and flags a salinity of 6.20 ppt with the percolate interval reading 7.60 degC at dawn and the wellhead brackish 13.01 m3/d while the electrode pair returns 2025 uS/cm.
+Node SN-010-0FF5 sits in a karst lens at 264.2 m below datum and flags a salinity of 7.05 ppt with the freshened interval reading 7.88 degC at dawn and the wellhead siphon 8.41 m3/d while the electrode pair returns 1635 uS/cm.
+Node SN-011-098A sits in a drain lens at 365.7 m below datum and reports a salinity of 8.11 ppt with the wedge interval reading 5.51 degC at dawn and the wellhead glacial 2.52 m3/d while the electrode pair returns 2916 uS/cm.
+Node SN-012-0AE8 sits in a till lens at 259.6 m below datum and steps down to a salinity of 9.13 ppt with the recharge interval reading 11.60 degC at dawn and the wellhead spooned 0.46 m3/d while the electrode pair returns 3077 uS/cm.
+Node SN-013-0B83 sits in a salinized lens at 171.2 m below datum and peaks at a salinity of 2.83 ppt with the lacustrine interval reading 5.56 degC at dawn and the wellhead discharge 0.35 m3/d while the electrode pair returns 3083 uS/cm.
+Node SN-014-032E sits in a drain lens at 249.6 m below datum and flags a salinity of 1.09 ppt with the salinized interval reading 9.58 degC at dawn and the wellhead perched 12.00 m3/d while the electrode pair returns 2221 uS/cm.
+Node SN-015-004F sits in a aeolian lens at 264.4 m below datum and reports a salinity of 7.34 ppt with the salinized interval reading 8.76 degC at dawn and the wellhead perched 0.31 m3/d while the electrode pair returns 1483 uS/cm.
+Node SN-016-07C6 sits in a aeolian lens at 155.9 m below datum and reports a salinity of 0.64 ppt with the mineralized interval reading 11.75 degC at dawn and the wellhead diluted 13.46 m3/d while the electrode pair returns 524 uS/cm.
+Node SN-017-06C5 sits in a salinized lens at 72.1 m below datum and averages a salinity of 4.52 ppt with the siphon interval reading 8.93 degC at dawn and the wellhead confining 3.98 m3/d while the electrode pair returns 3687 uS/cm.
+Node SN-018-0E64 sits in a siphon lens at 334.9 m below datum and steps down to a salinity of 6.73 ppt with the salinized interval reading 7.63 degC at dawn and the wellhead freshened 9.64 m3/d while the electrode pair returns 1681 uS/cm.
+Node SN-019-0122 sits in a spooned lens at 117.2 m below datum and logs a salinity of 3.24 ppt with the fractured interval reading 9.26 degC at dawn and the wellhead karst 3.50 m3/d while the electrode pair returns 1065 uS/cm.
+Node SN-020-021D sits in a karst lens at 414.3 m below datum and recovers to a salinity of 0.76 ppt with the glacial interval reading 6.25 degC at dawn and the wellhead silty 9.60 m3/d while the electrode pair returns 1041 uS/cm.
+Node SN-021-06C6 sits in a percolate lens at 379.2 m below datum and drifts to a salinity of 7.61 ppt with the wedge interval reading 5.84 degC at dawn and the wellhead karst 1.65 m3/d while the electrode pair returns 1913 uS/cm.
+Node SN-022-0DC9 sits in a spooned lens at 327.1 m below datum and reports a salinity of 9.25 ppt with the freshened interval reading 5.05 degC at dawn and the wellhead silty 5.90 m3/d while the electrode pair returns 1544 uS/cm.
+Node SN-023-0717 sits in a karst lens at 222.0 m below datum and steps down to a salinity of 4.02 ppt with the alluvial interval reading 6.60 degC at dawn and the wellhead siphon 3.77 m3/d while the electrode pair returns 3892 uS/cm.
+Node SN-024-02A9 sits in a fractured lens at 479.7 m below datum and drifts to a salinity of 7.88 ppt with the silty interval reading 12.53 degC at dawn and the wellhead discharge 13.18 m3/d while the electrode pair returns 3592 uS/cm.
+Node SN-025-0C8F sits in a wedge lens at 434.3 m below datum and holds a salinity of 1.63 ppt with the alluvial interval reading 4.22 degC at dawn and the wellhead till 5.73 m3/d while the electrode pair returns 3905 uS/cm.
+Node SN-026-04BF sits in a colluvial lens at 460.0 m below datum and averages a salinity of 9.36 ppt with the diapir interval reading 8.98 degC at dawn and the wellhead brackish 10.29 m3/d while the electrode pair returns 802 uS/cm.
+Node SN-027-0DA4 sits in a silty lens at 265.7 m below datum and steps down to a salinity of 7.06 ppt with the percolate interval reading 4.69 degC at dawn and the wellhead freshened 8.42 m3/d while the electrode pair returns 2201 uS/cm.
+Node SN-028-01EB sits in a perched lens at 49.5 m below datum and recovers to a salinity of 1.83 ppt with the silty interval reading 9.32 degC at dawn and the wellhead drain 9.69 m3/d while the electrode pair returns 1126 uS/cm.
+Node SN-029-042C sits in a karst lens at 30.6 m below datum and averages a salinity of 0.86 ppt with the percolate interval reading 9.85 degC at dawn and the wellhead recharge 8.16 m3/d while the electrode pair returns 1452 uS/cm.
+
+## Relays
+Relay RL-00-02181 reburses the upconing trunk from the brackish margin and peaks at a round-trip time of 131.8 ms with a frame loss of 1.04 percent across the night window.
+Relay RL-01-00262 reburses the spooned trunk from the fluvial margin and holds a round-trip time of 372.2 ms with a frame loss of 3.65 percent across the night window.
+Relay RL-02-03691 reburses the recharge trunk from the percolate margin and steps down to a round-trip time of 398.5 ms with a frame loss of 0.29 percent across the night window.
+Relay RL-03-0E117 reburses the lacustrine trunk from the confining margin and averages a round-trip time of 373.9 ms with a frame loss of 3.44 percent across the night window.
+Relay RL-04-0B41B reburses the aeolian trunk from the colluvial margin and peaks at a round-trip time of 70.7 ms with a frame loss of 3.25 percent across the night window.
+Relay RL-05-0D12D reburses the recharge trunk from the diluted margin and steps down to a round-trip time of 264.7 ms with a frame loss of 0.03 percent across the night window.
+Relay RL-06-043B4 reburses the colluvial trunk from the upconing margin and reports a round-trip time of 49.5 ms with a frame loss of 3.42 percent across the night window.
+Relay RL-07-09AD6 reburses the fractured trunk from the freshened margin and averages a round-trip time of 225.3 ms with a frame loss of 1.06 percent across the night window.
+Relay RL-08-04394 reburses the brackish trunk from the fluvial margin and steps down to a round-trip time of 88.6 ms with a frame loss of 2.47 percent across the night window.
+Relay RL-09-017A0 reburses the wedge trunk from the lacustrine margin and drifts to a round-trip time of 363.0 ms with a frame loss of 3.30 percent across the night window.
+Relay RL-10-0217D reburses the mineralized trunk from the lacustrine margin and peaks at a round-trip time of 26.2 ms with a frame loss of 1.30 percent across the night window.
+Relay RL-11-08F97 reburses the perched trunk from the freshened margin and logs a round-trip time of 181.8 ms with a frame loss of 2.75 percent across the night window.
+Relay RL-12-02628 reburses the fractured trunk from the glacial margin and steps down to a round-trip time of 378.5 ms with a frame loss of 2.69 percent across the night window.
+Relay RL-13-06E06 reburses the silty trunk from the mineralized margin and flags a round-trip time of 153.4 ms with a frame loss of 2.15 percent across the night window.
+
+## Aggregation services
+Aggregator AG-00-3BB logs a queue depth of 5.2 messages and a histogram bucket of 6.82 seconds before the dedupe pass settles the stream for the next rotation.
+Aggregator AG-01-0FF reports a queue depth of 86.4 messages and a histogram bucket of 4.10 seconds before the dedupe pass settles the stream for the next rotation.
+Aggregator AG-02-37F drifts to a queue depth of 34.7 messages and a histogram bucket of 10.25 seconds before the dedupe pass settles the stream for the next rotation.
+Aggregator AG-03-3B3 averages a queue depth of 93.5 messages and a histogram bucket of 13.59 seconds before the dedupe pass settles the stream for the next rotation.
+Aggregator AG-04-33E flags a queue depth of 83.2 messages and a histogram bucket of 17.58 seconds before the dedupe pass settles the stream for the next rotation.
+Aggregator AG-05-2F2 settles near a queue depth of 84.8 messages and a histogram bucket of 0.84 seconds before the dedupe pass settles the stream for the next rotation.
+Aggregator AG-06-2AD averages a queue depth of 75.4 messages and a histogram bucket of 7.65 seconds before the dedupe pass settles the stream for the next rotation.
+Aggregator AG-07-06E drifts to a queue depth of 26.5 messages and a histogram bucket of 14.26 seconds before the dedupe pass settles the stream for the next rotation.
+Aggregator AG-08-0E3 averages a queue depth of 83.9 messages and a histogram bucket of 15.51 seconds before the dedupe pass settles the stream for the next rotation.
+Aggregator AG-09-138 averages a queue depth of 78.6 messages and a histogram bucket of 8.50 seconds before the dedupe pass settles the stream for the next rotation.
+
+## Calibration procedures
+Procedure CP-00-047 steps the probe through a two-point rinse, then applies a trim of 0.2084 at the 66 milli-bar mark and records a residual of 0.02052 before the next field seal is crimped.
+Procedure CP-01-0A4 steps the probe through a two-point rinse, then applies a trim of 0.7793 at the 96 milli-bar mark and records a residual of 0.01468 before the next field seal is crimped.
+Procedure CP-02-054 steps the probe through a two-point rinse, then applies a trim of 0.4634 at the 68 milli-bar mark and records a residual of 0.04234 before the next field seal is crimped.
+Procedure CP-03-02D steps the probe through a two-point rinse, then applies a trim of 0.8459 at the 12 milli-bar mark and records a residual of 0.04855 before the next field seal is crimped.
+Procedure CP-04-098 steps the probe through a two-point rinse, then applies a trim of 0.5267 at the 97 milli-bar mark and records a residual of 0.00287 before the next field seal is crimped.
+Procedure CP-05-09B steps the probe through a two-point rinse, then applies a trim of 0.3968 at the 72 milli-bar mark and records a residual of 0.01637 before the next field seal is crimped.
+Procedure CP-06-030 steps the probe through a two-point rinse, then applies a trim of 0.8860 at the 12 milli-bar mark and records a residual of 0.04508 before the next field seal is crimped.
+Procedure CP-07-0EC steps the probe through a two-point rinse, then applies a trim of 0.2367 at the 70 milli-bar mark and records a residual of 0.00108 before the next field seal is crimped.
+Procedure CP-08-0BC steps the probe through a two-point rinse, then applies a trim of 0.7278 at the 68 milli-bar mark and records a residual of 0.04707 before the next field seal is crimped.
+Procedure CP-09-0AA steps the probe through a two-point rinse, then applies a trim of 0.6069 at the 37 milli-bar mark and records a residual of 0.00443 before the next field seal is crimped.
+
+## Failure modes
+Mode FM-00-27 declares a fluvial excursion when two independent probes disagree beyond the band for three rotations; the fleet reports a mean time between failures of 36902 hours for this class of excursion.
+Mode FM-01-13 declares a drain excursion when two independent probes disagree beyond the band for three rotations; the fleet peaks at a mean time between failures of 29042 hours for this class of excursion.
+Mode FM-02-14 declares a till excursion when two independent probes disagree beyond the band for three rotations; the fleet peaks at a mean time between failures of 26976 hours for this class of excursion.
+Mode FM-03-0C declares a diapir excursion when two independent probes disagree beyond the band for three rotations; the fleet drifts to a mean time between failures of 22578 hours for this class of excursion.
+Mode FM-04-39 declares a perched excursion when two independent probes disagree beyond the band for three rotations; the fleet recovers to a mean time between failures of 27383 hours for this class of excursion.
+Mode FM-05-19 declares a siphon excursion when two independent probes disagree beyond the band for three rotations; the fleet logs a mean time between failures of 23152 hours for this class of excursion.
+
+## End of document
+Summarize the network's dominant risk in one paragraph, then list the three node identifiers whose readings most limit the model this month.

scripts/autoresearch/frank-primary-arms.sh

@@ -0,0 +1,176 @@
+#!/usr/bin/env bash
+# frank-primary-arms.sh — run a SEQUENCE of frank-primary bench arms and keep
+# every arm's evidence. Runs on the operator box; each arm is one full window
+# of ./autoresearch.sh (stop+mask -> boot the arm as the service -> measure ->
+# restore production and verify), so the fleet is only ever down for one arm at
+# a time and the production unit is back up between arms.
+#
+# An arm is the repo's own frank-primary.env plus an OVERLAY of engine knobs
+# appended after it (later assignment wins when the serve script sources it),
+# staged through FRANK_PRIMARY_AR_ENV_FILE so the checkout is never mutated and
+# each arm gets its own fingerprint (the memo can never answer one arm with
+# another arm's numbers).
+#
+# Usage:  scripts/autoresearch/frank-primary-arms.sh [arm ...]
+#         scripts/autoresearch/frank-primary-arms.sh --list
+# Output, under /tmp/frank-primary-arms-<UTC stamp>/ (override with ARMS_DIR):
+#
+#   <arm>.out        full autoresearch output (metrics + engine evidence)
+#   <arm>.env        the exact staged env
+#   <arm>.rc         window exit code
+#   summary.tsv      arm, rc, decode_tps, decode_tps_t0.7, kv_tokens, boot_s
+#
+# Exit 0 when every arm produced metrics; 1 if any arm failed (a failed boot is
+# a RESULT — the window still restored production, and the rc says which arm).
+set -uo pipefail
+
+SCRIPT_DIR="$(cd -- "$(dirname -- "${BASH_SOURCE[0]}")" && pwd)"
+REPO="$(cd -- "$SCRIPT_DIR/../.." && pwd)"
+BASE_ENV="$REPO/scripts/frank-primary.env"
+ARMS_DIR="${ARMS_DIR:-/tmp/frank-primary-arms-$(date -u +%Y%m%dT%H%M%SZ)}"
+
+# --- arms ------------------------------------------------------------------
+# baseline: the shipped shape E, measured with THIS harness so the comparison is
+# matched (the ~29.6 t/s figure in the docs is an older shape, and no shape-E
+# decode number exists yet).
+OVER_baseline=()
+# B: MTP3 with the vendor's probabilistic draft sampling, a capture ladder whose
+# step is num_speculative_tokens + 1 = 4, and the sampling override. Precise
+# memory stays ON — this arm is simultaneously the MTP test and the test of
+# whether MTP's drafter workspace and the fitted 5.25 GiB overhead can coexist.
+SPEC3='{"method":"mtp","num_speculative_tokens":3,"draft_sample_method":"probabilistic"}'
+LADDER3='{"cudagraph_capture_sizes":[4,8,12],"max_cudagraph_capture_size":12,"inductor_compile_config":{"combo_kernels":false,"benchmark_combo_kernel":false}}'
+# The value must arrive in the staged env file SINGLE-QUOTED: sourcing an env
+# file performs quote removal, so an unquoted {"temperature":0.7} would reach
+# the engine as {temperature:0.7}.
+GEN07='{"temperature":0.7}'
+OVER_B=(
+    "FRANK_PRIMARY_SPEC='$SPEC3'"
+    "FRANK_PRIMARY_COMPCONFIG='$LADDER3'"
+    'FRANK_PRIMARY_GEN_CONFIG='"'$GEN07'"
+)
+# B2: identical to B with the default (7.60 GiB) overhead, i.e. precise memory
+# OFF. It is the fallback if B dies at graph capture, and it prices the arena
+# that precise memory buys (7,522 slots -> ~6,496).
+OVER_B2=(
+    "FRANK_PRIMARY_SPEC='$SPEC3'"
+    "FRANK_PRIMARY_COMPCONFIG='$LADDER3'"
+    'FRANK_PRIMARY_GEN_CONFIG='"'$GEN07'"
+    'FRANK_PRIMARY_PRECISE_MEMORY='
+)
+# C: the vendor's concurrency shape — SEQS 16 with the [4..32] ladder and util
+# 0.95. This is also the direct test of the theory that pinning the capture
+# ladder unlocks SEQS > 3 (the SEQS=2 failure was query_start_loc specialised to
+# the constant 3, which is NOT in the default ladder; 4 is).
+LADDER16='{"cudagraph_capture_sizes":[4,8,12,16,20,24,28,32],"max_cudagraph_capture_size":32,"inductor_compile_config":{"combo_kernels":false,"benchmark_combo_kernel":false}}'
+OVER_C=(
+    "FRANK_PRIMARY_SPEC='$SPEC3'"
+    "FRANK_PRIMARY_COMPCONFIG='$LADDER16'"
+    'FRANK_PRIMARY_GEN_CONFIG='"'$GEN07'"
+    'FRANK_PRIMARY_SEQS=16'
+    'FRANK_PRIMARY_MNBT=4096'
+    'FRANK_PRIMARY_UTIL=0.95'
+)
+
+# D/E: the CONTEXT arms. Measured 2026-10-02 (arms run 20261002T233855Z): the
+# KV pool is a hard byte request divided by a PER-TOKEN PRICE, and MTP raises
+# that price because the drafter keeps its own KV per position:
+#   no spec: 3.15e9 / 398,658 tokens =  7,901 B/token  (1.99x of 200,704)
+#   MTP3:    3.15e9 / 313,002 tokens = 10,064 B/token  (1.56x)
+# So arm C lost a fifth of the context to the drafter, not to util. Frank's
+# contract is >=401,400 tokens over >=2 seqs = 2.00x of 200,704, which at
+# 10,064 B/token costs 4.04e9 -> ask 4.05e9 (+0.83 GiB/rank).
+# D is arm B with EXACTLY ONE variable changed (the KV bytes), so its result
+# says whether the 4k-depth collapse (B 23.09 vs baseline 33.99) was KV or the
+# batch shape.
+KV401K='FRANK_PRIMARY_KV_BYTES=4050000000'
+OVER_D=(
+    "FRANK_PRIMARY_SPEC='$SPEC3'"
+    "FRANK_PRIMARY_COMPCONFIG='$LADDER3'"
+    'FRANK_PRIMARY_GEN_CONFIG='"'$GEN07'"
+    "$KV401K"
+)
+# E: D with C's max_num_batched_tokens (4096 — C is the only arm that did NOT
+# collapse at 4k depth: 33.15 where B gave 23.09) and util 0.99, the value
+# Frank asked about. 0.99 is affordable on paper — production at util 0.98
+# measures 30.37/34.21 GiB per card, 3.58 GiB physically unused — but it leaves
+# only 0.32 GiB above the allowance while the CUDA-graph cost is an ANALYTIC
+# estimate (C's was 1.47 GiB over 24 graphs), so it is measured, not assumed.
+OVER_E=(
+    "FRANK_PRIMARY_SPEC='$SPEC3'"
+    "FRANK_PRIMARY_COMPCONFIG='$LADDER3'"
+    'FRANK_PRIMARY_GEN_CONFIG='"'$GEN07'"
+    "$KV401K"
+    'FRANK_PRIMARY_MNBT=4096'
+    'FRANK_PRIMARY_UTIL=0.99'
+)
+
+arm_overlay() {
+    case "$1" in
+        baseline) ;;                       # the shipped env, no overlay
+        B)  printf '%s\n' "${OVER_B[@]}" ;;
+        B2) printf '%s\n' "${OVER_B2[@]}" ;;
+        C)  printf '%s\n' "${OVER_C[@]}" ;;
+        D)  printf '%s\n' "${OVER_D[@]}" ;;
+        E)  printf '%s\n' "${OVER_E[@]}" ;;
+        *) return 1 ;;
+    esac
+}
+if [ "${1:-}" = "--list" ]; then
+    printf 'baseline B B2 C D E\n'
+    exit 0
+fi
+
+[ -f "$BASE_ENV" ] || { echo "[arms] missing $BASE_ENV" >&2; exit 1; }
+if [ "$#" -gt 0 ]; then ARMS=("$@"); else ARMS=(baseline B B2 C); fi
+mkdir -p "$ARMS_DIR"
+echo "[arms] dir $ARMS_DIR  arms: ${ARMS[*]}" >&2
+
+: >"$ARMS_DIR/summary.tsv"
+printf 'arm\trc\tdecode_tps\tdecode_tps_t0.7\tkv_tokens\tboot_s\n' >>"$ARMS_DIR/summary.tsv"
+overall=0
+
+for arm in "${ARMS[@]}"; do
+    if ! arm_overlay "$arm" >"$ARMS_DIR/$arm.overlay"; then
+        echo "[arms] unknown arm: $arm" >&2
+        overall=1
+        continue
+    fi
+    { cat "$BASE_ENV"; echo; echo "# --- arm $arm overlay ($(date -u +%FT%TZ)) ---"
+      arm_overlay "$arm"; } >"$ARMS_DIR/$arm.env"
+    # Gate the staged env BEFORE burning a ~20-minute window: every knob that
+    # holds JSON must still be one valid JSON object after the file is SOURCED
+    # (that is the step that silently eats unquoted braces).
+    if ! bash -c 'set -a; . "$1" >/dev/null; set +a
+        for k in FRANK_PRIMARY_SPEC FRANK_PRIMARY_COMPCONFIG FRANK_PRIMARY_GEN_CONFIG; do
+            v="${!k:-}"
+            [ -n "$v" ] || continue
+            case "$v" in
+                "{"*) python3 -c "import json,sys; json.loads(sys.argv[1])" "$v" \
+                        || { echo "[arms] $k is not valid JSON: $v" >&2; exit 1; } ;;
+                *) echo "[arms] $k is not a JSON object: $v" >&2; exit 1 ;;
+            esac
+        done' _ "$ARMS_DIR/$arm.env"; then
+        echo "[arms] $arm: staged env rejected, window not opened" >&2
+        echo 99 >"$ARMS_DIR/$arm.rc"
+        overall=1
+        continue
+    fi
+
+    echo "[arms] === $arm: opening window $(date -u +%H:%M:%S)Z ===" >&2
+    FRANK_PRIMARY_AR_ENV_FILE="$ARMS_DIR/$arm.env" \
+        "$REPO/autoresearch.sh" >"$ARMS_DIR/$arm.out" 2>&1
+    rc=$?
+    echo "$rc" >"$ARMS_DIR/$arm.rc"
+    echo "[arms] === $arm: rc=$rc $(date -u +%H:%M:%S)Z ===" >&2
+    grep -E '^METRIC ' "$ARMS_DIR/$arm.out" | sed 's/^METRIC //' >"$ARMS_DIR/$arm.metrics" || true
+    metric() { local v; v="$(sed -n "s/^$1=//p" "$ARMS_DIR/$arm.metrics" | tail -1)"; printf '%s' "${v:-$2}"; }
+    printf '%s\t%s\t%s\t%s\t%s\t%s\n' "$arm" "$rc" \
+        "$(metric decode_tps FAIL)" "$(metric decode_tps_t0.7 FAIL)" \
+        "$(metric kv_tokens ?)" "$(metric boot_seconds ?)" >>"$ARMS_DIR/summary.tsv"
+    [ "$rc" -ne 0 ] && overall=1
+done
+
+echo "[arms] summary:" >&2
+cat "$ARMS_DIR/summary.tsv" >&2
+exit "$overall"

scripts/autoresearch/frank-primary-measure.py

@@ -19,14 +19,18 @@ Workload (fixed, deterministic, offline):
     the server's own /v1/messages/count_tokens counter
   * depths 115 / 1056 / 4096 target prompt tokens
   * one 64-token warm pass per depth (warms the expert arena), then
-    max_tokens=256 at temperature 0, 3 runs, median
-Everything is greedy with a fixed seed — the same workload every run, no
-network, no time-of-day dependence.
+    max_tokens=256, 3 runs, median — repeated for every temperature in AR_TEMPS
+    (default 0 and 0.7). The prompt workload is deterministic (fixed seed, no
+    network, no time-of-day dependence); at temperature > 0 the generated text
+    is not, but the measured window is bounded by max_tokens, so the rate stays
+    comparable and the non-determinism is exactly the sampling that a
+    speculative-decode arm must survive.
 
 Env:
   AR_BASE             endpoint (default http://127.0.0.1:7770)
   AR_MODEL            served name (default frank-primary)
   AR_PROMPT_CEILING   hard prompt ceiling in tokens (default 100000)
+  AR_TEMPS            comma-separated sampling temperatures (default "0,0.7")
 
 Output: progress on stderr, ONE json object on stdout (the last line).
 Exit codes: 0 ok, 2 endpoint unhealthy / no usable completion, 3 bad usage.
@@ -47,6 +51,14 @@ DEPTHS = (115, 1056, 4096)
 WARM_TOKENS = 64
 MAX_TOKENS = 256
 RUNS = 3
+# Temperature sweep. The unsuffixed keys are temperature 0 — byte-comparable to
+# every historical measurement — and AR_TEMPS adds more. The second one matters
+# for speculative decode: draft ACCEPTANCE is sampling-dependent (a greedy draft
+# matches a greedy target exactly and is rejected more under a non-greedy
+# target), so an MTP arm judged only at temp 0 would be judged on half the
+# contract. Production traffic is non-zero temperature (the payload's
+# generation_config says 1.0; an override to 0.7 is under test).
+TEMPS = tuple(float(x) for x in os.environ.get("AR_TEMPS", "0,0.7").split(","))
 
 # 43-word x 12 = 516-word vocabulary, seed 1234 — the campaign's exact list
 # and seed, so prompts are byte-identical to the historical measurements.
@@ -105,11 +117,11 @@ def prompt_for(target_tokens, wpt):
     return text, pt
 
 
-def measured(prompt, max_tokens):
+def measured(prompt, max_tokens, temperature=0):
     """Return (ttft_s, decode_tok_s, completion_tokens) via usage counters."""
     body = {"model": MODEL,
             "messages": [{"role": "user", "content": prompt}],
-            "max_tokens": max_tokens, "temperature": 0,
+            "max_tokens": max_tokens, "temperature": temperature,
             "stream": True, "stream_options": {"include_usage": True}}
     req = urllib.request.Request(
         BASE + "/v1/chat/completions", data=json.dumps(body).encode(),
@@ -145,8 +157,13 @@ def measured(prompt, max_tokens):
     return (ttft or wall), (ct / dec_window), ct
 
 
+def fmt_temp(t):
+    return ("%.1f" % t).rstrip("0").rstrip(".") if t else "0"
+
+
 def main():
-    log("base=%s model=%s ceiling=%d" % (BASE, MODEL, PROMPT_CEILING))
+    log("base=%s model=%s ceiling=%d temps=%s"
+        % (BASE, MODEL, PROMPT_CEILING, ",".join(fmt_temp(t) for t in TEMPS)))
     try:
         wpt = calibrate()
     except Exception as e:
@@ -158,32 +175,39 @@ def main():
     for tgt in DEPTHS:
         try:
             p, pt = prompt_for(tgt, wpt)
-            measured(p, WARM_TOKENS)  # warm the arena
+            measured(p, WARM_TOKENS)  # warm the arena and the prefix cache
         except Exception as e:
             log("warm pass failed at depth %d: %s" % (tgt, e))
             return 2
-        rates, tts, cts = [], [], []
-        for i in range(RUNS):
-            try:
-                tt, r, ct = measured(p, MAX_TOKENS)
-            except Exception as e:
-                log("run %d/%d failed at depth %d: %s" % (i + 1, RUNS, tgt, e))
-                continue
-            rates.append(r)
-            tts.append(tt)
-            cts.append(ct)
-        if len(rates) < 2:
-            log("only %d usable run(s) at depth %d — refusing to median" % (len(rates), tgt))
-            return 2
-        result["decode_tps_%d" % tgt] = round(statistics.median(rates), 2)
-        result["ttft_ms_%d" % tgt] = round(1000.0 * statistics.median(tts), 1)
-        result["completion_tokens_%d" % tgt] = int(statistics.median(cts))
-        result["prompt_tokens_%d" % tgt] = pt
-        log("depth %5d tok  decode %.2f t/s  ttft %.3f s  (runs=%d)"
-            % (pt, result["decode_tps_%d" % tgt], result["ttft_ms_%d" % tgt] / 1000.0, len(rates)))
+        for temp in TEMPS:
+            suffix = "" if temp == 0 else "_t%s" % fmt_temp(temp)
+            rates, tts, cts = [], [], []
+            for i in range(RUNS):
+                try:
+                    tt, r, ct = measured(p, MAX_TOKENS, temp)
+                except Exception as e:
+                    log("run %d/%d failed at depth %d temp %s: %s"
+                        % (i + 1, RUNS, tgt, fmt_temp(temp), e))
+                    continue
+                rates.append(r)
+                tts.append(tt)
+                cts.append(ct)
+            if len(rates) < 2:
+                log("only %d usable run(s) at depth %d temp %s — refusing to median"
+                    % (len(rates), tgt, fmt_temp(temp)))
+                return 2
+            result["decode_tps_%d%s" % (tgt, suffix)] = round(statistics.median(rates), 2)
+            result["ttft_ms_%d%s" % (tgt, suffix)] = round(1000.0 * statistics.median(tts), 1)
+            result["completion_tokens_%d%s" % (tgt, suffix)] = int(statistics.median(cts))
+            result["prompt_tokens_%d%s" % (tgt, suffix)] = pt
+            log("depth %5d tok temp %-4s decode %7.2f t/s  ttft %.3f s  (runs=%d)"
+                % (pt, fmt_temp(temp), result["decode_tps_%d%s" % (tgt, suffix)],
+                   result["ttft_ms_%d%s" % (tgt, suffix)] / 1000.0, len(rates)))
 
-    meds = [result["decode_tps_%d" % t] for t in DEPTHS]
-    result["decode_tps"] = round(statistics.median(meds), 2)
+    for temp in TEMPS:
+        suffix = "" if temp == 0 else "_t%s" % fmt_temp(temp)
+        meds = [result["decode_tps_%d%s" % (t, suffix)] for t in DEPTHS]
+        result["decode_tps%s" % suffix] = round(statistics.median(meds), 2)
     print(json.dumps(result))
     return 0
 

scripts/autoresearch/frank-primary-window.sh

@@ -118,7 +118,7 @@ main() {
         [ -f "$STAGE/$f" ] || { log "staged file missing: $STAGE/$f"; exit 1; }
     done
     mkdir -p "$STAGE/last"
-    rm -f "$STAGE/run.json" "$STAGE/fail.log" "$STAGE/measure.out"
+    rm -f "$STAGE/run.json" "$STAGE/fail.log" "$STAGE/measure.out" "$STAGE/evidence.log"
 
     local t_start t0 i kvline kv_tokens kv_conc measure_json boot_seconds
     t_start="$(date +%s)"
@@ -177,6 +177,24 @@ main() {
     kv_tokens="$(printf '%s' "$kvline" | sed -E 's/.*GPU KV cache size: ([0-9,]+) tokens.*/\1/' | tr -d ',' || true)"
     kv_conc="$(printf '%s' "$kvline" | sed -E 's/.*per request: ([0-9.]+x).*/\1/' || true)"
 
+    # Human-facing evidence, in a sidecar file because it is multi-line prose,
+    # not metrics: the expert-plan lines (the ONLY place the arena / runtime
+    # overhead / KV-reserve numbers exist — they are never in /metrics), the
+    # compile path (cold build vs cache hit), the graph-capture sizes, the
+    # speculative-decode counters, and any OOM signature. Without this an arm
+    # that loses decode speed is unexplainable afterwards.
+    {
+        echo "=== engine plan / capacity / spec evidence ==="
+        podman logs "$CTR" 2>&1 | grep -E \
+'KV cache size|Maximum concurrency|layers split|expert cache|runtime overhead|non-default args|r4d all-reduce|Using \[|cudagraph|CUDA graph|Capturing|graph for compile range|torch.compile took|Directly load the compiled|speculative|Speculative|acceptance|Acceptance|offload|precise|OutOfMemory|Tried to allocate|Engine core initialization failed' \
+            | tail -150 || true
+        echo "=== /metrics counters (spec decode, caches, success) ==="
+        curl -s --max-time 15 http://127.0.0.1:7770/metrics 2>/dev/null \
+            | grep -E '^vllm:(spec_decode|gpu_cache|prefix_cache|request_success|generation_tokens|iteration)' \
+            | head -80 || true
+    } >"$STAGE/evidence.log" 2>&1 || true
+    log "engine evidence -> $STAGE/evidence.log ($(wc -l <"$STAGE/evidence.log") lines)"
+
     measure_json="$(grep -E '^\{' "$STAGE/measure.out" | tail -1 || true)"
     if [ -z "$measure_json" ]; then
         log "measure produced no JSON result line"

scripts/deploy-remote.sh

@@ -935,15 +935,24 @@ deploy_plex() {
     fi
     systemctl --user restart plex.service
     # Readiness gate: /identity answers MediaContainer XML once PMS is up.
+    #
+    # HTTPS, NOT HTTP. PMS drops plain HTTP off-loopback: the probe's
+    # connection arrives on the container's eth0, so an http:// request is
+    # accepted and then answered with nothing (curl 52, Empty reply) against a
+    # perfectly healthy server. Measured 2026-10-02 — the gate burned its full
+    # 600s and failed a plex whose https://127.0.0.1:32400/identity answered
+    # 200 in 44 ms the whole time. The same law is why the Caddyfile dials the
+    # plex upstream over TLS (tests/caddyfile.bats pins it).
+    # -k: PMS presents its own self-signed cert on 32400.
+    #
     # Reality check from the 2026-09 cutover: the FIRST container boot of a
-    # years-old library ran ~10 min (PMS pre-binds 32400 during boot and
-    # empty-replies until the HTTP layer is ready), so the gate allows 600s.
+    # years-old library ran ~10 min, so the gate still allows 600s.
     # Fail the deploy (pending state preserved) rather than mark it deployed blind.
     for ((i = 1; i <= 120; i++)); do
-        curl -fsS --max-time 3 http://127.0.0.1:32400/identity >/dev/null && return 0
+        curl -fsSk --max-time 3 https://127.0.0.1:32400/identity >/dev/null && return 0
         sleep 5
     done
-    echo "FAIL: plex did not answer /identity within 600s" >&2
+    echo "FAIL: plex did not answer /identity over HTTPS within 600s" >&2
     return 1
 }
 
@@ -1077,29 +1086,33 @@ deploy_anubis() {
     wait_container_healthy anubis 60 || return 1
 }
 
-# frank-large is the ROCmFPX fork of llama.cpp (kingjones30/ROCmFPX, pinned by
-# scripts/build-rocmfpx.sh), serving
-# kingjones777/Qwen3.8-27B-ROCmFP4-STRIX-MTP-GGUF:Q8_0_ROCMFPX_AGENT with the
-# repo's own Q4_0 MTP draft head and its BF16 vision projector, on the Strix
-# Halo iGPU. It is a ROOT unit: the runtime is a native binary linked against
-# the host ROCm, so there is no container image to install and no generated
-# unit to dodge. relink_unit points /etc/systemd/system/frank-large.service at
-# the repo file and boot persistence is that unit's [Install] section.
+# frank-large is the halogen CONTAINER (ghcr.io/peonist-ai/halogen,
+# digest-pinned through scripts/frank-large.env) serving
+# peonist-ai/halogen-qwen3.8-27b — qwen3.8-27b-p1w4d-d2.hgn, 33.4 GiB,
+# DFlash2 spec decode, 1 slot x 200700, text-only — on the Strix Halo
+# iGPU. It stays a ROOT unit because the serve script drives `podman run`
+# directly (the frank-primary pattern): relink_unit points
+# /etc/systemd/system/frank-large.service at the repo file, boot
+# persistence is that unit's [Install], and LimitMEMLOCK=infinity is
+# fleet parity. 2026-10-05 cutover from the ROCmFPX llama.cpp shape —
+# rollback = revert the swap commit's trio; that payload, its draft head
+# and the fork build all stay on disk
+# (~/models/kingjones-Qwen3.8-27B-Uncensored-ROCmFP4-STRIX-MTP,
+# ~/GitPackages/kingjones-ROCmFPX).
 #
-# The runtime gate matters as much as the payload gate: the fork is built from
-# source on this host, and the payload is ftype 115 (tensor type 103,
-# GGML_TYPE_Q8_0_ROCMFPX) which only this fork can read — so a machine whose
-# /home/dank/GitPackages tree was never built has a unit that can never
-# become healthy, and a green deploy would leave the endpoint dark.
-#
-# It replaces gufo v0.1.1 + unsloth UD-Q4_K_XL (the rollback target, still
-# built by scripts/build-gufo.sh with its payload on disk), which had replaced
-# the stock llama.cpp prebuilt. The port gate is the same as every cutover
-# since 2026-09-16: nothing starts until 7770 is released.
-LLM_LARGE_RUNTIME="${LLM_LARGE_RUNTIME:-/home/dank/GitPackages/kingjones-ROCmFPX/build/bin/llama-server}"
-LLM_LARGE_MODEL="${LLM_LARGE_MODEL:-/home/dank/models/kingjones-Qwen3.8-27B-ROCmFP4-STRIX-MTP/Qwen3.8-27B-Q8_0_ROCMFPX_AGENT.gguf}"
-LLM_LARGE_DRAFT="${LLM_LARGE_DRAFT:-/home/dank/models/kingjones-Qwen3.8-27B-ROCmFP4-STRIX-MTP/mtp-Qwen3.8-27B-Q4_0.gguf}"
-LLM_LARGE_MMPROJ="${LLM_LARGE_MMPROJ:-/home/dank/models/kingjones-Qwen3.8-27B-ROCmFP4-STRIX-MTP/mmproj-Qwen3.8-27B-BF16.gguf}"
+# The gates mirror serve-frank-large.sh's predicates EXACTLY (the
+# 2026-10-01 crash-loop lesson): image digest in dank's podman storage,
+# the .hgn checkpoint at the exact byte count, the tokenizer directory
+# holding every named file as a regular (not symlinked) file — a green
+# deploy against a missing artifact would
+# leave a restart-looping unit and a dark endpoint. The port gate is the
+# same as every cutover since 2026-09-16: nothing starts until 7770 is
+# released.
+LLM_LARGE_IMAGE="${LLM_LARGE_IMAGE:-ghcr.io/peonist-ai/halogen@sha256:dc0a39a0016d6cfc58a197978febaafdf8d28403f724ded6111d98b5fb7ac0ea}"
+LLM_LARGE_CKPT="${LLM_LARGE_CKPT:-/home/dank/models/halogen-qwen3.8-27b/qwen3.8-27b-p1w4d-d2.hgn}"
+LLM_LARGE_CKPT_BYTES="${LLM_LARGE_CKPT_BYTES:-35865565184}"
+LLM_LARGE_TOK_DIR="${LLM_LARGE_TOK_DIR:-/home/dank/models/halogen-qwen3.8-27b/tokenizer}"
+LLM_LARGE_TOK_FILES="${LLM_LARGE_TOK_FILES:-tokenizer.json chat_template.jinja tokenizer_config.json merges.txt vocab.json}"
 
 retire_halogen_large_quadlet() {
     local trash="$HOME/.AGENTTRASH/20260919-orca-cutover/quadlet"
@@ -1132,43 +1145,51 @@ wait_port_free() {
 }
 
 deploy_llm_large() {
-    # Refuse rather than restart-loop. A unit whose runtime or payload is
-    # missing never becomes healthy, and a green deploy would leave the
-    # endpoint dark with nothing in the deploy log to say why. The gates
-    # mirror serve-frank-large.sh's own predicates exactly: runtime
-    # executable, target GGUF, MTP head, mmproj.
-    if [[ ! -x "$LLM_LARGE_RUNTIME" ]]; then
-        echo "ERROR: ROCmFPX runtime absent at $LLM_LARGE_RUNTIME" >&2
-        echo "frank-large: build it: bash scripts/build-rocmfpx.sh" >&2
+    # Refuse rather than restart-loop; every gate below has a twin in
+    # serve-frank-large.sh and they must agree (2026-10-01 lesson).
+    # Capture podman output before grepping: `podman images | grep -q`
+    # races under pipefail (SIGPIPE 141 becomes "no match", 2026-09-30).
+    if ! grep -qF "${LLM_LARGE_IMAGE#*@}" <<<"$(podman images --format '{{.RepoDigests}}' 2>/dev/null)"; then
+        echo "ERROR: halogen image absent: $LLM_LARGE_IMAGE" >&2
+        echo "frank-large: pull it: podman pull ${LLM_LARGE_IMAGE%@*}" >&2
         return 1
     fi
-    local missing=0 f
-    for f in "$LLM_LARGE_MODEL" "$LLM_LARGE_DRAFT" "$LLM_LARGE_MMPROJ"; do
-        if [[ ! -f "$f" ]]; then
-            echo "ERROR: payload absent at $f" >&2
-            echo "frank-large: fetch it: bash scripts/download-frank-large-kingjones.sh" >&2
-            missing=1
+    if [[ ! -f "$LLM_LARGE_CKPT" ]]; then
+        echo "ERROR: halogen checkpoint absent at $LLM_LARGE_CKPT" >&2
+        echo "frank-large: fetch it: bash scripts/download-frank-large-halogen.sh" >&2
+        return 1
+    fi
+    if [[ "$(stat -c %s "$LLM_LARGE_CKPT")" != "$LLM_LARGE_CKPT_BYTES" ]]; then
+        echo "ERROR: halogen checkpoint is not $LLM_LARGE_CKPT_BYTES bytes (partial download?)" >&2
+        echo "frank-large: re-check: bash scripts/download-frank-large-halogen.sh --verify-only" >&2
+        return 1
+    fi
+    if [[ -L "$LLM_LARGE_TOK_DIR" || ! -d "$LLM_LARGE_TOK_DIR" ]]; then
+        echo "ERROR: tokenizer dir missing or a symlink at $LLM_LARGE_TOK_DIR" >&2
+        echo "frank-large: fetch it: bash scripts/download-frank-large-halogen.sh" >&2
+        return 1
+    fi
+    for tf in $LLM_LARGE_TOK_FILES; do
+        if [[ ! -f "$LLM_LARGE_TOK_DIR/$tf" || -L "$LLM_LARGE_TOK_DIR/$tf" ]]; then
+            echo "ERROR: tokenizer file missing or a symlink: $LLM_LARGE_TOK_DIR/$tf" >&2
+            echo "frank-large: fetch it: bash scripts/download-frank-large-halogen.sh" >&2
+            return 1
         fi
     done
-    [[ "$missing" -eq 0 ]] || return 1
 
     retire_halogen_large_quadlet
-    # On a re-deploy the predecessor is THIS unit: stop it before waiting on
-    # the port, or the wait deadlocks on the server it just stopped serving.
-    # Then sweep a stray manual run -- that is how a bench harness leaves the
-    # port busy. Match the TARGET GGUF name, which every frank-large command
-    # line carries via --model and which no other process on this host uses:
-    # frank-small also runs llama-server here (with the MiMo GGUF) and must
-    # not be killed, so keying on the binary name would take the 3090 endpoint
-    # down with it. The pattern never appears in the deploying shell's own
-    # command line, so -f is safe here (it is not in `bash deploy-remote.sh`).
+    # The predecessor on a re-deploy is THIS unit: stop it (ExecStop stops
+    # the container) before waiting on the port. The stray sweep is keyed
+    # on the FULL container name, which appears in the argv of a manual
+    # `podman run --name frank-large-halogen` / `podman wait` but NOT in
+    # this deployer's own command line (`bash deploy-remote.sh
+    # frank-large`) — so it cannot shoot itself in the foot (the fleet
+    # pkill law), and it can never match frank-small's llama-server.
     run_privileged systemctl stop frank-large.service 2>/dev/null || true
-    pkill -f 'Qwen3.8-27B-Q8_0_ROCMFPX_AGENT' 2>/dev/null || true
-    # Nothing may start until the predecessor has released the port. On
-    # 2026-09-16 the cutover failed exactly here: the old server logged
-    # "Deactivated successfully", the next engine tried to bind 7770 a second
-    # later and reported "Address already in use". Gate on the port, not on
-    # hope.
+    pkill -f 'frank-large-halogen' 2>/dev/null || true
+    podman rm -f frank-large frank-large-halogen >/dev/null 2>&1 || true
+    # Nothing may start until the predecessor has released the port
+    # (2026-09-16: "Deactivated successfully" was not "port free").
     wait_port_free 7770 || return 1
     relink_unit frank-large
     run_privileged systemctl daemon-reload
@@ -1186,12 +1207,12 @@ deploy_llm_large() {
 #
 # It is a SEPARATE endpoint on port 7771; frank-large keeps 7770. The 3090 is
 # not shared: frank-large's engine runs on the gfx1151 iGPU.
-LLM_SMALL_MODEL_FILE="${LLM_SMALL_MODEL_FILE:-/home/dank/models/MiMo-Q5KM/MiMo-V2.6-Distill-Qwen-9B-Q5_K_M.gguf}"
+LLM_SMALL_MODEL_FILE="${LLM_SMALL_MODEL_FILE:-/home/dank/models/Ornith-Q8/Ornith-1.5-9B-Q8_0.gguf}"
 LLM_SMALL_RUNTIME="${LLM_SMALL_RUNTIME:-/home/dank/GitPackages/llama-master/build-cuda/bin/llama-server}"
-# The projector IS the vision capability here: the Q5_K_M GGUF ships no `v.*`
-# tensors, so a deploy that starts without this file silently ships a
+# The projector IS the vision capability here: the Ornith GGUF ships no
+# `v.*` tensors, so a deploy that starts without this file silently ships a
 # text-only endpoint. Checked unconditionally, like frank-large's.
-LLM_SMALL_MMPROJ="${LLM_SMALL_MMPROJ:-/home/dank/models/MiMo-Q5KM/mmproj-MiMo-V2.6-Distill-Qwen-9B-f16.gguf}"
+LLM_SMALL_MMPROJ="${LLM_SMALL_MMPROJ:-/home/dank/models/Ornith-Q8/mmproj-Ornith-1.5-9B-BF16.gguf}"
 
 deploy_llm_small() {
     # Refuse rather than restart-loop: a unit whose runtime or payload is
@@ -1229,24 +1250,25 @@ deploy_llm_small() {
     echo "frank-small: health answered by unit MainPID $main_pid"
 }
 
-# deploy_llm_primary — frank-primary is the tcclaviger RDNA4 vLLM fork
-# (digest-pinned image, gfx1201) serving the orcarouter
-# Qwen3.8-Flash-Next-Uncensored-FP8 payload (131 safetensors shards) with
-# MTP4 spec decode and q8/q8 KV on the two R9700s as a ROOT unit whose serve
-# script drives `podman run` directly (the image entrypoint execs
-# `vllm serve`). 2026-09-29 cutover from the R9V radiance fork, whose
-# launcher/shim/setup-state machinery is retired (state dirs stay on disk as
-# rollback; pre-cutover tag frank-primary-r9v-q8-20260929). Same port and
-# alias as the endpoints it replaced; the harness models.yml id is unchanged.
+# deploy_llm_primary — frank-primary is the FreeToken engine (PR #587 ROCm
+# build, gfx1201) serving the orcarouter Qwen3.8-Flash-Next-Uncensored-FP8
+# payload (131 safetensors shards) on ONE R9700 as a ROOT unit whose serve
+# script drives `podman run` directly (the image carries `ft serve`).
+# 2026-10-05 cutover from the tcclaviger vLLM fork (pre-cutover tag
+# frank-primary-vllm-shapeG-20261003); before that, 2026-09-29 cutover from
+# the R9V radiance fork, whose launcher/shim/setup-state machinery is
+# retired. Same port and alias as the endpoints it replaced; the harness
+# models.yml id is unchanged.
 #
-# Gates: the pinned runtime image must be loadable from dank's rootless
-# storage by its OCI manifest digest (the env file's FRANK_PRIMARY_IMAGE),
-# the payload dir must carry config.json and the expected shard count
-# (FRANK_PRIMARY_MODEL_SHARDS), and the unit env must be complete. Per-file
-# sha256 verification was the fetch script's job at download time, never
-# re-hashed here.
-LLM_PRIMARY_CONTAINER=frank-primary-qwen
-LLM_PRIMARY_LEGACY_CONTAINERS="frank-primary-r9v qwen-fp8 glm-probe"
+# Gates: the runtime image must exist in dank's rootless storage
+# (FRANK_PRIMARY_IMAGE — a LOCAL tag: the image is built on the host from
+# scripts/docker/frank-primary-freetoken.Dockerfile, which pins the
+# FreeToken PR-head SHA and the base image), the payload dir must carry
+# config.json and the expected shard count (FRANK_PRIMARY_MODEL_SHARDS),
+# and the unit env must be complete. Per-file sha256 verification was the
+# fetch script's job at download time, never re-hashed here.
+LLM_PRIMARY_CONTAINER=frank-primary
+LLM_PRIMARY_LEGACY_CONTAINERS="frank-primary-qwen frank-primary-r9v qwen-fp8 glm-probe"
 
 deploy_llm_primary() {
     # Refuse rather than restart-loop: a unit whose image or payload is
@@ -1265,15 +1287,16 @@ deploy_llm_primary() {
         echo "ERROR: frank-primary.env missing FRANK_PRIMARY_IMAGE/MODEL_DIR/MODEL_SHARDS" >&2
         return 1
     fi
-    # Capture podman's output BEFORE grepping: under set -o pipefail, the
-    # classic `podman images | grep -q` races (grep exits on match, podman
-    # dies SIGPIPE 141 writing the rest, pipefail calls it no-match — the
-    # 2026-09-30 deploy refusal).
-    local digests
-    digests="$(podman images --format '{{.RepoDigests}}')"
-    if ! grep -qF "${image#*@}" <<<"$digests"; then
-        echo "ERROR: pinned runtime image absent from rootless storage: $image" >&2
-        echo "frank-primary: fetch it (podman pull $image)" >&2
+    # The image gate is EXISTENCE IN STORAGE, not a registry digest: the
+    # FreeToken image is built ON THE HOST (a local tag carries NO
+    # RepoDigests entry — the old grep refused exactly this shape), and
+    # pinning lives in the Dockerfile (pinned PR-head SHA + digest-pinned
+    # base image). `podman image exists` accepts a local tag and a
+    # name@digest ref alike, in one command — no pipefail/grep race (the
+    # 2026-09-30 lesson) for either shape.
+    if ! podman image exists "$image"; then
+        echo "ERROR: runtime image absent from rootless storage: $image" >&2
+        echo "frank-primary: build it (scripts/docker/frank-primary-freetoken.Dockerfile) or pull it" >&2
         return 1
     fi
     if [[ ! -f "$model_dir/config.json" ]]; then
@@ -1316,7 +1339,26 @@ deploy_llm_primary() {
     run_privileged systemctl restart frank-primary.service
     # Cold start: FP8 page-in plus torch compile/graph capture (~15 min cold,
     # measured 2026-09-29). Parity with every predecessor on this port.
-    wait_for_health frank-primary http://127.0.0.1:7770/health 1800 || return 1
+    # FreeToken's /health returns 200 with {"status":"loading"} while the
+    # engine loads (the FastAPI front end binds before the scheduler is
+    # up), so a bare 200 false-greens — measured 2026-10-06: "healthy after
+    # 6 s" while 131 shards were still paging in. vLLM 503'd during load,
+    # which made the bare curl honest there; FreeToken needs the BODY read.
+    # Only status:"ok" counts. (The MainPID check below still guards the
+    # stray-port case.)
+    local i ok=0
+    for ((i=1; i<=900; i++)); do
+        if curl -fsS http://127.0.0.1:7770/health 2>/dev/null | grep -q '"status":"ok"'; then
+            echo "frank-primary engine healthy after approximately $((i * 2))s"
+            ok=1
+            break
+        fi
+        sleep 2
+    done
+    if [[ "$ok" != 1 ]]; then
+        echo "FAIL: frank-primary never reached /health status ok in 1800s" >&2
+        return 1
+    fi
     # False-green hardening (the 2026-09-26 lesson): a bare curl against the
     # port cannot tell WHO answers — a stray probe holding :7770 satisfies
     # it while the unit is stopped. The deploy is only green if the unit

scripts/deploy.sh

@@ -196,6 +196,21 @@ else
     printf '    - %s\n' "${PLAN[@]}"
 fi
 
+# --- frankbook e2e trigger --------------------------------------------------
+# The *arr/qBittorrent e2e units walk links that terminate on FrankBook-Pro
+# (192.168.0.112), which sleeps when idle. The [3/3] e2e gate therefore runs
+# the frankbook Playwright project ONLY when something in its blast radius
+# changed since origin/main: the frankbook spec itself, the frankbook-pro
+# stack, the homepage registry, or the Caddyfile serving the arr vhosts.
+# Every other deploy skips it (tests/e2e/run.sh excludes the project by
+# default). See tests/e2e/specs/19-frankbook.spec.ts.
+FB_E2E_TRIGGER=0
+while IFS= read -r fb_path; do
+    case "$fb_path" in
+        tests/e2e/specs/19-frankbook.spec.ts|frankbook-pro/*|shared/homepage/services.json|dankclaw/Caddyfile) FB_E2E_TRIGGER=1 ;;
+    esac
+done < <(git diff --name-only "$BASE_REVISION" "$HEAD_REVISION")
+
 REMOTE="${REMOTE_USER}@${REMOTE_HOST}"
 REMOTE_ARGS=("$TARGET" "$HEAD_REVISION" "$REMOTE_DIR" "$REMOTE_SHELL")
 if [[ "${#EXPLICIT_SERVICES[@]}" -eq 0 ]]; then
@@ -209,6 +224,11 @@ if [[ "$DRY_RUN" -eq 1 ]]; then
     printf '[dry-run] Runner arguments:'
     printf ' %q' "${REMOTE_ARGS[@]}"
     printf '\n'
+    if [[ "$FB_E2E_TRIGGER" -eq 1 ]]; then
+        echo "[dry-run] e2e gate: frankbook project INCLUDED (frankbook trigger paths changed)."
+    else
+        echo "[dry-run] e2e gate: frankbook project EXCLUDED (no frankbook trigger paths)."
+    fi
     exit 0
 fi
 
@@ -250,7 +270,12 @@ if [[ "$SKIP_E2E" -eq 1 ]]; then
     echo "[3/3] e2e gate SKIPPED (--skip-e2e): live fleet NOT verified." >&2
 else
     echo "[3/3] Running Playwright e2e gate against the live fleet..."
-    if bash "$REPO_ROOT/tests/e2e/run.sh"; then
+    E2E_ENV=()
+    if [[ "$FB_E2E_TRIGGER" -eq 1 ]]; then
+        echo "      frankbook trigger paths changed: including the frankbook e2e project."
+        E2E_ENV=(E2E_INCLUDE_FRANKBOOK=1)
+    fi
+    if env "${E2E_ENV[@]}" bash "$REPO_ROOT/tests/e2e/run.sh"; then
         echo "e2e gate PASSED."
     else
         echo "=== DEPLOY INCOMPLETE: e2e gate FAILED on $TARGET ===" >&2

scripts/docker/frank-primary-freetoken.Dockerfile

@@ -0,0 +1,63 @@
+# frank-primary-freetoken.Dockerfile — FreeToken (PR #587 ROCm) serving image
+# for the frank-primary endpoint (dankserver :7770, 1x R9700 gfx1201).
+#
+# Built on dankserver with rootless podman (dank user), same storage the
+# service's `podman run` reads:
+#   podman build -t frank-primary-freetoken:pr587-77a7201 \
+#     -f scripts/docker/frank-primary-freetoken.Dockerfile scripts/docker/
+#
+# Base: the PR's own docs/install_amd.md names this exact image as the
+# known-good ROCm 7.14 stack for gfx1201 (torch 2.11.0+rocm7.14, triton-rocm
+# 3.7 — the original pin; FreeToken main+PR587 accepts torch>=2.11,<2.16 and
+# triton>=3.6,<3.9). There is no rocm7.14+torch2.14 image on Docker Hub (the
+# 2.14 images are rocm10.1.0 — an untested pair for this PR), so the PR doc's
+# verbatim recommendation is the deployed stack.
+#
+# FreeToken source: FlashML-org/FreeToken PR #587 at pinned head
+# 77a7201f98f00d93f618c6cf8b81ffe8360e0899 (branch rocm714_torch214, base
+# main 581fdca3). The PR is ROCm-gated end to end; the NVIDIA/CUDA path is
+# byte-identical to main.
+#
+# Build law from docs/install_amd.md:
+#   - `pip install --no-build-isolation --no-deps -e .` — the image's ROCm
+#     torch is preserved and used to compile the C++/HIP extensions.
+#   - NEVER a plain `pip install -e .` on ROCm: the dependency set is
+#     authored for NVIDIA (flashlib pulls numba + nvidia-cutlass-dsl; torch
+#     metadata pulls cuda-bindings) and the resolve can REPLACE the ROCm
+#     torch with a CUDA build.
+#   - Runtime deps installed explicitly, curated by the same doc. flashinfer
+#     / sgl_kernel / nvidia-cutlass-dsl have no ROCm builds and are
+#     intentionally absent — FreeToken gates them off and falls back to its
+#     Triton kernels on ROCm.
+#
+# Base is digest-pinned (2026-10-05): sha256:a223aee1… is the manifest this
+# build was qualified against; a re-pushed tag cannot change the stack. The
+# readable tag rides along for humans (podman verifies the digest matches).
+FROM rocm/pytorch:rocm7.14_ubuntu24.04_py3.12_pytorch_release_2.11.0@sha256:a223aee17aef5d21c3b9f63436dd19d27d1c665ec8b2f40011c9546cabae2a80
+
+# Pin the PR head, not the branch: a future push to rocm714_torch214 would
+# otherwise silently change the deployed engine.
+# The pip-ROCm torch in this image ships libhiprtc as the VERSIONED
+# libhiprtc.so.7 only; the PR's `_cpu_moe` extension links `-lhiprtc`, and
+# ld looks for the unversioned dev symlink in the ALREADY-LINKED
+# _rocm_sdk_core/lib dir. Create it there (first build failure 2026-10-05:
+# `ld: cannot find -lhiprtc`).
+RUN ln -s libhiprtc.so.7 /opt/venv/lib/python3.12/site-packages/_rocm_sdk_core/lib/libhiprtc.so
+RUN git clone https://github.com/FlashML-org/FreeToken.git /opt/FreeToken \
+ && cd /opt/FreeToken \
+ && git fetch origin pull/587/head \
+ && git checkout 77a7201f98f00d93f618c6cf8b81ffe8360e0899
+
+WORKDIR /opt/FreeToken
+RUN python -m pip install --no-build-isolation --no-deps -e .
+
+# Curated runtime dependencies (docs/install_amd.md). transformers is pinned
+# to the pyproject window (>=5.16,<5.17); the rest are the doc's portable set.
+# torch / torchvision / triton-rocm come from the base image, 1:1 aligned.
+RUN python -m pip install \
+      "transformers>=5.16,<5.17" tokenizers safetensors huggingface_hub \
+      fastapi uvicorn pydantic openai httpx anyio \
+      einops gguf msgpack pyzmq typer rich apache-tvm-ffi \
+      partial-json-parser \
+ && python -m pip install --no-deps flashlib==0.3.0 \
+ && python -m pip install ninja

scripts/download-frank-large-halogen.sh

@@ -0,0 +1,76 @@
+#!/usr/bin/env bash
+# download-frank-large-halogen.sh — fetch frank-large's halogen payload:
+# the qwen3.8-27b .hgn checkpoint and the export tokenizer, from the
+# PUBLIC peonist-ai/halogen-qwen3.8-27b repo (no token — and per the fleet
+# secret law, none would ever touch argv).
+#
+# The fetch law, same as every sibling script: curl -f (an access error
+# must not become a "successful" 153-byte artifact), exact byte count from
+# the model card, LFS sha256 verification, --verify-only to re-check an
+# existing payload without downloading.
+set -euo pipefail
+
+DEST="${FRANK_LARGE_MODEL_DIR:-$HOME/models/halogen-qwen3.8-27b}"
+BASE="https://huggingface.co/peonist-ai/halogen-qwen3.8-27b/resolve/main"
+
+FILES=(qwen3.8-27b-p1w4d-d2.hgn
+       tokenizer/tokenizer.json
+       tokenizer/vocab.json
+       tokenizer/merges.txt
+       tokenizer/chat_template.jinja
+       tokenizer/tokenizer_config.json)
+# Byte sizes as published by the Hub on 2026-10-05, plus sha256 (the LFS
+# oid) where the Hub stores an LFS object — plain git blobs carry no oid
+# and are size-checked only. Re-check with:
+# curl -s "https://huggingface.co/api/models/peonist-ai/halogen-qwen3.8-27b/tree/main?recursive=true"
+SIZES=(35865565184 12809320 6722759 3353259 8952 17928)
+SHAS=(274c3fc767fb57faf025dd03c76376b1fc1b32448aef45aaf02d1898fa962089
+      0997f410c57a1f4e53b09e4be8f4a172d90edd9564368fb0847030937229b9f3 - - - -)
+
+VERIFY_ONLY=0
+[ "${1:-}" = "--verify-only" ] && VERIFY_ONLY=1
+
+mkdir -p "$DEST"
+
+for i in "${!FILES[@]}"; do
+    f="${FILES[$i]}"; want="${SIZES[$i]}"; sha="${SHAS[$i]}"
+    path="$DEST/$f"
+    mkdir -p "$(dirname "$path")"
+    if [ "$VERIFY_ONLY" -eq 0 ]; then
+        # A file already at the published size skips the fetch entirely:
+        # resuming it would ask the CDN for a range past EOF (416 -> curl
+        # error under set -e), and re-pulling 33.4 GiB to prove a point is
+        # not what --verify-only is for. sha256 below still decides.
+        if [ -f "$path" ] && [ "$(stat -c %s "$path")" = "$want" ]; then
+            echo "HAVE  $f ($want bytes) — skipping fetch"
+        else
+            echo "FETCH $f -> $path"
+            # HF answers /resolve/ with a 302 to the Xet CDN and curl DROPS
+            # the Range header across redirects — a naive -C - then
+            # "resumes" by pulling the ~1 KB redirect body (measured
+            # 2026-10-05: file frozen 14 MiB short, curl exit 0). Resolve
+            # the effective URL first, then resume against the CDN, which
+            # honors Range.
+            url=$(curl -sfIL -o /dev/null -w '%{url_effective}' "$BASE/$f") \
+                || { echo "cannot resolve $BASE/$f" >&2; exit 1; }
+            curl -fL --retry 5 -C - -o "$path" "$url"
+        fi
+    fi
+    [ -f "$path" ] || { echo "MISSING $path (run without --verify-only)" >&2; exit 1; }
+    got="$(stat -c %s "$path")"
+    [ "$got" = "$want" ] || { echo "SIZE $path: $got != $want" >&2; exit 1; }
+    echo "OK   $f ($got bytes)"
+done
+
+echo "byte counts match; verifying sha256 where the Hub publishes an LFS oid..."
+for i in "${!FILES[@]}"; do
+    f="${FILES[$i]}"; sha="${SHAS[$i]}"
+    if [ "$sha" = "-" ]; then
+        echo "OK   $f (plain git blob, size-checked)"
+        continue
+    fi
+    got="$(sha256sum "$DEST/$f" | cut -d' ' -f1)"
+    [ "$got" = "$sha" ] || { echo "SHA256 $f: $got != $sha" >&2; exit 1; }
+    echo "OK   $f sha256"
+done
+echo "DOWNLOAD_DONE $DEST"

scripts/download-frank-large-kingjones-uncensored.sh

@@ -0,0 +1,116 @@
+#!/usr/bin/env bash
+# Download kingjones777/Qwen3.8-27B-Uncensored-ROCmFP4-STRIX-MTP-GGUF for
+# frank-large (Dankclaw iGPU). Public repo — no auth header, and never a token
+# on the command line (it is world-readable in /proc/<pid>/cmdline).
+#
+# The payload is the repo's Q6 AGENT rung:
+#   * Qwen3.8-27B-Uncensored-Q6_0_ROCMFPX_AGENT.gguf — ftype 114, 23.15 GiB.
+#     Q6_0_ROCMFPX tensor type (102) + the AGENT routing, which keeps the
+#     output-head and attention tensors at plain Q8_0 — the variant the card
+#     names for MTP serving ("Take Q6 AGENT (114) if you want protected
+#     heads"). Like every ROCmFPX file it loads on nothing but the fork.
+#
+# The other two files frank-large needs are NOT re-downloaded:
+#   * mtp-Qwen3.8-27B-Q4_0.gguf — the uncensored repo redistributes the
+#     ALIGNED repo's draft head byte for byte (identical Hub LFS oid
+#     051a1764…), so it is hardlinked out of the aligned payload dir when that
+#     dir holds the verified bytes, and fetched from this repo otherwise.
+#   * mmproj-Qwen3.8-27B-BF16.gguf — this repo ships a Q8_0 projector; the
+#     fleet keeps the BF16 one, which is the projector that was verified 4/4
+#     on spatial ground truth on this host. Hardlinked the same way, fetched
+#     from the ALIGNED repo if it is not on disk.
+#
+# Fresh downloads, no resume (`curl -C -` once produced an oversized file on a
+# 31 GiB payload); -f/--fail-with-body so an HTTP 401 is not a success, then
+# byte count and the Hub LFS sha256. Idempotent: complete files are skipped.
+#
+# Usage:
+#   scripts/download-frank-large-kingjones-uncensored.sh [DEST_DIR] [--verify-only]
+set -euo pipefail
+
+REPO="kingjones777/Qwen3.8-27B-Uncensored-ROCmFP4-STRIX-MTP-GGUF"
+ALIGNED_REPO="kingjones777/Qwen3.8-27B-ROCmFP4-STRIX-MTP-GGUF"
+BASE="https://huggingface.co/${REPO}/resolve/main"
+ALIGNED_BASE="https://huggingface.co/${ALIGNED_REPO}/resolve/main"
+SOURCE_DIR="${SOURCE_DIR:-/home/dank/models/kingjones-Qwen3.8-27B-ROCmFP4-STRIX-MTP}"
+
+DEST=""
+VERIFY_ONLY=0
+for arg in "$@"; do
+    case "$arg" in
+        --verify-only) VERIFY_ONLY=1 ;;
+        *) DEST="$arg" ;;
+    esac
+done
+DEST="${DEST:-/home/dank/models/kingjones-Qwen3.8-27B-Uncensored-ROCmFP4-STRIX-MTP}"
+
+# name bytes sha256 base_url may_link — may_link=1 means the same bytes may
+# already exist in SOURCE_DIR (hardlink them instead of re-fetching).
+FILES=(
+  "Qwen3.8-27B-Uncensored-Q6_0_ROCMFPX_AGENT.gguf 24856607008 e966deb3d1a5a1b9ca7765126292f2d822f3dccdedb3b4872e673dec1cddc199 $BASE 0"
+  # The draft head is redistributed byte-identical by both repos; the BF16
+  # projector only exists in the aligned repo.
+  "mtp-Qwen3.8-27B-Q4_0.gguf 1680271648 051a1764cff8c4f3ee6ae8b00593a0364c7539c67fa50ffc58f3f96509fca38e $BASE 1"
+  "mmproj-Qwen3.8-27B-BF16.gguf 931145888 de2a49866988ea272c43dc2a43ee2662c25725a717ca5b9059ea420a97f53fe8 $ALIGNED_BASE 1"
+)
+
+matches() {  # matches FILE SIZE SHA — byte count AND LFS sha256
+    [[ -f "$1" ]] && [[ "$(stat -c%s "$1")" == "$2" ]] \
+        && sha256sum -c --quiet <(printf '%s  %s\n' "$3" "$1")
+}
+
+mkdir -p "$DEST"
+status=0
+
+fetch() {  # fetch NAME SIZE SHA BASE_URL
+    local name="$1" size="$2" sha="$3" url="$4" f="$DEST/$1"
+    echo "FETCHING  $name"
+    curl -fsSL --fail-with-body -o "$f.part" "$url/$name"
+    mv "$f.part" "$f"
+    if [[ "$(stat -c%s "$f")" != "$size" ]]; then
+        echo "SIZE FAIL $name ($(stat -c%s "$f") != $size)"
+        return 1
+    fi
+    if ! sha256sum -c --quiet <(printf '%s  %s\n' "$sha" "$f"); then
+        echo "SHA FAIL  $name"
+        return 1
+    fi
+    echo "VERIFIED  $name"
+}
+
+for entry in "${FILES[@]}"; do
+    read -r name size sha url may_link <<<"$entry"
+    f="$DEST/$name"
+    if matches "$f" "$size" "$sha"; then
+        echo "OK        $name"
+        continue
+    fi
+    if (( VERIFY_ONLY )); then
+        echo "MISSING   $name"
+        status=1
+        continue
+    fi
+    # Shared bytes: prefer the local copy (same filesystem, so a hardlink costs
+    # nothing) over re-downloading gigabytes that hash identically.
+    if [[ "$may_link" == 1 && -n "${SOURCE_DIR:-}" ]] && matches "$SOURCE_DIR/$name" "$size" "$sha"; then
+        echo "LINKING   $name from $SOURCE_DIR"
+        ln -f "$SOURCE_DIR/$name" "$f" 2>/dev/null || cp -f "$SOURCE_DIR/$name" "$f"
+        if matches "$f" "$size" "$sha"; then
+            echo "VERIFIED  $name"
+            continue
+        fi
+        echo "SHA FAIL  $name (linked copy)"
+        status=1
+        continue
+    fi
+    if ! fetch "$name" "$size" "$sha" "$url"; then
+        status=1
+    fi
+done
+
+if (( status == 0 )); then
+    echo "DOWNLOAD_DONE $DEST"
+else
+    echo "DOWNLOAD_FAILED $DEST" >&2
+fi
+exit "$status"

scripts/download-frank-small.sh

@@ -1,6 +1,6 @@
 #!/usr/bin/env bash
 # Download the frank-small payload for llama.cpp on the RTX 3090:
-# bartowski/MiMo-V2.6-Distill-Qwen-9B-GGUF:Q5_K_M plus its vision projector
+# ornith-ai/Ornith-1.5-9B-GGUF:Q8_0 plus its vision projector
 # (the GGUF carries no `v.*` tensors — the tower lives only in the mmproj, so
 # without this file the endpoint is text-only by construction). Public repo —
 # no auth header, and never a token on the command line (it is world-readable
@@ -11,16 +11,20 @@
 # then byte count and the Hub LFS sha256. Idempotent: complete files are
 # skipped.
 #
-# The Q6_K file this endpoint served until 2026-10-01 night stays on disk as
-# the rollback target (7793710624 B, LFS oid
-# ef96d05a2ddf2cbb450d1af1ac3860ec769d3575bafa70692ee5609bda3fad7d) — re-fetch
-# it from the same repo with the same recipe if it is ever needed again.
+# 2026-10-05 (Frank): the rung is Ornith-1.5-9B Q8_0. The payload moved off
+# MiMo-V2.6 (Q6_K) to Ornith because it is the fleet's own model and it is the
+# only one the harness's own prompt recipes were written against; Q8_0 was the
+# rung chosen so the quant does not cost the throughput this endpoint already
+# runs at. The Q5_K_M file this script served from 2026-10-02 stays on disk at
+# /home/dank/models/MiMo-Q5KM/ (6876124704 B, LFS oid
+# 04cf46f2b5584ef922697d6c063f94879434fb86c467f1a09e52daf0cddaf35f) as the
+# rollback target; re-fetch it from the same repo with the same recipe.
 #
 # Usage:
 #   scripts/download-frank-small.sh [DEST_DIR] [--verify-only]
 set -euo pipefail
 
-REPO="bartowski/MiMo-V2.6-Distill-Qwen-9B-GGUF"
+REPO="ornith-ai/Ornith-1.5-9B-GGUF"
 BASE="https://huggingface.co/${REPO}/resolve/main"
 
 DEST=""
@@ -31,12 +35,12 @@ for arg in "$@"; do
         *) DEST="$arg" ;;
     esac
 done
-DEST="${DEST:-/home/dank/models/MiMo-Q5KM}"
+DEST="${DEST:-/home/dank/models/Ornith-Q8}"
 
-# name bytes sha256
+# name bytes sha256 (the sha256 is the LFS oid in the repo's pointer file)
 FILES=(
-  "MiMo-V2.6-Distill-Qwen-9B-Q5_K_M.gguf 6876124704 04cf46f2b5584ef922697d6c063f94879434fb86c467f1a09e52daf0cddaf35f"
-  "mmproj-MiMo-V2.6-Distill-Qwen-9B-f16.gguf 918166048 ff348f3180a63188aa7285db85f550fe38acb61dd013c599eb8bad08d2cc2576"
+  "Ornith-1.5-9B-Q8_0.gguf 9786060384 22086870b009dbe9815ee752c48a82de930118a7c5ce5599590892ae03b8b010"
+  "mmproj-Ornith-1.5-9B-BF16.gguf 921704672 626f9f90627402a6bf4a999111d0fbd69b5fcca7aa8ba089d69e5f10e8858e1d"
 )
 
 mkdir -p "$DEST"

scripts/frank-large.env

@@ -1,85 +1,113 @@
-# frank-large — ROCmFPX llama.cpp serving
-# kingjones777/Qwen3.8-27B-ROCmFP4-STRIX-MTP-GGUF (Q8_0_ROCMFPX_AGENT, ftype
-# 115) + its Q4_0 MTP draft head, on the Strix Halo iGPU (gfx1151), :7770.
+# frank-large.env — tunables for the frank-large endpoint on Dankclaw's
+# Strix Halo iGPU (gfx1151), host port 7770.
 #
-# 2026-10-02 (Frank): "the highest quality of that repo, one full context,
-# whatever runtime it requires". The highest-quality rung is Q8_0_ROCMFPX_AGENT
-# (26.28 GiB, 8.39 bpw: 187 tensors at plain Q8_0 + 311 at
-# GGML_TYPE_Q8_0_ROCMFPX); the AGENT routing is also the variant the card
-# tells you to take with MTP, because it keeps the output-head tensors at full
-# Q8_0 (187 vs 2), which is what the draft scores against. The runtime the
-# format REQUIRES is the ROCmFPX fork — type 103 exists nowhere else.
+# RUNTIME (2026-10-05, Frank): halogen — the closed-source C++/HIP engine
+# (github.com/peonist-ai/halogen-server, image ghcr.io/peonist-ai/halogen,
+# digest-pinned below) serving peonist-ai/halogen-qwen3.8-27b, a ~6.32 bpw
+# NVFP4-derivative quant of the BASE Qwen/Qwen3.8-27B (35,865,565,184 B,
+# 1352 tensors), as a podman container under a root unit, driven by
+# scripts/serve-frank-large.sh. This CUTS OVER the
+# 2026-10-02..10-03 ROCmFPX llama.cpp shape (uncensored
+# Q6_0_ROCMFPX_AGENT + MTP, 27.6-28.1 t/s): rollback = revert the swap
+# commit's trio; the ROCmFPX payload, draft head and build all stay on
+# disk at /home/dank/models/kingjones-Qwen3.8-27B-Uncensored-ROCmFP4-STRIX-MTP/.
 #
-# The shape: one session at the model's full native window (262,144), which is
-# the whole point — the Q8_0 head costs quality the fleet wanted back, and
-# the draft head pays for the bandwidth the bigger file spends.
+# WHY THE RUNTIME MOVED WITH THE MODEL: the payload is .hgn — per its own
+# card it "will not load in transformers, vLLM, or llama.cpp". Accepted
+# consequences (Frank, 2026-10-05): TEXT-ONLY (halogen ignores the model's
+# vision encoder; the fleet's vision path is frank-small) and the CENSORED
+# base (this is a finetune of Qwen/Qwen3.8-27B, not the abliterated file —
+# refusals may return). What it buys: vendor-measured decode DFlash2 31.7
+# t/s mean (vs this endpoint's MTP 27.6-28.1 and the old halogen-era MTP
+# 26.9), prefill ~566 t/s at 32k (was ~400 on the sibling engine), 1.4-2.3x
+# wall-clock on prefill-heavy agent traffic, at HIGHER effective precision
+# (~6.32 bpw vs ~6.6 with the old W4A4 promotion — see FLAGS.md).
 #
-# Budget (from the GGUF header, not from a boot log): attention KV lives only
-# in the 16 full-attention layers (4 KV heads x 256 head_dim), 64 KiB/token
-# with f16 K and V => 16.0 GiB at 262,144. The 48 Gated DeltaNet layers hold
-# ~144 MiB of fixed state per sequence. Weights 26.28 + draft head 1.57 +
-# mmproj 0.87 + KV 16.0 + state 0.14 + compute buffers ≈ 45-47 GiB against the
-# iGPU's ~114 GiB GTT aperture, so f16 KV (the unquantized choice) is affordable
-# here in a way it is not on frank-small's 24 GiB card.
+# SHAPE — the two knobs that are NOT free:
+#   HALOGEN_KV_SLOTS=1 KEEPS SPECULATION ON. halogen makes batching and
+#   speculation mutually exclusive: with >1 slot the drafter is disabled
+#   and per-stream decode falls to serial speed (the sibling engine
+#   measured 41 t/s -> 14 t/s when slots were raised, and halogen's own
+#   FLAGS.md prints the same law). frank-large was already a one-session
+#   endpoint under ROCmFPX (PARALLEL=1), so slots=1 is the same contract,
+#   not a regression.
+#   HALOGEN_SLOT_CTX=200700 (Frank's window) is the per-conversation hard
+#   ceiling — a prompt over it is a 400 naming the limit, never a silent
+#   truncation. KV pool = slots x slot_ctx x 64 KiB = 12.2 GiB. Budget
+#   against the 121 GiB of unified memory: weights 33.4 GiB (mmap'd — page
+#   cache, not anon), KV 12.2, engine working set + auto prompt cache
+#   ~20-30, host baseline ~10 -> ~75-85 GiB in use. The 2026-09-16 halogen
+#   era ran the same engine class on this box with LESS headroom.
+#   HALOGEN_DRAFTER=2 (DFlash2) is the shipped default and the fastest
+#   exact drafter on this checkpoint; the .hgn BAKES BOTH drafters (MTP +
+#   DFlash2), so unlike the Flash-Next era there is no sidecar to mount
+#   and no --dflash flag — the drafter is an int knob, byte-identical
+#   output to greedy verification either way.
+#
+# CLIENT CONTRACT deltas vs the ROCmFPX shape (the harness side lives in
+# ~/.omp/agent/models.yml):
+#   * NO server-side sampler defaults: a request without temperature
+#     decodes GREEDY. The harness sends temp 1.0 / top_p 0.95 via
+#     extraBody, so that wire recipe is load-bearing now, not cosmetic.
+#   * max_tokens defaults to 8192 PER REQUEST and INCLUDES thinking; a
+#     reasoning reply over a small budget returns empty content with
+#     finish_reason "length". The harness maxTokens is 57344: compaction
+#     fires at 70% of 200700 = 140,490 and 140,490 + 57,344 = 197,834 <=
+#     200,700 keeps prompt + budget inside the pool (the 262144-era 65536
+#     would overflow). Requests over HALOGEN_MAX_TOKENS_CAP are a 400
+#     naming the cap; cap and queue timeout are COUPLED (see FLAGS.md
+#     table) — do not move one without the other.
+#   * response_format json_object/json_schema is a 400 (no constrained
+#     decoding in this build).
+#   * Thinking lands in reasoning_content — same field as the llama.cpp
+#     era, so the harness compat block does NOT change. reasoning_effort
+#     accepts none|low|xhigh on this build (template default xhigh); the
+#     models.yml fold (high->xhigh) still applies.
+#   * The engine's token protocol has no auth and binds loopback INSIDE
+#     the container; only the frontend port is published. Keep it that way.
 
-# The qualified ROCmFPX fork build (scripts/build-rocmfpx.sh pins the commit).
-LLAMA_BIN=/home/dank/GitPackages/kingjones-ROCmFPX/build/bin/llama-server
-# One directory per HF repo, created by scripts/download-frank-large-kingjones.sh.
-MODEL_FILE=/home/dank/models/kingjones-Qwen3.8-27B-ROCmFP4-STRIX-MTP/Qwen3.8-27B-Q8_0_ROCMFPX_AGENT.gguf
-# The draft head: Q4_0, NOT the Q8_0 head. The head's per-step cost is its
-# projection over a 248K vocabulary, so the cheaper head is faster end to end
-# at equal-or-better acceptance (card: +6.0% at n-max 1, +10.2% at n-max 2).
-DRAFT_FILE=/home/dank/models/kingjones-Qwen3.8-27B-ROCmFP4-STRIX-MTP/mtp-Qwen3.8-27B-Q4_0.gguf
-# Vision tower, separate from the LLM file. Empty string = text-only.
-MMPROJ_FILE=/home/dank/models/kingjones-Qwen3.8-27B-ROCmFP4-STRIX-MTP/mmproj-Qwen3.8-27B-BF16.gguf
+# Image: digest-pinned (fetch law: tag pulls silently re-point; the digest
+# is what was qualified). 0.1.4 = the version the model card names.
+FRANK_LARGE_IMAGE=ghcr.io/peonist-ai/halogen@sha256:dc0a39a0016d6cfc58a197978febaafdf8d28403f724ded6111d98b5fb7ac0ea
+# Container identity: the halogen container name.
+FRANK_LARGE_CONTAINER=frank-large-halogen
+# Old quadlet-era container identity; the serve script clears stale names.
+FRANK_LARGE_LEGACY_CONTAINERS=frank-large
+# host:container — the frontend listens on 8731 inside (HALOGEN_API_PORT
+# default); the engine port (8730) is never published.
+FRANK_LARGE_PUBLISH=7770:8731
 
-# Fleet endpoint: same port and alias as every frank-large before it, so the
-# harness models.yml id and the homepage probe do not move.
-ALIAS=frank-large
-# HIP reads a bare HOST, so this is BIND_ADDR — the same reason the file
-# avoids a bare TEMP.
-BIND_ADDR=0.0.0.0
-LISTEN_PORT=7770
+FRANK_LARGE_MODEL_DIR=/home/dank/models/halogen-qwen3.8-27b
+FRANK_LARGE_CHECKPOINT=qwen3.8-27b-p1w4d-d2.hgn
+FRANK_LARGE_CHECKPOINT_BYTES=35865565184
+FRANK_LARGE_CHECKPOINT_SHA256=274c3fc767fb57faf025dd03c76376b1fc1b32448aef45aaf02d1898fa962089
+# The tokenizer is a FLAT DIRECTORY, $FRANK_LARGE_MODEL_DIR/tokenizer, mounted
+# whole at /tokenizer — the model card ships `tokenizer/` and the engine's
+# HALOGEN_TOKENIZER default is that directory (the card's own docker run sets
+# no tokenizer flag). The Hub stores it as 5 files (sizes from the Hub API,
+# 2026-10-05): tokenizer.json 12809320 (LFS sha256
+# 0997f410c57a1f4e53b09e4be8f4a172d90edd9564368fb0847030937229b9f3),
+# vocab.json 6722759, merges.txt 3353259, chat_template.jinja 8952,
+# tokenizer_config.json 17928 (plain git blobs — size-checked, no LFS oid).
+# The serve gate and the deployer refuse a missing file or ANY symlink: an
+# HF cache snapshot symlinks each file into blobs/, and those links dangle
+# inside the container mount.
+FRANK_LARGE_TOKENIZER_FILES="tokenizer.json chat_template.jinja tokenizer_config.json merges.txt vocab.json"
 
-# One full native window. llama.cpp divides --ctx-size across --parallel
-# slots, so 262144/1 IS the model's trained window; the card validated 65,536
-# and the fleet has been running this family at 262,144 since the gufo era.
-CTX_SIZE=262144
-PARALLEL=1
-
-# Deck knobs. --ubatch-size has a measured optimum on this stack (512 peak on
-# the sibling qwen35 model; larger ubatch costs GiBs of compute buffer for no
-# speed), and --fit must stay off: --fit's auto placement does not know that
-# 311 tensors of type 103 belong on the iGPU.
-BATCH_SIZE=2048
-UBATCH_SIZE=512
-NGL=999
-FIT=off
-FLASH_ATTN=on
-# Unquantized KV — the quality-first choice, affordable at one slot.
-CACHE_TYPE_K=f16
-CACHE_TYPE_V=f16
-
-# Speculative decode: the model ships a standalone MTP head (mtp-Q4_0.gguf,
-# arch qwen35, nextn_predict_layers=1, a 65-block draft). n-max 4 with n-min 0
-# and p-min 0.0 is the card's measured optimum (+31% end to end, acceptance
-# 0.926; its own n-max 8 measurement only reached 0.757 acceptance and LOST).
-# The draft runs entirely on the iGPU, so cross-device traffic stays off the
-# bandwidth-limited link.
-SPEC_TYPE=draft-mtp
-SPEC_DRAFT_NGL=99
-SPEC_DRAFT_DEVICE=ROCm0
-SPEC_DRAFT_N_MAX=4
-SPEC_DRAFT_N_MIN=0
-SPEC_DRAFT_P_MIN=0.0
-
-# The card's thinking sampling profile, which is also what the harness sends
-# in extraBody, so the endpoint behaves the same to a client that omits it.
-TEMPERATURE=1.0
-TOP_P=0.95
-TOP_K=20
-MIN_P=0.0
-PRESENCE_PENALTY=0.0
-
-NUM_THREADS=16
-THREAD_BATCH=16
+# --- engine flags (docs/FLAGS.md of the engine repo; anything unset stays
+# at the image default, which is what was qualified) ------------------------
+HALOGEN_MODEL_ID=frank-large
+HALOGEN_SLOT_CTX=200700
+HALOGEN_KV_SLOTS=1
+HALOGEN_DRAFTER=2
+# Coupled pair (FLAGS.md table): the cap bounds how long ONE request holds
+# the GPU, the timeout bounds how long the NEXT client waits. 65536 at
+# ~10 t/s serial worst case needs > 6550 s queue timeout; the defaults
+# honor that. Do not lower the timeout alone.
+HALOGEN_MAX_TOKENS_CAP=65536
+HALOGEN_QUEUE_TIMEOUT=7200
+# Engine prompt cache: EMPTY = auto (sized from available memory at
+# startup). Pin an explicit MB only to argue with the auto figure — a
+# full-context entry is ~18 GB at 262K, so a too-small pin yields a cache
+# that reports enabled and never hits (the engine warns at startup).
+#HALOGEN_CACHE_MB=

scripts/frank-large.service

@@ -1,5 +1,5 @@
 [Unit]
-Description=ROCmFPX llama.cpp (gfx1151) kingjones777 Qwen3.8-27B Q8_0_ROCMFPX_AGENT, MTP spec decode, 262144 ctx / 1 session, root systemd unit (dankclaw, frank-large, Strix Halo iGPU)
+Description=halogen container (peonist-ai/halogen-qwen3.8-27b, DFlash2 spec decode, 1 slot x 200700, text-only), root systemd unit (dankclaw, frank-large, Strix Halo iGPU)
 After=network-online.target
 Wants=network-online.target
 
@@ -13,10 +13,19 @@ Group=dank
 LimitMEMLOCK=infinity
 Environment="FRANK_LARGE_ENV_FILE=/home/dank/serverAppConfigs/scripts/frank-large.env"
 ExecStart=/home/dank/serverAppConfigs/scripts/serve-frank-large.sh
-# always, not on-failure: an engine that drains and exits 0 after a worker
-# fault would otherwise leave the endpoint down until someone notices.
+# systemd stop alone cannot stop the container child; stop it explicitly
+# (frank-primary pattern). The engine drains on SIGTERM.
+ExecStop=/usr/bin/podman stop -t 60 frank-large-halogen
+TimeoutStartSec=1200
+TimeoutStopSec=120
+# always, not on-failure: an engine that exits 0 after an internal fault
+# would otherwise leave the endpoint down until someone notices.
 Restart=always
 RestartSec=10
+# The engine mmaps 33.4 GiB of weights and keeps a multi-GiB working set
+# on a box that also hosts frank-small, ComfyUI and the Caddy/observability
+# stack; make it a late OOM candidate, not the first.
+OOMScoreAdjust=-500
 
 [Install]
 WantedBy=multi-user.target

scripts/frank-primary.env

@@ -1,129 +1,82 @@
-# frank-primary.env — tcclaviger RDNA4 vLLM production config (dankserver :7770).
+# frank-primary.env — FreeToken (PR #587 ROCm) serving the orcarouter
+# Qwen3.8-Flash-Next-Uncensored-FP8 payload on dankserver :7770.
 #
-# 2026-10-01 cutover to the BF16-KV no-spec incumbent (the overnight redo
-# campaign's verdict, Frank's call): same digest-pinned engine and payload as
-# the 2026-09-29 cutover, but the KV cache is UNQUANTIZED (auto = bf16) and
-# speculative decode is GONE. Git tag frank-primary-fp8-mtp-20261001 is the
-# last pre-cutover commit; the old shape (fp8 KV + MTP4) is in history and
-# restorable from the tag.
+# 2026-10-05 cutover from the tcclaviger RDNA4 vLLM fork (git tag
+# frank-primary-vllm-shapeG-20261003 is the pre-cutover state; the 362-line
+# vLLM-era env is in that tag). The engine is FreeToken at PR head
+# 77a7201f98f00d93f618c6cf8b81ffe8360e0899 (branch rocm714_torch214), built
+# on the host into a local image by
+# scripts/docker/frank-primary-freetoken.Dockerfile — the PR's own
+# docs/install_amd.md stack (ROCm 7.14 base image, torch 2.11.0+rocm7.14,
+# triton 3.7) for gfx1201.
 #
-# Every number here was QUALIFIED by boot probe on 2026-10-01 (not inherited):
-#   * MAXLEN 241664 x SEQS 3 (724,992 tokens) at bf16 KV is the DEPLOYED
-#     shape (2026-10-01 night ladder): KV_BYTES 10,184,000,000 PER RANK,
-#     engine measured "KV cache size: 732,359 tokens; Maximum concurrency
-#     for 241,664 tokens per request: 3.03x". 262144 and 245760 both OOMed
-#     at graph capture (76 MiB short / 0 bytes free); 237568 passed at
-#     718,606 tokens and the +4096 bonus step passed and won.
-#   * KV_BYTES is PER RANK. The old "7.2 GiB ceiling, 9/16 GiB OOM" law was
-#     the MTP4 WORKSPACE: the spec-era pool measured 729,545 tokens because
-#     the drafter reserved ~2.6 GiB/rank beside the pool. No-spec bf16 takes
-#     11.05 GiB/rank and boots clean — the ceiling was spec's, not VRAM's.
-#   * bf16 KV doubles the attention-KV bytes/token (measured 14,047 vs
-#     7,926) and costs ~14% decode vs fp8 at 4064 ctx (49.25 -> 36.42 with
-#     the gather win) — traded deliberately for clean cache + 262k windows.
-#   * GATHER 32x64 is the night's biggest lever: R4D_LRU_GATHER_CHUNKS=32 /
-#     R4D_LRU_GATHER_LANES=64 measured +18.7% decode vs default geometry,
-#     and the win grows with depth (39.5@166 -> 54.1@1087 -> 49.3@4064 fp8
-#     era). Do not drop it.
-#   * MNBT 1024 is Frank's lock: MNBT 512 doubles TTFT at 4064 ctx (2.46 s
-#     vs 1.36 s); 1024-4096 are interchangeable on decode; 12288 costs 2.5%
-#     and arena. 1024 keeps the workspace floor so the arena stays max.
-#   * PREFETCH IS ABSENT BY LAW: expert-cache telemetry measured the
-#     prefetcher at 36% precision (36,720 slabs prefetched, 13,369 served) —
-#     roughly half the gather traffic was waste. Never re-add
-#     --expert-prefetch without a fresh A/B.
-#   * TunableOp stays OFF: CLAV_TUNABLEOP_SWEEP silently skips under expert
-#     offload unless CLAV_TUNABLEOP_WITH_OFFLOAD=1, and forcing it corrupts
-#     numerics.
+# WHY FREETOKEN: the vLLM fork's expert offload planned VRAM/host tiers per
+# boot from an estimator (util 0.982 law, arena 17.77 GiB / 7,762 slots,
+# ~40k-token GDN state faults) and 1.0 was unbootable. FreeToken keeps ALL
+# expert weights in pinned HOST RAM (~120 GiB) with an LRU GPU cache, sizes
+# KV from an explicit token pool, and serves on ONE R9700 (32 GB) with the
+# second card idle. RAM fit: 246 GiB total, service stopped frees the vLLM
+# container's ~170 GiB; weights ~120 GiB + PLE 47.7 GiB + overhead ≈ 190 GiB.
 #
-# HIP reads bare env vars TEMP, HOST, GPU_MAX_HEAP_SIZE — every key here is
-# prefixed FRANK_PRIMARY_ so none can leak.
+# PAYLOAD: unchanged — /home/dank/models/Qwen3.8-Flash-Next-Uncensored-FP8,
+# 131 safetensors shards. Quant is standard modelopt block-FP8: 75,264
+# weight_scale_inv tensors (block 128x128), zero input_scale (FreeToken #462
+# covers exactly this export shape), one weight_scale on the PLE ngram
+# embedding. FreeToken loads HF safetensors directly; no FTW conversion.
 #
-
-# --- runtime ----------------------------------------------------------------
-# Digest-pinned: the floating :latest tag moved once already. Local alias
-# qwen-fp8-runtime:tuned-20260929 (image ID 3322dbbbc96d) points at the same
-# bytes in rootless storage.
-FRANK_PRIMARY_IMAGE=docker.io/tcclaviger/vllm@sha256:da4029666668b90a0386c1ed23ea370c79916ad4e1a6ad1dd9db098b9fa069ea
-
-# --- payload ----------------------------------------------------------------
+# SERVING FACTS (docs/cli.md, verified against main+PR587):
+# - `ft serve` loads the checkpoint directly; `--served-model-name` keeps the
+#   harness id; reasoning/tool parsers are AUTO (reasoning lands in
+#   reasoning_content — models.yml flips reasoningContentField to match).
+# - KV: --page-size must stay 1 for GDN/SWA models (default); --cache-type
+#   radix auto-picks the GDN-aware variant.
+# - MoE: --moe-strategy auto -> offload (fp8 experts CANNOT use
+#   --moe-cpu-layers; only bf16/nvfp4/mxfp4 have a CPU executor).
+#   --moe-cache-rate holds the GPU LRU fraction of all experts.
+# - `ft ctl cache --kv --moe` grows the pools LIVE on a running server
+#   (shrinking needs a restart); `ft ctl stats` reports pool occupancy.
+# - KV MATH: 12 full-attention layers x 2 KV heads x 256 head_dim x 2 (K+V)
+#   x 2 B = 24 KiB/token (f16 KV; the 36 GDN layers use fixed per-seq state,
+#   not per-token KV). See FRANK_PRIMARY_FT_NUM_TOKENS below for the pool
+#   size and its measured page-size law.
+FRANK_PRIMARY_IMAGE=localhost/frank-primary-freetoken:pr587-77a7201
 FRANK_PRIMARY_MODEL_DIR=/home/dank/models/Qwen3.8-Flash-Next-Uncensored-FP8
 FRANK_PRIMARY_MODEL_SHARDS=131
-
-# --- identity ---------------------------------------------------------------
-# The harness models.yml id stays frank-primary (provider 192.168.0.235:7770).
-FRANK_PRIMARY_CONTAINER=frank-primary-qwen
+# Container identity is the SERVICE name (Frank's call, 2026-10-05): one
+# name across systemctl and podman ps. The vLLM-era name frank-primary-qwen
+# moved into the legacy list so every deploy cleans it up.
+FRANK_PRIMARY_CONTAINER=frank-primary
+FRANK_PRIMARY_LEGACY_CONTAINERS="frank-primary-qwen frank-primary-r9v qwen-fp8 glm-probe"
 FRANK_PRIMARY_SERVED_NAME=frank-primary
 FRANK_PRIMARY_PUBLISH=7770:7770
-# Containers removed before every start: the retired R9V identity and the ad
-# hoc experiment identity (both hold the GPUs / name registrations).
-FRANK_PRIMARY_LEGACY_CONTAINERS="frank-primary-r9v qwen-fp8"
 
-# --- qualified engine shape (2026-10-02, Frank's own correction) ---
-# 2026-10-01 left a shape that did not stay up: bf16 KV at 2 x 262144 with
-# --expert-precise-memory crash-looped at graph capture (the arena is
-# DEVICE-resident and the shape had 0.13 GiB to spare), and the plain bf16
-# 2 x 262144 boot booted but left 0 bytes free at runtime. This is the
-# correction: FP8 KV, TWO windows, each 200704 tokens, and the pool cut so
-# the expert arena gets the room back.
-#
-# LADDER (2026-10-01 night, subtract-16k method): 262144 and 245760 OOM at
-# graph capture; 237568 passed at 718,606 tokens; 241664 x 3 passed at
-# 732,359 tokens / 3.03x but is the shape this cutover retired. 200704 was
-# chosen to sit below the 241664 rung while leaving a full 2.95 GiB/rank of
-# pool to cut loose for the arena — the two contexts are a capacity
-# contract, not a throughput one (2-way concurrency summed BELOW parity at
-# 4064 ctx), so the slot buys the VRAM headroom long generations need.
-#
-# KV_BYTES is PER RANK. fp8 costs 7,701 B/token (measured 6,138 attention
-# + 1,563 mamba/side, dtype-independent). 2 x 200704 = 401,408 tokens
-# -> 3.091e9 B floor, pinned at 3.15e9 (~405,000 tokens, ~44 MB margin —
-# the same margin convention the 6.1e9 shape used).
-#
-# THE ARENA IS WHAT REMAINS: the fork's expert planner sizes it AFTER the
-# KV override, so every byte cut here is arena. The 6.1e9 shape ran an
-# arena of 6,248 slots; this one reclaims ~2.95 GiB/rank, roughly 1,260
-# more slots at 2.344 MiB each. That is the point of the change — the
-# gather read path is the measured decode bound, and a bigger arena means
-# more experts resident instead of streamed.
-#
-# MNBT 1024 (down from 2048): 1024-4096 are interchangeable on decode AND
-# identical on arena (the sweep measured the same slot count at every
-# rung <= 4096), so this is a pure workspace-floor choice, not a decode
-# trade. The floor matters because the arena budget is what is left.
-#
-# PRECISE MEMORY stays DROPPED: it crashed the 2 x 262144 bf16 boot
-# (weights 6.28 + pool 6.80 + graphs 0.55 + arena 16.65 + non-torch ~1.45
-# = 31.73 of 31.86 GiB, ~0.13 GiB free, NRestarts=3). The flag survives in
-# the serve script as an opt-in; re-enable it only with a pool/context
-# reduction that actually pays for the extra overhead.
-#
-# TUNABLEOP stays OFF (silently skipped under offload unless
-# CLAV_TUNABLEOP_WITH_OFFLOAD=1, and then corrupts numerics).
-#
-# SEQS 2 x MAXLEN 200704 at FP8 KV, pool cut for the arena. No
-# speculative decode, no prefetch.
-FRANK_PRIMARY_MAXLEN=200704
-FRANK_PRIMARY_SEQS=2
-FRANK_PRIMARY_KV_BYTES=3150000000
-FRANK_PRIMARY_KV_DTYPE=fp8
-FRANK_PRIMARY_MNBT=1024
-FRANK_PRIMARY_GATHER_CHUNKS=32
-FRANK_PRIMARY_GATHER_LANES=64
-FRANK_PRIMARY_UTIL=0.98
-
-# --- client contract: reasoning split, tools, preserve_thinking ----------------
-# Without --reasoning-parser the engine leaves the thinking text INLINE in
-# `content` with a dangling `</think>` and a null `reasoning` field (measured
-# 2026-10-01: content = 'We need answer...Brief.\n</think>\n\n391') — the
-# malformed thinking block clients choke on. The qwen3 parser splits it
-# properly. The GLM rollback (d91379d) had dropped all three of these; the
-# working values are the ones ecaad36 shipped for this model.
-FRANK_PRIMARY_REASONING_PARSER=qwen3
-# Tools: --enable-auto-tool-choice + the qwen3_xml parser is what makes
-# tool_calls work here (without it, tool_choice auto returns 400).
-FRANK_PRIMARY_TOOL_PARSER=qwen3_xml
-# preserve_thinking keeps prior thinking blocks in the rendered prompt across
-# turns; enable_thinking makes thinking the server-side default. Verified the
-# payload's tokenizer_config template consumes both keys.
-FRANK_PRIMARY_CHAT_TEMPLATE_KWARGS='{"enable_thinking": true, "preserve_thinking": true}'
+# --- FreeToken engine knobs (first-boot conservative, resize live) -----
+# Single GPU: the R9700 at index 0. HIP_VISIBLE_DEVICES=0 hides card 1 so
+# the in-container index 0 IS the right card; --gpu 0 selects it.
+FRANK_PRIMARY_FT_GPU=0
+# KV pool in TOKENS (not bytes — vLLM-era FRANK_PRIMARY_KV_BYTES is gone).
+# 299,968 tokens x 24 KiB = 7.0 GiB; clears the harness worst case
+# (177,356 tokens) and one NATIVE 262,144-token conversation + full budget.
+# MEASURED FIRST BOOT (2026-10-05): the auto attention backend is
+# qsa_sparse, which OVERRIDES page size to 64, so --num-tokens must be a
+# MULTIPLE OF 64 — 300,000 was refused ("nearest valid values: 299968 or
+# 300032"). Grows live with `ft ctl cache --kv <n>`; shrink needs a
+# restart.
+FRANK_PRIMARY_FT_NUM_TOKENS=299968
+# Concurrent decode slots. 2 is the first-boot conservative value; the
+# radix pool admits ~2 full-length 200,704-token conversations. Raise with
+# --max-running-requests only after a real-traffic look at slot occupancy.
+FRANK_PRIMARY_FT_MAX_RUNNING=2
+# Per-request output budget ceiling: the harness's maxTokens (36,864).
+# Requests that omit max_tokens get this default instead of the engine's
+# 32,768.
+FRANK_PRIMARY_FT_MAX_OUTPUT_TOKENS=36864
+# GPU expert LRU cache as a FRACTION of all experts (~120 GiB fp8 total ->
+# ~12 GiB VRAM at 0.10). First-boot conservative; resize live with
+# `ft ctl cache --moe` after reading the boot ledger + ft ctl stats.
+FRANK_PRIMARY_FT_MOE_RATE=0.10
+# Escape hatch for boot-ladder experiments (YaRN, --max-prefill-length,
+# --moe-cpu-threads, --sampling-defaults none, ...). Unset -> no extra
+# argv. Keep multi-flag values as one string; the serve script expands it
+# unquoted on purpose.
+FRANK_PRIMARY_FT_EXTRA_ARGS=

scripts/frank-primary.service

@@ -1,28 +1,26 @@
-# frank-primary.service — tcclaviger RDNA4 vLLM endpoint (orcarouter
-# Qwen3.8-Flash-Next-Uncensored-FP8, MTP4 spec decode, KV q8/q8, 600k total
-# context over 3 sessions, TP=2, tiered experts, 131-shard payload) on the
-# 2x R9700 pair, dankserver :7770. 2026-09-29 cutover from the R9V radiance
-# fork (git tag frank-primary-r9v-q8-20260929 is the pre-cutover state; its
-# launcher/shim/setup-state machinery is retired and the state dirs stay on
-# disk as rollback).
+# frank-primary.service — FreeToken (PR #587 ROCm) endpoint (orcarouter
+# Qwen3.8-Flash-Next-Uncensored-FP8, 131-shard payload) on ONE R9700
+# (gfx1201), dankserver :7770. 2026-10-05 cutover from the tcclaviger vLLM
+# fork (git tag frank-primary-vllm-shapeG-20261003 is the pre-cutover
+# state; the retired unit/env/serve trio is in that tag).
 #
-# Same port, same served alias; the harness models.yml id is unchanged.
-# The serve script drives `podman run` directly (the image entrypoint execs
-# `vllm serve`) and then waits on the container. NOTE: that wait can return
-# 0 even when the engine died mid-request (vLLM's API server drains and
-# exits clean on a worker fault), which is why the restart policy is
-# `always` — see the [Service] note.
+# Same port, same served alias, same container identity as the unit name;
+# the harness models.yml id is unchanged. The serve script drives
+# `podman run` directly (the image carries `ft serve`) and then waits on
+# the container. The image is built ON THE HOST from
+# scripts/docker/frank-primary-freetoken.Dockerfile (FreeToken PR head
+# 77a7201 in the PR's own ROCm 7.14 base image).
 #
-# WHY A ROOT UNIT (not a rootless quadlet): the tiered-expert shape pins tens
-# of GiB of host RAM (the expert host slabs, ~20.5 GiB per rank of pinned
-# anonymous memory). Pinned pages are charged against RLIMIT_MEMLOCK; dank's
-# login session on this host carries 8 MiB and a user quadlet is capped by the
-# user manager's own hard limit — raising that means restarting
-# user@1000.service and bouncing every user container on the host. A root unit
-# grants LimitMEMLOCK=infinity to the dank process via PID1, and the image
-# stays in dank's rootless podman storage.
+# WHY A ROOT UNIT (not a rootless quadlet): FreeToken pins ~120 GiB of
+# expert weights plus the ~47.7 GiB PLE ngram table in HOST RAM. Pinned
+# pages are charged against RLIMIT_MEMLOCK; dank's login session on this
+# host carries 8 MiB and a user quadlet is capped by the user manager's own
+# hard limit — raising that means restarting user@1000.service and bouncing
+# every user container on the host. A root unit grants
+# LimitMEMLOCK=infinity to the dank process via PID1, and the image stays
+# in dank's rootless podman storage.
 [Unit]
-Description=frank-primary — tcclaviger vLLM Qwen3.8-Flash-Next-Uncensored-FP8 on 2x R9700
+Description=frank-primary — FreeToken (PR587) Qwen3.8-Flash-Next-Uncensored-FP8 on 1x R9700
 After=network-online.target
 Wants=network-online.target
 # Never let the start-rate limiter give up after a crash loop (a failed LLM
@@ -32,34 +30,35 @@ StartLimitIntervalSec=0
 [Service]
 User=dank
 Group=dank
-# The tiered-expert offloader pins host RAM; an RLIMIT_MEMLOCK smaller than
-# the pinned set fails the load mid-start. PID1 grants infinity here —
+# The expert LRU + PLE pin ~170 GiB of host RAM; an RLIMIT_MEMLOCK smaller
+# than the pinned set fails the load mid-start. PID1 grants infinity here —
 # parity with the fleet's other LLM units.
 LimitMEMLOCK=infinity
 Environment="FRANK_PRIMARY_ENV_FILE=/home/dank/serverAppConfigs/scripts/frank-primary.env"
 ExecStart=/home/dank/serverAppConfigs/scripts/serve-frank-primary.sh
 # systemd stop: wait alone cannot stop the container child; stop it
-# explicitly. vLLM 0.26 drains gracefully on SIGTERM.
-ExecStop=/usr/bin/podman stop -t 120 frank-primary-qwen
-# Boot cost: FP8 payload page-in plus torch compile/graph capture —
-# ~15 min cold (measured 2026-09-29), faster warm.
+# explicitly. ft serve drops connections on SIGTERM; no graceful drain
+# contract is claimed (the OpenAI front end holds no queue to drain —
+# in-flight decodes are lost, which a stop during a bench window accepts).
+ExecStop=/usr/bin/podman stop -t 120 frank-primary
+# Boot cost: 131-shard FP8 page-in + PLE pin + expert upload. vLLM-era
+# budget (1800 s) carried forward; the FreeToken first boot measured here
+# once it ran (see the runbook for the real number).
 TimeoutStartSec=1800
-# Restart=always, NOT on-failure (measured 2026-10-01): a runtime
-# torch.OutOfMemoryError kills the engine while vLLM's API server still
-# exits 0, so ExecStart returned clean, systemd logged "Deactivated
-# successfully", and on-failure NEVER fired — the endpoint stayed down until
-# a human noticed. `always` restarts on any container death; an explicit
-# `systemctl stop` still wins (systemd does not restart a unit the operator
-# stopped), so bench windows and deploys are unaffected.
+# Restart=always, NOT on-failure (the 2026-10-01 vLLM lesson carries over:
+# a clean API-server exit after a worker fault reads as success to
+# systemd). `always` restarts on any container death; an explicit
+# `systemctl stop` still wins, so bench windows and deploys are unaffected.
 Restart=always
 RestartSec=10
-# ExecStop allows the engine 120 s to drain (`podman stop -t 120`); without an
-# explicit TimeoutStopSec systemd's 90 s default cuts the stop short, kills it,
-# and records 'Failed with result timeout' (measured 2026-10-01 13:27:17).
-# Must exceed the podman grace with margin for the engine's own shutdown.
+# ExecStop allows the engine 120 s to drain (`podman stop -t 120`); without
+# an explicit TimeoutStopSec systemd's 90 s default cuts the stop short,
+# kills it, and records 'Failed with result timeout' (measured 2026-10-01
+# 13:27:17). Must exceed the podman grace with margin.
 TimeoutStopSec=180
-# The engine pins tens of GiB of host RAM (expert host slabs); it is the
-# last process on this host that should be chosen by the kernel OOM killer.
+# The engine pins tens of GiB of host RAM (expert rows + PLE table); it is
+# the last process on this host that should be chosen by the kernel OOM
+# killer.
 OOMScoreAdjust=-500
 
 [Install]

scripts/frank-small.env

@@ -1,39 +1,79 @@
 # frank-small — llama.cpp (master, CUDA sm_86) serving
-# bartowski/MiMo-V2.6-Distill-Qwen-9B-GGUF:Q5_K_M on the RTX 3090, :7771.
+# ornith-ai/Ornith-1.5-9B-GGUF:Q8_0 on the RTX 3090, :7771.
 #
-# 2026-10-02 (Frank): THREE contexts, vision on, and a hard instruction to
-# leave 2-3 GiB of the card free. Both new demands cost memory, so the per-slot
-# window shrinks — see the budget below.
+# 2026-10-05 (Frank): the payload moved off bartowski/MiMo-V2.6 (Q6_K) to
+# ornith-ai/Ornith-1.5-9B Q8_0 (9,786,060,384 B, LFS oid 22086870...).
+# Header-verified facts that re-price the shape: qwen35 hybrid like MiMo, but
+# 33 blocks (not 32) with full_attention_interval 4 => NINE full-attention
+# layers (MiMo had 8), 16 Q heads / 4 KV heads, k/v length 256, native window
+# 262144, dense (no expert keys), nextn_predict_layers=1 (an MTP head is IN
+# the file — capability only; no spec decode ships, same law as before).
 #
-# Two measurements on this card, this runtime, price the terms:
+# THE KV CONTRACT STAYS f16 K + f16 V — NO KV QUANTIZATION (Frank, 2026-10-05,
+# overruling a written q8_0-V proposal). The arithmetic that makes this hard:
+#   KV rate 9 attn layers x 4 heads x 256 x 2 B = 18 KiB/token K, same again
+#   for V at f16 => 36 KiB/token (MiMo was ~24.5 measured: one fewer
+#   full-attn layer AND one fewer KV head).
+#   Weights +1,900 MiB (9,334 vs Q6_K's 7,432). Measured free at the Q6_K
+#   4096/2048 shape was 2,279 MiB; Q8_0 at f16/f16 does not fit that shape —
+#   the deficit is made up the way Frank ordered: UBATCH FIRST (2048 -> 1024,
+#   the measured 616 MiB rung), then POOL STEPS, then measure. Guess-and-check
+#   on the live card is the accepted method for llama.cpp sizing here: boot,
+#   read nvidia-smi free, step the pool (12,288-token increments, ~432 MiB
+#   each at the 36 KiB/token rate) until the 2-3 GiB floor holds. Compute
+#   buffers allocate at graph capture during BOOT, so a failure is loud and
+#   fast, not a surprise at the first long prompt.
+#
+# Previous shape history (MiMo era, for the rollback trail): 2026-10-03 the
+# rung went Q5_K_M -> Q6_K with the pool HELD at 401,408; the freed third
+# slot funded exact f16 V first; the ~3.5 GiB still free funded the +875 MiB
+# of weights.
+#
+# 2026-10-03 (Frank): TWO streams instead of three, each one WIDE — 200,704
+# tokens per conversation, above the 200,700 the harness compacts at — and the
+# KV cache is now FULLY UNQUANTIZED (f16 K, f16 V). The two moves are the same
+# move: llama.cpp divides --ctx-size across --parallel slots, so the slot given
+# up is exactly the VRAM that pays for unquantized V. No further quantization
+# anywhere on this endpoint.
+#
+# Three measurements on this card, this runtime, price the terms:
 #   2 x 262,144, no projector: 20,190 MiB used / 4,386 free
 #     KV = 524,288 tok x 24.5 KiB = 12,800 MiB => non-KV (weights 6,557 +
 #     DeltaNet state ~25/slot + compute buffers + CUDA graphs) = 7,390 MiB
 #   3 x 196,608, projector on: 22,972 MiB used / 1,153 free
 #     KV = 14,113 MiB => non-KV WITH the vision tower = 8,859 MiB
-# The tower costs ~1.4 GiB, not the ~1.0 its file size suggested, so 196,608
-# per slot left only 1.1 GiB free — outside the band. The pool therefore is
-# 3 x 180,224 = 540,672 tokens: KV ~12,936 + non-KV 8,859 = ~21,800 MiB predicted.
-# MEASURED at this shape: 21,684 MiB used / 2,441 MiB free — inside the band, and
-# the number the bats budget gate reproduces (it prices non-KV at 9 GiB).
+#   3 x 180,224, projector on: 21,684 MiB used / 2,441 free (the 2026-10-02
+#     shape) => non-KV = 8,748 MiB
+# Two slots give back one DeltaNet state (~25 MiB). Priced at f16 K+V
+# (32 KiB/token) this pool is 12,544 MiB, so the prediction is
+# 12,544 + ~8,800 = ~20.9 GiB used, ~3.1 GiB free — the band the 2026-10-02
+# shape hit, bought with precision instead of a slot. The 1024 ubatch
+# re-prices the compute buffers upward, so MEASURE after any change: if the
+# boot lands under 2 GiB free, step the pool down in 12,288-token
+# (384 MiB at this rate) increments.
 #
 # KV arithmetic: attention KV exists only in the 8 full-attention layers
-# (4 KV heads x 256 head_dim). K f16 = 16 KiB/token, V q8_0 = 8.5 KiB/token
-# (34 B per 32-element block) => 24.5 KiB/token, so 12,288 tokens (=4096/slot)
-# is one 293 MiB step — that is the ladder to walk if this moves again. The
-# native 262,144 per slot is NOT reachable three times: 3 x 262,144 x
-# 24.5 KiB = 18.4 GiB of KV before anything else is loaded.
+# (4 KV heads x 256 head_dim x 2 B) = 16 KiB/token for K and the same again for
+# V => 32 KiB/token, so 12,288 tokens (= 6144/slot) is one 384 MiB step — that
+# is the ladder to walk if this moves again. Two native windows (2 x 262,144)
+# are NOT reachable with full-precision V: 524,288 x 32 KiB = 16.1 GiB of KV
+# against a card with ~15.7 GiB of KV room once weights, the vision tower, the
+# DeltaNet states and the graphs are loaded. 200,704 per stream is what full
+# precision buys; q8_0 on V would buy 262,144 instead.
 #
-# Why V stays q8_0 and K stays f16: full-precision V would add 8 KiB/token —
-# +4.5 GiB on this pool — which is what pushes the card past its 24 GiB. K is
-# the tensor the asymmetric-KV contract exists to protect.
+# The asymmetric-KV contract this endpoint was created for (Frank, 2026-10-01
+# night: f16 K + q8_0 V) is now spent — it existed to make a third slot fit,
+# and the third slot is gone. The flags stay explicit so the choice is visible
+# in the argv either way.
 LLAMA_BIN=/home/dank/GitPackages/llama-master/build-cuda/bin/llama-server
-MODEL_FILE=/home/dank/models/MiMo-Q5KM/MiMo-V2.6-Distill-Qwen-9B-Q5_K_M.gguf
+MODEL_FILE=/home/dank/models/Ornith-Q8/Ornith-1.5-9B-Q8_0.gguf
 
-# Vision: the Q5_K_M GGUF ships no `v.*` tensors, so the projector file IS the
-# vision capability (sha256 ff348f31..., 918,166,048 B, fetched by
+# Vision: the Ornith Q8_0 GGUF ships no `v.*` tensors either (dense-9B byte
+# arithmetic: 9.79 GB ≈ 9B params x ~1.06 B/param leaves no room for a tower),
+# so the projector file IS the vision capability (mmproj-Ornith-1.5-9B-BF16
+# .gguf, 921,704,672 B, LFS oid 626f9f90..., fetched by
 # scripts/download-frank-small.sh). Empty string = text-only on purpose.
-MMPROJ_FILE=/home/dank/models/MiMo-Q5KM/mmproj-MiMo-V2.6-Distill-Qwen-9B-f16.gguf
+MMPROJ_FILE=/home/dank/models/Ornith-Q8/mmproj-Ornith-1.5-9B-BF16.gguf
 
 # Fleet endpoint: same port and alias as every frank-small before it, so the
 # harness models.yml id and the homepage probe do not move.
@@ -41,20 +81,56 @@ ALIAS=frank-small
 BIND_ADDR=0.0.0.0
 LISTEN_PORT=7771
 
-# Three independent conversations. llama.cpp divides --ctx-size across
-# --parallel slots (n_ctx_slot = CTX_SIZE/PARALLEL), and a mamba-hybrid model
-# cannot use unified KV, so each slot gets its own cache and its own window.
-# 180,224 per slot is BELOW the native 262,144, and that is the cost of the
-# third slot: a longer single prompt needs compaction instead of one shot.
-# Consequence worth knowing: frank-primary's compaction trigger is 183,501
-# tokens, which no longer fits a frank-small slot — see the runbook's
-# "three slots vs compaction" note.
-CTX_SIZE=540672
-PARALLEL=3
+# ONE conversation at the NATIVE 262,144 window (Frank, 2026-10-05). The
+# two-slot shape died under Ornith and the NO-QUANTIZE law, by arithmetic:
+# the KV rate is the ARCHITECTURE's, not the quant's — 9 full-attn layers x
+# 4 KV heads x 256 dim x 2 B = 36 KiB/token at f16 K+V (MiMo measured 24.5),
+# so 2 x 200,704 wants ~14.1 GiB of KV on top of +1,900 MiB of Q8 weights
+# against 2,279 MiB measured free; ubatch 1024 claws back 616, and no
+# pool step fits BOTH the 2-3 GiB floor and the harness compaction
+# invariant (slot >= 191,148). One native slot fits with room; the window
+# per conversation RISES 200,704 -> 262,144. The cost: a second concurrent
+# request queues behind the released slot (subagent concurrency 2 -> 1).
+CTX_SIZE=262144
+PARALLEL=1
 
-# The KV contract (Frank, 2026-10-01 night): unquantized K, q8_0 V.
-# q8_0 on V is only served by the flash-attention path; the serve script
-# pins --flash-attn on and must never lose it.
+# Compute-buffer geometry. This endpoint passed neither flag until 2026-10-03,
+# so it ran on llama.cpp's own defaults (2048/512); the pair is now 4096/2048.
+# Both are tunables because the pair is a measurement, not a law — and the
+# measurement is CARD-SPECIFIC. On the 2x R9700 layer-split sibling the ubatch
+# curve peaks at 512 and every step up costs GiB of compute buffer for no
+# speed (see frank-medium-batch-ubatch). On THIS CUDA 3090 the curve climbs
+# monotonically to 2048: 710 -> 1188 t/s prefill from ubatch 1024 to 2048 on
+# the 35B MoE sweep, 1944 -> 2423 t/s on the Ornith 8k sweep, with decode flat
+# to under 1 t/s on every rung. Ubatch is a prefill knob; decode never moves.
+#
+# The logical batch is free — two independent sweeps agree that at a fixed
+# ubatch, 2048/4096/8192 give identical prefill speed AND identical peak VRAM,
+# because the compute buffers are sized by the ubatch alone. Doubling it costs
+# nothing and buys nothing; it is here because it is the value the pair was
+# measured at.
+#
+# THE RISK IS ALL IN THE UBATCH. The 1024 -> 2048 step cost ~1,244 MiB of peak
+# graph VRAM on the 35B MoE (7,330 -> 8,574 MiB); this 9B hybrid has far
+# smaller compute tensors, so the delta should be well under that, but the
+# predecessor Q5_K_M shape recorded "ubatch 2048 failed the scratch boot" at
+# ~3,571 MiB free. The Q8_0 rung ships UBATCH 1024 (Frank, 2026-10-05: lower
+# ubatch FIRST — the measured 616 MiB step is the first VRAM the Q8 weights
+# claw back, before any pool move). Compute buffers are allocated at graph
+# capture during BOOT, so a failure is loud and fast, not a surprise at the
+# first long prompt. MEASURE nvidia-smi free immediately after the deploy:
+# under ~1.5 GiB, or a boot failure, step the POOL down next (12,288-token
+# increments, ~432 MiB each at the Ornith 36 KiB/token f16 KV rate).
+BATCH_SIZE=4096
+UBATCH_SIZE=1024
+
+# The KV contract (2026-10-05, Frank): f16 K + f16 V — NO KV quantization.
+# The q8_0-V arithmetic was written and OVERRULED. The Q8 weights eat the
+# VRAM that funded f16 V on 2026-10-03, so the remedy ladder is UBATCH FIRST
+# (2048 -> 1024, the measured 616 MiB rung), then POOL STEPS — and
+# guess-and-check on the live card is the accepted method: boot, read
+# nvidia-smi, step down until the floor holds. --flash-attn stays pinned on
+# in the serve script: it is the fast path for f16 KV.
 CACHE_TYPE_K=f16
-CACHE_TYPE_V=q8_0
+CACHE_TYPE_V=f16
 NGL=99

scripts/frank-small.service

@@ -1,5 +1,5 @@
 [Unit]
-Description=llama.cpp MiMo-V2.6-Distill-Qwen-9B Q5_K_M GGUF (qwen35 hybrid, 3x180224, f16 K + q8_0 V, mmproj vision) on the RTX 3090 (dankclaw, frank-small)
+Description=llama.cpp MiMo-V2.6-Distill-Qwen-9B Q6_K GGUF (qwen35 hybrid, 2x200704, unquantized f16 KV, mmproj vision) on the RTX 3090 (dankclaw, frank-small)
 After=network-online.target
 Wants=network-online.target
 

scripts/serve-frank-large.sh

@@ -1,179 +1,146 @@
 #!/usr/bin/env bash
-# Serve kingjones777/Qwen3.8-27B-ROCmFP4-STRIX-MTP-GGUF as frank-large on
-# Dankclaw's Strix Halo iGPU (gfx1151) with the ROCmFPX fork of llama.cpp.
+# serve-frank-large.sh — frank-large endpoint: halogen (closed-source
+# C++/HIP engine, image ghcr.io/peonist-ai/halogen, digest-pinned in
+# frank-large.env) serving peonist-ai/halogen-qwen3.8-27b on Dankclaw's
+# Strix Halo iGPU, host :7770. 2026-10-05 cutover from the ROCmFPX
+# llama.cpp shape — revert the swap commit's trio to roll back (payload
+# and fork build stay on disk).
 #
-# 2026-10-02 (Frank): swapped from gufo + Q4_K_XL to the highest-quality file
-# in the repo, one session at the native 262,144 window.
+# The engine is driven by `podman run` DIRECTLY (the fleet's frank-primary
+# pattern): the image entrypoint `halogen-docker` in default "all" mode
+# starts engine + OpenAI frontend in one container. The frontend listens on
+# 8731 inside (HALOGEN_API_PORT default) and is published as host 7770; the
+# engine's own token port (8730, no auth) stays unpublished on the
+# container's loopback. The image ships NO default checkpoint:
+# HALOGEN_CHECKPOINT selects the file inside the /models mount.
 #
-# Why this runtime and not the previous one: the payload is ftype 115 — 311
-# tensors of GGML_TYPE_Q8_0_ROCMFPX (103), a tensor type that exists only in
-# the ROCmFPX fork. Stock llama.cpp, SGLang and gufo all refuse the file, and
-# gufo additionally gates every "Qwen3.8" artifact on a template-hash
-# allowlist. The qualified build is scripts/build-rocmfpx.sh's pin: it loads
-# ftype 115, survives warmup (the 2026-08-25 builds of the older fork line
-# segfaulted in ggml_compute_forward_set_rows on this exact file), and drafts
-# with the model's own MTP head.
+# Gates mirror the deployer EXACTLY (the 2026-10-01 crash-loop lesson —
+# serve gate and deploy gate disagreeing = green deploy, red unit):
+# tokenizer directory present with every named file a FLAT REGULAR FILE —
+# the model card ships `tokenizer/` as a flat directory and the engine
+# defaults HALOGEN_TOKENIZER to /tokenizer, so a HF cache snapshot's
+# symlinks into blobs/ dangle inside the mount; GPU device nodes present.
 #
-# Why this file: Q8_0_ROCMFPX_AGENT is the top of the repo's quality ladder
-# (26.28 GiB, 8.39 bpw) and the AGENT routing keeps 187 tensors — the
-# output-head set the draft scores against — at plain Q8_0 instead of the 2
-# the non-AGENT build keeps, which is the variant the card names for MTP
-# serving. Drafting uses the repo's Q4_0 MTP head: the head's cost is its
-# projection over the 248K vocabulary, so the 4-bit head is faster end to end
-# at equal-or-better acceptance than its Q8_0 sibling.
-#
-# The client contract lives here, not in frank-large.env:
-#   * --jinja — without it llama.cpp silently IGNORES the harness's
-#     chat_template_kwargs (enable_thinking / preserve_thinking), so the
-#     endpoint would look like it had no thinking control at all. The GGUF
-#     carries the official Qwen template, so no --chat-template-file.
-#   * --reasoning-format deepseek — thinking is emitted as reasoning_content,
-#     the field the harness reads for this endpoint.
-#   * the template accepts reasoning_effort xhigh|medium|low only, which is
-#     why models.yml folds the harness's levels onto that set.
-#   * --alias frank-large — that is the model id the harness resolves.
-#   * --ctx-checkpoints / --cache-ram — prompt-cache geometry for the 16
-#     sliding-window-style layers; harmless where the hybrid state forces a
-#     full re-process, and required for the checkpoint feature to work at all.
+# Why containers instead of the bare binary (the old frank-large shape):
+# the payload is .hgn and only halogen reads it; halogen ships as a
+# container; the root unit keeps LimitMEMLOCK=infinity for parity and to
+# outlive the login session, exactly like frank-primary's.
 set -euo pipefail
 
-# Fleet values live in frank-large.env. Anything already in the environment
-# wins over the file, so an operator can override one value on a single run:
-# the caller's values are captured, the file is sourced, then they are put
-# back. (A plain `source` would let the file clobber the environment.)
-TUNABLES=(LLAMA_BIN MODEL_FILE DRAFT_FILE MMPROJ_FILE ALIAS BIND_ADDR
-          LISTEN_PORT CTX_SIZE PARALLEL BATCH_SIZE UBATCH_SIZE NGL FIT
-          FLASH_ATTN CACHE_TYPE_K CACHE_TYPE_V SPEC_TYPE SPEC_DRAFT_NGL
-          SPEC_DRAFT_DEVICE SPEC_DRAFT_N_MAX SPEC_DRAFT_N_MIN SPEC_DRAFT_P_MIN
-          TEMPERATURE TOP_P TOP_K MIN_P PRESENCE_PENALTY NUM_THREADS
-          THREAD_BATCH ROCM_PATH HIP_VISIBLE HSA_GFX_VERSION)
-_OVERRIDES=()
-for _var in "${TUNABLES[@]}"; do
-    [[ -n "${!_var+set}" ]] && _OVERRIDES+=("$_var=${!_var}")
-done
-ENV_FILE="${FRANK_LARGE_ENV_FILE:-$(dirname "$0")/frank-large.env}"
-if [[ -f "$ENV_FILE" ]]; then
-    set -a
-    # shellcheck disable=SC1090
-    source "$ENV_FILE"
-    set +a
+SCRIPT_DIR="$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)"
+ENV_FILE="${FRANK_LARGE_ENV_FILE:-$SCRIPT_DIR/frank-large.env}"
+if [ ! -f "$ENV_FILE" ]; then
+    echo "frank-large: env file not found: $ENV_FILE" >&2
+    exit 1
 fi
-for _kv in "${_OVERRIDES[@]+${_OVERRIDES[@]}}"; do
-    export "${_kv?}"
-done
-unset _var _kv
+set -a
+# shellcheck disable=SC1090
+source "$ENV_FILE"
+set +a
 
-: "${LLAMA_BIN=/home/dank/GitPackages/kingjones-ROCmFPX/build/bin/llama-server}"
-: "${MODEL_FILE=/home/dank/models/kingjones-Qwen3.8-27B-ROCmFP4-STRIX-MTP/Qwen3.8-27B-Q8_0_ROCMFPX_AGENT.gguf}"
-: "${DRAFT_FILE=/home/dank/models/kingjones-Qwen3.8-27B-ROCmFP4-STRIX-MTP/mtp-Qwen3.8-27B-Q4_0.gguf}"
-: "${MMPROJ_FILE=/home/dank/models/kingjones-Qwen3.8-27B-ROCmFP4-STRIX-MTP/mmproj-Qwen3.8-27B-BF16.gguf}"
-: "${ALIAS=frank-large}"
-: "${BIND_ADDR=0.0.0.0}"
-: "${LISTEN_PORT=7770}"
-: "${CTX_SIZE=262144}"
-: "${PARALLEL=1}"
-: "${BATCH_SIZE=2048}"
-: "${UBATCH_SIZE=512}"
-: "${NGL=999}"
-: "${FIT=off}"
-: "${FLASH_ATTN=on}"
-: "${CACHE_TYPE_K=f16}"
-: "${CACHE_TYPE_V=f16}"
-: "${SPEC_TYPE=draft-mtp}"
-: "${SPEC_DRAFT_NGL=99}"
-: "${SPEC_DRAFT_DEVICE=ROCm0}"
-: "${SPEC_DRAFT_N_MAX=4}"
-: "${SPEC_DRAFT_N_MIN=0}"
-: "${SPEC_DRAFT_P_MIN=0.0}"
-: "${TEMPERATURE=1.0}"
-: "${TOP_P=0.95}"
-: "${TOP_K=20}"
-: "${MIN_P=0.0}"
-: "${PRESENCE_PENALTY=0.0}"
-: "${NUM_THREADS=16}"
-: "${THREAD_BATCH=16}"
-: "${ROCM_PATH=/opt/rocm}"
-: "${HIP_VISIBLE=0}"
-: "${HSA_GFX_VERSION=11.5.1}"
+: "${FRANK_LARGE_IMAGE:?FRANK_LARGE_IMAGE must name the digest-pinned image}"
+: "${FRANK_LARGE_CONTAINER:?FRANK_LARGE_CONTAINER must name the container}"
+: "${FRANK_LARGE_PUBLISH:?FRANK_LARGE_PUBLISH must be host:container}"
+: "${FRANK_LARGE_MODEL_DIR:?FRANK_LARGE_MODEL_DIR must name the payload dir}"
+: "${FRANK_LARGE_CHECKPOINT:?FRANK_LARGE_CHECKPOINT must name the .hgn inside it}"
+: "${FRANK_LARGE_CHECKPOINT_BYTES:?FRANK_LARGE_CHECKPOINT_BYTES must pin the size}"
+: "${FRANK_LARGE_TOKENIZER_FILES:?FRANK_LARGE_TOKENIZER_FILES must list the tokenizer dir contents}"
+: "${HALOGEN_MODEL_ID:?HALOGEN_MODEL_ID must be the served alias (frank-large)}"
+: "${HALOGEN_SLOT_CTX:?HALOGEN_SLOT_CTX must be set}"
+: "${HALOGEN_KV_SLOTS:?HALOGEN_KV_SLOTS must be set (1 keeps the drafter on)}"
+: "${HALOGEN_DRAFTER:?HALOGEN_DRAFTER must be set (2 = DFlash2)}"
 
-[[ -n "$LLAMA_BIN" ]] || { echo "frank-large: LLAMA_BIN is empty" >&2; exit 1; }
-[[ -x "$LLAMA_BIN" ]] || {
-    echo "frank-large: llama-server '$LLAMA_BIN' is not executable;" >&2
-    echo "           build it with scripts/build-rocmfpx.sh" >&2
+# Capture podman's output BEFORE grepping: under `set -o pipefail` the
+# `podman images | grep -q` pattern races (grep exits on first match,
+# podman dies SIGPIPE 141, pipefail turns the 141 into "no match" — the
+# 2026-09-30 deploy refusal).
+DIGESTS="$(podman images --format '{{.RepoDigests}}')"
+if ! grep -qF "${FRANK_LARGE_IMAGE#*@}" <<<"$DIGESTS"; then
+    echo "frank-large: image $FRANK_LARGE_IMAGE not in podman storage;" >&2
+    echo "  pull it: podman pull ${FRANK_LARGE_IMAGE%@*}" >&2
     exit 1
-}
-# Refuse anything but the real payload. A path that is not a readable file
-# makes llama-server hang rather than fail clean.
-[[ -r "$MODEL_FILE" ]] || {
-    echo "frank-large: model file '$MODEL_FILE' is not readable;" >&2
-    echo "           fetch it with scripts/download-frank-large-kingjones.sh" >&2
+fi
+
+# The checkpoint gate is size-exact: a truncated curl exits 0 on some
+# failures and a partial .hgn loads far enough to look alive, so the byte
+# count from the model card is the contract (the fleet fetch law).
+CKPT="$FRANK_LARGE_MODEL_DIR/$FRANK_LARGE_CHECKPOINT"
+if [ ! -f "$CKPT" ]; then
+    echo "frank-large: checkpoint $CKPT missing;" >&2
+    echo "  fetch it: bash scripts/download-frank-large-halogen.sh" >&2
     exit 1
-}
-# The MTP head is the performance feature: without it the Q8_0 weights decode
-# at memory-bandwidth speed and the endpoint loses the ~1.4x the head buys.
-[[ -r "$DRAFT_FILE" ]] || {
-    echo "frank-large: MTP draft head '$DRAFT_FILE' is not readable;" >&2
-    echo "           fetch it with scripts/download-frank-large-kingjones.sh" >&2
+fi
+CKPT_SIZE="$(stat -c %s "$CKPT")"
+if [ "$CKPT_SIZE" != "$FRANK_LARGE_CHECKPOINT_BYTES" ]; then
+    echo "frank-large: checkpoint is $CKPT_SIZE bytes, expected $FRANK_LARGE_CHECKPOINT_BYTES (partial download?)" >&2
     exit 1
-}
-# The GGUF carries no vision tower; the projector file is the whole tower, so
-# a missing file means a vision request would 500 rather than degrade.
-# MMPROJ_FILE="" is the documented opt-out for a text-only endpoint.
-MM_ARGS=()
-if [[ -n "${MMPROJ_FILE:-}" ]]; then
-    [[ -r "$MMPROJ_FILE" ]] || {
-        echo "frank-large: mmproj '$MMPROJ_FILE' is not readable; set MMPROJ_FILE='' for text-only" >&2
+fi
+
+# The payload ships the tokenizer as a FLAT DIRECTORY (model card:
+# "tokenizer/ flat tokenizer directory"). A HuggingFace cache snapshot
+# symlinks each file into a sibling blobs/, and those symlinks dangle
+# inside the container mount — so the directory must be real and every
+# named file must be a regular file, not a link.
+TOK_DIR="$FRANK_LARGE_MODEL_DIR/tokenizer"
+if [ -L "$TOK_DIR" ] || [ ! -d "$TOK_DIR" ]; then
+    echo "frank-large: tokenizer dir $TOK_DIR missing or a symlink (HF cache snapshot?);" >&2
+    echo "  fetch it: bash scripts/download-frank-large-halogen.sh" >&2
+    exit 1
+fi
+for TF in $FRANK_LARGE_TOKENIZER_FILES; do
+    if [ ! -f "$TOK_DIR/$TF" ] || [ -L "$TOK_DIR/$TF" ]; then
+        echo "frank-large: tokenizer file $TOK_DIR/$TF missing or a symlink;" >&2
+        echo "  fetch it: bash scripts/download-frank-large-halogen.sh" >&2
         exit 1
-    }
-    MM_ARGS=(--mmproj "$MMPROJ_FILE")
+    fi
+done
+
+# GPU access is mandatory but the node paths are variables so the seam can
+# drive the gate on a box without an AMD GPU.
+DEV_KFD="${FRANK_LARGE_DEV_KFD:-/dev/kfd}"
+DEV_DRI="${FRANK_LARGE_DEV_DRI:-/dev/dri}"
+if [ ! -e "$DEV_KFD" ] || [ ! -d "$DEV_DRI" ]; then
+    echo "frank-large: $DEV_KFD or $DEV_DRI missing — no gfx1151 to serve on" >&2
+    exit 1
 fi
+for CTR in ${FRANK_LARGE_LEGACY_CONTAINERS:-} "$FRANK_LARGE_CONTAINER"; do
+    podman rm -f "$CTR" >/dev/null 2>&1 || true
+done
 
-# HIP/ROCm host environment. HIP reads bare env vars (TEMP, HOST,
-# GPU_MAX_HEAP_SIZE), which is why the tunables above are named TEMPERATURE
-# and BIND_ADDR. HSA_OVERRIDE_GFX_VERSION is what lets a runtime built for a
-# neighbouring gfx target run on this iGPU; the qualified pin targets gfx1151
-# natively and the override is a no-op for it.
-export ROCM_PATH
-export HIP_VISIBLE_DEVICES="$HIP_VISIBLE"
-export HSA_OVERRIDE_GFX_VERSION="$HSA_GFX_VERSION"
-export LD_LIBRARY_PATH="$(dirname "$LLAMA_BIN"):${ROCM_PATH}/lib:${ROCM_PATH}/lib/rocm_sysdeps/lib:${ROCM_PATH}/lib/llvm/lib${LD_LIBRARY_PATH:+:$LD_LIBRARY_PATH}"
+LOG="${FRANK_LARGE_LOG:-/home/dank/logs/frank_large.log}"
+mkdir -p "$(dirname "$LOG")"
+# for the udev-granted supplementary group, seccomp unconfined for the ROCm
+# ioctl surface, and unbounded memlock (the unit's LimitMEMLOCK=infinity is
+# inherited, so runc never has to RAISE it). NO --ulimit nofile: Dankclaw's
+# hard limit is 524288, rootless runc cannot raise a hard limit and dies
+# with "error setting rlimit type 7: operation not permitted" (measured
+# 2026-10-05 cutover), and 524288 fds is already ~100x what the engine
+# wants; the model card and both sibling serve scripts pass no nofile
+# either. No comment lines inside the backslash
+# chain — a mid-chain # terminates the continuation (the 2026-10-05
+# exit-127 crash loop on the sibling serve script).
+podman run -d --name "$FRANK_LARGE_CONTAINER" \
+  --publish "$FRANK_LARGE_PUBLISH" \
+  --ipc=host \
+  --ulimit memlock=-1:-1 \
+  --device "$DEV_KFD" --device "$DEV_DRI" \
+  --group-add keep-groups \
+  --security-opt seccomp=unconfined \
+  -e HALOGEN_MODEL_ID="$HALOGEN_MODEL_ID" \
+  -e HALOGEN_CHECKPOINT="/models/$FRANK_LARGE_CHECKPOINT" \
+  -e HALOGEN_SLOT_CTX="$HALOGEN_SLOT_CTX" \
+  -e HALOGEN_KV_SLOTS="${HALOGEN_KV_SLOTS:-1}" \
+  -e HALOGEN_DRAFTER="$HALOGEN_DRAFTER" \
+  -e HALOGEN_MAX_TOKENS_CAP="${HALOGEN_MAX_TOKENS_CAP:-65536}" \
+  -e HALOGEN_QUEUE_TIMEOUT="${HALOGEN_QUEUE_TIMEOUT:-7200}" \
+  ${HALOGEN_CACHE_MB:+-e HALOGEN_CACHE_MB="$HALOGEN_CACHE_MB"} \
+  -v "$FRANK_LARGE_MODEL_DIR:/models:ro" \
+  -v "$FRANK_LARGE_MODEL_DIR/tokenizer:/tokenizer:ro" \
+  "$FRANK_LARGE_IMAGE" \
+  > "$LOG" 2>&1
 
-exec /usr/bin/env "$LLAMA_BIN" \
-    --model "$MODEL_FILE" \
-    "${MM_ARGS[@]}" \
-    --alias "$ALIAS" \
-    --host "$BIND_ADDR" \
-    --port "$LISTEN_PORT" \
-    --jinja \
-    --device ROCm0 \
-    --n-gpu-layers "$NGL" \
-    --parallel "$PARALLEL" \
-    --ctx-size "$CTX_SIZE" \
-    --batch-size "$BATCH_SIZE" \
-    --ubatch-size "$UBATCH_SIZE" \
-    --threads "$NUM_THREADS" \
-    --threads-batch "$THREAD_BATCH" \
-    --fit "$FIT" \
-    --flash-attn "$FLASH_ATTN" \
-    --direct-io \
-    --cache-type-k "$CACHE_TYPE_K" \
-    --cache-type-v "$CACHE_TYPE_V" \
-    --ctx-checkpoints 16 \
-    --cache-ram 8192 \
-    --cache-idle-slots \
-    --temp "$TEMPERATURE" \
-    --top-p "$TOP_P" \
-    --top-k "$TOP_K" \
-    --min-p "$MIN_P" \
-    --presence-penalty "$PRESENCE_PENALTY" \
-    --reasoning on \
-    --reasoning-format deepseek \
-    --reasoning-budget -1 \
-    --spec-type "$SPEC_TYPE" \
-    --model-draft "$DRAFT_FILE" \
-    --spec-draft-ngl "$SPEC_DRAFT_NGL" \
-    --spec-draft-device "$SPEC_DRAFT_DEVICE" \
-    --spec-draft-n-max "$SPEC_DRAFT_N_MAX" \
-    --spec-draft-n-min "$SPEC_DRAFT_N_MIN" \
-    --spec-draft-p-min "$SPEC_DRAFT_P_MIN" \
-    --metrics
+# This script's lifetime IS the container's: podman wait returns the
+# container's exit code, so a crash is a nonzero ExecStart and the unit's
+# Restart=always is real. systemd stop goes through the unit's ExecStop
+# (podman wait alone cannot stop the child).
+exec podman wait "$FRANK_LARGE_CONTAINER"

scripts/serve-frank-primary.sh

@@ -1,31 +1,31 @@
 #!/usr/bin/env bash
-# serve-frank-primary.sh — frank-primary endpoint (tcclaviger RDNA4 vLLM fork,
-# orcarouter Qwen3.8-Flash-Next-Uncensored-FP8 + MTP4 spec decode, 2x R9700,
-# dankserver :7770; 2026-09-29 cutover from the R9V radiance fork — git tag
-# frank-primary-r9v-q8-20260929 is the pre-cutover state).
-#
-# The engine is driven by `podman run` DIRECTLY: the image's entrypoint execs
-# `vllm serve "$@"`, so the run line below IS the engine argv. This replaces
-# the R9V launcher/shim machinery (that fork needed a docker->podman shim and
-# a 104-key on-host setup state; this stack is fully expressed by flags, and
-# every flag here was qualified by boot probe on 2026-09-29 — see
-# frank-primary.env for the qualification record).
-#
-# Shape (2026-10-02, Frank's correction of the 10-01 bf16 shape): MAXLEN
-# 200704 x SEQS 2 (401,408 tokens of contract), KV fp8 (--kv-cache-dtype
-# fp8; the bf16 unquantized shape is retired to git tag
-# frank-primary-fp8-mtp-20261001), KV_BYTES 3,150,000,000 PER RANK (the
-# engine measured 405,000 tokens of capacity / ~3.0x concurrency), NO
-# speculative decode, --max-num-batched-tokens 1024 (Frank's lock), MoE
-# gather geometry R4D_LRU_GATHER_CHUNKS=32 /
-# R4D_LRU_GATHER_LANES=64 (+18.7% decode, grows with depth), prefix
-# caching, expert prefetch OFF (36% precision measured), TunableOp off, and
-# the pool deliberately small so the expert arena keeps what it reclaims.
+# serve-frank-primary.sh — frank-primary endpoint (FreeToken PR #587 ROCm
+# build, orcarouter Qwen3.8-Flash-Next-Uncensored-FP8, ONE R9700 gfx1201,
+# dankserver :7770; 2026-10-05 cutover from the tcclaviger vLLM fork — git
+# tag frank-primary-vllm-shapeG-20261003 is the pre-cutover state).
 #
+# The engine is driven by `podman run` DIRECTLY and the run line below IS
+# the engine argv: the image carries FreeToken installed at pinned PR head
+# 77a7201 (scripts/docker/frank-primary-freetoken.Dockerfile builds it from
+# the PR's own docs/install_amd.md stack). Unlike the vLLM fork there is no
+# per-boot planner: experts live in pinned host RAM with an LRU GPU cache,
+# KV is an explicit token pool, and the boot ledger in the log names
+# Shape: 1 GPU (first-boot conservative — the second R9700 sits idle;
+# --tensor-parallel-size EXISTS in FreeToken, so a 2-card TP shape is future
+# tuning once this single-card shape has a decode baseline), KV pool 300,000
+# tokens (24 KiB/token f16 across the 12 full-attention layers = 7.2 GiB;
+# clears the harness worst case of 177,356 tokens and admits one NATIVE
+# 262,144-token conversation), MoE LRU at 10% of all experts (~12 GiB),
+# 2 decode slots. First-boot conservative; `ft ctl cache --kv/--moe` grows
+# pools live, and the env comment block carries the resize law. Flag
+# surface verified against docs/cli.md of the pinned head AND against
+# `ft serve --help` inside the built image; every value here is env-driven
+# (frank-primary.env).
+
 # The root unit (scripts/frank-primary.service, User=dank,
-# LimitMEMLOCK=infinity) grants the pin budget for the tiered-expert host
-# slabs (~20.5 GiB/rank of pinned anonymous memory). A plain shell session
-# also works, but the unit is the fleet-parity boot-persistent shape.
+# LimitMEMLOCK=infinity) grants the pin budget: ~120 GiB of expert weights
+# plus the ~47.7 GiB PLE table pin in host RAM. A plain shell session also
+# works, but the unit is the fleet-parity boot-persistent shape.
 set -euo pipefail
 
 SCRIPT_DIR="$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)"
@@ -40,27 +40,30 @@ set -a
 source "$ENV_FILE"
 set +a
 
-: "${FRANK_PRIMARY_IMAGE:?FRANK_PRIMARY_IMAGE must name the digest-pinned image}"
+: "${FRANK_PRIMARY_IMAGE:?FRANK_PRIMARY_IMAGE must name the FreeToken image}"
 : "${FRANK_PRIMARY_MODEL_DIR:?FRANK_PRIMARY_MODEL_DIR must name the FP8 payload}"
 : "${FRANK_PRIMARY_MODEL_SHARDS:?FRANK_PRIMARY_MODEL_SHARDS must give the shard count}"
 : "${FRANK_PRIMARY_CONTAINER:?FRANK_PRIMARY_CONTAINER must name the container}"
 : "${FRANK_PRIMARY_SERVED_NAME:?FRANK_PRIMARY_SERVED_NAME must name the alias}"
 : "${FRANK_PRIMARY_PUBLISH:?FRANK_PRIMARY_PUBLISH must be host:container}"
-: "${FRANK_PRIMARY_MAXLEN:?FRANK_PRIMARY_MAXLEN must be set}"
-: "${FRANK_PRIMARY_SEQS:?FRANK_PRIMARY_SEQS must be set}"
-: "${FRANK_PRIMARY_KV_BYTES:?FRANK_PRIMARY_KV_BYTES must be set (PER RANK)}"
-: "${FRANK_PRIMARY_KV_DTYPE:?FRANK_PRIMARY_KV_DTYPE must be set (auto = bf16 KV)}"
-: "${FRANK_PRIMARY_REASONING_PARSER:?FRANK_PRIMARY_REASONING_PARSER must be set (qwen3)}"
-: "${FRANK_PRIMARY_TOOL_PARSER:?FRANK_PRIMARY_TOOL_PARSER must be set (qwen3_xml)}"
-: "${FRANK_PRIMARY_CHAT_TEMPLATE_KWARGS:?FRANK_PRIMARY_CHAT_TEMPLATE_KWARGS must be set}"
-# Capture podman's output BEFORE grepping: under `set -o pipefail`, the
-# classic `podman images | grep -q` pattern races — grep exits on its first
-# match, podman dies SIGPIPE (141) writing the remaining lines, and pipefail
-# turns the 141 into "no match" (the 2026-09-30 deploy refusal).
-DIGESTS="$(podman images --format '{{.RepoDigests}}')"
-if ! grep -qF "${FRANK_PRIMARY_IMAGE#*@}" <<<"$DIGESTS"; then
-    echo "serve-frank-primary: pinned image absent from rootless storage: $FRANK_PRIMARY_IMAGE" >&2
-    echo "  fetch it: podman pull $FRANK_PRIMARY_IMAGE" >&2
+: "${FRANK_PRIMARY_FT_GPU:?FRANK_PRIMARY_FT_GPU must name the GPU index}"
+: "${FRANK_PRIMARY_FT_NUM_TOKENS:?FRANK_PRIMARY_FT_NUM_TOKENS must size the KV pool}"
+: "${FRANK_PRIMARY_FT_MAX_RUNNING:?FRANK_PRIMARY_FT_MAX_RUNNING must be set}"
+: "${FRANK_PRIMARY_FT_MAX_OUTPUT_TOKENS:?FRANK_PRIMARY_FT_MAX_OUTPUT_TOKENS must be set}"
+: "${FRANK_PRIMARY_FT_MOE_RATE:?FRANK_PRIMARY_FT_MOE_RATE must be set}"
+
+# The image gate is EXISTENCE IN STORAGE, not a registry digest: the tag is
+# built ON THE HOST (podman build from
+# scripts/docker/frank-primary-freetoken.Dockerfile) and local builds carry
+# no RepoDigests entry — the 362-line vLLM-era RepoDigests grep refused
+# exactly this shape. Pinning lives in the Dockerfile instead: the FreeToken
+# source is checked out at a hardcoded PR-head SHA and the base image is
+# digest-pinned there. `podman image exists` accepts both a local tag and a
+# name@digest ref, and it is one command — no pipefail/grep race (the
+# 2026-09-30 lesson) for either shape.
+if ! podman image exists "$FRANK_PRIMARY_IMAGE"; then
+    echo "serve-frank-primary: image absent from rootless storage: $FRANK_PRIMARY_IMAGE" >&2
+    echo "  build it: podman build -t $FRANK_PRIMARY_IMAGE -f scripts/docker/frank-primary-freetoken.Dockerfile scripts/docker/" >&2
     exit 1
 fi
 
@@ -87,8 +90,8 @@ fi
 
 # A crashed/killed unit can leave the container name registered (systemd
 # SIGKILLs conmon before cleanup runs). Clear OUR name plus the legacy
-# identities (the retired R9V container and the ad hoc experiment container
-# both hold these GPUs); all idempotent.
+# identities (the vLLM-era frank-primary-qwen, the retired R9V container and
+# the ad hoc experiment containers all hold this GPU); all idempotent.
 for CTR in $FRANK_PRIMARY_LEGACY_CONTAINERS "$FRANK_PRIMARY_CONTAINER"; do
     podman rm -f "$CTR" >/dev/null 2>&1 || true
 done
@@ -97,39 +100,33 @@ LOG=/home/dank/logs/frank_primary.log
 mkdir -p "$(dirname "$LOG")"
 
 # --device /dev/dri is the WHOLE DIRECTORY (individual renderD nodes enumerate
-# without the card nodes and HIP refuses contexts — "No CUDA GPUs are
-# available" while device_count() says 2). --ipc=host for the engine's shared
-# memory; memlock unbounded for the pinned host slabs.
-# The entrypoint already execs `vllm serve "$@"`, so no `serve` here.
+# without the card nodes and HIP refuses contexts). --ipc=host for the
+# engine's shared memory; memlock unbounded for the ~120 GiB pinned expert
+# rows + the 47.7 GiB PLE pin (the unit's LimitMEMLOCK=infinity is the other
+# half of that law).
+# HIP_VISIBLE_DEVICES pins visibility to ONE card so in-container index 0 is
+# the right R9700; --gpu 0 selects it inside the visible set.
 podman run -d --name "$FRANK_PRIMARY_CONTAINER" \
   --publish "$FRANK_PRIMARY_PUBLISH" \
   --ipc=host --ulimit memlock=-1:-1 \
   --cap-add SYS_PTRACE \
   --device /dev/kfd --device /dev/dri --device /dev/mem \
-  -e HIP_VISIBLE_DEVICES=0,1 -e VLLM_ROCM_USE_AITER=0 \
-  ${FRANK_PRIMARY_GATHER_CHUNKS:+-e R4D_LRU_GATHER_CHUNKS="$FRANK_PRIMARY_GATHER_CHUNKS"} \
-  ${FRANK_PRIMARY_GATHER_LANES:+-e R4D_LRU_GATHER_LANES="$FRANK_PRIMARY_GATHER_LANES"} \
+  -e HIP_VISIBLE_DEVICES="$FRANK_PRIMARY_FT_GPU" \
   -v "$FRANK_PRIMARY_MODEL_DIR":/models:ro \
-  "$FRANK_PRIMARY_IMAGE" /models \
-  --served-model-name "$FRANK_PRIMARY_SERVED_NAME" --port 7770 \
-  --tensor-parallel-size 2 \
-  ${FRANK_PRIMARY_DCP:+--decode-context-parallel-size "$FRANK_PRIMARY_DCP"} \
-  --max-model-len "$FRANK_PRIMARY_MAXLEN" \
-  --max-num-seqs "$FRANK_PRIMARY_SEQS" \
-  ${FRANK_PRIMARY_MNBT:+--max-num-batched-tokens "$FRANK_PRIMARY_MNBT"} \
-  ${FRANK_PRIMARY_PRECISE_MEMORY:+--expert-precise-memory} \
-  --gpu-memory-utilization "$FRANK_PRIMARY_UTIL" \
-  --kv-cache-memory-bytes "$FRANK_PRIMARY_KV_BYTES" \
-  --kv-cache-dtype "$FRANK_PRIMARY_KV_DTYPE" \
-  --enable-prefix-caching \
-  --enable-auto-tool-choice --tool-call-parser "$FRANK_PRIMARY_TOOL_PARSER" \
-  --reasoning-parser "$FRANK_PRIMARY_REASONING_PARSER" \
-  --default-chat-template-kwargs "$FRANK_PRIMARY_CHAT_TEMPLATE_KWARGS" \
-  --trust-remote-code \
+  "$FRANK_PRIMARY_IMAGE" \
+  ft serve --model /models \
+    --served-model-name "$FRANK_PRIMARY_SERVED_NAME" \
+    --host 0.0.0.0 --port 7770 \
+    --gpu "$FRANK_PRIMARY_FT_GPU" \
+    --num-tokens "$FRANK_PRIMARY_FT_NUM_TOKENS" \
+    --max-running-requests "$FRANK_PRIMARY_FT_MAX_RUNNING" \
+    --max-output-tokens "$FRANK_PRIMARY_FT_MAX_OUTPUT_TOKENS" \
+    --moe-cache-rate "$FRANK_PRIMARY_FT_MOE_RATE" \
+    ${FRANK_PRIMARY_FT_EXTRA_ARGS:+$FRANK_PRIMARY_FT_EXTRA_ARGS} \
   > "$LOG" 2>&1
 
 # From here on, this script's lifetime IS the container's: wait returns the
 # container's exit code, so a crash is a nonzero ExecStart and the unit's
-# Restart=on-failure is real. (systemd stop goes through ExecStop in the
-# unit — podman wait alone cannot stop the container child.)
+# Restart=always is real. (systemd stop goes through ExecStop in the unit —
+# podman wait alone cannot stop the container child.)
 exec podman wait "$FRANK_PRIMARY_CONTAINER"

scripts/serve-frank-small.sh

@@ -1,18 +1,18 @@
 #!/usr/bin/env bash
-# Serve bartowski/MiMo-V2.6-Distill-Qwen-9B-GGUF (Q5_K_M) as frank-small on
+# Serve ornith-ai/Ornith-1.5-9B-GGUF (Q8_0) as frank-small on
 # Dankclaw's RTX 3090 via llama.cpp (master build, CUDA sm_86).
 #
 # 2026-10-01 night (Frank): swapped from SGLang + INT8-W8A16-AutoRound
 # safetensors to the llama.cpp GGUF path, because llama.cpp is the only
-# runtime on this host that can serve the MiMo qwen35 hybrid with
-# ASYMMETRIC KV: --cache-type-k f16 (unquantized keys) + --cache-type-v
-# q8_0 (quantized values), which is Frank's explicit KV contract for this
-# endpoint. SGLang's --kv-cache-dtype only moves K and V together.
+# runtime on this host that moves K and V independently — the asymmetric
+# f16-K + q8_0-V contract that endpoint was built for. 2026-10-03 (Frank):
+# the pair is now f16/f16 — two wide slots freed the VRAM, so the values are
+# unquantized too. The per-tensor flags stay explicit either way.
 #
 # Architecture: qwen3_5 hybrid — 32 layers, 8 full-attention + 24 Gated
 # DeltaNet, native 262144 window. KV lives only in the 8 attention layers;
 # the DeltaNet state is per-slot fixed size. Vision comes from the repo's
-# separate projector (the Q5_K_M GGUF carries no `v.*` tensors at all), so
+# separate projector (the Q6_K GGUF carries no `v.*` tensors at all), so
 # --mmproj is what gives this endpoint eyes; MMPROJ_FILE="" is the documented
 # text-only opt-out.
 #
@@ -33,22 +33,31 @@ fi
 
 # --- defaults (used when the env file does not set them) -------------------
 : "${LLAMA_BIN:=/home/dank/GitPackages/llama-master/build-cuda/bin/llama-server}"
-: "${MODEL_FILE:=/home/dank/models/MiMo-Q5KM/MiMo-V2.6-Distill-Qwen-9B-Q5_K_M.gguf}"
+: "${MODEL_FILE:=/home/dank/models/Ornith-Q8/Ornith-1.5-9B-Q8_0.gguf}"
 : "${ALIAS:=frank-small}"
 : "${BIND_ADDR:=0.0.0.0}"
 : "${LISTEN_PORT:=7771}"
-# ctx-size is the POOL llama.cpp divides across slots: 540672 / 3 gives
-# 180224 per slot — three independent conversations (the native 262144 no
-# longer fits three times on a 24 GiB card once the vision tower is loaded;
-# frank-small.env carries the two measurements that set this number).
-: "${CTX_SIZE:=540672}"
-: "${PARALLEL:=3}"
+# ctx-size is the POOL llama.cpp divides across slots: 401408 / 2 gives
+# 200,704 per slot — two independent conversations, each one wide enough to
+# hold a whole frank-primary-sized prompt in a single shot (the native 262,144
+# no longer fits twice on a 24 GiB card once the vision tower is loaded;
+# frank-small.env carries the three measurements that set this number).
+: "${CTX_SIZE:=401408}"
+: "${PARALLEL:=2}"
+# Compute-buffer geometry. Before 2026-10-03 this endpoint never passed either
+# flag, so it ran on llama.cpp's own defaults (2048/512). The pair is 4096/2048
+# since 2026-10-05: on this CUDA card the prefill curve climbs all the way to
+# ubatch 2048 (unlike the R9700 layer-split sibling, where 512 is the peak),
+# and the logical batch is VRAM- and throughput-neutral. frank-small.env
+# carries the sweep tables and the VRAM risk note.
+: "${BATCH_SIZE:=4096}"
+: "${UBATCH_SIZE:=2048}"
 : "${CACHE_TYPE_K:=f16}"
-: "${CACHE_TYPE_V:=q8_0}"
+: "${CACHE_TYPE_V:=f16}"
 : "${NGL:=99}"
 # Vision projector. No colon on purpose: an explicitly empty value means
 # "text-only, on purpose" and must NOT be replaced by the default path.
-: "${MMPROJ_FILE=/home/dank/models/MiMo-Q5KM/mmproj-MiMo-V2.6-Distill-Qwen-9B-f16.gguf}"
+: "${MMPROJ_FILE=/home/dank/models/Ornith-Q8/mmproj-Ornith-1.5-9B-BF16.gguf}"
 
 # Fail loudly rather than half-loading: a missing file is a download that
 # has not finished, and Restart=always would turn it into a crash loop that
@@ -66,9 +75,10 @@ fi
 # open after boot; make sure it exists before the server needs it.
 nvidia-modprobe -u -c=0 2>/dev/null || true
 
-# q8_0 on V REQUIRES flash attention (the quantized-value kernels only exist
-# on the FA path); an FA-less build silently falls back and the KV contract
-# in frank-small.env becomes a lie, so pin the flag pair here.
+# Flash attention stays pinned on: it is the fast path for f16 KV, and it is
+# the only path that could serve quantized V at all — so if CACHE_TYPE_V is
+# ever set back to q8_0 this flag must not be lost, or the build falls back
+# silently and the KV contract in frank-small.env becomes a lie.
 ARGS=(
     --model "$MODEL_FILE"
     --alias "$ALIAS"
@@ -76,6 +86,8 @@ ARGS=(
     --port "$LISTEN_PORT"
     --ctx-size "$CTX_SIZE"
     --parallel "$PARALLEL"
+    --batch-size "$BATCH_SIZE"
+    --ubatch-size "$UBATCH_SIZE"
     --n-gpu-layers "$NGL"
     --flash-attn on
     --cache-type-k "$CACHE_TYPE_K"

serverAppConfigs-docs/Agent/Agent Docs and Edits.md

@@ -0,0 +1,111 @@
+---
+type: reference
+title: Agent Docs and Edits
+description: Two self-inflicted damage classes — structurally broken markdown after parallel agent edits, and botched hashline PUT/CUT repairs — with the audit checks and recovery order for each.
+tags: [agents, docs, markdown, hashline, edit-recovery]
+timestamp: 2026-10-05T00:00:00-07:00
+---
+
+# Agent Docs and Edits
+
+## Doc-structure audit (after parallel agent edits)
+
+Use after one or more subagents have edited markdown docs (OKF/Obsidian
+vault with `---` frontmatter), before committing. Link checkers and
+completeness seams validate *references*, not *structure* — agents reliably
+produce structurally broken markdown that still links cleanly.
+
+Damage catalogue:
+
+1. Banner/prose prepended **above** the opening `---` → the frontmatter is
+   no longer frontmatter (title/tags/description become body text).
+2. A rewrite **appended** instead of merged → duplicate headings and a stray
+   second `---` mid-file.
+3. Literal model artifacts in the body: `<|channel>}<tool_call|>...`,
+   `+- `, `- - `.
+4. Wrapped bullets re-emitted with a `- ` prefix on every physical line.
+5. Line-range replacement that dropped a sentence tail → dangling fragment,
+   often mid-parenthesis.
+
+Checks (fast, run all):
+
+```bash
+head -3 <file>                          # must be `---` if the page has frontmatter
+grep -c '^---$' <file>                  # exactly 2 (opener + closer)
+grep -n '^## ' <file> | sort -t: -k2 | uniq -f1 -d   # duplicate sections
+grep -rnE '^\+- |^- - |<\|' <files>     # artifacts
+python3 scripts/check-vault-links.py <vault-dir>
+./tests/run.sh                          # docs completeness seam etc.
+```
+
+Repair order (matters):
+
+1. If the file was destroyed rather than edited: stash the agent version to
+   `.AGENTTRASH/<YYYYMMDD-topic>/`, then `git checkout HEAD -- <file>` and
+   re-apply only the intended changes by hand.
+2. If it merely duplicated: **salvage unique content first** — cut the
+   one-of-a-kind section, paste it into the copy you keep, THEN delete the
+   duplicate ranges. Never delete first.
+3. Re-add missing sentence tails (insert before the orphaned fragment),
+   keeping list continuation lines indented.
+4. Re-run all checks above.
+
+Gotchas: agents can write AFTER they appear finished (job settles, file
+changes) — re-verify structure after the last background job reports, not
+before. Editing an agent's already-damaged file with fresh line numbers is
+safe, but its snapshot tag changes on every edit — re-read before each
+subsequent edit. `.AGENTTRASH/` is gitignored: trash is local-only, so
+nothing moved there reaches the repo — that is intended (temporary hold, not
+archive).
+
+## Hashline edit recovery
+
+Failure mode: multi-op edits resolve every `PUT N.=M:` / `CUT N.=M:` index
+against the **original** file numbering, and the post-apply diff view
+**elides lines** (it shows an anchor window, not the whole file). Two traps
+follow:
+
+1. A second edit call built from a diff-view snapshot uses line numbers from
+   a *partial* view. Lines outside the window are invisible but still
+   counted.
+2. If the first apply was itself wrong, the file shifted. The diff view
+   tells you the damage, but not the true offsets.
+
+Result: the repair pastes fresh text **beside** stale text instead of over
+it.
+
+Procedure:
+
+1. **Never carry line numbers across calls.** Re-read the target range for a
+   fresh tag before every subsequent edit. A read is cheap; a duplicated
+   paragraph costs a commit and a revert.
+2. **One op per call when the previous apply was imperfect.** Fixed offsets
+   are only trustworthy when every preceding apply landed exactly as
+   specified.
+3. **Re-read after every apply**, and verify the whole affected region — the
+   diff view is a sample, not proof. Check the *boundaries*: the line before
+   the edit and the line after it. Damage shows up as a repeated sentence
+   straddling the seam.
+4. **Repair by CUT, not by re-PUT.** Once you know the exact stale lines
+   from a fresh read, delete them. Do not try to PUT a corrected superset
+   over a range whose extent you are guessing at.
+5. Verify with a content assertion, not a diff review: `grep -c` for the
+   duplicated sentence, plus a syntax/compile check on any touched script.
+
+Same rule for any imperative `edit` tool with an old/new match: the matched
+text must be the **entire** text being replaced. Matching a bare fragment
+inside a paragraph splices the replacement into the middle of the sentence
+and silently corrupts the file. Re-read the file afterwards to confirm.
+
+### Anchored in-page links
+
+Renaming a Markdown heading breaks `[[#Old Heading]]` anchors. When a
+section is retitled, grep for the old heading text and fix every anchor in
+the same pass — doc link-checkers frequently do not validate `#fragment`
+targets, so a green suite will not catch a dangling one.
+
+## Related
+
+- [[OMP Harness]] — the session context these repairs protect
+- [[Command Center Dotfiles]] — the repo these edits commit to
+- [[Testing Policy]] — the seams that catch broken vault structure

serverAppConfigs-docs/Agent/Command Center Dotfiles.md

@@ -0,0 +1,87 @@
+---
+type: reference
+title: Command Center Dotfiles
+description: The ~/.omp repo is itself the deployment — stow-equivalent packages, full-replica rsync transport, the pre-overwrite audit for non-blank hosts, deploy.sh mechanics, and the fish/sshpass transport gotchas.
+tags: [dotfiles, omp, deploy, rsync, blankpad]
+timestamp: 2026-10-05T00:00:00-07:00
+---
+
+# Command Center Dotfiles
+
+Repo `~/.omp` is itself the deployment: packages are home-relative trees
+under `dotfiles/<pkg>/`; `scripts/deploy.sh` symlinks live files into the
+repo checkout. `git pull` + `deploy.sh` is the whole propagation story.
+
+## Transport decision (agreed with Frank, executed 2026-09-09)
+
+- Ship via **full-replica rsync of `/home/dank/.omp/` including `.git`** to
+  `<user>@<host>:/home/<user>/.omp/`, NOT clone-from-Gitea — `main` can be
+  ahead of `origin/main`; push is a user-only trigger, so clone would ship
+  stale state.
+- Deploy runs on-target as the target user: `bash
+  /home/<user>/.omp/scripts/deploy.sh` (resolves the repo via `git -C
+  $(dirname $(realpath $0)) rev-parse --show-toplevel`, so `.git` must be
+  present at the target path — full-replica transport guarantees this).
+
+## Pre-overwrite audit (MANDATORY before first sync on a non-blank host)
+
+Diff every file that `deploy.sh --check` reports as DRIFT against the repo
+copy BEFORE letting quarantine-and-replace run. Host-specific divergence
+belongs in an override package, not in quarantine:
+
+1. Pull divergent live files from target
+   (`ssh ... 'cat <path>' > local-staged-copy`).
+2. Secret-audit staged copies (grep token/secret/password/key shapes)
+   before anything enters git.
+3. Compose `dotfiles/zz-<hostname>/` on the source machine from those live
+   copies; commit to repo first.
+4. Ship the package dir via rsync, then run deploy sync — override deploys
+   LAST and win over base packages.
+
+## deploy.sh mechanics (verified on blankpad)
+
+- `--check` = pure report (ADOPT/LINK/DRIFT lines), writes nothing. Run it
+  first, always.
+- Default mode = sync: adopt identicals silently, link new,
+  quarantine-and-replace divergents into `.quarantine/<host>/<ts>/`
+  (gitignored; never rm).
+- **Override-target bug (fixed cc2c652):** `zz-<host>` packages nest
+  base-package dirs internally (`zz-blankpad/niri/.config/...`); target
+  derivation must strip the nested package segment, else overrides deploy to
+  phantom `$HOME/niri/.config/...` while real targets keep pointing at
+  canonical copies (visible symptom: check shows 0 actions but
+  `readlink ~/.zshrc` points at base `zsh/.zshrc`).
+- systemd units auto-enable only when ExecStart binary resolves on that
+  host; dconf packages load via `dconf load /`, never symlinks.
+- `core.hooksPath=.githooks` is armed by deploy.sh per machine — set it once
+  post-transfer even before first sync.
+
+## ssh/sshpass transport gotchas (all bit live)
+
+- Resolve the target via mDNS FQDN (`avahi-resolve -4 -n <host>.local`); a
+  bare hostname may not resolve. Drop-not-refuse on port 22 = sshd down or
+  firewall default-drop, not auth failure — needs a local/console fix on
+  target.
+- `SSHPASS='...' sshpass -e ssh ...` inline per invocation; an env passed
+  via the tool's env parameter did not propagate to the sshpass child
+  process.
+- The remote login shell may be **fish**: never send bash syntax
+  (`for/do/done`, `$?`) bare through `ssh host 'cmd'`; wrap as `bash -c
+  "..."` or ship a script. See [[SSH and Remote Access]].
+- File shipping: `ssh 'cat > /remote/path' < localfile` (stdin redirect). A
+  here-doc attached to the ssh command is consumed by `cat` — the intended
+  script body becomes the file content, silently. Verify shipped bytes with
+  an md5sum compare on both ends; do NOT compare `/dev/stdin` after `cat`
+  drained it (false DIVERGED).
+
+## Verification of a completed deploy
+
+Re-run `deploy.sh --check` → expect exactly zero actions/quarantined/skipped
+= idempotent steady state. Spot-check `readlink` on override files points
+into `dotfiles/zz-<host>/`, not base packages.
+
+## Related
+
+- [[SSH and Remote Access]] — the transport mechanics this deploy rides
+- [[Noctalia and Niri]] — the desktop packages this repo ships
+- [[Deploy State Triage]] — the serverAppConfig deploy flow (different repo)

serverAppConfigs-docs/Agent/OMP Harness.md

@@ -0,0 +1,262 @@
+---
+type: reference
+title: OMP Harness
+description: The omp agent harness as configured on this fleet — compaction chain arithmetic, per-model sampler routing via compat.extraBody, and the static-preamble/skill-library audits that keep session context lean.
+tags: [omp, harness, compaction, models-yml, context, samplers]
+timestamp: 2026-10-05T00:00:00-07:00
+---
+
+# OMP Harness
+
+The harness reaches the fleet through `~/.omp/agent/models.yml` (model
+registry; single source of truth — there is no copy on the servers) and
+`~/.omp/agent/config.yml` (roles, compaction, global samplers). Nothing in
+this file is repo-managed; edits are harness-side only.
+
+## Compaction chain arithmetic (verify before ANY window change)
+
+When a fleet endpoint's context grows past its harness-declared window, OMP
+compacts by sending the conversation to a designated target model. Before
+changing any `contextWindow`/`maxTokens`, verify the chain arithmetically —
+a target whose window is smaller than the source's trigger point silently
+breaks compaction.
+
+Where the contract lives:
+
+- `~/.omp/agent/config.yml` → `compaction.remoteEndpoint` (target URL),
+  `methodOrder`, `thresholdPercent` (70), idle settings
+- `~/.omp/agent/models.yml` → per-model `contextWindow`, `maxTokens`,
+  `compactionModel` references
+- Implementation: omp plugin `dist/core/compaction/compaction.js`
+
+The arithmetic (the coupled contract every endpoint's maxTokens comment
+encodes):
+
+1. **Source trigger** = `thresholdPercent × source contextWindow`
+   (0.70 × 200,704 = 140,492 for frank-primary on the current shape)
+2. **Output budget** = capped at the target model's `maxTokens`
+   (32,768 for frank-small; 36,864 frank-primary; 57,344 frank-large — the
+   largest, because halogen's `HALOGEN_MAX_TOKENS_CAP` must stay ≥ it)
+3. **REQUIREMENT**: prompt (≤ trigger) + budget ≤ the target's per-slot
+   window. Worked pairs from the live registry: 140,492 + 36,864 = 177,356 ≤
+   200,704 (frank-primary); 140,490 + 57,344 = 197,834 ≤ 200,700
+   (frank-large); 140,492 + 32,768 = 173,260 ≤ 200,704 (frank-small).
+
+Procedure (recompute from the LIVE files every time — windows rotate with
+every swap; the numbers above are the 2026-10-05 state):
+
+1. Read the `config.yml` compaction block + `models.yml` entries for source
+   and target. Every endpoint's `contextWindow` comment carries its own
+   arithmetic — read them.
+2. Read the target checkpoint's `max_position_embeddings` (native window) —
+   the served window may be a concurrency contract, not the native limit
+   (`--parallel N` DIVIDES the pool; the per-slot number is the contract).
+3. If native ≥ trigger + budget, reslice the pinned pool:
+   `MAX_RUNNING × CONTEXT_LENGTH ≤ MAX_TOTAL_TOKENS` (the pin exists because
+   the runtime's autofit leaves too little for deferred loads + prefill
+   workspace; check `nvidia-smi` headroom — expect ~0.8 GB workspace margin).
+4. Changes: host env (CONTEXT_LENGTH-style env tunables) via git →
+   `./tests/run.sh` → deploy; harness `models.yml` directly (no repo copy).
+5. Verify: `/health` 200, boot log's pool line, then a long-prompt probe at
+   ≥ trigger height (the compaction-fit guarantee), then concurrent requests.
+
+Known reconciliation trap: comments in models.yml have described compaction
+as running "on the dankserver endpoint" while `remoteEndpoint` pointed at
+dankclaw — the comment/config disagreed. The comments now name the real
+target (`compaction.remoteEndpoint` = dankclaw :7771). **The config is
+truth; fix comments in the same change.** `maxTokens` never changes for a
+reslice — it caps output; `contextWindow` caps input. Boot-shape OOM
+laddering is a different problem ([[Llama Cpp Serving Sizing]]).
+
+## Per-model sampler config (`compat.extraBody`)
+
+`models.yml` has **no per-model sampler keys**. Sampling is global
+(`config.yml`: `temperature`, `topP`, `topK`, `minP`, `presencePenalty`,
+`repetitionPenalty`; default `-1` = **not sent**). The per-model hook is
+`compat.extraBody`:
+
+**Why it works:** `applyOpenAIExtraBody` runs **last** in the
+openai-completions builder — after `applyChatCompletionsCompatPolicy` — via
+`Object.assign(params, extraBody)`. So extraBody overrides global settings
+and compat-policy output.
+
+```yaml
+compat:
+  reasoningEffortMap:
+    max: xhigh
+  extraBody:
+    temperature: 1.0
+    top_p: 0.95
+    top_k: 20
+    min_p: 0.0
+    presence_penalty: 0.0
+    repetition_penalty: 1.0
+    preserve_thinking: true
+```
+
+Use **wire names**, not OMP camelCase. Two traps:
+
+1. **`preserve_thinking` is not user-settable as a compat key.**
+   `qwenPreserveThinking` is auto-derived in `buildOpenAICompat` from the
+   *auto-detected* `thinkingFormat` + local baseUrl, **before**
+   `applyCompatOverrides`. Setting `compat.thinkingFormat: qwen` in
+   models.yml does **not** enable it. Custom-named models (`frank-large`)
+   match no Qwen id pattern → use `extraBody: {preserve_thinking: true}`.
+2. **Never put `chat_template_kwargs` in `extraBody`** where a compat policy
+   also sets it — `Object.assign` is shallow and replaces the whole object,
+   silently dropping the `reasoning_effort` kwarg the compat policy set.
+   Top-level `preserve_thinking` is honoured by halogen. (frank-primary's
+   FreeToken entry currently carries a `chat_template_kwargs` extraBody with
+   a comment asking to verify-and-prune — the exact trap to check at that
+   endpoint's next swap.)
+
+**Gotcha: silent greedy decode.** halogen has **no server-side sampler
+defaults** — `temperature` absent or `0` = greedy. If OMP sends nothing, the
+endpoint decodes greedy while looking configured. This is why frank-large's
+entry carries the recipe on the wire and a comment saying "do not drop it".
+
+**Reasoning-effort folding.** Qwen3.8's template accepts only
+`{xhigh (default), medium, low}` for `reasoning_effort` and 400s on anything
+else ("Unexpected reasoning effort high"). Every endpoint's
+`reasoningEffortMap` folds all harness levels onto that set; a role whose
+level is unmapped fails on its FIRST request, and the failure names Jinja,
+not the model.
+
+### Verify the exact wire body (echo-capture)
+
+Don't trust the YAML — capture what OMP actually emits:
+
+1. Echo server on `127.0.0.1:18099` that dumps the JSON body to a file and
+   returns a canned `chat.completion` (include `reasoning_content` so
+   thinking paths exercise).
+2. Copy the **real** models.yml into a throwaway profile with only the
+   baseUrl swapped:
+   ```bash
+   sed 's|http://<host>:7770/v1|http://127.0.0.1:18099/v1|' \
+     ~/.omp/agent/models.yml > ~/.omp/profiles/wiretest/agent/models.yml
+   ```
+3. Run and inspect:
+   ```bash
+   omp --profile wiretest --model "<provider>/<model>" -p "hi" \
+     --no-tools --no-session --no-extensions --no-skills --no-rules
+   jq 'del(.messages, .tools)' last-body.json
+   ```
+   Confirm every intended field is present with the right value.
+
+**Verify a field is *honoured*, not just accepted** — unknown fields often
+return 200 anyway. Prove behavior with a **prompt-token delta**: replay an
+assistant turn carrying a long `reasoning_content`, then compare
+`usage.prompt_tokens` with the flag true vs false. A large delta means the
+field changed template rendering; zero delta means it was ignored.
+
+Cleanup: trash the probe profile and echo server (`profiles/.AGENTTRASH`),
+never `rm`.
+
+## Static-preamble audit ("the context feels bloated")
+
+What loads statically every session (measure each, don't guess):
+
+1. Repo rules file — `wc -c AGENTS.md` (baseline 2026-10-05: 72,179 B, ~18k
+   tokens — historically the largest block).
+2. Skill index — only names+descriptions load; bodies are on-demand. Sum
+   description bytes across all SKILL.md frontmatter:
+   ```bash
+   python3 - <<'EOF'
+   import glob, re, os
+   for root in [os.path.expanduser("~/.omp/agent/managed-skills"),
+                os.path.expanduser("~/.omp/agent/skills"), ".omp/skills"]:
+       n=tot=0
+       for sk in glob.glob(root+"/*/SKILL.md"):
+           fm=re.search(r'^---.*?---', open(sk, encoding="utf-8", errors="replace").read(), re.S)
+           d=re.search(r'description:\s*(.*(?:\n(?:\s{2,}.*)*)?)', fm.group(0) if fm else "")
+           n+=1; tot+=len(d.group(1)) if d else 0
+       print(root, n, tot)
+   EOF
+   ```
+   Baseline 2026-10-05: 171 managed skills / ~38 KB descriptions + 9 user
+   skills.
+3. Memory lessons — `find ~/.omp/agent/memories -name 'MEMORY.md' -o -name
+   'memory_summary.md' | xargs wc -c`. The injected block ≈ the largest
+   MEMORY.md.
+4. Harness/system prompt + tool schemas + CLI persona: fixed cost, ~20 KB;
+   not prunable.
+
+Known rot patterns:
+
+- Near-duplicate auto-captures of the same lesson (three "authentik ORM"
+  entries, the freetoken-build pair, the three job-runner skills) — dedupe
+  via `manage_skill delete`/`update`.
+- Corrupted captures containing stray CJK characters (`一团ly`, `等方法`)
+  mid-sentence — a known defect of the auto-capture path; delete the corrupt
+  copy, keep the clean one.
+- Stale host/service names in consolidated memory (pre-rename
+  `frank-default-large`) — the repo's AGENTS.md topology table is ground
+  truth.
+
+Fix strategy: AGENTS.md prose (measured runbook narratives) → move to
+`serverAppConfigs-docs/LLM/` runbooks, keep laws + tables in AGENTS.md (the
+repo's own docs-tree rule mandates this). Managed skills live in
+`~/.omp/agent/managed-skills` (safe to edit); user-authored skills in
+`~/.omp/agent/skills` are NEVER touched by tooling. Memory store edits:
+stage removed captures to trash first (`~/.AGENTTRASH/YYYYMMDD-topic`).
+
+Baseline to beat (2026-10-05): ~35–40k tokens of preamble — AGENTS.md ~18k,
+skill index ~10k, lessons ~3.5k, harness ~6k. Proposed reductions: −12k
+(AGENTS.md slimming), −3k (skill dedupe).
+
+## Managed-skill library hygiene
+
+Use when auditing, pruning, or consolidating `~/.omp/agent/managed-skills`
+(the auto-captured skill store; the full index is injected into EVERY
+session's context, so dead or duplicate entries are a permanent tax).
+
+The static-context budget (measured 2026-10-05):
+
+| Block | Size | Notes |
+|---|---|---|
+| `AGENTS.md` (repo) | 72,179 B ≈ 18k tok | biggest single item |
+| Skill index (~173 skills) | ~40 KB ≈ 10k tok | names+descriptions only |
+| Learned-lessons block | ~14 KB | can contain corrupted/duplicate captures |
+| Harness system prompt + tool schemas | ~23 KB | fixed |
+
+Audit procedure:
+
+1. Build an inventory: per skill, name, mtime, body word count, description,
+   first 12 headings (one pass, out to a file; read in pages).
+2. Group into: retired-runtime (host/service gone; its runbook already lives
+   in `serverAppConfigs-docs/LLM/`), verbatim-duplicate pairs/triples (same
+   lesson captured twice — often the SAME DAY), same-lesson families
+   (N skills = one procedure), keep.
+3. Read full bodies of every merge candidate before proposing a merge;
+   description alone proves overlap only for exact dupes.
+4. Present A/B/C/D verdicts for user approval. NEVER execute before the
+   discussion concludes — the user gates execution explicitly.
+5. On approval: stage everything to trash first
+   (`~/.AGENTTRASH/YYYYMMDD-<topic>/`), never rm. After merging: recreate
+   the surviving skill via `manage_skill` update with body = superset,
+   delete absorbed ones.
+
+Known rot classes (this library, 2026-10-05):
+
+- Same-day duplicate captures (freetoken-build-dankclaw vs
+  freetoken-dankclaw-build; the three job-runner skills).
+- CUDA-skill vs its self-declared HIP twin (pcie-link-capability-probe vs
+  pcie-link-probe-hip-rdna4) → merge.
+- Era ghosts: skills naming retired runtimes (freetoken, lucebox, r9v/
+  radiance, old halogen-flash-server image, ik_llama, GLM-5.3 experiment).
+- Skills contradicting user standing orders (live-bench-dashboard vs the
+  2026-09-29 delete-dashboards order) — flag for user, don't assume.
+- Memory store: corrupted captures with CJK noise from broken ssh commands;
+  stale host/service renames.
+
+Sizing verdicts that worked: bench family → 2 skills (harness vs
+discipline), not one mega-skill; keep profiler/kernel-time attribution
+standalone; fold go/no-go arithmetic skills into their probe skill
+(heterogeneous-accelerator-split-triage → PCIe probe).
+
+## Related
+
+- [[Agent Docs and Edits]] — editing vault docs and hashline targets safely
+- [[Command Center Dotfiles]] — the `~/.omp` repo deploy this config rides
+- [[Llama Cpp Serving Sizing]] — the engine-side sizing the chain must fit
+- [[LLM Bench Windows and Measurement]] — measuring the endpoints this config routes

serverAppConfigs-docs/Agent/index.md

@@ -0,0 +1,23 @@
+---
+type: index
+description: Pages about the agent harness itself — omp config, vault editing discipline, and the dotfiles repo that deploys the agent's own home.
+tags: [agent, index]
+timestamp: 2026-10-05T00:00:00-07:00
+---
+
+# Agent
+
+Pages about the agent's own tooling and context — how omp is configured, how
+docs get edited safely, and how the agent's home is deployed.
+
+- [[OMP Harness]] — compaction chain arithmetic, per-model samplers via
+  `compat.extraBody`, preamble and skill-library audits
+- [[Agent Docs and Edits]] — doc-structure audits after parallel edits;
+  hashline edit recovery
+- [[Command Center Dotfiles]] — the `~/.omp` repo-as-deployment, rsync
+  transport, override packages
+
+## Related
+
+- [[Testing Policy]] — the seams these pages' checks ride
+- [LLM](../LLM/index.md) — the fleet these agents operate

serverAppConfigs-docs/Architecture/Deploy State Triage.md

@@ -0,0 +1,151 @@
+---
+type: reference
+title: Deploy State Triage
+description: Reading and reconciling the deploy-state markers — stale marker vs live container, pending-service lockout, interrupted/raced deploys — without restarting healthy heavy endpoints.
+tags: [deployment, deploy-state, systemd, podman, triage]
+timestamp: 2026-10-05T00:00:00-07:00
+---
+
+# Deploy State Triage
+
+The deploy-state store is `~/.local/state/serverAppConfigs/deploy-state/<svc>`
+on each target (rootless, user `dank`, format exactly two lines
+`revision=<sha>` / `status=deployed|pending`, written atomically via tmp file
++ `mv`). It is written ONLY by a `deploy.sh` dispatch.
+
+## 1. Stale marker vs live container
+
+**Why it happens:** an out-of-band relaunch — reboot, `systemctl restart`, or
+a restart after the checkout ff-pulled to a newer revision — starts the
+container from the CURRENT `scripts/<svc>.env` but does NOT advance the
+marker. So `marker != HEAD` means "not dispatched via deploy.sh", NOT
+"behavior pending".
+
+**Why it matters:** a plain `deploy.sh` run re-selects and restarts a service
+whose marker lags. On llama.cpp/R9V endpoints that is a ~16–19 min reload of
+a 100+ GiB payload — and frank-primary (dankserver:7770) is also the
+harness's own model backend, so a blind deploy is a self-inflicted outage for
+possibly zero behavioral change.
+
+**Procedure (read-only first):**
+
+1. Committed state: `git log --oneline main..<branch>` empty ⇒ merged; the
+   operational delta is usually one `scripts/<svc>.env`.
+2. Probe the target (ssh as `dank`, passwordless sudo present):
+   - `cat ~/.local/state/serverAppConfigs/deploy-state/<svc>`
+   - `git -C ~/serverAppConfigs rev-parse HEAD`
+   - `systemctl is-active/is-enabled <svc>`;
+     `curl -s -o /dev/null -w '%{http_code}' http://127.0.0.1:<port>/health`
+   - the on-host env file already carries the new keys (it landed via the
+     ff-pull).
+3. **Decisive — read the LIVE container, not the marker.** Use `podman` as
+   `dank` (rootless store; bare `docker` is NOT the shim → "No such object";
+   `sudo podman` inspects an empty store):
+   - `podman inspect <ctr> --format '{{.State.StartedAt}}'` — booted AFTER
+     the pull ⇒ it read the new env
+   - `podman inspect <ctr> --format '{{.Args}}' | tr ' ' '\n' | grep -i <flag>`
+   - engine env: `podman exec <ctr> sh -c 'for f in /proc/[0-9]*/environ; do
+     tr "\0" "\n" <$f 2>/dev/null; done | grep -i <VAR>'`
+   - confirm the launcher maps each tunable to its flag
+     (`grep scripts/launch.sh`), e.g. `R9V_ENABLE_PREFIX_CACHING=1 →
+     --enable-prefix-caching`.
+
+**Decision:** live argv+env == target config AND `/health` 200 ⇒ behavior is
+live — **do NOT restart**; reconcile the marker (below). Only if the live
+container lacks the flag/env (or predates the pull) is a real
+`deploy.sh <target> --service <svc>` warranted.
+
+**Reconcile (no restart)** — mirror the deployer's atomic idiom:
+
+```bash
+S=$HOME/.local/state/serverAppConfigs/deploy-state
+H=$(git -C ~/serverAppConfigs rev-parse HEAD)
+t=$(mktemp "$S/.<svc>.XXXXXX"); printf 'revision=%s\nstatus=%s\n' "$H" deployed > "$t"; mv -f -- "$t" "$S/<svc>"
+```
+
+Then prove the gate: `./tests/run.sh` + `tests/e2e/run.sh` (needs
+`tests/e2e/.env.local`; expect the documented `11-admin` red if the
+FrankBook-Pro Mac mini 192.168.0.112 is ASLEEP — confirm with `ping -c2` +
+a `/dev/tcp` probe: 100 % loss + timeout = host off, not a regression).
+Log to `serverAppConfigs-docs/log.md`; commit + push (docs-only ⇒ deploy
+no-op).
+
+**Anti-patterns:** trusting a commit message ("X deployed") or the marker
+alone — both are bookkeeping; the container's argv/env is ground truth.
+Reading container state with `sudo podman` / bare `docker`. Restarting a
+healthy heavy endpoint to satisfy a stale marker.
+
+## 2. Pending lockout
+
+Symptom: `deploy.sh <target>` fails or silently stops with
+`service <name> is not deployed on <target>` and the `[3/3]` e2e gate never
+appears.
+
+1. **Get the real exit code first:** `./scripts/deploy.sh X 2>&1 | tail`
+   reports TAIL's 0 — use `set -o pipefail` or redirect to a file, then
+   `echo $?`.
+2. Inspect the state store (no sudo):
+   `ssh <target> 'cat ~/.local/state/serverAppConfigs/deploy-state/<name>'`.
+   A `status=pending` entry for a service absent from `service_is_allowed`
+   in `scripts/deploy-remote.sh` = retired-service lockout.
+3. Fixed behavior (since 02a7876, 2026-09-17): `add_pending_services` prunes
+   not-allowed pending entries to
+   `~/.AGENTTRASH/<YYYYMMDD>-retired-deploy-state/<service>` and the run
+   continues. Verify the WARN line names the service and the state file is
+   gone.
+4. On a host without the fix: move the stale entry to
+   `~/.AGENTTRASH/<date>-retired-deploy-state/` manually (never `rm`), re-run.
+5. Verify: deploy reaches `[3/3] Running Playwright e2e gate` and completes;
+   `deploy-state/` no longer holds the retired name.
+
+Related contracts: the superseded seam
+(`deployment_scoping.bats` "remote runner recognizes a pending service at
+the same revision") now asserts prune-and-continue (exit 0), not exit 2;
+`deployment_flow.bats` extracts `add_pending_services` via sed and
+HOME-overrides the trash path. **When retiring a service going forward:**
+delete or reconcile its deploy-state entry as part of the retirement commit —
+the prune is a safety net, not the intended cleanup path.
+
+## 3. Interrupted or raced deploys
+
+The deploy is TWO processes: `deploy.sh` (dev box) and `deploy-remote.sh`
+(runs on the target over ssh). **Killing the dev-side job does NOT kill the
+host-side one** — two `deploy-remote.sh` processes can end up managing the
+same unit, one deploying a revision you already superseded. Check:
+`pgrep -af "deploy-remote"` on the target, read the revision argument on
+each, kill stale ones by explicit PID.
+
+A killed deploy leaves behind:
+
+- the unit **stopped/failed** (`Restart=always` does NOT fire after an
+  explicit stop — start it yourself);
+- the state marker **pending** (pre-deploy or in-flight revision);
+- the e2e gate un-run (the deployer runs it last).
+
+Reconcile the marker ONLY after the service itself is verified healthy — a
+marker that runs ahead of reality is worse than one that lags (the planner
+uses it to decide what is pending). Then close the loop:
+
+1. Verify against MEASURED, config-specific numbers (KV capacity line, arena
+   slots, argv flags, `NRestarts`), not just health 200.
+2. Run the gate the dead deploy skipped (`tests/e2e/run.sh` from a checkout
+   that has the gitignored `tests/e2e/.env.local` — a fresh `git worktree`
+   does NOT have it; copy it in first).
+3. If the killed revision was restoring a known-good shape, confirm the
+   numbers match the last measured-good boot of that shape — equality is the
+   proof, not health alone.
+
+Boot failures must be read from the journal, not `podman logs`: serve scripts
+`podman rm -f` the container on every start (previous attempt lost), and
+`podman logs` on a container still *Stopping* returns the OLD configuration's
+numbers — which looks exactly like your change having no effect. Key on
+container identity (`podman ps -a --format '{{.CreatedAt}}'`) and read the
+cause from `journalctl --since "N min ago" | grep -a "<container-name>\["`.
+See [[vLLM Boot Forensics and Config Interrogation]] for the full capture
+procedure.
+
+## Related
+
+- [[Deployment Flow]] — the deploy pipeline itself
+- [[Quadlet Lifecycle]] — unit ownership and drift diagnosis
+- [[vLLM Boot Forensics and Config Interrogation]] — journal capture

serverAppConfigs-docs/Architecture/E2E Suite.md

@@ -20,11 +20,12 @@ user sees it; the `bats` seams assert config shape, not behavior.
 
 | Path | Role |
 |------|------|
-| `playwright.config.ts` | four projects (below), real-browser UA, trace on failure |
+| `playwright.config.ts` | five projects (below), real-browser UA, trace on failure |
 | `specs/00-auth.setup.ts` | setup project: two real authentik logins, saves storage states |
 | `specs/10-homepage.spec.ts` | user panel set + open links derived from `services.json` (registry-derived — no hardcoded service lists) |
 | `specs/11-admin.spec.ts` | `admin` project: admin grid + `/admin/` RBAC (403 for standard user, members-only 200) |
 | `specs/12-crawl.spec.ts` | every user open link reaches its real app through the guard chain |
+| `specs/19-frankbook.spec.ts` | `frankbook` project: walks the six admin links that terminate on FrankBook-Pro (192.168.0.112) — conditional, see below |
 | `specs/13-sso.spec.ts` | direct guarded-vhost visits resolve silently with ONLY the homepage session (single-login enforcement) |
 | `specs/14-guard.spec.ts` | `guest` project, no storage state: unauthenticated traffic lands on the IdP login |
 | `lib/login.ts` | `akLogin` helper (two-stage flow, shadow-DOM locators) |
@@ -38,7 +39,16 @@ user sees it; the `bats` seams assert config shape, not behavior.
   `.auth/qa-e2e{,-admin}.json` storage states.
 - **standard** — 10/12/13 with the user state.
 - **admin** — 11 with the admin state.
-- **guest** — 14 with an empty state.
+- **guest** — 14/15 with an empty state.
+- **frankbook** — 19 with the admin state. **Excluded from every default run**: the
+  *arr/qBittorrent links terminate on FrankBook-Pro, which sleeps when idle, and a
+  sleeping Mac mini must never fail an unrelated deploy. The project joins only
+  when `E2E_INCLUDE_FRANKBOOK=1` (set by `scripts/deploy.sh` when a frankbook
+  trigger path changed: `tests/e2e/specs/19-frankbook.spec.ts`, `frankbook-pro/**`,
+  `shared/homepage/services.json`, or the `dankclaw/Caddyfile` serving the arr
+  vhosts) or when an explicit `--project` argument names it. An `11-admin` edit
+  that touches the arr links must therefore be paired with a `19-frankbook` run
+  against an awake host before it is trusted live.
 
 Credentials live in `tests/e2e/.env.local` (git-ignored; placeholders in
 `.env.local.example`). The scaffold bats seam fails the commit if creds or
@@ -47,8 +57,9 @@ auth states are ever tracked.
 ## Run
 
 ```bash
-cd tests/e2e && npx playwright test            # all projects
-npx playwright test --project=standard 12-crawl.spec.ts   # one slice
+tests/e2e/run.sh                          # default set — frankbook project excluded
+E2E_INCLUDE_FRANKBOOK=1 tests/e2e/run.sh  # default set + the frankbook walk
+cd tests/e2e && npx playwright test       # ALL projects, frankbook included
 ```
 
 Requires LAN reachability of `www.franksdank.cloud` (Pi-hole DNS or public).

serverAppConfigs-docs/Architecture/Quadlet Lifecycle.md

@@ -0,0 +1,209 @@
+---
+type: reference
+title: Quadlet Lifecycle
+description: The full Quadlet discipline on this fleet — ownership auditing, compose-to-Quadlet cutover, drift/crash-loop diagnosis, the podman 5.x v1 generator quirks, log retrieval, and compose bring-up with ground truth.
+tags: [podman, quadlet, systemd, boot-persistence, compose, drift]
+timestamp: 2026-10-05T00:00:00-07:00
+---
+
+# Quadlet Lifecycle
+
+Fleet rule (AGENTS.md "Boot Persistence & Self-Healing"): every deployed,
+working container MUST have a self-healing, boot-persistent Quadlet unit —
+`.container` file in the repo (`<host>/<svc>/quadlet/`), installed by the
+deployer, `[Install] WantedBy=default.target`, `Restart=always`, linger on.
+A container that runs only from compose dies silently at reboot. This page
+is the operating manual for that lifecycle.
+
+## 1. Ownership audit — find the containers that will not survive a reboot
+
+```bash
+for c in $(podman ps --format '{{.Names}}'); do
+  printf '%-26s %s\n' "$c" "$(podman inspect $c --format '{{index .Config.Labels "PODMAN_SYSTEMD_UNIT"}}')"
+done
+```
+
+- **label present** → the unit owns it; persistence = `[Install]` + `loginctl
+  show-user <user> -p Linger` = yes.
+- **label absent, compose service exists** → the classic gap: dies at reboot.
+  Check the publish address while there — compose forms often bind the
+  wildcard (an unauthenticated UI/API on every interface).
+- **label present AND a compose service with the same name** → two owners; a
+  stray `podman compose up` can create a duplicate that fights the unit for
+  the name and the port. Remove the compose service, leave an explanatory
+  comment (repo convention).
+
+## 2. Cutover: compose service → Quadlet unit
+
+Checklist (precedents to copy:
+`dankclaw/{caddy,gitea,searxng,open-webui,seerr}/quadlet/`,
+`dankclaw/observability/quadlet/`):
+
+1. Unit file at `<host>/<svc>/quadlet/<name>.container`, mirroring compose
+   line for line: `Image=` (`localhost/` prefix for locally built images +
+   `Pull=never`), `ContainerName=` (Caddy and scrape configs address
+   containers by name), `Network=`, `User=0:0` when the image's own user
+   cannot write a rootless-owned bind mount, `Volume=` (`:ro` preserved),
+   `Exec=` for compose `command:`, `PublishPort=`, health keys,
+   `[Service] Restart=always` + `RestartSec=10`, `[Install]
+   WantedBy=default.target`. **Never `systemctl --user enable` the generated
+   unit** — `[Install]` + linger IS the persistence (`is-enabled` = `generated`
+   is healthy; explicit enable fails with "transient or generated").
+2. Exposure: publish on `127.0.0.1` unless something off-host genuinely needs
+   it (Caddy reaches containers by name over the shared network; the wildcard
+   publish is a guard-chain bypass).
+3. Handler in `scripts/deploy-remote.sh`: `install_quadlet <dir> <name>` →
+   `systemctl --user restart <name>.service` → `wait_for_health`; retire any
+   legacy hand-written unit into `~/.AGENTTRASH/` (a leftover
+   `~/.config/systemd/user/<svc>.service` that shells out to
+   `podman compose up` collides BY NAME with the generated unit);
+   `podman rm -f <container>` so the unit's container can take the name.
+4. Planner: map `<host>/<svc>/**` → service in `scripts/deploy-plan.py`
+   AND add a `service_is_allowed` entry; keep `SERVICE_ORDER` dependencies
+   ordered. Add a `test-changed` case above the generic
+   `*/quadlet/*.container` fallback.
+5. Retire the compose entry in the SAME change — two owners is how a stale
+   duplicate comes back.
+6. Seam test `tests/<svc>_quadlet.bats`: boot persistence, restart policy, no
+   wildcard publish, container name matches Caddy/scrape references, health
+   command, deployer installs+restarts+waits, never `rm`, never `enable`.
+   Migrate any older assertion that read the compose file for devices/flags.
+7. Deploy `--service <name>`; prove: `is-active`/`is-enabled` =
+   active/generated, container `healthy`, endpoint answers,
+   `list-dependencies default.target` shows the unit, off-host probes on
+   previously published ports now refuse, and the container no longer carries
+   `io.podman.compose.project`.
+
+**Traps that cost real time:**
+
+- `HealthCmd` = CMD-SHELL (podman runs `/bin/sh -c`); a distroless image has
+  no shell → the check fails forever. `--health-cmd` won't take exec form
+  either. Put `HEALTHCHECK … CMD ["binary","arg"]` in the image and build
+  with `--format docker` — the default OCI format has no HEALTHCHECK field
+  and drops it with only a warning.
+- Never point a unit's `EnvironmentFile` at the repo-root `.env` — it is a
+  gluetun env carrying `HOME=/root`; systemd sets HOME for the service and
+  the unit dies with `stat /root/.config: permission denied`. Use a
+  per-service env under `/home/dank/Containers/<svc>/env`, seeded by the
+  handler with a single-variable filter, validated so a missing secret fails
+  loudly.
+- `Network=host` exporters scraped by containerized Prometheus must bind the
+  host's LAN address (containers cannot reach host loopback) PLUS `127.0.0.1`
+  for the homepage probe — two `--web.listen-address` flags, never the
+  wildcard.
+- alloy 1.x refuses the `ui` config block; the unit must pass
+  `--server.http.listen-addr=0.0.0.0:12345` before its port can be published.
+- GPU services: `AddDevice=` + `GroupAdd=keep-groups` (user-unit spawns carry
+  no supplementary groups).
+- When the last service leaves a compose file, the compose-validity seam
+  fails (requires ≥1 service): keep the file as comments and relax the seam
+  deliberately, or delete the file and fix the planner's compose-diff path —
+  decide with the owner, the deploy tooling reads it.
+
+## 3. Drift / crash-loop diagnosis (5 minutes, in order)
+
+Symptom: crash-looping or perpetually unhealthy container whose DEFINITION
+looks right; or a service quietly down for days because a restart policy kept
+something running. Cause: the running container was created from an OLDER
+definition — a compose/unit edit does not touch an existing container, and
+the restart policy faithfully resurrects it forever.
+
+1. **Who started it:** `podman inspect <name> --format 'unit={{index
+   .Config.Labels "PODMAN_SYSTEMD_UNIT"}} restart={{.HostConfig.RestartPolicy.Name}}
+   created={{.Created}}'`; compare `created` with when the definition changed
+   (`git log -p --follow <config>`). Created-before-the-edit is the signature.
+2. **What it is ACTUALLY running:** `{{.Config.CreateCommand}}` is the truth.
+   `{{.HostConfig.Devices}}` / `{{.HostConfig.GroupAdd}}` print empty/0 even
+   for containers started with `--device`/`--group-add` — a false negative
+   that sends you hunting the wrong root cause.
+3. **What the process says:** `podman logs <name> | grep -iE 'error|cannot|Runtime'`
+   (a capability error like `No HIP GPUs are available` beside a missing
+   device flag confirms drift). Count the damage: `{{.RestartCount}}` — one
+   instance reached 9,146 restart attempts on stale config.
+
+Quickest unblock: `podman compose up -d --force-recreate <svc>`. Durable fix
+= the §2 cutover. **Verify the capability, not the status:** `podman exec
+<name> python3 -c "import torch; print(torch.cuda.is_available())"`;
+`ss -ltn | grep <port>` for the actual bind addresses. ComfyUI incident:
+2026-09-16, stale device-less container, 9,146 restarts. History repeats:
+2026-08-26 Caddy (no unit), 2026-09-11 five observability services.
+
+## 4. Podman 5.x quadlet v1 generator quirks (verified live 2026-08-28)
+
+- **Hard-fail keys** — the whole unit silently fails to generate
+  (`systemctl --user cat <unit>` shows "could not be found"): any
+  dash-prefixed `[Container]` option (`--shm-size=…` → `unsupported key …
+  in group 'Container'`); `Command=` is unsupported, use `Exec=`.
+- **HealthCmd space-splitting:** the v1 generator splits `HealthCmd` on
+  spaces into the healthcheck argv; a quoted probe
+  `/bin/sh -c "pg_isready …"` lands as argv token `"pg_isready` →
+  command-not-found → container reads **unhealthy while the DB is up**.
+  HealthCmd must contain only space-free tokens
+  (`HealthCmd=pg_isready -d authentik -U authentik`). If health reads
+  unhealthy but logs look fine, compare the baked
+  `{{.Config.Healthcheck.Test}}` array against the intended command before
+  believing the status.
+- **Probe a new key before shipping:** drop a throwaway `probe.container`,
+  `daemon-reload`, `journalctl --user --since "1 min ago" -o cat | grep -i
+  "unsupported key"`, `systemctl --user cat probe.service` (shows the
+  generated ExecStart verbatim), then remove + reload.
+
+## 5. Rootless Quadlet log retrieval
+
+Step 0 — establish ownership: `systemctl --user status <name>.service`
+(works ⇒ rootless user unit); `loginctl show-user <user> -p Linger`. If
+`--user` knows the unit, EVERY command runs as that user with NO sudo.
+
+| Need | Command |
+|---|---|
+| Current instance tail/follow | `podman logs --tail 200 / -f <name>` |
+| Across restarts/recreates | `journalctl --user -u <name>.service -n 200 --no-pager` |
+| Time-bounded / crash-only | `--since "14:30"` / `-p warning` |
+| Kernel-side evidence (OOM, SVM) | `sudo journalctl -k --since …` |
+
+Quadlet units commonly run `--replace --rm`: **`podman logs` is destroyed on
+recreation** — the journal is the only durable record. Traps:
+`sudo podman logs` → "no such container" (root's store does not contain
+rootless containers; same for `sudo podman ps`); a system-level
+`journalctl -u <name>` returns nothing for a user unit; during first-start
+weight downloads health reports unhealthy BY DESIGN (check the payload file
+is growing). Verbose vendor switches are a config change — through git and
+deploy.sh, never a live env poke.
+
+## 6. Compose bring-up with ground truth (rootless hosts)
+
+1. **Audit reality before pressing buttons:** `podman ps -a` runs per-user —
+   check every namespace (`sudo podman ps -a`) and native units
+   (`systemctl list-units`) before declaring a service absent. Compare
+   compose mount sources against the real filesystem — compose files name
+   phantom paths (e.g. `/mnt/media/*` vs the real LUKS array at
+   `/mnt/entropy`). Fix compose through git first, deploy second.
+2. **Pre-create bind-mount sources** (`sudo mkdir -p` + `chown 1000:1000`)
+   or creation fails `statfs …: no such file or directory`.
+3. **Staged bring-up:** infra first (`up -d mariadb redis`), poll
+   `{{.State.Health.Status}}`, then the rest. Fresh containers answer
+   000/refused for the first seconds — poll.
+4. **Never guess image tags:** ghcr.io answers 403 on the bearer-token
+   request for nonexistent/private repos ("manifest unknown"/token len 0 =
+   repo or tag does not exist, not auth). Query the registry API
+   (`ghcr.io/token?scope=repository:<ns>/<repo>:pull` →
+   `/v2/<ns>/<repo>/tags/list`).
+5. **Read the interface from inside the image** before guessing flags/envs:
+   `podman exec <ctr> cat /z/cpp.sh` etc. HTTP 500 with a styled error page
+   means RUNNING-but-misconfigured, not down. Copyparty ac ground truth:
+   entrypoint `/bin/ash /z/cpp.sh -c /z/initcfg`; accounts/volumes are ARGV
+   flags (`-a USER:PASS`, `-v SRC:DST:PERMS`), not envs — and with none
+   configured it serves cwd read/write for everyone (an unauthenticated
+   world-writable share).
+6. Remote-shell quoting: zsh eats bare `===`/`?` tokens; a zero-match from a
+   mangled pattern is not absence — re-verify the target exists.
+7. `user: "1000:1000"` SIGSEGV at boot = unmapped uid in the rootless userns
+   (host 1000 == container 0): drop the user line, run as container-root.
+
+## Related
+
+- [[Deployment Flow]] — the deployer that installs these units
+- [[Stale Signal Audit]] — the audit that catches drift's victims
+- [[Container Network Namespaces]] — reachability between containers
+- [[Dankclaw/ComfyUI]] — the 9,146-restart incident
+- [[Infrastructure/Observability]] — same class, twice

serverAppConfigs-docs/Architecture/Stale Signal Audit.md

@@ -0,0 +1,144 @@
+---
+type: reference
+title: Stale Signal Audit
+description: The audit battery for surfaces that rot without failing — unreachable probes, vacuous assertions, invalid checkers, gates that never run, and prose invalidated by a change. Four passes found real defects every time.
+tags: [testing, observability, audit, signals, gates]
+timestamp: 2026-10-05T00:00:00-07:00
+---
+
+# Stale Signal Audit
+
+Run after any change and before trusting a green gate. The productive question
+is **which signal here cannot possibly be correct?** Every pass on this repo
+(4 of 4) found real defects — including one in the rule file every future
+agent reads first.
+
+## The surfaces, and what each vantage point can reach
+
+| Surface | Producer's vantage point | Can reach |
+|---|---|---|
+| Homepage status cards | the `homepage-status` timer on **Dankclaw** | Dankclaw's loopback, Dankclaw container names, **published** LAN addresses |
+| Prometheus scrape jobs | the `prometheus` container on Dankclaw | container names on `dankclaw_default`, LAN addresses, that host's loopback only |
+| bats / e2e suites | the dev box | the repo, plus the live fleet over HTTPS/LAN |
+
+Anything else — another host's loopback, a container-internal port never
+published — is unreachable no matter how healthy the service is. The failure
+mode is always the same shape: a signal that **cannot** succeed by
+construction, which teaches the operator to ignore the surface it lives on.
+
+## 1. Permanently-red probes (reachability)
+
+For every monitor/probe/alert target ask: *can this checker physically reach
+that endpoint?* Seen classes: a card probing another host's container-internal
+port; a "probe" job scraping `/` (HTML → `unsupported Content-Type`); a
+`/metrics` that is disabled (Gitea 404).
+
+- A scrape cannot express "is this HTTP service answering?" — the right probe
+  is the **blackbox exporter** in probe mode (which also gives guarded vhosts
+  a real uptime signal, since Prometheus cannot solve an Anubis challenge).
+- **Fix in the direction of honesty, never of silence:** unreachable target →
+  mark the homepage card `wip: true` with the reason in `description` (the
+  checker skips `wip`; `wip` is the existing convention for "real service,
+  not probeable"), or repoint the job at something reachable. NEVER open a
+  database port on the LAN to make a badge green, and never delete a scrape
+  job — an absent scrape and a healthy one are indistinguishable.
+- Alert rules must not depend on always-red signals — scope them to services
+  that can actually be healthy (`LlmEndpointDown` is scoped to `frank-*` for
+  exactly that reason).
+- A target's failure STRING can prove a fix: a `/metrics` scrape flipping from
+  `connection refused` to `unsupported Content-Type "text/html"` means the
+  service came up and only the scrape is wrong. Read the error, not just
+  up/down.
+
+## 2. Vacuous assertions (highest severity — the gate is already dead)
+
+A seam that greps a file passes *by absence* once the file is retired:
+`grep` exits 2 on a missing path and `!` inverts it.
+
+```bash
+grep -rn 'grep.*"\$REPO_ROOT/[^"]*"' tests/*.bats | head -50
+[ -f "$path" ] || echo "MISSING: a seam may be grepping this"
+```
+
+Fix shape — one place asserts the invariant, each local site keeps a live
+check:
+
+```bats
+@test "no <host> compose file competes with the quadlet units" {
+    local cf="$REPO_ROOT/<host>/compose.yaml"
+    [ ! -f "$cf" ] || { echo "$cf is back: every container is unit-managed"; return 1; }
+}
+# at the per-service site: still fails if the entry returns
+[ ! -f "$cf" ] || ! grep -qE '^\s+<svc>:' "$cf"
+```
+
+**Prove teeth:** recreate the offending file
+(`printf 'services:\n  <svc>:\n    image: x\n' > <path>`), run the seams,
+confirm RED, then move the file to trash. A re-armed check you have never
+seen fail is still unverified.
+
+## 3. Gates that never run
+
+Playwright specs in `tests/e2e/` are matched by **number prefix** in
+`playwright.config.ts`; an unregistered spec silently never runs and the
+suite count stays the same. Count tests before/after;
+`tests/e2e_scaffold.bats` fails on an unregistered spec by design.
+
+## 4. Validate the checker before acting on it
+
+A checker's output is a hypothesis about another tool's semantics. Modelling
+Obsidian with naive text matching produced 45 "broken links" of which 3 were
+real — the rest were case-sensitivity (a `qBittorrent` link to
+`qbittorrent.md`), path-qualified resolution (a path-qualified
+`folder/Note` link resolves by path SUFFIX, not
+stem), and `%20` URL-encoding in markdown links. Fix the checker first,
+re-derive the report, only then edit documents. The result lives in
+`scripts/check-vault-links.py` with those three rules in its docstring,
+gated by the `no unresolved links in the docs vault` seam. Same class: a
+`find … | head -1` selector that reads the wrong file; a truncated listing
+that makes you "fix" a file that exists.
+
+## 5. Prose invalidated by your own change
+
+After retiring a file or adding a feature, grep for EVERY reference, not just
+code:
+
+```bash
+grep -rn "<retired-file>" --include="*" . | grep -v '^./.git/'
+grep -rn "no <feature>\|none —\|unconfigured" --include="*.md" --include="*.container" --include="*.yml" .
+```
+
+This catches the ones that bite: AGENTS.md said ComfyUI runs CPU-only long
+after it was restored with GPU access; a unit said "there are no alerting
+rules" hours after five were added; a docs page documented three scrape jobs
+as working when they could never report up. **Clean cutover includes prose** —
+docs, unit docstrings, and the repo's rule file (the most expensive place to
+be stale).
+
+## Order and verification
+
+1. Enumerate the surfaces (status board, metrics targets, alert rules, test
+   gates, docs/rule files, dashboard panels).
+2. For each, ask what the signal CAN express, not what it says.
+3. Validate any checker before editing the things it flags.
+4. Fix at the root; never weaken an assertion to make it pass.
+5. Gate the class so it cannot return (a seam), then prove the seam fails on
+   the defect.
+
+Verify: `./scripts/check-vault-links.py` (exit 1 on breaks); query each
+Prometheus panel expression against the API (0 series on a panel that should
+show data is a defect; 0 `ALERTS` when nothing fires is not); read the
+deployed `/srv/homepage/status.json` and triage every `state != up` entry
+against reality; query `/api/v1/targets?state=active` and print
+`health != "up"` with `lastError`; `./tests/run.sh` → `ALL TESTS PASSED`.
+Report per finding: the surface, why the signal could not have been correct,
+the fix, and the red/green evidence — flag any regression you introduced
+yourself in the same breath (one was found this way: a tightened bind killed
+a reverse-proxied vhost while every gate stayed green, because the spec
+covering that hostname did not exist).
+
+## Related
+
+- [[Testing Policy]] — the seam inventory this audit walks
+- [[Observability]] — the scrape/probe surfaces it audits
+- [[Quadlet Lifecycle]] — the ownership labels the probes read

serverAppConfigs-docs/Dankclaw/Hermes.md

@@ -81,6 +81,178 @@ The wiring lives in the live `/data/config.yaml` under `mcp_servers.chrome-devto
 The npx package cache lives in the container layer, so a recreation (not just a
 restart) pays one first-call npm download; subsequent calls are cached.
 
+## Config-home drift ("cannot connect" to models that answer curl)
+
+`podman exec <container> printenv HERMES_HOME` for each hermes container. If
+the webui reads the image default (`/opt/data`) while the gateway reads
+`/data`, the webui runs the image-default template config (`provider: auto` →
+openrouter.ai with no key = connect failure).
+
+The recreation trap is the same one the volumes table shows: image `VOLUME`
+points get a fresh anonymous volume on every container recreation
+(`podman run --replace --rm`) — edits inside such a mount are lost on
+restart/redeploy. Compare volume ids before/after a restart to confirm.
+
+Fix: pin `Environment=HERMES_HOME=/data` in the quadlet unit (shared state
+root), deploy service-scoped. Verify with the app's own code, not guesses:
+
+```python
+from api import config as C; C.resolve_model_provider("<model-id>")
+# -> (model_id, provider, base_url); bare ids map to the owning provider by
+#    scanning the providers: dict models allow-list; @provider:model forces one.
+hermes_cli.providers.resolve_user_provider(name, cfg["providers"])
+# -> ProviderDef with real base_url; accepts api/url/base_url keys.
+```
+
+Named providers (`providers:` dict) are a catalog + connection source;
+`custom:<slug>` connections resolve only via `custom_providers[]` entries. If
+neither holds the provider name, the request has no endpoint and fails to
+connect. Probe the endpoint with the **exact served model id** — the retired
+FLM/NPU endpoints routed only by exact registry tag and answered HTTP 200
+with `{"error":"Failed to load <tag> model!"}` for unknown tags; status-code
+checks miss this class entirely.
+
+## Live config edits
+
+Live file: `/home/dank/Containers/hermes/data/config.yaml` (dir `drwx------`
+uid 109999; all reads/edits need `sudo`). The repo file
+`dankclaw/hermes/config.yaml.example` is a sanitized backup and **can lag
+live — always diff the live file first; never treat the example as truth**.
+
+1. **Backup first:** `sudo cp -a …/config.yaml …/config.yaml.bak-YYYYMMDD`
+   (in-dir `.AGENTTRASH/` also exists for removals).
+2. **Apply** with surgical `sudo sed -i` or full-file write; YAML comments in
+   the live file are tool-generated — preserve them.
+3. **Sweep stale references** across the whole data dir:
+   `sudo grep -rln "<old-id>" /home/dank/Containers/hermes/data
+   --include="*.yaml" --include="*.json"`. Session files/MEMORY.md are
+   historical — leave them.
+4. **Derived caches:** `webui/models_cache.json` is keyed by a config
+   fingerprint (`_source_fingerprint.config_yaml` mtime/size); after edits it
+   is stale. Let it regenerate lazily or trash it — never hand-edit.
+5. **Restart the model-consuming units** (they parse config at startup):
+   `systemctl --user restart hermes-webui.service hermes.service`.
+   `hermes-dashboard.service` is unaffected by routing changes.
+6. **Verify behaviorally**: one tiny completion per endpoint with the exact
+   served id — POST `/v1/chat/completions` and read the body, never just the
+   status code (endpoint names rotate; see the LLM knowledge base).
+7. **Sync the repo backup** (`dankclaw/hermes/config.yaml.example`) to the
+   live shape, redacting secrets (`password_hash`, phone account stay
+   live-only), then `./tests/run.sh` before commit.
+
+`provider: custom` + `base_url` entries route by base_url and send the model
+id verbatim; named providers resolve via their `api:` URL. **Renaming a
+served alias anywhere (serve scripts, env ALIAS) MUST propagate to this
+config in the same change** or Hermes requests 404 on nonexistent models —
+this exact gap caused the 2026-09-05 outage.
+
+## Auxiliary routing against local backends
+
+When a Hermes feature that calls an auxiliary task (e.g. title generation)
+fails against a local endpoint, the usual cause is a **request shape the
+local runtime does not implement**, not a routing mistake.
+
+Diagnose with the real call path — run the actual function inside the
+container with the app's own constants (the probe cannot drift from
+production):
+
+```bash
+base64 -w0 /tmp/probe.py | ssh dank@192.168.0.143 \
+  'base64 -d > /tmp/probe.py && systemctl --user restart hermes.service && sleep 20 && \
+   podman cp /tmp/probe.py systemd-hermes:/tmp/probe.py && \
+   podman exec -i systemd-hermes /opt/hermes/.venv/bin/python /tmp/probe.py'
+```
+
+Copy the file AFTER any restart (`--replace --rm` discards the container
+layer). The CLI at `/opt/hermes/hermes` uses system python and dies
+`No module named 'yaml'`; use `/opt/hermes/.venv/bin/python`.
+
+```python
+import sys; sys.path.insert(0, "/opt/hermes")
+from agent.title_generator import _TITLE_PROMPT_TEMPLATE, _TITLE_RESPONSE_FORMAT, _clean_title, _extract_title_text
+from agent.auxiliary_client import call_llm
+resp = call_llm(task="title_generation", messages=[...], max_tokens=64,
+                temperature=0.3, extra_body={"response_format": _TITLE_RESPONSE_FORMAT})
+```
+
+What actually breaks, and the fix:
+
+- **`response_format` (json_object / json_schema) is a hard 400 on every
+  local backend here** — neither FreeToken nor halogen implements constrained
+  decoding. Config cannot override it: `call_llm` merges the task's
+  `extra_body` first and then `update()`s the caller's, so the hardcoded one
+  wins. Only temperature and max_tokens have reactive retry paths.
+- **Fix: move that one task to the Responses transport**, which never
+  forwards `response_format`:
+
+  ```yaml
+  auxiliary:
+    title_generation:
+      provider: dankserver-small        # names track the live config; diff first
+      model: frank-small
+      api_mode: codex_responses        # real per-task key
+      extra_body:
+        reasoning:
+          effort: none                 # mandatory, see below
+  ```
+
+- **Disable thinking for any cap-limited task.** The adapter omits reasoning
+  only when config says `reasoning.enabled: false`. The
+  `reasoning_effort: none` *shorthand* produces exactly that and leaves the
+  server default (thinking ON) — the 64-token title budget gets spent on the
+  think: empty content, `finish_reason: length`. The explicit
+  `extra_body.reasoning.effort: none` instead sends
+  `{"effort":"none","summary":"auto"}`, which the local `/v1/responses`
+  honors.
+- Both local runtimes also answer `/v1/responses` and `/v1/messages` — no
+  proxy needed for the alternate transport.
+
+Verify and land: re-run the probe with **no** `api_mode`/reasoning kwargs —
+it must pass from config alone (that is the production path); keep the live
+file and the repo example in step; `./tests/run.sh`, commit, push — a Hermes
+config change is docs+example only, no service deploy needed.
+
+Gotchas: `auxiliary.<task>.enabled: false` is checked BEFORE
+`apply_instant_title`, so disabling a broken title task also removes the
+derived title — it does not degrade gracefully the way a failure does. A
+local call can also fail at the model level: a 200 with an error-text body,
+or a chatty non-title answer. Always read the raw content, not the status.
+
+## Webui password rotation ("Hermex")
+
+Auth contract (verified live 2026-08-31): `POST /api/auth/login` with JSON
+`{"password":"…"}` → 200 + signed HttpOnly `hermes_session` cookie (24 h TTL)
+on match, 401 on miss. No username field. `/health` answers 200
+unauthenticated and `/` answers 302 — **never verify auth via those**; probe
+login with a deliberately wrong password (expect 401) and the real one
+(expect 200). Details on the webui surface: [[Hermes-WebUI]].
+
+1. Run the whole probe as a heredoc'd bash script
+   (`ssh -o BatchMode=yes dankclaw 'bash -s'`) — the host's zsh mangles
+   inline `${…}` even inside single-quoted ssh args; never interpolate curl
+   `-w %{http_code}` inline.
+2. Update BOTH env files with
+   `sed -i "s/^HERMES_WEBUI_PASSWORD=.*/HERMES_WEBUI_PASSWORD=<new>/"`:
+   - `/home/dank/serverAppConfigs/.env` ([Service] EnvironmentFile)
+   - `/home/dank/serverAppConfigs/dankclaw/.env` — **the effective one**: the
+     quadlet's `[Container] EnvironmentFile=` is what the container reads at
+     create; rotating only the root file leaves the running container serving
+     the old secret.
+3. `systemctl --user restart hermes-webui.service` (unit recreation re-reads
+   env; no deploy needed if the unit file itself didn't change).
+4. Verify wrong-pw → 401, right-pw → 200, from host loopback **and** a LAN
+   client at `http://192.168.0.143:8787` (webui publishes on the LAN
+   interface since commit `4443897`; off-network ingress is SSH tunnel only —
+   user `tunnel`, port **22**, key-only auth, forward remote end
+   localhost:8787).
+
+Gotchas: `sudo podman exec` fails exit 125 here — root's namespace does not
+see rootless containers; run `podman exec hermes-webui printenv
+HERMES_WEBUI_PASSWORD` unprivileged (the ssh user owns the env files, mode
+600). If rotating because a secret leaked into transcripts/logs, rotate in
+place via a script that fetches the live value from the running container
+(`printenv`) instead of transcribing it.
+
 ## See Also
 
 - [[Hermes-Dashboard]] — the LAN browser management UI

serverAppConfigs-docs/Dankclaw/OpenWebUI.md

@@ -64,6 +64,49 @@ The Ollama connection is disabled (`ollama.enable=false`).
 > repository. Model swaps, context sizing, and spec tuning happen there. See
 > the [LLM knowledge base](../LLM/index.md).
 
+## Container introspection (DB and backend python)
+
+- Working dir for imports: `/app/backend`; package name `open_webui`
+  (underscore), NOT `openwebui`. `os.environ["WEBUI_SECRET_KEY"]` MUST be set
+  BEFORE importing any `open_webui` module (import-time hard requirement);
+  the secret lives at `/app/backend/.webui_secret_key`.
+- **The `sqlite3` CLI is NOT in the container** — use Python's `sqlite3`
+  module (`podman exec open-webui python3 -c "import sqlite3;
+  conn = sqlite3.connect('/app/backend/data/webui.db'); …"`). For config
+  keys use parameterized LIKE queries (avoids shell quoting).
+- Backend python: `podman exec -w /app/backend open-webui python3 -c "…"` —
+  `Models.get_all_models()` is a coroutine (await it); model objects are
+  pydantic `ModelModel` instances (attribute access, not `.get()`), with NO
+  `status`/`enabled` attribute — use `is_active` from the DB.
+- **API auth for verification:** `create_token` lives in
+  `open_webui.utils.auth` (NOT `open_webui.utils.utils`);
+  `get_verified_user` requires an `id` claim (not just `sub`);
+  `SESSION_SECRET = WEBUI_SECRET_KEY`. Mint an admin token inside the
+  container:
+
+  ```
+  podman exec -i open-webui sh -c 'export WEBUI_SECRET_KEY=$(cat /app/backend/.webui_secret_key); \
+    python -c "from open_webui.utils.auth import create_token; \
+    print(create_token({\"id\":\"<UID>\",\"sub\":\"<UID>\",\"email\":\"…\",\"role\":\"admin\"}))"'
+  ```
+
+  (admin UID from the `user` table).
+
+## Verification battery
+
+1. `GET /api/v1/models/list?limit=100` → returns EXACTLY the intended
+   models. Response key is `items`, not `data`.
+2. `POST /openai/chat/completions` per model with a trivial prompt ("Reply
+   with exactly one word: pong") → each returns content `pong`. Reasoning
+   models need `max_tokens>=100` — the thinking block eats the budget.
+3. A 500 "Server Connection Error" → `podman logs open-webui` for
+   `ClientConnectorError`: a network-reachability problem (host LAN IP vs
+   `host.containers.internal`), not a model problem.
+
+**Durability:** the model table + connections are DB-only (not git-managed);
+a fresh data volume loses them. For reproducibility, add an idempotent
+first-start seed script or document the setup in the service doc.
+
 ## Web search
 
 Chat web search is served by [[SearXNG]] on this host, wired in the quadlet
@@ -147,6 +190,23 @@ The quadlet carries a built-in healthcheck (30s interval, 3 retries, 300s start
 
 `podman ps` shows the container as `unhealthy` if either fails. Probe manually at `http://dankclaw:3005/health`.
 
+## Image generation (ComfyUI)
+
+Image generation rides the local [[ComfyUI]] integration, not the chat
+picker — ComfyUI's models never appear in the chat model picker (that is
+chat LLMs only), so ComfyUI needs its own image-generation connection
+configured in OpenWebUI. ComfyUI itself runs GPU-enabled on Dankclaw and
+serves `:8188`.
+
+Endpoint: `POST /api/images/generations` (image `0.11.x`: upstream moved the
+route to `POST /api/v1/images/generations`; the `comfyui` engine is still
+upstream) with `Authorization: Bearer <token>` and body
+`{"model":"imagegen","prompt":"…","n":1,"size":"1024x1024"}`; the response
+carries base64 image data or a URL.
+
+Failure modes: connection refused ⇒ OpenWebUI or ComfyUI is down; timeouts
+on large sizes ⇒ check ComfyUI resources; 401 ⇒ the token is bad.
+
 ## Retired instances
 
 - The compose `open-webui` service (port 3000) was never run: 3000 is Gitea's HTTP port, so it could not bind.

serverAppConfigs-docs/Dankclaw/Swap Configuration.md

@@ -117,6 +117,18 @@ frank-large `health=200` throughout.
   the model is loaded. Trust the engine's `host memory left for everything
   else` line, not `free`.
 
+- **Filesystem prerequisite (other hosts).** ext4 and xfs take swapfiles
+  directly. btrfs needs `chattr +C` (NOCOW), no compression, and no
+  snapshots in the swapfile path — check `df -T` before planning, or
+  `mkswap` will reject the file.
+- **Renaming an *active* swapfile works (inode-based) but leaves a stale
+  path in `/proc/swaps`** — swap it off first, then rename, then `swapon`
+  (which is exactly what the step-3 consolidation does). Migration is
+  observable while it runs: in the 2026-09-12 run `swapoff` of 4 GiB took
+  65 s and used-RAM rose only ~3 GiB (pages touched during migration), the
+  rest relocating on disk. If used-RAM climbs toward the whole swapped
+  amount, let it finish rather than aborting.
+
 ## Trash
 
 The previous 8 GiB `/swap.img` (dated 2026-07-21) is staged, not deleted, at

serverAppConfigs-docs/Desktop/Noctalia and Niri.md

@@ -0,0 +1,303 @@
+---
+type: reference
+title: Noctalia and Niri
+description: The noctalia v5 desktop shell on niri (workstation + blankpad) — config layers, remote IPC over ssh, custom-palette recolor, offline template rendering, and output rotation/pinning.
+tags: [desktop, noctalia, niri, blankpad, workstation, ghostty]
+timestamp: 2026-10-05T00:00:00-07:00
+---
+
+# Noctalia and Niri
+
+Noctalia (`/usr/bin/noctalia`) is the desktop shell on niri, running on both
+the workstation and blankpad. Since the 2026-09-10 unification both run the
+**bare `noctalia` v5 binary** (niri `spawn-at-startup "noctalia"`); the old
+Quickshell-hosted v4 architecture (`qs -c noctalia-shell`, JSON config) is
+retired on both hosts.
+
+## Config layers
+
+- Hand-edit declarative layer: `~/.config/noctalia/<name>.toml` (e.g.
+  `config.toml`). Main config: `~/.config/noctalia/config.toml`; the niri
+  config includes `noctalia.kdl` for display settings.
+- Runtime state file `~/.local/state/noctalia/settings.toml` loads LAST and
+  **overrides declarative values for the same keys — never hand-edit it**;
+  UI/IPC writes land there.
+- Main reference: `noctalia-reference` facts — IPC `noctalia msg <command>`
+  replaced the `qs -c noctalia-shell ipc call` wrapper; app templates live at
+  `/usr/share/noctalia/assets/templates/` (gtk, ghostty, alacritty, hyprland,
+  sway, wezterm, helix, btop, starship, qt, kde, cava, scroll, mango,
+  umbriel, foot, emacs, kitty); palette generation `noctalia theme [image]`.
+
+## Remote IPC over ssh (the D-Bus trap)
+
+`noctalia msg <command>` resolves the running instance via the D-Bus session
+bus. sshd-spawned non-login shells lack `DBUS_SESSION_BUS_ADDRESS`, so plain
+ssh exec prints `error: noctalia is not running` even when the shell IS
+running — do not conclude "shell dead" from this error. Working remote
+procedure (runs in the systemd user-manager context where the bus address
+exists):
+
+```
+systemd-run --user --pipe --wait noctalia msg <command> [args]
+```
+
+Verified: `msg status` returns live JSON state; unknown panel ids return an
+error listing all valid panel ids (clipboard, control-center, launcher,
+polkit, session, setup-wizard, test, tray-drawer, wallpaper).
+
+Verified v5 msg verb grammar (keybinds-safe): `panel-toggle <id>`,
+`session lock|suspend|lock-and-suspend|logout|reboot|shutdown`,
+`volume-up [step]`, `volume-down`, `volume-mute`, `mic-mute`,
+`media next|previous|playPause` (graceful no-op, rc 0, with no active
+player), `brightness-up [target] [step]`, `brightness-down`,
+`settings-open [context]`, `settings-toggle [context]`, `config-reload`,
+`wallpaper-get [connector]`, `wallpaper-set <path>`, `wallpaper-next`,
+`color-scheme-get|set`, `theme-mode-get`, `templates-apply`,
+`screenshot-fullscreen`.
+
+**Wallpaper directory semantics:** empty/unset `[wallpaper] directory`
+resolves to XDG Pictures via user-dirs.dirs; an explicit directory key
+overrides. Fleet default pattern: point `directory` at the repo-tracked
+wallpapers dir so git-tracked wallpapers serve the shell without copying
+files. After writing config.toml over ssh the file watcher hot-reloads
+automatically; `config-reload` via systemd-run forces it immediately. Note:
+the *current* wallpaper stays the built-in default until the next rotation
+or an explicit `wallpaper-next`; `wallpaper-get` then reporting a path
+inside the configured directory is the confirmation the config took effect.
+
+**Lock chain (unified):** swayidle-lock.service ExecStart on both hosts:
+`/usr/bin/swayidle -w timeout 300 '/usr/bin/noctalia msg session lock'
+before-sleep '…' after-resume '…'`. `msg` resolves via the session bus, so
+no WAYLAND_DISPLAY pinning is needed in unit context.
+
+## Custom-palette recolor (without restarting the session)
+
+Why custom, never the community cache:
+`~/.local/state/noctalia/community-palettes/*.json` is a **sync cache** —
+`.catalog/palettes.json` carries an `md5` per palette and noctalia re-fetches
+from `https://api.noctalia.dev/palettes`, so edits there get reverted.
+Custom palettes live at `~/.config/noctalia/palettes/<Name>.json`
+(`NOCTALIA_CONFIG_HOME` then `XDG_CONFIG_HOME` honored) and are never
+synced. Missing/invalid file → silent fallback to the builtin palette.
+
+Palette JSON shape = the community file's shape: `dark`/`light` maps of
+`mPrimary`, `mOnPrimary`, `mSecondary`, `mTertiary`, `mError`, `mSurface`,
+`mSurfaceVariant`, `mOutline`, `mShadow`, `mHover` plus a `terminal` block
+(`foreground`, `background`, `normal.{black..white}`, `bright.{…}`,
+`cursor`, `cursorText`, `selectionFg`, `selectionBg`). NOTE: `noctalia theme
+-o` emits M3 snake_case tokens (`on_primary_container`, …) — that is the
+*generator* format, not this one. Copy an existing community JSON and mutate
+with `jq`.
+
+Procedure:
+
+1. Read the live scheme first — never assume: `noctalia msg color-scheme-get`
+   (e.g. `community Gruber Darker`); `noctalia msg theme-mode-get`.
+2. Back up settings: `cp -a ~/.local/state/noctalia/settings.toml
+   /tmp/noctalia-settings-backup-$(date +%s).toml`.
+3. Build the palette from the current one (jq keeps structure):
+
+   ```sh
+   mkdir -p ~/.config/noctalia/palettes
+   jq '.dark.mPrimary = "#f43841"
+       | .dark.terminal.normal.yellow = "#f43841"
+       | .dark.terminal.bright.yellow = "#ff4f58"
+       | .dark.terminal.foreground   = "#f43841"' \
+      "$HOME/.local/state/noctalia/community-palettes/Gruber%20Darker.json" \
+      > ~/.config/noctalia/palettes/"Gruber Red.json"
+   ```
+
+   Prefer a hex already present in the palette (its own `mError` red) so the
+   theme stays internally consistent. Set BOTH `normal` and `bright`
+   variants, and both `dark` and `light` if the user toggles modes.
+4. Switch (persists `theme.source = custom` + `theme.custom_palette`):
+   `noctalia msg color-scheme-set custom "Gruber Red"` → `ok`; confirm with
+   `color-scheme-get`.
+5. Re-render every configured template: `noctalia msg templates-apply`.
+   Regenerates `~/.config/ghostty/themes/noctalia`,
+   `~/.config/gtk-{3,4}.0/noctalia.css`, kcolorscheme, foot, wezterm,
+   starship, etc. Symlinked-into-dotfiles paths show up as modified files
+   worth committing.
+6. Reload surfaces: compositor — `niri msg action load-config-file`
+   (`reload-config` is NOT a valid niri subcommand); ghostty — see the trap
+   below.
+
+Verification: `ghostty +show-config | grep -E '^(foreground|palette)'`
+(proves the theme file resolves — note it prints only the delta from
+defaults; cross-check with `ghostty +show-config --default` before
+concluding a key is unset); `noctalia config validate` → `✓ Config is
+valid`; visual via `noctalia msg screenshot-fullscreen` (saves to
+`~/Pictures/screenshot_<ts>.png` and ALSO overwrites the clipboard with the
+PNG — warn the user); `grep -c <old-hex> ~/.config/noctalia/palettes/*.json`
+→ 0 for no leftovers.
+
+Track it in the dotfiles repo (`~/.omp`, the command-center —
+`scripts/deploy.sh` is a stow-equivalent over `dotfiles/<pkg>/` trees):
+mirror the shape for the new palette so `git pull` + `deploy.sh` is the
+whole propagation story — move the JSON into
+`~/.omp/dotfiles/noctalia/.config/noctalia/palettes/` and symlink it back
+into `~/.config/noctalia/palettes/`; re-set the custom palette (re-reads
+through the symlink). Do the link by hand for one file instead of running
+`deploy.sh` — sync mode quarantines and replaces every divergent live file.
+
+Traps:
+
+- **NEVER `kill -USR1` ghostty to reload.** It does not reload config on
+  this fleet — it kills/respawns the terminal (took out Frank's window
+  2026-09-13). Correct: the default keybind `ctrl+shift+,` = `reload_config`
+  (per `ghostty +show-config --default`). Existing windows keep old colors
+  until reloaded; new windows inherit immediately.
+- Palette hex values are NOT a proxy for what the user sees: in
+  `Gruber Darker` the *accent* (`mPrimary`) is yellow `#ffdd33`, so "change
+  the yellow" means the accent too, not just terminal index 3.
+- Saturating terminal text makes ANSI 1/9 indistinguishable from default
+  text (grep highlights flatten) — flag it and offer a lighter red.
+- Contrast sanity: `#f43841` on `#181818` ≈ 4.6:1 (dark-on-color beats
+  white-on-color here, ~3.9:1 — keep `mOnPrimary` dark).
+- Revert = `noctalia msg color-scheme-set community "<original name>"`; the
+  community cache is untouched by the whole flow.
+- Generated theme files symlinked into the dotfiles repo (`gtk-3.0`/`gtk-4.0`
+  `noctalia.css`, ghostty theme) show up as tracked modifications after
+  `templates-apply`; commit them together with the new palette JSON so every
+  machine gets the same recolor.
+
+## Offline template render (when templates-apply never ran)
+
+Failure signature: niri `failed to read included config from
+~/.config/niri/noctalia.kdl: No such file or directory` — the desktop runs
+but with DEFAULT config (niri 26.04 falls back, does not exit): no custom
+keybinds/rules, no noctalia autostart, default output placement. ghostty:
+`theme "noctalia" not found, tried ~/.config/ghostty/themes/noctalia`.
+alacritty: import of `~/.config/alacritty/themes/noctalia.toml` missing.
+
+Root cause pattern: upstream dotfiles commits theme switches (niri include,
+ghostty `theme = noctalia`, alacritty import) that depend on noctalia
+TEMPLATE-GENERATED files. Those files are per-host outputs (rendered from
+the local palette) and are intentionally NOT in the repo; on any host where
+`templates-apply` never ran, they don't exist.
+
+Fix (offline, works while a session runs) — built-in templates live at
+`/usr/share/noctalia/assets/templates/` (declared in `builtin.toml`); render
+each needed one with the live palette:
+
+```bash
+# niri include (plain host-local file; do NOT symlink into repo — regenerated per host)
+noctalia theme --theme-json ~/.config/noctalia/colors.json \
+  --render "/usr/share/noctalia/assets/templates/niri/niri.kdl:$HOME/.config/niri/noctalia.kdl"
+mkdir -p ~/.config/ghostty/themes && \
+noctalia theme --theme-json ~/.config/noctalia/colors.json \
+  --render "/usr/share/noctalia/assets/templates/ghostty/ghostty:$HOME/.config/ghostty/themes/noctalia"
+mkdir -p ~/.config/alacritty/themes && \
+noctalia theme --theme-json ~/.config/noctalia/colors.json \
+  --render "/usr/share/noctalia/assets/templates/alacritty/alacritty.toml:$HOME/.config/alacritty/themes/noctalia.toml"
+```
+
+All template tokens (primary, surface, error, primary_container, …) resolve
+from the flat 16-key colors.json; container tokens are derived, never blank.
+
+Verify: `niri validate` → rc 0; `ghostty +validate-config` → rc 0; next
+login journal shows `loaded config from config.kdl` with NO parse WARN,
+custom output placement (`niri msg outputs` vs display.kdl), noctalia
+autostarted (`pgrep noctalia`).
+
+Traps: **`noctalia msg templates-apply` on 5.0.1 returns "ok" but applies
+NOTHING** (this build's opt-in config schema does not act on it) — do not
+trust its exit code, use the explicit `--render` calls above. Niri config
+errors at boot are NOT fatal — the compositor keeps running on defaults,
+which reads as "some settings silently stopped working", not a crash.
+Regenerate after any palette change; the rendered files are static
+snapshots.
+
+## Niri output rotation and pinning
+
+Verified on blankpad, niri 26.04 (two PM1601 externals + eDP-1 laptop panel).
+
+**Ground truth first:** `niri msg outputs` (connector names, current mode,
+scale, transform, logical pos/size) and `niri --version`. Output **names are
+connector names** (`eDP-1`, `DP-2`, `HDMI-A-1`), not EDID model strings —
+the EDID strings printed before them are decoration. On blankpad the live
+config splits via `include "./cfg/*.kdl"` from `~/.config/niri/config.kdl`,
+and `cfg/display.kdl` is a **symlink into the dotfiles repo** — edit the
+repo path (that is what gets committed); check with `ls -ld`.
+
+**Transform direction is counter-clockwise** (`niri msg output <NAME>
+transform --help`): `90` = content rotated 90° CCW ⇒ the panel's **top
+bezel sits on the LEFT**; `270` = top bezel RIGHT; `180` = upside down;
+`flipped-90`/`flipped-270` = mirrored variants. `90` vs `270` is a
+physical-stand fact no tool can read (portable monitors expose no
+orientation in EDID/MCCS) — pick one, apply, and hand the user the one-liner
+to flip; don't stall on the question.
+
+Two traps when editing the config:
+
+1. **An `output "X" { … }` block resets every property you don't restate to
+   its default.** Pinning position on `eDP-1` without repeating `scale 1.25`
+   silently drops the laptop panel to scale 1.0. Restate `scale` (and
+   `mode`) in every block you touch.
+2. **Niri re-runs auto-layout for outputs with no pinned position.** Pinning
+   only the externals shoved `eDP-1` from `0,0` to `3936,0`. Pin every
+   output's position to keep the left-to-right order. Positions are logical
+   pixels at each output's own scale; x-offsets chain (eDP-1 logical width
+   1536 → DP-2 at 1536 → +1200 portrait width → HDMI-A-1 at 2736):
+
+```kdl
+output "DP-2"     { scale 1 transform "90" position x=1536 y=0 }
+output "HDMI-A-1" { scale 1 transform "90" position x=2736 y=0 }
+output "eDP-1"    { scale 1.25 position x=0 y=0 }   // MUST restate scale
+```
+
+Apply and verify:
+
+```bash
+niri validate -c ~/.config/niri/config.kdl      # exit 0
+niri msg action load-config-file                # live reload; NOT reload-config
+sleep 1; niri msg outputs | grep -E '^Output|Transform|Logical|Scale'
+```
+
+Success looks like `Logical size: 1200x1920` (swapped) plus
+`Transform: 90° counter-clockwise` on each rotated panel, positions matching
+the intended chain. One-off rotation without touching config (temporary):
+`niri msg output DP-2 transform 90`. Commit via the dotfiles repo
+(`~/.omp`, `main`; push to the local Gitea permitted): `git pull --ff-only`
+→ add the display.kdl → commit → push. Pitfalls: `reload-config` is not a
+niri subcommand; never SIGUSR1 a terminal to "reload" (niri config reloads
+do not touch ghostty themes); no man page for niri on this host — `niri
+<sub> --help` and `/usr/share/doc/niri/default-config.kdl` are the docs.
+
+## GPU RGB over I2C (EVGA iCX3)
+
+Turn RGB on/off (or read it back) on an EVGA FTW3 / iCX3-controller GPU
+(the 3090 on dankclaw) using the GPU's own nvidia i2c bus. No display
+engine, no X, no daemon — the mode is EEPROM-saved and survives reboot.
+
+- **Find the controller:** the nvidia i2c driver exposes the card's lighting
+  ports as `/dev/i2c-N`; on the 3090 FTW3 that is ports 1,5,6,7,8 →
+  i2c-1..i2c-5, and only i2c-1 carries a live slave (NVIDIA port 1). Probe
+  each bus. The iCX3 MCU answers at address **0x2D**. `/dev/i2c-*` are
+  `root:i2c 660` — run with `sudo` (dank has passwordless sudo).
+- **Protocol (raw I2C_RDWR, NOT the SMBus ioctls)** — write the register
+  byte first, then the payload:
+  - enable `0xB2` write `04 C6 EB EA 15` (required once before mode writes)
+  - mode `0xC0` write `09 m0 m1 m2 m3 .. .. .. .. SYNC`; m0..m3 are the four
+    zone modes, `MODE_OFF = 0x00`; OFF = `09 00 00 00 00 FF FF FF FF 00`
+  - save `0x90` write `04 9E EB 00 90` (commit to EEPROM — the
+    reboot-persistence step)
+  - fw `0xB1` block read (firmware version)
+  - Read the mode back: write reg `0xC0`, then read 16 bytes; bytes 1..4 are
+    the zone modes (0x00 = OFF), bytes 6..9 the per-zone LED counts.
+- **Verify persistence:** after a reboot, read the mode — all four zone
+  modes `0x00` means the EEPROM save held (confirmed 2026-09-19 post-reboot).
+- **Tool:** a ~90-line I2C_RDWR reader/writer lived at `/tmp/evga_raw.c`
+  (`evga_raw <bus> mode|off|save|enable|fw`) — `/tmp` is wiped by reboot, so
+  recreate it if needed; OpenRGB's `EVGAGPUv3Controller.cpp` carries the
+  same protocol plus the full colour API.
+- Gotchas: raw I2C_RDWR only (the iCX3 answers the raw form, not the SMBus
+  ioctls); the `enable` write must precede the first mode write in a
+  session; without the `save` write the mode is RAM-only and resets on power
+  cycle.
+
+## Related
+
+- [[Command Center Dotfiles]] — the stow-equivalent deploy this page links into
+- [[Hermes]] — the agent that drove the CDP daemon alongside this desktop
+- [[GGUF Compatibility]] — unrelated but shares the "read the file, not the name" lesson

serverAppConfigs-docs/FrankBook-Pro/ARR-Stack.md

@@ -5,3 +5,115 @@ description: "The *arr automation stack (Prowlarr, Sonarr, Radarr, Lidarr, Reada
 tags: [arr-stack, prowlarr, sonarr, radarr, lidarr, readarr, qbittorrent, frankbook-pro, dankserver]
 timestamp: 2026-09-04T00:00:00Z
 ---
+
+# *arr Stack
+
+The *arr automation stack runs on **FrankBook-Pro** (the Mac mini) as quadlets
+inside its podman machine; all egress rides the gluetun WireGuard netns. The
+per-service pages ([[Prowlarr]], [[Sonarr]], [[Radarr]], [[Lidarr]],
+[[Readarr]], [[qBittorrent]]) carry deployment facts; this page holds the
+cross-cutting API and auth-posture procedures.
+
+## API version split (memorize before any *arr HTTP call)
+
+- Sonarr/Radarr: `/api/v3/…`; Lidarr/Readarr/Prowlarr: `/api/v1/…`.
+- Prowlarr is port **9696** — 8090 is Jackett.
+- **Empty body = endpoint absent**, not a failure: probe endpoints by status
+  code first; v1 has no `/api/v1/profile` endpoint at all.
+- `PUT /downloadclient` and `PUT /indexer/{id}` take a SINGLE resource object
+  (GET it, patch `settings`, PUT the whole object back). An array body fails
+  with `Path: $ | BytePositionInLine: 1`.
+- API keys live in each container's config:
+  `podman exec <svc> grep -o "<ApiKey>[^<]*" /config/config.xml | cut -d">" -f2`.
+  They are local-only secrets — they stop being secret the moment a vhost
+  serves the API unauthenticated.
+
+## Schema-first probing (validators are free documentation)
+
+Send the FULL schema template object with fields filled; v1 validators reject
+partials, and each error names the next required field — peel validators one
+live call at a time instead of guessing payload shapes:
+
+- `'Fields' must not be empty` → send a full schema copy, not `fields:[]`.
+- `'Base Url' must not be empty` when keys were camelCase → **PascalCase is
+  required** (`BaseUrl`, `ApiKey`) — schema keys, not the JSON wire keys.
+- `DefaultMetadataProfileId must be > 0` → fetch real ids from
+  `GET /api/v1/metadataprofile` (rows like `(1,'Standard')`);
+  `GET /api/v1/qualitydefinition` returns the quality CATALOG — ids ≠
+  profiles.
+- `targetType null` → configContract not registered in that build (this
+  Prowlarr build has no Prowlarr-plugin indexer; integration is
+  Torznab-family only).
+- `testingPackageId` in a PUT body downgrades validation failures to
+  warnings.
+- `syncLevel` is enum `ApplicationSyncLevel` — omit it or use the int; a
+  string is rejected.
+
+**Verification rules:** `POST /downloadclient` returns the COLLECTION, not
+the created object — verify via GET. Test endpoints return verdicts in the
+BODY at HTTP 200 (`{}` = pass, a list = fail) — judge from bodies, not
+status codes. Command POSTs (`MoviesSearch`, `ApplicationSync`) run
+server-side and respond only when the work finishes: curl `-m` timeouts
+(HTTP 000) during a search are NOT failures — poll queue/history afterward
+instead of re-firing; warm small commands (`RefreshMonitoredDownloads`)
+answer in ~20 ms, so a hang means live work in progress, not a dead API.
+
+**DB as arbiter:** when endpoint shapes are opaque, stop the container and
+edit `/opt/prowlarr/config/prowlarr.db` with `sudo python3` sqlite directly,
+then start. The indexers table columns are
+Id/Name/Implementation/Settings/ConfigContract/Enable/Priority/Added — the
+enable flag column is `Enable` (not `EnableRss`). Prowlarr's real application
+settings values live in `prowlarr.db` → `Applications.Settings` JSON (the
+`/api/v1/applications` GET responses carry `fields: []`).
+
+## Auth posture probe (ground truth is behavior, not config flags)
+
+`AuthenticationMethod=None` + `AuthenticationRequired=Enabled` in
+`config.xml` is ambiguous on paper; on the wire it means a login screen with
+**zero users** — first-account-wins.
+
+1. **Confirm which host runs the service before probing** (`podman ps` on
+   the target, not the box you ssh'd into first — the arrs live on
+   FrankBook-Pro).
+2. Probe an API endpoint that bypasses the SPA shell, locally and through
+   any vhost: `curl -s -o /dev/null -w "%{http_code}"
+   http://<host>:<port>/api/v3/system/status`. **SPA roots (`/`) always
+   serve a 200 shell — never conclude "open" or "gated" from `/`.**
+   401 = gated; 200 + JSON = API fully open (anyone on the net can drive it
+   with curl).
+3. For config truth, dump the whole file
+   (`podman exec <svc> cat /config/config.xml`), not a piped
+   `grep | sed -n 2p`: **`sed -n Np` exits 0 on zero matches**, so the
+   pipeline hides a miss and you conclude "empty field" when the grep itself
+   failed.
+4. Decision framing: when posture = gated-with-zero-users, present it as a
+   decision menu (None behind the front-door wall vs External/authentikarr),
+   not a silent change — first-account-wins means whoever logs in first owns
+   the instance.
+
+## The `None` + `Enabled` auth-config trap (v1 .NET apps)
+
+`<AuthenticationMethod>None</AuthenticationMethod>` +
+`<AuthenticationRequired>Enabled</AuthenticationRequired>` is
+self-contradictory: the UI Save is permanently disabled, and any save throws
+`DryIoc.ScopedItemException` in `AuthorizationMiddleware` → 500 on every
+page after restart. The valid no-auth representation is **absence of the
+`<AuthenticationRequired>` key**, not the literal string `Disabled`. Fix =
+delete the key from `/config/config.xml`
+(`sed -i "/<AuthenticationRequired>/d"`), restart the container. Tell-tale:
+one v1 app boots clean while siblings 500-sprawl → diff their config.xml
+auth blocks; the healthy one shows the correct shape. (Prowlarr runs
+`Forms` + `Enabled` healthily — the crash combo is `None`+`Enabled` only.)
+
+General rules that saved the session these came from: container
+`State=running` ≠ app serving — read `podman logs --tail 12 <c>` before
+theorizing; when a UI gate blocks saving, go under it (these apps keep
+plaintext XML config on disk — read/edit via `podman exec`).
+
+## See Also
+
+- [[Prowlarr]] — indexer feeds and the v2 sync model
+- [[Sonarr]] · [[Radarr]] · [[Lidarr]] · [[Readarr]] — the managers
+- [[qBittorrent]] — download client, 4.6.7 pin, E2E verification
+- [[FrankBook-Pro]] — the host and its podman machine
+- [[Homepage Status Probes]] — how the e2e gate treats this sleeping host

serverAppConfigs-docs/FrankBook-Pro/qbittorrent.md

@@ -111,6 +111,90 @@ Restart=always
   probe one ssh-context out returns the residential IP — a false leak alarm.
 - Ensure umask is set to `022` so downloaded files are readable by arr apps.
 
+## E2E verification after any pipeline change
+
+Run after ANY change to the torrent pipeline — WireGuard credential rotation,
+qB version change, gluetun reconfig, category re-seeding. Proves the full
+grab path (tunnel egress → qB download → category save path → clean
+removal). All commands from a dev box via `ssh frankbook-pro`; inside the
+Mac shell export `PATH="/usr/local/bin:/opt/homebrew/bin:$PATH"` for
+`podman machine ssh`.
+
+**Access note (2026-09-21):** the Mac's login user is **dank**@192.168.0.112
+(NOT core — `core` is the podman-machine VM user, reached only via
+`podman machine ssh`). blankpad has key auth for `dank@` (bootstrapped
+2026-09-21) and a `frankbook-pro` alias in `~/.ssh/config`. macOS sshd has a
+low MaxAuthTries: when password-bootstrapping use `-o
+PreferredAuthentications=password -o PubkeyAuthentication=no
+-o IdentitiesOnly=yes` or the key offers exhaust the attempts before the
+password is tried.
+
+**1. Egress gate — MUST run inside gluetun:**
+
+```bash
+podman machine ssh 'podman exec gluetun wget -qO- https://ipinfo.io/json'
+```
+
+Require `org` = AS9009 (M247 — ProtonVPN Panama PA-12 config); judge by org,
+never country alone. NEVER run this one ssh-context out (bare `wget` on the
+Mac host): it silently returns the residential Comcast IP and reads as a
+hard-gate leak that does not exist. A residential reading ⇒ re-run inside
+the container before diagnosing.
+
+**2. Login — generic cookie parse.** The QPW fetch is flaky (~1 in N returns
+empty); assert length before use — an empty-password run once produced a
+Forbidden loop that cost hours:
+
+```bash
+QPW=$(podman machine ssh 'cat /home/core/Containers/qbittorrent/env' | grep QBIT_WEBUI_PASSWORD | cut -d= -f2)
+[ ${#QPW} -eq 20 ] || exit 1
+SID=$(curl -si --data-urlencode "username=admin" --data-urlencode "password=$QPW" \
+  http://127.0.0.1:8080/api/v2/auth/login \
+  | grep -oE '[A-Za-z_]*SID[A-Za-z0-9_]*=[^;[:space:]]*' | head -1)
+curl -b "$SID" …   # pass the WHOLE match
+```
+
+WebUI from the LAN: `http://192.168.0.112:8080` (published on the gluetun
+unit). Username `admin`; password = the machine-local `QBIT_WEBUI_PASSWORD`
+(20 chars, generated once at provisioning by the deploy handler — never in
+the repo). Parse-bug 403s never reach qB as failed logins — no ban accrues;
+server-side truth is `/Volumes/COMPLETE/qbittorrent-data/qBittorrent/config/qBittorrent/logs/qbittorrent.log`
+(`WebAPI login success/failure` + source IP). Repeated REAL failures accrue
+toward qB's 5-failures/3600 s IP ban.
+
+**3. Add a legal test torrent.** Blender open-movie torrents (~300 MB, e.g.
+Big Buck Bunny) via `POST /torrents/add` with the raw `.torrent` file
+(`-F "file=@…"`) and `category=Finished Torrents`. Never leave a real-looking
+grab behind.
+
+**4. Poll and verify.** Poll `GET /torrents/info` (~20 s): state
+`downloading` → `moving` (INCOMPLETE→COMPLETE move) → `stalledUP`/`uploading`
+at progress=1. `save_path` must be `/Volumes/COMPLETE/Finished Torrents`
+(qB 4.x auto-creates unknown categories at DefaultSavePath; 5.x did NOT
+inherit category paths — one more reason for the 4.6.7 pin). Confirm on
+disk: `ls "/Volumes/COMPLETE/Finished Torrents/"`. Throughput ~17 MB/s is
+normal for the tunnel.
+
+**5. Cleanup — delete only your artifact:**
+
+```bash
+curl -s -o /dev/null -w "%{http_code}" -b "$SID" \
+  --data-urlencode "hashes=<literal-hash>" \
+  --data-urlencode "deleteFiles=true" \
+  http://127.0.0.1:8080/api/v2/torrents/delete   # expect 200
+```
+
+Target the known hash literally — the user's real grabs share the directory
+and MUST be untouched. Verify: hash absent from `/torrents/info`, your dir
+gone, all other entries intact.
+
+**Version-coupling constraints:** qB is pinned to 4.6.7 (Readarr's EOL
+client rejects qB ≥ 5's `204` login at runtime — see the pin rationale
+above). Do not unpin while Readarr exists. After ANY qB version flip,
+regenerate the `WebUI\Password_PBKDF2` line fresh (same password, 100k
+iterations) — a 5.x-written hash line makes 4.6.7 reject every login.
+Rewrite while qB is stopped.
+
 ## See Also
 
 - [[ARR-Stack]] — full *arr stack documentation

serverAppConfigs-docs/Infrastructure/Anubis.md

@@ -131,6 +131,55 @@ ungated), and gitea (git smart-HTTP clients cannot solve JS challenges —
    distinguish challenge from deny at this release; grep the body for
    `Access Denied` / `Making sure you` instead.
 
+## Guard-chain layer diagnosis (a user reports a guarded page "broken")
+
+Use when a guarded page (e.g. the `www.franksdank.cloud` homepage) reads as
+"broken / links gone / not loading" and the fault must be localized to
+server-side or client-side/session. Chain topology:
+`Browser → Caddy TLS edge → Anubis :8923 → Caddy :8016 → Authentik outpost
+→ Caddy :8017 (@home/@data handlers) → app/static`.
+
+Layer-by-layer checks (run from the host via ssh):
+
+1. **Caddy unit + guard containers**: `systemctl --user is-active
+   caddy.service`; `podman ps --format '{{.Names}}\t{{.Status}}' | grep -iE
+   'anubis|authentik|caddy'` — expect `Up … (healthy)`.
+2. **Anubis challenge page validity**: `curl -sk
+   https://www.franksdank.cloud/ --resolve www.franksdank.cloud:443:192.168.0.143`
+   — a healthy challenge contains `"anubis_challenge"` JSON with
+   `difficulty` and the main JS.
+3. **Outpost errors**: `podman logs --tail 40 authentik-outpost 2>&1 | grep
+   -iE 'error|exception|traceback|500|provider'` — routine dramatiq task
+   logs are normal.
+4. **Post-auth handlers from INSIDE the caddy container** (8016/8017 live in
+   its netns — `curl http://127.0.0.1:8017` from the HOST returns `000`
+   connection refused, which is EXPECTED, not a fault):
+   `podman exec caddy sh -c 'wget -q -S -T5 --header "Host:
+   www.franksdank.cloud" -O- http://127.0.0.1:8017/'` and the same for
+   `/services.json`, `/status.json`. Expect 200 + body. (BusyBox wget in
+   this container has NO `-H` flag — use `--header "Host: …"`.)
+5. **Files on disk**: `ls -la /srv/homepage/` and validate `status.json`
+   parses (`python3 -c "import json;json.load(open('/srv/homepage/status.json'))"`).
+
+Tool/behavior notes: the anubis container is distroless — `podman exec anubis
+sh` fails (`sh: executable not found`); inspect via its logs or the served
+challenge page. Anubis logs `x-real-ip: 10.89.1.22` (the Caddy container's
+podman IP) for ALL clients — rootless-podman published-port SNAT;
+clients are indistinguishable by IP and clearance is cookie-based. Do NOT
+mistake that for the bug. Homepage render: `app.js` does
+`Promise.all([fetch services.json, fetch status.json])`; on ANY fetch
+failure the catch sets `#summary` to `status unavailable: <err>` and the
+grid stays EMPTY — which a user reads as "links gone / not loading." A
+30-service status.json with a 9-service services.json renders fine
+(stateByName lookup).
+
+**Decision:** if every layer is healthy (challenge valid, outpost clean,
+handlers 200, files render), the fault is client-side/session. STOP
+thrashing the live system — repeated failed browser challenges share the
+masked 10.89.1.22 identity. Get the authenticated view: ask for the exact
+symptom (challenge spinning? login loop? empty grid? "status unavailable"
+text?) or the authentik login to drive the page in a real browser.
+
 ## Verify live
 
 ```

serverAppConfigs-docs/Infrastructure/Authentik.md

@@ -342,6 +342,168 @@ The silent re-auth path was proven end-to-end: after deleting
 `qa-e2e-admin`'s rows, the e2e admin spec ran against the stale storage
 state and passed with no login prompt (2 tests, 2.9 s).
 
+## Guard-chain flip recipe (adding a guarded vhost)
+
+Move a public vhost from `reverse_proxy <app>:<port>` onto the chain:
+client → TLS edge → Anubis :8923 → outpost decision → internal
+`http://:8017` listener → Host-matched handler → real upstream.
+
+**Order (never reorder):**
+
+1. **Provider first.** Create the outpost ProxyProvider before touching
+   routes, or authed traffic hits the outpost error page instead of the app.
+   Fleet convention for `external_host` is the exact browser-facing origin
+   (the inventory table carries both bare origins and trailing slashes;
+   routing matches on hostname, verified 2026-09-02).
+   `internal_host = http://caddy:8017`, all three flows set, and an
+   Application attached — the two provisioning rules above.
+2. **Route flip in ONE commit/deploy**: the vhost becomes
+   `reverse_proxy anubis:8923` with `header_up X-Real-Ip {remote_host}`, and
+   inside the `http://:8017` listener add `@<svc> host <vhost>` +
+   `handle @<svc> { reverse_proxy <upstream> }`. Anubis guard set, Caddy TLS
+   edge, `:8017` routing table, and provider list change together.
+3. Deploy via the repo deploy flow (commit first; deploy pushes main itself).
+
+**Verification (all three):**
+
+- Unauthenticated probe from a client: `curl -s https://<vhost>/` → HTTP 200
+  challenge page; grep the body for `challenge|not a bot|anubis` (~8
+  markers on a live challenge). A 502 here instead of the challenge = chain
+  mis-wired.
+- Post-auth routing proof: grep the LIVE config inside the container —
+  `podman exec caddy sh -c "grep -n -A3 <handler> /etc/caddy/Caddyfile"` —
+  after the deploy prints its reload line; confirm the reload actually ran,
+  don't trust exit 0.
+- Outpost reachability: `podman exec authentik-server getent hosts caddy`
+  resolves the compose DNS alias (10.89.1.x) that `internal_host` dials.
+
+**Pitfalls:** probing `http://localhost:8017/` from the host returns HTTP
+000 (connection refused) — the post-auth listener lives in the caddy
+container's netns; probe inside it or grep the config. Non-browser clients
+cannot solve the JS challenge — before flipping, audit service-to-service
+traffic: ingestion/push paths must target internal addresses (`loki:3100`),
+never a public hostname of a guarded service. Exempt-by-construction
+categories stay unguarded: the IdP itself (`auth.`), git smart-HTTP
+endpoints, services with their own complete auth — document the exemption
+in the vhost comment.
+
+## Driving the ORM (`ak shell`)
+
+All authentik object work happens through the server container's ORM shell —
+no repo change, no deploy (these providers are NOT repo-managed; capture new
+provider+application pairs as a blueprint or setup script to avoid drift).
+
+```bash
+ssh dankclaw 'podman exec -i authentik-server ak shell' <<'EOF' | sed -n '/229 objects/,$p'
+from authentik.core.models import User, Group
+...
+EOF
+```
+
+- The heredoc IS the program: `ak shell` reads all of stdin and execs it
+  (heredoc-through-ssh quoting breaks; the raw-stdin pattern is the proven
+  one).
+- Bootstrap prints JSON log lines, then
+  `### authentik shell (2026.8.3)` and `229 objects imported automatically…`
+  — pipe through `sed -n '/229 objects/,$p'` to drop the noise.
+- 229 models are pre-imported. Import paths (each wrong guess costs a failed
+  exec): `User`, `Group`, `Application`, `ApplicationEntitlement`,
+  `PropertyMapping` → `authentik.core.models`; `ProxyProvider` →
+  `authentik.providers.proxy.models` (NOT `authentik.outposts.models`);
+  `LDAPProvider` → `authentik.providers.ldap.models`;
+  `authentik.core.models.application` does NOT exist as an import path; there
+  is no `authentik.policies.group` module in this build.
+- For live DB truth prefer the postgres container
+  (`podman exec authentik-postgres psql -U authentik -d authentik -At -c
+  "select * from …"`) — **query with `select *` and stop guessing column
+  names**: 2026.x renamed columns (`Provider.objects.filter(type=…)` raises
+  FieldError — `type` moved to subclass tables; the outpost `enabled` column
+  may not exist under that name; provider pk is `oauth2provider_ptr_id`).
+
+## User provisioning playbook (user-level account)
+
+There is no `User.create_user` classmethod, and `is_superuser` is NOT a
+constructor kwarg (it is a property derived from group membership) — passing
+it raises `TypeError: User() got unexpected keyword arguments`:
+
+```python
+from authentik.core.models import User, Group
+u = User(username="ridley", email="ridley@franksdank.cloud", name="Ridley", is_active=True)
+u.set_password("...")        # password arrives on stdin only, never argv
+u.save()
+for g in ("Non-Admin Users", "jellyfin_users", "openwebui_users"):
+    u.groups.add(Group.objects.get(name=g))
+```
+
+`path` defaults to `users` (regular). The access map (verified live
+2026-10-04): there are NO per-app policy bindings on the proxy providers
+(all use the implicit-consent "Authorize Application" flow), so guarded-vhost
+access is decided entirely by GROUP MEMBERSHIP consumed downstream —
+
+- `homepage-admins` → Caddy RBAC for homepage `/admin` (`X-Authentik-Groups`
+  regex gate) and the homepage admin panel; regular users get only
+  `category: user` panels from `services.json`.
+- `openwebui_users` / `openwebui_admins` → ApplicationEntitlements
+  "Open WebUI Users"/"Open WebUI Admins" (bound by group), consumed by the
+  "Open WebUI roles" property mapping → `roles` claim. Verify live with
+  `u.app_entitlements(app)`.
+- `jellyfin_users` / `jellyfin_admins` → Jellyfin via the LDAP outpost
+  (filter `(memberOf=cn=jellyfin_users,ou=groups,dc=franksdank,dc=cloud)`).
+- `Non-Admin Users` = the plain-user marker group (precedents: alex,
+  qa-e2e).
+- EXCLUDE from a user-level account: `homepage-admins`, `jellyfin_admins`,
+  `openwebui_admins`, `authentik Admins`, `authentik Read-only`.
+- Other user-category vhosts (SearXNG, Gitea, Calibre-Web, RomM, Copyparty,
+  Plex, Seerr, Marreta) need no per-user group: any authenticated user
+  passes the outpost.
+
+Verify login behaviorally — `authenticate()` is the exact PasswordStage code
+path:
+
+```python
+from django.contrib.auth import authenticate
+assert authenticate(request=None, username="ridley", password="...")   # -> User
+assert authenticate(request=None, username="ridley", password="nope") is None
+```
+
+Look flows up by slug, not name:
+`Flow.objects.get(slug="default-authentication-flow")` (its `.name` is
+"Welcome to authentik!" — `.get(name=…)` fails). Stage order:
+Identification(10) → Password(20) → AuthenticatorValidate(30; no-op with no
+enrolled device) → UserLogin(100). With no MFA enrolled, nothing else blocks.
+
+More field gotchas (each cost a failed exec): `ApplicationEntitlement.application`
+can be null (guard with `getattr`); `LDAPProvider` has
+`search_mode`/`bind_mode`/`base_dn`, lacks `search_base`/`of_groups`/
+`organization`; `OAuth2Provider` has no `.scopes`; `Outpost.config` is an
+`OutpostConfig` object, not a dict; `Policy` has no uniform `.policy_cls`
+(use `type(p).__name__`); `User` has no `.status`.
+
+Never rotate `AUTHENTIK_SECRET_KEY` to fix claim issues — it invalidates
+every outpost session fleet-wide; purge the stale `proxysession` rows
+instead (section above).
+
+## Probing the outpost from inside the netns
+
+Rootless podman gives each container its own netns; **host `ss -ltn` cannot
+see listeners inside other containers' namespaces** — a missing listener on
+the host is an observation artifact, not a fault. Distroless images have no
+shell (`podman exec authentik-server sh` fails), so probe by joining the
+namespace:
+
+```
+podman run --rm --network container:<container> --entrypoint /bin/busybox \
+  docker.io/library/busybox:latest \
+  wget -q -T 4 -t 1 -O /dev/null -S "<url>" 2>&1 | head -8
+```
+
+- Quote URLs over ssh: the remote shell is zsh and bare `?` in query strings
+  triggers globbing (`no matches found`).
+- Boot transience: immediately after a restart the same probe can return 503
+  (configure task pending); poll until the answer changes before concluding.
+- `podman inspect --format "{{.Config.NetMode}}"` errors on this podman
+  version — the field is `.HostConfig.NetworkMode`.
+
 ## Failure modes
 
 | Symptom | Cause | Fix |

serverAppConfigs-docs/Infrastructure/Homepage Status Probes.md

@@ -0,0 +1,158 @@
+---
+type: reference
+title: Homepage Status Probes
+description: The homepage TUI status page — how probe targets are chosen (host-netns vantage), the three false-down shapes, triage procedure, local render check, deploy, and the e2e gate interactions (asleep Mac mini, raw-IP matrix).
+tags: [homepage, status, probes, e2e, homepage-status]
+timestamp: 2026-10-05T00:00:00-07:00
+---
+
+# Homepage Status Probes
+
+The homepage is a static site —
+`shared/homepage/{index.html,admin.html,app.js,style.css,services.json,docs.html}`
+— served from `/srv/homepage` on **dankclaw** only (`deploy_homepage` in
+`scripts/deploy-remote.sh` refuses other targets). A checker
+(`scripts/homepage-status.py`, run by the `homepage-status.timer` unit) probes
+each card's target and writes `/srv/homepage/status.json` (mode **0644** so
+Caddy can serve it — losing the mode 403s the page).
+
+## The checker's vantage and its three false-down shapes
+
+The checker probes TCP/HTTP **from the host netns**. It only has three states
+— `wip` | `up` | `down`; there is no "unknown", so an unprobeable service can
+never render honestly and must be dropped or re-pointed, not "fixed".
+
+1. **TLS-only app probed with `method:"http"`.** Plex Media Server serves TLS
+   only on `32400`; plain-HTTP GET → TLS alert → curl exit 52, code `000` →
+   card says down while PMS answers `200` over HTTPS. Fix:
+   `"method":"tcp"` (connect is the honest liveness signal; the public URL
+   stays HTTP because Caddy re-keys).
+2. **Retired port.** The Hermes card probed `127.0.0.1:8090` after the
+   compose→quadlet move moved the dashboard to `9119` and the webui to
+   `8787`; nothing listened on 8090, so a healthy stack read `down`.
+3. **Unpublished port / wrong netns.** `gitea-postgres` (`db:5432`, net
+   `systemd-gitea`), `searxng-valkey` (`valkey:6379`, net `systemd-searxng`)
+   and `alloy` (OTLP 12345) publish **no** ports. Container DNS aliases are
+   netns-scoped, so from the host these are unreachable *by construction* —
+   no probe value makes them honest. Drop those cards.
+
+A host-vantage probe can only reach a listener **published to the host**
+(rootlessport/pasta); unpublished ports are invisible from the host netns.
+See [[Container Host Reachability]] for the netns rules.
+
+## Triage procedure
+
+1. Read the card's probe target in `shared/homepage/services.json`; note
+   method (`tcp` vs `http`).
+2. Does that port have a **host-visible listener**?
+   `podman ps -a --format '{{.Names}}|{{.Status}}'`, then per container
+   `podman inspect <c> --format '{{json .NetworkSettings.Networks}}
+   {{.HostConfig.NetworkMode}}'` and `podman port <c>` (empty output =
+   published nothing → structurally unprobeable from the host).
+3. Probe from the **correct vantage**: from a bridge-net container the host
+   is its gateway IP (`/proc/net/route`, e.g. `10.89.1.1` on
+   `dankclaw_default`) — *not* `host.containers.internal` (`169.254.1.2`),
+   which belongs to the default `podman` net (`10.88.0.0/16`). Probe with
+   bash `/dev/tcp` (no `python3`/`ip` in these images); escape loop vars so
+   they expand **inside** `podman exec bash -c`.
+4. Classify: real outage (container exited — read `State.ExitCode`: 137 =
+   SIGKILL, 143 = SIGTERM/manual stop; `OOMKilled=false` rules out the OOM
+   killer) vs false down (running + unreachable target).
+5. Fix: re-point or re-method for HTTP/TLS mismatches; **drop** cards whose
+   only exposure is an internal netns; never invent a probe target.
+6. Verify by re-running the checker path, not by eyeballing: after deploy,
+   re-read `/srv/homepage/status.json` and diff the `down` list
+   before/after.
+
+Removing a card's `probe` key without a code change crashes the checker
+(`svc["probe"]` KeyError) — dropping the whole entry is the only zero-code
+fix. The `description` field is consumed only by `app.js` `descHTML()`, so
+adding a `details` object (role/spec/inputs/use) is schema-safe:
+`tests/homepage_config.bats` checks only name/category/section/probe.
+
+## Local render check (before every deploy)
+
+The page fetches `services.json` + `status.json` at load; `file://` is
+CORS-blocked, so serve over HTTP:
+
+```bash
+cd shared/homepage
+python3 -c "import json; s=json.load(open('services.json')); \
+  json.dump({'services':[{'name':x['name'],'state':'up'} for x in s]}, open('status.json','w'), indent=2)"
+python3 -m http.server 8899 --bind 127.0.0.1
+```
+
+Then in the browser: load `http://127.0.0.1:8899/index.html`, **wait ~1.5 s**
+(panels render after the async fetch; checking immediately returns 0 panels),
+extract via `tab.evaluate` — LLM cards render `.desc-role`/`.desc-spec`/
+`.desc-inputs`/`.desc-use` from the card's `details` object, other cards a
+plain `.desc`; `getComputedStyle(el).color` confirms tones. Screenshot for
+visual proof. Expected local noise: `deployed.js` 404s (baked in at deploy
+time) — harmless.
+
+The scratch `status.json` is untracked and NOT git-ignored — move it to
+`shared/homepage/.AGENTTRASH/` after the check (repo policy: never `rm`).
+
+CSS palette lives in `style.css` `:root` (`--fg` bright green, `--dim` dim,
+`--head` pale, `--link` cyan, `--wip` yellow, `--down` red). Add new tones as
+`--*` vars and include any new `.desc-*` class in the `html[data-bare="1"]`
+reset so `?bare=1` strips it to black.
+
+## Deploy
+
+```
+./scripts/deploy.sh dankclaw --service homepage --skip-e2e
+```
+
+Copies the assets to `/srv/homepage`, regenerates `deployed.js` (git commit
+datetime), restarts the `homepage-status` timer. Verify live:
+`ssh dank@192.168.0.143 'grep … /srv/homepage/…'`.
+
+**`--skip-e2e` is justified for homepage-only changes when the Mac mini is
+asleep**: the e2e admin crawl (`tests/e2e/specs/11-admin.spec.ts:24`) walks
+Prowlarr at `192.168.0.112:8989` on FrankBook-Pro; an asleep mini (ARP STALE,
+no remote wake) times out that test — a pre-existing environmental failure,
+not a regression. All non-Mac-mini e2e tests pass.
+
+## e2e raw-IP *arr failures — probe from the browser's box, not a peer
+
+When `11-admin` fails with `ERR_CONNECTION_TIMED_OUT`, the instinct is "the
+Mac is asleep". Sometimes true; not always. Playwright runs on the dev box,
+so a healthy probe from dankclaw proves nothing about the **browser's** path:
+
+```bash
+# 1. From the box that RUNS the browser:
+curl -s -o /dev/null -w '%{http_code}\n' --max-time 7 http://192.168.0.112:8989/
+# 2. From a LAN peer:
+ssh dankclaw 'for p in 8989 7878 9696; do curl -s -o /dev/null -w "$p %{http_code}\n" \
+  --max-time 6 http://192.168.0.112:$p/; done'
+# 3. Liveness (ICMP, not TCP):
+ping -c 2 -W 3 192.168.0.112
+```
+
+| peer probe | dev-box probe | ping | meaning |
+|---|---|---|---|
+| 200 | 200 | ok | transient — re-run the spec |
+| timeout | timeout | fails | Mac asleep; wake, ports return in seconds |
+| **200** | **rc=28** | **ok** | **path problem, not sleep** — ICMP allowed, TCP blocked. Waking cannot help |
+
+Third row: confirm the Mac is innocent through the dankclaw jump
+(`pgrep -fl socat`; `netstat -an | grep -c 8989`; loopback
+`curl http://127.0.0.1:8989/` → 200; `/usr/libexec/ApplicationFirewall/socketfilterfw
+--getglobalstate` → disabled). Forwarders up + loopback 200 ⇒ the block is
+between the dev box and the Mac.
+
+Don't hold a deploy hostage to a known-environmental red gate: verify the
+deployed service **directly** (its own ledger/health/real completion), state
+the gate was skipped or partially red and why (`--skip-e2e` prints the
+warning, so the skip is recorded), and report the residual gap plainly
+("29/30 e2e pass; 11-admin fails on Sonarr's raw IP from the dev box; peer
+probe returns 200; deploy verified by other means"). Do not weaken or skip
+the spec — it guards a real SSO-plus-RBAC contract.
+
+## Related
+
+- [[Container Host Reachability]] — the netns rules behind the vantage
+- [[Observability Provisioning]] — the monitoring stack reading the same fleet
+- [[Deploy State Triage]] — service-scoped deploys and the e2e gate
+- [[Testing Policy]] — the seam/e2e contract this page's checks live in

serverAppConfigs-docs/Infrastructure/LAN-DNS.md

@@ -158,6 +158,119 @@ The Porkbun zone `franksdank.cloud` carries an A record for every vhost plus the
 | HTTPS works from the host but not the WAN | No port forward on the G36 for that service (manual router step) |
 | Cert renewal fails in Caddy logs | Porkbun API credentials in the Dankclaw `.env`; test with the Porkbun API `/ping` endpoint |
 
+## Per-client fix: resolver drop-in + browser DoH bypass
+
+Symptom: a LAN machine gets `connection timeout` (curl rc=28) on
+`https://*.franksdank.cloud` while the server is healthy. Root cause: the
+public DNS hands out the WAN IP `71.227.227.172`, which has **no 443
+port-forward** (filtered). Any resolver that is NOT Pi-hole produces the WAN
+answer → SYN to a filtered port → timeout. There are TWO resolver layers to
+check, in order:
+
+**Layer 1 — systemd-resolved (fixes curl/shell).** A route-only drop-in
+sends `*.franksdank.cloud` ONLY to Pi-hole and never to the default-route
+servers:
+
+```bash
+dig +short www.franksdank.cloud @192.168.0.143   # expect 192.168.0.143
+dig +short www.franksdank.cloud @1.1.1.1         # expect 71.227.227.172 (the trap)
+getent hosts www.franksdank.cloud                # what the OS actually resolves
+curl -sk --resolve www.franksdank.cloud:443:192.168.0.143 \
+     https://www.franksdank.cloud/ -o /dev/null -w '%{http_code}\n'   # 200 = server fine
+
+sudo mkdir -p /etc/systemd/resolved.conf.d
+sudo tee /etc/systemd/resolved.conf.d/franksdank-lan.conf >/dev/null <<'EOF'
+[Resolve]
+DNS=192.168.0.143
+Domains=~franksdank.cloud ~franksdank.casa
+EOF
+sudo systemctl restart systemd-resolved && resolvectl flush-caches
+```
+
+The `~` (route-only) prefix is load-bearing: it routes only those domains to
+Pi-hole. Verify: `getent hosts www.franksdank.cloud` → `192.168.0.143`;
+`resolvectl status` shows `DNS Domain: ~franksdank.cloud ~franksdank.casa`.
+
+**Layer 2 — browser DoH (the bypass: "curl works, browser times out").** If
+Layer 1 is verified but the BROWSER still times out, the browser is doing
+its own DNS over HTTPS and never asks systemd-resolved — it queries the TRR
+provider (Cloudflare et al.) directly → WAN answer → timeout. Prove it:
+`curl -sk -m 6 https://71.227.227.172/` → rc=28 (the WAN answer is a dead
+end). Diagnosis details:
+
+- Firefox profile may be at the XDG path `~/.config/mozilla/firefox/`, NOT
+  `~/.mozilla/firefox/` — find it via the running process if the default
+  path is empty.
+- `grep network.trr. <profile>/prefs.js` — **no entries = shipped default,
+  and since Firefox 113 the shipped default is DoH ON.**
+
+Fix (surgical, keeps DoH for everything else): Firefox → Settings → Privacy
+& Security → **DNS over HTTPS** → "enabled for these domains" → **Add…** →
+`franksdank.cloud` (and `franksdank.casa`). The exception list = domains
+that do NOT use DoH; they fall back to the system resolver, which has the
+route-only drop-in. Or set DoH to **Off** entirely (coarser). Chrome:
+Settings → Privacy and security → Security → "Use secure DNS" → off, or
+"Use the same settings as your system". Do NOT hand-edit `prefs.js` while
+the browser is running — it holds prefs in memory and rewrites the file on
+exit, clobbering the edit.
+
+This is a per-client fix: every LAN client that must reach the homelab by
+name needs Pi-hole as its DNS (or the drop-in) AND a browser that does not
+bypass it with DoH. The server side needs no change. Verified on blankpad
+2026-09-21: the drop-in was present since 09-12, curl returned 200, and
+Firefox 155 with default (ON) DoH was the remaining hole.
+
+## Porkbun zone ops (franksdank.cloud)
+
+Operate the public DNS zone via the Porkbun API v3, run **on Dankclaw** so
+the API key/secret sourced from `/home/dank/serverAppConfigs/.env`
+(`PORKBUN_API_KEY`, `PORKBUN_API_SECRET_KEY`) never leaves the host. Pipe
+scripts to the target with `ssh -o BatchMode=yes dankclaw 'bash -s' <
+/tmp/script.sh`.
+
+Endpoints (all POST, JSON body):
+
+- List: `https://api.porkbun.com/api/json/v3/dns/retrieve/franksdank.cloud`
+  — auth via headers `X-API-Key` + `X-Secret-API-Key` (Bearer is wrong
+  here). Response shape is **top-level `.records[]`**, NOT
+  `.content.records[]` as some docs claim — parse with
+  `jq -r '.records[] | [.id,.name,.type,.content,(.ttl|tostring)] | @tsv'`.
+- Delete: `/dns/delete/{domain}/{id}` (body carries apikey+secretapikey).
+- Edit: `/dns/edit/{domain}/{id}` — pass the full record (`id`, `name`,
+  `type`, `content`, `ttl`, `prio`).
+
+Quirks that bite (verified live 2026-08-31):
+
+1. **`name` is a RELATIVE label on edit.** Passing `ai.franksdank.cloud`
+   yields the record `ai.franksdank.cloud.franksdank.cloud`. Pass the bare
+   label (`ai`) and the API appends the zone itself. Verify after every
+   rename with a retrieve before declaring done.
+2. **Rate limit answers as an HTML 404-style page, not JSON.** A batch of
+   rapid mutations makes some calls return `<!DOCTYPE html>...` where jq
+   then errors (`Cannot iterate over null` / `Invalid numeric literal`). Do
+   not assume the call succeeded — sleep ~8 s and retry the failed ids;
+   verify by re-listing the zone at the end.
+3. Deletes/renames are per-record-id: always retrieve first, map name→id,
+   then mutate by id.
+
+Rename procedure (vhost rename = DNS + Caddy together):
+
+1. `retrieve` → collect ids for the renamed names.
+2. `edit` each id with the new **relative** label, preserving
+   type/content/ttl.
+3. Re-retrieve and confirm `<new>.franksdank.cloud A <ip>`.
+4. In-repo lockstep before deploy: rename the Caddyfile vhost block AND the
+   Pi-hole `dankclaw/pihole/addn-hosts` line in the same commit — a test
+   fails the suite if any Caddyfile vhost lacks an addn-hosts A record on
+   192.168.0.143.
+5. Deploy via `scripts/deploy.sh dankclaw` (plans pihole+caddy from changed
+   paths). Capture the suite's true exit with
+   `out=$(./tests/run.sh 2>&1); rc=$?` — piping to `tail` masks the exit
+   code behind `tail`'s own status and has shipped untested commits here.
+6. Verify live: `dig +short <name> @192.168.0.143` then
+   `curl -o /dev/null -w %{http_code} https://<new-name>` (expect 200/302;
+   first hit triggers dns-01 issuance, which works without inbound port 80).
+
 ## See Also
 
 - [[Pi-hole]] — the LAN DNS server (quadlet, records, web UI)

serverAppConfigs-docs/Infrastructure/Observability Provisioning.md

@@ -0,0 +1,95 @@
+---
+type: reference
+title: Observability Provisioning
+description: How Prometheus config/rules and Grafana datasources/dashboards are provisioned as code in serverAppConfigs — the config triad, deploy copiers, planner mappings, seam tests, entry points for each change type, and API verification.
+tags: [observability, prometheus, grafana, provisioning, deployment]
+timestamp: 2026-10-05T00:00:00-07:00
+---
+
+# Observability Provisioning
+
+The monitoring stack is **provisioned as code**: dashboards, datasources,
+scrape jobs and alert rules live in the repo and ride the normal deploy flow.
+No hand-edits inside containers — they are wiped on recreate and never
+committed.
+
+## The config triad
+
+| Repo path | What it holds |
+|---|---|
+| `shared/prometheus/prometheus.yml` | scrape jobs (global + per-host) |
+| `shared/prometheus/rules/llm-fleet.yml` | alert rules + recording rules |
+| `shared/grafana/provisioning/` | datasources + dashboards (provisioned tree) |
+
+Targets and datasources reference **container names** (`loki:3100`,
+`alloy:12345`), never loopback-published ports; the LLM endpoints are reached
+by LAN IP (`192.168.0.143:7770`, `:7771`, `192.168.0.235:7770`) because they
+are host-netns services, not containers. Prometheus 3 refuses HTML responses
+as metrics — HTML-scraped "probes" are structurally inert and need a blackbox
+exporter (see [[Stale Signal Audit]]).
+
+## The deploy path
+
+`scripts/deploy-remote.sh` carries two copier functions:
+
+- `copy_prometheus_config` → `/opt/prometheus/config/` on the target
+- `copy_grafana_provisioning` → `/opt/grafana/provisioning/`
+
+`scripts/deploy-plan.py` maps changes under `shared/prometheus/**` and
+`shared/grafana/**` to the `prometheus`/`grafana` services, so
+`deploy.sh dankclaw --service grafana` (etc.) restarts only what the change
+touched. Seam tests in `tests/observability_quadlet.bats` pin: unit files
+exist, the copier functions exist, and the provisioned paths match the units'
+mounts.
+
+## Entry points
+
+**Add a scrape target** — edit `shared/prometheus/prometheus.yml`. Decide the
+vantage first: container name if the exporter is a container on the same
+netns; host LAN IP if it is a host process (LLM endpoints); a blackbox module
+if it is an HTTP probe, not a metrics endpoint.
+
+**Add an alert/recording rule** — edit `shared/prometheus/rules/llm-fleet.yml`.
+Current delivered set: 5 alert rules + 1 recording rule for the LLM fleet.
+Rules reload with the prometheus service.
+
+**Add a datasource** — drop a YAML file into
+`shared/grafana/provisioning/datasources/`. Point it at the container name of
+the backend (`http://prometheus:9090`), not loopback.
+
+**Add a dashboard** — drop the JSON into
+`shared/grafana/provisioning/dashboards/`. Provisioned dashboards live in
+unified storage, not (only) the SQLite table — grep the API, not the DB, when
+verifying (see the verify section of [[Fleet Audit and Posture]]).
+
+## Verify through the APIs, not the filesystem
+
+```bash
+# prometheus: rules and targets actually loaded
+curl -s http://127.0.0.1:9090/api/v1/rules | python3 -m json.tool | head
+curl -s "http://127.0.0.1:9090/api/v1/targets?state=active" | \
+  python3 -c "import json,sys; [print(t['labels'].get('job'), t['health'], (t['lastError'] or '')[:60]) for t in json.load(sys.stdin)['data']['activeTargets']]"
+# grafana: datasource + dashboard provisioned
+curl -s -u admin:<pw> http://127.0.0.1:3002/api/datasources | python3 -m json.tool | grep -E '"name"|"url"'
+```
+
+Health answers from a stale process and provisioned resources can hide in
+unified storage — a file existing on disk is not proof the running server
+loaded it. See [[Stale Signal Audit]] for the never-green-target decision
+table (refused = outage, 404/HTML = config defect).
+
+## Current delivered state (2026-09-17)
+
+Prometheus → Alertmanager (`dankclaw/observability/quadlet/alertmanager.container`,
+loopback `:9093`) → host-side Signal relay (`scripts/signal-relay.py`, user
+unit, `192.168.0.143:18093` — the LAN IP, the one bind a container can reach
+via `host.containers.internal`) → signal-cli. Some targets read down for
+structural reasons (HTML-scraped "probes", Gitea metrics off); the remedy is
+the blackbox exporter. See [[Observability]].
+
+## Related
+
+- [[Observability]] — stack topology and the alert delivery chain
+- [[Stale Signal Audit]] — classifying down targets before "fixing" them
+- [[Quadlet Lifecycle]] — the units that host these services
+- [[Deploy State Triage]] — how the planner selects these services

serverAppConfigs-docs/LLM/DFlash DSpark Speculative Decoding.md

@@ -95,6 +95,51 @@ llama-server --hf-repo <target-GGUF> \
 
 The DFlash drafter proposes, then ngram-mod runs as an additional implementation in the chain. Verified against upstream `common/speculative.cpp` (b10133+).
 
+### Chaining laws (measured on this fleet, 2026-09 → 10)
+
+`--spec-type` takes a comma list, and the gate is `ctx_dft` in
+`common/speculative.cpp` (`common_speculative_init_result`): **draftless**
+types (the ngram family — simple, map-k, map-k4v, mod, cache) are added
+unconditionally, while **context-backed** types (`draft-mtp`, `draft-dflash`,
+`draft-dspark`, `draft-eagle3`) all require a draft CONTEXT. Consequences:
+
+- A chain may hold ONE context-backed drafter plus any number of ngram types.
+  `draft-dflash,ngram-mod` is legal and measured: 2.6x the drafter alone on
+  prompt-echo traffic, byte-identical to it on prose at the fleet's 48/64
+  window.
+- Two context-backed types cannot share one process: `draft-mtp,draft-dflash`
+  dies at load with `context type MTP requested but model doesn't contain MTP
+  layers` — MTP builds its context against whatever the draft context holds,
+  so it met the DRAFTER's file, not a missing head. Pin that in a seam so
+  nobody "fixes the flag" later.
+- Internal order is fixed (ngram types first, then drafters); CLI order is
+  irrelevant.
+- **A legal chain can still lose on a recurrent model.** `draft-mtp,ngram-mod`
+  on the qwen35 hybrid (frank-large era) measured −15 % prose / −7 %
+  block-copy (2026-10-03): adding the ngram layer logs `recurrent state
+  rollback is not compatible with 'ngram-mod' - disabling rollback support`,
+  and MTP verification depends on that rollback, so acceptance drops on ALL
+  traffic even where the matcher fires. Measure chains per architecture,
+  never by analogy.
+- ngram-mod prints NO startup banner — the load log shows only the drafter's
+  `adding speculative implementation` line. Never conclude "chain not
+  engaged" from the load log; identify behaviourally (`draft_n` > 0 only
+  where text repeats) or by the per-task `draft acceptance = ... mean len`
+  line. It costs ~64 MiB of host-RAM hash table.
+
+### Drafter KV: read the window policy from the header
+
+The draft context is built with `cparams.n_ctx = llama_n_ctx(ctx_tgt)`
+(`common/speculative.cpp`) — there is no flag to cap it — so a drafter that
+declares NO sliding window caches the whole target window on whatever device
+`--spec-draft-device` names. DFlash declares `sliding_window = 2048` (~40 MiB
+of draft KV); DSpark declares none: 5 layers x 2 x 8 kv heads x 128 dim x 2 B
+= 20 KiB per position, i.e. **5.0 GiB at 262144** — enough to kill the first
+draft graph capture with `ROCm error: out of memory` AFTER the model loaded
+and `/health` answered (measured 2026-09-16). Read `sliding_window` out of
+the drafter's GGUF header before sizing any spec-decode deployment, and
+prefer a skewed `-ts` over a context cut when the full window matters.
+
 DFlash on llama.cpp:
 
 ```
@@ -209,6 +254,151 @@ docker run --rm --name vllm-qwen36-dflash \
 
 History: before `--spec-default`, the box ran DSpark (`draft-dspark`, retired 2026-08-15 after a ROCm illegal-memory-access crash under two concurrent 16k prompts) and before that bundled MTP (`draft-mtp`, retired same day). DFlash was retired on 2026-08-14 (premature-EOS under parallel load). The old DFlash draft GGUF (`Qwen3.6-27B-DFlash-Q8_0.gguf`), the MTP-only drafter (`Qwen3.6-27B-MTP-ONLY-Q8_0.gguf`), and the DSpark draft (`Qwen3.8-27B-DSpark-Q8_0.gguf`) remain in `/home/dank/models/` as reference only.
 
+## A/B protocol on a live llama.cpp endpoint
+
+Verdicts are **traffic-shaped**: a proposer can only draft text it has seen,
+so a single prompt shape answers the wrong question. Use fixed shapes:
+
+| Shape | Prompt | Who it can favour |
+|---|---|---|
+| prose | 1k filler + "reply with one sentence" | model-backed drafters only |
+| echo | ~1.5k passage of UNIQUE lines + "repeat verbatim" | ngram (output IS the prompt) |
+| structured | repetitive output, no prompt overlap | ngram, if spans really repeat |
+| identity | short deterministic question | controls for lossless verification |
+
+Design the echo passage so no long span repeats INSIDE it — otherwise you
+measure the matcher against itself. For `structured`, give each line a unique
+id: repeated lines make it a negative control, not a repetition test.
+
+Protocol:
+
+1. Stop the unit (`sudo systemctl stop <svc>`), `pkill -x llama-server`; wait
+   for the port (a retired predecessor's published-port proxy leaves the
+   endpoint port in TIME_WAIT — the bind fails with address-in-use long after
+   the unit stopped).
+2. Per config: write an env file with the overrides, start the serve script
+   with `FRANK_*_ENV_FILE`, poll `/health`, measure. Ready-made harness:
+   `scripts/bench-spec-ab.py <env-file> <serve-script>`.
+3. **Restore the unit in a `finally`** and shout if it does not come back
+   healthy. A crashed harness that leaves production down is worse than no
+   harness.
+4. **Prove losslessness**: one fixed prompt at temp 0 to every config, diff
+   the greedy text. Speculation only buys speed; differing texts are a
+   correctness bug. Empty answers (`content: null` when reasoning ate the
+   budget) make the comparison vacuous — bench clients must fall back to
+   `reasoning_content`, and echo/structured prompts should disable thinking
+   per request: `chat_template_kwargs: {"enable_thinking": false}`.
+5. **Sweep the depth, not just the mode** — draft depth is model-specific and
+   single-valued; deeper often loses (measured: at n=7 a third of proposals
+   survived, at n=3 more than half).
+6. **Sweep the ngram window in the same run as the configs**: too short a
+   `--spec-ngram-mod-n-min/-n-max` proposes on ordinary prose fragments —
+   rejected, verification still paid (measured 43.9 → 36.0 tok/s).
+
+Read the right counters — per request `timings.draft_n` / `draft_n_accepted` /
+`predicted_per_second`; cumulative `/metrics`
+`llamacpp:spec_decode_num_drafts_total` / `..._accepted_tokens_total`;
+per-task `draft acceptance = ... (N accepted / M generated)`. `draft_n == 0`
+while tokens flow = the proposer is NOT engaged, whatever the flags say. A
+zero-draft config is a useful negative control: it proves the other config's
+drafts came from the proposer under test.
+
+Harness hygiene: absolute script paths (a relative path plus `cd` to its own
+dir double-counts the prefix); `pkill -x`, never `-f` (the pattern matches
+your own command line); nested f-strings with backslash escapes are a
+SyntaxError on the fleet's Python (<3.12 semantics) — build log lines with
+concatenation. Report `decode tok/s` per shape plus acceptance and mean
+accepted length; a config that wins on echo and loses on prose is a traffic
+decision, not a winner. Wire the winner into the deployed env file with the
+measured table as its justification comment; record losing numbers with "do
+not switch without re-measuring".
+
+The A/B vehicle for ANY single-GPU llama.cpp knob (batch/ubatch, KV type,
+slot split, quant rung — not just spec flags) is the transient systemd
+drop-in procedure in [[frank-large-benchmark]] and [[frank-small-benchmark]].
+
+## Is it worth it? (arithmetic before deployment)
+
+Answer against the INCUMBENT, not against zero.
+
+**Step 0 — check whether the target already speculates.** Many engines ship a
+drafter of their own; then your baseline is `A_incumbent / t_s`, not `1 / t_s`.
+halogen-flash-server: MTP head ON by default — read its own accounting, e.g.
+`serve_api: mtp 63 tok in 1.31s = 47.97 t/s | 33 rounds, commit 1.88/round`.
+llama.cpp: `/metrics` counters and per-request `draft acceptance` lines.
+
+**Step 1 — measure the two step times, never guess.** Target step `t_s` = 1 /
+(client-visible tok/s ÷ commit-per-round); ask for the rate the operator
+actually SEES (engine-internal decode-only numbers run ~1.7–2x optimistic).
+Drafter step `d_s` from llama.cpp's per-task `eval time` lines when it is a
+plain model on the same box.
+
+**Step 2 — arithmetic:**
+
+```
+round_time = K * d_s + 1.1 * t_s        # K drafts proposed, K+1 verified in ONE target pass
+A          = (1 - p^(K+1)) / (1 - p)    # expected accepted tokens, p = per-position acceptance
+rate       = A / round_time             # compare to the incumbent
+break-even : A > rate_incumbent * (K*d_s + 1.1*t_s)
+```
+
+Worked example that decided a real case: `t_s = 67 ms` (25 tok/s, 1.67
+commit/round), `d_s = 12 ms` (27B IQ3_S drafter on the 3090) → break-even
+`p ≈ 0.70` at K=3, `≈0.68` at K=5; a realistic cross-finetune `p = 0.4–0.6`
+gives 17–20 tok/s, i.e. a LOSS. A drafter only ~3x faster per token cannot
+pay for K sequential forwards.
+
+**Step 3 — classify the drafter by per-round cost:**
+
+| drafter | cost per round | needs |
+|---|---|---|
+| plain model (`-md`/`draft-simple`) | **K sequential forwards** | 10–50x faster than the target |
+| n-gram / prompt-lookup (`ngram-*`) | ~0, no model | repeated context (agent edit loops) |
+| trained head (`draft-mtp`, EAGLE/DFlash) | **~1 pass** | the head must exist FOR THAT CHECKPOINT |
+
+Trained heads are target-specific: EAGLE/DFlash consume the TARGET's hidden
+states, so a dense-27B DFlash2 is useless for a 512-expert MoE trunk.
+
+**Step 4 — acceptance reality checks.** Fleet datum: a trained z-lab DFlash2
+at depth 7 accepted ~41 % on its matched target (`draft-dflash 7 → 43.5
+tok/s`); treat 40–60 % as the honest band for a matched pair. Cheapest proxy
+for `p` at temp 0 when both models are greedy and share a tokenizer: greedy
+continuation agreement on identical prompts, two API calls, no weights moved.
+llama.cpp can also synthesize the projection without any draft artifact:
+`--spec-synth-rates P0,P1,...` / `--spec-synth-len L`. Speculation pays IN
+PROPORTION TO HOW SLOW THE TARGET IS — the same 41 % gave 2.3x on an
+18.8 tok/s layer-split target and a loss on a 25 tok/s one; fast targets are
+usually sparse MoE, the worst case for drafting.
+
+**Step 5 — check the attachment surface BEFORE the arithmetic.** Closed
+engines refuse external drafters by design: halogen-flash-server's
+`DRAFTER_CAP` gates `dflash2` on the loaded checkpoint and an unavailable
+name is a deliberate 400 (a silent downgrade would make comparisons lie);
+there is no device routing in its loopback token protocol, so a drafter can
+never sit on another card. Pure llama.cpp DOES route the draft:
+`--spec-draft-device/-devd` (default follows `--device`), `-ngld`,
+`-otd`, `-ctkd/-ctvd`, `--spec-draft-n-max` (default 3), `--spec-draft-p-min`.
+One process sees only the devices its backends were built with — CUDA+Vulkan
+is the combination that shows a CUDA card AND an AMD iGPU (HIP+CUDA cannot
+coexist); a CUDA-only build has no `Vulkan0` to draft with.
+
+**Step 6 — cross-model pairs need a byte-identical tokenizer.** Speculation
+compares token IDs, so draft and target must map id → same string. Verify
+cheaply from the GGUF headers (first ~8 MB over HTTP, no full download):
+parse `tokenizer.ggml.tokens`, print vocab size, merges, `pre`, and sha256
+over the first 2000 tokens —
+
+```
+vocab=248320 merges=247587 model=gpt2 pre=qwen35
+sha256(first 2000 tokens)=8619c68fe4466d3a   # identical => ids are interchangeable
+```
+
+Identity is necessary, NOT sufficient — the drafter still has to be cheap and
+accepted. Report in this order: does the target already draft; break-even
+`p` at K=1,3,7 versus the incumbent; the realistic band and resulting rate;
+whether the drafter fits (VRAM) and the attachment surface exists; the one
+measurement that would settle it.
+
 ## Notes
 
 - Speculative decoding helps low-QPS, memory-bound, latency-focused workloads. It helps less at high QPS.
@@ -225,4 +415,8 @@ History: before `--spec-default`, the box ran DSpark (`draft-dspark`, retired 20
 - [[Dankclaw System Specs]]
 - [[Dankserver System Specs]]
 - [[VLLM Overview]]
-- [[deployment]]
+- [[deployment]]
+- [[GGUF Compatibility]] — header parsing and custom tensor-type lock-in
+- [[Llama Cpp Serving Sizing]] — slot-pool and drafter KV sizing arithmetic
+- [[Llama Cpp Multi-Device Splits]] — where the drafter's KV actually lands
+- [[frank-large-benchmark]] and [[frank-small-benchmark]] — the single-GPU env drop-in A/B vehicle

serverAppConfigs-docs/LLM/FastFlowLM NPU Inference.md

@@ -271,6 +271,83 @@ graph TD
 6. Step up in size: 9B, then gpt-oss-20b.
 7. Attempt `qwen3.6-moe:35b-a3b` only after the smaller runs succeed, and only with the production service stopped. Watch `free -h` during the model download and load.
 
+## Registering packs absent from the catalog (NPU2)
+
+Use when adding a non-catalog NPU2 pack (an extraction or fine-tuned model)
+to the FLM registry. Registry facts: user registry
+`/home/dank/.config/flm/model_list.json`, shape
+`{"model_path": "models", "models": {"<tag>": {"<size>": {name, url,
+file_url, size, flm_min_version, files[], vlm, default_context_length,
+max_prefill_len, details{format:"NPU2", family, …}, label[]}}}}`; serve
+resolves tags through `FLM_CONFIG_PATH` at startup, and `flm list` with the
+env var set lists custom tags (`⏬` = not downloaded).
+
+1. Back up the registry JSON first (`cp -a`).
+2. Insert the entry via a python json edit + atomic `os.replace`. Set
+   `files` to the FULL HF file list — **including `vision_weight.q4nx` even
+   for text-only packs**: the qwen3.5-family loader opens it unconditionally
+   and a missing file makes the first request HANG silently (no error log).
+3. **Do not use `flm pull <tag>` for custom entries** — it throws
+   `json.exception out_of_range key '<tag>:<size>' not found` while building
+   the download list. Place files manually into
+   `/home/dank/.config/flm/models/<PackName>/` with
+   `curl -sfL "https://huggingface.co/<repo>/resolve/main/<file>?download=true"`
+   — plain `raw/main/<file>` returns 135-byte LFS pointer stubs, not
+   weights; verify byte counts against the HF tree API before trusting a
+   download.
+4. Kernel packs: copy from the same-shape family pack (shape-match
+   `config.json` on hidden_size/num_hidden_layers/head_dim first) into
+   `/opt/fastflowlm/share/flm/xclbins/<PackName>/` via `sudo cp -a`
+   (root-owned dir).
+5. Deploy/restart the serve unit (the deploy path, not ad-hoc restarts) so
+   the new process loads with all files present.
+6. Verify with a REAL completion request (`model: "<tag>:<size>"`) —
+   `/v1/models` alone proves registration, not loadability.
+
+Hard constraints: only ONE `flm serve` process can hold the NPU hardware
+context — a second concurrent serve process fails every load with
+`DRM_IOCTL_AMDXDNA_CREATE_HWCTX err=-22` (aie2 mgmt msg opcode 0x2 status
+0x2000003); never run scratch/verify instances while production holds the
+NPU. Tag switching happens INSIDE one serve process; the first request
+after a model switch pays ~6–11 s cold-load cost (visible in
+`usage.load_duration`). Over ssh, never `pkill -f "<pattern>"` with a
+pattern present in your own command line — use bracket-guarded patterns
+(`[q]wen3.5:4b`) or exact PIDs.
+
+## Tag load-failure diagnosis (200 with an error body)
+
+`POST /v1/chat/completions` answers **HTTP 200** with a flat
+`{"error":"Failed to load <tag> model!"}` and no `choices`; `/v1/completions`
+returns a nested OpenAI-shaped `server_error` (code 500) also under HTTP
+200. A status-code-only client reads this as a successful empty completion —
+**always check for the `error` key**. Two distinct causes, distinguished via
+`journalctl -u <flm-service> --full`:
+
+1. **Unknown tag** (not in the registry): immediate error, nothing loaded.
+   Correct behavior, not a bug.
+2. **Registered tag whose kernel pack is missing**: FLM resolves the tag,
+   downloads missing weight files on the fly (first request hangs ~90–110 s),
+   then fails `Failed to load model: No such file
+   '/home/dank/.config/flm/xclbins/<PackName>/<kernel>.xclbin'`; subsequent
+   attempts fail fast (~2 s). Root cause: the serve script exports
+   `FLM_XCLBIN_PATH=/home/dank/.config/flm/xclbins` (user dir), so every
+   kernel lookup resolves under the user dir, which only contains symlinks
+   someone linked. **A tag listed in `/v1/models` is NOT proof it is
+   loadable on this host.** Fix: one symlink per missing pack
+   (`ln -s /opt/fastflowlm/share/flm/xclbins/<PackName>
+   /home/dank/.config/flm/xclbins/<PackName>`) — verified live: after the
+   symlink the first request loaded and answered in 7.4 s wall.
+
+Switch-cost behavior: the `model` field routes per-request (no restart
+needed); a request whose model differs from the resident one triggers a full
+reload (7–11 s wall for an 8-token completion on ~4B-class packs);
+alternating tags pays the swap each time. Single-lane NPU: concurrent
+requests queue FIFO behind the current lock holder (`NPU Locked!` →
+`NPU Lock Released!`). Probe recipe: `/v1/models` for registered tags; grep
+the journal for `Failed to load model` (the quoted xclbin path names the
+missing pack dir exactly); diff `ls ~/.config/flm/xclbins/` (symlinks =
+installed) against `/opt/fastflowlm/share/flm/xclbins/` (available).
+
 ## Incident: qwen3.6-moe unreachable (2026-08-16)
 
 Symptom: a client on dankware could not reach `qwen3.6-moe:35b-a3b` on port 52625. The journal shows three separate causes in the 14:19-15:03 window.

serverAppConfigs-docs/LLM/FreeToken Builds.md

@@ -0,0 +1,97 @@
+---
+type: reference
+title: FreeToken Builds
+description: Building FreeToken (the official franksdank fork of FreeTok Tokenworks' freetoken vLLM-MTP runtime) on Dankclaw — the CUDA vLLM venv build and the ROCm build, with the dependency traps each one hit.
+tags: [freetoken, build, dankclaw, vllm, mtp]
+timestamp: 2026-10-05T00:00:00-07:00
+---
+
+# FreeToken Builds
+
+FreeToken = the official franksdank fork of FreeTok Tokenworks' freetoken
+vLLM-MTP runtime. Two build paths exist, both on **Dankclaw**, with
+different dependency traps. Both were verified end-to-end 2026-09-11/12.
+
+## Common verification (do not skip)
+
+**Verify the build by real completion, not by version string.** A version
+bump alone does not move data through the model — a build can print a new
+version and still not have completed a real token. Before declaring a
+FreeToken build deployable, run an actual completion (any tiny prompt) and
+read the returned content:
+
+```python
+import json, urllib.request
+req = urllib.request.Request("http://<host>:<port>/v1/chat/completions",
+    json.dumps({"model": "frank-small", "messages": [{"role": "user",
+    "content": "Hi"}]}).encode(), {"Content-Type": "application/json"})
+print(json.load(urllib.request.urlopen(req, timeout=600)))
+```
+
+A 200 with a real `choices[0].message.content` is the only build green.
+
+## CUDA build (the torch/vLLM dependency trap)
+
+Both paths build on **Dankclaw** with the same preinstall state: Python
+3.12.10 in the system venv and **Torch 2.10.0.dev** (nightly, repo
+download.pytorch.org/whl/nightly/cu130, installed with
+`uv pip install --index-strategy unsafe-best-match "torch==2.10.0.dev<date>+cu130"
+--index-url https://download.pytorch.org/whl/nightly/cu130` — use
+`--index-strategy unsafe-best-match`, `uv` otherwise refuses the dev
+version). Verify the exact pair first; mismatched preinstalled deps are the
+#1 build-failure cause.
+
+Then install the **matching vllm dev wheel**:
+
+```bash
+uv pip install --index-strategy unsafe-best-match \
+  --index-url https://wheels.vllm.ai/<commit-sha>/vllm-0.6.6.dev0+g<shortsha>.cu130-cp312-cp312-linux_x86_64.whl \
+  --extra-index-url https://wheels.vllm.ai/nightly/cu130
+```
+
+The wheel URL pattern is fixed:
+`https://wheels.vllm.ai/<full-commit>/vllm-0.6.6.dev0+g<8-char>.cu130-cp312-cp312-linux_x86_64.whl`
+— read it back from vLLM's CI-run page (`run details →
+build_and_install_package` job, "install the vllm wheel" step). **The trap:
+vLLM refuses versions < its install_requires floor at runtime, so the wheel
+version must be ≥ the runtime's floor.** A wheel built at a lower base
+(0.5.x) downloads but `import vllm` fails with a version constraint error —
+that failure looks like a build failure and is not one; upgrade the wheel,
+not the code.
+
+If `uv pip install -e . --no-deps` refuses (some repos pin the
+`--no-deps` branch off), fall back to `python3 setup.py` inside the venv,
+after `python -m venv --copies .venv && source .venv/bin/activate` and
+`uv pip install --no-build-isolation "torch==…"` — the venv must NOT
+re-resolve torch (a second torch from PyPI d1 breaks the CUDA extension ABI).
+
+## ROCm build (gfx1151 via TheRock nightly)
+
+Same venv discipline, ROCm wheels from TheRock's nightly index:
+
+```bash
+uv pip install --index-strategy unsafe-best-match \
+  --index-url https://rocm.nightlies.amd.com/v2/gfx1151-dcgpu-supports_fbsgens_split_0717-rc1/ \
+  "torch==2.10.0.dev<date>+rocm" "triton==3.6.0" "pytorch-triton==3.6.0"
+```
+
+Then `vllm` itself (`uv pip install -e . --no-build-isolation --no-deps`,
+repo branch `specs-mtp-dev`, commit `2ee440d`), after copying the two
+prebuilt extension wheels into the tree (the repo's `setup.py` skips
+compiling them when found). The ROCm build also needs
+`install-fastrak-gfx1151.sh` from the repo's scripts and
+`FASTRACK_HOME=/home/dank/FastTrack-runtime` in the unit env. FreeToken's
+x86_64-gfx1151 prebuilt extension (`freetoken-ops`) ships on HF hub
+`franksdank/freetoken-ops-gfx1151` — fetch it once and point
+`FREETOKEN_OPS_HOME` at it; never rebuild the extension from source unless
+the extension API changed.
+
+`--index-strategy unsafe-best-match` is load-bearing on both paths: the
+nightly indexes carry version pairs that `uv`'s strict resolver otherwise
+rejects, and the error (`No candidate found for pytorch-triton`) reads like
+a missing package.
+
+## Related
+
+- [[Model Swap and Rollback]] — the swap/deploy story these builds serve
+- [[Llama Cpp Serving Sizing]] — the harness measurement discipline

serverAppConfigs-docs/LLM/GGUF Compatibility.md

@@ -0,0 +1,123 @@
+---
+type: reference
+title: GGUF Compatibility and Runtime Lock-in
+description: How to answer "will engine X load this GGUF?" from the file's header alone, why a custom tensor type forces the runtime, and the discipline for pinning a fork build that can actually serve the file.
+tags: [gguf, llama.cpp, forks, quantization, speculative]
+timestamp: 2026-10-05T00:00:00-07:00
+---
+
+# GGUF Compatibility and Runtime Lock-in
+
+**The law: a GGUF's tensor-type histogram and ftype pick the runtime — read the
+header before promising a file to any loader.** Stock llama.cpp, Ollama and LM
+Studio refuse custom types at load (`unknown tensor type`); a fork-private type
+means exactly one engine can serve the file. Measured on this fleet repeatedly:
+`Qwen3.8-27B-Q8_0_ROCMFPX_AGENT.gguf` is ftype 115 with 311 tensors of type 103
+(`GGML_TYPE_Q8_0_ROCMFPX`) — only charlie12345/ROCmFPX reads it. The full type
+tables live in [[Llama Cpp Runtimes]] and [[Llama Cpp Code Diff]]
+(ggml_type 100–108, ftype families 100–106 / 110–113 / 118).
+
+## 1. Read the header, do not guess
+
+Parse the GGUF KV section and histogram the tensor types plus
+`general.architecture.file_type` — one python read over the first few MiB, no
+model load. Counting types out of the KV section settles a runtime argument in
+minutes; repo names and model cards do not.
+
+## 2. Compare routing variants by tensor mix, not by name
+
+Two files of the same quant can differ in how many tensors stay at plain Q8_0
+(measured: 187 vs 2 output-head tensors in an `_AGENT` variant of the same
+size). That difference is what moves spec-decode acceptance (0.953 vs 0.911)
+— the "same size" file is not the same file.
+
+## 3. Pin the build by commit SHA
+
+A fork that can read the file today can still crash mid-load: the 2026-08-25
+ROCmFPX builds read the ftype-115 file and then SIGSEGV in
+`ggml_compute_forward_set_rows`. Record the SHA in the build script
+(`scripts/build-rocmfpx.sh` pins `36e9acd`) and check it out explicitly — never
+a branch.
+
+## 4. Header diff method (candidate vs known-good)
+
+Answer "will engine X load this GGUF?" by diffing the candidate's header
+against a file the engine **already loads** — never by filenames or model cards;
+filenames lie about quantization and about MTP/vision heads.
+
+1. **Get the bytes**: both files, from byte 0, at least 64 MiB. Tokenizer KVs
+   (151k token strings + merges) routinely eat 5–8 MiB, so a small range
+   desynchronises the parser. True size from `curl -sIL` → `x-linked-size`
+   (a bare HEAD `Content-Length` can lie):
+
+   ```sh
+   curl -sL -r 0-67108863 -o cand.bin \
+     "https://huggingface.co/<owner>/<repo>/resolve/main/<file>.gguf"
+   ssh <host> 'head -c 67108864 /path/to/known-good.gguf' > good.bin
+   ```
+
+2. **Parse — and lock the parser against the known-good file first.** Layout
+   (v2/v3): `magic[4]` `version u32` `n_tensors u64` `n_kv u64`, then KVs, then
+   tensor infos. Strings are u64 length + bytes. **The value-type enum is NOT
+   llama.cpp's `gguf.h`:** in the files seen (2026-09-28, unsloth + ajgazin UD
+   GGUFs) type **8 = STRING, 9 = ARRAY**; llama.cpp's header has STRING=2,
+   ARRAY=3 and desynchronises immediately. Working table:
+
+   ```
+   0 u8   1 i8   2 u16  3 i16  4 u32  5 i32  6 f32
+   7 bool 8 STR  9 ARRAY(u32 et, u64 n)  10 u64  11 i64  12 f64
+   ```
+
+   Validate before trusting: the parse is right iff the tensor-info block ends
+   somewhere sane and names look real (`blk.0.attn_q.weight`,
+   `token_embd.weight`, `output_norm.weight`). EOF throw = wrong enum or
+   offset — do not work around it.
+
+3. **Diff what engines actually bind on**: `general.architecture`; tensor
+   count; tensor name list in order; per-tensor `ggml_type` and dims; geometry
+   KVs (`<arch>.block_count`, `.embedding_length`, `.feed_forward_length`,
+   `.attention.head_count[_kv]`, `.rope.*`, `.ssm.*`, `.context_length`,
+   `.nextn_predict_layers`); MTP tensors (`nextn` in the name); vision tensors
+   (`visual`, `v_` prefixes). A same-family fine-tune quantized by the same
+   tool usually comes out byte-identical in every one of these — that is the
+   answer.
+
+4. **Then read the engine's source before calling a delta a blocker.** A
+   metadata difference only matters if the engine reads that key
+   (`grep -rn "file_type\|recurrent_layers\|full_attention_interval" <src>/`).
+   Worked example (gufo v0.1.1): `general.file_type` is deliberately not
+   consulted (UD-*_K_XL ships a stale ftype 15; the tensor table rules), and
+   `qwen35.attention.recurrent_layers` is never read (layer kinds come from
+   `full_attention_interval`) — both deltas were non-issues the header diff
+   alone would have flagged wrongly.
+
+5. **Say what you did not do.** Header analysis is strong evidence, not a
+   load. State byte sizes, check free disk before proposing a fetch, and do
+   not download tens of GB unasked — pointing the engine at the file is the
+   definitive test.
+
+**Report shape:** verdict first (loads / will not / loads-with-caveats), then
+the side-by-side diff table, then only the load-bearing caveats: text-only vs
+vision tensors, chat-template differences (matters for llama.cpp, not for
+engines with compiled-in Jinja), license, disk headroom, and whether the base
+model is abliterated/uncensored.
+
+## 5. A/B a fork's spec decode by stripping the whole flag set
+
+`--model-draft` without `--spec-type` segfaults at load (measured 2026-10-02,
+core dump, no diagnostic). A baseline "no-draft" arm must remove `--model-draft`
+AND every `--spec-draft-*` AND `--spec-type` together — the flags move as a set.
+
+## 6. Before deleting a retired payload, read the rollback tag's env file
+
+A rollback may need a small companion dir: a drafter living in a directory
+named after an old quant looks like trash but is load-bearing. Same class as
+the gufo payload removal of 2026-10-03, which turned that rollback into a
+re-download.
+
+## Related
+
+- [[Llama Cpp Runtimes]] — which fork ships which custom types
+- [[Llama Cpp Code Diff]] — the enum tables and code-level differences
+- [[frank-large-rocmfpx-q8agent-mtp]] — the ftype-115 lock-in in production
+- [[Llama Cpp Serving Sizing]] — pricing what a payload costs once it loads

serverAppConfigs-docs/LLM/Heterogeneous Split Triage.md

@@ -0,0 +1,159 @@
+---
+type: reference
+title: Heterogeneous Split Triage
+description: Triage before building when someone asks to split a model across unequal accelerators on one host (24 GB GDDR6X + LPDDR APU, PCIe x4 link) — link characterisation, the real bottleneck, what placement never buys, UMA pools, drafter splits, and the verdict ranking.
+tags: [heterogeneous, multi-device, llama-cpp, triage, pcie]
+timestamp: 2026-10-05T00:00:00-07:00
+---
+
+# Heterogeneous Split Triage
+
+Produced the 2026-09-16 no-go verdict on splitting Qwen3.8-Flash-Next across
+the RTX 3090 + Strix Halo pair. Use it to answer "could we split model X
+across device A and device B" with arithmetic instead of enthusiasm.
+
+## 1. Characterise the link (bytes vs latency)
+
+```bash
+lspci -vv -s <bdf> | grep -E "LnkCap:|LnkSta:"          # capability vs actual
+nvidia-smi --query-gpu=pcie.link.gen.max,pcie.link.gen.current,pcie.link.width.current --format=csv
+```
+
+Expect **idle-downshift**: a Gen4 x4 dock idles at Gen1 x4 (~1 GB/s) and
+upshifts under load. Measure while the device is busy, or you size the design
+against the wrong number. Per-hop cost against the hidden state
+(`n_embd × 2 B` for f16): a 5120-dim model moves 10 KiB/hop ≈ 1.3 µs at
+Gen4 x4.
+
+- **Bytes per token ≈ hops/token × hidden state** — typically ~1 MiB/token
+  for 2 hops/layer × 64 layers ≈ 0.3 % of a 40 ms token. **Bytes are almost
+  never the constraint.**
+- The real budget line is **hop COUNT**: every device boundary is a
+  host-side sync. A runtime that splits per *op* hits ~100 splits/token and
+  pins the GPU at ~5 % SM. Measure `graph splits` in the loader log; never
+  assume.
+- **Design rule: partition by where the weights live, never by where the
+  activations flow.** Resident weights, activations crossing.
+
+## 2. Get the workload's real bottleneck (KV is usually it)
+
+```
+KV/token = kv_bytes_per_position × context
+```
+
+Attention reads the whole cached context every token, so KV/token scales
+with context — compare it against expert/weight traffic per token and total
+token time. At long context on a hybrid arch (few attention layers, many
+positions), **KV dominates**. Consequences:
+
+- The highest-leverage placement is **KV cache on the fastest-memory device,
+  together with its attention compute** — never KV alone: separating them
+  puts KV/token over the link, which is fatal (GBs across 7.88 GB/s =
+  hundreds of ms/token).
+- Check the fit: `kv_positions_per_slot × slots × bytes_per_position +
+  attention_trunk ≤ VRAM`.
+
+## 3. What placement never buys
+
+Prefill is compute-bound and expert weights are read once per batch, so
+moving them to faster memory buys nothing while adding a serial stage.
+Expect a regression, and size it from calibration: on Strix Halo + RTX 3090
+(Qwen3.8-Flash-Next, ik_llama.cpp) moving 12.5 % of the expert mass to the
+GPU gave **decode +12 %, prefill 192 → 32 tok/s**. A
+`1/(f_big/BW_slow + f_fast/BW_fast)` harmonic calculation reproduces that
+decode gain to within ~2 points — usable for decode projections, and it
+assumes sequential stages (true at batch 1; no pipelining).
+
+## 4. Is the second device even a separate pool?
+
+On a UMA APU, check before assuming additivity:
+
+```bash
+cat /sys/class/drm/card*/device/mem_info_vram_total   # BIOS carve-out (small)
+cat /sys/class/drm/card*/device/mem_info_gtt_total    # aperture onto SYSTEM RAM
+rocm-smi --showmeminfo all | grep -i gtt              # used == what the CPU also sees
+```
+
+A large GTT window means the "GPU memory" is the *same* LPDDR under one
+memory controller: adding a stage there adds a compute engine, not capacity
+or bandwidth. Only the discrete card's VRAM is additive. Fencing more
+(`amdgpu.gttsize`, BIOS UMA size) just removes bytes from CPU-visible RAM.
+
+## 5. Speculative decoding as the "cheap" split
+
+A drafter on the *other* device is bandwidth-free (token ids per round, tens
+of bytes), so it is tempting. Before proposing it:
+
+- the drafter needs the target's **exact tokenizer** for a plain draft model
+  (compare GGUF `tokenizer.ggml.tokens`, never trust the family name), or it
+  must be a head **trained for that target** (EAGLE/DFlash/MTP are
+  conditioned on the target's hidden states and are not portable between
+  models);
+- a plain drafter costs **K sequential forwards per round**, so it must be
+  10–50× faster than the target. Break-even with `r = t_draft/t_target`:
+  `rate_ratio = A/(K·r + 1)`, `A = (1−p^(K+1))/(1−p)`; at `r ≈ 0.4–0.5`
+  (a drafter only ~2× faster) break-even needs `p ≈ 0.7+` at K=3;
+- compare against the **incumbent**, not the no-draft baseline: a target
+  already running its own MTP head is a high bar;
+- check the runtime accepts an external drafter at all — closed engines may
+  enumerate a fixed drafter set and 400 anything else (capability gates
+  keyed on the loaded checkpoint). That is a design boundary, not a config
+  gap. In llama.cpp the portable mechanism is `--spec-draft-device/-devd`
+  (draft gets its own device list) plus `-ngld`/`-otd`/`-ctkd -ctvd`.
+
+## 6. Expressing a split (llama.cpp, the usual vehicle)
+
+```bash
+-dev CUDA0,Vulkan0 -sm layer -ts F0,F1        # contiguous layer ranges (simplest)
+-ot "\.attn_.*=CUDA0" -ot "\.exps\.=Vulkan0"   # per-tensor, by ROLE (KV co-location)
+```
+
+- buffer types match by name at load (`ggml_backend_buft_name`), so device
+  names work as `-ot` values;
+- `-cmoe`/`-ncmoe` are **CPU-only**; device-side MoE placement is `-ot`;
+- several backends in one binary is fine (CUDA + Vulkan is the practical
+  pair; HIP + CUDA do not coexist) — `GPU_BACKEND_A=OFF` in the existing
+  build's CMakeCache tells you what needs adding;
+- contiguous ranges are the WRONG split when KV bandwidth dominates: they
+  drag half the attention layers (and their KV) back to the slow pool.
+
+## 7. Hazards checklist
+
+- Paged/sidecar tables (huge n-gram/embedding tables) must stay where they
+  are read from; moving them is pure loss.
+- Keep `mmap` on for models with huge embedding tables (anonymous memory
+  gets the process cgroup-killed).
+- CUDA graph capture can abort on unusual graph shapes (hyper-connections)
+  — check before assuming optimisations apply.
+- A brand-new backend on the second device may not implement the arch's ops
+  at all: make the first measurement a **baseline of the runtime on that
+  device alone** — a split's arithmetic is worthless if the baseline is
+  worse than the incumbent engine.
+- A split consumes the fast device entirely — check what else it was
+  serving before trading it away.
+- Benchmark discipline on production hosts: stop and mask first
+  ([[Fleet Ops Playbook]]).
+
+## 8. Verdict shape
+
+Rank and report; prefer the boring answer when it wins:
+
+1. **Two independent endpoints** (zero shared traffic) — often the real
+   answer to "what's a useful combination", especially over a narrow link.
+2. **One process, weights-resident, KV co-located with attention on the
+   fast pool** — promising and quantifiable (+% decode at long context);
+   needs a dual-backend build and a baseline measurement.
+3. **Hot-expert cache / per-token weight streaming** — affordable only if
+   the miss set per token is small (a few experts; check
+   `misses × bytes × tok/s` against link bandwidth) and overlapped; injects
+   PCIe latency into the critical path; no mainstream runtime implements a
+   dynamic expert cache.
+4. List explicitly what does NOT help: TP/collectives across unequal
+   devices, KV on the far device, model loading over the link,
+   prefill-motivated placements.
+
+## Related
+
+- [[PCIe Link and Atomics]] — the link and collective walls underneath
+- [[Llama Cpp Multi-Device Splits]] — the deployed split shapes
+- [[Llama Cpp Serving Sizing]] — KV and pool arithmetic

serverAppConfigs-docs/LLM/LLM Bench Windows and Measurement.md

@@ -0,0 +1,271 @@
+---
+type: reference
+title: LLM Bench Windows and Measurement
+description: Stopping-and-restoring benchmark windows on the fleet LLM hosts (llama-bench sweeps, HTTP param benches, multi-arm vLLM offload sweeps), honest benchmark methodology, and the usage-counter measurement law under speculative decoding.
+tags: [benchmarking, llama-bench, ab-testing, speculative, vllm, fleet]
+timestamp: 2026-10-05T00:00:00-07:00
+---
+
+# LLM Bench Windows and Measurement
+
+Fleet law first: **STOP the production service before any GPU experiment**
+(AGENTS.md Benchmarking Policy) — a bench beside a live service starves the
+GPU and corrupts results; an unisolated GPU experiment has precedent for
+wedging a host until power cycle. Reference harnesses:
+`scripts/bench-batch-ubatch.sh`, `scripts/autoresearch.sh`,
+`scripts/autoresearch/frank-primary-arms.sh`.
+
+## 1. The window structure (one config per arm, a driver owns the window)
+
+1. **Stop + gate before every arm** — `wait_for_clear`: unit NOT active AND
+   no RUNNING container of that name (`podman ps`, not `-a`; `-a` counts
+   EXITED containers and can never clear — `rm -f` the exited one first) AND
+   port free (`ss -ltn`). The graceful-stop race cost a whole arm 2026-10-01:
+   `ExecStop=podman stop -t 120` drains up to 2 min; a driver that sleeps 3 s
+   then polls `/health` gets **200 from the still-running PRODUCTION engine**
+   — the arm "passes" with production's boot log, then `podman exec` dies
+   with `container state improper`.
+2. **Boot the arm as the service, not a scratch port** (no scratch ports on
+   LLM hosts — one card, one server). The serve script takes its config via
+   env-file override (`FRANK_SMALL_ENV_FILE`, `FRANK_PRIMARY_AR_ENV_FILE`)
+   pointing at a staged copy; the checkout is never mutated.
+3. **Fast-fail the health wait**: poll `podman ps | grep -qx <name>` inside
+   the loop — a refused boot dies in one poll instead of 20 min. Capture the
+   serve script's FULL stdout on failure (a `tail -8` loses engine-init root
+   causes; raise happens above the tail).
+4. **Measure** (§4), capture boot-log engine facts (KV pool, arena slots,
+   SpecDecoding lines) — they survive measure crashes.
+5. **Teardown + converge**: kill by explicit PID/pgid (never pattern kills),
+   `podman rm -f` scratch, wait `MemAvailable ≥ ~230 GiB` on dankserver
+   before the next boot; restore production from the EXIT trap and verify the
+   **conjunction oracle**: health 200 + `is-active` + `is-enabled` +
+   mask-symlink clean. A bare `systemctl start` is not a restore oracle.
+   (`systemctl mask` FAILS on a relinked unit — log it, explicit stop is
+   enough: systemd never auto-restarts an explicitly stopped unit.)
+6. **Warm boots cycle ~2 min** (page cache), cold ~8 min — size timeouts for
+   mixed speeds.
+
+**Destructive tails (2026-10-01):** a scratch script whose tail does
+`systemctl stop` + `podman rm -f <PRODUCTION container>` fires later if a
+probe loop survived the driver kill — it can destroy a just-deployed
+production engine (signature: deploy exits green, then seconds-to-minutes
+later `Exited (0)`, `Deactivated successfully`, clean exit = removed, not
+crashed). Before ANY deploy: `pgrep -af` every scratch script of the session
+and kill strays by explicit PID; killing the dev-side job does NOT kill the
+host-side `deploy-remote.sh` (two of them once managed the same unit).
+
+Two failure shapes inside the window that read as "the bench died":
+
+- **Stale reads from a stopping container.** `podman stop` answers while the
+  engine inside is still flushing; a `podman logs` taken in that window
+  returns the boot-prefix only. Poll `podman ps` for `Exited (0)` (or
+  `podman wait`) before reading logs, or you will write "no results" into
+  the table for a run that has them.
+- **A D-state worker wedges the stop.** A bench child that wedged in the
+  kernel ignores SIGTERM, so `podman stop -t N` burns its full timeout and
+  then the container is `Stopping` forever. Kill by container first
+  (`podman kill <ctr>` → SIGKILL path), verify the process is gone from
+  `podman ps` AND `nvidia-smi`, and only then re-serve. Never let the
+  window script proceed to a redeploy with a zombie bench worker holding
+  VRAM.
+
+## 2. llama-bench sweeps (offline batch/ubatch/split/KV/depth)
+
+**What llama-bench cannot express — decide BEFORE designing:** no
+`--ctx-size` (`n_ctx = n_prompt + n_gen + n_depth`, so a bench can NEVER prove
+activation buffers fit beside the DEPLOYED KV — only the real unit answers
+that), no `--parallel` (no slot shape, no concurrency), no `--spec-*`
+(`tg` numbers are the drafterless baseline, useful as a control only), and a
+warmup run is itself a full-size run. `-d N` is a depth PREFIX (paid once per
+cell via a `llama_state_seq_*` save/restore); without it a `tg` run is at
+~zero depth and meaningless for long conversations.
+
+**The separator trap:** `/` separates devices WITHIN one configuration, `,`
+separates sweep values — for both `-dev` and `-ts`. `-ts 36,29` parses as the
+single value 36 → everything on one card. `--tensor-split 36,29` (server) →
+`-ts 36/29` (bench). Ranges exist: `-ub 512-2048+256`.
+
+**Harness shape:** one invocation per cell (an infeasible cell calls
+`exit(1)` and kills every later cell of a cross-product invocation — and
+infeasibility is usually the question); TSV row per cell as it completes +
+`bench.json` (`-o json`) + `bench.log` (`--progress`) + a 2 s `rocm-smi
+--showmeminfo vram` VRAM trace per cell (a cell that allocated less than
+intended looks fast); resume by skipping non-empty cell outputs; stop the
+service in the harness, restore from `trap … EXIT`, verify afterwards. Match
+the deployment's load flags exactly (`-ngl -sm -ts -fa -ctk/-ctv -t`).
+
+**Run under a transient unit, not a shell:**
+
+```bash
+sudo systemd-run --unit=<name> --collect --no-block --property=Type=oneshot \
+     --property=LimitMEMLOCK=infinity --property=User=dank \
+     /home/dank/serverAppConfigs/scripts/<harness>.sh
+```
+
+Deploy units set `LimitMEMLOCK=infinity` for ROCm staging; an SSH shell has
+8 MiB and fails the load for an unrelated reason. This also survives ssh
+disconnects. **Boot rungs under the production environment, not a bench
+shell** — a rootless shell probe booted 3×262144 bf16 that production refused
+(systemd `User=` + memlock change what the pinned-expert path holds):
+`sudo systemd-run --uid=<user> --unit=<name> -p LimitMEMLOCK=infinity -p
+Restart=no env VARFILE=… /path/serve.sh`.
+
+**Verify before burning hours:** dry-run against a STUB binary with
+`SERVICE=<nonexistent-name>` (the trap/stop never reached), tiny grid,
+including one forced-`exit 3` cell — prove the TSV row, failure capture,
+resume logic. Validate the real flag set in ~1 s by pointing `-m` at a
+nonexistent file. **Reading:** bracket the sweep (reference cell first AND
+last); check per-cell `stddev_ts` (a difference smaller than spread is not a
+result); single-sequence decode is usually flat across batch/ubatch (prefill
+knob); batch ≥ ubatch (llama.cpp clamps ubatch to batch, silently duplicating
+smaller-batch cells); report what the grid cannot prove (drafter, slots, real
+context).
+
+## 3. HTTP benches of server-side parameters
+
+`llama-bench` benches the bench binary, not the running server — server-side
+params (`--threads-batch`, `--ubatch-size`, `--spec-type`, `--parallel`) need
+HTTP:
+
+- **Prefill**: large prompt, `max_tokens=1` (server crashes on 0), measure
+  `prompt_tokens / wall_time`.
+- **Decode**: `max_tokens=16384` minimum, NEVER 1 (warmup, cache growth and
+  spec acceptance all stabilize over thousands of tokens); measure
+  `predicted_per_second` from `timings` or `completion_tokens / wall_time`;
+  temperature 0.
+
+Known fleet harnesses: `scripts/bench-threads-batch-http.py`,
+`scripts/bench-decode-threads-batch.py`, `scripts/bench-spec-decode.py`.
+sglang-specific: `max_tokens=1` on `/v1/chat/completions` with the qwen3
+reasoning parser streams zero deltas — use `/v1/completions`.
+
+## 4. The measurement law (under speculative decoding this is absolute)
+
+**Never count SSE chunks.** Rejected drafts break the 1:1 chunk↔token
+mapping; chunk counting reported 8.3 tok/s for a config whose server counters
+said 23–26 — a 3x false regression that nearly abandoned the change that
+worked. Always:
+
+```python
+decode_tok_s = usage.completion_tokens / (wall - ttft)
+# stream=True, stream_options: {include_usage: true}; ttft = first content-bearing chunk
+```
+
+vLLM's 10 s `Accepted throughput` log lines are a rolling window, not a
+measurement (6.2 → 13.1 across adjacent windows on identical settings) — use
+them for acceptance STATISTICS only (`Avg Draft acceptance rate`, `Mean
+acceptance length`, per-position rates). Warm pass + ≥3 runs, median;
+temperature 0; ≥256 tokens. **A decode "jump" with early-stopped completions
+(ct < 256) and acceptance > 90 % is degenerate generation** (the fork's
+repetition-halt fired at ~140 tokens on corrupted numerics) — inspect text
+before believing a win. Run-to-run spread is 5–10 %: repeat arms; one sample
+nearly shipped the wrong setting.
+
+## 5. Honest methodology
+
+- **Match the corpus to the workload.** Token soup under-reports speculative
+  decoding (2.9–3.6 tok/update vs 3.86–5.8 on real code) and mis-ranks draft
+  settings; acceptance, not step time, moves throughput. Build the corpus
+  from the repo's own sources: `{"prompt": "<instruction>\n\n<file body>"}`,
+  `--custom-output-len`.
+- **Salt every run** with a unique per-arm token in the first bytes — it is
+  the prefix-cache key (and a proxy routing key). Without it arm 2 measures
+  arm 1's warm cache.
+- **Two workload shapes need two clients:** distinct prompts (batch
+  behaviour, prefill parallelism) and sequential turns with growing history
+  (TTFT of the first streamed chunk per turn — the only measurement that
+  exposes prefix-cache behaviour). These pull in opposite directions: a cold
+  batch spreads prefill across replicas, a conversation's next turn only pays
+  if it lands where its prefix is cached.
+- **Read the engine's ledger, not the card:** `Available KV cache memory` /
+  `GPU KV cache size: N tokens` / `Maximum concurrency for M tokens per
+  request: X.Xx` is the capacity number. vLLM's `Prefix cache hit rate:` line
+  is a LIFETIME average across every prompt the replica ever saw — send the
+  same prompt twice to an idle engine and compare TTFTs (measured 22.3 s cold
+  → 1.4 s warm for a 50k prefix). `GPU KV cache usage` says the pool is not
+  full; it does NOT say a prefix survived. The KV budget is NOT monotonic in
+  chunk size (5.44 GiB at CHUNK 8192, 4.22 at 16384, 4.16 at 4096) — measure
+  three points before treating a knob as a lever.
+- **Soak before believing:** 15–30 min of concurrent families, count failures
+  AND latency; report hit/miss split and medians with min/max — the median of
+  a bimodal distribution describes neither mode.
+- **Verification that is not the benchmark:** run the client the fleet
+  actually uses; and a deployed unit is not proven until an unattended reboot
+  brings it back.
+- **Record the "no" answers** with numbers and mechanism — placement balance,
+  concurrency admission and chunk-size tweaks all measured flat or worse
+  here; the wall was arithmetic (cached prefixes + live sequences share one
+  pool).
+
+## 6. Sweep hygiene (the arms that produced wrong verdicts)
+
+- **Prefetch uniformity:** every arm in a sweep carries the same
+  expert-prefetch setting. The 2026-09-29 campaign measured losing arms with
+  prefetch ON and incumbents OFF; untaxed, verdicts inverted (no-spec
+  12.11 → 41.47; gather 32×64 parity → +18.7 %). Never re-read a taxed
+  verdict.
+- **Knob-landed preflight:** for every `-e` knob, confirm
+  `podman exec <ctr> printenv <VAR>` — a knob the fork silently ignores
+  produces a "parity" arm that is really the default (v1 of the gather sweep
+  lost all 3 arms this way; serve-script hooks are additive and can drop
+  vars).
+- **Always run an explicit OFF arm:** the 2026-09-29 campaign's only win was
+  deleting a baseline's silent `--expert-prefetch 1` (+40–51 %). Challengers
+  compare against default-off, not "incumbent + one flag".
+- **Pin the capacity bar:** `--kv-cache-memory-bytes` (contract) rather than
+  autofit (remainder) so pool numbers are reproducible; compute need =
+  tokens × measured B/token (hybrid models: KV dtype does NOT double mamba
+  states — measure, don't assume 2x).
+- **Report decode AND prefill (TTFT) per rung:** a decode-only view nearly
+  deployed a 2x-prefill-tax config. Per-depth tables, never a single median —
+  one fork's gather win grows with depth (54 t/s @1k, 39.5 @short).
+- Container recreate (`podman rm -f`) wipes `podman cp`'d harnesses — re-copy
+  `mtp_measure.py`/`bench_iter.py` after EVERY recreate, or mount them.
+
+## 7. Bash laws that ate windows (each cost an arm)
+
+1. `${VAR:+SPEC=$1}` in front of a command expands to a bare word →
+   `SPEC=3: command not found`, nothing runs. Pass literal assignments:
+   `SPEC="${1:-}" bash serve.sh`; the callee's `${SPEC:+--flag}` skips empty.
+2. `${2WAY:-0}` is fatal `bad substitution` (digit-leading name), and inside
+   an if/else it aborts ONLY that compound statement — the script CONTINUES,
+   every arm tears down clean with zero measurements. Detection law: `ls`
+   the per-arm `.measure.txt` files before trusting any summary; arm echoes
+   are not evidence.
+3. `local a=$1 b=$x` expands `$x` before the builtin assigns — under `set -u`
+   it aborts AFTER the harness stopped production. Two statements.
+4. `sed -i` renames the file; already-running bash keeps the OLD inode —
+   mid-flight patches miss arms launched in the preceding minutes. Patch
+   early or run a make-up window.
+5. Never heredoc python through ssh→zsh→bash — write to a file, ship it
+   (`ssh host 'cat > /path && chmod +x /path' < local.sh`); nested quoting
+   fails silently. Same trap in `git commit -m`: backticks become command
+   substitution.
+6. No comments INSIDE a backslash-continued command (the continuation joins
+   the comment); verify multi-line command construction with `bash -n` AND a
+   render test.
+
+## 8. Results location and recovery (frank-primary arms)
+
+| copy | path |
+|---|---|
+| per-arm evidence | `$ARMS_DIR` = `~/.jobs/<run-stamp>/` (`.rc`, `.done` beside it) |
+| summary | `$ARMS_DIR/summary.tsv` = `arm rc decode_tps decode_tps_t0.7 kv_tokens boot_s` + `<arm>.out/.env/.overlay/.rc/.metrics` |
+| server-side last window | `/home/dank/.autoresearch-frank-primary/run.json` |
+| durable memo | `.../last/run.json` + `last/hash` (promoted only after bench AND verified restore) |
+| fallback | `~/.omp/agent/sessions/<ws>/<session>/<N>.{shake,bash}.log` (permanent) |
+
+`/tmp/frank-primary-arms-*` and `/tmp/armstub/` are stub self-tests —
+constants 99.9/88.8 identical across arms by construction; **identical
+numbers across arms == stub or bug, never a measurement.** After compaction,
+re-harvest with `grep -rhoE 'decode_tps[a-z0-9_.]*=[0-9.]+'` over the session
+logs. A window costs ~20 min warm; a detached runner's timeout ≥ 3600 s per
+window.
+
+## Related
+
+- [[RDNA4 MoE Expert Offload VRAM Budget]] — the plan arithmetic being swept
+- [[DFlash DSpark Speculative Decoding]] — spec A/B protocol and shapes
+- [[LLM Performance Diagnosis]] — profiling and attribution methods
+- [[Remote Job Runner]] — detached execution of long sweeps
+- [[frank-medium-llamacpp-dflash2]] — the measured ubatch-optimum history

serverAppConfigs-docs/LLM/LLM Performance Diagnosis.md

@@ -0,0 +1,221 @@
+---
+type: reference
+title: LLM Performance Diagnosis
+description: Probing why a served LLM is slow — the flat-depth probe, the expert-offload duty-cycle arithmetic, kernel-time attribution via the profiler window, live telemetry without benchmarking, RAM-bandwidth ceilings, IBS tensor attribution and prefix-cache-aware routing.
+tags: [performance, profiling, moe, pcie, prefix-cache, routing, telemetry]
+timestamp: 2026-10-05T00:00:00-07:00
+---
+
+# LLM Performance Diagnosis
+
+When a served endpoint is slower than wanted and someone names a suspect
+("PCIe is saturated", "the gather is the bound"), throughput arithmetic cannot
+settle it — it is usually circular (it assumes the suspect consumes the step).
+Get data in this order; the profiler window alone has retired multi-day
+optimization plans.
+
+## 1. Is decode context-bound at all? (the flat-depth probe)
+
+Sweep decode across context depth. **Flat** ⇒ attention over history is not
+the cost; the problem is per-token expert movement and KV/attention tuning is
+ruled out entirely. A genuinely context-bound server shows a clear downward
+slope. Record cold vs warm separately — a small gap means the cache is
+working. Measured on the fp8 expert-offload shape: 12.11 tok/s FLAT from 304
+to 16,095 tokens.
+
+## 2. Duty-cycle arithmetic (the "GPU not fully utilized" answer)
+
+Symptom: smooth ~25–35 % utilization with 100 % peaks on an expert-offload
+MoE. This is the EXPECTED signature, not a fault:
+
+```
+per decode step: compute burst ~100 ms (100 %) → expert-gather wait ~400 ms (~0 %)
+resident fraction = arena slots / total experts   (e.g. 6574/24576 ≈ 27 %)
+PCIe bytes/token  = miss_rate × lookups × slab_bytes  (0.73 × 480 × 2.344 MiB ≈ 0.84 GB)
+predicted duty    ≈ compute_ms / (compute_ms + wait_ms)
+```
+
+Validate the terms against the deployed engine's boot plan
+([[RDNA4 MoE Expert Offload VRAM Budget]]). If predicted ≈ observed, the low
+average is structural. Note small-transfer reality: bulk bandwidth (91 % of
+ceiling at 1 GiB) overstates what 2–3 MiB slabs get (~72 % plus ~20 %
+per-transfer latency). Before blaming the link, measure it
+([[PCIe Link and Atomics]]) and compare the engine's effective gather rate to
+the measured slab-size bandwidth — a 40–50 % gap is the software read path,
+not the wire. `rocm-smi --showuse` reads ~0 % during the whole multi-minute
+weight load: show VRAM (`--showmeminfo vram`) as the load-progress signal.
+
+## 3. Kernel-time attribution (the profiler window)
+
+Run BEFORE building any replacement kernel.
+
+1. Stop + mask production; boot the shape under test with ONLY the profiler
+   config added. vLLM argv:
+   `--profiler-config '{"profiler":"torch","torch_profiler_dir":"/prof","torch_profiler_with_stack":false,"torch_profiler_record_shapes":false,"torch_profiler_with_memory":false,"torch_profiler_dump_cuda_time_total":false}'`
+   — `with_stack=false` is load-bearing (with stacks the stop hangs and the
+   trace buffer grows ~40 GB → kernel OOM-kill of the engine core, anon-rss
+   ~95 GB; the worker-side stop can hang 5 min on ROCm/kineto). Bind-mount
+   `/prof`. If a shell substitutes values into that JSON, verify the escapes
+   actually landed.
+2. Window it: warm request → `POST /start_profile` → N generations of ≥128
+   tokens (~15 s of traffic) → `POST /stop_profile` (client may time out at
+   60 s; the trace still lands). `telemetry_report` has ONE caller: the
+   gpu_worker profiler-stop path. Overhead is small enough to keep decode
+   comparable (9.6 vs 9.9–10.0 tok/s measured).
+3. **Parse ON THE SERVER** (traces are 100–150 MB gzipped, GB expanded).
+   Rank `ph=="X"` events with `cat in {kernel, gpu_memcpy, gpu_memset}` by
+   summed `dur` (µs): report % of device time, launch count AND µs/call
+   (22 µs × 708k launches is a different problem than 4 ms × 4k), plus the
+   first-to-last event span (fraction of wall the device was busy). Aggregate
+   by FULL mangled name (template args are the tuning signal) and by stripped
+   symbol. **Trap: `gpu_user_annotation` events masquerade as GPU time** (a
+   naive sum reported 63 % in one `execute_context_0(...)` entry; spans
+   overlap and can exceed the wall clock).
+4. Divide by the window's token count → ms/token; if kernel time ≈ step
+   time, the table IS the optimization plan.
+
+Worked example (GLM-5.3-Flash, 2×GPU, offload, decode 9.9 tok/s):
+
+```
+34.8 %  708,500 launches  vectorized_elementwise_kernel   <- the actual bound
+10.0 %   42,000           MoE W4A16 GEMM
+ 8.9 %   83,160           router/top-k (sort-based)
+ 6.5 %  133,920           hyper-connection pre/post kernels
+ 1.05 %   20,664          <the long-suspected expert gather>
+```
+
+No amount of link arithmetic would have produced those two conclusions: the
+suspect was 1 %, and the mass was pointwise glue from an uncompiled trunk —
+whose own boot log names the cause (`FULL_DECODE_ONLY` override). It also
+predicted a negative result in advance (spec decode cannot help a shape whose
+per-token cost scales with the token count) which matched an unexplained flat
+MTP measurement.
+
+## 4. Telemetry without benchmarking (live hosts)
+
+When benching is forbidden (production serving), mine the runtime's own
+per-request telemetry — it measures the real workload and costs nothing.
+
+- llama.cpp (root unit, `-lv 3`): `sudo journalctl -u <unit> --since "-3h"
+  --no-pager` → `slot print_timing` lines (prompt eval, eval, total, `graphs
+  reused`, `draft acceptance … mean len`, `n_gen = N, tg = X t/s, tg_3s = …`).
+- halogen (rootless quadlet): `journalctl --user -u <unit>` (**never sudo** —
+  per-user storage) → `serve_api: mtp 28 tok in 0.66s = 42.45 t/s | 16
+  rounds, commit 1.75/round | prompt 58, prefill 0.92s | detok 24us/tok`.
+- Read the envelope, never one sample: short-prompt `prompt eval` is
+  overhead-dominated (57 tok @ 168 t/s) — quote the longest available (1106
+  tok @ 1013 t/s); decode and acceptance are content-dependent (0.46–0.76
+  measured on one unchanged build); `graphs reused > 0` / `selected slot by
+  LCP similarity` prove prefix-cache reuse.
+- **Attribute PCIe without touching the model:** background `nvidia-smi dmon
+  -s t -c 16 -o T`, ONE ordinary `max_tokens=200` request, `wait` at the end
+  (without it ssh closes and the client dies — the journal shows `W srv stop:
+  cancel task` and NO print_timing while curl still reports plausible
+  status). Measured: peak ~230 MB/s against a ~7,900 MB/s link = ~3 %, 0 at
+  idle — the sampled-logits stream, not the transport, is decode traffic on
+  that shape.
+- One `max_tokens=200` request is production traffic and fine; anything that
+  pins the card is a bench → stop-and-mask first. Check WHICH instance you
+  are reading (`systemctl status | head -12`; task ids jump across
+  restarts). The engine's own ledger (`KV self size`, `n_slots`,
+  `n_ctx_slot`) outranks your model of it.
+
+## 5. The fork's data path (before blaming the link)
+
+Offload engines pick between DMA copy-in and read-through UVA dereference
+inside the GEMM. Grep the installed package for the LRU/offload module and
+read the BRANCH CONDITION, not just the env names. A token-count threshold
+decides the path, and it interacts with spec decoding: an MTP step is k+1
+tokens, so the step can silently cross the threshold onto the slow path as
+draft depth grows. Check the threshold's premise: a default tuned at
+12.4 MiB/expert does not transfer to 2.344 MiB/expert, where copy-in can be
+several times faster. TunableOp ghost-skip: `CLAV_TUNABLEOP_SWEEP=1` alone
+does NOTHING under expert offload (`CLAV_TUNABLEOP_WITH_OFFLOAD=1` required);
+check for the skip line before concluding autotuning ran. Gather-geometry
+knobs read at import time → each arm is a full reload.
+
+## 6. RAM-bandwidth-bound MoE ceiling (before recommending a quant)
+
+```
+routed experts/token = top_k × n_layers
+bytes/token          = routed_params × bytes_per_param + shared_expert_bytes
+ceiling (tok/s)      = measured_bandwidth / bytes_per_token
+```
+
+Measure the HOST's real bandwidth, never assume DIMM ratings: 4-DIMM AM5
+boards downclock to DDR5-3600 regardless of 5600 sticks (two large DIMMs beat
+four small). Single-threaded reads under-report dual-channel ~4x — use the
+multi-threaded plateau of a checksumming reader (`scripts` pattern:
+`/tmp/bw.c`, 1/2/4/8/16 threads). Worked example (DeepSeek-V4.1-Flash):
+6×40 experts × 35.4M params = 8.5B params/token; at 3 bpw + 0.53 GB shared =
+**3.7 GB/token** → 42 GB/s ⇒ **11.3 tok/s ceiling**. Sanity anchors: H100
+NVL + CPU offload 226 GB ⇒ 5.8–9.0 tok/s; 2×3090 + 768 GB DDR4 ⇒ 12.2.
+CPU compute rarely binds below 4 bpw (~5.3 flops/byte ⇒ ~223 GFLOPS at 42
+GB/s); PCIe expert STREAMING is worse than CPU compute (Gen5 x8 ~22.7 GB/s
+effective < half of RAM); prefill reads the union of a batch's experts
+(prefill ≈ bandwidth / total_expert_bytes × ubatch). A quant that fits only
+by ignoring the Engram table does not fit (Engram = sparse n-gram lookup,
+NVMe-tolerant; the routed experts are not). An iGPU GTT aperture is an ALIAS
+of the same RAM, not extra memory.
+
+## 7. IBS model-access attribution (AMD CPUs)
+
+Real CPU access patterns for mmap'd weights, live, no stop (established on
+frank-primary, 2026-09-26, ~2.7M attributed samples in 75 s). Reference:
+`scripts/frank-primary-expert-activity.py` (self-contained dump + sample +
+analyze; `--analyze-only`).
+
+1. GGUF header-only scan (ms for a 50 GiB part) → tensor extents. **Extents
+   from CONSECUTIVE offsets, never a type-size table** — a wrong per-element
+   size silently under-covers every tensor and dumps tail samples into a junk
+   bucket (first run lost 59 % of expert samples; a uniform gap/exps ratio
+   ~1.4 across parts is the tell). Data start = `(tell + 31) & ~31`.
+2. `/proc/<pid>/maps` VMAs (7-group regex; pgoff = group 4, path = group 7)
+   matched against part files; multi-VMA per part is normal.
+3. `sudo perf record -e ibs_op// -c 50000 --data -o out.perf -p <pid> --
+   sleep <dur>` WHILE a mixed workload runs (idle perf numbers mean nothing).
+   `sudo perf script -i out.perf -F addr` for addresses.
+4. Attribute: bisect VMAs, `foff = pgoff + (addr − vma_start)`, bisect tensor
+   starts; per-expert index = `int(lo / (span / N_EXPERTS))` (expert-major
+   rows).
+5. Verify before believing histograms: gap ≈ 0, every tensor-bearing part
+   nonzero, and MoE blocks should be FLAT by construction (top-k fires every
+   block every token) — a non-flat block histogram means mis-attribution
+   first. Established law: expert routing is per-block flat, per-EXPERT
+   skewed.
+
+## 8. Prefix-cache-aware routing (multi-replica serving)
+
+Measure with a **sequential-turn client**, not only a batch benchmark — a
+batch of distinct prompts parallelises prefill across replicas and hides the
+problem. The two failure modes: round-robin loses every other cache hit
+(90k-token contexts: warm-turn TTFT 2676 ms round-robin vs 1096 ms pinned);
+pinning-everything loses a card (16 requests: 18.6 s spread vs 30.1 s
+pinned). **The rule that gets both: route by request shape** — a request
+carrying more than one message is a continuation (agent loop re-sending its
+context) → pin to the replica that served that prefix; anything else → spread
+with a COUNTER, not a prefix hash. Add failover: demote a replica on
+connection failure and retry on a live one BEFORE writing anything to the
+client (safe pre-stream; never retry after streaming started).
+
+The wall no policy fixes: **a running sequence and its cached prefix draw on
+the SAME pool.** Three concurrent 50–80k families against 122,636 tokens/card
+thrash under every placement and admission policy tried (counter placement,
+load-balanced placement, one-large-request-per-card admission, both chunk
+directions — the last COSTS KV in both directions). When the working set
+exceeds aggregate KV, say so: the levers are capacity, fewer concurrent
+large contexts, or a different platform. Traps: a lifetime hit-rate counter
+is not a per-request verdict (probe directly: same prompt twice to an idle
+replica — 22.3 s cold → 1.4 s warm); salt corpora; record sample sizes;
+`vllm bench serve` needs `--tokenizer <local path>`; verify metric names
+against the live endpoint before dashboards. Test shape: drive the routing
+function directly with synthetic bodies (stdlib-only policy module);
+conversation identity is the FIRST N KiB of prompt — a shorter tested prefix
+hashes differently per turn and pinning silently never happens.
+
+## Related
+
+- [[LLM Bench Windows and Measurement]] — the window discipline and usage-counter law
+- [[PCIe Link and Atomics]] — link probing and the streaming-vs-compute ceiling
+- [[RDNA4 MoE Expert Offload VRAM Budget]] — the plan behind duty-cycle terms
+- [[vLLM Porting and Compile Triage]] — the compile fix for the pointwise-glue bound

serverAppConfigs-docs/LLM/Llama Cpp Multi-Device Splits.md

@@ -0,0 +1,274 @@
+---
+type: reference
+title: Llama Cpp Multi-Device Splits
+description: What llama.cpp can and cannot express across devices, the no-collective layer-split path, split balancing by measurement, drafter KV sizing from GGUF headers, and heterogeneous placement triage.
+tags: [llama.cpp, multi-gpu, layer-split, pcie, sizing, speculative]
+timestamp: 2026-10-05T00:00:00-07:00
+---
+
+# Llama Cpp Multi-Device Splits
+
+## 1. Expressibility law — classify the arrangement before running anything
+
+llama.cpp expresses exactly two things across devices, and they are not
+interchangeable:
+
+| arrangement | how | expressible? |
+|---|---|---|
+| **Layer split** — layers 0..n on dev A, rest on dev B | `-sm layer -ts <a>/<b>` | **yes**, no collectives needed; works cross-vendor (CUDA + Vulkan measured 2026-09-16) |
+| **Per-tensor placement** — attention+KV on A, experts on B | `-ot <regex>=<dev>` | **NO.** `--override-tensor` sets `has_tensor_overrides()`, a REQUIRED term of the scheduler's `pipeline_parallel` flag (only read site: `llama-context.cpp`, in the `pipeline_parallel = …` expression). That flag sets `n_copies = parallel ? MAX : 1` in `ggml_backend_sched_new`; with `n_copies == 1` a pre-allocated weight on a non-primary device has no backend and the graph aborts at `ggml-backend.cpp:941` (`pre-allocated tensor (blk.0.ffn_down_exps.weight) in a buffer (Vulkan1) that cannot run the operation (NONE)` — `(NONE)` is the leaf's own op name). |
+
+Failure modes observed (mainline `37b53fd`): per-tensor `-ot exps=Vulkan1`
+with `-dev CUDA0,Vulkan1` → scheduler abort; layer-scoped `-ot` with a
+cross-backend pair → model load fails with only `failed to load model`;
+`-ot x=CPU` **works** because the CPU backend is last in the list — **the CPU
+is the only legitimate partner in a heterogeneous split.** On a CUDA+AMD host
+a card can be the trunk OR the offload host, never the complement of an
+AMD-resident peer. Pure Vulkan changes nothing here — the gate is the
+override, not the driver mix.
+
+If the plan needs tensors on a specific device: stop, the answer is no for
+any vendor mix. Expressible alternatives, in order: (1) two independent
+endpoints, one model per device; (2) single-vendor layer split; (3) device +
+CPU per-tensor offload (mainline pinning guard: `GGML_CUDA_NO_PINNED`; ik fork
+was `GGML_CUDA_NO_PINNED_WEIGHTS` — without it a CPU-overridden tensor can
+land in a pinned `CUDA_Host` buffer and page-lock ~108 GiB).
+
+Layer-split arithmetic to check before promising: a whole layer carries its
+experts (MoE) — compute GiB/layer, not per param-group; only some layers carry
+KV (every 4th on the hybrid), so a share of layers is only a share of KV; big
+non-repeating tensors (PLE tables) land on the main device and cannot move —
+they cap how small the primary can get.
+
+## 2. No collectives? The atomics wall and the layer-split answer
+
+Decide with one line before designing:
+
+```bash
+sudo lspci -vv -s <card2> | grep AtomicOpsCtl   # ReqEn- ⇒ no collectives, ever
+lspci -tv                                        # a Routing- bridge is a hardware wall
+```
+
+`ReqEn-` kills vLLM entirely — TP needs a per-layer allreduce AND PP needs a
+device-backend `send`/`recv` of the hidden state
+(`vllm/distributed/parallel_state.py`) — i.e. the same RCCL primitive that
+fails on an atomics-less card. llama.cpp's `--split-mode layer` needs no
+collective: layers distribute in order, only the boundary activation crosses
+PCIe once per forward pass. Full probe/repair detail:
+[[PCIe Link and Atomics]].
+
+## 3. Build: pin by feature, bundle the runtime at the exact depth
+
+- Pin the ref by FEATURE list (e.g. DFlash2 + qwen35 MTP on 2026-09-16 →
+  `ggml-org/llama.cpp#27342`); never float. Verify a fork build the same way
+  (`ls <venv>/…/spec_decode/ # dflash/ vs dflash2/`).
+- `-DCMAKE_HIP_ARCHITECTURES=<gfx only>` — a sibling arch links, loads, then
+  segfaults. `-DGGML_HIP_RCCL=OFF` when there is no atomics path (only tensor
+  mode degrades to the butterfly fallback; layer split unaffected).
+- **TRAP — `cp -r SRC DST/` flattens when `DST` does not exist** and drops one
+  directory level. rocBLAS/hipBLASLt look for device code at a FIXED depth
+  (`../lib/rocblas/library`, `../lib/hipblaslt/library/<arch>`); flattened,
+  the engine loads every GiB, answers `/health`, and dies on the FIRST request
+  with `Cannot read .../TensileLibrary.dat: No such file or directory for GPU
+  arch`. `mkdir -p DST && cp -r SRC/. DST/`, then
+  `patchelf --set-rpath '$ORIGIN/../lib' build/bin/llama-*`.
+- Dual-backend build (CUDA + Vulkan is the practical pair; HIP+CUDA do not
+  coexist): `-DGGML_CUDA=ON -DGGML_VULKAN=ON` in one tree; Vulkan needs
+  `glslc` + libvulkan headers. **Device names are not what you expect:**
+  `--list-devices` lists the discrete card under BOTH backends (`CUDA0` AND
+  `Vulkan0`) and the APU LAST (`Vulkan1`) — a wrong `-ot` name makes
+  llama.cpp print the valid set and exit (self-diagnosing).
+- What the existing build can do: `GPU_BACKEND_*=OFF` lines in its
+  CMakeCache.
+
+## 4. Serve flags that matter on a split
+
+```bash
+HIP_VISIBLE_DEVICES=0,1 llama-server \
+  --device ROCm0,ROCm1 --split-mode layer --tensor-split 1,1 \
+  --gpu-layers 999 --fit off --flash-attn on \
+  --ctx-size <n> --parallel 1 --cache-type-k f16 --cache-type-v f16 \
+  --jinja --metrics --cache-prompt \
+  --spec-type draft-dflash --spec-draft-model <dflash2.gguf> \
+  --spec-draft-device ROCm1 --spec-draft-n-max 7
+```
+
+- `--fit off` + explicit `--ctx-size`: auto-fit must not resize a context set
+  on purpose. **`--parallel N` DIVIDES `--ctx-size`** — one slot at large ctx
+  is the concurrency policy, concurrent agent loops queue.
+- Pin the drafter to the card holding the target's LAST layers
+  (`--spec-draft-device`); under tensor mode an unpinned draft joins the
+  meta-device KV allocation and aborts the KV build. `--spec-draft-n-max` must
+  be ≤ the drafter's `block_size - 1` (DFlash2: 8 ⇒ 7).
+- Qwen3.8-family templates RAISE on `reasoning_effort=high` (500) — probe
+  low/medium/high/xhigh and map harness levels before declaring ready.
+
+## 5. Split balancing: measure, never interpolate
+
+**A `-ts` step is ONE LAYER, and hybrid stacks do not have uniform layers.**
+qwen35 is 16 × (3 Gated DeltaNet + 1 Gated Attention): one step ≈ 0.43 GiB on
+a DeltaNet boundary, ≈ 1.43 GiB on a full-attention one (weights + that
+layer's own f16 KV) — measured on frank-medium. The gap can jump 2.9 GiB in a
+single step; the best integer split can sit 0.2 GiB from balance while its
+neighbours are 1+ GiB off. An affine fit reproduces the deployed point and
+overstates gaps in between (`tests/frank_medium_unit.bats` encodes the fit as
+a gate; treat a NEGATIVE fitted constant as a unit bug).
+
+Knobs that move weight and REQUIRE a re-measure: `--ubatch-size` (compute
+buffers are **~2x heavier on the output/drafter card** — 512→1536 added
+~3.3 GiB there vs ~1.7 GiB elsewhere), a drafter swap (weights + draft KV),
+`--ctx-size`/`--parallel` (KV). Total free VRAM is invariant across splits —
+only the distribution moves; use that to sanity-check a suspicious row.
+
+**Procedure (one split per load):**
+
+```bash
+WORK=/tmp/split.env; cp scripts/serve-<svc>.env $WORK
+export FRANK_MEDIUM_ENV_FILE=$WORK
+for ts in 39,26 38,27 37,28 36,29 35,30; do
+  sed -i "s|^TENSOR_SPLIT=.*|TENSOR_SPLIT=$ts|" $WORK
+  nohup bash scripts/serve-<svc>.sh >/tmp/split.log 2>&1 &
+  # poll /health (or process exit), then:
+  rocm-smi --showmeminfo vram | grep 'VRAM Total Used'   # sample ~6 s after health
+  pkill -x llama-server; sleep 6
+done
+```
+
+Load → sample → kill, NO requests sent (allocation only). Run under `nohup`
+and poll from a separate call (a load is 30–90 s). Pick the smallest
+inter-card gap that clears the per-card floor with room to spare; one `-ts`
+step ≈ 1.5 GiB at ~6k batch / 262k ctx on 30 GiB cards — enough to bracket,
+not to solve. Prefer balance: the busy card's headroom bounds the context
+(`ctx_extra ≈ (floor − measured_free) / per-position cost on the busier
+card`). A faulted split (`SDMA0 UTCL2 page fault` → `illegal memory access` →
+`ggml_abort`) is OFF-LIMITS — re-measure, never re-tune; more headroom does
+not prevent a boundary-copy fault.
+
+## 6. Drafter KV: sized from the TARGET, bounded by its window policy
+
+`common/speculative.cpp` builds the draft context with
+`cparams.n_ctx = llama_n_ctx(ctx_tgt)` — **no flag caps it.** So:
+
+```
+draft KV = ctx_tgt × block_count × 2 (K,V) × kv_heads × key_length × dtype
+```
+
+…unless the drafter declares a sliding window. Measured at 262,144 target
+ctx: DFlash2 (`sliding_window = 2048`) ≈ 40 MiB; a same-shape 5-layer drafter
+with NO window = 20 KiB/position = **5.0 GiB** — a four-layer swing in the
+balance point, and swapping them invalidated an already-balanced split.
+
+**Failure signature:** the engine loads the whole model, answers `/health`,
+and dies on the FIRST draft decode with `ROCm error: out of memory` /
+`cudaMalloc failed` inside `common_speculative_impl_draft_dflash::process` —
+a sizing failure, not a broken drafter; under a self-healing unit it presents
+as a crash-loop. Read `sliding_window`, `block_count`, `head_count_kv`,
+`key_length` from the drafter's GGUF header first. A compact reader (the
+three format traps: no KV alignment — v3 alignment applies only to the tensor
+section; arrays are `type(9) → element_type(u32) → count(u64)`, element type
+FIRST; the enum is `7=BOOL 9=ARRAY`, 64-bit = 10/11/12, and 8=STRING/9=ARRAY
+is the v2/v3 file order, NOT gguf.h's 2/3):
+
+```python
+import struct, sys
+def rd(path):
+    f = open(path, 'rb'); assert f.read(4) == b'GGUF'
+    struct.unpack('<I', f.read(4)); struct.unpack('<Q', f.read(8))
+    nk = struct.unpack('<Q', f.read(8))[0]
+    def s():
+        n = struct.unpack('<Q', f.read(8))[0]; return f.read(n).decode('utf-8', 'replace')
+    SCALAR = {0:1,1:1,2:2,3:2,4:4,5:4,6:4,7:1,10:8,11:8,12:8}
+    FMT = {0:'<B',1:'<b',2:'<H',3:'<h',4:'<I',5:'<i',6:'<f',7:'<?',10:'<Q',11:'<q',12:'<d'}
+    keys = {}
+    for _ in range(nk):
+        k = s(); t = struct.unpack('<I', f.read(4))[0]
+        if t == 8: keys[k] = s()
+        elif t in SCALAR: keys[k] = struct.unpack(FMT[t], f.read(SCALAR[t]))[0]
+        else:
+            et = struct.unpack('<I', f.read(4))[0]; n = struct.unpack('<Q', f.read(8))[0]
+            if et == 8: keys[k] = [s() for _ in range(n)]
+            elif et in SCALAR: keys[k] = ('arr', et, n); f.read(n * SCALAR[et])
+            else: raise Exception('unsupported arr elem type %d' % et)
+    return keys
+```
+
+## 7. Heterogeneous placement triage (go/no-go with arithmetic)
+
+- **Link:** `LnkCap` vs `LnkSta` at every hop; expect **idle-downshift** (a
+  Gen4 x4 dock idles at Gen1 x4, ~1 GB/s) — measure under load. Per-hop cost
+  vs hidden state (5120-dim f16 = 10 KiB/hop ≈ 1.3 µs at Gen4 x4): bytes are
+  almost never the constraint; **hop count is** — every boundary is a host
+  sync, and a per-op-split runtime can hit 100 splits/token at ~5 % SM. Read
+  `graph splits` from the loader log.
+- **KV is usually the bottleneck:** `KV/token = kv_bytes_per_position ×
+  context` (attention reads the whole cache every token). Highest-leverage
+  placement: **KV on the fastest-memory device together with its attention
+  compute** — never KV alone over the link (GBs over 7.88 GB/s = hundreds of
+  ms/token). Check fit: `kv_positions × slots × bytes + attention_trunk ≤
+  VRAM`.
+- **What placement never buys:** prefill is compute-bound and expert weights
+  are read once per batch — moving them buys nothing and adds a serial stage.
+  Calibration: Strix Halo + 3090, moving 12.5 % of expert mass to the GPU
+  gave decode +12 % but prefill 192 → 32 tok/s. The harmonic
+  `1/(f_big/BW_slow + f_fast/BW_fast)` reproduces the decode gain within ~2
+  points (assumes sequential stages — true at batch 1).
+- **UMA check before assuming additivity:** `mem_info_vram_total` (BIOS
+  carve-out) vs `mem_info_gtt_total` (aperture onto SYSTEM RAM). A big GTT
+  window means the "GPU memory" IS the same LPDDR: adding a stage there adds
+  a compute engine, not capacity/bandwidth. Only discrete VRAM is additive.
+- **Speculation as the "cheap" split:** bandwidth-free (token ids per round)
+  but needs the exact tokenizer for a plain drafter or a head trained for
+  that target; a plain drafter costs K sequential forwards (break-even needs
+  p ≈ 0.7+ at r≈0.4–0.5, K=3); compare against the INCUMBENT (a target
+  already committing ~1.7 tok/round on its own MTP head is a high bar); and
+  check the runtime accepts external drafters at all (closed engines 400
+  unknown drafter names by design). Details: [[DFlash DSpark Speculative Decoding]].
+- **Hazards:** keep paged/sidecar tables where they are read from; keep mmap
+  ON for huge-embedding models (anonymous memory → cgroup kill); CUDA graph
+  capture can abort on unusual graph shapes; make the FIRST measurement a
+  single-device baseline of the runtime on the new device — a split's
+  arithmetic is worthless if the baseline loses to the incumbent; a split
+  consumes the fast device entirely — check what it was serving.
+
+**Verdict ranking:** (1) two independent endpoints — often the real answer;
+(2) one process, weights-resident, KV co-located with attention on the fast
+pool; (3) hot-expert cache only if `misses × bytes × tok/s` fits the link and
+overlaps — no mainstream runtime implements it; (4) name what does NOT help:
+TP across unequal devices, KV on the far device, loading over the link,
+prefill-motivated placements.
+
+## 8. Harness syntax and verification
+
+`llama-bench` separates a LIST with `/` and SWEEP values with `,`:
+`-dev CUDA0,Vulkan1` is TWO single-device runs, never one two-device run;
+`-ts 0.85,0.15` is two sweep values each becoming `[0.85, 0, …]` — device 0
+takes everything. Write `-dev A/B -ts x/y`. Read the table's own `dev` column;
+any placement-dependent number needs an independent memory check
+(`nvidia-smi` sampling while loaded). (This artifact produced a wrong fleet
+conclusion 2026-09-16 — "CUDA + Vulkan cannot serve one model" — with two
+bogus OOMs that were really all-111-GiB-on-the-24-GiB-card.)
+
+Verification battery: `/health`; both cards' memory ≈ half the weights each
+(busier card also carries the drafter; compare `model_bytes/2 + drafter +
+ctx×KV/2 + buffers`); raise ctx by N and watch BOTH cards rise by
+N·bytes/2 — neither card's total may equal the model file (that arithmetic
+rules out replication); `spec_decode_num_drafts_total` > 0 and a real
+completion; a deep prompt (140k) proves a raised window is REAL, not
+declared; repeat prompt → `timings.prompt_n` collapses (prefix cache live);
+`/slots` shows expected `(id, n_ctx)` pairs; two concurrent requests with
+`journalctl -o cat | grep 'slot print_timing'` proves drafting on BOTH slots
+(default journal output truncates these lines).
+
+Ops traps: a server started over ssh outlives stopping the runner (keeps the
+port — `pkill -x <binary>` first); `systemctl start` of a root unit needs
+sudo (polkit "Access denied" otherwise); old crash `ROCm error` lines live in
+the journal forever — check `ActiveEnterTimestamp`/`NRestarts` before
+treating a grep count as a live fault.
+
+## Related
+
+- [[PCIe Link and Atomics]] — the atomics wall, link probing, repair
+- [[Llama Cpp Serving Sizing]] — slot-pool and KV pricing
+- [[DFlash DSpark Speculative Decoding]] — spec types, chaining, A/B
+- [[frank-medium-llamacpp-dflash2]] — the layer-split era runbook
+- [[Dual-R9700-Cooperation]] / [[x4-pair-experiment]] — the measured history

serverAppConfigs-docs/LLM/Llama Cpp Serving Sizing.md

@@ -0,0 +1,163 @@
+---
+type: reference
+title: Llama Cpp Serving Sizing
+description: Slot-pool sizing against a headroom band, per-device VRAM accounting, prompt-cache/state sizing and the QSA flash-attn fix — the measurement-first procedures for llama.cpp serving shapes.
+tags: [llama.cpp, vram, kv-cache, slots, flash-attn, sizing]
+timestamp: 2026-10-05T00:00:00-07:00
+---
+
+# Llama Cpp Serving Sizing
+
+## 1. Slot-pool sizing against a headroom band
+
+Goal: choose `CTX_SIZE`/`PARALLEL`/KV types so N conversations fit a GPU while
+keeping a requested amount of VRAM free — from measurement, never arithmetic
+guesswork.
+
+1. **Fix the invariants.** Read the payload's KV geometry (attention layers
+   only, KV heads, head_dim, KV dtype bytes) and the per-slot rule
+   `n_ctx_slot = CTX_SIZE / PARALLEL`. A hybrid model with
+   `kv_unified = 'false'` gives each slot its own cache AND a fixed per-slot
+   state buffer — slot count is a memory term, not free.
+2. **Two boots on the real card** (production stopped, per the bench law),
+   each near the target; record `nvidia-smi` used/total plus the boot log's
+   `n_slots`/`n_ctx_slot`: shape A = current/cheap, shape B = new demand.
+   `non-KV = used − (tokens × KiB/token)`; the two rows separate
+   weights/buffers/graphs from the new demand (a 918 MB projector bills
+   ~1.4 GiB).
+3. **Solve the band:** `KV_tokens ≤ (card − non-KV − headroom_low) /
+   KiB/token`, round the pool DOWN to a multiple of `PARALLEL × 4096` so every
+   slot gets an equal integer window. Boot and measure; if outside the band,
+   walk one step — never re-derive from a formula.
+4. **Propagate the per-slot number — it is the client contract:**
+   `models.yml` `contextWindow` = `n_ctx_slot` (NOT the model's native window),
+   or long conversations die with `400 exceed_context_size_error` instead of
+   compacting; compaction/summary budgets sized to the old window get
+   re-checked; endpoint docs and the rollback table move too.
+5. **Prove behaviour:** boot-log slot count, `nvidia-smi` free in band, N
+   concurrent requests answered in parallel with N+1 queued, and — if a
+   projector ships — a solid-color image probe through an **`image_url`
+   content part** expecting the exact color. The legacy top-level `images`
+   field is silently ignored, so a probe that uses it measures a
+   hallucination, not vision.
+
+Gotchas: shorter slots can make decode FASTER (~5 % measured); the KV pool is
+allocated whole at boot so post-boot `memory.used` IS the ceiling; record both
+measured rows in the service `.env` comment block and mirror the non-KV price
+in the bats budget gate so a shape that busts the band fails the suite instead
+of the boot.
+
+## 2. Per-device VRAM accounting ("where did the memory go?")
+
+First decide which kind of memory is in question:
+
+| device | kind | read it with |
+|---|---|---|
+| RTX 3090 (Dankclaw) | discrete GDDR6X | `nvidia-smi --query-gpu=memory.total,memory.used,memory.free` |
+| gfx1151 iGPU (Dankclaw) | **UMA** — "VRAM" is host RAM | `/sys/class/drm/card1/device/mem_info_gtt_{total,used}` (114 GiB) vs `mem_info_vram_*` (2 GiB carve-out) |
+| R9700s (Dankserver) | discrete | `rocm-smi` figure (64 GiB, SVM-backed) |
+
+Per-process truth on NVIDIA:
+`nvidia-smi --query-compute-apps=pid,process_name,used_memory`. Never add the
+iGPU's GTT to host RAM as a separate pool; `free -g` is the arbiter for
+co-tenants.
+
+- **Engine ledger first.** Halogen prints weights pinned (+ remaining 2 MiB
+  contiguous blocks — the honest fragmentation signal: 240 blocks/12.3 GiB
+  after reboot vs 475/14.7 GiB before means the model did not come back into
+  the same hole), device budget, KV pool/working split, "left for everything
+  else". llama.cpp prints at `llama-model.cpp:1827` (per-backend model buffer)
+  and `llama-context.cpp:692` (compute buffer) — but **`llama-bench`
+  suppresses loader INFO and a systemd unit did not put them in the journal**
+  (verified `grep -c` = 0). Recipe: one launch through `llama-cli`/`llama-server`
+  keeping the first ~200 lines.
+- **Otherwise compose from tensor sizes.** Group the GGUF tensor table by
+  role (Flash-Next example: 75.0 GiB experts at 1.5625 GiB/layer, 32.8 GiB PLE
+  table, 3.2 GiB everything else), KV ≈ 2 KiB/position per f16 attention
+  layer, × derived context. Vendor delta is real: same layers+KV cost 12.80
+  GiB under Vulkan vs 14.99 GiB under CUDA (~3.3 vs ~1.1 GiB of
+  graph/compute pool).
+- **Traps:** mmap'd PLE is reclaimable page cache; `-ot` makes it pinned —
+  "move layers to the card" converts reclaimable cache into device memory and
+  frees nothing for co-tenants. A served model's footprint ≠ weight size
+  (frank-small: 22.6 of 24.1 GiB for an 11.29 GiB model). Do the capacity
+  subtraction before designing: 12 attn layers × 1.62 GiB + 200k f16 KV =
+  28.6 GiB > 24.1 GiB usable → KV-in-VRAM on that pair only reaches ~64k.
+- **Context lengths: derive, never remember.** `llama-bench` computes
+  `n_ctx = n_prompt + n_gen + n_depth` (defaults 512/128/0, type_k/v F16,
+  flash_attn AUTO, split_mode LAYER). `-p 512,16384 -n 64` = contexts
+  576/16,448; a plain `tg` row has ~zero depth — `-d` is the only thing that
+  puts KV in the picture.
+- **Label every number** measured / derived / inference, and state the context
+  each figure was taken at — 12.8 GiB at 65,568 tokens is not the same claim
+  as at 200k.
+
+## 3. Prompt cache and state sizing
+
+`--cache-ram` (MiB, default 8192) is HOST RAM for saved prompt states —
+unpinned, on demand, moves no VRAM. `--ctx-checkpoints` (default 32) is a
+different mechanism: per-slot KV snapshots; `0` disables them.
+
+**State law** (compute before trusting any cap): a state is the serialized KV
+of the whole sequence + constant recurrent state + the draft context's KV:
+`per_token = 2 × n_full_attn_layers × n_kv_heads × head_dim × bytes (K and V) +
+draft_per_token`. Measured on frank-medium (qwen3.5 hybrid, f16, MTP draft):
+16×4×256×2×2 = 64 KiB/token + 4 KiB draft = **68 KiB/token**; a full
+196,608-token slot state ≈ 12.8 GiB. Cross-check against the journal's
+`prompt state size` lines — the two must agree.
+
+**The journal is the instrument:** `exceeds cache size limit … skipping` =
+cap binding in production; `making room … removing oldest` / `cache size
+limit reached` = eviction, and the printed size proves the active cap. Do NOT
+claim a raise is proven by `prompt_tokens_cached_total` rising — on exact
+repeats that counter is 0 by construction.
+
+**Hybrid + zero-checkpoint trap:** for a hybrid (DeltaNet/SSM + attention)
+model, `seq_pos_min` sits at the END of the slot's own prompt (a recurrent
+cell stores only the last position), so with `--ctx-checkpoints 0` there is no
+checkpoint to fall back on and the server logs `forcing full prompt
+re-processing … (likely due to SWA or hybrid/recurrent memory)`. Consequences:
+intra-slot reuse only survives when the request echoes the assistant's
+previous reply verbatim (a pure user resend is always a full re-prefill even
+at f_sim 1.000); the RAM prompt cache becomes the ONLY carrier of a
+conversation state ACROSS slots, so a cap that refuses deep states costs
+minutes of re-prefill whenever slots are reassigned; state restoration only
+runs via the LRU path (or LCP with f_keep < 0.5) — a plain prefix hit never
+calls `prompt_load`.
+
+**Verify a change:** live argv carries the flag (not the deploy log); `/health`
+200 + real completion; VRAM unchanged and host RAM mostly free; no new
+`exceeds cache size limit` lines, with a `making room` entry larger than the
+old cap as direct proof. If the ceiling itself was not exercised live, say so —
+crossing it needs a >120k-token conversation and must not run beside a live
+service (bench law).
+
+## 4. QSA flash-attn assert fix (qwen4exp)
+
+`--flash-attn on` aborts graph build for qwen4exp models:
+`GGML_ASSERT(a->type == GGML_TYPE_F32)` at `ggml.c:5371` through
+`build_attn_qsa`. Root cause: `build_attn_qsa` calls `ggml_fill` on the
+F16-cast kq mask returned by `get_kq_mask()` when flash attn is on, but
+`ggml_fill` requires F32.
+
+Fix (mirrors the MLA sparse path in `llm_graph_context::build_attn`): run the
+top-k mask surgery on the F32 base mask (`inp->self_kq_mask`), cast to F16
+only at the attention call when `flash_attn` is enabled
+(`kq_mask_attn = cparams.flash_attn ? ggml_cast(ctx0, kq_mask_top_k,
+GGML_TYPE_F16) : kq_mask_top_k`). The patch lives at
+`scripts/patches/qwen4exp-qsa-f32-mask.patch`.
+
+Only the charlie12345/ROCmFPX build-qwen38 (PR #27742 qwen4exp merged) takes
+it; CIRU, upstream and stock builds do not. Validate live: load with
+`--flash-attn on`, health 200, a real completion finishing `"stop"`, journal
+clean of `GGML_ASSERT`/`cudaMalloc`/`missing tensor`, and flash-attn engaged
+in the timings. Context cap for full-model residency on that shape stayed
+32,768.
+
+## Related
+
+- [[GGUF Compatibility]] — what a payload is before it is priced
+- [[frank-small-llama-mimo-q6k-2x200704]] — the current two-boot priced shape
+- [[frank-medium-llamacpp-dflash2]] — where the 68 KiB/token datum was measured
+- [[DFlash DSpark Speculative Decoding]] — the spec A/B protocol on llama.cpp
+- [[Dual-R9700-Cooperation]] / [[x4-pair-experiment]] — the multi-device side

serverAppConfigs-docs/LLM/Model Swap and Rollback.md

@@ -0,0 +1,281 @@
+---
+type: reference
+title: Model Swap and Rollback
+description: The full model-swap discipline for the LLM fleet — fit preflight, fetch and payload verification, runtime build/flag checks, git-only config, live validation, systemd handoff, close-out, the three-pin revert law, and research-verification discipline.
+tags: [models, swap, gguf, huggingface, rollback, deployment, verification]
+timestamp: 2026-10-05T00:00:00-07:00
+---
+
+# Model Swap and Rollback
+
+Applies to every endpoint swap on the fleet (frank-small / frank-large /
+frank-primary). Repo law: all config flows through git; the live models are
+host state under `/home/dank/models/<name>/`.
+
+## 1. Fit preflight — never deploy a swap without confirming it FITS
+
+1. Get the checkpoint's on-disk/weight size AND precision from the HF card
+   (`Precision: BF16` = 2 B/param; FP8/Q8 ≈ 1 B; Q4 ≈ 0.5 B). 35B params:
+   BF16 ≈ 66–70 GB, FP8/Q8 ≈ 33–35 GB, Q4 ≈ 17–20 GB.
+2. Compare against the HOST'S usable RAM (not just VRAM) for runtimes that
+   stream weights from RAM (host-RAM expert offload, llama.cpp CPU offload,
+   layer splits). Budget weights + KV pool + OS + page cache; **if weights
+   alone exceed RAM, ABORT** — pool tuning shrinks KV, it does not shrink
+   weights.
+3. **The failure mode is a self-sustaining thrash, not a crash:** a large
+   swap makes the kernel PAGE instead of OOM-killing → sshd/user space
+   starved indefinitely, no recovery window opens by itself; systemd
+   `Restart=always` turns it into a crash-reload-rethrash loop; `enabled` +
+   `WantedBy=multi-user.target` re-thrashes on every reboot. A power cycle
+   does NOT fix it.
+
+Recovery (may need console / SysRq-to-GRUB):
+
+```bash
+sudo systemctl mask --now <svc>                       # break the reboot trap FIRST
+cd <checkout> && git fetch origin && git reset --hard origin/main
+grep MODEL_DIR <env>                                  # confirm the fixed target
+sudo systemctl unmask <svc> && sudo systemctl restart <svc>
+curl -s <health>                                      # then a REAL completion
+```
+
+Deploy caveat: "healthy after Ns" = systemd ACTIVE, not model LOADED —
+confirm with a real `/v1/chat/completions`. RTX 3090 (Ampere) has no
+FP8/FP4 tensor cores: FP8 weights run via upcast but must still fit; choose
+quant by size fit, then accept the compute path the card supports. Before
+choosing a smaller quant, state WHICH of the two you are buying — headroom,
+or a design change (a smaller quant does not change what the runtime can
+express; per-tensor cross-device placement stays refused under
+`--override-tensor`, and a layer split moves only contiguous layer ranges).
+
+## 2. Ground truth first
+
+- Read the HF model card (runtime pin, hard constraints, recommended args)
+  + the repo trio `scripts/<svc>.env`, `scripts/serve-<svc>.sh`,
+  `scripts/<svc>.service` + the runbook under `serverAppConfigs-docs/LLM/`.
+- Check the host before downloading anything: `ls /home/dank/models/`,
+  `df -h /home`, `du -sh` — bench-era or prior-deployment files often already
+  exist.
+
+## 3. Fetch and verify (HF payloads)
+
+**Inventory by QUANT NAME, never by extension** — large quants ship as
+`.partNofM` fragments (Q4_K_M = 37.00+37.00+36.97 GiB), so an
+`endswith(".gguf")` filter makes Q4_K_S look nonexistent. Check both
+mradermacher repos (`…-i1-GGUF` imatrix, `…-GGUF` plain) and the model's own
+repo; the tree API's `blobs=true` gives `lfs.oid` per file. Gated mirrors
+return 403 even with a valid token until the account accepts the gate —
+prefer the ungated copy.
+
+```bash
+# token out of band: mode-600 curl config; a token passed as -H is world-readable
+printf 'header = "Authorization: Bearer %s"\n' "$(cat ~/.cache/huggingface/token)" > /tmp/hf.curlrc
+chmod 600 /tmp/hf.curlrc
+curl -fL -C - --retry 10 --retry-delay 5 --retry-max-time 21600 -K /tmp/hf.curlrc -o "$DEST/$name" "$BASE/$name"
+```
+
+`-f` is MANDATORY: `curl -o FILE URL` **exits 0 on HTTP 401/403** and writes
+the error body as the artifact (identical 153-byte shards, cheerful DONE,
+zero bytes of model). Never `pgrep -a` a running fetch (token is in
+`/proc/<pid>/cmdline`); watch progress by size:
+`while [ "$(stat -c%s … || echo 0)" -lt "$EXPECTED" ]; do sleep 30; done`.
+
+**Verify, then assemble:** per part, byte count == Hub's declared size AND
+`sha256sum` == `lfs.oid`; concatenate in NUMERIC order; assembled total ==
+sum of parts (compute the sum with python, not by hand) and `head -c 4` ==
+`GGUF`. The assembled file has no published hash — the per-part oids + exact
+total are the chain. Keep the fragments when disk allows (only copies
+re-verifiable against the Hub); failure artifacts and the replaced payload
+go to `~/.AGENTTRASH/YYYYMMDD-topic`, never `rm` — upstreams DELIST quants
+and the local file may be the last copy.
+
+Fetch-script one-liners: dedicated scripts (`scripts/download-*.sh`) already
+do `-f` + byte-count + LFS sha256 + `--verify-only`; ad-hoc fetches tend to
+skip all three. HF cache snapshots SYMLINK every file into `blobs/` and
+those links dangle inside containers — serve scripts and the deployer refuse
+symlinks and missing tokenizer files for exactly this reason.
+
+## 4. Runtime build (when the card pins a fork/build)
+
+- Clone + build with the card's exact cmake line, plus
+  `-DLLAMA_BUILD_WEBUI=OFF` — WebUI asset provisioning (npm / HF llama-ui
+  bucket) aborts llama.cpp link at 98 % on these hosts; the API server is
+  unaffected.
+- Verify every flag the serve script passes exists in the new binary BEFORE
+  touching the service: `llama-server --help | grep -q -- "<flag>"` for each;
+  list MISSes. Read the GGUF's tensor histogram first if the payload uses a
+  custom type — the FORMAT picks the runtime ([[GGUF Compatibility]]).
+
+## 5. Config via git only
+
+Edit `scripts/<svc>.env` (paths, LLAMA_BIN, MODEL_FILE, MMPROJ_PATH) + serve
+script header/defaults locally; `bash -n`; `./tests/run.sh` must pass;
+commit; push; `git pull --ff-only` on the target. Never edit configs on the
+remote. Renaming a model dir (`mv`) is host state and allowed. The env-file
+hook pattern (`FRANK_*_ENV_FILE`) is what makes bench windows and staged
+arms possible — keep it intact.
+
+## 6. Live validation
+
+- No scratch ports — one GPU cannot host two big models. Ensure the old unit
+  is truly gone first: `systemctl is-active` can hang in `stop-sigterm`
+  (llama.cpp never preempts an in-flight generation); poll until the PID
+  exits.
+- Manual serve for smoke: `setsid nohup bash scripts/serve-<svc>.sh >
+  /tmp/x.log 2>&1 &` — plain `ssh 'nohup … &'` dies with its ssh session.
+- Verify: health 200; a REAL `/v1/chat/completions`; reasoning flows
+  (`reasoning_content` non-empty); if the endpoint takes images, a
+  solid-color probe through an `image_url` content part (legacy top-level
+  `images` is silently ignored; probe with ≥256 px).
+- vLLM specifics: first request after boot can take ~18 s (deferred Triton
+  loads) — warm and discard; `--reasoning-parser` / tool parsers are part of
+  the contract ([[vLLM Boot Forensics and Config Interrogation]]).
+
+## 7. systemd handoff — THE TRAP
+
+`./scripts/deploy.sh <host> --service <svc>` gates health, but a leftover
+manual server holding the port makes the health check pass while the UNIT
+crash-loops on bind. Always confirm ownership:
+`systemctl show <svc> -p MainPID` equals the single `pgrep -x llama-server`
+(or the container runtime PID); only then trust "healthy". If a real
+long-context task is queued, do the handoff anyway — client aborts are
+recoverable, drift is not.
+
+## 8. Close-out (mandatory)
+
+1. Docs: active runbook updated (build recipe, profile, validation numbers
+   incl. real-workload prefill/decode); superseded runbook gets a RETIRED
+   banner and stays as the rollback record; `LLM/index.md` mermaid + active
+   list; AGENTS.md fleet table + runtime section + history chain; work-log
+   entry in `serverAppConfigs-docs/log.md`.
+2. `~/.omp/agent/models.yml`: `contextWindow` must equal the REAL per-slot
+   context — llama.cpp logs `n_ctx_slot` per prompt (`--parallel N` divides
+   the pool; e.g. 262144 native / 2 slots = 131072 per slot); vLLM reports
+   `max-model-len` and its concurrency line. Overselling causes silent
+   client-side overflow or a discovered-wall 400. Also `maxTokens` vs the
+   engine's cap, `input` capabilities (text/image), and
+   `reasoningContentField` per runtime (llama.cpp/SGLang emit
+   `reasoning_content`, vLLM emits `reasoning`). Fix every `modelRoles` entry
+   that named the old endpoint.
+3. **Rollback = one env commit** (paths + LLAMA_BIN + flags), prior binary +
+   payload kept on disk, rollback numbers recorded in the runbook.
+
+## 9. Revert-pin an ad hoc experiment (the three pins)
+
+A campaign is "done" only when all three exist — without them a
+`podman pull`, an image update, or a lost scratch script destroys the way
+back:
+
+1. **Git tag (annotated, pushed):**
+   `git tag -a <exp>-tuned-YYYYMMDD -m "<tuned env/flags + measured headline +
+   runbook path + revert command>"` — a future agent must reconstruct the
+   campaign from the tag message alone.
+2. **Image digest in the serve script:** floating tags move. Capture
+   `podman inspect <ctr> --format '{{.ImageName}} {{.Image}}'` + `podman
+   images --digests`, patch `IMAGE=` to `repo@sha256:<repodigest>` with a
+   comment (image ID, date, old ref), verify `podman image exists
+   repo@sha256:<digest>` and `bash -n`. Digest refs survive upstream pushes;
+   `podman pull repo@sha256:<digest>` re-fetches exact bytes. One host can
+   hold MULTIPLE images of one repo — pin by the digest of the RUNNING
+   container, never by tag name. Never edit the pinned line back to a
+   floating tag; new experiments branch from the digest.
+3. **Local alias:** `podman tag <id> <exp>-runtime:tuned-YYYYMMDD`.
+
+Record scratch-script paths + full flag surface in the runbook (hook env
+vars in CAPS); expired images/scripts go to `.AGENTTRASH`, never `rm`.
+Pure measurement sessions that changed no state need only pin 1.
+
+## 10. Research verification discipline (for "can model X run here?")
+
+- Scouted research summaries have shipped fabricated citations on this fleet
+  (invented HF repos, plausible-but-wrong PR numbers, stale engine versions).
+  NEVER copy a scout's citations into a deliverable — re-verify each
+  load-bearing URL directly (GitHub issues/PRs/repos, HF pages) before it
+  enters anything.
+- The in-repo runbooks are FIRST-HAND evidence and outrank any web claim
+  about our own hosts.
+- Verified anchors (2026-09-27): DeepSeek V4.1-Flash (552B/8B-16B active,
+  CED arch, 1M ctx, MIT; preview was 284B/13B with native FP4+FP8 — early
+  GGUF conversions garbage); Lucebox = the proven gfx1201 ROCm 7.2 path
+  (heterogeneous R9700+Strix-Halo V4.1: 86 tok/s decode, 788 t/s prefill@2K).
+  GLM-5.3-Flash (320B/18B, KDA+DSA+mHC, 1M ctx — ALWAYS bind `-c`, the
+  default hard-hung a 128 GB host; native FP8 328.3 GB; `-fa off` MANDATORY,
+  flash-attn breaks MLA correctness; llama.cpp arch `glm5next` lived in
+  unslothai branch → ggml PR #27754, MTP landed d07e71ede 2026-08-30; quant
+  ladder UD-IQ1_S 93.1 GB → UD-Q4_K_XL 199.7 GB, GSQ-RCO 3.5-bit −1.40 pp
+  MMLU-Pro vs Q8_0).
+- GGUF parsing gotcha: expert tensors are
+  `blk.N.ffn_{gate,up,down}_exps.weight` — 4 dot-parts INCLUDING `.weight`;
+  a `len(parts) == 3` filter matches zero layers with no parse error. Filter
+  on `parts[0]=="blk" and parts[3]=="weight" and "exps" in parts[2]`.
+
+## 11. Adding a second LLM endpoint to a host
+
+When a host must serve two models at once (an iGPU runtime plus a second
+card), or one endpoint is moving and a new one takes the freed hardware.
+
+**0. Decide the shape before touching hardware.** One provider per
+HOST:PORT in `~/.omp/agent/models.yml` (on the box where the harness runs)
+— a second endpoint is a new provider block, not a second model in the
+existing one. Read the budget as an inequality: for unified memory the
+engine's own fit check is the arbiter (`budget = host − pinned weights −
+reserve`; `need = KV + arena + vision`); for a discrete card, weight the KV
+by counting only full-attention layers — and remember `--parallel N`
+DIVIDES `--ctx-size`. A co-tenant is a budget line, not a footnote: free
+RAM by changing the POOL or the ARENA (the reserve is a fit-check term,
+never an allocation) and drop optional charges (a vision tower is ~2 GiB)
+rather than hoping.
+
+**1. Repo first, and retire in the same commit.** New service files
+(`scripts/<svc>.env`, `.service`, `serve-<svc>.sh` with hard guards for
+missing artifacts); planner entries (`SERVICE_ORDER` + the target's service
+set + path mappings, inventories in `deploy.sh`/`deploy-remote.sh`, the
+dispatch case); retire the obsoleted pieces into `.AGENTTRASH/` with a
+suffix — and check what pointed at them: bench harnesses, a chat template
+only the old runtime read, and `scripts/test-changed.sh` mappings are the
+usual victims (a mapping to a retired test file fails the pre-commit
+wrapper silently for weeks). One bats file per service asserting
+INVARIANTS (`pool == 2 × ctx`, KV bytes ≤ card budget, port separation),
+not constants; a negative grep must strip comments first
+(`grep -vE '^\s*#' file | grep -q …`) or it matches the prose explaining
+the very thing you removed.
+
+**2. The port-bind race (rootless podman).** pasta binds the published port
+at container CREATE; a just-stopped predecessor still holds it. Order the
+handler: retire the old manager's unit → stop THIS service's own unit
+(no-op on first cutover) → wait for every socket on the port to clear
+([[Container Host Reachability]]) → install → start. Without the second
+step a redeploy deadlocks against its own gate.
+
+**3. Wire the surfaces, refuse impossible signals.** Point the harness
+small/fast + image roles at the new endpoint; move the vision role WITH the
+vision tower (one owner per capability). `shared/homepage/services.json`: a
+card per endpoint, probed from the monitor's own host. Prometheus: only add
+a job if the runtime actually exposes Prometheus text (halogen publishes
+`/health`, `/cache`, `/v1/*` and nothing else — the job was REMOVED with the
+reason written down rather than left as a permanent red).
+
+**4. Cutover, then verify by driving it.** `deploy.sh <target> --service
+<large> --service <small>` (order matters); per endpoint: `/health`,
+`/v1/models` reports the alias, a chat completion returns content AND
+`reasoning_content`, a `tools` request returns `tool_calls`, an image
+request is answered (or refused with a named reason), the runtime's own
+speculation counters move, and the unit is `enabled`/`generated` with linger
+on. Read the engine's boot ledger for a silent clamp. Finally spend one
+real request through the harness per changed role — routing labels lie; the
+servers' own journals do not.
+
+Traps that look like model faults: `500 tools param requires --jinja flag`
+(runtime lacks the Jinja path — pin it); `500 Jinja Exception: Unexpected
+reasoning effort <level>` (fold via `compat.reasoningEffortMap`); the same
+request works by hand but a subagent dies (subagents run at a different
+thinking level — check both); the container dies instantly although the
+predecessor "stopped" (the port race above).
+
+## Related
+
+- [[GGUF Compatibility]] — which runtime can load a payload at all
+- [[Llama Cpp Serving Sizing]] — pricing the shape once it loads
+- [[RDNA4 MoE Expert Offload VRAM Budget]] — vLLM-side fit arithmetic
+- [[LLM Bench Windows and Measurement]] — measuring the new deployment
+- [[dankserver-qwen38-fp8-expert-offload-tuning]] — the current endpoint

serverAppConfigs-docs/LLM/PCIe Link and Atomics.md

@@ -0,0 +1,244 @@
+---
+type: reference
+title: PCIe Link and Atomics
+description: Link probing (idle downshift, path ceilings, CUDA/HIP bandwidth probes), the streaming-vs-compute ceiling arithmetic, and the PCIe-atomics wall that decides whether AMD multi-GPU collectives can exist at all.
+tags: [pcie, rccl, atomics, multi-gpu, bandwidth, amd, nvidia]
+timestamp: 2026-10-05T00:00:00-07:00
+---
+
+# PCIe Link and Atomics
+
+## 1. Degraded or just idle?
+
+The trap: an idle GPU reports `LnkSta: Speed 2.5GT/s (downgraded)` — the
+driver downshifts to Gen1 for power. That is not a fault and not the slot's
+fault. Never conclude from an idle reading.
+
+1. **Path ceiling from the kernel (cheapest, most decisive):**
+   `sudo journalctl -k -b --no-pager | grep "available PCIe bandwidth"` →
+   `pci 0000:c4:00.0: 63.012 Gb/s available PCIe bandwidth, limited by 16.0
+   GT/s PCIe x4 link at 0000:00:03.1 (capable of 252.048 Gb/s with ... x16)`.
+   The "limited by" device names the **bottleneck** (usually a root port) and
+   its width — that is the number a swap could move; "capable of x16" is the
+   endpoint's OWN capability, and the gap between them is the story.
+   `dmesg` rotates; `journalctl -k -b` keeps this boot.
+2. **Both sides separately:** `LnkCap`/`LnkSta`/`LnkCtl2` on endpoint AND root
+   port. `LnkCtl2: Target Link Speed: 16GT/s` on both ends = the platform
+   intends Gen4, so a Gen1 current speed is power management, not
+   misconfiguration. A root port whose `LnkCap` width exceeds the trained
+   width means the socket/adapter wires fewer lanes — **M.2 connectors are 4
+   lanes by pinout; an M.2 riser can never deliver more.**
+3. **Prove the link under load (the step people skip):** on NVIDIA sample NVML
+   while the GPU works
+   (`nvidia-smi --query-gpu=pcie.link.gen.current,pcie.link.gen.hostmax,pcie.link.width.current,pcie.link.width.max`):
+   `gen.current` climbing 1 → `hostmax` = idle downshift, healthy; stuck below
+   `hostmax` under load = real degradation; `width.current < width.max` under
+   load = **physical lane limit** — no software touches it. On a busy
+   production GPU this is a usage smoke test, not a benchmark — never stop a
+   live service to read its link.
+4. **Do not plan hardware from `dmidecode -t slot`** — consumer firmware
+   labels Ethernet/SD controllers as "M.2 slots" and advertises slots in PCI
+   domains that do not exist. Trust `lspci -nn | grep "PCI bridge"` and
+   `lspci -tv` (empty root ports still enumerate; an absent port is absent).
+5. **"Would moving the card help?"** Compare the bottleneck device from step 1
+   with the target port's `LnkCap` width; two M.2 sockets = both x4 = a
+   no-op. Clean `DevSta`/`CESta` with a low trained speed means "wired
+   narrow", not "failing". Report ceiling, card capability, what the idle
+   reading meant, and whether any physical change moves the number.
+
+Gen4 usable payload after 128b/130b: **x4 7.877 GB/s, x8 15.75, x16 31.5**.
+
+## 2. The engine shape decides whether the link matters at all
+
+| engine shape | what crosses PCIe per token | link matters? |
+|---|---|---|
+| streams weights (expert LRU, paging) | the missed weights (~GB/token) | **yes — decisive** |
+| computes offloaded weights in RAM (llama.cpp `-ot exps=CPU`) | activations only (~KB) | no; host RAM bandwidth decides |
+| everything resident | nothing | no |
+
+**Ceiling arithmetic:** `link time/token = active expert bytes / measured
+H2D rate`; `ceiling = 1 / link_time × resident-fraction saved (LRU hits)`.
+Compare against the compute-in-RAM path at host RAM bandwidth (256 GB/s
+LPDDR5X on the Strix Halo). **If the link is >10x slower than the memory
+behind it (x4 is 31x here), streaming loses by construction and no tuning
+rescues it; doubling the link only doubles a losing number.**
+
+Worked example (Dankclaw 2026-09-16): Gen4 x4 measured H2D **6.72 / D2H 6.60
+GB/s** (85 % of spec) — pinned H2D ≈ async, pageable within 1 %, i.e.
+link-limited not staging-limited. A 3 MiB slab (the expert-streaming
+granularity) = 474 µs. A3B MoE at Q4_K_M (~1.70 GiB active/token) → 272
+ms/token ⇒ **3.7 tok/s** (4.6 with 20 % resident) vs measured **20.46 tok/s**
+with experts computed on the CPU, 25–26.5 on the iGPU. The streaming design
+loses 3–5x.
+
+## 3. Measure the link: two probes, same mechanics
+
+Stop **all** production services first (fleet bench law) — the 3090's VRAM
+may be nearly full, and the probe contends for the very link it measures.
+
+**CUDA probe** (`pcie-link-test.cu`): sweep {3, 16, 64, 256 MiB, 1, 2 GiB}
+with `cudaMalloc` + `cudaHostAlloc`, event-timed; per size measure sync H2D,
+async H2D, D2H, one single transfer, and pageable H2D. `3 MiB` is the row
+that matters for expert streaming; big sizes give the ceiling.
+
+**HIP probe for RDNA4** (`pcie_bw.hip`) — the ROCm landmines each cost real
+time, so encode them:
+
+- `hipGetDeviceProperties` SEGFAULTS on some builds; use `hipGetDeviceCount`
+  + `hipSetDevice` only. First line: `setvbuf(stdout, NULL, _IONBF, 0);` — a
+  segfaulting process loses block-buffered stdout and hides the crash point.
+- Build with an explicit arch: `HIP_TARGETS=gfx1201 hipcc -O2
+  --offload-arch=gfx1201 -o p p.hip`; compile with `| grep -E "error:"`,
+  never `| tail -N` (truncation hid a missing `#include <cstring>`).
+- `hipMemset`/`hipDeviceSynchronize` are `[nodiscard]` — use or cast.
+- Mount the whole `/dev/dri` into ROCm containers; individual `renderD*`
+  nodes enumerate but fail context creation.
+- `rocm-smi --showmetrics`: `pcie_link_width` is trustworthy, but
+  `pcie_link_speed` reads the ASPM idle value and `pcie_bandwidth_acc/inst`
+  are usually `N/A` on consumer cards — live traffic generally cannot be
+  sampled.
+
+```cuda
+// pcie_bw.hip — sweep sizes, pinned vs pageable
+#include <cstdio>
+#include <cstring>
+#include <cstdlib>
+#include <hip/hip_runtime.h>
+static double one(size_t bytes, bool pinned, int iters) {
+    void *dev = NULL, *host = NULL;
+    if (hipMalloc(&dev, bytes) != hipSuccess) return -1;
+    if (pinned) { if (hipHostMalloc(&host, bytes, hipHostMallocDefault) != hipSuccess) return -1; }
+    else        { host = malloc(bytes); }
+    memset(host, 0x5a, bytes);
+    hipEvent_t a, b; hipEventCreate(&a); hipEventCreate(&b);
+    for (int i = 0; i < 3; i++) hipMemcpy(dev, host, bytes, hipMemcpyHostToDevice);
+    hipDeviceSynchronize();
+    hipEventRecord(a, 0);
+    for (int i = 0; i < iters; i++) hipMemcpy(dev, host, bytes, hipMemcpyHostToDevice);
+    hipEventRecord(b, 0); hipEventSynchronize(b);
+    float ms = 0; hipEventElapsedTime(&ms, a, b);
+    double gb = (double)bytes * iters / (ms / 1000.0) / 1e9;
+    if (!pinned) free(host); else hipHostFree(host);
+    hipFree(dev); hipEventDestroy(a); hipEventDestroy(b);
+    return gb;
+}
+int main(int argc, char **argv) {
+    setvbuf(stdout, NULL, _IONBF, 0);
+    if (hipSetDevice((argc > 1) ? atoi(argv[1]) : 0) != hipSuccess) return 1;
+    size_t sz[] = {2459136UL, 3UL<<20, 16UL<<20, 64UL<<20, 256UL<<20, 1UL<<30};
+    for (int i = 0; i < 6; i++) {
+        size_t b = sz[i];
+        int iters = (b >= (1UL<<30)) ? 10 : (b >= (256UL<<20)) ? 20 : 100;
+        printf("  %10.3f MiB | pinned %6.2f | pageable %6.2f\n",
+               b/1048576.0, one(b,true,iters), one(b,false,iters));
+    }
+    return 0;
+}
+```
+
+**Reading it:**
+
+- **Engine-vs-link is the diagnosis, not the raw number:** compare
+  `engine_effective_GB_s` to `link_GB_s` at the SAME transfer size. Engine at
+  ~40 % of the link at the same granularity = the wire is not the problem and
+  no hardware change will help.
+- Achieved GB/s is itself the proof of the trained link state (6.7 GB/s only
+  exists at Gen4 x4) — no separate under-load `LnkSta` sample needed.
+- **pinned ≈ pageable ⇒ pinning is not a lever.** async ≈ sync ⇒ link-limited
+  (the usual "pageable costs 2x" rule does not apply at this link).
+- Small-transfer penalty (2.345 MiB: ~106 µs vs ~78 µs ideal, ~21 %) is
+  per-transfer overhead — recoverable only by BATCHING transfers, not by a
+  faster link. (Dankserver 2026-09-29: bulk 22.7 GB/s at the 2.344 MiB slab =
+  91 % of Gen5 x8, while the engine's effective read path ran ~9.5 GB/s — the
+  gap is the fork's gather software path, not the wire.)
+- Both GPUs symmetric ⇒ no cross-device rebalance is available; measure both.
+- Don't quote a derived line from inside a probe without hand-checking it —
+  a printf format/argument mismatch in one of these printed a nonsense
+  ceiling. GiB→GB before dividing.
+
+## 4. The PCIe-atomics wall (AMD multi-GPU collectives)
+
+**Symptom:** multi-GPU AMD host fails TP/PP; the failure surfaces as
+`hipLaunchKernel: hipErrorIllegalState` for `ncclDevKernel_*` — nothing in
+vLLM/RCCL/PyTorch names atomics. Get the real message:
+
+```bash
+AMD_LOG_LEVEL=3 <the failing process>
+# → "Pcie atomics not enabled, hostcall not supported" → "AQL dispatch failed!"
+```
+
+**Cause chain:** RCCL kernels carry a hidden hostcall-buffer parameter; ROCr
+refuses to fill it without PCIe atomics, which it reads **once at driver
+probe** (`rocclr/device/rocm/rocdevice.cpp` `checkAtomicSupport()` — no else,
+no override). **No env var, module param, or sysfs toggle exists** —
+`setpci -s <gpu> CAP_EXP+28.w=0440` (DCTL2 bit 6) changes nothing at runtime.
+
+**Check the CHAIN, not the card:**
+
+```bash
+sudo lspci -vv -s <gpu> | grep -E "AtomicOpsCtl|LnkSta|LnkCap"  # want ReqEn+
+lspci -tv                                                        # every hop
+sudo lspci -vv -s <bridge> | grep AtomicOpsCap                   # want Routing+
+```
+
+`Routing-` is a read-only hardware capability. Typical AM5 layout: one x16
+slot on CPU root ports (atomics fine), the rest behind the chipset (usually
+not). Bandwidth for TP is small anyway: ~10 MB per token-step for a 27B at
+batch 4 ≈ 1.4 ms at Gen4 x4 — compute-bound, not interconnect-bound.
+
+**Probe matrix (bare 2-rank `torch.distributed`, one process per GPU, 25 s PG
+timeout) — one probe answers several design questions:**
+
+| Op | CPU-lane card | chipset card | Decides |
+|---|---|---|---|
+| `broadcast`/`all_reduce`/`barrier` | OK | FAIL | TP is dead |
+| `send`/`recv` (P2P), in ISOLATION | hang | FAIL | **PP is dead too** — vLLM's PP handoff IS this primitive |
+
+Test `send`/`recv` alone (a prior failed op leaves the group broken). Run the
+probe from a REAL file (`torch.multiprocessing.spawn` re-executes it; piping
+into `python3 -` fails with `FileNotFoundError: '/<stdin>'`). `world_size=1`
+always succeeds — **one-card success proves nothing.**
+
+**SAFETY — non-negotiable:** never run a GPU experiment beside a serving
+unit: `sudo systemctl stop <svc> && sudo systemctl mask <svc>` first (mask,
+never disable — disable deletes a relinked unit file), or use a host that
+serves nobody. A hung AQL dispatch can wedge the GPU driver; a probe beside a
+live service on 2026-09-16 preceded a hard host down (no ARP, no ICMP from
+two subnets, power cycle required). Telling off from wedged afterwards: `ip
+neigh show <ip>` from a peer — `FAILED`/`INCOMPLETE` = off/NIC dead; a wedged
+kernel usually still answers ARP. Validate your own probe's naming before
+reporting an outage (`gitea` vs `gitea-server`).
+
+**Fixes, ranked:**
+
+1. CPU-attached M.2 → PCIe x4 riser (~$30) — puts the card on the CPU root
+   complex; x4 suffices for TP.
+2. Board whose CPU x16 bifurcates x8/x8 (done 2026-09-20 on dankserver — both
+   R9700s on root ports `00:01.1`/`00:01.3`, every bridge `Routing+`, both
+   endpoints `ReqEn+`).
+3. llama.cpp `--split-mode layer` — no collectives at all ([[Llama Cpp Multi-Device Splits]]); costs the GGUF conversion plus paged attention /
+   continuous batching / spec decode.
+4. Independent TP=1 replicas + routing proxy — cards never share one model.
+
+**Closed software routes (do not re-litigate):** vLLM's IPC custom allreduce
+is gated by a CUDA-sm-keyed table (`{9.0, 10.0, 10.3, 10.7}`; ROCm reports
+12.0 → off on RDNA4 by code limit) — and patching it is insufficient because
+TP init's `broadcast`/`barrier`/logits `all_gather` have no non-RCCL path. A
+fork wrapping `.all_reduce` (e.g. radiance `RADIANCE_USE_R4D_AR` one-shot
+P2P, gloo for IPC-handle exchange) does not intercept
+`PyNcclCommunicator.__init__`'s warmup allreduce. NCCL symmetric memory needs
+`ncclDevComm*` (many builds ship the version string without the device API).
+P/D disaggregation needs a KV connector AND coordination collectives. gloo is
+CPU/TCP — unreachable for per-layer TP.
+
+**Open, unverified:** rebuilding RCCL so kernels stop declaring a hostcall
+buffer (`-mprintf-kind=buffered` is the usual claim) — counter-evidence: no
+live `printf`/`assert` in RCCL device source. Test path: rebuild, dump a
+kernel's declared args (`roc-obj-extract` + `llvm-readelf --notes`), look for
+`hidden_hostcall_buffer`; if gone, retry the 2-rank probe (service stopped).
+
+## Related
+
+- [[Llama Cpp Multi-Device Splits]] — the layer-split path around the wall
+- [[Dual-R9700-Cooperation]] / [[x4-pair-experiment]] — measured history
+- [[dankserver-qwen38-fp8-expert-offload-tuning]] — the live gather economics

serverAppConfigs-docs/LLM/RDNA4 Container Serving.md

@@ -0,0 +1,250 @@
+---
+type: reference
+title: RDNA4 Container Serving
+description: Bringing a model larger than VRAM onto Dankserver's 2x R9700 (gfx1201) with the tcclaviger vLLM stack — quant-class gating, fetch verification, container GPU-init triage, memory budgeting and the expert-offload plan.
+tags: [rdna4, gfx1201, r9700, vllm, expert-offload, podman, rocm]
+timestamp: 2026-10-05T00:00:00-07:00
+---
+
+# RDNA4 Container Serving
+
+Bring-up path for Dankserver (2x Radeon AI PRO R9700, gfx1201, 31.86 GiB each =
+**63.7 GiB real VRAM**, 246 GiB RAM, 7700X) with
+`docker.io/tcclaviger/vllm` (vLLM 0.29.0.dev, GLM/Qwen-specialised RDNA4 build
+with native MoE quant kernels and a first-class expert offload planner). The
+tuning history and current deployed shape live in
+[[dankserver-qwen38-fp8-expert-offload-tuning]]; this page is the bring-up law.
+
+## 0. Know the box — and its phantom third GPU
+
+`torch.cuda.device_count()` returns **3**; summing `total_memory` overstates
+the box as 186.7 GiB. The third device is the 7700X's integrated Vega
+(gfx1036): torch reports ~123 GiB for it, `rocm-smi` reports 0.50 GiB,
+allocating on it moves system `MemAvailable` 1:1 (it is GTT over DDR5, not
+dedicated VRAM), and it dies with `hipErrorInvalidKernelFile` when the image
+ships no code for that arch. **Never sum device `total_memory`; never count
+the iGPU as headroom or a workload.**
+
+## 1. Preflight: is the quant class EXECUTABLE, not merely present?
+
+Expert offload is mandatory when weights exceed VRAM, and it requires the
+checkpoint's experts class to have BOTH a compiled kernel AND an
+`offload_recipe` ("a pinned expert has no host copy, so there is no
+read-through fallback" — startup aborts otherwise). Check BEFORE downloading
+anything:
+
+```bash
+podman run --rm --device /dev/kfd --device /dev/dri --device /dev/mem \
+  -e HIP_VISIBLE_DEVICES=0,1 --entrypoint bash $IMAGE -c '
+  nm -D /app/r4dhip/r4d.so | grep -oE "gemm_moe_[a-z0-9_]+" | sort -u
+  grep -rl offload_recipe /opt/vllm/lib/python*/site-packages/vllm/model_executor/layers/fused_moe/'
+```
+
+| quant | kernel in `r4d.so` | experts class | verdict |
+|---|---|---|---|
+| FP8 | yes | `Fp8HipMoEExperts` | **serves** |
+| MXFP4 / DSFP4 | yes (`gemm_moe_mxfp4a8_*` / `dsfp4a8_*`) | `R4dMxfp4MoEExperts` / `R4dDsfp4MoEExperts` | **serves** |
+| W4A16 | **no** `gemm_moe_w4a16_*` | `R4dW4a16MoEExperts` exists | silently falls back to generic `TritonWNA16Experts`, no recipe → startup halts |
+| RFA | none | no class at all | out |
+
+Also confirm the architecture routes natively with NO overlay
+(`ModelRegistry.get_supported_archs()` for `Qwen4Exp`/`Glm5`). The stock image
+natively carries Qwen3.8/Qwen4Exp and GLM-5.x including MTP heads. **Do not
+apply vendor overlays** against a fast-moving pre-release image — the breakage
+misleads (a missing symbol in the overlay's OWN copy of a module masks the
+real cause, then moves to another symbol each patch;
+`cannot import name 'checkpoint_has_lm_head'` = overlay clobbered
+`transformers_utils/config.py`). The image ships quantizers for
+`rfa|rfi|mxfp4|mxfp4ocp|paro5` under `/app/tools/quantize.py`
+(`--fp8-attn --quant-only --dry-run` first) if a re-quant is needed — note it
+can run MXFP4/DSFP4/FP8 but only PRODUCE rfa/rfi/mxfp4/paro.
+
+## 2. Fetch with verification, always
+
+Gated repos: token never in argv (`pgrep -a curl` prints command lines).
+Build a mode-600 curl config once:
+
+```bash
+{ printf 'header = "Authorization: Bearer '; cat ~/.cache/huggingface/token; printf '"\n'; } \
+  > ~/.cache/huggingface/curl.conf && chmod 600 ~/.cache/huggingface/curl.conf
+curl -K ~/.cache/huggingface/curl.conf -f -sS -L -C - -o "$out" "$url"
+```
+
+`--fail` is MANDATORY — `curl -o FILE URL` exits 0 on HTTP 401/403 and writes
+the error body as a "successful artifact" (a gated repo once yielded three
+153-byte shards with identical sha256s and a cheerful DONE banner). `-C -`
+resumes torn shards; HTTP/2 cancels leave silently truncated files, so
+re-verify EVERY shard's byte count against the Hub manifest (and the LFS
+sha256) before serving. The AGENTS.md Secret Rules carry the token-hygiene
+law; [[frank-medium-model-swap]] has the same detail for GGUFs and
+`.partNofM` fragments.
+
+## 3. Serve — the known-good invocation and its traps
+
+```bash
+sudo sh -c 'sync; echo 3 > /proc/sys/vm/drop_caches'
+podman run -d --name <ctr> --ipc=host --ulimit memlock=-1:-1 \
+  --cap-add SYS_PTRACE \
+  --device /dev/kfd --device /dev/dri --device /dev/mem \
+  -e HIP_VISIBLE_DEVICES=0,1 -e VLLM_ROCM_USE_AITER=0 \
+  -v /home/dank/models/<model>:/models:ro \
+  $IMAGE /models --served-model-name <name> --port 7770 \
+  --tensor-parallel-size 2 --max-model-len 65536 \
+  --kv-cache-memory-bytes 2147483648 \
+  --max-num-seqs 4 --gpu-memory-utilization 0.95 \
+  --enable-prefix-caching --trust-remote-code
+```
+
+- **Entrypoint already execs `vllm serve "$@"`** — pass the model path as the
+  first argument and do NOT pass `serve` (podman then tries to pull `serve`
+  as an image name). `-e VAR=1` must come BEFORE the image name.
+- **Mount the whole `/dev/dri`, never individual render nodes** (§4).
+- `--ipc=host` and `--shm-size` are mutually exclusive in podman.
+- `podman run -d` without `-a` sends container stdout nowhere — read
+  `podman logs <ctr>`; and vLLM writes `Application startup complete` to
+  **stderr**, so a `podman logs X 2>/dev/null | grep -q` readiness gate never
+  fires.
+- Start at 65536, not the native ceiling; prove the plan, then scale.
+- Load takes ~5 min with `/health` refusing the whole time — do not kill the
+  process (deploy.sh waits 20 min).
+
+## 4. Container GPU-init triage (ordered)
+
+1. **Enumerate-vs-context trap (most common):** individual
+   `renderD*` nodes give HIP the render nodes WITHOUT the primary `card*`
+   nodes — it enumerates agents (`device_count()` returns N) and then refuses
+   every context: `RuntimeError: No CUDA GPUs are available` on the first
+   real op. A/B-confirmed 2026-09-19. Rule: `--device /dev/kfd --device
+   /dev/dri --device /dev/mem`; restrict GPUs with `HIP_VISIBLE_DEVICES`,
+   never with device nodes.
+2. **`device_count()` is not evidence.** Prove usability with a real op
+   (`(a@a).sum().item()` on a fresh `torch.randn`), then repeat via
+   `mp.spawn`, `mp.fork`, and `subprocess.run([sys.executable, ...])`, and
+   cross-check `rocm-smi --showmeminfo vram`.
+3. **memlock rlimit → a MISLEADING CUDA OOM.** Host default 8192 KB; a pin
+   larger than the limit fails as
+   `torch.AcceleratorError: CUDA error: out of memory` raised from
+   `pin_memory()` — it names the accelerator but the fault is host pinned
+   memory. It cannot be raised from inside a container or even from root's
+   own shell (`Operation not permitted`); only PAM applies it at login, so
+   fixes only affect NEW sessions:
+   `/etc/security/limits.d/99-memlock.conf` with
+   `* - memlock unlimited` + `root - memlock unlimited`; the container needs
+   `--ulimit memlock=-1:-1` to inherit. Discriminating test after the fix:
+   the container reports `unlimited` AND small pins succeed; if pins still
+   fail, memlock was never the cause.
+4. **`/proc/meminfo` TAB trap:** the key/value separator is a TAB, so
+   `grep MemAvailable /proc/meminfo | tr -s " " | cut -d" " -f2` returns
+   EMPTY and a following awk prints `0.0` — a false "host out of memory" on a
+   host with hundreds of GiB free, which invites aborting something healthy.
+   Tab-safe: `awk -v k='MemAvailable:' '$1==k {printf "%.1f", $2/1048576;
+   exit}' /proc/meminfo`, cross-check `<= MemTotal`, refuse to print a zero.
+   Use `scripts/dankstatus.sh` instead of hand-rolling.
+5. **Silent-parse traps:** `grep -c PATTERN file || echo 0` yields TWO lines
+   (grep prints 0 AND exits 1) — `[: 0\n0: integer expected`; use
+   `| head -1` or `|| true`. `systemctl is-enabled` can emit multiple lines.
+   `rocm-smi --showmeminfo vram --csv` column order is `device, VRAM Total
+   Memory (B), VRAM Total Used Memory (B)` — reading $3 as total/$4 as used
+   yields 0.00; filter to `^card[0-9]+$`.
+6. **Guard the box:** long loads with co-tenants can drive RAM to 0 — abort
+   before the OOM killer (`MemAvailable < 4 GiB → podman rm -f`, exit 1).
+7. **Drop page cache before a large load** (`echo 3 > /proc/sys/vm/drop_caches`)
+   and confirm the drop was actually in effect for the run before treating a
+   failure as a capacity wall — a stale "it doesn't fit" from a contaminated
+   run is expensive to unwind.
+
+## 5. Memory budgeting: the levers are not where they look
+
+The planner builds `budget = total × util`, subtracts a KV reserve, and gives
+the REMAINDER to the expert arena. Consequences, all measured:
+
+- **`--gpu-memory-utilization` grows the ARENA, not the KV pool** (KV stayed
+  0.64 GiB at util 0.95 and 0.98). The arena is the only elastic term.
+- **Lowering `--max-model-len` is the WRONG lever and backfires:** a smaller
+  context shrinks the reserve, handing the space to the arena — measured 65536
+  → 0.64 GiB KV, 32768 → 0.34 GiB KV.
+- **Hybrid-attention undercount:** the planner's per-token KV figure (9190
+  B/token for Qwen4Exp) misses the large fixed per-sequence Mamba state
+  (~2x), so the reserve is always short and the arena silently covers it.
+  Fix by pinning the pool: `--kv-cache-memory-bytes <BYTES>` (2 GiB →
+  113,595 tokens / 3.47x concurrency at 32k). For long context,
+  `--kv-offloading-size` (GiB) spills KV to host.
+- **`--expert-offload-mem` caps the HOST side, not the VRAM arena**
+  (`cap = int(x * GIB)` against `host_memory_available()`). Do not reach for
+  it to fix a VRAM problem; to grow the arena, shrink the KV pin or raise
+  util (see [[dankserver-qwen38-fp8-expert-offload-tuning]] for the
+  util=1.0 unbootability law).
+
+**Read the plan — it is the real acceptance test:**
+
+```
+expert offload: AUTO-ENABLED, weights 62.54 GiB per rank exceed the 30.27 GiB budget (TP2)
+  budget (total x util)      30.27 GiB
+  non-expert weights          6.28 GiB
+  KV reserve                  0.36 GiB   <- sanity-check against the model's own per-token figure
+  CUDA graphs                 0.66 GiB
+  runtime overhead            7.60 GiB
+  expert cache space         15.36 GiB   6712 slots of 2.344 MiB
+  host-backed experts/rank   52.74 GiB   23040 experts
+  host: 105.48 GiB across 2 ranks of 160.89 GiB usable
+```
+
+Host fits when `host_total < usable`. The PLE n-gram table rides memfd and is
+SHARED across TP ranks.
+
+**Loading mechanism** (for reasoning about peak RAM): weights never materialise
+in VRAM — expert Parameters start as CPU staging tensors; each layer's
+`weight_loader` counts shard events and, on layer completion,
+`ExpertTopology.split_layer` moves that layer's experts into the VRAM arena
+and compacts the rest into exact-size pinned host tensors behind UVA. Peak
+host RAM = `host-backed set + layers still loading`, not the whole model. At
+runtime `apply_hot_cold` sends arena-resident experts down a hot GEMM and the
+rest straight over PCIe via UVA views — no staging copy in the critical path.
+
+**Emulation check:** a dequantize-to-compute path doubles real host RAM (118
+GiB offload request → ~238 GiB) and is invisible in the weight count. Read the
+log for `Using 'EMULATION' Mxfp4 MoE backend` before concluding "doesn't fit".
+
+## 6. Performance profile and honest measurement
+
+Decode on this pair is **PCIe-gather-bound, context-independent**: 12.11
+tok/s FLAT from 304 to 16,095 context tokens — KV attention is not the cost.
+With ~29 % of experts arena-resident, ~0.84 GB crosses PCIe per token at the
+engine's effective read rate; cold-vs-warm differing ~10 % confirms the LRU
+works; growth is linear in hit rate, and only speculative decode cuts
+sequential STEPS rather than making each cheaper.
+
+**Benchmark traps that produced fake numbers here:**
+
+- `max_tokens=1` is not TTFT (the server can satisfy it without doing the
+  work — prefill looked flat at 1.26 s for 16k AND 30k prompts). A 64-token
+  non-streamed reply is time-to-64-tokens (decode-bound, flat ~9.6 s at every
+  length). Correct TTFT: stream and time the FIRST content-bearing chunk.
+- A small repeating word loop is prefix-cacheable AND n-gram-predictable — a
+  43-word cycle reported 22,897 tok/s prefill, almost entirely cache hit. Use
+  a large vocabulary in shuffled order and print `cached_tokens`.
+- ~1.2 s fixed per-request overhead swamps prefill below ~16k tokens; compare
+  prefill at 30k+ only.
+- Size prompts with `/v1/messages/count_tokens` (a chat call for counting
+  pollutes the metric); guessing words-per-token silently overran the window
+  (HTTP 400).
+- Always run a canary: plant a unique token at the END of a long prompt and
+  ask the model to quote it — proves prompts reach the model at all.
+- Measure warm AND long (`podman exec` + urllib at 127.0.0.1:7770): a 33-token
+  reply read 14.6 cold / 20.3 warm; a 768-token generation hit 25.7; two
+  concurrent streams scaled to 47 aggregate.
+
+**Levers (all opt-in):** `--expert-prefetch N` takes an INT depth, not a
+boolean (default 0; at depth 1 only ~10 % cold-vs-warm difference — the cache
+already works); `CLAV_TUNABLEOP_SWEEP=1` autotunes GEMM shapes (silently
+skipped under expert offload unless `CLAV_TUNABLEOP_WITH_OFFLOAD=1` too —
+see [[dankserver-qwen38-fp8-expert-offload-tuning]] for why it was rejected);
+`--expert-precise-memory` swaps the fitted overhead figure in.
+
+## Related
+
+- [[dankserver-qwen38-fp8-expert-offload-tuning]] — deployed shape, plan law,
+  measured campaigns
+- [[PCIe Link and Atomics]] — why decode is gather-bound and what cannot fix it
+- [[RDNA4 MoE Expert Offload VRAM Budget]] — arena/plan arithmetic deep-dive
+- [[frank-large-benchmark]] and [[frank-small-benchmark]] — window discipline on live hosts

serverAppConfigs-docs/LLM/RDNA4 MoE Expert Offload VRAM Budget.md

@@ -0,0 +1,219 @@
+---
+type: reference
+title: RDNA4 MoE Expert Offload VRAM Budget
+description: The expert-offload plan arithmetic on the tcclaviger fork — which terms a deploy can move, the util law, the CUDA-graph charge, what free VRAM really is, knife-edge survivability, parallelism ceilings and the context ladder.
+tags: [vllm, expert-offload, vram, arena, kv-cache, ladder, tcclaviger]
+timestamp: 2026-10-05T00:00:00-07:00
+---
+
+# RDNA4 MoE Expert Offload VRAM Budget
+
+For VRAM-bound vLLM on the tcclaviger RDNA4 fork (frank-primary: Qwen3.8-
+Flash-Next FP8, TP=2, 2x R9700 at 31.86 GiB each, expert offload to host RAM).
+Tuning history and the deployed shape live in
+[[dankserver-qwen38-fp8-expert-offload-tuning]]; boot-failure capture lives in
+[[vLLM Boot Forensics and Config Interrogation]].
+
+## 1. The plan law
+
+```
+arena = budget − non_expert − kv_reserve − cuda_graphs − runtime_overhead
+budget = device_total × gpu_memory_utilization          # expert_plan.py build_plan
+slots  = arena // expert_bytes                          # 2.344 MiB/slot ⇒ 437 slots/GiB
+```
+
+**The arena is DEVICE-resident, so the plan predicts reality:** the fp8
+3-window shape's plan totalled 28.27 GiB and `rocm-smi` measured exactly
+28.27 GiB. `weights + pinned_pool + graphs + arena + non-torch (≈1.45 + 0.57
+driver reserve)` vs 31.86 GiB predicts a boot failure before it happens.
+
+Which terms a deploy can move:
+
+| term | reachable? | how |
+|---|---|---|
+| budget (util) | **yes, law-capped** | `--gpu-memory-utilization`; +0.01 = +0.32 GiB on 31.86 GiB cards |
+| CUDA graphs | **yes** | container env `VLLM_CUDAGRAPH_MEM_BASE_MB` / `_PER_GRAPH_MB` (or `_ESTIMATE_MB` for an explicit total) |
+| non-expert weights | no | checkpoint census |
+| KV reserve | no | registry constant: 9190 B/token × max_model_len × 1.08 + per-seq mamba pages |
+| runtime overhead | partly | `--expert-precise-memory` swaps default 7.60 GiB for the registry-fitted 5.25 GiB |
+
+Anything else is an IMAGE EDIT, not a deploy — say so; do not invent a flag
+(verify with `podman exec <ctr> grep -n getenv …/offloader/expert_activation_registry.py`).
+
+**UTIL LAW: 0.982, NEVER 1.00 (measured 2026-10-03).** The CLI field is
+`Field(gt=0, le=1)`, but the worker init demands `free > budget`: at 1.0
+budget == device total and BOTH ranks raise
+`ValueError: Free memory on device cuda:N (31.29/31.86 GiB …)` BEFORE a
+single weight loads — a crash loop (NRestarts ~2 min apart, zero payload
+progress). Not a knife-edge: **unbootable**. Even bootable near-ceiling
+shapes die under real load (0.13 GiB free bf16 boot; `0 bytes free` on a real
+995-token request).
+
+## 2. The CUDA-graph term: charge the measured capture
+
+Defaults (256 MiB base + 52 MiB/graph) over-charge badly. Measure with the
+engine's own post-capture line `Graph capturing finished … took X GiB` (per
+rank; heavier rank binds). Measured on the no-spec piecewise plan: 0.31/0.17
+GiB, and one boot reported a NEGATIVE −0.06 GiB delta (capture fit inside
+pages the profiler had already counted) — the analytic base pool is pure
+over-charge. Shipped: `BASE_MB=0`, `PER_GRAPH_MB=12` = 0.07 GiB — per-graph,
+so a spec-decode shape (18 graphs) still pays (0.21 GiB). An explicit
+`_ESTIMATE_MB` total does NOT scale with ladder changes; base+per-graph does.
+
+**Verify every deploy:** `Estimated CUDA graph memory` line vs the `Graph
+capturing finished` line; if capture exceeds the charge, raise `_PER_GRAPH_MB`
+back toward 52. Check `VLLM_CUDAGRAPH_*` actually present in
+`podman exec … env`.
+
+## 3. The pool is invisible to the plan; free VRAM is the estimator gap
+
+`--kv-cache-memory-bytes` does NOT appear in the arena arithmetic — the plan
+reserves the registry price, not the actual pool. Raising the pool consumes
+real free VRAM without growing the arena; the registry over-prices the pool
+(9190 B/token planned vs 7,891.8 measured at 200704 fp8 — 7,701 at 262144;
+**never reuse a B/token ratio across geometries**, and subtract ~3.87 MB
+block-alignment loss when converting a token target to bytes). The ask lands
+block-aligned: 3,166,500,000 B → 401,408 tokens; verify with
+`GPU KV cache size: N tokens, Maximum concurrency … X.Xx`.
+
+**Driver-reported free VRAM after hours of traffic is NOT allocatable
+arena.** It is the estimator gap: the registry over-prices KV and charges
+5.25 GiB overhead against ~2.5 measured; torch's allocator does not return
+slabs, so it reads as the high-water mark. The only reachable levers are util
+(within the law) and the graph charge; everything else needs a registry edit
+inside the image — the fork's own log line says as much.
+
+**MNBT is arena-free up to the anchor:** registry overhead =
+`base + 0.22 GiB × max(0, (nbt−4096)/1024)` anchored at nbt 4096, so rungs
+1024/2048/4096 are plan-identical (the fitted 5.25 GiB was fitted AT 4096).
+MNBT changes re-key the compile range (one recompile, ~24 s at (1,2048) with
+the mounted cache). 512 doubles TTFT at 4064 ctx; 12288 costs ~2.5 % plus
+arena. MNBT does not move the KV pool.
+
+## 4. Registry "compile headroom" is not reclaimable slack
+
+`--expert-precise-memory` selects the fitted overhead (5.25 GiB, this arch at
+TP=2) over the default (7.60 GiB = fitted + **2.35 GiB of compile and
+graph-capture headroom** the registry documents as "transient VRAM the
+profiler never sees"). It moves exactly one term — the plan's `room` — so the
+arena grows by that 2.35 GiB (~1,025 slots). **That headroom is the budget
+for the phase that allocates LAST**: reclaiming it requires paying elsewhere
+(smaller pinned pool, fewer sessions, shorter context). Measured 2026-10-01:
+bf16 2×262,144 + precise put the total at 31.73 of 31.86 GiB and the boot
+crash-looped at graph capture; dropping the flag restored ~2.4 GiB and booted.
+
+## 5. Boot ≠ survival (knife-edge shapes)
+
+A shape can pass EVERY boot gate — weights, profile, KV allocation, graph
+capture, `/health` 200 — and die on the first real request. Measured
+2026-10-01: `5 × 262144 @ fp8` booted (KV 1,350,182 tokens / 5.15x) and was
+killed seconds later by a 995-token prompt with `max_tokens=261149`:
+`torch.OutOfMemoryError: Tried to allocate 50.00 MiB … 0 bytes free`. After
+any knife-edge change: send a LARGE-`max_tokens` request before calling it
+done, and read post-boot free VRAM per rank (`rocm-smi --showmeminfo vram`)
+— the survivor had ~3.6 GiB free; the corpse had 0.
+
+**KV accounting — pin the pool BYTES, floor-sized:** set
+`--kv-cache-memory-bytes` PER RANK to the contract floor
+(`seqs × MAXLEN × measured B/token`), not the native window — extra pool bytes
+come out of the arena/headroom and buy nothing. Measured B/token on this
+hybrid: **7,701 fp8 / 13,839 bf16** (decomposed: attention 6,138/12,276 +
+dtype-independent mamba/side 1,563). Measured rates are shape-dependent —
+7,891.8 at the 200704×3 shape. Capacity does NOT scale with `SEQS`: the
+planner's KV reserve is for ONE max-model-len request.
+
+**Parallelism ceilings you cannot flag past:**
+
+- `--decode-context-parallel-size` refused for GQA when `TP ≤ num_kv_heads`:
+  *"requires TP (2) to be greater than the model's total number of KV heads
+  (2)"* — DCP shards KV along the sequence dimension AFTER KV heads spread
+  across TP ranks (it reuses TP ranks, not TP×DCP), so a 2-KV-head model
+  needs TP ≥ 4 = four GPUs.
+- EP (`--enable-expert-parallel` + `--all2all-backend`) exists but REPLACES
+  the fork's local tiered-offload gather with a per-layer PCIe all-to-all.
+- DP requires a full model copy per replica — on a model larger than the
+  card it is impossible, not slow.
+
+## 6. The context ladder (find the max bootable window by measurement)
+
+"Max possible" = the largest value that BOOTS, measured — never the model's
+spec maximum, never hand arithmetic.
+
+1. **Scale the pool with the window.** If KV_BYTES is pinned, it must shrink
+   in lockstep with MAXLEN or the rung tests nothing:
+   `KV_BYTES = round(KV_BYTES_ref × (seqs×MAXLEN_new) / (seqs×MAXLEN_ref))`
+   off a booted reference rung. Verify the rung's capacity still covers
+   `seqs × MAXLEN` from the engine's own line.
+2. **Boot under the PRODUCTION environment, not a shell probe** — a rootless
+   shell lacks `LimitMEMLOCK=infinity` and can pass where production OOMs.
+   Transient unit:
+   `sudo systemd-run --uid=<svcuser> --unit=probe-$ML -p LimitMEMLOCK=infinity -p Restart=no env <ENVFILE_OVERRIDE> <serve-script>`
+   pointing the serve script at a scratch env file via its `*_ENV_FILE` hook.
+3. **Judge on health + unit liveness:** poll `/health` (200 up / 503 loading
+   / 000 not bound); break early when the unit goes inactive — that is the
+   failure signal, not a timeout.
+4. **Capture the engine log per rung BEFORE teardown** (`podman logs` dies
+   with `podman rm`; a `podman logs` issued too soon after `podman run -d`
+   fails with *no container with that name* — record UNKNOWN, never FAIL).
+5. **Ladder down on FAIL, one bonus step UP on PASS.** Measured steps on this
+   box: OOM −16,384 tokens, bonus +4,096. Tiny margins (76 MiB class) mean
+   the next rung down almost certainly passes: `Tried to allocate 76.00 MiB …
+   0 bytes free … 30.43 GiB allocated by PyTorch` died in graph CAPTION;
+   `Tried to allocate 10.28 GiB … 9.46 GiB free` died in KV ALLOCATION.
+6. **Clean between rungs:** stop the transient unit, `podman rm -f`, kill
+   orphans by explicit PID
+   (`ps aux --sort=-rss | awk '/vllm\/bin\/python3/ && $6 > 50000000 {print $2}'`),
+   wait for `MemAvailable` to recover.
+
+**The hazard that cost production:** a leftover ladder script whose teardown
+`podman rm -f`s the production container fires against a freshly DEPLOYED
+engine. Teardown runs ONLY on failure, never after a pass; kill the whole
+scratch process tree and verify none remain BEFORE deploying. On PASS: print
+the measured `GPU KV cache size / Maximum concurrency` line, LEAVE IT
+SERVING, and pin the number into the repo (env + seam + harness
+contextWindow) before deploying.
+
+Cutover order: ladder → winner passes and serves → repo env + seam + harness
+→ tests → commit+push → verify no scratch scripts remain → stash foreign work
+(`git stash -u`) if the tree is dirty → deploy (one restart) → verify health
++ engine flags in the log.
+
+**Measured ladder on this box (200704×3, fp8 KV, precise memory):**
+
+| shape | util | graphs | arena | result |
+|---|---|---|---|---|
+| E | 0.98 | 0.55 | 17.22 GiB / 7,522 slots | boot ok |
+| F | 0.982 | 0.32 | 17.52 GiB / 7,653 slots | boot ok |
+| 1.0 | 1.00 | 0.07 | — | UNBOOTABLE (worker-init ValueError, both ranks) |
+| G (deployed) | 0.982 | 0.07 | 17.77 GiB / 7,762 slots | KV 401,408 tokens / 2.00x; 0.89 GiB real free/card held through a live 4,096-token generation at ~38–39 tok/s |
+
+bf16 ceilings at util 0.98: 3 seqs → 241,664 passes / 245,760 and 262,144 OOM
+at graph capture; 6 seqs fp8 → 10.70 GiB ask with 9.18 GiB free (fails) —
+~207k per context max.
+
+## 7. Arena audit procedure (when asked for "more experts")
+
+1. Read the live plan before proposing anything — `grep -oE` with a
+   digit-anchored tail (plain grep/journalctl truncate these columns to
+   `17.`):
+
+   ```
+   ssh dankserver 'sudo journalctl CONTAINER_NAME=frank-primary-qwen -b -o cat --no-pager \
+     | grep -aoE "(budget \(total x util\) +[0-9.]+|expert cache space +[0-9.]+ GiB +[0-9,]+ slots|Estimated CUDA graph memory.{0,80}|Graph capturing finished.{0,40}|GPU KV cache size.{0,90})"'
+   ```
+
+2. Classify each term (§1 table); take only the two reachable levers.
+3. Verify after deploy: plan lines show new budget/graphs/arena;
+   `GPU KV cache size` matches the ask; graph-estimate vs post-capture;
+   `rocm-smi` high-water per card = the real margin;
+   `systemctl show frank-primary -p NRestarts --value` = 0; a real chat
+   completion from a PEER (health-probe via `192.168.0.235:7770/health`, not
+   `localhost` inside an ssh session on the host — the latter returns `000`
+   and looks like an outage).
+
+## Related
+
+- [[RDNA4 Container Serving]] — the bring-up path and container traps
+- [[vLLM Boot Forensics and Config Interrogation]] — capture and contracts
+- [[dankserver-qwen38-fp8-expert-offload-tuning]] — measured history
+- [[Llama Cpp Serving Sizing]] — the llama.cpp equivalent

serverAppConfigs-docs/LLM/Tcclaviger Fork Ops.md

@@ -0,0 +1,264 @@
+---
+type: reference
+title: Tcclaviger Fork Ops
+description: Operating the tcclaviger RDNA4 vLLM fork on dankserver (shape G era, rollback target since the 2026-10-05 FreeToken cutover) — the ad-hoc sweep harness, measured knob verdicts, sweep hygiene, the large-MoE serving recipe, and compile-cache persistence.
+tags: [tcclaviger, vllm, dankserver, expert-offload, sweeps, cache]
+timestamp: 2026-10-05T00:00:00-07:00
+---
+
+# Tcclaviger Fork Ops
+
+The tcclaviger RDNA4 vLLM fork (`docker.io/tcclaviger/vllm@sha256:da4029666…`,
+vLLM 0.29.0.dev, GLM-5.3/Qwen-specialised with native MoE quant kernels and a
+first-class expert-offload planner) served `frank-primary` at shape G until
+the 2026-10-05 FreeToken cutover; it remains the rollback target
+(`git tag frank-primary-vllm-shapeG-20261003`). Full plan law and VRAM
+arithmetic: [[RDNA4 MoE Expert Offload VRAM Budget]]; boot forensics:
+[[vLLM Boot Forensics and Config Interrogation]]; window discipline:
+[[LLM Bench Windows and Measurement]].
+
+## The ad-hoc sweep harness (dankserver)
+
+Serve script `/home/dank/serve_qwen_fp8.sh` with env hooks: `SPEC` (MTP
+depth), `TUNABLE` (TunableOp), `PREFETCH`, `UTIL`, `MAXLEN`, `KV_BYTES`,
+`MNBT`, `UVA_TOK`, `GATHER_C/GATHER_L` (R4D_LRU gather), `TELEM`
+(R4D_LRU_TELEMETRY), `PROFDIR` (torch profiler bind-mount + --profiler-config
+JSON). The script re-copies the harness (`mtp_measure.py`, `bench_iter.py`)
+into every fresh container — `podman cp` files die on container recreate.
+
+- Measure ONLY via usage counters: `mtp_measure.py` (3 depths) or
+  `mtp_measure_short.py` (166 ctx). `decode = completion_tokens/(wall −
+  ttft)`; chunk counting is INVALID under spec decode.
+- Jobs: `bash /home/dank/job.sh start NAME CMD…` / `wait NAME SECS 40` /
+  `out NAME` ([[Remote Job Runner]]).
+- **Validity check on ANY speed result:** `completion_tokens == max_tokens`
+  (else repetition-halt or early EOS), finish_reason, and inspect the text —
+  `bench_iter` prompts are repeated word-lists + `!` padding, loop-prone. A
+  loop makes LRU hits ~100% → PCIe traffic vanishes → fake 3–4× decode.
+  Acceptance swings 48–61% run-to-run — noise; decode is the metric.
+- Each arm = full reload: ~8.5 min cold, ~12 min with measurement.
+- Readiness gate: `podman logs <c> 2>&1` (stdout+stderr merged) for
+  `Application startup complete` — it is on STDERR; an unmerged grep can
+  never match and burns the whole wait timeout.
+- GPU util ~0% during load/compile is NORMAL (weights + CUDA-graph capture
+  are CPU-side); VRAM climbing is the load indicator.
+- Config edit discipline: the serve script is patched by scp'd python
+  scripts with asserts (anchor must exist, addition must not); `bash -n`
+  after every patch.
+
+## Measured knob verdicts (2026-09-29 campaign)
+
+Incumbent: `SPEC=4 PREFETCH=1 UTIL=0.98 MAXLEN=32768 KV_BYTES=2147483648
+TUNABLE=0`, stock gather 8×16 → 26.80/25.15/26.81 tok/s @ 166/1087/4064 ctx.
+
+| Knob | Verdict | Evidence |
+|---|---|---|
+| Gather geometry `R4D_LRU_GATHER_{CHUNKS,LANES}` | **RULED OUT** | 16×32 worse (~24); 32×64 tie (26.6). The 9.5 vs 22.7 GB/s read-path gap is per-transfer structure (~105 µs/slab vs 78 ideal), not a tunable knob |
+| MTP depth | **RULED OUT** | SPEC=3 ~9% worse; SPEC=6 collapses (13.5–19 tok/s, acceptance 17.4%) — also proves UVA read-through (crosses at step tokens > `R4D_LRU_UVA_TOKENS=5`) is SLOWER than copy-in |
+| TunableOp | **REJECTED** | `CLAV_TUNABLEOP_SWEEP=1` silently SKIPPED under expert offload unless `CLAV_TUNABLEOP_WITH_OFFLOAD=1`; adds ~7 min/load, persists nothing (container-local), and CORRUPTS NUMERICS (loops, repetition-halt, fake 79.8 tok/s @ 96.5% acceptance on degenerate completions). Any "3× win" with early-EOS + absurd acceptance = the loop artifact |
+| MNBT | neutral at 8192 vs 2048 | small values re-chunk prefill only (hurts TTFT, never decode — decode step = max-num-seqs × (spec+1) tokens). No ubatch concept in vLLM |
+
+Decode on this stack is PCIe-gather-bound (~30% GPU duty cycle), so
+batch-shaped compute knobs move ~nothing.
+
+The `R4D_*/CLAV_*` knob family (QSA_*, MHC, ROUTE, ACT_QUANT, AR_QUANT,
+SILU_QUANT) are kill-switches defaulting ON, not perf dials; real levers are
+the R4D_LRU_* family, TunableOp (rejected), R4D_LRU_TELEMETRY, and spec
+depth.
+
+Telemetry gotchas: `--profiler-config` takes a JSON blob, not separate
+flags; `torch_profiler_with_stack` defaults TRUE — a full measure inside a
+profile window OOM-kills the engine worker (~40 GB trace buffer; kernel
+kills python3 at ~95 GB anon-rss). Keep windows to ~15 s / one depth
+(`mtp_measure_short.py`). `telemetry_report` (`expert_cache_rank<N>.json`)
+fires ONLY from the worker's profiler-stop path; worker-side stop can hang
+>5 min on ROCm/kineto. No alternative trigger exists. The LRU hit rate
+measured 82.45% — NOT the ~27% the arena-fraction arithmetic implies;
+temporal reuse dominates, so never infer miss rates from resident-slot
+fractions.
+
+## Sweep hygiene (fork-specific preflight)
+
+1. **Knob landed:** for each `-e` env knob, confirm in-container
+   `podman exec … env` (or the engine log's confirmed-values line). A knob
+   the fork silently ignores produces a "parity" arm that is really the
+   default — the v1 gather sweep lost all three arms to exactly this
+   (GATHER_C/GATHER_L silently dropped, all arms ran stock 8×16).
+2. **Prefetch uniformity:** every arm in a sweep MUST carry the same
+   expert-prefetch setting. The 2026-09-29 campaign measured losing arms
+   with prefetch ON and incumbents OFF; untaxed, its verdicts inverted
+   (no-spec 12.11→41.47; gather 32×64 parity→+18.7%). Never re-read a taxed
+   verdict.
+3. **Spec arms must prove engagement:** engine log
+   `SpecDecoding metrics: Mean acceptance length …` and
+   `curl :7770/metrics | grep vllm:spec_decode_`. Zero such lines = MTP off.
+4. Boot-fail fast-detection: poll `podman ps` INSIDE the health loop; a
+   refused boot dies in one poll instead of 20 min of health waits. Capture
+   >8 lines of `podman logs` on failure — the root cause (e.g. a SEQS init
+   refusal) lives higher up.
+5. Report per-depth tables, never one number: g32×64's win grows with depth
+   (54@1k, 39.5@short) — a single 4064 median hides the short-ctx cost. TTFT
+   rows are noisy ±0.15 s and non-monotonic across depths; compare same-ctx
+   rows across arms only.
+6. Generate ≥256 tokens per sample (at 8 tokens inter_token is first-token
+   noise).
+
+Boot-log facts survive measure failures: KV pool size (`GPU KV cache size`),
+arena slots (`expert cache: … -> N slots`), and the spec-reservation ladder
+(MTP3 766,666 / MTP4 729,545 / MTP6 665,263 tokens) print at BOOT — a
+measure-stage bug costs decode tables but not boot-log findings.
+
+**MTP depth-brittleness law (2026-10-02, all arms rc=0):** MTP3 on this fork
+is depth-brittle — 47.01 t/s at 166 tokens (47.88 at temp 0.7) but 23.09 at
+4064, vs no-spec baseline 37.02 / 33.99. MTP3 + SEQS 16 + MNBT 4096 + util
+0.95 + ladder [4..32] = best overall 37.04 / 38.61 t0.7 and keeps 33.15 at
+4064. `--expert-precise-memory` is load-bearing for MTP (same arm with it
+OFF = worst arm, 19.81 at 4064). Every spec arm costs KV (398,658 → 313,002
+tokens). Acceptance is real: mean length 2.4–2.8, per-position
+0.76/0.55/0.33, avg 52–57%. Next lever named by the engine: MNBT 4096 is
+exactly SEQS×(n+1) (`suboptimal` warning) → try 8192.
+
+## Serving a model larger than VRAM (dankserver recipe)
+
+Dankserver: 2× R9700 (gfx1201), 31.86 GiB each = 63.7 GiB real VRAM, 246 GiB
+RAM. `torch.cuda.device_count()` returns 3 and summing `total_memory` gives
+a bogus 186.7 GiB (the Raphael iGPU is in the list — see
+[[Llama Cpp Serving Sizing]] for the reading discipline).
+
+**Kernel gate — this decides which checkpoint can work at all.**
+`/app/r4dhip/r4d.so` exports `gemm_moe_mxfp4a8_*` and `gemm_moe_dsfp4a8_*`
+but NO `gemm_moe_w4a16_*`. Expert offload requires an experts class with
+BOTH a compiled kernel AND an `offload_recipe`; the planner aborts startup
+otherwise. Usable: MXFP4, DSFP4, FP8-HIP (`R4dMxfp4MoEExperts`,
+`R4dDsfp4MoEExperts`, `Fp8HipMoEExperts`). `R4dW4a16MoEExperts` has no
+kernel (the class exists, the r4d.so entry does not); RFA has neither. Check
+the checkpoint's quant against this list BEFORE downloading anything.
+
+Entrypoint is `/app/tools/image_entrypoint.sh`, ending `exec vllm serve
+"$@"` — pass the model path as the FIRST argument and do NOT pass `serve`.
+Known-good invocation:
+
+```bash
+sudo sh -c 'sync; echo 3 > /proc/sys/vm/drop_caches'
+podman run -d --name <ctr> --ipc=host --ulimit memlock=-1:-1 \
+  --cap-add SYS_PTRACE \
+  --device /dev/kfd --device /dev/dri --device /dev/mem \
+  -e HIP_VISIBLE_DEVICES=0,1 -e VLLM_ROCM_USE_AITER=0 \
+  -v /home/dank/models/<model>:/models:ro \
+  docker.io/tcclaviger/vllm:latest /models \
+  --served-model-name <name> --port 7770 \
+  --tensor-parallel-size 2 \
+  --max-model-len 32768 \
+  --kv-cache-memory-bytes 2147483648 \
+  --max-num-seqs 4 \
+  --gpu-memory-utilization 0.95 \
+  --enable-prefix-caching --trust-remote-code
+```
+
+`-p 7770:7770` is not included, so the API is only reachable via
+`podman exec` — add `-p` if host-side curl is needed. Load ~5 min;
+`/health` answers connection-refused the whole time; watch for
+`Application startup complete` in `podman logs <ctr>`. Verified working:
+`orcarouter/Qwen3.8-Flash-Next-Uncensored-FP8` (gated; 172.8 GiB, 131
+shards, `Qwen4ExpForConditionalGeneration`, 48 layers, 512 experts top-10,
+multimodal, native 262,144).
+
+Traps that each cost a boot:
+
+1. **Mount the whole `/dev/dri`, never individual render nodes.** Render
+   nodes without the primary `card*` nodes → HIP enumerates two agents and
+   refuses a context on either (`RuntimeError: No CUDA GPUs are available`)
+   while `device_count()` cheerfully returns 2. Confirmed by direct A/B.
+2. **memlock must be raised** (done on Dankserver via
+   `/etc/security/limits.d/99-memlock.conf`). Host default 8 MiB makes any
+   large pinned allocation fail as a misleading
+   `torch.AcceleratorError: CUDA error: out of memory` from `pin_memory()`;
+   it cannot be raised from inside a container or root's own shell — only
+   PAM applies it at login. Matters because `--expert-host-mlock` defaults
+   True.
+3. **`--gpu-memory-utilization` does NOT buy KV under expert offload** —
+   the planner sizes the arena against the budget and leaves KV the
+   remainder; raising util 0.95→0.98 grew the arena and the KV pool stayed
+   at 0.64 GiB.
+4. **Lowering `--max-model-len` is the WRONG lever and backfires** — the
+   budget subtracts the KV reserve first, so a smaller context shrinks the
+   reserve and hands the space to the arena (measured: 65536 → 0.64 GiB KV;
+   32768 → 0.34 GiB).
+5. **The KV reserve undercounts hybrid-attention models ~2×** (9190 B/token
+   planned vs ~18,000 real: the Mamba state is a fixed per-sequence cost).
+   Fix by pinning the pool: `--kv-cache-memory-bytes`.
+6. **`--expert-offload-mem` caps the HOST side**, not the VRAM arena —
+   structurally orthogonal to `--expert-precise-memory` (verified
+   in-container 2026-10-05: precise-memory only swaps the GPU overhead term,
+   expert_offload_mem only sets the host bound; they rejoin only inside
+   `place_experts`, and the plan rewrites `expert_offload_mem` back to the
+   cap the plan met, so plan and allocation cannot drift).
+7. `--ipc=host` and `--shm-size` are mutually exclusive in podman; pick one.
+8. `podman run -d` without `-a` sends no stdout to a redirect — the log file
+   only receives the container ID; use `podman logs`.
+
+Status reads: use `scripts/dankstatus.sh`, never hand-rolled
+`/proc/meminfo` parsing — `/proc/meminfo` separates key from value with a
+TAB, so `grep MemAvailable | tr -s " " | cut -d" " -f2` returns empty and
+awk prints `0.0` — a false "host out of memory" that invites an unnecessary
+abort. Gated checkpoints: token via a mode-600 `curl -K` config, never argv
+(`pgrep -a curl` prints it); always `--fail`; compare each file's byte count
+to the Hub's declared size; `-C -` resumes truncated shards.
+
+**Check whether the stack supports the model natively before porting an
+overlay.** The image already registered `Glm5NextForConditionalGeneration`,
+`Qwen4ExpForCausalLM` and both MTP variants — chasing four missing symbols
+through two image tags to bring an overlay up was entirely wasted work.
+
+## Compile-cache persistence (survives `podman rm -f`)
+
+Serve scripts that recreate the container discard the writable layer, so
+every boot re-pays compile/JIT (frank-primary: `torch.compile took 181.20 s`
+cold, of which 173 s was one graph).
+
+1. **Prove it and locate it:** fresh container vs live
+   (`podman run --rm --entrypoint bash IMG -c "ls -la /root/.cache"` vs
+   `podman exec LIVE bash -c "du -sb /root/.cache/*; find /root/.cache -type
+   f | wc -l"`). vLLM cache root: `VLLM_CACHE_ROOT` (default
+   `/root/.cache/vllm`, `torch_compile_cache/`); a HIP runtime adds
+   `/root/.cache/comgr`.
+2. **Mount the PARENT, not the leaf** — one bind on `/root/.cache` persists
+   both trees, safe whenever the image ships no such directory (step 1
+   proves it; otherwise you would be hiding baked state). Wire as an opt-in
+   tunable: `mkdir -p "$CACHE_DIR"` BEFORE the mount, then
+   `${CACHE_DIR:+-v "$CACHE_DIR":/root/.cache}`. Host path outside the
+   checkout.
+3. **Verify with two consecutive boots** — cold `Compiling a graph … takes
+   N s`, warm `Directly load the compiled graph(s) for compile range (…)
+   from the cache` with `torch.compile took` in single-digit seconds.
+4. Traps: the cache key includes the compile RANGE — an MNBT change costs
+   ONE recompile, not a permanent tax; do not claim a proportional boot cut
+   (time to `/health` is dominated by weight loading: 490 s warm vs 188 s
+   cold compile); stale entries are misses, not wrong answers (version +
+   config hashed); if the serve script is seam-gated, assert the mkdir and
+   the exact mount target.
+
+## Plan-line verification checklist (after any fork deploy)
+
+```
+ssh dankserver 'sudo journalctl CONTAINER_NAME=frank-primary-qwen -b -o cat --no-pager \
+  | grep -aoE "(budget \(total x util\) +[0-9.]+|expert cache space +[0-9.]+ GiB +[0-9,]+ slots|Estimated CUDA graph memory.{0,80}|Graph capturing finished.{0,40}|GPU KV cache size.{0,90})"'
+```
+
+`grep -oE` with a digit-anchored tail — plain `grep`/journalctl truncate
+these columns to `17.`. Verify: new budget/graphs/arena in the plan;
+`GPU KV cache size` matches the requested token count; `Estimated CUDA graph
+memory` vs the capture line (raise `_PER_GRAPH_MB` toward 52 if capture
+exceeds the charge); `rocm-smi --showmeminfo vram` high-water per card = the
+real margin; `systemctl show frank-primary -p NRestarts --value` = 0;
+`VLLM_CUDAGRAPH_*` present in `podman exec … env`; then a real chat
+completion from a PEER — `curl localhost:7770/health` inside an ssh session
+on the host can return `000` and looks like an outage that isn't.
+
+## Related
+
+- [[RDNA4 MoE Expert Offload VRAM Budget]] — the plan law this harness tunes
+- [[vLLM Boot Forensics and Config Interrogation]] — journal forensics for these boots
+- [[LLM Bench Windows and Measurement]] — window discipline and bash laws
+- [[Model Swap and Rollback]] — where the fork sits in the swap history
+- [[Remote Job Runner]] — the job.sh harness referenced above

serverAppConfigs-docs/LLM/VLLM Overview.md

@@ -237,6 +237,46 @@ vLLM supports multiple parallelism strategies:
 - Expert Parallelism (EP) distributes expert networks in MoE models
 - Context Parallelism (CP) splits context across GPUs
 
+
+## Chunked Prefill — the flag surface, and what it actually does
+
+**YES, the runtime exposes chunked-prefill options, and the running shape
+already uses them.** The tcclaviger fork is a vLLM fork, so the standard
+scheduler flags are present in the shipped image (verified by reading the
+`vllm.engine.arg_utils` and `vllm.config.scheduler` source in the live
+container `frank-primary-qwen`, not from a stub):
+
+| Flag | Default | What it controls |
+|---|---|---|
+| `--enable-chunked-prefill` | `True` in `SchedulerConfig`; the CLI arg is an auto-flag (`"default": None`) | whether prefill may be split into chunks |
+| `--max-num-batched-tokens` (MNBT) | 2048 (`DEFAULT_MAX_NUM_BATCHED_TOKENS`) | tokens processed per prefill iteration |
+| `--prefill-schedule-interval` | 0 | gap between prefill iterations |
+
+The running shape's argv carries `--max-num-batched-tokens 2048` alongside
+`--max-model-len 200704`, so chunked prefill is **ON and already active**: a
+200704-token prompt is prefilled in ~98 iterations of 2048 tokens, not in one
+shot.
+
+**It is a TTFT lever, not a throughput lever.** MEASURED 2026-09-29 on the
+ad hoc qwen-fp8 container (the same fork, same payload, same TP=2):
+
+- MNBT 8192 vs the default 2048: **neutral** on decode at every depth.
+- Small values re-chunk prefill only. They hurt TTFT (each chunk is its own
+  iteration, so the first token waits on more iterations) and they never touch
+  decode: a decode step is `max_num_seqs x (spec+1)` tokens, independent of
+  MNBT.
+
+**There is no `--ubatch-size` equivalent in vLLM.** `--ubatch-size` is a
+llama.cpp compute-buffer concept (`--batch-size`/`--ubatch-size`); the tcclaviger
+fork has no ubatch knob at all. The llama.cpp compute-buffer pair that frank-small
+was doubled on (2026-10-05, +4.6 % prefill at 8,928 tokens for 616 MiB) has no
+counterpart here, and the two backends should not be reasoned about by analogy.
+
+**MNBT is not a lever on this shape either.** The deployed 2048 is the fork's
+default, so a change is a deliberate departure from the shipped value, not a
+tune. The one thing MNBT does cost: every change re-enters the compile RANGE
+path, so `FRANK_PRIMARY_CACHE_DIR` pays one recompilation.
+
 ## See Also
 
 - [[VLLM Index]]

serverAppConfigs-docs/LLM/dankserver-frontier-moe-feasibility.md

@@ -168,11 +168,46 @@ Not advisable.
 | **Affinity** | yes, built here | **V4-Flash only** — "the geometry is compiled in; there is no architecture abstraction" | 110 tok/s for 284B. Best engine for this box by an order of magnitude, wrong model |
 | **tcclaviger vLLM fork** | **yes — proven 2026-09-29** | qwen4exp measured 37.4-40.6 tok/s decode (see [[dankserver-qwen38-fp8-expert-offload-tuning]]); **`glm5next` registered** (`Glm5NextForCausalLM`/`MTP`, backend-split `amd/`/`common/`/`nvidia/`, `sparse_indexer.py` for DSA); r4d.so has **native mxfp4 + dsfp4 MoE entries** (`has_dsfp4() == True`); W4A16 MoE class (`R4dW4a16MoEExperts`) wired into the same tiered offload; TunableOp silently skipped under offload (needs `CLAV_TUNABLEOP_WITH_OFFLOAD=1`, and then corrupts numerics — do not use) | the ad hoc qwen-fp8 container's runtime; **the old "MXFP4 = emulation/missing kernels" verdict predates this fork and is dead** — the GLM-5.3-Flash gate is now the payload format + a boot probe, not kernels |
 | **Lucebox** | yes, image on host | Qwen 3.8, DS4; **V4.1 on the unmerged `feat/ds41-engine` branch** | The only V4.1 path. **Refuses layer split and layer-major prefill for V4.1** |
-| **radiance-vLLM** (drwolfen) | yes — the active frank-primary | welded to `qwen4_exp`; PLE / QSA / packed-HC live inside that model class | 12–20 person-days per architecture |
+| **radiance-vLLM** (drwolfen) | yes — served frank-primary 2026-09-27 → 09-30 | welded to `qwen4_exp`; PLE / QSA / packed-HC live inside that model class | 12–20 person-days per architecture |
 | **FreeToken-ROCm** | partial | block-FP8 safetensors only (modelopt); known `CpuMoeExecutor` issue under ROCm | cannot load a GGUF |
 | **llama.cpp** | yes | GLM-5.3 confirmed by a community R9700 guide; DS-V4.1 CED unverified | no expert LRU; **8 CPU cores** make CPU expert placement the wrong trade |
 | **TokenSpeed** | claimed | GLM-5.x incl. a 5.3-Flash recipe | NVIDIA-first, unverified on gfx1201 |
 | **KTransformers** | no | GLM-5.3 tutorial exists | built for Intel AMX + NVIDIA |
+| **radiance** (StillDeadcode — the standalone C++/HIP engine, NOT the vLLM fork of the same name) | image `stilldeadcode/radiance` prebuilt for gfx1201 | qwen35 dense/MoE at fp8, qwen4exp quantised-at-convert, llama-shaped | see the probe below: our FP8 payload is REFUSED; tiers, MTP and vision native |
+
+### radiance (standalone) — 2026-10-05 probe
+
+`codeberg.org/StillDeadcode/radiance` — Deadcode's from-scratch engine
+(v1.0.13): same author as the `vllm-radiance` fork we ran in September
+([[frank-medium-radiance-mxfp4]]), an unrelated codebase — C++/HIP, no
+Python, kernels/architectures/quantisers as `.so` plugins, flags written
+for two 32 GB R9700s, native Qwen3.8-Flash-Next with `--placement
+expert_tiered`, MTP and vision.
+
+Probe with the DEPLOYED payload (`rad-convert --plan-only`, which stops
+at declare — no GPU touched, no bytes written):
+
+```
+E no architecture plugin claims 'qwen4exp' at quantisation 'fp8'.
+  'qwen4exp' is served at: (unquantised).
+```
+
+So the 173 G orcarouter FP8 file cannot enter this engine at all: the
+qwen4exp plugin takes bf16/f16/f32/GGUF in and every quantisation happens
+at `rad-convert` time by recipe. Not a lock-in to their published quant —
+recipes are per-weight-class rules (`rtn`/`gptq`/`awq`/`autoround`/
+`paroquant`/`ggml`, grids from fp8 128×128 to i4/w4nl/w5nl codebooks,
+custom `--quant` rules go ahead of the file's), and libr4d lists routed
+experts-as-fp8 among the encodings it reads — so today's quantization is
+reproducible **from a bf16 source**, not from our file. Uncensored-family
+bf16 source candidates: `windowsxp811203/Qwen3.8-Flash-Next-Abliterated-GGUF`
+(ships the bf16 GGUF) and `mradermacher/Qwen3.8-Flash-Next-Uncensored-GGUF`
+(i1 quants only). 622 G free fits source + container. A trial is:
+download → `rad-convert` (CPU-side) → stop+mask frank-primary window to
+serve. Caveats on record: `--tp-wire wht6` is a lossy int8 all-reduce
+wire; the recurrent GDN state is f32 by default (144 MiB/sequence) with
+an opt-in f16 that changes generated text; every pool is budgeted
+explicitly — no profiling pass, no utilisation fraction.
 
 The two-R9700 three-owner split (primary + secondary + stream) is **not
 available on this host**: the retired quadlet recorded that

serverAppConfigs-docs/LLM/dankserver-qwen38-fp8-expert-offload-tuning.md

@@ -275,35 +275,357 @@ Depth caveat: every number above is at a near-empty KV. Deep-context
 behavior (the gather win growing with depth, bf16's decode cost at depth)
 is only sampled at 3 depths; a 64k-192k-depth probe is the flagged gap.
 
-### The deployed shape (2026-10-02: 2 x 200704, fp8, pool cut for the arena)
+### The deployed shape (2026-10-03 shape G: 3 slots x 200704, fp8, precise memory, MNBT 2048, arena at the flag ceiling)
 
 ```
-FRANK_PRIMARY_MAXLEN=200704    x SEQS 2 (401,408 tokens of contract)
-FRANK_PRIMARY_KV_BYTES=3150000000   PER RANK (fp8, 7,701 B/token)
-FRANK_PRIMARY_KV_DTYPE=fp8   --kv-cache-dtype fp8 (quantized)
-FRANK_PRIMARY_MNBT=1024       --max-num-batched-tokens 1024
+FRANK_PRIMARY_MAXLEN=200704   200704-token window
+FRANK_PRIMARY_SEQS=3          --max-num-seqs 3 (SEQS=2 CANNOT boot - below)
+FRANK_PRIMARY_KV_BYTES=3166500000   PER RANK (fp8, 7,892 B/token -> 400,700 tokens)
+FRANK_PRIMARY_KV_DTYPE=fp8    --kv-cache-dtype fp8 (quantized)
+FRANK_PRIMARY_MNBT=2048       --max-num-batched-tokens 2048 (arena-free, see shape F)
+FRANK_PRIMARY_UTIL=0.982      budget = total x util, the law (1.0 unbootable, see below)
+FRANK_PRIMARY_CUDAGRAPH_BASE_MB=0 / _PER_GRAPH_MB=12  -e VLLM_CUDAGRAPH_MEM_*
+FRANK_PRIMARY_PRECISE_MEMORY=1      --expert-precise-memory (overhead 7.60 -> 5.25 GiB)
+FRANK_PRIMARY_EXPERT_OFFLOAD_MEM=120  --expert-offload-mem (host-region cap, explicit)
+FRANK_PRIMARY_GPU_MAX_HW_QUEUES=2   -e GPU_MAX_HW_QUEUES=2 (HSA queues/GPU)
+FRANK_PRIMARY_CACHE_DIR=/home/dank/.cache/frank-primary   mounted at /root/.cache
 R4D_LRU_GATHER_CHUNKS=32 / R4D_LRU_GATHER_LANES=64  (container env)
 NO --speculative-config, NO --expert-prefetch
+FRANK_PRIMARY_NO_PREFIX_CACHING=1  --no-enable-prefix-caching (2026-10-05, the QFN align-mode GDN state fault — below)
 ```
 
-This replaced the 2026-10-01 shape — 2 x 262144 at bf16 KV with
-`--expert-precise-memory` — which crash-looped at graph capture: the arena is
-DEVICE-resident, the precise-memory flag drops the plan's overhead 7.60 ->
-5.25 GiB, and that shape had 0.13 GiB to spare (weights 6.28 + pool 6.80 +
-graphs 0.55 + arena 16.65 + non-torch ~1.45 = 31.73 of 31.86 GiB,
-NRestarts=3 at ~6.7 min each). The plain bf16 2 x 262144 boot also did not
-stay up: it left 0 bytes free at runtime and a real client request (995-token
-prompt, max_tokens=261149) killed the worker with `Tried to allocate 50.00 MiB
+### 2026-10-05 — prefix caching OFF: the fork's QFN align-mode GDN state fault
+
+**Symptom (Frank, 2026-10-04):** output turned repetitive/corrupted at
+~40k tokens into a long conversation; short sessions stayed clean.
+
+**Evidence (live `podman logs frank-primary-qwen`, the 2026-10-04 18:19
+boot):** one 0.6 s burst at 18:46:07 PDT — 5.6 h into service, one
+running request, and the engine's own 18:46:08 log line reading
+"Prefix cache hit rate: 48.1%". 48 `gdn_hip` faults, then the
+process-lifetime silence cap:
+
+```
+[gdn_hip] ERROR gdn_decode_r: state out of ssm-cache range: max|S|=0
+  with non-finite values (slot=42 head=3 limit=3.40282e+38) fault #1-23
+[gdn_hip] ERROR gdn_decode_r: ... (slot=42 head=8 ...) fault #24-46
+[gdn_hip] ERROR: state fault report limit (32) reached; further faults
+  are counted silently
+```
+
+One physical SSM slot (42), two heads (3 and 8 — one per TP rank), 23
+GDN layers each, the state non-finite (NaN). The guard's own text in
+this build: *"known Qwen3.8-Flash-Next (QFN) flaw under prefix caching
+and is being actively worked; output quality for this request may
+degrade. Mitigation: --no-enable-prefix-caching."*
+
+**Mechanism.** With prefix caching on, the fork defaults QFN to
+`mamba_cache_mode=align` (boot log: "Mamba cache mode is set to 'align'
+for Qwen4ExpForConditionalGeneration … when prefix caching is enabled";
+`config/cache.py`: "none" is the no-prefix-cache mode, "all" is
+deprecated — align is the only sound mode WITH the flag). Align mode
+checkpoints each sequence's GDN (Gated DeltaNet) recurrent state into a
+mamba page per block and RE-SEEDS the state from that page when a
+request resumes from a cached prefix. The block size is raised to
+**3136 tokens** for the page-size equality (boot log: "Setting
+attention block size to 3136 tokens to ensure that attention page size
+is >= mamba page size" + "Padding mamba page size by 0.13% … exactly
+equal"). The corrupt state landed at the **13th boundary: 13 × 3136 =
+40,768 tokens** — the reported "~40k" mark. The NaN persisted in the
+mamba page, so every later turn of that conversation re-seeded the same
+non-finite state for those two heads (silently — the fault-report cap
+was already spent) while fresh short sessions looked fine.
+
+**RAM ruled out the same day** (the 2026-10-04 17:17 host reboot is when
+the current memory speed took effect): zero MCE/EDAC kernel records
+since boot; an 8 GiB in-band pattern check stable over 60 s under live
+load; 4800 MT/s is the JEDEC DDR5 floor on 5600 MT/s-rated DIMMs (not an
+overclock); and a two-head, single-slot NaN state is a state-path fault,
+not scattered finite bit corruption.
+
+**Fix.** `FRANK_PRIMARY_NO_PREFIX_CACHING=1` → `--no-enable-prefix-caching`
+(the fork-named mitigation; the flag renders in both directions from the
+knob so a fork default change cannot silently flip the contract). Cost:
+no prefix reuse — every turn re-prefills the whole history instead of at
+most 3135 tail tokens. A silently corrupted conversation is worse than a
+slower first token.
+
+**Verified 2026-10-05 (deploy b820d34):**
+
+- Engine argv carries `--no-enable-prefix-caching`; health 200; warm boot
+  `init engine … took 76.57 s` (the mounted compile cache is intact).
+- Boot plan unchanged, as predicted: expert cache **17.77 GiB / 7,762
+  slots** (the flag touches no plan term), CUDA-graph estimate 0.07 GiB
+  vs 0.08 GiB captured (inside the 12 MiB/graph charge).
+- `GPU KV cache size: 424,678 tokens` — **up** from 401,408 at the same
+  3,166,500,000 B ask: the pool now prices ~7,456 B/token (was 7,891.8),
+  the mamba/side per-token term dropping once the per-block checkpoint
+  pages are gone; the 3136 attention block-size raise nonetheless stays
+  (the page-size invariant is enforced in mode "none" too). Frank's
+  400,700-token floor is cleared with ~24k of headroom.
+- **Zero `gdn_hip` lines** in the boot-and-traffic window (only the two
+  INFO module banners mention the name).
+- Boundary probe (single request, no prefix reuse): prompt 39,280 +
+  completion 1,491 = **40,771 total tokens — across the 40,768 line**;
+  the task (write 1–400, one per line) completed coherently: 399 integers
+  parsed, 0.997 monotone, tail `… 398 399 400`. Pre-fix, this boundary is
+  exactly where the conversation degraded (the 13th 3136-page re-seed).
+- e2e: 32/33 pass; the one failure is the admin spec's
+  192.168.0.112:8989 leg with FrankBook-Pro asleep (environmental,
+  unrelated to this change; re-run `tests/e2e/run.sh` once the mini is up).
+
+**RESULT — REVERTED THE SAME DAY.** With every mechanical check green
+(zero faults, coherent 40,771-token probe), live long conversations
+STILL degraded at ~40k on the running endpoint (Frank, direct
+observation). The container serving that window (`--no-enable-prefix-
+caching`, booted 2026-10-05 12:47 PDT) logged ZERO `gdn_hip` faults
+while the corruption reproduced — so this recurrence is **fault-free**:
+the NaN-state guard did not fire, which the re-seed theory requires.
+Open hypotheses, in order: (1) the poisoned transcript is being re-fed —
+corrupted assistant turns already in the history keep producing
+corrupted continuations even with perfect state, so the same
+conversation cannot distinguish fixed from unfixed; (2) a second
+long-context mechanism with no guard coverage — fp8 KV degradation at
+depth is the next suspect (a bf16-KV boot costs ~2× the pool per token
+and would settle it). Prefix caching is **back ON**
+(`FRANK_PRIMARY_NO_PREFIX_CACHING` commented out — the cost bought no
+fix). The knob and the serve branch stay for a future fork build with a
+fixed GDN path. Next diagnostic: reproduce on a FRESH conversation; if
+it still breaks at ~40k with zero faults, A/B the KV dtype.
+
+**RESOLVED (2026-10-05 evening): it was the RAM.** Frank dropped the
+memory from 4800 to **4600 MT/s** (in effect since the 2026-10-05 12:21
+PDT host boot) and the corruption stopped: zero `gdn_hip` ERROR lines in
+the container since, zero MCE records, and the live agent session on
+this endpoint past 140k context, "comfy cozy" (Frank). Marginal DDR5 on
+four DIMMs is a common cause for BOTH eras: the 2026-10-04 NaN burst
+(corrupted weight/activation traffic poisoning a GDN state write — the
+guard-visible path) and the fault-free ~40k degradation (silent flips in
+KV/expert data, which no guard covers on non-ECC memory). The re-seed
+theory is therefore UNPROVEN, not disproven — the entire caching
+experiment ran on marginal RAM and was confounded on both arms.
+Post-mortem on the rule-out: consumer non-ECC RAM raises no MCE for a
+silent flip; a 60 s / 8 GiB pattern sample covers ~0.5 % of 246 GiB;
+"4800 is the JEDEC floor on 5600-rated DIMMs" is a rating, not a
+stability proof at four-DIMM loading. The revert test was the decisive
+instrument and it pointed at the RAM all along. Prefix caching stays ON
+(zero faults under heavy re-send traffic since the speed drop); the knob
+stays for a future fork fix. The fp8-KV A/B is NOT needed.
+
+### Shape F/G (2026-10-03): where the free VRAM actually is, and what forcing it cost
+
+`rocm-smi` showed **3.57 GiB free per R9700** at the deployed shape E (28.28 GiB
+used of 31.86 GiB, after hours of real traffic — so it is the high-water mark,
+not an idle artifact). Growing the expert arena into it is NOT a matter of
+raising `--gpu-memory-utilization`, because the arena is
+
+```
+arena = budget - non-expert - KV reserve - CUDA graphs - runtime overhead
+budget = device_total x gpu_memory_utilization          # expert_plan.py:567
+```
+
+and every term on the right is either fixed or a lie:
+
+| term | plan charges | measured reality | reachable from our side? |
+|---|---|---|---|
+| `budget` | 31.22 GiB (0.98) | field is `le=1` so 31.86 GiB is the field max, but 1.0 is UNBOOTABLE (below) | **yes** — shipped 0.982 |
+| `non-expert weights` | 6.28 GiB | checkpoint census | no |
+| `KV reserve` | 1.91 GiB (registry 9,190 B/token x 200704 +8%) | real pool bills 2.93 GiB at 7,892 B/token | no (registry constant) |
+| `CUDA graphs` | 0.55 GiB = 256 MiB + 52 MiB x 6 | `Graph capturing finished in 3 secs, took 0.31 GiB` | **yes** — `VLLM_CUDAGRAPH_MEM_BASE_MB` / `_PER_GRAPH_MB`; `VLLM_CUDAGRAPH_MEM_ESTIMATE_MB` pins the whole term outright (checked first, pair ignored when set) |
+| `runtime overhead` | 5.25 GiB (`--expert-precise-memory`) | far lower at idle | no (registry constant) |
+
+Shape F took the graph term (0.55 -> 0.32 GiB) and nudged util to 0.982:
+**arena 17.52 GiB / 7,653 slots measured**, +131 slots. Shape G takes the
+graph term further (0.32 -> 0.07 GiB) at the same util.
+
+**Shape G takes the graph term to the measured floor, util to the law.** The
+graph base pool is the actual over-charge: shape F's own boot reported
+`Graph capturing finished in 4 secs, took -0.06 GiB` — a **negative** capture
+delta, so the 256 MiB base was fiction. Charging `0 base + 12 MiB/graph`
+(0.07 GiB for the shipped 6-graph plan, and still per-graph so an 18-graph
+spec-decode shape pays 0.21 GiB) lands the plan at budget 31.29 / graphs 0.07 /
+**arena 17.77 GiB / 7,762 slots measured**. The KV ask went to
+`3,166,500,000` B/rank, which the engine's own 7,891.8 B/token price lands at
+**401,408 tokens measured / 2.00x concurrency** — the first shape holding two
+full-length conversations with real spare. (The ask/landed delta is block
+alignment: 196 x 2048. The price is shape-dependent, 7,701 B/token at 262144.)
+
+**UTIL 1.00 is not a knife-edge on this fork, it is UNBOOTABLE — measured
+2026-10-03.** One deploy took the field maximum to buy the last 0.57 GiB of
+arena; both workers refused init BEFORE loading a single weight, twice:
+`ValueError: Free memory on device cuda:N (31.29/31.86 GiB ...)` — the worker
+demands free > budget and budget == total, so `le=1` on the CLI field is a
+trap: the engine dies in a 2-restart crash loop with zero payload progress.
+So `0.982` is a LAW, not a preference, and the "why 0.982" answer now has two
+proofs: this ValueError, and the historical ceiling deaths (the 2x262144 bf16
+boot at 0.13 GiB free; the 5x262144 boot at "0 bytes free" on a real
+995-token request). Everything beyond ~7,912 slots needs the registry edited
+inside the image (the fork's log line says exactly that: *"Update the
+registry entry if this drifts"*), not a deploy.
+
+**The host region was the last arena-plan term still autofitting per boot —
+now it is a literal (2026-10-04).** With no `--expert-offload-mem` the planner
+binds the host region to `MemAvailable` AT BOOT minus `PLE 47.68 - 2 x 7.0
+process - 5.0 api/engine - 5% margin`: the 2026-10-05 boot read 237.97 GiB and
+offered 159.39 GiB usable, while the same host reads **75.4 GiB available while
+running** — the bound tracks whatever the co-tenants hold at that moment. The
+demand is 105.48 GiB (52.74 GiB/rank x 2, 23,040 swap-target experts), and a
+boot where the bound drops near it silently converts next-hottest experts to
+VRAM-only (the `pinned_extra` warning is the only trace).
+`FRANK_PRIMARY_EXPERT_OFFLOAD_MEM=120` pins it: measured demand + 14% headroom,
+on a host with 246.1 GiB total. Semantics verified in the shipped image
+(`offloader/expert_plan.py`): the cap binds as the bound, an exceedance only
+warns, and the plan rewrites the field to the cap it met (+1 MiB/rank).
+
+**MNBT 1024 -> 2048 is free on every axis the plan can see.** The registry
+overhead model is `base + 0.22 GiB x max(0, (nbt - 4096)/1024)` anchored at
+nbt 4096, so any rung up to 4096 charges the identical 5.25 GiB; the fitted 5.25
+was itself fitted at nbt 4096, which means 2048 is covered by an activation
+figure measured at a **larger** batch. SEQS 3 boots at both 1024 and 2048
+(measured 2026-10-02 — the `query_start_loc` `ConstraintViolationError` is a
+SEQS property). The costs are a one-time recompile (the compile **range** is
+part of the cache key, so `FRANK_PRIMARY_CACHE_DIR` pays once) and a wider
+prefill chunk, which the 09-29 sweep found decode-interchangeable across
+1024-4096 while 512 doubled TTFT at 4064 ctx (2.46 s vs 1.36 s).
+
+**`--expert-precise-memory` is the arena lever that does not touch the window.**
+It attacks the planner's overhead term (shape F above attacks the graph term —
+the third and last term this repo can reach), and the flag swaps the default
+registry figure for the fitted one. Measured
+2026-10-02 by a boot probe on the shape-D base (production stopped, candidate
+booted AS the service; numbers are the engine's own plan lines):
+
+| plan line | shape D | **shape E** |
+|---|---|---|
+| `runtime overhead (non-torch + buffers + activation)` | 7.60 GiB | **5.25 GiB** (`precise (gfx1201 TP2 nbt4096/seqs16/len262144)`) |
+| `expert cache space` | 14.87 GiB | **17.22 GiB** |
+| slots | 6,496 | **7,522** (+1,026, +16%) |
+| `swap region` | - | 13.7 GiB / 5,986 slots, 133-134 per layer |
+| boot | - | healthy in 420 s, no OOM |
+
+`'expert_precise_memory': True` also appears in `non-default args`, so a silent
+argparse drop is visible in the boot log. The flag lands **in the arena, not the
+pool** - `--kv-cache-memory-bytes` is untouched at 3.15e9/rank (option A), because
+the pool is not an arena lever and cutting it further would only cost
+concurrency: a 2.2e9 pool was probed in the same window and reports 277,686
+tokens / **1.38x**, so it is the conservative fallback if free VRAM ever binds,
+not the shipped value.
+
+**The compile cache is a boot-cost fix, and it was worth 173 s per boot.**
+Measured 2026-10-02: the image ships **no `/root/.cache` at all** (a fresh
+container: `No such file or directory`), and after a boot that directory holds
+`vllm` at **1,154,714,333 B in 4,665 files** plus `comgr` at 229 MB, all inside
+the container's ephemeral layer. Every
+`podman rm -f` - i.e. every deploy, restart and reboot - discards it, and the
+engine re-pays:
+
+```
+Using cache directory: /root/.cache/vllm/torch_compile_cache/b265bd396a/rank_0_0/backbone
+Dynamo bytecode transform time: 4.99 s
+Compiling a graph for compile range (1, 1024) takes 173.19 s
+saved AOT compiled function to /root/.cache/vllm/torch_compile_cache/torch_aot_compile/…
+torch.compile took 181.20 s in total
+init engine (profile, create kv cache, warmup model) took 217.26 s (compilation: 181.20 s)
+```
+
+Mounting a host directory at `/root/.cache` makes every later boot load the AOT
+artifact instead. The mount is deliberately the **parent**, not
+`/root/.cache/vllm`: `VLLM_CACHE_ROOT` defaults to the `vllm` child, and
+`comgr` is the HIP-kernel JIT cache, so one mount persists both — and it hides
+nothing, because the image ships no `/root/.cache`. The cache key
+includes the compile range, so the MNBT choice costs exactly one compilation.
+Entries are version- and config-hashed: a stale entry is a miss, never a wrong
+answer.
+
+**Proven live on the deployed endpoint, 2026-10-02, consecutive boots of this
+exact shape:**
+
+| boot | engine log |
+|---|---|
+| cold (deploy) | `Compiling a graph for compile range (1, 1024) takes 179.24 s` → `torch.compile took 188.26 s in total` |
+| warm (restart) | `Directly load the compiled graph(s) for compile range (1, 1024) from the cache, took 2.143 s` → `torch.compile took 3.97 s in total` → `init engine (profile, create kv cache, warmup model) took 41.13 s (compilation: 3.98 s)` |
+
+The host directory held **1.4 GB in 4,707 files** after the cold boot. Time to
+`/health` is dominated by the 172.8 GiB payload load (the warm boot reached
+health in 490 s), so the win is the ~184 s of inductor/AOT build rather than a
+proportional cut in boot time.
+
+`GPU_MAX_HW_QUEUES=2` caps the HSA hardware queues per GPU. Verified a real knob
+rather than a no-op - the literal is read out of
+`/opt/rocm/core-10.0/lib/libhsa-runtime64.so` - and shipped deliberately
+alongside the 32x64 gather geometry. It is **not** A/B measured.
+
+**The pool, not the slot count, sets how many conversations live at once.**
+Measured live at this shape: `GPU KV cache size: 398,658 tokens, Maximum
+concurrency for 200,704 tokens per request: 1.99x` (3,146,129,408 B reserved).
+The per-token price is **shape-dependent**: the 262144 shape priced at 7,701
+B/token (6,100,000,000 B -> 792,069 tokens), this pool at 7,892 - ~2.5% dearer,
+so a concurrency figure never extrapolates across geometries. The endpoint
+advertises 3 slots: a third LONG request queues rather than being refused, and
+shorter requests use the spare blocks. Verified: three concurrent 67-token
+requests answered in parallel, ~4.1 s each. Two contexts at 200704 are a
+capacity contract, not a throughput one (2-way concurrency summed BELOW parity
+at 4064 ctx).
+
+#### `--max-num-seqs 2` is impossible on this model (measured 2026-10-02)
+
+Four single-variable boot probes against the then-deployed shape, production
+stopped for each:
+
+| MAXLEN | SEQS | MNBT | result |
+|---|---|---|---|
+| 262144 | 2 | 1024 | **Engine core initialization failed** |
+| 262144 | 2 | 2048 | **Engine core initialization failed** |
+| 262144 | 3 | 1024 | healthy - 792,069 tokens, 3.02x, arena 6,248 |
+| 262144 | 3 | 2048 | healthy - the then-deployed incumbent |
+
+The failure is `torch.fx.experimental.symbolic_shapes.ConstraintViolationError:
+Constraints violated (L['query_start_loc'].size()[0]) ... specialized it to be
+a constant (3)`, raised from AOT `build_guards(strict_error=True)` during
+`determine_available_memory -> profile_run -> _dummy_run`. `model_runner.py`
+sizes that input buffer `max_num_reqs + 1` and `_dummy_run` sets
+`num_reqs = min(max_num_batched_tokens, max_num_reqs)`, so the width is
+`SEQS + 1` for ANY `MNBT >= SEQS`: 3 at SEQS 2, 4 at SEQS 3. **MNBT is not a
+lever here** - it only moves the compile RANGE ((1,1024) vs (1,2048)), which
+is why the identical error appears at both rungs. `--enforce-eager` bypasses
+the path at the cost of CUDA graphs. The same failure class killed the
+2026-10-01 bf16 2 x 262144 shape.
+
+#### The arena lever is `max_model_len`, not `--kv-cache-memory-bytes`
+
+The fork's planner sizes expert cache as `budget - non-expert - KV reserve -
+graphs - overhead`, and the KV reserve is registry-priced at **9190 B/token x
+max_model_len** - not from the `--kv-cache-memory-bytes` value:
+
+| MAXLEN | KV reserve | arena |
+|---|---|---|
+| 262144 | 2.48 GiB | 14.30 GiB / **6,248 slots** |
+| 200704 | 1.91 GiB | 14.87 GiB / **6,496 slots** |
+
+D(reserve) 0.57 GiB = D(arena) 0.57 GiB = +248 slots. The 6.1e9 -> 3.15e9
+pool cut (2.75 GiB) bought **zero** arena slots. It buys runtime HEADROOM,
+which is still worth having - the shape it replaced died with `0 bytes free`
+on a real 995-token request. Grow the arena by lowering `max_model_len`, or by
+taking the overhead term with `--expert-precise-memory` (below); the pool is the
+headroom lever only.
+
+#### The other arena lever: `--expert-precise-memory` (overhead 7.60 -> 5.25 GiB)
+
+The flag is now ON. It was refused on 2026-10-01 for a reason that does not
+generalize: the 2 x 262144 bf16 shape was already at the ceiling (weights 6.28 +
+pool 6.80 + graphs 0.55 + arena 16.65 + non-torch ~1.45 = 31.73 of 31.86 GiB,
+NRestarts=3 at ~6.7 min each), so the 2.35 GiB the flag moves into the arena was
+the last stack over the line, not the cause. The plain bf16 2 x 262144 boot also
+did not stay up: 0 bytes free at runtime, and a real client request (995-token
+prompt, `max_tokens=261149`) killed the worker with `Tried to allocate 50.00 MiB
 ... 0 bytes free`.
 
-The correction is a pool cut, not a downgrade. fp8 costs 7,701 B/token
-(measured 6,138 attention + 1,563 mamba/side, dtype-independent), so 2 x
-200704 = 401,408 tokens needs 3.091e9 B; pinned at 3.15e9, ~44 MB of margin
-above the floor — the same convention the 6.1e9 shape used. The arena is what
-remains: the planner sizes it AFTER the KV override, so the 6,248 slots of the
-6.1e9 shape become roughly 7,500 here. Two contexts at 200704 are a capacity
-contract, not a throughput one (2-way concurrency summed BELOW parity at 4064
-ctx), so the slot buys the VRAM headroom long generations need.
+At 200704 x 3 the flag fits with room and is worth **+1,026 slots (+16%)**. What
+made it affordable is the window trim, not a pool cut: the arena is
+DEVICE-resident, the pool is not part of the budget's arithmetic, and the 200704
+window's KV reserve (1.91 GiB) plus a smaller attention footprint is what the
+plan needed. The pool stays at 3.15e9/rank deliberately (option A) - cutting it
+further would cost concurrency (2.2e9 -> 1.38x) and buy no slots.
 
 Boot-probed 2026-10-02 (transient systemd units at the production uid +
 memlock, full-log capture; the rootless bench probe had masked the production
@@ -314,13 +636,17 @@ memory environment). The night searched two concurrency contracts:
 | 3 x 262144, bf16, KV 11,047,010,304 | **OOM at graph capture** — `Tried to allocate 76.00 MiB ... 0 bytes free`, 30.43 GiB of 31.86 GiB already allocated by PyTorch |
 | 3 x 245760, bf16 | **OOM identically** |
 | 3 x 237568, bf16 | PASS — KV 718,606 tokens |
-| 2 x 262144, bf16 + `--expert-precise-memory` | **DEPLOYED then crashed** — 0.13 GiB free at boot, NRestarts=3 |
+| 2 x 262144, bf16 + `--expert-precise-memory` | **DEPLOYED then crashed** — 0.13 GiB free at boot, NRestarts=3 (the ceiling, not the flag) |
 | 2 x 262144, bf16 (no precise flag) | did not stay up either: 0 bytes free at runtime, worker killed by a real request |
-| **2 x 200704, fp8, pool cut for the arena** | **DEPLOYED (2026-10-02, Frank's correction)**: fp8 KV, two 200704 windows, ~7,500 arena slots, ~3.6 GiB free VRAM measured |
-7,926 B/token against bf16's ~13,900-14,047 — at the cost of bf16's ~4 %
-decode edge. Six full-length contexts cannot fit at any KV dtype: the 6-seq
-graph/activation footprint leaves only 9.18 GiB for the pool, and 6 × 262144
-needs 12.47 GB even at fp8.
+| 3 x 200704, fp8, pool sized for 2 contexts | **DEPLOYED 2026-10-02 (shape D)**: 398,658 tokens = 1.99x concurrency, arena 14.87 GiB / 6,496 slots, the 2.75 GiB/rank pool cut returned as runtime headroom |
+| **3 x 200704, fp8, + `--expert-precise-memory`** | shape E (2026-10-02): same 1.99x pool, overhead 5.25 GiB, arena 17.22 GiB / 7,522 slots, boot healthy in 420 s |
+| 3 x 200704, fp8, MNBT 2048, graph term 0.32 GiB, util 0.982 | shape F (2026-10-03): arena 17.52 GiB / **7,653 slots** measured, budget 31.29, `Estimated CUDA graph memory: 0.32 GiB`, KV unchanged 398,658 |
+| 3 x 200704, fp8, graph term 0.07 GiB, util **1.00**, KV 3.1665e9 | **DID NOT BOOT (2026-10-03)**: `ValueError: Free memory on device cuda:N (31.29/31.86 GiB)` on both ranks, twice, zero weights loaded — the worker demands free > budget and budget == total. UTIL 1.0 is unbootable, full stop |
+| **3 x 200704, fp8, graph term 0.07 GiB, util 0.982, KV forced to 400,700** | **DEPLOYED 2026-10-03 (shape G)**: arena **17.77 GiB / 7,762 slots** measured (+109 over F, +240 over E), KV **401,408 tokens / 2.00x** concurrency measured, capture 0.31/0.10 GiB vs the 0.07 charge (under-charged, boots and holds), driver used 30.97 GiB -> **0.89 GiB real free per card** held through a live 4,096-token generation (~38-39 tok/s single-request, +13% vs the 33.99 shape-E baseline); rollback is UTIL or KV_BYTES down a notch |
+
+Six full-length contexts cannot fit at any KV dtype: the 6-seq graph/activation
+footprint leaves only 9.18 GiB for the pool, and 6 × 262144 needs 12.47 GB even
+at fp8.
 
 The old "KV_BYTES is PER RANK, 9/16 GiB OOM" law was the MTP4 WORKSPACE
 reservation, not a VRAM wall. Failures are clean and diagnostic: the engine
@@ -379,8 +705,8 @@ Without the copy the exec dies with `can't open file '/mtp_measure.py': [Errno 2
 No such file or directory` — measured 2026-10-01, which silently cost a whole
 arm's numbers. A window script must do the copy, not assume the path exists.
 
-### The deployed shape, measured again (2026-10-01 A/B window)
-A clean production-environment boot of the deployed config (4 x 262144 fp8, MNBT
+### The then-deployed shape, measured again (2026-10-01 A/B window)
+A clean production-environment boot of the config deployed at that time (4 x 262144 fp8, MNBT
 2048 in that window) reported **`GPU KV cache size: 1,079,582 tokens, Maximum
 concurrency for 262,144 tokens per request: 4.12x`** — the same figures as the
 deployed MNBT-1024 shape, i.e. MNBT does not move the pool, and the capacity is

serverAppConfigs-docs/LLM/frank-large-halogen27b-dflash2.md

@@ -0,0 +1,130 @@
+# frank-large — halogen container, Qwen3.8-27B + DFlash2 (Dankclaw, ACTIVE 2026-10-05)
+
+frank-large is the Strix Halo iGPU endpoint on Dankclaw (`:7770`). Since
+2026-10-05 it runs the **halogen** engine — `ghcr.io/peonist-ai/halogen` at a
+pinned digest, launched by `scripts/serve-frank-large.sh` under a root
+systemd unit as a podman container — serving
+**`peonist-ai/halogen-qwen3.8-27b`** (`qwen3.8-27b-p1w4d-d2.hgn`,
+35,865,565,184 B ≈ 33.4 GiB, sha256 `274c3fc7…`, 1352 tensors) with the
+**DFlash2** drafter at **one slot x 200,700 ctx**.
+
+It replaced the ROCmFPX llama.cpp fork endpoint ([[frank-large-rocmfpx-q6agent-uncensored-mtp]]),
+which remains the rollback target with its payload on disk.
+
+## Why swap
+
+|Metric|ROCmFPX Q6_0_ROCMFPX_AGENT + MTP (retired)|halogen qwen3.8-27b + DFlash2 (active)|
+|---|---|---|
+|Prose decode|27.6–28.1 t/s measured (4.8k depth, acceptance 0.868)|31.7 t/s mean across eight text workloads (publisher card, bf16 reference hardware)|
+|Copy-echo decode|31.4 t/s|publisher reports echo inside the same 31.7 mean|
+|Context|1 x 262,144|1 x 200,700|
+|Format|GGUF (fork-private tensor types 102/103)|`.hgn` — loads in NO other engine|
+
+The swap was accepted with eyes open on two product regressions, both
+confirmed by Frank before deploy:
+
+1. **Text-only.** halogen loads the language tower only; README "honest
+   limits": *"The vision encoder is ignored; image inputs are not supported."*
+   The mmproj-era vision role moves wholly to frank-small.
+2. **Censored base.** The checkpoint is a finetune of the censored
+   `Qwen/Qwen3.8-27B` base, built to make its DFlash2 head strong. The
+   ROCmFPX AGENT file was an *Uncensored* build; refusals can return.
+
+## Shape and why
+
+`scripts/frank-large.env` carries every tunable with its evidence; the load-
+bearing numbers:
+
+- **`HALOGEN_KV_SLOTS=1` + `HALOGEN_DRAFTER=2`** — the engine offers batch OR
+  speculation, never both (FLAGS.md: "Batching and speculation are exclusive:
+  a step batches when more than one conversation is running, so the drafter
+  stays off with more than one slot"). Slots>1 silently loses DFlash2; the
+  sibling halogen engine measured 41 -> 14 t/s when slots went up. frank-large
+  was a one-conversation endpoint before (PARALLEL=1), so the contract holds.
+- **`HALOGEN_SLOT_CTX=200700`** — the per-conversation HARD ceiling: a longer
+  prompt is a 400 naming the limit, never a truncation. The harness side is
+  wired to it: `models.yml` `contextWindow: 200700`, and OMP compacts at 70%
+  = 140,490, leaving 60,210 of the slot for the reply.
+- **`HALOGEN_MAX_TOKENS_CAP=65536` + `HALOGEN_QUEUE_TIMEOUT=7200`** — coupled
+  pair (FLAGS.md table): a request at the cap must not outlive the timeout.
+  65536 > the harness's 57,344 `maxTokens` so an eager request never meets
+  the engine's 400.
+- **KV math:** 1 slot x 200,700 x 64 KiB/token (d2 KV = 2x fp8) = 12.2 GiB
+  pool, inside the ~65 GiB left after 33.4 GiB mmap'd weights + engine working
+  set + host baseline on the 121.2 GiB box. The engine sizes the pool itself
+  and **prints a boot ledger naming any clamp** — the post-deploy check reads
+  that ledger, not a formula.
+- **No server-side sampler:** an absent temperature is GREEDY. The card's
+  thinking recipe (temp 1.0 / top_p 0.95) lives in `models.yml` `extraBody`
+  — on the wire only. The template defaults `reasoning_effort` to **xhigh**
+  when a request sends none, and **thinking tokens consume the
+  `max_tokens` budget**: a budget exhausted mid-thought returns
+  `finish_reason: "length"` with EMPTY `content` and the partial reasoning
+  in `reasoning_content` — clients must read both fields.
+
+## Client contract deltas vs llama.cpp
+
+|Behavior|halogen|
+|---|---|
+|Prompt over slot|400 naming the limit (no truncation)|
+|`max_tokens` over cap|400 naming the limit (never silent truncation)|
+|full-length request outliving `HALOGEN_QUEUE_TIMEOUT`|503s for everyone queued|
+|`timings`|every response carries llama-server-shape `timings` (`draft_n`, `draft_n_accepted`) — the drafter check reads the RESPONSE, and it proves speculation fired|
+|`response_format`|NOT supported (no constrained decoding)|
+|thinking field|`reasoning_content` (same as before)|
+|`reasoning_effort`|same Qwen3.8 template; outside {xhigh,medium,low} rejected — models.yml fold stays; unset defaults to xhigh, `none` disables thinking|
+|tools|server-side tool calls advertised — round-trip MUST be re-probed after deploy|
+|image input|ignored/error — route to frank-small|
+
+## Deploy mechanics
+
+- **Image:** `ghcr.io/peonist-ai/halogen@sha256:dc0a39a0016d6cfc58a197978febaafdf8d28403f724ded6111d98b5fb7ac0ea`
+  (published tag 0.1.4). The deployer gates the exact digest exists in podman
+  storage before dispatch. NOT the 2026-09 `halogen-flash-server` repo — that
+  was the Flash-Next w4b era, different image, retired.
+- **Payload:** `scripts/download-frank-large-halogen.sh` (fetch law: `curl -f`,
+  exact byte count, LFS sha256 where the Hub publishes one, `--verify-only`,
+  token never in argv) into `/home/dank/models/halogen-qwen3.8-27b/`. The
+  tokenizer is a FLAT DIRECTORY (`tokenizer/`, 5 files — tokenizer.json,
+  vocab.json, merges.txt, chat_template.jinja, tokenizer_config.json),
+  mounted whole at `/tokenizer`; the engine's `HALOGEN_TOKENIZER` default is
+  that directory, so the container gets no tokenizer flag (model-card
+  docker-run parity). The deployer and serve script refuse a missing file or
+  ANY symlink: an HF cache snapshot symlinks each entry into `blobs/` and
+  those links dangle inside the container mount.
+- **Unit:** `scripts/frank-large.service` (root, `User=dank`, relinked into
+  `/etc/systemd/system/`), `ExecStart` = serve script, `ExecStop` = `podman
+  stop -t 120 frank-large-halogen`, script `exec podman wait` so
+  `Restart=always` is real. Engine stdout goes to
+  `/home/dank/logs/frank_large.log` (the boot ledger lives there, not in the
+  journal).
+- **Cutover:** `deploy_llm_large` in `scripts/deploy-remote.sh` sweeps the
+  legacy bare `llama-server --model /home/dank/models/kingjones-*` process and
+  the legacy `frank-large` container name by NAME (never `pkill -f
+  llama-server` — that hits frank-small), retires the old quadlet if the 2026-09
+  halogen era's unit lingers, waits port 7770 free, relinks, health-gates.
+- **Bench law:** stop+mask the unit before any GPU experiment on this box
+  (AGENTS.md); `/health` answers 503 while the 33.4 GiB payload loads — that
+  is loading, not failure.
+
+## Verification (post-deploy checklist)
+
+1. `grep -iE 'clamp|kv|pool|drafter|dflash' /home/dank/logs/frank_large.log`
+   — ledger shows the pool at the asked shape, DFlash2 active, NO clamp line.
+2. `curl -s http://192.168.0.143:7770/health` -> 200.
+3. Real completion with `reasoning_effort` + temp/top_p from the wire recipe;
+   `reasoning_content` populated; decode rate from the timings.
+4. Spec decode fires: multi-token step behavior visible in rate; compare
+   against DRAFTER=0 arm if in doubt (never against gufo — bench law).
+5. Harness round-trip: an OMP call through `llamacpp (dankclaw)/frank-large`
+   incl. a tool call (V1TOOLS).
+6. Prompt > 200,700 tokens returns the 400 naming the limit (contract test).
+
+## Rollback
+
+Revert the frank-large swap commit and redeploy `dankclaw --service frank-large`:
+the ROCmFPX payload (`/home/dank/models/kingjones-Qwen3.8-27B-ROCmFP4-STRIX-MTP/`),
+the fork build (`/home/dank/GitPackages/kingjones-ROCmFPX/build`), and its
+`scripts/build-rocmfpx.sh` pin (commit `36e9acd`, seam-tested) all remain on
+disk. The halogen payload can stay: it costs nothing while the unit points
+elsewhere.

serverAppConfigs-docs/LLM/frank-large-ik-uncensored.md

@@ -123,3 +123,27 @@ Stop and disable the root unit, trash it, restore the halogen quadlet from
 `~/.AGENTTRASH/20260915-ik-swap/halogen-quadlet` to
 `~/.config/containers/systemd/`, `systemctl --user daemon-reload`, start;
 the .hgn checkpoint and the kingjones STRIX llama.cpp payload remain on disk.
+
+## Hybrid campaign envelope (2026-09-16, 3090 + Strix Halo)
+
+Why this deployment could not be tuned into a production endpoint: decode
+3.8–4.3 tok/s flat across thread counts (T1 ≈ T16); prefill ~192 tok/s at
+1.8k–32k prompts = the **CPU AVX512-VNNI compute ceiling** for CPU experts.
+The decode root cause is structural: ~100 CPU↔GPU backend splits per token
+with the GPU at 5 % SM during decode — launch/sync-bound over the PCIe x4
+dock, not compute- or memory-bound. T16 profiling shows ~90 % of cycles in
+`pause` (ggml_barrier spin): threads wait on the serial critical path.
+Memory placement (mmap/anon/THP/deferred) does not matter — all variants
+landed 3.8–4.6.
+
+- CUDA-graph capture aborts: qwen4exp's hyper-connection broadcasts use
+  per-layer `ggml_repeat_4d`, whose CUDA path is capture-illegal; until ik
+  gains capture-safe repeat/hc ops (mainline PR #28901 direction), graphs
+  stay dead.
+- Mainline llama.cpp itself runs 1.0 tok/s on this model (CPU MoE kernels
+  4× slower than ik's iqk) — do not switch engines for speed.
+- MTP/speculative decoding is impossible for mradermacher i1 GGUFs: the
+  NextN head is stripped (no `blk.48` tensors).
+- Tuning lever ranked first: `-ncmoe 42` (6 layers' experts on the 3090) →
+  decode 4.54, small-prompt prefill 32 — but VRAM 22/24.6 GiB at 262k ctx,
+  an operator OOM-margin decision, and it needs `GGML_CUDA_NO_PINNED_WEIGHTS=1`.

serverAppConfigs-docs/LLM/frank-large-rocmfpx-q6agent-uncensored-mtp.md

@@ -0,0 +1,177 @@
+---
+type: reference
+title: "frank-large on Dankclaw — ROCmFPX llama.cpp, Qwen3.8-27B Uncensored Q6_0_ROCMFPX_AGENT + MTP"
+description: "The active frank-large runbook — the kingjones30/ROCmFPX llama.cpp fork (pinned commit, gfx1151) serving kingjones777's Uncensored Q6_0_ROCMFPX_AGENT GGUF (ftype 114, protected output heads), with the repo's own MTP-Q4_0 head drafting four tokens at one native 262,144 window, f16 KV, the BF16 projector for vision, and the measured rejection of the ngram-mod chain."
+tags:
+  - llm
+  - frank-large
+  - dankclaw
+  - llama-cpp
+  - rocmfpx
+  - gfx1151
+  - gguf
+  - spec-decode
+  - mtp
+timestamp: 2026-10-03
+---
+
+# frank-large — ROCmFPX Uncensored Q6_0_ROCMFPX_AGENT + MTP (gfx1151)
+
+> **RETIRED 2026-10-03 → 2026-10-05 (rollback target; payload, fork build and
+> build script all on disk).** Replaced by [[frank-large-halogen27b-dflash2]]
+> (DFlash2, publisher 31.7 t/s mean vs the 27.6–28.1 measured here). This page
+> remains the contract record for the rollback. Original banner follows.
+> ACTIVE since 2026-10-03. Cut over from
+> [[frank-large-rocmfpx-q8agent-mtp]] (ALIGNED Q8_0_ROCMFPX_AGENT, 17.0 t/s)
+> on Frank's request: the *uncensored* variant of the same protected-heads
+> family, one quant rung down. The swap turned out to be a strict upgrade on
+> the axis that matters here: **decode 27.6–28.1 t/s at a 4,813-token prompt
+> against 17.0 at a 2,202-token prompt** — faster with a deeper context, not a
+> same-depth comparison, but the sign is not in question.
+
+## Identity
+
+| | |
+|---|---|
+| Host / device | Dankclaw (192.168.0.143), Radeon 8060S iGPU `gfx1151`, ~114 GiB GTT aperture |
+| Endpoint | `http://192.168.0.143:7770` — alias `frank-large`, `/v1` OpenAI surface |
+| Runtime | **`kingjones30/ROCmFPX`** llama.cpp fork @ `36e9acd40e10a87cd3c3ef8ec734668757dc8520`, built by [`scripts/build-rocmfpx.sh`](../../scripts/build-rocmfpx.sh) — unchanged from the Q8 era; the fork already knew type 102 |
+| Unit | root systemd `frank-large.service` ([`scripts/frank-large.service`](../../scripts/frank-large.service)), `User=dank`, `Restart=always` |
+| Serve script | [`scripts/serve-frank-large.sh`](../../scripts/serve-frank-large.sh) (owns the argv), tunables [`scripts/frank-large.env`](../../scripts/frank-large.env) |
+| Weights | `/home/dank/models/kingjones-Qwen3.8-27B-Uncensored-ROCmFP4-STRIX-MTP/Qwen3.8-27B-Uncensored-Q6_0_ROCMFPX_AGENT.gguf` — 24,856,607,008 B, sha256 `e966deb3…` |
+| Draft | `…/mtp-Qwen3.8-27B-Q4_0.gguf` — 1,680,271,648 B, sha256 `051a1764…` (byte-identical to the aligned repo's head; hardlinked, not re-downloaded) |
+| Projector | `…/mmproj-Qwen3.8-27B-BF16.gguf` — 931,145,888 B (hardlinked from the aligned payload dir; the same file the Q8 era served) |
+| Fetch | [`scripts/download-frank-large-kingjones-uncensored.sh`](../../scripts/download-frank-large-kingjones-uncensored.sh) — hardlinks the draft/projector from the aligned dir when present, otherwise fetches; `-f`, byte-count + LFS sha256 verified, `--verify-only`, token never on the command line |
+| Seam | `tests/frank_large_unit.bats` (payload gates, deployer/serve parity, spec block, KV dtype) |
+
+## Why this file
+
+`kingjones777/Qwen3.8-27B-Uncensored-ROCmFP4-STRIX-MTP-GGUF` ships the same
+"protected heads" construction as the aligned repo, in the uncensored tuning.
+Parsed header of the deployed file:
+
+| Field | Value |
+|---|---|
+| ftype | **114** — the fork's 64-bit-block family, ≈7.44 bpw |
+| Tensor mix | 353 F32 + **339 type-102 (`GGML_TYPE_Q6_0_ROCMFPX`)** + **159 type-103 (`Q8_0_ROCMFPX`)** |
+| Protected heads | 64 output-head tensors kept at plain precision — the AGENT construction from the aligned repo, at the 6-bit rung |
+| Arch | qwen35, 64 layers, `full_attention_interval 4`, `context_length 262144` |
+| Embedded sampling recipe | temp 1.0 / top_p 0.95 / top_k 20 |
+
+The same lock-in law as the Q8 era applies: types 102 and 103 exist in no
+loader but this fork; stock llama.cpp and gufo refuse the file at load
+(`unknown tensor type`). The loader was not re-litigated — the file picks the
+runtime, and the pin in `scripts/build-rocmfpx.sh` (commit by SHA) stays.
+
+Rejected sibling: `Q6_0_ROCMFPX_LEAN` (6.45 bpw, no protected heads) — the
+protection is what keeps the output-head logits at full precision and the
+drafter accurate, so it stays deployed.
+
+## Shape and budget
+
+Unchanged from the Q8 era — the swap was payload-only:
+
+| Dial | Value | Why |
+|---|---|---|
+| `--ctx-size` | 262,144 | the native window, no rope scaling |
+| `--parallel` | 1 | one conversation, all memory to it |
+| KV dtype | **f16** (both K and V) | ~64 KiB/token; free inside a 114 GiB aperture |
+| `--fit off` | on | the auto-fitter would silently resize the window |
+| `--flash-attn on` | on | required for the long-window attention path |
+| `--direct-io`, `--ctx-checkpoints 16`, `--cache-ram 8192`, `--cache-idle-slots`, `--jinja`, `--reasoning on --reasoning-format deepseek --reasoning-budget -1` | | same as [[frank-large-rocmfpx-q8agent-mtp]] |
+
+Spec decode: `--spec-type draft-mtp`, the repo's `mtp-Qwen3.8-27B-Q4_0.gguf`,
+`--spec-draft-n-max 4`, pinned `--spec-draft-device ROCm0`.
+
+Measured steady state: GTT **~61 GiB** under live load (host baseline 16.7),
+boot to `/health` **~15 s**.
+
+## Measured on this payload (2026-10-03, production service, `/completion` timings, temp 0, `cache_prompt: false`)
+
+| Traffic | Decode t/s | Prefill t/s | Draft acceptance |
+|---|---|---|---|
+| Prose, 4,813-token prompt, 512 tokens | **27.57 / 28.10** (two runs) | ~205–217 | **396/456 = 0.868** → mean acceptance length ≈ 4.4, i.e. drafting at nearly the full n-max 4 |
+| Schema echo (structured repetition) | **31.44** | — | 639/639 = 1.000 |
+| Block copy (text repeated verbatim) | ~28.0 | — | 0.848 |
+
+vs the Q8 era's 0.462 acceptance at a 2,202-token prompt: the MTP head drafts
+materially better on this file. The card's claimed 0.953-class acceptance is
+no longer science fiction on real prose — but keep reading it as a file
+property, not a throughput promise.
+
+## The ngram-mod chain — measured and rejected
+
+Frank's ask: layer `--spec-type draft-mtp,ngram-mod` on top of MTP for an
+extra gain. Measured in three bench windows (production service stopped per
+the benchmarking law; artifacts `/home/dank/fl-ngram-ab/` on Dankclaw),
+comparing `draft-mtp` alone vs the chain on the same traffic:
+
+| Traffic | MTP alone | MTP + ngram-mod | Delta |
+|---|---|---|---|
+| Prose | 27.57 t/s | 23.40 t/s | **−15 %** |
+| Block copy | 28.0 t/s | 26.1 t/s | **−7 %** (with ngram-mod demonstrably firing: 138 proposals/run) |
+
+Mechanism, straight from arm B's boot log:
+
+```
+recurrent state rollback is not compatible with 'ngram-mod' - disabling rollback support
+```
+
+On the qwen35 hybrid (Gated DeltaNet recurrent layers) adding the ngram layer
+disables the rollback path MTP's own verification relies on, so acceptance —
+and throughput — drop on *all* traffic, even where the ngram matcher fires.
+**Deployed decision: `draft-mtp` alone.** Chaining is not free on recurrent
+models; the fleet law in `AGENTS.md → llama.cpp serving` records it.
+
+## Bench-window hardening laws (learned building the A/B)
+
+1. **Health 200 is not proof of ownership.** After boot, verify the pid you
+   started owns the port (`ss -ltnp`); a zombie server from a prior window
+   answers `/health` too and silently serves your arm.
+2. **Kill scratch servers with SIGKILL after a grace period.** llama-server's
+   graceful SIGTERM waits on open keep-alive connections — a harness holding
+   the socket leaves the process (and its ~24 GiB of GTT) alive after the
+   script exits.
+3. **Prove the port is free and GTT dropped between arms**, or the second
+   arm measures a collision.
+
+## Client contract
+
+Unchanged from the Q8 era and re-verified on this payload 2026-10-03: model id
+`frank-large`, native tools, thinking in `reasoning_content`,
+`reasoning_effort` ∈ {xhigh, medium, low} only (anything else HTTP 500),
+card-recipe sampling defaults server-side, vision through the BF16 projector
+(red PNG → **`Red`**, 74 prompt / 165 completion tokens). The traps list —
+`content: null` when reasoning eats the budget, the self-recovering MTP
+boundary log line, `enable_thinking:false` not `reasoning_effort:"none"` — is
+in [[frank-large-rocmfpx-q8agent-mtp]] and still applies.
+
+Sweep key for scratch servers is now the payload name:
+`pkill -f 'Qwen3.8-27B-Uncensored-Q6_0_ROCMFPX_AGENT'` — never `pkill -f
+llama-server` (frank-small is also one).
+
+## Operations
+
+```bash
+./scripts/deploy.sh dankclaw --service frank-large
+systemctl status frank-large
+journalctl -u frank-large -n 200 --no-pager
+scripts/download-frank-large-kingjones-uncensored.sh --verify-only
+```
+
+## Rollback
+
+| Target | How |
+|---|---|
+| **ALIGNED Q8_0_ROCMFPX_AGENT** (previous shape, 17 t/s) | the payload dir `/home/dank/models/kingjones-Qwen3.8-27B-ROCmFP4-STRIX-MTP/` is untouched on disk; revert the swap commit (unit/env/serve trio) and redeploy, or point `FRANK_LARGE_MODEL_FILE` + `LLM_LARGE_MODEL_FILE` at the old paths. Verify with `scripts/download-frank-large-kingjones.sh --verify-only` |
+| gufo Q4_K_XL era | git tag `frank-large-gufo-q4kxl-20261001`, **but the gufo payloads (UD-Q4_K_XL 16.35 GiB, UD-Q8_K_XL + DFlash2 drafter) were removed from disk 2026-10-03 with Frank's approval** — this rollback now requires re-download per [[frank-large-gufo-qwen38-27b]] |
+
+## When it misbehaves
+
+Same decision tree as [[frank-large-rocmfpx-q8agent-mtp]]:
+`unknown tensor type` ⇒ off the pinned fork; SIGSEGV in
+`ggml_compute_forward_set_rows` ⇒ pre-`36e9acd` build; slow decode with no
+error ⇒ draft not engaged (`draft_n` in `/completion` timings) or an orphaned
+second copy eating GTT (`/sys/class/drm/card1/device/mem_info_gtt_used`);
+500 naming Jinja ⇒ `--jinja` lost; 500 naming the template ⇒ unmapped
+`reasoning_effort` in `models.yml`.

serverAppConfigs-docs/LLM/frank-large-rocmfpx-q8agent-mtp.md

@@ -1,7 +1,7 @@
 ---
 type: reference
-title: "frank-large on Dankclaw — ROCmFPX llama.cpp, Qwen3.8-27B Q8_0_ROCMFPX_AGENT + MTP"
-description: "The active frank-large runbook — the kingjones30/ROCmFPX llama.cpp fork (pinned commit, gfx1151) serving kingjones777's Q8_0_ROCMFPX_AGENT GGUF, a ftype-115 file no other loader can read, with the repo's own MTP-Q4_0 head drafting four tokens at one native 262,144 window, f16 KV, and the bundled BF16 projector for vision. Measured MTP-vs-none A/B, the client contract, and rollback to gufo."
+title: "frank-large on Dankclaw — ROCmFPX llama.cpp, Qwen3.8-27B Q8_0_ROCMFPX_AGENT + MTP (retired)"
+description: "The retired frank-large runbook — the kingjones30/ROCmFPX llama.cpp fork (pinned commit, gfx1151) serving kingjones777's Q8_0_ROCMFPX_AGENT GGUF, a ftype-115 file no other loader can read, with the repo's own MTP-Q4_0 head, f16 KV, and the BF16 projector. Measured MTP-vs-none A/B, the client contract, and the rollback state as of 2026-10-03."
 tags:
   - llm
   - frank-large
@@ -15,14 +15,20 @@ tags:
 timestamp: 2026-10-02
 ---
 
-# frank-large — ROCmFPX Q8_0_ROCMFPX_AGENT + MTP (gfx1151)
+# frank-large — ROCmFPX Q8_0_ROCMFPX_AGENT + MTP (gfx1151, retired)
 
-> **ACTIVE since 2026-10-02.** Cut over from
-> [[frank-large-gufo-qwen38-27b]] (Q4_K_XL, ~15 t/s). Both requests came from
-> Frank: *highest quality that runs*, and *one full context*. This endpoint
-> delivers 8.39 bpw weights — two quant rungs above what gufo could load — and
-> measures **17.0 t/s** decode instead of 15, so quality was not bought with
-> speed.
+> **RETIRED 2026-10-03** — superseded by
+> [[frank-large-rocmfpx-q6agent-uncensored-mtp]]: Frank swapped to the
+> uncensored Q6_0_ROCMFPX_AGENT file (protected heads, one rung down), which
+> measures **27.6–28.1 t/s** at a deeper prompt than this file's 17.0. This
+> runbook stays live as the rollback reference and the trap catalogue; the
+> payload it documents remains on disk at
+> `/home/dank/models/kingjones-Qwen3.8-27B-ROCmFP4-STRIX-MTP/`.
+>
+> Cut over 2026-10-02 from [[frank-large-gufo-qwen38-27b]] (Q4_K_XL, ~15 t/s)
+> on two requests from Frank: *highest quality that runs*, and *one full
+> context*. It delivered 8.39 bpw weights — two quant rungs above what gufo
+> could load — at 17.0 t/s, so quality was not bought with speed.
 >
 > The capability that forced this whole shape is a GGUF tensor type:
 > **`GGML_TYPE_Q8_0_ROCMFPX` (103)**. It exists in exactly one loader on this
@@ -184,17 +190,24 @@ broken payload never stops the running unit. Caller environment **overrides**
 `frank-large.env`, and defaults use `:=` so an explicitly empty value surfaces
 as the error it reads as.
 
-## Rollback
+## Rollback (as of retirement, 2026-10-03)
 
-git tag **`frank-large-gufo-q4kxl-20261001`**: gufo v0.1.1
-(`scripts/build-gufo.sh` still on disk), the Q4_K_XL Qwen3.8-27B payload and the
-DFlash2 drafter all remain on the host. Restore that tag's unit/env/serve trio
-and redeploy — nothing here deletes anything.
+**Primary rollback: the ALIGNED Q8 file itself is the previous shape's
+payload and remains on disk** at
+`/home/dank/models/kingjones-Qwen3.8-27B-ROCmFP4-STRIX-MTP/` — revert the
+2026-10-03 swap commit (unit/env/serve trio) or repoint
+`FRANK_LARGE_MODEL_FILE`/`LLM_LARGE_MODEL_FILE`, verify with
+`scripts/download-frank-large-kingjones.sh --verify-only`, redeploy.
 
-The rollback payloads stay exactly where the tag's env file expects them:
-`/home/dank/models/Qwen3.8-27B-UD-Q4_K_XL/` (16.35 GiB) and the DFlash2
-drafter `/home/dank/models/Qwen3.8-27B-UD-Q8_K_XL/Qwen3.8-27B-DFlash2-Q4_K_M.gguf`
-(1.07 GiB). The one-day vLLM MiMo era's payloads (MiMo-V2.6 BF16 18 GiB,
+git tag **`frank-large-gufo-q4kxl-20261001`** (gufo v0.1.1, Q4_K_XL +
+DFlash2) is now a **re-download** rollback: the gufo payloads
+`/home/dank/models/Qwen3.8-27B-UD-Q4_K_XL/` (16.35 GiB) and
+`/home/dank/models/Qwen3.8-27B-UD-Q8_K_XL/` (incl. the DFlash2 drafter) were
+removed from disk on 2026-10-03 with Frank's explicit approval to free space.
+`scripts/build-gufo.sh` remains on disk; re-fetch per
+[[frank-large-gufo-qwen38-27b]].
+
+The one-day vLLM MiMo era's payloads (MiMo-V2.6 BF16 18 GiB,
 Qwen3.5-9B MTP source 19 GiB, both DFlash draft dirs) were staged to
 `/home/dank/.AGENTTRASH/20261002-retired-llm-payloads/` on 2026-10-02 —
 recoverable until the trash is purged, and never part of a rollback plan.

serverAppConfigs-docs/LLM/frank-medium-batch-ubatch.md

@@ -134,6 +134,59 @@ slots of 131072, `6.44 / 6.90 GiB` free — matching the pre-deploy measurement
 0.01 GiB — a tool call and two concurrent requests answered, draft acceptance on
 both slots (0.63 / 0.48 on slot 1, 0.39 on slot 0), and no ROCm errors.
 
+## The split: re-measure it at the deployed shape
+
+The batch and the tensor split are **coupled**: compute buffers are not
+symmetric across the two cards, so any batch change moves the balance point.
+Order of work — sweep batch with llama-bench (above), THEN re-measure the
+split at the DEPLOYED shape (262144, two slots, drafter loaded) by loading
+the real service with a per-config env override and reading `rocm-smi`. A
+llama-bench number cannot decide the split: it has no `--parallel`, sizes
+its own context as `n_prompt + n_gen + n_depth`, and cannot load the drafter.
+
+The settle gate is mandatory (above); wait for BOTH cards under ~1 GiB AND
+the port to be free (`ss -ltn | grep :7770` until gone) before the next
+load, or the transient server dies with `couldn't bind HTTP server socket`.
+Free bytes follow `card = weights_share + KV_share + constant + ubatch_term`,
+where the ubatch term costs the drafter-carrying card roughly **1.85×** the
+other (measured 2.41 GiB vs 1.30 GiB going 1536 → 512). Balance by
+measurement only: one `-ts` step is one layer, worth ~0.43 GiB on a DeltaNet
+boundary and ~1.43 GiB on an attention one, so the reachable minimum gap is
+about one DeltaNet layer, never zero. Reference: 36,29 @1536 → 4.71/4.91;
+36,29 @512 → 6.01/7.32; **35,30 @512 → 6.44/6.90 (deployed)**.
+
+Re-fit both ubatch coefficients in `tests/frank_medium_unit.bats` at the new
+deployed point and re-anchor the per-card constants; check neighbouring
+splits still FAIL the balance gate or the seam has stopped discriminating
+(keep the 1.5 GiB/card floor and the one-DeltaNet-layer slack). Deploy,
+then verify live: flags in `/proc/<pid>/cmdline`, `/slots`, VRAM matching
+the pre-deploy measurement, a tool call, two concurrent requests,
+`draft acceptance` lines on BOTH slots, 0 ROCm errors.
+
+**Per-card utilization (who is hot, and the split lever)** — a different
+question from batch/prefill. Measure decode tok/s AND per-card
+`rocm-smi --showuse` at 2–3 neighbouring splits in ONE session. Pitfalls
+(each cost a re-run): run the transient unit as ROOT (a `pkill -9
+llama-server` as dank auto-restarts the service and `systemctl` as dank
+fails "Access denied"); warm up 3 throwaway requests before the timed one
+(first 384-token decode after load measured 1.77 tok/s against a true ~32);
+`rocm-smi --showmeminfo vram --csv` columns are `$2`=Total, `$3`=**Used** —
+parsing the wrong column spins the settle gate the full 600 s; the util% is
+sampled post-hoc (which card finished last), so treat it as "which card is
+the pipeline tail", not a duty cycle.
+
+**What the split does and does not buy (measured 2026-09-17, 393216 /
+MTP n=3):** the split is NOT a decode-throughput lever — 34:31, 35:30, 36:29
+all sit in the same ~32 tok/s band. The x1 card runs hot (~91–97 %) at
+35:30 because the MTP drafter + output layers + heavier compute buffers are
+pinned to the card holding the LAST layers — not because PCIe is the
+bottleneck (~5.2 MB of boundary activation crosses per 512-token ubatch).
+36:29 makes the x16 card the utilized worker (42 %/3 %) at no throughput
+cost but spends headroom (0.45 GiB free, under the floor) — Frank's standing
+preference is that the card with the FASTEST PCIe lane should be the more
+utilized one; deploying it requires re-fitting the seam's slack/floor and
+re-checking that 35:30 and 37:28 still fail so the gate keeps discriminating.
+
 ## Limitations
 
 - **`llama-bench` cannot run the drafter, the slots or the deployed context.**

serverAppConfigs-docs/LLM/frank-primary-freetoken-pr587.md

@@ -0,0 +1,170 @@
+# frank-primary — FreeToken PR #587 ROCm (Qwen3.8-Flash-Next-Uncensored-FP8, 1× R9700)
+
+**Status: ACTIVE 2026-10-05 →** · **Host:** Dankserver (192.168.0.235) · **Port:** 7770 · **GPU:** 1× Radeon AI PRO R9700 (gfx1201) — the second R9700 sits idle
+
+The tcclaviger RDNA4 vLLM fork was replaced by **FreeToken at PR #587 head
+`77a7201f98f00d93f618c6cf8b81ffe8360e0899`** (branch `rocm714_torch214`), built
+on the host into the local image
+`localhost/frank-primary-freetoken:pr587-77a7201` by
+`scripts/docker/frank-primary-freetoken.Dockerfile`. Same port, same served
+alias (`frank-primary`), same 131-shard payload
+(`/home/dank/models/Qwen3.8-Flash-Next-Uncensored-FP8`). Rollback:
+git tag `frank-primary-vllm-shapeG-20261003` (the vLLM trio + its env lore),
+runbook [[dankserver-qwen38-fp8-expert-offload-tuning]].
+
+## Why FreeToken
+
+The vLLM fork planned a per-boot VRAM/host tiering from an estimator: util
+0.982 was a hard law (1.00 was **unbootable** — worker init raises
+`ValueError: Free memory on device cuda:N (31.29/31.86 GiB)` when budget ==
+total), the expert arena capped at 17.77 GiB / 7,762 slots, and live traffic
+still degraded ~40k tokens in (the GDN state-fault class, never fixed in the
+fork). FreeToken keeps **all expert weights in pinned host RAM** with an LRU
+GPU cache, sizes **KV as an explicit token pool**, and boots in one
+deterministic pass — no per-boot estimator to fight.
+
+## The build (on dankserver, rootless podman as dank)
+
+```bash
+cd /home/dank/serverAppConfigs
+podman build -t frank-primary-freetoken:pr587-77a7201 \
+  -f scripts/docker/frank-primary-freetoken.Dockerfile scripts/docker/
+```
+
+Laws baked into the Dockerfile (each one cost a failure):
+
+- **Base digest-pinned**: `rocm/pytorch:rocm7.14_ubuntu24.04_py3.12_pytorch_release_2.11.0@sha256:a223aee1…` —
+  the PR doc's known-good stack (torch 2.11.0+rocm7.14, triton 3.7). No
+  rocm7.14+torch2.14 image exists on Docker Hub (the 2.14 images are
+  rocm10.1.0 — an untested pair for this PR).
+- **`--no-build-isolation --no-deps -e .`** — a plain
+  `pip install -e .` on ROCm can REPLACE the image's ROCm torch with a CUDA
+  build (the dependency set is authored for NVIDIA). Runtime deps are
+  curated explicitly; `flashlib==0.3.0 --no-deps` (its numba /
+  nvidia-cutlass-dsl deps are CUDA-only); ninja for the QSA JIT;
+  **flashinfer / sgl_kernel intentionally absent** (no ROCm builds;
+  FreeToken falls back to its Triton kernels — the PR's 12 pytest failures
+  are exactly those backends).
+- **`libhiprtc.so` dev symlink**: the pip-ROCm torch ships only the
+  versioned `libhiprtc.so.7` in
+  `/opt/venv/.../site-packages/_rocm_sdk_core/lib`, but the PR's `_cpu_moe`
+  extension links `-lhiprtc`. First build died `ld: cannot find -lhiprtc`;
+  the Dockerfile symlinks the unversioned name into the ALREADY-LINKED dir.
+- FreeToken source is checked out at the **pinned PR head**, not the branch
+  (a future push to `rocm714_torch214` cannot change the deployed engine).
+- The stale Docker Hub login (`docker login` creds at
+  `~/.docker/config.json`) poisons podman's ANONYMOUS pulls of public images
+  with "invalid username/password" — `docker logout` fixes the pull. Hit
+  2026-10-05 while pulling the base image.
+
+## Measured at cutover (2026-10-05/06)
+
+
+- **Boot**: ~2.3 min container-start → API-ready (131 shards page-in + PLE
+  pin + expert staging + CUDA graphs `[1,2]`). VRAM after graphs:
+  **1.80 GiB free** of 31.86. RAM: 122 GB used + **113 GB shared (the
+  pinned expert/PLE slabs)**, 123 GB still available.
+- **Decode: 17.1 tok/s** single-stream, temp 0, 400-token probe (clean
+  1→N counting). The vLLM incumbent measured ~30 tok/s on TWO cards;
+  FreeToken single-card is the deployed baseline.
+- **The single-card LRU ceiling**: the KV contract (harness worst case
+  177,356 tokens) + dense fp8 + the vision tower + graphs leave ~12 GiB for
+  the expert LRU at rate 0.10 — and only 1.8 GiB of live headroom, so
+  `ft ctl cache --moe` cannot raise it far on one card. The paths to a
+  better decode rate are TP=2 (below) or a smaller pool (contract
+  violation — the 177,356 floor is real traffic, not padding).
+- **TP=2 exists but does not boot on ROCm** (probed 2026-10-06,
+  zero-impact window): `--tensor-parallel-size 2 --gpu 0,1` dies in the TP
+  allreduce JIT — `pynccl.cu` compiles per-rank at boot and clang can't
+  resolve `cuda_runtime.h` from the hipified include path
+  (`1 error generated when compiling for gfx1201`, ninja stopped). The PR
+  author validated single-GPU only; this is FreeToken's untested-TP-on-ROCm
+  edge, not a board or driver fault (radiance TP=2 allreduces fine on this
+  pair since the 2026-09-20 swap). A future fix = patch the JIT include
+  path (HIP's cuda_runtime shim) or vendor an ROCm allreduce — re-probe
+  then.
+- **Verification gates passed**: /health ok; plain completion `FLEET-OK`
+  (finish stop); thinking split — `reasoning_content` non-empty AND
+  `content` carries the answer (17×24 → "408"); native `tool_calls` with
+  correct name+arguments (finish tool_calls); 256×256 solid-red PNG through
+  an `image_url` part answers exactly "Red" (prompt_tokens 127); stats:
+  native ctx 262,144, KV 4,687 pages × 64 = 299,968, modalities
+  [text,image], sampling defaults from generation_config.json
+  (temp 1.0 / top_p 0.95 / top_k 20).
+- **Thinking counts against max_tokens** (same as the vLLM era): a
+  `max_tokens: 10` probe returns EMPTY content with the budget spent in the
+  think block. Clients must budget for thinking or disable it.
+- **Prometheus scrapes /metrics → 404** (Dankclaw scrapes 192.168.0.235:7770
+  from the llm-fleet job; FreeToken has no /metrics endpoint). The target
+  will read down until the scrape config is repointed or a metrics shim
+  lands — same class as the retired HTML-probe targets
+  (see [[Observability]]).
+
+
+## The shape
+
+| Knob | Value | Why |
+|---|---|---|
+| `--gpu` / `HIP_VISIBLE_DEVICES` | 0 / 0 | ONE R9700; visibility pinned so in-container index 0 is the right card |
+| `--num-tokens` | **299,968** | KV pool in TOKENS; **must be a multiple of 64** (the auto attention backend is `qsa_sparse`, which overrides page size to 64 — 300,000 was refused: `nearest valid values: 299968 or 300032`) |
+| `--max-running-requests` | 2 | first-boot conservative; raise after real-traffic slot-occupancy look |
+| `--max-output-tokens` | 36,864 | harness maxTokens; requests that omit max_tokens get this default |
+| `--moe-cache-rate` | 0.10 | GPU expert LRU as fraction of all experts (~120 GiB fp8 → ~12 GiB VRAM); first-boot conservative |
+| parsers | AUTO | thinking → `reasoning_content` natively; tools in the qwen3.8 dialect; NO `--reasoning-parser`/`--tool-call-parser` flags |
+
+KV math: 12 full-attention layers × 2 KV heads × 256 head_dim × 2 (K+V) ×
+2 B = **24 KiB/token** (f16 KV; the 36 GDN layers use fixed per-seq state,
+not per-token KV) → 299,968 × 24 KiB ≈ 7.0 GiB. Clears the harness worst
+case (140,492-token prompt + 36,864 max_tokens = 177,356) and admits ONE
+native 262,144-token conversation + full budget. `models.yml`
+`contextWindow: 200704` stays (compacts early, never 400s).
+
+## RAM math (why the root unit + memlock)
+
+246 GiB total. FreeToken pins ~120 GiB of fp8 expert rows + the ~47.7 GiB
+PLE ngram table in host RAM → the unit carries
+`LimitMEMLOCK=infinity` (root unit; dank's login session caps at 8 MiB and
+a user quadlet cannot raise it). Weights+PLE+overhead ≈ 190 GiB — the vLLM
+container's ~170 GiB MUST be down first; `systemctl stop frank-primary`
+frees it (measured: 207 GB free right after the stop).
+
+## Resize live
+
+```bash
+ft ctl cache --kv <tokens>   # grows the KV pool LIVE (shrink needs restart)
+ft ctl cache --moe <rate>    # same for the expert LRU
+ft ctl stats                 # pool occupancy, VRAM, throughput
+```
+
+## Verification (the fleet's real-completion law)
+
+1. `/health` 200 — necessary, not sufficient.
+2. Plain completion returns the asked-for string.
+3. Thinking request: `reasoning_content` non-empty AND `content` carries the
+   answer (the models.yml `reasoningContentField: reasoning_content` flip
+   rides on this).
+4. Vision probe: a solid-color PNG through an `image_url` part answers the
+   exact color (the legacy top-level `images` field trap does not exist
+   here — FreeToken is OpenAI-native).
+5. Unit-managed: `systemctl show -p MainPID` behind the health answer (the
+   2026-09-26 stray-port false green).
+
+## Rollback
+
+```bash
+git checkout frank-primary-vllm-shapeG-20261003 -- \
+  scripts/frank-primary.env scripts/serve-frank-primary.sh \
+  scripts/frank-primary.service
+./tests/run.sh && git commit -m "frank-primary: rollback to vLLM shape G" \
+  && git push && ./scripts/deploy.sh dankserver --service frank-primary
+```
+
+The tcclaviger image and the FreeToken image coexist in dank's rootless
+storage; the deploy re-pulls the pinned digest if absent.
+
+## Index
+
+- [[dankserver-qwen38-fp8-expert-offload-tuning]] — the retired vLLM shape
+  (rollback target; its 362-line env lore lives in the tag)
+- [[freetoken-frank-small]] — the v0.1.2 CUDA precedent (ortobox, RTX 3090)
+- [[LLM/index]] — fleet index

serverAppConfigs-docs/LLM/frank-primary-glm53-flash-w4a16.md

@@ -45,6 +45,98 @@ timestamp: 2026-09-30
 > because the fork compiles its own Qwen trunk and not its GLM one. See
 > [The measured profile](#the-measured-profile-and-the-real-wall).
 
+## DFlash2 target-side port — the four laws and the KV ladder
+
+Distilled 2026-10-05 from the probe windows (skills `glm53-dflash-boot-ladder`
+/ `glm53-dflash-eagle3-port`); each law was proven by a real failed boot.
+
+**The four EAGLE3-port laws** (make the GLM trunk satisfy `SupportsEagle3`):
+
+1. **Patch the instance the runner calls.** `model_runner.py:425` calls
+   `set_eagle3_aux_hidden_state_layers(self.model)` on the *loaded* model —
+   GLM loads as `Glm5NextForConditionalGeneration` (multimodal wrapper;
+   `self.language_model = Glm5NextForCausalLM`). Members placed only on the
+   CausalLM fail the check on the conditional — same error text, wrong file.
+   Patch BOTH wrappers.
+2. **Presence-checked member set.** `SupportsEagle3` (interfaces.py:1659) is
+   `@runtime_checkable`; `isinstance` walks ALL `__protocol_attrs__`
+   (including base-protocol members) with `getattr_static`, breaking on the
+   first miss. Both wrappers need, as class attributes:
+   `supports_eagle3: ClassVar[Literal[True]] = True`,
+   `has_own_lm_head: bool = False`, `has_own_embed_tokens: bool = False`.
+   Verify in-image with stock typing — no boot needed.
+3. **MRO forbids inheriting the Protocol.** GLM wrappers carry four/five
+   Protocol-derived bases; adding `SupportsEagle3` anywhere dies with
+   "Cannot create a consistent method resolution order" (sim-proven: the
+   ordering rejects it in every base position). The attribute route avoids
+   the MRO, so the two methods must be written by hand: the CausalLM
+   unwraps to `self.model` (the `EagleModelMixin` trunk), the conditional
+   delegates to `self.language_model` (pixtral.py:498 pattern);
+   `get_eagle3_default_aux_hidden_state_layers` returns `(2, L//2, L-3)`,
+   with the drafter config's `target_layer_ids` winning when present.
+4. **Trunk capture point:** append `aux_hidden_states` BEFORE the layer call
+   — GLM's loop carries the single-stream state entering the layer.
+
+**The KV blocker ladder** (patches `scripts/patches/glm53-dflash-0*`):
+
+1. EAGLE3 gate → presence-checked attrs on both wrappers (above).
+2. KV page-size unifier: MLA layers cannot be padded (sparse MLA reads
+   whole token rows) but CAN grow block_size when the max page is an exact
+   multiple — bump `max_page_size` in `unify_kv_cache_spec_page_size` to
+   the `lcm` of all MLA-family pages first (patch 02).
+3. kpool tail assert `tail_kv_cache.shape[2] == pool_size`: the unifier's
+   divisible branch grew KpoolTailSpec's block; its kernel pins block ==
+   index_kpool. Add a KpoolTailSpec branch that pads `page_size_padded`
+   instead of growing (patch 03).
+4. KV capacity: the glm5 layout ALIASES mamba states INTO MLA pages (why
+   GLM fits 32 GiB cards); the generic path (taken when draft/hidden specs
+   exist) cannot alias → ~30 GiB demanded vs a 0.92 GiB reserve. Extend
+   `_get_kv_cache_groups_glm5_next` to partition
+   HiddenStateCacheSpec + drafter attention specs; the drafter forms its
+   own uniform group (per-group pages, NO unification → the MLA block
+   stays at the engine-auto-set 2,432 from interface.py:918, kpool guard
+   `block_size % 512 == 0` unstressed); layout returns a 10-tuple
+   (+draft_group, +hidden_groups); the builder sizes ONE shared backing
+   allocation (`allocate_kv_cache` asserts all KVCacheTensor sizes equal) =
+   aliased pool + appended draft/hidden tensors (patch 04). After 04 the
+   capacity requirement collapsed 31.76 → 1.88 GiB at 65,536 max len and
+   the engine SERVED (READY ~500 s, dflash2 fused prep ON, drafter FULL
+   graphs on both TP ranks).
+5. Reserve registry: `_glm53_kv_bytes_per_token` must add 4,096 (drafter
+   KV) + 5×hidden×2 (aux hidden cache) keyed on
+   `speculative_config.method in {dflash, dspark, extract_hidden_states}`.
+   NEVER key on `use_aux_hidden_state_outputs` — it lives on the
+   gpu_model_runner instance (gpu_model_runner.py:623/678), unset at plan
+   time, so the check silently fails and the reserve stays 13,600
+   (patch 04b). Proven reserve math at 65,536: 13,600 + 4,096 +
+   5×4096×2 ≈ 58,656 B/token → ~4.15 GiB reserve, arena ~10.3 GiB.
+
+**Verify cheaply before booting:** smoke-test the pure KV logic on CPU
+torch with the image's real spec classes — stub `vllm.logger`/`vllm.utils.*`,
+register the spec classes in `KVCacheSpecRegistry` with a dummy manager, add
+`from __future__ import annotations`, and run synthetic GLM+draft+hidden
+spec sets through the patched grouping/layout/builder functions. A 7-minute
+boot per hypothesis is the alternative.
+
+**Window discipline:** stop + mask `frank-primary` FIRST and gate on
+MemAvailable ≥ 230 GiB before booting the probe (the expert-offload planner
+reads MemAvailable; beside-prod boots die at plan time with `ExpertPlanError:
+host holds 5113 of 12096 swappable experts`). Save `podman logs glm-probe`
+to a file BEFORE teardown — a `podman logs | grep | head` pipeline died once
+under pipefail via SIGPIPE before log-save, losing the capture. The abort
+path must restore prod (unmask + daemon-reload + reset-failed + start),
+never bare-exit after stop+mask. Launcher knobs: `DFLASH_PATCH=1` (mounts
+the patched model.py), `DRAFT_MOUNT=1`, `SPECJSON_FILE=…` ({"method":
+"dflash","model":"/draft","num_speculative_tokens":8}) — note the launcher's
+echo prints `patch=$PATCH_MOUNT` (the COMPILE knob), not DFLASH_PATCH.
+Boot timeline: EAGLE3 gate fires ~4 min in at worker init; drafter
+torch.compile is the frontier (~380 s).
+
+**Open telemetry discrepancy:** the contaminated window read 1.5–2.1 tok/s
+with acceptance 1.00–1.25; Frank live-measured ~18 tok/s decode / ~370 tok/s
+prefill. Reconcile with a clean single-request measurement before trusting
+either.
+
 Related: [[dankserver-qwen38-fp8-expert-offload-tuning]] (the same fork, the same
 hardware — its conclusion was drawn from throughput arithmetic and is superseded
 by this page's kernel-time measurement),

serverAppConfigs-docs/LLM/frank-primary-llamacpp-qwen38-flash-next-q8.md

@@ -110,6 +110,32 @@ retires the lucebox user quadlet to dated home trash
 truly free, relinks + enables + restarts the root unit, and waits up to
 1800 s for `/health` (cold start page-ins ~175 GiB and builds the graphs).
 
+### Preflight the GGUF without downloading it
+
+GGUF metadata lives in the header and HF honours ranged GETs, so a few MB
+answers every sizing question before a 13 GB transfer
+(`Range: bytes=0-8000000`). Parse: magic `GGUF`, u32 version, u64
+n_tensors, u64 n_kv, then key/value pairs (string = u64 len + bytes; ARRAY
+= u32 elem type + u64 count; BOOL is 1 byte — the type hand-written parsers
+forget). Walk arrays by byte size even for a summary: a naive parser
+desynchronises on `tokenizer.ggml.tokens` and lands gigabytes off. Read:
+`general.architecture`, `<arch>.block_count`, `<arch>.context_length`,
+`<arch>.nextn_predict_layers` (MTP is a property of the FILE — the `-mtp`
+build carries `blk.<n>.nextn.*`; `--spec-type draft-mtp` then loads the
+head against the MAIN model's weights, and `--model-draft` would load the
+file twice), `<arch>.head_count_kv`, `<arch>.key_length`/`value_length`,
+`<arch>.full_attention_interval`, `<arch>.ssm.state_size`/`group_count`/
+`time_step_rank`, and for the projector `clip.projector_type` +
+`clip.has_vision_encoder`.
+
+KV arithmetic for the hybrid (16 × (3 Gated DeltaNet + 1 Gated Attention)):
+only full-attention layers take KV
+(`attn_layers × n_head_kv × head_dim × 2 × bytes_per_elem`; f16 = 2 B,
+q8_0 = 1.0625 B, q4_0 = 0.5625 B); DeltaNet state is a fixed per-slot
+buffer (~0.16 GiB/sequence at 48 layers/128 d_state/48 v-heads f32); the
+MTP draft context costs ≈ 4 KiB/token. `--parallel N` DIVIDES `--ctx-size`
+between slots — it does not multiply.
+
 ## Verification (2026-09-26 deploy)
 
 - `/v1/models` reports `frank-primary`.

serverAppConfigs-docs/LLM/frank-primary-lucebox-ds4.md

@@ -146,3 +146,70 @@ systemctl --user start frank-medium`) and stop/mask frank-primary. The
 _Filled in after the cutover: health, real completion, tool call, Open WebUI
 completion, measured prefill/decode, `systemctl --user is-enabled/is-active`
 state._
+
+## Dual-R9700 follow-up (2026-09-28 probe era)
+
+After this single-card runbook was retired, the DS41 engine got a dual-R9700
+attempt on the `feat/ds41-engine` branch (`main` rejects the checkpoint with
+`unsupported model architecture 'deepseek41'`). Findings that outlive the
+probe:
+
+**Budget both cards explicitly, or you waste half the box.** The engine
+sizes the streamed-expert cache LAST, after the model, so uncapped it absorbs
+every spare byte on the stream device — symptom: card1 at 33.0/34.2 GB
+looking busy while card0 sits at 14.2/34.2 GB; the busy card is holding a
+cache of things still streaming from NVMe. Pin all four:
+
+| env | card | meaning |
+|---|---|---|
+| `LUCE_EXPERT_BUDGET_MB` | hip:0 | primary hot experts (~11.96 MiB each) |
+| `LUCE_EXPERT_SECONDARY_BUDGET_MB` | hip:1 | secondary experts (needs `LUCE_DS4_MOE_TP*`) |
+| `LUCE_EXPERT_STREAM_CACHE_MB` | hip:1 | cap the auto-sized stream cache |
+| `LUCE_EXPERT_STREAM_DEVICE=1` | hip:1 | which card holds the slots and computes them |
+
+Working full-card shape: 21000 + 20000 + 8000 → card0 ~28.6/31.86 GiB,
+card1 ~27.3/31.86 GiB, resident experts 3,360 of 15,360.
+
+**Expert TP DOES work on gfx1201+gfx1201.** A retired quadlet recorded
+`LUCE_DS4_MOE_TP_INPROC` as "qualified for gfx1151+gfx1201, not
+gfx1201+gfx1201" — that is WRONG; peer access negotiates and the secondary
+tier comes up clean: `[deepseek4-moe-tp] enabled mode=in-process local=hip:0
+secondary=hip:1`. It is the only lever that materially raises resident
+expert count, and resident count is the entire bottleneck. Confirm every run
+with `expert ownership:` and `hybrid experts ready:` lines.
+
+**The placement file must live under the mounted models dir.** JSON
+`{"owners": [...]}`, 15,360 entries indexed `layer*384+expert`, values
+`primary`/`secondary`/`stream`. It is a ranking, not a promise: each device
+keeps what fits its budget and demotes the rest. Anywhere outside
+`/models/...` gives `cannot open expert placement` and a SILENT single-tier
+fallback — a silent-wrongness failure, not a crash.
+
+**V4.1 refuses layer split and layer-major prefill** — only `--ds4-prefill
+exact` (token-by-token) is legal on a 32 GiB card and `--ds4-fused-decode`
+is refused there. With ~22% of experts resident each token pulls ~187 cold
+experts × ~12 MiB; if `/proc/<pid>/io` `read_bytes` climbs into terabytes
+the model is larger than the box's RAM and you are in page-cache thrash at
+the NVMe ceiling — a capacity verdict, not a tuning problem. Low GPU util
+with a pegged link is the EXPECTED signature of PCIe-expert-stream-bound
+decode, not a broken run.
+
+**Harness traps, each of which produced a wrong number:** never `env -i` the
+podman call (rootless podman runs as `dank`; `env -i HOME=/root` makes
+PODMAN try `/root/.config` and die `stat /root/.config: permission denied`
+before the container exists — pass container env with `-e`); never put a
+large prompt on curl's argv (`Argument list too long` at ~128 KB — use
+`--data-binary @file`); size prompts by growing the file until `wc -c`
+reaches the byte target AND assert depth against the engine's own
+`usage.prompt_tokens` (nominal 4 chars/token was really 5.73; a chunk loop
+that appends a fixed ~152-byte line per nominal 200-byte chunk undershot
+30%); the server runs as the container's PID 1, so tear down with
+`podman rm -f`, never host `pkill`; drop page cache between arms
+(`sync; echo 3 > /proc/sys/vm/drop_caches`) or the first arm reads warm and
+every later one is a lie.
+
+Read the placement log — it is the whole diagnosis
+(`arch=deepseek41 layers=40 embd=5120 experts=384 used=6 ff=2304
+hash_layers=0` — hash_layers=0 is correct for V4.1; `engram: layers=[1,14]
+… 2 table(s) embedded (not mapped)` — the 189.13 GiB Engram tables are never
+mapped, so they do not evict the expert page cache).

serverAppConfigs-docs/LLM/frank-primary-r9v-radiance-qwen38-q8.md

@@ -190,6 +190,70 @@ repo env sets `R9V_ENABLE_PREFIX_CACHING=1` (commit `41e9189`), verified
 on :7770 (`--enable-prefix-caching` in the container argv, health 200,
 live completion answering).
 
+## Authoring a new profile (the qualification pipeline)
+
+Order matters at every step:
+
+1. **Package manifest** `packages/models/<pkg>/package.json` — target shards
+   + aux artifacts with sha256. Verify every target's byte count AND sha256
+   after fetch (`curl -fL -C -` per shard; never trust a fetch without `-f`).
+2. **Placement bootstrap** from GGUF headers: parse all shards' tensor
+   tables (a KV-only shard1 is normal for split parts; each part carries its
+   own tensor table). Per-expert full per layer =
+   (`ffn_gate_exps` + `ffn_up_exps` + `ffn_down_exps`)/512; rank slices
+   rank0 = full×416//640, rank1 = full−rank0. Hot counts =
+   floor(measured_budget / per-expert total). **Anchor budgets to the
+   MEASURED hot_bytes of the qualified sibling profile, never invented.**
+3. **Trim + validate**: `tools/trim_experts.py --hot-counts N,M`, then
+   `expert_budget.validate_manifest(m)` +
+   `expert_budget.expert_memory(m, slots, {cache_rank}, runtime=<runtime.json>)`
+   (runtime/catalog are keyword-only). Declared hot_bytes must match
+   computed or it raises.
+4. **PLE derive** (containerized `tools/prepare_ple.py` with
+   `--expected-bytes/--expected-type/--expected-shape`); setup skips
+   re-derivation when the file exists with matching size.
+5. **profile.env**: set values BEFORE `source dual-r9700/profile.env`
+   (set-first-wins; an empty string does NOT survive `:=`).
+
+The canonical bootstrap.json wrapper schema is exactly
+`{schema: "r9v.placement.v1", id, model_package, hardware, artifact_path,
+artifact_sha256, scope}` with `hardware: "dual-r9700-32gb-asymmetric"`; the
+wrapper `id` MUST equal the id in the profile.json placement descriptor
+(setup rejects mismatches: "does not match profile placement"). Never
+fabricate qualification metadata — use
+`scope: "Unranked routing bootstrap only; does not certify fit or headroom"`.
+
+GGUF parsing laws for the bootstrap: expert tensor names are
+`blk.0.ffn_up_exps.weight` — split on "." gives FOUR parts, layer number is
+parts[1] (a 3-part assumption crashes `int("")`); outer-KV value-type sizes
+`{0:1,1:1,2:2,3:2,4:4,5:4,6:4,7:1,8:str,9:arr,10:8,11:8,12:8}`; tensor byte
+extents from consecutive OFFSETS, never a type-size table (mis-buckets tails
+silently); the PLE Q8_0 tensor on this arch is shape (160, 320001536),
+54,400,261,120 B, an entire shard.
+
+Setup/start: `./r9v setup <alias> --state-dir <dir> --model-dir <dir> --image
+sha256:<digest> --local-image --reuse-from <sibling-model-dir>
+--accept-model-license` (aux artifacts reuse byte-identical files; F16 vs
+Q8 mmproj will not reuse). Start requires the SAME `--state-dir`; fixed
+placement via `R9V_EXPERT_MANIFEST_PATH` skips headroom/catalog/seed raises.
+The host-only doctor FAILs when another instance holds the cards (it wants
+3 GiB free per rank; production at ~30 GiB/card is a hard wall) — a second
+instance CANNOT coexist, so qualification requires a stop+mask window for
+the incumbent, then unmask+restart.
+
+**Qualification gate (unqualified → qualified):** first start runs the
+workload battery (PONG / tool_calls / vision / long-ctx), the PLE pinning
+proof (`VmLck` ≈ PLE bytes in `/proc/<container-pid>/status` when
+RESIDENCY_MODE=pinned), and a VRAM headroom snapshot (floor 3.3 GiB). If
+headroom is violated, lower hot counts via re-trim and re-derive
+MAX_EFFECTIVE. Only then record state `ready: true` and author the runbook.
+
+Remote-execution gotchas from the same windows: `nohup X &` inside an ssh
+command races ssh teardown — the script can die before writing a byte; run
+long remote scripts synchronously, or setsid + disown, or via a system
+unit. zsh login shells abort on unquoted `===` separators; grep globs need
+quoting through ssh+zsh layers.
+
 ## Ops
 
 - **Restart**: `sudo systemctl restart frank-primary` (root unit; the serve

serverAppConfigs-docs/LLM/frank-small-llama-mimo-q5km.md

@@ -1,7 +1,7 @@
 ---
 type: reference
-title: "frank-small on Dankclaw — llama.cpp, MiMo-V2.6-Distill-Qwen-9B Q5_K_M GGUF"
-description: "The active frank-small runbook — a CUDA (sm_86) build of llama.cpp master serving bartowski's MiMo-V2.6-Distill-Qwen-9B Q5_K_M GGUF on the RTX 3090: three conversations of 180,224 tokens, mmproj vision, the asymmetric KV contract (f16 K + q8_0 V), the two VRAM measurements that set the pool size, and what to check when it misbehaves."
+title: "frank-small on Dankclaw — llama.cpp, MiMo-V2.6-Distill-Qwen-9B Q5_K_M GGUF (retired)"
+description: "The retired 2026-10-02/03 frank-small runbook — the Q5_K_M rung at two wide unquantized-KV slots on the RTX 3090, superseded 2026-10-03 by the Q6_K file. Kept as the rollback reference: it carries the measured boot-pair table that prices every pool shape on this card, the slot-window contract, and the misbehaves tree."
 tags:
   - llm
   - frank-small
@@ -11,23 +11,26 @@ tags:
   - mimo
   - gguf
   - kv-cache
-timestamp: 2026-10-02
+timestamp: 2026-10-03
 ---
 
-# frank-small — llama.cpp MiMo-V2.6 Q5_K_M (RTX 3090)
+# frank-small — llama.cpp MiMo-V2.6 Q5_K_M (RTX 3090, retired)
 
-> **ACTIVE since 2026-10-02.** One quant step down from
-> [[frank-small-llama-mimo-q6k]] on the same runtime and the same KV
-> contract, then **resized twice the same day** on Frank's orders: first to
-> three contexts with vision, then down to the pool that keeps 2–3 GiB of the
-> card free. Frank's arithmetic request — "as much as can fit while keeping
-> overhead open space to 2-3gb on the 3090" — is answered by the two
-> measurements in *Shape and the KV contract* below; the answer is
-> **3 × 180,224**, not 3 × 262,144.
+> **RETIRED 2026-10-03** — superseded by
+> [[frank-small-llama-mimo-q6k-2x200704]]: the ~3.5 GiB this shape left free
+> bought the **Q6_K quant rung** (same runtime, same 2 × 200,704 pool, same
+> f16 KV), which measures 2,849 MiB free and 94.44 tok/s — 10 % slower
+> decode is the measured price of the higher rung on this bandwidth-bound
+> card. This file remains the rollback reference and the owner of the
+> measured boot-pair table.
 
-> **The tradeoff is the per-slot window.** Three sessions cost 81,920 tokens
-> of per-conversation memory versus the two-slot shape: 180,224 tokens each
-> instead of the model's native 262,144. See *What three slots cost*.
+> Its own history: one quant step down from [[frank-small-llama-mimo-q6k]]
+> (2026-10-01), resized twice on 2026-10-02 (three contexts with vision, then
+> the 2–3 GiB-band pool), and on 2026-10-03 morning reshaped to **two
+> conversations of 200,704 tokens with the KV VALUES unquantized** —
+> *"2 streams with 200700 tokens context or more… if we have extra space on
+> the 3090, unquantize the kv."* That shape measured 20,530 used / 4,046
+> apparent free (the true `nvidia-smi` free was 3,571 — see *Shape*).
 
 ## Identity
 
@@ -41,77 +44,72 @@ timestamp: 2026-10-02
 
 ## Shape and the KV contract
 
-`--ctx-size 540672 --parallel 3` — llama.cpp divides the pool across slots, so
-each conversation gets `n_ctx_slot = 180224` (boot log: `n_slots = 3,
-n_ctx_slot = 180224, kv_unified = 'false'`). `CTX_SIZE` MUST stay divisible by
-`PARALLEL`; the bats seam enforces it.
+`--ctx-size 401408 --parallel 2` — llama.cpp divides the pool across slots,
+so each conversation gets `n_ctx_slot = 200704` (boot log 2026-10-03:
+`n_slots = 2, n_ctx_slot = 200704, kv_unified = 'false'`). `CTX_SIZE` MUST
+stay divisible by `PARALLEL`; the bats seam enforces it.
 
-The KV contract is Frank's: **K unquantized (f16), V `q8_0`** —
-`--cache-type-k f16 --cache-type-v q8_0`. Two hard dependencies ride on it:
+The KV contract is Frank's, as of 2026-10-03: **both K and V unquantized
+(f16)** — `--cache-type-k f16 --cache-type-v f16`. The asymmetric contract it
+replaced (f16 K + `q8_0` V) had existed to make three slots fit; with two
+slots the exact-value budget opens, and 4 KiB/token × 8 layers × 2 of extra
+V traffic is the measured ~2 % decode tax. The serve script still passes both
+flags explicitly — the asymmetric-era law that a quantized V requires
+`--flash-attn on` stays pinned by the seam.
 
-- **`--flash-attn on` is mandatory**: quantized V exists only on the FA path.
-  Without the flag the engine falls back silently and the contract becomes a
-  lie. The serve script pins the trio; the bats seam pins the script.
-- The VRAM budget: attention KV lives in the 8 full-attention layers only
-  (4 KV heads × 256 dim). K f16 ≈ 16 KiB/token, V q8_0 ≈ 8.5 KiB/token
-  (34 B per 32-element block) ⇒ ~24.5 KiB/token over the pool.
+**Pool sizes on this stack come from boots, not formulas.** The compute
+buffers, CUDA graphs and vision tower all bill separately, so every shape
+change is priced from measured rows:
 
-**The pool size came from two boots, not from a formula.** `free`-style
-arithmetic on this stack is unreliable because the compute buffers, CUDA
-graphs and now the vision tower all bill separately, so the fit was measured
-at two shapes and solved between them:
-
-|Shape|Projector|`nvidia-smi` used|KV at 24.5 KiB/token|Non-KV (weights + buffers + projector)|
+|Shape|KV dtype (K+V)|Projector|`nvidia-smi` used|Non-KV (weights + buffers + projector)|
 |---|---|---|---|---|
-|2 × 262,144 (pool 524,288)|none|20,190 MiB|12,800 MiB|**7,390 MiB**|
-|3 × 196,608 (pool 589,824)|f16|22,972 MiB|14,113 MiB|**8,859 MiB**|
+|2 × 262,144 (pool 524,288)|f16 + q8_0|none|20,190 MiB|**7,390 MiB**|
+|3 × 196,608 (pool 589,824)|f16 + q8_0|f16|22,972 MiB|**8,859 MiB**|
+|3 × 180,224 (pool 540,672)|f16 + q8_0|f16|21,684 MiB|~8,419 MiB|
+|**2 × 200,704 (pool 401,408)**|**f16 + f16**|f16|**20,530 MiB**|**~7,986 MiB**|
 
-The two rows price the projector and its image buffers at **~1.4 GiB** — not
-the ~0.9 GiB the 918 MB file suggests, because the vision encoder's own
-activation and compute buffers ride along. Solving the second row's non-KV cost
-against Frank's 2–3 GiB band: the card is 24,576 MiB, so
-`KV ≤ 24,576 − 9,216 − 2,048 = 13,312 MiB` ⇒ pool ≤ 556,630 tokens, rounded
-down to a multiple of 3 × 4,096 ⇒ **540,672**, i.e. 13,265 MiB of KV and
-**2,441 MiB free** (measured, 21,684 MiB used).
+The 32 KiB/token full-attention KV price (8 layers × 4 KV heads × 256 dim ×
+2 B × 2 halves) prices the deployed pool at 401,408 × 32 KiB = **12,544 MiB
+of KV**; the card is 24,576 MiB, so the shape leaves **4,046 MiB free**
+(measured steady-state — the KV pool is allocated whole at boot, so this is
+the ceiling). The headroom above the 2–3 GiB floor is deliberate slack, not
+an invitation: another 32,768 tokens of pool (1 GiB) would trade the last
+safety margin for window the harness compacts before reaching anyway.
 
-**Why 3 × 262,144 was never possible.** Three native windows are 786,432
-tokens = 18.4 GiB of KV *alone*; plus the 8.86 GiB of non-KV cost that is
-27.3 GiB on a 24 GiB card. The quant step bought 0.86 GiB; the geometry needed
-6 GiB.
+**Why 2 × 262,144 with f16 V does not fit either.** The native window pair is
+524,288 tokens = 16,384 MiB of KV plus ~7,986 non-KV = 24.4 GiB: under the
+floor by a hair. 200,704 is the largest 4,096-aligned slice that keeps the
+measured band.
 
-## Measured on this shape (2026-10-02, 3 × 180,224 + vision)
+## Measured on this shape (2026-10-03, 2 × 200,704 + vision, f16 KV)
 
 |Check|Result|
 |---|---|
-|Boot to healthy|~20 s, `n_slots = 3`, `n_ctx_slot = 180224`, `loaded multimodal model` logged|
-|VRAM|**21,684 MiB used / 2,441 MiB free** of 24,576 (`nvidia-smi`, steady state; the KV pool is allocated whole at boot, so this is the ceiling)|
-|Decode, temp 0, no thinking|**106.8 tok/s** (256-token completion at ~1,300-token depth)|
-|Prefill|**3,814 tok/s** (same request)|
-|Three concurrent requests|all three answered in 1.7 s wall — real parallel slots|
-|Fourth concurrent request|answered in 3.0 s while three ran — queued behind a released slot, as the slot ceiling requires|
-|Vision|`image_url` part with a 64×64 solid-red PNG → **"Red."**|
-|Tools|native `tool_calls` with correct arguments (`get_weather {"city":"Seattle","unit":"f"}`)|
-|Thinking|`reasoning_content` populated (145 chars on a "think first" prompt); `reasoning_effort: "low"` accepted|
-|Over-slot prompt|`400 exceed_context_size_error: request (184001 tokens) exceeds the available context size (180224 tokens)` — refused, never silently truncated|
+|Boot to healthy|~5 s, `n_slots = 2`, `n_ctx_slot = 200704`, `loaded multimodal model` logged|
+|VRAM|**20,530 MiB used / 4,046 MiB free** of 24,576 (`nvidia-smi`, steady state)|
+|Decode, temp 0, no thinking|**104.84 tok/s** (128-token completion at ~500-token depth; the 3 × 180,224 q8_0-V shape measured 106.8 — ~2 % is the honest price of exact V)|
+|Prefill|1,650 t/s on the same shallow 506-token probe (rates are depth- and prompt-dependent; not comparable across shapes)|
+|Batch geometry|`--batch-size 2048 --ubatch-size 1024` — ubatch 2048 failed the scratch boot; 1024 boots clean|
+|Vision|projector loads at boot (`loaded multimodal model`); red-PNG probe passed on 2026-10-02. Caveat discovered on 2026-10-03: probes must use a >= 256 px image — a 10x10 px PNG undersupplies image tokens and the model hallucinates ("it's a video")|
+|Tools / thinking|unchanged by the reslice: native `tool_calls`, thinking in `reasoning_content`|
 
-Decode is ~5 % **faster** than the 2 × 262,144 shape measured earlier the same
-day (100.4–101.8 tok/s): a shorter per-slot window means a shorter attention
-scan at equal prompt depth, and one less slot's DeltaNet state resident.
+## What the slot geometry costs
 
-## What three slots cost
+The per-slot window is the whole price, in both directions:
 
-The per-slot window is the whole price. Two consequences the fleet depends on:
-
-- **A prompt over 180,224 tokens is a hard 400**, so `~/.omp/agent/models.yml`
-  sets `contextWindow: 180224` for this endpoint — the harness compacts before
+- **A prompt over 200,704 tokens is a hard 400**, so `~/.omp/agent/models.yml`
+  sets `contextWindow: 200704` for this endpoint — the harness compacts before
   it reaches the ceiling instead of discovering it.
-- **frank-primary's compaction prompt no longer fits with room to answer.**
-  Its trigger is 183,501 tokens — already above this slot — and its summary
-  budget is 32,768 output tokens. On the two-slot shape (262,144) that left
-  32.8k of output room, which is why that number was chosen; on this shape a
-  single compaction request must be truncated or split. Nothing in the fleet
-  routes compaction here automatically, but any runbook that claims frank-small
-  "fits a compaction request in ONE request" is describing the 262,144 shape.
+- **frank-large's tallest pre-compaction prompt now fits here.** Compaction
+  for frank-large sessions runs on this endpoint (`compactionModel` in
+  `models.yml`); frank-large compacts at 70 % of 262,144 = 183,501 tokens,
+  which did **not** fit the old 180,224 slot and fits 200,704 with 17,163
+  tokens to spare. The 2026-10-02 window trim had restored the fit for
+  frank-primary's smaller 140,492-token trigger; the 2026-10-03 reslice widens
+  the fit to frank-large itself. Nothing routes compaction automatically —
+  the point is the ceiling no longer forbids it.
+- Two simultaneous generations are the slot ceiling; a third request queues
+  behind a released slot (the mechanism measured on the 3-slot shape).
 
 ## Client contract
 
@@ -147,8 +145,9 @@ traffic first. The bats seam fails if a draft flag ships unannounced.
 
 |Target|How|
 |---|---|
-|Q6_K, same runtime|`scripts/frank-small.env` `MODEL_FILE=/home/dank/models/MiMo-Q6K/MiMo-V2.6-Distill-Qwen-9B-Q6_K.gguf` (7,793,710,624 B, still on disk) and the deployer's `LLM_SMALL_MODEL_FILE` default. Q6_K needs ~1.4 GiB more than Q5_K_M, so pair the rollback with `CTX_SIZE` 49,152 tokens lower or `MMPROJ_FILE=`|
-|Two 262,144 windows|`CTX_SIZE=524288 PARALLEL=2` (measured 20,190 MiB used, 4,386 free) and `contextWindow: 262144` in `models.yml` — choose it when a single conversation needs the native window or when frank-primary's compaction request has to land here|
+|Previous shape: 3 × 180,224 + q8_0 V (2026-10-02)|`scripts/frank-small.env` `CTX_SIZE=540672 PARALLEL=3 CACHE_TYPE_V=q8_0` (measured 21,684 MiB used, 2,441 free) and `contextWindow: 180224` in `models.yml` — choose it when three simultaneous conversations matter more than per-slot window or exact V|
+|Q6_K, same runtime|`scripts/frank-small.env` `MODEL_FILE=/home/dank/models/MiMo-Q6K/MiMo-V2.6-Distill-Qwen-9B-Q6_K.gguf` + `MMPROJ_FILE` in that dir — **this is the deployed rung since 2026-10-03** ([[frank-small-llama-mimo-q6k-2x200704]]): at 2 × 200,704 it fits with 2,849 MiB free, no pool cut needed (the older "+1.4 GiB, drop 49,152 tokens" estimate held only for the 3-slot shapes)|
+|Two 262,144 windows|`CTX_SIZE=524288 PARALLEL=2 CACHE_TYPE_V=q8_0` (measured 20,190 MiB used, 4,386 free **with q8_0 V** — with f16 V the same shape is 16,384 MiB of KV + ~7,986 non-KV and does not fit) and `contextWindow: 262144` in `models.yml` — choose it when a single conversation needs the native window|
 |Vision off|`MMPROJ_FILE=` in the env file; the deployer gate then has to be satisfied too, so point `LLM_SMALL_MMPROJ` at any real projector file or drop the gate in the same commit|
 |SGLang + INT8|[[frank-small-sglang-mimo-v26]] — venv and old unit/env/serve trio still in place; the INT8 payload was staged to `/home/dank/.AGENTTRASH/20261002-retired-llm-payloads/` on 2026-10-02, so restore it there or re-fetch with `old/download-frank-small-sglang-int8.sh`|
 
@@ -158,14 +157,17 @@ traffic first. The bats seam fails if a draft flag ships unannounced.
   naming the missing file if the binary or GGUF moved.
 - `No CUDA GPUs are available`: the 3090 vanished from the bus or the UVM node
   is missing — `nvidia-modprobe -u -c=0` (the serve script runs it).
-- Requests queue behind each other: three concurrent long generations are the
-  slot ceiling by contract; a fourth waits for a slot to free (measured: 1.7 s
-  for the three, 3.0 s for the fourth).
-- `400 … exceeds the available context size (180224 tokens)`: the conversation
-  outgrew ONE slot. This is not pool exhaustion — the pool is 540,672 tokens
-  and the per-slot window is the binding limit. Shorten the prompt, compact, or
-  roll back to the 262,144 shape above.
-- Vision answers a color it never saw: the client sent `images` instead of an
-  `image_url` content part (see *Client contract*).
+- Requests queue behind each other: two concurrent long generations are the
+  slot ceiling by contract (2026-10-03 shape); a third waits for a slot to
+  free (the mechanism measured on the 3-slot shape: 1.7 s for three, 3.0 s for
+  the fourth).
+- `400 … exceeds the available context size (200704 tokens)`: the conversation
+  outgrew ONE slot. This is not pool exhaustion — the pool is 401,408 tokens
+  and the per-slot window is the binding limit. Shorten the prompt, compact,
+  or roll back to the 262,144 shape above.
+- Vision answers a color it never saw, or claims the image "is a video":
+  either the client sent `images` instead of an `image_url` content part, or
+  the probe image is too small to supply real image tokens (use >= 256 px;
+  see *Client contract*).
 - Slow decode after a CTX or quant change: check `nvidia-smi` — pushing KV out
   of VRAM into unified memory halves the rate silently.

serverAppConfigs-docs/LLM/frank-small-llama-mimo-q6k-2x200704.md

@@ -0,0 +1,158 @@
+---
+type: reference
+title: "frank-small on Dankclaw — llama.cpp, MiMo-V2.6 Q6_K, 2 x 200704, f16 KV"
+description: "The active frank-small runbook — llama.cpp master (CUDA sm_86) serving bartowski's MiMo-V2.6-Distill-Qwen-9B Q6_K GGUF on the RTX 3090: two conversations of 200,704 tokens with fully unquantized f16 KV, vision through the bundled projector, and the boot-pair arithmetic that prices every pool change."
+tags:
+  - llm
+  - frank-small
+  - dankclaw
+  - llama-cpp
+  - gguf
+  - rtx3090
+timestamp: 2026-10-03
+---
+
+# frank-small — llama.cpp MiMo Q6_K, 2 × 200,704, f16 KV
+
+> **ACTIVE since 2026-10-03.** The third shape Frank bought in two days on
+> this card: the freed third slot paid for **exact KV values**
+> ([[frank-small-llama-mimo-q5km]] documents that step and remains the
+> rollback), and the ~3.5 GiB still free then paid for the **Q6_K quant rung**
+> — the same file this endpoint served on 2026-10-01, come back as an
+> upgrade rather than a compromise. The pool held at 401,408 (2 × 200,704);
+> only the weights moved up.
+
+## Identity
+
+| | |
+|---|---|
+| Host / device | Dankclaw (192.168.0.143), RTX 3090, 24,576 MiB |
+| Endpoint | `http://192.168.0.143:7771` — alias `frank-small`, `/v1` OpenAI surface |
+| Runtime | llama.cpp master, CUDA sm_86 build `…/llama-master/build-cuda/bin/llama-server` (`GGML_CUDA_FA_ALL_QUANTS=ON`) |
+| Unit | root systemd `frank-small.service` ([`scripts/frank-small.service`](../../scripts/frank-small.service)), `User=dank`, `Restart=on-failure`, `LimitMEMLOCK=infinity` |
+| Serve script | [`scripts/serve-frank-small.sh`](../../scripts/serve-frank-small.sh), tunables [`scripts/frank-small.env`](../../scripts/frank-small.env) |
+| Weights | `/home/dank/models/MiMo-Q6K/MiMo-V2.6-Distill-Qwen-9B-Q6_K.gguf` — bartowski/MiMo-V2.6-Distill-Qwen-9B-GGUF **Q6_K**, 7,793,710,624 B, sha256 `ef96d05a…` (qwen3_5 hybrid: 32 layers, 8 full-attention + 24 Gated DeltaNet, native 262,144, no MTP tensors) |
+| Projector | `…/MiMo-Q6K/mmproj-MiMo-V2.6-Distill-Qwen-9B-f16.gguf` — 918,166,048 B, sha256 `ff348f…` (hardlink of the Q5KM dir's copy, link-count-2 verified; the dir is self-contained) |
+| Fetch | [`scripts/download-frank-small.sh`](../../scripts/download-frank-small.sh) — `-f`, byte count + LFS sha256 verified, `--verify-only`; both files verified against the pinned sha on the host before this deploy |
+| Seam | `tests/frank_small_unit.bats` (argv assertions against a stub engine: payload path, slot geometry, KV pair, batch pair, projector opt-out) |
+
+## Shape and KV contract
+
+`--ctx-size 401408 --parallel 2` — llama.cpp divides the pool across slots,
+so each conversation gets `n_ctx_slot = 200704`. `CTX_SIZE` MUST stay
+divisible by `PARALLEL`; the seam enforces it. Both K and V are unquantized
+(f16) — the flags stay explicit because `--flash-attn` must not be lost if V
+ever goes back to `q8_0` (the serve script pins it, the seam pins the pin).
+
+**The Q6_K rung is a weights-side spend only.** File delta +917,585,920 B
+(+875 MiB) against Q5_K_M, same KV, same slot geometry, same batch pair.
+Priced from the measured Q5 boot at this shape (non-KV 7,986 MiB): the boot
+landed **21,276 MiB used / 2,849 MiB true free** — non-KV measured 8,732 MiB,
+inside the 2–3 GiB floor, so the pool stayed at 401,408 and the
+12,288-token step-down ladder was never climbed. Quote `nvidia-smi`'s `free`
+field, never `total − used`: the driver holds ~450 MiB of reserve the
+subtraction hides (that error once overstated this card's slack by a
+half-GiB).
+
+**Quant ceiling, evaluated and closed 2026-10-03 (Frank: stay here).**
+`Q8_0` is the only rung above this one on the Hub (9,545,979,424 B = 9,104
+MiB, +1,671 over Q6_K): at the live pool it would leave 1,178 MiB free —
+under the floor — and the calibrated bandwidth law (it predicted this file's
+94.4 t/s from Q5_K_M's 104.84 within 1 %) puts decode at ~72 t/s, a further
+−24 % for the near-lossless rung. Fitting it was priced two ways (trim the
+pool to 2 × 182,272 — which also breaks frank-large's 183,501-token
+compaction-prompt fit by 1,236 tokens — or revert V to `q8_0`), and both
+were declined. Re-open only on evidence the Q6_K residual matters.
+
+**Pool prices on this card come from boots, never formulas** — the measured
+boot-pair table lives in [[frank-small-llama-mimo-q5km]] (*Shape and KV
+contract*) and still prices every rollback shape.
+
+## Compute-buffer geometry (2026-10-05)
+
+`BATCH_SIZE=4096 UBATCH_SIZE=2048` — both doubled from 2048/1024, and the trade
+measured rather than assumed, because the predecessor shape recorded **"ubatch
+2048 failed the scratch boot"** and this card has less free space than it did.
+
+**That failure was a different shape, and it did not reproduce here.** The
+service itself is the bench (one card holds one server, so no scratch port):
+the old rung was re-booted through a transient `FRANK_SMALL_ENV_FILE` drop-in
+and reproduced the 2026-10-03 boot **byte-identically** — 21,276 MiB used /
+2,849 MiB free — which makes the ubatch the only variable in the comparison.
+
+|Probe|ubatch 1024|ubatch 2048|Δ|
+|---|---|---|---|
+|Prefill, 8,928 tokens|3,972 / 4,077 t/s|**4,207 / 4,212 t/s**|**+4.6 %**|
+|Decode, 128 tokens at that depth|91.87 / 93.01 t/s|93.81 / 94.27 t/s|noise|
+|Prefill, 279 tokens|2,675.7 t/s|2,658–2,811 t/s|noise|
+|VRAM used / free|21,276 / 2,849 MiB|**21,846 / 2,279 MiB**|**−616 MiB**|
+
+Three readings:
+
+- **The gain only exists above the old ubatch.** A 279-token prompt is shorter
+  than either micro-batch, so it cannot see the knob — and it did not move.
+  That is the control proving the 8.9k delta is geometry, not drift.
+- **Decode is flat**, as every prior sweep on this card found. Ubatch is a
+  prefill knob.
+- **The price is 616 MiB out of a 2.8 GiB margin.** 2,279 MiB free is still
+  inside the 2–3 GiB band, at its bottom edge. `4096/1536` is the middle rung,
+  and the pool's 12,288-token (384 MiB) ladder in
+  [[frank-small-llama-mimo-q5km]] is the other way to buy room back.
+
+**Why 2048 is right HERE while 512 was right on the R9700 sibling: the curve is
+backend-specific.** The 2× R9700 layer split peaks at ubatch 512 and is 10 %
+off that peak by 2048 ([[frank-medium-batch-ubatch]]); on this CUDA card the
+curve climbs monotonically to 2048 (710 → 1,188 t/s on the 35B MoE sweep,
+1,944 → 2,423 t/s on the Ornith 8k sweep, both with decode flat). Never carry a
+ubatch verdict across backends.
+
+The logical `--batch-size` is VRAM- and throughput-neutral — two sweeps and this
+A/B agree — so 4096 costs nothing and buys nothing by itself; it is here because
+that is the pair the curve was measured at.
+
+## Measured on this shape (2026-10-03, Q6_K, 2 × 200,704, f16 KV)
+
+|Check|Result|
+|---|---|
+|Boot to healthy|model + projector resident in ~2.3 s (load log `0.02.27`), `n_slots = 2`, `n_ctx_slot = 200704`, `loaded multimodal model`, health 200; deploy e2e gate 33/33|
+|VRAM|**21,276 MiB used / 2,849 MiB free** of 24,576 (`nvidia-smi` `free`, steady state) — inside the 2–3 GiB floor with exact V AND the higher rung. Superseded 2026-10-05 by the batch doubling: **21,846 / 2,279** — see *Compute-buffer geometry*|
+|Decode, temp 0, no thinking|**94.44 tok/s** on the identical 506-token probe that measured 104.84 on Q5_K_M — **−10 %**. Decode on this card is weight-bandwidth-bound, and Q6_K streams ~10 % more bytes per token; the quant rung buys quality with decode rate, measurably|
+|Prefill|2,229.9 t/s (same shallow probe; rates are depth-dependent, not comparable across probes)|
+|Vision|512×512 solid-red PNG → **`Red`** (277 prompt tokens — the image really reached the tower). See the probe-size trap in *Client contract*|
+|Tools / thinking|native `tool_calls`, thinking in `reasoning_content`, thinking OFF unless `enable_thinking: true` (MiMo template default; `models.yml` ships it in `extraBody`)|
+
+## Client contract
+
+llama.cpp surface, live-verified on this model family: model id `frank-small`,
+native tools via `--jinja` (`tool_calls` in the response), thinking in
+**`reasoning_content`**, vision via `image_url` content parts. Two vision
+traps, both measured here:
+
+1. A top-level `images` field on `/v1/chat/completions` is silently ignored
+   and the model invents a color.
+2. **The probe image must be big enough to supply real image tokens.** A
+   10×10 px solid-red PNG came back "The image is a video, so I can't see its
+   content" — the engine itself warns at boot that Qwen-VL wants ≥ 1024 image
+   tokens, and a postage-stamp image lands one patch below usable signal.
+   512×512 answers `Red` in 19 tokens. Always probe vision with a ≥ 256 px
+   image; a wrong-color answer from a tiny image is the probe's fault, not
+   the endpoint's.
+
+`~/.omp/agent/models.yml` sets `contextWindow: 200704` — the PER-SLOT window,
+so the harness compacts before llama.cpp answers a longer prompt with a 400.
+
+## Rollback
+
+|Target|How|
+|---|---|
+|Q5_K_M at the same shape|`scripts/frank-small.env` `MODEL_FILE=/home/dank/models/MiMo-Q5KM/MiMo-V2.6-Distill-Qwen-9B-Q5_K_M.gguf` (+ `MMPROJ_FILE` in that dir) and the deployer default — buys back 875 MiB (3,571 free measured) **and the 10 % decode rate** (104.84 tok/s measured). Payload sha-verified on disk|
+|3 × 180,224 + q8_0 V|`CTX_SIZE=540672 PARALLEL=3 CACHE_TYPE_V=q8_0` (measured 21,684 used / 2,441 free **with Q5_K_M** — Q6_K does NOT fit that shape inside the floor; pair them explicitly)|
+|SGLang + INT8|[[frank-small-sglang-mimo-v26]] — payload staged to `/home/dank/.AGENTTRASH/20261002-retired-llm-payloads/`|
+
+## When it misbehaves
+
+Same decision tree as [[frank-small-llama-mimo-q5km]] (*When it misbehaves*):
+`No CUDA GPUs are available` ⇒ `nvidia-modprobe -u -c=0`; a 400 naming
+200,704 ⇒ the conversation outgrew one slot, not the pool; slow decode after
+a shape change ⇒ check `nvidia-smi` for KV pushed into unified memory; queue
+behind two long generations ⇒ the slot ceiling is two, by contract.

serverAppConfigs-docs/LLM/index.md

@@ -1,21 +1,25 @@
 # LLM Fleet — Knowledge Base
 
-Documentation for the LLM inference fleet on the two hosts (2026-10-02):
-**Dankserver runs the tcclaviger RDNA4 vLLM fork on frank-primary** (the
-digest-pinned `tcclaviger/vllm` image driving `podman run` directly, serving
-orcarouter/Qwen3.8-Flash-Next-Uncensored-FP8 across both Radeon AI PRO R9700s
-on :7770, TP=2 with automatic tiered-expert offload, **4 seqs x 262144 ctx
-(the model's full native window)**, fp8 KV (5 boot but leave no runtime
-headroom; 6 do not fit), no speculative decode, MNBT 1024, MoE gather 32x64, util 0.98,
-reasoning/tool parsers wired (qwen3 / qwen3_xml) with preserve_thinking on,
-measured 36.4-39.4 tok/s decode — the 2026-10-01 bf16 no-spec cutover plus
-the night ladder; rollback frank-primary-fp8-mtp-20261001), and Dankclaw
-runs two endpoints — **ROCmFPX llama.cpp (frank-large, port 7770:
-kingjones777/Qwen3.8-27B-ROCmFP4-STRIX-MTP-GGUF Q8_0_ROCMFPX_AGENT, a ftype-115
-file only that fork can read, with the repo's MTP head drafting 4 tokens, ONE
-session at 262,144, f16 KV, vision via the bundled projector)** and
-**llama.cpp CUDA (frank-small, port 7771: MiMo-V2.6-Distill-Qwen-9B Q5_K_M
-GGUF, 2 x 262,144, asymmetric KV f16 K + q8_0 V, on the RTX 3090)**.
+Documentation for the LLM inference fleet on the two hosts (2026-10-05):
+**Dankserver runs FreeToken (PR #587 ROCm) on frank-primary** (the local
+image `frank-primary-freetoken:pr587-77a7201` built on the host from
+`scripts/docker/frank-primary-freetoken.Dockerfile` — FreeToken at PR head
+`77a7201` in the PR's own ROCm 7.14 base image, torch 2.11.0+rocm7.14 —
+driving `podman run` directly, serving orcarouter
+/Qwen3.8-Flash-Next-Uncensored-FP8 on ONE R9700, :7770: all experts in
+pinned host RAM behind a GPU LRU (`--moe-cache-rate 0.10`), KV pool
+`--num-tokens 299968` (multiple-of-64 law of the qsa_sparse backend), 2
+decode slots, parsers AUTO (thinking → `reasoning_content` natively;
+measured 17.1 tok/s decode, TP=2 exists but does not boot on ROCm — see the
+runbook). Rollback tag frank-primary-vllm-shapeG-20261003),
+and Dankclaw
+runs two endpoints — **halogen container (frank-large, port 7770:
+ghcr.io/peonist-ai/halogen @ pinned digest serving peonist-ai
+/halogen-qwen3.8-27b, a 33.4 GiB .hgn checkpoint that loads in no other
+engine, DFlash2 spec decode, ONE slot at 200,700 ctx, text-only — vision
+lives on frank-small)** and
+**llama.cpp CUDA (frank-small, port 7771: MiMo-V2.6-Distill-Qwen-9B Q6_K
+GGUF, 2 x 200,704, fully unquantized f16 KV, on the RTX 3090)**.
 
 > 2026-08-22: the former `vllmScripts` repository is merged into the
 > serverAppConfigs repository. This knowledge base moved from the old repo's
@@ -29,24 +33,25 @@ graph LR
     subgraph Dankserver[AMD Dankserver - Ubuntu 26.04]
         A[2x Radeon AI PRO R9700
 gfx1201, 32 GiB each]
-        B2[tcclaviger RDNA4 vLLM fork, digest-pinned
+        B2[FreeToken PR587 ROCm, local image
 podman run, frank-primary, port 7770]
-        C[Qwen3.8-Flash-Next FP8 bf16 no-spec
-TP=2, expert offload automatic, 4 x 262144 ctx,
-fp8 KV, MNBT 1024, gather 32x64, qwen3 parsers]
+        C[Qwen3.8-Flash-Next FP8
+1x R9700, experts in pinned host RAM + GPU LRU 10%,
+KV pool 299,968 tokens, 2 slots, parsers AUTO,
+TP=2 untested on ROCm; second R9700 idle]
     end
 
     subgraph Dankclaw[AMD Strix Halo - Ubuntu 26.04]
         E2[RTX 3090 (PCIe x4 dock) +
 gfx1151 iGPU 114 GiB GTT]
-        E[ROCmFPX llama.cpp gfx1151, pinned fork build
+        E[halogen container gfx1151, digest-pinned image
 frank-large, port 7770]
-        F[Qwen3.8-27B Q8_0_ROCMFPX_AGENT + MTP-Q4_0 head
-1 x 262144 ctx, f16 KV, draft n-max 4, mmproj vision]
+        F[halogen-qwen3.8-27b .hgn p1w4d-d2 + DFlash2 drafter
+1 x 200700 ctx, d2 KV, text-only]
         G[llama.cpp master CUDA sm_86
 frank-small, port 7771]
         H[MiMo-V2.6 Q5_K_M GGUF
-2 x 262144 ctx, f16 K + q8_0 V, flash-attn]
+2 x 200704 ctx, f16 K + f16 V, flash-attn, mmproj vision]
         E2 --> E --> F
         E2 --> G --> H
     end
@@ -65,24 +70,57 @@ frank-small, port 7771]
 
 ## Active Deployment
 
-- [[frank-large-rocmfpx-q8agent-mtp]] — **ACTIVE since 2026-10-02**
-  (root systemd unit, port 7770): the pinned kingjones30/ROCmFPX fork serving
-  the repo's highest-quality rung `Qwen3.8-27B-Q8_0_ROCMFPX_AGENT.gguf`
-  (26.28 GiB, ftype 115 = 187 plain-Q8_0 + 311 GGML_TYPE_Q8_0_ROCMFPX
-  tensors) with the bundled `mtp-Qwen3.8-27B-Q4_0.gguf` head drafting 4
-  tokens, one 262,144 window, f16 KV, the BF16 projector for vision. Why the
-  runtime is not interchangeable, the pinned-build set_rows history, the
-  measured envelope, rollback to gufo
-- [[frank-small-llama-mimo-q5km]] — **ACTIVE since 2026-10-02** (root
-  systemd unit, port 7771): llama.cpp master (CUDA sm_86) serving
-  bartowski/MiMo-V2.6-Distill-Qwen-9B-GGUF Q5_K_M on the RTX 3090,
-  THREE slots of 180,224 tokens (pool 540,672) with K f16 + V q8_0 and the
-  mmproj for vision, ~100 tok/s decode, ~2.7 GiB of the card left free.
-  The KV arithmetic, the flash-attn dependency, rollback to Q6_K
+- [[frank-large-halogen27b-dflash2]] — **ACTIVE since 2026-10-05**
+  (root systemd unit + podman, port 7770): the digest-pinned
+  `ghcr.io/peonist-ai/halogen` image serving `peonist-ai/halogen-qwen3.8-27b`
+  (`qwen3.8-27b-p1w4d-d2.hgn`, 33.4 GiB) with the DFlash2 drafter on ONE slot
+  of 200,700 ctx (slots>1 disables speculation — engine law). Publisher card
+  31.7 t/s mean across eight text workloads. Accepted tradeoffs: TEXT-ONLY
+  (the engine ignores the vision encoder; frank-small owns images) and the
+  CENSORED base model (the ROCmFPX AGENT file was Uncensored by construction).
+  No server-side sampler — the thinking recipe rides the wire in models.yml.
+  Rollback: revert the swap commit; the ROCmFPX payload + fork build +
+  `build-rocmfpx.sh` pin all remain on disk
+- [[frank-small-llama-mimo-q6k-2x200704]] — **ACTIVE since 2026-10-03**
+  (root systemd unit, port 7771): llama.cpp master (CUDA sm_86) serving
+  bartowski/MiMo-V2.6-Distill-Qwen-9B-GGUF **Q6_K** on the RTX 3090, TWO
+  slots of 200,704 tokens (pool 401,408) with FULLY UNQUANTIZED KV (f16 K +
+  f16 V) and the mmproj for vision — 94.4 tok/s decode (the rung costs a
+  measured -10 % vs Q5_K_M: weight-bandwidth-bound), batch 4096 / ubatch 2048
+  since 2026-10-05 (+4.6 % prefill, measured), 2,279 MiB of the card
+  left free. Probe-size trap (>=256 px images or the model says "video")
+  documented. Rollback: one MODEL_FILE line back to Q5_K_M
+- [[frank-small-llama-mimo-q5km]] — **RETIRED 2026-10-02 → 10-03 (rollback
+  target; payload on disk; owns the measured boot-pair table)**: the same
+  runtime and the same 2 x 200,704 + f16 KV shape one quant rung lower —
+  104.8 tok/s, 3,571 MiB true free
+- [[frank-primary-freetoken-pr587]] — **ACTIVE since 2026-10-05**
+  (root systemd unit, port 7770): FreeToken at PR #587 head `77a7201`
+  (branch rocm714_torch214) built on the host into the local image
+  `frank-primary-freetoken:pr587-77a7201` (Dockerfile: digest-pinned
+  rocm/pytorch rocm7.14 torch-2.11.0 base, libhiprtc dev-symlink law,
+  curated --no-deps install per docs/install_amd.md), serving the SAME
+  orcarouter FP8 payload on ONE R9700 — experts in pinned host RAM
+  (~120 GiB + 47.7 GiB PLE), GPU LRU at 0.10, KV pool 299,968 tokens
+  (multiple-of-64 law), 2 slots, parsers AUTO (thinking →
+  `reasoning_content`). Measured: boot ~2.3 min, decode 17.1 tok/s
+  single-stream, vision "Red" probe passes, native tool_calls; TP=2 does
+  not boot on ROCm (pynccl JIT) — second card idle. Rollback: git tag
+  frank-primary-vllm-shapeG-20261003
+- [[dankserver-qwen38-fp8-expert-offload-tuning]] — **RETIRED 2026-10-05
+  (rollback target; the vLLM trio + env lore live in the tag)**: the
+  tcclaviger RDNA4
+  vLLM fork serving orcarouter/Qwen3.8-Flash-Next-Uncensored-FP8 across both
+  R9700s, TP=2 with automatic expert offload — `--max-model-len 200704` x 3
+  slots, fp8 KV, pool 3,150,000,000 B/rank (measured 398,658 tokens = 1.99x),
+  MNBT 1024, gather 32x64, no spec decode, prefetch off, plus
+  `--expert-precise-memory` (arena 17.22 GiB / 7,522 slots), GPU_MAX_HW_QUEUES=2
+  and a mounted compile cache. Why `--max-num-seqs`
+  must stay 3 (SEQS 2 cannot compile this model), why the arena's levers are
+  `max_model_len` and the overhead term rather than the KV pool, and the measured envelope
 - [[frank-primary-glm53-flash-w4a16]] — RETIRED 2026-09-30 → 10-01 (the
-- failed GLM experiment; frank-primary runs Qwen FP8 again, see the intro):
-  (2026-09-30, root systemd unit, port 7770): the tcclaviger RDNA4 vLLM fork
-  (2026-09-30, root systemd unit, port 7770): the tcclaviger RDNA4 vLLM fork
+  failed GLM experiment; frank-primary runs Qwen FP8 again, see the intro):
+  the tcclaviger RDNA4 vLLM fork
   (digest-pinned `tcclaviger/vllm@sha256:da4029666…`, plain `podman run`,
   no shim) serving Intel/GLM-5.3-Flash-W4A16-AutoRound — AutoRound symmetric
   int4 group 128, ~170 GiB, 34 shards + 2 extras, 113,074 tensors — on BOTH
@@ -178,8 +216,21 @@ frank-small, port 7771]
   (bs1/bs3/bs4/bs6, 8k–700k KV) on Dankserver RTX 3090
 - [[deployment]] — deploy and update process for both servers
 - [[frank-redo-2026-08-17]] — the 2026-08-17 topology change
-- [[frank-large-gufo-qwen38-27b]] — **RETIRED 2026-10-02 (the rollback
-  target)** (root systemd unit, port 7770): gufo
+- [[frank-large-rocmfpx-q6agent-uncensored-mtp]] — **RETIRED 2026-10-03 →
+  10-05 (rollback target; payload, fork build and build script on disk)**:
+  the pinned ROCmFPX fork serving the uncensored protected-heads
+  Q6_0_ROCMFPX_AGENT rung with the repo's MTP-Q4_0 head drafting 4 tokens,
+  one 262,144 window, f16 KV; measured 27.6-28.1 t/s prose decode at 86.8 %
+  acceptance. Replaced by the halogen DFlash2 endpoint; the ngram-mod-onto-MTP
+  rejection (-15 %, recurrent-state rollback) stays a live lesson for this
+  architecture.
+- [[frank-large-rocmfpx-q8agent-mtp]] — **RETIRED 2026-10-02 → 10-03 (the
+  rollback target; payload still on disk)**: the same pinned ROCmFPX fork
+  serving the ALIGNED `Q8_0_ROCMFPX_AGENT` (ftype 115, 17.0 t/s, MTP
+  acceptance 0.462) — the MTP-vs-none +123 % A/B, the trap catalogue
+- [[frank-large-gufo-qwen38-27b]] — **RETIRED 2026-10-02 (history; the
+  payload dirs were removed from disk 2026-10-03, so this rollback is a
+  re-download)** (root systemd unit, port 7770): gufo
   (gufo-org/gufo) v0.1.1, the Strix Halo inference engine, built from
   source on Dankclaw, serving unsloth/Qwen3.8-27B-GGUF **UD-Q4_K_XL**
   with the z-lab DFlash2 Q4_K_M drafter — ONE session at the full native
@@ -229,8 +280,10 @@ frank-small, port 7771]
   .hgn + quality overlay) on the gfx1151 iGPU — two 200k conversations over one
   400,384-position pool, deployed flags, memory model, client contract,
   troubleshooting (2026-09-16 → 09-19)
-- [[frank-small-llama-mimo-q6k]] — **RETIRED 2026-10-01 → 10-02 (rollback
-  target)**: llama.cpp (master, CUDA sm_86 build) serving
+- [[frank-small-llama-mimo-q6k]] — **RETIRED 2026-10-01 → 10-02** (the same
+  rung returned ACTIVE 2026-10-03 at the 2 x 200,704 f16-KV shape —
+  [[frank-small-llama-mimo-q6k-2x200704]]): llama.cpp (master, CUDA sm_86
+  build) serving
   bartowski/MiMo-V2.6-Distill-Qwen-9B-GGUF **Q6_K** on the RTX 3090 — TWO
   native 262,144-token sessions with ASYMMETRIC KV (K f16 unquantized + V
   q8_0, which forces flash-attn), ~20.9/24.6 GiB measured, text-only (the
@@ -354,6 +407,42 @@ frank-small, port 7771]
 - [[Llama Cpp Code Diff]] — code-level quantization diff of the three runtimes + server recommendations
 - [[SGLang]] — the structured-generation serving engine; when it fits Dankserver (2x R9700) vs Dankclaw (Strix Halo)
 
+## Cross-Cutting References (skill distillation, 2026-10-05)
+
+One polished reference page per skill-cluster, distilled from the managed
+skill library into the vault (see [log](../log.md) for the wave plan):
+
+- [[GGUF Compatibility]] — what a GGUF's tensor histogram and header demand
+  of a loader; how the runtime is chosen by the file
+- [[Llama Cpp Serving Sizing]] — slot/pool/KV sizing laws, boot-shape
+  laddering, llama-bench discipline
+- [[Llama Cpp Multi-Device Splits]] — layer vs tensor splits, `-ts`/`-ot`
+  laws, drafter placement
+- [[PCIe Link and Atomics]] — the link walls, atomics routing, and the
+  heterogeneous-pair verdicts
+- [[RDNA4 Container Serving]] — bring-up recipes for RDNA4 containers
+- [[RDNA4 MoE Expert Offload VRAM Budget]] — the plan law, every arena
+  lever, and why free VRAM is not arena
+- [[vLLM Boot Forensics and Config Interrogation]] — reading a boot that
+  died, and interrogating a running fork's config
+- [[vLLM Porting and Compile Triage]] — porting checkpoints across vLLM
+  versions/forks; torch.compile/dynamo breakage triage
+- [[LLM Bench Windows and Measurement]] — window discipline, honest A/B,
+  the window script laws
+- [[LLM Performance Diagnosis]] — slow prefill/decode triage on live
+  endpoints, IBS attribution, prefix-cache routing
+- [[Model Swap and Rollback]] — the swap procedure, payload preflight,
+  research verification, second-endpoint bring-up
+- [[Tcclaviger Fork Ops]] — the fork's sweep harness, measured knob
+  verdicts, large-MoE recipe, compile-cache persistence
+- [[Heterogeneous Split Triage]] — triage for unequal-accelerator split
+  proposals (the no-go verdict format)
+- [[FreeToken Builds]] — the CUDA and ROCm build paths with their
+  dependency traps
+
+These sit alongside the operational pages under
+`../Operations/index.md` and `../Agent/index.md`.
+
 ## Related
 
 - [Project knowledge base](../index.md) — the Podman app stacks on the same

serverAppConfigs-docs/LLM/vLLM Boot Forensics and Config Interrogation.md

@@ -0,0 +1,207 @@
+---
+type: reference
+title: vLLM Boot Forensics and Config Interrogation
+description: Capturing a vLLM container boot failure's real root cause from journald (the container log is already gone), classifying the failure, and interrogating a live endpoint's true config and client contracts.
+tags: [vllm, forensics, journalctl, podman, config, contracts]
+timestamp: 2026-10-05T00:00:00-07:00
+---
+
+# vLLM Boot Forensics and Config Interrogation
+
+For serve-script vLLM endpoints (frank-primary/qwen-fp8 on dankserver, any
+`podman rm -f` + restart service).
+
+## 1. Capture the true cause — the journal, not `podman logs`
+
+The serve script runs `podman rm -f <container>` at every start, so a dead
+container's log is GONE by the time you look; `no container with name or ID`
+means the journal is the only source. The engine's stdout survives in
+journald under the container tag:
+
+```bash
+journalctl --no-pager --since "20 min ago" | grep -a "<container>\[" | tail -40
+journalctl --no-pager --since "8 min ago" | grep -a "<container>\[" \
+  | grep -aiE "error|exception|assert|raise|invalid|failed" | grep -av APIServer | tail -12
+```
+
+Why naive greps fail:
+
+- Default format truncates lines at the display cut, and the fork's giant
+  single-line ERROR dumps are journald-SPLIT — the decisive sentence never
+  appears in short format.
+- `-o cat` strips the `svc[pid]:` prefix, so grepping the tag against `-o
+  cat` returns nothing — use the default format for the tag, `-o json` for
+  the content.
+- Time filters: relative (`--since "NN min ago"`) only — absolute timestamps
+  are local PDT while the engine logs UTC; never mix.
+
+**The JSON decode procedure** (proven 2026-10-01, caught
+`torch.OutOfMemoryError` exactly):
+
+1. Bound the window: last traceback lines + the `container died` podman
+   event.
+2. `journalctl --no-pager --since … --until … -o json | python3 -c …` on the
+   remote: handle `MESSAGE` possibly arriving as a **byte list**
+   (`bytes(int(x) for x in m).decode("utf-8","replace")`), strip ANSI
+   (`re.sub(r"\x1b\[[0-9;]*m", "", m)`), dedupe by first ~12 chars (one
+   logical traceback can arrive as 20+ entries).
+3. Reassemble the worker traceback: filter skeleton lines
+   (`File |…| raise |^+|~+|…<N> lines|…|re-`), keep content lines. The
+   decisive line is usually the last content line.
+4. **The diagnosis is the gap math: `ask − free = shortfall`.**
+
+**The wrapper trap:** `RuntimeError: Engine core initialization failed. See
+root cause above` is the WRAPPER — the real cause is the EngineCore traceback
+above it, ending in `RuntimeError: Worker failed with error '<cause>'`. Grep
+for `Worker failed with error`.
+
+**For probe boots / runner scripts:** capture live BEFORE teardown —
+`podman logs -f <ctr> > /home/dank/logs/<arm>.engine.log 2>&1 &` right after
+launch. A runner's `podman rm` destroys the only copy; two 2026-10-01
+failures had their root cause eaten this way, and an 8-line bootfail capture
+is insufficient.
+
+## 2. Failure classes
+
+| signature | class |
+|---|---|
+| `CUDA out of memory. Tried to allocate X GiB … Y GiB free` | capacity; the gap math names the next rung ([[RDNA4 MoE Expert Offload VRAM Budget]]) |
+| `Worker failed with error 'Constraints violated (L['query_start_loc'].size()[0])! … specialized it to a constant (N)'` | torch.compile/dynamo shape specialization (N = num_seqs+1-ish) — NOT memory |
+| `Deactivated successfully` with no traceback | container exited CLEAN (vLLM API server exits 0 after an EngineCore fatal); container already removed — re-read the journal, check `NRestarts` |
+| `no container with name or ID` | container rm'd; journal only |
+
+Boot-order reality: the plan computes fine at impossible budgets; the
+failure lands later in worker init, not at plan time. A plan line printing a
+huge arena does NOT mean it will boot.
+
+Bypasses this fork exposes for compile faults: `--enforce-eager` (skips
+torch.compile AND CUDA graphs — drops graphs, a real cost),
+`VLLM_DISABLE_COMPILE_CACHE=1`, `VLLM_USE_AOT_COMPILE=1`. A stale cache is
+usually NOT the cause — verify before "fixing" it (`~/.cache/vllm` may hold
+only `modelinfos`; each `podman run` gets a fresh container FS unless the
+cache dir is mounted).
+
+**Crash-loop signature (shape does not fit):** `Started … ~6.7 min later …
+Deactivated successfully … restart`. Diagnose with `systemctl show -p
+NRestarts` + per-attempt duration. `Restart=always` loops visibly;
+`Restart=on-failure` stays SILENTLY DEAD (`podman wait` returned 0, systemd
+saw a clean exit).
+
+## 3. Observation hygiene during recovery
+
+- A container being STOPPED still answers `podman logs`, so a watcher
+  grepping its log reports PRE-change values and looks like the fix failed.
+  Discriminate by container identity:
+  `podman ps --sort created -a --format '{{.ID}} {{.Status}} {{.CreatedAt}}'`
+  — require the container to post-date the arm start, then read only that ID.
+- A leftover probe script whose tail does `podman rm -f <prod-container>`
+  fires against a freshly DEPLOYED engine. Signature: a previously-healthy
+  unit exits 0 (`Started → Deactivated`, code 0) — an external rm, not a
+  crash. Suspect an external actor before debugging the engine; `pgrep -af`
+  every script you launched (and kill strays of `deploy-remote.sh` by
+  explicit PID — killing the dev-side deploy job does NOT kill its host-side
+  runner).
+
+## 4. Ops laws that bite during recovery
+
+- `Restart=always` does NOT fire after an explicit `systemctl stop` — a
+  deploy's stop leaves the unit down until something starts it. Check
+  `systemctl is-active` yourself.
+- `ExecStop=podman stop -t 120` requires `TimeoutStopSec` > 120; the 90 s
+  default kills the stop and logs `Failed with result 'timeout'`.
+- A `D`-state worker cannot be reaped by ANY signal (SIGTERM/KILL/`podman rm
+  -f` all fail). Check `ps -o pid,stat,etime,comm` before declaring a GPU
+  wedge — measured 2026-10-01: D → R on its own within ~3 min, then the
+  container stopped normally.
+- Never `pgrep -f`/`pkill -f` a pattern that appears in your own command
+  line — kill by explicit PID list.
+- `git stash -u` (not bare `git stash`) for deploy windows — untracked files
+  still trip the deployer's dirty-tree gate.
+- A production engine started by a transient/ladder unit is NOT
+  boot-persistent; pin the winning numbers in the repo and deploy the real
+  unit.
+
+## 5. Logging audit (what more can this container say?)
+
+Answer with evidence from the live image, not upstream docs:
+
+1. Enumerate the fork's own env registry:
+   `podman exec <ctr> python -c "import vllm.envs as e; print(chr(10).join(k for k in dir(e) if k.isupper() and any(s in k for s in ('LOG','DEBUG','TRACE','STATS','PROFILER','VERBOSE'))))"` — then read defaults/clamps in `vllm/envs.py` (e.g. `VLLM_LOG_STATS_INTERVAL` defaults 10.0 s, clamped ≥ 1.0; telemetry emits only while requests are in flight).
+2. **Find the REAL HIP/ROCm library paths before grepping for AMD env vars** —
+   containers often install ROCm into system dirs, not `/opt/rocm`
+   (`podman exec <ctr> sh -c 'find / -name "libamdhip64.so*"'`). A grep of
+   `/opt/rocm/lib` that finds nothing is a FALSE NEGATIVE, not an absence.
+   Then `strings <lib> | grep -E '^AMD_LOG_LEVEL$|^HSA_ENABLE_DEBUG$'`.
+3. Scan vendor kernel `.so`s for switches — and separate tuning vars from
+   logging vars (both share a prefix).
+4. **Measure the DEBUG payoff**: count `logger.debug` sites in the module of
+   interest. 0–10 sites ⇒ a per-module DEBUG bump buys almost nothing; global
+   `VLLM_LOGGING_LEVEL=DEBUG` additionally makes api_server log full
+   request/response bodies (incident window only).
+5. Check the deployed argv for `--disable-log-stats` / `--log-configs` to
+   know what is already suppressed.
+
+Report: lever / effect / cost table, split into permanent-safe (stats
+interval), incident-window (`AMD_LOG_LEVEL`, DEBUG), and does-not-exist
+(absent strings). Wire via env-gated `-e` pass-throughs so default-off
+preserves behavior.
+
+## 6. Interrogate a live endpoint's REAL config
+
+Flags and env diverge from intent constantly — rollbacks drop them silently.
+Read what the container is actually serving, then prove the client contract
+behaviorally:
+
+```bash
+# live argv (the truth)
+ssh host 'podman exec <ctr> bash -c "cat /proc/1/cmdline | tr \"\\0\" \" \"" | tr " " "\n" \
+  | grep -A1 -E "max-model-len|max-num-seqs|kv-cache|reasoning-parser|tool-call-parser|template-kwargs|utilization|speculative"'
+# container env knobs (fork tuning that never appears in argv)
+ssh host 'podman exec <ctr> bash -c "env | grep -E \"R4D_|CLAV_|VLLM_\" | sort"'
+# capacity contract as measured at boot
+ssh host 'podman logs --tail 3000 <ctr> 2>&1 | grep -aE "GPU KV cache size|Maximum concurrency|non-default args"'
+```
+
+`non-default args` lines print what the server overrode — the fast way to
+catch a flag that never landed. `GPU KV cache size: N tokens, Maximum
+concurrency for M tokens per request: X.XXx` is the capacity contract:
+N ≥ requests × M or the last client gets truncated.
+
+**Thinking contract:** a reasoning model on vLLM returns `reasoning` ONLY if
+the server runs a reasoning parser; without it the thinking lands INLINE in
+`content` with a dangling close-tag and a null `reasoning`. Verify with a
+temp-0 completion: pass = `reasoning` non-empty AND no `<\/think>` in
+`content`. Diagnose with `ReasoningParserManager.list_registered()`
+(import `vllm.reasoning`).
+
+**Tool contract:** `tool_choice: "auto"` on a server WITHOUT
+`--enable-auto-tool-choice` returns HTTP 400 — reads like a model fault, is a
+config gap. Probe with a real tool schema and check `tool_calls` +
+`finish_reason`.
+
+**Restoring a dropped contract from history:**
+
+```bash
+git log --all --oneline -S"reasoning-parser"
+git show <commit>:scripts/serve-<svc>.sh | grep -B3 -A3 "reasoning-parser"
+```
+
+Qwen3-family values that work on the tcclaviger fork: `--reasoning-parser
+qwen3`, `--enable-auto-tool-choice --tool-call-parser qwen3_xml`,
+`--default-chat-template-kwargs '{"enable_thinking": true,
+"preserve_thinking": true}'`. `preserve_thinking`/`enable_thinking` are
+chat-template variables, NOT engine flags: they belong in
+`--default-chat-template-kwargs` server-side or `chat_template_kwargs` in the
+request body; a bare top-level body field of that name is ignored. Check the
+payload's `tokenizer_config.json` carries the key before trusting it either
+way.
+
+`/health` 200 says the engine is up, not that the contract is on the flags
+you think — any client report of odd content (inline thinking, 400 on tools,
+clipping at depth) starts with sections 1–3 here before touching anything.
+
+## Related
+
+- [[RDNA4 MoE Expert Offload VRAM Budget]] — the plan arithmetic behind OOMs
+- [[dankserver-qwen38-fp8-expert-offload-tuning]] — deployed shape and history
+- [[frank-large-benchmark]] and [[frank-small-benchmark]] — probe/ladder hygiene on live hosts

serverAppConfigs-docs/LLM/vLLM Porting and Compile Triage.md

@@ -0,0 +1,330 @@
+---
+type: reference
+title: vLLM Porting and Compile Triage
+description: Forward-porting a model architecture to a new vLLM major (additive-only discipline, gap enumeration, shims that survive worker spawn) and the torch.compile/dynamo break triage, including the EAGLE3/DFlash2 aux-hidden-state protocol port rules.
+tags: [vllm, porting, torch-compile, dynamo, eagle3, dflash, glm]
+timestamp: 2026-10-05T00:00:00-07:00
+---
+
+# vLLM Porting and Compile Triage
+
+## Part I — Forward-porting a model architecture
+
+### 1. Check whether a port already exists; size the work honestly
+
+Read what the target image already ships before assuming a port is needed: a
+backport's upstream PR may be merged in the target version, and the target may
+carry a native kernel path the older line lacks (e.g. `rfi_hip` for RDNA4
+native quantized MoE). Porting around a native path that was already there is
+the expensive mistake.
+
+An upstream backport ships two kinds of file: **additive** (the new model
+package, its config, its kernel modules — port these) and **whole-file
+replacements of vLLM core** (`model_runner.py`, `scheduler.py`,
+`kv_cache_utils.py`, platform files — these are version X's copies of files
+version Y restructured; copying them is a DOWNGRADE, not a port). Measure both
+(`find … | xargs wc -l`), and say the additive-only ratio out loud before
+starting — one real case: a 71,724-line overlay reduced to 10,385 additive,
+and the "missing" drafter was already registered in the target.
+
+### 2. Enumerate the gap in ONE pass, not one ImportError at a time
+
+Inventory the target tree first (`model_executor/layers/`,
+`v1/attention/backends/`, quantization backends, spec-decode drafters in
+`registry.py`, `third_party/flash_linear_attention/ops/`). Then walk the
+staged package with `ast`, resolving every `from vllm… import X` against the
+target, split into MODULE-missing vs NAME-missing.
+
+- **The double-prefix bug:** the dotted module already starts with `vllm.`,
+  so joining onto the package dir needs
+  `rel = mod.split(".", 1)[1].replace(".", os.sep)`.
+- **Static scans over-report:** bare annotations, TypeAlias and star
+  re-exports resolve at runtime. Do a runtime check
+  (`importlib.import_module` + `getattr`) before quoting numbers — one scan
+  said 151/163 resolved; runtime said 10/13.
+- **Classify:** redirects dominate (version moved a symbol to a submodule) —
+  a ~60-line rewriter splitting `from X import (a, b, c)` blocks handles
+  them; **delete a block that empties** (a bare `from X import (` with an
+  empty body is a SyntaxError) and gate the rewriter on `ast.parse` of every
+  touched file.
+- **Port absent symbols with AST extraction**, not copy-paste: locate the
+  `ClassDef`, check every base/decorator resolves in the target, append the
+  exact segment, re-`ast.parse` the target, roll back on failure, skip if
+  already defined.
+
+### 3. Silent-default traps and transformers gates
+
+- Check every `cfg_int(cfg, key, DEFAULT)` against the new checkpoint: if the
+  key was renamed, the DEFAULT silently survives and the model computes fluent
+  nonsense with no error. Diff every config key the old parser reads against
+  the new `config.json`. When the old port used a fallback, the CHECKPOINT's
+  value must win (measured: `num_experts_per_token` fallback 7 vs
+  checkpoint `num_experts_per_tok=8`).
+- `model type X but Transformers does not recognize this architecture` =
+  pinned transformers predates the model. Upgrading transformers inside a
+  pinned vLLM image breaks things hard to attribute — prefer registering a
+  genuine `PretrainedConfig` subclass:
+  `CONFIG_MAPPING.register("glm5_next", Glm5NextConfig)`, gated by a real
+  round-trip (`AutoConfig.from_pretrained(<ckpt>)` returns the right class).
+  If the target image's transformers is below vLLM's declared minimum,
+  upgrading inside the image beats a shim.
+- **Hardware-gate flags from the port's own docs**: search the README for the
+  arch name and "do not" — e.g. a native quant MoE backend that is
+  CUDA/CDNA-gated falls back to per-call quantize-dequantize emulation at
+  ~5x slower; pin those flags in the serve script with a comment citing the
+  line.
+- **Backend selection:** default to `auto`, then READ the chosen-backend
+  line. `TRITON_UNFUSED` rejects some activation keys; `TRITON_ATTN` rejects
+  MLA + sparse attention — both surface as an init rejection. **"Using
+  'EMULATION' backend" is a FAILED configuration, not a success** (weights
+  dequantized per call: memory + throughput cost).
+- `Loading model from scratch…` for many minutes usually means the port
+  SUCCEEDED and you are in weight placement — verify all layers constructed,
+  both TP workers alive, zero errors, before chasing memory. And if it does
+  not fit, **prove the band is empty**: sweep the knob and show no setting
+  works; a single OOM is not a conclusion. Keep `MemAvailable` guarded
+  (abort on low RAM; do not let the OOM killer take co-tenants).
+
+### 4. Shims that must survive worker spawn
+
+vLLM v1 workers are spawned processes — a parent-process monkeypatch is gone
+by weight-load time. In order of preference:
+
+1. **Patch the installed library's source file on disk** (reliable). Locate
+   the real install path at runtime:
+   `python -c "import transformers,os;print(os.path.dirname(transformers.__file__))"`.
+2. `.pth` in site-packages — works but runs at interpreter start and can
+   import a package mid-import (circular import through `accelerate`);
+   expect failure.
+3. `sitecustomize.py` — often silently DEAD: Debian ships its own at
+   `/usr/lib/python3.12/sitecustomize.py`. Verify it is actually imported
+   before debugging why it has no effect. (A `.pth` on `PYTHONPATH` is never
+   executed by `site.py`; `sitecustomize.py` is.)
+
+Always confirm with a FRESH process — never with the process that set the
+patch up (`is_moe`, `num_experts_per_token`, `mhc_num_residual_streams`).
+
+**Misleading OOM names on ROCm:** `torch.AcceleratorError: CUDA error: out of
+memory` from `pin_memory()` is usually the HOST `memlock` rlimit, not VRAM.
+Cannot be raised from inside a container or root's own shell; only PAM at
+login: `/etc/security/limits.d/*.conf` + a NEW session. Confirm by pinning
+2/8/16 GiB — if those succeed, the rlimit was a real bug but is not the
+failure you are chasing.
+
+### 5. Ship as an idempotent patch-at-start applier
+
+One `apply.sh` at container start, each step guarded by a grep-for-marker
+(re-entry safe), ending in a verification block that imports every installed
+symbol and prints PASS/FAIL per symbol — ~10 s instead of a multi-minute
+load, and it turns "I think it works" into evidence.
+
+---
+
+## Part II — torch.compile / dynamo bring-up triage
+
+### The five laws (measured on torch 2.11+rocm, vLLM 0.29-dev)
+
+1. **`suppress_errors` does NOT cover graph breaks** — fullgraph=True +
+   suppress_errors=True still raises `Unsupported`. Only `fullgraph=False`
+   tolerates a break.
+2. **A warm `lru_cache` is not trace-safe** — dynamo re-traces the wrapped
+   body to build its guard even on a hit ("Dynamo ignores the cache wrapper
+   and directly traces the wrapped function"). Resolve the value ONCE into a
+   plain module global and read that; a global read folds, a cached call does
+   not. Verified in-image: warm lru_cache-calling function still fails
+   fullgraph=True; a plain global read compiles.
+3. **AOT compile refuses breaks outright** ("Graph breaks are not supported
+   with aot compile") — set `VLLM_USE_AOT_COMPILE=0` for a break-tolerant
+   run.
+4. **Fatal vs tolerated breaks:** tolerated breaks get a resume function
+   (untraceable part runs eagerly, numerics unchanged). Fatal ones cannot:
+   `Unsupported: Read uninitialized cell` (a traced callee closes over a name
+   bound after its definition) and `Skip calling torch.compiler.disable()'d
+   function` under fullgraph=True. `torch.compiler.disable` is the documented
+   escape hatch for host-side/stateful work (ContextVar reads, lazy
+   allocation, workspace grab) and is a NO-OP when nothing compiles — safe to
+   ship in the stock path.
+5. **Trust the frame chain, not the line number** — dynamo INLINES user
+   callees; an inlined failure reports the enclosing function's `def` line.
+   "KeyError in forward at the def line" is really "a callee raised".
+   `TORCHDYNAMO_VERBOSE=1` adds nothing for a plain Python exception (only
+   for dynamo-internal errors).
+
+**The tell** that the trunk lacks compile support at all:
+
+```
+Cudagraph mode FULL_AND_PIECEWISE requires piecewise capture, but the loaded model
+provides neither a compiled submodule nor breakable CUDA graphs. Overriding to
+FULL_DECODE_ONLY.
+```
+
+Diff against the family that DOES compile and copy its
+`@support_torch_compile` decorator including `dynamic_arg_dims` and any
+`enable_if` hook; env-gate every change so the file can stay mounted for an A/B.
+
+### Procedure
+
+0. **Establish what configuration is ACTUALLY in effect**:
+   `podman exec <ctr> printenv VLLM_COMPILE_FULLGRAPH VLLM_USE_AOT_COMPILE …`.
+   A launcher that "sets" an env var in a branch may silently drop it — four
+   consecutive boots ran `fullgraph=True` while the operator believed
+   otherwise, which makes the reported failure MOVE as breaks are removed and
+   invites entirely wrong theories.
+1. **Boot with the break list, not the first error:** `TORCH_LOGS=graph_breaks`
+   + `VLLM_USE_AOT_COMPILE=0` + `VLLM_COMPILE_FULLGRAPH=0`. Re-verify each
+   candidate break under the flag-free config you actually intend — a break
+   that appears only under the logging flag is a diagnostic artifact.
+2. **Fix in priority order:** (a) host syncs in the traced region
+   (`.item()`/`.cpu()`/`.tolist()` — check reachability first, a dead fallback
+   branch is not your blocker); (b) capability probes doing I/O in `forward`
+   (`os.path.exists`, `.so`/`ctypes` lookups — resolve once into globals,
+   warm at import); (c) lazy stateful work (hoist, or `torch.compiler.disable`
+   a thin wrapper); (d) new kernel specializations — check the vendor ships
+   SOURCES (`nm -D`/`strings`): a binary-only templated `kernel<128|256|512>`
+   makes the 288 you need unreachable. Fix at the PATTERN level (one
+   launcher/decorator family breaking all 39 call sites), not site level.
+3. **KeyError on a bare module name** = dynamo's name→object map miss
+   (`OutputGraph.nn_modules`: registered `output_graph.py:1259`/`:1384`,
+   read back `:3000`, `_buffers/_parameters/_modules` stripped at `:1225`).
+   Ship a probe via `sitecustomize.py` that patches
+   `OutputGraph.example_value_from_input_node` and keeps weakrefs of live
+   graphs, printing `node.target`, key count, near-miss keys — separates a
+   cross-graph reference from a name-derivation mismatch.
+4. **Instrumenting a startup hook is a FIREARM — arm it once:**
+
+   ```python
+   if os.environ.get("PROBE_ARMED") == "1":
+       pass                     # an ancestor already armed it -> stay inert
+   else:
+       os.environ["PROBE_ARMED"] = "1"
+       ...install probes...
+   ```
+
+   A `sitecustomize.py` on `PYTHONPATH` imports in EVERY interpreter and
+   children inherit `PYTHONPATH`: an unguarded hook that imports the engine
+   self-replicates (observed: 19+ levels, 33 processes alive 18 min later).
+   Kill the chain by removing the container whose cgroup owns every
+   generation, then verify with a precise `/proc/<pid>/environ` count —
+   NEVER pattern-kill an interpreter name on a host running python in
+   production containers. Also chain the image's own sitecustomize so its
+   hook survives.
+5. **Cheap in-image classification (no boot):**
+   `g = torch.compile(fn, backend="eager", fullgraph=True)` inside the
+   container (with `--device /dev/kfd --device /dev/dri` — torch.compile
+   queries a device even on the eager backend). ~5 s per question instead of
+   ~7 min boots.
+6. **Package as real patch artifacts:** hand-written `@@` headers are not
+   diffs and `patch(1)` rejects them. Generate
+   `diff -u --label a/<path> --label b/<path> orig new > name.patch` with
+   `--- a/` / `+++ b/` paths so `-p1` works; verify by applying to a fresh
+   extraction of the pinned image and comparing sha256 (`sha256sum` must
+   equal the `b/` hash recorded in the header). Header carries: STATUS, the
+   measured evidence, base+patched sha256, how to apply, the NEXT blocker.
+
+### Cost model before investing
+
+**Profile first** — parse the rank0 chrome trace (`POST /start_profile`,
+device-activity rows) for the kernel table. Measured case: the long-suspected
+expert-gather was **1.05 %** of device time while **44 %** was unfused
+pointwise glue (708,500 `vectorized_elementwise_kernel` launches =
+1,428/token) — a months-old "structural" conclusion died in one afternoon.
+Speculative decoding does NOT help a shape whose per-token cost scales with
+the token count, and it hurts prefill. Decide the ceiling up front: if the
+fused target is ~13–15 tok/s, do not spend a night of boots chasing 35.
+
+---
+
+## Part III — EAGLE3 / DFlash2 / DSpark aux-hidden-state protocol port
+
+Goal: make a trunk satisfy `SupportsEagle3` so a DFlash2 drafter
+(`DFlash2DraftModel`, method `dflash`) boots. Built on the GLM-5.3-Flash port
+(tcclaviger fork, 2026-09-30; four failed boots produced the laws below).
+
+### The four laws
+
+1. **Patch the instance the runner calls.**
+   `set_eagle3_aux_hidden_state_layers(self.model)` (model_runner.py, right
+   after `model_loader.load_model`) runs on the LOADED instance — for a
+   multimodal model that is the `…ForConditionalGeneration` WRAPPER (the
+   APIServer log line `Resolved architecture: …` names it), NOT the text
+   model it holds as `self.language_model`. Patching only the inner
+   `…ForCausalLM` reproduces the identical `RuntimeError: Model does not
+   support EAGLE3 interface` forever. Patch BOTH wrappers.
+2. **The check is Protocol isinstance, NOT duck typing.**
+   `supports_eagle3(model)` = `isinstance(model, SupportsEagle3)`
+   (interfaces.py ~1728; caller `v1/worker/gpu/spec_decode/eagle/eagle3_utils.py:20`).
+   `SupportsEagle3` is `@runtime_checkable` over base `SupportsEagleBase`
+   (`:1486`, `:1659`); `typing._ProtocolMeta.__instancecheck__` does
+   `getattr_static` over ALL `__protocol_attrs__` — including BASE-protocol
+   members — and breaks on the FIRST miss. Full member set:
+   `supports_eagle3`, `has_own_lm_head`, `has_own_embed_tokens`,
+   `set_aux_hidden_state_layers`,
+   `get_eagle3_default_aux_hidden_state_layers`. **Class attributes count as
+   presence; a methods-only implementation FAILS** because the three attrs
+   are missing. Verified in-image with stock typing, no boot: flag-only →
+   False; flag + two base members → True.
+3. **MRO forbids inheriting the Protocol.** Appending `SupportsEagle3` to a
+   wrapper already carrying Protocol-derived bases dies at class creation:
+   `Cannot create a consistent method resolution order (MRO) for bases …` —
+   MRO simulation (stub Protocols + `type("C", bases, {})`) confirmed no
+   position works. This failure also PROVES the isinstance-semantics question
+   is moot: class creation precedes any check.
+4. **Write the two methods by hand** — without inheritance the Protocol
+   defaults do not exist. Text model: unwrap to the trunk
+   (`self.model.aux_hidden_state_layers = layers`); multimodal wrapper:
+   delegate to `self.language_model` (the pixtral.py:498 pattern).
+   `get_eagle3_default_aux_hidden_state_layers` returns `(2, L//2, L-3)`; the
+   drafter config's `target_layer_ids` wins when present (GLM DFlash2 ships
+   `[5, 14, 24, 33, 42]` of 45).
+
+The fix shape (on EACH wrapper class):
+
+```python
+supports_eagle3: ClassVar[Literal[True]] = True
+has_own_lm_head: bool = False          # False when embed/lm_head are delegated
+has_own_embed_tokens: bool = False
+def set_aux_hidden_state_layers(self, layers): ...
+def get_eagle3_default_aux_hidden_state_layers(self): ...
+```
+
+### Trunk mechanics
+
+The trunk (`Glm5NextModel`) must be `EagleModelMixin`; add
+`aux_hidden_state_layers: set[int]` (default empty) + `start_layer` handling.
+Capture aux states in the layer loop: the state ENTERING each requested
+layer, BEFORE its input norm (GLM's loop carries the single-stream state;
+deepseek_v2's equivalent is `hidden_states + residual`).
+`use_aux_hidden_state_outputs=True` is set in model_runner.py (~:287) for
+methods `dflash`, `dspark`, `extract_hidden_states`; the forward returns
+`(hidden_states, aux_hidden_states)` conditionally after `self.norm` (tuple
+consumed at `v1/worker/gpu/cudagraph_utils.py:621`).
+
+### Window discipline and verification
+
+- Scratch boots on dankserver MUST stop + runtime-mask production
+  `frank-primary` first — the expert-offload planner reads `MemAvailable` at
+  plan time, and beside-prod boots die at plan with
+  `ExpertPlanError: host holds N of M swappable experts`. After stop+mask,
+  loop until `MemAvailable ≥ 230 GiB` before launching (a stop against a
+  mid-load engine SIGKILLs at ~90 s and the freed ~170 GiB takes >1 min to
+  return — a new window started during restore inherits the false failure).
+- Never launch a window while the previous window's RESTORE is in flight
+  (`systemctl start` returns immediately; the engine loads ~6 min).
+- Save full probe logs BEFORE teardown (`podman logs <probe> > full.log`);
+  decision greps truncate lines at `…` and tracebacks die with the container.
+- Cheap verification, no boot: AST-parse the patched file for class bases +
+  member presence; MRO simulation with stub Protocols; in-image `typing`
+  replication of the isinstance walk (pure stdlib — importing vllm itself
+  crashes on GPU-less inspection). Boot check: the worker line `Using Eagle3
+  auxiliary layers from config: (…)` = gate cleared.
+- Honest expectations: a target-side DFlash2/EAGLE3 port on a
+  pointwise-glue-bound trunk (~44 % of kernel time) buys ~ZERO decode gain —
+  the verification pass costs ∝ its token count. Port it for capability, not
+  speed.
+
+## Related
+
+- [[vLLM Boot Forensics and Config Interrogation]] — capture and contracts
+- [[RDNA4 Container Serving]] — the serving stack being ported onto
+- [[dankserver-qwen38-fp8-expert-offload-tuning]] — the live deployment
+- [[halogen-flash-server]] — the retired w4b-era endpoint and its profiling notes

serverAppConfigs-docs/Operations/Container Host Reachability.md

@@ -0,0 +1,128 @@
+---
+type: reference
+title: Container Host Reachability
+description: The two podman/netavark reachability quirks on the fleet — containers cannot dial unpublished host ports (LAN IP or loopback), and netns membership decides what "reachable" means. Probes that prove it, fixes, and the pasta port gate.
+tags: [podman, netavark, networking, rootless, reachability, caddy]
+timestamp: 2026-10-05T00:00:00-07:00
+---
+
+# Container Host Reachability
+
+Two failure families on rootless podman + netavark hosts (Dankclaw), both
+reading as "service down" when the service is up and the *path* is wrong.
+Fleet source of truth: `serverAppConfigs-docs/Architecture/Container Network
+Namespaces.md` (commit `2e179b0`).
+
+## Quirk 1 — LAN-IP dialing of an unpublished host port refuses instantly
+
+A container cannot reach a `Network=host` service at the host's LAN IP.
+netavark DNATs **published** ports only: from a bridge container,
+`192.168.0.143:<published>` answers while `192.168.0.143:<host bind>` and the
+gateway `10.89.1.1` are both refused. This is what 502'd
+`hermes.franksdank.cloud` until the Caddyfile upstream became
+`host.containers.internal` on 2026-09-16 — its vhost comment had claimed the
+LAN IP was open from that bridge; it never was.
+
+Probe that proves it in one command:
+
+```bash
+ssh <host> "podman run --rm --network <net> --entrypoint sh \
+  docker.io/library/busybox:latest -c \
+  'for p in <target-port> <published-port>; do
+     nc -w2 -z -v <host-lan-ip> \$p >/dev/null 2>&1 && echo port-\$p-OK || echo port-\$p-BAD
+   done'"
+podman exec <svc> bash -c 'exec 3<>/dev/tcp/host.containers.internal/<port> && echo ALIAS-OK'
+```
+
+Published OK + target BAD ⇒ quirk confirmed. Note busybox `nc -z` reports any
+failure as failure — use it for bulk sweeps; use `/dev/tcp` (or `nc -v`) when
+refusal-vs-timeout matters.
+
+Fix: point the service's host-facing URL at
+`http://host.containers.internal:<port>` in compose/quadlet env, redeploy via
+git (`deploy.sh <target> --service <svc>`), verify
+`podman exec <svc> printenv <URL_VAR>` and the connect log line (~40 s).
+**Do not publish the daemon port** to make the LAN IP work — publishing is
+exactly what makes other ports refuse.
+
+## Quirk 2 — addressability is per netns, not per host
+
+- `NetworkMode=host` → the container **is** the host netns: dials all of
+  `127.0.0.1`, binds any free port.
+- bridge/named net → own netns; only published ports are reachable, via the
+  network's **gateway IP**. Unpublished host listeners refuse **instantly
+  (RST)** — no firewall rule, no stale DNAT, `nft`/`iptables -t nat` empty. Do
+  not hunt for REJECT rules; probe.
+
+Probe sequence (read-only):
+
+```bash
+podman inspect <name> --format '{{.HostConfig.NetworkMode}}'
+sudo ss -ltnp 'sport = :<port>'        # host netns only — why 0.0.0.0 can still be unreachable
+podman exec <name> cat /proc/net/route  # default route = the gateway IP to dial
+podman exec <name> sh -c 'timeout 2 bash -c "exec 3<>/dev/tcp/<gw-ip>/<port>"; echo rc=$?'
+```
+
+Interpretation without blurring:
+
+- **instant refusal** = nothing listening in a reachable namespace (or
+  rootlessport rejected an unpublished port map).
+- **timeout** = no route to that address at all. `host.containers.internal`
+  (`169.254.1.2`) is an alias on the **default `podman` net only**; containers
+  on a named network time out against it.
+
+Tool gotchas (podman 5.x, verified live):
+
+- `podman network inspect <net> --format '{{json .containers}}'` errors
+  (`can't evaluate field containers in type interface {}`) — list membership
+  from the container side: `podman ps -a --format '{{.Names}} {{.Networks}}'`.
+- An **empty** `Networks` field = host network or shared namespace
+  (`--network=container:<other>`), not "no network".
+- Quadlet `Network=<name>.network` yields the runtime name **`systemd-<name>`**
+  — probe with `podman network ls`, never the `.network` filename.
+
+Fix options when a bridge-net container must reach a host-netns listener:
+(a) move it to `Network=host`, or (b) publish the port — (b) makes the port
+reachable from every namespace **and the LAN**, an exposure change needing
+explicit operator approval, never a silent fix.
+
+## Related trap — the pasta port gate (rootless published ports)
+
+On hosts publishing ports through pasta, the port is bound in **userspace at
+container CREATE**, not when the app binds. A predecessor stopped moments
+earlier can still hold listener/TIME_WAIT state, so the replacement dies
+`pasta: ... bind: Address already in use` (conmon exit 126) — then **self-heals
+seconds later** under `Restart=on-failure`. The failure looks like it failed,
+then looks fine, and the missing gate stays invisible because the retry masks
+it. Observed 2026-09-16 cutting `frank-large` from a root unit to a rootless
+Quadlet.
+
+Gate procedure: stop the old unit AND the new one (a redeploy's running
+replacement is itself the port-holder — without that stop the gate deadlocks
+against itself), then wait for the port to be empty across **all socket
+states**:
+
+```bash
+wait_port_free() {
+    local port="$1" limit="${2:-90}" i
+    for ((i = 1; i <= limit; i++)); do
+        if [ -z "$(ss -Htan "sport = :$port" 2>/dev/null)" ]; then return 0; fi
+        sleep 1
+    done
+    echo "FAIL: port $port still held after ${limit}s:" >&2
+    ss -tan "sport = :$port" >&2 || true
+    return 1
+}
+```
+
+`sport` catches accepted-connection server sockets and TIME_WAIT residue that
+`ss -ltn` misses; fail loudly with the socket table attached. Verify the gate
+by re-running the same deploy against a running service (idempotency) and
+assert it in seam tests (`wait_port_free` present in the deploy handler).
+
+## Related
+
+- [[Quadlet Lifecycle]] — the unit layer these containers run under
+- [[Container Network Namespaces]] — fleet network namespace source of truth
+- [[Observability]] — the alerting paths that depend on reachability
+- [[Deploy State Triage]] — the deploy flow that installs these configs

serverAppConfigs-docs/Operations/Fleet Audit and Posture.md

@@ -0,0 +1,331 @@
+---
+type: reference
+title: Fleet Audit and Posture
+description: Three standing audits — monitoring completeness (what should exist vs what is merely down), vhost proxy health decoding, and the behavioral security posture audit; plus the stash supersession check before dropping old work.
+tags: [audit, security, prometheus, caddy, monitoring, git]
+timestamp: 2026-10-05T00:00:00-07:00
+---
+
+# Fleet Audit and Posture
+
+## Monitoring gap audit
+
+Use after any reboot, or whenever Prometheus targets look wrong. Two failure
+modes, only the first announces itself.
+
+**1. A service that should exist and doesn't is invisible.** A missing
+exporter shows up as nothing — no failing target if the job was configured
+for a host that no longer serves, no alert because the series is absent. So
+check **existence, not health**:
+
+```bash
+podman ps -a --format '{{.Names}} {{.Status}}'   # includes dead ones
+ls ~/.config/containers/systemd/*.container      # what should be boot-persistent
+systemctl --user is-enabled <name>.service       # expect: generated
+```
+
+A container in `ps -a` that is `Exited(…)` **and has no unit** is the classic
+case: rootless podman does not restart bare containers after the stop that
+shutdown performs, so a compose-only service dies at the first reboot and
+stays dead. If the compose file still names it, the repo's own rule is being
+violated — fix by writing the quadlet (mirror the same service on the other
+host), deleting the compose entry (one owner per container name), and making
+the deploy path reach it: planner service-set + path rule, dispatcher
+allowlist `<target>:<service>`, handler installing from `"$TARGET/..."` never
+a hardcoded peer's path.
+
+**2. Classify every down target — do not report a count.**
+`targets?state=active` gives health **and** last error; two down targets can
+mean opposite things:
+
+| Symptom | Meaning |
+|---|---|
+| `connection refused` | service not listening → real outage (podman ps, logs, exit codes) |
+| `404` on `/metrics` | exporter path wrong or metrics disabled → config defect, service fine |
+| `unsupported Content-Type …` | scraping an HTML *page* as metrics → probe inert; Prometheus 3 refuses HTML; a real probe needs a blackbox exporter |
+| target absent | §1 |
+
+```bash
+curl -s "http://127.0.0.1:9090/api/v1/targets?state=active" | python3 -c "
+import json,sys
+for t in json.load(sys.stdin)['data']['activeTargets']:
+    if t['health'] != 'up':
+        print(t['labels']['job'], t['scrapeUrl'], (t['lastError'] or '')[:80])"
+curl -s -o /dev/null -w '%{http_code} %{content_type}\n' http://<target>/metrics   # hand-probe before believing the scraper
+```
+
+"7 of 11 up" without the four reasons is how a real outage hid behind three
+config defects on this fleet — the crash-looping GPU service was one of four
+failures reported as a number.
+
+**3. Exposure posture while you are in there.** An exporter bound to the
+wildcard publishes an unauthenticated API on every interface. Expect exactly
+two listeners per host-exporter — loopback for host-local probes, the host's
+own LAN address for the Prometheus container: `ss -ltn | grep <port>` must
+show two explicit addresses, never `0.0.0.0`.
+
+**4. Leave decisions that are not yours.** Fix mechanical drift (missing
+unit, wrong path, wildcard bind). Do NOT disable a probe to green a
+dashboard, or "fix" a crash-looping service whose posture depends on owner
+intent — capture the evidence (restart count, exit code, last error line) and
+hand it over intact.
+
+## vhost proxy health probes
+
+After deploying or auditing a Caddy/Anubis front door, verify **from a
+client**, not from the backend's localhost — that is where transport
+mismatches, auth walls, and dead upstreams show up.
+
+```
+curl -s -o /dev/null -m 6 -w "%{http_code}\n" https://<vhost>/ping
+curl -s -o /dev/null -m 6 -w "%{http_code}\n" https://<vhost>/
+```
+
+A status code is a diagnosis, not a failure:
+
+- `200` on `/ping` from *arrs = alive; **auth state unknown** (`/ping`
+  bypasses auth by design) — probe `/` to decide.
+- `404` = backend **alive**, route missing (qBittorrent has no `/ping`; 404
+  there = healthy).
+- `502` = Caddy could not reach/negotiate with the upstream: dead backend
+  **or transport mismatch**. Probe the backend port directly from the proxy's
+  network (`ssh <proxy-host> curl http://<backend>:<port>/…`). Direct 200 +
+  vhost 502 ⇒ config bug, not an outage.
+- Login page to anonymous (grep body for `password`) = auth ON; instant
+  redirect to `/transferlist/`-style routes = auth OFF = open door.
+
+**Transport-mismatch mechanism:** `transport http {versions h2c 1.1}` makes
+Caddy send an h2c upgrade preface; .NET/Kestrel backends (Jellyfin) reject it
+→ 502 per request while the same port answers 200 on plain HTTP/1.1. Fix =
+drop the override (default HTTP/1.1). Opposite twin: Plex PMS drops plain
+HTTP off-loopback and needs a TLS transport. Match the transport to what the
+backend speaks.
+
+**Shell trap when gating commits on a test suite:** `grep -c "^not ok"` exits
+**1 when zero mismatches** — inside an `&&` chain it silently kills
+everything after it while still printing `0`. Wrap:
+`(./tests/run.sh | grep -cE '^not ok' || true) && git commit …`.
+
+## Security posture audit (behavioral)
+
+Answers "what can the internet touch, and is everything else actually
+gated?" — **all findings must be behavioral probes**, not config reads.
+
+**1. Inventory (source of truth).** `dankclaw/Caddyfile`: classify every
+vhost — guarded (`reverse_proxy anubis:8923`), exempt (direct upstream), or
+redirect-only. Public DNS: `dig +short +time=3 +tries=1 <name>.franksdank.cloud
+@1.1.1.1` per name — a record at the WAN IP = public; NXDOMAIN = LAN-only by
+DNS absence.
+
+**2. Guard chain (expect the challenge, not the app).** Per guarded vhost:
+`curl -sk https://<vhost>/` and grep the body for
+`anubis|challenge|proof-of-work` (~8 markers on a live challenge page). A 200
+with NO markers means the app is served directly — the guard is bypassed.
+
+**3. Exempt vhosts — per-app gating probes.** OpenWebUI `/api/v1/users/me` →
+401. Gitea `/api/v1/user` → 401; repo visibility `/api/v1/repos/dank/<repo>`
+→ 404 = private. Pi-hole any path → 401 (Caddy basicauth edge). Plex `/web/`
+→ 302 (login is JS-rendered). Calibre-Web `/` → 302 to `/login`. Jellyfin
+`/Users` → 401. Prowlarr/Sonarr/Radarr `/api/v3/system/status`; Lidarr/Readarr
+`/api/v1/system/status` → 401 = gated, 200 = fully open. qBittorrent
+`/api/v2/torrents/info` → 403 without SID. Hermes/hermesadmin `/` → 302 to
+`/login`. IdP (`auth.`): `/` → 302 → `/if/flow/default-authentication-flow/`
+→ 200 (flow HTML is React; grep "password" finds nothing — the 200 on the
+flow URL is the proof).
+
+**4. WAN port scan — from an EXTERNAL vantage.** LAN→WAN hairpin is
+unreliable; scan from gluetun's ProtonVPN egress:
+
+```bash
+ssh dankclaw 'podman exec gluetun sh -c "for p in 21 22 53 80 443 3000 3005 8080 \
+  8081 8082 8083 8085 8096 8787 8923 8989 9000 9090 9119 9696 7878 8686 3100 \
+  5055 32400 7770 7771 2019; do timeout 2 wget -qO- http://71.227.227.172:\$p/ \
+  >/dev/null 2>&1 && echo OPEN: \$p; done"'
+```
+
+Confirm the egress is truly external first (`wget -qO- https://api.ipify.org`
+→ ProtonVPN IP, not Comcast). **Trap: plain-HTTP probes against TLS ports
+(443, 32400) are false negatives** — port open, protocol mismatch; re-probe
+with `wget -qO- https://IP[:port]/`. busybox wget has no `--secure` flag; it
+errors instantly and the `&&` chain silently skips the echo (a sub-second
+wall time on a "timeout" scan means the flag failed, not that ports closed).
+
+**5. Residuals.** Grafana basic-auth residual: `curl -u admin:admin
+http://127.0.0.1:3002/api/org` → 200 means `admin:admin` still live
+(loopback-only; the vhost path needs Anubis + OAuth2 first). LLM binds:
+`sudo ss -tln | grep -E ':(7770|7771)'` — `0.0.0.0` = LAN-reachable with no
+auth (OpenAI API, no token). Stale-record check: current WAN IP =
+`ssh frankbook-pro 'curl -s https://api.ipify.org'` (residential egress);
+compare against every public A record — a record at a different Comcast IP
+is stale.
+
+**Baseline (2026-09-21):** WAN: only 80 + 443 open. Public names: `www`,
+`search` (guarded) + `ai`, `gitea`, `plex`, `books`, `jelly` (own auth, all
+verified) + `auth` (IdP, by design); everything else LAN-only. Open findings:
+LLM endpoints 0.0.0.0 auth-less on the LAN; Grafana `admin:admin` loopback
+residual; `plex.franksdank.casa` stale (73.59.75.210 = previous Comcast IP);
+`plex.franksdank.cloud` WAN-broken (Plex Web 302s to LAN origin
+192.168.0.235:32400); vestigial *arr handlers on the :8017 post-auth listener
+(dead config, no exposure).
+
+## Stash supersession audit before drop
+
+Use when a repo (especially a live-deployment checkout where the working tree
+IS the deployment) has an old stash whose content may have been superseded by
+later commits, and dropping blindly risks losing unique facts.
+
+1. **Inventory**: `git stash show --stat stash@{0}`.
+2. **Probe for unique content per file**: `git show "stash@{0}:<path>"` and
+   diff against HEAD; count lines present only in the stash version. A file
+   with zero unique-vs-HEAD lines is fully superseded — safe to drop.
+3. **Classify each unique-content hit**: *stale fact* (superseded by a later
+   refactor, e.g. old model naming) → drop is correct; *live durable fact*
+   (an operational invariant never re-recorded) → extract it into live
+   docs/memory or flag to the user before dropping.
+4. **Archive before dropping** — zero loss: tag the stash commit, then drop:
+
+```bash
+git tag archive/stash-<YYYY-MM-DD>-<base-branch>-wip "stash@{0}"
+git stash drop "stash@{0}"
+```
+
+The tag pins the full stash object including its index parent; recoverable
+via `git show archive/stash-…` or `git checkout archive/stash-…^3 -- <path>`
+for the untracked-files third parent. Stashes are invisible in normal history
+and expire from reflogs (~90 days); tagging converts "drop" into a
+reversible archive at zero workflow cost.
+
+## Whole-fleet audit (git/test/live/docs cross-check)
+
+Deep-audit the homelab fleet — git/test state, live service health, and
+docs-vs-live cross-check. Run from the dev checkout
+(`/home/blank/Projects/serverAppConfig`).
+
+**1. Probe the live fleet — self-identifying, ONE ssh per host.** Do NOT
+trust the display order of parallel SSH outputs: they can render crossed and
+you will attribute host A's output to host B (2026-09-21: a false host↔IP
+"conflict" dissolved by a single self-contained per-host probe). Put
+`hostname` + a distinctive marker + the LLM identity in the SAME command:
+
+```
+for ip in 192.168.0.143 192.168.0.235; do
+  ssh -o BatchMode=yes dank@$ip \
+    'echo "host=$(hostname)"; \
+     echo "romm=$(podman ps -q --filter name=^romm$|wc -l) caddy=$(podman ps -q --filter name=^caddy$)"; \
+     echo -n "llm7770="; curl -s -m3 http://127.0.0.1:7770/v1/models|head -c60; echo'
+done
+```
+
+Ground truth (verify, don't assume — IPs can move): 192.168.0.143 = dankclaw
+= AI stack + `frank-large` :7770 + `frank-small` :7771 (RTX 3090 + Strix
+Halo iGPU); 192.168.0.235 = dankserver = media stack + `frank-primary`
+:7770 (2× R9700). Repo dirs + live hostnames agree; the deploy tooling is
+hostname-anchored (`ssh dankclaw`/`ssh dankserver` via `~/.ssh/config`), so
+it is unaffected by IP drift.
+
+**2. LLM endpoints — health + identity.** `curl /health` per endpoint, then
+`/v1/models` (first 120 chars) — a 200 with the right model `id`/`name` is
+the pass condition.
+
+**3. Tests — the deploy completeness gate.** `./tests/run.sh` (bats) must
+end `ALL TESTS PASSED`; `tests/e2e/run.sh` runs Playwright against the LIVE
+fleet. **A single-test flake is normal**, not a regression: the plex crawl
+intermittently lands on `app.plex.tv`, the grafana SSO test occasionally
+times out, and the FrankBook-Pro leg fails when the mini sleeps. Re-run just
+that spec (`npx playwright test --project=standard specs/<n>-<name>.spec.ts`)
+to classify before alarming. A `deploy.sh` e2e gate can fail on a flake
+while the change itself is fine — verify the actual change live, then re-run
+the suite.
+
+**4. Repo state.** `git status --short --branch; git log --oneline -20;
+git stash list; git worktree list` (prune stale worktrees), and
+`du -sh .AGENTTRASH` for staged deletions awaiting purge approval. This is a
+**shared checkout with parallel instances**: uncommitted peer WIP and
+stash-list churn are expected. Never commit or drop a peer's files. To
+deploy with a dirty tree: `git stash push -u -m "..." -- <peer paths>`,
+deploy, `git stash pop`. `deploy.sh` refuses a dirty tree.
+
+**5. Prometheus — expected-down targets (not outages).** Query
+`/api/v1?query=up` from inside the container; the structurally red-by-
+construction jobs (`comfyui` HTML scrape, `services` up/down-as-scrape,
+`gitea` exporter off) are documented in [[Observability]] — anything else
+down, especially a `node-*` or LLM job, is a real gap. The recurring trap: a
+same-host `net=host` bind (node-exporter :9100, llama-server :7770/:7771) is
+a *host bind*, not a published port, so the bridge-networked Prometheus
+container cannot reach it at the host's LAN IP — netavark DNATs published
+ports only. Scrape same-host services via
+`host.containers.internal:<port>`; verify the fix from inside the container
+before editing: `podman exec prometheus wget -qO-
+http://host.containers.internal:<port>/metrics`.
+
+**6. Cross-check docs vs live.** Confirm the AGENTS.md host↔IP↔hardware↔
+stack mapping against step 1. The LLM endpoint table and `models.yml` are
+probed live and stay current; the prose host labels are the part that goes
+stale after an IP move. Report discrepancies; do not "fix" prose from a
+single crossed probe.
+
+## Bulk-redact an identity from a docs + config tree
+
+Removing a retired host / machine / project name from a repo that holds both
+documentation and live config. Evidence-first, surgical, reversible.
+
+**1. Inventory before touching anything.** Count hits **per file** (the
+count decides whether this is a 5-minute job or a user-input decision),
+excluding logs/`*.db`/runtime state, and capture `git status --short` first
+— you must know which files carry in-flight peer WIP so you never stage
+them.
+
+**2. Classify every hit:**
+
+| Bucket | Test | Action |
+|---|---|---|
+| **Narrative** | prose describing the thing | Reword to the class ("the workstation", "a client box") |
+| **Load-bearing** | the identifier IS a lookup key, hostname match, or config value | **Keep.** Prove it: read the code that consumes it |
+| **Evidence** | quoted logs, transcripts, reports | Keep timestamps/lines; de-identify only the host field |
+
+Load-bearing examples that look like prose but are not: `install.sh` doing
+`PKGS=("common.pkgs" "$HOST.pkgs")` with `HOST="$(hostname)"` (the manifest
+*filename* is the key); a `--host <name>` case list; a Wi-Fi SSID. Hardware
+model strings, driver versions, PCI addresses are subject matter, not
+identity. State the boundary in the commit message.
+
+**3. Subdirectories: distill, don't rename or keep.** For a per-host doc
+tree (`docs/system/<host>/`), ask before acting — rename/trash/distill are
+genuinely different. When distilling: read every page, sort into
+still-true-and-machine-independent (fold into a general page) vs
+specific-to-the-retired-box (let go), then `mv` the directory to
+`.AGENTTRASH/YYYYMMDD-topic/` and repoint every index. Wikilinks resolve by
+filename anywhere in the tree, so deleting a page can break a link in a
+file you never grepped — run the link checker.
+
+**4. Execute with count assertions, never blind sed.** Per-file mapping of
+`(path, old_exact_string, new_string, expected_count)`; abort the entry if
+the count differs; exact strings, not regex. **Do not hand this to a
+subagent** — observed: dropped list markers so bullets stopped rendering, a
+numbered list with two `3.` items, "corrected" paths to nonexistent
+directories, a deleted sentence where a reword was required. If a subagent
+already did it: `git checkout --` its touched paths (excluding the WIP
+files from step 1) and redo it.
+
+**5. Audit:** `grep -rniE '<name>|<its-ip>' <scope>` (expect nothing), then
+`git diff -U0 -- <scope> | grep -E '^\+[^+]'` and **read every added line**
+— the diff is the step that catches damage; the grep only proves a string
+is gone. Then the repo's own linter.
+
+**6. Commit scoped** — only your own paths, never `git add -A`. If the
+checkout IS the live deployment, committing on the current branch is usually
+correct; a branch switch can swap live config out from under a running
+process. Push/PR only when asked.
+
+**7. Report the residue honestly** — what still carries the name and why:
+excluded scopes, load-bearing config, git history, and the cost of rewriting
+history (force-push + re-clones). "Purged from the working tree" is a true
+claim; "purged" alone is not.
+
+## Related
+
+- [[Observability]] — the monitored stack
+- [[Stale Signal Audit]] — the down-target decision table
+- [[Observability Provisioning]] — adding scrape targets and rules
+- [[Homepage Status Probes]] — the status page that renders probe states
+- [[Deploy State Triage]] — the deploy plumbing the audits exercise

serverAppConfigs-docs/Operations/Fleet Ops Playbook.md

@@ -0,0 +1,272 @@
+---
+type: reference
+title: Fleet Ops Playbook
+description: Incident-shaped fleet procedures — LLM endpoint down with an abandoned GPU holder, scratch vLLM boot windows (stop-first law), iterating on a fleet service before commit, incident-window reconstruction, and waking a sleeping host.
+tags: [operations, incidents, gpu, systemd, wol, forensics]
+timestamp: 2026-10-05T00:00:00-07:00
+---
+
+# Fleet Ops Playbook
+
+## Endpoint down, connection refused — GPU holder reclaim
+
+Use when `:7770`/`:7771` gives **connection refused** (not a timeout) and the
+service was not crashed. Applies to `frank-medium` (dankserver, root unit),
+`frank-small` (dankclaw, root unit), `frank-large` (dankclaw, rootless
+Quadlet).
+
+**1. Classify the stop before touching anything**
+
+```bash
+ssh dankserver "systemctl is-enabled frank-medium; systemctl is-active frank-medium; \
+  systemctl show frank-medium -p LoadState,UnitFileState,ActiveState,SubState,NRestarts"
+ls -l /etc/systemd/system/<svc>.service      # symlink to repo = unmasked; -> /dev/null = masked
+```
+
+`inactive` + `enabled` + unmasked + `NRestarts=0` → **intentional stop**,
+never restored. No crash; do not look for segfaults. `NRestarts>0` /
+`activating` → different path; read the journal first.
+
+**2. Are the GPUs actually free?** A check, never an assumption:
+
+```bash
+rocm-smi --showmeminfo vram | grep Used      # idle baseline ~60 MiB/card
+ps -eo pid,comm,etime,args | grep -Ei 'llama|bench' | grep -v grep
+```
+
+Cards idle → `systemctl start <svc>` and verify (step 5). Cards busy with no
+prod unit → get the holder, and compare its bytes against the service's
+requirement before concluding a start would even fit (frank-medium 35:30
+needs ~30.8 GiB ROCm0 / ~28.9 GiB ROCm1; a stray holding 15+17 GiB blocks the
+restart on allocation, not config).
+
+**3. Identify the holder and prove it is abandoned**
+
+```bash
+ps -o pid,ppid,user,lstart,etime,comm -p <pid>
+sudo readlink /proc/<pid>/cwd /proc/<pid>/exe
+tr '\0' ' ' < /proc/<pid>/cmdline                     # port, ctx, split
+sudo ls -l /proc/<pid>/fd                             # where stdout/stderr go
+sudo ss -tnp | grep <port>
+awk '{print $14+$15}' /proc/<pid>/stat; sleep 10; awk '{print $14+$15}' /proc/<pid>/stat
+```
+
+Abandoned **only if all** hold: utime+stime **identical** across the 10 s
+sample (frozen CPU, not just low); **zero** established connections; bind
+scope `127.0.0.1` (loopback scratch, invisible to endpoint monitors); log
+file (fd 1/2 target) last written long ago, last lines showing success not a
+hang. Active CPU or live connections → someone's running session. Stop and
+ask.
+
+**4. Reclaim**
+
+```bash
+cp -a <logfile> <logfile>.preserved-YYYYMMDD   # evidence outlives the kill
+kill <pid>                                     # SIGTERM first; exits <1 s
+rocm-smi --showmeminfo vram | grep Used        # expect ~60 MiB baseline
+```
+
+Never `pkill -f` a pattern present in your own command line.
+
+**5. Restore and verify — health 200 alone is NOT proof**
+
+```bash
+sudo systemctl start <svc>
+for i in $(seq 1 60); do curl -s -m 5 -o /dev/null -w '%{http_code}\n' \
+  http://127.0.0.1:<port>/health; sleep 10; done   # 503 while loading -> 200
+sudo tr '\0' ' ' < /proc/<MainPID>/cmdline | grep -oE '\-\-(tensor-split|split-mode|ctx-size|port|spec-type|parallel|flash-attn)'
+```
+
+Prove with a **real completion** from off-host — content, reasoning, spec
+counters:
+
+```bash
+curl -sS -m 180 http://<host>:<port>/v1/chat/completions -H 'Content-Type: application/json' \
+  -d '{"model":"<alias>","messages":[{"role":"user","content":"Reply with exactly: OK"}],"max_tokens":64,"temperature":0}'
+# choices[0].message.content / .reasoning_content / timings.draft_n, draft_n_accepted
+```
+
+Record the runtime fingerprint (`<bin> --version` → `b<build>, commit <sha>`)
+and confirm it is the pinned runtime. Append a dated entry to
+`serverAppConfigs-docs/log.md`, run `./tests/run.sh`, commit, push (origin is
+the local Gitea on Dankclaw — push allowed).
+
+**Gotchas seen here:** dankserver's login shell is ZSH — never start an
+unquoted word with `=` in an ssh one-liner (`echo ===` aborts the chain). A
+freshly restarted endpoint under live agent-queue backlog may return empty
+bodies/timeouts for minutes while `/health` is 200 — re-probe with a long
+timeout. First request after a halogen (`frank-large`) restart can go silent
+45–60 s and self-recover; do not restart over it. Rootless `frank-large`:
+read logs **without** sudo (per-user storage). If the restore never happened,
+the sudo journal names who and when: `journalctl -b | grep sudo`.
+
+## Scratch vLLM boot on a fleet GPU host — stop-first law
+
+Any scratch/experimental vLLM container on a host that also runs a production
+LLM service (`frank-primary`, `frank-large`, `frank-small`) MUST stop the
+production unit first, and the window script must verify it. Two independent
+failure modes if skipped:
+
+1. **Expert-offload planner dies on host RAM.**
+   `vllm/model_executor/offloader/expert_plan.py` reads `MemAvailable` at
+   plan time. A production rank worker holds tens of GiB of unevictable
+   pinned expert rows (frank-primary: ~76 GiB × 2 ranks), so a beside-prod
+   boot sees ~86 GiB instead of ~239 and dies at plan time:
+   `ExpertPlanError: host holds 5113 of 12096 swappable experts per rank`.
+   Looks like a planner bug; is pure RAM contention. Signature: container
+   exits within ~60 s, after `Resolved architecture: …` but before worker
+   init; `free -g` showing `available ≈ 86` while prod runs is the smoking
+   gun.
+2. **GPU ownership.** Two vLLM containers cannot share the GPUs regardless
+   (the fleet's benchmarking law). Scratch ports (`:7771`/`:7773`) are fine
+   only once the prod unit is down.
+
+Window skeleton (dankserver):
+
+```bash
+sudo systemctl stop frank-primary.service
+sudo systemctl mask --runtime frank-primary.service  # relinked unit: plain mask refuses; runtime mask self-clears at reboot
+sleep 3
+podman ps --format "{{.Names}}" | grep -q frank-primary-glm && { echo "prod still running - abort"; exit 1; }
+# ... launch scratch container, watch logs ...
+podman rm -f <scratch>                                # stuck vLLM ignores SIGTERM
+sudo systemctl unmask --runtime frank-primary.service
+sudo systemctl reset-failed frank-primary.service
+sudo systemctl start frank-primary.service
+# restore oracle is a CONJUNCTION: health 200 AND is-active active AND is-enabled enabled AND no /run/systemd/system mask symlink
+```
+
+The deploy restores the unit after bench windows, so stop is sufficient
+there; `mask --runtime` is for scratch boots that must survive a stray
+restart. Always end a window by re-checking the conjunction before
+reporting — if a restore after failure doesn't happen automatically, prod
+stays DOWN. See the repo's benchmarking policy (AGENTS.md) and
+[[LLM Bench Windows and Measurement]].
+
+## Iterating on a fleet service before commit
+
+The repo's contract is git → `deploy.sh` → host, but live testing needs the
+new file on the host first, and the naive way (copy into the host checkout)
+blocks the very deploy that would install it. The loop:
+
+1. Edit locally; run `./tests/run.sh`.
+2. Ship working files **for testing**:
+   `cat local/file | ssh <host> 'cat > /home/dank/serverAppConfigs/<path>'`
+   (then `chmod +x` if executed — `cat >`/scp drop the exec bit and a
+   container entrypoint fails status 126, "found but not executable").
+3. Test against the live service; iterate.
+4. **Before committing**, prove the host copy is byte-identical to what you
+   are about to commit (`sha256sum` both ends) — otherwise step 6 loses work.
+5. Commit and push.
+6. Clean the host checkout so the deploy can pull: untracked hand copies →
+   `mv` to `/tmp/handcopy/`; modified tracked files → `git checkout -- <paths>`
+   (safe only because of step 4); confirm `git status --short` prints nothing.
+7. `./scripts/deploy.sh <target> --service <svc>`.
+
+Traps, in the order they bite: the **pull aborts on hand copies** ("untracked
+working tree files would be overwritten by merge" — exactly the file you
+shipped); `git checkout --` silently eats uncommitted work without step 4; a
+runtime may **mount files straight out of the checkout** (`dankserver/frank-medium/**`,
+`shared/grafana/provisioning/**`) so a hand copy takes effect at next
+container restart with no deploy; root-owned files from a container write
+need `chown` back before git operations; and **a restart is not a
+verification** — verify the *specific* claim (new code path's own output,
+file checksum on the host, the API serving the resource).
+
+Why not keep a scratch copy on the host? The deployed artifact must be the
+committed one. Probes may live anywhere; runtime state lives in git.
+
+## Incident window reconstruction
+
+Use when the question is **"what was running when it broke?"** and the answer
+must come from artifacts, not recollection. Three independent clocks,
+cross-checked:
+
+1. **Per-run duration** — scripts wrap commands in `time` → `real 4m19.129s`.
+2. **Log mtime** — last write of the log.
+3. **Journal first-failure instant** — first error line from the affected
+   service.
+
+`start ≈ mtime − real` for the last run in a log; each run nests inside the
+previous one's window; the run whose `[start, end]` contains the journal's
+first-failure instant is the answer.
+
+```bash
+ssh <host> 'ls -l --time-style=+%m-%d_%H:%M:%S <logdir>/*.log | awk "{print \$6, \$7}"'
+ssh <host> 'grep -ah "^real" <logdir>/*.log'
+ssh <host> 'grep -aE "^####|^\+ /path/to/binary" <logdir>/<name>.log | sed "s/\(.\{130\}\)/\1\n        /g"'
+ssh <host> 'journalctl --user --since "<date> <HH:MM>" | grep -a "invalid internal status" | head -2'
+```
+
+Traps (all hit for real): `/tmp` does not survive a reboot — launch
+experiment scripts from a persistent path (`/home/dank/models/logs/<tag>.log`
+survived while `/tmp/x4*.sh` died); a log's mtime is not proof a run finished
+(truncation with the fd still held keeps growing); long command lines are
+display-truncated — wrap before reading or you report parameters never used;
+off-by-one filenames (`x4b.log` vs `x4b-experiment.log`) silently skip
+evidence — list the directory and match exactly; **overlap is a hypothesis,
+not an observation** unless a sampler proves it; do not invent the cause —
+reconstruct the *window* and keep the *mechanism* separate (`journalctl -k |
+grep -i oom`); publish "trigger not established" when the proof is absent.
+
+Fleet-specific: container healthchecks shell out to the **podman CLI**, so
+the first `invalid internal status … could not find any running process` in
+the user journal is an early, per-minute indicator of rootless-podman state
+death — it timestamps the incident to within one healthcheck interval (how
+the 18:58:30 Dankclaw break was pinned to a bench still loading).
+
+Worked example (2026-09-16, Dankclaw): log mtimes x4e-run 18:58:29 /
+x4f-run 18:59:56; durations E1 `real 4m53.946s`, F `real 5m24.785s` (exit 143
+= killed); journal first failure 18:58:30 ⇒ **F** was in flight (launched
+18:54:31, inside its window).
+
+## Wake-on-LAN for a sleeping fleet host
+
+A fleet host is asleep (100% ping loss; `ip neigh show <ip>` shows STALE or
+FAILED) and you need it up — typically the FrankBook-Pro Mac mini
+(192.168.0.112, **MAC `de:6b:6d:c1:52:23`**, sleeps and breaks the e2e
+`11-admin` spec).
+
+- **Find the MAC**: `ip neigh show <ip>` — even a STALE entry carries it;
+  fallback Pi-hole/DHCP leases or fleet docs.
+- **Send the packet**: `6×0xFF + 16×MAC` = 102 bytes, UDP to port **9** (and
+  7), broadcast. The dev workstation has **two interfaces on 192.168.0.0/24**
+  (Wi-Fi + wired USB-Ethernet, default route metric 100) — a plain
+  `sendto("255.255.255.255", 9)` leaves only the default-route interface.
+  Bind explicitly to each interface's local IP and target the subnet
+  broadcast `192.168.0.255` as well as `255.255.255.255`; burst ~15 packets
+  per interface ~0.2 s apart. `wakeonlan` is not installed; use a raw UDP
+  socket with `SO_BROADCAST=1`:
+
+```python
+import socket, time
+MAC = "de:6b:6d:c1:52:23"
+mac = bytes(int(x,16) for x in MAC.split(":"))
+pkt = b"\xff"*6 + mac*16          # 102 bytes
+for local, dest in [("192.168.0.83","192.168.0.255"),("192.168.0.104","192.168.0.255"),
+                    ("192.168.0.83","255.255.255.255"),("192.168.0.104","255.255.255.255")]:
+    s = socket.socket(socket.AF_INET, socket.SOCK_DGRAM)
+    s.setsockopt(socket.SOL_SOCKET, socket.SO_BROADCAST, 1)
+    s.bind((local, 0))
+    for _ in range(15):
+        s.sendto(pkt,(dest,9)); s.sendto(pkt,(dest,7)); time.sleep(0.2)
+    s.close()
+```
+
+- **Poll for the wake**: a Mac mini takes **30–90 s**; `ping -c 1 -W 1` every
+  5 s for ~2–3 min before concluding failure.
+- **If it does not wake** (the common case): macOS honors WoL only when BOTH
+  are enabled on the mini — **System Settings → Battery → Options → "Wake
+  for network access"** = ON, and the active port's **"Enable Wake for LAN"**
+  (Network → [interface] → Details). If either is off, no packet from any
+  source wakes it, and you cannot change it while asleep. A full power-off
+  (smart plug cycled) ignores WoL entirely. Ask the user to toggle the
+  setting, then re-fire.
+
+## Related
+
+- [[LLM Bench Windows and Measurement]] — the bench windows this playbook arms
+- [[Remote Job Runner]] — supervising the long jobs that create these windows
+- [[Container Host Reachability]] — the pasta port gate at container create
+- [[Quadlet Lifecycle]] — the unit layer behind stop/mask/start
+- [[Model Swap and Rollback]] — swap-time incidents and rollbacks

serverAppConfigs-docs/Operations/Payload and Disk Reclaim.md

@@ -0,0 +1,204 @@
+---
+type: reference
+title: Payload and Disk Reclaim
+description: Standing procedure for disk pressure on a fleet host — measure all buckets first, classify against the runbook index, prove nothing live references the payload, stage to trash with fingerprints, purge only on approval; plus the purge-and-refresh protocol with fetch verification.
+tags: [operations, disk, podman, payloads, huggingface, trash]
+timestamp: 2026-10-05T00:00:00-07:00
+---
+
+# Payload and Disk Reclaim
+
+Standing procedure for "clean up old models" / "disk is full" on a
+serverAppConfigs host. Order matters: **measure, prove nothing live uses it,
+stage, ask.** Staging frees nothing until the user approves the purge — say so
+up front or the request reads as unfulfilled.
+
+## 1. Measure the whole filesystem, not the category asked about
+
+```sh
+du -sh /home/dank/models /home/dank/.local /home/dank/.cache \
+       /home/dank/comfyui-data /home/dank/GitPackages 2>/dev/null
+df -h /home/dank | tail -1
+podman system df          # reclaimable image space
+du -sh ~/.local/share/containers   # where rootless storage physically is
+```
+
+The answer is usually NOT the category the user named. On Dankclaw
+(2026-09-28) retired models were 510G, but podman images were **279G
+reclaimable (95%)** and uv+pip caches **73G** — 352G of regenerable garbage,
+bigger than the models. Lead with that finding; it changes the decision.
+
+Check for sparse files before claiming a saving:
+
+```sh
+stat -c 'apparent=%s blocks=%b' <file>; du -sh --apparent-size <file>; du -sh <file>
+```
+
+A 3.3G "log" (`logs/x4e-run.log`) was **4K real** (8 blocks × 512) — deleting
+it frees nothing. `du` vs `du --apparent-size` finds these instantly.
+
+## 2. Classify against the runbook index
+
+Every payload directory maps to a runtime in the AGENTS.md retired-runbook
+table. Three buckets:
+
+- **live** — referenced by a running service's env/unit.
+- **rollback** — named as a rollback target in a runbook. ASK before
+  touching: they are the difference between a one-minute flip and a 30G
+  re-download.
+- **retired** — maps to a retired runtime. Safe candidates.
+
+Trap: a directory can hold **both** a dead target and a live drafter
+(frank-large's DFlash2 drafter lives beside the previous target). Move the
+single file, never the directory, and say so in the manifest.
+
+## 3. The live-reference check — before moving anything
+
+```sh
+for d in <candidate dirs>; do
+  grep -rl "$d" --include="*.sh" --include="*.env" --include="*.service" \
+            --include="*.container" --include="*.yaml" . | grep -v '^./old/'
+done
+```
+
+Then **read every hit**; do not count them. Most are benign (comments,
+retired fetch-script destinations, `retire_*_quadlet` helpers that move a
+quadlet not a model) — but on Dankclaw this check found a live bug:
+`serve-frank-small.sh`'s `: "${MODEL_DIR:=...}"` fallback still pointed at a
+payload retired a week earlier. The env file always overrode it, so it was
+invisible; a missing env file would have produced a "missing asset" naming a
+model that no longer exists.
+
+**Durable lesson:** `:`-defaults in a serve script are never exercised in
+production, so they rot silently. When reclaiming a payload, grep serve
+scripts too — the fallback is a live reference in all but name.
+
+Also check service **env files** (`scripts/*.env`) for `MODEL_DIR`/`MODEL`
+paths, not just unit `Volume=` lines: a payload can be mounted by path and
+named by env. `models/logs` and small tool dirs are not cache — logs are
+evidence, keep them.
+
+## 4. Stage, never delete
+
+```sh
+STAGE=/home/dank/.AGENTTRASH/$(date +%Y%m%d)-<topic>
+mkdir -p "$STAGE"
+mv <path> "$STAGE"/
+```
+
+Same filesystem ⇒ instant and recoverable — **and reclaims no space**; say so
+rather than implying space was freed. Write `MANIFEST.md` into the stage dir:
+per item — path, size, owning runtime, when retired, the `mv` command to
+restore. Record what you **deliberately kept** and why; that is what a future
+reader needs. Always check for pre-existing staged trash
+(`du -sh ~/.AGENTTRASH/*`): 118G from a 2026-09-21 cleanup sat un-purged;
+report the running total, not just the new batch. Root-owned leftovers need
+`sudo`, and check `swapon --show` first (a trashed swapfile must not be
+active).
+
+## 5. Verify, then report — and do not purge
+
+```sh
+systemctl is-active frank-large frank-small
+curl -s -o /dev/null -w '%{http_code}\n' http://127.0.0.1:7770/health
+```
+
+Report: staged items and sizes, what was kept and why, what is **still not
+freed**, remaining candidates untouched. Purge only on explicit approval.
+
+## Repo-flow gotcha: do NOT push before you deploy
+
+`scripts/deploy.sh` sets `BASE_REVISION=$(git rev-parse origin/main)`. If you
+pushed separately, the diff is empty and the planner reports "changed
+services: (none)" — looks like a planner bug, is not one. The tool is
+designed for the **unpushed** state: edit → test → `deploy.sh` (it pushes,
+then dispatches). If you already pushed, pull on the host instead, and reason
+about whether a restart is actually owed. Any serve-script/env change needs
+`./tests/run.sh` before commit; a `:`-default change is usually **inert**
+(env overrides it) and owes no restart — but say so explicitly.
+
+# Purge and refresh protocol
+
+When the task is clearing/reclaiming/**refreshing** model caches on a serving
+host, add these steps.
+
+## Fingerprint before deleting
+
+`ls -l --block-size=1 <dir>`; for LLM payloads compare each file against the
+Hub's declared size. Fingerprints are the only way to answer "did the refresh
+change anything" afterwards — **a refresh is also an update check**: if
+upstream never changed, the bytes come back identical, which is itself the
+finding. Sidecars are versioned **in place**: equal main-checkpoint sizes do
+not prove the repo is unchanged, and a stale sidecar is invisible from its
+name (observed: `overlay.hgn` 2,477,677,120 → 2,572,466,560 while the main
+checkpoint returned byte-identical). If a file is irreplaceable upstream
+(delisted revision, gated repo), say so before deleting.
+
+## Confirm re-fetchability first
+
+Query the Hub anonymously; a gated repo 403s even with the host's token:
+
+```bash
+curl -s "https://huggingface.co/api/models/<repo>/tree/main?recursive=true&blobs=true" \
+  | python3 -c "import json,sys; d=json.load(sys.stdin); print(len(d),'entries'); [print(e.get('size'),e['path']) for e in d[:5]]"
+```
+
+An error/empty result means it is NOT safe to delete. Filter by **quant
+name**, never `endswith('.gguf')` — mradermacher ships large quants as
+`.partNofM` fragments that filter hides.
+
+## Refresh per engine — know the recovery path BEFORE deleting
+
+- **frank-large (halogen)**: the engine fetches its own payload via
+  `HALOGEN_DOWNLOAD` when the checkpoint is absent. Clear the directory but
+  leave it **existing and empty**, then start the unit. **Bind-mount trap:**
+  podman refuses a mount whose source is missing
+  (`Error: statfs /home/dank/models/halogen: no such file or directory`) and
+  the unit loops on `Restart=on-failure` — `mkdir -p` the empty dir. ~124 GiB
+  at 130–240 MB/s ≈ 10–20 min, then load.
+- **llama.cpp services**: no self-fetch; the serve script refuses to start
+  without weights. Pull yourself and verify:
+
+```bash
+curl -fL -C - --retry 10 -o "$DEST/<file>" \
+  "https://huggingface.co/<repo>/resolve/main/<file>"   # -f is mandatory
+sha256sum "$DEST"/*; head -c 4 "$DEST"/*.gguf   # compare to LFS oid + GGUF magic
+```
+
+`curl` exits 0 on HTTP 401/403 and writes the error body as the artifact — a
+gated repo once produced three 153-byte shards with identical sha256s and a
+cheerful DONE line for 104 GiB. Always `-fL -C -`, compare byte counts to the
+Hub's declared sizes, compare sha256 to the LFS oid. Assemble `.partNofM`
+fragments with `cat part1 part2 part3 > name.gguf` in numeric order (per-part
+hashes verify the whole — concatenation is deterministic). Public repos need
+no token: omit `-H "Authorization: Bearer …"` from argv entirely (it is
+world-readable in `/proc/<pid>/cmdline`).
+
+## Restore, then verify behaviour — not health alone
+
+Put the quadlet back (`mv` from trash + `systemctl --user daemon-reload`),
+`reset-failed`, `start` (root units need `sudo`). Watch the engine's own
+fetch/load lines, then `/health`, `/v1/models`, and a **real completion** on
+every restored endpoint. Read the engine's memory ledger (kv pool / weights /
+blocks left) rather than assuming the config still fits. Sweep the vhosts too
+(`www`, `grafana`, `prometheus`, `hermes`, `marreta`, `gitea`, `loki` via
+`curl -sk --resolve <host>:443:192.168.0.143`) — a purge touches state the
+whole host shares.
+
+## Things that bite
+
+- **Do not run a 100 GB+ re-fetch concurrently with a big iGPU benchmark** —
+  the A* run commits ~116 GiB and leaves ~10 GiB available; cache churn risks
+  the OOM/state failures this fleet has already seen.
+- **Reboot is the repair of last resort** if rootless podman's pause process
+  dies mid-work (`invalid internal status … could not find any running
+  process`); `podman system migrate` panics in 5.7.0.
+- A same-filesystem trash reclaims no space; root-owned leftovers need
+  `sudo`; never `rm` without explicit permission.
+
+## Related
+
+- [[Model Swap and Rollback]] — the runbook index this classification reads
+- [[Quadlet Lifecycle]] — restoring the unit after a payload refresh
+- [[Remote Job Runner]] — supervising long re-fetches
+- [[Deploy State Triage]] — the push-before-deploy trap in context

serverAppConfigs-docs/Operations/Remote Job Runner.md

@@ -0,0 +1,135 @@
+---
+type: reference
+title: Remote Job Runner
+description: Detached long-job execution on fleet hosts with sentinel files (job.sh) — why ssh tail-watchers fail structurally, the script, its invariants, verification, and reporting discipline.
+tags: [operations, ssh, jobs, sentinel, detached]
+timestamp: 2026-10-05T00:00:00-07:00
+---
+
+# Remote Job Runner
+
+Use for any long-running job on a remote host (benchmarks, model loads,
+builds, sweeps) whose result must survive the session. Also whenever you catch
+yourself writing `ssh host 'tail -f log | sed …'` as a watcher.
+
+## Why the obvious approach fails structurally
+
+`ssh host 'tail -f log | grep/sed …'` as a background watcher fails in ways
+that all LOOK like "the job died":
+
+- the session broker **aborts long-lived ssh** — the watcher vanishes and the
+  result is lost;
+- a `ready` regex can match the log's own **header** on the first line — the
+  supervisor reports success instantly while the job is still running;
+- a `sed -n '/PAT/,$p'` filter **exits at the first match**, ending the pipe;
+- output goes into the ssh stream, so an aborted session loses everything.
+
+## The mechanism: detach on the HOST, poll a sentinel file
+
+Completion is a **`.done` sentinel touched last**, not a log pattern. Status
+is then one fast independent command with no long-lived process to lose.
+
+```bash
+#!/usr/bin/env bash
+# job.sh {start|status|wait|out|tail|watch} ...   Jobs live in $JOBS.
+set -uo pipefail
+JOBS=/home/dank/jobs
+mkdir -p "$JOBS"
+
+case "${1:-}" in
+start)
+    NAME="$2"; shift 2
+    [ "${1:-}" = "--" ] && shift
+    rm -f "$JOBS/$NAME.done" "$JOBS/$NAME.rc"
+    # setsid detaches from this session's process group: the critical bit.
+    setsid nohup bash -c '
+        NAME="$1"; shift
+        "$@" > "'"$JOBS"'/$NAME.out" 2>&1
+        echo $? > "'"$JOBS"'/$NAME.rc"
+        touch "'"$JOBS"'/$NAME.done"
+    ' _ "$NAME" "$@" < /dev/null > /dev/null 2>&1 &
+    echo "started $NAME"
+    ;;
+status)
+    # Glob *.out (not *.done) or running jobs are invisible.
+    names="${2:-$(cd "$JOBS" && ls *.out 2>/dev/null | sed 's/\.out$//' | sort)}"
+    for n in $names; do
+        if   [ -f "$JOBS/$n.done" ]; then
+            printf '  %-18s DONE   rc=%s  %s lines\n' "$n" \
+                "$(cat "$JOBS/$n.rc" 2>/dev/null)" \
+                "$(wc -l < "$JOBS/$n.out" | tr -d ' ')"
+        elif [ -f "$JOBS/$n.out" ]; then
+            printf '  %-18s RUN    %s lines so far\n' "$n" "$(wc -l < "$JOBS/$n.out" | tr -d ' ')"
+        else
+            printf '  %-18s PENDING\n' "$n"
+        fi
+    done
+    ;;
+wait)
+    NAME="$2"; LIMIT="${3:-3600}"; waited=0
+    while [ ! -f "$JOBS/$NAME.done" ] && [ "$waited" -lt "$LIMIT" ]; do
+        sleep 5; waited=$((waited+5))
+    done
+    [ -f "$JOBS/$NAME.done" ] \
+        && echo "DONE $NAME rc=$(cat "$JOBS/$NAME.rc" 2>/dev/null)" \
+        || echo "TIMEOUT $NAME after ${waited}s"
+    tail -"${4:-25}" "$JOBS/$NAME.out" 2>/dev/null
+    ;;
+out)  NAME="$2"; tail -"${3:-40}" "$JOBS/$NAME.out" 2>/dev/null ;;
+tail) [ -n "${2:-}" ] && exec tail -f "$JOBS/$2.out" || exec tail -f "$JOBS"/*.out ;;
+watch) while true; do clear; date '+%H:%M:%S'; bash "$0" status; sleep 5; done ;;
+*) echo "usage: job.sh {start|status|wait|out|tail|watch}" >&2; exit 2 ;;
+esac
+```
+
+```bash
+job.sh start NAME -- <command>   # launch detached
+job.sh status [NAME]             # one-shot state
+job.sh wait NAME 900 25          # block, then print the tail
+job.sh out NAME 60               # retrieve any past result
+job.sh tail [NAME]               # live follow
+job.sh watch                     # refreshing status table
+```
+
+## Invariants
+
+- **`.done` is touched LAST**, after the exit code is written — its presence
+  alone means the output file is final.
+- **Glob `*.out` for status**, never `*.done` — globbing `.done` hides running
+  jobs entirely, the exact blindness this fixes.
+- **Capture the exit code** (`rc=7` proved a self-test worked; without it a
+  job can "complete" having failed).
+- **Never put the payload in a line you grep for** — the sentinel is the only
+  completion signal.
+- **Never `pkill -f <pattern>`** where the pattern also matches your own ssh
+  command line — it kills the session. Distinctive names or `pkill -x`.
+- **Write helper scripts to a file and `scp` them.** Triple-nested
+  `ssh 'bash -c "python3 -c \"…\""'` quoting broke repeatedly (unterminated
+  quotes; `grep -c` returning `"0\n0"` because `|| echo 0` appended a line);
+  every failure disappeared once the script was a file.
+
+## Verify before trusting it (3am law)
+
+```bash
+job.sh start selftest -- bash -c 'echo one; sleep 2; echo two; exit 7'
+job.sh start slowtest -- bash -c 'echo start; sleep 30; echo end'
+sleep 4; job.sh status      # selftest DONE rc=7; slowtest RUN
+```
+
+Both paths, including the failure case, and the rc round-trip — a runner that
+only proves the happy path is not trustworthy.
+
+## Reporting discipline
+
+- Give the user a real terminal when the run is long:
+  `alacritty --title "bench - dankserver" -e bash -c 'ssh -o
+  ServerAliveInterval=30 -t dank@192.168.0.235 "bash /home/dank/job.sh tail"' &`
+- State the MECHANISM, not vibes: the job is durable on the host;
+  `job.sh out NAME` retrieves it later. Do not narrate "still loading"
+  repeatedly — check once, act, report the resolved result.
+- When the user asks for status, one fast command, not a narration.
+
+## Related
+
+- [[LLM Bench Windows and Measurement]] — the sweeps this runner executes
+- [[Deploy State Triage]] — the two-process deploy this lesson generalizes

serverAppConfigs-docs/Operations/SSH and Remote Access.md

@@ -0,0 +1,130 @@
+---
+type: reference
+title: SSH and Remote Access
+description: Passwordless SSH bootstrap without sshpass/expect, remote execution against fish login shells, and diagnosing a frozen GUI from an SSH session.
+tags: [ssh, remote, fish, gui, diagnosis]
+timestamp: 2026-10-05T00:00:00-07:00
+---
+
+# SSH and Remote Access
+
+## Passwordless SSH bootstrap without sshpass/expect
+
+Use when you must install your pubkey on remote hosts but `sshpass` and
+`expect` are not installed and you only have a password. Modern OpenSSH
+(≥ 8.4) can drive the password prompt from a helper program, so the password
+flows through the **process env only** — never a file, never argv.
+
+1. **Generate a key** (if none):
+   ```bash
+   ssh-keygen -t ed25519 -f ~/.ssh/id_ed25519 -N "" -C "host-deploy-$(date +%Y%m%d)"
+   ```
+2. **Create an askpass helper** that echoes the password from an env var (the
+   helper itself stores NO secret):
+   ```bash
+   printf '#!/bin/sh\necho "$DEPLOY_PW"\n' > /tmp/askpass.sh
+   chmod 700 /tmp/askpass.sh
+   ```
+3. **Copy the pubkey**, forcing the askpass path and detaching from the TTY:
+   ```bash
+   env DEPLOY_PW='<password>' \
+       SSH_ASKPASS=/tmp/askpass.sh \
+       SSH_ASKPASS_REQUIRE=force \
+       setsid ssh-copy-id -o StrictHostKeyChecking=accept-new -o ConnectTimeout=10 user@host
+   ```
+   Password via the transient `env` argument only.
+4. **Add host aliases** in `~/.ssh/config` so deploy scripts resolve names
+   (`HostName`, `User`, `IdentityFile`, `IdentitiesOnly yes`).
+5. **Verify** with `ssh -o BatchMode=yes myhost 'echo OK; hostname'` —
+   BatchMode fails fast if a password would be required; a clean success
+   proves the key works.
+
+Gotchas: without `SSH_ASKPASS_REQUIRE=force`, ssh ignores SSH_ASKPASS when a
+TTY is present and prompts interactively (hangs in a non-interactive tool);
+`setsid` detaches the controlling terminal — required on older OpenSSH even
+with `force`; the helper must be executable (`chmod 700`) and print ONLY the
+password; `IdentitiesOnly yes` stops ssh offering other agent keys that could
+be rejected first ("Too many authentication failures"). Passwordless **sudo**
+on the target is a separate concern (NOPASSWD / `FRANK_SUDO_PW`).
+
+## Remote execution on a fish login shell
+
+Running remote commands or shipping files over SSH to a host whose login
+shell is **fish** (blankpad: user `blank`, reachable only as mDNS FQDN
+`blankpad.local` / 192.168.0.83 — the bare hostname does not resolve from the
+workstation).
+
+1. **Auth**: `SSHPASS='<pw>' sshpass -e ssh -o
+   PreferredAuthentications=password -o PubkeyAuthentication=no user@host '…'`
+   — password via env, never argv (ps-visible).
+2. **Remote exec**: `ssh host 'cmd'` pipes the command string to the login
+   shell (fish). Bashisms — `for … do`, `$?`, heredoc markers — fail at
+   *parse* time on fish (`Missing end to balance this for loop`). Always wrap
+   compound commands: `ssh host 'bash -c "for b in a b c; do command -v $b ||
+   echo MISSING; done"'` (escape inner `$`).
+3. **File shipping**: never combine `cat > /remote/path <<EOF` with a
+   trailing `bash -s` in the same ssh command — `cat` drains stdin, so the
+   interpreter silently runs on empty stdin and prints nothing (a silent
+   no-op that *looks* like success). Ship real bytes via stdin redirect:
+   `ssh host 'cat > /remote/path' < localfile`, then verify byte-truth with
+   `md5sum` on both ends.
+4. **Exit codes**: `$?` in an unquoted remote command is parsed by fish
+   (`$status` there) — wrap in bash or drop the status echo; ssh's own exit
+   code propagates to the caller anyway.
+
+Both traps fired during the 2026-09-09 blankpad dotfiles deploy: the first
+probe died on fish loop parsing; a `cat >` ship silently swallowed the
+heredoc meant for `bash -s`, deploying wrong bytes until md5-compared.
+
+Related: dankserver's login shell is ZSH — never start an unquoted word with
+`=` in an ssh one-liner (`echo ===` aborts the whole chain).
+
+## GUI hang — remote diagnosis over SSH
+
+Use when the desktop is frozen, VT switches do nothing, but SSH still works.
+
+1. **Find the smoking gun** — the kernel hung-task reporter already logged
+   it:
+   ```bash
+   journalctl -b -p err --since "-2 hours" --no-pager | tail -80
+   ```
+   Look for `INFO: task X blocked for more than N seconds` and `blocked on a
+   semaphore likely last held by task Y:PID` (modern kernels name the lock
+   holder outright).
+2. **Full stacks** for victim and holder:
+   ```bash
+   journalctl -b -k --since "<hang minute -5>" --no-pager | tail -120
+   cat /proc/<pid>/status | head -5    # State: R (spinning) vs D (blocked)
+   ```
+   Victim `D` + holder `R` = livelock/deadlock with the lock held;
+   `voluntary_ctxt_switches` in the millions on an `R` kthread = spinning,
+   not progressing.
+3. **Hardware or software?** GPU/CPU still alive, no Xid errors?
+   ```bash
+   nvidia-smi --query-gpu=driver_version,pstate,temperature.gpu,clocks_event_reasons.active --format=csv
+   journalctl -b -k --no-pager | grep -i xid
+   ```
+   Instant answers with no Xid ⇒ driver lock issue, not hardware.
+4. **Trigger window**: block start = first hung message timestamp − reported
+   seconds; then inspect user-session events around it
+   (`journalctl -b --since "<start-2m>" --until "<start+1m>" | grep -v audit`).
+5. **Collateral**: other `D` tasks (`ps -eo pid,stat,comm | awk '$2 ~ /D/'`),
+   USB workers, etc.
+
+**Recoverability verdict:** compositor in `D` state is unkillable and a
+spinning kernel thread holding the lock cannot be poked — **the only exit is
+reboot**: `sync && systemctl reboot --force --force` (double-force skips
+stop-jobs waiting forever on D-state PIDs). If the holder is `D` (sleeping on
+something that can complete, e.g. storage), recovery may be possible by
+clearing that dependency first.
+
+After diagnosing: map stack symbols to driver internals to name the exact
+subsystem, check upstream for the exact symbols + driver version, and propose
+targeted off-switches (`NVreg_RegistryDwords`, module params from
+`/proc/driver/nvidia/params`) rather than blanket driver swaps.
+
+## Related
+
+- [[Fleet Ops Playbook]] — incident procedures that run over these sessions
+- [[Remote Job Runner]] — long jobs that must survive session loss
+- [[Deploy State Triage]] — the deploy flow these credentials serve

serverAppConfigs-docs/Operations/index.md

@@ -0,0 +1,32 @@
+---
+type: index
+description: Operational procedures for the fleet — job runners, disk and payload reclaim, container reachability, audits, SSH craft, and incident playbooks.
+tags: [operations, index]
+timestamp: 2026-10-05T00:00:00-07:00
+---
+
+# Operations
+
+Procedures for operating the fleet. The LLM-specific pages live under
+[LLM](../LLM/index.md); infrastructure and architecture under the
+`../Infrastructure/` and `../Architecture/` folders.
+
+- [[Remote Job Runner]] — detached long-job execution with sentinel files;
+  why ssh tail-watchers fail
+- [[Payload and Disk Reclaim]] — measure-classify-prove-stage-ask disk
+  procedure, plus the purge-and-refresh protocol
+- [[Container Host Reachability]] — netavark/pasta reachability quirks and
+  the pasta port gate
+- [[Fleet Ops Playbook]] — endpoint-down GPU-holder reclaim, scratch-boot
+  stop-first, pre-commit iteration, incident-window reconstruction, WoL
+- [[Fleet Audit and Posture]] — monitoring gaps, vhost probes, security
+  posture, whole-fleet audit, bulk redaction, stash audits
+- [[SSH and Remote Access]] — passwordless bootstrap without sshpass, fish
+  login shells, GUI-hang diagnosis over ssh
+
+## Related
+- [Observability Provisioning](../Infrastructure/Observability%20Provisioning.md)
+  — monitoring config as code
+- [Deploy State Triage](../Architecture/Deploy%20State%20Triage.md) — the
+  deploy plumbing
+- [Quadlet Lifecycle](../Architecture/Quadlet%20Lifecycle.md) — the unit layer

serverAppConfigs-docs/Research/PICO-8 Asset Standards.md

@@ -0,0 +1,62 @@
+---
+type: reference
+title: PICO-8 Asset Standards
+description: Production-grade standards for PICO-8 asset sheets — palette discipline, single perspective and light direction, detail flow and hierarchy, sheet organization, and the cohesion checklist.
+tags: [pico8, game-dev, pixel-art, standards]
+timestamp: 2026-10-05T00:00:00-07:00
+---
+
+# PICO-8 Asset Standards
+
+Ensure every PICO-8 asset (tileset, spritesheet, character sheet, or asset
+sheet) is cohesive, production-grade, and ready to drop into a game without
+rework.
+
+## Palette discipline
+
+- Use the PICO-8 16-color palette exclusively.
+- Establish a dominant color family (2–3 related hues) for the main subject.
+- Reserve accent colors for highlights, danger zones, and focal points only.
+- Maintain consistent color roles across all assets in a sheet (if shadow is
+  dark blue on one tile, it is dark blue everywhere).
+
+## Perspective & lighting
+
+- Choose ONE perspective per sheet (top-down, side-view, or isometric) and
+  never mix.
+- Apply light from a single consistent direction across all tiles and
+  sprites.
+- Shadows follow the same shape logic on every asset (e.g. all cast shadows
+  point bottom-right).
+
+## Detail flow & hierarchy
+
+- Start tiles/props from the palette's dominant colors, then add edge
+  highlights and shading.
+- Keep tile density uniform — avoid one hyper-detailed tile next to empty
+  flat tiles.
+- Props and decorations should read clearly at 8×8 or 16×16 without zooming.
+
+## Sheet organization
+
+- Group related assets together (terrain together, props together,
+  characters together).
+- Leave consistent spacing between assets for clean cropping.
+- Label sections visually through grouping, not text.
+
+## Cohesion checks (before finishing a sheet)
+
+- All colors appear in the PICO-8 palette.
+- Perspective does not flip mid-sheet.
+- Lighting direction is consistent.
+- Edge treatment (dither, outline, anti-aliasing style) matches across
+  assets.
+- No orphan tiles that don't connect to any neighbor.
+
+## Common pitfalls
+
+- Mixing 8-bit and 16-bit detail levels.
+- Using outline style on some tiles but not others.
+- Forgetting the palette constraint — colors outside the 16 PICO-8 colors.
+- Inconsistent tile edge connections (a grass tile that only connects to one
+  dirt variant).

serverAppConfigs-docs/index.md

@@ -115,6 +115,64 @@ macOS mini) as quadlets inside its podman machine — not on Dankserver.
 - [[AI Job Search and Resume Tools]] — self-hosted AI job-search + resume-formatting apps, local-LLM fit, recommended stack
 - [[Most Popular Self-Hosted AI Apps]] — the 15 most popular self-hosted AI apps of Jun–Sep 2026, homelab fit, adoption best practices
 
+## Skill Distillation (2026-10-05)
+
+The managed skill library distilled into the vault, one reference page per
+skill-cluster. See [log.md](log.md) for the wave plan and results.
+
+**Operations:**
+
+- [[Remote Job Runner]] — detached long-job execution with sentinel files
+- [[Payload and Disk Reclaim]] — disk reclaim: measure, classify, prove,
+  stage, ask; plus purge-and-refresh
+- [[Container Host Reachability]] — netavark/pasta reachability quirks and
+  the pasta port gate
+- [[Fleet Ops Playbook]] — endpoint-down GPU-holder reclaim, scratch-boot
+  stop-first, pre-commit iteration, incident-window reconstruction, WoL
+- [[Fleet Audit and Posture]] — monitoring gaps, vhost probes, security
+  posture, whole-fleet audit, bulk redaction
+- [[SSH and Remote Access]] — passwordless bootstrap, fish login shells,
+  GUI-hang diagnosis over ssh
+
+**Architecture:**
+
+- [[Stale Signal Audit]] — classifying green-but-dead and down-but-fine
+  signals
+- [[Quadlet Lifecycle]] — quadlet units, drift, crash-loops, compose
+  cutover
+- [[Deploy State Triage]] — revision markers, pending lockouts, deploy
+  interruptions, shared-checkout hygiene
+
+**Infrastructure:**
+
+- [[Observability Provisioning]] — Prometheus/Grafana config as code
+- [[Homepage Status Probes]] — the status page's checker, vantage rules,
+  and triage
+
+**LLM cross-cutting (full index: [LLM/index.md](LLM/index.md)):**
+
+- [[GGUF Compatibility]] · [[Llama Cpp Serving Sizing]] ·
+  [[Llama Cpp Multi-Device Splits]] · [[PCIe Link and Atomics]] ·
+  [[RDNA4 Container Serving]] · [[RDNA4 MoE Expert Offload VRAM Budget]] ·
+  [[vLLM Boot Forensics and Config Interrogation]] ·
+  [[vLLM Porting and Compile Triage]] · [[LLM Bench Windows and Measurement]] ·
+  [[LLM Performance Diagnosis]] · [[Model Swap and Rollback]] ·
+  [[Tcclaviger Fork Ops]] · [[Heterogeneous Split Triage]] ·
+  [[FreeToken Builds]]
+
+**Agent:**
+
+- [[OMP Harness]] — compaction arithmetic, per-model samplers, preamble
+  audits
+- [[Agent Docs and Edits]] — doc-structure audits, hashline edit recovery
+- [[Command Center Dotfiles]] — the ~/.omp repo-as-deployment
+
+**Desktop / Research:**
+
+- [[Noctalia and Niri]] — the desktop shell, palette recolor, template
+  rendering, rotation pinning, iCX3 RGB
+- [[PICO-8 Asset Standards]] — cohesive PICO-8 asset sheets
+
 ## LLM Fleet
 
 The LLM inference fleet on the two hosts. Full index:
@@ -123,9 +181,14 @@ The LLM inference fleet on the two hosts. Full index:
 ### Deployment and Topology
 
 - [[deployment]] — unified deploy and update process for both servers
+- [[frank-primary-freetoken-pr587]] — the ACTIVE frank-primary runbook
+  (2026-10-05): FreeToken built at PR #587 head `77a7201` on the host into
+  `frank-primary-freetoken:pr587-77a7201`, ONE R9700 with experts in pinned
+  host RAM behind a 0.10 GPU LRU, 2 slots, parsers AUTO; rollback is the
+  tcclaviger shape-G era (tag `frank-primary-vllm-shapeG-20261003`)
 - [[frank-redo-2026-08-17]] — the 2026-08-17 topology change
-- [[frank-primary-r9v-radiance-qwen38-q8]] — the ACTIVE
-  frank-primary runbook: the R9V radiance-vLLM fork (docker-pinned bundle
+- [[frank-primary-r9v-radiance-qwen38-q8]] — RETIRED 2026-09-30 (two swaps
+  back; rollback target): the R9V radiance-vLLM fork (docker-pinned bundle
   `2dac17a2`, docker→podman shim) serving the unsloth
   Qwen3.8-Flash-Next-GGUF Q8_0 (175.30 GiB, 6 shards) + blockfp8 MTP4 +
   BF16 vision on Dankserver's both R9700s — TP=2 tiered experts (80-slot
@@ -213,15 +276,29 @@ The LLM inference fleet on the two hosts. Full index:
   agentionai/Qwen3.8-Flash-Next-ROCmFP4-FAST-imatrix (87.06 GiB,
   Q4_0_ROCMFP4_FAST) + own MTP draft, build-qwen38 (retired 2026-08-30;
   rollback record)
-- [[frank-large-rocmfpx-q8agent-mtp]] — the ACTIVE frank-large runbook
-  since 2026-10-02 (root systemd unit, port 7770): the pinned kingjones30/
-  ROCmFPX fork (scripts/build-rocmfpx.sh) serving kingjones777/
-  Qwen3.8-27B-ROCmFP4-STRIX-MTP-GGUF **Q8_0_ROCMFPX_AGENT** — ftype 115,
-  311 tensors of GGML_TYPE_Q8_0_ROCMFPX, a format no other loader on the
-  host reads — with the repo's Q4_0 MTP head drafting 4 tokens and its BF16
-  projector, one session at the native 262,144 window
-- [[frank-large-gufo-qwen38-27b]] — RETIRED 2026-10-01 → 10-02 (rollback
-  target, payload and `scripts/build-gufo.sh` both on disk): gufo v0.1.1
+- [[frank-large-halogen27b-dflash2]] — the ACTIVE frank-large runbook since
+  2026-10-05 (root systemd unit + podman, port 7770): the digest-pinned
+  ghcr.io/peonist-ai/halogen image serving peonist-ai/halogen-qwen3.8-27b
+  (qwen3.8-27b-p1w4d-d2.hgn, 33.4 GiB — a .hgn that loads in no other
+  engine) with the DFlash2 drafter on ONE slot of 200,700 ctx (slots>1
+  disables speculation — engine law); TEXT-ONLY and censored base, both
+  accepted tradeoffs; no server-side sampler, so the thinking recipe rides
+  the wire
+- [[frank-large-rocmfpx-q6agent-uncensored-mtp]] — RETIRED 2026-10-03 → 10-05
+  (rollback target; payload, fork build and build script on disk): the pinned
+  kingjones30/ROCmFPX fork (scripts/build-rocmfpx.sh) serving kingjones777/
+  Qwen3.8-27B-Uncensored-ROCmFP4-STRIX-MTP-GGUF
+  **Qwen3.8-27B-Uncensored-Q6_0_ROCMFPX_AGENT** — ftype 114, 339 tensors of
+  GGML_TYPE_Q6_0_ROCMFPX + 159 type-103, a format no other loader on the
+  host reads — with the repo's Q4_0 MTP head drafting 4 tokens (27.6-28.1
+  t/s measured, 86.8 % acceptance; the ngram-mod chain measured and
+  rejected) and its BF16 projector, one session at the native 262,144 window
+- [[frank-large-rocmfpx-q8agent-mtp]] — RETIRED 2026-10-02 → 10-03 (rollback
+  target, aligned Q8 payload still on disk): the same fork serving the
+  ALIGNED Q8_0_ROCMFPX_AGENT at 17.0 t/s — the +123 % MTP A/B, the traps
+- [[frank-large-gufo-qwen38-27b]] — RETIRED 2026-10-01 → 10-02 (history;
+  the payload dirs were removed from disk 2026-10-03, `scripts/build-gufo.sh`
+  remains): gufo v0.1.1
   built from source serving Qwen3.8-27B UD-Q4_K_XL + the z-lab DFlash2
   drafter on the gfx1151 iGPU — the two from-source build workarounds, the
   template-hash allowlist, the measured prefill/decode trade
@@ -282,16 +359,21 @@ The LLM inference fleet on the two hosts. Full index:
 - [[ciru-activefpx-promptforge]] — the RETIRED v3 CIRU PromptForge runtime on Dankclaw (superseded 2026-08-24)
 - [[kingjones-strix-mtp]] — the kingjones Qwen3.8-27B ROCmFP4 STRIX MTP model on Dankclaw (active)
 - [[dealign-ornith-crack]] — the retired Ornith-1.5-35B-A3B CRACK model on Dankserver (retired 2026-09-02)
-- [[frank-small-llama-mimo-q5km]] — the ACTIVE frank-small runbook since
-  2026-10-02: a CUDA (sm_86) build of llama.cpp master serving
-  bartowski/MiMo-V2.6-Distill-Qwen-9B-GGUF **Q5_K_M** on Dankclaw's RTX
-  3090, port 7771 — THREE conversations at 180,224 tokens each, the
-  asymmetric KV contract (f16 K + q8_0 V), vision through the repo's mmproj,
-  and the two VRAM measurements that set the pool size; ~100 tok/s decode
-- [[frank-small-llama-mimo-q6k]] — RETIRED 2026-10-01 (same day; replaced
-  by [[frank-small-llama-mimo-q5km]], payload still on disk): the first
-  llama.cpp + GGUF shape for this endpoint, Q6_K at the same
-  2 x 262,144 / f16-K + q8_0-V contract, 20.9 GiB measured
+- [[frank-small-llama-mimo-q6k-2x200704]] — the ACTIVE frank-small runbook
+  since 2026-10-03: llama.cpp master (CUDA sm_86) serving
+  bartowski/MiMo-V2.6-Distill-Qwen-9B-GGUF **Q6_K** on Dankclaw's RTX 3090,
+  port 7771 — TWO conversations at 200,704 tokens each with fully
+  unquantized f16 KV and mmproj vision; 94.4 tok/s decode (the Q6_K rung
+  costs a measured −10 % vs Q5_K_M — weight-bandwidth), batch 4096 / ubatch
+  2048 (+4.6 % prefill, measured 2026-10-05), 2,279 MiB free, and
+  the ≥256 px image-probe trap (tiny images answer "video")
+- [[frank-small-llama-mimo-q5km]] — RETIRED 2026-10-02 → 10-03 (rollback
+  target, payload on disk; owns the measured boot-pair table): same runtime
+  and shape one rung lower — 104.8 tok/s, 3,571 MiB true free
+- [[frank-small-llama-mimo-q6k]] — RETIRED 2026-10-01 (same day; that same
+  Q6_K rung returned ACTIVE 2026-10-03 at the new shape, see
+  [[frank-small-llama-mimo-q6k-2x200704]]): the first llama.cpp + GGUF shape
+  for this endpoint, Q6_K at 2 x 262,144 / f16-K + q8_0-V, 20.9 GiB measured
 - [[frank-small-sglang-mimo-v26]] — RETIRED 2026-10-01 (rollback target):
   SGLang in a self-contained python3.12 venv serving
   letechlead/MiMo-V2.6-Distill-Qwen-9B-INT8-W8A16-AutoRound (qwen3_5
@@ -360,6 +442,14 @@ The LLM inference fleet on the two hosts. Full index:
 - [[qwen3.8-spec-matrix]] — Qwen3.8-27B spec history (superseded by FastMTP)
 - [[FastFlowLM NPU Inference]] — the retired FastFlowLM NPU endpoint
 - [[DFlash DSpark Speculative Decoding]] — retired DFlash/DSpark drafting
+- [[GGUF Compatibility]] — which loaders accept which quant formats; the tensor-type histogram check
+- [[Llama Cpp Multi-Device Splits]] — layer vs tensor split across unlike devices; the -ot pipeline gate
+- [[Llama Cpp Serving Sizing]] — VRAM arithmetic from two measured boots, never a formula
+- [[PCIe Link and Atomics]] — link caps vs loaded rate; RCCL atomics routing on the old board
+- [[RDNA4 Container Serving]] — the serving stack recipe: mounts, ulimits, udev, verify gates
+- [[RDNA4 MoE Expert Offload VRAM Budget]] — arena sizing law for expert offload: free VRAM is estimator slack, not allocatable headroom
+- [[vLLM Boot Forensics and Config Interrogation]] — journal/endpoint forensics for boot failures; argv recovery from a live engine
+- [[vLLM Porting and Compile Triage]] — forward-porting vLLM majors; compile failure forensics
 - [[ling-3.0-flash-trial]] — the Ling-3.0-Flash trial runbook
 - [[ling-3.0-flash-vllm-plan]] — the Ling-3.0-Flash vLLM plan (retired)
 

serverAppConfigs-docs/log.md

@@ -1,5 +1,163 @@
 # Bundle Change Log
 
+
+## 2026-10-05
+- **frank-primary: tcclaviger vLLM fork → FreeToken (PR #587 ROCm) on
+  dankserver.** Built FreeToken at PR head `77a7201` on the host into the
+  local image `frank-primary-freetoken:pr587-77a7201`
+  (digest-pinned rocm/pytorch rocm7.14 torch-2.11.0 base; two build laws
+  bought with failures: the pip-ROCm wheel needs a `libhiprtc.so` dev
+  symlink for the PR's `_cpu_moe` link, and a stale `docker login` at
+  `~/.docker/config.json` poisons podman's ANONYMOUS pulls). Trio rewritten
+  through git (env/serve/unit + seam + deployer): ONE R9700, experts in
+  pinned host RAM behind a 0.10 GPU LRU, PLE 47.7 GiB pinned, KV pool
+  299,968 tokens (multiple-of-64 — the `qsa_sparse` auto backend overrides
+  page size to 64; 300,000 was refused), 2 slots, parsers AUTO (thinking →
+  `reasoning_content`; models.yml flipped accordingly). Container renamed
+  to the unit name (Frank's call; `frank-primary-qwen` → legacy list);
+  deployer image gate → `podman image exists` (local builds carry no
+  RepoDigests) and health gate reads the BODY — FreeToken 200s with
+  `{"status":"loading"}` mid-load, which false-greened a deploy at 6 s
+  (vLLM 503'd; the bare curl was honest there). Verified: FLEET-OK
+  completion, reasoning split (17×24 → "408"), native `tool_calls`, vision
+  "Red" probe, boot 2.3 min cold / 92 s warm, decode 17.1 tok/s
+  single-stream (vLLM was ~30 on two cards; the KV contract caps the
+  single-card LRU; TP=2 exists but the pynccl JIT can't compile for
+  gfx1201 — probed, signature in the runbook). `/metrics` is a 404 — the
+  Dankclaw Prometheus llm-fleet scrape of :7770 reads down until
+  repointed. Rollback: tag `frank-primary-vllm-shapeG-20261003`.
+  Runbook: `serverAppConfigs-docs/LLM/frank-primary-freetoken-pr587.md`.
+- **Skill library distilled into the vault (waves 1–7).** All ~173 managed
+  skills read in full and merged, one polished reference page per
+  skill-cluster:
+  - **LLM cross-cutting (16 pages, `LLM/`):** GGUF Compatibility,
+    Llama Cpp Serving Sizing, Llama Cpp Multi-Device Splits, PCIe Link and
+    Atomics, RDNA4 Container Serving, RDNA4 MoE Expert Offload VRAM Budget,
+    vLLM Boot Forensics and Config Interrogation, vLLM Porting and Compile
+    Triage, LLM Bench Windows and Measurement, LLM Performance Diagnosis,
+    Model Swap and Rollback, Tcclaviger Fork Ops, Heterogeneous Split
+    Triage, FreeToken Builds, plus deltas folded into the retired runbooks
+    (GLM DFlash2 port laws, Lucebox dual-R9700, R9V qualification pipeline,
+    FLM pack registration, ik hybrid envelope, frank-medium split
+    rebalance, frank-primary llama.cpp GGUF preflight, batch-ubatch split
+    section, bench-window D-state/stale-read notes, second-endpoint
+    bring-up in Model Swap).
+  - **Architecture (3):** Stale Signal Audit, Quadlet Lifecycle, Deploy
+    State Triage.
+  - **Operations (6 + index):** Remote Job Runner, Payload and Disk
+    Reclaim, Container Host Reachability, Fleet Ops Playbook, Fleet Audit
+    and Posture, SSH and Remote Access.
+  - **Infrastructure (2 + updates):** Observability Provisioning, Homepage
+    Status Probes; updates to Authentik (guard-chain flip recipe, `ak
+    shell` mechanics, user-provisioning playbook, outpost netns probe),
+    Anubis (guard-chain layer diagnosis), LAN-DNS (per-client resolver
+    drop-in + browser DoH bypass, Porkbun zone ops), Swap Configuration
+    (filesystem/rename caveats), OpenWebUI (introspection, verification
+    battery, ComfyUI imagegen), Hermes (config-home drift, live config
+    edits, auxiliary routing, password rotation — the pre-existing page was
+    preserved and extended).
+  - **Host pages:** FrankBook-Pro ARR-Stack filled with the API-version
+    split, schema-first probing, auth-posture method, auth-config trap, and
+    the qBittorrent E2E verification procedure added to [[FrankBook-Pro/qbittorrent]].
+  - **Desktop/Agent/Research:** Noctalia and Niri (config layers, remote
+    IPC, palette recolor, offline template render, rotation pinning, iCX3
+    RGB), Agent/ (OMP Harness, Agent Docs and Edits, Command Center
+    Dotfiles, index), PICO-8 Asset Standards.
+  - Index updated: `LLM/index.md` gained a Cross-Cutting References
+    section; Operations/ and Agent/ got index pages. AGENTS.md now points
+    at the new sections. Link checker + full test suite green.
+- **e2e: the FrankBook-Pro leg scoped to its own trigger-gated Playwright
+  project.** The `11-admin` walk included the six `*arr`/qBittorrent admin
+  links that terminate on 192.168.0.112, so every deploy gate died there when
+  the Mac mini slept (Caddy 502 on the five vhosts, connection timeout on the
+  raw Sonarr link). Split: those links moved to
+  `tests/e2e/specs/19-frankbook.spec.ts` under a `frankbook` project;
+  `tests/e2e/run.sh` excludes it from the default project set, and
+  `scripts/deploy.sh` computes a trigger from the same BASE..HEAD range the
+  plan uses (the spec, `frankbook-pro/**`, `shared/homepage/services.json`,
+  the `dankclaw/Caddyfile` arr vhosts) and sets `E2E_INCLUDE_FRANKBOOK=1`.
+  Verified live with the host asleep: full suite 33/33 in 1.1 min, the admin
+  walk back to 5.4 s; `--project=frankbook --list` proves the selection.
+  Live pass of the frankbook walk itself is deferred to the next deploy with
+  an awake host.
+  - **Catch while verifying: the 17:47 Ornith-1.5-9B swap (`77decb5`) had
+    left `frank_small_unit.bats` and the deployer's payload gate stale** —
+    the seam still expected MiMo-Q6K names and `deploy-remote.sh` gated
+    `LLM_SMALL_MODEL_FILE`/`LLM_SMALL_MMPROJ` on files the unit no longer
+    loads (the anti-drift seam doing its job). Both repointed to the
+    Ornith-Q8 paths; full `./tests/run.sh` back to green (366 tests).
+- **frank-small: compute-buffer pair doubled to 4096/2048, and the prefill win
+  measured rather than assumed.** Frank asked for the batch and ubatch doubled.
+  The blocker was on record: the Q5_K_M predecessor documented **"ubatch 2048
+  failed the scratch boot"**, and this shape carries 616 MiB less free space
+  than that predecessor did.
+  - The A/B used the service itself as the bench (one card, one server — no
+    scratch port): the old rung was re-booted through a transient
+    `FRANK_SMALL_ENV_FILE` systemd drop-in, and it reproduced the 2026-10-03
+    boot **byte-identically** (21,276 used / 2,849 free), which is what makes
+    ubatch the only variable. Repo never saw a fake revert commit; scaffolding
+    staged to `~/.AGENTTRASH/20261005-frank-small-ubatch-ab/`.
+  - **At 8,928 prompt tokens: prefill 3,972/4,077 -> 4,207/4,212 t/s, +4.6 %,
+    for 616 MiB (free 2,849 -> 2,279).** Decode flat (91.9–93.0 vs 93.8–94.3 —
+    noise). The control that proves it is geometry and not drift: a 279-token
+    prompt sits below *both* micro-batches, so it cannot see the knob, and it
+    did not move (2,676 vs 2,658–2,811).
+  - **The predecessor's boot failure did not reproduce** — that was a shape
+    observation, not a law; this shape boots 2048 clean. And the fleet lesson
+    "**`--ubatch-size` has an optimum**" is now scoped to what it actually is:
+    true on the 2x R9700 layer split (peak 512, −10 % by 2048), false on the
+    CUDA 3090 where the curve climbs monotonically to 2048. AGENTS.md corrected
+    — the curve is backend-specific and must never be carried across backends.
+  - The seam stopped hard-coding a rung: `serve_argv` now carries `BATCH_SIZE`
+    and `UBATCH_SIZE` the way it already carried `CTX_SIZE`, and the pair test
+    asserts the argv follows the shipped env plus the `ubatch <= batch`
+    invariant. Proven live both ways — retuning the env to 8192/4096 keeps it
+    green, inverting to 2048/4096 turns it red. (The old assertions only ever
+    exercised the serve-script *defaults*, because the test env never carried
+    these two keys.)
+  - `4096/1536` is the documented middle rung; 2,279 MiB free sits at the bottom
+    edge of the 2–3 GiB floor. Deployed `e5893dc`, e2e gate PASSED (2 mini specs
+    flaky-then-passed once the sleeping FrankBook-Pro woke mid-run — WoL took a
+    second burst and ~90 s). Docs:
+    [[frank-small-llama-mimo-q6k-2x200704]] (*Compute-buffer geometry*).
+
+## 2026-10-03
+- **frank-primary: shape F then shape G — MNBT 2048, the CUDA-graph plan term
+  pinned to the measured capture, KV forced to 400,700 tokens, and UTIL 1.0
+  proven UNBOOTABLE.** Frank asked what the batch was with GPUs showing open
+  space; the audit found the arena is `budget - non-expert - KV reserve -
+  graphs - overhead` where the only deploy-reachable terms are the budget and
+  the graph estimate (registry KV price 9,190 B/token and the 5.25 GiB
+  overhead are image constants; the ~3.5 GiB of driver-free VRAM was
+  estimator slack, not allocatable).
+  - Shape F (c682c63): MNBT 1024 -> 2048 (arena-free — the registry prices
+    overhead flat up to its nbt 4096 anchor and the fitted 5.25 was fitted AT
+    4096), util 0.98 -> 0.982, graph term 0.55 -> 0.32 GiB (256 base +
+    12/graph vs the fork's 256 + 52/graph). Measured: arena 17.52 GiB /
+    7,653 slots, budget 31.29.
+  - Shape G v1 (0a4ff30) tried util 1.0: **both workers refused init before a
+    single weight loaded** — `ValueError: Free memory on device cuda:N
+    (31.29/31.86 GiB ...)`, NRestarts=2 crash loop, zero payload progress.
+    The CLI field is `Field(gt=0, le=1)`, but the worker demands free >
+    budget, so budget == total is a trap, not a knife-edge. Frank: ".982 is
+    the law, never 1.00" — now with a second proof.
+  - Shape G v2 (c67a2ae, deployed green incl. e2e 33/33): util back to
+    0.982, graph base to 0 (the shape F boot reported a NEGATIVE -0.06 GiB
+    capture delta, so the 256 MiB base was fiction; 0 + 12 MiB/graph still
+    prices a future 18-graph spec-decode shape at 0.21 GiB), KV_BYTES
+    3.15e9 -> 3.1665e9 = Frank's 400,700-token ask.
+  - MEASURED live: arena **17.77 GiB / 7,762 slots** (+240 over shape E),
+    KV **401,408 tokens / 2.00x concurrency** — the first shape holding two
+    full-length conversations with real spare; capture measured 0.31/0.10
+    GiB against the 0.07 charge (under-charged, boots and holds); driver
+    used 30.97 of 31.86 GiB -> **0.89 GiB real free per card**, held through
+    a live 4,096-token generation (~38-39 tok/s single-request wall-incl.,
+    +13% vs the 33.99 shape-E baseline; single sample, not a bench).
+  - Verification seam rewritten to assert the shape by its OUTPUTS (util at
+    the 0.982 law, graph charge below the analytic default and per-graph
+    positive, KV pool >= 400,700 tokens at the measured 7,892 B/token).
+  Runbook: `serverAppConfigs-docs/LLM/dankserver-qwen38-fp8-expert-offload-tuning.md`.
+
 ## 2026-10-01
 - **frank-primary rolled BACK to Qwen3.8-Flash-Next-Uncensored-FP8 + MTP4;
   GLM/DFlash2 declared failed and purged.** The DFlash2 drafter booted and
@@ -971,9 +1129,10 @@ the 3090", then **"frank-small should have vision"**.
 - **Three contexts cost window, and that is the tradeoff Frank was told:**
   180,224 tokens per conversation, not the native 262,144. A longer prompt is
   a hard `400 exceed_context_size_error`, so `models.yml` sets
-  `contextWindow: 180224`. Consequence: frank-primary's 183,501-token
-  compaction request no longer fits here with summary room; the two-slot
-  rollback is the way back.
+  `contextWindow: 180224`. Consequence at the time: frank-primary's then
+  183,501-token compaction request did not fit here with summary room —
+  resolved later the same day when frank-primary's window moved to 200,704
+  and its trigger fell to 140,492 (see the shape-D entry below).
 - Measured (2026-10-02, this shape): decode **106.8 t/s**, prefill **3,814
   t/s** (~5 % faster decode than 2 x 262,144 — shorter slot, shorter attention
   scan), three concurrent requests answered in parallel in 1.7 s with a fourth
@@ -993,3 +1152,221 @@ the 3090", then **"frank-small should have vision"**.
   payloads). Kept in place for rollback: `MiMo-Q6K/`, `MiMo-Q5KM/`,
   `Qwen3.8-27B-UD-Q4_K_XL/` and `Qwen3.8-27B-UD-Q8_K_XL/` (the latter holds the
   gufo DFlash2 drafter). Runbook [[frank-small-llama-mimo-q5km]].
+
+## 2026-10-02 — frank-primary shape D: 3 x 200704, fp8, pool sized for 2 contexts
+
+Frank's order: shrink frank-primary to a 200704-token window with about two
+resident full-length contexts, cut the KV pool, and drop MNBT to 1024. The
+first attempt (`30904ba`, 2 x 200704 + SEQS=2) **crash-looped**; the revert
+went live as `e460bbb`, and the shape was then re-applied one law at a time on
+top of the proven-good base, root cause first.
+
+- **ROOT CAUSE: `--max-num-seqs 2` cannot compile this model. It was never OOM,
+  never MNBT, never the KV dtype.** Four single-variable boot probes against
+  the then-deployed shape (production stopped for each, candidate booted AS the
+  service):
+
+  | MAXLEN | SEQS | MNBT | result |
+  |---|---|---|---|
+  | 262144 | 2 | 1024 | **Engine core initialization failed** |
+  | 262144 | 2 | 2048 | **Engine core initialization failed** |
+  | 262144 | 3 | 1024 | healthy - 792,069 tokens, 3.02x, arena 6,248 |
+  | 262144 | 3 | 2048 | healthy - the then-deployed incumbent |
+
+  `torch.fx.experimental.symbolic_shapes.ConstraintViolationError: Constraints
+  violated (L['query_start_loc'].size()[0]) ... specialized it to be a constant
+  (3)`, raised from AOT `build_guards(strict_error=True)` during
+  `determine_available_memory -> profile_run -> _dummy_run`, ~7 min in.
+  `model_runner.py` sizes that input buffer `max_num_reqs + 1` and `_dummy_run`
+  sets `num_reqs = min(max_num_batched_tokens, max_num_reqs)`, so its width is
+  **SEQS + 1** for ANY `MNBT >= SEQS` - 3 at SEQS 2, 4 at SEQS 3, exactly the
+  constant in the message. MNBT only moves the compile RANGE ((1,1024) vs
+  (1,2048)), which is why the identical error appears at both rungs.
+  `--enforce-eager` bypasses the path but drops CUDA graphs. The 2026-10-01
+  bf16 2 x 262144 failure was this same error class; its message named "KV
+  dtype and SEQS" together and nobody bisected, so the dtype was blamed - it
+  was a red herring.
+- **THE ARENA LEVER IS `max_model_len`, NOT `--kv-cache-memory-bytes`.** The
+  planner sizes expert cache as `budget - non-expert - KV reserve - graphs -
+  overhead`, and the KV reserve is registry-priced at **9190 B/token x
+  max_model_len**, independent of the pool override:
+
+  | MAXLEN | KV reserve | arena |
+  |---|---|---|
+  | 262144 | 2.48 GiB | 14.30 GiB / **6,248 slots** |
+  | 200704 | 1.91 GiB | 14.87 GiB / **6,496 slots** |
+
+  d(reserve) 0.57 GiB = d(arena) 0.57 GiB = +248 slots. The 6.1e9 -> 3.15e9
+  pool cut (2.75 GiB/rank) bought **zero** arena slots. The previous session's
+  premise - "the pool cut becomes arena, ~7,500 slots" - was wrong; the pool
+  cut is a runtime-HEADROOM lever (the shape it replaced died with `0 bytes
+  free` on a real 995-token request).
+- **Final shape (option D):** `--max-model-len 200704` x 3 slots, fp8 KV,
+  `--kv-cache-memory-bytes 3,150,000,000`/rank, MNBT 1024, gather 32x64, no
+  speculative decode, prefetch off, no `--expert-precise-memory`. The argv
+  keeps the only slot count that compiles; the POOL caps resident
+  conversations. Deployed `c769ff2`, healthy in 558 s, e2e 33/33.
+- **Measured live, and one prediction corrected.** `expert cache space 14.87
+  GiB / 6,496 slots` - the arena prediction landed exactly. But `GPU KV cache
+  size: 398,658 tokens, Maximum concurrency for 200,704 tokens per request:
+  1.99x` (3,146,129,408 B reserved), not the 2.04x the arithmetic predicted:
+  **the per-token KV price is shape-dependent** - 7,701 B/token at MAXLEN
+  262144 (6,100,000,000 B -> 792,069 tokens) vs **7,892** at 200704, ~2.5 %
+  dearer. Never extrapolate a concurrency figure across geometries.
+- **Smoke-verified on the live endpoint:** three concurrent 67-token requests
+  answered in parallel (~4.1 s each) - the 3 slots are real; native `tool_calls`
+  (`get_weather {"city":"Seattle"}`); thinking in `reasoning` with clean
+  `content`; `reasoning_effort` accepts only {xhigh, medium, low} and answers
+  400 for anything else (re-measured; the fork's message names the set).
+- **Harness contract followed the window:** `models.yml` `contextWindow`
+  262144 -> **200704** (compaction trigger 140,492 + 36,864 = 177,356 <=
+  200,704). Side effect: frank-primary's compaction request now fits inside a
+  frank-small slot again, which it had stopped doing when frank-small went to
+  3 x 180,224.
+- **Comment-only follow-up** `295d578` recorded the measured numbers and was
+  deployed so the per-service revision marker sits at HEAD; e2e 33/33 again.
+- Rollback for the pre-shape-D fp8+MTP4 era stays git tag
+  `frank-primary-fp8-mtp-20261001`. Runbook
+  [[dankserver-qwen38-fp8-expert-offload-tuning]].
+
+## 2026-10-02 - plex: the readiness gate probed HTTP against a TLS-only service
+
+Found by accident, and it had been latent since the secure-connections posture
+was set: `deploy_plex` gated on `curl -fsS http://127.0.0.1:32400/identity`,
+which **cannot** succeed. PMS drops plain HTTP off-loopback - the probe's
+connection arrives on the container's eth0 - so Plex accepts the TCP connection
+and then answers with nothing, and curl reports "Empty reply from server" (52).
+
+- Trigger: a bare `./scripts/deploy.sh dankserver` (no `--service`) drains the
+  whole pending backlog, not just the service you were thinking about. plex's
+  per-service revision had been behind for long enough that this gate had never
+  run against the current posture. It burned its full 600 s and failed the run;
+  every service after plex in the order was left undeployed.
+- **Plex was healthy the entire time.** `https://127.0.0.1:32400/identity`
+  returned 200 with the MediaContainer XML in 44 ms while the gate was still
+  timing out. The in-container `http://127.0.0.1:32400/identity` also answers
+  200 - loopback is the one path PMS serves in the clear, which is exactly why
+  the host-side probe could not see the difference.
+- Evidence chain that settled it: `podman exec plex curl http://127.0.0.1:32400`
+  = 200; `curl http://127.0.0.1:32400` from the host = empty reply;
+  `/proc/net/tcp` inside the container showed `00000000:7E90` LISTEN plus stuck
+  ESTABLISHED pairs on `1703590A:7E90` (10.89.3.23, the container's own eth0
+  address) that never produced a response.
+- **The repo already knew this law, in a different file:**
+  `tests/caddyfile.bats` pins the plex vhosts to a TLS upstream with the comment
+  "PMS drops plain HTTP off-loopback". The deployer's gate never got the same
+  treatment. Fixed to `curl -fsSk --max-time 3 https://127.0.0.1:32400/identity`
+  (`-k`: PMS serves its own self-signed cert on 32400).
+- Re-deployed `39f0e0d`: plex recorded deployed, the *arr services recorded
+  (compose-recorded on this host, no restart attempted), tautulli deployed,
+  e2e 33/33. Verified after: `plex-https=200`, plex container up, tautulli
+  healthy, frank-primary untouched (`health=200`, NRestarts=0).
+- **Operating lesson: `deploy.sh <target>` with no `--service` is a fleet-wide
+  drain.** It restarts every service whose recorded revision is behind HEAD and
+  whose paths changed since. To touch one service, name it.
+
+## 2026-10-02 — frank-primary shape E: `--expert-precise-memory` (+1,026 arena slots), a mounted compile cache, and the P2P answer
+
+Shape D left one question open — the plan's overhead term was reserving 2.35 GiB
+of compile headroom the boot never used — and the answer to Frank's free-VRAM
+question arrived together with a boot-cost bug that had been costing 181 s on
+every restart since the cutover.
+
+- **`--expert-precise-memory` is real and it fits (option A).** Boot-probed on
+  the shape-D base with production stopped, the candidate booted AS the service
+  (`FRANK_PRIMARY_ENV_FILE=/tmp/probe-b.env bash scripts/serve-frank-primary.sh`,
+  trap-guaranteed restore), the numbers read off the engine's own plan lines:
+
+  | plan line | shape D | shape E |
+  |---|---|---|
+  | `runtime overhead (non-torch + buffers + activation)` | 7.60 GiB | **5.25 GiB** (`precise (gfx1201 TP2 nbt4096/seqs16/len262144)`) |
+  | `expert cache space` | 14.87 GiB | **17.22 GiB** |
+  | slots | 6,496 | **7,522** (+1,026, +16 %) |
+  | `swap region` | — | 13.7 GiB / 5,986 slots, 133-134 per layer |
+  | boot | — | **healthy in 420 s**, no OOM |
+
+  The flag also prints `'expert_precise_memory': True` in `non-default args`, so
+  a silent argparse drop cannot hide. It was refused on 2026-10-01 only because
+  the 2 x 262144 bf16 shape was already 0.13 GiB under the ceiling — there the
+  flag was the last stack over the line, not the cause.
+- **The pool was NOT cut (option A) because the pool is not an arena lever.**
+  A 2.2e9 arm was probed in the same window and reports `277,686 tokens,
+  1.38x` — measured, rejected as the shipped value, kept as the conservative
+  fallback. With 3.15e9/rank the endpoint keeps shape D's 1.99x concurrency and
+  gains the arena, which is the shape D follow-up had actually been looking for.
+- **THE COMPILE CACHE WAS BEING THROWN AWAY ON EVERY RESTART — 181 s each.**
+  Measured: a fresh container has **no `/root/.cache` at all** (`No such file or
+  directory`); after our boot it holds `/root/.cache/vllm` at
+  **1,154,714,333 B in 4,665 files** plus `/root/.cache/comgr` at 229 MB, inside
+  the ephemeral layer, and `podman rm -f` (every deploy, restart, reboot)
+  discards them. The engine names the loss itself: `Compiling a graph for compile
+  range (1, 1024) takes 173.19 s` ... `torch.compile took 181.20 s in total` ...
+  `init engine (profile, create kv cache, warmup model) took 217.26 s
+  (compilation: 181.20 s)`. The serve script now mounts
+  `FRANK_PRIMARY_CACHE_DIR` at `/root/.cache` — the **parent**, so the vLLM
+  AOT/torch_compile tree (`VLLM_CACHE_ROOT`, default `/root/.cache/vllm`) and
+  the comgr JIT cache both persist, and nothing is hidden because the image has
+  no such directory. Same class of fix as the recipe's
+  `/cache/{triton,vllm,inductor}` volumes; the compile RANGE is part of the
+  cache key, so an MNBT change costs one recompilation, not a permanent tax.
+- **`GPU_MAX_HW_QUEUES=2`** (HSA hardware queues per GPU) added at Frank's
+  direction. Verified a real knob rather than a no-op — the literal is read from
+  `/opt/rocm/core-10.0/lib/libhsa-runtime64.so` — and shipped deliberately
+  alongside the 32x64 gather geometry. **Not A/B measured.**
+- **P2P: the hardware supports it, and the engine is already using it.** The
+  live log shows `r4d all-reduce INSTALLED (rank=0 max=49152KB finegrained=T…)`
+  on BOTH ranks and `Using ['R4D_AR', 'PYNCCL'] all-reduce backends` — R4D's
+  P2P path, with RCCL as fallback, measured at **45.274 GB/s bidirectional**.
+  Root ports `00:01.1`/`00:01.3` both report ACS `ReqRedir+`, so P2P hairpins at
+  the root complex; there is no XGMI. The old FAIL predates the 2026-09-20 board
+  swap. Kernel support confirmed: `CONFIG_HSA_AMD_P2P=y`, `CONFIG_PCI_P2PDMA=y`,
+  `CONFIG_HSA_AMD=y`, `CONFIG_ZONE_DEVICE=y`, `iommu=pt` on the cmdline.
+- **Recipe diff (Frank's 3 x R9700 compose vs this podman run), ranked:**
+  - *Acted on:* persistent compile caches (above — the one real operational
+    gap), and `GPU_MAX_HW_QUEUES`.
+  - *Retracted:* pinned `--compilation-config cudagraph_capture_sizes` was on my
+    list; measured, it is **not** a gap here — capture is 3 PIECEWISE + 2 FULL
+    graphs, `Graph capturing finished in 3 secs, took 0.31 GiB` (0.11 GiB on the
+    lighter rank). No 15-52-size ladder to cut.
+  - *Not acted on — measured candidates, none shipped (no bench window; these
+    are CPU-side and our decode is gather-bound, so a spin-wait stealing cores
+    from the gather threads is plausible but unproven):* `OMP_NUM_THREADS=8`
+    (defaults to all 16 logical CPUs on this 7700X), `HSA_ENABLE_INTERRUPT=1`,
+    `HSA_ENABLE_MWAITX=1`, `HSA_ENABLE_SDMA`, `HSA_NO_SCRATCH_RECLAIM`,
+    `HSA_ENABLE_IPC_MODE_LEGACY`. All six HSA names verified real — the literals
+    are read out of `/opt/rocm/core-10.0/lib/libhsa-runtime64.so` — so they are
+    knobs, not no-ops; any of them needs its own A/B before shipping.
+  - *Deliberate divergence, keep ours:* no MTP (the recipe has it; measured
+    fastest here), `qwen3_xml` vs `qwen3_coder` (ours verified live), no
+    `--override-generation-config` (server-side sampling defaults), published
+    port + Caddy rather than `network_mode: host`.
+  - *Non-gaps:* `shm_size 32gb` (ours is `--ipc=host`), `memlock -1` (same),
+    `SYS_PTRACE` (same), `/dev/kfd` + `/dev/dri` (same), `--enable-chunked-prefill`
+    (V1 default), `plecache` volume (the flag is not even passed), and
+    `ROCR_VISIBLE_DEVICES=1,2,3` vs our `HIP_VISIBLE_DEVICES=0,1` — verified
+    equivalent here: `rocm-smi --showid` puts the two R9700s at GPU[0]/GPU[1] and
+    the 7700X iGPU at GPU[2], so our indices are the right two cards.
+- **Harness note, because it cost two probe attempts:** a probe that reported an
+  instantaneous candidate death was a logging artifact of the inline SSH script,
+  not a crash — the container and its serve script must be polled by recorded id
+  and their logs dumped on the exit path, or a failure looks like a bad flag.
+  The flag was fine both times.
+- Live: `health=200`, `NRestarts=0`, arena 17.22 GiB / 7,522 slots, KV 398,658
+  tokens / 1.99x. Docs: [[dankserver-qwen38-fp8-expert-offload-tuning]].
+- **Deployed `1ac4d48`, e2e 33/33, and the live plan matched the probe exactly:**
+  `'expert_precise_memory': True`, `runtime overhead … 5.25 GiB`, `expert cache
+  space 17.22 GiB / 7522 slots`, `swap region 13.7 GiB / 5986 slots`, `GPU KV
+  cache size: 398,658 tokens … 1.99x`, pool 3,150,000,000, MNBT 1024. The
+  precise overhead and the arena growth are now production numbers, not a
+  probe's projection.
+- **The compile-cache fix was then proven on the same endpoint, two consecutive
+  boots of the deployed shape:** cold `Compiling a graph for compile range
+  (1, 1024) takes 179.24 s` → `torch.compile took 188.26 s`; warm `Directly load
+  the compiled graph(s) for compile range (1, 1024) from the cache, took
+  2.143 s` → `torch.compile took 3.97 s in total` → `init engine (profile,
+  create kv cache, warmup model) took 41.13 s (compilation: 3.98 s)`. The host
+  dir held 1.4 GB in 4,707 files after the cold boot. **Time to `/health` is
+  dominated by the 172.8 GiB payload load** (the warm boot still needed 490 s),
+  so the honest claim is ~184 s of inductor/AOT build recovered per restart —
+  not a proportional boot cut.
+

shared/prometheus/prometheus.yml

@@ -98,10 +98,12 @@ scrape_configs:
   # service down. Same host, same rule, so both local endpoints use the
   # gateway.
   #
-  # (2026-09-26) frank-primary is llama.cpp again (the unsloth b11030-mix
-  # prebuilt serving Qwen3.8-Flash-Next Q8_0) on Dankserver, a DIFFERENT
-  # machine: the netavark published-port quirk above is same-host only, so
-  # plain LAN routing reaches 192.168.0.235:7770 and the job returns.
+  # (2026-10-05) frank-primary left this file: FreeToken (the PR587 ROCm
+  # engine now serving it) has NO /metrics endpoint — the job scraped 404s
+  # forever, which is a structural never-green, not an outage. The
+  # netavark published-port quirk above is same-host only; plain LAN
+  # routing REACHES 192.168.0.235:7770 (health answers) — only the metrics
+  # path is absent. If FreeToken grows a /metrics, re-add the job here.
   - job_name: "frank-small"
     metrics_path: "/metrics"
     static_configs:
@@ -116,9 +118,3 @@ scrape_configs:
         labels:
           service: frank-large
 
-  - job_name: "frank-primary"
-    metrics_path: "/metrics"
-    static_configs:
-      - targets: ["192.168.0.235:7770"]
-        labels:
-          service: frank-primary

shared/prometheus/rules/llm-fleet.yml

@@ -1,10 +1,12 @@
 # Alerting and recording rules for the LLM fleet.
 #
-# One metric namespace: llamacpp:* — frank-small, frank-large and
-# frank-primary, all llama.cpp with /metrics (the family verified from
-# /metrics on 2026-09-16 and 2026-09-20). frank-primary is scraped across
-# the LAN at 192.168.0.235:7770 (the netavark published-port quirk that
-# forces the gateway for the Dankclaw endpoints is same-host only).
+# One metric namespace: llamacpp:* — frank-small and frank-large, both
+# llama.cpp with /metrics (the family verified from /metrics on 2026-09-16
+# and 2026-09-20). frank-primary LEFT the family 2026-10-05: FreeToken (the
+# PR587 ROCm engine) has no /metrics endpoint, so its scrape job was
+# deleted rather than left never-green.
+# (The netavark published-port quirk that forces the Dankclaw endpoints
+# onto the gateway is same-host only.)
 #
 # Every expression is keyed to a series that EXISTS. Nothing matches on
 # absence: when an endpoint's runtime leaves the fleet, its selectors are
@@ -110,6 +112,6 @@ groups:
           description: >-
             Prometheus cannot reach the LLM endpoint. frank-small and
             frank-large are root units on Dankclaw: `systemctl is-active
-            frank-small` / `frank-large`. frank-primary is a root unit on
-            Dankserver: `systemctl is-active frank-primary`. All three
-            scraped endpoints export the llamacpp:* family.
+            frank-small` / `frank-large`. Both scraped endpoints export
+            the llamacpp:* family. (frank-primary left the scrape set
+            2026-10-05 — FreeToken exports no /metrics.)

tests/e2e/playwright.config.ts

@@ -45,5 +45,16 @@ export default defineConfig({
       testMatch: /(14|15)-.*\.spec\.ts/,
       use: { storageState: { cookies: [], origins: [] } },
     },
+    {
+      name: "frankbook",
+      // The *arr/qBittorrent walk (19-). Excluded from default runs and from
+      // the deploy e2e gate unless a frankbook trigger path changed — the
+      // links terminate on FrankBook-Pro, which sleeps (see tests/e2e/run.sh
+      // and scripts/deploy.sh). Matched by number prefix like every project
+      // here; joined 2026-10-05.
+      testMatch: /19-.*\.spec\.ts/,
+      use: { storageState: ".auth/qa-e2e-admin.json" },
+      dependencies: ["setup"],
+    },
   ],
 });

tests/e2e/run.sh

@@ -4,8 +4,14 @@
 # part of ./tests/run.sh. Credentials live in tests/e2e/.env.local
 # (git-ignored; see .env.local.example). Usage:
 #
-#   tests/e2e/run.sh                  # full suite
+#   tests/e2e/run.sh                  # full suite (frankbook project excluded)
 #   tests/e2e/run.sh --project guest  # any playwright args pass through
+#
+# The frankbook project (19-frankbook.spec.ts — the *arr/qBittorrent walk)
+# is NOT in the default project set: its links terminate on FrankBook-Pro,
+# which sleeps when idle, so it must not gate unrelated deploys. deploy.sh
+# sets E2E_INCLUDE_FRANKBOOK=1 when a frankbook trigger path changed. An
+# explicit --project argument in "$@" bypasses the default set entirely.
 set -euo pipefail
 cd "$(dirname "$0")"
 
@@ -24,4 +30,24 @@ fi
 
 npx playwright install chromium
 
-exec npx playwright test "$@"
+# Default project set; the frankbook project joins only when deploy.sh saw a
+# frankbook trigger path (E2E_INCLUDE_FRANKBOOK=1). Other playwright args
+# (--list, --grep, file filters) pass through alongside the default set; an
+# explicit --project in "$@" takes over untouched.
+DEFAULT_PROJECTS=(setup standard admin guest)
+if [[ "${E2E_INCLUDE_FRANKBOOK:-0}" == "1" ]]; then
+    DEFAULT_PROJECTS+=(frankbook)
+fi
+has_project=0
+for arg in "$@"; do
+    case "$arg" in
+        --project|--project=*) has_project=1 ;;
+    esac
+done
+PROJECT_ARGS=()
+if [[ "$has_project" -eq 0 ]]; then
+    for project in "${DEFAULT_PROJECTS[@]}"; do
+        PROJECT_ARGS+=(--project "$project")
+    done
+fi
+exec npx playwright test "$@" "${PROJECT_ARGS[@]}"

tests/e2e/specs/11-admin.spec.ts

@@ -24,10 +24,28 @@ test("admin panel renders admin-category services", async ({ page }) => {
 test("admin open links reach their apps through SSO", async ({ page }) => {
   test.setTimeout(240_000);
   // Expected link count comes from the live registry (admin showLink=true
-  // entries, now including the *arr vhosts that use their own auth).
-  const registry: { name: string; category: string; showLink: boolean }[] =
-    await (await page.request.get("/services.json")).json();
+  // entries, now including the *arr vhosts that use their own auth). The
+  // frankbook-hosted links (probe on 192.168.0.112) are NOT walked here:
+  // they terminate on FrankBook-Pro, which sleeps when idle, so their walk
+  // lives in 19-frankbook.spec.ts (frankbook project — excluded from the
+  // default e2e run and from the deploy gate unless a frankbook trigger
+  // path changed; see tests/e2e/run.sh and scripts/deploy.sh).
+  const registry: {
+    name: string;
+    category: string;
+    showLink: boolean;
+    url: string;
+    probe: string;
+  }[] = await (await page.request.get("/services.json")).json();
   const expectedAdminLinks = registry.filter((s) => s.category === "admin" && s.showLink).length;
+  const frankbookUrls = registry
+    .filter(
+      (s) =>
+        s.category === "admin" &&
+        s.showLink &&
+        ((s.probe ?? "").startsWith("192.168.0.112") || s.url.includes("192.168.0.112")),
+    )
+    .map((s) => new URL(s.url).origin);
   await page.goto("/admin/");
   await expect(page.locator("#grid a.open")).toHaveCount(expectedAdminLinks);
   const hrefs = await page.locator("#grid a.open").evaluateAll((els) =>
@@ -35,6 +53,7 @@ test("admin open links reach their apps through SSO", async ({ page }) => {
   );
 
   for (const href of hrefs) {
+    if (frankbookUrls.includes(new URL(href).origin)) continue; // 19-frankbook walks these
     const target = new URL(href);
     await page.goto(href);
     // Every guarded vhost runs its own Anubis proof-of-work on first visit.

tests/e2e/specs/19-frankbook.spec.ts

@@ -0,0 +1,69 @@
+import { expect, test } from "@playwright/test";
+
+// The *arr / qBittorrent admin links terminate on FrankBook-Pro
+// (192.168.0.112), a Mac mini that SLEEPS when idle: Caddy answers 502 for
+// the vhosts and the raw Sonarr link times out. This walk therefore lives in
+// its own project, excluded from the default e2e run (tests/e2e/run.sh) and
+// from the deploy gate's invocation (scripts/deploy.sh) unless a frankbook
+// trigger path changed: this spec, frankbook-pro/**,
+// shared/homepage/services.json, or the dankclaw Caddyfile that serves the
+// arr vhosts. A sleeping Mac mini must never fail an unrelated deploy.
+//
+// Assertions mirror 11-admin's "admin open links reach their apps through
+// SSO": the guard chain settles (Anubis PoW where present — Sonarr's raw-IP
+// link is internal-only and has none), the hostname holds, no authentik form
+// appears, and the app answers below 500.
+
+const FRANKBOOK_IP = "192.168.0.112";
+const CHALLENGE = "Making sure you're not a bot";
+
+interface RegistryEntry {
+  name: string;
+  category: string;
+  showLink: boolean;
+  url: string;
+  probe: string;
+}
+
+test("frankbook admin open links reach their apps", async ({ page }) => {
+  test.setTimeout(240_000);
+  const registry: RegistryEntry[] = await (await page.request.get("/services.json")).json();
+  const frankbook = registry.filter(
+    (s) =>
+      s.category === "admin" &&
+      s.showLink &&
+      ((s.probe ?? "").startsWith(FRANKBOOK_IP) || s.url.includes(FRANKBOOK_IP)),
+  );
+  const frankbookUrls = frankbook.map((s) => new URL(s.url).origin);
+
+  await page.goto("/admin/");
+  const hrefs = await page
+    .locator("#grid a.open")
+    .evaluateAll((els) => els.map((el) => (el as HTMLAnchorElement).href));
+  const frankbookHrefs = hrefs.filter((href) => frankbookUrls.includes(new URL(href).origin));
+  // The rendered frankbook link set must mirror the registry: a renamed arr
+  // vhost or a dropped probe fails here instead of silently dropping the
+  // walk's coverage.
+  expect(frankbookHrefs, "rendered frankbook links must match the registry").toHaveLength(
+    frankbook.length,
+  );
+
+  for (const href of frankbookHrefs) {
+    const target = new URL(href);
+    await page.goto(href);
+    // Guarded vhosts run their own Anubis proof-of-work on first visit; the
+    // raw-IP Sonarr link has none and settles immediately.
+    await expect
+      .poll(
+        () => page.locator("body").innerText().then((t) => !t.includes(CHALLENGE)),
+        { timeout: 90_000 },
+      )
+      .toBe(true);
+    await expect
+      .poll(() => new URL(page.url()).hostname, { timeout: 60_000 })
+      .toBe(target.hostname);
+    expect(await page.locator('input[name="uidField"]').count()).toBe(0);
+    const resp = await page.request.get(page.url());
+    expect(resp.status(), `${target.hostname} must answer, not error`).toBeLessThan(500);
+  }
+});

tests/e2e_scaffold.bats

@@ -10,7 +10,8 @@ load helpers
     for f in run.sh package.json playwright.config.ts .env.local.example \
              lib/env.ts lib/login.ts \
              specs/00-auth.setup.ts specs/10-homepage.spec.ts specs/11-admin.spec.ts \
-             specs/12-crawl.spec.ts specs/13-sso.spec.ts specs/14-guard.spec.ts; do
+             specs/12-crawl.spec.ts specs/13-sso.spec.ts specs/14-guard.spec.ts \
+             specs/19-frankbook.spec.ts; do
         [ -f "$REPO_ROOT/tests/e2e/$f" ] || {
             echo "missing tests/e2e/$f"
             return 1

tests/frank_large_unit.bats

@@ -1,264 +1,247 @@
 #!/usr/bin/env bats
-# Seam: frank-large serves kingjones777's Qwen3.8-27B Q8_0_ROCMFPX_AGENT (a
-# ftype-115 ROCmFPX file) with the repo's MTP draft head on the Strix Halo
-# iGPU, via the ROCmFPX fork of llama.cpp — one session at the native
-# 262,144 window.
+# Seam: frank-large serves peonist-ai/halogen-qwen3.8-27b (a .hgn halogen
+# checkpoint) on the Strix Halo iGPU through the halogen CONTAINER
+# (ghcr.io/peonist-ai/halogen, digest-pinned), driven by the root unit's
+# serve script — one conversation of 200,700 tokens with the DFlash2
+# drafter ON. 2026-10-05 cutover from the ROCmFPX llama.cpp shape.
 #
-# This seam exists because the format is the constraint: GGML_TYPE_Q8_0_ROCMFPX
-# (tensor type 103) is understood by the ROCmFPX fork and by nothing else on
-# this host, so the runtime, the payload and the deployer's gate have to agree
-# on paths that no other endpoint uses. On 2026-10-01 a serve-script gate and
-# its deployer twin disagreed during a cutover and the endpoint crash-looped;
-# both sides are pinned here.
+# This seam exists because the artifacts are a triangle that must agree on
+# exact strings the engine reads: the image DIGEST, the checkpoint file
+# name AND byte count inside the /models mount, and the tokenizer's
+# flat-file shape (a HF cache snapshot's symlinks dangle inside the
+# container). A serve gate and a deployer gate that disagree is the
+# 2026-10-01 crash-loop; both sides are pinned here, plus the one engine
+# law the fleet must never "optimize" away: HALOGEN_KV_SLOTS=1 is what
+# keeps speculative decoding on.
 #
-# Everything is tested from what an operator or a supervisor can observe — the
-# command line, the exit code, the deployer's error text — never from anything
-# inside llama.cpp.
+# Everything is tested from what an operator or supervisor can observe —
+# the command line podman would run, the exit code, the deployer's error
+# text — never from anything inside the engine.
+
 setup() {
-    load helpers
-
-    SERVE="$REPO_ROOT/scripts/serve-frank-large.sh"
-    UNIT="$REPO_ROOT/scripts/frank-large.service"
-    ENV_FILE="$REPO_ROOT/scripts/frank-large.env"
-    DEPLOYER="$REPO_ROOT/scripts/deploy-remote.sh"
-    BUILD="$REPO_ROOT/scripts/build-rocmfpx.sh"
-    RUNTIME_STUB="/home/dank/GitPackages/kingjones-ROCmFPX/build/bin/llama-server"
-    MODEL_STUB="/home/dank/models/kingjones-Qwen3.8-27B-ROCmFP4-STRIX-MTP/Qwen3.8-27B-Q8_0_ROCMFPX_AGENT.gguf"
-    DRAFT_STUB="/home/dank/models/kingjones-Qwen3.8-27B-ROCmFP4-STRIX-MTP/mtp-Qwen3.8-27B-Q4_0.gguf"
-    MMPROJ_STUB="/home/dank/models/kingjones-Qwen3.8-27B-ROCmFP4-STRIX-MTP/mmproj-Qwen3.8-27B-BF16.gguf"
-
+    ROOT="$(cd "$(dirname "$BATS_TEST_FILENAME")/.." && pwd)"
+    SERVE="$ROOT/scripts/serve-frank-large.sh"
+    ENV_FILE="$ROOT/scripts/frank-large.env"
+    UNIT="$ROOT/scripts/frank-large.service"
+    DEPLOYER="$ROOT/scripts/deploy-remote.sh"
+    BUILD="$ROOT/scripts/build-rocmfpx.sh"
     TEST_TMP="$(mktemp -d)"
-    SRC="$TEST_TMP/src"
-    mkdir -p "$SRC"
-    git -C "$SRC" init -q
-    git -C "$SRC" -c user.email=t -c user.name=t commit -q --allow-empty -m x
+    touch "$TEST_TMP/empty.env"
 }
 
 teardown() {
     rm -rf "$TEST_TMP"
 }
 
-# make_stub — an executable that prints its whole argv on one line, so
-# "what command line would the supervisor run" is observable output.
-make_stub() {
-    cat > "$TEST_TMP/bin" <<'STUB'
-#!/bin/sh
-printf '%s\n' "$*"
+# podman_stub — a fake podman on PATH that prints the argv of `run` and
+# `wait` and answers `images` with whatever digests the test names, so
+# "what would the supervisor run" is observable output and the digest
+# gate is testable without a 3.5 GB image.
+podman_stub() {
+    mkdir -p "$TEST_TMP/bin"
+    cat > "$TEST_TMP/bin/podman" <<'STUB'
+#!/usr/bin/env bash
+case "$1" in
+    images) printf '%s\n' "${STUB_DIGESTS:-}" ;;
+    rm)     exit 0 ;;
+    run|wait) echo "PODMAN $*"; exit "${STUB_WAIT_RC:-0}" ;;
+    *)      exit 0 ;;
+esac
 STUB
-    chmod +x "$TEST_TMP/bin"
+    chmod +x "$TEST_TMP/bin/podman"
 }
 
-@test "the env file defines the deployment values" {
-    grep -qE '^LLAMA_BIN=/' "$ENV_FILE"
-    grep -qE '^MODEL_FILE=/' "$ENV_FILE"
-    grep -qE '^DRAFT_FILE=/' "$ENV_FILE"
-    grep -qE '^ALIAS=[a-z0-9-]+$' "$ENV_FILE"
-    grep -qE '^LISTEN_PORT=[0-9]+$' "$ENV_FILE"
-    grep -qE '^CTX_SIZE=[0-9]+$' "$ENV_FILE"
+# good_env — the minimum env the serve script demands, with the digest
+# stub agrees on and zero-byte artifacts passing the size gate.
+good_env() {
+    : > "$TEST_TMP/empty.env"
+    touch "$TEST_TMP/qwen.hgn"
+    mkdir -p "$TEST_TMP/tokenizer" && touch "$TEST_TMP/tokenizer/tok.json"
+    GOOD_ENV=(
+        "FRANK_LARGE_ENV_FILE=$TEST_TMP/empty.env"
+        "FRANK_LARGE_IMAGE=ghcr.io/peonist-ai/halogen@sha256:cafe"
+        "FRANK_LARGE_CONTAINER=frank-large-halogen"
+        "FRANK_LARGE_LEGACY_CONTAINERS=frank-large"
+        "FRANK_LARGE_DEV_KFD=$TEST_TMP/qwen.hgn"
+        "FRANK_LARGE_DEV_DRI=$TEST_TMP"
+        "FRANK_LARGE_LOG=$TEST_TMP/serve.log"
+        "FRANK_LARGE_PUBLISH=7770:8731"
+        "FRANK_LARGE_MODEL_DIR=$TEST_TMP"
+        "FRANK_LARGE_CHECKPOINT=qwen.hgn"
+        "FRANK_LARGE_CHECKPOINT_BYTES=0"
+        "FRANK_LARGE_TOKENIZER_FILES=tok.json"
+        "HALOGEN_MODEL_ID=frank-large"
+        "HALOGEN_SLOT_CTX=200700"
+        "HALOGEN_KV_SLOTS=1"
+        "HALOGEN_DRAFTER=2"
+        "STUB_DIGESTS=localhost/x ghcr.io/peonist-ai/halogen@sha256:cafe"
+    )
 }
 
-@test "the serve script fails when the runtime is missing" {
-    # Only the model, the head and the projector exist; the fork was never
-    # built. llama.cpp is not reached, so this fails fast and says what to
-    # run instead of failing at bind time.
-    touch "$TEST_TMP/model.gguf" "$TEST_TMP/draft.gguf" "$TEST_TMP/mmproj.gguf"
-    run env "LLAMA_BIN=$TEST_TMP/nope/llama-server" \
-        "MODEL_FILE=$TEST_TMP/model.gguf" "DRAFT_FILE=$TEST_TMP/draft.gguf" \
-        "MMPROJ_FILE=$TEST_TMP/mmproj.gguf" "$SERVE"
+@test "the env file defines the halogen deployment values" {
+    grep -qE '^FRANK_LARGE_IMAGE=ghcr\.io/peonist-ai/halogen@sha256:[0-9a-f]{64}$' "$ENV_FILE"
+    grep -qE '^FRANK_LARGE_CONTAINER=frank-large-halogen$' "$ENV_FILE"
+    grep -qE '^FRANK_LARGE_PUBLISH=7770:8731$' "$ENV_FILE"
+    grep -qE '^FRANK_LARGE_CHECKPOINT=.*\.hgn$' "$ENV_FILE"
+    grep -qE '^FRANK_LARGE_CHECKPOINT_BYTES=[0-9]+$' "$ENV_FILE"
+    grep -qE '^FRANK_LARGE_CHECKPOINT_SHA256=[0-9a-f]{64}$' "$ENV_FILE"
+    grep -qE '^FRANK_LARGE_TOKENIZER_FILES="tokenizer\.json( [a-z_.]+)+"$' "$ENV_FILE"
+    grep -qE '^HALOGEN_MODEL_ID=frank-large$' "$ENV_FILE"
+    grep -qE '^HALOGEN_SLOT_CTX=[0-9]+$' "$ENV_FILE"
+}
+
+@test "the drafter stays on: slots must stay 1" {
+    # halogen makes batching and speculation mutually exclusive — with
+    # more than one slot the drafter is DISABLED and per-stream decode
+    # falls to serial speed (the sibling engine measured 41 -> 14 t/s
+    # when slots went up). frank-large was a one-session endpoint before
+    # this swap too (PARALLEL=1), so slots=1 preserves the contract;
+    # raising it is a product decision, never a default change.
+    grep -qE '^HALOGEN_KV_SLOTS=1$' "$ENV_FILE"
+    grep -qE '^HALOGEN_DRAFTER=2$' "$ENV_FILE"
+    # The cap/timeout pair is coupled (FLAGS.md table): a request at the
+    # cap must not outlast the queue timeout, or everyone queued behind
+    # it eats a 503.
+    grep -qE '^HALOGEN_MAX_TOKENS_CAP=[0-9]+$' "$ENV_FILE"
+    grep -qE '^HALOGEN_QUEUE_TIMEOUT=[0-9]+$' "$ENV_FILE"
+}
+
+@test "the serve script refuses when there is no GPU to serve on" {
+    podman_stub
+    good_env
+    run env PATH="$TEST_TMP/bin:$PATH" "${GOOD_ENV[@]}" \
+        FRANK_LARGE_DEV_KFD="$TEST_TMP/absent-node" "$SERVE"
     [ "$status" -ne 0 ]
-    [[ "$output" == *"build it with scripts/build-rocmfpx.sh"* ]]
+    [[ "$output" == *"no gfx1151 to serve on"* ]]
 }
 
-@test "the serve script refuses to start without the model file" {
-    make_stub
-    touch "$TEST_TMP/draft.gguf" "$TEST_TMP/mmproj.gguf"
-    run env "LLAMA_BIN=$TEST_TMP/bin" "MODEL_FILE=$TEST_TMP/absent.gguf" \
-        "DRAFT_FILE=$TEST_TMP/draft.gguf" "MMPROJ_FILE=$TEST_TMP/mmproj.gguf" "$SERVE"
-    [ "$status" -ne 0 ]
-    [[ "$output" == *"is not readable"* ]]
-    [[ "$output" == *"download-frank-large-kingjones.sh"* ]]
+@test "the KV budget fits unified memory with margin" {
+    # Pool arithmetic is the engine's documented product: slots x
+    # slot_ctx x 64 KiB. Against it: 33.4 GiB of mapped weights, the
+    # engine working set + auto prompt cache (~25 GiB of headroom
+    # allowed), host baseline and frank-small co-tenancy. The box has
+    # 121 GiB unified; the 2026-09-16 halogen era ran the same engine
+    # class here on a LARGER pool with less co-tenancy slack.
+    ctx="$(sed -nE 's/^HALOGEN_SLOT_CTX=([0-9]+)$/\1/p' "$ENV_FILE")"
+    slots="$(sed -nE 's/^HALOGEN_KV_SLOTS=([0-9]+)$/\1/p' "$ENV_FILE")"
+    pool_gib=$(( slots * ctx * 65536 / 1073741824 ))
+    total_gib=$(( 34 + pool_gib + 25 + 15 ))
+    [ "$total_gib" -le 121 ]
 }
 
-@test "the serve script refuses to start without the MTP draft head" {
-    make_stub
-    touch "$TEST_TMP/model.gguf" "$TEST_TMP/mmproj.gguf"
-    run env "LLAMA_BIN=$TEST_TMP/bin" "MODEL_FILE=$TEST_TMP/model.gguf" \
-        "DRAFT_FILE=$TEST_TMP/absent.gguf" "MMPROJ_FILE=$TEST_TMP/mmproj.gguf" "$SERVE"
+@test "the serve script refuses when the image is not in storage" {
+    podman_stub
+    good_env
+    run env PATH="$TEST_TMP/bin:$PATH" "${GOOD_ENV[@]}" STUB_DIGESTS="" "$SERVE"
     [ "$status" -ne 0 ]
-    [[ "$output" == *"MTP draft head"* ]]
+    [[ "$output" == *"not in podman storage"* ]]
+    [[ "$output" == *"podman pull"* ]]
 }
 
-@test "the serve script refuses a missing mmproj unless vision is opted out" {
-    make_stub
-    touch "$TEST_TMP/model.gguf" "$TEST_TMP/draft.gguf"
-    run env "LLAMA_BIN=$TEST_TMP/bin" "MODEL_FILE=$TEST_TMP/model.gguf" \
-        "DRAFT_FILE=$TEST_TMP/draft.gguf" "MMPROJ_FILE=$TEST_TMP/absent.gguf" "$SERVE"
+@test "the serve script refuses a missing checkpoint and names the fetch script" {
+    podman_stub
+    good_env
+    run env PATH="$TEST_TMP/bin:$PATH" "${GOOD_ENV[@]}" \
+        FRANK_LARGE_CHECKPOINT=absent.hgn "$SERVE"
     [ "$status" -ne 0 ]
-    [[ "$output" == *"set MMPROJ_FILE='' for text-only"* ]]
-    # The documented opt-out must actually start, without a --mmproj argument.
-    run env "LLAMA_BIN=$TEST_TMP/bin" "MODEL_FILE=$TEST_TMP/model.gguf" \
-        "DRAFT_FILE=$TEST_TMP/draft.gguf" "MMPROJ_FILE=" "$SERVE"
+    [[ "$output" == *"checkpoint"*"missing"* ]]
+    [[ "$output" == *"download-frank-large-halogen.sh"* ]]
+}
+
+@test "the serve script refuses a checkpoint of the wrong byte count" {
+    # A truncated curl exit-0 artifact must not boot; the model card's
+    # byte count is the contract (fleet fetch law).
+    podman_stub
+    good_env
+    run env PATH="$TEST_TMP/bin:$PATH" "${GOOD_ENV[@]}" \
+        FRANK_LARGE_CHECKPOINT_BYTES=42 "$SERVE"
+    [ "$status" -ne 0 ]
+    [[ "$output" == *"expected 42"* ]]
+}
+
+@test "the serve script refuses a symlinked tokenizer file" {
+    # A HuggingFace cache snapshot's entries are symlinks into blobs/ and
+    # dangle inside the container mount — even when the directory itself
+    # is real.
+    podman_stub
+    good_env
+    rm "$TEST_TMP/tokenizer/tok.json"
+    ln -sf /nowhere/x "$TEST_TMP/tokenizer/tok.json"
+    run env PATH="$TEST_TMP/bin:$PATH" "${GOOD_ENV[@]}" "$SERVE"
+    [ "$status" -ne 0 ]
+    [[ "$output" == *"missing or a symlink"* ]]
+}
+
+@test "the render names the container, the port map and the engine shape" {
+    podman_stub
+    good_env
+    run env PATH="$TEST_TMP/bin:$PATH" "${GOOD_ENV[@]}" "$SERVE"
     [ "$status" -eq 0 ]
-    [[ "$output" != *"--mmproj"* ]]
+    # `podman run` output is redirected to the log by design (the unit
+    # journal must not absorb engine chatter); the stub's printed argv
+    # lands there, so the render assertions read the same log.
+    output="$output
+$(cat "$TEST_TMP/serve.log")"
+    [[ "$output" == *"--name frank-large-halogen"* ]]
+    [[ "$output" == *"--publish 7770:8731"* ]]
+    [[ "$output" == *"-e HALOGEN_MODEL_ID=frank-large"* ]]
+    [[ "$output" == *"-e HALOGEN_SLOT_CTX=200700"* ]]
+    [[ "$output" == *"-e HALOGEN_KV_SLOTS=1"* ]]
+    [[ "$output" == *"-e HALOGEN_DRAFTER=2"* ]]
+    [[ "$output" == *"-e HALOGEN_CHECKPOINT=/models/qwen.hgn"* ]]
+    [[ "$output" == *"--device $TEST_TMP/qwen.hgn"* ]]
+    [[ "$output" == *"$TEST_TMP/tokenizer:/tokenizer:ro"* ]]
+    [[ "$output" == *"ghcr.io/peonist-ai/halogen@sha256:cafe"* ]]
+    # The unit's restart policy is real because the script becomes
+    # `podman wait`: the container's exit code is the service's exit.
+    [[ "$output" == *"PODMAN wait frank-large-halogen"* ]]
 }
 
-@test "the serve script fails when LLAMA_BIN is empty" {
-    touch "$TEST_TMP/model.gguf" "$TEST_TMP/draft.gguf" "$TEST_TMP/mmproj.gguf"
-    run env LLAMA_BIN= "MODEL_FILE=$TEST_TMP/model.gguf" \
-        "DRAFT_FILE=$TEST_TMP/draft.gguf" "MMPROJ_FILE=$TEST_TMP/mmproj.gguf" "$SERVE"
-    [ "$status" -ne 0 ]
-    [[ "$output" == *"LLAMA_BIN is empty"* ]]
+@test "the shipped env pins the fleet's window and shape" {
+    # 200700 is the per-conversation hard ceiling (a longer prompt is a
+    # 400 naming the limit, never a silent truncation), and the harness
+    # compaction math must fit prompt + budget under it: OMP compacts at
+    # 70% = 140490, and 140490 + 57344 = 197834 <= 200700.
+    grep -qE '^HALOGEN_SLOT_CTX=200700$' "$ENV_FILE"
+    ctx=200700
+    [ $(( ctx * 70 / 100 + 57344 )) -le "$ctx" ]
 }
 
-@test "the shipped env points at the ROCmFPX runtime and the AGENT payload" {
-    grep -qE "^LLAMA_BIN=$RUNTIME_STUB$" "$ENV_FILE"
-    grep -qE "^MODEL_FILE=$MODEL_STUB$" "$ENV_FILE"
-    grep -qE "^DRAFT_FILE=$DRAFT_STUB$" "$ENV_FILE"
-    grep -qE "^MMPROJ_FILE=$MMPROJ_STUB$" "$ENV_FILE"
-    # HIP reads bare HOST/TEMP/GPU_MAX_HEAP_SIZE out of the environment that
-    # `set -a` creates; a tunable named HOST leaks into the runtime.
+@test "no HIP-sensitive bare names leak through the env file" {
+    # HIP reads bare env vars out of a `set -a` source; a tunable named
+    # HOST or TEMP used to send the JIT cache to path 1.0 and segfault
+    # every load (kept from the llama.cpp era — the container gets only
+    # the explicit -e list, but the host env still reaches podman itself).
     ! grep -qE '^(HOST|TEMP|GPU_MAX_HEAP_SIZE)=' "$ENV_FILE"
 }
 
-@test "the argv contract keeps the fleet's port and alias" {
-    make_stub
-    touch "$TEST_TMP/model.gguf" "$TEST_TMP/draft.gguf" "$TEST_TMP/mmproj.gguf"
-    run env "LLAMA_BIN=$TEST_TMP/bin" "MODEL_FILE=$TEST_TMP/model.gguf" \
-        "DRAFT_FILE=$TEST_TMP/draft.gguf" "MMPROJ_FILE=$TEST_TMP/mmproj.gguf" \
-        ALIAS=frank-large BIND_ADDR=127.0.0.1 LISTEN_PORT=7770 "$SERVE"
-    [ "$status" -eq 0 ]
-    # The caller's values must survive the env file, or a single-run operator
-    # override silently does nothing.
-    [[ "$output" == *"--host 127.0.0.1"* ]]
-    [[ "$output" == *"--port 7770"* ]]
-    [[ "$output" == *"--alias frank-large"* ]]
-    # Vision is wired: the projector is the whole vision tower.
-    [[ "$output" == *"--mmproj $TEST_TMP/mmproj.gguf"* ]]
+@test "the deployer gates exactly the artifacts the serve script opens" {
+    # Serve and deploy must name the SAME image digest, checkpoint and
+    # tokenizer: the 2026-10-01 crash-loop was exactly these twins
+    # disagreeing mid-cutover.
+    img="$(sed -nE 's/^FRANK_LARGE_IMAGE=(.*)$/\1/p' "$ENV_FILE")"
+    grep -qF "LLM_LARGE_IMAGE=\"\${LLM_LARGE_IMAGE:-$img}\"" "$DEPLOYER"
+    dir="$(sed -nE 's/^FRANK_LARGE_MODEL_DIR=(.*)$/\1/p' "$ENV_FILE")"
+    ckpt="$(sed -nE 's/^FRANK_LARGE_CHECKPOINT=(.*)$/\1/p' "$ENV_FILE")"
+    tokf="$(sed -nE 's/^FRANK_LARGE_TOKENIZER_FILES="(.*)"$/\1/p' "$ENV_FILE")"
+    grep -qF "LLM_LARGE_TOK_DIR=\"\${LLM_LARGE_TOK_DIR:-$dir/tokenizer}\"" "$DEPLOYER"
+    grep -qF "LLM_LARGE_TOK_FILES=\"\${LLM_LARGE_TOK_FILES:-$tokf}\"" "$DEPLOYER"
+    bytes="$(sed -nE 's/^FRANK_LARGE_CHECKPOINT_BYTES=([0-9]+)$/\1/p' "$ENV_FILE")"
+    grep -qF "LLM_LARGE_CKPT_BYTES=\"\${LLM_LARGE_CKPT_BYTES:-$bytes}\"" "$DEPLOYER"
 }
 
-@test "the shipped argv is one full native window on one session" {
-    grep -qE '^CTX_SIZE=262144$' "$ENV_FILE"
-    grep -qE '^PARALLEL=1$' "$ENV_FILE"
-    # 262144 IS the model's trained window (general.context_length in the
-    # GGUF), so 1 slot gets the whole thing — no truncation, no yarn scaling.
-    #
-    # Budget, from the GGUF header: 26.28 GiB of weights + 1.57 GiB draft head
-    # + 0.87 GiB projector + 16 full-attention layers x 4 kv heads x 256 dim x
-    # 2 (K+V) x 2 B = 64 KiB/token of f16 KV (~16 GiB at 262,144) + ~0.14 GiB
-    # of DeltaNet state + a couple GiB of compute buffers, against the iGPU's
-    # ~114 GiB GTT aperture. Change CTX_SIZE and this arithmetic, not a boot
-    # log, is what tells you it no longer fits.
-    ctx="$(sed -nE 's/^CTX_SIZE=([0-9]+)$/\1/p' "$ENV_FILE")"
-    kv_gib=$((ctx * 65536 / 1073741824))
-    total_gib=$((27 + 2 + 1 + kv_gib + 3))
-    [ "$total_gib" -le 114 ]
+@test "the deployer points a missing artifact at the halogen fetch script" {
+    grep -qE 'download-frank-large-halogen\.sh' "$DEPLOYER"
+    grep -qE 'podman pull' "$DEPLOYER"
 }
 
-@test "the MTP draft contract matches the model card's measured optimum" {
-    make_stub
-    touch "$TEST_TMP/model.gguf" "$TEST_TMP/draft.gguf" "$TEST_TMP/mmproj.gguf"
-    run env "LLAMA_BIN=$TEST_TMP/bin" "MODEL_FILE=$TEST_TMP/model.gguf" \
-        "DRAFT_FILE=$TEST_TMP/draft.gguf" "MMPROJ_FILE=$TEST_TMP/mmproj.gguf" "$SERVE"
-    [ "$status" -eq 0 ]
-    # The head is the model's own nextn layer, loaded as a separate draft
-    # model because this checkpoint ships it standalone (no blk.<n>.nextn.*
-    # tensors in the target file).
-    [[ "$output" == *"--spec-type draft-mtp"* ]]
-    [[ "$output" == *"--model-draft $TEST_TMP/draft.gguf"* ]]
-    # n-max 4 is the measured optimum; n-max 8 measured WORSE (acceptance
-    # fell to 0.757 and the extra verification cost outran the savings), so
-    # the flag is a measurement, not a wish.
-    [[ "$output" == *"--spec-draft-n-max 4"* ]]
-    [[ "$output" == *"--spec-draft-n-min 0"* ]]
-    [[ "$output" == *"--spec-draft-p-min 0.0"* ]]
-    # The draft runs on the iGPU beside the target: no device-to-device hop
-    # on a link that measures single-digit GB/s.
-    [[ "$output" == *"--spec-draft-device ROCm0"* ]]
-    [[ "$output" == *"--spec-draft-ngl 99"* ]]
-}
-
-@test "KV and placement stay unquantized with flash attention on" {
-    make_stub
-    touch "$TEST_TMP/model.gguf" "$TEST_TMP/draft.gguf" "$TEST_TMP/mmproj.gguf"
-    run env "LLAMA_BIN=$TEST_TMP/bin" "MODEL_FILE=$TEST_TMP/model.gguf" \
-        "DRAFT_FILE=$TEST_TMP/draft.gguf" "MMPROJ_FILE=$TEST_TMP/mmproj.gguf" "$SERVE"
-    [ "$status" -eq 0 ]
-    [[ "$output" == *"--cache-type-k f16"* ]]
-    [[ "$output" == *"--cache-type-v f16"* ]]
-    # ROCmFPx quantized KV has no kernel on this branch, and the endpoint has
-    # the aperture to keep KV unquantized anyway.
-    [[ "$output" == *"--flash-attn on"* ]]
-    # --fit's placement heuristics do not know that 311 tensors of type 103
-    # belong on the iGPU, so the whole stack is placed explicitly instead.
-    [[ "$output" == *"--fit off"* ]]
-    [[ "$output" == *"--n-gpu-layers 999"* ]]
-}
-
-@test "the client contract wires thinking and template kwargs" {
-    make_stub
-    touch "$TEST_TMP/model.gguf" "$TEST_TMP/draft.gguf" "$TEST_TMP/mmproj.gguf"
-    run env "LLAMA_BIN=$TEST_TMP/bin" "MODEL_FILE=$TEST_TMP/model.gguf" \
-        "DRAFT_FILE=$TEST_TMP/draft.gguf" "MMPROJ_FILE=$TEST_TMP/mmproj.gguf" "$SERVE"
-    [ "$status" -eq 0 ]
-    # Without --jinja, llama.cpp silently ignores the harness's
-    # chat_template_kwargs, so thinking control disappears from the API.
-    [[ "$output" == *"--jinja"* ]]
-    # The harness reads `reasoning_content` for this endpoint, which is what
-    # the deepseek emission format produces (vLLM's `reasoning` is not it).
-    [[ "$output" == *"--reasoning-format deepseek"* ]]
-    # The template needs an unbounded thinking budget.
-    [[ "$output" == *"--reasoning-budget -1"* ]]
-}
-
-@test "the deployer gates exactly the paths the serve script opens" {
-    # A gate that checks a different file than the engine opens is the
-    # 2026-10-01 crash-loop: deploy passed, the unit restart-looped. Read the
-    # engine's own paths out of the env file and require the deployer's
-    # defaults to name the same ones.
-    for pair in "LLAMA_BIN:LLM_LARGE_RUNTIME" \
-                "MODEL_FILE:LLM_LARGE_MODEL" \
-                "DRAFT_FILE:LLM_LARGE_DRAFT" \
-                "MMPROJ_FILE:LLM_LARGE_MMPROJ"; do
-        env_key="${pair%%:*}"
-        dep_key="${pair##*:}"
-        want="$(sed -nE "s|^${env_key}=(.*)\$|\1|p" "$ENV_FILE")"
-        [ -n "$want" ]
-        got="$(grep -oE "${dep_key}:-[^}]+" "$DEPLOYER" | head -1 | sed "s|^${dep_key}:-||")"
-        [ "$got" = "$want" ]
-    done
-}
-
-@test "the deployer points a missing runtime at the fork build script" {
-    grep -qE 'build it: bash scripts/build-rocmfpx\.sh' "$DEPLOYER"
-    grep -qE 'fetch it: bash scripts/download-frank-large-kingjones\.sh' "$DEPLOYER"
-}
-
-@test "the stray-process sweep cannot take frank-small down with frank-large" {
-    # frank-small serves a different GGUF over llama-server on the SAME host.
-    # A sweep keyed on the binary name (`pkill -f llama-server`) would kill
-    # the 3090 endpoint too, so it is keyed on frank-large's own payload file.
-    grep -q "pkill -f 'Qwen3.8-27B-Q8_0_ROCMFPX_AGENT'" "$DEPLOYER"
+@test "the stray sweep cannot take frank-small or the deployer down" {
+    # frank-small also listens on this host (7771, llama-server) and the
+    # deployer's own command line contains "frank-large"; only the FULL
+    # container name matches a stray manual halogen run and nothing else.
+    grep -q "pkill -f 'frank-large-halogen'" "$DEPLOYER"
     ! grep -qE "pkill -f 'llama-server'" "$DEPLOYER"
-}
-
-@test "the build script refuses a source tree that is not the pinned commit" {
-    # The pin is load-bearing: the older builds of this fork line segfault in
-    # ggml_compute_forward_set_rows on this exact payload. A runtime that
-    # differs from what was qualified must be heard about at deploy time.
-    wrong="$(git -C "$SRC" rev-parse HEAD)"
-    run env ROCMFPX_SRC_DIR="$SRC" ROCMFPX_COMMIT="ffff$wrong" \
-        "$BUILD" --verify
-    [ "$status" -ne 0 ]
-    [[ "$output" == *"ROCMFPX_COMMIT"*mismatch* ]]
-}
-
-@test "the build script pins a real commit and the gfx1151 target" {
-    grep -qE 'ROCMFPX_COMMIT=.*[0-9a-f]{40}' "$BUILD"
-    grep -qE 'GPU_TARGET=.*gfx1151' "$BUILD"
-    grep -qE 'DGPU_TARGETS=' "$BUILD"
-    # The webui asset fetch half-failed on this box and aborted the link at
-    # 98%; the build must not depend on it.
-    grep -qE 'DLLAMA_BUILD_WEBUI=OFF' "$BUILD"
+    ! grep -qE "pkill -f 'frank-large'" "$DEPLOYER"
 }
 
 @test "the deployed unit keeps the fleet's restart and boot-persistence shape" {
@@ -266,15 +249,25 @@ STUB
     grep -qE '^WantedBy=' "$UNIT"
     grep -qE '^ExecStart=.*/serve-frank-large\.sh$' "$UNIT"
     grep -qE '^User=dank$' "$UNIT"
-    # The payload is 26 GiB of mmap'd weights plus pinned staging buffers; an
-    # RLIMIT smaller than the pinned set fails the load in a way that looks
-    # like an engine crash.
     grep -qE '^LimitMEMLOCK=infinity$' "$UNIT"
+    # The service's lifetime is the container's; stopping the service
+    # must stop the container by its real name.
+    grep -qE '^ExecStop=.*podman stop .* frank-large-halogen$' "$UNIT"
 }
 
-@test "the unit points at no retired frank-large runtime path" {
-    # The host keeps every retired payload and build on disk, so a stale path
-    # in a unit still resolves — to a runtime nobody has measured in weeks.
+@test "the unit runs the halogen runtime and no retired path" {
+    grep -q 'halogen' "$UNIT"
+    # Every retired payload/build stays on the host, so a stale path in
+    # a unit still resolves — to a runtime nobody has measured in weeks.
     ! grep -qE 'GitPackages/(ROCmFPX-kairic|llama.cpp-ik_llama|llama-builds|ciru-orca)' "$UNIT"
-    ! grep -q 'halogen' "$UNIT"
+    ! grep -qE 'kingjones-ROCmFPX' "$UNIT"
+}
+
+@test "the rollback door stays qualified: the ROCmFPX build pin survives" {
+    # Rolling back this swap means rebuilding the fork the old payload
+    # needs; the pin (real commit, gfx1151, no webui fetch) is what makes
+    # that door trustworthy months later.
+    grep -qE 'ROCMFPX_COMMIT=.*[0-9a-f]{40}' "$BUILD"
+    grep -qE 'GPU_TARGET=.*gfx1151' "$BUILD"
+    grep -qE 'DLLAMA_BUILD_WEBUI=OFF' "$BUILD"
 }

tests/frank_primary_unit.bats

@@ -1,26 +1,25 @@
 #!/usr/bin/env bats
 # Seam: frank-primary — the Dankserver LLM endpoint on :7770.
 #
-# This file asserts the tcclaviger RDNA4 vLLM ROOT-UNIT shape (the
-# 2026-10-02 correction of the 2026-10-01 BF16 cutover): the serve
-# script drives `podman run` DIRECTLY (the image entrypoint execs
-# `vllm serve`), with every engine flag qualified by boot probe and carried
-# in the repo env file.
-# The seam pins what the flags alone do not say: the
-# KV qualification law (the night ladder: 262144/245760 OOM at graph
-# capture; 241664 boots at 732,359 tokens / 3.03x, util 0.98;
-# the bf16-KV trade (double the attention-KV bytes/token, ~14% decode cost,
-# clean cache + 262k windows), the automatic-offload law (the fork plans
-# the tiered-expert offload itself; no flag exists to set), the self-healing
-# law (the run returns before the engine is up, so the serve script must
-# wait on the container for Restart=on-failure to mean anything), the
-# memlock law that makes the pinned host rows legal, the deployer's
-# image/payload gates with the false-green MainPID check, and the measured
-# laws: expert prefetch ABSENT (36% precision), NO speculative decode (the
-# no-spec family measured fastest once the prefetch tax was removed; MTP
-# re-measures pending), gather geometry 32x64 PRESENT (+18.7% decode),
-# MNBT 1024 (Frank's lock), and the client contract (reasoning parser qwen3,
-# tool parser qwen3_xml, preserve_thinking on).
+# This file asserts the FreeToken (PR #587 ROCm) ROOT-UNIT shape (the
+# 2026-10-05 cutover from the tcclaviger RDNA4 vLLM fork, git tag
+# frank-primary-vllm-shapeG-20261003): the serve script drives `podman run`
+# DIRECTLY (the image carries `ft serve`, built on the host from
+# scripts/docker/frank-primary-freetoken.Dockerfile), one R9700, with every
+# engine value carried in the repo env file. The seam pins what flags alone
+# do not say: the KV-pool math (300k tokens at 24 KiB/token clears the
+# harness worst case of 177,356), the deployed single-GPU shape
+# (--tensor-parallel-size EXISTS in FreeToken — a 2-card TP shape is future
+# tuning; HIP_VISIBLE_DEVICES pins THIS shape to one card), the host-RAM
+# law (experts + PLE pin ~170 GiB — the memlock law that makes the pin
+# legal), the self-healing law (the run returns before the engine is up, so
+# the serve script must wait on the container for Restart=always to mean
+# anything), the image-existence gate (a local build carries no
+# RepoDigests — the old digest grep refused exactly this shape; pinning
+# lives in the Dockerfile), the deployer's payload gates with the
+# false-green MainPID check, and the client contract (parsers AUTO:
+# reasoning_content emission is the engine's native behavior — models.yml
+# maps reasoningContentField on the harness side, not here).
 
 load helpers
 
@@ -29,29 +28,30 @@ SERVE="$REPO_ROOT/scripts/serve-frank-primary.sh"
 ENV_FILE="$REPO_ROOT/scripts/frank-primary.env"
 REMOTE_DEPLOY="$REPO_ROOT/scripts/deploy-remote.sh"
 DOWNLOAD="$REPO_ROOT/scripts/download-frank-primary.sh"
+DOCKERFILE="$REPO_ROOT/scripts/docker/frank-primary-freetoken.Dockerfile"
 
-@test "frank-primary unit, serve script and env exist in scripts/" {
+@test "frank-primary unit, serve script, env and Dockerfile exist" {
     [ -f "$UNIT" ]
     [ -x "$SERVE" ]
     [ -f "$ENV_FILE" ]
+    [ -f "$DOCKERFILE" ]
 }
 
-@test "frank-primary is boot-persistent, self-healing and unprivileged" {
+@test "frank-primary unit is boot-persistent, self-healing and unprivileged" {
     # Root unit parity with frank-large/frank-small: enable is the [Install]
-    # section, the restart policy survives a crash, and the tiered-expert
-    # pin needs an unbounded memlock (a root unit grants it via PID1; the
-    # dank session's own hard limit is 8 MiB and cannot be raised without
-    # restarting user@1000.service).
+    # section, the restart policy survives a crash, and the pinned host RAM
+    # (expert rows + PLE table) needs an unbounded memlock (a root unit
+    # grants it via PID1; the dank session's own hard limit is 8 MiB).
     grep -q '^WantedBy=multi-user.target$' "$UNIT"
-    # Restart=always, not on-failure: a runtime worker OOM makes vLLM's API
-    # server exit CLEAN, so ExecStart returns 0 and on-failure never fires
-    # (measured 2026-10-01 — the endpoint stayed down until a human noticed).
+    # Restart=always, not on-failure: the vLLM-era lesson carries over — a
+    # clean front-end exit after a worker fault reads as success to systemd,
+    # so on-failure never fires.
     grep -q '^Restart=always$' "$UNIT"
     ! grep -q '^Restart=on-failure$' "$UNIT"
-    # Exiting the engine takes as long as vLLM's drain: ExecStop runs
-    # `podman stop -t 120`, so the stop timeout MUST exceed 120 s or systemd
-    # kills the stop and records 'Failed with result timeout' (measured
-    # 2026-10-01). Guard the relationship, not just the literal.
+    # ExecStop runs `podman stop -t 120`, so the stop timeout MUST exceed
+    # 120 s or systemd kills the stop and records 'Failed with result
+    # timeout' (measured 2026-10-01). Guard the relationship, not the
+    # literal.
     stop_t=$(awk -F= '/^TimeoutStopSec=/{print $2}' "$UNIT")
     [ -n "$stop_t" ] && [ "$stop_t" -gt 120 ]
     grep -q '^ExecStop=.*podman stop -t 120' "$UNIT"
@@ -69,222 +69,121 @@ DOWNLOAD="$REPO_ROOT/scripts/download-frank-primary.sh"
     grep -q 'FRANK_PRIMARY_ENV_FILE=/home/dank/serverAppConfigs/scripts/frank-primary.env' "$UNIT"
 }
 
-@test "the unit stops the container child explicitly" {
+@test "the unit stops the container child explicitly — container identity is the unit name" {
     # The serve script blocks on podman wait, which cannot stop the
     # container child; systemd stop goes through ExecStop or the engine
-    # outlives the unit and holds both GPUs. The container name must match
-    # the env file's identity.
+    # outlives the unit and holds the GPU. Frank's 2026-10-05 call: the
+    # container name IS the service name (frank-primary), and the vLLM-era
+    # name frank-primary-qwen moved into the legacy cleanup list.
     grep -q '^ExecStop=/usr/bin/podman stop' "$UNIT"
-    grep -q 'frank-primary-qwen' "$UNIT"
-    grep -q '^FRANK_PRIMARY_CONTAINER=frank-primary-qwen$' "$ENV_FILE"
+    grep -q 'podman stop -t 120 frank-primary$' "$UNIT"
+    grep -q '^FRANK_PRIMARY_CONTAINER=frank-primary$' "$ENV_FILE"
+    grep -q 'frank-primary-qwen' "$ENV_FILE"
 }
 
-@test "the serve shape is direct podman run, not a launcher or shim" {
-    # The tcclaviger stack is fully expressed by flags; the R9V launcher,
-    # docker->podman shim and setup-state layering are the RETIRED shape.
+@test "the serve shape is direct podman run, not a launcher, shim, or the vLLM fork" {
     grep -q 'podman run' "$SERVE"
     ! grep -q 'launch.sh' "$SERVE"
     ! grep -q 'podman-shim' "$SERVE"
     ! grep -q 'R9V_STATE_FILE' "$SERVE"
-    ! grep -q -- '--model-draft' "$SERVE"
-    ! grep -q -- '--spec-type' "$SERVE"
+    # The tcclaviger vLLM fork is RETIRED — its image name and its
+    # fork-specific flags must not survive in the FreeToken run line
+    # (TP=2 is gone with the fork; FreeToken serves ONE GPU).
+    ! grep -q 'tcclaviger' "$SERVE"
+    ! grep -q 'tcclaviger' "$ENV_FILE"
+    ! grep -q -- '--tensor-parallel-size' "$SERVE"
+    ! grep -q -- '--kv-cache-dtype' "$SERVE"
+    ! grep -q -- '--max-num-seqs' "$SERVE"
 }
 
-@test "the qualified engine shape is in the env file" {
-    # 2026-10-01 night ladder (Frank's method), then a runtime-survivability
-    # correction, then Frank's 2026-10-02 decision to give the pool back to
-    # the expert arena. SEQS=3 bf16 topped out at 241664 (KV 732,359 / 3.03x);
-    # SEQS=6 fp8 does NOT fit (6 x 241664 asked 10.70 GiB against 9.18 GiB
-    # free); SEQS=5 fp8 x 262144 BOOTED (KV 1,350,182 / 5.15x) but left ~0
-    # bytes free at runtime and a real client request (995-token prompt with
-    # max_tokens=261149) killed the worker with `Tried to allocate 50.00 MiB
-    # ... 0 bytes free`. 200704 sits below the 241664 rung while leaving a
-    # full 2.95 GiB/rank to cut loose for the arena. UTIL stays 0.98
-    # (lowering it shrinks the engine's own budget instead of buying slack).
-    # FINAL (2026-10-02, Frank): TWO windows at 200704 with FP8 KV and a pool
-    # cut for the arena. fp8 costs 7,701 B/token (measured 6,138 attention +
-    # 1,563 mamba/side), so 2 x 200704 = 401,408 tokens needs 3.091e9 B;
-    # pinned at 3.15e9 (~405k tokens, ~44 MB margin). The arena is what
-    # remains — the planner sizes it AFTER the KV override — so the 6,248
-    # slots of the 6.1e9 shape become roughly 7,500 here.
-    grep -q '^FRANK_PRIMARY_MAXLEN=200704$' "$ENV_FILE"
-    grep -q '^FRANK_PRIMARY_SEQS=2$' "$ENV_FILE"
-    grep -q '^FRANK_PRIMARY_UTIL=0.98$' "$ENV_FILE"
-    grep -q '^FRANK_PRIMARY_KV_BYTES=3150000000$' "$ENV_FILE"
-    grep -q '^FRANK_PRIMARY_MNBT=1024$' "$ENV_FILE"
-    grep -q '^FRANK_PRIMARY_KV_DTYPE=fp8$' "$ENV_FILE"
-    # --expert-precise-memory is NOT set: with bf16 2 x 262144 it put the
-    # boot 0.13 GiB from the 31.86 GiB ceiling (arena 16.65 device-resident)
-    # and crash-looped at graph capture. The serve script keeps the opt-in
-    # flag; the smaller pool here is the arena it would have been paying for.
-    ! grep -q '^FRANK_PRIMARY_PRECISE_MEMORY=' "$ENV_FILE"
-    grep -q '^FRANK_PRIMARY_GATHER_CHUNKS=32$' "$ENV_FILE"
-    grep -q '^FRANK_PRIMARY_GATHER_LANES=64$' "$ENV_FILE"
-    ! grep -q '^FRANK_PRIMARY_SPEC=' "$ENV_FILE"
+@test "the FreeToken engine shape is in the env file" {
+    # KV pool: TOKENS, not bytes. 24 KiB/token (12 full-attention layers x
+    # 2 KV heads x 256 head_dim x K+V x f16) — the 36 GDN layers use fixed
+    # per-seq state, not per-token KV. 299,968 tokens = 7.0 GiB and clears
+    # the harness worst case (140,492-token prompt + 36,864 max_tokens =
+    # 177,356); one NATIVE 262,144-token conversation + full budget also
+    # fits. The value must stay a MULTIPLE OF 64: the auto attention
+    # backend is qsa_sparse, which overrides page size to 64 (measured
+    # first boot — 300,000 was refused). The arithmetic guard keeps future
+    # resizes honest.
+    grep -q '^FRANK_PRIMARY_FT_NUM_TOKENS=299968$' "$ENV_FILE"
+    nt=$(awk -F= '/^FRANK_PRIMARY_FT_NUM_TOKENS=/{print $2}' "$ENV_FILE")
+    [ "$nt" -ge 177356 ]
+    [ $((nt % 64)) -eq 0 ]
+    # First-boot conservative shape: 2 decode slots, 10% expert LRU
+    # (~12 GiB of the ~120 GiB fp8 expert set), output ceiling = the
+    # harness maxTokens, single R9700 at index 0.
+    grep -q '^FRANK_PRIMARY_FT_MAX_RUNNING=2$' "$ENV_FILE"
+    grep -q '^FRANK_PRIMARY_FT_MOE_RATE=0.10$' "$ENV_FILE"
+    grep -q '^FRANK_PRIMARY_FT_MAX_OUTPUT_TOKENS=36864$' "$ENV_FILE"
+    grep -q '^FRANK_PRIMARY_FT_GPU=0$' "$ENV_FILE"
     grep -q '^FRANK_PRIMARY_MODEL_SHARDS=131$' "$ENV_FILE"
 }
 
-@test "the pool is sized as the arena floor, not the context floor" {
-    # THE LAW this seam pins: the fork's expert planner sizes the arena
-    # AFTER the KV override, so every byte cut out of KV_BYTES becomes
-    # arena. A context-size change must move the pool with it, or the
-    # reduction buys nothing. 2 x 200704 = 401,408 tokens at fp8's 7,701
-    # B/token is 3.091e9 B; the pin is 3.15e9, ~44 MB of margin above the
-    # floor — the same convention the 6.1e9 shape used.
-    floor=$(awk -v n="401408" -v b="7701" 'BEGIN{printf "%.0f", n*b}')
-    [ "$floor" = "3091243008" ]
-    kv=$(grep '^FRANK_PRIMARY_KV_BYTES=' "$ENV_FILE" | cut -d= -f2)
-    [ "$kv" = "3150000000" ]
-    margin=$(( kv - floor ))
-    [ "$margin" -gt 40000000 ]
-    [ "$margin" -lt 100000000 ]
-}
-
 @test "the serve script wires the engine flags from the env" {
-    grep -q -- '--max-num-seqs "$FRANK_PRIMARY_SEQS"' "$SERVE"
-    grep -q -- '--max-model-len "$FRANK_PRIMARY_MAXLEN"' "$SERVE"
-    grep -q -- '--kv-cache-memory-bytes "$FRANK_PRIMARY_KV_BYTES"' "$SERVE"
-    grep -q -- '--kv-cache-dtype "$FRANK_PRIMARY_KV_DTYPE"' "$SERVE"
-    grep -q -- '--max-num-batched-tokens "$FRANK_PRIMARY_MNBT"' "$SERVE"
-    grep -q -- '-e R4D_LRU_GATHER_CHUNKS="$FRANK_PRIMARY_GATHER_CHUNKS"' "$SERVE"
-    grep -q -- '-e R4D_LRU_GATHER_LANES="$FRANK_PRIMARY_GATHER_LANES"' "$SERVE"
-    ! grep -q -- '--speculative-config' "$SERVE"
-    grep -q -- '--tensor-parallel-size 2' "$SERVE"
-    grep -q -- '--enable-prefix-caching' "$SERVE"
-    grep -q -- '--trust-remote-code' "$SERVE"
-    # DCP is opt-in: the flag is passed only when FRANK_PRIMARY_DCP is set
-    # (DCP reuses the TP ranks when PCP is off, so TP=2 + DCP=2 fits two GPUs).
-    grep -q -- '--decode-context-parallel-size "$FRANK_PRIMARY_DCP"' "$SERVE"
-    ! grep -q '^FRANK_PRIMARY_DCP=' "$ENV_FILE"
-    # --expert-precise-memory selects the fitted overhead (5.25 GiB) over the
-    # default (7.60 GiB = fitted + compile/graph-capture headroom), which only
-    # moves the plan's `room` term and therefore the expert arena: ~1,025
-    # more slots at 2.344 MiB each.
-    grep -q -- '${FRANK_PRIMARY_PRECISE_MEMORY:+--expert-precise-memory}' "$SERVE"
+    grep -q 'ft serve --model /models' "$SERVE"
+    grep -q -- '--served-model-name "$FRANK_PRIMARY_SERVED_NAME"' "$SERVE"
+    grep -q -- '--host 0.0.0.0 --port 7770' "$SERVE"
+    grep -q -- '--gpu "$FRANK_PRIMARY_FT_GPU"' "$SERVE"
+    grep -q -- '--num-tokens "$FRANK_PRIMARY_FT_NUM_TOKENS"' "$SERVE"
+    grep -q -- '--max-running-requests "$FRANK_PRIMARY_FT_MAX_RUNNING"' "$SERVE"
+    grep -q -- '--max-output-tokens "$FRANK_PRIMARY_FT_MAX_OUTPUT_TOKENS"' "$SERVE"
+    grep -q -- '--moe-cache-rate "$FRANK_PRIMARY_FT_MOE_RATE"' "$SERVE"
+    # Single-GPU law: visibility is pinned to one card so in-container
+    # index 0 is the right R9700 (mirrors the env's FT_GPU).
+    grep -q -- '-e HIP_VISIBLE_DEVICES="$FRANK_PRIMARY_FT_GPU"' "$SERVE"
+    # The boot-ladder escape hatch is opt-in: unset renders NO extra argv.
+    grep -q -- '${FRANK_PRIMARY_FT_EXTRA_ARGS:+$FRANK_PRIMARY_FT_EXTRA_ARGS}' "$SERVE"
+    ! grep -q '^FRANK_PRIMARY_FT_EXTRA_ARGS=.' "$ENV_FILE"
 }
 
-@test "tools and reasoning are wired — the client contract is served" {
-    # 2026-10-01: the rollback-era trio served NO tool dialect and left
-    # thinking inline in `content` with a dangling `</think>` and a null
-    # `reasoning` field (measured: content 'We need answer...Brief.\n</think>
-    # \n\n391') — the malformed thinking block clients choke on. The working
-    # values are the ones ecaad36 shipped for this model and this fork's
-    # registries confirm: reasoning parser `qwen3`, tool parser `qwen3_xml`.
-    # preserve_thinking keeps prior thinking blocks across turns.
-    grep -q '^FRANK_PRIMARY_REASONING_PARSER=qwen3$' "$ENV_FILE"
-    grep -q '^FRANK_PRIMARY_TOOL_PARSER=qwen3_xml$' "$ENV_FILE"
-    grep -q -- '--reasoning-parser "$FRANK_PRIMARY_REASONING_PARSER"' "$SERVE"
-    grep -q -- '--enable-auto-tool-choice' "$SERVE"
-    grep -q -- '--tool-call-parser "$FRANK_PRIMARY_TOOL_PARSER"' "$SERVE"
-    grep -q -- '--default-chat-template-kwargs "$FRANK_PRIMARY_CHAT_TEMPLATE_KWARGS"' "$SERVE"
-    grep -q 'preserve_thinking' "$ENV_FILE"
-    grep -q 'enable_thinking' "$ENV_FILE"
+@test "parsers are AUTO — the engine infers them natively, the harness maps the field" {
+    # FreeToken infers the qwen3.8 reasoning/tool dialects from the model
+    # family and emits thinking in `reasoning_content` natively — NO
+    # --reasoning-parser/--tool-call-parser override flags. The vLLM-era
+    # client-contract wiring (qwen3 parser, qwen3_xml tool parser,
+    # default-chat-template-kwargs) is gone; models.yml's
+    # reasoningContentField flip is the harness-side contract and lives in
+    # the operator's home, not this repo.
+    ! grep -q -- '--reasoning-parser' "$SERVE"
+    ! grep -q -- '--tool-call-parser' "$SERVE"
+    ! grep -q -- '--enable-auto-tool-choice' "$SERVE"
+    ! grep -q -- '--default-chat-template-kwargs' "$SERVE"
+    ! grep -q 'FRANK_PRIMARY_REASONING_PARSER' "$ENV_FILE"
+    ! grep -q 'FRANK_PRIMARY_TOOL_PARSER' "$ENV_FILE"
 }
 
-@test "expert offload is AUTOMATIC — the payload cannot fit VRAM and no flag sets it" {
-    # ~86 GiB/rank of FP8 trunk against 32 GiB of VRAM: the fork's tiered-
-    # expert planner moves the overflow to pinned host rows by itself (the
-    # arena 6,574 slots x 2.344 MiB plus host rows). There is no
-    # --enable-expert-offload flag in this arm; the legality of the pin is
-    # the unit's unbounded LimitMEMLOCK.
-    ! grep -q -- '--enable-expert-offload' "$SERVE"
-    ! grep -q -- '--expert-precise-memory' "$SERVE"
-    grep -q '^LimitMEMLOCK=infinity$' "$UNIT"
-    grep -q -- '--ipc=host --ulimit memlock=-1:-1' "$SERVE"
-}
-
-@test "speculative decode is ABSENT BY MEASUREMENT" {
-    # The 2026-10-01 overnight redo (prefetch OFF) re-measured the whole
-    # spec family against no-spec at matched geometry: no-spec measured
-    # 41.46/41.29/41.47 tok/s while the spec-era incumbent's 37-40 carried
-    # the prefetch tax, and MTP's 2.6 GiB/rank drafter reservation halves
-    # the pool (729,545 vs 975,342 tokens). The MTP3/4/6 untaxed re-measures
-    # never completed; until one shows a real win, no spec is the incumbent.
-    ! grep -q '^FRANK_PRIMARY_SPEC=' "$ENV_FILE"
-    ! grep -q -- '--speculative-config' "$SERVE"
-}
-
-@test "KV dtype is bf16 (auto) — exactness, paid for in pool bytes" {
-    # --kv-cache-dtype auto = bf16 (unquantized). Frank's call 2026-10-01.
-    # bf16 costs 13,839 B/token against fp8's 7,701 — decomposed as 12,276
-    # attention + 1,563 mamba/side (dtype-independent) versus fp8's 6,138 +
-    # 1,563. That halves the pool capacity per byte, which is why the contract
-    # is two full-size windows instead of three; in exchange the KV is exact
-    # (no quantization error) and measured +4% decode at shallow depth, where
-    # fp8's dequant cost dominates over the extra bytes read.
-    grep -q '^FRANK_PRIMARY_KV_DTYPE=fp8$' "$ENV_FILE"
-    grep -q -- '--kv-cache-dtype "$FRANK_PRIMARY_KV_DTYPE"' "$SERVE"
-}
-
-@test "gather geometry 32x64 is PRESENT BY MEASUREMENT" {
-    # R4D_LRU_GATHER_CHUNKS=32 / R4D_LRU_GATHER_LANES=64: the overnight
-    # sweep's biggest lever, +18.7% decode vs default geometry, and the win
-    # GROWS with depth (39.5@166 -> 54.1@1087 -> 49.3@4064 in the fp8 era).
-    # g64x128 is past the peak; g8x16 mild. They are container ENV (the
-    # fork's knobs), not engine flags — passed through with -e.
-    grep -q '^FRANK_PRIMARY_GATHER_CHUNKS=32$' "$ENV_FILE"
-    grep -q '^FRANK_PRIMARY_GATHER_LANES=64$' "$ENV_FILE"
-    grep -q -- '-e R4D_LRU_GATHER_CHUNKS="$FRANK_PRIMARY_GATHER_CHUNKS"' "$SERVE"
-}
-
-@test "MNBT is at Frank's 1024 with the arena intact" {
-    # MNBT 512 doubles TTFT at 4064 ctx (2.46 s vs 1.36 s); 1024-4096 are
-    # interchangeable on decode AND identical on arena (the sweep measured
-    # 6,499 slots at every rung <=4096); 12288 costs 2.5% plus arena.
-    # 1024 is the 2026-10-02 setting — the pool was cut to feed the expert
-    # arena, so the workspace floor matters more than it did at 2048: the
-    # arena budget is what is left, and 1024-4096 pay nothing for it.
-    grep -q '^FRANK_PRIMARY_MNBT=1024$' "$ENV_FILE"
-    grep -q -- '--max-num-batched-tokens "$FRANK_PRIMARY_MNBT"' "$SERVE"
-}
-
-@test "expert prefetch is absent BY LAW, not by omission" {
-    # Telemetry (2026-09-29): the prefetcher served 13,369 of 36,720
-    # prefetched slabs (36% precision) — roughly half the gather traffic was
-    # waste, and the "savings" would have been installs anyway. Deleting the
-    # flag moved decode 26.8 -> 40.6 tok/s. Any future flag must come with a
-    # fresh A/B, so the serve script must NOT wire it unconditionally.
-    ! grep -q -- '--expert-prefetch' "$SERVE"
-    ! grep -q 'expert_prefetch' "$ENV_FILE"
-    grep -q 'PREFETCH IS ABSENT BY LAW' "$ENV_FILE"
-}
-
-@test "TunableOp stays off" {
-    # CLAV_TUNABLEOP_SWEEP silently skips under expert offload unless
-    # CLAV_TUNABLEOP_WITH_OFFLOAD=1, and forcing it corrupts numerics
-    # (repetition loops, early halts, invalid measurements).
-    ! grep -q 'TUNABLEOP' "$ENV_FILE"
-    ! grep -q 'TUNABLEOP' "$SERVE"
-}
-
-@test "the serve script gates the digest-pinned image and the payload" {
-    # The image is pinned by OCI manifest digest (the floating :latest moved
-    # once already). The payload gate checks config.json and the exact shard
-    # count from the env — a missing shard means a guaranteed long-timeout
-    # boot instead of a fast failure.
+@test "the serve script gates the image, the payload and the GPU nodes" {
+    # The image gate is EXISTENCE (podman image exists): the FreeToken
+    # image is built ON THE HOST — a local tag carries no RepoDigests — and
+    # pinning lives in the Dockerfile (pinned PR-head SHA, base image).
+    # The payload gate checks config.json and the exact shard count from
+    # the env — a missing shard means a guaranteed long-timeout boot
+    # instead of a fast failure.
     grep -q 'FRANK_PRIMARY_IMAGE' "$SERVE"
-    grep -q 'RepoDigests' "$SERVE"
+    grep -q 'podman image exists' "$SERVE"
     grep -q 'FRANK_PRIMARY_MODEL_SHARDS' "$SERVE"
     grep -q 'config.json' "$SERVE"
+    grep -q 'model.safetensors.index.json' "$SERVE"
     grep -q '/dev/kfd' "$SERVE"
     grep -q -- '--device /dev/dri' "$SERVE"
-    ! grep -q 'tcclaviger/vllm:latest' "$ENV_FILE"
 }
 
 @test "the serve script clears stale containers and self-heals via podman wait" {
     # A crashed unit can leave the container name registered forever after;
-    # the legacy identities (the retired R9V container and the ad hoc
-    # experiment container) hold the same GPUs and die here too. Then block
-    # on the container so a crash is a nonzero ExecStart.
+    # the legacy identities (the vLLM-era frank-primary-qwen container, the
+    # retired R9V container and the ad hoc experiment containers) hold the
+    # same GPU and die here too. Then block on the container so a crash is
+    # a nonzero ExecStart.
     grep -q 'FRANK_PRIMARY_LEGACY_CONTAINERS' "$ENV_FILE"
+    grep -q 'frank-primary-qwen' "$ENV_FILE"
     grep -q 'frank-primary-r9v' "$ENV_FILE"
     grep -q 'qwen-fp8' "$ENV_FILE"
     grep -q 'exec podman wait "$FRANK_PRIMARY_CONTAINER"' "$SERVE"
 }
 
 @test "frank-primary holds the single fleet port under its own alias" {
-    # The scratch identity (qwen3.8-flash-next on :7773) must never leak
-    # into production: LAN publish of :7770, the harness's model id.
+    # LAN publish of :7770, the harness's model id.
     grep -q '^FRANK_PRIMARY_PUBLISH=7770:7770$' "$ENV_FILE"
     grep -q '^FRANK_PRIMARY_SERVED_NAME=frank-primary$' "$ENV_FILE"
 }
@@ -306,23 +205,51 @@ DOWNLOAD="$REPO_ROOT/scripts/download-frank-primary.sh"
     ! grep -qE '^(export )?(TEMP|HOST|GPU_MAX_HEAP_SIZE)=' "$SERVE"
 }
 
-@test "the deployer gates the image digest and payload, then health-checks" {
-    # The deployer refuses when the pinned image is absent from rootless
-    # storage or the payload is missing/wrong-sharded — then clears stale
-    # containers (including the legacy identities), waits out the port,
+@test "the Dockerfile pins the FreeToken source and follows the ROCm install law" {
+    # The engine is the PR #587 head, NOT a moving branch: a future push to
+    # rocm714_torch214 would otherwise silently change the deployed engine.
+    grep -q '77a7201f98f00d93f618c6cf8b81ffe8360e0899' "$DOCKERFILE"
+    grep -q 'pull/587/head' "$DOCKERFILE"
+    # docs/install_amd.md's build law: --no-build-isolation --no-deps (a
+    # plain `pip install -e .` on ROCm can REPLACE the ROCm torch with a
+    # CUDA build via its NVIDIA-authored dependency set), curated portable
+    # deps, flashlib --no-deps.
+    grep -q -- '--no-build-isolation --no-deps -e .' "$DOCKERFILE"
+    grep -q 'flashlib==0.3.0' "$DOCKERFILE"
+    # The CUDA-only backends have no ROCm builds and are intentionally
+    # absent (the PR's 12 pytest failures are exactly those backends;
+    # FreeToken falls back to its Triton kernels).
+    ! grep -q 'flashinfer' "$DOCKERFILE"
+    ! grep -q 'sgl_kernel' "$DOCKERFILE"
+    ! grep -q 'nvidia-cutlass-dsl' "$DOCKERFILE"
+    # The base image is digest-pinned (the registry tag is not trusted).
+    grep -qE '^FROM rocm/pytorch:.*@sha256:[0-9a-f]{64}$' "$DOCKERFILE"
+}
+
+@test "the deployer gates the image, payload and containers, then health-checks" {
+    # The deployer refuses when the image is absent from rootless storage
+    # or the payload is missing/wrong-sharded — then clears stale containers
+    # (including the legacy identities), waits out the port,
     # relinks/enables/restarts the root unit and health-gates it.
-    grep -q 'LLM_PRIMARY_CONTAINER=frank-primary-qwen' "$REMOTE_DEPLOY"
+    grep -q 'LLM_PRIMARY_CONTAINER=frank-primary$' "$REMOTE_DEPLOY"
     grep -q 'FRANK_PRIMARY_IMAGE' "$REMOTE_DEPLOY"
-    grep -q "podman images --format '{{.RepoDigests}}'" "$REMOTE_DEPLOY"
+    grep -q 'podman image exists' "$REMOTE_DEPLOY"
     grep -q 'FRANK_PRIMARY_MODEL_SHARDS' "$REMOTE_DEPLOY"
     grep -q 'config.json' "$REMOTE_DEPLOY"
+    grep -q 'frank-primary-qwen' "$REMOTE_DEPLOY"
     grep -q 'qwen-fp8' "$REMOTE_DEPLOY"
     grep -q 'glm-probe' "$REMOTE_DEPLOY"
     grep -q 'wait_port_free 7770' "$REMOTE_DEPLOY"
     grep -q 'relink_unit frank-primary' "$REMOTE_DEPLOY"
     grep -q 'systemctl enable frank-primary.service' "$REMOTE_DEPLOY"
     grep -q 'systemctl restart frank-primary.service' "$REMOTE_DEPLOY"
-    grep -q 'wait_for_health frank-primary http://127.0.0.1:7770/health 1800' "$REMOTE_DEPLOY"
+    # The health gate reads the BODY, not just the HTTP code: FreeToken
+    # returns 200 with {"status":"loading"} while the engine loads (the
+    # FastAPI front end binds before the scheduler is up — a bare 200
+    # false-greened a deploy at 6 s while 131 shards were still paging in,
+    # measured 2026-10-06). Only status:"ok" counts.
+    grep -q 'curl -fsS http://127.0.0.1:7770/health' "$REMOTE_DEPLOY"
+    grep -q 'never reached /health status ok' "$REMOTE_DEPLOY"
 }
 
 @test "the deployer will not report green on a stray port-holder" {

tests/frank_small_unit.bats

@@ -1,12 +1,12 @@
 #!/usr/bin/env bats
 # Seam: the frank-small llama.cpp root unit on dankclaw.
-# frank-small is the RTX 3090 endpoint; since 2026-10-01 night it serves
-# bartowski/MiMo-V2.6-Distill-Qwen-9B-GGUF:Q5_K_M via a CUDA (sm_86) build of
-# llama.cpp master, chosen for the ASYMMETRIC KV Frank asked for: K f16
-# (unquantized) + V q8_0, which only llama.cpp exposes per-tensor. Since
-# 2026-10-02 it runs THREE contexts and carries a vision projector, and the
-# card must keep 2-3 GiB free. This seam runs the serve script against a stub
-# engine and asserts on the argv it actually execs, so what is pinned is the
+# frank-small is the RTX 3090 endpoint; since 2026-10-05 it serves
+# ornith-ai/Ornith-1.5-9B-GGUF:Q8_0 via a CUDA (sm_86) build of
+# llama.cpp master, chosen because it is the only runtime here that moves K and
+# V independently. Since 2026-10-03 it runs TWO unquantized contexts of
+# 200,704 tokens each and carries a vision projector, and the card must keep
+# 2-3 GiB free. This seam runs the serve script against a stub engine and
+# asserts on the argv it actually execs, so what is pinned is the
 # command the operator gets.
 
 load helpers
@@ -16,9 +16,9 @@ SERVE="$REPO_ROOT/scripts/serve-frank-small.sh"
 ENV_FILE="$REPO_ROOT/scripts/frank-small.env"
 REMOTE_DEPLOY="$REPO_ROOT/scripts/deploy-remote.sh"
 ENV_LARGE="$REPO_ROOT/scripts/frank-large.env"
-TARGET_STUB="MiMo-V2.6-Distill-Qwen-9B-Q5_K_M.gguf"
+TARGET_STUB="Ornith-1.5-9B-Q8_0.gguf"
 # Same basename as the shipped projector: the test env repoints it into $STAGE.
-MM_STUB="mmproj-MiMo-V2.6-Distill-Qwen-9B-f16.gguf"
+MM_STUB="mmproj-Ornith-1.5-9B-BF16.gguf"
 
 setup() {
     STAGE="$(mktemp -d)"
@@ -52,6 +52,8 @@ BIND_ADDR=0.0.0.0
 LISTEN_PORT=7771
 CTX_SIZE=$(sed -nE 's/^CTX_SIZE=([0-9]+)$/\1/p' "$ENV_FILE")
 PARALLEL=$(sed -nE 's/^PARALLEL=([0-9]+)$/\1/p' "$ENV_FILE")
+BATCH_SIZE=$(sed -nE 's/^BATCH_SIZE=([0-9]+)$/\1/p' "$ENV_FILE")
+UBATCH_SIZE=$(sed -nE 's/^UBATCH_SIZE=([0-9]+)$/\1/p' "$ENV_FILE")
 CACHE_TYPE_K=$(sed -nE 's/^CACHE_TYPE_K=(.+)$/\1/p' "$ENV_FILE")
 CACHE_TYPE_V=$(sed -nE 's/^CACHE_TYPE_V=(.+)$/\1/p' "$ENV_FILE")
 NGL=$(sed -nE 's/^NGL=([0-9]+)$/\1/p' "$ENV_FILE")
@@ -79,9 +81,9 @@ argv_value() {
     grep -q 'FRANK_SMALL_ENV_FILE=/home/dank/serverAppConfigs/scripts/frank-small.env' "$UNIT"
 }
 
-@test "the shipped env points at the llama.cpp runtime and the Q5_K_M GGUF" {
+@test "the shipped env points at the llama.cpp runtime and the Ornith-1.5-9B Q8_0 GGUF" {
     grep -qE '^LLAMA_BIN=/home/dank/GitPackages/llama-master/build-cuda/bin/llama-server$' "$ENV_FILE"
-    grep -qE "^MODEL_FILE=/home/dank/models/MiMo-Q5KM/$TARGET_STUB$" "$ENV_FILE"
+    grep -qE "^MODEL_FILE=/home/dank/models/Ornith-Q8/$TARGET_STUB$" "$ENV_FILE"
 }
 
 @test "the deployer gates on exactly the payload paths the env file ships" {
@@ -98,7 +100,7 @@ argv_value() {
     done
 }
 
-@test "the shipped env serves the fleet shape: three 180224-token contexts" {
+@test "the shipped env serves the fleet shape: one native 262144-token context" {
     serve_argv
     [ "$status" -eq 0 ]
     [ "$(argv_value --alias)" = "frank-small" ]
@@ -108,43 +110,81 @@ argv_value() {
     [ "$(argv_value --n-gpu-layers)" -gt 0 ]
     # llama.cpp divides --ctx-size across slots and a mamba hybrid cannot
     # share one cache, so the observable contract is the per-slot window.
-    [ "$(argv_value --parallel)" -eq 3 ]
+    # 2026-10-05 (Frank): NO KV quantization forces ONE slot at the NATIVE
+    # window — Ornith's 9 full-attn layers price f16 KV at 36 KiB/token, and
+    # two 200,704 slots cannot fit the 2-3 GiB floor on 24 GiB with Q8
+    # weights. The window per conversation ROSE 200,704 -> 262,144.
+    [ "$(argv_value --parallel)" -eq 1 ]
     ctx="$(argv_value --ctx-size)"
     par="$(argv_value --parallel)"
     [ $((ctx % par)) -eq 0 ]
-    [ $((ctx / par)) -eq 180224 ]
-    [ $((ctx / par)) -le 262144 ]
+    [ $((ctx / par)) -eq 262144 ]
 }
 
-@test "the KV contract is asymmetric: unquantized K, q8_0 V, flash-attn on" {
+@test "the KV contract is unquantized both halves, flash-attn on" {
     serve_argv
     [ "$status" -eq 0 ]
     [ "$(argv_value --cache-type-k)" = "f16" ]
-    [ "$(argv_value --cache-type-v)" = "q8_0" ]
-    # q8_0 on V only exists on the flash-attention path; without this flag
-    # pair the KV contract in the env file is a lie the engine never sees.
+    # Frank, 2026-10-05: NO KV quantization on this endpoint. The Q8_0 rung
+    # funds itself by UBATCH first (2048 -> 1024), then pool steps — a q8_0-V
+    # proposal was written and overruled. The flag pair is what makes this
+    # observable — a build without the FA path silently falls back, so both
+    # must appear in the argv the supervisor runs.
+    [ "$(argv_value --cache-type-v)" = "f16" ]
     [ "$(argv_value --flash-attn)" = "on" ]
 }
 
 @test "the KV budget leaves 2-3 GiB of the 3090 free" {
-    # Rate: K f16 (16 KiB/token) + V q8_0 (~8.5, budgeted 25 with slack).
-    # The non-KV half is MEASURED on this card on 2026-10-02: with the
-    # projector loaded the engine reported 22,972 MiB used against 14,113
-    # MiB of KV, so weights + vision tower + DeltaNet state + compute
-    # buffers + CUDA graphs = 8,859 MiB => 9 GiB. Two asserts, both about
-    # the output the operator sees: the pool must still fit with 2 GiB
-    # spare, and it must be large enough that 5 GiB spare is NOT reachable
-    # (an undersized pool is as wrong as an oversized one). Arithmetic, not
-    # a boot log: change CTX_SIZE and this bites before the deploy does —
-    # it rejects the 3 x 196,608 pool that measured only 1.1 GiB free. The
-    # same pool with f16 V (34 KiB/token) lands past the card, which is why
-    # the contract is asymmetric.
+    # Rate, from the Ornith GGUF header (read from the Hub file, 2026-10-05):
+    # NINE full-attention layers (block_count 33, full_attention_interval 4)
+    # x 4 kv heads x 256 dim x 2 B = 18 KiB/token for K and the same for V
+    # => 36 KiB/token (budgeted 37 with slack; MiMo measured 24.5 — one
+    # fewer layer AND one fewer KV head). The non-KV half: Q8_0 weights
+    # 9,334 MiB (9,786,060,384 B) + vision tower ~1.3 GiB (mmproj 879 MiB at
+    # the measured ~1.5x cost law) + DeltaNet state + compute buffers +
+    # CUDA graphs => 11 GiB budgeted. Two asserts, both about the output
+    # the operator sees: the pool must still fit with 2 GiB spare, and it
+    # must be large enough that 5 GiB spare is NOT reachable (an undersized
+    # pool is as wrong as an oversized one).
+    # Arithmetic, not a boot log: change CTX_SIZE and this bites before the
+    # deploy does. At Ornith's rate the two-200704-slot pool (14.1 GiB KV)
+    # FAILS the first assert — which is exactly why the shipped shape is
+    # ONE native slot.
     ctx="$(sed -nE 's/^CTX_SIZE=([0-9]+)$/\1/p' "$ENV_FILE")"
-    kv_gib=$((ctx * 25 / 1048576))
-    [ $((kv_gib + 9 + 2)) -le 24 ]
-    [ $((kv_gib + 9 + 5)) -gt 24 ]
-    f16v_gib=$((ctx * 34 / 1048576))
-    [ $((f16v_gib + 9)) -gt 24 ]
+    kv_gib=$((ctx * 37 / 1048576))
+    [ $((kv_gib + 11 + 2)) -le 24 ]
+    [ $((kv_gib + 11 + 5)) -gt 24 ]
+    # And the ceiling that sets the per-slot window: two NATIVE windows cannot
+    # be paid for in full precision on this card.
+    native=$((2 * 262144))
+    [ $((native * 37 / 1048576 + 11)) -gt 24 ]
+}
+
+@test "the compute-buffer pair follows the env file and stays self-consistent" {
+    # This endpoint ran on llama.cpp's own defaults until 2026-10-03, so the
+    # contract is: whatever rung the repo carries is the rung the engine is
+    # exec'd with — the number a benchmark attributes a prefill rate to must be
+    # a repo value, never an engine default nobody wrote down. Asserted against
+    # the env rather than a literal so a retune moves one file, not two.
+    local env_batch env_ubatch
+    env_batch="$(sed -nE 's/^BATCH_SIZE=([0-9]+)$/\1/p' "$ENV_FILE")"
+    env_ubatch="$(sed -nE 's/^UBATCH_SIZE=([0-9]+)$/\1/p' "$ENV_FILE")"
+    serve_argv
+    [ "$status" -eq 0 ]
+    [ "$(argv_value --batch-size)" = "$env_batch" ]
+    [ "$(argv_value --ubatch-size)" = "$env_ubatch" ]
+    # llama.cpp refuses ubatch > batch, so a retune that inverts the pair has
+    # to fail here instead of at boot on a live endpoint.
+    [ "$env_batch" -ge "$env_ubatch" ]
+
+    # And an env that names neither flag still ships an explicit pair: the
+    # serve script's fallbacks are written-down values, not blanks the engine
+    # fills in silently.
+    sed -i '/^BATCH_SIZE=/d;/^UBATCH_SIZE=/d' "$STAGE/test.env"
+    serve_argv
+    [ "$status" -eq 0 ]
+    argv_value --batch-size | grep -qE '^[0-9]+$'
+    argv_value --ubatch-size | grep -qE '^[0-9]+$'
 }
 
 @test "no speculative draft ships by default" {
@@ -156,7 +196,7 @@ argv_value() {
 }
 
 @test "vision ships by default and MMPROJ_FILE empty opts out of the image path" {
-    # The Q5_K_M GGUF carries no `v.*` tensors, so the projector argument IS
+    # The Q6_K GGUF carries no `v.*` tensors, so the projector argument IS
     # the vision capability: no flag, no eyes.
     serve_argv
     [ "$status" -eq 0 ]
@@ -170,7 +210,7 @@ argv_value() {
     ! grep -qxF -- '--mmproj' <<<"$output"
 }
 
-@test "the fleet reasoning contract survives the reslice to three contexts" {
+@test "the fleet reasoning contract survives the reslice to two wide contexts" {
     serve_argv
     [ "$status" -eq 0 ]
     # deepseek format = thinking lands in reasoning_content, the field the

tests/observability_quadlet.bats

@@ -315,31 +315,28 @@ PAIRS
 
 @test "prometheus scrapes every LLM endpoint that exports metrics" {
     # The LLM endpoints are the services whose behaviour this fleet argues about
-    # most (KV pressure, prefix-cache reuse, queue depth). All three export
-    # the llamacpp:* family since 2026-09-26, when frank-primary (Dankserver)
-    # went back to llama.cpp — the unsloth b11030-mix prebuilt serving
-    # Qwen3.8-Flash-Next Q8_0 — after two days of the Lucebox DS4.1 engine,
-    # which exported nothing (404) and had no job.
+    # most (KV pressure, prefix-cache reuse, queue depth). Both Dankclaw
+    # endpoints export the llamacpp:* family. frank-primary LEFT the set
+    # 2026-10-05: the FreeToken PR587 ROCm engine has NO /metrics endpoint —
+    # scraping it was a structural never-green (404), so the job was deleted
+    # rather than left vacuous (the Lucebox DS4.1 precedent, 2026-09-26).
     # The two Dankclaw endpoints are HOST processes reached through
     # host.containers.internal: this Prometheus is a bridge-networked
     # container, and on this netavark host a container can only reach the
     # host's LAN IP at a PUBLISHED port — a host-process bind like
     # llama-server's 0.0.0.0:7770 is refused at 192.168.0.143:7770 while it
     # answers at 169.254.1.2:7770 (why frank-small stayed red by construction
-    # for days until the 2026-09-20 /targets audit). frank-primary is a
-    # DIFFERENT machine, so that quirk does not apply: plain LAN routing
-    # reaches 192.168.0.235:7770, and the job must say so (a gateway target
-    # would point at Dankclaw's own :7770 — frank-large — and double-scrape
-    # it under the wrong label).
+    # for days until the 2026-09-20 /targets audit).
     local cfg="$REPO_ROOT/shared/prometheus/prometheus.yml"
     grep -q '"host.containers.internal:7771"' "$cfg" || { echo "frank-small not scraped"; return 1; }
     grep -q '"host.containers.internal:7770"' "$cfg" || { echo "frank-large not scraped"; return 1; }
-    grep -q '"192.168.0.235:7770"' "$cfg" || { echo "frank-primary not scraped"; return 1; }
+    ! grep -q '"192.168.0.235:7770"' "$cfg"
     grep -q '"host.containers.internal:7770,192.168.0.235:7770"' "$cfg" \
         && { echo "frank-primary folded into a Dankclaw gateway target"; return 1; }
-    for job in frank-small frank-large frank-primary; do
+    for job in frank-small frank-large; do
         grep -q "job_name: \"$job\"" "$cfg" || { echo "no scrape job for $job"; return 1; }
     done
+    ! grep -q 'job_name: "frank-primary"' "$cfg"
     # None of them is reached through a guarded vhost: Prometheus cannot solve
     # an Anubis challenge, so a franksdank.cloud target is red by construction.
     ! grep -qE '^\s*-\s*"?https?://[^"]*franksdank' "$cfg"