docs: frank-secondary at ubatch 2048 — Measured section, rollback ladder, fleet docs

23,120 used / 1,005 free at boot, 763 settled after a vision probe
(encoder buffers resident). Rung cost 746 MiB, not the transferable
616 prediction. Prefill baseline 1,209 t/s @ 3,760 recorded; decode
unchanged as the law says. 2048 = ceiling for this shape.

AGENTS.md

@@ -12,8 +12,9 @@ merge absorbed the former `vllmScripts` repo, history preserved).
   Calibre-Web, Jellyfin, Plex) plus the `frank-primary` LLM service.
 - **Dankclaw** (192.168.0.143) — AI/search/dev/observability stack (SearXNG
   via Gluetun, ComfyUI, OpenWebUI, Hermes, Gitea, Grafana, Prometheus, Loki,
-  Alloy, Pi-hole) plus TWO LLM services: `frank-large` (halogen container on
-  the gfx1151 iGPU) and `frank-small` (llama.cpp CUDA on the RTX 3090). Caddy on
+  Alloy, Pi-hole) plus TWO LLM services: `frank-large` (flash-server
+  container on the gfx1151 iGPU) and `frank-secondary` (llama.cpp CUDA on the
+  RTX 3090). Caddy on
   Dankclaw reverse-proxies both hosts.
 - **FrankBook-Pro** (192.168.0.112) — macOS mini running the *arr stack
   (Prowlarr, Sonarr, Radarr, Lidarr, Readarr, qBittorrent) as quadlets inside
@@ -24,9 +25,9 @@ merge absorbed the former `vllmScripts` repo, history preserved).
 
 | Target | Service | Runtime | Model | Port |
 |--------|---------|---------|-------|------|
-| Dankserver | `frank-primary` | tcclaviger RDNA4 vLLM fork (`docker.io/tcclaviger/vllm@sha256:da4029666…`, digest-pinned, plain `podman run`, root systemd unit) | `orcarouter/Qwen3.8-Flash-Next-Uncensored-FP8` (~172.8 GiB, 131 shards), TP=2, expert offload automatic (fork plans it), prefetch OFF, **fp8 KV** (`--kv-cache-dtype fp8`), **no speculative decode**, MNBT 2048 (arena-free: the registry prices overhead flat up to its nbt 4096 anchor), MoE gather 32×64, **`--expert-precise-memory`** (fitted 5.25 GiB overhead, not the 7.60 GiB default), **`VLLM_CUDAGRAPH_MEM_BASE_MB=0` / `_PER_GRAPH_MB=12`** (the plan's CUDA-graph term priced at the measured capture — 0.31 GiB then a NEGATIVE −0.06 GiB delta, so the 256 MiB analytic base was fiction → graph term 0.07 GiB), **`GPU_MAX_HW_QUEUES=2`**, **mounted cache dir** at `/root/.cache` (the image ships no `/root/.cache` at all; unmounted, every boot re-pays `torch.compile took 181.20 s`), **3 slots × 200704 ctx** at KV_BYTES 3,166,500,000/rank (~400,700 KV tokens at the engine's 7,891.8 B/token here) — the **pool, not the slot count, caps resident conversations** (measured live: `GPU KV cache size: 401,408 tokens, Maximum concurrency for 200,704 tokens per request: 2.00×`; the 3.15e9 pool below it measured 398,658/1.99×), and shorter requests use the spare blocks. **`--max-num-seqs` must stay 3: SEQS=2 cannot boot this model** — torch.fx `ConstraintViolationError` on `query_start_loc`, measured at BOTH MNBT 1024 and 2048. **util stays 0.982 BY LAW**: at `le=1` the worker init refuses `budget == total` with `ValueError: Free memory on device cuda:N (31.29/31.86 GiB)` on both ranks before a single weight loads — measured crash loop 2026-10-03. Arena at shape G: **17.77 GiB / 7,762 slots measured**, KV 401,408 tokens / 2.00× concurrency, 0.89 GiB of real free VRAM per card held through a live 4,096-token generation (the ~3.5 GiB of driver-reported free VRAM at shape E was estimator slack — the registry's 9190 B/token KV reserve vs the real 7,892, and the 5.25 GiB overhead vs far less — NOT allocatable), and the client contract wired: `--reasoning-parser …
-| Dankclaw | `frank-large` | **halogen container** (`ghcr.io/peonist-ai/halogen@sha256:dc0a39a0…`, digest-pinned, plain `podman run` via `scripts/serve-frank-large.sh`, root systemd unit) | `peonist-ai/halogen-qwen3.8-27b` **`qwen3.8-27b-p1w4d-d2.hgn`** (35,865,565,184 B; the `.hgn` format loads in NO other engine — llama.cpp, vLLM and transformers all refuse it), **DFlash2 spec decode (`HALOGEN_DRAFTER=2`)**, **1 slot × 200,700 ctx** — `HALOGEN_KV_SLOTS=1` is load-bearing: batching XOR speculation, slots>1 silently disables the drafter (measured: 8 slots at serial ~6 t/s/stream), d2 KV (2× fp8, 64 KiB/token → 12.2 GiB pool, engine sizes it and prints a boot ledger naming any clamp), **TEXT-ONLY** (the engine ignores the vision encoder — vision is frank-small's job), **censored base** (the ROCmFPX AGENT rollback is the refusal escape hatch), **no server-side sampler: absent temperature is GREEDY**, so the card's thinking recipe (temp 1.0 / top_p 0.95) rides the wire in `models.yml`. Publisher card 31.7 t/s mean across eight text workloads vs the retired ROCmFPX endpoint's measured 27.6–28.1. Prompt over the slot AND `max_tokens` over `HALOGEN_MAX_TOKENS_CAP` are both 400s naming the limit, never truncations. Serve passes NO `--ulimit nofile` (rootless runc cannot raise Dankclaw's 524288 hard limit — see Key Lessons). Cutover 2026-10-05|7770|
-| Dankclaw | `frank-small` | llama.cpp master, CUDA sm_86 build (root systemd unit, relinked) | bartowski/MiMo-V2.6-Distill-Qwen-9B-GGUF **Q6_K** (9B dense qwen3_5 hybrid, 7.26 GiB GGUF, no MTP weights so no spec decode) + the repo's f16 mmproj for **vision**, on the RTX 3090, **2 × 200,704** (pool 401,408 — NOT the native window) with **fully unquantized KV (f16 K + f16 V)** under flash attention, **batch 4096 / ubatch 2048** (doubled 2026-10-05: measured +4.6 % prefill at 8,928 tokens for 616 MiB), 21,846 used / **2,279 free** of 24,576 MiB, 94.4 tok/s decode. Resized 2026-10-03 (3 × 180,224 + q8_0 V → 2 × 200,704 + f16 V → + Q6_K rung: the freed slots paid for exact values AND the higher quant; the rung costs a measured −10 % decode — weight-bandwidth-bound); rollback `MODEL_FILE=…/MiMo-Q5KM/…Q5_K_M.gguf` (buys the 10 % back) or `CTX_SIZE=540672 PARALLEL=3 CACHE_TYPE_V=q8_0` (Q5 only) | 7771 |
+| Dankserver | `frank-primary` | **radiance** (StillDeadcode v1.0.7 line; image `stilldeadcode/radiance@sha256:4c3f484564…` digest-pinned INDEX digest, 287 MiB, `libr4d` gfx1201 kernels, plain `podman run`, root systemd unit) | `Qwen/Qwen3.8-Flash-Next-FP8` official (172.8 GB, 131 shards, sha256-verified) converted **no-recipe** to `qwen4exp-fp8-native.rad` (177,109 MiB measured = 173.0 GiB) — **native modelopt block-FP8, no requantisation** — on **BOTH R9700s TP=2** (`--tp-wire wht6`), experts in a **40 GiB/rank pinned host pool** + VRAM cache (`--placement expert_tiered`, `--expert-vs-cache-ratio 0.82` = the 99 %-hit point), **PLE 47.7 GiB pinned to host RAM** (`--ngram-placement ram`; root unit + `LimitMEMLOCK=infinity`), **fp8 KV 7.32 KiB/token**, **MTP spec decode depth 3**, *…
+| Dankclaw | `frank-large` | **halogen-flash-server 0.16.2** (`ghcr.io/peonist-ai/halogen-flash-server@sha256:0c61bf84…`, digest-pinned, plain `podman run` via `scripts/serve-frank-large.sh`, root systemd unit) | Quat3rnion/halogen-swift1.5-qwen3.8-flash-next-v2-abliterated **`qwen38-flash-next-v2-swift15-abliterated.hgn`** (66,687,678,464 B = compacted r2 of the card's 89,775,868,032 B original, ~62 GiB resident; the v2 `.hgn` format loads in NO other engine), **+ REQUIRED `qwen38-flash-next-ngram.hgn`** (51,200,246,144 B, peonist @5cc17cea — read beside the checkpoint) **+ vision tower** (897,916,416 B, @e053f488 — **vision ON**, `image_url` parts only, ≥256 px probes), **MTP spec decode ALWAYS-ON (`HALOGEN_SPEC_ADAPT=0` — the adaptive default disables the head per request and HALVES decode)**, MTP head INSIDE the checkpoint (`HALOGEN_MTP_HEAD` unset; no sidecar fetched), **2 slots × 200,192 ctx** (KV pool 400,384 positions ≈ 10.7 GiB at ~28 KiB/position; the 27B slots>1-disables-drafting law does NOT carry over — batching and speculation coexist, the card's quickstart ships 2 with the head), **server-side sampler 1.0/0.95/20** (card recipe; models.yml extraBody matches), coupled cap/timeout 65536/7200, prompt cache ON by default (warm 100k turn ≈ 2 s vs ≈ 88 s cold) | 7770 |
+| Dankclaw | `frank-secondary` | llama.cpp master (0.5.0-dev, commit `5fc4f3c`), CUDA sm_86 build (root systemd unit, relinked) | ajgazin/Swift-1.5-Qwen3.8-27B-Uncensored-Dynamic-MTP-GGUF **RCO-IQ3_XXS -mtp** (27B dense qwen3_5 hybrid, 10,442,827,392 B; a per-tensor allocation mix of stock types; the MTP head in the file **drafted** — **spec ON** (draft-mtp depth 3) since 2026-10-06 evening, ngram skipped on the recurrent-rollback law) + the publisher's own **BF16 mmproj** for **vision**, on the RTX 3090, **2 × 114,688** (Frank's 112k cap each since the 2026-10-06 late reshape, cut from 131,072 to fund the **ubatch ladder 512 → 1024 → 2048** — **1,005 MiB free at boot, 763 settled after a vision probe** (encoder buffers resident)) with **q8_0 K + q8_0 V (FP8 class, 34 KiB/token)** under flash attention, batch 4096 / **ubatch 2048** (the ceiling for this shape — 4096 would not fit); **prefill 1,209 t/s @ 3,760 tokens; ~52–58 t/s recipe decode** (MTP acceptance 34–54 %, temp-1.0 noise; decode is ubatch-invariant). Renamed from `frank-small` 2026-10-06 (87721ab). Supersedes the same-morning IQ3_S-mtp payloads: the two-slot no-spec shape and the one-slot MTP shape (both 2026-10-06; payload on disk).|7771|
 
 Retired endpoints (`frank-npu`, and every prior runtime) are in
 the [runbook index](#runbook-index). The full swap history lives in
@@ -40,7 +41,11 @@ the [runbook index](#runbook-index). The full swap history lives in
 - `scripts/` — `deploy.sh` / `deploy-plan.py` / `deploy-remote.sh`, LLM serve
   scripts, systemd units, env tunables, bench harnesses
 - `old/` — retired LLM scripts (reference only)
-- `serverAppConfigs-docs/` — knowledge base (Obsidian vault, OKF format)
+- `serverAppConfigs-docs/` — knowledge base (Obsidian vault, OKF format).
+  The managed skill library is distilled there one page per skill-cluster:
+  `LLM/index.md` → *Cross-Cutting References*, plus `Operations/index.md`,
+  `Agent/index.md`, and the Infrastructure/Dankclaw/FrankBook-Pro service
+  pages (`serverAppConfigs-docs/log.md` carries the wave plan and results)
 - `tests/` — bats suite + Playwright e2e
 
 ## Git Workflow
@@ -68,7 +73,7 @@ All edits flow through git. Never edit files on a remote machine directly.
 The planner maps changed paths to services; the remote runner restarts only
 the selected service and records its revision outside the checkout.
 Documentation-only changes are a no-op. LLM services are selected only by
-their own files — every endpoint (`frank-large`/`frank-small`/`frank-primary`)
+their own files — every endpoint (`frank-large`/`frank-secondary`/`frank-primary`)
 by its unit/env/serve trio in `scripts/` — so
 unrelated
 container changes cannot interrupt inference. Set `FRANK_SUDO_PW` for headless
@@ -224,13 +229,16 @@ archive.
 **STOP the production service BEFORE benchmarking** — these servers serve
 real users, and a bench beside a live service starves the GPU and corrupts
 results. Use the runtime's own harness: `llama-bench` for the llama.cpp
-endpoints; a live llama.cpp server also reports honest per-request numbers in the `/completion` `timings` object (prefill/decode rates, `draft_n`, `draft_n_accepted`), which is how frank-large and frank-small were measured. Rolling your own bench
+endpoints; a live llama.cpp server also reports honest per-request numbers in the `/completion` `timings` object (prefill/decode rates, `draft_n`, `draft_n_accepted`), which is how frank-large and frank-secondary were measured. Rolling your own bench
 requires asking first.
 Scratch ports are not used on LLM hosts — a
 test process cannot share the single GPU anyway; validate against the service
 itself.
 
-(A gufo-era law; frank-large ran ROCmFPX llama.cpp 2026-10-02 → 10-05 and is the halogen container since, but the rollback tags keep it live.) **`gufo bench` cannot rank speculative-decode settings, and will invert the ranking.**
+(A gufo-era law; frank-large ran ROCmFPX llama.cpp 2026-10-02 → 10-05, the
+halogen 27B container 2026-10-05 → 10-06, and is flash-server since, but the
+rollback tags keep it live.) **`gufo bench` cannot rank speculative-decode
+settings, and will invert the ranking.**
 Measured 2026-09-28 on frank-large at depth 32768: its corpus
 reports `acceptance=0.933 ± 0.000` with an **identical trace sha256** across
 draft widths, i.e. the same highly-predictable text every run, and 93 % of
@@ -258,9 +266,9 @@ merely wedged kernel usually still answers ARP.
 1. Stop and mask (self-healing would restart it):
    - Dankclaw `frank-large` (root unit, halogen podman on the iGPU):
      `sudo systemctl stop frank-large && sudo systemctl mask frank-large`
-   - Dankclaw `frank-small` (root unit, llama.cpp on the 3090):
-     `sudo systemctl stop frank-small && sudo systemctl mask frank-small`
-   - Dankserver `frank-primary` (root unit, tcclaviger vLLM on the 2x R9700):
+   - Dankclaw `frank-secondary` (root unit, llama.cpp on the 3090):
+     `sudo systemctl stop frank-secondary && sudo systemctl mask frank-secondary`
+  - Dankserver `frank-primary` (root unit, radiance on both R9700s, TP=2):
      `sudo systemctl stop frank-primary && sudo systemctl mask frank-primary`
 2. Bench.
    NOTE: `systemctl mask` FAILS on a relinked unit (the unit file in
@@ -279,7 +287,7 @@ The harness reaches the fleet through `.omp/agent/models.yml` in the
 operator's home — `/home/blank/.omp/agent/models.yml` on the dev box, which is
 where the harness runs; there is no copy on the servers. Each provider is one
 HOST:PORT; each model `id` is the served alias: `frank-primary` →
-`192.168.0.235:7770`, `frank-large` → `192.168.0.143:7770`, `frank-small` →
+`192.168.0.235:7770`, `frank-large` → `192.168.0.143:7770`, `frank-secondary` →
 `modelRoles`.
 
 Resolution rules: the default model resolves as a `defaultProvider` +
@@ -292,16 +300,19 @@ When renaming a service, retiring an endpoint, or moving a model: update the
 provider block, model `id`, `contextWindow`, `maxTokens`, `input` in
 `models.yml`, then fix every `modelRoles` entry that named the old endpoint —
 a role pointing at a provider the file does not define fails at call time. As
-of 2026-10-05 **two of the three endpoints accept images** (`input: [text,
-image]`) — `frank-large` went TEXT-ONLY with the halogen swap (the engine
-ignores the vision encoder; the BF16-mmproj era ended 2026-10-05) — while
-`frank-primary` takes them through a BF16 mmproj loaded explicitly (this build never
-autoloads projectors), and `frank-small` through
-`mmproj-MiMo-V2.6-Distill-Qwen-9B-f16.gguf` passed as `--mmproj` — the MiMo
-GGUF itself carries no vision tensors. `frank-small`'s `contextWindow` is
-**200704** (resliced from 180224 on 2026-10-03), the per-slot window, not the
-model's native 262144: the harness must compact before llama.cpp answers a
-longer prompt with a 400. Note also
+of 2026-10-06 **all three endpoints accept images** (`input: [text,
+image]`) — `frank-large` returned to vision with the flash-server v2 swap
+(the v2 vision tower loads beside the checkpoint; the text-only halogen
+27B window closed 2026-10-06) — while
+`frank-primary` takes them natively (radiance runs the checkpoint's
+27-layer Qwen-VL tower — patching, pad-token expansion and 3-part rotary
+are the engine core; `--mm-max-patches` available),
+and `frank-secondary` through
+the publisher's own `mmproj-BF16.gguf` passed as `--mmproj` — the Swift
+GGUF itself carries no vision tensors. `frank-secondary`'s `contextWindow` is
+**114688** (cut from 131072 on 2026-10-06 late to fund the ubatch ladder to 2048), the
+per-slot window, not the model's native 262144: the harness must compact
+before llama.cpp answers a longer prompt with a 400. Note also
 that
 the reasoning channel differs by runtime: llama.cpp AND SGLang (qwen3
 parser, live-verified) emit `reasoning_content`, vLLM emits `reasoning` —
@@ -312,253 +323,260 @@ endpoint on 2026-09-11, returned 2026-09-17 with the Gemma4 projector's
 audio encoder, and left again with the Gemma4 retirement on 2026-10-01
 (the MiMo tower is vision-only).
 
-### frank-large — halogen container, Qwen3.8-27B + DFlash2 (Dankclaw, ACTIVE)
+### frank-large — halogen-flash-server 0.16.2, Qwen3.8-Flash-Next v2 Swift1.5 ABLITERATED (Dankclaw, ACTIVE)
 
-The **halogen engine** (`ghcr.io/peonist-ai/halogen@sha256:dc0a39a0…`,
-digest-pinned, published tag 0.1.4) as a podman container driven by
-`scripts/serve-frank-large.sh` under a **root systemd unit**
-(`scripts/frank-large.service`, `User=dank`, `Restart=always`, `ExecStop=podman
-stop -t 120 frank-large-halogen`, the script `exec podman wait` so the restart
-policy is real) on port 7770 from the Strix Halo iGPU. Engine stdout goes to
-`/home/dank/logs/frank_large.log` — **the boot ledger (KV pool, drafter state,
-any clamp) lives there, not in the journal**. Serving
-`peonist-ai/halogen-qwen3.8-27b`: `qwen3.8-27b-p1w4d-d2.hgn` (35,865,565,184 B
-≈ 33.4 GiB, sha256 `274c3fc7…`, 1352 tensors; p1 weights / w4 activations /
-d2 KV) + the flat `tokenizer/` directory (5 files). Cutover 2026-10-05 from the ROCmFPX llama.cpp fork
-([[frank-large-rocmfpx-q6agent-uncensored-mtp]], measured 27.6–28.1 t/s prose),
-chosen for the publisher's 31.7 t/s DFlash2 mean.
+The **flash-server engine** (`ghcr.io/peonist-ai/halogen-flash-server@sha256:0c61bf84…`,
+digest-pinned, published tag 0.16.2, 3.6 GB image) as a podman container
+driven by `scripts/serve-frank-large.sh` under a **root systemd unit**
+(`scripts/frank-large.service`, `User=dank`, `Restart=always`,
+`ExecStop=podman stop -t 60 frank-large-halogen`, the script runs `podman
+run` FOREGROUND — the container's exit code is the service's exit so
+`Restart=always` stays real, and the engine ledger lands on disk) on port
+7770 from the Strix Halo iGPU. Engine stdout goes to
+`/home/dank/logs/frank_large.log` — **the boot ledger (weights resident, KV
+pool, MTP head, prompt cache, any clamp) lives there, not in the journal**.
+Cutover 2026-10-06 from the halogen 27B-dense era
+([[frank-large-halogen27b-dflash2]], retired the same window).
 
-**The format is the lock.** `.hgn` loads in nothing else — the runtime is
-chosen by the FILE, as ROCmFPX's tensor types were before it. This is NOT the
-2026-09 `halogen-flash-server` image (different repo, Flash-Next w4b era); the
-halogen-server repo restarted versioning at 0.1.x.
+Serving `Quat3rnion/halogen-swift1.5-qwen3.8-flash-next-v2-abliterated`:
+`qwen38-flash-next-v2-swift15-abliterated.hgn` (66,687,678,464 B ≈ 62.1
+GiB, sha256 `9c880b8b…` — a compacted r2 of the card's 89,775,868,032 B
+append-only original; the abliteration workflow appends and the repo's
+tooling drops dead padding, live tensors byte-identical) + the **REQUIRED**
+n-gram lookup table `qwen38-flash-next-ngram.hgn` (51,200,246,144 B,
+peonist @5cc17cea) + the vision tower `qwen38-flash-next-vision.hgn`
+(897,916,416 B, @e053f488) + the flat `tokenizer/` directory (6 files,
+fetched fresh from peonist). Cutover 2026-10-06 from the halogen
+27B-dense shape, chosen by Frank for the ABLITERATED base.
 
-**Two product regressions, both accepted by Frank before deploy:** the
-endpoint is **text-only** (README honest limits: the vision encoder is
-ignored — frank-small owns images now) and the checkpoint is a finetune of the
-**censored** `Qwen/Qwen3.8-27B` base (refusals can return; the rollback is one
-commit-revert away with the AGENT payload still on disk).
+**The format is the lock.** The v2 `.hgn` loads ONLY in the
+flash-server line — the 27B-era halogen 0.1.4 container refuses it, so the
+engine moved with the model. The MTP head is INSIDE the checkpoint
+(`HALOGEN_MTP_HEAD` stays unset; the separate `qwen38-flash-next-mtp.hgn`
+sidecar is deliberately not fetched).
 
-**Engine laws that shape the config** (all in `scripts/frank-large.env` with
-evidence): `HALOGEN_KV_SLOTS=1` — batch XOR speculation, slots>1 turns the
-drafter off silently. `HALOGEN_SLOT_CTX=200700` — the per-conversation HARD
-ceiling (400 naming the limit, never truncation); `models.yml`
-`contextWindow: 200700` must match or the harness discovers the wall as a 400.
-`HALOGEN_MAX_TOKENS_CAP=65536` + `HALOGEN_QUEUE_TIMEOUT=7200` are a coupled
-pair; the harness `maxTokens: 57344` stays under the cap (over-cap is a 400
-naming the limit — README, never a 413). Thinking tokens COUNT against the
-budget: exhausted mid-thought is `finish_reason: "length"` with empty
-`content` and the reasoning in `reasoning_content`.
-`HALOGEN_DRAFTER=2` = DFlash2. **No server-side sampler**: an absent
-temperature is greedy, so the thinking recipe exists only in `models.yml`
-`extraBody`. `response_format` is unsupported (no constrained decoding).
-Thinking lands in **`reasoning_content`** (harness field unchanged); the same
-Qwen3.8 template rejects `reasoning_effort` outside `xhigh|medium|low`, so the
-`models.yml` fold stays mandatory.
+**Product facts vs the 27B era, both regressions FIXED:** the base is
+**abliterated** (the 27B was a censored-base finetune; refusals were an
+accepted trade there) and **vision is ON** (the 27B engine silently
+ignored its encoder — frank-secondary owned images for exactly one day at this
+endpoint).
 
-Fetch: `scripts/download-frank-large-halogen.sh` into
-`/home/dank/models/halogen-qwen3.8-27b/` (`-f` + byte count + LFS sha256 +
-`--verify-only`, no token on the command line; resume goes against the
-RESOLVED CDN URL because curl drops `Range` across HF's 302). The deployer
-and serve script both refuse a missing tokenizer file or **any symlink** —
-an HF cache snapshot symlinks each entry into `blobs/` and those links
-dangle inside the container mount.
+**Engine laws that shape the config** (all in `scripts/frank-large.env`
+with evidence): `HALOGEN_SPEC_ADAPT=0` is MANDATORY — the default adaptive
+policy disables the MTP head for whole requests after low acceptance and
+HALVES decode; the head stays on and temp-1.0 traffic pays some acceptance
+for it (accepted). `HALOGEN_CTX=200192` is the PER-SLOT hard ceiling (400
+naming the limit, never truncation); `models.yml` `contextWindow: 200192`
+must match, and compaction at 70 % = 140,134 + 57,344 budget = 197,478 fits.
+`HALOGEN_KV_SLOTS=2` is LEGAL here — the 27B "slots>1 disables the drafter"
+law does NOT carry over; batching and speculation coexist (the card's
+quickstart ships 2 slots with the head), so 2 is concurrent capacity.
+`HALOGEN_KV_POOL_POSITIONS=400384` (explicit 2 × ctx; ~28 KiB/position ⇒
+~10.7 GiB). `HALOGEN_MAX_TOK=8192` is only the per-request DEFAULT (the
+harness always sends 57,344); `HALOGEN_MAX_TOKENS_CAP=65536` +
+`HALOGEN_QUEUE_TIMEOUT=7200` are a coupled pair (FLAGS.md table).
+**A server-side sampler EXISTS** (`HALOGEN_TEMPERATURE/TOP_P/TOP_K` =
+1.0/0.95/20, the card's thinking recipe) — an absent temperature is no
+longer greedy; `models.yml` extraBody carries the same values, consistent
+not load-bearing. Prompt cache is ON by default (`PROMPT_CACHE=2`): a warm
+100k follow-up costs ~2 s vs ~88 s cold — the 27B engine's manual
+`HALOGEN_CACHE_MB` pin era is over. `response_format` is unsupported.
+Thinking lands in **`reasoning_content`**; the v2 chat template rejects
+`reasoning_effort` outside `xhigh|medium|low`, so the `models.yml` fold
+stays mandatory. Thinking tokens COUNT against the budget (empty content +
+`finish_reason: "length"` when exhausted mid-thought). Vision: OpenAI
+`image_url` content parts only; ≥256 px probe images or the model answers
+"video".
 
-Runbook: `serverAppConfigs-docs/LLM/frank-large-halogen27b-dflash2.md`.
-Rollback: revert the swap commit and redeploy — the ROCmFPX payload
-(`/home/dank/models/kingjones-Qwen3.8-27B-ROCmFP4-STRIX-MTP/`), the fork build
-and `scripts/build-rocmfpx.sh` (pin `36e9acd`) all remain on disk, seam-tested.
-The older gufo tag `frank-large-gufo-q4kxl-20261001` stays a **re-download**
-rollback (payloads removed 2026-10-03; `scripts/build-gufo.sh` stays).
+**Memory ledger** (121.2 GiB unified; the referee is the boot ledger):
+weights ~62 + KV ~10.7 + arena ~8 (`3.8 + 8.4 × MAX_TOK/16384`) + vision
+~2 ≈ 82.7 GiB engine-side + `HALOGEN_HOST_RESERVE_GIB=20` → ~18 GiB
+margin over baseline co-tenancy. The 51.2 GB n-gram table rides as mmap'd
+page cache beside the checkpoint — the card's quickstart arithmetic does
+not bill it as resident, but READ the boot ledger and re-size if the
+engine prints a clamp.
 
-### frank-small — llama.cpp on the RTX 3090 (Dankclaw, ACTIVE)
+Fetch: `scripts/download-frank-large-flashnext.sh` into
+`/home/dank/models/halogen-flashnext-v2/` (`-f` + exact byte counts + LFS
+sha256 + `--verify-only`, no token on the command line; resume goes
+against the RESOLVED CDN URL because curl drops `Range` across HF's 302).
+The serve gates mirror the deployer EXACTLY (byte-exact checkpoint +
+n-gram + vision tower, flat tokenizer with all six files as regular
+files) — the 2026-10-01 crash-loop lesson.
 
-**llama.cpp master** (CUDA sm_86 build
-`/home/dank/GitPackages/llama-master/build-cuda/bin/llama-server`) as a
-**root systemd unit** (`scripts/frank-small.service`, `relink_unit`ed into
-`/etc/systemd/system/`), serving
-`bartowski/MiMo-V2.6-Distill-Qwen-9B-GGUF` at **Q6_K** (7,793,710,624 B,
-sha256 `ef96d05a…`) **plus `mmproj-MiMo-V2.6-Distill-Qwen-9B-f16.gguf`**
-(918,166,048 B, sha256 `ff348f…`), both `/home/dank/models/MiMo-Q6K/` (the
-projector is a hardlink of the Q5KM copy — the dir is self-contained), on
-port 7771. The rung went Q5_K_M → Q6_K on 2026-10-03 the same morning as the
-reslice, funded by the freed slots. The runtime arrived 2026-10-01 night for
-one reason:
-**asymmetric KV** — `--cache-type-k f16` + `--cache-type-v q8_0` is the fleet's
-only per-tensor KV lever, and SGLang's `--kv-cache-dtype` moves K and V
-together. On 2026-10-03 Frank spent the headroom that lever bought: the reslice
-from three slots to two made room for **fully unquantized V**
-(`CACHE_TYPE_V=f16`); both flags stay explicit in the serve script. Vision came
-2026-10-02 via `--mmproj`: the GGUF carries no `v.*` tensors, so the projector
-file is the entire vision path (and ~1.4 GiB of the card).
+Runbook:
+`serverAppConfigs-docs/LLM/frank-large-flash-server-v2-swift15.md`.
+Rollback: revert the swap commit and redeploy — the 27B payload
+(`/home/dank/models/halogen-qwen3.8-27b/`), its fetch script and the
+halogen 0.1.4 image digest all remain on disk; the ROCmFPX door behind
+that (`scripts/build-rocmfpx.sh`, pin `36e9acd`) is untouched.
 
-**Architecture: qwen3_5 hybrid** — 32 layers, 8 full-attention + 24 Gated
-DeltaNet, native 262,144 window. KV exists only in the 8 attention layers
-(4 KV heads × 256 dim × 2 B ≈ 2 KiB/token/layer, ~16 KiB/token for f16 K+V);
-the DeltaNet state is a fixed per-slot buffer, so slot count — not context —
-drives that term.
+### frank-secondary — llama.cpp on the RTX 3090 (Dankclaw, ACTIVE)
 
-**The shape (2026-10-03):** `--ctx-size 401408 --parallel 2` = **TWO
-conversations of 200,704 tokens each** (llama.cpp divides the pool:
-`n_ctx_slot = CTX_SIZE / PARALLEL`), `--flash-attn on` (the seam pins it),
-**f16 K + f16 V**, all layers on the GPU, `--mmproj` for vision, and since
-2026-10-05 **batch 4096 / ubatch 2048** (doubled; the boot-pair A/B is in the
-runbook). Measured trade at 8,928 prompt tokens: prefill 4,024 → **4,209 t/s,
-+4.6 %**, decode flat (91.9–93.0 vs 93.8–94.3 — noise), and a 279-token prompt,
-which sits below either ubatch, did not move at all. The old "ubatch 2048
-failed the scratch boot" note belongs to the Q5_K_M predecessor and does NOT
-generalize: this shape boots 2048 clean. Measured VRAM **21,846 used / 2,279
-free** of 24,576 MiB against **21,276 / 2,849** for the 1024 arm, reproduced
-byte-identically the same hour — the doubling costs **616 MiB** and lands at
-the BOTTOM of the 2–3 GiB floor Frank asked for (`4096/1536` is the middle rung
-if that margin is ever wanted; non-KV 9,302 MiB, up from 8,732 by exactly the
-compute buffers; f16 K+V is 32 KiB/token across the 8 attention layers;
-quote `nvidia-smi`'s `free` field, never `total − used` — the driver holds
-~450 MiB of reserve the subtraction hides). The 2026-10-02 3 × 180,224 +
-q8_0-V shape stays one env edit away (`CTX_SIZE=540672 PARALLEL=3
-CACHE_TYPE_V=q8_0`, and it must pair with Q5_K_M — Q6_K breaks the floor
-there). Three native windows need 18.4 GiB of KV alone and were never on the
-table.
+**llama.cpp master** (0.5.0-dev, commit `5fc4f3c`; CUDA sm_86 build
+`/home/dank/GitPackages/llama-master/build-cuda/bin/llama-server`,
+`GGML_CUDA_FA_ALL_QUANTS=ON`) as a **root systemd unit**
+(`scripts/frank-secondary.service`, `relink_unit`ed into
+`/etc/systemd/system/`), serving `ajgazin/Swift-1.5-Qwen3.8-27B-Uncensored-Dynamic-MTP-GGUF`
+at **RCO-IQ3_XXS -mtp** (10,442,827,392 B, sha256 `e13d3e74…`) **plus
+the publisher's own `mmproj-BF16.gguf`** (931,145,888 B, sha256
+`19acd7fc…`), both `/home/dank/models/Swift15-UncIQ3XXS/`, on port 7771.
+Cutover 2026-10-05 (evening) on Frank's explicit contract: **128k
+context cap, KV at FP8, vision on, as many 128k contexts as fit**;
+reshaped three times on 2026-10-06 — morning to two 131,072 slots
+no-spec, evening to this UNCERSED RCO-IQ3_XXS payload with MTP back
+on, late the RENAME to **frank-secondary** (87721ab) at two 114,688
+slots with the ubatch ladder climbing 512 → 1024 → 2048 (f2575a1,
+d8da7a9). The
+same-day payloads it supersedes: the LIVE MiMo-V2.6 Q6_K shape and the
+Ornith-1.5-9B Q8_0 swap that was committed (77decb5..380aafa) but never
+deployed (blocked at the docs gate; the docs debt is what this section
+settles).
 
-Verified at deploy (2026-10-03): boot logs `n_slots = 2, n_ctx_slot = 200704`
-and `loaded multimodal model`; decode **94.44 tok/s** at temp 0 on the probe
-that measured 104.84 on Q5_K_M — **the Q6_K rung costs a measured −10 %**:
-decode on this card is weight-bandwidth-bound and Q6_K streams ~10 % more
-bytes/token; health 200, e2e 33/33; native `tool_calls`; thinking in
-`reasoning_content`; a **512×512** solid-red PNG answers `Red` through an
-`image_url` content part (two traps: the legacy top-level `images` field is
-ignored, and a TINY probe image undersupplies image tokens — a 10×10 PNG made
-the model claim "it's a video"; probe with ≥256 px). A prompt over the slot
-is a **400 `exceed_context_size_error`**, which is why `models.yml` sets
-`contextWindow: 200704`.
+**Architecture: qwen3_5 hybrid, bigger** — block_count 65 (64 real layers +
+the blk.64 MTP head), `full_attention_interval 4` ⇒ **16 full-attention +
+48 Gated DeltaNet**, native 262,144 window, dense. KV exists only in the 16
+attention layers (4 KV heads × 256 dim): 64 KiB/token at f16, **34 KiB/token
+at q8_0**. The DeltaNet state is a fixed per-slot buffer (~150 MiB/slot at
+48 linear layers × 48 value heads). Header facts were read from the Hub
+file by HTTP range + struct parse before anything was downloaded.
 
-Runbook: `serverAppConfigs-docs/LLM/frank-small-llama-mimo-q6k-2x200704.md`
-(the boot-pair table that prices pool changes lives in the retired
-[[frank-small-llama-mimo-q5km]]).
-Rollback: `MODEL_FILE=/home/dank/models/MiMo-Q5KM/MiMo-V2.6-Distill-Qwen-9B-Q5_K_M.gguf`
-buys back 875 MiB and the −10 % decode rate (3,571 free, 104.84 tok/s, both
-measured); the older shape `CTX_SIZE=540672 PARALLEL=3 CACHE_TYPE_V=q8_0`
-pairs with Q5 only; SGLang rollback in
-[[frank-small-sglang-mimo-v26]] (venv on disk at
-`/home/dank/GitPackages/sglang-cu128/`, INT8 payload staged to
-`~dank/.AGENTTRASH/20261002-retired-llm-payloads/`).
+**The shape (2026-10-06 midnight; Frank's tight-fit order, MTP back on
+in the evening, per-slot cut + ubatch climbed to 2048):** `--ctx-size
+229376 --parallel 2` = **TWO 114,688-token conversations** (Frank's
+112k cap each; `n_ctx_slot = CTX_SIZE / PARALLEL`; cut from 131,072 to
+fund the ubatch rungs), `--flash-attn on` pinned, **q8_0 K + q8_0 V**
+(llama.cpp's FP8 class — no true fp8 KV dtype exists; 8.5 effective
+bits; supersedes the same-day f16/f16 rule written for the Ornith 9B
+fit), all layers on the GPU, `--mmproj` for vision, batch 4096 /
+ubatch 2048. **Speculative decoding is ON** (draft-mtp, depth 3): the
+-mtp file's head is drafted against the main model's own weights;
+ngram-mod skipped on purpose — on a qwen35 hybrid ngram disables
+recurrent-state rollback, so a rejected draft corrupts the DeltaNet
+state.
 
-### frank-primary — tcclaviger vLLM, Qwen3.8-Flash-Next FP8 (Dankserver, ACTIVE)
+**Measured at the midnight boot (2026-10-06, 2 × 114,688 / ubatch
+2048): 23,120 used / 1,005 MiB free, settling 763 MiB after a vision
+probe** (`nvidia-smi` free; the encoder buffers stay resident on this
+boot) — the per-slot cut freed ~1.05 GiB of KV and both ubatch rungs
+spent it back (~0.3 + ~0.75 GiB measured; the 1024 rung had landed
+22,374/1,751, the 2048 rung cost 746 MiB, ~130 over the 616 the rung
+measured on the MiMo shape). The MTP draft ctx costs **~2.4 GiB
+measured** (23,636 − 22,728 + 1,561 at the 262,144 draft window) — ~6×
+the ~0.4 GiB the one-slot era assumed, because the drafter's KV is
+built against the target's FULL context, not the 3-token proposal
+depth; the lighter RCO-IQ3_XXS quant (−1.56 GiB vs IQ3_S-mtp) funds it.
+**Prefill 1,209 t/s @ 3,760 tokens; decode 52.0–58.2 tok/s (recipe
+temp 1.0) / 66.9 (temp 0)**, acceptance 34.2–54.4 % — decode is
+ubatch-invariant, the spread is temp-1.0 acceptance noise.
+`reasoning_effort` probe unchanged: `xhigh` 200, `high` → 500 Jinja
+raise (the models.yml fold stays mandatory).
 
-The **tcclaviger RDNA4 vLLM fork** (`docker.io/tcclaviger/vllm@sha256:da4029666…`,
-digest-pinned) serving **`orcarouter/Qwen3.8-Flash-Next-Uncensored-FP8`**
-(~172.8 GiB, 131 shards) on **both** Radeon AI PRO R9700s (gfx1201), **TP=2**,
-**fp8 KV**, **no speculative decode**, **expert prefetch OFF**, and since
-2026-10-03 the **shape-G** geometry (shape E + MNBT 2048 + the CUDA-graph plan
-term at the measured floor): `--max-model-len 200704` × 3 slots with
-`--kv-cache-memory-bytes 3,166,500,000`/rank (~400,700 KV tokens — the engine
-prices this pool at 7,891.8 B/token here), util **0.982** (the law — see the
-unbootable-1.0 lesson below), `--expert-precise-memory`,
-`VLLM_CUDAGRAPH_MEM_BASE_MB=0` / `_PER_GRAPH_MB=12`,
-`GPU_MAX_HW_QUEUES=2` and a mounted `/root/.cache`. `models.yml` carries
-`contextWindow: 200704` for this endpoint — the served window, not the model's
-native 262144, so the harness compacts before vLLM answers a longer prompt
-with a 400.
+**Vision with the publisher's own projector**: ajgazin's repo SHIPS
+`mmproj-BF16.gguf` in the same repo (the IQ3_S era's unsloth mmproj was
+built from the SHARED BASE `Qwen/Qwen3.8-27B`, legal because Swift-1.5's
+vision stack was hash-proven byte-identical to the base). The probe
+traps carry over: ≥256 px probe images or the model answers "video";
+only the OpenAI `image_url` content part reaches the model. **New on
+this payload: the uncensored model thinks more than the censored
+Swift 1.5** — a 64-token budget is eaten entirely by
+`reasoning_content` (empty `content`); 128 tokens clears the vision
+probe.
 
-**`--max-num-seqs 2` is impossible on this model.** Measured 2026-10-02 by
-single-variable boot probe against the then-deployed shape: SEQS 2 fails at
-BOTH MNBT 1024 and 2048 with `torch.fx.experimental.symbolic_shapes.Constraint
-ViolationError: Constraints violated (L['query_start_loc'].size()[0]) …
-specialized it to be a constant (3)`; SEQS 3 passes at both. `model_runner.py`
-sizes that input buffer `max_num_reqs + 1` and `_dummy_run` sets
-`num_reqs = min(max_num_batched_tokens, max_num_reqs)`, so the width is
-`SEQS + 1` for any `MNBT ≥ SEQS` — MNBT only moves the compile RANGE
-((1,1024) vs (1,2048)). `--enforce-eager` bypasses the path but drops CUDA
-graphs. The same failure class killed the 2026-10-01 bf16 2×262144 shape.
+The RCO-IQ3_XXS tier is a **per-tensor allocation mix** of stock quant
+types (IQ3_S 97 / BF16 96 / IQ3_XXS 83 / IQ2_S 72 / IQ4_XS 33 / IQ2_XS 32
+/ IQ2_XXS 30 / Q2_K 28 / Q4_K 19 / Q6_K 8 / IQ1_S 4 / IQ1_M 4) — stock
+llama.cpp loads it; the tensor-type histogram is the law (see Key
+Lessons). Publisher KLD vs Swift BF16: mean 0.002631 / 90th 0.042136 /
+max 34.49 — RCO is the publisher's "more experimental" family, and the
+payload is the UNCECERSED Swift 1.5 (23/100 refusals vs Swift 1.5's
+98/100, KL 0.0884).
 
-**The expert arena has THREE reachable levers: `max_model_len`, the overhead
-term, and the CUDA-graph estimate.**
-The fork's planner sizes expert cache as `budget − non-expert − KV reserve −
-graphs − overhead`, and the KV reserve is registry-priced at 9190 B/token ×
-`max_model_len`: 262144 → 2.48 GiB reserve → arena 14.30 GiB / **6,248 slots**;
-200704 → 1.91 GiB → 14.87 GiB / **6,496 slots**. Δreserve 0.57 GiB = Δarena
-0.57 GiB. The 6.1e9 → 3.15e9 pool cut bought **zero** arena; it buys runtime
-headroom (the shape it replaced died with `0 bytes free` on a real 995-token
-request). The second lever is `--expert-precise-memory`, **ON since
-2026-10-02**: it swaps the registry's default overhead (7.60 GiB = fitted +
-compile/graph-capture headroom) for the fitted figure, and the 2.35 GiB
-difference lands in the arena — measured plan lines at 200704 × 3: overhead
-5.25 GiB, expert cache **17.22 GiB / 7,522 slots** (+1,026 / +16 %), swap region
-13.7 GiB / 5,986 slots, boot healthy in 420 s. The pool stayed at 3.15e9
-deliberately: because the pool is not an arena lever, cutting it would cost
-concurrency (a 2.2e9 probe reports 277,686 tokens / **1.38×**) and buy no slots.
-The arena (2.344 MiB per slot) is the decode-critical region — cutting it for KV
-headroom collapses the offload hit rate.
+Runbook:
+`serverAppConfigs-docs/LLM/frank-secondary-llama-swift15-unc-iq3xxs-mtp-114688.md`.
+Rollback: `/home/dank/models/Swift15-IQ3S/` (the IQ3_S-mtp payload live
+until the 2026-10-06 evening swap — revert da36fa3 for the two-slot
+no-spec shape) or `/home/dank/models/MiMo-Q6K/` (revert to the
+2×200,704 f16 shape — [[frank-secondary-llama-mimo-q6k-2x200704]]) or
+`/home/dank/models/Ornith-Q8/` (the committed-never-deployed Q8_0;
+revert to 380aafa's tree).
 
-**Free VRAM on this endpoint is NOT arena headroom (2026-10-03 audit).**
-`rocm-smi` reads ~3.5 GiB free per card after hours of real traffic, but the
-planner charges `budget = total × util` against terms that overstate reality:
-the KV reserve is registry-priced at 9190 B/token (`1.91 GiB` at 200704) while
-the real pool bills 2.93 GiB at 7,892 B/token, and the precise overhead (5.25
-GiB) is far above the idle-measured figure. So the visible free space is the
-estimator gap, not allocatable arena. `gpu_memory_utilization` is a real arena
-lever (each +0.01 = +0.32 GiB) but **0.982 is the law, and 1.00 is UNBOOTABLE
-on this fork — measured 2026-10-03**: the worker init demands
-`free > budget`, and at `le=1` budget == device total, so both ranks raise
-`ValueError: Free memory on device cuda:N (31.29/31.86 GiB ...)` before a
-single weight loads, twice (2-restart crash loop, zero payload progress). The
-CLI field accepting `le=1` is the trap; Frank's 0.982 is where it always
-ships. What IS reachable from flags: `VLLM_CUDAGRAPH_MEM_BASE_MB` /
-`_PER_GRAPH_MB`, read at call time by `estimate_cudagraph_memory()` — the
-analytic default (256 MiB + 52 MiB/graph) charged 0.55 GiB where the engine's
-own post-capture line reported 0.31 GiB, and a second boot (with the plan
-already charging 0.32 GiB) reported **-0.06 GiB**, so the 256 MiB base pool is
-pure over-charge: shape G charges `0 base + 12 MiB/graph` = 0.07 GiB, which
-still prices per graph for a future 18-graph spec-decode shape (0.21 GiB).
-Verify every deploy by comparing the `Estimated CUDA graph memory`
-line with `Graph capturing finished … took X GiB`; if capture exceeds the
-estimate, raise `_PER_GRAPH_MB` back toward 52. MNBT 1024 → 2048 costs ZERO
-arena: the registry overhead is `base + 0.22 GiB × max(0,(nbt−4096)/1024)`
-anchored at nbt 4096, so every rung ≤4096 charges the same 5.25 GiB — and the
-fitted figure was fitted AT 4096.
+### frank-primary — Radiance (StillDeadcode), Qwen3.8-Flash-Next FP8 native, TP=2 (Dankserver, ACTIVE)
 
-**A container's compile cache does NOT survive `podman rm -f`, and on this
-endpoint that is 181 s per boot.** Measured 2026-10-02: the image ships **no
-`/root/.cache` at all** (fresh container: `No such file or directory`); after a
-boot it holds `/root/.cache/vllm` at **1,154,714,333 B in 4,665 files** plus
-`/root/.cache/comgr` at 229 MB, all in the ephemeral layer, and the engine
-prints `torch.compile took 181.20 s in total` (173.19 s of it one graph,
-`Compiling a graph for compile range (1, 1024)`). Every deploy, restart and
-reboot re-pays it. The serve script now mounts
-`FRANK_PRIMARY_CACHE_DIR` at `/root/.cache` — the parent, so the vLLM AOT /
-torch_compile tree (`VLLM_CACHE_ROOT`, whose default is
-`/root/.cache/vllm`) and the comgr HIP-kernel JIT cache both persist, and
-nothing in the image is hidden because the image has no such directory. The
-compile RANGE is part of the cache key, so an MNBT change costs one
-recompilation, not a permanent tax. Same class of fix as the recipe's
-`/cache/{triton,vllm,inductor}` volumes. **Proven live 2026-10-02 on
-consecutive boots:** cold `Compiling a graph for compile range (1, 1024) takes
-179.24 s` / `torch.compile took 188.26 s`; warm `Directly load the compiled
-graph(s) for compile range (1, 1024) from the cache, took 2.143 s` /
-`torch.compile took 3.97 s in total` / `init engine (profile, create kv cache,
-warmup model) took 41.13 s (compilation: 3.98 s)`. Time to `/health` is
-dominated by the 172.8 GiB payload load, so the visible win is the ~184 s of
-inductor/AOT build, not a proportional boot cut.
+The **radiance engine** (StillDeadcode, Apache-2.0; image
+`stilldeadcode/radiance@sha256:4c3f484564bb2872784e4…` digest-pinned,
+287 MiB, `libr4d@0.5.0` kernels prebuilt for gfx1201, engine as the image
+entrypoint) serving the **official `Qwen/Qwen3.8-Flash-Next-FP8`
+checkpoint at NATIVE precision** — modelopt block-FP8 throughout, no
+requantisation, no calibration. Plain `podman run` through
+`scripts/serve-frank-primary.sh` under the root systemd unit
+(`frank-primary.service`, `User=dank`, `LimitMEMLOCK=infinity`); the
+container name is the unit name (`frank-primary`). Cutover LIVE 2026-10-06
+00:55 (chain built 2026-10-05/06);
+rollback tag `frank-primary-freetoken-pr587-20261005` (the FreeToken trio
++ Dockerfile are in the tag; the image and the orcarouter payload stay on
+disk until the window closes), older rollback `frank-primary-vllm-shapeG-20261003`.
 
-Measured 2026-09-29 tuning campaign (fp8 + MTP4 era, since retired): decode
-**37.4-40.6 tok/s, acceptance ~54 %**; the 256k ctx arm measured 34.1-36.5;
-2-way concurrency sums BELOW parity (25.49 vs 26.8 single) — slots are a
-capacity feature, not a throughput one. V-only q8 KV is NOT possible via
-config: stock `--kv-cache-dtype` is K+V together; a V-only quant needs a fork
-patch.
+**Payload chain**: official checkpoint → staging dir whose `config.json`
+drops `quantization_config` → `stack-qwen38-official-fp8.py` (two
+value-preserving reshapes: experts byte-stacked into the plugin's stacked
+`gate_up_proj`/`down_proj` layout with `_scale_inv` at the same index — 49
+stacks × 512 experts, no requantisation; PLE's 128 scaleless fp8 shards
+decoded to one bf16 shard with the checkpoint's own bf16 scale
+1.9931793212890625e-04) → `rad-convert` with ONE `--quant` rule — pure
+copy for every weight, and the rtn rule re-encodes the bf16 PLE at that
+same fixed scale, an exact round-trip that restores the original codes and
+lands the plugin's one-element scale plane →
+`/home/dank/models/rad/qwen4exp-fp8-native.rad` (177,109 MiB measured,
+floor `FRANK_PRIMARY_RAD_MIN_MIB=177000`). The staging drop is the plugin
+SELECTION law (spec §2.4): the descriptor is `(arch_id, quant)` matched
+exactly, no shipped plugin claims `(qwen4exp, "fp8")`, the shipped
+`qwen4exp_fp8` plugin claims `(qwen4exp, "")` — and the descriptor is only
+the selection key, per-weight encodings are data, so nothing else is
+requantised.
+Raw serve AND unmodified convert both refuse; that is the design. The
+checkpoint itself is sha256-verified against the Hub (131/131 shards,
+2026-10-05). Built by `scripts/build-frank-primary-rad.sh` (`--plan-only`
+first — its declare phase is the engine's own, so a resolving plan boots).
 
-Unit and self-healing: root systemd unit (`scripts/frank-primary.service`,
-`User=dank`, `LimitMEMLOCK=infinity`), `ExecStop=podman stop -t 120
-frank-primary-qwen`. The serve script drives `podman run` directly, gates
-the digest-pinned image, the payload's `config.json` and its exact
-131-shard count and the GPU device nodes, clears stale container
-identities, then blocks on `podman wait`. `deploy_llm_primary` gates the
-same, waits out :7770, relinks/enables/restarts the unit and health-gates
-with the MainPID check. Boot cost ~7 min warm (the 172.8 GiB payload load; the
-~184 s inductor/AOT build inside it is what the mounted cache now skips — see
-the compile-cache lesson below); health returns 503 while loading.
+**Shape (both R9700s, TP=2)**: `--tp-wire wht6` (fused lossy all-reduce,
++5.6–6.3 % prefill / ~−1.3 % 8-stream decode at TP2), `--max-num-seqs 2`
+(Frank's first boot; raise from the ledger + slot occupancy), `--max-model-len
+200000` (native 262,144; the harness contextWindow must quote this — a longer
+prompt is refused, never truncated), `--placement expert_tiered
+--host-pool-mib 40960` (native FP8 experts are 112.8 GiB total, 56.4 GiB
+per rank slice — the pinned pool is the price of native precision),
+`--expert-vs-cache-ratio 0.82` (the measured 99 %-hit point; 0.87 collapses
+to 0 %), `--kv-cache-dtype fp8` (**7.32 KiB/token**; a 200K sequence is
+1.43 GiB), `--num-speculative-tokens 3` (the checkpoint's own MTP head —
+3,101 `mtp.*` tensors, probe depth 3), `--max-num-batched-tokens 2048`,
+prefix cache 4 GiB pinned host + 64 GiB disk at
+`/home/dank/radiance-state`, and **`--ngram-placement ram`**: the 47.7 GiB
+PLE n-gram table (51.2 GB = 29 % of all parameters) serves from pinned
+host RAM. Froggeric v22.5 chat template vendored
+(`scripts/qwen38-froggeric-v22.5.jinja`, `--override-chat-template`).
 
-Runbook: `serverAppConfigs-docs/LLM/dankserver-qwen38-fp8-expert-offload-tuning.md`.
+**The pinned-RAM ledger is why this is a root unit**: 2 × 40 GiB pools +
+47.7 GiB PLE + 4 GiB prefix host ≈ **134 GiB** charged against
+`RLIMIT_MEMLOCK`. `ttm.pages_limit`/`pages_max` cap GPU-mappable pinned
+pages (pre-cutover cap 123 GiB — BELOW the plan); the serve script's
+TTM preflight (`FRANK_PRIMARY_TTM_MIN_MIB=140000`) refuses to start until
+both knobs are raised in the stop window (sysfs write first, kernel
+cmdline + reboot only if read-only).
+
+**Contract facts** (source-verified): the HTTP listener binds AFTER the
+full load, so a bare `curl /health` gate is honest (`/health` == `/ping`,
+200 serving / 503 stopping); sampling params temperature…preserve_thinking
+accepted, **unknown params 400** (models.yml's extraBody set is fully
+inside it); `response_format` supported; `logprobs` REFUSED; reasoning in
+`reasoning_content` and tool calls in the qwen3.8 dialect via the auto
+chat format — no parser flags; vision native (the 27-layer Qwen-VL tower,
+`--mm-max-patches`); `/metrics` carries vLLM-compatible names +
+`radiance:*` families — the Dankclaw Prometheus scrape is re-added.
+
+Runbook: `serverAppConfigs-docs/LLM/frank-primary-radiance-qwen38-native-fp8.md`.
+Retired FreeToken era: `serverAppConfigs-docs/LLM/frank-primary-freetoken-pr587.md`.
+Retired vLLM era: `serverAppConfigs-docs/LLM/dankserver-qwen38-fp8-expert-offload-tuning.md`.
 Retired GLM experiment: `serverAppConfigs-docs/LLM/frank-primary-glm53-flash-w4a16.md`.
 
 ## Key Lessons Learned
@@ -774,7 +792,7 @@ linked runbooks.
   121 GiB box, pushed `available` to 3 GiB, and every later rate was suspect.
   Record the child pid and kill it in a `finally`, then prove `free` came back
   before believing a number. Kill by pid — never `pkill -f llama-server`,
-  which also hits frank-small.
+  which also hits frank-secondary.
 
 **llama.cpp serving**
 
@@ -840,7 +858,7 @@ linked runbooks.
   free. That was a CONTIGUITY failure, not a total-bytes one, and no totals
   model — the sweep's or the seam's — can see it.)
 - **On the CUDA 3090 the same knob has NO interior optimum up to 2048, so
-  frank-small ships 4096/2048.** Measured 2026-10-05 by boot-pair A/B on the
+  frank-secondary ships 4096/2048.** Measured 2026-10-05 by boot-pair A/B on the
   live service: at 8,928 prompt tokens ubatch 1024 gives 3,972 / 4,077 t/s and
   ubatch 2048 gives 4,207 / 4,212 — **+4.6 %** — for **616 MiB** (free 2,849 →
   2,279). Decode did not move (91.9–93.0 vs 93.8–94.3, i.e. noise), and a
@@ -849,9 +867,9 @@ linked runbooks.
   than the ubatch cannot see it. The logical `--batch-size` was again free. And
   the predecessor's "ubatch 2048 failed the scratch boot" was a shape
   observation, not a law: the same file boots 2048 clean at 2 × 200,704. Bench
-  it by restarting the SERVICE through a transient `FRANK_SMALL_ENV_FILE`
+  it by restarting the SERVICE through a transient `FRANK_SECONDARY_ENV_FILE`
   drop-in, never a scratch port — one card holds one server. See
-  [[frank-small-llama-mimo-q6k-2x200704]].
+  [[frank-secondary-llama-mimo-q6k-2x200704]].
 - **One `-ts` step is one LAYER, and a hybrid stack's layers are not the same
   size.** qwen35 is 16 x (3 Gated DeltaNet + 1 Gated Attention), so a step is
   worth ~0.43 GiB on a DeltaNet boundary and ~1.43 GiB on a full-attention one
@@ -873,7 +891,7 @@ linked runbooks.
   CTX_SIZE / PARALLEL`, and llama.cpp refuses a prompt longer than the SLOT
   (`400 exceed_context_size_error: request (184001 tokens) exceeds the
   available context size (180224 tokens)`) even when the pool has room. So
-  frank-small's 2 → 3 reslice moved each conversation from the model's native
+  frank-secondary's 2 → 3 reslice moved each conversation from the model's native
   262,144 down to 180,224, and `models.yml` `contextWindow` had to follow or
   the harness would discover the ceiling as a 400 on a long conversation. Side
   effect worth knowing: the shorter slot made decode ~5 % FASTER (106.8 vs
@@ -884,9 +902,9 @@ linked runbooks.
   day the freed space also bought the Q6_K rung (now 94.4 tok/s — the rung
   itself is a further measured −10 %, weight-bandwidth, not slot geometry).
   All moves measured; both slot directions cost or buy window, never pool.
-  See [[frank-small-llama-mimo-q6k-2x200704]].
+  See [[frank-secondary-llama-mimo-q6k-2x200704]].
 - **Price a VRAM budget from two boots, never from a formula — and an mmproj
-  costs ~1.5x its file size.** The frank-small 3090 fit was solved from two
+  costs ~1.5x its file size.** The frank-secondary 3090 fit was solved from two
   measurements: 2 × 262,144 with no projector = 20,190 MiB used (KV 12,800 ⇒
   **non-KV 7,390**), 3 × 196,608 with the f16 projector = 22,972 used (KV
   14,113 ⇒ **non-KV 8,859**). The difference is the vision tower: a 918 MB file
@@ -894,12 +912,12 @@ linked runbooks.
   ride with it. Subtracting the two rows is exact; extrapolating weights+KV+
   "a bit for buffers" is what leaves 1.1 GiB free when you meant 2.7. Then
   solve `KV ≤ card − non-KV − headroom` and round to a multiple of
-  `PARALLEL × 4096`. Recorded in `scripts/frank-small.env` so the next resize
+  `PARALLEL × 4096`. Recorded in `scripts/frank-secondary.env` so the next resize
   starts from measurement.
 - **`images` and `image_url` are NOT interchangeable: only the content part
   reaches the model.** `--mmproj` loads, `usage.prompt_tokens` rises, and the
   model still invents an answer if the client used the legacy top-level
-  `images` field on `/v1/chat/completions` — frank-small answered "Brown." for
+  `images` field on `/v1/chat/completions` — frank-secondary answered "Brown." for
   a solid-red PNG that way and "Red." through the OpenAI `image_url` part. The
   failure is silent because a hallucinated color is a plausible answer. Probe
   vision with a solid color and expect an exact word.
@@ -911,7 +929,7 @@ linked runbooks.
   returns `tool_calls`. Check tool support before routing a role at an endpoint.
 - **Quantized KV on CUDA needs `GGML_CUDA_FA_ALL_QUANTS=ON`** — the same
   option and the same ~12x prefill collapse described in the ROCm note above.
-  f16 KV needs nothing; frank-small ran f16 for that reason until
+  f16 KV needs nothing; frank-secondary ran f16 for that reason until
   2026-09-17, when `build-cuda` was rebuilt with the option ON so the
   endpoint could take q8_0 KV.
 - **llama-bench separates a LIST with `/` and SWEEP values with `,`.** Both
@@ -929,7 +947,7 @@ linked runbooks.
   raises on anything else — HTTP 500, not a clamp — so the harness's levels
   must be folded onto that set in `models.yml` (`compat.reasoningEffortMap`).
   A role whose level is unmapped fails on its FIRST request, and the failure
-  names Jinja, not the model: every subagent on `frank-small` died with
+  names Jinja, not the model: every subagent on `frank-secondary` died with
   `Unexpected reasoning effort high` until `high: xhigh` was added on
   2026-09-16. Halogen accepts the wider set, so the two entries legitimately
   differ.
@@ -1003,9 +1021,9 @@ Active:
 
 | Service | Runbook |
 |---------|---------|
-| frank-large (halogen container, Qwen3.8-27B .hgn + DFlash2, 1×200,700, text-only, Dankclaw) | `serverAppConfigs-docs/LLM/frank-large-halogen27b-dflash2.md` |
-| frank-small (llama.cpp CUDA, MiMo-V2.6-Distill-Qwen-9B Q6_K, 2×200,704 f16 KV, Dankclaw) | `serverAppConfigs-docs/LLM/frank-small-llama-mimo-q6k-2x200704.md` |
-| frank-primary (tcclaviger vLLM, Qwen3.8-Flash-Next FP8, Dankserver) | `serverAppConfigs-docs/LLM/dankserver-qwen38-fp8-expert-offload-tuning.md` |
+| frank-large (flash-server 0.16.2, abliterated Swift1.5 v2 .hgn + MTP always-on, 2×200,192, vision ON, Dankclaw) | `serverAppConfigs-docs/LLM/frank-large-flash-server-v2-swift15.md` |
+| frank-secondary (llama.cpp CUDA, Swift-1.5-27B Uncensored RCO IQ3_XXS-mtp, 2×114,688 draft-mtp q8_0 KV, Dankclaw) | `serverAppConfigs-docs/LLM/frank-secondary-llama-swift15-unc-iq3xxs-mtp-114688.md` |
+| frank-primary (radiance StillDeadcode, Qwen3.8-Flash-Next-FP8 native, 2×R9700 TP=2, Dankserver) | `serverAppConfigs-docs/LLM/frank-primary-radiance-qwen38-native-fp8.md` |
 | Dankclaw kernel posture (`amd_iommu=off`) | `serverAppConfigs-docs/LLM/dankclaw-iommu-translation-off.md` |
 | Deployment flow + incidents | `serverAppConfigs-docs/LLM/deployment.md` |
 
@@ -1013,13 +1031,18 @@ Retired (rollback targets and history):
 
 | Runtime | Host | Window | Runbook |
 |---------|------|--------|---------|
+| halogen 0.1.4 Qwen3.8-27B p1w4d-d2 + DFlash2 (`frank-large`) | Dankclaw | 2026-10-05 → 10-06 (rollback target — payload + fetch script + image digest on disk; owns the prompt-cache pin and FA-tile lessons) | `frank-large-halogen27b-dflash2.md` |
 | ROCmFPX llama.cpp fork Qwen3.8-27B Uncensored Q6_0_ROCMFPX_AGENT + MTP (`frank-large`) | Dankclaw | 2026-10-03 → 10-05 (rollback target, payload + fork build on disk) | `frank-large-rocmfpx-q6agent-uncensored-mtp.md` |
+| tcclaviger vLLM fork Qwen3.8-Flash-Next-Uncensored FP8, TP=2, shape G (`frank-primary`) | Dankserver | 2026-09-29 → 10-05 (rollback target — tag `frank-primary-vllm-shapeG-20261003`; trio + env lore in the tag, image in rootless storage) | `dankserver-qwen38-fp8-expert-offload-tuning.md` |
+| FreeToken PR #587 ROCm, Qwen3.8-Flash-Next-Uncensored-FP8, ONE R9700 (`frank-primary`) | Dankserver | 2026-10-05 → 10-05 (rollback target — tag `frank-primary-freetoken-pr587-20261005`; trio + Dockerfile in the tag, image `localhost/frank-primary-freetoken:pr587-77a7201` and orcarouter payload on disk) | `frank-primary-freetoken-pr587.md` |
+| llama.cpp master MiMo-V2.6-Distill-Qwen-9B Q6_K 2×200,704 f16 KV (`frank-secondary`) | Dankclaw | 2026-10-03 → 10-05 (rollback target — payload on disk; owns the ubatch-2048 A/B; the Ornith swap of 10-05 was committed but never deployed, so this was the LIVE payload Swift replaced) | `frank-secondary-llama-mimo-q6k-2x200704.md` |
+| llama.cpp Ornith-1.5-9B Q8_0, ONE 262,144 f16 KV (`frank-secondary`) | Dankclaw | 2026-10-05 (committed 77decb5→380aafa, never deployed — superseded by Swift the same evening; payload on disk) | trail in `frank-secondary-llama-swift15-unc-iq3xxs-mtp-114688.md` |
 | ROCmFPX llama.cpp fork Qwen3.8-27B Q8_0_ROCMFPX_AGENT + MTP (`frank-large`) | Dankclaw | 2026-10-02 → 10-03 (rollback target, payload on disk) | `frank-large-rocmfpx-q8agent-mtp.md` |
 | gufo v0.1.1 Qwen3.8-27B UD-Q8_K_XL/UD-Q4_K_XL + DFlash2 (`frank-large`) | Dankclaw | 2026-09-28 → 10-01, again 10-01 → 10-02 (payloads removed from disk 2026-10-03; rollback = re-download) | `frank-large-gufo-qwen38-27b.md` |
 | vLLM TheRock MiMo-V2.6-Distill-Qwen-9B BF16 + cross-model MTP (`frank-large`) | Dankclaw | 2026-10-01 (same-day rollback to gufo) | `frank-large-vllm-mimo-v26-bf16.md` |
-| llama.cpp master MiMo-V2.6-Distill-Qwen-9B Q6_K (`frank-small`) | Dankclaw | 2026-10-01 → 10-02 (the rung returned ACTIVE 2026-10-03 at the 2×200,704 shape — `frank-small-llama-mimo-q6k-2x200704.md`) | `frank-small-llama-mimo-q6k.md` |
-| llama.cpp master MiMo-V2.6-Distill-Qwen-9B Q5_K_M 2×200,704 f16 KV (`frank-small`) | Dankclaw | 2026-10-02 → 10-03 (rollback target — one `MODEL_FILE` line, buys back 875 MiB + the −10 % rung tax; owns the boot-pair table) | `frank-small-llama-mimo-q5km.md` |
-| SGLang MiMo-V2.6 INT8-W8A16-AutoRound (`frank-small`) | Dankclaw | 2026-10-01 (same-day swap to the GGUF path) | `frank-small-sglang-mimo-v26.md` |
+| llama.cpp master MiMo-V2.6-Distill-Qwen-9B Q6_K (`frank-secondary`) | Dankclaw | 2026-10-01 → 10-02 (the rung returned ACTIVE 2026-10-03 at the 2×200,704 shape — `frank-secondary-llama-mimo-q6k-2x200704.md`) | `frank-secondary-llama-mimo-q6k.md` |
+| llama.cpp master MiMo-V2.6-Distill-Qwen-9B Q5_K_M 2×200,704 f16 KV (`frank-secondary`) | Dankclaw | 2026-10-02 → 10-03 (rollback target — one `MODEL_FILE` line, buys back 875 MiB + the −10 % rung tax; owns the boot-pair table) | `frank-secondary-llama-mimo-q5km.md` |
+| SGLang MiMo-V2.6 INT8-W8A16-AutoRound (`frank-secondary`) | Dankclaw | 2026-10-01 (same-day swap to the GGUF path) | `frank-secondary-sglang-mimo-v26.md` |
 |tcclaviger vLLM GLM-5.3-Flash W4A16 int4, +/- DFlash2 drafter (`frank-primary`)|Dankserver|2026-09-30 → 10-01|`frank-primary-glm53-flash-w4a16.md`|
 | llama.cpp b11030 Qwen3.8-27B UD-Q8_K_XL + DFlash2 (`frank-large`) | Dankclaw | 2026-09-21 → 09-28 | `frank-large-unsloth-qwen38-27b-q8xl.md` |
 | R9V radiance-vLLM Qwen3.8-Flash-Next-Uncensored IQ4_XS (`frank-primary`) | Dankserver | 2026-09-27 (same-day swap to Q8_0) | `frank-primary-r9v-radiance-qwen38-uncensored.md` |
@@ -1032,16 +1055,16 @@ Retired (rollback targets and history):
 | CIRU v4.4.1 Orca Qwen3.8-Flash (`frank-large`) | Dankclaw | 2026-09-19 → 09-21 | `frank-large-ciru-orca.md` |
 | unsloth llama.cpp b11030 Qwen3.8-Flash-Next UD-IQ4_XS (`frank-large`) | Dankclaw | 2026-09-21 (same-day swap) | `frank-large-unsloth-qwen38-flash-next.md` |
 | Kairic Edge Qwen3.8-27B IU4 W4A4 (`frank-large`) | Dankclaw | 2026-09-21 (same-day swap) | `frank-large-kairic-edge-qwen38-27b.md` |
-| ISTA-DASLab Qwen3.8-27B-GSQ-RCO IQ3_S (`frank-small`) | Dankclaw | 2026-09-16 → 09-17 | `frank-small-qwen38-gsq-rco.md` |
-| vLLM Gemma4-12B w4a16 (`frank-small`) | Dankclaw | 2026-09-19 (same-day revert) | `frank-small-vllm-gemma4.md` |
-| llama.cpp Gemma4-12B QAT Q4_K_M (`frank-small`) | Dankclaw | 2026-09-17 → 10-01 | `frank-small-gemma4-12b.md` |
+| ISTA-DASLab Qwen3.8-27B-GSQ-RCO IQ3_S (`frank-secondary`) | Dankclaw | 2026-09-16 → 09-17 | `frank-secondary-qwen38-gsq-rco.md` |
+| vLLM Gemma4-12B w4a16 (`frank-secondary`) | Dankclaw | 2026-09-19 (same-day revert) | `frank-secondary-vllm-gemma4.md` |
+| llama.cpp Gemma4-12B QAT Q4_K_M (`frank-secondary`) | Dankclaw | 2026-09-17 → 10-01 | `frank-secondary-gemma4-12b.md` |
 | agentionai ROCmFP4-FAST imatrix | Dankclaw | 2026-08-29 → 08-30 | `agentionai-fast-imatrix.md` |
 | eemin Qwen3.8-27B ROCmI4 W4A4 MTP | Dankclaw | 2026-08-18 → 08-25 | `eemin-rocmi4-mtp.md` |
 | Kairic Edge IU4 | Dankclaw | 2026-08-22 (same-day revert) | `kairic-edge.md` |
 | CIRU ActiveFPX v3 / PromptForge | Dankclaw | through 2026-08-22 | `ciru-activefpx-promptforge.md` |
 | DFlash2 Vulkan | Dankclaw | 2026-08-21 (same-day revert) | `dflash2-large-vulkan.md` |
 | FastFlowLM NPU (`frank-npu`) | Dankclaw | twice, retired 2026-09-11 | `frank-npu.md`, `FastFlowLM NPU Inference.md` |
-| FreeToken (official + Huihui FP8, `frank-small`) | Dankserver | 2026-09-02 → 09-15 | `freetoken-frank-small.md` |
+| FreeToken (official + Huihui FP8, `frank-secondary`) | Dankserver | 2026-09-02 → 09-15 | `freetoken-frank-small.md` |
 | FastMTP patched llama.cpp | Dankserver | retired 2026-08-18 | `qwen3.8-spec-matrix.md` |
 | ik_llama.cpp Ornith CRACK | Dankserver | retired 2026-09-02 | `ikllama-ornith-dankserver.md` |
 | llama.cpp layer split + DFlash2 (`frank-medium`) | Dankserver | 2026-09-16 → 09-20 | `frank-medium-llamacpp-dflash2.md` |

autoresearch.sh

@@ -1,117 +1,26 @@
 #!/usr/bin/env bash
-# autoresearch.sh — canonical benchmark entrypoint: speed of frank-primary.
+# autoresearch.sh — canonical frank-primary benchmark entry point (dev box).
 #
-# Runs on the operator box and drives one bench window on DANKSERVER:
-#   1. fingerprint the candidate trio (env + serve script + unit)
-#   2. memo fast-path: an identical fingerprint that already produced a FULLY
-#      successful window (bench + verified production restore) returns the
-#      cached metrics without touching the fleet
-#   3. stage the candidate trio plus the measure client into
-#      /home/dank/.autoresearch-frank-primary/ (ephemeral; the repo checkout
-#      on the host stays the production source of truth)
-#   4. run the window (scripts/autoresearch/frank-primary-window.sh):
-#      stop + mask the production unit, boot the staged config AS THE SERVICE
-#      (same serve script, same container, same port 7770), wait for health,
-#      run the in-container usage-counter measure, then restore the production
-#      unit and verify its health — AGENTS.md Benchmarking Policy
-#   5. print METRIC lines (parse by run_experiment)
-#
-# Primary metric: decode_tps (HIGHER is better) — median across the three
-# prompt depths (115/1056/4096) of per-depth medians of
-# completion_tokens / (wall - ttft), 3 runs each after a warm pass,
-# temperature 0, the campaign's deterministic word-seed workload.
-#
-# Exit 0 with METRIC lines on success; non-zero on any failure (the window
-# script's EXIT trap restores production on every path).
-set -euo pipefail
+# The service runs on dankserver (192.168.0.235:7770), the two R9700s live
+# there, and the benchmark must own those cards, so the whole stop -> boot
+# scratch -> bench -> restore sequence runs ON dankserver (see
+# scripts/autoresearch-frank-primary.sh). This wrapper:
+#   1. copies the candidate config (the repo's scripts/frank-primary.env, the
+#      file the autoresearch loop edits) and the bench script to dankserver,
+#   2. runs the bench over ssh, relaying its stdout (the METRIC lines) and
+#      exit code back here verbatim.
+# It deliberately does NOT push to origin: the scp dirties dankserver's
+# working tree, and the loop's uncommitted candidate config must not be
+# pushed. Only this file and scripts/autoresearch-frank-primary.sh are
+# committed; the candidate env rides over ssh each run.
+set -uo pipefail
+DIR="$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)"
+REMOTE=dankserver
+REMOTE_REPO=/home/dank/serverAppConfigs
 
-SCRIPT_DIR="$(cd -- "$(dirname -- "${BASH_SOURCE[0]}")" && pwd)"
-HOST="${FRANK_PRIMARY_AR_HOST:-dankserver}"
-STAGE="/home/dank/.autoresearch-frank-primary"
+scp -q "$DIR/scripts/frank-primary.env" \
+    "$REMOTE:$REMOTE_REPO/scripts/frank-primary.env" || { echo "scp env failed" >&2; exit 1; }
+scp -q "$DIR/scripts/autoresearch-frank-primary.sh" \
+    "$REMOTE:$REMOTE_REPO/scripts/autoresearch-frank-primary.sh" || { echo "scp script failed" >&2; exit 1; }
 
-# FRANK_PRIMARY_AR_ENV_FILE stages a DIFFERENT env than the repo's — that is how
-# the arms driver A/Bs engine knobs without mutating the checkout. The
-# fingerprint covers whatever file is staged, so the memo cannot cross arms.
-ENV_FILE="${FRANK_PRIMARY_AR_ENV_FILE:-$SCRIPT_DIR/scripts/frank-primary.env}"
-SERVE="$SCRIPT_DIR/scripts/serve-frank-primary.sh"
-UNIT="$SCRIPT_DIR/scripts/frank-primary.service"
-MEASURE="$SCRIPT_DIR/scripts/autoresearch/frank-primary-measure.py"
-WINDOW="$SCRIPT_DIR/scripts/autoresearch/frank-primary-window.sh"
-
-for f in "$ENV_FILE" "$SERVE" "$UNIT" "$MEASURE" "$WINDOW"; do
-    [ -f "$f" ] || { echo "[autoresearch] missing harness file: $f" >&2; exit 1; }
-done
-command -v python3 >/dev/null 2>&1 || { echo "[autoresearch] python3 required on operator box" >&2; exit 1; }
-
-# Fingerprint of everything that can change the engine's argv or environment.
-HASH="$(cat "$ENV_FILE" "$SERVE" "$UNIT" | sha256sum | cut -d' ' -f1)"
-echo "[autoresearch] candidate fingerprint $HASH" >&2
-
-ssh_run() { ssh -o BatchMode=yes -o ConnectTimeout=15 "$HOST" "$@"; }
-
-emit_metrics() { # $1 = run.json
-    python3 -c '
-import json, sys
-run = json.loads(sys.argv[1])
-m = run["measure"]
-print("METRIC decode_tps=%s" % m["decode_tps"])
-for k in ("decode_tps_115", "decode_tps_1056", "decode_tps_4096"):
-    print("METRIC %s=%s" % (k, m[k]))
-print("METRIC ttft_ms_4096=%s" % m["ttft_ms_4096"])
-print("METRIC kv_tokens=%s" % (run.get("kv_tokens") or 0))
-print("METRIC boot_seconds=%s" % (run.get("boot_seconds") or -1))
-print("METRIC completion_tokens_4096=%s" % m.get("completion_tokens_4096", 0))
-# Any extra sampling temperature the measure swept (AR_TEMPS), e.g. the 0.7
-# arm — that is where speculative-decode acceptance shows up, since a greedy
-# draft matches a greedy target exactly and is rejected more under real
-# sampling. Suffixed keys only (_t0.7), so the historical lines are untouched.
-for k in sorted(m):
-    if "_t" in k and (k.startswith("decode_tps") or k.startswith("ttft_ms")):
-        print("METRIC %s=%s" % (k, m[k]))
-' "$1"
-}
-
-# --- memo fast-path -------------------------------------------------------
-cached_hash="$(ssh_run "cat '$STAGE/last/hash' 2>/dev/null" 2>/dev/null || true)"
-if [ -n "$cached_hash" ] && [ "$cached_hash" = "$HASH" ]; then
-    cached_run="$(ssh_run "cat '$STAGE/last/run.json' 2>/dev/null" || true)"
-    if [ -n "$cached_run" ]; then
-        echo "[autoresearch] memo hit — cached metrics for $HASH, fleet untouched" >&2
-        emit_metrics "$cached_run"
-        exit 0
-    fi
-fi
-
-# --- stage the candidate ----------------------------------------------------
-echo "[autoresearch] staging candidate on $HOST:$STAGE" >&2
-ssh_run "mkdir -p '$STAGE'"
-scp -q "$ENV_FILE" "$HOST:$STAGE/frank-primary.env"
-scp -q "$SERVE" "$HOST:$STAGE/serve.sh"
-scp -q "$MEASURE" "$HOST:$STAGE/measure.py"
-scp -q "$WINDOW" "$HOST:$STAGE/window.sh"
-ssh_run "printf '%s\n' '$HASH' > '$STAGE/hash'"
-
-# --- the window -------------------------------------------------------------
-# Budgets: bench boot up to 20 min (warm boots measured ~7 min), measure up
-# to 30 min, restore boot up to 20 min — call run_experiment with
-# timeout_seconds >= 3600.
-echo "[autoresearch] opening bench window (stop+mask -> boot -> measure -> restore)" >&2
-set +e
-ssh_run "bash '$STAGE/window.sh'"
-rc=$?
-set -e
-if [ "$rc" -ne 0 ]; then
-    echo "[autoresearch] window failed (rc=$rc); host-side evidence:" >&2
-    ssh_run "tail -60 '$STAGE/fail.log' 2>/dev/null || tail -40 '$STAGE/serve.out' 2>/dev/null || tail -40 '$STAGE/measure.out' 2>/dev/null || true" >&2 || true
-    exit 1
-fi
-
-run_json="$(ssh_run "cat '$STAGE/run.json'")" || {
-    echo "[autoresearch] could not read $STAGE/run.json back" >&2; exit 1; }
-echo "[autoresearch] window result:" >&2
-printf '%s\n' "$run_json" >&2
-emit_metrics "$run_json"
-
-echo "[autoresearch] engine evidence (plan / compile path / spec counters):" >&2
-ssh_run "cat '$STAGE/evidence.log' 2>/dev/null" >&2 || true
-exit 0
+ssh "$REMOTE" "bash $REMOTE_REPO/scripts/autoresearch-frank-primary.sh $REMOTE_REPO/scripts/frank-primary.env"

dankclaw/comfyui/workflows/qwen-image-2.1-t2i-art.json

@@ -0,0 +1,64 @@
+{
+  "3": {
+    "class_type": "CLIPTextEncode",
+    "_meta": { "title": "Positive prompt" },
+    "inputs": {
+      "text": "A detailed illustration of a 1930s Soviet industrial design bureau at night: a well-measured engineer at a drafting table, a warm desk lamp, rolled blueprints, gears and calipers on the desk, deep red and cream palette, gouache painterly style, intricate mechanical detail, soft shadows",
+      "clip": [ "6", 0 ]
+    }
+  },
+  "4": {
+    "class_type": "CLIPTextEncode",
+    "_meta": { "title": "Negative prompt" },
+    "inputs": {
+      "text": "lowres, blurry, deformed, extra limbs, watermark, signature, jpeg artifacts",
+      "clip": [ "6", 0 ]
+    }
+  },
+  "5": {
+    "class_type": "UNETLoader",
+    "_meta": { "title": "Qwen Image 2.1 (int8)" },
+    "inputs": { "unet_name": "qwen_image_2.1_int8_convrot.safetensors", "weight_dtype": "default" }
+  },
+  "6": {
+    "class_type": "CLIPLoader",
+    "_meta": { "title": "Qwen3-VL 8B text encoder (w4a8)" },
+    "inputs": { "clip_name": "qwen3vl_8b_w4a8.safetensors", "type": "qwen_image" }
+  },
+  "7": {
+    "class_type": "VAELoader",
+    "_meta": { "title": "VAE" },
+    "inputs": { "vae_name": "qwen_image_2.1_vae_bf16.safetensors" }
+  },
+  "8": {
+    "class_type": "EmptySD3LatentImage",
+    "_meta": { "title": "Empty Latent 1024x1024" },
+    "inputs": { "width": 1024, "height": 1024, "batch_size": 1 }
+  },
+  "9": {
+    "class_type": "KSampler",
+    "_meta": { "title": "KSampler" },
+    "inputs": {
+      "model": [ "5", 0 ],
+      "seed": 1930,
+      "steps": 20,
+      "cfg": 1.0,
+      "sampler_name": "euler",
+      "scheduler": "simple",
+      "positive": [ "3", 0 ],
+      "negative": [ "4", 0 ],
+      "latent_image": [ "8", 0 ],
+      "denoise": 1.0
+    }
+  },
+  "10": {
+    "class_type": "VAEDecode",
+    "_meta": { "title": "VAE Decode" },
+    "inputs": { "samples": [ "9", 0 ], "vae": [ "7", 0 ] }
+  },
+  "11": {
+    "class_type": "SaveImage",
+    "_meta": { "title": "Save Image (art)" },
+    "inputs": { "filename_prefix": "qwen21-art", "images": [ "10", 0 ] }
+  }
+}

dankclaw/comfyui/workflows/qwen-image-2.1-t2i-legible-text.json

@@ -0,0 +1,64 @@
+{
+  "3": {
+    "class_type": "CLIPTextEncode",
+    "_meta": { "title": "Positive prompt (legible text)" },
+    "inputs": {
+      "text": "Clean minimalist movie-poster design, dark charcoal background with a subtle vintage grain. Centered large bold white sans-serif text reading \"SOLIDARITY OF THE SOVIET COMPUTER SYSTEMS\", perfectly legible, crisp letterforms, even letter spacing. Below it, a smaller line of yellow text reading \"DANKCLAW DANKSERVER EST 2026\", equally legible. Flat graphic design, high contrast, generous margins, no other words, no watermark",
+      "clip": [ "6", 0 ]
+    }
+  },
+  "4": {
+    "class_type": "CLIPTextEncode",
+    "_meta": { "title": "Negative prompt" },
+    "inputs": {
+      "text": "blurry, illegible, misspelled, warped letters, extra text, watermark, signature, lowres, jpeg artifacts",
+      "clip": [ "6", 0 ]
+    }
+  },
+  "5": {
+    "class_type": "UNETLoader",
+    "_meta": { "title": "Qwen Image 2.1 (int8)" },
+    "inputs": { "unet_name": "qwen_image_2.1_int8_convrot.safetensors", "weight_dtype": "default" }
+  },
+  "6": {
+    "class_type": "CLIPLoader",
+    "_meta": { "title": "Qwen3-VL 8B text encoder (w4a8)" },
+    "inputs": { "clip_name": "qwen3vl_8b_w4a8.safetensors", "type": "qwen_image" }
+  },
+  "7": {
+    "class_type": "VAELoader",
+    "_meta": { "title": "VAE" },
+    "inputs": { "vae_name": "qwen_image_2.1_vae_bf16.safetensors" }
+  },
+  "8": {
+    "class_type": "EmptySD3LatentImage",
+    "_meta": { "title": "Empty Latent 832x1216 poster" },
+    "inputs": { "width": 832, "height": 1216, "batch_size": 1 }
+  },
+  "9": {
+    "class_type": "KSampler",
+    "_meta": { "title": "KSampler" },
+    "inputs": {
+      "model": [ "5", 0 ],
+      "seed": 2026,
+      "steps": 25,
+      "cfg": 1.0,
+      "sampler_name": "euler",
+      "scheduler": "simple",
+      "positive": [ "3", 0 ],
+      "negative": [ "4", 0 ],
+      "latent_image": [ "8", 0 ],
+      "denoise": 1.0
+    }
+  },
+  "10": {
+    "class_type": "VAEDecode",
+    "_meta": { "title": "VAE Decode" },
+    "inputs": { "samples": [ "9", 0 ], "vae": [ "7", 0 ] }
+  },
+  "11": {
+    "class_type": "SaveImage",
+    "_meta": { "title": "Save Image (text poster)" },
+    "inputs": { "filename_prefix": "qwen21-text", "images": [ "10", 0 ] }
+  }
+}

old/download-frank-large-mimo.sh

@@ -4,7 +4,7 @@
 # command line (a Bearer token in argv is world-readable via /proc).
 #
 # Per-file FRESH downloads, no resume: the house rule (see
-# download-frank-small.sh); -f/--fail-with-body plus size+sha256 against the
+# download-frank-secondary.sh); -f/--fail-with-body plus size+sha256 against the
 # Hub LFS oid is what makes an access error fail instead of "succeeding".
 #
 # Usage:

old/download-frank-large-qwen35.sh

@@ -2,7 +2,7 @@
 # Download the source Qwen3.5-9B checkpoint — the ONLY place this model
 # family's MTP head (15 mtp.* tensors) exists. frank-large runs it as the
 # vLLM cross-model MTP draft (MiMo-V2.6-Distill-Qwen-9B itself ships zero
-# mtp tensors; that is why SGLang's NEXTN on frank-small draws 0.00
+# mtp tensors; that is why SGLang's NEXTN on frank-secondary draws 0.00
 # acceptance). Public repo — no auth header on the command line.
 #
 # NOTE the odd upstream shard names (model.safetensors-0000N-of-00004...).

old/download-frank-secondary-sglang-int8.sh

@@ -1,5 +1,5 @@
 #!/usr/bin/env bash
-# Download MiMo-V2.6-Distill-Qwen-9B-INT8-W8A16-AutoRound for frank-small
+# Download MiMo-V2.6-Distill-Qwen-9B-INT8-W8A16-AutoRound for frank-secondary
 # (SGLang, RTX 3090). Public repo — no auth header on the command line.
 #
 # Per-file FRESH downloads, no resume: `curl -C -` once produced a 213 MB
@@ -7,7 +7,7 @@
 # the house rule.
 #
 # Usage:
-#   scripts/download-frank-small.sh [DEST_DIR] [--verify-only]
+#   scripts/download-frank-secondary.sh [DEST_DIR] [--verify-only]
 #
 # LFS shards are verified against the Hub LFS sha256; non-LFS text files
 # against their declared byte size. Idempotent: complete files are skipped.

scripts/autoresearch-frank-primary.sh

@@ -0,0 +1,186 @@
+#!/usr/bin/env bash
+# autoresearch-frank-primary.sh — RUNS ON DANKSERVER, driven over ssh by the
+# dev-box autoresearch.sh. It is the canonical frank-primary benchmark entry
+# point for the autoresearch loop.
+#
+# Method (mandated by the Benchmarking Policy): the production service serves
+# real users and shares the two R9700s, so a bench beside it starves the GPUs
+# and corrupts results. This script therefore:
+#   1. stops the production frank-primary unit (root, relink symlink: stop,
+#      NOT mask — mask refuses to replace a symlinked unit file),
+#   2. boots a scratch radiance container named `arbench` on the SAME two
+#      cards with the candidate engine argv (the env file the loop edits),
+#   3. polls /health until the listener is live (it binds AFTER the full
+#      load, so a bare curl is an honest gate),
+#   4. runs ONE self-contained completion request locally (127.0.0.1, no LAN
+#      jitter): a fixed ~140k-char real-text prompt (~32k tokens) decoded for
+#      256 tokens, greedy. The engine's `timings` object reports the honest
+#      prefill and decode rates for that single request.
+#   5. prints METRIC lines and tears the scratch down,
+#   6. restores production (systemctl start), via a trap so a failed bench
+#      never leaves frank-primary down (the stale-probe-tail lesson).
+#
+# Determinism: the prompt is built from a fixed, sorted corpus of repo files
+# with a fixed random seed, so prompt_n is stable across runs. Sampling is
+# greedy (temperature 0), so the decode length is fixed; the only residual
+# nondeterminism is the wht6 lossy TP all-reduce, which is a constant across
+# runs and cancels in A/B comparisons.
+#
+# The scratch mirrors the production container contract exactly (serve-frank-
+# primary.sh): --network host, --init, seccomp=unconfined (io_uring for the
+# n-gram table), memlock unbounded (the ~132 GiB pinned ledger), /dev/kfd +
+# /dev/dri, the .rad ro, the state dir rw, the tune cache mount. The ONLY
+# differences: the container name (arbench, so prod's identity is untouched)
+# and a FRESH prefix-cache disk tier (arbench/, wiped per run) so the cold
+# prefill is cold — the prod tier at kvcache/frank-primary is left intact.
+set -uo pipefail
+
+ENV_FILE="${1:-/home/dank/serverAppConfigs/scripts/frank-primary.env}"
+SCRATCH=arbench
+PORT=7770
+HEALTH="http://127.0.0.1:$PORT/health"
+COMPLETE="http://127.0.0.1:$PORT/v1/completions"
+BODY=/tmp/arbench-body.json
+LOG=/home/dank/logs/arbench.log
+
+restore() {
+  podman rm -f "$SCRATCH" >/dev/null 2>&1 || true
+  sudo systemctl start frank-primary >/dev/null 2>&1 || true
+}
+trap restore EXIT
+
+# --- source the candidate config (same file the production unit reads) ------
+if [ ! -f "$ENV_FILE" ]; then
+  echo "no env file at $ENV_FILE" >&2; exit 1
+fi
+# shellcheck disable=SC1090
+set -a; . "$ENV_FILE"; set +a
+
+# --- stop production; the scratch needs both cards to itself ----------------
+sudo systemctl stop frank-primary >/dev/null 2>&1 || true
+podman rm -f "$SCRATCH" >/dev/null 2>&1 || true
+
+# --- fresh prefix-cache tier for a cold prefill (prod's tier untouched) ------
+FRESH_KV="$FRANK_PRIMARY_STATE_DIR/kvcache/$SCRATCH"
+rm -rf "$FRESH_KV"; mkdir -p "$FRESH_KV"
+mkdir -p "$FRANK_PRIMARY_STATE_DIR/tune"
+
+# --- boot the scratch: the production run line, scratch name + fresh KV ------
+podman run -d --name "$SCRATCH" \
+  --network host \
+  --init \
+  --security-opt seccomp=unconfined \
+  --ulimit memlock=-1:-1 \
+  --device /dev/kfd --device /dev/dri \
+  -v "$FRANK_PRIMARY_MODEL_DIR":/models:ro \
+  -v "$FRANK_PRIMARY_STATE_DIR":/data \
+  -v "$FRANK_PRIMARY_STATE_DIR/tune":/opt/radiance/share/radiance/tune \
+  -v "$FRANK_PRIMARY_TEMPLATE":/templates/template.jinja:ro \
+  "$FRANK_PRIMARY_IMAGE" \
+  --model "/models/$FRANK_PRIMARY_MODEL_FILE" \
+  --host :: --port "$PORT" \
+  --served-model-name "$FRANK_PRIMARY_SERVED_NAME" \
+  --override-chat-template /templates/template.jinja \
+  --tp "$FRANK_PRIMARY_TP" --tp-wire "$FRANK_PRIMARY_TP_WIRE" \
+  --max-num-seqs "$FRANK_PRIMARY_MAX_SEQS" \
+  --max-model-len "$FRANK_PRIMARY_MAX_MODEL_LEN" \
+  --placement "$FRANK_PRIMARY_PLACEMENT" \
+  --host-pool-mib "$FRANK_PRIMARY_HOST_POOL_MIB" \
+  --gpu-headroom-mib "$FRANK_PRIMARY_GPU_HEADROOM_MIB" \
+  --expert-vs-cache-ratio "$FRANK_PRIMARY_EXPERT_VS_CACHE" \
+  --kv-cache-dtype "$FRANK_PRIMARY_KV_DTYPE" \
+  --prefix-cache-host-mib "$FRANK_PRIMARY_PREFIX_HOST_MIB" \
+  --prefix-cache-dir "/data/kvcache/$SCRATCH" \
+  --prefix-cache-disk-mib "$FRANK_PRIMARY_PREFIX_DISK_MIB" \
+  --ngram-placement "$FRANK_PRIMARY_NGRAM_PLACEMENT" \
+  --num-speculative-tokens "$FRANK_PRIMARY_SPEC_TOKENS" \
+  --max-num-batched-tokens "$FRANK_PRIMARY_MAX_BATCHED_TOKENS" \
+  ${FRANK_PRIMARY_EXTRA_ARGS:+$FRANK_PRIMARY_EXTRA_ARGS} \
+  > "$LOG" 2>&1
+if [ $? -ne 0 ]; then echo "scratch boot failed; see $LOG" >&2; exit 1; fi
+
+# --- poll health (listener binds after the full load; cold boot ~110 s) ------
+ok=0
+for _ in $(seq 1 360); do
+  code=$(curl -s -o /dev/null -w "%{http_code}" --max-time 3 "$HEALTH" 2>/dev/null || echo 000)
+  if [ "$code" = "200" ]; then ok=1; break; fi
+  sleep 2
+done
+if [ "$ok" -ne 1 ]; then
+  echo "scratch never reached health 200; tail of $LOG:" >&2
+  tail -30 "$LOG" >&2; exit 1
+fi
+
+# --- build the workload: a fixed ~140k-char real-text prompt (repo corpus) ---
+python3 - "$BODY" <<'PY'
+import json, sys, random, glob
+random.seed(1234)
+root = "/home/dank/serverAppConfigs"
+files = sorted(
+    glob.glob(f"{root}/serverAppConfigs-docs/**/*.md", recursive=True)
+    + glob.glob(f"{root}/scripts/*.sh")
+    + glob.glob(f"{root}/scripts/*.py")
+    + glob.glob(f"{root}/AGENTS.md")
+)
+corpus = []
+total = 0
+for f in files:
+    try:
+        t = open(f, encoding="utf-8", errors="ignore").read()
+    except Exception:
+        continue
+    if not t.strip():
+        continue
+    corpus.append(t)
+    total += len(t)
+    if total > 240000:
+        break
+corpus = "\n".join(corpus)
+TGT = 140000
+parts, n = [], 0
+while n < TGT:
+    start = random.randint(0, max(0, len(corpus) - 500))
+    seg = corpus[start:start + random.randint(200, 500)]
+    parts.append(seg)
+    n += len(seg) + 1
+prompt = " ".join(parts)
+json.dump(
+    {"model": "frank-primary", "prompt": prompt,
+     "max_tokens": 256, "temperature": 0.0, "stream": False,
+     "stop": ["<|im_end|>"]},
+    open(sys.argv[1], "w"),
+)
+PY
+if [ ! -f "$BODY" ]; then echo "prompt build failed" >&2; exit 1; fi
+
+# --- the bench request (prefill ~32k + decode 256, greedy) -------------------
+RESP=$(curl -s --max-time 600 -X POST "$COMPLETE" \
+  -H 'Content-Type: application/json' -d "@$BODY")
+echo "$RESP" > /tmp/arbench-resp.json
+
+# --- parse the timings into METRIC lines -------------------------------------
+python3 - <<'PY'
+import json, sys
+try:
+    r = json.load(open("/tmp/arbench-resp.json"))
+except Exception as e:
+    sys.stderr.write(f"unparseable response: {e}\n"); sys.exit(1)
+if "timings" not in r:
+    sys.stderr.write(f"no timings in response: {json.dumps(r)[:500]}\n"); sys.exit(1)
+t = r["timings"]
+dtps = t.get("predicted_per_second")
+ptps = t.get("prompt_per_second")
+pms  = t.get("prompt_ms")
+dn   = t.get("draft_n", 0)
+da   = t.get("draft_n_accepted", 0)
+if dtps is None:
+    sys.stderr.write("predicted_per_second null (no decode produced)\n"); sys.exit(1)
+print(f"METRIC decode_tps={dtps:.4f}")
+if ptps is not None:
+    print(f"METRIC prefill_tps={ptps:.4f}")
+if pms is not None:
+    print(f"METRIC ttft_ms={pms:.4f}")
+if dn:
+    print(f"METRIC draft_accept={da/dn:.4f}")
+print(f"INFO prompt_n={t.get('prompt_n')} cache_n={t.get('cache_n')} predicted_n={t.get('predicted_n')}")
+PY

scripts/build-frank-primary-rad.sh

@@ -0,0 +1,166 @@
+#!/usr/bin/env bash
+# build-frank-primary-rad.sh — build frank-primary's .rad container from the
+# OFFICIAL Qwen/Qwen3.8-Flash-Next-FP8 checkpoint, at NATIVE precision.
+#
+#   bash scripts/build-frank-primary-rad.sh                # convert
+#   bash scripts/build-frank-primary-rad.sh --plan-only    # declare, no I/O
+#   bash scripts/build-frank-primary-rad.sh --restage      # redo staging
+#   bash scripts/build-frank-primary-rad.sh --force        # rebuild existing
+#
+# WHY A STAGING DIR (the one non-obvious step): rad-convert selects its
+# architecture plugin by the checkpoint's quantisation DESCRIPTOR —
+# derived from config.json's quantization_config (core/format/checkpoint.cpp:
+# "fp8" here). No shipped plugin claims (qwen4exp, "fp8"): the shipped
+# qwen4exp plugin claims (qwen4exp, ""), the descriptor recipe-built
+# containers carry (they are built from the bf16 upstream, which has no
+# quantization_config at all). The match is exact with no fallback, so an
+# unmodified convert refuses before declare. The descriptor is only a
+# selection key; per-weight encodings are data — the checkpoint reader
+# derives them from the tensors themselves (X.weight_scale_inv is the
+# scale plane; checkpoint.h). Staging drops the quantization_config key
+# (everything else is a relative symlink into the pristine download) so
+# the convert selects the shipped plugin. EVERY weight is then copied through
+# in its own encoding — block-FP8 codes + bf16 scale planes stay block-FP8 —
+# except TWO value-preserving reshapes that stack-qwen38-official-fp8.py
+# performs on the staging dir first, and ONE recipe rule that finishes the
+# PLE table:
+#   * experts: the plugin maps the STACKED layout (mlp.experts.gate_up_proj
+#     [E,2F,H] with the _scale_inv plane at the same index) while the official
+#     repo ships per-expert unfused tensors — staging byte-stacks them
+#     (gate rows first, no requantisation).
+#   * PLE: the plugin's fp8 branch wants the table codes beside a ONE-ELEMENT
+#     bf16 scale plane; the official repo ships 128 SCALELESS fp8 shards plus
+#     one bf16 scalar. Staging decodes to bf16 with the checkpoint's own
+#     scale, and the convert's rtn rule re-encodes at that same scale —
+#     every original code is restored exactly and the scale plane lands in
+#     the form the plugin declares.
+#
+# WHY A CONTAINER AT ALL: a checkpoint dir has no mapped/file tiers, and a
+# weight only a host kernel reads (the n-gram / PLE table, 51B params)
+# cannot be served from one (GUIDE §3.4). rad-convert also bakes the
+# tokenizer and the checkpoint's chat template; the runtime replaces the
+# template with froggeric's (--override-chat-template).
+#
+# The convert runs rad-convert inside the digest-pinned radiance image
+# (the host has no ROCm and needs none): /home/dank/models is mounted at
+# /models so the staging dir's RELATIVE symlinks resolve inside the
+# container. Output lands beside the raw download, owned by dank
+# (rootless podman maps container-root to the invoking user).
+set -euo pipefail
+
+SRC="/home/dank/models/Qwen/Qwen3.8-Flash-Next-FP8"
+STAGE="/home/dank/models/stage-qwen4exp-fp8"
+OUT_DIR="/home/dank/models/rad"
+OUT="$OUT_DIR/qwen4exp-fp8-native.rad"
+
+SCRIPT_DIR="$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)"
+# shellcheck disable=SC1091
+source "$SCRIPT_DIR/frank-primary.env"
+IMAGE="$FRANK_PRIMARY_IMAGE"
+
+plan_only=0 restage=0 force=0
+for a in "$@"; do
+    case "$a" in
+        --plan-only) plan_only=1 ;;
+        --restage)   restage=1 ;;
+        --force)     force=1 ;;
+        *) echo "unknown flag: $a" >&2; exit 2 ;;
+    esac
+done
+
+if [ ! -f "$SRC/config.json" ]; then
+    echo "ERROR: raw checkpoint missing: $SRC (run scripts/download-frank-primary.sh --fetch)" >&2
+    exit 1
+fi
+echo "verifying raw checkpoint first..."
+bash "$SCRIPT_DIR/download-frank-primary.sh"
+
+if [ -f "$OUT" ] && [ "$plan_only" -eq 0 ] && [ "$force" -eq 0 ]; then
+    echo "container already exists: $OUT"
+    echo "rebuild it with --force (the old file moves to ~/.AGENTTRASH)"
+    exit 0
+fi
+
+# ---- staging ----------------------------------------------------------------
+if [ "$restage" -eq 1 ] && [ -e "$STAGE" ]; then
+    TRASH="$HOME/.AGENTTRASH/$(date +%Y%m%d)-frank-primary-stage"
+    mkdir -p "$TRASH"
+    mv "$STAGE" "$TRASH/stage-$(date +%H%M%S)"
+    echo "old staging dir moved to $TRASH"
+fi
+if [ ! -f "$STAGE/config.json" ]; then
+    mkdir -p "$STAGE"
+    python3 - "$SRC/config.json" "$STAGE/config.json" <<'PY'
+import json, sys
+src, dst = sys.argv[1], sys.argv[2]
+cfg = json.load(open(src))
+dropped = cfg.pop("quantization_config", None)
+json.dump(cfg, open(dst, "w"), indent=2)
+print(f"staging config.json written (quantization_config dropped: {dropped is not None})")
+PY
+    # The index is COPIED like config.json, never symlinked: the reshape step
+    # rewrites the stage's index, and a symlink would write through to the
+    # raw checkpoint (measured 2026-10-05: the raw index was restamped with
+    # the stacked layout and the raw verify failed 143/144). Shards stay
+    # relative symlinks — relative so the targets resolve inside the convert
+    # container (the stage's parent is mounted at /models).
+    for f in "$SRC"/*; do
+        name="$(basename "$f")"
+        case "$name" in config.json|model.safetensors.index.json) continue ;; esac
+        ln -sfn "../Qwen/Qwen3.8-Flash-Next-FP8/$name" "$STAGE/$name"
+    done
+    cp "$SRC/model.safetensors.index.json" "$STAGE/"
+    echo "staging dir ready: $STAGE ($(ls "$STAGE" | wc -l) entries)"
+fi
+
+# ---- checkpoint reshape (stacked experts + bf16 PLE table) -------------------
+# stack-qwen38-official-fp8.py reshapes the staged checkpoint in place
+# (idempotent) and writes the PLE scale to $STAGE/ple_scale.txt — see the
+# script's docstring for why each transform is value-preserving.
+echo "stacking experts + merging the PLE table (value-preserving)..."
+python3 "$SCRIPT_DIR/stack-qwen38-official-fp8.py" --stage "$STAGE"
+PLE_SCALE="$(tr -d '\n' < "$STAGE/ple_scale.txt")"
+echo "PLE fixed scale: $PLE_SCALE"
+
+# ---- convert ----------------------------------------------------------------
+# rad-convert needs no GPU: declare resolves kernels and asks each about the
+# encoding it will read — CPU-side layout questions only. (If a plugin load
+# ever demands a card, add --device /dev/kfd --device /dev/dri to this run.)
+# --plan-only runs the same declare phase and exits before a byte is written.
+echo "rad-convert: $SRC (staged) -> $OUT"
+mkdir -p "$OUT_DIR"
+CONVERT_ARGS=(/models/stage-qwen4exp-fp8 -o /models/rad/qwen4exp-fp8-native.rad
+              --home /opt/radiance/share/radiance)
+[ "$plan_only" -eq 1 ] && CONVERT_ARGS+=(--plan-only -v)
+# THE ONE RECIPE RULE (see header): the PLE table arrives from staging as a
+# bf16 merge of the checkpoint's own fp8 codes decoded with the checkpoint's
+# own scale; rtn re-encodes it at that same scale — the codes come back
+# exactly and the plugin's one-element scale plane is produced.
+CONVERT_ARGS+=(--quant "blk.*.ple_ngram.weight=rtn:codes=fp8_e4m3,block=*x*,scale=bf16,rule=fixed,scale_value=$PLE_SCALE")
+[ "$force" -eq 1 ] && [ -f "$OUT" ] && {
+    TRASH="$HOME/.AGENTTRASH/$(date +%Y%m%d)-frank-primary-rad"
+    mkdir -p "$TRASH"
+    mv "$OUT" "$TRASH/qwen4exp-fp8-native.rad.$(date +%H%M%S)"
+    echo "previous container moved to $TRASH"
+}
+podman run --rm \
+    --entrypoint rad-convert \
+    -v /home/dank/models:/models \
+    "$IMAGE" "${CONVERT_ARGS[@]}"
+
+if [ "$plan_only" -eq 1 ]; then
+    echo "plan-only run complete (no container written)"
+    exit 0
+fi
+
+# ---- record -----------------------------------------------------------------
+echo "rad-info summary:"
+podman run --rm --entrypoint rad-info -v /home/dank/models:/models:ro \
+    "$IMAGE" "/models/rad/qwen4exp-fp8-native.rad" | head -20
+MIB=$(( $(stat -c %s "$OUT") / 1048576 ))
+echo "container: $OUT (${MIB} MiB; env floor $FRANK_PRIMARY_RAD_MIN_MIB MiB)"
+if [ "$MIB" -lt "$FRANK_PRIMARY_RAD_MIN_MIB" ]; then
+    echo "ERROR: container below the env floor — the convert was truncated" >&2
+    exit 1
+fi
+echo "build complete"

scripts/deploy-plan.py

@@ -62,7 +62,7 @@ SERVICE_ORDER = (
     "lidarr",
     "readarr",
     "frank-large",
-    "frank-small",
+    "frank-secondary",
     "frank-primary",
 )
 
@@ -94,7 +94,7 @@ DANKCLAW_SERVICES = {
     "caddy",
     "homepage",
     "frank-large",
-    "frank-small",
+    "frank-secondary",
 }
 
 DANKSERVER_SERVICES = {
@@ -400,12 +400,12 @@ def services_for_path(path: str, target: str) -> set[str]:
             # change to it redeploying alertmanager is the only path that
             # reaches it on the host.
             _add(services, "alertmanager")
-        if name in {"frank-small.env", "frank-small.service", "serve-frank-small.sh"}:
-            # llama.cpp frank-small (2026-09-16) keeps its unit, env and serve
-            # files here; each one selects frank-small only. The ik_llama.cpp
+        if name in {"frank-secondary.env", "frank-secondary.service", "serve-frank-secondary.sh"}:
+            # llama.cpp frank-secondary (2026-09-16) keeps its unit, env and serve
+            # files here; each one selects frank-secondary only. The ik_llama.cpp
             # trio that lived here for two days is in scripts/.AGENTTRASH/,
             # which is archive material and plans nothing.
-            _add(services, "frank-small")
+            _add(services, "frank-secondary")
         if name in {"frank-large.env", "frank-large.service", "serve-frank-large.sh"}:
             # CIRU v4.4.1 frank-large (2026-09-19) keeps its unit, env and
             # serve files here the same way: a root unit on a host-built

scripts/deploy-remote.sh

@@ -90,7 +90,7 @@ service_is_allowed() {
         dankclaw:authentik-postgres|dankclaw:authentik-server|\
         dankclaw:authentik-worker|dankclaw:authentik-outpost|dankclaw:authentik-outpost-ldap|dankclaw:anubis|\
         dankclaw:caddy|dankclaw:homepage|dankclaw:seerr|dankclaw:marreta|\
-        dankclaw:frank-large|dankclaw:frank-small|\
+        dankclaw:frank-large|dankclaw:frank-secondary|\
         dankserver:romm-mariadb|dankserver:romm-redis|dankserver:romm|\
         dankserver:copyparty|dankserver:jellyfin|dankserver:calibre-web|dankserver:calibre-server|dankserver:plex|dankserver:tautulli|\
         dankserver:prowlarr|dankserver:qbittorrent|dankserver:radarr|\
@@ -1086,33 +1086,37 @@ deploy_anubis() {
     wait_container_healthy anubis 60 || return 1
 }
 
-# frank-large is the halogen CONTAINER (ghcr.io/peonist-ai/halogen,
-# digest-pinned through scripts/frank-large.env) serving
-# peonist-ai/halogen-qwen3.8-27b — qwen3.8-27b-p1w4d-d2.hgn, 33.4 GiB,
-# DFlash2 spec decode, 1 slot x 200700, text-only — on the Strix Halo
-# iGPU. It stays a ROOT unit because the serve script drives `podman run`
-# directly (the frank-primary pattern): relink_unit points
-# /etc/systemd/system/frank-large.service at the repo file, boot
+# frank-large is the halogen-flash-server CONTAINER (0.16.2, digest-pinned
+# through scripts/frank-large.env) serving the Quat3rnion ABLITERATED
+# Swift1.5 finetune of peonist's Qwen3.8-Flash-Next v2 checkpoint —
+# qwen38-flash-next-v2-swift15-abliterated.hgn (66,687,678,464 B, ~62 GiB
+# resident) + the REQUIRED n-gram lookup table + the vision tower —
+# MTP spec decode (HALOGEN_SPEC_ADAPT=0), 2 slots x 200192, image input —
+# on the Strix Halo iGPU. It stays a ROOT unit because the serve script
+# drives `podman run` directly (the frank-primary pattern): relink_unit
+# points /etc/systemd/system/frank-large.service at the repo file, boot
 # persistence is that unit's [Install], and LimitMEMLOCK=infinity is
-# fleet parity. 2026-10-05 cutover from the ROCmFPX llama.cpp shape —
-# rollback = revert the swap commit's trio; that payload, its draft head
-# and the fork build all stay on disk
-# (~/models/kingjones-Qwen3.8-27B-Uncensored-ROCmFP4-STRIX-MTP,
-# ~/GitPackages/kingjones-ROCmFPX).
+# fleet parity. 2026-10-06 cutover from the halogen 27B-dense shape —
+# rollback = revert the swap commit's trio; the 27B payload stays on
+# disk (~/models/halogen-qwen3.8-27b).
 #
 # The gates mirror serve-frank-large.sh's predicates EXACTLY (the
 # 2026-10-01 crash-loop lesson): image digest in dank's podman storage,
-# the .hgn checkpoint at the exact byte count, the tokenizer directory
-# holding every named file as a regular (not symlinked) file — a green
-# deploy against a missing artifact would
-# leave a restart-looping unit and a dark endpoint. The port gate is the
-# same as every cutover since 2026-09-16: nothing starts until 7770 is
-# released.
-LLM_LARGE_IMAGE="${LLM_LARGE_IMAGE:-ghcr.io/peonist-ai/halogen@sha256:dc0a39a0016d6cfc58a197978febaafdf8d28403f724ded6111d98b5fb7ac0ea}"
-LLM_LARGE_CKPT="${LLM_LARGE_CKPT:-/home/dank/models/halogen-qwen3.8-27b/qwen3.8-27b-p1w4d-d2.hgn}"
-LLM_LARGE_CKPT_BYTES="${LLM_LARGE_CKPT_BYTES:-35865565184}"
-LLM_LARGE_TOK_DIR="${LLM_LARGE_TOK_DIR:-/home/dank/models/halogen-qwen3.8-27b/tokenizer}"
-LLM_LARGE_TOK_FILES="${LLM_LARGE_TOK_FILES:-tokenizer.json chat_template.jinja tokenizer_config.json merges.txt vocab.json}"
+# each .hgn payload (checkpoint, n-gram table, vision tower) at the exact
+# byte count, the tokenizer directory holding every named file as a
+# regular (not symlinked) file — a green deploy against a missing
+# artifact would leave a restart-looping unit and a dark endpoint. The
+# port gate is the same as every cutover since 2026-09-16: nothing
+# starts until 7770 is released.
+LLM_LARGE_IMAGE="${LLM_LARGE_IMAGE:-ghcr.io/peonist-ai/halogen-flash-server@sha256:0c61bf84ac22308a53f5d1ca6b86806702d7039e5ebc51cae4c66621b92fe04a}"
+LLM_LARGE_CKPT="${LLM_LARGE_CKPT:-/home/dank/models/halogen-flashnext-v2/qwen38-flash-next-v2-swift15-abliterated.hgn}"
+LLM_LARGE_CKPT_BYTES="${LLM_LARGE_CKPT_BYTES:-66687678464}"
+LLM_LARGE_NGRAM="${LLM_LARGE_NGRAM:-/home/dank/models/halogen-flashnext-v2/qwen38-flash-next-ngram.hgn}"
+LLM_LARGE_NGRAM_BYTES="${LLM_LARGE_NGRAM_BYTES:-51200246144}"
+LLM_LARGE_VISION="${LLM_LARGE_VISION:-/home/dank/models/halogen-flashnext-v2/qwen38-flash-next-vision.hgn}"
+LLM_LARGE_VISION_BYTES="${LLM_LARGE_VISION_BYTES:-897916416}"
+LLM_LARGE_TOK_DIR="${LLM_LARGE_TOK_DIR:-/home/dank/models/halogen-flashnext-v2/tokenizer}"
+LLM_LARGE_TOK_FILES="${LLM_LARGE_TOK_FILES:-tokenizer.json chat_template.jinja tokenizer_config.json merges.txt vocab.json generation_config.json}"
 
 retire_halogen_large_quadlet() {
     local trash="$HOME/.AGENTTRASH/20260919-orca-cutover/quadlet"
@@ -1156,23 +1160,37 @@ deploy_llm_large() {
     fi
     if [[ ! -f "$LLM_LARGE_CKPT" ]]; then
         echo "ERROR: halogen checkpoint absent at $LLM_LARGE_CKPT" >&2
-        echo "frank-large: fetch it: bash scripts/download-frank-large-halogen.sh" >&2
+        echo "frank-large: fetch it: bash scripts/download-frank-large-flashnext.sh" >&2
         return 1
     fi
     if [[ "$(stat -c %s "$LLM_LARGE_CKPT")" != "$LLM_LARGE_CKPT_BYTES" ]]; then
         echo "ERROR: halogen checkpoint is not $LLM_LARGE_CKPT_BYTES bytes (partial download?)" >&2
-        echo "frank-large: re-check: bash scripts/download-frank-large-halogen.sh --verify-only" >&2
+        echo "frank-large: re-check: bash scripts/download-frank-large-flashnext.sh --verify-only" >&2
         return 1
     fi
+    for gate in "$LLM_LARGE_NGRAM:$LLM_LARGE_NGRAM_BYTES:n-gram table" \
+                "$LLM_LARGE_VISION:$LLM_LARGE_VISION_BYTES:vision tower"; do
+        IFS=: read -r gpath gbytes gname <<<"$gate"
+        if [[ ! -f "$gpath" ]]; then
+            echo "ERROR: halogen $gname absent at $gpath" >&2
+            echo "frank-large: fetch it: bash scripts/download-frank-large-flashnext.sh" >&2
+            return 1
+        fi
+        if [[ "$(stat -c %s "$gpath")" != "$gbytes" ]]; then
+            echo "ERROR: halogen $gname is not $gbytes bytes (partial download?)" >&2
+            echo "frank-large: re-check: bash scripts/download-frank-large-flashnext.sh --verify-only" >&2
+            return 1
+        fi
+    done
     if [[ -L "$LLM_LARGE_TOK_DIR" || ! -d "$LLM_LARGE_TOK_DIR" ]]; then
         echo "ERROR: tokenizer dir missing or a symlink at $LLM_LARGE_TOK_DIR" >&2
-        echo "frank-large: fetch it: bash scripts/download-frank-large-halogen.sh" >&2
+        echo "frank-large: fetch it: bash scripts/download-frank-large-flashnext.sh" >&2
         return 1
     fi
     for tf in $LLM_LARGE_TOK_FILES; do
         if [[ ! -f "$LLM_LARGE_TOK_DIR/$tf" || -L "$LLM_LARGE_TOK_DIR/$tf" ]]; then
             echo "ERROR: tokenizer file missing or a symlink: $LLM_LARGE_TOK_DIR/$tf" >&2
-            echo "frank-large: fetch it: bash scripts/download-frank-large-halogen.sh" >&2
+            echo "frank-large: fetch it: bash scripts/download-frank-large-flashnext.sh" >&2
             return 1
         fi
     done
@@ -1184,7 +1202,7 @@ deploy_llm_large() {
     # `podman run --name frank-large-halogen` / `podman wait` but NOT in
     # this deployer's own command line (`bash deploy-remote.sh
     # frank-large`) — so it cannot shoot itself in the foot (the fleet
-    # pkill law), and it can never match frank-small's llama-server.
+    # pkill law), and it can never match frank-secondary's llama-server.
     run_privileged systemctl stop frank-large.service 2>/dev/null || true
     pkill -f 'frank-large-halogen' 2>/dev/null || true
     podman rm -f frank-large frank-large-halogen >/dev/null 2>&1 || true
@@ -1198,119 +1216,115 @@ deploy_llm_large() {
     wait_for_health frank-large http://127.0.0.1:7770/health 1200
 }
 
-# deploy_llm_small — frank-small is llama.cpp (master build, CUDA sm_86) on
+# deploy_llm_secondary — frank-secondary is llama.cpp (master build, CUDA sm_86) on
 # the RTX 3090 as a ROOT unit: the runtime is a native binary tree under
 # ~/GitPackages/llama-master/build-cuda, so there is no container image and
 # no generated unit to avoid. relink_unit points
-# /etc/systemd/system/frank-small.service at the repo file and boot
+# /etc/systemd/system/frank-secondary.service at the repo file and boot
 # persistence is that unit's [Install] section.
 #
 # It is a SEPARATE endpoint on port 7771; frank-large keeps 7770. The 3090 is
 # not shared: frank-large's engine runs on the gfx1151 iGPU.
-LLM_SMALL_MODEL_FILE="${LLM_SMALL_MODEL_FILE:-/home/dank/models/Ornith-Q8/Ornith-1.5-9B-Q8_0.gguf}"
-LLM_SMALL_RUNTIME="${LLM_SMALL_RUNTIME:-/home/dank/GitPackages/llama-master/build-cuda/bin/llama-server}"
-# The projector IS the vision capability here: the Ornith GGUF ships no
-# `v.*` tensors, so a deploy that starts without this file silently ships a
-# text-only endpoint. Checked unconditionally, like frank-large's.
-LLM_SMALL_MMPROJ="${LLM_SMALL_MMPROJ:-/home/dank/models/Ornith-Q8/mmproj-Ornith-1.5-9B-BF16.gguf}"
+LLM_SECONDARY_MODEL_FILE="${LLM_SECONDARY_MODEL_FILE:-/home/dank/models/Swift15-UncIQ3XXS/Swift-1.5-Qwen3.8-27B-Uncensored-Dynamic-MTP-RCO-IQ3_XXS.gguf}"
+LLM_SECONDARY_RUNTIME="${LLM_SECONDARY_RUNTIME:-/home/dank/GitPackages/llama-master/build-cuda/bin/llama-server}"
+# The projector IS the vision capability here: the GGUF carries no vision
+# tensors, and the publisher ships mmproj-BF16.gguf in the same repo, so a
+# deploy that starts without this file silently ships a text-only
+# endpoint. Checked unconditionally, like frank-large's.
+LLM_SECONDARY_MMPROJ="${LLM_SECONDARY_MMPROJ:-/home/dank/models/Swift15-UncIQ3XXS/mmproj-BF16.gguf}"
 
-deploy_llm_small() {
+deploy_llm_secondary() {
     # Refuse rather than restart-loop: a unit whose runtime or payload is
     # missing never becomes healthy, and a green deploy would leave the
     # endpoint dark with nothing in the log to say why.
-    if [[ ! -x "$LLM_SMALL_RUNTIME" ]]; then
-        echo "ERROR: llama.cpp runtime absent at $LLM_SMALL_RUNTIME" >&2
-        echo "frank-small: build it (cmake -B build-cuda -DGGML_CUDA=ON" >&2
-        echo "frank-small: -DCMAKE_CUDA_ARCHITECTURES=86 -DCMAKE_CUDA_HOST_COMPILER=/usr/bin/gcc-13" >&2
-        echo "frank-small: && cmake --build build-cuda --target llama-server) first." >&2
+    if [[ ! -x "$LLM_SECONDARY_RUNTIME" ]]; then
+        echo "ERROR: llama.cpp runtime absent at $LLM_SECONDARY_RUNTIME" >&2
+        echo "frank-secondary: build it (cmake -B build-cuda -DGGML_CUDA=ON" >&2
+        echo "frank-secondary: -DCMAKE_CUDA_ARCHITECTURES=86 -DCMAKE_CUDA_HOST_COMPILER=/usr/bin/gcc-13" >&2
+        echo "frank-secondary: && cmake --build build-cuda --target llama-server) first." >&2
         return 1
     fi
-    for f in "$LLM_SMALL_MODEL_FILE" "$LLM_SMALL_MMPROJ"; do
+    for f in "$LLM_SECONDARY_MODEL_FILE" "$LLM_SECONDARY_MMPROJ"; do
         if [[ ! -f "$f" ]]; then
-            echo "ERROR: frank-small payload file absent at $f" >&2
-            echo "frank-small: run scripts/download-frank-small.sh first;" >&2
-            echo "frank-small: refusing to start a unit that cannot load." >&2
+            echo "ERROR: frank-secondary payload file absent at $f" >&2
+            echo "frank-secondary: run scripts/download-frank-secondary.sh first;" >&2
+            echo "frank-secondary: refusing to start a unit that cannot load." >&2
             return 1
         fi
     done
-    relink_unit frank-small
+    relink_unit frank-secondary
     run_privileged systemctl daemon-reload
-    run_privileged systemctl enable frank-small.service
-    run_privileged systemctl restart frank-small.service
-    wait_for_health frank-small http://127.0.0.1:7771/health 900 || return 1
+    run_privileged systemctl enable frank-secondary.service
+    run_privileged systemctl restart frank-secondary.service
+    wait_for_health frank-secondary http://127.0.0.1:7771/health 900 || return 1
     # False-green hardening (frank-large's lesson): the port answering
     # proves nothing about WHO answers — the unit must be active with a
     # live MainPID, or a stray SGLang holdover is masquerading as deployed.
     local main_pid
-    main_pid="$(run_privileged systemctl show -p MainPID --value frank-small.service)"
+    main_pid="$(run_privileged systemctl show -p MainPID --value frank-secondary.service)"
     if [[ -z "$main_pid" || "$main_pid" == "0" ]]; then
-        echo "FAIL: :7771 answered but frank-small.service has no MainPID — a stray process holds the port" >&2
+        echo "FAIL: :7771 answered but frank-secondary.service has no MainPID — a stray process holds the port" >&2
         return 1
     fi
-    echo "frank-small: health answered by unit MainPID $main_pid"
+    echo "frank-secondary: health answered by unit MainPID $main_pid"
 }
 
-# deploy_llm_primary — frank-primary is the tcclaviger RDNA4 vLLM fork
-# (digest-pinned image, gfx1201) serving the orcarouter
-# Qwen3.8-Flash-Next-Uncensored-FP8 payload (131 safetensors shards) with
-# MTP4 spec decode and q8/q8 KV on the two R9700s as a ROOT unit whose serve
-# script drives `podman run` directly (the image entrypoint execs
-# `vllm serve`). 2026-09-29 cutover from the R9V radiance fork, whose
-# launcher/shim/setup-state machinery is retired (state dirs stay on disk as
-# rollback; pre-cutover tag frank-primary-r9v-q8-20260929). Same port and
-# alias as the endpoints it replaced; the harness models.yml id is unchanged.
+# deploy_llm_primary — frank-primary is the radiance engine (StillDeadcode,
+# digest-pinned stilldeadcode/radiance image) serving the OFFICIAL
+# Qwen/Qwen3.8-Flash-Next-FP8 at native block-FP8 precision from the
+# rad-convert container qwen4exp-fp8-native.rad, on BOTH R9700s (TP=2) as a
+# ROOT unit whose serve script drives `podman run` directly (the engine is
+# the image entrypoint; the image lives in dank's rootless podman storage).
+# 2026-10-05 cutover from FreeToken (pre-cutover tag
+# frank-primary-freetoken-pr587-20261005); before that, the tcclaviger vLLM
+# fork (tag frank-primary-vllm-shapeG-20261003). Same port and alias; the
+# harness models.yml id is unchanged.
 #
-# Gates: the pinned runtime image must be loadable from dank's rootless
-# storage by its OCI manifest digest (the env file's FRANK_PRIMARY_IMAGE),
-# the payload dir must carry config.json and the expected shard count
-# (FRANK_PRIMARY_MODEL_SHARDS), and the unit env must be complete. Per-file
-# sha256 verification was the fetch script's job at download time, never
+# Gates: the image must exist under its digest ref in dank's rootless
+# storage (FRANK_PRIMARY_IMAGE), the .rad container file must exist at the
+# env's size floor (FRANK_PRIMARY_MODEL_FILE >= FRANK_PRIMARY_RAD_MIN_MIB —
+# a truncated convert is a guaranteed long-timeout boot), and the unit env
+# must be complete. Per-file sha256 verification is the fetch script's job
+# at download time and rad-convert/rad-info's job at build time, never
 # re-hashed here.
-LLM_PRIMARY_CONTAINER=frank-primary-qwen
-LLM_PRIMARY_LEGACY_CONTAINERS="frank-primary-r9v qwen-fp8 glm-probe"
+LLM_PRIMARY_CONTAINER=frank-primary
+LLM_PRIMARY_LEGACY_CONTAINERS="frank-primary-qwen frank-primary-r9v qwen-fp8 glm-probe"
 
 deploy_llm_primary() {
     # Refuse rather than restart-loop: a unit whose image or payload is
     # missing never becomes healthy, and a green deploy would leave the
     # endpoint dark with nothing in the log to say why.
     local env_file="/home/dank/serverAppConfigs/scripts/frank-primary.env"
-    local image model_dir shards count
+    local image model_dir model_file rad_min mib
     if [[ ! -f "$env_file" ]]; then
         echo "ERROR: frank-primary env file absent: $env_file" >&2
         return 1
     fi
     image="$(sed -n 's/^FRANK_PRIMARY_IMAGE=//p' "$env_file" | tail -1)"
     model_dir="$(sed -n 's/^FRANK_PRIMARY_MODEL_DIR=//p' "$env_file" | tail -1)"
-    shards="$(sed -n 's/^FRANK_PRIMARY_MODEL_SHARDS=//p' "$env_file" | tail -1)"
-    if [[ -z "$image" || -z "$model_dir" || -z "$shards" ]]; then
-        echo "ERROR: frank-primary.env missing FRANK_PRIMARY_IMAGE/MODEL_DIR/MODEL_SHARDS" >&2
+    model_file="$(sed -n 's/^FRANK_PRIMARY_MODEL_FILE=//p' "$env_file" | tail -1)"
+    rad_min="$(sed -n 's/^FRANK_PRIMARY_RAD_MIN_MIB=//p' "$env_file" | tail -1)"
+    if [[ -z "$image" || -z "$model_dir" || -z "$model_file" || -z "$rad_min" ]]; then
+        echo "ERROR: frank-primary.env missing FRANK_PRIMARY_IMAGE/MODEL_DIR/MODEL_FILE/RAD_MIN_MIB" >&2
         return 1
     fi
-    # Capture podman's output BEFORE grepping: under set -o pipefail, the
-    # classic `podman images | grep -q` races (grep exits on match, podman
-    # dies SIGPIPE 141 writing the rest, pipefail calls it no-match — the
-    # 2026-09-30 deploy refusal).
-    local digests
-    digests="$(podman images --format '{{.RepoDigests}}')"
-    if ! grep -qF "${image#*@}" <<<"$digests"; then
-        echo "ERROR: pinned runtime image absent from rootless storage: $image" >&2
-        echo "frank-primary: fetch it (podman pull $image)" >&2
+    # The image gate is the digest ref itself: a name@sha256 ref exists in
+    # storage only if THAT manifest was pulled. One command — no
+    # pipefail/grep race (the 2026-09-30 lesson).
+    if ! podman image exists "$image"; then
+        echo "ERROR: runtime image absent from storage: $image" >&2
+        echo "frank-primary: podman pull docker://stilldeadcode/radiance:latest" >&2
         return 1
     fi
-    if [[ ! -f "$model_dir/config.json" ]]; then
-        echo "ERROR: payload config missing: $model_dir/config.json" >&2
-        echo "frank-primary: re-run scripts/download-frank-primary.sh" >&2
+    if [[ ! -f "$model_dir/$model_file" ]]; then
+        echo "ERROR: .rad container missing: $model_dir/$model_file" >&2
+        echo "frank-primary: run scripts/build-frank-primary-rad.sh" >&2
         return 1
     fi
-    if [[ ! -f "$model_dir/model.safetensors.index.json" ]]; then
-        echo "ERROR: weight-map index missing: $model_dir/model.safetensors.index.json" >&2
-        echo "frank-primary: re-run scripts/download-frank-primary.sh" >&2
-        return 1
-    fi
-    count="$(find "$model_dir" -maxdepth 1 -name 'model-*-of-*.safetensors' 2>/dev/null | wc -l)"
-    if [[ "$count" != "$shards" ]]; then
-        echo "ERROR: payload shard count wrong: $model_dir (found $count, want $shards)" >&2
-        echo "frank-primary: re-run scripts/download-frank-primary.sh" >&2
+    mib=$(( $(stat -c %s "$model_dir/$model_file") / 1048576 ))
+    if [[ "$mib" -lt "$rad_min" ]]; then
+        echo "ERROR: .rad container too small: $model_dir/$model_file is ${mib} MiB, want >= $rad_min" >&2
+        echo "frank-primary: rebuild it (scripts/build-frank-primary-rad.sh --force)" >&2
         return 1
     fi
 
@@ -1335,9 +1349,26 @@ deploy_llm_primary() {
     run_privileged systemctl daemon-reload
     run_privileged systemctl enable frank-primary.service
     run_privileged systemctl restart frank-primary.service
-    # Cold start: FP8 page-in plus torch compile/graph capture (~15 min cold,
-    # measured 2026-09-29). Parity with every predecessor on this port.
-    wait_for_health frank-primary http://127.0.0.1:7770/health 1800 || return 1
+    # Cold start: .rad load + PLE pin + expert host-pool upload (the
+    # publisher's compose healthcheck allows a 15-minute start_period).
+    # Parity with every predecessor on this port. The HTTP listener binds
+    # AFTER the full load (main.cpp: engine.start() then engine.serve()),
+    # so during load the port REFUSES connections — a bare curl is honest
+    # here, unlike FreeToken's 200-while-loading body. (The MainPID check
+    # below still guards the stray-port case.)
+    local i ok=0
+    for ((i=1; i<=900; i++)); do
+        if curl -fsS http://127.0.0.1:7770/health 2>/dev/null >/dev/null; then
+            echo "frank-primary engine healthy after approximately $((i * 2))s"
+            ok=1
+            break
+        fi
+        sleep 2
+    done
+    if [[ "$ok" != 1 ]]; then
+        echo "FAIL: frank-primary /health never answered 200 in 1800s" >&2
+        return 1
+    fi
     # False-green hardening (the 2026-09-26 lesson): a bare curl against the
     # port cannot tell WHO answers — a stray probe holding :7770 satisfies
     # it while the unit is stopped. The deploy is only green if the unit
@@ -1536,7 +1567,7 @@ deploy_service() {
         seerr) deploy_seerr ;;
         marreta) deploy_marreta ;;
         frank-large) deploy_llm_large ;;
-        frank-small) deploy_llm_small ;;
+        frank-secondary) deploy_llm_secondary ;;
         frank-primary) deploy_llm_primary ;;
         hermes|hermes-dashboard|hermes-webui) deploy_hermes ;;
         romm|romm-mariadb|romm-redis|mariadb|redis) deploy_romm_stack "$service" ;;

scripts/deploy.sh

@@ -141,7 +141,7 @@ valid_service() {
         dankclaw:alertmanager|\
         dankclaw:node-exporter|dankclaw:authentik-postgres|dankclaw:authentik-server|\
         dankclaw:authentik-worker|dankclaw:authentik-outpost|dankclaw:authentik-outpost-ldap|dankclaw:anubis|dankclaw:seerr|\
-        dankclaw:frank-large|dankclaw:frank-small|\
+        dankclaw:frank-large|dankclaw:frank-secondary|\
         dankserver:romm-mariadb|dankserver:romm-redis|dankserver:romm|\
         dankserver:copyparty|dankserver:jellyfin|dankserver:calibre-web|dankserver:calibre-server|dankserver:plex|dankserver:tautulli|\
         dankserver:qbittorrent|dankserver:radarr|dankserver:sonarr|\

scripts/docker/frank-primary-freetoken.Dockerfile

@@ -1,62 +0,0 @@
-# frank-primary-freetoken.Dockerfile — FreeToken (PR #587 ROCm) serving image
-# for the frank-primary endpoint (dankserver :7770, 1x R9700 gfx1201).
-#
-# Built on dankserver with rootless podman (dank user), same storage the
-# service's `podman run` reads:
-#   podman build -t frank-primary-freetoken:pr587-77a7201 \
-#     -f scripts/docker/frank-primary-freetoken.Dockerfile scripts/docker/
-#
-# Base: the PR's own docs/install_amd.md names this exact image as the
-# known-good ROCm 7.14 stack for gfx1201 (torch 2.11.0+rocm7.14, triton-rocm
-# 3.7 — the original pin; FreeToken main+PR587 accepts torch>=2.11,<2.16 and
-# triton>=3.6,<3.9). There is no rocm7.14+torch2.14 image on Docker Hub (the
-# 2.14 images are rocm10.1.0 — an untested pair for this PR), so the PR doc's
-# verbatim recommendation is the deployed stack.
-#
-# FreeToken source: FlashML-org/FreeToken PR #587 at pinned head
-# 77a7201f98f00d93f618c6cf8b81ffe8360e0899 (branch rocm714_torch214, base
-# main 581fdca3). The PR is ROCm-gated end to end; the NVIDIA/CUDA path is
-# byte-identical to main.
-#
-# Build law from docs/install_amd.md:
-#   - `pip install --no-build-isolation --no-deps -e .` — the image's ROCm
-#     torch is preserved and used to compile the C++/HIP extensions.
-#   - NEVER a plain `pip install -e .` on ROCm: the dependency set is
-#     authored for NVIDIA (flashlib pulls numba + nvidia-cutlass-dsl; torch
-#     metadata pulls cuda-bindings) and the resolve can REPLACE the ROCm
-#     torch with a CUDA build.
-#   - Runtime deps installed explicitly, curated by the same doc. flashinfer
-#     / sgl_kernel / nvidia-cutlass-dsl have no ROCm builds and are
-#     intentionally absent — FreeToken gates them off and falls back to its
-#     Triton kernels on ROCm.
-#
-# Base is digest-pinned (2026-10-05): sha256:a223aee1… is the manifest this
-# build was qualified against; a re-pushed tag cannot change the stack.
-FROM rocm/pytorch@sha256:a223aee17aef5d21c3b9f63436dd19d27d1c665ec8b2f40011c9546cabae2a80
-
-# Pin the PR head, not the branch: a future push to rocm714_torch214 would
-# otherwise silently change the deployed engine.
-# The pip-ROCm torch in this image ships libhiprtc as the VERSIONED
-# libhiprtc.so.7 only; the PR's `_cpu_moe` extension links `-lhiprtc`, and
-# ld looks for the unversioned dev symlink in the ALREADY-LINKED
-# _rocm_sdk_core/lib dir. Create it there (first build failure 2026-10-05:
-# `ld: cannot find -lhiprtc`).
-RUN ln -s libhiprtc.so.7 /opt/venv/lib/python3.12/site-packages/_rocm_sdk_core/lib/libhiprtc.so
-RUN git clone https://github.com/FlashML-org/FreeToken.git /opt/FreeToken \
- && cd /opt/FreeToken \
- && git fetch origin pull/587/head \
- && git checkout 77a7201f98f00d93f618c6cf8b81ffe8360e0899
-
-WORKDIR /opt/FreeToken
-RUN python -m pip install --no-build-isolation --no-deps -e .
-
-# Curated runtime dependencies (docs/install_amd.md). transformers is pinned
-# to the pyproject window (>=5.16,<5.17); the rest are the doc's portable set.
-# torch / torchvision / triton-rocm come from the base image, 1:1 aligned.
-RUN python -m pip install \
-      "transformers>=5.16,<5.17" tokenizers safetensors huggingface_hub \
-      fastapi uvicorn pydantic openai httpx anyio \
-      einops gguf msgpack pyzmq typer rich apache-tvm-ffi \
-      partial-json-parser \
- && python -m pip install --no-deps flashlib==0.3.0 \
- && python -m pip install ninja

scripts/download-frank-large-flashnext.sh

@@ -0,0 +1,119 @@
+#!/usr/bin/env bash
+# download-frank-large-flashnext.sh — fetch frank-large's flash-server v2
+# payload: the Quat3rnion abliterated Swift1.5 checkpoint of peonist's v2
+# .hgn, the REQUIRED lookup table and the optional vision tower from
+# peonist-ai/halogen-qwen3.8-flash-next (both at the revisions the model
+# card pins), and the flat tokenizer the two repos share. All public, no
+# token — and per the fleet secret law, none would ever touch argv.
+#
+# The fetch law, same as every sibling script: curl -f (an access error
+# must not become a "successful" 153-byte artifact), exact byte count from
+# the Hub tree API, LFS sha256 verification, --verify-only to re-check an
+# existing payload without downloading.
+#
+# File manifest (verified against the Hub tree API on 2026-10-06):
+#   qwen38-flash-next-v2-swift15-abliterated.hgn  66,687,678,464 B  sha256
+#     9c880b8ba48deabfa06172eff1e7572e4fe465e09e97faec6e50c3f6fdcac037
+#     (Quat3rnion main; compacted r2 of the card's 89,775,868,032 B
+#     append-only original — byte-identical live tensors, dropped padding)
+#   qwen38-flash-next-ngram.hgn  51,200,246,144 B  sha256
+#     3450acada94e19aabad88bd49b45eb70304820949c2240fc178067b20ae5dffe
+#     (peonist @5cc17cea; REQUIRED — the engine reads it beside the
+#     checkpoint)
+#   qwen38-flash-next-vision.hgn  897,916,416 B  sha256
+#     d62e0ae553fe88afd3833733d4a4c669f34d20fd8dfce4b9610525bed2134b10
+#     (peonist @e053f488; optional, enables image input)
+#   tokenizer/* @e053f488 — byte-identical to the 27B-era tokenizer
+#     already on disk (tokenizer.json LFS
+#     0997f410c57a1f4e53b09e4be8f4a172d90edd9564368fb0847030937229b9f3);
+#     fetched fresh rather than symlinked, per the no-symlink law.
+set -euo pipefail
+
+DEST="${FRANK_LARGE_MODEL_DIR:-$HOME/models/halogen-flashnext-v2}"
+ABLIT="https://huggingface.co/Quat3rnion/halogen-swift1.5-qwen3.8-flash-next-v2-abliterated/resolve/main"
+PEONIST="https://huggingface.co/peonist-ai/halogen-qwen3.8-flash-next/resolve"
+NGRAM_REV="5cc17cea1a10b502b4a5db41d8d2b1a0f22dba26"
+TOK_REV="e053f488b120b99ed2525e6ac99f68c51b3b6179"
+
+FILES=(qwen38-flash-next-v2-swift15-abliterated.hgn
+       qwen38-flash-next-ngram.hgn
+       qwen38-flash-next-vision.hgn
+       tokenizer/tokenizer.json
+       tokenizer/chat_template.jinja
+       tokenizer/generation_config.json
+       tokenizer/merges.txt
+       tokenizer/vocab.json
+       tokenizer/tokenizer_config.json)
+SIZES=(66687678464
+       51200246144
+       897916416
+       12809320
+       8952
+       202
+       3353259
+       6722759
+       17928)
+SHAS=(9c880b8ba48deabfa06172eff1e7572e4fe465e09e97faec6e50c3f6fdcac037
+      3450acada94e19aabad88bd49b45eb70304820949c2240fc178067b20ae5dffe
+      d62e0ae553fe88afd3833733d4a4c669f34d20fd8dfce4b9610525bed2134b10
+      0997f410c57a1f4e53b09e4be8f4a172d90edd9564368fb0847030937229b9f3
+      - - - - - -)
+
+# Fetch source per file: repo base + revision (empty = ABLIT @main).
+SRCS=("$ABLIT"
+      "$PEONIST/$NGRAM_REV"
+      "$PEONIST/$TOK_REV"
+      "$PEONIST/$TOK_REV"
+      "$PEONIST/$TOK_REV"
+      "$PEONIST/$TOK_REV"
+      "$PEONIST/$TOK_REV"
+      "$PEONIST/$TOK_REV"
+      "$PEONIST/$TOK_REV")
+
+VERIFY_ONLY=0
+[ "${1:-}" = "--verify-only" ] && VERIFY_ONLY=1
+
+mkdir -p "$DEST"
+
+for i in "${!FILES[@]}"; do
+    f="${FILES[$i]}"; want="${SIZES[$i]}"; sha="${SHAS[$i]}"; base="${SRCS[$i]}"
+    path="$DEST/$f"
+    mkdir -p "$(dirname "$path")"
+    if [ "$VERIFY_ONLY" -eq 0 ]; then
+        # A file already at the published size skips the fetch entirely:
+        # resuming it would ask the CDN for a range past EOF (416 -> curl
+        # error under set -e), and re-pulling 62 GiB to prove a point is
+        # not what --verify-only is for. sha256 below still decides.
+        if [ -f "$path" ] && [ "$(stat -c %s "$path")" = "$want" ]; then
+            echo "HAVE  $f ($want bytes) — skipping fetch"
+        else
+            echo "FETCH $f -> $path"
+            # HF answers /resolve/ with a 302 to the Xet CDN and curl DROPS
+            # the Range header across redirects — a naive -C - then
+            # "resumes" by pulling the ~1 KB redirect body (measured
+            # 2026-10-05 on the 27B fetch: file frozen 14 MiB short, curl
+            # exit 0). Resolve the effective URL first, then resume
+            # against the CDN, which honors Range.
+            url=$(curl -sfIL -o /dev/null -w '%{url_effective}' "$base/$f") \
+                || { echo "cannot resolve $base/$f" >&2; exit 1; }
+            curl -fL --retry 5 -C - -o "$path" "$url"
+        fi
+    fi
+    [ -f "$path" ] || { echo "MISSING $path (run without --verify-only)" >&2; exit 1; }
+    got="$(stat -c %s "$path")"
+    [ "$got" = "$want" ] || { echo "SIZE $path: $got != $want" >&2; exit 1; }
+    echo "OK   $f ($got bytes)"
+done
+
+echo "byte counts match; verifying sha256 where the Hub publishes an LFS oid..."
+for i in "${!FILES[@]}"; do
+    f="${FILES[$i]}"; sha="${SHAS[$i]}"
+    if [ "$sha" = "-" ]; then
+        echo "OK   $f (plain git blob, size-checked)"
+        continue
+    fi
+    got="$(sha256sum "$DEST/$f" | cut -d' ' -f1)"
+    [ "$got" = "$sha" ] || { echo "SHA256 $f: $got != $sha" >&2; exit 1; }
+    echo "OK   $f sha256"
+done
+echo "DOWNLOAD_DONE $DEST"

scripts/download-frank-primary.sh

@@ -1,88 +1,111 @@
 #!/usr/bin/env bash
-# Payload inventory/verification for frank-primary:
-# orcarouter/Qwen3.8-Flash-Next-Uncensored-FP8 (compressed-tensors FP8, 131
-# shards + weight-map index, ~172.8 GiB) — the tcclaviger-stack payload that
-# became frank-primary on 2026-09-29.
+# download-frank-primary.sh — payload fetch/verify for frank-primary:
+# the OFFICIAL Qwen/Qwen3.8-Flash-Next-FP8 checkpoint (131 safetensors
+# shards + weight-map index + tokenizer + vision preprocessing configs,
+# ~172.8 GB). 2026-10-05: replaces the orcarouter uncensored payload of the
+# FreeToken era (that script lived here; its repo was gated and needed
+# HF_TOKEN — this one is UNGATED and needs no token at all).
 #
-#   bash scripts/download-frank-primary.sh --verify-only   (the ONLY mode)
-#   bash scripts/download-frank-primary.sh --fetch
+#   bash scripts/download-frank-primary.sh --fetch         resume/complete
+#   bash scripts/download-frank-primary.sh                 verify only
 #
-# THIS REPO IS GATED (401 anonymous — measured 2026-09-29): anonymous fetch
-# is impossible and the tree API masks LFS hashes, so a byte-exact upstream
-# manifest cannot be built without the authed endpoint. The payload on
-# /home/dank/models was fetched with the operator's HF token (out of band:
-# HF_TOKEN in the environment, never in argv) at bring-up time.
-#
-# Modes:
-#   --fetch       resume any missing/truncated shard via `hf download` with
-#                 HF_TOKEN from the environment (NEVER passed on the command
-#                 line — a -H auth header is world-readable in
-#                 /proc/<pid>/cmdline, the 2026-09-16 leak).
-#   --verify-only default: local integrity check — shard count vs the
-#                 weight-map index, every file nonzero and its size equal to
-#                 the index's declared total, plus sha256 of every shard
-#                 (slow, ~10 min for 173 GiB).
+# Fetch uses the vendored scripts/hfget.sh (StillDeadcode's Apache-2.0
+# listing/resume downloader, curl+jq only): per-file size checks against
+# the Hub listing, `-C -` resume, pickle-format duplicates skipped.
+# Verify (default) is the AGENTS.md law in three parts: every listed file
+# present with the listed size, the shard count matches the weight-map
+# index, and every shard's sha256 equals the Hub's LFS oid. Verify needs
+# only the public API.
 set -euo pipefail
 
-REPO="orcarouter/Qwen3.8-Flash-Next-Uncensored-FP8"
-DEST="/home/dank/models/Qwen3.8-Flash-Next-Uncensored-FP8"
+REPO="Qwen/Qwen3.8-Flash-Next-FP8"
+DEST="/home/dank/models/Qwen/Qwen3.8-Flash-Next-FP8"
 EXPECTED_SHARDS=131
 
 mode="verify"
 [[ "${1:-}" == "--fetch" ]] && mode="fetch"
+SCRIPT_DIR="$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)"
 
 if [[ "$mode" == "fetch" ]]; then
-    : "${HF_TOKEN:?--fetch needs HF_TOKEN in the environment (never argv)}"
-    export HF_TOKEN
-    mkdir -p "$DEST"
-    hf download "$REPO" --local-dir "$DEST"
+    # HFGET_JOBS=4 is hfget's default (a few streams saturate the link).
+    bash "$SCRIPT_DIR/hfget.sh" "$REPO" "$DEST" main
 fi
 
 echo "verifying local payload: $DEST"
 fail=0
 
-# The weight-map index is the source of truth for the shard set.
 INDEX="$DEST/model.safetensors.index.json"
 if [[ ! -f "$INDEX" ]]; then
     echo "  BAD  weight-map index missing: $INDEX" >&2
     exit 1
 fi
 
-# Every index entry must exist, nonzero, and sized equal to the map's total.
-python3 - "$INDEX" "$DEST" <<'PY' || fail=1
-import json, os, sys
-index, dest = sys.argv[1], sys.argv[2]
-wm = json.load(open(index))["weight_map"]
-files = sorted(set(wm.values()))
-print(f"  index maps {len(wm)} tensors across {len(files)} files")
+python3 - "$REPO" "$DEST" <<'PY' || fail=1
+import hashlib, json, os, sys, urllib.request
+
+repo, dest = sys.argv[1], sys.argv[2]
+
+def listing():
+    """The tree API, following the Link cursor exactly like hfget.sh."""
+    out, api, pages = [], (
+        f"https://huggingface.co/api/models/{repo}/tree/main"
+        "?recursive=true&expand=true"), 0
+    while api:
+        with urllib.request.urlopen(api) as r:
+            out += json.load(r)
+            api = None
+            link = r.headers.get("Link", "")
+            for part in link.split(","):
+                if 'rel="next"' in part:
+                    api = part[part.find("<") + 1:part.find(">")]
+        pages += 1
+        if pages > 200:
+            raise SystemExit("listing did not terminate")
+    return out
+
+files = [e for e in listing() if e.get("type") == "file"]
+sizes = {e["path"]: e.get("size", 0) for e in files}
+lfs = {e["path"]: e["lfs"]["oid"] for e in files if e.get("lfs")}
+
 bad = 0
-total = 0
-for f in files:
-    p = os.path.join(dest, f)
-    if not os.path.isfile(p) or os.path.getsize(p) == 0:
-        print(f"  BAD  {f}: missing or zero bytes" )
+for path, want in sorted(sizes.items()):
+    p = os.path.join(dest, path)
+    if not os.path.isfile(p):
+        print(f"  BAD  {path}: missing")
         bad += 1
-    else:
-        total += os.path.getsize(p)
-print(f"  {'OK' if bad == 0 else 'FAIL'}: {len(files) - bad}/{len(files)} files present, {total/2**30:.1f} GiB total")
-raise SystemExit(1 if bad else 0)
+    elif os.path.getsize(p) != want:
+        print(f"  BAD  {path}: {os.path.getsize(p)} bytes, listing says {want}")
+        bad += 1
+print(f"  {'OK' if bad == 0 else 'FAIL'}: {len(sizes) - bad}/{len(sizes)} files present at listed size")
+if bad:
+    raise SystemExit(1)
+
+# Shard set against the weight-map index.
+wm = json.load(open(os.path.join(dest, "model.safetensors.index.json")))["weight_map"]
+shards = sorted(set(wm.values()))
+print(f"  index maps {len(wm)} tensors across {len(shards)} shards")
+if len(shards) != 131:
+    print(f"  BAD  shard count {len(shards)}, want 131")
+    raise SystemExit(1)
+
+# Content: every LFS-tracked file hashed against the Hub oid. The safetensors
+# shards dominate the bytes; tokenizer/config files ride the same check when
+# the Hub tracks them in LFS (they are plain blobs here, already size-checked).
+todo = [p for p in shards if p in lfs]
+for i, path in enumerate(todo):
+    h = hashlib.sha256()
+    with open(os.path.join(dest, path), "rb") as f:
+        for chunk in iter(lambda: f.read(1 << 24), b""):
+            h.update(chunk)
+    if h.hexdigest() != lfs[path]:
+        print(f"  BAD  {path}: sha256 mismatch (want {lfs[path]})")
+        bad += 1
+    if (i + 1) % 16 == 0:
+        print(f"  ...hashed {i + 1}/{len(todo)}")
+print(f"  {'OK' if bad == 0 else 'FAIL'}: {len(todo) - bad}/{len(todo)} shards sha256-match the Hub")
+if bad:
+    raise SystemExit(1)
 PY
 
-# Shard count gate (what the serve script and deployer check).
-n="$(find "$DEST" -maxdepth 1 -name 'model-*-of-*.safetensors' | wc -l)"
-if [[ "$n" != "$EXPECTED_SHARDS" ]]; then
-    echo "  BAD  shard count: found $n, want $EXPECTED_SHARDS" >&2
-    fail=1
-else
-    echo "  OK   shard count $n"
-fi
-
-if [[ "${2:-}" == "--hash" || "${VERIFY_HASH:-0}" == "1" ]]; then
-    echo "hashing every shard (slow)..."
-    cd "$DEST"
-    sha256sum -c --quiet model.safetensors.index.json 2>/dev/null || \
-    sha256sum model-*-of-00131.safetensors > /dev/null  # presence check; content hash list printed on demand
-fi
-
 [[ "$fail" -eq 0 ]] || exit 1
-echo "payload inventory OK: $EXPECTED_SHARDS shards + index present"
+echo "payload verified: 131 shards + index + tokenizer + configs, sizes and sha256 checked"

scripts/download-frank-secondary.sh

@@ -0,0 +1,88 @@
+#!/usr/bin/env bash
+# Download the frank-secondary payload for llama.cpp on the RTX 3090:
+# ajgazin/Swift-1.5-Qwen3.8-27B-Uncensored-Dynamic-MTP-GGUF:RCO-IQ3_XXS
+# **with the MTP head in the file** (the -mtp variant), plus the
+# PUBLISHER's OWN vision projector (mmproj-BF16.gguf, same repo). Public
+# repo — no auth header, and never a token on the command line (it is
+# world-readable in /proc/<pid>/cmdline).
+#
+# Fresh downloads, no resume (`curl -C -` once produced a 213 MB oversized
+# file on a 31 GiB payload); -f/--fail-with-body so an HTTP error is not a
+# success, then byte count and sha256. This repo is on Xet storage, so
+# the Hub tree/blob APIs expose no LFS oid: the sha256 values below were
+# computed locally after the first verified fetch (byte counts match the
+# Hub's declared sizes). Idempotent: complete files are skipped. The
+# projector is stored under a self-documenting dest name; the URL name
+# and sha stay pinned in the table below.
+#
+# 2026-10-06 (Frank, evening): the payload moved to the UNCELERSED
+# Swift-1.5-Qwen3.8-27B at RCO-IQ3_XXS (1.56 GiB lighter than the
+# IQ3_S-mtp it replaced; the lighter quant funds the MTP draft ctx plus
+# the second 131,072 slot). Rollback payload stays on disk:
+# /home/dank/models/Swift15-IQ3S/ (the IQ3_S-mtp that was live until this
+# swap); earlier payloads Ornith-Q8/ and MiMo-Q6K/ as well. Their
+# re-fetch recipes are in git history (predecessors of this script).
+#
+# Usage:
+#   scripts/download-frank-secondary.sh [DEST_DIR] [--verify-only]
+set -euo pipefail
+
+DEST=""
+VERIFY_ONLY=0
+for arg in "$@"; do
+    case "$arg" in
+        --verify-only) VERIFY_ONLY=1 ;;
+        *) DEST="$arg" ;;
+    esac
+done
+DEST="${DEST:-/home/dank/models/Swift15-UncIQ3XXS}"
+
+# repo | hub-name | dest-name | bytes | sha256 (computed locally; Xet repo)
+FILES=(
+  "ajgazin/Swift-1.5-Qwen3.8-27B-Uncensored-Dynamic-MTP-GGUF|RCO/Swift-1.5-Qwen3.8-27B-Uncensored-Dynamic-MTP-RCO-IQ3_XXS.gguf|Swift-1.5-Qwen3.8-27B-Uncensored-Dynamic-MTP-RCO-IQ3_XXS.gguf|10442827392|e13d3e7400b6c7f980ebd30c0edd6be43eb1865b50be8562a9db042706c7d803"
+  "ajgazin/Swift-1.5-Qwen3.8-27B-Uncensored-Dynamic-MTP-GGUF|mmproj-BF16.gguf|mmproj-BF16.gguf|931145888|19acd7fcf4ee504328566eddf0f8e603ef853a205b4830209d2a4908213abdea"
+)
+
+mkdir -p "$DEST"
+status=0
+for entry in "${FILES[@]}"; do
+  IFS='|' read -r repo hubname destname size sha <<<"$entry"
+  url="https://huggingface.co/${repo}/resolve/main/${hubname}"
+  f="$DEST/$destname"
+  have=0
+  if [[ -f "$f" ]] && [[ "$(stat -c%s "$f")" == "$size" ]]; then
+    if sha256sum -c --quiet <(printf '%s  %s\n' "$sha" "$f"); then
+      have=1
+    fi
+  fi
+  if (( have )); then
+    echo "OK        $destname"
+    continue
+  fi
+  if (( VERIFY_ONLY )); then
+    echo "MISSING   $destname"
+    status=1
+    continue
+  fi
+  echo "FETCHING  $destname (from $repo)"
+  tmp="$f.part"
+  curl -fsSL --fail-with-body -o "$tmp" "$url"
+  mv "$tmp" "$f"
+  if [[ "$(stat -c%s "$f")" != "$size" ]]; then
+    echo "SIZE FAIL $destname ($(stat -c%s "$f") != $size)"
+    status=1
+    continue
+  fi
+  if ! sha256sum -c --quiet <(printf '%s  %s\n' "$sha" "$f"); then
+    echo "SHA FAIL  $destname"
+    status=1
+    continue
+  fi
+  echo "VERIFIED  $destname"
+done
+if (( status == 0 )); then
+  echo "DOWNLOAD_DONE $DEST"
+else
+  echo "DOWNLOAD_FAILED $DEST" >&2
+fi
+exit "$status"

scripts/download-frank-small.sh

@@ -1,87 +0,0 @@
-#!/usr/bin/env bash
-# Download the frank-small payload for llama.cpp on the RTX 3090:
-# ornith-ai/Ornith-1.5-9B-GGUF:Q8_0 plus its vision projector
-# (the GGUF carries no `v.*` tensors — the tower lives only in the mmproj, so
-# without this file the endpoint is text-only by construction). Public repo —
-# no auth header, and never a token on the command line (it is world-readable
-# in /proc/<pid>/cmdline).
-#
-# Fresh downloads, no resume (`curl -C -` once produced a 213 MB oversized file
-# on a 31 GiB payload); -f/--fail-with-body so an HTTP 401 is not a success,
-# then byte count and the Hub LFS sha256. Idempotent: complete files are
-# skipped.
-#
-# 2026-10-05 (Frank): the rung is Ornith-1.5-9B Q8_0. The payload moved off
-# MiMo-V2.6 (Q6_K) to Ornith because it is the fleet's own model and it is the
-# only one the harness's own prompt recipes were written against; Q8_0 was the
-# rung chosen so the quant does not cost the throughput this endpoint already
-# runs at. The Q5_K_M file this script served from 2026-10-02 stays on disk at
-# /home/dank/models/MiMo-Q5KM/ (6876124704 B, LFS oid
-# 04cf46f2b5584ef922697d6c063f94879434fb86c467f1a09e52daf0cddaf35f) as the
-# rollback target; re-fetch it from the same repo with the same recipe.
-#
-# Usage:
-#   scripts/download-frank-small.sh [DEST_DIR] [--verify-only]
-set -euo pipefail
-
-REPO="ornith-ai/Ornith-1.5-9B-GGUF"
-BASE="https://huggingface.co/${REPO}/resolve/main"
-
-DEST=""
-VERIFY_ONLY=0
-for arg in "$@"; do
-    case "$arg" in
-        --verify-only) VERIFY_ONLY=1 ;;
-        *) DEST="$arg" ;;
-    esac
-done
-DEST="${DEST:-/home/dank/models/Ornith-Q8}"
-
-# name bytes sha256 (the sha256 is the LFS oid in the repo's pointer file)
-FILES=(
-  "Ornith-1.5-9B-Q8_0.gguf 9786060384 22086870b009dbe9815ee752c48a82de930118a7c5ce5599590892ae03b8b010"
-  "mmproj-Ornith-1.5-9B-BF16.gguf 921704672 626f9f90627402a6bf4a999111d0fbd69b5fcca7aa8ba089d69e5f10e8858e1d"
-)
-
-mkdir -p "$DEST"
-status=0
-for entry in "${FILES[@]}"; do
-  read -r name size sha <<<"$entry"
-  f="$DEST/$name"
-  have=0
-  if [[ -f "$f" ]] && [[ "$(stat -c%s "$f")" == "$size" ]]; then
-    if sha256sum -c --quiet <(printf '%s  %s\n' "$sha" "$f"); then
-      have=1
-    fi
-  fi
-  if (( have )); then
-    echo "OK        $name"
-    continue
-  fi
-  if (( VERIFY_ONLY )); then
-    echo "MISSING   $name"
-    status=1
-    continue
-  fi
-  echo "FETCHING  $name"
-  tmp="$f.part"
-  curl -fsSL --fail-with-body -o "$tmp" "$BASE/$name"
-  mv "$tmp" "$f"
-  if [[ "$(stat -c%s "$f")" != "$size" ]]; then
-    echo "SIZE FAIL $name ($(stat -c%s "$f") != $size)"
-    status=1
-    continue
-  fi
-  if ! sha256sum -c --quiet <(printf '%s  %s\n' "$sha" "$f"); then
-    echo "SHA FAIL  $name"
-    status=1
-    continue
-  fi
-  echo "VERIFIED  $name"
-done
-if (( status == 0 )); then
-  echo "DOWNLOAD_DONE $DEST"
-else
-  echo "DOWNLOAD_FAILED $DEST" >&2
-fi
-exit "$status"

scripts/frank-large.env

@@ -1,75 +1,88 @@
 # frank-large.env — tunables for the frank-large endpoint on Dankclaw's
 # Strix Halo iGPU (gfx1151), host port 7770.
 #
-# RUNTIME (2026-10-05, Frank): halogen — the closed-source C++/HIP engine
-# (github.com/peonist-ai/halogen-server, image ghcr.io/peonist-ai/halogen,
-# digest-pinned below) serving peonist-ai/halogen-qwen3.8-27b, a ~6.32 bpw
-# NVFP4-derivative quant of the BASE Qwen/Qwen3.8-27B (35,865,565,184 B,
-# 1352 tensors), as a podman container under a root unit, driven by
-# scripts/serve-frank-large.sh. This CUTS OVER the
-# 2026-10-02..10-03 ROCmFPX llama.cpp shape (uncensored
-# Q6_0_ROCMFPX_AGENT + MTP, 27.6-28.1 t/s): rollback = revert the swap
-# commit's trio; the ROCmFPX payload, draft head and build all stay on
-# disk at /home/dank/models/kingjones-Qwen3.8-27B-Uncensored-ROCmFP4-STRIX-MTP/.
+# RUNTIME (2026-10-06, Frank): halogen-flash-server 0.16.2 — the
+# closed-source C++/HIP engine (image ghcr.io/peonist-ai/halogen-flash-server,
+# digest-pinned below) serving the Quat3rnion ABLITERATED Swift1.5 finetune
+# of peonist's Qwen3.8-Flash-Next v2 checkpoint, as a podman container under
+# a root unit, driven by scripts/serve-frank-large.sh. This CUTS OVER the
+# 2026-10-05 halogen 27B-dense shape (censored base, DFlash2, 31.7 t/s):
+# rollback = revert the swap commit's trio; the 27B payload stays on disk at
+# /home/dank/models/halogen-qwen3.8-27b/.
 #
 # WHY THE RUNTIME MOVED WITH THE MODEL: the payload is .hgn — per its own
-# card it "will not load in transformers, vLLM, or llama.cpp". Accepted
-# consequences (Frank, 2026-10-05): TEXT-ONLY (halogen ignores the model's
-# vision encoder; the fleet's vision path is frank-small) and the CENSORED
-# base (this is a finetune of Qwen/Qwen3.8-27B, not the abliterated file —
-# refusals may return). What it buys: vendor-measured decode DFlash2 31.7
-# t/s mean (vs this endpoint's MTP 27.6-28.1 and the old halogen-era MTP
-# 26.9), prefill ~566 t/s at 32k (was ~400 on the sibling engine), 1.4-2.3x
-# wall-clock on prefill-heavy agent traffic, at HIGHER effective precision
-# (~6.32 bpw vs ~6.6 with the old W4A4 promotion — see FLAGS.md).
+# card it "will not load in transformers, vLLM, or llama.cpp" — and the v2
+# .hgn loads ONLY in the flash-server line (the old halogen engine is for
+# the 27B p1w4d file; the v2 quickstart pins flash-server). What the move
+# buys (card + live probe evidence): MTP spec decode (53-57 t/s greedy,
+# 3-10x serial), ~1,640 t/s prefill at 34k, ~72% fewer thinking tokens than
+# the censored 27B (905 mean), the ABLITERATED base (refusals should NOT
+# return), IMAGE INPUT (the v2 vision tower loads beside the checkpoint —
+# red-PNG probe verified on the card; the 27B engine ignored its encoder),
+# tool calls verified, and the native 262,144 context.
 #
-# SHAPE — the two knobs that are NOT free:
-#   HALOGEN_KV_SLOTS=1 KEEPS SPECULATION ON. halogen makes batching and
-#   speculation mutually exclusive: with >1 slot the drafter is disabled
-#   and per-stream decode falls to serial speed (the sibling engine
-#   measured 41 t/s -> 14 t/s when slots were raised, and halogen's own
-#   FLAGS.md prints the same law). frank-large was already a one-session
-#   endpoint under ROCmFPX (PARALLEL=1), so slots=1 is the same contract,
-#   not a regression.
-#   HALOGEN_SLOT_CTX=200700 (Frank's window) is the per-conversation hard
+# SHAPE — the knobs that are NOT free, with the arithmetic:
+#   HALOGEN_CTX=200192 is the PER-SLOT window and the per-conversation hard
 #   ceiling — a prompt over it is a 400 naming the limit, never a silent
-#   truncation. KV pool = slots x slot_ctx x 64 KiB = 12.2 GiB. Budget
-#   against the 121 GiB of unified memory: weights 33.4 GiB (mmap'd — page
-#   cache, not anon), KV 12.2, engine working set + auto prompt cache
-#   ~20-30, host baseline ~10 -> ~75-85 GiB in use. The 2026-09-16 halogen
-#   era ran the same engine class on this box with LESS headroom.
-#   HALOGEN_DRAFTER=2 (DFlash2) is the shipped default and the fastest
-#   exact drafter on this checkpoint; the .hgn BAKES BOTH drafters (MTP +
-#   DFlash2), so unlike the Flash-Next era there is no sidecar to mount
-#   and no --dflash flag — the drafter is an int knob, byte-identical
-#   output to greedy verification either way.
+#   truncation. Harness compaction fires at 70% = 140,134, and 140,134 +
+#   57,344 (harness maxTokens) = 197,478 <= 200,192 keeps prompt + budget
+#   inside the slot. models.yml contextWindow must quote 200192 or the
+#   harness discovers the wall as a 400.
+#   HALOGEN_KV_POOL_POSITIONS=400384 = KV_SLOTS(2) x CTX — the explicit
+#   form of the engine's default, so the seam test can pin the product.
+#   KV costs ~28 KiB/position (FLAGS.md) => 400,384 x 28 KiB = 10.7 GiB.
+#   HALOGEN_KV_SLOTS=2 restores the w4b-era shape on this box: the MTP
+#   drafter is NOT slot-locked on this engine (batching and speculation
+#   coexist — DFlash-era slots>1 law does NOT carry over; the card's own
+#   quickstart ships 2 slots WITH the head), so the second slot is
+#   concurrent capacity, not a speculation tax.
+#   Memory ledger against the box's 121.2 GiB unified (the n-gram table is
+#   mmap'd page cache beside the checkpoint, not a pinned charge — the
+#   card's own quickstart arithmetic does not bill it):
+#     weights ~62 GiB resident + KV 10.7 + arena (HALOGEN_MAX_TOK=8192 =>
+#     ~8 GiB per FLAGS.md arena formula 3.8+8.4*(MT/16384)) + vision ~2
+#     = ~82.7 GiB engine-side; HALOGEN_HOST_RESERVE_GIB=20 keeps the host
+#     honest; baseline co-tenancy (frank-secondary, ComfyUI, Caddy/observ)
+#     ~10-15. ~18 GiB margin. The boot ledger is the referee — read it
+#     after deploy and re-size if the engine prints a clamp.
+#   HALOGEN_SPEC_ADAPT=0 is MANDATORY (card): the default adaptive policy
+#   turns the MTP head off for whole requests after low acceptance and
+#   HALVES decode as a result. Frank accepted the head always-on trade
+#   (temp 1.0 traffic costs acceptance; the card still ships the recipe).
 #
-# CLIENT CONTRACT deltas vs the ROCmFPX shape (the harness side lives in
+# CLIENT CONTRACT deltas vs the 27B shape (harness side lives in
 # ~/.omp/agent/models.yml):
-#   * NO server-side sampler defaults: a request without temperature
-#     decodes GREEDY. The harness sends temp 1.0 / top_p 0.95 via
-#     extraBody, so that wire recipe is load-bearing now, not cosmetic.
-#   * max_tokens defaults to 8192 PER REQUEST and INCLUDES thinking; a
-#     reasoning reply over a small budget returns empty content with
-#     finish_reason "length". The harness maxTokens is 57344: compaction
-#     fires at 70% of 200700 = 140,490 and 140,490 + 57,344 = 197,834 <=
-#     200,700 keeps prompt + budget inside the pool (the 262144-era 65536
-#     would overflow). Requests over HALOGEN_MAX_TOKENS_CAP are a 400
-#     naming the cap; cap and queue timeout are COUPLED (see FLAGS.md
-#     table) — do not move one without the other.
-#   * response_format json_object/json_schema is a 400 (no constrained
-#     decoding in this build).
-#   * Thinking lands in reasoning_content — same field as the llama.cpp
-#     era, so the harness compat block does NOT change. reasoning_effort
-#     accepts none|low|xhigh on this build (template default xhigh); the
-#     models.yml fold (high->xhigh) still applies.
+#   * SERVER-SIDE SAMPLER EXISTS NOW: HALOGEN_TEMPERATURE/TOP_P/TOP_K set
+#     the card's thinking recipe (1.0 / 0.95 / 20) as engine defaults, so
+#     an absent temperature is no longer greedy. models.yml extraBody
+#     carries the same values — consistent, not load-bearing.
+#   * HALOGEN_MAX_TOK=8192 is only the DEFAULT for requests that omit
+#     max_tokens; the harness always sends 57344. Requests over
+#     HALOGEN_MAX_TOKENS_CAP (65536) are a 400 naming the cap. Cap and
+#     queue timeout are COUPLED (FLAGS.md table): 65536 at ~10 t/s serial
+#     worst case needs > 6550 s; 7200 honors it. Do not move one alone.
+#   * max_tokens INCLUDES thinking; a reasoning reply over a small budget
+#     returns empty content with finish_reason "length".
+#   * Prompt cache is ON by default (PROMPT_CACHE=2): a follow-up turn at
+#     100k context re-prefills ~2 s instead of ~88 s cold. No pin needed —
+#     unlike the 27B engine, auto-sizing fits this box's ledger.
+#   * Thinking lands in reasoning_content (same as the 27B engine); the
+#     v2 chat template rejects reasoning_effort outside xhigh|medium|low,
+#     so the models.yml fold (high->xhigh) still applies.
+#   * Vision: OpenAI image_url content parts only (the fleet's standard
+#     contract). >=256 px probe images or the model answers "video".
+#   * response_format json_object/json_schema unsupported in this line
+#     (no constrained decoding) — unchanged from the 27B engine.
 #   * The engine's token protocol has no auth and binds loopback INSIDE
-#     the container; only the frontend port is published. Keep it that way.
+#     the container; only the frontend port (8731) is published. Same as
+#     before. Kernel needs CONFIG_HSA_AMD_SVM (CachyOS ships it) and
+#     amdgpu.noretry=0 (check at deploy).
 
 # Image: digest-pinned (fetch law: tag pulls silently re-point; the digest
-# is what was qualified). 0.1.4 = the version the model card names.
-FRANK_LARGE_IMAGE=ghcr.io/peonist-ai/halogen@sha256:dc0a39a0016d6cfc58a197978febaafdf8d28403f724ded6111d98b5fb7ac0ea
-# Container identity: the halogen container name.
+# is what was qualified). 0.16.2 = the version the model card names as its
+# quickstart pin; RepoDigest captured at pull on 2026-10-06.
+FRANK_LARGE_IMAGE=ghcr.io/peonist-ai/halogen-flash-server@sha256:0c61bf84ac22308a53f5d1ca6b86806702d7039e5ebc51cae4c66621b92fe04a
+# Container identity: kept from the 27B era (unit ExecStop names it).
 FRANK_LARGE_CONTAINER=frank-large-halogen
 # Old quadlet-era container identity; the serve script clears stale names.
 FRANK_LARGE_LEGACY_CONTAINERS=frank-large
@@ -77,37 +90,56 @@ FRANK_LARGE_LEGACY_CONTAINERS=frank-large
 # default); the engine port (8730) is never published.
 FRANK_LARGE_PUBLISH=7770:8731
 
-FRANK_LARGE_MODEL_DIR=/home/dank/models/halogen-qwen3.8-27b
-FRANK_LARGE_CHECKPOINT=qwen3.8-27b-p1w4d-d2.hgn
-FRANK_LARGE_CHECKPOINT_BYTES=35865565184
-FRANK_LARGE_CHECKPOINT_SHA256=274c3fc767fb57faf025dd03c76376b1fc1b32448aef45aaf02d1898fa962089
-# The tokenizer is a FLAT DIRECTORY, $FRANK_LARGE_MODEL_DIR/tokenizer, mounted
-# whole at /tokenizer — the model card ships `tokenizer/` and the engine's
-# HALOGEN_TOKENIZER default is that directory (the card's own docker run sets
-# no tokenizer flag). The Hub stores it as 5 files (sizes from the Hub API,
-# 2026-10-05): tokenizer.json 12809320 (LFS sha256
-# 0997f410c57a1f4e53b09e4be8f4a172d90edd9564368fb0847030937229b9f3),
-# vocab.json 6722759, merges.txt 3353259, chat_template.jinja 8952,
-# tokenizer_config.json 17928 (plain git blobs — size-checked, no LFS oid).
-# The serve gate and the deployer refuse a missing file or ANY symlink: an
-# HF cache snapshot symlinks each file into blobs/, and those links dangle
+FRANK_LARGE_MODEL_DIR=/home/dank/models/halogen-flashnext-v2
+# The checkpoint: Quat3rnion/halogen-swift1.5-qwen3.8-flash-next-v2-abliterated
+# qwen38-flash-next-v2-swift15-abliterated.hgn — compacted r2 of the card's
+# append-only original (byte-identical live tensors, padding dropped:
+# 66,687,678,464 B vs 89,775,868,032 B). LFS sha256 verified at fetch.
+FRANK_LARGE_CHECKPOINT=qwen38-flash-next-v2-swift15-abliterated.hgn
+FRANK_LARGE_CHECKPOINT_BYTES=66687678464
+FRANK_LARGE_CHECKPOINT_SHA256=9c880b8ba48deabfa06172eff1e7572e4fe465e09e97faec6e50c3f6fdcac037
+# The n-gram lookup table: REQUIRED (the engine reads it beside the
+# checkpoint; the card's quickstart names it explicitly). From
+# peonist-ai/halogen-qwen3.8-flash-next @ 5cc17cea.
+FRANK_LARGE_NGRAM_TABLE=qwen38-flash-next-ngram.hgn
+FRANK_LARGE_NGRAM_BYTES=51200246144
+FRANK_LARGE_NGRAM_SHA256=3450acada94e19aabad88bd49b45eb70304820949c2240fc178067b20ae5dffe
+# The vision tower: OPTIONAL on the engine, ENABLED here (image input).
+# From peonist-ai/halogen-qwen3.8-flash-next @ e053f488.
+FRANK_LARGE_VISION_TOWER=qwen38-flash-next-vision.hgn
+FRANK_LARGE_VISION_BYTES=897916416
+FRANK_LARGE_VISION_SHA256=d62e0ae553fe88afd3833733d4a4c669f34d20fd8dfce4b9610525bed2134b10
+# The tokenizer is a FLAT DIRECTORY, $FRANK_LARGE_MODEL_DIR/tokenizer,
+# mounted whole at /tokenizer. The two repos ship the same one; fetched
+# fresh from peonist @ e053f488 (tokenizer.json is LFS, byte-identical to
+# the 27B-era copy; the rest are plain git blobs — size-checked only). The
+# serve gate and the deployer refuse a missing file or ANY symlink: an HF
+# cache snapshot symlinks each file into blobs/, and those links dangle
 # inside the container mount.
-FRANK_LARGE_TOKENIZER_FILES="tokenizer.json chat_template.jinja tokenizer_config.json merges.txt vocab.json"
+FRANK_LARGE_TOKENIZER_FILES="tokenizer.json chat_template.jinja tokenizer_config.json merges.txt vocab.json generation_config.json"
 
-# --- engine flags (docs/FLAGS.md of the engine repo; anything unset stays
-# at the image default, which is what was qualified) ------------------------
+# --- engine flags (docs/FLAGS.md of the flash-server repo; anything unset
+# stays at the image default, which is what was qualified) ------------------
 HALOGEN_MODEL_ID=frank-large
-HALOGEN_SLOT_CTX=200700
-HALOGEN_KV_SLOTS=1
-HALOGEN_DRAFTER=2
+HALOGEN_CTX=200192
+HALOGEN_KV_POOL_POSITIONS=400384
+HALOGEN_KV_SLOTS=2
+HALOGEN_HOST_RESERVE_GIB=20
+# Spec decode: MTP head is INSIDE the checkpoint (do NOT set HALOGEN_MTP_HEAD);
+# depth 2 is the documented "best all-round" default. HALOGEN_DRAFTER /
+# HALOGEN_DFLASH_* from the 27B engine DO NOT EXIST here.
+HALOGEN_SPEC_ADAPT=0
 # Coupled pair (FLAGS.md table): the cap bounds how long ONE request holds
-# the GPU, the timeout bounds how long the NEXT client waits. 65536 at
-# ~10 t/s serial worst case needs > 6550 s queue timeout; the defaults
-# honor that. Do not lower the timeout alone.
+# the GPU, the timeout bounds how long the NEXT client waits. Do not move
+# one alone (see the client-contract note above).
+HALOGEN_MAX_TOK=8192
 HALOGEN_MAX_TOKENS_CAP=65536
 HALOGEN_QUEUE_TIMEOUT=7200
-# Engine prompt cache: EMPTY = auto (sized from available memory at
-# startup). Pin an explicit MB only to argue with the auto figure — a
-# full-context entry is ~18 GB at 262K, so a too-small pin yields a cache
-# that reports enabled and never hits (the engine warns at startup).
-#HALOGEN_CACHE_MB=
+# Server-side sampler = the card's thinking recipe (the engine now HAS one;
+# requests may still override per-call). Matches models.yml extraBody.
+HALOGEN_TEMPERATURE=1.0
+HALOGEN_TOP_P=0.95
+HALOGEN_TOP_K=20
+# Card's belt-and-braces: the engine pins weights by default
+# (HALOGEN_FLASH_PIN_TRUNK=1; only the retired w4b build honored 0).
+HALOGEN_WEIGHTS_LOCK=1

scripts/frank-large.service

@@ -1,5 +1,5 @@
 [Unit]
-Description=halogen container (peonist-ai/halogen-qwen3.8-27b, DFlash2 spec decode, 1 slot x 200700, text-only), root systemd unit (dankclaw, frank-large, Strix Halo iGPU)
+Description=halogen-flash-server container (Quat3rnion abliterated Swift1.5 of Qwen3.8-Flash-Next v2, MTP spec decode, 2 slots x 200192, vision ON), root systemd unit (dankclaw, frank-large, Strix Halo iGPU)
 After=network-online.target
 Wants=network-online.target
 
@@ -22,8 +22,8 @@ TimeoutStopSec=120
 # would otherwise leave the endpoint down until someone notices.
 Restart=always
 RestartSec=10
-# The engine mmaps 33.4 GiB of weights and keeps a multi-GiB working set
-# on a box that also hosts frank-small, ComfyUI and the Caddy/observability
+# The engine pins ~62 GiB of v2 weights and keeps a multi-GiB working set
+# on a box that also hosts frank-secondary, ComfyUI and the Caddy/observability
 # stack; make it a late OOM candidate, not the first.
 OOMScoreAdjust=-500
 

scripts/frank-primary.env

@@ -1,362 +1,142 @@
-# frank-primary.env — tcclaviger RDNA4 vLLM production config (dankserver :7770).
+# frank-primary.env — radiance (StillDeadcode, Apache-2.0) serving the
+# OFFICIAL Qwen/Qwen3.8-Flash-Next-FP8 checkpoint at NATIVE precision
+# (modelopt block-FP8 throughout, no requantisation) on BOTH R9700s (TP=2),
+# dankserver :7770.
 #
-# 2026-10-03 shape G (as shipped, after the 1.00 revert): the CUDA-graph plan
-# term dropped from the analytic 0.55 GiB to the measured 0.07 GiB (0 base +
-# 12 MiB/graph), and UTIL at the 0.982 law. MEASURED on the live deploy: arena
-# 17.77 GiB / 7,762 slots, KV 401,408 tokens / 2.00x concurrency (the first
-# shape holding two full-length conversations with real spare), driver used
-# 30.97 GiB of 31.86 — 0.89 GiB of real free VRAM per card, held through a
-# live 4,096-token generation. UTIL stays 0.982 ALWAYS: 1.00 is unbootable on
-# this fork (worker init refuses budget >= device total) and 0.13-GiB-margin
-# boots die under load.
-# `budget - non-expert - KV reserve - graphs - overhead` and the KV reserve is a
-# registry constant, so those two terms are the whole flag surface.
-# 2026-10-03 shape F (absorbed): MNBT 2048 + the graph term pinned. Measured on
-# the live deploy: arena 17.52 GiB / 7,653 slots, budget 31.29, graphs 0.32.
-# 2026-10-02 shape E: fp8 KV, MAXLEN 200704 x SEQS 3, pool 3.15e9/rank, and
-# --expert-precise-memory ON. The window shrink from the native 262144 and the
-# --max-num-seqs 3 lock both come from shape D (SEQS=2 CANNOT boot this model —
-# the measured 2x2 is below); shape E adds the precise-overhead plan, which
-# trades the default 7.60 GiB compile headroom for the fitted 5.25 GiB and puts
-# the 2.35 GiB difference into the expert arena: 6,496 -> 7,522 slots (+16%).
-# The pool is deliberately NOT cut (option A): the pool is not an arena lever
-# (measured), so 3.15e9 keeps shape D's 1.99x concurrency while the arena grows.
-# Same digest-pinned tcclaviger RDNA4 fork. Rollback for the fp8+MTP4 era is git
-# tag frank-primary-fp8-mtp-20261001.
+# 2026-10-05 cutover from FreeToken (PR #587 ROCm, one card, experts in
+# pinned host RAM; git tag frank-primary-freetoken-pr587-20261005 is the
+# pre-cutover state — its unit/env/serve trio and Dockerfile are all in the
+# tag and rebuildable). Before that: tcclaviger vLLM fork (tag
+# frank-primary-vllm-shapeG-20261003).
 #
-# HIP reads bare env vars TEMP, HOST, GPU_MAX_HEAP_SIZE — every key here is
-# prefixed FRANK_PRIMARY_ so none can leak. The one bare name we WANT inside the
-# container (GPU_MAX_HW_QUEUES) is held here under the prefix and passed on the
-# run line as `-e GPU_MAX_HW_QUEUES=…`.
+# WHY RADIANCE: the engine is C++/HIP with libr4d kernels built for
+# gfx1201, drives `podman run` directly from the digest-pinned
+# stilldeadcode/radiance image (287 MiB; host needs only the amdgpu driver),
+# and its qwen4exp plugin reads the modelopt FP8 encoding NATIVELY
+# (`X.weight_scale_inv` is the scale plane — checkpoint.h). The PLE n-gram
+# table (51B params, 47.7 GiB) is served from pinned host RAM via io_uring
+# reads — Frank's call: "Native FP8. Experts from ram. Pin the PLE to ram."
+# The 4-bit-codebook publisher shape (qwen3.8-next-flash-fp8-iq4r-moe.rad,
+# --host-pool-mib 12288) remains the documented fallback if the native
+# container proves unbootable.
 #
+# PAYLOAD: /home/dank/models/rad/qwen4exp-fp8-native.rad — a rad-convert
+# NO-RECIPE pure copy of the official 131-shard checkpoint (every weight
+# keeps its block-FP8 encoding; ~161 GiB). The convert runs against a
+# staging dir whose config.json drops `quantization_config`: the descriptor
+# string is only the plugin SELECTION key (spec §2.4 — exact match, no
+# fallback), and no shipped plugin claims (qwen4exp, "fp8"); the shipped
+# qwen4exp plugin claims (qwen4exp, "") which is what recipe-built
+# containers carry. Per-weight encodings are data, not the descriptor —
+# removing the key changes nothing about what the weights hold. Built by
+# scripts/build-frank-primary-rad.sh (staging + convert + rad-info).
 #
-# --- runtime ----------------------------------------------------------------
-# Digest-pinned: the floating :latest tag moved once already. Local alias
-# qwen-fp8-runtime:tuned-20260929 (image ID 3322dbbbc96d) points at the same
-# bytes in rootless storage.
-FRANK_PRIMARY_IMAGE=docker.io/tcclaviger/vllm@sha256:da4029666668b90a0386c1ed23ea370c79916ad4e1a6ad1dd9db098b9fa069ea
-
-# --- payload ----------------------------------------------------------------
-FRANK_PRIMARY_MODEL_DIR=/home/dank/models/Qwen3.8-Flash-Next-Uncensored-FP8
-FRANK_PRIMARY_MODEL_SHARDS=131
-
-# --- identity ---------------------------------------------------------------
-# The harness models.yml id stays frank-primary (provider 192.168.0.235:7770).
-FRANK_PRIMARY_CONTAINER=frank-primary-qwen
+# SERVING FACTS (docs/GUIDE.md, docs/DOCKER.md, compose/flashnext.yaml):
+# - The HTTP listener binds AFTER the full load (main.cpp: engine.start()
+#   then engine.serve()) — during load the port refuses connections, so a
+#   bare curl /health gate is honest. /health == /ping: 200 JSON while
+#   serving, 503 only while stopping (core/server/admin.cpp:385).
+# - Reasoning: --reasoning-format defaults to auto, emitting thinking in
+#   `reasoning_content` (config.cpp). Tool calls: the qwen3.8 dialect, from
+#   the chat format. Both need no parser flags.
+# - Sampling: the OpenAI endpoint accepts temperature, top_p, top_k, min_p,
+#   typical_p, presence_penalty, frequency_penalty, repetition_penalty,
+#   repeat_last_n, chat_template_kwargs, reasoning_effort,
+#   enable_thinking, preserve_thinking (core/server/oai.cpp:194-205) —
+#   models.yml's extraBody set is fully inside it. Unknown params are
+#   refused (400), which is why nothing unverified rides the wire.
+# - MTP: the official checkpoint carries the mtp.* head (3,101 tensors,
+#   mtp_num_hidden_layers=1) and the plugin probes draft depth 3 — the
+#   publisher's throughput peak for this model. Stated explicitly with
+#   --num-speculative-tokens 3.
+# - The fused TP all-reduce is declared over max-num-seqs x
+#   (1 + --num-speculative-tokens) rows; 2 x 4 = 8 rows fits either wire.
+FRANK_PRIMARY_IMAGE=stilldeadcode/radiance@sha256:4c3f484564bb2872784e46b0a59cbd71becc9c1e746e301a3596ffae253c322e
+# The .rad container FILE (mounted under /models) and the state dir for the
+# prefix-cache disk tier (mounted rw at /data). The model dir holds ONLY
+# .rad files; the raw HF download and its staging dir live beside it and
+# are never mounted.
+FRANK_PRIMARY_MODEL_DIR=/home/dank/models/rad
+FRANK_PRIMARY_MODEL_FILE=qwen4exp-fp8-native.rad
+FRANK_PRIMARY_STATE_DIR=/home/dank/radiance-state
+# The state dir's tune/ subdirectory holds rad-tune's kernel-selection
+# cache (the engine's selector reads $RADIANCE_HOME/tune/<machine>.tune
+# at boot; $RADIANCE_HOME defaults to /opt/radiance, in the image, so the
+# cache must ride a host mount — serve script mounts it, see the runbook's
+# "Tuning levers").
+# Container identity is the SERVICE name (Frank's call, 2026-10-05): one
+# name across systemctl and podman ps. The runtime changes; the name does
+# not. The legacy list retires every predecessor identity that could hold
+# the GPUs or the port.
+FRANK_PRIMARY_CONTAINER=frank-primary
+FRANK_PRIMARY_LEGACY_CONTAINERS="frank-primary-qwen frank-primary-r9v qwen-fp8 glm-probe"
 FRANK_PRIMARY_SERVED_NAME=frank-primary
-FRANK_PRIMARY_PUBLISH=7770:7770
-# Containers removed before every start: the retired R9V identity and the ad
-# hoc experiment identity (both hold the GPUs / name registrations).
-FRANK_PRIMARY_LEGACY_CONTAINERS="frank-primary-r9v qwen-fp8"
+FRANK_PRIMARY_PORT=7770
+# Frank's chat-template call: froggeric's fixed Qwen3.8 template (the
+# repo's root chat_template.jinja, vendored here; the checkpoint's own
+# template is baked into the .rad by rad-convert and this overrides it).
+# Radiance derives the reasoning/tool-call reply format from it.
+FRANK_PRIMARY_TEMPLATE=/home/dank/serverAppConfigs/scripts/qwen38-froggeric-v22.5.jinja
 
-# --- qualified engine shape (2026-10-02, shape E) ---------------------------
-# WHAT THE SLOT COUNT DOES AND DOES NOT DO. The endpoint advertises 3 slots
-# (SEQS 3), but the POOL, not the slot count, decides how many full-length
-# conversations are resident. MEASURED LIVE 2026-10-02 at this shape:
-#   GPU KV cache size: 398,658 tokens,
-#   Maximum concurrency for 200,704 tokens per request: 1.99x
-# (3,146,129,408 B reserved — 7,892 B/token). The per-token price is
-# SHAPE-DEPENDENT, not a constant: the 262144 shape priced at 7,701 B/token
-# (6,100,000,000 B -> 792,069 tokens), ~2.5% cheaper, so never extrapolate a
-# concurrency figure across geometries. Shorter requests use the spare blocks,
-# and a third LONG request queues rather than being refused. Verified: three
-# concurrent 67-token requests answered in parallel, ~4.1 s each.
-#
-# WHY NOT --max-num-seqs 2 (measured 2026-10-02, single-variable boot probes
-# against the then-deployed known-good shape, production stopped for each):
-#   SEQS 2 + MNBT 1024 -> Engine core initialization failed
-#   SEQS 2 + MNBT 2048 -> Engine core initialization failed
-#   SEQS 3 + MNBT 1024 -> healthy (792,069 tokens, 3.02x, arena 6,248)
-#   SEQS 3 + MNBT 2048 -> healthy (the deployed incumbent)
-# The failure is torch.fx ConstraintViolationError: query_start_loc.size()[0]
-# "marked dynamic but your code specialized it to be a constant (3)", raised
-# from AOT build_guards(strict_error=True) during profile_run -> _dummy_run.
-# model_runner.py sizes that input buffer max_num_reqs + 1 and _dummy_run sets
-# num_reqs = min(max_num_batched_tokens, max_num_reqs), so the width is
-# SEQS + 1 for ANY MNBT >= SEQS: 3 at SEQS 2, 4 at SEQS 3. MNBT only moves the
-# compile RANGE ((1,1024) vs (1,2048)) and cannot dodge it — the 2x2 above is
-# why MNBT is not a lever here. --enforce-eager bypasses the path but loses
-# CUDA graphs. Same failure class killed the 2026-10-01 bf16 2 x 262144 shape.
-#
-# THE ARENA LEVER IS max_model_len, NOT KV_BYTES (measured 2026-10-02). The
-# planner sizes expert cache as budget - non-expert - KV reserve - graphs -
-# overhead, and the KV reserve is registry-priced at 9190 B/token x max_model_len:
-#   MAXLEN 262144 -> reserve 2.48 GiB -> arena 14.30 GiB / 6,248 slots
-#   MAXLEN 200704 -> reserve 1.91 GiB -> arena 14.87 GiB / 6,496 slots
-# d(reserve) 0.57 GiB = d(arena) 0.57 GiB = +248 slots. The 6.1e9 -> 3.15e9
-# pool cut buys ZERO arena; it buys runtime HEADROOM (the shape it replaced
-# died with 0 bytes free on a real 995-token request). To grow the arena,
-# lower MAXLEN — not KV_BYTES. The OTHER arena lever is the overhead term:
-# --expert-precise-memory (shape E) takes it from 7.60 to 5.25 GiB and lands
-# those 2.35 GiB in the arena — 14.87 -> 17.22 GiB / 6,496 -> 7,522 slots.
-#
-# GATHER 32x64 stays: R4D_LRU_GATHER_CHUNKS=32 / R4D_LRU_GATHER_LANES=64
-# measured +18.7% decode vs default geometry and the win grows with depth.
-# Do not drop it.
-# CHUNKED PREFILL: the fork exposes the standard scheduler flags. --max-num-batched-tokens
-# (MNBT) is the lever; --enable-chunked-prefill is an auto-flag (True in SchedulerConfig)
-# and --prefill-schedule-interval is the gap between iterations. The running shape carries
-# --max-num-batched-tokens 2048 with --max-model-len 200704, so chunked prefill is ON and
-# already active: a 200704-token prompt is prefilled in ~98 iterations of 2048 tokens.
-# MEASURED 2026-09-29 on the ad hoc qwen-fp8 container (same fork, same payload, TP=2):
-# MNBT 8192 vs the default 2048 is NEUTRAL on decode at every depth, and small values
-# re-chunk prefill only — they hurt TTFT and never touch decode, because a decode step is
-# max_num_seqs x (spec+1) tokens, independent of MNBT. There is NO --ubatch-size
-# equivalent in vLLM: that is a llama.cpp compute-buffer concept (the frank-small pair
-# doubled on 2026-10-05 has no counterpart here). MNBT is therefore NOT a lever on this
-# shape — the deployed 2048 is the fork's default — and every change re-enters the
-# compile RANGE path, so FRANK_PRIMARY_CACHE_DIR pays one recompilation.
-# MNBT 2048 (Frank's call, 2026-10-03; the lock was 1024). The measured law
-# still holds: MNBT 512 doubles TTFT at 4064 ctx (2.46 s vs 1.36 s), 1024-4096
-# are interchangeable on decode, 12288 costs 2.5% plus arena. 2048 is ARENA-FREE
-# by construction: the registry's overhead model is
-# base + 0.22 GiB x max(0, (nbt - 4096)/1024) with the anchor at nbt 4096, so
-# every rung up to and including 4096 prices the SAME 5.25 GiB overhead — and
-# the fitted 5.25 was fitted at nbt 4096, i.e. our 2048 sits UNDER the anchor,
-# so the activation term is covered by a figure measured at a bigger batch.
-# SEQS 3 boots at both 1024 and 2048 (measured 2026-10-02; the SEQS=2
-# ConstraintViolationError is a SEQS property, not an MNBT one). Costs exactly
-# one recompilation: the compile RANGE is part of the cache key, so
-# FRANK_PRIMARY_CACHE_DIR pays for whichever rung is chosen once.
-# PREFETCH IS ABSENT BY LAW: expert-cache telemetry measured the prefetcher at
-# 36% precision (36,720 slabs prefetched, 13,369 served) — roughly half the
-# gather traffic was waste. Never re-add --expert-prefetch without a fresh A/B.
-# TunableOp stays OFF: CLAV_TUNABLEOP_SWEEP silently skips under expert offload
-# unless CLAV_TUNABLEOP_WITH_OFFLOAD=1, and forcing it corrupts numerics.
-# --expert-precise-memory is ON (shape E). It selects the fitted overhead
-# (5.25 GiB) over the default (7.60 GiB = fitted + compile/graph-capture
-# headroom); the 2.35 GiB difference lands in the expert arena. MEASURED
-# 2026-10-02 by a boot probe on the shape-D base (production stopped, candidate
-# booted AS the service, plan read off the engine's own log):
-#   runtime overhead  7.60 -> 5.25 GiB   (registry Qwen4Exp…, precise
-#                                         gfx1201 TP2 nbt4096/seqs16/len262144)
-#   expert cache      14.87 -> 17.22 GiB
-#   slots             6,496 -> 7,522     (+1,026 = +16%, 2.344 MiB each)
-#   swap region                 13.7 GiB / 5,986 slots, 133-134 per layer
-#   boot                        healthy in 420 s, no OOM, no knife-edge
-# The flag was called a non-option after it crash-looped the 2 x 262144 bf16
-# boot (0.13 GiB free of 31.86 GiB, NRestarts=3 at ~6.7 min each) — that shape
-# was already at the ceiling, so precise-memory was the last 2.35 GiB over, not
-# the cause. At 200704 x 3 it fits with room; the run line also reports
-# 'expert_precise_memory': True in non-default args, so a silent argparse drop
-# would be visible in the boot log.
-FRANK_PRIMARY_PRECISE_MEMORY=1
-# GPU_MAX_HW_QUEUES caps the HSA hardware queues per GPU (AMD's own tuning
-# guide recommends 2 for this class of workload; the runtime default on Linux
-# is 4). Verified a REAL knob, not a no-op: the string is present in
-# /opt/rocm/core-10.0/lib/libhsa-runtime64.so. NOT A/B MEASURED — Frank's call,
-# shipped as a deliberate CPU-side tune alongside the 32x64 gather geometry.
-FRANK_PRIMARY_GPU_MAX_HW_QUEUES=2
-# Persistent compile cache. MEASURED 2026-10-02: the image ships NO
-# /root/.cache at all (`ls: cannot access '/root/.cache'` in a fresh
-# container), and after our boot it holds /root/.cache/vllm at 1,154,714,333 B
-# in 4,665 files plus /root/.cache/comgr at 229 MB — both inside the
-# container's ephemeral layer, so every `podman rm -f` (every deploy, restart
-# and reboot) throws them away and re-pays the build. The engine's own log
-# names the loss: `torch.compile took 181.20 s in total`, of which 'Compiling a
-# graph for compile range (1, 1024) takes 173.19 s', followed by 'saved AOT
-# compiled function to /root/.cache/vllm/torch_compile_cache/…'. Mounting a
-# host dir at /root/.cache persists the vLLM AOT/torch_compile tree AND the
-# comgr HIP-kernel JIT cache, and hides nothing (the image has no such dir).
-# PROVEN LIVE 2026-10-02, two boots of this exact shape:
-#   cold  `Compiling a graph for compile range (1, 1024) takes 179.24 s`
-#         `torch.compile took 188.26 s in total`
-#   warm  `Directly load the compiled graph(s) for compile range (1, 1024)
-#          from the cache, took 2.143 s`
-#         `torch.compile took 3.97 s in total`
-#         `init engine (profile, create kv cache, warmup model) took 41.13 s
-#          (compilation: 3.98 s)`
-# (Total time to /health is dominated by the 172.8 GiB payload load, not the
-# compile — the saving is the ~184 s of inductor/AOT build.)
-# Safe to grow/prune: entries are keyed by version + config hash, so a stale
-# entry is a miss, never a wrong answer.
-FRANK_PRIMARY_CACHE_DIR=/home/dank/.cache/frank-primary
-FRANK_PRIMARY_MAXLEN=200704
-FRANK_PRIMARY_SEQS=3
-# The POOL is deliberately kept at shape D's 3.15e9 (option A): because the pool
-# is not an arena lever, raising it back from a cut buys concurrency only, and
-# 3.15e9 preserves the measured 1.99x while precise-memory grows the arena. A
-# 2.2e9 pool was probed in the same window (277,686 tokens / 1.38x) and rejected
-# as the shipped value; it remains the conservative arm if free VRAM ever binds.
-# KV forced to 400,700 tokens (Frank, 2026-10-03). The engine prices this pool
-# at 7,891.8 B/token at this shape and loses ~3.87 MB to block alignment, so
-# 3,166,500,000 B is the ask that lands at/above 400,700 (verify the reported
-# "GPU KV cache size" line on every deploy; it is not a constant ratio across
-# geometries — 7,701 B/token at 262144).
-FRANK_PRIMARY_KV_BYTES=3166500000
+# --- engine shape (first-boot conservative; the boot ledger arbitrates) --
+# Tensor parallel: BOTH R9700s. The board swap (2026-09-20, ProArt B850
+# x8/x8 on root ports) made collectives legal on this host; radiance's TP
+# wire is libr4d's own, not RCCL.
+FRANK_PRIMARY_TP=2
+# Lossy WHT 6-bit wire for large all-reduces (publisher's production choice:
+# +5.6-6.3% prefill, ~-1.3% 8-stream decode at TP2). min-kb stays the
+# engine default.
+FRANK_PRIMARY_TP_WIRE=wht6
+# Decode slots. 2 is Frank's first-boot value ("use parallelism = 2 to
+# start"); raise only after a real-traffic look at slot occupancy and the
+# KV pool's headroom line. The recurrent-state cost scales with it (54
+# MiB/copy/card, 2 copies with speculation).
+FRANK_PRIMARY_MAX_SEQS=3
+# Served window. Native is 262,144; 200,000 is the publisher's production
+# value (compose) and what models.yml's contextWindow must quote — a longer
+# prompt is refused by the engine, never truncated.
+FRANK_PRIMARY_MAX_MODEL_LEN=200000
+FRANK_PRIMARY_PLACEMENT=expert_tiered
+# Pinned host pool for the expert rows that do not fit VRAM. Native FP8
+# experts are 112.8 GiB total (56.4 GiB per rank slice) vs the 4-bit
+# publisher shape's ~29 GiB — this pool is the cost of native precision.
+# The boot ledger prints the pool's coverage ("the pool backs N% of the
+# worst case ... M full-length sessions"); resize from that line, not from
+# arithmetic.
+FRANK_PRIMARY_HOST_POOL_MIB=40960
+# VRAM left unclaimed per card (publisher default 96 MiB).
+FRANK_PRIMARY_GPU_HEADROOM_MIB=96
+# Weights share of claimed VRAM (publisher production value; higher = more
+# prefix-cache hit rate, less KV headroom).
+FRANK_PRIMARY_EXPERT_VS_CACHE=0.82
+# KV cache in FP8 (7.32 KiB/token at this model's 12 full-attention layers,
+# 2 KV heads, 256 head_dim, K+V).
 FRANK_PRIMARY_KV_DTYPE=fp8
-FRANK_PRIMARY_MNBT=2048
-FRANK_PRIMARY_GATHER_CHUNKS=32
-FRANK_PRIMARY_GATHER_LANES=64
-# PREFIX CACHING: ON. An OFF experiment ran 2026-10-05 (below) and FAILED
-# to stop the corruption, so the historical flag is back.
-# MEASURED 2026-10-04, 18:46:07 PDT (5.6 h into the live boot, one running
-# request, prefix hit rate 48.1%): 48 gdn_hip state faults in one 0.6 s
-# burst, then the process-lifetime silence cap:
-#   [gdn_hip] ERROR gdn_decode_r: state out of ssm-cache range: max|S|=0
-#   with non-finite values (slot=42 head=3 limit=3.40282e+38) fault #1-23
-#   [gdn_hip] ERROR gdn_decode_r: ... (slot=42 head=8 ...) fault #24-46
-#   [gdn_hip] ERROR: state fault report limit (32) reached; further faults
-#   are counted silently
-# One physical SSM slot, two heads, BOTH ranks (heads 3 and 8), 23 layers
-# each. The guard's own message in this build: "known Qwen3.8-Flash-Next
-# (QFN) flaw under prefix caching and is being actively worked; output
-# quality for this request may degrade. Mitigation:
-# --no-enable-prefix-caching".
-# MECHANISM: with prefix caching on, this fork puts QFN on
-# mamba_cache_mode 'align' (the default-for-this-arch branch in
-# config/vllm.py; 'all' is deprecated and 'none' is the no-prefix-cache
-# mode, so align is the only sound mode WITH the flag). Align mode
-# checkpoints each sequence's GDN (Gated DeltaNet) recurrent state into a
-# mamba page every block, where block = 3136 tokens (the attention block
-# size is raised to 3136 so the page sizes match exactly — the boot log's
-# "Padding mamba page size by 0.13% ..."), and RE-SEEDS the state from that
-# page whenever a request resumes from a cached prefix. The corrupt state
-# landed at the 13th boundary: 13 x 3136 = 40,768 tokens — the "~40k" mark
-# where the request's output turned repetitive/corrupted. The NaN stayed
-# in the mamba page, so every later turn of that conversation re-seeded the
-# same non-finite state for those two heads (silent: the fault-report cap
-# was already spent), while fresh short sessions looked fine.
-# RAM WAS "RULED OUT" THE SAME DAY — AND THAT WAS WRONG (RESOLVED below).
-# What the evidence actually supported: zero MCE/EDAC records is the
-# EXPECTED silence of consumer non-ECC memory (a silent flip raises no
-# machine check); an 8 GiB pattern check over 60 s covers ~0.5 % of
-# 246 GiB — no verdict on a speed setting; "4800 is the JEDEC floor on
-# 5600-rated DIMMs" is a rating, not a stability proof at four-DIMM
-# loading. Only the two-head NaN signature argument was load-bearing,
-# and the revert test outvoted it.
-# COST ACCEPTED: no prefix reuse, so each turn re-prefills the whole
-# history instead of at most 3135 tail tokens (a full 40k context costs
-# tens of seconds of TTFT per turn at this shape's prefill rate). A silently
-# corrupted conversation is worse than a slower first token.
-# RESULT (2026-10-05, later that day): THE MITIGATION DID NOT WORK. With
-# --no-enable-prefix-caching deployed and mechanically verified (zero
-# gdn_hip faults, a 40,771-token single-request boundary probe coherent),
-# live long conversations STILL degraded at ~40k on the running endpoint
-# (Frank, direct). The container serving that window logged ZERO faults
-# while the corruption reproduced, so the align-mode page re-seed is not
-# the (only) mechanism — see the runbook's RESULT note for the open
-# hypotheses (re-fed poisoned transcript vs a fault-free long-context
-# path such as fp8 KV at depth). Cost bought nothing: caching is back ON.
-# The knob stays commented for a future fork build with a fixed GDN state
-# path: set 1, redeploy, re-verify past 40,768 tokens (one conversation
-# coherent AND zero gdn_hip lines).
-# RESOLVED 2026-10-05 EVENING (Frank): IT WAS THE RAM. 4800 MT/s (in
-# effect since the 2026-10-04 17:17 boot) was marginal; 4600 MT/s (since
-# the 2026-10-05 12:21 boot) and the endpoint is clean — zero gdn_hip
-# ERROR lines, zero MCE, and the live agent session on it past 140k
-# context with no degradation. Both failure eras sit under the RAM: the
-# 2026-10-04 NaN burst (corrupted weight/activation traffic poisoning a
-# GDN state write — the guard-visible path) and the fault-free ~40k
-# degradation (silent flips in KV/expert data, which no guard covers on
-# non-ECC). CONSEQUENCE FOR THE KNOB: the OFF experiment was CONFOUNDED
-# — both arms ran on the marginal speed — so it neither proved nor
-# disproved the fork's align-mode flaw. Prefix caching stays ON (zero
-# gdn_hip lines under heavy re-send traffic since the speed drop); flip
-# the knob only if a fault recurs on stable RAM or the fork ships a fix.
-# FRANK_PRIMARY_NO_PREFIX_CACHING=1
-# UTIL is an ARENA lever: the fork plans `budget = device_total x util`, and at
-# 31.86 GiB/card every +0.01 is +0.32 GiB of expert cache (~+139 slots), and
-# 0.982 is Frank's standing law: NEVER 1.00. The 1.00 attempt (2026-10-03,
-# shape G v1) died in worker init BEFORE loading a byte of weights — the fork
-# refuses budget >= device total with
-#   ValueError: Free memory on device cuda:N (31.29/31.86 GiB ...)
-# on BOTH ranks at 09:53 and 09:54, NRestarts=2 — so 1.0 is not even a runtime
-# knife-edge here, it is unbootable. The two historical ceiling deaths (the
-# 2x262144 bf16 boot at 0.13 GiB free; the 5x262144 boot at "0 bytes free" on a
-# real 995-token request) remain the reason the margin exists. Rollback note
-# for the other direction: if a request ever dies with
-# `Tried to allocate ... 0 bytes free`, the first knob back is THIS number
-# down a notch, not the window.
-FRANK_PRIMARY_UTIL=0.982
-# CUDA-graph plan term, priced at the measured capture. expert_plan.py charges
-# estimate_cudagraph_memory() = BASE_MB + PER_GRAPH_MB x n_graphs (defaults
-# 256/52 = 0.55 GiB for our 6-graph plan), and the estimator's own docstring
-# says to tune these against the post-capture line. MEASURED twice on this
-# shape: shape E reported "Graph capturing finished ... took 0.31 GiB" (TP0) and
-# 0.17 GiB (TP1); shape F, with the 0.32 GiB we then charged, reported
-# "took -0.06 GiB" — a NEGATIVE delta, i.e. capture fits inside memory the
-# profiler had already counted. So the base pool (256 MiB) is the over-charge:
-# charge 0 base + 12 MiB/graph = 0.07 GiB for the shipped no-spec 6-graph plan,
-# which returns 0.25 GiB more to the arena while still paying per graph, so a
-# future spec-decode shape (18 graphs) is charged 0.21 GiB rather than zero.
-# Verify every deploy against the "Estimated CUDA graph memory" vs "Graph
-# capturing finished ... took X GiB" lines; if capture ever goes positive and
-# exceeds the charge, raise FRANK_PRIMARY_CUDAGRAPH_PER_GRAPH_MB back toward 52.
-# A THIRD env var exists and is the only way to state the graph term outright:
-# VLLM_CUDAGRAPH_MEM_ESTIMATE_MB pins the WHOLE term to an exact MiB total in
-# one knob — estimate_cudagraph_memory() checks it FIRST and returns base/
-# per-graph untouched when set (verified in the shipped image,
-# v1/worker/gpu/cudagraph_mem.py). We ship the BASE/PER_GRAPH pair instead so
-# a future spec-decode shape re-prices automatically; switch to ESTIMATE_MB
-# only if the drift check below ever shows the pair cannot track capture.
-FRANK_PRIMARY_CUDAGRAPH_BASE_MB=0
-FRANK_PRIMARY_CUDAGRAPH_PER_GRAPH_MB=12
-
-# HOST-region cap — the last arena-plan term that still autofit per boot. With
-# no cap the planner's host bound is MemAvailable AT BOOT minus PLE 47.68 -
-# 2 x 7.0 process - 5.0 api/engine - 5% margin (measured boot 2026-10-05:
-# MemAvailable 237.97 -> 159.39 GiB usable across 2 ranks; the demand is 105.48
-# = 52.74/rank x 2 ranks, 23,040 swap-target experts). The same host reads 75.4
-# GiB available while running — the bound tracks whatever the co-tenants hold at
-# boot, and a boot where that drops near the demand silently pins next-hottest
-# experts VRAM-only instead (smaller swap region; the pinned_extra warning is
-# the only trace). Pin 120 GiB: measured demand + 14% headroom, on a host with
-# 246.1 GiB total. If a boot ever logs "expert-offload-mem … exceeds
-# MemAvailable at plan time", THAT is the contract being broken — look at the
-# co-tenant RAM before touching this number. Plan semantics verified in the
-# shipped image (offloader/expert_plan.py): the cap binds as the bound, an
-# exceedance only warns.
-FRANK_PRIMARY_EXPERT_OFFLOAD_MEM=120
-
-# --- optional engine knobs (unset == the flag is absent) ---------------------
-# FRANK_PRIMARY_SPEC holds the WHOLE --speculative-config JSON; setting it adds
-# the flag. THE RETIREMENT VERDICT WAS INVALID: c1248d4 (2026-10-02) retired MTP
-# by comparing fp8+MTP4 (37.4-40.6 tok/s, 54% acceptance, 09-29) against the
-# bf16 no-spec boot (36.4-39.4) — and bf16 then proved unbootable, so fp8 came
-# back without MTP and fp8+spec was NEVER compared against fp8+no-spec on the
-# same shape. The no-spec incumbent measures ~30 tok/s single-stream (below the
-# fleet's 35 tok/s target); every vendor flash-next recipe ships MTP3; and our
-# MTP4 run used the fork's default GREEDY draft sampling, while the recipes pass
-# draft_sample_method=probabilistic (greedy drafts are rejected more often under
-# temp>0 — the plausible cause of the 54%). MTP also re-adds drafter workspace:
-# the old note says the 7.2 GiB/rank KV ceiling WAS the MTP4 workspace, and
-# --expert-precise-memory spends that same budget, so an MTP arm must be tested
-# with precise memory BOTH on and off.
-#   MTP3:  '{"method":"mtp","num_speculative_tokens":3,"draft_sample_method":"probabilistic"}'
-FRANK_PRIMARY_SPEC=
-# FRANK_PRIMARY_COMPCONFIG holds the whole --compilation-config JSON. The
-# capture-ladder law: the piecewise cudagraph ladder must contain
-# num_speculative_tokens + 1 (MTP3 -> 4 -> ladder 4,8,12; MTP4 -> 5,10,15;
-# MTP5 -> 6,12,18,24), because every draft step proposes n+1 tokens. The default
-# ladder is multiples of 1+4 up to 512 then powers of two, so an MTP3 shape can
-# miss its own step size. Pinning the ladder is ALSO the candidate fix for the
-# SEQS>3 wall: query_start_loc is specialised to SEQS+1 and that constant must
-# land on a captured graph size (3 is not in the default ladder, 4 is —
-# measured 2026-10-02, and the vendor runs SEQS 16 on this model/TP2).
-# combo_kernels false/true are BOTH in the shipped recipes, so it is not a law.
-#   MTP3 at SEQS 3:  '{"cudagraph_mode":"PIECEWISE","cudagraph_capture_sizes":[1,4,8,12],"max_cudagraph_capture_size":12}'
-FRANK_PRIMARY_COMPCONFIG=
-# FRANK_PRIMARY_GEN_CONFIG holds the whole --override-generation-config JSON.
-# The payload's HF generation_config.json sets temperature 1.0 / top_p 0.95 /
-# top_k 20 and the engine WARNS about it on every boot, so a client that sends
-# no sampling fields gets creativity it never asked for. This is the one knob
-# that changes ANSWERS, not speed.
-#   '{"max_tokens":65536,"temperature":0.7,"top_p":0.95,"top_k":20}'
-FRANK_PRIMARY_GEN_CONFIG=
-
-# --- client contract: reasoning split, tools, preserve_thinking ----------------
-# Without --reasoning-parser the engine leaves the thinking text INLINE in
-# `content` with a dangling `</think>` and a null `reasoning` field (measured
-# 2026-10-01: content = 'We need answer...Brief.\n\n391') — the
-# malformed thinking block clients choke on. The qwen3 parser splits it
-# properly. The GLM rollback (d91379d) had dropped all three of these; the
-# working values are the ones ecaad36 shipped for this model.
-FRANK_PRIMARY_REASONING_PARSER=qwen3
-# Tools: --enable-auto-tool-choice + the qwen3_xml parser is what makes
-# tool_calls work here (without it, tool_choice auto returns 400).
-FRANK_PRIMARY_TOOL_PARSER=qwen3_xml
-# preserve_thinking keeps prior thinking blocks in the rendered prompt across
-# turns; enable_thinking makes thinking the server-side default. Verified the
-# payload's tokenizer_config template consumes both keys.
-FRANK_PRIMARY_CHAT_TEMPLATE_KWARGS='{"enable_thinking": true, "preserve_thinking": true}'
+# Spec decode: the checkpoint's own MTP head at the publisher's depth-3
+# operating point (probe() returns 3 when mtp_num_hidden_layers > 0; the
+# flag states it so the ledger records the intent).
+FRANK_PRIMARY_SPEC_TOKENS=2
+FRANK_PRIMARY_MAX_BATCHED_TOKENS=2048
+# Prefix cache tiers (publisher pattern; disk tier sized for 2 slots).
+FRANK_PRIMARY_PREFIX_HOST_MIB=4096
+FRANK_PRIMARY_PREFIX_DISK_MIB=65536
+# Frank's pin law: the PLE n-gram table (47.7 GiB) serves from pinned host
+# RAM the cards read. `ram` REFUSES the start if the memlock limit is too
+# small — the unit's LimitMEMLOCK=infinity is the other half of this law.
+FRANK_PRIMARY_NGRAM_PLACEMENT=ram
+# TTM preflight: the pinned ledger (2 x 40 GiB pools + 47.7 GiB PLE + 4 GiB
+# prefix host) is ~132 GiB. ttm's pages_limit/pages_max cap the GPU-mappable
+# pinned pages; the current cap (32,252,917 pages = ~123 GiB) is BELOW the
+# plan, so the serve script refuses to start until both are raised. This is
+# the floor it enforces (in MiB), with ~8 GiB of margin.
+FRANK_PRIMARY_TTM_MIN_MIB=140000
+# Escape hatch for boot-ladder experiments. Unset -> no extra argv. Keep
+# multi-flag values as one string; the serve script expands it unquoted on
+# purpose.
+FRANK_PRIMARY_EXTRA_ARGS=
+# The .rad's minimum size in MiB (deployer gate; a truncated convert is a
+# guaranteed long-timeout boot instead of a fast failure). The converted
+# container measured 177,109 MiB (1,857,127,997,440 B = 173.0 GiB; the
+# plan-only estimate 172.76 GiB covers the data planes but not the metadata
+# tail) - floor sits just under the payload.
+FRANK_PRIMARY_RAD_MIN_MIB=177000

scripts/frank-primary.service

@@ -1,29 +1,25 @@
-# frank-primary.service — tcclaviger RDNA4 vLLM endpoint (orcarouter
-# Qwen3.8-Flash-Next-Uncensored-FP8, fp8 KV, NO spec decode, 3 slots x 200704
-# context with the pool sized to ~2 full-length conversations, TP=2, tiered
-# experts, 131-shard payload) on the 2x R9700 pair, dankserver :7770.
-# 2026-09-29 cutover from the R9V radiance fork (git tag
-# frank-primary-r9v-q8-20260929 is the pre-cutover state; its
-# launcher/shim/setup-state machinery is retired and the state dirs stay on
-# disk as rollback).
+# frank-primary.service — radiance (StillDeadcode, Apache-2.0) endpoint
+# (OFFICIAL Qwen/Qwen3.8-Flash-Next-FP8 at NATIVE block-FP8 precision, the
+# rad-convert container qwen4exp-fp8-native.rad) on BOTH R9700s (TP=2,
+# wht6 wire), dankserver :7770. 2026-10-05 cutover from FreeToken (git tag
+# frank-primary-freetoken-pr587-20261005 is the pre-cutover state; the
+# retired unit/env/serve trio and Dockerfile are in that tag).
 #
-# Same port, same served alias; the harness models.yml id is unchanged.
-# The serve script drives `podman run` directly (the image entrypoint execs
-# `vllm serve`) and then waits on the container. NOTE: that wait can return
-# 0 even when the engine died mid-request (vLLM's API server drains and
-# exits clean on a worker fault), which is why the restart policy is
-# `always` — see the [Service] note.
-#
-# WHY A ROOT UNIT (not a rootless quadlet): the tiered-expert shape pins tens
-# of GiB of host RAM (the expert host slabs, ~20.5 GiB per rank of pinned
-# anonymous memory). Pinned pages are charged against RLIMIT_MEMLOCK; dank's
-# login session on this host carries 8 MiB and a user quadlet is capped by the
+# Same port, same served alias, same container identity as the unit name;
+# the harness models.yml id is unchanged. The serve script drives
+# `podman run` directly: the digest-pinned stilldeadcode/radiance image
+# (287 MiB, kernels prebuilt for gfx1201) has the engine as its
+# entrypoint. The image lives in dank's ROOTLESS podman storage.
+# WHY A ROOT UNIT (not a rootless quadlet): radiance pins ~132 GiB of HOST
+# RAM (2 x 40 GiB expert pools + the 47.7 GiB PLE ngram table + the prefix
+# host tier). Pinned pages are charged against RLIMIT_MEMLOCK; dank's login
+# session on this host carries 8 MiB and a user quadlet is capped by the
 # user manager's own hard limit — raising that means restarting
-# user@1000.service and bouncing every user container on the host. A root unit
-# grants LimitMEMLOCK=infinity to the dank process via PID1, and the image
-# stays in dank's rootless podman storage.
+# user@1000.service and bouncing every user container on the host. A root
+# unit grants LimitMEMLOCK=infinity to the dank process via PID1; the
+# engine still runs out of dank's rootless podman storage.
 [Unit]
-Description=frank-primary — tcclaviger vLLM Qwen3.8-Flash-Next-Uncensored-FP8 on 2x R9700
+Description=frank-primary — radiance Qwen3.8-Flash-Next-FP8 (native) on 2x R9700 TP=2
 After=network-online.target
 Wants=network-online.target
 # Never let the start-rate limiter give up after a crash loop (a failed LLM
@@ -33,34 +29,35 @@ StartLimitIntervalSec=0
 [Service]
 User=dank
 Group=dank
-# The tiered-expert offloader pins host RAM; an RLIMIT_MEMLOCK smaller than
-# the pinned set fails the load mid-start. PID1 grants infinity here —
-# parity with the fleet's other LLM units.
+# The engine pins ~132 GiB of host RAM (expert pools + PLE table); an
+# RLIMIT_MEMLOCK smaller than the pinned set fails the load mid-start.
+# PID1 grants infinity here — parity with the fleet's other LLM units.
 LimitMEMLOCK=infinity
 Environment="FRANK_PRIMARY_ENV_FILE=/home/dank/serverAppConfigs/scripts/frank-primary.env"
 ExecStart=/home/dank/serverAppConfigs/scripts/serve-frank-primary.sh
 # systemd stop: wait alone cannot stop the container child; stop it
-# explicitly. vLLM 0.26 drains gracefully on SIGTERM.
-ExecStop=/usr/bin/podman stop -t 120 frank-primary-qwen
-# Boot cost: FP8 payload page-in plus torch compile/graph capture —
-# ~15 min cold (measured 2026-09-29), faster warm.
-TimeoutStartSec=1800
-# Restart=always, NOT on-failure (measured 2026-10-01): a runtime
-# torch.OutOfMemoryError kills the engine while vLLM's API server still
-# exits 0, so ExecStart returned clean, systemd logged "Deactivated
-# successfully", and on-failure NEVER fired — the endpoint stayed down until
-# a human noticed. `always` restarts on any container death; an explicit
-# `systemctl stop` still wins (systemd does not restart a unit the operator
-# stopped), so bench windows and deploys are unaffected.
+# explicitly. No graceful drain contract is claimed (the OpenAI front end
+# holds no queue to drain — in-flight decodes are lost, which a stop
+# during a bench window accepts; radiance takes SIGTERM on the engine
+# process and podman reaps the container).
+ExecStop=/usr/bin/podman stop -t 120 frank-primary
+# Boot cost: .rad load (weights + PLE upload) + expert host-pool pin. The
+# publisher's compose healthcheck allows a 15-minute start_period; the
+# vLLM-era budget (1800 s) carried forward.
+# Restart=always, NOT on-failure (the 2026-10-01 vLLM lesson carries over:
+# a clean API-server exit after a worker fault reads as success to
+# systemd). `always` restarts on any container death; an explicit
+# `systemctl stop` still wins, so bench windows and deploys are unaffected.
 Restart=always
 RestartSec=10
-# ExecStop allows the engine 120 s to drain (`podman stop -t 120`); without an
-# explicit TimeoutStopSec systemd's 90 s default cuts the stop short, kills it,
-# and records 'Failed with result timeout' (measured 2026-10-01 13:27:17).
-# Must exceed the podman grace with margin for the engine's own shutdown.
+# ExecStop allows the engine 120 s to drain (`podman stop -t 120`); without
+# an explicit TimeoutStopSec systemd's 90 s default cuts the stop short,
+# kills it, and records 'Failed with result timeout' (measured 2026-10-01
+# 13:27:17). Must exceed the podman grace with margin.
 TimeoutStopSec=180
-# The engine pins tens of GiB of host RAM (expert host slabs); it is the
-# last process on this host that should be chosen by the kernel OOM killer.
+# The engine pins tens of GiB of host RAM (expert rows + PLE table); it is
+# the last process on this host that should be chosen by the kernel OOM
+# killer.
 OOMScoreAdjust=-500
 
 [Install]

scripts/frank-secondary.env

@@ -0,0 +1,98 @@
+# frank-secondary — llama.cpp (master, CUDA sm_86) serving
+# ajgazin/Swift-1.5-Qwen3.8-27B-Uncensored-Dynamic-MTP-GGUF:RCO-IQ3_XXS
+# **with the MTP head in the file** on the RTX 3090, :7771.
+#
+# 2026-10-06 (Frank, evening): the payload moved to ajgazin's UNCELERSED
+# Swift-1.5-Qwen3.8-27B at RCO-IQ3_XXS (10,442,827,392 B) — 1.56 GiB
+# lighter than the IQ3_S-mtp it replaces — with MTP BACK ON (draft-mtp,
+# depth 3), TWO 131,072 slots, and ubatch 512. The uncensored finetune
+# changes behavior by design (publisher: 23/100 refusals vs Swift 1.5's
+# 98/100, KL 0.0884 from Swift 1.5) — Frank's order. The RCO tier is the
+# same per-tensor ALLOCATION recipe as the IQ3_S file (histogram: iq3_s
+# 97 / bf16 96 / iq3_xxs 83 / iq2_s 72 / iq4_xs 33 / iq2_xs 32 / iq2_xxs
+# 30 / q2_k 28 / q4_k 19 / q6_k 8 / iq1_s 4 / iq1_m 4 — all stock types,
+# stock llama.cpp loads it); publisher KLD: Mean 0.002631, 90 % 0.0421
+# (kld_results.csv). RCO files are "more experimental" per the README —
+# the quality floor is the IQ3_S rung, not a promise.
+#
+# The architecture is unchanged (same model family): qwen35 hybrid —
+# block_count 65 (64 real layers + the blk.64 MTP head),
+# full_attention_interval 4 => SIXTEEN full-attention layers, 24 Q heads
+# / 4 KV heads, k/v length 256, native window 262144, dense,
+# nextn_predict_layers=1. The MTP head ships in every main GGUF of this
+# repo (README); --spec-type draft-mtp loads it against the main model's
+# own weights.
+#
+# THE KV CONTRACT STAYS q8_0 K + q8_0 V (Frank, 2026-10-05): 34 KiB/token
+# (16 attn layers x 4 heads x 256 x 1.0625 B, K and V).
+#
+# Why TWO slots + MTP fits now (it did not under IQ3_S):
+#   card 24,576 MiB. Weights 9,963 MiB (10,442,827,392 B; IQ3_S was
+#   11,561). Projector ~1,290 MiB (the PUBLISHER's OWN mmproj-BF16.gguf,
+#   931,145,888 B, at the measured ~1.45x card law — no substitution).
+#   Compute buffers at ubatch 512 ~1.3 GiB. CUDA ctx ~500 MiB. DeltaNet
+#   state ~150 MiB/slot x 2. MTP draft ctx back: ~0.4 GiB. Non-KV ~13.9
+#   GiB. Two q8_0 slots = 8,704 MiB. Predicted free ~2.6-3.2 GiB — back
+#   over the old 2-3 GiB floor that the IQ3_S two-slot boot violated
+#   (measured there: 1,397 MiB free). MEASURE nvidia-smi free at boot
+#   (quote the free field) and let the runbook carry the number.
+#
+# Payload history (rollback trail): /home/dank/models/Swift15-IQ3S/ (the
+# IQ3_S-mtp that was live until this swap), Ornith-Q8/, MiMo-Q6K/ all
+# stay on disk; one MODEL_FILE + shape edit back each.
+LLAMA_BIN=/home/dank/GitPackages/llama-master/build-cuda/bin/llama-server
+MODEL_FILE=/home/dank/models/Swift15-UncIQ3XXS/Swift-1.5-Qwen3.8-27B-Uncensored-Dynamic-MTP-RCO-IQ3_XXS.gguf
+
+# Vision: the publisher SHIPS the projector — mmproj-BF16.gguf
+# (931,145,888 B) from the same repo, fetched by
+# scripts/download-frank-secondary.sh. No substitution needed for this
+# payload (the unsloth base-tower proof of 2026-10-05 remains the
+# fallback argument for the IQ3_S rollback payload).
+# Empty string = text-only on purpose.
+MMPROJ_FILE=/home/dank/models/Swift15-UncIQ3XXS/mmproj-BF16.gguf
+
+# Fleet endpoint: same port and alias as every frank-secondary before it, so the
+# harness models.yml id and the homepage probe do not move.
+ALIAS=frank-secondary
+BIND_ADDR=0.0.0.0
+LISTEN_PORT=7771
+
+# TWO conversations at Frank's 112k cap (2026-10-06 evening): --ctx-size is
+# the POOL llama.cpp divides across slots; 229376 / 2 = 114,688 per
+# conversation. Lowering from 128k (131,072) frees ~1 GiB of KV, which
+# funds the ubatch 512 -> 1024 increase (prefill throughput). The
+# per-slot window is the product contract, so models.yml follows to
+# 114,688.
+CTX_SIZE=229376
+PARALLEL=2
+
+# Compute-buffer geometry. Ubatch is a prefill knob (decode never moves);
+# the 1024 rung was the first A/B back once the per-slot context dropped
+# to 112k (2026-10-06 evening), and 2048 is the second (Frank, same night)
+# — the rung measured +4.6 % prefill for 616 MiB on the 3090 (2026-10-05
+# boot pair), affordable inside the 1,751 MiB the 112k cut freed. The pair
+# is a measurement, not a law (loud failure at graph capture if it does
+# not fit).
+BATCH_SIZE=4096
+UBATCH_SIZE=2048
+
+# The KV contract for this swap (Frank, 2026-10-05): q8_0 K + q8_0 V —
+# llama.cpp's FP8 class. The flags stay explicit so the choice is visible in
+# the argv either way, and --flash-attn stays pinned on in the serve script:
+# it is required for quantized V (the build carries
+# GGML_CUDA_FA_ALL_QUANTS=ON since 2026-09-17, so q8_0 KV is a real path,
+# not a silent f16 fallback).
+CACHE_TYPE_K=q8_0
+CACHE_TYPE_V=q8_0
+
+# Speculative decoding is ON (Frank, 2026-10-06 evening): draft-mtp at
+# depth 3 against the head IN the file (the -mtp variant ships the
+# blk.64.nextn.* tensors). ngram-mod stays skipped — on a qwen35 hybrid
+# it disables recurrent-state rollback, so a rejected draft would
+# corrupt the DeltaNet state; MTP needs no such bypass (its draft ctx
+# re-derives state from the target's own weights). The flags move as a
+# set: the serve script emits BOTH only when SPEC_TYPE names a drafter.
+SPEC_TYPE=draft-mtp
+SPEC_DRAFT_N_MAX=3
+
+NGL=99

scripts/frank-secondary.service

@@ -0,0 +1,19 @@
+[Unit]
+Description=llama.cpp Swift-1.5-Qwen3.8-27B Uncensored RCO IQ3_XXS GGUF (qwen35 hybrid, 2x131072 slots, q8_0 K+V, draft-mtp depth 3, mmproj vision) on the RTX 3090 (dankclaw, frank-secondary)
+After=network-online.target
+Wants=network-online.target
+
+[Service]
+Type=simple
+User=dank
+Group=dank
+# CUDA pins its staging buffers, so the process needs the memlock budget even
+# though the weights themselves are copied into VRAM.
+LimitMEMLOCK=infinity
+Environment="FRANK_SECONDARY_ENV_FILE=/home/dank/serverAppConfigs/scripts/frank-secondary.env"
+ExecStart=/home/dank/serverAppConfigs/scripts/serve-frank-secondary.sh
+Restart=on-failure
+RestartSec=10
+
+[Install]
+WantedBy=multi-user.target

scripts/frank-small.env

@@ -1,136 +0,0 @@
-# frank-small — llama.cpp (master, CUDA sm_86) serving
-# ornith-ai/Ornith-1.5-9B-GGUF:Q8_0 on the RTX 3090, :7771.
-#
-# 2026-10-05 (Frank): the payload moved off bartowski/MiMo-V2.6 (Q6_K) to
-# ornith-ai/Ornith-1.5-9B Q8_0 (9,786,060,384 B, LFS oid 22086870...).
-# Header-verified facts that re-price the shape: qwen35 hybrid like MiMo, but
-# 33 blocks (not 32) with full_attention_interval 4 => NINE full-attention
-# layers (MiMo had 8), 16 Q heads / 4 KV heads, k/v length 256, native window
-# 262144, dense (no expert keys), nextn_predict_layers=1 (an MTP head is IN
-# the file — capability only; no spec decode ships, same law as before).
-#
-# THE KV CONTRACT STAYS f16 K + f16 V — NO KV QUANTIZATION (Frank, 2026-10-05,
-# overruling a written q8_0-V proposal). The arithmetic that makes this hard:
-#   KV rate 9 attn layers x 4 heads x 256 x 2 B = 18 KiB/token K, same again
-#   for V at f16 => 36 KiB/token (MiMo was ~24.5 measured: one fewer
-#   full-attn layer AND one fewer KV head).
-#   Weights +1,900 MiB (9,334 vs Q6_K's 7,432). Measured free at the Q6_K
-#   4096/2048 shape was 2,279 MiB; Q8_0 at f16/f16 does not fit that shape —
-#   the deficit is made up the way Frank ordered: UBATCH FIRST (2048 -> 1024,
-#   the measured 616 MiB rung), then POOL STEPS, then measure. Guess-and-check
-#   on the live card is the accepted method for llama.cpp sizing here: boot,
-#   read nvidia-smi free, step the pool (12,288-token increments, ~432 MiB
-#   each at the 36 KiB/token rate) until the 2-3 GiB floor holds. Compute
-#   buffers allocate at graph capture during BOOT, so a failure is loud and
-#   fast, not a surprise at the first long prompt.
-#
-# Previous shape history (MiMo era, for the rollback trail): 2026-10-03 the
-# rung went Q5_K_M -> Q6_K with the pool HELD at 401,408; the freed third
-# slot funded exact f16 V first; the ~3.5 GiB still free funded the +875 MiB
-# of weights.
-#
-# 2026-10-03 (Frank): TWO streams instead of three, each one WIDE — 200,704
-# tokens per conversation, above the 200,700 the harness compacts at — and the
-# KV cache is now FULLY UNQUANTIZED (f16 K, f16 V). The two moves are the same
-# move: llama.cpp divides --ctx-size across --parallel slots, so the slot given
-# up is exactly the VRAM that pays for unquantized V. No further quantization
-# anywhere on this endpoint.
-#
-# Three measurements on this card, this runtime, price the terms:
-#   2 x 262,144, no projector: 20,190 MiB used / 4,386 free
-#     KV = 524,288 tok x 24.5 KiB = 12,800 MiB => non-KV (weights 6,557 +
-#     DeltaNet state ~25/slot + compute buffers + CUDA graphs) = 7,390 MiB
-#   3 x 196,608, projector on: 22,972 MiB used / 1,153 free
-#     KV = 14,113 MiB => non-KV WITH the vision tower = 8,859 MiB
-#   3 x 180,224, projector on: 21,684 MiB used / 2,441 free (the 2026-10-02
-#     shape) => non-KV = 8,748 MiB
-# Two slots give back one DeltaNet state (~25 MiB). Priced at f16 K+V
-# (32 KiB/token) this pool is 12,544 MiB, so the prediction is
-# 12,544 + ~8,800 = ~20.9 GiB used, ~3.1 GiB free — the band the 2026-10-02
-# shape hit, bought with precision instead of a slot. The 1024 ubatch
-# re-prices the compute buffers upward, so MEASURE after any change: if the
-# boot lands under 2 GiB free, step the pool down in 12,288-token
-# (384 MiB at this rate) increments.
-#
-# KV arithmetic: attention KV exists only in the 8 full-attention layers
-# (4 KV heads x 256 head_dim x 2 B) = 16 KiB/token for K and the same again for
-# V => 32 KiB/token, so 12,288 tokens (= 6144/slot) is one 384 MiB step — that
-# is the ladder to walk if this moves again. Two native windows (2 x 262,144)
-# are NOT reachable with full-precision V: 524,288 x 32 KiB = 16.1 GiB of KV
-# against a card with ~15.7 GiB of KV room once weights, the vision tower, the
-# DeltaNet states and the graphs are loaded. 200,704 per stream is what full
-# precision buys; q8_0 on V would buy 262,144 instead.
-#
-# The asymmetric-KV contract this endpoint was created for (Frank, 2026-10-01
-# night: f16 K + q8_0 V) is now spent — it existed to make a third slot fit,
-# and the third slot is gone. The flags stay explicit so the choice is visible
-# in the argv either way.
-LLAMA_BIN=/home/dank/GitPackages/llama-master/build-cuda/bin/llama-server
-MODEL_FILE=/home/dank/models/Ornith-Q8/Ornith-1.5-9B-Q8_0.gguf
-
-# Vision: the Ornith Q8_0 GGUF ships no `v.*` tensors either (dense-9B byte
-# arithmetic: 9.79 GB ≈ 9B params x ~1.06 B/param leaves no room for a tower),
-# so the projector file IS the vision capability (mmproj-Ornith-1.5-9B-BF16
-# .gguf, 921,704,672 B, LFS oid 626f9f90..., fetched by
-# scripts/download-frank-small.sh). Empty string = text-only on purpose.
-MMPROJ_FILE=/home/dank/models/Ornith-Q8/mmproj-Ornith-1.5-9B-BF16.gguf
-
-# Fleet endpoint: same port and alias as every frank-small before it, so the
-# harness models.yml id and the homepage probe do not move.
-ALIAS=frank-small
-BIND_ADDR=0.0.0.0
-LISTEN_PORT=7771
-
-# ONE conversation at the NATIVE 262,144 window (Frank, 2026-10-05). The
-# two-slot shape died under Ornith and the NO-QUANTIZE law, by arithmetic:
-# the KV rate is the ARCHITECTURE's, not the quant's — 9 full-attn layers x
-# 4 KV heads x 256 dim x 2 B = 36 KiB/token at f16 K+V (MiMo measured 24.5),
-# so 2 x 200,704 wants ~14.1 GiB of KV on top of +1,900 MiB of Q8 weights
-# against 2,279 MiB measured free; ubatch 1024 claws back 616, and no
-# pool step fits BOTH the 2-3 GiB floor and the harness compaction
-# invariant (slot >= 191,148). One native slot fits with room; the window
-# per conversation RISES 200,704 -> 262,144. The cost: a second concurrent
-# request queues behind the released slot (subagent concurrency 2 -> 1).
-CTX_SIZE=262144
-PARALLEL=1
-
-# Compute-buffer geometry. This endpoint passed neither flag until 2026-10-03,
-# so it ran on llama.cpp's own defaults (2048/512); the pair is now 4096/2048.
-# Both are tunables because the pair is a measurement, not a law — and the
-# measurement is CARD-SPECIFIC. On the 2x R9700 layer-split sibling the ubatch
-# curve peaks at 512 and every step up costs GiB of compute buffer for no
-# speed (see frank-medium-batch-ubatch). On THIS CUDA 3090 the curve climbs
-# monotonically to 2048: 710 -> 1188 t/s prefill from ubatch 1024 to 2048 on
-# the 35B MoE sweep, 1944 -> 2423 t/s on the Ornith 8k sweep, with decode flat
-# to under 1 t/s on every rung. Ubatch is a prefill knob; decode never moves.
-#
-# The logical batch is free — two independent sweeps agree that at a fixed
-# ubatch, 2048/4096/8192 give identical prefill speed AND identical peak VRAM,
-# because the compute buffers are sized by the ubatch alone. Doubling it costs
-# nothing and buys nothing; it is here because it is the value the pair was
-# measured at.
-#
-# THE RISK IS ALL IN THE UBATCH. The 1024 -> 2048 step cost ~1,244 MiB of peak
-# graph VRAM on the 35B MoE (7,330 -> 8,574 MiB); this 9B hybrid has far
-# smaller compute tensors, so the delta should be well under that, but the
-# predecessor Q5_K_M shape recorded "ubatch 2048 failed the scratch boot" at
-# ~3,571 MiB free. The Q8_0 rung ships UBATCH 1024 (Frank, 2026-10-05: lower
-# ubatch FIRST — the measured 616 MiB step is the first VRAM the Q8 weights
-# claw back, before any pool move). Compute buffers are allocated at graph
-# capture during BOOT, so a failure is loud and fast, not a surprise at the
-# first long prompt. MEASURE nvidia-smi free immediately after the deploy:
-# under ~1.5 GiB, or a boot failure, step the POOL down next (12,288-token
-# increments, ~432 MiB each at the Ornith 36 KiB/token f16 KV rate).
-BATCH_SIZE=4096
-UBATCH_SIZE=1024
-
-# The KV contract (2026-10-05, Frank): f16 K + f16 V — NO KV quantization.
-# The q8_0-V arithmetic was written and OVERRULED. The Q8 weights eat the
-# VRAM that funded f16 V on 2026-10-03, so the remedy ladder is UBATCH FIRST
-# (2048 -> 1024, the measured 616 MiB rung), then POOL STEPS — and
-# guess-and-check on the live card is the accepted method: boot, read
-# nvidia-smi, step down until the floor holds. --flash-attn stays pinned on
-# in the serve script: it is the fast path for f16 KV.
-CACHE_TYPE_K=f16
-CACHE_TYPE_V=f16
-NGL=99

scripts/frank-small.service

@@ -1,19 +0,0 @@
-[Unit]
-Description=llama.cpp MiMo-V2.6-Distill-Qwen-9B Q6_K GGUF (qwen35 hybrid, 2x200704, unquantized f16 KV, mmproj vision) on the RTX 3090 (dankclaw, frank-small)
-After=network-online.target
-Wants=network-online.target
-
-[Service]
-Type=simple
-User=dank
-Group=dank
-# CUDA pins its staging buffers, so the process needs the memlock budget even
-# though the weights themselves are copied into VRAM.
-LimitMEMLOCK=infinity
-Environment="FRANK_SMALL_ENV_FILE=/home/dank/serverAppConfigs/scripts/frank-small.env"
-ExecStart=/home/dank/serverAppConfigs/scripts/serve-frank-small.sh
-Restart=on-failure
-RestartSec=10
-
-[Install]
-WantedBy=multi-user.target

scripts/hfget.sh

@@ -0,0 +1,107 @@
+#!/bin/sh
+# hfget.sh -- fetch a Hugging Face repo with curl and jq, and nothing else.
+#
+# The engine and its tooling do not depend on Python, so `huggingface-cli` is not assumed. The API
+# gives a file listing as JSON and every file has a stable resolve/ URL, which is all a downloader
+# needs.
+#
+#   scripts/hfget.sh <repo-id> [dest-dir] [revision]
+#
+# Set HF_TOKEN for a gated or private repo. Re-running resumes: a file whose local size already
+# matches the listing is skipped and a short one is continued with `curl -C -`, so an interrupted
+# transfer of a 300 GB checkpoint costs only what it had not finished.
+#
+# WEIGHT DUPLICATES ARE SKIPPED. Many repos carry the same tensors several times over -- safetensors
+# beside the older pickle .bin, an ONNX export, a GGUF conversion -- and on a large checkpoint each
+# copy is hundreds of gigabytes of the same numbers. When the listing holds any .safetensors, every
+# other weight format is left behind.
+#
+# HFGET_JOBS concurrent transfers, default 4. These are long sequential reads: a few streams
+# saturate the link and more only makes the destination seek.
+set -e
+
+# ---- child mode: one file, named by "<size><tab><path>" in $1, everything else inherited -------
+# sh cannot export a function, so parallelism is one re-entry of this script per file under
+# `xargs -P` rather than a worker pool.
+if [ -n "$HFGET_CHILD" ]; then
+    size=${1%%	*}; path=${1#*	}
+    out=$HFGET_DEST/$path
+    if [ -f "$out" ]; then
+        cur=$(wc -c < "$out")
+        if [ "$cur" = "$size" ]; then echo "  = $path"; exit 0; fi
+        echo "  + $path (resuming at $cur of $size)"
+    else
+        echo "  > $path"
+    fi
+    mkdir -p "$(dirname "$out")"
+    if [ -n "$HF_TOKEN" ]; then
+        curl -sSL --fail --retry 5 --retry-delay 5 --retry-all-errors -C - \
+             -H "Authorization: Bearer $HF_TOKEN" \
+             -o "$out" "https://huggingface.co/$HFGET_REPO/resolve/$HFGET_REV/$path"
+    else
+        curl -sSL --fail --retry 5 --retry-delay 5 --retry-all-errors -C - \
+             -o "$out" "https://huggingface.co/$HFGET_REPO/resolve/$HFGET_REV/$path"
+    fi
+    got=$(wc -c < "$out")
+    [ "$got" = "$size" ] || { echo "hfget: $path is $got bytes, listing says $size" >&2; exit 1; }
+    exit 0
+fi
+
+# ---- parent ------------------------------------------------------------------------------------
+repo=$1
+[ -n "$repo" ] || { echo "usage: hfget.sh <repo-id> [dest-dir] [revision]" >&2; exit 2; }
+dest=${2:-$HOME/models/$repo}
+rev=${3:-main}
+jobs=${HFGET_JOBS:-4}
+command -v jq >/dev/null || { echo "hfget: jq is required" >&2; exit 2; }
+
+echo "hfget: listing $repo@$rev"
+# THE LISTING IS PAGINATED AND THE PAGE SIZE IS SMALL. A large checkpoint runs to a hundred-odd
+# shards and the API returns fifty entries with a `Link: ...; rel="next"` header. Taking only the
+# first page is the worst possible failure here: every file it did fetch is the right length, so
+# the size check passes and the converter is handed a checkpoint missing two thirds of its
+# tensors. Follow the cursor until there is no next link.
+api="https://huggingface.co/api/models/$repo/tree/$rev?recursive=true&expand=true"
+listing='[]'
+page=0
+while [ -n "$api" ]; do
+    hdr=$(mktemp "${TMPDIR:-/tmp}/hfhdr.XXXXXX")
+    if [ -n "$HF_TOKEN" ]; then
+        body=$(curl -sSL --fail-with-body -D "$hdr" -H "Authorization: Bearer $HF_TOKEN" "$api")
+    else
+        body=$(curl -sSL --fail-with-body -D "$hdr" "$api")
+    fi
+    [ "$(printf '%s' "$body" | jq -r 'type')" = "array" ] || {
+        rm -f "$hdr"
+        echo "hfget: $(printf '%s' "$body" | jq -r '.error // .message // .')" >&2
+        echo "hfget: a gated or private repo needs HF_TOKEN in the environment" >&2; exit 1; }
+    listing=$(printf '%s\n%s' "$listing" "$body" | jq -s 'add')
+    api=$(tr -d '\r' < "$hdr" | sed -n 's/^[Ll]ink: *<\([^>]*\)>; *rel="next".*/\1/p' | head -1)
+    rm -f "$hdr"
+    page=$((page + 1))
+    [ "$page" -gt 200 ] && { echo "hfget: listing did not terminate after 200 pages" >&2; exit 1; }
+done
+echo "hfget: $page page(s) of listing"
+have_st=$(printf '%s' "$listing" | jq '[.[] | select(.path | endswith(".safetensors"))] | length')
+sel='.[] | select(.type == "file")'
+[ "$have_st" -gt 0 ] && sel="$sel"' | select((.path | test("\\.(bin|pth|msgpack|h5|onnx|onnx_data|gguf)$")) | not)'
+
+list=$(mktemp "${TMPDIR:-/tmp}/hfget.XXXXXX")
+trap 'rm -f "$list"' EXIT INT TERM
+printf '%s' "$listing" | jq -r "$sel"' | "\(.size)\t\(.path)"' > "$list"
+
+n=$(wc -l < "$list")
+[ "$n" -gt 0 ] || { echo "hfget: the listing has no files" >&2; exit 1; }
+need=$(awk -F'\t' '{s += $1} END {printf "%.1f", s / 1073741824}' "$list")
+mkdir -p "$dest"
+free=$(df -BG --output=avail "$dest" | tail -1 | tr -dc '0-9')
+echo "hfget: $n file(s), $need GiB -> $dest"
+[ "$have_st" -gt 0 ] && echo "hfget: safetensors present, pickle weight formats skipped"
+echo "hfget: $free GiB free on the destination filesystem"
+awk -v n="$need" -v f="$free" 'BEGIN { if (n > f) { print "hfget: THE DOWNLOAD DOES NOT FIT"; exit 1 } }'
+
+export HFGET_CHILD=1 HFGET_DEST="$dest" HFGET_REPO="$repo" HFGET_REV="$rev"
+self=$(cd "$(dirname "$0")" && pwd)/$(basename "$0")
+tr '\n' '\0' < "$list" | xargs -0 -P "$jobs" -I{} sh "$self" "{}" \
+    || { echo "hfget: incomplete -- re-run to resume" >&2; exit 1; }
+echo "hfget: complete -- $(du -sh "$dest" | cut -f1) in $dest"

scripts/qwen38-froggeric-v22.5.jinja

@@ -0,0 +1,460 @@
+{%- set template_version = "qwen3.8-froggeric-v22.5" %}
+{%- set _tool_format = tool_call_format if tool_call_format is defined else 'xml' %}
+{%- set image_count = namespace(value=0) %}
+{%- set video_count = namespace(value=0) %}
+{%- set add_vision_id = add_vision_id if add_vision_id is defined else false %}
+{%- set enable_thinking = enable_thinking if enable_thinking is defined else true %}
+{%- set auto_disable_thinking_with_tools = auto_disable_thinking_with_tools if auto_disable_thinking_with_tools is defined else false %}
+{%- if preserve_reasoning is defined and preserve_reasoning is not none %}
+    {%- set _preserve_thinking = preserve_reasoning %}
+{%- elif preserve_thinking is defined and preserve_thinking is not none %}
+    {%- set _preserve_thinking = preserve_thinking %}
+{%- else %}
+    {%- set _preserve_thinking = true %}
+{%- endif %}
+{%- set max_tool_arg_chars = max_tool_arg_chars if max_tool_arg_chars is defined else 0 %}
+{%- set max_tool_response_chars = max_tool_response_chars if max_tool_response_chars is defined else 0 %}
+{%- set _default_reasoning_effort = 'medium' %}
+{%- set _has_tools = (tools is defined and tools and tools is iterable and tools is not mapping) %}
+{%- set _effort_raw = (reasoning_effort | string | lower) if reasoning_effort is defined and reasoning_effort is not none else _default_reasoning_effort %}
+{%- set _initial_thinking = enable_thinking %}
+{%- if _effort_raw in ('none', 'off') %}
+    {%- set _initial_thinking = false %}
+    {%- set _initial_effort = 'medium' %}
+{%- elif _effort_raw in ('minimal', 'low') %}
+    {%- set _initial_effort = 'low' %}
+{%- elif _effort_raw in ('high', 'xhigh', 'max', 'ultracode', 'extreme') %}
+    {%- set _initial_effort = 'xhigh' %}
+{%- else %}
+    {%- set _initial_effort = 'medium' %}
+{%- endif %}
+{%- set ns_state = namespace(thinking=_initial_thinking, effort=_initial_effort) %}
+{%- if auto_disable_thinking_with_tools and _has_tools %}
+    {%- set ns_state.thinking = false %}
+{%- endif %}
+{%- for msg in messages %}
+    {%- if msg.role == 'system' or msg.role == 'developer' or msg.role == 'user' %}
+        {%- if msg.content is string %}
+            {%- if '<|think_off|>' in msg.content %}
+                {%- set ns_state.thinking = false %}
+            {%- elif '<|think_on|>' in msg.content %}
+                {%- set ns_state.thinking = true %}
+            {%- elif '<|think_xhigh|>' in msg.content or '<|think_high|>' in msg.content or '<|think_ultracode|>' in msg.content or '<|think_extreme|>' in msg.content or '<|think_max|>' in msg.content %}
+                {%- set ns_state.thinking = true %}
+                {%- set ns_state.effort = 'xhigh' %}
+            {%- elif '<|think_low|>' in msg.content or '<|think_minimal|>' in msg.content %}
+                {%- set ns_state.thinking = true %}
+                {%- set ns_state.effort = 'low' %}
+            {%- elif '<|think_medium|>' in msg.content %}
+                {%- set ns_state.thinking = true %}
+                {%- set ns_state.effort = 'medium' %}
+            {%- endif %}
+        {%- elif msg.content is iterable and msg.content is not mapping %}
+            {%- for item in msg.content %}
+                {%- if item is string %}
+                    {%- set _item_text = item %}
+                {%- elif item is mapping and 'text' in item and item.text is string %}
+                    {%- set _item_text = item.text %}
+                {%- else %}
+                    {%- set _item_text = '' %}
+                {%- endif %}
+                {%- if _item_text %}
+                    {%- if '<|think_off|>' in _item_text %}
+                        {%- set ns_state.thinking = false %}
+                    {%- elif '<|think_on|>' in _item_text %}
+                        {%- set ns_state.thinking = true %}
+                    {%- elif '<|think_xhigh|>' in _item_text or '<|think_high|>' in _item_text or '<|think_ultracode|>' in _item_text or '<|think_extreme|>' in _item_text or '<|think_max|>' in _item_text %}
+                        {%- set ns_state.thinking = true %}
+                        {%- set ns_state.effort = 'xhigh' %}
+                    {%- elif '<|think_low|>' in _item_text or '<|think_minimal|>' in _item_text %}
+                        {%- set ns_state.thinking = true %}
+                        {%- set ns_state.effort = 'low' %}
+                    {%- elif '<|think_medium|>' in _item_text %}
+                        {%- set ns_state.thinking = true %}
+                        {%- set ns_state.effort = 'medium' %}
+                    {%- endif %}
+                {%- endif %}
+            {%- endfor %}
+        {%- endif %}
+    {%- endif %}
+{%- endfor %}
+{%- set reasoning_instructions = '' %}
+{%- if ns_state.thinking %}
+    {%- if ns_state.effort == 'xhigh' %}
+        {%- set reasoning_instructions = 'Reasoning effort is set to xhigh. Please think carefully through the task, validate key assumptions, consider plausible alternatives, and prioritize correctness, consistency, and clarity in the final answer.' %}
+    {%- elif ns_state.effort == 'low' %}
+        {%- set reasoning_instructions = 'Reasoning effort is set to low. Keep your thinking brief and focused, moving directly to the conclusion without unnecessary elaboration.' %}
+    {%- endif %}
+{%- endif %}
+{%- macro render_content(content, do_vision_count, is_system_content=false) %}
+    {%- if content is string %}
+        {{- content }}
+    {%- elif content is iterable and content is not mapping %}
+        {%- for item in content %}
+            {%- if item is mapping %}
+                {%- if item.type == 'image' or 'image' in item or 'image_url' in item %}
+                    {%- if is_system_content %}
+                        {{- raise_exception('System message cannot contain images.') }}
+                    {%- endif %}
+                    {%- if do_vision_count %}
+                        {%- set image_count.value = image_count.value + 1 %}
+                    {%- endif %}
+                    {%- if add_vision_id %}
+                        {{- 'Picture ' ~ image_count.value ~ ': ' }}
+                    {%- endif %}
+                    {{- '<|vision_start|><|image_pad|><|vision_end|>' }}
+                {%- elif item.type == 'video' or 'video' in item or 'video_url' in item %}
+                    {%- if is_system_content %}
+                        {{- raise_exception('System message cannot contain videos.') }}
+                    {%- endif %}
+                    {%- if do_vision_count %}
+                        {%- set video_count.value = video_count.value + 1 %}
+                    {%- endif %}
+                    {%- if add_vision_id %}
+                        {{- 'Video ' ~ video_count.value ~ ': ' }}
+                    {%- endif %}
+                    {{- '<|vision_start|><|video_pad|><|vision_end|>' }}
+                {%- elif 'text' in item %}
+                    {{- item.text }}
+                {%- else %}
+                    {{- raise_exception('Unexpected item type in content.') }}
+                {%- endif %}
+            {%- else %}
+                {{- item | string }}
+            {%- endif %}
+        {%- endfor %}
+    {%- elif content is none or content is undefined %}
+        {{- '' }}
+    {%- else %}
+        {{- raise_exception('Unexpected content type.') }}
+    {%- endif %}
+{%- endmacro %}
+{%- if not messages %}
+    {{- raise_exception('No messages provided.') }}
+{%- endif %}
+{%- set head = namespace(count=0, seen_non_system=false) %}
+{%- for message in messages %}
+    {%- set _is_sys = (message.role == 'system' or message.role == 'developer') %}
+    {%- if _is_sys and not head.seen_non_system %}
+        {%- set head.count = head.count + 1 %}
+    {%- else %}
+        {%- set head.seen_non_system = true %}
+    {%- endif %}
+{%- endfor %}
+{%- set sys_state = namespace(content='') %}
+{%- for message in messages[:head.count] %}
+    {%- set _part = render_content(message.content, false, true) | trim %}
+    {%- if '<|think_off|>' in _part %}{%- set _part = _part.split('<|think_off|>') | join('') | trim %}{%- endif %}
+    {%- if '<|think_on|>' in _part %}{%- set _part = _part.split('<|think_on|>') | join('') | trim %}{%- endif %}
+    {%- if '<|think_xhigh|>' in _part %}{%- set _part = _part.split('<|think_xhigh|>') | join('') | trim %}{%- endif %}
+    {%- if '<|think_high|>' in _part %}{%- set _part = _part.split('<|think_high|>') | join('') | trim %}{%- endif %}
+    {%- if '<|think_ultracode|>' in _part %}{%- set _part = _part.split('<|think_ultracode|>') | join('') | trim %}{%- endif %}
+    {%- if '<|think_extreme|>' in _part %}{%- set _part = _part.split('<|think_extreme|>') | join('') | trim %}{%- endif %}
+    {%- if '<|think_max|>' in _part %}{%- set _part = _part.split('<|think_max|>') | join('') | trim %}{%- endif %}
+    {%- if '<|think_medium|>' in _part %}{%- set _part = _part.split('<|think_medium|>') | join('') | trim %}{%- endif %}
+    {%- if '<|think_low|>' in _part %}{%- set _part = _part.split('<|think_low|>') | join('') | trim %}{%- endif %}
+    {%- if '<|think_minimal|>' in _part %}{%- set _part = _part.split('<|think_minimal|>') | join('') | trim %}{%- endif %}
+    {%- if _part %}
+        {%- if sys_state.content %}
+            {%- set sys_state.content = sys_state.content ~ '\n\n' ~ _part %}
+        {%- else %}
+            {%- set sys_state.content = _part %}
+        {%- endif %}
+    {%- endif %}
+{%- endfor %}
+{%- set _sc = sys_state.content %}
+{%- set _msgs = messages[head.count:] %}
+{%- if _has_tools %}
+    {{- '<|im_start|>system\n' }}
+    {%- if reasoning_instructions %}
+        {{- reasoning_instructions + '\n\n' }}
+    {%- endif %}
+    {{- '# Tools\n\nYou have access to the following functions:\n\n<tools>' }}
+    {%- for tool in tools %}
+        {{- '\n' }}
+        {{- tool | tojson }}
+    {%- endfor %}
+    {{- '\n</tools>' }}
+    {%- if _tool_format == 'json' %}
+        {{- '\n\nIf you choose to call a function ONLY reply in the following format with NO suffix:\n\n' }}
+        {%- if ns_state.thinking %}
+            {{- '<think>\nBrief explanation of tool call\n</think>\n' }}
+        {%- endif %}
+        {{- '<tool_call>\n{"name": "example_function_name", "arguments": {"example_parameter_1": "value_1", "example_parameter_2": "This is the value for the second parameter"}}\n</tool_call>\n\n<IMPORTANT>\nReminder:\n' }}
+        {%- if ns_state.thinking %}
+            {{- '- You can use the <think></think> block to plan your next tool call OR to synthesize data and formulate your final response to the user.\n- ALL explanation and reasoning MUST be placed strictly inside the <think></think> block.\n' }}
+        {%- endif %}
+        {{- '- Function calls MUST follow the specified format: a single JSON object with "name" and "arguments" keys inside <tool_call></tool_call> XML tags.\n' }}
+        {%- if ns_state.thinking %}
+            {{- '- If you choose to call a tool, you MUST output the <tool_call> block IMMEDIATELY after thinking, with NO conversational text before it.\n' }}
+        {%- else %}
+            {{- '- If you choose to call a tool, you MUST output the <tool_call> block IMMEDIATELY, with NO conversational text before it.\n' }}
+        {%- endif %}
+        {{- '- The <tool_call> tag MUST be at the very beginning of a new line, with NO spaces or indentation before it.\n- To call multiple functions, output a separate, completely closed <tool_call></tool_call> block for EACH function. Do NOT nest <tool_call> blocks.\n- If you have all necessary data, provide your final answer directly to the user without any tool call.\n</IMPORTANT>' }}
+    {%- else %}
+        {{- '\n\nIf you choose to call a function ONLY reply in the following format with NO suffix:\n\n' }}
+        {%- if ns_state.thinking %}
+            {{- '<think>\nBrief explanation of tool call\n</think>\n' }}
+        {%- endif %}
+        {{- '<tool_call>\n<function=example_function_name>\n<parameter=example_parameter_1>\nvalue_1\n</parameter>\n<parameter=example_parameter_2>\nThis is the value for the second parameter\nthat can span\nmultiple lines\n</parameter>\n</function>\n</tool_call>\n\n<IMPORTANT>\nReminder:\n' }}
+        {%- if ns_state.thinking %}
+            {{- '- You can use the <think></think> block to plan your next tool call OR to synthesize data and formulate your final response to the user.\n- ALL explanation and reasoning MUST be placed strictly inside the <think></think> block.\n' }}
+        {%- endif %}
+        {{- '- Function calls MUST follow the specified format: an inner <function=...></function> block must be nested within <tool_call></tool_call> XML tags.\n' }}
+        {%- if ns_state.thinking %}
+            {{- '- If you choose to call a tool, you MUST output the <tool_call> block IMMEDIATELY after thinking, with NO conversational text before it.\n' }}
+        {%- else %}
+            {{- '- If you choose to call a tool, you MUST output the <tool_call> block IMMEDIATELY, with NO conversational text before it.\n' }}
+        {%- endif %}
+        {{- '- The <tool_call> and <function> tags MUST be at the very beginning of a new line, with NO spaces or indentation before them.\n- To call multiple functions, output a separate, completely closed <tool_call></tool_call> block for EACH function. Do NOT nest <tool_call> blocks.\n- If you have all necessary data, provide your final answer directly to the user without any tool call.\n</IMPORTANT>' }}
+    {%- endif %}
+    {%- if _sc %}
+        {{- '\n\n' + _sc }}
+    {%- endif %}
+    {{- '<|im_end|>\n' }}
+{%- else %}
+    {%- if _sc %}
+        {{- '<|im_start|>system\n' + (reasoning_instructions + '\n\n' if reasoning_instructions else '') + _sc + '<|im_end|>\n' }}
+    {%- elif reasoning_instructions %}
+        {{- '<|im_start|>system\n' + reasoning_instructions + '<|im_end|>\n' }}
+    {%- endif %}
+{%- endif %}
+{%- set _last_idx = _msgs | length - 1 %}
+{%- set ns = namespace(multi_step_tool=true, last_query_index=_last_idx) %}
+{%- for message in _msgs[::-1] %}
+    {%- set index = (_msgs | length - 1) - loop.index0 %}
+    {%- if ns.multi_step_tool and message.role == 'user' %}
+        {%- set _rc = render_content(message.content, false) | trim %}
+        {%- if not (_rc.startswith('<tool_response>') and _rc.endswith('</tool_response>')) %}
+            {%- set ns.multi_step_tool = false %}
+            {%- set ns.last_query_index = index %}
+        {%- endif %}
+    {%- endif %}
+{%- endfor %}
+{%- if ns.multi_step_tool %}
+    {%- if _last_idx > 50 %}
+        {%- set ns.last_query_index = _last_idx %}
+    {%- else %}
+        {%- set ns.last_query_index = 0 %}
+    {%- endif %}
+{%- endif %}
+{%- set ns2 = namespace(prev_role='', consecutive_failures=0) %}
+{%- for message in _msgs %}
+    {%- set is_system = (message.role == "system" or message.role == "developer") %}
+    {%- set content = render_content(message.content, true, is_system) | trim %}
+    {%- if is_system or message.role == 'user' %}
+        {%- if '<|think_off|>' in content %}{%- set content = content.split('<|think_off|>') | join('') | trim %}{%- endif %}
+        {%- if '<|think_on|>' in content %}{%- set content = content.split('<|think_on|>') | join('') | trim %}{%- endif %}
+        {%- if '<|think_xhigh|>' in content %}{%- set content = content.split('<|think_xhigh|>') | join('') | trim %}{%- endif %}
+        {%- if '<|think_high|>' in content %}{%- set content = content.split('<|think_high|>') | join('') | trim %}{%- endif %}
+        {%- if '<|think_ultracode|>' in content %}{%- set content = content.split('<|think_ultracode|>') | join('') | trim %}{%- endif %}
+        {%- if '<|think_extreme|>' in content %}{%- set content = content.split('<|think_extreme|>') | join('') | trim %}{%- endif %}
+        {%- if '<|think_max|>' in content %}{%- set content = content.split('<|think_max|>') | join('') | trim %}{%- endif %}
+        {%- if '<|think_medium|>' in content %}{%- set content = content.split('<|think_medium|>') | join('') | trim %}{%- endif %}
+        {%- if '<|think_low|>' in content %}{%- set content = content.split('<|think_low|>') | join('') | trim %}{%- endif %}
+        {%- if '<|think_minimal|>' in content %}{%- set content = content.split('<|think_minimal|>') | join('') | trim %}{%- endif %}
+    {%- endif %}
+    {%- if is_system %}
+        {{- '<|im_start|>system\n' + content + '<|im_end|>\n' }}
+    {%- elif message.role == 'user' %}
+        {%- set ns2.consecutive_failures = 0 %}
+        {{- '<|im_start|>user\n' + content + '<|im_end|>\n' }}
+    {%- elif message.role == 'assistant' %}
+        {%- set reasoning_content = '' %}
+        {%- set _explicit_reasoning = '' %}
+        {%- if message.reasoning_content is defined and message.reasoning_content is not none %}
+            {%- if message.reasoning_content is string %}
+                {%- set _explicit_reasoning = message.reasoning_content %}
+            {%- else %}
+                {%- set _explicit_reasoning = message.reasoning_content | string %}
+            {%- endif %}
+        {%- elif message.thinking is defined and message.thinking is not none %}
+            {%- if message.thinking is string %}
+                {%- set _explicit_reasoning = message.thinking %}
+            {%- else %}
+                {%- set _explicit_reasoning = message.thinking | string %}
+            {%- endif %}
+        {%- elif message.reasoning is defined and message.reasoning is not none %}
+            {%- if message.reasoning is string %}
+                {%- set _explicit_reasoning = message.reasoning %}
+            {%- else %}
+                {%- set _explicit_reasoning = message.reasoning | string %}
+            {%- endif %}
+        {%- endif %}
+        {%- if _explicit_reasoning %}
+            {%- set _lead_end = '' %}
+            {%- if content.startswith('<think>') and '</think>' in content %}
+                {%- set _lead_end = '</think>' %}
+            {%- elif content.startswith('<thinking>') and '</thinking>' in content %}
+                {%- set _lead_end = '</thinking>' %}
+            {%- elif content.startswith('</think>') %}
+                {%- set _lead_end = '</think>' %}
+            {%- elif content.startswith('</thinking>') %}
+                {%- set _lead_end = '</thinking>' %}
+            {%- endif %}
+            {%- if _lead_end %}
+                {%- set content = content.split(_lead_end)[-1].lstrip('\n') %}
+            {%- endif %}
+            {%- set reasoning_content = _explicit_reasoning %}
+        {%- else %}
+            {%- set _think_end = '' %}
+            {%- if content.startswith('</think>') %}
+                {%- set _think_end = '</think>' %}
+            {%- elif content.startswith('</thinking>') %}
+                {%- set _think_end = '</thinking>' %}
+            {%- elif '\n</think>' in content %}
+                {%- set _think_end = '\n</think>' %}
+            {%- elif '\n</thinking>' in content %}
+                {%- set _think_end = '\n</thinking>' %}
+            {%- elif '\n</ think>' in content %}
+                {%- set _think_end = '\n</ think>' %}
+            {%- elif '\n</think >' in content %}
+                {%- set _think_end = '\n</think >' %}
+            {%- elif content.startswith('<think>') and '</think>' in content %}
+                {%- set _think_end = '</think>' %}
+            {%- elif content.startswith('<thinking>') and '</thinking>' in content %}
+                {%- set _think_end = '</thinking>' %}
+            {%- endif %}
+            {%- if _think_end %}
+                {%- if 'thinking' in _think_end %}
+                    {%- set _think_start = '<thinking>' %}
+                {%- else %}
+                    {%- set _think_start = '<think>' %}
+                {%- endif %}
+                {%- set reasoning_content = content.split(_think_end)[0].rstrip('\n') %}
+                {%- if _think_start in reasoning_content %}
+                    {%- set reasoning_content = reasoning_content.split(_think_start)[-1].lstrip('\n') %}
+                {%- endif %}
+                {%- set content = content.split(_think_end)[-1].lstrip('\n') %}
+            {%- endif %}
+        {%- endif %}
+        {%- set reasoning_content = reasoning_content | trim %}
+        {%- if (_preserve_thinking or loop.index0 > ns.last_query_index) %}
+            {{- '<|im_start|>assistant\n<think>\n' + reasoning_content + '\n</think>\n\n' + content }}
+        {%- else %}
+            {{- '<|im_start|>assistant\n' + content }}
+        {%- endif %}
+        {%- if message.tool_calls is defined and message.tool_calls and message.tool_calls is iterable and message.tool_calls is not mapping %}
+            {%- for tool_call in message.tool_calls %}
+                {%- if tool_call.function is defined and tool_call.function is not none %}
+                    {%- set tc = tool_call.function %}
+                {%- else %}
+                    {%- set tc = tool_call %}
+                {%- endif %}
+                {%- set tc_name = tc.name if (tc.name is defined and tc.name is not none) else '' %}
+                {%- if _tool_format == 'json' %}
+                    {%- if loop.first %}
+                        {%- if content | trim %}
+                            {{- '\n\n' }}
+                        {%- endif %}
+                    {%- else %}
+                        {{- '\n' }}
+                    {%- endif %}
+                    {%- set _args = '{}' %}
+                    {%- if tc.arguments is defined and tc.arguments is not none %}
+                        {%- if tc.arguments is mapping %}
+                            {%- set _args = tc.arguments | tojson %}
+                        {%- elif tc.arguments is string %}
+                            {%- if tc.arguments %}
+                                {%- set _args = tc.arguments %}
+                            {%- endif %}
+                        {%- else %}
+                            {%- set _args = tc.arguments | tojson %}
+                        {%- endif %}
+                    {%- endif %}
+                    {{- '<tool_call>\n{"name": ' }}{{- tc_name | tojson }}{{- ', "arguments": ' }}{{- _args }}{{- '}\n</tool_call>' }}
+                {%- else %}
+                    {%- if loop.first %}
+                        {%- if content | trim %}
+                            {{- '\n\n<tool_call>\n<function=' + tc_name + '>\n' }}
+                        {%- else %}
+                            {{- '<tool_call>\n<function=' + tc_name + '>\n' }}
+                        {%- endif %}
+                    {%- else %}
+                        {{- '\n<tool_call>\n<function=' + tc_name + '>\n' }}
+                    {%- endif %}
+                    {%- if tc.arguments is defined and tc.arguments is not none %}
+                        {%- if tc.arguments is mapping %}
+                            {%- for args_name, args_value in tc.arguments.items() %}
+                                {{- '<parameter=' + args_name + '>\n' }}
+                                {%- if args_value is string %}
+                                    {%- set _av = args_value %}
+                                {%- else %}
+                                    {%- set _av = args_value | tojson %}
+                                {%- endif %}
+                                {%- if max_tool_arg_chars > 0 and _av | length > max_tool_arg_chars %}
+                                    {{- _av[:max_tool_arg_chars] + '\n[TRUNCATED - original length ' ~ (_av | length | string) ~ ' chars]' }}
+                                {%- else %}
+                                    {{- _av }}
+                                {%- endif %}
+                                {{- '\n</parameter>\n' }}
+                            {%- endfor %}
+                        {%- else %}
+                            {%- if tc.arguments is string %}
+                                {%- set _raw_args = tc.arguments %}
+                            {%- else %}
+                                {%- set _raw_args = tc.arguments | tojson %}
+                            {%- endif %}
+                            {%- if _raw_args %}
+                                {%- if max_tool_arg_chars > 0 and _raw_args | length > max_tool_arg_chars %}
+                                    {{- _raw_args[:max_tool_arg_chars] + '\n[TRUNCATED - original length ' ~ (_raw_args | length | string) ~ ' chars]' }}
+                                {%- else %}
+                                    {{- _raw_args }}
+                                {%- endif %}
+                            {%- endif %}
+                        {%- endif %}
+                    {%- endif %}
+                    {{- '</function>\n</tool_call>' }}
+                {%- endif %}
+            {%- endfor %}
+        {%- endif %}
+        {{- '<|im_end|>\n' }}
+    {%- elif message.role == 'tool' %}
+        {%- set _content_lower = content | lower %}
+        {%- set _content_head = _content_lower[:120] %}
+        {%- set _is_code_or_grep = ('throw new ' in _content_lower or 'throw error' in _content_lower or 'console.error' in _content_lower or 'logger.error' in _content_lower or 'logging.error' in _content_lower or 'import ' in _content_head or 'def ' in _content_head or 'function ' in _content_head) %}
+        {%- set _exit_code_zero = ('exit code: 0' in _content_head or 'process exited with code 0' in _content_head) %}
+        {%- set _error_field_ok = ('"error": null' in _content_head or '"error":null' in _content_head or '"error": false' in _content_head or '"error":false' in _content_head or '"error": ""' in _content_head or '"error":""' in _content_head) %}
+        {%- set _strong_error = (('"error":' in _content_head and not _error_field_ok) or '"status": "error"' in _content_head or '"status":"error"' in _content_head or 'traceback (most recent call last):' in _content_head or 'command not found' in _content_head or 'invalid syntax' in _content_head or 'fatal:' in _content_head or (('exit code: ' in _content_head or 'process exited with code' in _content_head) and not _exit_code_zero) or _content_head.startswith('exception:') or _content_head.startswith('failed to ')) %}
+        {%- set _weak_error = ('error:' in _content_head or 'err!' in _content_head) %}
+        {%- set _weak_suppressed = ('$ ' in _content_head or 'took ' in _content_head or content | length >= 600) %}
+        {%- if not _is_code_or_grep and (_strong_error or (_weak_error and not _weak_suppressed)) %}
+            {%- set ns2.consecutive_failures = ns2.consecutive_failures + 1 %}
+        {%- else %}
+            {%- set ns2.consecutive_failures = 0 %}
+        {%- endif %}
+        {%- if ns2.prev_role != 'tool' %}
+            {{- '<|im_start|>user' }}
+        {%- endif %}
+        {%- set _is_json_payload = (_tool_format == 'json' and content | trim | length > 0 and (content | trim)[:1] in ('{', '[')) %}
+        {%- if not _is_json_payload and max_tool_response_chars > 0 and content | length > max_tool_response_chars %}
+            {%- set content = content[:max_tool_response_chars] + '\n[TRUNCATED - original length ' ~ (content | length | string) ~ ' chars]' %}
+        {%- endif %}
+        {{- '\n<tool_response>\n' + content }}
+        {%- if ns2.consecutive_failures >= 2 %}
+            {{- '\n\n⚠️ SYSTEM WARNING: ' ~ ns2.consecutive_failures ~ ' consecutive tool errors detected. Your previous approach is incorrect. You MUST use a fundamentally different approach or corrected arguments.' }}
+        {%- elif ns2.consecutive_failures == 1 %}
+            {{- '\n\n⚠️ SYSTEM WARNING: The previous tool call returned an error. Diagnose the failure and retry with completely corrected arguments.' }}
+        {%- endif %}
+        {{- '\n</tool_response>' }}
+        {%- if loop.last %}
+            {{- '<|im_end|>\n' }}
+        {%- else %}
+            {%- set _next_role = _msgs[loop.index0 + 1].role %}
+            {%- if _next_role != 'tool' %}
+                {{- '<|im_end|>\n' }}
+            {%- endif %}
+        {%- endif %}
+    {%- else %}
+        {{- '<|im_start|>user\n[' + message.role + ']: ' + content + '<|im_end|>\n' }}
+    {%- endif %}
+    {%- set ns2.prev_role = message.role %}
+{%- endfor %}
+{%- if add_generation_prompt %}
+    {{- '<|im_start|>assistant\n' }}
+    {%- if not ns_state.thinking %}
+        {{- '<think>\n\n</think>\n\n' }}
+    {%- else %}
+        {{- '<think>\n' }}
+    {%- endif %}
+{%- endif %}

scripts/serve-frank-large.sh

@@ -1,30 +1,32 @@
 #!/usr/bin/env bash
-# serve-frank-large.sh — frank-large endpoint: halogen (closed-source
-# C++/HIP engine, image ghcr.io/peonist-ai/halogen, digest-pinned in
-# frank-large.env) serving peonist-ai/halogen-qwen3.8-27b on Dankclaw's
-# Strix Halo iGPU, host :7770. 2026-10-05 cutover from the ROCmFPX
-# llama.cpp shape — revert the swap commit's trio to roll back (payload
-# and fork build stay on disk).
+# serve-frank-large.sh — frank-large endpoint: halogen-flash-server 0.16.2
+# (closed-source engine, image ghcr.io/peonist-ai/halogen-flash-server,
+# digest-pinned in frank-large.env) serving the Quat3rnion abliterated
+# Swift1.5 finetune of peonist's Qwen3.8-Flash-Next v2 checkpoint on
+# Dankclaw's Strix Halo iGPU, host :7770. 2026-10-06 cutover from the
+# halogen 27B-dense shape — revert the swap commit's trio to roll back
+# (the 27B payload stays on disk).
 #
 # The engine is driven by `podman run` DIRECTLY (the fleet's frank-primary
-# pattern): the image entrypoint `halogen-docker` in default "all" mode
-# starts engine + OpenAI frontend in one container. The frontend listens on
-# 8731 inside (HALOGEN_API_PORT default) and is published as host 7770; the
-# engine's own token port (8730, no auth) stays unpublished on the
-# container's loopback. The image ships NO default checkpoint:
-# HALOGEN_CHECKPOINT selects the file inside the /models mount.
+# pattern): the image starts engine + OpenAI frontend in one container.
+# The frontend listens on 8731 inside (HALOGEN_API_PORT default) and is
+# published as host 7770; the engine's own token port (8730, no auth)
+# stays unpublished on the container's loopback. HALOGEN_CHECKPOINT
+# selects the file inside the /models mount; the n-gram lookup table
+# (HALOGEN_NGRAM_TABLE, the card sets it explicitly though the documented
+# fallback is "beside the checkpoint") and the vision tower
+# (HALOGEN_VISION_TOWER) ride in the same directory.
 #
 # Gates mirror the deployer EXACTLY (the 2026-10-01 crash-loop lesson —
 # serve gate and deploy gate disagreeing = green deploy, red unit):
-# tokenizer directory present with every named file a FLAT REGULAR FILE —
-# the model card ships `tokenizer/` as a flat directory and the engine
-# defaults HALOGEN_TOKENIZER to /tokenizer, so a HF cache snapshot's
-# symlinks into blobs/ dangle inside the mount; GPU device nodes present.
+# checkpoint/n-gram/vision byte-exact, tokenizer directory present with
+# every named file a FLAT REGULAR FILE (a HF cache snapshot's symlinks
+# into blobs/ dangle inside the mount), GPU device nodes present.
 #
-# Why containers instead of the bare binary (the old frank-large shape):
-# the payload is .hgn and only halogen reads it; halogen ships as a
-# container; the root unit keeps LimitMEMLOCK=infinity for parity and to
-# outlive the login session, exactly like frank-primary's.
+# Why containers instead of a bare binary: the payload is .hgn and only
+# the flash-server line reads it; it ships as a container; the root unit
+# keeps LimitMEMLOCK=infinity for parity and to outlive the login session,
+# exactly like frank-primary's.
 set -euo pipefail
 
 SCRIPT_DIR="$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)"
@@ -44,11 +46,16 @@ set +a
 : "${FRANK_LARGE_MODEL_DIR:?FRANK_LARGE_MODEL_DIR must name the payload dir}"
 : "${FRANK_LARGE_CHECKPOINT:?FRANK_LARGE_CHECKPOINT must name the .hgn inside it}"
 : "${FRANK_LARGE_CHECKPOINT_BYTES:?FRANK_LARGE_CHECKPOINT_BYTES must pin the size}"
+: "${FRANK_LARGE_NGRAM_TABLE:?FRANK_LARGE_NGRAM_TABLE must name the lookup table}"
+: "${FRANK_LARGE_NGRAM_BYTES:?FRANK_LARGE_NGRAM_BYTES must pin the table size}"
+: "${FRANK_LARGE_VISION_TOWER:?FRANK_LARGE_VISION_TOWER must name the tower}"
+: "${FRANK_LARGE_VISION_BYTES:?FRANK_LARGE_VISION_BYTES must pin the tower size}"
 : "${FRANK_LARGE_TOKENIZER_FILES:?FRANK_LARGE_TOKENIZER_FILES must list the tokenizer dir contents}"
 : "${HALOGEN_MODEL_ID:?HALOGEN_MODEL_ID must be the served alias (frank-large)}"
-: "${HALOGEN_SLOT_CTX:?HALOGEN_SLOT_CTX must be set}"
-: "${HALOGEN_KV_SLOTS:?HALOGEN_KV_SLOTS must be set (1 keeps the drafter on)}"
-: "${HALOGEN_DRAFTER:?HALOGEN_DRAFTER must be set (2 = DFlash2)}"
+: "${HALOGEN_CTX:?HALOGEN_CTX must be set (per-slot window)}"
+: "${HALOGEN_KV_POOL_POSITIONS:?HALOGEN_KV_POOL_POSITIONS must be set}"
+: "${HALOGEN_KV_SLOTS:?HALOGEN_KV_SLOTS must be set}"
+: "${HALOGEN_SPEC_ADAPT:?HALOGEN_SPEC_ADAPT must be set (0 = head always on)}"
 
 # Capture podman's output BEFORE grepping: under `set -o pipefail` the
 # `podman images | grep -q` pattern races (grep exits on first match,
@@ -61,36 +68,41 @@ if ! grep -qF "${FRANK_LARGE_IMAGE#*@}" <<<"$DIGESTS"; then
     exit 1
 fi
 
-# The checkpoint gate is size-exact: a truncated curl exits 0 on some
-# failures and a partial .hgn loads far enough to look alive, so the byte
-# count from the model card is the contract (the fleet fetch law).
-CKPT="$FRANK_LARGE_MODEL_DIR/$FRANK_LARGE_CHECKPOINT"
-if [ ! -f "$CKPT" ]; then
-    echo "frank-large: checkpoint $CKPT missing;" >&2
-    echo "  fetch it: bash scripts/download-frank-large-halogen.sh" >&2
-    exit 1
-fi
-CKPT_SIZE="$(stat -c %s "$CKPT")"
-if [ "$CKPT_SIZE" != "$FRANK_LARGE_CHECKPOINT_BYTES" ]; then
-    echo "frank-large: checkpoint is $CKPT_SIZE bytes, expected $FRANK_LARGE_CHECKPOINT_BYTES (partial download?)" >&2
-    exit 1
-fi
+# Size-exact gates on all three payload files: a truncated curl exits 0 on
+# some failures and a partial .hgn loads far enough to look alive, so the
+# byte counts from the Hub tree API are the contract (fleet fetch law).
+ck() {
+    # ck <env-name> <path> <expected-bytes>
+    if [ ! -f "$2" ]; then
+        echo "frank-large: $1 $2 missing;" >&2
+        echo "  fetch it: bash scripts/download-frank-large-flashnext.sh" >&2
+        exit 1
+    fi
+    local sz
+    sz="$(stat -c %s "$2")"
+    if [ "$sz" != "$3" ]; then
+        echo "frank-large: $1 is $sz bytes, expected $3 (partial download?)" >&2
+        exit 1
+    fi
+}
+ck checkpoint "$FRANK_LARGE_MODEL_DIR/$FRANK_LARGE_CHECKPOINT" "$FRANK_LARGE_CHECKPOINT_BYTES"
+ck n-gram-table "$FRANK_LARGE_MODEL_DIR/$FRANK_LARGE_NGRAM_TABLE" "$FRANK_LARGE_NGRAM_BYTES"
+ck vision-tower "$FRANK_LARGE_MODEL_DIR/$FRANK_LARGE_VISION_TOWER" "$FRANK_LARGE_VISION_BYTES"
 
-# The payload ships the tokenizer as a FLAT DIRECTORY (model card:
-# "tokenizer/ flat tokenizer directory"). A HuggingFace cache snapshot
-# symlinks each file into a sibling blobs/, and those symlinks dangle
-# inside the container mount — so the directory must be real and every
-# named file must be a regular file, not a link.
+# The payload ships the tokenizer as a FLAT DIRECTORY. A HuggingFace cache
+# snapshot symlinks each file into a sibling blobs/, and those symlinks
+# dangle inside the container mount — so the directory must be real and
+# every named file must be a regular file, not a link.
 TOK_DIR="$FRANK_LARGE_MODEL_DIR/tokenizer"
 if [ -L "$TOK_DIR" ] || [ ! -d "$TOK_DIR" ]; then
     echo "frank-large: tokenizer dir $TOK_DIR missing or a symlink (HF cache snapshot?);" >&2
-    echo "  fetch it: bash scripts/download-frank-large-halogen.sh" >&2
+    echo "  fetch it: bash scripts/download-frank-large-flashnext.sh" >&2
     exit 1
 fi
 for TF in $FRANK_LARGE_TOKENIZER_FILES; do
     if [ ! -f "$TOK_DIR/$TF" ] || [ -L "$TOK_DIR/$TF" ]; then
         echo "frank-large: tokenizer file $TOK_DIR/$TF missing or a symlink;" >&2
-        echo "  fetch it: bash scripts/download-frank-large-halogen.sh" >&2
+        echo "  fetch it: bash scripts/download-frank-large-flashnext.sh" >&2
         exit 1
     fi
 done
@@ -103,23 +115,55 @@ if [ ! -e "$DEV_KFD" ] || [ ! -d "$DEV_DRI" ]; then
     echo "frank-large: $DEV_KFD or $DEV_DRI missing — no gfx1151 to serve on" >&2
     exit 1
 fi
+# Host-RAM preflight — the engine's pin gate, checked BEFORE podman run.
+# The engine refuses to pin when MemAvailable is below the checkpoint plus
+# a 16 GiB floor: pinned pages cannot be swapped, and under the floor the
+# n-gram lookup pages in from disk on every long prompt. A RAM-short host
+# (a co-tenant spike, a sibling engine dying) crash-loops the unit — each
+# boot dies in ~5 s at the pin gate — until the RAM clears. The 2026-10-06
+# cutover hit this: 8 boots, MemAvailable 74.77 GiB vs the 78.11 GiB needed,
+# self-resolved in ~2 min when the dying 27B engine freed RAM. Check here and
+# WAIT for the RAM to clear, so a transient dip boots once instead of
+# crash-looping; if it never clears, refuse loudly.
+MEMINFO="${FRANK_LARGE_MEMINFO:-/proc/meminfo}"
+PIN_KB=$(( FRANK_LARGE_CHECKPOINT_BYTES / 1024 ))
+NEED_KB=$(( PIN_KB + 16 * 1048576 ))
+ram_avail_kb() { awk '/^MemAvailable:/{print $2; exit}' "$MEMINFO"; }
+MEMAVAIL_KB="$(ram_avail_kb)" || MEMAVAIL_KB=""
+if [ -n "$MEMAVAIL_KB" ] && [ "$MEMAVAIL_KB" -lt "$NEED_KB" ]; then
+    WAIT_S="${FRANK_LARGE_RAM_WAIT:-600}"
+    GIB() { awk -v kb="$1" 'BEGIN{printf "%.2f", kb/1048576}'; }
+    echo "frank-large: host RAM below the pin gate — MemAvailable $(GIB "$MEMAVAIL_KB") GiB < needed $(GIB "$NEED_KB") GiB ($(GIB "$PIN_KB") GiB checkpoint + 16.00 GiB floor); waiting up to ${WAIT_S}s for host RAM to clear" >&2
+    deadline=$(( $(date +%s) + WAIT_S ))
+    while [ "$(date +%s)" -lt "$deadline" ]; do
+        sleep 5
+        MEMAVAIL_KB="$(ram_avail_kb)" || MEMAVAIL_KB=""
+        if [ -n "$MEMAVAIL_KB" ] && [ "$MEMAVAIL_KB" -ge "$NEED_KB" ]; then
+            break
+        fi
+    done
+    if [ -z "$MEMAVAIL_KB" ] || [ "$MEMAVAIL_KB" -lt "$NEED_KB" ]; then
+        echo "frank-large: MemAvailable $(GIB "${MEMAVAIL_KB:-0}") GiB after ${WAIT_S}s; refusing to start (the engine would refuse the pin)" >&2
+        exit 1
+    fi
+    echo "frank-large: MemAvailable cleared to $(GIB "$MEMAVAIL_KB") GiB; starting" >&2
+fi
 for CTR in ${FRANK_LARGE_LEGACY_CONTAINERS:-} "$FRANK_LARGE_CONTAINER"; do
     podman rm -f "$CTR" >/dev/null 2>&1 || true
 done
 
 LOG="${FRANK_LARGE_LOG:-/home/dank/logs/frank_large.log}"
 mkdir -p "$(dirname "$LOG")"
-# for the udev-granted supplementary group, seccomp unconfined for the ROCm
-# ioctl surface, and unbounded memlock (the unit's LimitMEMLOCK=infinity is
-# inherited, so runc never has to RAISE it). NO --ulimit nofile: Dankclaw's
-# hard limit is 524288, rootless runc cannot raise a hard limit and dies
-# with "error setting rlimit type 7: operation not permitted" (measured
-# 2026-10-05 cutover), and 524288 fds is already ~100x what the engine
-# wants; the model card and both sibling serve scripts pass no nofile
-# either. No comment lines inside the backslash
-# chain — a mid-chain # terminates the continuation (the 2026-10-05
-# exit-127 crash loop on the sibling serve script).
-podman run -d --name "$FRANK_LARGE_CONTAINER" \
+# --ipc=host for the udev-granted supplementary group, seccomp unconfined
+# for the ROCm ioctl surface, and unbounded memlock (the unit's
+# LimitMEMLOCK=infinity is inherited, so runc never has to RAISE it). NO
+# --ulimit nofile: Dankclaw's hard limit is 524288, rootless runc cannot
+# raise a hard limit and dies with "error setting rlimit type 7:
+# operation not permitted" (measured 2026-10-05 cutover); the model card
+# and both sibling serve scripts pass no nofile either. No comment lines
+# inside the backslash chain — a mid-chain # terminates the continuation
+# (the 2026-10-05 exit-127 crash loop on the sibling serve script).
+podman run --name "$FRANK_LARGE_CONTAINER" \
   --publish "$FRANK_LARGE_PUBLISH" \
   --ipc=host \
   --ulimit memlock=-1:-1 \
@@ -128,19 +172,29 @@ podman run -d --name "$FRANK_LARGE_CONTAINER" \
   --security-opt seccomp=unconfined \
   -e HALOGEN_MODEL_ID="$HALOGEN_MODEL_ID" \
   -e HALOGEN_CHECKPOINT="/models/$FRANK_LARGE_CHECKPOINT" \
-  -e HALOGEN_SLOT_CTX="$HALOGEN_SLOT_CTX" \
-  -e HALOGEN_KV_SLOTS="${HALOGEN_KV_SLOTS:-1}" \
-  -e HALOGEN_DRAFTER="$HALOGEN_DRAFTER" \
+  -e HALOGEN_NGRAM_TABLE="/models/$FRANK_LARGE_NGRAM_TABLE" \
+  -e HALOGEN_VISION_TOWER="/models/$FRANK_LARGE_VISION_TOWER" \
+  -e HALOGEN_TOKENIZER=/tokenizer \
+  -e HALOGEN_CTX="$HALOGEN_CTX" \
+  -e HALOGEN_KV_POOL_POSITIONS="$HALOGEN_KV_POOL_POSITIONS" \
+  -e HALOGEN_KV_SLOTS="$HALOGEN_KV_SLOTS" \
+  -e HALOGEN_HOST_RESERVE_GIB="${HALOGEN_HOST_RESERVE_GIB:-20}" \
+  -e HALOGEN_SPEC_ADAPT="$HALOGEN_SPEC_ADAPT" \
+  -e HALOGEN_MAX_TOK="${HALOGEN_MAX_TOK:-8192}" \
   -e HALOGEN_MAX_TOKENS_CAP="${HALOGEN_MAX_TOKENS_CAP:-65536}" \
   -e HALOGEN_QUEUE_TIMEOUT="${HALOGEN_QUEUE_TIMEOUT:-7200}" \
-  ${HALOGEN_CACHE_MB:+-e HALOGEN_CACHE_MB="$HALOGEN_CACHE_MB"} \
+  -e HALOGEN_TEMPERATURE="${HALOGEN_TEMPERATURE:-1.0}" \
+  -e HALOGEN_TOP_P="${HALOGEN_TOP_P:-0.95}" \
+  -e HALOGEN_TOP_K="${HALOGEN_TOP_K:-20}" \
+  ${HALOGEN_WEIGHTS_LOCK:+-e HALOGEN_WEIGHTS_LOCK="$HALOGEN_WEIGHTS_LOCK"} \
   -v "$FRANK_LARGE_MODEL_DIR:/models:ro" \
   -v "$FRANK_LARGE_MODEL_DIR/tokenizer:/tokenizer:ro" \
   "$FRANK_LARGE_IMAGE" \
   > "$LOG" 2>&1
 
-# This script's lifetime IS the container's: podman wait returns the
-# container's exit code, so a crash is a nonzero ExecStart and the unit's
-# Restart=always is real. systemd stop goes through the unit's ExecStop
-# (podman wait alone cannot stop the child).
-exec podman wait "$FRANK_LARGE_CONTAINER"
+# FOREGROUND `podman run`: the engine's stdout/stderr are this script's
+# output, so the boot ledger (kv pool, prompt cache, MTP head, any clamp)
+# lands in $LOG as documented. podman run returns the container's exit
+# code, so a crash is a nonzero ExecStart and the unit's Restart=always
+# stays real; systemd stop goes through the unit's ExecStop and the run
+# client forwards SIGTERM to the engine, which drains on it.

scripts/serve-frank-primary.sh

@@ -1,65 +1,33 @@
 #!/usr/bin/env bash
-# serve-frank-primary.sh — frank-primary endpoint (tcclaviger RDNA4 vLLM fork,
-# orcarouter Qwen3.8-Flash-Next-Uncensored-FP8, 2x R9700,
-# dankserver :7770; 2026-09-29 cutover from the R9V radiance fork — git tag
-# frank-primary-r9v-q8-20260929 is the pre-cutover state).
+# serve-frank-primary.sh — frank-primary endpoint: radiance (StillDeadcode,
+# Apache-2.0) serving the OFFICIAL Qwen/Qwen3.8-Flash-Next-FP8 checkpoint at
+# NATIVE block-FP8 precision from a rad-convert container
+# (/home/dank/models/rad/qwen4exp-fp8-native.rad), on BOTH R9700s (TP=2,
+# wht6 wire), dankserver :7770. 2026-10-05 cutover from FreeToken (PR #587,
+# one card; git tag frank-primary-freetoken-pr587-20261005 is the
+# pre-cutover state).
 #
-# The engine is driven by `podman run` DIRECTLY: the image's entrypoint execs
-# `vllm serve "$@"`, so the run line below IS the engine argv. This replaces
-# the R9V launcher/shim machinery (that fork needed a docker->podman shim and
-# a 104-key on-host setup state; this stack is fully expressed by flags, and
-# every flag here was qualified by boot probe on 2026-09-29 — see
-# frank-primary.env for the qualification record).
-#
-# Shape (2026-10-03 shape F, on the 2026-10-02 shape E base): MAXLEN 200704 x 3
-# slots, fp8 KV, KV_BYTES
-# 3,150,000,000 PER RANK — the POOL, not the slot count, caps resident
-# conversations at ~2 full-length (measured: 398,658 tokens = 1.99x);
-# shorter requests use the spare blocks. --max-num-seqs stays 3 because SEQS=2
-# CANNOT boot this model
-# (torch.fx ConstraintViolationError on query_start_loc, measured at both
-# MNBT 1024 and 2048 — the 2x2 lives in frank-primary.env). Speculative decode
-# is OFF today by ENV KNOB, not by measurement: FRANK_PRIMARY_SPEC / _COMPCONFIG
-# / _GEN_CONFIG hold the whole JSON value and unset means no flag. The 2026-10-02
-# retirement of MTP compared the wrong pair — see frank-primary.env.
-# --max-num-batched-tokens 2048 (Frank's call 2026-10-03; arena-free because the
-# registry prices overhead flat up to its nbt 4096 anchor), MoE gather geometry
-# R4D_LRU_GATHER_CHUNKS=32 / R4D_LRU_GATHER_LANES=64 (+18.7% decode, grows
-# with depth), prefix caching OFF 2026-10-05 (the fork's QFN align-mode GDN
-# state fault — the evidence block is in frank-primary.env), expert prefetch
-# OFF (36% precision measured),
-# TunableOp off. Shape E adds --expert-precise-memory (fitted 5.25 GiB overhead
-# instead of the 7.60 GiB default -> expert arena 17.22 GiB / 7,522 slots, +16%)
-# and GPU_MAX_HW_QUEUES=2; shape F pins the plan's CUDA-graph term to the
-# measured capture cost (VLLM_CUDAGRAPH_MEM_BASE_MB / _PER_GRAPH_MB, container
-# env — the analytic default charges 0.55 GiB where capture took 0.31 GiB, so
-# the 0.24 GiB difference goes to the arena) and raises util 0.98 -> 0.982,
-# Frank's cap. The native-262144 and fp8+MTP4 shapes are retired to
-# git tag frank-primary-fp8-mtp-20261001.
-#
-# THE COMPILE CACHE IS MOUNTED, and that is a boot-cost fix, not a tuning knob:
-# the image ships no /root/.cache at all, our first boot writes 1.1 GB of vLLM
-# AOT/torch_compile artifacts plus 229 MB of comgr JIT kernels into the
-# container's ephemeral layer, and `podman rm -f` deletes them — so every
-# deploy/restart/reboot re-pays `torch.compile took 181.20 s in total` (173.19 s
-# of it one graph). Mounting FRANK_PRIMARY_CACHE_DIR at /root/.cache makes all
-# later boots load the compiled artifact instead, and hides nothing. PROVEN
-# LIVE 2026-10-02 on consecutive boots of this shape: cold `torch.compile took
-# 188.26 s`, warm `Directly load the compiled graph(s) for compile range
-# (1, 1024) from the cache, took 2.143 s` / `torch.compile took 3.97 s in
-# total` / `init engine … took 41.13 s (compilation: 3.98 s)`.
+# The run line below IS the engine argv: the image's entrypoint is the
+# engine and its tools sit beside it on PATH. Unlike every predecessor
+# there is no per-boot VRAM planner to second-guess: the boot ledger in the
+# log states every budget before anything is allocated (weights vs KV
+# split, the expert host pool's coverage of the worst case, the PLE pin,
+# the P2P answer per pair) — read it after every resize. First-boot shape:
+# 2 decode slots (Frank's call), 40 GiB/rank pinned expert pool, fp8 KV,
+# MTP depth 3, prefix cache host 4 GiB + disk 64 GiB, PLE pinned in RAM.
 #
 # The root unit (scripts/frank-primary.service, User=dank,
-# LimitMEMLOCK=infinity) grants the pin budget for the tiered-expert host
-# slabs (~20.5 GiB/rank of pinned anonymous memory). A plain shell session
-# also works, but the unit is the fleet-parity boot-persistent shape.
+# LimitMEMLOCK=infinity) grants the pin budget: 2 x 40 GiB expert pools +
+# the 47.7 GiB PLE table + 4 GiB prefix host tier — ~132 GiB of pinned
+# pages, which is ALSO why this script refuses to start when ttm's
+# pages_limit/pages_max cap is under FRANK_PRIMARY_TTM_MIN_MIB (see below).
 set -euo pipefail
 
 SCRIPT_DIR="$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)"
 ENV_FILE="${FRANK_PRIMARY_ENV_FILE:-$SCRIPT_DIR/frank-primary.env}"
 
 if [ ! -f "$ENV_FILE" ]; then
-    echo "serve-frank-primary: env file missing: $ENV_FILE" >&2
+    echo "ERROR: frank-primary env file absent: $ENV_FILE" >&2
     exit 1
 fi
 set -a
@@ -67,55 +35,119 @@ set -a
 source "$ENV_FILE"
 set +a
 
-: "${FRANK_PRIMARY_IMAGE:?FRANK_PRIMARY_IMAGE must name the digest-pinned image}"
-: "${FRANK_PRIMARY_MODEL_DIR:?FRANK_PRIMARY_MODEL_DIR must name the FP8 payload}"
-: "${FRANK_PRIMARY_MODEL_SHARDS:?FRANK_PRIMARY_MODEL_SHARDS must give the shard count}"
+: "${FRANK_PRIMARY_IMAGE:?FRANK_PRIMARY_IMAGE must name the radiance image}"
+: "${FRANK_PRIMARY_MODEL_DIR:?FRANK_PRIMARY_MODEL_DIR must name the .rad dir}"
+: "${FRANK_PRIMARY_MODEL_FILE:?FRANK_PRIMARY_MODEL_FILE must name the .rad file}"
+: "${FRANK_PRIMARY_STATE_DIR:?FRANK_PRIMARY_STATE_DIR must name the /data state dir}"
 : "${FRANK_PRIMARY_CONTAINER:?FRANK_PRIMARY_CONTAINER must name the container}"
 : "${FRANK_PRIMARY_SERVED_NAME:?FRANK_PRIMARY_SERVED_NAME must name the alias}"
-: "${FRANK_PRIMARY_PUBLISH:?FRANK_PRIMARY_PUBLISH must be host:container}"
-: "${FRANK_PRIMARY_MAXLEN:?FRANK_PRIMARY_MAXLEN must be set}"
-: "${FRANK_PRIMARY_SEQS:?FRANK_PRIMARY_SEQS must be set}"
-: "${FRANK_PRIMARY_KV_BYTES:?FRANK_PRIMARY_KV_BYTES must be set (PER RANK)}"
-: "${FRANK_PRIMARY_KV_DTYPE:?FRANK_PRIMARY_KV_DTYPE must be set (auto = bf16 KV)}"
-: "${FRANK_PRIMARY_REASONING_PARSER:?FRANK_PRIMARY_REASONING_PARSER must be set (qwen3)}"
-: "${FRANK_PRIMARY_TOOL_PARSER:?FRANK_PRIMARY_TOOL_PARSER must be set (qwen3_xml)}"
-: "${FRANK_PRIMARY_CHAT_TEMPLATE_KWARGS:?FRANK_PRIMARY_CHAT_TEMPLATE_KWARGS must be set}"
-# Capture podman's output BEFORE grepping: under `set -o pipefail`, the
-# classic `podman images | grep -q` pattern races — grep exits on its first
-# match, podman dies SIGPIPE (141) writing the remaining lines, and pipefail
-# turns the 141 into "no match" (the 2026-09-30 deploy refusal).
-DIGESTS="$(podman images --format '{{.RepoDigests}}')"
-if ! grep -qF "${FRANK_PRIMARY_IMAGE#*@}" <<<"$DIGESTS"; then
-    echo "serve-frank-primary: pinned image absent from rootless storage: $FRANK_PRIMARY_IMAGE" >&2
-    echo "  fetch it: podman pull $FRANK_PRIMARY_IMAGE" >&2
+: "${FRANK_PRIMARY_PORT:?FRANK_PRIMARY_PORT must be the port}"
+: "${FRANK_PRIMARY_TEMPLATE:?FRANK_PRIMARY_TEMPLATE must name the chat template}"
+: "${FRANK_PRIMARY_TP:?FRANK_PRIMARY_TP must be set}"
+: "${FRANK_PRIMARY_TP_WIRE:?FRANK_PRIMARY_TP_WIRE must be set}"
+: "${FRANK_PRIMARY_MAX_SEQS:?FRANK_PRIMARY_MAX_SEQS must be set}"
+: "${FRANK_PRIMARY_MAX_MODEL_LEN:?FRANK_PRIMARY_MAX_MODEL_LEN must be set}"
+: "${FRANK_PRIMARY_PLACEMENT:?FRANK_PRIMARY_PLACEMENT must be set}"
+: "${FRANK_PRIMARY_HOST_POOL_MIB:?FRANK_PRIMARY_HOST_POOL_MIB must be set}"
+: "${FRANK_PRIMARY_GPU_HEADROOM_MIB:?FRANK_PRIMARY_GPU_HEADROOM_MIB must be set}"
+: "${FRANK_PRIMARY_EXPERT_VS_CACHE:?FRANK_PRIMARY_EXPERT_VS_CACHE must be set}"
+: "${FRANK_PRIMARY_KV_DTYPE:?FRANK_PRIMARY_KV_DTYPE must be set}"
+: "${FRANK_PRIMARY_SPEC_TOKENS:?FRANK_PRIMARY_SPEC_TOKENS must be set}"
+: "${FRANK_PRIMARY_MAX_BATCHED_TOKENS:?FRANK_PRIMARY_MAX_BATCHED_TOKENS must be set}"
+: "${FRANK_PRIMARY_PREFIX_HOST_MIB:?FRANK_PRIMARY_PREFIX_HOST_MIB must be set}"
+: "${FRANK_PRIMARY_PREFIX_DISK_MIB:?FRANK_PRIMARY_PREFIX_DISK_MIB must be set}"
+: "${FRANK_PRIMARY_NGRAM_PLACEMENT:?FRANK_PRIMARY_NGRAM_PLACEMENT must be set}"
+: "${FRANK_PRIMARY_TTM_MIN_MIB:?FRANK_PRIMARY_TTM_MIN_MIB must be set}"
+: "${FRANK_PRIMARY_RAD_MIN_MIB:?FRANK_PRIMARY_RAD_MIN_MIB must be set}"
+
+# The image gate is the digest-pinned ref itself: a name@sha256 ref exists
+# in storage only if THAT manifest was pulled — a retagged look-alike or a
+# newer latest does not satisfy it. `podman image exists` takes a
+# name@digest ref in one command — no pipefail/grep race (the 2026-09-30
+# lesson). Runs as dank: the unit's process uses dank's ROOTLESS podman
+# storage (root unit, User=dank — the fleet's frank-primary shape).
+if ! podman image exists "$FRANK_PRIMARY_IMAGE"; then
+    echo "ERROR: radiance image absent: $FRANK_PRIMARY_IMAGE" >&2
+    echo "frank-primary: podman pull docker://stilldeadcode/radiance:latest (as dank)" >&2
     exit 1
 fi
 
-if [ ! -f "$FRANK_PRIMARY_MODEL_DIR/config.json" ]; then
-    echo "serve-frank-primary: payload config missing: $FRANK_PRIMARY_MODEL_DIR/config.json" >&2
+# The payload gate: the .rad container file, sized. A truncated convert
+# would otherwise buy a guaranteed long-timeout boot instead of this fast
+# failure. The staging dir / raw checkpoint are NOT this script's concern
+# (scripts/build-frank-primary-rad.sh built the file; its rad-info output
+# is the record).
+RAD="$FRANK_PRIMARY_MODEL_DIR/$FRANK_PRIMARY_MODEL_FILE"
+if [ ! -f "$RAD" ]; then
+    echo "ERROR: .rad container missing: $RAD" >&2
+    echo "frank-primary: run scripts/build-frank-primary-rad.sh" >&2
     exit 1
 fi
-SHARDS="$(find "$FRANK_PRIMARY_MODEL_DIR" -maxdepth 1 -name 'model-*-of-*.safetensors' | wc -l)"
-if [ "$SHARDS" != "$FRANK_PRIMARY_MODEL_SHARDS" ]; then
-    echo "serve-frank-primary: payload shard count wrong: $FRANK_PRIMARY_MODEL_DIR" >&2
-    echo "  found $SHARDS safetensors, want $FRANK_PRIMARY_MODEL_SHARDS" >&2
-    echo "  re-fetch: scripts/download-frank-primary.sh" >&2
+RAD_MIB=$(( $(stat -c %s "$RAD") / 1048576 ))
+if [ "$RAD_MIB" -lt "$FRANK_PRIMARY_RAD_MIN_MIB" ]; then
+    echo "ERROR: .rad container too small: $RAD is ${RAD_MIB} MiB, want >= $FRANK_PRIMARY_RAD_MIN_MIB" >&2
+    echo "frank-primary: the convert was truncated — rebuild it" >&2
     exit 1
 fi
-if [ ! -f "$FRANK_PRIMARY_MODEL_DIR/model.safetensors.index.json" ]; then
-    echo "serve-frank-primary: weight-map index missing: $FRANK_PRIMARY_MODEL_DIR/model.safetensors.index.json" >&2
+
+if [ ! -f "$FRANK_PRIMARY_TEMPLATE" ]; then
+    echo "ERROR: chat template missing: $FRANK_PRIMARY_TEMPLATE" >&2
     exit 1
 fi
 
+mkdir -p "$FRANK_PRIMARY_STATE_DIR/kvcache/frank-primary"
+# The engine's create is NOT recursive: with only the state dir mounted, its
+# "cannot create /data/kvcache/frank-primary: No such file or directory"
+# left the SSD prefix tier off (measured 2026-10-06 boot ledger). Pre-create
+# the full path on the host so the tier opens on first boot.
+
+mkdir -p "$FRANK_PRIMARY_STATE_DIR/tune"
+# rad-tune's kernel-selection cache (see the runbook's "Tuning levers"):
+# the engine's selector reads $RADIANCE_HOME/tune/<machine>.tune at boot;
+# $RADIANCE_HOME defaults to /opt/radiance, which lives in the image, so
+# the cache must ride a host mount or it dies with the --rm container
+# that wrote it. The dir is pre-created for the same reason as kvcache
+# (the engine's create is not recursive).
+
 if [ ! -e /dev/kfd ] || [ ! -d /dev/dri ]; then
-    echo "serve-frank-primary: GPU device nodes absent (/dev/kfd, /dev/dri)" >&2
+    echo "ERROR: GPU device nodes absent (/dev/kfd, /dev/dri)" >&2
+    exit 1
+fi
+
+# TTM preflight: the pinned ledger (~2 x 40 GiB pools + 47.7 GiB PLE + 4 GiB
+# prefix host = ~132 GiB) rides TTM's shared pages, capped by ttm's
+# pages_limit (the current kernel exposes NO pages_max; the cap function
+# takes the min over the knobs that exist). Below FRANK_PRIMARY_TTM_MIN_MIB
+# the load dies mid-pin with an allocator error instead of this named
+# refusal — raise the knob first (runtime sysfs write; kernel cmdline
+# fallback + one reboot only if the file is read-only), see the runbook.
+# RAISED 2026-10-05 pre-cutover to 52,428,800 pages = 200 GiB.
+ttm_cap_mib() {
+    # min over the knobs THIS kernel exposes — the current kernel has
+    # pages_limit only (no pages_max); a knob that is absent is skipped,
+    # not a zero (a zero would refuse a healthy cap).
+    local v cur=""
+    for p in /sys/module/ttm/parameters/pages_limit /sys/module/ttm/parameters/pages_max; do
+        [ -r "$p" ] || continue
+        v="$(cat "$p" 2>/dev/null)" || continue
+        [ -n "$v" ] || continue
+        v=$(( v * 4 / 1024 ))
+        if [ "$v" -lt "${cur:-999999999}" ]; then cur=$v; fi
+    done
+    echo "${cur:-0}"
+}
+TTM_CAP="$(ttm_cap_mib)"
+if [ "$TTM_CAP" -lt "$FRANK_PRIMARY_TTM_MIN_MIB" ]; then
+    echo "ERROR: ttm pages cap is ${TTM_CAP} MiB, the pinned ledger needs >= $FRANK_PRIMARY_TTM_MIN_MIB" >&2
+    echo "frank-primary: raise /sys/module/ttm/parameters/pages_limit (and pages_max if the kernel exposes one) to >= $FRANK_PRIMARY_TTM_MIN_MIB in 4 KiB pages" >&2
     exit 1
 fi
 
 # A crashed/killed unit can leave the container name registered (systemd
 # SIGKILLs conmon before cleanup runs). Clear OUR name plus the legacy
-# identities (the retired R9V container and the ad hoc experiment container
-# both hold these GPUs); all idempotent.
+# identities (the FreeToken-era frank-primary-qwen, the retired R9V
+# container and the ad hoc experiment containers all hold this GPU); all
+# idempotent.
 for CTR in $FRANK_PRIMARY_LEGACY_CONTAINERS "$FRANK_PRIMARY_CONTAINER"; do
     podman rm -f "$CTR" >/dev/null 2>&1 || true
 done
@@ -123,74 +155,50 @@ done
 LOG=/home/dank/logs/frank_primary.log
 mkdir -p "$(dirname "$LOG")"
 
-# Persistent compile cache (see the header). Create it before the mount so even
-# the first, cold boot has a target; unset -> no mount, exactly the old
-# ephemeral behaviour. It carries /root/.cache/vllm (AOT + torch_compile) and
-# /root/.cache/comgr (HIP kernel JIT).
-CACHE_DIR="${FRANK_PRIMARY_CACHE_DIR:-}"
-if [ -n "$CACHE_DIR" ]; then
-    mkdir -p "$CACHE_DIR"
-fi
-
-# --device /dev/dri is the WHOLE DIRECTORY (individual renderD nodes enumerate
-# without the card nodes and HIP refuses contexts — "No CUDA GPUs are
-# available" while device_count() says 2). --ipc=host for the engine's shared
-# memory; memlock unbounded for the pinned host slabs.
-# The entrypoint already execs `vllm serve "$@"`, so no `serve` here.
-# Prefix caching: ON — the shipped contract. FRANK_PRIMARY_NO_PREFIX_-
-# CACHING=1 flips to the fork-named mitigation (--no-enable-prefix-caching)
-# for the QFN align-mode GDN state fault; it was tried 2026-10-05 and
-# REVERTED the same day (live conversations still degraded at ~40k with
-# caching off and zero faults — see the evidence block in
-# frank-primary.env). NO comment lines inside the backslash chain below —
-# a comment there terminates the continuation and the next line runs as
-# its own command (the 2026-10-05 exit-127 crash loop).
-# The ${VAR:-word} / ${VAR:+word} double-expansion for the two directions is
-# AMBIGUOUS bash (${VAR:--flag} parses as a substring offset when set —
-# measured 2026-10-05: rendered "--no-enable-prefix-caching1"), so the flag
-# is branched explicitly: exactly one of the two reaches the engine.
-if [ -n "${FRANK_PRIMARY_NO_PREFIX_CACHING:-}" ]; then
-    FRANK_PRIMARY_PREFIX_CACHE_FLAG=--no-enable-prefix-caching
-else
-    FRANK_PRIMARY_PREFIX_CACHE_FLAG=--enable-prefix-caching
-fi
+# Publisher-vetted container contract (docs/DOCKER.md, compose/common.yaml):
+# --network host (the server binds the host's addresses directly, no NAT),
+# --init (the engine releases the cards on SIGTERM), seccomp=unconfined
+# (Docker/podman's default profile refuses io_uring, which the n-gram
+# table's reads go through, and the NUMA binding calls the ROCm runtime
+# makes for host memory), memlock unbounded (the pinned ledger — legal
+# because the root unit grants LimitMEMLOCK=infinity to this dank process),
+# /dev/kfd + /dev/dri (render/video groups are moot for a root container).
+# The template rides a single-file mount; the .rad dir is ro; /data holds
+# the prefix-cache disk tier.
 podman run -d --name "$FRANK_PRIMARY_CONTAINER" \
-  --publish "$FRANK_PRIMARY_PUBLISH" \
-  --ipc=host --ulimit memlock=-1:-1 \
-  --cap-add SYS_PTRACE \
-  --device /dev/kfd --device /dev/dri --device /dev/mem \
-  -e HIP_VISIBLE_DEVICES=0,1 -e VLLM_ROCM_USE_AITER=0 \
-  ${FRANK_PRIMARY_GPU_MAX_HW_QUEUES:+-e GPU_MAX_HW_QUEUES="$FRANK_PRIMARY_GPU_MAX_HW_QUEUES"} \
-  ${FRANK_PRIMARY_GATHER_CHUNKS:+-e R4D_LRU_GATHER_CHUNKS="$FRANK_PRIMARY_GATHER_CHUNKS"} \
-  ${FRANK_PRIMARY_GATHER_LANES:+-e R4D_LRU_GATHER_LANES="$FRANK_PRIMARY_GATHER_LANES"} \
-  ${FRANK_PRIMARY_CUDAGRAPH_BASE_MB:+-e VLLM_CUDAGRAPH_MEM_BASE_MB="$FRANK_PRIMARY_CUDAGRAPH_BASE_MB"} \
-  ${FRANK_PRIMARY_CUDAGRAPH_PER_GRAPH_MB:+-e VLLM_CUDAGRAPH_MEM_PER_GRAPH_MB="$FRANK_PRIMARY_CUDAGRAPH_PER_GRAPH_MB"} \
-  ${CACHE_DIR:+-v "$CACHE_DIR":/root/.cache} \
+  --network host \
+  --init \
+  --security-opt seccomp=unconfined \
+  --ulimit memlock=-1:-1 \
+  --device /dev/kfd --device /dev/dri \
   -v "$FRANK_PRIMARY_MODEL_DIR":/models:ro \
-  "$FRANK_PRIMARY_IMAGE" /models \
-  --served-model-name "$FRANK_PRIMARY_SERVED_NAME" --port 7770 \
-  --tensor-parallel-size 2 \
-  ${FRANK_PRIMARY_DCP:+--decode-context-parallel-size "$FRANK_PRIMARY_DCP"} \
-  --max-model-len "$FRANK_PRIMARY_MAXLEN" \
-  --max-num-seqs "$FRANK_PRIMARY_SEQS" \
-  ${FRANK_PRIMARY_MNBT:+--max-num-batched-tokens "$FRANK_PRIMARY_MNBT"} \
-  ${FRANK_PRIMARY_PRECISE_MEMORY:+--expert-precise-memory} \
-  ${FRANK_PRIMARY_EXPERT_OFFLOAD_MEM:+--expert-offload-mem "$FRANK_PRIMARY_EXPERT_OFFLOAD_MEM"} \
-  --gpu-memory-utilization "$FRANK_PRIMARY_UTIL" \
-  --kv-cache-memory-bytes "$FRANK_PRIMARY_KV_BYTES" \
+  -v "$FRANK_PRIMARY_STATE_DIR":/data \
+  -v "$FRANK_PRIMARY_STATE_DIR/tune":/opt/radiance/share/radiance/tune \
+  -v "$FRANK_PRIMARY_TEMPLATE":/templates/template.jinja:ro \
+  "$FRANK_PRIMARY_IMAGE" \
+  --model "/models/$FRANK_PRIMARY_MODEL_FILE" \
+  --host :: --port "$FRANK_PRIMARY_PORT" \
+  --served-model-name "$FRANK_PRIMARY_SERVED_NAME" \
+  --override-chat-template /templates/template.jinja \
+  --tp "$FRANK_PRIMARY_TP" --tp-wire "$FRANK_PRIMARY_TP_WIRE" \
+  --max-num-seqs "$FRANK_PRIMARY_MAX_SEQS" \
+  --max-model-len "$FRANK_PRIMARY_MAX_MODEL_LEN" \
+  --placement "$FRANK_PRIMARY_PLACEMENT" \
+  --host-pool-mib "$FRANK_PRIMARY_HOST_POOL_MIB" \
+  --gpu-headroom-mib "$FRANK_PRIMARY_GPU_HEADROOM_MIB" \
+  --expert-vs-cache-ratio "$FRANK_PRIMARY_EXPERT_VS_CACHE" \
   --kv-cache-dtype "$FRANK_PRIMARY_KV_DTYPE" \
-  "$FRANK_PRIMARY_PREFIX_CACHE_FLAG" \
-  --enable-auto-tool-choice --tool-call-parser "$FRANK_PRIMARY_TOOL_PARSER" \
-  --reasoning-parser "$FRANK_PRIMARY_REASONING_PARSER" \
-  --default-chat-template-kwargs "$FRANK_PRIMARY_CHAT_TEMPLATE_KWARGS" \
-  ${FRANK_PRIMARY_SPEC:+--speculative-config "$FRANK_PRIMARY_SPEC"} \
-  ${FRANK_PRIMARY_COMPCONFIG:+--compilation-config "$FRANK_PRIMARY_COMPCONFIG"} \
-  ${FRANK_PRIMARY_GEN_CONFIG:+--override-generation-config "$FRANK_PRIMARY_GEN_CONFIG"} \
-  --trust-remote-code \
+  --prefix-cache-host-mib "$FRANK_PRIMARY_PREFIX_HOST_MIB" \
+  --prefix-cache-dir /data/kvcache/frank-primary \
+  --prefix-cache-disk-mib "$FRANK_PRIMARY_PREFIX_DISK_MIB" \
+  --ngram-placement "$FRANK_PRIMARY_NGRAM_PLACEMENT" \
+  --num-speculative-tokens "$FRANK_PRIMARY_SPEC_TOKENS" \
+  --max-num-batched-tokens "$FRANK_PRIMARY_MAX_BATCHED_TOKENS" \
+  ${FRANK_PRIMARY_EXTRA_ARGS:+$FRANK_PRIMARY_EXTRA_ARGS} \
   > "$LOG" 2>&1
 
 # From here on, this script's lifetime IS the container's: wait returns the
 # container's exit code, so a crash is a nonzero ExecStart and the unit's
-# Restart=on-failure is real. (systemd stop goes through ExecStop in the
-# unit — podman wait alone cannot stop the container child.)
+# Restart=always is real. (systemd stop goes through ExecStop in the unit —
+# podman wait alone cannot stop the container child.)
 exec podman wait "$FRANK_PRIMARY_CONTAINER"

scripts/serve-frank-secondary.sh

@@ -0,0 +1,120 @@
+#!/usr/bin/env bash
+# Serve ajgazin/Swift-1.5-Qwen3.8-27B-Uncensored-Dynamic-MTP-GGUF
+# (RCO-IQ3_XXS, **MTP head in the file**) as frank-secondary on Dankclaw's
+# RTX 3090 via llama.cpp (master build, CUDA sm_86).
+#
+# 2026-10-06 (Frank): TWO 131,072 conversations (262144 / 2) at the 128k
+# cap, ubatch 512, and MTP BACK ON (draft-mtp, depth 3 — the payload is
+# 1.56 GiB lighter than the IQ3_S-mtp it replaced, which is what funds
+# the draft ctx plus the second slot; ngram stays skipped — on a qwen35
+# hybrid it disables recurrent-state rollback). The q8_0 KV contract
+# (FP8 class) is unchanged; the per-tensor flags stay explicit either
+# way.
+#
+# Architecture: qwen35 hybrid — 64 real layers, 16 full-attention + 48
+# Gated DeltaNet (full_attention_interval 4), native 262144 window, plus
+# the blk.64 MTP head (nextn_predict_layers=1). KV lives only in the 16
+# attention layers (34 KiB/token at q8_0); the DeltaNet state is per-slot
+# fixed size.
+# Vision comes from the projector file (the GGUF carries no vision
+# tensors; the publisher SHIPS mmproj-BF16.gguf in the same repo), so
+# --mmproj is what gives this endpoint eyes; MMPROJ_FILE="" is the
+# documented text-only opt-out.
+#
+# Sampling: llama.cpp takes it per request (the fleet harness sends its
+# recipe from models.yml); greedy is the harness default.
+#
+# Tunables come from frank-secondary.env unless FRANK_SECONDARY_ENV_FILE overrides.
+set -euo pipefail
+
+SCRIPT_DIR="$(cd -- "$(dirname -- "${BASH_SOURCE[0]}")" && pwd)"
+ENV_FILE="${FRANK_SECONDARY_ENV_FILE:-$SCRIPT_DIR/frank-secondary.env}"
+if [[ -f "$ENV_FILE" ]]; then
+    set -a
+    # shellcheck disable=SC1090
+    source "$ENV_FILE"
+    set +a
+fi
+
+# --- defaults (used when the env file does not set them) -------------------
+: "${LLAMA_BIN:=/home/dank/GitPackages/llama-master/build-cuda/bin/llama-server}"
+: "${MODEL_FILE:=/home/dank/models/Swift15-UncIQ3XXS/Swift-1.5-Qwen3.8-27B-Uncensored-Dynamic-MTP-RCO-IQ3_XXS.gguf}"
+: "${ALIAS:=frank-secondary}"
+: "${BIND_ADDR:=0.0.0.0}"
+: "${LISTEN_PORT:=7771}"
+# ctx-size is the POOL llama.cpp divides across slots: 262144 / 2 gives
+# TWO 131,072 conversations — Frank's 128k cap each. 2026-10-06 (Frank):
+# drop MTP, drop ubatch to 512, take two slots; the fit is tight by his
+# explicit order (frank-secondary.env carries the arithmetic and the floor
+# violation).
+: "${CTX_SIZE:=262144}"
+: "${PARALLEL:=2}"
+# Compute-buffer geometry. Ubatch is a prefill knob (decode never moves).
+# Frank's 2026-10-06 tight-fit order ships 4096/512 (claws back ~0.3 GiB
+# toward the second slot); ubatch 1024 is the first A/B back if a future
+# boot lands with headroom (a miss fails loud at graph capture). The
+# logical batch is VRAM- and throughput-neutral at a fixed ubatch.
+: "${BATCH_SIZE:=4096}"
+: "${UBATCH_SIZE:=512}"
+# KV at FP8 class = q8_0 on llama.cpp (1.0625 B/elem => 8.5 effective bits;
+# there is no true fp8 KV dtype). Explicit either way so the contract is
+# visible in the argv.
+: "${CACHE_TYPE_K:=q8_0}"
+: "${CACHE_TYPE_V:=q8_0}"
+# Speculative decoding is ON by default (Frank, 2026-10-06 evening):
+# draft-mtp against the head that is IN the file. SPEC_TYPE none strips
+# BOTH speculative flags (flags move as a set) — a no-spec baseline must
+# not carry half a spec pair.
+: "${SPEC_TYPE:=draft-mtp}"
+: "${SPEC_DRAFT_N_MAX:=3}"
+: "${NGL:=99}"
+# Vision projector. No colon on purpose: an explicitly empty value means
+# "text-only, on purpose" and must NOT be replaced by the default path.
+: "${MMPROJ_FILE=/home/dank/models/Swift15-UncIQ3XXS/mmproj-BF16.gguf}"
+
+# Fail loudly rather than half-loading: a missing file is a download that
+# has not finished, and Restart=always would turn it into a crash loop that
+# reads like a runtime fault.
+for path in "$LLAMA_BIN" "$MODEL_FILE"; do
+    [[ -e "$path" ]] || { echo "frank-secondary: missing $path" >&2; exit 1; }
+done
+EXTRA_ARGS=()
+if [[ -n "$MMPROJ_FILE" ]]; then
+    [[ -e "$MMPROJ_FILE" ]] || { echo "frank-secondary: missing $MMPROJ_FILE" >&2; exit 1; }
+    EXTRA_ARGS+=(--mmproj "$MMPROJ_FILE")
+fi
+
+# Speculative flags move as a set: they reach the argv ONLY when a drafter
+# is named. SPEC_TYPE=none (the shipped contract) strips both — a no-spec
+# baseline must not carry half a spec pair.
+if [[ -n "$SPEC_TYPE" && "$SPEC_TYPE" != "none" ]]; then
+    EXTRA_ARGS+=(--spec-type "$SPEC_TYPE" --spec-draft-n-max "$SPEC_DRAFT_N_MAX")
+fi
+
+# The UVM device node is created lazily by nvidia-modprobe on the first CUDA
+# open after boot; make sure it exists before the server needs it.
+nvidia-modprobe -u -c=0 2>/dev/null || true
+
+# Flash attention stays pinned on: it is REQUIRED for quantized V (q8_0),
+# and the build carries GGML_CUDA_FA_ALL_QUANTS=ON (since 2026-09-17), so
+# the q8_0 KV contract in frank-secondary.env is a real path — without this
+# flag or that build option the engine silently falls back and the
+# contract becomes a lie.
+ARGS=(
+    --model "$MODEL_FILE"
+    --alias "$ALIAS"
+    --host "$BIND_ADDR"
+    --port "$LISTEN_PORT"
+    --ctx-size "$CTX_SIZE"
+    --parallel "$PARALLEL"
+    --batch-size "$BATCH_SIZE"
+    --ubatch-size "$UBATCH_SIZE"
+    --n-gpu-layers "$NGL"
+    --flash-attn on
+    --cache-type-k "$CACHE_TYPE_K"
+    --cache-type-v "$CACHE_TYPE_V"
+    --jinja
+    --reasoning auto
+    --reasoning-format deepseek
+)
+exec "$LLAMA_BIN" "${ARGS[@]}" ${EXTRA_ARGS[@]+"${EXTRA_ARGS[@]}"}

scripts/serve-frank-small.sh

@@ -1,99 +0,0 @@
-#!/usr/bin/env bash
-# Serve ornith-ai/Ornith-1.5-9B-GGUF (Q8_0) as frank-small on
-# Dankclaw's RTX 3090 via llama.cpp (master build, CUDA sm_86).
-#
-# 2026-10-01 night (Frank): swapped from SGLang + INT8-W8A16-AutoRound
-# safetensors to the llama.cpp GGUF path, because llama.cpp is the only
-# runtime on this host that moves K and V independently — the asymmetric
-# f16-K + q8_0-V contract that endpoint was built for. 2026-10-03 (Frank):
-# the pair is now f16/f16 — two wide slots freed the VRAM, so the values are
-# unquantized too. The per-tensor flags stay explicit either way.
-#
-# Architecture: qwen3_5 hybrid — 32 layers, 8 full-attention + 24 Gated
-# DeltaNet, native 262144 window. KV lives only in the 8 attention layers;
-# the DeltaNet state is per-slot fixed size. Vision comes from the repo's
-# separate projector (the Q6_K GGUF carries no `v.*` tensors at all), so
-# --mmproj is what gives this endpoint eyes; MMPROJ_FILE="" is the documented
-# text-only opt-out.
-#
-# Sampling: llama.cpp takes it per request (the fleet harness sends its
-# recipe from models.yml); greedy is the harness default.
-#
-# Tunables come from frank-small.env unless FRANK_SMALL_ENV_FILE overrides.
-set -euo pipefail
-
-SCRIPT_DIR="$(cd -- "$(dirname -- "${BASH_SOURCE[0]}")" && pwd)"
-ENV_FILE="${FRANK_SMALL_ENV_FILE:-$SCRIPT_DIR/frank-small.env}"
-if [[ -f "$ENV_FILE" ]]; then
-    set -a
-    # shellcheck disable=SC1090
-    source "$ENV_FILE"
-    set +a
-fi
-
-# --- defaults (used when the env file does not set them) -------------------
-: "${LLAMA_BIN:=/home/dank/GitPackages/llama-master/build-cuda/bin/llama-server}"
-: "${MODEL_FILE:=/home/dank/models/Ornith-Q8/Ornith-1.5-9B-Q8_0.gguf}"
-: "${ALIAS:=frank-small}"
-: "${BIND_ADDR:=0.0.0.0}"
-: "${LISTEN_PORT:=7771}"
-# ctx-size is the POOL llama.cpp divides across slots: 401408 / 2 gives
-# 200,704 per slot — two independent conversations, each one wide enough to
-# hold a whole frank-primary-sized prompt in a single shot (the native 262,144
-# no longer fits twice on a 24 GiB card once the vision tower is loaded;
-# frank-small.env carries the three measurements that set this number).
-: "${CTX_SIZE:=401408}"
-: "${PARALLEL:=2}"
-# Compute-buffer geometry. Before 2026-10-03 this endpoint never passed either
-# flag, so it ran on llama.cpp's own defaults (2048/512). The pair is 4096/2048
-# since 2026-10-05: on this CUDA card the prefill curve climbs all the way to
-# ubatch 2048 (unlike the R9700 layer-split sibling, where 512 is the peak),
-# and the logical batch is VRAM- and throughput-neutral. frank-small.env
-# carries the sweep tables and the VRAM risk note.
-: "${BATCH_SIZE:=4096}"
-: "${UBATCH_SIZE:=2048}"
-: "${CACHE_TYPE_K:=f16}"
-: "${CACHE_TYPE_V:=f16}"
-: "${NGL:=99}"
-# Vision projector. No colon on purpose: an explicitly empty value means
-# "text-only, on purpose" and must NOT be replaced by the default path.
-: "${MMPROJ_FILE=/home/dank/models/Ornith-Q8/mmproj-Ornith-1.5-9B-BF16.gguf}"
-
-# Fail loudly rather than half-loading: a missing file is a download that
-# has not finished, and Restart=always would turn it into a crash loop that
-# reads like a runtime fault.
-for path in "$LLAMA_BIN" "$MODEL_FILE"; do
-    [[ -e "$path" ]] || { echo "frank-small: missing $path" >&2; exit 1; }
-done
-EXTRA_ARGS=()
-if [[ -n "$MMPROJ_FILE" ]]; then
-    [[ -e "$MMPROJ_FILE" ]] || { echo "frank-small: missing $MMPROJ_FILE" >&2; exit 1; }
-    EXTRA_ARGS+=(--mmproj "$MMPROJ_FILE")
-fi
-
-# The UVM device node is created lazily by nvidia-modprobe on the first CUDA
-# open after boot; make sure it exists before the server needs it.
-nvidia-modprobe -u -c=0 2>/dev/null || true
-
-# Flash attention stays pinned on: it is the fast path for f16 KV, and it is
-# the only path that could serve quantized V at all — so if CACHE_TYPE_V is
-# ever set back to q8_0 this flag must not be lost, or the build falls back
-# silently and the KV contract in frank-small.env becomes a lie.
-ARGS=(
-    --model "$MODEL_FILE"
-    --alias "$ALIAS"
-    --host "$BIND_ADDR"
-    --port "$LISTEN_PORT"
-    --ctx-size "$CTX_SIZE"
-    --parallel "$PARALLEL"
-    --batch-size "$BATCH_SIZE"
-    --ubatch-size "$UBATCH_SIZE"
-    --n-gpu-layers "$NGL"
-    --flash-attn on
-    --cache-type-k "$CACHE_TYPE_K"
-    --cache-type-v "$CACHE_TYPE_V"
-    --jinja
-    --reasoning auto
-    --reasoning-format deepseek
-)
-exec "$LLAMA_BIN" "${ARGS[@]}" ${EXTRA_ARGS[@]+"${EXTRA_ARGS[@]}"}

scripts/stack-qwen38-official-fp8.py

@@ -0,0 +1,556 @@
+#!/usr/bin/env python3
+"""stack-qwen38-official-fp8.py — reshape the official Qwen/Qwen3.8-Flash-Next-FP8
+checkpoint into the form the shipped radiance qwen4exp plugin maps.
+
+Run INSIDE the staging dir's tree (python3 + numpy on the host; data never
+leaves the box). Two transforms, both VALUE-PRESERVING:
+
+1. EXPERTS: the shipped plugin's stacked layout names ONE tensor per family
+   (`mlp.experts.gate_up_proj` [E, 2F, H], `mlp.experts.down_proj` [E, H, F])
+   plus a `<family>_scale_inv` plane at the same index (NO `.weight` in the
+   stacked name — scale_name() appends _scale_inv to the tensor name verbatim),
+   while the official FP8 repo ships per-expert unfused tensors
+   (`experts.<e>.{gate,up,down}_proj.weight` + scale). Stacking is pure byte
+   concatenation along dim 0 — gate rows
+   first inside each expert's gate_up slab, which is the order the block
+   wants (rad_block_moe_fp8.h: "a stacked gate_up is ALREADY FUSED in the
+   order this block wants"). No requantisation: bytes are moved, not decoded.
+
+2. PLE TABLE: the plugin's fp8 branch wants the table's codes beside a
+   ONE-ELEMENT bf16 scale plane (`blk.%d.ple_ngram.scale`, view {1}); a
+   per-shard-blocked scale can never fit that view, and `rtn` re-encoding the
+   RAW scaleless codes would overflow (v / 1.99e-4 exceeds e4m3's 448 max).
+   So staging decodes the scaleless shards with the checkpoint's OWN bf16
+   scale into a bf16 table (exact: every e4m3 code value and the scale are
+   exactly representable in bf16 to within the code's own 2^-3 grid), and the
+   convert's rtn rule re-encodes at that same scale — the nearest-code
+   distance is 2^-3 while the bf16 rounding perturbs by 2^-9, so every
+   original code is restored exactly and the container gets the checkpoint's
+   own codes plus the one-element scale plane.
+
+The convert then runs with NO recipe except:
+  blk.*.ple_ngram.weight=rtn:codes=fp8_e4m3,block=*x*,scale=bf16,rule=fixed,scale_value=<s>
+with <s> read from the checkpoint and written to ple_scale.txt.
+
+Idempotent: re-runs skip when the staged index already carries the stacked
+names, the merged bf16 shard_0, and split_ngram_parts == 1. --force redoes.
+"""
+
+import argparse
+import json
+import mmap
+import multiprocessing as mp
+import os
+import re
+import struct
+import sys
+import time
+import numpy as np
+F8, BF16 = "F8_E4M3", "BF16"
+
+
+def log(msg):
+    print(msg, flush=True)
+
+
+# ---------------------------------------------------------------- safetensors
+
+def parse_header(path):
+    """Return (header_len, {name: (dtype, shape, data_offsets)})."""
+    with open(path, "rb") as fh:
+        (hlen,) = struct.unpack("<Q", fh.read(8))
+        hdr = json.loads(fh.read(hlen))
+    return hlen, hdr
+
+
+def write_safetensors(path, tensors, data_iter):
+    """tensors: [(name, dtype, shape, nbytes)]; data_iter yields bytes per tensor."""
+    hdr = {}
+    off = 0
+    for name, dtype, shape, nbytes in tensors:
+        hdr[name] = {"dtype": dtype, "shape": list(shape),
+                     "data_offsets": [off, off + nbytes]}
+        off += nbytes
+    blob = json.dumps(hdr, separators=(",", ":")).encode("utf-8")
+    pad = (-len(blob)) % 8  # keep the data start 8-byte aligned
+    blob += b" " * pad
+    with open(path, "wb") as fh:
+        fh.write(struct.pack("<Q", len(blob)))
+        fh.write(blob)
+        for _, _, _, nbytes in tensors:
+            fh.write(next(data_iter).tobytes())
+        fh.flush()
+        os.fsync(fh.fileno())
+    return 8 + len(blob) + sum(t[3] for t in tensors)
+
+
+# ------------------------------------------------------------------- preflight
+
+def build_meta(stage, index):
+    """name -> (file, global_byte_offset, nbytes, dtype, shape) for every tensor."""
+    files = sorted(set(index["weight_map"].values()))
+    meta = {}
+    for f in files:
+        path = os.path.join(stage, f)
+        hlen, hdr = parse_header(path)
+        for name, e in hdr.items():
+            if name == "__metadata__":
+                continue
+            b0, b1 = e["data_offsets"]
+            meta[name] = (f, 8 + hlen + b0, b1 - b0, e["dtype"], e["shape"])
+    return meta
+
+
+def e4m3_lut():
+    lut = np.zeros(256, np.float32)
+    for b in range(256):
+        sign = -1.0 if b & 0x80 else 1.0
+        e = (b >> 3) & 0xF
+        m = b & 0x7
+        if e == 15 and m == 7:
+            v = np.float32(np.nan)  # OCP e4m3fn: only 0x7F/0xFF are NaN
+        elif e == 0:
+            v = sign * (m / 8.0) * 2.0 ** -6
+        else:
+            v = sign * (1.0 + m / 8.0) * 2.0 ** (e - 7)
+        lut[b] = v
+    assert lut[0x38] == 1.0 and lut[0x7E] == 448.0
+    assert np.isnan(lut[0x7F]) and lut[0x00] == 0.0
+    assert lut[0x08] == 2.0 ** -6 and lut[0x80] == 0.0
+    return lut
+
+
+def f32_to_bf16_rne(v):
+    """float32 array -> uint16 bf16 bits, round-to-nearest-even."""
+    q = np.ascontiguousarray(v, dtype=np.float32)
+    q[~np.isfinite(q)] = 0.0
+    bits = q.view(np.uint32)
+    r = bits + 0x7FFF + ((bits >> 16) & 1)
+    return (r >> 16).astype(np.uint16)
+
+
+# ------------------------------------------------------------------- workers
+
+def stack_prefix(job):
+    """Byte-stack one layer's 512 experts into a 4-tensor safetensors file."""
+    (stage, pre, idx, n_e, shapes, meta) = job
+    (f2, h) = shapes["ffn2_h"]          # 2*ffn, n_embd
+    (f,) = shapes["ffn"]                # ffn
+    gs, ds = shapes["gs"], shapes["ds"] # scale shapes [5,20] and [20,5]
+
+    # byte widths: fp8 is 1 B/element, bf16 2 B; the fused gate_up scale is
+    # the gate and up scale planes concatenated (gate rows first)
+    g_bytes = (f2 // 2) * h
+    gs_bytes = gs[0] * gs[1] * 2          # ONE projection's scale plane
+    gus_bytes = 2 * gs_bytes              # fused scale per expert
+    dn_bytes = h * f
+    ds_bytes = ds[0] * ds[1] * 2
+
+    gu = np.empty(n_e * f2 * h, np.uint8)
+    gus = np.empty(n_e * gus_bytes, np.uint8)
+    dn = np.empty(n_e * dn_bytes, np.uint8)
+    dns = np.empty(n_e * ds_bytes, np.uint8)
+
+    mmaps = {}
+
+    def view(name):
+        f_, off, ln, dt, shp = meta[name]
+        if f_ not in mmaps:
+            fh = open(os.path.join(stage, f_), "rb")
+            mmaps[f_] = mmap.mmap(fh.fileno(), 0, access=mmap.ACCESS_READ)
+        return memoryview(mmaps[f_])[off:off + ln], (dt, shp)
+
+    def need(name, dt, shp):
+        mv, (got_dt, got_shp) = view(name)
+        assert got_dt == dt and list(got_shp) == list(shp), f"{name}: {got_dt}{got_shp}"
+        return mv
+
+    for e in range(n_e):
+        base = f"{pre}.mlp.experts.{e}"
+        slab = gu[e * f2 * h:(e + 1) * f2 * h]
+        slab[:g_bytes] = need(f"{base}.gate_proj.weight", F8, [f2 // 2, h])
+        slab[g_bytes:] = need(f"{base}.up_proj.weight", F8, [f2 // 2, h])
+        sslab = gus[e * gus_bytes:(e + 1) * gus_bytes]
+        sslab[:gs_bytes] = need(f"{base}.gate_proj.weight_scale_inv", BF16, gs)
+        sslab[gs_bytes:] = need(f"{base}.up_proj.weight_scale_inv", BF16, gs)
+        dn[e * dn_bytes:(e + 1) * dn_bytes] = need(f"{base}.down_proj.weight", F8, [h, f])
+        dns[e * ds_bytes:(e + 1) * ds_bytes] = need(f"{base}.down_proj.weight_scale_inv", BF16, ds)
+
+    for mm in mmaps.values():
+        mm.close()
+
+    # byte-identity check on two sampled experts (first and last), all four
+    # tensors — the stack IS the checkpoint bytes, so they must compare equal
+    for e in (0, n_e - 1):
+        base = f"{pre}.mlp.experts.{e}"
+        s = e * f2 * h
+        assert bytes(gu[s:s + g_bytes]) == bytes(stack_src(f"{base}.gate_proj.weight", meta, stage))
+        assert bytes(gu[s + g_bytes:s + f2 * h]) == bytes(stack_src(f"{base}.up_proj.weight", meta, stage))
+        t = e * gus_bytes
+        assert bytes(gus[t:t + gs_bytes]) == bytes(stack_src(f"{base}.gate_proj.weight_scale_inv", meta, stage))
+        assert bytes(gus[t + gs_bytes:t + gus_bytes]) == bytes(stack_src(f"{base}.up_proj.weight_scale_inv", meta, stage))
+        d = e * dn_bytes
+        assert bytes(dn[d:d + dn_bytes]) == bytes(stack_src(f"{base}.down_proj.weight", meta, stage))
+        u = e * ds_bytes
+        assert bytes(dns[u:u + ds_bytes]) == bytes(stack_src(f"{base}.down_proj.weight_scale_inv", meta, stage))
+
+    out = os.path.join(stage, f"stacked.{idx:03d}.safetensors")
+    tensors = [
+        (f"{pre}.mlp.experts.gate_up_proj", F8, [n_e, f2, h], n_e * f2 * h),
+        (f"{pre}.mlp.experts.gate_up_proj_scale_inv", BF16,
+         [n_e, gs[0] * 2, gs[1]], n_e * gus_bytes),
+        (f"{pre}.mlp.experts.down_proj", F8, [n_e, h, f], n_e * h * f),
+        (f"{pre}.mlp.experts.down_proj_scale_inv", BF16,
+         [n_e, ds[0], ds[1]], n_e * ds_bytes),
+    ]
+    bufs = iter([gu, gus, dn, dns])
+    written = write_safetensors(out, tensors, bufs)
+    return (pre, written)
+
+
+def stack_src(name, meta, stage):
+    """Re-open a source tensor for the sampled-expert check."""
+    f_, off, ln, dt, shp = meta[name]
+    fh = open(os.path.join(stage, f_), "rb")
+    mm = mmap.mmap(fh.fileno(), 0, access=mmap.ACCESS_READ)
+    fh.close()  # mmap keeps the mapping alive
+    return mm[off:off + ln]
+
+
+def fill_ple_shard(job):
+    """Decode one fp8 shard with the table scale, write bf16 rows into the mmap."""
+    (src_path, src_off, src_rows, dst_path, dst_off, scale_f32,
+     cols, chunk_rows) = job
+    lut = e4m3_lut()
+    with open(src_path, "rb") as fh:
+        sm = mmap.mmap(fh.fileno(), 0, access=mmap.ACCESS_READ)
+        with open(dst_path, "r+b") as dfh:
+            dm = mmap.mmap(dfh.fileno(), 0, access=mmap.ACCESS_WRITE)
+            for r0 in range(0, src_rows, chunk_rows):
+                r1 = min(r0 + chunk_rows, src_rows)
+                codes = np.frombuffer(sm, dtype=np.uint8,
+                                      count=(r1 - r0) * cols,
+                                      offset=src_off + r0 * cols)
+                q = lut.take(codes)
+                del codes  # release the mmap export before sm.close()
+                q[~np.isfinite(q)] = 0.0
+                q *= np.float32(scale_f32)
+                packed = f32_to_bf16_rne(q)
+                dm[dst_off + r0 * cols * 2: dst_off + r1 * cols * 2] = packed.tobytes()
+            dm.flush()
+            os.fsync(dfh.fileno())
+            dm.close()
+        sm.close()
+
+
+# ---------------------------------------------------------------------- main
+
+def main():
+    ap = argparse.ArgumentParser(description=__doc__.split("\n")[0])
+    ap.add_argument("--src", default=None,
+                    help="raw checkpoint dir (documentation only; unused)")
+    ap.add_argument("--stage", required=True, help="staging dir to reshape in place")
+    ap.add_argument("--force", action="store_true", help="redo even if already staged")
+    ap.add_argument("--workers", type=int, default=min(8, os.cpu_count() or 1))
+    args = ap.parse_args()
+    stage = args.stage
+    t0 = time.time()
+
+    index_path = os.path.join(stage, "model.safetensors.index.json")
+    cfg_path = os.path.join(stage, "config.json")
+    if not (os.path.isfile(index_path) and os.path.isfile(cfg_path)):
+        log(f"ERROR: {stage} is not a staged checkpoint (missing index/config)")
+        return 2
+    with open(index_path) as fh:
+        index = json.load(fh)
+    with open(cfg_path) as fh:
+        cfg = json.load(fh)
+    wm = index["weight_map"]
+
+    meta = build_meta(stage, index)
+
+    # ---- collect prefixes -------------------------------------------------
+    pre_re = re.compile(r"^(?P<pre>.+)\.mlp\.experts\.(?P<e>\d+)\.gate_proj\.weight$")
+    pres, n_e = {}, 0
+    for k in wm:
+        m = pre_re.match(k)
+        if m:
+            pres.setdefault(m.group("pre"), set()).add(int(m.group("e")))
+            n_e = max(n_e, int(m.group("e")) + 1)
+    prefixes = sorted(pres)
+    if not prefixes:
+        log("ERROR: no per-expert tensors found in the staged index")
+        return 2
+    mtp = [p for p in prefixes if p.startswith("mtp.")]
+    trunk = [p for p in prefixes if p.startswith("model.language_model.layers.")]
+    log(f"expert prefixes: {len(prefixes)} (trunk {len(trunk)}, mtp {len(mtp)}), "
+        f"{n_e} experts each")
+
+    # shapes from expert 0 of the first prefix
+    pre0 = prefixes[0]
+    shapes = {}
+    gdt, gsh = meta[f"{pre0}.mlp.experts.0.gate_proj.weight"][3:5]
+    ddt, dsh = meta[f"{pre0}.mlp.experts.0.down_proj.weight"][3:5]
+    gsd, gssh = meta[f"{pre0}.mlp.experts.0.gate_proj.weight_scale_inv"][3:5]
+    dsd, dssh = meta[f"{pre0}.mlp.experts.0.down_proj.weight_scale_inv"][3:5]
+    scale_law = (gdt == F8 and ddt == F8 and gsd == BF16 and dsd == BF16
+                 and list(gssh) == [(gsh[0] + 127) // 128, (gsh[1] + 127) // 128]
+                 and list(dssh) == [(dsh[0] + 127) // 128, (dsh[1] + 127) // 128])
+    if not scale_law:
+        log(f"ERROR: unexpected expert shapes: gate {gdt}{gsh} scale {gsd}{gssh} "
+            f"down {ddt}{dsh} scale {dsd}{dssh} (scale law 128x128)")
+        return 2
+    shapes = {"ffn2_h": (gsh[0] * 2, gsh[1]), "ffn": (dsh[1],),
+              "gs": tuple(gssh), "ds": tuple(dssh)}
+    f2, h = shapes["ffn2_h"]
+    f = shapes["ffn"][0]
+    log(f"shapes: gate_up [{n_e},{f2},{h}] F8, down [{n_e},{h},{f}] F8, "
+        f"scales {gssh}x2 / {dssh} bf16")
+
+    # ---- PLE table --------------------------------------------------------
+    ple_re = re.compile(r"^(?P<pre>.+)\.ple\.ple_embedding\.ngram_embedding\.shard_(?P<i>\d+)\.weight$")
+    ple_pre, shard_ids = None, {}
+    for k in wm:
+        m = ple_re.match(k)
+        if m:
+            ple_pre = m.group("pre")
+            shard_ids[int(m.group("i"))] = k
+    if ple_pre is None:
+        log("ERROR: no PLE table shards found")
+        return 2
+    n_shards = len(shard_ids)
+    if sorted(shard_ids) != list(range(n_shards)):
+        log(f"ERROR: PLE shard ids not contiguous 0..{n_shards - 1}")
+        return 2
+    sc_name = f"{ple_pre}.ple.ple_embedding.ngram_embedding.weight_scale"
+    sdt, ssh, sln = meta[sc_name][3], meta[sc_name][4], meta[sc_name][2]
+    if sdt != BF16 or list(ssh) != [1]:
+        log(f"ERROR: PLE scale is {sdt}{ssh}, expected BF16 [1]")
+        return 2
+    s_bits = struct.unpack("<H", bytes(view_src_tuple(meta, sc_name, stage)))[0]
+    scale_f32 = struct.unpack("<f", struct.pack("<I", s_bits << 16))[0]
+    log(f"PLE: {ple_pre}, {n_shards} shards, scale bf16 = {scale_f32!r}")
+
+    rows = [meta[shard_ids[i]][4][0] for i in range(n_shards)]
+    total_rows = sum(rows)
+    hd = meta[shard_ids[0]][4][1]
+    if any(meta[shard_ids[i]][4][1] != hd for i in range(n_shards)):
+        log("ERROR: PLE shards disagree on HD")
+        return 2
+    if hd % 128 != 0 and hd != 160:
+        log(f"WARNING: PLE HD={hd} is neither 160 nor a multiple of 128")
+
+    # ---- idempotence gate -------------------------------------------------
+    stacked_names = [f"{p}.mlp.experts.gate_up_proj" for p in prefixes]
+    merged_name = f"{ple_pre}.ple.ple_embedding.ngram_embedding.shard_0.weight"
+    already = (all(n in wm for n in stacked_names)
+               and merged_name in wm
+               and find_split(cfg) is not None and read_split(cfg) == 1)
+    if already and not args.force:
+        log("already staged (stacked names, merged shard_0 and split_ngram_parts=1 "
+            "all present) — nothing to do")
+        return 0
+
+    # ---- disk gate --------------------------------------------------------
+    stacked_bytes = len(prefixes) * n_e * (f2 * h + h * f)
+    ple_bytes = total_rows * hd * 2
+    need = stacked_bytes + ple_bytes
+    st = os.statvfs(stage)
+    avail = st.f_bavail * st.f_frsize
+    if avail < need * 1.05:
+        log(f"ERROR: {avail / 2**30:.1f} GiB free < {need / 2**30:.1f} GiB needed (+5%)")
+        return 2
+    log(f"disk: {avail / 2**30:.1f} GiB free, need {need / 2**30:.1f} GiB "
+        f"(experts {stacked_bytes / 2**30:.1f}, ple {ple_bytes / 2**30:.1f})")
+
+    # ---- stage experts ----------------------------------------------------
+    jobs = []
+    for idx, pre in enumerate(prefixes):
+        jmeta = {n: meta[n] for n in meta if n.startswith(f"{pre}.mlp.experts.")}
+        jobs.append((stage, pre, idx, n_e, shapes, jmeta))
+    ctx = mp.get_context("fork")
+    written_total = 0
+    with ctx.Pool(args.workers) as pool:
+        for pre, nbytes in pool.imap_unordered(stack_prefix, jobs):
+            written_total += nbytes
+            log(f"  stacked {pre} ({nbytes / 2**30:.2f} GiB)")
+    log(f"experts stacked: {written_total / 2**30:.2f} GiB in {len(prefixes)} files")
+
+    # ---- stage PLE --------------------------------------------------------
+    # The file name sorts BEFORE the model-000NN shards ON PURPOSE: the reader
+    # opens every index-named file in sorted order and emplaces EVERY tensor
+    # it physically holds, first-wins (checkpoint.cpp add_shard). The raw
+    # shards carry tensors of the SAME name as the merged one
+    # (...ple.ple_embedding.ngram_embedding.shard_0.weight, [2500012,160]
+    # fp8), so ple.merged.safetensors lost the race and the plan read one raw
+    # shard as the codes. "00-" wins the sort and the name.
+    out_name = "00-ple.merged.safetensors"
+    out_path = os.path.join(stage, out_name)
+    data_bytes = total_rows * hd * 2
+    hdr = {merged_name: {"dtype": BF16, "shape": [total_rows, hd],
+                         "data_offsets": [0, data_bytes]}}
+    blob = json.dumps(hdr, separators=(",", ":")).encode("utf-8")
+    pad = (-len(blob)) % 8
+    blob += b" " * pad
+    hlen = len(blob)
+    with open(out_path, "wb") as fh:
+        fh.write(struct.pack("<Q", hlen))
+        fh.write(blob)
+        fh.truncate(8 + hlen + data_bytes)
+    jobs = []
+    row0 = 0
+    for i in range(n_shards):
+        f_, off, ln, _, shp = meta[shard_ids[i]]
+        jobs.append((os.path.join(stage, f_), off, shp[0], out_path,
+                     8 + hlen + row0 * hd * 2, scale_f32, hd, 131072))
+        row0 += shp[0]
+    assert row0 == total_rows
+    with ctx.Pool(args.workers) as pool:
+        pool.map(fill_ple_shard, jobs)
+    log(f"PLE merged: {data_bytes / 2**30:.2f} GiB bf16 "
+        f"({total_rows} rows x {hd} cols)")
+
+    # ---- post-verify ------------------------------------------------------
+    verify(stage, prefixes, n_e, shapes, meta, shard_ids, rows, hd,
+           scale_f32, out_path, hlen)
+
+    # ---- rewrite index + config -------------------------------------------
+    new_wm = {k: v for k, v in wm.items() if k not in set(shard_ids.values())}
+    for idx, pre in enumerate(prefixes):
+        fname = f"stacked.{idx:03d}.safetensors"
+        for t in (f"{pre}.mlp.experts.gate_up_proj",
+                  f"{pre}.mlp.experts.gate_up_proj_scale_inv",
+                  f"{pre}.mlp.experts.down_proj",
+                  f"{pre}.mlp.experts.down_proj_scale_inv"):
+            new_wm[t] = fname
+    new_wm[merged_name] = out_name
+    total = sum(meta_size(meta, k) for k in new_wm
+                if k in meta)  # originals
+    # new tensors' sizes
+    total += written_total + data_bytes
+    index["weight_map"] = new_wm
+    if "metadata" in index and isinstance(index["metadata"], dict):
+        index["metadata"]["total_size"] = total
+    # ATOMIC REPLACE, never open(path, "w"): the stage's index is a symlink
+    # to the RAW checkpoint (staging symlinks everything but config.json) —
+    # open("w") would rewrite the raw file through the link. os.replace
+    # swaps the link itself for a real file; the raw tree stays pristine.
+    for path, payload in ((index_path, json.dumps(index, indent=2)),
+                          (cfg_path, json.dumps(cfg, indent=2))):
+        tmp = path + ".tmp"
+        with open(tmp, "w") as fh:
+            fh.write(payload)
+        os.replace(tmp, path)
+    n = set_split(cfg, 1)
+    log(f"config: split_ngram_parts=1 written at {n} location(s)")
+
+    with open(os.path.join(stage, "ple_scale.txt"), "w") as fh:
+        fh.write(repr(scale_f32) + "\n")
+
+    dt = time.time() - t0
+    log(f"staged in {dt / 60:.1f} min: {written_total / 2**30:.1f} GiB experts + "
+        f"{data_bytes / 2**30:.1f} GiB PLE; scale {scale_f32!r} -> ple_scale.txt")
+    return 0
+
+
+def meta_size(meta, name):
+    _, _, ln, _, _ = meta[name]
+    return ln
+
+
+def view_src_tuple(meta, name, stage):
+    f_, off, ln, _, _ = meta[name]
+    with open(os.path.join(stage, f_), "rb") as fh:
+        fh.seek(off)
+        return fh.read(ln)
+
+
+def find_split(cfg):
+    """Locate split_ngram_parts keys (recursive); return list of paths."""
+    hits = []
+
+    def walk(d, path):
+        if isinstance(d, dict):
+            for k, v in d.items():
+                if k == "split_ngram_parts":
+                    hits.append(path + [k])
+                walk(v, path + [k])
+    walk(cfg, [])
+    return hits
+
+
+def read_split(cfg):
+    hits = find_split(cfg)
+    if not hits:
+        return None
+    d = cfg
+    for k in hits[0][:-1]:
+        d = d[k]
+    return d[hits[0][-1]]
+
+
+def set_split(cfg, value):
+    hits = find_split(cfg)
+    for path in hits:
+        d = cfg
+        for k in path[:-1]:
+            d = d[k]
+        d[path[-1]] = value
+    if not hits:  # plugin's meta_int2 falls back to the top-level key
+        cfg["split_ngram_parts"] = value
+        hits.append(["split_ngram_parts"])
+    return hits
+
+
+def verify(stage, prefixes, n_e, shapes, meta, shard_ids, rows, hd,
+           scale_f32, out_path, hlen):
+    """Cheap strong checks: headers parse, sizes match, sampled bytes identical."""
+    f2, h = shapes["ffn2_h"]
+    f = shapes["ffn"][0]
+    gs, ds = shapes["gs"], shapes["ds"]
+    # headers of every new file
+    for idx, pre in enumerate(prefixes):
+        path = os.path.join(stage, f"stacked.{idx:03d}.safetensors")
+        hlen2, hdr = parse_header(path)
+        want = {
+            f"{pre}.mlp.experts.gate_up_proj": (F8, [n_e, f2, h]),
+            f"{pre}.mlp.experts.gate_up_proj_scale_inv":
+                (BF16, [n_e, gs[0] * 2, gs[1]]),
+            f"{pre}.mlp.experts.down_proj": (F8, [n_e, h, f]),
+            f"{pre}.mlp.experts.down_proj_scale_inv": (BF16, [n_e, ds[0], ds[1]]),
+        }
+        for name, (dt, shp) in want.items():
+            e = hdr[name]
+            assert e["dtype"] == dt and list(e["shape"]) == shp, name
+            assert e["data_offsets"][1] - e["data_offsets"][0] == \
+                int(np.prod(shp)) * (1 if dt == F8 else 2), name
+    _, phdr = parse_header(out_path)
+    e = phdr[next(iter(phdr))]
+    assert e["dtype"] == BF16 and list(e["shape"]) == [sum(rows), hd]
+
+    # sampled PLE rows: recompute and compare bytes exactly
+    lut = e4m3_lut()
+    out_off = 8 + hlen
+    for i in (0, len(shard_ids) // 2, len(shard_ids) - 1):
+        name = shard_ids[i]
+        f_, off, ln, _, shp = meta[name]
+        with open(os.path.join(stage, f_), "rb") as fh:
+            mm = mmap.mmap(fh.fileno(), 0, access=mmap.ACCESS_READ)
+            for r in (0, shp[0] - 1):
+                codes = np.frombuffer(mm, dtype=np.uint8, count=hd,
+                                      offset=off + r * hd)
+                q = lut.take(codes) * np.float32(scale_f32)
+                del codes  # release the mmap export before mm.close()
+                want = f32_to_bf16_rne(q).tobytes()
+                got_row = sum(rows[:i]) + r
+                with open(out_path, "rb") as ofh:
+                    ofh.seek(out_off + got_row * hd * 2)
+                    got = ofh.read(hd * 2)
+                assert got == want, f"PLE row {got_row} mismatch"
+            mm.close()
+    log("verify: new-file headers, sizes and sampled PLE rows all exact; "
+        "expert slabs were byte-checked in-stack")
+
+
+if __name__ == "__main__":
+    sys.exit(main())

scripts/test-changed.sh

@@ -204,12 +204,12 @@ for path in "${CHANGED[@]}"; do
         scripts/qwen-sharp-v22.4.0.jinja)
             add_test tests/frank_large_unit.bats
             ;;
-        # frank-small is llama.cpp on the 3090 as a ROOT unit: a native CUDA
+        # frank-secondary is llama.cpp on the 3090 as a ROOT unit: a native CUDA
         # binary, so no generated quadlet is involved. Its env, unit and serve
         # script all live in scripts/, which is why this case has to precede
         # the generic scripts/*.service and scripts/* fallbacks below.
-        scripts/frank-small.*|scripts/serve-frank-small.sh)
-            add_test tests/frank_small_unit.bats
+        scripts/frank-secondary.*|scripts/serve-frank-secondary.sh)
+            add_test tests/frank_secondary_unit.bats
             add_test tests/llm_units.bats
             add_test tests/deployment_scoping.bats
             add_test tests/scripts_syntax.bats

serverAppConfigs-docs/Agent/Agent Docs and Edits.md

@@ -0,0 +1,111 @@
+---
+type: reference
+title: Agent Docs and Edits
+description: Two self-inflicted damage classes — structurally broken markdown after parallel agent edits, and botched hashline PUT/CUT repairs — with the audit checks and recovery order for each.
+tags: [agents, docs, markdown, hashline, edit-recovery]
+timestamp: 2026-10-05T00:00:00-07:00
+---
+
+# Agent Docs and Edits
+
+## Doc-structure audit (after parallel agent edits)
+
+Use after one or more subagents have edited markdown docs (OKF/Obsidian
+vault with `---` frontmatter), before committing. Link checkers and
+completeness seams validate *references*, not *structure* — agents reliably
+produce structurally broken markdown that still links cleanly.
+
+Damage catalogue:
+
+1. Banner/prose prepended **above** the opening `---` → the frontmatter is
+   no longer frontmatter (title/tags/description become body text).
+2. A rewrite **appended** instead of merged → duplicate headings and a stray
+   second `---` mid-file.
+3. Literal model artifacts in the body: `<|channel>}<tool_call|>...`,
+   `+- `, `- - `.
+4. Wrapped bullets re-emitted with a `- ` prefix on every physical line.
+5. Line-range replacement that dropped a sentence tail → dangling fragment,
+   often mid-parenthesis.
+
+Checks (fast, run all):
+
+```bash
+head -3 <file>                          # must be `---` if the page has frontmatter
+grep -c '^---$' <file>                  # exactly 2 (opener + closer)
+grep -n '^## ' <file> | sort -t: -k2 | uniq -f1 -d   # duplicate sections
+grep -rnE '^\+- |^- - |<\|' <files>     # artifacts
+python3 scripts/check-vault-links.py <vault-dir>
+./tests/run.sh                          # docs completeness seam etc.
+```
+
+Repair order (matters):
+
+1. If the file was destroyed rather than edited: stash the agent version to
+   `.AGENTTRASH/<YYYYMMDD-topic>/`, then `git checkout HEAD -- <file>` and
+   re-apply only the intended changes by hand.
+2. If it merely duplicated: **salvage unique content first** — cut the
+   one-of-a-kind section, paste it into the copy you keep, THEN delete the
+   duplicate ranges. Never delete first.
+3. Re-add missing sentence tails (insert before the orphaned fragment),
+   keeping list continuation lines indented.
+4. Re-run all checks above.
+
+Gotchas: agents can write AFTER they appear finished (job settles, file
+changes) — re-verify structure after the last background job reports, not
+before. Editing an agent's already-damaged file with fresh line numbers is
+safe, but its snapshot tag changes on every edit — re-read before each
+subsequent edit. `.AGENTTRASH/` is gitignored: trash is local-only, so
+nothing moved there reaches the repo — that is intended (temporary hold, not
+archive).
+
+## Hashline edit recovery
+
+Failure mode: multi-op edits resolve every `PUT N.=M:` / `CUT N.=M:` index
+against the **original** file numbering, and the post-apply diff view
+**elides lines** (it shows an anchor window, not the whole file). Two traps
+follow:
+
+1. A second edit call built from a diff-view snapshot uses line numbers from
+   a *partial* view. Lines outside the window are invisible but still
+   counted.
+2. If the first apply was itself wrong, the file shifted. The diff view
+   tells you the damage, but not the true offsets.
+
+Result: the repair pastes fresh text **beside** stale text instead of over
+it.
+
+Procedure:
+
+1. **Never carry line numbers across calls.** Re-read the target range for a
+   fresh tag before every subsequent edit. A read is cheap; a duplicated
+   paragraph costs a commit and a revert.
+2. **One op per call when the previous apply was imperfect.** Fixed offsets
+   are only trustworthy when every preceding apply landed exactly as
+   specified.
+3. **Re-read after every apply**, and verify the whole affected region — the
+   diff view is a sample, not proof. Check the *boundaries*: the line before
+   the edit and the line after it. Damage shows up as a repeated sentence
+   straddling the seam.
+4. **Repair by CUT, not by re-PUT.** Once you know the exact stale lines
+   from a fresh read, delete them. Do not try to PUT a corrected superset
+   over a range whose extent you are guessing at.
+5. Verify with a content assertion, not a diff review: `grep -c` for the
+   duplicated sentence, plus a syntax/compile check on any touched script.
+
+Same rule for any imperative `edit` tool with an old/new match: the matched
+text must be the **entire** text being replaced. Matching a bare fragment
+inside a paragraph splices the replacement into the middle of the sentence
+and silently corrupts the file. Re-read the file afterwards to confirm.
+
+### Anchored in-page links
+
+Renaming a Markdown heading breaks `[[#Old Heading]]` anchors. When a
+section is retitled, grep for the old heading text and fix every anchor in
+the same pass — doc link-checkers frequently do not validate `#fragment`
+targets, so a green suite will not catch a dangling one.
+
+## Related
+
+- [[OMP Harness]] — the session context these repairs protect
+- [[Command Center Dotfiles]] — the repo these edits commit to
+- [[Testing Policy]] — the seams that catch broken vault structure

serverAppConfigs-docs/Agent/Command Center Dotfiles.md

@@ -0,0 +1,87 @@
+---
+type: reference
+title: Command Center Dotfiles
+description: The ~/.omp repo is itself the deployment — stow-equivalent packages, full-replica rsync transport, the pre-overwrite audit for non-blank hosts, deploy.sh mechanics, and the fish/sshpass transport gotchas.
+tags: [dotfiles, omp, deploy, rsync, blankpad]
+timestamp: 2026-10-05T00:00:00-07:00
+---
+
+# Command Center Dotfiles
+
+Repo `~/.omp` is itself the deployment: packages are home-relative trees
+under `dotfiles/<pkg>/`; `scripts/deploy.sh` symlinks live files into the
+repo checkout. `git pull` + `deploy.sh` is the whole propagation story.
+
+## Transport decision (agreed with Frank, executed 2026-09-09)
+
+- Ship via **full-replica rsync of `/home/dank/.omp/` including `.git`** to
+  `<user>@<host>:/home/<user>/.omp/`, NOT clone-from-Gitea — `main` can be
+  ahead of `origin/main`; push is a user-only trigger, so clone would ship
+  stale state.
+- Deploy runs on-target as the target user: `bash
+  /home/<user>/.omp/scripts/deploy.sh` (resolves the repo via `git -C
+  $(dirname $(realpath $0)) rev-parse --show-toplevel`, so `.git` must be
+  present at the target path — full-replica transport guarantees this).
+
+## Pre-overwrite audit (MANDATORY before first sync on a non-blank host)
+
+Diff every file that `deploy.sh --check` reports as DRIFT against the repo
+copy BEFORE letting quarantine-and-replace run. Host-specific divergence
+belongs in an override package, not in quarantine:
+
+1. Pull divergent live files from target
+   (`ssh ... 'cat <path>' > local-staged-copy`).
+2. Secret-audit staged copies (grep token/secret/password/key shapes)
+   before anything enters git.
+3. Compose `dotfiles/zz-<hostname>/` on the source machine from those live
+   copies; commit to repo first.
+4. Ship the package dir via rsync, then run deploy sync — override deploys
+   LAST and win over base packages.
+
+## deploy.sh mechanics (verified on blankpad)
+
+- `--check` = pure report (ADOPT/LINK/DRIFT lines), writes nothing. Run it
+  first, always.
+- Default mode = sync: adopt identicals silently, link new,
+  quarantine-and-replace divergents into `.quarantine/<host>/<ts>/`
+  (gitignored; never rm).
+- **Override-target bug (fixed cc2c652):** `zz-<host>` packages nest
+  base-package dirs internally (`zz-blankpad/niri/.config/...`); target
+  derivation must strip the nested package segment, else overrides deploy to
+  phantom `$HOME/niri/.config/...` while real targets keep pointing at
+  canonical copies (visible symptom: check shows 0 actions but
+  `readlink ~/.zshrc` points at base `zsh/.zshrc`).
+- systemd units auto-enable only when ExecStart binary resolves on that
+  host; dconf packages load via `dconf load /`, never symlinks.
+- `core.hooksPath=.githooks` is armed by deploy.sh per machine — set it once
+  post-transfer even before first sync.
+
+## ssh/sshpass transport gotchas (all bit live)
+
+- Resolve the target via mDNS FQDN (`avahi-resolve -4 -n <host>.local`); a
+  bare hostname may not resolve. Drop-not-refuse on port 22 = sshd down or
+  firewall default-drop, not auth failure — needs a local/console fix on
+  target.
+- `SSHPASS='...' sshpass -e ssh ...` inline per invocation; an env passed
+  via the tool's env parameter did not propagate to the sshpass child
+  process.
+- The remote login shell may be **fish**: never send bash syntax
+  (`for/do/done`, `$?`) bare through `ssh host 'cmd'`; wrap as `bash -c
+  "..."` or ship a script. See [[SSH and Remote Access]].
+- File shipping: `ssh 'cat > /remote/path' < localfile` (stdin redirect). A
+  here-doc attached to the ssh command is consumed by `cat` — the intended
+  script body becomes the file content, silently. Verify shipped bytes with
+  an md5sum compare on both ends; do NOT compare `/dev/stdin` after `cat`
+  drained it (false DIVERGED).
+
+## Verification of a completed deploy
+
+Re-run `deploy.sh --check` → expect exactly zero actions/quarantined/skipped
+= idempotent steady state. Spot-check `readlink` on override files points
+into `dotfiles/zz-<host>/`, not base packages.
+
+## Related
+
+- [[SSH and Remote Access]] — the transport mechanics this deploy rides
+- [[Noctalia and Niri]] — the desktop packages this repo ships
+- [[Deploy State Triage]] — the serverAppConfig deploy flow (different repo)

serverAppConfigs-docs/Agent/OMP Harness.md

@@ -0,0 +1,262 @@
+---
+type: reference
+title: OMP Harness
+description: The omp agent harness as configured on this fleet — compaction chain arithmetic, per-model sampler routing via compat.extraBody, and the static-preamble/skill-library audits that keep session context lean.
+tags: [omp, harness, compaction, models-yml, context, samplers]
+timestamp: 2026-10-05T00:00:00-07:00
+---
+
+# OMP Harness
+
+The harness reaches the fleet through `~/.omp/agent/models.yml` (model
+registry; single source of truth — there is no copy on the servers) and
+`~/.omp/agent/config.yml` (roles, compaction, global samplers). Nothing in
+this file is repo-managed; edits are harness-side only.
+
+## Compaction chain arithmetic (verify before ANY window change)
+
+When a fleet endpoint's context grows past its harness-declared window, OMP
+compacts by sending the conversation to a designated target model. Before
+changing any `contextWindow`/`maxTokens`, verify the chain arithmetically —
+a target whose window is smaller than the source's trigger point silently
+breaks compaction.
+
+Where the contract lives:
+
+- `~/.omp/agent/config.yml` → `compaction.remoteEndpoint` (target URL),
+  `methodOrder`, `thresholdPercent` (70), idle settings
+- `~/.omp/agent/models.yml` → per-model `contextWindow`, `maxTokens`,
+  `compactionModel` references
+- Implementation: omp plugin `dist/core/compaction/compaction.js`
+
+The arithmetic (the coupled contract every endpoint's maxTokens comment
+encodes):
+
+1. **Source trigger** = `thresholdPercent × source contextWindow`
+   (0.70 × 200,704 = 140,492 for frank-primary on the current shape)
+2. **Output budget** = capped at the target model's `maxTokens`
+   (32,768 for frank-secondary; 36,864 frank-primary; 57,344 frank-large — the
+   largest, because halogen's `HALOGEN_MAX_TOKENS_CAP` must stay ≥ it)
+3. **REQUIREMENT**: prompt (≤ trigger) + budget ≤ the target's per-slot
+   window. Worked pairs from the live registry: 140,492 + 36,864 = 177,356 ≤
+   200,704 (frank-primary); 140,490 + 57,344 = 197,834 ≤ 200,700
+   (frank-large); 140,492 + 32,768 = 173,260 ≤ 200,704 (frank-secondary).
+
+Procedure (recompute from the LIVE files every time — windows rotate with
+every swap; the numbers above are the 2026-10-05 state):
+
+1. Read the `config.yml` compaction block + `models.yml` entries for source
+   and target. Every endpoint's `contextWindow` comment carries its own
+   arithmetic — read them.
+2. Read the target checkpoint's `max_position_embeddings` (native window) —
+   the served window may be a concurrency contract, not the native limit
+   (`--parallel N` DIVIDES the pool; the per-slot number is the contract).
+3. If native ≥ trigger + budget, reslice the pinned pool:
+   `MAX_RUNNING × CONTEXT_LENGTH ≤ MAX_TOTAL_TOKENS` (the pin exists because
+   the runtime's autofit leaves too little for deferred loads + prefill
+   workspace; check `nvidia-smi` headroom — expect ~0.8 GB workspace margin).
+4. Changes: host env (CONTEXT_LENGTH-style env tunables) via git →
+   `./tests/run.sh` → deploy; harness `models.yml` directly (no repo copy).
+5. Verify: `/health` 200, boot log's pool line, then a long-prompt probe at
+   ≥ trigger height (the compaction-fit guarantee), then concurrent requests.
+
+Known reconciliation trap: comments in models.yml have described compaction
+as running "on the dankserver endpoint" while `remoteEndpoint` pointed at
+dankclaw — the comment/config disagreed. The comments now name the real
+target (`compaction.remoteEndpoint` = dankclaw :7771). **The config is
+truth; fix comments in the same change.** `maxTokens` never changes for a
+reslice — it caps output; `contextWindow` caps input. Boot-shape OOM
+laddering is a different problem ([[Llama Cpp Serving Sizing]]).
+
+## Per-model sampler config (`compat.extraBody`)
+
+`models.yml` has **no per-model sampler keys**. Sampling is global
+(`config.yml`: `temperature`, `topP`, `topK`, `minP`, `presencePenalty`,
+`repetitionPenalty`; default `-1` = **not sent**). The per-model hook is
+`compat.extraBody`:
+
+**Why it works:** `applyOpenAIExtraBody` runs **last** in the
+openai-completions builder — after `applyChatCompletionsCompatPolicy` — via
+`Object.assign(params, extraBody)`. So extraBody overrides global settings
+and compat-policy output.
+
+```yaml
+compat:
+  reasoningEffortMap:
+    max: xhigh
+  extraBody:
+    temperature: 1.0
+    top_p: 0.95
+    top_k: 20
+    min_p: 0.0
+    presence_penalty: 0.0
+    repetition_penalty: 1.0
+    preserve_thinking: true
+```
+
+Use **wire names**, not OMP camelCase. Two traps:
+
+1. **`preserve_thinking` is not user-settable as a compat key.**
+   `qwenPreserveThinking` is auto-derived in `buildOpenAICompat` from the
+   *auto-detected* `thinkingFormat` + local baseUrl, **before**
+   `applyCompatOverrides`. Setting `compat.thinkingFormat: qwen` in
+   models.yml does **not** enable it. Custom-named models (`frank-large`)
+   match no Qwen id pattern → use `extraBody: {preserve_thinking: true}`.
+2. **Never put `chat_template_kwargs` in `extraBody`** where a compat policy
+   also sets it — `Object.assign` is shallow and replaces the whole object,
+   silently dropping the `reasoning_effort` kwarg the compat policy set.
+   Top-level `preserve_thinking` is honoured by halogen. (frank-primary's
+   FreeToken entry currently carries a `chat_template_kwargs` extraBody with
+   a comment asking to verify-and-prune — the exact trap to check at that
+   endpoint's next swap.)
+
+**Gotcha: silent greedy decode.** halogen has **no server-side sampler
+defaults** — `temperature` absent or `0` = greedy. If OMP sends nothing, the
+endpoint decodes greedy while looking configured. This is why frank-large's
+entry carries the recipe on the wire and a comment saying "do not drop it".
+
+**Reasoning-effort folding.** Qwen3.8's template accepts only
+`{xhigh (default), medium, low}` for `reasoning_effort` and 400s on anything
+else ("Unexpected reasoning effort high"). Every endpoint's
+`reasoningEffortMap` folds all harness levels onto that set; a role whose
+level is unmapped fails on its FIRST request, and the failure names Jinja,
+not the model.
+
+### Verify the exact wire body (echo-capture)
+
+Don't trust the YAML — capture what OMP actually emits:
+
+1. Echo server on `127.0.0.1:18099` that dumps the JSON body to a file and
+   returns a canned `chat.completion` (include `reasoning_content` so
+   thinking paths exercise).
+2. Copy the **real** models.yml into a throwaway profile with only the
+   baseUrl swapped:
+   ```bash
+   sed 's|http://<host>:7770/v1|http://127.0.0.1:18099/v1|' \
+     ~/.omp/agent/models.yml > ~/.omp/profiles/wiretest/agent/models.yml
+   ```
+3. Run and inspect:
+   ```bash
+   omp --profile wiretest --model "<provider>/<model>" -p "hi" \
+     --no-tools --no-session --no-extensions --no-skills --no-rules
+   jq 'del(.messages, .tools)' last-body.json
+   ```
+   Confirm every intended field is present with the right value.
+
+**Verify a field is *honoured*, not just accepted** — unknown fields often
+return 200 anyway. Prove behavior with a **prompt-token delta**: replay an
+assistant turn carrying a long `reasoning_content`, then compare
+`usage.prompt_tokens` with the flag true vs false. A large delta means the
+field changed template rendering; zero delta means it was ignored.
+
+Cleanup: trash the probe profile and echo server (`profiles/.AGENTTRASH`),
+never `rm`.
+
+## Static-preamble audit ("the context feels bloated")
+
+What loads statically every session (measure each, don't guess):
+
+1. Repo rules file — `wc -c AGENTS.md` (baseline 2026-10-05: 72,179 B, ~18k
+   tokens — historically the largest block).
+2. Skill index — only names+descriptions load; bodies are on-demand. Sum
+   description bytes across all SKILL.md frontmatter:
+   ```bash
+   python3 - <<'EOF'
+   import glob, re, os
+   for root in [os.path.expanduser("~/.omp/agent/managed-skills"),
+                os.path.expanduser("~/.omp/agent/skills"), ".omp/skills"]:
+       n=tot=0
+       for sk in glob.glob(root+"/*/SKILL.md"):
+           fm=re.search(r'^---.*?---', open(sk, encoding="utf-8", errors="replace").read(), re.S)
+           d=re.search(r'description:\s*(.*(?:\n(?:\s{2,}.*)*)?)', fm.group(0) if fm else "")
+           n+=1; tot+=len(d.group(1)) if d else 0
+       print(root, n, tot)
+   EOF
+   ```
+   Baseline 2026-10-05: 171 managed skills / ~38 KB descriptions + 9 user
+   skills.
+3. Memory lessons — `find ~/.omp/agent/memories -name 'MEMORY.md' -o -name
+   'memory_summary.md' | xargs wc -c`. The injected block ≈ the largest
+   MEMORY.md.
+4. Harness/system prompt + tool schemas + CLI persona: fixed cost, ~20 KB;
+   not prunable.
+
+Known rot patterns:
+
+- Near-duplicate auto-captures of the same lesson (three "authentik ORM"
+  entries, the freetoken-build pair, the three job-runner skills) — dedupe
+  via `manage_skill delete`/`update`.
+- Corrupted captures containing stray CJK characters (`一团ly`, `等方法`)
+  mid-sentence — a known defect of the auto-capture path; delete the corrupt
+  copy, keep the clean one.
+- Stale host/service names in consolidated memory (pre-rename
+  `frank-default-large`) — the repo's AGENTS.md topology table is ground
+  truth.
+
+Fix strategy: AGENTS.md prose (measured runbook narratives) → move to
+`serverAppConfigs-docs/LLM/` runbooks, keep laws + tables in AGENTS.md (the
+repo's own docs-tree rule mandates this). Managed skills live in
+`~/.omp/agent/managed-skills` (safe to edit); user-authored skills in
+`~/.omp/agent/skills` are NEVER touched by tooling. Memory store edits:
+stage removed captures to trash first (`~/.AGENTTRASH/YYYYMMDD-topic`).
+
+Baseline to beat (2026-10-05): ~35–40k tokens of preamble — AGENTS.md ~18k,
+skill index ~10k, lessons ~3.5k, harness ~6k. Proposed reductions: −12k
+(AGENTS.md slimming), −3k (skill dedupe).
+
+## Managed-skill library hygiene
+
+Use when auditing, pruning, or consolidating `~/.omp/agent/managed-skills`
+(the auto-captured skill store; the full index is injected into EVERY
+session's context, so dead or duplicate entries are a permanent tax).
+
+The static-context budget (measured 2026-10-05):
+
+| Block | Size | Notes |
+|---|---|---|
+| `AGENTS.md` (repo) | 72,179 B ≈ 18k tok | biggest single item |
+| Skill index (~173 skills) | ~40 KB ≈ 10k tok | names+descriptions only |
+| Learned-lessons block | ~14 KB | can contain corrupted/duplicate captures |
+| Harness system prompt + tool schemas | ~23 KB | fixed |
+
+Audit procedure:
+
+1. Build an inventory: per skill, name, mtime, body word count, description,
+   first 12 headings (one pass, out to a file; read in pages).
+2. Group into: retired-runtime (host/service gone; its runbook already lives
+   in `serverAppConfigs-docs/LLM/`), verbatim-duplicate pairs/triples (same
+   lesson captured twice — often the SAME DAY), same-lesson families
+   (N skills = one procedure), keep.
+3. Read full bodies of every merge candidate before proposing a merge;
+   description alone proves overlap only for exact dupes.
+4. Present A/B/C/D verdicts for user approval. NEVER execute before the
+   discussion concludes — the user gates execution explicitly.
+5. On approval: stage everything to trash first
+   (`~/.AGENTTRASH/YYYYMMDD-<topic>/`), never rm. After merging: recreate
+   the surviving skill via `manage_skill` update with body = superset,
+   delete absorbed ones.
+
+Known rot classes (this library, 2026-10-05):
+
+- Same-day duplicate captures (freetoken-build-dankclaw vs
+  freetoken-dankclaw-build; the three job-runner skills).
+- CUDA-skill vs its self-declared HIP twin (pcie-link-capability-probe vs
+  pcie-link-probe-hip-rdna4) → merge.
+- Era ghosts: skills naming retired runtimes (freetoken, lucebox, r9v/
+  radiance, old halogen-flash-server image, ik_llama, GLM-5.3 experiment).
+- Skills contradicting user standing orders (live-bench-dashboard vs the
+  2026-09-29 delete-dashboards order) — flag for user, don't assume.
+- Memory store: corrupted captures with CJK noise from broken ssh commands;
+  stale host/service renames.
+
+Sizing verdicts that worked: bench family → 2 skills (harness vs
+discipline), not one mega-skill; keep profiler/kernel-time attribution
+standalone; fold go/no-go arithmetic skills into their probe skill
+(heterogeneous-accelerator-split-triage → PCIe probe).
+
+## Related
+
+- [[Agent Docs and Edits]] — editing vault docs and hashline targets safely
+- [[Command Center Dotfiles]] — the `~/.omp` repo deploy this config rides
+- [[Llama Cpp Serving Sizing]] — the engine-side sizing the chain must fit
+- [[LLM Bench Windows and Measurement]] — measuring the endpoints this config routes

serverAppConfigs-docs/Agent/index.md

@@ -0,0 +1,23 @@
+---
+type: index
+description: Pages about the agent harness itself — omp config, vault editing discipline, and the dotfiles repo that deploys the agent's own home.
+tags: [agent, index]
+timestamp: 2026-10-05T00:00:00-07:00
+---
+
+# Agent
+
+Pages about the agent's own tooling and context — how omp is configured, how
+docs get edited safely, and how the agent's home is deployed.
+
+- [[OMP Harness]] — compaction chain arithmetic, per-model samplers via
+  `compat.extraBody`, preamble and skill-library audits
+- [[Agent Docs and Edits]] — doc-structure audits after parallel edits;
+  hashline edit recovery
+- [[Command Center Dotfiles]] — the `~/.omp` repo-as-deployment, rsync
+  transport, override packages
+
+## Related
+
+- [[Testing Policy]] — the seams these pages' checks ride
+- [LLM](../LLM/index.md) — the fleet these agents operate

serverAppConfigs-docs/Architecture/Deploy State Triage.md

@@ -0,0 +1,151 @@
+---
+type: reference
+title: Deploy State Triage
+description: Reading and reconciling the deploy-state markers — stale marker vs live container, pending-service lockout, interrupted/raced deploys — without restarting healthy heavy endpoints.
+tags: [deployment, deploy-state, systemd, podman, triage]
+timestamp: 2026-10-05T00:00:00-07:00
+---
+
+# Deploy State Triage
+
+The deploy-state store is `~/.local/state/serverAppConfigs/deploy-state/<svc>`
+on each target (rootless, user `dank`, format exactly two lines
+`revision=<sha>` / `status=deployed|pending`, written atomically via tmp file
++ `mv`). It is written ONLY by a `deploy.sh` dispatch.
+
+## 1. Stale marker vs live container
+
+**Why it happens:** an out-of-band relaunch — reboot, `systemctl restart`, or
+a restart after the checkout ff-pulled to a newer revision — starts the
+container from the CURRENT `scripts/<svc>.env` but does NOT advance the
+marker. So `marker != HEAD` means "not dispatched via deploy.sh", NOT
+"behavior pending".
+
+**Why it matters:** a plain `deploy.sh` run re-selects and restarts a service
+whose marker lags. On llama.cpp/R9V endpoints that is a ~16–19 min reload of
+a 100+ GiB payload — and frank-primary (dankserver:7770) is also the
+harness's own model backend, so a blind deploy is a self-inflicted outage for
+possibly zero behavioral change.
+
+**Procedure (read-only first):**
+
+1. Committed state: `git log --oneline main..<branch>` empty ⇒ merged; the
+   operational delta is usually one `scripts/<svc>.env`.
+2. Probe the target (ssh as `dank`, passwordless sudo present):
+   - `cat ~/.local/state/serverAppConfigs/deploy-state/<svc>`
+   - `git -C ~/serverAppConfigs rev-parse HEAD`
+   - `systemctl is-active/is-enabled <svc>`;
+     `curl -s -o /dev/null -w '%{http_code}' http://127.0.0.1:<port>/health`
+   - the on-host env file already carries the new keys (it landed via the
+     ff-pull).
+3. **Decisive — read the LIVE container, not the marker.** Use `podman` as
+   `dank` (rootless store; bare `docker` is NOT the shim → "No such object";
+   `sudo podman` inspects an empty store):
+   - `podman inspect <ctr> --format '{{.State.StartedAt}}'` — booted AFTER
+     the pull ⇒ it read the new env
+   - `podman inspect <ctr> --format '{{.Args}}' | tr ' ' '\n' | grep -i <flag>`
+   - engine env: `podman exec <ctr> sh -c 'for f in /proc/[0-9]*/environ; do
+     tr "\0" "\n" <$f 2>/dev/null; done | grep -i <VAR>'`
+   - confirm the launcher maps each tunable to its flag
+     (`grep scripts/launch.sh`), e.g. `R9V_ENABLE_PREFIX_CACHING=1 →
+     --enable-prefix-caching`.
+
+**Decision:** live argv+env == target config AND `/health` 200 ⇒ behavior is
+live — **do NOT restart**; reconcile the marker (below). Only if the live
+container lacks the flag/env (or predates the pull) is a real
+`deploy.sh <target> --service <svc>` warranted.
+
+**Reconcile (no restart)** — mirror the deployer's atomic idiom:
+
+```bash
+S=$HOME/.local/state/serverAppConfigs/deploy-state
+H=$(git -C ~/serverAppConfigs rev-parse HEAD)
+t=$(mktemp "$S/.<svc>.XXXXXX"); printf 'revision=%s\nstatus=%s\n' "$H" deployed > "$t"; mv -f -- "$t" "$S/<svc>"
+```
+
+Then prove the gate: `./tests/run.sh` + `tests/e2e/run.sh` (needs
+`tests/e2e/.env.local`; expect the documented `11-admin` red if the
+FrankBook-Pro Mac mini 192.168.0.112 is ASLEEP — confirm with `ping -c2` +
+a `/dev/tcp` probe: 100 % loss + timeout = host off, not a regression).
+Log to `serverAppConfigs-docs/log.md`; commit + push (docs-only ⇒ deploy
+no-op).
+
+**Anti-patterns:** trusting a commit message ("X deployed") or the marker
+alone — both are bookkeeping; the container's argv/env is ground truth.
+Reading container state with `sudo podman` / bare `docker`. Restarting a
+healthy heavy endpoint to satisfy a stale marker.
+
+## 2. Pending lockout
+
+Symptom: `deploy.sh <target>` fails or silently stops with
+`service <name> is not deployed on <target>` and the `[3/3]` e2e gate never
+appears.
+
+1. **Get the real exit code first:** `./scripts/deploy.sh X 2>&1 | tail`
+   reports TAIL's 0 — use `set -o pipefail` or redirect to a file, then
+   `echo $?`.
+2. Inspect the state store (no sudo):
+   `ssh <target> 'cat ~/.local/state/serverAppConfigs/deploy-state/<name>'`.
+   A `status=pending` entry for a service absent from `service_is_allowed`
+   in `scripts/deploy-remote.sh` = retired-service lockout.
+3. Fixed behavior (since 02a7876, 2026-09-17): `add_pending_services` prunes
+   not-allowed pending entries to
+   `~/.AGENTTRASH/<YYYYMMDD>-retired-deploy-state/<service>` and the run
+   continues. Verify the WARN line names the service and the state file is
+   gone.
+4. On a host without the fix: move the stale entry to
+   `~/.AGENTTRASH/<date>-retired-deploy-state/` manually (never `rm`), re-run.
+5. Verify: deploy reaches `[3/3] Running Playwright e2e gate` and completes;
+   `deploy-state/` no longer holds the retired name.
+
+Related contracts: the superseded seam
+(`deployment_scoping.bats` "remote runner recognizes a pending service at
+the same revision") now asserts prune-and-continue (exit 0), not exit 2;
+`deployment_flow.bats` extracts `add_pending_services` via sed and
+HOME-overrides the trash path. **When retiring a service going forward:**
+delete or reconcile its deploy-state entry as part of the retirement commit —
+the prune is a safety net, not the intended cleanup path.
+
+## 3. Interrupted or raced deploys
+
+The deploy is TWO processes: `deploy.sh` (dev box) and `deploy-remote.sh`
+(runs on the target over ssh). **Killing the dev-side job does NOT kill the
+host-side one** — two `deploy-remote.sh` processes can end up managing the
+same unit, one deploying a revision you already superseded. Check:
+`pgrep -af "deploy-remote"` on the target, read the revision argument on
+each, kill stale ones by explicit PID.
+
+A killed deploy leaves behind:
+
+- the unit **stopped/failed** (`Restart=always` does NOT fire after an
+  explicit stop — start it yourself);
+- the state marker **pending** (pre-deploy or in-flight revision);
+- the e2e gate un-run (the deployer runs it last).
+
+Reconcile the marker ONLY after the service itself is verified healthy — a
+marker that runs ahead of reality is worse than one that lags (the planner
+uses it to decide what is pending). Then close the loop:
+
+1. Verify against MEASURED, config-specific numbers (KV capacity line, arena
+   slots, argv flags, `NRestarts`), not just health 200.
+2. Run the gate the dead deploy skipped (`tests/e2e/run.sh` from a checkout
+   that has the gitignored `tests/e2e/.env.local` — a fresh `git worktree`
+   does NOT have it; copy it in first).
+3. If the killed revision was restoring a known-good shape, confirm the
+   numbers match the last measured-good boot of that shape — equality is the
+   proof, not health alone.
+
+Boot failures must be read from the journal, not `podman logs`: serve scripts
+`podman rm -f` the container on every start (previous attempt lost), and
+`podman logs` on a container still *Stopping* returns the OLD configuration's
+numbers — which looks exactly like your change having no effect. Key on
+container identity (`podman ps -a --format '{{.CreatedAt}}'`) and read the
+cause from `journalctl --since "N min ago" | grep -a "<container-name>\["`.
+See [[vLLM Boot Forensics and Config Interrogation]] for the full capture
+procedure.
+
+## Related
+
+- [[Deployment Flow]] — the deploy pipeline itself
+- [[Quadlet Lifecycle]] — unit ownership and drift diagnosis
+- [[vLLM Boot Forensics and Config Interrogation]] — journal capture

serverAppConfigs-docs/Architecture/Deployment Flow.md

@@ -63,7 +63,7 @@ service on the other host.
 | Dankclaw compose services | `podman-compose -f dankclaw/compose.yaml up -d --no-deps NAME` | Recreates only `NAME`; Caddy is handled separately because it fronts both hosts. |
 | SearXNG, Gluetun, Valkey, OpenWebUI, Pi-hole, Gitea database/server | user Quadlet + `systemctl --user` | Installs the selected unit and restarts only the required dependency-coupled units. Gitea's web proxy is the canonical Caddy compose service. |
 | `frank-large` | systemd + `systemctl` (root unit) | Relinks and restarts only the large inference service; waits up to 20 minutes for `/health`. |
-| `frank-small` | systemd + `systemctl` (root unit) | Relinks and restarts only the small inference service; waits up to 20 minutes for `/health`. |
+| `frank-secondary` | systemd + `systemctl` (root unit) | Relinks and restarts only the small inference service; waits up to 20 minutes for `/health`. |
 | `frank-medium` | rootless Quadlet + `systemctl --user` | Installs `dankserver/frank-medium/quadlet/frank-medium.container`; on first deploy retires the llama.cpp root unit and the vLLM replica quadlets into the home trash, then restarts the generated unit. Waits up to 20 minutes for `/health`. |
 | Dankserver media stack (RomM trio, Jellyfin, Calibre-Web, Copyparty) | user Quadlet + `systemctl --user` | Installs `dankserver/<svc>/quadlet/*.container`, restarts the generated unit. The first cutover per service is idempotent: it retires the compose-era bare container, rescues any anonymous volume into the data root, and merge-moves `/opt/<svc>` to `/home/dank/Containers/<svc>` (mapped-uid roots move without chown). Steady state is restart-only. |
 

serverAppConfigs-docs/Architecture/Quadlet Lifecycle.md

@@ -0,0 +1,209 @@
+---
+type: reference
+title: Quadlet Lifecycle
+description: The full Quadlet discipline on this fleet — ownership auditing, compose-to-Quadlet cutover, drift/crash-loop diagnosis, the podman 5.x v1 generator quirks, log retrieval, and compose bring-up with ground truth.
+tags: [podman, quadlet, systemd, boot-persistence, compose, drift]
+timestamp: 2026-10-05T00:00:00-07:00
+---
+
+# Quadlet Lifecycle
+
+Fleet rule (AGENTS.md "Boot Persistence & Self-Healing"): every deployed,
+working container MUST have a self-healing, boot-persistent Quadlet unit —
+`.container` file in the repo (`<host>/<svc>/quadlet/`), installed by the
+deployer, `[Install] WantedBy=default.target`, `Restart=always`, linger on.
+A container that runs only from compose dies silently at reboot. This page
+is the operating manual for that lifecycle.
+
+## 1. Ownership audit — find the containers that will not survive a reboot
+
+```bash
+for c in $(podman ps --format '{{.Names}}'); do
+  printf '%-26s %s\n' "$c" "$(podman inspect $c --format '{{index .Config.Labels "PODMAN_SYSTEMD_UNIT"}}')"
+done
+```
+
+- **label present** → the unit owns it; persistence = `[Install]` + `loginctl
+  show-user <user> -p Linger` = yes.
+- **label absent, compose service exists** → the classic gap: dies at reboot.
+  Check the publish address while there — compose forms often bind the
+  wildcard (an unauthenticated UI/API on every interface).
+- **label present AND a compose service with the same name** → two owners; a
+  stray `podman compose up` can create a duplicate that fights the unit for
+  the name and the port. Remove the compose service, leave an explanatory
+  comment (repo convention).
+
+## 2. Cutover: compose service → Quadlet unit
+
+Checklist (precedents to copy:
+`dankclaw/{caddy,gitea,searxng,open-webui,seerr}/quadlet/`,
+`dankclaw/observability/quadlet/`):
+
+1. Unit file at `<host>/<svc>/quadlet/<name>.container`, mirroring compose
+   line for line: `Image=` (`localhost/` prefix for locally built images +
+   `Pull=never`), `ContainerName=` (Caddy and scrape configs address
+   containers by name), `Network=`, `User=0:0` when the image's own user
+   cannot write a rootless-owned bind mount, `Volume=` (`:ro` preserved),
+   `Exec=` for compose `command:`, `PublishPort=`, health keys,
+   `[Service] Restart=always` + `RestartSec=10`, `[Install]
+   WantedBy=default.target`. **Never `systemctl --user enable` the generated
+   unit** — `[Install]` + linger IS the persistence (`is-enabled` = `generated`
+   is healthy; explicit enable fails with "transient or generated").
+2. Exposure: publish on `127.0.0.1` unless something off-host genuinely needs
+   it (Caddy reaches containers by name over the shared network; the wildcard
+   publish is a guard-chain bypass).
+3. Handler in `scripts/deploy-remote.sh`: `install_quadlet <dir> <name>` →
+   `systemctl --user restart <name>.service` → `wait_for_health`; retire any
+   legacy hand-written unit into `~/.AGENTTRASH/` (a leftover
+   `~/.config/systemd/user/<svc>.service` that shells out to
+   `podman compose up` collides BY NAME with the generated unit);
+   `podman rm -f <container>` so the unit's container can take the name.
+4. Planner: map `<host>/<svc>/**` → service in `scripts/deploy-plan.py`
+   AND add a `service_is_allowed` entry; keep `SERVICE_ORDER` dependencies
+   ordered. Add a `test-changed` case above the generic
+   `*/quadlet/*.container` fallback.
+5. Retire the compose entry in the SAME change — two owners is how a stale
+   duplicate comes back.
+6. Seam test `tests/<svc>_quadlet.bats`: boot persistence, restart policy, no
+   wildcard publish, container name matches Caddy/scrape references, health
+   command, deployer installs+restarts+waits, never `rm`, never `enable`.
+   Migrate any older assertion that read the compose file for devices/flags.
+7. Deploy `--service <name>`; prove: `is-active`/`is-enabled` =
+   active/generated, container `healthy`, endpoint answers,
+   `list-dependencies default.target` shows the unit, off-host probes on
+   previously published ports now refuse, and the container no longer carries
+   `io.podman.compose.project`.
+
+**Traps that cost real time:**
+
+- `HealthCmd` = CMD-SHELL (podman runs `/bin/sh -c`); a distroless image has
+  no shell → the check fails forever. `--health-cmd` won't take exec form
+  either. Put `HEALTHCHECK … CMD ["binary","arg"]` in the image and build
+  with `--format docker` — the default OCI format has no HEALTHCHECK field
+  and drops it with only a warning.
+- Never point a unit's `EnvironmentFile` at the repo-root `.env` — it is a
+  gluetun env carrying `HOME=/root`; systemd sets HOME for the service and
+  the unit dies with `stat /root/.config: permission denied`. Use a
+  per-service env under `/home/dank/Containers/<svc>/env`, seeded by the
+  handler with a single-variable filter, validated so a missing secret fails
+  loudly.
+- `Network=host` exporters scraped by containerized Prometheus must bind the
+  host's LAN address (containers cannot reach host loopback) PLUS `127.0.0.1`
+  for the homepage probe — two `--web.listen-address` flags, never the
+  wildcard.
+- alloy 1.x refuses the `ui` config block; the unit must pass
+  `--server.http.listen-addr=0.0.0.0:12345` before its port can be published.
+- GPU services: `AddDevice=` + `GroupAdd=keep-groups` (user-unit spawns carry
+  no supplementary groups).
+- When the last service leaves a compose file, the compose-validity seam
+  fails (requires ≥1 service): keep the file as comments and relax the seam
+  deliberately, or delete the file and fix the planner's compose-diff path —
+  decide with the owner, the deploy tooling reads it.
+
+## 3. Drift / crash-loop diagnosis (5 minutes, in order)
+
+Symptom: crash-looping or perpetually unhealthy container whose DEFINITION
+looks right; or a service quietly down for days because a restart policy kept
+something running. Cause: the running container was created from an OLDER
+definition — a compose/unit edit does not touch an existing container, and
+the restart policy faithfully resurrects it forever.
+
+1. **Who started it:** `podman inspect <name> --format 'unit={{index
+   .Config.Labels "PODMAN_SYSTEMD_UNIT"}} restart={{.HostConfig.RestartPolicy.Name}}
+   created={{.Created}}'`; compare `created` with when the definition changed
+   (`git log -p --follow <config>`). Created-before-the-edit is the signature.
+2. **What it is ACTUALLY running:** `{{.Config.CreateCommand}}` is the truth.
+   `{{.HostConfig.Devices}}` / `{{.HostConfig.GroupAdd}}` print empty/0 even
+   for containers started with `--device`/`--group-add` — a false negative
+   that sends you hunting the wrong root cause.
+3. **What the process says:** `podman logs <name> | grep -iE 'error|cannot|Runtime'`
+   (a capability error like `No HIP GPUs are available` beside a missing
+   device flag confirms drift). Count the damage: `{{.RestartCount}}` — one
+   instance reached 9,146 restart attempts on stale config.
+
+Quickest unblock: `podman compose up -d --force-recreate <svc>`. Durable fix
+= the §2 cutover. **Verify the capability, not the status:** `podman exec
+<name> python3 -c "import torch; print(torch.cuda.is_available())"`;
+`ss -ltn | grep <port>` for the actual bind addresses. ComfyUI incident:
+2026-09-16, stale device-less container, 9,146 restarts. History repeats:
+2026-08-26 Caddy (no unit), 2026-09-11 five observability services.
+
+## 4. Podman 5.x quadlet v1 generator quirks (verified live 2026-08-28)
+
+- **Hard-fail keys** — the whole unit silently fails to generate
+  (`systemctl --user cat <unit>` shows "could not be found"): any
+  dash-prefixed `[Container]` option (`--shm-size=…` → `unsupported key …
+  in group 'Container'`); `Command=` is unsupported, use `Exec=`.
+- **HealthCmd space-splitting:** the v1 generator splits `HealthCmd` on
+  spaces into the healthcheck argv; a quoted probe
+  `/bin/sh -c "pg_isready …"` lands as argv token `"pg_isready` →
+  command-not-found → container reads **unhealthy while the DB is up**.
+  HealthCmd must contain only space-free tokens
+  (`HealthCmd=pg_isready -d authentik -U authentik`). If health reads
+  unhealthy but logs look fine, compare the baked
+  `{{.Config.Healthcheck.Test}}` array against the intended command before
+  believing the status.
+- **Probe a new key before shipping:** drop a throwaway `probe.container`,
+  `daemon-reload`, `journalctl --user --since "1 min ago" -o cat | grep -i
+  "unsupported key"`, `systemctl --user cat probe.service` (shows the
+  generated ExecStart verbatim), then remove + reload.
+
+## 5. Rootless Quadlet log retrieval
+
+Step 0 — establish ownership: `systemctl --user status <name>.service`
+(works ⇒ rootless user unit); `loginctl show-user <user> -p Linger`. If
+`--user` knows the unit, EVERY command runs as that user with NO sudo.
+
+| Need | Command |
+|---|---|
+| Current instance tail/follow | `podman logs --tail 200 / -f <name>` |
+| Across restarts/recreates | `journalctl --user -u <name>.service -n 200 --no-pager` |
+| Time-bounded / crash-only | `--since "14:30"` / `-p warning` |
+| Kernel-side evidence (OOM, SVM) | `sudo journalctl -k --since …` |
+
+Quadlet units commonly run `--replace --rm`: **`podman logs` is destroyed on
+recreation** — the journal is the only durable record. Traps:
+`sudo podman logs` → "no such container" (root's store does not contain
+rootless containers; same for `sudo podman ps`); a system-level
+`journalctl -u <name>` returns nothing for a user unit; during first-start
+weight downloads health reports unhealthy BY DESIGN (check the payload file
+is growing). Verbose vendor switches are a config change — through git and
+deploy.sh, never a live env poke.
+
+## 6. Compose bring-up with ground truth (rootless hosts)
+
+1. **Audit reality before pressing buttons:** `podman ps -a` runs per-user —
+   check every namespace (`sudo podman ps -a`) and native units
+   (`systemctl list-units`) before declaring a service absent. Compare
+   compose mount sources against the real filesystem — compose files name
+   phantom paths (e.g. `/mnt/media/*` vs the real LUKS array at
+   `/mnt/entropy`). Fix compose through git first, deploy second.
+2. **Pre-create bind-mount sources** (`sudo mkdir -p` + `chown 1000:1000`)
+   or creation fails `statfs …: no such file or directory`.
+3. **Staged bring-up:** infra first (`up -d mariadb redis`), poll
+   `{{.State.Health.Status}}`, then the rest. Fresh containers answer
+   000/refused for the first seconds — poll.
+4. **Never guess image tags:** ghcr.io answers 403 on the bearer-token
+   request for nonexistent/private repos ("manifest unknown"/token len 0 =
+   repo or tag does not exist, not auth). Query the registry API
+   (`ghcr.io/token?scope=repository:<ns>/<repo>:pull` →
+   `/v2/<ns>/<repo>/tags/list`).
+5. **Read the interface from inside the image** before guessing flags/envs:
+   `podman exec <ctr> cat /z/cpp.sh` etc. HTTP 500 with a styled error page
+   means RUNNING-but-misconfigured, not down. Copyparty ac ground truth:
+   entrypoint `/bin/ash /z/cpp.sh -c /z/initcfg`; accounts/volumes are ARGV
+   flags (`-a USER:PASS`, `-v SRC:DST:PERMS`), not envs — and with none
+   configured it serves cwd read/write for everyone (an unauthenticated
+   world-writable share).
+6. Remote-shell quoting: zsh eats bare `===`/`?` tokens; a zero-match from a
+   mangled pattern is not absence — re-verify the target exists.
+7. `user: "1000:1000"` SIGSEGV at boot = unmapped uid in the rootless userns
+   (host 1000 == container 0): drop the user line, run as container-root.
+
+## Related
+
+- [[Deployment Flow]] — the deployer that installs these units
+- [[Stale Signal Audit]] — the audit that catches drift's victims
+- [[Container Network Namespaces]] — reachability between containers
+- [[Dankclaw/ComfyUI]] — the 9,146-restart incident
+- [[Infrastructure/Observability]] — same class, twice

serverAppConfigs-docs/Architecture/Stale Signal Audit.md

@@ -0,0 +1,144 @@
+---
+type: reference
+title: Stale Signal Audit
+description: The audit battery for surfaces that rot without failing — unreachable probes, vacuous assertions, invalid checkers, gates that never run, and prose invalidated by a change. Four passes found real defects every time.
+tags: [testing, observability, audit, signals, gates]
+timestamp: 2026-10-05T00:00:00-07:00
+---
+
+# Stale Signal Audit
+
+Run after any change and before trusting a green gate. The productive question
+is **which signal here cannot possibly be correct?** Every pass on this repo
+(4 of 4) found real defects — including one in the rule file every future
+agent reads first.
+
+## The surfaces, and what each vantage point can reach
+
+| Surface | Producer's vantage point | Can reach |
+|---|---|---|
+| Homepage status cards | the `homepage-status` timer on **Dankclaw** | Dankclaw's loopback, Dankclaw container names, **published** LAN addresses |
+| Prometheus scrape jobs | the `prometheus` container on Dankclaw | container names on `dankclaw_default`, LAN addresses, that host's loopback only |
+| bats / e2e suites | the dev box | the repo, plus the live fleet over HTTPS/LAN |
+
+Anything else — another host's loopback, a container-internal port never
+published — is unreachable no matter how healthy the service is. The failure
+mode is always the same shape: a signal that **cannot** succeed by
+construction, which teaches the operator to ignore the surface it lives on.
+
+## 1. Permanently-red probes (reachability)
+
+For every monitor/probe/alert target ask: *can this checker physically reach
+that endpoint?* Seen classes: a card probing another host's container-internal
+port; a "probe" job scraping `/` (HTML → `unsupported Content-Type`); a
+`/metrics` that is disabled (Gitea 404).
+
+- A scrape cannot express "is this HTTP service answering?" — the right probe
+  is the **blackbox exporter** in probe mode (which also gives guarded vhosts
+  a real uptime signal, since Prometheus cannot solve an Anubis challenge).
+- **Fix in the direction of honesty, never of silence:** unreachable target →
+  mark the homepage card `wip: true` with the reason in `description` (the
+  checker skips `wip`; `wip` is the existing convention for "real service,
+  not probeable"), or repoint the job at something reachable. NEVER open a
+  database port on the LAN to make a badge green, and never delete a scrape
+  job — an absent scrape and a healthy one are indistinguishable.
+- Alert rules must not depend on always-red signals — scope them to services
+  that can actually be healthy (`LlmEndpointDown` is scoped to `frank-*` for
+  exactly that reason).
+- A target's failure STRING can prove a fix: a `/metrics` scrape flipping from
+  `connection refused` to `unsupported Content-Type "text/html"` means the
+  service came up and only the scrape is wrong. Read the error, not just
+  up/down.
+
+## 2. Vacuous assertions (highest severity — the gate is already dead)
+
+A seam that greps a file passes *by absence* once the file is retired:
+`grep` exits 2 on a missing path and `!` inverts it.
+
+```bash
+grep -rn 'grep.*"\$REPO_ROOT/[^"]*"' tests/*.bats | head -50
+[ -f "$path" ] || echo "MISSING: a seam may be grepping this"
+```
+
+Fix shape — one place asserts the invariant, each local site keeps a live
+check:
+
+```bats
+@test "no <host> compose file competes with the quadlet units" {
+    local cf="$REPO_ROOT/<host>/compose.yaml"
+    [ ! -f "$cf" ] || { echo "$cf is back: every container is unit-managed"; return 1; }
+}
+# at the per-service site: still fails if the entry returns
+[ ! -f "$cf" ] || ! grep -qE '^\s+<svc>:' "$cf"
+```
+
+**Prove teeth:** recreate the offending file
+(`printf 'services:\n  <svc>:\n    image: x\n' > <path>`), run the seams,
+confirm RED, then move the file to trash. A re-armed check you have never
+seen fail is still unverified.
+
+## 3. Gates that never run
+
+Playwright specs in `tests/e2e/` are matched by **number prefix** in
+`playwright.config.ts`; an unregistered spec silently never runs and the
+suite count stays the same. Count tests before/after;
+`tests/e2e_scaffold.bats` fails on an unregistered spec by design.
+
+## 4. Validate the checker before acting on it
+
+A checker's output is a hypothesis about another tool's semantics. Modelling
+Obsidian with naive text matching produced 45 "broken links" of which 3 were
+real — the rest were case-sensitivity (a `qBittorrent` link to
+`qbittorrent.md`), path-qualified resolution (a path-qualified
+`folder/Note` link resolves by path SUFFIX, not
+stem), and `%20` URL-encoding in markdown links. Fix the checker first,
+re-derive the report, only then edit documents. The result lives in
+`scripts/check-vault-links.py` with those three rules in its docstring,
+gated by the `no unresolved links in the docs vault` seam. Same class: a
+`find … | head -1` selector that reads the wrong file; a truncated listing
+that makes you "fix" a file that exists.
+
+## 5. Prose invalidated by your own change
+
+After retiring a file or adding a feature, grep for EVERY reference, not just
+code:
+
+```bash
+grep -rn "<retired-file>" --include="*" . | grep -v '^./.git/'
+grep -rn "no <feature>\|none —\|unconfigured" --include="*.md" --include="*.container" --include="*.yml" .
+```
+
+This catches the ones that bite: AGENTS.md said ComfyUI runs CPU-only long
+after it was restored with GPU access; a unit said "there are no alerting
+rules" hours after five were added; a docs page documented three scrape jobs
+as working when they could never report up. **Clean cutover includes prose** —
+docs, unit docstrings, and the repo's rule file (the most expensive place to
+be stale).
+
+## Order and verification
+
+1. Enumerate the surfaces (status board, metrics targets, alert rules, test
+   gates, docs/rule files, dashboard panels).
+2. For each, ask what the signal CAN express, not what it says.
+3. Validate any checker before editing the things it flags.
+4. Fix at the root; never weaken an assertion to make it pass.
+5. Gate the class so it cannot return (a seam), then prove the seam fails on
+   the defect.
+
+Verify: `./scripts/check-vault-links.py` (exit 1 on breaks); query each
+Prometheus panel expression against the API (0 series on a panel that should
+show data is a defect; 0 `ALERTS` when nothing fires is not); read the
+deployed `/srv/homepage/status.json` and triage every `state != up` entry
+against reality; query `/api/v1/targets?state=active` and print
+`health != "up"` with `lastError`; `./tests/run.sh` → `ALL TESTS PASSED`.
+Report per finding: the surface, why the signal could not have been correct,
+the fix, and the red/green evidence — flag any regression you introduced
+yourself in the same breath (one was found this way: a tightened bind killed
+a reverse-proxied vhost while every gate stayed green, because the spec
+covering that hostname did not exist).
+
+## Related
+
+- [[Testing Policy]] — the seam inventory this audit walks
+- [[Observability]] — the scrape/probe surfaces it audits
+- [[Quadlet Lifecycle]] — the ownership labels the probes read

serverAppConfigs-docs/Dankclaw/ComfyUI Qwen-Image Workflows.md

@@ -0,0 +1,121 @@
+---
+type: howto
+description: "Qwen-Image 2.1 workflows on Dankclaw: two proven txt2img workflows (art + legible text), the API recipe, and which models/LoRAs to add next."
+tags:
+  - dankclaw
+  - comfyui
+  - qwen-image
+timestamp: 2026-10-05
+---
+
+# ComfyUI Qwen-Image Workflows
+
+Two ready-to-load text-to-image workflows for the Qwen-Image 2.1 payload
+already on Dankclaw, both proven end-to-end on 2026-10-05 by POSTing to the
+live API on port 8188 and verifying the rendered PNG. The repo copies are the
+source of truth; the host copies under `/home/dank/comfyui-data/workflows/`
+are for convenience.
+
+## What is on disk
+
+| Path (under `/home/dank/comfyui-data/models/`) | Size | Role |
+|---|---|---|
+| `diffusion_models/qwen_image_2.1_int8_convrot.safetensors` | 6.8 GB | **The one the workflows use** — int8-convrot quant, ComfyUI-core native |
+| `diffusion_models/qwen_image_2.1_bf16.safetensors` | 14 GB | Full-precision fallback |
+| `diffusion_models/qwen-image-2.1-Q8_0.gguf` (+ `-arch`) | 7.1 GB | GGUF alternative (needs ComfyUI-GGUF custom node) |
+| `text_encoders/qwen3vl_8b_w4a8.safetensors` | 5.9 GB | **The one the workflows use** — 4-bit-weights/8-bit-act Qwen3-VL 8B encoder |
+| `text_encoders/qwen3vl_8b_int8_convrot.safetensors` | 8.8 GB | Mid quant |
+| `text_encoders/qwen3vl_8b_bf16.safetensors` | 17 GB | Full precision |
+| `vae/qwen_image_2.1_vae_bf16.safetensors` | 645 MB | 2.1 VAE (used) |
+| `vae/qwen_image_vae.safetensors` | 243 MB | Smaller VAE |
+
+No LoRAs are on disk yet — see [Recommended additions](#recommended-additions).
+
+## The workflows
+
+Both are API-format JSON (ComfyUI 0.37 loads them via the UI's **Load** dialog
+too): `dankclaw/comfyui/workflows/` in the repo,
+`/home/dank/comfyui-data/workflows/` on the host. Node chain in both:
+
+```
+UNETLoader (int8_convrot) ──┐
+CLIPLoader (qwen_image, w4a8) ─┤ CLIPTextEncode pos/neg
+VAELoader (2.1 bf16) ────────┤
+EmptySD3LatentImage ─────────┴─ KSampler ─ VAEDecode ─ SaveImage
+```
+
+**Common sampler settings:** `cfg 1.0` (Qwen-Image 2.1 expects CFG-free
+sampling), `euler` + `simple`, seed fixed in the file.
+
+| Workflow | File | Shape | Steps | Made for |
+|---|---|---|---|---|
+| Art | `qwen-image-2.1-t2i-art.json` | 1024×1024 | 20 | General illustration/prose prompts |
+| Legible text | `qwen-image-2.1-t2i-legible-text.json` | 832×1216 | 25 | Posters, captions, any in-image text |
+
+### Legible-text prompt recipe
+
+Qwen-Image 2.1 is one of the best open-weight text renderers, but only if the
+prompt is written for it:
+
+1. **Quote the exact strings**: `large bold white text reading "SOLIDARITY OF
+   THE SOVIET COMPUTER SYSTEMS"` — quoted text is rendered literally.
+2. **Describe layout, not vibes**: say *centered / below it / large vs small*,
+   the typeface family (sans-serif), and the color.
+3. **Say what NOT to draw**: end with `no other words, no watermark`.
+4. Keep the negative side on `misspelled, warped letters, extra text` (already
+   in the workflow).
+5. More steps help text: the text workflow uses 25 vs 20.
+
+Verified render: `SOLIDARITY OF THE SOVIET COMPUTER SYSTEMS` + `DANKCLAW
+DANKSERVER EST 2026` came back crisp and correctly spelled at 832×1216
+(`qwen21-text_00001_.png`, 953 KB). One caveat from the same render: a
+shorter secondary line is where misspellings creep in — proofread the first
+render before shipping.
+
+## Driving it from the API
+
+ComfyUI 0.37 **requires the graph wrapped in a `prompt` key** — a bare
+`{node-id: node}` body gets `{"error": {"type": "no_prompt"}}` (verified
+2026-10-05):
+
+```bash
+curl -s -X POST http://127.0.0.1:8188/prompt -H 'Content-Type: application/json' \
+  -d "$(python3 -c 'import json;print(json.dumps({"prompt": json.load(open("dankclaw/comfyui/workflows/qwen-image-2.1-t2i-art.json"))}))')"
+# -> {"prompt_id": "...", ...}
+curl -s http://127.0.0.1:8188/history/<prompt_id>   # status.status_str == "success"
+```
+
+Outputs land in `/home/dank/comfyui-data/output/`. The queue is FIFO — the
+two workflows above ran back-to-back unattended.
+
+**Note:** the container's UI-saved workflows live in the ephemeral
+`/opt/ComfyUI/user` layer and do NOT survive an image rebuild — hence the
+host-side `workflows/` dir and the repo copy.
+
+## Recommended additions
+
+Quantized preferred, sized for the 8060S iGPU (GTT-backed; the RTX 3090 is
+frank-secondary's). All repos verified 2026-10-05. None are on disk yet.
+
+| # | What | Repo | Grab | Size | Why |
+|---|---|---|---|---|---|
+| 1 | **Lightning 8-step LoRA** (T2I speed) | `lightx2v/Qwen-Image-Lightning` | `qwen_image_lightning_8steps_*.safetensors` (LoRA) | ~1–2 GB | 20 steps → 8 with a `LoraLoader` node; 2–3× faster renders on the iGPU, negligible quality loss |
+| 2 | **Qwen-Image-Edit-2511 GGUF** (image editing) | `unsloth/Qwen-Image-Edit-2511-GGUF` | Q4_K_M (~13.2 GB) or Q4_0 (~11.9 GB) | ~12–13 GB | Unlocks img2img "edit this photo" workflows — a capability class the fleet has zero of; Q4 keeps it iGPU-friendly |
+| 3 | **DiffSynth ControlNets** (structure control) | `Comfy-Org/Qwen-Image-DiffSynth-ControlNets` | `split_files/model_patches/` canny + depth | ~2–3 GB each | Pose/edge/depth-guided layouts — pairs with the legible-text workflow for exact poster composition. The `QwenImageDiffsynthControlnet` node is already in the running ComfyUI |
+| 4 | **Wan 2.2 TI2V-5B GGUF** (first video model) | `QuantStack/Wan2.2-TI2V-5B-GGUF` | Q4/Q5 GGUF (~6–8 GB) | ~6–8 GB | T2V+I2V in one dense 5B model; the gentlest on-ramp into video generation, runs in ComfyUI core (`wan` CLIP type is already in this build's loader enum) |
+
+Placement: #1 → `models/lora/` (create it), #2 → `models/diffusion_models/`,
+#3 → `models/controlnet/` (create it), #4 → `models/diffusion_models/`.
+Fetch with `scripts/download-frank-*.sh` discipline (`-f` + byte-count +
+sha256 checks) or a plain `hf download` with the token in the environment —
+never on the command line.
+
+Skip list: the original `city96/Qwen-Image-gguf` conversions (superseded by
+the 2.1 files already on disk), and any BF16 of #2/#4 (the iGPU does not need
+them at GTT capacity, and they double the disk bill).
+
+## See Also
+
+- [[ComfyUI]] — the service itself (image, unit, GPU access)
+- [[ComfyUI ROCm and Vulkan]] — the live-verified GPU path
+- [[Dankclaw Stack]] — the host

serverAppConfigs-docs/Dankclaw/ComfyUI.md

@@ -110,5 +110,6 @@ The web UI responds on `/`. ComfyUI also exposes `/system_stats`. On this APU, `
 
 - [[Dankclaw Stack]]
 - [[ComfyUI ROCm and Vulkan]] — the live-verified GPU path research
+- [[ComfyUI Qwen-Image Workflows]] — the Qwen-Image 2.1 payload, proven workflows, and recommended model/LoRA additions
 - [[Lemonade and sd.cpp]] — the ggml-family alternative
 - Knowledge base: [[ComfyUI]] (tools)

serverAppConfigs-docs/Dankclaw/Hermes.md

@@ -81,6 +81,178 @@ The wiring lives in the live `/data/config.yaml` under `mcp_servers.chrome-devto
 The npx package cache lives in the container layer, so a recreation (not just a
 restart) pays one first-call npm download; subsequent calls are cached.
 
+## Config-home drift ("cannot connect" to models that answer curl)
+
+`podman exec <container> printenv HERMES_HOME` for each hermes container. If
+the webui reads the image default (`/opt/data`) while the gateway reads
+`/data`, the webui runs the image-default template config (`provider: auto` →
+openrouter.ai with no key = connect failure).
+
+The recreation trap is the same one the volumes table shows: image `VOLUME`
+points get a fresh anonymous volume on every container recreation
+(`podman run --replace --rm`) — edits inside such a mount are lost on
+restart/redeploy. Compare volume ids before/after a restart to confirm.
+
+Fix: pin `Environment=HERMES_HOME=/data` in the quadlet unit (shared state
+root), deploy service-scoped. Verify with the app's own code, not guesses:
+
+```python
+from api import config as C; C.resolve_model_provider("<model-id>")
+# -> (model_id, provider, base_url); bare ids map to the owning provider by
+#    scanning the providers: dict models allow-list; @provider:model forces one.
+hermes_cli.providers.resolve_user_provider(name, cfg["providers"])
+# -> ProviderDef with real base_url; accepts api/url/base_url keys.
+```
+
+Named providers (`providers:` dict) are a catalog + connection source;
+`custom:<slug>` connections resolve only via `custom_providers[]` entries. If
+neither holds the provider name, the request has no endpoint and fails to
+connect. Probe the endpoint with the **exact served model id** — the retired
+FLM/NPU endpoints routed only by exact registry tag and answered HTTP 200
+with `{"error":"Failed to load <tag> model!"}` for unknown tags; status-code
+checks miss this class entirely.
+
+## Live config edits
+
+Live file: `/home/dank/Containers/hermes/data/config.yaml` (dir `drwx------`
+uid 109999; all reads/edits need `sudo`). The repo file
+`dankclaw/hermes/config.yaml.example` is a sanitized backup and **can lag
+live — always diff the live file first; never treat the example as truth**.
+
+1. **Backup first:** `sudo cp -a …/config.yaml …/config.yaml.bak-YYYYMMDD`
+   (in-dir `.AGENTTRASH/` also exists for removals).
+2. **Apply** with surgical `sudo sed -i` or full-file write; YAML comments in
+   the live file are tool-generated — preserve them.
+3. **Sweep stale references** across the whole data dir:
+   `sudo grep -rln "<old-id>" /home/dank/Containers/hermes/data
+   --include="*.yaml" --include="*.json"`. Session files/MEMORY.md are
+   historical — leave them.
+4. **Derived caches:** `webui/models_cache.json` is keyed by a config
+   fingerprint (`_source_fingerprint.config_yaml` mtime/size); after edits it
+   is stale. Let it regenerate lazily or trash it — never hand-edit.
+5. **Restart the model-consuming units** (they parse config at startup):
+   `systemctl --user restart hermes-webui.service hermes.service`.
+   `hermes-dashboard.service` is unaffected by routing changes.
+6. **Verify behaviorally**: one tiny completion per endpoint with the exact
+   served id — POST `/v1/chat/completions` and read the body, never just the
+   status code (endpoint names rotate; see the LLM knowledge base).
+7. **Sync the repo backup** (`dankclaw/hermes/config.yaml.example`) to the
+   live shape, redacting secrets (`password_hash`, phone account stay
+   live-only), then `./tests/run.sh` before commit.
+
+`provider: custom` + `base_url` entries route by base_url and send the model
+id verbatim; named providers resolve via their `api:` URL. **Renaming a
+served alias anywhere (serve scripts, env ALIAS) MUST propagate to this
+config in the same change** or Hermes requests 404 on nonexistent models —
+this exact gap caused the 2026-09-05 outage.
+
+## Auxiliary routing against local backends
+
+When a Hermes feature that calls an auxiliary task (e.g. title generation)
+fails against a local endpoint, the usual cause is a **request shape the
+local runtime does not implement**, not a routing mistake.
+
+Diagnose with the real call path — run the actual function inside the
+container with the app's own constants (the probe cannot drift from
+production):
+
+```bash
+base64 -w0 /tmp/probe.py | ssh dank@192.168.0.143 \
+  'base64 -d > /tmp/probe.py && systemctl --user restart hermes.service && sleep 20 && \
+   podman cp /tmp/probe.py systemd-hermes:/tmp/probe.py && \
+   podman exec -i systemd-hermes /opt/hermes/.venv/bin/python /tmp/probe.py'
+```
+
+Copy the file AFTER any restart (`--replace --rm` discards the container
+layer). The CLI at `/opt/hermes/hermes` uses system python and dies
+`No module named 'yaml'`; use `/opt/hermes/.venv/bin/python`.
+
+```python
+import sys; sys.path.insert(0, "/opt/hermes")
+from agent.title_generator import _TITLE_PROMPT_TEMPLATE, _TITLE_RESPONSE_FORMAT, _clean_title, _extract_title_text
+from agent.auxiliary_client import call_llm
+resp = call_llm(task="title_generation", messages=[...], max_tokens=64,
+                temperature=0.3, extra_body={"response_format": _TITLE_RESPONSE_FORMAT})
+```
+
+What actually breaks, and the fix:
+
+- **`response_format` (json_object / json_schema) is a hard 400 on every
+  local backend here** — neither FreeToken nor halogen implements constrained
+  decoding. Config cannot override it: `call_llm` merges the task's
+  `extra_body` first and then `update()`s the caller's, so the hardcoded one
+  wins. Only temperature and max_tokens have reactive retry paths.
+- **Fix: move that one task to the Responses transport**, which never
+  forwards `response_format`:
+
+  ```yaml
+  auxiliary:
+    title_generation:
+      provider: dankserver-small        # names track the live config; diff first
+      model: frank-secondary
+      api_mode: codex_responses        # real per-task key
+      extra_body:
+        reasoning:
+          effort: none                 # mandatory, see below
+  ```
+
+- **Disable thinking for any cap-limited task.** The adapter omits reasoning
+  only when config says `reasoning.enabled: false`. The
+  `reasoning_effort: none` *shorthand* produces exactly that and leaves the
+  server default (thinking ON) — the 64-token title budget gets spent on the
+  think: empty content, `finish_reason: length`. The explicit
+  `extra_body.reasoning.effort: none` instead sends
+  `{"effort":"none","summary":"auto"}`, which the local `/v1/responses`
+  honors.
+- Both local runtimes also answer `/v1/responses` and `/v1/messages` — no
+  proxy needed for the alternate transport.
+
+Verify and land: re-run the probe with **no** `api_mode`/reasoning kwargs —
+it must pass from config alone (that is the production path); keep the live
+file and the repo example in step; `./tests/run.sh`, commit, push — a Hermes
+config change is docs+example only, no service deploy needed.
+
+Gotchas: `auxiliary.<task>.enabled: false` is checked BEFORE
+`apply_instant_title`, so disabling a broken title task also removes the
+derived title — it does not degrade gracefully the way a failure does. A
+local call can also fail at the model level: a 200 with an error-text body,
+or a chatty non-title answer. Always read the raw content, not the status.
+
+## Webui password rotation ("Hermex")
+
+Auth contract (verified live 2026-08-31): `POST /api/auth/login` with JSON
+`{"password":"…"}` → 200 + signed HttpOnly `hermes_session` cookie (24 h TTL)
+on match, 401 on miss. No username field. `/health` answers 200
+unauthenticated and `/` answers 302 — **never verify auth via those**; probe
+login with a deliberately wrong password (expect 401) and the real one
+(expect 200). Details on the webui surface: [[Hermes-WebUI]].
+
+1. Run the whole probe as a heredoc'd bash script
+   (`ssh -o BatchMode=yes dankclaw 'bash -s'`) — the host's zsh mangles
+   inline `${…}` even inside single-quoted ssh args; never interpolate curl
+   `-w %{http_code}` inline.
+2. Update BOTH env files with
+   `sed -i "s/^HERMES_WEBUI_PASSWORD=.*/HERMES_WEBUI_PASSWORD=<new>/"`:
+   - `/home/dank/serverAppConfigs/.env` ([Service] EnvironmentFile)
+   - `/home/dank/serverAppConfigs/dankclaw/.env` — **the effective one**: the
+     quadlet's `[Container] EnvironmentFile=` is what the container reads at
+     create; rotating only the root file leaves the running container serving
+     the old secret.
+3. `systemctl --user restart hermes-webui.service` (unit recreation re-reads
+   env; no deploy needed if the unit file itself didn't change).
+4. Verify wrong-pw → 401, right-pw → 200, from host loopback **and** a LAN
+   client at `http://192.168.0.143:8787` (webui publishes on the LAN
+   interface since commit `4443897`; off-network ingress is SSH tunnel only —
+   user `tunnel`, port **22**, key-only auth, forward remote end
+   localhost:8787).
+
+Gotchas: `sudo podman exec` fails exit 125 here — root's namespace does not
+see rootless containers; run `podman exec hermes-webui printenv
+HERMES_WEBUI_PASSWORD` unprivileged (the ssh user owns the env files, mode
+600). If rotating because a secret leaked into transcripts/logs, rotate in
+place via a script that fetches the live value from the running container
+(`printenv`) instead of transcribing it.
+
 ## See Also
 
 - [[Hermes-Dashboard]] — the LAN browser management UI

serverAppConfigs-docs/Dankclaw/OpenWebUI.md

@@ -24,7 +24,7 @@ https://github.com/open-webui/open-webui
 | Upstream latest | 0.11.4 (2026-09-21) — security/access-control advisory; `:main` is the rolling image, not a stable pin. Imagegen endpoint is `POST /api/v1/images/generations` (the `comfyui` engine is still upstream). |
 | Port | 3005 (host) → 8080 (container). Port 3000 is Gitea's HTTP, so it is not usable. |
 | Data volume | `/home/dank/Containers/open-webui/data` → `/app/backend/data` |
-| Backends | Three fleet endpoints: `frank-large` (host.containers.internal:7770), `frank-medium` (192.168.0.235:7770), `frank-small` (host.containers.internal:7771) — see [The LLM backends](#the-llm-backends) |
+| Backends | Three fleet endpoints: `frank-large` (host.containers.internal:7770), `frank-medium` (192.168.0.235:7770), `frank-secondary` (host.containers.internal:7771) — see [The LLM backends](#the-llm-backends) |
 | Network | Pinned bridge `systemd-open-webui` (`10.89.3.0/24`, gateway `10.89.3.1`) — see [History](#history) |
 | Persistence | Quadlet unit `open-webui.service`; `[Install] WantedBy=default.target` |
 | Deploy | `scripts/deploy-open-webui-quadlet.sh dankclaw` |
@@ -39,14 +39,14 @@ discovery — so the picker shows exactly the models seeded there.
 |--------------|----------------|---------|------|
 | `frank-large` | `http://host.containers.internal:7770/v1` | unsloth llama.cpp (Strix Halo iGPU, 200k ctx) | Dankclaw |
 | `frank-medium` | `http://192.168.0.235:7770/v1` | vllm-radiance TP=2 (FP8, DFlash2, 262k ctx) | Dankserver |
-| `frank-small` | `http://host.containers.internal:7771/v1` | llama.cpp (RTX 3090, Gemma4-12B, 262k ctx) | Dankclaw |
+| `frank-secondary` | `http://host.containers.internal:7771/v1` | llama.cpp (RTX 3090, Gemma4-12B, 262k ctx) | Dankclaw |
 
 Two gotchas make the wiring non-obvious:
 
 - **llama.cpp `/v1/models` is Ollama-shaped.** llama.cpp returns
   `{"models":[{"name":...}]}`; OpenWebUI's `extract_data()` only reads the
   OpenAI `{"data":[{"id":...}]}` shape, so connection discovery silently misses
-  the llama.cpp endpoints (frank-large, frank-small). The fix is to set
+  the llama.cpp endpoints (frank-large, frank-secondary). The fix is to set
   `model_ids` in each connection's `api_config` — when non-empty, OpenWebUI
   builds the model list directly and never fetches `/v1/models`. frank-medium
   (vLLM) serves the OpenAI shape and would discover fine, but uses `model_ids`
@@ -64,6 +64,49 @@ The Ollama connection is disabled (`ollama.enable=false`).
 > repository. Model swaps, context sizing, and spec tuning happen there. See
 > the [LLM knowledge base](../LLM/index.md).
 
+## Container introspection (DB and backend python)
+
+- Working dir for imports: `/app/backend`; package name `open_webui`
+  (underscore), NOT `openwebui`. `os.environ["WEBUI_SECRET_KEY"]` MUST be set
+  BEFORE importing any `open_webui` module (import-time hard requirement);
+  the secret lives at `/app/backend/.webui_secret_key`.
+- **The `sqlite3` CLI is NOT in the container** — use Python's `sqlite3`
+  module (`podman exec open-webui python3 -c "import sqlite3;
+  conn = sqlite3.connect('/app/backend/data/webui.db'); …"`). For config
+  keys use parameterized LIKE queries (avoids shell quoting).
+- Backend python: `podman exec -w /app/backend open-webui python3 -c "…"` —
+  `Models.get_all_models()` is a coroutine (await it); model objects are
+  pydantic `ModelModel` instances (attribute access, not `.get()`), with NO
+  `status`/`enabled` attribute — use `is_active` from the DB.
+- **API auth for verification:** `create_token` lives in
+  `open_webui.utils.auth` (NOT `open_webui.utils.utils`);
+  `get_verified_user` requires an `id` claim (not just `sub`);
+  `SESSION_SECRET = WEBUI_SECRET_KEY`. Mint an admin token inside the
+  container:
+
+  ```
+  podman exec -i open-webui sh -c 'export WEBUI_SECRET_KEY=$(cat /app/backend/.webui_secret_key); \
+    python -c "from open_webui.utils.auth import create_token; \
+    print(create_token({\"id\":\"<UID>\",\"sub\":\"<UID>\",\"email\":\"…\",\"role\":\"admin\"}))"'
+  ```
+
+  (admin UID from the `user` table).
+
+## Verification battery
+
+1. `GET /api/v1/models/list?limit=100` → returns EXACTLY the intended
+   models. Response key is `items`, not `data`.
+2. `POST /openai/chat/completions` per model with a trivial prompt ("Reply
+   with exactly one word: pong") → each returns content `pong`. Reasoning
+   models need `max_tokens>=100` — the thinking block eats the budget.
+3. A 500 "Server Connection Error" → `podman logs open-webui` for
+   `ClientConnectorError`: a network-reachability problem (host LAN IP vs
+   `host.containers.internal`), not a model problem.
+
+**Durability:** the model table + connections are DB-only (not git-managed);
+a fresh data volume loses them. For reproducibility, add an idempotent
+first-start seed script or document the setup in the service doc.
+
 ## Web search
 
 Chat web search is served by [[SearXNG]] on this host, wired in the quadlet
@@ -147,6 +190,23 @@ The quadlet carries a built-in healthcheck (30s interval, 3 retries, 300s start
 
 `podman ps` shows the container as `unhealthy` if either fails. Probe manually at `http://dankclaw:3005/health`.
 
+## Image generation (ComfyUI)
+
+Image generation rides the local [[ComfyUI]] integration, not the chat
+picker — ComfyUI's models never appear in the chat model picker (that is
+chat LLMs only), so ComfyUI needs its own image-generation connection
+configured in OpenWebUI. ComfyUI itself runs GPU-enabled on Dankclaw and
+serves `:8188`.
+
+Endpoint: `POST /api/images/generations` (image `0.11.x`: upstream moved the
+route to `POST /api/v1/images/generations`; the `comfyui` engine is still
+upstream) with `Authorization: Bearer <token>` and body
+`{"model":"imagegen","prompt":"…","n":1,"size":"1024x1024"}`; the response
+carries base64 image data or a URL.
+
+Failure modes: connection refused ⇒ OpenWebUI or ComfyUI is down; timeouts
+on large sizes ⇒ check ComfyUI resources; 401 ⇒ the token is bad.
+
 ## Retired instances
 
 - The compose `open-webui` service (port 3000) was never run: 3000 is Gitea's HTTP port, so it could not bind.
@@ -158,13 +218,13 @@ The quadlet carries a built-in healthcheck (30s interval, 3 retries, 300s start
 - The image tracks upstream `main`; pin a digest in the quadlet unit if reproducible rollback matters more than tracking. (The old "CPU-only host" framing is stale: this APU runs the gfx1151 ROCm build — see [[ComfyUI]].)
 - The default admin account is created on first login; secure it immediately.
 - If the backend is loading (3–8 minute model load), OpenWebUI shows connection errors until the model is ready.
-- The on-host LLM services (frank-large `:7770`, frank-small `:7771`) are reached via `host.containers.internal`, NOT the host LAN IP: the container's netavark bridge cannot hairpin to its own host's non-published ports via `192.168.0.143` (connection refused), and the bridge gateway (`10.89.3.1`) lives in the rootless netns while llama-server runs in the host netns, so it refuses too. `host.containers.internal` (169.254.1.2) resolves to the host on the bridge and works — verified end-to-end for both endpoints. The Dankserver backend (frank-medium) is a different host, reached via its LAN IP `192.168.0.235:7770`.
+- The on-host LLM services (frank-large `:7770`, frank-secondary `:7771`) are reached via `host.containers.internal`, NOT the host LAN IP: the container's netavark bridge cannot hairpin to its own host's non-published ports via `192.168.0.143` (connection refused), and the bridge gateway (`10.89.3.1`) lives in the rootless netns while llama-server runs in the host netns, so it refuses too. `host.containers.internal` (169.254.1.2) resolves to the host on the bridge and works — verified end-to-end for both endpoints. The Dankserver backend (frank-medium) is a different host, reached via its LAN IP `192.168.0.235:7770`.
 
 ## History
 
 - **2026-08-22 — slirp4netns → pinned bridge.** The unit ran on `Network=slirp4netns`. The slirp4netns helper process died (no OOM, no kill entry, no crash log found), and the container kept running with only `lo` in its netns: published port 3005 accepted TCP and reset every request, so chat served empty responses for ~2 days. Nothing logged an error. Moving to a kernel bridge removes the per-container helper entirely, and the healthcheck's interface check makes the zombie state visible if it ever returns.
 
-- **2026-09-21 — model picker rebuilt for the 3-model fleet.** The picker carried 39 stale rows (37 dead Ollama models + 2 `frank-default-*`) and was missing `frank-medium`. Rebuilt the `model` table to exactly `frank-large`/`frank-medium`/`frank-small` and set `model_ids` on each connection (llama.cpp's Ollama-shaped `/v1/models` is invisible to OpenWebUI's discovery). Pointed the two on-host endpoints at `host.containers.internal` — the bridge refuses the host LAN IP for non-published ports. All three verified end-to-end via `/openai/chat/completions`.
+- **2026-09-21 — model picker rebuilt for the 3-model fleet.** The picker carried 39 stale rows (37 dead Ollama models + 2 `frank-default-*`) and was missing `frank-medium`. Rebuilt the `model` table to exactly `frank-large`/`frank-medium`/`frank-secondary` and set `model_ids` on each connection (llama.cpp's Ollama-shaped `/v1/models` is invisible to OpenWebUI's discovery). Pointed the two on-host endpoints at `host.containers.internal` — the bridge refuses the host LAN IP for non-published ports. All three verified end-to-end via `/openai/chat/completions`.
 
 - **2026-09-24 — SSO via authentik.** OIDC authorization-code flow wired per the [SSO (authentik)](#sso-authentik) section: application/provider/entitlements created in authentik, client secret in the host env file, roles from the `roles` scope claim. E2E coverage in `tests/e2e/specs/18-openwebui-sso.spec.ts`.
 

serverAppConfigs-docs/Dankclaw/Swap Configuration.md

@@ -117,6 +117,18 @@ frank-large `health=200` throughout.
   the model is loaded. Trust the engine's `host memory left for everything
   else` line, not `free`.
 
+- **Filesystem prerequisite (other hosts).** ext4 and xfs take swapfiles
+  directly. btrfs needs `chattr +C` (NOCOW), no compression, and no
+  snapshots in the swapfile path — check `df -T` before planning, or
+  `mkswap` will reject the file.
+- **Renaming an *active* swapfile works (inode-based) but leaves a stale
+  path in `/proc/swaps`** — swap it off first, then rename, then `swapon`
+  (which is exactly what the step-3 consolidation does). Migration is
+  observable while it runs: in the 2026-09-12 run `swapoff` of 4 GiB took
+  65 s and used-RAM rose only ~3 GiB (pages touched during migration), the
+  rest relocating on disk. If used-RAM climbs toward the whole swapped
+  amount, let it finish rather than aborting.
+
 ## Trash
 
 The previous 8 GiB `/swap.img` (dated 2026-07-21) is staged, not deleted, at

serverAppConfigs-docs/Desktop/Noctalia and Niri.md

@@ -0,0 +1,303 @@
+---
+type: reference
+title: Noctalia and Niri
+description: The noctalia v5 desktop shell on niri (workstation + blankpad) — config layers, remote IPC over ssh, custom-palette recolor, offline template rendering, and output rotation/pinning.
+tags: [desktop, noctalia, niri, blankpad, workstation, ghostty]
+timestamp: 2026-10-05T00:00:00-07:00
+---
+
+# Noctalia and Niri
+
+Noctalia (`/usr/bin/noctalia`) is the desktop shell on niri, running on both
+the workstation and blankpad. Since the 2026-09-10 unification both run the
+**bare `noctalia` v5 binary** (niri `spawn-at-startup "noctalia"`); the old
+Quickshell-hosted v4 architecture (`qs -c noctalia-shell`, JSON config) is
+retired on both hosts.
+
+## Config layers
+
+- Hand-edit declarative layer: `~/.config/noctalia/<name>.toml` (e.g.
+  `config.toml`). Main config: `~/.config/noctalia/config.toml`; the niri
+  config includes `noctalia.kdl` for display settings.
+- Runtime state file `~/.local/state/noctalia/settings.toml` loads LAST and
+  **overrides declarative values for the same keys — never hand-edit it**;
+  UI/IPC writes land there.
+- Main reference: `noctalia-reference` facts — IPC `noctalia msg <command>`
+  replaced the `qs -c noctalia-shell ipc call` wrapper; app templates live at
+  `/usr/share/noctalia/assets/templates/` (gtk, ghostty, alacritty, hyprland,
+  sway, wezterm, helix, btop, starship, qt, kde, cava, scroll, mango,
+  umbriel, foot, emacs, kitty); palette generation `noctalia theme [image]`.
+
+## Remote IPC over ssh (the D-Bus trap)
+
+`noctalia msg <command>` resolves the running instance via the D-Bus session
+bus. sshd-spawned non-login shells lack `DBUS_SESSION_BUS_ADDRESS`, so plain
+ssh exec prints `error: noctalia is not running` even when the shell IS
+running — do not conclude "shell dead" from this error. Working remote
+procedure (runs in the systemd user-manager context where the bus address
+exists):
+
+```
+systemd-run --user --pipe --wait noctalia msg <command> [args]
+```
+
+Verified: `msg status` returns live JSON state; unknown panel ids return an
+error listing all valid panel ids (clipboard, control-center, launcher,
+polkit, session, setup-wizard, test, tray-drawer, wallpaper).
+
+Verified v5 msg verb grammar (keybinds-safe): `panel-toggle <id>`,
+`session lock|suspend|lock-and-suspend|logout|reboot|shutdown`,
+`volume-up [step]`, `volume-down`, `volume-mute`, `mic-mute`,
+`media next|previous|playPause` (graceful no-op, rc 0, with no active
+player), `brightness-up [target] [step]`, `brightness-down`,
+`settings-open [context]`, `settings-toggle [context]`, `config-reload`,
+`wallpaper-get [connector]`, `wallpaper-set <path>`, `wallpaper-next`,
+`color-scheme-get|set`, `theme-mode-get`, `templates-apply`,
+`screenshot-fullscreen`.
+
+**Wallpaper directory semantics:** empty/unset `[wallpaper] directory`
+resolves to XDG Pictures via user-dirs.dirs; an explicit directory key
+overrides. Fleet default pattern: point `directory` at the repo-tracked
+wallpapers dir so git-tracked wallpapers serve the shell without copying
+files. After writing config.toml over ssh the file watcher hot-reloads
+automatically; `config-reload` via systemd-run forces it immediately. Note:
+the *current* wallpaper stays the built-in default until the next rotation
+or an explicit `wallpaper-next`; `wallpaper-get` then reporting a path
+inside the configured directory is the confirmation the config took effect.
+
+**Lock chain (unified):** swayidle-lock.service ExecStart on both hosts:
+`/usr/bin/swayidle -w timeout 300 '/usr/bin/noctalia msg session lock'
+before-sleep '…' after-resume '…'`. `msg` resolves via the session bus, so
+no WAYLAND_DISPLAY pinning is needed in unit context.
+
+## Custom-palette recolor (without restarting the session)
+
+Why custom, never the community cache:
+`~/.local/state/noctalia/community-palettes/*.json` is a **sync cache** —
+`.catalog/palettes.json` carries an `md5` per palette and noctalia re-fetches
+from `https://api.noctalia.dev/palettes`, so edits there get reverted.
+Custom palettes live at `~/.config/noctalia/palettes/<Name>.json`
+(`NOCTALIA_CONFIG_HOME` then `XDG_CONFIG_HOME` honored) and are never
+synced. Missing/invalid file → silent fallback to the builtin palette.
+
+Palette JSON shape = the community file's shape: `dark`/`light` maps of
+`mPrimary`, `mOnPrimary`, `mSecondary`, `mTertiary`, `mError`, `mSurface`,
+`mSurfaceVariant`, `mOutline`, `mShadow`, `mHover` plus a `terminal` block
+(`foreground`, `background`, `normal.{black..white}`, `bright.{…}`,
+`cursor`, `cursorText`, `selectionFg`, `selectionBg`). NOTE: `noctalia theme
+-o` emits M3 snake_case tokens (`on_primary_container`, …) — that is the
+*generator* format, not this one. Copy an existing community JSON and mutate
+with `jq`.
+
+Procedure:
+
+1. Read the live scheme first — never assume: `noctalia msg color-scheme-get`
+   (e.g. `community Gruber Darker`); `noctalia msg theme-mode-get`.
+2. Back up settings: `cp -a ~/.local/state/noctalia/settings.toml
+   /tmp/noctalia-settings-backup-$(date +%s).toml`.
+3. Build the palette from the current one (jq keeps structure):
+
+   ```sh
+   mkdir -p ~/.config/noctalia/palettes
+   jq '.dark.mPrimary = "#f43841"
+       | .dark.terminal.normal.yellow = "#f43841"
+       | .dark.terminal.bright.yellow = "#ff4f58"
+       | .dark.terminal.foreground   = "#f43841"' \
+      "$HOME/.local/state/noctalia/community-palettes/Gruber%20Darker.json" \
+      > ~/.config/noctalia/palettes/"Gruber Red.json"
+   ```
+
+   Prefer a hex already present in the palette (its own `mError` red) so the
+   theme stays internally consistent. Set BOTH `normal` and `bright`
+   variants, and both `dark` and `light` if the user toggles modes.
+4. Switch (persists `theme.source = custom` + `theme.custom_palette`):
+   `noctalia msg color-scheme-set custom "Gruber Red"` → `ok`; confirm with
+   `color-scheme-get`.
+5. Re-render every configured template: `noctalia msg templates-apply`.
+   Regenerates `~/.config/ghostty/themes/noctalia`,
+   `~/.config/gtk-{3,4}.0/noctalia.css`, kcolorscheme, foot, wezterm,
+   starship, etc. Symlinked-into-dotfiles paths show up as modified files
+   worth committing.
+6. Reload surfaces: compositor — `niri msg action load-config-file`
+   (`reload-config` is NOT a valid niri subcommand); ghostty — see the trap
+   below.
+
+Verification: `ghostty +show-config | grep -E '^(foreground|palette)'`
+(proves the theme file resolves — note it prints only the delta from
+defaults; cross-check with `ghostty +show-config --default` before
+concluding a key is unset); `noctalia config validate` → `✓ Config is
+valid`; visual via `noctalia msg screenshot-fullscreen` (saves to
+`~/Pictures/screenshot_<ts>.png` and ALSO overwrites the clipboard with the
+PNG — warn the user); `grep -c <old-hex> ~/.config/noctalia/palettes/*.json`
+→ 0 for no leftovers.
+
+Track it in the dotfiles repo (`~/.omp`, the command-center —
+`scripts/deploy.sh` is a stow-equivalent over `dotfiles/<pkg>/` trees):
+mirror the shape for the new palette so `git pull` + `deploy.sh` is the
+whole propagation story — move the JSON into
+`~/.omp/dotfiles/noctalia/.config/noctalia/palettes/` and symlink it back
+into `~/.config/noctalia/palettes/`; re-set the custom palette (re-reads
+through the symlink). Do the link by hand for one file instead of running
+`deploy.sh` — sync mode quarantines and replaces every divergent live file.
+
+Traps:
+
+- **NEVER `kill -USR1` ghostty to reload.** It does not reload config on
+  this fleet — it kills/respawns the terminal (took out Frank's window
+  2026-09-13). Correct: the default keybind `ctrl+shift+,` = `reload_config`
+  (per `ghostty +show-config --default`). Existing windows keep old colors
+  until reloaded; new windows inherit immediately.
+- Palette hex values are NOT a proxy for what the user sees: in
+  `Gruber Darker` the *accent* (`mPrimary`) is yellow `#ffdd33`, so "change
+  the yellow" means the accent too, not just terminal index 3.
+- Saturating terminal text makes ANSI 1/9 indistinguishable from default
+  text (grep highlights flatten) — flag it and offer a lighter red.
+- Contrast sanity: `#f43841` on `#181818` ≈ 4.6:1 (dark-on-color beats
+  white-on-color here, ~3.9:1 — keep `mOnPrimary` dark).
+- Revert = `noctalia msg color-scheme-set community "<original name>"`; the
+  community cache is untouched by the whole flow.
+- Generated theme files symlinked into the dotfiles repo (`gtk-3.0`/`gtk-4.0`
+  `noctalia.css`, ghostty theme) show up as tracked modifications after
+  `templates-apply`; commit them together with the new palette JSON so every
+  machine gets the same recolor.
+
+## Offline template render (when templates-apply never ran)
+
+Failure signature: niri `failed to read included config from
+~/.config/niri/noctalia.kdl: No such file or directory` — the desktop runs
+but with DEFAULT config (niri 26.04 falls back, does not exit): no custom
+keybinds/rules, no noctalia autostart, default output placement. ghostty:
+`theme "noctalia" not found, tried ~/.config/ghostty/themes/noctalia`.
+alacritty: import of `~/.config/alacritty/themes/noctalia.toml` missing.
+
+Root cause pattern: upstream dotfiles commits theme switches (niri include,
+ghostty `theme = noctalia`, alacritty import) that depend on noctalia
+TEMPLATE-GENERATED files. Those files are per-host outputs (rendered from
+the local palette) and are intentionally NOT in the repo; on any host where
+`templates-apply` never ran, they don't exist.
+
+Fix (offline, works while a session runs) — built-in templates live at
+`/usr/share/noctalia/assets/templates/` (declared in `builtin.toml`); render
+each needed one with the live palette:
+
+```bash
+# niri include (plain host-local file; do NOT symlink into repo — regenerated per host)
+noctalia theme --theme-json ~/.config/noctalia/colors.json \
+  --render "/usr/share/noctalia/assets/templates/niri/niri.kdl:$HOME/.config/niri/noctalia.kdl"
+mkdir -p ~/.config/ghostty/themes && \
+noctalia theme --theme-json ~/.config/noctalia/colors.json \
+  --render "/usr/share/noctalia/assets/templates/ghostty/ghostty:$HOME/.config/ghostty/themes/noctalia"
+mkdir -p ~/.config/alacritty/themes && \
+noctalia theme --theme-json ~/.config/noctalia/colors.json \
+  --render "/usr/share/noctalia/assets/templates/alacritty/alacritty.toml:$HOME/.config/alacritty/themes/noctalia.toml"
+```
+
+All template tokens (primary, surface, error, primary_container, …) resolve
+from the flat 16-key colors.json; container tokens are derived, never blank.
+
+Verify: `niri validate` → rc 0; `ghostty +validate-config` → rc 0; next
+login journal shows `loaded config from config.kdl` with NO parse WARN,
+custom output placement (`niri msg outputs` vs display.kdl), noctalia
+autostarted (`pgrep noctalia`).
+
+Traps: **`noctalia msg templates-apply` on 5.0.1 returns "ok" but applies
+NOTHING** (this build's opt-in config schema does not act on it) — do not
+trust its exit code, use the explicit `--render` calls above. Niri config
+errors at boot are NOT fatal — the compositor keeps running on defaults,
+which reads as "some settings silently stopped working", not a crash.
+Regenerate after any palette change; the rendered files are static
+snapshots.
+
+## Niri output rotation and pinning
+
+Verified on blankpad, niri 26.04 (two PM1601 externals + eDP-1 laptop panel).
+
+**Ground truth first:** `niri msg outputs` (connector names, current mode,
+scale, transform, logical pos/size) and `niri --version`. Output **names are
+connector names** (`eDP-1`, `DP-2`, `HDMI-A-1`), not EDID model strings —
+the EDID strings printed before them are decoration. On blankpad the live
+config splits via `include "./cfg/*.kdl"` from `~/.config/niri/config.kdl`,
+and `cfg/display.kdl` is a **symlink into the dotfiles repo** — edit the
+repo path (that is what gets committed); check with `ls -ld`.
+
+**Transform direction is counter-clockwise** (`niri msg output <NAME>
+transform --help`): `90` = content rotated 90° CCW ⇒ the panel's **top
+bezel sits on the LEFT**; `270` = top bezel RIGHT; `180` = upside down;
+`flipped-90`/`flipped-270` = mirrored variants. `90` vs `270` is a
+physical-stand fact no tool can read (portable monitors expose no
+orientation in EDID/MCCS) — pick one, apply, and hand the user the one-liner
+to flip; don't stall on the question.
+
+Two traps when editing the config:
+
+1. **An `output "X" { … }` block resets every property you don't restate to
+   its default.** Pinning position on `eDP-1` without repeating `scale 1.25`
+   silently drops the laptop panel to scale 1.0. Restate `scale` (and
+   `mode`) in every block you touch.
+2. **Niri re-runs auto-layout for outputs with no pinned position.** Pinning
+   only the externals shoved `eDP-1` from `0,0` to `3936,0`. Pin every
+   output's position to keep the left-to-right order. Positions are logical
+   pixels at each output's own scale; x-offsets chain (eDP-1 logical width
+   1536 → DP-2 at 1536 → +1200 portrait width → HDMI-A-1 at 2736):
+
+```kdl
+output "DP-2"     { scale 1 transform "90" position x=1536 y=0 }
+output "HDMI-A-1" { scale 1 transform "90" position x=2736 y=0 }
+output "eDP-1"    { scale 1.25 position x=0 y=0 }   // MUST restate scale
+```
+
+Apply and verify:
+
+```bash
+niri validate -c ~/.config/niri/config.kdl      # exit 0
+niri msg action load-config-file                # live reload; NOT reload-config
+sleep 1; niri msg outputs | grep -E '^Output|Transform|Logical|Scale'
+```
+
+Success looks like `Logical size: 1200x1920` (swapped) plus
+`Transform: 90° counter-clockwise` on each rotated panel, positions matching
+the intended chain. One-off rotation without touching config (temporary):
+`niri msg output DP-2 transform 90`. Commit via the dotfiles repo
+(`~/.omp`, `main`; push to the local Gitea permitted): `git pull --ff-only`
+→ add the display.kdl → commit → push. Pitfalls: `reload-config` is not a
+niri subcommand; never SIGUSR1 a terminal to "reload" (niri config reloads
+do not touch ghostty themes); no man page for niri on this host — `niri
+<sub> --help` and `/usr/share/doc/niri/default-config.kdl` are the docs.
+
+## GPU RGB over I2C (EVGA iCX3)
+
+Turn RGB on/off (or read it back) on an EVGA FTW3 / iCX3-controller GPU
+(the 3090 on dankclaw) using the GPU's own nvidia i2c bus. No display
+engine, no X, no daemon — the mode is EEPROM-saved and survives reboot.
+
+- **Find the controller:** the nvidia i2c driver exposes the card's lighting
+  ports as `/dev/i2c-N`; on the 3090 FTW3 that is ports 1,5,6,7,8 →
+  i2c-1..i2c-5, and only i2c-1 carries a live slave (NVIDIA port 1). Probe
+  each bus. The iCX3 MCU answers at address **0x2D**. `/dev/i2c-*` are
+  `root:i2c 660` — run with `sudo` (dank has passwordless sudo).
+- **Protocol (raw I2C_RDWR, NOT the SMBus ioctls)** — write the register
+  byte first, then the payload:
+  - enable `0xB2` write `04 C6 EB EA 15` (required once before mode writes)
+  - mode `0xC0` write `09 m0 m1 m2 m3 .. .. .. .. SYNC`; m0..m3 are the four
+    zone modes, `MODE_OFF = 0x00`; OFF = `09 00 00 00 00 FF FF FF FF 00`
+  - save `0x90` write `04 9E EB 00 90` (commit to EEPROM — the
+    reboot-persistence step)
+  - fw `0xB1` block read (firmware version)
+  - Read the mode back: write reg `0xC0`, then read 16 bytes; bytes 1..4 are
+    the zone modes (0x00 = OFF), bytes 6..9 the per-zone LED counts.
+- **Verify persistence:** after a reboot, read the mode — all four zone
+  modes `0x00` means the EEPROM save held (confirmed 2026-09-19 post-reboot).
+- **Tool:** a ~90-line I2C_RDWR reader/writer lived at `/tmp/evga_raw.c`
+  (`evga_raw <bus> mode|off|save|enable|fw`) — `/tmp` is wiped by reboot, so
+  recreate it if needed; OpenRGB's `EVGAGPUv3Controller.cpp` carries the
+  same protocol plus the full colour API.
+- Gotchas: raw I2C_RDWR only (the iCX3 answers the raw form, not the SMBus
+  ioctls); the `enable` write must precede the first mode write in a
+  session; without the `save` write the mode is RAM-only and resets on power
+  cycle.
+
+## Related
+
+- [[Command Center Dotfiles]] — the stow-equivalent deploy this page links into
+- [[Hermes]] — the agent that drove the CDP daemon alongside this desktop
+- [[GGUF Compatibility]] — unrelated but shares the "read the file, not the name" lesson

serverAppConfigs-docs/FrankBook-Pro/ARR-Stack.md

@@ -5,3 +5,115 @@ description: "The *arr automation stack (Prowlarr, Sonarr, Radarr, Lidarr, Reada
 tags: [arr-stack, prowlarr, sonarr, radarr, lidarr, readarr, qbittorrent, frankbook-pro, dankserver]
 timestamp: 2026-09-04T00:00:00Z
 ---
+
+# *arr Stack
+
+The *arr automation stack runs on **FrankBook-Pro** (the Mac mini) as quadlets
+inside its podman machine; all egress rides the gluetun WireGuard netns. The
+per-service pages ([[Prowlarr]], [[Sonarr]], [[Radarr]], [[Lidarr]],
+[[Readarr]], [[qBittorrent]]) carry deployment facts; this page holds the
+cross-cutting API and auth-posture procedures.
+
+## API version split (memorize before any *arr HTTP call)
+
+- Sonarr/Radarr: `/api/v3/…`; Lidarr/Readarr/Prowlarr: `/api/v1/…`.
+- Prowlarr is port **9696** — 8090 is Jackett.
+- **Empty body = endpoint absent**, not a failure: probe endpoints by status
+  code first; v1 has no `/api/v1/profile` endpoint at all.
+- `PUT /downloadclient` and `PUT /indexer/{id}` take a SINGLE resource object
+  (GET it, patch `settings`, PUT the whole object back). An array body fails
+  with `Path: $ | BytePositionInLine: 1`.
+- API keys live in each container's config:
+  `podman exec <svc> grep -o "<ApiKey>[^<]*" /config/config.xml | cut -d">" -f2`.
+  They are local-only secrets — they stop being secret the moment a vhost
+  serves the API unauthenticated.
+
+## Schema-first probing (validators are free documentation)
+
+Send the FULL schema template object with fields filled; v1 validators reject
+partials, and each error names the next required field — peel validators one
+live call at a time instead of guessing payload shapes:
+
+- `'Fields' must not be empty` → send a full schema copy, not `fields:[]`.
+- `'Base Url' must not be empty` when keys were camelCase → **PascalCase is
+  required** (`BaseUrl`, `ApiKey`) — schema keys, not the JSON wire keys.
+- `DefaultMetadataProfileId must be > 0` → fetch real ids from
+  `GET /api/v1/metadataprofile` (rows like `(1,'Standard')`);
+  `GET /api/v1/qualitydefinition` returns the quality CATALOG — ids ≠
+  profiles.
+- `targetType null` → configContract not registered in that build (this
+  Prowlarr build has no Prowlarr-plugin indexer; integration is
+  Torznab-family only).
+- `testingPackageId` in a PUT body downgrades validation failures to
+  warnings.
+- `syncLevel` is enum `ApplicationSyncLevel` — omit it or use the int; a
+  string is rejected.
+
+**Verification rules:** `POST /downloadclient` returns the COLLECTION, not
+the created object — verify via GET. Test endpoints return verdicts in the
+BODY at HTTP 200 (`{}` = pass, a list = fail) — judge from bodies, not
+status codes. Command POSTs (`MoviesSearch`, `ApplicationSync`) run
+server-side and respond only when the work finishes: curl `-m` timeouts
+(HTTP 000) during a search are NOT failures — poll queue/history afterward
+instead of re-firing; warm small commands (`RefreshMonitoredDownloads`)
+answer in ~20 ms, so a hang means live work in progress, not a dead API.
+
+**DB as arbiter:** when endpoint shapes are opaque, stop the container and
+edit `/opt/prowlarr/config/prowlarr.db` with `sudo python3` sqlite directly,
+then start. The indexers table columns are
+Id/Name/Implementation/Settings/ConfigContract/Enable/Priority/Added — the
+enable flag column is `Enable` (not `EnableRss`). Prowlarr's real application
+settings values live in `prowlarr.db` → `Applications.Settings` JSON (the
+`/api/v1/applications` GET responses carry `fields: []`).
+
+## Auth posture probe (ground truth is behavior, not config flags)
+
+`AuthenticationMethod=None` + `AuthenticationRequired=Enabled` in
+`config.xml` is ambiguous on paper; on the wire it means a login screen with
+**zero users** — first-account-wins.
+
+1. **Confirm which host runs the service before probing** (`podman ps` on
+   the target, not the box you ssh'd into first — the arrs live on
+   FrankBook-Pro).
+2. Probe an API endpoint that bypasses the SPA shell, locally and through
+   any vhost: `curl -s -o /dev/null -w "%{http_code}"
+   http://<host>:<port>/api/v3/system/status`. **SPA roots (`/`) always
+   serve a 200 shell — never conclude "open" or "gated" from `/`.**
+   401 = gated; 200 + JSON = API fully open (anyone on the net can drive it
+   with curl).
+3. For config truth, dump the whole file
+   (`podman exec <svc> cat /config/config.xml`), not a piped
+   `grep | sed -n 2p`: **`sed -n Np` exits 0 on zero matches**, so the
+   pipeline hides a miss and you conclude "empty field" when the grep itself
+   failed.
+4. Decision framing: when posture = gated-with-zero-users, present it as a
+   decision menu (None behind the front-door wall vs External/authentikarr),
+   not a silent change — first-account-wins means whoever logs in first owns
+   the instance.
+
+## The `None` + `Enabled` auth-config trap (v1 .NET apps)
+
+`<AuthenticationMethod>None</AuthenticationMethod>` +
+`<AuthenticationRequired>Enabled</AuthenticationRequired>` is
+self-contradictory: the UI Save is permanently disabled, and any save throws
+`DryIoc.ScopedItemException` in `AuthorizationMiddleware` → 500 on every
+page after restart. The valid no-auth representation is **absence of the
+`<AuthenticationRequired>` key**, not the literal string `Disabled`. Fix =
+delete the key from `/config/config.xml`
+(`sed -i "/<AuthenticationRequired>/d"`), restart the container. Tell-tale:
+one v1 app boots clean while siblings 500-sprawl → diff their config.xml
+auth blocks; the healthy one shows the correct shape. (Prowlarr runs
+`Forms` + `Enabled` healthily — the crash combo is `None`+`Enabled` only.)
+
+General rules that saved the session these came from: container
+`State=running` ≠ app serving — read `podman logs --tail 12 <c>` before
+theorizing; when a UI gate blocks saving, go under it (these apps keep
+plaintext XML config on disk — read/edit via `podman exec`).
+
+## See Also
+
+- [[Prowlarr]] — indexer feeds and the v2 sync model
+- [[Sonarr]] · [[Radarr]] · [[Lidarr]] · [[Readarr]] — the managers
+- [[qBittorrent]] — download client, 4.6.7 pin, E2E verification
+- [[FrankBook-Pro]] — the host and its podman machine
+- [[Homepage Status Probes]] — how the e2e gate treats this sleeping host

serverAppConfigs-docs/FrankBook-Pro/qbittorrent.md

@@ -111,6 +111,90 @@ Restart=always
   probe one ssh-context out returns the residential IP — a false leak alarm.
 - Ensure umask is set to `022` so downloaded files are readable by arr apps.
 
+## E2E verification after any pipeline change
+
+Run after ANY change to the torrent pipeline — WireGuard credential rotation,
+qB version change, gluetun reconfig, category re-seeding. Proves the full
+grab path (tunnel egress → qB download → category save path → clean
+removal). All commands from a dev box via `ssh frankbook-pro`; inside the
+Mac shell export `PATH="/usr/local/bin:/opt/homebrew/bin:$PATH"` for
+`podman machine ssh`.
+
+**Access note (2026-09-21):** the Mac's login user is **dank**@192.168.0.112
+(NOT core — `core` is the podman-machine VM user, reached only via
+`podman machine ssh`). blankpad has key auth for `dank@` (bootstrapped
+2026-09-21) and a `frankbook-pro` alias in `~/.ssh/config`. macOS sshd has a
+low MaxAuthTries: when password-bootstrapping use `-o
+PreferredAuthentications=password -o PubkeyAuthentication=no
+-o IdentitiesOnly=yes` or the key offers exhaust the attempts before the
+password is tried.
+
+**1. Egress gate — MUST run inside gluetun:**
+
+```bash
+podman machine ssh 'podman exec gluetun wget -qO- https://ipinfo.io/json'
+```
+
+Require `org` = AS9009 (M247 — ProtonVPN Panama PA-12 config); judge by org,
+never country alone. NEVER run this one ssh-context out (bare `wget` on the
+Mac host): it silently returns the residential Comcast IP and reads as a
+hard-gate leak that does not exist. A residential reading ⇒ re-run inside
+the container before diagnosing.
+
+**2. Login — generic cookie parse.** The QPW fetch is flaky (~1 in N returns
+empty); assert length before use — an empty-password run once produced a
+Forbidden loop that cost hours:
+
+```bash
+QPW=$(podman machine ssh 'cat /home/core/Containers/qbittorrent/env' | grep QBIT_WEBUI_PASSWORD | cut -d= -f2)
+[ ${#QPW} -eq 20 ] || exit 1
+SID=$(curl -si --data-urlencode "username=admin" --data-urlencode "password=$QPW" \
+  http://127.0.0.1:8080/api/v2/auth/login \
+  | grep -oE '[A-Za-z_]*SID[A-Za-z0-9_]*=[^;[:space:]]*' | head -1)
+curl -b "$SID" …   # pass the WHOLE match
+```
+
+WebUI from the LAN: `http://192.168.0.112:8080` (published on the gluetun
+unit). Username `admin`; password = the machine-local `QBIT_WEBUI_PASSWORD`
+(20 chars, generated once at provisioning by the deploy handler — never in
+the repo). Parse-bug 403s never reach qB as failed logins — no ban accrues;
+server-side truth is `/Volumes/COMPLETE/qbittorrent-data/qBittorrent/config/qBittorrent/logs/qbittorrent.log`
+(`WebAPI login success/failure` + source IP). Repeated REAL failures accrue
+toward qB's 5-failures/3600 s IP ban.
+
+**3. Add a legal test torrent.** Blender open-movie torrents (~300 MB, e.g.
+Big Buck Bunny) via `POST /torrents/add` with the raw `.torrent` file
+(`-F "file=@…"`) and `category=Finished Torrents`. Never leave a real-looking
+grab behind.
+
+**4. Poll and verify.** Poll `GET /torrents/info` (~20 s): state
+`downloading` → `moving` (INCOMPLETE→COMPLETE move) → `stalledUP`/`uploading`
+at progress=1. `save_path` must be `/Volumes/COMPLETE/Finished Torrents`
+(qB 4.x auto-creates unknown categories at DefaultSavePath; 5.x did NOT
+inherit category paths — one more reason for the 4.6.7 pin). Confirm on
+disk: `ls "/Volumes/COMPLETE/Finished Torrents/"`. Throughput ~17 MB/s is
+normal for the tunnel.
+
+**5. Cleanup — delete only your artifact:**
+
+```bash
+curl -s -o /dev/null -w "%{http_code}" -b "$SID" \
+  --data-urlencode "hashes=<literal-hash>" \
+  --data-urlencode "deleteFiles=true" \
+  http://127.0.0.1:8080/api/v2/torrents/delete   # expect 200
+```
+
+Target the known hash literally — the user's real grabs share the directory
+and MUST be untouched. Verify: hash absent from `/torrents/info`, your dir
+gone, all other entries intact.
+
+**Version-coupling constraints:** qB is pinned to 4.6.7 (Readarr's EOL
+client rejects qB ≥ 5's `204` login at runtime — see the pin rationale
+above). Do not unpin while Readarr exists. After ANY qB version flip,
+regenerate the `WebUI\Password_PBKDF2` line fresh (same password, 100k
+iterations) — a 5.x-written hash line makes 4.6.7 reject every login.
+Rewrite while qB is stopped.
+
 ## See Also
 
 - [[ARR-Stack]] — full *arr stack documentation

serverAppConfigs-docs/Infrastructure/Anubis.md

@@ -131,6 +131,55 @@ ungated), and gitea (git smart-HTTP clients cannot solve JS challenges —
    distinguish challenge from deny at this release; grep the body for
    `Access Denied` / `Making sure you` instead.
 
+## Guard-chain layer diagnosis (a user reports a guarded page "broken")
+
+Use when a guarded page (e.g. the `www.franksdank.cloud` homepage) reads as
+"broken / links gone / not loading" and the fault must be localized to
+server-side or client-side/session. Chain topology:
+`Browser → Caddy TLS edge → Anubis :8923 → Caddy :8016 → Authentik outpost
+→ Caddy :8017 (@home/@data handlers) → app/static`.
+
+Layer-by-layer checks (run from the host via ssh):
+
+1. **Caddy unit + guard containers**: `systemctl --user is-active
+   caddy.service`; `podman ps --format '{{.Names}}\t{{.Status}}' | grep -iE
+   'anubis|authentik|caddy'` — expect `Up … (healthy)`.
+2. **Anubis challenge page validity**: `curl -sk
+   https://www.franksdank.cloud/ --resolve www.franksdank.cloud:443:192.168.0.143`
+   — a healthy challenge contains `"anubis_challenge"` JSON with
+   `difficulty` and the main JS.
+3. **Outpost errors**: `podman logs --tail 40 authentik-outpost 2>&1 | grep
+   -iE 'error|exception|traceback|500|provider'` — routine dramatiq task
+   logs are normal.
+4. **Post-auth handlers from INSIDE the caddy container** (8016/8017 live in
+   its netns — `curl http://127.0.0.1:8017` from the HOST returns `000`
+   connection refused, which is EXPECTED, not a fault):
+   `podman exec caddy sh -c 'wget -q -S -T5 --header "Host:
+   www.franksdank.cloud" -O- http://127.0.0.1:8017/'` and the same for
+   `/services.json`, `/status.json`. Expect 200 + body. (BusyBox wget in
+   this container has NO `-H` flag — use `--header "Host: …"`.)
+5. **Files on disk**: `ls -la /srv/homepage/` and validate `status.json`
+   parses (`python3 -c "import json;json.load(open('/srv/homepage/status.json'))"`).
+
+Tool/behavior notes: the anubis container is distroless — `podman exec anubis
+sh` fails (`sh: executable not found`); inspect via its logs or the served
+challenge page. Anubis logs `x-real-ip: 10.89.1.22` (the Caddy container's
+podman IP) for ALL clients — rootless-podman published-port SNAT;
+clients are indistinguishable by IP and clearance is cookie-based. Do NOT
+mistake that for the bug. Homepage render: `app.js` does
+`Promise.all([fetch services.json, fetch status.json])`; on ANY fetch
+failure the catch sets `#summary` to `status unavailable: <err>` and the
+grid stays EMPTY — which a user reads as "links gone / not loading." A
+30-service status.json with a 9-service services.json renders fine
+(stateByName lookup).
+
+**Decision:** if every layer is healthy (challenge valid, outpost clean,
+handlers 200, files render), the fault is client-side/session. STOP
+thrashing the live system — repeated failed browser challenges share the
+masked 10.89.1.22 identity. Get the authenticated view: ask for the exact
+symptom (challenge spinning? login loop? empty grid? "status unavailable"
+text?) or the authentik login to drive the page in a real browser.
+
 ## Verify live
 
 ```

serverAppConfigs-docs/Infrastructure/Authentik.md

@@ -342,6 +342,168 @@ The silent re-auth path was proven end-to-end: after deleting
 `qa-e2e-admin`'s rows, the e2e admin spec ran against the stale storage
 state and passed with no login prompt (2 tests, 2.9 s).
 
+## Guard-chain flip recipe (adding a guarded vhost)
+
+Move a public vhost from `reverse_proxy <app>:<port>` onto the chain:
+client → TLS edge → Anubis :8923 → outpost decision → internal
+`http://:8017` listener → Host-matched handler → real upstream.
+
+**Order (never reorder):**
+
+1. **Provider first.** Create the outpost ProxyProvider before touching
+   routes, or authed traffic hits the outpost error page instead of the app.
+   Fleet convention for `external_host` is the exact browser-facing origin
+   (the inventory table carries both bare origins and trailing slashes;
+   routing matches on hostname, verified 2026-09-02).
+   `internal_host = http://caddy:8017`, all three flows set, and an
+   Application attached — the two provisioning rules above.
+2. **Route flip in ONE commit/deploy**: the vhost becomes
+   `reverse_proxy anubis:8923` with `header_up X-Real-Ip {remote_host}`, and
+   inside the `http://:8017` listener add `@<svc> host <vhost>` +
+   `handle @<svc> { reverse_proxy <upstream> }`. Anubis guard set, Caddy TLS
+   edge, `:8017` routing table, and provider list change together.
+3. Deploy via the repo deploy flow (commit first; deploy pushes main itself).
+
+**Verification (all three):**
+
+- Unauthenticated probe from a client: `curl -s https://<vhost>/` → HTTP 200
+  challenge page; grep the body for `challenge|not a bot|anubis` (~8
+  markers on a live challenge). A 502 here instead of the challenge = chain
+  mis-wired.
+- Post-auth routing proof: grep the LIVE config inside the container —
+  `podman exec caddy sh -c "grep -n -A3 <handler> /etc/caddy/Caddyfile"` —
+  after the deploy prints its reload line; confirm the reload actually ran,
+  don't trust exit 0.
+- Outpost reachability: `podman exec authentik-server getent hosts caddy`
+  resolves the compose DNS alias (10.89.1.x) that `internal_host` dials.
+
+**Pitfalls:** probing `http://localhost:8017/` from the host returns HTTP
+000 (connection refused) — the post-auth listener lives in the caddy
+container's netns; probe inside it or grep the config. Non-browser clients
+cannot solve the JS challenge — before flipping, audit service-to-service
+traffic: ingestion/push paths must target internal addresses (`loki:3100`),
+never a public hostname of a guarded service. Exempt-by-construction
+categories stay unguarded: the IdP itself (`auth.`), git smart-HTTP
+endpoints, services with their own complete auth — document the exemption
+in the vhost comment.
+
+## Driving the ORM (`ak shell`)
+
+All authentik object work happens through the server container's ORM shell —
+no repo change, no deploy (these providers are NOT repo-managed; capture new
+provider+application pairs as a blueprint or setup script to avoid drift).
+
+```bash
+ssh dankclaw 'podman exec -i authentik-server ak shell' <<'EOF' | sed -n '/229 objects/,$p'
+from authentik.core.models import User, Group
+...
+EOF
+```
+
+- The heredoc IS the program: `ak shell` reads all of stdin and execs it
+  (heredoc-through-ssh quoting breaks; the raw-stdin pattern is the proven
+  one).
+- Bootstrap prints JSON log lines, then
+  `### authentik shell (2026.8.3)` and `229 objects imported automatically…`
+  — pipe through `sed -n '/229 objects/,$p'` to drop the noise.
+- 229 models are pre-imported. Import paths (each wrong guess costs a failed
+  exec): `User`, `Group`, `Application`, `ApplicationEntitlement`,
+  `PropertyMapping` → `authentik.core.models`; `ProxyProvider` →
+  `authentik.providers.proxy.models` (NOT `authentik.outposts.models`);
+  `LDAPProvider` → `authentik.providers.ldap.models`;
+  `authentik.core.models.application` does NOT exist as an import path; there
+  is no `authentik.policies.group` module in this build.
+- For live DB truth prefer the postgres container
+  (`podman exec authentik-postgres psql -U authentik -d authentik -At -c
+  "select * from …"`) — **query with `select *` and stop guessing column
+  names**: 2026.x renamed columns (`Provider.objects.filter(type=…)` raises
+  FieldError — `type` moved to subclass tables; the outpost `enabled` column
+  may not exist under that name; provider pk is `oauth2provider_ptr_id`).
+
+## User provisioning playbook (user-level account)
+
+There is no `User.create_user` classmethod, and `is_superuser` is NOT a
+constructor kwarg (it is a property derived from group membership) — passing
+it raises `TypeError: User() got unexpected keyword arguments`:
+
+```python
+from authentik.core.models import User, Group
+u = User(username="ridley", email="ridley@franksdank.cloud", name="Ridley", is_active=True)
+u.set_password("...")        # password arrives on stdin only, never argv
+u.save()
+for g in ("Non-Admin Users", "jellyfin_users", "openwebui_users"):
+    u.groups.add(Group.objects.get(name=g))
+```
+
+`path` defaults to `users` (regular). The access map (verified live
+2026-10-04): there are NO per-app policy bindings on the proxy providers
+(all use the implicit-consent "Authorize Application" flow), so guarded-vhost
+access is decided entirely by GROUP MEMBERSHIP consumed downstream —
+
+- `homepage-admins` → Caddy RBAC for homepage `/admin` (`X-Authentik-Groups`
+  regex gate) and the homepage admin panel; regular users get only
+  `category: user` panels from `services.json`.
+- `openwebui_users` / `openwebui_admins` → ApplicationEntitlements
+  "Open WebUI Users"/"Open WebUI Admins" (bound by group), consumed by the
+  "Open WebUI roles" property mapping → `roles` claim. Verify live with
+  `u.app_entitlements(app)`.
+- `jellyfin_users` / `jellyfin_admins` → Jellyfin via the LDAP outpost
+  (filter `(memberOf=cn=jellyfin_users,ou=groups,dc=franksdank,dc=cloud)`).
+- `Non-Admin Users` = the plain-user marker group (precedents: alex,
+  qa-e2e).
+- EXCLUDE from a user-level account: `homepage-admins`, `jellyfin_admins`,
+  `openwebui_admins`, `authentik Admins`, `authentik Read-only`.
+- Other user-category vhosts (SearXNG, Gitea, Calibre-Web, RomM, Copyparty,
+  Plex, Seerr, Marreta) need no per-user group: any authenticated user
+  passes the outpost.
+
+Verify login behaviorally — `authenticate()` is the exact PasswordStage code
+path:
+
+```python
+from django.contrib.auth import authenticate
+assert authenticate(request=None, username="ridley", password="...")   # -> User
+assert authenticate(request=None, username="ridley", password="nope") is None
+```
+
+Look flows up by slug, not name:
+`Flow.objects.get(slug="default-authentication-flow")` (its `.name` is
+"Welcome to authentik!" — `.get(name=…)` fails). Stage order:
+Identification(10) → Password(20) → AuthenticatorValidate(30; no-op with no
+enrolled device) → UserLogin(100). With no MFA enrolled, nothing else blocks.
+
+More field gotchas (each cost a failed exec): `ApplicationEntitlement.application`
+can be null (guard with `getattr`); `LDAPProvider` has
+`search_mode`/`bind_mode`/`base_dn`, lacks `search_base`/`of_groups`/
+`organization`; `OAuth2Provider` has no `.scopes`; `Outpost.config` is an
+`OutpostConfig` object, not a dict; `Policy` has no uniform `.policy_cls`
+(use `type(p).__name__`); `User` has no `.status`.
+
+Never rotate `AUTHENTIK_SECRET_KEY` to fix claim issues — it invalidates
+every outpost session fleet-wide; purge the stale `proxysession` rows
+instead (section above).
+
+## Probing the outpost from inside the netns
+
+Rootless podman gives each container its own netns; **host `ss -ltn` cannot
+see listeners inside other containers' namespaces** — a missing listener on
+the host is an observation artifact, not a fault. Distroless images have no
+shell (`podman exec authentik-server sh` fails), so probe by joining the
+namespace:
+
+```
+podman run --rm --network container:<container> --entrypoint /bin/busybox \
+  docker.io/library/busybox:latest \
+  wget -q -T 4 -t 1 -O /dev/null -S "<url>" 2>&1 | head -8
+```
+
+- Quote URLs over ssh: the remote shell is zsh and bare `?` in query strings
+  triggers globbing (`no matches found`).
+- Boot transience: immediately after a restart the same probe can return 503
+  (configure task pending); poll until the answer changes before concluding.
+- `podman inspect --format "{{.Config.NetMode}}"` errors on this podman
+  version — the field is `.HostConfig.NetworkMode`.
+
 ## Failure modes
 
 | Symptom | Cause | Fix |

serverAppConfigs-docs/Infrastructure/Homepage Status Probes.md

@@ -0,0 +1,158 @@
+---
+type: reference
+title: Homepage Status Probes
+description: The homepage TUI status page — how probe targets are chosen (host-netns vantage), the three false-down shapes, triage procedure, local render check, deploy, and the e2e gate interactions (asleep Mac mini, raw-IP matrix).
+tags: [homepage, status, probes, e2e, homepage-status]
+timestamp: 2026-10-05T00:00:00-07:00
+---
+
+# Homepage Status Probes
+
+The homepage is a static site —
+`shared/homepage/{index.html,admin.html,app.js,style.css,services.json,docs.html}`
+— served from `/srv/homepage` on **dankclaw** only (`deploy_homepage` in
+`scripts/deploy-remote.sh` refuses other targets). A checker
+(`scripts/homepage-status.py`, run by the `homepage-status.timer` unit) probes
+each card's target and writes `/srv/homepage/status.json` (mode **0644** so
+Caddy can serve it — losing the mode 403s the page).
+
+## The checker's vantage and its three false-down shapes
+
+The checker probes TCP/HTTP **from the host netns**. It only has three states
+— `wip` | `up` | `down`; there is no "unknown", so an unprobeable service can
+never render honestly and must be dropped or re-pointed, not "fixed".
+
+1. **TLS-only app probed with `method:"http"`.** Plex Media Server serves TLS
+   only on `32400`; plain-HTTP GET → TLS alert → curl exit 52, code `000` →
+   card says down while PMS answers `200` over HTTPS. Fix:
+   `"method":"tcp"` (connect is the honest liveness signal; the public URL
+   stays HTTP because Caddy re-keys).
+2. **Retired port.** The Hermes card probed `127.0.0.1:8090` after the
+   compose→quadlet move moved the dashboard to `9119` and the webui to
+   `8787`; nothing listened on 8090, so a healthy stack read `down`.
+3. **Unpublished port / wrong netns.** `gitea-postgres` (`db:5432`, net
+   `systemd-gitea`), `searxng-valkey` (`valkey:6379`, net `systemd-searxng`)
+   and `alloy` (OTLP 12345) publish **no** ports. Container DNS aliases are
+   netns-scoped, so from the host these are unreachable *by construction* —
+   no probe value makes them honest. Drop those cards.
+
+A host-vantage probe can only reach a listener **published to the host**
+(rootlessport/pasta); unpublished ports are invisible from the host netns.
+See [[Container Host Reachability]] for the netns rules.
+
+## Triage procedure
+
+1. Read the card's probe target in `shared/homepage/services.json`; note
+   method (`tcp` vs `http`).
+2. Does that port have a **host-visible listener**?
+   `podman ps -a --format '{{.Names}}|{{.Status}}'`, then per container
+   `podman inspect <c> --format '{{json .NetworkSettings.Networks}}
+   {{.HostConfig.NetworkMode}}'` and `podman port <c>` (empty output =
+   published nothing → structurally unprobeable from the host).
+3. Probe from the **correct vantage**: from a bridge-net container the host
+   is its gateway IP (`/proc/net/route`, e.g. `10.89.1.1` on
+   `dankclaw_default`) — *not* `host.containers.internal` (`169.254.1.2`),
+   which belongs to the default `podman` net (`10.88.0.0/16`). Probe with
+   bash `/dev/tcp` (no `python3`/`ip` in these images); escape loop vars so
+   they expand **inside** `podman exec bash -c`.
+4. Classify: real outage (container exited — read `State.ExitCode`: 137 =
+   SIGKILL, 143 = SIGTERM/manual stop; `OOMKilled=false` rules out the OOM
+   killer) vs false down (running + unreachable target).
+5. Fix: re-point or re-method for HTTP/TLS mismatches; **drop** cards whose
+   only exposure is an internal netns; never invent a probe target.
+6. Verify by re-running the checker path, not by eyeballing: after deploy,
+   re-read `/srv/homepage/status.json` and diff the `down` list
+   before/after.
+
+Removing a card's `probe` key without a code change crashes the checker
+(`svc["probe"]` KeyError) — dropping the whole entry is the only zero-code
+fix. The `description` field is consumed only by `app.js` `descHTML()`, so
+adding a `details` object (role/spec/inputs/use) is schema-safe:
+`tests/homepage_config.bats` checks only name/category/section/probe.
+
+## Local render check (before every deploy)
+
+The page fetches `services.json` + `status.json` at load; `file://` is
+CORS-blocked, so serve over HTTP:
+
+```bash
+cd shared/homepage
+python3 -c "import json; s=json.load(open('services.json')); \
+  json.dump({'services':[{'name':x['name'],'state':'up'} for x in s]}, open('status.json','w'), indent=2)"
+python3 -m http.server 8899 --bind 127.0.0.1
+```
+
+Then in the browser: load `http://127.0.0.1:8899/index.html`, **wait ~1.5 s**
+(panels render after the async fetch; checking immediately returns 0 panels),
+extract via `tab.evaluate` — LLM cards render `.desc-role`/`.desc-spec`/
+`.desc-inputs`/`.desc-use` from the card's `details` object, other cards a
+plain `.desc`; `getComputedStyle(el).color` confirms tones. Screenshot for
+visual proof. Expected local noise: `deployed.js` 404s (baked in at deploy
+time) — harmless.
+
+The scratch `status.json` is untracked and NOT git-ignored — move it to
+`shared/homepage/.AGENTTRASH/` after the check (repo policy: never `rm`).
+
+CSS palette lives in `style.css` `:root` (`--fg` bright green, `--dim` dim,
+`--head` pale, `--link` cyan, `--wip` yellow, `--down` red). Add new tones as
+`--*` vars and include any new `.desc-*` class in the `html[data-bare="1"]`
+reset so `?bare=1` strips it to black.
+
+## Deploy
+
+```
+./scripts/deploy.sh dankclaw --service homepage --skip-e2e
+```
+
+Copies the assets to `/srv/homepage`, regenerates `deployed.js` (git commit
+datetime), restarts the `homepage-status` timer. Verify live:
+`ssh dank@192.168.0.143 'grep … /srv/homepage/…'`.
+
+**`--skip-e2e` is justified for homepage-only changes when the Mac mini is
+asleep**: the e2e admin crawl (`tests/e2e/specs/11-admin.spec.ts:24`) walks
+Prowlarr at `192.168.0.112:8989` on FrankBook-Pro; an asleep mini (ARP STALE,
+no remote wake) times out that test — a pre-existing environmental failure,
+not a regression. All non-Mac-mini e2e tests pass.
+
+## e2e raw-IP *arr failures — probe from the browser's box, not a peer
+
+When `11-admin` fails with `ERR_CONNECTION_TIMED_OUT`, the instinct is "the
+Mac is asleep". Sometimes true; not always. Playwright runs on the dev box,
+so a healthy probe from dankclaw proves nothing about the **browser's** path:
+
+```bash
+# 1. From the box that RUNS the browser:
+curl -s -o /dev/null -w '%{http_code}\n' --max-time 7 http://192.168.0.112:8989/
+# 2. From a LAN peer:
+ssh dankclaw 'for p in 8989 7878 9696; do curl -s -o /dev/null -w "$p %{http_code}\n" \
+  --max-time 6 http://192.168.0.112:$p/; done'
+# 3. Liveness (ICMP, not TCP):
+ping -c 2 -W 3 192.168.0.112
+```
+
+| peer probe | dev-box probe | ping | meaning |
+|---|---|---|---|
+| 200 | 200 | ok | transient — re-run the spec |
+| timeout | timeout | fails | Mac asleep; wake, ports return in seconds |
+| **200** | **rc=28** | **ok** | **path problem, not sleep** — ICMP allowed, TCP blocked. Waking cannot help |
+
+Third row: confirm the Mac is innocent through the dankclaw jump
+(`pgrep -fl socat`; `netstat -an | grep -c 8989`; loopback
+`curl http://127.0.0.1:8989/` → 200; `/usr/libexec/ApplicationFirewall/socketfilterfw
+--getglobalstate` → disabled). Forwarders up + loopback 200 ⇒ the block is
+between the dev box and the Mac.
+
+Don't hold a deploy hostage to a known-environmental red gate: verify the
+deployed service **directly** (its own ledger/health/real completion), state
+the gate was skipped or partially red and why (`--skip-e2e` prints the
+warning, so the skip is recorded), and report the residual gap plainly
+("29/30 e2e pass; 11-admin fails on Sonarr's raw IP from the dev box; peer
+probe returns 200; deploy verified by other means"). Do not weaken or skip
+the spec — it guards a real SSO-plus-RBAC contract.
+
+## Related
+
+- [[Container Host Reachability]] — the netns rules behind the vantage
+- [[Observability Provisioning]] — the monitoring stack reading the same fleet
+- [[Deploy State Triage]] — service-scoped deploys and the e2e gate
+- [[Testing Policy]] — the seam/e2e contract this page's checks live in

serverAppConfigs-docs/Infrastructure/Homepage.md

@@ -23,7 +23,7 @@ The homepage is live on Dankclaw. `./scripts/deploy.sh dankclaw` copies the stat
 page (`shared/homepage/docs.html`) — access walkthrough, the running-model
 guide, media how-tos, and troubleshooting. Per-service `/docs/<svc>/` pages
 remain a separate thread. The three LLM cards (`frank-large`, `frank-medium`,
-`frank-small`) moved from admin-only to the **user** page with live model
+`frank-secondary`) moved from admin-only to the **user** page with live model
 descriptions and a `docs` link to `/docs#models`.
 
 **Sections + WIP split (2026-09-21):** both pages now group their panels
@@ -111,7 +111,7 @@ flowchart TD
 ## Service classification
 
 **User services** (root page), in section order:
-- **AI** — OpenWebUI, and the three LLM fleet cards (frank-large, frank-medium, frank-small — status + docs link, no Open link).
+- **AI** — OpenWebUI, and the three LLM fleet cards (frank-large, frank-medium, frank-secondary — status + docs link, no Open link).
 - **Web Apps** — SearXNG, Gitea, Copyparty, Marreta.
 - **Media Library Apps** — Calibre-Web, RomM, Plex, Jellyfin, Seerr.
 

serverAppConfigs-docs/Infrastructure/LAN-DNS.md

@@ -158,6 +158,119 @@ The Porkbun zone `franksdank.cloud` carries an A record for every vhost plus the
 | HTTPS works from the host but not the WAN | No port forward on the G36 for that service (manual router step) |
 | Cert renewal fails in Caddy logs | Porkbun API credentials in the Dankclaw `.env`; test with the Porkbun API `/ping` endpoint |
 
+## Per-client fix: resolver drop-in + browser DoH bypass
+
+Symptom: a LAN machine gets `connection timeout` (curl rc=28) on
+`https://*.franksdank.cloud` while the server is healthy. Root cause: the
+public DNS hands out the WAN IP `71.227.227.172`, which has **no 443
+port-forward** (filtered). Any resolver that is NOT Pi-hole produces the WAN
+answer → SYN to a filtered port → timeout. There are TWO resolver layers to
+check, in order:
+
+**Layer 1 — systemd-resolved (fixes curl/shell).** A route-only drop-in
+sends `*.franksdank.cloud` ONLY to Pi-hole and never to the default-route
+servers:
+
+```bash
+dig +short www.franksdank.cloud @192.168.0.143   # expect 192.168.0.143
+dig +short www.franksdank.cloud @1.1.1.1         # expect 71.227.227.172 (the trap)
+getent hosts www.franksdank.cloud                # what the OS actually resolves
+curl -sk --resolve www.franksdank.cloud:443:192.168.0.143 \
+     https://www.franksdank.cloud/ -o /dev/null -w '%{http_code}\n'   # 200 = server fine
+
+sudo mkdir -p /etc/systemd/resolved.conf.d
+sudo tee /etc/systemd/resolved.conf.d/franksdank-lan.conf >/dev/null <<'EOF'
+[Resolve]
+DNS=192.168.0.143
+Domains=~franksdank.cloud ~franksdank.casa
+EOF
+sudo systemctl restart systemd-resolved && resolvectl flush-caches
+```
+
+The `~` (route-only) prefix is load-bearing: it routes only those domains to
+Pi-hole. Verify: `getent hosts www.franksdank.cloud` → `192.168.0.143`;
+`resolvectl status` shows `DNS Domain: ~franksdank.cloud ~franksdank.casa`.
+
+**Layer 2 — browser DoH (the bypass: "curl works, browser times out").** If
+Layer 1 is verified but the BROWSER still times out, the browser is doing
+its own DNS over HTTPS and never asks systemd-resolved — it queries the TRR
+provider (Cloudflare et al.) directly → WAN answer → timeout. Prove it:
+`curl -sk -m 6 https://71.227.227.172/` → rc=28 (the WAN answer is a dead
+end). Diagnosis details:
+
+- Firefox profile may be at the XDG path `~/.config/mozilla/firefox/`, NOT
+  `~/.mozilla/firefox/` — find it via the running process if the default
+  path is empty.
+- `grep network.trr. <profile>/prefs.js` — **no entries = shipped default,
+  and since Firefox 113 the shipped default is DoH ON.**
+
+Fix (surgical, keeps DoH for everything else): Firefox → Settings → Privacy
+& Security → **DNS over HTTPS** → "enabled for these domains" → **Add…** →
+`franksdank.cloud` (and `franksdank.casa`). The exception list = domains
+that do NOT use DoH; they fall back to the system resolver, which has the
+route-only drop-in. Or set DoH to **Off** entirely (coarser). Chrome:
+Settings → Privacy and security → Security → "Use secure DNS" → off, or
+"Use the same settings as your system". Do NOT hand-edit `prefs.js` while
+the browser is running — it holds prefs in memory and rewrites the file on
+exit, clobbering the edit.
+
+This is a per-client fix: every LAN client that must reach the homelab by
+name needs Pi-hole as its DNS (or the drop-in) AND a browser that does not
+bypass it with DoH. The server side needs no change. Verified on blankpad
+2026-09-21: the drop-in was present since 09-12, curl returned 200, and
+Firefox 155 with default (ON) DoH was the remaining hole.
+
+## Porkbun zone ops (franksdank.cloud)
+
+Operate the public DNS zone via the Porkbun API v3, run **on Dankclaw** so
+the API key/secret sourced from `/home/dank/serverAppConfigs/.env`
+(`PORKBUN_API_KEY`, `PORKBUN_API_SECRET_KEY`) never leaves the host. Pipe
+scripts to the target with `ssh -o BatchMode=yes dankclaw 'bash -s' <
+/tmp/script.sh`.
+
+Endpoints (all POST, JSON body):
+
+- List: `https://api.porkbun.com/api/json/v3/dns/retrieve/franksdank.cloud`
+  — auth via headers `X-API-Key` + `X-Secret-API-Key` (Bearer is wrong
+  here). Response shape is **top-level `.records[]`**, NOT
+  `.content.records[]` as some docs claim — parse with
+  `jq -r '.records[] | [.id,.name,.type,.content,(.ttl|tostring)] | @tsv'`.
+- Delete: `/dns/delete/{domain}/{id}` (body carries apikey+secretapikey).
+- Edit: `/dns/edit/{domain}/{id}` — pass the full record (`id`, `name`,
+  `type`, `content`, `ttl`, `prio`).
+
+Quirks that bite (verified live 2026-08-31):
+
+1. **`name` is a RELATIVE label on edit.** Passing `ai.franksdank.cloud`
+   yields the record `ai.franksdank.cloud.franksdank.cloud`. Pass the bare
+   label (`ai`) and the API appends the zone itself. Verify after every
+   rename with a retrieve before declaring done.
+2. **Rate limit answers as an HTML 404-style page, not JSON.** A batch of
+   rapid mutations makes some calls return `<!DOCTYPE html>...` where jq
+   then errors (`Cannot iterate over null` / `Invalid numeric literal`). Do
+   not assume the call succeeded — sleep ~8 s and retry the failed ids;
+   verify by re-listing the zone at the end.
+3. Deletes/renames are per-record-id: always retrieve first, map name→id,
+   then mutate by id.
+
+Rename procedure (vhost rename = DNS + Caddy together):
+
+1. `retrieve` → collect ids for the renamed names.
+2. `edit` each id with the new **relative** label, preserving
+   type/content/ttl.
+3. Re-retrieve and confirm `<new>.franksdank.cloud A <ip>`.
+4. In-repo lockstep before deploy: rename the Caddyfile vhost block AND the
+   Pi-hole `dankclaw/pihole/addn-hosts` line in the same commit — a test
+   fails the suite if any Caddyfile vhost lacks an addn-hosts A record on
+   192.168.0.143.
+5. Deploy via `scripts/deploy.sh dankclaw` (plans pihole+caddy from changed
+   paths). Capture the suite's true exit with
+   `out=$(./tests/run.sh 2>&1); rc=$?` — piping to `tail` masks the exit
+   code behind `tail`'s own status and has shipped untested commits here.
+6. Verify live: `dig +short <name> @192.168.0.143` then
+   `curl -o /dev/null -w %{http_code} https://<new-name>` (expect 200/302;
+   first hit triggers dns-01 issuance, which works without inbound port 80).
+
 ## See Also
 
 - [[Pi-hole]] — the LAN DNS server (quadlet, records, web UI)

serverAppConfigs-docs/Infrastructure/Observability Provisioning.md

@@ -0,0 +1,95 @@
+---
+type: reference
+title: Observability Provisioning
+description: How Prometheus config/rules and Grafana datasources/dashboards are provisioned as code in serverAppConfigs — the config triad, deploy copiers, planner mappings, seam tests, entry points for each change type, and API verification.
+tags: [observability, prometheus, grafana, provisioning, deployment]
+timestamp: 2026-10-05T00:00:00-07:00
+---
+
+# Observability Provisioning
+
+The monitoring stack is **provisioned as code**: dashboards, datasources,
+scrape jobs and alert rules live in the repo and ride the normal deploy flow.
+No hand-edits inside containers — they are wiped on recreate and never
+committed.
+
+## The config triad
+
+| Repo path | What it holds |
+|---|---|
+| `shared/prometheus/prometheus.yml` | scrape jobs (global + per-host) |
+| `shared/prometheus/rules/llm-fleet.yml` | alert rules + recording rules |
+| `shared/grafana/provisioning/` | datasources + dashboards (provisioned tree) |
+
+Targets and datasources reference **container names** (`loki:3100`,
+`alloy:12345`), never loopback-published ports; the LLM endpoints are reached
+by LAN IP (`192.168.0.143:7770`, `:7771`, `192.168.0.235:7770`) because they
+are host-netns services, not containers. Prometheus 3 refuses HTML responses
+as metrics — HTML-scraped "probes" are structurally inert and need a blackbox
+exporter (see [[Stale Signal Audit]]).
+
+## The deploy path
+
+`scripts/deploy-remote.sh` carries two copier functions:
+
+- `copy_prometheus_config` → `/opt/prometheus/config/` on the target
+- `copy_grafana_provisioning` → `/opt/grafana/provisioning/`
+
+`scripts/deploy-plan.py` maps changes under `shared/prometheus/**` and
+`shared/grafana/**` to the `prometheus`/`grafana` services, so
+`deploy.sh dankclaw --service grafana` (etc.) restarts only what the change
+touched. Seam tests in `tests/observability_quadlet.bats` pin: unit files
+exist, the copier functions exist, and the provisioned paths match the units'
+mounts.
+
+## Entry points
+
+**Add a scrape target** — edit `shared/prometheus/prometheus.yml`. Decide the
+vantage first: container name if the exporter is a container on the same
+netns; host LAN IP if it is a host process (LLM endpoints); a blackbox module
+if it is an HTTP probe, not a metrics endpoint.
+
+**Add an alert/recording rule** — edit `shared/prometheus/rules/llm-fleet.yml`.
+Current delivered set: 5 alert rules + 1 recording rule for the LLM fleet.
+Rules reload with the prometheus service.
+
+**Add a datasource** — drop a YAML file into
+`shared/grafana/provisioning/datasources/`. Point it at the container name of
+the backend (`http://prometheus:9090`), not loopback.
+
+**Add a dashboard** — drop the JSON into
+`shared/grafana/provisioning/dashboards/`. Provisioned dashboards live in
+unified storage, not (only) the SQLite table — grep the API, not the DB, when
+verifying (see the verify section of [[Fleet Audit and Posture]]).
+
+## Verify through the APIs, not the filesystem
+
+```bash
+# prometheus: rules and targets actually loaded
+curl -s http://127.0.0.1:9090/api/v1/rules | python3 -m json.tool | head
+curl -s "http://127.0.0.1:9090/api/v1/targets?state=active" | \
+  python3 -c "import json,sys; [print(t['labels'].get('job'), t['health'], (t['lastError'] or '')[:60]) for t in json.load(sys.stdin)['data']['activeTargets']]"
+# grafana: datasource + dashboard provisioned
+curl -s -u admin:<pw> http://127.0.0.1:3002/api/datasources | python3 -m json.tool | grep -E '"name"|"url"'
+```
+
+Health answers from a stale process and provisioned resources can hide in
+unified storage — a file existing on disk is not proof the running server
+loaded it. See [[Stale Signal Audit]] for the never-green-target decision
+table (refused = outage, 404/HTML = config defect).
+
+## Current delivered state (2026-09-17)
+
+Prometheus → Alertmanager (`dankclaw/observability/quadlet/alertmanager.container`,
+loopback `:9093`) → host-side Signal relay (`scripts/signal-relay.py`, user
+unit, `192.168.0.143:18093` — the LAN IP, the one bind a container can reach
+via `host.containers.internal`) → signal-cli. Some targets read down for
+structural reasons (HTML-scraped "probes", Gitea metrics off); the remedy is
+the blackbox exporter. See [[Observability]].
+
+## Related
+
+- [[Observability]] — stack topology and the alert delivery chain
+- [[Stale Signal Audit]] — classifying down targets before "fixing" them
+- [[Quadlet Lifecycle]] — the units that host these services
+- [[Deploy State Triage]] — how the planner selects these services

serverAppConfigs-docs/Infrastructure/Prometheus.md

@@ -38,7 +38,7 @@ Authentik; post-auth the Caddy vhost routes straight to `prometheus:9090`) —
 | `node-dankserver` | `192.168.0.235:9100` | host metrics from [[Node-Exporter]] — a different machine, so the LAN address works |
 | `grafana`, `loki` | `grafana:3000`, `loki:3100` | container names on the `dankclaw_default` network; both serve real `/metrics` |
 | `frank-medium` | `192.168.0.235:7770` | the LLM fleet — `vllm:` series; a different machine, so the LAN address works |
-| `frank-large`, `frank-small` | `host.containers.internal:7770`, `:7771` | the LLM fleet — `llamacpp:` series; same-host net=host binds, so via the gateway |
+| `frank-large`, `frank-secondary` | `host.containers.internal:7770`, `:7771` | the LLM fleet — `llamacpp:` series; same-host net=host binds, so via the gateway |
 
 **Three jobs read down, and they are not outages** (audited 2026-09-16):
 

serverAppConfigs-docs/LLM/DFlash DSpark Speculative Decoding.md

@@ -215,9 +215,9 @@ The DFlash draft is not a standalone model. It must pair with its target model.
 
 ## Deployment on Dankserver (NVIDIA RTX 3090)
 
-The Dankserver llama.cpp deployment (`frank-small`, Qwen3.6-35B-A3B) is **active** on Dankserver (port 7770) as of the 2026-08-17 redo (the earlier 2026-08-16/17 retirement of this service was reversed when the PP split was retired). The vLLM material below documents the archived AWQ/DFlash path on the same hardware.
+The Dankserver llama.cpp deployment (`frank-secondary`, Qwen3.6-35B-A3B) is **active** on Dankserver (port 7770) as of the 2026-08-17 redo (the earlier 2026-08-16/17 retirement of this service was reversed when the PP split was retired). The vLLM material below documents the archived AWQ/DFlash path on the same hardware.
 
-> **DFlash retired from active service.** `DFlash` (draft-dflash) was tested initially but did not deliver the desired speed gains under the `frank-small` workload (see ngram-mod benchmarks below). The `Alittlehammmer/Qwen3.6-35B-A3B-DFlash` draft GGUF remains in `/home/dank/models/` as reference material only.
+> **DFlash retired from active service.** `DFlash` (draft-dflash) was tested initially but did not deliver the desired speed gains under the `frank-secondary` workload (see ngram-mod benchmarks below). The `Alittlehammmer/Qwen3.6-35B-A3B-DFlash` draft GGUF remains in `/home/dank/models/` as reference material only.
 
 The vLLM material below documents the archived AWQ/DFlash path on the same hardware. The RTX 3090 is Ampere (compute capability 8.6). Use these conditions:
 
@@ -315,7 +315,7 @@ not switch without re-measuring".
 
 The A/B vehicle for ANY single-GPU llama.cpp knob (batch/ubatch, KV type,
 slot split, quant rung — not just spec flags) is the transient systemd
-drop-in procedure in [[frank-large-benchmark]] and [[frank-small-benchmark]].
+drop-in procedure in [[frank-large-benchmark]] and [[frank-secondary-benchmark]].
 
 ## Is it worth it? (arithmetic before deployment)
 
@@ -419,4 +419,4 @@ measurement that would settle it.
 - [[GGUF Compatibility]] — header parsing and custom tensor-type lock-in
 - [[Llama Cpp Serving Sizing]] — slot-pool and drafter KV sizing arithmetic
 - [[Llama Cpp Multi-Device Splits]] — where the drafter's KV actually lands
-- [[frank-large-benchmark]] and [[frank-small-benchmark]] — the single-GPU env drop-in A/B vehicle
+- [[frank-large-benchmark]] and [[frank-secondary-benchmark]] — the single-GPU env drop-in A/B vehicle

serverAppConfigs-docs/LLM/Dankclaw System Specs.md

@@ -46,7 +46,7 @@ Dankclaw is an AMD APU server for CIRU ROCmFPX llama.cpp inference. The host run
 | Runtime | FastFlowLM 0.9.45 + XRT (installed 2026-08-16; **retired 2026-09-11** with the `amd_iommu=off` decision) |
 | Status | Not in service. Research only. See [[FastFlowLM NPU Inference]] |
 
-The NPU is retired (2026-09-11). The `frank-small` name now serves Gemma4-12B QAT on the RTX 3090 via llama.cpp (see [[frank-small-gemma4-12b]]).
+The NPU is retired (2026-09-11). The `frank-secondary` name now serves Gemma4-12B QAT on the RTX 3090 via llama.cpp (see [[frank-secondary-gemma4-12b]]).
 
 ### ROCm
 

serverAppConfigs-docs/LLM/Dankserver System Specs.md

@@ -17,7 +17,7 @@ timestamp: 2026-09-20
 
 Dankserver is a media/library server and the `frank-medium` LLM host. The
 host runs Ubuntu 26.04 (Resolute Raccoon). The RTX 3090 that served the
-FreeToken/CUDA `frank-small` era was removed on 2026-09-15 and replaced with
+FreeToken/CUDA `frank-secondary` era was removed on 2026-09-15 and replaced with
 two AMD Radeon AI PRO R9700 cards.
 
 ## Hardware Specifications
@@ -107,8 +107,8 @@ hijack `libhsa-runtime64` were purged on 2026-09-15 — see
 The active inference service is `frank-medium.service`: mainline llama-server on
 port 7770, layer-split 32:33 across both R9700s with the DFlash2 drafter pinned to
 ROCm1. See [[frank-medium-llamacpp-dflash2]]. The former
-`frank-small.service` (FreeToken/CUDA) was retired with the 2026-09-15 GPU swap;
-its unit is archived in `~/.AGENTTRASH/20260915-frank-small-retire/host-units/`,
+`frank-secondary.service` (FreeToken/CUDA) was retired with the 2026-09-15 GPU swap;
+its unit is archived in `~/.AGENTTRASH/20260915-frank-secondary-retire/host-units/`,
 and the vllm-radiance replica trio in `~/.AGENTTRASH/20260916-vllm-radiance-retire/`.
 See [[deployment]].
 

serverAppConfigs-docs/LLM/FastFlowLM NPU Inference.md

@@ -13,12 +13,12 @@ timestamp: 2026-09-03
 
 > **SUPERSEDED 2026-08-17.** The Frank redo retired this topology.
 > See [[frank-redo-2026-08-17]] and [[deployment]] for the active configuration.
-> The `frank-small` name and port 52625 then belonged to the vLLM
+> The `frank-secondary` name and port 52625 then belonged to the vLLM
 > service on Dankclaw (nameistoken Quark W8A8); this page is archive.
 
 # FastFlowLM NPU Inference
 
-FastFlowLM (FLM) runs LLMs entirely on the AMD Ryzen AI NPU. It targets XDNA2 chips: Strix, Strix Halo, Kraken, and Gorgon Point. The project describes itself as "Ollama, but deeply optimized for NPUs." As of 2026-08-16, FLM is installed on Dankclaw, the NPU validates, and `frank-small` (port 52625) serves in production. See Promotion.
+FastFlowLM (FLM) runs LLMs entirely on the AMD Ryzen AI NPU. It targets XDNA2 chips: Strix, Strix Halo, Kraken, and Gorgon Point. The project describes itself as "Ollama, but deeply optimized for NPUs." As of 2026-08-16, FLM is installed on Dankclaw, the NPU validates, and `frank-secondary` (port 52625) serves in production. See Promotion.
 
 ## Verdict (2026-08-16)
 
@@ -203,16 +203,16 @@ Notes:
 
 ## Promotion (2026-08-16)
 
-`frank-small` now runs the NPU endpoint on Dankclaw (was the Dankserver llama.cpp service). Unit: `scripts/frank-small.service` (symlinked into `/etc/systemd/system`), wrapper `scripts/serve-flm-small.sh`, tunables `scripts/frank-small.env`. `LimitMEMLOCK=infinity` is set in the unit. Default model: `gemma4-it:e4b`. Swap `FLM_MODEL` in the env to switch.
+`frank-secondary` now runs the NPU endpoint on Dankclaw (was the Dankserver llama.cpp service). Unit: `scripts/frank-secondary.service` (symlinked into `/etc/systemd/system`), wrapper `scripts/serve-flm-small.sh`, tunables `scripts/frank-secondary.env`. `LimitMEMLOCK=infinity` is set in the unit. Default model: `gemma4-it:e4b`. Swap `FLM_MODEL` in the env to switch.
 
 ### Boot policy (2026-08-17)
 
-The `frank-small` unit is disabled for boot. Only the PP pair (`frank-large` + `frank-pp-worker`) starts at boot. The unit file was removed from `/etc/systemd/system`; the running FLM service keeps serving until stopped or rebooted. To restore the endpoint, run these commands on Dankclaw:
+The `frank-secondary` unit is disabled for boot. Only the PP pair (`frank-large` + `frank-pp-worker`) starts at boot. The unit file was removed from `/etc/systemd/system`; the running FLM service keeps serving until stopped or rebooted. To restore the endpoint, run these commands on Dankclaw:
 
 ```
-sudo ln -sf /home/dank/vllmScripts/scripts/frank-small.service /etc/systemd/system/frank-small.service
+sudo ln -sf /home/dank/vllmScripts/scripts/frank-secondary.service /etc/systemd/system/frank-secondary.service
 sudo systemctl daemon-reload
-sudo systemctl enable frank-small
+sudo systemctl enable frank-secondary
 ```
 
 Live config (2026-08-16): `FLM_PMODE=turbo`, `FLM_ASR=1` (loads Whisper-V3-Turbo-NPU2; exposes `/v1/audio/transcriptions`), `FLM_CTX_LEN=131072` (verified max — Gemma4 natively supports 128K and the NPU budget fits it; KV occupancy measured at 0.11% for a 123-token exchange, decode 12.0 t/s, 13 GiB headroom).
@@ -271,12 +271,89 @@ graph TD
 6. Step up in size: 9B, then gpt-oss-20b.
 7. Attempt `qwen3.6-moe:35b-a3b` only after the smaller runs succeed, and only with the production service stopped. Watch `free -h` during the model download and load.
 
+## Registering packs absent from the catalog (NPU2)
+
+Use when adding a non-catalog NPU2 pack (an extraction or fine-tuned model)
+to the FLM registry. Registry facts: user registry
+`/home/dank/.config/flm/model_list.json`, shape
+`{"model_path": "models", "models": {"<tag>": {"<size>": {name, url,
+file_url, size, flm_min_version, files[], vlm, default_context_length,
+max_prefill_len, details{format:"NPU2", family, …}, label[]}}}}`; serve
+resolves tags through `FLM_CONFIG_PATH` at startup, and `flm list` with the
+env var set lists custom tags (`⏬` = not downloaded).
+
+1. Back up the registry JSON first (`cp -a`).
+2. Insert the entry via a python json edit + atomic `os.replace`. Set
+   `files` to the FULL HF file list — **including `vision_weight.q4nx` even
+   for text-only packs**: the qwen3.5-family loader opens it unconditionally
+   and a missing file makes the first request HANG silently (no error log).
+3. **Do not use `flm pull <tag>` for custom entries** — it throws
+   `json.exception out_of_range key '<tag>:<size>' not found` while building
+   the download list. Place files manually into
+   `/home/dank/.config/flm/models/<PackName>/` with
+   `curl -sfL "https://huggingface.co/<repo>/resolve/main/<file>?download=true"`
+   — plain `raw/main/<file>` returns 135-byte LFS pointer stubs, not
+   weights; verify byte counts against the HF tree API before trusting a
+   download.
+4. Kernel packs: copy from the same-shape family pack (shape-match
+   `config.json` on hidden_size/num_hidden_layers/head_dim first) into
+   `/opt/fastflowlm/share/flm/xclbins/<PackName>/` via `sudo cp -a`
+   (root-owned dir).
+5. Deploy/restart the serve unit (the deploy path, not ad-hoc restarts) so
+   the new process loads with all files present.
+6. Verify with a REAL completion request (`model: "<tag>:<size>"`) —
+   `/v1/models` alone proves registration, not loadability.
+
+Hard constraints: only ONE `flm serve` process can hold the NPU hardware
+context — a second concurrent serve process fails every load with
+`DRM_IOCTL_AMDXDNA_CREATE_HWCTX err=-22` (aie2 mgmt msg opcode 0x2 status
+0x2000003); never run scratch/verify instances while production holds the
+NPU. Tag switching happens INSIDE one serve process; the first request
+after a model switch pays ~6–11 s cold-load cost (visible in
+`usage.load_duration`). Over ssh, never `pkill -f "<pattern>"` with a
+pattern present in your own command line — use bracket-guarded patterns
+(`[q]wen3.5:4b`) or exact PIDs.
+
+## Tag load-failure diagnosis (200 with an error body)
+
+`POST /v1/chat/completions` answers **HTTP 200** with a flat
+`{"error":"Failed to load <tag> model!"}` and no `choices`; `/v1/completions`
+returns a nested OpenAI-shaped `server_error` (code 500) also under HTTP
+200. A status-code-only client reads this as a successful empty completion —
+**always check for the `error` key**. Two distinct causes, distinguished via
+`journalctl -u <flm-service> --full`:
+
+1. **Unknown tag** (not in the registry): immediate error, nothing loaded.
+   Correct behavior, not a bug.
+2. **Registered tag whose kernel pack is missing**: FLM resolves the tag,
+   downloads missing weight files on the fly (first request hangs ~90–110 s),
+   then fails `Failed to load model: No such file
+   '/home/dank/.config/flm/xclbins/<PackName>/<kernel>.xclbin'`; subsequent
+   attempts fail fast (~2 s). Root cause: the serve script exports
+   `FLM_XCLBIN_PATH=/home/dank/.config/flm/xclbins` (user dir), so every
+   kernel lookup resolves under the user dir, which only contains symlinks
+   someone linked. **A tag listed in `/v1/models` is NOT proof it is
+   loadable on this host.** Fix: one symlink per missing pack
+   (`ln -s /opt/fastflowlm/share/flm/xclbins/<PackName>
+   /home/dank/.config/flm/xclbins/<PackName>`) — verified live: after the
+   symlink the first request loaded and answered in 7.4 s wall.
+
+Switch-cost behavior: the `model` field routes per-request (no restart
+needed); a request whose model differs from the resident one triggers a full
+reload (7–11 s wall for an 8-token completion on ~4B-class packs);
+alternating tags pays the swap each time. Single-lane NPU: concurrent
+requests queue FIFO behind the current lock holder (`NPU Locked!` →
+`NPU Lock Released!`). Probe recipe: `/v1/models` for registered tags; grep
+the journal for `Failed to load model` (the quoted xclbin path names the
+missing pack dir exactly); diff `ls ~/.config/flm/xclbins/` (symlinks =
+installed) against `/opt/fastflowlm/share/flm/xclbins/` (available).
+
 ## Incident: qwen3.6-moe unreachable (2026-08-16)
 
 Symptom: a client on dankware could not reach `qwen3.6-moe:35b-a3b` on port 52625. The journal shows three separate causes in the 14:19-15:03 window.
 
 1. **EXEC 203 crash loop (14:19-14:20).** The exec bit was lost on `serve-flm-small.sh` on the target. The unit failed `status=203/EXEC: Permission denied` four times, then started after the bit was restored. The git mode is `100755` (commit `3f42210`), and a `git pull` restores the bit. Verify with `git ls-files -s scripts/serve-flm-small.sh`.
-2. **Wrong model ID (14:29-14:52).** The client config still used the model ID `frank-small` (a service alias). FLM rejects that tag with `[ERROR] Model tag 'frank-small' is not supported.` Each such request also triggered the unknown-tag fallback below.
+2. **Wrong model ID (14:29-14:52).** The client config still used the model ID `frank-secondary` (a service alias). FLM rejects that tag with `[ERROR] Model tag 'frank-secondary' is not supported.` Each such request also triggered the unknown-tag fallback below.
 3. **Swap churn (14:48-15:03).** The swap `gemma4-it:e4b` → `qwen3.6-moe:35b-a3b` restarted the service several times. Each load of the 23 GB model takes about 1.5-2.5 minutes at 262144 ctx. The endpoint does not serve during a load.
 
 Fix: the client model ID must be the FLM registry tag `qwen3.6-moe:35b-a3b`. The working pi agent entry in `~/.pi/agent/models.json` is `baseUrl http://192.168.0.143:52625/v1`, `id qwen3.6-moe:35b-a3b`, `contextWindow 262144`, `reasoning true`, `input [text, image]`. Verified end-to-end from dankware (192.168.0.226) at 15:32 on 2026-08-16.
@@ -313,7 +390,7 @@ pi sessions failed against this endpoint with `Stream ended without finish_reaso
 - **The FLM serve process is not under systemd.** A reboot of Dankclaw kills it. The PP head service is enabled and returns. The FLM process must be restarted by hand. A unit file is a natural next step.
 - **gpt-oss reasoning budget.** The model can spend its full token budget on `reasoning_content` and return empty content. Clients must handle or cap this.
 - **No llama.cpp GGUF path.** FLM uses its own pre-compiled kernels. Our UD-Q8_K_XL and HauhauCS builds are not importable. NPU work means FLM-specific model tags.
-- **NPU production role (2026-08-16).** `frank-small` serves `qwen3.6-moe:35b-a3b` on port 52625 (swap churn and the unknown-tag incident are recorded above). The remaining NPU content (benchmarks, runbook steps) is still research until a step lands and is recorded.
+- **NPU production role (2026-08-16).** `frank-secondary` serves `qwen3.6-moe:35b-a3b` on port 52625 (swap churn and the unknown-tag incident are recorded above). The remaining NPU content (benchmarks, runbook steps) is still research until a step lands and is recorded.
 
 ## Citations
 

serverAppConfigs-docs/LLM/FreeToken Builds.md

@@ -0,0 +1,97 @@
+---
+type: reference
+title: FreeToken Builds
+description: Building FreeToken (the official franksdank fork of FreeTok Tokenworks' freetoken vLLM-MTP runtime) on Dankclaw — the CUDA vLLM venv build and the ROCm build, with the dependency traps each one hit.
+tags: [freetoken, build, dankclaw, vllm, mtp]
+timestamp: 2026-10-05T00:00:00-07:00
+---
+
+# FreeToken Builds
+
+FreeToken = the official franksdank fork of FreeTok Tokenworks' freetoken
+vLLM-MTP runtime. Two build paths exist, both on **Dankclaw**, with
+different dependency traps. Both were verified end-to-end 2026-09-11/12.
+
+## Common verification (do not skip)
+
+**Verify the build by real completion, not by version string.** A version
+bump alone does not move data through the model — a build can print a new
+version and still not have completed a real token. Before declaring a
+FreeToken build deployable, run an actual completion (any tiny prompt) and
+read the returned content:
+
+```python
+import json, urllib.request
+req = urllib.request.Request("http://<host>:<port>/v1/chat/completions",
+    json.dumps({"model": "frank-secondary", "messages": [{"role": "user",
+    "content": "Hi"}]}).encode(), {"Content-Type": "application/json"})
+print(json.load(urllib.request.urlopen(req, timeout=600)))
+```
+
+A 200 with a real `choices[0].message.content` is the only build green.
+
+## CUDA build (the torch/vLLM dependency trap)
+
+Both paths build on **Dankclaw** with the same preinstall state: Python
+3.12.10 in the system venv and **Torch 2.10.0.dev** (nightly, repo
+download.pytorch.org/whl/nightly/cu130, installed with
+`uv pip install --index-strategy unsafe-best-match "torch==2.10.0.dev<date>+cu130"
+--index-url https://download.pytorch.org/whl/nightly/cu130` — use
+`--index-strategy unsafe-best-match`, `uv` otherwise refuses the dev
+version). Verify the exact pair first; mismatched preinstalled deps are the
+#1 build-failure cause.
+
+Then install the **matching vllm dev wheel**:
+
+```bash
+uv pip install --index-strategy unsafe-best-match \
+  --index-url https://wheels.vllm.ai/<commit-sha>/vllm-0.6.6.dev0+g<shortsha>.cu130-cp312-cp312-linux_x86_64.whl \
+  --extra-index-url https://wheels.vllm.ai/nightly/cu130
+```
+
+The wheel URL pattern is fixed:
+`https://wheels.vllm.ai/<full-commit>/vllm-0.6.6.dev0+g<8-char>.cu130-cp312-cp312-linux_x86_64.whl`
+— read it back from vLLM's CI-run page (`run details →
+build_and_install_package` job, "install the vllm wheel" step). **The trap:
+vLLM refuses versions < its install_requires floor at runtime, so the wheel
+version must be ≥ the runtime's floor.** A wheel built at a lower base
+(0.5.x) downloads but `import vllm` fails with a version constraint error —
+that failure looks like a build failure and is not one; upgrade the wheel,
+not the code.
+
+If `uv pip install -e . --no-deps` refuses (some repos pin the
+`--no-deps` branch off), fall back to `python3 setup.py` inside the venv,
+after `python -m venv --copies .venv && source .venv/bin/activate` and
+`uv pip install --no-build-isolation "torch==…"` — the venv must NOT
+re-resolve torch (a second torch from PyPI d1 breaks the CUDA extension ABI).
+
+## ROCm build (gfx1151 via TheRock nightly)
+
+Same venv discipline, ROCm wheels from TheRock's nightly index:
+
+```bash
+uv pip install --index-strategy unsafe-best-match \
+  --index-url https://rocm.nightlies.amd.com/v2/gfx1151-dcgpu-supports_fbsgens_split_0717-rc1/ \
+  "torch==2.10.0.dev<date>+rocm" "triton==3.6.0" "pytorch-triton==3.6.0"
+```
+
+Then `vllm` itself (`uv pip install -e . --no-build-isolation --no-deps`,
+repo branch `specs-mtp-dev`, commit `2ee440d`), after copying the two
+prebuilt extension wheels into the tree (the repo's `setup.py` skips
+compiling them when found). The ROCm build also needs
+`install-fastrak-gfx1151.sh` from the repo's scripts and
+`FASTRACK_HOME=/home/dank/FastTrack-runtime` in the unit env. FreeToken's
+x86_64-gfx1151 prebuilt extension (`freetoken-ops`) ships on HF hub
+`franksdank/freetoken-ops-gfx1151` — fetch it once and point
+`FREETOKEN_OPS_HOME` at it; never rebuild the extension from source unless
+the extension API changed.
+
+`--index-strategy unsafe-best-match` is load-bearing on both paths: the
+nightly indexes carry version pairs that `uv`'s strict resolver otherwise
+rejects, and the error (`No candidate found for pytorch-triton`) reads like
+a missing package.
+
+## Related
+
+- [[Model Swap and Rollback]] — the swap/deploy story these builds serve
+- [[Llama Cpp Serving Sizing]] — the harness measurement discipline

serverAppConfigs-docs/LLM/Heterogeneous Split Triage.md

@@ -0,0 +1,159 @@
+---
+type: reference
+title: Heterogeneous Split Triage
+description: Triage before building when someone asks to split a model across unequal accelerators on one host (24 GB GDDR6X + LPDDR APU, PCIe x4 link) — link characterisation, the real bottleneck, what placement never buys, UMA pools, drafter splits, and the verdict ranking.
+tags: [heterogeneous, multi-device, llama-cpp, triage, pcie]
+timestamp: 2026-10-05T00:00:00-07:00
+---
+
+# Heterogeneous Split Triage
+
+Produced the 2026-09-16 no-go verdict on splitting Qwen3.8-Flash-Next across
+the RTX 3090 + Strix Halo pair. Use it to answer "could we split model X
+across device A and device B" with arithmetic instead of enthusiasm.
+
+## 1. Characterise the link (bytes vs latency)
+
+```bash
+lspci -vv -s <bdf> | grep -E "LnkCap:|LnkSta:"          # capability vs actual
+nvidia-smi --query-gpu=pcie.link.gen.max,pcie.link.gen.current,pcie.link.width.current --format=csv
+```
+
+Expect **idle-downshift**: a Gen4 x4 dock idles at Gen1 x4 (~1 GB/s) and
+upshifts under load. Measure while the device is busy, or you size the design
+against the wrong number. Per-hop cost against the hidden state
+(`n_embd × 2 B` for f16): a 5120-dim model moves 10 KiB/hop ≈ 1.3 µs at
+Gen4 x4.
+
+- **Bytes per token ≈ hops/token × hidden state** — typically ~1 MiB/token
+  for 2 hops/layer × 64 layers ≈ 0.3 % of a 40 ms token. **Bytes are almost
+  never the constraint.**
+- The real budget line is **hop COUNT**: every device boundary is a
+  host-side sync. A runtime that splits per *op* hits ~100 splits/token and
+  pins the GPU at ~5 % SM. Measure `graph splits` in the loader log; never
+  assume.
+- **Design rule: partition by where the weights live, never by where the
+  activations flow.** Resident weights, activations crossing.
+
+## 2. Get the workload's real bottleneck (KV is usually it)
+
+```
+KV/token = kv_bytes_per_position × context
+```
+
+Attention reads the whole cached context every token, so KV/token scales
+with context — compare it against expert/weight traffic per token and total
+token time. At long context on a hybrid arch (few attention layers, many
+positions), **KV dominates**. Consequences:
+
+- The highest-leverage placement is **KV cache on the fastest-memory device,
+  together with its attention compute** — never KV alone: separating them
+  puts KV/token over the link, which is fatal (GBs across 7.88 GB/s =
+  hundreds of ms/token).
+- Check the fit: `kv_positions_per_slot × slots × bytes_per_position +
+  attention_trunk ≤ VRAM`.
+
+## 3. What placement never buys
+
+Prefill is compute-bound and expert weights are read once per batch, so
+moving them to faster memory buys nothing while adding a serial stage.
+Expect a regression, and size it from calibration: on Strix Halo + RTX 3090
+(Qwen3.8-Flash-Next, ik_llama.cpp) moving 12.5 % of the expert mass to the
+GPU gave **decode +12 %, prefill 192 → 32 tok/s**. A
+`1/(f_big/BW_slow + f_fast/BW_fast)` harmonic calculation reproduces that
+decode gain to within ~2 points — usable for decode projections, and it
+assumes sequential stages (true at batch 1; no pipelining).
+
+## 4. Is the second device even a separate pool?
+
+On a UMA APU, check before assuming additivity:
+
+```bash
+cat /sys/class/drm/card*/device/mem_info_vram_total   # BIOS carve-out (small)
+cat /sys/class/drm/card*/device/mem_info_gtt_total    # aperture onto SYSTEM RAM
+rocm-smi --showmeminfo all | grep -i gtt              # used == what the CPU also sees
+```
+
+A large GTT window means the "GPU memory" is the *same* LPDDR under one
+memory controller: adding a stage there adds a compute engine, not capacity
+or bandwidth. Only the discrete card's VRAM is additive. Fencing more
+(`amdgpu.gttsize`, BIOS UMA size) just removes bytes from CPU-visible RAM.
+
+## 5. Speculative decoding as the "cheap" split
+
+A drafter on the *other* device is bandwidth-free (token ids per round, tens
+of bytes), so it is tempting. Before proposing it:
+
+- the drafter needs the target's **exact tokenizer** for a plain draft model
+  (compare GGUF `tokenizer.ggml.tokens`, never trust the family name), or it
+  must be a head **trained for that target** (EAGLE/DFlash/MTP are
+  conditioned on the target's hidden states and are not portable between
+  models);
+- a plain drafter costs **K sequential forwards per round**, so it must be
+  10–50× faster than the target. Break-even with `r = t_draft/t_target`:
+  `rate_ratio = A/(K·r + 1)`, `A = (1−p^(K+1))/(1−p)`; at `r ≈ 0.4–0.5`
+  (a drafter only ~2× faster) break-even needs `p ≈ 0.7+` at K=3;
+- compare against the **incumbent**, not the no-draft baseline: a target
+  already running its own MTP head is a high bar;
+- check the runtime accepts an external drafter at all — closed engines may
+  enumerate a fixed drafter set and 400 anything else (capability gates
+  keyed on the loaded checkpoint). That is a design boundary, not a config
+  gap. In llama.cpp the portable mechanism is `--spec-draft-device/-devd`
+  (draft gets its own device list) plus `-ngld`/`-otd`/`-ctkd -ctvd`.
+
+## 6. Expressing a split (llama.cpp, the usual vehicle)
+
+```bash
+-dev CUDA0,Vulkan0 -sm layer -ts F0,F1        # contiguous layer ranges (simplest)
+-ot "\.attn_.*=CUDA0" -ot "\.exps\.=Vulkan0"   # per-tensor, by ROLE (KV co-location)
+```
+
+- buffer types match by name at load (`ggml_backend_buft_name`), so device
+  names work as `-ot` values;
+- `-cmoe`/`-ncmoe` are **CPU-only**; device-side MoE placement is `-ot`;
+- several backends in one binary is fine (CUDA + Vulkan is the practical
+  pair; HIP + CUDA do not coexist) — `GPU_BACKEND_A=OFF` in the existing
+  build's CMakeCache tells you what needs adding;
+- contiguous ranges are the WRONG split when KV bandwidth dominates: they
+  drag half the attention layers (and their KV) back to the slow pool.
+
+## 7. Hazards checklist
+
+- Paged/sidecar tables (huge n-gram/embedding tables) must stay where they
+  are read from; moving them is pure loss.
+- Keep `mmap` on for models with huge embedding tables (anonymous memory
+  gets the process cgroup-killed).
+- CUDA graph capture can abort on unusual graph shapes (hyper-connections)
+  — check before assuming optimisations apply.
+- A brand-new backend on the second device may not implement the arch's ops
+  at all: make the first measurement a **baseline of the runtime on that
+  device alone** — a split's arithmetic is worthless if the baseline is
+  worse than the incumbent engine.
+- A split consumes the fast device entirely — check what else it was
+  serving before trading it away.
+- Benchmark discipline on production hosts: stop and mask first
+  ([[Fleet Ops Playbook]]).
+
+## 8. Verdict shape
+
+Rank and report; prefer the boring answer when it wins:
+
+1. **Two independent endpoints** (zero shared traffic) — often the real
+   answer to "what's a useful combination", especially over a narrow link.
+2. **One process, weights-resident, KV co-located with attention on the
+   fast pool** — promising and quantifiable (+% decode at long context);
+   needs a dual-backend build and a baseline measurement.
+3. **Hot-expert cache / per-token weight streaming** — affordable only if
+   the miss set per token is small (a few experts; check
+   `misses × bytes × tok/s` against link bandwidth) and overlapped; injects
+   PCIe latency into the critical path; no mainstream runtime implements a
+   dynamic expert cache.
+4. List explicitly what does NOT help: TP/collectives across unequal
+   devices, KV on the far device, model loading over the link,
+   prefill-motivated placements.
+
+## Related
+
+- [[PCIe Link and Atomics]] — the link and collective walls underneath
+- [[Llama Cpp Multi-Device Splits]] — the deployed split shapes
+- [[Llama Cpp Serving Sizing]] — KV and pool arithmetic

serverAppConfigs-docs/LLM/LLM Bench Windows and Measurement.md

@@ -0,0 +1,271 @@
+---
+type: reference
+title: LLM Bench Windows and Measurement
+description: Stopping-and-restoring benchmark windows on the fleet LLM hosts (llama-bench sweeps, HTTP param benches, multi-arm vLLM offload sweeps), honest benchmark methodology, and the usage-counter measurement law under speculative decoding.
+tags: [benchmarking, llama-bench, ab-testing, speculative, vllm, fleet]
+timestamp: 2026-10-05T00:00:00-07:00
+---
+
+# LLM Bench Windows and Measurement
+
+Fleet law first: **STOP the production service before any GPU experiment**
+(AGENTS.md Benchmarking Policy) — a bench beside a live service starves the
+GPU and corrupts results; an unisolated GPU experiment has precedent for
+wedging a host until power cycle. Reference harnesses:
+`scripts/bench-batch-ubatch.sh`, `scripts/autoresearch.sh`,
+`scripts/autoresearch/frank-primary-arms.sh`.
+
+## 1. The window structure (one config per arm, a driver owns the window)
+
+1. **Stop + gate before every arm** — `wait_for_clear`: unit NOT active AND
+   no RUNNING container of that name (`podman ps`, not `-a`; `-a` counts
+   EXITED containers and can never clear — `rm -f` the exited one first) AND
+   port free (`ss -ltn`). The graceful-stop race cost a whole arm 2026-10-01:
+   `ExecStop=podman stop -t 120` drains up to 2 min; a driver that sleeps 3 s
+   then polls `/health` gets **200 from the still-running PRODUCTION engine**
+   — the arm "passes" with production's boot log, then `podman exec` dies
+   with `container state improper`.
+2. **Boot the arm as the service, not a scratch port** (no scratch ports on
+   LLM hosts — one card, one server). The serve script takes its config via
+   env-file override (`FRANK_SECONDARY_ENV_FILE`, `FRANK_PRIMARY_AR_ENV_FILE`)
+   pointing at a staged copy; the checkout is never mutated.
+3. **Fast-fail the health wait**: poll `podman ps | grep -qx <name>` inside
+   the loop — a refused boot dies in one poll instead of 20 min. Capture the
+   serve script's FULL stdout on failure (a `tail -8` loses engine-init root
+   causes; raise happens above the tail).
+4. **Measure** (§4), capture boot-log engine facts (KV pool, arena slots,
+   SpecDecoding lines) — they survive measure crashes.
+5. **Teardown + converge**: kill by explicit PID/pgid (never pattern kills),
+   `podman rm -f` scratch, wait `MemAvailable ≥ ~230 GiB` on dankserver
+   before the next boot; restore production from the EXIT trap and verify the
+   **conjunction oracle**: health 200 + `is-active` + `is-enabled` +
+   mask-symlink clean. A bare `systemctl start` is not a restore oracle.
+   (`systemctl mask` FAILS on a relinked unit — log it, explicit stop is
+   enough: systemd never auto-restarts an explicitly stopped unit.)
+6. **Warm boots cycle ~2 min** (page cache), cold ~8 min — size timeouts for
+   mixed speeds.
+
+**Destructive tails (2026-10-01):** a scratch script whose tail does
+`systemctl stop` + `podman rm -f <PRODUCTION container>` fires later if a
+probe loop survived the driver kill — it can destroy a just-deployed
+production engine (signature: deploy exits green, then seconds-to-minutes
+later `Exited (0)`, `Deactivated successfully`, clean exit = removed, not
+crashed). Before ANY deploy: `pgrep -af` every scratch script of the session
+and kill strays by explicit PID; killing the dev-side job does NOT kill the
+host-side `deploy-remote.sh` (two of them once managed the same unit).
+
+Two failure shapes inside the window that read as "the bench died":
+
+- **Stale reads from a stopping container.** `podman stop` answers while the
+  engine inside is still flushing; a `podman logs` taken in that window
+  returns the boot-prefix only. Poll `podman ps` for `Exited (0)` (or
+  `podman wait`) before reading logs, or you will write "no results" into
+  the table for a run that has them.
+- **A D-state worker wedges the stop.** A bench child that wedged in the
+  kernel ignores SIGTERM, so `podman stop -t N` burns its full timeout and
+  then the container is `Stopping` forever. Kill by container first
+  (`podman kill <ctr>` → SIGKILL path), verify the process is gone from
+  `podman ps` AND `nvidia-smi`, and only then re-serve. Never let the
+  window script proceed to a redeploy with a zombie bench worker holding
+  VRAM.
+
+## 2. llama-bench sweeps (offline batch/ubatch/split/KV/depth)
+
+**What llama-bench cannot express — decide BEFORE designing:** no
+`--ctx-size` (`n_ctx = n_prompt + n_gen + n_depth`, so a bench can NEVER prove
+activation buffers fit beside the DEPLOYED KV — only the real unit answers
+that), no `--parallel` (no slot shape, no concurrency), no `--spec-*`
+(`tg` numbers are the drafterless baseline, useful as a control only), and a
+warmup run is itself a full-size run. `-d N` is a depth PREFIX (paid once per
+cell via a `llama_state_seq_*` save/restore); without it a `tg` run is at
+~zero depth and meaningless for long conversations.
+
+**The separator trap:** `/` separates devices WITHIN one configuration, `,`
+separates sweep values — for both `-dev` and `-ts`. `-ts 36,29` parses as the
+single value 36 → everything on one card. `--tensor-split 36,29` (server) →
+`-ts 36/29` (bench). Ranges exist: `-ub 512-2048+256`.
+
+**Harness shape:** one invocation per cell (an infeasible cell calls
+`exit(1)` and kills every later cell of a cross-product invocation — and
+infeasibility is usually the question); TSV row per cell as it completes +
+`bench.json` (`-o json`) + `bench.log` (`--progress`) + a 2 s `rocm-smi
+--showmeminfo vram` VRAM trace per cell (a cell that allocated less than
+intended looks fast); resume by skipping non-empty cell outputs; stop the
+service in the harness, restore from `trap … EXIT`, verify afterwards. Match
+the deployment's load flags exactly (`-ngl -sm -ts -fa -ctk/-ctv -t`).
+
+**Run under a transient unit, not a shell:**
+
+```bash
+sudo systemd-run --unit=<name> --collect --no-block --property=Type=oneshot \
+     --property=LimitMEMLOCK=infinity --property=User=dank \
+     /home/dank/serverAppConfigs/scripts/<harness>.sh
+```
+
+Deploy units set `LimitMEMLOCK=infinity` for ROCm staging; an SSH shell has
+8 MiB and fails the load for an unrelated reason. This also survives ssh
+disconnects. **Boot rungs under the production environment, not a bench
+shell** — a rootless shell probe booted 3×262144 bf16 that production refused
+(systemd `User=` + memlock change what the pinned-expert path holds):
+`sudo systemd-run --uid=<user> --unit=<name> -p LimitMEMLOCK=infinity -p
+Restart=no env VARFILE=… /path/serve.sh`.
+
+**Verify before burning hours:** dry-run against a STUB binary with
+`SERVICE=<nonexistent-name>` (the trap/stop never reached), tiny grid,
+including one forced-`exit 3` cell — prove the TSV row, failure capture,
+resume logic. Validate the real flag set in ~1 s by pointing `-m` at a
+nonexistent file. **Reading:** bracket the sweep (reference cell first AND
+last); check per-cell `stddev_ts` (a difference smaller than spread is not a
+result); single-sequence decode is usually flat across batch/ubatch (prefill
+knob); batch ≥ ubatch (llama.cpp clamps ubatch to batch, silently duplicating
+smaller-batch cells); report what the grid cannot prove (drafter, slots, real
+context).
+
+## 3. HTTP benches of server-side parameters
+
+`llama-bench` benches the bench binary, not the running server — server-side
+params (`--threads-batch`, `--ubatch-size`, `--spec-type`, `--parallel`) need
+HTTP:
+
+- **Prefill**: large prompt, `max_tokens=1` (server crashes on 0), measure
+  `prompt_tokens / wall_time`.
+- **Decode**: `max_tokens=16384` minimum, NEVER 1 (warmup, cache growth and
+  spec acceptance all stabilize over thousands of tokens); measure
+  `predicted_per_second` from `timings` or `completion_tokens / wall_time`;
+  temperature 0.
+
+Known fleet harnesses: `scripts/bench-threads-batch-http.py`,
+`scripts/bench-decode-threads-batch.py`, `scripts/bench-spec-decode.py`.
+sglang-specific: `max_tokens=1` on `/v1/chat/completions` with the qwen3
+reasoning parser streams zero deltas — use `/v1/completions`.
+
+## 4. The measurement law (under speculative decoding this is absolute)
+
+**Never count SSE chunks.** Rejected drafts break the 1:1 chunk↔token
+mapping; chunk counting reported 8.3 tok/s for a config whose server counters
+said 23–26 — a 3x false regression that nearly abandoned the change that
+worked. Always:
+
+```python
+decode_tok_s = usage.completion_tokens / (wall - ttft)
+# stream=True, stream_options: {include_usage: true}; ttft = first content-bearing chunk
+```
+
+vLLM's 10 s `Accepted throughput` log lines are a rolling window, not a
+measurement (6.2 → 13.1 across adjacent windows on identical settings) — use
+them for acceptance STATISTICS only (`Avg Draft acceptance rate`, `Mean
+acceptance length`, per-position rates). Warm pass + ≥3 runs, median;
+temperature 0; ≥256 tokens. **A decode "jump" with early-stopped completions
+(ct < 256) and acceptance > 90 % is degenerate generation** (the fork's
+repetition-halt fired at ~140 tokens on corrupted numerics) — inspect text
+before believing a win. Run-to-run spread is 5–10 %: repeat arms; one sample
+nearly shipped the wrong setting.
+
+## 5. Honest methodology
+
+- **Match the corpus to the workload.** Token soup under-reports speculative
+  decoding (2.9–3.6 tok/update vs 3.86–5.8 on real code) and mis-ranks draft
+  settings; acceptance, not step time, moves throughput. Build the corpus
+  from the repo's own sources: `{"prompt": "<instruction>\n\n<file body>"}`,
+  `--custom-output-len`.
+- **Salt every run** with a unique per-arm token in the first bytes — it is
+  the prefix-cache key (and a proxy routing key). Without it arm 2 measures
+  arm 1's warm cache.
+- **Two workload shapes need two clients:** distinct prompts (batch
+  behaviour, prefill parallelism) and sequential turns with growing history
+  (TTFT of the first streamed chunk per turn — the only measurement that
+  exposes prefix-cache behaviour). These pull in opposite directions: a cold
+  batch spreads prefill across replicas, a conversation's next turn only pays
+  if it lands where its prefix is cached.
+- **Read the engine's ledger, not the card:** `Available KV cache memory` /
+  `GPU KV cache size: N tokens` / `Maximum concurrency for M tokens per
+  request: X.Xx` is the capacity number. vLLM's `Prefix cache hit rate:` line
+  is a LIFETIME average across every prompt the replica ever saw — send the
+  same prompt twice to an idle engine and compare TTFTs (measured 22.3 s cold
+  → 1.4 s warm for a 50k prefix). `GPU KV cache usage` says the pool is not
+  full; it does NOT say a prefix survived. The KV budget is NOT monotonic in
+  chunk size (5.44 GiB at CHUNK 8192, 4.22 at 16384, 4.16 at 4096) — measure
+  three points before treating a knob as a lever.
+- **Soak before believing:** 15–30 min of concurrent families, count failures
+  AND latency; report hit/miss split and medians with min/max — the median of
+  a bimodal distribution describes neither mode.
+- **Verification that is not the benchmark:** run the client the fleet
+  actually uses; and a deployed unit is not proven until an unattended reboot
+  brings it back.
+- **Record the "no" answers** with numbers and mechanism — placement balance,
+  concurrency admission and chunk-size tweaks all measured flat or worse
+  here; the wall was arithmetic (cached prefixes + live sequences share one
+  pool).
+
+## 6. Sweep hygiene (the arms that produced wrong verdicts)
+
+- **Prefetch uniformity:** every arm in a sweep carries the same
+  expert-prefetch setting. The 2026-09-29 campaign measured losing arms with
+  prefetch ON and incumbents OFF; untaxed, verdicts inverted (no-spec
+  12.11 → 41.47; gather 32×64 parity → +18.7 %). Never re-read a taxed
+  verdict.
+- **Knob-landed preflight:** for every `-e` knob, confirm
+  `podman exec <ctr> printenv <VAR>` — a knob the fork silently ignores
+  produces a "parity" arm that is really the default (v1 of the gather sweep
+  lost all 3 arms this way; serve-script hooks are additive and can drop
+  vars).
+- **Always run an explicit OFF arm:** the 2026-09-29 campaign's only win was
+  deleting a baseline's silent `--expert-prefetch 1` (+40–51 %). Challengers
+  compare against default-off, not "incumbent + one flag".
+- **Pin the capacity bar:** `--kv-cache-memory-bytes` (contract) rather than
+  autofit (remainder) so pool numbers are reproducible; compute need =
+  tokens × measured B/token (hybrid models: KV dtype does NOT double mamba
+  states — measure, don't assume 2x).
+- **Report decode AND prefill (TTFT) per rung:** a decode-only view nearly
+  deployed a 2x-prefill-tax config. Per-depth tables, never a single median —
+  one fork's gather win grows with depth (54 t/s @1k, 39.5 @short).
+- Container recreate (`podman rm -f`) wipes `podman cp`'d harnesses — re-copy
+  `mtp_measure.py`/`bench_iter.py` after EVERY recreate, or mount them.
+
+## 7. Bash laws that ate windows (each cost an arm)
+
+1. `${VAR:+SPEC=$1}` in front of a command expands to a bare word →
+   `SPEC=3: command not found`, nothing runs. Pass literal assignments:
+   `SPEC="${1:-}" bash serve.sh`; the callee's `${SPEC:+--flag}` skips empty.
+2. `${2WAY:-0}` is fatal `bad substitution` (digit-leading name), and inside
+   an if/else it aborts ONLY that compound statement — the script CONTINUES,
+   every arm tears down clean with zero measurements. Detection law: `ls`
+   the per-arm `.measure.txt` files before trusting any summary; arm echoes
+   are not evidence.
+3. `local a=$1 b=$x` expands `$x` before the builtin assigns — under `set -u`
+   it aborts AFTER the harness stopped production. Two statements.
+4. `sed -i` renames the file; already-running bash keeps the OLD inode —
+   mid-flight patches miss arms launched in the preceding minutes. Patch
+   early or run a make-up window.
+5. Never heredoc python through ssh→zsh→bash — write to a file, ship it
+   (`ssh host 'cat > /path && chmod +x /path' < local.sh`); nested quoting
+   fails silently. Same trap in `git commit -m`: backticks become command
+   substitution.
+6. No comments INSIDE a backslash-continued command (the continuation joins
+   the comment); verify multi-line command construction with `bash -n` AND a
+   render test.
+
+## 8. Results location and recovery (frank-primary arms)
+
+| copy | path |
+|---|---|
+| per-arm evidence | `$ARMS_DIR` = `~/.jobs/<run-stamp>/` (`.rc`, `.done` beside it) |
+| summary | `$ARMS_DIR/summary.tsv` = `arm rc decode_tps decode_tps_t0.7 kv_tokens boot_s` + `<arm>.out/.env/.overlay/.rc/.metrics` |
+| server-side last window | `/home/dank/.autoresearch-frank-primary/run.json` |
+| durable memo | `.../last/run.json` + `last/hash` (promoted only after bench AND verified restore) |
+| fallback | `~/.omp/agent/sessions/<ws>/<session>/<N>.{shake,bash}.log` (permanent) |
+
+`/tmp/frank-primary-arms-*` and `/tmp/armstub/` are stub self-tests —
+constants 99.9/88.8 identical across arms by construction; **identical
+numbers across arms == stub or bug, never a measurement.** After compaction,
+re-harvest with `grep -rhoE 'decode_tps[a-z0-9_.]*=[0-9.]+'` over the session
+logs. A window costs ~20 min warm; a detached runner's timeout ≥ 3600 s per
+window.
+
+## Related
+
+- [[RDNA4 MoE Expert Offload VRAM Budget]] — the plan arithmetic being swept
+- [[DFlash DSpark Speculative Decoding]] — spec A/B protocol and shapes
+- [[LLM Performance Diagnosis]] — profiling and attribution methods
+- [[Remote Job Runner]] — detached execution of long sweeps
+- [[frank-medium-llamacpp-dflash2]] — the measured ubatch-optimum history

serverAppConfigs-docs/LLM/LLM Performance Diagnosis.md

@@ -0,0 +1,221 @@
+---
+type: reference
+title: LLM Performance Diagnosis
+description: Probing why a served LLM is slow — the flat-depth probe, the expert-offload duty-cycle arithmetic, kernel-time attribution via the profiler window, live telemetry without benchmarking, RAM-bandwidth ceilings, IBS tensor attribution and prefix-cache-aware routing.
+tags: [performance, profiling, moe, pcie, prefix-cache, routing, telemetry]
+timestamp: 2026-10-05T00:00:00-07:00
+---
+
+# LLM Performance Diagnosis
+
+When a served endpoint is slower than wanted and someone names a suspect
+("PCIe is saturated", "the gather is the bound"), throughput arithmetic cannot
+settle it — it is usually circular (it assumes the suspect consumes the step).
+Get data in this order; the profiler window alone has retired multi-day
+optimization plans.
+
+## 1. Is decode context-bound at all? (the flat-depth probe)
+
+Sweep decode across context depth. **Flat** ⇒ attention over history is not
+the cost; the problem is per-token expert movement and KV/attention tuning is
+ruled out entirely. A genuinely context-bound server shows a clear downward
+slope. Record cold vs warm separately — a small gap means the cache is
+working. Measured on the fp8 expert-offload shape: 12.11 tok/s FLAT from 304
+to 16,095 tokens.
+
+## 2. Duty-cycle arithmetic (the "GPU not fully utilized" answer)
+
+Symptom: smooth ~25–35 % utilization with 100 % peaks on an expert-offload
+MoE. This is the EXPECTED signature, not a fault:
+
+```
+per decode step: compute burst ~100 ms (100 %) → expert-gather wait ~400 ms (~0 %)
+resident fraction = arena slots / total experts   (e.g. 6574/24576 ≈ 27 %)
+PCIe bytes/token  = miss_rate × lookups × slab_bytes  (0.73 × 480 × 2.344 MiB ≈ 0.84 GB)
+predicted duty    ≈ compute_ms / (compute_ms + wait_ms)
+```
+
+Validate the terms against the deployed engine's boot plan
+([[RDNA4 MoE Expert Offload VRAM Budget]]). If predicted ≈ observed, the low
+average is structural. Note small-transfer reality: bulk bandwidth (91 % of
+ceiling at 1 GiB) overstates what 2–3 MiB slabs get (~72 % plus ~20 %
+per-transfer latency). Before blaming the link, measure it
+([[PCIe Link and Atomics]]) and compare the engine's effective gather rate to
+the measured slab-size bandwidth — a 40–50 % gap is the software read path,
+not the wire. `rocm-smi --showuse` reads ~0 % during the whole multi-minute
+weight load: show VRAM (`--showmeminfo vram`) as the load-progress signal.
+
+## 3. Kernel-time attribution (the profiler window)
+
+Run BEFORE building any replacement kernel.
+
+1. Stop + mask production; boot the shape under test with ONLY the profiler
+   config added. vLLM argv:
+   `--profiler-config '{"profiler":"torch","torch_profiler_dir":"/prof","torch_profiler_with_stack":false,"torch_profiler_record_shapes":false,"torch_profiler_with_memory":false,"torch_profiler_dump_cuda_time_total":false}'`
+   — `with_stack=false` is load-bearing (with stacks the stop hangs and the
+   trace buffer grows ~40 GB → kernel OOM-kill of the engine core, anon-rss
+   ~95 GB; the worker-side stop can hang 5 min on ROCm/kineto). Bind-mount
+   `/prof`. If a shell substitutes values into that JSON, verify the escapes
+   actually landed.
+2. Window it: warm request → `POST /start_profile` → N generations of ≥128
+   tokens (~15 s of traffic) → `POST /stop_profile` (client may time out at
+   60 s; the trace still lands). `telemetry_report` has ONE caller: the
+   gpu_worker profiler-stop path. Overhead is small enough to keep decode
+   comparable (9.6 vs 9.9–10.0 tok/s measured).
+3. **Parse ON THE SERVER** (traces are 100–150 MB gzipped, GB expanded).
+   Rank `ph=="X"` events with `cat in {kernel, gpu_memcpy, gpu_memset}` by
+   summed `dur` (µs): report % of device time, launch count AND µs/call
+   (22 µs × 708k launches is a different problem than 4 ms × 4k), plus the
+   first-to-last event span (fraction of wall the device was busy). Aggregate
+   by FULL mangled name (template args are the tuning signal) and by stripped
+   symbol. **Trap: `gpu_user_annotation` events masquerade as GPU time** (a
+   naive sum reported 63 % in one `execute_context_0(...)` entry; spans
+   overlap and can exceed the wall clock).
+4. Divide by the window's token count → ms/token; if kernel time ≈ step
+   time, the table IS the optimization plan.
+
+Worked example (GLM-5.3-Flash, 2×GPU, offload, decode 9.9 tok/s):
+
+```
+34.8 %  708,500 launches  vectorized_elementwise_kernel   <- the actual bound
+10.0 %   42,000           MoE W4A16 GEMM
+ 8.9 %   83,160           router/top-k (sort-based)
+ 6.5 %  133,920           hyper-connection pre/post kernels
+ 1.05 %   20,664          <the long-suspected expert gather>
+```
+
+No amount of link arithmetic would have produced those two conclusions: the
+suspect was 1 %, and the mass was pointwise glue from an uncompiled trunk —
+whose own boot log names the cause (`FULL_DECODE_ONLY` override). It also
+predicted a negative result in advance (spec decode cannot help a shape whose
+per-token cost scales with the token count) which matched an unexplained flat
+MTP measurement.
+
+## 4. Telemetry without benchmarking (live hosts)
+
+When benching is forbidden (production serving), mine the runtime's own
+per-request telemetry — it measures the real workload and costs nothing.
+
+- llama.cpp (root unit, `-lv 3`): `sudo journalctl -u <unit> --since "-3h"
+  --no-pager` → `slot print_timing` lines (prompt eval, eval, total, `graphs
+  reused`, `draft acceptance … mean len`, `n_gen = N, tg = X t/s, tg_3s = …`).
+- halogen (rootless quadlet): `journalctl --user -u <unit>` (**never sudo** —
+  per-user storage) → `serve_api: mtp 28 tok in 0.66s = 42.45 t/s | 16
+  rounds, commit 1.75/round | prompt 58, prefill 0.92s | detok 24us/tok`.
+- Read the envelope, never one sample: short-prompt `prompt eval` is
+  overhead-dominated (57 tok @ 168 t/s) — quote the longest available (1106
+  tok @ 1013 t/s); decode and acceptance are content-dependent (0.46–0.76
+  measured on one unchanged build); `graphs reused > 0` / `selected slot by
+  LCP similarity` prove prefix-cache reuse.
+- **Attribute PCIe without touching the model:** background `nvidia-smi dmon
+  -s t -c 16 -o T`, ONE ordinary `max_tokens=200` request, `wait` at the end
+  (without it ssh closes and the client dies — the journal shows `W srv stop:
+  cancel task` and NO print_timing while curl still reports plausible
+  status). Measured: peak ~230 MB/s against a ~7,900 MB/s link = ~3 %, 0 at
+  idle — the sampled-logits stream, not the transport, is decode traffic on
+  that shape.
+- One `max_tokens=200` request is production traffic and fine; anything that
+  pins the card is a bench → stop-and-mask first. Check WHICH instance you
+  are reading (`systemctl status | head -12`; task ids jump across
+  restarts). The engine's own ledger (`KV self size`, `n_slots`,
+  `n_ctx_slot`) outranks your model of it.
+
+## 5. The fork's data path (before blaming the link)
+
+Offload engines pick between DMA copy-in and read-through UVA dereference
+inside the GEMM. Grep the installed package for the LRU/offload module and
+read the BRANCH CONDITION, not just the env names. A token-count threshold
+decides the path, and it interacts with spec decoding: an MTP step is k+1
+tokens, so the step can silently cross the threshold onto the slow path as
+draft depth grows. Check the threshold's premise: a default tuned at
+12.4 MiB/expert does not transfer to 2.344 MiB/expert, where copy-in can be
+several times faster. TunableOp ghost-skip: `CLAV_TUNABLEOP_SWEEP=1` alone
+does NOTHING under expert offload (`CLAV_TUNABLEOP_WITH_OFFLOAD=1` required);
+check for the skip line before concluding autotuning ran. Gather-geometry
+knobs read at import time → each arm is a full reload.
+
+## 6. RAM-bandwidth-bound MoE ceiling (before recommending a quant)
+
+```
+routed experts/token = top_k × n_layers
+bytes/token          = routed_params × bytes_per_param + shared_expert_bytes
+ceiling (tok/s)      = measured_bandwidth / bytes_per_token
+```
+
+Measure the HOST's real bandwidth, never assume DIMM ratings: 4-DIMM AM5
+boards downclock to DDR5-3600 regardless of 5600 sticks (two large DIMMs beat
+four small). Single-threaded reads under-report dual-channel ~4x — use the
+multi-threaded plateau of a checksumming reader (`scripts` pattern:
+`/tmp/bw.c`, 1/2/4/8/16 threads). Worked example (DeepSeek-V4.1-Flash):
+6×40 experts × 35.4M params = 8.5B params/token; at 3 bpw + 0.53 GB shared =
+**3.7 GB/token** → 42 GB/s ⇒ **11.3 tok/s ceiling**. Sanity anchors: H100
+NVL + CPU offload 226 GB ⇒ 5.8–9.0 tok/s; 2×3090 + 768 GB DDR4 ⇒ 12.2.
+CPU compute rarely binds below 4 bpw (~5.3 flops/byte ⇒ ~223 GFLOPS at 42
+GB/s); PCIe expert STREAMING is worse than CPU compute (Gen5 x8 ~22.7 GB/s
+effective < half of RAM); prefill reads the union of a batch's experts
+(prefill ≈ bandwidth / total_expert_bytes × ubatch). A quant that fits only
+by ignoring the Engram table does not fit (Engram = sparse n-gram lookup,
+NVMe-tolerant; the routed experts are not). An iGPU GTT aperture is an ALIAS
+of the same RAM, not extra memory.
+
+## 7. IBS model-access attribution (AMD CPUs)
+
+Real CPU access patterns for mmap'd weights, live, no stop (established on
+frank-primary, 2026-09-26, ~2.7M attributed samples in 75 s). Reference:
+`scripts/frank-primary-expert-activity.py` (self-contained dump + sample +
+analyze; `--analyze-only`).
+
+1. GGUF header-only scan (ms for a 50 GiB part) → tensor extents. **Extents
+   from CONSECUTIVE offsets, never a type-size table** — a wrong per-element
+   size silently under-covers every tensor and dumps tail samples into a junk
+   bucket (first run lost 59 % of expert samples; a uniform gap/exps ratio
+   ~1.4 across parts is the tell). Data start = `(tell + 31) & ~31`.
+2. `/proc/<pid>/maps` VMAs (7-group regex; pgoff = group 4, path = group 7)
+   matched against part files; multi-VMA per part is normal.
+3. `sudo perf record -e ibs_op// -c 50000 --data -o out.perf -p <pid> --
+   sleep <dur>` WHILE a mixed workload runs (idle perf numbers mean nothing).
+   `sudo perf script -i out.perf -F addr` for addresses.
+4. Attribute: bisect VMAs, `foff = pgoff + (addr − vma_start)`, bisect tensor
+   starts; per-expert index = `int(lo / (span / N_EXPERTS))` (expert-major
+   rows).
+5. Verify before believing histograms: gap ≈ 0, every tensor-bearing part
+   nonzero, and MoE blocks should be FLAT by construction (top-k fires every
+   block every token) — a non-flat block histogram means mis-attribution
+   first. Established law: expert routing is per-block flat, per-EXPERT
+   skewed.
+
+## 8. Prefix-cache-aware routing (multi-replica serving)
+
+Measure with a **sequential-turn client**, not only a batch benchmark — a
+batch of distinct prompts parallelises prefill across replicas and hides the
+problem. The two failure modes: round-robin loses every other cache hit
+(90k-token contexts: warm-turn TTFT 2676 ms round-robin vs 1096 ms pinned);
+pinning-everything loses a card (16 requests: 18.6 s spread vs 30.1 s
+pinned). **The rule that gets both: route by request shape** — a request
+carrying more than one message is a continuation (agent loop re-sending its
+context) → pin to the replica that served that prefix; anything else → spread
+with a COUNTER, not a prefix hash. Add failover: demote a replica on
+connection failure and retry on a live one BEFORE writing anything to the
+client (safe pre-stream; never retry after streaming started).
+
+The wall no policy fixes: **a running sequence and its cached prefix draw on
+the SAME pool.** Three concurrent 50–80k families against 122,636 tokens/card
+thrash under every placement and admission policy tried (counter placement,
+load-balanced placement, one-large-request-per-card admission, both chunk
+directions — the last COSTS KV in both directions). When the working set
+exceeds aggregate KV, say so: the levers are capacity, fewer concurrent
+large contexts, or a different platform. Traps: a lifetime hit-rate counter
+is not a per-request verdict (probe directly: same prompt twice to an idle
+replica — 22.3 s cold → 1.4 s warm); salt corpora; record sample sizes;
+`vllm bench serve` needs `--tokenizer <local path>`; verify metric names
+against the live endpoint before dashboards. Test shape: drive the routing
+function directly with synthetic bodies (stdlib-only policy module);
+conversation identity is the FIRST N KiB of prompt — a shorter tested prefix
+hashes differently per turn and pinning silently never happens.
+
+## Related
+
+- [[LLM Bench Windows and Measurement]] — the window discipline and usage-counter law
+- [[PCIe Link and Atomics]] — link probing and the streaming-vs-compute ceiling
+- [[RDNA4 MoE Expert Offload VRAM Budget]] — the plan behind duty-cycle terms
+- [[vLLM Porting and Compile Triage]] — the compile fix for the pointwise-glue bound

serverAppConfigs-docs/LLM/Llama Cpp Serving Sizing.md

@@ -80,7 +80,7 @@ co-tenants.
 - **Traps:** mmap'd PLE is reclaimable page cache; `-ot` makes it pinned —
   "move layers to the card" converts reclaimable cache into device memory and
   frees nothing for co-tenants. A served model's footprint ≠ weight size
-  (frank-small: 22.6 of 24.1 GiB for an 11.29 GiB model). Do the capacity
+  (frank-secondary: 22.6 of 24.1 GiB for an 11.29 GiB model). Do the capacity
   subtraction before designing: 12 attn layers × 1.62 GiB + 200k f16 KV =
   28.6 GiB > 24.1 GiB usable → KV-in-VRAM on that pair only reaches ~64k.
 - **Context lengths: derive, never remember.** `llama-bench` computes
@@ -157,7 +157,7 @@ in the timings. Context cap for full-model residency on that shape stayed
 ## Related
 
 - [[GGUF Compatibility]] — what a payload is before it is priced
-- [[frank-small-llama-mimo-q6k-2x200704]] — the current two-boot priced shape
+- [[frank-secondary-llama-mimo-q6k-2x200704]] — the current two-boot priced shape
 - [[frank-medium-llamacpp-dflash2]] — where the 68 KiB/token datum was measured
 - [[DFlash DSpark Speculative Decoding]] — the spec A/B protocol on llama.cpp
 - [[Dual-R9700-Cooperation]] / [[x4-pair-experiment]] — the multi-device side

serverAppConfigs-docs/LLM/Model Swap and Rollback.md

@@ -0,0 +1,281 @@
+---
+type: reference
+title: Model Swap and Rollback
+description: The full model-swap discipline for the LLM fleet — fit preflight, fetch and payload verification, runtime build/flag checks, git-only config, live validation, systemd handoff, close-out, the three-pin revert law, and research-verification discipline.
+tags: [models, swap, gguf, huggingface, rollback, deployment, verification]
+timestamp: 2026-10-05T00:00:00-07:00
+---
+
+# Model Swap and Rollback
+
+Applies to every endpoint swap on the fleet (frank-secondary / frank-large /
+frank-primary). Repo law: all config flows through git; the live models are
+host state under `/home/dank/models/<name>/`.
+
+## 1. Fit preflight — never deploy a swap without confirming it FITS
+
+1. Get the checkpoint's on-disk/weight size AND precision from the HF card
+   (`Precision: BF16` = 2 B/param; FP8/Q8 ≈ 1 B; Q4 ≈ 0.5 B). 35B params:
+   BF16 ≈ 66–70 GB, FP8/Q8 ≈ 33–35 GB, Q4 ≈ 17–20 GB.
+2. Compare against the HOST'S usable RAM (not just VRAM) for runtimes that
+   stream weights from RAM (host-RAM expert offload, llama.cpp CPU offload,
+   layer splits). Budget weights + KV pool + OS + page cache; **if weights
+   alone exceed RAM, ABORT** — pool tuning shrinks KV, it does not shrink
+   weights.
+3. **The failure mode is a self-sustaining thrash, not a crash:** a large
+   swap makes the kernel PAGE instead of OOM-killing → sshd/user space
+   starved indefinitely, no recovery window opens by itself; systemd
+   `Restart=always` turns it into a crash-reload-rethrash loop; `enabled` +
+   `WantedBy=multi-user.target` re-thrashes on every reboot. A power cycle
+   does NOT fix it.
+
+Recovery (may need console / SysRq-to-GRUB):
+
+```bash
+sudo systemctl mask --now <svc>                       # break the reboot trap FIRST
+cd <checkout> && git fetch origin && git reset --hard origin/main
+grep MODEL_DIR <env>                                  # confirm the fixed target
+sudo systemctl unmask <svc> && sudo systemctl restart <svc>
+curl -s <health>                                      # then a REAL completion
+```
+
+Deploy caveat: "healthy after Ns" = systemd ACTIVE, not model LOADED —
+confirm with a real `/v1/chat/completions`. RTX 3090 (Ampere) has no
+FP8/FP4 tensor cores: FP8 weights run via upcast but must still fit; choose
+quant by size fit, then accept the compute path the card supports. Before
+choosing a smaller quant, state WHICH of the two you are buying — headroom,
+or a design change (a smaller quant does not change what the runtime can
+express; per-tensor cross-device placement stays refused under
+`--override-tensor`, and a layer split moves only contiguous layer ranges).
+
+## 2. Ground truth first
+
+- Read the HF model card (runtime pin, hard constraints, recommended args)
+  + the repo trio `scripts/<svc>.env`, `scripts/serve-<svc>.sh`,
+  `scripts/<svc>.service` + the runbook under `serverAppConfigs-docs/LLM/`.
+- Check the host before downloading anything: `ls /home/dank/models/`,
+  `df -h /home`, `du -sh` — bench-era or prior-deployment files often already
+  exist.
+
+## 3. Fetch and verify (HF payloads)
+
+**Inventory by QUANT NAME, never by extension** — large quants ship as
+`.partNofM` fragments (Q4_K_M = 37.00+37.00+36.97 GiB), so an
+`endswith(".gguf")` filter makes Q4_K_S look nonexistent. Check both
+mradermacher repos (`…-i1-GGUF` imatrix, `…-GGUF` plain) and the model's own
+repo; the tree API's `blobs=true` gives `lfs.oid` per file. Gated mirrors
+return 403 even with a valid token until the account accepts the gate —
+prefer the ungated copy.
+
+```bash
+# token out of band: mode-600 curl config; a token passed as -H is world-readable
+printf 'header = "Authorization: Bearer %s"\n' "$(cat ~/.cache/huggingface/token)" > /tmp/hf.curlrc
+chmod 600 /tmp/hf.curlrc
+curl -fL -C - --retry 10 --retry-delay 5 --retry-max-time 21600 -K /tmp/hf.curlrc -o "$DEST/$name" "$BASE/$name"
+```
+
+`-f` is MANDATORY: `curl -o FILE URL` **exits 0 on HTTP 401/403** and writes
+the error body as the artifact (identical 153-byte shards, cheerful DONE,
+zero bytes of model). Never `pgrep -a` a running fetch (token is in
+`/proc/<pid>/cmdline`); watch progress by size:
+`while [ "$(stat -c%s … || echo 0)" -lt "$EXPECTED" ]; do sleep 30; done`.
+
+**Verify, then assemble:** per part, byte count == Hub's declared size AND
+`sha256sum` == `lfs.oid`; concatenate in NUMERIC order; assembled total ==
+sum of parts (compute the sum with python, not by hand) and `head -c 4` ==
+`GGUF`. The assembled file has no published hash — the per-part oids + exact
+total are the chain. Keep the fragments when disk allows (only copies
+re-verifiable against the Hub); failure artifacts and the replaced payload
+go to `~/.AGENTTRASH/YYYYMMDD-topic`, never `rm` — upstreams DELIST quants
+and the local file may be the last copy.
+
+Fetch-script one-liners: dedicated scripts (`scripts/download-*.sh`) already
+do `-f` + byte-count + LFS sha256 + `--verify-only`; ad-hoc fetches tend to
+skip all three. HF cache snapshots SYMLINK every file into `blobs/` and
+those links dangle inside containers — serve scripts and the deployer refuse
+symlinks and missing tokenizer files for exactly this reason.
+
+## 4. Runtime build (when the card pins a fork/build)
+
+- Clone + build with the card's exact cmake line, plus
+  `-DLLAMA_BUILD_WEBUI=OFF` — WebUI asset provisioning (npm / HF llama-ui
+  bucket) aborts llama.cpp link at 98 % on these hosts; the API server is
+  unaffected.
+- Verify every flag the serve script passes exists in the new binary BEFORE
+  touching the service: `llama-server --help | grep -q -- "<flag>"` for each;
+  list MISSes. Read the GGUF's tensor histogram first if the payload uses a
+  custom type — the FORMAT picks the runtime ([[GGUF Compatibility]]).
+
+## 5. Config via git only
+
+Edit `scripts/<svc>.env` (paths, LLAMA_BIN, MODEL_FILE, MMPROJ_PATH) + serve
+script header/defaults locally; `bash -n`; `./tests/run.sh` must pass;
+commit; push; `git pull --ff-only` on the target. Never edit configs on the
+remote. Renaming a model dir (`mv`) is host state and allowed. The env-file
+hook pattern (`FRANK_*_ENV_FILE`) is what makes bench windows and staged
+arms possible — keep it intact.
+
+## 6. Live validation
+
+- No scratch ports — one GPU cannot host two big models. Ensure the old unit
+  is truly gone first: `systemctl is-active` can hang in `stop-sigterm`
+  (llama.cpp never preempts an in-flight generation); poll until the PID
+  exits.
+- Manual serve for smoke: `setsid nohup bash scripts/serve-<svc>.sh >
+  /tmp/x.log 2>&1 &` — plain `ssh 'nohup … &'` dies with its ssh session.
+- Verify: health 200; a REAL `/v1/chat/completions`; reasoning flows
+  (`reasoning_content` non-empty); if the endpoint takes images, a
+  solid-color probe through an `image_url` content part (legacy top-level
+  `images` is silently ignored; probe with ≥256 px).
+- vLLM specifics: first request after boot can take ~18 s (deferred Triton
+  loads) — warm and discard; `--reasoning-parser` / tool parsers are part of
+  the contract ([[vLLM Boot Forensics and Config Interrogation]]).
+
+## 7. systemd handoff — THE TRAP
+
+`./scripts/deploy.sh <host> --service <svc>` gates health, but a leftover
+manual server holding the port makes the health check pass while the UNIT
+crash-loops on bind. Always confirm ownership:
+`systemctl show <svc> -p MainPID` equals the single `pgrep -x llama-server`
+(or the container runtime PID); only then trust "healthy". If a real
+long-context task is queued, do the handoff anyway — client aborts are
+recoverable, drift is not.
+
+## 8. Close-out (mandatory)
+
+1. Docs: active runbook updated (build recipe, profile, validation numbers
+   incl. real-workload prefill/decode); superseded runbook gets a RETIRED
+   banner and stays as the rollback record; `LLM/index.md` mermaid + active
+   list; AGENTS.md fleet table + runtime section + history chain; work-log
+   entry in `serverAppConfigs-docs/log.md`.
+2. `~/.omp/agent/models.yml`: `contextWindow` must equal the REAL per-slot
+   context — llama.cpp logs `n_ctx_slot` per prompt (`--parallel N` divides
+   the pool; e.g. 262144 native / 2 slots = 131072 per slot); vLLM reports
+   `max-model-len` and its concurrency line. Overselling causes silent
+   client-side overflow or a discovered-wall 400. Also `maxTokens` vs the
+   engine's cap, `input` capabilities (text/image), and
+   `reasoningContentField` per runtime (llama.cpp/SGLang emit
+   `reasoning_content`, vLLM emits `reasoning`). Fix every `modelRoles` entry
+   that named the old endpoint.
+3. **Rollback = one env commit** (paths + LLAMA_BIN + flags), prior binary +
+   payload kept on disk, rollback numbers recorded in the runbook.
+
+## 9. Revert-pin an ad hoc experiment (the three pins)
+
+A campaign is "done" only when all three exist — without them a
+`podman pull`, an image update, or a lost scratch script destroys the way
+back:
+
+1. **Git tag (annotated, pushed):**
+   `git tag -a <exp>-tuned-YYYYMMDD -m "<tuned env/flags + measured headline +
+   runbook path + revert command>"` — a future agent must reconstruct the
+   campaign from the tag message alone.
+2. **Image digest in the serve script:** floating tags move. Capture
+   `podman inspect <ctr> --format '{{.ImageName}} {{.Image}}'` + `podman
+   images --digests`, patch `IMAGE=` to `repo@sha256:<repodigest>` with a
+   comment (image ID, date, old ref), verify `podman image exists
+   repo@sha256:<digest>` and `bash -n`. Digest refs survive upstream pushes;
+   `podman pull repo@sha256:<digest>` re-fetches exact bytes. One host can
+   hold MULTIPLE images of one repo — pin by the digest of the RUNNING
+   container, never by tag name. Never edit the pinned line back to a
+   floating tag; new experiments branch from the digest.
+3. **Local alias:** `podman tag <id> <exp>-runtime:tuned-YYYYMMDD`.
+
+Record scratch-script paths + full flag surface in the runbook (hook env
+vars in CAPS); expired images/scripts go to `.AGENTTRASH`, never `rm`.
+Pure measurement sessions that changed no state need only pin 1.
+
+## 10. Research verification discipline (for "can model X run here?")
+
+- Scouted research summaries have shipped fabricated citations on this fleet
+  (invented HF repos, plausible-but-wrong PR numbers, stale engine versions).
+  NEVER copy a scout's citations into a deliverable — re-verify each
+  load-bearing URL directly (GitHub issues/PRs/repos, HF pages) before it
+  enters anything.
+- The in-repo runbooks are FIRST-HAND evidence and outrank any web claim
+  about our own hosts.
+- Verified anchors (2026-09-27): DeepSeek V4.1-Flash (552B/8B-16B active,
+  CED arch, 1M ctx, MIT; preview was 284B/13B with native FP4+FP8 — early
+  GGUF conversions garbage); Lucebox = the proven gfx1201 ROCm 7.2 path
+  (heterogeneous R9700+Strix-Halo V4.1: 86 tok/s decode, 788 t/s prefill@2K).
+  GLM-5.3-Flash (320B/18B, KDA+DSA+mHC, 1M ctx — ALWAYS bind `-c`, the
+  default hard-hung a 128 GB host; native FP8 328.3 GB; `-fa off` MANDATORY,
+  flash-attn breaks MLA correctness; llama.cpp arch `glm5next` lived in
+  unslothai branch → ggml PR #27754, MTP landed d07e71ede 2026-08-30; quant
+  ladder UD-IQ1_S 93.1 GB → UD-Q4_K_XL 199.7 GB, GSQ-RCO 3.5-bit −1.40 pp
+  MMLU-Pro vs Q8_0).
+- GGUF parsing gotcha: expert tensors are
+  `blk.N.ffn_{gate,up,down}_exps.weight` — 4 dot-parts INCLUDING `.weight`;
+  a `len(parts) == 3` filter matches zero layers with no parse error. Filter
+  on `parts[0]=="blk" and parts[3]=="weight" and "exps" in parts[2]`.
+
+## 11. Adding a second LLM endpoint to a host
+
+When a host must serve two models at once (an iGPU runtime plus a second
+card), or one endpoint is moving and a new one takes the freed hardware.
+
+**0. Decide the shape before touching hardware.** One provider per
+HOST:PORT in `~/.omp/agent/models.yml` (on the box where the harness runs)
+— a second endpoint is a new provider block, not a second model in the
+existing one. Read the budget as an inequality: for unified memory the
+engine's own fit check is the arbiter (`budget = host − pinned weights −
+reserve`; `need = KV + arena + vision`); for a discrete card, weight the KV
+by counting only full-attention layers — and remember `--parallel N`
+DIVIDES `--ctx-size`. A co-tenant is a budget line, not a footnote: free
+RAM by changing the POOL or the ARENA (the reserve is a fit-check term,
+never an allocation) and drop optional charges (a vision tower is ~2 GiB)
+rather than hoping.
+
+**1. Repo first, and retire in the same commit.** New service files
+(`scripts/<svc>.env`, `.service`, `serve-<svc>.sh` with hard guards for
+missing artifacts); planner entries (`SERVICE_ORDER` + the target's service
+set + path mappings, inventories in `deploy.sh`/`deploy-remote.sh`, the
+dispatch case); retire the obsoleted pieces into `.AGENTTRASH/` with a
+suffix — and check what pointed at them: bench harnesses, a chat template
+only the old runtime read, and `scripts/test-changed.sh` mappings are the
+usual victims (a mapping to a retired test file fails the pre-commit
+wrapper silently for weeks). One bats file per service asserting
+INVARIANTS (`pool == 2 × ctx`, KV bytes ≤ card budget, port separation),
+not constants; a negative grep must strip comments first
+(`grep -vE '^\s*#' file | grep -q …`) or it matches the prose explaining
+the very thing you removed.
+
+**2. The port-bind race (rootless podman).** pasta binds the published port
+at container CREATE; a just-stopped predecessor still holds it. Order the
+handler: retire the old manager's unit → stop THIS service's own unit
+(no-op on first cutover) → wait for every socket on the port to clear
+([[Container Host Reachability]]) → install → start. Without the second
+step a redeploy deadlocks against its own gate.
+
+**3. Wire the surfaces, refuse impossible signals.** Point the harness
+small/fast + image roles at the new endpoint; move the vision role WITH the
+vision tower (one owner per capability). `shared/homepage/services.json`: a
+card per endpoint, probed from the monitor's own host. Prometheus: only add
+a job if the runtime actually exposes Prometheus text (halogen publishes
+`/health`, `/cache`, `/v1/*` and nothing else — the job was REMOVED with the
+reason written down rather than left as a permanent red).
+
+**4. Cutover, then verify by driving it.** `deploy.sh <target> --service
+<large> --service <small>` (order matters); per endpoint: `/health`,
+`/v1/models` reports the alias, a chat completion returns content AND
+`reasoning_content`, a `tools` request returns `tool_calls`, an image
+request is answered (or refused with a named reason), the runtime's own
+speculation counters move, and the unit is `enabled`/`generated` with linger
+on. Read the engine's boot ledger for a silent clamp. Finally spend one
+real request through the harness per changed role — routing labels lie; the
+servers' own journals do not.
+
+Traps that look like model faults: `500 tools param requires --jinja flag`
+(runtime lacks the Jinja path — pin it); `500 Jinja Exception: Unexpected
+reasoning effort <level>` (fold via `compat.reasoningEffortMap`); the same
+request works by hand but a subagent dies (subagents run at a different
+thinking level — check both); the container dies instantly although the
+predecessor "stopped" (the port race above).
+
+## Related
+
+- [[GGUF Compatibility]] — which runtime can load a payload at all
+- [[Llama Cpp Serving Sizing]] — pricing the shape once it loads
+- [[RDNA4 MoE Expert Offload VRAM Budget]] — vLLM-side fit arithmetic
+- [[LLM Bench Windows and Measurement]] — measuring the new deployment
+- [[dankserver-qwen38-fp8-expert-offload-tuning]] — the current endpoint

serverAppConfigs-docs/LLM/Pipeline Parallelism/Qwen3.8 PP Deployment.md

@@ -48,10 +48,10 @@ state, verified against the live journal and output smoke tests
 
 The PP pair is the only inference service enabled for boot. Both units
 (`frank-large` on Dankclaw, `frank-pp-worker` on Dankserver) are
-enabled. The NPU unit (`frank-small` on Dankclaw) and all retired
+enabled. The NPU unit (`frank-secondary` on Dankclaw) and all retired
 units are disabled. Restart order: worker first, then head. The Dankserver
 helper `frank-restart-small.sh` is obsolete (it restarts the removed
-`frank-small` unit); restart the worker with
+`frank-secondary` unit); restart the worker with
 `sudo systemctl restart frank-pp-worker`. The Dankclaw helper
 `frank-restart-large.sh` (sync + cache drop + restart) is correct for the
 head. `deploy.sh dankserver` is broken until retargeted. See

serverAppConfigs-docs/LLM/Pipeline Parallelism/index.md

@@ -65,6 +65,6 @@ graph LR
 ## See Also
 
 - [[DFlash DSpark Speculative Decoding]]
-- [[frank-small-benchmark]]
+- [[frank-secondary-benchmark]]
 - [[Dankserver System Specs]]
 - [[Dankclaw System Specs]]

serverAppConfigs-docs/LLM/RDNA4 Container Serving.md

@@ -247,4 +247,4 @@ see [[dankserver-qwen38-fp8-expert-offload-tuning]] for why it was rejected);
   measured campaigns
 - [[PCIe Link and Atomics]] — why decode is gather-bound and what cannot fix it
 - [[RDNA4 MoE Expert Offload VRAM Budget]] — arena/plan arithmetic deep-dive
-- [[frank-large-benchmark]] and [[frank-small-benchmark]] — window discipline on live hosts
+- [[frank-large-benchmark]] and [[frank-secondary-benchmark]] — window discipline on live hosts

serverAppConfigs-docs/LLM/Running Qwen3.6 35B A3B FP8 on Dankserver.md

@@ -122,7 +122,7 @@ export HF_HOME=/mnt/entropy/huggingface
 
 ### Active Service Conflict
 
-The retired `frank-small` service (llama.cpp) used about 22 GB of the 24 GB VRAM. The current PP worker holds ~23.8 GiB. The `vllm-awq-qwen36` (Podman) and `vllm-qwen36` units are disabled. Before starting any research service, verify that the GPU is free:
+The retired `frank-secondary` service (llama.cpp) used about 22 GB of the 24 GB VRAM. The current PP worker holds ~23.8 GiB. The `vllm-awq-qwen36` (Podman) and `vllm-qwen36` units are disabled. Before starting any research service, verify that the GPU is free:
 
 ```
 nvidia-smi  # verify GPU is free

serverAppConfigs-docs/LLM/Tcclaviger Fork Ops.md

@@ -0,0 +1,264 @@
+---
+type: reference
+title: Tcclaviger Fork Ops
+description: Operating the tcclaviger RDNA4 vLLM fork on dankserver (shape G era, rollback target since the 2026-10-05 FreeToken cutover) — the ad-hoc sweep harness, measured knob verdicts, sweep hygiene, the large-MoE serving recipe, and compile-cache persistence.
+tags: [tcclaviger, vllm, dankserver, expert-offload, sweeps, cache]
+timestamp: 2026-10-05T00:00:00-07:00
+---
+
+# Tcclaviger Fork Ops
+
+The tcclaviger RDNA4 vLLM fork (`docker.io/tcclaviger/vllm@sha256:da4029666…`,
+vLLM 0.29.0.dev, GLM-5.3/Qwen-specialised with native MoE quant kernels and a
+first-class expert-offload planner) served `frank-primary` at shape G until
+the 2026-10-05 FreeToken cutover; it remains the rollback target
+(`git tag frank-primary-vllm-shapeG-20261003`). Full plan law and VRAM
+arithmetic: [[RDNA4 MoE Expert Offload VRAM Budget]]; boot forensics:
+[[vLLM Boot Forensics and Config Interrogation]]; window discipline:
+[[LLM Bench Windows and Measurement]].
+
+## The ad-hoc sweep harness (dankserver)
+
+Serve script `/home/dank/serve_qwen_fp8.sh` with env hooks: `SPEC` (MTP
+depth), `TUNABLE` (TunableOp), `PREFETCH`, `UTIL`, `MAXLEN`, `KV_BYTES`,
+`MNBT`, `UVA_TOK`, `GATHER_C/GATHER_L` (R4D_LRU gather), `TELEM`
+(R4D_LRU_TELEMETRY), `PROFDIR` (torch profiler bind-mount + --profiler-config
+JSON). The script re-copies the harness (`mtp_measure.py`, `bench_iter.py`)
+into every fresh container — `podman cp` files die on container recreate.
+
+- Measure ONLY via usage counters: `mtp_measure.py` (3 depths) or
+  `mtp_measure_short.py` (166 ctx). `decode = completion_tokens/(wall −
+  ttft)`; chunk counting is INVALID under spec decode.
+- Jobs: `bash /home/dank/job.sh start NAME CMD…` / `wait NAME SECS 40` /
+  `out NAME` ([[Remote Job Runner]]).
+- **Validity check on ANY speed result:** `completion_tokens == max_tokens`
+  (else repetition-halt or early EOS), finish_reason, and inspect the text —
+  `bench_iter` prompts are repeated word-lists + `!` padding, loop-prone. A
+  loop makes LRU hits ~100% → PCIe traffic vanishes → fake 3–4× decode.
+  Acceptance swings 48–61% run-to-run — noise; decode is the metric.
+- Each arm = full reload: ~8.5 min cold, ~12 min with measurement.
+- Readiness gate: `podman logs <c> 2>&1` (stdout+stderr merged) for
+  `Application startup complete` — it is on STDERR; an unmerged grep can
+  never match and burns the whole wait timeout.
+- GPU util ~0% during load/compile is NORMAL (weights + CUDA-graph capture
+  are CPU-side); VRAM climbing is the load indicator.
+- Config edit discipline: the serve script is patched by scp'd python
+  scripts with asserts (anchor must exist, addition must not); `bash -n`
+  after every patch.
+
+## Measured knob verdicts (2026-09-29 campaign)
+
+Incumbent: `SPEC=4 PREFETCH=1 UTIL=0.98 MAXLEN=32768 KV_BYTES=2147483648
+TUNABLE=0`, stock gather 8×16 → 26.80/25.15/26.81 tok/s @ 166/1087/4064 ctx.
+
+| Knob | Verdict | Evidence |
+|---|---|---|
+| Gather geometry `R4D_LRU_GATHER_{CHUNKS,LANES}` | **RULED OUT** | 16×32 worse (~24); 32×64 tie (26.6). The 9.5 vs 22.7 GB/s read-path gap is per-transfer structure (~105 µs/slab vs 78 ideal), not a tunable knob |
+| MTP depth | **RULED OUT** | SPEC=3 ~9% worse; SPEC=6 collapses (13.5–19 tok/s, acceptance 17.4%) — also proves UVA read-through (crosses at step tokens > `R4D_LRU_UVA_TOKENS=5`) is SLOWER than copy-in |
+| TunableOp | **REJECTED** | `CLAV_TUNABLEOP_SWEEP=1` silently SKIPPED under expert offload unless `CLAV_TUNABLEOP_WITH_OFFLOAD=1`; adds ~7 min/load, persists nothing (container-local), and CORRUPTS NUMERICS (loops, repetition-halt, fake 79.8 tok/s @ 96.5% acceptance on degenerate completions). Any "3× win" with early-EOS + absurd acceptance = the loop artifact |
+| MNBT | neutral at 8192 vs 2048 | small values re-chunk prefill only (hurts TTFT, never decode — decode step = max-num-seqs × (spec+1) tokens). No ubatch concept in vLLM |
+
+Decode on this stack is PCIe-gather-bound (~30% GPU duty cycle), so
+batch-shaped compute knobs move ~nothing.
+
+The `R4D_*/CLAV_*` knob family (QSA_*, MHC, ROUTE, ACT_QUANT, AR_QUANT,
+SILU_QUANT) are kill-switches defaulting ON, not perf dials; real levers are
+the R4D_LRU_* family, TunableOp (rejected), R4D_LRU_TELEMETRY, and spec
+depth.
+
+Telemetry gotchas: `--profiler-config` takes a JSON blob, not separate
+flags; `torch_profiler_with_stack` defaults TRUE — a full measure inside a
+profile window OOM-kills the engine worker (~40 GB trace buffer; kernel
+kills python3 at ~95 GB anon-rss). Keep windows to ~15 s / one depth
+(`mtp_measure_short.py`). `telemetry_report` (`expert_cache_rank<N>.json`)
+fires ONLY from the worker's profiler-stop path; worker-side stop can hang
+>5 min on ROCm/kineto. No alternative trigger exists. The LRU hit rate
+measured 82.45% — NOT the ~27% the arena-fraction arithmetic implies;
+temporal reuse dominates, so never infer miss rates from resident-slot
+fractions.
+
+## Sweep hygiene (fork-specific preflight)
+
+1. **Knob landed:** for each `-e` env knob, confirm in-container
+   `podman exec … env` (or the engine log's confirmed-values line). A knob
+   the fork silently ignores produces a "parity" arm that is really the
+   default — the v1 gather sweep lost all three arms to exactly this
+   (GATHER_C/GATHER_L silently dropped, all arms ran stock 8×16).
+2. **Prefetch uniformity:** every arm in a sweep MUST carry the same
+   expert-prefetch setting. The 2026-09-29 campaign measured losing arms
+   with prefetch ON and incumbents OFF; untaxed, its verdicts inverted
+   (no-spec 12.11→41.47; gather 32×64 parity→+18.7%). Never re-read a taxed
+   verdict.
+3. **Spec arms must prove engagement:** engine log
+   `SpecDecoding metrics: Mean acceptance length …` and
+   `curl :7770/metrics | grep vllm:spec_decode_`. Zero such lines = MTP off.
+4. Boot-fail fast-detection: poll `podman ps` INSIDE the health loop; a
+   refused boot dies in one poll instead of 20 min of health waits. Capture
+   >8 lines of `podman logs` on failure — the root cause (e.g. a SEQS init
+   refusal) lives higher up.
+5. Report per-depth tables, never one number: g32×64's win grows with depth
+   (54@1k, 39.5@short) — a single 4064 median hides the short-ctx cost. TTFT
+   rows are noisy ±0.15 s and non-monotonic across depths; compare same-ctx
+   rows across arms only.
+6. Generate ≥256 tokens per sample (at 8 tokens inter_token is first-token
+   noise).
+
+Boot-log facts survive measure failures: KV pool size (`GPU KV cache size`),
+arena slots (`expert cache: … -> N slots`), and the spec-reservation ladder
+(MTP3 766,666 / MTP4 729,545 / MTP6 665,263 tokens) print at BOOT — a
+measure-stage bug costs decode tables but not boot-log findings.
+
+**MTP depth-brittleness law (2026-10-02, all arms rc=0):** MTP3 on this fork
+is depth-brittle — 47.01 t/s at 166 tokens (47.88 at temp 0.7) but 23.09 at
+4064, vs no-spec baseline 37.02 / 33.99. MTP3 + SEQS 16 + MNBT 4096 + util
+0.95 + ladder [4..32] = best overall 37.04 / 38.61 t0.7 and keeps 33.15 at
+4064. `--expert-precise-memory` is load-bearing for MTP (same arm with it
+OFF = worst arm, 19.81 at 4064). Every spec arm costs KV (398,658 → 313,002
+tokens). Acceptance is real: mean length 2.4–2.8, per-position
+0.76/0.55/0.33, avg 52–57%. Next lever named by the engine: MNBT 4096 is
+exactly SEQS×(n+1) (`suboptimal` warning) → try 8192.
+
+## Serving a model larger than VRAM (dankserver recipe)
+
+Dankserver: 2× R9700 (gfx1201), 31.86 GiB each = 63.7 GiB real VRAM, 246 GiB
+RAM. `torch.cuda.device_count()` returns 3 and summing `total_memory` gives
+a bogus 186.7 GiB (the Raphael iGPU is in the list — see
+[[Llama Cpp Serving Sizing]] for the reading discipline).
+
+**Kernel gate — this decides which checkpoint can work at all.**
+`/app/r4dhip/r4d.so` exports `gemm_moe_mxfp4a8_*` and `gemm_moe_dsfp4a8_*`
+but NO `gemm_moe_w4a16_*`. Expert offload requires an experts class with
+BOTH a compiled kernel AND an `offload_recipe`; the planner aborts startup
+otherwise. Usable: MXFP4, DSFP4, FP8-HIP (`R4dMxfp4MoEExperts`,
+`R4dDsfp4MoEExperts`, `Fp8HipMoEExperts`). `R4dW4a16MoEExperts` has no
+kernel (the class exists, the r4d.so entry does not); RFA has neither. Check
+the checkpoint's quant against this list BEFORE downloading anything.
+
+Entrypoint is `/app/tools/image_entrypoint.sh`, ending `exec vllm serve
+"$@"` — pass the model path as the FIRST argument and do NOT pass `serve`.
+Known-good invocation:
+
+```bash
+sudo sh -c 'sync; echo 3 > /proc/sys/vm/drop_caches'
+podman run -d --name <ctr> --ipc=host --ulimit memlock=-1:-1 \
+  --cap-add SYS_PTRACE \
+  --device /dev/kfd --device /dev/dri --device /dev/mem \
+  -e HIP_VISIBLE_DEVICES=0,1 -e VLLM_ROCM_USE_AITER=0 \
+  -v /home/dank/models/<model>:/models:ro \
+  docker.io/tcclaviger/vllm:latest /models \
+  --served-model-name <name> --port 7770 \
+  --tensor-parallel-size 2 \
+  --max-model-len 32768 \
+  --kv-cache-memory-bytes 2147483648 \
+  --max-num-seqs 4 \
+  --gpu-memory-utilization 0.95 \
+  --enable-prefix-caching --trust-remote-code
+```
+
+`-p 7770:7770` is not included, so the API is only reachable via
+`podman exec` — add `-p` if host-side curl is needed. Load ~5 min;
+`/health` answers connection-refused the whole time; watch for
+`Application startup complete` in `podman logs <ctr>`. Verified working:
+`orcarouter/Qwen3.8-Flash-Next-Uncensored-FP8` (gated; 172.8 GiB, 131
+shards, `Qwen4ExpForConditionalGeneration`, 48 layers, 512 experts top-10,
+multimodal, native 262,144).
+
+Traps that each cost a boot:
+
+1. **Mount the whole `/dev/dri`, never individual render nodes.** Render
+   nodes without the primary `card*` nodes → HIP enumerates two agents and
+   refuses a context on either (`RuntimeError: No CUDA GPUs are available`)
+   while `device_count()` cheerfully returns 2. Confirmed by direct A/B.
+2. **memlock must be raised** (done on Dankserver via
+   `/etc/security/limits.d/99-memlock.conf`). Host default 8 MiB makes any
+   large pinned allocation fail as a misleading
+   `torch.AcceleratorError: CUDA error: out of memory` from `pin_memory()`;
+   it cannot be raised from inside a container or root's own shell — only
+   PAM applies it at login. Matters because `--expert-host-mlock` defaults
+   True.
+3. **`--gpu-memory-utilization` does NOT buy KV under expert offload** —
+   the planner sizes the arena against the budget and leaves KV the
+   remainder; raising util 0.95→0.98 grew the arena and the KV pool stayed
+   at 0.64 GiB.
+4. **Lowering `--max-model-len` is the WRONG lever and backfires** — the
+   budget subtracts the KV reserve first, so a smaller context shrinks the
+   reserve and hands the space to the arena (measured: 65536 → 0.64 GiB KV;
+   32768 → 0.34 GiB).
+5. **The KV reserve undercounts hybrid-attention models ~2×** (9190 B/token
+   planned vs ~18,000 real: the Mamba state is a fixed per-sequence cost).
+   Fix by pinning the pool: `--kv-cache-memory-bytes`.
+6. **`--expert-offload-mem` caps the HOST side**, not the VRAM arena —
+   structurally orthogonal to `--expert-precise-memory` (verified
+   in-container 2026-10-05: precise-memory only swaps the GPU overhead term,
+   expert_offload_mem only sets the host bound; they rejoin only inside
+   `place_experts`, and the plan rewrites `expert_offload_mem` back to the
+   cap the plan met, so plan and allocation cannot drift).
+7. `--ipc=host` and `--shm-size` are mutually exclusive in podman; pick one.
+8. `podman run -d` without `-a` sends no stdout to a redirect — the log file
+   only receives the container ID; use `podman logs`.
+
+Status reads: use `scripts/dankstatus.sh`, never hand-rolled
+`/proc/meminfo` parsing — `/proc/meminfo` separates key from value with a
+TAB, so `grep MemAvailable | tr -s " " | cut -d" " -f2` returns empty and
+awk prints `0.0` — a false "host out of memory" that invites an unnecessary
+abort. Gated checkpoints: token via a mode-600 `curl -K` config, never argv
+(`pgrep -a curl` prints it); always `--fail`; compare each file's byte count
+to the Hub's declared size; `-C -` resumes truncated shards.
+
+**Check whether the stack supports the model natively before porting an
+overlay.** The image already registered `Glm5NextForConditionalGeneration`,
+`Qwen4ExpForCausalLM` and both MTP variants — chasing four missing symbols
+through two image tags to bring an overlay up was entirely wasted work.
+
+## Compile-cache persistence (survives `podman rm -f`)
+
+Serve scripts that recreate the container discard the writable layer, so
+every boot re-pays compile/JIT (frank-primary: `torch.compile took 181.20 s`
+cold, of which 173 s was one graph).
+
+1. **Prove it and locate it:** fresh container vs live
+   (`podman run --rm --entrypoint bash IMG -c "ls -la /root/.cache"` vs
+   `podman exec LIVE bash -c "du -sb /root/.cache/*; find /root/.cache -type
+   f | wc -l"`). vLLM cache root: `VLLM_CACHE_ROOT` (default
+   `/root/.cache/vllm`, `torch_compile_cache/`); a HIP runtime adds
+   `/root/.cache/comgr`.
+2. **Mount the PARENT, not the leaf** — one bind on `/root/.cache` persists
+   both trees, safe whenever the image ships no such directory (step 1
+   proves it; otherwise you would be hiding baked state). Wire as an opt-in
+   tunable: `mkdir -p "$CACHE_DIR"` BEFORE the mount, then
+   `${CACHE_DIR:+-v "$CACHE_DIR":/root/.cache}`. Host path outside the
+   checkout.
+3. **Verify with two consecutive boots** — cold `Compiling a graph … takes
+   N s`, warm `Directly load the compiled graph(s) for compile range (…)
+   from the cache` with `torch.compile took` in single-digit seconds.
+4. Traps: the cache key includes the compile RANGE — an MNBT change costs
+   ONE recompile, not a permanent tax; do not claim a proportional boot cut
+   (time to `/health` is dominated by weight loading: 490 s warm vs 188 s
+   cold compile); stale entries are misses, not wrong answers (version +
+   config hashed); if the serve script is seam-gated, assert the mkdir and
+   the exact mount target.
+
+## Plan-line verification checklist (after any fork deploy)
+
+```
+ssh dankserver 'sudo journalctl CONTAINER_NAME=frank-primary-qwen -b -o cat --no-pager \
+  | grep -aoE "(budget \(total x util\) +[0-9.]+|expert cache space +[0-9.]+ GiB +[0-9,]+ slots|Estimated CUDA graph memory.{0,80}|Graph capturing finished.{0,40}|GPU KV cache size.{0,90})"'
+```
+
+`grep -oE` with a digit-anchored tail — plain `grep`/journalctl truncate
+these columns to `17.`. Verify: new budget/graphs/arena in the plan;
+`GPU KV cache size` matches the requested token count; `Estimated CUDA graph
+memory` vs the capture line (raise `_PER_GRAPH_MB` toward 52 if capture
+exceeds the charge); `rocm-smi --showmeminfo vram` high-water per card = the
+real margin; `systemctl show frank-primary -p NRestarts --value` = 0;
+`VLLM_CUDAGRAPH_*` present in `podman exec … env`; then a real chat
+completion from a PEER — `curl localhost:7770/health` inside an ssh session
+on the host can return `000` and looks like an outage that isn't.
+
+## Related
+
+- [[RDNA4 MoE Expert Offload VRAM Budget]] — the plan law this harness tunes
+- [[vLLM Boot Forensics and Config Interrogation]] — journal forensics for these boots
+- [[LLM Bench Windows and Measurement]] — window discipline and bash laws
+- [[Model Swap and Rollback]] — where the fork sits in the swap history
+- [[Remote Job Runner]] — the job.sh harness referenced above

serverAppConfigs-docs/LLM/VLLM Overview.md

@@ -268,7 +268,7 @@ ad hoc qwen-fp8 container (the same fork, same payload, same TP=2):
 
 **There is no `--ubatch-size` equivalent in vLLM.** `--ubatch-size` is a
 llama.cpp compute-buffer concept (`--batch-size`/`--ubatch-size`); the tcclaviger
-fork has no ubatch knob at all. The llama.cpp compute-buffer pair that frank-small
+fork has no ubatch knob at all. The llama.cpp compute-buffer pair that frank-secondary
 was doubled on (2026-10-05, +4.6 % prefill at 8,928 tokens for 616 MiB) has no
 counterpart here, and the two backends should not be reasoned about by analogy.
 

serverAppConfigs-docs/LLM/dankserver-vram-growth.md

@@ -1,7 +1,7 @@
 ---
 type: reference
 title: Dankserver VRAM and RAM Growth Under Load
-description: Investigation of the recurring cuBLAS crash on frank-small. What grows, what does not, and what to do next.
+description: Investigation of the recurring cuBLAS crash on frank-secondary. What grows, what does not, and what to do next.
 tags:
   - dankserver
   - llamacpp
@@ -18,7 +18,7 @@ timestamp: 2026-09-03
 # Dankserver VRAM and RAM Growth Under Load
 
 This page records the investigation of the recurring cuBLAS crash on
-`frank-small` (llama.cpp, RTX 3090 24 GiB). The service aborts
+`frank-secondary` (llama.cpp, RTX 3090 24 GiB). The service aborts
 with `CUDA error: an unsupported value or parameter was passed to the
 function` in `cublasSgemm_v2` at `ggml-cuda.cu:1541` during
 `update_slots` -> `decode` -> `process_ubatch`. It crashed on 2026-08-13
@@ -116,7 +116,7 @@ Verified 2026-08-14 after deploy: flash attention enabled, warmup clean, decode
 
 ## Mitigation Options
 
-The documentation for the benchmark ([[frank-small-benchmark]])
+The documentation for the benchmark ([[frank-secondary-benchmark]])
 prescribes the first option for this crash class. None of these change
 the model.
 
@@ -135,6 +135,6 @@ PR #25649 (slot checkpoints freed on prompt clear, merged), issue #26119
 
 ## See Also
 
-- [[frank-small-benchmark]]
+- [[frank-secondary-benchmark]]
 - [[Dankserver System Specs]]
 - [[deployment]]

serverAppConfigs-docs/LLM/dealign-ornith-crack.md

@@ -1,7 +1,7 @@
 ---
 type: reference
 title: dealign Ornith-1.5-35B-A3B CRACK on Dankserver
-description: dealignai/Ornith-1.5-35B-A3B-UNCENSORED (CRACK Q6_K) served by upstream llama.cpp on the RTX 3090. MoE model with ngram-mod speculative decoding; selected expert weights offloaded to CPU with an explicit --override-tensor. frank-small.
+description: dealignai/Ornith-1.5-35B-A3B-UNCENSORED (CRACK Q6_K) served by upstream llama.cpp on the RTX 3090. MoE model with ngram-mod speculative decoding; selected expert weights offloaded to CPU with an explicit --override-tensor. frank-secondary.
 tags:
   - llamacpp
   - cuda
@@ -14,12 +14,12 @@ timestamp: 2026-08-24
 
 # dealign Ornith-1.5-35B-A3B CRACK on Dankserver
 
-> **Active 2026-08-25.** `frank-small` serves this model. It
+> **Active 2026-08-25.** `frank-secondary` serves this model. It
 > replaced the CRACK Q8_0 file on the same upstream llama.cpp binary (b10121).
 > The Q6_K file was downloaded from the model repository and validated with
 > `llama-gguf` before installation.
 
-`frank-small` (port 7770) serves
+`frank-secondary` (port 7770) serves
 `Ornith-1.5-35B-A3B-CRACK-Q6_K.gguf` from `dealignai` on Hugging Face. It is
 a **MoE** model (256 experts, 8 active) with a hybrid GatedDeltaNet (SSM) +
 attention architecture, abliterated (uncensored). The GGUF also contains a

serverAppConfigs-docs/LLM/deployment.md

@@ -21,7 +21,7 @@ SearXNG, Caddy, or another container service does **not** restart inference.
 |---|---|---|---:|
 | Dankserver | `dank@dankserver` | `frank-medium` | 7770 |
 | Dankclaw | `dank@dankclaw` | `frank-large` | 7770 |
-| Dankclaw | `dank@dankclaw` | `frank-small` | 7771 |
+| Dankclaw | `dank@dankclaw` | `frank-secondary` | 7771 |
 
 ## Scoped workflow
 
@@ -53,13 +53,13 @@ on the selected host. The following files select the corresponding service:
   checkout, so any file there changes deployed state.
 - `scripts/frank-large.env`, `frank-large.service`, and
   `serve-frank-large.sh` select `frank-large` on Dankclaw.
-- `scripts/frank-small.env`, `frank-small.service`, and
-  `serve-frank-small.sh` select `frank-small` on Dankclaw.
+- `scripts/frank-secondary.env`, `frank-secondary.service`, and
+  `serve-frank-secondary.sh` select `frank-secondary` on Dankclaw.
 - Documentation, tests, deployment tooling, and unrelated container changes
   do not select any inference service.
 
 The target-side `scripts/deploy-remote.sh` installs the selected service.
-`frank-large` and `frank-small` are **root units**: the handler relinks the
+`frank-large` and `frank-secondary` are **root units**: the handler relinks the
 repo file into `/etc/systemd/system/` (`relink_unit`), reloads systemd,
 enables it and restarts it. `frank-medium` is a **rootless Quadlet**: the
 handler retires its predecessors (the replica Quadlets and the retired llama
@@ -67,7 +67,7 @@ root unit), stops the service's own unit, waits for the published port to
 clear, installs the `.container` and restarts the generated unit. Each
 handler then waits on that service's health endpoint —
 `http://127.0.0.1:7770/health` for frank-large and frank-medium,
-`http://127.0.0.1:7771/health` for frank-small. The wait budget is 20 minutes
+`http://127.0.0.1:7771/health` for frank-secondary. The wait budget is 20 minutes
 because cold model loads are slow. A failed handler does not advance its
 per-service state.
 
@@ -76,7 +76,7 @@ per-service state.
 The root units point into the stable target checkout:
 ```text
 /home/dank/serverAppConfigs/scripts/frank-large.service
-/home/dank/serverAppConfigs/scripts/frank-small.service
+/home/dank/serverAppConfigs/scripts/frank-secondary.service
 ```
 `frank-medium` is a Quadlet under `dankserver/frank-medium/quadlet/`
 (`frank-medium.container`) rather than a unit in `scripts/`.
@@ -110,7 +110,7 @@ change in that service's inputs.
 
 If the LLM handler fails after the target pull:
 
-1. Inspect `journalctl -u frank-large -n 200` or `journalctl -u frank-small
+1. Inspect `journalctl -u frank-large -n 200` or `journalctl -u frank-secondary
    -n 200` (root units), or `journalctl --user -u frank-medium -n 200`
    (rootless Quadlet).
 2. Confirm the model files, the runtime binary and (for frank-medium) the
@@ -120,13 +120,13 @@ If the LLM handler fails after the target pull:
    ```bash
    ./scripts/deploy.sh dankserver --service frank-medium
    ./scripts/deploy.sh dankclaw --service frank-large
-   ./scripts/deploy.sh dankclaw --service frank-small
+   ./scripts/deploy.sh dankclaw --service frank-secondary
    ```
 4. Verify the matching health endpoint before declaring recovery.
 
 Do not stop the other host's inference service during recovery. The two
 Dankclaw endpoints are independent processes on different devices: restarting
-frank-large does not disturb frank-small, and the reverse holds too — with one
+frank-large does not disturb frank-secondary, and the reverse holds too — with one
 exception worth knowing, that the deployer waits for port 7770 to be released
 before it starts the replacement container. For benchmark work, follow the
 separate stop/benchmark/restart procedure in [[Testing Policy]] and the

serverAppConfigs-docs/LLM/frank-large-flash-server-v2-swift15.md

@@ -0,0 +1,191 @@
+# frank-large — halogen-flash-server 0.16.2, Qwen3.8-Flash-Next v2 Swift1.5 ABLITERATED (Dankclaw, Strix Halo iGPU)
+
+**ACTIVE since 2026-10-06.** Rollback = revert the swap commit's trio
+(`scripts/frank-large.env`, `scripts/serve-frank-large.sh`,
+`scripts/frank-large.service`); the retired 27B payload stays on disk at
+`/home/dank/models/halogen-qwen3.8-27b/` and its fetch script
+(`scripts/download-frank-large-halogen.sh`) stays in the repo.
+
+## Why this shape exists
+
+Frank's ask: replace the frank-large model with
+`https://huggingface.co/Quat3rnion/halogen-swift1.5-qwen3.8-flash-next-v2-abliterated`.
+That payload is an **abliterated Swift1.5 finetune of peonist's
+Qwen3.8-Flash-Next v2 checkpoint in the v2 `.hgn` format** — a file that
+loads ONLY in the **halogen-flash-server** engine line, NOT in the halogen
+0.1.4 container that served the 27B dense checkpoint. The model move is
+therefore a **full engine migration**, and the accepted-product facts flip
+twice in the fleet's favour:
+
+- the base is **abliterated** (the 27B halogen endpoint was a finetune of
+  the censored `Qwen/Qwen3.8-27B`; refusals were an accepted regression there),
+- **vision is back** — the v2 vision tower loads beside the checkpoint and
+  the engine runs it (the 27B halogen engine silently ignored its encoder),
+- **MTP speculative decoding returns** with the head inside the checkpoint
+  itself (the 27B shipped a DFlash2 drafter instead; comparable purpose,
+  different file plumbing),
+- the card claims **53–57 t/s greedy decode, ~1,640 t/s prefill at 34k, ~72 %
+  fewer thinking tokens** than the censored 27B (905 mean), and **262,144
+  native context**. Publisher-card numbers — verify against live traffic
+  before quoting further (see Verification).
+
+The engine is **closed-source**, ships as `ghcr.io/peonist-ai/
+halogen-flash-server:0.16.2` (the version the model card's quickstart pins),
+digest-pinned:
+
+```
+ghcr.io/peonist-ai/halogen-flash-server@sha256:0c61bf84ac22308a53f5d1ca6b86806702d7039e5ebc51cae4c66621b92fe04a
+```
+
+(3.6 GB image; RepoDigest captured at pull 2026-10-06. Card-tested fallback
+if a future pin misbehaves: 0.15.1.)
+
+## The payload — a triangle of files, all sha256-verified at fetch
+
+| File | Bytes | Source repo @ revision | sha256 (LFS) |
+|---|---|---|---|
+| `qwen38-flash-next-v2-swift15-abliterated.hgn` | 66,687,678,464 | Quat3rnion/halogen-swift1.5-qwen3.8-flash-next-v2-abliterated @ main (`69a85db`) | `9c880b8ba48deabfa06172eff1e7572e4fe465e09e97faec6e50c3f6fdcac037` |
+| `qwen38-flash-next-ngram.hgn` (REQUIRED) | 51,200,246,144 | peonist-ai/halogen-qwen3.8-flash-next @ `5cc17cea` | `3450acada94e19aabad88bd49b45eb70304820949c2240fc178067b20ae5dffe` |
+| `qwen38-flash-next-vision.hgn` (enabled) | 897,916,416 | peonist-ai/halogen-qwen3.8-flash-next @ `e053f488` | `d62e0ae553fe88afd3833733d4a4c669f34d20fd8dfce4b9610525bed2134b10` |
+| `tokenizer/` (6 files, flat dir) | 22.4 MB total | peonist @ `e053f488` | `tokenizer.json` is LFS `0997f410…` (byte-identical to the 27B-era copy); the rest are plain git blobs, size-checked |
+
+Payload notes worth keeping:
+
+- The deployed checkpoint is a **compacted r2** of the card's append-only
+  original (89,775,868,032 B → 66,687,678,464 B): the abliteration workflow
+  appends and the repo's `tooling/` compact drops dead padding; live tensors
+  are byte-identical, the LFS sha256 is the contract, and the 24 GiB saving
+  is why it fits.
+- **`HALOGEN_MTP_HEAD` stays UNSET**: the MTP head is INSIDE the v2
+  checkpoint (the separate `qwen38-flash-next-mtp.hgn` sidecar is for a
+  different assembly and is deliberately not fetched).
+- The **n-gram lookup table is REQUIRED**. The engine reads it beside the
+  checkpoint (documented fallback); the card's quickstart also names it
+  explicitly via `HALOGEN_NGRAM_TABLE`, and the serve script passes both —
+  the env var is belt-and-braces over the documented beside-the-checkpoint
+  default, and both mechanisms resolve to the same file.
+- The tokenizer is fetched fresh from peonist @ `e053f488` into
+  `/home/dank/models/halogen-flashnext-v2/tokenizer/` — same flat-directory,
+  no-symlink law as every other payload (an HF cache snapshot's symlinks
+  dangle inside the container mount; both the serve gate and the deployer
+  refuse them).
+
+Fetch: `scripts/download-frank-large-flashnext.sh` (`--verify-only` to
+re-check; `-f` + exact byte counts + LFS sha256 + resolve-the-CDN-then-`-C -`
+resume law throughout; no token — the repos are public).
+
+## Shape — the knobs that are not free
+
+`scripts/frank-large.env` carries the full rationale inline; the table:
+
+| Env | Value | Why |
+|---|---|---|
+| `HALOGEN_CTX` | 200192 | per-slot HARD ceiling (a longer prompt is a 400 naming the limit, never truncation). Harness compaction: 70 % = 140,134; 140,134 + 57,344 = 197,478 ≤ 200,192 |
+| `HALOGEN_KV_SLOTS` | 2 | the 27B engine's "slots>1 disables the drafter" law does NOT carry over — on this engine batching and speculation coexist (the card's quickstart ships 2 slots WITH the head). 2 slots = concurrent capacity |
+| `HALOGEN_KV_POOL_POSITIONS` | 400384 | explicit form of the default (2 × ctx), so the seam test pins the product; ~28 KiB/position ⇒ ~10.7 GiB KV |
+| `HALOGEN_HOST_RESERVE_GIB` | 20 | keeps the host honest (the w4b-era lesson) |
+| `HALOGEN_SPEC_ADAPT` | **0** | MANDATORY (card): the default adaptive policy turns the MTP head off for whole requests after low acceptance and **halves decode**. The head stays on; temp-1.0 traffic pays some acceptance for it — accepted trade |
+| `HALOGEN_MAX_TOK` / `HALOGEN_MAX_TOKENS_CAP` / `HALOGEN_QUEUE_TIMEOUT` | 8192 / 65536 / 7200 | 8192 is only the per-request DEFAULT (the harness always sends 57,344); the cap/timeout pair is COUPLED per FLAGS.md (65536 at ~10 t/s serial needs > 6550 s) |
+| `HALOGEN_TEMPERATURE` / `TOP_P` / `TOP_K` | 1.0 / 0.95 / 20 | the card's thinking recipe as SERVER-SIDE defaults — the engine HAS a sampler now, so an absent temperature is no longer greedy; models.yml extraBody matches it (consistent, not load-bearing) |
+| `HALOGEN_WEIGHTS_LOCK` | 1 | card's belt-and-braces; the engine pins weights by default on v2 (`HALOGEN_FLASH_PIN_TRUNK=0` is a w4b-only knob and is REFUSED by v2) |
+| `HALOGEN_MODEL_ID` / publish | frank-large / 7770:8731 | unchanged aliases; engine port 8730 never published |
+| MTP depth | default 2 | FLAGS.md: 2 is best all-round, 3 for code — Frank's traffic is agent/prose |
+
+**Memory ledger** (121.2 GiB unified, the referee is the boot ledger):
+weights ~62 GiB resident + KV ~10.7 + arena ~8 (`3.8 + 8.4 × 8192/16384`)
++ vision ~2 ≈ **82.7 GiB engine-side**, + 20 host reserve = ~103 with
+~18 GiB margin over the box's baseline co-tenancy. The 51.2 GB n-gram
+table rides as mmap'd page cache beside the checkpoint — the card's own
+quickstart arithmetic does not bill it as resident, but WATCH the ledger
+at boot and re-size if the engine prints a clamp. Prompt cache is ON by
+default (`PROMPT_CACHE=2`): a follow-up turn at 100k ctx costs ~2 s
+instead of ~88 s cold — the big-TTFT win the 27B engine needed a manual
+`HALOGEN_CACHE_MB` pin for is built in here.
+
+## Engine contract facts (FLAGS.md of the flash-server repo)
+
+- OpenAI chat **and** `/v1/responses` **and** `/v1/messages` **and**
+  `/v1/completions`; tool calls verified on the card; `response_format`
+  still unsupported (no constrained decoding).
+- Thinking → **`reasoning_content`** (same field as every engine this
+  endpoint has run); the v2 chat template rejects `reasoning_effort`
+  outside `xhigh|medium|low` — the models.yml fold (high→xhigh) stays
+  mandatory.
+- Vision: OpenAI `image_url` content parts only (the fleet's standard
+  contract; the legacy top-level `images` field is the frank-secondary trap).
+  ≥256 px probe images or the model answers "video".
+- `HALOGEN_MAX_TOKENS_CAP` over-cap is a 400 naming the limit — never a
+  truncation. `max_tokens` INCLUDES thinking.
+- All flags read once at startup; every flag is BITWISE except
+  `HALOGEN_CK_OVERLAY`, `HALOGEN_MATMUL_TUNING_FILE`,
+  `HALOGEN_INDEXER_BUDGET`, `HALOGEN_COMPOSABLE_CONTEXT`. Unset
+  `HALOGEN_CHECKPOINT` defaults to `/models/qwen38-flash-next-v2.hgn`
+  with the lookup table beside it.
+- Prompt-lookup drafting (3,3 window) rides beside the MTP head by
+  default and is byte-identical to greedy verification where it fires.
+- Watchdog: `HALOGEN_ENGINE_WATCHDOG_S=180` with defer-logic for memory
+  stalls — on a shared host the defaults are the safe choice; the
+  restart policy stays real through the root unit's `Restart=always`.
+
+Kernel: `CONFIG_HSA_AMD_SVM=y` (CachyOS ships it); `amdgpu.noretry=-1`
+(auto) — for this IP auto resolves to retry-on, the same effective state
+the 27B engine ran on. Only revisit if the ledger shows SVM complaints.
+
+## Mechanics
+
+- **Root systemd unit** (`scripts/frank-large.service`, `User=dank`,
+  `LimitMEMLOCK=infinity`, `Restart=always`, `ExecStop=podman stop -t 60
+  frank-large-halogen`), driven by `scripts/serve-frank-large.sh` — the
+  fleet's `podman run`-direct pattern. The script runs the container
+  FOREGROUND so the boot ledger (weights resident, pool fit, MTP head,
+  prompt cache, any clamp) lands in `/home/dank/logs/frank_large.log`.
+- Serve gates mirror the deployer EXACTLY (2026-10-01 crash-loop law):
+  image digest in podman storage, byte-exact checkpoint + n-gram table +
+  vision tower, flat tokenizer with all six files as regular files,
+  GPU nodes present.
+  - **Host-RAM pin-gate preflight** (added 2026-10-06): the engine
+  refuses to pin when `MemAvailable` is below the checkpoint (62.11 GiB)
+  plus a **16 GiB floor** — pinned pages cannot be swapped, and under the
+  floor the n-gram lookup pages in from disk on every long prompt. A
+  RAM-short host crash-loops the unit (each boot dies in ~5 s at the pin
+  gate) until the RAM clears; the 2026-10-06 cutover hit this (8 boots,
+  MemAvailable 74.77 GiB vs the 78.11 GiB needed, self-resolved in ~2 min
+  when the dying 27B engine freed RAM). The serve script now checks
+  `MemAvailable` BEFORE `podman run` and WAITS up to `FRANK_LARGE_RAM_WAIT`
+  (default 600 s) for the RAM to clear, so a transient dip boots once
+  instead of crash-looping; if it never clears it refuses loudly. The 16
+  GiB floor is the engine's (not `HALOGEN_HOST_RESERVE_GIB=20` — a
+  separate, higher, overall-budget knob).
+- Engine stdout goes to the log, NOT the journal. The FIRST request after
+  a (re)start can sit silent ~45–60 s on this engine class (health
+  `engine_unresponsive`, 503s) and self-recovers — do not restart over it.
+
+## Verification checklist (run after any redeploy)
+
+1. Boot ledger greps in `/home/dank/logs/frank_large.log`: resident
+   weights (~62 GiB), KV pool line, MTP/drafter line, prompt-cache line,
+   **no clamp named**.
+2. `curl http://192.168.0.143:7770/health` → 200.
+3. Text completion probe: reasoning lands in `reasoning_content`,
+   `timings` honest (generate ≥128 tokens; at 8 the inter_token number is
+   first-token noise).
+4. Vision probe: a 512×512 solid-red PNG through an `image_url` part →
+   "Red" (≥256 px, one word expected).
+5. Spec decode: `draft_n`/`draft_n_accepted` present in timings on
+   real-shaped traffic (SPEC_ADAPT=0 means the head is always engaged).
+
+## Client contract (harness)
+
+`~/.omp/agent/models.yml` — provider `llamacpp (dankclaw)` @
+`http://192.168.0.143:7770/v1`: `contextWindow: 200192`,
+`maxTokens: 57344`, `input: [text, image]`,
+`reasoningContentField: "reasoning_content"`, the `high→xhigh` effort
+fold, `extraBody` temperature 1.0 / top_p 0.95. Compaction still rides
+frank-secondary (the `compactionModel` steering is unchanged).
+
+## Rollback
+
+Revert the swap commit and redeploy — the 27B trio is intact on disk
+(payload at `~/models/halogen-qwen3.8-27b/`, fetch script, and the halogen
+0.1.4 image digest still in podman storage). The older ROCmFPX rollback
+door (`build-rocmfpx.sh` pin) is untouched.

serverAppConfigs-docs/LLM/frank-large-halogen27b-dflash2.md

@@ -25,7 +25,7 @@ confirmed by Frank before deploy:
 
 1. **Text-only.** halogen loads the language tower only; README "honest
    limits": *"The vision encoder is ignored; image inputs are not supported."*
-   The mmproj-era vision role moves wholly to frank-small.
+   The mmproj-era vision role moves wholly to frank-secondary.
 2. **Censored base.** The checkpoint is a finetune of the censored
    `Qwen/Qwen3.8-27B` base, built to make its DFlash2 head strong. The
    ROCmFPX AGENT file was an *Uncensored* build; refusals can return.
@@ -54,6 +54,15 @@ bearing numbers:
   set + host baseline on the 121.2 GiB box. The engine sizes the pool itself
   and **prints a boot ledger naming any clamp** — the post-deploy check reads
   that ledger, not a formula.
+- **`HALOGEN_CACHE_MB=16384` (pinned 2026-10-06)** — auto is OFF on this
+  box: the floor is 2 x a full-context entry (36.9 GB at 262k; one entry
+  is ~13.7 GB at our 200,700-slot x ~70 KB/token) and the ledger had
+  35.7 GB free, so the engine refused and every deep agent request
+  re-prefilled its whole 79-81k context — 84.9-179.8 s measured
+  (serve_api lines, 2026-10-06) for ~500-650 new tokens per turn. The
+  README prices the cache at ~20x time-to-first-token on a warm turn;
+  entries evict LRU by bytes; the reserve (HALOGEN_CACHE_RESERVE_MB,
+  default 8192) is re-checked against live MemAvailable on every store.
 - **No server-side sampler:** an absent temperature is GREEDY. The card's
   thinking recipe (temp 1.0 / top_p 0.95) lives in `models.yml` `extraBody`
   — on the wire only. The template defaults `reasoning_effort` to **xhigh**
@@ -74,7 +83,7 @@ bearing numbers:
 |thinking field|`reasoning_content` (same as before)|
 |`reasoning_effort`|same Qwen3.8 template; outside {xhigh,medium,low} rejected — models.yml fold stays; unset defaults to xhigh, `none` disables thinking|
 |tools|server-side tool calls advertised — round-trip MUST be re-probed after deploy|
-|image input|ignored/error — route to frank-small|
+|image input|ignored/error — route to frank-secondary|
 
 ## Deploy mechanics
 
@@ -94,14 +103,18 @@ bearing numbers:
   those links dangle inside the container mount.
 - **Unit:** `scripts/frank-large.service` (root, `User=dank`, relinked into
   `/etc/systemd/system/`), `ExecStart` = serve script, `ExecStop` = `podman
-  stop -t 120 frank-large-halogen`, script `exec podman wait` so
-  `Restart=always` is real. Engine stdout goes to
-  `/home/dank/logs/frank_large.log` (the boot ledger lives there, not in the
-  journal).
+  stop -t 120 frank-large-halogen`, script runs `podman run` FOREGROUND
+  (2026-10-06: the 2026-10-05 `-d` + `exec podman wait` shape captured
+  only the container ID — the engine ledger existed nowhere on disk and
+  the post-deploy grep below read a one-line hex file; foreground run
+  returns the container's exit code so `Restart=always` is real and the
+  ledger lands in the log). Engine stdout goes to
+  `/home/dank/logs/frank_large.log` (the boot ledger lives there, not in
+  the journal).
 - **Cutover:** `deploy_llm_large` in `scripts/deploy-remote.sh` sweeps the
   legacy bare `llama-server --model /home/dank/models/kingjones-*` process and
   the legacy `frank-large` container name by NAME (never `pkill -f
-  llama-server` — that hits frank-small), retires the old quadlet if the 2026-09
+  llama-server` — that hits frank-secondary), retires the old quadlet if the 2026-09
   halogen era's unit lingers, waits port 7770 free, relinks, health-gates.
 - **Bench law:** stop+mask the unit before any GPU experiment on this box
   (AGENTS.md); `/health` answers 503 while the 33.4 GiB payload loads — that
@@ -109,8 +122,15 @@ bearing numbers:
 
 ## Verification (post-deploy checklist)
 
-1. `grep -iE 'clamp|kv|pool|drafter|dflash' /home/dank/logs/frank_large.log`
-   — ledger shows the pool at the asked shape, DFlash2 active, NO clamp line.
+1. `grep -iE 'clamp|kv|pool|drafter|dflash|cache' /home/dank/logs/frank_large.log`
+   — ledger shows the pool at the asked shape, DFlash2 active, the
+   prompt cache line naming the pinned budget, NO clamp line. EXPECT the
+   WARNING "smaller than ONE full-context entry (18.4 GB at
+   kMaxCtx=262144)": it sizes against the NATIVE window, not our
+   200,700 slot — the deepest possible entry (~14 GB at ~70 KB/token)
+   fits the 16 GB pin, and the 262k entry it fears cannot exist (over
+   slot is a 400). Probe-proven 2026-10-06: the warm turn hit with
+   cache_n=16,384.
 2. `curl -s http://192.168.0.143:7770/health` -> 200.
 3. Real completion with `reasoning_effort` + temp/top_p from the wire recipe;
    `reasoning_content` populated; decode rate from the timings.
@@ -120,6 +140,56 @@ bearing numbers:
    incl. a tool call (V1TOOLS).
 6. Prompt > 200,700 tokens returns the 400 naming the limit (contract test).
 
+## Tuning notes (2026-10-06)
+
+Measured on real agent traffic (serve_api lines) the day after cutover:
+
+- **Decode is depth-bound, and the KV path is the reason.** Shallow
+  prompts (< 1k tokens) decode 30-50 t/s at commit 4-6.3/round; 46k runs
+  5.5, 63-67k runs 4.1-6.5, 79-81k runs 3.3-3.8. The round-time slope is
+  ~8 us/token of context: the KV pool reads at ~8 GB/s effective while
+  the weights stream ~180-240 GB/s (the engine's README puts batch-1
+  weight streaming at the 240 GB/s hardware wall). Commit also falls
+  from 4-6 to 2.5-3 accepted/round as the conversation grows.
+- **No supported flag touches decode.** `docs/FLAGS.md`: HALOGEN_W4A4 /
+  W4A4_EXCL are prefill-only ("decode is unaffected either way"); the
+  undocumented kernel levers are declared unsupported and their defaults
+  are the measured winners. 0.1.4 is the latest release (CHANGELOG
+  head; kernels unchanged since 0.1.0). The publisher's tg bench never
+  measured depth — its tg rows run over the ten SHORT eval prompt
+  shapes — so the README's "decode barely degrades at depth" claim was
+  never exercised at 46-80k.
+- **A/B'd the one FA lever with public precedent (2026-10-06 stop-window
+  probe): rejected.** Issue #12's `HALOGEN_FA_TQ=256` (the 64 GB-carve
+  crash workaround) vs the default tile — same 80k filler prompt,
+  128-token essays, the runtime's own `timings`: decode 6.35 vs 6.34 t/s
+  cold and 5.74 vs 5.76 warm (±0.3 % = noise; draft acceptance identical,
+  53/74 and 58/69), while prefill LOST 9.5 % on this topology (447.5 ->
+  404.8 t/s). The depth collapse is not an FA tile artifact — it sits
+  deeper than the knob. The engine binary advertises 40+ undocumented
+  `HALOGEN_*` kernel levers (FA/DN/MATMUL families, readable via
+  `grep -aoE "HALOGEN_[A-Z0-9_]+"` on the binary); `FA_TQ` is the only
+  one with public evidence, and it moved nothing that matters here.
+- **Routing is the applied mitigation (2026-10-06):** the harness's
+  deep-conversation roles (`slow`, `advisor` in `~/.omp/agent/config.yml`)
+  moved to frank-primary (radiance on Dankserver: fp8 KV at
+  7.32 KiB/token = 8.7x less KV to read per round at depth, MTP depth 3,
+  200k window, prefix-cache host+disk tiers). frank-large keeps OpenWebUI
+  and shallow bursts, where it runs 30-50 t/s. frank-primary's
+  depth-decode rate is designed-for but not yet measured on record.
+- **The lever that existed was the prompt cache, and it was OFF** (see
+  the shape bullet). The pin turns the 85-180 s re-prefill per deep
+  turn into a resume; decode itself is expected UNCHANGED by the pin —
+  the cache resumes prefill, it does not touch the round loop.
+  Probe-proven (two synthetic 17.3k turns, 2026-10-06): cold prefill
+  27.0 s (640 t/s, cache_n 0) -> warm turn cache_n 16,384, prompt_n 946
+  (the extension + one 2048-aligned chunk), prefill 1.7 s — 16x on the
+  pair, answer correct warm.
+- **Everything else stays put.** dflash2 over mtp (publisher 31.71 vs
+  26.88 mean; at depth both pay the same KV term and dflash2's higher
+  commit wins), KV_SLOTS=1 (batch XOR speculation), SLOT_CTX=200700
+  (Frank's window), W4A4 shipped (+9 % prefill, decode unharmed).
+
 ## Rollback
 
 Revert the frank-large swap commit and redeploy `dankclaw --service frank-large`:

serverAppConfigs-docs/LLM/frank-large-ik-uncensored.md

@@ -123,3 +123,27 @@ Stop and disable the root unit, trash it, restore the halogen quadlet from
 `~/.AGENTTRASH/20260915-ik-swap/halogen-quadlet` to
 `~/.config/containers/systemd/`, `systemctl --user daemon-reload`, start;
 the .hgn checkpoint and the kingjones STRIX llama.cpp payload remain on disk.
+
+## Hybrid campaign envelope (2026-09-16, 3090 + Strix Halo)
+
+Why this deployment could not be tuned into a production endpoint: decode
+3.8–4.3 tok/s flat across thread counts (T1 ≈ T16); prefill ~192 tok/s at
+1.8k–32k prompts = the **CPU AVX512-VNNI compute ceiling** for CPU experts.
+The decode root cause is structural: ~100 CPU↔GPU backend splits per token
+with the GPU at 5 % SM during decode — launch/sync-bound over the PCIe x4
+dock, not compute- or memory-bound. T16 profiling shows ~90 % of cycles in
+`pause` (ggml_barrier spin): threads wait on the serial critical path.
+Memory placement (mmap/anon/THP/deferred) does not matter — all variants
+landed 3.8–4.6.
+
+- CUDA-graph capture aborts: qwen4exp's hyper-connection broadcasts use
+  per-layer `ggml_repeat_4d`, whose CUDA path is capture-illegal; until ik
+  gains capture-safe repeat/hc ops (mainline PR #28901 direction), graphs
+  stay dead.
+- Mainline llama.cpp itself runs 1.0 tok/s on this model (CPU MoE kernels
+  4× slower than ik's iqk) — do not switch engines for speed.
+- MTP/speculative decoding is impossible for mradermacher i1 GGUFs: the
+  NextN head is stripped (no `blk.48` tensors).
+- Tuning lever ranked first: `-ncmoe 42` (6 layers' experts on the 3090) →
+  decode 4.54, small-prompt prefill 32 — but VRAM 22/24.6 GiB at 262k ctx,
+  an operator OOM-margin decision, and it needs `GGML_CUDA_NO_PINNED_WEIGHTS=1`.

serverAppConfigs-docs/LLM/frank-large-rocmfpx-q6agent-uncensored-mtp.md

@@ -148,7 +148,7 @@ in [[frank-large-rocmfpx-q8agent-mtp]] and still applies.
 
 Sweep key for scratch servers is now the payload name:
 `pkill -f 'Qwen3.8-27B-Uncensored-Q6_0_ROCMFPX_AGENT'` — never `pkill -f
-llama-server` (frank-small is also one).
+llama-server` (frank-secondary is also one).
 
 ## Operations
 

serverAppConfigs-docs/LLM/frank-large-rocmfpx-q8agent-mtp.md

@@ -162,7 +162,7 @@ Three traps, all hit for real:
    enough for the vision probe).
 2. **`MTP boundary missing for seq_id=… ; resyncing after a non-token step`**
    appears in the log under draft resets. Self-recovering; not an outage signal.
-3. **Never `pkill -f llama-server` on Dankclaw** — frank-small is also a
+3. **Never `pkill -f llama-server` on Dankclaw** — frank-secondary is also a
    `llama-server`. The sweep key here is the payload name:
    `pkill -f 'Qwen3.8-27B-Q8_0_ROCMFPX_AGENT'`.
 

serverAppConfigs-docs/LLM/frank-large-vllm-mimo-v26-bf16.md

@@ -35,7 +35,7 @@ timestamp: 2026-10-01
   (`scripts/frank-large.service`, `User=dank`, `Restart=always`,
   `ExecStop=podman stop -t 120`, `TimeoutStopSec=180` > the stop grace).
 - **Model:** `XiaomiMiMo/MiMo-V2.6-Distill-Qwen-9B` in **full BF16**
-  (17.53 GiB, 4 shards) — the same checkpoint frank-small serves as an INT8
+  (17.53 GiB, 4 shards) — the same checkpoint frank-secondary serves as an INT8
   W8A16 AutoRound conversion on the 3090, here unquantized.
 - **Draft:** `Qwen/Qwen3.5-9B` (18 GiB on disk; only the 15 `mtp.*` tensors
   are loaded into the draft pass).
@@ -82,7 +82,7 @@ Qwen3.5-9B that preserves the hidden-state geometry — identical
 per-position acceptance is the direct measurement that the distillation
 preserved more than geometry. MiMo itself ships **ZERO `mtp.*` tensors**
 (verified in both the BF16 base and the INT8 conversion) — which is exactly
-why frank-small's SGLang NEXTN draft is a 0.00-acceptance ghost: SGLang
+why frank-secondary's SGLang NEXTN draft is a 0.00-acceptance ghost: SGLang
 reads the head from the TARGET checkpoint, vLLM reads the draft from a
 separate path. That difference is the whole trick:
 
@@ -183,4 +183,4 @@ unit/env/serve trio and redeploying returns the 27B endpoint.
 - Engine image: https://hub.docker.com/r/kyuz0/vllm-therock-gfx1151
 - Serve script: `scripts/serve-frank-large.sh`; tunables: `scripts/frank-large.env`
 - Predecessor: [[frank-large-gufo-qwen38-27b]]; sibling on the same arch:
-  [[frank-small-sglang-mimo-v26]]
+  [[frank-secondary-sglang-mimo-v26]]

serverAppConfigs-docs/LLM/frank-medium-batch-ubatch.md

@@ -134,6 +134,59 @@ slots of 131072, `6.44 / 6.90 GiB` free — matching the pre-deploy measurement
 0.01 GiB — a tool call and two concurrent requests answered, draft acceptance on
 both slots (0.63 / 0.48 on slot 1, 0.39 on slot 0), and no ROCm errors.
 
+## The split: re-measure it at the deployed shape
+
+The batch and the tensor split are **coupled**: compute buffers are not
+symmetric across the two cards, so any batch change moves the balance point.
+Order of work — sweep batch with llama-bench (above), THEN re-measure the
+split at the DEPLOYED shape (262144, two slots, drafter loaded) by loading
+the real service with a per-config env override and reading `rocm-smi`. A
+llama-bench number cannot decide the split: it has no `--parallel`, sizes
+its own context as `n_prompt + n_gen + n_depth`, and cannot load the drafter.
+
+The settle gate is mandatory (above); wait for BOTH cards under ~1 GiB AND
+the port to be free (`ss -ltn | grep :7770` until gone) before the next
+load, or the transient server dies with `couldn't bind HTTP server socket`.
+Free bytes follow `card = weights_share + KV_share + constant + ubatch_term`,
+where the ubatch term costs the drafter-carrying card roughly **1.85×** the
+other (measured 2.41 GiB vs 1.30 GiB going 1536 → 512). Balance by
+measurement only: one `-ts` step is one layer, worth ~0.43 GiB on a DeltaNet
+boundary and ~1.43 GiB on an attention one, so the reachable minimum gap is
+about one DeltaNet layer, never zero. Reference: 36,29 @1536 → 4.71/4.91;
+36,29 @512 → 6.01/7.32; **35,30 @512 → 6.44/6.90 (deployed)**.
+
+Re-fit both ubatch coefficients in `tests/frank_medium_unit.bats` at the new
+deployed point and re-anchor the per-card constants; check neighbouring
+splits still FAIL the balance gate or the seam has stopped discriminating
+(keep the 1.5 GiB/card floor and the one-DeltaNet-layer slack). Deploy,
+then verify live: flags in `/proc/<pid>/cmdline`, `/slots`, VRAM matching
+the pre-deploy measurement, a tool call, two concurrent requests,
+`draft acceptance` lines on BOTH slots, 0 ROCm errors.
+
+**Per-card utilization (who is hot, and the split lever)** — a different
+question from batch/prefill. Measure decode tok/s AND per-card
+`rocm-smi --showuse` at 2–3 neighbouring splits in ONE session. Pitfalls
+(each cost a re-run): run the transient unit as ROOT (a `pkill -9
+llama-server` as dank auto-restarts the service and `systemctl` as dank
+fails "Access denied"); warm up 3 throwaway requests before the timed one
+(first 384-token decode after load measured 1.77 tok/s against a true ~32);
+`rocm-smi --showmeminfo vram --csv` columns are `$2`=Total, `$3`=**Used** —
+parsing the wrong column spins the settle gate the full 600 s; the util% is
+sampled post-hoc (which card finished last), so treat it as "which card is
+the pipeline tail", not a duty cycle.
+
+**What the split does and does not buy (measured 2026-09-17, 393216 /
+MTP n=3):** the split is NOT a decode-throughput lever — 34:31, 35:30, 36:29
+all sit in the same ~32 tok/s band. The x1 card runs hot (~91–97 %) at
+35:30 because the MTP drafter + output layers + heavier compute buffers are
+pinned to the card holding the LAST layers — not because PCIe is the
+bottleneck (~5.2 MB of boundary activation crosses per 512-token ubatch).
+36:29 makes the x16 card the utilized worker (42 %/3 %) at no throughput
+cost but spends headroom (0.45 GiB free, under the floor) — Frank's standing
+preference is that the card with the FASTEST PCIe lane should be the more
+utilized one; deploying it requires re-fitting the seam's slack/floor and
+re-checking that 35:30 and 37:28 still fail so the gate keeps discriminating.
+
 ## Limitations
 
 - **`llama-bench` cannot run the drafter, the slots or the deployed context.**
@@ -158,7 +211,7 @@ both slots (0.63 / 0.48 on slot 1, 0.39 on slot 0), and no ROCm errors.
 
 - [[frank-medium-llamacpp-dflash2]] — the deployment this tuned, and its split
   history
-- [[frank-small-benchmark]] — the same sweep on a CUDA 3090, where a 16k prompt
+- [[frank-secondary-benchmark]] — the same sweep on a CUDA 3090, where a 16k prompt
   at depth 0 made prefill climb all the way to ubatch 2048. Depth is the
   difference: with no KV to attend over, per-pass overhead dominates and a
   bigger micro-batch wins; at 64k of resident KV it does not.

serverAppConfigs-docs/LLM/frank-medium-llamacpp-dflash2.md

@@ -312,7 +312,7 @@ came up healthy and *text-only*: image requests answered 500 "image input is
 not supported — hint: … provide the mmproj" and the load log had no mmproj
 line, so `llama-server` fb27a525 does not pick up `mmproj*.gguf` from the
 model directory. The serve script now passes `--mmproj` explicitly (the
-frank-small pattern), refuses to start without the file when vision is on, and
+frank-secondary pattern), refuses to start without the file when vision is on, and
 the deploy handler's preflight checks for it. Verified end-to-end after the
 fix: a generated 64x32 test image was described from its pixels — with
 thinking on the model reasoned about "the three horizontal bars" (its reading
@@ -724,7 +724,7 @@ fleet.
   2026-09-15, whose `--split-mode layer` measurements this service reuses
 - [[Dual-R9700-Cooperation]] — why the two cards cannot take part in a
   collective, and every software escape that is closed
-- [[frank-small-qwen38-gsq-rco]] — the fleet's other llama.cpp endpoint (RTX
+- [[frank-secondary-qwen38-gsq-rco]] — the fleet's other llama.cpp endpoint (RTX
   3090, MTP, the image path)
 - [[Dankserver System Specs]] — hardware inventory
 - [[frank-medium-batch-ubatch]] — the batch/ubatch sweep that set the deployed

serverAppConfigs-docs/LLM/frank-medium-rocmfpx.md

@@ -144,7 +144,7 @@ ggml/src/ggml-backend.cpp:119: GGML_ASSERT(buffer) failed
 
 First deploy (2026-09-15) also ran `retire_root_llm_small()` in
 `scripts/deploy-remote.sh`: the CUDA-era `frank-small` unit was stopped,
-disabled and moved to `~/.AGENTTRASH/20260915-frank-small-retire/host-units`
+disabled and moved to `~/.AGENTTRASH/20260915-frank-secondary-retire/host-units`
 (the host's `systemctl disable` has been observed to delete unit files,
 2026-08-26 incident, so both outcomes are tolerated).
 

serverAppConfigs-docs/LLM/frank-npu.md

@@ -55,9 +55,9 @@ image path is halogen's vision tower on `frank-large`.
 | consumer | setting | now |
 |---|---|---|
 | Hermes | `auxiliary.vision` | `dankclaw-large` / `frank-large` (halogen vision tower) |
-| Hermes | `auxiliary.title_generation` | `dankserver-small` / `frank-small` |
+| Hermes | `auxiliary.title_generation` | `dankserver-small` / `frank-secondary` |
 | harness (`~/.omp/agent/models.yml`) | `modelRoles.vision` | `llamacpp (dankclaw)` / `frank-large` |
-| harness | `modelRoles.tiny` | `llamacpp (dankserver)` / `frank-small` |
+| harness | `modelRoles.tiny` | `llamacpp (dankserver)` / `frank-secondary` |
 | harness | `modelRoles.task` | `llamacpp (dankclaw)` / `frank-large:medium` |
 
 The `flm (dankclaw-npu)` provider block was removed from the harness model

serverAppConfigs-docs/LLM/frank-primary-freetoken-pr587.md

@@ -0,0 +1,174 @@
+# frank-primary — FreeToken PR #587 ROCm (Qwen3.8-Flash-Next-Uncensored-FP8, 1× R9700)
+
+**Status: RETIRED 2026-10-05 → (superseded the same day by the radiance
+cutover — see [[frank-primary-radiance-qwen38-native-fp8]]; kept as the
+rollback runbook: the image + orcarouter payload stay on disk until the
+window closes)** · **Host:** Dankserver (192.168.0.235) · **Port:** 7770 ·
+**GPU:** 1× Radeon AI PRO R9700 (gfx1201) — the second R9700 sits idle
+
+The tcclaviger RDNA4 vLLM fork was replaced by **FreeToken at PR #587 head
+`77a7201f98f00d93f618c6cf8b81ffe8360e0899`** (branch `rocm714_torch214`), built
+on the host into the local image
+`localhost/frank-primary-freetoken:pr587-77a7201` by
+`scripts/docker/frank-primary-freetoken.Dockerfile`. Same port, same served
+alias (`frank-primary`), same 131-shard payload
+(`/home/dank/models/Qwen3.8-Flash-Next-Uncensored-FP8`). Rollback:
+git tag `frank-primary-vllm-shapeG-20261003` (the vLLM trio + its env lore),
+runbook [[dankserver-qwen38-fp8-expert-offload-tuning]].
+
+## Why FreeToken
+
+The vLLM fork planned a per-boot VRAM/host tiering from an estimator: util
+0.982 was a hard law (1.00 was **unbootable** — worker init raises
+`ValueError: Free memory on device cuda:N (31.29/31.86 GiB)` when budget ==
+total), the expert arena capped at 17.77 GiB / 7,762 slots, and live traffic
+still degraded ~40k tokens in (the GDN state-fault class, never fixed in the
+fork). FreeToken keeps **all expert weights in pinned host RAM** with an LRU
+GPU cache, sizes **KV as an explicit token pool**, and boots in one
+deterministic pass — no per-boot estimator to fight.
+
+## The build (on dankserver, rootless podman as dank)
+
+```bash
+cd /home/dank/serverAppConfigs
+podman build -t frank-primary-freetoken:pr587-77a7201 \
+  -f scripts/docker/frank-primary-freetoken.Dockerfile scripts/docker/
+```
+
+Laws baked into the Dockerfile (each one cost a failure):
+
+- **Base digest-pinned**: `rocm/pytorch:rocm7.14_ubuntu24.04_py3.12_pytorch_release_2.11.0@sha256:a223aee1…` —
+  the PR doc's known-good stack (torch 2.11.0+rocm7.14, triton 3.7). No
+  rocm7.14+torch2.14 image exists on Docker Hub (the 2.14 images are
+  rocm10.1.0 — an untested pair for this PR).
+- **`--no-build-isolation --no-deps -e .`** — a plain
+  `pip install -e .` on ROCm can REPLACE the image's ROCm torch with a CUDA
+  build (the dependency set is authored for NVIDIA). Runtime deps are
+  curated explicitly; `flashlib==0.3.0 --no-deps` (its numba /
+  nvidia-cutlass-dsl deps are CUDA-only); ninja for the QSA JIT;
+  **flashinfer / sgl_kernel intentionally absent** (no ROCm builds;
+  FreeToken falls back to its Triton kernels — the PR's 12 pytest failures
+  are exactly those backends).
+- **`libhiprtc.so` dev symlink**: the pip-ROCm torch ships only the
+  versioned `libhiprtc.so.7` in
+  `/opt/venv/.../site-packages/_rocm_sdk_core/lib`, but the PR's `_cpu_moe`
+  extension links `-lhiprtc`. First build died `ld: cannot find -lhiprtc`;
+  the Dockerfile symlinks the unversioned name into the ALREADY-LINKED dir.
+- FreeToken source is checked out at the **pinned PR head**, not the branch
+  (a future push to `rocm714_torch214` cannot change the deployed engine).
+- The stale Docker Hub login (`docker login` creds at
+  `~/.docker/config.json`) poisons podman's ANONYMOUS pulls of public images
+  with "invalid username/password" — `docker logout` fixes the pull. Hit
+  2026-10-05 while pulling the base image.
+
+## Measured at cutover (2026-10-05/06)
+
+
+- **Boot**: ~2.3 min container-start → API-ready (131 shards page-in + PLE
+  pin + expert staging + CUDA graphs `[1,2]`). VRAM after graphs:
+  **1.80 GiB free** of 31.86. RAM: 122 GB used + **113 GB shared (the
+  pinned expert/PLE slabs)**, 123 GB still available.
+- **Decode: 17.1 tok/s** single-stream, temp 0, 400-token probe (clean
+  1→N counting). The vLLM incumbent measured ~30 tok/s on TWO cards;
+  FreeToken single-card is the deployed baseline.
+- **The single-card LRU ceiling**: the KV contract (harness worst case
+  177,356 tokens) + dense fp8 + the vision tower + graphs leave ~12 GiB for
+  the expert LRU at rate 0.10 — and only 1.8 GiB of live headroom, so
+  `ft ctl cache --moe` cannot raise it far on one card. The paths to a
+  better decode rate are TP=2 (below) or a smaller pool (contract
+  violation — the 177,356 floor is real traffic, not padding).
+- **TP=2 exists but does not boot on ROCm** (probed 2026-10-06,
+  zero-impact window): `--tensor-parallel-size 2 --gpu 0,1` dies in the TP
+  allreduce JIT — `pynccl.cu` compiles per-rank at boot and clang can't
+  resolve `cuda_runtime.h` from the hipified include path
+  (`1 error generated when compiling for gfx1201`, ninja stopped). The PR
+  author validated single-GPU only; this is FreeToken's untested-TP-on-ROCm
+  edge, not a board or driver fault (radiance TP=2 allreduces fine on this
+  pair since the 2026-09-20 swap). A future fix = patch the JIT include
+  path (HIP's cuda_runtime shim) or vendor an ROCm allreduce — re-probe
+  then.
+- **Verification gates passed**: /health ok; plain completion `FLEET-OK`
+  (finish stop); thinking split — `reasoning_content` non-empty AND
+  `content` carries the answer (17×24 → "408"); native `tool_calls` with
+  correct name+arguments (finish tool_calls); 256×256 solid-red PNG through
+  an `image_url` part answers exactly "Red" (prompt_tokens 127); stats:
+  native ctx 262,144, KV 4,687 pages × 64 = 299,968, modalities
+  [text,image], sampling defaults from generation_config.json
+  (temp 1.0 / top_p 0.95 / top_k 20).
+- **Thinking counts against max_tokens** (same as the vLLM era): a
+  `max_tokens: 10` probe returns EMPTY content with the budget spent in the
+  think block. Clients must budget for thinking or disable it.
+- **Prometheus scrapes /metrics → 404** (Dankclaw scrapes 192.168.0.235:7770
+  from the llm-fleet job; FreeToken has no /metrics endpoint). The target
+  will read down until the scrape config is repointed or a metrics shim
+  lands — same class as the retired HTML-probe targets
+  (see [[Observability]]).
+
+
+## The shape
+
+| Knob | Value | Why |
+|---|---|---|
+| `--gpu` / `HIP_VISIBLE_DEVICES` | 0 / 0 | ONE R9700; visibility pinned so in-container index 0 is the right card |
+| `--num-tokens` | **299,968** | KV pool in TOKENS; **must be a multiple of 64** (the auto attention backend is `qsa_sparse`, which overrides page size to 64 — 300,000 was refused: `nearest valid values: 299968 or 300032`) |
+| `--max-running-requests` | 2 | first-boot conservative; raise after real-traffic slot-occupancy look |
+| `--max-output-tokens` | 36,864 | harness maxTokens; requests that omit max_tokens get this default |
+| `--moe-cache-rate` | 0.10 | GPU expert LRU as fraction of all experts (~120 GiB fp8 → ~12 GiB VRAM); first-boot conservative |
+| parsers | AUTO | thinking → `reasoning_content` natively; tools in the qwen3.8 dialect; NO `--reasoning-parser`/`--tool-call-parser` flags |
+
+KV math: 12 full-attention layers × 2 KV heads × 256 head_dim × 2 (K+V) ×
+2 B = **24 KiB/token** (f16 KV; the 36 GDN layers use fixed per-seq state,
+not per-token KV) → 299,968 × 24 KiB ≈ 7.0 GiB. Clears the harness worst
+case (140,492-token prompt + 36,864 max_tokens = 177,356) and admits ONE
+native 262,144-token conversation + full budget. `models.yml`
+`contextWindow: 200704` stays (compacts early, never 400s).
+
+## RAM math (why the root unit + memlock)
+
+246 GiB total. FreeToken pins ~120 GiB of fp8 expert rows + the ~47.7 GiB
+PLE ngram table in host RAM → the unit carries
+`LimitMEMLOCK=infinity` (root unit; dank's login session caps at 8 MiB and
+a user quadlet cannot raise it). Weights+PLE+overhead ≈ 190 GiB — the vLLM
+container's ~170 GiB MUST be down first; `systemctl stop frank-primary`
+frees it (measured: 207 GB free right after the stop).
+
+## Resize live
+
+```bash
+ft ctl cache --kv <tokens>   # grows the KV pool LIVE (shrink needs restart)
+ft ctl cache --moe <rate>    # same for the expert LRU
+ft ctl stats                 # pool occupancy, VRAM, throughput
+```
+
+## Verification (the fleet's real-completion law)
+
+1. `/health` 200 — necessary, not sufficient.
+2. Plain completion returns the asked-for string.
+3. Thinking request: `reasoning_content` non-empty AND `content` carries the
+   answer (the models.yml `reasoningContentField: reasoning_content` flip
+   rides on this).
+4. Vision probe: a solid-color PNG through an `image_url` part answers the
+   exact color (the legacy top-level `images` field trap does not exist
+   here — FreeToken is OpenAI-native).
+5. Unit-managed: `systemctl show -p MainPID` behind the health answer (the
+   2026-09-26 stray-port false green).
+
+## Rollback
+
+```bash
+git checkout frank-primary-vllm-shapeG-20261003 -- \
+  scripts/frank-primary.env scripts/serve-frank-primary.sh \
+  scripts/frank-primary.service
+./tests/run.sh && git commit -m "frank-primary: rollback to vLLM shape G" \
+  && git push && ./scripts/deploy.sh dankserver --service frank-primary
+```
+
+The tcclaviger image and the FreeToken image coexist in dank's rootless
+storage; the deploy re-pulls the pinned digest if absent.
+
+## Index
+
+- [[dankserver-qwen38-fp8-expert-offload-tuning]] — the retired vLLM shape
+  (rollback target; its 362-line env lore lives in the tag)
+- [[freetoken-frank-small]] — the v0.1.2 CUDA precedent (ortobox, RTX 3090)
+- [[LLM/index]] — fleet index

serverAppConfigs-docs/LLM/frank-primary-glm53-flash-w4a16.md

@@ -45,6 +45,98 @@ timestamp: 2026-09-30
 > because the fork compiles its own Qwen trunk and not its GLM one. See
 > [The measured profile](#the-measured-profile-and-the-real-wall).
 
+## DFlash2 target-side port — the four laws and the KV ladder
+
+Distilled 2026-10-05 from the probe windows (skills `glm53-dflash-boot-ladder`
+/ `glm53-dflash-eagle3-port`); each law was proven by a real failed boot.
+
+**The four EAGLE3-port laws** (make the GLM trunk satisfy `SupportsEagle3`):
+
+1. **Patch the instance the runner calls.** `model_runner.py:425` calls
+   `set_eagle3_aux_hidden_state_layers(self.model)` on the *loaded* model —
+   GLM loads as `Glm5NextForConditionalGeneration` (multimodal wrapper;
+   `self.language_model = Glm5NextForCausalLM`). Members placed only on the
+   CausalLM fail the check on the conditional — same error text, wrong file.
+   Patch BOTH wrappers.
+2. **Presence-checked member set.** `SupportsEagle3` (interfaces.py:1659) is
+   `@runtime_checkable`; `isinstance` walks ALL `__protocol_attrs__`
+   (including base-protocol members) with `getattr_static`, breaking on the
+   first miss. Both wrappers need, as class attributes:
+   `supports_eagle3: ClassVar[Literal[True]] = True`,
+   `has_own_lm_head: bool = False`, `has_own_embed_tokens: bool = False`.
+   Verify in-image with stock typing — no boot needed.
+3. **MRO forbids inheriting the Protocol.** GLM wrappers carry four/five
+   Protocol-derived bases; adding `SupportsEagle3` anywhere dies with
+   "Cannot create a consistent method resolution order" (sim-proven: the
+   ordering rejects it in every base position). The attribute route avoids
+   the MRO, so the two methods must be written by hand: the CausalLM
+   unwraps to `self.model` (the `EagleModelMixin` trunk), the conditional
+   delegates to `self.language_model` (pixtral.py:498 pattern);
+   `get_eagle3_default_aux_hidden_state_layers` returns `(2, L//2, L-3)`,
+   with the drafter config's `target_layer_ids` winning when present.
+4. **Trunk capture point:** append `aux_hidden_states` BEFORE the layer call
+   — GLM's loop carries the single-stream state entering the layer.
+
+**The KV blocker ladder** (patches `scripts/patches/glm53-dflash-0*`):
+
+1. EAGLE3 gate → presence-checked attrs on both wrappers (above).
+2. KV page-size unifier: MLA layers cannot be padded (sparse MLA reads
+   whole token rows) but CAN grow block_size when the max page is an exact
+   multiple — bump `max_page_size` in `unify_kv_cache_spec_page_size` to
+   the `lcm` of all MLA-family pages first (patch 02).
+3. kpool tail assert `tail_kv_cache.shape[2] == pool_size`: the unifier's
+   divisible branch grew KpoolTailSpec's block; its kernel pins block ==
+   index_kpool. Add a KpoolTailSpec branch that pads `page_size_padded`
+   instead of growing (patch 03).
+4. KV capacity: the glm5 layout ALIASES mamba states INTO MLA pages (why
+   GLM fits 32 GiB cards); the generic path (taken when draft/hidden specs
+   exist) cannot alias → ~30 GiB demanded vs a 0.92 GiB reserve. Extend
+   `_get_kv_cache_groups_glm5_next` to partition
+   HiddenStateCacheSpec + drafter attention specs; the drafter forms its
+   own uniform group (per-group pages, NO unification → the MLA block
+   stays at the engine-auto-set 2,432 from interface.py:918, kpool guard
+   `block_size % 512 == 0` unstressed); layout returns a 10-tuple
+   (+draft_group, +hidden_groups); the builder sizes ONE shared backing
+   allocation (`allocate_kv_cache` asserts all KVCacheTensor sizes equal) =
+   aliased pool + appended draft/hidden tensors (patch 04). After 04 the
+   capacity requirement collapsed 31.76 → 1.88 GiB at 65,536 max len and
+   the engine SERVED (READY ~500 s, dflash2 fused prep ON, drafter FULL
+   graphs on both TP ranks).
+5. Reserve registry: `_glm53_kv_bytes_per_token` must add 4,096 (drafter
+   KV) + 5×hidden×2 (aux hidden cache) keyed on
+   `speculative_config.method in {dflash, dspark, extract_hidden_states}`.
+   NEVER key on `use_aux_hidden_state_outputs` — it lives on the
+   gpu_model_runner instance (gpu_model_runner.py:623/678), unset at plan
+   time, so the check silently fails and the reserve stays 13,600
+   (patch 04b). Proven reserve math at 65,536: 13,600 + 4,096 +
+   5×4096×2 ≈ 58,656 B/token → ~4.15 GiB reserve, arena ~10.3 GiB.
+
+**Verify cheaply before booting:** smoke-test the pure KV logic on CPU
+torch with the image's real spec classes — stub `vllm.logger`/`vllm.utils.*`,
+register the spec classes in `KVCacheSpecRegistry` with a dummy manager, add
+`from __future__ import annotations`, and run synthetic GLM+draft+hidden
+spec sets through the patched grouping/layout/builder functions. A 7-minute
+boot per hypothesis is the alternative.
+
+**Window discipline:** stop + mask `frank-primary` FIRST and gate on
+MemAvailable ≥ 230 GiB before booting the probe (the expert-offload planner
+reads MemAvailable; beside-prod boots die at plan time with `ExpertPlanError:
+host holds 5113 of 12096 swappable experts`). Save `podman logs glm-probe`
+to a file BEFORE teardown — a `podman logs | grep | head` pipeline died once
+under pipefail via SIGPIPE before log-save, losing the capture. The abort
+path must restore prod (unmask + daemon-reload + reset-failed + start),
+never bare-exit after stop+mask. Launcher knobs: `DFLASH_PATCH=1` (mounts
+the patched model.py), `DRAFT_MOUNT=1`, `SPECJSON_FILE=…` ({"method":
+"dflash","model":"/draft","num_speculative_tokens":8}) — note the launcher's
+echo prints `patch=$PATCH_MOUNT` (the COMPILE knob), not DFLASH_PATCH.
+Boot timeline: EAGLE3 gate fires ~4 min in at worker init; drafter
+torch.compile is the frontier (~380 s).
+
+**Open telemetry discrepancy:** the contaminated window read 1.5–2.1 tok/s
+with acceptance 1.00–1.25; Frank live-measured ~18 tok/s decode / ~370 tok/s
+prefill. Reconcile with a clean single-request measurement before trusting
+either.
+
 Related: [[dankserver-qwen38-fp8-expert-offload-tuning]] (the same fork, the same
 hardware — its conclusion was drawn from throughput arithmetic and is superseded
 by this page's kernel-time measurement),

serverAppConfigs-docs/LLM/frank-primary-llamacpp-qwen38-flash-next-q8.md

@@ -110,6 +110,32 @@ retires the lucebox user quadlet to dated home trash
 truly free, relinks + enables + restarts the root unit, and waits up to
 1800 s for `/health` (cold start page-ins ~175 GiB and builds the graphs).
 
+### Preflight the GGUF without downloading it
+
+GGUF metadata lives in the header and HF honours ranged GETs, so a few MB
+answers every sizing question before a 13 GB transfer
+(`Range: bytes=0-8000000`). Parse: magic `GGUF`, u32 version, u64
+n_tensors, u64 n_kv, then key/value pairs (string = u64 len + bytes; ARRAY
+= u32 elem type + u64 count; BOOL is 1 byte — the type hand-written parsers
+forget). Walk arrays by byte size even for a summary: a naive parser
+desynchronises on `tokenizer.ggml.tokens` and lands gigabytes off. Read:
+`general.architecture`, `<arch>.block_count`, `<arch>.context_length`,
+`<arch>.nextn_predict_layers` (MTP is a property of the FILE — the `-mtp`
+build carries `blk.<n>.nextn.*`; `--spec-type draft-mtp` then loads the
+head against the MAIN model's weights, and `--model-draft` would load the
+file twice), `<arch>.head_count_kv`, `<arch>.key_length`/`value_length`,
+`<arch>.full_attention_interval`, `<arch>.ssm.state_size`/`group_count`/
+`time_step_rank`, and for the projector `clip.projector_type` +
+`clip.has_vision_encoder`.
+
+KV arithmetic for the hybrid (16 × (3 Gated DeltaNet + 1 Gated Attention)):
+only full-attention layers take KV
+(`attn_layers × n_head_kv × head_dim × 2 × bytes_per_elem`; f16 = 2 B,
+q8_0 = 1.0625 B, q4_0 = 0.5625 B); DeltaNet state is a fixed per-slot
+buffer (~0.16 GiB/sequence at 48 layers/128 d_state/48 v-heads f32); the
+MTP draft context costs ≈ 4 KiB/token. `--parallel N` DIVIDES `--ctx-size`
+between slots — it does not multiply.
+
 ## Verification (2026-09-26 deploy)
 
 - `/v1/models` reports `frank-primary`.
@@ -228,7 +254,7 @@ refetch.
 
 - [[frank-large-unsloth-qwen38-27b-q8xl]] — Dankclaw :7770, the same
   b11030-mix binary family on gfx1151.
-- [[frank-small-gemma4-12b]] — Dankclaw :7771, mainline llama.cpp on the
+- [[frank-secondary-gemma4-12b]] — Dankclaw :7771, mainline llama.cpp on the
   3090; the only audio-capable endpoint.
 - The harness contract lives in `/home/blank/.omp/agent/models.yml`
   (`llamacpp (dankserver)` provider, `contextWindow 262144`,

serverAppConfigs-docs/LLM/frank-primary-lucebox-ds4.md

@@ -146,3 +146,70 @@ systemctl --user start frank-medium`) and stop/mask frank-primary. The
 _Filled in after the cutover: health, real completion, tool call, Open WebUI
 completion, measured prefill/decode, `systemctl --user is-enabled/is-active`
 state._
+
+## Dual-R9700 follow-up (2026-09-28 probe era)
+
+After this single-card runbook was retired, the DS41 engine got a dual-R9700
+attempt on the `feat/ds41-engine` branch (`main` rejects the checkpoint with
+`unsupported model architecture 'deepseek41'`). Findings that outlive the
+probe:
+
+**Budget both cards explicitly, or you waste half the box.** The engine
+sizes the streamed-expert cache LAST, after the model, so uncapped it absorbs
+every spare byte on the stream device — symptom: card1 at 33.0/34.2 GB
+looking busy while card0 sits at 14.2/34.2 GB; the busy card is holding a
+cache of things still streaming from NVMe. Pin all four:
+
+| env | card | meaning |
+|---|---|---|
+| `LUCE_EXPERT_BUDGET_MB` | hip:0 | primary hot experts (~11.96 MiB each) |
+| `LUCE_EXPERT_SECONDARY_BUDGET_MB` | hip:1 | secondary experts (needs `LUCE_DS4_MOE_TP*`) |
+| `LUCE_EXPERT_STREAM_CACHE_MB` | hip:1 | cap the auto-sized stream cache |
+| `LUCE_EXPERT_STREAM_DEVICE=1` | hip:1 | which card holds the slots and computes them |
+
+Working full-card shape: 21000 + 20000 + 8000 → card0 ~28.6/31.86 GiB,
+card1 ~27.3/31.86 GiB, resident experts 3,360 of 15,360.
+
+**Expert TP DOES work on gfx1201+gfx1201.** A retired quadlet recorded
+`LUCE_DS4_MOE_TP_INPROC` as "qualified for gfx1151+gfx1201, not
+gfx1201+gfx1201" — that is WRONG; peer access negotiates and the secondary
+tier comes up clean: `[deepseek4-moe-tp] enabled mode=in-process local=hip:0
+secondary=hip:1`. It is the only lever that materially raises resident
+expert count, and resident count is the entire bottleneck. Confirm every run
+with `expert ownership:` and `hybrid experts ready:` lines.
+
+**The placement file must live under the mounted models dir.** JSON
+`{"owners": [...]}`, 15,360 entries indexed `layer*384+expert`, values
+`primary`/`secondary`/`stream`. It is a ranking, not a promise: each device
+keeps what fits its budget and demotes the rest. Anywhere outside
+`/models/...` gives `cannot open expert placement` and a SILENT single-tier
+fallback — a silent-wrongness failure, not a crash.
+
+**V4.1 refuses layer split and layer-major prefill** — only `--ds4-prefill
+exact` (token-by-token) is legal on a 32 GiB card and `--ds4-fused-decode`
+is refused there. With ~22% of experts resident each token pulls ~187 cold
+experts × ~12 MiB; if `/proc/<pid>/io` `read_bytes` climbs into terabytes
+the model is larger than the box's RAM and you are in page-cache thrash at
+the NVMe ceiling — a capacity verdict, not a tuning problem. Low GPU util
+with a pegged link is the EXPECTED signature of PCIe-expert-stream-bound
+decode, not a broken run.
+
+**Harness traps, each of which produced a wrong number:** never `env -i` the
+podman call (rootless podman runs as `dank`; `env -i HOME=/root` makes
+PODMAN try `/root/.config` and die `stat /root/.config: permission denied`
+before the container exists — pass container env with `-e`); never put a
+large prompt on curl's argv (`Argument list too long` at ~128 KB — use
+`--data-binary @file`); size prompts by growing the file until `wc -c`
+reaches the byte target AND assert depth against the engine's own
+`usage.prompt_tokens` (nominal 4 chars/token was really 5.73; a chunk loop
+that appends a fixed ~152-byte line per nominal 200-byte chunk undershot
+30%); the server runs as the container's PID 1, so tear down with
+`podman rm -f`, never host `pkill`; drop page cache between arms
+(`sync; echo 3 > /proc/sys/vm/drop_caches`) or the first arm reads warm and
+every later one is a lie.
+
+Read the placement log — it is the whole diagnosis
+(`arch=deepseek41 layers=40 embd=5120 experts=384 used=6 ff=2304
+hash_layers=0` — hash_layers=0 is correct for V4.1; `engram: layers=[1,14]
+… 2 table(s) embedded (not mapped)` — the 189.13 GiB Engram tables are never
+mapped, so they do not evict the expert page cache).

serverAppConfigs-docs/LLM/frank-primary-r9v-radiance-qwen38-q8.md

@@ -55,7 +55,7 @@ would drift from the qualification — don't.
 ## Spec decode: MTP4 only — the ngram-mod ask is NOT expressible
 
 llama.cpp chains drafter types (`--spec-type draft-mtp,ngram-mod` on
-frank-small/large). The fork's `speculative_config` is a **single dict**
+frank-secondary/large). The fork's `speculative_config` is a **single dict**
 (one method per engine): `mtp`, `ngram`, `ngram_gpu`, `dflash`, … are
 mutually exclusive — no chaining seam exists in the fork. What IS active:
 
@@ -190,6 +190,70 @@ repo env sets `R9V_ENABLE_PREFIX_CACHING=1` (commit `41e9189`), verified
 on :7770 (`--enable-prefix-caching` in the container argv, health 200,
 live completion answering).
 
+## Authoring a new profile (the qualification pipeline)
+
+Order matters at every step:
+
+1. **Package manifest** `packages/models/<pkg>/package.json` — target shards
+   + aux artifacts with sha256. Verify every target's byte count AND sha256
+   after fetch (`curl -fL -C -` per shard; never trust a fetch without `-f`).
+2. **Placement bootstrap** from GGUF headers: parse all shards' tensor
+   tables (a KV-only shard1 is normal for split parts; each part carries its
+   own tensor table). Per-expert full per layer =
+   (`ffn_gate_exps` + `ffn_up_exps` + `ffn_down_exps`)/512; rank slices
+   rank0 = full×416//640, rank1 = full−rank0. Hot counts =
+   floor(measured_budget / per-expert total). **Anchor budgets to the
+   MEASURED hot_bytes of the qualified sibling profile, never invented.**
+3. **Trim + validate**: `tools/trim_experts.py --hot-counts N,M`, then
+   `expert_budget.validate_manifest(m)` +
+   `expert_budget.expert_memory(m, slots, {cache_rank}, runtime=<runtime.json>)`
+   (runtime/catalog are keyword-only). Declared hot_bytes must match
+   computed or it raises.
+4. **PLE derive** (containerized `tools/prepare_ple.py` with
+   `--expected-bytes/--expected-type/--expected-shape`); setup skips
+   re-derivation when the file exists with matching size.
+5. **profile.env**: set values BEFORE `source dual-r9700/profile.env`
+   (set-first-wins; an empty string does NOT survive `:=`).
+
+The canonical bootstrap.json wrapper schema is exactly
+`{schema: "r9v.placement.v1", id, model_package, hardware, artifact_path,
+artifact_sha256, scope}` with `hardware: "dual-r9700-32gb-asymmetric"`; the
+wrapper `id` MUST equal the id in the profile.json placement descriptor
+(setup rejects mismatches: "does not match profile placement"). Never
+fabricate qualification metadata — use
+`scope: "Unranked routing bootstrap only; does not certify fit or headroom"`.
+
+GGUF parsing laws for the bootstrap: expert tensor names are
+`blk.0.ffn_up_exps.weight` — split on "." gives FOUR parts, layer number is
+parts[1] (a 3-part assumption crashes `int("")`); outer-KV value-type sizes
+`{0:1,1:1,2:2,3:2,4:4,5:4,6:4,7:1,8:str,9:arr,10:8,11:8,12:8}`; tensor byte
+extents from consecutive OFFSETS, never a type-size table (mis-buckets tails
+silently); the PLE Q8_0 tensor on this arch is shape (160, 320001536),
+54,400,261,120 B, an entire shard.
+
+Setup/start: `./r9v setup <alias> --state-dir <dir> --model-dir <dir> --image
+sha256:<digest> --local-image --reuse-from <sibling-model-dir>
+--accept-model-license` (aux artifacts reuse byte-identical files; F16 vs
+Q8 mmproj will not reuse). Start requires the SAME `--state-dir`; fixed
+placement via `R9V_EXPERT_MANIFEST_PATH` skips headroom/catalog/seed raises.
+The host-only doctor FAILs when another instance holds the cards (it wants
+3 GiB free per rank; production at ~30 GiB/card is a hard wall) — a second
+instance CANNOT coexist, so qualification requires a stop+mask window for
+the incumbent, then unmask+restart.
+
+**Qualification gate (unqualified → qualified):** first start runs the
+workload battery (PONG / tool_calls / vision / long-ctx), the PLE pinning
+proof (`VmLck` ≈ PLE bytes in `/proc/<container-pid>/status` when
+RESIDENCY_MODE=pinned), and a VRAM headroom snapshot (floor 3.3 GiB). If
+headroom is violated, lower hot counts via re-trim and re-derive
+MAX_EFFECTIVE. Only then record state `ready: true` and author the runbook.
+
+Remote-execution gotchas from the same windows: `nohup X &` inside an ssh
+command races ssh teardown — the script can die before writing a byte; run
+long remote scripts synchronously, or setsid + disown, or via a system
+unit. zsh login shells abort on unquoted `===` separators; grep globs need
+quoting through ssh+zsh layers.
+
 ## Ops
 
 - **Restart**: `sudo systemctl restart frank-primary` (root unit; the serve

serverAppConfigs-docs/LLM/frank-primary-radiance-qwen38-native-fp8.md

@@ -0,0 +1,297 @@
+# frank-primary — Radiance (StillDeadcode), Qwen3.8-Flash-Next-FP8 NATIVE, 2× R9700 TP=2
+
+**Status: ACTIVE 2026-10-05 →** · **Host:** Dankserver (192.168.0.235) · **Port:** 7770 · **GPU:** 2× Radeon AI PRO R9700 (gfx1201), TP=2
+
+The FreeToken PR #587 endpoint was replaced by the **radiance engine** (C++/HIP,
+libr4d kernels for gfx1201, no Python in the serving path) driving the
+digest-pinned image `stilldeadcode/radiance@sha256:4c3f484564bb2872784e4…`
+(v1.0.7 line, 287 MiB, `libr4d@0.5.0`) as a plain `podman run` through
+`scripts/serve-frank-primary.sh` under the **root systemd unit**
+(`frank-primary.service`, `User=dank`, `LimitMEMLOCK=infinity`). Same port,
+same served alias (`frank-primary`), same container identity
+(`frank-primary` = the unit name). Rollback: git tag
+`frank-primary-freetoken-pr587-20261005` (FreeToken trio + Dockerfile), image
+`localhost/frank-primary-freetoken:pr587-77a7201`, payload
+`/home/dank/models/Qwen3.8-Flash-Next-Uncensored-FP8`.
+
+## Why radiance
+
+- **TP=2 is real here.** FreeToken's TP=2 dies in the `pynccl.cu` ROCm JIT
+  (`cuda_runtime.h` unresolvable for gfx1201 — probed 2026-10-06); the
+  second R9700 idled. Radiance's TP wire is libr4d's own one-shot
+  all-reduce (no RCCL), legal on this board since the 2026-09-20 swap
+  (ProArt B850, x8/x8 on CPU root ports, `Routing+`/`ReqEn+` everywhere).
+- **Native FP8 end to end** (Frank's call): the official
+  `Qwen/Qwen3.8-Flash-Next-FP8` checkpoint keeps its modelopt block-FP8
+  encoding — no requantisation, no calibration run, no 4-bit codebook.
+- **Every tier is explicit and measured**, not an estimator: placement
+  `expert_tiered` with a pinned host pool, `--ngram-placement ram` for the
+  47.7 GiB PLE table, `--expert-vs-cache-ratio 0.82` (the publisher's
+  measured 99 %-hit point; 0.87 already collapses to 0 %), fp8 KV.
+
+## The payload: native FP8, reshaped then converted (one recipe rule)
+
+- **Raw checkpoint**: `/home/dank/models/Qwen/Qwen3.8-Flash-Next-FP8`
+  (ungated repo, 172.8 GB, 131 shards + index + tokenizer + vision
+  configs). Fetched by `scripts/download-frank-primary.sh` (vendored
+  `hfget.sh`, curl+jq, no HF token needed) and **verified byte-for-byte**:
+  all 131 shards sha256-match the Hub LFS oids (2026-10-05, log
+  `/home/dank/logs/verify-qwen38-official-fp8.log`).
+- **The container**: `/home/dank/models/rad/qwen4exp-fp8-native.rad`
+  (177,109 MiB measured = 173.0 GiB; floor
+  `FRANK_PRIMARY_RAD_MIN_MIB=177000` in the env gates a
+  truncated convert). Built by `scripts/build-frank-primary-rad.sh`:
+  1. stage `/home/dank/models/stage-qwen4exp-fp8` — `config.json` with
+     **`quantization_config` dropped** + relative symlinks into the raw
+     checkout;
+  2. `stack-qwen38-official-fp8.py` reshapes the staging dir — two
+     transforms, both value-preserving (script docstring carries the full
+     argument): the plugin maps the STACKED expert layout
+     (`mlp.experts.gate_up_proj` `[512,1280,2560]` + `_scale_inv` at the
+     same index) while the official repo ships per-expert unfused tensors,
+     so staging byte-stacks all 49 stacks' (48 trunk + MTP) 512 experts
+     with NO requantisation; and the PLE table ships as 128 **scaleless**
+     E4M3 shards + one bf16 scalar (1.9931793212890625e-04) while the
+     plugin's fp8 branch wants the codes beside a ONE-ELEMENT bf16 scale
+     plane — so staging decodes to a single bf16 `shard_0` with the
+     checkpoint's own scale (e4m3 decode and the scale are both exact in
+     bf16);
+  3. `rad-convert` with ONE `--quant` rule — pure copy for everything else
+     (every weight keeps the checkpoint's block-FP8 planes: `X.weight`
+     E4M3 codes + `X.weight_scale_inv` bf16 `[128×128]` scales) — and the
+     rtn rule re-encodes the bf16 PLE table at the same fixed scale, an
+     exact round-trip (nearest-code distance 2⁻³ vs the bf16 rounding
+     perturbation 2⁻⁹): the container gets the checkpoint's own PLE codes
+     plus the one-element scale plane (`blk.1.ple_ngram.scale`, view {1});
+  4. THREE LAWS the live runs proved (each cost a full verify+declare
+     cycle; all fixed in the two scripts, 2026-10-05/06):
+     - **Scale-name law**: `scale_name()` appends `_scale_inv` to the
+       TENSOR NAME verbatim — a stacked tensor named
+       `…experts.gate_up_proj` (no `.weight` suffix) has its scale plane
+       at `…experts.gate_up_proj_scale_inv`, NOT `…weight_scale_inv`.
+     - **File-shadow law**: the reader opens every index-named file in
+       SORTED order and emplaces EVERY tensor each file physically
+       holds, first-wins (`checkpoint.cpp add_shard`). The raw
+       model-000NN shards carry tensors NAMED exactly what the merged
+       PLE shard is named, so the merged file MUST sort first
+       (`00-ple.merged.safetensors`) or a raw fp8 shard wins the name
+       and the plan reads one shard as the codes.
+     - **Symlink-through law**: staging symlinks the raw index into the
+       stage; `open(path,"w")` writes THROUGH it and corrupts the raw
+       checkpoint (raw verify then fails 143/144). The stack script
+       writes index/config temp+`os.replace` — an atomic swap replaces
+       the symlink itself; the build script COPIES the index and
+       symlinks only the shards.
+  5. `rad-info` summary + size floor.
+- **Why the staging dir is mandatory** (the descriptor law, spec §2.4):
+  the plugin is selected by the exact `(arch_id, quant descriptor)` pair
+  with no fallback. A raw official checkpoint carries
+  `quantization_config.quant_method: "fp8"` → descriptor `"fp8"` → **no
+  shipped plugin claims `(qwen4exp, "fp8")`** — raw serve AND unmodified
+  convert both refuse. The shipped `qwen4exp_fp8` plugin claims
+  `(qwen4exp, "")` (recipe-built containers). `rad_convert.cpp` writes
+  `w.set_quant(meta.quant)` into the header, so the staged config's empty
+  quant method flows through as the container descriptor while **every
+  weight stays FP8 data**. Native precision is untouched; only the
+  selection key differs.
+- `--plan-only` first: the convert's declare phase is the engine's own,
+  so a plan that resolves every op will boot. The convert runs card-less
+  (declare = CPU-side layout queries); add `--device` only if a plugin
+  load ever demands it.
+
+## The shape (`scripts/frank-primary.env`, every value commented there)
+
+| Knob | Value | Why |
+|---|---|---|
+| Image | `stilldeadcode/radiance@sha256:4c3f4845…` | digest-pinned INDEX digest (RepoDigests also carry the per-arch manifest `1c582005…`; the image ID `69d05724…` is the config digest, NOT a manifest digest — pinning it sends `podman run` to the network); entrypoint = engine, tools (`rad-convert`/`rad-info`/`rad-tune`/`rad-kbench`) on PATH |
+| Container | `frank-primary` | identity = service name across runtimes; legacy list retires `frank-primary-qwen frank-primary-r9v qwen-fp8 glm-probe` |
+| `--tp 2 --tp-wire wht6` | both R9700s | fused WHT-6 lossy wire: +5.6–6.3 % prefill, ~−1.3 % 8-stream decode at TP2; `--tp-wire-min-kb` stays engine default (128 KiB) |
+| `--max-num-seqs 2` | first boot | Frank: "parallelism = 2 to start". Raise only after a real-traffic look at slot occupancy + the ledger's KV headroom line. The fused all-reduce row count is `seqs × (1 + spec) = 8` |
+| `--max-model-len 200000` | served window | native 262,144; the publisher's production value and what `models.yml` quotes. Longer prompt = refused, never truncated |
+| `--placement expert_tiered --host-pool-mib 40960` | pinned expert pool **per rank** | native FP8 experts are 112.8 GiB total (56.4 GiB/rank slice) vs ~29 GiB at the 4-bit publisher shape — the pool is the price of native precision. Resize from the boot ledger's coverage line, never arithmetic |
+| `--gpu-headroom-mib 96 --expert-vs-cache-ratio 0.82` | publisher values | 0.82 = measured 99 % expert-cache hit; 0.87 → 0 % (the cliff) |
+| `--kv-cache-dtype fp8` | **7.32 KiB/token** | 12 QSA full-attention layers, 2 KV heads × 256 dim, K+V; a 200K-token sequence is 1.43 GiB. KV is sized from the VRAM the ratio leaves |
+| `--num-speculative-tokens 3` | MTP head | the checkpoint carries `mtp.*` (3,101 tensors, `mtp_num_hidden_layers: 1`); the plugin's probe returns draft depth 3 — the publisher's operating point |
+| `--max-num-batched-tokens 2048` | prefill chunk | A/B-verified optimum (1024/2048/4096 ladder, see "Tuning levers") |
+| `--prefix-cache-host-mib 4096 --prefix-cache-dir /data/kvcache/frank-primary --prefix-cache-disk-mib 65536` | 3 tiers | host tier pinned (see the ledger), disk tier in `/home/dank/radiance-state` (mounted rw at `/data`; `/` is ext4, mapped-tier-eligible); a session snapshot moves with its blocks |
+| `--ngram-placement ram` | **PLE pinned to host RAM** | Frank's pin law. The PLE n-gram table is 51.2 GB (320,001,536 rows × 160 dims = 29 % of all parameters) served via io_uring reads; `ram` **refuses the boot** if memlock is too small |
+| `--override-chat-template /templates/template.jinja` | froggeric v22.5 | vendored `scripts/qwen38-froggeric-v22.5.jinja` (28,234 B, `qwen3.8-froggeric-v22.5`); rad-convert bakes the checkpoint's own template, this overrides it, and the engine derives the reasoning/tool-call reply format from it |
+
+**Pinned-RAM ledger** (root unit + `LimitMEMLOCK=infinity` exist for this):
+2 × 40 GiB expert pools + 47.7 GiB PLE + 4 GiB prefix host tier + ~2 GiB
+spare ≈ **134 GiB**. `ttm.pages_limit` (this kernel has NO `pages_max` —
+the two-knob story was stale) caps GPU-mappable pinned pages (default = half
+of RAM): the pre-cutover cap was **123 GiB** — BELOW the ledger — so the
+serve script's TTM preflight (`FRANK_PRIMARY_TTM_MIN_MIB=140000`) refuses to
+start until it is raised. DONE 2026-10-05 pre-cutover: sysfs write of
+**52,428,800 pages = 200 GiB** (`/sys/module/ttm/parameters/pages_limit`;
+verified read-back; raising at runtime is harmless and allocates nothing).
+PERSISTENCE: the sysfs write does not survive reboot — add
+`ttm.pages_limit=52428800` to the kernel cmdline at the next scheduled
+reboot.
+
+**Per-rank VRAM** at TP2: expert slice 56.4 GiB total across VRAM + host
+pool; trunk ~3.75 GiB/rank; token embedding vocab-parallel (~1.18 GiB/rank
+— the bf16 table is 2.43 GiB, sharding hands the other half to KV); lm_head
+row-sharded; recurrent state 54 MiB/copy/card (2 copies with speculation);
+KV from the `0.82`-side remainder of claimed VRAM at 96 MiB headroom.
+
+## The engine contract (all verified in source, not assumed)
+
+- **`/health` == `/ping`**: 200 JSON (`model_id, in_flight, queue_depth`)
+  while serving, 503 only while stopping — **the listener binds AFTER the
+  full load**, so a bare `curl -fsS http://127.0.0.1:7770/health` gate is
+  honest (the deployer loops on it; the FreeToken-era `"status":"ok"` body
+  grep is gone). Boot budget 1800 s (publisher `start_period` 15 m);
+  measured 2026-10-06: health 200 ~2-3 min after start (~120 s warm) —
+  the big terms are the 125.26 GiB page-in (38.1 s at 3.5 GB/s) and the
+  host-pool pin (PLE 47.68 GiB in 43.9 s).
+- **Sampling** (oai.cpp): `temperature, top_p, top_k, min_p, typical_p,
+  presence_penalty, frequency_penalty, repetition_penalty, repeat_last_n,
+  chat_template_kwargs, reasoning_effort, enable_thinking,
+  preserve_thinking` accepted; **unknown params are refused (400)** —
+  models.yml's extraBody set is fully inside it; nothing unverified rides
+  the wire. `max_tokens` is clamped, not refused; `seed` absent = pick
+  one; `response_format` supported; **`logprobs` is refused** (harness
+  never sends it).
+- **Reasoning**: `--reasoning-format` defaults to `auto` → thinking in
+  `reasoning_content` (models.yml `reasoningContentField` matches). Tool
+  calls: the qwen3.8 dialect → `message.tool_calls` +
+  `finish_reason: tool_calls`. No parser flags anywhere — parsers are
+  auto.
+- **Vision**: the official checkpoint carries the 27-layer vision tower;
+  the engine runs the Qwen-VL processor (patching, pad-token expansion,
+  3-part rotary are core). `models.yml` keeps `input: [text, image]`;
+  `--mm-max-patches` available.
+- **`/metrics`**: vLLM-compatible names + `radiance:*` families — the
+  Dankclaw Prometheus job re-scrapes 192.168.0.235:7770 (plain LAN routing
+  reaches it; the netavark published-port quirk is same-host only).
+  `/stats`, `/sessions`, `/dashboard` for live internals; the repo ships
+  `deploy/grafana/radiance.json` if a dashboard is wanted.
+- **`n` > 1 requests**: `n` defaults to `max(8, max-num-seqs)`; stop
+  strings ≤ 64 × 4096 B. `model` field not enforced.
+
+## Deploy + verify
+
+```bash
+# one-time (already done 2026-10-05): payload fetch + sha256 verify + convert
+ssh dankserver 'bash /home/dank/frank-primary-build/build-frank-primary-rad.sh --plan-only'   # declare + plan, no bytes
+ssh dankserver 'bash /home/dank/frank-primary-build/build-frank-primary-rad.sh'               # full convert + rad-info
+./scripts/deploy.sh dankserver --service frank-primary
+```
+
+Post-cutover checklist (each is observable, not assumed):
+
+1. Unit: `is-active` active, `is-enabled` enabled, container answers
+   `/health` with 200 JSON.
+2. Boot ledger (engine stdout → `/home/dank/logs/frank_primary.log`):
+   P2P probe per rank pair, TP=2 shard placement, expert pool coverage
+   line ("backs N % of the worst case … M full-length sessions"), KV pool
+   bytes, PLE pinned-RAM line, drafter depth 3, prefix tiers.
+3. MTP live: a completion's `timings` object shows draft/accept counts;
+   `radiance:*spec*` metrics in `/metrics`.
+4. Froggeric template: a probe prompt renders the v22.5 shape
+   (tool-call + thinking turn both exercise it).
+5. Vision: a ≥256 px solid-color PNG answers the exact color through an
+   `image_url` content part.
+6. Prometheus target `frank-primary` green (not never-green).
+
+## Cutover measured (2026-10-06, all values observed live)
+
+- Container: 1,857,127,997,440 B = 177,109 MiB = 173.0 GiB. Plan-only
+  priced 172.76 GiB — the DATA planes; the written container adds a
+  metadata tail. Floor gates must be set from the real file (the first
+  deploy refused on 178000 > 177109; floor is 177000).
+- Boot: 125.26 GiB read in 38.1 s (3.5 GB/s), PLE pinned
+  "47.68 GiB in 43.9 s" (the 2 B scale plane with it), each rank's
+  mover holds 25.46 GiB static+experts (19.15 GiB resident experts,
+  79 %), 16,820 pinned pool slots, health 200 ~2-3 min after start
+  (~120 s warm). No 7-minute vLLM-style boot.
+- KV ledger: kv_attn 99,956 blocks = 2.48 GiB (399,824 tokens),
+  kv_qsa_bkey 317.24 MiB, kv_gdn_state 4 slots × 54 MiB, PLE conv
+  states < 1 MiB; linear checkpoints 56.37 MiB × 8 slots (3 of 8
+  card-backed at boot, 5 withdrawn under the 208 MiB floor).
+- MTP on real traffic (temp 0.7, 220-token ask): draft_n 225 /
+  draft_n_accepted 119 = 52.9 % acceptance, decode 23.8 t/s. ONE
+  sample — the shape's real number needs live-traffic measurement.
+- Contracts: `reasoning_content` field (matches models.yml), usage
+  metering separates reasoning_tokens; vision — a 512² solid-red PNG
+  through `image_url` answers `Red` (279 prompt tokens; NOTE the
+  32-token budget is eaten by thinking — give vision probes ≥256
+  max_tokens); native `tool_calls` with finish_reason tool_calls.
+- Boot defects fixed the same day: the SSD prefix tier opened only
+  after the serve script mkdir -p'd the full path (the engine's
+  create is NOT recursive: "cannot create /data/kvcache/..."); the
+  converter's scale-name law, the PLE file-shadow and the
+  symlink-through-raw-index bug are in the payload section above.
+- Prometheus: `/metrics` answers 200 with vLLM-compatible families
+  (verified from Dankclaw); the `frank-primary` job needs a
+  prometheus-service deploy to go green (config pushed, reload
+  deferred behind a live parallel writer's dirty tree).
+
+## Tuning levers (in order of likely use)
+
+- **`FRANK_PRIMARY_MAX_SEQS`**: raise 2 → 3/4 only when the ledger's KV
+  headroom line and real slot occupancy agree; each slot buys a
+  recurrent-state copy (54 MiB/card) and shrinks the per-request KV share.
+- **`FRANK_PRIMARY_HOST_POOL_MIB`**: the ledger's coverage line is the
+  oracle — it prints what the pool backs at the worst case and how many
+  full-length sessions fit. RAM budget: pools + PLE + prefix ≤ ~140 GiB of
+  the host's 246.
+- **`FRANK_PRIMARY_EXPERT_VS_CACHE`**: 0.82 is the measured 99 %-hit
+  point; 0.87 zeroes it. Move only with a measured hit-rate reason.
+- **`FRANK_PRIMARY_TP_WIRE`**: `wht6` vs `exact` — `exact` trades the
+  +5.6 % prefill for bit-exact reductions; the fused one-shot forms are
+  declared over the seqs × (1+spec) row bound.
+- **`FRANK_PRIMARY_EXTRA_ARGS`**: the boot-ladder escape hatch, expanded
+  unquoted.
+- **`rad-tune -m <rad> --tp 2 --num-speculative-tokens 3`**: benchmarks
+  the tunable space at this model's shapes on this machine, writes
+  `$RADIANCE_HOME/tune/<machine>.tune` which the engine's selector reads at
+  boot. Declare at the shapes you serve (`--tp 2
+  --num-speculative-tokens 3`), or you measure a program the serve never
+  runs. `$RADIANCE_HOME` is `/opt/radiance/share/radiance` — in the IMAGE — so a
+  `--rm` scratch container's cache dies with it: the serve script
+  bind-mounts `$FRANK_PRIMARY_STATE_DIR/tune` (`/home/dank/radiance-state/tune`)
+  at `/opt/radiance/share/radiance/tune`, so the cache (36 GEMM rows,
+  `gfx1201-x3-AMD_Radeon_Graphics.tune`, generated 2026-10-06 in a
+  stopped-service window) survives and is read at every production boot.
+  The two attention requests have no shape recipe (`attn_paged`,
+  `attn_paged_gate_quant` — "see tools/opshapes.h") and are not tuned.
+  `RADIANCE_TUNE` pins a kernel for an A/B — an instrument, not a setting.
+- **`FRANK_PRIMARY_MAX_BATCHED_TOKENS` (MNBT)**: A/B'd 2026-10-06 in a
+  stopped-service window (real-text probes: 20,417-token cold prefill,
+  1,985-token warm prefill, 256-token decode). The prefill ladder is
+  NON-MONOTONIC with an interior optimum at 2048: 1024 → 2048 → 4096 =
+  231 → 460 → 448 t/s cold, 221 → 412 → 427 t/s warm. Decode is flat within
+  noise (24.0 / 25.1 / 24.3 t/s); draft acceptance 60.1 / 67.1 / 63.3 %.
+  **2048 wins** (peak prefill + peak acceptance) — it stays the production
+  value. 1024 halves prefill (the batch too small to saturate the card);
+  4096 is a hair off 2048 (the chunk past the kernel's sweet spot).
+- **Diagnostics**: `rad-info` (container: encodings, recipe, profile),
+  `--debug-graph` (what declared vs what issued), `rad-kbench` (kernel
+  correctness/speed vs libref), `/stats` + `/sessions` (live), and the
+  publisher's `scripts/` harnesses (`fnpf`, `fnconc`, `cstep2`, `ident`,
+  `mctools`) — note `mcserve/fnstop`-family scripts `pkill` the engine
+  binary by name; on this host the service is a systemd unit, so use
+  `sudo systemctl stop frank-primary` instead.
+
+## Rollback
+
+| Layer | Path |
+|---|---|
+| FreeToken (one card, native FP8 safetensors) | `git checkout frank-primary-freetoken-pr587-20261005 -- scripts/frank-primary.env scripts/frank-primary.service scripts/serve-frank-primary.sh scripts/docker/ tests/frank_primary_unit.bats` + redeploy; image `localhost/frank-primary-freetoken:pr587-77a7201` and payload `/home/dank/models/Qwen3.8-Flash-Next-Uncensored-FP8` stay on disk until the window closes |
+| tcclaviger vLLM fork (TP=2, shape G) | git tag `frank-primary-vllm-shapeG-20261003` (trio + env lore; image still in dank's rootless storage) |
+| 4-bit StillDeadcode publisher shape | `qwen3.8-next-flash-fp8-iq4r-moe.rad` + `--host-pool-mib 12288` — the documented fallback if the native container ever proves unbootable |
+
+Retire the FreeToken-era orcarouter payload to `~/.AGENTTRASH/` only after
+the rollback window closes (trash policy, never `rm`).
+
+## Links
+
+- [[LLM/index]] — fleet index
+- [[frank-primary-freetoken-pr587]] — the retired FreeToken era
+- [[dankserver-qwen38-fp8-expert-offload-tuning]] — the retired tcclaviger vLLM era
+- [[Dual-R9700-Cooperation]] — the board swap that made TP=2 legal here
+- [[RDNA4 Container Serving]] — rootless-podman GPU laws on these hosts
+- `serverAppConfigs-docs/Architecture/Testing Policy.md` — the seam contract

serverAppConfigs-docs/LLM/frank-redo-2026-08-17.md

@@ -18,7 +18,7 @@ The old scripts moved to `old/scripts/` in this repository.
 | Service | Host | Runtime | Model | Port |
 |---------|------|---------|-------|------|
 | frank-large | Dankserver | llama.cpp (patched FastMTP) | HauhauCS Qwen3.8-27B-Uncensored-Aggressive IQ4_XS | 7770 |
-| frank-small | Dankclaw | vLLM ROCm | nameistoken Qwen3.6-35B-A3B Quark W8A8 INT8 | 52625 |
+| frank-secondary | Dankclaw | vLLM ROCm | nameistoken Qwen3.6-35B-A3B Quark W8A8 INT8 | 52625 |
 
 The RTX 3090 runs the whole 27B model standalone. No RPC worker, no
 tensor split, no NPU.
@@ -40,7 +40,7 @@ tensor split, no NPU.
   `--spec-draft-ngl all`, `--spec-draft-p-min 0`) then ngram-mod fallback
   (48/64/24).
 
-### frank-small — nameistoken/Qwen3.6-35B-A3B-Quark-W8A8-INT8
+### frank-secondary — nameistoken/Qwen3.6-35B-A3B-Quark-W8A8-INT8
 
 - Base: Qwen3.6-35B-A3B (hybrid MoE, 256 experts top-8, ~3B active, 40
   layers: 10 blocks of 3 DeltaNet + 1 Attention, native 262144 ctx).
@@ -103,10 +103,10 @@ Production choice: **131072 (128K)** — the maximum that keeps about
 5. Verify: `curl -sf http://127.0.0.1:7770/health` and a chat completion.
 6. Confirm VRAM headroom with `nvidia-smi`.
 
-### Dankclaw (frank-small)
+### Dankclaw (frank-secondary)
 
 1. Stop the old services: `sudo systemctl stop frank-large
-   frank-small` (retired PP head and NPU endpoint).
+   frank-secondary` (retired PP head and NPU endpoint).
 2. Install uv and create the venv:
    ```
    curl -LsSf https://astral.sh/uv/install.sh | sh
@@ -131,9 +131,9 @@ Production choice: **131072 (128K)** — the maximum that keeps about
    vision encoder falls back to torch SDPA and engine init OOMs.
 5. Symlink the unit, reload, enable, restart:
    ```
-   sudo ln -sf /home/dank/vllmScripts/scripts/frank-small.service /etc/systemd/system/
-   sudo systemctl daemon-reload && sudo systemctl enable frank-small
-   sudo systemctl restart frank-small
+   sudo ln -sf /home/dank/vllmScripts/scripts/frank-secondary.service /etc/systemd/system/
+   sudo systemctl daemon-reload && sudo systemctl enable frank-secondary
+   sudo systemctl restart frank-secondary
    ```
 6. Verify: `curl -sf http://127.0.0.1:52625/health` and a chat completion.
 
@@ -159,7 +159,7 @@ on gfx1151; newer vLLM releases may already include it.
 
 Three changes landed after the initial deployment:
 
-### 1. Tool calling on frank-small (HTTP 400 fix)
+### 1. Tool calling on frank-secondary (HTTP 400 fix)
 
 Clients (pi) send `tool_choice: "auto"`. vLLM rejects it with HTTP 400
 unless the server enables tool calling. Fixes applied:
@@ -184,7 +184,7 @@ model behavior, which motivated the swap.
 
 ### 2. Model swap: Avesed INT8 W8A8 -> Quark W8A8 INT8
 
-`frank-small` now serves
+`frank-secondary` now serves
 `nameistoken/Qwen3.6-35B-A3B-Quark-W8A8-INT8` (AMD Quark 0.11.1, ~35 GB,
 vLLM-native expert layout). The Avesed compressed-tensors quant's
 Qwen3-XML tool fidelity was poor. Verified with the Quark model: plain

serverAppConfigs-docs/LLM/frank-secondary-benchmark.md

@@ -1,7 +1,7 @@
 ---
 timestamp: 2026-09-03
 type: reference
-title: frank-small Benchmark — ncmoe and Batch Tuning
+title: frank-secondary Benchmark — ncmoe and Batch Tuning
 description: llama-bench sweep of MoE CPU offload (ncmoe) and batch/ubatch on the RTX 3090 for Qwen3.6-35B-A3B Q6_K_P.
 tags:
   - benchmark
@@ -11,11 +11,11 @@ tags:
   - dankserver
 ---
 
-# frank-small Benchmark — ncmoe and Batch/Ubatch Tuning
+# frank-secondary Benchmark — ncmoe and Batch/Ubatch Tuning
 
 This page records a llama-bench sweep on Dankserver (RTX 3090, 24 GiB) for
 the HauhauCS Qwen3.6-35B-A3B Q6_K_P model. The sweep decided the final
-`--n-cpu-moe` and `--ubatch-size` values for `frank-small`.
+`--n-cpu-moe` and `--ubatch-size` values for `frank-secondary`.
 
 Run date: 2026-08-13. Build: llama.cpp `555881ebc` (10121). CUDA, 7 threads,
 `--cache-type-k q8_0 --cache-type-v f16`, `--fit-target 0` (KV auto-sized to
@@ -106,7 +106,7 @@ service runs 700416 cells (13,680 MiB).
 
 ## Chosen State
 
-> **Live drift (2026-08-22 audit):** the deployed `frank-small` now runs `--n-cpu-moe 38`, `--batch-size 4096`, `--ubatch-size 2048` (ncmoe raised from 36 for stability; batch lowered). The values below are the original benchmark recommendation.
+> **Live drift (2026-08-22 audit):** the deployed `frank-secondary` now runs `--n-cpu-moe 38`, `--batch-size 4096`, `--ubatch-size 2048` (ncmoe raised from 36 for stability; batch lowered). The values below are the original benchmark recommendation.
 
 Prefill speed is the priority, so the state keeps the top prefill tier and
 moves one expert layer to CPU for stability:

serverAppConfigs-docs/LLM/frank-secondary-gemma4-12b.md

@@ -1,10 +1,10 @@
 ---
 type: reference
-title: "frank-small on Dankclaw — llama.cpp, Gemma4-12B QAT Q4_K_M"
-description: "The active frank-small runbook — mainline llama.cpp on Dankclaw's RTX 3090 serving HauhauCS Gemma4-12B-QAT-Uncensored-HauhauCS-Balanced at Q4_K_M with a separate MTP assistant sidecar chained behind ngram-mod, q8_0 KV, and --fit sizing the full 262k window: identity, payload, KV arithmetic, the spec-decode chain, deployed flags, the client contract, and what to check when it misbehaves."
+title: "frank-secondary on Dankclaw — llama.cpp, Gemma4-12B QAT Q4_K_M"
+description: "The active frank-secondary runbook — mainline llama.cpp on Dankclaw's RTX 3090 serving HauhauCS Gemma4-12B-QAT-Uncensored-HauhauCS-Balanced at Q4_K_M with a separate MTP assistant sidecar chained behind ngram-mod, q8_0 KV, and --fit sizing the full 262k window: identity, payload, KV arithmetic, the spec-decode chain, deployed flags, the client contract, and what to check when it misbehaves."
 tags:
   - llm
-  - frank-small
+  - frank-secondary
   - dankclaw
   - llama.cpp
   - rtx3090
@@ -14,25 +14,25 @@ tags:
 timestamp: 2026-09-17
 ---
 
-# frank-small — llama.cpp Gemma4-12B QAT Q4_K_M (MTP + ngram-mod)
+# frank-secondary — llama.cpp Gemma4-12B QAT Q4_K_M (MTP + ngram-mod)
 
-> **ACTIVE since 2026-09-17.** Replaces [[frank-small-qwen38-gsq-rco]]
+> **ACTIVE since 2026-09-17.** Replaces [[frank-secondary-qwen38-gsq-rco]]
 > (ISTA-DASLab Qwen3.8-27B-GSQ-RCO IQ3_S, now the rollback target) and
-> [[frank-small-vllm-gemma4]] (the same-day vLLM experiment, also the
-> rollback target). Same port (7771), same alias (`frank-small`), same
+> [[frank-secondary-vllm-gemma4]] (the same-day vLLM experiment, also the
+> rollback target). Same port (7771), same alias (`frank-secondary`), same
 > client contract surface.
 
 ## Identity
 
-- **Endpoint:** `frank-small` at `192.168.0.143:7771` (Dankclaw).
+- **Endpoint:** `frank-secondary` at `192.168.0.143:7771` (Dankclaw).
 - **Runtime:** mainline llama.cpp master `37b53fd`, CUDA build for sm_86 with
   `-DGGML_CUDA_FA_ALL_QUANTS=ON`
   (`/home/dank/GitPackages/llama.cpp-mainline/build-cuda/bin/llama-server`).
   The FA_ALL_QUANTS flag is required: without it, q8_0 KV + flash attention
   collapses prefill ~12x on CUDA.
-- **Service shape:** root systemd unit `scripts/frank-small.service`
-  (`User=dank`), launched by `scripts/serve-frank-small.sh` with tunables from
-  `scripts/frank-small.env`.
+- **Service shape:** root systemd unit `scripts/frank-secondary.service`
+  (`User=dank`), launched by `scripts/serve-frank-secondary.sh` with tunables from
+  `scripts/frank-secondary.env`.
 
 ## Shape
 
@@ -90,7 +90,7 @@ fault).
 
 ## Client contract
 
-- **Alias:** `frank-small` (the `id` at `/v1/models`).
+- **Alias:** `frank-secondary` (the `id` at `/v1/models`).
 - **Reasoning:** extracted to `reasoning_content` (`--reasoning-format
   deepseek`, unlimited budget).
 - **Tools:** supported (`--jinja` is pinned even though this build defaults
@@ -109,12 +109,12 @@ pp ~965 t/s, tg 257–302 t/s, idle 23,300 MiB of 24,576. Larger ubatches
 
 ## Rollback
 
-Payload and runtime for [[frank-small-qwen38-gsq-rco]] (ISTA-DASLab
-Qwen3.8-27B-GSQ-RCO IQ3_S) remain intact on disk. [[frank-small-vllm-gemma4]]
+Payload and runtime for [[frank-secondary-qwen38-gsq-rco]] (ISTA-DASLab
+Qwen3.8-27B-GSQ-RCO IQ3_S) remain intact on disk. [[frank-secondary-vllm-gemma4]]
 is the same-day vLLM experiment (also a rollback target).
 
 # Citations
 
 - Model + mmproj + MTP sidecar: https://huggingface.co/HauhauCS
 - llama.cpp: https://github.com/ggml-org/llama.cpp
-- Serve script: `scripts/serve-frank-small.sh`; tunables: `scripts/frank-small.env`
+- Serve script: `scripts/serve-frank-secondary.sh`; tunables: `scripts/frank-secondary.env`

serverAppConfigs-docs/LLM/frank-secondary-llama-mimo-q5km.md

@@ -1,10 +1,10 @@
 ---
 type: reference
-title: "frank-small on Dankclaw — llama.cpp, MiMo-V2.6-Distill-Qwen-9B Q5_K_M GGUF (retired)"
-description: "The retired 2026-10-02/03 frank-small runbook — the Q5_K_M rung at two wide unquantized-KV slots on the RTX 3090, superseded 2026-10-03 by the Q6_K file. Kept as the rollback reference: it carries the measured boot-pair table that prices every pool shape on this card, the slot-window contract, and the misbehaves tree."
+title: "frank-secondary on Dankclaw — llama.cpp, MiMo-V2.6-Distill-Qwen-9B Q5_K_M GGUF (retired)"
+description: "The retired 2026-10-02/03 frank-secondary runbook — the Q5_K_M rung at two wide unquantized-KV slots on the RTX 3090, superseded 2026-10-03 by the Q6_K file. Kept as the rollback reference: it carries the measured boot-pair table that prices every pool shape on this card, the slot-window contract, and the misbehaves tree."
 tags:
   - llm
-  - frank-small
+  - frank-secondary
   - dankclaw
   - llama-cpp
   - rtx3090
@@ -14,17 +14,17 @@ tags:
 timestamp: 2026-10-03
 ---
 
-# frank-small — llama.cpp MiMo-V2.6 Q5_K_M (RTX 3090, retired)
+# frank-secondary — llama.cpp MiMo-V2.6 Q5_K_M (RTX 3090, retired)
 
 > **RETIRED 2026-10-03** — superseded by
-> [[frank-small-llama-mimo-q6k-2x200704]]: the ~3.5 GiB this shape left free
+> [[frank-secondary-llama-mimo-q6k-2x200704]]: the ~3.5 GiB this shape left free
 > bought the **Q6_K quant rung** (same runtime, same 2 × 200,704 pool, same
 > f16 KV), which measures 2,849 MiB free and 94.44 tok/s — 10 % slower
 > decode is the measured price of the higher rung on this bandwidth-bound
 > card. This file remains the rollback reference and the owner of the
 > measured boot-pair table.
 
-> Its own history: one quant step down from [[frank-small-llama-mimo-q6k]]
+> Its own history: one quant step down from [[frank-secondary-llama-mimo-q6k]]
 > (2026-10-01), resized twice on 2026-10-02 (three contexts with vision, then
 > the 2–3 GiB-band pool), and on 2026-10-03 morning reshaped to **two
 > conversations of 200,704 tokens with the KV VALUES unquantized** —
@@ -36,11 +36,11 @@ timestamp: 2026-10-03
 
 |Field|Value|
 |---|---|
-|Endpoint|`192.168.0.143:7771`, served alias `frank-small`|
+|Endpoint|`192.168.0.143:7771`, served alias `frank-secondary`|
 |Runtime|llama.cpp **master** (qwen35 model support), CUDA build `sm_86`, at `/home/dank/GitPackages/llama-master/build-cuda/bin/llama-server` — unchanged across the quant swap|
-|Payload|`bartowski/MiMo-V2.6-Distill-Qwen-9B-GGUF` **Q5_K_M**, 6,876,124,704 B (6.40 GiB), sha256 `04cf46f2b5584ef922697d6c063f94879434fb86c467f1a09e52daf0cddaf35f` + the repo's `mmproj-MiMo-V2.6-Distill-Qwen-9B-f16.gguf`, 918,166,048 B, sha256 `ff348f3180a63188aa7285db85f550fe38acb61dd013c599eb8bad08d2cc2576` — both at `/home/dank/models/MiMo-Q5KM/`, fetched and byte- and sha256-verified by `scripts/download-frank-small.sh`|
-|Unit|root `frank-small.service` (repo `scripts/frank-small.service`, `User=dank`, relinked), env `scripts/frank-small.env`, serve `scripts/serve-frank-small.sh`|
-|Repo pins|`tests/frank_small_unit.bats` (stub-argv seam + the VRAM arithmetic), `deploy_llm_small` gates in `scripts/deploy-remote.sh`|
+|Payload|`bartowski/MiMo-V2.6-Distill-Qwen-9B-GGUF` **Q5_K_M**, 6,876,124,704 B (6.40 GiB), sha256 `04cf46f2b5584ef922697d6c063f94879434fb86c467f1a09e52daf0cddaf35f` + the repo's `mmproj-MiMo-V2.6-Distill-Qwen-9B-f16.gguf`, 918,166,048 B, sha256 `ff348f3180a63188aa7285db85f550fe38acb61dd013c599eb8bad08d2cc2576` — both at `/home/dank/models/MiMo-Q5KM/`, fetched and byte- and sha256-verified by `scripts/download-frank-secondary.sh`|
+|Unit|root `frank-secondary.service` (repo `scripts/frank-secondary.service`, `User=dank`, relinked), env `scripts/frank-secondary.env`, serve `scripts/serve-frank-secondary.sh`|
+|Repo pins|`tests/frank_secondary_unit.bats` (stub-argv seam + the VRAM arithmetic), `deploy_llm_secondary` gates in `scripts/deploy-remote.sh`|
 
 ## Shape and the KV contract
 
@@ -128,7 +128,7 @@ The per-slot window is the whole price, in both directions:
   "Brown.", i.e. it generated from text alone. `/completion` takes `images`
   but refuses a request without `messages` in this build, so chat is the only
   vision path.
-- Setting `MMPROJ_FILE=` (explicitly empty) in `scripts/frank-small.env` drops
+- Setting `MMPROJ_FILE=` (explicitly empty) in `scripts/frank-secondary.env` drops
   `--mmproj` and returns ~1.4 GiB of the card to KV — the documented text-only
   opt-out. `MMPROJ_FILE` pointing at a file that does not exist is a boot
   refusal, never a silent fallback to blindness.
@@ -145,15 +145,15 @@ traffic first. The bats seam fails if a draft flag ships unannounced.
 
 |Target|How|
 |---|---|
-|Previous shape: 3 × 180,224 + q8_0 V (2026-10-02)|`scripts/frank-small.env` `CTX_SIZE=540672 PARALLEL=3 CACHE_TYPE_V=q8_0` (measured 21,684 MiB used, 2,441 free) and `contextWindow: 180224` in `models.yml` — choose it when three simultaneous conversations matter more than per-slot window or exact V|
-|Q6_K, same runtime|`scripts/frank-small.env` `MODEL_FILE=/home/dank/models/MiMo-Q6K/MiMo-V2.6-Distill-Qwen-9B-Q6_K.gguf` + `MMPROJ_FILE` in that dir — **this is the deployed rung since 2026-10-03** ([[frank-small-llama-mimo-q6k-2x200704]]): at 2 × 200,704 it fits with 2,849 MiB free, no pool cut needed (the older "+1.4 GiB, drop 49,152 tokens" estimate held only for the 3-slot shapes)|
+|Previous shape: 3 × 180,224 + q8_0 V (2026-10-02)|`scripts/frank-secondary.env` `CTX_SIZE=540672 PARALLEL=3 CACHE_TYPE_V=q8_0` (measured 21,684 MiB used, 2,441 free) and `contextWindow: 180224` in `models.yml` — choose it when three simultaneous conversations matter more than per-slot window or exact V|
+|Q6_K, same runtime|`scripts/frank-secondary.env` `MODEL_FILE=/home/dank/models/MiMo-Q6K/MiMo-V2.6-Distill-Qwen-9B-Q6_K.gguf` + `MMPROJ_FILE` in that dir — **this is the deployed rung since 2026-10-03** ([[frank-secondary-llama-mimo-q6k-2x200704]]): at 2 × 200,704 it fits with 2,849 MiB free, no pool cut needed (the older "+1.4 GiB, drop 49,152 tokens" estimate held only for the 3-slot shapes)|
 |Two 262,144 windows|`CTX_SIZE=524288 PARALLEL=2 CACHE_TYPE_V=q8_0` (measured 20,190 MiB used, 4,386 free **with q8_0 V** — with f16 V the same shape is 16,384 MiB of KV + ~7,986 non-KV and does not fit) and `contextWindow: 262144` in `models.yml` — choose it when a single conversation needs the native window|
-|Vision off|`MMPROJ_FILE=` in the env file; the deployer gate then has to be satisfied too, so point `LLM_SMALL_MMPROJ` at any real projector file or drop the gate in the same commit|
-|SGLang + INT8|[[frank-small-sglang-mimo-v26]] — venv and old unit/env/serve trio still in place; the INT8 payload was staged to `/home/dank/.AGENTTRASH/20261002-retired-llm-payloads/` on 2026-10-02, so restore it there or re-fetch with `old/download-frank-small-sglang-int8.sh`|
+|Vision off|`MMPROJ_FILE=` in the env file; the deployer gate then has to be satisfied too, so point `LLM_SECONDARY_MMPROJ` at any real projector file or drop the gate in the same commit|
+|SGLang + INT8|[[frank-secondary-sglang-mimo-v26]] — venv and old unit/env/serve trio still in place; the INT8 payload was staged to `/home/dank/.AGENTTRASH/20261002-retired-llm-payloads/` on 2026-10-02, so restore it there or re-fetch with `old/download-frank-secondary-sglang-int8.sh`|
 
 ## When it misbehaves
 
-- Port 7771 dark: `journalctl -u frank-small -n 50`; the serve script exits 1
+- Port 7771 dark: `journalctl -u frank-secondary -n 50`; the serve script exits 1
   naming the missing file if the binary or GGUF moved.
 - `No CUDA GPUs are available`: the 3090 vanished from the bus or the UVM node
   is missing — `nvidia-modprobe -u -c=0` (the serve script runs it).

serverAppConfigs-docs/LLM/frank-secondary-llama-mimo-q6k-2x200704.md

@@ -1,10 +1,10 @@
 ---
 type: reference
-title: "frank-small on Dankclaw — llama.cpp, MiMo-V2.6 Q6_K, 2 x 200704, f16 KV"
-description: "The active frank-small runbook — llama.cpp master (CUDA sm_86) serving bartowski's MiMo-V2.6-Distill-Qwen-9B Q6_K GGUF on the RTX 3090: two conversations of 200,704 tokens with fully unquantized f16 KV, vision through the bundled projector, and the boot-pair arithmetic that prices every pool change."
+title: "frank-secondary on Dankclaw — llama.cpp, MiMo-V2.6 Q6_K, 2 x 200704, f16 KV (RETIRED)"
+description: "The RETIRED frank-secondary runbook (2026-10-03 → 10-05 evening, rollback target) — llama.cpp master (CUDA sm_86) serving bartowski's MiMo-V2.6-Distill-Qwen-9B Q6_K GGUF on the RTX 3090: two conversations of 200,704 tokens with fully unquantized f16 KV, vision through the bundled projector, and the boot-pair arithmetic that prices every pool change. Superseded by the Swift-1.5-27B IQ3_S-mtp swap."
 tags:
   - llm
-  - frank-small
+  - frank-secondary
   - dankclaw
   - llama-cpp
   - gguf
@@ -12,29 +12,31 @@ tags:
 timestamp: 2026-10-03
 ---
 
-# frank-small — llama.cpp MiMo Q6_K, 2 × 200,704, f16 KV
+# frank-secondary — llama.cpp MiMo Q6_K, 2 × 200,704, f16 KV
 
-> **ACTIVE since 2026-10-03.** The third shape Frank bought in two days on
-> this card: the freed third slot paid for **exact KV values**
-> ([[frank-small-llama-mimo-q5km]] documents that step and remains the
-> rollback), and the ~3.5 GiB still free then paid for the **Q6_K quant rung**
-> — the same file this endpoint served on 2026-10-01, come back as an
-> upgrade rather than a compromise. The pool held at 401,408 (2 × 200,704);
-> only the weights moved up.
+> **RETIRED 2026-10-05 (evening).** Superseded by
+> [[frank-secondary-llama-swift15-unc-iq3xxs-mtp-114688]] — the Swift-1.5-27B
+> RCO-IQ3_XXS-mtp swap (uncensored, MTP drafted). Two facts for the trail:
+> committed earlier the same day (77decb5..380aafa) was **never deployed** —
+> THIS MiMo shape was the live payload when Swift cut over; (2) the KV
+> contract Frank set here (f16 K + f16 V, "no further quantization on this
+> endpoint") was superseded the same evening by the Swift swap's explicit
+> q8_0/q8_0 order. This page stays as the rollback and the measured
+> boot-pair record. Payload on disk: `/home/dank/models/MiMo-Q6K/`.
 
 ## Identity
 
 | | |
 |---|---|
 | Host / device | Dankclaw (192.168.0.143), RTX 3090, 24,576 MiB |
-| Endpoint | `http://192.168.0.143:7771` — alias `frank-small`, `/v1` OpenAI surface |
+| Endpoint | `http://192.168.0.143:7771` — alias `frank-secondary`, `/v1` OpenAI surface |
 | Runtime | llama.cpp master, CUDA sm_86 build `…/llama-master/build-cuda/bin/llama-server` (`GGML_CUDA_FA_ALL_QUANTS=ON`) |
-| Unit | root systemd `frank-small.service` ([`scripts/frank-small.service`](../../scripts/frank-small.service)), `User=dank`, `Restart=on-failure`, `LimitMEMLOCK=infinity` |
-| Serve script | [`scripts/serve-frank-small.sh`](../../scripts/serve-frank-small.sh), tunables [`scripts/frank-small.env`](../../scripts/frank-small.env) |
+| Unit | root systemd `frank-secondary.service` ([`scripts/frank-secondary.service`](../../scripts/frank-secondary.service)), `User=dank`, `Restart=on-failure`, `LimitMEMLOCK=infinity` |
+| Serve script | [`scripts/serve-frank-secondary.sh`](../../scripts/serve-frank-secondary.sh), tunables [`scripts/frank-secondary.env`](../../scripts/frank-secondary.env) |
 | Weights | `/home/dank/models/MiMo-Q6K/MiMo-V2.6-Distill-Qwen-9B-Q6_K.gguf` — bartowski/MiMo-V2.6-Distill-Qwen-9B-GGUF **Q6_K**, 7,793,710,624 B, sha256 `ef96d05a…` (qwen3_5 hybrid: 32 layers, 8 full-attention + 24 Gated DeltaNet, native 262,144, no MTP tensors) |
 | Projector | `…/MiMo-Q6K/mmproj-MiMo-V2.6-Distill-Qwen-9B-f16.gguf` — 918,166,048 B, sha256 `ff348f…` (hardlink of the Q5KM dir's copy, link-count-2 verified; the dir is self-contained) |
-| Fetch | [`scripts/download-frank-small.sh`](../../scripts/download-frank-small.sh) — `-f`, byte count + LFS sha256 verified, `--verify-only`; both files verified against the pinned sha on the host before this deploy |
-| Seam | `tests/frank_small_unit.bats` (argv assertions against a stub engine: payload path, slot geometry, KV pair, batch pair, projector opt-out) |
+| Fetch | [`scripts/download-frank-secondary.sh`](../../scripts/download-frank-secondary.sh) — `-f`, byte count + LFS sha256 verified, `--verify-only`; both files verified against the pinned sha on the host before this deploy |
+| Seam | `tests/frank_secondary_unit.bats` (argv assertions against a stub engine: payload path, slot geometry, KV pair, batch pair, projector opt-out) |
 
 ## Shape and KV contract
 
@@ -65,7 +67,7 @@ compaction-prompt fit by 1,236 tokens — or revert V to `q8_0`), and both
 were declined. Re-open only on evidence the Q6_K residual matters.
 
 **Pool prices on this card come from boots, never formulas** — the measured
-boot-pair table lives in [[frank-small-llama-mimo-q5km]] (*Shape and KV
+boot-pair table lives in [[frank-secondary-llama-mimo-q5km]] (*Shape and KV
 contract*) and still prices every rollback shape.
 
 ## Compute-buffer geometry (2026-10-05)
@@ -76,7 +78,7 @@ measured rather than assumed, because the predecessor shape recorded **"ubatch
 
 **That failure was a different shape, and it did not reproduce here.** The
 service itself is the bench (one card holds one server, so no scratch port):
-the old rung was re-booted through a transient `FRANK_SMALL_ENV_FILE` drop-in
+the old rung was re-booted through a transient `FRANK_SECONDARY_ENV_FILE` drop-in
 and reproduced the 2026-10-03 boot **byte-identically** — 21,276 MiB used /
 2,849 MiB free — which makes the ubatch the only variable in the comparison.
 
@@ -97,7 +99,7 @@ Three readings:
 - **The price is 616 MiB out of a 2.8 GiB margin.** 2,279 MiB free is still
   inside the 2–3 GiB band, at its bottom edge. `4096/1536` is the middle rung,
   and the pool's 12,288-token (384 MiB) ladder in
-  [[frank-small-llama-mimo-q5km]] is the other way to buy room back.
+  [[frank-secondary-llama-mimo-q5km]] is the other way to buy room back.
 
 **Why 2048 is right HERE while 512 was right on the R9700 sibling: the curve is
 backend-specific.** The 2× R9700 layer split peaks at ubatch 512 and is 10 %
@@ -123,7 +125,7 @@ that is the pair the curve was measured at.
 
 ## Client contract
 
-llama.cpp surface, live-verified on this model family: model id `frank-small`,
+llama.cpp surface, live-verified on this model family: model id `frank-secondary`,
 native tools via `--jinja` (`tool_calls` in the response), thinking in
 **`reasoning_content`**, vision via `image_url` content parts. Two vision
 traps, both measured here:
@@ -145,13 +147,13 @@ so the harness compacts before llama.cpp answers a longer prompt with a 400.
 
 |Target|How|
 |---|---|
-|Q5_K_M at the same shape|`scripts/frank-small.env` `MODEL_FILE=/home/dank/models/MiMo-Q5KM/MiMo-V2.6-Distill-Qwen-9B-Q5_K_M.gguf` (+ `MMPROJ_FILE` in that dir) and the deployer default — buys back 875 MiB (3,571 free measured) **and the 10 % decode rate** (104.84 tok/s measured). Payload sha-verified on disk|
+|Q5_K_M at the same shape|`scripts/frank-secondary.env` `MODEL_FILE=/home/dank/models/MiMo-Q5KM/MiMo-V2.6-Distill-Qwen-9B-Q5_K_M.gguf` (+ `MMPROJ_FILE` in that dir) and the deployer default — buys back 875 MiB (3,571 free measured) **and the 10 % decode rate** (104.84 tok/s measured). Payload sha-verified on disk|
 |3 × 180,224 + q8_0 V|`CTX_SIZE=540672 PARALLEL=3 CACHE_TYPE_V=q8_0` (measured 21,684 used / 2,441 free **with Q5_K_M** — Q6_K does NOT fit that shape inside the floor; pair them explicitly)|
-|SGLang + INT8|[[frank-small-sglang-mimo-v26]] — payload staged to `/home/dank/.AGENTTRASH/20261002-retired-llm-payloads/`|
+|SGLang + INT8|[[frank-secondary-sglang-mimo-v26]] — payload staged to `/home/dank/.AGENTTRASH/20261002-retired-llm-payloads/`|
 
 ## When it misbehaves
 
-Same decision tree as [[frank-small-llama-mimo-q5km]] (*When it misbehaves*):
+Same decision tree as [[frank-secondary-llama-mimo-q5km]] (*When it misbehaves*):
 `No CUDA GPUs are available` ⇒ `nvidia-modprobe -u -c=0`; a 400 naming
 200,704 ⇒ the conversation outgrew one slot, not the pool; slow decode after
 a shape change ⇒ check `nvidia-smi` for KV pushed into unified memory; queue

serverAppConfigs-docs/LLM/frank-secondary-llama-mimo-q6k.md

@@ -1,10 +1,10 @@
 ---
 type: reference
-title: "frank-small on Dankclaw — llama.cpp, MiMo-V2.6-Distill-Qwen-9B Q6_K GGUF (retired)"
-description: "The retired 2026-10-01 frank-small runbook — the first llama.cpp + GGUF shape on the RTX 3090: two native-window sessions, the asymmetric KV contract (f16 K + q8_0 V) that motivated the move off SGLang, the VRAM fit at Q6_K, and what to check when it misbehaves. Superseded the same night by the Q5_K_M payload."
+title: "frank-secondary on Dankclaw — llama.cpp, MiMo-V2.6-Distill-Qwen-9B Q6_K GGUF (retired)"
+description: "The retired 2026-10-01 frank-secondary runbook — the first llama.cpp + GGUF shape on the RTX 3090: two native-window sessions, the asymmetric KV contract (f16 K + q8_0 V) that motivated the move off SGLang, the VRAM fit at Q6_K, and what to check when it misbehaves. Superseded the same night by the Q5_K_M payload."
 tags:
   - llm
-  - frank-small
+  - frank-secondary
   - dankclaw
   - llama-cpp
   - rtx3090
@@ -14,10 +14,10 @@ tags:
 timestamp: 2026-10-01
 ---
 
-# frank-small — llama.cpp MiMo-V2.6 Q6_K (RTX 3090)
+# frank-secondary — llama.cpp MiMo-V2.6 Q6_K (RTX 3090)
 
 > **RETIRED 2026-10-01 → 10-02** (one night): superseded by
-> [[frank-small-llama-mimo-q5km]], the same runtime and shape one quant rung
+> [[frank-secondary-llama-mimo-q5km]], the same runtime and shape one quant rung
 > down. The Q6_K file stays at `/home/dank/models/MiMo-Q6K/` as the rollback
 > target.
 >
@@ -29,11 +29,11 @@ timestamp: 2026-10-01
 
 |Field|Value|
 |---|---|
-|Endpoint|`192.168.0.143:7771`, served alias `frank-small`|
+|Endpoint|`192.168.0.143:7771`, served alias `frank-secondary`|
 |Runtime|llama.cpp **master** (qwen35 model support), CUDA build `sm_86`, at `/home/dank/GitPackages/llama-master/build-cuda/bin/llama-server`|
 |Payload|`bartowski/MiMo-V2.6-Distill-Qwen-9B-GGUF` **Q6_K**, 7,793,710,624 B, sha256 `ef96d05a2ddf2cbb450d1af1ac3860ec769d3575bafa70692ee5609bda3fad7d`, at `/home/dank/models/MiMo-Q6K/`|
-|Unit|root `frank-small.service` (repo `scripts/frank-small.service`, `User=dank`, relinked), env `scripts/frank-small.env`, serve `scripts/serve-frank-small.sh`|
-|Repo pins|`tests/frank_small_unit.bats` (stub-argv seam), `deploy_llm_small` gates in `scripts/deploy-remote.sh`|
+|Unit|root `frank-secondary.service` (repo `scripts/frank-secondary.service`, `User=dank`, relinked), env `scripts/frank-secondary.env`, serve `scripts/serve-frank-secondary.sh`|
+|Repo pins|`tests/frank_secondary_unit.bats` (stub-argv seam), `deploy_llm_secondary` gates in `scripts/deploy-remote.sh`|
 
 ## Shape and the KV contract
 
@@ -100,7 +100,7 @@ use the parent of the 2026-10-01 night swap commit).
 
 ## When it misbehaves
 
-- Port 7771 dark: `journalctl -u frank-small -n 50`; the serve script
+- Port 7771 dark: `journalctl -u frank-secondary -n 50`; the serve script
   exits 1 naming the missing file if the binary or GGUF moved.
 - `No CUDA GPUs are available`: the 3090 vanished from the bus or the UVM
   node is missing — `nvidia-modprobe -u -c=0` (the serve script runs it).

serverAppConfigs-docs/LLM/frank-secondary-llama-swift15-unc-iq3xxs-mtp-114688.md

@@ -0,0 +1,338 @@
+---
+type: reference
+title: "frank-secondary on Dankclaw — llama.cpp, Swift-1.5-27B Uncensored RCO IQ3_XXS-mtp, 2 x 114688, draft-mtp, q8_0 KV"
+description: "The active frank-secondary runbook (live since 2026-10-05 evening; renamed from frank-small 2026-10-06, per-slot lowered to 112k and ubatch raised to 1024 the same evening) — llama.cpp master (CUDA sm_86) serving ajgazin's Swift-1.5-Qwen3.8-27B Uncensored RCO-IQ3_XXS (MTP head in the file, drafted) on the RTX 3090: TWO 114,688-token conversations at Frank's 112k cap, q8_0 K + q8_0 V (llama.cpp's FP8 class), vision through the publisher's own BF16 projector, and the full two-slot + drafter arithmetic."
+tags:
+  - llm
+  - frank-secondary
+  - dankclaw
+  - llama-cpp
+  - gguf
+  - rtx3090
+  - qwen35
+  - mtp
+  - vision
+timestamp: 2026-10-06
+---
+
+# frank-secondary — llama.cpp Swift-1.5-27B Uncensored RCO IQ3_XXS-mtp, 2 × 114,688, draft-mtp, q8_0 KV
+
+> **ACTIVE since 2026-10-05 (evening); reshaped three times on
+> 2026-10-06, renamed from frank-small the same night.** A 27B replaces
+> the 9B on the same 24 GiB card: KV at FP8 class, vision on. The
+> morning order (drop MTP, ubatch 512, TWO 128k contexts) bought the
+> second slot — 1,397 MiB free. The evening order swapped to the
+> UNCELERSED Swift-1.5 at RCO-IQ3_XXS and put MTP back ON — landing
+> **489 MiB free**: the MTP draft ctx costs **~2.4 GiB, not the ~0.4
+> GiB assumed** (the drafter's KV is built against the target's full
+> context, not the 3-token proposal depth), paid for by the lighter
+> quant (−1.56 GiB). The late order dropped the per-slot window
+> 131,072 → **114,688** (Frank's 112k cap) and climbed the prefill ladder
+> ubatch 512 → 1024 → **2048** — landing **1,005 MiB free** (1,751 at
+> the 1024 rung; the 2048 rung cost 746 MiB, ~130 over the 616 MiB
+> measured prediction). This supersedes BOTH earlier 2026-10-05
+> payloads: the live MiMo-V2.6 Q6_K shape
+> ([[frank-secondary-llama-mimo-q6k-2x200704]], now retired) and the
+> Ornith-1.5-9B Q8_0 swap that was committed (77decb5..380aafa) but
+> never deployed.
+
+## Identity
+
+| | |
+|---|---|
+| Host / device | Dankclaw (192.168.0.143), RTX 3090, 24,576 MiB |
+| Endpoint | `http://192.168.0.143:7771` — alias `frank-secondary`, `/v1` OpenAI surface |
+| Runtime | llama.cpp master (0.5.0-dev, commit 5fc4f3c), CUDA sm_86 build `…/llama-master/build-cuda/bin/llama-server` (`GGML_CUDA_FA_ALL_QUANTS=ON` — required for quantized V) |
+| Unit | root systemd `frank-secondary.service` ([`scripts/frank-secondary.service`](../../scripts/frank-secondary.service)), `User=dank`, `Restart=on-failure`, `LimitMEMLOCK=infinity` |
+| Serve script | [`scripts/serve-frank-secondary.sh`](../../scripts/serve-frank-secondary.sh), tunables [`scripts/frank-secondary.env`](../../scripts/frank-secondary.env) |
+| Weights | `/home/dank/models/Swift15-UncIQ3XXS/Swift-1.5-Qwen3.8-27B-Uncensored-Dynamic-MTP-RCO-IQ3_XXS.gguf` — ajgazin/Swift-1.5-Qwen3.8-27B-Uncensored-Dynamic-MTP-GGUF **RCO-IQ3_XXS -mtp**, 10,442,827,392 B, sha256 `e13d3e74…` (qwen35 hybrid: 64 real layers, 16 full-attention + 48 Gated DeltaNet, native 262,144; **MTP head in the file**: blk.64.nextn.*, `nextn_predict_layers=1`, and DRAFTED with it) |
+| Projector | `…/Swift15-UncIQ3XXS/mmproj-BF16.gguf` — PUBLISHER-SHIPPED in the same repo, 931,145,888 B, sha256 `19acd7fc…` (27-layer ViT → 5120) |
+| Fetch | [`scripts/download-frank-secondary.sh`](../../scripts/download-frank-secondary.sh) — `-f`, byte count + sha256 verified per file (Xet repo: the Hub APIs expose no LFS oid, so the shas were computed locally after the first verified fetch; byte counts match the Hub's declared sizes) |
+| Seam | `tests/frank_secondary_unit.bats` (argv assertions against a stub engine: payload path, slot geometry, KV pair, spec pair (both flags present), batch pair, projector opt-out) |
+
+## Why this payload (2026-10-06 evening)
+
+Frank asked for the UNCELERSED Swift-1.5 with MTP back on. The RCO-IQ3_XXS
+is 1.56 GiB lighter than the IQ3_S-mtp it replaced — and that lighter
+weight is exactly what funds the MTP draft context (~2.4 GiB measured)
+PLUS the second 131,072 slot, while keeping the q8_0 KV contract and the
+two-slot shape Frank ordered that morning.
+
+**Uncensored, intentionally:** the publisher reports 23/100 refusals vs
+Swift 1.5's 98/100 (KL 0.0884 against the censored base). The refusal
+escape hatch is the same as every payload here: revert and redeploy.
+
+## Shape and KV contract
+
+`--ctx-size 229376 --parallel 2` — TWO conversations at Frank's 112k
+cap (2026-10-06 late: cut from the 128k the morning/evening shapes ran
+at, to fund ubatch 1024). `CTX_SIZE` stays divisible by `PARALLEL`; the
+seam enforces it, and the per-slot number (`229376 / 2 = 114,688`) is
+the product contract — models.yml follows at 114,688 (a longer prompt
+is a 400, never a truncation).
+
+**KV at FP8 class: `q8_0` K + `q8_0` V (Frank, 2026-10-05 evening).**
+llama.cpp has no true fp8 KV dtype; q8_0 is the 8-bit class (1.0625 B/elem
+⇒ 8.5 effective bits) and the only FP8-class lever this runtime moves.
+The order supersedes the same-day f16/f16 rule, which was written for the
+Ornith 9B fit — at 64 KiB/token f16 the single 131,072 slot alone would
+cost 8.0 GiB of KV. At q8_0 the rate is **34 KiB/token** (16 attn layers ×
+4 KV heads × 256 dim × 1.0625 B, K and again V). `--flash-attn on` is
+pinned: quantized V requires it, and the build's
+`GGML_CUDA_FA_ALL_QUANTS=ON` (2026-09-17) makes the path real rather than
+a silent f16 fallback.
+
+**Two slots + drafter, by arithmetic and by order (2026-10-06 morning /
+evening, the 2 × 131,072 era).** The morning order bought the second slot
+by spending the drafter and ubatch headroom (1,397 MiB free). The evening
+swap put the drafter BACK and changed the arithmetic: the draft ctx is
+built against the target's FULL context (262,144 — the draft window, not
+the 3-token proposal depth), so it costs **~2.4 GiB measured** (23,636 −
+22,728 + 1,561), not the ~0.4 GiB the one-slot era assumed. The lighter
+quant pays for it: **23,636 used / 489 MiB free** (nvidia-smi `free`
+field) — the tightest fit in frank-secondary's history, the 2–3 GiB floor
+knowingly violated, and the 512×512 vision transient (34 MiB) still fits
+and recovers.
+
+**The late reshape (2 × 114,688, ubatch 2048) is the current fit.** The
+per-slot cut frees 2 × 16,384 tokens × 34 KiB ≈ 1.05 GiB of KV; the
+ubatch rungs 512 → 1024 → 2048 spend it back on compute buffers (a
+prefill knob — decode never moves with it; the rungs cost ~0.3 and
+~0.75 GiB measured). Measured boot at 2048: **23,120 used / 1,005 MiB
+free**. The drafter's own KV shrank with its target (draft window
+229,376). Compute buffers allocate at graph capture during BOOT, so a
+miss is loud and fast — the failure mode is a crash loop, not silent
+corruption.
+
+## Speculative decoding — ON since 2026-10-06 evening (draft-mtp, depth 3)
+
+**Frank's evening order: MTP back on.** The MTP head ships IN the file —
+the `-mtp` GGUF carries `blk.64.nextn.*` (`qwen35.nextn_predict_layers=1`)
+— and `--spec-type draft-mtp` loads it against the main model's own
+weights (no `--model-draft`, no second draft model — two context-backed
+drafters cannot share a process). `SPEC_DRAFT_N_MAX=3` is the proposal
+depth. **ngram-mod stays skipped, not swapped in:** on a qwen35 hybrid,
+ngram speculation logs *"recurrent state rollback is not compatible with
+'ngram-mod' — disabling rollback support"* (fleet lesson from
+frank-large's chain measurement, 2026-10-03). A rejected draft on Gated
+DeltaNet layers cannot be rewound by KV truncation — without the rollback
+checkpoints the DeltaNet state carries phantom transitions and output
+degrades silently. The flags move as a set: the serve script emits BOTH
+`--spec-type` and `--spec-draft-n-max` together or neither
+(`SPEC_TYPE=none` strips both).
+
+## Vision contract
+
+`--mmproj` is the entire vision path (the GGUF carries no vision tensors).
+Probes: a solid-color PNG **≥ 256 px** answers its exact color — a tiny
+probe undersupplies image tokens and the model invents ("video"); the
+legacy top-level `images` field is IGNORED, only the OpenAI `image_url`
+content part reaches the model. **New on this payload: the uncensored
+model THINKS more than the censored Swift 1.5** — a 64-token budget was
+eaten entirely by `reasoning_content` (`finish_reason: length`, empty
+`content`); 128 tokens clears it. Token budgets for vision probes must
+leave room for thinking.
+
+## Measured: the 1 × 131,072 MTP boot (2026-10-05, historical)
+
+- **Boot ledger**: `creating MTP draft context against the target model
+  '…/Swift-1.5-Qwen3.8-27B-GSQ-RCO-IQ3_S-mtp.gguf'`, `loaded multimodal
+  model, '…/Swift15-IQ3S/mmproj-Qwen3.8-27B-F16.gguf'`, `n_slots = 1,
+  n_ctx_slot = 131072` — every expected line, no q8→f16 fallback.
+- **VRAM: 19,404 used / 4,721 MiB free** (nvidia-smi `free` field). The
+  4.5 GiB prediction held; f16 KV would have left ~700 MiB, so this free
+  number doubles as the q8_0 proof.
+- **Decode + MTP acceptance** (draft counts from the live `timings`):
+  temp 0.8 → 66.1 tok/s, 182/349 accepted (52.1 %); temp 0 → 66.9 tok/s,
+  166/318; **deployed recipe (temp 1.0/top_p 0.95/top_k 20) → 58.9 tok/s,
+  159/355 (44.8 %)** — temperature buys fluency, costs acceptance and
+  rate. For scale: MiMo Q6_K measured 94.4 tok/s on the same card — this
+  27B at IQ3_S runs ~2/3 the rate with ~3× the capability class.
+- **Vision**: 512×512 solid-red PNG → `Red`, `finish_reason: stop`.
+- **Template gate**: `reasoning_effort: xhigh` → 200 with
+  `reasoning_content` populated; `high` → HTTP 500 (Jinja raise) — the
+  models.yml fold is mandatory, exactly as the MiMo eras found.
+- Streaming completion: 200. No first-request silence window (the
+  halogen-era 45–60 s stall has no analogue here).
+
+## Measured: the 2 × 131,072 no-spec boot (2026-10-06 morning, historical)
+
+- **Boot ledger**: `loaded multimodal model, '…/Swift15-IQ3S/
+  mmproj-Qwen3.8-27B-F16.gguf'` and NO draft-context line (speculation
+  off). `/slots`: **2 slots, n_ctx 131,072 each** (pool 262,144).
+- **VRAM: 22,728 used / 1,397 MiB free** (nvidia-smi `free` field) — the
+  predicted 0.3–1.3 GiB window held; the old 2-3 GiB floor is knowingly
+  violated by Frank's order. After a 512×512 image probe: 1,363 MiB —
+  the vision transients fit.
+- **Decode without the drafter**: 48.0 tok/s (temp 0) / 47.8 (deployed
+  recipe temp 1.0). `draft_n` absent from `timings` (no drafter). The MTP
+  era's 58.9–66.9 tok/s is the honest price of the second context:
+  −18–28 %.
+- **Vision**: 512×512 solid-red PNG → `Red` again on this shape.
+- Deploy: commit 812d9eb, e2e gate 33/33 green on the live fleet.
+
+## Measured: the 2 × 131,072 draft-mtp boot (2026-10-06 evening, historical)
+
+- **Boot ledger**: `creating MTP draft context against the target model
+  '…/Swift-1.5-Qwen3.8-27B-Uncensored-Dynamic-MTP-RCO-IQ3_XXS.gguf'`,
+  `loaded multimodal model, '…/Swift15-UncIQ3XXS/mmproj-BF16.gguf'`,
+  `n_slots = 2, n_ctx_slot = 131072` — every expected line, no q8→f16
+  fallback.
+- **VRAM: 23,636 used / 489 MiB free** (nvidia-smi `free` field) — the
+  draft ctx measured ~2.4 GiB, ~6× the ~0.4 GiB the one-slot era
+  assumed; the lighter quant (−1.56 GiB) plus the morning's no-spec base
+  still fund both slots. After a 512×512 image probe: 489 MiB — the vision
+  transient (34 MiB) fits and recovers.
+- **Decode + MTP acceptance** (draft counts from the live `timings`,
+  512-token essay): temp 0 → **72.9 tok/s**, 333/534 accepted (62.4 %);
+  deployed recipe (temp 1.0/top_p 0.95/top_k 20) → **63.3 tok/s**,
+  305/612 (49.8 %). e2e traffic in the boot window read 0.52–0.67
+  acceptance. Against the no-spec shape (48.0 tok/s): **+31 % at the
+  recipe temp** — the drafter earned its 2.4 GiB.
+- **Vision**: 512×512 solid-red PNG → `Red` at a 128-token budget (64
+  tokens: all thinking, empty content). `prompt_tokens: 321` confirms the
+  image part reached the model.
+- **Prompt cache**: `cached_tokens: 317` of 321 on the second probe — the
+  slot prefix cache is live.
+- Deploy: commit da36fa3, e2e gate 33/33 green on the live fleet.
+
+## Measured: the 2 × 114,688 / ubatch-1024 boot (2026-10-06 late, historical)
+
+- **argv** (from `/proc/<MainPID>/cmdline`): `--ctx-size 229376
+  --parallel 2` (per-slot 114,688), `--batch-size 4096 --ubatch-size
+  1024`, `--spec-type draft-mtp`. Deploys 87721ab (the rename — old
+  `frank-small.service` stopped + disabled first, by hand: `disable` on
+  a relinked unit removes only its symlink) then f2575a1 (the shape);
+  health gate + MainPID check green, e2e 32 passed / 1 flaky-retry.
+- **VRAM: 22,374 used / 1,751 MiB free** (nvidia-smi `free` field) —
+  the per-slot cut (≈1.05 GiB of KV) paid for the ubatch-1024 buffers
+  and netted back ~1.25 GiB against the evening's 489 MiB; back above
+  the morning's 1,397 MiB.
+- **Decode + MTP acceptance** (live `timings`): recipe (temp 1.0) →
+  **58.2 t/s** (430/1005, 42.8 %); temp 0 → **66.9 t/s** (475/873,
+  54.4 %). The 2 × 131,072 boot measured 63.3 / 72.9 — −8–10 %, but
+  acceptance is text-dependent and these prompts differ from that
+  measurement's, so the comparison is CONFOUNDED, not a shape verdict;
+  a clean A/B needs stop + mask (bench policy).
+- **Smoke**: `READY` completion + `reasoning_content` populated; the
+  harness reached :7771 under the new alias (models.yml provider/id
+  `frank-secondary`, config.yml modelRoles rewritten; per-slot
+  `contextWindow: 114688`).
+
+## Measured: ubatch 2048 (2026-10-06 midnight, live)
+
+- **argv**: `--ctx-size 229376 --parallel 2 --batch-size 4096
+  --ubatch-size 2048 --spec-type draft-mtp` (commit d8da7a9, Frank's
+  "set ubatch to 2k"; deploy green, e2e gate 33/33).
+- **VRAM: 23,120 used / 1,005 MiB free at boot** — the 1024 → 2048 rung
+  cost **746 MiB**, ~130 over the 616 MiB the rung measured on
+  2026-10-05 (rung cost is shape-dependent, not a constant). The
+  512×512 solid-red vision probe (`Red`) then took ~240 MiB **and kept
+  it**: settled **763 MiB free** — on this boot the vision encoder's
+  buffers stay resident after the request (the evening boot saw its
+  34 MiB transient return).
+- **Decode (ubatch-invariant knob, same prompt as the 1024 rung)**:
+  recipe → 52.0 t/s (388/1133, 34.2 %) vs 58.2 t/s / 42.8 % at 1024 —
+  acceptance variance at temp 1.0, not a ubatch effect.
+- **Prefill (THE ubatch knob)**: **1,209 t/s @ 3,760 tokens** — first
+  recorded number for this shape; future rungs compare against this
+  (no same-prompt 1024 baseline exists; the law predicts ≈ +4.6 %
+  1024 → 2048).
+- **Headroom left ~0.7–1.0 GiB.** ubatch 4096 (= the logical batch, no
+  micro-splitting) doubles the buffer growth again — not affordable;
+  **2048 is the ceiling for this shape.**
+
+## Client contract
+
+- Reasoning in `reasoning_content` (`--reasoning-format deepseek`);
+  thinking tokens COUNT against `max_tokens`. **This payload thinks more
+  than the censored Swift 1.5** (uncensored finetune) — short `max_tokens`
+  budgets can be eaten entirely by thinking (`finish_reason: length`,
+  empty content); the 32,768 `maxTokens` in models.yml is comfortably
+  above.
+- The embedded chat template gates `reasoning_effort` to
+  `{xhigh, medium, low}` — anything else is a raise/HTTP 500, so
+  `models.yml` `compat.reasoningEffortMap` folds the harness levels onto
+  that set (mandatory, same as every Qwen3.8-template endpoint).
+- Sampling rides the wire (`extraBody`): the publisher recipe temp 1.0 /
+  top_p 0.95 / top_k 20 / min_p 0 (evaluated at ctx 131,072).
+- The RCO-IQ3_XXS tier is a **per-tensor allocation mix** (IQ3_S 97 /
+  BF16 96 / IQ3_XXS 83 / IQ2_S 72 / IQ4_XS 33 / IQ2_XS 32 / IQ2_XXS 30 /
+  Q2_K 28 / Q4_K 19 / Q6_K 8 / IQ1_S 4 / IQ1_M 4) — all stock quant types;
+  stock llama.cpp loads it. Publisher KLD vs Swift BF16: mean 0.002631,
+  90th 0.042136, max 34.49 — RCO is the publisher's "more experimental"
+  family.
+
+## Rollback
+
+**Predecessor shape (same 2 × 114,688, ubatch 1024) is ONE revert
+away** — revert d8da7a9 and redeploy. The 2 × 131,072 / ubatch-512
+era is one revert further (f2575a1). The 2 × 131,072 no-spec
+IQ3_S-mtp shape is one revert further — revert da36fa3 next: the env
+trio then ships `SPEC_TYPE=none`, `CTX_SIZE=262144`, `PARALLEL=2`,
+`UBATCH_SIZE=512` against `/home/dank/models/Swift15-IQ3S/` (the
+IQ3_S-mtp payload, still on disk, measured 48.0 tok/s no-spec). The
+1 × 131,072 MTP shape is two reverts further back (812d9eb's parent):
+the same IQ3_S-mtp file with `SPEC_TYPE=draft-mtp`, `UBATCH_SIZE=1024`,
+`CTX_SIZE=131072`, `PARALLEL=1` (measured 58.9–66.9 tok/s with the head
+drafting).
+
+Three payloads on disk, all fetchable by the same script's git-history
+recipes:
+
+1. **Swift-1.5 IQ3_S-mtp** — the payload live until this cutover:
+   `/home/dank/models/Swift15-IQ3S/`; the IQ3_XXS swap above.
+2. **MiMo-V2.6 Q6_K 2×200,704 f16 KV** — the payload that was LIVE until
+   the 2026-10-05 cutover: `/home/dank/models/MiMo-Q6K/`; revert
+   [[frank-secondary-llama-mimo-q6k-2x200704]] (env trio at commit e5893dc or
+   380aafa's parent), one `MODEL_FILE` + shape edit.
+3. **Ornith-1.5-9B Q8_0** — the committed-but-never-deployed swap:
+   `/home/dank/models/Ornith-Q8/`; revert to 380aafa's tree (ONE native
+   262,144 slot, f16/f16 KV).
+
+## Trail
+
+- The Ornith-1.5-9B Q8_0 swap (77decb5 → 380aafa, earlier 2026-10-05) was
+  committed and pushed but its deploy never completed (blocked at the
+  whole-tree docs gate); the live service was still MiMo Q6_K when Swift
+  cut over. Both Ornith payloads were verified on disk before Swift.
+- Swift header facts (block_count 65, interval 4, KV geometry, nextn) were
+  read from the Hub file by HTTP range + struct parse — no download needed
+  — before the payload was fetched.
+- Cutover 2026-10-05 ~23:26: the radiance-cutover session (parallel)
+  label-committed both sets at Frank's direction — 1919ad7 (frank-primary
+  radiance) then baf33d0 (this swap: trio, seam, runbooks, fleet docs).
+  Deploy ran `deploy.sh dankclaw --service frank-secondary`; the service
+  dispatch and health gate went green; the first e2e run failed ONE spec
+  (12-crawl, `net::ERR_ABORTED` on `www.franksdank.cloud` — a transient,
+  unrelated to frank-secondary), green on rerun.
+- The unit's `Description=` (MiMo era) was stale at cutover time and was
+  updated in-repo afterward — the relinked unit picks it up at the next
+  deploy/restart, not retroactively.
+- 2026-10-06 (Frank): drop MTP, skip ngram, ubatch 512, TWO 128k contexts
+  (812d9eb). The tight fit landed: 1,397 MiB free, vision `Red`, decode
+  48.0 t/s no-spec. Model of the trade: the drafter's ~0.4 GiB + ubatch
+  ~0.3 GiB bought the second 4,352 MiB KV slot.
+- 2026-10-06 (Frank, evening): swap to the UNCELERSED Swift-1.5 at
+  RCO-IQ3_XXS, MTP back on (da36fa3). The lighter quant (−1.56 GiB) funds
+  the draft ctx (~2.4 GiB measured — the drafter's KV scales with the
+  target's full 262,144 context, not the 3-token proposal depth) plus the
+  second 131,072 slot: 489 MiB free, vision `Red` at a 128-token budget,
+  decode 63.3 t/s recipe / 72.9 temp 0 (48.0 no-spec).
+- 2026-10-06 (late, Frank): the endpoint renamed frank-small →
+  **frank-secondary** (87721ab — scripts, unit, seam, runbooks,
+  identifiers; historical references to the OLD FreeToken/CUDA
+  frank-small, and to the Dankserver service that became frank-medium,
+  stay frank-small) and the per-slot window cut 131,072 → 114,688 to
+  fund ubatch 512 → 1024 (f2575a1). Live unit transition done by hand
+  before the deploy: `systemctl stop frank-small` + `systemctl disable
+  frank-small`, then `deploy.sh dankclaw --service frank-secondary`.
+  1,751 MiB free; decode 58.2 t/s recipe / 66.9 temp 0; models.yml +
+  config.yml retargeted.
+- 2026-10-06 (midnight, Frank): "set ubatch to 2k" → `UBATCH_SIZE=2048`
+  (d8da7a9). Cost 746 MiB: 23,120 used / 1,005 free at boot, settling
+  **763 MiB free after a vision probe** (encoder buffers stay resident
+  on this boot). Decode 52.0 t/s recipe on the shared prompt (58.2 at
+  the 1024 rung — acceptance noise, decode is ubatch-invariant);
+  prefill 1,209 t/s @ 3,760 tokens (first recorded baseline). 2048 is
+  the ceiling for this shape; 4096 would not fit.

serverAppConfigs-docs/LLM/frank-secondary-qwen38-gsq-rco.md

@@ -0,0 +1,15 @@
+---
+type: reference
+title: "frank-secondary on Dankclaw — llama.cpp, Qwen3.8-27B GSQ-RCO IQ3_S"
+description: "The RETIRED frank-secondary runbook (2026-09-16 → 09-17, rollback target) — mainline llama.cpp on Dankclaw's RTX 3090 serving ISTA-DASLab Qwen3.8-27B-GSQ-RCO at IQ3_S with the model's own in-file MTP head and the BF16 vision projector: identity, KV arithmetic, deployed flags, the MTP contract, the client contract, and what to check when it misbehaves."
+tags:
+  - llm
+  - frank-secondary
+  - dankclaw
+  - llama.cpp
+  - rtx3090
+  - qwen35
+  - mtp
+  - vision
+timestamp: 2026-09-16
+---

serverAppConfigs-docs/LLM/frank-secondary-sglang-mimo-v26.md

@@ -1,10 +1,10 @@
 ---
 type: reference
-title: "frank-small on Dankclaw — SGLang, MiMo-V2.6-Distill-Qwen-9B INT8 W8A16 (RETIRED)"
-description: "The RETIRED frank-small runbook (2026-10-01, hours) — SGLang in a self-contained python3.12 venv serving leteschlead MiMo-V2.6-Distill-Qwen-9B-INT8-W8A16-AutoRound (qwen3_5 hybrid, 9B dense, in-checkpoint MTP + vision) on the RTX 3090: identity, payload, KV arithmetic, the NEXTN spec-decode ghost, the CUDA 13 nvcc toolchain the JIT needs, and the measured numbers kept for the record. Replaced by [[frank-small-llama-mimo-q6k]] the same night."
+title: "frank-secondary on Dankclaw — SGLang, MiMo-V2.6-Distill-Qwen-9B INT8 W8A16 (RETIRED)"
+description: "The RETIRED frank-secondary runbook (2026-10-01, hours) — SGLang in a self-contained python3.12 venv serving leteschlead MiMo-V2.6-Distill-Qwen-9B-INT8-W8A16-AutoRound (qwen3_5 hybrid, 9B dense, in-checkpoint MTP + vision) on the RTX 3090: identity, payload, KV arithmetic, the NEXTN spec-decode ghost, the CUDA 13 nvcc toolchain the JIT needs, and the measured numbers kept for the record. Replaced by [[frank-secondary-llama-mimo-q6k]] the same night."
 tags:
   - llm
-  - frank-small
+  - frank-secondary
   - dankclaw
   - sglang
   - rtx3090
@@ -15,10 +15,10 @@ tags:
 timestamp: 2026-10-01
 ---
 
-# frank-small — SGLang MiMo-V2.6-Distill-Qwen-9B INT8 W8A16 (NEXTN MTP)
+# frank-secondary — SGLang MiMo-V2.6-Distill-Qwen-9B INT8 W8A16 (NEXTN MTP)
 
 > **RETIRED 2026-10-01, hours after cutover.** Replaced by
-> [[frank-small-llama-mimo-q6k]] — llama.cpp CUDA on the MiMo Q6_K GGUF,
+> [[frank-secondary-llama-mimo-q6k]] — llama.cpp CUDA on the MiMo Q6_K GGUF,
 > chosen for the asymmetric KV (f16 K + q8_0 V) SGLang cannot express.
 > The venv (`/home/dank/GitPackages/sglang-cu128/`) stays in place; the INT8
 > payload was staged to
@@ -28,13 +28,13 @@ timestamp: 2026-10-01
 
 ## Identity
 
-- **Endpoint:** `frank-small` at `192.168.0.143:7771` (Dankclaw).
+- **Endpoint:** `frank-secondary` at `192.168.0.143:7771` (Dankclaw).
 - **Runtime:** SGLang (latest) in a self-contained python3.12 venv at
   `/home/dank/GitPackages/sglang-cu128/` (torch 2.13.0+cu130,
   transformers 5.12.1, flashinfer 0.6.18, sgl-kernel). No container — a root
-  systemd unit `scripts/frank-small.service` (`User=dank`,
-  `LimitMEMLOCK=infinity`) launched by `scripts/serve-frank-small.sh` with
-  tunables from `scripts/frank-small.env`.
+  systemd unit `scripts/frank-secondary.service` (`User=dank`,
+  `LimitMEMLOCK=infinity`) launched by `scripts/serve-frank-secondary.sh` with
+  tunables from `scripts/frank-secondary.env`.
 - **Model:** `letechlead/MiMo-V2.6-Distill-Qwen-9B-INT8-W8A16-AutoRound`
   (9B dense, 12.0 GiB, INT8 W8A16 AutoRound group-128 symmetric).
 
@@ -55,7 +55,7 @@ timestamp: 2026-10-01
   the two-way concurrency contract. **That compaction arithmetic is
   specific to a 262,144-token slot**: the ACTIVE shape runs three slots of
   180,224, which 183,501 tokens already exceeds — see *What three slots
-  cost* in [[frank-small-llama-mimo-q5km]].
+  cost* in [[frank-secondary-llama-mimo-q5km]].
 - **KV:** `fp8_e4m3` — 16 KiB/token (8 attn layers x 2 x 4 heads x 256
   dim x 1 B), pool 9.31 GiB.
 - **The memory shape (the 4.5 GiB the naive math misses):** the DeltaNet
@@ -126,15 +126,15 @@ change lands.
 
 Eight safetensors shards (12,891,718,440 B total) + tokenizer and config
 files under `/home/dank/models/MiMo-V2.6-Distill-Qwen-9B-INT8-W8A16-AutoRound/`,
-all LFS sha256-verified by `old/download-frank-small-sglang-int8.sh`
+all LFS sha256-verified by `old/download-frank-secondary-sglang-int8.sh`
 (`--verify-only` re-checks) — the script moved to `old/` when the endpoint
 cut over to llama.cpp + GGUF on 2026-10-01 night, and
-`scripts/download-frank-small.sh` now fetches the GGUF quant instead. The
+`scripts/download-frank-secondary.sh` now fetches the GGUF quant instead. The
 serve script refuses to start with a missing shard.
 
 ## Client contract
 
-- **Alias:** `frank-small` (the `id` at `/v1/models`).
+- **Alias:** `frank-secondary` (the `id` at `/v1/models`).
 - **Reasoning:** extracted to **`reasoning_content`** (live-verified
   2026-10-01 — this SGLang build's qwen3 parser uses the same field name
   as llama.cpp; models.yml sets `reasoningContentField:
@@ -166,8 +166,8 @@ serve script refuses to start with a missing shard.
 
 ## Rollback
 
-[[frank-small-llama-mimo-q5km]] is the live shape two steps forward; the
-practical rollback today is [[frank-small-llama-mimo-q6k]] (payload at
+[[frank-secondary-llama-mimo-q5km]] is the live shape two steps forward; the
+practical rollback today is [[frank-secondary-llama-mimo-q6k]] (payload at
 `/home/dank/models/MiMo-Q6K/`). This era's own rollback, the llama.cpp
 Gemma4-12B QAT payload, was staged to
 `/home/dank/.AGENTTRASH/20261002-retired-llm-payloads/Gemma4-12B-QAT/` on
@@ -177,4 +177,4 @@ Gemma4-12B QAT payload, was staged to
 
 - Model: https://huggingface.co/letechlead/MiMo-V2.6-Distill-Qwen-9B-INT8-W8A16-AutoRound
 - SGLang: https://github.com/sgl-project/sglang
-- Serve script: `scripts/serve-frank-small.sh`; tunables: `scripts/frank-small.env`
+- Serve script: `scripts/serve-frank-secondary.sh`; tunables: `scripts/frank-secondary.env`