docs(openwebui): frank-dragoon enabled in the picker beside frank-zealot

Live DB trio applied on Dankclaw (enable idx 0, model row + global
access_grant, restart); verified /api/models as admin and role=user,
chat answers pong. Battery rewritten to gate on /api/models —
/api/v1/models/list lists only workspace models on 0.6.x.

AGENTS.md

@@ -12,8 +12,9 @@ merge absorbed the former `vllmScripts` repo, history preserved).
   Calibre-Web, Jellyfin, Plex) plus the `frank-primary` LLM service.
 - **Dankclaw** (192.168.0.143) — AI/search/dev/observability stack (SearXNG
   via Gluetun, ComfyUI, OpenWebUI, Hermes, Gitea, Grafana, Prometheus, Loki,
-  Alloy, Pi-hole) plus TWO LLM services: `frank-large` (halogen container on
-  the gfx1151 iGPU) and `frank-small` (llama.cpp CUDA on the RTX 3090). Caddy on
+  Alloy, Pi-hole) plus TWO LLM services: `frank-large` (flash-server
+  container on the gfx1151 iGPU) and `frank-secondary` (llama.cpp CUDA on the
+  RTX 3090). Caddy on
   Dankclaw reverse-proxies both hosts.
 - **FrankBook-Pro** (192.168.0.112) — macOS mini running the *arr stack
   (Prowlarr, Sonarr, Radarr, Lidarr, Readarr, qBittorrent) as quadlets inside
@@ -24,9 +25,9 @@ merge absorbed the former `vllmScripts` repo, history preserved).
 
 | Target | Service | Runtime | Model | Port |
 |--------|---------|---------|-------|------|
-| Dankserver | `frank-primary` | tcclaviger RDNA4 vLLM fork (`docker.io/tcclaviger/vllm@sha256:da4029666…`, digest-pinned, plain `podman run`, root systemd unit) | `orcarouter/Qwen3.8-Flash-Next-Uncensored-FP8` (~172.8 GiB, 131 shards), TP=2, expert offload automatic (fork plans it), prefetch OFF, **fp8 KV** (`--kv-cache-dtype fp8`), **no speculative decode**, MNBT 2048 (arena-free: the registry prices overhead flat up to its nbt 4096 anchor), MoE gather 32×64, **`--expert-precise-memory`** (fitted 5.25 GiB overhead, not the 7.60 GiB default), **`VLLM_CUDAGRAPH_MEM_BASE_MB=0` / `_PER_GRAPH_MB=12`** (the plan's CUDA-graph term priced at the measured capture — 0.31 GiB then a NEGATIVE −0.06 GiB delta, so the 256 MiB analytic base was fiction → graph term 0.07 GiB), **`GPU_MAX_HW_QUEUES=2`**, **mounted cache dir** at `/root/.cache` (the image ships no `/root/.cache` at all; unmounted, every boot re-pays `torch.compile took 181.20 s`), **3 slots × 200704 ctx** at KV_BYTES 3,166,500,000/rank (~400,700 KV tokens at the engine's 7,891.8 B/token here) — the **pool, not the slot count, caps resident conversations** (measured live: `GPU KV cache size: 401,408 tokens, Maximum concurrency for 200,704 tokens per request: 2.00×`; the 3.15e9 pool below it measured 398,658/1.99×), and shorter requests use the spare blocks. **`--max-num-seqs` must stay 3: SEQS=2 cannot boot this model** — torch.fx `ConstraintViolationError` on `query_start_loc`, measured at BOTH MNBT 1024 and 2048. **util stays 0.982 BY LAW**: at `le=1` the worker init refuses `budget == total` with `ValueError: Free memory on device cuda:N (31.29/31.86 GiB)` on both ranks before a single weight loads — measured crash loop 2026-10-03. Arena at shape G: **17.77 GiB / 7,762 slots measured**, KV 401,408 tokens / 2.00× concurrency, 0.89 GiB of real free VRAM per card held through a live 4,096-token generation (the ~3.5 GiB of driver-reported free VRAM at shape E was estimator slack — the registry's 9190 B/token KV reserve vs the real 7,892, and the 5.25 GiB overhead vs far less — NOT allocatable), and the client contract wired: `--reasoning-parser …
-| Dankclaw | `frank-large` | **halogen container** (`ghcr.io/peonist-ai/halogen@sha256:dc0a39a0…`, digest-pinned, plain `podman run` via `scripts/serve-frank-large.sh`, root systemd unit) | `peonist-ai/halogen-qwen3.8-27b` **`qwen3.8-27b-p1w4d-d2.hgn`** (35,865,565,184 B; the `.hgn` format loads in NO other engine — llama.cpp, vLLM and transformers all refuse it), **DFlash2 spec decode (`HALOGEN_DRAFTER=2`)**, **1 slot × 200,700 ctx** — `HALOGEN_KV_SLOTS=1` is load-bearing: batching XOR speculation, slots>1 silently disables the drafter (measured: 8 slots at serial ~6 t/s/stream), d2 KV (2× fp8, 64 KiB/token → 12.2 GiB pool, engine sizes it and prints a boot ledger naming any clamp), **TEXT-ONLY** (the engine ignores the vision encoder — vision is frank-small's job), **censored base** (the ROCmFPX AGENT rollback is the refusal escape hatch), **no server-side sampler: absent temperature is GREEDY**, so the card's thinking recipe (temp 1.0 / top_p 0.95) rides the wire in `models.yml`. Publisher card 31.7 t/s mean across eight text workloads vs the retired ROCmFPX endpoint's measured 27.6–28.1. Prompt over the slot AND `max_tokens` over `HALOGEN_MAX_TOKENS_CAP` are both 400s naming the limit, never truncations. Serve passes NO `--ulimit nofile` (rootless runc cannot raise Dankclaw's 524288 hard limit — see Key Lessons). Cutover 2026-10-05|7770|
-| Dankclaw | `frank-small` | llama.cpp master, CUDA sm_86 build (root systemd unit, relinked) | bartowski/MiMo-V2.6-Distill-Qwen-9B-GGUF **Q6_K** (9B dense qwen3_5 hybrid, 7.26 GiB GGUF, no MTP weights so no spec decode) + the repo's f16 mmproj for **vision**, on the RTX 3090, **2 × 200,704** (pool 401,408 — NOT the native window) with **fully unquantized KV (f16 K + f16 V)** under flash attention, **batch 4096 / ubatch 2048** (doubled 2026-10-05: measured +4.6 % prefill at 8,928 tokens for 616 MiB), 21,846 used / **2,279 free** of 24,576 MiB, 94.4 tok/s decode. Resized 2026-10-03 (3 × 180,224 + q8_0 V → 2 × 200,704 + f16 V → + Q6_K rung: the freed slots paid for exact values AND the higher quant; the rung costs a measured −10 % decode — weight-bandwidth-bound); rollback `MODEL_FILE=…/MiMo-Q5KM/…Q5_K_M.gguf` (buys the 10 % back) or `CTX_SIZE=540672 PARALLEL=3 CACHE_TYPE_V=q8_0` (Q5 only) | 7771 |
+| Dankserver | `frank-primary` | **radiance** (StillDeadcode, **pin 1.0.13** digest-pinned `stilldeadcode/radiance@sha256:4c3f484564…`, plain `podman run`, root systemd unit) | ukisai/Swift1.5-Qwen3.8-Flash-Next BF16 (335.3 GiB, 397/397 sha256-verified) converted by `scripts/convert-swift15-w4dn8.sh` to `qwen4exp-swift15-w4dn8-i8-hc8m.rad` (**116,416 MiB**): **all 49,152 experts GPTQ w4** (10M-token calibration on engine 1.2.0) + **24 whole bf16-protected experts/layer** (energy-share law), int8 trunk/lm_head, fp8 HC, PLE n-gram fp8 fixed-scale, on **BOTH R9700s TP=2** (`--tp-wire wht6`), experts in a **40 GiB/rank pinned host pool** + VRAM cache (`--placement expert_tiered`, explicit `--vram-weights-mib 26071 --vram-kv-mib 3543`), **47.7 GiB n-gram copy pinned to host RAM** (`--ngram-placement ram`; root unit + `LimitMEMLOCK=infinity`), **fp8 KV** (inherited; pool 3.46 GiB), **MTP spec decode depth 2**, `--max-num-seqs 3`, `--max-model-len 200000`, MNBT 2048, Froggeric v22.5 template. KLD-gated (mean KL 0.0131, top-1 97.92 % vs BF16); decode ~190 t/s live (gate bar 30.8). Cutover 2026-10-08 from fp8-native (payload on disk = rollback). | 7770 |
+| Dankclaw | `frank-large` | **halogen-flash-server 0.16.2** (`ghcr.io/peonist-ai/halogen-flash-server@sha256:0c61bf84…`, digest-pinned, plain `podman run` via `scripts/serve-frank-large.sh`, root systemd unit) | Quat3rnion/halogen-swift1.5-qwen3.8-flash-next-v2-abliterated **`qwen38-flash-next-v2-swift15-abliterated.hgn`** (66,687,678,464 B = compacted r2 of the card's 89,775,868,032 B original, ~62 GiB resident; the v2 `.hgn` format loads in NO other engine), **+ REQUIRED `qwen38-flash-next-ngram.hgn`** (51,200,246,144 B, peonist @5cc17cea — read beside the checkpoint) **+ vision tower** (897,916,416 B, @e053f488 — **vision ON**, `image_url` parts only, ≥256 px probes), **MTP spec decode ALWAYS-ON (`HALOGEN_SPEC_ADAPT=0` — the adaptive default disables the drafter per request and HALVES decode)**, MTP head INSIDE the checkpoint (`HALOGEN_MTP_HEAD` unset; no sidecar fetched), **2 slots × 262,144 ctx** (KV pool 524,288 positions ≈ 14.0 GiB at ~28 KiB/position; the 27B slots>1-disables-drafting law does NOT carry over — batching and speculation coexist, the card's quickstart ships 2 with the head), harness window 217,088 = ceiling − 45k estimate headroom, **server-side sampler 1.0/0.95/20** (card recipe; models.yml extraBody matches), coupled cap/timeout 65536/7200, prompt cache ON by default (warm 100k turn ≈ 2 s vs ≈ 88 s cold)|7770|
+| Dankclaw | `frank-secondary` | llama.cpp master (0.5.0-dev, commit `5fc4f3c`), CUDA sm_86 build (root systemd unit, relinked) | HauhauCS/Qwen3.5-9B-Uncensored-HauhauCS-Aggressive **Q6_K** (7,359,259,008 B; dense `qwen35` hybrid: 32 layers, 8 full-attention + 24 Gated DeltaNet, 4 KV heads × 256 dim, vocab 248,320, native 262,144, **no MTP tensors in the file**) + the publisher's own **BF16 mmproj** (921,704,448 B) for **vision**, on the RTX 3090, **TWO 200,704-token slots** (`--ctx-size 401408 --parallel 2`), **exact f16 K + f16 V** under flash attention (32 KiB/token over the 8 full layers = 12.25 GiB pool), batch 4096 / **ubatch 2048** (measured CUDA optimum), `--log-verbosity 4`, **MTP OFF** (forced: no `nextn` tensors exist). Priced from the MiMo Q6_K boots (same geometry, same card, same shape): 21,276/2,849 at ubatch 1024, 21,846/2,279 at 2048. Cutover 2026-10-07 (late) from Gemma4-26B-A4B Q4_K_M 1×200,704. All three endpoints route harness **compaction** here. | 7771 |
 
 Retired endpoints (`frank-npu`, and every prior runtime) are in
 the [runbook index](#runbook-index). The full swap history lives in
@@ -40,7 +41,11 @@ the [runbook index](#runbook-index). The full swap history lives in
 - `scripts/` — `deploy.sh` / `deploy-plan.py` / `deploy-remote.sh`, LLM serve
   scripts, systemd units, env tunables, bench harnesses
 - `old/` — retired LLM scripts (reference only)
-- `serverAppConfigs-docs/` — knowledge base (Obsidian vault, OKF format)
+- `serverAppConfigs-docs/` — knowledge base (Obsidian vault, OKF format).
+  The managed skill library is distilled there one page per skill-cluster:
+  `LLM/index.md` → *Cross-Cutting References*, plus `Operations/index.md`,
+  `Agent/index.md`, and the Infrastructure/Dankclaw/FrankBook-Pro service
+  pages (`serverAppConfigs-docs/log.md` carries the wave plan and results)
 - `tests/` — bats suite + Playwright e2e
 
 ## Git Workflow
@@ -68,7 +73,7 @@ All edits flow through git. Never edit files on a remote machine directly.
 The planner maps changed paths to services; the remote runner restarts only
 the selected service and records its revision outside the checkout.
 Documentation-only changes are a no-op. LLM services are selected only by
-their own files — every endpoint (`frank-large`/`frank-small`/`frank-primary`)
+their own files — every endpoint (`frank-large`/`frank-secondary`/`frank-primary`)
 by its unit/env/serve trio in `scripts/` — so
 unrelated
 container changes cannot interrupt inference. Set `FRANK_SUDO_PW` for headless
@@ -224,13 +229,16 @@ archive.
 **STOP the production service BEFORE benchmarking** — these servers serve
 real users, and a bench beside a live service starves the GPU and corrupts
 results. Use the runtime's own harness: `llama-bench` for the llama.cpp
-endpoints; a live llama.cpp server also reports honest per-request numbers in the `/completion` `timings` object (prefill/decode rates, `draft_n`, `draft_n_accepted`), which is how frank-large and frank-small were measured. Rolling your own bench
+endpoints; a live llama.cpp server also reports honest per-request numbers in the `/completion` `timings` object (prefill/decode rates, `draft_n`, `draft_n_accepted`), which is how frank-large and frank-secondary were measured. Rolling your own bench
 requires asking first.
 Scratch ports are not used on LLM hosts — a
 test process cannot share the single GPU anyway; validate against the service
 itself.
 
-(A gufo-era law; frank-large ran ROCmFPX llama.cpp 2026-10-02 → 10-05 and is the halogen container since, but the rollback tags keep it live.) **`gufo bench` cannot rank speculative-decode settings, and will invert the ranking.**
+(A gufo-era law; frank-large ran ROCmFPX llama.cpp 2026-10-02 → 10-05, the
+halogen 27B container 2026-10-05 → 10-06, and is flash-server since, but the
+rollback tags keep it live.) **`gufo bench` cannot rank speculative-decode
+settings, and will invert the ranking.**
 Measured 2026-09-28 on frank-large at depth 32768: its corpus
 reports `acceptance=0.933 ± 0.000` with an **identical trace sha256** across
 draft widths, i.e. the same highly-predictable text every run, and 93 % of
@@ -258,9 +266,9 @@ merely wedged kernel usually still answers ARP.
 1. Stop and mask (self-healing would restart it):
    - Dankclaw `frank-large` (root unit, halogen podman on the iGPU):
      `sudo systemctl stop frank-large && sudo systemctl mask frank-large`
-   - Dankclaw `frank-small` (root unit, llama.cpp on the 3090):
-     `sudo systemctl stop frank-small && sudo systemctl mask frank-small`
-   - Dankserver `frank-primary` (root unit, tcclaviger vLLM on the 2x R9700):
+   - Dankclaw `frank-secondary` (root unit, llama.cpp on the 3090):
+     `sudo systemctl stop frank-secondary && sudo systemctl mask frank-secondary`
+  - Dankserver `frank-primary` (root unit, radiance on both R9700s, TP=2):
      `sudo systemctl stop frank-primary && sudo systemctl mask frank-primary`
 2. Bench.
    NOTE: `systemctl mask` FAILS on a relinked unit (the unit file in
@@ -278,9 +286,17 @@ deletes the unit file from `/etc/systemd/system/`, not just the symlink.
 The harness reaches the fleet through `.omp/agent/models.yml` in the
 operator's home — `/home/blank/.omp/agent/models.yml` on the dev box, which is
 where the harness runs; there is no copy on the servers. Each provider is one
-HOST:PORT; each model `id` is the served alias: `frank-primary` →
-`192.168.0.235:7770`, `frank-large` → `192.168.0.143:7770`, `frank-small` →
-`modelRoles`.
+HOST:PORT; each model `id` is the served alias. **Official model ids are
+the StarCraft unit names (2026-10-08) and since 2026-10-09 they are also the
+ids on the wire:** `frank-archon` → `192.168.0.235:7770`
+(`FRANK_PRIMARY_SERVED_NAME`), `frank-dragoon` → `192.168.0.143:7770`
+(`HALOGEN_MODEL_ID`), `frank-zealot` → `192.168.0.143:7771` (`ALIAS`). The
+SERVICES, units, scripts, env files and runbooks keep the old names
+(`frank-primary`, `frank-large`, `frank-secondary`); only the served alias,
+the `models.yml`/`config.yml` ids, the homepage cards, the OpenWebUI grants
+and the Hermes routing were renamed. Verify a rename on `/v1/models`, never
+on a chat response: radiance echoes the **requested** model name, so a 200
+proves nothing about the served id (llama.cpp and halogen ignore it).
 
 Resolution rules: the default model resolves as a `defaultProvider` +
 `defaultModel` pair (the provider must serve the model); a subagent override
@@ -292,16 +308,21 @@ When renaming a service, retiring an endpoint, or moving a model: update the
 provider block, model `id`, `contextWindow`, `maxTokens`, `input` in
 `models.yml`, then fix every `modelRoles` entry that named the old endpoint —
 a role pointing at a provider the file does not define fails at call time. As
-of 2026-10-05 **two of the three endpoints accept images** (`input: [text,
-image]`) — `frank-large` went TEXT-ONLY with the halogen swap (the engine
-ignores the vision encoder; the BF16-mmproj era ended 2026-10-05) — while
-`frank-primary` takes them through a BF16 mmproj loaded explicitly (this build never
-autoloads projectors), and `frank-small` through
-`mmproj-MiMo-V2.6-Distill-Qwen-9B-f16.gguf` passed as `--mmproj` — the MiMo
-GGUF itself carries no vision tensors. `frank-small`'s `contextWindow` is
-**200704** (resliced from 180224 on 2026-10-03), the per-slot window, not the
-model's native 262144: the harness must compact before llama.cpp answers a
-longer prompt with a 400. Note also
+of 2026-10-06 **all three endpoints accept images** (`input: [text,
+image]`) — `frank-large` returned to vision with the flash-server v2 swap
+(the v2 vision tower loads beside the checkpoint; the text-only halogen
+27B window closed 2026-10-06) — while
+`frank-primary` takes them natively (radiance runs the checkpoint's
+27-layer Qwen-VL tower — patching, pad-token expansion and 3-part rotary
+are the engine core; `--mm-max-patches` available),
+and `frank-secondary` through
+the publisher's own BF16 mmproj passed as `--mmproj` — the model is natively
+multimodal but the Q6_K GGUF carries no vision tensors (same arrangement the
+Gemma4 era used). `frank-secondary`'s `contextWindow` is
+**200704** — the PER-SLOT window (`--ctx-size 401408 --parallel 2` since the
+2026-10-07 Qwen3.5-9B swap), not the pool and not the model's native 262144:
+the harness must compact before llama.cpp answers a longer prompt with a 400.
+Note also
 that
 the reasoning channel differs by runtime: llama.cpp AND SGLang (qwen3
 parser, live-verified) emit `reasoning_content`, vLLM emits `reasoning` —
@@ -310,255 +331,249 @@ There is **no audio**
 input anywhere in the fleet: that capability left with the FLM NPU
 endpoint on 2026-09-11, returned 2026-09-17 with the Gemma4 projector's
 audio encoder, and left again with the Gemma4 retirement on 2026-10-01
-(the MiMo tower is vision-only).
+(neither the MiMo nor the Qwen3.5 tower carries audio).
 
-### frank-large — halogen container, Qwen3.8-27B + DFlash2 (Dankclaw, ACTIVE)
+### frank-large — halogen-flash-server 0.16.2, Qwen3.8-Flash-Next v2 Swift1.5 ABLITERATED (Dankclaw, ACTIVE)
 
-The **halogen engine** (`ghcr.io/peonist-ai/halogen@sha256:dc0a39a0…`,
-digest-pinned, published tag 0.1.4) as a podman container driven by
-`scripts/serve-frank-large.sh` under a **root systemd unit**
-(`scripts/frank-large.service`, `User=dank`, `Restart=always`, `ExecStop=podman
-stop -t 120 frank-large-halogen`, the script `exec podman wait` so the restart
-policy is real) on port 7770 from the Strix Halo iGPU. Engine stdout goes to
-`/home/dank/logs/frank_large.log` — **the boot ledger (KV pool, drafter state,
-any clamp) lives there, not in the journal**. Serving
-`peonist-ai/halogen-qwen3.8-27b`: `qwen3.8-27b-p1w4d-d2.hgn` (35,865,565,184 B
-≈ 33.4 GiB, sha256 `274c3fc7…`, 1352 tensors; p1 weights / w4 activations /
-d2 KV) + the flat `tokenizer/` directory (5 files). Cutover 2026-10-05 from the ROCmFPX llama.cpp fork
-([[frank-large-rocmfpx-q6agent-uncensored-mtp]], measured 27.6–28.1 t/s prose),
-chosen for the publisher's 31.7 t/s DFlash2 mean.
+The **flash-server engine** (`ghcr.io/peonist-ai/halogen-flash-server@sha256:0c61bf84…`,
+digest-pinned, published tag 0.16.2, 3.6 GB image) as a podman container
+driven by `scripts/serve-frank-large.sh` under a **root systemd unit**
+(`scripts/frank-large.service`, `User=dank`, `Restart=always`,
+`ExecStop=podman stop -t 60 frank-large-halogen`, the script runs `podman
+run` FOREGROUND — the container's exit code is the service's exit so
+`Restart=always` stays real, and the engine ledger lands on disk) on port
+7770 from the Strix Halo iGPU. Engine stdout goes to
+`/home/dank/logs/frank_large.log` — **the boot ledger (weights resident, KV
+pool, MTP head, prompt cache, any clamp) lives there, not in the journal**.
+Cutover 2026-10-06 from the halogen 27B-dense era
+([[frank-large-halogen27b-dflash2]], retired the same window).
 
-**The format is the lock.** `.hgn` loads in nothing else — the runtime is
-chosen by the FILE, as ROCmFPX's tensor types were before it. This is NOT the
-2026-09 `halogen-flash-server` image (different repo, Flash-Next w4b era); the
-halogen-server repo restarted versioning at 0.1.x.
+Serving `Quat3rnion/halogen-swift1.5-qwen3.8-flash-next-v2-abliterated`:
+`qwen38-flash-next-v2-swift15-abliterated.hgn` (66,687,678,464 B ≈ 62.1
+GiB, sha256 `9c880b8b…` — a compacted r2 of the card's 89,775,868,032 B
+append-only original; the abliteration workflow appends and the repo's
+tooling drops dead padding, live tensors byte-identical) + the **REQUIRED**
+n-gram lookup table `qwen38-flash-next-ngram.hgn` (51,200,246,144 B,
+peonist @5cc17cea) + the vision tower `qwen38-flash-next-vision.hgn`
+(897,916,416 B, @e053f488) + the flat `tokenizer/` directory (6 files,
+fetched fresh from peonist). Cutover 2026-10-06 from the halogen
+27B-dense shape, chosen by Frank for the ABLITERATED base.
 
-**Two product regressions, both accepted by Frank before deploy:** the
-endpoint is **text-only** (README honest limits: the vision encoder is
-ignored — frank-small owns images now) and the checkpoint is a finetune of the
-**censored** `Qwen/Qwen3.8-27B` base (refusals can return; the rollback is one
-commit-revert away with the AGENT payload still on disk).
+**The format is the lock.** The v2 `.hgn` loads ONLY in the
+flash-server line — the 27B-era halogen 0.1.4 container refuses it, so the
+engine moved with the model. The MTP head is INSIDE the checkpoint
+(`HALOGEN_MTP_HEAD` stays unset; the separate `qwen38-flash-next-mtp.hgn`
+sidecar is deliberately not fetched).
 
-**Engine laws that shape the config** (all in `scripts/frank-large.env` with
-evidence): `HALOGEN_KV_SLOTS=1` — batch XOR speculation, slots>1 turns the
-drafter off silently. `HALOGEN_SLOT_CTX=200700` — the per-conversation HARD
-ceiling (400 naming the limit, never truncation); `models.yml`
-`contextWindow: 200700` must match or the harness discovers the wall as a 400.
-`HALOGEN_MAX_TOKENS_CAP=65536` + `HALOGEN_QUEUE_TIMEOUT=7200` are a coupled
-pair; the harness `maxTokens: 57344` stays under the cap (over-cap is a 400
-naming the limit — README, never a 413). Thinking tokens COUNT against the
-budget: exhausted mid-thought is `finish_reason: "length"` with empty
-`content` and the reasoning in `reasoning_content`.
-`HALOGEN_DRAFTER=2` = DFlash2. **No server-side sampler**: an absent
-temperature is greedy, so the thinking recipe exists only in `models.yml`
-`extraBody`. `response_format` is unsupported (no constrained decoding).
-Thinking lands in **`reasoning_content`** (harness field unchanged); the same
-Qwen3.8 template rejects `reasoning_effort` outside `xhigh|medium|low`, so the
-`models.yml` fold stays mandatory.
+**Product facts vs the 27B era, both regressions FIXED:** the base is
+**abliterated** (the 27B was a censored-base finetune; refusals were an
+accepted trade there) and **vision is ON** (the 27B engine silently
+ignored its encoder — frank-secondary owned images for exactly one day at this
+endpoint).
 
-Fetch: `scripts/download-frank-large-halogen.sh` into
-`/home/dank/models/halogen-qwen3.8-27b/` (`-f` + byte count + LFS sha256 +
-`--verify-only`, no token on the command line; resume goes against the
-RESOLVED CDN URL because curl drops `Range` across HF's 302). The deployer
-and serve script both refuse a missing tokenizer file or **any symlink** —
-an HF cache snapshot symlinks each entry into `blobs/` and those links
-dangle inside the container mount.
+**Engine laws that shape the config** (all in `scripts/frank-large.env`
+with evidence): `HALOGEN_SPEC_ADAPT=0` is MANDATORY — the default adaptive
+policy disables the MTP head for whole requests after low acceptance and
+HALVES decode; the head stays on and temp-1.0 traffic pays some acceptance
+for it (accepted). `HALOGEN_CTX=262144` is the PER-SLOT hard ceiling (400
+naming the limit, never truncation) — RAISED from 200192 on 2026-10-07
+under the 400-zone law: the harness clamps max_tokens to
+`min(45056, contextWindow − est − 4096)`, its chars/4 est UNDER-counts
+the rendered prompt by ~3 % (measured: est 152,938 → server 157,388,
+request 43,158 vs room 42,804), so a harness window == engine ceiling
+made every request over ~146k est a guaranteed 400. `models.yml`
+`contextWindow: 217088` leaves the 45k gap that absorbs the estimate
+error (worst-case request ≈ 219k ≤ 262144); compaction fires at 85 % of
+the harness window = 184,524 est ≈ 190k measured (Frank's 180k target).
+`HALOGEN_KV_SLOTS=2` is LEGAL here — the 27B "slots>1 disables the drafter"
+law does NOT carry over; batching and speculation coexist (the card's
+quickstart ships 2 slots with the head), so 2 is concurrent capacity.
+`HALOGEN_KV_POOL_POSITIONS=524288` (explicit 2 × ctx; ~28 KiB/position ⇒
+14.0 GiB). `HALOGEN_MAX_TOK=8192` is only the per-request DEFAULT (the
+harness sends at most its `maxTokens` 45056); `HALOGEN_MAX_TOKENS_CAP=65536` +
+`HALOGEN_QUEUE_TIMEOUT=7200` are a coupled pair (FLAGS.md table).
+**A server-side sampler EXISTS** (`HALOGEN_TEMPERATURE/TOP_P/TOP_K` =
+1.0/0.95/20, the card's thinking recipe) — an absent temperature is no
+longer greedy; `models.yml` extraBody carries the same values, consistent
+not load-bearing. Prompt cache is ON by default (`PROMPT_CACHE=2`): a warm
+100k follow-up costs ~2 s vs ~88 s cold — the 27B engine's manual
+`HALOGEN_CACHE_MB` pin era is over. `response_format` is unsupported.
+Thinking lands in **`reasoning_content`**; the v2 chat template rejects
+`reasoning_effort` outside `xhigh|medium|low`, so the `models.yml` fold
+stays mandatory. Thinking tokens COUNT against the budget (empty content +
+`finish_reason: "length"` when exhausted mid-thought). Vision: OpenAI
+`image_url` content parts only; ≥256 px probe images or the model answers
+"video".
 
-Runbook: `serverAppConfigs-docs/LLM/frank-large-halogen27b-dflash2.md`.
-Rollback: revert the swap commit and redeploy — the ROCmFPX payload
-(`/home/dank/models/kingjones-Qwen3.8-27B-ROCmFP4-STRIX-MTP/`), the fork build
-and `scripts/build-rocmfpx.sh` (pin `36e9acd`) all remain on disk, seam-tested.
-The older gufo tag `frank-large-gufo-q4kxl-20261001` stays a **re-download**
-rollback (payloads removed 2026-10-03; `scripts/build-gufo.sh` stays).
+**Memory ledger** (121.2 GiB unified; the referee is the boot ledger):
+weights ~62 + KV 14.0 + arena ~8 (`3.8 + 8.4 × MAX_TOK/16384`) + vision
+~2 ≈ 86 GiB engine-side + `HALOGEN_HOST_RESERVE_GIB=20` → ~15 GiB
+margin over baseline co-tenancy. The 51.2 GB n-gram table rides as mmap'd
+page cache beside the checkpoint — the card's quickstart arithmetic does
+not bill it as resident, but READ the boot ledger and re-size if the
+engine prints a clamp.
 
-### frank-small — llama.cpp on the RTX 3090 (Dankclaw, ACTIVE)
+Fetch: `scripts/download-frank-large-flashnext.sh` into
+`/home/dank/models/halogen-flashnext-v2/` (`-f` + exact byte counts + LFS
+sha256 + `--verify-only`, no token on the command line; resume goes
+against the RESOLVED CDN URL because curl drops `Range` across HF's 302).
+The serve gates mirror the deployer EXACTLY (byte-exact checkpoint +
+n-gram + vision tower, flat tokenizer with all six files as regular
+files) — the 2026-10-01 crash-loop lesson.
 
-**llama.cpp master** (CUDA sm_86 build
-`/home/dank/GitPackages/llama-master/build-cuda/bin/llama-server`) as a
-**root systemd unit** (`scripts/frank-small.service`, `relink_unit`ed into
+Runbook:
+`serverAppConfigs-docs/LLM/frank-large-flash-server-v2-swift15.md`.
+Rollback: revert the swap commit and redeploy — the 27B payload
+(`/home/dank/models/halogen-qwen3.8-27b/`), its fetch script and the
+halogen 0.1.4 image digest all remain on disk; the ROCmFPX door behind
+that (`scripts/build-rocmfpx.sh`, pin `36e9acd`) is untouched.
+
+### frank-secondary — llama.cpp on the RTX 3090 (Dankclaw, ACTIVE)
+
+**llama.cpp master** (0.5.0-dev, commit `5fc4f3c`; CUDA sm_86 build
+`/home/dank/GitPackages/llama-master/build-cuda/bin/llama-server`,
+`GGML_CUDA_FA_ALL_QUANTS=ON`) as a **root systemd unit**
+(`scripts/frank-secondary.service`, `relink_unit`ed into
 `/etc/systemd/system/`), serving
-`bartowski/MiMo-V2.6-Distill-Qwen-9B-GGUF` at **Q6_K** (7,793,710,624 B,
-sha256 `ef96d05a…`) **plus `mmproj-MiMo-V2.6-Distill-Qwen-9B-f16.gguf`**
-(918,166,048 B, sha256 `ff348f…`), both `/home/dank/models/MiMo-Q6K/` (the
-projector is a hardlink of the Q5KM copy — the dir is self-contained), on
-port 7771. The rung went Q5_K_M → Q6_K on 2026-10-03 the same morning as the
-reslice, funded by the freed slots. The runtime arrived 2026-10-01 night for
-one reason:
-**asymmetric KV** — `--cache-type-k f16` + `--cache-type-v q8_0` is the fleet's
-only per-tensor KV lever, and SGLang's `--kv-cache-dtype` moves K and V
-together. On 2026-10-03 Frank spent the headroom that lever bought: the reslice
-from three slots to two made room for **fully unquantized V**
-(`CACHE_TYPE_V=f16`); both flags stay explicit in the serve script. Vision came
-2026-10-02 via `--mmproj`: the GGUF carries no `v.*` tensors, so the projector
-file is the entire vision path (and ~1.4 GiB of the card).
+`HauhauCS/Qwen3.5-9B-Uncensored-HauhauCS-Aggressive` at **Q6_K**
+(7,359,259,008 B, sha256 `c0ba7beb…` computed locally — the Hub carries no
+LFS oid) **plus the publisher's own BF16 mmproj** (921,704,448 B), both
+`/home/dank/models/Qwen35-9B-HauhauCS-Q6K/`, on port 7771. Cutover
+2026-10-07 (late) from the Gemma4-26B-A4B Q4_K_M 1×200,704 shape (same day).
 
-**Architecture: qwen3_5 hybrid** — 32 layers, 8 full-attention + 24 Gated
-DeltaNet, native 262,144 window. KV exists only in the 8 attention layers
-(4 KV heads × 256 dim × 2 B ≈ 2 KiB/token/layer, ~16 KiB/token for f16 K+V);
-the DeltaNet state is a fixed per-slot buffer, so slot count — not context —
-drives that term.
+**Architecture: qwen35 dense hybrid** — read from the GGUF's own header, not
+the model card: 32 layers, `full_attention_interval = 4` ⇒ **8 full-attention
+layers (3, 7, 11, …, 31) + 24 Gated DeltaNet linear layers**, 16 attention
+heads / **4 KV heads** × 256 dim, `ssm` conv_kernel 4 / state_size 128 /
+group_count 16 / time_step_rank 32 / inner_size 4096, native 262,144, vocab
+248,320. Tensor types are Q6_K × 250 + F32 × 177 — standard quants only, so
+the stock master build loads it (arch `qwen35` is registered in this build's
+`src/llama-arch.cpp`). **KV accounting:** the pool scales with the 8 full
+layers only — 8 × 4 × 256 = 8,192 elements/token per cache, so f16 K + f16 V
+bills **32 KiB/token** = 12.25 GiB at the 401,408 pool; the 24 DeltaNet
+layers bill a fixed recurrent state per SLOT, not per token, which is exactly
+why two 200k slots fit where the 26B MoE fit one.
 
-**The shape (2026-10-03):** `--ctx-size 401408 --parallel 2` = **TWO
-conversations of 200,704 tokens each** (llama.cpp divides the pool:
-`n_ctx_slot = CTX_SIZE / PARALLEL`), `--flash-attn on` (the seam pins it),
-**f16 K + f16 V**, all layers on the GPU, `--mmproj` for vision, and since
-2026-10-05 **batch 4096 / ubatch 2048** (doubled; the boot-pair A/B is in the
-runbook). Measured trade at 8,928 prompt tokens: prefill 4,024 → **4,209 t/s,
-+4.6 %**, decode flat (91.9–93.0 vs 93.8–94.3 — noise), and a 279-token prompt,
-which sits below either ubatch, did not move at all. The old "ubatch 2048
-failed the scratch boot" note belongs to the Q5_K_M predecessor and does NOT
-generalize: this shape boots 2048 clean. Measured VRAM **21,846 used / 2,279
-free** of 24,576 MiB against **21,276 / 2,849** for the 1024 arm, reproduced
-byte-identically the same hour — the doubling costs **616 MiB** and lands at
-the BOTTOM of the 2–3 GiB floor Frank asked for (`4096/1536` is the middle rung
-if that margin is ever wanted; non-KV 9,302 MiB, up from 8,732 by exactly the
-compute buffers; f16 K+V is 32 KiB/token across the 8 attention layers;
-quote `nvidia-smi`'s `free` field, never `total − used` — the driver holds
-~450 MiB of reserve the subtraction hides). The 2026-10-02 3 × 180,224 +
-q8_0-V shape stays one env edit away (`CTX_SIZE=540672 PARALLEL=3
-CACHE_TYPE_V=q8_0`, and it must pair with Q5_K_M — Q6_K breaks the floor
-there). Three native windows need 18.4 GiB of KV alone and were never on the
-table.
+**The shape (2026-10-07):** `--ctx-size 401408 --parallel 2` = TWO
+conversations of 200,704 (Frank's cap; the harness `contextWindow` quotes
+the per-slot number), `--flash-attn on`, **f16 K + f16 V** (exact KV),
+batch 4096 / **ubatch 2048** (the measured CUDA optimum: +4.6 % prefill,
+decode flat, 616 MiB — MiMo-era A/B on this card), `--log-verbosity 4`,
+`--mmproj` for vision, **speculative decoding OFF** (`SPEC_TYPE=none`).
 
-Verified at deploy (2026-10-03): boot logs `n_slots = 2, n_ctx_slot = 200704`
-and `loaded multimodal model`; decode **94.44 tok/s** at temp 0 on the probe
-that measured 104.84 on Q5_K_M — **the Q6_K rung costs a measured −10 %**:
-decode on this card is weight-bandwidth-bound and Q6_K streams ~10 % more
-bytes/token; health 200, e2e 33/33; native `tool_calls`; thinking in
-`reasoning_content`; a **512×512** solid-red PNG answers `Red` through an
-`image_url` content part (two traps: the legacy top-level `images` field is
-ignored, and a TINY probe image undersupplies image tokens — a 10×10 PNG made
-the model claim "it's a video"; probe with ≥256 px). A prompt over the slot
-is a **400 `exceed_context_size_error`**, which is why `models.yml` sets
-`contextWindow: 200704`.
+**Why MTP is off**: it is not a choice — **this Q6_K file carries no
+`nextn`/`mtp` tensors** (verified from the tensor-name list), and MTP is a
+property of the file, not of the flags. The publisher card advertises "MTP
+support" for the base architecture; the quantized export dropped the head.
 
-Runbook: `serverAppConfigs-docs/LLM/frank-small-llama-mimo-q6k-2x200704.md`
-(the boot-pair table that prices pool changes lives in the retired
-[[frank-small-llama-mimo-q5km]]).
-Rollback: `MODEL_FILE=/home/dank/models/MiMo-Q5KM/MiMo-V2.6-Distill-Qwen-9B-Q5_K_M.gguf`
-buys back 875 MiB and the −10 % decode rate (3,571 free, 104.84 tok/s, both
-measured); the older shape `CTX_SIZE=540672 PARALLEL=3 CACHE_TYPE_V=q8_0`
-pairs with Q5 only; SGLang rollback in
-[[frank-small-sglang-mimo-v26]] (venv on disk at
-`/home/dank/GitPackages/sglang-cu128/`, INT8 payload staged to
-`~dank/.AGENTTRASH/20261002-retired-llm-payloads/`).
+**Measured boots** (nvidia-smi, the referee): the deployed boot is
+**21,436 used / 2,689 free** (ubatch 2048) — the pricing boots were the
+MiMo-V2.6-Distill-Qwen-9B pair at this exact geometry (**21,276 / 2,849** at
+ubatch 1024, **21,846 / 2,279** at 2048), so the 415 MiB lighter weights
+bought back what the projection predicted. The engine ledger confirms the KV
+law to the MiB: `CUDA0 KV buffer size = 12544.00 MiB` (`K (f16): 6272.00`,
+`V (f16): 6272.00`), compute buffers 1,136 MiB on CUDA0 + 848 MiB host, the
+mmproj 879 MiB + 248 MiB. Decode is ubatch-invariant (Swift-era 3090 law). The
+Q6_K rung costs ~10 % decode vs Q5_K_M (weight bandwidth; MiMo-measured 94.4
+vs 104.8 t/s).
 
-### frank-primary — tcclaviger vLLM, Qwen3.8-Flash-Next FP8 (Dankserver, ACTIVE)
+**Vision**: the publisher's own BF16 mmproj via `--mmproj` (the model is
+natively multimodal; the Q6_K file carries no vision tensors). Probe traps
+carry over: ≥256 px probe images or the model answers "video", and only the
+OpenAI `image_url` content part reaches the model.
 
-The **tcclaviger RDNA4 vLLM fork** (`docker.io/tcclaviger/vllm@sha256:da4029666…`,
-digest-pinned) serving **`orcarouter/Qwen3.8-Flash-Next-Uncensored-FP8`**
-(~172.8 GiB, 131 shards) on **both** Radeon AI PRO R9700s (gfx1201), **TP=2**,
-**fp8 KV**, **no speculative decode**, **expert prefetch OFF**, and since
-2026-10-03 the **shape-G** geometry (shape E + MNBT 2048 + the CUDA-graph plan
-term at the measured floor): `--max-model-len 200704` × 3 slots with
-`--kv-cache-memory-bytes 3,166,500,000`/rank (~400,700 KV tokens — the engine
-prices this pool at 7,891.8 B/token here), util **0.982** (the law — see the
-unbootable-1.0 lesson below), `--expert-precise-memory`,
-`VLLM_CUDAGRAPH_MEM_BASE_MB=0` / `_PER_GRAPH_MB=12`,
-`GPU_MAX_HW_QUEUES=2` and a mounted `/root/.cache`. `models.yml` carries
-`contextWindow: 200704` for this endpoint — the served window, not the model's
-native 262144, so the harness compacts before vLLM answers a longer prompt
-with a 400.
+**Compaction hub**: `models.yml` routes `compactionModel` for
+`frank-archon`, `frank-dragoon` and `frank-zealot` (the 3090 endpoint) here — 200,704 is the
+only window in the fleet that fits a worst-case compaction input (~170k
+tokens). With `--parallel 2` a compaction request no longer serializes behind
+a live generation (the Gemma4 era's `--parallel 1` law).
 
-**`--max-num-seqs 2` is impossible on this model.** Measured 2026-10-02 by
-single-variable boot probe against the then-deployed shape: SEQS 2 fails at
-BOTH MNBT 1024 and 2048 with `torch.fx.experimental.symbolic_shapes.Constraint
-ViolationError: Constraints violated (L['query_start_loc'].size()[0]) …
-specialized it to be a constant (3)`; SEQS 3 passes at both. `model_runner.py`
-sizes that input buffer `max_num_reqs + 1` and `_dummy_run` sets
-`num_reqs = min(max_num_batched_tokens, max_num_reqs)`, so the width is
-`SEQS + 1` for any `MNBT ≥ SEQS` — MNBT only moves the compile RANGE
-((1,1024) vs (1,2048)). `--enforce-eager` bypasses the path but drops CUDA
-graphs. The same failure class killed the 2026-10-01 bf16 2×262144 shape.
+Runbook:
+`serverAppConfigs-docs/LLM/frank-secondary-llama-qwen35-9b-q6k-2x200704.md`.
+Rollback: revert the swap commit — the Gemma4 payload
+(`/home/dank/models/Gemma4-26B-A4B/`), the MiMo Q6_K payload
+(`/home/dank/models/MiMo-Q6K/`), the Swift payload
+(`/home/dank/models/Swift15-UncIQ3XXS/`) and the Ornith Q8_0 payload
+(`/home/dank/models/Ornith-Q8/`) all remain on disk.
 
-**The expert arena has THREE reachable levers: `max_model_len`, the overhead
-term, and the CUDA-graph estimate.**
-The fork's planner sizes expert cache as `budget − non-expert − KV reserve −
-graphs − overhead`, and the KV reserve is registry-priced at 9190 B/token ×
-`max_model_len`: 262144 → 2.48 GiB reserve → arena 14.30 GiB / **6,248 slots**;
-200704 → 1.91 GiB → 14.87 GiB / **6,496 slots**. Δreserve 0.57 GiB = Δarena
-0.57 GiB. The 6.1e9 → 3.15e9 pool cut bought **zero** arena; it buys runtime
-headroom (the shape it replaced died with `0 bytes free` on a real 995-token
-request). The second lever is `--expert-precise-memory`, **ON since
-2026-10-02**: it swaps the registry's default overhead (7.60 GiB = fitted +
-compile/graph-capture headroom) for the fitted figure, and the 2.35 GiB
-difference lands in the arena — measured plan lines at 200704 × 3: overhead
-5.25 GiB, expert cache **17.22 GiB / 7,522 slots** (+1,026 / +16 %), swap region
-13.7 GiB / 5,986 slots, boot healthy in 420 s. The pool stayed at 3.15e9
-deliberately: because the pool is not an arena lever, cutting it would cost
-concurrency (a 2.2e9 probe reports 277,686 tokens / **1.38×**) and buy no slots.
-The arena (2.344 MiB per slot) is the decode-critical region — cutting it for KV
-headroom collapses the offload hit rate.
 
-**Free VRAM on this endpoint is NOT arena headroom (2026-10-03 audit).**
-`rocm-smi` reads ~3.5 GiB free per card after hours of real traffic, but the
-planner charges `budget = total × util` against terms that overstate reality:
-the KV reserve is registry-priced at 9190 B/token (`1.91 GiB` at 200704) while
-the real pool bills 2.93 GiB at 7,892 B/token, and the precise overhead (5.25
-GiB) is far above the idle-measured figure. So the visible free space is the
-estimator gap, not allocatable arena. `gpu_memory_utilization` is a real arena
-lever (each +0.01 = +0.32 GiB) but **0.982 is the law, and 1.00 is UNBOOTABLE
-on this fork — measured 2026-10-03**: the worker init demands
-`free > budget`, and at `le=1` budget == device total, so both ranks raise
-`ValueError: Free memory on device cuda:N (31.29/31.86 GiB ...)` before a
-single weight loads, twice (2-restart crash loop, zero payload progress). The
-CLI field accepting `le=1` is the trap; Frank's 0.982 is where it always
-ships. What IS reachable from flags: `VLLM_CUDAGRAPH_MEM_BASE_MB` /
-`_PER_GRAPH_MB`, read at call time by `estimate_cudagraph_memory()` — the
-analytic default (256 MiB + 52 MiB/graph) charged 0.55 GiB where the engine's
-own post-capture line reported 0.31 GiB, and a second boot (with the plan
-already charging 0.32 GiB) reported **-0.06 GiB**, so the 256 MiB base pool is
-pure over-charge: shape G charges `0 base + 12 MiB/graph` = 0.07 GiB, which
-still prices per graph for a future 18-graph spec-decode shape (0.21 GiB).
-Verify every deploy by comparing the `Estimated CUDA graph memory`
-line with `Graph capturing finished … took X GiB`; if capture exceeds the
-estimate, raise `_PER_GRAPH_MB` back toward 52. MNBT 1024 → 2048 costs ZERO
-arena: the registry overhead is `base + 0.22 GiB × max(0,(nbt−4096)/1024)`
-anchored at nbt 4096, so every rung ≤4096 charges the same 5.25 GiB — and the
-fitted figure was fitted AT 4096.
+### frank-primary — Radiance (StillDeadcode), Swift1.5-Qwen3.8-Flash-Next GPTQ w4, TP=2 (Dankserver, ACTIVE)
 
-**A container's compile cache does NOT survive `podman rm -f`, and on this
-endpoint that is 181 s per boot.** Measured 2026-10-02: the image ships **no
-`/root/.cache` at all** (fresh container: `No such file or directory`); after a
-boot it holds `/root/.cache/vllm` at **1,154,714,333 B in 4,665 files** plus
-`/root/.cache/comgr` at 229 MB, all in the ephemeral layer, and the engine
-prints `torch.compile took 181.20 s in total` (173.19 s of it one graph,
-`Compiling a graph for compile range (1, 1024)`). Every deploy, restart and
-reboot re-pays it. The serve script now mounts
-`FRANK_PRIMARY_CACHE_DIR` at `/root/.cache` — the parent, so the vLLM AOT /
-torch_compile tree (`VLLM_CACHE_ROOT`, whose default is
-`/root/.cache/vllm`) and the comgr HIP-kernel JIT cache both persist, and
-nothing in the image is hidden because the image has no such directory. The
-compile RANGE is part of the cache key, so an MNBT change costs one
-recompilation, not a permanent tax. Same class of fix as the recipe's
-`/cache/{triton,vllm,inductor}` volumes. **Proven live 2026-10-02 on
-consecutive boots:** cold `Compiling a graph for compile range (1, 1024) takes
-179.24 s` / `torch.compile took 188.26 s`; warm `Directly load the compiled
-graph(s) for compile range (1, 1024) from the cache, took 2.143 s` /
-`torch.compile took 3.97 s in total` / `init engine (profile, create kv cache,
-warmup model) took 41.13 s (compilation: 3.98 s)`. Time to `/health` is
-dominated by the 172.8 GiB payload load, so the visible win is the ~184 s of
-inductor/AOT build, not a proportional boot cut.
+The **radiance engine** (StillDeadcode, Apache-2.0; image
+`stilldeadcode/radiance@sha256:4c3f484564…` digest-pinned, **pin 1.0.13**
+for serve/convert/KLD; `libr4d` gfx1201 kernels, plain `podman run`, root
+systemd unit, container name = unit name) serving
+**ukisai/Swift1.5-Qwen3.8-Flash-Next (BF16, 335.3 GiB, 131 shards,
+397/397 sha256-verified)** converted by `scripts/convert-swift15-w4dn8.sh`
+with recipe `scripts/recipes/qwen4exp-swift15-w4dn8-i8-hc8m.recipe` to
+`qwen4exp-swift15-w4dn8-i8-hc8m.rad` (~114 GiB): **all 49,152 experts GPTQ
+w4** (act-order, damp 0.05, 10,000,000-token calibration captured on engine
+**1.2.0** via `RADIANCE_CALIB_DIR`/`RADIANCE_CALIB_TOKENS` — the only pin
+with the token knob; the RADGRAM2 grams are format-identical across pins and
+cross the boundary by design), **24 whole experts per layer cast bf16** (the
+PROTECTED set, derived from the calibration grams' energy share),
+**int8 trunk/lm_head**, **fp8 HC mixtures** (`--hc-mix-dtype`), PLE n-gram
+table at fp8 fixed scale (47.7 GiB in the container, from 95.4 GiB bf16).
+The plugin's **one-table-one-encoding law** (measured at rad-convert
+declare): one expert table takes ONE encoding; only whole bf16 experts are
+legal beside it — no w4/fp8 split of the down/gate_up pair (24-value
+`rad_arch_quant_desc` array, one per layer). KLD gate vs the BF16
+reference (recorded on the same 1.0.13 pin) is the precision referee;
+the throughput gate (decode ≥ 30.8 t/s) was passed at 4.5× the bar
+(122–173 t/s at 44.5k depth). Measured decode/prefill in the runbook.
+Rollback: revert the swap commit — the fp8-native `.rad`
+(`qwen4exp-fp8-native.rad`) and every prior payload remain on disk.
 
-Measured 2026-09-29 tuning campaign (fp8 + MTP4 era, since retired): decode
-**37.4-40.6 tok/s, acceptance ~54 %**; the 256k ctx arm measured 34.1-36.5;
-2-way concurrency sums BELOW parity (25.49 vs 26.8 single) — slots are a
-capacity feature, not a throughput one. V-only q8 KV is NOT possible via
-config: stock `--kv-cache-dtype` is K+V together; a V-only quant needs a fork
-patch.
+**Shape (both R9700s, TP=2)**: `--tp-wire wht6`, `--max-num-seqs 3`,
+`--max-model-len 200000` (the harness contextWindow must quote this),
+`--placement expert_tiered --host-pool-mib 40960` + **explicit VRAM
+partition `--vram-weights-mib 26071 --vram-kv-mib 3543`** (the vram-pair
+replaced `--expert-vs-cache-ratio`; the pair sizes the fp8-KV shape at the
+deployed boot), `--kv-cache-dtype fp8` (inherited from the fp8-native
+env), `--num-speculative-tokens 2`,
+`--max-num-batched-tokens 2048`, prefix cache 4 GiB pinned host + 64 GiB
+disk at `/home/dank/radiance-state`, **`--ngram-placement ram`**, Froggeric
+v22.5 chat template vendored (`scripts/qwen38-froggeric-v22.5.jinja`,
+`--override-chat-template`). Boot ledger: KV pool 3.46 GiB (~480k fp8
+tokens), two expert pools 57.84 GiB, cards settled 1.41 GiB free.
 
-Unit and self-healing: root systemd unit (`scripts/frank-primary.service`,
-`User=dank`, `LimitMEMLOCK=infinity`), `ExecStop=podman stop -t 120
-frank-primary-qwen`. The serve script drives `podman run` directly, gates
-the digest-pinned image, the payload's `config.json` and its exact
-131-shard count and the GPU device nodes, clears stale container
-identities, then blocks on `podman wait`. `deploy_llm_primary` gates the
-same, waits out :7770, relinks/enables/restarts the unit and health-gates
-with the MainPID check. Boot cost ~7 min warm (the 172.8 GiB payload load; the
-~184 s inductor/AOT build inside it is what the mounted cache now skips — see
-the compile-cache lesson below); health returns 503 while loading.
+Shape notes carried from the fp8-native era and unchanged: wht6 gains,
+MNDT 2048 law (2560 collapses decode to 2.6–7.7 t/s), the 64 GiB disk
+prefix cache, and the rule that a prompt over `--max-model-len` is
+refused, never truncated. What changed: the fp8-native era's NVMe expert
+streaming (the 3.5 GB/s wall) is gone — the w4 resident set fits the
+40 GiB pool, so decode ran 122–173 t/s in the campaign arms and ~190 t/s
+on the deployed shape (512-token streamed probe). The campaign also
+measured the int8-KV + spec-3 arm (136.0 t/s median at 44.5k depth,
+accept 0.7246); production shipped the fp8-KV + spec-2 pair at cutover —
+the KLD gate scored at the fp8-KV shape, so the gate and the boot agree.
+Disk-tier sizing law: the engine forces ~9.6 GiB of disk-tier weight
+buffers + arena onto the weights side, so `weights + kv` is capped near
+29.6 GiB/card; boot with the pair, never `--kv-cache-size-mib`.
 
-Runbook: `serverAppConfigs-docs/LLM/dankserver-qwen38-fp8-expert-offload-tuning.md`.
+**The pinned-RAM ledger is why this is a root unit**: 2 × 40 GiB expert
+pools + the 47.7 GiB n-gram copy (`--ngram-placement ram` is pinned RAM
+the card reads) + 4 GiB prefix host ≈ **132 GiB** charged against
+`RLIMIT_MEMLOCK` — the TTM floor is set just above it.
+`ttm.pages_limit`/`pages_max` cap GPU-mappable pinned pages; the serve
+script's TTM preflight (`FRANK_PRIMARY_TTM_MIN_MIB=140000`) refuses to
+start until both knobs are raised in the stop window (sysfs write first,
+kernel cmdline + reboot only if read-only).
+
+**Contract facts** (source-verified): the HTTP listener binds AFTER the
+full load, so a bare `curl /health` gate is honest (`/health` == `/ping`,
+200 serving / 503 stopping); sampling params temperature…preserve_thinking
+accepted, **unknown params 400** (models.yml's extraBody set is fully
+inside it); `response_format` supported; `logprobs` REFUSED; reasoning in
+`reasoning_content` and tool calls in the qwen3.8 dialect via the auto
+chat format — no parser flags; vision native (the 27-layer Qwen-VL tower,
+`--mm-max-patches`); `/metrics` carries vLLM-compatible names +
+`radiance:*` families — the Dankclaw Prometheus scrape is re-added.
+
+Runbook: `serverAppConfigs-docs/LLM/frank-primary-radiance-swift15-w4dn8.md`.
+Retired fp8-native era: `serverAppConfigs-docs/LLM/frank-primary-radiance-qwen38-native-fp8.md`.
+Retired FreeToken era: `serverAppConfigs-docs/LLM/frank-primary-freetoken-pr587.md`.
+Retired vLLM era: `serverAppConfigs-docs/LLM/dankserver-qwen38-fp8-expert-offload-tuning.md`.
 Retired GLM experiment: `serverAppConfigs-docs/LLM/frank-primary-glm53-flash-w4a16.md`.
 
 ## Key Lessons Learned
@@ -774,7 +789,7 @@ linked runbooks.
   121 GiB box, pushed `available` to 3 GiB, and every later rate was suspect.
   Record the child pid and kill it in a `finally`, then prove `free` came back
   before believing a number. Kill by pid — never `pkill -f llama-server`,
-  which also hits frank-small.
+  which also hits frank-secondary.
 
 **llama.cpp serving**
 
@@ -840,7 +855,7 @@ linked runbooks.
   free. That was a CONTIGUITY failure, not a total-bytes one, and no totals
   model — the sweep's or the seam's — can see it.)
 - **On the CUDA 3090 the same knob has NO interior optimum up to 2048, so
-  frank-small ships 4096/2048.** Measured 2026-10-05 by boot-pair A/B on the
+  frank-secondary ships 4096/2048.** Measured 2026-10-05 by boot-pair A/B on the
   live service: at 8,928 prompt tokens ubatch 1024 gives 3,972 / 4,077 t/s and
   ubatch 2048 gives 4,207 / 4,212 — **+4.6 %** — for **616 MiB** (free 2,849 →
   2,279). Decode did not move (91.9–93.0 vs 93.8–94.3, i.e. noise), and a
@@ -849,9 +864,9 @@ linked runbooks.
   than the ubatch cannot see it. The logical `--batch-size` was again free. And
   the predecessor's "ubatch 2048 failed the scratch boot" was a shape
   observation, not a law: the same file boots 2048 clean at 2 × 200,704. Bench
-  it by restarting the SERVICE through a transient `FRANK_SMALL_ENV_FILE`
+  it by restarting the SERVICE through a transient `FRANK_SECONDARY_ENV_FILE`
   drop-in, never a scratch port — one card holds one server. See
-  [[frank-small-llama-mimo-q6k-2x200704]].
+  [[frank-secondary-llama-mimo-q6k-2x200704]].
 - **One `-ts` step is one LAYER, and a hybrid stack's layers are not the same
   size.** qwen35 is 16 x (3 Gated DeltaNet + 1 Gated Attention), so a step is
   worth ~0.43 GiB on a DeltaNet boundary and ~1.43 GiB on a full-attention one
@@ -873,7 +888,7 @@ linked runbooks.
   CTX_SIZE / PARALLEL`, and llama.cpp refuses a prompt longer than the SLOT
   (`400 exceed_context_size_error: request (184001 tokens) exceeds the
   available context size (180224 tokens)`) even when the pool has room. So
-  frank-small's 2 → 3 reslice moved each conversation from the model's native
+  frank-secondary's 2 → 3 reslice moved each conversation from the model's native
   262,144 down to 180,224, and `models.yml` `contextWindow` had to follow or
   the harness would discover the ceiling as a 400 on a long conversation. Side
   effect worth knowing: the shorter slot made decode ~5 % FASTER (106.8 vs
@@ -884,9 +899,9 @@ linked runbooks.
   day the freed space also bought the Q6_K rung (now 94.4 tok/s — the rung
   itself is a further measured −10 %, weight-bandwidth, not slot geometry).
   All moves measured; both slot directions cost or buy window, never pool.
-  See [[frank-small-llama-mimo-q6k-2x200704]].
+  See [[frank-secondary-llama-mimo-q6k-2x200704]].
 - **Price a VRAM budget from two boots, never from a formula — and an mmproj
-  costs ~1.5x its file size.** The frank-small 3090 fit was solved from two
+  costs ~1.5x its file size.** The frank-secondary 3090 fit was solved from two
   measurements: 2 × 262,144 with no projector = 20,190 MiB used (KV 12,800 ⇒
   **non-KV 7,390**), 3 × 196,608 with the f16 projector = 22,972 used (KV
   14,113 ⇒ **non-KV 8,859**). The difference is the vision tower: a 918 MB file
@@ -894,12 +909,12 @@ linked runbooks.
   ride with it. Subtracting the two rows is exact; extrapolating weights+KV+
   "a bit for buffers" is what leaves 1.1 GiB free when you meant 2.7. Then
   solve `KV ≤ card − non-KV − headroom` and round to a multiple of
-  `PARALLEL × 4096`. Recorded in `scripts/frank-small.env` so the next resize
+  `PARALLEL × 4096`. Recorded in `scripts/frank-secondary.env` so the next resize
   starts from measurement.
 - **`images` and `image_url` are NOT interchangeable: only the content part
   reaches the model.** `--mmproj` loads, `usage.prompt_tokens` rises, and the
   model still invents an answer if the client used the legacy top-level
-  `images` field on `/v1/chat/completions` — frank-small answered "Brown." for
+  `images` field on `/v1/chat/completions` — frank-secondary answered "Brown." for
   a solid-red PNG that way and "Red." through the OpenAI `image_url` part. The
   failure is silent because a hallucinated color is a plausible answer. Probe
   vision with a solid color and expect an exact word.
@@ -911,7 +926,7 @@ linked runbooks.
   returns `tool_calls`. Check tool support before routing a role at an endpoint.
 - **Quantized KV on CUDA needs `GGML_CUDA_FA_ALL_QUANTS=ON`** — the same
   option and the same ~12x prefill collapse described in the ROCm note above.
-  f16 KV needs nothing; frank-small ran f16 for that reason until
+  f16 KV needs nothing; frank-secondary ran f16 for that reason until
   2026-09-17, when `build-cuda` was rebuilt with the option ON so the
   endpoint could take q8_0 KV.
 - **llama-bench separates a LIST with `/` and SWEEP values with `,`.** Both
@@ -929,7 +944,7 @@ linked runbooks.
   raises on anything else — HTTP 500, not a clamp — so the harness's levels
   must be folded onto that set in `models.yml` (`compat.reasoningEffortMap`).
   A role whose level is unmapped fails on its FIRST request, and the failure
-  names Jinja, not the model: every subagent on `frank-small` died with
+  names Jinja, not the model: every subagent on `frank-secondary` died with
   `Unexpected reasoning effort high` until `high: xhigh` was added on
   2026-09-16. Halogen accepts the wider set, so the two entries legitimately
   differ.
@@ -1003,9 +1018,9 @@ Active:
 
 | Service | Runbook |
 |---------|---------|
-| frank-large (halogen container, Qwen3.8-27B .hgn + DFlash2, 1×200,700, text-only, Dankclaw) | `serverAppConfigs-docs/LLM/frank-large-halogen27b-dflash2.md` |
-| frank-small (llama.cpp CUDA, MiMo-V2.6-Distill-Qwen-9B Q6_K, 2×200,704 f16 KV, Dankclaw) | `serverAppConfigs-docs/LLM/frank-small-llama-mimo-q6k-2x200704.md` |
-| frank-primary (tcclaviger vLLM, Qwen3.8-Flash-Next FP8, Dankserver) | `serverAppConfigs-docs/LLM/dankserver-qwen38-fp8-expert-offload-tuning.md` |
+| frank-large (flash-server 0.16.2, abliterated Swift1.5 v2 .hgn + MTP always-on, 2×262,144, vision ON, Dankclaw) | `serverAppConfigs-docs/LLM/frank-large-flash-server-v2-swift15.md` |
+| frank-secondary (llama.cpp CUDA, Qwen3.5-9B-Uncensored Q6_K, 2×200,704 exact f16 KV MTP-off, vision ON, Dankclaw) | `serverAppConfigs-docs/LLM/frank-secondary-llama-qwen35-9b-q6k-2x200704.md` |
+| frank-primary (radiance StillDeadcode, Qwen3.8-Flash-Next-FP8 native, 2×R9700 TP=2, Dankserver) | `serverAppConfigs-docs/LLM/frank-primary-radiance-qwen38-native-fp8.md` |
 | Dankclaw kernel posture (`amd_iommu=off`) | `serverAppConfigs-docs/LLM/dankclaw-iommu-translation-off.md` |
 | Deployment flow + incidents | `serverAppConfigs-docs/LLM/deployment.md` |
 
@@ -1013,13 +1028,19 @@ Retired (rollback targets and history):
 
 | Runtime | Host | Window | Runbook |
 |---------|------|--------|---------|
+| llama.cpp CUDA Gemma4-26B-A4B QAT Q4_K_M, 1×200,704 f16-K/q8_0-V MTP-off (`frank-secondary`) | Dankclaw | 2026-10-07 → 10-07 (same-day swap; rollback target — payload on disk; owns the `attention_k_eq_v` pair-billing KV law and the gemma4-assistant drafter-cost boots) | `frank-secondary-llama-gemma4-q4km-200704.md` |
 | ROCmFPX llama.cpp fork Qwen3.8-27B Uncensored Q6_0_ROCMFPX_AGENT + MTP (`frank-large`) | Dankclaw | 2026-10-03 → 10-05 (rollback target, payload + fork build on disk) | `frank-large-rocmfpx-q6agent-uncensored-mtp.md` |
+| tcclaviger vLLM fork Qwen3.8-Flash-Next-Uncensored FP8, TP=2, shape G (`frank-primary`) | Dankserver | 2026-09-29 → 10-05 (rollback target — tag `frank-primary-vllm-shapeG-20261003`; trio + env lore in the tag, image in rootless storage) | `dankserver-qwen38-fp8-expert-offload-tuning.md` |
+| FreeToken PR #587 ROCm, Qwen3.8-Flash-Next-Uncensored-FP8, ONE R9700 (`frank-primary`) | Dankserver | 2026-10-05 → 10-05 (rollback target — tag `frank-primary-freetoken-pr587-20261005`; trio + Dockerfile in the tag, image `localhost/frank-primary-freetoken:pr587-77a7201` and orcarouter payload on disk) | `frank-primary-freetoken-pr587.md` |
+| llama.cpp master MiMo-V2.6-Distill-Qwen-9B Q6_K 2×200,704 f16 KV (`frank-secondary`) | Dankclaw | 2026-10-03 → 10-05 (rollback target — payload on disk; owns the ubatch-2048 A/B; the Ornith swap of 10-05 was committed but never deployed, so this was the LIVE payload Swift replaced) | `frank-secondary-llama-mimo-q6k-2x200704.md` |
+| llama.cpp Ornith-1.5-9B Q8_0, ONE 262,144 f16 KV (`frank-secondary`) | Dankclaw | 2026-10-05 (committed 77decb5→380aafa, never deployed — superseded by Swift the same evening; payload on disk) | trail in `frank-secondary-llama-swift15-unc-iq3xxs-mtp-114688.md` |
+| llama.cpp CUDA Swift-1.5-27B Uncensored RCO IQ3_XXS-mtp, 2×114,688 draft-mtp q8_0 KV (`frank-secondary`) | Dankclaw | 2026-10-06 → 2026-10-07 (rollback target — payload on disk; owns the 3090 ubatch ladder A/B and the 34 KiB/token qwen35 KV law) | `frank-secondary-llama-swift15-unc-iq3xxs-mtp-114688.md` |
 | ROCmFPX llama.cpp fork Qwen3.8-27B Q8_0_ROCMFPX_AGENT + MTP (`frank-large`) | Dankclaw | 2026-10-02 → 10-03 (rollback target, payload on disk) | `frank-large-rocmfpx-q8agent-mtp.md` |
 | gufo v0.1.1 Qwen3.8-27B UD-Q8_K_XL/UD-Q4_K_XL + DFlash2 (`frank-large`) | Dankclaw | 2026-09-28 → 10-01, again 10-01 → 10-02 (payloads removed from disk 2026-10-03; rollback = re-download) | `frank-large-gufo-qwen38-27b.md` |
 | vLLM TheRock MiMo-V2.6-Distill-Qwen-9B BF16 + cross-model MTP (`frank-large`) | Dankclaw | 2026-10-01 (same-day rollback to gufo) | `frank-large-vllm-mimo-v26-bf16.md` |
-| llama.cpp master MiMo-V2.6-Distill-Qwen-9B Q6_K (`frank-small`) | Dankclaw | 2026-10-01 → 10-02 (the rung returned ACTIVE 2026-10-03 at the 2×200,704 shape — `frank-small-llama-mimo-q6k-2x200704.md`) | `frank-small-llama-mimo-q6k.md` |
-| llama.cpp master MiMo-V2.6-Distill-Qwen-9B Q5_K_M 2×200,704 f16 KV (`frank-small`) | Dankclaw | 2026-10-02 → 10-03 (rollback target — one `MODEL_FILE` line, buys back 875 MiB + the −10 % rung tax; owns the boot-pair table) | `frank-small-llama-mimo-q5km.md` |
-| SGLang MiMo-V2.6 INT8-W8A16-AutoRound (`frank-small`) | Dankclaw | 2026-10-01 (same-day swap to the GGUF path) | `frank-small-sglang-mimo-v26.md` |
+| llama.cpp master MiMo-V2.6-Distill-Qwen-9B Q6_K (`frank-secondary`) | Dankclaw | 2026-10-01 → 10-02 (the rung returned ACTIVE 2026-10-03 at the 2×200,704 shape — `frank-secondary-llama-mimo-q6k-2x200704.md`) | `frank-secondary-llama-mimo-q6k.md` |
+| llama.cpp master MiMo-V2.6-Distill-Qwen-9B Q5_K_M 2×200,704 f16 KV (`frank-secondary`) | Dankclaw | 2026-10-02 → 10-03 (rollback target — one `MODEL_FILE` line, buys back 875 MiB + the −10 % rung tax; owns the boot-pair table) | `frank-secondary-llama-mimo-q5km.md` |
+| SGLang MiMo-V2.6 INT8-W8A16-AutoRound (`frank-secondary`) | Dankclaw | 2026-10-01 (same-day swap to the GGUF path) | `frank-secondary-sglang-mimo-v26.md` |
 |tcclaviger vLLM GLM-5.3-Flash W4A16 int4, +/- DFlash2 drafter (`frank-primary`)|Dankserver|2026-09-30 → 10-01|`frank-primary-glm53-flash-w4a16.md`|
 | llama.cpp b11030 Qwen3.8-27B UD-Q8_K_XL + DFlash2 (`frank-large`) | Dankclaw | 2026-09-21 → 09-28 | `frank-large-unsloth-qwen38-27b-q8xl.md` |
 | R9V radiance-vLLM Qwen3.8-Flash-Next-Uncensored IQ4_XS (`frank-primary`) | Dankserver | 2026-09-27 (same-day swap to Q8_0) | `frank-primary-r9v-radiance-qwen38-uncensored.md` |
@@ -1032,16 +1053,16 @@ Retired (rollback targets and history):
 | CIRU v4.4.1 Orca Qwen3.8-Flash (`frank-large`) | Dankclaw | 2026-09-19 → 09-21 | `frank-large-ciru-orca.md` |
 | unsloth llama.cpp b11030 Qwen3.8-Flash-Next UD-IQ4_XS (`frank-large`) | Dankclaw | 2026-09-21 (same-day swap) | `frank-large-unsloth-qwen38-flash-next.md` |
 | Kairic Edge Qwen3.8-27B IU4 W4A4 (`frank-large`) | Dankclaw | 2026-09-21 (same-day swap) | `frank-large-kairic-edge-qwen38-27b.md` |
-| ISTA-DASLab Qwen3.8-27B-GSQ-RCO IQ3_S (`frank-small`) | Dankclaw | 2026-09-16 → 09-17 | `frank-small-qwen38-gsq-rco.md` |
-| vLLM Gemma4-12B w4a16 (`frank-small`) | Dankclaw | 2026-09-19 (same-day revert) | `frank-small-vllm-gemma4.md` |
-| llama.cpp Gemma4-12B QAT Q4_K_M (`frank-small`) | Dankclaw | 2026-09-17 → 10-01 | `frank-small-gemma4-12b.md` |
+| ISTA-DASLab Qwen3.8-27B-GSQ-RCO IQ3_S (`frank-secondary`) | Dankclaw | 2026-09-16 → 09-17 | `frank-secondary-qwen38-gsq-rco.md` |
+| vLLM Gemma4-12B w4a16 (`frank-secondary`) | Dankclaw | 2026-09-19 (same-day revert) | `frank-secondary-vllm-gemma4.md` |
+| llama.cpp Gemma4-12B QAT Q4_K_M (`frank-secondary`) | Dankclaw | 2026-09-17 → 10-01 | `frank-secondary-gemma4-12b.md` |
 | agentionai ROCmFP4-FAST imatrix | Dankclaw | 2026-08-29 → 08-30 | `agentionai-fast-imatrix.md` |
 | eemin Qwen3.8-27B ROCmI4 W4A4 MTP | Dankclaw | 2026-08-18 → 08-25 | `eemin-rocmi4-mtp.md` |
 | Kairic Edge IU4 | Dankclaw | 2026-08-22 (same-day revert) | `kairic-edge.md` |
 | CIRU ActiveFPX v3 / PromptForge | Dankclaw | through 2026-08-22 | `ciru-activefpx-promptforge.md` |
 | DFlash2 Vulkan | Dankclaw | 2026-08-21 (same-day revert) | `dflash2-large-vulkan.md` |
 | FastFlowLM NPU (`frank-npu`) | Dankclaw | twice, retired 2026-09-11 | `frank-npu.md`, `FastFlowLM NPU Inference.md` |
-| FreeToken (official + Huihui FP8, `frank-small`) | Dankserver | 2026-09-02 → 09-15 | `freetoken-frank-small.md` |
+| FreeToken (official + Huihui FP8, `frank-secondary`) | Dankserver | 2026-09-02 → 09-15 | `freetoken-frank-small.md` |
 | FastMTP patched llama.cpp | Dankserver | retired 2026-08-18 | `qwen3.8-spec-matrix.md` |
 | ik_llama.cpp Ornith CRACK | Dankserver | retired 2026-09-02 | `ikllama-ornith-dankserver.md` |
 | llama.cpp layer split + DFlash2 (`frank-medium`) | Dankserver | 2026-09-16 → 09-20 | `frank-medium-llamacpp-dflash2.md` |

autoresearch.sh

@@ -1,117 +1,26 @@
 #!/usr/bin/env bash
-# autoresearch.sh — canonical benchmark entrypoint: speed of frank-primary.
+# autoresearch.sh — canonical frank-primary benchmark entry point (dev box).
 #
-# Runs on the operator box and drives one bench window on DANKSERVER:
-#   1. fingerprint the candidate trio (env + serve script + unit)
-#   2. memo fast-path: an identical fingerprint that already produced a FULLY
-#      successful window (bench + verified production restore) returns the
-#      cached metrics without touching the fleet
-#   3. stage the candidate trio plus the measure client into
-#      /home/dank/.autoresearch-frank-primary/ (ephemeral; the repo checkout
-#      on the host stays the production source of truth)
-#   4. run the window (scripts/autoresearch/frank-primary-window.sh):
-#      stop + mask the production unit, boot the staged config AS THE SERVICE
-#      (same serve script, same container, same port 7770), wait for health,
-#      run the in-container usage-counter measure, then restore the production
-#      unit and verify its health — AGENTS.md Benchmarking Policy
-#   5. print METRIC lines (parse by run_experiment)
-#
-# Primary metric: decode_tps (HIGHER is better) — median across the three
-# prompt depths (115/1056/4096) of per-depth medians of
-# completion_tokens / (wall - ttft), 3 runs each after a warm pass,
-# temperature 0, the campaign's deterministic word-seed workload.
-#
-# Exit 0 with METRIC lines on success; non-zero on any failure (the window
-# script's EXIT trap restores production on every path).
-set -euo pipefail
+# The service runs on dankserver (192.168.0.235:7770), the two R9700s live
+# there, and the benchmark must own those cards, so the whole stop -> boot
+# scratch -> bench -> restore sequence runs ON dankserver (see
+# scripts/autoresearch-frank-primary.sh). This wrapper:
+#   1. copies the candidate config (the repo's scripts/frank-primary.env, the
+#      file the autoresearch loop edits) and the bench script to dankserver,
+#   2. runs the bench over ssh, relaying its stdout (the METRIC lines) and
+#      exit code back here verbatim.
+# It deliberately does NOT push to origin: the scp dirties dankserver's
+# working tree, and the loop's uncommitted candidate config must not be
+# pushed. Only this file and scripts/autoresearch-frank-primary.sh are
+# committed; the candidate env rides over ssh each run.
+set -uo pipefail
+DIR="$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)"
+REMOTE=dankserver
+REMOTE_REPO=/home/dank/serverAppConfigs
 
-SCRIPT_DIR="$(cd -- "$(dirname -- "${BASH_SOURCE[0]}")" && pwd)"
-HOST="${FRANK_PRIMARY_AR_HOST:-dankserver}"
-STAGE="/home/dank/.autoresearch-frank-primary"
+scp -q "$DIR/scripts/frank-primary.env" \
+    "$REMOTE:$REMOTE_REPO/scripts/frank-primary.env" || { echo "scp env failed" >&2; exit 1; }
+scp -q "$DIR/scripts/autoresearch-frank-primary.sh" \
+    "$REMOTE:$REMOTE_REPO/scripts/autoresearch-frank-primary.sh" || { echo "scp script failed" >&2; exit 1; }
 
-# FRANK_PRIMARY_AR_ENV_FILE stages a DIFFERENT env than the repo's — that is how
-# the arms driver A/Bs engine knobs without mutating the checkout. The
-# fingerprint covers whatever file is staged, so the memo cannot cross arms.
-ENV_FILE="${FRANK_PRIMARY_AR_ENV_FILE:-$SCRIPT_DIR/scripts/frank-primary.env}"
-SERVE="$SCRIPT_DIR/scripts/serve-frank-primary.sh"
-UNIT="$SCRIPT_DIR/scripts/frank-primary.service"
-MEASURE="$SCRIPT_DIR/scripts/autoresearch/frank-primary-measure.py"
-WINDOW="$SCRIPT_DIR/scripts/autoresearch/frank-primary-window.sh"
-
-for f in "$ENV_FILE" "$SERVE" "$UNIT" "$MEASURE" "$WINDOW"; do
-    [ -f "$f" ] || { echo "[autoresearch] missing harness file: $f" >&2; exit 1; }
-done
-command -v python3 >/dev/null 2>&1 || { echo "[autoresearch] python3 required on operator box" >&2; exit 1; }
-
-# Fingerprint of everything that can change the engine's argv or environment.
-HASH="$(cat "$ENV_FILE" "$SERVE" "$UNIT" | sha256sum | cut -d' ' -f1)"
-echo "[autoresearch] candidate fingerprint $HASH" >&2
-
-ssh_run() { ssh -o BatchMode=yes -o ConnectTimeout=15 "$HOST" "$@"; }
-
-emit_metrics() { # $1 = run.json
-    python3 -c '
-import json, sys
-run = json.loads(sys.argv[1])
-m = run["measure"]
-print("METRIC decode_tps=%s" % m["decode_tps"])
-for k in ("decode_tps_115", "decode_tps_1056", "decode_tps_4096"):
-    print("METRIC %s=%s" % (k, m[k]))
-print("METRIC ttft_ms_4096=%s" % m["ttft_ms_4096"])
-print("METRIC kv_tokens=%s" % (run.get("kv_tokens") or 0))
-print("METRIC boot_seconds=%s" % (run.get("boot_seconds") or -1))
-print("METRIC completion_tokens_4096=%s" % m.get("completion_tokens_4096", 0))
-# Any extra sampling temperature the measure swept (AR_TEMPS), e.g. the 0.7
-# arm — that is where speculative-decode acceptance shows up, since a greedy
-# draft matches a greedy target exactly and is rejected more under real
-# sampling. Suffixed keys only (_t0.7), so the historical lines are untouched.
-for k in sorted(m):
-    if "_t" in k and (k.startswith("decode_tps") or k.startswith("ttft_ms")):
-        print("METRIC %s=%s" % (k, m[k]))
-' "$1"
-}
-
-# --- memo fast-path -------------------------------------------------------
-cached_hash="$(ssh_run "cat '$STAGE/last/hash' 2>/dev/null" 2>/dev/null || true)"
-if [ -n "$cached_hash" ] && [ "$cached_hash" = "$HASH" ]; then
-    cached_run="$(ssh_run "cat '$STAGE/last/run.json' 2>/dev/null" || true)"
-    if [ -n "$cached_run" ]; then
-        echo "[autoresearch] memo hit — cached metrics for $HASH, fleet untouched" >&2
-        emit_metrics "$cached_run"
-        exit 0
-    fi
-fi
-
-# --- stage the candidate ----------------------------------------------------
-echo "[autoresearch] staging candidate on $HOST:$STAGE" >&2
-ssh_run "mkdir -p '$STAGE'"
-scp -q "$ENV_FILE" "$HOST:$STAGE/frank-primary.env"
-scp -q "$SERVE" "$HOST:$STAGE/serve.sh"
-scp -q "$MEASURE" "$HOST:$STAGE/measure.py"
-scp -q "$WINDOW" "$HOST:$STAGE/window.sh"
-ssh_run "printf '%s\n' '$HASH' > '$STAGE/hash'"
-
-# --- the window -------------------------------------------------------------
-# Budgets: bench boot up to 20 min (warm boots measured ~7 min), measure up
-# to 30 min, restore boot up to 20 min — call run_experiment with
-# timeout_seconds >= 3600.
-echo "[autoresearch] opening bench window (stop+mask -> boot -> measure -> restore)" >&2
-set +e
-ssh_run "bash '$STAGE/window.sh'"
-rc=$?
-set -e
-if [ "$rc" -ne 0 ]; then
-    echo "[autoresearch] window failed (rc=$rc); host-side evidence:" >&2
-    ssh_run "tail -60 '$STAGE/fail.log' 2>/dev/null || tail -40 '$STAGE/serve.out' 2>/dev/null || tail -40 '$STAGE/measure.out' 2>/dev/null || true" >&2 || true
-    exit 1
-fi
-
-run_json="$(ssh_run "cat '$STAGE/run.json'")" || {
-    echo "[autoresearch] could not read $STAGE/run.json back" >&2; exit 1; }
-echo "[autoresearch] window result:" >&2
-printf '%s\n' "$run_json" >&2
-emit_metrics "$run_json"
-
-echo "[autoresearch] engine evidence (plan / compile path / spec counters):" >&2
-ssh_run "cat '$STAGE/evidence.log' 2>/dev/null" >&2 || true
-exit 0
+ssh "$REMOTE" "bash $REMOTE_REPO/scripts/autoresearch-frank-primary.sh $REMOTE_REPO/scripts/frank-primary.env"

dankclaw/Caddyfile

@@ -337,7 +337,16 @@ http://:8017 {
 			try_files /docs.html
 			file_server
 		}
-		@data path /services.json /status.json /app.js /style.css /deployed.js
+		# /whoami echoes the outpost-injected identity headers so the page
+		# can gate the admin button client-side (the /admin* 403 above stays
+		# the real gate; this only decides what is shown). The outpost strips
+		# client-supplied X-Authentik-* before injecting, so this is spoof-proof.
+		handle /whoami {
+			templates
+			header Content-Type application/json
+			respond "{\"user\":\"{header.X-Authentik-Username}\",\"groups\":\"{header.X-Authentik-Groups}\"}"
+		}
+		@data path /services.json /status.json /app.js /style.css /deployed.js /img/* /sfx/*
 		handle @data {
 			file_server
 		}

dankclaw/comfyui/workflows/qwen-image-2.1-t2i-art.json

@@ -0,0 +1,64 @@
+{
+  "3": {
+    "class_type": "CLIPTextEncode",
+    "_meta": { "title": "Positive prompt" },
+    "inputs": {
+      "text": "A detailed illustration of a 1930s Soviet industrial design bureau at night: a well-measured engineer at a drafting table, a warm desk lamp, rolled blueprints, gears and calipers on the desk, deep red and cream palette, gouache painterly style, intricate mechanical detail, soft shadows",
+      "clip": [ "6", 0 ]
+    }
+  },
+  "4": {
+    "class_type": "CLIPTextEncode",
+    "_meta": { "title": "Negative prompt" },
+    "inputs": {
+      "text": "lowres, blurry, deformed, extra limbs, watermark, signature, jpeg artifacts",
+      "clip": [ "6", 0 ]
+    }
+  },
+  "5": {
+    "class_type": "UNETLoader",
+    "_meta": { "title": "Qwen Image 2.1 (int8)" },
+    "inputs": { "unet_name": "qwen_image_2.1_int8_convrot.safetensors", "weight_dtype": "default" }
+  },
+  "6": {
+    "class_type": "CLIPLoader",
+    "_meta": { "title": "Qwen3-VL 8B text encoder (w4a8)" },
+    "inputs": { "clip_name": "qwen3vl_8b_w4a8.safetensors", "type": "qwen_image" }
+  },
+  "7": {
+    "class_type": "VAELoader",
+    "_meta": { "title": "VAE" },
+    "inputs": { "vae_name": "qwen_image_2.1_vae_bf16.safetensors" }
+  },
+  "8": {
+    "class_type": "EmptySD3LatentImage",
+    "_meta": { "title": "Empty Latent 1024x1024" },
+    "inputs": { "width": 1024, "height": 1024, "batch_size": 1 }
+  },
+  "9": {
+    "class_type": "KSampler",
+    "_meta": { "title": "KSampler" },
+    "inputs": {
+      "model": [ "5", 0 ],
+      "seed": 1930,
+      "steps": 20,
+      "cfg": 1.0,
+      "sampler_name": "euler",
+      "scheduler": "simple",
+      "positive": [ "3", 0 ],
+      "negative": [ "4", 0 ],
+      "latent_image": [ "8", 0 ],
+      "denoise": 1.0
+    }
+  },
+  "10": {
+    "class_type": "VAEDecode",
+    "_meta": { "title": "VAE Decode" },
+    "inputs": { "samples": [ "9", 0 ], "vae": [ "7", 0 ] }
+  },
+  "11": {
+    "class_type": "SaveImage",
+    "_meta": { "title": "Save Image (art)" },
+    "inputs": { "filename_prefix": "qwen21-art", "images": [ "10", 0 ] }
+  }
+}

dankclaw/comfyui/workflows/qwen-image-2.1-t2i-legible-text.json

@@ -0,0 +1,64 @@
+{
+  "3": {
+    "class_type": "CLIPTextEncode",
+    "_meta": { "title": "Positive prompt (legible text)" },
+    "inputs": {
+      "text": "Clean minimalist movie-poster design, dark charcoal background with a subtle vintage grain. Centered large bold white sans-serif text reading \"SOLIDARITY OF THE SOVIET COMPUTER SYSTEMS\", perfectly legible, crisp letterforms, even letter spacing. Below it, a smaller line of yellow text reading \"DANKCLAW DANKSERVER EST 2026\", equally legible. Flat graphic design, high contrast, generous margins, no other words, no watermark",
+      "clip": [ "6", 0 ]
+    }
+  },
+  "4": {
+    "class_type": "CLIPTextEncode",
+    "_meta": { "title": "Negative prompt" },
+    "inputs": {
+      "text": "blurry, illegible, misspelled, warped letters, extra text, watermark, signature, lowres, jpeg artifacts",
+      "clip": [ "6", 0 ]
+    }
+  },
+  "5": {
+    "class_type": "UNETLoader",
+    "_meta": { "title": "Qwen Image 2.1 (int8)" },
+    "inputs": { "unet_name": "qwen_image_2.1_int8_convrot.safetensors", "weight_dtype": "default" }
+  },
+  "6": {
+    "class_type": "CLIPLoader",
+    "_meta": { "title": "Qwen3-VL 8B text encoder (w4a8)" },
+    "inputs": { "clip_name": "qwen3vl_8b_w4a8.safetensors", "type": "qwen_image" }
+  },
+  "7": {
+    "class_type": "VAELoader",
+    "_meta": { "title": "VAE" },
+    "inputs": { "vae_name": "qwen_image_2.1_vae_bf16.safetensors" }
+  },
+  "8": {
+    "class_type": "EmptySD3LatentImage",
+    "_meta": { "title": "Empty Latent 832x1216 poster" },
+    "inputs": { "width": 832, "height": 1216, "batch_size": 1 }
+  },
+  "9": {
+    "class_type": "KSampler",
+    "_meta": { "title": "KSampler" },
+    "inputs": {
+      "model": [ "5", 0 ],
+      "seed": 2026,
+      "steps": 25,
+      "cfg": 1.0,
+      "sampler_name": "euler",
+      "scheduler": "simple",
+      "positive": [ "3", 0 ],
+      "negative": [ "4", 0 ],
+      "latent_image": [ "8", 0 ],
+      "denoise": 1.0
+    }
+  },
+  "10": {
+    "class_type": "VAEDecode",
+    "_meta": { "title": "VAE Decode" },
+    "inputs": { "samples": [ "9", 0 ], "vae": [ "7", 0 ] }
+  },
+  "11": {
+    "class_type": "SaveImage",
+    "_meta": { "title": "Save Image (text poster)" },
+    "inputs": { "filename_prefix": "qwen21-text", "images": [ "10", 0 ] }
+  }
+}

dankclaw/hermes/config.yaml.example

@@ -10,15 +10,15 @@ model:
   provider: dankclaw-large
   base_url: http://192.168.0.143:7770/v1
   api_key: not-needed
-  default: frank-large
+  default: frank-dragoon
 
 providers:
   dankclaw-large:
     api: http://192.168.0.143:7770/v1
     discover_models: false
     models:
-      frank-large:
-        context_length: 114688
+      frank-dragoon:
+        context_length: 262144
   dankserver-small:
     api: http://192.168.0.235:7770/v1
     discover_models: false
@@ -29,7 +29,7 @@ providers:
 # Auxiliary routing from the live configuration. Named providers route via
 # their `api:` URL; `custom` entries route by their explicit `base_url`.
 auxiliary:
-  vision: {provider: dankclaw-large, model: frank-large}
+  vision: {provider: dankclaw-large, model: frank-dragoon}
   web_extract: {provider: dankserver-small, model: frank-medium}
   compression: {provider: dankserver-small, model: frank-medium}
   skills_hub: {provider: dankserver-small, model: frank-medium}
@@ -50,8 +50,8 @@ auxiliary:
         effort: none
   memory_query_rewrite: {provider: custom, model: frank-medium, base_url: http://192.168.0.235:7770/v1}
   tts_audio_tags: {provider: custom, model: frank-medium, base_url: http://192.168.0.235:7770/v1}
-  triage_specifier: {provider: dankclaw-large, model: frank-large}
-  kanban_decomposer: {provider: dankclaw-large, model: frank-large}
+  triage_specifier: {provider: dankclaw-large, model: frank-dragoon}
+  kanban_decomposer: {provider: dankclaw-large, model: frank-dragoon}
   profile_describer: {provider: dankserver-small, model: frank-medium}
   curator: {provider: dankserver-small, model: frank-medium}
 

old/download-frank-large-mimo.sh

@@ -4,7 +4,7 @@
 # command line (a Bearer token in argv is world-readable via /proc).
 #
 # Per-file FRESH downloads, no resume: the house rule (see
-# download-frank-small.sh); -f/--fail-with-body plus size+sha256 against the
+# download-frank-secondary.sh); -f/--fail-with-body plus size+sha256 against the
 # Hub LFS oid is what makes an access error fail instead of "succeeding".
 #
 # Usage:

old/download-frank-large-qwen35.sh

@@ -2,7 +2,7 @@
 # Download the source Qwen3.5-9B checkpoint — the ONLY place this model
 # family's MTP head (15 mtp.* tensors) exists. frank-large runs it as the
 # vLLM cross-model MTP draft (MiMo-V2.6-Distill-Qwen-9B itself ships zero
-# mtp tensors; that is why SGLang's NEXTN on frank-small draws 0.00
+# mtp tensors; that is why SGLang's NEXTN on frank-secondary draws 0.00
 # acceptance). Public repo — no auth header on the command line.
 #
 # NOTE the odd upstream shard names (model.safetensors-0000N-of-00004...).

old/download-frank-secondary-sglang-int8.sh

@@ -1,5 +1,5 @@
 #!/usr/bin/env bash
-# Download MiMo-V2.6-Distill-Qwen-9B-INT8-W8A16-AutoRound for frank-small
+# Download MiMo-V2.6-Distill-Qwen-9B-INT8-W8A16-AutoRound for frank-secondary
 # (SGLang, RTX 3090). Public repo — no auth header on the command line.
 #
 # Per-file FRESH downloads, no resume: `curl -C -` once produced a 213 MB
@@ -7,7 +7,7 @@
 # the house rule.
 #
 # Usage:
-#   scripts/download-frank-small.sh [DEST_DIR] [--verify-only]
+#   scripts/download-frank-secondary.sh [DEST_DIR] [--verify-only]
 #
 # LFS shards are verified against the Hub LFS sha256; non-LFS text files
 # against their declared byte size. Idempotent: complete files are skipped.

scripts/apply-protected-cast.sh

@@ -0,0 +1,43 @@
+#!/usr/bin/env bash
+# apply-protected-cast.sh — RUNS ON DANKSERVER. Rewrite the recipe's
+# PROTECTED-BEGIN/END section from Swift's OWN calibration grams: run
+# derive-protected-experts.py over the calib dir, splice its cast lines into
+# the marked section of the recipe in place, keep the pre-edit recipe in the
+# campaign trash. Refuses to run twice (marker text says derived).
+set -euo pipefail
+CALIB=${1:-/home/dank/corpora/calib}
+RECIPE=${2:-/home/dank/serverAppConfigs-swift15/scripts/recipes/qwen4exp-swift15-w4dn8-i8-hc8m.recipe}
+PY=/home/dank/venvs/radcorpus/bin/python
+DERIVE=/home/dank/serverAppConfigs-swift15/scripts/derive-protected-experts.py
+TS=$(date +%Y%m%d-%H%M%S)
+
+grep -q "derive-protected-experts.py rewrote" "$RECIPE" && {
+    echo "recipe already carries derived cast lines; refusing to re-derive" >&2; exit 1; }
+grep -q "PROTECTED-BEGIN" "$RECIPE" || { echo "no PROTECTED markers in $RECIPE" >&2; exit 1; }
+
+"$PY" "$DERIVE" "$CALIB" > "/tmp/protected-cast-$TS.txt"
+N=$(grep -c 'cast dtype=bf16' "/tmp/protected-cast-$TS.txt")
+[ "$N" -ge 2 ] || { echo "derive produced $N cast lines; refusing" >&2; exit 1; }
+echo "derived $N protected experts -> /tmp/protected-cast-$TS.txt"
+
+cp -f "$RECIPE" "/home/dank/.AGENTTRASH/$TS-qwen4exp-swift15-w4dn8-i8-hc8m.recipe"
+"$PY" - "$RECIPE" "/tmp/protected-cast-$TS.txt" <<'PYEOF'
+import sys
+recipe, castfile = sys.argv[1], sys.argv[2]
+lines = open(recipe).read().splitlines()
+cast = [l for l in open(castfile).read().splitlines() if 'cast dtype=bf16' in l]
+out, insec = [], False
+for l in lines:
+    if 'PROTECTED-BEGIN' in l:
+        insec = True
+        out.append(l)
+        out.extend(cast)
+    elif 'PROTECTED-END' in l:
+        insec = False
+        out.append(l)
+        out.append('# (derive-protected-experts.py rewrote this section from Swift\'s own grams)')
+    elif not insec:
+        out.append(l)
+open(recipe, 'w').write('\n'.join(out) + '\n')
+print(f'spliced {len(cast)} cast lines into {recipe}')
+PYEOF

scripts/autoresearch-bench.py

@@ -104,7 +104,7 @@ def stream_completion(url: str, model: str, prompt: str):
 def main():
     ap = argparse.ArgumentParser()
     ap.add_argument("--url", default="http://127.0.0.1:7770")
-    ap.add_argument("--model", default="frank-large")
+    ap.add_argument("--model", default="frank-dragoon")
     args = ap.parse_args()
 
     with urllib.request.urlopen(args.url.rstrip("/") + "/v1/models",

scripts/autoresearch-frank-primary.sh

@@ -0,0 +1,187 @@
+#!/usr/bin/env bash
+# autoresearch-frank-primary.sh — RUNS ON DANKSERVER, driven over ssh by the
+# dev-box autoresearch.sh. It is the canonical frank-primary benchmark entry
+# point for the autoresearch loop.
+#
+# Method (mandated by the Benchmarking Policy): the production service serves
+# real users and shares the two R9700s, so a bench beside it starves the GPUs
+# and corrupts results. This script therefore:
+#   1. stops the production frank-primary unit (root, relink symlink: stop,
+#      NOT mask — mask refuses to replace a symlinked unit file),
+#   2. boots a scratch radiance container named `arbench` on the SAME two
+#      cards with the candidate engine argv (the env file the loop edits),
+#   3. polls /health until the listener is live (it binds AFTER the full
+#      load, so a bare curl is an honest gate),
+#   4. runs ONE self-contained completion request locally (127.0.0.1, no LAN
+#      jitter): a fixed ~140k-char real-text prompt (~32k tokens) decoded for
+#      256 tokens, greedy. The engine's `timings` object reports the honest
+#      prefill and decode rates for that single request.
+#   5. prints METRIC lines and tears the scratch down,
+#   6. restores production (systemctl start), via a trap so a failed bench
+#      never leaves frank-primary down (the stale-probe-tail lesson).
+#
+# Determinism: the prompt is built from a fixed, sorted corpus of repo files
+# with a fixed random seed, so prompt_n is stable across runs. Sampling is
+# greedy (temperature 0), so the decode length is fixed; the only residual
+# nondeterminism is the wht6 lossy TP all-reduce, which is a constant across
+# runs and cancels in A/B comparisons.
+#
+# The scratch mirrors the production container contract exactly (serve-frank-
+# primary.sh): --network host, --init, seccomp=unconfined (io_uring for the
+# n-gram table), memlock unbounded (the ~132 GiB pinned ledger), /dev/kfd +
+# /dev/dri, the .rad ro, the state dir rw, the tune cache mount. The ONLY
+# differences: the container name (arbench, so prod's identity is untouched)
+# and a FRESH prefix-cache disk tier (arbench/, wiped per run) so the cold
+# prefill is cold — the prod tier at kvcache/frank-primary is left intact.
+set -uo pipefail
+
+ENV_FILE="${1:-/home/dank/serverAppConfigs/scripts/frank-primary.env}"
+SCRATCH=arbench
+PORT=7770
+HEALTH="http://127.0.0.1:$PORT/health"
+COMPLETE="http://127.0.0.1:$PORT/v1/completions"
+BODY=/tmp/arbench-body.json
+LOG=/home/dank/logs/arbench.log
+
+restore() {
+  podman rm -f "$SCRATCH" >/dev/null 2>&1 || true
+  sudo systemctl start frank-primary >/dev/null 2>&1 || true
+}
+trap restore EXIT
+
+# --- source the candidate config (same file the production unit reads) ------
+if [ ! -f "$ENV_FILE" ]; then
+  echo "no env file at $ENV_FILE" >&2; exit 1
+fi
+# shellcheck disable=SC1090
+set -a; . "$ENV_FILE"; set +a
+
+# --- stop production; the scratch needs both cards to itself ----------------
+sudo systemctl stop frank-primary >/dev/null 2>&1 || true
+podman rm -f "$SCRATCH" >/dev/null 2>&1 || true
+
+# --- fresh prefix-cache tier for a cold prefill (prod's tier untouched) ------
+FRESH_KV="$FRANK_PRIMARY_STATE_DIR/kvcache/$SCRATCH"
+rm -rf "$FRESH_KV"; mkdir -p "$FRESH_KV"
+mkdir -p "$FRANK_PRIMARY_STATE_DIR/tune"
+
+# --- boot the scratch: the production run line, scratch name + fresh KV ------
+podman run -d --name "$SCRATCH" \
+  --network host \
+  --init \
+  --security-opt seccomp=unconfined \
+  --ulimit memlock=-1:-1 \
+  --device /dev/kfd --device /dev/dri \
+  -v "$FRANK_PRIMARY_MODEL_DIR":/models:ro \
+  -v "$FRANK_PRIMARY_STATE_DIR":/data \
+  -v "$FRANK_PRIMARY_STATE_DIR/tune":/opt/radiance/share/radiance/tune \
+  -v "$FRANK_PRIMARY_TEMPLATE":/templates/template.jinja:ro \
+  "$FRANK_PRIMARY_IMAGE" \
+  --model "/models/$FRANK_PRIMARY_MODEL_FILE" \
+  --host :: --port "$PORT" \
+  --served-model-name "$FRANK_PRIMARY_SERVED_NAME" \
+  --override-chat-template /templates/template.jinja \
+  --tp "$FRANK_PRIMARY_TP" --tp-wire "$FRANK_PRIMARY_TP_WIRE" \
+  --max-num-seqs "$FRANK_PRIMARY_MAX_SEQS" \
+  --max-model-len "$FRANK_PRIMARY_MAX_MODEL_LEN" \
+  --placement "$FRANK_PRIMARY_PLACEMENT" \
+  --host-pool-mib "$FRANK_PRIMARY_HOST_POOL_MIB" \
+  --gpu-headroom-mib "$FRANK_PRIMARY_GPU_HEADROOM_MIB" \
+  --vram-weights-mib "$FRANK_PRIMARY_VRAM_WEIGHTS_MIB" \
+  --vram-kv-mib "$FRANK_PRIMARY_VRAM_KV_MIB" \
+  --kv-cache-dtype "$FRANK_PRIMARY_KV_DTYPE" \
+  --prefix-cache-host-mib "$FRANK_PRIMARY_PREFIX_HOST_MIB" \
+  --prefix-cache-dir "/data/kvcache/$SCRATCH" \
+  --prefix-cache-disk-mib "$FRANK_PRIMARY_PREFIX_DISK_MIB" \
+  --ngram-placement "$FRANK_PRIMARY_NGRAM_PLACEMENT" \
+  --num-speculative-tokens "$FRANK_PRIMARY_SPEC_TOKENS" \
+  --max-num-batched-tokens "$FRANK_PRIMARY_MAX_BATCHED_TOKENS" \
+  ${FRANK_PRIMARY_EXTRA_ARGS:+$FRANK_PRIMARY_EXTRA_ARGS} \
+  > "$LOG" 2>&1
+if [ $? -ne 0 ]; then echo "scratch boot failed; see $LOG" >&2; exit 1; fi
+
+# --- poll health (listener binds after the full load; cold boot ~110 s) ------
+ok=0
+for _ in $(seq 1 360); do
+  code=$(curl -s -o /dev/null -w "%{http_code}" --max-time 3 "$HEALTH" 2>/dev/null || echo 000)
+  if [ "$code" = "200" ]; then ok=1; break; fi
+  sleep 2
+done
+if [ "$ok" -ne 1 ]; then
+  echo "scratch never reached health 200; tail of $LOG:" >&2
+  tail -30 "$LOG" >&2; exit 1
+fi
+
+# --- build the workload: a fixed ~140k-char real-text prompt (repo corpus) ---
+python3 - "$BODY" <<'PY'
+import json, sys, random, glob
+random.seed(1234)
+root = "/home/dank/serverAppConfigs"
+files = sorted(
+    glob.glob(f"{root}/serverAppConfigs-docs/**/*.md", recursive=True)
+    + glob.glob(f"{root}/scripts/*.sh")
+    + glob.glob(f"{root}/scripts/*.py")
+    + glob.glob(f"{root}/AGENTS.md")
+)
+corpus = []
+total = 0
+for f in files:
+    try:
+        t = open(f, encoding="utf-8", errors="ignore").read()
+    except Exception:
+        continue
+    if not t.strip():
+        continue
+    corpus.append(t)
+    total += len(t)
+    if total > 240000:
+        break
+corpus = "\n".join(corpus)
+TGT = 140000
+parts, n = [], 0
+while n < TGT:
+    start = random.randint(0, max(0, len(corpus) - 500))
+    seg = corpus[start:start + random.randint(200, 500)]
+    parts.append(seg)
+    n += len(seg) + 1
+prompt = " ".join(parts)
+json.dump(
+    {"model": "frank-archon", "prompt": prompt,
+     "max_tokens": 256, "temperature": 0.0, "stream": False,
+     "stop": ["<|im_end|>"]},
+    open(sys.argv[1], "w"),
+)
+PY
+if [ ! -f "$BODY" ]; then echo "prompt build failed" >&2; exit 1; fi
+
+# --- the bench request (prefill ~32k + decode 256, greedy) -------------------
+RESP=$(curl -s --max-time 600 -X POST "$COMPLETE" \
+  -H 'Content-Type: application/json' -d "@$BODY")
+echo "$RESP" > /tmp/arbench-resp.json
+
+# --- parse the timings into METRIC lines -------------------------------------
+python3 - <<'PY'
+import json, sys
+try:
+    r = json.load(open("/tmp/arbench-resp.json"))
+except Exception as e:
+    sys.stderr.write(f"unparseable response: {e}\n"); sys.exit(1)
+if "timings" not in r:
+    sys.stderr.write(f"no timings in response: {json.dumps(r)[:500]}\n"); sys.exit(1)
+t = r["timings"]
+dtps = t.get("predicted_per_second")
+ptps = t.get("prompt_per_second")
+pms  = t.get("prompt_ms")
+dn   = t.get("draft_n", 0)
+da   = t.get("draft_n_accepted", 0)
+if dtps is None:
+    sys.stderr.write("predicted_per_second null (no decode produced)\n"); sys.exit(1)
+print(f"METRIC decode_tps={dtps:.4f}")
+if ptps is not None:
+    print(f"METRIC prefill_tps={ptps:.4f}")
+if pms is not None:
+    print(f"METRIC ttft_ms={pms:.4f}")
+if dn:
+    print(f"METRIC draft_accept={da/dn:.4f}")
+print(f"INFO prompt_n={t.get('prompt_n')} cache_n={t.get('cache_n')} predicted_n={t.get('predicted_n')}")
+PY

scripts/autoresearch/frank-primary-measure.py

@@ -45,7 +45,7 @@ import urllib.error
 import urllib.request
 
 BASE = os.environ.get("AR_BASE", "http://127.0.0.1:7770")
-MODEL = os.environ.get("AR_MODEL", "frank-primary")
+MODEL = os.environ.get("AR_MODEL", "frank-archon")
 PROMPT_CEILING = int(os.environ.get("AR_PROMPT_CEILING", "100000"))
 DEPTHS = (115, 1056, 4096)
 WARM_TOKENS = 64

scripts/bench-decode.sh

@@ -57,7 +57,7 @@ run_bench() {
     # Warmup (small request)
     curl -s -m 60 http://127.0.0.1:$PORT/v1/chat/completions \
         -H "Content-Type: application/json" \
-        -d '{"model":"frank-large","messages":[{"role":"user","content":"Hi"}],"max_tokens":16,"stream":false}' > /dev/null 2>&1 || true
+        -d '{"model":"frank-dragoon","messages":[{"role":"user","content":"Hi"}],"max_tokens":16,"stream":false}' > /dev/null 2>&1 || true
     
     # Build 8K prompt
     local prompt

scripts/bench-probe.py

@@ -0,0 +1,48 @@
+#!/usr/bin/env python3
+"""bench-probe.py — RUNS ON DANKSERVER. The campaign's decode probe: one
+/completion request, a prompt assembled from calib.jsonl to ~TARGET tokens,
+temperature 0, max_tokens 512. Reports the engine's own timings: decode rate,
+prefill rate, draft accept. Run 3x; the median is the arm's number."""
+import json, sys, urllib.request
+
+TARGET = 43600  # tokens
+BYTES_PER_TOK = 3.8
+URL = "http://localhost:7770/v1/completions"
+
+def build_prompt():
+    want = int(TARGET * BYTES_PER_TOK)
+    parts = []
+    n = 0
+    with open("/home/dank/corpora/out/calib.jsonl") as f:
+        for line in f:
+            d = json.loads(line)
+            p = d.get("prompt") or d.get("messages", [{}])[0].get("content", "")
+            parts.append(p)
+            n += len(p.encode())
+            if n >= want:
+                break
+    return "".join(parts)
+
+def probe(salt):
+    prompt = f"[probe {salt}]\n" + build_prompt()
+    body = json.dumps({
+        "model": "frank-archon",
+        "prompt": prompt, "temperature": 0.0, "max_tokens": 512,
+    }).encode()
+    req = urllib.request.Request(URL, data=body,
+        headers={"Content-Type": "application/json"})
+    with urllib.request.urlopen(req, timeout=1800) as r:
+        d = json.load(r)
+    t = d.get("timings", {})
+    return {
+        "prompt_n": t.get("prompt_n"),
+        "prefill_tps": round(t.get("prompt_per_second", 0), 1),
+        "pred_n": t.get("predicted_n"),
+        "decode_tps": round(t.get("predicted_per_second", 0), 2),
+        "draft_n": t.get("draft_n"),
+        "accept": (round(t["draft_n_accepted"] / t["draft_n"], 4)
+                   if t.get("draft_n") else None),
+    }
+
+if __name__ == "__main__":
+    print(json.dumps(probe(sys.argv[1] if len(sys.argv) > 1 else "a")))

scripts/bench-threads-batch-http.py

@@ -39,7 +39,7 @@ def send_prefill_request():
     with open(PROMPT_FILE, "r") as f:
         prompt = f.read()
     body = json.dumps({
-        "model": "frank-large",
+        "model": "frank-dragoon",
         "messages": [{"role": "user", "content": prompt}],
         "max_tokens": 1,
         "stream": False

scripts/build-frank-primary-rad.sh

@@ -0,0 +1,166 @@
+#!/usr/bin/env bash
+# build-frank-primary-rad.sh — build frank-primary's .rad container from the
+# OFFICIAL Qwen/Qwen3.8-Flash-Next-FP8 checkpoint, at NATIVE precision.
+#
+#   bash scripts/build-frank-primary-rad.sh                # convert
+#   bash scripts/build-frank-primary-rad.sh --plan-only    # declare, no I/O
+#   bash scripts/build-frank-primary-rad.sh --restage      # redo staging
+#   bash scripts/build-frank-primary-rad.sh --force        # rebuild existing
+#
+# WHY A STAGING DIR (the one non-obvious step): rad-convert selects its
+# architecture plugin by the checkpoint's quantisation DESCRIPTOR —
+# derived from config.json's quantization_config (core/format/checkpoint.cpp:
+# "fp8" here). No shipped plugin claims (qwen4exp, "fp8"): the shipped
+# qwen4exp plugin claims (qwen4exp, ""), the descriptor recipe-built
+# containers carry (they are built from the bf16 upstream, which has no
+# quantization_config at all). The match is exact with no fallback, so an
+# unmodified convert refuses before declare. The descriptor is only a
+# selection key; per-weight encodings are data — the checkpoint reader
+# derives them from the tensors themselves (X.weight_scale_inv is the
+# scale plane; checkpoint.h). Staging drops the quantization_config key
+# (everything else is a relative symlink into the pristine download) so
+# the convert selects the shipped plugin. EVERY weight is then copied through
+# in its own encoding — block-FP8 codes + bf16 scale planes stay block-FP8 —
+# except TWO value-preserving reshapes that stack-qwen38-official-fp8.py
+# performs on the staging dir first, and ONE recipe rule that finishes the
+# PLE table:
+#   * experts: the plugin maps the STACKED layout (mlp.experts.gate_up_proj
+#     [E,2F,H] with the _scale_inv plane at the same index) while the official
+#     repo ships per-expert unfused tensors — staging byte-stacks them
+#     (gate rows first, no requantisation).
+#   * PLE: the plugin's fp8 branch wants the table codes beside a ONE-ELEMENT
+#     bf16 scale plane; the official repo ships 128 SCALELESS fp8 shards plus
+#     one bf16 scalar. Staging decodes to bf16 with the checkpoint's own
+#     scale, and the convert's rtn rule re-encodes at that same scale —
+#     every original code is restored exactly and the scale plane lands in
+#     the form the plugin declares.
+#
+# WHY A CONTAINER AT ALL: a checkpoint dir has no mapped/file tiers, and a
+# weight only a host kernel reads (the n-gram / PLE table, 51B params)
+# cannot be served from one (GUIDE §3.4). rad-convert also bakes the
+# tokenizer and the checkpoint's chat template; the runtime replaces the
+# template with froggeric's (--override-chat-template).
+#
+# The convert runs rad-convert inside the digest-pinned radiance image
+# (the host has no ROCm and needs none): /home/dank/models is mounted at
+# /models so the staging dir's RELATIVE symlinks resolve inside the
+# container. Output lands beside the raw download, owned by dank
+# (rootless podman maps container-root to the invoking user).
+set -euo pipefail
+
+SRC="/home/dank/models/Qwen/Qwen3.8-Flash-Next-FP8"
+STAGE="/home/dank/models/stage-qwen4exp-fp8"
+OUT_DIR="/home/dank/models/rad"
+OUT="$OUT_DIR/qwen4exp-fp8-native.rad"
+
+SCRIPT_DIR="$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)"
+# shellcheck disable=SC1091
+source "$SCRIPT_DIR/frank-primary.env"
+IMAGE="$FRANK_PRIMARY_IMAGE"
+
+plan_only=0 restage=0 force=0
+for a in "$@"; do
+    case "$a" in
+        --plan-only) plan_only=1 ;;
+        --restage)   restage=1 ;;
+        --force)     force=1 ;;
+        *) echo "unknown flag: $a" >&2; exit 2 ;;
+    esac
+done
+
+if [ ! -f "$SRC/config.json" ]; then
+    echo "ERROR: raw checkpoint missing: $SRC (run scripts/download-frank-primary.sh --fetch)" >&2
+    exit 1
+fi
+echo "verifying raw checkpoint first..."
+bash "$SCRIPT_DIR/download-frank-primary.sh"
+
+if [ -f "$OUT" ] && [ "$plan_only" -eq 0 ] && [ "$force" -eq 0 ]; then
+    echo "container already exists: $OUT"
+    echo "rebuild it with --force (the old file moves to ~/.AGENTTRASH)"
+    exit 0
+fi
+
+# ---- staging ----------------------------------------------------------------
+if [ "$restage" -eq 1 ] && [ -e "$STAGE" ]; then
+    TRASH="$HOME/.AGENTTRASH/$(date +%Y%m%d)-frank-primary-stage"
+    mkdir -p "$TRASH"
+    mv "$STAGE" "$TRASH/stage-$(date +%H%M%S)"
+    echo "old staging dir moved to $TRASH"
+fi
+if [ ! -f "$STAGE/config.json" ]; then
+    mkdir -p "$STAGE"
+    python3 - "$SRC/config.json" "$STAGE/config.json" <<'PY'
+import json, sys
+src, dst = sys.argv[1], sys.argv[2]
+cfg = json.load(open(src))
+dropped = cfg.pop("quantization_config", None)
+json.dump(cfg, open(dst, "w"), indent=2)
+print(f"staging config.json written (quantization_config dropped: {dropped is not None})")
+PY
+    # The index is COPIED like config.json, never symlinked: the reshape step
+    # rewrites the stage's index, and a symlink would write through to the
+    # raw checkpoint (measured 2026-10-05: the raw index was restamped with
+    # the stacked layout and the raw verify failed 143/144). Shards stay
+    # relative symlinks — relative so the targets resolve inside the convert
+    # container (the stage's parent is mounted at /models).
+    for f in "$SRC"/*; do
+        name="$(basename "$f")"
+        case "$name" in config.json|model.safetensors.index.json) continue ;; esac
+        ln -sfn "../Qwen/Qwen3.8-Flash-Next-FP8/$name" "$STAGE/$name"
+    done
+    cp "$SRC/model.safetensors.index.json" "$STAGE/"
+    echo "staging dir ready: $STAGE ($(ls "$STAGE" | wc -l) entries)"
+fi
+
+# ---- checkpoint reshape (stacked experts + bf16 PLE table) -------------------
+# stack-qwen38-official-fp8.py reshapes the staged checkpoint in place
+# (idempotent) and writes the PLE scale to $STAGE/ple_scale.txt — see the
+# script's docstring for why each transform is value-preserving.
+echo "stacking experts + merging the PLE table (value-preserving)..."
+python3 "$SCRIPT_DIR/stack-qwen38-official-fp8.py" --stage "$STAGE"
+PLE_SCALE="$(tr -d '\n' < "$STAGE/ple_scale.txt")"
+echo "PLE fixed scale: $PLE_SCALE"
+
+# ---- convert ----------------------------------------------------------------
+# rad-convert needs no GPU: declare resolves kernels and asks each about the
+# encoding it will read — CPU-side layout questions only. (If a plugin load
+# ever demands a card, add --device /dev/kfd --device /dev/dri to this run.)
+# --plan-only runs the same declare phase and exits before a byte is written.
+echo "rad-convert: $SRC (staged) -> $OUT"
+mkdir -p "$OUT_DIR"
+CONVERT_ARGS=(/models/stage-qwen4exp-fp8 -o /models/rad/qwen4exp-fp8-native.rad
+              --home /opt/radiance/share/radiance)
+[ "$plan_only" -eq 1 ] && CONVERT_ARGS+=(--plan-only -v)
+# THE ONE RECIPE RULE (see header): the PLE table arrives from staging as a
+# bf16 merge of the checkpoint's own fp8 codes decoded with the checkpoint's
+# own scale; rtn re-encodes it at that same scale — the codes come back
+# exactly and the plugin's one-element scale plane is produced.
+CONVERT_ARGS+=(--quant "blk.*.ple_ngram.weight=rtn:codes=fp8_e4m3,block=*x*,scale=bf16,rule=fixed,scale_value=$PLE_SCALE")
+[ "$force" -eq 1 ] && [ -f "$OUT" ] && {
+    TRASH="$HOME/.AGENTTRASH/$(date +%Y%m%d)-frank-primary-rad"
+    mkdir -p "$TRASH"
+    mv "$OUT" "$TRASH/qwen4exp-fp8-native.rad.$(date +%H%M%S)"
+    echo "previous container moved to $TRASH"
+}
+podman run --rm \
+    --entrypoint rad-convert \
+    -v /home/dank/models:/models \
+    "$IMAGE" "${CONVERT_ARGS[@]}"
+
+if [ "$plan_only" -eq 1 ]; then
+    echo "plan-only run complete (no container written)"
+    exit 0
+fi
+
+# ---- record -----------------------------------------------------------------
+echo "rad-info summary:"
+podman run --rm --entrypoint rad-info -v /home/dank/models:/models:ro \
+    "$IMAGE" "/models/rad/qwen4exp-fp8-native.rad" | head -20
+MIB=$(( $(stat -c %s "$OUT") / 1048576 ))
+echo "container: $OUT (${MIB} MiB; env floor $FRANK_PRIMARY_RAD_MIN_MIB MiB)"
+if [ "$MIB" -lt "$FRANK_PRIMARY_RAD_MIN_MIB" ]; then
+    echo "ERROR: container below the env floor — the convert was truncated" >&2
+    exit 1
+fi
+echo "build complete"

scripts/calib-push.py

@@ -0,0 +1,110 @@
+#!/usr/bin/env python3
+"""calib-push.py — drive a calibration corpus through a radiance scratch
+server's prefill so its tap writes layer Hessians (docs/MOE-W4.md).
+
+Runs ON DANKSERVER next to the scratch server (the one autoresearch- /
+serve-style scratch container booted with RADIANCE_CALIB_DIR set).
+Reads the corpus JSONL produced by make-calib-corpus.py
+({"prompt": "..."} a line), POSTs each prompt to /v1/completions with
+max_tokens=1 (prefill-only cost, one token out), at N-way concurrency
+matching the scratch server's --max-num-seqs budget.
+
+Corpus rounds repeat until the engine prints the tap's completion line
+("calibration wrote layer Hessians") in `podman logs <container>` — that
+is the ground truth that RADIANCE_CALIB_TOKENS was reached. After that
+line, further prompts are pure waste, so the driver exits.
+
+Mechanics notes:
+- requests carry no sampling params that matter (max_tokens=1, greedy).
+- a per-request timeout of 30 min: at ~1k tok/s prefill a 12k-token
+  prompt is a few seconds; the timeout only catches a wedged engine.
+- failures print and continue: the tap pools over tokens, a few lost
+  prompts do not bias the Hessians (the corpus is drawn round-robin over
+  source classes, so loss is class-neutral as long as failures are rare;
+  a systematic 4xx is loud in the log and must be fixed, not ignored).
+
+Usage: calib-push.py [--port 7770] [--parallel 64] [--container arbench]
+       [--corpus /home/dank/corpora/out/calib.jsonl] [--max-rounds 4]
+"""
+import argparse
+import concurrent.futures
+import json
+import subprocess
+import sys
+import time
+import urllib.request
+
+DONE_MARK = "calibration wrote"
+
+
+def tap_done(container):
+    """True if the engine logged the tap's completion line."""
+    try:
+        out = subprocess.run(
+            ["podman", "logs", "--tail", "50", container],
+            capture_output=True, text=True, timeout=30,
+        )
+    except Exception:
+        return False
+    return DONE_MARK in (out.stdout + out.stderr)
+
+
+def push_one(url, prompt):
+    body = json.dumps({
+        "model": "frank-archon",
+        "prompt": prompt,
+        "max_tokens": 1,
+        "temperature": 0,
+        "stream": False,
+    }).encode()
+    req = urllib.request.Request(
+        url, data=body, headers={"Content-Type": "application/json"})
+    try:
+        with urllib.request.urlopen(req, timeout=1800) as r:
+            r.read()
+        return True
+    except Exception as e:
+        print(f"FAIL {str(e)[:120]}", flush=True)
+        return False
+
+
+def main():
+    ap = argparse.ArgumentParser()
+    ap.add_argument("--port", type=int, default=7770)
+    ap.add_argument("--parallel", type=int, default=64)
+    ap.add_argument("--container", default="arbench")
+    ap.add_argument("--corpus", default="/home/dank/corpora/out/calib.jsonl")
+    ap.add_argument("--max-rounds", type=int, default=4)
+    a = ap.parse_args()
+    url = f"http://127.0.0.1:{a.port}/v1/completions"
+
+    prompts = []
+    with open(a.corpus) as f:
+        for line in f:
+            line = line.strip()
+            if line:
+                prompts.append(json.loads(line)["prompt"])
+    print(f"corpus: {len(prompts)} prompts", flush=True)
+
+    for rnd in range(1, a.max_rounds + 1):
+        if tap_done(a.container):
+            break
+        t0 = time.time()
+        ok = 0
+        with concurrent.futures.ThreadPoolExecutor(a.parallel) as ex:
+            for good in ex.map(lambda p: push_one(url, p), prompts):
+                ok += 1 if good else 0
+        dt = time.time() - t0
+        print(f"round {rnd}: {ok}/{len(prompts)} ok in {dt:.0f}s "
+              f"({sum(len(p) for p in prompts)/4/dt:.0f} chars/s est)",
+              flush=True)
+        if tap_done(a.container):
+            break
+    else:
+        print("max rounds exhausted, tap not done", flush=True)
+        sys.exit(1)
+    print("TAP_DONE: calibration wrote layer Hessians", flush=True)
+
+
+if __name__ == "__main__":
+    main()

scripts/convert-swift15-w4dn8.sh

@@ -0,0 +1,73 @@
+#!/usr/bin/env bash
+# convert-swift15-w4dn8.sh — RUNS ON DANKSERVER. The final recipe convert of
+# the Swift 1.5 checkpoint: gate_up experts through GPTQ against the 10M-token
+# calibration Grams, down projections block-fp8, protected experts bf16,
+# trunk int8, PLE fixed-scale fp8 (scripts/recipes/
+# qwen4exp-swift15-w4dn8-i8-hc8m.recipe).
+#
+# The convert engine is 1.0.13 — the SAME pin that will serve the result, so
+# the .rad crosses no version boundary. Only the Gram files cross versions
+# (1.2.0 wrote them; the RADGRAM2 magic, gram.r%d.%s.bin naming and the
+# calib_gram/calib_resolve/calib_stem symbols are byte-identical between the
+# two libquant.so). rad-convert runs on the host backend (no GPU visible in
+# the container) — production can stay up through this script.
+#
+# Prereqs (campaign order): calibration complete (calib-push printed TAP_DONE),
+# the protected-expert cast lines rewritten into the recipe by
+# derive-protected-experts.py, and the 335 GiB bf16 container deleted —
+# the peak-disk arithmetic needs that space (checkpoint 335.3 + output ~132
+# GiB on the volume; --reuse is NOT used, see the recipe header).
+set -euo pipefail
+
+PIN_1013=sha256:4c3f484564bb2872784e46b0a59cbd71becc9c1e746e301a3596ffae253c322e
+IMAGE="stilldeadcode/radiance@$PIN_1013"
+CKPT=/home/dank/models/ukisai/Swift1.5-Qwen3.8-Flash-Next
+RECIPE=/home/dank/serverAppConfigs-swift15/scripts/recipes/qwen4exp-swift15-w4dn8-i8-hc8m.recipe
+CALIB=/home/dank/corpora/calib
+OUT=/home/dank/models/rad/qwen4exp-swift15-w4dn8-i8-hc8m.rad
+LOG=/home/dank/logs/swift15_convert.log
+# gate_up ~37.6 + down ~18.8 + PLE 47.7 + trunk/embeddings/head ~10 GiB (all-w4 expert table;
+# the plugin's one-encoding law dropped the block-fp8 down tier -- see the recipe header).
+MIN_MIB=${FRANK_SWIFT15_RAD_MIN_MIB:-108000}
+
+# Gate 1: inputs. A missing Gram directory converts UNCALIBRATED with a
+# per-tensor stderr note, which is silently worse than failing here.
+[ -d "$CKPT" ] || { echo "no checkpoint at $CKPT" >&2; exit 1; }
+[ -f "$RECIPE" ] || { echo "no recipe at $RECIPE" >&2; exit 1; }
+GRAMS=$(find "$CALIB" -maxdepth 1 -name 'gram.r*.bin' 2>/dev/null | wc -l)
+[ "$GRAMS" -ge 48 ] || { echo "calib dir $CALIB holds $GRAMS gram files (<48)" >&2; exit 1; }
+if [ -e "$OUT" ]; then echo "$OUT already exists; move it to trash first" >&2; exit 1; fi
+if [ -e /home/dank/models/rad/qwen4exp-swift15-bf16.rad ]; then
+    echo "the bf16 container is still on disk; delete it first (peak disk)" >&2; exit 1
+fi
+
+# Gate 2: disk. Output ~132 GiB plus convert staging, 20 GiB of slack.
+FREE_MIB=$(df --output=avail -m /home/dank/models | tail -1 | tr -d ' ')
+[ "$FREE_MIB" -ge $((MIN_MIB + 20480)) ] || {
+    echo "only ${FREE_MIB} MiB free on /home/dank/models" >&2; exit 1; }
+
+# The recipe names $CALIB; rad-convert expands it from the environment
+# (MOE-W4.md:275). The recipe's cast section must name THIS checkpoint's
+# protected experts — derive-protected-experts.py rewrote it.
+echo "convert: $CKPT -> $OUT (gptq calib=$CALIB, $(date -Is))" | tee "$LOG"
+podman run --rm --name swift15-convert \
+    -v /home/dank/models:/models \
+    -v /home/dank/serverAppConfigs-swift15:/campaign:ro \
+    -v "$CALIB:/data/calib:ro" \
+    -e CALIB=/data/calib \
+    --entrypoint /opt/radiance/bin/rad-convert \
+    "$IMAGE" "/models/${CKPT#/home/dank/models/}" \
+    --recipe /campaign/"${RECIPE#/home/dank/serverAppConfigs-swift15/}" \
+    -o /models/rad/"$(basename "$OUT")" 2>&1 | tee -a "$LOG"
+rc=${PIPESTATUS[0]}
+[ "$rc" -eq 0 ] || { echo "rad-convert rc=$rc; log: $LOG" >&2; exit "$rc"; }
+
+# Gate 3: the artifact. rad-info's recipe echo is the receipt that GPTQ ran.
+SIZE_MIB=$(( $(stat -c %s "$OUT") / 1048576 ))
+[ "$SIZE_MIB" -ge "$MIN_MIB" ] || {
+    echo "$OUT is ${SIZE_MIB} MiB < floor ${MIN_MIB}" >&2; exit 1; }
+podman run --rm --entrypoint /opt/radiance/bin/rad-info \
+    -v /home/dank/models:/models:ro "$IMAGE" /models/rad/"$(basename "$OUT")" \
+    2>&1 | tee -a "$LOG" | grep -c 'gptq' >/dev/null \
+    || echo "WARNING: rad-info shows no gptq rule in the recorded recipe" >&2
+echo "CONVERT_DONE ${SIZE_MIB} MiB -> $OUT"

scripts/deep-probe.py

@@ -0,0 +1,55 @@
+#!/usr/bin/env python3
+"""deep-probe.py — one chat completion at ~43.6k prompt tokens, temp 0, 512
+generated, streamed; decode = completion_tokens / (total - ttft)."""
+import json, time, urllib.request
+
+TARGET_BYTES = int(43600 * 3.8)
+
+def build():
+    parts, n = [], 0
+    with open("/home/dank/corpora/out/calib.jsonl") as f:
+        for line in f:
+            p = json.loads(line)["prompt"]
+            parts.append(p)
+            n += len(p.encode())
+            if n >= TARGET_BYTES:
+                break
+    return "".join(parts)
+
+def main(salt):
+    body = json.dumps({
+        "model": "frank-archon",
+        "messages": [{"role": "user", "content": f"[probe {salt}] Continue and summarize this material in English, 400 words.\n\n" + build()}],
+        "temperature": 0.0, "max_tokens": 512,
+        "stream": True, "stream_options": {"include_usage": True},
+    }).encode()
+    req = urllib.request.Request("http://localhost:7770/v1/chat/completions",
+        data=body, headers={"Content-Type": "application/json"})
+    t0 = time.time(); ttft = None; ctok = 0; usage = {}
+    with urllib.request.urlopen(req, timeout=2400) as r:
+        for raw in r:
+            line = raw.decode().strip()
+            if not line.startswith("data:"):
+                continue
+            d = line[5:].strip()
+            if d == "[DONE]":
+                break
+            j = json.loads(d)
+            if j.get("usage"):
+                usage = j["usage"]
+            for ch in j.get("choices", []):
+                delta = (ch.get("delta") or {})
+                piece = delta.get("content") or delta.get("reasoning_content") or ""
+                if piece and ttft is None:
+                    ttft = time.time() - t0
+                ctok += 1
+    total = time.time() - t0
+    gen = usage.get("completion_tokens") or ctok
+    dec = gen / (total - (ttft or 0)) if total > (ttft or 0) else 0
+    print(json.dumps({"salt": salt, "prompt_tokens": usage.get("prompt_tokens"),
+                      "completion_tokens": gen, "ttft_s": round(ttft or 0, 2),
+                      "total_s": round(total, 1), "decode_tps": round(dec, 2)}))
+
+if __name__ == "__main__":
+    import sys
+    main(sys.argv[1] if len(sys.argv) > 1 else "x")

scripts/deploy-plan.py

@@ -62,7 +62,7 @@ SERVICE_ORDER = (
     "lidarr",
     "readarr",
     "frank-large",
-    "frank-small",
+    "frank-secondary",
     "frank-primary",
 )
 
@@ -94,7 +94,7 @@ DANKCLAW_SERVICES = {
     "caddy",
     "homepage",
     "frank-large",
-    "frank-small",
+    "frank-secondary",
 }
 
 DANKSERVER_SERVICES = {
@@ -400,12 +400,12 @@ def services_for_path(path: str, target: str) -> set[str]:
             # change to it redeploying alertmanager is the only path that
             # reaches it on the host.
             _add(services, "alertmanager")
-        if name in {"frank-small.env", "frank-small.service", "serve-frank-small.sh"}:
-            # llama.cpp frank-small (2026-09-16) keeps its unit, env and serve
-            # files here; each one selects frank-small only. The ik_llama.cpp
+        if name in {"frank-secondary.env", "frank-secondary.service", "serve-frank-secondary.sh"}:
+            # llama.cpp frank-secondary (2026-09-16) keeps its unit, env and serve
+            # files here; each one selects frank-secondary only. The ik_llama.cpp
             # trio that lived here for two days is in scripts/.AGENTTRASH/,
             # which is archive material and plans nothing.
-            _add(services, "frank-small")
+            _add(services, "frank-secondary")
         if name in {"frank-large.env", "frank-large.service", "serve-frank-large.sh"}:
             # CIRU v4.4.1 frank-large (2026-09-19) keeps its unit, env and
             # serve files here the same way: a root unit on a host-built

scripts/deploy-remote.sh

@@ -90,7 +90,7 @@ service_is_allowed() {
         dankclaw:authentik-postgres|dankclaw:authentik-server|\
         dankclaw:authentik-worker|dankclaw:authentik-outpost|dankclaw:authentik-outpost-ldap|dankclaw:anubis|\
         dankclaw:caddy|dankclaw:homepage|dankclaw:seerr|dankclaw:marreta|\
-        dankclaw:frank-large|dankclaw:frank-small|\
+        dankclaw:frank-large|dankclaw:frank-secondary|\
         dankserver:romm-mariadb|dankserver:romm-redis|dankserver:romm|\
         dankserver:copyparty|dankserver:jellyfin|dankserver:calibre-web|dankserver:calibre-server|dankserver:plex|dankserver:tautulli|\
         dankserver:prowlarr|dankserver:qbittorrent|dankserver:radarr|\
@@ -322,6 +322,12 @@ deploy_homepage() {
     if [[ -d shared/homepage/fonts ]]; then
         run_privileged cp -r shared/homepage/fonts /srv/homepage/
     fi
+    if [[ -d shared/homepage/img ]]; then
+        run_privileged cp -r shared/homepage/img /srv/homepage/
+    fi
+    if [[ -d shared/homepage/sfx ]]; then
+        run_privileged cp -r shared/homepage/sfx /srv/homepage/
+    fi
     local deployed_at
     deployed_at="$(git log -1 --format=%cI)"
     printf 'window.DEPLOYED_AT = "%s";\n' "$deployed_at" > /tmp/deployed.js
@@ -1086,33 +1092,37 @@ deploy_anubis() {
     wait_container_healthy anubis 60 || return 1
 }
 
-# frank-large is the halogen CONTAINER (ghcr.io/peonist-ai/halogen,
-# digest-pinned through scripts/frank-large.env) serving
-# peonist-ai/halogen-qwen3.8-27b — qwen3.8-27b-p1w4d-d2.hgn, 33.4 GiB,
-# DFlash2 spec decode, 1 slot x 200700, text-only — on the Strix Halo
-# iGPU. It stays a ROOT unit because the serve script drives `podman run`
-# directly (the frank-primary pattern): relink_unit points
-# /etc/systemd/system/frank-large.service at the repo file, boot
+# frank-large is the halogen-flash-server CONTAINER (0.16.2, digest-pinned
+# through scripts/frank-large.env) serving the Quat3rnion ABLITERATED
+# Swift1.5 finetune of peonist's Qwen3.8-Flash-Next v2 checkpoint —
+# qwen38-flash-next-v2-swift15-abliterated.hgn (66,687,678,464 B, ~62 GiB
+# resident) + the REQUIRED n-gram lookup table + the vision tower —
+# MTP spec decode (HALOGEN_SPEC_ADAPT=0), 2 slots x 200192, image input —
+# on the Strix Halo iGPU. It stays a ROOT unit because the serve script
+# drives `podman run` directly (the frank-primary pattern): relink_unit
+# points /etc/systemd/system/frank-large.service at the repo file, boot
 # persistence is that unit's [Install], and LimitMEMLOCK=infinity is
-# fleet parity. 2026-10-05 cutover from the ROCmFPX llama.cpp shape —
-# rollback = revert the swap commit's trio; that payload, its draft head
-# and the fork build all stay on disk
-# (~/models/kingjones-Qwen3.8-27B-Uncensored-ROCmFP4-STRIX-MTP,
-# ~/GitPackages/kingjones-ROCmFPX).
+# fleet parity. 2026-10-06 cutover from the halogen 27B-dense shape —
+# rollback = revert the swap commit's trio; the 27B payload stays on
+# disk (~/models/halogen-qwen3.8-27b).
 #
 # The gates mirror serve-frank-large.sh's predicates EXACTLY (the
 # 2026-10-01 crash-loop lesson): image digest in dank's podman storage,
-# the .hgn checkpoint at the exact byte count, the tokenizer directory
-# holding every named file as a regular (not symlinked) file — a green
-# deploy against a missing artifact would
-# leave a restart-looping unit and a dark endpoint. The port gate is the
-# same as every cutover since 2026-09-16: nothing starts until 7770 is
-# released.
-LLM_LARGE_IMAGE="${LLM_LARGE_IMAGE:-ghcr.io/peonist-ai/halogen@sha256:dc0a39a0016d6cfc58a197978febaafdf8d28403f724ded6111d98b5fb7ac0ea}"
-LLM_LARGE_CKPT="${LLM_LARGE_CKPT:-/home/dank/models/halogen-qwen3.8-27b/qwen3.8-27b-p1w4d-d2.hgn}"
-LLM_LARGE_CKPT_BYTES="${LLM_LARGE_CKPT_BYTES:-35865565184}"
-LLM_LARGE_TOK_DIR="${LLM_LARGE_TOK_DIR:-/home/dank/models/halogen-qwen3.8-27b/tokenizer}"
-LLM_LARGE_TOK_FILES="${LLM_LARGE_TOK_FILES:-tokenizer.json chat_template.jinja tokenizer_config.json merges.txt vocab.json}"
+# each .hgn payload (checkpoint, n-gram table, vision tower) at the exact
+# byte count, the tokenizer directory holding every named file as a
+# regular (not symlinked) file — a green deploy against a missing
+# artifact would leave a restart-looping unit and a dark endpoint. The
+# port gate is the same as every cutover since 2026-09-16: nothing
+# starts until 7770 is released.
+LLM_LARGE_IMAGE="${LLM_LARGE_IMAGE:-ghcr.io/peonist-ai/halogen-flash-server@sha256:0c61bf84ac22308a53f5d1ca6b86806702d7039e5ebc51cae4c66621b92fe04a}"
+LLM_LARGE_CKPT="${LLM_LARGE_CKPT:-/home/dank/models/halogen-flashnext-v2/qwen38-flash-next-v2-swift15-abliterated.hgn}"
+LLM_LARGE_CKPT_BYTES="${LLM_LARGE_CKPT_BYTES:-66687678464}"
+LLM_LARGE_NGRAM="${LLM_LARGE_NGRAM:-/home/dank/models/halogen-flashnext-v2/qwen38-flash-next-ngram.hgn}"
+LLM_LARGE_NGRAM_BYTES="${LLM_LARGE_NGRAM_BYTES:-51200246144}"
+LLM_LARGE_VISION="${LLM_LARGE_VISION:-/home/dank/models/halogen-flashnext-v2/qwen38-flash-next-vision.hgn}"
+LLM_LARGE_VISION_BYTES="${LLM_LARGE_VISION_BYTES:-897916416}"
+LLM_LARGE_TOK_DIR="${LLM_LARGE_TOK_DIR:-/home/dank/models/halogen-flashnext-v2/tokenizer}"
+LLM_LARGE_TOK_FILES="${LLM_LARGE_TOK_FILES:-tokenizer.json chat_template.jinja tokenizer_config.json merges.txt vocab.json generation_config.json}"
 
 retire_halogen_large_quadlet() {
     local trash="$HOME/.AGENTTRASH/20260919-orca-cutover/quadlet"
@@ -1156,23 +1166,37 @@ deploy_llm_large() {
     fi
     if [[ ! -f "$LLM_LARGE_CKPT" ]]; then
         echo "ERROR: halogen checkpoint absent at $LLM_LARGE_CKPT" >&2
-        echo "frank-large: fetch it: bash scripts/download-frank-large-halogen.sh" >&2
+        echo "frank-large: fetch it: bash scripts/download-frank-large-flashnext.sh" >&2
         return 1
     fi
     if [[ "$(stat -c %s "$LLM_LARGE_CKPT")" != "$LLM_LARGE_CKPT_BYTES" ]]; then
         echo "ERROR: halogen checkpoint is not $LLM_LARGE_CKPT_BYTES bytes (partial download?)" >&2
-        echo "frank-large: re-check: bash scripts/download-frank-large-halogen.sh --verify-only" >&2
+        echo "frank-large: re-check: bash scripts/download-frank-large-flashnext.sh --verify-only" >&2
         return 1
     fi
+    for gate in "$LLM_LARGE_NGRAM:$LLM_LARGE_NGRAM_BYTES:n-gram table" \
+                "$LLM_LARGE_VISION:$LLM_LARGE_VISION_BYTES:vision tower"; do
+        IFS=: read -r gpath gbytes gname <<<"$gate"
+        if [[ ! -f "$gpath" ]]; then
+            echo "ERROR: halogen $gname absent at $gpath" >&2
+            echo "frank-large: fetch it: bash scripts/download-frank-large-flashnext.sh" >&2
+            return 1
+        fi
+        if [[ "$(stat -c %s "$gpath")" != "$gbytes" ]]; then
+            echo "ERROR: halogen $gname is not $gbytes bytes (partial download?)" >&2
+            echo "frank-large: re-check: bash scripts/download-frank-large-flashnext.sh --verify-only" >&2
+            return 1
+        fi
+    done
     if [[ -L "$LLM_LARGE_TOK_DIR" || ! -d "$LLM_LARGE_TOK_DIR" ]]; then
         echo "ERROR: tokenizer dir missing or a symlink at $LLM_LARGE_TOK_DIR" >&2
-        echo "frank-large: fetch it: bash scripts/download-frank-large-halogen.sh" >&2
+        echo "frank-large: fetch it: bash scripts/download-frank-large-flashnext.sh" >&2
         return 1
     fi
     for tf in $LLM_LARGE_TOK_FILES; do
         if [[ ! -f "$LLM_LARGE_TOK_DIR/$tf" || -L "$LLM_LARGE_TOK_DIR/$tf" ]]; then
             echo "ERROR: tokenizer file missing or a symlink: $LLM_LARGE_TOK_DIR/$tf" >&2
-            echo "frank-large: fetch it: bash scripts/download-frank-large-halogen.sh" >&2
+            echo "frank-large: fetch it: bash scripts/download-frank-large-flashnext.sh" >&2
             return 1
         fi
     done
@@ -1184,7 +1208,7 @@ deploy_llm_large() {
     # `podman run --name frank-large-halogen` / `podman wait` but NOT in
     # this deployer's own command line (`bash deploy-remote.sh
     # frank-large`) — so it cannot shoot itself in the foot (the fleet
-    # pkill law), and it can never match frank-small's llama-server.
+    # pkill law), and it can never match frank-secondary's llama-server.
     run_privileged systemctl stop frank-large.service 2>/dev/null || true
     pkill -f 'frank-large-halogen' 2>/dev/null || true
     podman rm -f frank-large frank-large-halogen >/dev/null 2>&1 || true
@@ -1198,119 +1222,116 @@ deploy_llm_large() {
     wait_for_health frank-large http://127.0.0.1:7770/health 1200
 }
 
-# deploy_llm_small — frank-small is llama.cpp (master build, CUDA sm_86) on
+# deploy_llm_secondary — frank-secondary is llama.cpp (master build, CUDA sm_86) on
 # the RTX 3090 as a ROOT unit: the runtime is a native binary tree under
 # ~/GitPackages/llama-master/build-cuda, so there is no container image and
 # no generated unit to avoid. relink_unit points
-# /etc/systemd/system/frank-small.service at the repo file and boot
+# /etc/systemd/system/frank-secondary.service at the repo file and boot
 # persistence is that unit's [Install] section.
 #
 # It is a SEPARATE endpoint on port 7771; frank-large keeps 7770. The 3090 is
 # not shared: frank-large's engine runs on the gfx1151 iGPU.
-LLM_SMALL_MODEL_FILE="${LLM_SMALL_MODEL_FILE:-/home/dank/models/Ornith-Q8/Ornith-1.5-9B-Q8_0.gguf}"
-LLM_SMALL_RUNTIME="${LLM_SMALL_RUNTIME:-/home/dank/GitPackages/llama-master/build-cuda/bin/llama-server}"
-# The projector IS the vision capability here: the Ornith GGUF ships no
-# `v.*` tensors, so a deploy that starts without this file silently ships a
-# text-only endpoint. Checked unconditionally, like frank-large's.
-LLM_SMALL_MMPROJ="${LLM_SMALL_MMPROJ:-/home/dank/models/Ornith-Q8/mmproj-Ornith-1.5-9B-BF16.gguf}"
+LLM_SECONDARY_MODEL_FILE="${LLM_SECONDARY_MODEL_FILE:-/home/dank/models/Qwen35-9B-HauhauCS-Q6K/Qwen3.5-9B-Uncensored-HauhauCS-Aggressive-Q6_K.gguf}"
+LLM_SECONDARY_RUNTIME="${LLM_SECONDARY_RUNTIME:-/home/dank/GitPackages/llama-master/build-cuda/bin/llama-server}"
+# The projector IS the vision capability here: the model is natively
+# multimodal and the Q6_K file carries no vision tensors — the publisher ships
+# its own BF16 projector in the same repo, so a deploy that starts without
+# this file silently ships a text-only endpoint. Checked unconditionally, like
+# frank-large's.
+LLM_SECONDARY_MMPROJ="${LLM_SECONDARY_MMPROJ:-/home/dank/models/Qwen35-9B-HauhauCS-Q6K/mmproj-Qwen3.5-9B-Uncensored-HauhauCS-Aggressive-BF16.gguf}"
 
-deploy_llm_small() {
+deploy_llm_secondary() {
     # Refuse rather than restart-loop: a unit whose runtime or payload is
     # missing never becomes healthy, and a green deploy would leave the
     # endpoint dark with nothing in the log to say why.
-    if [[ ! -x "$LLM_SMALL_RUNTIME" ]]; then
-        echo "ERROR: llama.cpp runtime absent at $LLM_SMALL_RUNTIME" >&2
-        echo "frank-small: build it (cmake -B build-cuda -DGGML_CUDA=ON" >&2
-        echo "frank-small: -DCMAKE_CUDA_ARCHITECTURES=86 -DCMAKE_CUDA_HOST_COMPILER=/usr/bin/gcc-13" >&2
-        echo "frank-small: && cmake --build build-cuda --target llama-server) first." >&2
+    if [[ ! -x "$LLM_SECONDARY_RUNTIME" ]]; then
+        echo "ERROR: llama.cpp runtime absent at $LLM_SECONDARY_RUNTIME" >&2
+        echo "frank-secondary: build it (cmake -B build-cuda -DGGML_CUDA=ON" >&2
+        echo "frank-secondary: -DCMAKE_CUDA_ARCHITECTURES=86 -DCMAKE_CUDA_HOST_COMPILER=/usr/bin/gcc-13" >&2
+        echo "frank-secondary: && cmake --build build-cuda --target llama-server) first." >&2
         return 1
     fi
-    for f in "$LLM_SMALL_MODEL_FILE" "$LLM_SMALL_MMPROJ"; do
+    for f in "$LLM_SECONDARY_MODEL_FILE" "$LLM_SECONDARY_MMPROJ"; do
         if [[ ! -f "$f" ]]; then
-            echo "ERROR: frank-small payload file absent at $f" >&2
-            echo "frank-small: run scripts/download-frank-small.sh first;" >&2
-            echo "frank-small: refusing to start a unit that cannot load." >&2
+            echo "ERROR: frank-secondary payload file absent at $f" >&2
+            echo "frank-secondary: run scripts/download-frank-secondary.sh first;" >&2
+            echo "frank-secondary: refusing to start a unit that cannot load." >&2
             return 1
         fi
     done
-    relink_unit frank-small
+    relink_unit frank-secondary
     run_privileged systemctl daemon-reload
-    run_privileged systemctl enable frank-small.service
-    run_privileged systemctl restart frank-small.service
-    wait_for_health frank-small http://127.0.0.1:7771/health 900 || return 1
+    run_privileged systemctl enable frank-secondary.service
+    run_privileged systemctl restart frank-secondary.service
+    wait_for_health frank-secondary http://127.0.0.1:7771/health 900 || return 1
     # False-green hardening (frank-large's lesson): the port answering
     # proves nothing about WHO answers — the unit must be active with a
     # live MainPID, or a stray SGLang holdover is masquerading as deployed.
     local main_pid
-    main_pid="$(run_privileged systemctl show -p MainPID --value frank-small.service)"
+    main_pid="$(run_privileged systemctl show -p MainPID --value frank-secondary.service)"
     if [[ -z "$main_pid" || "$main_pid" == "0" ]]; then
-        echo "FAIL: :7771 answered but frank-small.service has no MainPID — a stray process holds the port" >&2
+        echo "FAIL: :7771 answered but frank-secondary.service has no MainPID — a stray process holds the port" >&2
         return 1
     fi
-    echo "frank-small: health answered by unit MainPID $main_pid"
+    echo "frank-secondary: health answered by unit MainPID $main_pid"
 }
 
-# deploy_llm_primary — frank-primary is the tcclaviger RDNA4 vLLM fork
-# (digest-pinned image, gfx1201) serving the orcarouter
-# Qwen3.8-Flash-Next-Uncensored-FP8 payload (131 safetensors shards) with
-# MTP4 spec decode and q8/q8 KV on the two R9700s as a ROOT unit whose serve
-# script drives `podman run` directly (the image entrypoint execs
-# `vllm serve`). 2026-09-29 cutover from the R9V radiance fork, whose
-# launcher/shim/setup-state machinery is retired (state dirs stay on disk as
-# rollback; pre-cutover tag frank-primary-r9v-q8-20260929). Same port and
-# alias as the endpoints it replaced; the harness models.yml id is unchanged.
+# deploy_llm_primary — frank-primary is the radiance engine (StillDeadcode,
+# digest-pinned stilldeadcode/radiance image) serving the OFFICIAL
+# Qwen/Qwen3.8-Flash-Next-FP8 at native block-FP8 precision from the
+# rad-convert container qwen4exp-fp8-native.rad, on BOTH R9700s (TP=2) as a
+# ROOT unit whose serve script drives `podman run` directly (the engine is
+# the image entrypoint; the image lives in dank's rootless podman storage).
+# 2026-10-05 cutover from FreeToken (pre-cutover tag
+# frank-primary-freetoken-pr587-20261005); before that, the tcclaviger vLLM
+# fork (tag frank-primary-vllm-shapeG-20261003). Same port and alias; the
+# harness models.yml id is unchanged.
 #
-# Gates: the pinned runtime image must be loadable from dank's rootless
-# storage by its OCI manifest digest (the env file's FRANK_PRIMARY_IMAGE),
-# the payload dir must carry config.json and the expected shard count
-# (FRANK_PRIMARY_MODEL_SHARDS), and the unit env must be complete. Per-file
-# sha256 verification was the fetch script's job at download time, never
+# Gates: the image must exist under its digest ref in dank's rootless
+# storage (FRANK_PRIMARY_IMAGE), the .rad container file must exist at the
+# env's size floor (FRANK_PRIMARY_MODEL_FILE >= FRANK_PRIMARY_RAD_MIN_MIB —
+# a truncated convert is a guaranteed long-timeout boot), and the unit env
+# must be complete. Per-file sha256 verification is the fetch script's job
+# at download time and rad-convert/rad-info's job at build time, never
 # re-hashed here.
-LLM_PRIMARY_CONTAINER=frank-primary-qwen
-LLM_PRIMARY_LEGACY_CONTAINERS="frank-primary-r9v qwen-fp8 glm-probe"
+LLM_PRIMARY_CONTAINER=frank-primary
+LLM_PRIMARY_LEGACY_CONTAINERS="frank-primary-qwen frank-primary-r9v qwen-fp8 glm-probe"
 
 deploy_llm_primary() {
     # Refuse rather than restart-loop: a unit whose image or payload is
     # missing never becomes healthy, and a green deploy would leave the
     # endpoint dark with nothing in the log to say why.
     local env_file="/home/dank/serverAppConfigs/scripts/frank-primary.env"
-    local image model_dir shards count
+    local image model_dir model_file rad_min mib
     if [[ ! -f "$env_file" ]]; then
         echo "ERROR: frank-primary env file absent: $env_file" >&2
         return 1
     fi
     image="$(sed -n 's/^FRANK_PRIMARY_IMAGE=//p' "$env_file" | tail -1)"
     model_dir="$(sed -n 's/^FRANK_PRIMARY_MODEL_DIR=//p' "$env_file" | tail -1)"
-    shards="$(sed -n 's/^FRANK_PRIMARY_MODEL_SHARDS=//p' "$env_file" | tail -1)"
-    if [[ -z "$image" || -z "$model_dir" || -z "$shards" ]]; then
-        echo "ERROR: frank-primary.env missing FRANK_PRIMARY_IMAGE/MODEL_DIR/MODEL_SHARDS" >&2
+    model_file="$(sed -n 's/^FRANK_PRIMARY_MODEL_FILE=//p' "$env_file" | tail -1)"
+    rad_min="$(sed -n 's/^FRANK_PRIMARY_RAD_MIN_MIB=//p' "$env_file" | tail -1)"
+    if [[ -z "$image" || -z "$model_dir" || -z "$model_file" || -z "$rad_min" ]]; then
+        echo "ERROR: frank-primary.env missing FRANK_PRIMARY_IMAGE/MODEL_DIR/MODEL_FILE/RAD_MIN_MIB" >&2
         return 1
     fi
-    # Capture podman's output BEFORE grepping: under set -o pipefail, the
-    # classic `podman images | grep -q` races (grep exits on match, podman
-    # dies SIGPIPE 141 writing the rest, pipefail calls it no-match — the
-    # 2026-09-30 deploy refusal).
-    local digests
-    digests="$(podman images --format '{{.RepoDigests}}')"
-    if ! grep -qF "${image#*@}" <<<"$digests"; then
-        echo "ERROR: pinned runtime image absent from rootless storage: $image" >&2
-        echo "frank-primary: fetch it (podman pull $image)" >&2
+    # The image gate is the digest ref itself: a name@sha256 ref exists in
+    # storage only if THAT manifest was pulled. One command — no
+    # pipefail/grep race (the 2026-09-30 lesson).
+    if ! podman image exists "$image"; then
+        echo "ERROR: runtime image absent from storage: $image" >&2
+        echo "frank-primary: podman pull docker://stilldeadcode/radiance:latest" >&2
         return 1
     fi
-    if [[ ! -f "$model_dir/config.json" ]]; then
-        echo "ERROR: payload config missing: $model_dir/config.json" >&2
-        echo "frank-primary: re-run scripts/download-frank-primary.sh" >&2
+    if [[ ! -f "$model_dir/$model_file" ]]; then
+        echo "ERROR: .rad container missing: $model_dir/$model_file" >&2
+        echo "frank-primary: run scripts/build-frank-primary-rad.sh" >&2
         return 1
     fi
-    if [[ ! -f "$model_dir/model.safetensors.index.json" ]]; then
-        echo "ERROR: weight-map index missing: $model_dir/model.safetensors.index.json" >&2
-        echo "frank-primary: re-run scripts/download-frank-primary.sh" >&2
-        return 1
-    fi
-    count="$(find "$model_dir" -maxdepth 1 -name 'model-*-of-*.safetensors' 2>/dev/null | wc -l)"
-    if [[ "$count" != "$shards" ]]; then
-        echo "ERROR: payload shard count wrong: $model_dir (found $count, want $shards)" >&2
-        echo "frank-primary: re-run scripts/download-frank-primary.sh" >&2
+    mib=$(( $(stat -c %s "$model_dir/$model_file") / 1048576 ))
+    if [[ "$mib" -lt "$rad_min" ]]; then
+        echo "ERROR: .rad container too small: $model_dir/$model_file is ${mib} MiB, want >= $rad_min" >&2
+        echo "frank-primary: rebuild it (scripts/build-frank-primary-rad.sh --force)" >&2
         return 1
     fi
 
@@ -1335,9 +1356,26 @@ deploy_llm_primary() {
     run_privileged systemctl daemon-reload
     run_privileged systemctl enable frank-primary.service
     run_privileged systemctl restart frank-primary.service
-    # Cold start: FP8 page-in plus torch compile/graph capture (~15 min cold,
-    # measured 2026-09-29). Parity with every predecessor on this port.
-    wait_for_health frank-primary http://127.0.0.1:7770/health 1800 || return 1
+    # Cold start: .rad load + PLE pin + expert host-pool upload (the
+    # publisher's compose healthcheck allows a 15-minute start_period).
+    # Parity with every predecessor on this port. The HTTP listener binds
+    # AFTER the full load (main.cpp: engine.start() then engine.serve()),
+    # so during load the port REFUSES connections — a bare curl is honest
+    # here, unlike FreeToken's 200-while-loading body. (The MainPID check
+    # below still guards the stray-port case.)
+    local i ok=0
+    for ((i=1; i<=900; i++)); do
+        if curl -fsS http://127.0.0.1:7770/health 2>/dev/null >/dev/null; then
+            echo "frank-primary engine healthy after approximately $((i * 2))s"
+            ok=1
+            break
+        fi
+        sleep 2
+    done
+    if [[ "$ok" != 1 ]]; then
+        echo "FAIL: frank-primary /health never answered 200 in 1800s" >&2
+        return 1
+    fi
     # False-green hardening (the 2026-09-26 lesson): a bare curl against the
     # port cannot tell WHO answers — a stray probe holding :7770 satisfies
     # it while the unit is stopped. The deploy is only green if the unit
@@ -1536,7 +1574,7 @@ deploy_service() {
         seerr) deploy_seerr ;;
         marreta) deploy_marreta ;;
         frank-large) deploy_llm_large ;;
-        frank-small) deploy_llm_small ;;
+        frank-secondary) deploy_llm_secondary ;;
         frank-primary) deploy_llm_primary ;;
         hermes|hermes-dashboard|hermes-webui) deploy_hermes ;;
         romm|romm-mariadb|romm-redis|mariadb|redis) deploy_romm_stack "$service" ;;

scripts/deploy.sh

@@ -141,7 +141,7 @@ valid_service() {
         dankclaw:alertmanager|\
         dankclaw:node-exporter|dankclaw:authentik-postgres|dankclaw:authentik-server|\
         dankclaw:authentik-worker|dankclaw:authentik-outpost|dankclaw:authentik-outpost-ldap|dankclaw:anubis|dankclaw:seerr|\
-        dankclaw:frank-large|dankclaw:frank-small|\
+        dankclaw:frank-large|dankclaw:frank-secondary|\
         dankserver:romm-mariadb|dankserver:romm-redis|dankserver:romm|\
         dankserver:copyparty|dankserver:jellyfin|dankserver:calibre-web|dankserver:calibre-server|dankserver:plex|dankserver:tautulli|\
         dankserver:qbittorrent|dankserver:radarr|dankserver:sonarr|\

scripts/derive-protected-experts.py

@@ -0,0 +1,117 @@
+#!/usr/bin/env python3
+"""derive-protected-experts.py — rewrite the recipe's protected-expert section
+from a calibration recording. RUNS ON DANKSERVER (venv: numpy).
+
+The law (docs/MOE-W4.md "Protected experts", restated in the recipe):
+an expert whose share of its layer's down-input energy is at least 3% AND
+at least three times its share of the routing is kept bf16; the count must
+be EVEN on each rank's half of the layer (the four-bit GEMM takes the rest
+as two equal tables by parity) — an odd half gains its next most energetic
+non-qualifying expert.
+
+Data: the calibration dir's per-expert down Grams,
+gram.r<rank>.blk.L.ffn_down_exps.<e>.bin — 32-byte RADGRAM2 header
+(magic, k, rank, rows, domain) then k*k f32, named with the checkpoint's
+GLOBAL expert numbering (arch/common/rad_block_moe_fp8.h: ge =
+expert_base + e). rows (header) is the expert's routing count; the trace
+of the Gram is its down-input energy. Rank halves: 0-255, 256-511
+(contiguous expert_base split at TP=2).
+
+Output: the cast-rule lines (recipe format) on stdout, a report on stderr.
+
+Usage: derive-protected-experts.py <calib dir> [--min-share 0.03]
+       [--min-route-mult 3.0]
+"""
+import argparse
+import glob
+import os
+import re
+import struct
+import sys
+
+import numpy as np
+
+NAME_RE = re.compile(r"^gram\.r(\d)\.blk\.(\d+)\.ffn_down_exps\.(\d+)\.bin$")
+
+
+def read_gram(path):
+    """(k, rank, rows, trace) from a RADGRAM2 file (full matrix on disk)."""
+    with open(path, "rb") as f:
+        hdr = f.read(32)
+        magic, k, rank, rows, domain = struct.unpack("<8sIIQI", hdr[:28])
+        if magic != b"RADGRAM2":
+            raise SystemExit(f"not a RADGRAM2 file: {path}")
+        n = k * k
+        a = np.frombuffer(f.read(n * 4), dtype=np.float32)
+        if a.size != n:
+            raise SystemExit(f"short gram file: {path}")
+        tr = float(a.reshape(k, k).diagonal().sum())
+    return k, rank, int(rows), tr
+
+
+def main():
+    ap = argparse.ArgumentParser()
+    ap.add_argument("calib_dir")
+    ap.add_argument("--min-share", type=float, default=0.03)
+    ap.add_argument("--min-route-mult", type=float, default=3.0)
+    a = ap.parse_args()
+
+    # layer -> global expert -> (rank, rows, energy)
+    data = {}
+    for path in sorted(glob.glob(os.path.join(
+            a.calib_dir, "gram.r*.ffn_down_exps.*.bin"))):
+        m = NAME_RE.match(os.path.basename(path))
+        if not m:
+            continue
+        rank, layer, expert = int(m.group(1)), int(m.group(2)), int(m.group(3))
+        k, _, rows, tr = read_gram(path)
+        data.setdefault(layer, {})[expert] = (rank, rows, tr)
+
+    if not data:
+        raise SystemExit("no per-expert down grams found")
+
+    lines, report = [], []
+    total = 0
+    for layer in sorted(data):
+        experts = data[layer]
+        tot_e = sum(v[2] for v in experts.values())
+        tot_r = sum(v[1] for v in experts.values())
+        if tot_e <= 0 or tot_r <= 0:
+            report.append(f"blk.{layer}: empty recording, skipped")
+            continue
+        qual, pool = [], []
+        for e, (rank, rows, tr) in experts.items():
+            es, rs = tr / tot_e, rows / max(tot_r, 1)
+            rec = (e, rank, es, rs)
+            if es >= a.min_share and es >= a.min_route_mult * rs:
+                qual.append(rec)
+            elif es > 0:
+                pool.append(rec)
+        chosen = {r[0]: (r[2], r[3]) for r in qual}
+        for half in (0, 1):  # rank halves by the contiguous expert_base split
+            n_half = sum(1 for e in chosen if e // 256 == half)
+            if n_half % 2:
+                pool.sort(key=lambda r: -r[2])
+                for cand in pool:
+                    if cand[1] == half and cand[0] not in chosen:
+                        chosen[cand[0]] = (cand[2], cand[3])
+                        report.append(
+                            f"blk.{layer}: half {half} odd -> added e{cand[0]} "
+                            f"(energy {cand[2]*100:.2f}%)")
+                        break
+        for e in sorted(chosen):
+            es, rs = chosen[e]
+            lines.append(f"blk.{layer}.ffn_*_exps.{e}.weight"
+                         f"         cast dtype=bf16")
+            report.append(f"blk.{layer} e{e:3d} (rank {e // 256}): "
+                          f"energy {es*100:5.2f}%, routing {rs*100:5.3f}%, "
+                          f"ratio {es/max(rs,1e-12):6.1f}")
+        total += len(chosen)
+
+    print("\n".join(lines))
+    print("\n".join(report), file=sys.stderr)
+    print(f"total protected: {total}", file=sys.stderr)
+
+
+if __name__ == "__main__":
+    main()

scripts/docker/frank-primary-freetoken.Dockerfile

@@ -1,62 +0,0 @@
-# frank-primary-freetoken.Dockerfile — FreeToken (PR #587 ROCm) serving image
-# for the frank-primary endpoint (dankserver :7770, 1x R9700 gfx1201).
-#
-# Built on dankserver with rootless podman (dank user), same storage the
-# service's `podman run` reads:
-#   podman build -t frank-primary-freetoken:pr587-77a7201 \
-#     -f scripts/docker/frank-primary-freetoken.Dockerfile scripts/docker/
-#
-# Base: the PR's own docs/install_amd.md names this exact image as the
-# known-good ROCm 7.14 stack for gfx1201 (torch 2.11.0+rocm7.14, triton-rocm
-# 3.7 — the original pin; FreeToken main+PR587 accepts torch>=2.11,<2.16 and
-# triton>=3.6,<3.9). There is no rocm7.14+torch2.14 image on Docker Hub (the
-# 2.14 images are rocm10.1.0 — an untested pair for this PR), so the PR doc's
-# verbatim recommendation is the deployed stack.
-#
-# FreeToken source: FlashML-org/FreeToken PR #587 at pinned head
-# 77a7201f98f00d93f618c6cf8b81ffe8360e0899 (branch rocm714_torch214, base
-# main 581fdca3). The PR is ROCm-gated end to end; the NVIDIA/CUDA path is
-# byte-identical to main.
-#
-# Build law from docs/install_amd.md:
-#   - `pip install --no-build-isolation --no-deps -e .` — the image's ROCm
-#     torch is preserved and used to compile the C++/HIP extensions.
-#   - NEVER a plain `pip install -e .` on ROCm: the dependency set is
-#     authored for NVIDIA (flashlib pulls numba + nvidia-cutlass-dsl; torch
-#     metadata pulls cuda-bindings) and the resolve can REPLACE the ROCm
-#     torch with a CUDA build.
-#   - Runtime deps installed explicitly, curated by the same doc. flashinfer
-#     / sgl_kernel / nvidia-cutlass-dsl have no ROCm builds and are
-#     intentionally absent — FreeToken gates them off and falls back to its
-#     Triton kernels on ROCm.
-#
-# Base is digest-pinned (2026-10-05): sha256:a223aee1… is the manifest this
-# build was qualified against; a re-pushed tag cannot change the stack.
-FROM rocm/pytorch@sha256:a223aee17aef5d21c3b9f63436dd19d27d1c665ec8b2f40011c9546cabae2a80
-
-# Pin the PR head, not the branch: a future push to rocm714_torch214 would
-# otherwise silently change the deployed engine.
-# The pip-ROCm torch in this image ships libhiprtc as the VERSIONED
-# libhiprtc.so.7 only; the PR's `_cpu_moe` extension links `-lhiprtc`, and
-# ld looks for the unversioned dev symlink in the ALREADY-LINKED
-# _rocm_sdk_core/lib dir. Create it there (first build failure 2026-10-05:
-# `ld: cannot find -lhiprtc`).
-RUN ln -s libhiprtc.so.7 /opt/venv/lib/python3.12/site-packages/_rocm_sdk_core/lib/libhiprtc.so
-RUN git clone https://github.com/FlashML-org/FreeToken.git /opt/FreeToken \
- && cd /opt/FreeToken \
- && git fetch origin pull/587/head \
- && git checkout 77a7201f98f00d93f618c6cf8b81ffe8360e0899
-
-WORKDIR /opt/FreeToken
-RUN python -m pip install --no-build-isolation --no-deps -e .
-
-# Curated runtime dependencies (docs/install_amd.md). transformers is pinned
-# to the pyproject window (>=5.16,<5.17); the rest are the doc's portable set.
-# torch / torchvision / triton-rocm come from the base image, 1:1 aligned.
-RUN python -m pip install \
-      "transformers>=5.16,<5.17" tokenizers safetensors huggingface_hub \
-      fastapi uvicorn pydantic openai httpx anyio \
-      einops gguf msgpack pyzmq typer rich apache-tvm-ffi \
-      partial-json-parser \
- && python -m pip install --no-deps flashlib==0.3.0 \
- && python -m pip install ninja

scripts/download-frank-large-flashnext.sh

@@ -0,0 +1,119 @@
+#!/usr/bin/env bash
+# download-frank-large-flashnext.sh — fetch frank-large's flash-server v2
+# payload: the Quat3rnion abliterated Swift1.5 checkpoint of peonist's v2
+# .hgn, the REQUIRED lookup table and the optional vision tower from
+# peonist-ai/halogen-qwen3.8-flash-next (both at the revisions the model
+# card pins), and the flat tokenizer the two repos share. All public, no
+# token — and per the fleet secret law, none would ever touch argv.
+#
+# The fetch law, same as every sibling script: curl -f (an access error
+# must not become a "successful" 153-byte artifact), exact byte count from
+# the Hub tree API, LFS sha256 verification, --verify-only to re-check an
+# existing payload without downloading.
+#
+# File manifest (verified against the Hub tree API on 2026-10-06):
+#   qwen38-flash-next-v2-swift15-abliterated.hgn  66,687,678,464 B  sha256
+#     9c880b8ba48deabfa06172eff1e7572e4fe465e09e97faec6e50c3f6fdcac037
+#     (Quat3rnion main; compacted r2 of the card's 89,775,868,032 B
+#     append-only original — byte-identical live tensors, dropped padding)
+#   qwen38-flash-next-ngram.hgn  51,200,246,144 B  sha256
+#     3450acada94e19aabad88bd49b45eb70304820949c2240fc178067b20ae5dffe
+#     (peonist @5cc17cea; REQUIRED — the engine reads it beside the
+#     checkpoint)
+#   qwen38-flash-next-vision.hgn  897,916,416 B  sha256
+#     d62e0ae553fe88afd3833733d4a4c669f34d20fd8dfce4b9610525bed2134b10
+#     (peonist @e053f488; optional, enables image input)
+#   tokenizer/* @e053f488 — byte-identical to the 27B-era tokenizer
+#     already on disk (tokenizer.json LFS
+#     0997f410c57a1f4e53b09e4be8f4a172d90edd9564368fb0847030937229b9f3);
+#     fetched fresh rather than symlinked, per the no-symlink law.
+set -euo pipefail
+
+DEST="${FRANK_LARGE_MODEL_DIR:-$HOME/models/halogen-flashnext-v2}"
+ABLIT="https://huggingface.co/Quat3rnion/halogen-swift1.5-qwen3.8-flash-next-v2-abliterated/resolve/main"
+PEONIST="https://huggingface.co/peonist-ai/halogen-qwen3.8-flash-next/resolve"
+NGRAM_REV="5cc17cea1a10b502b4a5db41d8d2b1a0f22dba26"
+TOK_REV="e053f488b120b99ed2525e6ac99f68c51b3b6179"
+
+FILES=(qwen38-flash-next-v2-swift15-abliterated.hgn
+       qwen38-flash-next-ngram.hgn
+       qwen38-flash-next-vision.hgn
+       tokenizer/tokenizer.json
+       tokenizer/chat_template.jinja
+       tokenizer/generation_config.json
+       tokenizer/merges.txt
+       tokenizer/vocab.json
+       tokenizer/tokenizer_config.json)
+SIZES=(66687678464
+       51200246144
+       897916416
+       12809320
+       8952
+       202
+       3353259
+       6722759
+       17928)
+SHAS=(9c880b8ba48deabfa06172eff1e7572e4fe465e09e97faec6e50c3f6fdcac037
+      3450acada94e19aabad88bd49b45eb70304820949c2240fc178067b20ae5dffe
+      d62e0ae553fe88afd3833733d4a4c669f34d20fd8dfce4b9610525bed2134b10
+      0997f410c57a1f4e53b09e4be8f4a172d90edd9564368fb0847030937229b9f3
+      - - - - - -)
+
+# Fetch source per file: repo base + revision (empty = ABLIT @main).
+SRCS=("$ABLIT"
+      "$PEONIST/$NGRAM_REV"
+      "$PEONIST/$TOK_REV"
+      "$PEONIST/$TOK_REV"
+      "$PEONIST/$TOK_REV"
+      "$PEONIST/$TOK_REV"
+      "$PEONIST/$TOK_REV"
+      "$PEONIST/$TOK_REV"
+      "$PEONIST/$TOK_REV")
+
+VERIFY_ONLY=0
+[ "${1:-}" = "--verify-only" ] && VERIFY_ONLY=1
+
+mkdir -p "$DEST"
+
+for i in "${!FILES[@]}"; do
+    f="${FILES[$i]}"; want="${SIZES[$i]}"; sha="${SHAS[$i]}"; base="${SRCS[$i]}"
+    path="$DEST/$f"
+    mkdir -p "$(dirname "$path")"
+    if [ "$VERIFY_ONLY" -eq 0 ]; then
+        # A file already at the published size skips the fetch entirely:
+        # resuming it would ask the CDN for a range past EOF (416 -> curl
+        # error under set -e), and re-pulling 62 GiB to prove a point is
+        # not what --verify-only is for. sha256 below still decides.
+        if [ -f "$path" ] && [ "$(stat -c %s "$path")" = "$want" ]; then
+            echo "HAVE  $f ($want bytes) — skipping fetch"
+        else
+            echo "FETCH $f -> $path"
+            # HF answers /resolve/ with a 302 to the Xet CDN and curl DROPS
+            # the Range header across redirects — a naive -C - then
+            # "resumes" by pulling the ~1 KB redirect body (measured
+            # 2026-10-05 on the 27B fetch: file frozen 14 MiB short, curl
+            # exit 0). Resolve the effective URL first, then resume
+            # against the CDN, which honors Range.
+            url=$(curl -sfIL -o /dev/null -w '%{url_effective}' "$base/$f") \
+                || { echo "cannot resolve $base/$f" >&2; exit 1; }
+            curl -fL --retry 5 -C - -o "$path" "$url"
+        fi
+    fi
+    [ -f "$path" ] || { echo "MISSING $path (run without --verify-only)" >&2; exit 1; }
+    got="$(stat -c %s "$path")"
+    [ "$got" = "$want" ] || { echo "SIZE $path: $got != $want" >&2; exit 1; }
+    echo "OK   $f ($got bytes)"
+done
+
+echo "byte counts match; verifying sha256 where the Hub publishes an LFS oid..."
+for i in "${!FILES[@]}"; do
+    f="${FILES[$i]}"; sha="${SHAS[$i]}"
+    if [ "$sha" = "-" ]; then
+        echo "OK   $f (plain git blob, size-checked)"
+        continue
+    fi
+    got="$(sha256sum "$DEST/$f" | cut -d' ' -f1)"
+    [ "$got" = "$sha" ] || { echo "SHA256 $f: $got != $sha" >&2; exit 1; }
+    echo "OK   $f sha256"
+done
+echo "DOWNLOAD_DONE $DEST"

scripts/download-frank-primary-swift15.sh

@@ -0,0 +1,113 @@
+#!/usr/bin/env bash
+# download-frank-primary-swift15.sh — payload fetch/verify for the planned
+# frank-primary swap to the Swift 1.5 finetune: ukisai/Swift1.5-Qwen3.8-Flash-Next
+# (BF16 source, 149 files, 360,037,355,038 bytes = 335.3 GiB: 131 safetensors
+# shards + weight-map index + tokenizer + vision preprocessing configs + chat
+# template + licenses). Key set verified identical to the official
+# Qwen/Qwen3.8-Flash-Next base (0 added / 0 missing), so the qwen4exp recipes
+# apply verbatim; the checkpoint ships BF16 with NO quantization_config.
+# Repo is UNGATED — no token needed (never put one on the command line).
+#
+#   bash scripts/download-frank-primary-swift15.sh --fetch         resume/complete
+#   bash scripts/download-frank-primary-swift15.sh                 verify only
+#
+# Fetch uses the vendored scripts/hfget.sh (StillDeadcode's Apache-2.0
+# listing/resume downloader, curl+jq only): per-file size checks against
+# the Hub listing, `-C -` resume, pickle-format duplicates skipped.
+# Verify (default) is the AGENTS.md law in three parts: every listed file
+# present with the listed size, the shard count matches the weight-map
+# index, and every shard's sha256 equals the Hub's LFS oid. Verify needs
+# only the public API.
+set -euo pipefail
+
+REPO="ukisai/Swift1.5-Qwen3.8-Flash-Next"
+DEST="/home/dank/models/ukisai/Swift1.5-Qwen3.8-Flash-Next"
+EXPECTED_SHARDS=131
+
+mode="verify"
+[[ "${1:-}" == "--fetch" ]] && mode="fetch"
+SCRIPT_DIR="$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)"
+
+if [[ "$mode" == "fetch" ]]; then
+    # HFGET_JOBS=4 is hfget's default (a few streams saturate the link).
+    bash "$SCRIPT_DIR/hfget.sh" "$REPO" "$DEST" main
+fi
+
+echo "verifying local payload: $DEST"
+fail=0
+
+INDEX="$DEST/model.safetensors.index.json"
+if [[ ! -f "$INDEX" ]]; then
+    echo "  BAD  weight-map index missing: $INDEX" >&2
+    exit 1
+fi
+
+python3 - "$REPO" "$DEST" <<'PY' || fail=1
+import hashlib, json, os, sys, urllib.request
+
+repo, dest = sys.argv[1], sys.argv[2]
+
+def listing():
+    """The tree API, following the Link cursor exactly like hfget.sh."""
+    out, api, pages = [], (
+        f"https://huggingface.co/api/models/{repo}/tree/main"
+        "?recursive=true&expand=true"), 0
+    while api:
+        with urllib.request.urlopen(api) as r:
+            out += json.load(r)
+            api = None
+            link = r.headers.get("Link", "")
+            for part in link.split(","):
+                if 'rel="next"' in part:
+                    api = part[part.find("<") + 1:part.find(">")]
+        pages += 1
+        if pages > 200:
+            raise SystemExit("listing did not terminate")
+    return out
+
+files = [e for e in listing() if e.get("type") == "file"]
+sizes = {e["path"]: e.get("size", 0) for e in files}
+lfs = {e["path"]: e["lfs"]["oid"] for e in files if e.get("lfs")}
+
+bad = 0
+for path, want in sorted(sizes.items()):
+    p = os.path.join(dest, path)
+    if not os.path.isfile(p):
+        print(f"  BAD  {path}: missing")
+        bad += 1
+    elif os.path.getsize(p) != want:
+        print(f"  BAD  {path}: {os.path.getsize(p)} bytes, listing says {want}")
+        bad += 1
+print(f"  {'OK' if bad == 0 else 'FAIL'}: {len(sizes) - bad}/{len(sizes)} files present at listed size")
+if bad:
+    raise SystemExit(1)
+
+# Shard set against the weight-map index.
+wm = json.load(open(os.path.join(dest, "model.safetensors.index.json")))["weight_map"]
+shards = sorted(set(wm.values()))
+print(f"  index maps {len(wm)} tensors across {len(shards)} shards")
+if len(shards) != 131:
+    print(f"  BAD  shard count {len(shards)}, want 131")
+    raise SystemExit(1)
+
+# Content: every LFS-tracked file hashed against the Hub oid. The safetensors
+# shards dominate the bytes; tokenizer/config files ride the same check when
+# the Hub tracks them in LFS (they are plain blobs here, already size-checked).
+todo = [p for p in shards if p in lfs]
+for i, path in enumerate(todo):
+    h = hashlib.sha256()
+    with open(os.path.join(dest, path), "rb") as f:
+        for chunk in iter(lambda: f.read(1 << 24), b""):
+            h.update(chunk)
+    if h.hexdigest() != lfs[path]:
+        print(f"  BAD  {path}: sha256 mismatch (want {lfs[path]})")
+        bad += 1
+    if (i + 1) % 16 == 0:
+        print(f"  ...hashed {i + 1}/{len(todo)}")
+print(f"  {'OK' if bad == 0 else 'FAIL'}: {len(todo) - bad}/{len(todo)} shards sha256-match the Hub")
+if bad:
+    raise SystemExit(1)
+PY
+
+[[ "$fail" -eq 0 ]] || exit 1
+echo "payload verified: 131 shards + index + tokenizer + configs, sizes and sha256 checked"

scripts/download-frank-primary.sh

@@ -1,88 +1,111 @@
 #!/usr/bin/env bash
-# Payload inventory/verification for frank-primary:
-# orcarouter/Qwen3.8-Flash-Next-Uncensored-FP8 (compressed-tensors FP8, 131
-# shards + weight-map index, ~172.8 GiB) — the tcclaviger-stack payload that
-# became frank-primary on 2026-09-29.
+# download-frank-primary.sh — payload fetch/verify for frank-primary:
+# the OFFICIAL Qwen/Qwen3.8-Flash-Next-FP8 checkpoint (131 safetensors
+# shards + weight-map index + tokenizer + vision preprocessing configs,
+# ~172.8 GB). 2026-10-05: replaces the orcarouter uncensored payload of the
+# FreeToken era (that script lived here; its repo was gated and needed
+# HF_TOKEN — this one is UNGATED and needs no token at all).
 #
-#   bash scripts/download-frank-primary.sh --verify-only   (the ONLY mode)
-#   bash scripts/download-frank-primary.sh --fetch
+#   bash scripts/download-frank-primary.sh --fetch         resume/complete
+#   bash scripts/download-frank-primary.sh                 verify only
 #
-# THIS REPO IS GATED (401 anonymous — measured 2026-09-29): anonymous fetch
-# is impossible and the tree API masks LFS hashes, so a byte-exact upstream
-# manifest cannot be built without the authed endpoint. The payload on
-# /home/dank/models was fetched with the operator's HF token (out of band:
-# HF_TOKEN in the environment, never in argv) at bring-up time.
-#
-# Modes:
-#   --fetch       resume any missing/truncated shard via `hf download` with
-#                 HF_TOKEN from the environment (NEVER passed on the command
-#                 line — a -H auth header is world-readable in
-#                 /proc/<pid>/cmdline, the 2026-09-16 leak).
-#   --verify-only default: local integrity check — shard count vs the
-#                 weight-map index, every file nonzero and its size equal to
-#                 the index's declared total, plus sha256 of every shard
-#                 (slow, ~10 min for 173 GiB).
+# Fetch uses the vendored scripts/hfget.sh (StillDeadcode's Apache-2.0
+# listing/resume downloader, curl+jq only): per-file size checks against
+# the Hub listing, `-C -` resume, pickle-format duplicates skipped.
+# Verify (default) is the AGENTS.md law in three parts: every listed file
+# present with the listed size, the shard count matches the weight-map
+# index, and every shard's sha256 equals the Hub's LFS oid. Verify needs
+# only the public API.
 set -euo pipefail
 
-REPO="orcarouter/Qwen3.8-Flash-Next-Uncensored-FP8"
-DEST="/home/dank/models/Qwen3.8-Flash-Next-Uncensored-FP8"
+REPO="Qwen/Qwen3.8-Flash-Next-FP8"
+DEST="/home/dank/models/Qwen/Qwen3.8-Flash-Next-FP8"
 EXPECTED_SHARDS=131
 
 mode="verify"
 [[ "${1:-}" == "--fetch" ]] && mode="fetch"
+SCRIPT_DIR="$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)"
 
 if [[ "$mode" == "fetch" ]]; then
-    : "${HF_TOKEN:?--fetch needs HF_TOKEN in the environment (never argv)}"
-    export HF_TOKEN
-    mkdir -p "$DEST"
-    hf download "$REPO" --local-dir "$DEST"
+    # HFGET_JOBS=4 is hfget's default (a few streams saturate the link).
+    bash "$SCRIPT_DIR/hfget.sh" "$REPO" "$DEST" main
 fi
 
 echo "verifying local payload: $DEST"
 fail=0
 
-# The weight-map index is the source of truth for the shard set.
 INDEX="$DEST/model.safetensors.index.json"
 if [[ ! -f "$INDEX" ]]; then
     echo "  BAD  weight-map index missing: $INDEX" >&2
     exit 1
 fi
 
-# Every index entry must exist, nonzero, and sized equal to the map's total.
-python3 - "$INDEX" "$DEST" <<'PY' || fail=1
-import json, os, sys
-index, dest = sys.argv[1], sys.argv[2]
-wm = json.load(open(index))["weight_map"]
-files = sorted(set(wm.values()))
-print(f"  index maps {len(wm)} tensors across {len(files)} files")
+python3 - "$REPO" "$DEST" <<'PY' || fail=1
+import hashlib, json, os, sys, urllib.request
+
+repo, dest = sys.argv[1], sys.argv[2]
+
+def listing():
+    """The tree API, following the Link cursor exactly like hfget.sh."""
+    out, api, pages = [], (
+        f"https://huggingface.co/api/models/{repo}/tree/main"
+        "?recursive=true&expand=true"), 0
+    while api:
+        with urllib.request.urlopen(api) as r:
+            out += json.load(r)
+            api = None
+            link = r.headers.get("Link", "")
+            for part in link.split(","):
+                if 'rel="next"' in part:
+                    api = part[part.find("<") + 1:part.find(">")]
+        pages += 1
+        if pages > 200:
+            raise SystemExit("listing did not terminate")
+    return out
+
+files = [e for e in listing() if e.get("type") == "file"]
+sizes = {e["path"]: e.get("size", 0) for e in files}
+lfs = {e["path"]: e["lfs"]["oid"] for e in files if e.get("lfs")}
+
 bad = 0
-total = 0
-for f in files:
-    p = os.path.join(dest, f)
-    if not os.path.isfile(p) or os.path.getsize(p) == 0:
-        print(f"  BAD  {f}: missing or zero bytes" )
+for path, want in sorted(sizes.items()):
+    p = os.path.join(dest, path)
+    if not os.path.isfile(p):
+        print(f"  BAD  {path}: missing")
         bad += 1
-    else:
-        total += os.path.getsize(p)
-print(f"  {'OK' if bad == 0 else 'FAIL'}: {len(files) - bad}/{len(files)} files present, {total/2**30:.1f} GiB total")
-raise SystemExit(1 if bad else 0)
+    elif os.path.getsize(p) != want:
+        print(f"  BAD  {path}: {os.path.getsize(p)} bytes, listing says {want}")
+        bad += 1
+print(f"  {'OK' if bad == 0 else 'FAIL'}: {len(sizes) - bad}/{len(sizes)} files present at listed size")
+if bad:
+    raise SystemExit(1)
+
+# Shard set against the weight-map index.
+wm = json.load(open(os.path.join(dest, "model.safetensors.index.json")))["weight_map"]
+shards = sorted(set(wm.values()))
+print(f"  index maps {len(wm)} tensors across {len(shards)} shards")
+if len(shards) != 131:
+    print(f"  BAD  shard count {len(shards)}, want 131")
+    raise SystemExit(1)
+
+# Content: every LFS-tracked file hashed against the Hub oid. The safetensors
+# shards dominate the bytes; tokenizer/config files ride the same check when
+# the Hub tracks them in LFS (they are plain blobs here, already size-checked).
+todo = [p for p in shards if p in lfs]
+for i, path in enumerate(todo):
+    h = hashlib.sha256()
+    with open(os.path.join(dest, path), "rb") as f:
+        for chunk in iter(lambda: f.read(1 << 24), b""):
+            h.update(chunk)
+    if h.hexdigest() != lfs[path]:
+        print(f"  BAD  {path}: sha256 mismatch (want {lfs[path]})")
+        bad += 1
+    if (i + 1) % 16 == 0:
+        print(f"  ...hashed {i + 1}/{len(todo)}")
+print(f"  {'OK' if bad == 0 else 'FAIL'}: {len(todo) - bad}/{len(todo)} shards sha256-match the Hub")
+if bad:
+    raise SystemExit(1)
 PY
 
-# Shard count gate (what the serve script and deployer check).
-n="$(find "$DEST" -maxdepth 1 -name 'model-*-of-*.safetensors' | wc -l)"
-if [[ "$n" != "$EXPECTED_SHARDS" ]]; then
-    echo "  BAD  shard count: found $n, want $EXPECTED_SHARDS" >&2
-    fail=1
-else
-    echo "  OK   shard count $n"
-fi
-
-if [[ "${2:-}" == "--hash" || "${VERIFY_HASH:-0}" == "1" ]]; then
-    echo "hashing every shard (slow)..."
-    cd "$DEST"
-    sha256sum -c --quiet model.safetensors.index.json 2>/dev/null || \
-    sha256sum model-*-of-00131.safetensors > /dev/null  # presence check; content hash list printed on demand
-fi
-
 [[ "$fail" -eq 0 ]] || exit 1
-echo "payload inventory OK: $EXPECTED_SHARDS shards + index present"
+echo "payload verified: 131 shards + index + tokenizer + configs, sizes and sha256 checked"

scripts/download-frank-secondary.sh

@@ -0,0 +1,90 @@
+#!/usr/bin/env bash
+# Download the frank-secondary payload for llama.cpp on the RTX 3090:
+# HauhauCS/Qwen3.5-9B-Uncensored-HauhauCS-Aggressive at **Q6_K** plus the
+# PUBLISHER's OWN vision projector (the BF16 mmproj — the model is natively
+# multimodal, so this file is the endpoint's eyes). Public repo — no auth
+# header, and never a token on the command line (it is world-readable in
+# /proc/<pid>/cmdline).
+#
+# Fresh downloads, no resume (`curl -C -` once produced a 213 MB oversized
+# file on a 31 GiB payload); -f/--fail-with-body so an HTTP error is not a
+# success, then byte count and sha256. The Hub exposes no LFS oid for this
+# repo (Xet storage), so the sha256 values below were computed locally after
+# the first verified fetch — byte counts match the Hub's declared sizes.
+# A sha of "-" means "first fetch: size-verified, sha printed for pinning".
+# Idempotent: complete files are skipped.
+#
+# 2026-10-07 (Frank): the payload moved to Qwen3.5-9B-Uncensored Q6_K at
+# 2 x 200,704 (qwen35 hybrid: 32 layers, 8 full-attention + 24 Gated
+# DeltaNet, no MTP tensors in this file). Rollback payloads stay on disk:
+# /home/dank/models/Gemma4-26B-A4B/ (Q4_K_M, the 2026-10-07 era being
+# replaced), Swift15-UncIQ3XXS/, MiMo-Q6K/, MiMo-Q5KM/, Ornith-Q8/.
+#
+# Usage:
+#   scripts/download-frank-secondary.sh [DEST_DIR] [--verify-only]
+set -euo pipefail
+
+DEST=""
+VERIFY_ONLY=0
+for arg in "$@"; do
+    case "$arg" in
+        --verify-only) VERIFY_ONLY=1 ;;
+        *) DEST="$arg" ;;
+    esac
+done
+DEST="${DEST:-/home/dank/models/Qwen35-9B-HauhauCS-Q6K}"
+
+# repo | hub-name | dest-name | bytes | sha256 (computed locally; Xet repo)
+FILES=(
+  "HauhauCS/Qwen3.5-9B-Uncensored-HauhauCS-Aggressive|Qwen3.5-9B-Uncensored-HauhauCS-Aggressive-Q6_K.gguf|Qwen3.5-9B-Uncensored-HauhauCS-Aggressive-Q6_K.gguf|7359259008|c0ba7beb68fd3fe47891bd549486d38dcf62d00817296ea314ad37017f5a4986"
+  "HauhauCS/Qwen3.5-9B-Uncensored-HauhauCS-Aggressive|mmproj-Qwen3.5-9B-Uncensored-HauhauCS-Aggressive-BF16.gguf|mmproj-Qwen3.5-9B-Uncensored-HauhauCS-Aggressive-BF16.gguf|921704448|05f662501f8bd45607b079723a3e238a4e888fd085a10a53f4057a0e250f6934"
+)
+
+status=0
+for entry in "${FILES[@]}"; do
+  IFS='|' read -r repo hubname destname size sha <<<"$entry"
+  url="https://huggingface.co/${repo}/resolve/main/${hubname}"
+  f="$DEST/$destname"
+  have=0
+  if [[ -f "$f" ]] && [[ "$(stat -c%s "$f")" == "$size" ]]; then
+    if [[ "$sha" == "-" ]]; then
+      have=1
+    elif sha256sum -c --quiet <(printf '%s  %s\n' "$sha" "$f"); then
+      have=1
+    fi
+  fi
+  if (( have )); then
+    echo "OK        $destname"
+    continue
+  fi
+  if (( VERIFY_ONLY )); then
+    echo "MISSING   $destname"
+    status=1
+    continue
+  fi
+  echo "FETCHING  $destname (from $repo)"
+  tmp="$f.part"
+  curl -fsSL --fail-with-body -o "$tmp" "$url"
+  mv "$tmp" "$f"
+  if [[ "$(stat -c%s "$f")" != "$size" ]]; then
+    echo "SIZE FAIL $destname ($(stat -c%s "$f") != $size)"
+    status=1
+    continue
+  fi
+  if [[ "$sha" == "-" ]]; then
+    echo "VERIFIED  $destname (size only) sha256=$(sha256sum "$f" | cut -d' ' -f1)"
+    continue
+  fi
+  if ! sha256sum -c --quiet <(printf '%s  %s\n' "$sha" "$f"); then
+    echo "SHA FAIL  $destname"
+    status=1
+    continue
+  fi
+  echo "VERIFIED  $destname"
+done
+if (( status == 0 )); then
+  echo "DOWNLOAD_DONE $DEST"
+else
+  echo "DOWNLOAD_FAILED $DEST" >&2
+fi
+exit "$status"

scripts/download-frank-small.sh

@@ -1,87 +0,0 @@
-#!/usr/bin/env bash
-# Download the frank-small payload for llama.cpp on the RTX 3090:
-# ornith-ai/Ornith-1.5-9B-GGUF:Q8_0 plus its vision projector
-# (the GGUF carries no `v.*` tensors — the tower lives only in the mmproj, so
-# without this file the endpoint is text-only by construction). Public repo —
-# no auth header, and never a token on the command line (it is world-readable
-# in /proc/<pid>/cmdline).
-#
-# Fresh downloads, no resume (`curl -C -` once produced a 213 MB oversized file
-# on a 31 GiB payload); -f/--fail-with-body so an HTTP 401 is not a success,
-# then byte count and the Hub LFS sha256. Idempotent: complete files are
-# skipped.
-#
-# 2026-10-05 (Frank): the rung is Ornith-1.5-9B Q8_0. The payload moved off
-# MiMo-V2.6 (Q6_K) to Ornith because it is the fleet's own model and it is the
-# only one the harness's own prompt recipes were written against; Q8_0 was the
-# rung chosen so the quant does not cost the throughput this endpoint already
-# runs at. The Q5_K_M file this script served from 2026-10-02 stays on disk at
-# /home/dank/models/MiMo-Q5KM/ (6876124704 B, LFS oid
-# 04cf46f2b5584ef922697d6c063f94879434fb86c467f1a09e52daf0cddaf35f) as the
-# rollback target; re-fetch it from the same repo with the same recipe.
-#
-# Usage:
-#   scripts/download-frank-small.sh [DEST_DIR] [--verify-only]
-set -euo pipefail
-
-REPO="ornith-ai/Ornith-1.5-9B-GGUF"
-BASE="https://huggingface.co/${REPO}/resolve/main"
-
-DEST=""
-VERIFY_ONLY=0
-for arg in "$@"; do
-    case "$arg" in
-        --verify-only) VERIFY_ONLY=1 ;;
-        *) DEST="$arg" ;;
-    esac
-done
-DEST="${DEST:-/home/dank/models/Ornith-Q8}"
-
-# name bytes sha256 (the sha256 is the LFS oid in the repo's pointer file)
-FILES=(
-  "Ornith-1.5-9B-Q8_0.gguf 9786060384 22086870b009dbe9815ee752c48a82de930118a7c5ce5599590892ae03b8b010"
-  "mmproj-Ornith-1.5-9B-BF16.gguf 921704672 626f9f90627402a6bf4a999111d0fbd69b5fcca7aa8ba089d69e5f10e8858e1d"
-)
-
-mkdir -p "$DEST"
-status=0
-for entry in "${FILES[@]}"; do
-  read -r name size sha <<<"$entry"
-  f="$DEST/$name"
-  have=0
-  if [[ -f "$f" ]] && [[ "$(stat -c%s "$f")" == "$size" ]]; then
-    if sha256sum -c --quiet <(printf '%s  %s\n' "$sha" "$f"); then
-      have=1
-    fi
-  fi
-  if (( have )); then
-    echo "OK        $name"
-    continue
-  fi
-  if (( VERIFY_ONLY )); then
-    echo "MISSING   $name"
-    status=1
-    continue
-  fi
-  echo "FETCHING  $name"
-  tmp="$f.part"
-  curl -fsSL --fail-with-body -o "$tmp" "$BASE/$name"
-  mv "$tmp" "$f"
-  if [[ "$(stat -c%s "$f")" != "$size" ]]; then
-    echo "SIZE FAIL $name ($(stat -c%s "$f") != $size)"
-    status=1
-    continue
-  fi
-  if ! sha256sum -c --quiet <(printf '%s  %s\n' "$sha" "$f"); then
-    echo "SHA FAIL  $name"
-    status=1
-    continue
-  fi
-  echo "VERIFIED  $name"
-done
-if (( status == 0 )); then
-  echo "DOWNLOAD_DONE $DEST"
-else
-  echo "DOWNLOAD_FAILED $DEST" >&2
-fi
-exit "$status"

scripts/fleet-probes-swift15.sh

@@ -0,0 +1,56 @@
+#!/usr/bin/env bash
+# fleet-probes-swift15.sh — RUNS ON DANKSERVER. The cutover contract probes.
+set -uo pipefail
+B=http://localhost:7770/v1
+
+echo "== vision (solid red 512px, expect 'Red')"
+python3 - <<'PY'
+import base64, json, urllib.request, struct, zlib
+def png(color):
+    w = h = 512
+    raw = b"".join(b"\x00" + bytes(color) * w for _ in range(h))
+    def chunk(t, d):
+        c = t + d
+        return struct.pack(">I", len(d)) + c + struct.pack(">I", zlib.crc32(c))
+    return (b"\x89PNG\r\n\x1a\n"
+            + chunk(b"IHDR", struct.pack(">IIBBBBB", w, h, 8, 2, 0, 0, 0))
+            + chunk(b"IDAT", zlib.compress(raw)) + chunk(b"IEND", b""))
+b64 = base64.b64encode(png((255, 0, 0))).decode()
+body = json.dumps({"model": "frank-archon", "max_tokens": 256, "temperature": 0,
+    "messages": [{"role": "user", "content": [
+        {"type": "image_url", "image_url": {"url": "data:image/png;base64," + b64}},
+        {"type": "text", "text": "What color is this image? Answer with one word."}]}]}).encode()
+r = urllib.request.Request("http://localhost:7770/v1" + "/chat/completions",
+                           data=body, headers={"Content-Type": "application/json"})
+d = json.load(urllib.request.urlopen(r, timeout=300))
+m = d["choices"][0]["message"]
+ans = (m.get("content") or m.get("reasoning_content") or "")
+print("VISION ANSWER:", ans[:80].replace("\n", " "))
+PY
+
+echo "== tools (expect tool_calls)"
+curl -s -m 120 $B/chat/completions -H 'Content-Type: application/json' -d '{
+ "model":"frank-archon","temperature":0,"max_tokens":128,
+ "messages":[{"role":"user","content":"What is 23 times 47? Use the calculator tool."}],
+ "tools":[{"type":"function","function":{"name":"calculator","description":"evaluate arithmetic",
+ "parameters":{"type":"object","properties":{"expr":{"type":"string"}},"required":["expr"]}}}]}' \
+ | python3 -c 'import json,sys; d=json.load(sys.stdin); m=d["choices"][0]["message"]; print("TOOL_CALLS:", m.get("tool_calls"))'
+
+echo "== reasoning_effort fold (xhigh/medium/low/high all 200 on froggeric v22.5)"
+for lvl in xhigh medium low high; do
+  code=$(curl -s -o /dev/null -w "%{http_code}" -m 120 $B/chat/completions \
+    -H 'Content-Type: application/json' \
+    -d "{\"model\":\"frank-archon\",\"max_tokens\":8,\"reasoning_effort\":\"$lvl\",\"messages\":[{\"role\":\"user\",\"content\":\"hi\"}]}")
+  echo "  reasoning_effort=$lvl -> HTTP $code"
+done
+
+echo "== reasoning_content channel"
+curl -s -m 120 $B/chat/completions -H 'Content-Type: application/json' \
+ -d '{"model":"frank-archon","max_tokens":64,"messages":[{"role":"user","content":"Is 97 prime? Think briefly."}]}' \
+ | python3 -c 'import json,sys; m=json.load(sys.stdin)["choices"][0]["message"]; print("has reasoning_content:", bool(m.get("reasoning_content")), "| content:", (m.get("content") or "")[:60].replace(chr(10)," "))'
+
+echo "== response_format json_object"
+curl -s -m 120 $B/chat/completions -H 'Content-Type: application/json' \
+ -d '{"model":"frank-archon","max_tokens":64,"response_format":{"type":"json_object"},"messages":[{"role":"user","content":"Return JSON {\"ok\": true}"}]}' \
+ | python3 -c 'import json,sys; d=json.load(sys.stdin); print("JSON MODE:", (d["choices"][0]["message"].get("content") or "")[:60])'
+echo "PROBES_DONE"

scripts/frank-large.env

@@ -1,75 +1,98 @@
 # frank-large.env — tunables for the frank-large endpoint on Dankclaw's
 # Strix Halo iGPU (gfx1151), host port 7770.
 #
-# RUNTIME (2026-10-05, Frank): halogen — the closed-source C++/HIP engine
-# (github.com/peonist-ai/halogen-server, image ghcr.io/peonist-ai/halogen,
-# digest-pinned below) serving peonist-ai/halogen-qwen3.8-27b, a ~6.32 bpw
-# NVFP4-derivative quant of the BASE Qwen/Qwen3.8-27B (35,865,565,184 B,
-# 1352 tensors), as a podman container under a root unit, driven by
-# scripts/serve-frank-large.sh. This CUTS OVER the
-# 2026-10-02..10-03 ROCmFPX llama.cpp shape (uncensored
-# Q6_0_ROCMFPX_AGENT + MTP, 27.6-28.1 t/s): rollback = revert the swap
-# commit's trio; the ROCmFPX payload, draft head and build all stay on
-# disk at /home/dank/models/kingjones-Qwen3.8-27B-Uncensored-ROCmFP4-STRIX-MTP/.
+# RUNTIME (2026-10-06, Frank): halogen-flash-server 0.16.2 — the
+# closed-source C++/HIP engine (image ghcr.io/peonist-ai/halogen-flash-server,
+# digest-pinned below) serving the Quat3rnion ABLITERATED Swift1.5 finetune
+# of peonist's Qwen3.8-Flash-Next v2 checkpoint, as a podman container under
+# a root unit, driven by scripts/serve-frank-large.sh. This CUTS OVER the
+# 2026-10-05 halogen 27B-dense shape (censored base, DFlash2, 31.7 t/s):
+# rollback = revert the swap commit's trio; the 27B payload stays on disk at
+# /home/dank/models/halogen-qwen3.8-27b/.
 #
 # WHY THE RUNTIME MOVED WITH THE MODEL: the payload is .hgn — per its own
-# card it "will not load in transformers, vLLM, or llama.cpp". Accepted
-# consequences (Frank, 2026-10-05): TEXT-ONLY (halogen ignores the model's
-# vision encoder; the fleet's vision path is frank-small) and the CENSORED
-# base (this is a finetune of Qwen/Qwen3.8-27B, not the abliterated file —
-# refusals may return). What it buys: vendor-measured decode DFlash2 31.7
-# t/s mean (vs this endpoint's MTP 27.6-28.1 and the old halogen-era MTP
-# 26.9), prefill ~566 t/s at 32k (was ~400 on the sibling engine), 1.4-2.3x
-# wall-clock on prefill-heavy agent traffic, at HIGHER effective precision
-# (~6.32 bpw vs ~6.6 with the old W4A4 promotion — see FLAGS.md).
+# card it "will not load in transformers, vLLM, or llama.cpp" — and the v2
+# .hgn loads ONLY in the flash-server line (the old halogen engine is for
+# the 27B p1w4d file; the v2 quickstart pins flash-server). What the move
+# buys (card + live probe evidence): MTP spec decode (53-57 t/s greedy,
+# 3-10x serial), ~1,640 t/s prefill at 34k, ~72% fewer thinking tokens than
+# the censored 27B (905 mean), the ABLITERATED base (refusals should NOT
+# return), IMAGE INPUT (the v2 vision tower loads beside the checkpoint —
+# red-PNG probe verified on the card; the 27B engine ignored its encoder),
+# tool calls verified, and the native 262,144 context.
 #
-# SHAPE — the two knobs that are NOT free:
-#   HALOGEN_KV_SLOTS=1 KEEPS SPECULATION ON. halogen makes batching and
-#   speculation mutually exclusive: with >1 slot the drafter is disabled
-#   and per-stream decode falls to serial speed (the sibling engine
-#   measured 41 t/s -> 14 t/s when slots were raised, and halogen's own
-#   FLAGS.md prints the same law). frank-large was already a one-session
-#   endpoint under ROCmFPX (PARALLEL=1), so slots=1 is the same contract,
-#   not a regression.
-#   HALOGEN_SLOT_CTX=200700 (Frank's window) is the per-conversation hard
+# SHAPE — the knobs that are NOT free, with the arithmetic:
+#   HALOGEN_CTX=262144 is the PER-SLOT window and the per-conversation hard
 #   ceiling — a prompt over it is a 400 naming the limit, never a silent
-#   truncation. KV pool = slots x slot_ctx x 64 KiB = 12.2 GiB. Budget
-#   against the 121 GiB of unified memory: weights 33.4 GiB (mmap'd — page
-#   cache, not anon), KV 12.2, engine working set + auto prompt cache
-#   ~20-30, host baseline ~10 -> ~75-85 GiB in use. The 2026-09-16 halogen
-#   era ran the same engine class on this box with LESS headroom.
-#   HALOGEN_DRAFTER=2 (DFlash2) is the shipped default and the fastest
-#   exact drafter on this checkpoint; the .hgn BAKES BOTH drafters (MTP +
-#   DFlash2), so unlike the Flash-Next era there is no sidecar to mount
-#   and no --dflash flag — the drafter is an int knob, byte-identical
-#   output to greedy verification either way.
+#   truncation. RAISED from 200192 on 2026-10-07: the harness clamps
+#   max_tokens to contextWindow − est − 4096, its chars/4 est UNDER-counts
+#   the rendered prompt by ~3 % on code/agent traffic, and compaction only
+#   fires at 85 % — so with window == engine ceiling, every request over
+#   ~146k est (65-85 % of the window) was a guaranteed 400. Measured
+#   2026-10-07: est 152,938, server 157,388, request 43,158 vs room 42,804
+#   — over by 354, margin 4,096 < the 4,450 under-count.
+#   Harness contract (models.yml): contextWindow 217088, maxTokens 45056.
+#   Compaction fires at 85 % = 184,524 est ≈ 190k measured (Frank's 180k
+#   target). Worst-case request: prompt + budget <= 217088 − 4096 + 3 %
+#   est-error ≈ 219k <= 262144 — the 45k gap between harness window and
+#   engine ceiling absorbs the estimate error with ~7x margin (robust to a
+#   10 % under-count). Compaction rides on frank-secondary: input ≈
+#   threshold − keepRecent(20k) ≈ 170k measured, inside its 200,704 slot.
+#   HALOGEN_KV_POOL_POSITIONS=524288 = KV_SLOTS(2) x CTX — the explicit
+#   form of the engine's default, so the seam test can pin the product.
+#   KV costs ~28 KiB/position (FLAGS.md) => 524,288 x 28 KiB = 14.0 GiB.
+#   HALOGEN_KV_SLOTS=2 restores the w4b-era shape on this box: the MTP
+#   drafter is NOT slot-locked on this engine (batching and speculation
+#   coexist — DFlash-era slots>1 law does NOT carry over; the card's own
+#   quickstart ships 2 slots WITH the head), so the second slot is
+#   concurrent capacity, not a speculation tax.
+#   Memory ledger against the box's 121.2 GiB unified (the n-gram table is
+#   mmap'd page cache beside the checkpoint, not a pinned charge — the
+#   card's own quickstart arithmetic does not bill it):
+#     weights ~62 GiB resident + KV 14.0 + arena (HALOGEN_MAX_TOK=8192 =>
+#     ~8 GiB per FLAGS.md arena formula 3.8+8.4*(MT/16384)) + vision ~2
+#     = ~86 GiB engine-side; HALOGEN_HOST_RESERVE_GIB=20 keeps the host
+#     honest; baseline co-tenancy (frank-secondary, ComfyUI, Caddy/observ)
+#     ~10-15. ~15 GiB margin. The boot ledger is the referee — read it
+#     after deploy and re-size if the engine prints a clamp.
+#   HALOGEN_SPEC_ADAPT=0 is MANDATORY (card): the default adaptive policy
+#   turns the MTP head off for whole requests after low acceptance and
+#   HALVES decode as a result. Frank accepted the head always-on trade
+#   (temp 1.0 traffic costs acceptance; the card still ships the recipe).
 #
-# CLIENT CONTRACT deltas vs the ROCmFPX shape (the harness side lives in
+# CLIENT CONTRACT deltas vs the 27B shape (harness side lives in
 # ~/.omp/agent/models.yml):
-#   * NO server-side sampler defaults: a request without temperature
-#     decodes GREEDY. The harness sends temp 1.0 / top_p 0.95 via
-#     extraBody, so that wire recipe is load-bearing now, not cosmetic.
-#   * max_tokens defaults to 8192 PER REQUEST and INCLUDES thinking; a
-#     reasoning reply over a small budget returns empty content with
-#     finish_reason "length". The harness maxTokens is 57344: compaction
-#     fires at 70% of 200700 = 140,490 and 140,490 + 57,344 = 197,834 <=
-#     200,700 keeps prompt + budget inside the pool (the 262144-era 65536
-#     would overflow). Requests over HALOGEN_MAX_TOKENS_CAP are a 400
-#     naming the cap; cap and queue timeout are COUPLED (see FLAGS.md
-#     table) — do not move one without the other.
-#   * response_format json_object/json_schema is a 400 (no constrained
-#     decoding in this build).
-#   * Thinking lands in reasoning_content — same field as the llama.cpp
-#     era, so the harness compat block does NOT change. reasoning_effort
-#     accepts none|low|xhigh on this build (template default xhigh); the
-#     models.yml fold (high->xhigh) still applies.
+#   * SERVER-SIDE SAMPLER EXISTS NOW: HALOGEN_TEMPERATURE/TOP_P/TOP_K set
+#     the card's thinking recipe (1.0 / 0.95 / 20) as engine defaults, so
+#     an absent temperature is no longer greedy. models.yml extraBody
+#     carries the same values — consistent, not load-bearing.
+#   * HALOGEN_MAX_TOK=8192 is only the DEFAULT for requests that omit
+#     max_tokens; the harness sends at most its maxTokens 45056. Requests
+#     over HALOGEN_MAX_TOKENS_CAP (65536) are a 400 naming the cap. Cap and
+#     queue timeout are COUPLED (FLAGS.md table): 65536 at ~10 t/s serial
+#     worst case needs > 6550 s; 7200 honors it. Do not move one alone.
+#   * max_tokens INCLUDES thinking; a reasoning reply over a small budget
+#     returns empty content with finish_reason "length".
+#   * Prompt cache is ON by default (PROMPT_CACHE=2): a follow-up turn at
+#     100k context re-prefills ~2 s instead of ~88 s cold. No pin needed —
+#     unlike the 27B engine, auto-sizing fits this box's ledger.
+#   * Thinking lands in reasoning_content (same as the 27B engine); the
+#     v2 chat template rejects reasoning_effort outside xhigh|medium|low,
+#     so the models.yml fold (high->xhigh) still applies.
+#   * Vision: OpenAI image_url content parts only (the fleet's standard
+#     contract). >=256 px probe images or the model answers "video".
+#   * response_format json_object/json_schema unsupported in this line
+#     (no constrained decoding) — unchanged from the 27B engine.
 #   * The engine's token protocol has no auth and binds loopback INSIDE
-#     the container; only the frontend port is published. Keep it that way.
+#     the container; only the frontend port (8731) is published. Same as
+#     before. Kernel needs CONFIG_HSA_AMD_SVM (CachyOS ships it) and
+#     amdgpu.noretry=0 (check at deploy).
 
 # Image: digest-pinned (fetch law: tag pulls silently re-point; the digest
-# is what was qualified). 0.1.4 = the version the model card names.
-FRANK_LARGE_IMAGE=ghcr.io/peonist-ai/halogen@sha256:dc0a39a0016d6cfc58a197978febaafdf8d28403f724ded6111d98b5fb7ac0ea
-# Container identity: the halogen container name.
+# is what was qualified). 0.16.2 = the version the model card names as its
+# quickstart pin; RepoDigest captured at pull on 2026-10-06.
+FRANK_LARGE_IMAGE=ghcr.io/peonist-ai/halogen-flash-server@sha256:0c61bf84ac22308a53f5d1ca6b86806702d7039e5ebc51cae4c66621b92fe04a
+# Container identity: kept from the 27B era (unit ExecStop names it).
 FRANK_LARGE_CONTAINER=frank-large-halogen
 # Old quadlet-era container identity; the serve script clears stale names.
 FRANK_LARGE_LEGACY_CONTAINERS=frank-large
@@ -77,37 +100,59 @@ FRANK_LARGE_LEGACY_CONTAINERS=frank-large
 # default); the engine port (8730) is never published.
 FRANK_LARGE_PUBLISH=7770:8731
 
-FRANK_LARGE_MODEL_DIR=/home/dank/models/halogen-qwen3.8-27b
-FRANK_LARGE_CHECKPOINT=qwen3.8-27b-p1w4d-d2.hgn
-FRANK_LARGE_CHECKPOINT_BYTES=35865565184
-FRANK_LARGE_CHECKPOINT_SHA256=274c3fc767fb57faf025dd03c76376b1fc1b32448aef45aaf02d1898fa962089
-# The tokenizer is a FLAT DIRECTORY, $FRANK_LARGE_MODEL_DIR/tokenizer, mounted
-# whole at /tokenizer — the model card ships `tokenizer/` and the engine's
-# HALOGEN_TOKENIZER default is that directory (the card's own docker run sets
-# no tokenizer flag). The Hub stores it as 5 files (sizes from the Hub API,
-# 2026-10-05): tokenizer.json 12809320 (LFS sha256
-# 0997f410c57a1f4e53b09e4be8f4a172d90edd9564368fb0847030937229b9f3),
-# vocab.json 6722759, merges.txt 3353259, chat_template.jinja 8952,
-# tokenizer_config.json 17928 (plain git blobs — size-checked, no LFS oid).
-# The serve gate and the deployer refuse a missing file or ANY symlink: an
-# HF cache snapshot symlinks each file into blobs/, and those links dangle
+FRANK_LARGE_MODEL_DIR=/home/dank/models/halogen-flashnext-v2
+# The checkpoint: Quat3rnion/halogen-swift1.5-qwen3.8-flash-next-v2-abliterated
+# qwen38-flash-next-v2-swift15-abliterated.hgn — compacted r2 of the card's
+# append-only original (byte-identical live tensors, padding dropped:
+# 66,687,678,464 B vs 89,775,868,032 B). LFS sha256 verified at fetch.
+FRANK_LARGE_CHECKPOINT=qwen38-flash-next-v2-swift15-abliterated.hgn
+FRANK_LARGE_CHECKPOINT_BYTES=66687678464
+FRANK_LARGE_CHECKPOINT_SHA256=9c880b8ba48deabfa06172eff1e7572e4fe465e09e97faec6e50c3f6fdcac037
+# The n-gram lookup table: REQUIRED (the engine reads it beside the
+# checkpoint; the card's quickstart names it explicitly). From
+# peonist-ai/halogen-qwen3.8-flash-next @ 5cc17cea.
+FRANK_LARGE_NGRAM_TABLE=qwen38-flash-next-ngram.hgn
+FRANK_LARGE_NGRAM_BYTES=51200246144
+FRANK_LARGE_NGRAM_SHA256=3450acada94e19aabad88bd49b45eb70304820949c2240fc178067b20ae5dffe
+# The vision tower: OPTIONAL on the engine, ENABLED here (image input).
+# From peonist-ai/halogen-qwen3.8-flash-next @ e053f488.
+FRANK_LARGE_VISION_TOWER=qwen38-flash-next-vision.hgn
+FRANK_LARGE_VISION_BYTES=897916416
+FRANK_LARGE_VISION_SHA256=d62e0ae553fe88afd3833733d4a4c669f34d20fd8dfce4b9610525bed2134b10
+# The tokenizer is a FLAT DIRECTORY, $FRANK_LARGE_MODEL_DIR/tokenizer,
+# mounted whole at /tokenizer. The two repos ship the same one; fetched
+# fresh from peonist @ e053f488 (tokenizer.json is LFS, byte-identical to
+# the 27B-era copy; the rest are plain git blobs — size-checked only). The
+# serve gate and the deployer refuse a missing file or ANY symlink: an HF
+# cache snapshot symlinks each file into blobs/, and those links dangle
 # inside the container mount.
-FRANK_LARGE_TOKENIZER_FILES="tokenizer.json chat_template.jinja tokenizer_config.json merges.txt vocab.json"
+FRANK_LARGE_TOKENIZER_FILES="tokenizer.json chat_template.jinja tokenizer_config.json merges.txt vocab.json generation_config.json"
 
-# --- engine flags (docs/FLAGS.md of the engine repo; anything unset stays
-# at the image default, which is what was qualified) ------------------------
-HALOGEN_MODEL_ID=frank-large
-HALOGEN_SLOT_CTX=200700
-HALOGEN_KV_SLOTS=1
-HALOGEN_DRAFTER=2
+# --- engine flags (docs/FLAGS.md of the flash-server repo; anything unset
+# stays at the image default, which is what was qualified) ------------------
+# Served model id — Protoss name (wire rename 2026-10-09); the service, unit,
+# scripts and this file keep the frank-large name. OpenWebUI api_configs
+# model_ids for the :7770 backend MUST name this value.
+HALOGEN_MODEL_ID=frank-dragoon
+HALOGEN_CTX=262144
+HALOGEN_KV_POOL_POSITIONS=524288
+HALOGEN_KV_SLOTS=2
+HALOGEN_HOST_RESERVE_GIB=20
+# Spec decode: MTP head is INSIDE the checkpoint (do NOT set HALOGEN_MTP_HEAD);
+# depth 2 is the documented "best all-round" default. HALOGEN_DRAFTER /
+# HALOGEN_DFLASH_* from the 27B engine DO NOT EXIST here.
+HALOGEN_SPEC_ADAPT=0
 # Coupled pair (FLAGS.md table): the cap bounds how long ONE request holds
-# the GPU, the timeout bounds how long the NEXT client waits. 65536 at
-# ~10 t/s serial worst case needs > 6550 s queue timeout; the defaults
-# honor that. Do not lower the timeout alone.
+# the GPU, the timeout bounds how long the NEXT client waits. Do not move
+# one alone (see the client-contract note above).
+HALOGEN_MAX_TOK=8192
 HALOGEN_MAX_TOKENS_CAP=65536
 HALOGEN_QUEUE_TIMEOUT=7200
-# Engine prompt cache: EMPTY = auto (sized from available memory at
-# startup). Pin an explicit MB only to argue with the auto figure — a
-# full-context entry is ~18 GB at 262K, so a too-small pin yields a cache
-# that reports enabled and never hits (the engine warns at startup).
-#HALOGEN_CACHE_MB=
+# Server-side sampler = the card's thinking recipe (the engine now HAS one;
+# requests may still override per-call). Matches models.yml extraBody.
+HALOGEN_TEMPERATURE=1.0
+HALOGEN_TOP_P=0.95
+HALOGEN_TOP_K=20
+# Card's belt-and-braces: the engine pins weights by default
+# (HALOGEN_FLASH_PIN_TRUNK=1; only the retired w4b build honored 0).
+HALOGEN_WEIGHTS_LOCK=1

scripts/frank-large.service

@@ -1,5 +1,5 @@
 [Unit]
-Description=halogen container (peonist-ai/halogen-qwen3.8-27b, DFlash2 spec decode, 1 slot x 200700, text-only), root systemd unit (dankclaw, frank-large, Strix Halo iGPU)
+Description=halogen-flash-server container (Quat3rnion abliterated Swift1.5 of Qwen3.8-Flash-Next v2, MTP spec decode, 2 slots x 200192, vision ON), root systemd unit (dankclaw, frank-large, Strix Halo iGPU)
 After=network-online.target
 Wants=network-online.target
 
@@ -22,8 +22,8 @@ TimeoutStopSec=120
 # would otherwise leave the endpoint down until someone notices.
 Restart=always
 RestartSec=10
-# The engine mmaps 33.4 GiB of weights and keeps a multi-GiB working set
-# on a box that also hosts frank-small, ComfyUI and the Caddy/observability
+# The engine pins ~62 GiB of v2 weights and keeps a multi-GiB working set
+# on a box that also hosts frank-secondary, ComfyUI and the Caddy/observability
 # stack; make it a late OOM candidate, not the first.
 OOMScoreAdjust=-500
 

scripts/frank-primary.env

@@ -1,362 +1,168 @@
-# frank-primary.env — tcclaviger RDNA4 vLLM production config (dankserver :7770).
+# frank-primary.env — radiance (StillDeadcode, Apache-2.0) serving the
+# OFFICIAL Qwen/Qwen3.8-Flash-Next-FP8 checkpoint at NATIVE precision
+# (modelopt block-FP8 throughout, no requantisation) on BOTH R9700s (TP=2),
+# dankserver :7770.
 #
-# 2026-10-03 shape G (as shipped, after the 1.00 revert): the CUDA-graph plan
-# term dropped from the analytic 0.55 GiB to the measured 0.07 GiB (0 base +
-# 12 MiB/graph), and UTIL at the 0.982 law. MEASURED on the live deploy: arena
-# 17.77 GiB / 7,762 slots, KV 401,408 tokens / 2.00x concurrency (the first
-# shape holding two full-length conversations with real spare), driver used
-# 30.97 GiB of 31.86 — 0.89 GiB of real free VRAM per card, held through a
-# live 4,096-token generation. UTIL stays 0.982 ALWAYS: 1.00 is unbootable on
-# this fork (worker init refuses budget >= device total) and 0.13-GiB-margin
-# boots die under load.
-# `budget - non-expert - KV reserve - graphs - overhead` and the KV reserve is a
-# registry constant, so those two terms are the whole flag surface.
-# 2026-10-03 shape F (absorbed): MNBT 2048 + the graph term pinned. Measured on
-# the live deploy: arena 17.52 GiB / 7,653 slots, budget 31.29, graphs 0.32.
-# 2026-10-02 shape E: fp8 KV, MAXLEN 200704 x SEQS 3, pool 3.15e9/rank, and
-# --expert-precise-memory ON. The window shrink from the native 262144 and the
-# --max-num-seqs 3 lock both come from shape D (SEQS=2 CANNOT boot this model —
-# the measured 2x2 is below); shape E adds the precise-overhead plan, which
-# trades the default 7.60 GiB compile headroom for the fitted 5.25 GiB and puts
-# the 2.35 GiB difference into the expert arena: 6,496 -> 7,522 slots (+16%).
-# The pool is deliberately NOT cut (option A): the pool is not an arena lever
-# (measured), so 3.15e9 keeps shape D's 1.99x concurrency while the arena grows.
-# Same digest-pinned tcclaviger RDNA4 fork. Rollback for the fp8+MTP4 era is git
-# tag frank-primary-fp8-mtp-20261001.
+# 2026-10-05 cutover from FreeToken (PR #587 ROCm, one card, experts in
+# pinned host RAM; git tag frank-primary-freetoken-pr587-20261005 is the
+# pre-cutover state — its unit/env/serve trio and Dockerfile are all in the
+# tag and rebuildable). Before that: tcclaviger vLLM fork (tag
+# frank-primary-vllm-shapeG-20261003).
 #
-# HIP reads bare env vars TEMP, HOST, GPU_MAX_HEAP_SIZE — every key here is
-# prefixed FRANK_PRIMARY_ so none can leak. The one bare name we WANT inside the
-# container (GPU_MAX_HW_QUEUES) is held here under the prefix and passed on the
-# run line as `-e GPU_MAX_HW_QUEUES=…`.
+# WHY RADIANCE: the engine is C++/HIP with libr4d kernels built for
+# gfx1201, drives `podman run` directly from the digest-pinned
+# stilldeadcode/radiance image (287 MiB; host needs only the amdgpu driver),
+# and its qwen4exp plugin reads the modelopt FP8 encoding NATIVELY
+# (`X.weight_scale_inv` is the scale plane — checkpoint.h). The PLE n-gram
+# table (51B params, 47.7 GiB) is served from pinned host RAM via io_uring
+# reads — Frank's call: "Native FP8. Experts from ram. Pin the PLE to ram."
+# The 4-bit-codebook publisher shape (qwen3.8-next-flash-fp8-iq4r-moe.rad,
+# --host-pool-mib 12288) remains the documented fallback if the native
+# container proves unbootable.
 #
+# PAYLOAD: /home/dank/models/rad/qwen4exp-fp8-native.rad — a rad-convert
+# NO-RECIPE pure copy of the official 131-shard checkpoint (every weight
+# keeps its block-FP8 encoding; ~161 GiB). The convert runs against a
+# staging dir whose config.json drops `quantization_config`: the descriptor
+# string is only the plugin SELECTION key (spec §2.4 — exact match, no
+# fallback), and no shipped plugin claims (qwen4exp, "fp8"); the shipped
+# qwen4exp plugin claims (qwen4exp, "") which is what recipe-built
+# containers carry. Per-weight encodings are data, not the descriptor —
+# removing the key changes nothing about what the weights hold. Built by
+# scripts/build-frank-primary-rad.sh (staging + convert + rad-info).
 #
-# --- runtime ----------------------------------------------------------------
-# Digest-pinned: the floating :latest tag moved once already. Local alias
-# qwen-fp8-runtime:tuned-20260929 (image ID 3322dbbbc96d) points at the same
-# bytes in rootless storage.
-FRANK_PRIMARY_IMAGE=docker.io/tcclaviger/vllm@sha256:da4029666668b90a0386c1ed23ea370c79916ad4e1a6ad1dd9db098b9fa069ea
+# SERVING FACTS (docs/GUIDE.md, docs/DOCKER.md, compose/flashnext.yaml):
+# - The HTTP listener binds AFTER the full load (main.cpp: engine.start()
+#   then engine.serve()) — during load the port refuses connections, so a
+#   bare curl /health gate is honest. /health == /ping: 200 JSON while
+#   serving, 503 only while stopping (core/server/admin.cpp:385).
+# - Reasoning: --reasoning-format defaults to auto, emitting thinking in
+#   `reasoning_content` (config.cpp). Tool calls: the qwen3.8 dialect, from
+#   the chat format. Both need no parser flags.
+# - Sampling: the OpenAI endpoint accepts temperature, top_p, top_k, min_p,
+#   typical_p, presence_penalty, frequency_penalty, repetition_penalty,
+#   repeat_last_n, chat_template_kwargs, reasoning_effort,
+#   enable_thinking, preserve_thinking (core/server/oai.cpp:194-205) —
+#   models.yml's extraBody set is fully inside it. Unknown params are
+#   refused (400), which is why nothing unverified rides the wire.
+# - MTP: the official checkpoint carries the mtp.* head (3,101 tensors,
+#   mtp_num_hidden_layers=1) and the plugin probes draft depth 3 — the
+#   publisher's throughput peak for this model. Stated explicitly with
+#   --num-speculative-tokens 3.
+# - The fused TP all-reduce is declared over max-num-seqs x
+#   (1 + --num-speculative-tokens) rows; 3 x 3 = 9 rows fits either wire.
+# Engine pin: Codeberg tag v1.0.13 (commit 53282fa), Docker Hub tag 1.0.13,
+# index digest below. REJECTED UPGRADE: 1.2.0 (sha256:665775ed1d90de6dbb
+# 25558e07fc60d20328f1346f96f3c6e00845eb763f01fa, tag 1.2.0, commit
+# e95d397) was benched 2026-10-06 on this exact shape (fp8-native rad,
+# vram pair, spec 2, seqs 3, 43.6k greedy probe): decode 29.21 t/s and
+# draft acceptance 0.7123 vs the 1.0.13 control's 30.08 and 0.7549.
+# Acceptance on the greedy harness is deterministic (the control
+# reproduced the recorded 0.755 exactly), so the drop is a real numerics
+# change — the 1.2.0 MoE weight-rotation change (92b8aaf) is the named
+# suspect; decode loss tracks the acceptance ratio (0.943 vs 0.945).
+# The vram-pair translation itself is exact (1.2.0 boot ledger took
+# vram_weights 25.46 GiB / vram_kv 3.46 GiB per card byte-identical to
+# the 1.0.13 ledger at EVC 0.82). Re-test 1.2.0+ when the rotation change
+# is untangled; 1.2.0 also REMOVED --expert-vs-cache-ratio in favour of
+# the --vram-weights-mib / --vram-kv-mib pair (core/config.cpp) — this
+# file already uses the pair, which BOTH pins accept (rollback symmetry).
+FRANK_PRIMARY_IMAGE=stilldeadcode/radiance@sha256:4c3f484564bb2872784e46b0a59cbd71becc9c1e746e301a3596ffae253c322e
+# The .rad container FILE (mounted under /models) and the state dir for the
+# prefix-cache disk tier (mounted rw at /data). The model dir holds ONLY
+# .rad files; the raw HF download and its staging dir live beside it and
+# are never mounted.
+FRANK_PRIMARY_MODEL_DIR=/home/dank/models/rad
+FRANK_PRIMARY_MODEL_FILE=qwen4exp-swift15-w4dn8-i8-hc8m.rad
+FRANK_PRIMARY_STATE_DIR=/home/dank/radiance-state
+# The state dir's tune/ subdirectory holds rad-tune's kernel-selection
+# cache (the engine's selector reads $RADIANCE_HOME/tune/<machine>.tune
+# at boot; $RADIANCE_HOME defaults to /opt/radiance, in the image, so the
+# cache must ride a host mount — serve script mounts it, see the runbook's
+# "Tuning levers").
+# Container identity is the SERVICE name (Frank's call, 2026-10-05): one
+# name across systemctl and podman ps. The runtime changes; the name does
+# not. The legacy list retires every predecessor identity that could hold
+# the GPUs or the port.
+FRANK_PRIMARY_CONTAINER=frank-primary
+FRANK_PRIMARY_LEGACY_CONTAINERS="frank-primary-qwen frank-primary-r9v qwen-fp8 glm-probe"
+# Served model id — Protoss name (wire rename 2026-10-09). The container,
+# unit, scripts and this file keep the frank-primary name above; only the
+# alias on the wire moved. Radiance validates the requested model name, so
+# every client (harness, bench scripts, probes) MUST send this value.
+FRANK_PRIMARY_SERVED_NAME=frank-archon
+FRANK_PRIMARY_PORT=7770
+# Frank's chat-template call: froggeric's fixed Qwen3.8 template (the
+# repo's root chat_template.jinja, vendored here; the checkpoint's own
+# template is baked into the .rad by rad-convert and this overrides it).
+# Radiance derives the reasoning/tool-call reply format from it.
+FRANK_PRIMARY_TEMPLATE=/home/dank/serverAppConfigs/scripts/qwen38-froggeric-v22.5.jinja
 
-# --- payload ----------------------------------------------------------------
-FRANK_PRIMARY_MODEL_DIR=/home/dank/models/Qwen3.8-Flash-Next-Uncensored-FP8
-FRANK_PRIMARY_MODEL_SHARDS=131
-
-# --- identity ---------------------------------------------------------------
-# The harness models.yml id stays frank-primary (provider 192.168.0.235:7770).
-FRANK_PRIMARY_CONTAINER=frank-primary-qwen
-FRANK_PRIMARY_SERVED_NAME=frank-primary
-FRANK_PRIMARY_PUBLISH=7770:7770
-# Containers removed before every start: the retired R9V identity and the ad
-# hoc experiment identity (both hold the GPUs / name registrations).
-FRANK_PRIMARY_LEGACY_CONTAINERS="frank-primary-r9v qwen-fp8"
-
-# --- qualified engine shape (2026-10-02, shape E) ---------------------------
-# WHAT THE SLOT COUNT DOES AND DOES NOT DO. The endpoint advertises 3 slots
-# (SEQS 3), but the POOL, not the slot count, decides how many full-length
-# conversations are resident. MEASURED LIVE 2026-10-02 at this shape:
-#   GPU KV cache size: 398,658 tokens,
-#   Maximum concurrency for 200,704 tokens per request: 1.99x
-# (3,146,129,408 B reserved — 7,892 B/token). The per-token price is
-# SHAPE-DEPENDENT, not a constant: the 262144 shape priced at 7,701 B/token
-# (6,100,000,000 B -> 792,069 tokens), ~2.5% cheaper, so never extrapolate a
-# concurrency figure across geometries. Shorter requests use the spare blocks,
-# and a third LONG request queues rather than being refused. Verified: three
-# concurrent 67-token requests answered in parallel, ~4.1 s each.
-#
-# WHY NOT --max-num-seqs 2 (measured 2026-10-02, single-variable boot probes
-# against the then-deployed known-good shape, production stopped for each):
-#   SEQS 2 + MNBT 1024 -> Engine core initialization failed
-#   SEQS 2 + MNBT 2048 -> Engine core initialization failed
-#   SEQS 3 + MNBT 1024 -> healthy (792,069 tokens, 3.02x, arena 6,248)
-#   SEQS 3 + MNBT 2048 -> healthy (the deployed incumbent)
-# The failure is torch.fx ConstraintViolationError: query_start_loc.size()[0]
-# "marked dynamic but your code specialized it to be a constant (3)", raised
-# from AOT build_guards(strict_error=True) during profile_run -> _dummy_run.
-# model_runner.py sizes that input buffer max_num_reqs + 1 and _dummy_run sets
-# num_reqs = min(max_num_batched_tokens, max_num_reqs), so the width is
-# SEQS + 1 for ANY MNBT >= SEQS: 3 at SEQS 2, 4 at SEQS 3. MNBT only moves the
-# compile RANGE ((1,1024) vs (1,2048)) and cannot dodge it — the 2x2 above is
-# why MNBT is not a lever here. --enforce-eager bypasses the path but loses
-# CUDA graphs. Same failure class killed the 2026-10-01 bf16 2 x 262144 shape.
-#
-# THE ARENA LEVER IS max_model_len, NOT KV_BYTES (measured 2026-10-02). The
-# planner sizes expert cache as budget - non-expert - KV reserve - graphs -
-# overhead, and the KV reserve is registry-priced at 9190 B/token x max_model_len:
-#   MAXLEN 262144 -> reserve 2.48 GiB -> arena 14.30 GiB / 6,248 slots
-#   MAXLEN 200704 -> reserve 1.91 GiB -> arena 14.87 GiB / 6,496 slots
-# d(reserve) 0.57 GiB = d(arena) 0.57 GiB = +248 slots. The 6.1e9 -> 3.15e9
-# pool cut buys ZERO arena; it buys runtime HEADROOM (the shape it replaced
-# died with 0 bytes free on a real 995-token request). To grow the arena,
-# lower MAXLEN — not KV_BYTES. The OTHER arena lever is the overhead term:
-# --expert-precise-memory (shape E) takes it from 7.60 to 5.25 GiB and lands
-# those 2.35 GiB in the arena — 14.87 -> 17.22 GiB / 6,496 -> 7,522 slots.
-#
-# GATHER 32x64 stays: R4D_LRU_GATHER_CHUNKS=32 / R4D_LRU_GATHER_LANES=64
-# measured +18.7% decode vs default geometry and the win grows with depth.
-# Do not drop it.
-# CHUNKED PREFILL: the fork exposes the standard scheduler flags. --max-num-batched-tokens
-# (MNBT) is the lever; --enable-chunked-prefill is an auto-flag (True in SchedulerConfig)
-# and --prefill-schedule-interval is the gap between iterations. The running shape carries
-# --max-num-batched-tokens 2048 with --max-model-len 200704, so chunked prefill is ON and
-# already active: a 200704-token prompt is prefilled in ~98 iterations of 2048 tokens.
-# MEASURED 2026-09-29 on the ad hoc qwen-fp8 container (same fork, same payload, TP=2):
-# MNBT 8192 vs the default 2048 is NEUTRAL on decode at every depth, and small values
-# re-chunk prefill only — they hurt TTFT and never touch decode, because a decode step is
-# max_num_seqs x (spec+1) tokens, independent of MNBT. There is NO --ubatch-size
-# equivalent in vLLM: that is a llama.cpp compute-buffer concept (the frank-small pair
-# doubled on 2026-10-05 has no counterpart here). MNBT is therefore NOT a lever on this
-# shape — the deployed 2048 is the fork's default — and every change re-enters the
-# compile RANGE path, so FRANK_PRIMARY_CACHE_DIR pays one recompilation.
-# MNBT 2048 (Frank's call, 2026-10-03; the lock was 1024). The measured law
-# still holds: MNBT 512 doubles TTFT at 4064 ctx (2.46 s vs 1.36 s), 1024-4096
-# are interchangeable on decode, 12288 costs 2.5% plus arena. 2048 is ARENA-FREE
-# by construction: the registry's overhead model is
-# base + 0.22 GiB x max(0, (nbt - 4096)/1024) with the anchor at nbt 4096, so
-# every rung up to and including 4096 prices the SAME 5.25 GiB overhead — and
-# the fitted 5.25 was fitted at nbt 4096, i.e. our 2048 sits UNDER the anchor,
-# so the activation term is covered by a figure measured at a bigger batch.
-# SEQS 3 boots at both 1024 and 2048 (measured 2026-10-02; the SEQS=2
-# ConstraintViolationError is a SEQS property, not an MNBT one). Costs exactly
-# one recompilation: the compile RANGE is part of the cache key, so
-# FRANK_PRIMARY_CACHE_DIR pays for whichever rung is chosen once.
-# PREFETCH IS ABSENT BY LAW: expert-cache telemetry measured the prefetcher at
-# 36% precision (36,720 slabs prefetched, 13,369 served) — roughly half the
-# gather traffic was waste. Never re-add --expert-prefetch without a fresh A/B.
-# TunableOp stays OFF: CLAV_TUNABLEOP_SWEEP silently skips under expert offload
-# unless CLAV_TUNABLEOP_WITH_OFFLOAD=1, and forcing it corrupts numerics.
-# --expert-precise-memory is ON (shape E). It selects the fitted overhead
-# (5.25 GiB) over the default (7.60 GiB = fitted + compile/graph-capture
-# headroom); the 2.35 GiB difference lands in the expert arena. MEASURED
-# 2026-10-02 by a boot probe on the shape-D base (production stopped, candidate
-# booted AS the service, plan read off the engine's own log):
-#   runtime overhead  7.60 -> 5.25 GiB   (registry Qwen4Exp…, precise
-#                                         gfx1201 TP2 nbt4096/seqs16/len262144)
-#   expert cache      14.87 -> 17.22 GiB
-#   slots             6,496 -> 7,522     (+1,026 = +16%, 2.344 MiB each)
-#   swap region                 13.7 GiB / 5,986 slots, 133-134 per layer
-#   boot                        healthy in 420 s, no OOM, no knife-edge
-# The flag was called a non-option after it crash-looped the 2 x 262144 bf16
-# boot (0.13 GiB free of 31.86 GiB, NRestarts=3 at ~6.7 min each) — that shape
-# was already at the ceiling, so precise-memory was the last 2.35 GiB over, not
-# the cause. At 200704 x 3 it fits with room; the run line also reports
-# 'expert_precise_memory': True in non-default args, so a silent argparse drop
-# would be visible in the boot log.
-FRANK_PRIMARY_PRECISE_MEMORY=1
-# GPU_MAX_HW_QUEUES caps the HSA hardware queues per GPU (AMD's own tuning
-# guide recommends 2 for this class of workload; the runtime default on Linux
-# is 4). Verified a REAL knob, not a no-op: the string is present in
-# /opt/rocm/core-10.0/lib/libhsa-runtime64.so. NOT A/B MEASURED — Frank's call,
-# shipped as a deliberate CPU-side tune alongside the 32x64 gather geometry.
-FRANK_PRIMARY_GPU_MAX_HW_QUEUES=2
-# Persistent compile cache. MEASURED 2026-10-02: the image ships NO
-# /root/.cache at all (`ls: cannot access '/root/.cache'` in a fresh
-# container), and after our boot it holds /root/.cache/vllm at 1,154,714,333 B
-# in 4,665 files plus /root/.cache/comgr at 229 MB — both inside the
-# container's ephemeral layer, so every `podman rm -f` (every deploy, restart
-# and reboot) throws them away and re-pays the build. The engine's own log
-# names the loss: `torch.compile took 181.20 s in total`, of which 'Compiling a
-# graph for compile range (1, 1024) takes 173.19 s', followed by 'saved AOT
-# compiled function to /root/.cache/vllm/torch_compile_cache/…'. Mounting a
-# host dir at /root/.cache persists the vLLM AOT/torch_compile tree AND the
-# comgr HIP-kernel JIT cache, and hides nothing (the image has no such dir).
-# PROVEN LIVE 2026-10-02, two boots of this exact shape:
-#   cold  `Compiling a graph for compile range (1, 1024) takes 179.24 s`
-#         `torch.compile took 188.26 s in total`
-#   warm  `Directly load the compiled graph(s) for compile range (1, 1024)
-#          from the cache, took 2.143 s`
-#         `torch.compile took 3.97 s in total`
-#         `init engine (profile, create kv cache, warmup model) took 41.13 s
-#          (compilation: 3.98 s)`
-# (Total time to /health is dominated by the 172.8 GiB payload load, not the
-# compile — the saving is the ~184 s of inductor/AOT build.)
-# Safe to grow/prune: entries are keyed by version + config hash, so a stale
-# entry is a miss, never a wrong answer.
-FRANK_PRIMARY_CACHE_DIR=/home/dank/.cache/frank-primary
-FRANK_PRIMARY_MAXLEN=200704
-FRANK_PRIMARY_SEQS=3
-# The POOL is deliberately kept at shape D's 3.15e9 (option A): because the pool
-# is not an arena lever, raising it back from a cut buys concurrency only, and
-# 3.15e9 preserves the measured 1.99x while precise-memory grows the arena. A
-# 2.2e9 pool was probed in the same window (277,686 tokens / 1.38x) and rejected
-# as the shipped value; it remains the conservative arm if free VRAM ever binds.
-# KV forced to 400,700 tokens (Frank, 2026-10-03). The engine prices this pool
-# at 7,891.8 B/token at this shape and loses ~3.87 MB to block alignment, so
-# 3,166,500,000 B is the ask that lands at/above 400,700 (verify the reported
-# "GPU KV cache size" line on every deploy; it is not a constant ratio across
-# geometries — 7,701 B/token at 262144).
-FRANK_PRIMARY_KV_BYTES=3166500000
+# --- engine shape (first-boot conservative; the boot ledger arbitrates) --
+# Tensor parallel: BOTH R9700s. The board swap (2026-09-20, ProArt B850
+# x8/x8 on root ports) made collectives legal on this host; radiance's TP
+# wire is libr4d's own, not RCCL.
+FRANK_PRIMARY_TP=2
+# Lossy WHT 6-bit wire for large all-reduces (publisher's production choice:
+# +5.6-6.3% prefill, ~-1.3% 8-stream decode at TP2). min-kb stays the
+# engine default.
+FRANK_PRIMARY_TP_WIRE=wht6
+# Decode slots. 2 is Frank's first-boot value ("use parallelism = 2 to
+# start"); raise only after a real-traffic look at slot occupancy and the
+# KV pool's headroom line. The recurrent-state cost scales with it (54
+# MiB/copy/card, 2 copies with speculation).
+FRANK_PRIMARY_MAX_SEQS=3
+# Served window. Native is 262,144; 200,000 is the publisher's production
+# value (compose) and what models.yml's contextWindow must quote — a longer
+# prompt is refused by the engine, never truncated.
+FRANK_PRIMARY_MAX_MODEL_LEN=200000
+FRANK_PRIMARY_PLACEMENT=expert_tiered
+# Pinned host pool for the expert rows that do not fit VRAM. Native FP8
+# experts are 112.8 GiB total (56.4 GiB per rank slice) vs the 4-bit
+# publisher shape's ~29 GiB — this pool is the cost of native precision.
+# The boot ledger prints the pool's coverage ("the pool backs N% of the
+# worst case ... M full-length sessions"); resize from that line, not from
+# arithmetic.
+FRANK_PRIMARY_HOST_POOL_MIB=40960
+# VRAM left unclaimed per card (publisher default 96 MiB).
+FRANK_PRIMARY_GPU_HEADROOM_MIB=96
+# VRAM split of claimed memory per card, MiB (0 = derive from the card).
+# Engine 1.2.0 REMOVED --expert-vs-cache-ratio; these two knobs replace
+# it. The values transcribe the live 1.0.13 boot ledger at ratio 0.82:
+# vram_weights 25.46 GiB, vram_kv 3.46 GiB per card -> 26071 / 3543.
+# Bracket arms scale the KV term linearly in (1-r) from the 0.82 anchor
+# with weights fixed: ratio 0.90 -> KV 1968, ratio 0.95 -> KV 984. The
+# boot ledger arbitrates; re-derive after any payload change.
+FRANK_PRIMARY_VRAM_WEIGHTS_MIB=26071
+FRANK_PRIMARY_VRAM_KV_MIB=3543
+# KV cache in FP8 (7.32 KiB/token at this model's 12 full-attention layers,
+# 2 KV heads, 256 head_dim, K+V).
 FRANK_PRIMARY_KV_DTYPE=fp8
-FRANK_PRIMARY_MNBT=2048
-FRANK_PRIMARY_GATHER_CHUNKS=32
-FRANK_PRIMARY_GATHER_LANES=64
-# PREFIX CACHING: ON. An OFF experiment ran 2026-10-05 (below) and FAILED
-# to stop the corruption, so the historical flag is back.
-# MEASURED 2026-10-04, 18:46:07 PDT (5.6 h into the live boot, one running
-# request, prefix hit rate 48.1%): 48 gdn_hip state faults in one 0.6 s
-# burst, then the process-lifetime silence cap:
-#   [gdn_hip] ERROR gdn_decode_r: state out of ssm-cache range: max|S|=0
-#   with non-finite values (slot=42 head=3 limit=3.40282e+38) fault #1-23
-#   [gdn_hip] ERROR gdn_decode_r: ... (slot=42 head=8 ...) fault #24-46
-#   [gdn_hip] ERROR: state fault report limit (32) reached; further faults
-#   are counted silently
-# One physical SSM slot, two heads, BOTH ranks (heads 3 and 8), 23 layers
-# each. The guard's own message in this build: "known Qwen3.8-Flash-Next
-# (QFN) flaw under prefix caching and is being actively worked; output
-# quality for this request may degrade. Mitigation:
-# --no-enable-prefix-caching".
-# MECHANISM: with prefix caching on, this fork puts QFN on
-# mamba_cache_mode 'align' (the default-for-this-arch branch in
-# config/vllm.py; 'all' is deprecated and 'none' is the no-prefix-cache
-# mode, so align is the only sound mode WITH the flag). Align mode
-# checkpoints each sequence's GDN (Gated DeltaNet) recurrent state into a
-# mamba page every block, where block = 3136 tokens (the attention block
-# size is raised to 3136 so the page sizes match exactly — the boot log's
-# "Padding mamba page size by 0.13% ..."), and RE-SEEDS the state from that
-# page whenever a request resumes from a cached prefix. The corrupt state
-# landed at the 13th boundary: 13 x 3136 = 40,768 tokens — the "~40k" mark
-# where the request's output turned repetitive/corrupted. The NaN stayed
-# in the mamba page, so every later turn of that conversation re-seeded the
-# same non-finite state for those two heads (silent: the fault-report cap
-# was already spent), while fresh short sessions looked fine.
-# RAM WAS "RULED OUT" THE SAME DAY — AND THAT WAS WRONG (RESOLVED below).
-# What the evidence actually supported: zero MCE/EDAC records is the
-# EXPECTED silence of consumer non-ECC memory (a silent flip raises no
-# machine check); an 8 GiB pattern check over 60 s covers ~0.5 % of
-# 246 GiB — no verdict on a speed setting; "4800 is the JEDEC floor on
-# 5600-rated DIMMs" is a rating, not a stability proof at four-DIMM
-# loading. Only the two-head NaN signature argument was load-bearing,
-# and the revert test outvoted it.
-# COST ACCEPTED: no prefix reuse, so each turn re-prefills the whole
-# history instead of at most 3135 tail tokens (a full 40k context costs
-# tens of seconds of TTFT per turn at this shape's prefill rate). A silently
-# corrupted conversation is worse than a slower first token.
-# RESULT (2026-10-05, later that day): THE MITIGATION DID NOT WORK. With
-# --no-enable-prefix-caching deployed and mechanically verified (zero
-# gdn_hip faults, a 40,771-token single-request boundary probe coherent),
-# live long conversations STILL degraded at ~40k on the running endpoint
-# (Frank, direct). The container serving that window logged ZERO faults
-# while the corruption reproduced, so the align-mode page re-seed is not
-# the (only) mechanism — see the runbook's RESULT note for the open
-# hypotheses (re-fed poisoned transcript vs a fault-free long-context
-# path such as fp8 KV at depth). Cost bought nothing: caching is back ON.
-# The knob stays commented for a future fork build with a fixed GDN state
-# path: set 1, redeploy, re-verify past 40,768 tokens (one conversation
-# coherent AND zero gdn_hip lines).
-# RESOLVED 2026-10-05 EVENING (Frank): IT WAS THE RAM. 4800 MT/s (in
-# effect since the 2026-10-04 17:17 boot) was marginal; 4600 MT/s (since
-# the 2026-10-05 12:21 boot) and the endpoint is clean — zero gdn_hip
-# ERROR lines, zero MCE, and the live agent session on it past 140k
-# context with no degradation. Both failure eras sit under the RAM: the
-# 2026-10-04 NaN burst (corrupted weight/activation traffic poisoning a
-# GDN state write — the guard-visible path) and the fault-free ~40k
-# degradation (silent flips in KV/expert data, which no guard covers on
-# non-ECC). CONSEQUENCE FOR THE KNOB: the OFF experiment was CONFOUNDED
-# — both arms ran on the marginal speed — so it neither proved nor
-# disproved the fork's align-mode flaw. Prefix caching stays ON (zero
-# gdn_hip lines under heavy re-send traffic since the speed drop); flip
-# the knob only if a fault recurs on stable RAM or the fork ships a fix.
-# FRANK_PRIMARY_NO_PREFIX_CACHING=1
-# UTIL is an ARENA lever: the fork plans `budget = device_total x util`, and at
-# 31.86 GiB/card every +0.01 is +0.32 GiB of expert cache (~+139 slots), and
-# 0.982 is Frank's standing law: NEVER 1.00. The 1.00 attempt (2026-10-03,
-# shape G v1) died in worker init BEFORE loading a byte of weights — the fork
-# refuses budget >= device total with
-#   ValueError: Free memory on device cuda:N (31.29/31.86 GiB ...)
-# on BOTH ranks at 09:53 and 09:54, NRestarts=2 — so 1.0 is not even a runtime
-# knife-edge here, it is unbootable. The two historical ceiling deaths (the
-# 2x262144 bf16 boot at 0.13 GiB free; the 5x262144 boot at "0 bytes free" on a
-# real 995-token request) remain the reason the margin exists. Rollback note
-# for the other direction: if a request ever dies with
-# `Tried to allocate ... 0 bytes free`, the first knob back is THIS number
-# down a notch, not the window.
-FRANK_PRIMARY_UTIL=0.982
-# CUDA-graph plan term, priced at the measured capture. expert_plan.py charges
-# estimate_cudagraph_memory() = BASE_MB + PER_GRAPH_MB x n_graphs (defaults
-# 256/52 = 0.55 GiB for our 6-graph plan), and the estimator's own docstring
-# says to tune these against the post-capture line. MEASURED twice on this
-# shape: shape E reported "Graph capturing finished ... took 0.31 GiB" (TP0) and
-# 0.17 GiB (TP1); shape F, with the 0.32 GiB we then charged, reported
-# "took -0.06 GiB" — a NEGATIVE delta, i.e. capture fits inside memory the
-# profiler had already counted. So the base pool (256 MiB) is the over-charge:
-# charge 0 base + 12 MiB/graph = 0.07 GiB for the shipped no-spec 6-graph plan,
-# which returns 0.25 GiB more to the arena while still paying per graph, so a
-# future spec-decode shape (18 graphs) is charged 0.21 GiB rather than zero.
-# Verify every deploy against the "Estimated CUDA graph memory" vs "Graph
-# capturing finished ... took X GiB" lines; if capture ever goes positive and
-# exceeds the charge, raise FRANK_PRIMARY_CUDAGRAPH_PER_GRAPH_MB back toward 52.
-# A THIRD env var exists and is the only way to state the graph term outright:
-# VLLM_CUDAGRAPH_MEM_ESTIMATE_MB pins the WHOLE term to an exact MiB total in
-# one knob — estimate_cudagraph_memory() checks it FIRST and returns base/
-# per-graph untouched when set (verified in the shipped image,
-# v1/worker/gpu/cudagraph_mem.py). We ship the BASE/PER_GRAPH pair instead so
-# a future spec-decode shape re-prices automatically; switch to ESTIMATE_MB
-# only if the drift check below ever shows the pair cannot track capture.
-FRANK_PRIMARY_CUDAGRAPH_BASE_MB=0
-FRANK_PRIMARY_CUDAGRAPH_PER_GRAPH_MB=12
-
-# HOST-region cap — the last arena-plan term that still autofit per boot. With
-# no cap the planner's host bound is MemAvailable AT BOOT minus PLE 47.68 -
-# 2 x 7.0 process - 5.0 api/engine - 5% margin (measured boot 2026-10-05:
-# MemAvailable 237.97 -> 159.39 GiB usable across 2 ranks; the demand is 105.48
-# = 52.74/rank x 2 ranks, 23,040 swap-target experts). The same host reads 75.4
-# GiB available while running — the bound tracks whatever the co-tenants hold at
-# boot, and a boot where that drops near the demand silently pins next-hottest
-# experts VRAM-only instead (smaller swap region; the pinned_extra warning is
-# the only trace). Pin 120 GiB: measured demand + 14% headroom, on a host with
-# 246.1 GiB total. If a boot ever logs "expert-offload-mem … exceeds
-# MemAvailable at plan time", THAT is the contract being broken — look at the
-# co-tenant RAM before touching this number. Plan semantics verified in the
-# shipped image (offloader/expert_plan.py): the cap binds as the bound, an
-# exceedance only warns.
-FRANK_PRIMARY_EXPERT_OFFLOAD_MEM=120
-
-# --- optional engine knobs (unset == the flag is absent) ---------------------
-# FRANK_PRIMARY_SPEC holds the WHOLE --speculative-config JSON; setting it adds
-# the flag. THE RETIREMENT VERDICT WAS INVALID: c1248d4 (2026-10-02) retired MTP
-# by comparing fp8+MTP4 (37.4-40.6 tok/s, 54% acceptance, 09-29) against the
-# bf16 no-spec boot (36.4-39.4) — and bf16 then proved unbootable, so fp8 came
-# back without MTP and fp8+spec was NEVER compared against fp8+no-spec on the
-# same shape. The no-spec incumbent measures ~30 tok/s single-stream (below the
-# fleet's 35 tok/s target); every vendor flash-next recipe ships MTP3; and our
-# MTP4 run used the fork's default GREEDY draft sampling, while the recipes pass
-# draft_sample_method=probabilistic (greedy drafts are rejected more often under
-# temp>0 — the plausible cause of the 54%). MTP also re-adds drafter workspace:
-# the old note says the 7.2 GiB/rank KV ceiling WAS the MTP4 workspace, and
-# --expert-precise-memory spends that same budget, so an MTP arm must be tested
-# with precise memory BOTH on and off.
-#   MTP3:  '{"method":"mtp","num_speculative_tokens":3,"draft_sample_method":"probabilistic"}'
-FRANK_PRIMARY_SPEC=
-# FRANK_PRIMARY_COMPCONFIG holds the whole --compilation-config JSON. The
-# capture-ladder law: the piecewise cudagraph ladder must contain
-# num_speculative_tokens + 1 (MTP3 -> 4 -> ladder 4,8,12; MTP4 -> 5,10,15;
-# MTP5 -> 6,12,18,24), because every draft step proposes n+1 tokens. The default
-# ladder is multiples of 1+4 up to 512 then powers of two, so an MTP3 shape can
-# miss its own step size. Pinning the ladder is ALSO the candidate fix for the
-# SEQS>3 wall: query_start_loc is specialised to SEQS+1 and that constant must
-# land on a captured graph size (3 is not in the default ladder, 4 is —
-# measured 2026-10-02, and the vendor runs SEQS 16 on this model/TP2).
-# combo_kernels false/true are BOTH in the shipped recipes, so it is not a law.
-#   MTP3 at SEQS 3:  '{"cudagraph_mode":"PIECEWISE","cudagraph_capture_sizes":[1,4,8,12],"max_cudagraph_capture_size":12}'
-FRANK_PRIMARY_COMPCONFIG=
-# FRANK_PRIMARY_GEN_CONFIG holds the whole --override-generation-config JSON.
-# The payload's HF generation_config.json sets temperature 1.0 / top_p 0.95 /
-# top_k 20 and the engine WARNS about it on every boot, so a client that sends
-# no sampling fields gets creativity it never asked for. This is the one knob
-# that changes ANSWERS, not speed.
-#   '{"max_tokens":65536,"temperature":0.7,"top_p":0.95,"top_k":20}'
-FRANK_PRIMARY_GEN_CONFIG=
-
-# --- client contract: reasoning split, tools, preserve_thinking ----------------
-# Without --reasoning-parser the engine leaves the thinking text INLINE in
-# `content` with a dangling `</think>` and a null `reasoning` field (measured
-# 2026-10-01: content = 'We need answer...Brief.\n\n391') — the
-# malformed thinking block clients choke on. The qwen3 parser splits it
-# properly. The GLM rollback (d91379d) had dropped all three of these; the
-# working values are the ones ecaad36 shipped for this model.
-FRANK_PRIMARY_REASONING_PARSER=qwen3
-# Tools: --enable-auto-tool-choice + the qwen3_xml parser is what makes
-# tool_calls work here (without it, tool_choice auto returns 400).
-FRANK_PRIMARY_TOOL_PARSER=qwen3_xml
-# preserve_thinking keeps prior thinking blocks in the rendered prompt across
-# turns; enable_thinking makes thinking the server-side default. Verified the
-# payload's tokenizer_config template consumes both keys.
-FRANK_PRIMARY_CHAT_TEMPLATE_KWARGS='{"enable_thinking": true, "preserve_thinking": true}'
+# Spec decode: the checkpoint's own MTP head at the publisher's depth-3
+# operating point (probe() returns 3 when mtp_num_hidden_layers > 0; the
+# flag states it so the ledger records the intent).
+FRANK_PRIMARY_SPEC_TOKENS=2
+FRANK_PRIMARY_MAX_BATCHED_TOKENS=2048
+# Prefix cache tiers (publisher pattern; disk tier sized for 2 slots).
+FRANK_PRIMARY_PREFIX_HOST_MIB=4096
+FRANK_PRIMARY_PREFIX_DISK_MIB=65536
+# Frank's pin law: the PLE n-gram table (47.7 GiB) serves from pinned host
+# RAM the cards read. `ram` REFUSES the start if the memlock limit is too
+# small — the unit's LimitMEMLOCK=infinity is the other half of this law.
+FRANK_PRIMARY_NGRAM_PLACEMENT=ram
+# TTM preflight: the pinned ledger (2 x 40 GiB pools + 47.7 GiB PLE + 4 GiB
+# prefix host) is ~132 GiB. ttm's pages_limit/pages_max cap the GPU-mappable
+# pinned pages; the current cap (32,252,917 pages = ~123 GiB) is BELOW the
+# plan, so the serve script refuses to start until both are raised. This is
+# the floor it enforces (in MiB), with ~8 GiB of margin.
+FRANK_PRIMARY_TTM_MIN_MIB=140000
+# Escape hatch for boot-ladder experiments. Unset -> no extra argv. Keep
+# multi-flag values as one string; the serve script expands it unquoted on
+# purpose.
+FRANK_PRIMARY_EXTRA_ARGS=
+# The .rad's minimum size in MiB (deployer gate; a truncated convert is a
+# guaranteed long-timeout boot instead of a fast failure). The converted
+# container measured 177,109 MiB (1,857,127,997,440 B = 173.0 GiB; the
+# plan-only estimate 172.76 GiB covers the data planes but not the metadata
+# tail) - floor sits just under the payload.
+FRANK_PRIMARY_RAD_MIN_MIB=110000

scripts/frank-primary.service

@@ -1,29 +1,25 @@
-# frank-primary.service — tcclaviger RDNA4 vLLM endpoint (orcarouter
-# Qwen3.8-Flash-Next-Uncensored-FP8, fp8 KV, NO spec decode, 3 slots x 200704
-# context with the pool sized to ~2 full-length conversations, TP=2, tiered
-# experts, 131-shard payload) on the 2x R9700 pair, dankserver :7770.
-# 2026-09-29 cutover from the R9V radiance fork (git tag
-# frank-primary-r9v-q8-20260929 is the pre-cutover state; its
-# launcher/shim/setup-state machinery is retired and the state dirs stay on
-# disk as rollback).
+# frank-primary.service — radiance (StillDeadcode, Apache-2.0) endpoint
+# (OFFICIAL Qwen/Qwen3.8-Flash-Next-FP8 at NATIVE block-FP8 precision, the
+# rad-convert container qwen4exp-fp8-native.rad) on BOTH R9700s (TP=2,
+# wht6 wire), dankserver :7770. 2026-10-05 cutover from FreeToken (git tag
+# frank-primary-freetoken-pr587-20261005 is the pre-cutover state; the
+# retired unit/env/serve trio and Dockerfile are in that tag).
 #
-# Same port, same served alias; the harness models.yml id is unchanged.
-# The serve script drives `podman run` directly (the image entrypoint execs
-# `vllm serve`) and then waits on the container. NOTE: that wait can return
-# 0 even when the engine died mid-request (vLLM's API server drains and
-# exits clean on a worker fault), which is why the restart policy is
-# `always` — see the [Service] note.
-#
-# WHY A ROOT UNIT (not a rootless quadlet): the tiered-expert shape pins tens
-# of GiB of host RAM (the expert host slabs, ~20.5 GiB per rank of pinned
-# anonymous memory). Pinned pages are charged against RLIMIT_MEMLOCK; dank's
-# login session on this host carries 8 MiB and a user quadlet is capped by the
+# Same port, same served alias, same container identity as the unit name;
+# the harness models.yml id is unchanged. The serve script drives
+# `podman run` directly: the digest-pinned stilldeadcode/radiance image
+# (287 MiB, kernels prebuilt for gfx1201) has the engine as its
+# entrypoint. The image lives in dank's ROOTLESS podman storage.
+# WHY A ROOT UNIT (not a rootless quadlet): radiance pins ~132 GiB of HOST
+# RAM (2 x 40 GiB expert pools + the 47.7 GiB PLE ngram table + the prefix
+# host tier). Pinned pages are charged against RLIMIT_MEMLOCK; dank's login
+# session on this host carries 8 MiB and a user quadlet is capped by the
 # user manager's own hard limit — raising that means restarting
-# user@1000.service and bouncing every user container on the host. A root unit
-# grants LimitMEMLOCK=infinity to the dank process via PID1, and the image
-# stays in dank's rootless podman storage.
+# user@1000.service and bouncing every user container on the host. A root
+# unit grants LimitMEMLOCK=infinity to the dank process via PID1; the
+# engine still runs out of dank's rootless podman storage.
 [Unit]
-Description=frank-primary — tcclaviger vLLM Qwen3.8-Flash-Next-Uncensored-FP8 on 2x R9700
+Description=frank-primary — radiance Qwen3.8-Flash-Next-FP8 (native) on 2x R9700 TP=2
 After=network-online.target
 Wants=network-online.target
 # Never let the start-rate limiter give up after a crash loop (a failed LLM
@@ -33,34 +29,35 @@ StartLimitIntervalSec=0
 [Service]
 User=dank
 Group=dank
-# The tiered-expert offloader pins host RAM; an RLIMIT_MEMLOCK smaller than
-# the pinned set fails the load mid-start. PID1 grants infinity here —
-# parity with the fleet's other LLM units.
+# The engine pins ~132 GiB of host RAM (expert pools + PLE table); an
+# RLIMIT_MEMLOCK smaller than the pinned set fails the load mid-start.
+# PID1 grants infinity here — parity with the fleet's other LLM units.
 LimitMEMLOCK=infinity
 Environment="FRANK_PRIMARY_ENV_FILE=/home/dank/serverAppConfigs/scripts/frank-primary.env"
 ExecStart=/home/dank/serverAppConfigs/scripts/serve-frank-primary.sh
 # systemd stop: wait alone cannot stop the container child; stop it
-# explicitly. vLLM 0.26 drains gracefully on SIGTERM.
-ExecStop=/usr/bin/podman stop -t 120 frank-primary-qwen
-# Boot cost: FP8 payload page-in plus torch compile/graph capture —
-# ~15 min cold (measured 2026-09-29), faster warm.
-TimeoutStartSec=1800
-# Restart=always, NOT on-failure (measured 2026-10-01): a runtime
-# torch.OutOfMemoryError kills the engine while vLLM's API server still
-# exits 0, so ExecStart returned clean, systemd logged "Deactivated
-# successfully", and on-failure NEVER fired — the endpoint stayed down until
-# a human noticed. `always` restarts on any container death; an explicit
-# `systemctl stop` still wins (systemd does not restart a unit the operator
-# stopped), so bench windows and deploys are unaffected.
+# explicitly. No graceful drain contract is claimed (the OpenAI front end
+# holds no queue to drain — in-flight decodes are lost, which a stop
+# during a bench window accepts; radiance takes SIGTERM on the engine
+# process and podman reaps the container).
+ExecStop=/usr/bin/podman stop -t 120 frank-primary
+# Boot cost: .rad load (weights + PLE upload) + expert host-pool pin. The
+# publisher's compose healthcheck allows a 15-minute start_period; the
+# vLLM-era budget (1800 s) carried forward.
+# Restart=always, NOT on-failure (the 2026-10-01 vLLM lesson carries over:
+# a clean API-server exit after a worker fault reads as success to
+# systemd). `always` restarts on any container death; an explicit
+# `systemctl stop` still wins, so bench windows and deploys are unaffected.
 Restart=always
 RestartSec=10
-# ExecStop allows the engine 120 s to drain (`podman stop -t 120`); without an
-# explicit TimeoutStopSec systemd's 90 s default cuts the stop short, kills it,
-# and records 'Failed with result timeout' (measured 2026-10-01 13:27:17).
-# Must exceed the podman grace with margin for the engine's own shutdown.
+# ExecStop allows the engine 120 s to drain (`podman stop -t 120`); without
+# an explicit TimeoutStopSec systemd's 90 s default cuts the stop short,
+# kills it, and records 'Failed with result timeout' (measured 2026-10-01
+# 13:27:17). Must exceed the podman grace with margin.
 TimeoutStopSec=180
-# The engine pins tens of GiB of host RAM (expert host slabs); it is the
-# last process on this host that should be chosen by the kernel OOM killer.
+# The engine pins tens of GiB of host RAM (expert rows + PLE table); it is
+# the last process on this host that should be chosen by the kernel OOM
+# killer.
 OOMScoreAdjust=-500
 
 [Install]

scripts/frank-secondary.env

@@ -0,0 +1,62 @@
+# frank-secondary — llama.cpp (master, CUDA sm_86) serving
+# HauhauCS/Qwen3.5-9B-Uncensored-HauhauCS-Aggressive at Q6_K on the RTX 3090,
+# :7771.
+#
+# 2026-10-07 (Frank): swap from Gemma4-26B-A4B QAT Q4_K_M (one 200,704 slot)
+# to Qwen3.5-9B Q6_K at TWO 200,704-token conversations, EXACT (f16) KV,
+# MTP disabled (SPEC_TYPE=none), log verbosity 4.
+#
+# Architecture — read from the GGUF's own header (HTTP range read, not the
+# model card): arch qwen35, 32 layers, dense 9B, full_attention_interval 4
+# ⇒ 8 full-attention layers (3,7,11,...,31) + 24 Gated DeltaNet linear layers,
+# 16 attention heads / 4 KV heads × 256 dim, ssm conv_kernel 4 / state_size
+# 128 / group_count 16 / time_step_rank 32 / inner_size 4096, native 262,144,
+# vocab 248,320, file_type 18 (Q6_K). Tensor-type histogram: Q6_K×250 +
+# F32×177 — standard types only, so the stock master build loads it (no
+# custom-quant lock-in like the ROCmFPX era). NO nextn/mtp tensors exist, so
+# in-file MTP is impossible here; SPEC_TYPE=none is forced by the payload.
+#
+# KV rate law: 8 full layers × 4 KV heads × 256 dim = 8,192 elements per
+# token per cache. f16 K + f16 V = 32,768 B/token = 32 KiB/token, so the
+# 401,408-token pool is 12.25 GiB. The 24 DeltaNet layers cost a fixed
+# recurrent state per SLOT (not per token) — that is why two 200k slots fit
+# where the 26B MoE fit only one.
+#
+# Feasibility comes from the predecessor's MEASURED boots, not a formula:
+# MiMo-V2.6-Distill-Qwen-9B (identical qwen35 geometry, 7,793,710,624 B) at
+# 2×200,704 f16 K + f16 V measured 21,276 used / 2,849 free at ubatch 1024
+# and 21,846 / 2,279 at ubatch 2048. This file is 415 MiB lighter in weights,
+# its BF16 mmproj is 3 MiB heavier ⇒ projected ~21,430 used / ~2,700 free.
+# The verbosity-4 boot ledger is the referee.
+#
+# Ubatch 2048 is the measured CUDA optimum (measured on the MiMo predecessor
+# at this exact arch + shape: +4.6 % prefill at 8.9k depth, decode flat, cost
+# 616 MiB). 1024 is the fallback rung if the boot lands under the 2 GiB floor.
+LLAMA_BIN=/home/dank/GitPackages/llama-master/build-cuda/bin/llama-server
+MODEL_FILE=/home/dank/models/Qwen35-9B-HauhauCS-Q6K/Qwen3.5-9B-Uncensored-HauhauCS-Aggressive-Q6_K.gguf
+MMPROJ_FILE=/home/dank/models/Qwen35-9B-HauhauCS-Q6K/mmproj-Qwen3.5-9B-Uncensored-HauhauCS-Aggressive-BF16.gguf
+# Served model id — the harness/family name (Protoss rename completed to the
+# wire 2026-10-09: harness ids + homepage cards 2026-10-08, /v1/models aliases
+# this day). The SERVICE, unit, scripts and this env file keep the
+# frank-secondary name; only the alias on the wire changed. Consumers that
+# key on the served id: OpenWebUI (model row, access_grant, api_configs
+# model_ids) and the harness models.yml.
+ALIAS=frank-zealot
+BIND_ADDR=0.0.0.0
+LISTEN_PORT=7771
+# CTX_SIZE is the POOL llama.cpp divides across slots: 401,408 / 2 = 200,704
+# per conversation. MUST stay divisible by PARALLEL (the seam enforces it) and
+# the harness contextWindow quotes the PER-SLOT number — a longer prompt is
+# answered with a 400, never truncated.
+CTX_SIZE=401408
+PARALLEL=2
+BATCH_SIZE=4096
+UBATCH_SIZE=2048
+CACHE_TYPE_K=f16
+CACHE_TYPE_V=f16
+SPEC_TYPE=none
+NGL=99
+# Log verbosity threshold (llama.cpp -lv). 4 = debug: KV layout, buffer sizes
+# and graph capture land in the journal, which is what the VRAM-fit forensics
+# read. Levels above 4 flood the journal.
+VERBOSITY=4

scripts/frank-secondary.service

@@ -0,0 +1,19 @@
+[Unit]
+Description=llama.cpp Qwen3.5-9B-Uncensored-HauhauCS-Aggressive Q6_K GGUF (qwen35 hybrid, 2x200704 slots, f16 K+V, MTP off, mmproj vision) on the RTX 3090 (dankclaw, frank-secondary)
+After=network-online.target
+Wants=network-online.target
+
+[Service]
+Type=simple
+User=dank
+Group=dank
+# CUDA pins its staging buffers, so the process needs the memlock budget even
+# though the weights themselves are copied into VRAM.
+LimitMEMLOCK=infinity
+Environment="FRANK_SECONDARY_ENV_FILE=/home/dank/serverAppConfigs/scripts/frank-secondary.env"
+ExecStart=/home/dank/serverAppConfigs/scripts/serve-frank-secondary.sh
+Restart=on-failure
+RestartSec=10
+
+[Install]
+WantedBy=multi-user.target

scripts/frank-small.env

@@ -1,136 +0,0 @@
-# frank-small — llama.cpp (master, CUDA sm_86) serving
-# ornith-ai/Ornith-1.5-9B-GGUF:Q8_0 on the RTX 3090, :7771.
-#
-# 2026-10-05 (Frank): the payload moved off bartowski/MiMo-V2.6 (Q6_K) to
-# ornith-ai/Ornith-1.5-9B Q8_0 (9,786,060,384 B, LFS oid 22086870...).
-# Header-verified facts that re-price the shape: qwen35 hybrid like MiMo, but
-# 33 blocks (not 32) with full_attention_interval 4 => NINE full-attention
-# layers (MiMo had 8), 16 Q heads / 4 KV heads, k/v length 256, native window
-# 262144, dense (no expert keys), nextn_predict_layers=1 (an MTP head is IN
-# the file — capability only; no spec decode ships, same law as before).
-#
-# THE KV CONTRACT STAYS f16 K + f16 V — NO KV QUANTIZATION (Frank, 2026-10-05,
-# overruling a written q8_0-V proposal). The arithmetic that makes this hard:
-#   KV rate 9 attn layers x 4 heads x 256 x 2 B = 18 KiB/token K, same again
-#   for V at f16 => 36 KiB/token (MiMo was ~24.5 measured: one fewer
-#   full-attn layer AND one fewer KV head).
-#   Weights +1,900 MiB (9,334 vs Q6_K's 7,432). Measured free at the Q6_K
-#   4096/2048 shape was 2,279 MiB; Q8_0 at f16/f16 does not fit that shape —
-#   the deficit is made up the way Frank ordered: UBATCH FIRST (2048 -> 1024,
-#   the measured 616 MiB rung), then POOL STEPS, then measure. Guess-and-check
-#   on the live card is the accepted method for llama.cpp sizing here: boot,
-#   read nvidia-smi free, step the pool (12,288-token increments, ~432 MiB
-#   each at the 36 KiB/token rate) until the 2-3 GiB floor holds. Compute
-#   buffers allocate at graph capture during BOOT, so a failure is loud and
-#   fast, not a surprise at the first long prompt.
-#
-# Previous shape history (MiMo era, for the rollback trail): 2026-10-03 the
-# rung went Q5_K_M -> Q6_K with the pool HELD at 401,408; the freed third
-# slot funded exact f16 V first; the ~3.5 GiB still free funded the +875 MiB
-# of weights.
-#
-# 2026-10-03 (Frank): TWO streams instead of three, each one WIDE — 200,704
-# tokens per conversation, above the 200,700 the harness compacts at — and the
-# KV cache is now FULLY UNQUANTIZED (f16 K, f16 V). The two moves are the same
-# move: llama.cpp divides --ctx-size across --parallel slots, so the slot given
-# up is exactly the VRAM that pays for unquantized V. No further quantization
-# anywhere on this endpoint.
-#
-# Three measurements on this card, this runtime, price the terms:
-#   2 x 262,144, no projector: 20,190 MiB used / 4,386 free
-#     KV = 524,288 tok x 24.5 KiB = 12,800 MiB => non-KV (weights 6,557 +
-#     DeltaNet state ~25/slot + compute buffers + CUDA graphs) = 7,390 MiB
-#   3 x 196,608, projector on: 22,972 MiB used / 1,153 free
-#     KV = 14,113 MiB => non-KV WITH the vision tower = 8,859 MiB
-#   3 x 180,224, projector on: 21,684 MiB used / 2,441 free (the 2026-10-02
-#     shape) => non-KV = 8,748 MiB
-# Two slots give back one DeltaNet state (~25 MiB). Priced at f16 K+V
-# (32 KiB/token) this pool is 12,544 MiB, so the prediction is
-# 12,544 + ~8,800 = ~20.9 GiB used, ~3.1 GiB free — the band the 2026-10-02
-# shape hit, bought with precision instead of a slot. The 1024 ubatch
-# re-prices the compute buffers upward, so MEASURE after any change: if the
-# boot lands under 2 GiB free, step the pool down in 12,288-token
-# (384 MiB at this rate) increments.
-#
-# KV arithmetic: attention KV exists only in the 8 full-attention layers
-# (4 KV heads x 256 head_dim x 2 B) = 16 KiB/token for K and the same again for
-# V => 32 KiB/token, so 12,288 tokens (= 6144/slot) is one 384 MiB step — that
-# is the ladder to walk if this moves again. Two native windows (2 x 262,144)
-# are NOT reachable with full-precision V: 524,288 x 32 KiB = 16.1 GiB of KV
-# against a card with ~15.7 GiB of KV room once weights, the vision tower, the
-# DeltaNet states and the graphs are loaded. 200,704 per stream is what full
-# precision buys; q8_0 on V would buy 262,144 instead.
-#
-# The asymmetric-KV contract this endpoint was created for (Frank, 2026-10-01
-# night: f16 K + q8_0 V) is now spent — it existed to make a third slot fit,
-# and the third slot is gone. The flags stay explicit so the choice is visible
-# in the argv either way.
-LLAMA_BIN=/home/dank/GitPackages/llama-master/build-cuda/bin/llama-server
-MODEL_FILE=/home/dank/models/Ornith-Q8/Ornith-1.5-9B-Q8_0.gguf
-
-# Vision: the Ornith Q8_0 GGUF ships no `v.*` tensors either (dense-9B byte
-# arithmetic: 9.79 GB ≈ 9B params x ~1.06 B/param leaves no room for a tower),
-# so the projector file IS the vision capability (mmproj-Ornith-1.5-9B-BF16
-# .gguf, 921,704,672 B, LFS oid 626f9f90..., fetched by
-# scripts/download-frank-small.sh). Empty string = text-only on purpose.
-MMPROJ_FILE=/home/dank/models/Ornith-Q8/mmproj-Ornith-1.5-9B-BF16.gguf
-
-# Fleet endpoint: same port and alias as every frank-small before it, so the
-# harness models.yml id and the homepage probe do not move.
-ALIAS=frank-small
-BIND_ADDR=0.0.0.0
-LISTEN_PORT=7771
-
-# ONE conversation at the NATIVE 262,144 window (Frank, 2026-10-05). The
-# two-slot shape died under Ornith and the NO-QUANTIZE law, by arithmetic:
-# the KV rate is the ARCHITECTURE's, not the quant's — 9 full-attn layers x
-# 4 KV heads x 256 dim x 2 B = 36 KiB/token at f16 K+V (MiMo measured 24.5),
-# so 2 x 200,704 wants ~14.1 GiB of KV on top of +1,900 MiB of Q8 weights
-# against 2,279 MiB measured free; ubatch 1024 claws back 616, and no
-# pool step fits BOTH the 2-3 GiB floor and the harness compaction
-# invariant (slot >= 191,148). One native slot fits with room; the window
-# per conversation RISES 200,704 -> 262,144. The cost: a second concurrent
-# request queues behind the released slot (subagent concurrency 2 -> 1).
-CTX_SIZE=262144
-PARALLEL=1
-
-# Compute-buffer geometry. This endpoint passed neither flag until 2026-10-03,
-# so it ran on llama.cpp's own defaults (2048/512); the pair is now 4096/2048.
-# Both are tunables because the pair is a measurement, not a law — and the
-# measurement is CARD-SPECIFIC. On the 2x R9700 layer-split sibling the ubatch
-# curve peaks at 512 and every step up costs GiB of compute buffer for no
-# speed (see frank-medium-batch-ubatch). On THIS CUDA 3090 the curve climbs
-# monotonically to 2048: 710 -> 1188 t/s prefill from ubatch 1024 to 2048 on
-# the 35B MoE sweep, 1944 -> 2423 t/s on the Ornith 8k sweep, with decode flat
-# to under 1 t/s on every rung. Ubatch is a prefill knob; decode never moves.
-#
-# The logical batch is free — two independent sweeps agree that at a fixed
-# ubatch, 2048/4096/8192 give identical prefill speed AND identical peak VRAM,
-# because the compute buffers are sized by the ubatch alone. Doubling it costs
-# nothing and buys nothing; it is here because it is the value the pair was
-# measured at.
-#
-# THE RISK IS ALL IN THE UBATCH. The 1024 -> 2048 step cost ~1,244 MiB of peak
-# graph VRAM on the 35B MoE (7,330 -> 8,574 MiB); this 9B hybrid has far
-# smaller compute tensors, so the delta should be well under that, but the
-# predecessor Q5_K_M shape recorded "ubatch 2048 failed the scratch boot" at
-# ~3,571 MiB free. The Q8_0 rung ships UBATCH 1024 (Frank, 2026-10-05: lower
-# ubatch FIRST — the measured 616 MiB step is the first VRAM the Q8 weights
-# claw back, before any pool move). Compute buffers are allocated at graph
-# capture during BOOT, so a failure is loud and fast, not a surprise at the
-# first long prompt. MEASURE nvidia-smi free immediately after the deploy:
-# under ~1.5 GiB, or a boot failure, step the POOL down next (12,288-token
-# increments, ~432 MiB each at the Ornith 36 KiB/token f16 KV rate).
-BATCH_SIZE=4096
-UBATCH_SIZE=1024
-
-# The KV contract (2026-10-05, Frank): f16 K + f16 V — NO KV quantization.
-# The q8_0-V arithmetic was written and OVERRULED. The Q8 weights eat the
-# VRAM that funded f16 V on 2026-10-03, so the remedy ladder is UBATCH FIRST
-# (2048 -> 1024, the measured 616 MiB rung), then POOL STEPS — and
-# guess-and-check on the live card is the accepted method: boot, read
-# nvidia-smi, step down until the floor holds. --flash-attn stays pinned on
-# in the serve script: it is the fast path for f16 KV.
-CACHE_TYPE_K=f16
-CACHE_TYPE_V=f16
-NGL=99

scripts/frank-small.service

@@ -1,19 +0,0 @@
-[Unit]
-Description=llama.cpp MiMo-V2.6-Distill-Qwen-9B Q6_K GGUF (qwen35 hybrid, 2x200704, unquantized f16 KV, mmproj vision) on the RTX 3090 (dankclaw, frank-small)
-After=network-online.target
-Wants=network-online.target
-
-[Service]
-Type=simple
-User=dank
-Group=dank
-# CUDA pins its staging buffers, so the process needs the memlock budget even
-# though the weights themselves are copied into VRAM.
-LimitMEMLOCK=infinity
-Environment="FRANK_SMALL_ENV_FILE=/home/dank/serverAppConfigs/scripts/frank-small.env"
-ExecStart=/home/dank/serverAppConfigs/scripts/serve-frank-small.sh
-Restart=on-failure
-RestartSec=10
-
-[Install]
-WantedBy=multi-user.target

scripts/hfget.sh

@@ -0,0 +1,107 @@
+#!/bin/sh
+# hfget.sh -- fetch a Hugging Face repo with curl and jq, and nothing else.
+#
+# The engine and its tooling do not depend on Python, so `huggingface-cli` is not assumed. The API
+# gives a file listing as JSON and every file has a stable resolve/ URL, which is all a downloader
+# needs.
+#
+#   scripts/hfget.sh <repo-id> [dest-dir] [revision]
+#
+# Set HF_TOKEN for a gated or private repo. Re-running resumes: a file whose local size already
+# matches the listing is skipped and a short one is continued with `curl -C -`, so an interrupted
+# transfer of a 300 GB checkpoint costs only what it had not finished.
+#
+# WEIGHT DUPLICATES ARE SKIPPED. Many repos carry the same tensors several times over -- safetensors
+# beside the older pickle .bin, an ONNX export, a GGUF conversion -- and on a large checkpoint each
+# copy is hundreds of gigabytes of the same numbers. When the listing holds any .safetensors, every
+# other weight format is left behind.
+#
+# HFGET_JOBS concurrent transfers, default 4. These are long sequential reads: a few streams
+# saturate the link and more only makes the destination seek.
+set -e
+
+# ---- child mode: one file, named by "<size><tab><path>" in $1, everything else inherited -------
+# sh cannot export a function, so parallelism is one re-entry of this script per file under
+# `xargs -P` rather than a worker pool.
+if [ -n "$HFGET_CHILD" ]; then
+    size=${1%%	*}; path=${1#*	}
+    out=$HFGET_DEST/$path
+    if [ -f "$out" ]; then
+        cur=$(wc -c < "$out")
+        if [ "$cur" = "$size" ]; then echo "  = $path"; exit 0; fi
+        echo "  + $path (resuming at $cur of $size)"
+    else
+        echo "  > $path"
+    fi
+    mkdir -p "$(dirname "$out")"
+    if [ -n "$HF_TOKEN" ]; then
+        curl -sSL --fail --retry 5 --retry-delay 5 --retry-all-errors -C - \
+             -H "Authorization: Bearer $HF_TOKEN" \
+             -o "$out" "https://huggingface.co/$HFGET_REPO/resolve/$HFGET_REV/$path"
+    else
+        curl -sSL --fail --retry 5 --retry-delay 5 --retry-all-errors -C - \
+             -o "$out" "https://huggingface.co/$HFGET_REPO/resolve/$HFGET_REV/$path"
+    fi
+    got=$(wc -c < "$out")
+    [ "$got" = "$size" ] || { echo "hfget: $path is $got bytes, listing says $size" >&2; exit 1; }
+    exit 0
+fi
+
+# ---- parent ------------------------------------------------------------------------------------
+repo=$1
+[ -n "$repo" ] || { echo "usage: hfget.sh <repo-id> [dest-dir] [revision]" >&2; exit 2; }
+dest=${2:-$HOME/models/$repo}
+rev=${3:-main}
+jobs=${HFGET_JOBS:-4}
+command -v jq >/dev/null || { echo "hfget: jq is required" >&2; exit 2; }
+
+echo "hfget: listing $repo@$rev"
+# THE LISTING IS PAGINATED AND THE PAGE SIZE IS SMALL. A large checkpoint runs to a hundred-odd
+# shards and the API returns fifty entries with a `Link: ...; rel="next"` header. Taking only the
+# first page is the worst possible failure here: every file it did fetch is the right length, so
+# the size check passes and the converter is handed a checkpoint missing two thirds of its
+# tensors. Follow the cursor until there is no next link.
+api="https://huggingface.co/api/models/$repo/tree/$rev?recursive=true&expand=true"
+listing='[]'
+page=0
+while [ -n "$api" ]; do
+    hdr=$(mktemp "${TMPDIR:-/tmp}/hfhdr.XXXXXX")
+    if [ -n "$HF_TOKEN" ]; then
+        body=$(curl -sSL --fail-with-body -D "$hdr" -H "Authorization: Bearer $HF_TOKEN" "$api")
+    else
+        body=$(curl -sSL --fail-with-body -D "$hdr" "$api")
+    fi
+    [ "$(printf '%s' "$body" | jq -r 'type')" = "array" ] || {
+        rm -f "$hdr"
+        echo "hfget: $(printf '%s' "$body" | jq -r '.error // .message // .')" >&2
+        echo "hfget: a gated or private repo needs HF_TOKEN in the environment" >&2; exit 1; }
+    listing=$(printf '%s\n%s' "$listing" "$body" | jq -s 'add')
+    api=$(tr -d '\r' < "$hdr" | sed -n 's/^[Ll]ink: *<\([^>]*\)>; *rel="next".*/\1/p' | head -1)
+    rm -f "$hdr"
+    page=$((page + 1))
+    [ "$page" -gt 200 ] && { echo "hfget: listing did not terminate after 200 pages" >&2; exit 1; }
+done
+echo "hfget: $page page(s) of listing"
+have_st=$(printf '%s' "$listing" | jq '[.[] | select(.path | endswith(".safetensors"))] | length')
+sel='.[] | select(.type == "file")'
+[ "$have_st" -gt 0 ] && sel="$sel"' | select((.path | test("\\.(bin|pth|msgpack|h5|onnx|onnx_data|gguf)$")) | not)'
+
+list=$(mktemp "${TMPDIR:-/tmp}/hfget.XXXXXX")
+trap 'rm -f "$list"' EXIT INT TERM
+printf '%s' "$listing" | jq -r "$sel"' | "\(.size)\t\(.path)"' > "$list"
+
+n=$(wc -l < "$list")
+[ "$n" -gt 0 ] || { echo "hfget: the listing has no files" >&2; exit 1; }
+need=$(awk -F'\t' '{s += $1} END {printf "%.1f", s / 1073741824}' "$list")
+mkdir -p "$dest"
+free=$(df -BG --output=avail "$dest" | tail -1 | tr -dc '0-9')
+echo "hfget: $n file(s), $need GiB -> $dest"
+[ "$have_st" -gt 0 ] && echo "hfget: safetensors present, pickle weight formats skipped"
+echo "hfget: $free GiB free on the destination filesystem"
+awk -v n="$need" -v f="$free" 'BEGIN { if (n > f) { print "hfget: THE DOWNLOAD DOES NOT FIT"; exit 1 } }'
+
+export HFGET_CHILD=1 HFGET_DEST="$dest" HFGET_REPO="$repo" HFGET_REV="$rev"
+self=$(cd "$(dirname "$0")" && pwd)/$(basename "$0")
+tr '\n' '\0' < "$list" | xargs -0 -P "$jobs" -I{} sh "$self" "{}" \
+    || { echo "hfget: incomplete -- re-run to resume" >&2; exit 1; }
+echo "hfget: complete -- $(du -sh "$dest" | cut -f1) in $dest"

scripts/kld-gate-swift15.sh

@@ -0,0 +1,91 @@
+#!/usr/bin/env bash
+# kld-gate-swift15.sh — RUNS ON DANKSERVER. Score the quantized candidate
+# against the BF16 reference and apply the pre-approved gate:
+#   mean KL <= 0.03 AND top-1 agreement >= 95%.
+# Same engine (1.0.13) and same boot shape as the reference RECORD
+# (run-swift15-bf16.sh kld): disk-tier, no speculation, deterministic, fp8 KV.
+# --kld-ref reads its tokens from the reference (no --kld-corpus here — the
+# (run-swift15-bf16.sh kld): 1.0.13, tp 2, disk-tier, fp8 KV, no speculation,
+# deterministic, max-model-len 16384, vram-kv 3072, 4 slots. Two deviations
+# are forced by the w4 file: MNBT 2048 (the w4 kernels stream from the file
+# tier ONLY at MNBT<=2048 — at the recording's 32768 the engine demands a
+# widened form: "DID NOT FIT ... stored form is not the container's"), and
+# pool 40960 (the recording's 12 GiB pool predates the w4 tables; at 12288 the
+# gate_up slab cannot be cached). Chunk size is softmax-invariant, so the
+# MNBT delta is float-order noise riding on the KL floor.
+set -uo pipefail
+PIN=sha256:4c3f484564bb2872784e46b0a59cbd71becc9c1e746e301a3596ffae253c322e
+C=swift15gate
+LOG=/home/dank/logs/swift15_kldgate.log
+KLD_REF_DIR=/home/dank/corpora/kld-ref
+KLD_OUT=/home/dank/radiance-state/kld-gate.json
+
+[ -d "$KLD_REF_DIR" ] || { echo "missing reference: $KLD_REF_DIR"; exit 2; }
+: > "$KLD_OUT"
+cp -f /home/dank/serverAppConfigs/scripts/qwen38-froggeric-v22.5.jinja \
+      /home/dank/models/tpl.jinja 2>/dev/null || true
+sudo systemctl stop frank-primary >/dev/null 2>&1 || true
+podman rm -f "$C" >/dev/null 2>&1 || true
+
+podman run --rm --name "$C" \
+    --network host --init \
+    --security-opt seccomp=unconfined \
+    --ulimit memlock=-1:-1 \
+    --device /dev/kfd --device /dev/dri \
+    -v /home/dank/models:/models:ro \
+    -v /home/dank/radiance-state:/data \
+    -v "$KLD_REF_DIR:/data/kld-ref:ro" \
+    "stilldeadcode/radiance@$PIN" \
+    --model /models/rad/qwen4exp-swift15-w4dn8-i8-hc8m.rad \
+    --tp 2 --placement expert_tiered --weights-disk-tier \
+    --host-pool-mib 40960 --deterministic --gpu-headroom-mib 768 \
+    --num-speculative-tokens 0 \
+    --kv-cache-dtype fp8 --max-model-len 16384 \
+    --ngram-placement ram \
+    --override-chat-template /models/tpl.jinja \
+    --kld-ref /data/kld-ref --kld-out /data/kld-gate.json \
+    --vram-kv-mib 3072 \
+    --max-num-seqs 4 --max-num-batched-tokens 2048 \
+    2>&1 | tee "$LOG"
+rc=${PIPESTATUS[0]}
+[ "$rc" -eq 0 ] || { echo "KLD GATE BOOT FAILED rc=$rc; log: $LOG"; exit "$rc"; }
+
+python3 - "$KLD_OUT" "$LOG" <<'PY'
+import json, sys, re
+out, log = sys.argv[1], sys.argv[2]
+def num(pat):
+    m = re.findall(pat, open(log, errors="replace").read(), re.I)
+    return float(m[-1]) if m else None
+kl = top1 = None
+try:
+    d = json.load(open(out))
+    def find(obj, keys):
+        if isinstance(obj, dict):
+            for k, v in obj.items():
+                if any(s in k.lower() for s in keys) and isinstance(v, (int, float)):
+                    return float(v)
+            for v in obj.values():
+                r = find(v, keys)
+                if r is not None: return r
+        return None
+    kl = find(d, ("mean_kl", "mean kl", "kl_mean", "mean"))
+    top1 = find(d, ("top1", "top_1", "top-1", "agreement"))
+    if top1 is not None and top1 <= 1.0:
+        top1 *= 100.0
+except Exception as e:
+    print("report JSON unusable (%s); falling back to log" % e)
+if kl is None:
+    kl = num(r"mean[_ ]kl[^0-9]*([0-9]*\.?[0-9]+(?:e[-+][0-9]+)?)")
+if top1 is None:
+    top1 = num(r"top[-_ ]?1[^0-9]*([0-9]*\.?[0-9]+)")
+if kl is None or top1 is None:
+    print("KLD GATE PARSE FAILED: mean_kl=%s top1=%s — inspect %s / %s" % (kl, top1, out, log))
+    sys.exit(2)
+print("KLD GATE: mean KL = %.4f (max 0.03) | top-1 = %.2f%% (min 95.0)" % (kl, top1))
+ok = (kl <= 0.03) and (top1 >= 95.0)
+print("KLD GATE " + ("PASS" if ok else "FAIL"))
+sys.exit(0 if ok else 1)
+PY
+gate=$?
+echo "kld-gate complete rc=$gate; log: $LOG report: $KLD_OUT"
+exit "$gate"

scripts/kld-then-calib.sh

@@ -0,0 +1,40 @@
+#!/bin/bash
+# kld-then-calib.sh — RUNS ON DANKSERVER. Campaign chain: wait for the KLD
+# record to finish, verify it, then boot the 1.2.0 calibration server and
+# push the 10M-token corpus. One GPU, so the phases serialize; prod is
+# restored by each run script's trap.
+set -u
+KLOG=/home/dank/logs/swift15-kld-driver.log
+CLOG=/home/dank/logs/swift15-calib-driver.log
+S=/home/dank/serverAppConfigs-swift15/scripts
+# Phase 1: KLD record (SKIP_KLD=1 when the record is already complete —
+# kld.json's rows count is the receipt; a mid-run script swap can kill the
+# wrapper with rc=127 after the engine exited clean).
+if [ "${SKIP_KLD:-0}" != 1 ]; then
+    while ! grep -q "exited rc=" "$KLOG" 2>/dev/null; do sleep 60; done
+    RC=$(grep -o "exited rc=[0-9]*" "$KLOG" | tail -1)
+    if [ "$RC" != "exited rc=0" ]; then echo "KLD FAILED ($RC); calib NOT started"; exit 1; fi
+fi
+N=$(find /home/dank/corpora/kld-ref -type f | wc -l)
+SZ=$(du -sm /home/dank/corpora/kld-ref | cut -f1)
+echo "KLD OK files=$N size_mib=$SZ"
+[ "$N" -ge 5 ] && [ "$SZ" -ge 1000 ] || { echo "kld-ref looks thin; calib NOT started"; exit 1; }
+grep -q '"rows": 50235' /home/dank/corpora/kld-ref/kld.json || {
+    echo "kld.json rows != 50235; calib NOT started"; exit 1; }
+
+# Phase 2: calibration serve (1.2.0) + corpus push.
+nohup bash "$S/run-swift15-bf16.sh" calib > "$CLOG" 2>&1 &
+CP=$!
+for _ in $(seq 1 160); do
+    sleep 15
+    curl -sf localhost:7770/health >/dev/null 2>&1 && break
+done
+if ! curl -sf localhost:7770/health >/dev/null 2>&1; then
+    echo "CALIB BOOT FAILED; see /home/dank/logs/swift15_bf16.log"; kill "$CP" 2>/dev/null; exit 1
+fi
+echo "CALIB UP, pushing corpus"
+python3 "$S/calib-push.py" --port 7770 --parallel 16 --container swift15ref --max-rounds 6
+PUSH=$?
+kill "$CP" 2>/dev/null
+sleep 10
+echo "CALIB_CHAIN_DONE push=$PUSH grams=$(find /home/dank/corpora/calib -name 'gram.r*.bin' 2>/dev/null | wc -l)"

scripts/make-calib-corpus.py

@@ -0,0 +1,317 @@
+#!/usr/bin/env python3
+"""Build the two Swift1.5 campaign corpora (calibration + KLD).
+
+calib.jsonl — 10M tokens of agent-shaped RAW text (45% code, 30% prose,
+25% tool-call JSON), one {"prompt": text} per line, pushed to
+/v1/completions with max_tokens=1 during the calibration run (the Hessian
+tap is of the prompt tokens; docs/MOE-W4.md step 2). Code and prose mix
+the repo checkout with HF dataset parquets (Frank's call 2026-10-06:
+local material is only ~1.6M unique tokens; MOE-W4 wants 10M):
+nampdn-ai/tiny-codes (code) and Salesforce/wikitext wikitext-103-raw
+(prose). Tool calls are synthesized from commands harvested from the
+repo's own scripts plus a fleet-ops set, with argument-shaped variants.
+
+kld.jsonl — 1.8M tokens, one {"prompt", "score_from", "source"} per
+line; each prompt is a multi-turn conversation rendered through the
+Froggeric v22.5 template with a >= --kld-min-prompt (8192) token target,
+and the last 256 tokens (the tail of the assistant answer) are the
+scored span: score_from = n - 256. At 8k-token prompts the 1.8M corpus
+is ~220 prompts = ~56k scored positions ~= 25 GB of record, matching
+the published Flash-Next KLD scale (83k positions, MOE-W4.md). The KLD
+reference run and every candidate run consume the SAME file (TOOLS.md
+--kld-record / --kld-ref). The KLD corpus is LOCAL material only, so the
+file-level disjointness rule (fnv1a64(path) % 5 == 0 -> KLD pool) is
+untouched by the HF additions.
+
+Deps: tokenizers, jinja2, pyarrow (venv /home/dank/venvs/radcorpus).
+Usage: make-calib-corpus.py --repo ROOT --tokenizer-json PATH
+    --template PATH --out-dir DIR [--hf-code-parquet F,F]
+    [--hf-prose-parquet F,F] [--calib-tokens 10000000]
+    [--kld-tokens 1800000]
+"""
+import argparse
+import json
+import os
+import random
+import re
+
+from jinja2 import Environment, FileSystemLoader
+from tokenizers import Tokenizer
+
+CODE_EXT = (".sh", ".py", ".jinja", ".recipe", ".env", ".example",
+            ".yaml", ".yml", ".bats", ".service", ".container", ".toml")
+JSONL_EXTS = (".jsonl", ".json")
+DOC_DIR = "serverAppConfigs-docs"
+
+FLEET_CMDS = [
+    "systemctl status frank-primary --no-pager",
+    "journalctl -u frank-primary -n 100 --no-pager",
+    "podman ps --format '{{.Names}} {{.Status}}'",
+    "podman logs --tail 50 frank-primary",
+    "curl -s http://192.168.0.235:7770/health",
+    "curl -s http://192.168.0.235:7770/metrics | grep radiance_",
+    "df -h /home/dank | tail -1",
+    "free -g",
+    "sudo systemctl restart frank-primary",
+    "cd /home/dank/serverAppConfigs && git pull --ff-only",
+    "./scripts/deploy.sh dankserver --service frank-primary",
+    "./tests/run.sh 2>&1 | tail -5",
+    "jq . /home/dank/models/rad/index.json | head -40",
+    "python3 -c \"import json; print(json.load(open('config.json'))['num_hidden_layers'])\"",
+    "ip neigh show 192.168.0.235",
+    "amdgpu_top --device-info 2>/dev/null | head -20",
+    "grep -E 'MemAvailable' /proc/meminfo",
+    "sha256sum model-00001-of-00131.safetensors",
+    "git log --oneline -10",
+    "awk '{s+=$1} END {print s}' shard_sizes.txt",
+]
+TOOL_NAMES = ["bash", "read", "grep", "web_search", "edit", "write"]
+WORKDIRS = ["/home/dank", "/home/dank/serverAppConfigs", "/home/dank/models",
+            "/etc/systemd/system", "/var/log"]
+DESCS = ["check service state", "tail the engine ledger", "list containers",
+          "query health", "read metrics", "check disk", "check memory",
+          "restart the endpoint", "sync the checkout", "run deployer",
+          "run the test seams", "inspect the index", "probe the peer"]
+
+
+def fnv(s: str) -> int:
+    h = 0xcbf29ce484222325
+    for b in s.encode():
+        h = ((h ^ b) * 0x100000001b3) & 0xFFFFFFFFFFFFFFFF
+    return h
+
+
+def harvest_cmds(repo: str, rng: random.Random) -> list[str]:
+    cmds = list(FLEET_CMDS)
+    pat = re.compile(r"^\s*(?:sudo\s+)?([a-z][\w./-]*(?:\s+\S+){1,6})\s*$")
+    for root, dirs, files in os.walk(repo):
+        dirs[:] = [d for d in dirs if not d.startswith(".")]
+        for f in files:
+            if not f.endswith(".sh"):
+                continue
+            try:
+                for line in open(os.path.join(root, f), encoding="utf-8",
+                                 errors="replace"):
+                    m = pat.match(line.rstrip())
+                    if m and len(m.group(1)) > 12:
+                        cmds.append("ssh dank@192.168.0.235 '"
+                                    + m.group(1)[:90] + "'")
+            except OSError:
+                pass
+    rng.shuffle(cmds)
+    return cmds[:4000]
+
+
+def tool_call(rng: random.Random, cmds: list[str]) -> str:
+    """One agent turn: 2-4 tool calls with argument-shaped JSON."""
+    parts = []
+    for _ in range(rng.randint(2, 4)):
+        args: dict = {"command": rng.choice(cmds)}
+        if rng.random() < 0.4:
+            args["cwd"] = rng.choice(WORKDIRS)
+        if rng.random() < 0.3:
+            args["timeout"] = rng.choice([10, 30, 60, 300, 600])
+        if rng.random() < 0.5:
+            args["description"] = rng.choice(DESCS)
+        parts.append('<tool_call>{"name": "%s", "arguments": %s}<|im_end|>'
+                     % (rng.choice(TOOL_NAMES),
+                        json.dumps(args, ensure_ascii=False)))
+    return "\n".join(parts)
+
+
+def load_parquet_texts(path: str, kind: str, rng: random.Random,
+                       cap: int) -> list[str]:
+    import pyarrow.parquet as pq
+    d = pq.read_table(path).to_pydict()
+    n = len(next(iter(d.values())))
+    idx = range(n) if n <= cap else rng.sample(range(n), cap)
+    out = []
+    for i in idx:
+        if kind == "code":
+            if "content" in d:
+                frag = d["content"][i]
+            else:
+                frag = ("// %s\n### Instruction:\n%s\n### Response:\n%s"
+                        % (d.get("file_name", [""])[i], d["prompt"][i],
+                           d["code"][i] or d["response"][i]))
+        else:
+            frag = d["text"][i]
+        if frag and frag.strip():
+            out.append(frag[:8000])
+    rng.shuffle(out)
+    return out[:cap]
+
+
+def main() -> None:
+    ap = argparse.ArgumentParser(description=__doc__)
+    ap.add_argument("--repo", required=True)
+    ap.add_argument("--tokenizer-json", required=True)
+    ap.add_argument("--template", required=True)
+    ap.add_argument("--out-dir", required=True)
+    ap.add_argument("--calib-tokens", type=int, default=10_000_000)
+    ap.add_argument("--kld-tokens", type=int, default=1_800_000)
+    ap.add_argument("--chunk-tokens", type=int, default=8192)
+    ap.add_argument("--score-tail", type=int, default=256)
+    ap.add_argument("--kld-min-prompt", type=int, default=8192)
+    ap.add_argument("--hf-code-parquet", default="")
+    ap.add_argument("--hf-prose-parquet", default="")
+    ap.add_argument("--seed", type=int, default=20261006)
+    a = ap.parse_args()
+
+    rng = random.Random(a.seed)
+    tokenizer = Tokenizer.from_file(a.tokenizer_json)
+    env = Environment(loader=FileSystemLoader(os.path.dirname(a.template)),
+                      trim_blocks=True, lstrip_blocks=True)
+    env.filters["tojson"] = lambda v: json.dumps(v, ensure_ascii=False)
+    env.filters["raise_exception"] = lambda m: (_ for _ in ()).throw(
+        ValueError(m))
+    tname = os.path.basename(a.template)
+
+    code, prose, jsonl = [], [], []
+    for root, dirs, files in os.walk(a.repo):
+        dirs[:] = [d for d in dirs if not d.startswith(".")]
+        for f in sorted(files):
+            p = os.path.relpath(os.path.join(root, f), a.repo)
+            if DOC_DIR in p and f.endswith(".md"):
+                prose.append(p)
+            elif p.startswith(DOC_DIR):
+                pass
+            elif f.endswith(CODE_EXT):
+                code.append(p)
+            elif f.endswith(JSONL_EXTS) and f != "package-lock.json":
+                jsonl.append(p)
+    # file-level split: bucket 0 -> KLD pool, 1..4 -> calibration pool
+    pools: dict[str, tuple[list, list]] = {}
+    for cls, lst in (("code", code), ("prose", prose), ("tools", jsonl)):
+        pools[cls] = ([p for p in lst if fnv(p) % 5 == 0],
+                      [p for p in lst if fnv(p) % 5 != 0])
+
+    def text_of(p: str) -> str:
+        path = os.path.join(a.repo, p)
+        raw = open(path, encoding="utf-8", errors="replace").read()
+        if p.endswith(JSONL_EXTS):
+            lines = [json.loads(l)["content"] for l in raw.splitlines()
+                     if l.strip() and "content" in l]
+            raw = "\n".join(lines)
+        toks = tokenizer.encode(raw, add_special_tokens=False).ids
+        return tokenizer.decode(toks[:2400], skip_special_tokens=True)
+
+    cmds = harvest_cmds(a.repo, rng)
+    # fragment pools per class, corpus-split first
+    frags: dict[str, dict[str, list[str]]] = {}
+    for cls in pools:
+        kld_files, cal_files = pools[cls]
+        fr = {}
+        src = {"kld": kld_files, "cal": cal_files}
+        if cls == "tools":
+            fr["kld"] = [tool_call(rng, cmds) for _ in range(600)]
+            fr["cal"] = [tool_call(rng, cmds) for _ in range(40000)]
+        else:
+            fr["kld"] = [text_of(p) for p in src["kld"]]
+            fr["cal"] = [text_of(p) for p in src["cal"]]
+        frags[cls] = fr
+    # per-class fragment caps: prose rows are short (~66 tok) so need many;
+    # codeparrot files are long (~2.6k tok) so a small cap suffices
+    hfcap = {"code": 40000, "prose": 60000, "tools": 0}
+    for cls, spec in (("code", a.hf_code_parquet),
+                      ("prose", a.hf_prose_parquet)):
+        for path in [p for p in spec.split(",") if p]:
+            frags[cls]["cal"].extend(
+                load_parquet_texts(path, cls, rng, cap=hfcap[cls]))
+    for cls in frags:
+        rng.shuffle(frags[cls]["cal"])
+
+    def toks(s: str) -> int:
+        return len(tokenizer.encode(s, add_special_tokens=False).ids)
+
+    def emit_calib() -> tuple[int, list[dict]]:
+        targets = {"code": .45, "prose": .30, "tools": .25}
+        rows: list[dict] = []
+        for cls, frac in targets.items():
+            want = int(a.calib_tokens * frac)
+            pool = frags[cls]["cal"]
+            if not pool:
+                continue
+            got, i, buf, nbuf = 0, 0, [], 0
+            while got < want:
+                frag = pool[i % len(pool)]
+                i += 1
+                n = toks(frag)
+                if nbuf + n > a.chunk_tokens and buf:
+                    rows.append({"prompt": "\n".join(buf), "cls": cls})
+                    got += nbuf
+                    buf, nbuf = [], 0
+                    if got >= want:
+                        break
+                buf.append(frag)
+                nbuf += n
+                if i > len(pool) * 40:
+                    break  # pool exhausted with repeats; stop
+            if buf and got < want:
+                rows.append({"prompt": "\n".join(buf), "cls": cls})
+                got += nbuf
+            print(f"calib {cls}: {got} tokens, {i} fragments")
+        rng.shuffle(rows)
+        return len(rows), [{k: v for k, v in r.items() if k == "prompt"}
+                           for r in rows]
+
+    def emit_kld() -> tuple[int, list[dict]]:
+        rows, total = [], 0
+        classes = ["code", "prose", "tools"]
+        i = {c: 0 for c in classes}
+        guard = 0
+        while total < a.kld_tokens and guard < 5000:
+            guard += 1
+            msgs, srcs = [], []
+            for t in range(rng.choice([2, 3, 4])):
+                cls = classes[(len(rows) + t) % 3]
+                pool = frags[cls]["kld"]
+                if not pool:
+                    continue
+                frag = pool[i[cls] % len(pool)]
+                i[cls] += 2
+                msgs.append({"role": "user",
+                             "content": "Summarize and explain this "
+                                        f"engineering material:\n{frag}"})
+                msgs.append({"role": "assistant",
+                             "content": frag[:4000]})
+                srcs.append(cls)
+            rendered = env.get_template(tname).render(
+                messages=msgs, add_generation_prompt=False)
+            n = toks(rendered)
+            if n < a.kld_min_prompt:
+                continue
+            rows.append({"prompt": rendered,
+                         "score_from": n - a.score_tail,
+                         "source": "+".join(sorted(set(srcs)))})
+            total += n
+        return len(rows), rows
+
+    os.makedirs(a.out_dir, exist_ok=True)
+    nc, rows = emit_calib()
+    with open(os.path.join(a.out_dir, "calib.jsonl"), "w") as f:
+        for r in rows:
+            f.write(json.dumps(r) + "\n")
+    nk, krows = emit_kld()
+    with open(os.path.join(a.out_dir, "kld.jsonl"), "w") as f:
+        for r in krows:
+            f.write(json.dumps(r) + "\n")
+    scored = sum(len(tokenizer.encode(r["prompt"], add_special_tokens=False).ids)
+                 - r["score_from"] for r in krows)
+    manifest = {
+        "seed": a.seed, "template": tname,
+        "tokenizer": os.path.basename(a.tokenizer_json),
+        "calib_prompts": nc, "kld_prompts": nk,
+        "kld_scored_positions": scored,
+        "kld_score_tail": a.score_tail,
+        "disjoint_file_pools": "local KLD material only; HF code feeds calib",
+        "calib_mix": {"code": .45, "prose": .30, "tools": .25},
+        "hf_code": a.hf_code_parquet, "hf_prose": a.hf_prose_parquet,
+    }
+    with open(os.path.join(a.out_dir, "manifest.json"), "w") as f:
+        json.dump(manifest, f, indent=1)
+    print(json.dumps(manifest, indent=1))
+
+
+if __name__ == "__main__":
+    main()

scripts/qwen38-froggeric-v22.5.jinja

@@ -0,0 +1,460 @@
+{%- set template_version = "qwen3.8-froggeric-v22.5" %}
+{%- set _tool_format = tool_call_format if tool_call_format is defined else 'xml' %}
+{%- set image_count = namespace(value=0) %}
+{%- set video_count = namespace(value=0) %}
+{%- set add_vision_id = add_vision_id if add_vision_id is defined else false %}
+{%- set enable_thinking = enable_thinking if enable_thinking is defined else true %}
+{%- set auto_disable_thinking_with_tools = auto_disable_thinking_with_tools if auto_disable_thinking_with_tools is defined else false %}
+{%- if preserve_reasoning is defined and preserve_reasoning is not none %}
+    {%- set _preserve_thinking = preserve_reasoning %}
+{%- elif preserve_thinking is defined and preserve_thinking is not none %}
+    {%- set _preserve_thinking = preserve_thinking %}
+{%- else %}
+    {%- set _preserve_thinking = true %}
+{%- endif %}
+{%- set max_tool_arg_chars = max_tool_arg_chars if max_tool_arg_chars is defined else 0 %}
+{%- set max_tool_response_chars = max_tool_response_chars if max_tool_response_chars is defined else 0 %}
+{%- set _default_reasoning_effort = 'medium' %}
+{%- set _has_tools = (tools is defined and tools and tools is iterable and tools is not mapping) %}
+{%- set _effort_raw = (reasoning_effort | string | lower) if reasoning_effort is defined and reasoning_effort is not none else _default_reasoning_effort %}
+{%- set _initial_thinking = enable_thinking %}
+{%- if _effort_raw in ('none', 'off') %}
+    {%- set _initial_thinking = false %}
+    {%- set _initial_effort = 'medium' %}
+{%- elif _effort_raw in ('minimal', 'low') %}
+    {%- set _initial_effort = 'low' %}
+{%- elif _effort_raw in ('high', 'xhigh', 'max', 'ultracode', 'extreme') %}
+    {%- set _initial_effort = 'xhigh' %}
+{%- else %}
+    {%- set _initial_effort = 'medium' %}
+{%- endif %}
+{%- set ns_state = namespace(thinking=_initial_thinking, effort=_initial_effort) %}
+{%- if auto_disable_thinking_with_tools and _has_tools %}
+    {%- set ns_state.thinking = false %}
+{%- endif %}
+{%- for msg in messages %}
+    {%- if msg.role == 'system' or msg.role == 'developer' or msg.role == 'user' %}
+        {%- if msg.content is string %}
+            {%- if '<|think_off|>' in msg.content %}
+                {%- set ns_state.thinking = false %}
+            {%- elif '<|think_on|>' in msg.content %}
+                {%- set ns_state.thinking = true %}
+            {%- elif '<|think_xhigh|>' in msg.content or '<|think_high|>' in msg.content or '<|think_ultracode|>' in msg.content or '<|think_extreme|>' in msg.content or '<|think_max|>' in msg.content %}
+                {%- set ns_state.thinking = true %}
+                {%- set ns_state.effort = 'xhigh' %}
+            {%- elif '<|think_low|>' in msg.content or '<|think_minimal|>' in msg.content %}
+                {%- set ns_state.thinking = true %}
+                {%- set ns_state.effort = 'low' %}
+            {%- elif '<|think_medium|>' in msg.content %}
+                {%- set ns_state.thinking = true %}
+                {%- set ns_state.effort = 'medium' %}
+            {%- endif %}
+        {%- elif msg.content is iterable and msg.content is not mapping %}
+            {%- for item in msg.content %}
+                {%- if item is string %}
+                    {%- set _item_text = item %}
+                {%- elif item is mapping and 'text' in item and item.text is string %}
+                    {%- set _item_text = item.text %}
+                {%- else %}
+                    {%- set _item_text = '' %}
+                {%- endif %}
+                {%- if _item_text %}
+                    {%- if '<|think_off|>' in _item_text %}
+                        {%- set ns_state.thinking = false %}
+                    {%- elif '<|think_on|>' in _item_text %}
+                        {%- set ns_state.thinking = true %}
+                    {%- elif '<|think_xhigh|>' in _item_text or '<|think_high|>' in _item_text or '<|think_ultracode|>' in _item_text or '<|think_extreme|>' in _item_text or '<|think_max|>' in _item_text %}
+                        {%- set ns_state.thinking = true %}
+                        {%- set ns_state.effort = 'xhigh' %}
+                    {%- elif '<|think_low|>' in _item_text or '<|think_minimal|>' in _item_text %}
+                        {%- set ns_state.thinking = true %}
+                        {%- set ns_state.effort = 'low' %}
+                    {%- elif '<|think_medium|>' in _item_text %}
+                        {%- set ns_state.thinking = true %}
+                        {%- set ns_state.effort = 'medium' %}
+                    {%- endif %}
+                {%- endif %}
+            {%- endfor %}
+        {%- endif %}
+    {%- endif %}
+{%- endfor %}
+{%- set reasoning_instructions = '' %}
+{%- if ns_state.thinking %}
+    {%- if ns_state.effort == 'xhigh' %}
+        {%- set reasoning_instructions = 'Reasoning effort is set to xhigh. Please think carefully through the task, validate key assumptions, consider plausible alternatives, and prioritize correctness, consistency, and clarity in the final answer.' %}
+    {%- elif ns_state.effort == 'low' %}
+        {%- set reasoning_instructions = 'Reasoning effort is set to low. Keep your thinking brief and focused, moving directly to the conclusion without unnecessary elaboration.' %}
+    {%- endif %}
+{%- endif %}
+{%- macro render_content(content, do_vision_count, is_system_content=false) %}
+    {%- if content is string %}
+        {{- content }}
+    {%- elif content is iterable and content is not mapping %}
+        {%- for item in content %}
+            {%- if item is mapping %}
+                {%- if item.type == 'image' or 'image' in item or 'image_url' in item %}
+                    {%- if is_system_content %}
+                        {{- raise_exception('System message cannot contain images.') }}
+                    {%- endif %}
+                    {%- if do_vision_count %}
+                        {%- set image_count.value = image_count.value + 1 %}
+                    {%- endif %}
+                    {%- if add_vision_id %}
+                        {{- 'Picture ' ~ image_count.value ~ ': ' }}
+                    {%- endif %}
+                    {{- '<|vision_start|><|image_pad|><|vision_end|>' }}
+                {%- elif item.type == 'video' or 'video' in item or 'video_url' in item %}
+                    {%- if is_system_content %}
+                        {{- raise_exception('System message cannot contain videos.') }}
+                    {%- endif %}
+                    {%- if do_vision_count %}
+                        {%- set video_count.value = video_count.value + 1 %}
+                    {%- endif %}
+                    {%- if add_vision_id %}
+                        {{- 'Video ' ~ video_count.value ~ ': ' }}
+                    {%- endif %}
+                    {{- '<|vision_start|><|video_pad|><|vision_end|>' }}
+                {%- elif 'text' in item %}
+                    {{- item.text }}
+                {%- else %}
+                    {{- raise_exception('Unexpected item type in content.') }}
+                {%- endif %}
+            {%- else %}
+                {{- item | string }}
+            {%- endif %}
+        {%- endfor %}
+    {%- elif content is none or content is undefined %}
+        {{- '' }}
+    {%- else %}
+        {{- raise_exception('Unexpected content type.') }}
+    {%- endif %}
+{%- endmacro %}
+{%- if not messages %}
+    {{- raise_exception('No messages provided.') }}
+{%- endif %}
+{%- set head = namespace(count=0, seen_non_system=false) %}
+{%- for message in messages %}
+    {%- set _is_sys = (message.role == 'system' or message.role == 'developer') %}
+    {%- if _is_sys and not head.seen_non_system %}
+        {%- set head.count = head.count + 1 %}
+    {%- else %}
+        {%- set head.seen_non_system = true %}
+    {%- endif %}
+{%- endfor %}
+{%- set sys_state = namespace(content='') %}
+{%- for message in messages[:head.count] %}
+    {%- set _part = render_content(message.content, false, true) | trim %}
+    {%- if '<|think_off|>' in _part %}{%- set _part = _part.split('<|think_off|>') | join('') | trim %}{%- endif %}
+    {%- if '<|think_on|>' in _part %}{%- set _part = _part.split('<|think_on|>') | join('') | trim %}{%- endif %}
+    {%- if '<|think_xhigh|>' in _part %}{%- set _part = _part.split('<|think_xhigh|>') | join('') | trim %}{%- endif %}
+    {%- if '<|think_high|>' in _part %}{%- set _part = _part.split('<|think_high|>') | join('') | trim %}{%- endif %}
+    {%- if '<|think_ultracode|>' in _part %}{%- set _part = _part.split('<|think_ultracode|>') | join('') | trim %}{%- endif %}
+    {%- if '<|think_extreme|>' in _part %}{%- set _part = _part.split('<|think_extreme|>') | join('') | trim %}{%- endif %}
+    {%- if '<|think_max|>' in _part %}{%- set _part = _part.split('<|think_max|>') | join('') | trim %}{%- endif %}
+    {%- if '<|think_medium|>' in _part %}{%- set _part = _part.split('<|think_medium|>') | join('') | trim %}{%- endif %}
+    {%- if '<|think_low|>' in _part %}{%- set _part = _part.split('<|think_low|>') | join('') | trim %}{%- endif %}
+    {%- if '<|think_minimal|>' in _part %}{%- set _part = _part.split('<|think_minimal|>') | join('') | trim %}{%- endif %}
+    {%- if _part %}
+        {%- if sys_state.content %}
+            {%- set sys_state.content = sys_state.content ~ '\n\n' ~ _part %}
+        {%- else %}
+            {%- set sys_state.content = _part %}
+        {%- endif %}
+    {%- endif %}
+{%- endfor %}
+{%- set _sc = sys_state.content %}
+{%- set _msgs = messages[head.count:] %}
+{%- if _has_tools %}
+    {{- '<|im_start|>system\n' }}
+    {%- if reasoning_instructions %}
+        {{- reasoning_instructions + '\n\n' }}
+    {%- endif %}
+    {{- '# Tools\n\nYou have access to the following functions:\n\n<tools>' }}
+    {%- for tool in tools %}
+        {{- '\n' }}
+        {{- tool | tojson }}
+    {%- endfor %}
+    {{- '\n</tools>' }}
+    {%- if _tool_format == 'json' %}
+        {{- '\n\nIf you choose to call a function ONLY reply in the following format with NO suffix:\n\n' }}
+        {%- if ns_state.thinking %}
+            {{- '<think>\nBrief explanation of tool call\n</think>\n' }}
+        {%- endif %}
+        {{- '<tool_call>\n{"name": "example_function_name", "arguments": {"example_parameter_1": "value_1", "example_parameter_2": "This is the value for the second parameter"}}\n</tool_call>\n\n<IMPORTANT>\nReminder:\n' }}
+        {%- if ns_state.thinking %}
+            {{- '- You can use the <think></think> block to plan your next tool call OR to synthesize data and formulate your final response to the user.\n- ALL explanation and reasoning MUST be placed strictly inside the <think></think> block.\n' }}
+        {%- endif %}
+        {{- '- Function calls MUST follow the specified format: a single JSON object with "name" and "arguments" keys inside <tool_call></tool_call> XML tags.\n' }}
+        {%- if ns_state.thinking %}
+            {{- '- If you choose to call a tool, you MUST output the <tool_call> block IMMEDIATELY after thinking, with NO conversational text before it.\n' }}
+        {%- else %}
+            {{- '- If you choose to call a tool, you MUST output the <tool_call> block IMMEDIATELY, with NO conversational text before it.\n' }}
+        {%- endif %}
+        {{- '- The <tool_call> tag MUST be at the very beginning of a new line, with NO spaces or indentation before it.\n- To call multiple functions, output a separate, completely closed <tool_call></tool_call> block for EACH function. Do NOT nest <tool_call> blocks.\n- If you have all necessary data, provide your final answer directly to the user without any tool call.\n</IMPORTANT>' }}
+    {%- else %}
+        {{- '\n\nIf you choose to call a function ONLY reply in the following format with NO suffix:\n\n' }}
+        {%- if ns_state.thinking %}
+            {{- '<think>\nBrief explanation of tool call\n</think>\n' }}
+        {%- endif %}
+        {{- '<tool_call>\n<function=example_function_name>\n<parameter=example_parameter_1>\nvalue_1\n</parameter>\n<parameter=example_parameter_2>\nThis is the value for the second parameter\nthat can span\nmultiple lines\n</parameter>\n</function>\n</tool_call>\n\n<IMPORTANT>\nReminder:\n' }}
+        {%- if ns_state.thinking %}
+            {{- '- You can use the <think></think> block to plan your next tool call OR to synthesize data and formulate your final response to the user.\n- ALL explanation and reasoning MUST be placed strictly inside the <think></think> block.\n' }}
+        {%- endif %}
+        {{- '- Function calls MUST follow the specified format: an inner <function=...></function> block must be nested within <tool_call></tool_call> XML tags.\n' }}
+        {%- if ns_state.thinking %}
+            {{- '- If you choose to call a tool, you MUST output the <tool_call> block IMMEDIATELY after thinking, with NO conversational text before it.\n' }}
+        {%- else %}
+            {{- '- If you choose to call a tool, you MUST output the <tool_call> block IMMEDIATELY, with NO conversational text before it.\n' }}
+        {%- endif %}
+        {{- '- The <tool_call> and <function> tags MUST be at the very beginning of a new line, with NO spaces or indentation before them.\n- To call multiple functions, output a separate, completely closed <tool_call></tool_call> block for EACH function. Do NOT nest <tool_call> blocks.\n- If you have all necessary data, provide your final answer directly to the user without any tool call.\n</IMPORTANT>' }}
+    {%- endif %}
+    {%- if _sc %}
+        {{- '\n\n' + _sc }}
+    {%- endif %}
+    {{- '<|im_end|>\n' }}
+{%- else %}
+    {%- if _sc %}
+        {{- '<|im_start|>system\n' + (reasoning_instructions + '\n\n' if reasoning_instructions else '') + _sc + '<|im_end|>\n' }}
+    {%- elif reasoning_instructions %}
+        {{- '<|im_start|>system\n' + reasoning_instructions + '<|im_end|>\n' }}
+    {%- endif %}
+{%- endif %}
+{%- set _last_idx = _msgs | length - 1 %}
+{%- set ns = namespace(multi_step_tool=true, last_query_index=_last_idx) %}
+{%- for message in _msgs[::-1] %}
+    {%- set index = (_msgs | length - 1) - loop.index0 %}
+    {%- if ns.multi_step_tool and message.role == 'user' %}
+        {%- set _rc = render_content(message.content, false) | trim %}
+        {%- if not (_rc.startswith('<tool_response>') and _rc.endswith('</tool_response>')) %}
+            {%- set ns.multi_step_tool = false %}
+            {%- set ns.last_query_index = index %}
+        {%- endif %}
+    {%- endif %}
+{%- endfor %}
+{%- if ns.multi_step_tool %}
+    {%- if _last_idx > 50 %}
+        {%- set ns.last_query_index = _last_idx %}
+    {%- else %}
+        {%- set ns.last_query_index = 0 %}
+    {%- endif %}
+{%- endif %}
+{%- set ns2 = namespace(prev_role='', consecutive_failures=0) %}
+{%- for message in _msgs %}
+    {%- set is_system = (message.role == "system" or message.role == "developer") %}
+    {%- set content = render_content(message.content, true, is_system) | trim %}
+    {%- if is_system or message.role == 'user' %}
+        {%- if '<|think_off|>' in content %}{%- set content = content.split('<|think_off|>') | join('') | trim %}{%- endif %}
+        {%- if '<|think_on|>' in content %}{%- set content = content.split('<|think_on|>') | join('') | trim %}{%- endif %}
+        {%- if '<|think_xhigh|>' in content %}{%- set content = content.split('<|think_xhigh|>') | join('') | trim %}{%- endif %}
+        {%- if '<|think_high|>' in content %}{%- set content = content.split('<|think_high|>') | join('') | trim %}{%- endif %}
+        {%- if '<|think_ultracode|>' in content %}{%- set content = content.split('<|think_ultracode|>') | join('') | trim %}{%- endif %}
+        {%- if '<|think_extreme|>' in content %}{%- set content = content.split('<|think_extreme|>') | join('') | trim %}{%- endif %}
+        {%- if '<|think_max|>' in content %}{%- set content = content.split('<|think_max|>') | join('') | trim %}{%- endif %}
+        {%- if '<|think_medium|>' in content %}{%- set content = content.split('<|think_medium|>') | join('') | trim %}{%- endif %}
+        {%- if '<|think_low|>' in content %}{%- set content = content.split('<|think_low|>') | join('') | trim %}{%- endif %}
+        {%- if '<|think_minimal|>' in content %}{%- set content = content.split('<|think_minimal|>') | join('') | trim %}{%- endif %}
+    {%- endif %}
+    {%- if is_system %}
+        {{- '<|im_start|>system\n' + content + '<|im_end|>\n' }}
+    {%- elif message.role == 'user' %}
+        {%- set ns2.consecutive_failures = 0 %}
+        {{- '<|im_start|>user\n' + content + '<|im_end|>\n' }}
+    {%- elif message.role == 'assistant' %}
+        {%- set reasoning_content = '' %}
+        {%- set _explicit_reasoning = '' %}
+        {%- if message.reasoning_content is defined and message.reasoning_content is not none %}
+            {%- if message.reasoning_content is string %}
+                {%- set _explicit_reasoning = message.reasoning_content %}
+            {%- else %}
+                {%- set _explicit_reasoning = message.reasoning_content | string %}
+            {%- endif %}
+        {%- elif message.thinking is defined and message.thinking is not none %}
+            {%- if message.thinking is string %}
+                {%- set _explicit_reasoning = message.thinking %}
+            {%- else %}
+                {%- set _explicit_reasoning = message.thinking | string %}
+            {%- endif %}
+        {%- elif message.reasoning is defined and message.reasoning is not none %}
+            {%- if message.reasoning is string %}
+                {%- set _explicit_reasoning = message.reasoning %}
+            {%- else %}
+                {%- set _explicit_reasoning = message.reasoning | string %}
+            {%- endif %}
+        {%- endif %}
+        {%- if _explicit_reasoning %}
+            {%- set _lead_end = '' %}
+            {%- if content.startswith('<think>') and '</think>' in content %}
+                {%- set _lead_end = '</think>' %}
+            {%- elif content.startswith('<thinking>') and '</thinking>' in content %}
+                {%- set _lead_end = '</thinking>' %}
+            {%- elif content.startswith('</think>') %}
+                {%- set _lead_end = '</think>' %}
+            {%- elif content.startswith('</thinking>') %}
+                {%- set _lead_end = '</thinking>' %}
+            {%- endif %}
+            {%- if _lead_end %}
+                {%- set content = content.split(_lead_end)[-1].lstrip('\n') %}
+            {%- endif %}
+            {%- set reasoning_content = _explicit_reasoning %}
+        {%- else %}
+            {%- set _think_end = '' %}
+            {%- if content.startswith('</think>') %}
+                {%- set _think_end = '</think>' %}
+            {%- elif content.startswith('</thinking>') %}
+                {%- set _think_end = '</thinking>' %}
+            {%- elif '\n</think>' in content %}
+                {%- set _think_end = '\n</think>' %}
+            {%- elif '\n</thinking>' in content %}
+                {%- set _think_end = '\n</thinking>' %}
+            {%- elif '\n</ think>' in content %}
+                {%- set _think_end = '\n</ think>' %}
+            {%- elif '\n</think >' in content %}
+                {%- set _think_end = '\n</think >' %}
+            {%- elif content.startswith('<think>') and '</think>' in content %}
+                {%- set _think_end = '</think>' %}
+            {%- elif content.startswith('<thinking>') and '</thinking>' in content %}
+                {%- set _think_end = '</thinking>' %}
+            {%- endif %}
+            {%- if _think_end %}
+                {%- if 'thinking' in _think_end %}
+                    {%- set _think_start = '<thinking>' %}
+                {%- else %}
+                    {%- set _think_start = '<think>' %}
+                {%- endif %}
+                {%- set reasoning_content = content.split(_think_end)[0].rstrip('\n') %}
+                {%- if _think_start in reasoning_content %}
+                    {%- set reasoning_content = reasoning_content.split(_think_start)[-1].lstrip('\n') %}
+                {%- endif %}
+                {%- set content = content.split(_think_end)[-1].lstrip('\n') %}
+            {%- endif %}
+        {%- endif %}
+        {%- set reasoning_content = reasoning_content | trim %}
+        {%- if (_preserve_thinking or loop.index0 > ns.last_query_index) %}
+            {{- '<|im_start|>assistant\n<think>\n' + reasoning_content + '\n</think>\n\n' + content }}
+        {%- else %}
+            {{- '<|im_start|>assistant\n' + content }}
+        {%- endif %}
+        {%- if message.tool_calls is defined and message.tool_calls and message.tool_calls is iterable and message.tool_calls is not mapping %}
+            {%- for tool_call in message.tool_calls %}
+                {%- if tool_call.function is defined and tool_call.function is not none %}
+                    {%- set tc = tool_call.function %}
+                {%- else %}
+                    {%- set tc = tool_call %}
+                {%- endif %}
+                {%- set tc_name = tc.name if (tc.name is defined and tc.name is not none) else '' %}
+                {%- if _tool_format == 'json' %}
+                    {%- if loop.first %}
+                        {%- if content | trim %}
+                            {{- '\n\n' }}
+                        {%- endif %}
+                    {%- else %}
+                        {{- '\n' }}
+                    {%- endif %}
+                    {%- set _args = '{}' %}
+                    {%- if tc.arguments is defined and tc.arguments is not none %}
+                        {%- if tc.arguments is mapping %}
+                            {%- set _args = tc.arguments | tojson %}
+                        {%- elif tc.arguments is string %}
+                            {%- if tc.arguments %}
+                                {%- set _args = tc.arguments %}
+                            {%- endif %}
+                        {%- else %}
+                            {%- set _args = tc.arguments | tojson %}
+                        {%- endif %}
+                    {%- endif %}
+                    {{- '<tool_call>\n{"name": ' }}{{- tc_name | tojson }}{{- ', "arguments": ' }}{{- _args }}{{- '}\n</tool_call>' }}
+                {%- else %}
+                    {%- if loop.first %}
+                        {%- if content | trim %}
+                            {{- '\n\n<tool_call>\n<function=' + tc_name + '>\n' }}
+                        {%- else %}
+                            {{- '<tool_call>\n<function=' + tc_name + '>\n' }}
+                        {%- endif %}
+                    {%- else %}
+                        {{- '\n<tool_call>\n<function=' + tc_name + '>\n' }}
+                    {%- endif %}
+                    {%- if tc.arguments is defined and tc.arguments is not none %}
+                        {%- if tc.arguments is mapping %}
+                            {%- for args_name, args_value in tc.arguments.items() %}
+                                {{- '<parameter=' + args_name + '>\n' }}
+                                {%- if args_value is string %}
+                                    {%- set _av = args_value %}
+                                {%- else %}
+                                    {%- set _av = args_value | tojson %}
+                                {%- endif %}
+                                {%- if max_tool_arg_chars > 0 and _av | length > max_tool_arg_chars %}
+                                    {{- _av[:max_tool_arg_chars] + '\n[TRUNCATED - original length ' ~ (_av | length | string) ~ ' chars]' }}
+                                {%- else %}
+                                    {{- _av }}
+                                {%- endif %}
+                                {{- '\n</parameter>\n' }}
+                            {%- endfor %}
+                        {%- else %}
+                            {%- if tc.arguments is string %}
+                                {%- set _raw_args = tc.arguments %}
+                            {%- else %}
+                                {%- set _raw_args = tc.arguments | tojson %}
+                            {%- endif %}
+                            {%- if _raw_args %}
+                                {%- if max_tool_arg_chars > 0 and _raw_args | length > max_tool_arg_chars %}
+                                    {{- _raw_args[:max_tool_arg_chars] + '\n[TRUNCATED - original length ' ~ (_raw_args | length | string) ~ ' chars]' }}
+                                {%- else %}
+                                    {{- _raw_args }}
+                                {%- endif %}
+                            {%- endif %}
+                        {%- endif %}
+                    {%- endif %}
+                    {{- '</function>\n</tool_call>' }}
+                {%- endif %}
+            {%- endfor %}
+        {%- endif %}
+        {{- '<|im_end|>\n' }}
+    {%- elif message.role == 'tool' %}
+        {%- set _content_lower = content | lower %}
+        {%- set _content_head = _content_lower[:120] %}
+        {%- set _is_code_or_grep = ('throw new ' in _content_lower or 'throw error' in _content_lower or 'console.error' in _content_lower or 'logger.error' in _content_lower or 'logging.error' in _content_lower or 'import ' in _content_head or 'def ' in _content_head or 'function ' in _content_head) %}
+        {%- set _exit_code_zero = ('exit code: 0' in _content_head or 'process exited with code 0' in _content_head) %}
+        {%- set _error_field_ok = ('"error": null' in _content_head or '"error":null' in _content_head or '"error": false' in _content_head or '"error":false' in _content_head or '"error": ""' in _content_head or '"error":""' in _content_head) %}
+        {%- set _strong_error = (('"error":' in _content_head and not _error_field_ok) or '"status": "error"' in _content_head or '"status":"error"' in _content_head or 'traceback (most recent call last):' in _content_head or 'command not found' in _content_head or 'invalid syntax' in _content_head or 'fatal:' in _content_head or (('exit code: ' in _content_head or 'process exited with code' in _content_head) and not _exit_code_zero) or _content_head.startswith('exception:') or _content_head.startswith('failed to ')) %}
+        {%- set _weak_error = ('error:' in _content_head or 'err!' in _content_head) %}
+        {%- set _weak_suppressed = ('$ ' in _content_head or 'took ' in _content_head or content | length >= 600) %}
+        {%- if not _is_code_or_grep and (_strong_error or (_weak_error and not _weak_suppressed)) %}
+            {%- set ns2.consecutive_failures = ns2.consecutive_failures + 1 %}
+        {%- else %}
+            {%- set ns2.consecutive_failures = 0 %}
+        {%- endif %}
+        {%- if ns2.prev_role != 'tool' %}
+            {{- '<|im_start|>user' }}
+        {%- endif %}
+        {%- set _is_json_payload = (_tool_format == 'json' and content | trim | length > 0 and (content | trim)[:1] in ('{', '[')) %}
+        {%- if not _is_json_payload and max_tool_response_chars > 0 and content | length > max_tool_response_chars %}
+            {%- set content = content[:max_tool_response_chars] + '\n[TRUNCATED - original length ' ~ (content | length | string) ~ ' chars]' %}
+        {%- endif %}
+        {{- '\n<tool_response>\n' + content }}
+        {%- if ns2.consecutive_failures >= 2 %}
+            {{- '\n\n⚠️ SYSTEM WARNING: ' ~ ns2.consecutive_failures ~ ' consecutive tool errors detected. Your previous approach is incorrect. You MUST use a fundamentally different approach or corrected arguments.' }}
+        {%- elif ns2.consecutive_failures == 1 %}
+            {{- '\n\n⚠️ SYSTEM WARNING: The previous tool call returned an error. Diagnose the failure and retry with completely corrected arguments.' }}
+        {%- endif %}
+        {{- '\n</tool_response>' }}
+        {%- if loop.last %}
+            {{- '<|im_end|>\n' }}
+        {%- else %}
+            {%- set _next_role = _msgs[loop.index0 + 1].role %}
+            {%- if _next_role != 'tool' %}
+                {{- '<|im_end|>\n' }}
+            {%- endif %}
+        {%- endif %}
+    {%- else %}
+        {{- '<|im_start|>user\n[' + message.role + ']: ' + content + '<|im_end|>\n' }}
+    {%- endif %}
+    {%- set ns2.prev_role = message.role %}
+{%- endfor %}
+{%- if add_generation_prompt %}
+    {{- '<|im_start|>assistant\n' }}
+    {%- if not ns_state.thinking %}
+        {{- '<think>\n\n</think>\n\n' }}
+    {%- else %}
+        {{- '<think>\n' }}
+    {%- endif %}
+{%- endif %}

scripts/recipes/qwen4exp-swift15-w4dn8-i8-hc8m.recipe

@@ -0,0 +1,105 @@
+# qwen4exp-swift15-w4dn8-i8-hc8m.recipe -- the Swift 1.5 finetune of Qwen3.8-Flash-Next
+# (ukisai/Swift1.5-Qwen3.8-Flash-Next, BF16 source, 335.3 GiB, key set identical to the official
+# base) as a radiance container for frank-primary: EVERY routed expert -- gate_up AND down --
+# through GPTQ into the w4nl64a8h table. The plugin's one-table-one-encoding law
+# ("one table of experts takes one encoding, and only a whole expert kept plain bf16 is served
+# beside it", qwen4exp_fp8 rad_arch_declare) forbids the original split tier (down block-fp8 +
+# gate_up w4): the served kernel is w4nl64a8h for the whole expert set, bf16 experts beside it.
+# The rest is qwen4exp-w5dn64-i8-hc8m.recipe verbatim: int8 trunk linears, block-fp8 hc mixers
+# and MTP fc, the fixed-scale n-gram table, the 2-bit draft head.
+#
+# Why down is the calibrated half: the calibration tap rides the down projection, so its GPTQ
+# rounding propagates against real down-input Hessians; gate_up shares the same w4 table
+# uncalibrated. The published w5dn64 comparison (mean KL 0.0789 -> 0.0717, top-1 92.85% ->
+# 93.34%, decode +4%, prefill -19%) is the 5-bit upper bound; this tier trades part of that
+# quality back for prefill.
+#
+#   CALIB=DIR rad-convert <Swift1.5-Qwen3.8-Flash-Next checkpoint> \
+#       --recipe scripts/recipes/qwen4exp-swift15-w4dn8-i8-hc8m.recipe -o qwen4exp-swift15-w4dn8-i8-hc8m.rad
+#
+# DIR is the calibration directory the bf16 container wrote (RADIANCE_CALIB_DIR, 10M tokens of
+# traffic-shaped corpus). Everything not named here is served as the checkpoint holds it (bf16:
+# embeddings, the vision tower, the QSA indexer); the first rule that matches a weight decides it.
+# The bf16 container is NOT passed with --reuse: the peak-disk arithmetic does not fit (checkpoint +
+# container + container + output > purge-freed space), and --reuse only saves a re-read.
+
+# The protected experts, cast to bf16 ahead of both expert rules. THE NAMES BELOW ARE THE BASE
+# CHECKPOINT'S (from its 1M-token recording, qwen4exp-w5dn64-i8-hc8m.recipe) -- a warm start only.
+# The Swift 1.5 finetune's routing distribution is not assumed equal: the derivation step rewrites
+# this section from Swift's OWN calibration recording before this recipe is ever run, by the base's
+# law -- an expert whose share of its layer's down-input energy is at least 3% and at least three
+# times its share of the routing, an even count on each rank's half of the layer (the four-bit GEMM
+# takes the rest as two equal tables by parity) by adding that half's next most energetic expert.
+# PROTECTED-BEGIN (rewritten from Swift's OWN grams by derive-protected-experts.py)
+blk.1.ffn_*_exps.271.weight         cast dtype=bf16
+blk.1.ffn_*_exps.355.weight         cast dtype=bf16
+blk.2.ffn_*_exps.279.weight         cast dtype=bf16
+blk.2.ffn_*_exps.413.weight         cast dtype=bf16
+blk.34.ffn_*_exps.146.weight         cast dtype=bf16
+blk.34.ffn_*_exps.153.weight         cast dtype=bf16
+blk.34.ffn_*_exps.407.weight         cast dtype=bf16
+blk.34.ffn_*_exps.453.weight         cast dtype=bf16
+blk.37.ffn_*_exps.288.weight         cast dtype=bf16
+blk.37.ffn_*_exps.377.weight         cast dtype=bf16
+blk.38.ffn_*_exps.406.weight         cast dtype=bf16
+blk.38.ffn_*_exps.420.weight         cast dtype=bf16
+blk.42.ffn_*_exps.37.weight         cast dtype=bf16
+blk.42.ffn_*_exps.90.weight         cast dtype=bf16
+blk.42.ffn_*_exps.302.weight         cast dtype=bf16
+blk.42.ffn_*_exps.388.weight         cast dtype=bf16
+blk.44.ffn_*_exps.315.weight         cast dtype=bf16
+blk.44.ffn_*_exps.350.weight         cast dtype=bf16
+blk.46.ffn_*_exps.290.weight         cast dtype=bf16
+blk.46.ffn_*_exps.419.weight         cast dtype=bf16
+blk.47.ffn_*_exps.119.weight         cast dtype=bf16
+blk.47.ffn_*_exps.143.weight         cast dtype=bf16
+blk.47.ffn_*_exps.399.weight         cast dtype=bf16
+blk.47.ffn_*_exps.445.weight         cast dtype=bf16
+# PROTECTED-END
+# (derive-protected-experts.py rewrote this section from Swift's own grams)
+
+# Every other routed expert, trunk and MTP layer alike: u4 codes into the w4nl table, rotated by
+# the 128-wide Hadamard, an E4M3 scale a group of 64 under the fixed 2^-13 searched for least
+# error as stored, chosen by GPTQ against $CALIB and swept three times by coordinate descent.
+# The calibration Grams are down-input Hessians, so down is the calibrated half of the table;
+# gate_up has no recording of its own and quantises uncalibrated (said on stderr, as the base
+# recipe says). The MTP layer has no recording either. Expert mass: gate_up 150.00 GiB -> ~37.6,
+# down 75.00 GiB -> ~18.8: ~56 GiB of experts, ~28 GiB a rank at TP=2 -- well inside the pool.
+blk.*.ffn_*_exps.*.weight            gptq table=w4nl group=64 scale=fp8_e4m3 scale2=f32 block2=*x* scale2_value=0.0001220703125 transform=fwht128 rule=search cd=3 calib=$CALIB
+
+# Every connection's two mixing matrices -- the 96 trunk connections, the final mixer and the MTP
+# head's three -- as E4M3 rows with an f32 scale a group of each row. A 128x128 block cannot cover
+# them: lowrank is 320, 2.5 blocks. Down is [320, 10240] and takes a scale every 128 columns; up is
+# [10240, 320] and takes four scales a row, 80 columns each.
+*_hc_down.weight                     rtn  codes=fp8_e4m3 group=128 scale=f32
+*_hc_up.weight                       rtn  codes=fp8_e4m3 group=80 scale=f32
+
+# The MTP head's fc_hidden and fc_embedding, the same rows. They only ever change the drafts.
+mtp.fc_hidden.weight                 rtn  codes=fp8_e4m3 group=128 scale=f32
+mtp.fc_embedding.weight              rtn  codes=fp8_e4m3 group=128 scale=f32
+
+# The trunk's block linears as int8, a scale per 128 columns of a row searched for least error.
+# The QSA indexer's projection is not one of them: its top-k is a hard choice between blocks, and
+# keeping it as the checkpoint holds it costs 20 MB.
+blk.*.attn_qg.weight                 rtn  codes=i8 group=128 scale=bf16 clamp=sym rule=search
+blk.*.attn_k.weight                  rtn  codes=i8 group=128 scale=bf16 clamp=sym rule=search
+blk.*.attn_v.weight                  rtn  codes=i8 group=128 scale=bf16 clamp=sym rule=search
+blk.*.attn_output.weight             rtn  codes=i8 group=128 scale=bf16 clamp=sym rule=search
+blk.*.ssm_inz.weight                 rtn  codes=i8 group=128 scale=bf16 clamp=sym rule=search
+blk.*.ssm_out.weight                 rtn  codes=i8 group=128 scale=bf16 clamp=sym rule=search
+blk.*.ffn_gate_up_shexp.weight       rtn  codes=i8 group=128 scale=bf16 clamp=sym rule=search
+blk.*.ffn_down_shexp.weight          rtn  codes=i8 group=128 scale=bf16 clamp=sym rule=search
+
+# The lm_head, int8 with a bf16 scale a row per 128 columns (logits_gemm_i8): at E4M3 it costs
+# ~0.6 points of top-1 on its own.
+output.weight                        rtn  codes=i8 group=128 scale=bf16 clamp=sym rule=search
+
+# The n-gram table at one fixed E4M3 scale; qwen4exp-w4.recipe says where the value comes from.
+# The value sits on a three-octave plateau (2^-13..2^-10, rel_l2 0.02662) with 1.7x headroom over
+# the base table's absmax; the prep phase verifies Swift's PLE absmax lands inside the plateau
+# before this rule is trusted.
+blk.*.ple_ngram.weight               rtn  codes=fp8_e4m3 block=*x* scale=bf16 rule=fixed scale_value=1.99317932128906e-4
+
+# The MTP head's 2-bit draft head. INERT for this checkpoint: the HF weight map has no
+# mtp.draft_head.weight (same as the base), kept for shape parity with the family.
+mtp.draft_head.weight                rtn  codes=u2 zero=u8 group=128 scale=f16

scripts/run-swift15-bf16.sh

@@ -0,0 +1,116 @@
+#!/usr/bin/env bash
+# run-swift15-bf16.sh — RUNS ON DANKSERVER. Boot the no-recipe bf16 Swift1.5
+# container for the two offline campaign phases. The engine owns the GPUs, so
+# the production unit must be stopped first (this script does it and restores
+# on exit, autoresearch pattern: stop, never mask — relinked unit).
+#
+#   kld    --kld-record the reference logits from the bf16 model
+#          (runs the corpus and EXITS; engine writes DIR and leaves)
+#   calib  serve with the calibration tap; a separate driver
+#          (scripts/calib-push.py) pushes the corpus; the engine writes the
+#          Hessians at RADIANCE_CALIB_TOKENS and the tap switches itself off.
+#          Stop this script after calib-push prints TAP_DONE.
+#
+# Shape = the vendor's bf16 calibration lines (GUIDE.md §4.3:528 + MOE-W4.md
+# "calibrate the checkpoint as it ships"): expert_tiered + weights-disk-tier,
+# no speculation, deterministic. Large prefill steps because a disk-tier step
+# costs the same at 1 token as at 32K. The container declared --max-ctx 4096;
+# --max-model-len 16384 exceeds it on purpose (the KLD prompts run to 12,007
+# tokens, calib to 8,242) — the boot ledger arbitrates the QSA buffers; if
+# declare refuses, re-convert with a wider --max-ctx.
+set -uo pipefail
+
+MODE="${1:?usage: run-swift15-bf16.sh kld|calib}"
+PIN_1013=sha256:4c3f484564bb2872784e46b0a59cbd71becc9c1e746e301a3596ffae253c322e
+PIN_120=sha256:665775ed1d90de6dbb25558e07fc60d20328f1346f96f3c6e00845eb763f01fa
+RAD=/home/dank/models/rad/qwen4exp-swift15-bf16.rad
+C=swift15ref
+LOG=/home/dank/logs/swift15_bf16.log
+KLD_REF_DIR=/home/dank/corpora/kld-ref
+CALIB_DIR=/home/dank/corpora/calib
+KLD_CORPUS=/home/dank/corpora/out/kld.jsonl
+
+restore() { sudo systemctl start frank-primary >/dev/null 2>&1 || true; }
+trap restore EXIT
+sudo systemctl stop frank-primary >/dev/null 2>&1 || true
+podman rm -f "$C" >/dev/null 2>&1 || true
+
+ARGS=(--model /models/rad/qwen4exp-swift15-bf16.rad
+      --tp 2 --placement expert_tiered --weights-disk-tier
+      --host-pool-mib 12288 --deterministic --gpu-headroom-mib 768
+      --num-speculative-tokens 0
+      --kv-cache-dtype fp8 --max-model-len 16384
+      --ngram-placement ram
+      --override-chat-template /models/tpl.jinja)
+
+case "$MODE" in
+  kld)
+    # 1.0.13: the engine that will serve the candidate — the reference is
+    # recorded on the same engine that scores it. The weights slab DERIVES as
+    # card - headroom - kv, and the disk tier refuses under 9.57 GiB (4.21 of
+    # layer buffers + 5.36 fixed), so kv is explicit SMALL: recording is
+    # prefill-only, self-batched at MNBT 32768 = two 16k prompts at a time;
+    # 4 slots x 16k fp8 (29 KiB/pos, 1.7 GiB) + headroom fits 3072.
+    IMAGE="stilldeadcode/radiance@$PIN_1013"
+    mkdir -p "$KLD_REF_DIR"
+    ARGS+=(--kld-record /data/kld-ref --kld-corpus /data/kld.jsonl
+           --vram-kv-mib 3072
+           --max-num-seqs 4 --max-num-batched-tokens 32768)
+    ;;
+  calib)
+    # 1.2.0: the calibration tap (RADIANCE_CALIB_DIR + RADIANCE_CALIB_TOKENS)
+    # lives in the 1.2.0 qwen4exp plugin; 1.0.13 has only the DIR half with an
+    # unparameterisable threshold. Grams are bf16 activations — engine-invariant.
+    # Same KV ceiling as kld: the disk-tier weights floor (9.57 GiB) plus
+    # ~18 GiB of fixed per-rank claims cap the pool near 3.5 GiB on a 32 GiB
+    # card — pool minus the 2.17 GiB fixed holds ~237k fp8 tokens a rank, so
+    # the vendor's big-step calibration shape is affordable: MOE-W4.md:244
+    # measures 640 tok/s at a 32k step against 930 at 55k, and the doc's own
+    # run drives MNBT 65536. 8 slots x 16k = 131k resident tokens fits the
+    # pool; the step batches 65k. KLD's observed rate confirmed step size,
+    # not slot count, is the throughput lever.
+    IMAGE="stilldeadcode/radiance@$PIN_120"
+    mkdir -p "$CALIB_DIR"
+    ARGS+=(--port 7770 --vram-kv-mib 3072
+           --max-num-seqs 8 --max-num-batched-tokens 65536
+           --prefix-cache-dir /data/calib-kvcache)
+    ;;
+  probe)
+    IMAGE="stilldeadcode/radiance@$PIN_1013"
+    ARGS+=(--port 7770 --max-num-seqs 8 --max-num-batched-tokens 4096)
+    ;;
+  *) echo "unknown mode: $MODE" >&2; exit 2 ;;
+esac
+
+# The prod template copy, readable from the container: mount the models tree
+# ro plus the state dir rw (KLD record + prefix tier land there).
+cp -f /home/dank/serverAppConfigs/scripts/qwen38-froggeric-v22.5.jinja \
+      /home/dank/models/tpl.jinja 2>/dev/null || true
+
+KLD_MOUNT=()
+if [ "$MODE" = kld ]; then
+    # Nested under the /data mount, so these must come after it (line order
+    # in podman run below): the corpus read-only, the record dir writable.
+    KLD_MOUNT=(-v "$KLD_CORPUS:/data/kld.jsonl:ro"
+               -v "$KLD_REF_DIR:/data/kld-ref")
+fi
+EXTRA_ENV=()
+if [ "$MODE" = calib ]; then
+    EXTRA_ENV=(-v "$CALIB_DIR:/data/calib"
+               -e RADIANCE_CALIB_DIR=/data/calib
+               -e RADIANCE_CALIB_TOKENS=10000000)
+fi
+
+podman run --rm --name "$C" \
+    --network host --init \
+    --security-opt seccomp=unconfined \
+    --ulimit memlock=-1:-1 \
+    --device /dev/kfd --device /dev/dri \
+    -v /home/dank/models:/models:ro \
+    -v /home/dank/radiance-state:/data \
+    "${KLD_MOUNT[@]}" \
+    "${EXTRA_ENV[@]}" \
+    "$IMAGE" "${ARGS[@]}" 2>&1 | tee "$LOG"
+rc=${PIPESTATUS[0]}
+echo "swift15-bf16 $MODE exited rc=$rc; log: $LOG"
+exit "$rc"

scripts/serve-frank-large.sh

@@ -1,30 +1,32 @@
 #!/usr/bin/env bash
-# serve-frank-large.sh — frank-large endpoint: halogen (closed-source
-# C++/HIP engine, image ghcr.io/peonist-ai/halogen, digest-pinned in
-# frank-large.env) serving peonist-ai/halogen-qwen3.8-27b on Dankclaw's
-# Strix Halo iGPU, host :7770. 2026-10-05 cutover from the ROCmFPX
-# llama.cpp shape — revert the swap commit's trio to roll back (payload
-# and fork build stay on disk).
+# serve-frank-large.sh — frank-large endpoint: halogen-flash-server 0.16.2
+# (closed-source engine, image ghcr.io/peonist-ai/halogen-flash-server,
+# digest-pinned in frank-large.env) serving the Quat3rnion abliterated
+# Swift1.5 finetune of peonist's Qwen3.8-Flash-Next v2 checkpoint on
+# Dankclaw's Strix Halo iGPU, host :7770. 2026-10-06 cutover from the
+# halogen 27B-dense shape — revert the swap commit's trio to roll back
+# (the 27B payload stays on disk).
 #
 # The engine is driven by `podman run` DIRECTLY (the fleet's frank-primary
-# pattern): the image entrypoint `halogen-docker` in default "all" mode
-# starts engine + OpenAI frontend in one container. The frontend listens on
-# 8731 inside (HALOGEN_API_PORT default) and is published as host 7770; the
-# engine's own token port (8730, no auth) stays unpublished on the
-# container's loopback. The image ships NO default checkpoint:
-# HALOGEN_CHECKPOINT selects the file inside the /models mount.
+# pattern): the image starts engine + OpenAI frontend in one container.
+# The frontend listens on 8731 inside (HALOGEN_API_PORT default) and is
+# published as host 7770; the engine's own token port (8730, no auth)
+# stays unpublished on the container's loopback. HALOGEN_CHECKPOINT
+# selects the file inside the /models mount; the n-gram lookup table
+# (HALOGEN_NGRAM_TABLE, the card sets it explicitly though the documented
+# fallback is "beside the checkpoint") and the vision tower
+# (HALOGEN_VISION_TOWER) ride in the same directory.
 #
 # Gates mirror the deployer EXACTLY (the 2026-10-01 crash-loop lesson —
 # serve gate and deploy gate disagreeing = green deploy, red unit):
-# tokenizer directory present with every named file a FLAT REGULAR FILE —
-# the model card ships `tokenizer/` as a flat directory and the engine
-# defaults HALOGEN_TOKENIZER to /tokenizer, so a HF cache snapshot's
-# symlinks into blobs/ dangle inside the mount; GPU device nodes present.
+# checkpoint/n-gram/vision byte-exact, tokenizer directory present with
+# every named file a FLAT REGULAR FILE (a HF cache snapshot's symlinks
+# into blobs/ dangle inside the mount), GPU device nodes present.
 #
-# Why containers instead of the bare binary (the old frank-large shape):
-# the payload is .hgn and only halogen reads it; halogen ships as a
-# container; the root unit keeps LimitMEMLOCK=infinity for parity and to
-# outlive the login session, exactly like frank-primary's.
+# Why containers instead of a bare binary: the payload is .hgn and only
+# the flash-server line reads it; it ships as a container; the root unit
+# keeps LimitMEMLOCK=infinity for parity and to outlive the login session,
+# exactly like frank-primary's.
 set -euo pipefail
 
 SCRIPT_DIR="$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)"
@@ -44,11 +46,16 @@ set +a
 : "${FRANK_LARGE_MODEL_DIR:?FRANK_LARGE_MODEL_DIR must name the payload dir}"
 : "${FRANK_LARGE_CHECKPOINT:?FRANK_LARGE_CHECKPOINT must name the .hgn inside it}"
 : "${FRANK_LARGE_CHECKPOINT_BYTES:?FRANK_LARGE_CHECKPOINT_BYTES must pin the size}"
+: "${FRANK_LARGE_NGRAM_TABLE:?FRANK_LARGE_NGRAM_TABLE must name the lookup table}"
+: "${FRANK_LARGE_NGRAM_BYTES:?FRANK_LARGE_NGRAM_BYTES must pin the table size}"
+: "${FRANK_LARGE_VISION_TOWER:?FRANK_LARGE_VISION_TOWER must name the tower}"
+: "${FRANK_LARGE_VISION_BYTES:?FRANK_LARGE_VISION_BYTES must pin the tower size}"
 : "${FRANK_LARGE_TOKENIZER_FILES:?FRANK_LARGE_TOKENIZER_FILES must list the tokenizer dir contents}"
-: "${HALOGEN_MODEL_ID:?HALOGEN_MODEL_ID must be the served alias (frank-large)}"
-: "${HALOGEN_SLOT_CTX:?HALOGEN_SLOT_CTX must be set}"
-: "${HALOGEN_KV_SLOTS:?HALOGEN_KV_SLOTS must be set (1 keeps the drafter on)}"
-: "${HALOGEN_DRAFTER:?HALOGEN_DRAFTER must be set (2 = DFlash2)}"
+: "${HALOGEN_MODEL_ID:?HALOGEN_MODEL_ID must be the served alias (frank-dragoon)}"
+: "${HALOGEN_CTX:?HALOGEN_CTX must be set (per-slot window)}"
+: "${HALOGEN_KV_POOL_POSITIONS:?HALOGEN_KV_POOL_POSITIONS must be set}"
+: "${HALOGEN_KV_SLOTS:?HALOGEN_KV_SLOTS must be set}"
+: "${HALOGEN_SPEC_ADAPT:?HALOGEN_SPEC_ADAPT must be set (0 = head always on)}"
 
 # Capture podman's output BEFORE grepping: under `set -o pipefail` the
 # `podman images | grep -q` pattern races (grep exits on first match,
@@ -61,36 +68,41 @@ if ! grep -qF "${FRANK_LARGE_IMAGE#*@}" <<<"$DIGESTS"; then
     exit 1
 fi
 
-# The checkpoint gate is size-exact: a truncated curl exits 0 on some
-# failures and a partial .hgn loads far enough to look alive, so the byte
-# count from the model card is the contract (the fleet fetch law).
-CKPT="$FRANK_LARGE_MODEL_DIR/$FRANK_LARGE_CHECKPOINT"
-if [ ! -f "$CKPT" ]; then
-    echo "frank-large: checkpoint $CKPT missing;" >&2
-    echo "  fetch it: bash scripts/download-frank-large-halogen.sh" >&2
-    exit 1
-fi
-CKPT_SIZE="$(stat -c %s "$CKPT")"
-if [ "$CKPT_SIZE" != "$FRANK_LARGE_CHECKPOINT_BYTES" ]; then
-    echo "frank-large: checkpoint is $CKPT_SIZE bytes, expected $FRANK_LARGE_CHECKPOINT_BYTES (partial download?)" >&2
-    exit 1
-fi
+# Size-exact gates on all three payload files: a truncated curl exits 0 on
+# some failures and a partial .hgn loads far enough to look alive, so the
+# byte counts from the Hub tree API are the contract (fleet fetch law).
+ck() {
+    # ck <env-name> <path> <expected-bytes>
+    if [ ! -f "$2" ]; then
+        echo "frank-large: $1 $2 missing;" >&2
+        echo "  fetch it: bash scripts/download-frank-large-flashnext.sh" >&2
+        exit 1
+    fi
+    local sz
+    sz="$(stat -c %s "$2")"
+    if [ "$sz" != "$3" ]; then
+        echo "frank-large: $1 is $sz bytes, expected $3 (partial download?)" >&2
+        exit 1
+    fi
+}
+ck checkpoint "$FRANK_LARGE_MODEL_DIR/$FRANK_LARGE_CHECKPOINT" "$FRANK_LARGE_CHECKPOINT_BYTES"
+ck n-gram-table "$FRANK_LARGE_MODEL_DIR/$FRANK_LARGE_NGRAM_TABLE" "$FRANK_LARGE_NGRAM_BYTES"
+ck vision-tower "$FRANK_LARGE_MODEL_DIR/$FRANK_LARGE_VISION_TOWER" "$FRANK_LARGE_VISION_BYTES"
 
-# The payload ships the tokenizer as a FLAT DIRECTORY (model card:
-# "tokenizer/ flat tokenizer directory"). A HuggingFace cache snapshot
-# symlinks each file into a sibling blobs/, and those symlinks dangle
-# inside the container mount — so the directory must be real and every
-# named file must be a regular file, not a link.
+# The payload ships the tokenizer as a FLAT DIRECTORY. A HuggingFace cache
+# snapshot symlinks each file into a sibling blobs/, and those symlinks
+# dangle inside the container mount — so the directory must be real and
+# every named file must be a regular file, not a link.
 TOK_DIR="$FRANK_LARGE_MODEL_DIR/tokenizer"
 if [ -L "$TOK_DIR" ] || [ ! -d "$TOK_DIR" ]; then
     echo "frank-large: tokenizer dir $TOK_DIR missing or a symlink (HF cache snapshot?);" >&2
-    echo "  fetch it: bash scripts/download-frank-large-halogen.sh" >&2
+    echo "  fetch it: bash scripts/download-frank-large-flashnext.sh" >&2
     exit 1
 fi
 for TF in $FRANK_LARGE_TOKENIZER_FILES; do
     if [ ! -f "$TOK_DIR/$TF" ] || [ -L "$TOK_DIR/$TF" ]; then
         echo "frank-large: tokenizer file $TOK_DIR/$TF missing or a symlink;" >&2
-        echo "  fetch it: bash scripts/download-frank-large-halogen.sh" >&2
+        echo "  fetch it: bash scripts/download-frank-large-flashnext.sh" >&2
         exit 1
     fi
 done
@@ -103,23 +115,55 @@ if [ ! -e "$DEV_KFD" ] || [ ! -d "$DEV_DRI" ]; then
     echo "frank-large: $DEV_KFD or $DEV_DRI missing — no gfx1151 to serve on" >&2
     exit 1
 fi
+# Host-RAM preflight — the engine's pin gate, checked BEFORE podman run.
+# The engine refuses to pin when MemAvailable is below the checkpoint plus
+# a 16 GiB floor: pinned pages cannot be swapped, and under the floor the
+# n-gram lookup pages in from disk on every long prompt. A RAM-short host
+# (a co-tenant spike, a sibling engine dying) crash-loops the unit — each
+# boot dies in ~5 s at the pin gate — until the RAM clears. The 2026-10-06
+# cutover hit this: 8 boots, MemAvailable 74.77 GiB vs the 78.11 GiB needed,
+# self-resolved in ~2 min when the dying 27B engine freed RAM. Check here and
+# WAIT for the RAM to clear, so a transient dip boots once instead of
+# crash-looping; if it never clears, refuse loudly.
+MEMINFO="${FRANK_LARGE_MEMINFO:-/proc/meminfo}"
+PIN_KB=$(( FRANK_LARGE_CHECKPOINT_BYTES / 1024 ))
+NEED_KB=$(( PIN_KB + 16 * 1048576 ))
+ram_avail_kb() { awk '/^MemAvailable:/{print $2; exit}' "$MEMINFO"; }
+MEMAVAIL_KB="$(ram_avail_kb)" || MEMAVAIL_KB=""
+if [ -n "$MEMAVAIL_KB" ] && [ "$MEMAVAIL_KB" -lt "$NEED_KB" ]; then
+    WAIT_S="${FRANK_LARGE_RAM_WAIT:-600}"
+    GIB() { awk -v kb="$1" 'BEGIN{printf "%.2f", kb/1048576}'; }
+    echo "frank-large: host RAM below the pin gate — MemAvailable $(GIB "$MEMAVAIL_KB") GiB < needed $(GIB "$NEED_KB") GiB ($(GIB "$PIN_KB") GiB checkpoint + 16.00 GiB floor); waiting up to ${WAIT_S}s for host RAM to clear" >&2
+    deadline=$(( $(date +%s) + WAIT_S ))
+    while [ "$(date +%s)" -lt "$deadline" ]; do
+        sleep 5
+        MEMAVAIL_KB="$(ram_avail_kb)" || MEMAVAIL_KB=""
+        if [ -n "$MEMAVAIL_KB" ] && [ "$MEMAVAIL_KB" -ge "$NEED_KB" ]; then
+            break
+        fi
+    done
+    if [ -z "$MEMAVAIL_KB" ] || [ "$MEMAVAIL_KB" -lt "$NEED_KB" ]; then
+        echo "frank-large: MemAvailable $(GIB "${MEMAVAIL_KB:-0}") GiB after ${WAIT_S}s; refusing to start (the engine would refuse the pin)" >&2
+        exit 1
+    fi
+    echo "frank-large: MemAvailable cleared to $(GIB "$MEMAVAIL_KB") GiB; starting" >&2
+fi
 for CTR in ${FRANK_LARGE_LEGACY_CONTAINERS:-} "$FRANK_LARGE_CONTAINER"; do
     podman rm -f "$CTR" >/dev/null 2>&1 || true
 done
 
 LOG="${FRANK_LARGE_LOG:-/home/dank/logs/frank_large.log}"
 mkdir -p "$(dirname "$LOG")"
-# for the udev-granted supplementary group, seccomp unconfined for the ROCm
-# ioctl surface, and unbounded memlock (the unit's LimitMEMLOCK=infinity is
-# inherited, so runc never has to RAISE it). NO --ulimit nofile: Dankclaw's
-# hard limit is 524288, rootless runc cannot raise a hard limit and dies
-# with "error setting rlimit type 7: operation not permitted" (measured
-# 2026-10-05 cutover), and 524288 fds is already ~100x what the engine
-# wants; the model card and both sibling serve scripts pass no nofile
-# either. No comment lines inside the backslash
-# chain — a mid-chain # terminates the continuation (the 2026-10-05
-# exit-127 crash loop on the sibling serve script).
-podman run -d --name "$FRANK_LARGE_CONTAINER" \
+# --ipc=host for the udev-granted supplementary group, seccomp unconfined
+# for the ROCm ioctl surface, and unbounded memlock (the unit's
+# LimitMEMLOCK=infinity is inherited, so runc never has to RAISE it). NO
+# --ulimit nofile: Dankclaw's hard limit is 524288, rootless runc cannot
+# raise a hard limit and dies with "error setting rlimit type 7:
+# operation not permitted" (measured 2026-10-05 cutover); the model card
+# and both sibling serve scripts pass no nofile either. No comment lines
+# inside the backslash chain — a mid-chain # terminates the continuation
+# (the 2026-10-05 exit-127 crash loop on the sibling serve script).
+podman run --name "$FRANK_LARGE_CONTAINER" \
   --publish "$FRANK_LARGE_PUBLISH" \
   --ipc=host \
   --ulimit memlock=-1:-1 \
@@ -128,19 +172,29 @@ podman run -d --name "$FRANK_LARGE_CONTAINER" \
   --security-opt seccomp=unconfined \
   -e HALOGEN_MODEL_ID="$HALOGEN_MODEL_ID" \
   -e HALOGEN_CHECKPOINT="/models/$FRANK_LARGE_CHECKPOINT" \
-  -e HALOGEN_SLOT_CTX="$HALOGEN_SLOT_CTX" \
-  -e HALOGEN_KV_SLOTS="${HALOGEN_KV_SLOTS:-1}" \
-  -e HALOGEN_DRAFTER="$HALOGEN_DRAFTER" \
+  -e HALOGEN_NGRAM_TABLE="/models/$FRANK_LARGE_NGRAM_TABLE" \
+  -e HALOGEN_VISION_TOWER="/models/$FRANK_LARGE_VISION_TOWER" \
+  -e HALOGEN_TOKENIZER=/tokenizer \
+  -e HALOGEN_CTX="$HALOGEN_CTX" \
+  -e HALOGEN_KV_POOL_POSITIONS="$HALOGEN_KV_POOL_POSITIONS" \
+  -e HALOGEN_KV_SLOTS="$HALOGEN_KV_SLOTS" \
+  -e HALOGEN_HOST_RESERVE_GIB="${HALOGEN_HOST_RESERVE_GIB:-20}" \
+  -e HALOGEN_SPEC_ADAPT="$HALOGEN_SPEC_ADAPT" \
+  -e HALOGEN_MAX_TOK="${HALOGEN_MAX_TOK:-8192}" \
   -e HALOGEN_MAX_TOKENS_CAP="${HALOGEN_MAX_TOKENS_CAP:-65536}" \
   -e HALOGEN_QUEUE_TIMEOUT="${HALOGEN_QUEUE_TIMEOUT:-7200}" \
-  ${HALOGEN_CACHE_MB:+-e HALOGEN_CACHE_MB="$HALOGEN_CACHE_MB"} \
+  -e HALOGEN_TEMPERATURE="${HALOGEN_TEMPERATURE:-1.0}" \
+  -e HALOGEN_TOP_P="${HALOGEN_TOP_P:-0.95}" \
+  -e HALOGEN_TOP_K="${HALOGEN_TOP_K:-20}" \
+  ${HALOGEN_WEIGHTS_LOCK:+-e HALOGEN_WEIGHTS_LOCK="$HALOGEN_WEIGHTS_LOCK"} \
   -v "$FRANK_LARGE_MODEL_DIR:/models:ro" \
   -v "$FRANK_LARGE_MODEL_DIR/tokenizer:/tokenizer:ro" \
   "$FRANK_LARGE_IMAGE" \
   > "$LOG" 2>&1
 
-# This script's lifetime IS the container's: podman wait returns the
-# container's exit code, so a crash is a nonzero ExecStart and the unit's
-# Restart=always is real. systemd stop goes through the unit's ExecStop
-# (podman wait alone cannot stop the child).
-exec podman wait "$FRANK_LARGE_CONTAINER"
+# FOREGROUND `podman run`: the engine's stdout/stderr are this script's
+# output, so the boot ledger (kv pool, prompt cache, MTP head, any clamp)
+# lands in $LOG as documented. podman run returns the container's exit
+# code, so a crash is a nonzero ExecStart and the unit's Restart=always
+# stays real; systemd stop goes through the unit's ExecStop and the run
+# client forwards SIGTERM to the engine, which drains on it.

scripts/serve-frank-primary.sh

@@ -1,65 +1,33 @@
 #!/usr/bin/env bash
-# serve-frank-primary.sh — frank-primary endpoint (tcclaviger RDNA4 vLLM fork,
-# orcarouter Qwen3.8-Flash-Next-Uncensored-FP8, 2x R9700,
-# dankserver :7770; 2026-09-29 cutover from the R9V radiance fork — git tag
-# frank-primary-r9v-q8-20260929 is the pre-cutover state).
+# serve-frank-primary.sh — frank-primary endpoint: radiance (StillDeadcode,
+# Apache-2.0) serving the OFFICIAL Qwen/Qwen3.8-Flash-Next-FP8 checkpoint at
+# NATIVE block-FP8 precision from a rad-convert container
+# (/home/dank/models/rad/qwen4exp-fp8-native.rad), on BOTH R9700s (TP=2,
+# wht6 wire), dankserver :7770. 2026-10-05 cutover from FreeToken (PR #587,
+# one card; git tag frank-primary-freetoken-pr587-20261005 is the
+# pre-cutover state).
 #
-# The engine is driven by `podman run` DIRECTLY: the image's entrypoint execs
-# `vllm serve "$@"`, so the run line below IS the engine argv. This replaces
-# the R9V launcher/shim machinery (that fork needed a docker->podman shim and
-# a 104-key on-host setup state; this stack is fully expressed by flags, and
-# every flag here was qualified by boot probe on 2026-09-29 — see
-# frank-primary.env for the qualification record).
-#
-# Shape (2026-10-03 shape F, on the 2026-10-02 shape E base): MAXLEN 200704 x 3
-# slots, fp8 KV, KV_BYTES
-# 3,150,000,000 PER RANK — the POOL, not the slot count, caps resident
-# conversations at ~2 full-length (measured: 398,658 tokens = 1.99x);
-# shorter requests use the spare blocks. --max-num-seqs stays 3 because SEQS=2
-# CANNOT boot this model
-# (torch.fx ConstraintViolationError on query_start_loc, measured at both
-# MNBT 1024 and 2048 — the 2x2 lives in frank-primary.env). Speculative decode
-# is OFF today by ENV KNOB, not by measurement: FRANK_PRIMARY_SPEC / _COMPCONFIG
-# / _GEN_CONFIG hold the whole JSON value and unset means no flag. The 2026-10-02
-# retirement of MTP compared the wrong pair — see frank-primary.env.
-# --max-num-batched-tokens 2048 (Frank's call 2026-10-03; arena-free because the
-# registry prices overhead flat up to its nbt 4096 anchor), MoE gather geometry
-# R4D_LRU_GATHER_CHUNKS=32 / R4D_LRU_GATHER_LANES=64 (+18.7% decode, grows
-# with depth), prefix caching OFF 2026-10-05 (the fork's QFN align-mode GDN
-# state fault — the evidence block is in frank-primary.env), expert prefetch
-# OFF (36% precision measured),
-# TunableOp off. Shape E adds --expert-precise-memory (fitted 5.25 GiB overhead
-# instead of the 7.60 GiB default -> expert arena 17.22 GiB / 7,522 slots, +16%)
-# and GPU_MAX_HW_QUEUES=2; shape F pins the plan's CUDA-graph term to the
-# measured capture cost (VLLM_CUDAGRAPH_MEM_BASE_MB / _PER_GRAPH_MB, container
-# env — the analytic default charges 0.55 GiB where capture took 0.31 GiB, so
-# the 0.24 GiB difference goes to the arena) and raises util 0.98 -> 0.982,
-# Frank's cap. The native-262144 and fp8+MTP4 shapes are retired to
-# git tag frank-primary-fp8-mtp-20261001.
-#
-# THE COMPILE CACHE IS MOUNTED, and that is a boot-cost fix, not a tuning knob:
-# the image ships no /root/.cache at all, our first boot writes 1.1 GB of vLLM
-# AOT/torch_compile artifacts plus 229 MB of comgr JIT kernels into the
-# container's ephemeral layer, and `podman rm -f` deletes them — so every
-# deploy/restart/reboot re-pays `torch.compile took 181.20 s in total` (173.19 s
-# of it one graph). Mounting FRANK_PRIMARY_CACHE_DIR at /root/.cache makes all
-# later boots load the compiled artifact instead, and hides nothing. PROVEN
-# LIVE 2026-10-02 on consecutive boots of this shape: cold `torch.compile took
-# 188.26 s`, warm `Directly load the compiled graph(s) for compile range
-# (1, 1024) from the cache, took 2.143 s` / `torch.compile took 3.97 s in
-# total` / `init engine … took 41.13 s (compilation: 3.98 s)`.
+# The run line below IS the engine argv: the image's entrypoint is the
+# engine and its tools sit beside it on PATH. Unlike every predecessor
+# there is no per-boot VRAM planner to second-guess: the boot ledger in the
+# log states every budget before anything is allocated (weights vs KV
+# split, the expert host pool's coverage of the worst case, the PLE pin,
+# the P2P answer per pair) — read it after every resize. First-boot shape:
+# 2 decode slots (Frank's call), 40 GiB/rank pinned expert pool, fp8 KV,
+# MTP depth 3, prefix cache host 4 GiB + disk 64 GiB, PLE pinned in RAM.
 #
 # The root unit (scripts/frank-primary.service, User=dank,
-# LimitMEMLOCK=infinity) grants the pin budget for the tiered-expert host
-# slabs (~20.5 GiB/rank of pinned anonymous memory). A plain shell session
-# also works, but the unit is the fleet-parity boot-persistent shape.
+# LimitMEMLOCK=infinity) grants the pin budget: 2 x 40 GiB expert pools +
+# the 47.7 GiB PLE table + 4 GiB prefix host tier — ~132 GiB of pinned
+# pages, which is ALSO why this script refuses to start when ttm's
+# pages_limit/pages_max cap is under FRANK_PRIMARY_TTM_MIN_MIB (see below).
 set -euo pipefail
 
 SCRIPT_DIR="$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)"
 ENV_FILE="${FRANK_PRIMARY_ENV_FILE:-$SCRIPT_DIR/frank-primary.env}"
 
 if [ ! -f "$ENV_FILE" ]; then
-    echo "serve-frank-primary: env file missing: $ENV_FILE" >&2
+    echo "ERROR: frank-primary env file absent: $ENV_FILE" >&2
     exit 1
 fi
 set -a
@@ -67,55 +35,120 @@ set -a
 source "$ENV_FILE"
 set +a
 
-: "${FRANK_PRIMARY_IMAGE:?FRANK_PRIMARY_IMAGE must name the digest-pinned image}"
-: "${FRANK_PRIMARY_MODEL_DIR:?FRANK_PRIMARY_MODEL_DIR must name the FP8 payload}"
-: "${FRANK_PRIMARY_MODEL_SHARDS:?FRANK_PRIMARY_MODEL_SHARDS must give the shard count}"
+: "${FRANK_PRIMARY_IMAGE:?FRANK_PRIMARY_IMAGE must name the radiance image}"
+: "${FRANK_PRIMARY_MODEL_DIR:?FRANK_PRIMARY_MODEL_DIR must name the .rad dir}"
+: "${FRANK_PRIMARY_MODEL_FILE:?FRANK_PRIMARY_MODEL_FILE must name the .rad file}"
+: "${FRANK_PRIMARY_STATE_DIR:?FRANK_PRIMARY_STATE_DIR must name the /data state dir}"
 : "${FRANK_PRIMARY_CONTAINER:?FRANK_PRIMARY_CONTAINER must name the container}"
 : "${FRANK_PRIMARY_SERVED_NAME:?FRANK_PRIMARY_SERVED_NAME must name the alias}"
-: "${FRANK_PRIMARY_PUBLISH:?FRANK_PRIMARY_PUBLISH must be host:container}"
-: "${FRANK_PRIMARY_MAXLEN:?FRANK_PRIMARY_MAXLEN must be set}"
-: "${FRANK_PRIMARY_SEQS:?FRANK_PRIMARY_SEQS must be set}"
-: "${FRANK_PRIMARY_KV_BYTES:?FRANK_PRIMARY_KV_BYTES must be set (PER RANK)}"
-: "${FRANK_PRIMARY_KV_DTYPE:?FRANK_PRIMARY_KV_DTYPE must be set (auto = bf16 KV)}"
-: "${FRANK_PRIMARY_REASONING_PARSER:?FRANK_PRIMARY_REASONING_PARSER must be set (qwen3)}"
-: "${FRANK_PRIMARY_TOOL_PARSER:?FRANK_PRIMARY_TOOL_PARSER must be set (qwen3_xml)}"
-: "${FRANK_PRIMARY_CHAT_TEMPLATE_KWARGS:?FRANK_PRIMARY_CHAT_TEMPLATE_KWARGS must be set}"
-# Capture podman's output BEFORE grepping: under `set -o pipefail`, the
-# classic `podman images | grep -q` pattern races — grep exits on its first
-# match, podman dies SIGPIPE (141) writing the remaining lines, and pipefail
-# turns the 141 into "no match" (the 2026-09-30 deploy refusal).
-DIGESTS="$(podman images --format '{{.RepoDigests}}')"
-if ! grep -qF "${FRANK_PRIMARY_IMAGE#*@}" <<<"$DIGESTS"; then
-    echo "serve-frank-primary: pinned image absent from rootless storage: $FRANK_PRIMARY_IMAGE" >&2
-    echo "  fetch it: podman pull $FRANK_PRIMARY_IMAGE" >&2
+: "${FRANK_PRIMARY_PORT:?FRANK_PRIMARY_PORT must be the port}"
+: "${FRANK_PRIMARY_TEMPLATE:?FRANK_PRIMARY_TEMPLATE must name the chat template}"
+: "${FRANK_PRIMARY_TP:?FRANK_PRIMARY_TP must be set}"
+: "${FRANK_PRIMARY_TP_WIRE:?FRANK_PRIMARY_TP_WIRE must be set}"
+: "${FRANK_PRIMARY_MAX_SEQS:?FRANK_PRIMARY_MAX_SEQS must be set}"
+: "${FRANK_PRIMARY_MAX_MODEL_LEN:?FRANK_PRIMARY_MAX_MODEL_LEN must be set}"
+: "${FRANK_PRIMARY_PLACEMENT:?FRANK_PRIMARY_PLACEMENT must be set}"
+: "${FRANK_PRIMARY_HOST_POOL_MIB:?FRANK_PRIMARY_HOST_POOL_MIB must be set}"
+: "${FRANK_PRIMARY_GPU_HEADROOM_MIB:?FRANK_PRIMARY_GPU_HEADROOM_MIB must be set}"
+: "${FRANK_PRIMARY_VRAM_WEIGHTS_MIB:?FRANK_PRIMARY_VRAM_WEIGHTS_MIB must be set}"
+: "${FRANK_PRIMARY_VRAM_KV_MIB:?FRANK_PRIMARY_VRAM_KV_MIB must be set}"
+: "${FRANK_PRIMARY_KV_DTYPE:?FRANK_PRIMARY_KV_DTYPE must be set}"
+: "${FRANK_PRIMARY_SPEC_TOKENS:?FRANK_PRIMARY_SPEC_TOKENS must be set}"
+: "${FRANK_PRIMARY_MAX_BATCHED_TOKENS:?FRANK_PRIMARY_MAX_BATCHED_TOKENS must be set}"
+: "${FRANK_PRIMARY_PREFIX_HOST_MIB:?FRANK_PRIMARY_PREFIX_HOST_MIB must be set}"
+: "${FRANK_PRIMARY_PREFIX_DISK_MIB:?FRANK_PRIMARY_PREFIX_DISK_MIB must be set}"
+: "${FRANK_PRIMARY_NGRAM_PLACEMENT:?FRANK_PRIMARY_NGRAM_PLACEMENT must be set}"
+: "${FRANK_PRIMARY_TTM_MIN_MIB:?FRANK_PRIMARY_TTM_MIN_MIB must be set}"
+: "${FRANK_PRIMARY_RAD_MIN_MIB:?FRANK_PRIMARY_RAD_MIN_MIB must be set}"
+
+# The image gate is the digest-pinned ref itself: a name@sha256 ref exists
+# in storage only if THAT manifest was pulled — a retagged look-alike or a
+# newer latest does not satisfy it. `podman image exists` takes a
+# name@digest ref in one command — no pipefail/grep race (the 2026-09-30
+# lesson). Runs as dank: the unit's process uses dank's ROOTLESS podman
+# storage (root unit, User=dank — the fleet's frank-primary shape).
+if ! podman image exists "$FRANK_PRIMARY_IMAGE"; then
+    echo "ERROR: radiance image absent: $FRANK_PRIMARY_IMAGE" >&2
+    echo "frank-primary: podman pull docker://stilldeadcode/radiance:latest (as dank)" >&2
     exit 1
 fi
 
-if [ ! -f "$FRANK_PRIMARY_MODEL_DIR/config.json" ]; then
-    echo "serve-frank-primary: payload config missing: $FRANK_PRIMARY_MODEL_DIR/config.json" >&2
+# The payload gate: the .rad container file, sized. A truncated convert
+# would otherwise buy a guaranteed long-timeout boot instead of this fast
+# failure. The staging dir / raw checkpoint are NOT this script's concern
+# (scripts/build-frank-primary-rad.sh built the file; its rad-info output
+# is the record).
+RAD="$FRANK_PRIMARY_MODEL_DIR/$FRANK_PRIMARY_MODEL_FILE"
+if [ ! -f "$RAD" ]; then
+    echo "ERROR: .rad container missing: $RAD" >&2
+    echo "frank-primary: run scripts/build-frank-primary-rad.sh" >&2
     exit 1
 fi
-SHARDS="$(find "$FRANK_PRIMARY_MODEL_DIR" -maxdepth 1 -name 'model-*-of-*.safetensors' | wc -l)"
-if [ "$SHARDS" != "$FRANK_PRIMARY_MODEL_SHARDS" ]; then
-    echo "serve-frank-primary: payload shard count wrong: $FRANK_PRIMARY_MODEL_DIR" >&2
-    echo "  found $SHARDS safetensors, want $FRANK_PRIMARY_MODEL_SHARDS" >&2
-    echo "  re-fetch: scripts/download-frank-primary.sh" >&2
+RAD_MIB=$(( $(stat -c %s "$RAD") / 1048576 ))
+if [ "$RAD_MIB" -lt "$FRANK_PRIMARY_RAD_MIN_MIB" ]; then
+    echo "ERROR: .rad container too small: $RAD is ${RAD_MIB} MiB, want >= $FRANK_PRIMARY_RAD_MIN_MIB" >&2
+    echo "frank-primary: the convert was truncated — rebuild it" >&2
     exit 1
 fi
-if [ ! -f "$FRANK_PRIMARY_MODEL_DIR/model.safetensors.index.json" ]; then
-    echo "serve-frank-primary: weight-map index missing: $FRANK_PRIMARY_MODEL_DIR/model.safetensors.index.json" >&2
+
+if [ ! -f "$FRANK_PRIMARY_TEMPLATE" ]; then
+    echo "ERROR: chat template missing: $FRANK_PRIMARY_TEMPLATE" >&2
     exit 1
 fi
 
+mkdir -p "$FRANK_PRIMARY_STATE_DIR/kvcache/frank-primary"
+# The engine's create is NOT recursive: with only the state dir mounted, its
+# "cannot create /data/kvcache/frank-primary: No such file or directory"
+# left the SSD prefix tier off (measured 2026-10-06 boot ledger). Pre-create
+# the full path on the host so the tier opens on first boot.
+
+mkdir -p "$FRANK_PRIMARY_STATE_DIR/tune"
+# rad-tune's kernel-selection cache (see the runbook's "Tuning levers"):
+# the engine's selector reads $RADIANCE_HOME/tune/<machine>.tune at boot;
+# $RADIANCE_HOME defaults to /opt/radiance, which lives in the image, so
+# the cache must ride a host mount or it dies with the --rm container
+# that wrote it. The dir is pre-created for the same reason as kvcache
+# (the engine's create is not recursive).
+
 if [ ! -e /dev/kfd ] || [ ! -d /dev/dri ]; then
-    echo "serve-frank-primary: GPU device nodes absent (/dev/kfd, /dev/dri)" >&2
+    echo "ERROR: GPU device nodes absent (/dev/kfd, /dev/dri)" >&2
+    exit 1
+fi
+
+# TTM preflight: the pinned ledger (~2 x 40 GiB pools + 47.7 GiB PLE + 4 GiB
+# prefix host = ~132 GiB) rides TTM's shared pages, capped by ttm's
+# pages_limit (the current kernel exposes NO pages_max; the cap function
+# takes the min over the knobs that exist). Below FRANK_PRIMARY_TTM_MIN_MIB
+# the load dies mid-pin with an allocator error instead of this named
+# refusal — raise the knob first (runtime sysfs write; kernel cmdline
+# fallback + one reboot only if the file is read-only), see the runbook.
+# RAISED 2026-10-05 pre-cutover to 52,428,800 pages = 200 GiB.
+ttm_cap_mib() {
+    # min over the knobs THIS kernel exposes — the current kernel has
+    # pages_limit only (no pages_max); a knob that is absent is skipped,
+    # not a zero (a zero would refuse a healthy cap).
+    local v cur=""
+    for p in /sys/module/ttm/parameters/pages_limit /sys/module/ttm/parameters/pages_max; do
+        [ -r "$p" ] || continue
+        v="$(cat "$p" 2>/dev/null)" || continue
+        [ -n "$v" ] || continue
+        v=$(( v * 4 / 1024 ))
+        if [ "$v" -lt "${cur:-999999999}" ]; then cur=$v; fi
+    done
+    echo "${cur:-0}"
+}
+TTM_CAP="$(ttm_cap_mib)"
+if [ "$TTM_CAP" -lt "$FRANK_PRIMARY_TTM_MIN_MIB" ]; then
+    echo "ERROR: ttm pages cap is ${TTM_CAP} MiB, the pinned ledger needs >= $FRANK_PRIMARY_TTM_MIN_MIB" >&2
+    echo "frank-primary: raise /sys/module/ttm/parameters/pages_limit (and pages_max if the kernel exposes one) to >= $FRANK_PRIMARY_TTM_MIN_MIB in 4 KiB pages" >&2
     exit 1
 fi
 
 # A crashed/killed unit can leave the container name registered (systemd
 # SIGKILLs conmon before cleanup runs). Clear OUR name plus the legacy
-# identities (the retired R9V container and the ad hoc experiment container
-# both hold these GPUs); all idempotent.
+# identities (the FreeToken-era frank-primary-qwen, the retired R9V
+# container and the ad hoc experiment containers all hold this GPU); all
+# idempotent.
 for CTR in $FRANK_PRIMARY_LEGACY_CONTAINERS "$FRANK_PRIMARY_CONTAINER"; do
     podman rm -f "$CTR" >/dev/null 2>&1 || true
 done
@@ -123,74 +156,51 @@ done
 LOG=/home/dank/logs/frank_primary.log
 mkdir -p "$(dirname "$LOG")"
 
-# Persistent compile cache (see the header). Create it before the mount so even
-# the first, cold boot has a target; unset -> no mount, exactly the old
-# ephemeral behaviour. It carries /root/.cache/vllm (AOT + torch_compile) and
-# /root/.cache/comgr (HIP kernel JIT).
-CACHE_DIR="${FRANK_PRIMARY_CACHE_DIR:-}"
-if [ -n "$CACHE_DIR" ]; then
-    mkdir -p "$CACHE_DIR"
-fi
-
-# --device /dev/dri is the WHOLE DIRECTORY (individual renderD nodes enumerate
-# without the card nodes and HIP refuses contexts — "No CUDA GPUs are
-# available" while device_count() says 2). --ipc=host for the engine's shared
-# memory; memlock unbounded for the pinned host slabs.
-# The entrypoint already execs `vllm serve "$@"`, so no `serve` here.
-# Prefix caching: ON — the shipped contract. FRANK_PRIMARY_NO_PREFIX_-
-# CACHING=1 flips to the fork-named mitigation (--no-enable-prefix-caching)
-# for the QFN align-mode GDN state fault; it was tried 2026-10-05 and
-# REVERTED the same day (live conversations still degraded at ~40k with
-# caching off and zero faults — see the evidence block in
-# frank-primary.env). NO comment lines inside the backslash chain below —
-# a comment there terminates the continuation and the next line runs as
-# its own command (the 2026-10-05 exit-127 crash loop).
-# The ${VAR:-word} / ${VAR:+word} double-expansion for the two directions is
-# AMBIGUOUS bash (${VAR:--flag} parses as a substring offset when set —
-# measured 2026-10-05: rendered "--no-enable-prefix-caching1"), so the flag
-# is branched explicitly: exactly one of the two reaches the engine.
-if [ -n "${FRANK_PRIMARY_NO_PREFIX_CACHING:-}" ]; then
-    FRANK_PRIMARY_PREFIX_CACHE_FLAG=--no-enable-prefix-caching
-else
-    FRANK_PRIMARY_PREFIX_CACHE_FLAG=--enable-prefix-caching
-fi
+# Publisher-vetted container contract (docs/DOCKER.md, compose/common.yaml):
+# --network host (the server binds the host's addresses directly, no NAT),
+# --init (the engine releases the cards on SIGTERM), seccomp=unconfined
+# (Docker/podman's default profile refuses io_uring, which the n-gram
+# table's reads go through, and the NUMA binding calls the ROCm runtime
+# makes for host memory), memlock unbounded (the pinned ledger — legal
+# because the root unit grants LimitMEMLOCK=infinity to this dank process),
+# /dev/kfd + /dev/dri (render/video groups are moot for a root container).
+# The template rides a single-file mount; the .rad dir is ro; /data holds
+# the prefix-cache disk tier.
 podman run -d --name "$FRANK_PRIMARY_CONTAINER" \
-  --publish "$FRANK_PRIMARY_PUBLISH" \
-  --ipc=host --ulimit memlock=-1:-1 \
-  --cap-add SYS_PTRACE \
-  --device /dev/kfd --device /dev/dri --device /dev/mem \
-  -e HIP_VISIBLE_DEVICES=0,1 -e VLLM_ROCM_USE_AITER=0 \
-  ${FRANK_PRIMARY_GPU_MAX_HW_QUEUES:+-e GPU_MAX_HW_QUEUES="$FRANK_PRIMARY_GPU_MAX_HW_QUEUES"} \
-  ${FRANK_PRIMARY_GATHER_CHUNKS:+-e R4D_LRU_GATHER_CHUNKS="$FRANK_PRIMARY_GATHER_CHUNKS"} \
-  ${FRANK_PRIMARY_GATHER_LANES:+-e R4D_LRU_GATHER_LANES="$FRANK_PRIMARY_GATHER_LANES"} \
-  ${FRANK_PRIMARY_CUDAGRAPH_BASE_MB:+-e VLLM_CUDAGRAPH_MEM_BASE_MB="$FRANK_PRIMARY_CUDAGRAPH_BASE_MB"} \
-  ${FRANK_PRIMARY_CUDAGRAPH_PER_GRAPH_MB:+-e VLLM_CUDAGRAPH_MEM_PER_GRAPH_MB="$FRANK_PRIMARY_CUDAGRAPH_PER_GRAPH_MB"} \
-  ${CACHE_DIR:+-v "$CACHE_DIR":/root/.cache} \
+  --network host \
+  --init \
+  --security-opt seccomp=unconfined \
+  --ulimit memlock=-1:-1 \
+  --device /dev/kfd --device /dev/dri \
   -v "$FRANK_PRIMARY_MODEL_DIR":/models:ro \
-  "$FRANK_PRIMARY_IMAGE" /models \
-  --served-model-name "$FRANK_PRIMARY_SERVED_NAME" --port 7770 \
-  --tensor-parallel-size 2 \
-  ${FRANK_PRIMARY_DCP:+--decode-context-parallel-size "$FRANK_PRIMARY_DCP"} \
-  --max-model-len "$FRANK_PRIMARY_MAXLEN" \
-  --max-num-seqs "$FRANK_PRIMARY_SEQS" \
-  ${FRANK_PRIMARY_MNBT:+--max-num-batched-tokens "$FRANK_PRIMARY_MNBT"} \
-  ${FRANK_PRIMARY_PRECISE_MEMORY:+--expert-precise-memory} \
-  ${FRANK_PRIMARY_EXPERT_OFFLOAD_MEM:+--expert-offload-mem "$FRANK_PRIMARY_EXPERT_OFFLOAD_MEM"} \
-  --gpu-memory-utilization "$FRANK_PRIMARY_UTIL" \
-  --kv-cache-memory-bytes "$FRANK_PRIMARY_KV_BYTES" \
+  -v "$FRANK_PRIMARY_STATE_DIR":/data \
+  -v "$FRANK_PRIMARY_STATE_DIR/tune":/opt/radiance/share/radiance/tune \
+  -v "$FRANK_PRIMARY_TEMPLATE":/templates/template.jinja:ro \
+  "$FRANK_PRIMARY_IMAGE" \
+  --model "/models/$FRANK_PRIMARY_MODEL_FILE" \
+  --host :: --port "$FRANK_PRIMARY_PORT" \
+  --served-model-name "$FRANK_PRIMARY_SERVED_NAME" \
+  --override-chat-template /templates/template.jinja \
+  --tp "$FRANK_PRIMARY_TP" --tp-wire "$FRANK_PRIMARY_TP_WIRE" \
+  --max-num-seqs "$FRANK_PRIMARY_MAX_SEQS" \
+  --max-model-len "$FRANK_PRIMARY_MAX_MODEL_LEN" \
+  --placement "$FRANK_PRIMARY_PLACEMENT" \
+  --host-pool-mib "$FRANK_PRIMARY_HOST_POOL_MIB" \
+  --gpu-headroom-mib "$FRANK_PRIMARY_GPU_HEADROOM_MIB" \
+  --vram-weights-mib "$FRANK_PRIMARY_VRAM_WEIGHTS_MIB" \
+  --vram-kv-mib "$FRANK_PRIMARY_VRAM_KV_MIB" \
   --kv-cache-dtype "$FRANK_PRIMARY_KV_DTYPE" \
-  "$FRANK_PRIMARY_PREFIX_CACHE_FLAG" \
-  --enable-auto-tool-choice --tool-call-parser "$FRANK_PRIMARY_TOOL_PARSER" \
-  --reasoning-parser "$FRANK_PRIMARY_REASONING_PARSER" \
-  --default-chat-template-kwargs "$FRANK_PRIMARY_CHAT_TEMPLATE_KWARGS" \
-  ${FRANK_PRIMARY_SPEC:+--speculative-config "$FRANK_PRIMARY_SPEC"} \
-  ${FRANK_PRIMARY_COMPCONFIG:+--compilation-config "$FRANK_PRIMARY_COMPCONFIG"} \
-  ${FRANK_PRIMARY_GEN_CONFIG:+--override-generation-config "$FRANK_PRIMARY_GEN_CONFIG"} \
-  --trust-remote-code \
+  --prefix-cache-host-mib "$FRANK_PRIMARY_PREFIX_HOST_MIB" \
+  --prefix-cache-dir /data/kvcache/frank-primary \
+  --prefix-cache-disk-mib "$FRANK_PRIMARY_PREFIX_DISK_MIB" \
+  --ngram-placement "$FRANK_PRIMARY_NGRAM_PLACEMENT" \
+  --num-speculative-tokens "$FRANK_PRIMARY_SPEC_TOKENS" \
+  --max-num-batched-tokens "$FRANK_PRIMARY_MAX_BATCHED_TOKENS" \
+  ${FRANK_PRIMARY_EXTRA_ARGS:+$FRANK_PRIMARY_EXTRA_ARGS} \
   > "$LOG" 2>&1
 
 # From here on, this script's lifetime IS the container's: wait returns the
 # container's exit code, so a crash is a nonzero ExecStart and the unit's
-# Restart=on-failure is real. (systemd stop goes through ExecStop in the
-# unit — podman wait alone cannot stop the container child.)
+# Restart=always is real. (systemd stop goes through ExecStop in the unit —
+# podman wait alone cannot stop the container child.)
 exec podman wait "$FRANK_PRIMARY_CONTAINER"

scripts/serve-frank-secondary.sh

@@ -0,0 +1,132 @@
+#!/usr/bin/env bash
+# Serve HauhauCS/Qwen3.5-9B-Uncensored-HauhauCS-Aggressive (Q6_K) as
+# frank-secondary on Dankclaw's RTX 3090 via llama.cpp (master build, CUDA sm_86).
+#
+# 2026-10-07 (Frank): Qwen3.5-9B swap at TWO 200,704-token conversations
+# (--ctx-size 401408 --parallel 2) with EXACT f16 KV — the 9B rung frees the
+# room the 26B MoE Gemma4 shape lacked, so quantized V retires. MTP stays off:
+# this GGUF carries no MTP tensors at all (verified from its tensor-name
+# list), so there is no in-file head to draft with. Verbosity 4 ships so the
+# VRAM-fit forensics (KV layout, buffer sizes, graph capture) land in the journal.
+#
+# Architecture: qwen35 — dense 9B, 32 layers with full_attention_interval 4
+# (8 full-attention layers at 3,7,...,31 + 24 Gated DeltaNet linear layers),
+# 16 heads / 4 KV heads × 256 dim, native 262,144, vocab 248,320. The KV pool
+# scales with the 8 full layers: 32 KiB/token at f16 K + f16 V (12.25 GiB at
+# the 401,408 pool); the DeltaNet layers cost a fixed recurrent state per
+# slot. Feasibility is priced from the MiMo-V2.6-Distill-Qwen-9B boots (same
+# geometry, same card, same shape): 21,276/2,849 at ubatch 1024, 21,846/2,279
+# at ubatch 2048 — the boot ledger, not arithmetic, is the referee.
+# Vision comes from the publisher's own BF16 projector (the model is natively
+# multimodal), so --mmproj is what gives this endpoint eyes; MMPROJ_FILE="" is
+# the documented text-only opt-out.
+#
+# Sampling: llama.cpp takes it per request (the fleet harness sends the
+# publisher recipe from models.yml: thinking temp 0.6 / top_p 0.95 / top_k 20).
+#
+# Tunables come from frank-secondary.env unless FRANK_SECONDARY_ENV_FILE overrides.
+set -euo pipefail
+
+SCRIPT_DIR="$(cd -- "$(dirname -- "${BASH_SOURCE[0]}")" && pwd)"
+ENV_FILE="${FRANK_SECONDARY_ENV_FILE:-$SCRIPT_DIR/frank-secondary.env}"
+if [[ -f "$ENV_FILE" ]]; then
+    set -a
+    # shellcheck disable=SC1090
+    source "$ENV_FILE"
+    set +a
+fi
+
+# --- defaults (used when the env file does not set them) -------------------
+: "${LLAMA_BIN:=/home/dank/GitPackages/llama-master/build-cuda/bin/llama-server}"
+: "${MODEL_FILE:=/home/dank/models/Qwen35-9B-HauhauCS-Q6K/Qwen3.5-9B-Uncensored-HauhauCS-Aggressive-Q6_K.gguf}"
+: "${ALIAS:=frank-zealot}"
+: "${BIND_ADDR:=0.0.0.0}"
+: "${LISTEN_PORT:=7771}"
+# ctx-size is the POOL llama.cpp divides across slots: 401,408 / 2 = 200,704
+# per conversation, inside the model's trained 262,144. The harness
+# contextWindow MUST quote the PER-SLOT number — a longer prompt is answered
+# with a 400, never truncated.
+: "${CTX_SIZE:=401408}"
+: "${PARALLEL:=2}"
+# Compute-buffer geometry. Ubatch is a prefill knob (decode never moves).
+# 2048 is the measured CUDA optimum (MiMo era, same arch/shape: +4.6 % prefill
+# at 8.9k depth, decode flat, 616 MiB). The logical batch is VRAM- and
+# throughput-neutral at a fixed ubatch.
+: "${BATCH_SIZE:=4096}"
+: "${UBATCH_SIZE:=2048}"
+# Exact (unquantized) KV: f16 K + f16 V, restoring the MiMo-era contract that
+# the 26B Gemma4 shape could not afford. 32 KiB/token at this geometry
+# (8 full-attention layers × 4 KV heads × 256 dim); 12.25 GiB at the 401,408
+# pool. Explicit either way so the contract is visible in the argv, and
+# --flash-attn stays required for any quantized-V fallback (q8_0 = llama.cpp's
+# FP8 class; no true fp8 KV dtype exists in this build).
+: "${CACHE_TYPE_K:=f16}"
+: "${CACHE_TYPE_V:=f16}"
+# Speculative decoding is OFF: this Q6_K file carries NO nextn/mtp tensors, so
+# there is no in-file head (the law: MTP is a property of the file, not of the
+# flags). SPEC_TYPE=none strips BOTH speculative flags (flags move as a set);
+# naming a drafter requires SPEC_DRAFT_FILE alongside SPEC_TYPE — the Gemma4
+# assistant on disk is arch gemma4-assistant and cannot drafter-match qwen35.
+: "${SPEC_TYPE:=none}"
+: "${SPEC_DRAFT_N_MAX:=3}"
+: "${SPEC_DRAFT_FILE:=}"
+: "${NGL:=99}"
+# Log verbosity threshold (llama.cpp -lv). 3 is the engine default (info);
+# 4 adds debug detail (KV layout, buffer sizes, graph capture) which is what
+# the VRAM-fit forensics need. Levels above 4 flood the journal.
+: "${VERBOSITY:=3}"
+# Vision projector. No colon on purpose: an explicitly empty value means
+# "text-only, on purpose" and must NOT be replaced by the default path.
+: "${MMPROJ_FILE=/home/dank/models/Qwen35-9B-HauhauCS-Q6K/mmproj-Qwen3.5-9B-Uncensored-HauhauCS-Aggressive-BF16.gguf}"
+
+# Fail loudly rather than half-loading: a missing file is a download that
+# has not finished, and Restart=always would turn it into a crash loop that
+# reads like a runtime fault.
+for path in "$LLAMA_BIN" "$MODEL_FILE"; do
+    [[ -e "$path" ]] || { echo "frank-secondary: missing $path" >&2; exit 1; }
+done
+EXTRA_ARGS=()
+if [[ -n "$MMPROJ_FILE" ]]; then
+    [[ -e "$MMPROJ_FILE" ]] || { echo "frank-secondary: missing $MMPROJ_FILE" >&2; exit 1; }
+    EXTRA_ARGS+=(--mmproj "$MMPROJ_FILE")
+fi
+
+# Speculative flags move as a set: they reach the argv ONLY when a drafter
+# is named. SPEC_TYPE=none (the shipped contract) strips both — a no-spec
+# baseline must not carry half a spec pair.
+if [[ -n "$SPEC_TYPE" && "$SPEC_TYPE" != "none" ]]; then
+    EXTRA_ARGS+=(--spec-type "$SPEC_TYPE" --spec-draft-n-max "$SPEC_DRAFT_N_MAX")
+if [[ -n "$SPEC_DRAFT_FILE" ]]; then
+    [[ -e "$SPEC_DRAFT_FILE" ]] || { echo "frank-secondary: missing $SPEC_DRAFT_FILE" >&2; exit 1; }
+    EXTRA_ARGS+=(--spec-draft-model "$SPEC_DRAFT_FILE")
+fi
+fi
+
+# The UVM device node is created lazily by nvidia-modprobe on the first CUDA
+# open after boot; make sure it exists before the server needs it.
+nvidia-modprobe -u -c=0 2>/dev/null || true
+
+# Flash attention stays pinned on: it is REQUIRED for quantized V (q8_0),
+# and the build carries GGML_CUDA_FA_ALL_QUANTS=ON (since 2026-09-17), so
+# the q8_0 KV contract in frank-secondary.env is a real path — without this
+# flag or that build option the engine silently falls back and the
+# contract becomes a lie.
+ARGS=(
+    --model "$MODEL_FILE"
+    --alias "$ALIAS"
+    --host "$BIND_ADDR"
+    --port "$LISTEN_PORT"
+    --ctx-size "$CTX_SIZE"
+    --parallel "$PARALLEL"
+    --batch-size "$BATCH_SIZE"
+    --ubatch-size "$UBATCH_SIZE"
+    --n-gpu-layers "$NGL"
+    --flash-attn on
+    --cache-type-k "$CACHE_TYPE_K"
+    --cache-type-v "$CACHE_TYPE_V"
+    --log-verbosity "$VERBOSITY"
+    --jinja
+    --reasoning auto
+    --reasoning-format deepseek
+)
+exec "$LLAMA_BIN" "${ARGS[@]}" ${EXTRA_ARGS[@]+"${EXTRA_ARGS[@]}"}

scripts/serve-frank-small.sh

@@ -1,99 +0,0 @@
-#!/usr/bin/env bash
-# Serve ornith-ai/Ornith-1.5-9B-GGUF (Q8_0) as frank-small on
-# Dankclaw's RTX 3090 via llama.cpp (master build, CUDA sm_86).
-#
-# 2026-10-01 night (Frank): swapped from SGLang + INT8-W8A16-AutoRound
-# safetensors to the llama.cpp GGUF path, because llama.cpp is the only
-# runtime on this host that moves K and V independently — the asymmetric
-# f16-K + q8_0-V contract that endpoint was built for. 2026-10-03 (Frank):
-# the pair is now f16/f16 — two wide slots freed the VRAM, so the values are
-# unquantized too. The per-tensor flags stay explicit either way.
-#
-# Architecture: qwen3_5 hybrid — 32 layers, 8 full-attention + 24 Gated
-# DeltaNet, native 262144 window. KV lives only in the 8 attention layers;
-# the DeltaNet state is per-slot fixed size. Vision comes from the repo's
-# separate projector (the Q6_K GGUF carries no `v.*` tensors at all), so
-# --mmproj is what gives this endpoint eyes; MMPROJ_FILE="" is the documented
-# text-only opt-out.
-#
-# Sampling: llama.cpp takes it per request (the fleet harness sends its
-# recipe from models.yml); greedy is the harness default.
-#
-# Tunables come from frank-small.env unless FRANK_SMALL_ENV_FILE overrides.
-set -euo pipefail
-
-SCRIPT_DIR="$(cd -- "$(dirname -- "${BASH_SOURCE[0]}")" && pwd)"
-ENV_FILE="${FRANK_SMALL_ENV_FILE:-$SCRIPT_DIR/frank-small.env}"
-if [[ -f "$ENV_FILE" ]]; then
-    set -a
-    # shellcheck disable=SC1090
-    source "$ENV_FILE"
-    set +a
-fi
-
-# --- defaults (used when the env file does not set them) -------------------
-: "${LLAMA_BIN:=/home/dank/GitPackages/llama-master/build-cuda/bin/llama-server}"
-: "${MODEL_FILE:=/home/dank/models/Ornith-Q8/Ornith-1.5-9B-Q8_0.gguf}"
-: "${ALIAS:=frank-small}"
-: "${BIND_ADDR:=0.0.0.0}"
-: "${LISTEN_PORT:=7771}"
-# ctx-size is the POOL llama.cpp divides across slots: 401408 / 2 gives
-# 200,704 per slot — two independent conversations, each one wide enough to
-# hold a whole frank-primary-sized prompt in a single shot (the native 262,144
-# no longer fits twice on a 24 GiB card once the vision tower is loaded;
-# frank-small.env carries the three measurements that set this number).
-: "${CTX_SIZE:=401408}"
-: "${PARALLEL:=2}"
-# Compute-buffer geometry. Before 2026-10-03 this endpoint never passed either
-# flag, so it ran on llama.cpp's own defaults (2048/512). The pair is 4096/2048
-# since 2026-10-05: on this CUDA card the prefill curve climbs all the way to
-# ubatch 2048 (unlike the R9700 layer-split sibling, where 512 is the peak),
-# and the logical batch is VRAM- and throughput-neutral. frank-small.env
-# carries the sweep tables and the VRAM risk note.
-: "${BATCH_SIZE:=4096}"
-: "${UBATCH_SIZE:=2048}"
-: "${CACHE_TYPE_K:=f16}"
-: "${CACHE_TYPE_V:=f16}"
-: "${NGL:=99}"
-# Vision projector. No colon on purpose: an explicitly empty value means
-# "text-only, on purpose" and must NOT be replaced by the default path.
-: "${MMPROJ_FILE=/home/dank/models/Ornith-Q8/mmproj-Ornith-1.5-9B-BF16.gguf}"
-
-# Fail loudly rather than half-loading: a missing file is a download that
-# has not finished, and Restart=always would turn it into a crash loop that
-# reads like a runtime fault.
-for path in "$LLAMA_BIN" "$MODEL_FILE"; do
-    [[ -e "$path" ]] || { echo "frank-small: missing $path" >&2; exit 1; }
-done
-EXTRA_ARGS=()
-if [[ -n "$MMPROJ_FILE" ]]; then
-    [[ -e "$MMPROJ_FILE" ]] || { echo "frank-small: missing $MMPROJ_FILE" >&2; exit 1; }
-    EXTRA_ARGS+=(--mmproj "$MMPROJ_FILE")
-fi
-
-# The UVM device node is created lazily by nvidia-modprobe on the first CUDA
-# open after boot; make sure it exists before the server needs it.
-nvidia-modprobe -u -c=0 2>/dev/null || true
-
-# Flash attention stays pinned on: it is the fast path for f16 KV, and it is
-# the only path that could serve quantized V at all — so if CACHE_TYPE_V is
-# ever set back to q8_0 this flag must not be lost, or the build falls back
-# silently and the KV contract in frank-small.env becomes a lie.
-ARGS=(
-    --model "$MODEL_FILE"
-    --alias "$ALIAS"
-    --host "$BIND_ADDR"
-    --port "$LISTEN_PORT"
-    --ctx-size "$CTX_SIZE"
-    --parallel "$PARALLEL"
-    --batch-size "$BATCH_SIZE"
-    --ubatch-size "$UBATCH_SIZE"
-    --n-gpu-layers "$NGL"
-    --flash-attn on
-    --cache-type-k "$CACHE_TYPE_K"
-    --cache-type-v "$CACHE_TYPE_V"
-    --jinja
-    --reasoning auto
-    --reasoning-format deepseek
-)
-exec "$LLAMA_BIN" "${ARGS[@]}" ${EXTRA_ARGS[@]+"${EXTRA_ARGS[@]}"}

scripts/stack-qwen38-official-fp8.py

@@ -0,0 +1,556 @@
+#!/usr/bin/env python3
+"""stack-qwen38-official-fp8.py — reshape the official Qwen/Qwen3.8-Flash-Next-FP8
+checkpoint into the form the shipped radiance qwen4exp plugin maps.
+
+Run INSIDE the staging dir's tree (python3 + numpy on the host; data never
+leaves the box). Two transforms, both VALUE-PRESERVING:
+
+1. EXPERTS: the shipped plugin's stacked layout names ONE tensor per family
+   (`mlp.experts.gate_up_proj` [E, 2F, H], `mlp.experts.down_proj` [E, H, F])
+   plus a `<family>_scale_inv` plane at the same index (NO `.weight` in the
+   stacked name — scale_name() appends _scale_inv to the tensor name verbatim),
+   while the official FP8 repo ships per-expert unfused tensors
+   (`experts.<e>.{gate,up,down}_proj.weight` + scale). Stacking is pure byte
+   concatenation along dim 0 — gate rows
+   first inside each expert's gate_up slab, which is the order the block
+   wants (rad_block_moe_fp8.h: "a stacked gate_up is ALREADY FUSED in the
+   order this block wants"). No requantisation: bytes are moved, not decoded.
+
+2. PLE TABLE: the plugin's fp8 branch wants the table's codes beside a
+   ONE-ELEMENT bf16 scale plane (`blk.%d.ple_ngram.scale`, view {1}); a
+   per-shard-blocked scale can never fit that view, and `rtn` re-encoding the
+   RAW scaleless codes would overflow (v / 1.99e-4 exceeds e4m3's 448 max).
+   So staging decodes the scaleless shards with the checkpoint's OWN bf16
+   scale into a bf16 table (exact: every e4m3 code value and the scale are
+   exactly representable in bf16 to within the code's own 2^-3 grid), and the
+   convert's rtn rule re-encodes at that same scale — the nearest-code
+   distance is 2^-3 while the bf16 rounding perturbs by 2^-9, so every
+   original code is restored exactly and the container gets the checkpoint's
+   own codes plus the one-element scale plane.
+
+The convert then runs with NO recipe except:
+  blk.*.ple_ngram.weight=rtn:codes=fp8_e4m3,block=*x*,scale=bf16,rule=fixed,scale_value=<s>
+with <s> read from the checkpoint and written to ple_scale.txt.
+
+Idempotent: re-runs skip when the staged index already carries the stacked
+names, the merged bf16 shard_0, and split_ngram_parts == 1. --force redoes.
+"""
+
+import argparse
+import json
+import mmap
+import multiprocessing as mp
+import os
+import re
+import struct
+import sys
+import time
+import numpy as np
+F8, BF16 = "F8_E4M3", "BF16"
+
+
+def log(msg):
+    print(msg, flush=True)
+
+
+# ---------------------------------------------------------------- safetensors
+
+def parse_header(path):
+    """Return (header_len, {name: (dtype, shape, data_offsets)})."""
+    with open(path, "rb") as fh:
+        (hlen,) = struct.unpack("<Q", fh.read(8))
+        hdr = json.loads(fh.read(hlen))
+    return hlen, hdr
+
+
+def write_safetensors(path, tensors, data_iter):
+    """tensors: [(name, dtype, shape, nbytes)]; data_iter yields bytes per tensor."""
+    hdr = {}
+    off = 0
+    for name, dtype, shape, nbytes in tensors:
+        hdr[name] = {"dtype": dtype, "shape": list(shape),
+                     "data_offsets": [off, off + nbytes]}
+        off += nbytes
+    blob = json.dumps(hdr, separators=(",", ":")).encode("utf-8")
+    pad = (-len(blob)) % 8  # keep the data start 8-byte aligned
+    blob += b" " * pad
+    with open(path, "wb") as fh:
+        fh.write(struct.pack("<Q", len(blob)))
+        fh.write(blob)
+        for _, _, _, nbytes in tensors:
+            fh.write(next(data_iter).tobytes())
+        fh.flush()
+        os.fsync(fh.fileno())
+    return 8 + len(blob) + sum(t[3] for t in tensors)
+
+
+# ------------------------------------------------------------------- preflight
+
+def build_meta(stage, index):
+    """name -> (file, global_byte_offset, nbytes, dtype, shape) for every tensor."""
+    files = sorted(set(index["weight_map"].values()))
+    meta = {}
+    for f in files:
+        path = os.path.join(stage, f)
+        hlen, hdr = parse_header(path)
+        for name, e in hdr.items():
+            if name == "__metadata__":
+                continue
+            b0, b1 = e["data_offsets"]
+            meta[name] = (f, 8 + hlen + b0, b1 - b0, e["dtype"], e["shape"])
+    return meta
+
+
+def e4m3_lut():
+    lut = np.zeros(256, np.float32)
+    for b in range(256):
+        sign = -1.0 if b & 0x80 else 1.0
+        e = (b >> 3) & 0xF
+        m = b & 0x7
+        if e == 15 and m == 7:
+            v = np.float32(np.nan)  # OCP e4m3fn: only 0x7F/0xFF are NaN
+        elif e == 0:
+            v = sign * (m / 8.0) * 2.0 ** -6
+        else:
+            v = sign * (1.0 + m / 8.0) * 2.0 ** (e - 7)
+        lut[b] = v
+    assert lut[0x38] == 1.0 and lut[0x7E] == 448.0
+    assert np.isnan(lut[0x7F]) and lut[0x00] == 0.0
+    assert lut[0x08] == 2.0 ** -6 and lut[0x80] == 0.0
+    return lut
+
+
+def f32_to_bf16_rne(v):
+    """float32 array -> uint16 bf16 bits, round-to-nearest-even."""
+    q = np.ascontiguousarray(v, dtype=np.float32)
+    q[~np.isfinite(q)] = 0.0
+    bits = q.view(np.uint32)
+    r = bits + 0x7FFF + ((bits >> 16) & 1)
+    return (r >> 16).astype(np.uint16)
+
+
+# ------------------------------------------------------------------- workers
+
+def stack_prefix(job):
+    """Byte-stack one layer's 512 experts into a 4-tensor safetensors file."""
+    (stage, pre, idx, n_e, shapes, meta) = job
+    (f2, h) = shapes["ffn2_h"]          # 2*ffn, n_embd
+    (f,) = shapes["ffn"]                # ffn
+    gs, ds = shapes["gs"], shapes["ds"] # scale shapes [5,20] and [20,5]
+
+    # byte widths: fp8 is 1 B/element, bf16 2 B; the fused gate_up scale is
+    # the gate and up scale planes concatenated (gate rows first)
+    g_bytes = (f2 // 2) * h
+    gs_bytes = gs[0] * gs[1] * 2          # ONE projection's scale plane
+    gus_bytes = 2 * gs_bytes              # fused scale per expert
+    dn_bytes = h * f
+    ds_bytes = ds[0] * ds[1] * 2
+
+    gu = np.empty(n_e * f2 * h, np.uint8)
+    gus = np.empty(n_e * gus_bytes, np.uint8)
+    dn = np.empty(n_e * dn_bytes, np.uint8)
+    dns = np.empty(n_e * ds_bytes, np.uint8)
+
+    mmaps = {}
+
+    def view(name):
+        f_, off, ln, dt, shp = meta[name]
+        if f_ not in mmaps:
+            fh = open(os.path.join(stage, f_), "rb")
+            mmaps[f_] = mmap.mmap(fh.fileno(), 0, access=mmap.ACCESS_READ)
+        return memoryview(mmaps[f_])[off:off + ln], (dt, shp)
+
+    def need(name, dt, shp):
+        mv, (got_dt, got_shp) = view(name)
+        assert got_dt == dt and list(got_shp) == list(shp), f"{name}: {got_dt}{got_shp}"
+        return mv
+
+    for e in range(n_e):
+        base = f"{pre}.mlp.experts.{e}"
+        slab = gu[e * f2 * h:(e + 1) * f2 * h]
+        slab[:g_bytes] = need(f"{base}.gate_proj.weight", F8, [f2 // 2, h])
+        slab[g_bytes:] = need(f"{base}.up_proj.weight", F8, [f2 // 2, h])
+        sslab = gus[e * gus_bytes:(e + 1) * gus_bytes]
+        sslab[:gs_bytes] = need(f"{base}.gate_proj.weight_scale_inv", BF16, gs)
+        sslab[gs_bytes:] = need(f"{base}.up_proj.weight_scale_inv", BF16, gs)
+        dn[e * dn_bytes:(e + 1) * dn_bytes] = need(f"{base}.down_proj.weight", F8, [h, f])
+        dns[e * ds_bytes:(e + 1) * ds_bytes] = need(f"{base}.down_proj.weight_scale_inv", BF16, ds)
+
+    for mm in mmaps.values():
+        mm.close()
+
+    # byte-identity check on two sampled experts (first and last), all four
+    # tensors — the stack IS the checkpoint bytes, so they must compare equal
+    for e in (0, n_e - 1):
+        base = f"{pre}.mlp.experts.{e}"
+        s = e * f2 * h
+        assert bytes(gu[s:s + g_bytes]) == bytes(stack_src(f"{base}.gate_proj.weight", meta, stage))
+        assert bytes(gu[s + g_bytes:s + f2 * h]) == bytes(stack_src(f"{base}.up_proj.weight", meta, stage))
+        t = e * gus_bytes
+        assert bytes(gus[t:t + gs_bytes]) == bytes(stack_src(f"{base}.gate_proj.weight_scale_inv", meta, stage))
+        assert bytes(gus[t + gs_bytes:t + gus_bytes]) == bytes(stack_src(f"{base}.up_proj.weight_scale_inv", meta, stage))
+        d = e * dn_bytes
+        assert bytes(dn[d:d + dn_bytes]) == bytes(stack_src(f"{base}.down_proj.weight", meta, stage))
+        u = e * ds_bytes
+        assert bytes(dns[u:u + ds_bytes]) == bytes(stack_src(f"{base}.down_proj.weight_scale_inv", meta, stage))
+
+    out = os.path.join(stage, f"stacked.{idx:03d}.safetensors")
+    tensors = [
+        (f"{pre}.mlp.experts.gate_up_proj", F8, [n_e, f2, h], n_e * f2 * h),
+        (f"{pre}.mlp.experts.gate_up_proj_scale_inv", BF16,
+         [n_e, gs[0] * 2, gs[1]], n_e * gus_bytes),
+        (f"{pre}.mlp.experts.down_proj", F8, [n_e, h, f], n_e * h * f),
+        (f"{pre}.mlp.experts.down_proj_scale_inv", BF16,
+         [n_e, ds[0], ds[1]], n_e * ds_bytes),
+    ]
+    bufs = iter([gu, gus, dn, dns])
+    written = write_safetensors(out, tensors, bufs)
+    return (pre, written)
+
+
+def stack_src(name, meta, stage):
+    """Re-open a source tensor for the sampled-expert check."""
+    f_, off, ln, dt, shp = meta[name]
+    fh = open(os.path.join(stage, f_), "rb")
+    mm = mmap.mmap(fh.fileno(), 0, access=mmap.ACCESS_READ)
+    fh.close()  # mmap keeps the mapping alive
+    return mm[off:off + ln]
+
+
+def fill_ple_shard(job):
+    """Decode one fp8 shard with the table scale, write bf16 rows into the mmap."""
+    (src_path, src_off, src_rows, dst_path, dst_off, scale_f32,
+     cols, chunk_rows) = job
+    lut = e4m3_lut()
+    with open(src_path, "rb") as fh:
+        sm = mmap.mmap(fh.fileno(), 0, access=mmap.ACCESS_READ)
+        with open(dst_path, "r+b") as dfh:
+            dm = mmap.mmap(dfh.fileno(), 0, access=mmap.ACCESS_WRITE)
+            for r0 in range(0, src_rows, chunk_rows):
+                r1 = min(r0 + chunk_rows, src_rows)
+                codes = np.frombuffer(sm, dtype=np.uint8,
+                                      count=(r1 - r0) * cols,
+                                      offset=src_off + r0 * cols)
+                q = lut.take(codes)
+                del codes  # release the mmap export before sm.close()
+                q[~np.isfinite(q)] = 0.0
+                q *= np.float32(scale_f32)
+                packed = f32_to_bf16_rne(q)
+                dm[dst_off + r0 * cols * 2: dst_off + r1 * cols * 2] = packed.tobytes()
+            dm.flush()
+            os.fsync(dfh.fileno())
+            dm.close()
+        sm.close()
+
+
+# ---------------------------------------------------------------------- main
+
+def main():
+    ap = argparse.ArgumentParser(description=__doc__.split("\n")[0])
+    ap.add_argument("--src", default=None,
+                    help="raw checkpoint dir (documentation only; unused)")
+    ap.add_argument("--stage", required=True, help="staging dir to reshape in place")
+    ap.add_argument("--force", action="store_true", help="redo even if already staged")
+    ap.add_argument("--workers", type=int, default=min(8, os.cpu_count() or 1))
+    args = ap.parse_args()
+    stage = args.stage
+    t0 = time.time()
+
+    index_path = os.path.join(stage, "model.safetensors.index.json")
+    cfg_path = os.path.join(stage, "config.json")
+    if not (os.path.isfile(index_path) and os.path.isfile(cfg_path)):
+        log(f"ERROR: {stage} is not a staged checkpoint (missing index/config)")
+        return 2
+    with open(index_path) as fh:
+        index = json.load(fh)
+    with open(cfg_path) as fh:
+        cfg = json.load(fh)
+    wm = index["weight_map"]
+
+    meta = build_meta(stage, index)
+
+    # ---- collect prefixes -------------------------------------------------
+    pre_re = re.compile(r"^(?P<pre>.+)\.mlp\.experts\.(?P<e>\d+)\.gate_proj\.weight$")
+    pres, n_e = {}, 0
+    for k in wm:
+        m = pre_re.match(k)
+        if m:
+            pres.setdefault(m.group("pre"), set()).add(int(m.group("e")))
+            n_e = max(n_e, int(m.group("e")) + 1)
+    prefixes = sorted(pres)
+    if not prefixes:
+        log("ERROR: no per-expert tensors found in the staged index")
+        return 2
+    mtp = [p for p in prefixes if p.startswith("mtp.")]
+    trunk = [p for p in prefixes if p.startswith("model.language_model.layers.")]
+    log(f"expert prefixes: {len(prefixes)} (trunk {len(trunk)}, mtp {len(mtp)}), "
+        f"{n_e} experts each")
+
+    # shapes from expert 0 of the first prefix
+    pre0 = prefixes[0]
+    shapes = {}
+    gdt, gsh = meta[f"{pre0}.mlp.experts.0.gate_proj.weight"][3:5]
+    ddt, dsh = meta[f"{pre0}.mlp.experts.0.down_proj.weight"][3:5]
+    gsd, gssh = meta[f"{pre0}.mlp.experts.0.gate_proj.weight_scale_inv"][3:5]
+    dsd, dssh = meta[f"{pre0}.mlp.experts.0.down_proj.weight_scale_inv"][3:5]
+    scale_law = (gdt == F8 and ddt == F8 and gsd == BF16 and dsd == BF16
+                 and list(gssh) == [(gsh[0] + 127) // 128, (gsh[1] + 127) // 128]
+                 and list(dssh) == [(dsh[0] + 127) // 128, (dsh[1] + 127) // 128])
+    if not scale_law:
+        log(f"ERROR: unexpected expert shapes: gate {gdt}{gsh} scale {gsd}{gssh} "
+            f"down {ddt}{dsh} scale {dsd}{dssh} (scale law 128x128)")
+        return 2
+    shapes = {"ffn2_h": (gsh[0] * 2, gsh[1]), "ffn": (dsh[1],),
+              "gs": tuple(gssh), "ds": tuple(dssh)}
+    f2, h = shapes["ffn2_h"]
+    f = shapes["ffn"][0]
+    log(f"shapes: gate_up [{n_e},{f2},{h}] F8, down [{n_e},{h},{f}] F8, "
+        f"scales {gssh}x2 / {dssh} bf16")
+
+    # ---- PLE table --------------------------------------------------------
+    ple_re = re.compile(r"^(?P<pre>.+)\.ple\.ple_embedding\.ngram_embedding\.shard_(?P<i>\d+)\.weight$")
+    ple_pre, shard_ids = None, {}
+    for k in wm:
+        m = ple_re.match(k)
+        if m:
+            ple_pre = m.group("pre")
+            shard_ids[int(m.group("i"))] = k
+    if ple_pre is None:
+        log("ERROR: no PLE table shards found")
+        return 2
+    n_shards = len(shard_ids)
+    if sorted(shard_ids) != list(range(n_shards)):
+        log(f"ERROR: PLE shard ids not contiguous 0..{n_shards - 1}")
+        return 2
+    sc_name = f"{ple_pre}.ple.ple_embedding.ngram_embedding.weight_scale"
+    sdt, ssh, sln = meta[sc_name][3], meta[sc_name][4], meta[sc_name][2]
+    if sdt != BF16 or list(ssh) != [1]:
+        log(f"ERROR: PLE scale is {sdt}{ssh}, expected BF16 [1]")
+        return 2
+    s_bits = struct.unpack("<H", bytes(view_src_tuple(meta, sc_name, stage)))[0]
+    scale_f32 = struct.unpack("<f", struct.pack("<I", s_bits << 16))[0]
+    log(f"PLE: {ple_pre}, {n_shards} shards, scale bf16 = {scale_f32!r}")
+
+    rows = [meta[shard_ids[i]][4][0] for i in range(n_shards)]
+    total_rows = sum(rows)
+    hd = meta[shard_ids[0]][4][1]
+    if any(meta[shard_ids[i]][4][1] != hd for i in range(n_shards)):
+        log("ERROR: PLE shards disagree on HD")
+        return 2
+    if hd % 128 != 0 and hd != 160:
+        log(f"WARNING: PLE HD={hd} is neither 160 nor a multiple of 128")
+
+    # ---- idempotence gate -------------------------------------------------
+    stacked_names = [f"{p}.mlp.experts.gate_up_proj" for p in prefixes]
+    merged_name = f"{ple_pre}.ple.ple_embedding.ngram_embedding.shard_0.weight"
+    already = (all(n in wm for n in stacked_names)
+               and merged_name in wm
+               and find_split(cfg) is not None and read_split(cfg) == 1)
+    if already and not args.force:
+        log("already staged (stacked names, merged shard_0 and split_ngram_parts=1 "
+            "all present) — nothing to do")
+        return 0
+
+    # ---- disk gate --------------------------------------------------------
+    stacked_bytes = len(prefixes) * n_e * (f2 * h + h * f)
+    ple_bytes = total_rows * hd * 2
+    need = stacked_bytes + ple_bytes
+    st = os.statvfs(stage)
+    avail = st.f_bavail * st.f_frsize
+    if avail < need * 1.05:
+        log(f"ERROR: {avail / 2**30:.1f} GiB free < {need / 2**30:.1f} GiB needed (+5%)")
+        return 2
+    log(f"disk: {avail / 2**30:.1f} GiB free, need {need / 2**30:.1f} GiB "
+        f"(experts {stacked_bytes / 2**30:.1f}, ple {ple_bytes / 2**30:.1f})")
+
+    # ---- stage experts ----------------------------------------------------
+    jobs = []
+    for idx, pre in enumerate(prefixes):
+        jmeta = {n: meta[n] for n in meta if n.startswith(f"{pre}.mlp.experts.")}
+        jobs.append((stage, pre, idx, n_e, shapes, jmeta))
+    ctx = mp.get_context("fork")
+    written_total = 0
+    with ctx.Pool(args.workers) as pool:
+        for pre, nbytes in pool.imap_unordered(stack_prefix, jobs):
+            written_total += nbytes
+            log(f"  stacked {pre} ({nbytes / 2**30:.2f} GiB)")
+    log(f"experts stacked: {written_total / 2**30:.2f} GiB in {len(prefixes)} files")
+
+    # ---- stage PLE --------------------------------------------------------
+    # The file name sorts BEFORE the model-000NN shards ON PURPOSE: the reader
+    # opens every index-named file in sorted order and emplaces EVERY tensor
+    # it physically holds, first-wins (checkpoint.cpp add_shard). The raw
+    # shards carry tensors of the SAME name as the merged one
+    # (...ple.ple_embedding.ngram_embedding.shard_0.weight, [2500012,160]
+    # fp8), so ple.merged.safetensors lost the race and the plan read one raw
+    # shard as the codes. "00-" wins the sort and the name.
+    out_name = "00-ple.merged.safetensors"
+    out_path = os.path.join(stage, out_name)
+    data_bytes = total_rows * hd * 2
+    hdr = {merged_name: {"dtype": BF16, "shape": [total_rows, hd],
+                         "data_offsets": [0, data_bytes]}}
+    blob = json.dumps(hdr, separators=(",", ":")).encode("utf-8")
+    pad = (-len(blob)) % 8
+    blob += b" " * pad
+    hlen = len(blob)
+    with open(out_path, "wb") as fh:
+        fh.write(struct.pack("<Q", hlen))
+        fh.write(blob)
+        fh.truncate(8 + hlen + data_bytes)
+    jobs = []
+    row0 = 0
+    for i in range(n_shards):
+        f_, off, ln, _, shp = meta[shard_ids[i]]
+        jobs.append((os.path.join(stage, f_), off, shp[0], out_path,
+                     8 + hlen + row0 * hd * 2, scale_f32, hd, 131072))
+        row0 += shp[0]
+    assert row0 == total_rows
+    with ctx.Pool(args.workers) as pool:
+        pool.map(fill_ple_shard, jobs)
+    log(f"PLE merged: {data_bytes / 2**30:.2f} GiB bf16 "
+        f"({total_rows} rows x {hd} cols)")
+
+    # ---- post-verify ------------------------------------------------------
+    verify(stage, prefixes, n_e, shapes, meta, shard_ids, rows, hd,
+           scale_f32, out_path, hlen)
+
+    # ---- rewrite index + config -------------------------------------------
+    new_wm = {k: v for k, v in wm.items() if k not in set(shard_ids.values())}
+    for idx, pre in enumerate(prefixes):
+        fname = f"stacked.{idx:03d}.safetensors"
+        for t in (f"{pre}.mlp.experts.gate_up_proj",
+                  f"{pre}.mlp.experts.gate_up_proj_scale_inv",
+                  f"{pre}.mlp.experts.down_proj",
+                  f"{pre}.mlp.experts.down_proj_scale_inv"):
+            new_wm[t] = fname
+    new_wm[merged_name] = out_name
+    total = sum(meta_size(meta, k) for k in new_wm
+                if k in meta)  # originals
+    # new tensors' sizes
+    total += written_total + data_bytes
+    index["weight_map"] = new_wm
+    if "metadata" in index and isinstance(index["metadata"], dict):
+        index["metadata"]["total_size"] = total
+    # ATOMIC REPLACE, never open(path, "w"): the stage's index is a symlink
+    # to the RAW checkpoint (staging symlinks everything but config.json) —
+    # open("w") would rewrite the raw file through the link. os.replace
+    # swaps the link itself for a real file; the raw tree stays pristine.
+    for path, payload in ((index_path, json.dumps(index, indent=2)),
+                          (cfg_path, json.dumps(cfg, indent=2))):
+        tmp = path + ".tmp"
+        with open(tmp, "w") as fh:
+            fh.write(payload)
+        os.replace(tmp, path)
+    n = set_split(cfg, 1)
+    log(f"config: split_ngram_parts=1 written at {n} location(s)")
+
+    with open(os.path.join(stage, "ple_scale.txt"), "w") as fh:
+        fh.write(repr(scale_f32) + "\n")
+
+    dt = time.time() - t0
+    log(f"staged in {dt / 60:.1f} min: {written_total / 2**30:.1f} GiB experts + "
+        f"{data_bytes / 2**30:.1f} GiB PLE; scale {scale_f32!r} -> ple_scale.txt")
+    return 0
+
+
+def meta_size(meta, name):
+    _, _, ln, _, _ = meta[name]
+    return ln
+
+
+def view_src_tuple(meta, name, stage):
+    f_, off, ln, _, _ = meta[name]
+    with open(os.path.join(stage, f_), "rb") as fh:
+        fh.seek(off)
+        return fh.read(ln)
+
+
+def find_split(cfg):
+    """Locate split_ngram_parts keys (recursive); return list of paths."""
+    hits = []
+
+    def walk(d, path):
+        if isinstance(d, dict):
+            for k, v in d.items():
+                if k == "split_ngram_parts":
+                    hits.append(path + [k])
+                walk(v, path + [k])
+    walk(cfg, [])
+    return hits
+
+
+def read_split(cfg):
+    hits = find_split(cfg)
+    if not hits:
+        return None
+    d = cfg
+    for k in hits[0][:-1]:
+        d = d[k]
+    return d[hits[0][-1]]
+
+
+def set_split(cfg, value):
+    hits = find_split(cfg)
+    for path in hits:
+        d = cfg
+        for k in path[:-1]:
+            d = d[k]
+        d[path[-1]] = value
+    if not hits:  # plugin's meta_int2 falls back to the top-level key
+        cfg["split_ngram_parts"] = value
+        hits.append(["split_ngram_parts"])
+    return hits
+
+
+def verify(stage, prefixes, n_e, shapes, meta, shard_ids, rows, hd,
+           scale_f32, out_path, hlen):
+    """Cheap strong checks: headers parse, sizes match, sampled bytes identical."""
+    f2, h = shapes["ffn2_h"]
+    f = shapes["ffn"][0]
+    gs, ds = shapes["gs"], shapes["ds"]
+    # headers of every new file
+    for idx, pre in enumerate(prefixes):
+        path = os.path.join(stage, f"stacked.{idx:03d}.safetensors")
+        hlen2, hdr = parse_header(path)
+        want = {
+            f"{pre}.mlp.experts.gate_up_proj": (F8, [n_e, f2, h]),
+            f"{pre}.mlp.experts.gate_up_proj_scale_inv":
+                (BF16, [n_e, gs[0] * 2, gs[1]]),
+            f"{pre}.mlp.experts.down_proj": (F8, [n_e, h, f]),
+            f"{pre}.mlp.experts.down_proj_scale_inv": (BF16, [n_e, ds[0], ds[1]]),
+        }
+        for name, (dt, shp) in want.items():
+            e = hdr[name]
+            assert e["dtype"] == dt and list(e["shape"]) == shp, name
+            assert e["data_offsets"][1] - e["data_offsets"][0] == \
+                int(np.prod(shp)) * (1 if dt == F8 else 2), name
+    _, phdr = parse_header(out_path)
+    e = phdr[next(iter(phdr))]
+    assert e["dtype"] == BF16 and list(e["shape"]) == [sum(rows), hd]
+
+    # sampled PLE rows: recompute and compare bytes exactly
+    lut = e4m3_lut()
+    out_off = 8 + hlen
+    for i in (0, len(shard_ids) // 2, len(shard_ids) - 1):
+        name = shard_ids[i]
+        f_, off, ln, _, shp = meta[name]
+        with open(os.path.join(stage, f_), "rb") as fh:
+            mm = mmap.mmap(fh.fileno(), 0, access=mmap.ACCESS_READ)
+            for r in (0, shp[0] - 1):
+                codes = np.frombuffer(mm, dtype=np.uint8, count=hd,
+                                      offset=off + r * hd)
+                q = lut.take(codes) * np.float32(scale_f32)
+                del codes  # release the mmap export before mm.close()
+                want = f32_to_bf16_rne(q).tobytes()
+                got_row = sum(rows[:i]) + r
+                with open(out_path, "rb") as ofh:
+                    ofh.seek(out_off + got_row * hd * 2)
+                    got = ofh.read(hd * 2)
+                assert got == want, f"PLE row {got_row} mismatch"
+            mm.close()
+    log("verify: new-file headers, sizes and sampled PLE rows all exact; "
+        "expert slabs were byte-checked in-stack")
+
+
+if __name__ == "__main__":
+    sys.exit(main())

scripts/test-changed.sh

@@ -204,12 +204,12 @@ for path in "${CHANGED[@]}"; do
         scripts/qwen-sharp-v22.4.0.jinja)
             add_test tests/frank_large_unit.bats
             ;;
-        # frank-small is llama.cpp on the 3090 as a ROOT unit: a native CUDA
+        # frank-secondary is llama.cpp on the 3090 as a ROOT unit: a native CUDA
         # binary, so no generated quadlet is involved. Its env, unit and serve
         # script all live in scripts/, which is why this case has to precede
         # the generic scripts/*.service and scripts/* fallbacks below.
-        scripts/frank-small.*|scripts/serve-frank-small.sh)
-            add_test tests/frank_small_unit.bats
+        scripts/frank-secondary.*|scripts/serve-frank-secondary.sh)
+            add_test tests/frank_secondary_unit.bats
             add_test tests/llm_units.bats
             add_test tests/deployment_scoping.bats
             add_test tests/scripts_syntax.bats

scripts/verify-swift15-ple.py

@@ -0,0 +1,87 @@
+#!/usr/bin/env python3
+"""Streaming absmax audit of the Swift1.5 PLE n-gram table (BF16).
+
+The PLE fp8 encoding uses ONE fixed scale (1.9931793212890625e-04, the
+official checkpoint's plateau value). fp8_e4m3 saturates at 448, so the
+fixed scale is legal iff absmax <= 448 * SCALE. This walks every
+ngram_embedding.shard_*.weight tensor in the checkpoint (mmap, no copy),
+computes the absmax from the BF16 bit pattern (clear the sign bit, max
+the remaining uint16 — BF16 magnitudes are monotone in their bits), and
+reports the headroom. Exit 0 only when headroom >= 1.5x.
+
+Run from any cwd EXCEPT /tmp on dankserver (/tmp/inspect.py shadows the
+stdlib). Usage: verify-swift15-ple.py <ckpt-dir>
+"""
+import json
+import mmap
+import os
+import struct
+import sys
+
+SCALE = 1.9931793212890625e-04  # bf16 0x3951, the official plateau scale
+FP8_E4M3_MAX = 448.0
+
+import numpy as np
+gmax = 0.0
+gfile = ""
+over_cap = 0
+total = 0
+
+ckpt = sys.argv[1]
+index = json.load(open(os.path.join(ckpt, "model.safetensors.index.json")))
+shards: dict[str, list[str]] = {}
+for k, f in index["weight_map"].items():
+    if "ngram_embedding.shard_" in k and k.endswith(".weight"):
+        shards.setdefault(f, []).append(k)
+
+files = len(shards)
+tensors = sum(len(v) for v in shards.values())
+print(f"n-gram shards: {files} files, {tensors} tensors")
+
+import math
+
+cap = FP8_E4M3_MAX * SCALE
+# largest bf16 <= cap: truncate the float32 bits (cap > 0); bf16
+# magnitudes are monotone in their bits, so |w| > cap <=> bits > cap_bits
+cap_bits = int((np.float32(cap).view(np.uint32) >> 16) & 0xFFFF)
+for fname, keys in sorted(shards.items()):
+    path = os.path.join(ckpt, fname)
+    with open(path, "rb") as fh:
+        hlen = struct.unpack("<Q", fh.read(8))[0]
+        header = json.loads(fh.read(hlen))
+        base = 8 + hlen
+        mm = mmap.mmap(fh.fileno(), 0, prot=mmap.PROT_READ)
+        for k in keys:
+            meta = header[k]
+            off, end = meta["data_offsets"]
+            a = np.frombuffer(mm, dtype=np.uint16, count=(end - off) // 2,
+                              offset=base + off)
+            v = a & 0x7FFF
+            m = int(v.max())
+            over_cap += int((v > np.uint16(cap_bits)).sum())
+            total += v.size
+            del a
+            if m > gmax:
+                gmax = m
+                gfile = k
+        mm.close()
+
+# rebuild the BF16 value: the 16 bits are the HIGH half of the float32
+absmax = np.frombuffer(b"\x00\x00" + np.uint16(gmax).tobytes(),
+                        dtype=np.float32)[0]
+
+print(f"absmax        = {absmax!r}  (tensor: {gfile})")
+print(f"2^log2        = 2^{math.log2(absmax):.3f}")
+print(f"fp8 cap       = 448 * {SCALE:.8g} = {cap:.8g}")
+print(f"headroom      = {cap / absmax:.3f}x")
+ok = cap / absmax >= 1.5
+print("VERDICT:", "PASS (>= 1.5x headroom over the fixed scale)" if ok
+      else "FAIL — fixed scale does not cover the Swift PLE")
+print(f"values > cap  = {over_cap} of {total} ({over_cap / total:.3e})")
+nxt_bits = int((np.float32(absmax / FP8_E4M3_MAX).view(np.uint32)
+                >> 16) & 0xFFFF) + 1
+nxt_scale = np.frombuffer(b"\x00\x00" + np.uint16(nxt_bits).tobytes(),
+                          dtype=np.float32)[0]
+print(f"next-bf16 scale covering absmax: {nxt_scale:.8g} "
+      f"(0x{nxt_bits:04x}, headroom {FP8_E4M3_MAX * nxt_scale / absmax:.4f}x)")
+sys.exit(0 if ok else 1)

serverAppConfigs-docs/Agent/Agent Docs and Edits.md

@@ -0,0 +1,111 @@
+---
+type: reference
+title: Agent Docs and Edits
+description: Two self-inflicted damage classes — structurally broken markdown after parallel agent edits, and botched hashline PUT/CUT repairs — with the audit checks and recovery order for each.
+tags: [agents, docs, markdown, hashline, edit-recovery]
+timestamp: 2026-10-05T00:00:00-07:00
+---
+
+# Agent Docs and Edits
+
+## Doc-structure audit (after parallel agent edits)
+
+Use after one or more subagents have edited markdown docs (OKF/Obsidian
+vault with `---` frontmatter), before committing. Link checkers and
+completeness seams validate *references*, not *structure* — agents reliably
+produce structurally broken markdown that still links cleanly.
+
+Damage catalogue:
+
+1. Banner/prose prepended **above** the opening `---` → the frontmatter is
+   no longer frontmatter (title/tags/description become body text).
+2. A rewrite **appended** instead of merged → duplicate headings and a stray
+   second `---` mid-file.
+3. Literal model artifacts in the body: `<|channel>}<tool_call|>...`,
+   `+- `, `- - `.
+4. Wrapped bullets re-emitted with a `- ` prefix on every physical line.
+5. Line-range replacement that dropped a sentence tail → dangling fragment,
+   often mid-parenthesis.
+
+Checks (fast, run all):
+
+```bash
+head -3 <file>                          # must be `---` if the page has frontmatter
+grep -c '^---$' <file>                  # exactly 2 (opener + closer)
+grep -n '^## ' <file> | sort -t: -k2 | uniq -f1 -d   # duplicate sections
+grep -rnE '^\+- |^- - |<\|' <files>     # artifacts
+python3 scripts/check-vault-links.py <vault-dir>
+./tests/run.sh                          # docs completeness seam etc.
+```
+
+Repair order (matters):
+
+1. If the file was destroyed rather than edited: stash the agent version to
+   `.AGENTTRASH/<YYYYMMDD-topic>/`, then `git checkout HEAD -- <file>` and
+   re-apply only the intended changes by hand.
+2. If it merely duplicated: **salvage unique content first** — cut the
+   one-of-a-kind section, paste it into the copy you keep, THEN delete the
+   duplicate ranges. Never delete first.
+3. Re-add missing sentence tails (insert before the orphaned fragment),
+   keeping list continuation lines indented.
+4. Re-run all checks above.
+
+Gotchas: agents can write AFTER they appear finished (job settles, file
+changes) — re-verify structure after the last background job reports, not
+before. Editing an agent's already-damaged file with fresh line numbers is
+safe, but its snapshot tag changes on every edit — re-read before each
+subsequent edit. `.AGENTTRASH/` is gitignored: trash is local-only, so
+nothing moved there reaches the repo — that is intended (temporary hold, not
+archive).
+
+## Hashline edit recovery
+
+Failure mode: multi-op edits resolve every `PUT N.=M:` / `CUT N.=M:` index
+against the **original** file numbering, and the post-apply diff view
+**elides lines** (it shows an anchor window, not the whole file). Two traps
+follow:
+
+1. A second edit call built from a diff-view snapshot uses line numbers from
+   a *partial* view. Lines outside the window are invisible but still
+   counted.
+2. If the first apply was itself wrong, the file shifted. The diff view
+   tells you the damage, but not the true offsets.
+
+Result: the repair pastes fresh text **beside** stale text instead of over
+it.
+
+Procedure:
+
+1. **Never carry line numbers across calls.** Re-read the target range for a
+   fresh tag before every subsequent edit. A read is cheap; a duplicated
+   paragraph costs a commit and a revert.
+2. **One op per call when the previous apply was imperfect.** Fixed offsets
+   are only trustworthy when every preceding apply landed exactly as
+   specified.
+3. **Re-read after every apply**, and verify the whole affected region — the
+   diff view is a sample, not proof. Check the *boundaries*: the line before
+   the edit and the line after it. Damage shows up as a repeated sentence
+   straddling the seam.
+4. **Repair by CUT, not by re-PUT.** Once you know the exact stale lines
+   from a fresh read, delete them. Do not try to PUT a corrected superset
+   over a range whose extent you are guessing at.
+5. Verify with a content assertion, not a diff review: `grep -c` for the
+   duplicated sentence, plus a syntax/compile check on any touched script.
+
+Same rule for any imperative `edit` tool with an old/new match: the matched
+text must be the **entire** text being replaced. Matching a bare fragment
+inside a paragraph splices the replacement into the middle of the sentence
+and silently corrupts the file. Re-read the file afterwards to confirm.
+
+### Anchored in-page links
+
+Renaming a Markdown heading breaks `[[#Old Heading]]` anchors. When a
+section is retitled, grep for the old heading text and fix every anchor in
+the same pass — doc link-checkers frequently do not validate `#fragment`
+targets, so a green suite will not catch a dangling one.
+
+## Related
+
+- [[OMP Harness]] — the session context these repairs protect
+- [[Command Center Dotfiles]] — the repo these edits commit to
+- [[Testing Policy]] — the seams that catch broken vault structure

serverAppConfigs-docs/Agent/Command Center Dotfiles.md

@@ -0,0 +1,87 @@
+---
+type: reference
+title: Command Center Dotfiles
+description: The ~/.omp repo is itself the deployment — stow-equivalent packages, full-replica rsync transport, the pre-overwrite audit for non-blank hosts, deploy.sh mechanics, and the fish/sshpass transport gotchas.
+tags: [dotfiles, omp, deploy, rsync, blankpad]
+timestamp: 2026-10-05T00:00:00-07:00
+---
+
+# Command Center Dotfiles
+
+Repo `~/.omp` is itself the deployment: packages are home-relative trees
+under `dotfiles/<pkg>/`; `scripts/deploy.sh` symlinks live files into the
+repo checkout. `git pull` + `deploy.sh` is the whole propagation story.
+
+## Transport decision (agreed with Frank, executed 2026-09-09)
+
+- Ship via **full-replica rsync of `/home/dank/.omp/` including `.git`** to
+  `<user>@<host>:/home/<user>/.omp/`, NOT clone-from-Gitea — `main` can be
+  ahead of `origin/main`; push is a user-only trigger, so clone would ship
+  stale state.
+- Deploy runs on-target as the target user: `bash
+  /home/<user>/.omp/scripts/deploy.sh` (resolves the repo via `git -C
+  $(dirname $(realpath $0)) rev-parse --show-toplevel`, so `.git` must be
+  present at the target path — full-replica transport guarantees this).
+
+## Pre-overwrite audit (MANDATORY before first sync on a non-blank host)
+
+Diff every file that `deploy.sh --check` reports as DRIFT against the repo
+copy BEFORE letting quarantine-and-replace run. Host-specific divergence
+belongs in an override package, not in quarantine:
+
+1. Pull divergent live files from target
+   (`ssh ... 'cat <path>' > local-staged-copy`).
+2. Secret-audit staged copies (grep token/secret/password/key shapes)
+   before anything enters git.
+3. Compose `dotfiles/zz-<hostname>/` on the source machine from those live
+   copies; commit to repo first.
+4. Ship the package dir via rsync, then run deploy sync — override deploys
+   LAST and win over base packages.
+
+## deploy.sh mechanics (verified on blankpad)
+
+- `--check` = pure report (ADOPT/LINK/DRIFT lines), writes nothing. Run it
+  first, always.
+- Default mode = sync: adopt identicals silently, link new,
+  quarantine-and-replace divergents into `.quarantine/<host>/<ts>/`
+  (gitignored; never rm).
+- **Override-target bug (fixed cc2c652):** `zz-<host>` packages nest
+  base-package dirs internally (`zz-blankpad/niri/.config/...`); target
+  derivation must strip the nested package segment, else overrides deploy to
+  phantom `$HOME/niri/.config/...` while real targets keep pointing at
+  canonical copies (visible symptom: check shows 0 actions but
+  `readlink ~/.zshrc` points at base `zsh/.zshrc`).
+- systemd units auto-enable only when ExecStart binary resolves on that
+  host; dconf packages load via `dconf load /`, never symlinks.
+- `core.hooksPath=.githooks` is armed by deploy.sh per machine — set it once
+  post-transfer even before first sync.
+
+## ssh/sshpass transport gotchas (all bit live)
+
+- Resolve the target via mDNS FQDN (`avahi-resolve -4 -n <host>.local`); a
+  bare hostname may not resolve. Drop-not-refuse on port 22 = sshd down or
+  firewall default-drop, not auth failure — needs a local/console fix on
+  target.
+- `SSHPASS='...' sshpass -e ssh ...` inline per invocation; an env passed
+  via the tool's env parameter did not propagate to the sshpass child
+  process.
+- The remote login shell may be **fish**: never send bash syntax
+  (`for/do/done`, `$?`) bare through `ssh host 'cmd'`; wrap as `bash -c
+  "..."` or ship a script. See [[SSH and Remote Access]].
+- File shipping: `ssh 'cat > /remote/path' < localfile` (stdin redirect). A
+  here-doc attached to the ssh command is consumed by `cat` — the intended
+  script body becomes the file content, silently. Verify shipped bytes with
+  an md5sum compare on both ends; do NOT compare `/dev/stdin` after `cat`
+  drained it (false DIVERGED).
+
+## Verification of a completed deploy
+
+Re-run `deploy.sh --check` → expect exactly zero actions/quarantined/skipped
+= idempotent steady state. Spot-check `readlink` on override files points
+into `dotfiles/zz-<host>/`, not base packages.
+
+## Related
+
+- [[SSH and Remote Access]] — the transport mechanics this deploy rides
+- [[Noctalia and Niri]] — the desktop packages this repo ships
+- [[Deploy State Triage]] — the serverAppConfig deploy flow (different repo)

serverAppConfigs-docs/Agent/OMP Harness.md

@@ -0,0 +1,274 @@
+---
+type: reference
+title: OMP Harness
+description: The omp agent harness as configured on this fleet — compaction chain arithmetic, per-model sampler routing via compat.extraBody, and the static-preamble/skill-library audits that keep session context lean.
+tags: [omp, harness, compaction, models-yml, context, samplers]
+timestamp: 2026-10-05T00:00:00-07:00
+---
+
+# OMP Harness
+
+The harness reaches the fleet through `~/.omp/agent/models.yml` (model
+registry; single source of truth — there is no copy on the servers) and
+`~/.omp/agent/config.yml` (roles, compaction, global samplers). Nothing in
+this file is repo-managed; edits are harness-side only.
+
+**Model ids are the StarCraft unit names (official 2026-10-08, on the wire
+since 2026-10-09):** `frank-archon` (dankserver :7770, radiance),
+`frank-dragoon` (dankclaw :7770, flash-server), `frank-zealot` (dankclaw
+:7771, the 3090 compaction hub) — formerly `frank-primary`, `frank-large`,
+`frank-secondary`. The SERVICES keep the old names (units, scripts, env
+files, runbook pages); pages written in the old era name the service,
+the harness names the model. The wire now matches: each endpoint's
+`/v1/models` answers with the Protoss name, so `models.yml`, the OpenWebUI
+picker/grants, the homepage docs and the raw API all use one id per model
+(see [[OpenWebUI]] for the grant re-key and the [[frank-primary-radiance-swift15-w4dn8]]
+note on radiance echoing the requested name).
+
+## Compaction chain arithmetic (verify before ANY window change)
+
+When a fleet endpoint's context grows past its harness-declared window, OMP
+compacts by sending the conversation to a designated target model. Before
+changing any `contextWindow`/`maxTokens`, verify the chain arithmetically —
+a target whose window is smaller than the source's trigger point silently
+breaks compaction.
+
+Where the contract lives:
+
+- `~/.omp/agent/config.yml` → `compaction.remoteEndpoint` (target URL),
+  `methodOrder`, `thresholdPercent` (70), idle settings
+- `~/.omp/agent/models.yml` → per-model `contextWindow`, `maxTokens`,
+  `compactionModel` references
+- Implementation: omp plugin `dist/core/compaction/compaction.js`
+
+The arithmetic (the coupled contract every endpoint's maxTokens comment
+encodes):
+
+1. **Source trigger** = `thresholdPercent × source contextWindow`
+   (0.70 × 200,704 = 140,492 for frank-primary on the current shape)
+2. **Output budget** = capped at the target model's `maxTokens`
+   (32,768 for frank-secondary; 36,864 frank-primary; 57,344 frank-large — the
+   largest, because halogen's `HALOGEN_MAX_TOKENS_CAP` must stay ≥ it)
+3. **REQUIREMENT**: prompt (≤ trigger) + budget ≤ the target's per-slot
+   window. Worked pairs from the live registry: 140,492 + 36,864 = 177,356 ≤
+   200,704 (frank-primary); 140,490 + 57,344 = 197,834 ≤ 200,700
+   (frank-large); 140,492 + 32,768 = 173,260 ≤ 200,704 (frank-secondary).
+
+Procedure (recompute from the LIVE files every time — windows rotate with
+every swap; the numbers above are the 2026-10-05 state):
+
+1. Read the `config.yml` compaction block + `models.yml` entries for source
+   and target. Every endpoint's `contextWindow` comment carries its own
+   arithmetic — read them.
+2. Read the target checkpoint's `max_position_embeddings` (native window) —
+   the served window may be a concurrency contract, not the native limit
+   (`--parallel N` DIVIDES the pool; the per-slot number is the contract).
+3. If native ≥ trigger + budget, reslice the pinned pool:
+   `MAX_RUNNING × CONTEXT_LENGTH ≤ MAX_TOTAL_TOKENS` (the pin exists because
+   the runtime's autofit leaves too little for deferred loads + prefill
+   workspace; check `nvidia-smi` headroom — expect ~0.8 GB workspace margin).
+4. Changes: host env (CONTEXT_LENGTH-style env tunables) via git →
+   `./tests/run.sh` → deploy; harness `models.yml` directly (no repo copy).
+5. Verify: `/health` 200, boot log's pool line, then a long-prompt probe at
+   ≥ trigger height (the compaction-fit guarantee), then concurrent requests.
+
+Known reconciliation trap: comments in models.yml have described compaction
+as running "on the dankserver endpoint" while `remoteEndpoint` pointed at
+dankclaw — the comment/config disagreed. The comments now name the real
+target (`compaction.remoteEndpoint` = dankclaw :7771). **The config is
+truth; fix comments in the same change.** `maxTokens` never changes for a
+reslice — it caps output; `contextWindow` caps input. Boot-shape OOM
+laddering is a different problem ([[Llama Cpp Serving Sizing]]).
+
+## Per-model sampler config (`compat.extraBody`)
+
+`models.yml` has **no per-model sampler keys**. Sampling is global
+(`config.yml`: `temperature`, `topP`, `topK`, `minP`, `presencePenalty`,
+`repetitionPenalty`; default `-1` = **not sent**). The per-model hook is
+`compat.extraBody`:
+
+**Why it works:** `applyOpenAIExtraBody` runs **last** in the
+openai-completions builder — after `applyChatCompletionsCompatPolicy` — via
+`Object.assign(params, extraBody)`. So extraBody overrides global settings
+and compat-policy output.
+
+```yaml
+compat:
+  reasoningEffortMap:
+    max: xhigh
+  extraBody:
+    temperature: 1.0
+    top_p: 0.95
+    top_k: 20
+    min_p: 0.0
+    presence_penalty: 0.0
+    repetition_penalty: 1.0
+    preserve_thinking: true
+```
+
+Use **wire names**, not OMP camelCase. Two traps:
+
+1. **`preserve_thinking` is not user-settable as a compat key.**
+   `qwenPreserveThinking` is auto-derived in `buildOpenAICompat` from the
+   *auto-detected* `thinkingFormat` + local baseUrl, **before**
+   `applyCompatOverrides`. Setting `compat.thinkingFormat: qwen` in
+   models.yml does **not** enable it. Custom-named models (`frank-large`)
+   match no Qwen id pattern → use `extraBody: {preserve_thinking: true}`.
+2. **Never put `chat_template_kwargs` in `extraBody`** where a compat policy
+   also sets it — `Object.assign` is shallow and replaces the whole object,
+   silently dropping the `reasoning_effort` kwarg the compat policy set.
+   Top-level `preserve_thinking` is honoured by halogen. (frank-primary's
+   FreeToken entry currently carries a `chat_template_kwargs` extraBody with
+   a comment asking to verify-and-prune — the exact trap to check at that
+   endpoint's next swap.)
+
+**Gotcha: silent greedy decode.** halogen has **no server-side sampler
+defaults** — `temperature` absent or `0` = greedy. If OMP sends nothing, the
+endpoint decodes greedy while looking configured. This is why frank-large's
+entry carries the recipe on the wire and a comment saying "do not drop it".
+
+**Reasoning-effort folding.** Qwen3.8's template accepts only
+`{xhigh (default), medium, low}` for `reasoning_effort` and 400s on anything
+else ("Unexpected reasoning effort high"). Every endpoint's
+`reasoningEffortMap` folds all harness levels onto that set; a role whose
+level is unmapped fails on its FIRST request, and the failure names Jinja,
+not the model.
+
+### Verify the exact wire body (echo-capture)
+
+Don't trust the YAML — capture what OMP actually emits:
+
+1. Echo server on `127.0.0.1:18099` that dumps the JSON body to a file and
+   returns a canned `chat.completion` (include `reasoning_content` so
+   thinking paths exercise).
+2. Copy the **real** models.yml into a throwaway profile with only the
+   baseUrl swapped:
+   ```bash
+   sed 's|http://<host>:7770/v1|http://127.0.0.1:18099/v1|' \
+     ~/.omp/agent/models.yml > ~/.omp/profiles/wiretest/agent/models.yml
+   ```
+3. Run and inspect:
+   ```bash
+   omp --profile wiretest --model "<provider>/<model>" -p "hi" \
+     --no-tools --no-session --no-extensions --no-skills --no-rules
+   jq 'del(.messages, .tools)' last-body.json
+   ```
+   Confirm every intended field is present with the right value.
+
+**Verify a field is *honoured*, not just accepted** — unknown fields often
+return 200 anyway. Prove behavior with a **prompt-token delta**: replay an
+assistant turn carrying a long `reasoning_content`, then compare
+`usage.prompt_tokens` with the flag true vs false. A large delta means the
+field changed template rendering; zero delta means it was ignored.
+
+Cleanup: trash the probe profile and echo server (`profiles/.AGENTTRASH`),
+never `rm`.
+
+## Static-preamble audit ("the context feels bloated")
+
+What loads statically every session (measure each, don't guess):
+
+1. Repo rules file — `wc -c AGENTS.md` (baseline 2026-10-05: 72,179 B, ~18k
+   tokens — historically the largest block).
+2. Skill index — only names+descriptions load; bodies are on-demand. Sum
+   description bytes across all SKILL.md frontmatter:
+   ```bash
+   python3 - <<'EOF'
+   import glob, re, os
+   for root in [os.path.expanduser("~/.omp/agent/managed-skills"),
+                os.path.expanduser("~/.omp/agent/skills"), ".omp/skills"]:
+       n=tot=0
+       for sk in glob.glob(root+"/*/SKILL.md"):
+           fm=re.search(r'^---.*?---', open(sk, encoding="utf-8", errors="replace").read(), re.S)
+           d=re.search(r'description:\s*(.*(?:\n(?:\s{2,}.*)*)?)', fm.group(0) if fm else "")
+           n+=1; tot+=len(d.group(1)) if d else 0
+       print(root, n, tot)
+   EOF
+   ```
+   Baseline 2026-10-05: 171 managed skills / ~38 KB descriptions + 9 user
+   skills.
+3. Memory lessons — `find ~/.omp/agent/memories -name 'MEMORY.md' -o -name
+   'memory_summary.md' | xargs wc -c`. The injected block ≈ the largest
+   MEMORY.md.
+4. Harness/system prompt + tool schemas + CLI persona: fixed cost, ~20 KB;
+   not prunable.
+
+Known rot patterns:
+
+- Near-duplicate auto-captures of the same lesson (three "authentik ORM"
+  entries, the freetoken-build pair, the three job-runner skills) — dedupe
+  via `manage_skill delete`/`update`.
+- Corrupted captures containing stray CJK characters (`一团ly`, `等方法`)
+  mid-sentence — a known defect of the auto-capture path; delete the corrupt
+  copy, keep the clean one.
+- Stale host/service names in consolidated memory (pre-rename
+  `frank-default-large`) — the repo's AGENTS.md topology table is ground
+  truth.
+
+Fix strategy: AGENTS.md prose (measured runbook narratives) → move to
+`serverAppConfigs-docs/LLM/` runbooks, keep laws + tables in AGENTS.md (the
+repo's own docs-tree rule mandates this). Managed skills live in
+`~/.omp/agent/managed-skills` (safe to edit); user-authored skills in
+`~/.omp/agent/skills` are NEVER touched by tooling. Memory store edits:
+stage removed captures to trash first (`~/.AGENTTRASH/YYYYMMDD-topic`).
+
+Baseline to beat (2026-10-05): ~35–40k tokens of preamble — AGENTS.md ~18k,
+skill index ~10k, lessons ~3.5k, harness ~6k. Proposed reductions: −12k
+(AGENTS.md slimming), −3k (skill dedupe).
+
+## Managed-skill library hygiene
+
+Use when auditing, pruning, or consolidating `~/.omp/agent/managed-skills`
+(the auto-captured skill store; the full index is injected into EVERY
+session's context, so dead or duplicate entries are a permanent tax).
+
+The static-context budget (measured 2026-10-05):
+
+| Block | Size | Notes |
+|---|---|---|
+| `AGENTS.md` (repo) | 72,179 B ≈ 18k tok | biggest single item |
+| Skill index (~173 skills) | ~40 KB ≈ 10k tok | names+descriptions only |
+| Learned-lessons block | ~14 KB | can contain corrupted/duplicate captures |
+| Harness system prompt + tool schemas | ~23 KB | fixed |
+
+Audit procedure:
+
+1. Build an inventory: per skill, name, mtime, body word count, description,
+   first 12 headings (one pass, out to a file; read in pages).
+2. Group into: retired-runtime (host/service gone; its runbook already lives
+   in `serverAppConfigs-docs/LLM/`), verbatim-duplicate pairs/triples (same
+   lesson captured twice — often the SAME DAY), same-lesson families
+   (N skills = one procedure), keep.
+3. Read full bodies of every merge candidate before proposing a merge;
+   description alone proves overlap only for exact dupes.
+4. Present A/B/C/D verdicts for user approval. NEVER execute before the
+   discussion concludes — the user gates execution explicitly.
+5. On approval: stage everything to trash first
+   (`~/.AGENTTRASH/YYYYMMDD-<topic>/`), never rm. After merging: recreate
+   the surviving skill via `manage_skill` update with body = superset,
+   delete absorbed ones.
+
+Known rot classes (this library, 2026-10-05):
+
+- Same-day duplicate captures (freetoken-build-dankclaw vs
+  freetoken-dankclaw-build; the three job-runner skills).
+- CUDA-skill vs its self-declared HIP twin (pcie-link-capability-probe vs
+  pcie-link-probe-hip-rdna4) → merge.
+- Era ghosts: skills naming retired runtimes (freetoken, lucebox, r9v/
+  radiance, old halogen-flash-server image, ik_llama, GLM-5.3 experiment).
+- Skills contradicting user standing orders (live-bench-dashboard vs the
+  2026-09-29 delete-dashboards order) — flag for user, don't assume.
+- Memory store: corrupted captures with CJK noise from broken ssh commands;
+  stale host/service renames.
+
+Sizing verdicts that worked: bench family → 2 skills (harness vs
+discipline), not one mega-skill; keep profiler/kernel-time attribution
+standalone; fold go/no-go arithmetic skills into their probe skill
+(heterogeneous-accelerator-split-triage → PCIe probe).
+
+## Related
+
+- [[Agent Docs and Edits]] — editing vault docs and hashline targets safely
+- [[Command Center Dotfiles]] — the `~/.omp` repo deploy this config rides
+- [[Llama Cpp Serving Sizing]] — the engine-side sizing the chain must fit
+- [[LLM Bench Windows and Measurement]] — measuring the endpoints this config routes

serverAppConfigs-docs/Agent/index.md

@@ -0,0 +1,23 @@
+---
+type: index
+description: Pages about the agent harness itself — omp config, vault editing discipline, and the dotfiles repo that deploys the agent's own home.
+tags: [agent, index]
+timestamp: 2026-10-05T00:00:00-07:00
+---
+
+# Agent
+
+Pages about the agent's own tooling and context — how omp is configured, how
+docs get edited safely, and how the agent's home is deployed.
+
+- [[OMP Harness]] — compaction chain arithmetic, per-model samplers via
+  `compat.extraBody`, preamble and skill-library audits
+- [[Agent Docs and Edits]] — doc-structure audits after parallel edits;
+  hashline edit recovery
+- [[Command Center Dotfiles]] — the `~/.omp` repo-as-deployment, rsync
+  transport, override packages
+
+## Related
+
+- [[Testing Policy]] — the seams these pages' checks ride
+- [LLM](../LLM/index.md) — the fleet these agents operate

serverAppConfigs-docs/Architecture/Deploy State Triage.md

@@ -0,0 +1,151 @@
+---
+type: reference
+title: Deploy State Triage
+description: Reading and reconciling the deploy-state markers — stale marker vs live container, pending-service lockout, interrupted/raced deploys — without restarting healthy heavy endpoints.
+tags: [deployment, deploy-state, systemd, podman, triage]
+timestamp: 2026-10-05T00:00:00-07:00
+---
+
+# Deploy State Triage
+
+The deploy-state store is `~/.local/state/serverAppConfigs/deploy-state/<svc>`
+on each target (rootless, user `dank`, format exactly two lines
+`revision=<sha>` / `status=deployed|pending`, written atomically via tmp file
++ `mv`). It is written ONLY by a `deploy.sh` dispatch.
+
+## 1. Stale marker vs live container
+
+**Why it happens:** an out-of-band relaunch — reboot, `systemctl restart`, or
+a restart after the checkout ff-pulled to a newer revision — starts the
+container from the CURRENT `scripts/<svc>.env` but does NOT advance the
+marker. So `marker != HEAD` means "not dispatched via deploy.sh", NOT
+"behavior pending".
+
+**Why it matters:** a plain `deploy.sh` run re-selects and restarts a service
+whose marker lags. On llama.cpp/R9V endpoints that is a ~16–19 min reload of
+a 100+ GiB payload — and frank-primary (dankserver:7770) is also the
+harness's own model backend, so a blind deploy is a self-inflicted outage for
+possibly zero behavioral change.
+
+**Procedure (read-only first):**
+
+1. Committed state: `git log --oneline main..<branch>` empty ⇒ merged; the
+   operational delta is usually one `scripts/<svc>.env`.
+2. Probe the target (ssh as `dank`, passwordless sudo present):
+   - `cat ~/.local/state/serverAppConfigs/deploy-state/<svc>`
+   - `git -C ~/serverAppConfigs rev-parse HEAD`
+   - `systemctl is-active/is-enabled <svc>`;
+     `curl -s -o /dev/null -w '%{http_code}' http://127.0.0.1:<port>/health`
+   - the on-host env file already carries the new keys (it landed via the
+     ff-pull).
+3. **Decisive — read the LIVE container, not the marker.** Use `podman` as
+   `dank` (rootless store; bare `docker` is NOT the shim → "No such object";
+   `sudo podman` inspects an empty store):
+   - `podman inspect <ctr> --format '{{.State.StartedAt}}'` — booted AFTER
+     the pull ⇒ it read the new env
+   - `podman inspect <ctr> --format '{{.Args}}' | tr ' ' '\n' | grep -i <flag>`
+   - engine env: `podman exec <ctr> sh -c 'for f in /proc/[0-9]*/environ; do
+     tr "\0" "\n" <$f 2>/dev/null; done | grep -i <VAR>'`
+   - confirm the launcher maps each tunable to its flag
+     (`grep scripts/launch.sh`), e.g. `R9V_ENABLE_PREFIX_CACHING=1 →
+     --enable-prefix-caching`.
+
+**Decision:** live argv+env == target config AND `/health` 200 ⇒ behavior is
+live — **do NOT restart**; reconcile the marker (below). Only if the live
+container lacks the flag/env (or predates the pull) is a real
+`deploy.sh <target> --service <svc>` warranted.
+
+**Reconcile (no restart)** — mirror the deployer's atomic idiom:
+
+```bash
+S=$HOME/.local/state/serverAppConfigs/deploy-state
+H=$(git -C ~/serverAppConfigs rev-parse HEAD)
+t=$(mktemp "$S/.<svc>.XXXXXX"); printf 'revision=%s\nstatus=%s\n' "$H" deployed > "$t"; mv -f -- "$t" "$S/<svc>"
+```
+
+Then prove the gate: `./tests/run.sh` + `tests/e2e/run.sh` (needs
+`tests/e2e/.env.local`; expect the documented `11-admin` red if the
+FrankBook-Pro Mac mini 192.168.0.112 is ASLEEP — confirm with `ping -c2` +
+a `/dev/tcp` probe: 100 % loss + timeout = host off, not a regression).
+Log to `serverAppConfigs-docs/log.md`; commit + push (docs-only ⇒ deploy
+no-op).
+
+**Anti-patterns:** trusting a commit message ("X deployed") or the marker
+alone — both are bookkeeping; the container's argv/env is ground truth.
+Reading container state with `sudo podman` / bare `docker`. Restarting a
+healthy heavy endpoint to satisfy a stale marker.
+
+## 2. Pending lockout
+
+Symptom: `deploy.sh <target>` fails or silently stops with
+`service <name> is not deployed on <target>` and the `[3/3]` e2e gate never
+appears.
+
+1. **Get the real exit code first:** `./scripts/deploy.sh X 2>&1 | tail`
+   reports TAIL's 0 — use `set -o pipefail` or redirect to a file, then
+   `echo $?`.
+2. Inspect the state store (no sudo):
+   `ssh <target> 'cat ~/.local/state/serverAppConfigs/deploy-state/<name>'`.
+   A `status=pending` entry for a service absent from `service_is_allowed`
+   in `scripts/deploy-remote.sh` = retired-service lockout.
+3. Fixed behavior (since 02a7876, 2026-09-17): `add_pending_services` prunes
+   not-allowed pending entries to
+   `~/.AGENTTRASH/<YYYYMMDD>-retired-deploy-state/<service>` and the run
+   continues. Verify the WARN line names the service and the state file is
+   gone.
+4. On a host without the fix: move the stale entry to
+   `~/.AGENTTRASH/<date>-retired-deploy-state/` manually (never `rm`), re-run.
+5. Verify: deploy reaches `[3/3] Running Playwright e2e gate` and completes;
+   `deploy-state/` no longer holds the retired name.
+
+Related contracts: the superseded seam
+(`deployment_scoping.bats` "remote runner recognizes a pending service at
+the same revision") now asserts prune-and-continue (exit 0), not exit 2;
+`deployment_flow.bats` extracts `add_pending_services` via sed and
+HOME-overrides the trash path. **When retiring a service going forward:**
+delete or reconcile its deploy-state entry as part of the retirement commit —
+the prune is a safety net, not the intended cleanup path.
+
+## 3. Interrupted or raced deploys
+
+The deploy is TWO processes: `deploy.sh` (dev box) and `deploy-remote.sh`
+(runs on the target over ssh). **Killing the dev-side job does NOT kill the
+host-side one** — two `deploy-remote.sh` processes can end up managing the
+same unit, one deploying a revision you already superseded. Check:
+`pgrep -af "deploy-remote"` on the target, read the revision argument on
+each, kill stale ones by explicit PID.
+
+A killed deploy leaves behind:
+
+- the unit **stopped/failed** (`Restart=always` does NOT fire after an
+  explicit stop — start it yourself);
+- the state marker **pending** (pre-deploy or in-flight revision);
+- the e2e gate un-run (the deployer runs it last).
+
+Reconcile the marker ONLY after the service itself is verified healthy — a
+marker that runs ahead of reality is worse than one that lags (the planner
+uses it to decide what is pending). Then close the loop:
+
+1. Verify against MEASURED, config-specific numbers (KV capacity line, arena
+   slots, argv flags, `NRestarts`), not just health 200.
+2. Run the gate the dead deploy skipped (`tests/e2e/run.sh` from a checkout
+   that has the gitignored `tests/e2e/.env.local` — a fresh `git worktree`
+   does NOT have it; copy it in first).
+3. If the killed revision was restoring a known-good shape, confirm the
+   numbers match the last measured-good boot of that shape — equality is the
+   proof, not health alone.
+
+Boot failures must be read from the journal, not `podman logs`: serve scripts
+`podman rm -f` the container on every start (previous attempt lost), and
+`podman logs` on a container still *Stopping* returns the OLD configuration's
+numbers — which looks exactly like your change having no effect. Key on
+container identity (`podman ps -a --format '{{.CreatedAt}}'`) and read the
+cause from `journalctl --since "N min ago" | grep -a "<container-name>\["`.
+See [[vLLM Boot Forensics and Config Interrogation]] for the full capture
+procedure.
+
+## Related
+
+- [[Deployment Flow]] — the deploy pipeline itself
+- [[Quadlet Lifecycle]] — unit ownership and drift diagnosis
+- [[vLLM Boot Forensics and Config Interrogation]] — journal capture

serverAppConfigs-docs/Architecture/Deployment Flow.md

@@ -63,7 +63,7 @@ service on the other host.
 | Dankclaw compose services | `podman-compose -f dankclaw/compose.yaml up -d --no-deps NAME` | Recreates only `NAME`; Caddy is handled separately because it fronts both hosts. |
 | SearXNG, Gluetun, Valkey, OpenWebUI, Pi-hole, Gitea database/server | user Quadlet + `systemctl --user` | Installs the selected unit and restarts only the required dependency-coupled units. Gitea's web proxy is the canonical Caddy compose service. |
 | `frank-large` | systemd + `systemctl` (root unit) | Relinks and restarts only the large inference service; waits up to 20 minutes for `/health`. |
-| `frank-small` | systemd + `systemctl` (root unit) | Relinks and restarts only the small inference service; waits up to 20 minutes for `/health`. |
+| `frank-secondary` | systemd + `systemctl` (root unit) | Relinks and restarts only the small inference service; waits up to 20 minutes for `/health`. |
 | `frank-medium` | rootless Quadlet + `systemctl --user` | Installs `dankserver/frank-medium/quadlet/frank-medium.container`; on first deploy retires the llama.cpp root unit and the vLLM replica quadlets into the home trash, then restarts the generated unit. Waits up to 20 minutes for `/health`. |
 | Dankserver media stack (RomM trio, Jellyfin, Calibre-Web, Copyparty) | user Quadlet + `systemctl --user` | Installs `dankserver/<svc>/quadlet/*.container`, restarts the generated unit. The first cutover per service is idempotent: it retires the compose-era bare container, rescues any anonymous volume into the data root, and merge-moves `/opt/<svc>` to `/home/dank/Containers/<svc>` (mapped-uid roots move without chown). Steady state is restart-only. |
 

serverAppConfigs-docs/Architecture/Quadlet Lifecycle.md

@@ -0,0 +1,209 @@
+---
+type: reference
+title: Quadlet Lifecycle
+description: The full Quadlet discipline on this fleet — ownership auditing, compose-to-Quadlet cutover, drift/crash-loop diagnosis, the podman 5.x v1 generator quirks, log retrieval, and compose bring-up with ground truth.
+tags: [podman, quadlet, systemd, boot-persistence, compose, drift]
+timestamp: 2026-10-05T00:00:00-07:00
+---
+
+# Quadlet Lifecycle
+
+Fleet rule (AGENTS.md "Boot Persistence & Self-Healing"): every deployed,
+working container MUST have a self-healing, boot-persistent Quadlet unit —
+`.container` file in the repo (`<host>/<svc>/quadlet/`), installed by the
+deployer, `[Install] WantedBy=default.target`, `Restart=always`, linger on.
+A container that runs only from compose dies silently at reboot. This page
+is the operating manual for that lifecycle.
+
+## 1. Ownership audit — find the containers that will not survive a reboot
+
+```bash
+for c in $(podman ps --format '{{.Names}}'); do
+  printf '%-26s %s\n' "$c" "$(podman inspect $c --format '{{index .Config.Labels "PODMAN_SYSTEMD_UNIT"}}')"
+done
+```
+
+- **label present** → the unit owns it; persistence = `[Install]` + `loginctl
+  show-user <user> -p Linger` = yes.
+- **label absent, compose service exists** → the classic gap: dies at reboot.
+  Check the publish address while there — compose forms often bind the
+  wildcard (an unauthenticated UI/API on every interface).
+- **label present AND a compose service with the same name** → two owners; a
+  stray `podman compose up` can create a duplicate that fights the unit for
+  the name and the port. Remove the compose service, leave an explanatory
+  comment (repo convention).
+
+## 2. Cutover: compose service → Quadlet unit
+
+Checklist (precedents to copy:
+`dankclaw/{caddy,gitea,searxng,open-webui,seerr}/quadlet/`,
+`dankclaw/observability/quadlet/`):
+
+1. Unit file at `<host>/<svc>/quadlet/<name>.container`, mirroring compose
+   line for line: `Image=` (`localhost/` prefix for locally built images +
+   `Pull=never`), `ContainerName=` (Caddy and scrape configs address
+   containers by name), `Network=`, `User=0:0` when the image's own user
+   cannot write a rootless-owned bind mount, `Volume=` (`:ro` preserved),
+   `Exec=` for compose `command:`, `PublishPort=`, health keys,
+   `[Service] Restart=always` + `RestartSec=10`, `[Install]
+   WantedBy=default.target`. **Never `systemctl --user enable` the generated
+   unit** — `[Install]` + linger IS the persistence (`is-enabled` = `generated`
+   is healthy; explicit enable fails with "transient or generated").
+2. Exposure: publish on `127.0.0.1` unless something off-host genuinely needs
+   it (Caddy reaches containers by name over the shared network; the wildcard
+   publish is a guard-chain bypass).
+3. Handler in `scripts/deploy-remote.sh`: `install_quadlet <dir> <name>` →
+   `systemctl --user restart <name>.service` → `wait_for_health`; retire any
+   legacy hand-written unit into `~/.AGENTTRASH/` (a leftover
+   `~/.config/systemd/user/<svc>.service` that shells out to
+   `podman compose up` collides BY NAME with the generated unit);
+   `podman rm -f <container>` so the unit's container can take the name.
+4. Planner: map `<host>/<svc>/**` → service in `scripts/deploy-plan.py`
+   AND add a `service_is_allowed` entry; keep `SERVICE_ORDER` dependencies
+   ordered. Add a `test-changed` case above the generic
+   `*/quadlet/*.container` fallback.
+5. Retire the compose entry in the SAME change — two owners is how a stale
+   duplicate comes back.
+6. Seam test `tests/<svc>_quadlet.bats`: boot persistence, restart policy, no
+   wildcard publish, container name matches Caddy/scrape references, health
+   command, deployer installs+restarts+waits, never `rm`, never `enable`.
+   Migrate any older assertion that read the compose file for devices/flags.
+7. Deploy `--service <name>`; prove: `is-active`/`is-enabled` =
+   active/generated, container `healthy`, endpoint answers,
+   `list-dependencies default.target` shows the unit, off-host probes on
+   previously published ports now refuse, and the container no longer carries
+   `io.podman.compose.project`.
+
+**Traps that cost real time:**
+
+- `HealthCmd` = CMD-SHELL (podman runs `/bin/sh -c`); a distroless image has
+  no shell → the check fails forever. `--health-cmd` won't take exec form
+  either. Put `HEALTHCHECK … CMD ["binary","arg"]` in the image and build
+  with `--format docker` — the default OCI format has no HEALTHCHECK field
+  and drops it with only a warning.
+- Never point a unit's `EnvironmentFile` at the repo-root `.env` — it is a
+  gluetun env carrying `HOME=/root`; systemd sets HOME for the service and
+  the unit dies with `stat /root/.config: permission denied`. Use a
+  per-service env under `/home/dank/Containers/<svc>/env`, seeded by the
+  handler with a single-variable filter, validated so a missing secret fails
+  loudly.
+- `Network=host` exporters scraped by containerized Prometheus must bind the
+  host's LAN address (containers cannot reach host loopback) PLUS `127.0.0.1`
+  for the homepage probe — two `--web.listen-address` flags, never the
+  wildcard.
+- alloy 1.x refuses the `ui` config block; the unit must pass
+  `--server.http.listen-addr=0.0.0.0:12345` before its port can be published.
+- GPU services: `AddDevice=` + `GroupAdd=keep-groups` (user-unit spawns carry
+  no supplementary groups).
+- When the last service leaves a compose file, the compose-validity seam
+  fails (requires ≥1 service): keep the file as comments and relax the seam
+  deliberately, or delete the file and fix the planner's compose-diff path —
+  decide with the owner, the deploy tooling reads it.
+
+## 3. Drift / crash-loop diagnosis (5 minutes, in order)
+
+Symptom: crash-looping or perpetually unhealthy container whose DEFINITION
+looks right; or a service quietly down for days because a restart policy kept
+something running. Cause: the running container was created from an OLDER
+definition — a compose/unit edit does not touch an existing container, and
+the restart policy faithfully resurrects it forever.
+
+1. **Who started it:** `podman inspect <name> --format 'unit={{index
+   .Config.Labels "PODMAN_SYSTEMD_UNIT"}} restart={{.HostConfig.RestartPolicy.Name}}
+   created={{.Created}}'`; compare `created` with when the definition changed
+   (`git log -p --follow <config>`). Created-before-the-edit is the signature.
+2. **What it is ACTUALLY running:** `{{.Config.CreateCommand}}` is the truth.
+   `{{.HostConfig.Devices}}` / `{{.HostConfig.GroupAdd}}` print empty/0 even
+   for containers started with `--device`/`--group-add` — a false negative
+   that sends you hunting the wrong root cause.
+3. **What the process says:** `podman logs <name> | grep -iE 'error|cannot|Runtime'`
+   (a capability error like `No HIP GPUs are available` beside a missing
+   device flag confirms drift). Count the damage: `{{.RestartCount}}` — one
+   instance reached 9,146 restart attempts on stale config.
+
+Quickest unblock: `podman compose up -d --force-recreate <svc>`. Durable fix
+= the §2 cutover. **Verify the capability, not the status:** `podman exec
+<name> python3 -c "import torch; print(torch.cuda.is_available())"`;
+`ss -ltn | grep <port>` for the actual bind addresses. ComfyUI incident:
+2026-09-16, stale device-less container, 9,146 restarts. History repeats:
+2026-08-26 Caddy (no unit), 2026-09-11 five observability services.
+
+## 4. Podman 5.x quadlet v1 generator quirks (verified live 2026-08-28)
+
+- **Hard-fail keys** — the whole unit silently fails to generate
+  (`systemctl --user cat <unit>` shows "could not be found"): any
+  dash-prefixed `[Container]` option (`--shm-size=…` → `unsupported key …
+  in group 'Container'`); `Command=` is unsupported, use `Exec=`.
+- **HealthCmd space-splitting:** the v1 generator splits `HealthCmd` on
+  spaces into the healthcheck argv; a quoted probe
+  `/bin/sh -c "pg_isready …"` lands as argv token `"pg_isready` →
+  command-not-found → container reads **unhealthy while the DB is up**.
+  HealthCmd must contain only space-free tokens
+  (`HealthCmd=pg_isready -d authentik -U authentik`). If health reads
+  unhealthy but logs look fine, compare the baked
+  `{{.Config.Healthcheck.Test}}` array against the intended command before
+  believing the status.
+- **Probe a new key before shipping:** drop a throwaway `probe.container`,
+  `daemon-reload`, `journalctl --user --since "1 min ago" -o cat | grep -i
+  "unsupported key"`, `systemctl --user cat probe.service` (shows the
+  generated ExecStart verbatim), then remove + reload.
+
+## 5. Rootless Quadlet log retrieval
+
+Step 0 — establish ownership: `systemctl --user status <name>.service`
+(works ⇒ rootless user unit); `loginctl show-user <user> -p Linger`. If
+`--user` knows the unit, EVERY command runs as that user with NO sudo.
+
+| Need | Command |
+|---|---|
+| Current instance tail/follow | `podman logs --tail 200 / -f <name>` |
+| Across restarts/recreates | `journalctl --user -u <name>.service -n 200 --no-pager` |
+| Time-bounded / crash-only | `--since "14:30"` / `-p warning` |
+| Kernel-side evidence (OOM, SVM) | `sudo journalctl -k --since …` |
+
+Quadlet units commonly run `--replace --rm`: **`podman logs` is destroyed on
+recreation** — the journal is the only durable record. Traps:
+`sudo podman logs` → "no such container" (root's store does not contain
+rootless containers; same for `sudo podman ps`); a system-level
+`journalctl -u <name>` returns nothing for a user unit; during first-start
+weight downloads health reports unhealthy BY DESIGN (check the payload file
+is growing). Verbose vendor switches are a config change — through git and
+deploy.sh, never a live env poke.
+
+## 6. Compose bring-up with ground truth (rootless hosts)
+
+1. **Audit reality before pressing buttons:** `podman ps -a` runs per-user —
+   check every namespace (`sudo podman ps -a`) and native units
+   (`systemctl list-units`) before declaring a service absent. Compare
+   compose mount sources against the real filesystem — compose files name
+   phantom paths (e.g. `/mnt/media/*` vs the real LUKS array at
+   `/mnt/entropy`). Fix compose through git first, deploy second.
+2. **Pre-create bind-mount sources** (`sudo mkdir -p` + `chown 1000:1000`)
+   or creation fails `statfs …: no such file or directory`.
+3. **Staged bring-up:** infra first (`up -d mariadb redis`), poll
+   `{{.State.Health.Status}}`, then the rest. Fresh containers answer
+   000/refused for the first seconds — poll.
+4. **Never guess image tags:** ghcr.io answers 403 on the bearer-token
+   request for nonexistent/private repos ("manifest unknown"/token len 0 =
+   repo or tag does not exist, not auth). Query the registry API
+   (`ghcr.io/token?scope=repository:<ns>/<repo>:pull` →
+   `/v2/<ns>/<repo>/tags/list`).
+5. **Read the interface from inside the image** before guessing flags/envs:
+   `podman exec <ctr> cat /z/cpp.sh` etc. HTTP 500 with a styled error page
+   means RUNNING-but-misconfigured, not down. Copyparty ac ground truth:
+   entrypoint `/bin/ash /z/cpp.sh -c /z/initcfg`; accounts/volumes are ARGV
+   flags (`-a USER:PASS`, `-v SRC:DST:PERMS`), not envs — and with none
+   configured it serves cwd read/write for everyone (an unauthenticated
+   world-writable share).
+6. Remote-shell quoting: zsh eats bare `===`/`?` tokens; a zero-match from a
+   mangled pattern is not absence — re-verify the target exists.
+7. `user: "1000:1000"` SIGSEGV at boot = unmapped uid in the rootless userns
+   (host 1000 == container 0): drop the user line, run as container-root.
+
+## Related
+
+- [[Deployment Flow]] — the deployer that installs these units
+- [[Stale Signal Audit]] — the audit that catches drift's victims
+- [[Container Network Namespaces]] — reachability between containers
+- [[Dankclaw/ComfyUI]] — the 9,146-restart incident
+- [[Infrastructure/Observability]] — same class, twice

serverAppConfigs-docs/Architecture/Stale Signal Audit.md

@@ -0,0 +1,144 @@
+---
+type: reference
+title: Stale Signal Audit
+description: The audit battery for surfaces that rot without failing — unreachable probes, vacuous assertions, invalid checkers, gates that never run, and prose invalidated by a change. Four passes found real defects every time.
+tags: [testing, observability, audit, signals, gates]
+timestamp: 2026-10-05T00:00:00-07:00
+---
+
+# Stale Signal Audit
+
+Run after any change and before trusting a green gate. The productive question
+is **which signal here cannot possibly be correct?** Every pass on this repo
+(4 of 4) found real defects — including one in the rule file every future
+agent reads first.
+
+## The surfaces, and what each vantage point can reach
+
+| Surface | Producer's vantage point | Can reach |
+|---|---|---|
+| Homepage status cards | the `homepage-status` timer on **Dankclaw** | Dankclaw's loopback, Dankclaw container names, **published** LAN addresses |
+| Prometheus scrape jobs | the `prometheus` container on Dankclaw | container names on `dankclaw_default`, LAN addresses, that host's loopback only |
+| bats / e2e suites | the dev box | the repo, plus the live fleet over HTTPS/LAN |
+
+Anything else — another host's loopback, a container-internal port never
+published — is unreachable no matter how healthy the service is. The failure
+mode is always the same shape: a signal that **cannot** succeed by
+construction, which teaches the operator to ignore the surface it lives on.
+
+## 1. Permanently-red probes (reachability)
+
+For every monitor/probe/alert target ask: *can this checker physically reach
+that endpoint?* Seen classes: a card probing another host's container-internal
+port; a "probe" job scraping `/` (HTML → `unsupported Content-Type`); a
+`/metrics` that is disabled (Gitea 404).
+
+- A scrape cannot express "is this HTTP service answering?" — the right probe
+  is the **blackbox exporter** in probe mode (which also gives guarded vhosts
+  a real uptime signal, since Prometheus cannot solve an Anubis challenge).
+- **Fix in the direction of honesty, never of silence:** unreachable target →
+  mark the homepage card `wip: true` with the reason in `description` (the
+  checker skips `wip`; `wip` is the existing convention for "real service,
+  not probeable"), or repoint the job at something reachable. NEVER open a
+  database port on the LAN to make a badge green, and never delete a scrape
+  job — an absent scrape and a healthy one are indistinguishable.
+- Alert rules must not depend on always-red signals — scope them to services
+  that can actually be healthy (`LlmEndpointDown` is scoped to `frank-*` for
+  exactly that reason).
+- A target's failure STRING can prove a fix: a `/metrics` scrape flipping from
+  `connection refused` to `unsupported Content-Type "text/html"` means the
+  service came up and only the scrape is wrong. Read the error, not just
+  up/down.
+
+## 2. Vacuous assertions (highest severity — the gate is already dead)
+
+A seam that greps a file passes *by absence* once the file is retired:
+`grep` exits 2 on a missing path and `!` inverts it.
+
+```bash
+grep -rn 'grep.*"\$REPO_ROOT/[^"]*"' tests/*.bats | head -50
+[ -f "$path" ] || echo "MISSING: a seam may be grepping this"
+```
+
+Fix shape — one place asserts the invariant, each local site keeps a live
+check:
+
+```bats
+@test "no <host> compose file competes with the quadlet units" {
+    local cf="$REPO_ROOT/<host>/compose.yaml"
+    [ ! -f "$cf" ] || { echo "$cf is back: every container is unit-managed"; return 1; }
+}
+# at the per-service site: still fails if the entry returns
+[ ! -f "$cf" ] || ! grep -qE '^\s+<svc>:' "$cf"
+```
+
+**Prove teeth:** recreate the offending file
+(`printf 'services:\n  <svc>:\n    image: x\n' > <path>`), run the seams,
+confirm RED, then move the file to trash. A re-armed check you have never
+seen fail is still unverified.
+
+## 3. Gates that never run
+
+Playwright specs in `tests/e2e/` are matched by **number prefix** in
+`playwright.config.ts`; an unregistered spec silently never runs and the
+suite count stays the same. Count tests before/after;
+`tests/e2e_scaffold.bats` fails on an unregistered spec by design.
+
+## 4. Validate the checker before acting on it
+
+A checker's output is a hypothesis about another tool's semantics. Modelling
+Obsidian with naive text matching produced 45 "broken links" of which 3 were
+real — the rest were case-sensitivity (a `qBittorrent` link to
+`qbittorrent.md`), path-qualified resolution (a path-qualified
+`folder/Note` link resolves by path SUFFIX, not
+stem), and `%20` URL-encoding in markdown links. Fix the checker first,
+re-derive the report, only then edit documents. The result lives in
+`scripts/check-vault-links.py` with those three rules in its docstring,
+gated by the `no unresolved links in the docs vault` seam. Same class: a
+`find … | head -1` selector that reads the wrong file; a truncated listing
+that makes you "fix" a file that exists.
+
+## 5. Prose invalidated by your own change
+
+After retiring a file or adding a feature, grep for EVERY reference, not just
+code:
+
+```bash
+grep -rn "<retired-file>" --include="*" . | grep -v '^./.git/'
+grep -rn "no <feature>\|none —\|unconfigured" --include="*.md" --include="*.container" --include="*.yml" .
+```
+
+This catches the ones that bite: AGENTS.md said ComfyUI runs CPU-only long
+after it was restored with GPU access; a unit said "there are no alerting
+rules" hours after five were added; a docs page documented three scrape jobs
+as working when they could never report up. **Clean cutover includes prose** —
+docs, unit docstrings, and the repo's rule file (the most expensive place to
+be stale).
+
+## Order and verification
+
+1. Enumerate the surfaces (status board, metrics targets, alert rules, test
+   gates, docs/rule files, dashboard panels).
+2. For each, ask what the signal CAN express, not what it says.
+3. Validate any checker before editing the things it flags.
+4. Fix at the root; never weaken an assertion to make it pass.
+5. Gate the class so it cannot return (a seam), then prove the seam fails on
+   the defect.
+
+Verify: `./scripts/check-vault-links.py` (exit 1 on breaks); query each
+Prometheus panel expression against the API (0 series on a panel that should
+show data is a defect; 0 `ALERTS` when nothing fires is not); read the
+deployed `/srv/homepage/status.json` and triage every `state != up` entry
+against reality; query `/api/v1/targets?state=active` and print
+`health != "up"` with `lastError`; `./tests/run.sh` → `ALL TESTS PASSED`.
+Report per finding: the surface, why the signal could not have been correct,
+the fix, and the red/green evidence — flag any regression you introduced
+yourself in the same breath (one was found this way: a tightened bind killed
+a reverse-proxied vhost while every gate stayed green, because the spec
+covering that hostname did not exist).
+
+## Related
+
+- [[Testing Policy]] — the seam inventory this audit walks
+- [[Observability]] — the scrape/probe surfaces it audits
+- [[Quadlet Lifecycle]] — the ownership labels the probes read

serverAppConfigs-docs/Dankclaw/ComfyUI Qwen-Image Workflows.md

@@ -0,0 +1,121 @@
+---
+type: howto
+description: "Qwen-Image 2.1 workflows on Dankclaw: two proven txt2img workflows (art + legible text), the API recipe, and which models/LoRAs to add next."
+tags:
+  - dankclaw
+  - comfyui
+  - qwen-image
+timestamp: 2026-10-05
+---
+
+# ComfyUI Qwen-Image Workflows
+
+Two ready-to-load text-to-image workflows for the Qwen-Image 2.1 payload
+already on Dankclaw, both proven end-to-end on 2026-10-05 by POSTing to the
+live API on port 8188 and verifying the rendered PNG. The repo copies are the
+source of truth; the host copies under `/home/dank/comfyui-data/workflows/`
+are for convenience.
+
+## What is on disk
+
+| Path (under `/home/dank/comfyui-data/models/`) | Size | Role |
+|---|---|---|
+| `diffusion_models/qwen_image_2.1_int8_convrot.safetensors` | 6.8 GB | **The one the workflows use** — int8-convrot quant, ComfyUI-core native |
+| `diffusion_models/qwen_image_2.1_bf16.safetensors` | 14 GB | Full-precision fallback |
+| `diffusion_models/qwen-image-2.1-Q8_0.gguf` (+ `-arch`) | 7.1 GB | GGUF alternative (needs ComfyUI-GGUF custom node) |
+| `text_encoders/qwen3vl_8b_w4a8.safetensors` | 5.9 GB | **The one the workflows use** — 4-bit-weights/8-bit-act Qwen3-VL 8B encoder |
+| `text_encoders/qwen3vl_8b_int8_convrot.safetensors` | 8.8 GB | Mid quant |
+| `text_encoders/qwen3vl_8b_bf16.safetensors` | 17 GB | Full precision |
+| `vae/qwen_image_2.1_vae_bf16.safetensors` | 645 MB | 2.1 VAE (used) |
+| `vae/qwen_image_vae.safetensors` | 243 MB | Smaller VAE |
+
+No LoRAs are on disk yet — see [Recommended additions](#recommended-additions).
+
+## The workflows
+
+Both are API-format JSON (ComfyUI 0.37 loads them via the UI's **Load** dialog
+too): `dankclaw/comfyui/workflows/` in the repo,
+`/home/dank/comfyui-data/workflows/` on the host. Node chain in both:
+
+```
+UNETLoader (int8_convrot) ──┐
+CLIPLoader (qwen_image, w4a8) ─┤ CLIPTextEncode pos/neg
+VAELoader (2.1 bf16) ────────┤
+EmptySD3LatentImage ─────────┴─ KSampler ─ VAEDecode ─ SaveImage
+```
+
+**Common sampler settings:** `cfg 1.0` (Qwen-Image 2.1 expects CFG-free
+sampling), `euler` + `simple`, seed fixed in the file.
+
+| Workflow | File | Shape | Steps | Made for |
+|---|---|---|---|---|
+| Art | `qwen-image-2.1-t2i-art.json` | 1024×1024 | 20 | General illustration/prose prompts |
+| Legible text | `qwen-image-2.1-t2i-legible-text.json` | 832×1216 | 25 | Posters, captions, any in-image text |
+
+### Legible-text prompt recipe
+
+Qwen-Image 2.1 is one of the best open-weight text renderers, but only if the
+prompt is written for it:
+
+1. **Quote the exact strings**: `large bold white text reading "SOLIDARITY OF
+   THE SOVIET COMPUTER SYSTEMS"` — quoted text is rendered literally.
+2. **Describe layout, not vibes**: say *centered / below it / large vs small*,
+   the typeface family (sans-serif), and the color.
+3. **Say what NOT to draw**: end with `no other words, no watermark`.
+4. Keep the negative side on `misspelled, warped letters, extra text` (already
+   in the workflow).
+5. More steps help text: the text workflow uses 25 vs 20.
+
+Verified render: `SOLIDARITY OF THE SOVIET COMPUTER SYSTEMS` + `DANKCLAW
+DANKSERVER EST 2026` came back crisp and correctly spelled at 832×1216
+(`qwen21-text_00001_.png`, 953 KB). One caveat from the same render: a
+shorter secondary line is where misspellings creep in — proofread the first
+render before shipping.
+
+## Driving it from the API
+
+ComfyUI 0.37 **requires the graph wrapped in a `prompt` key** — a bare
+`{node-id: node}` body gets `{"error": {"type": "no_prompt"}}` (verified
+2026-10-05):
+
+```bash
+curl -s -X POST http://127.0.0.1:8188/prompt -H 'Content-Type: application/json' \
+  -d "$(python3 -c 'import json;print(json.dumps({"prompt": json.load(open("dankclaw/comfyui/workflows/qwen-image-2.1-t2i-art.json"))}))')"
+# -> {"prompt_id": "...", ...}
+curl -s http://127.0.0.1:8188/history/<prompt_id>   # status.status_str == "success"
+```
+
+Outputs land in `/home/dank/comfyui-data/output/`. The queue is FIFO — the
+two workflows above ran back-to-back unattended.
+
+**Note:** the container's UI-saved workflows live in the ephemeral
+`/opt/ComfyUI/user` layer and do NOT survive an image rebuild — hence the
+host-side `workflows/` dir and the repo copy.
+
+## Recommended additions
+
+Quantized preferred, sized for the 8060S iGPU (GTT-backed; the RTX 3090 is
+frank-secondary's). All repos verified 2026-10-05. None are on disk yet.
+
+| # | What | Repo | Grab | Size | Why |
+|---|---|---|---|---|---|
+| 1 | **Lightning 8-step LoRA** (T2I speed) | `lightx2v/Qwen-Image-Lightning` | `qwen_image_lightning_8steps_*.safetensors` (LoRA) | ~1–2 GB | 20 steps → 8 with a `LoraLoader` node; 2–3× faster renders on the iGPU, negligible quality loss |
+| 2 | **Qwen-Image-Edit-2511 GGUF** (image editing) | `unsloth/Qwen-Image-Edit-2511-GGUF` | Q4_K_M (~13.2 GB) or Q4_0 (~11.9 GB) | ~12–13 GB | Unlocks img2img "edit this photo" workflows — a capability class the fleet has zero of; Q4 keeps it iGPU-friendly |
+| 3 | **DiffSynth ControlNets** (structure control) | `Comfy-Org/Qwen-Image-DiffSynth-ControlNets` | `split_files/model_patches/` canny + depth | ~2–3 GB each | Pose/edge/depth-guided layouts — pairs with the legible-text workflow for exact poster composition. The `QwenImageDiffsynthControlnet` node is already in the running ComfyUI |
+| 4 | **Wan 2.2 TI2V-5B GGUF** (first video model) | `QuantStack/Wan2.2-TI2V-5B-GGUF` | Q4/Q5 GGUF (~6–8 GB) | ~6–8 GB | T2V+I2V in one dense 5B model; the gentlest on-ramp into video generation, runs in ComfyUI core (`wan` CLIP type is already in this build's loader enum) |
+
+Placement: #1 → `models/lora/` (create it), #2 → `models/diffusion_models/`,
+#3 → `models/controlnet/` (create it), #4 → `models/diffusion_models/`.
+Fetch with `scripts/download-frank-*.sh` discipline (`-f` + byte-count +
+sha256 checks) or a plain `hf download` with the token in the environment —
+never on the command line.
+
+Skip list: the original `city96/Qwen-Image-gguf` conversions (superseded by
+the 2.1 files already on disk), and any BF16 of #2/#4 (the iGPU does not need
+them at GTT capacity, and they double the disk bill).
+
+## See Also
+
+- [[ComfyUI]] — the service itself (image, unit, GPU access)
+- [[ComfyUI ROCm and Vulkan]] — the live-verified GPU path
+- [[Dankclaw Stack]] — the host

serverAppConfigs-docs/Dankclaw/ComfyUI.md

@@ -110,5 +110,6 @@ The web UI responds on `/`. ComfyUI also exposes `/system_stats`. On this APU, `
 
 - [[Dankclaw Stack]]
 - [[ComfyUI ROCm and Vulkan]] — the live-verified GPU path research
+- [[ComfyUI Qwen-Image Workflows]] — the Qwen-Image 2.1 payload, proven workflows, and recommended model/LoRA additions
 - [[Lemonade and sd.cpp]] — the ggml-family alternative
 - Knowledge base: [[ComfyUI]] (tools)

serverAppConfigs-docs/Dankclaw/Hermes.md

@@ -81,6 +81,178 @@ The wiring lives in the live `/data/config.yaml` under `mcp_servers.chrome-devto
 The npx package cache lives in the container layer, so a recreation (not just a
 restart) pays one first-call npm download; subsequent calls are cached.
 
+## Config-home drift ("cannot connect" to models that answer curl)
+
+`podman exec <container> printenv HERMES_HOME` for each hermes container. If
+the webui reads the image default (`/opt/data`) while the gateway reads
+`/data`, the webui runs the image-default template config (`provider: auto` →
+openrouter.ai with no key = connect failure).
+
+The recreation trap is the same one the volumes table shows: image `VOLUME`
+points get a fresh anonymous volume on every container recreation
+(`podman run --replace --rm`) — edits inside such a mount are lost on
+restart/redeploy. Compare volume ids before/after a restart to confirm.
+
+Fix: pin `Environment=HERMES_HOME=/data` in the quadlet unit (shared state
+root), deploy service-scoped. Verify with the app's own code, not guesses:
+
+```python
+from api import config as C; C.resolve_model_provider("<model-id>")
+# -> (model_id, provider, base_url); bare ids map to the owning provider by
+#    scanning the providers: dict models allow-list; @provider:model forces one.
+hermes_cli.providers.resolve_user_provider(name, cfg["providers"])
+# -> ProviderDef with real base_url; accepts api/url/base_url keys.
+```
+
+Named providers (`providers:` dict) are a catalog + connection source;
+`custom:<slug>` connections resolve only via `custom_providers[]` entries. If
+neither holds the provider name, the request has no endpoint and fails to
+connect. Probe the endpoint with the **exact served model id** — the retired
+FLM/NPU endpoints routed only by exact registry tag and answered HTTP 200
+with `{"error":"Failed to load <tag> model!"}` for unknown tags; status-code
+checks miss this class entirely.
+
+## Live config edits
+
+Live file: `/home/dank/Containers/hermes/data/config.yaml` (dir `drwx------`
+uid 109999; all reads/edits need `sudo`). The repo file
+`dankclaw/hermes/config.yaml.example` is a sanitized backup and **can lag
+live — always diff the live file first; never treat the example as truth**.
+
+1. **Backup first:** `sudo cp -a …/config.yaml …/config.yaml.bak-YYYYMMDD`
+   (in-dir `.AGENTTRASH/` also exists for removals).
+2. **Apply** with surgical `sudo sed -i` or full-file write; YAML comments in
+   the live file are tool-generated — preserve them.
+3. **Sweep stale references** across the whole data dir:
+   `sudo grep -rln "<old-id>" /home/dank/Containers/hermes/data
+   --include="*.yaml" --include="*.json"`. Session files/MEMORY.md are
+   historical — leave them.
+4. **Derived caches:** `webui/models_cache.json` is keyed by a config
+   fingerprint (`_source_fingerprint.config_yaml` mtime/size); after edits it
+   is stale. Let it regenerate lazily or trash it — never hand-edit.
+5. **Restart the model-consuming units** (they parse config at startup):
+   `systemctl --user restart hermes-webui.service hermes.service`.
+   `hermes-dashboard.service` is unaffected by routing changes.
+6. **Verify behaviorally**: one tiny completion per endpoint with the exact
+   served id — POST `/v1/chat/completions` and read the body, never just the
+   status code (endpoint names rotate; see the LLM knowledge base).
+7. **Sync the repo backup** (`dankclaw/hermes/config.yaml.example`) to the
+   live shape, redacting secrets (`password_hash`, phone account stay
+   live-only), then `./tests/run.sh` before commit.
+
+`provider: custom` + `base_url` entries route by base_url and send the model
+id verbatim; named providers resolve via their `api:` URL. **Renaming a
+served alias anywhere (serve scripts, env ALIAS) MUST propagate to this
+config in the same change** or Hermes requests 404 on nonexistent models —
+this exact gap caused the 2026-09-05 outage.
+
+## Auxiliary routing against local backends
+
+When a Hermes feature that calls an auxiliary task (e.g. title generation)
+fails against a local endpoint, the usual cause is a **request shape the
+local runtime does not implement**, not a routing mistake.
+
+Diagnose with the real call path — run the actual function inside the
+container with the app's own constants (the probe cannot drift from
+production):
+
+```bash
+base64 -w0 /tmp/probe.py | ssh dank@192.168.0.143 \
+  'base64 -d > /tmp/probe.py && systemctl --user restart hermes.service && sleep 20 && \
+   podman cp /tmp/probe.py systemd-hermes:/tmp/probe.py && \
+   podman exec -i systemd-hermes /opt/hermes/.venv/bin/python /tmp/probe.py'
+```
+
+Copy the file AFTER any restart (`--replace --rm` discards the container
+layer). The CLI at `/opt/hermes/hermes` uses system python and dies
+`No module named 'yaml'`; use `/opt/hermes/.venv/bin/python`.
+
+```python
+import sys; sys.path.insert(0, "/opt/hermes")
+from agent.title_generator import _TITLE_PROMPT_TEMPLATE, _TITLE_RESPONSE_FORMAT, _clean_title, _extract_title_text
+from agent.auxiliary_client import call_llm
+resp = call_llm(task="title_generation", messages=[...], max_tokens=64,
+                temperature=0.3, extra_body={"response_format": _TITLE_RESPONSE_FORMAT})
+```
+
+What actually breaks, and the fix:
+
+- **`response_format` (json_object / json_schema) is a hard 400 on every
+  local backend here** — neither FreeToken nor halogen implements constrained
+  decoding. Config cannot override it: `call_llm` merges the task's
+  `extra_body` first and then `update()`s the caller's, so the hardcoded one
+  wins. Only temperature and max_tokens have reactive retry paths.
+- **Fix: move that one task to the Responses transport**, which never
+  forwards `response_format`:
+
+  ```yaml
+  auxiliary:
+    title_generation:
+      provider: dankserver-small        # names track the live config; diff first
+      model: frank-secondary
+      api_mode: codex_responses        # real per-task key
+      extra_body:
+        reasoning:
+          effort: none                 # mandatory, see below
+  ```
+
+- **Disable thinking for any cap-limited task.** The adapter omits reasoning
+  only when config says `reasoning.enabled: false`. The
+  `reasoning_effort: none` *shorthand* produces exactly that and leaves the
+  server default (thinking ON) — the 64-token title budget gets spent on the
+  think: empty content, `finish_reason: length`. The explicit
+  `extra_body.reasoning.effort: none` instead sends
+  `{"effort":"none","summary":"auto"}`, which the local `/v1/responses`
+  honors.
+- Both local runtimes also answer `/v1/responses` and `/v1/messages` — no
+  proxy needed for the alternate transport.
+
+Verify and land: re-run the probe with **no** `api_mode`/reasoning kwargs —
+it must pass from config alone (that is the production path); keep the live
+file and the repo example in step; `./tests/run.sh`, commit, push — a Hermes
+config change is docs+example only, no service deploy needed.
+
+Gotchas: `auxiliary.<task>.enabled: false` is checked BEFORE
+`apply_instant_title`, so disabling a broken title task also removes the
+derived title — it does not degrade gracefully the way a failure does. A
+local call can also fail at the model level: a 200 with an error-text body,
+or a chatty non-title answer. Always read the raw content, not the status.
+
+## Webui password rotation ("Hermex")
+
+Auth contract (verified live 2026-08-31): `POST /api/auth/login` with JSON
+`{"password":"…"}` → 200 + signed HttpOnly `hermes_session` cookie (24 h TTL)
+on match, 401 on miss. No username field. `/health` answers 200
+unauthenticated and `/` answers 302 — **never verify auth via those**; probe
+login with a deliberately wrong password (expect 401) and the real one
+(expect 200). Details on the webui surface: [[Hermes-WebUI]].
+
+1. Run the whole probe as a heredoc'd bash script
+   (`ssh -o BatchMode=yes dankclaw 'bash -s'`) — the host's zsh mangles
+   inline `${…}` even inside single-quoted ssh args; never interpolate curl
+   `-w %{http_code}` inline.
+2. Update BOTH env files with
+   `sed -i "s/^HERMES_WEBUI_PASSWORD=.*/HERMES_WEBUI_PASSWORD=<new>/"`:
+   - `/home/dank/serverAppConfigs/.env` ([Service] EnvironmentFile)
+   - `/home/dank/serverAppConfigs/dankclaw/.env` — **the effective one**: the
+     quadlet's `[Container] EnvironmentFile=` is what the container reads at
+     create; rotating only the root file leaves the running container serving
+     the old secret.
+3. `systemctl --user restart hermes-webui.service` (unit recreation re-reads
+   env; no deploy needed if the unit file itself didn't change).
+4. Verify wrong-pw → 401, right-pw → 200, from host loopback **and** a LAN
+   client at `http://192.168.0.143:8787` (webui publishes on the LAN
+   interface since commit `4443897`; off-network ingress is SSH tunnel only —
+   user `tunnel`, port **22**, key-only auth, forward remote end
+   localhost:8787).
+
+Gotchas: `sudo podman exec` fails exit 125 here — root's namespace does not
+see rootless containers; run `podman exec hermes-webui printenv
+HERMES_WEBUI_PASSWORD` unprivileged (the ssh user owns the env files, mode
+600). If rotating because a secret leaked into transcripts/logs, rotate in
+place via a script that fetches the live value from the running container
+(`printenv`) instead of transcribing it.
+
 ## See Also
 
 - [[Hermes-Dashboard]] — the LAN browser management UI

serverAppConfigs-docs/Dankclaw/OpenWebUI.md

@@ -24,29 +24,48 @@ https://github.com/open-webui/open-webui
 | Upstream latest | 0.11.4 (2026-09-21) — security/access-control advisory; `:main` is the rolling image, not a stable pin. Imagegen endpoint is `POST /api/v1/images/generations` (the `comfyui` engine is still upstream). |
 | Port | 3005 (host) → 8080 (container). Port 3000 is Gitea's HTTP, so it is not usable. |
 | Data volume | `/home/dank/Containers/open-webui/data` → `/app/backend/data` |
-| Backends | Three fleet endpoints: `frank-large` (host.containers.internal:7770), `frank-medium` (192.168.0.235:7770), `frank-small` (host.containers.internal:7771) — see [The LLM backends](#the-llm-backends) |
+| Backends | Picker exposes two models — `frank-zealot` (host.containers.internal:7771) and `frank-dragoon` (host.containers.internal:7770, added 2026-10-09). The `frank-archon` (Dankserver) connection remains configured in the DB, disabled — see [The LLM backends](#the-llm-backends) |
 | Network | Pinned bridge `systemd-open-webui` (`10.89.3.0/24`, gateway `10.89.3.1`) — see [History](#history) |
 | Persistence | Quadlet unit `open-webui.service`; `[Install] WantedBy=default.target` |
 | Deploy | `scripts/deploy-open-webui-quadlet.sh dankclaw` |
 
 ## The LLM backends
 
-OpenWebUI serves the three fleet LLM endpoints. The model picker is built from
-the `model` table in `webui.db` (one row per model), NOT from live connection
-discovery — so the picker shows exactly the models seeded there.
+The picker is built from the `model` table in `webui.db` (one row per model)
+plus the `model_ids` of each **enabled** OpenAI connection, NOT from live
+connection discovery — so the picker shows exactly what is seeded there.
+As of 2026-10-09 the picker exposes **two models: `frank-zealot` and
+`frank-dragoon`**.
 
-| Picker model | Connection URL | Runtime | Host |
-|--------------|----------------|---------|------|
-| `frank-large` | `http://host.containers.internal:7770/v1` | unsloth llama.cpp (Strix Halo iGPU, 200k ctx) | Dankclaw |
-| `frank-medium` | `http://192.168.0.235:7770/v1` | vllm-radiance TP=2 (FP8, DFlash2, 262k ctx) | Dankserver |
-| `frank-small` | `http://host.containers.internal:7771/v1` | llama.cpp (RTX 3090, Gemma4-12B, 262k ctx) | Dankclaw |
+| Connection (`openai.api_configs` idx) | URL | Enabled | `model_ids` | Runtime | Host |
+|---|---|---|---|---|---|
+| 0 | `http://host.containers.internal:7770/v1` | yes (2026-10-09) | `frank-dragoon` | halogen flash-server (Strix Halo iGPU) | Dankclaw |
+| 1 | `http://192.168.0.235:7770/v1` | no (2026-10-07) | `frank-archon` | radiance TP=2 (2× R9700) | Dankserver |
+| 2 | `http://host.containers.internal:7771/v1` | yes | `frank-zealot` | llama.cpp CUDA (RTX 3090, 2×200,704) | Dankclaw |
+
+The `model` table carries one decoration row per picker model:
+`frank-zealot` (context length 200,704 = the per-slot window; description:
+Qwen3.5-9B Q6_K + BF16 vision mmproj) and `frank-dragoon` (context length
+262,144 = the per-slot `HALOGEN_CTX`; description: Swift1.5 v2 abliterated,
+MTP, vision ON).
+
+A third gate sits *after* the table, and it is the one that bites family
+accounts (2026-10-07): a model row is only visible to a non-admin user when
+its **access permission** grants them. A seeded row defaults to admin-only,
+so the picker renders "No results found" for every regular user even though
+the connection is enabled, `model_ids` is correct, and the endpoint answers
+`/v1/models` — and `GET /api/v1/models` returns `[]` for that user while an
+admin token returns the model. Fix: set the row's `access_control` to public
+(the `access` column on the `model` row) via the admin UI (Admin Panel →
+Models → visibility) or the DB. The verification battery below must therefore
+run **as a non-admin user**, not only as an admin.
 
 Two gotchas make the wiring non-obvious:
 
 - **llama.cpp `/v1/models` is Ollama-shaped.** llama.cpp returns
   `{"models":[{"name":...}]}`; OpenWebUI's `extract_data()` only reads the
   OpenAI `{"data":[{"id":...}]}` shape, so connection discovery silently misses
-  the llama.cpp endpoints (frank-large, frank-small). The fix is to set
+  the llama.cpp endpoints (frank-large, frank-secondary). The fix is to set
   `model_ids` in each connection's `api_config` — when non-empty, OpenWebUI
   builds the model list directly and never fetches `/v1/models`. frank-medium
   (vLLM) serves the OpenAI shape and would discover fine, but uses `model_ids`
@@ -62,7 +81,72 @@ The Ollama connection is disabled (`ollama.enable=false`).
 
 > All three endpoints are bare-metal systemd services managed by this
 > repository. Model swaps, context sizing, and spec tuning happen there. See
-> the [LLM knowledge base](../LLM/index.md).
+> the [LLM knowledge base](../LLM/index.md). Disabling a connection here only
+> hides it from this picker — the endpoint itself keeps serving every other
+> client (the harness, Hermes, API consumers).
+
+## Container introspection (DB and backend python)
+
+- Working dir for imports: `/app/backend`; package name `open_webui`
+  (underscore), NOT `openwebui`. `os.environ["WEBUI_SECRET_KEY"]` MUST be set
+  BEFORE importing any `open_webui` module (import-time hard requirement);
+  the secret lives at `/app/backend/.webui_secret_key`.
+- **The `sqlite3` CLI is NOT in the container** — use Python's `sqlite3`
+  module (`podman exec open-webui python3 -c "import sqlite3;
+  conn = sqlite3.connect('/app/backend/data/webui.db'); …"`). For config
+  keys use parameterized LIKE queries (avoids shell quoting).
+- Backend python: `podman exec -w /app/backend open-webui python3 -c "…"` —
+  `Models.get_all_models()` is a coroutine (await it); model objects are
+  pydantic `ModelModel` instances (attribute access, not `.get()`), with NO
+  `status`/`enabled` attribute — use `is_active` from the DB.
+- **API auth for verification:** `create_token` lives in
+  `open_webui.utils.auth` (NOT `open_webui.utils.utils`);
+  `get_verified_user` requires an `id` claim (not just `sub`);
+  `SESSION_SECRET = WEBUI_SECRET_KEY`. Mint an admin token inside the
+  container:
+
+  ```
+  podman exec -i open-webui sh -c 'export WEBUI_SECRET_KEY=$(cat /app/backend/.webui_secret_key); \
+    python -c "from open_webui.utils.auth import create_token; \
+    print(create_token({\"id\":\"<UID>\",\"sub\":\"<UID>\",\"email\":\"…\",\"role\":\"admin\"}))"'
+  ```
+
+  (admin UID from the `user` table).
+
+## Verification battery
+
+1. `GET /api/models` (the picker endpoint; response key `data`) → lists
+   EXACTLY the intended models, **as an admin AND as a `role='user'` token**.
+   Do NOT gate on `GET /api/v1/models/list` (the Admin Panel → Models page):
+   on the 0.6.x image it returns `items: []` for connection-backed models
+   (workspace-list only; measured 2026-10-09 on both picker ids).
+2. `POST /api/chat/completions` per model with a trivial prompt ("Reply
+   with exactly one word: pong") → each returns content `pong`. Reasoning
+   models need a generous budget: frank-secondary (Qwen3.5 template) returns
+   empty content + `finish_reason: "length"` at `max_tokens 200` (the
+   thinking block eats it) and answers `pong` at 1500.
+3. A 500 "Server Connection Error" → `podman logs open-webui` for
+   `ClientConnectorError`: a network-reachability problem (host LAN IP vs
+   `host.containers.internal`), not a model problem.
+
+**Durability:** the model table + connections are DB-only (not git-managed);
+a fresh data volume loses them — including the public access grant above.
+For reproducibility, add an idempotent first-start seed script or document
+the setup in the service doc.
+
+## API access (auth-gated)
+
+Every `/api/*` endpoint requires an authenticated session (SSO login JWT)
+or a per-user API key (`auth.enable_api_keys=true`; keys are minted per
+account under Profile → API Keys). Anonymous web visitors get the `pending`
+role (`ui.default_user_role=pending`), which cannot chat. Verified
+2026-10-07 from loopback `:3005` and over the public vhost:
+`GET /api/v1/models/list`, `GET /api/v1/models` and
+`POST /api/chat/completions` all return **401** with no key (also with a
+bogus bearer token); only `/api/config`, `/api/version` and `/health`
+answer publicly, and none of them expose the model list. An external
+program needs an account + API key — there is no keyless access.
+
 
 ## Web search
 
@@ -106,6 +190,18 @@ browser-driven and only touches the two exempt vhosts (`ai.` and `auth.`).
   into the existing local account with the same email — the pre-SSO admin
   (frank8465@protonmail.com) keeps its data. `ENABLE_OAUTH_SIGNUP=true` lets
   other `openwebui_users` create accounts on first login.
+- **First login mints role `pending`.** `ui.default_user_role=pending` (config
+  DB, verified 2026-10-09), so a freshly-provisioned `openwebui_users` account
+  completes SSO and then cannot chat (picker empty) until its role is flipped:
+  `UPDATE "user" SET role='user' WHERE email='<user>@franksdank.cloud';` —
+  run with host `python3` over ssh against
+  `/home/dank/Containers/open-webui/data/webui.db` (no `sqlite3` CLI on
+  Dankclaw; table is `user`, singular; no `active` column). No per-user model
+  grant is needed: both picker models (`frank-zealot`, `frank-dragoon`) carry
+  the global `access_grant` row `(model, <id>, user, *, read)`, so any
+  `role='user'` account sees them. Flip on FIRST-LOGIN SIGHTING — the row does
+  not exist until the account signs in (2026-10-09: five provisioned accounts
+  had no row).
 - **IdP reachability:** the container is on its own bridge (10.89.3.0/24) and
   cannot reach `authentik-server:9000` by name. It reaches the IdP over the
   public URL `https://auth.franksdank.cloud` — Caddy publishes `0.0.0.0:443`
@@ -147,6 +243,23 @@ The quadlet carries a built-in healthcheck (30s interval, 3 retries, 300s start
 
 `podman ps` shows the container as `unhealthy` if either fails. Probe manually at `http://dankclaw:3005/health`.
 
+## Image generation (ComfyUI)
+
+Image generation rides the local [[ComfyUI]] integration, not the chat
+picker — ComfyUI's models never appear in the chat model picker (that is
+chat LLMs only), so ComfyUI needs its own image-generation connection
+configured in OpenWebUI. ComfyUI itself runs GPU-enabled on Dankclaw and
+serves `:8188`.
+
+Endpoint: `POST /api/images/generations` (image `0.11.x`: upstream moved the
+route to `POST /api/v1/images/generations`; the `comfyui` engine is still
+upstream) with `Authorization: Bearer <token>` and body
+`{"model":"imagegen","prompt":"…","n":1,"size":"1024x1024"}`; the response
+carries base64 image data or a URL.
+
+Failure modes: connection refused ⇒ OpenWebUI or ComfyUI is down; timeouts
+on large sizes ⇒ check ComfyUI resources; 401 ⇒ the token is bad.
+
 ## Retired instances
 
 - The compose `open-webui` service (port 3000) was never run: 3000 is Gitea's HTTP port, so it could not bind.
@@ -158,18 +271,64 @@ The quadlet carries a built-in healthcheck (30s interval, 3 retries, 300s start
 - The image tracks upstream `main`; pin a digest in the quadlet unit if reproducible rollback matters more than tracking. (The old "CPU-only host" framing is stale: this APU runs the gfx1151 ROCm build — see [[ComfyUI]].)
 - The default admin account is created on first login; secure it immediately.
 - If the backend is loading (3–8 minute model load), OpenWebUI shows connection errors until the model is ready.
-- The on-host LLM services (frank-large `:7770`, frank-small `:7771`) are reached via `host.containers.internal`, NOT the host LAN IP: the container's netavark bridge cannot hairpin to its own host's non-published ports via `192.168.0.143` (connection refused), and the bridge gateway (`10.89.3.1`) lives in the rootless netns while llama-server runs in the host netns, so it refuses too. `host.containers.internal` (169.254.1.2) resolves to the host on the bridge and works — verified end-to-end for both endpoints. The Dankserver backend (frank-medium) is a different host, reached via its LAN IP `192.168.0.235:7770`.
+- The on-host LLM services (Dankclaw `:7770`, `:7771`) are reached via `host.containers.internal`, NOT the host LAN IP: the container's netavark bridge cannot hairpin to its own host's non-published ports via `192.168.0.143` (connection refused), and the bridge gateway (`10.89.3.1`) lives in the rootless netns while llama-server runs in the host netns, so it refuses too. `host.containers.internal` (169.254.1.2) resolves to the host on the bridge and works — verified end-to-end for both endpoints. The Dankserver backend is a different host, reached via its LAN IP `192.168.0.235:7770`.
 
 ## History
 
 - **2026-08-22 — slirp4netns → pinned bridge.** The unit ran on `Network=slirp4netns`. The slirp4netns helper process died (no OOM, no kill entry, no crash log found), and the container kept running with only `lo` in its netns: published port 3005 accepted TCP and reset every request, so chat served empty responses for ~2 days. Nothing logged an error. Moving to a kernel bridge removes the per-container helper entirely, and the healthcheck's interface check makes the zombie state visible if it ever returns.
 
-- **2026-09-21 — model picker rebuilt for the 3-model fleet.** The picker carried 39 stale rows (37 dead Ollama models + 2 `frank-default-*`) and was missing `frank-medium`. Rebuilt the `model` table to exactly `frank-large`/`frank-medium`/`frank-small` and set `model_ids` on each connection (llama.cpp's Ollama-shaped `/v1/models` is invisible to OpenWebUI's discovery). Pointed the two on-host endpoints at `host.containers.internal` — the bridge refuses the host LAN IP for non-published ports. All three verified end-to-end via `/openai/chat/completions`.
+- **2026-09-21 — model picker rebuilt for the 3-model fleet.** The picker carried 39 stale rows (37 dead Ollama models + 2 `frank-default-*`) and was missing `frank-medium`. Rebuilt the `model` table to exactly `frank-large`/`frank-medium`/`frank-secondary` and set `model_ids` on each connection (llama.cpp's Ollama-shaped `/v1/models` is invisible to OpenWebUI's discovery). Pointed the two on-host endpoints at `host.containers.internal` — the bridge refuses the host LAN IP for non-published ports. All three verified end-to-end via `/openai/chat/completions`.
 
 - **2026-09-24 — SSO via authentik.** OIDC authorization-code flow wired per the [SSO (authentik)](#sso-authentik) section: application/provider/entitlements created in authentik, client secret in the host env file, roles from the `roles` scope claim. E2E coverage in `tests/e2e/specs/18-openwebui-sso.spec.ts`.
 
 - **2026-09-24 — flipped to SSO-only.** The OIDC round-trip verified in a real browser (login view renders no password form, only "Continue with authentik", and the flow lands on a signed-in session), so the local username/password path was retired: `ENABLE_LOGIN_FORM=false` + `ENABLE_PASSWORD_AUTH=false` in the quadlet and `ui.enable_login_form` set to `false` in the config DB (the DB wins over env, so both were changed together). `qa-e2e-admin` was demoted out of the `authentik Admins` group (it keeps `openwebui_users` / `homepage-admins` for app access).
 
+- **2026-10-07 — picker narrowed to `frank-secondary` only.** Per Frank: the only model accessible in OpenWebUI is `frank-secondary`. Rebuilt the `model` table to a single `frank-secondary` row (the old rows `frank-large`/`frank-primary`/`frank-small` are deleted), set `openai.api_configs` indices 0 (frank-large) and 1 (frank-primary) to `enable: false`, and retargeted index 2's `model_ids` from the stale `frank-small` label to `frank-secondary` — llama.cpp serves by alias, so the old label addressed a model name the server does not know. Pre-change snapshot: `/home/dank/Containers/open-webui/data/webui.db.bak-20261007` (+ `-wal`/`-shm`). Verified: `/api/v1/models/list` returns exactly `["frank-secondary"]`, chat completion answers `pong` (needs `max_tokens≥1500` — the Qwen3.5 thinking block consumes a 200-token budget), and anonymous API probes are all 401 (see [API access](#api-access-auth-gated)).
+
+- **2026-10-09 — re-keyed to the Protoss ids (wire rename).** The endpoints
+  began serving `frank-zealot` / `frank-dragoon` / `frank-archon`, so three
+  id-keyed objects had to move in the same window as the alias flip:
+  `model.id`+`name` (`frank-secondary` → `frank-zealot`, `params`
+  `{"context":{"length":200704}}` and the `meta.description` kept),
+  `access_grant` (`resource_type='model'`, `resource_id`, principal
+  `user`/`*`, permission `read`), and `openai.api_configs` per-backend
+  `model_ids` (index 2 = `:7771` is the enabled family backend; indices 0/1
+  = `:7770` and Dankserver `:7770` stay `enable: false`). Snapshot first:
+  `/home/dank/Containers/open-webui/data/webui.db.bak-20261009`. Apply from
+  inside the container (`podman exec -i open-webui python3 -`, sqlite3; the
+  image has no sqlite3 binary) then `systemctl --user restart open-webui.service`
+  (it is a rootless quadlet) to drop the cached model list. Verify in a real
+  browser as the family QA account: the welcome screen selects
+  `frank-zealot` and answers. **A stale id fails silently** — the picker
+  simply lists nothing, because OpenWebUI intersects the backend's
+  `/v1/models` with `model_ids`.
+
+  Law: **the model id is a foreign key in three tables.** Any future alias
+  change on an endpoint OpenWebUI fronts must ship with this re-key, in the
+  same deploy.
+
+- **2026-10-09 — composer is `contenteditable`, not `<textarea>`.** A
+  Playwright probe that waits for `textarea` after SSO times out on the
+  welcome screen: OpenWebUI 0.6 renders the composer as a contenteditable
+  div. The stable anchor is the model button (`getByRole("button", {
+  name: /selected model/i })`) and the welcome text "How can I help you
+  today?"; that snapshot is also the cheapest proof the picker is populated.
+
+- **2026-10-09 — `frank-dragoon` added to the picker.** Per Frank: expose the
+  halogen endpoint (service `frank-large`, `host.containers.internal:7770`)
+  beside `frank-zealot`. Three id-keyed writes (the trio the 10-07 rebuild
+  proved): `openai.api_configs` idx 0 `enable: true` (its `model_ids` already
+  carried `frank-dragoon` from the wire rename), a `model` row (`id`/`name`
+  `frank-dragoon`, `params.context.length` 262144 = the per-slot
+  `HALOGEN_CTX`, description of the v2 abliterated payload), and the global
+  `access_grant` `(model, frank-dragoon, user, *, read)`. Snapshot
+  `webui.db.bak-20261009b`, apply with host `python3` over ssh, then
+  `systemctl --user restart open-webui.service`. Verified: `/api/models`
+  lists `frank-dragoon` as admin AND as the QA `role='user'` account, and
+  `POST /api/chat/completions` through the app answers `pong` at
+  `max_tokens 1500`. Halogen's `/v1/models` is OpenAI-shaped, so discovery
+  would have worked too — `model_ids` stays authoritative.
+
 ## See Also
 
 - [[Dankclaw Stack]]

serverAppConfigs-docs/Dankclaw/Swap Configuration.md

@@ -117,6 +117,18 @@ frank-large `health=200` throughout.
   the model is loaded. Trust the engine's `host memory left for everything
   else` line, not `free`.
 
+- **Filesystem prerequisite (other hosts).** ext4 and xfs take swapfiles
+  directly. btrfs needs `chattr +C` (NOCOW), no compression, and no
+  snapshots in the swapfile path — check `df -T` before planning, or
+  `mkswap` will reject the file.
+- **Renaming an *active* swapfile works (inode-based) but leaves a stale
+  path in `/proc/swaps`** — swap it off first, then rename, then `swapon`
+  (which is exactly what the step-3 consolidation does). Migration is
+  observable while it runs: in the 2026-09-12 run `swapoff` of 4 GiB took
+  65 s and used-RAM rose only ~3 GiB (pages touched during migration), the
+  rest relocating on disk. If used-RAM climbs toward the whole swapped
+  amount, let it finish rather than aborting.
+
 ## Trash
 
 The previous 8 GiB `/swap.img` (dated 2026-07-21) is staged, not deleted, at

serverAppConfigs-docs/Desktop/Noctalia and Niri.md

@@ -0,0 +1,303 @@
+---
+type: reference
+title: Noctalia and Niri
+description: The noctalia v5 desktop shell on niri (workstation + blankpad) — config layers, remote IPC over ssh, custom-palette recolor, offline template rendering, and output rotation/pinning.
+tags: [desktop, noctalia, niri, blankpad, workstation, ghostty]
+timestamp: 2026-10-05T00:00:00-07:00
+---
+
+# Noctalia and Niri
+
+Noctalia (`/usr/bin/noctalia`) is the desktop shell on niri, running on both
+the workstation and blankpad. Since the 2026-09-10 unification both run the
+**bare `noctalia` v5 binary** (niri `spawn-at-startup "noctalia"`); the old
+Quickshell-hosted v4 architecture (`qs -c noctalia-shell`, JSON config) is
+retired on both hosts.
+
+## Config layers
+
+- Hand-edit declarative layer: `~/.config/noctalia/<name>.toml` (e.g.
+  `config.toml`). Main config: `~/.config/noctalia/config.toml`; the niri
+  config includes `noctalia.kdl` for display settings.
+- Runtime state file `~/.local/state/noctalia/settings.toml` loads LAST and
+  **overrides declarative values for the same keys — never hand-edit it**;
+  UI/IPC writes land there.
+- Main reference: `noctalia-reference` facts — IPC `noctalia msg <command>`
+  replaced the `qs -c noctalia-shell ipc call` wrapper; app templates live at
+  `/usr/share/noctalia/assets/templates/` (gtk, ghostty, alacritty, hyprland,
+  sway, wezterm, helix, btop, starship, qt, kde, cava, scroll, mango,
+  umbriel, foot, emacs, kitty); palette generation `noctalia theme [image]`.
+
+## Remote IPC over ssh (the D-Bus trap)
+
+`noctalia msg <command>` resolves the running instance via the D-Bus session
+bus. sshd-spawned non-login shells lack `DBUS_SESSION_BUS_ADDRESS`, so plain
+ssh exec prints `error: noctalia is not running` even when the shell IS
+running — do not conclude "shell dead" from this error. Working remote
+procedure (runs in the systemd user-manager context where the bus address
+exists):
+
+```
+systemd-run --user --pipe --wait noctalia msg <command> [args]
+```
+
+Verified: `msg status` returns live JSON state; unknown panel ids return an
+error listing all valid panel ids (clipboard, control-center, launcher,
+polkit, session, setup-wizard, test, tray-drawer, wallpaper).
+
+Verified v5 msg verb grammar (keybinds-safe): `panel-toggle <id>`,
+`session lock|suspend|lock-and-suspend|logout|reboot|shutdown`,
+`volume-up [step]`, `volume-down`, `volume-mute`, `mic-mute`,
+`media next|previous|playPause` (graceful no-op, rc 0, with no active
+player), `brightness-up [target] [step]`, `brightness-down`,
+`settings-open [context]`, `settings-toggle [context]`, `config-reload`,
+`wallpaper-get [connector]`, `wallpaper-set <path>`, `wallpaper-next`,
+`color-scheme-get|set`, `theme-mode-get`, `templates-apply`,
+`screenshot-fullscreen`.
+
+**Wallpaper directory semantics:** empty/unset `[wallpaper] directory`
+resolves to XDG Pictures via user-dirs.dirs; an explicit directory key
+overrides. Fleet default pattern: point `directory` at the repo-tracked
+wallpapers dir so git-tracked wallpapers serve the shell without copying
+files. After writing config.toml over ssh the file watcher hot-reloads
+automatically; `config-reload` via systemd-run forces it immediately. Note:
+the *current* wallpaper stays the built-in default until the next rotation
+or an explicit `wallpaper-next`; `wallpaper-get` then reporting a path
+inside the configured directory is the confirmation the config took effect.
+
+**Lock chain (unified):** swayidle-lock.service ExecStart on both hosts:
+`/usr/bin/swayidle -w timeout 300 '/usr/bin/noctalia msg session lock'
+before-sleep '…' after-resume '…'`. `msg` resolves via the session bus, so
+no WAYLAND_DISPLAY pinning is needed in unit context.
+
+## Custom-palette recolor (without restarting the session)
+
+Why custom, never the community cache:
+`~/.local/state/noctalia/community-palettes/*.json` is a **sync cache** —
+`.catalog/palettes.json` carries an `md5` per palette and noctalia re-fetches
+from `https://api.noctalia.dev/palettes`, so edits there get reverted.
+Custom palettes live at `~/.config/noctalia/palettes/<Name>.json`
+(`NOCTALIA_CONFIG_HOME` then `XDG_CONFIG_HOME` honored) and are never
+synced. Missing/invalid file → silent fallback to the builtin palette.
+
+Palette JSON shape = the community file's shape: `dark`/`light` maps of
+`mPrimary`, `mOnPrimary`, `mSecondary`, `mTertiary`, `mError`, `mSurface`,
+`mSurfaceVariant`, `mOutline`, `mShadow`, `mHover` plus a `terminal` block
+(`foreground`, `background`, `normal.{black..white}`, `bright.{…}`,
+`cursor`, `cursorText`, `selectionFg`, `selectionBg`). NOTE: `noctalia theme
+-o` emits M3 snake_case tokens (`on_primary_container`, …) — that is the
+*generator* format, not this one. Copy an existing community JSON and mutate
+with `jq`.
+
+Procedure:
+
+1. Read the live scheme first — never assume: `noctalia msg color-scheme-get`
+   (e.g. `community Gruber Darker`); `noctalia msg theme-mode-get`.
+2. Back up settings: `cp -a ~/.local/state/noctalia/settings.toml
+   /tmp/noctalia-settings-backup-$(date +%s).toml`.
+3. Build the palette from the current one (jq keeps structure):
+
+   ```sh
+   mkdir -p ~/.config/noctalia/palettes
+   jq '.dark.mPrimary = "#f43841"
+       | .dark.terminal.normal.yellow = "#f43841"
+       | .dark.terminal.bright.yellow = "#ff4f58"
+       | .dark.terminal.foreground   = "#f43841"' \
+      "$HOME/.local/state/noctalia/community-palettes/Gruber%20Darker.json" \
+      > ~/.config/noctalia/palettes/"Gruber Red.json"
+   ```
+
+   Prefer a hex already present in the palette (its own `mError` red) so the
+   theme stays internally consistent. Set BOTH `normal` and `bright`
+   variants, and both `dark` and `light` if the user toggles modes.
+4. Switch (persists `theme.source = custom` + `theme.custom_palette`):
+   `noctalia msg color-scheme-set custom "Gruber Red"` → `ok`; confirm with
+   `color-scheme-get`.
+5. Re-render every configured template: `noctalia msg templates-apply`.
+   Regenerates `~/.config/ghostty/themes/noctalia`,
+   `~/.config/gtk-{3,4}.0/noctalia.css`, kcolorscheme, foot, wezterm,
+   starship, etc. Symlinked-into-dotfiles paths show up as modified files
+   worth committing.
+6. Reload surfaces: compositor — `niri msg action load-config-file`
+   (`reload-config` is NOT a valid niri subcommand); ghostty — see the trap
+   below.
+
+Verification: `ghostty +show-config | grep -E '^(foreground|palette)'`
+(proves the theme file resolves — note it prints only the delta from
+defaults; cross-check with `ghostty +show-config --default` before
+concluding a key is unset); `noctalia config validate` → `✓ Config is
+valid`; visual via `noctalia msg screenshot-fullscreen` (saves to
+`~/Pictures/screenshot_<ts>.png` and ALSO overwrites the clipboard with the
+PNG — warn the user); `grep -c <old-hex> ~/.config/noctalia/palettes/*.json`
+→ 0 for no leftovers.
+
+Track it in the dotfiles repo (`~/.omp`, the command-center —
+`scripts/deploy.sh` is a stow-equivalent over `dotfiles/<pkg>/` trees):
+mirror the shape for the new palette so `git pull` + `deploy.sh` is the
+whole propagation story — move the JSON into
+`~/.omp/dotfiles/noctalia/.config/noctalia/palettes/` and symlink it back
+into `~/.config/noctalia/palettes/`; re-set the custom palette (re-reads
+through the symlink). Do the link by hand for one file instead of running
+`deploy.sh` — sync mode quarantines and replaces every divergent live file.
+
+Traps:
+
+- **NEVER `kill -USR1` ghostty to reload.** It does not reload config on
+  this fleet — it kills/respawns the terminal (took out Frank's window
+  2026-09-13). Correct: the default keybind `ctrl+shift+,` = `reload_config`
+  (per `ghostty +show-config --default`). Existing windows keep old colors
+  until reloaded; new windows inherit immediately.
+- Palette hex values are NOT a proxy for what the user sees: in
+  `Gruber Darker` the *accent* (`mPrimary`) is yellow `#ffdd33`, so "change
+  the yellow" means the accent too, not just terminal index 3.
+- Saturating terminal text makes ANSI 1/9 indistinguishable from default
+  text (grep highlights flatten) — flag it and offer a lighter red.
+- Contrast sanity: `#f43841` on `#181818` ≈ 4.6:1 (dark-on-color beats
+  white-on-color here, ~3.9:1 — keep `mOnPrimary` dark).
+- Revert = `noctalia msg color-scheme-set community "<original name>"`; the
+  community cache is untouched by the whole flow.
+- Generated theme files symlinked into the dotfiles repo (`gtk-3.0`/`gtk-4.0`
+  `noctalia.css`, ghostty theme) show up as tracked modifications after
+  `templates-apply`; commit them together with the new palette JSON so every
+  machine gets the same recolor.
+
+## Offline template render (when templates-apply never ran)
+
+Failure signature: niri `failed to read included config from
+~/.config/niri/noctalia.kdl: No such file or directory` — the desktop runs
+but with DEFAULT config (niri 26.04 falls back, does not exit): no custom
+keybinds/rules, no noctalia autostart, default output placement. ghostty:
+`theme "noctalia" not found, tried ~/.config/ghostty/themes/noctalia`.
+alacritty: import of `~/.config/alacritty/themes/noctalia.toml` missing.
+
+Root cause pattern: upstream dotfiles commits theme switches (niri include,
+ghostty `theme = noctalia`, alacritty import) that depend on noctalia
+TEMPLATE-GENERATED files. Those files are per-host outputs (rendered from
+the local palette) and are intentionally NOT in the repo; on any host where
+`templates-apply` never ran, they don't exist.
+
+Fix (offline, works while a session runs) — built-in templates live at
+`/usr/share/noctalia/assets/templates/` (declared in `builtin.toml`); render
+each needed one with the live palette:
+
+```bash
+# niri include (plain host-local file; do NOT symlink into repo — regenerated per host)
+noctalia theme --theme-json ~/.config/noctalia/colors.json \
+  --render "/usr/share/noctalia/assets/templates/niri/niri.kdl:$HOME/.config/niri/noctalia.kdl"
+mkdir -p ~/.config/ghostty/themes && \
+noctalia theme --theme-json ~/.config/noctalia/colors.json \
+  --render "/usr/share/noctalia/assets/templates/ghostty/ghostty:$HOME/.config/ghostty/themes/noctalia"
+mkdir -p ~/.config/alacritty/themes && \
+noctalia theme --theme-json ~/.config/noctalia/colors.json \
+  --render "/usr/share/noctalia/assets/templates/alacritty/alacritty.toml:$HOME/.config/alacritty/themes/noctalia.toml"
+```
+
+All template tokens (primary, surface, error, primary_container, …) resolve
+from the flat 16-key colors.json; container tokens are derived, never blank.
+
+Verify: `niri validate` → rc 0; `ghostty +validate-config` → rc 0; next
+login journal shows `loaded config from config.kdl` with NO parse WARN,
+custom output placement (`niri msg outputs` vs display.kdl), noctalia
+autostarted (`pgrep noctalia`).
+
+Traps: **`noctalia msg templates-apply` on 5.0.1 returns "ok" but applies
+NOTHING** (this build's opt-in config schema does not act on it) — do not
+trust its exit code, use the explicit `--render` calls above. Niri config
+errors at boot are NOT fatal — the compositor keeps running on defaults,
+which reads as "some settings silently stopped working", not a crash.
+Regenerate after any palette change; the rendered files are static
+snapshots.
+
+## Niri output rotation and pinning
+
+Verified on blankpad, niri 26.04 (two PM1601 externals + eDP-1 laptop panel).
+
+**Ground truth first:** `niri msg outputs` (connector names, current mode,
+scale, transform, logical pos/size) and `niri --version`. Output **names are
+connector names** (`eDP-1`, `DP-2`, `HDMI-A-1`), not EDID model strings —
+the EDID strings printed before them are decoration. On blankpad the live
+config splits via `include "./cfg/*.kdl"` from `~/.config/niri/config.kdl`,
+and `cfg/display.kdl` is a **symlink into the dotfiles repo** — edit the
+repo path (that is what gets committed); check with `ls -ld`.
+
+**Transform direction is counter-clockwise** (`niri msg output <NAME>
+transform --help`): `90` = content rotated 90° CCW ⇒ the panel's **top
+bezel sits on the LEFT**; `270` = top bezel RIGHT; `180` = upside down;
+`flipped-90`/`flipped-270` = mirrored variants. `90` vs `270` is a
+physical-stand fact no tool can read (portable monitors expose no
+orientation in EDID/MCCS) — pick one, apply, and hand the user the one-liner
+to flip; don't stall on the question.
+
+Two traps when editing the config:
+
+1. **An `output "X" { … }` block resets every property you don't restate to
+   its default.** Pinning position on `eDP-1` without repeating `scale 1.25`
+   silently drops the laptop panel to scale 1.0. Restate `scale` (and
+   `mode`) in every block you touch.
+2. **Niri re-runs auto-layout for outputs with no pinned position.** Pinning
+   only the externals shoved `eDP-1` from `0,0` to `3936,0`. Pin every
+   output's position to keep the left-to-right order. Positions are logical
+   pixels at each output's own scale; x-offsets chain (eDP-1 logical width
+   1536 → DP-2 at 1536 → +1200 portrait width → HDMI-A-1 at 2736):
+
+```kdl
+output "DP-2"     { scale 1 transform "90" position x=1536 y=0 }
+output "HDMI-A-1" { scale 1 transform "90" position x=2736 y=0 }
+output "eDP-1"    { scale 1.25 position x=0 y=0 }   // MUST restate scale
+```
+
+Apply and verify:
+
+```bash
+niri validate -c ~/.config/niri/config.kdl      # exit 0
+niri msg action load-config-file                # live reload; NOT reload-config
+sleep 1; niri msg outputs | grep -E '^Output|Transform|Logical|Scale'
+```
+
+Success looks like `Logical size: 1200x1920` (swapped) plus
+`Transform: 90° counter-clockwise` on each rotated panel, positions matching
+the intended chain. One-off rotation without touching config (temporary):
+`niri msg output DP-2 transform 90`. Commit via the dotfiles repo
+(`~/.omp`, `main`; push to the local Gitea permitted): `git pull --ff-only`
+→ add the display.kdl → commit → push. Pitfalls: `reload-config` is not a
+niri subcommand; never SIGUSR1 a terminal to "reload" (niri config reloads
+do not touch ghostty themes); no man page for niri on this host — `niri
+<sub> --help` and `/usr/share/doc/niri/default-config.kdl` are the docs.
+
+## GPU RGB over I2C (EVGA iCX3)
+
+Turn RGB on/off (or read it back) on an EVGA FTW3 / iCX3-controller GPU
+(the 3090 on dankclaw) using the GPU's own nvidia i2c bus. No display
+engine, no X, no daemon — the mode is EEPROM-saved and survives reboot.
+
+- **Find the controller:** the nvidia i2c driver exposes the card's lighting
+  ports as `/dev/i2c-N`; on the 3090 FTW3 that is ports 1,5,6,7,8 →
+  i2c-1..i2c-5, and only i2c-1 carries a live slave (NVIDIA port 1). Probe
+  each bus. The iCX3 MCU answers at address **0x2D**. `/dev/i2c-*` are
+  `root:i2c 660` — run with `sudo` (dank has passwordless sudo).
+- **Protocol (raw I2C_RDWR, NOT the SMBus ioctls)** — write the register
+  byte first, then the payload:
+  - enable `0xB2` write `04 C6 EB EA 15` (required once before mode writes)
+  - mode `0xC0` write `09 m0 m1 m2 m3 .. .. .. .. SYNC`; m0..m3 are the four
+    zone modes, `MODE_OFF = 0x00`; OFF = `09 00 00 00 00 FF FF FF FF 00`
+  - save `0x90` write `04 9E EB 00 90` (commit to EEPROM — the
+    reboot-persistence step)
+  - fw `0xB1` block read (firmware version)
+  - Read the mode back: write reg `0xC0`, then read 16 bytes; bytes 1..4 are
+    the zone modes (0x00 = OFF), bytes 6..9 the per-zone LED counts.
+- **Verify persistence:** after a reboot, read the mode — all four zone
+  modes `0x00` means the EEPROM save held (confirmed 2026-09-19 post-reboot).
+- **Tool:** a ~90-line I2C_RDWR reader/writer lived at `/tmp/evga_raw.c`
+  (`evga_raw <bus> mode|off|save|enable|fw`) — `/tmp` is wiped by reboot, so
+  recreate it if needed; OpenRGB's `EVGAGPUv3Controller.cpp` carries the
+  same protocol plus the full colour API.
+- Gotchas: raw I2C_RDWR only (the iCX3 answers the raw form, not the SMBus
+  ioctls); the `enable` write must precede the first mode write in a
+  session; without the `save` write the mode is RAM-only and resets on power
+  cycle.
+
+## Related
+
+- [[Command Center Dotfiles]] — the stow-equivalent deploy this page links into
+- [[Hermes]] — the agent that drove the CDP daemon alongside this desktop
+- [[GGUF Compatibility]] — unrelated but shares the "read the file, not the name" lesson

serverAppConfigs-docs/FrankBook-Pro/ARR-Stack.md

@@ -5,3 +5,115 @@ description: "The *arr automation stack (Prowlarr, Sonarr, Radarr, Lidarr, Reada
 tags: [arr-stack, prowlarr, sonarr, radarr, lidarr, readarr, qbittorrent, frankbook-pro, dankserver]
 timestamp: 2026-09-04T00:00:00Z
 ---
+
+# *arr Stack
+
+The *arr automation stack runs on **FrankBook-Pro** (the Mac mini) as quadlets
+inside its podman machine; all egress rides the gluetun WireGuard netns. The
+per-service pages ([[Prowlarr]], [[Sonarr]], [[Radarr]], [[Lidarr]],
+[[Readarr]], [[qBittorrent]]) carry deployment facts; this page holds the
+cross-cutting API and auth-posture procedures.
+
+## API version split (memorize before any *arr HTTP call)
+
+- Sonarr/Radarr: `/api/v3/…`; Lidarr/Readarr/Prowlarr: `/api/v1/…`.
+- Prowlarr is port **9696** — 8090 is Jackett.
+- **Empty body = endpoint absent**, not a failure: probe endpoints by status
+  code first; v1 has no `/api/v1/profile` endpoint at all.
+- `PUT /downloadclient` and `PUT /indexer/{id}` take a SINGLE resource object
+  (GET it, patch `settings`, PUT the whole object back). An array body fails
+  with `Path: $ | BytePositionInLine: 1`.
+- API keys live in each container's config:
+  `podman exec <svc> grep -o "<ApiKey>[^<]*" /config/config.xml | cut -d">" -f2`.
+  They are local-only secrets — they stop being secret the moment a vhost
+  serves the API unauthenticated.
+
+## Schema-first probing (validators are free documentation)
+
+Send the FULL schema template object with fields filled; v1 validators reject
+partials, and each error names the next required field — peel validators one
+live call at a time instead of guessing payload shapes:
+
+- `'Fields' must not be empty` → send a full schema copy, not `fields:[]`.
+- `'Base Url' must not be empty` when keys were camelCase → **PascalCase is
+  required** (`BaseUrl`, `ApiKey`) — schema keys, not the JSON wire keys.
+- `DefaultMetadataProfileId must be > 0` → fetch real ids from
+  `GET /api/v1/metadataprofile` (rows like `(1,'Standard')`);
+  `GET /api/v1/qualitydefinition` returns the quality CATALOG — ids ≠
+  profiles.
+- `targetType null` → configContract not registered in that build (this
+  Prowlarr build has no Prowlarr-plugin indexer; integration is
+  Torznab-family only).
+- `testingPackageId` in a PUT body downgrades validation failures to
+  warnings.
+- `syncLevel` is enum `ApplicationSyncLevel` — omit it or use the int; a
+  string is rejected.
+
+**Verification rules:** `POST /downloadclient` returns the COLLECTION, not
+the created object — verify via GET. Test endpoints return verdicts in the
+BODY at HTTP 200 (`{}` = pass, a list = fail) — judge from bodies, not
+status codes. Command POSTs (`MoviesSearch`, `ApplicationSync`) run
+server-side and respond only when the work finishes: curl `-m` timeouts
+(HTTP 000) during a search are NOT failures — poll queue/history afterward
+instead of re-firing; warm small commands (`RefreshMonitoredDownloads`)
+answer in ~20 ms, so a hang means live work in progress, not a dead API.
+
+**DB as arbiter:** when endpoint shapes are opaque, stop the container and
+edit `/opt/prowlarr/config/prowlarr.db` with `sudo python3` sqlite directly,
+then start. The indexers table columns are
+Id/Name/Implementation/Settings/ConfigContract/Enable/Priority/Added — the
+enable flag column is `Enable` (not `EnableRss`). Prowlarr's real application
+settings values live in `prowlarr.db` → `Applications.Settings` JSON (the
+`/api/v1/applications` GET responses carry `fields: []`).
+
+## Auth posture probe (ground truth is behavior, not config flags)
+
+`AuthenticationMethod=None` + `AuthenticationRequired=Enabled` in
+`config.xml` is ambiguous on paper; on the wire it means a login screen with
+**zero users** — first-account-wins.
+
+1. **Confirm which host runs the service before probing** (`podman ps` on
+   the target, not the box you ssh'd into first — the arrs live on
+   FrankBook-Pro).
+2. Probe an API endpoint that bypasses the SPA shell, locally and through
+   any vhost: `curl -s -o /dev/null -w "%{http_code}"
+   http://<host>:<port>/api/v3/system/status`. **SPA roots (`/`) always
+   serve a 200 shell — never conclude "open" or "gated" from `/`.**
+   401 = gated; 200 + JSON = API fully open (anyone on the net can drive it
+   with curl).
+3. For config truth, dump the whole file
+   (`podman exec <svc> cat /config/config.xml`), not a piped
+   `grep | sed -n 2p`: **`sed -n Np` exits 0 on zero matches**, so the
+   pipeline hides a miss and you conclude "empty field" when the grep itself
+   failed.
+4. Decision framing: when posture = gated-with-zero-users, present it as a
+   decision menu (None behind the front-door wall vs External/authentikarr),
+   not a silent change — first-account-wins means whoever logs in first owns
+   the instance.
+
+## The `None` + `Enabled` auth-config trap (v1 .NET apps)
+
+`<AuthenticationMethod>None</AuthenticationMethod>` +
+`<AuthenticationRequired>Enabled</AuthenticationRequired>` is
+self-contradictory: the UI Save is permanently disabled, and any save throws
+`DryIoc.ScopedItemException` in `AuthorizationMiddleware` → 500 on every
+page after restart. The valid no-auth representation is **absence of the
+`<AuthenticationRequired>` key**, not the literal string `Disabled`. Fix =
+delete the key from `/config/config.xml`
+(`sed -i "/<AuthenticationRequired>/d"`), restart the container. Tell-tale:
+one v1 app boots clean while siblings 500-sprawl → diff their config.xml
+auth blocks; the healthy one shows the correct shape. (Prowlarr runs
+`Forms` + `Enabled` healthily — the crash combo is `None`+`Enabled` only.)
+
+General rules that saved the session these came from: container
+`State=running` ≠ app serving — read `podman logs --tail 12 <c>` before
+theorizing; when a UI gate blocks saving, go under it (these apps keep
+plaintext XML config on disk — read/edit via `podman exec`).
+
+## See Also
+
+- [[Prowlarr]] — indexer feeds and the v2 sync model
+- [[Sonarr]] · [[Radarr]] · [[Lidarr]] · [[Readarr]] — the managers
+- [[qBittorrent]] — download client, 4.6.7 pin, E2E verification
+- [[FrankBook-Pro]] — the host and its podman machine
+- [[Homepage Status Probes]] — how the e2e gate treats this sleeping host

serverAppConfigs-docs/FrankBook-Pro/qbittorrent.md

@@ -111,6 +111,90 @@ Restart=always
   probe one ssh-context out returns the residential IP — a false leak alarm.
 - Ensure umask is set to `022` so downloaded files are readable by arr apps.
 
+## E2E verification after any pipeline change
+
+Run after ANY change to the torrent pipeline — WireGuard credential rotation,
+qB version change, gluetun reconfig, category re-seeding. Proves the full
+grab path (tunnel egress → qB download → category save path → clean
+removal). All commands from a dev box via `ssh frankbook-pro`; inside the
+Mac shell export `PATH="/usr/local/bin:/opt/homebrew/bin:$PATH"` for
+`podman machine ssh`.
+
+**Access note (2026-09-21):** the Mac's login user is **dank**@192.168.0.112
+(NOT core — `core` is the podman-machine VM user, reached only via
+`podman machine ssh`). blankpad has key auth for `dank@` (bootstrapped
+2026-09-21) and a `frankbook-pro` alias in `~/.ssh/config`. macOS sshd has a
+low MaxAuthTries: when password-bootstrapping use `-o
+PreferredAuthentications=password -o PubkeyAuthentication=no
+-o IdentitiesOnly=yes` or the key offers exhaust the attempts before the
+password is tried.
+
+**1. Egress gate — MUST run inside gluetun:**
+
+```bash
+podman machine ssh 'podman exec gluetun wget -qO- https://ipinfo.io/json'
+```
+
+Require `org` = AS9009 (M247 — ProtonVPN Panama PA-12 config); judge by org,
+never country alone. NEVER run this one ssh-context out (bare `wget` on the
+Mac host): it silently returns the residential Comcast IP and reads as a
+hard-gate leak that does not exist. A residential reading ⇒ re-run inside
+the container before diagnosing.
+
+**2. Login — generic cookie parse.** The QPW fetch is flaky (~1 in N returns
+empty); assert length before use — an empty-password run once produced a
+Forbidden loop that cost hours:
+
+```bash
+QPW=$(podman machine ssh 'cat /home/core/Containers/qbittorrent/env' | grep QBIT_WEBUI_PASSWORD | cut -d= -f2)
+[ ${#QPW} -eq 20 ] || exit 1
+SID=$(curl -si --data-urlencode "username=admin" --data-urlencode "password=$QPW" \
+  http://127.0.0.1:8080/api/v2/auth/login \
+  | grep -oE '[A-Za-z_]*SID[A-Za-z0-9_]*=[^;[:space:]]*' | head -1)
+curl -b "$SID" …   # pass the WHOLE match
+```
+
+WebUI from the LAN: `http://192.168.0.112:8080` (published on the gluetun
+unit). Username `admin`; password = the machine-local `QBIT_WEBUI_PASSWORD`
+(20 chars, generated once at provisioning by the deploy handler — never in
+the repo). Parse-bug 403s never reach qB as failed logins — no ban accrues;
+server-side truth is `/Volumes/COMPLETE/qbittorrent-data/qBittorrent/config/qBittorrent/logs/qbittorrent.log`
+(`WebAPI login success/failure` + source IP). Repeated REAL failures accrue
+toward qB's 5-failures/3600 s IP ban.
+
+**3. Add a legal test torrent.** Blender open-movie torrents (~300 MB, e.g.
+Big Buck Bunny) via `POST /torrents/add` with the raw `.torrent` file
+(`-F "file=@…"`) and `category=Finished Torrents`. Never leave a real-looking
+grab behind.
+
+**4. Poll and verify.** Poll `GET /torrents/info` (~20 s): state
+`downloading` → `moving` (INCOMPLETE→COMPLETE move) → `stalledUP`/`uploading`
+at progress=1. `save_path` must be `/Volumes/COMPLETE/Finished Torrents`
+(qB 4.x auto-creates unknown categories at DefaultSavePath; 5.x did NOT
+inherit category paths — one more reason for the 4.6.7 pin). Confirm on
+disk: `ls "/Volumes/COMPLETE/Finished Torrents/"`. Throughput ~17 MB/s is
+normal for the tunnel.
+
+**5. Cleanup — delete only your artifact:**
+
+```bash
+curl -s -o /dev/null -w "%{http_code}" -b "$SID" \
+  --data-urlencode "hashes=<literal-hash>" \
+  --data-urlencode "deleteFiles=true" \
+  http://127.0.0.1:8080/api/v2/torrents/delete   # expect 200
+```
+
+Target the known hash literally — the user's real grabs share the directory
+and MUST be untouched. Verify: hash absent from `/torrents/info`, your dir
+gone, all other entries intact.
+
+**Version-coupling constraints:** qB is pinned to 4.6.7 (Readarr's EOL
+client rejects qB ≥ 5's `204` login at runtime — see the pin rationale
+above). Do not unpin while Readarr exists. After ANY qB version flip,
+regenerate the `WebUI\Password_PBKDF2` line fresh (same password, 100k
+iterations) — a 5.x-written hash line makes 4.6.7 reject every login.
+Rewrite while qB is stopped.
+
 ## See Also
 
 - [[ARR-Stack]] — full *arr stack documentation

serverAppConfigs-docs/Infrastructure/Anubis.md

@@ -131,6 +131,55 @@ ungated), and gitea (git smart-HTTP clients cannot solve JS challenges —
    distinguish challenge from deny at this release; grep the body for
    `Access Denied` / `Making sure you` instead.
 
+## Guard-chain layer diagnosis (a user reports a guarded page "broken")
+
+Use when a guarded page (e.g. the `www.franksdank.cloud` homepage) reads as
+"broken / links gone / not loading" and the fault must be localized to
+server-side or client-side/session. Chain topology:
+`Browser → Caddy TLS edge → Anubis :8923 → Caddy :8016 → Authentik outpost
+→ Caddy :8017 (@home/@data handlers) → app/static`.
+
+Layer-by-layer checks (run from the host via ssh):
+
+1. **Caddy unit + guard containers**: `systemctl --user is-active
+   caddy.service`; `podman ps --format '{{.Names}}\t{{.Status}}' | grep -iE
+   'anubis|authentik|caddy'` — expect `Up … (healthy)`.
+2. **Anubis challenge page validity**: `curl -sk
+   https://www.franksdank.cloud/ --resolve www.franksdank.cloud:443:192.168.0.143`
+   — a healthy challenge contains `"anubis_challenge"` JSON with
+   `difficulty` and the main JS.
+3. **Outpost errors**: `podman logs --tail 40 authentik-outpost 2>&1 | grep
+   -iE 'error|exception|traceback|500|provider'` — routine dramatiq task
+   logs are normal.
+4. **Post-auth handlers from INSIDE the caddy container** (8016/8017 live in
+   its netns — `curl http://127.0.0.1:8017` from the HOST returns `000`
+   connection refused, which is EXPECTED, not a fault):
+   `podman exec caddy sh -c 'wget -q -S -T5 --header "Host:
+   www.franksdank.cloud" -O- http://127.0.0.1:8017/'` and the same for
+   `/services.json`, `/status.json`. Expect 200 + body. (BusyBox wget in
+   this container has NO `-H` flag — use `--header "Host: …"`.)
+5. **Files on disk**: `ls -la /srv/homepage/` and validate `status.json`
+   parses (`python3 -c "import json;json.load(open('/srv/homepage/status.json'))"`).
+
+Tool/behavior notes: the anubis container is distroless — `podman exec anubis
+sh` fails (`sh: executable not found`); inspect via its logs or the served
+challenge page. Anubis logs `x-real-ip: 10.89.1.22` (the Caddy container's
+podman IP) for ALL clients — rootless-podman published-port SNAT;
+clients are indistinguishable by IP and clearance is cookie-based. Do NOT
+mistake that for the bug. Homepage render: `app.js` does
+`Promise.all([fetch services.json, fetch status.json])`; on ANY fetch
+failure the catch sets `#summary` to `status unavailable: <err>` and the
+grid stays EMPTY — which a user reads as "links gone / not loading." A
+30-service status.json with a 9-service services.json renders fine
+(stateByName lookup).
+
+**Decision:** if every layer is healthy (challenge valid, outpost clean,
+handlers 200, files render), the fault is client-side/session. STOP
+thrashing the live system — repeated failed browser challenges share the
+masked 10.89.1.22 identity. Get the authenticated view: ask for the exact
+symptom (challenge spinning? login loop? empty grid? "status unavailable"
+text?) or the authentik login to drive the page in a real browser.
+
 ## Verify live
 
 ```

serverAppConfigs-docs/Infrastructure/Authentik.md

@@ -342,6 +342,178 @@ The silent re-auth path was proven end-to-end: after deleting
 `qa-e2e-admin`'s rows, the e2e admin spec ran against the stale storage
 state and passed with no login prompt (2 tests, 2.9 s).
 
+## Guard-chain flip recipe (adding a guarded vhost)
+
+Move a public vhost from `reverse_proxy <app>:<port>` onto the chain:
+client → TLS edge → Anubis :8923 → outpost decision → internal
+`http://:8017` listener → Host-matched handler → real upstream.
+
+**Order (never reorder):**
+
+1. **Provider first.** Create the outpost ProxyProvider before touching
+   routes, or authed traffic hits the outpost error page instead of the app.
+   Fleet convention for `external_host` is the exact browser-facing origin
+   (the inventory table carries both bare origins and trailing slashes;
+   routing matches on hostname, verified 2026-09-02).
+   `internal_host = http://caddy:8017`, all three flows set, and an
+   Application attached — the two provisioning rules above.
+2. **Route flip in ONE commit/deploy**: the vhost becomes
+   `reverse_proxy anubis:8923` with `header_up X-Real-Ip {remote_host}`, and
+   inside the `http://:8017` listener add `@<svc> host <vhost>` +
+   `handle @<svc> { reverse_proxy <upstream> }`. Anubis guard set, Caddy TLS
+   edge, `:8017` routing table, and provider list change together.
+3. Deploy via the repo deploy flow (commit first; deploy pushes main itself).
+
+**Verification (all three):**
+
+- Unauthenticated probe from a client: `curl -s https://<vhost>/` → HTTP 200
+  challenge page; grep the body for `challenge|not a bot|anubis` (~8
+  markers on a live challenge). A 502 here instead of the challenge = chain
+  mis-wired.
+- Post-auth routing proof: grep the LIVE config inside the container —
+  `podman exec caddy sh -c "grep -n -A3 <handler> /etc/caddy/Caddyfile"` —
+  after the deploy prints its reload line; confirm the reload actually ran,
+  don't trust exit 0.
+- Outpost reachability: `podman exec authentik-server getent hosts caddy`
+  resolves the compose DNS alias (10.89.1.x) that `internal_host` dials.
+
+**Pitfalls:** probing `http://localhost:8017/` from the host returns HTTP
+000 (connection refused) — the post-auth listener lives in the caddy
+container's netns; probe inside it or grep the config. Non-browser clients
+cannot solve the JS challenge — before flipping, audit service-to-service
+traffic: ingestion/push paths must target internal addresses (`loki:3100`),
+never a public hostname of a guarded service. Exempt-by-construction
+categories stay unguarded: the IdP itself (`auth.`), git smart-HTTP
+endpoints, services with their own complete auth — document the exemption
+in the vhost comment.
+
+## Driving the ORM (`ak shell`)
+
+All authentik object work happens through the server container's ORM shell —
+no repo change, no deploy (these providers are NOT repo-managed; capture new
+provider+application pairs as a blueprint or setup script to avoid drift).
+
+```bash
+ssh dankclaw 'podman exec -i authentik-server ak shell' <<'EOF' | sed -n '/229 objects/,$p'
+from authentik.core.models import User, Group
+...
+EOF
+```
+
+- The heredoc IS the program: `ak shell` reads all of stdin and execs it
+  (heredoc-through-ssh quoting breaks; the raw-stdin pattern is the proven
+  one).
+- Bootstrap prints JSON log lines, then
+  `### authentik shell (2026.8.3)` and `229 objects imported automatically…`
+  — pipe through `sed -n '/229 objects/,$p'` to drop the noise.
+- 229 models are pre-imported. Import paths (each wrong guess costs a failed
+  exec): `User`, `Group`, `Application`, `ApplicationEntitlement`,
+  `PropertyMapping` → `authentik.core.models`; `ProxyProvider` →
+  `authentik.providers.proxy.models` (NOT `authentik.outposts.models`);
+  `LDAPProvider` → `authentik.providers.ldap.models`;
+  `authentik.core.models.application` does NOT exist as an import path; there
+  is no `authentik.policies.group` module in this build.
+- For live DB truth prefer the postgres container
+  (`podman exec authentik-postgres psql -U authentik -d authentik -At -c
+  "select * from …"`) — **query with `select *` and stop guessing column
+  names**: 2026.x renamed columns (`Provider.objects.filter(type=…)` raises
+  FieldError — `type` moved to subclass tables; the outpost `enabled` column
+  may not exist under that name; provider pk is `oauth2provider_ptr_id`).
+
+## User provisioning playbook (user-level account)
+
+There is no `User.create_user` classmethod, and `is_superuser` is NOT a
+constructor kwarg (it is a property derived from group membership) — passing
+it raises `TypeError: User() got unexpected keyword arguments`:
+
+```python
+from authentik.core.models import User, Group
+u = User(username="ridley", email="ridley@franksdank.cloud", name="Ridley", is_active=True)
+u.set_password("...")        # password arrives on stdin only, never argv
+u.save()
+for g in ("Non-Admin Users", "jellyfin_users", "openwebui_users"):
+    u.groups.add(Group.objects.get(name=g))
+```
+
+`path` defaults to `users` (regular). The access map (verified live
+2026-10-04): there are NO per-app policy bindings on the proxy providers
+(all use the implicit-consent "Authorize Application" flow), so guarded-vhost
+access is decided entirely by GROUP MEMBERSHIP consumed downstream —
+
+- `homepage-admins` → Caddy RBAC for homepage `/admin` (`X-Authentik-Groups`
+  regex gate) and the homepage admin panel; regular users get only
+  `category: user` panels from `services.json`.
+- `openwebui_users` / `openwebui_admins` → ApplicationEntitlements
+  "Open WebUI Users"/"Open WebUI Admins" (bound by group), consumed by the
+  "Open WebUI roles" property mapping → `roles` claim. Verify live with
+  `u.app_entitlements(app)`.
+- `jellyfin_users` / `jellyfin_admins` → Jellyfin via the LDAP outpost
+  (filter `(memberOf=cn=jellyfin_users,ou=groups,dc=franksdank,dc=cloud)`).
+- `Non-Admin Users` = the plain-user marker group (precedents: qa-e2e, and
+  every human account).
+- **The human-user template** (ridley, kassie, alex, artem, kirby, ry, nic,
+  debo, freya — kirby/ry/nic/debo/freya (pk 17–21) created 2026-10-09, kassie
+  and alex created 2026-10-08, artem (pk 16) created 2026-10-07): `username` +
+  `<user>@franksdank.cloud` + display name +
+  `is_active=True` + the password set via `set_password`, membership in
+  `Non-Admin Users` + `jellyfin_users` + `openwebui_users`. `alex` was
+  recreated from scratch on 2026-10-08 into this shape (the old row had an
+  empty email and no `openwebui_users`; audit first — the row carried no MFA
+  device, token, entitlement, or proxy session, and events survive a user
+  delete because `Event.user` is a denormalized JSON field, not a FK).
+- EXCLUDE from a user-level account: `homepage-admins`, `jellyfin_admins`,
+  `openwebui_admins`, `authentik Admins`, `authentik Read-only`.
+- Other user-category vhosts (SearXNG, Gitea, Calibre-Web, RomM, Copyparty,
+  Plex, Seerr, Marreta) need no per-user group: any authenticated user
+  passes the outpost.
+
+Verify login behaviorally — `authenticate()` is the exact PasswordStage code
+path:
+
+```python
+from django.contrib.auth import authenticate
+assert authenticate(request=None, username="ridley", password="...")   # -> User
+assert authenticate(request=None, username="ridley", password="nope") is None
+```
+
+Look flows up by slug, not name:
+`Flow.objects.get(slug="default-authentication-flow")` (its `.name` is
+"Welcome to authentik!" — `.get(name=…)` fails). Stage order:
+Identification(10) → Password(20) → AuthenticatorValidate(30; no-op with no
+enrolled device) → UserLogin(100). With no MFA enrolled, nothing else blocks.
+
+More field gotchas (each cost a failed exec): `ApplicationEntitlement.application`
+can be null (guard with `getattr`); `LDAPProvider` has
+`search_mode`/`bind_mode`/`base_dn`, lacks `search_base`/`of_groups`/
+`organization`; `OAuth2Provider` has no `.scopes`; `Outpost.config` is an
+`OutpostConfig` object, not a dict; `Policy` has no uniform `.policy_cls`
+(use `type(p).__name__`); `User` has no `.status`.
+
+Never rotate `AUTHENTIK_SECRET_KEY` to fix claim issues — it invalidates
+every outpost session fleet-wide; purge the stale `proxysession` rows
+instead (section above).
+
+## Probing the outpost from inside the netns
+
+Rootless podman gives each container its own netns; **host `ss -ltn` cannot
+see listeners inside other containers' namespaces** — a missing listener on
+the host is an observation artifact, not a fault. Distroless images have no
+shell (`podman exec authentik-server sh` fails), so probe by joining the
+namespace:
+
+```
+podman run --rm --network container:<container> --entrypoint /bin/busybox \
+  docker.io/library/busybox:latest \
+  wget -q -T 4 -t 1 -O /dev/null -S "<url>" 2>&1 | head -8
+```
+
+- Quote URLs over ssh: the remote shell is zsh and bare `?` in query strings
+  triggers globbing (`no matches found`).
+- Boot transience: immediately after a restart the same probe can return 503
+  (configure task pending); poll until the answer changes before concluding.
+- `podman inspect --format "{{.Config.NetMode}}"` errors on this podman
+  version — the field is `.HostConfig.NetworkMode`.
+
 ## Failure modes
 
 | Symptom | Cause | Fix |

serverAppConfigs-docs/Infrastructure/Homepage Status Probes.md

@@ -0,0 +1,158 @@
+---
+type: reference
+title: Homepage Status Probes
+description: The homepage TUI status page — how probe targets are chosen (host-netns vantage), the three false-down shapes, triage procedure, local render check, deploy, and the e2e gate interactions (asleep Mac mini, raw-IP matrix).
+tags: [homepage, status, probes, e2e, homepage-status]
+timestamp: 2026-10-05T00:00:00-07:00
+---
+
+# Homepage Status Probes
+
+The homepage is a static site —
+`shared/homepage/{index.html,admin.html,app.js,style.css,services.json,docs.html}`
+— served from `/srv/homepage` on **dankclaw** only (`deploy_homepage` in
+`scripts/deploy-remote.sh` refuses other targets). A checker
+(`scripts/homepage-status.py`, run by the `homepage-status.timer` unit) probes
+each card's target and writes `/srv/homepage/status.json` (mode **0644** so
+Caddy can serve it — losing the mode 403s the page).
+
+## The checker's vantage and its three false-down shapes
+
+The checker probes TCP/HTTP **from the host netns**. It only has three states
+— `wip` | `up` | `down`; there is no "unknown", so an unprobeable service can
+never render honestly and must be dropped or re-pointed, not "fixed".
+
+1. **TLS-only app probed with `method:"http"`.** Plex Media Server serves TLS
+   only on `32400`; plain-HTTP GET → TLS alert → curl exit 52, code `000` →
+   card says down while PMS answers `200` over HTTPS. Fix:
+   `"method":"tcp"` (connect is the honest liveness signal; the public URL
+   stays HTTP because Caddy re-keys).
+2. **Retired port.** The Hermes card probed `127.0.0.1:8090` after the
+   compose→quadlet move moved the dashboard to `9119` and the webui to
+   `8787`; nothing listened on 8090, so a healthy stack read `down`.
+3. **Unpublished port / wrong netns.** `gitea-postgres` (`db:5432`, net
+   `systemd-gitea`), `searxng-valkey` (`valkey:6379`, net `systemd-searxng`)
+   and `alloy` (OTLP 12345) publish **no** ports. Container DNS aliases are
+   netns-scoped, so from the host these are unreachable *by construction* —
+   no probe value makes them honest. Drop those cards.
+
+A host-vantage probe can only reach a listener **published to the host**
+(rootlessport/pasta); unpublished ports are invisible from the host netns.
+See [[Container Host Reachability]] for the netns rules.
+
+## Triage procedure
+
+1. Read the card's probe target in `shared/homepage/services.json`; note
+   method (`tcp` vs `http`).
+2. Does that port have a **host-visible listener**?
+   `podman ps -a --format '{{.Names}}|{{.Status}}'`, then per container
+   `podman inspect <c> --format '{{json .NetworkSettings.Networks}}
+   {{.HostConfig.NetworkMode}}'` and `podman port <c>` (empty output =
+   published nothing → structurally unprobeable from the host).
+3. Probe from the **correct vantage**: from a bridge-net container the host
+   is its gateway IP (`/proc/net/route`, e.g. `10.89.1.1` on
+   `dankclaw_default`) — *not* `host.containers.internal` (`169.254.1.2`),
+   which belongs to the default `podman` net (`10.88.0.0/16`). Probe with
+   bash `/dev/tcp` (no `python3`/`ip` in these images); escape loop vars so
+   they expand **inside** `podman exec bash -c`.
+4. Classify: real outage (container exited — read `State.ExitCode`: 137 =
+   SIGKILL, 143 = SIGTERM/manual stop; `OOMKilled=false` rules out the OOM
+   killer) vs false down (running + unreachable target).
+5. Fix: re-point or re-method for HTTP/TLS mismatches; **drop** cards whose
+   only exposure is an internal netns; never invent a probe target.
+6. Verify by re-running the checker path, not by eyeballing: after deploy,
+   re-read `/srv/homepage/status.json` and diff the `down` list
+   before/after.
+
+Removing a card's `probe` key without a code change crashes the checker
+(`svc["probe"]` KeyError) — dropping the whole entry is the only zero-code
+fix. The `description` field is consumed only by `app.js` `descHTML()`, so
+adding a `details` object (role/spec/inputs/use) is schema-safe:
+`tests/homepage_config.bats` checks only name/category/section/probe.
+
+## Local render check (before every deploy)
+
+The page fetches `services.json` + `status.json` at load; `file://` is
+CORS-blocked, so serve over HTTP:
+
+```bash
+cd shared/homepage
+python3 -c "import json; s=json.load(open('services.json')); \
+  json.dump({'services':[{'name':x['name'],'state':'up'} for x in s]}, open('status.json','w'), indent=2)"
+python3 -m http.server 8899 --bind 127.0.0.1
+```
+
+Then in the browser: load `http://127.0.0.1:8899/index.html`, **wait ~1.5 s**
+(panels render after the async fetch; checking immediately returns 0 panels),
+extract via `tab.evaluate` — LLM cards render `.desc-role`/`.desc-spec`/
+`.desc-inputs`/`.desc-use` from the card's `details` object, other cards a
+plain `.desc`; `getComputedStyle(el).color` confirms tones. Screenshot for
+visual proof. Expected local noise: `deployed.js` 404s (baked in at deploy
+time) — harmless.
+
+The scratch `status.json` is untracked and NOT git-ignored — move it to
+`shared/homepage/.AGENTTRASH/` after the check (repo policy: never `rm`).
+
+CSS palette lives in `style.css` `:root` (`--fg` bright green, `--dim` dim,
+`--head` pale, `--link` cyan, `--wip` yellow, `--down` red). Add new tones as
+`--*` vars and include any new `.desc-*` class in the `html[data-bare="1"]`
+reset so `?bare=1` strips it to black.
+
+## Deploy
+
+```
+./scripts/deploy.sh dankclaw --service homepage --skip-e2e
+```
+
+Copies the assets to `/srv/homepage`, regenerates `deployed.js` (git commit
+datetime), restarts the `homepage-status` timer. Verify live:
+`ssh dank@192.168.0.143 'grep … /srv/homepage/…'`.
+
+**`--skip-e2e` is justified for homepage-only changes when the Mac mini is
+asleep**: the e2e admin crawl (`tests/e2e/specs/11-admin.spec.ts:24`) walks
+Prowlarr at `192.168.0.112:8989` on FrankBook-Pro; an asleep mini (ARP STALE,
+no remote wake) times out that test — a pre-existing environmental failure,
+not a regression. All non-Mac-mini e2e tests pass.
+
+## e2e raw-IP *arr failures — probe from the browser's box, not a peer
+
+When `11-admin` fails with `ERR_CONNECTION_TIMED_OUT`, the instinct is "the
+Mac is asleep". Sometimes true; not always. Playwright runs on the dev box,
+so a healthy probe from dankclaw proves nothing about the **browser's** path:
+
+```bash
+# 1. From the box that RUNS the browser:
+curl -s -o /dev/null -w '%{http_code}\n' --max-time 7 http://192.168.0.112:8989/
+# 2. From a LAN peer:
+ssh dankclaw 'for p in 8989 7878 9696; do curl -s -o /dev/null -w "$p %{http_code}\n" \
+  --max-time 6 http://192.168.0.112:$p/; done'
+# 3. Liveness (ICMP, not TCP):
+ping -c 2 -W 3 192.168.0.112
+```
+
+| peer probe | dev-box probe | ping | meaning |
+|---|---|---|---|
+| 200 | 200 | ok | transient — re-run the spec |
+| timeout | timeout | fails | Mac asleep; wake, ports return in seconds |
+| **200** | **rc=28** | **ok** | **path problem, not sleep** — ICMP allowed, TCP blocked. Waking cannot help |
+
+Third row: confirm the Mac is innocent through the dankclaw jump
+(`pgrep -fl socat`; `netstat -an | grep -c 8989`; loopback
+`curl http://127.0.0.1:8989/` → 200; `/usr/libexec/ApplicationFirewall/socketfilterfw
+--getglobalstate` → disabled). Forwarders up + loopback 200 ⇒ the block is
+between the dev box and the Mac.
+
+Don't hold a deploy hostage to a known-environmental red gate: verify the
+deployed service **directly** (its own ledger/health/real completion), state
+the gate was skipped or partially red and why (`--skip-e2e` prints the
+warning, so the skip is recorded), and report the residual gap plainly
+("29/30 e2e pass; 11-admin fails on Sonarr's raw IP from the dev box; peer
+probe returns 200; deploy verified by other means"). Do not weaken or skip
+the spec — it guards a real SSO-plus-RBAC contract.
+
+## Related
+
+- [[Container Host Reachability]] — the netns rules behind the vantage
+- [[Observability Provisioning]] — the monitoring stack reading the same fleet
+- [[Deploy State Triage]] — service-scoped deploys and the e2e gate
+- [[Testing Policy]] — the seam/e2e contract this page's checks live in

serverAppConfigs-docs/Infrastructure/Homepage.md

@@ -23,7 +23,7 @@ The homepage is live on Dankclaw. `./scripts/deploy.sh dankclaw` copies the stat
 page (`shared/homepage/docs.html`) — access walkthrough, the running-model
 guide, media how-tos, and troubleshooting. Per-service `/docs/<svc>/` pages
 remain a separate thread. The three LLM cards (`frank-large`, `frank-medium`,
-`frank-small`) moved from admin-only to the **user** page with live model
+`frank-secondary`) moved from admin-only to the **user** page with live model
 descriptions and a `docs` link to `/docs#models`.
 
 **Sections + WIP split (2026-09-21):** both pages now group their panels
@@ -36,6 +36,211 @@ deliberately unprobeable) via the `wip` value (`"open"` / `"closed"`).
 Tautulli (Plex statistics, `192.168.0.235:8181`) joined the admin page's
 Media Library Apps section.
 
+**Amber CRT theme + API docs (2026-10-07):** the palette moved from green
+phosphor to amber CRT-AMBIOS (`--fg #ffb000` on `--bg #0a0600`), and the font
+stack now leads with **VT323** loaded from the Google Fonts CDN (the deploy
+also ships `shared/homepage/img/` beside the page, so `/img/*` serves the docs
+screenshots — the post-auth listener serves files only through the `@data`
+path allowlist, so new top-level assets must be added there (the `@home`
+catch-all answers anything else with `index.html`, i.e. a PNG arrives as
+`200 text/html` and renders broken; pinned in `homepage_caddy.bats` — this
+exact bug bit the screenshots on 2026-10-08). The docs page gained a full
+(`/docs#api-access`): per-user OpenWebUI API keys (User menu → Settings →
+Account → API keys), the base URL `https://ai.franksdank.cloud/api/v1`, and
+copy-paste configs for **Opencode** (`~/.config/opencode/opencode.json`) and
+**Pi** (`~/.pi/agent/models.json`) pointing at `frank-secondary`. Shipped
+together with the OpenWebUI side: `features.api_keys=true` for regular users
+and the restored `frank-secondary` model row + public `read` access grant
+without which the picker showed "No results found" for every family account
+(see [[OpenWebUI]]).
+
+**Capability equivalence notes (2026-10-08):** each fleet card (and the
+`/docs#models` section) carries a `details.equiv` line — frank-large
+"equivalent to Claude Sonnet 5 / OpenAI-Luna (Summer 2026 frontier
+intelligence)", frank-primary "equivalent to Claude Opus 4.8 / OpenAI-Terra
+(same era)", frank-secondary "equivalent to last year's frontier level
+(2025)". Rendered by `descHTML()` as `.desc-equiv` (head color, italic);
+pinned in `homepage_config.bats`.
+
+**Protoss rename + Model Information component (2026-10-08):** the model
+cards became the StarCraft trio — `frank-large` → **frank-dragoon**
+(iGPU), `frank-primary` → **frank-archon** (Dankserver), `frank-secondary`
+→ **frank-zealot** (3090). The endpoint/API ids did NOT change in this wave; only the
+family-facing names. The three cards fused into one **Model Information**
+component (`app.js` `renderModelInfo()`, `.model-info`/`.mi-*` styles):
+Protoss unit art (`/img/unit-<unit>.jpg`, hex-clipped), per-unit accent
+colours (Dragoon gold, Archon cyan, Zealot blue), smaller type (15px vs
+19px body), and clickable unit-voice lines that play the original
+StarCraft `.ogg` quotations from `/sfx/` (fetched from
+starcraft.fandom.com via its MediaWiki API; the quote→file mapping lives
+in the `model.sfx`/`model.voice.sfx` fields). New `services.json`
+per-model fields: `unit`, `tagline`, `sfx[]`, `voice`, `supervision`
+(dragoon/archon autonomous-safe, zealot tight-supervision), and `warn`
+(uncensored legal note, rendered as a hover tooltip, not wall text).
+A compaction-model recommendation for frank-zealot rides in its `use`
+text. The Caddy `@data` allowlist grew `/sfx/*`; `deploy_homepage` copies
+`sfx/`. A `wip: "down"` badge state ("WIP: DOWN", red) was added; Seerr,
+Calibre-Web, RomM, and Jellyfin are WIP: DOWN with their Open links
+removed. Card notes added: Copyparty holds every other media app's
+content (including the WIP ones); Gitea and Plex need logins separate
+from the dankcloud SSO; SearXNG is obfuscated, not anonymous (upstream
+API keys tie to Frank). The user docs `/docs#models` section carries the
+same renames, api-id mapping, supervision and compaction notes.
+
+**Icons, card polish, admin identity gate (2026-10-08, later wave):** every
+app card now carries hex-clipped art beside its name (`services.json` `art`
+field, files under `img/apps/`): 19 brand marks tinted amber from the
+Simple Icons CDN plus 19 hand-improvised monogram SVGs (amber hex + VT323
+initials) for apps with no mark — Copyparty, Marreta, Hermes (+dashboard),
+node-exporter, Gluetun, OpenWebUI, Calibre-Web, RomM, Seerr, Loki, Alloy,
+Alertmanager, Valkey, Prowlarr, Readarr, Lidarr, Tautulli, ComfyUI. Model
+cards render in the order zealot → dragoon → archon (registry order is the
+contract; `homepage_config.bats` pins it). Dragoon and zealot share one
+common uncensored legal warning (adult content, harmful/hateful/illegal
+descriptions, "sensitive offensive cybersecurity topics", severe legal
+risk, "don't ruin this server for everybody"), rendered as the hover
+label; archon's ask-Frank rule is a bordered red `.mi-perm` callout.
+SearXNG's obfuscation-not-anonymity note moved out of its description into
+a card-level `warn` (`.svc-warn` hover label) and the same hover treatment
+in the docs. Gitea's note now documents self-service signup — no need to
+ask Frank. Footer: docs / admin / account settings are bordered glowing
+`.foot-btn` buttons; **the admin button is identity-gated** — Caddy's
+`/whoami` echoes the outpost-injected `X-Authentik-Username`/`X-Authentik-Groups`
+headers (the outpost strips client-supplied copies, so the echo is
+spoof-proof), and `app.js` shows the button only to `homepage-admins`
+members. The `/admin*` 403 stays the real gate; the e2e specs pin both
+sides (standard user: hidden + 403; admin: visible). Docs were swept for
+naming consistency: Protoss names primary everywhere, bracketed api ids
+at the model headings, and a note that a future switchover renames the
+ids themselves (OpenWebUI picker + raw API + models.yml all move together
+— not done yet).
+
+**Icons v2, clickable tiles, header nav (2026-10-08, third wave):** five
+icons replaced on request — Copyparty gets an improvised amber folder,
+Marreta a crumbling wall (bricks + red crack), Calibre-Web a book,
+OpenWebUI its official logo (`openwebui.com/logo.png`), Seerr the official
+`seerr.dev/os_logo_filled.svg`. Tiles with an open link are now clickable
+whole (`.panel.clickable`, hover lift + glow; clicks on inner links pass
+through). Any panel whose live state is DOWN gets a red gradient wash +
+slow pulsing glow (`.panel.down`, and `.mi-card:has(.badge.down)` for
+model cards). docs/admin/account-settings moved from the footer to the
+header (`nav.top-links`). The zealot compaction advice moved out of the
+Use paragraph into a bordered glowing `model.compaction` callout
+(`.mi-comp`), pinned by the component seam test.
+
+**Protoss theme, brand-color icons, portraits (2026-10-08, fourth wave):**
+the full-page angular treatment: chamfered 14 px-cut panels and model
+cards, hex-lattice + amber-glow backdrop (`body::after`), scanline
+overlay (`body::before`), diamond-glyph section titles with gradient
+underlines, the gradient MODEL INFORMATION gateway, a blinking cursor
+beside the title, amber scrollbars, cyan text selection, cyan hover glow
+on clickable tiles, and a red pulsing glow on down tiles/cards (the
+`data-state="down"` attribute `applyStatus` now sets). Icons recut to
+official brand colors (Simple Icons CDN): SearXNG, Gitea, Plex, Jellyfin,
+Sonarr, Radarr, Lidarr, Readarr, qBittorrent, Prowlarr, Seerr, OpenWebUI,
+Calibre-Web (amber glyph on amber tile). Handcrafted monograms carry
+per-app colors: Copyparty gold, Marreta brick, Grafana orange, Loki
+indigo, Neo4j network, Alloy violet, Promtail pink, Alertmanager red,
+Prometheus orange, Node Exporter teal, cAdvisor sky, Guake yellow,
+Authentik blue, Seer green radar, RomM a purple GameBoy. New per-service
+fields: `art_bg` (art backdrop — OpenWebUI's dark logo needs `#f5f5f5`),
+`warn` on OpenWebUI (searches ride the Seerr→SearXNG path),
+`model.compaction` (the amber "set this as compaction model" callout
+frank-zealot carries). frank-dragoon's card portrait is the SC2 unit
+portrait (fandom wiki `Dragoon_portrait.jpg`, webp→jpeg). The whole
+effect layer is stripped under `data-bare="1"` (the e2e bare page).
+Unit: `tests/homepage_config.bats` (referenced art files exist, model
+blocks well-formed, deployer ships the `img/` tree).
+
+**Docs reorder + model source links (2026-10-08, fifth wave):** the docs
+page section order is now getting-in → troubleshooting → media → chat →
+models → api-access (non-AI first, AI last). The *arr stack (Prowlarr,
+Sonarr, Radarr, Lidarr, Readarr, qBittorrent) is out of the family docs
+entirely — admin tooling; Seerr is presented as the front door for
+requests. Each fleet model gained a Hugging Face source link: card field
+`model.hf` rendered as the cyan `model on huggingface ↗` line under the
+role, and the docs `Model:` bullets link the checkpoint names (zealot →
+HauhauCS/Qwen3.5-9B-Uncensored-HauhauCS-Aggressive, dragoon →
+Quat3rnion/halogen-swift1.5-qwen3.8-flash-next-v2-abliterated, archon →
+Qwen/Qwen3.8-Flash-Next-FP8).
+
+**Navy Protoss console, maximal cornering, font swap (2026-10-08, sixth
+wave):** the amber CRT palette is retired. The page is now deep space
+navy (`#050d16`) with warp-blue/psychic-cyan accents (`--proto-blue
+#3fa9ff`, `--proto-cyan #7fe7ff`) — StarCraft Protoss console colours.
+Cornering went maximal: every panel, model card, badge, button, callout
+and the MODEL INFORMATION gateway are octagon-cut (all four corners
+chamfered, 18–24 px), and panels/model cards gained a second inset
+octagonal frame line (`::before`) — a double-bevel gateway look. Section
+titles are chamfered plates with the diamond glyph inside the plate.
+Fonts: body is **Share Tech Mono** (legible retro terminal), headings
+are **Rajdhani 600** (the StarCraft UI face) — VT323 is retired; all
+three pages load the pair from Google Fonts. Four tiles that had
+reverted to initials got pictograms: Calibre-Web an open book (orange),
+RomM a purple GameBoy, Copyparty a blue folder with upload arrow,
+Marreta a crumbling brick wall with a red crack (`img/apps/app-*.svg`).
+frank-dragoon's card accent is phase-crystal teal (`#5fe0d0`), matching
+the new palette. Bare mode strips the frames, octagon cuts and fonts.
+
+**Simplified console (2026-10-08, seventh wave):** "too busy — clean up
+and simplify/solidify." The sixth-wave effect stack was stripped to
+essentials. Removed: the scanline overlay (`body::before`), the hex
+lattice + twin radial backdrop (one quiet pylon glow above the horizon
+remains), the panel/model-card inner frame lines, all ambient text
+glows (body, title, model names, callouts), the blinking title cursor,
+the gradient section-title plates (headings are plain Rajdhani with a
+hollow diamond glyph + steel underline), the gradient `border-image`
+header rule (solid 2px warp-blue), the pulsing down-state animation
+(down tiles and cards now show a STATIC red border + tint), and the
+button/hover glows. Kept: octagon-cut panels (softer 14 px), unit-color
+model cards, the pictograms, and the fonts. Motion is reserved for
+state — the only animation left is the speaking-voice pulse. The result
+is flat navy panels on crisp 1px borders: same Protoss direction, half
+the pixels.
+
+**Model ids official (2026-10-08):** the card ids `frank-zealot`,
+`frank-dragoon`, `frank-archon` are now the harness's official model ids
+in `models.yml`/`config.yml` (see [[OMP Harness]]); the
+systemd services keep the `frank-primary`/`frank-large`/
+`frank-secondary` names. Homepage, docs page and harness now agree on
+one name per model.
+
+**Api ids renamed on the wire (2026-10-09):** the promise in
+`docs.html` is kept — the served model ids are now `frank-zealot`,
+`frank-dragoon`, `frank-archon`, so the docs page no longer carries
+`api id:` annotations, and the curl/opencode/Pi examples send
+`frank-zealot`. The systemd services keep the old names. `docs.html`
+names the fleet trio once each; `homepage_config.bats` pins the card
+ids, which did not change.
+
+**Console polish (2026-10-08, eighth wave):** five fixes — octagon
+clip-paths replaced with `border-radius` (the clipped corners read as
+gaps in the 1px border); section-title diamonds moved from absolute
+positioning to a fixed-width inline glyph (aligned, wrap-safe); closed
+WIP items grey out (dim type, desaturated art, muted border via
+`:has(.badge.wip-closed)`); the Dragoon tagline is one spliced track
+(`sfx/dragoon-tagline.ogg` — loudnorm-matched, silence-trimmed, 350 ms
+beat; the two SC2 source oggs are trashed); privacy notes became
+**`<details>` disclosure chips** ("⚠ privacy note ▸") pinned to the
+bottom of each box (`margin-top:auto` in a flex-column panel/card) —
+collapsed by default, tap-to-expand, the touch-safe replacement for the
+unreachable hover tooltips. The panel click-to-open handler ignores
+`.svc-warn` taps so the chip never navigates. Same chips on the docs
+page (privacy + uncensored-model warnings). The API-access section was
+reblessed against the live Open WebUI 0.6 UI: steps now match the real
+Show/copy/rotate flow and `img/openwebui-api-keys.png` is a fresh
+screenshot (old shot trashed). The troubleshooting DNS bullet dropped
+its Pi-hole specifics — family members off the LAN cannot use
+house-DNS advice; the bullet now just says the names resolve on the
+home network.
+
+**Archon card runs the fleet's own model (2026-10-08):** the
+`frank-archon` homepage card and docs Model bullet now say the payload
+is our in-house GPTQ-w4 quant of Swift1.5-Qwen3.8-Flash-Next and link
+our HF publication (`frank8465/Swift-1.5-Qwen3.8-Flash-Next-GPTQ-W4A8-Radiance-AMD-R9700`)
+in place of the FP8-era base-model link; capacity line says three
+submissions (`--max-num-seqs 3`).
+
 ## Design decision — 2026-08-22
 
 The live up/down check runs server-side. A checker on Dankclaw probes each backend and writes a same-origin `status.json`. The page fetches only that file and renders the badges. The browser never calls a backend directly. The checker runs as a systemd timer every 3 minutes.
@@ -111,7 +316,7 @@ flowchart TD
 ## Service classification
 
 **User services** (root page), in section order:
-- **AI** — OpenWebUI, and the three LLM fleet cards (frank-large, frank-medium, frank-small — status + docs link, no Open link).
+- **AI** — OpenWebUI, and the three LLM fleet cards (frank-large, frank-primary, frank-secondary — status + docs link, no Open link).
 - **Web Apps** — SearXNG, Gitea, Copyparty, Marreta.
 - **Media Library Apps** — Calibre-Web, RomM, Plex, Jellyfin, Seerr.
 
@@ -131,7 +336,10 @@ shared/homepage/
 ├── style.css         # full theme; a toggle strips to bare HTML
 ├── app.js            # loads status.json once, renders the panels
 ├── services.json     # service config (user + admin)
-└── fonts/            # Terminus (woff2) for @font-face
+├── img/              # served at /img: docs screenshots, unit portraits
+│   ├── apps/         #   (unit-*.jpg) and app card icons (apps/*.svg)
+├── sfx/              # StarCraft unit voice .ogg files served at /sfx
+└── fonts/            # optional self-hosted fonts (deployed with the page; VT323 comes from the Google Fonts CDN, this dir is absent)
 ```
 
 `status.json` is generated at runtime in `/srv/homepage/`. It is not committed.

serverAppConfigs-docs/Infrastructure/LAN-DNS.md

@@ -158,6 +158,119 @@ The Porkbun zone `franksdank.cloud` carries an A record for every vhost plus the
 | HTTPS works from the host but not the WAN | No port forward on the G36 for that service (manual router step) |
 | Cert renewal fails in Caddy logs | Porkbun API credentials in the Dankclaw `.env`; test with the Porkbun API `/ping` endpoint |
 
+## Per-client fix: resolver drop-in + browser DoH bypass
+
+Symptom: a LAN machine gets `connection timeout` (curl rc=28) on
+`https://*.franksdank.cloud` while the server is healthy. Root cause: the
+public DNS hands out the WAN IP `71.227.227.172`, which has **no 443
+port-forward** (filtered). Any resolver that is NOT Pi-hole produces the WAN
+answer → SYN to a filtered port → timeout. There are TWO resolver layers to
+check, in order:
+
+**Layer 1 — systemd-resolved (fixes curl/shell).** A route-only drop-in
+sends `*.franksdank.cloud` ONLY to Pi-hole and never to the default-route
+servers:
+
+```bash
+dig +short www.franksdank.cloud @192.168.0.143   # expect 192.168.0.143
+dig +short www.franksdank.cloud @1.1.1.1         # expect 71.227.227.172 (the trap)
+getent hosts www.franksdank.cloud                # what the OS actually resolves
+curl -sk --resolve www.franksdank.cloud:443:192.168.0.143 \
+     https://www.franksdank.cloud/ -o /dev/null -w '%{http_code}\n'   # 200 = server fine
+
+sudo mkdir -p /etc/systemd/resolved.conf.d
+sudo tee /etc/systemd/resolved.conf.d/franksdank-lan.conf >/dev/null <<'EOF'
+[Resolve]
+DNS=192.168.0.143
+Domains=~franksdank.cloud ~franksdank.casa
+EOF
+sudo systemctl restart systemd-resolved && resolvectl flush-caches
+```
+
+The `~` (route-only) prefix is load-bearing: it routes only those domains to
+Pi-hole. Verify: `getent hosts www.franksdank.cloud` → `192.168.0.143`;
+`resolvectl status` shows `DNS Domain: ~franksdank.cloud ~franksdank.casa`.
+
+**Layer 2 — browser DoH (the bypass: "curl works, browser times out").** If
+Layer 1 is verified but the BROWSER still times out, the browser is doing
+its own DNS over HTTPS and never asks systemd-resolved — it queries the TRR
+provider (Cloudflare et al.) directly → WAN answer → timeout. Prove it:
+`curl -sk -m 6 https://71.227.227.172/` → rc=28 (the WAN answer is a dead
+end). Diagnosis details:
+
+- Firefox profile may be at the XDG path `~/.config/mozilla/firefox/`, NOT
+  `~/.mozilla/firefox/` — find it via the running process if the default
+  path is empty.
+- `grep network.trr. <profile>/prefs.js` — **no entries = shipped default,
+  and since Firefox 113 the shipped default is DoH ON.**
+
+Fix (surgical, keeps DoH for everything else): Firefox → Settings → Privacy
+& Security → **DNS over HTTPS** → "enabled for these domains" → **Add…** →
+`franksdank.cloud` (and `franksdank.casa`). The exception list = domains
+that do NOT use DoH; they fall back to the system resolver, which has the
+route-only drop-in. Or set DoH to **Off** entirely (coarser). Chrome:
+Settings → Privacy and security → Security → "Use secure DNS" → off, or
+"Use the same settings as your system". Do NOT hand-edit `prefs.js` while
+the browser is running — it holds prefs in memory and rewrites the file on
+exit, clobbering the edit.
+
+This is a per-client fix: every LAN client that must reach the homelab by
+name needs Pi-hole as its DNS (or the drop-in) AND a browser that does not
+bypass it with DoH. The server side needs no change. Verified on blankpad
+2026-09-21: the drop-in was present since 09-12, curl returned 200, and
+Firefox 155 with default (ON) DoH was the remaining hole.
+
+## Porkbun zone ops (franksdank.cloud)
+
+Operate the public DNS zone via the Porkbun API v3, run **on Dankclaw** so
+the API key/secret sourced from `/home/dank/serverAppConfigs/.env`
+(`PORKBUN_API_KEY`, `PORKBUN_API_SECRET_KEY`) never leaves the host. Pipe
+scripts to the target with `ssh -o BatchMode=yes dankclaw 'bash -s' <
+/tmp/script.sh`.
+
+Endpoints (all POST, JSON body):
+
+- List: `https://api.porkbun.com/api/json/v3/dns/retrieve/franksdank.cloud`
+  — auth via headers `X-API-Key` + `X-Secret-API-Key` (Bearer is wrong
+  here). Response shape is **top-level `.records[]`**, NOT
+  `.content.records[]` as some docs claim — parse with
+  `jq -r '.records[] | [.id,.name,.type,.content,(.ttl|tostring)] | @tsv'`.
+- Delete: `/dns/delete/{domain}/{id}` (body carries apikey+secretapikey).
+- Edit: `/dns/edit/{domain}/{id}` — pass the full record (`id`, `name`,
+  `type`, `content`, `ttl`, `prio`).
+
+Quirks that bite (verified live 2026-08-31):
+
+1. **`name` is a RELATIVE label on edit.** Passing `ai.franksdank.cloud`
+   yields the record `ai.franksdank.cloud.franksdank.cloud`. Pass the bare
+   label (`ai`) and the API appends the zone itself. Verify after every
+   rename with a retrieve before declaring done.
+2. **Rate limit answers as an HTML 404-style page, not JSON.** A batch of
+   rapid mutations makes some calls return `<!DOCTYPE html>...` where jq
+   then errors (`Cannot iterate over null` / `Invalid numeric literal`). Do
+   not assume the call succeeded — sleep ~8 s and retry the failed ids;
+   verify by re-listing the zone at the end.
+3. Deletes/renames are per-record-id: always retrieve first, map name→id,
+   then mutate by id.
+
+Rename procedure (vhost rename = DNS + Caddy together):
+
+1. `retrieve` → collect ids for the renamed names.
+2. `edit` each id with the new **relative** label, preserving
+   type/content/ttl.
+3. Re-retrieve and confirm `<new>.franksdank.cloud A <ip>`.
+4. In-repo lockstep before deploy: rename the Caddyfile vhost block AND the
+   Pi-hole `dankclaw/pihole/addn-hosts` line in the same commit — a test
+   fails the suite if any Caddyfile vhost lacks an addn-hosts A record on
+   192.168.0.143.
+5. Deploy via `scripts/deploy.sh dankclaw` (plans pihole+caddy from changed
+   paths). Capture the suite's true exit with
+   `out=$(./tests/run.sh 2>&1); rc=$?` — piping to `tail` masks the exit
+   code behind `tail`'s own status and has shipped untested commits here.
+6. Verify live: `dig +short <name> @192.168.0.143` then
+   `curl -o /dev/null -w %{http_code} https://<new-name>` (expect 200/302;
+   first hit triggers dns-01 issuance, which works without inbound port 80).
+
 ## See Also
 
 - [[Pi-hole]] — the LAN DNS server (quadlet, records, web UI)

serverAppConfigs-docs/Infrastructure/Observability Provisioning.md

@@ -0,0 +1,95 @@
+---
+type: reference
+title: Observability Provisioning
+description: How Prometheus config/rules and Grafana datasources/dashboards are provisioned as code in serverAppConfigs — the config triad, deploy copiers, planner mappings, seam tests, entry points for each change type, and API verification.
+tags: [observability, prometheus, grafana, provisioning, deployment]
+timestamp: 2026-10-05T00:00:00-07:00
+---
+
+# Observability Provisioning
+
+The monitoring stack is **provisioned as code**: dashboards, datasources,
+scrape jobs and alert rules live in the repo and ride the normal deploy flow.
+No hand-edits inside containers — they are wiped on recreate and never
+committed.
+
+## The config triad
+
+| Repo path | What it holds |
+|---|---|
+| `shared/prometheus/prometheus.yml` | scrape jobs (global + per-host) |
+| `shared/prometheus/rules/llm-fleet.yml` | alert rules + recording rules |
+| `shared/grafana/provisioning/` | datasources + dashboards (provisioned tree) |
+
+Targets and datasources reference **container names** (`loki:3100`,
+`alloy:12345`), never loopback-published ports; the LLM endpoints are reached
+by LAN IP (`192.168.0.143:7770`, `:7771`, `192.168.0.235:7770`) because they
+are host-netns services, not containers. Prometheus 3 refuses HTML responses
+as metrics — HTML-scraped "probes" are structurally inert and need a blackbox
+exporter (see [[Stale Signal Audit]]).
+
+## The deploy path
+
+`scripts/deploy-remote.sh` carries two copier functions:
+
+- `copy_prometheus_config` → `/opt/prometheus/config/` on the target
+- `copy_grafana_provisioning` → `/opt/grafana/provisioning/`
+
+`scripts/deploy-plan.py` maps changes under `shared/prometheus/**` and
+`shared/grafana/**` to the `prometheus`/`grafana` services, so
+`deploy.sh dankclaw --service grafana` (etc.) restarts only what the change
+touched. Seam tests in `tests/observability_quadlet.bats` pin: unit files
+exist, the copier functions exist, and the provisioned paths match the units'
+mounts.
+
+## Entry points
+
+**Add a scrape target** — edit `shared/prometheus/prometheus.yml`. Decide the
+vantage first: container name if the exporter is a container on the same
+netns; host LAN IP if it is a host process (LLM endpoints); a blackbox module
+if it is an HTTP probe, not a metrics endpoint.
+
+**Add an alert/recording rule** — edit `shared/prometheus/rules/llm-fleet.yml`.
+Current delivered set: 5 alert rules + 1 recording rule for the LLM fleet.
+Rules reload with the prometheus service.
+
+**Add a datasource** — drop a YAML file into
+`shared/grafana/provisioning/datasources/`. Point it at the container name of
+the backend (`http://prometheus:9090`), not loopback.
+
+**Add a dashboard** — drop the JSON into
+`shared/grafana/provisioning/dashboards/`. Provisioned dashboards live in
+unified storage, not (only) the SQLite table — grep the API, not the DB, when
+verifying (see the verify section of [[Fleet Audit and Posture]]).
+
+## Verify through the APIs, not the filesystem
+
+```bash
+# prometheus: rules and targets actually loaded
+curl -s http://127.0.0.1:9090/api/v1/rules | python3 -m json.tool | head
+curl -s "http://127.0.0.1:9090/api/v1/targets?state=active" | \
+  python3 -c "import json,sys; [print(t['labels'].get('job'), t['health'], (t['lastError'] or '')[:60]) for t in json.load(sys.stdin)['data']['activeTargets']]"
+# grafana: datasource + dashboard provisioned
+curl -s -u admin:<pw> http://127.0.0.1:3002/api/datasources | python3 -m json.tool | grep -E '"name"|"url"'
+```
+
+Health answers from a stale process and provisioned resources can hide in
+unified storage — a file existing on disk is not proof the running server
+loaded it. See [[Stale Signal Audit]] for the never-green-target decision
+table (refused = outage, 404/HTML = config defect).
+
+## Current delivered state (2026-09-17)
+
+Prometheus → Alertmanager (`dankclaw/observability/quadlet/alertmanager.container`,
+loopback `:9093`) → host-side Signal relay (`scripts/signal-relay.py`, user
+unit, `192.168.0.143:18093` — the LAN IP, the one bind a container can reach
+via `host.containers.internal`) → signal-cli. Some targets read down for
+structural reasons (HTML-scraped "probes", Gitea metrics off); the remedy is
+the blackbox exporter. See [[Observability]].
+
+## Related
+
+- [[Observability]] — stack topology and the alert delivery chain
+- [[Stale Signal Audit]] — classifying down targets before "fixing" them
+- [[Quadlet Lifecycle]] — the units that host these services
+- [[Deploy State Triage]] — how the planner selects these services

serverAppConfigs-docs/Infrastructure/Prometheus.md

@@ -38,7 +38,7 @@ Authentik; post-auth the Caddy vhost routes straight to `prometheus:9090`) —
 | `node-dankserver` | `192.168.0.235:9100` | host metrics from [[Node-Exporter]] — a different machine, so the LAN address works |
 | `grafana`, `loki` | `grafana:3000`, `loki:3100` | container names on the `dankclaw_default` network; both serve real `/metrics` |
 | `frank-medium` | `192.168.0.235:7770` | the LLM fleet — `vllm:` series; a different machine, so the LAN address works |
-| `frank-large`, `frank-small` | `host.containers.internal:7770`, `:7771` | the LLM fleet — `llamacpp:` series; same-host net=host binds, so via the gateway |
+| `frank-large`, `frank-secondary` | `host.containers.internal:7770`, `:7771` | the LLM fleet — `llamacpp:` series; same-host net=host binds, so via the gateway |
 
 **Three jobs read down, and they are not outages** (audited 2026-09-16):
 

serverAppConfigs-docs/LLM/DFlash DSpark Speculative Decoding.md

@@ -215,9 +215,9 @@ The DFlash draft is not a standalone model. It must pair with its target model.
 
 ## Deployment on Dankserver (NVIDIA RTX 3090)
 
-The Dankserver llama.cpp deployment (`frank-small`, Qwen3.6-35B-A3B) is **active** on Dankserver (port 7770) as of the 2026-08-17 redo (the earlier 2026-08-16/17 retirement of this service was reversed when the PP split was retired). The vLLM material below documents the archived AWQ/DFlash path on the same hardware.
+The Dankserver llama.cpp deployment (`frank-secondary`, Qwen3.6-35B-A3B) is **active** on Dankserver (port 7770) as of the 2026-08-17 redo (the earlier 2026-08-16/17 retirement of this service was reversed when the PP split was retired). The vLLM material below documents the archived AWQ/DFlash path on the same hardware.
 
-> **DFlash retired from active service.** `DFlash` (draft-dflash) was tested initially but did not deliver the desired speed gains under the `frank-small` workload (see ngram-mod benchmarks below). The `Alittlehammmer/Qwen3.6-35B-A3B-DFlash` draft GGUF remains in `/home/dank/models/` as reference material only.
+> **DFlash retired from active service.** `DFlash` (draft-dflash) was tested initially but did not deliver the desired speed gains under the `frank-secondary` workload (see ngram-mod benchmarks below). The `Alittlehammmer/Qwen3.6-35B-A3B-DFlash` draft GGUF remains in `/home/dank/models/` as reference material only.
 
 The vLLM material below documents the archived AWQ/DFlash path on the same hardware. The RTX 3090 is Ampere (compute capability 8.6). Use these conditions:
 
@@ -315,7 +315,7 @@ not switch without re-measuring".
 
 The A/B vehicle for ANY single-GPU llama.cpp knob (batch/ubatch, KV type,
 slot split, quant rung — not just spec flags) is the transient systemd
-drop-in procedure in [[frank-large-benchmark]] and [[frank-small-benchmark]].
+drop-in procedure in [[frank-large-benchmark]] and [[frank-secondary-benchmark]].
 
 ## Is it worth it? (arithmetic before deployment)
 
@@ -419,4 +419,4 @@ measurement that would settle it.
 - [[GGUF Compatibility]] — header parsing and custom tensor-type lock-in
 - [[Llama Cpp Serving Sizing]] — slot-pool and drafter KV sizing arithmetic
 - [[Llama Cpp Multi-Device Splits]] — where the drafter's KV actually lands
-- [[frank-large-benchmark]] and [[frank-small-benchmark]] — the single-GPU env drop-in A/B vehicle
+- [[frank-large-benchmark]] and [[frank-secondary-benchmark]] — the single-GPU env drop-in A/B vehicle

serverAppConfigs-docs/LLM/Dankclaw System Specs.md

@@ -46,7 +46,7 @@ Dankclaw is an AMD APU server for CIRU ROCmFPX llama.cpp inference. The host run
 | Runtime | FastFlowLM 0.9.45 + XRT (installed 2026-08-16; **retired 2026-09-11** with the `amd_iommu=off` decision) |
 | Status | Not in service. Research only. See [[FastFlowLM NPU Inference]] |
 
-The NPU is retired (2026-09-11). The `frank-small` name now serves Gemma4-12B QAT on the RTX 3090 via llama.cpp (see [[frank-small-gemma4-12b]]).
+The NPU is retired (2026-09-11). The `frank-secondary` name now serves Gemma4-12B QAT on the RTX 3090 via llama.cpp (see [[frank-secondary-gemma4-12b]]).
 
 ### ROCm
 

serverAppConfigs-docs/LLM/Dankserver System Specs.md

@@ -17,7 +17,7 @@ timestamp: 2026-09-20
 
 Dankserver is a media/library server and the `frank-medium` LLM host. The
 host runs Ubuntu 26.04 (Resolute Raccoon). The RTX 3090 that served the
-FreeToken/CUDA `frank-small` era was removed on 2026-09-15 and replaced with
+FreeToken/CUDA `frank-secondary` era was removed on 2026-09-15 and replaced with
 two AMD Radeon AI PRO R9700 cards.
 
 ## Hardware Specifications
@@ -107,8 +107,8 @@ hijack `libhsa-runtime64` were purged on 2026-09-15 — see
 The active inference service is `frank-medium.service`: mainline llama-server on
 port 7770, layer-split 32:33 across both R9700s with the DFlash2 drafter pinned to
 ROCm1. See [[frank-medium-llamacpp-dflash2]]. The former
-`frank-small.service` (FreeToken/CUDA) was retired with the 2026-09-15 GPU swap;
-its unit is archived in `~/.AGENTTRASH/20260915-frank-small-retire/host-units/`,
+`frank-secondary.service` (FreeToken/CUDA) was retired with the 2026-09-15 GPU swap;
+its unit is archived in `~/.AGENTTRASH/20260915-frank-secondary-retire/host-units/`,
 and the vllm-radiance replica trio in `~/.AGENTTRASH/20260916-vllm-radiance-retire/`.
 See [[deployment]].
 

serverAppConfigs-docs/LLM/FastFlowLM NPU Inference.md

@@ -13,12 +13,12 @@ timestamp: 2026-09-03
 
 > **SUPERSEDED 2026-08-17.** The Frank redo retired this topology.
 > See [[frank-redo-2026-08-17]] and [[deployment]] for the active configuration.
-> The `frank-small` name and port 52625 then belonged to the vLLM
+> The `frank-secondary` name and port 52625 then belonged to the vLLM
 > service on Dankclaw (nameistoken Quark W8A8); this page is archive.
 
 # FastFlowLM NPU Inference
 
-FastFlowLM (FLM) runs LLMs entirely on the AMD Ryzen AI NPU. It targets XDNA2 chips: Strix, Strix Halo, Kraken, and Gorgon Point. The project describes itself as "Ollama, but deeply optimized for NPUs." As of 2026-08-16, FLM is installed on Dankclaw, the NPU validates, and `frank-small` (port 52625) serves in production. See Promotion.
+FastFlowLM (FLM) runs LLMs entirely on the AMD Ryzen AI NPU. It targets XDNA2 chips: Strix, Strix Halo, Kraken, and Gorgon Point. The project describes itself as "Ollama, but deeply optimized for NPUs." As of 2026-08-16, FLM is installed on Dankclaw, the NPU validates, and `frank-secondary` (port 52625) serves in production. See Promotion.
 
 ## Verdict (2026-08-16)
 
@@ -203,16 +203,16 @@ Notes:
 
 ## Promotion (2026-08-16)
 
-`frank-small` now runs the NPU endpoint on Dankclaw (was the Dankserver llama.cpp service). Unit: `scripts/frank-small.service` (symlinked into `/etc/systemd/system`), wrapper `scripts/serve-flm-small.sh`, tunables `scripts/frank-small.env`. `LimitMEMLOCK=infinity` is set in the unit. Default model: `gemma4-it:e4b`. Swap `FLM_MODEL` in the env to switch.
+`frank-secondary` now runs the NPU endpoint on Dankclaw (was the Dankserver llama.cpp service). Unit: `scripts/frank-secondary.service` (symlinked into `/etc/systemd/system`), wrapper `scripts/serve-flm-small.sh`, tunables `scripts/frank-secondary.env`. `LimitMEMLOCK=infinity` is set in the unit. Default model: `gemma4-it:e4b`. Swap `FLM_MODEL` in the env to switch.
 
 ### Boot policy (2026-08-17)
 
-The `frank-small` unit is disabled for boot. Only the PP pair (`frank-large` + `frank-pp-worker`) starts at boot. The unit file was removed from `/etc/systemd/system`; the running FLM service keeps serving until stopped or rebooted. To restore the endpoint, run these commands on Dankclaw:
+The `frank-secondary` unit is disabled for boot. Only the PP pair (`frank-large` + `frank-pp-worker`) starts at boot. The unit file was removed from `/etc/systemd/system`; the running FLM service keeps serving until stopped or rebooted. To restore the endpoint, run these commands on Dankclaw:
 
 ```
-sudo ln -sf /home/dank/vllmScripts/scripts/frank-small.service /etc/systemd/system/frank-small.service
+sudo ln -sf /home/dank/vllmScripts/scripts/frank-secondary.service /etc/systemd/system/frank-secondary.service
 sudo systemctl daemon-reload
-sudo systemctl enable frank-small
+sudo systemctl enable frank-secondary
 ```
 
 Live config (2026-08-16): `FLM_PMODE=turbo`, `FLM_ASR=1` (loads Whisper-V3-Turbo-NPU2; exposes `/v1/audio/transcriptions`), `FLM_CTX_LEN=131072` (verified max — Gemma4 natively supports 128K and the NPU budget fits it; KV occupancy measured at 0.11% for a 123-token exchange, decode 12.0 t/s, 13 GiB headroom).
@@ -271,12 +271,89 @@ graph TD
 6. Step up in size: 9B, then gpt-oss-20b.
 7. Attempt `qwen3.6-moe:35b-a3b` only after the smaller runs succeed, and only with the production service stopped. Watch `free -h` during the model download and load.
 
+## Registering packs absent from the catalog (NPU2)
+
+Use when adding a non-catalog NPU2 pack (an extraction or fine-tuned model)
+to the FLM registry. Registry facts: user registry
+`/home/dank/.config/flm/model_list.json`, shape
+`{"model_path": "models", "models": {"<tag>": {"<size>": {name, url,
+file_url, size, flm_min_version, files[], vlm, default_context_length,
+max_prefill_len, details{format:"NPU2", family, …}, label[]}}}}`; serve
+resolves tags through `FLM_CONFIG_PATH` at startup, and `flm list` with the
+env var set lists custom tags (`⏬` = not downloaded).
+
+1. Back up the registry JSON first (`cp -a`).
+2. Insert the entry via a python json edit + atomic `os.replace`. Set
+   `files` to the FULL HF file list — **including `vision_weight.q4nx` even
+   for text-only packs**: the qwen3.5-family loader opens it unconditionally
+   and a missing file makes the first request HANG silently (no error log).
+3. **Do not use `flm pull <tag>` for custom entries** — it throws
+   `json.exception out_of_range key '<tag>:<size>' not found` while building
+   the download list. Place files manually into
+   `/home/dank/.config/flm/models/<PackName>/` with
+   `curl -sfL "https://huggingface.co/<repo>/resolve/main/<file>?download=true"`
+   — plain `raw/main/<file>` returns 135-byte LFS pointer stubs, not
+   weights; verify byte counts against the HF tree API before trusting a
+   download.
+4. Kernel packs: copy from the same-shape family pack (shape-match
+   `config.json` on hidden_size/num_hidden_layers/head_dim first) into
+   `/opt/fastflowlm/share/flm/xclbins/<PackName>/` via `sudo cp -a`
+   (root-owned dir).
+5. Deploy/restart the serve unit (the deploy path, not ad-hoc restarts) so
+   the new process loads with all files present.
+6. Verify with a REAL completion request (`model: "<tag>:<size>"`) —
+   `/v1/models` alone proves registration, not loadability.
+
+Hard constraints: only ONE `flm serve` process can hold the NPU hardware
+context — a second concurrent serve process fails every load with
+`DRM_IOCTL_AMDXDNA_CREATE_HWCTX err=-22` (aie2 mgmt msg opcode 0x2 status
+0x2000003); never run scratch/verify instances while production holds the
+NPU. Tag switching happens INSIDE one serve process; the first request
+after a model switch pays ~6–11 s cold-load cost (visible in
+`usage.load_duration`). Over ssh, never `pkill -f "<pattern>"` with a
+pattern present in your own command line — use bracket-guarded patterns
+(`[q]wen3.5:4b`) or exact PIDs.
+
+## Tag load-failure diagnosis (200 with an error body)
+
+`POST /v1/chat/completions` answers **HTTP 200** with a flat
+`{"error":"Failed to load <tag> model!"}` and no `choices`; `/v1/completions`
+returns a nested OpenAI-shaped `server_error` (code 500) also under HTTP
+200. A status-code-only client reads this as a successful empty completion —
+**always check for the `error` key**. Two distinct causes, distinguished via
+`journalctl -u <flm-service> --full`:
+
+1. **Unknown tag** (not in the registry): immediate error, nothing loaded.
+   Correct behavior, not a bug.
+2. **Registered tag whose kernel pack is missing**: FLM resolves the tag,
+   downloads missing weight files on the fly (first request hangs ~90–110 s),
+   then fails `Failed to load model: No such file
+   '/home/dank/.config/flm/xclbins/<PackName>/<kernel>.xclbin'`; subsequent
+   attempts fail fast (~2 s). Root cause: the serve script exports
+   `FLM_XCLBIN_PATH=/home/dank/.config/flm/xclbins` (user dir), so every
+   kernel lookup resolves under the user dir, which only contains symlinks
+   someone linked. **A tag listed in `/v1/models` is NOT proof it is
+   loadable on this host.** Fix: one symlink per missing pack
+   (`ln -s /opt/fastflowlm/share/flm/xclbins/<PackName>
+   /home/dank/.config/flm/xclbins/<PackName>`) — verified live: after the
+   symlink the first request loaded and answered in 7.4 s wall.
+
+Switch-cost behavior: the `model` field routes per-request (no restart
+needed); a request whose model differs from the resident one triggers a full
+reload (7–11 s wall for an 8-token completion on ~4B-class packs);
+alternating tags pays the swap each time. Single-lane NPU: concurrent
+requests queue FIFO behind the current lock holder (`NPU Locked!` →
+`NPU Lock Released!`). Probe recipe: `/v1/models` for registered tags; grep
+the journal for `Failed to load model` (the quoted xclbin path names the
+missing pack dir exactly); diff `ls ~/.config/flm/xclbins/` (symlinks =
+installed) against `/opt/fastflowlm/share/flm/xclbins/` (available).
+
 ## Incident: qwen3.6-moe unreachable (2026-08-16)
 
 Symptom: a client on dankware could not reach `qwen3.6-moe:35b-a3b` on port 52625. The journal shows three separate causes in the 14:19-15:03 window.
 
 1. **EXEC 203 crash loop (14:19-14:20).** The exec bit was lost on `serve-flm-small.sh` on the target. The unit failed `status=203/EXEC: Permission denied` four times, then started after the bit was restored. The git mode is `100755` (commit `3f42210`), and a `git pull` restores the bit. Verify with `git ls-files -s scripts/serve-flm-small.sh`.
-2. **Wrong model ID (14:29-14:52).** The client config still used the model ID `frank-small` (a service alias). FLM rejects that tag with `[ERROR] Model tag 'frank-small' is not supported.` Each such request also triggered the unknown-tag fallback below.
+2. **Wrong model ID (14:29-14:52).** The client config still used the model ID `frank-secondary` (a service alias). FLM rejects that tag with `[ERROR] Model tag 'frank-secondary' is not supported.` Each such request also triggered the unknown-tag fallback below.
 3. **Swap churn (14:48-15:03).** The swap `gemma4-it:e4b` → `qwen3.6-moe:35b-a3b` restarted the service several times. Each load of the 23 GB model takes about 1.5-2.5 minutes at 262144 ctx. The endpoint does not serve during a load.
 
 Fix: the client model ID must be the FLM registry tag `qwen3.6-moe:35b-a3b`. The working pi agent entry in `~/.pi/agent/models.json` is `baseUrl http://192.168.0.143:52625/v1`, `id qwen3.6-moe:35b-a3b`, `contextWindow 262144`, `reasoning true`, `input [text, image]`. Verified end-to-end from dankware (192.168.0.226) at 15:32 on 2026-08-16.
@@ -313,7 +390,7 @@ pi sessions failed against this endpoint with `Stream ended without finish_reaso
 - **The FLM serve process is not under systemd.** A reboot of Dankclaw kills it. The PP head service is enabled and returns. The FLM process must be restarted by hand. A unit file is a natural next step.
 - **gpt-oss reasoning budget.** The model can spend its full token budget on `reasoning_content` and return empty content. Clients must handle or cap this.
 - **No llama.cpp GGUF path.** FLM uses its own pre-compiled kernels. Our UD-Q8_K_XL and HauhauCS builds are not importable. NPU work means FLM-specific model tags.
-- **NPU production role (2026-08-16).** `frank-small` serves `qwen3.6-moe:35b-a3b` on port 52625 (swap churn and the unknown-tag incident are recorded above). The remaining NPU content (benchmarks, runbook steps) is still research until a step lands and is recorded.
+- **NPU production role (2026-08-16).** `frank-secondary` serves `qwen3.6-moe:35b-a3b` on port 52625 (swap churn and the unknown-tag incident are recorded above). The remaining NPU content (benchmarks, runbook steps) is still research until a step lands and is recorded.
 
 ## Citations
 

serverAppConfigs-docs/LLM/FreeToken Builds.md

@@ -0,0 +1,97 @@
+---
+type: reference
+title: FreeToken Builds
+description: Building FreeToken (the official franksdank fork of FreeTok Tokenworks' freetoken vLLM-MTP runtime) on Dankclaw — the CUDA vLLM venv build and the ROCm build, with the dependency traps each one hit.
+tags: [freetoken, build, dankclaw, vllm, mtp]
+timestamp: 2026-10-05T00:00:00-07:00
+---
+
+# FreeToken Builds
+
+FreeToken = the official franksdank fork of FreeTok Tokenworks' freetoken
+vLLM-MTP runtime. Two build paths exist, both on **Dankclaw**, with
+different dependency traps. Both were verified end-to-end 2026-09-11/12.
+
+## Common verification (do not skip)
+
+**Verify the build by real completion, not by version string.** A version
+bump alone does not move data through the model — a build can print a new
+version and still not have completed a real token. Before declaring a
+FreeToken build deployable, run an actual completion (any tiny prompt) and
+read the returned content:
+
+```python
+import json, urllib.request
+req = urllib.request.Request("http://<host>:<port>/v1/chat/completions",
+    json.dumps({"model": "frank-secondary", "messages": [{"role": "user",
+    "content": "Hi"}]}).encode(), {"Content-Type": "application/json"})
+print(json.load(urllib.request.urlopen(req, timeout=600)))
+```
+
+A 200 with a real `choices[0].message.content` is the only build green.
+
+## CUDA build (the torch/vLLM dependency trap)
+
+Both paths build on **Dankclaw** with the same preinstall state: Python
+3.12.10 in the system venv and **Torch 2.10.0.dev** (nightly, repo
+download.pytorch.org/whl/nightly/cu130, installed with
+`uv pip install --index-strategy unsafe-best-match "torch==2.10.0.dev<date>+cu130"
+--index-url https://download.pytorch.org/whl/nightly/cu130` — use
+`--index-strategy unsafe-best-match`, `uv` otherwise refuses the dev
+version). Verify the exact pair first; mismatched preinstalled deps are the
+#1 build-failure cause.
+
+Then install the **matching vllm dev wheel**:
+
+```bash
+uv pip install --index-strategy unsafe-best-match \
+  --index-url https://wheels.vllm.ai/<commit-sha>/vllm-0.6.6.dev0+g<shortsha>.cu130-cp312-cp312-linux_x86_64.whl \
+  --extra-index-url https://wheels.vllm.ai/nightly/cu130
+```
+
+The wheel URL pattern is fixed:
+`https://wheels.vllm.ai/<full-commit>/vllm-0.6.6.dev0+g<8-char>.cu130-cp312-cp312-linux_x86_64.whl`
+— read it back from vLLM's CI-run page (`run details →
+build_and_install_package` job, "install the vllm wheel" step). **The trap:
+vLLM refuses versions < its install_requires floor at runtime, so the wheel
+version must be ≥ the runtime's floor.** A wheel built at a lower base
+(0.5.x) downloads but `import vllm` fails with a version constraint error —
+that failure looks like a build failure and is not one; upgrade the wheel,
+not the code.
+
+If `uv pip install -e . --no-deps` refuses (some repos pin the
+`--no-deps` branch off), fall back to `python3 setup.py` inside the venv,
+after `python -m venv --copies .venv && source .venv/bin/activate` and
+`uv pip install --no-build-isolation "torch==…"` — the venv must NOT
+re-resolve torch (a second torch from PyPI d1 breaks the CUDA extension ABI).
+
+## ROCm build (gfx1151 via TheRock nightly)
+
+Same venv discipline, ROCm wheels from TheRock's nightly index:
+
+```bash
+uv pip install --index-strategy unsafe-best-match \
+  --index-url https://rocm.nightlies.amd.com/v2/gfx1151-dcgpu-supports_fbsgens_split_0717-rc1/ \
+  "torch==2.10.0.dev<date>+rocm" "triton==3.6.0" "pytorch-triton==3.6.0"
+```
+
+Then `vllm` itself (`uv pip install -e . --no-build-isolation --no-deps`,
+repo branch `specs-mtp-dev`, commit `2ee440d`), after copying the two
+prebuilt extension wheels into the tree (the repo's `setup.py` skips
+compiling them when found). The ROCm build also needs
+`install-fastrak-gfx1151.sh` from the repo's scripts and
+`FASTRACK_HOME=/home/dank/FastTrack-runtime` in the unit env. FreeToken's
+x86_64-gfx1151 prebuilt extension (`freetoken-ops`) ships on HF hub
+`franksdank/freetoken-ops-gfx1151` — fetch it once and point
+`FREETOKEN_OPS_HOME` at it; never rebuild the extension from source unless
+the extension API changed.
+
+`--index-strategy unsafe-best-match` is load-bearing on both paths: the
+nightly indexes carry version pairs that `uv`'s strict resolver otherwise
+rejects, and the error (`No candidate found for pytorch-triton`) reads like
+a missing package.
+
+## Related
+
+- [[Model Swap and Rollback]] — the swap/deploy story these builds serve
+- [[Llama Cpp Serving Sizing]] — the harness measurement discipline

serverAppConfigs-docs/LLM/Heterogeneous Split Triage.md

@@ -0,0 +1,159 @@
+---
+type: reference
+title: Heterogeneous Split Triage
+description: Triage before building when someone asks to split a model across unequal accelerators on one host (24 GB GDDR6X + LPDDR APU, PCIe x4 link) — link characterisation, the real bottleneck, what placement never buys, UMA pools, drafter splits, and the verdict ranking.
+tags: [heterogeneous, multi-device, llama-cpp, triage, pcie]
+timestamp: 2026-10-05T00:00:00-07:00
+---
+
+# Heterogeneous Split Triage
+
+Produced the 2026-09-16 no-go verdict on splitting Qwen3.8-Flash-Next across
+the RTX 3090 + Strix Halo pair. Use it to answer "could we split model X
+across device A and device B" with arithmetic instead of enthusiasm.
+
+## 1. Characterise the link (bytes vs latency)
+
+```bash
+lspci -vv -s <bdf> | grep -E "LnkCap:|LnkSta:"          # capability vs actual
+nvidia-smi --query-gpu=pcie.link.gen.max,pcie.link.gen.current,pcie.link.width.current --format=csv
+```
+
+Expect **idle-downshift**: a Gen4 x4 dock idles at Gen1 x4 (~1 GB/s) and
+upshifts under load. Measure while the device is busy, or you size the design
+against the wrong number. Per-hop cost against the hidden state
+(`n_embd × 2 B` for f16): a 5120-dim model moves 10 KiB/hop ≈ 1.3 µs at
+Gen4 x4.
+
+- **Bytes per token ≈ hops/token × hidden state** — typically ~1 MiB/token
+  for 2 hops/layer × 64 layers ≈ 0.3 % of a 40 ms token. **Bytes are almost
+  never the constraint.**
+- The real budget line is **hop COUNT**: every device boundary is a
+  host-side sync. A runtime that splits per *op* hits ~100 splits/token and
+  pins the GPU at ~5 % SM. Measure `graph splits` in the loader log; never
+  assume.
+- **Design rule: partition by where the weights live, never by where the
+  activations flow.** Resident weights, activations crossing.
+
+## 2. Get the workload's real bottleneck (KV is usually it)
+
+```
+KV/token = kv_bytes_per_position × context
+```
+
+Attention reads the whole cached context every token, so KV/token scales
+with context — compare it against expert/weight traffic per token and total
+token time. At long context on a hybrid arch (few attention layers, many
+positions), **KV dominates**. Consequences:
+
+- The highest-leverage placement is **KV cache on the fastest-memory device,
+  together with its attention compute** — never KV alone: separating them
+  puts KV/token over the link, which is fatal (GBs across 7.88 GB/s =
+  hundreds of ms/token).
+- Check the fit: `kv_positions_per_slot × slots × bytes_per_position +
+  attention_trunk ≤ VRAM`.
+
+## 3. What placement never buys
+
+Prefill is compute-bound and expert weights are read once per batch, so
+moving them to faster memory buys nothing while adding a serial stage.
+Expect a regression, and size it from calibration: on Strix Halo + RTX 3090
+(Qwen3.8-Flash-Next, ik_llama.cpp) moving 12.5 % of the expert mass to the
+GPU gave **decode +12 %, prefill 192 → 32 tok/s**. A
+`1/(f_big/BW_slow + f_fast/BW_fast)` harmonic calculation reproduces that
+decode gain to within ~2 points — usable for decode projections, and it
+assumes sequential stages (true at batch 1; no pipelining).
+
+## 4. Is the second device even a separate pool?
+
+On a UMA APU, check before assuming additivity:
+
+```bash
+cat /sys/class/drm/card*/device/mem_info_vram_total   # BIOS carve-out (small)
+cat /sys/class/drm/card*/device/mem_info_gtt_total    # aperture onto SYSTEM RAM
+rocm-smi --showmeminfo all | grep -i gtt              # used == what the CPU also sees
+```
+
+A large GTT window means the "GPU memory" is the *same* LPDDR under one
+memory controller: adding a stage there adds a compute engine, not capacity
+or bandwidth. Only the discrete card's VRAM is additive. Fencing more
+(`amdgpu.gttsize`, BIOS UMA size) just removes bytes from CPU-visible RAM.
+
+## 5. Speculative decoding as the "cheap" split
+
+A drafter on the *other* device is bandwidth-free (token ids per round, tens
+of bytes), so it is tempting. Before proposing it:
+
+- the drafter needs the target's **exact tokenizer** for a plain draft model
+  (compare GGUF `tokenizer.ggml.tokens`, never trust the family name), or it
+  must be a head **trained for that target** (EAGLE/DFlash/MTP are
+  conditioned on the target's hidden states and are not portable between
+  models);
+- a plain drafter costs **K sequential forwards per round**, so it must be
+  10–50× faster than the target. Break-even with `r = t_draft/t_target`:
+  `rate_ratio = A/(K·r + 1)`, `A = (1−p^(K+1))/(1−p)`; at `r ≈ 0.4–0.5`
+  (a drafter only ~2× faster) break-even needs `p ≈ 0.7+` at K=3;
+- compare against the **incumbent**, not the no-draft baseline: a target
+  already running its own MTP head is a high bar;
+- check the runtime accepts an external drafter at all — closed engines may
+  enumerate a fixed drafter set and 400 anything else (capability gates
+  keyed on the loaded checkpoint). That is a design boundary, not a config
+  gap. In llama.cpp the portable mechanism is `--spec-draft-device/-devd`
+  (draft gets its own device list) plus `-ngld`/`-otd`/`-ctkd -ctvd`.
+
+## 6. Expressing a split (llama.cpp, the usual vehicle)
+
+```bash
+-dev CUDA0,Vulkan0 -sm layer -ts F0,F1        # contiguous layer ranges (simplest)
+-ot "\.attn_.*=CUDA0" -ot "\.exps\.=Vulkan0"   # per-tensor, by ROLE (KV co-location)
+```
+
+- buffer types match by name at load (`ggml_backend_buft_name`), so device
+  names work as `-ot` values;
+- `-cmoe`/`-ncmoe` are **CPU-only**; device-side MoE placement is `-ot`;
+- several backends in one binary is fine (CUDA + Vulkan is the practical
+  pair; HIP + CUDA do not coexist) — `GPU_BACKEND_A=OFF` in the existing
+  build's CMakeCache tells you what needs adding;
+- contiguous ranges are the WRONG split when KV bandwidth dominates: they
+  drag half the attention layers (and their KV) back to the slow pool.
+
+## 7. Hazards checklist
+
+- Paged/sidecar tables (huge n-gram/embedding tables) must stay where they
+  are read from; moving them is pure loss.
+- Keep `mmap` on for models with huge embedding tables (anonymous memory
+  gets the process cgroup-killed).
+- CUDA graph capture can abort on unusual graph shapes (hyper-connections)
+  — check before assuming optimisations apply.
+- A brand-new backend on the second device may not implement the arch's ops
+  at all: make the first measurement a **baseline of the runtime on that
+  device alone** — a split's arithmetic is worthless if the baseline is
+  worse than the incumbent engine.
+- A split consumes the fast device entirely — check what else it was
+  serving before trading it away.
+- Benchmark discipline on production hosts: stop and mask first
+  ([[Fleet Ops Playbook]]).
+
+## 8. Verdict shape
+
+Rank and report; prefer the boring answer when it wins:
+
+1. **Two independent endpoints** (zero shared traffic) — often the real
+   answer to "what's a useful combination", especially over a narrow link.
+2. **One process, weights-resident, KV co-located with attention on the
+   fast pool** — promising and quantifiable (+% decode at long context);
+   needs a dual-backend build and a baseline measurement.
+3. **Hot-expert cache / per-token weight streaming** — affordable only if
+   the miss set per token is small (a few experts; check
+   `misses × bytes × tok/s` against link bandwidth) and overlapped; injects
+   PCIe latency into the critical path; no mainstream runtime implements a
+   dynamic expert cache.
+4. List explicitly what does NOT help: TP/collectives across unequal
+   devices, KV on the far device, model loading over the link,
+   prefill-motivated placements.
+
+## Related
+
+- [[PCIe Link and Atomics]] — the link and collective walls underneath
+- [[Llama Cpp Multi-Device Splits]] — the deployed split shapes
+- [[Llama Cpp Serving Sizing]] — KV and pool arithmetic

serverAppConfigs-docs/LLM/LLM Bench Windows and Measurement.md

@@ -0,0 +1,271 @@
+---
+type: reference
+title: LLM Bench Windows and Measurement
+description: Stopping-and-restoring benchmark windows on the fleet LLM hosts (llama-bench sweeps, HTTP param benches, multi-arm vLLM offload sweeps), honest benchmark methodology, and the usage-counter measurement law under speculative decoding.
+tags: [benchmarking, llama-bench, ab-testing, speculative, vllm, fleet]
+timestamp: 2026-10-05T00:00:00-07:00
+---
+
+# LLM Bench Windows and Measurement
+
+Fleet law first: **STOP the production service before any GPU experiment**
+(AGENTS.md Benchmarking Policy) — a bench beside a live service starves the
+GPU and corrupts results; an unisolated GPU experiment has precedent for
+wedging a host until power cycle. Reference harnesses:
+`scripts/bench-batch-ubatch.sh`, `scripts/autoresearch.sh`,
+`scripts/autoresearch/frank-primary-arms.sh`.
+
+## 1. The window structure (one config per arm, a driver owns the window)
+
+1. **Stop + gate before every arm** — `wait_for_clear`: unit NOT active AND
+   no RUNNING container of that name (`podman ps`, not `-a`; `-a` counts
+   EXITED containers and can never clear — `rm -f` the exited one first) AND
+   port free (`ss -ltn`). The graceful-stop race cost a whole arm 2026-10-01:
+   `ExecStop=podman stop -t 120` drains up to 2 min; a driver that sleeps 3 s
+   then polls `/health` gets **200 from the still-running PRODUCTION engine**
+   — the arm "passes" with production's boot log, then `podman exec` dies
+   with `container state improper`.
+2. **Boot the arm as the service, not a scratch port** (no scratch ports on
+   LLM hosts — one card, one server). The serve script takes its config via
+   env-file override (`FRANK_SECONDARY_ENV_FILE`, `FRANK_PRIMARY_AR_ENV_FILE`)
+   pointing at a staged copy; the checkout is never mutated.
+3. **Fast-fail the health wait**: poll `podman ps | grep -qx <name>` inside
+   the loop — a refused boot dies in one poll instead of 20 min. Capture the
+   serve script's FULL stdout on failure (a `tail -8` loses engine-init root
+   causes; raise happens above the tail).
+4. **Measure** (§4), capture boot-log engine facts (KV pool, arena slots,
+   SpecDecoding lines) — they survive measure crashes.
+5. **Teardown + converge**: kill by explicit PID/pgid (never pattern kills),
+   `podman rm -f` scratch, wait `MemAvailable ≥ ~230 GiB` on dankserver
+   before the next boot; restore production from the EXIT trap and verify the
+   **conjunction oracle**: health 200 + `is-active` + `is-enabled` +
+   mask-symlink clean. A bare `systemctl start` is not a restore oracle.
+   (`systemctl mask` FAILS on a relinked unit — log it, explicit stop is
+   enough: systemd never auto-restarts an explicitly stopped unit.)
+6. **Warm boots cycle ~2 min** (page cache), cold ~8 min — size timeouts for
+   mixed speeds.
+
+**Destructive tails (2026-10-01):** a scratch script whose tail does
+`systemctl stop` + `podman rm -f <PRODUCTION container>` fires later if a
+probe loop survived the driver kill — it can destroy a just-deployed
+production engine (signature: deploy exits green, then seconds-to-minutes
+later `Exited (0)`, `Deactivated successfully`, clean exit = removed, not
+crashed). Before ANY deploy: `pgrep -af` every scratch script of the session
+and kill strays by explicit PID; killing the dev-side job does NOT kill the
+host-side `deploy-remote.sh` (two of them once managed the same unit).
+
+Two failure shapes inside the window that read as "the bench died":
+
+- **Stale reads from a stopping container.** `podman stop` answers while the
+  engine inside is still flushing; a `podman logs` taken in that window
+  returns the boot-prefix only. Poll `podman ps` for `Exited (0)` (or
+  `podman wait`) before reading logs, or you will write "no results" into
+  the table for a run that has them.
+- **A D-state worker wedges the stop.** A bench child that wedged in the
+  kernel ignores SIGTERM, so `podman stop -t N` burns its full timeout and
+  then the container is `Stopping` forever. Kill by container first
+  (`podman kill <ctr>` → SIGKILL path), verify the process is gone from
+  `podman ps` AND `nvidia-smi`, and only then re-serve. Never let the
+  window script proceed to a redeploy with a zombie bench worker holding
+  VRAM.
+
+## 2. llama-bench sweeps (offline batch/ubatch/split/KV/depth)
+
+**What llama-bench cannot express — decide BEFORE designing:** no
+`--ctx-size` (`n_ctx = n_prompt + n_gen + n_depth`, so a bench can NEVER prove
+activation buffers fit beside the DEPLOYED KV — only the real unit answers
+that), no `--parallel` (no slot shape, no concurrency), no `--spec-*`
+(`tg` numbers are the drafterless baseline, useful as a control only), and a
+warmup run is itself a full-size run. `-d N` is a depth PREFIX (paid once per
+cell via a `llama_state_seq_*` save/restore); without it a `tg` run is at
+~zero depth and meaningless for long conversations.
+
+**The separator trap:** `/` separates devices WITHIN one configuration, `,`
+separates sweep values — for both `-dev` and `-ts`. `-ts 36,29` parses as the
+single value 36 → everything on one card. `--tensor-split 36,29` (server) →
+`-ts 36/29` (bench). Ranges exist: `-ub 512-2048+256`.
+
+**Harness shape:** one invocation per cell (an infeasible cell calls
+`exit(1)` and kills every later cell of a cross-product invocation — and
+infeasibility is usually the question); TSV row per cell as it completes +
+`bench.json` (`-o json`) + `bench.log` (`--progress`) + a 2 s `rocm-smi
+--showmeminfo vram` VRAM trace per cell (a cell that allocated less than
+intended looks fast); resume by skipping non-empty cell outputs; stop the
+service in the harness, restore from `trap … EXIT`, verify afterwards. Match
+the deployment's load flags exactly (`-ngl -sm -ts -fa -ctk/-ctv -t`).
+
+**Run under a transient unit, not a shell:**
+
+```bash
+sudo systemd-run --unit=<name> --collect --no-block --property=Type=oneshot \
+     --property=LimitMEMLOCK=infinity --property=User=dank \
+     /home/dank/serverAppConfigs/scripts/<harness>.sh
+```
+
+Deploy units set `LimitMEMLOCK=infinity` for ROCm staging; an SSH shell has
+8 MiB and fails the load for an unrelated reason. This also survives ssh
+disconnects. **Boot rungs under the production environment, not a bench
+shell** — a rootless shell probe booted 3×262144 bf16 that production refused
+(systemd `User=` + memlock change what the pinned-expert path holds):
+`sudo systemd-run --uid=<user> --unit=<name> -p LimitMEMLOCK=infinity -p
+Restart=no env VARFILE=… /path/serve.sh`.
+
+**Verify before burning hours:** dry-run against a STUB binary with
+`SERVICE=<nonexistent-name>` (the trap/stop never reached), tiny grid,
+including one forced-`exit 3` cell — prove the TSV row, failure capture,
+resume logic. Validate the real flag set in ~1 s by pointing `-m` at a
+nonexistent file. **Reading:** bracket the sweep (reference cell first AND
+last); check per-cell `stddev_ts` (a difference smaller than spread is not a
+result); single-sequence decode is usually flat across batch/ubatch (prefill
+knob); batch ≥ ubatch (llama.cpp clamps ubatch to batch, silently duplicating
+smaller-batch cells); report what the grid cannot prove (drafter, slots, real
+context).
+
+## 3. HTTP benches of server-side parameters
+
+`llama-bench` benches the bench binary, not the running server — server-side
+params (`--threads-batch`, `--ubatch-size`, `--spec-type`, `--parallel`) need
+HTTP:
+
+- **Prefill**: large prompt, `max_tokens=1` (server crashes on 0), measure
+  `prompt_tokens / wall_time`.
+- **Decode**: `max_tokens=16384` minimum, NEVER 1 (warmup, cache growth and
+  spec acceptance all stabilize over thousands of tokens); measure
+  `predicted_per_second` from `timings` or `completion_tokens / wall_time`;
+  temperature 0.
+
+Known fleet harnesses: `scripts/bench-threads-batch-http.py`,
+`scripts/bench-decode-threads-batch.py`, `scripts/bench-spec-decode.py`.
+sglang-specific: `max_tokens=1` on `/v1/chat/completions` with the qwen3
+reasoning parser streams zero deltas — use `/v1/completions`.
+
+## 4. The measurement law (under speculative decoding this is absolute)
+
+**Never count SSE chunks.** Rejected drafts break the 1:1 chunk↔token
+mapping; chunk counting reported 8.3 tok/s for a config whose server counters
+said 23–26 — a 3x false regression that nearly abandoned the change that
+worked. Always:
+
+```python
+decode_tok_s = usage.completion_tokens / (wall - ttft)
+# stream=True, stream_options: {include_usage: true}; ttft = first content-bearing chunk
+```
+
+vLLM's 10 s `Accepted throughput` log lines are a rolling window, not a
+measurement (6.2 → 13.1 across adjacent windows on identical settings) — use
+them for acceptance STATISTICS only (`Avg Draft acceptance rate`, `Mean
+acceptance length`, per-position rates). Warm pass + ≥3 runs, median;
+temperature 0; ≥256 tokens. **A decode "jump" with early-stopped completions
+(ct < 256) and acceptance > 90 % is degenerate generation** (the fork's
+repetition-halt fired at ~140 tokens on corrupted numerics) — inspect text
+before believing a win. Run-to-run spread is 5–10 %: repeat arms; one sample
+nearly shipped the wrong setting.
+
+## 5. Honest methodology
+
+- **Match the corpus to the workload.** Token soup under-reports speculative
+  decoding (2.9–3.6 tok/update vs 3.86–5.8 on real code) and mis-ranks draft
+  settings; acceptance, not step time, moves throughput. Build the corpus
+  from the repo's own sources: `{"prompt": "<instruction>\n\n<file body>"}`,
+  `--custom-output-len`.
+- **Salt every run** with a unique per-arm token in the first bytes — it is
+  the prefix-cache key (and a proxy routing key). Without it arm 2 measures
+  arm 1's warm cache.
+- **Two workload shapes need two clients:** distinct prompts (batch
+  behaviour, prefill parallelism) and sequential turns with growing history
+  (TTFT of the first streamed chunk per turn — the only measurement that
+  exposes prefix-cache behaviour). These pull in opposite directions: a cold
+  batch spreads prefill across replicas, a conversation's next turn only pays
+  if it lands where its prefix is cached.
+- **Read the engine's ledger, not the card:** `Available KV cache memory` /
+  `GPU KV cache size: N tokens` / `Maximum concurrency for M tokens per
+  request: X.Xx` is the capacity number. vLLM's `Prefix cache hit rate:` line
+  is a LIFETIME average across every prompt the replica ever saw — send the
+  same prompt twice to an idle engine and compare TTFTs (measured 22.3 s cold
+  → 1.4 s warm for a 50k prefix). `GPU KV cache usage` says the pool is not
+  full; it does NOT say a prefix survived. The KV budget is NOT monotonic in
+  chunk size (5.44 GiB at CHUNK 8192, 4.22 at 16384, 4.16 at 4096) — measure
+  three points before treating a knob as a lever.
+- **Soak before believing:** 15–30 min of concurrent families, count failures
+  AND latency; report hit/miss split and medians with min/max — the median of
+  a bimodal distribution describes neither mode.
+- **Verification that is not the benchmark:** run the client the fleet
+  actually uses; and a deployed unit is not proven until an unattended reboot
+  brings it back.
+- **Record the "no" answers** with numbers and mechanism — placement balance,
+  concurrency admission and chunk-size tweaks all measured flat or worse
+  here; the wall was arithmetic (cached prefixes + live sequences share one
+  pool).
+
+## 6. Sweep hygiene (the arms that produced wrong verdicts)
+
+- **Prefetch uniformity:** every arm in a sweep carries the same
+  expert-prefetch setting. The 2026-09-29 campaign measured losing arms with
+  prefetch ON and incumbents OFF; untaxed, verdicts inverted (no-spec
+  12.11 → 41.47; gather 32×64 parity → +18.7 %). Never re-read a taxed
+  verdict.
+- **Knob-landed preflight:** for every `-e` knob, confirm
+  `podman exec <ctr> printenv <VAR>` — a knob the fork silently ignores
+  produces a "parity" arm that is really the default (v1 of the gather sweep
+  lost all 3 arms this way; serve-script hooks are additive and can drop
+  vars).
+- **Always run an explicit OFF arm:** the 2026-09-29 campaign's only win was
+  deleting a baseline's silent `--expert-prefetch 1` (+40–51 %). Challengers
+  compare against default-off, not "incumbent + one flag".
+- **Pin the capacity bar:** `--kv-cache-memory-bytes` (contract) rather than
+  autofit (remainder) so pool numbers are reproducible; compute need =
+  tokens × measured B/token (hybrid models: KV dtype does NOT double mamba
+  states — measure, don't assume 2x).
+- **Report decode AND prefill (TTFT) per rung:** a decode-only view nearly
+  deployed a 2x-prefill-tax config. Per-depth tables, never a single median —
+  one fork's gather win grows with depth (54 t/s @1k, 39.5 @short).
+- Container recreate (`podman rm -f`) wipes `podman cp`'d harnesses — re-copy
+  `mtp_measure.py`/`bench_iter.py` after EVERY recreate, or mount them.
+
+## 7. Bash laws that ate windows (each cost an arm)
+
+1. `${VAR:+SPEC=$1}` in front of a command expands to a bare word →
+   `SPEC=3: command not found`, nothing runs. Pass literal assignments:
+   `SPEC="${1:-}" bash serve.sh`; the callee's `${SPEC:+--flag}` skips empty.
+2. `${2WAY:-0}` is fatal `bad substitution` (digit-leading name), and inside
+   an if/else it aborts ONLY that compound statement — the script CONTINUES,
+   every arm tears down clean with zero measurements. Detection law: `ls`
+   the per-arm `.measure.txt` files before trusting any summary; arm echoes
+   are not evidence.
+3. `local a=$1 b=$x` expands `$x` before the builtin assigns — under `set -u`
+   it aborts AFTER the harness stopped production. Two statements.
+4. `sed -i` renames the file; already-running bash keeps the OLD inode —
+   mid-flight patches miss arms launched in the preceding minutes. Patch
+   early or run a make-up window.
+5. Never heredoc python through ssh→zsh→bash — write to a file, ship it
+   (`ssh host 'cat > /path && chmod +x /path' < local.sh`); nested quoting
+   fails silently. Same trap in `git commit -m`: backticks become command
+   substitution.
+6. No comments INSIDE a backslash-continued command (the continuation joins
+   the comment); verify multi-line command construction with `bash -n` AND a
+   render test.
+
+## 8. Results location and recovery (frank-primary arms)
+
+| copy | path |
+|---|---|
+| per-arm evidence | `$ARMS_DIR` = `~/.jobs/<run-stamp>/` (`.rc`, `.done` beside it) |
+| summary | `$ARMS_DIR/summary.tsv` = `arm rc decode_tps decode_tps_t0.7 kv_tokens boot_s` + `<arm>.out/.env/.overlay/.rc/.metrics` |
+| server-side last window | `/home/dank/.autoresearch-frank-primary/run.json` |
+| durable memo | `.../last/run.json` + `last/hash` (promoted only after bench AND verified restore) |
+| fallback | `~/.omp/agent/sessions/<ws>/<session>/<N>.{shake,bash}.log` (permanent) |
+
+`/tmp/frank-primary-arms-*` and `/tmp/armstub/` are stub self-tests —
+constants 99.9/88.8 identical across arms by construction; **identical
+numbers across arms == stub or bug, never a measurement.** After compaction,
+re-harvest with `grep -rhoE 'decode_tps[a-z0-9_.]*=[0-9.]+'` over the session
+logs. A window costs ~20 min warm; a detached runner's timeout ≥ 3600 s per
+window.
+
+## Related
+
+- [[RDNA4 MoE Expert Offload VRAM Budget]] — the plan arithmetic being swept
+- [[DFlash DSpark Speculative Decoding]] — spec A/B protocol and shapes
+- [[LLM Performance Diagnosis]] — profiling and attribution methods
+- [[Remote Job Runner]] — detached execution of long sweeps
+- [[frank-medium-llamacpp-dflash2]] — the measured ubatch-optimum history

serverAppConfigs-docs/LLM/LLM Performance Diagnosis.md

@@ -0,0 +1,221 @@
+---
+type: reference
+title: LLM Performance Diagnosis
+description: Probing why a served LLM is slow — the flat-depth probe, the expert-offload duty-cycle arithmetic, kernel-time attribution via the profiler window, live telemetry without benchmarking, RAM-bandwidth ceilings, IBS tensor attribution and prefix-cache-aware routing.
+tags: [performance, profiling, moe, pcie, prefix-cache, routing, telemetry]
+timestamp: 2026-10-05T00:00:00-07:00
+---
+
+# LLM Performance Diagnosis
+
+When a served endpoint is slower than wanted and someone names a suspect
+("PCIe is saturated", "the gather is the bound"), throughput arithmetic cannot
+settle it — it is usually circular (it assumes the suspect consumes the step).
+Get data in this order; the profiler window alone has retired multi-day
+optimization plans.
+
+## 1. Is decode context-bound at all? (the flat-depth probe)
+
+Sweep decode across context depth. **Flat** ⇒ attention over history is not
+the cost; the problem is per-token expert movement and KV/attention tuning is
+ruled out entirely. A genuinely context-bound server shows a clear downward
+slope. Record cold vs warm separately — a small gap means the cache is
+working. Measured on the fp8 expert-offload shape: 12.11 tok/s FLAT from 304
+to 16,095 tokens.
+
+## 2. Duty-cycle arithmetic (the "GPU not fully utilized" answer)
+
+Symptom: smooth ~25–35 % utilization with 100 % peaks on an expert-offload
+MoE. This is the EXPECTED signature, not a fault:
+
+```
+per decode step: compute burst ~100 ms (100 %) → expert-gather wait ~400 ms (~0 %)
+resident fraction = arena slots / total experts   (e.g. 6574/24576 ≈ 27 %)
+PCIe bytes/token  = miss_rate × lookups × slab_bytes  (0.73 × 480 × 2.344 MiB ≈ 0.84 GB)
+predicted duty    ≈ compute_ms / (compute_ms + wait_ms)
+```
+
+Validate the terms against the deployed engine's boot plan
+([[RDNA4 MoE Expert Offload VRAM Budget]]). If predicted ≈ observed, the low
+average is structural. Note small-transfer reality: bulk bandwidth (91 % of
+ceiling at 1 GiB) overstates what 2–3 MiB slabs get (~72 % plus ~20 %
+per-transfer latency). Before blaming the link, measure it
+([[PCIe Link and Atomics]]) and compare the engine's effective gather rate to
+the measured slab-size bandwidth — a 40–50 % gap is the software read path,
+not the wire. `rocm-smi --showuse` reads ~0 % during the whole multi-minute
+weight load: show VRAM (`--showmeminfo vram`) as the load-progress signal.
+
+## 3. Kernel-time attribution (the profiler window)
+
+Run BEFORE building any replacement kernel.
+
+1. Stop + mask production; boot the shape under test with ONLY the profiler
+   config added. vLLM argv:
+   `--profiler-config '{"profiler":"torch","torch_profiler_dir":"/prof","torch_profiler_with_stack":false,"torch_profiler_record_shapes":false,"torch_profiler_with_memory":false,"torch_profiler_dump_cuda_time_total":false}'`
+   — `with_stack=false` is load-bearing (with stacks the stop hangs and the
+   trace buffer grows ~40 GB → kernel OOM-kill of the engine core, anon-rss
+   ~95 GB; the worker-side stop can hang 5 min on ROCm/kineto). Bind-mount
+   `/prof`. If a shell substitutes values into that JSON, verify the escapes
+   actually landed.
+2. Window it: warm request → `POST /start_profile` → N generations of ≥128
+   tokens (~15 s of traffic) → `POST /stop_profile` (client may time out at
+   60 s; the trace still lands). `telemetry_report` has ONE caller: the
+   gpu_worker profiler-stop path. Overhead is small enough to keep decode
+   comparable (9.6 vs 9.9–10.0 tok/s measured).
+3. **Parse ON THE SERVER** (traces are 100–150 MB gzipped, GB expanded).
+   Rank `ph=="X"` events with `cat in {kernel, gpu_memcpy, gpu_memset}` by
+   summed `dur` (µs): report % of device time, launch count AND µs/call
+   (22 µs × 708k launches is a different problem than 4 ms × 4k), plus the
+   first-to-last event span (fraction of wall the device was busy). Aggregate
+   by FULL mangled name (template args are the tuning signal) and by stripped
+   symbol. **Trap: `gpu_user_annotation` events masquerade as GPU time** (a
+   naive sum reported 63 % in one `execute_context_0(...)` entry; spans
+   overlap and can exceed the wall clock).
+4. Divide by the window's token count → ms/token; if kernel time ≈ step
+   time, the table IS the optimization plan.
+
+Worked example (GLM-5.3-Flash, 2×GPU, offload, decode 9.9 tok/s):
+
+```
+34.8 %  708,500 launches  vectorized_elementwise_kernel   <- the actual bound
+10.0 %   42,000           MoE W4A16 GEMM
+ 8.9 %   83,160           router/top-k (sort-based)
+ 6.5 %  133,920           hyper-connection pre/post kernels
+ 1.05 %   20,664          <the long-suspected expert gather>
+```
+
+No amount of link arithmetic would have produced those two conclusions: the
+suspect was 1 %, and the mass was pointwise glue from an uncompiled trunk —
+whose own boot log names the cause (`FULL_DECODE_ONLY` override). It also
+predicted a negative result in advance (spec decode cannot help a shape whose
+per-token cost scales with the token count) which matched an unexplained flat
+MTP measurement.
+
+## 4. Telemetry without benchmarking (live hosts)
+
+When benching is forbidden (production serving), mine the runtime's own
+per-request telemetry — it measures the real workload and costs nothing.
+
+- llama.cpp (root unit, `-lv 3`): `sudo journalctl -u <unit> --since "-3h"
+  --no-pager` → `slot print_timing` lines (prompt eval, eval, total, `graphs
+  reused`, `draft acceptance … mean len`, `n_gen = N, tg = X t/s, tg_3s = …`).
+- halogen (rootless quadlet): `journalctl --user -u <unit>` (**never sudo** —
+  per-user storage) → `serve_api: mtp 28 tok in 0.66s = 42.45 t/s | 16
+  rounds, commit 1.75/round | prompt 58, prefill 0.92s | detok 24us/tok`.
+- Read the envelope, never one sample: short-prompt `prompt eval` is
+  overhead-dominated (57 tok @ 168 t/s) — quote the longest available (1106
+  tok @ 1013 t/s); decode and acceptance are content-dependent (0.46–0.76
+  measured on one unchanged build); `graphs reused > 0` / `selected slot by
+  LCP similarity` prove prefix-cache reuse.
+- **Attribute PCIe without touching the model:** background `nvidia-smi dmon
+  -s t -c 16 -o T`, ONE ordinary `max_tokens=200` request, `wait` at the end
+  (without it ssh closes and the client dies — the journal shows `W srv stop:
+  cancel task` and NO print_timing while curl still reports plausible
+  status). Measured: peak ~230 MB/s against a ~7,900 MB/s link = ~3 %, 0 at
+  idle — the sampled-logits stream, not the transport, is decode traffic on
+  that shape.
+- One `max_tokens=200` request is production traffic and fine; anything that
+  pins the card is a bench → stop-and-mask first. Check WHICH instance you
+  are reading (`systemctl status | head -12`; task ids jump across
+  restarts). The engine's own ledger (`KV self size`, `n_slots`,
+  `n_ctx_slot`) outranks your model of it.
+
+## 5. The fork's data path (before blaming the link)
+
+Offload engines pick between DMA copy-in and read-through UVA dereference
+inside the GEMM. Grep the installed package for the LRU/offload module and
+read the BRANCH CONDITION, not just the env names. A token-count threshold
+decides the path, and it interacts with spec decoding: an MTP step is k+1
+tokens, so the step can silently cross the threshold onto the slow path as
+draft depth grows. Check the threshold's premise: a default tuned at
+12.4 MiB/expert does not transfer to 2.344 MiB/expert, where copy-in can be
+several times faster. TunableOp ghost-skip: `CLAV_TUNABLEOP_SWEEP=1` alone
+does NOTHING under expert offload (`CLAV_TUNABLEOP_WITH_OFFLOAD=1` required);
+check for the skip line before concluding autotuning ran. Gather-geometry
+knobs read at import time → each arm is a full reload.
+
+## 6. RAM-bandwidth-bound MoE ceiling (before recommending a quant)
+
+```
+routed experts/token = top_k × n_layers
+bytes/token          = routed_params × bytes_per_param + shared_expert_bytes
+ceiling (tok/s)      = measured_bandwidth / bytes_per_token
+```
+
+Measure the HOST's real bandwidth, never assume DIMM ratings: 4-DIMM AM5
+boards downclock to DDR5-3600 regardless of 5600 sticks (two large DIMMs beat
+four small). Single-threaded reads under-report dual-channel ~4x — use the
+multi-threaded plateau of a checksumming reader (`scripts` pattern:
+`/tmp/bw.c`, 1/2/4/8/16 threads). Worked example (DeepSeek-V4.1-Flash):
+6×40 experts × 35.4M params = 8.5B params/token; at 3 bpw + 0.53 GB shared =
+**3.7 GB/token** → 42 GB/s ⇒ **11.3 tok/s ceiling**. Sanity anchors: H100
+NVL + CPU offload 226 GB ⇒ 5.8–9.0 tok/s; 2×3090 + 768 GB DDR4 ⇒ 12.2.
+CPU compute rarely binds below 4 bpw (~5.3 flops/byte ⇒ ~223 GFLOPS at 42
+GB/s); PCIe expert STREAMING is worse than CPU compute (Gen5 x8 ~22.7 GB/s
+effective < half of RAM); prefill reads the union of a batch's experts
+(prefill ≈ bandwidth / total_expert_bytes × ubatch). A quant that fits only
+by ignoring the Engram table does not fit (Engram = sparse n-gram lookup,
+NVMe-tolerant; the routed experts are not). An iGPU GTT aperture is an ALIAS
+of the same RAM, not extra memory.
+
+## 7. IBS model-access attribution (AMD CPUs)
+
+Real CPU access patterns for mmap'd weights, live, no stop (established on
+frank-primary, 2026-09-26, ~2.7M attributed samples in 75 s). Reference:
+`scripts/frank-primary-expert-activity.py` (self-contained dump + sample +
+analyze; `--analyze-only`).
+
+1. GGUF header-only scan (ms for a 50 GiB part) → tensor extents. **Extents
+   from CONSECUTIVE offsets, never a type-size table** — a wrong per-element
+   size silently under-covers every tensor and dumps tail samples into a junk
+   bucket (first run lost 59 % of expert samples; a uniform gap/exps ratio
+   ~1.4 across parts is the tell). Data start = `(tell + 31) & ~31`.
+2. `/proc/<pid>/maps` VMAs (7-group regex; pgoff = group 4, path = group 7)
+   matched against part files; multi-VMA per part is normal.
+3. `sudo perf record -e ibs_op// -c 50000 --data -o out.perf -p <pid> --
+   sleep <dur>` WHILE a mixed workload runs (idle perf numbers mean nothing).
+   `sudo perf script -i out.perf -F addr` for addresses.
+4. Attribute: bisect VMAs, `foff = pgoff + (addr − vma_start)`, bisect tensor
+   starts; per-expert index = `int(lo / (span / N_EXPERTS))` (expert-major
+   rows).
+5. Verify before believing histograms: gap ≈ 0, every tensor-bearing part
+   nonzero, and MoE blocks should be FLAT by construction (top-k fires every
+   block every token) — a non-flat block histogram means mis-attribution
+   first. Established law: expert routing is per-block flat, per-EXPERT
+   skewed.
+
+## 8. Prefix-cache-aware routing (multi-replica serving)
+
+Measure with a **sequential-turn client**, not only a batch benchmark — a
+batch of distinct prompts parallelises prefill across replicas and hides the
+problem. The two failure modes: round-robin loses every other cache hit
+(90k-token contexts: warm-turn TTFT 2676 ms round-robin vs 1096 ms pinned);
+pinning-everything loses a card (16 requests: 18.6 s spread vs 30.1 s
+pinned). **The rule that gets both: route by request shape** — a request
+carrying more than one message is a continuation (agent loop re-sending its
+context) → pin to the replica that served that prefix; anything else → spread
+with a COUNTER, not a prefix hash. Add failover: demote a replica on
+connection failure and retry on a live one BEFORE writing anything to the
+client (safe pre-stream; never retry after streaming started).
+
+The wall no policy fixes: **a running sequence and its cached prefix draw on
+the SAME pool.** Three concurrent 50–80k families against 122,636 tokens/card
+thrash under every placement and admission policy tried (counter placement,
+load-balanced placement, one-large-request-per-card admission, both chunk
+directions — the last COSTS KV in both directions). When the working set
+exceeds aggregate KV, say so: the levers are capacity, fewer concurrent
+large contexts, or a different platform. Traps: a lifetime hit-rate counter
+is not a per-request verdict (probe directly: same prompt twice to an idle
+replica — 22.3 s cold → 1.4 s warm); salt corpora; record sample sizes;
+`vllm bench serve` needs `--tokenizer <local path>`; verify metric names
+against the live endpoint before dashboards. Test shape: drive the routing
+function directly with synthetic bodies (stdlib-only policy module);
+conversation identity is the FIRST N KiB of prompt — a shorter tested prefix
+hashes differently per turn and pinning silently never happens.
+
+## Related
+
+- [[LLM Bench Windows and Measurement]] — the window discipline and usage-counter law
+- [[PCIe Link and Atomics]] — link probing and the streaming-vs-compute ceiling
+- [[RDNA4 MoE Expert Offload VRAM Budget]] — the plan behind duty-cycle terms
+- [[vLLM Porting and Compile Triage]] — the compile fix for the pointwise-glue bound

serverAppConfigs-docs/LLM/Llama Cpp Serving Sizing.md

@@ -80,7 +80,7 @@ co-tenants.
 - **Traps:** mmap'd PLE is reclaimable page cache; `-ot` makes it pinned —
   "move layers to the card" converts reclaimable cache into device memory and
   frees nothing for co-tenants. A served model's footprint ≠ weight size
-  (frank-small: 22.6 of 24.1 GiB for an 11.29 GiB model). Do the capacity
+  (frank-secondary: 22.6 of 24.1 GiB for an 11.29 GiB model). Do the capacity
   subtraction before designing: 12 attn layers × 1.62 GiB + 200k f16 KV =
   28.6 GiB > 24.1 GiB usable → KV-in-VRAM on that pair only reaches ~64k.
 - **Context lengths: derive, never remember.** `llama-bench` computes
@@ -157,7 +157,7 @@ in the timings. Context cap for full-model residency on that shape stayed
 ## Related
 
 - [[GGUF Compatibility]] — what a payload is before it is priced
-- [[frank-small-llama-mimo-q6k-2x200704]] — the current two-boot priced shape
+- [[frank-secondary-llama-mimo-q6k-2x200704]] — the current two-boot priced shape
 - [[frank-medium-llamacpp-dflash2]] — where the 68 KiB/token datum was measured
 - [[DFlash DSpark Speculative Decoding]] — the spec A/B protocol on llama.cpp
 - [[Dual-R9700-Cooperation]] / [[x4-pair-experiment]] — the multi-device side

serverAppConfigs-docs/LLM/Model Swap and Rollback.md

@@ -0,0 +1,281 @@
+---
+type: reference
+title: Model Swap and Rollback
+description: The full model-swap discipline for the LLM fleet — fit preflight, fetch and payload verification, runtime build/flag checks, git-only config, live validation, systemd handoff, close-out, the three-pin revert law, and research-verification discipline.
+tags: [models, swap, gguf, huggingface, rollback, deployment, verification]
+timestamp: 2026-10-05T00:00:00-07:00
+---
+
+# Model Swap and Rollback
+
+Applies to every endpoint swap on the fleet (frank-secondary / frank-large /
+frank-primary). Repo law: all config flows through git; the live models are
+host state under `/home/dank/models/<name>/`.
+
+## 1. Fit preflight — never deploy a swap without confirming it FITS
+
+1. Get the checkpoint's on-disk/weight size AND precision from the HF card
+   (`Precision: BF16` = 2 B/param; FP8/Q8 ≈ 1 B; Q4 ≈ 0.5 B). 35B params:
+   BF16 ≈ 66–70 GB, FP8/Q8 ≈ 33–35 GB, Q4 ≈ 17–20 GB.
+2. Compare against the HOST'S usable RAM (not just VRAM) for runtimes that
+   stream weights from RAM (host-RAM expert offload, llama.cpp CPU offload,
+   layer splits). Budget weights + KV pool + OS + page cache; **if weights
+   alone exceed RAM, ABORT** — pool tuning shrinks KV, it does not shrink
+   weights.
+3. **The failure mode is a self-sustaining thrash, not a crash:** a large
+   swap makes the kernel PAGE instead of OOM-killing → sshd/user space
+   starved indefinitely, no recovery window opens by itself; systemd
+   `Restart=always` turns it into a crash-reload-rethrash loop; `enabled` +
+   `WantedBy=multi-user.target` re-thrashes on every reboot. A power cycle
+   does NOT fix it.
+
+Recovery (may need console / SysRq-to-GRUB):
+
+```bash
+sudo systemctl mask --now <svc>                       # break the reboot trap FIRST
+cd <checkout> && git fetch origin && git reset --hard origin/main
+grep MODEL_DIR <env>                                  # confirm the fixed target
+sudo systemctl unmask <svc> && sudo systemctl restart <svc>
+curl -s <health>                                      # then a REAL completion
+```
+
+Deploy caveat: "healthy after Ns" = systemd ACTIVE, not model LOADED —
+confirm with a real `/v1/chat/completions`. RTX 3090 (Ampere) has no
+FP8/FP4 tensor cores: FP8 weights run via upcast but must still fit; choose
+quant by size fit, then accept the compute path the card supports. Before
+choosing a smaller quant, state WHICH of the two you are buying — headroom,
+or a design change (a smaller quant does not change what the runtime can
+express; per-tensor cross-device placement stays refused under
+`--override-tensor`, and a layer split moves only contiguous layer ranges).
+
+## 2. Ground truth first
+
+- Read the HF model card (runtime pin, hard constraints, recommended args)
+  + the repo trio `scripts/<svc>.env`, `scripts/serve-<svc>.sh`,
+  `scripts/<svc>.service` + the runbook under `serverAppConfigs-docs/LLM/`.
+- Check the host before downloading anything: `ls /home/dank/models/`,
+  `df -h /home`, `du -sh` — bench-era or prior-deployment files often already
+  exist.
+
+## 3. Fetch and verify (HF payloads)
+
+**Inventory by QUANT NAME, never by extension** — large quants ship as
+`.partNofM` fragments (Q4_K_M = 37.00+37.00+36.97 GiB), so an
+`endswith(".gguf")` filter makes Q4_K_S look nonexistent. Check both
+mradermacher repos (`…-i1-GGUF` imatrix, `…-GGUF` plain) and the model's own
+repo; the tree API's `blobs=true` gives `lfs.oid` per file. Gated mirrors
+return 403 even with a valid token until the account accepts the gate —
+prefer the ungated copy.
+
+```bash
+# token out of band: mode-600 curl config; a token passed as -H is world-readable
+printf 'header = "Authorization: Bearer %s"\n' "$(cat ~/.cache/huggingface/token)" > /tmp/hf.curlrc
+chmod 600 /tmp/hf.curlrc
+curl -fL -C - --retry 10 --retry-delay 5 --retry-max-time 21600 -K /tmp/hf.curlrc -o "$DEST/$name" "$BASE/$name"
+```
+
+`-f` is MANDATORY: `curl -o FILE URL` **exits 0 on HTTP 401/403** and writes
+the error body as the artifact (identical 153-byte shards, cheerful DONE,
+zero bytes of model). Never `pgrep -a` a running fetch (token is in
+`/proc/<pid>/cmdline`); watch progress by size:
+`while [ "$(stat -c%s … || echo 0)" -lt "$EXPECTED" ]; do sleep 30; done`.
+
+**Verify, then assemble:** per part, byte count == Hub's declared size AND
+`sha256sum` == `lfs.oid`; concatenate in NUMERIC order; assembled total ==
+sum of parts (compute the sum with python, not by hand) and `head -c 4` ==
+`GGUF`. The assembled file has no published hash — the per-part oids + exact
+total are the chain. Keep the fragments when disk allows (only copies
+re-verifiable against the Hub); failure artifacts and the replaced payload
+go to `~/.AGENTTRASH/YYYYMMDD-topic`, never `rm` — upstreams DELIST quants
+and the local file may be the last copy.
+
+Fetch-script one-liners: dedicated scripts (`scripts/download-*.sh`) already
+do `-f` + byte-count + LFS sha256 + `--verify-only`; ad-hoc fetches tend to
+skip all three. HF cache snapshots SYMLINK every file into `blobs/` and
+those links dangle inside containers — serve scripts and the deployer refuse
+symlinks and missing tokenizer files for exactly this reason.
+
+## 4. Runtime build (when the card pins a fork/build)
+
+- Clone + build with the card's exact cmake line, plus
+  `-DLLAMA_BUILD_WEBUI=OFF` — WebUI asset provisioning (npm / HF llama-ui
+  bucket) aborts llama.cpp link at 98 % on these hosts; the API server is
+  unaffected.
+- Verify every flag the serve script passes exists in the new binary BEFORE
+  touching the service: `llama-server --help | grep -q -- "<flag>"` for each;
+  list MISSes. Read the GGUF's tensor histogram first if the payload uses a
+  custom type — the FORMAT picks the runtime ([[GGUF Compatibility]]).
+
+## 5. Config via git only
+
+Edit `scripts/<svc>.env` (paths, LLAMA_BIN, MODEL_FILE, MMPROJ_PATH) + serve
+script header/defaults locally; `bash -n`; `./tests/run.sh` must pass;
+commit; push; `git pull --ff-only` on the target. Never edit configs on the
+remote. Renaming a model dir (`mv`) is host state and allowed. The env-file
+hook pattern (`FRANK_*_ENV_FILE`) is what makes bench windows and staged
+arms possible — keep it intact.
+
+## 6. Live validation
+
+- No scratch ports — one GPU cannot host two big models. Ensure the old unit
+  is truly gone first: `systemctl is-active` can hang in `stop-sigterm`
+  (llama.cpp never preempts an in-flight generation); poll until the PID
+  exits.
+- Manual serve for smoke: `setsid nohup bash scripts/serve-<svc>.sh >
+  /tmp/x.log 2>&1 &` — plain `ssh 'nohup … &'` dies with its ssh session.
+- Verify: health 200; a REAL `/v1/chat/completions`; reasoning flows
+  (`reasoning_content` non-empty); if the endpoint takes images, a
+  solid-color probe through an `image_url` content part (legacy top-level
+  `images` is silently ignored; probe with ≥256 px).
+- vLLM specifics: first request after boot can take ~18 s (deferred Triton
+  loads) — warm and discard; `--reasoning-parser` / tool parsers are part of
+  the contract ([[vLLM Boot Forensics and Config Interrogation]]).
+
+## 7. systemd handoff — THE TRAP
+
+`./scripts/deploy.sh <host> --service <svc>` gates health, but a leftover
+manual server holding the port makes the health check pass while the UNIT
+crash-loops on bind. Always confirm ownership:
+`systemctl show <svc> -p MainPID` equals the single `pgrep -x llama-server`
+(or the container runtime PID); only then trust "healthy". If a real
+long-context task is queued, do the handoff anyway — client aborts are
+recoverable, drift is not.
+
+## 8. Close-out (mandatory)
+
+1. Docs: active runbook updated (build recipe, profile, validation numbers
+   incl. real-workload prefill/decode); superseded runbook gets a RETIRED
+   banner and stays as the rollback record; `LLM/index.md` mermaid + active
+   list; AGENTS.md fleet table + runtime section + history chain; work-log
+   entry in `serverAppConfigs-docs/log.md`.
+2. `~/.omp/agent/models.yml`: `contextWindow` must equal the REAL per-slot
+   context — llama.cpp logs `n_ctx_slot` per prompt (`--parallel N` divides
+   the pool; e.g. 262144 native / 2 slots = 131072 per slot); vLLM reports
+   `max-model-len` and its concurrency line. Overselling causes silent
+   client-side overflow or a discovered-wall 400. Also `maxTokens` vs the
+   engine's cap, `input` capabilities (text/image), and
+   `reasoningContentField` per runtime (llama.cpp/SGLang emit
+   `reasoning_content`, vLLM emits `reasoning`). Fix every `modelRoles` entry
+   that named the old endpoint.
+3. **Rollback = one env commit** (paths + LLAMA_BIN + flags), prior binary +
+   payload kept on disk, rollback numbers recorded in the runbook.
+
+## 9. Revert-pin an ad hoc experiment (the three pins)
+
+A campaign is "done" only when all three exist — without them a
+`podman pull`, an image update, or a lost scratch script destroys the way
+back:
+
+1. **Git tag (annotated, pushed):**
+   `git tag -a <exp>-tuned-YYYYMMDD -m "<tuned env/flags + measured headline +
+   runbook path + revert command>"` — a future agent must reconstruct the
+   campaign from the tag message alone.
+2. **Image digest in the serve script:** floating tags move. Capture
+   `podman inspect <ctr> --format '{{.ImageName}} {{.Image}}'` + `podman
+   images --digests`, patch `IMAGE=` to `repo@sha256:<repodigest>` with a
+   comment (image ID, date, old ref), verify `podman image exists
+   repo@sha256:<digest>` and `bash -n`. Digest refs survive upstream pushes;
+   `podman pull repo@sha256:<digest>` re-fetches exact bytes. One host can
+   hold MULTIPLE images of one repo — pin by the digest of the RUNNING
+   container, never by tag name. Never edit the pinned line back to a
+   floating tag; new experiments branch from the digest.
+3. **Local alias:** `podman tag <id> <exp>-runtime:tuned-YYYYMMDD`.
+
+Record scratch-script paths + full flag surface in the runbook (hook env
+vars in CAPS); expired images/scripts go to `.AGENTTRASH`, never `rm`.
+Pure measurement sessions that changed no state need only pin 1.
+
+## 10. Research verification discipline (for "can model X run here?")
+
+- Scouted research summaries have shipped fabricated citations on this fleet
+  (invented HF repos, plausible-but-wrong PR numbers, stale engine versions).
+  NEVER copy a scout's citations into a deliverable — re-verify each
+  load-bearing URL directly (GitHub issues/PRs/repos, HF pages) before it
+  enters anything.
+- The in-repo runbooks are FIRST-HAND evidence and outrank any web claim
+  about our own hosts.
+- Verified anchors (2026-09-27): DeepSeek V4.1-Flash (552B/8B-16B active,
+  CED arch, 1M ctx, MIT; preview was 284B/13B with native FP4+FP8 — early
+  GGUF conversions garbage); Lucebox = the proven gfx1201 ROCm 7.2 path
+  (heterogeneous R9700+Strix-Halo V4.1: 86 tok/s decode, 788 t/s prefill@2K).
+  GLM-5.3-Flash (320B/18B, KDA+DSA+mHC, 1M ctx — ALWAYS bind `-c`, the
+  default hard-hung a 128 GB host; native FP8 328.3 GB; `-fa off` MANDATORY,
+  flash-attn breaks MLA correctness; llama.cpp arch `glm5next` lived in
+  unslothai branch → ggml PR #27754, MTP landed d07e71ede 2026-08-30; quant
+  ladder UD-IQ1_S 93.1 GB → UD-Q4_K_XL 199.7 GB, GSQ-RCO 3.5-bit −1.40 pp
+  MMLU-Pro vs Q8_0).
+- GGUF parsing gotcha: expert tensors are
+  `blk.N.ffn_{gate,up,down}_exps.weight` — 4 dot-parts INCLUDING `.weight`;
+  a `len(parts) == 3` filter matches zero layers with no parse error. Filter
+  on `parts[0]=="blk" and parts[3]=="weight" and "exps" in parts[2]`.
+
+## 11. Adding a second LLM endpoint to a host
+
+When a host must serve two models at once (an iGPU runtime plus a second
+card), or one endpoint is moving and a new one takes the freed hardware.
+
+**0. Decide the shape before touching hardware.** One provider per
+HOST:PORT in `~/.omp/agent/models.yml` (on the box where the harness runs)
+— a second endpoint is a new provider block, not a second model in the
+existing one. Read the budget as an inequality: for unified memory the
+engine's own fit check is the arbiter (`budget = host − pinned weights −
+reserve`; `need = KV + arena + vision`); for a discrete card, weight the KV
+by counting only full-attention layers — and remember `--parallel N`
+DIVIDES `--ctx-size`. A co-tenant is a budget line, not a footnote: free
+RAM by changing the POOL or the ARENA (the reserve is a fit-check term,
+never an allocation) and drop optional charges (a vision tower is ~2 GiB)
+rather than hoping.
+
+**1. Repo first, and retire in the same commit.** New service files
+(`scripts/<svc>.env`, `.service`, `serve-<svc>.sh` with hard guards for
+missing artifacts); planner entries (`SERVICE_ORDER` + the target's service
+set + path mappings, inventories in `deploy.sh`/`deploy-remote.sh`, the
+dispatch case); retire the obsoleted pieces into `.AGENTTRASH/` with a
+suffix — and check what pointed at them: bench harnesses, a chat template
+only the old runtime read, and `scripts/test-changed.sh` mappings are the
+usual victims (a mapping to a retired test file fails the pre-commit
+wrapper silently for weeks). One bats file per service asserting
+INVARIANTS (`pool == 2 × ctx`, KV bytes ≤ card budget, port separation),
+not constants; a negative grep must strip comments first
+(`grep -vE '^\s*#' file | grep -q …`) or it matches the prose explaining
+the very thing you removed.
+
+**2. The port-bind race (rootless podman).** pasta binds the published port
+at container CREATE; a just-stopped predecessor still holds it. Order the
+handler: retire the old manager's unit → stop THIS service's own unit
+(no-op on first cutover) → wait for every socket on the port to clear
+([[Container Host Reachability]]) → install → start. Without the second
+step a redeploy deadlocks against its own gate.
+
+**3. Wire the surfaces, refuse impossible signals.** Point the harness
+small/fast + image roles at the new endpoint; move the vision role WITH the
+vision tower (one owner per capability). `shared/homepage/services.json`: a
+card per endpoint, probed from the monitor's own host. Prometheus: only add
+a job if the runtime actually exposes Prometheus text (halogen publishes
+`/health`, `/cache`, `/v1/*` and nothing else — the job was REMOVED with the
+reason written down rather than left as a permanent red).
+
+**4. Cutover, then verify by driving it.** `deploy.sh <target> --service
+<large> --service <small>` (order matters); per endpoint: `/health`,
+`/v1/models` reports the alias, a chat completion returns content AND
+`reasoning_content`, a `tools` request returns `tool_calls`, an image
+request is answered (or refused with a named reason), the runtime's own
+speculation counters move, and the unit is `enabled`/`generated` with linger
+on. Read the engine's boot ledger for a silent clamp. Finally spend one
+real request through the harness per changed role — routing labels lie; the
+servers' own journals do not.
+
+Traps that look like model faults: `500 tools param requires --jinja flag`
+(runtime lacks the Jinja path — pin it); `500 Jinja Exception: Unexpected
+reasoning effort <level>` (fold via `compat.reasoningEffortMap`); the same
+request works by hand but a subagent dies (subagents run at a different
+thinking level — check both); the container dies instantly although the
+predecessor "stopped" (the port race above).
+
+## Related
+
+- [[GGUF Compatibility]] — which runtime can load a payload at all
+- [[Llama Cpp Serving Sizing]] — pricing the shape once it loads
+- [[RDNA4 MoE Expert Offload VRAM Budget]] — vLLM-side fit arithmetic
+- [[LLM Bench Windows and Measurement]] — measuring the new deployment
+- [[dankserver-qwen38-fp8-expert-offload-tuning]] — the current endpoint

serverAppConfigs-docs/LLM/Pipeline Parallelism/Qwen3.8 PP Deployment.md

@@ -48,10 +48,10 @@ state, verified against the live journal and output smoke tests
 
 The PP pair is the only inference service enabled for boot. Both units
 (`frank-large` on Dankclaw, `frank-pp-worker` on Dankserver) are
-enabled. The NPU unit (`frank-small` on Dankclaw) and all retired
+enabled. The NPU unit (`frank-secondary` on Dankclaw) and all retired
 units are disabled. Restart order: worker first, then head. The Dankserver
 helper `frank-restart-small.sh` is obsolete (it restarts the removed
-`frank-small` unit); restart the worker with
+`frank-secondary` unit); restart the worker with
 `sudo systemctl restart frank-pp-worker`. The Dankclaw helper
 `frank-restart-large.sh` (sync + cache drop + restart) is correct for the
 head. `deploy.sh dankserver` is broken until retargeted. See

serverAppConfigs-docs/LLM/Pipeline Parallelism/index.md

@@ -65,6 +65,6 @@ graph LR
 ## See Also
 
 - [[DFlash DSpark Speculative Decoding]]
-- [[frank-small-benchmark]]
+- [[frank-secondary-benchmark]]
 - [[Dankserver System Specs]]
 - [[Dankclaw System Specs]]

serverAppConfigs-docs/LLM/Quantization Production.md

@@ -0,0 +1,260 @@
+# Quantization Production — making our own quants, per card and runtime
+
+**Research page (2026-10-08).** Question: how do WE produce quantizations
+for the RTX 3090, the R9700 (gfx1201), and Strix Halo (gfx1151); what is
+native to each runtime; and the state of the named prior-art families
+(REAP, Heretic, abliteration, HauhauCS K_P "Perfect"). Marked facts:
+**[repo-verified]** = evidenced by a file in this repo; **[verified]** =
+checked first-hand against the cited primary source today; the rest are
+survey claims to re-verify before acting.
+
+## Runtime-native quant matrix
+
+| Card | Runtime | Container | Native weight encodings | Production tooling we own |
+|---|---|---|---|---|
+| RTX 3090 (sm_86) | llama.cpp CUDA | GGUF | K-quants (Q4_K_M…Q8_0), I-quants (IQ1_S…IQ4_XS), ML, f16/bf16; any tensor type per-tensor | `llama-quantize` + `ggml-imatrix` (built by `scripts/build-llama-rdna4.sh` targets) **[repo-verified]** |
+| R9700 (gfx1201) | radiance (StillDeadcode) | `.rad` | per-tensor encodings from a recipe: GPTQ w4 (`w4nl64a8h`), bf16, int8, fp8_e4m3, u2, MXFP4/MXFP6 (vLLM-radiance era) | full pipeline: `rad-convert` + calibration tap + `derive-protected-experts.py` + `kld-gate-swift15.sh` **[repo-verified]** — CPU host-backend, the card is irrelevant to production |
+| R9700 (gfx1201) | vLLM (tcclaviger/R4D) | safetensors | native block-FP8, MXFP4/DSFP4, GPTQ/AWQ, compressed-tensors | GPTQModel/AutoRound are hardware-agnostic torch (CPU-scale); AMD decode kernels are the gate **[survey]** |
+| Strix Halo (gfx1151) | halogen-flash-server (peonist, **closed**) | `.hgn` | vendor-defined (w4b era, v2 checkpoint+ngram+vision files) | **NONE public** — no `.hgn` quantizer exists; every `.hgn` on the fleet was made by peonist/Quat3rnion **[repo-verified gap]** |
+| Strix Halo (gfx1151) | llama.cpp / gufo / Kairic / ROCmFPX forks | GGUF | K/I quants (gufo UD-* = imatrix mixed profiles), custom types: `Q4_0_ROCMFP4_STRIX`, `Q8_0_ROCMFPX` (type 103), Kairic IU4 | we built the RUNTIMES (`scripts/build-kairic-edge-gfx1151.sh`, ROCmFPX pin `36e9acd`), never the custom-type quantizers — those files came from agentionai/kingjones **[repo-verified]** |
+
+- **The lock is asymmetric: llama.cpp on the 3090 is a CHOICE.** radiance
+  demands `.rad`, halogen demands `.hgn` (the v2 checkpoint loads in no
+  other engine); sm_86 decodes GGUF, GPTQ/AWQ-Marlin, INT8, weight-only
+  FP8, and EXL3 natively — see the 3090 menu under Per-card playbooks.
+
+Laws that steer everything here:
+- **Production is hardware-agnostic; decode is not.** GPTQ/imatrix/ablation
+  surgery runs in torch/CPU anywhere. The card only picks which ENCODED
+  FORM it can DECODE — hence the one-table-one-encoding law, the GGUF
+  custom-type histogram law ([[GGUF Compatibility]]), and the fact that
+  our w4 convert ran 5 h 50 m on the host backend while the GPUs served
+  nobody (`scripts/convert-swift15-w4dn8.sh`) **[repo-verified]**.
+- **A custom tensor type locks the file to its fork** (ROCmFPX type 103
+  precedent). K_P quants deliberately avoid that trap (below).
+
+## The named families
+
+### REAP — Router-weighted Expert Activation Pruning (Cerebras, 2025)
+- arXiv 2510.13999; `github.com/CerebrasResearch/reap` **[verified]**.
+- One-shot MoE surgery: score every expert by router gate-values ×
+  activation energy, prune the low-utility tail, renumber+export a
+  SMALLER MoE checkpoint (safetensors; GGUF by conversion). Published:
+  GLM-4.5-Air −25 % (82B→~A12B class), GLM-4.7-REAP-268B-A32B-FP8;
+  near-lossless at 25–50 % compression on generative benchmarks, and
+  pruning BEATS expert-merging there **[verified from paper/README]**.
+- Fleet relevance: it is the **inverse of our protected-experts law**.
+  `derive-protected-experts.py` ranks experts by calibration-gram energy
+  share and keeps the hottest 24/layer at bf16; REAP's saliency is the
+  same statistic and deletes the coldest. REAP the Flash-Next MoE (all
+  49,152 experts w4, 512/layer) by 25–50 % → resident set ~58–87 GiB →
+  the expert pool fits with room for KV/cache, killing the disk tier.
+  A REAPed checkpoint re-enters our normal pipeline: stage → calibrate →
+  rad-convert → KLD gate. Hardware-agnostic; runs on the host.
+
+### Heretic (`p-e-w/heretic`) — automated abliteration + layer pruning
+- `github.com/p-e-w/heretic`, AGPLv3 **[verified]**. NOT a quantizer.
+  Fully-automatic censorship removal: directional ablation (refusal
+  direction from PCA of harmful/harmless residuals), per-layer ablation
+  weights, optional LAYER DROPPING (model shrinks), auto-tuned by
+  co-minimizing refusals and KL divergence from the original. Runs on
+  our 3090 (torch, ~24–48 GB working set for 9–27B classes; long batch
+  probes) **[verified README; runtime cost survey]**.
+- Output is a normal HF checkpoint (fp16/bf16) → feeds GGUF, safetensors
+  (vLLM), and `.rad` conversion. Order law: **surgery → calibrate →
+  quantize**; calibrate GPTQ/imatrix on the SURGICAL output, and keep
+  refusal-shaped tokens out of the calibration corpus.
+- Fleet note: our abliterated bases (Quat3rnion v2, HauhauCS
+  Aggressive/Balanced, ukisai Swift1.5) are this family's output; the
+  Quat3rnion repo lore "the abliteration workflow appends, compaction
+  drops dead padding" is exactly Heretic-style layer-editing leaving
+  dead blocks (89.8 → 66.7 GiB r2) **[repo-verified lore]**.
+
+### Ablation / abliteration (directional ablation)
+- Arditi et al. 2024, arXiv 2406.11717: refusal is mediated by a single
+  direction; orthogonalizing weights against it removes refusals with a
+  rank-1 edit. Open implementations: `p-e-w/heretic` (automated, above),
+  Artio/abliteration-toolkits, `EleutherAI/repeng`-style rep-overlap
+  variants **[verified (tool existence: heretic; rest survey)]**.
+- Hardware-agnostic fp-editing. On a MoE the edit touches attention/norm
+  weights, so it composes with any expert quantization.
+- Fleet status: frank-large runs an abliterated base **by Frank's choice**
+  (AGENTS: censored-base refusals were the accepted trade of the 27B era);
+  frank-archon's ukisai Swift1.5 is an abliterated finetune of Qwen3.8.
+
+### HauhauCS QX_K_P "Perfect" quants
+- Our frank-secondary publisher. Method, from the publisher's own cards
+  **[verified]**: (1) abliterate + finetune ("Aggressive"/"Balanced"),
+  (2) quantize with **imatrix calibration on the ABLITERATED weights**,
+  (3) K_P = a **per-model, per-tensor mixed-precision profile** —
+  "model-specific analysis to selectively preserve quality where it
+  matters most", e.g. Q8_K_P = a census of Q8_K + F32 + F16 tensors
+  (FriendliAI's safetensor census of a Q8_K_P shows exactly that
+  399/353/99 tensor split). Result: +1–2 quality rungs for +5–15 %
+  size. **It is a standard GGUF** — llama-quantize with per-tensor type
+  overrides (`--tensor-type`), imatrix on; no fork, no custom types
+  (LM Studio just fails to NAME the mixed file, shows "?").
+- Reproducible with stock tooling: `ggml-imatrix` on the target model +
+  `llama-quantize --tensor-type <pattern>=<type>` per the model's
+  sensitivity census. Both binaries already build from
+  `scripts/build-llama-rdna4.sh` **[repo-verified]**.
+- Our own ladder precedent: the fleet's UD-Q8_K_XL (unsloth "Dynamic" =
+  same family: imatrix + importance-tiered per-tensor types) and
+  agentionai FAST imatrix ROCmFP4-FAST (we ran the imatrix step
+  ourselves on the v2 joined-PLE file — the one self-made quant in repo
+  history) **[repo-verified]**.
+
+## Per-card playbooks
+
+### RTX 3090 — the unlocked card (frank-secondary runs llama.cpp BY CHOICE)
+The 3090 is the only fleet device with no format lock: radiance needs
+`.rad`, halogen needs `.hgn`, and the v2 checkpoint loads in NO other
+engine. sm_86 + CUDA is the industry's default target, so the runtime
+menu is open and every self-produced artifact format has a native
+decoder here. This makes the 3090 the natural **quant R&D bench**:
+produce on the host, DECODE AND SCORE on the 3090 against the fleet's
+own KLD/perplexity gates.
+
+Runtime menu on sm_86 (verified against each project's support matrix):
+
+| Runtime | Native quant on Ampere | Notes |
+|---|---|---|
+| llama.cpp CUDA (deployed) | K/I/ML quants, f16 | imatrix + `--tensor-type` K_P ladder |
+| vLLM CUDA | GPTQ INT4 + AWQ via **Marlin** (sm_80/86 native), INT8 W8A8 SmoothQuant, **FP8 as W8A16 weight-only via Marlin** (FP8 W8A8 compute is Ada/Hopper-only) | full OpenAI server, batching |
+| SGLang | GPTQ/AWQ (vLLM kernel lineage); Ampere FP8-via-Marlin tracked upstream | CUDA-first engine |
+| ExLlamaV3 | **EXL3** trellis (QTIP) 2–8 bpw; runs on Ampere incl. the 3090 | the quantizer (`convert.py`) runs on this same card |
+| TensorRT-LLM | INT4 AWQ/GPTQ (FP16 acts), INT8 SmoothQuant; FP8 and INT4-AWQ-with-FP8-acts need SM≥89, NVFP4 is Blackwell | engine-build workflow |
+
+Hardware ceilings that no software unlocks on this card: no FP8 matmul
+hardware (sm_86 < 89), no FP4 (Blackwell). Ampere's fast lanes are
+**INT4 (Marlin/AWQ/GPTQ), INT8 (SmoothQuant), weight-only FP8, f16**.
+
+Playbook A — GGUF ladder (status quo, llama.cpp kept):
+1. Surgery on host/3090: abliterate/prune (Heretic) — torch.
+2. `ggml-imatrix` (CUDA build) over a calibration corpus (our
+   `/home/dank/corpora` style set; ≥1M tokens; the fleet's KLD corpus
+   is a reasonable seed).
+3. `llama-quantize` F16 → target rung; K_P-style `--tensor-type`
+   overrides for attention/KV/first-layer upgrades; `--use-repack` for
+   IQ rungs.
+4. Validate: perplexity + KLD vs BF16 + the K-quant rank checks.
+
+Playbook B — safetensors ladder (the unlocked option): GPTQModel /
+AutoRound / AWQ run CPU-side, emit GPTQ/AWQ/FP8 checkpoints that vLLM
+or SGLang serves on the 3090 with Marlin kernels — same KLD-gate
+discipline as the radiance pipeline. A REAPed + abliterated Flash-Next
+cut to GPTQ-w4 for the 3090 is a bench experiment that needs no new
+hardware; EXL3 (`convert.py` on the 3090, exllama serve) is the
+quality-per-bit extreme of the same menu.
+Caveats before any swap: the serving contract (OpenAI server,
+`reasoning_content`, vision mmproj, compaction-hub window
+200,704 ×2 slots) must be met by the challenger runtime, and the
+24 GB card runs ONE server — benching follows the stop-and-mask law.
+
+#### Worked example — the frank-archon w4dn8 recipe ported to the 3090
+(audit 2026-10-08, target `Qwen3.6-35B-A3B`)
+**The methodology ports; the encoding must be translated.** llama.cpp
+decodes no GPTQ — the recipe re-expresses on GGUF-native terms:
+imatrix on our corpora (replaces the radiance grams; runs the 65 GiB
+BF16 on host CPU), IQ4-class experts, hot-expert protection as
+`--tensor-type "=f16"` patterns over the derive-the-hot-experts law
+(statistic: imatrix importances), Q6_K trunk, KLD/perplexity gate at
+the deployed slot shape. Production stays host-side; the 3090 is only
+the decode target.
+KV is cheap on this arch: 10 full layers × 2 KV heads × 256 =
+**20 KiB/token f16** (3.8 GiB per 200k slot); DeltaNet state is
+per-slot, tens of MiB. Size verdict (weights ≈ 32.2 B experts + 3 B
+trunk):
+- Archon-faithful (w4 experts + bf16 protected + i8 trunk) ≈ 22 GiB →
+  27 GiB with KV+buffers → **over 24; that shape is 2×3090-class.**
+- Fits 1×200,704 text-only: IQ4_XS experts (17.4) + Q6_K trunk + q8_0
+  KV (1.9; `GGML_CUDA_FA_ALL_QUANTS=ON` build) ≈ 21.5 ✓
+- Compaction-hub shape (2×200,704 + mmproj): experts at IQ3_M (14.4) ≈
+  24.0 — the exact Swift IQ3_XXS 2-slot edge, MTP precedent exists.
+- Literal GPTQ-w4 encoding decodes only via vLLM/SGLang Marlin (Playbook
+  B); compressed-tensors expresses the protected set per-module.
+Shelf check: HauhauCS publishes this model's K_P ladder (IQ2_M 10.86,
+IQ4_XS 17.44, Q4_K_M 19.71, Q4_K_P 21.82, Q5_K_P 26.1, Q6_K_P 28.5,
+Q8_K_P 40.6 GiB + mmproj 0.84) — Q4_K_P IS the protected-tensor blend.
+The self-made delta: calibration on OUR corpora, our hot-expert law,
+and a KLD gate scored at the serving shape.
+
+### R9700 — frank-primary (radiance)
+The full self-quant pipeline already exists end to end: fetch BF16 →
+record KLD reference (1.0.13) → capture calibration grams (1.2.0 tap,
+`RADIANCE_CALIB_DIR/TOKENS`) → derive protected experts (energy-share) →
+`convert-swift15-w4dn8.sh` (1.0.13, host backend) → KLD gate (MNBT 2048
+law, pool 40960, mean KL ≤ 0.03, top-1 ≥ 95 %). New model = new recipe
++ new `.so` arch plugin; every step is scripted. Prior art to fold in:
+REAP the expert set BEFORE step 3; ablate BEFORE calibration.
+Native encodings beyond w4: i8 trunk, fp8 HC/PLE/KV, bf16 protected,
+u2 draft — all recipe-expressible (see the w4dn8 runbook).
+
+#### Recipe portability: the w4dn8 shape onto other MoEs (2026-10-08 audit)
+The pin 1.0.13 container ships SIX arch plugins
+(`/opt/radiance/share/radiance/architectures/`): `qwen4exp_fp8`,
+**`qwen35moe_fp8`**, `qwen35_fp8`, `qwen35_bf16`, `llama_fp8`,
+`llama_dense_fp8` (census via `podman run --rm --entrypoint sh <image> -c`
+— the image has no `ls`/`strings`; grep the `.so` files directly, and
+the image lives in ROOTLESS storage as `docker.io/stilldeadcode/radiance`).
+`grep -c` receipts: `qwen35moe_fp8.so` carries the `RADIANCE_CALIB` tap
+(2) and gram export (2) plus MTP (4) and NO ngram (0) — i.e. the full
+calibration→GPTQ path exists for the qwen3.5-MoE family; GPTQ itself
+lives in `rad-convert`, not the plugin (0 hits in both `.so`).
+
+Worked example — `Qwen/Qwen3.6-35B-A3B` (config: `qwen3_5_moe_text`,
+40 layers, `full_attention_interval=4` DeltaNet hybrid, **256 experts
+top-8**, moe_intermediate 512, hidden 2048, MTP ×1, 262k ctx, vocab
+248320, multimodal wrapper with vision tower):
+- Expert set ~32 B params → w4 ≈ 16 GiB; trunk (attn/DeltaNet/embed/
+  lm_head ~3 B) at i8 ≈ 3 GiB ⇒ **~20 GiB total — fully VRAM-resident
+  on ONE R9700** (32 GiB card): no TP=2, no host pool, no disk tier,
+  no `--ngram-placement`, no 47.7 GiB PLE copy, no HC term. The recipe
+  loses three lines (PLE/HC/protected-count scales with 256 experts).
+- Convert cost scales with the file: Flash-Next's 335 GiB took 5 h 50 m
+  host-backend ⇒ ~70 GiB BF16 ≈ 1–1.5 h. KLD reference boot is trivial
+  (BF16 ~65 GiB in host pool beside the pool KV needs).
+- Unresolved before any real campaign: does `qwen35moe` decode the
+  `Qwen3_5MoeForConditionalGeneration` vision wrapper (assume text-only
+  until proven), and does its MTP cast to the fp8/u2 encodings like
+  qwen4exp. The MNBT-2048 disk-tier law is MOOT for a resident model.
+
+### Strix Halo — frank-large (halogen) and friends
+- `.hgn`: **closed production door.** No public quantizer exists
+  (verified absence across peonist-ai repos); the format's lock is the
+  engine's lock. We consume vendor artifacts; the repo-side tooling is
+  compaction/verification only.
+- Self-quant work on this card therefore runs through **GGUF runtimes**:
+  the same imatrix/llama-quantize flow as the 3090 (HIP/Vulkan decode of
+  K/I quants, gufo UD-* precedent, `--use-repack` off on ROCm), or the
+  custom-type forks (ROCmFP4 STRIX, Kairic IU4) whose PRODUCERS we have
+  never been — the quantizers for types 101/103 ship in those forks'
+  trees; building one is a fork build, not a flag.
+- Native-FP4 note: RDNA4/Strix get hardware FP4 matmuls (hipBLASLt
+  MXFP4/DSFP4), which is the engine-level path (radiance/vLLM), not a
+  GGUF decode.
+
+## Sources
+- REAP: arXiv 2510.13999, github.com/CerebrasResearch/reap,
+  cerebras/GLM-4.5-Air-REAP-82B-A12B-FP8, cerebras/GLM-4.7-REAP-268B-A32B-FP8.
+- Heretic: github.com/p-e-w/heretic (AGPLv3).
+- Abliteration: arXiv 2406.11717 (Arditi et al.).
+- K_P: HauhauCS model cards (e.g. `HauhauCS/Gemma-4-E4B-Uncensored-HauhauCS-Aggressive`
+  README "What are K_P quants?"), FriendliAI Q8_K_P tensor census,
+  fleet payload tree (BF16/Q4_K_M/Q6_K/Q8_0 + mmproj ladder in
+  `HauhauCS/Qwen3.5-9B-Uncensored-HauhauCS-Aggressive`).
+- EXL3: github.com/turboderp-org/exllamav3 (QTIP lineage; CUDA; Ampere
+  decode incl. 3090 documented in v1.0 performance write-ups).
+- Ampere capability edges: vLLM FP8 docs ("only Hopper and Ada …
+  W8A8; Ampere supported for W8A16 weight-only utilizing Marlin"),
+  vLLM Supported-Hardware matrix (Marlin GPTQ/AWQ on SM 8.0/8.6),
+  TensorRT-LLM Support Matrix (FP8 and INT4-AWQ-with-FP8-acts need
+  SM≥89; NVFP4 Blackwell), SGLang issue #12887 (Ampere FP8-via-Marlin).
+- Repo: `scripts/convert-swift15-w4dn8.sh`,
+  `scripts/recipes/qwen4exp-swift15-w4dn8-i8-hc8m.recipe`,
+  `scripts/kld-gate-swift15.sh`, `scripts/build-llama-rdna4.sh`,
+  `scripts/build-kairic-edge-gfx1151.sh`, [[agentionai-fast-imatrix]],
+  [[kingjones-strix-rocmfp4]], [[qwen3.8-strix-optimizations]],
+  [[frank-primary-radiance-swift15-w4dn8]],
+  [[frank-secondary-llama-qwen35-9b-q6k-2x200704]],
+  [[frank-large-flash-server-v2-swift15]], [[GGUF Compatibility]].

serverAppConfigs-docs/LLM/RDNA4 Container Serving.md

@@ -247,4 +247,4 @@ see [[dankserver-qwen38-fp8-expert-offload-tuning]] for why it was rejected);
   measured campaigns
 - [[PCIe Link and Atomics]] — why decode is gather-bound and what cannot fix it
 - [[RDNA4 MoE Expert Offload VRAM Budget]] — arena/plan arithmetic deep-dive
-- [[frank-large-benchmark]] and [[frank-small-benchmark]] — window discipline on live hosts
+- [[frank-large-benchmark]] and [[frank-secondary-benchmark]] — window discipline on live hosts

serverAppConfigs-docs/LLM/Running Qwen3.6 35B A3B FP8 on Dankserver.md

@@ -122,7 +122,7 @@ export HF_HOME=/mnt/entropy/huggingface
 
 ### Active Service Conflict
 
-The retired `frank-small` service (llama.cpp) used about 22 GB of the 24 GB VRAM. The current PP worker holds ~23.8 GiB. The `vllm-awq-qwen36` (Podman) and `vllm-qwen36` units are disabled. Before starting any research service, verify that the GPU is free:
+The retired `frank-secondary` service (llama.cpp) used about 22 GB of the 24 GB VRAM. The current PP worker holds ~23.8 GiB. The `vllm-awq-qwen36` (Podman) and `vllm-qwen36` units are disabled. Before starting any research service, verify that the GPU is free:
 
 ```
 nvidia-smi  # verify GPU is free

serverAppConfigs-docs/LLM/Tcclaviger Fork Ops.md

@@ -0,0 +1,264 @@
+---
+type: reference
+title: Tcclaviger Fork Ops
+description: Operating the tcclaviger RDNA4 vLLM fork on dankserver (shape G era, rollback target since the 2026-10-05 FreeToken cutover) — the ad-hoc sweep harness, measured knob verdicts, sweep hygiene, the large-MoE serving recipe, and compile-cache persistence.
+tags: [tcclaviger, vllm, dankserver, expert-offload, sweeps, cache]
+timestamp: 2026-10-05T00:00:00-07:00
+---
+
+# Tcclaviger Fork Ops
+
+The tcclaviger RDNA4 vLLM fork (`docker.io/tcclaviger/vllm@sha256:da4029666…`,
+vLLM 0.29.0.dev, GLM-5.3/Qwen-specialised with native MoE quant kernels and a
+first-class expert-offload planner) served `frank-primary` at shape G until
+the 2026-10-05 FreeToken cutover; it remains the rollback target
+(`git tag frank-primary-vllm-shapeG-20261003`). Full plan law and VRAM
+arithmetic: [[RDNA4 MoE Expert Offload VRAM Budget]]; boot forensics:
+[[vLLM Boot Forensics and Config Interrogation]]; window discipline:
+[[LLM Bench Windows and Measurement]].
+
+## The ad-hoc sweep harness (dankserver)
+
+Serve script `/home/dank/serve_qwen_fp8.sh` with env hooks: `SPEC` (MTP
+depth), `TUNABLE` (TunableOp), `PREFETCH`, `UTIL`, `MAXLEN`, `KV_BYTES`,
+`MNBT`, `UVA_TOK`, `GATHER_C/GATHER_L` (R4D_LRU gather), `TELEM`
+(R4D_LRU_TELEMETRY), `PROFDIR` (torch profiler bind-mount + --profiler-config
+JSON). The script re-copies the harness (`mtp_measure.py`, `bench_iter.py`)
+into every fresh container — `podman cp` files die on container recreate.
+
+- Measure ONLY via usage counters: `mtp_measure.py` (3 depths) or
+  `mtp_measure_short.py` (166 ctx). `decode = completion_tokens/(wall −
+  ttft)`; chunk counting is INVALID under spec decode.
+- Jobs: `bash /home/dank/job.sh start NAME CMD…` / `wait NAME SECS 40` /
+  `out NAME` ([[Remote Job Runner]]).
+- **Validity check on ANY speed result:** `completion_tokens == max_tokens`
+  (else repetition-halt or early EOS), finish_reason, and inspect the text —
+  `bench_iter` prompts are repeated word-lists + `!` padding, loop-prone. A
+  loop makes LRU hits ~100% → PCIe traffic vanishes → fake 3–4× decode.
+  Acceptance swings 48–61% run-to-run — noise; decode is the metric.
+- Each arm = full reload: ~8.5 min cold, ~12 min with measurement.
+- Readiness gate: `podman logs <c> 2>&1` (stdout+stderr merged) for
+  `Application startup complete` — it is on STDERR; an unmerged grep can
+  never match and burns the whole wait timeout.
+- GPU util ~0% during load/compile is NORMAL (weights + CUDA-graph capture
+  are CPU-side); VRAM climbing is the load indicator.
+- Config edit discipline: the serve script is patched by scp'd python
+  scripts with asserts (anchor must exist, addition must not); `bash -n`
+  after every patch.
+
+## Measured knob verdicts (2026-09-29 campaign)
+
+Incumbent: `SPEC=4 PREFETCH=1 UTIL=0.98 MAXLEN=32768 KV_BYTES=2147483648
+TUNABLE=0`, stock gather 8×16 → 26.80/25.15/26.81 tok/s @ 166/1087/4064 ctx.
+
+| Knob | Verdict | Evidence |
+|---|---|---|
+| Gather geometry `R4D_LRU_GATHER_{CHUNKS,LANES}` | **RULED OUT** | 16×32 worse (~24); 32×64 tie (26.6). The 9.5 vs 22.7 GB/s read-path gap is per-transfer structure (~105 µs/slab vs 78 ideal), not a tunable knob |
+| MTP depth | **RULED OUT** | SPEC=3 ~9% worse; SPEC=6 collapses (13.5–19 tok/s, acceptance 17.4%) — also proves UVA read-through (crosses at step tokens > `R4D_LRU_UVA_TOKENS=5`) is SLOWER than copy-in |
+| TunableOp | **REJECTED** | `CLAV_TUNABLEOP_SWEEP=1` silently SKIPPED under expert offload unless `CLAV_TUNABLEOP_WITH_OFFLOAD=1`; adds ~7 min/load, persists nothing (container-local), and CORRUPTS NUMERICS (loops, repetition-halt, fake 79.8 tok/s @ 96.5% acceptance on degenerate completions). Any "3× win" with early-EOS + absurd acceptance = the loop artifact |
+| MNBT | neutral at 8192 vs 2048 | small values re-chunk prefill only (hurts TTFT, never decode — decode step = max-num-seqs × (spec+1) tokens). No ubatch concept in vLLM |
+
+Decode on this stack is PCIe-gather-bound (~30% GPU duty cycle), so
+batch-shaped compute knobs move ~nothing.
+
+The `R4D_*/CLAV_*` knob family (QSA_*, MHC, ROUTE, ACT_QUANT, AR_QUANT,
+SILU_QUANT) are kill-switches defaulting ON, not perf dials; real levers are
+the R4D_LRU_* family, TunableOp (rejected), R4D_LRU_TELEMETRY, and spec
+depth.
+
+Telemetry gotchas: `--profiler-config` takes a JSON blob, not separate
+flags; `torch_profiler_with_stack` defaults TRUE — a full measure inside a
+profile window OOM-kills the engine worker (~40 GB trace buffer; kernel
+kills python3 at ~95 GB anon-rss). Keep windows to ~15 s / one depth
+(`mtp_measure_short.py`). `telemetry_report` (`expert_cache_rank<N>.json`)
+fires ONLY from the worker's profiler-stop path; worker-side stop can hang
+>5 min on ROCm/kineto. No alternative trigger exists. The LRU hit rate
+measured 82.45% — NOT the ~27% the arena-fraction arithmetic implies;
+temporal reuse dominates, so never infer miss rates from resident-slot
+fractions.
+
+## Sweep hygiene (fork-specific preflight)
+
+1. **Knob landed:** for each `-e` env knob, confirm in-container
+   `podman exec … env` (or the engine log's confirmed-values line). A knob
+   the fork silently ignores produces a "parity" arm that is really the
+   default — the v1 gather sweep lost all three arms to exactly this
+   (GATHER_C/GATHER_L silently dropped, all arms ran stock 8×16).
+2. **Prefetch uniformity:** every arm in a sweep MUST carry the same
+   expert-prefetch setting. The 2026-09-29 campaign measured losing arms
+   with prefetch ON and incumbents OFF; untaxed, its verdicts inverted
+   (no-spec 12.11→41.47; gather 32×64 parity→+18.7%). Never re-read a taxed
+   verdict.
+3. **Spec arms must prove engagement:** engine log
+   `SpecDecoding metrics: Mean acceptance length …` and
+   `curl :7770/metrics | grep vllm:spec_decode_`. Zero such lines = MTP off.
+4. Boot-fail fast-detection: poll `podman ps` INSIDE the health loop; a
+   refused boot dies in one poll instead of 20 min of health waits. Capture
+   >8 lines of `podman logs` on failure — the root cause (e.g. a SEQS init
+   refusal) lives higher up.
+5. Report per-depth tables, never one number: g32×64's win grows with depth
+   (54@1k, 39.5@short) — a single 4064 median hides the short-ctx cost. TTFT
+   rows are noisy ±0.15 s and non-monotonic across depths; compare same-ctx
+   rows across arms only.
+6. Generate ≥256 tokens per sample (at 8 tokens inter_token is first-token
+   noise).
+
+Boot-log facts survive measure failures: KV pool size (`GPU KV cache size`),
+arena slots (`expert cache: … -> N slots`), and the spec-reservation ladder
+(MTP3 766,666 / MTP4 729,545 / MTP6 665,263 tokens) print at BOOT — a
+measure-stage bug costs decode tables but not boot-log findings.
+
+**MTP depth-brittleness law (2026-10-02, all arms rc=0):** MTP3 on this fork
+is depth-brittle — 47.01 t/s at 166 tokens (47.88 at temp 0.7) but 23.09 at
+4064, vs no-spec baseline 37.02 / 33.99. MTP3 + SEQS 16 + MNBT 4096 + util
+0.95 + ladder [4..32] = best overall 37.04 / 38.61 t0.7 and keeps 33.15 at
+4064. `--expert-precise-memory` is load-bearing for MTP (same arm with it
+OFF = worst arm, 19.81 at 4064). Every spec arm costs KV (398,658 → 313,002
+tokens). Acceptance is real: mean length 2.4–2.8, per-position
+0.76/0.55/0.33, avg 52–57%. Next lever named by the engine: MNBT 4096 is
+exactly SEQS×(n+1) (`suboptimal` warning) → try 8192.
+
+## Serving a model larger than VRAM (dankserver recipe)
+
+Dankserver: 2× R9700 (gfx1201), 31.86 GiB each = 63.7 GiB real VRAM, 246 GiB
+RAM. `torch.cuda.device_count()` returns 3 and summing `total_memory` gives
+a bogus 186.7 GiB (the Raphael iGPU is in the list — see
+[[Llama Cpp Serving Sizing]] for the reading discipline).
+
+**Kernel gate — this decides which checkpoint can work at all.**
+`/app/r4dhip/r4d.so` exports `gemm_moe_mxfp4a8_*` and `gemm_moe_dsfp4a8_*`
+but NO `gemm_moe_w4a16_*`. Expert offload requires an experts class with
+BOTH a compiled kernel AND an `offload_recipe`; the planner aborts startup
+otherwise. Usable: MXFP4, DSFP4, FP8-HIP (`R4dMxfp4MoEExperts`,
+`R4dDsfp4MoEExperts`, `Fp8HipMoEExperts`). `R4dW4a16MoEExperts` has no
+kernel (the class exists, the r4d.so entry does not); RFA has neither. Check
+the checkpoint's quant against this list BEFORE downloading anything.
+
+Entrypoint is `/app/tools/image_entrypoint.sh`, ending `exec vllm serve
+"$@"` — pass the model path as the FIRST argument and do NOT pass `serve`.
+Known-good invocation:
+
+```bash
+sudo sh -c 'sync; echo 3 > /proc/sys/vm/drop_caches'
+podman run -d --name <ctr> --ipc=host --ulimit memlock=-1:-1 \
+  --cap-add SYS_PTRACE \
+  --device /dev/kfd --device /dev/dri --device /dev/mem \
+  -e HIP_VISIBLE_DEVICES=0,1 -e VLLM_ROCM_USE_AITER=0 \
+  -v /home/dank/models/<model>:/models:ro \
+  docker.io/tcclaviger/vllm:latest /models \
+  --served-model-name <name> --port 7770 \
+  --tensor-parallel-size 2 \
+  --max-model-len 32768 \
+  --kv-cache-memory-bytes 2147483648 \
+  --max-num-seqs 4 \
+  --gpu-memory-utilization 0.95 \
+  --enable-prefix-caching --trust-remote-code
+```
+
+`-p 7770:7770` is not included, so the API is only reachable via
+`podman exec` — add `-p` if host-side curl is needed. Load ~5 min;
+`/health` answers connection-refused the whole time; watch for
+`Application startup complete` in `podman logs <ctr>`. Verified working:
+`orcarouter/Qwen3.8-Flash-Next-Uncensored-FP8` (gated; 172.8 GiB, 131
+shards, `Qwen4ExpForConditionalGeneration`, 48 layers, 512 experts top-10,
+multimodal, native 262,144).
+
+Traps that each cost a boot:
+
+1. **Mount the whole `/dev/dri`, never individual render nodes.** Render
+   nodes without the primary `card*` nodes → HIP enumerates two agents and
+   refuses a context on either (`RuntimeError: No CUDA GPUs are available`)
+   while `device_count()` cheerfully returns 2. Confirmed by direct A/B.
+2. **memlock must be raised** (done on Dankserver via
+   `/etc/security/limits.d/99-memlock.conf`). Host default 8 MiB makes any
+   large pinned allocation fail as a misleading
+   `torch.AcceleratorError: CUDA error: out of memory` from `pin_memory()`;
+   it cannot be raised from inside a container or root's own shell — only
+   PAM applies it at login. Matters because `--expert-host-mlock` defaults
+   True.
+3. **`--gpu-memory-utilization` does NOT buy KV under expert offload** —
+   the planner sizes the arena against the budget and leaves KV the
+   remainder; raising util 0.95→0.98 grew the arena and the KV pool stayed
+   at 0.64 GiB.
+4. **Lowering `--max-model-len` is the WRONG lever and backfires** — the
+   budget subtracts the KV reserve first, so a smaller context shrinks the
+   reserve and hands the space to the arena (measured: 65536 → 0.64 GiB KV;
+   32768 → 0.34 GiB).
+5. **The KV reserve undercounts hybrid-attention models ~2×** (9190 B/token
+   planned vs ~18,000 real: the Mamba state is a fixed per-sequence cost).
+   Fix by pinning the pool: `--kv-cache-memory-bytes`.
+6. **`--expert-offload-mem` caps the HOST side**, not the VRAM arena —
+   structurally orthogonal to `--expert-precise-memory` (verified
+   in-container 2026-10-05: precise-memory only swaps the GPU overhead term,
+   expert_offload_mem only sets the host bound; they rejoin only inside
+   `place_experts`, and the plan rewrites `expert_offload_mem` back to the
+   cap the plan met, so plan and allocation cannot drift).
+7. `--ipc=host` and `--shm-size` are mutually exclusive in podman; pick one.
+8. `podman run -d` without `-a` sends no stdout to a redirect — the log file
+   only receives the container ID; use `podman logs`.
+
+Status reads: use `scripts/dankstatus.sh`, never hand-rolled
+`/proc/meminfo` parsing — `/proc/meminfo` separates key from value with a
+TAB, so `grep MemAvailable | tr -s " " | cut -d" " -f2` returns empty and
+awk prints `0.0` — a false "host out of memory" that invites an unnecessary
+abort. Gated checkpoints: token via a mode-600 `curl -K` config, never argv
+(`pgrep -a curl` prints it); always `--fail`; compare each file's byte count
+to the Hub's declared size; `-C -` resumes truncated shards.
+
+**Check whether the stack supports the model natively before porting an
+overlay.** The image already registered `Glm5NextForConditionalGeneration`,
+`Qwen4ExpForCausalLM` and both MTP variants — chasing four missing symbols
+through two image tags to bring an overlay up was entirely wasted work.
+
+## Compile-cache persistence (survives `podman rm -f`)
+
+Serve scripts that recreate the container discard the writable layer, so
+every boot re-pays compile/JIT (frank-primary: `torch.compile took 181.20 s`
+cold, of which 173 s was one graph).
+
+1. **Prove it and locate it:** fresh container vs live
+   (`podman run --rm --entrypoint bash IMG -c "ls -la /root/.cache"` vs
+   `podman exec LIVE bash -c "du -sb /root/.cache/*; find /root/.cache -type
+   f | wc -l"`). vLLM cache root: `VLLM_CACHE_ROOT` (default
+   `/root/.cache/vllm`, `torch_compile_cache/`); a HIP runtime adds
+   `/root/.cache/comgr`.
+2. **Mount the PARENT, not the leaf** — one bind on `/root/.cache` persists
+   both trees, safe whenever the image ships no such directory (step 1
+   proves it; otherwise you would be hiding baked state). Wire as an opt-in
+   tunable: `mkdir -p "$CACHE_DIR"` BEFORE the mount, then
+   `${CACHE_DIR:+-v "$CACHE_DIR":/root/.cache}`. Host path outside the
+   checkout.
+3. **Verify with two consecutive boots** — cold `Compiling a graph … takes
+   N s`, warm `Directly load the compiled graph(s) for compile range (…)
+   from the cache` with `torch.compile took` in single-digit seconds.
+4. Traps: the cache key includes the compile RANGE — an MNBT change costs
+   ONE recompile, not a permanent tax; do not claim a proportional boot cut
+   (time to `/health` is dominated by weight loading: 490 s warm vs 188 s
+   cold compile); stale entries are misses, not wrong answers (version +
+   config hashed); if the serve script is seam-gated, assert the mkdir and
+   the exact mount target.
+
+## Plan-line verification checklist (after any fork deploy)
+
+```
+ssh dankserver 'sudo journalctl CONTAINER_NAME=frank-primary-qwen -b -o cat --no-pager \
+  | grep -aoE "(budget \(total x util\) +[0-9.]+|expert cache space +[0-9.]+ GiB +[0-9,]+ slots|Estimated CUDA graph memory.{0,80}|Graph capturing finished.{0,40}|GPU KV cache size.{0,90})"'
+```
+
+`grep -oE` with a digit-anchored tail — plain `grep`/journalctl truncate
+these columns to `17.`. Verify: new budget/graphs/arena in the plan;
+`GPU KV cache size` matches the requested token count; `Estimated CUDA graph
+memory` vs the capture line (raise `_PER_GRAPH_MB` toward 52 if capture
+exceeds the charge); `rocm-smi --showmeminfo vram` high-water per card = the
+real margin; `systemctl show frank-primary -p NRestarts --value` = 0;
+`VLLM_CUDAGRAPH_*` present in `podman exec … env`; then a real chat
+completion from a PEER — `curl localhost:7770/health` inside an ssh session
+on the host can return `000` and looks like an outage that isn't.
+
+## Related
+
+- [[RDNA4 MoE Expert Offload VRAM Budget]] — the plan law this harness tunes
+- [[vLLM Boot Forensics and Config Interrogation]] — journal forensics for these boots
+- [[LLM Bench Windows and Measurement]] — window discipline and bash laws
+- [[Model Swap and Rollback]] — where the fork sits in the swap history
+- [[Remote Job Runner]] — the job.sh harness referenced above

serverAppConfigs-docs/LLM/VLLM Overview.md

@@ -268,7 +268,7 @@ ad hoc qwen-fp8 container (the same fork, same payload, same TP=2):
 
 **There is no `--ubatch-size` equivalent in vLLM.** `--ubatch-size` is a
 llama.cpp compute-buffer concept (`--batch-size`/`--ubatch-size`); the tcclaviger
-fork has no ubatch knob at all. The llama.cpp compute-buffer pair that frank-small
+fork has no ubatch knob at all. The llama.cpp compute-buffer pair that frank-secondary
 was doubled on (2026-10-05, +4.6 % prefill at 8,928 tokens for 616 MiB) has no
 counterpart here, and the two backends should not be reasoned about by analogy.
 

serverAppConfigs-docs/LLM/dankserver-vram-growth.md

@@ -1,7 +1,7 @@
 ---
 type: reference
 title: Dankserver VRAM and RAM Growth Under Load
-description: Investigation of the recurring cuBLAS crash on frank-small. What grows, what does not, and what to do next.
+description: Investigation of the recurring cuBLAS crash on frank-secondary. What grows, what does not, and what to do next.
 tags:
   - dankserver
   - llamacpp
@@ -18,7 +18,7 @@ timestamp: 2026-09-03
 # Dankserver VRAM and RAM Growth Under Load
 
 This page records the investigation of the recurring cuBLAS crash on
-`frank-small` (llama.cpp, RTX 3090 24 GiB). The service aborts
+`frank-secondary` (llama.cpp, RTX 3090 24 GiB). The service aborts
 with `CUDA error: an unsupported value or parameter was passed to the
 function` in `cublasSgemm_v2` at `ggml-cuda.cu:1541` during
 `update_slots` -> `decode` -> `process_ubatch`. It crashed on 2026-08-13
@@ -116,7 +116,7 @@ Verified 2026-08-14 after deploy: flash attention enabled, warmup clean, decode
 
 ## Mitigation Options
 
-The documentation for the benchmark ([[frank-small-benchmark]])
+The documentation for the benchmark ([[frank-secondary-benchmark]])
 prescribes the first option for this crash class. None of these change
 the model.
 
@@ -135,6 +135,6 @@ PR #25649 (slot checkpoints freed on prompt clear, merged), issue #26119
 
 ## See Also
 
-- [[frank-small-benchmark]]
+- [[frank-secondary-benchmark]]
 - [[Dankserver System Specs]]
 - [[deployment]]

serverAppConfigs-docs/LLM/dealign-ornith-crack.md

@@ -1,7 +1,7 @@
 ---
 type: reference
 title: dealign Ornith-1.5-35B-A3B CRACK on Dankserver
-description: dealignai/Ornith-1.5-35B-A3B-UNCENSORED (CRACK Q6_K) served by upstream llama.cpp on the RTX 3090. MoE model with ngram-mod speculative decoding; selected expert weights offloaded to CPU with an explicit --override-tensor. frank-small.
+description: dealignai/Ornith-1.5-35B-A3B-UNCENSORED (CRACK Q6_K) served by upstream llama.cpp on the RTX 3090. MoE model with ngram-mod speculative decoding; selected expert weights offloaded to CPU with an explicit --override-tensor. frank-secondary.
 tags:
   - llamacpp
   - cuda
@@ -14,12 +14,12 @@ timestamp: 2026-08-24
 
 # dealign Ornith-1.5-35B-A3B CRACK on Dankserver
 
-> **Active 2026-08-25.** `frank-small` serves this model. It
+> **Active 2026-08-25.** `frank-secondary` serves this model. It
 > replaced the CRACK Q8_0 file on the same upstream llama.cpp binary (b10121).
 > The Q6_K file was downloaded from the model repository and validated with
 > `llama-gguf` before installation.
 
-`frank-small` (port 7770) serves
+`frank-secondary` (port 7770) serves
 `Ornith-1.5-35B-A3B-CRACK-Q6_K.gguf` from `dealignai` on Hugging Face. It is
 a **MoE** model (256 experts, 8 active) with a hybrid GatedDeltaNet (SSM) +
 attention architecture, abliterated (uncensored). The GGUF also contains a

serverAppConfigs-docs/LLM/deployment.md

@@ -21,7 +21,7 @@ SearXNG, Caddy, or another container service does **not** restart inference.
 |---|---|---|---:|
 | Dankserver | `dank@dankserver` | `frank-medium` | 7770 |
 | Dankclaw | `dank@dankclaw` | `frank-large` | 7770 |
-| Dankclaw | `dank@dankclaw` | `frank-small` | 7771 |
+| Dankclaw | `dank@dankclaw` | `frank-secondary` | 7771 |
 
 ## Scoped workflow
 
@@ -53,13 +53,13 @@ on the selected host. The following files select the corresponding service:
   checkout, so any file there changes deployed state.
 - `scripts/frank-large.env`, `frank-large.service`, and
   `serve-frank-large.sh` select `frank-large` on Dankclaw.
-- `scripts/frank-small.env`, `frank-small.service`, and
-  `serve-frank-small.sh` select `frank-small` on Dankclaw.
+- `scripts/frank-secondary.env`, `frank-secondary.service`, and
+  `serve-frank-secondary.sh` select `frank-secondary` on Dankclaw.
 - Documentation, tests, deployment tooling, and unrelated container changes
   do not select any inference service.
 
 The target-side `scripts/deploy-remote.sh` installs the selected service.
-`frank-large` and `frank-small` are **root units**: the handler relinks the
+`frank-large` and `frank-secondary` are **root units**: the handler relinks the
 repo file into `/etc/systemd/system/` (`relink_unit`), reloads systemd,
 enables it and restarts it. `frank-medium` is a **rootless Quadlet**: the
 handler retires its predecessors (the replica Quadlets and the retired llama
@@ -67,7 +67,7 @@ root unit), stops the service's own unit, waits for the published port to
 clear, installs the `.container` and restarts the generated unit. Each
 handler then waits on that service's health endpoint —
 `http://127.0.0.1:7770/health` for frank-large and frank-medium,
-`http://127.0.0.1:7771/health` for frank-small. The wait budget is 20 minutes
+`http://127.0.0.1:7771/health` for frank-secondary. The wait budget is 20 minutes
 because cold model loads are slow. A failed handler does not advance its
 per-service state.
 
@@ -76,7 +76,7 @@ per-service state.
 The root units point into the stable target checkout:
 ```text
 /home/dank/serverAppConfigs/scripts/frank-large.service
-/home/dank/serverAppConfigs/scripts/frank-small.service
+/home/dank/serverAppConfigs/scripts/frank-secondary.service
 ```
 `frank-medium` is a Quadlet under `dankserver/frank-medium/quadlet/`
 (`frank-medium.container`) rather than a unit in `scripts/`.
@@ -110,7 +110,7 @@ change in that service's inputs.
 
 If the LLM handler fails after the target pull:
 
-1. Inspect `journalctl -u frank-large -n 200` or `journalctl -u frank-small
+1. Inspect `journalctl -u frank-large -n 200` or `journalctl -u frank-secondary
    -n 200` (root units), or `journalctl --user -u frank-medium -n 200`
    (rootless Quadlet).
 2. Confirm the model files, the runtime binary and (for frank-medium) the
@@ -120,13 +120,13 @@ If the LLM handler fails after the target pull:
    ```bash
    ./scripts/deploy.sh dankserver --service frank-medium
    ./scripts/deploy.sh dankclaw --service frank-large
-   ./scripts/deploy.sh dankclaw --service frank-small
+   ./scripts/deploy.sh dankclaw --service frank-secondary
    ```
 4. Verify the matching health endpoint before declaring recovery.
 
 Do not stop the other host's inference service during recovery. The two
 Dankclaw endpoints are independent processes on different devices: restarting
-frank-large does not disturb frank-small, and the reverse holds too — with one
+frank-large does not disturb frank-secondary, and the reverse holds too — with one
 exception worth knowing, that the deployer waits for port 7770 to be released
 before it starts the replacement container. For benchmark work, follow the
 separate stop/benchmark/restart procedure in [[Testing Policy]] and the

serverAppConfigs-docs/LLM/frank-large-flash-server-v2-swift15.md

@@ -0,0 +1,194 @@
+# frank-large — halogen-flash-server 0.16.2, Qwen3.8-Flash-Next v2 Swift1.5 ABLITERATED (Dankclaw, Strix Halo iGPU)
+
+**ACTIVE since 2026-10-06.** Rollback = revert the swap commit's trio
+(`scripts/frank-large.env`, `scripts/serve-frank-large.sh`,
+`scripts/frank-large.service`); the retired 27B payload stays on disk at
+`/home/dank/models/halogen-qwen3.8-27b/` and its fetch script
+(`scripts/download-frank-large-halogen.sh`) stays in the repo.
+
+## Why this shape exists
+
+Frank's ask: replace the frank-large model with
+`https://huggingface.co/Quat3rnion/halogen-swift1.5-qwen3.8-flash-next-v2-abliterated`.
+That payload is an **abliterated Swift1.5 finetune of peonist's
+Qwen3.8-Flash-Next v2 checkpoint in the v2 `.hgn` format** — a file that
+loads ONLY in the **halogen-flash-server** engine line, NOT in the halogen
+0.1.4 container that served the 27B dense checkpoint. The model move is
+therefore a **full engine migration**, and the accepted-product facts flip
+twice in the fleet's favour:
+
+- the base is **abliterated** (the 27B halogen endpoint was a finetune of
+  the censored `Qwen/Qwen3.8-27B`; refusals were an accepted regression there),
+- **vision is back** — the v2 vision tower loads beside the checkpoint and
+  the engine runs it (the 27B halogen engine silently ignored its encoder),
+- **MTP speculative decoding returns** with the head inside the checkpoint
+  itself (the 27B shipped a DFlash2 drafter instead; comparable purpose,
+  different file plumbing),
+- the card claims **53–57 t/s greedy decode, ~1,640 t/s prefill at 34k, ~72 %
+  fewer thinking tokens** than the censored 27B (905 mean), and **262,144
+  native context**. Publisher-card numbers — verify against live traffic
+  before quoting further (see Verification).
+
+The engine is **closed-source**, ships as `ghcr.io/peonist-ai/
+halogen-flash-server:0.16.2` (the version the model card's quickstart pins),
+digest-pinned:
+
+```
+ghcr.io/peonist-ai/halogen-flash-server@sha256:0c61bf84ac22308a53f5d1ca6b86806702d7039e5ebc51cae4c66621b92fe04a
+```
+
+(3.6 GB image; RepoDigest captured at pull 2026-10-06. Card-tested fallback
+if a future pin misbehaves: 0.15.1.)
+
+## The payload — a triangle of files, all sha256-verified at fetch
+
+| File | Bytes | Source repo @ revision | sha256 (LFS) |
+|---|---|---|---|
+| `qwen38-flash-next-v2-swift15-abliterated.hgn` | 66,687,678,464 | Quat3rnion/halogen-swift1.5-qwen3.8-flash-next-v2-abliterated @ main (`69a85db`) | `9c880b8ba48deabfa06172eff1e7572e4fe465e09e97faec6e50c3f6fdcac037` |
+| `qwen38-flash-next-ngram.hgn` (REQUIRED) | 51,200,246,144 | peonist-ai/halogen-qwen3.8-flash-next @ `5cc17cea` | `3450acada94e19aabad88bd49b45eb70304820949c2240fc178067b20ae5dffe` |
+| `qwen38-flash-next-vision.hgn` (enabled) | 897,916,416 | peonist-ai/halogen-qwen3.8-flash-next @ `e053f488` | `d62e0ae553fe88afd3833733d4a4c669f34d20fd8dfce4b9610525bed2134b10` |
+| `tokenizer/` (6 files, flat dir) | 22.4 MB total | peonist @ `e053f488` | `tokenizer.json` is LFS `0997f410…` (byte-identical to the 27B-era copy); the rest are plain git blobs, size-checked |
+
+Payload notes worth keeping:
+
+- The deployed checkpoint is a **compacted r2** of the card's append-only
+  original (89,775,868,032 B → 66,687,678,464 B): the abliteration workflow
+  appends and the repo's `tooling/` compact drops dead padding; live tensors
+  are byte-identical, the LFS sha256 is the contract, and the 24 GiB saving
+  is why it fits.
+- **`HALOGEN_MTP_HEAD` stays UNSET**: the MTP head is INSIDE the v2
+  checkpoint (the separate `qwen38-flash-next-mtp.hgn` sidecar is for a
+  different assembly and is deliberately not fetched).
+- The **n-gram lookup table is REQUIRED**. The engine reads it beside the
+  checkpoint (documented fallback); the card's quickstart also names it
+  explicitly via `HALOGEN_NGRAM_TABLE`, and the serve script passes both —
+  the env var is belt-and-braces over the documented beside-the-checkpoint
+  default, and both mechanisms resolve to the same file.
+- The tokenizer is fetched fresh from peonist @ `e053f488` into
+  `/home/dank/models/halogen-flashnext-v2/tokenizer/` — same flat-directory,
+  no-symlink law as every other payload (an HF cache snapshot's symlinks
+  dangle inside the container mount; both the serve gate and the deployer
+  refuse them).
+
+Fetch: `scripts/download-frank-large-flashnext.sh` (`--verify-only` to
+re-check; `-f` + exact byte counts + LFS sha256 + resolve-the-CDN-then-`-C -`
+resume law throughout; no token — the repos are public).
+
+## Shape — the knobs that are not free
+
+`scripts/frank-large.env` carries the full rationale inline; the table:
+
+| Env | Value | Why |
+|---|---|---|
+| `HALOGEN_CTX` | 262144 | per-slot HARD ceiling (the checkpoint's native window; a longer prompt is a 400 naming the limit, never truncation). **RAISED from 200192 on 2026-10-07 — the 400-zone law:** the harness clamps `max_tokens = min(45056, contextWindow − est − 4096)`, its chars/4 est UNDER-counts the rendered prompt by ~3 % (measured twice: est 152,938 → server 157,388, request over room by 354; est 158,906 → server 165,301, over by 2,300), and compaction only fires at 85 % of the harness window — so with window == engine ceiling, every request in the est ~146k–184k zone was a guaranteed 400. Harness window 217,088 = ceiling − 45k absorbs the error: worst-case request ≈ 219k ≤ 262,144 (≈10 % under-count headroom) |
+| `HALOGEN_KV_SLOTS` | 2 | the 27B engine's "slots>1 disables the drafter" law does NOT carry over — on this engine batching and speculation coexist (the card's quickstart ships 2 slots WITH the head). 2 slots = concurrent capacity |
+| `HALOGEN_KV_POOL_POSITIONS` | 524288 | explicit form of the default (2 × ctx), so the seam test pins the product; ~28 KiB/position ⇒ 14.0 GiB KV |
+| `HALOGEN_HOST_RESERVE_GIB` | 20 | keeps the host honest (the w4b-era lesson) |
+| `HALOGEN_SPEC_ADAPT` | **0** | MANDATORY (card): the default adaptive policy turns the MTP head off for whole requests after low acceptance and **halves decode**. The head stays on; temp-1.0 traffic pays some acceptance for it — accepted trade |
+| `HALOGEN_MAX_TOK` / `HALOGEN_MAX_TOKENS_CAP` / `HALOGEN_QUEUE_TIMEOUT` | 8192 / 65536 / 7200 | 8192 is only the per-request DEFAULT (the harness sends at most its `maxTokens` 45,056); the cap/timeout pair is COUPLED per FLAGS.md (65536 at ~10 t/s serial needs > 6550 s) |
+| `HALOGEN_TEMPERATURE` / `TOP_P` / `TOP_K` | 1.0 / 0.95 / 20 | the card's thinking recipe as SERVER-SIDE defaults — the engine HAS a sampler now, so an absent temperature is no longer greedy; models.yml extraBody matches it (consistent, not load-bearing) |
+| `HALOGEN_WEIGHTS_LOCK` | 1 | card's belt-and-braces; the engine pins weights by default on v2 (`HALOGEN_FLASH_PIN_TRUNK=0` is a w4b-only knob and is REFUSED by v2) |
+| `HALOGEN_MODEL_ID` / publish | **frank-dragoon** / 7770:8731 | served id renamed 2026-10-09 (service/unit/env keep `frank-large`); engine port 8730 never published |
+| MTP depth | default 2 | FLAGS.md: 2 is best all-round, 3 for code — Frank's traffic is agent/prose |
+
+**Memory ledger** (121.2 GiB unified, the referee is the boot ledger):
+weights ~62 GiB resident + KV 14.0 + arena ~8 (`3.8 + 8.4 × 8192/16384`)
++ vision ~2 ≈ **86 GiB engine-side**, + 20 host reserve = ~106 with
+~15 GiB margin over the box's baseline co-tenancy. The 51.2 GB n-gram
+table rides as mmap'd page cache beside the checkpoint — the card's own
+quickstart arithmetic does not bill it as resident, but WATCH the ledger
+at boot and re-size if the engine prints a clamp. Prompt cache is ON by
+default (`PROMPT_CACHE=2`): a follow-up turn at 100k ctx costs ~2 s
+instead of ~88 s cold — the big-TTFT win the 27B engine needed a manual
+`HALOGEN_CACHE_MB` pin for is built in here.
+
+## Engine contract facts (FLAGS.md of the flash-server repo)
+
+- OpenAI chat **and** `/v1/responses` **and** `/v1/messages` **and**
+  `/v1/completions`; tool calls verified on the card; `response_format`
+  still unsupported (no constrained decoding).
+- Thinking → **`reasoning_content`** (same field as every engine this
+  endpoint has run); the v2 chat template rejects `reasoning_effort`
+  outside `xhigh|medium|low` — the models.yml fold (high→xhigh) stays
+  mandatory.
+- Vision: OpenAI `image_url` content parts only (the fleet's standard
+  contract; the legacy top-level `images` field is the frank-secondary trap).
+  ≥256 px probe images or the model answers "video".
+- `HALOGEN_MAX_TOKENS_CAP` over-cap is a 400 naming the limit — never a
+  truncation. `max_tokens` INCLUDES thinking.
+- All flags read once at startup; every flag is BITWISE except
+  `HALOGEN_CK_OVERLAY`, `HALOGEN_MATMUL_TUNING_FILE`,
+  `HALOGEN_INDEXER_BUDGET`, `HALOGEN_COMPOSABLE_CONTEXT`. Unset
+  `HALOGEN_CHECKPOINT` defaults to `/models/qwen38-flash-next-v2.hgn`
+  with the lookup table beside it.
+- Prompt-lookup drafting (3,3 window) rides beside the MTP head by
+  default and is byte-identical to greedy verification where it fires.
+- Watchdog: `HALOGEN_ENGINE_WATCHDOG_S=180` with defer-logic for memory
+  stalls — on a shared host the defaults are the safe choice; the
+  restart policy stays real through the root unit's `Restart=always`.
+
+Kernel: `CONFIG_HSA_AMD_SVM=y` (CachyOS ships it); `amdgpu.noretry=-1`
+(auto) — for this IP auto resolves to retry-on, the same effective state
+the 27B engine ran on. Only revisit if the ledger shows SVM complaints.
+
+## Mechanics
+
+- **Root systemd unit** (`scripts/frank-large.service`, `User=dank`,
+  `LimitMEMLOCK=infinity`, `Restart=always`, `ExecStop=podman stop -t 60
+  frank-large-halogen`), driven by `scripts/serve-frank-large.sh` — the
+  fleet's `podman run`-direct pattern. The script runs the container
+  FOREGROUND so the boot ledger (weights resident, pool fit, MTP head,
+  prompt cache, any clamp) lands in `/home/dank/logs/frank_large.log`.
+- Serve gates mirror the deployer EXACTLY (2026-10-01 crash-loop law):
+  image digest in podman storage, byte-exact checkpoint + n-gram table +
+  vision tower, flat tokenizer with all six files as regular files,
+  GPU nodes present.
+  - **Host-RAM pin-gate preflight** (added 2026-10-06): the engine
+  refuses to pin when `MemAvailable` is below the checkpoint (62.11 GiB)
+  plus a **16 GiB floor** — pinned pages cannot be swapped, and under the
+  floor the n-gram lookup pages in from disk on every long prompt. A
+  RAM-short host crash-loops the unit (each boot dies in ~5 s at the pin
+  gate) until the RAM clears; the 2026-10-06 cutover hit this (8 boots,
+  MemAvailable 74.77 GiB vs the 78.11 GiB needed, self-resolved in ~2 min
+  when the dying 27B engine freed RAM). The serve script now checks
+  `MemAvailable` BEFORE `podman run` and WAITS up to `FRANK_LARGE_RAM_WAIT`
+  (default 600 s) for the RAM to clear, so a transient dip boots once
+  instead of crash-looping; if it never clears it refuses loudly. The 16
+  GiB floor is the engine's (not `HALOGEN_HOST_RESERVE_GIB=20` — a
+  separate, higher, overall-budget knob).
+- Engine stdout goes to the log, NOT the journal. The FIRST request after
+  a (re)start can sit silent ~45–60 s on this engine class (health
+  `engine_unresponsive`, 503s) and self-recovers — do not restart over it.
+
+## Verification checklist (run after any redeploy)
+
+1. Boot ledger greps in `/home/dank/logs/frank_large.log`: resident
+   weights (~62 GiB), KV pool line, MTP/drafter line, prompt-cache line,
+   **no clamp named**.
+2. `curl http://192.168.0.143:7770/health` → 200.
+3. Text completion probe: reasoning lands in `reasoning_content`,
+   `timings` honest (generate ≥128 tokens; at 8 the inter_token number is
+   first-token noise).
+4. Vision probe: a 512×512 solid-red PNG through an `image_url` part →
+   "Red" (≥256 px, one word expected).
+5. Spec decode: `draft_n`/`draft_n_accepted` present in timings on
+   real-shaped traffic (SPEC_ADAPT=0 means the head is always engaged).
+
+## Client contract (harness)
+
+`~/.omp/agent/models.yml` — provider `llamacpp (dankclaw)` @
+`http://192.168.0.143:7770/v1`: `contextWindow: 217088` — deliberately
+BELOW the 262,144 engine ceiling: the 45k gap absorbs the harness's
+~3 % chars/4 est under-count (see the `HALOGEN_CTX` row above) —
+`maxTokens: 45056`, `input: [text, image]`,
+`reasoningContentField: "reasoning_content"`, the `high→xhigh` effort
+fold, `extraBody` temperature 1.0 / top_p 0.95. Compaction still rides
+frank-secondary (the `compactionModel` steering is unchanged; the input
+≈ 170k fits its 200,704 slot).
+
+## Rollback
+
+Revert the swap commit and redeploy — the 27B trio is intact on disk
+(payload at `~/models/halogen-qwen3.8-27b/`, fetch script, and the halogen
+0.1.4 image digest still in podman storage). The older ROCmFPX rollback
+door (`build-rocmfpx.sh` pin) is untouched.

serverAppConfigs-docs/LLM/frank-large-halogen27b-dflash2.md

@@ -25,7 +25,7 @@ confirmed by Frank before deploy:
 
 1. **Text-only.** halogen loads the language tower only; README "honest
    limits": *"The vision encoder is ignored; image inputs are not supported."*
-   The mmproj-era vision role moves wholly to frank-small.
+   The mmproj-era vision role moves wholly to frank-secondary.
 2. **Censored base.** The checkpoint is a finetune of the censored
    `Qwen/Qwen3.8-27B` base, built to make its DFlash2 head strong. The
    ROCmFPX AGENT file was an *Uncensored* build; refusals can return.
@@ -54,6 +54,15 @@ bearing numbers:
   set + host baseline on the 121.2 GiB box. The engine sizes the pool itself
   and **prints a boot ledger naming any clamp** — the post-deploy check reads
   that ledger, not a formula.
+- **`HALOGEN_CACHE_MB=16384` (pinned 2026-10-06)** — auto is OFF on this
+  box: the floor is 2 x a full-context entry (36.9 GB at 262k; one entry
+  is ~13.7 GB at our 200,700-slot x ~70 KB/token) and the ledger had
+  35.7 GB free, so the engine refused and every deep agent request
+  re-prefilled its whole 79-81k context — 84.9-179.8 s measured
+  (serve_api lines, 2026-10-06) for ~500-650 new tokens per turn. The
+  README prices the cache at ~20x time-to-first-token on a warm turn;
+  entries evict LRU by bytes; the reserve (HALOGEN_CACHE_RESERVE_MB,
+  default 8192) is re-checked against live MemAvailable on every store.
 - **No server-side sampler:** an absent temperature is GREEDY. The card's
   thinking recipe (temp 1.0 / top_p 0.95) lives in `models.yml` `extraBody`
   — on the wire only. The template defaults `reasoning_effort` to **xhigh**
@@ -74,7 +83,7 @@ bearing numbers:
 |thinking field|`reasoning_content` (same as before)|
 |`reasoning_effort`|same Qwen3.8 template; outside {xhigh,medium,low} rejected — models.yml fold stays; unset defaults to xhigh, `none` disables thinking|
 |tools|server-side tool calls advertised — round-trip MUST be re-probed after deploy|
-|image input|ignored/error — route to frank-small|
+|image input|ignored/error — route to frank-secondary|
 
 ## Deploy mechanics
 
@@ -94,14 +103,18 @@ bearing numbers:
   those links dangle inside the container mount.
 - **Unit:** `scripts/frank-large.service` (root, `User=dank`, relinked into
   `/etc/systemd/system/`), `ExecStart` = serve script, `ExecStop` = `podman
-  stop -t 120 frank-large-halogen`, script `exec podman wait` so
-  `Restart=always` is real. Engine stdout goes to
-  `/home/dank/logs/frank_large.log` (the boot ledger lives there, not in the
-  journal).
+  stop -t 120 frank-large-halogen`, script runs `podman run` FOREGROUND
+  (2026-10-06: the 2026-10-05 `-d` + `exec podman wait` shape captured
+  only the container ID — the engine ledger existed nowhere on disk and
+  the post-deploy grep below read a one-line hex file; foreground run
+  returns the container's exit code so `Restart=always` is real and the
+  ledger lands in the log). Engine stdout goes to
+  `/home/dank/logs/frank_large.log` (the boot ledger lives there, not in
+  the journal).
 - **Cutover:** `deploy_llm_large` in `scripts/deploy-remote.sh` sweeps the
   legacy bare `llama-server --model /home/dank/models/kingjones-*` process and
   the legacy `frank-large` container name by NAME (never `pkill -f
-  llama-server` — that hits frank-small), retires the old quadlet if the 2026-09
+  llama-server` — that hits frank-secondary), retires the old quadlet if the 2026-09
   halogen era's unit lingers, waits port 7770 free, relinks, health-gates.
 - **Bench law:** stop+mask the unit before any GPU experiment on this box
   (AGENTS.md); `/health` answers 503 while the 33.4 GiB payload loads — that
@@ -109,8 +122,15 @@ bearing numbers:
 
 ## Verification (post-deploy checklist)
 
-1. `grep -iE 'clamp|kv|pool|drafter|dflash' /home/dank/logs/frank_large.log`
-   — ledger shows the pool at the asked shape, DFlash2 active, NO clamp line.
+1. `grep -iE 'clamp|kv|pool|drafter|dflash|cache' /home/dank/logs/frank_large.log`
+   — ledger shows the pool at the asked shape, DFlash2 active, the
+   prompt cache line naming the pinned budget, NO clamp line. EXPECT the
+   WARNING "smaller than ONE full-context entry (18.4 GB at
+   kMaxCtx=262144)": it sizes against the NATIVE window, not our
+   200,700 slot — the deepest possible entry (~14 GB at ~70 KB/token)
+   fits the 16 GB pin, and the 262k entry it fears cannot exist (over
+   slot is a 400). Probe-proven 2026-10-06: the warm turn hit with
+   cache_n=16,384.
 2. `curl -s http://192.168.0.143:7770/health` -> 200.
 3. Real completion with `reasoning_effort` + temp/top_p from the wire recipe;
    `reasoning_content` populated; decode rate from the timings.
@@ -120,6 +140,56 @@ bearing numbers:
    incl. a tool call (V1TOOLS).
 6. Prompt > 200,700 tokens returns the 400 naming the limit (contract test).
 
+## Tuning notes (2026-10-06)
+
+Measured on real agent traffic (serve_api lines) the day after cutover:
+
+- **Decode is depth-bound, and the KV path is the reason.** Shallow
+  prompts (< 1k tokens) decode 30-50 t/s at commit 4-6.3/round; 46k runs
+  5.5, 63-67k runs 4.1-6.5, 79-81k runs 3.3-3.8. The round-time slope is
+  ~8 us/token of context: the KV pool reads at ~8 GB/s effective while
+  the weights stream ~180-240 GB/s (the engine's README puts batch-1
+  weight streaming at the 240 GB/s hardware wall). Commit also falls
+  from 4-6 to 2.5-3 accepted/round as the conversation grows.
+- **No supported flag touches decode.** `docs/FLAGS.md`: HALOGEN_W4A4 /
+  W4A4_EXCL are prefill-only ("decode is unaffected either way"); the
+  undocumented kernel levers are declared unsupported and their defaults
+  are the measured winners. 0.1.4 is the latest release (CHANGELOG
+  head; kernels unchanged since 0.1.0). The publisher's tg bench never
+  measured depth — its tg rows run over the ten SHORT eval prompt
+  shapes — so the README's "decode barely degrades at depth" claim was
+  never exercised at 46-80k.
+- **A/B'd the one FA lever with public precedent (2026-10-06 stop-window
+  probe): rejected.** Issue #12's `HALOGEN_FA_TQ=256` (the 64 GB-carve
+  crash workaround) vs the default tile — same 80k filler prompt,
+  128-token essays, the runtime's own `timings`: decode 6.35 vs 6.34 t/s
+  cold and 5.74 vs 5.76 warm (±0.3 % = noise; draft acceptance identical,
+  53/74 and 58/69), while prefill LOST 9.5 % on this topology (447.5 ->
+  404.8 t/s). The depth collapse is not an FA tile artifact — it sits
+  deeper than the knob. The engine binary advertises 40+ undocumented
+  `HALOGEN_*` kernel levers (FA/DN/MATMUL families, readable via
+  `grep -aoE "HALOGEN_[A-Z0-9_]+"` on the binary); `FA_TQ` is the only
+  one with public evidence, and it moved nothing that matters here.
+- **Routing is the applied mitigation (2026-10-06):** the harness's
+  deep-conversation roles (`slow`, `advisor` in `~/.omp/agent/config.yml`)
+  moved to frank-primary (radiance on Dankserver: fp8 KV at
+  7.32 KiB/token = 8.7x less KV to read per round at depth, MTP depth 3,
+  200k window, prefix-cache host+disk tiers). frank-large keeps OpenWebUI
+  and shallow bursts, where it runs 30-50 t/s. frank-primary's
+  depth-decode rate is designed-for but not yet measured on record.
+- **The lever that existed was the prompt cache, and it was OFF** (see
+  the shape bullet). The pin turns the 85-180 s re-prefill per deep
+  turn into a resume; decode itself is expected UNCHANGED by the pin —
+  the cache resumes prefill, it does not touch the round loop.
+  Probe-proven (two synthetic 17.3k turns, 2026-10-06): cold prefill
+  27.0 s (640 t/s, cache_n 0) -> warm turn cache_n 16,384, prompt_n 946
+  (the extension + one 2048-aligned chunk), prefill 1.7 s — 16x on the
+  pair, answer correct warm.
+- **Everything else stays put.** dflash2 over mtp (publisher 31.71 vs
+  26.88 mean; at depth both pay the same KV term and dflash2's higher
+  commit wins), KV_SLOTS=1 (batch XOR speculation), SLOT_CTX=200700
+  (Frank's window), W4A4 shipped (+9 % prefill, decode unharmed).
+
 ## Rollback
 
 Revert the frank-large swap commit and redeploy `dankclaw --service frank-large`:

serverAppConfigs-docs/LLM/frank-large-ik-uncensored.md

@@ -123,3 +123,27 @@ Stop and disable the root unit, trash it, restore the halogen quadlet from
 `~/.AGENTTRASH/20260915-ik-swap/halogen-quadlet` to
 `~/.config/containers/systemd/`, `systemctl --user daemon-reload`, start;
 the .hgn checkpoint and the kingjones STRIX llama.cpp payload remain on disk.
+
+## Hybrid campaign envelope (2026-09-16, 3090 + Strix Halo)
+
+Why this deployment could not be tuned into a production endpoint: decode
+3.8–4.3 tok/s flat across thread counts (T1 ≈ T16); prefill ~192 tok/s at
+1.8k–32k prompts = the **CPU AVX512-VNNI compute ceiling** for CPU experts.
+The decode root cause is structural: ~100 CPU↔GPU backend splits per token
+with the GPU at 5 % SM during decode — launch/sync-bound over the PCIe x4
+dock, not compute- or memory-bound. T16 profiling shows ~90 % of cycles in
+`pause` (ggml_barrier spin): threads wait on the serial critical path.
+Memory placement (mmap/anon/THP/deferred) does not matter — all variants
+landed 3.8–4.6.
+
+- CUDA-graph capture aborts: qwen4exp's hyper-connection broadcasts use
+  per-layer `ggml_repeat_4d`, whose CUDA path is capture-illegal; until ik
+  gains capture-safe repeat/hc ops (mainline PR #28901 direction), graphs
+  stay dead.
+- Mainline llama.cpp itself runs 1.0 tok/s on this model (CPU MoE kernels
+  4× slower than ik's iqk) — do not switch engines for speed.
+- MTP/speculative decoding is impossible for mradermacher i1 GGUFs: the
+  NextN head is stripped (no `blk.48` tensors).
+- Tuning lever ranked first: `-ncmoe 42` (6 layers' experts on the 3090) →
+  decode 4.54, small-prompt prefill 32 — but VRAM 22/24.6 GiB at 262k ctx,
+  an operator OOM-margin decision, and it needs `GGML_CUDA_NO_PINNED_WEIGHTS=1`.

serverAppConfigs-docs/LLM/frank-large-rocmfpx-q6agent-uncensored-mtp.md

@@ -148,7 +148,7 @@ in [[frank-large-rocmfpx-q8agent-mtp]] and still applies.
 
 Sweep key for scratch servers is now the payload name:
 `pkill -f 'Qwen3.8-27B-Uncensored-Q6_0_ROCMFPX_AGENT'` — never `pkill -f
-llama-server` (frank-small is also one).
+llama-server` (frank-secondary is also one).
 
 ## Operations