feat(media): the movie-library relayout toolchain — census, plan, executor

The movie library on the entropy volume is release-folded: the video inside
`Some.Movie.2019.1080p…-FOO/` is named after the release, and Jellyfin matches
a movie by folder and file name, so the noise cost posters, extras and versions.
Three scripts own the fix, and nothing else in the repo moves a media file.

  * media-census.py  the read-only inputs: a stat pass, an ffprobe header pass
                     (nice'd, so it is safe beside a live endpoint), and the
                     Plex metadata cache read that outranks any filename guess
  * media-plan.py    the Jellyfin-format planner: one folder per movie, the
                     video named exactly like the folder, extras in the
                     canonical subfolders, copies of one film as versions. It
                     opens no media file, so a whole-library plan costs seconds,
                     and it ends in gates — bad-names, occupied, double-claim,
                     left-unmoved, unplanned — that must all read zero before a
                     human points the executor at 24 TB
  * media-apply.py   the executor: rename(2) only, the intent fsynced to a
                     write-ahead journal BEFORE the move it authorises, so an
                     interrupted run is finishable and every run is undoable.
                     Refusals are all-or-nothing and it never overwrites a file

Laws the fixture pins (tests/media_relayout.bats, 16 tests, each proven red
against a reverted copy of the rule it guards):

  * a Plex year of 0 means "indexed, never matched" — trusting it produced
    folders "Title (0)": items with no year, no poster, no identity
  * a Plex title carrying "/" is sanitised, or "V/H/S/2" became a five-level
    tree neither matcher reads as a movie
  * a folder inside a movie's folder is bonus territory even when its name is
    misspelled ("Exrtas"), unless the file's own name carries another year
  * matching words split on everything but a letter or digit — the earlier
    tokenizer left "(1960)/extras" glued, so a film could not be found by the
    folder holding it and one box set's featurettes drifted into a neighbour
  * a name with no words in it is bonus material, never a movie folder named 1

Docs: the runbook page owns the procedure, the gates and the rollback; AGENTS.md
links it from the runbook index and the summary table.

AGENTS.md

@@ -26,8 +26,8 @@ merge absorbed the former `vllmScripts` repo, history preserved).
 | Target | Service | Runtime | Payload | Port |
 |--------|---------|---------|---------|------|
 | Dankserver | `frank-archon` | radiance, digest-pinned, root unit, TP=2 | publisher `qwen3.8-next-flash-fp8-iq4r-moe.rad` | 7770 |
-| Dankclaw | `frank-dragoon` | halogen-flash-server 0.16.2, digest-pinned, root unit | `qwen38-flash-next-v2-abliterated.hgn` + n-gram + vision tower | 7770 |
+| Dankclaw | `frank-dragoon` | halogen-flash-server 0.18.20, digest-pinned, root unit | `qwen38-flash-next-v2-abliterated.hgn` + n-gram + vision tower | 7770 |
-| Dankclaw | `frank-zealot` | llama.cpp PrismML fork build 10777, root unit — the only loader for this payload | Ternary-Bonsai-2-9B PQ2_0 + Q8_0 mmproj + rank-1 refusal adapter | 7771 |
+| Dankclaw | `frank-zealot` | llama.cpp PrismML fork build 12118, root unit — the only loader for this payload | Ternary-Bonsai-2-9B PQ2_0 + Q8_0 mmproj + rank-1 refusal adapter | 7771 |
 
 Payload identity, shape, eras, laws and rollback live in ONE page each:
 [[frank-archon]], [[frank-dragoon]], [[frank-zealot]]. `scripts/<service>.env`
@@ -293,8 +293,8 @@ means off or NIC dead; a merely wedged kernel usually still answers ARP.
 2. Bench against the service itself. No scratch ports on an LLM host — one
    process owns the card. Use the runtime's own harness (`llama-bench`; a
    live llama.cpp server also reports honest `timings` per request, including
-   `draft_n`/`draft_n_accepted` — but on frank-zealot's PrismML fork build
+   `draft_n`/`draft_n_accepted` — but on frank-zealot's PrismML fork that object
-   10777 that object only reaches you on **`/v1/chat/completions`**: a greedy
+   only reaches you on **`/v1/chat/completions`** (measured on build 10777): a greedy
    `ignore_eos` run on the raw `/completion` path there answers **500 "The
    model produced output that does not match the expected Content-only
    format"** (measured 2026-10-18). halogen has no `/completion` path at all,
@@ -439,16 +439,16 @@ disagrees with it is the bug.
 ### frank-zealot — llama.cpp (PrismML fork), RTX 3090, :7771
 
 - Owns: `scripts/frank-zealot.env`, `scripts/serve-frank-zealot.sh`,
-  `scripts/frank-zealot.service`; fork build `llama-prism-b10777` is the ONLY
+  `scripts/frank-zealot.service`; fork build `llama-prism-0244744` (build 12118)
-  loader (the payload carries the fork's custom PQ2_0 tensor type plus
+  is the ONLY loader — `llama-prism-b10777` stays on disk as its rollback (the payload carries the fork's custom PQ2_0 tensor type plus
   `prism.hadamard.*` dequant metadata — stock llama.cpp refuses it, and a
   loader ignoring that metadata dequantizes wrong); payload
   `Bonsai2-9B-MTP/…PQ2_0.gguf` + the publisher's **Q8_0** mmproj (the BF16
   tower can crash the server with concurrent image requests) + the rank-1
   refusal-direction adapter applied at load by `--lora-scaled …:1`. The
   adapter's rollback is `LORA_FILE=` (empty) in the env — one line, no rebuild
-  — and build 10777 is also the gate that lets it load: older forks refuse it
+  — and the fork BUILD is also the gate that lets it load: anything older than
-  with `unsupported prism.hadamard.version: 2`.
+  b10777 refuses it with `unsupported prism.hadamard.version: 2`.
 - Shape: `--ctx-size 602112 --parallel 3` = **three conversations at
   200,704 each**, `CACHE_TYPE_K=f16` + `CACHE_TYPE_V=q8_0` (asymmetric: 25,088
   B/token, 14,406 MiB of pool), batch 6144 / ubatch 1024,
@@ -741,6 +741,13 @@ linked runbooks.
   `/etc/default/grub`. It replaced `amd_iommu=off`, and the local A/B on byte-identical traffic
   says identity mode costs nothing, so the XDNA2 NPU returned for free.
   [[dankclaw-iommu-translation-off]]
+- **A deploy handler that only `start`s a quadlet no-ops an image bump.**
+  `systemctl --user start` on an already-active unit does nothing, so installing
+  a new digest under a running service keeps the OLD container serving while the
+  deploy, its health gate and the Playwright suite all pass — the authentik
+  handler did exactly that on 2026-10-30 (2026.8.3 → 2026.8.5 moved only the
+  outpost, the one handler that restarted). A pin change is applied by
+  `restart`, never `start`.
 - **Rootless Quadlet containers run with `--replace --rm`**: `podman logs` covers only the CURRENT
   instance; history across recreations lives in `journalctl --user -u <name>.service`. Read
   rootless storage WITHOUT sudo — it is per-user storage, so `sudo podman logs` finds nothing.
@@ -800,6 +807,7 @@ per-attempt page and no per-swap ledger anywhere else.
 | `frank-tts` — cloned-voice speech, Dankclaw | `serverAppConfigs-docs/Dankclaw/Frank TTS.md` |
 | Dankclaw kernel posture (`iommu=pt` since 2026-10-18) | `serverAppConfigs-docs/LLM/dankclaw-iommu-translation-off.md` |
 | Deployment flow | `serverAppConfigs-docs/Architecture/Deployment Flow.md` |
+| Movie-library relayout (Jellyfin format, Dankserver) | `serverAppConfigs-docs/Dankserver/Media Relayout.md` |
 
 There is no page for a retired endpoint and no change diary. A payload swap
 appends one row to the live endpoint's **Eras** table; when the endpoint itself
@@ -841,5 +849,6 @@ history.
 | Writing a doc | **One owner per fact** — edit the endpoint/service page; everywhere else links. Never restate a number. |
 | Benchmarking (only when ordered) | **Stop and mask the service first**, bench, unmask, restart, verify health. |
 | Deleting a file | **Move it to trash** — `.AGENTTRASH/` (in-repo) or `~/.AGENTTRASH/YYYYMMDD-topic`. Never `rm`. |
+| Relayout the movie library | **Census → plan → read the exceptions → `scripts/media-apply.py`** — renames only, journal before every move; never hand-rename, never copy. |
 | Tests exist | **Run them** — verify tests pass after every change. Do not commit failing tests. |
 | Real secrets | **Never commit** — host `.env` files; `.env.example` holds placeholders only. |

dankclaw/authentik/quadlet/authentik-outpost-ldap.container

@@ -23,7 +23,7 @@ Network=dankclaw_default
 # Image=ghcr.io/goauthentik/ldap:2026.8.3 (amd64 digest; podman 5.7.0 refuses
 # the tag's nested manifest list, so pin the single-platform digest).
 EnvironmentFile=/home/dank/serverAppConfigs/dankclaw/.env
-Image=ghcr.io/goauthentik/ldap@sha256:2e7b259eb8c37faac916946ddf48ec681ee9d3a9f724fe8521bf161973f13cf9
+Image=ghcr.io/goauthentik/ldap:2026.8.5@sha256:4c75bd27918418d2d6fbbfe4581bbd5a199fe38942d24559c4483edd64a1ea87
 Environment=AUTHENTIK_HOST=https://auth.franksdank.cloud
 Environment=AUTHENTIK_INSECURE=false
 Environment=AUTHENTIK_TOKEN=${AUTHENTIK_LDAP_OUTPOST_TOKEN}

dankclaw/authentik/quadlet/authentik-outpost.container

@@ -20,7 +20,7 @@ Wants=authentik-server.service
 
 [Container]
 ContainerName=authentik-outpost
-Image=ghcr.io/goauthentik/server:2026.8.3@sha256:ab9b4e8cc4ab3f8d1198d2db6aeea66bafea1963b3f2843589e0d163f97d9849
+Image=ghcr.io/goauthentik/server:2026.8.5@sha256:602227a86455d6772e3d16b7d852713c4f864f40df5df894abd1285bb53979a5
 Exec=worker
 Network=dankclaw_default
 EnvironmentFile=/home/dank/serverAppConfigs/dankclaw/.env

dankclaw/authentik/quadlet/authentik-postgres.container

@@ -14,7 +14,7 @@ After=network.target
 
 [Container]
 ContainerName=authentik-postgres
-Image=docker.io/library/postgres:16-alpine@sha256:721873c34ceb9f8d8fc265984940dc982404c105f19ad51be9fdc5970a6080ea
+Image=docker.io/library/postgres:16-alpine@sha256:0cefaabd882e42bcb2a2f3dc56d3ce4770560ba62fd7f197f1a2e008fd5731ab
 Network=dankclaw_default
 EnvironmentFile=/home/dank/serverAppConfigs/dankclaw/.env
 Environment=POSTGRES_DB=authentik

dankclaw/authentik/quadlet/authentik-server.container

@@ -19,7 +19,7 @@ Wants=authentik-postgres.service
 
 [Container]
 ContainerName=authentik-server
-Image=ghcr.io/goauthentik/server:2026.8.3@sha256:ab9b4e8cc4ab3f8d1198d2db6aeea66bafea1963b3f2843589e0d163f97d9849
+Image=ghcr.io/goauthentik/server:2026.8.5@sha256:602227a86455d6772e3d16b7d852713c4f864f40df5df894abd1285bb53979a5
 Exec=server
 Network=dankclaw_default
 EnvironmentFile=/home/dank/serverAppConfigs/dankclaw/.env

dankclaw/authentik/quadlet/authentik-worker.container

@@ -16,7 +16,7 @@ Wants=authentik-server.service
 
 [Container]
 ContainerName=authentik-worker
-Image=ghcr.io/goauthentik/server:2026.8.3@sha256:ab9b4e8cc4ab3f8d1198d2db6aeea66bafea1963b3f2843589e0d163f97d9849
+Image=ghcr.io/goauthentik/server:2026.8.5@sha256:602227a86455d6772e3d16b7d852713c4f864f40df5df894abd1285bb53979a5
 Exec=worker
 Network=dankclaw_default
 EnvironmentFile=/home/dank/serverAppConfigs/dankclaw/.env

dankclaw/gitea/quadlet/gitea-postgres.container

@@ -15,7 +15,7 @@ After=network.target
 [Container]
 ContainerName=gitea-postgres
 Network=gitea.network
-Image=docker.io/library/postgres:16-alpine@sha256:721873c34ceb9f8d8fc265984940dc982404c105f19ad51be9fdc5970a6080ea
+Image=docker.io/library/postgres:16-alpine@sha256:0cefaabd882e42bcb2a2f3dc56d3ce4770560ba62fd7f197f1a2e008fd5731ab
 EnvironmentFile=/home/dank/Containers/gitea/env
 Environment=POSTGRES_USER=gitea
 Environment=POSTGRES_PASSWORD=${GITEA_DB_PASSWORD}

dankclaw/hermes/quadlet/chrome-devtools-protocol.container

@@ -25,7 +25,7 @@ Description=Chrome DevTools Protocol daemon (headless-shell) for hermes
 [Container]
 ContainerName=chrome-devtools-protocol
 Network=host
-Image=docker.io/chromedp/headless-shell@sha256:2d349b544a1ea6b5b5fd7c0fe99215ff662339c57407ee2e8c0a11af93516b04
+Image=docker.io/chromedp/headless-shell@sha256:7bee201478b2f6266aae01243740ee85c16281e32f42d3aed9fdf04f4eace7a1
 Entrypoint=/headless-shell/headless-shell
 Exec=--no-sandbox --disable-gpu --use-gl=angle --use-angle=swiftshader --remote-debugging-address=127.0.0.1 --remote-debugging-port=9222
 ShmSize=1073741824

dankclaw/marreta/quadlet/marreta-browser.container

@@ -23,7 +23,7 @@ After=gluetun.service network.target
 [Container]
 ContainerName=marreta_browser
 Network=gluetun.container
-Image=docker.io/lightpanda/browser:nightly@sha256:979c36ed869f1016b8dcf22939a0633c262f2c30e892a03a9f720eba2ff1931e
+Image=docker.io/lightpanda/browser:nightly@sha256:e72730e364d1fbf352c1149f29ffa4b278f3b1a55ff0dec3262298ebe624ec7a
 
 [Service]
 TimeoutStartSec=300

dankclaw/observability/quadlet/alertmanager.container

@@ -33,7 +33,7 @@ Wants=network-online.target
 
 [Container]
 ContainerName=alertmanager
-Image=docker.io/prom/alertmanager:latest@sha256:e9733bafb1bdef9b00e25a21f8f99dc26a22224bf16641ad754d1649f4c3357a
+Image=docker.io/prom/alertmanager:latest@sha256:39d8bb3966c694a124428a3a88fbcffd40bd069129ecd47c059f8a3f17c55f3b
 Network=dankclaw_default
 User=0:0
 Exec=--config.file=/etc/alertmanager/alertmanager.yml --storage.path=/alertmanager

dankclaw/observability/quadlet/grafana.container

@@ -34,7 +34,7 @@ Wants=network-online.target prometheus.service
 
 [Container]
 ContainerName=grafana
-Image=docker.io/grafana/grafana:latest@sha256:dbbc38ea21103cefb50396340cd88afec779c7e070b0a7bf5a75b4ce46651d3f
+Image=docker.io/grafana/grafana:latest@sha256:488984b716c0862639f91a698adbe2862677a5a07bad42da5f03f3030883b205
 Network=dankclaw_default
 User=0:0
 PublishPort=127.0.0.1:3002:3000

dankclaw/open-webui/quadlet/open-webui.container

@@ -27,7 +27,7 @@ After=network.target
 [Container]
 ContainerName=open-webui
 Network=open-webui.network
-Image=ghcr.io/open-webui/open-webui:main@sha256:4860ba005ae8f19f08d1e12b0868855b7b339ea0dad9927f196820fc0de04575
+Image=ghcr.io/open-webui/open-webui:main@sha256:0b399a8ba34558e181d316750455f2a95376a38cd60a61b9c942e138898944bd
 PublishPort=3005:8080
 Environment=OPENAI_API_BASE_URL=http://192.168.0.143:7770
 Environment=ANONYMIZED_TELEMETRY=False

dankclaw/searxng/quadlet/gluetun.container

@@ -16,7 +16,7 @@ After=searxng-network.service network.target
 [Container]
 ContainerName=gluetun
 Network=searxng.network
-Image=docker.io/qmcgaw/gluetun:latest@sha256:c2ee4191762fca6b6a3c3359a731198a95d9c9af4db32d466e223bf3a84e779b
+Image=docker.io/qmcgaw/gluetun:latest@sha256:4af0c804003481e100180e1f8d4d5e9c48163fedc0f88d061881b615aef9d183
 AddCapability=NET_ADMIN
 AddDevice=/dev/net/tun:/dev/net/tun
 Environment=VPN_SERVICE_PROVIDER=${VPN_SERVICE_PROVIDER}

dankclaw/searxng/quadlet/searxng-valkey.container

@@ -16,7 +16,7 @@ After=searxng-network.service network.target
 ContainerName=searxng-valkey
 Network=searxng.network
 NetworkAlias=valkey
-Image=docker.io/valkey/valkey:9-alpine@sha256:48332870af354a799964c0012ae1194a0bf2bf894eb508f945810596dc2d8d11
+Image=docker.io/valkey/valkey:9-alpine@sha256:bf09d58e96d6b4524a15396b65b242bbe18beebbcc15a04d8ee75ddac62fae0b
 Volume=/opt/searxng/valkey:/data
 
 [Service]

dankclaw/searxng/quadlet/searxng.container

@@ -16,7 +16,7 @@ After=gluetun.service searxng-valkey.service network.target
 [Container]
 ContainerName=searxng
 Network=gluetun.container
-Image=ghcr.io/searxng/searxng:latest@sha256:5ed0323b5f5f8701264ef65e239a2bd83cb57098f2b971fa30bf79c06eeae271
+Image=ghcr.io/searxng/searxng:latest@sha256:f4aa660c54d062dedb268be84e821ea24e39f48c7a496af8dd02cc9eedb93847
 Environment=SEARXNG_SECRET=${SEARXNG_SECRET}
 Environment=SEARXNG_BASE_URL=${SEARXNG_BASE_URL}
 Volume=/opt/searxng/config:/etc/searxng

dankclaw/seerr/quadlet/seerr.container

@@ -29,7 +29,7 @@ Wants=network-online.target
 
 [Container]
 ContainerName=seerr
-Image=ghcr.io/seerr-team/seerr:v3.4.1@sha256:f4768de5f616248d723e05891f3345a1402123775d03bf0890dbfedc0831bda1
+Image=ghcr.io/seerr-team/seerr:v3.5.1@sha256:d8fab675b55ad293ded98fd71efcfd46054cbad9746534ae1cc4794a88c0b3be
 Network=dankclaw_default
 PublishPort=127.0.0.1:5055:5055
 Volume=/home/dank/Containers/seerr/data:/app/config

dankserver/calibre-server/quadlet/calibre-server.container

@@ -43,7 +43,7 @@ After=network.target
 
 [Container]
 ContainerName=calibre-server
-Image=lscr.io/linuxserver/calibre:9.15.0@sha256:59e4b0e036e62f228f8bcfae1ade42f47047719885fe454ddbad38e4597b987b
+Image=lscr.io/linuxserver/calibre:9.17.0@sha256:b198fc03a6abc6be7e251a9958e50e74634b00e447c7b25e92c191c8077ba378
 Network=pasta
 EnvironmentFile=/home/dank/Containers/calibre-server/env
 Environment=PUID=1000

dankserver/calibre-web/quadlet/calibre-web.container

@@ -16,7 +16,7 @@ After=network.target
 
 [Container]
 ContainerName=calibre-web
-Image=lscr.io/linuxserver/calibre-web:0.6.27-ls402@sha256:802729966faed5e32d671cd4f92ba6196e9e754a6c0f740ce1ee9acd86b10e55
+Image=lscr.io/linuxserver/calibre-web:0.6.27-ls407@sha256:470b4ed9a7d6ea1475fef17dbcf957cb60880910296d226bd0dca652c05f68ca
 Network=pasta
 Environment=DOCKER_MODS=linuxserver/mods:universal-calibre
 Environment=LANG=en_US.UTF-8

dankserver/jellyfin/quadlet/jellyfin.container

@@ -16,7 +16,7 @@ After=network.target
 
 [Container]
 ContainerName=jellyfin
-Image=ghcr.io/jellyfin/jellyfin:latest@sha256:c3efa69a22f2691fc88d6bb035e87a8560e395a6047f062fb381c61c3185e8f5
+Image=ghcr.io/jellyfin/jellyfin:latest@sha256:fea863d0cd9ec445985d9c75fc6ed64e38a1d2766cf0c250a33d553f4fc1b771
 Network=dankserver_appnet
 Environment=DEBIAN_FRONTEND=noninteractive
 Environment=DOTNET_SYSTEM_IO_DISABLEFILELOCKING=true

dankserver/plex/quadlet/plex.container

@@ -4,7 +4,7 @@ After=network-online.target
 Wants=network-online.target
 
 [Container]
-Image=lscr.io/linuxserver/plex:1.43.5.11029-6bf8af3f0-ls329@sha256:8f655bad7dcbc5398a78f70a5af12600b4ed3424f901566ed1eaa6710e9e2334
+Image=lscr.io/linuxserver/plex:1.43.5.11079-b3b13e506-ls330@sha256:01a53d9c7adcbb3701fbb0a4230c0a37c85453759b6b05a67356c3c7a09de08d
 ContainerName=plex
 Network=dankserver_appnet:alias=plex
 Environment=PUID=0

dankserver/romm/quadlet/romm-mariadb.container

@@ -14,7 +14,7 @@ After=network.target
 
 [Container]
 ContainerName=romm-mariadb
-Image=docker.io/library/mariadb:11@sha256:f16d7a344413e519dff518f27bdb221388f84b0d274c97c4fb5f2cb0dc1b34fa
+Image=docker.io/library/mariadb:11@sha256:c4124d1afa875e47f2892edf26b482fa54f796e2f342d3717a04ca091a22a9bd
 Network=dankserver_appnet:alias=mariadb
 EnvironmentFile=/home/dank/Containers/romm/env
 Environment=MARIADB_DATABASE=romm

dankserver/romm/quadlet/romm-redis.container

@@ -12,7 +12,7 @@ After=network.target
 
 [Container]
 ContainerName=romm-redis
-Image=docker.io/library/redis:7-alpine@sha256:858f009f9709ce576febc734aa78b8f6d624b82571f9ddb6bda4377c833b3499
+Image=docker.io/library/redis:7-alpine@sha256:8da19c7399dca900c6822a1cf30f9d2285585e2c4b457ff4095b0f4500054d3f
 Network=dankserver_appnet:alias=redis
 Volume=/home/dank/Containers/romm/redis:/data
 HealthCmd=redis-cli ping

dankserver/romm/quadlet/romm.container

@@ -18,7 +18,7 @@ After=romm-mariadb.service romm-redis.service network.target
 
 [Container]
 ContainerName=romm
-Image=ghcr.io/rommapp/romm:latest@sha256:03295f95a6290e752641f9553ac4198b0e5b71b879d57f7b2371b98f608feb54
+Image=ghcr.io/rommapp/romm:latest@sha256:8677fb7b81b1d3db20b849e989a1dadc2a11159e51eb34a2acdcb01102274edf
 Network=dankserver_appnet
 EnvironmentFile=/home/dank/Containers/romm/env
 Environment=DB_HOST=mariadb

dankserver/tautulli/quadlet/tautulli.container

@@ -4,7 +4,7 @@ After=network-online.target
 Wants=network-online.target
 
 [Container]
-Image=lscr.io/linuxserver/tautulli:v2.18.1-ls245@sha256:c66d3afa8f7a701a48dd1b1f57b22eca0483d895d670fe11ecff783f6a9bbd73
+Image=lscr.io/linuxserver/tautulli:v2.18.2-ls250@sha256:529c7eabcc40554c1dc48f33f1f3cc3b1438462d73e0ec2dac8ac0b66aeeb8d4
 ContainerName=tautulli
 Network=dankserver_appnet:alias=tautulli
 Environment=PUID=0

scripts/calib-push.py

@@ -22,6 +22,17 @@ Mechanics notes:
   prompts do not bias the Hessians (the corpus is drawn round-robin over
   source classes, so loss is class-neutral as long as failures are rare;
   a systematic 4xx is loud in the log and must be fixed, not ignored).
+- a round that seals no tap bytes is NORMAL and continues: RADIANCE_CALIB_TOKENS
+  is a pooled floor and this campaign's corpus is 10.03M tokens against a 12M
+  floor, so round 1 seals nothing by construction. A tap that LOSES bytes fails
+  the run, and a run that never sees the engine's completion line exits non-zero.
+  The first version failed on a flat round, which aborted exactly one round too
+  early (2026-10-29).
+- before any prompt, the tap must carry the rig's own marker
+  (`--tap-marker`, written by run-qwen38-bf16.sh calib) naming THIS
+  container, and that container must be running. Measured 2026-10-29:
+  with neither check, four rounds of 10M tokens went into restored
+  production after the rig's boot had died, and recorded nothing.
 
 Usage: calib-push.py [--port 7770] [--parallel 64] [--container arbench]
        [--corpus /home/dank/corpora/out/calib.jsonl] [--max-rounds 4]
@@ -29,11 +40,18 @@ Usage: calib-push.py [--port 7770] [--parallel 64] [--container arbench]
 import argparse
 import concurrent.futures
 import json
+import os
 import subprocess
 import sys
 import time
 import urllib.request
 
+_HERE = os.path.dirname(os.path.abspath(__file__)) if "__file__" in globals() else os.getcwd()
+if _HERE not in sys.path:
+    sys.path.insert(0, _HERE)
+
+import qwen38_common as qc  # noqa: E402 - sibling module, same directory
+
 DONE_MARK = "calibration wrote"
 
 
@@ -49,9 +67,9 @@ def tap_done(container):
     return DONE_MARK in (out.stdout + out.stderr)
 
 
-def push_one(url, prompt):
+def push_one(url, prompt, model):
     body = json.dumps({
-        "model": "frank-archon",
+        "model": model,
         "prompt": prompt,
         "max_tokens": 1,
         "temperature": 0,
@@ -68,38 +86,151 @@ def push_one(url, prompt):
         return False
 
 
-def main():
+def push_round(url, prompts, model, parallel):
+    """Push the whole corpus once; returns how many requests were accepted."""
+    ok = 0
+    with concurrent.futures.ThreadPoolExecutor(parallel) as ex:
+        for good in ex.map(lambda p: push_one(url, p, model), prompts):
+            ok += 1 if good else 0
+    return ok
+
+
+def tap_bytes(path):
+    """Bytes the tap has sealed so far; 0 when the directory is absent.
+
+    A tap whose directory is wrong, full or read-only stops writing while the
+    engine keeps answering 200 — the driver then re-pushes the whole corpus for
+    every round it is allowed and the loss is invisible until the hours are gone.
+    """
+    total = 0
+    try:
+        for root, _, files in os.walk(path):
+            for f in files:
+                try:
+                    total += os.path.getsize(os.path.join(root, f))
+                except OSError:
+                    pass
+    except OSError:
+        return 0
+    return total
+
+
+def round_verdict(before, after, done):
+    """What one round's tap bytes mean.
+
+    'done'   the engine logged that it sealed the tap — the run is over
+    'shrink' the tap LOST bytes: reset, pruned or replaced under us
+    'grew'   sealed bytes this round
+    'flat'   sealed nothing — which is legitimate: RADIANCE_CALIB_TOKENS is a
+             pooled floor, and a round that does not reach it has nothing to
+             write yet (the campaign's corpus is 10.03M tokens against a 12M
+             floor, so round 1 is always flat). Failing on it aborted the run
+             exactly one round too early.
+    """
+    if done:
+        return "done"
+    if after < before:
+        return "shrink"
+    if after == before:
+        return "flat"
+    return "grew"
+
+
+def require_rig(a):
+    """Refuse unless the rig that opened this tap is up and serving right now.
+
+    Three facts, in order: the runner wrote its marker into the tap dir, the
+    marker names the container this pusher targets, and that container is
+    running. A marker without a live container is a boot that died; a live
+    container without a marker is some other engine on the port — on 2026-10-29
+    it was restored production, which took four rounds of 10M tokens before the
+    old driver admitted it had recorded nothing.
+    """
+    marker = os.path.join(a.tap_dir, a.tap_marker)
+    if not os.path.isfile(marker):
+        qc.refuse(f"no rig marker at {marker}: this tap was not opened by "
+                  "run-qwen38-bf16.sh calib, so a push here would feed whatever "
+                  "answers the port and record nothing")
+    try:
+        lines = open(marker, encoding="utf-8", errors="replace").read().splitlines()
+    except OSError as exc:
+        qc.refuse(f"cannot read the rig marker {marker}: {exc}")
+    meta = dict(line.split("=", 1) for line in lines if "=" in line)
+    who = meta.get("container", "")
+    if who != a.container:
+        qc.refuse(f"the rig marker names container {who!r} and this pusher targets "
+                  f"{a.container!r} — wrong tap, or a stale marker")
+    try:
+        p = subprocess.run(["podman", "inspect", "-f", "{{.State.Running}}", a.container],
+                           capture_output=True, text=True, timeout=30)
+    except Exception as exc:
+        qc.refuse(f"cannot inspect the rig container {a.container}: {exc}")
+    if p.returncode != 0 or p.stdout.strip() != "true":
+        qc.refuse(f"the rig container {a.container} is not running — whatever is "
+                  "answering the port, it is not the calibration rig")
+
+
+def parse_args():
     ap = argparse.ArgumentParser()
     ap.add_argument("--port", type=int, default=7770)
     ap.add_argument("--parallel", type=int, default=64)
     ap.add_argument("--container", default="arbench")
+    ap.add_argument("--model", default="frank-archon")
+    ap.add_argument("--tap-dir", default="/home/dank/campaign/qwen38-w5nl64/calib")
+    ap.add_argument("--tap-marker", default=".rig-boot",
+                    help="the file run-qwen38-bf16.sh calib writes into the tap dir; without it the pusher refuses, because an unmarked tap is not a rig")
     ap.add_argument("--corpus", default="/home/dank/corpora/out/calib.jsonl")
     ap.add_argument("--max-rounds", type=int, default=4)
-    a = ap.parse_args()
+    return ap.parse_args()
-    url = f"http://127.0.0.1:{a.port}/v1/completions"
 
+
+def load_prompts(a) -> list[str]:
+    if not os.path.isfile(a.corpus):
+        qc.refuse(f"no corpus at {a.corpus}")
     prompts = []
-    with open(a.corpus) as f:
+    for lineno, rec in qc.iter_jsonl(a.corpus):
-        for line in f:
+        if not isinstance(rec.get("prompt"), str):
-            line = line.strip()
+            qc.refuse(f"{a.corpus}:{lineno} has no string 'prompt' field")
-            if line:
+        prompts.append(rec["prompt"])
-                prompts.append(json.loads(line)["prompt"])
+    if not prompts:
-    print(f"corpus: {len(prompts)} prompts", flush=True)
+        qc.refuse(f"the corpus at {a.corpus} holds no prompts")
+    return prompts
+
+
+def main():
+    a = parse_args()
+    url = f"http://127.0.0.1:{a.port}/v1/completions"
+    prompts = load_prompts(a)
+    require_rig(a)
+    chars = sum(len(p) for p in prompts)
+    print(f"corpus: {len(prompts)} prompts, ~{chars // 4} tokens (chars/4)",
+          flush=True)
 
     for rnd in range(1, a.max_rounds + 1):
         if tap_done(a.container):
             break
+        before = tap_bytes(a.tap_dir)
         t0 = time.time()
-        ok = 0
+        ok = push_round(url, prompts, a.model, a.parallel)
-        with concurrent.futures.ThreadPoolExecutor(a.parallel) as ex:
+        dt = max(time.time() - t0, 1e-9)
-            for good in ex.map(lambda p: push_one(url, p), prompts):
-                ok += 1 if good else 0
-        dt = time.time() - t0
         print(f"round {rnd}: {ok}/{len(prompts)} ok in {dt:.0f}s "
-              f"({sum(len(p) for p in prompts)/4/dt:.0f} chars/s est)",
+              f"({chars / 4 / dt:.0f} tokens/s est)", flush=True)
-              flush=True)
+        if ok == 0:
+            print("FAIL: not one prompt was accepted; fix the engine or the argv "
+                  "before spending another round", flush=True)
+            sys.exit(1)
         if tap_done(a.container):
             break
+        after = tap_bytes(a.tap_dir)
+        verdict = round_verdict(before, after, False)
+        if verdict == "shrink":
+            print(f"FAIL: the tap at {a.tap_dir} LOST bytes this round "
+                  f"({before} B before, {after} B after) — it was reset or replaced "
+                  "under us", flush=True)
+            sys.exit(1)
+        if verdict == "flat":
+            print(f"round {rnd}: the tap sealed nothing yet (byte floor not reached; "
+                  f"{before} B) — continuing", flush=True)
     else:
         print("max rounds exhausted, tap not done", flush=True)
         sys.exit(1)

scripts/census-frank-dragoon.sh

@@ -21,7 +21,7 @@
 # deployed knob has since dropped to 262144.
 set -euo pipefail
 
-IMG=ghcr.io/peonist-ai/halogen-flash-server@sha256:0c61bf84ac22308a53f5d1ca6b86806702d7039e5ebc51cae4c66621b92fe04a
+IMG=ghcr.io/peonist-ai/halogen-flash-server@sha256:e35133b206da3094cc4d70c78800b26235bd9124b9aeb0e029d4a7c153b9cff0
 CUR=/home/dank/models/halogen-flashnext-v2
 NEW=/home/dank/models/halogen-v2-abliterated
 CUR_CK=qwen38-flash-next-v2-swift15-abliterated.hgn

scripts/convert-qwen38-w5nl64.sh

@@ -47,7 +47,12 @@
 set -u -o pipefail
 
 SELF="$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)"
-IMAGE=${IMAGE:-stilldeadcode/radiance@sha256:d590e300234bf6d2f942a0b8d456d183153fd808f28820e6c2fa69c9389178ed}
+# THE TOOLCHAIN PIN. `:latest` on 2026-10-30 is v1.14.0 (revision 4a61d7296bc5),
+# the same image the tap producer boots: a tap must be converted by the line that
+# is current, and rad-convert's `gptq:calib=<dir>` input contract is version-stable
+# (RADGRAM2 + gram.r<rank>.<stem>.bin), so nothing else about the recipe moves with
+# it. Production stays on its own pin — this is the campaign's toolchain.
+IMAGE=${IMAGE:-stilldeadcode/radiance@sha256:69add58f7de1ed2bca2b1e20a4cc75717557df3f73569a178eca5aafe8c44093}
 CKPT=${CKPT:-/home/dank/models/Qwen3.8-Flash-Next}
 CAMPAIGN=${CAMPAIGN:-/home/dank/campaign/qwen38-w5nl64}
 RECIPE=${RECIPE:-$SELF/recipes/qwen4exp-qwen38-w5nl64-bf16trunk.recipe}
@@ -71,10 +76,12 @@ PLAN_ONLY=0
 KEEP_RECIPE=0
 while [ "$#" -gt 0 ]; do
     case "$1" in
-      --arm) ARMS+=("$2"); shift 2 ;;
+      --arm) [ "$#" -ge 2 ] || { echo "--arm needs a value (cd3|cd6|ao1)" >&2; exit 2; }
+             ARMS+=("$2"); shift 2 ;;
       --dry-run) DRY=1; shift ;;
       --plan-only) PLAN_ONLY=1; shift ;;
-      --target-count) TARGET_COUNT="$2"; shift 2 ;;
+      --target-count) [ "$#" -ge 2 ] || { echo "--target-count needs a value" >&2; exit 2; }
+             TARGET_COUNT="$2"; shift 2 ;;
       --keep-recipe) KEEP_RECIPE=1; shift ;;
       *) echo "unknown argument: $1" >&2; exit 2 ;;
     esac
@@ -124,6 +131,13 @@ fi
 for a in "${ARMS[@]}"; do
     [ -e "$OUTDIR/qwen38-w5nl64-$a.rad" ] && {
         echo "$OUTDIR/qwen38-w5nl64-$a.rad already exists; move it to trash first" >&2; exit 2; }
+    # A killed run leaves its container behind (--rm only fires when podman itself
+    # exits), and the next arm then dies on "name already in use" AFTER the plan
+    # phase; refuse here, where the fix is one `podman rm -f`.
+    podman container exists "qwen38-convert-$a" && {
+        echo "container qwen38-convert-$a still exists (a killed run?);" \
+             "'podman rm -f qwen38-convert-$a' and re-run" >&2
+        exit 2; }
 done
 
 # --- the protected plane, spliced into a per-run copy of the template --------------------------
@@ -203,6 +217,7 @@ for arm in "${ARMS[@]}"; do
     rc=0; run_convert "$arm" || rc=$?
     [ "$rc" -eq 0 ] || { echo "arm $arm: rad-convert rc=$rc; log: $STATE/convert-$arm-$TS.log" >&2; overall=1; continue; }
     out="$OUTDIR/qwen38-w5nl64-$arm.rad"
+    [ -f "$out" ] || { echo "arm $arm: rad-convert exited 0 but wrote no $out; log: $STATE/convert-$arm-$TS.log" >&2; overall=1; continue; }
     got_mib=$(( $(stat -c %s "$out") / 1048576 ))
     # The receipt: the artifact is what the plan predicted, within the metadata the plan does not
     # count. An arm that came out materially smaller lost a plane somewhere.

scripts/deploy-remote.sh

@@ -1362,11 +1362,17 @@ deploy_authentik() {
     # generated". Boot persistence comes from [Install] WantedBy=default.target
     # inside each .container file, which the generator honors at user-manager
     # start (linger must stay on for the deploy user).
-    systemctl --user start authentik-postgres.service
+    # RESTART, not start: `start` on an already-active unit is a no-op, so an
+    # image bump would install the new digest and keep serving the old
+    # container while every gate stayed green (measured 2026-10-30: server and
+    # worker stayed on the previous release beside an outpost that moved,
+    # because only the outpost handler restarts). The generator has already
+    # re-rendered the unit from the .container, so a restart is what applies it.
+    systemctl --user restart authentik-postgres.service
     wait_container_healthy authentik-postgres 120 || return 1
-    systemctl --user start authentik-server.service
+    systemctl --user restart authentik-server.service
     wait_container_healthy authentik-server 300 || return 1
-    systemctl --user start authentik-worker.service
+    systemctl --user restart authentik-worker.service
 }
 
 deploy_authentik_outpost() {
@@ -1482,7 +1488,7 @@ deploy_anubis() {
 # artifact would leave a restart-looping unit and a dark endpoint. The
 # port gate is the same as every cutover since 2026-09-16: nothing
 # starts until 7770 is released.
-LLM_DRAGOON_IMAGE="${LLM_DRAGOON_IMAGE:-ghcr.io/peonist-ai/halogen-flash-server@sha256:0c61bf84ac22308a53f5d1ca6b86806702d7039e5ebc51cae4c66621b92fe04a}"
+LLM_DRAGOON_IMAGE="${LLM_DRAGOON_IMAGE:-ghcr.io/peonist-ai/halogen-flash-server@sha256:e35133b206da3094cc4d70c78800b26235bd9124b9aeb0e029d4a7c153b9cff0}"
 # The payload dir is its own twin because the NPU lane's containment gate below
 # reads it; every artifact path under it stays a full literal as usual.
 LLM_DRAGOON_MODEL_DIR="${LLM_DRAGOON_MODEL_DIR:-/home/dank/models/halogen-v2-abliterated}"

scripts/download-frank-dragoon-npu.sh

@@ -8,8 +8,10 @@
 # (/opt/halogen/npu/models.txt inside the image) and refuses to load a model
 # whose files do not match.
 #
-# The manifest below IS that catalog block (dumped from the pinned 0.16.2
+# The manifest below IS that catalog block (dumped from the pinned 0.18.20
-# image on 2026-10-19), so the files land in the layout the entrypoint
+# image on 2026-10-30; the 0.16.2 block this replaced named different device
+# programs — 10 `u*.elf` and a 5,086,496 B consts.hnpb — so a version bump
+# re-fetches the set or the entrypoint refuses the boot), so the files land in the layout the entrypoint
 # expects — <dir>/<model-id>/devices/*, <dir>/<model-id>/<model-id>.hnpw and
 # <dir>/<model-id>/tokenizer/tokenizer.json — and the endpoint can keep
 # /models read-only and open no outbound connection at boot (HALOGEN_DOWNLOAD
@@ -57,21 +59,25 @@ manifest() {
     case "$1" in
       qwen3-embedding-0.6b)
         NPU_REPO="peonist-ai/halogen-npu-qwen3-embedding-0.6b"
-        NPU_REV="4960fd9d4423dd15373ad21c83db48dbdb3b2798"
+        NPU_REV="8fd44c9f18861618a509bea75c51ff8a37e3bd5d"
         FILES=(
         "devices/NOTICE.md 3248 0f267b3d2523ddf14997ee58660a492a9eaf10ccf6b08bf19dfdc85b9a478b91"
-        "devices/consts.hnpb 5086496 ae430e41545a0b1012938cda373b0c450364d3be631111cca3a8c43679f93336"
+        "devices/consts.hnpb 8495512 5af7c1f2f24d45a9054872bba882461cf7f6ffda832e785b700643d6467ad88b"
-        "devices/devices.hnpm 24047 4e9bca9d95d084fb89ce5b99f579a00666c8ca1efae2613c53e3a4878cc9f174"
+        "devices/devices.hnpm 35465 85d870f6b900de1829659955f0dc701d57984580f5e5ec251add5d6969f03e3d"
-        "devices/u0.elf 4931968 b38f165886cb037b7d3951d9cb878473eaba58a432acd45416ff85fd88aa4e46"
+        "devices/u0.elf 2975008 f74df12eb8c79e4e90cbedcc1dd8f142219772d400e3a142fde8a95378b105eb"
-        "devices/u1.elf 5456000 b1811feee851b8af833b37721da92099d89772110815bbbe2c3fa1c276f0ff42"
+        "devices/u1.elf 3266848 ac5a3f89f934951b315b62ab53e1d0a74dbe6396134192641c405d08da332754"
-        "devices/u2.elf 6505552 406ea9a95b470628a7382e376f4a72cdf4d7aa4c59b4ddf0a4bdd139d63d6ff2"
+        "devices/u2.elf 4015328 5e4f0d3342f262816289866d7e8c966f56a9f938c2e12e7341718cc7312aa480"
-        "devices/u3.elf 6817216 bc205d51b523d770151a0c61968f7c70bf624412827580bb6fe0aaf4650dc2ee"
+        "devices/u3.elf 4326976 94cf5d269fa286d91c26bf4654cedf101c047d23c4495b74c9c129cdc33b210f"
-        "devices/u4.elf 7444032 a5a91338a2603acdcd5c11df532033fda2d3f5479ea44ae87f98fafbdff9a62d"
+        "devices/u4.elf 4953792 cbd2796050d4c675f5cb3da117111087d510173e26f262f1b36aab66dd49dc91"
-        "devices/u5.elf 7432416 6c2bf41835569e0eba25ded7c05c7264ae69c6700bdeb4791057d884f2a4c126"
+        "devices/u5.elf 4940112 7bf6a4470546bb888da02af395c07d92a24fb6d43aef27eb2151b3a9eda19e4d"
-        "devices/u6.elf 4941232 919d0038ca088eafddf195f3466ff38db27d3635be1cffa5d4d4b5140902a68e"
+        "devices/u6.elf 6174544 10b9d50b635993e1f9c23775c0b94901d922c1e433cd10910e9f97174235f5ac"
-        "devices/u7.elf 5488304 d904cf8afae9428dd167960549e18cce5441745f18c62f8c50975dffc077d3c2"
+        "devices/u7.elf 2983776 e0b06063b7379d05c85b3f435b819d96fda1223d8025d6d15958854b5711ace9"
-        "devices/u8.elf 4941232 db3aef69b17825854ee0dd2df475125a3f5c872f95c686793d6af1ee83596fa7"
+        "devices/u8.elf 3298656 3bad204f782a68b97e27c1172530b13d7fbb0b495fa95f9504530a1048560644"
-        "devices/u9.elf 5488304 d182727e551084db8fb9600441d8aa373c7f09a3d267ae0a5e921ef92326db1c"
+        "devices/u9.elf 2983776 875e313280358ce116e2a51a29a2c42794b5a9fadb3d48db702c90d18d838c2c"
+        "devices/u10.elf 3298656 ef2e0f399f149a5476955bcf233c6fa2e896fcef1f86ecb0d8113c952c41a8b2"
+        "devices/u11.elf 3298656 68a2519eee2d6a61ccc4ca2007e3186d44c9eb4e81162d742fe2f1756ec5a528"
+        "devices/u12.elf 3298656 2bdc6abfa58cac1d5795a3ef52fe8a9ab4e7c2a4792822d615499a913401f728"
+        "devices/u13.elf 3298656 4a3ce14f7ba15987b9bcbb4a010083e8aef68cd87c20cbe43403a98263179fe2"
         "qwen3-embedding-0.6b.hnpw 813032248 b9f9901429df7b46ec17153aa3899f738adbea8e4356080da0c6e6586fb750d1"
         "tokenizer/tokenizer.json 11423705 def76fb086971c7867b829c23a26261e38d9d74e02139253b38aeb9df8b4b50a"
         ) ;;
@@ -99,7 +105,18 @@ for MODEL in ${MODELS//,/ }; do
                 # 62 GiB fetch hit this on 2026-10-18).
                 url=$(curl -sfIL -o /dev/null -w '%{url_effective}' "$HF/$NPU_REPO/resolve/$NPU_REV/$f") \
                     || { echo "cannot resolve $NPU_REPO/$f" >&2; exit 1; }
-                curl -fL --retry 5 -C - -o "$path" "$url"
+                # Resume ONLY into a genuinely partial file. A staged file of a
+                # DIFFERENT size is an artifact of another revision (a version
+                # bump re-cuts the device programs), and `-C -` against it
+                # appends the tail to the old bytes: the result is the right
+                # BYTE COUNT with the wrong content, which reads as a corrupt
+                # program, not as a fetch fault (measured 2026-10-30 on
+                # consts.hnpb). Fetch those from zero.
+                resume=()
+                if [ -f "$path" ] && [ "$(stat -c %s "$path")" -lt "$want" ]; then
+                    resume=(-C -)
+                fi
+                curl -fL --retry 5 "${resume[@]+"${resume[@]}"}" -o "$path" "$url"
             fi
         fi
         [ -f "$path" ] || { echo "MISSING $path (run without --verify-only)" >&2; exit 1; }

scripts/frank-dragoon.env

@@ -10,8 +10,11 @@
 # The .hgn format is the lock: it loads ONLY in the flash-server line, so the
 # engine moves with the payload, never ahead of it.
 # Image digest-pinned (a tag pull silently re-points; the digest is what was
-# qualified — 0.16.2 is the version the model card names as its quickstart pin).
+# qualified). 0.18.20 since 2026-10-30 (was 0.16.2, the model card's quickstart
-FRANK_DRAGOON_IMAGE=ghcr.io/peonist-ai/halogen-flash-server@sha256:0c61bf84ac22308a53f5d1ca6b86806702d7039e5ebc51cae4c66621b92fe04a
+# pin) — the vendor has moved two minor versions and the payload lock is what
+# the deploy's health gate plus the boot ledger decide; rollback is the
+# previous digest, 0c61bf84…, on this same line.
+FRANK_DRAGOON_IMAGE=ghcr.io/peonist-ai/halogen-flash-server@sha256:e35133b206da3094cc4d70c78800b26235bd9124b9aeb0e029d4a7c153b9cff0
 # Container identity kept from the 27B era: the unit's ExecStop names it.
 FRANK_DRAGOON_CONTAINER=frank-dragoon-halogen
 # Predecessor identity the serve script clears if it is still holding the GPUs.

scripts/frank-zealot.env

@@ -1,12 +1,12 @@
-# frank-zealot — llama.cpp (PrismML fork build 10777, CUDA) on the RTX 3090,
+# frank-zealot — llama.cpp (PrismML fork build 12118 = 0.7.0-dev, commit
-# :7771. Payload: Ternary-Bonsai-2-9B PQ2_0 with the MTP head in the file, plus
+# 024474448, CUDA) on the RTX 3090, :7771. Payload: Ternary-Bonsai-2-9B PQ2_0 with the MTP head in the file, plus
 # the refusal-direction adapter applied at load.
 #
 # ONE OWNER: shape, payload identity, eras, laws and every measured number
 # live in serverAppConfigs-docs/LLM/frank-zealot.md. This file carries the
 # knobs and only the reason a reader editing THIS line cannot recover
 # elsewhere. Where the two disagree, this file wins on values.
-LLAMA_BIN=/home/dank/GitPackages/llama-prism-b10777/llama-server
+LLAMA_BIN=/home/dank/GitPackages/llama-prism-0244744/bin/llama-server
 # The fork is a LOCK, not a preference: custom tensor type 142 + prism.hadamard.*
 # metadata. Stock llama.cpp refuses the file; a loader ignoring the Hadamard
 # metadata dequantizes wrong, silently.
@@ -64,7 +64,8 @@ CACHE_TYPE_V=q8_0
 # weights, so there is no --spec-draft-model. Its bill is pool × 4,096 B of
 # draft KV plus ~0.45 GiB of buffers and a tripled recurrent state — paid for
 # by CACHE_TYPE_V=q8_0 and UBATCH_SIZE=1024 above. Build gate: quantized V
-# beside MTP needs fork build >= b10776; LLAMA_BIN is 10777.
+# beside MTP needs fork build >= b10776; LLAMA_BIN is 12118 (b10777 is kept on
+# disk as the rollback path).
 # SPEC_TYPE=none is the rollback and strips the WHOLE set (--spec-type and
 # every --spec-draft-*); half a set has segfaulted at load.
 SPEC_TYPE=draft-mtp

scripts/kld-gate-qwen38.sh

@@ -1,44 +1,53 @@
 #!/usr/bin/env bash
 # kld-gate-qwen38.sh — RUNS ON DANKSERVER (or anywhere the reports are). The campaign's
-# acceptance gate: three KLD reports against ONE reference recording, one verdict.
+# acceptance gate: one verdict from the KLD reports of the containers that ride one reference
+# recording.
 #
 # The reference is the BF16 container built from the vendor checkpoint (run-qwen38-bf16.sh
 # kldref, in the same drained window that records the calibration grams). Every scored
 # container is measured against that one recording, so the KL scale runs from 0 = the
-# checkpoint itself downward in faithfulness. Three containers ride it:
+# checkpoint itself downward in faithfulness.
 #
-#   fp8.json    the published FP8-native release   — the achievable end of the interval
+# TWO MODES, because the disk decides how many anchors can exist at once:
-#   iq4r.json   the payload frank-archon serves    — the bar to beat
-#   cand.json   the arm under test                 — the subject
 #
-# The FP8 number is what makes the interval absolute instead of relative: "half the gap"
+#   gate IQ4R.json CAND.json                    candidate vs the served payload
-# is half of the distance between the 4-bit payload and the near-lossless release, not half
+#   gate FP8.json IQ4R.json CAND.json           the same, inside the absolute interval
-# of an invented distance to the checkpoint. That is also why the FP8 payload is FETCHED and
-# scored, never converted: it is a second opinion about what this checkpoint tolerates, not a
-# step of the campaign. It lands LAST (disk sequencing) — by then the 335 GiB bf16 container
-# is gone, so its 172.8 GiB fits.
 #
-# The law, in order of severity, on the mean KL of one slice (--slice all | a corpus source):
+# iq4r.json is the payload frank-archon serves — the bar to beat. When the FP8-native release
-#   1. KL(fp8) < KL(iq4r)                 the anchors are ordered. If the near-lossless
+# (172.8 GiB, fetched last because of the peak-disk law) is on disk, pass its report first:
-#                                         release does NOT sit closer to the checkpoint than
+# "half the gap" is then half of the distance between the 4-bit payload and the near-lossless
-#                                         the 4-bit payload, the corpus, the tokenizer or the
+# release, not half of an invented distance to the checkpoint. Without it the gate still
-#                                         boot shape is wrong and no arm number means anything.
+# answers the campaign's question — does the candidate beat what is served — and says so.
-#   2. KL(fp8) < KL(cand) < KL(iq4r)      the candidate lies strictly inside the interval —
+#
-#                                         outside it, something is mis-measured, not better
+# The law, on the summary of one slice (--slice all | a corpus source):
-#   3. KL(cand) <= KL(iq4r) - R x gap     it closes at least R (--recover, default 0.5) of the
+#
-#                                         gap = KL(iq4r) - KL(fp8): at least half the way from
+#   candidate vs served (2 reports)
-#                                         what is served toward the release (agreed 2026-10-27)
+#   1. mean KL(cand) < mean KL(iq4r)         the served payload is beaten on the corpus
-#   4. top-1(cand) >= top-1(iq4r)         the agreement the harness keys on does not fall
+#   2. p99.9(cand) <= p99.9(iq4r)            and the tail is no worse — a mean can hide a few
+#                                            positions taking the whole hit
+#   3. top-1(cand) >= top-1(iq4r)            the agreement the harness keys on does not fall
+#   4. every source slice passes 1-3         a win bought by regressing one distribution
+#                                            (code, prose, tools) is not a win
+#
+#   inside the interval (3 reports), laws 1-4 above (1-3 renumbered to the interval and the
+#   recovery fraction, as agreed 2026-10-27) plus:
+#   5. p99.9(cand) <= p99.9(iq4r)            the tail law
+#   6. every source slice passes            the slice law
 #
 # Booting the containers is NOT this script's job: `run-qwen38-bf16.sh score RAD OUT.json`
 # writes the reports and owns the single engine argv (same pin, --tp 2, --kv-cache-dtype fp8,
-# same step size, --deterministic, no speculation) that makes the three numbers comparable.
+# same step size, --deterministic, no speculation) that makes the numbers comparable.
 # This script is arithmetic, so it re-runs against old reports and is testable without a GPU.
 #
-# Usage: kld-gate-qwen38.sh FP8.json IQ4R.json CAND.json [--slice NAME] [--recover F]
+# Usage: kld-gate-qwen38.sh IQ4R.json CAND.json [--slice NAME]
+#        kld-gate-qwen38.sh FP8.json IQ4R.json CAND.json [--slice NAME] [--recover F]
 set -uo pipefail
 
-[ "$#" -ge 3 ] || { echo "usage: kld-gate-qwen38.sh FP8.json IQ4R.json CAND.json [--slice NAME] [--recover F]" >&2; exit 2; }
+if [ "$#" -lt 2 ]; then
+    echo "usage: kld-gate-qwen38.sh IQ4R.json CAND.json [--slice NAME]" >&2
+    echo "       kld-gate-qwen38.sh FP8.json IQ4R.json CAND.json [--slice NAME] [--recover F]" >&2
+    exit 2
+fi
 python3 - "$@" <<'PY'
 import json, sys
 
@@ -52,13 +61,15 @@ while i < len(a):
     else:
         files.append(a[i])
         i += 1
-if len(files) != 3:
+if len(files) not in (2, 3):
-    sys.exit("usage: kld-gate-qwen38.sh FP8.json IQ4R.json CAND.json [--slice NAME] [--recover F]")
+    sys.exit("usage: kld-gate-qwen38.sh IQ4R.json CAND.json [--slice NAME]\n"
+             "       kld-gate-qwen38.sh FP8.json IQ4R.json CAND.json [--slice NAME] [--recover F]")
 o = dict(zip(opts[0::2], opts[1::2]))
 slice_ = o.get("--slice", "all")
 recover = float(o.get("--recover", 0.5))
 if not 0.0 < recover <= 1.0:
     sys.exit(f"--recover must be in (0,1], got {recover}")
+interval = len(files) == 3
 
 
 def die(msg):
@@ -66,11 +77,15 @@ def die(msg):
     raise SystemExit(2)
 
 
-def read(path):
+def load(path):
     try:
-        d = json.load(open(path))
+        return json.load(open(path))
     except Exception as e:
         die(f"cannot read the KLD report {path}: {e}")
+
+
+def summary(path, d):
+    """The slice this gate grades: 'all', or one corpus source."""
     if slice_ == "all":
         g = d.get("all")
     else:
@@ -80,23 +95,25 @@ def read(path):
                 f"{', '.join(sorted(d.get('by_source') or {})) or '(none)'}")
     if g is None or "kld" not in g or "top1_agreement" not in g:
         die(f"{path} has no {slice_!r} summary — wrong report version?")
-    return d, g
+    return g
 
 
+docs = {label: load(path) for label, path in
+        zip(("fp8", "iq4r", "cand") if interval else ("iq4r", "cand"), files)}
 rows = []
-for label, path in zip(("fp8", "iq4r", "cand"), files):
+for label, path in zip(("fp8", "iq4r", "cand") if interval else ("iq4r", "cand"), files):
-    d, g = read(path)
+    g = summary(path, docs[label])
-    rows.append((label, path, d.get("model", "?"), g["kld"]["mean"], g["kld"]["p99_9"],
+    rows.append((label, path, docs[label].get("model", "?"), g["kld"]["mean"],
-                 g["top1_agreement"], g["positions"]))
+                 g["kld"]["p99_9"], g["top1_agreement"], g["positions"]))
 
-print(f"reference: BF16 container; slice: {slice_}")
+print(f"reference: BF16 container; slice: {slice_}; "
+      + ("inside the FP8 interval" if interval else "candidate vs served (no FP8 anchor)"))
 print(f"{'':6} {'model':38} {'KL mean':>10} {'KL p99.9':>10} {'top-1':>8}  positions")
 for label, path, model, kl, p999, top1, n in rows:
     print(f"{label:6} {str(model)[:38]:38} {kl:10.6f} {p999:10.4f} {100*top1:7.2f}%  {n}")
-r = {label: (kl, top1) for label, _p, _m, kl, _q, top1, _n in rows}
+r = {label: (kl, p999, top1) for label, _p, _m, kl, p999, top1, _n in rows}
-kl_fp8, kl_iq4r, kl_cand = r["fp8"][0], r["iq4r"][0], r["cand"][0]
+kl_cand, p_cand, t_cand = r["cand"]
-t_iq4r, t_cand = r["iq4r"][1], r["cand"][1]
+kl_iq4r, p_iq4r, t_iq4r = r["iq4r"]
-gap = kl_iq4r - kl_fp8
 ok = True
 
 
@@ -106,17 +123,64 @@ def law(n, text, passed):
     print(f"  [{'PASS' if passed else 'FAIL'}] {n}. {text}")
 
 
-law(1, f"anchors ordered: fp8 {kl_fp8:.6f} < iq4r {kl_iq4r:.6f} (gap {gap:.6f})",
+def tail_law(n):
-    gap > 0)
+    law(n, f"candidate KL p99.9 {p_cand:.4f} <= served iq4r {p_iq4r:.4f}",
-law(2, f"candidate inside the interval: {kl_fp8:.6f} < {kl_cand:.6f} < {kl_iq4r:.6f}",
+        p_cand <= p_iq4r)
-    kl_fp8 < kl_cand < kl_iq4r)
+
-need = kl_iq4r - recover * gap
+
-law(3, f"candidate closes {100*max((kl_iq4r - kl_cand)/gap, 0):.1f}% of the "
+# --- the slice law: every corpus source must pass the metric laws -------------
-       f"{gap:.6f} gap (needs {100*recover:.0f}%, i.e. KL <= {need:.6f})"
+def slice_law(n):
-       if gap > 0 else "gap is not positive; cannot score recovery",
+    """A win that regresses one source class is not a win: check them all."""
-    gap > 0 and kl_cand <= need)
+    mine = docs["cand"].get("by_source") or {}
-law(4, f"candidate top-1 {100*t_cand:.2f}% >= served iq4r top-1 {100*t_iq4r:.2f}%",
+    theirs = docs["iq4r"].get("by_source") or {}
-    t_cand >= t_iq4r)
+    if not mine or not theirs:
+        law(n, "no per-source summaries to check", True)
+        return
+    missing = sorted(set(theirs) - set(mine))
+    extra = sorted(set(mine) - set(theirs))
+    if missing or extra:
+        law(n, "same source slices in both reports"
+            + (f" (missing from cand: {', '.join(missing)})" if missing else "")
+            + (f" (only in cand: {', '.join(extra)})" if extra else ""), False)
+        return
+    bad = []
+    for src in sorted(theirs):
+        c, q = mine[src], theirs[src]
+        worse_mean = c["kld"]["mean"] >= q["kld"]["mean"]
+        worse_tail = c["kld"]["p99_9"] > q["kld"]["p99_9"]
+        worse_top1 = c["top1_agreement"] < q["top1_agreement"]
+        if worse_mean or worse_tail or worse_top1:
+            why = ",".join(m for m, w in (("mean", worse_mean), ("p99.9", worse_tail),
+                                          ("top-1", worse_top1)) if w)
+            bad.append(f"{src} ({why})")
+    law(n, "every source slice passes (mean, p99.9, top-1)"
+        + (f" — regressed: {'; '.join(bad)}" if bad else ""), not bad)
+
+
+if interval:
+    kl_fp8 = r["fp8"][0]
+    gap = kl_iq4r - kl_fp8
+    law(1, f"anchors ordered: fp8 {kl_fp8:.6f} < iq4r {kl_iq4r:.6f} (gap {gap:.6f})",
+        gap > 0)
+    law(2, f"candidate inside the interval: {kl_fp8:.6f} < {kl_cand:.6f} < {kl_iq4r:.6f}",
+        kl_fp8 < kl_cand < kl_iq4r)
+    need = kl_iq4r - recover * gap
+    law(3, f"candidate closes {100*max((kl_iq4r - kl_cand)/gap, 0):.1f}% of the "
+           f"{gap:.6f} gap (needs {100*recover:.0f}%, i.e. KL <= {need:.6f})"
+           if gap > 0 else "gap is not positive; cannot score recovery",
+        gap > 0 and kl_cand <= need)
+    law(4, f"candidate top-1 {100*t_cand:.2f}% >= served iq4r top-1 {100*t_iq4r:.2f}%",
+        t_cand >= t_iq4r)
+    tail_law(5)
+    slice_law(6)
+else:
+    law(1, f"candidate beats the served payload: KL {kl_cand:.6f} < iq4r {kl_iq4r:.6f}",
+        kl_cand < kl_iq4r)
+    tail_law(2)
+    law(3, f"candidate top-1 {100*t_cand:.2f}% >= served iq4r top-1 {100*t_iq4r:.2f}%",
+        t_cand >= t_iq4r)
+    slice_law(4)
+
 print("KLD GATE " + ("PASS" if ok else "FAIL"))
 sys.exit(0 if ok else 1)
 PY

scripts/make-calib-corpus.py

@@ -34,14 +34,25 @@ import json
 import os
 import random
 import re
+import sys
 
 from jinja2 import Environment, FileSystemLoader
-from tokenizers import Tokenizer
+
+_HERE = os.path.dirname(os.path.abspath(__file__)) if "__file__" in globals() else os.getcwd()
+if _HERE not in sys.path:
+    sys.path.insert(0, _HERE)
+
+import qwen38_common as qc  # noqa: E402 - sibling module, same directory
 
 CODE_EXT = (".sh", ".py", ".jinja", ".recipe", ".env", ".example",
             ".yaml", ".yml", ".bats", ".service", ".container", ".toml")
 JSONL_EXTS = (".jsonl", ".json")
 DOC_DIR = "serverAppConfigs-docs"
+FRAGMENT_TOKENS = 2400   # what one fragment may contribute
+CLASS_MIX = {"code": .45, "prose": .30, "tools": .25}
+HF_CAP = {"code": 40000, "prose": 60000, "tools": 0}
+POOL_CYCLES = 40         # how many times a class pool may repeat before giving up
+KLD_GUARD = 5000         # rendered conversations tried before declaring the pool short
 
 FLEET_CMDS = [
     "systemctl status frank-archon --no-pager",
@@ -142,7 +153,7 @@ def load_parquet_texts(path: str, kind: str, rng: random.Random,
     return out[:cap]
 
 
-def main() -> None:
+def parse_args() -> argparse.Namespace:
     ap = argparse.ArgumentParser(description=__doc__)
     ap.add_argument("--repo", required=True)
     ap.add_argument("--tokenizer-json", required=True)
@@ -156,22 +167,26 @@ def main() -> None:
     ap.add_argument("--hf-code-parquet", default="")
     ap.add_argument("--hf-prose-parquet", default="")
     ap.add_argument("--seed", type=int, default=20261006)
-    a = ap.parse_args()
+    return ap.parse_args()
 
-    rng = random.Random(a.seed)
+
-    tokenizer = Tokenizer.from_file(a.tokenizer_json)
+def load_template(template: str) -> tuple[Environment, str]:
-    env = Environment(loader=FileSystemLoader(os.path.dirname(a.template)),
+    """The chat template as the engine will render it, plus its short name."""
+    env = Environment(loader=FileSystemLoader(os.path.dirname(template)),
                       trim_blocks=True, lstrip_blocks=True)
     env.filters["tojson"] = lambda v: json.dumps(v, ensure_ascii=False)
     env.filters["raise_exception"] = lambda m: (_ for _ in ()).throw(
         ValueError(m))
-    tname = os.path.basename(a.template)
+    return env, os.path.basename(template)
 
+
+def classify(repo: str) -> dict[str, list[str]]:
+    """Walk the checkout into the three source classes the mix draws from."""
     code, prose, jsonl = [], [], []
-    for root, dirs, files in os.walk(a.repo):
+    for root, dirs, files in os.walk(repo):
         dirs[:] = [d for d in dirs if not d.startswith(".")]
         for f in sorted(files):
-            p = os.path.relpath(os.path.join(root, f), a.repo)
+            p = os.path.relpath(os.path.join(root, f), repo)
             if DOC_DIR in p and f.endswith(".md"):
                 prose.append(p)
             elif p.startswith(DOC_DIR):
@@ -180,137 +195,185 @@ def main() -> None:
                 code.append(p)
             elif f.endswith(JSONL_EXTS) and f != "package-lock.json":
                 jsonl.append(p)
-    # file-level split: bucket 0 -> KLD pool, 1..4 -> calibration pool
+    return {"code": code, "prose": prose, "tools": jsonl}
-    pools: dict[str, tuple[list, list]] = {}
-    for cls, lst in (("code", code), ("prose", prose), ("tools", jsonl)):
-        pools[cls] = ([p for p in lst if fnv(p) % 5 == 0],
-                      [p for p in lst if fnv(p) % 5 != 0])
 
-    def text_of(p: str) -> str:
-        path = os.path.join(a.repo, p)
-        raw = open(path, encoding="utf-8", errors="replace").read()
-        if p.endswith(JSONL_EXTS):
-            lines = [json.loads(l)["content"] for l in raw.splitlines()
-                     if l.strip() and "content" in l]
-            raw = "\n".join(lines)
-        toks = tokenizer.encode(raw, add_special_tokens=False).ids
-        return tokenizer.decode(toks[:2400], skip_special_tokens=True)
 
+def split_pools(classes: dict[str, list[str]]) -> dict[str, tuple[list, list]]:
+    """File-level split: bucket 0 -> KLD pool, everything else -> calibration."""
+    return {cls: ([p for p in lst if fnv(p) % 5 == 0],
+                  [p for p in lst if fnv(p) % 5 != 0])
+            for cls, lst in classes.items()}
+
+
+def jsonl_content(raw: str) -> str:
+    """The content fields of a content-keyed JSONL, or "" for anything else.
+
+    A .json in this repo is not necessarily a content-keyed JSONL: the substring
+    test passes on any line that merely mentions "content", and a list-shaped
+    document raises TypeError — both used to crash the whole build. Skip what
+    does not fit instead.
+    """
+    lines = []
+    for line in raw.splitlines():
+        if not line.strip() or "content" not in line:
+            continue
+        try:
+            rec = json.loads(line)
+        except json.JSONDecodeError:
+            continue
+        if isinstance(rec, dict) and isinstance(rec.get("content"), str):
+            lines.append(rec["content"])
+    return "\n".join(lines)
+
+
+def fragment_text(repo: str, rel: str, tok) -> str:
+    """One file as a fragment: its first FRAGMENT_TOKENS tokens, decoded."""
+    raw = open(os.path.join(repo, rel), encoding="utf-8",
+               errors="replace").read()
+    if rel.endswith(JSONL_EXTS):
+        raw = jsonl_content(raw)
+    ids = tok.encode(raw, add_special_tokens=False).ids
+    return tok.decode(ids[:FRAGMENT_TOKENS], skip_special_tokens=True)
+
+
+def build_frags(a: argparse.Namespace, tok,
+                rng: random.Random) -> dict[str, dict[str, list[str]]]:
+    """Both fragment pools per class, seeded so the same seed rebuilds the corpus."""
+    pools = split_pools(classify(a.repo))
     cmds = harvest_cmds(a.repo, rng)
-    # fragment pools per class, corpus-split first
     frags: dict[str, dict[str, list[str]]] = {}
-    for cls in pools:
+    for cls, (kld_files, cal_files) in pools.items():
-        kld_files, cal_files = pools[cls]
-        fr = {}
-        src = {"kld": kld_files, "cal": cal_files}
         if cls == "tools":
-            fr["kld"] = [tool_call(rng, cmds) for _ in range(600)]
+            frags[cls] = {"kld": [tool_call(rng, cmds) for _ in range(600)],
-            fr["cal"] = [tool_call(rng, cmds) for _ in range(40000)]
+                          "cal": [tool_call(rng, cmds) for _ in range(40000)]}
         else:
-            fr["kld"] = [text_of(p) for p in src["kld"]]
+            frags[cls] = {"kld": [fragment_text(a.repo, p, tok) for p in kld_files],
-            fr["cal"] = [text_of(p) for p in src["cal"]]
+                          "cal": [fragment_text(a.repo, p, tok) for p in cal_files]}
-        frags[cls] = fr
     # per-class fragment caps: prose rows are short (~66 tok) so need many;
     # codeparrot files are long (~2.6k tok) so a small cap suffices
-    hfcap = {"code": 40000, "prose": 60000, "tools": 0}
     for cls, spec in (("code", a.hf_code_parquet),
                       ("prose", a.hf_prose_parquet)):
         for path in [p for p in spec.split(",") if p]:
             frags[cls]["cal"].extend(
-                load_parquet_texts(path, cls, rng, cap=hfcap[cls]))
+                load_parquet_texts(path, cls, rng, cap=HF_CAP[cls]))
     for cls in frags:
         rng.shuffle(frags[cls]["cal"])
+    return frags
 
-    def toks(s: str) -> int:
-        return len(tokenizer.encode(s, add_special_tokens=False).ids)
 
-    def emit_calib() -> tuple[int, list[dict]]:
+def pack_class(cls: str, pool: list[str], want: int, tok,
-        targets = {"code": .45, "prose": .30, "tools": .25}
+               chunk_tokens: int) -> list[dict]:
-        rows: list[dict] = []
+    """Greedily pack `pool` into chunk_tokens-sized calibration prompts."""
-        for cls, frac in targets.items():
+    if not pool:
-            want = int(a.calib_tokens * frac)
+        return []
-            pool = frags[cls]["cal"]
+    rows: list[dict] = []
+    # Count every fragment once: the loop cycles the pool up to POOL_CYCLES
+    # times, and re-tokenizing a 2.4k-token fragment on each pass dominated the
+    # build's wall time.
+    counts = [qc.n_tokens(tok, f) for f in pool]
+    got = 0
+    i = 0
+    buf: list[str] = []
+    nbuf = 0
+    while got < want:
+        j = i % len(pool)
+        frag = pool[j]
+        i += 1
+        n = counts[j]
+        if nbuf + n > chunk_tokens and buf:
+            rows.append({"prompt": "\n".join(buf), "cls": cls})
+            got += nbuf
+            buf, nbuf = [], 0
+            if got >= want:
+                break
+        buf.append(frag)
+        nbuf += n
+        if i > len(pool) * POOL_CYCLES:
+            break  # pool exhausted with repeats; stop
+    if buf and got < want:
+        rows.append({"prompt": "\n".join(buf), "cls": cls})
+        got += nbuf
+    print(f"calib {cls}: {got} tokens, {i} fragments")
+    return rows
+
+
+def emit_calib(a: argparse.Namespace, frags, tok,
+               rng: random.Random) -> tuple[int, list[dict]]:
+    """calib.jsonl rows: the class mix, shuffled, prompts only."""
+    rows: list[dict] = []
+    for cls, frac in CLASS_MIX.items():
+        rows.extend(pack_class(cls, frags[cls]["cal"], int(a.calib_tokens * frac),
+                               tok, a.chunk_tokens))
+    rng.shuffle(rows)
+    return len(rows), [{"prompt": r["prompt"]} for r in rows]
+
+
+def emit_kld(a: argparse.Namespace, frags, tok, rng: random.Random,
+             env: Environment, tname: str) -> tuple[int, list[dict], int]:
+    """kld.jsonl rows: multi-turn conversations, each with a scored tail."""
+    template = env.get_template(tname)
+    rows: list[dict] = []
+    total = scored_total = 0
+    classes = ["code", "prose", "tools"]
+    cursor = {c: 0 for c in classes}
+    guard = 0
+    while total < a.kld_tokens and guard < KLD_GUARD:
+        guard += 1
+        msgs, srcs = [], []
+        for t in range(rng.choice([2, 3, 4])):
+            cls = classes[(len(rows) + t) % 3]
+            pool = frags[cls]["kld"]
             if not pool:
                 continue
-            got, i, buf, nbuf = 0, 0, [], 0
+            frag = pool[cursor[cls] % len(pool)]
-            while got < want:
+            cursor[cls] += 1
-                frag = pool[i % len(pool)]
+            msgs.append({"role": "user",
-                i += 1
+                         "content": "Summarize and explain this "
-                n = toks(frag)
+                                    f"engineering material:\n{frag}"})
-                if nbuf + n > a.chunk_tokens and buf:
+            msgs.append({"role": "assistant",
-                    rows.append({"prompt": "\n".join(buf), "cls": cls})
+                         "content": frag[:4000]})
-                    got += nbuf
+            srcs.append(cls)
-                    buf, nbuf = [], 0
+        rendered = template.render(messages=msgs, add_generation_prompt=False)
-                    if got >= want:
+        n = qc.n_tokens(tok, rendered)
-                        break
+        if n < a.kld_min_prompt:
-                buf.append(frag)
+            continue
-                nbuf += n
+        rows.append({"prompt": rendered,
-                if i > len(pool) * 40:
+                     "score_from": n - a.score_tail,
-                    break  # pool exhausted with repeats; stop
+                     "source": "+".join(sorted(set(srcs)))})
-            if buf and got < want:
+        total += n
-                rows.append({"prompt": "\n".join(buf), "cls": cls})
+        scored_total += a.score_tail
-                got += nbuf
+    return len(rows), rows, scored_total
-            print(f"calib {cls}: {got} tokens, {i} fragments")
-        rng.shuffle(rows)
-        return len(rows), [{k: v for k, v in r.items() if k == "prompt"}
-                           for r in rows]
 
-    def emit_kld() -> tuple[int, list[dict]]:
-        rows, total = [], 0
-        classes = ["code", "prose", "tools"]
-        i = {c: 0 for c in classes}
-        guard = 0
-        while total < a.kld_tokens and guard < 5000:
-            guard += 1
-            msgs, srcs = [], []
-            for t in range(rng.choice([2, 3, 4])):
-                cls = classes[(len(rows) + t) % 3]
-                pool = frags[cls]["kld"]
-                if not pool:
-                    continue
-                frag = pool[i[cls] % len(pool)]
-                i[cls] += 2
-                msgs.append({"role": "user",
-                             "content": "Summarize and explain this "
-                                        f"engineering material:\n{frag}"})
-                msgs.append({"role": "assistant",
-                             "content": frag[:4000]})
-                srcs.append(cls)
-            rendered = env.get_template(tname).render(
-                messages=msgs, add_generation_prompt=False)
-            n = toks(rendered)
-            if n < a.kld_min_prompt:
-                continue
-            rows.append({"prompt": rendered,
-                         "score_from": n - a.score_tail,
-                         "source": "+".join(sorted(set(srcs)))})
-            total += n
-        return len(rows), rows
 
-    os.makedirs(a.out_dir, exist_ok=True)
+def manifest(a: argparse.Namespace, tname: str, nc: int, nk: int,
-    nc, rows = emit_calib()
+             scored: int) -> dict:
-    with open(os.path.join(a.out_dir, "calib.jsonl"), "w") as f:
+    return {
-        for r in rows:
-            f.write(json.dumps(r) + "\n")
-    nk, krows = emit_kld()
-    with open(os.path.join(a.out_dir, "kld.jsonl"), "w") as f:
-        for r in krows:
-            f.write(json.dumps(r) + "\n")
-    scored = sum(len(tokenizer.encode(r["prompt"], add_special_tokens=False).ids)
-                 - r["score_from"] for r in krows)
-    manifest = {
         "seed": a.seed, "template": tname,
         "tokenizer": os.path.basename(a.tokenizer_json),
         "calib_prompts": nc, "kld_prompts": nk,
         "kld_scored_positions": scored,
         "kld_score_tail": a.score_tail,
         "disjoint_file_pools": "local KLD material only; HF code feeds calib",
-        "calib_mix": {"code": .45, "prose": .30, "tools": .25},
+        "calib_mix": dict(CLASS_MIX),
         "hf_code": a.hf_code_parquet, "hf_prose": a.hf_prose_parquet,
     }
-    with open(os.path.join(a.out_dir, "manifest.json"), "w") as f:
+
-        json.dump(manifest, f, indent=1)
+
-    print(json.dumps(manifest, indent=1))
+def main() -> None:
+    a = parse_args()
+    rng = random.Random(a.seed)
+    tok = qc.load_tokenizer(a.tokenizer_json)
+    env, tname = load_template(a.template)
+
+    frags = build_frags(a, tok, rng)
+    os.makedirs(a.out_dir, exist_ok=True)
+    nc, calib_rows = emit_calib(a, frags, tok, rng)
+    qc.write_jsonl(os.path.join(a.out_dir, "calib.jsonl"), calib_rows)
+    nk, kld_rows, scored = emit_kld(a, frags, tok, rng, env, tname)
+    qc.write_jsonl(os.path.join(a.out_dir, "kld.jsonl"), kld_rows)
+    man = manifest(a, tname, nc, nk, scored)
+    qc.write_json(os.path.join(a.out_dir, "manifest.json"), man)
+    print(json.dumps(man, indent=1))
 
 
 if __name__ == "__main__":

scripts/media-apply.py

@@ -0,0 +1,321 @@
+#!/usr/bin/env python3
+"""media-apply.py — the only thing that moves movie-library files.
+
+Reads the plan written by scripts/media-plan.py (rows: `action  src  dst  kind`,
+plus `mkdir` rows) and performs it with rename(2) — bytes are never copied, so
+an interrupted run has destroyed nothing. Every run is resumable and every run
+is undoable: the journal is a write-ahead log, one JSON line per intent, fsynced
+BEFORE the rename it authorises. That ordering is the whole design: a crash
+leaves either an intent whose move happened or an intent whose move never ran,
+and both are recognised on the next run. A file that moved with no journal line
+cannot have been moved by this tool, and saying so is better than guessing.
+
+Three subcommands:
+
+    apply    --plan PLAN [--root ROOT] [--journal J] [--apply]
+             no --apply => report only and write NOTHING, not even folders
+    rollback --journal J [--root ROOT] [--apply]
+             back to the paths the census saw; folders this run created are
+             removed only when empty, never anything else
+    verify   --plan PLAN --journal J [--root ROOT]
+             every planned file is at its target with the journalled byte count
+
+Refusals (exit 2, before the first rename, with nothing touched): a target
+already occupied by a foreign file, a source that has gone, a duplicate target,
+a path outside --root, a journal that was already rolled back, a move that would
+cross a filesystem (rename cannot do it and a silent copy at 24 TB would not).
+
+--force relaxes only the guards that are opinion (journal already rolled back,
+paths outside --root); it never allows overwriting a file.
+
+The journal defaults to `media-relayout.jsonl` BESIDE the library root, never
+inside it: a log inside the library is a file Jellyfin will index.
+"""
+import argparse
+import datetime
+import json
+import os
+import sys
+
+DEFAULT_ROOT = os.environ.get("MEDIA_ROOT", "/mnt/entropy/Movies")
+
+
+class Refuse(Exception):
+    """A gate the operator has to see; every refusal leaves the library alone."""
+
+
+def journal_default(root):
+    return os.path.join(os.path.dirname(os.path.abspath(root)), "media-relayout.jsonl")
+
+
+class Journal:
+    """Append-only JSONL of intents. Writes are fsynced before the tool acts."""
+
+    def __init__(self, path, writing):
+        self.path, self.writing = path, writing
+        self.records = []
+        if os.path.exists(path):
+            for line in open(path):
+                line = line.strip()
+                if line:
+                    self.records.append(json.loads(line))
+        self._fh = None
+
+    def already(self, dst, src):
+        """An intent for this exact (src, dst) was written by an earlier run."""
+        return any(r.get("op") == "move" and r.get("dst") == dst
+                   and r.get("src") == src for r in self.records)
+
+    @property
+    def rolled_back(self):
+        return any(r.get("op") == "rollback" for r in self.records)
+
+    def record(self, **fields):
+        self.records.append(fields)
+        if not self.writing:
+            return
+        if self._fh is None:
+            self._fh = open(self.path, "a")
+        self._fh.write(json.dumps(fields) + "\n")
+        self._fh.flush()
+        os.fsync(self._fh.fileno())
+
+    def close(self):
+        if self._fh:
+            self._fh.close()
+
+
+def read_plan(path, root, force):
+    """Plan rows: (mkdirs, keeps, moves), header dropped, all paths under root."""
+    mkdirs, keeps, moves = [], [], []
+    with open(path) as fh:
+        header = fh.readline()
+        if not header.startswith("action\t"):
+            raise Refuse(f"{path} is not a plan (first column is "
+                         f"{header.split(chr(9))[0]!r}, expected 'action') —"
+                         " regenerate it with scripts/media-plan.py")
+        for line in fh:
+            f = line.rstrip("\n").split("\t")
+            if f[0] == "mkdir":
+                mkdirs.append(f[2])
+            elif f[0] == "keep":
+                keeps.append(f[1])
+            elif f[0] == "move":
+                moves.append((f[1], f[2], f[3] if len(f) > 3 else ""))
+    if not force:
+        for p in mkdirs + keeps + [p for m in moves for p in m[:2]]:
+            if p and not (p == root or p.startswith(root + os.sep)):
+                raise Refuse(f"{p} is outside --root {root}"
+                             " (a plan built for another library?)")
+    return mkdirs, keeps, moves
+
+
+def do_apply(args, root, journal, mkdirs, keeps, moves):
+    """Preflight everything, then folders, then renames. Never a partial start."""
+    problems = []
+    for p in keeps:
+        if not os.path.exists(p):
+            problems.append(f"missing: keep row source is gone: {p}")
+    seen = {}
+    for src, dst, _kind in moves:
+        if src == dst:
+            problems.append(f"no-op: move row whose target is its source: {src}")
+        if dst in seen:
+            problems.append(f"double-claim: {dst} claimed by {seen[dst]} and {src}")
+        seen[dst] = src
+        if journal.already(dst, src) and os.path.exists(dst) and not os.path.exists(src):
+            continue                                    # this intent already ran
+        if not os.path.exists(src):
+            if os.path.exists(dst):
+                problems.append(f"missing: {src} is gone and {dst} carries no journal"
+                                " line — it moved outside this tool, refusing to guess")
+            else:
+                problems.append(f"missing: {src} is gone and {dst} is not there either")
+            continue
+        if os.path.exists(dst):
+            problems.append(f"occupied: target already holds another file: {dst}")
+    if problems:
+        print(f"refusing: {len(problems)} problem(s), nothing moved")
+        for p in sorted(problems)[:20]:
+            print("  " + p)
+        if len(problems) > 20:
+            print(f"  ... and {len(problems) - 20} more")
+        return 2
+
+    if not args.apply:
+        print(f"dry run: {len(moves)} moves, {len(mkdirs)} folders, {len(keeps)} files stay"
+              " — nothing written (pass --apply to run)")
+        for src, dst, _k in moves[:5]:
+            print(f"  would move {src} -> {dst}")
+        return 0
+
+    made = {r.get("dst") for r in journal.records if r.get("op") == "mkdir"}
+    dirs_new = 0
+    for d in sorted(mkdirs, key=lambda q: q.count(os.sep)):
+        if d in made or os.path.isdir(d):
+            continue
+        journal.record(op="mkdir", dst=d)
+        os.makedirs(d, exist_ok=True)
+        dirs_new += 1
+
+    # same-filesystem gate: rename cannot cross a device, and a silent copy of a
+    # library this size is not an option anyone would choose.
+    cross = []
+    for src, dst, _k in moves:
+        if journal.already(dst, src) and os.path.exists(dst) and not os.path.exists(src):
+            continue
+        parent = os.path.dirname(dst)
+        if os.path.exists(parent) and os.stat(parent).st_dev != os.stat(src).st_dev:
+            cross.append(dst)
+    if cross:
+        print(f"refusing: {len(cross)} moves would cross a filesystem"
+              f" (first: {cross[0]}) — this tool only renames")
+        return 2
+
+    moved = done = 0
+    for src, dst, _kind in moves:
+        if journal.already(dst, src):
+            if os.path.exists(dst) and not os.path.exists(src):
+                done += 1
+                continue
+            if os.path.exists(src) and not os.path.exists(dst):
+                os.rename(src, dst)             # the intent ran, the move did not
+                moved += 1
+                continue
+        st = os.stat(src)
+        journal.record(op="move", src=src, dst=dst, bytes=st.st_size, mode=st.st_mode,
+                       mtime=st.st_mtime, ino=st.st_ino, dev=st.st_dev)
+        os.rename(src, dst)
+        moved += 1
+    print(f"applied: {moved} moved, {done} already done in an earlier run,"
+          f" {dirs_new} folders created, {len(keeps)} files stayed put")
+    print(f"journal: {journal.path} — undo with: media-apply.py rollback"
+          f" --journal {journal.path} --root {root} --apply")
+    return 0
+
+
+def do_verify(root, journal, moves, keeps):
+    """Every planned file sits at its target carrying the journalled bytes."""
+    journalled = {(r["src"], r["dst"]): r for r in journal.records
+                  if r.get("op") == "move"}
+    todo = moves or [(r["src"], r["dst"], "") for r in journal.records
+                     if r.get("op") == "move"]
+    bad, missing, bytes_total = [], [], 0
+    for src, dst, _kind in todo:
+        rec = journalled.get((src, dst))
+        if not os.path.exists(dst):
+            missing.append(dst)
+            continue
+        size = os.path.getsize(dst)
+        bytes_total += size
+        if rec and rec.get("bytes") not in (None, size):
+            bad.append(f"{dst}: {size} bytes on disk, {rec['bytes']} at move time")
+        if os.path.exists(src) and src != dst:
+            bad.append(f"{src}: still in place as well as at {dst}")
+    missing += [p for p in keeps if not os.path.exists(p)]
+    if bad or missing:
+        print(f"verify FAILED: {len(missing)} absent, {len(bad)} wrong")
+        for line in (missing + bad)[:20]:
+            print("  " + line)
+        return 2
+    empty = [r["dst"] for r in journal.records
+             if r.get("op") == "mkdir" and os.path.isdir(r["dst"])
+             and not os.listdir(r["dst"])]
+    print(f"verified: {len(todo)} files at their targets"
+          f" ({bytes_total / 2**40:.2f} TiB), {len(keeps)} kept")
+    if empty:
+        print(f"note: {len(empty)} folders are empty (left in place — see the trash law)")
+    return 0
+
+
+def do_rollback(args, root, journal):
+    """Undo every journalled move, newest first, after a full preflight."""
+    moves = [r for r in journal.records if r.get("op") == "move"]
+    made = [r["dst"] for r in journal.records if r.get("op") == "mkdir"]
+    problems = []
+    for r in reversed(moves):
+        src, dst = r["src"], r["dst"]
+        if not args.force and not (src.startswith(root + os.sep)
+                                   or dst.startswith(root + os.sep)):
+            problems.append(f"outside --root: {dst}")
+            continue
+        if os.path.exists(dst) and os.path.exists(src):
+            problems.append(f"occupied: both paths exist, refusing to choose:"
+                            f" {src} | {dst}")
+        elif os.path.exists(dst):
+            pass
+        elif os.path.exists(src):
+            continue                                    # already undone
+        else:
+            problems.append(f"missing: {dst} is gone and {src} never came back")
+    if problems:
+        print(f"refusing: {len(problems)} problem(s), nothing moved back")
+        for p in problems[:20]:
+            print("  " + p)
+        return 2
+    if not args.apply:
+        print(f"dry run: would move {len(moves)} files back to their census paths"
+              " (pass --apply)")
+        return 0
+    undone = 0
+    for r in reversed(moves):
+        if os.path.exists(r["dst"]) and not os.path.exists(r["src"]):
+            os.makedirs(os.path.dirname(r["src"]), exist_ok=True)
+            os.rename(r["dst"], r["src"])
+            undone += 1
+    dirs_gone = 0
+    for d in sorted(set(made), key=lambda q: -q.count(os.sep)):
+        if d == root or not os.path.isdir(d) or os.listdir(d):
+            continue
+        try:
+            os.rmdir(d)
+            dirs_gone += 1
+        except OSError as e:
+            print(f"left in place: {d} ({e})")
+    journal.record(op="rollback", undone=undone, dirs_removed=dirs_gone,
+                   t=datetime.datetime.now(datetime.timezone.utc).isoformat())
+    print(f"rolled back: {undone} files back where the census found them,"
+          f" {dirs_gone} empty folders removed")
+    return 0
+
+
+def main():
+    ap = argparse.ArgumentParser(description="apply or undo a media relayout plan")
+    ap.add_argument("cmd", choices=["apply", "rollback", "verify"])
+    ap.add_argument("--plan")
+    ap.add_argument("--root", default=DEFAULT_ROOT)
+    ap.add_argument("--journal")
+    ap.add_argument("--apply", action="store_true", help="actually change the library")
+    ap.add_argument("--force", action="store_true", help="relax the opinion guards")
+    args = ap.parse_args()
+
+    root = os.path.abspath(args.root)
+    if not os.path.isdir(root):
+        raise Refuse(f"no library at {root}")
+    journal_path = args.journal or journal_default(root)
+    journal = Journal(journal_path, args.apply)
+    try:
+        if args.cmd in ("rollback", "verify") and not os.path.exists(journal_path):
+            raise Refuse(f"no journal at {journal_path} — nothing to {args.cmd}")
+        if args.cmd == "rollback" and not args.force and journal.rolled_back:
+            raise Refuse("this journal has already been rolled back"
+                         " (--force to replay it)")
+        if args.cmd in ("apply", "verify") and not args.plan:
+            raise Refuse(f"{args.cmd} needs --plan")
+        mkdirs, keeps, moves = (read_plan(args.plan, root, args.force)
+                                if args.plan else ([], [], []))
+        if args.cmd == "apply":
+            return do_apply(args, root, journal, mkdirs, keeps, moves)
+        if args.cmd == "verify":
+            return do_verify(root, journal, moves, keeps)
+        return do_rollback(args, root, journal)
+    finally:
+        journal.close()
+
+
+if __name__ == "__main__":
+    try:
+        sys.exit(main())
+    except Refuse as r:
+        print(f"refusing: {r}")
+        sys.exit(2)

scripts/media-census.py

@@ -0,0 +1,203 @@
+#!/usr/bin/env python3
+r"""Read-only census of a movie library — the inputs a relayout plan is built from.
+
+  media-census.py enumerate [--root DIR] [--out TSV]
+  media-census.py probe     [--enum TSV] [--out TSV] [--jobs N]
+  media-census.py plex      [--root DIR] [--db PATH] [--out TSV]
+
+Run it on the machine that mounts the library. Nothing here writes inside the
+library and nothing is deleted; the only outputs are the three TSVs below, and
+scripts/media-plan.py reads them instead of touching media.
+
+  enum    bytes <TAB> ext <TAB> path                  every file under the root
+  probed  bytes <TAB> path <TAB> dur <TAB> vcodec <TAB> WxH <TAB> fps <TAB> vbit
+          <TAB> pix <TAB> audio <TAB> subs <TAB> spare <TAB> format
+          one row per video-container file; `audio` is "codec:lang+codec:lang",
+          `subs` the subtitle-stream count, `spare` the audio streams carrying
+          neither a default nor a forced tag
+  plex    file <TAB> leaf_title <TAB> … (see PLEX_COLS)
+          the titles Plex already matched — the authority over any filename guess
+
+`probe` is the slow step (every container is opened for its header, ~10 min for
+4,800 files on a spindle array at 6 workers) so it is `nice`d: a census beside a
+live service must not starve it. `enumerate` is a stat pass, so re-censusing
+before a plan is cheap — and a plan is only ever as good as these three files.
+"""
+import argparse
+import csv
+import json
+import os
+import re
+import sqlite3
+import subprocess
+import sys
+from concurrent.futures import ThreadPoolExecutor
+
+# Containers ffprobe can read. .iso/.img are deliberately NOT here: a disc image is
+# a mounted volume, not a stream, and the planner treats it as a movie file instead.
+PROBE_EXT = {".mkv", ".mp4", ".avi", ".m4v", ".mov", ".ts", ".m2ts", ".mpg",
+             ".mpeg", ".webm", ".wmv", ".divx", ".ogm", ".bdmv", ".vob"}
+# Trash and thumbnail caches are not library content, and this repo's trash policy
+# means .AGENTTRASH can sit inside a movie folder at any time.
+SKIP_DIR = re.compile(r"(^|/)(\.AGENTTRASH|\.thumbnails|\.Trash-\d+|@eaDir|lost\+found)(/|$)")
+PLEX_DB = ("/var/lib/plexmediaserver/Library/Application Support/Plex Media Server"
+           "/Plug-in Support/Databases/com.plexapp.plugins.library.db")
+PLEX_COLS = ["file", "leaf_title", "leaf_orig", "leaf_edition", "leaf_year",
+             "parent_title", "parent_orig", "parent_year", "parent_edition",
+             "container", "vcodec", "acodec", "width", "height"]
+PLEX_SQL = """
+select p.file, coalesce(mi.title,''), coalesce(mi.original_title,''),
+       coalesce(mi.edition_title,''), coalesce(mi.year,0),
+       coalesce(par.title,''), coalesce(par.original_title,''),
+       coalesce(par.year,0), coalesce(par.edition_title,''),
+       m.container, m.video_codec, m.audio_codec, m.width, m.height
+from media_parts p
+join media_items m on m.id = p.media_item_id
+join metadata_items mi on mi.id = m.metadata_item_id
+left join metadata_items par on par.id = mi.parent_id
+where p.file like ? and p.deleted_at is null and mi.deleted_at is null
+"""
+
+
+def enumerate_lib(root, out):
+    """One row per real file under root: size, extension, path."""
+    rows = []
+    for dirpath, dirnames, filenames in os.walk(root):
+        dirnames[:] = [d for d in dirnames
+                       if not SKIP_DIR.search(os.path.join(dirpath, d))]
+        for fn in filenames:
+            p = os.path.join(dirpath, fn)
+            if SKIP_DIR.search(p):
+                continue
+            try:
+                st = os.lstat(p)
+            except OSError:
+                continue
+            if not os.path.isfile(p) or os.path.islink(p):
+                continue                # links and sockets are not library content
+            rows.append((st.st_size, os.path.splitext(fn)[1].lower(), p))
+    with open(out, "w") as fh:
+        for size, ext, p in rows:
+            fh.write(f"{size}\t{ext}\t{p}\n")
+    vid = [r for r in rows if r[1] in PROBE_EXT]
+    print(f"files={len(rows)} bytes={sum(r[0] for r in rows)} "
+          f"probes={len(vid)} probe_bytes={sum(r[0] for r in vid)} -> {out}")
+
+
+def probe_one(path):
+    """ffprobe header read -> the probed TSV row (never the whole file)."""
+    try:
+        os.nice(19)                     # a census must never outrank the media server
+    except OSError:
+        pass
+    try:
+        size = os.stat(path).st_size
+    except OSError:
+        return None
+    cmd = ["ffprobe", "-v", "error", "-print_format", "json",
+           "-show_format", "-show_streams", path]
+    try:
+        done = subprocess.run(cmd, capture_output=True, text=True, timeout=180)
+        data = json.loads(done.stdout or "{}")
+    except (subprocess.TimeoutExpired, subprocess.SubprocessError,
+            json.JSONDecodeError, OSError):
+        data = {}
+    st = data.get("streams", [])
+    fmt = data.get("format", {})
+    v = next((s for s in st if s.get("codec_type") == "video"), None)
+    aud = [s for s in st if s.get("codec_type") == "audio"]
+    subs = [s for s in st if s.get("codec_type") == "subtitle"]
+    fps = "-"
+    if v and v.get("r_frame_rate"):
+        try:
+            n, d = (int(x) for x in v["r_frame_rate"].split("/", 1))
+            fps = f"{round(n / d * 100) / 100:g}" if d else "0"
+        except ValueError:
+            pass
+
+    def num(x):
+        try:
+            return round(float(x))
+        except (TypeError, ValueError):
+            return 0
+
+    return [size, path, num(fmt.get("duration")),
+            (v or {}).get("codec_name", "-"),
+            f"{v['width']}x{v['height']}" if v else "-",
+            fps, num(fmt.get("bit_rate")), (v or {}).get("pix_fmt", "-"),
+            "+".join(f"{a.get('codec_name', '?')}:{a.get('language', '?')}" for a in aud),
+            len(subs),
+            sum(1 for a in aud if not re.search(
+                r"default|forced", json.dumps(a.get("tags", {})), re.I)),
+            fmt.get("format_name", "-") or "-"]
+
+
+def probe(enum, out, jobs):
+    """Probe every video-container row of the enum TSV, in parallel."""
+    paths = []
+    with open(enum) as fh:
+        for line in fh:
+            f = line.rstrip("\n").split("\t", 2)
+            if len(f) == 3 and f[1] in PROBE_EXT:
+                paths.append(f[2])
+    print(f"probing {len(paths)} files with {jobs} workers", flush=True)
+    done = unreadable = 0
+    with open(out, "w") as fh, ThreadPoolExecutor(max_workers=jobs) as pool:
+        for row in pool.map(probe_one, paths):
+            if row is None:
+                continue
+            if row[3] == "-":
+                unreadable += 1         # header unreadable: the row still gets written
+            done += 1
+            fh.write("\t".join(str(x) for x in row) + "\n")
+            if done % 250 == 0:
+                print(f"  {done}/{len(paths)}", flush=True)
+    print(f"probed={done} unreadable={unreadable} -> {out}")
+
+
+def plex(root, db, out):
+    """Plex's matched titles for files under root, from its DB, read-only."""
+    uri = f"file:{db}?mode=ro&immutable=1"
+    try:
+        con = sqlite3.connect(uri, uri=True)
+    except sqlite3.Error as e:
+        sys.exit(f"cannot open the Plex database read-only ({db}): {e}\n"
+                 "Plex runs rootless here: run this as the container's owner, never "
+                 "with sudo — rootless storage is per-user, so root sees nothing.")
+    with open(out, "w", newline="") as fh:
+        w = csv.writer(fh, delimiter="\t")
+        w.writerow(PLEX_COLS)
+        n = 0
+        for r in con.execute(PLEX_SQL, (root + "/%",)):
+            w.writerow([str(x).replace("\t", " ").replace("\n", " ") for x in r])
+            n += 1
+    print(f"plex-matched={n} -> {out}")
+
+
+def main():
+    ap = argparse.ArgumentParser(description=__doc__.splitlines()[0])
+    sub = ap.add_subparsers(dest="cmd", required=True)
+    root = {"default": os.environ.get("MEDIA_ROOT", "/mnt/entropy/Movies"),
+            "help": "library root"}
+    e = sub.add_parser("enumerate", help="stat every file under the root")
+    e.add_argument("--root", **root)
+    e.add_argument("--out", default="/tmp/media-enum.tsv")
+    p = sub.add_parser("probe", help="ffprobe every video container")
+    p.add_argument("--enum", default="/tmp/media-enum.tsv")
+    p.add_argument("--out", default="/tmp/media-probed.tsv")
+    p.add_argument("--jobs", type=int, default=6)
+    x = sub.add_parser("plex", help="export Plex's matched titles for the root")
+    x.add_argument("--root", **root)
+    x.add_argument("--db", default=PLEX_DB)
+    x.add_argument("--out", default="/tmp/plex-movies.tsv")
+    a = ap.parse_args()
+    if a.cmd == "enumerate":
+        enumerate_lib(a.root, a.out)
+    elif a.cmd == "probe":
+        probe(a.enum, a.out, a.jobs)
+    else:
+        plex(a.root, a.db, a.out)
+
+
+if __name__ == "__main__":
+    main()

scripts/media-plan.py

@@ -0,0 +1,882 @@
+#!/usr/bin/env python3
+r"""Plan a Jellyfin-format relayout of a movie library.  DRY RUN ONLY: nothing moves.
+
+  media-plan.py [--root DIR] [--enum TSV] [--probed TSV] [--plex TSV]
+                [--out TSV] [--exceptions TXT]
+
+Inputs are the TSVs scripts/media-census.py exports, plus the optional Plex
+metadata-cache export whose clean titles outrank the filename parser; the title
+authority for everything Plex never matched is scripts/media-titles.py beside this
+file. No media file is opened, so planning the whole library costs seconds.
+
+PLAN is a TSV the executor (scripts/media-apply.py) replays verbatim —
+`action<TAB>src<TAB>dst<TAB>kind` with action in mkdir | move | keep, ordered so
+parents exist before children. EXCEPTIONS is the human review list: every file the
+plan could not name a film for, every collision it refused, the READMEs the
+executor must write into the piles it created.
+
+Target contract (jellyfin.org/docs/general/server/media/movies):
+  * one folder per movie, named  Title (year)
+  * the video inside is named exactly like the folder
+  * extra copies of one film are versions: "<base> - <label>.ext"; the base must
+    match the folder name character-for-character or Jellyfin splits them apart
+  * extras live in canonical subfolders (trailers, featurettes, deleted scenes,
+    interviews, samples, shorts, clips, other, extras, backdrops, theme-music)
+  * sidecars <base>.<lang>.<flag>.<ext>; images <base>-<poster|backdrop|...>
+  * <>:"/\|?* break the matcher, so they never survive here
+  * VIDEO_TS/BDMV folders support neither multiple versions nor parts
+"""
+import argparse
+import collections
+import csv
+import importlib.util
+import os
+import re
+
+# Inputs are the census TSVs scripts/media-census.py writes; the engine beside
+# this file is the title authority. Nothing here opens a media file.
+def parse_args():
+    ap = argparse.ArgumentParser(description=__doc__.splitlines()[0])
+    ap.add_argument("--root", default=os.environ.get("MEDIA_ROOT", "/mnt/entropy/Movies"),
+                    help="library root the plan moves files inside")
+    ap.add_argument("--enum", default="/tmp/media-enum.tsv", help="file census TSV")
+    ap.add_argument("--probed", default="/tmp/media-probed.tsv", help="ffprobe census TSV")
+    ap.add_argument("--plex", default="/tmp/plex-movies.tsv",
+                    help="Plex metadata export (clean titles); absent = parser only")
+    ap.add_argument("--out", default="/tmp/jf-plan.tsv")
+    ap.add_argument("--exceptions", default="/tmp/jf-exceptions.txt")
+    return ap.parse_args()
+
+
+ARGS = parse_args()
+ROOT = ARGS.root
+ENUM, PROBE, PLEX = ARGS.enum, ARGS.probed, ARGS.plex
+PLAN, EXCEPTS = ARGS.out, ARGS.exceptions
+ENGINE = os.path.join(os.path.dirname(os.path.abspath(__file__)), "media-titles.py")
+
+spec = importlib.util.spec_from_file_location("eng", ENGINE)
+eng = importlib.util.module_from_spec(spec)
+spec.loader.exec_module(eng)
+
+VIDEO_EXT = {".mkv", ".mp4", ".avi", ".m4v", ".mov", ".ts", ".m2ts", ".mpg",
+             ".mpeg", ".webm", ".wmv", ".divx", ".ogm", ".flv", ".vob", ".iso",
+             ".img", ".mpv", ".3gp", ".rm", ".rmvb", ".m2v", ".mpv"}
+DISC_DIRS = {"BDMV", "VIDEO_TS", "CERTIFICATE", "AACS", "MLPI", "CLIPINF"}
+SUB_EXT = {".srt", ".ass", ".ssa", ".sub", ".vtt"}
+IMG_EXT = {".jpg", ".jpeg", ".png", ".webp", ".bmp", ".gif", ".tbn"}
+AUDIO_DIR_RE = re.compile(r"soundtrack|score|\bost\b|music|song|album", re.I)
+SUB_KEEP_RE = re.compile(r"^(?!.*\d)(?!.*[._])(?!.*\[)(?!.*\()([A-Za-z][A-Za-z -]{1,24})$")
+MUSIC_TOP_RE = re.compile(r"(?i)soundtrack|score|\bost\b|music|album|song")
+SUB_DIR_OK = re.compile(r"(?i)^$|subs?|subtitles?|srt|custom ?subs?(\[.*\])?"
+                        r"|(additional|extra) languages|languages|resources")
+AUDIO_DIR_OK = re.compile(r"(?i)^$|audio(s)?|extra audio|additional audio"
+                          r"|(additional|extra) languages|commentar(y|ies)"
+                          r"|audio commentary|audio tracks?")
+
+
+AUDIO_EXT = {".mp3", ".flac", ".m4a", ".aac", ".dts", ".wav", ".ac3", ".mka",
+             ".opus", ".ogg", ".wma"}
+NFO_EXT = {".nfo"}
+IMG_PRIMARY = {"poster", "folder", "cover", "default", "movie", "show"}
+IMG_OTHER = {"backdrop": "backdrop", "fanart": "backdrop", "background": "backdrop",
+             "banner": "banner", "logo": "logo", "clearlogo": "logo",
+             "thumb": "thumb", "landscape": "thumb", "art": "backdrop",
+             "clearart": "backdrop", "jacket": "poster", "cdart": "poster",
+             "discart": "poster", "disc": "poster"}
+RES_RE = re.compile(r"\b(2160|4320|4k|1080|720|576|480|360|240)(i|p)\b", re.I)
+# The characters Jellyfin's matcher cannot carry in a file or folder name. "/" is
+# one of them and it is the dangerous one: Plex titles KEEP it ("V/H/S/2", and this
+# library holds six of those films), so leaving it in a name turns ONE movie folder
+# into a nested tree — Movies/V/H/S/2 (2013)/… — and Jellyfin then sees four
+# folders, none of them a movie.
+BAD_CHARS = re.compile(r'[<>:"/|?*]')
+PART_RE = re.compile(r"(?:^|[\s._-])(cd|dvd|part|pt|disc|disk|disco)"
+                     r"[\s._-]?([1-9abcd])(?:$|[\s._-])", re.I)
+LANGS = {"en", "eng", "it", "ita", "ja", "jpn", "de", "deu", "fr", "fra", "es",
+         "spa", "pt", "por", "ru", "rus", "zh", "chi", "ko", "kor", "sv", "no",
+         "da", "nl", "pl", "cs", "hu", "fi", "tr", "el", "he", "ar", "uk", "ro",
+         "bg", "hr", "sr", "sk", "et", "lv", "lt", "id", "th", "vi", "fa", "la"}
+FLAGS = {"default", "forced", "foreign", "sdh", "cc"}
+NOISE = {"sub", "subs", "subtitle", "subtitles", "srt", "ass", "ext", "external",
+         "the", "a", "an", "of", "and", "in", "on", "to", "for", "with", "ita",
+         "eng", "english", "italian", "dual", "audio", "forced", "default"}
+# Canonical title where the library only ever carries a localised name.
+OVERRIDES = {"28 giorni dopo": "28 Days Later",
+             "28 settimane dopo": "28 Weeks Later"}
+
+rows = []
+for line in open(ENUM):
+    size, ext, path = line.rstrip("\n").split("\t")
+    rows.append({"bytes": int(size), "ext": ext.lower(), "path": path})
+ext_by = {r["path"]: r["ext"] for r in rows}
+size_by = {r["path"]: r["bytes"] for r in rows}
+
+
+def rel(p):
+    return p[len(ROOT) + 1:] if p.startswith(ROOT + "/") else p
+
+
+def parts_of(p):
+    return [q for q in rel(p).split("/") if q]
+
+
+def is_qb(p):
+    return ".!q" in p.lower()
+
+
+def disc_root(p, fold=True):
+    """Absolute root of the disc tree containing p (the MOVIE folder that holds
+    BDMV/ or VIDEO_TS/), else None. `fold` also folds bookkeeping dirs that sit
+    beside the tree (CERTIFICATE/, AACS/) into the same movie folder."""
+    ps = parts_of(p)
+    kinds = DISC_DIRS if fold else {"BDMV", "VIDEO_TS"}
+    for i, q in enumerate(ps):
+        if q.upper() in kinds:
+            return ROOT + "/" + "/".join(ps[:i])
+    return None
+
+
+def row_name(r):
+    """The name a disc-tree entry should be identified by: the engine stores the
+    disc's own folder for it, because "BDMV/index.bdmv" is not a title."""
+    return r.get("name_src") or r["path"]
+
+
+def sanitize(name):
+    name = BAD_CHARS.sub("-", name).replace("\\", "-")
+    name = re.sub(r"\s+", " ", name).strip().strip(". ").strip()
+    return (name[:180] or "untitled")
+
+
+def ok_year(y):
+    """True for a year a real film could carry. Plex stores 0 for a file it indexed
+    but never MATCHED, and "0" is a digit string: trusting it named 67 movie
+    folders "Title (0)" — Jellyfin items with no identity, no year and no poster."""
+    return bool(re.fullmatch(r"(18|19|20)\d\d", y or "")) and 1870 <= int(y) <= 2035
+
+
+WORD_SPLIT = re.compile(r"[^0-9a-z]+")
+
+
+def words_of(text):
+    """The matching words of a name — or of a whole path, which is how callers
+    pass a file together with the folders it sits in.
+
+    Split on everything that is not a letter or a digit: eng.norm does not touch
+    "/", so feeding it "…/Psycho (1960)/Extras" left glued tokens like
+    "(1960)/extras" and "(1954)" that no other name could ever agree with, and a
+    film could not be found by the folder that holds it — which is how one box
+    set's featurettes ended up in another film's folder. Both sides of every
+    comparison below go through here. A bare number counts only when it could be
+    a release year, so release junk ("1080", "2160", "2", "7") never poses as
+    agreement between two names.
+    """
+    out = set()
+    for w in WORD_SPLIT.split(eng.norm(text)):
+        if not w or w in eng.STOP:
+            continue
+        if w.isdigit() and not 1870 <= int(w) <= 2100:
+            continue
+        out.add(w)
+    return out
+
+
+# ------------------------------------------------------------------ Plex titles
+# Plex has matched a large part of the library's media files to real titles
+# ("Alien", not "Alien.Directors.Cut.1979.1080p.BRrip.x265…"). Where the cache
+# holds a MATCH, its title wins; the filename parser carries everything else
+# (extras, disc trees, TV, and whatever Plex never matched).
+def plex_titles():
+    if not os.path.exists(PLEX):
+        return {}
+    out = {}
+    with open(PLEX, newline="") as fh:
+        for r in csv.DictReader(fh, delimiter="\t"):
+            t, y = r["leaf_title"].strip(), r["leaf_year"].strip()
+            # Only a match is an authority, and a match carries a year: an
+            # unmatched row echoes the filename, which the parser reads better
+            # than a copied release name. The junk filter stays for the matches
+            # whose title still carries the release.
+            if t and ok_year(y) and not re.search(
+                    r"2160p|1080p|x26[45]|hevc|bluray|web-?dl", t.lower()):
+                out[r["file"]] = (t, y)
+    return out
+
+
+PLEX_T = plex_titles()
+
+# ------------------------------------------------------------- classification
+erows = eng.load(PROBE)
+for r in erows:
+    for k, v in OVERRIDES.items():
+        if k in r["title"]:
+            r["title"] = v
+            r["key"] = eng.key_of(r["title"])
+            r["base"] = eng.base_words(r["title"], r["key"])
+marks = {r["path"] for r in erows if eng.is_watermark(r["path"], r["bytes"])}
+extras = {r["path"] for r in erows
+          if r["path"] not in marks
+          and eng.is_extra(r["path"], r["title"], r["dur"], ROOT,
+                           matched=r["path"] in PLEX_T)}
+eps = collections.defaultdict(list)
+for r in erows:
+    if r["path"] in marks | extras:
+        continue
+    if eng.ep_of(r["path"]):
+        eps[eng.key_of(r["title"])].append(r)
+tv_keys = {k for k, v in eps.items() if len(v) >= 5}
+tv_set = {r["path"] for k, v in eps.items() if k in tv_keys for r in v}
+# A folder INSIDE a movie's folder is bonus territory even when its own name is
+# not bonus vocabulary: "Saboteur (1942)/Exrtas/Saboteur - A Closer Look" is a
+# misspelled Extras folder, so the engine's folder rule never fired and the
+# featurette became a movie folder of its own — whose words then out-scored the
+# real Saboteur and drew that box set's other extras into it. The rule needs no
+# metadata: the folder holding the longest film owns the shorter video filed
+# under it, provided that shorter file's own name speaks the film's name — a
+# whole other film filed there stays a film, exactly as the engine's folder rule
+# exempts it. A different year in the file's own name IS that other film
+# speaking, so it vetoes the rule.
+def longest_of(rows):
+    """The longest video in each folder: that folder's feature."""
+    out = {}
+    for r in rows:
+        if r["dur"] > 0:
+            d = os.path.dirname(r["path"])
+            if d not in out or r["dur"] > out[d]["dur"]:
+                out[d] = r
+    return out
+
+
+def not_disc(r):
+    """A disc tree's innards belong to the tree, never to the folder's film."""
+    return not any(c in DISC_DIRS for c in r["path"].split("/"))
+
+
+FEATURE_OF = longest_of([r for r in erows if r["path"] not in marks and not_disc(r)])
+
+
+def under_a_feature(path, dur):
+    """True when a folder above this file holds a longer film that this file's
+    own name names."""
+    d = os.path.dirname(os.path.dirname(path))
+    while len(d) > len(ROOT):
+        f = FEATURE_OF.get(d)
+        if f:
+            if f["dur"] < dur:
+                return False
+            theirs = (words_of(os.path.basename(f["path"]))
+                      | words_of(os.path.basename(d)))
+            if not words_of(os.path.basename(path)) & theirs:
+                return False
+            y1 = re.search(r"(?:18|19|20)\d\d", os.path.basename(path))
+            y2 = re.search(r"(?:18|19|20)\d\d", os.path.basename(f["path"]))
+            return not (y1 and y2 and y1.group() != y2.group())
+        d = os.path.dirname(d)
+    return False
+
+
+extras |= {r["path"] for r in erows
+           if r["dur"] > 0 and r["path"] not in marks | extras | tv_set
+           and not_disc(r) and under_a_feature(r["path"], r["dur"])}
+film_rows = [r for r in erows if r["path"] not in marks | extras | tv_set]
+
+buckets = collections.defaultdict(list)
+for r in film_rows:
+    buckets[eng.key_of(r["title"])].append(r)
+
+
+def res_label(r):
+    m = RES_RE.search(r["res"] or "")
+    if m:
+        n, kind = m.group(1).lower(), m.group(2).lower()
+        return "2160p" if n in ("2160", "4k") else f"{n}{kind}"
+    try:
+        w = int((r["res"] or "0x0").split("x")[0])
+    except (ValueError, IndexError):
+        w = 0
+    return "2160p" if w >= 3000 else ("1080p" if w >= 1400 else "SD")
+
+
+def version_label(r):
+    bits = [res_label(r)]
+    vc = (r["vcodec"] or "").lower()
+    if vc in ("av1", "libaom", "libsvtav1"):
+        bits.append("AV1")
+    elif vc in ("vc1", "wmv3"):
+        bits.append("VC1")
+    if r["bytes"] > 40 * 2**30:
+        bits.append("REMUX")
+    if "imax" in r["path"].lower():
+        bits.append("IMAX")
+    return sanitize(" ".join(bits))
+
+
+def group_name(items):
+    titles = collections.Counter(r["title"] for r in items)
+    ascii_t = [t for t in titles if t.isascii()]
+    title = (collections.Counter(ascii_t) if ascii_t else titles).most_common(1)[0][0]
+    years = [r["year"] for r in items if r["year"] != "?"] or \
+            [r["soft"] for r in items if r["soft"]]
+    year = collections.Counter(years).most_common(1)[0][0] if years else None
+    return sanitize(f"{title} ({year})" if year else title)
+
+
+groups = []
+for key, items in buckets.items():
+    for c in eng.dur_clusters(items):
+        g = {"rows": c, "base": group_name(c)}
+        known = [PLEX_T[r["path"]] for r in c if r["path"] in PLEX_T]
+        if known:
+            # one film, one name: a Plex match for any copy names every copy
+            t, yr = collections.Counter(known).most_common(1)[0][0]
+            g["plex"] = t
+            g["base"] = sanitize(f"{t} ({yr})" if ok_year(yr) else t)
+        groups.append(g)
+clash = {b for b, n in collections.Counter(g["base"] for g in groups).items() if n > 1}
+for g in groups:
+    if g["base"] in clash:
+        # two genuinely different films share title+year: separate folders, and
+        # say so in the name, because Jellyfin can only match one item per folder
+        g["base"] = sanitize(g["base"] + " - " + version_label(g["rows"][0]))
+        g["flag"] = "same title and year as another film"
+
+moves, mkdirs, problems, readmes = [], set(), [], []
+# --------------------------------------------------------------- unnamed groups
+# A movie folder has to READ as a title. The engine's runtime veto says any file
+# over 45 minutes that is not named like an extra is a film whatever it is called
+# — true for Jellyfin, but a 118-minute documentary named "Extra 1" then produced
+# a movie folder named "1", and folders named "1", "2", "3" are Jellyfin items
+# with no identity whose numeric keys went on to capture every subtitle named
+# "3_Eng.srt" in the library. A year proves a dated feature ("300 (2006)" is a
+# real film), so the gate is: a year, or a word that is not release noise. With
+# neither, the rows are bonus material until a human names them.
+def named(g):
+    if re.search(r"\(\d{4}\)$", g["base"]):
+        return True
+    return any(w.isalpha() and len(w) >= 2 for w in eng.name_words(g["base"]))
+
+
+unnamed = [g for g in groups if not named(g)]
+groups = [g for g in groups if named(g)]
+for g in unnamed:
+    for r in g["rows"]:
+        extras.add(r["path"])
+        problems.append(
+            f"UNNAMED film {rel(r['path'])} parses to {g['base']!r} with no year: "
+            f"bonus pile, not a folder — give the file a real name and re-plan")
+
+
+group_of = {}
+for g in groups:
+    g["dir"] = f"{ROOT}/{g['base']}"
+    for r in g["rows"]:
+        group_of[r["path"]] = g
+        dr = disc_root(r["path"])
+        if dr:
+            group_of[dr] = g
+by_path = {r["path"]: r for r in erows}
+
+# -------------------------------------------------------------- bookkeeping
+claimed = {}
+existing = set(rows_map := {r["path"] for r in rows})
+
+
+at_target, booked_src = set(), set()
+
+
+def book(src, dst, kind):
+    if src == dst:
+        at_target.add(src)
+        booked_src.add(src)
+        return
+    if os.environ.get("JF_TRACE") and (src in booked_src or dst.lower() in claimed):
+        why = "rebook" if src in booked_src else "collision"
+        problems.append(f"TRACE {why} kind={kind} src={rel(src)} dst={rel(dst)}")
+    if src in booked_src:          # one source may only ever move once
+        problems.append(f"RE-BOOK {rel(src)} -> {rel(dst)} (already planned)")
+        return
+    if dst.lower() in claimed:
+        problems.append(f"COLLISION {rel(dst)}\n    from {rel(src)}\n    and  {rel(claimed[dst.lower()])}")
+        return
+    claimed[dst.lower()] = src
+    booked_src.add(src)
+    comp = dst.split("/")
+    for i in range(1, len(comp)):
+        mkdirs.add("/".join(comp[:i]))
+    moves.append((src, dst, kind))
+
+
+def uniq_dir(d, stem, ext, hint=""):
+    """A destination that no other source already took."""
+    cand = f"{d}/{stem}{ext}"
+    if cand.lower() not in claimed:
+        return cand
+    n = 2
+    while True:
+        cand = f"{d}/{stem} {n}{ext}"
+        if cand.lower() not in claimed:
+            return cand
+        n += 1
+
+
+def media_dir(p):
+    dr = disc_root(p)
+    return os.path.dirname(dr) if dr else os.path.dirname(p)
+
+
+# ------------------------------------------------------------------ films
+# film / disc moves: the largest copy carries the folder's own name, the rest
+# become Jellyfin versions; a disc tree moves as a whole into the movie folder
+
+
+def vext(p):
+    return ext_by.get(p) or os.path.splitext(p)[1].lower()
+disc_children = collections.defaultdict(list)
+for q in rows:
+    dr = disc_root(q["path"])
+    if dr:
+        disc_children[dr].append(q["path"])
+for g in groups:
+    primary_taken = False
+    disc_kinds = set()
+    for r in sorted(g["rows"], key=lambda r: -r["bytes"]):
+        dr = disc_root(r["path"])
+        if dr:
+            inner = os.path.basename(dr).upper()
+            if inner in disc_kinds:
+                problems.append(
+                    f"DISC-VERSION {g['base']}: extra disc tree {rel(dr)} cannot "
+                    f"share the folder — discs have no multiple versions")
+                continue
+            disc_kinds.add(inner)
+            for src in disc_children[dr]:
+                tail = os.path.relpath(src, dr)
+                fold = disc_root(src, fold=False)
+                if fold != dr and os.path.dirname(src) != dr:
+                    tail = os.path.basename(dr) + "/" + tail
+                book(src, f"{g['dir']}/{tail}", "disc-file")
+            continue
+        label = "" if not primary_taken else f" - {version_label(r)}"
+        dst = f"{g['dir']}/{g['base']}{label}{vext(r['path'])}"
+        n = 2
+        while dst.lower() in claimed:
+            dst = f"{g['dir']}/{g['base']}{label} {n}{vext(r['path'])}"
+            n += 1
+        r["dst"] = dst
+        primary_taken = True
+        book(r["path"], dst, "film")
+    g["primary"] = primary_taken
+
+# ------------------------------------------------------------------ extras
+EXTRA_KIND = [("trailers", r"trailer|teaser|preview|promo|tv spot|advert"),
+              ("deleted scenes", r"deleted|alt scene|gag reel|blooper|outtake"),
+              ("interviews", r"interview|cast and crew|session piece|commentary"),
+              ("featurettes", r"featurette|behind the scenes|making of|documentary"
+                              r"|heritage|archival|video archive|music video"),
+              ("samples", r"sample"),
+              ("shorts", r"\bshort\b|short film"),
+              ("clips", r"\bclip\b|scene selection|montage|epilogue|isolated score"),
+              ("other", r"menu|logo|gallery|still|trivia|storyboard|screen test"
+                        r"|audition|test footage|watermark|workprint|poster"
+                        r"|picture in picture|singalong|karaoke|original score")]
+THEME_RE = re.compile(r"theme|opening song|ending song|soundtrack|score"
+                      r"|\bost\b|music|song", re.I)
+BACKDROP_RE = re.compile(r"backdrop|intro", re.I)
+
+
+def extras_kind(path):
+    low = path.lower()
+    if THEME_RE.search(low):
+        return "theme-music"
+    if BACKDROP_RE.search(low):
+        return "backdrops"
+    for name, rx in EXTRA_KIND:
+        if re.search(rx, low):
+            return name
+    return "extras"
+
+
+def attachable_title(path):
+    t, _y = eng.title_of(path)
+    return eng.key_of(t) if t and not eng.is_watermark(path, size_by.get(path, 0)) else None
+
+
+for g in groups:
+    g["tops"] = {parts_of(r["path"])[0] for r in g["rows"]}
+top_groups = collections.defaultdict(list)
+for g in groups:
+    for t in g["tops"]:
+        top_groups[t].append(g)
+group_by_key = collections.defaultdict(list)
+for g in groups:
+    for r in g["rows"]:
+        group_by_key[eng.key_of(r["title"])].append(g)
+group_by_dir = {}
+for g in groups:
+    for r in g["rows"]:
+        group_by_dir.setdefault(media_dir(r["path"]), g)
+        dr = disc_root(r["path"])
+        if dr:
+            group_by_dir.setdefault(os.path.dirname(dr), g)
+
+
+def group_words(g):
+    """The words that identify a film group: its titles and the names of the
+    folders its files live in."""
+    if "words" not in g:
+        w = set()
+        if g.get("plex"):
+            w |= words_of(g["plex"])
+        for r in g["rows"]:
+            w |= words_of(r["title"])
+            w |= words_of(os.path.basename(media_dir(r["path"])))
+        g["words"] = w
+    return g["words"]
+
+
+def score_words(text, groups):
+    """Ranks film groups by how many of their words a path carries."""
+    words = words_of(text)
+    out = [(len(words & group_words(g)), g) for g in groups]
+    out.sort(key=lambda x: -x[0])
+    return out
+
+
+def group_for_orphan(path):
+    """Which movie folder an extra/sidecar belongs to. A collection folder holding
+    several films must be resolved by WORDS, never by position: taking the first
+    film of the folder attached every loose subtitle in a box set to whichever
+    movie happened to sort first."""
+    stem = os.path.splitext(os.path.basename(path))[0]
+    key = attachable_title(path)
+    if key and group_by_key.get(key):
+        cands = group_by_key[key]
+        top = parts_of(path)[0] if len(parts_of(path)) > 1 else ""
+        same_top = [g for g in cands if any(parts_of(r["path"])[0] == top
+                                            for r in g["rows"] if len(parts_of(r["path"])) > 1)]
+        pool = same_top or cands
+        return pool[0] if len(pool) == 1 else score_words(stem, pool)[0][1]
+    gtop = parts_of(path)[0] if len(parts_of(path)) > 1 else ""
+    pool = top_groups.get(gtop, [])
+    if len(pool) > 1:                    # a loose collection folder (box set, bonus
+        n, g = score_words(stem + " " + os.path.dirname(path), pool)[0]
+        return g if n >= 2 else None     # album) is resolved by words, not position
+    if len(pool) == 1:
+        return pool[0]
+    d = os.path.dirname(path)
+    while d and d != ROOT:
+        if d in group_by_dir:
+            return group_by_dir[d]
+        d = os.path.dirname(d)
+    return None
+
+
+tv_tops = {parts_of(p)[0] for p in tv_set}
+def film_folder_for(path):
+    """A movie folder for a lone feature that has no group: derived from its own
+    title when a year proves it, else None (goes to the review pile)."""
+    t, y = eng.title_of(path)
+    if not t or y == "?":
+        return None
+    d = uniq_dir(ROOT, sanitize(f"{t} ({y})"), "", "")
+    mkdirs.add(d)
+    return d
+
+
+for p in sorted(extras):
+    if disc_root(p):
+        continue
+    if len(parts_of(p)) > 1 and parts_of(p)[0] in tv_tops:
+        continue                        # a special inside a TV set: staged with it
+    g = group_for_orphan(p)
+    ext = ext_by.get(p, os.path.splitext(p)[1].lower())
+    stem = os.path.splitext(os.path.basename(p))[0]
+    if g is None:
+        solo = film_folder_for(p)
+        if solo:
+            dst = f"{solo}/{sanitize(stem)}{ext}"
+            problems.append(f"REVIEW lone feature {rel(p)}: gets its own folder "
+                            f"{os.path.basename(solo)}")
+        else:
+            dst = f"{ROOT}/_unsorted bonus/{extras_kind(p)}/{sanitize(stem)}{ext}"
+            problems.append(f"NO-FILM extra {rel(p)}")
+    else:
+        dst = f"{g['dir']}/{extras_kind(p)}/{sanitize(stem)}{ext}"
+    book(p, uniq_dir(os.path.dirname(dst), os.path.basename(dst)[:-(len(ext) or 1)],
+                     ext, os.path.dirname(p)), "extra")
+
+# Disc trees the engine never saw (a MakeMKV scratch folder, an AACS remnant):
+# they belong to the film of the folder they sit in, kept as a named subtree.
+for q in rows:
+    pth = q["path"]
+    dr = disc_root(pth)
+    if not dr or pth in by_path or pth in booked_src:
+        continue
+    g = group_by_dir.get(dr) or (top_groups[parts_of(pth)[0]][0]
+                                 if len(top_groups.get(parts_of(pth)[0], [])) == 1 else None)
+    if g is None:
+        continue
+    inner = os.path.basename(dr)
+    problems.append(f"REVIEW disc remnant {rel(dr)} moves inside {g['base']} "
+                    f"(a second master: compare before keeping both)")
+    for src in disc_children[dr]:
+        if src not in booked_src:
+            book(src, f"{g['dir']}/{inner}/{os.path.relpath(src, dr)}", "disc-file")
+
+
+# ----------------------------------------------------------------- sidecars
+media_by_dir = collections.defaultdict(list)
+for r in erows:
+    if r["path"] in marks:
+        continue
+    dst = r.get("dst")
+    if dst:
+        media_by_dir[media_dir(r["path"])].append(r)
+
+
+
+def sidecar_target(path, e):
+    d = os.path.dirname(path)
+    stem = os.path.splitext(os.path.basename(path))[0]
+    if d == ROOT:
+        # A file loose in the library ROOT names no film. media_by_dir[ROOT] is the
+        # set of root-level MOVIE files, and scoring a stray name against it
+        # attached every root-level leftover to whichever root-level film was
+        # largest — the plan sent ".DS_Store" into "Titanic (1997)/other/" that
+        # way. A root-level stray is nobody's sidecar: it goes to the review pile.
+        return None, None
+    cands = media_by_dir.get(d, [])
+    if not cands:
+        nd = os.path.dirname(d)          # a subs/ or Extras/ folder beside the media
+        while nd and nd != ROOT and not media_by_dir.get(nd):
+            nd = os.path.dirname(nd)
+        if media_by_dir.get(nd):
+            if nd != d:                  # a NAMED subfolder (Subs/, Extras/, the
+                g = group_for_orphan(path)   # box set's own root) carries evidence
+                dirs = {os.path.dirname(r.get("dst", "")) for r in media_by_dir[nd]}
+                if g is None or g["dir"] not in dirs:
+                    # ...but only a STRONG word match outranks the folder the file
+                    # sits in. A weak one is an accident: "3_Eng.srt" scored 1
+                    # against a junk group named "3" and escaped its own movie's
+                    # Subs/ folder, while "03 xXx 3 Return Of Xander Cage" scores 4
+                    # and genuinely belongs to the other film of the set.
+                    n = score_words(stem, [g])[0][0] if g else 0
+                    if n >= 2:
+                        return (g, None)
+                cands = media_by_dir[nd]
+    if not cands:
+        g = group_for_orphan(path)
+        return (g, None) if g else (None, None)
+    if len(cands) > 1:
+        w = [t for t in eng.norm(stem).split() if t not in NOISE]
+        cands = sorted(cands, key=lambda r: (
+            -len([t for t in eng.norm(os.path.basename(r["path"])).split() if t in w]),
+            -r["bytes"]))
+    return (None, cands[0])
+
+
+for q in rows:
+    p, e = q["path"], q["ext"]
+    if p in by_path or p in extras or p in marks or is_qb(p) or disc_root(p):
+        continue
+    if os.path.basename(p).startswith("._"):
+        # macOS AppleDouble resource forks: a few hundred bytes carrying a name
+        # COPIED FROM ANOTHER FILM. Word-matching them to "their" film scattered
+        # dead files through four movie folders (and they are video extensions in
+        # the AppleDouble case, so they also skipped every other filter). They
+        # claim nothing: one pile, and they stay in it — nothing is deleted here.
+        book(p, uniq_dir(f"{ROOT}/_macos sidecars", sanitize(os.path.basename(p))[:120],
+                         "", os.path.dirname(p)), "junk")
+        problems.append(f"JUNK AppleDouble sidecar {rel(p)} (macOS resource fork)")
+        continue
+    if e == ".iso":         # a DVD image: Jellyfin plays it as a movie file
+        g = group_for_orphan(p)
+        stem = os.path.splitext(os.path.basename(p))[0]
+        if g is not None:
+            book(p, uniq_dir(g["dir"], g["base"], e, stem) if g.get("primary") is not None
+                 else f"{g['dir']}/{g['base']}{e}", "iso")
+        else:
+            solo = film_folder_for(p)
+            if solo:
+                book(p, f"{solo}/{sanitize(stem)}.iso", "iso")
+                problems.append(f"REVIEW disc image {rel(p)}: DVD rip, alone in its "
+                                f"folder -> {os.path.basename(solo)}")
+            else:
+                book(p, f"{ROOT}/_unsorted files/{sanitize(os.path.basename(p))}", "orphan")
+                problems.append(f"NO-FILM disc image {rel(p)}")
+        continue
+    if e in VIDEO_EXT:
+        # A video the engine never probed (unreadable header) and not an AppleDouble
+        # remnant: the plan does not guess at it, so it stays where it is.
+        continue
+    stem = os.path.splitext(os.path.basename(p))[0]
+    g, media = sidecar_target(p, e)
+    if media is not None:
+        base = os.path.splitext(os.path.basename(media["dst"]))[0]
+        d = os.path.dirname(media["dst"])
+    elif g is not None:
+        base, d = g["base"], g["dir"]
+    else:
+        # Nothing names this file's film. NEVER scatter it: an archive set or an
+        # album tree must move as the unit it is, so the whole loose folder goes
+        # to the review pile and a film folder's own leftovers stay with it.
+        ps = parts_of(p)
+        if len(ps) < 2:
+            dst = f"{ROOT}/_unsorted files/{sanitize(stem)}{e}"
+        elif len(top_groups.get(ps[0], [])):
+            dst = f"{top_groups[ps[0]][0]['dir']}/other/{'/'.join(sanitize(c) for c in ps[1:])}"
+        else:
+            dst = f"{ROOT}/_unsorted files/{'/'.join(sanitize(c) for c in ps)}"
+        book(dst=dst, src=p, kind="orphan")
+        problems.append(f"NO-OWNER sidecar {rel(p)}")
+        continue
+    if e in IMG_EXT:
+        low = eng.norm(stem)
+        word = next((w for w in IMG_PRIMARY if re.search(rf"\b{w}\b", low)), None)
+        kind = "poster" if word else next(
+            (IMG_OTHER[w] for w in IMG_OTHER if re.search(rf"\b{w}\b", low)), None)
+        if re.search(r"\btheme\b", low):
+            dst = f"{d}/theme{e}"
+        else:
+            dst = f"{d}/{base}-{kind or 'art'}{e}" if kind else f"{d}/{sanitize(stem)}{e}"
+        book(p, uniq_dir(os.path.dirname(dst), os.path.basename(dst)[:-(len(e) or 1)],
+                         e, os.path.dirname(p)), "image")
+    elif e in NFO_EXT:
+        book(p, uniq_dir(d, base, ".nfo", os.path.dirname(p)), "nfo")
+    elif e in SUB_EXT | AUDIO_EXT:
+        toks = [t for t in re.split(r"[._\-\s]+", stem) if t]
+        langs = [t.lower() for t in toks if t.lower() in LANGS]
+        flags = [t.lower() for t in toks if t.lower() in FLAGS]
+        keep = [sanitize(t) for t in toks
+                if t.lower() not in langs + flags + list(NOISE)
+                and not re.fullmatch(r"\d{3,4}p?|cd\d+|\d{4}|\d+ch|\d+.\d", t.lower())]
+        bits = [base] + ([sanitize(" ".join(keep[:4]))] if keep else []) + langs + flags
+        if e in SUB_EXT:
+            # subtitles: <movie>/srt[/language]
+            d = f"{d}/srt"
+            pd = os.path.basename(os.path.dirname(p))
+            if SUB_KEEP_RE.match(pd) and not SUB_DIR_OK.match(pd):
+                d = f"{d}/{sanitize(pd)}"      # a real language folder survives
+        elif e in AUDIO_EXT and AUDIO_DIR_RE.search(os.path.dirname(p)):
+            d = f"{d}/theme-music"                 # bonus audio: one canonical folder
+            pd = os.path.basename(os.path.dirname(p))
+            if not MUSIC_TOP_RE.search(pd):        # an album folder keeps its name
+                d = f"{d}/{sanitize(pd)}"
+        else:
+                pd = os.path.basename(os.path.dirname(p))
+                if not AUDIO_DIR_OK.match(pd):
+                    d = f"{d}/{sanitize(pd)}"
+        stem2 = ".".join(bits) or sanitize(os.path.basename(p))
+        book(p, uniq_dir(d, stem2, e, os.path.basename(p)), "track")
+    else:
+        book(p, uniq_dir(f"{d}/other", sanitize(os.path.basename(p))[:120], e,
+                         os.path.dirname(p)), "misc")
+
+readmes.append(f"README in {ROOT}: this library was relaid out into Jellyfin's "
+               "one-folder-per-movie layout; `_release-group watermarks/`, "
+               "`_unsorted files/` and `_unsorted bonus/` hold files no film claims.")
+for key in tv_keys:
+    nm = "_TV needs a TV library - " + collections.Counter(
+        r["title"] for r in eps[key]).most_common(1)[0][0]
+    readmes.append(f"README in {ROOT}/{sanitize(nm)}: these are TV episodes found in "
+                   "the MOVIES library; move the folder to a TV library source, else "
+                   "Jellyfin groups them wrongly as one movie.")
+
+# --------------------------------------------------------------------- TV
+for key in tv_keys:
+    items = eps[key]
+    title = collections.Counter(r["title"] for r in items).most_common(1)[0][0]
+    year = collections.Counter(r["year"] for r in items if r["year"] != "?")
+    yr = year.most_common(1)[0][0] if year else None
+    base = f"_TV needs a TV library - {sanitize(title)}" + (f" ({yr})" if yr else "")
+    tops = {parts_of(r["path"])[0] for r in items}
+    members = sorted([r["path"] for r in items]
+                     + [q for q in extras | marks if len(parts_of(q)) > 1
+                        and parts_of(q)[0] in tops],
+                     key=lambda q: q)
+    for src in members:
+        rel_parts = rel(src).split("/")
+        tail = "/".join(sanitize(c) for c in rel_parts[:-1] + [rel_parts[-1]])
+        book(src, f"{ROOT}/{base}/{tail}", "tv")
+for p in sorted(marks):
+    stem, ext = os.path.splitext(os.path.basename(p))
+    book(p, uniq_dir(f"{ROOT}/_release-group watermarks", sanitize(stem)[:120], ext,
+                     os.path.dirname(p)), "watermark")
+for q in rows:
+    if is_qb(q["path"]):
+        problems.append(f"SKIPPED qBittorrent partial {rel(q['path'])}")
+        book(q["path"], q["path"], "qb-partial")     # stays: resume data
+
+# --------------------------------------------------------------- debug hook
+if os.environ.get("JF_ISO"):
+    x = ROOT + "/High Anxiety (Mel Brooks) [1977] nordic subs  DVDrip/High Anxiety [1977].iso"
+    print("attachable", attachable_title(x), "| dir", os.path.dirname(x) in group_by_dir,
+          "| g", group_for_orphan(x) and group_for_orphan(x)["base"])
+    raise SystemExit
+if os.environ.get("JF_DEBUG"):
+    import json
+    unp = sorted({r["path"] for r in rows} - {m[0] for m in moves} - at_target)
+    print("at_target", len(at_target))
+    print("kinds-unplanned", collections.Counter(os.path.splitext(u)[1] for u in unp).most_common(8))
+    for up in unp[:20]:
+        d = os.path.dirname(up)
+        print("UNP", up[:90], "| mediadir?", d in media_by_dir, "| by_path", up in by_path,
+              "| extras", up in extras, "| discroot", disc_root(up), "| ext", os.path.splitext(up)[1])
+
+# -------------------------------------------------------------- validation
+srcs = [m[0] for m in moves]
+dup_src = {s for s, n in collections.Counter(srcs).items() if n > 1}
+for s in dup_src:
+    problems.append(f"DOUBLE-CLAIM source {rel(s)}")
+all_src = set(srcs)
+untouched = {r["path"] for r in rows} - all_src
+bad_names, into_alive, lost = [], [], []
+for src, dst, kind in moves:
+    for comp in dst[len(ROOT) + 1:].split("/"):
+        if (BAD_CHARS.search(comp) or comp != comp.strip(". ") or not comp
+                or len(comp) > 200):
+            bad_names.append(f"{kind}: {comp!r} in {rel(dst)}")
+    if dst in untouched:                       # something staying put is there
+        into_alive.append(f"{rel(src)} -> {rel(dst)} (occupied by a file that stays)")
+    if dst.lower() in {s.lower() for s in untouched}:
+        into_alive.append(f"{rel(src)} -> {rel(dst)} (case-collides with a survivor)")
+    if dst.startswith(src + "/") and src in all_src:
+        lost.append(f"{rel(src)} moves into its own subtree")
+# every accounted-for source must exist exactly once in the plan
+covered = {r["path"] for r in rows}
+not_planned = sorted(covered - all_src - at_target)
+
+with open(PLAN, "w") as out:
+    out.write("action\tsrc\tdst\tkind\n")
+    for d in sorted(mkdirs, key=lambda q: (q.count("/"), q)):
+        if d.startswith(ROOT + "/") and d != ROOT:
+            out.write(f"mkdir\t\t{d}\tdir\n")
+    for p in sorted(booked_src - {m[0] for m in moves}):
+        out.write(f"keep\t{p}\t{p}\n")
+    for src, dst, kind in sorted(moves, key=lambda m: m[0]):
+        out.write(f"move\t{src}\t{dst}\t{kind}\n")
+
+kinds = collections.Counter(k for _s, _d, k in moves)
+films_n = sum(1 for _s, _d, k in moves if k == "film")
+folders = len({os.path.dirname(d) for _s, d, k in moves if k in ("film", "disc-file")})
+with open(EXCEPTS, "w") as out:
+    out.write(f"groups={len(groups)} moves={len(moves)} folders={folders}\n")
+    out.write(f"kinds={dict(kinds)}\n")
+    out.write(f"at-target-already={len(at_target)} unplanned={len(not_planned)}\n")
+    out.write("\n=== WHAT THE EXECUTOR MUST ALSO WRITE ===\n")
+    out.write("\n".join(sorted(set(readmes))) + "\n")
+    out.write("\n=== PROBLEMS ===\n")
+    out.write("\n".join(sorted(set(problems))) + "\n")
+    out.write("\n=== BAD NAMES ===\n" + "\n".join(sorted(set(bad_names))) + "\n")
+    out.write("\n=== TARGETS OCCUPIED BY SURVIVORS ===\n"
+              + "\n".join(sorted(set(into_alive))) + "\n")
+    out.write("\n=== LEFTOVER SOURCES (first 60) ===\n"
+              + "\n".join(rel(p) for p in not_planned[:60]) + "\n")
+print(f"groups={len(groups)} folders={folders} moves={len(moves)} "
+      f"mkdirs={len(mkdirs)} film-files={films_n}")
+print(f"kinds={dict(kinds)}")
+print(f"problems={len(set(problems))} bad-names={len(set(bad_names))} "
+      f"occupied={len(set(into_alive))} double-claim={len(dup_src)}")
+print(f"left-unmoved={len(not_planned)}")
+print(f"tv-keys={len(tv_keys)} extras={len(extras)} marks={len(marks)} "
+      f"disc-trees={len(disc_children)}")

scripts/media-titles.py

@@ -0,0 +1,563 @@
+#!/usr/bin/env python3
+r"""The fleet's release-filename title parser: the grouping authority the relayout
+plan uses for every file the media server never matched.
+
+  report:  media-titles.py report [--probed TSV] [--out TSV]
+           the similar-copies census (DUP / REVIEW / EP-DUP groups) over the
+           ffprobe census scripts/media-census.py writes.
+  parse:   media-titles.py parse NAME [NAME…]
+           what title_of() makes of a name — the way to check a naming accident
+           before it merges two films into one folder.
+
+scripts/media-plan.py imports this file; over there the paths are arguments.
+
+Method — three independent signals, so a naming accident cannot merge two films:
+  * franchise KEY  : first two significant title tokens. Stable across parts of a
+    collection ("resident evil", "jump street") but NOT film-specific — it only
+    selects candidates.
+  * DURATION       : the identity check. Same master encodes differ by at most a
+    couple of seconds; different films / cuts do not collide.
+  * SIZE SPREAD + YEAR: decide the tier. Two files same size AND same duration
+    are byte-twins (the hash pass confirms); same duration but different sizes are
+    two different releases of one film (deletable redundancy).
+
+TV episodes (SxxEyy in the name) are grouped separately: their uniform runtime
+would otherwise collapse a whole series into one "duplicate" group.
+"""
+import argparse
+import collections, os, re, sys, unicodedata
+
+# Library root: the folder directly under this one names the film, and every
+# folder below it is structure (Extras/, Featurettes/, Disc 1/, BDMV/). Tests pass
+# their own; nothing here reads media, only the census TSVs about it.
+ROOT = os.environ.get("MEDIA_ROOT", "/mnt/entropy/Movies")
+
+# Release-tag vocabulary: never part of a title, so the title stops here.
+JUNK = {"5.1", "7.1", "2.0", "1.0", "6.1", "4.0", "extras"}
+JUNK_RE = re.compile(
+    r"^("
+    r"\d{3,5}p|\d{1,2}k|uhd|fhd|imax|x ?26[45]|hevc|h ?26[45]|avc|av1|x ?av1|vp[89]"
+    r"|divx|xvid|mpeg-?2?|\d{1,2}-?bit|hdr ?10\+?|hdr|dolby ?vision|dvh?|sdr|hybrid"
+    r"|re-?mux|bd-?re-?mux|blu-?ray|bdrip|brrip|bdmux|bd|web-?(rip|dl)?|hdtv|hdrip"
+    r"|dvd-?(rip|r|scr)?|hdcam|telesync|screener|ppv|re-?encode|extras?"
+    r"|ddp? ?\d(\.\d)?|dd\d(\.\d)?|dts-?hd(\.ma)?|dts|truehd|atmos|e-?ac-?3?|ac-?3"
+    r"|aac\d?(\.\d)?|lpcm|flac|opus|mp3|pcm|\d\.\d(\.\d)?|5 ?1|7 ?1|2 ?0|1 ?0|6 ?1|4 ?0"
+    r"|subs?-?(it|en|eng|ita)?|subbed|dual(-?audio)?|multi|dub(bed)?"
+    r"|ita|eng|en|it|de|fr|es|pl|jp|jap|lat|esp|ger|fre|rus|zho|hin|tam|tel"
+    r"|repack|proper|internal|limited|unrated|extended|director'?s|cut|remaster(ed)?|uncut"
+    r"|special|edition|version|crc32|sha1|sfv|par2|nfo|nzb|mux|complete|unpack"
+    r"|certified|scene|p2p|part\d*|cd\d+|disc\d+|disk\d+|vob|idx|srt|ass|ssa|sup"
+    r"|m3u|cue|log|txt|mkv|mp4|avi|imdb\d[\d.]*|yts(\.\w+)?|rarbg|collection"
+    r")$", re.I)
+RES_RE = re.compile(r"^\d{3,5}p$|^\d{1,2}k$|^\d+x\d+$", re.I)
+YEAR_RE = re.compile(r"^(19|20)\d{2}$")
+SEP_RE = re.compile(r"[._\s\-\[\](){}+!~,;:&@#/*'\"`´|]+")
+IDX_RE = re.compile(r"^\d{1,3}[a-z]?$")
+WORD_RE = re.compile(r"[A-Za-z0-9']+$")
+EP_RE = re.compile(r"(?i)(?<![a-z0-9])(s(?:eason[\s._-]*)?\d{1,2}[\s._-]*e(?:p(?:isode)?[\s._-]*)?"
+                   r"\d{1,3}|s\d{1,2}\s?e\d{1,3}|\d{1,2}x\d{2,3}|part[\s._-]*\d{1,2})")
+STOP = {"a", "an", "the", "il", "la", "le", "i", "un", "une", "el", "los", "las", "de",
+        "del", "du", "di", "e", "ed", "y", "and", "et", "for", "of", "to", "in", "on",
+        "vs", "avec", "con"}
+
+
+def norm(s):
+    s = unicodedata.normalize("NFKD", s)
+    return "".join(c for c in s if not unicodedata.combining(c)).lower().strip()
+
+
+def fold_sep(s):
+    """norm(), plus: every character that is left over is not a letter, digit or
+    apostrophe, so it is a SEPARATOR whatever codepoint it hides behind — not part
+    of the word. Release groups file titles with lookalikes for punctuation the
+    filesystem forbids: "V ∕ H ∕ S ∕ 2" carries U+2215 DIVISION SLASH, "S꞉ Viral"
+    U+A789, and en dashes hide in "Star Wars – A New Hope". NFKD does NOT fold
+    those (no decomposition mapping), and the scrubber inside title_of DELETES
+    what it does not recognise, so five different V/H/S films squashed into the one
+    title "vhs2" and would have shared one folder. Deleting is the wrong operation:
+    they separate. Accents are gone by then — norm() removed the combining marks —
+    so "Amélie" stays "amelie" and never splits into "ame lie"."""
+    return " ".join("".join(c if c.isalnum() or c == "'" else " "
+                            for c in norm(s)).split())
+
+def title_of(path):
+    """(title, year) from the basename.
+
+    Two passes. (1) Tokenize; every 4-digit year becomes a sentinel in place.
+    (2) Find the boundary: the first release-noise token (resolution / codec /
+    source / audio / language / group vocabulary) ends the title, so trailing
+    tags can never leak in. (3) Collect up to that boundary: leading index
+    numbers ("03."), leading year sentinels ("1940 - Fantasia") and leading noise
+    are skipped while nothing is collected; once a real word appears a sentinel
+    CLOSES the title. A title sitting BEFORE the year sentinel ("300.2006.…" or
+    "12. Monkeys. 1995") is a numeric title and is prepended to what follows.
+    """
+    # fold_sep BEFORE tokenizing, so a lookalike separator separates (see its
+    # docstring): the tokens below are then plain alnum/apostrophe words.
+    name = fold_sep(os.path.splitext(os.path.basename(path))[0])
+    raw = [x for x in SEP_RE.split(name) if x]
+    years = [t for t in raw if YEAR_RE.match(t)]
+    two_year = len(years) > 1                       # "2001…1968": one of them is the title
+    toks, year = [], None
+    for t in raw:
+        if YEAR_RE.match(t):
+            year = t                                    # last year wins
+            toks.append("\x00")
+        else:
+            toks.append(t)
+    stop, seen_word = len(toks), False
+    for i, t in enumerate(toks):
+        if t == "\x00":
+            continue
+        c = norm(re.sub(r"[^A-Za-z0-9']", "", t))
+        if (not c or RES_RE.match(c) or JUNK_RE.match(c) or c in JUNK
+                or not WORD_RE.match(c)) and not (IDX_RE.match(t) and not seen_word):
+            stop = i
+            break
+        if c.isalpha():
+            seen_word = True
+    # A leading number is a track index ("01. Bye Bye Baby", "05. Casablanca") unless
+    # it is three digits or more ("007", "000", "1408") or the name carries no year
+    # at all ("12 Monkeys", "831"). A 1-2 digit prefix beside a year is an index;
+    # the rare real title that looks like one ("0 Things I Hate About You (1999)")
+    # loses its zero to the parser — Plex's matched title overrides it anyway.
+    lead = re.split(r"[\s._-]", name, maxsplit=1)[0]
+    track_idx = bool(re.fullmatch(r"\d{1,2}[a-z]?", lead)) and bool(year)
+    keep, num_title = [], None
+    for t in toks[:stop]:
+        if t == "\x00":
+            if keep:
+                break                                   # year closes the title
+            if two_year and t is not None and years[0] != year:
+                # two DIFFERENT years ("2001…1968"): the leading one is the title;
+                # "1940 - Fantasia (1940)" repeats one year, so it is not a title
+                num_title = num_title or years[0]
+            continue                                    # leading year: not the title
+        if track_idx and IDX_RE.match(t) and not keep and t == lead:
+            continue                                    # "03." / "1c." ordering
+        c = norm(re.sub(r"[^A-Za-z0-9']", "", t))
+        if not keep and not c.isalpha():
+            num_title = num_title or (c or None)
+            continue                                    # numeric title candidate
+        keep.append(c)
+        if len(keep) >= 8:
+            break
+    keep = [k for k in keep if not re.fullmatch(r"(disc|disk|cd|dvd|part)\d*", k)]
+    title = " ".join(keep)
+    if not keep:
+        # nothing collected: a pure-numeric leading token is the title even when it
+        # sits where the release year sits ("1917.2019.2160p…"), and "300.2006…" too
+        for t in raw:
+            c = norm(re.sub(r"[^A-Za-z0-9']", "", t))
+            if re.fullmatch(r"(?:[12]\d{3}|\d{1,3})", c) and (c != year or two_year):
+                return c, year or "?"    # never "2160p" / "1080": they are not pure digits
+    if (num_title and title and not keep[0].isdigit()
+            and (two_year or not track_idx)):
+        # a leading number that only echoes the release year ("1917.2019.2160p…") is
+        # not a title; one that stands alone ("300.2006…", "1900") is
+        title = f"{num_title} {title}"
+    title = title or num_title or norm(name).replace("\x00", " ").strip()[:40]
+    return title, year or "?"
+
+
+# Bonus-material vocabulary: names that say nothing about the film, so an extras
+# file otherwise pairs with an extras file of another movie. "Short", "Spot" and
+# "Clip" are deliberately absent — they are film titles (Short Circuit, Spot,
+# Clips) — and the running-time guard in is_extra() carries the rest.
+SEP_CLS = r"[\s._-]+"
+
+
+def _bonus_rx(word):
+    return r"(?<![a-z0-9])" + re.sub(r"[ \-]+", lambda _: SEP_CLS, word.lower()) + r"(?![a-z0-9])"
+
+
+EXTRA_WORDS = [
+    "trailer", "teaser", "featurette", "featurettes", "promo", "promotional",
+    "tv spot", "advert", "commercial", "behind the scenes", "making of",
+    "deleted scene", "deleted scenes", "deleted footage", "gag reel", "blooper",
+    "blooper reel", "bloopers", "outtake", "outtakes", "scene selection", "menu",
+    "menus", "gallery", "still gallery", "stills", "poster", "posters",
+    "montage", "preview", "previews", "epilogue", "visual effects",
+    "music video", "singalong", "karaoke", "trivia", "storyboard", "workprint",
+    "test footage", "screen test", "audition", "cast and crew", "isolated score",
+    "picture in picture", "original score", "session piece", "sample",
+    "watermark",
+]
+# Words that describe bonus material which can run LONG (a feature-length
+# commentary-track cut, a documentary about the film). Trusted only below
+# LONG_BONUS_MAX, so a 40-minute "commentary" release stays a film.
+EXTRA_LONG_WORDS = ["interview", "interviews", "commentary", "video archive",
+                    "archival", "documentary"]
+EXTRA_NAME_RE = re.compile(
+    "|".join([_bonus_rx(w) for w in EXTRA_WORDS] + [r"scene[\s._-]?\d{1,3}"]), re.I)
+EXTRA_LONG_NAME_RE = re.compile(
+    "|".join(_bonus_rx(w) for w in EXTRA_LONG_WORDS), re.I)
+# Bonus-folder vocabulary. fullmatch = the folder IS an extras subtree; a bare
+# search = it only mentions extras ("... + Extras" box sets hold feature films).
+EXTRA_FOLDER_RE = re.compile(
+    r"(?:extras?|bonus(?:[\s._-]material)?|specials?|special[\s._-]featurettes?|"
+    r"featurettes?|deleted(?:[\s._-](?:scenes?|footage))?|trailers?|"
+    r"making[\s._-]of|behind[\s._-]the[\s._-]scenes|shorts?|interviews?|"
+    r"archiv(?:al|es)|video[\s._-]archive)", re.I)
+VIDEO_SUFFIX = (".mkv", ".mp4", ".avi", ".m4v", ".mov", ".m2ts", ".mpg", ".webm")
+
+# Anything longer than this is a feature film whatever it is called: a 118-minute
+# file named "The Menu" or "Trailer" is a movie, not bonus material.
+SHORT_MAX = 45 * 60
+
+# Long-shaped bonus vocabulary is only believed below this: a 40-minute
+# commentary cut is an alternate version of the film, not a featurette.
+LONG_BONUS_MAX = 25 * 60
+
+
+def name_words(x):
+    """Bare words of a file or folder name: no release metadata, no year."""
+    x = os.path.splitext(x)[0] if x.lower().endswith(VIDEO_SUFFIX) else x
+    return [w for w in SEP_RE.split(fold_sep(x))
+            if w and not YEAR_RE.match(w) and not RES_RE.match(w)
+            and not JUNK_RE.match(w) and w not in JUNK]
+
+
+def mentions(short, long_):
+    """True when one word list appears in the other, anywhere: a release names
+    the film at the front ("Alien (1979) - Trailer") or in the middle
+    ("Movies/Alien (1979)/Extra - Logo.mkv"), so a start-of-string rule fails."""
+    a, b = sorted([short, long_], key=len)
+    if not a:
+        return False
+    return any(b[i:i + len(a)] == a for i in range(len(b) - len(a) + 1))
+
+
+
+def folder_is_bonus(folder):
+    """True when a folder name carries bonus vocabulary as WHOLE words: the
+    vocabulary is matched against a padded word string, so "Extras" decides and
+    a word that merely CONTAINS one ("Extraordinary", "The Extras") does not."""
+    return bool(EXTRA_FOLDER_RE.search(" " + " ".join(SEP_RE.split(norm(folder))) + " "))
+
+
+def is_extra(path, title="", dur=0.0, root=ROOT, matched=False):
+    """True for bonus material, which must never be counted as a film: a file
+    called "Trailer", "Menu", "Sample" or "Gallery" carries no film identity, so
+    extras otherwise pair with each other across the whole library.
+
+    Evidence, strongest first:
+      * RUNTIME VETOES EVERYTHING. The film "The Menu", the film "Trailer", the
+        franchise "Trailer Park Boys" and "ET — The Extra-Terrestrial" all carry
+        bonus vocabulary and run 100+ minutes: anything over SHORT_MAX is a film —
+        unless it sits in an explicit bonus folder and metadata never matched it.
+        Bonus words that describe long-form material (commentary, documentary,
+        interview) are believed only
+        under LONG_BONUS_MAX, so a feature-length commentary cut stays a film.
+      * The file's OWN name carries bonus vocabulary -> bonus.
+      * ANY folder of the path below the film's own root folder carries bonus
+        vocabulary ("Alien (1979)/Extras/", ".../Featurettes/Deleted Scenes/",
+        "Angel's Egg (1985)/04b. Angel's Egg Extras/") -> bonus. The ROOT folder
+        is exempt because it names the film, and a film may be titled with a
+        bonus word; and a file whose own name IS that folder's film name is the
+        feature, not the bonus ("Ghost in the Shell Movies/Extras/
+        Ghost in the Shell - CM3.mp4" is a bonus, but a whole other film filed
+        there stays a film).
+    Bonus vocabulary is matched as whole words only, so the film
+    "Angel's Egg - Extra - Logo" inside "04b. Angel's Egg Extras" is reached by
+    the folder rule, and "…Extras" as a fragment of a word is never evidence.
+    """
+    parts = [q for q in path.split("/") if q]
+    if not parts:
+        return False
+    name = (os.path.splitext(parts[-1])[0]
+            if parts[-1].lower().endswith(VIDEO_SUFFIX) else "")
+    # A bonus folder anywhere BELOW the film's own folder decides for its
+    # contents; the film's own folder is the film's name, whatever words it
+    # contains, so it is exempt. It is the first component AFTER ROOT's own
+    # components — those are the mount path, not folders of the library.
+    head = len([q for q in root.split("/") if q]) + 1
+    bonus_folder = False
+    for folder in parts[head:-1]:
+        if folder_is_bonus(folder):
+            bonus_folder = not mentions(name_words(name), name_words(folder))
+            break
+    if dur and dur > SHORT_MAX:
+        # Runtime vetoes vocabulary, with ONE exception: an explicit bonus folder
+        # and no metadata match. A 90-minute "Feature Length Rough Cut" sitting in
+        # a movie's Featurettes/ is a disc extra, and nothing in the library claims
+        # it as a film. Long-form bonus words still describe a film, so a
+        # feature-length commentary cut or documentary keeps its runtime veto.
+        if bonus_folder and not matched and not EXTRA_LONG_NAME_RE.search(name):
+            return True
+        return False
+    if EXTRA_NAME_RE.search(name):
+        return True
+    if dur and dur <= LONG_BONUS_MAX and EXTRA_LONG_NAME_RE.search(name):
+        return True
+    return bonus_folder
+
+# Release-group watermark clips: a group drops a tiny ad ("ETRG.mp4",
+# "RARBG.com.mp4") into many different films, so its name carries no film
+# identity at all and it would otherwise pair with itself across the library.
+WATERMARK_MAX = 20 * 2**20
+WATERMARK_RE = re.compile(r"(?i)^(rarbg(\.com)?|etrg|tgx|fbt|cpbg|yify|yts|"
+                          r"rarbg\.com|playnomore|imprs|gauntlet|dfx|macmote|"
+                          r"psi-axiz|acm|tlf|sa8|ebp|gtv|w4g|don|aAF|enjoy|hd4u)$")
+
+
+def is_watermark(path, size):
+    """True for a release group's own watermark video, not a film or an extra."""
+    return (size < WATERMARK_MAX
+            and bool(WATERMARK_RE.match(os.path.splitext(os.path.basename(path))[0])))
+
+
+
+def key_of(title):
+    """Franchise key: first two alphabetic tokens (numeric-only title = itself).
+    Deliberately NOT film-specific — it only selects candidates for the duration
+    check, so a two-token collision (two films starting "star trek") cannot by
+    itself create a duplicate group."""
+    toks = [t for t in title.split() if t]
+    words = [t for t in toks if not t.isdigit()]
+    if not words:
+        return " ".join(toks)
+    sig = [t for t in words if t not in STOP] or words
+    return " ".join(sig[:2])
+
+
+def ep_of(path):
+    """(show, episode-token) when the basename carries SxxEyy / 1x04 / Part N."""
+    name = os.path.splitext(os.path.basename(path))[0]
+    m = EP_RE.search(name)
+    if not m:
+        return None
+    show, _ = title_of(path)
+    return show, norm(m.group(1)).replace(" ", "")
+
+
+def base_words(title, key):
+    """Title tokens beyond the franchise key — the FILM identity part.
+    "evil dead rise" under key "evil dead" -> {rise}; "28 days later" -> {}."""
+    kt = set(key.split())
+    return frozenset(t for t in title.split() if t not in kt and t not in STOP
+                     and not t.isdigit() and not YEAR_RE.match(t))
+
+
+def folder_year(path):
+    """Soft year evidence: a 19xx/20xx token in any parent directory name
+    (collection folders are written both "(2002)" and "[2009]"; a range such as
+    "(1998-2000)" deliberately yields nothing)."""
+    for d in reversed(os.path.dirname(path).split("/")):
+        ys = [y for y in re.split(r"[^0-9]+", d) if YEAR_RE.match(y)]
+        if len(ys) == 1:
+            return ys[0]
+    return None
+
+
+def same_film(a, b):
+    """Two files are copies of ONE film only when the film-identity tokens agree
+    (one side may carry none) AND no year evidence contradicts them. The key says
+    the franchise matches; this says the actual film matches — it is what keeps
+    'Evil Dead' (2013) apart from 'Evil Dead Rise' (2023) and the four Austin
+    Powers apart when their runtimes happen to agree."""
+    ba, bb = a["base"], b["base"]
+    if ba and bb and ba != bb:
+        return False
+    ya = a["year"] if a["year"] != "?" else a["soft"]
+    yb = b["year"] if b["year"] != "?" else b["soft"]
+    if ya and yb and ya != yb:
+        return False
+    return True
+
+
+def dur_close(a, b):
+    if a <= 0 or b <= 0:
+        return True                                   # unknown duration: let titles decide
+    return abs(a - b) <= max(2.0, 0.003 * max(a, b))
+
+
+# Blu-ray folder trees: ffprobe only sees the clip it is pointed at, so each tree
+# is folded into ONE entry, described by its longest main-feature clip (measured
+# with ffprobe on the largest BDMV/STREAM file) and sized by the whole tree.
+BDMV_MAIN = {
+    "Avengers.Age.of.Ultron.2015.2160p.BluRay.HEVC.TrueHD.7.1.Atmos-TERMiNAL":
+        dict(dur=8478.2, vcodec="hevc", res="3840x2160", pix="yuv420p10le", fps=23.976),
+    "Full.Metal.Jacket.1987.2160p.BluRay.HEVC.DTS-HD.MA.5.1-EATDIK":
+        dict(dur=6993.3, vcodec="hevc", res="3840x2160", pix="yuv420p10le", fps=23.976),
+}
+
+
+def load(probed):
+    """Probe rows -> film entries. AppleDouble (._*) sidecars are dropped; each
+    BDMV tree collapses to one entry carrying the tree's total size."""
+    trees = {}
+    rows = []
+    for line in open(probed):
+        f = line.rstrip("\n").split("\t")
+        if len(f) < 12:
+            continue
+        b, p, dur, vc, res, fps, vbit, pix, aud, subs, _d, fmt = f[:12]
+        base = os.path.basename(p)
+        if base.startswith("._") or base == ".DS_Store":
+            continue                                    # macOS sidecar, not media
+        m = re.search(r"/(BDMV|VIDEO_TS)/", p)
+        root = p[:m.start()] if m else None
+        if root or re.search(r"/(CERTIFICATE|AACS|MLPI|CLIPINF|PLAYLIST|BACKUP|REGISTRL)/", p):
+            if not root:
+                continue      # disc bookkeeping: folds into its tree, never a film
+        if root:
+            t = trees.setdefault(root, dict(bytes=0, audio=set(), subs=0))
+            t["bytes"] += int(b)
+            t["audio"].update(a for a in aud.split("+") if a)
+            t["subs"] += int(subs or 0)
+            continue
+        try:
+            dur = float(dur)
+        except ValueError:
+            dur = 0.0
+        rows.append(dict(bytes=int(b), path=p, dur=dur, vcodec=vc, res=res, fps=fps,
+                         vbit=int(float(vbit or 0)), pix=pix, audio=aud, subs=subs, fmt=fmt))
+    for root, t in trees.items():
+        folder = os.path.basename(root)
+        m = BDMV_MAIN.get(folder, {})
+        rows.append(dict(bytes=t["bytes"], path=root + "/BDMV/index.bdmv",
+                         name_src=folder, dur=m.get("dur", 0.0),
+                         vcodec=m.get("vcodec", "hevc"), res=m.get("res", "-"),
+                         fps=m.get("fps", 0), vbit=0, pix=m.get("pix", "-"),
+                         audio="+".join(sorted(t["audio"])), subs=str(t["subs"]),
+                         fmt="BDMV tree"))
+    for r in rows:
+        r["title"], r["year"] = title_of(r.get("name_src") or r["path"])
+        r["key"] = key_of(r["title"])
+        r["base"] = base_words(r["title"], r["key"])
+        r["soft"] = folder_year(r["path"])
+        r["ep"] = ep_of(r["path"])
+    return rows
+
+
+def dur_clusters(items):
+    """Group files that are the SAME FILM and the SAME MASTER duration. Candidate
+    buckets come from the franchise key; same_film then rejects the different
+    installments whose runtimes happen to agree."""
+    clusters = []
+    for r in sorted(items, key=lambda r: -r["dur"]):
+        for c in clusters:
+            if dur_close(c[0]["dur"], r["dur"]) and same_film(c[0], r):
+                c.append(r)
+                break
+        else:
+            clusters.append([r])
+    return clusters
+
+
+def tier(groups):
+    """Split same-film/same-duration clusters: DUP = different releases of one
+    film (real, deletable redundancy); REVIEW = identical size AND duration
+    (byte-twins — the hash pass is the authority) or clashing year evidence."""
+    dup, review = [], []
+    for g in groups:
+        g.sort(key=lambda r: -r["bytes"])
+        years = {(r["year"] if r["year"] != "?" else r["soft"]) for r in g}
+        years.discard(None)
+        spread = (g[0]["bytes"] - g[-1]["bytes"]) / g[0]["bytes"] if g[0]["bytes"] else 0
+        if len(years) > 1 or spread < 0.005:
+            review.append(g)
+        else:
+            dup.append(g)
+    return dup, review
+
+
+def redun(g):
+    return sum(r["bytes"] for r in sorted(g, key=lambda r: -r["bytes"])[1:])
+
+
+def line(r):
+    return (f"{r['bytes']/2**30:8.2f} GiB {r['res']:>9} {r['vcodec']:<7} "
+            f"{r['pix']:<11} {r['dur']/60:5.0f}m {r['audio'][:34]:<34}")
+
+
+def dump(tag, groups, out, note=""):
+    for gi, g in enumerate(groups, 1):
+        g.sort(key=lambda r: -r["bytes"])
+        print(f"\n### {tag}-{gi}. {g[0]['title']} ({g[0]['year']}) — {len(g)} files, "
+              f"{redun(g)/2**30:.1f} GiB redundant {note}".rstrip())
+        for rank, r in enumerate(g):
+            print(f"  {line(r)} [{r['year']}] "
+                  f"{r['path'].replace(ROOT + '/', '')}")
+            out.write(f"{tag}\t{gi}\t{rank}\t{r['bytes']}\t{r['res']}\t{r['vcodec']}\t"
+                      f"{r['pix']}\t{int(r['dur'])}\t{r['audio']}\t{r['title']}|{r['year']}\t"
+                      f"{r['path']}\n")
+
+
+def report(probed, out_path):
+    rows = load(probed)
+    for r in rows:
+        r["mark"] = is_watermark(r["path"], r["bytes"])
+        r["extra"] = not r["mark"] and is_extra(r["path"], r["title"], r["dur"])
+    marks = [r for r in rows if r["mark"]]
+    movies = [r for r in rows if not r["ep"] and not r["extra"] and not r["mark"]]
+    episodes = [r for r in rows if r["ep"] and not r["extra"] and not r["mark"]]
+    extras = [r for r in rows if r["extra"]]
+    print(f"census: {len(rows)} video entries = {len(movies)} films + {len(episodes)} "
+          f"TV episodes + {len(extras)} bonus/trailers "
+          f"({sum(r['bytes'] for r in extras)/2**30:.1f} GiB) "
+          f"+ {len(marks)} release-group watermarks")
+
+    out = open(out_path, "w")
+
+    buckets = collections.defaultdict(list)
+    for r in movies:
+        buckets[r["key"]].append(r)
+    cand = {k: v for k, v in buckets.items() if len(v) > 1}
+    clusters = [c for v in cand.values() for c in dur_clusters(v) if len(c) > 1]
+    dup, review = tier(clusters)
+    dup.sort(key=lambda g: -redun(g))
+    review.sort(key=lambda g: -redun(g))
+    dump("DUP", dup, out)
+    dump("REVIEW", review, out, "— same key+duration, sizes near-identical OR years clash: RECHECK")
+
+
+    ebuf = collections.defaultdict(list)
+    for r in episodes:
+        ebuf[(r["ep"][0], r["ep"][1])].append(r)
+    edup, ereview = tier([c for v in ebuf.values() if len(v) > 1
+                          for c in dur_clusters(v) if len(c) > 1])
+    edup.sort(key=lambda g: -redun(g))
+    dump("EP-DUP", edup, out)
+
+    shows = collections.Counter()
+    for r in episodes:
+        shows[r["ep"][0]] += 1
+    multi = {s: n for s, n in shows.items() if n > 4}
+    print("\n=== TV-series content inside Movies/ (>4 episodes) ===")
+    for s, n in sorted(multi.items(), key=lambda kv: -kv[1]):
+        print(f"  {n:4d} episodes  {s}")
+    out.close()
+
+    print(f"\nDUP groups={len(dup)} files={sum(len(g) for g in dup)} "
+          f"redundant={sum(redun(g) for g in dup)/2**40:.2f} TiB", file=sys.stderr)
+    print(f"REVIEW groups={len(review)} "
+          f"redundant={sum(redun(g) for g in review)/2**40:.2f} TiB", file=sys.stderr)
+    print(f"EP-DUP groups={len(edup)} "
+          f"ep-redundant={sum(redun(g) for g in edup)/2**30:.1f} GiB", file=sys.stderr)
+
+
+def parse(names):
+    for n in names:
+        t, y = title_of(n)
+        print(f"{t!r} ({y})\tkey={key_of(t)!r}\t{n[:70]}")
+
+
+def main():
+    ap = argparse.ArgumentParser(description=__doc__.splitlines()[0])
+    sub = ap.add_subparsers(dest="cmd", required=True)
+    rep = sub.add_parser("report", help="similar-copy census over a probe TSV")
+    rep.add_argument("--probed", default="/tmp/media-probed.tsv")
+    rep.add_argument("--out", default="/tmp/media-similar.tsv")
+    par = sub.add_parser("parse", help="show the parsed title/year of a name")
+    par.add_argument("names", nargs="+")
+    a = ap.parse_args()
+    if a.cmd == "report":
+        report(a.probed, a.out)
+    else:
+        parse(a.names)
+
+
+if __name__ == "__main__":
+    main()

scripts/qwen38-calib-window.sh

@@ -0,0 +1,143 @@
+#!/usr/bin/env bash
+# qwen38-calib-window.sh — RUNS ON DANKSERVER. The calibration window, and nothing else:
+# boot the BF16 rig, wait until it is actually serving, push the corpus until the engine says
+# the tap is full, stop the rig, restore frank-archon, and bank the reference container.
+#
+# The reference recording is already taken (kld-ref/kld.json exists), so this window is the
+# calibration tap alone — the one job that needs both cards after it. The kldref phase is NOT
+# re-run here: the chained launcher that used to do that moved the banked kld-ref into trash
+# first, which is how a 16-hour phase gets paid for twice.
+#
+# Every phase is gated on the previous one, because the last time this ran unguarded
+# (2026-10-29) the rig's boot died of an OOM, the health loop quietly fell through, and the
+# pusher fed four rounds of 10M tokens into RESTORED PRODUCTION with nothing recorded:
+#
+#   1. the runner must still be alive and its container must answer /health AND own the
+#      listening socket, or this script kills it, restores the service and exits non-zero —
+#      no push, ever. The socket clause is not decoration: the runner stops frank-archon
+#      first, the dying process keeps its listener for a moment, and a rig container that
+#      is still loading has not bound the port yet, so `podman inspect` + `/health` both
+#      report success for the WRONG process (measured 2026-10-30: `health ok waited=0s`,
+#      then the pusher refused on the container name and the window died at second zero);
+#   2. the pusher refuses on its own unless the rig wrote its marker into the tap dir, the
+#      marker names this container, and the container is running (scripts/calib-push.py
+#      --tap-marker), so a second lock survives a mistake in this file;
+#   3. the reference container is banked into trash only when BOTH the serve and the push
+#      exited 0 and the tap dir holds grams. The checkpoint that reproduces it stays.
+#
+# knobs: Q38_WINDOW_CALIB_CORPUS (calib-7168.jsonl — the re-chunked file the rig's pool can
+#        hold), Q38_WINDOW_HEALTH_TIMEOUT (seconds to wait for /health), Q38_WINDOW_ROUNDS
+#        (pusher rounds; one is 10M tokens), Q38_WINDOW_KEEP_RAD=1 to leave the .rad in place.
+# Usage: scripts/qwen38-calib-window.sh [--dry-run]
+set -uo pipefail
+cd "$(dirname "${BASH_SOURCE[0]}")/.." || exit 9
+
+DRY=0
+[ "${1:-}" = "--dry-run" ] && DRY=1
+say() { printf '%s %s\n' "$(date -Is)" "$*"; }
+
+MODELS="${FRANK_QWEN38_MODELS:-/home/dank/models}"
+CAMPAIGN="${FRANK_QWEN38_CAMPAIGN:-/home/dank/campaign/qwen38-w5nl64}"
+RAD="${FRANK_QWEN38_RAD:-$MODELS/rad/qwen38-bf16.rad}"
+CORPUS="${Q38_WINDOW_CALIB_CORPUS:-/home/dank/corpora/out/calib-7168.jsonl}"
+CALIB="${FRANK_QWEN38_CALIB_DIR:-$CAMPAIGN/calib}"
+HEALTH_TIMEOUT="${Q38_WINDOW_HEALTH_TIMEOUT:-3600}"
+ROUNDS="${Q38_WINDOW_ROUNDS:-6}"
+RIG=qwen38-bf16
+SERVICE=frank-archon
+PY="${PY:-python3}"
+
+# ---- preflight: every fact by identity, one owner (scripts/qwen38-preflight.sh) ----
+# The checks that used to live here were proxies, and each lied at least once: the space check
+# asked a directory that merely LOOKED related, `/health` answered for a dying production
+# process, `podman inspect` named a container that existed but had bound nothing. The shared
+# script binds each fact to its object — cwd, the digest the tag actually resolves to, free
+# space on the tap's own filesystem, every /dev/kfd holder's HOST pid against the service's,
+# the :7770 listener's host pid, the corpus and the reference recording. A resumed window
+# keeps an existing tap, so the tap-empty fact is waived.
+[ -f "$RAD" ]    || { echo "REFUSED: no BF16 reference container at $RAD — nothing to serve; scripts/build-qwen38-bf16-rad.sh" >&2; exit 2; }
+FRANK_QWEN38_CAMPAIGN="$CAMPAIGN" \
+Q38_WINDOW_CALIB_CORPUS="$CORPUS" \
+Q38_CALIB_MIN_FREE_MIB="${Q38_CALIB_MIN_FREE_MIB:-61440}" \
+    bash scripts/qwen38-preflight.sh --allow-existing-tap || exit 2
+! [[ "$HEALTH_TIMEOUT" =~ ^[0-9]+$ ]] && { echo "REFUSED: Q38_WINDOW_HEALTH_TIMEOUT must be seconds (got: $HEALTH_TIMEOUT)" >&2; exit 2; }
+! [[ "$ROUNDS" =~ ^[0-9]+$ ]] && { echo "REFUSED: Q38_WINDOW_ROUNDS must be a round count (got: $ROUNDS)" >&2; exit 2; }
+
+if [ "$DRY" = 1 ]; then
+    echo "dry run: calib window — the service is left running, nothing is mounted or written"
+    echo "would run: bash scripts/run-qwen38-bf16.sh calib          (rig boots, writes $CALIB/.rig-boot)"
+    echo "would wait: /health up within ${HEALTH_TIMEOUT}s, else kill the rig, restore $SERVICE, exit 1"
+    echo "would run: $PY scripts/calib-push.py --container $RIG --corpus $CORPUS --max-rounds $ROUNDS"
+    echo "would bank: $RAD -> ~/.AGENTTRASH/$(date +%Y%m%d)-qwen38-bf16-rad/  (only if both phases exit 0)"
+    exit 0
+fi
+
+say "calib window start pid=$$ corpus=$CORPUS rounds=$ROUNDS"
+[ -n "$(ls -A "$CALIB" 2>/dev/null)" ] && say "tap dir $CALIB is not empty; the engine's token floor is pooled, so an earlier partial tap still counts"
+
+# ---- phase 1: the rig -------------------------------------------------------
+say "phase=calib serve begin"
+bash scripts/run-qwen38-bf16.sh calib & spid=$!
+
+rig_up() {
+    podman inspect -f '{{.State.Running}}' "$RIG" 2>/dev/null | grep -qx true || return 1
+    curl -sf --max-time 10 -o /dev/null http://127.0.0.1:7770/health || return 1
+    # The answer must come from the CONTAINER. The runner stops frank-archon first, and the
+    # dying production process keeps its listener for a moment while the freshly created
+    # rig container is still loading — so `container running` + `/health` 200 both hold
+    # before the rig has bound anything. Measured 2026-10-30: `health ok waited=0s`, then the
+    # pusher refused on the container name and the window died at second zero. Ask the kernel
+    # who owns the listening socket and require it to be a rig process.
+    local lpid
+    lpid=$(ss -H -ltnp 'sport = :7770' 2>/dev/null | grep -o 'pid=[0-9]*' | head -1 | cut -d= -f2)
+    [ -n "${lpid:-}" ] || return 1
+    podman top "$RIG" pid 2>/dev/null | grep -qw "$lpid"
+}
+
+waited=0
+until rig_up; do
+    if ! kill -0 "$spid" 2>/dev/null; then
+        wait "$spid"; crc=$?
+        say "phase=calib serve end rc=$crc (the rig exited before it ever answered /health)"
+        echo "FAIL: the rig boot died (rc=$crc) — nothing was pushed; read /home/dank/logs/qwen38_bf16_calib.log" >&2
+        exit 1
+    fi
+    if [ "$waited" -ge "$HEALTH_TIMEOUT" ]; then
+        say "phase=calib health never came after ${HEALTH_TIMEOUT}s; killing the rig"
+        podman stop "$RIG" >/dev/null 2>&1 || true
+        wait "$spid"
+        echo "FAIL: the rig never answered /health within ${HEALTH_TIMEOUT}s — nothing was pushed" >&2
+        exit 1
+    fi
+    sleep 15
+    waited=$((waited + 15))
+done
+say "phase=calib health ok waited=${waited}s"
+
+# ---- phase 2: the tap -------------------------------------------------------
+say "phase=calib push begin"
+"$PY" scripts/calib-push.py --port 7770 --parallel 8 --container "$RIG" \
+    --tap-dir "$CALIB" --corpus "$CORPUS" --max-rounds "$ROUNDS"
+prc=$?
+say "phase=calib push end rc=$prc"
+
+podman stop "$RIG" >/dev/null 2>&1 || true
+wait "$spid"; crc=$?
+say "phase=calib serve end rc=$crc"
+
+if [ "$prc" -ne 0 ] || [ "$crc" -ne 0 ]; then
+    echo "FAIL: serve rc=$crc push rc=$prc — the tap is NOT banked and $RAD stays: $(basename "$RAD")" >&2
+    exit 1
+fi
+[ -n "$(ls -A "$CALIB" 2>/dev/null)" ] || { echo "FAIL: both phases exited 0 but the tap dir $CALIB is empty" >&2; exit 1; }
+say "TAP BANKED: $(ls -1 "$CALIB" | wc -l) file(s) in $CALIB"
+
+# ---- phase 3: bank the reference container ----------------------------------
+if [ "${Q38_WINDOW_KEEP_RAD:-0}" = "1" ]; then
+    say "reference .rad kept (Q38_WINDOW_KEEP_RAD=1)"
+else
+    T="${Q38_WINDOW_TRASH_ROOT:-$HOME/.AGENTTRASH}/$(date +%Y%m%d)-qwen38-bf16-rad"
+    mkdir -p "$T" || { echo "FAIL: cannot create trash dir $T" >&2; exit 1; }
+    mv "$RAD" "$T/" && say "reference .rad banked in $T (the checkpoint that rebuilds it stays)"
+fi
+say "calib window end"

scripts/qwen38-preflight.sh

@@ -0,0 +1,136 @@
+#!/usr/bin/env bash
+# qwen38-preflight.sh — the facts a calibration window must have BEFORE it takes the cards,
+# each checked by IDENTITY and not by proxy. Every line is PASS or FAIL with the reason.
+#
+# Why this file exists: on 2026-10-30 a window died at second zero because `/health` answered
+# 200 from the *production* process that was still draining its listener, while the freshly
+# created rig container had not bound anything. Other runs died from a port that answered for
+# the wrong process, a /dev/kfd held by an orphaned engine, a tap dir on a 35 GiB filesystem,
+# and a launcher whose cwd silently lacked the repo. None of those were math problems; all of
+# them were a check that measured something adjacent to the thing it meant.
+#
+# Usage:  bash scripts/qwen38-preflight.sh [--allow-existing-tap]
+# Exit:   0 all PASS · 2 any FAIL · 3 a fact could not be evaluated (missing tool)
+set -uo pipefail
+
+SELF_DIR="$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)"
+REPO_ROOT="$(cd "$SELF_DIR/.." && pwd)"
+
+CAMPAIGN="${FRANK_QWEN38_CAMPAIGN:-/home/dank/campaign/qwen38-w5nl64}"
+TAP="$CAMPAIGN/calib"
+CORPUS="${Q38_WINDOW_CALIB_CORPUS:-/home/dank/corpora/out/calib-7168.jsonl}"
+NEED_MIB="${Q38_CALIB_MIN_FREE_MIB:-61440}"
+ALLOW_EXISTING=0
+[ "${1:-}" = "--allow-existing-tap" ] && ALLOW_EXISTING=1
+
+# The producer pin is the calib image; the window fails closed if the tag on this host is not it.
+WANT_DIGEST="${Q38_PREFLIGHT_IMAGE_DIGEST:-sha256:69add58f7de1ed2bca2b1e20a4cc75717557df3f73569a178eca5aafe8c44093}"
+IMAGE_REF="${Q38_CALIB_IMAGE:-stilldeadcode/radiance@$WANT_DIGEST}"
+SERVICE="frank-archon"
+PORT=7770
+
+fails=0
+ok()   { printf 'PASS  %-14s %s\n' "$1" "$2"; }
+bad()  { printf 'FAIL  %-14s %s\n' "$1" "$2"; fails=$((fails + 1)); }
+skip() { printf 'WARN  %-14s %s\n' "$1" "$2"; }
+
+# 1. cwd — a detached launcher that lost its cwd fails here in a second, not in the log later.
+if [ "$(cd "$REPO_ROOT" && pwd)" = "$(pwd)" ] && [ -x "$REPO_ROOT/scripts/run-qwen38-bf16.sh" ]; then
+    ok cwd "running from the repo root ($REPO_ROOT)"
+else
+    bad cwd "cwd is $(pwd), not the repo root ($REPO_ROOT) — every relative script path would break"
+fi
+
+# 2. image — resolve the REF to a digest. A stale local `:latest` is a different model of the world.
+if command -v podman >/dev/null 2>&1; then
+    # An image carries EVERY digest it was pulled under (a tag plus the by-digest pull), so this
+    # asks whether the pin is one of them — reading RepoDigests[0] reports a healthy image as wrong.
+    got="$(podman image inspect -f '{{range .RepoDigests}}{{println .}}{{end}}' "$IMAGE_REF" 2>/dev/null \
+           | sed 's/.*@//' | grep -c "^${WANT_DIGEST}$")"
+    if [ "${got:-0}" -ge 1 ]; then
+        ok image "$IMAGE_REF carries ${WANT_DIGEST:0:19}…"
+    elif podman image exists "$IMAGE_REF" 2>/dev/null; then
+        bad image "$IMAGE_REF is present but does not carry $WANT_DIGEST (a retagged or older pull)"
+    else
+        bad image "$IMAGE_REF is not present on this host"
+    fi
+else
+    skip image "podman not on PATH — cannot resolve $IMAGE_REF"
+fi
+
+# 3. space — df the FILE PATH the seal writes to, never a directory that merely looks related.
+if [ -d "$TAP" ] || mkdir -p "$TAP" 2>/dev/null; then
+    avail="$(df --output=avail -B 1M "$TAP" 2>/dev/null | tail -1 | tr -d ' ')"
+    if [[ "${avail:-}" =~ ^[0-9]+$ ]] && [ "$avail" -ge "$NEED_MIB" ]; then
+        ok space "$avail MiB free on $(df --output=target "$TAP" | tail -1) (need $NEED_MIB)"
+    else
+        bad space "${avail:-unknown} MiB free on $TAP, need $NEED_MIB — one seal writes ~43 GB"
+    fi
+    # 4. tap cleanliness — stale grams from another run must never be mixed in.
+    n=$(ls -A "$TAP" 2>/dev/null | grep -c . || true)
+    if [ "$n" -eq 0 ] || [ "$ALLOW_EXISTING" = 1 ]; then
+        ok tap-empty "$n entries in $TAP"
+    else
+        bad tap-empty "$n entries in $TAP — rotate them to .AGENTTRASH first"
+    fi
+else
+    bad space "cannot create $TAP"
+fi
+
+# 5. device — every /dev/kfd holder must be a process of the service we are about to stop. An
+#    orphaned engine from a previous run makes the next boot die with `startup stopped: device
+#    error`. Identity comes from podman's own host-pid list: `podman top <ctr> pid` prints
+#    CONTAINER-internal pids (1, 7, …), which compare against nothing, and a rootless cgroup
+#    carries the container ID, never the service name — both are proxies that lie.
+PROD_CTR="${Q38_PREFLIGHT_PROD_CONTAINER:-$SERVICE}"
+service_pids() { podman top "$PROD_CTR" hpid 2>/dev/null | tail -n +2; }
+if [ -e /dev/kfd ]; then
+    mine="$(service_pids | tr '\n' ' ')"
+    foreign=""
+    for p in $(fuser /dev/kfd 2>/dev/null); do
+        case " $mine " in
+            *" $p "*) : ;;                                          # the service we will stop
+            *) foreign="$foreign $p:$(cat /proc/$p/comm 2>/dev/null)" ;;
+        esac
+    done
+    if [ -z "$foreign" ]; then
+        ok device "no foreign /dev/kfd holders"
+    else
+        bad device "foreign holders:$foreign — kill by pid, never by pattern"
+    fi
+else
+    skip device "/dev/kfd absent — not an AMD host?"
+fi
+
+# 6. port — a listener on :7770 must be a process of the service we stop, never an unknown one.
+lpid="$(ss -H -ltnp "sport = :$PORT" 2>/dev/null | grep -o 'pid=[0-9]*' | head -1 | cut -d= -f2)"
+if [ -z "${lpid:-}" ]; then
+    ok port ":$PORT is free"
+else
+    mine="$(service_pids | tr '\n' ' ')"
+    case " $mine " in
+        *" $lpid "*) ok port ":$PORT is held by $SERVICE (host pid $lpid) — the window stops it" ;;
+        *)           bad port ":$PORT is held by host pid $lpid ($(cat /proc/$lpid/comm 2>/dev/null)), not $SERVICE" ;;
+    esac
+fi
+
+# 7. corpus — the push source must exist and be the re-chunked file the window documents.
+if [ -s "$CORPUS" ]; then
+    ok corpus "$(wc -l < "$CORPUS") prompts in $CORPUS"
+else
+    bad corpus "$CORPUS is missing or empty"
+fi
+
+# 8. reference — the KL gate has nothing to compare against without it.
+if [ -s "$CAMPAIGN/kld-ref/kld.json" ]; then
+    ok reference "$(du -h "$CAMPAIGN/kld-ref/kld.json" | cut -f1) kld.json present"
+else
+    bad reference "$CAMPAIGN/kld-ref/kld.json is missing"
+fi
+
+echo
+if [ "$fails" -gt 0 ]; then
+    echo "PREFLIGHT FAILED: $fails fact(s). The window must not take the cards."
+    exit 2
+fi
+echo "PREFLIGHT PASSED: every fact checked by identity."

scripts/qwen38_common.py

@@ -0,0 +1,94 @@
+#!/usr/bin/env python3
+"""Shared plumbing for the qwen38 campaign corpus tools.
+
+The three tools move the same two shapes — a JSONL of prompt records and a
+one-artifact JSON manifest — and they all tokenize with the SAME tokenizer. That
+last part is not a detail: a KLD reference is only meaningful against the exact
+token sequence the corpus recorded, so `n_tokens` exists once, here, and every
+tool counts the same way. A window that decode/encode does not round-trip is a
+refusal, never a warning.
+
+Loaded as a sibling module. Each tool puts its own directory on `sys.path`
+first, so `python3 scripts/<tool>.py` and the driver's container mount
+(`runpy.run_path('/work/<tool>.py')`, `-v scripts:/work:ro`) both resolve it —
+run the tools by path, not piped through stdin.
+"""
+import contextlib
+import json
+import os
+import sys
+import tempfile
+
+
+def refuse(msg: str) -> None:
+    """Exit 2 with a REFUSED line — the code argparse uses for unusable input."""
+    print(f"REFUSED: {msg}", file=sys.stderr)
+    sys.exit(2)
+
+
+def load_tokenizer(path: str):
+    try:
+        from tokenizers import Tokenizer
+    except ImportError as exc:  # the corpus tools carry their own venv
+        refuse(f"tokenizers is not importable by {sys.executable} — run the "
+               f"corpus venv's python (on dankserver: "
+               f"/home/dank/venvs/radcorpus/bin/python) or set "
+               f"FRANK_QWEN38_TOKENIZER: {exc}")
+    if not os.path.isfile(path):
+        refuse(f"no tokenizer at {path}")
+    return Tokenizer.from_file(path)
+
+
+def n_tokens(tok, text: str) -> int:
+    """The corpus's own count: no special tokens added, prompt side only."""
+    return len(tok.encode(text, add_special_tokens=False).ids)
+
+
+def iter_jsonl(path: str):
+    """Yield `(lineno, record)` for every non-blank line; refuse on a bad line.
+
+    Field validation stays with the caller: the three corpora carry different
+    fields (`prompt` alone, `prompt`+`cls`, `prompt`+`score_from`+`source`), and
+    a record missing the one the tool needs is that tool's refusal to make.
+    """
+    with open(path, encoding="utf-8", errors="replace") as fh:
+        for lineno, line in enumerate(fh, 1):
+            if not line.strip():
+                continue
+            try:
+                rec = json.loads(line)
+            except json.JSONDecodeError as exc:
+                refuse(f"{path}:{lineno} is not JSON: {exc}")
+            yield lineno, rec
+
+
+def write_jsonl(path: str, rows) -> None:
+    with open(path, "w", encoding="utf-8") as fh:
+        for row in rows:
+            fh.write(json.dumps(row, ensure_ascii=False) + "\n")
+
+
+@contextlib.contextmanager
+def atomic_writer(path: str):
+    """Write `path` through a temporary sibling and rename on success.
+
+    An interrupted run then leaves no half corpus behind, and the temporary name
+    keeps the one-artifact-one-name law the driver applies to a score report.
+    """
+    d = os.path.dirname(os.path.abspath(path)) or "."
+    fd, tmp = tempfile.mkstemp(dir=d, prefix="." + os.path.basename(path) + "-")
+    try:
+        with os.fdopen(fd, "w", encoding="utf-8") as fh:
+            yield fh
+        os.replace(tmp, path)
+    except BaseException:
+        if os.path.exists(tmp):
+            os.unlink(tmp)
+        raise
+    os.chmod(path, 0o644)
+
+
+def write_json(path: str, obj) -> None:
+    with atomic_writer(path) as fh:
+        json.dump(obj, fh, indent=1)
+        fh.write("\n")

scripts/rechunk-kld-corpus.py

@@ -38,12 +38,19 @@ one-artifact-one-name law run-qwen38-bf16.sh applies to a score report. The file
 is written through a temporary name and renamed, so an interrupted run leaves no
 half corpus behind. A JSON summary of what was written goes to stdout.
 """
-
 import argparse
+import dataclasses
 import json
 import os
 import sys
-import tempfile
+
+_HERE = os.path.dirname(os.path.abspath(__file__)) if "__file__" in globals() else os.getcwd()
+if _HERE not in sys.path:
+    sys.path.insert(0, _HERE)
+
+import qwen38_common as qc
+
+MISSING = object()  # "the parent record carried no source field"
 
 
 def parse_args() -> argparse.Namespace:
@@ -69,95 +76,96 @@ def parse_args() -> argparse.Namespace:
     return ap.parse_args()
 
 
-def load_tokenizer(path: str):
+@dataclasses.dataclass(frozen=True)
-    try:
+class Window:
-        from tokenizers import Tokenizer
+    """One scoring document cut out of a parent record."""
-    except ImportError as exc:  # the corpus scripts carry their own venv
+    prompt: str
-        raise SystemExit(
+    score_from: int
-            f"REFUSED: tokenizers is not importable by {sys.executable} — run the "
+    n_tokens: int
-            f"corpus venv's python (on dankserver: /home/dank/venvs/radcorpus/bin/python) "
+    clamped: bool
-            f"or set FRANK_QWEN38_TOKENIZER: {exc}") from None
+    source: object = MISSING
-    if not os.path.isfile(path):
+
-        raise SystemExit(f"REFUSED: no tokenizer at {path}")
+    def record(self) -> dict:
-    return Tokenizer.from_file(path)
+        rec = {"prompt": self.prompt, "score_from": self.score_from}
+        if self.source is not MISSING:
+            rec["source"] = self.source
+        return rec
 
 
-def refuse(msg: str) -> None:
+def validate(a: argparse.Namespace) -> None:
-    print(f"REFUSED: {msg}", file=sys.stderr)
+    """Everything that makes this run unusable, named before a token is cut."""
-    sys.exit(2)  # 2 is the code argparse uses for unusable arguments
-
-
-def main() -> None:
-    a = parse_args()
     if a.max_doc_tokens <= 0:
-        refuse(f"--max-doc-tokens must be positive (got {a.max_doc_tokens})")
+        qc.refuse(f"--max-doc-tokens must be positive (got {a.max_doc_tokens})")
     if a.score_tail <= 0:
-        refuse(f"--score-tail must be positive (got {a.score_tail})")
+        qc.refuse(f"--score-tail must be positive (got {a.score_tail})")
     if a.max_doc_tokens <= a.score_tail:
-        refuse(f"--max-doc-tokens {a.max_doc_tokens} cannot hold a {a.score_tail}-position "
+        qc.refuse(f"--max-doc-tokens {a.max_doc_tokens} cannot hold a "
-               "score tail: every window would be all tail and no context")
+                  f"{a.score_tail}-position score tail: every window would be all "
+                  "tail and no context")
     if not os.path.isfile(a.corpus):
-        refuse(f"no corpus at {a.corpus}")
+        qc.refuse(f"no corpus at {a.corpus}")
     if os.path.realpath(a.out) == os.path.realpath(a.corpus):
-        refuse(f"--out {a.out} is the corpus itself; move it to trash first — the trash law "
+        qc.refuse(f"--out {a.out} is the corpus itself; move it to trash first — the "
-               "means the old segmentation stays recoverable")
+                  "trash law means the old segmentation stays recoverable")
     if os.path.exists(a.out) and os.path.getsize(a.out) > 0 and not a.force:
-        refuse(f"{a.out} exists and is not empty; pass --force after moving it to trash")
+        qc.refuse(f"{a.out} exists and is not empty; pass --force after moving it to trash")
 
-    tok = load_tokenizer(a.tokenizer_json)
 
+def cut(rec: dict, tok, a: argparse.Namespace,
+        where: str) -> tuple[list[Window], int]:
+    """Cut one record's token ids into windows, proving each one round-trips.
+
+    The round-trip encode is the whole reason this tool may claim the reference
+    scores the corpus's own token sequence: decoding a window to text and
+    encoding it back must return the identical ids, or the score is of a
+    different document than the one on disk.
+    """
+    ids = tok.encode(rec["prompt"], add_special_tokens=False).ids
+    windows: list[Window] = []
+    for start in range(0, max(len(ids), 1), a.max_doc_tokens):
+        window = ids[start:start + a.max_doc_tokens]
+        if not window:
+            continue
+        text = tok.decode(window)
+        if tok.encode(text, add_special_tokens=False).ids != window:
+            qc.refuse(f"{where} window at token {start} does not round-trip through "
+                      "text — the reference would score a different token sequence "
+                      "than the corpus holds")
+        windows.append(Window(prompt=text,
+                              score_from=max(len(window) - a.score_tail, 0),
+                              n_tokens=len(window),
+                              clamped=len(window) < a.score_tail,
+                              source=rec["source"] if "source" in rec else MISSING))
+    return windows, len(ids)
+
+
+def rechunk(a: argparse.Namespace, tok) -> dict:
+    """Write the re-chunked corpus and return the summary it printed."""
     docs_in = docs_out = tokens_in = tokens_out = 0
-    longest_child = 0
+    longest_child = clamped = scored = 0
-    clamped = scored = 0
     sources: dict[str, int] = {}
-    fd, tmp = tempfile.mkstemp(dir=os.path.dirname(os.path.abspath(a.out)) or ".",
+    with qc.atomic_writer(a.out) as out:
-                              prefix=".rechunk-")
+        for lineno, rec in qc.iter_jsonl(a.corpus):
-    try:
+            where = f"{a.corpus}:{lineno}"
-        with os.fdopen(fd, "w", encoding="utf-8") as out:
+            if not isinstance(rec.get("prompt"), str):
-            with open(a.corpus, encoding="utf-8") as src:
+                qc.refuse(f"{where} has no string 'prompt' field")
-                for lineno, line in enumerate(src, 1):
+            windows, n_parent = cut(rec, tok, a, where)
-                    if not line.strip():
+            docs_in += 1
-                        continue
+            tokens_in += n_parent
-                    try:
+            for w in windows:
-                        rec = json.loads(line)
+                out.write(json.dumps(w.record(), ensure_ascii=False) + "\n")
-                    except json.JSONDecodeError as exc:
+                docs_out += 1
-                        refuse(f"{a.corpus}:{lineno} is not JSON: {exc}")
+                tokens_out += w.n_tokens
-                    prompt = rec.get("prompt")
+                longest_child = max(longest_child, w.n_tokens)
-                    if not isinstance(prompt, str):
+                clamped += 1 if w.clamped else 0
-                        refuse(f"{a.corpus}:{lineno} has no string 'prompt' field")
+                scored += w.n_tokens - w.score_from
-                    ids = tok.encode(prompt, add_special_tokens=False).ids
+                if "source" in rec:
-                    docs_in += 1
+                    for cls in str(rec["source"]).split("+"):
-                    tokens_in += len(ids)
+                        sources[cls] = sources.get(cls, 0) + 1
-                    for start in range(0, max(len(ids), 1), a.max_doc_tokens):
+    if tokens_out != tokens_in:
-                        window = ids[start:start + a.max_doc_tokens]
+        raise SystemExit(f"REFUSED: the windows hold {tokens_out} tokens and the corpus "
-                        if not window:
+                         f"has {tokens_in} — re-chunking lost or invented tokens")
-                            continue
+    return {
-                        score_from = max(len(window) - a.score_tail, 0)
-                        if len(window) < a.score_tail:
-                            clamped += 1
-                        scored += len(window) - score_from
-                        child = {"prompt": tok.decode(window),
-                                 "score_from": score_from}
-                        if "source" in rec:
-                            child["source"] = rec["source"]
-                        out.write(json.dumps(child, ensure_ascii=False) + "\n")
-                        docs_out += 1
-                        tokens_out += len(window)
-                        longest_child = max(longest_child, len(window))
-                        if "source" in rec:
-                            for cls in str(rec["source"]).split("+"):
-                                sources[cls] = sources.get(cls, 0) + 1
-        if tokens_out != tokens_in:
-            raise SystemExit(f"REFUSED: the windows hold {tokens_out} tokens and the corpus "
-                             f"has {tokens_in} — re-chunking lost or invented tokens")
-        os.replace(tmp, a.out)
-    except BaseException:
-        if os.path.exists(tmp):
-            os.unlink(tmp)
-        raise
-    os.chmod(a.out, 0o644)
-
-    print(json.dumps({
         "corpus": a.corpus,
         "out": a.out,
         "tokenizer": os.path.basename(a.tokenizer_json),
@@ -171,7 +179,13 @@ def main() -> None:
         "scored_positions": scored,
         "clamped_score_from": clamped,
         "source_documents": sources,
-    }, indent=1))
+    }
+
+
+def main() -> None:
+    a = parse_args()
+    validate(a)
+    print(json.dumps(rechunk(a, qc.load_tokenizer(a.tokenizer_json)), indent=1))
 
 
 if __name__ == "__main__":

scripts/run-qwen38-bf16.sh

@@ -28,10 +28,18 @@
 #                     (RADIANCE_CALIB_DIR + RADIANCE_CALIB_TOKENS=12000000). A
 #                     separate driver — scripts/calib-push.py --port 7770
 #                     --container qwen38-bf16 --corpus
-#                     /home/dank/corpora/out/calib.jsonl — pushes the corpus; at
+#                     /home/dank/corpora/out/calib-7168.jsonl — pushes the corpus;
-#                     12M pooled positions the engine writes the Grams and the
+#                     at the floor the engine writes the Grams and the tap switches
-#                     tap switches itself off (calib-push prints TAP_DONE). Stop
+#                     itself off (calib-push prints TAP_DONE). Stop this script
-#                     this script after that; the trap brings frank-archon back.
+#                     after that; the trap brings frank-archon back. THE FLOOR IS
+#                     THE CAMPAIGN'S RULE (12 M tokens, Frank 2026-10-30): the
+#                     served iq4r this arm competes with was calibrated on a 10 M-
+#                     token set, so the arm is calibrated at least as deeply. One
+#                     round of the re-chunked corpus is ~9.31 M tokens, so sealing
+#                     takes round 1 plus part of round 2 — a round that seals
+#                     nothing is NORMAL (the floor is pooled), and the pusher's
+#                     round_verdict treats a flat round as "continue", never as a
+#                     failure. One seal writes ~40 GiB.
 #
 # Shape: the vendor's bf16 calibration lines — expert_tiered placement with the
 # weights disk tier, deterministic (the resident set pinned to the loader's plan,
@@ -92,7 +100,8 @@
 #        the container reads through /models), FRANK_QWEN38_LOG_DIR, and the step
 #        and slot counts inherited as FRANK_ARCHON_MAX_BATCHED_TOKENS /
 #        FRANK_ARCHON_MAX_SEQS — no per-mode literals (the audit law: a mode that
-#        carries its own step measures a shape nothing serves) —
+#        carries its own step measures a shape nothing serves; calib is exempt, see
+#        Q38_CALIB_MNBT / Q38_CALIB_SEQS below — its output is never compared) —
 #        and FRANK_QWEN38_MAX_MODEL_LEN — the window every mode boots at (32,768;
 #        see the corpus census above, and raise it together with the corpus).
 # kldref refuses (exit 2) when the longest record of the corpus it is about to
@@ -112,6 +121,18 @@ SCRIPT_DIR="$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)"
 # shellcheck disable=SC1091
 source "$SCRIPT_DIR/frank-archon.env"
 IMAGE="${FRANK_ARCHON_IMAGE:?FRANK_ARCHON_IMAGE must name the radiance image}"
+# THE CALIBRATION PIN: the LATEST radiance, pinned by digest because a tag is a
+# moving target and a tap must name the producer that wrote it. `:latest` on
+# 2026-10-30 resolves to v1.14.0 (revision 4a61d7296bc5), and on that image a
+# calib boot prints the arming line — "recorded passes: off, the program
+# accumulates calibration Grams (gram_accum)" — where the production pin
+# (1.8.1) prints nothing of the kind: a 17-hour run on 1.8.1 wrote ZERO bytes
+# into the tap dir while /health, /stats and the push all looked healthy
+# (measured 2026-10-30). The tap code is identical in both trees' sources, so
+# the shipped 1.8.1 build is what drops it; do not spend another window on it.
+# Any other image must be named deliberately with --image.
+CALIB_IMAGE="${Q38_CALIB_IMAGE:-stilldeadcode/radiance@sha256:69add58f7de1ed2bca2b1e20a4cc75717557df3f73569a178eca5aafe8c44093}"
+IMAGE_EXPLICIT=0
 
 MODE=""
 POS=()
@@ -120,7 +141,8 @@ while [ $# -gt 0 ]; do
     case "$1" in
         kldref|calib|score) MODE="$1" ;;
         --dry-run)          DRY=1 ;;
-        --image)            IMAGE="$2"; shift ;;
+        --image)            [ "$#" -ge 2 ] || { echo "--image needs a reference" >&2; exit 2; }
+                            IMAGE="$2"; IMAGE_EXPLICIT=1; shift ;;
         *)
             # score's two positional arguments, and nothing else may be bare.
             if [ "$MODE" = score ] && [ "${#POS[@]}" -lt 2 ]; then
@@ -150,7 +172,7 @@ TEMPLATE="$SCRIPT_DIR/qwen38-froggeric-v22.5.jinja"
 LOG_DIR="${FRANK_QWEN38_LOG_DIR:-/home/dank/logs}"
 LOG="$LOG_DIR/qwen38_bf16_${MODE}.log"
 KLD_REF_DIR="$CAMPAIGN/kld-ref"
-CALIB_DIR="$CAMPAIGN/calib"
+CALIB_DIR="${FRANK_QWEN38_CALIB_DIR:-$CAMPAIGN/calib}"
 C=qwen38-bf16
 SERVICE=frank-archon
 # The card split, taken from the file that owns it instead of a private pair:
@@ -184,6 +206,24 @@ if ! [[ "$MNBT" =~ ^[0-9]+$ ]]; then
     exit 2
 fi
 
+# Calib's shape is NOT a measuring shape. kldref and score produce KL numbers that are only
+# comparable across boots at an identical shape, which is why they inherit production's step and
+# slots to the letter. The tap's output is layer Hessians, never a number any report compares, so
+# its step and slot count are its own — and they are the tap's dominant cost: a 7,168-token
+# document cut into four 2,048-token steps re-streams each layer's routed expert union four
+# times from the container, so one wide step is worth up to ~4x the read volume. Measured
+# 2026-10-29 at the inherited 2,048/4: prefill 71 tok/s and a 47-hour tap; the probe exists to
+# find out what the wide step is worth before the campaign pays for two days of drained serving.
+# Both default to the inherited value, so nothing moves unless a probe asks it to.
+CALIB_MNBT="${Q38_CALIB_MNBT:-$MNBT}"
+CALIB_SEQS="${Q38_CALIB_SEQS:-$MAX_SEQS}"
+for pair in "Q38_CALIB_MNBT=$CALIB_MNBT" "Q38_CALIB_SEQS=$CALIB_SEQS"; do
+    if ! [[ "${pair#*=}" =~ ^[0-9]+$ ]]; then
+        echo "REFUSED: ${pair%%=*} must be a number (got: ${pair#*=})" >&2
+        exit 2
+    fi
+done
+
 # The window every mode boots at. Over-length is a REFUSAL, not a truncation (the
 # env file says so for production), so this is the number the corpus must fit in.
 MAX_MODEL_LEN="${FRANK_QWEN38_MAX_MODEL_LEN:-32768}"
@@ -192,6 +232,25 @@ if ! [[ "$MAX_MODEL_LEN" =~ ^[0-9]+$ ]]; then
     exit 2
 fi
 
+# Where the n-gram table lives. On the bf16 payload it is 95.37 GiB and `ram`
+# PINS it: the calibration boot died at "startup stopped: out of memory" with
+# 230.22 GiB pinned of 246.07 GiB and 8.06 GiB available (measured 2026-10-29),
+# while the reference recording and every score boot had already succeeded with
+# the same pool and slots. The difference that matters is the job: calib only
+# needs the routed weights prefilled, so it reads the table from the container's
+# own file tier and the pin never happens; kldref and score keep production's
+# placement, because their numbers are only comparable when every other term of
+# the shape is identical. FRANK_QWEN38_NGRAM_PLACEMENT overrides either default.
+case "$MODE" in
+    calib) NGRAM_PLACEMENT="${FRANK_QWEN38_NGRAM_PLACEMENT:-disk}" ;;
+    *)     NGRAM_PLACEMENT="${FRANK_QWEN38_NGRAM_PLACEMENT:-ram}" ;;
+esac
+case "$NGRAM_PLACEMENT" in
+    auto|vram|ram|disk) ;;
+    *) echo "REFUSED: FRANK_QWEN38_NGRAM_PLACEMENT must be auto|vram|ram|disk (got: $NGRAM_PLACEMENT)" >&2
+       exit 2 ;;
+esac
+
 # Every container this campaign boots is read through the ro /models mount, so
 # the host path must live under it; the container-side spelling is what goes
 # into the argv. Same rule for the container a score pass opens.
@@ -229,7 +288,7 @@ ARGS=(--tp 2 --placement expert_tiered --weights-disk-tier
       --host-pool-mib "$HOST_POOL_MIB" --gpu-headroom-mib "$GPU_HEADROOM_MIB"
       --deterministic --num-speculative-tokens 0
       --kv-cache-dtype fp8 --max-model-len "$MAX_MODEL_LEN"
-      --ngram-placement ram
+      --ngram-placement "$NGRAM_PLACEMENT"
       --override-chat-template /models/tpl.jinja)
 MOUNTS=(-v "$MODELS:/models:ro")
 TAPENV=()
@@ -250,7 +309,8 @@ case "$MODE" in
     # time the engine reads a recording and writes a report instead of the
     # reverse. The reference dir mounts READ-ONLY — a scoring pass must not be
     # able to touch the zero point — and the report gets its own writable bind.
-    if [ "$OUT_DIR" = "$KLD_REF_DIR" ] || same_dir "$OUT_DIR" "$KLD_REF_DIR"; then
+    if [ "$OUT_DIR" = "$KLD_REF_DIR" ] || same_dir "$OUT_DIR" "$KLD_REF_DIR" \
+       || [ "$(realpath -m "$OUT_DIR")" = "$(realpath -m "$KLD_REF_DIR")" ]; then
         echo "REFUSED: score output $SCORE_OUT is inside the reference dir $KLD_REF_DIR, which this mode mounts read-only" >&2
         exit 2
     fi
@@ -274,11 +334,45 @@ case "$MODE" in
     # buys throughput with the expert slab, so price it like any other
     # VRAM knob. The slots follow frank-archon.env's FRANK_ARCHON_MAX_SEQS
     # through MAX_SEQS, exactly as kldref and score do.
+    [ "$IMAGE_EXPLICIT" = 1 ] || IMAGE="$CALIB_IMAGE"
+    # ONE TAP SEAL WRITES ~40 GiB. Measured 2026-10-30: a 8,192-token floor wrote
+    # 39.4 GiB into the tap dir within seconds; pointed at a 37 GiB root it filled
+    # the filesystem to zero, after which every write came back short ("calibration
+    # cannot write", "write was short") and the system itself began refusing
+    # writes. The room is demanded up front, on the tap's own filesystem.
+    if [ "$DRY" != 1 ]; then
+        avail_mib=$(df --output=avail -B 1M "$CALIB_DIR" 2>/dev/null | tail -1 | tr -d ' ')
+        [[ "$avail_mib" =~ ^[0-9]+$ ]] || avail_mib=0
+        need_mib="${Q38_CALIB_MIN_FREE_MIB:-61440}"
+        if [ "$avail_mib" -lt "$need_mib" ]; then
+            echo "REFUSED: tap dir $CALIB_DIR has ${avail_mib} MiB free; one seal writes ~40 GiB and this run needs ${need_mib} MiB (Q38_CALIB_MIN_FREE_MIB overrides)" >&2
+            exit 2
+        fi
+    fi
     ARGS+=(--model "$MODEL_IN" --port 7770
-           --max-num-seqs "$MAX_SEQS" --max-num-batched-tokens "$MNBT")
+           --max-num-seqs "$CALIB_SEQS" --max-num-batched-tokens "$CALIB_MNBT")
+    # The tap is only real if this directory exists and the engine can write it:
+    # without the check a typo'd CALIB_DIR auto-creates a root-owned empty dir in
+    # whatever path was given, and the run burns hours serving prompts into a tap
+    # nobody reads. RADIANCE_CALIB_TOKENS is the whole corpus' floor, not a round's.
+    [ "$DRY" = 1 ] || [ -w "$CALIB_DIR" ] || {
+        echo "REFUSED: calib tap dir $CALIB_DIR is not writable (mkdir it, chown it, then re-run)" >&2
+        exit 2
+    }
     MOUNTS+=(-v "$CALIB_DIR:/data/calib")
     TAPENV=(-e RADIANCE_CALIB_DIR=/data/calib
-            -e RADIANCE_CALIB_TOKENS=12000000)
+            -e "RADIANCE_CALIB_TOKENS=${FRANK_QWEN38_CALIB_TOKENS:-12000000}")
+    # The tap's own authorization. A pusher must see this marker before it sends
+    # a prompt (calib-push.py --tap-marker), it names the container so a push
+    # aimed anywhere else refuses, and a boot that fails removes it again below.
+    # Measured 2026-10-29: with no marker, the chain pushed 4 rounds x 10M tokens
+    # into RESTORED PRODUCTION after the rig's boot died, and nothing recorded.
+    if [ "$DRY" = 1 ]; then
+        :
+    else
+        printf 'container=%s\nimage=%s\nmode=calib\nstarted=%s\npid=%s\n' \
+            "$C" "$IMAGE" "$(date -Is)" "$$" > "$CALIB_DIR/.rig-boot"
+    fi
     ;;
 esac
 
@@ -352,7 +446,27 @@ cp -f "$TEMPLATE" "$TPL" || {
 [ "$MODE" = kldref ] && mkdir -p "$KLD_REF_DIR"
 [ "$MODE" = calib ]  && mkdir -p "$CALIB_DIR"
 
-restore() { sudo systemctl start "$SERVICE" >/dev/null 2>&1 || true; }
+# Stop the campaign container BEFORE handing the cards back: a killed run leaves
+# its conmon alive (--rm only fires when podman itself exits), and production
+# then boots beside a container still holding the GPUs — measured 2026-10-29,
+# where the service reported `active` while its engine starved on the rig's VRAM.
+restore() {
+    podman rm -f "$C" >/dev/null 2>&1 || true
+    # Removing the container does NOT kill its engine. Measured 2026-10-30: a rig
+    # radiance kept /dev/kfd after `podman rm -f`, production's
+    # ar_oneshot_2rank_wht6 init then failed `device error`, the engine exited 0,
+    # and Restart=always looped it 100 times with /health never answering. The
+    # service is stopped for the whole run, so any radiance still holding the
+    # device here is an orphan — kill it BY PID (pkill -f matches the caller's
+    # own command line, which is how a session gets killed instead of the orphan).
+    for p in $(fuser /dev/kfd 2>/dev/null); do
+        if [ "$(cat "/proc/$p/comm" 2>/dev/null)" = radiance ]; then
+            echo "restore: killing orphaned radiance pid $p still holding /dev/kfd" >&2
+            kill -9 "$p" 2>/dev/null || true
+        fi
+    done
+    sudo systemctl start "$SERVICE" >/dev/null 2>&1 || true
+}
 trap restore EXIT
 echo "run-qwen38-bf16 $MODE: stopping $SERVICE ($(date -Is)); log: $LOG"
 sudo systemctl stop "$SERVICE" >/dev/null 2>&1 || true
@@ -360,5 +474,12 @@ podman rm -f "$C" >/dev/null 2>&1 || true
 
 "${CMD[@]}" 2>&1 | tee "$LOG"
 rc=${PIPESTATUS[0]}
+# A boot that failed must not leave the tap marked as rig-authorized: the marker
+# is what calib-push.py trusts, and the point of it is that a dead rig can never
+# be pushed into again.
+if [ "$MODE" = calib ] && [ "$rc" -ne 0 ] && [ -f "$CALIB_DIR/.rig-boot" ]; then
+    mv -f "$CALIB_DIR/.rig-boot" "$CALIB_DIR/.rig-boot.failed-$(date +%Y%m%d-%H%M%S)" 2>/dev/null || true
+    echo "the rig boot failed (rc=$rc): the tap marker is retired, so no pusher can take this tap for a live rig"
+fi
 echo "run-qwen38-bf16 $MODE exited rc=$rc; $SERVICE restored; log: $LOG"
 exit "$rc"

scripts/serve-frank-dragoon.sh

@@ -1,5 +1,5 @@
 #!/usr/bin/env bash
-# serve-frank-dragoon.sh — frank-dragoon endpoint: halogen-flash-server 0.16.2
+# serve-frank-dragoon.sh — frank-dragoon endpoint: halogen-flash-server 0.18.20
 # (closed-source engine, image ghcr.io/peonist-ai/halogen-flash-server,
 # digest-pinned in frank-dragoon.env) serving Quat3rnion's NATIVE v2
 # conversion (2026-10-11) of the abliterated Qwen3.8-Flash-Next on Dankclaw's

scripts/serve-frank-zealot.sh

@@ -1,7 +1,7 @@
 #!/usr/bin/env bash
 # Serve prism-ml/Ternary-Bonsai-2-9B-gguf (PQ2_0 ternary quant WITH the MTP head
 # in the file) as frank-zealot on Dankclaw's RTX 3090 via llama.cpp (PrismML
-# vendor build 10777, CUDA sm_86).
+# vendor build 12118 (0.7.0-dev, commit 024474448), CUDA sm_86).
 #
 # 2026-10-18 (Frank): Bonsai-2 27B PQ2_0 → the 9B MTP export, then the same
 # evening TWO 262,144-token conversations (--ctx-size 524288 --parallel 2) over
@@ -78,7 +78,7 @@ if [[ -f "$ENV_FILE" ]]; then
 fi
 
 # --- defaults (used when the env file does not set them) -------------------
-: "${LLAMA_BIN:=/home/dank/GitPackages/llama-prism-b10777/llama-server}"
+: "${LLAMA_BIN:=/home/dank/GitPackages/llama-prism-0244744/bin/llama-server}"
 : "${MODEL_FILE:=/home/dank/models/Bonsai2-9B-MTP/Ternary-Bonsai-2-9B-PQ2_0-MTP-Q8_0.gguf}"
 : "${ALIAS:=frank-zealot}"
 : "${BIND_ADDR:=0.0.0.0}"
@@ -119,7 +119,7 @@ fi
 # above are what pay for it. The draft cache dtype IS a further knob
 # (--spec-draft-type-k / --spec-draft-type-v, both default f16 — verified in this
 # build's --help) and stays at its default here. Card gate: quantized V alongside
-# MTP needs fork build >= prism-b10776, which is why LLAMA_BIN is the 10777
+# MTP needs fork build >= prism-b10776; LLAMA_BIN is 12118 (b10777 kept as rollback)
 # prebuilt; the type-142 loader lock keeps it pinned regardless.
 # SPEC_TYPE=none is the rollback and strips the whole set (flags move together).
 : "${SPEC_TYPE:=draft-mtp}"

scripts/stack-qwen38-official-fp8.py

@@ -337,8 +337,12 @@ def main():
     if any(meta[shard_ids[i]][4][1] != hd for i in range(n_shards)):
         log("ERROR: PLE shards disagree on HD")
         return 2
-    if hd % 128 != 0 and hd != 160:
+    if hd != 160 and hd % 128 != 0:
-        log(f"WARNING: PLE HD={hd} is neither 160 nor a multiple of 128")
+        # The pack below writes lanes of 128 codes; any other HD would produce a
+        # table that decodes to the wrong rows. The old warning let it continue.
+        log(f"ERROR: PLE HD={hd} is neither 160 nor a multiple of 128, and the "
+            "128-lane pack cannot express it")
+        return 2
 
     # ---- idempotence gate -------------------------------------------------
     stacked_names = [f"{p}.mlp.experts.gate_up_proj" for p in prefixes]
@@ -521,12 +525,16 @@ def verify(stage, prefixes, n_e, shapes, meta, shard_ids, rows, hd,
         }
         for name, (dt, shp) in want.items():
             e = hdr[name]
-            assert e["dtype"] == dt and list(e["shape"]) == shp, name
+            if e["dtype"] != dt or list(e["shape"]) != shp:
-            assert e["data_offsets"][1] - e["data_offsets"][0] == \
+                raise RuntimeError(f"{name}: header is {e['dtype']}{e['shape']}, expected {dt}{shp}")
-                int(np.prod(shp)) * (1 if dt == F8 else 2), name
+            if e["data_offsets"][1] - e["data_offsets"][0] != int(np.prod(shp)) * (1 if dt == F8 else 2):
+                raise RuntimeError(f"{name}: data span is {e['data_offsets']}, shape {shp} needs "
+                                   f"{int(np.prod(shp)) * (1 if dt == F8 else 2)} B")
     _, phdr = parse_header(out_path)
     e = phdr[next(iter(phdr))]
-    assert e["dtype"] == BF16 and list(e["shape"]) == [sum(rows), hd]
+    if e["dtype"] != BF16 or list(e["shape"]) != [sum(rows), hd]:
+        raise RuntimeError(f"PLE table header is {e['dtype']}{e['shape']}, "
+                           f"expected {BF16}[{sum(rows)}, {hd}]")
 
     # sampled PLE rows: recompute and compare bytes exactly
     lut = e4m3_lut()
@@ -546,7 +554,8 @@ def verify(stage, prefixes, n_e, shapes, meta, shard_ids, rows, hd,
                 with open(out_path, "rb") as ofh:
                     ofh.seek(out_off + got_row * hd * 2)
                     got = ofh.read(hd * 2)
-                assert got == want, f"PLE row {got_row} mismatch"
+                if got != want:
+                    raise RuntimeError(f"PLE row {got_row} does not match the decoded source")
             mm.close()
     log("verify: new-file headers, sizes and sampled PLE rows all exact; "
         "expert slabs were byte-checked in-stack")

serverAppConfigs-docs/Dankserver/Media Relayout.md

@@ -0,0 +1,130 @@
+---
+type: reference
+title: Media Relayout
+description: "Census, plan and apply the movie library into Jellyfin format: the gates to read, and the rollback."
+tags: [dankserver, media-server, jellyfin, plex, procedure]
+timestamp: 2026-10-30T00:00:00Z
+---
+
+# Media Relayout
+
+The movie library on [[Entropy Volume]] arrived as release folders —
+`Some.Movie.2019.1080p.WiKiTaLx.x265-FOO/` with the video inside named after
+the release. [[Jellyfin]] matches a movie by folder and file name, so the
+release noise cost posters, extras and versions. These three scripts turn that
+library into Jellyfin's own shape, and nothing else in the repo moves a media
+file.
+
+| Script | Touches media? | Output |
+|---|---|---|
+| `scripts/media-census.py` | reads headers only | `enum`, `probed`, `plex` TSVs |
+| `scripts/media-plan.py` | no — TSVs in, TSVs out | the plan + the exception report |
+| `scripts/media-apply.py` | **renames, never copies** | the moves, plus a journal |
+
+The seam is `tests/media_relayout.bats`; the target contract is quoted from
+[jellyfin.org/docs/general/server/media/movies](https://jellyfin.org/docs/general/server/media/movies)
+in `media-plan.py`'s docstring.
+
+## Procedure
+
+Run the census on the machine that mounts the library. `probe` is the slow step
+(a header read per container) and is `nice`d — it is safe beside a live service
+because it never competes for the GPU and never writes inside the library.
+
+```bash
+C=scripts/media-census.py
+$C enumerate --root /mnt/entropy/Movies --out /tmp/media-enum.tsv
+$C probe     --enum /tmp/media-enum.tsv --out /tmp/media-probed.tsv --jobs 6
+$C plex      --root /mnt/entropy/Movies --out /tmp/plex-movies.tsv
+```
+
+`plex` reads the Plex metadata cache (`--db` defaults to the path the
+[[Plex]] container bind-mounts at `/var/lib/plexmediaserver`) with
+`mode=ro&immutable=1`: a live server is never locked or written.
+
+Then plan — this reads only the three TSVs, so it costs seconds and can run
+anywhere, including a laptop:
+
+```bash
+scripts/media-plan.py --root /mnt/entropy/Movies \
+    --enum /tmp/media-enum.tsv --probed /tmp/media-probed.tsv \
+    --plex /tmp/plex-movies.tsv --out /tmp/jf-plan.tsv --exceptions /tmp/jf-exceptions.txt
+```
+
+**Read `/tmp/jf-exceptions.txt` before applying.** It is not a log: every line
+is a file the plan could not name a film for, a pile that needs a README, or a
+collision it refused. The printed gate lines are the go/no-go:
+`problems`, `bad-names`, `occupied`, `double-claim`, `left-unmoved` must all be
+0 and `unplanned=0` must appear in the report. Any non-zero number means a file
+that would be filed wrong, and the plan is not run until it is understood.
+
+```bash
+scripts/media-apply.py apply    --plan /tmp/jf-plan.tsv --root /mnt/entropy/Movies   # dry, writes nothing
+scripts/media-apply.py apply    --plan /tmp/jf-plan.tsv --root /mnt/entropy/Movies --apply
+scripts/media-apply.py verify   --plan /tmp/jf-plan.tsv --journal …/media-relayout.jsonl
+scripts/media-apply.py rollback --journal …/media-relayout.jsonl --apply
+```
+
+After it lands, both servers need a scan or the change is invisible:
+[[Jellyfin]] *Library → Scan All Libraries*, and Plex's library scan
+(`podman exec plex …` or the web UI). `verify` reporting empty folders is normal
+for the folders the piles own; they are left in place per the deletion policy.
+
+## Laws
+
+- **Plex's matched title outranks the filename**, and a Plex year of `0` means
+  *indexed but never matched* — trusting it named folders `Title (0)`, which is
+  a Jellyfin item with no year, no poster and no identity. A year that is not a
+  plausible year is no information.
+- **`<>:"/\|?*` never survive a name.** Plex's own title for *V/H/S/2* is
+  `V/H/S/2`; kept verbatim it became `Movies/V/H/S/2 (2013)/` — a tree five
+  levels deep that neither matcher reads as a movie.
+- **A folder inside a movie's folder is bonus territory even when its name is
+  misspelled.** `Saboteur (1942)/Exrtas/Saboteur - A Closer Look` is a typo for
+  `Extras`; the vocabulary rule never fired, the featurette became a movie
+  folder, and its words then out-scored the real film and drew that box set's
+  other extras into it. The rule that replaced it needs no metadata: the folder
+  holding the longest film owns the shorter video filed under it, unless that
+  file's own name carries a different year — which is another film speaking.
+- **Matching words are split on everything that is not a letter or a digit.**
+  Splitting on separators but not on `/` left tokens like `(1960)/extras` that
+  no other name could ever agree with, so a film could not be found by the
+  folder holding it, and extras drifted into a neighbour. A bare number counts
+  only when it could be a release year, so `1080` and `2160` never pose as
+  agreement between two names.
+- **A file whose name carries no words is bonus material, not a folder.**
+  `.Extra 1 ( 1H 27min multisub ).mkv` parses to `1`; a movie folder named `1`
+  is not a movie. It becomes a review line telling a human to name the file.
+- **The executor renames and nothing else.** `rename(2)` cannot cross a
+  filesystem, so a plan that would is refused rather than silently copying tens
+  of terabytes.
+- **The journal is a write-ahead log: the intent is fsynced before the rename.**
+  An interrupted run therefore leaves only two states — an intent whose move
+  happened, and an intent whose move never ran — and the next run finishes the
+  job without moving anything twice. A file that moved with no journal line did
+  not move through this tool, and the tool refuses to guess on its behalf.
+- **The journal lives BESIDE the library root, never inside it** (default:
+  `media-relayout.jsonl` next to the root): a log inside the library is a file
+  Jellyfin indexes.
+- **Refusals are all-or-nothing.** An occupied target, a vanished source or a
+  duplicate destination stops the run *before the first rename*, so the operator
+  fixes one thing and re-runs instead of untangling a half-applied plan.
+  `--force` relaxes only the opinion guards; it never permits overwriting.
+
+## Rollback
+
+One command, from the journal of the run:
+`media-apply.py rollback --journal …/media-relayout.jsonl --apply`. It restores
+every source path, removes only the folders that run created *and that are now
+empty*, and refuses everything if any moved file cannot be accounted for. A
+rolled-back journal refuses to roll back twice.
+
+## See Also
+
+[[Jellyfin]], [[Plex]], [[Entropy Volume]], [[Testing Policy]], [[Storage Policy]]
+
+## Citations
+
+- Jellyfin Movie documentation — folder/file naming, extras subfolders,
+  `Movie (2013) - 216p.mkv` version and `- 4K.mkv` handling:
+  <https://jellyfin.org/docs/general/server/media/movies>

serverAppConfigs-docs/LLM/Dankclaw System Specs.md

@@ -10,7 +10,7 @@ timestamp: 2026-09-03
 
 ## Server Overview
 
-Dankclaw is an AMD APU + RTX 3090 inference host. It ran CIRU ROCmFPX llama.cpp inference through 2026-08-22 ([[frank-dragoon]], retired). The host runs Ubuntu 26.04 with an AMD Ryzen AI MAX+ 395 Strix Halo APU. The APU uses unified memory — CPU and GPU share the same physical RAM pool. Today it serves [[frank-dragoon]] (halogen-flash-server 0.16.2 on the iGPU, port 7770) and [[frank-zealot]] (llama.cpp PrismML fork on the RTX 3090, port 7771).
+Dankclaw is an AMD APU + RTX 3090 inference host. It ran CIRU ROCmFPX llama.cpp inference through 2026-08-22 ([[frank-dragoon]], retired). The host runs Ubuntu 26.04 with an AMD Ryzen AI MAX+ 395 Strix Halo APU. The APU uses unified memory — CPU and GPU share the same physical RAM pool. Today it serves [[frank-dragoon]] (halogen-flash-server 0.18.20 on the iGPU, port 7770) and [[frank-zealot]] (llama.cpp PrismML fork on the RTX 3090, port 7771).
 ## Hardware Specifications
 
 ### CPU
@@ -105,7 +105,7 @@ cable between claw and server is required for 10G.
 
 - Unified memory eliminates separate VRAM limits. The GPU uses system RAM.
 - The GPU budget is the GTT aperture: ~114 GiB (`mem_info_gtt_total` in `/sys/class/drm/card1/device/` = 122,402,963,456 B; `card1` is the amdgpu iGPU, `PCI_ID=1002:1586`; `llama-server --list-devices` reports 116,732 MiB). rocm-smi's smaller SVM/dedicated carve-out is **not** the budget — the 2026-08 "64 GiB rocm-smi" lesson is stale. Do not use `free` or `dmidecode` for the GPU budget.
-- The active services are `frank-dragoon` on 7770 — **halogen-flash-server 0.16.2** serving the native v2 abliterated `.hgn` in a rootless podman container ([[frank-dragoon]]) — and `frank-zealot` on 7771 ([[frank-zealot]]). The gufo v0.1.1 / UD-Q8_K_XL runtime this page once described is retired (2026-09-28 → 10-01, [[frank-dragoon]]); the gufo build tree is still on disk at `/home/dank/GitPackages/gufo-v0.1.1/`.
+- The active services are `frank-dragoon` on 7770 — **halogen-flash-server 0.18.20** serving the native v2 abliterated `.hgn` in a rootless podman container ([[frank-dragoon]]) — and `frank-zealot` on 7771 ([[frank-zealot]]). The gufo v0.1.1 / UD-Q8_K_XL runtime this page once described is retired (2026-09-28 → 10-01, [[frank-dragoon]]); the gufo build tree is still on disk at `/home/dank/GitPackages/gufo-v0.1.1/`.
 - The gufo-era note that the active runtime links against the host ROCm (`/opt/rocm` → `/etc/alternatives/rocm` → 7.2.4) through its RPATH belongs to that retired source build. The serving runtimes do not depend on the host ROCm that way: halogen runs inside its digest-pinned image, and `frank-zealot` is a CUDA build on the 3090. The retired CIRU venv's TheRock 7.15 SDK (`LD_LIBRARY_PATH`) was never in the serving path either.
 - The retired vLLM venvs (`/home/dank/vllm-venv`, `/home/dank/vllm-vk-venv`) were moved to `/home/dank/.AGENTTRASH/` on 2026-08-18.
 - 424 GB free on `/` (`df -h`, 2026-10-18) is ample for model storage.

serverAppConfigs-docs/LLM/Radiance Architecture Plugins.md

@@ -0,0 +1,287 @@
+---
+type: "howto"
+title: "Radiance Architecture Plugins — declaring a model"
+description: "Writing a radiance architecture plugin: declaring weights, buffers, KV groups and ops, then issuing the step."
+tags:
+  - llm
+  - radiance
+  - plugins
+  - howto
+timestamp: 2026-10-30
+---
+
+# Radiance architecture plugins
+
+An architecture plugin declares a model in two phases: `rad_arch_declare` names every weight,
+buffer, KV group and op **once**, against the model's metadata; `rad_arch_step` issues the ops for
+one step. The engine picks the plugin by the exact `(rad_arch_id, rad_arch_quant)` pair — no
+fallback — so one family at two weight formats is two `.so` files, and branching on a quant
+descriptor inside one plugin is the anti-pattern the split exists to prevent.
+
+The in-tree plugins are the templates: `arch/llama_fp8/llama_fp8.cpp` (the smallest complete
+decoder), `arch/gemma3`, `arch/qwen35_bf16`, and the header-only blocks in `arch/common/`
+(`rad_block_attn*.h`, `rad_block_mlp*.h`, `rad_block_moe_fp8.h`, `rad_block_gdn*.h`,
+`rad_block_dflash2.h`, `rad_block_mtp*.h`, `rad_block_vit.h`, `rad_block_ple.h`,
+`rad_block_glm_*.h`, `rad_qsa.h`, …). Include those by `<arch/rad_arch.h>` — they are installed
+beside the ABI and the in-tree plugins include them by that spelling.
+
+## Exports
+
+```c
+const char* rad_arch_id(void);              /* "llama", "qwen35", ... */
+const char* rad_arch_quant(void);           /* "" unquantised, else the descriptor served */
+int         rad_arch_serves_imports(void);  /* optional: 0 declines an imported quantisation */
+int         rad_arch_probe(const RadModelMeta* meta, RadArchProbe* out);   /* optional */
+int         rad_arch_declare(RadBuilder* b, const RadModelMeta* meta, const RadBuildCtx* ctx);
+void        rad_arch_step(struct RadCtx* c, const struct RadBatch* batch);
+const RadChatFormat*  rad_arch_chat_format(const RadModelMeta* meta);      /* optional */
+const RadAudioFormat* rad_arch_audio_format(const RadModelMeta* meta);     /* optional */
+```
+
+Everything but declare/step is optional. `RAD_ARCH_PLUGIN(NS, ID, QUANT, VERSION, DESC)` in
+`arch/common/rad_arch.h` emits the whole export block, including `rad_plugin_info` (kind
+`RAD_PLUGIN_ARCH`). The plugin's **name must be unique**; the architecture id may repeat.
+
+`rad_arch_serves_imports` exists because a checkpoint whose quantisation was imported weight by
+weight (Quark, ParoQuant, GPTQ, NVFP4) names its producing tool, which no plugin claims: the family's
+one quantised plugin serves it, and two are refused by name. A second quantised plugin that reads
+only its own containers returns 0.
+
+## Declare: the metadata
+
+`RadModelMeta` is what the loader read from the container's header before any plugin ran: `arch_id`,
+`name`, `quant`, `n_layers`, `n_embd`, `n_head`, `n_head_kv`, **`head_dim`**, `n_ff`, `n_vocab`,
+`n_ctx_train`, expert counts, `rms_eps`, `rope_theta`, and a free-form `kv_key`/`kv_val` table for
+everything else (`rad_meta_geti/getf/gets`). Read dimensions from here — one plugin covers every size
+in a family.
+
+**`head_dim` is read, never derived.** On the Qwen hybrids it is 256 while `hidden / heads` is 213;
+a derived value silently breaks projection widths, KV page size and the attention kernel's band.
+
+`RadBuildCtx` is the deployment the plugin must declare against:
+
+| field | meaning |
+|---|---|
+| `rank`, `world_size` | declare runs once per rank; **dimensions are already divided** — the core inserts no collectives |
+| `max_tok`, `max_seqs`, `max_ctx`, `max_spec` | the step bounds: buffer sizing, kernel bands, the scheduler's window |
+| `max_mixed_tok` | the most a multi-sequence step may carry; a wider step is one sequence (prefill span) |
+| `tp_wire_lossy`, `tp_wire_min_bytes` | invert into `all_reduce`'s `exact`; below the floor a message must go exact |
+| `kv_dtype`, `kv_quant` | the attention cache width and, under fp8, its narrow layout (`RAD_KVQ_V4`, `RAD_KVQ_K6V4`) |
+| `attn_prefill_fp8` | `RAD_ATTN_FP8_*` bits to pass as `attn_paged`'s `prefill_fp8` |
+| `kv_resident_blocks`, `kv_stream_ok` | KV streaming: the ring depth, and permission to declare the streamed-only ops |
+| `state_dtype` | `RAD_F16` when `--state-cache-dtype f16` is served by this plugin |
+| `is_draft`, `scope` | a drafter/encoder declare; `scope` prefixes declared names (`draft`, `vision`, `audio`) |
+| `shape_probe` | a **sizing** declare at a smaller `max_tok`; must not touch state the real declare wrote |
+| `max_enc_patches`, `max_audio_frames` | the encoder pass sizes |
+| `max_out_rows` | logits rows the KL mode needs (0 = the plugin's own rule) |
+| `cvec` | the loaded control vector, or null |
+| `tp_attn_rows` | row-split prefill attention requested |
+| `draft_coupling`, `draft_tree` | declare the coupled/tree forms |
+| `weight_prefetch_off` | declare no weight prefetch beside collectives |
+
+## Declare: weights
+
+```c
+static const RadWeightDecl d = {
+    .dtype = RAD_BF16, .rank = 2, .shape = {n_embd, n_ff},
+    .access = RAD_ACCESS_PER_TOKEN, .shard = RAD_SHARD_ROW,
+    .group = grp_layer(l), .optional = 0,
+    .n_row_parts = 0, .source = nullptr, .planes = nullptr,
+};
+rad_weight w = rad_decl_weight(b, "blk.0.mlp.gate.weight", &d);
+```
+
+- **`access`** tells the placement planner how the weight is read: `PER_TOKEN` (dense),
+  `PER_REQUEST` (an encoder tower, read once per request), `CONDITIONAL` (routed experts),
+  `RARE`, `VOCAB` (lm_head/embedding: read every token but huge, and one is a gather).
+- **`shard`**: `NONE` (replicated), `ROW` (split dim 0), `COL` (split dim 1, needs a following
+  all-reduce). The plugin says; the core does not infer. Uneven splits use
+  `rad_weight_shard_span`/`_parts` (a width that does not divide into whole 128-column blocks; the
+  delta net's `[q|k|v|z]` of unequal part widths).
+- **`group`** is the unit of movement (`layer`, `expert`, `slot`); an expert's gate/up/down share one,
+  because the unit of transfer is an expert, not a tensor.
+- **`row_parts`** describes what a `SHARD_ROW` weight stacks along dim 0 (`[gate|up]`,
+  `[q|k|v|z]`). **A tensor-parallel rank owns a slice of each stacked projection**, never a
+  contiguous slice of the stack; the parts cannot be inferred from a fused container tensor, and a
+  contiguous slice produces fluent wrong text.
+- **`source`/`planes`** view an existing stored weight: a block-fp8 linear's `codes,scale` are one
+  logical weight taken by a GEMM as two operands. Null `source` is the weight of its own name; null
+  `planes` takes every plane.
+- A dtype other than the plane's is legal only as an exact widening (a bf16 norm read as f32).
+
+The **name map** (`rad_decl_name_map`) is the plugin's map from checkpoint tensor names to declared
+weight names, and it is the only place convert-time fusions live: `COPY`, `CONCAT`, `WNORM` (a
+weight-normalised pair folded to the plain weight), `ROWS`, `TRANSPOSE`, `EMA` (an EnCodec codebook),
+`TAP`, `COLS`. `rad-convert` reads this map from the plugin, so it needs no knowledge of the model.
+`RAD_MAP_COPY` with several sources means "first present wins" (a tied lm_head); `CONCAT` requires
+all. Ask for a weight's encoding with `rad_weight_encoding` **after** its map entry when the graph
+depends on the format (fp8 GEMM vs plain, rotated expert kernels).
+
+## Declare: buffers and KV groups
+
+`rad_decl_buffer` names every activation: `RAD_BUF_TRANSIENT` (liveness-analysed, may share bytes),
+`RAD_BUF_PERSIST` (across steps: a calibration accumulator, a routing histogram), or
+`RAD_BUF_DERIVED` (computed once per step by the core from the batch — `derive` names the
+derivation, e.g. `conv_state_index`). Buffers live in one arena sized once, not an allocator per
+step.
+
+`rad_decl_kv_group` declares each state kind: `RAD_KV_FULL` (paged per-token, growing),
+`RAD_KV_WINDOW` (bounded by a window), `RAD_KV_LINEAR` (per-sequence fixed state, GDN),
+`RAD_KV_CONV` (a rolling window of `width-1+num_spec`), `RAD_KV_RING` (window floor set at runtime).
+The **block size is not declared** — it comes from the resolved attention kernel's constraints; read
+it back with `rad_kv_block_size`. `rad_bind_layer_kv` binds each layer so the block manager knows how
+many state instances a sequence owns; `rad_kv_group_zero` (a rank with no heads),
+`rad_kv_group_replicated` (every rank holds the same bytes), and `rad_kv_group_dense_read` (read over
+the whole history every step, so it cannot stream) are the three refinements.
+
+## Declare: ops
+
+```c
+rad_op h = RAD_OP(b, "gemm_w4a8_nt",
+                  RAD_PARAMS(RAD_RANGE("M", 1, max_tok), RAD_INT("N", 8192), RAD_INT("K", 5120)),
+                  RAD_WEIGHTS(w));
+rad_op_reads (b, h, RAD_BUFS(b_x), n);
+rad_op_writes(b, h, RAD_BUFS(b_y), n);
+```
+
+- Parameter names and operand order come from `docs/OPS.md`; a new name is allowed only when the
+  vocabulary has none, and the first plugin to declare it fixes its schema.
+- `RAD_RANGE` on exactly one parameter per op gives the selector its bucket table. The core carries
+  every parameter opaquely except `RAD_PROLE_SEQ_CHUNK` (cut steps on it) and `RAD_PROLE_CAPACITY`.
+- `rad_op_reads`/`rad_op_writes` are the buffer planner's liveness. An undeclared use is assumed live
+  for the whole program: sound but wasteful.
+- **Fusion is selection.** Ask for the fused op; if `rad_op_resolved` is false, emit the chain. A
+  fusion's `replaces` hook is a checkable equivalence, not a rewrite.
+- `rad_op_declared(b, "op")` asks whether any library declares an optional op; use it before
+  emitting a fused form only an optional library defines.
+- Buffers touched from the **second lane** go through `rad_buf_concurrent`, and an architecture whose
+  lane-1 work lies in one stretch declares `rad_lane1_window_begin/end` around it. The runtime
+  refuses a lane-1 op outside the window; the window lets buffers used only outside it share bytes.
+
+Step output: `rad_declare_logits` names the buffer the sampler reads (its dim 0 bounds `n_out`; a
+[n_tok, vocab] logits buffer at a large vocabulary is gigabytes — declare the gather +
+`logits_gemm` shape instead). An embedding program declares `rad_declare_pooled` instead (one row a
+sequence, no sampler, no generation). `rad_declare_scores` opts into host-side scored steps.
+
+## Declare: drafters, encoders, generators
+
+- `rad_declare_drafter(RadDrafterDecl)` — depth, block, proposal shape. `rad_drafter_shared_kv` (a
+  drafter with no cache of its own), `rad_drafter_tree` (a tree over the proposals),
+  `rad_drafter_short_block` (a second, shorter pass) are refinements. Declare the drafter **last**.
+- `rad_declare_encoder(RadEncoderDecl)` — a vision/audio tower, media preprocessed by the core.
+- `rad_declare_generator` / `rad_declare_audio_generator` / `rad_declare_video_generator` — image,
+  audio and video generation programs; the engine sizes the step from the probe's `gen_*` fields
+  before declare.
+- Optional format exports let a model state what it knows and the server learn nothing new:
+  `rad_arch_chat_format` (a data description of reasoning/tool-call markup, replacing template
+  analysis), `rad_arch_audio_format` (how a speech model hears: clip tokens, prompt, language,
+  transcript start), `rad_arch_text_prepare` (rewrite request text into training form),
+  `rad_arch_video_positions`/`rad_arch_video_pack` (video pass geometry and packing),
+  `rad_arch_encode_geom` (encoder pass geometry). Each struct grows only at its end behind
+  `struct_size`, so adding a field does not move the ABI.
+
+**Media preprocessing is the core's**, not the plugin's: resizing, patching, placeholder expansion
+and positions in six processor families (Qwen-VL, Gemma 4, LFM2-VL, Unlimited-OCR, GLM-5.3, Pixtral
+for Nemotron), chosen by `vision_config.model_type` plus owning-model checks; audio through Gemma 4's
+extractor or Qwen3-ASR's Whisper one, chosen by `audio_config.model_type`. A vision model with another
+pipeline needs a core change.
+
+## Step: issuing the graph
+
+`RadBatch` is built once per step and shared by every block: `phase` (`PREFILL`/`DECODE`/`MIXED`),
+`n_tok`, `n_seq`, `token_ids`, `positions`, `cu_seqlens`, `seq_ids`, `lora_slot`, `n_out`/`out_ids`,
+`q_lens`/`ctx_lens`/`max_q_len`/`max_ctx_len`, the KV-group array, the speculative fields
+(`n_spec`, `draft_pass`, …).
+
+```c
+static void step(RadCtx* c, const RadBatch* batch) {
+    if (batch->draft_pass > 0) { m.ds.step(c, batch); return; }      /* query pass */
+    if (batch->draft_pass < 0) { m.ds.context(c, batch); return; }   /* context pass */
+    RAD_ISSUE(c, m.op_embed, praw(batch->token_ids, RAD_I32, batch->n_tok),
+              RAD_W(m.w_tok), brows(m.b_x, batch->n_tok));
+    for (auto& layer : m.layers) { layer.attn.step(c, batch); layer.mlp.step(c, batch); }
+    /* ...heads; the lm_head only over batch->n_out rows... */
+}
+```
+
+- `rad_issue(c, op, operands…, n)` / `RAD_ISSUE` (n = the batch's ranged extent) /
+  `RAD_ISSUE_N` (an explicit row count). `rad_step_fail(c, what)` sets the status helpers return.
+- **A mixed step is the trap.** Continuous batching produces steps with a prefill chunk beside decode
+  rows. A block that issues one path for the whole step sends its decode rows down the chunked path,
+  which commits recurrent state directly and takes no `num_accepted`; a rejected draft cannot be
+  rolled back, and the model stays fluent while drifting. Issue the recurrent path over
+  `RadBatch::n_seq_decode` sequences and the chunked path over the rest (the core orders steps
+  decode-first, so each half is contiguous; `cu_seqlens` holds absolute offsets, so pass
+  `cu + n_seq_decode` with the token operands at their base).
+- A step issued on lane 1 outside the declared window fails by name. An op the core appends behind a
+  declared one (runtime LoRA) counts at that op's position.
+- **Zero rows is not a call**: issuing an op over zero rows (a zero ranged extent, or all-empty
+  sequences) returns `RAD_E_SHAPE` and writes nothing. The caller skips the issue, as the
+  architectures' decode and prefill halves do.
+- `n_out` is 0 for a chunk that finishes no sequence — do not run the lm_head.
+
+## Probe: what the plugin measured
+
+`rad_arch_probe(meta, out)` runs before declare. Its fields are the engine's permissions:
+
+| field | effect |
+|---|---|
+| `draft_depth`, `draft_depth_fixed` | the container's spec depth; a block drafter's trained block is the only depth it serves |
+| `shape_probe_ok` | the plugin tolerates a sizing declare, so the activation arena can be lent |
+| `enc_merge`, `audio_frames` | encoder pass granularity; longest audio clip |
+| `gen_rows`, `gen_patches`, `gen_audio`, `gen_ctx`, `gen_video` | generator sizing before declare |
+| `kv_quant_ok` | bits of the narrow KV layouts this model's kernels serve |
+| `chain_pays`, `tree_pays` | the plugin measured chained launches / draft trees faster |
+| `kv_stream_ok` | the architecture can stream a KV group (sparse, block-addressed, with `kv_resolve`) |
+| `control_vector_ok` | the plugin applies `RadBuildCtx::cvec` |
+| `tp_attn_rows_ok` | the plugin serves row-split prefill attention |
+| `state_f16_ok` | the plugin declares recurrent state at f16 and its kernels read it |
+
+A flag whose probe bit is 0 is **refused at startup with the plugin named** (`--kv-cache-dtype
+fp8_v4`, `--control-vector`, `--tp-attn-split rows`, `--state-cache-dtype f16`, `--kv-resident`).
+That is the design: a deployment switch the plugin cannot honour is an error, never a silent
+substitution.
+
+## Worked pattern: the smallest complete plugin
+
+`arch/llama_fp8/llama_fp8.cpp` builds twice from one source — `llama_fp8` (quant `fp8_e4m3`) and
+`llama_dense_fp8` (quant `""`), differing only in the descriptor and the graph-dump text. It shows
+the whole shape:
+
+1. `struct Layer { AttnFP8 attn; MlpFP8 mlp; }`, `struct Model` holding `Names`, `Geom`, buffer and
+   op handles.
+2. `declare()`: name map entries first, then weights (with `rad_weight_encoding` where the head's
+   format decides the kernel), buffers, KV group, the embedding and its all-reduce, then each layer's
+   `declare()` from the shared blocks, the final norm, gather and lm_head, **the drafter last**, and
+   `rad_note` lines explaining the shape.
+3. `step()`: draft passes first (sign selects query/context), then embedding (+ exact all-reduce
+   when needed), each layer's `step()`, the final norm, and the head only over `n_out`.
+4. `probe()`: reads `draft.block_size` from the metadata and refuses any other depth for a block
+   drafter.
+5. `RAD_ARCH_PLUGIN(...)` at the bottom.
+
+Add a new family by copying a neighbour, replacing the name map and geometry, and composing the
+blocks you need; add a new *format* by cloning the plugin with another `(id, quant)` pair and the
+ops that format needs.
+
+## Verifying a new plugin
+
+```sh
+radiance --model m.rad --debug-graph          # every op, its resolved kernel, and why
+radiance --model m.rad --debug-placement      # the plan
+rad-convert $SRC -o m.rad --plan-only -v      # a convert that plans will boot
+rad-kbench -m m.rad --kernels mylib,libr4d,libref --report k.md --bench
+rad-tune -m m.rad --tp 2                      # only at the shapes you serve
+```
+
+`tests/arch_test.cpp` drives a plugin's builder the way the core does, including lane-window
+liveness, so the pinned graphs and steps are the ones the core plans. A plugin that declares a weight
+form no kernel `layout` accepts is a declare-time error naming the weight, encoding and kernel — the
+same check `rad-convert` runs before writing.
+
+## Related
+
+- [[Radiance Plugins]] — kernel libraries, encodings, the build path, ABI law.
+- [[Radiance Conversion]] — how the name map drives a convert, and `--set`/descriptor selection.
+- [[Radiance Model Families]] — the existing `(id, quant)` pairs and their recipes.
+- [[Radiance Tools]] — `rad-kbench`, `rad-tune`, `rad-info --plugins`.

serverAppConfigs-docs/LLM/Radiance Conversion.md

@@ -0,0 +1,354 @@
+---
+type: "howto"
+title: "Radiance Conversion — checkpoints into .rad containers"
+description: "Turning a checkpoint into a .rad container: rad-convert, the recipe language, quantisers, reuse and export."
+tags:
+  - llm
+  - radiance
+  - quantization
+  - howto
+timestamp: 2026-10-30
+---
+
+# Radiance conversion
+
+`rad-convert` turns a Hugging Face checkpoint into a `.rad` container, choosing each weight's
+encoding from a **recipe**. `rad-export` wraps it for finetunes (a merged checkpoint or a PEFT
+adapter) with the stock model's recipe. Neither opens a GPU: conversion is CPU-only and safe beside a
+running server.
+
+Fetch checkpoints with `scripts/hfget.sh <org/repo> <dir>` (resumable; reads `.safetensors`,
+skips `.bin`/ONNX/GGUF when safetensors exist; `HF_TOKEN` for gated repos; writes a
+`.hfget-manifest` of content keys). `radiance --model <owner/name>` runs the same fetch itself.
+
+## What a conversion does
+
+1. Read `config.json` (or GGUF metadata), and the shard index if present.
+2. Load the architecture, kernel and quantiser plugins and run **declare**, as the engine does.
+3. Resolve every op and ask each resolved kernel's `layout` hook about the planned encoding. A
+   recipe no kernel can serve is refused **here**, before any I/O.
+4. Write each logical weight once: the checkpoint's bytes where no rule matches, otherwise the
+   planes the matching quantiser writes, a block of rows at a time.
+5. Measure every quantised weight against its source through the core's decoder and print the worst.
+6. Bake the tokeniser, chat template, metadata, recipe and any draft head.
+
+```sh
+build/bin/rad-convert /srv/models/hf/Qwen/Qwen3.8-27B -o qwen3.8-27b-bf16.rad \
+    --recipe data/recipes/q38-27b-bf16.recipe
+build/bin/rad-convert /srv/models/hf/Qwen/Qwen3.8-Flash-Next -o flashnext-bf16.rad --max-ctx 4096
+```
+
+## Every `rad-convert` option
+
+| option | meaning |
+|---|---|
+| `<input>` | a `.gguf`, a `.safetensors`, an `index.json`, or a checkpoint directory |
+| `-o, --out FILE` | the container to write |
+| `--recipe FILE` | one rule a line: `PATTERN QUANTISER key=value …`; repeatable; first match decides |
+| `--quant RULE` | `PATTERN=QUANTISER[:key=value,…]`, placed **ahead of** the recipe's rules; repeatable |
+| `--list-quantizers` | every loaded quantiser and the options it takes, then exit |
+| `--arch ID` | override the architecture id read from the checkpoint |
+| `--home DIR` | `$RADIANCE_HOME` (kernels, architectures, quantizers) |
+| `--kernels A:B` | kernel libraries to load, first wins; declare checks the recipe against them |
+| `--tokenizer F` | `tokenizer.json`; its declared chain is baked instead of reconstructed from a pre-tokenizer name. Required for a GGUF, without it the chain is a guess |
+| `--tokenizer2 F` | a second `tokenizer.json` baked as the second vocab (a generator reading its prompt through two tokenisers); keep `tokenizer_config.json` beside it |
+| `--add FILE` | merge one more `.safetensors`, tensor names kept; repeatable; a name in two files is refused |
+| `--config DIR` | read `config.json`, `generation_config.json`, `tokenizer.json` from DIR for a single-file checkpoint |
+| `--chat-template F` | a Jinja template baked in, overriding the checkpoint's (or its absence); trailing newlines stripped; without one `/v1/chat/completions` answers 501 |
+| `--imatrix F` | llama.cpp imatrix: expert-popularity profile, per-column importances, error weighting; required by `ggml` types `iq2_xxs`, `iq2_xs`, `iq1_s` |
+| `--set K=V` | a metadata key, seen by declare **and** written into the container; repeatable |
+| `--max-tok N` | the `max_tok` declare runs at (default 8192, or a generator's largest pass) |
+| `--max-ctx N` | the context declare runs at (default: the checkpoint's training context) |
+| `--max-spec N` | the `max_spec` declare runs at (default 4, or the block a drafter states); `0` leaves a draft head out |
+| `--draft-model DIR` | a drafter shipped as its own repository, merged under `draft.` |
+| `--kv-cache-dtype T` | `bf16` or `fp8`: the cache width declare runs at; the container is identical at either |
+| `--reuse FILE` | copy every weight an earlier container holds with the same encoding, quantiser, options and shape |
+| `--changed FILE` | with `--reuse`: checkpoint tensors (one name a line) that differ; every weight reading one is converted |
+| `--in-place` | with `--reuse`: extend that container; new weights go past its end, the header last; keeps `<file>.pre-append` |
+| `--lora FILE\|DIR` | merge a PEFT adapter into the checkpoint tensors before converting (`W + strength · scale · B·A`); repeatable, merged in order; each merged tensor joins `--changed` |
+| `--lora-strength F` | strength for every `--lora` (default 1) |
+| `--lora-map FILE` | JSON map for adapters whose module names differ: `{"strip":[prefix…],"modules":{"<pat>":[target…]}}`; one `*` matches a block index; several targets mean a fused module (rows split in order) |
+| `--plan-only` | declare, plan every encoding, print the plan, stop before writing |
+| `--report N` | print the N worst quantisation errors (default 20) |
+| `-v, --verbose` | per-weight detail |
+
+`--reuse` matches a weight on its **encoding, quantiser, options and shape — not on its source
+bytes** — and carries whatever calibration made the old planes. Use it only when source and
+calibration are unchanged, or name the changed tensors with `--changed`. `--in-place` refuses to
+change a weight the container holds or to add an expert.
+
+## The recipe language
+
+```
+# PATTERN                      QUANTISER  OPTIONS
+blk.*.ffn_*_exps.*.weight      gptq       codes=i4 group=128 scale=bf16 transform=fwht128 calib=$CALIB
+blk.*.ple_ngram.weight         rtn        codes=fp8_e4m3 block=*x* scale=bf16 rule=fixed scale_value=1.99317932128906e-4
+output.weight                  rtn        codes=fp8_e4m3 block=128x128 scale=bf16
+blk.*.weight                   rtn        codes=fp8_e4m3 block=128x128 scale=bf16
+```
+
+- One rule a line: `PATTERN QUANTISER [key=value …]`. `#` starts a comment.
+- `PATTERN` is a glob over the **whole logical weight name**: `*` crosses dots, `?` is one character.
+- **The first matching rule decides.** No rule → the checkpoint's own encoding, which must be one
+  the engine serves as is: `bf16`, `f16`, `f32`, or block-scaled FP8 (`X.weight` with
+  `X.weight_scale_inv`). Anything else is named and refused.
+- `$NAME` / `${NAME}` in a value comes from the environment; an **unset variable is refused**, never
+  expanded empty (so an empty `$CALIB` cannot silently produce an uncalibrated convert).
+- A quantiser may **decline** a weight (`RAD_E_UNSUPPORTED`), which keeps the checkpoint's encoding
+  — `rtn` declines a rank-1 tensor, so a pattern that also matches a norm leaves the norm alone.
+- An option the quantiser does not declare is refused, by name, before anything is quantised.
+- Two plugins naming the same quantiser are refused, both named.
+
+## Quantisers
+
+| quantiser | what it does |
+|---|---|
+| `rtn` | round to nearest on a grid; each block's scale chosen by `rule` |
+| `gptq` | error feedback against the layer's input Hessian (`calib=DIR`, damped by `damp` = 0.01 of the mean diagonal by default); `act_order=1` quantises columns by descending input energy and stores the permutation; `cd=N` adds coordinate-descent sweeps |
+| `awq` | per-input-column scaling to a searched power, then a per-block clip search, both against the Hessian; column scales are the last scale level |
+| `autoround` | sign-gradient descent on a rounding offset per value and a range factor per block; `iters` steps; slow (a 27B dense model takes most of a working day at defaults; `iters=50` a quarter) |
+| `paroquant` | channel scaling plus `stages` Givens rotations learned against the Hessian, then AutoRound on the rotated weight; the `givens` transform |
+| `m2xfp4` | M2XFP's weight-side Sg-EM: MXFP4 plus one metadata byte per group of 32; its own `m2xfp4` scheme; `adaptive=0` pins the bias, `wide=1` widens it |
+| `ggml` | a llama.cpp format (`q4_0`, `q4_1`, `q5_0`, `q5_1`, `q8_0`, `q2_K`…`q6_K`, `iq4_nl`, `iq4_xs`, `iq2_xxs`, `iq2_xs`, `iq2_s`, `iq3_xxs`, `iq3_s`, `iq1_s`, `iq1_m`), by llama.cpp's own search, imatrix-weighted, codes byte for byte |
+| `cast` | a float weight in another float dtype (`dtype=` f32, bf16, f16, fp8_e4m3, fp8_e5m2), no scale |
+| `import` | rad-convert's own, not a plugin: write the numbers a quantised release ships, in the spelling the kernels read, instead of quantising again |
+
+Every quantiser except `m2xfp4`, `ggml`, `cast` and `import` takes the grid options:
+
+| option | values |
+|---|---|
+| `codes` | `i2`…`i8`, `i16`, `i32` symmetric; `u1`…`u8`, `u16` with `zero=`; `fp8_e4m3`, `fp8_e5m2`, `fp6_e2m3`, `fp6_e3m2`, `fp4_e2m1` |
+| `table` | a codebook the codes index: `nf4`, `iq4nl`, `w4nl`, `w5nl`, `binary`, `ternary`, or `;`-separated values |
+| `group=N` / `block=RxC` | where the scales sit; `*` is a whole extent |
+| `scale` | f32, bf16, f16, fp8_e4m3, e8m0; under `scale2` also an integer sub-scale `u4`…`u8`, `i6`, `i8` |
+| `scale2`, `group2` / `block2` | a second scale level the first is relative to (default the whole tensor) |
+| `scale2_value` | the second level fixed at this value, exact in `scale2`'s dtype |
+| `zero` | an integer zero point per block — an asymmetric grid |
+| `transform=fwhtN` | rotate each row by the Hadamard first |
+| `rule` | `absmax`, `mx` (OCP shared exponent), `fixed` (`scale_value=`), `search` (least squared error over candidates and refits, imatrix-weighted) |
+| `clamp` | a signed code's range: `full` or `sym` |
+
+Named grids: MXFP4 = `codes=fp4_e2m1 group=32 scale=e8m0`; NVFP4 =
+`codes=fp4_e2m1 group=16 scale=fp8_e4m3 scale2=f32`; NF4 = `table=nf4 group=64`; a K-quant-like grid
+= `codes=i4 group=32 scale=u6 scale2=f16 group2=256`; a ternary BitNet weight = `table=ternary`.
+
+**A container can hold only what some kernel reads.** rad-convert checks every planned encoding
+against the `layout` hooks of the kernels declare resolved and refuses one no kernel accepts. libr4d
+reads the plain floats; block-scaled fp8 and fp8 rows; int8 block rows plain or rotated; routed
+experts as fp8, int4 (`i4*bf16[1x128]/fwht128`) or the `w4nl`/`w5nl` codebooks under E4M3 scales;
+the dense w4, w8 and mxfp4 GEMM rows; and the 2-bit draft head. See [[Radiance Plugins]] for the
+encoding model.
+
+## Importing a release someone else quantised
+
+These forms are read weight by weight and written in the spelling the kernels read; a recipe rule
+re-quantises from the exact dequantisation where the family's kernels want another form.
+
+| `quantization_config` | read as | notes |
+|---|---|---|
+| `quark` MXFP4 (fp4, group 32, e8m0) | `fp4_e2m1*e8m0[1x32]` | byte for byte; also servable as it stands |
+| `paroquant_mxfp4` | `fp4_e2m1*e8m0[1x32]/givens` | codes/exponents byte for byte plus learned rotations as `givens` tables |
+| `paroquant`, `int5-bitplane`, plain `paroquant` | `u5*f16[1x128]-u8[1x128]/givens`, `u4*…` | AWQ-packed nibbles re-laid as a code bit stream; f16 scales, integer zeros, rotations |
+| `compressed-tensors` `pack-quantized`, 4-bit `int`, group, act-order none/static/weight | `i4*<scale>[1xG]` | `weight_packed` made signed beside `weight_scale`; group/dynamic reorder is refused |
+| same with `symmetric:false` (AWQ) | `u4*<scale>[1xG]-u4[1xG]` | nibbles plus `weight_zero_point`, bit for bit |
+| `auto-round` `auto_round:auto_gptq`, `gptq` v1 | `i4*bf16[1x128]` or `i8*bf16[1x128]` | scales rounded to bf16; convert-time; `import` keeps shipped f16 |
+| `nvfp4-pack-quantized` | `fp4_e2m1*f32[1x16]` | `weight_scale / weight_global_scale`; **no kernel serves NVFP4**, so it always takes a rule |
+| `modelopt` NVFP4 / FP8 / MIXED_PRECISION | NVFP4 `fp4_e2m1*f32[1x16]`; FP8 `fp8_e4m3*f32[1x*]` | ModelOpt's second scale **multiplies**; `input_scale` is not read |
+| compressed-tensors all-FP8 groups | `fp8_e4m3*<scale>[1x*]` or `[RxC]` | nothing imported: stock FP8 with its scale; a per-tensor scale is written once a row |
+| GGUF `Q8_0`, `Q4_K`, `Q6_K` | `i8*f16[1x32]`, affine u4/u6…, `i6*i8…` | blocks split bit for bit; **not servable as they stand** |
+
+The width is read **per tensor**, so a mixed release imports int4 experts and int8 projections
+together. What an import cannot read (an asymmetric GPTQ zero point, a group-reordering act-order
+index, a GPTQ group under 128, a missing/zero global scale, an unlisted format) is refused by name.
+The release's `quantization_config` names the tool, not a served form, so it is kept as
+`imported_quantization_config.*`.
+
+`import` options: `codes=uN` makes a symmetric grid unsigned under a zero point of `2^(N-1)`;
+`zero=` the zero plane's dtype (`u4`, `u8`); `scale=` f16/bf16/f32, each shipped scale rounded to
+nearest; `block=RxC` repeats an FP8 scale shipped per row/tensor over that grid. It reads an imported
+integer grid or FP8 with a per-row/per-tensor scale and keeps a plain float weight as it is.
+
+**NVFP4 to MXFP4 rounds twice** (e4m3 per 16 → power of two per 32), so it loses more than MXFP4 made
+from bf16. Prefer a calibrated MXFP4 container when one exists; the quality verdict is a KL gate, not
+a file size.
+
+## The descriptor law (`--set`)
+
+The architecture plugin is chosen by the exact `(arch id, quant descriptor)` pair. A convert-time
+choice that must also hold at serve time is stated once with `--set`, which hands it to declare and
+writes it into the container. Storage format is not such a choice (the encoding is in the container
+and the plugin reads it there), but the **serving plugin** is: a `quantization_config.*` key
+re-derives the descriptor.
+
+```sh
+rad-convert <bf16 Qwen3.8-27B> -o q38-27b-fp8.rad \
+    --recipe data/recipes/q38-27b-fp8block-df2.recipe \
+    --set quantization_config.quant_method=fp8 --set quantization_config.fmt=e4m3 \
+    --set "quantization_config.weight_block_size=128 128" \
+    --set quantization_config.activation_scheme=dynamic
+```
+
+A bf16 checkpoint whose recipe writes Qwen's block FP8 must therefore be declared *as that format*;
+otherwise the selection pair does not match any plugin and the convert (or the boot) is refused. The
+same law bites the other way when a release's own `quant_method` names a pair no plugin claims.
+
+## Inspect and validate the result
+
+```sh
+rad-info out.rad                  # architecture, source quant, recipe, sections
+rad-info --recipe out.rad         # the expanded rules — did the intended one win?
+rad-info -v out.rad | head -40    # every tensor, its encoding, and what made it
+rad-kbench -m out.rad --home $RADIANCE_HOME --report kbench.md
+radiance --model out.rad --debug-graph     # every op resolves
+```
+
+The converter's own error report (the worst `--report N` weights) measures each quantised weight
+against its source through the core decoder. `rad-kbench` adds the artifact test: weight operands
+read from the container and passed through each kernel's `layout`/`relayout` hooks. Both are on
+[[Radiance Tools]].
+
+## Calibration (GPTQ, AWQ, AutoRound, ParoQuant)
+
+`calib=DIR` is a directory an engine run wrote with `RADIANCE_CALIB_DIR` set: each layer's **Gram
+matrices of its input**, in the domain the kernel multiplies it in. A block that characterises an
+activation declares an f32 `[K,K]` PERSIST buffer and a `gram_accum` op; at `RADIANCE_CALIB_TOKENS`
+(default 65536) the run writes the files.
+
+Three-step bootstrap: convert with plain `rtn` → serve that container with `RADIANCE_CALIB_DIR` →
+push real prompts through prefill → convert again with the Hessians. The second moment is robust to
+the first container's small quantisation error. A weight with no file is quantised uncalibrated and
+the converter says so; `awq`, `autoround` and `paroquant` fall back to the imatrix diagonal when one
+is given.
+
+- The accumulator must be **PERSIST**: a transient buffer may share bytes and be overwritten between
+  the launches it sums.
+- Files are per rank; the consumer sums them. Under expert parallelism a rank's `down` input covers
+  only its experts' tokens; `gate_up` input is the full token set on every rank, so the sum scales
+  `H` by world size — GPTQ is invariant to a positive scale.
+- Under tensor parallelism a column-split linear sees only its rank's columns, so a tap may write
+  `gram.r<rank>.<stem>.part.bin`; libquant places the parts on the diagonal in rank order, leaving
+  cross-rank products zero. They must cover K and be in the model's own domain.
+- `--imatrix` supplies the expert-popularity profile the placement planner ranks units by, the
+  per-column importances a quantiser uses, and the error report's weighting.
+
+## Reuse, drafters, merging
+
+```sh
+# re-quantise only what changed
+rad-convert $SRC -o new.rad --recipe new.recipe --reuse old.rad --changed changed.txt
+
+# merge a drafter shipped as its own repo (DFlash2, DSpark, MTP assistant)
+rad-convert $SRC -o out.rad --draft-model /srv/models/z-lab/Qwen3.8-27B-DFlash2 \
+    --recipe data/recipes/q38-27b-mxfp4-df2.recipe
+
+# merge a PEFT adapter at convert time
+rad-convert $SRC -o tuned.rad --lora ~/run/adapter --lora-strength 1.0 \
+    --recipe data/recipes/q38-27b-mxfp4-df2.recipe
+```
+
+- A drafter's tensors go in under `draft.`, its config under `draft.` keys. It has no embedding and
+  no `lm_head` of its own; it uses the target's.
+- `--lora-map` exists for adapters trained on names the checkpoint does not use; a fused target
+  (`qkv`) takes several targets and splits the adapter's rows in order.
+- `--add` + `--config` + `--tokenizer2` assemble a ComfyUI-style release of separate single files
+  (Anima: diffusion model, text encoder, VAE). A name in two files is refused.
+
+## `rad-export` — finetunes without hand-writing the command
+
+```sh
+rad-export ~/outputs/my-run/checkpoint-400 --dry-run   # what it would run
+rad-export ~/outputs/my-run/checkpoint-400             # adapter -> <name>.rad
+rad-export ~/exports/merged -o ~/models/mine.rad --recipe q38-27b-fp8block-df2
+```
+
+| flag | default | meaning |
+|---|---|---|
+| `SRC` | required | a merged 16-bit checkpoint (`config.json` + `*.safetensors`) or a PEFT adapter (`adapter_config.json` + `adapter_model.safetensors`) |
+| `-o, --out FILE` | SRC's folder name + `.rad`, beside SRC or in `out_dir` | the container |
+| `--recipe NAME\|FILE\|auto` | `auto` | a recipe name, a file, or the one recipe whose header names the base model, takes a bf16 checkpoint and needs no calibration |
+| `--base DIR` | the adapter's `base_model_name_or_path`, from disk or the HF cache, else downloaded | the base a LoRA merges into |
+| `--draft DIR\|ORG/NAME` | the config, else the Hub repo the recipe's usage line names | the drafter when the recipe merges one; a usage line with a placeholder asks for `--draft` |
+| `--image IMAGE` | `$RADIANCE_IMAGE`, the config's, then `stilldeadcode/radiance` | run `rad-convert` in this image (docker, CPU-only, no network, **not pulled**) |
+| `--rad-convert PATH` | — | run this binary instead of docker; recipes come from its `share/radiance` |
+| `--memory M` | — | docker memory limit for the convert |
+| `--dry-run` | — | resolve recipe, drafter and paths; print the `rad-convert` line; write nothing |
+| `--keep-merged` | — | keep the merged checkpoint beside OUT |
+| `--config FILE` | `~/.config/rad-export/config.json` | the config below |
+| `--list-recipes [--json]` | — | list the recipes the image/installed tool carries |
+
+- Input: a merged 16-bit checkpoint (`config.json` + `*.safetensors`) or a PEFT LoRA adapter
+  (`adapter_config.json` + `adapter_model.safetensors`).
+- An adapter is merged **exactly as PEFT's `merge_and_unload`** computes it on CPU: `scale` =
+  alpha/r (alpha/√r with rsLoRA), `rank_pattern`/`alpha_pattern` honoured, delta in fp32 rounded to
+  the adapter's dtype. It streams tensor by tensor into 1 GiB shards, so a 27B merge needs a few GiB
+  of RAM, not the model twice. Plain linear LoRA only: embedding LoRA, trained biases, DoRA, saved
+  modules, convolution LoRA, a rank other than the config's, and a missing target are refused before
+  anything is written. The adapter's tokeniser/template replace the base's where shipped.
+- The recipe comes from the image (`/opt/radiance/share/radiance/recipes`) or the installed
+  `rad-convert`'s share; `auto` matches the base model's name in recipe headers and asks when several
+  fit. A recipe needing calibration is listed but never picked by `auto` — a finetune's Hessians are
+  not the stock model's.
+- It runs `rad-convert` on the CPU, under Docker as the caller with no network, and writes the log to
+  `OUT.log`.
+- `~/.config/rad-export/config.json` pins what auto cannot know: `image`, `out_dir`,
+  `recipes{model:recipe}`, `drafts{recipe:dir}`, `bases{repo:dir}`.
+- It is Python 3 with no dependencies; merging needs `torch`/`safetensors`, fetching a missing base or
+  drafter needs `huggingface_hub`. The image carries no Python; extract the script with
+  `docker run --rm --entrypoint busybox stilldeadcode/radiance cat /opt/radiance/bin/rad-export`.
+
+## Scoring a container (the KL mode)
+
+```sh
+# once, on the bf16 container
+radiance --model bf16.rad --kld-record /srv/kld/q38 --kld-corpus corpus.jsonl
+# each candidate, same flags otherwise
+radiance --model quant.rad --kld-ref /srv/kld/q38 --kld-out report.json
+```
+
+- The corpus is JSONL, one `{"prompt", "score_from"?, "source"?}` a line; only `prompt` is required.
+  Positions `[score_from, n-1)` are scored; `source` groups the report. Keep the scoring corpus
+  disjoint from the calibration corpus.
+- `--kld-record` writes the reference's log-probabilities (f16 over the vocabulary, the 16 most
+  likely tokens exactly, the token ids). DIR is never overwritten and is read only once `kld.json`
+  exists, which is written last.
+- `--kld-ref` scores this model's f32 logits against stored rows: mean, median, 99/99.9/99.99th
+  percentile KL, top-1 agreement, and both models' perplexity. `--kld-out` writes the report; `FILE.rows`
+  is `[positions, 4]` f32 (KL, both NLLs, argmax agreement) for position-by-position comparison.
+- The mode turns speculation and the prefix cache off, turns the step pipeline off, and refuses an
+  explicit spec depth above 0. The logits buffer holds a row per token of a step, so a large
+  `--max-num-batched-tokens` costs pinned host memory as well as VRAM.
+- `--kld-lora NAME` scores every document through a served adapter, against a reference recorded
+  from the merged checkpoint; `--kld-lora-sources PREFIX` runs only matching documents through it,
+  the rest as the base, in the same steps — each document's rows should be byte-identical to the
+  all-adapter and all-base runs.
+
+The gate the fleet uses (mean KL ≤ 0.03, top-1 ≥ 95 %) is on [[frank-archon]] and
+[[Quantization Production]]; the mode here is only the instrument.
+
+## Convert laws paid for in full
+
+- **`--plan-only` first.** A plan that resolves every op will boot; it costs seconds against an hour
+  of I/O on a large checkpoint.
+- **Declare runs at the step and context bounds, and some graphs depend on them.** Qwen4-Exp sizes
+  its QSA buffers by `--max-ctx` and refuses a `--max-tok`/context pair whose score plane would pass
+  2 GiB; the defaults (8192 tokens, the checkpoint's training context) fit most models, and a
+  generator's largest pass may override `--max-tok`.
+- **Scale names follow the tensor.** `scale_name()` appends `_scale_inv` to the *tensor* name, so a
+  stacked `…experts.gate_up_proj` keeps its scales at `…gate_up_proj_scale_inv`.
+- **File shadowing: first wins.** The reader opens index-named files sorted and emplaces every tensor
+  each file physically holds, first-wins. A merged PLE shard must sort first
+  (`00-ple.merged.safetensors`) or a raw shard wins the name.
+- **Never write through a staging symlink.** A checkpoint may symlink its raw index; `open(path,"w")`
+  writes through it and corrupts the checkpoint. Write a temp file and `os.replace`.
+- **One encoding per expert table.** A whole expert kept bf16 is the only thing legal beside a
+  table; a split tier (down=fp8, gate_up=w4) is unservable.
+- **The descriptor pair has no fallback** (above), and **convert is card-less**: GPTQ solves on CPU
+  BLAS, so the GPU is irrelevant to production and the card only decides what can be decoded.
+
+## Related
+
+- [[Radiance Model Families]] — the (arch id, quant descriptor) table and the command per model.
+- [[Radiance Plugins]] — quantiser and kernel-plugin contracts behind these options.
+- [[Radiance Tools]] — `rad-info`, `rad-kbench`, `rad-tune`.
+- [[Quantization Production]] — the fleet's calibration/KLD campaign and its gates.
+- [[frank-archon]] — the deployment whose recipes and convert laws these are.

serverAppConfigs-docs/LLM/Radiance Engine.md

@@ -0,0 +1,140 @@
+---
+type: "reference"
+title: "Radiance Engine — the inference server behind frank-archon"
+description: "The radiance inference engine: its shape, plugin model, container format, and how the reference pages fit together."
+tags:
+  - llm
+  - radiance
+  - reference
+timestamp: 2026-10-30
+---
+
+# Radiance Engine — shape, plugin model, container
+
+Radiance is StillDeadcode's inference server for AMD GPUs, written in C++20 and HIP: language
+models, plus image, video, speech and music generation and embeddings. One HTTP front end speaks
+OpenAI's chat, completions, Responses, images, audio and video APIs, Anthropic's Messages API and
+llama-server's native one. Nothing in the serving path is Python.
+
+The fleet's only radiance endpoint is [[frank-archon]] (2× R9700, TP=2, Dankserver). **This page
+and its siblings own the engine reference; the live deployment shape, its Eras and its measured
+laws stay on the endpoint page.** The numbers here describe upstream `main` as read 2026-10-30
+(release 1.14.0, plugin ABI 31). The fleet pin is older (1.8.1): a flag below that post-dates the pin
+is a boot refusal, never a silent no-op, and the endpoint page records the CLI families that pin
+itself retired relative to its own rollback target.
+
+Source: `https://codeberg.org/StillDeadcode/radiance` at `8da043c`. Engine docs live in `docs/`
+(`GUIDE.md` for serving, `TOOLS.md` for tools, `PLUGIN.md` for plugins, `ARCHITECTURES.md` per
+family, `OPS.md` for the op vocabulary, `spec.md` for the design).
+
+## The four moving parts
+
+```mermaid
+graph LR
+    A[Checkpoint<br/>safetensors / GGUF] -->|rad-convert + recipe| B[.rad container]
+    B --> C[radiance engine]
+    C --> D[OpenAI / Anthropic /<br/>llama-server wire APIs]
+    P1[kernel library .so<br/>libr4d, libavx, libref] --> C
+    P2[architecture .so<br/>one per arch id × quant descriptor] --> C
+    P3[quantiser .so<br/>libquant] -->|rad-convert| A
+```
+
+- **Engine** — `core/`: declare phase, placement/memory planner, scheduler, KV manager, sampler,
+  tokeniser, HTTP server, media preprocessing.
+- **Kernel libraries** — implement ops for hardware. `libr4d` targets RDNA4 (`gfx1200`,
+  `gfx1201`); `libavx` serves the host backend (AVX-512/AVX2) with no GPU at all; `libref` is the
+  slow reference implementation every other library is checked against.
+- **Architecture plugins** — declare one model family at one weight format: its weights, buffers,
+  KV groups and the ops it issues each step. The engine picks one by an **exact (architecture id,
+  quantisation descriptor) pair, with no fallback**.
+- **Quantiser plugins** — `libquant`'s `rtn`, `gptq`, `awq`, `autoround`, `paroquant`, `m2xfp4`,
+  `ggml`, `cast`, `import`; they write a weight's numbers into an encoding's planes.
+- **Tools** — `rad-info`, `rad-convert`, `rad-export`, `rad-kbench`, `rad-tune`, `rad-schemas`,
+  `rad-tokcheck`, plus `rad-chatparse`, `rad-avx-check`, `r4d_selftest` in the build tree.
+
+## Two phases, one static graph
+
+`rad_arch_declare` runs once per rank and names every weight, buffer, KV group and op, in order.
+The core resolves every op against the kernel hierarchy per band, per domain, and completes the
+whole plan before a byte of weights is read. `rad_arch_step` then only *issues* handles obtained at
+declare; it may not allocate and may not synchronise. A model whose declare succeeds is fully known
+statically, which is what makes `--debug-graph`, the buffer liveness plan and `rad-convert`'s
+plan-only check possible.
+
+Consequences a plugin author must respect:
+
+- An op that never resolves is `RAD_NULL_HANDLE` and only fails if issued. A fusion is a *selection*:
+  ask for the fused op, emit the chain when `rad_op_resolved` is false.
+- Declaration order is the buffer planner's liveness. A drafter must be declared after the trunk
+  it runs behind, or trunk and drafter buffers silently share bytes.
+- Every op in the graph must have a row in a loaded kernel library. `libref` implements the
+  conventional vocabulary in `docs/OPS.md`; device-only ops (most fusions) exist only in `libr4d`.
+
+## The `.rad` container (RAD2)
+
+A container is one file holding everything the server needs: weights as **canonical encoding
+planes**, the tokeniser (and optionally a second vocab), the chat template, metadata, the recipe it
+was made with, and any draft head. It names no kernel library.
+
+- Planes are kernel-independent: codes, scales, zero points, codebooks — row-major, bit-packed
+  sub-byte where needed. A kernel rearranges them into its own layout at load (`layout`/`relayout`
+  hooks), losslessly, so a kernel library can change layouts without reconverting the model.
+- Encodings are data, not an enum: `RadEncoding` names the scheme (`plain`, `affine`, or a
+  quantiser's own), the planes by role, the scale grouping, and a transform (`fwhtN`, `perm`,
+  `givens`). That is how one file can hold block-FP8, int4, MXFP4, NVFP4-imported, ParoQuant
+  rotations and llama.cpp grids.
+- `rad-info` is the reader: sections (strings, metadata, directory, planes, encodings, vocab,
+  profile, data), `--meta`, `--vocab`, `--recipe`, `--plugins`, and `-v` for every tensor with its
+  encoding and what made it.
+
+See [[Radiance Conversion]] for producing one and [[Radiance Tools]] for inspecting it.
+
+## Selection key and version law
+
+- A checkpoint's architecture id is its `config.json` `model_type` without underscores; `rad-convert
+  --arch ID` overrides it. `rad_arch_quant` supplies the quantisation descriptor (empty for an
+  unquantised checkpoint).
+- **No fallback.** A raw official FP8 checkpoint whose `quantization_config.quant_method` is `fp8`
+  is not the pair the `qwen4exp_fp8` plugin claims (`(qwen4exp, "")`), so both serve and convert
+  refuse it until the descriptor is changed with `--set` or a staged `config.json`. The deployment
+  lesson is on [[frank-archon]]; the mechanism is on [[Radiance Conversion]].
+- `RAD_ABI_VERSION` covers every header a plugin compiles against. A plugin reporting a different
+  number is refused at load, with both numbers printed. It moves on any shape change, not on a new
+  optional export; rebuild when it changes. Release version and ABI are independent: `/version`
+  reports the release, `rad-info --plugins` reports what loaded.
+
+## Environment
+
+| variable | meaning |
+|---|---|
+| `RADIANCE_HOME` | plugin search path: `:`-separated homes, each holding `kernels/`, `architectures/`, `quantizers/`; first wins, same-name files shadow later homes. Compiled-in default is `<prefix>/share/radiance` |
+| `RADIANCE_KERNELS` | default for `--kernels`: ordered kernel libraries, first wins; unnamed libraries are not loaded |
+| `RADIANCE_TUNE` | pin a kernel's tuned axes for a measurement (a measurement aid, not a deployment setting) |
+| `RADIANCE_CALIB_DIR` | a serve run records per-layer input Gram matrices here (the GPTQ calibration tap) |
+| `RADIANCE_CALIB_TOKENS` | tokens after which the tap stops and writes (default 65536) |
+| `RADIANCE_LOG`, `RADIANCE_LOG_STEPS` | log level; per-step token counts |
+| `RADIANCE_DEBUG_RESID` | residual-stream probe per (layer, site) |
+| `RADIANCE_DFLASH_Q_TEMP` | temperature of a DFlash2 coupled walk's candidate softmax |
+
+The full list of debug knobs (`RADIANCE_DEBUG_*`, `RADIANCE_DUMP_*`, `RADIANCE_OP_ORACLE`, …) is in
+the source's `getenv` calls; none is a deployment setting.
+
+## Where the pages are
+
+| Page | Owns |
+|---|---|
+| [[Radiance Serve Options]] | every serve flag, grouped, with defaults and the law behind each knob |
+| [[Radiance HTTP API]] | endpoints, request fields, streaming, tool calls, refusal rules |
+| [[Radiance Conversion]] | `rad-convert`, the recipe language, quantisers, imports, reuse, `rad-export`, KL scoring |
+| [[Radiance Model Families]] | which plugin claims which model, and the conversion command per family |
+| [[Radiance Plugins]] | kernel and quantiser plugins: ABI, rows, hooks, encodings, build and checks |
+| [[Radiance Architecture Plugins]] | declaring a model: weights, buffers, KV groups, ops, step, probe |
+| [[Radiance Tools]] | `rad-info`, `rad-kbench`, `rad-tune`, `rad-schemas`, `rad-tokcheck`, harnesses |
+| [[Radiance Operations]] | the container contract, host prerequisites, readiness, ledgers, restarts |
+| [[frank-archon]] | the fleet's live radiance endpoint: shape, Eras, rollback, measured laws |
+
+## Sources
+
+- `README.md`, `docs/GUIDE.md` §1–§4, `docs/TOOLS.md`, `docs/PLUGIN.md`, `docs/DOCKER.md`,
+  `docs/ARCHITECTURES.md`, `docs/OPS.md`, `spec.md` §2–§4, `abi/*.h`, `core/config.cpp`,
+  `core/rad_core.h` at commit `8da043c` (codeberg.org/StillDeadcode/radiance).

serverAppConfigs-docs/LLM/Radiance HTTP API.md

@@ -0,0 +1,198 @@
+---
+type: "reference"
+title: "Radiance HTTP API — endpoints, fields, refusals"
+description: "The HTTP surface radiance serves: endpoints, request fields, streaming, tools, and refusal rules."
+tags:
+  - llm
+  - radiance
+  - api
+  - reference
+timestamp: 2026-10-30
+---
+
+# Radiance HTTP API
+
+One server serves one model. It answers OpenAI's chat/completions/Responses/images/audio/video APIs,
+Anthropic's Messages API and llama-server's native one, so their clients and SDKs work unchanged.
+`model` in a request is accepted and **not enforced**.
+
+## Endpoints
+
+| route | what it serves |
+|---|---|
+| `POST /v1/chat/completions` | chat: tools, reasoning, streaming, grammars |
+| `POST /v1/completions` | raw text completion |
+| `POST /v1/embeddings` | an embedding head, else 501 |
+| `POST /v1/images/generations`, `/v1/images/edits` | OpenAI images API; needs an image model, else 501 |
+| `POST /v1/systemone` | LiquidAI d1 decisions: named yes/no, choice and score questions as probabilities; needs a d1 container, else 501 |
+| `POST /v1/audio/transcriptions` | OpenAI speech-to-text; needs a transcribing plugin, else 501 |
+| `GET /v1/realtime` (WebSocket) | streaming transcription session |
+| `POST /v1/audio/speech` | speech (Breeze-TTS-2, drama-3) or a song (MiniMax-Music3), whole or streamed |
+| `GET /v1/audio/speech/stream` (WebSocket) | text streamed in, each sentence's audio out |
+| `POST /v1/videos`; `GET /v1/videos[/:id[/content]]`; `DELETE /v1/videos/:id` | video jobs made in the background, downloaded as MP4 |
+| `POST /v1/messages`, `/v1/messages/count_tokens` | Anthropic Messages, for Claude Code and the Anthropic SDKs |
+| `POST /v1/responses`, `/v1/responses/input_tokens` | OpenAI Responses, for Codex and SDK clients |
+| `POST /completion`, `/completions`, `/infill` | llama-server native completion and fill-in-the-middle |
+| `GET /v1/models`, `/v1/models/:id` | what is loaded, with `max_model_len`, `root`, `capabilities` |
+| `GET /health`, `/v1/health`, `/ping`, `POST /ping` | liveness |
+| `GET /load`, `/version`, `/props`, `/server_info`, `/diagnostics` | load; release version; llama-server props; full configuration; the startup report as JSON (`/diagnostics` is same-machine-only without an API key) |
+| `POST /apply-template`, `/v1/chat/completions/input_tokens` | the rendered prompt; its token count |
+| `GET /metrics` | Prometheus |
+| `POST /tokenize`, `/detokenize`; `GET /get_tokenizer_info` | vLLM's shape and llama-server's at once |
+| `POST /reset_prefix_cache`; `DELETE /sessions/<label>`, `/sessions?id=<id>` | drop every cached prefix; forget a stored conversation |
+| `GET /`, `/dashboard`; `GET /stats`, `/graph`, `/sessions` | the dashboard and the JSON it is drawn from |
+
+Unversioned shorthands exist for chat, chat input tokens, embeddings, images, audio transcriptions,
+audio speech, models, responses and completions, for a client given the bare host as base URL.
+`/completions` **without** `/v1` is llama-server's native completion. A path nothing serves is a JSON
+404.
+
+A text prompt to `/v1/completions`, `/v1/embeddings` or `/tokenize` reads control tokens as tokens
+(as vLLM and llama.cpp do), so a client rendering its own template can send `<|im_start|>user\n…`.
+The model's declared BOS is prepended unless the prompt already opens with it.
+
+## Chat
+
+```sh
+curl -s localhost:8000/v1/chat/completions -H 'Content-Type: application/json' \
+  -d '{"model":"m","messages":[{"role":"user","content":"hi"}],"max_tokens":200,"temperature":0.7}'
+```
+
+- **No chat template → 501** on this route (the container carries one; `--override-chat-template`
+  replaces it). Use `/v1/completions` instead.
+- Thinking off has three spellings: `chat_template_kwargs.enable_thinking=false`,
+  top-level `enable_thinking=false`, or `reasoning_effort:"none"`.
+- Reasoning returns in `choices[].message.reasoning_content`, separate from `content`.
+  **Reasoning counts against `max_tokens`**; a reply that runs out inside it has empty `content` and
+  `finish_reason:"length"`. `usage.completion_tokens_details.reasoning_tokens` tells that case from a
+  model that answered nothing.
+- `thinking_token_budget` (aliases `thinking_budget`, `thinking_budget_tokens`,
+  `reasoning_budget_tokens`) closes the reasoning block at N tokens and forces the close, so it holds
+  under speculation and at any temperature. It needs a one-token end marker; otherwise the response
+  has no `completion_tokens_details` and a budget is a 400.
+- `continue_final_message` continues a conversation ending in an assistant message instead of
+  answering it; `add_generation_prompt` opens a new turn.
+- Tool calls: `tools`, `tool_choice` (`auto`, `none`, `required`, named function),
+  `parallel_tool_calls`. `required`/named hold the whole reply to calls after the reasoning block and
+  cannot combine with `response_format` or `grammar`.
+- Structured output: `response_format` (`json_object` or `json_schema`), llama-server's `json_schema`
+  shorthand, or a raw GBNF `grammar`. A chat template that drops the grammar makes the request a
+  refusal.
+
+## Streaming
+
+```sh
+curl -N localhost:8000/v1/chat/completions -H 'Content-Type: application/json' \
+  -d '{"model":"m","messages":[{"role":"user","content":"Count to ten."}],
+       "max_tokens":100,"stream":true,"stream_options":{"include_usage":true}}'
+```
+
+Server-sent events in OpenAI's delta format, terminated by `data: [DONE]`. **Nothing is sent while
+the prompt is read**: the first event (the `role: assistant` chunk) goes out with the first generated
+token, so a client timing its first event measures time-to-first-token. `return_progress` adds
+llama-server's `prompt_progress` chunks during prefill; `timings_per_token` adds `timings` to every
+chunk.
+
+## Completions, embeddings, images, audio, video, tokens
+
+- **Completions** — `/v1/completions` takes `prompt` (string or token ids), `echo`, `logprobs`,
+  `preserved_tokens`. `/completion` is llama-server's native route; `/infill` is fill-in-the-middle.
+- **Embeddings** — `input` is text, a batch, or token ids; `encoding_format` `float`/`base64`;
+  `dimensions` truncates to the first N and renormalises; `prompt_name` prepends the model's named
+  prompt. Pooling is the model's; the engine does not renormalise.
+- **Images** — `/v1/images/generations` returns `b64_json`; `size` defaults to 2048×2048. Edits take
+  one picture or up to ten (`image[]`), multipart as OpenAI's SDKs send them, with an optional mask
+  (repaint only where transparent). `partial_images` streams previews as SSE.
+- **Speech to text** — `/v1/audio/transcriptions`; `/v1/realtime` streams audio in and words back as
+  they are spoken. The model's prompt shape comes from its plugin's audio format, not from the server.
+- **Speech and music** — `/v1/audio/speech` (whole or streamed) with `instructions` describing a voice
+  (Breeze-TTS-2) or speakers written in the input (`<|speaker:1|>`, `[laughs]` for drama-3); a music
+  model takes the lyrics as `input` and the description as `instructions`. `/v1/audio/speech/stream`
+  streams text in and audio out as each sentence completes.
+- **Video** — `/v1/videos` is a job: a video with its stereo soundtrack from a prompt, 5–14 s, up to
+  1344×768, made in the background; `GET /v1/videos/:id/content` downloads the MP4.
+- **Tokenising** — `POST /tokenize` accepts vLLM's shape (`prompt` or `messages` in, `prompt` out) and
+  llama-server's (`content` in/out, `with_pieces`). `POST /detokenize` reverses it.
+  `GET /get_tokenizer_info` reports the tokeniser configuration.
+
+## Request fields
+
+Accepted fields, in OpenAI's, vLLM's and llama-server's spellings:
+
+| field | notes |
+|---|---|
+| `model` | accepted, not enforced |
+| `messages` / `prompt` / `input` | chat / completion / embedding |
+| `max_tokens`, `max_completion_tokens`, `n_predict` | default is what the context leaves; `-1` is llama-server's "no limit"; clamped to context and to `--max-tokens-cap` |
+| `temperature`, `top_p`, `top_k`, `min_p`, `typical_p` | sampler |
+| `presence_penalty`, `frequency_penalty`, `repetition_penalty`/`repeat_penalty`, `repeat_last_n` | penalties |
+| `xtc_probability`, `xtc_threshold`, `dry_*` | XTC and DRY |
+| `seed` | absent means "pick one", not 0 |
+| `n` | 1..`--max-n`; prefix cache makes the fan-out nearly free after the first |
+| `stop` | string or array, bounded by `--max-stop-strings`/`--max-stop-bytes` |
+| `priority` | scheduler hint |
+| `ignore_eos` | generate to `max_tokens` whatever the model emits |
+| `stream`, `stream_options.include_usage`, `timings_per_token`, `return_progress` | streaming controls |
+| `samplers` | llama-server's order; this engine's chain is penalties, DRY, top-k, top-p, min-p, typical, XTC, with temperature first or last; any other list is refused by name |
+| `session_id` | labels the conversation on `/sessions`, dashboard and `DELETE` |
+| `control_vector` | `true` applies the deployment's vector to this request; absent takes the deployment default; a deployment with no vector refuses `true` |
+| `logprobs`, `top_logprobs`, `n_probs` | up to 20 alternatives; not with `echo` |
+| `tools`, `tool_choice`, `parallel_tool_calls`, legacy `functions`/`function_call` | chat only |
+| `add_generation_prompt`, `chat_template_kwargs`, `continue_final_message` | chat only |
+| `reasoning_format`, `reasoning_effort`, `enable_thinking`, `thinking_token_budget`, `preserve_thinking` | chat only; levels are the template's (`capabilities.reasoning_efforts`); `none`/`minimal` turn thinking off |
+| `response_format`, `json_schema`, `grammar` | structured output; `grammar` and `response_format` are mutually exclusive |
+| `echo`, `preserved_tokens` | `/v1/completions` only |
+| `encoding_format`, `dimensions`, `prompt_name`, `image_tokens` | `/v1/embeddings` only |
+| `images_config` | image models only: a named processor mode or an object restating served sizes |
+
+## What is refused, ignored, and clamped
+
+The contract: **a 200 means every field that could change the answer was honoured.** Fields fall
+into four classes.
+
+- **Inert** — accepted and ignored because they cannot change the answer: OpenAI/OpenRouter
+  bookkeeping (`user`, `metadata`, `store`, `service_tier`, `prompt_cache_key`, …) and llama-server's
+  own speed/slot knobs (`cache_prompt`, `id_slot`, `n_cache_reuse`, `backend_sampling`,
+  `speculative.*`, …). The prefix cache and speculation here are the server's and never change output.
+- **Guarded** — known, unimplemented, and able to change the answer: accepted at their off value,
+  refused by name otherwise. `logit_bias` (empty), `mirostat` (0), `top_n_sigma` (≤0),
+  `dynatemp_range` (≤0), `min_keep` (0), `post_sampling_probs` (false), `best_of` (≤1), `suffix`
+  (empty), `lora` (empty — an adapter is chosen by `model`), `modalities`/`audio` (`["text"]`/null),
+  `n_indent` (0), `t_max_predict_ms` (≤0), `return_tokens` (false), `grammar_lazy`,
+  `grammar_triggers`, `preserved_tokens` (empty on chat).
+- **Ignored** — anything else unknown, logged at debug. An unknown field is far more often a client's
+  extension than a misunderstood request.
+- **`max_tokens` is the exception**: if prompt + `max_tokens` exceeds the context, it is **clamped**
+  and served, ending `finish_reason:"length"`, because refusing would produce no tokens instead of
+  fewer. A prompt that does not fit on its own is a 400 `context_length_exceeded` naming
+  `n_prompt_tokens`, `n_ctx`, and how many tokens must go.
+
+## Anthropic, Responses, llama-server
+
+- `/v1/messages` is translated into a chat request, so chat defaults, refusals and limits apply.
+  Streamed (`message_start`, content blocks, `message_delta`, `message_stop`) or not:
+  `ANTHROPIC_BASE_URL=http://host:8000 ANTHROPIC_API_KEY=$KEY claude` works. `/v1/messages/count_tokens`
+  returns the prompt length.
+- `/v1/responses` maps `instructions`/`input` (strings, messages, `function_call`,
+  `function_call_output`, `reasoning` items) into the chat request; `max_output_tokens`,
+  `reasoning.effort`, `text.format` map to chat fields; undefined fields pass through. The event stream
+  carries `output_index`, `content_index`, `sequence_number`, and opens with `response.created` and
+  `response.in_progress`. A reply cut at `max_output_tokens` is `incomplete`. Function tools a
+  `namespace` groups are named `<namespace>__<name>`; built-in tools and freeform custom tools are
+  left out.
+- `/completion`, `/completions`, `/infill` are llama-server's own shapes, for llama.vim/llama.vscode.
+
+## Auth, CORS, errors
+
+- `--api-key KEY` requires `Authorization: Bearer KEY` or `X-Api-Key: KEY` on every request except
+  `/health`, `/v1/health`, `/ping` and the dashboard page. An empty key is refused at startup.
+- CORS headers are sent unless `--no-cors`.
+- Errors are JSON. Over-context is 400 `context_length_exceeded`; a full queue is 429 with
+  `Retry-After`; a capability the model lacks is 501; a guarded field is 400 naming the field and its
+  accepted off value.
+
+## Related
+
+- [[Radiance Serve Options]] — the defaults every omitted field takes.
+- [[Radiance Operations]] — `/health` semantics, metrics, dashboard.
+- [[frank-archon]] — the harness contract built on this surface.

serverAppConfigs-docs/LLM/Radiance Model Families.md

@@ -0,0 +1,142 @@
+---
+type: "reference"
+title: "Radiance Model Families — arch ids, descriptors and recipes"
+description: "Which radiance plugin claims which model family, and the conversion command and recipe of record for each."
+tags:
+  - llm
+  - radiance
+  - models
+  - reference
+timestamp: 2026-10-30
+---
+
+# Radiance model families
+
+The selection key is the exact **(architecture id, quantisation descriptor)** pair, so a family at
+two weight formats is two `.so` files. A checkpoint's id is its `config.json` `model_type` without
+underscores (`qwen3_5` → `qwen35`); `rad-convert --arch ID` overrides it. `data/recipes/` is the
+authority: **read the recipe's header** — it names the exact `rad-convert` line, its `--set` keys
+and whether it needs calibration.
+
+## The plugin table
+
+| plugin | id | quant | what it serves |
+|---|---|---|---|
+| `llama_fp8` | `llama` | `fp8_e4m3` | plain LLaMA decoder (MiniCPM5, Llama 3, derivatives reporting `model_type: llama`) from a checkpoint holding E4M3 codes + `weight_scale_inv` |
+| `llama_dense_fp8` | `llama` | *(empty)* | the same graph from a bf16 checkpoint, linears made block fp8 by a recipe |
+| `qwen35_bf16` | `qwen35` | *(empty)* | Qwen3.5/3.6/3.8 dense hybrid (GDN + gated attention), bf16 |
+| `qwen35_fp8` | `qwen35` | `fp8_e4m3` | the same, block-scaled FP8 weights and activations |
+| `qwen35_w8a8` | `qwen35` | `w8a8_fwht128` | int8 trunk: i8 weights, bf16 scale per 128 of K, 128-wide Hadamard, int8 activations |
+| `qwen35moe_fp8` | `qwen35moe` | `fp8_e4m3` | `qwen35_fp8` with a routed feed-forward |
+| `qwen4exp_fp8` | `qwen4exp` | *(empty)* | Qwen4-Exp / Qwen3.8-Flash-Next: a bf16 checkpoint quantised at convert, or Qwen's FP8 release as it ships |
+| `glm5next` | `glm5next` | *(empty)* | GLM-5.3-Flash: four mHC residual streams, 34 KDA + 11 sparse MLA layers with a DSA indexer, 288 MXFP4 experts top-8, block-FP8 attention/dense at f32 scales, MTP drafter, vision tower |
+| `hyv3_bf16` | `hyv3` | *(empty)* | Tencent Hy-Memory-30B-A3B bf16; bf16/W4/W8A8/block-FP8/MXFP4 by recipe |
+| `hyv3_fp8` | `hyv3` | `fp8` | Hy-Memory-30B-A3B-FP8: imported per-tensor E4M3, one dense layer then sigmoid-routed MoE |
+| `hyv3_gguf` | `hyv3` | `gguf_ftype_15` | a llama.cpp Hy-Memory Q4_K_M GGUF, dequantised and recipe-converted |
+| `gemma4_bf16` | `gemma4` | `compressed-tensors` | Gemma 4 31B dense (and E2B/E4B at one rank): sliding-window + proportional-RoPE global attention, GELU, softcapped logits; int4/Q8 codes per 128; assistant (shared-KV), DFlash or DFlash2 drafter; vision |
+| `gemma4` | `gemma4` | *(empty)* | Gemma 4 text from Google's bf16 checkpoint (26B-A4B MoE, dense sizes) with z-lab's DFlash |
+| `gemma3` | `gemma3` | *(empty)* | Gemma 3 dense (TranslateGemma 27B): sliding-window + global attention, GemmaRMSNorm, GELU; bf16 or int8/MXFP4/asym-int4 linears; DFlash2 drafter |
+| `lfm2_bf16` | `lfm2` | *(empty)* | LiquidAI LFM2 / LFM2-VL, bf16, one rank |
+| `embgemma2` | `embeddinggemma2` | *(empty)* | EmbeddingGemma 2: bidirectional encoder with Gemma 4 vision tower and audio conformer, pooled |
+| `nomicbert` | `nomicbert` | *(empty)* | nomic-embed-text-v2-moe: bidirectional post-norm encoder, 8 experts top-2 on every second layer, mean-pooled |
+| `nemotron35_embed` | `mistral3` | *(empty)* | Nemotron 3.5 Embed 1B/8B and Rerank 8B: bf16 text/image retrieval with a Pixtral tower |
+| `qwenimage21` | `qwenimage21` | *(empty)* | Qwen-Image 2.1 / 2.1-Turbo image generator and editor (diffusers pipeline) |
+| `krea2` | `krea2` | *(empty)* | Krea 2 Raw/Turbo: Qwen3-VL-4B tapped layers, 12.9B single-stream DiT, Wan 2.1 VAE |
+| `sdxl` | `sdxl` | *(empty)* | Stable Diffusion XL and finetunes (Illustrious, NoobAI, Pony), single-file checkpoint |
+| `flux3` | `flux3` | *(empty)* | FLUX.3 [klein] and base: image generation and edits from up to ten pictures |
+| `flux3distilled` | `flux3distilled` | *(empty)* | the same plugin under the step-distilled release's own pipeline name |
+| `anima` | `anima` | *(empty)* | Anima Base/Aesthetic/Turbo from the release's three single files |
+| `qwen3asr` | `qwen3asr` | *(empty)* | Qwen3-ASR: Whisper-style audio encoder over a dense Qwen3 decoder, speech to text |
+| `minimaxmusic3` | `minimaxmusic3` | *(empty)* | MiniMax-Music3: Qwen3-8B frame model, depth decoder, flow-matching DiT, DAC vocoder |
+| `breeze` | `breeze` | *(empty)* | Breeze-TTS-2: T5Gemma 2 text encoder, Qwen3 backbone, 16-codebook depth decoder, Qwen3-TTS 12 Hz codec |
+| `fishomni` | `fishomni` | `bsgemm-moe` | Fish Audio drama-3: Qwen3.5-35B-A3B backbone, block-FP8 experts at f32 scales, 13-layer codebook tower, Fish S2 DAC/codec |
+| `minimaxh3` | `minimaxh3` | *(empty)* | MiniMax-H3 video with stereo sound: Qwen3-VL-32B text encoder, 33B DiT, tiled ViT decoder, BigVGAN audio |
+| `ltx2` | `ltx2` | *(empty)* | LTX-2.5 base/distilled: Gemma 4 LM + text connectors, duration head, audio-video DiT, causal video VAE, audio VAE |
+
+## Recipes of record, by family
+
+`$SRC` is a downloaded checkpoint; `$CALIB` is a calibration directory (see
+[[Radiance Conversion]]). Paths are shortened to the recipe name.
+
+| target | command |
+|---|---|
+| Qwen3.8-27B bf16 | `rad-convert $SRC -o q38-27b-bf16.rad --recipe data/recipes/q38-27b-bf16.recipe` |
+| Qwen3.8-27B block fp8 + DFlash2 | `rad-convert $SRC -o q38-27b-fp8.rad --recipe data/recipes/q38-27b-fp8block-df2.recipe --draft-model $DFLASH2 --set quantization_config.quant_method=fp8 --set quantization_config.fmt=e4m3 --set "quantization_config.weight_block_size=128 128" --set quantization_config.activation_scheme=dynamic` |
+| Qwen3.8-27B MXFP4 + DFlash2 | `rad-convert $SRC -o q38-27b-mxfp4.rad --recipe data/recipes/q38-27b-mxfp4-df2.recipe --draft-model $DFLASH2 --set quantization_config.quant_method=fp8 --set quantization_config.fmt=e4m3` |
+| Qwen3.8-27B w4a16 (an int4 release) | `rad-convert $SRC -o q38-27b-w4a16.rad --recipe data/recipes/q38-27b-w4a16.recipe` |
+| Qwen3.8-27B Quark AWQ-MXFP4 | `rad-convert $SRC -o q38-27b-quark.rad --recipe data/recipes/q38-27b-quark-mxfp4-df2.recipe --draft-model $DFLASH2` |
+| Qwen3.8-27B ParoQuant int4/int5/MXFP4 | `rad-convert $SRC -o q38-27b-paro.rad --recipe data/recipes/q38-27b-paro-mxfp4-df2.recipe --draft-model $DFLASH2` |
+| Qwen3.8-27B NVFP4 (compressed-tensors or ModelOpt) | `--recipe data/recipes/q38-27b-nvfp4-df2.recipe` or `q38-27b-modelopt-nvfp4.recipe`; every NVFP4 weight must take a rule (no kernel serves NVFP4) |
+| Qwen3.8-Flash-Next bf16 reference | `rad-convert $SRC -o flashnext-bf16.rad --max-ctx 4096` (no recipe; served with `--weights-disk-tier`) |
+| Qwen3.8-Flash-Next four-bit production | `CALIB=$CALIB rad-convert $SRC -o w4nl64-i8lm.rad --recipe data/recipes/qwen4exp-w4nl64-i8-hc8m.recipe` (GPTQ experts + int8 trunk) |
+| Qwen3.8-Flash-Next uncalibrated four-bit | `--recipe data/recipes/qwen4exp-w4.recipe` (rtn, no calibration) |
+| Flash-Next from an AutoRound int4 release | `--recipe data/recipes/qwen4exp-autoround-i8-hc8m.recipe` |
+| Gemma 4 31B w4a8 (no calibration) | `--recipe data/recipes/gemma4-31b-dense-bf16-w4a8.recipe --set quantization_config.quant_method=compressed-tensors --draft-model $ASSISTANT` |
+| Gemma 4 31B w4a8 + DFlash / DFlash2 | `gemma4-31b-dense-bf16-w4a8-df.recipe` / `-w4a8-df2.recipe`, same `--set` |
+| Gemma 4 31B Q8 grid | `gemma4-31b-dense-bf16-q8-df.recipe` |
+| Gemma 4 26B-A4B + DFlash | `gemma4-26b-a4b-df.recipe`; `-pad768` when the experts were padded to 768 (down may then be block fp8) |
+| Gemma 4 E4B (Nemotron Bolt) | `gemma4-e4b-bf16-df.recipe`, `-i8g-`, `-a4-`, `-w4a8-`, `-w8a8-` variants |
+| TranslateGemma 27B | `translategemma-27b-i8a8.recipe` (calibration container), then `CALIB=$CALIB translategemma-27b-mxfp4-gptq.recipe` (or `-df2`) |
+| GLM-5.3-Flash | `rad-convert $AMD_QUARK -o glm53.rad --recipe data/recipes/glm53-mxfp4.recipe` |
+| Hy-Memory-30B-A3B | `hy-memory-30b-a3b-bf16-fp8.recipe`, `-mxfp4`, `-w4`, `-w8`, `hy-memory-30b-a3b-fp8.recipe`, `hy-memory-30b-a3b-gguf-q4k-to-w4.recipe` |
+| MiniCPM5-2B | `minicpm5-2b-dspark.recipe` (or `-bf16-dspark`) with `--draft-model $DSPARK` |
+| Qwen3.6-35B-A3B | `q36-35b-a3b-fp8.recipe` |
+| Qwen-Image 2.1 | `qwen-image-2.1-fp8.recipe` (Turbo uses the same file) or `-bf16` |
+| Krea 2 | `krea-2-fp8.recipe`, `krea-2-bf16.recipe`, `krea-2-fp8-unscaled.recipe` for a plain-FP8 finetune file |
+| FLUX.3 [klein] | `flux-3-klein-fp8.recipe` |
+| SDXL / finetunes | `sdxl-bf16.recipe --tokenizer <openai/clip-vit-large-patch14>/tokenizer.json` |
+| Anima | `anima-fp8.recipe` (or `-bf16`) + `--add` ×2 + `--config` + `--tokenizer2` + `--arch anima --set anima.variant=aesthetic` |
+| LTX-2.5 | `ltx-2.5-fp8.recipe` (or `-bf16`); full DiT: symlink a folder whose `transformer/` is `transformer_full/` and `--set scheduler.use_dynamic_shifting=1 --set scheduler.shift_terminal=0.1` |
+| MiniMax-H3 / FastH3 | `minimax-h3-fp8.recipe`, `minimax-h3-w8a8.recipe`, `minimax-h3-curve-*` for curve-form checkpoints |
+| MiniMax-Music3 | `minimax-music3-bf16.recipe` |
+| Breeze-TTS-2 | `breeze-tts2-bf16.recipe` |
+| drama-3 | `drama3-fp8.recipe` |
+| nomic-embed-v2-moe | `nomic-embed-v2-moe-bf16.recipe` |
+| Unlimited-OCR | `unlimited-ocr-bf16.recipe` |
+| Qwen3-ASR | no container: serve the checkpoint directory as it ships |
+| EmbeddingGemma 2, Nemotron Embed/Rerank | serve the checkpoint, or convert bf16 with no recipe |
+
+A checkpoint served at its own precision needs no container at all:
+`radiance --model /srv/models/hf/Qwen/Qwen3.8-27B-FP8`. Convert when you want a tier only a container
+has (quantised encodings, a drafter, a second vocab) or a smaller file.
+
+## Family-specific traps
+
+- **`head_dim` is read, never derived.** On the Qwen hybrids it is 256 while `hidden_size /
+  num_attention_heads` is 213; a derived value silently breaks projection widths, KV page size and the
+  attention kernel's band.
+- **A fused projection shards by part, not by contiguous rows.** A tensor-parallel rank owns a slice
+  of *each* stacked projection in a fused `[gate|up]` or `[q|k|v|z]`; a contiguous slice would hand
+  rank 0 the whole gate, and the model would produce fluent wrong text. This is declared through
+  `RadWeightDecl::row_parts`, never inferred from the container.
+- **Gemma 4 26B-A4B:** the routed experts' `gate_up` may be block fp8, but `down` stays bf16 unless the
+  experts were padded to 768 — hence the `-pad768` recipe.
+- **GLM-5.3-Flash:** every weight is served as stored except the latent attention's small `kv_b`,
+  decoded to bf16. Its KV cache is bf16 only (the plugin refuses `fp8` at startup), and at 172 GiB it
+  needs `--tp 8`.
+- **TranslateGemma 27B:** the checkpoint's template takes structured turns naming two languages and
+  refuses the plain turn the server renders at startup. The container therefore serves a plain Gemma 3
+  template, and a client sends on `/v1/completions` the prompt the checkpoint's own template renders.
+- **ParoQuant containers made before 1.5.0 are refused at startup.** Convert again with the old
+  container as `--reuse`; every tensor that did not change is copied.
+- **SDXL carries no tokeniser.** Pass CLIP's `tokenizer.json` beside the single file. A Krea 2
+  finetune shipped as one DiT file in Krea's key names is rewritten into a directory whose
+  `transformer/` holds it (`scripts/krea2_native.py`), with the release's other parts copied in.
+- **Anima** ships as three single files: `--add` the text encoder and VAE, `--config` a Qwen3-0.6B-Base
+  directory (config + tokenizer), `--tokenizer2` ComfyUI's T5 tokenizer, `--arch anima`,
+  `--set anima.variant=…`.
+- **drama-3's codec is `codec.pth`**, a `torch.save` file read as data, never unpickled.
+- **Generators that load LoRA files declare their module aliases**; the checkpoint and a ComfyUI port
+  disagree in ways that load without error and generate garbage. The per-plugin alias table is in
+  `docs/ARCHITECTURES.md` ("Adapter key names of the generators").
+- **Licences travel with the checkpoint, not the engine.** Qwen-Image 2.1 is non-commercial;
+  MiniMax-H3's licence excludes the EU, UK, Republic of Korea and USA; drama-3's community licence
+  needs a commercial licence to serve speech to others, and cloning a real person needs consent;
+  Breeze-TTS-2 is research/non-commercial.
+
+## Related
+
+- [[Radiance Conversion]] — the tool, recipe language and quantisers used above.
+- [[Radiance Serve Options]] — the flags each family's compose file uses.
+- [[Radiance Architecture Plugins]] — writing a plugin for a family not in this table.
+- [[Quantization Production]] — the fleet's campaign and gates.

serverAppConfigs-docs/LLM/Radiance Operations.md

@@ -0,0 +1,240 @@
+---
+type: "howto"
+title: "Radiance Operations — running, reading, restarting"
+description: "Running a radiance server: the container contract, host prerequisites, readiness, ledgers and safe restarts."
+tags:
+  - llm
+  - radiance
+  - operations
+  - howto
+timestamp: 2026-10-30
+---
+
+# Radiance operations
+
+This is the engine-level runbook. The fleet's deployed shape, its unit and its deploy path live on
+[[frank-archon]]; the flags themselves on [[Radiance Serve Options]].
+
+## Install
+
+```sh
+git clone https://codeberg.org/StillDeadcode/radiance.git && cd radiance
+./build.sh                                   # configure, build for the cards present, run fast tests
+./build.sh --no-hip                          # host backend only
+cmake -S . -B build -G Ninja -DRAD_GPU_TARGETS='gfx1201'   # cards this machine lacks
+cmake --install build                        # after -DCMAKE_INSTALL_PREFIX=/opt/radiance at first configure
+```
+
+Needs Linux, git, a C++20 compiler, CMake 3.21+, Ninja; for device kernels ROCm 10.1 or 7.2
+(`hipcc`, `ROCM_PATH` if not `/opt/rocm`).
+
+- **Name the prefix at the first configure.** The default plugin path `<prefix>/share/radiance` is
+  compiled in from that configure; `cmake --install build --prefix DIR` alone gives a tree that looks
+  for its plugins elsewhere.
+- The prefix gets `bin/`, `include/radiance/`, `lib/cmake/radiance/`, `libexec/radiance/`,
+  `share/radiance/` (plugins and recipes) and `share/doc/radiance/`.
+- With ROCm 10.1 the build turns VOPD pairing off because its LLVM 24 pairs in heap-address order, so
+  two builds of one source can differ. The Docker image's compiler has the fix and keeps pairing
+  (faster fp8 prefill GEMMs); declare such a compiler with `-DRAD_HIP_VOPD_REPEATABLE=ON`. ROCm 7.2 is
+  unaffected.
+
+Run without Docker:
+
+```sh
+export RADIANCE_HOME=$PWD/build/radiance_home   # plugins; an install finds its own
+build/bin/radiance --model /srv/models/rad/minicpm5-2b-fp8.rad --port 8000 <flags>
+build/bin/radiance --model StillDeadcode/minicpm5-2b-fp8 --port 8000 <flags>
+```
+
+`--model` also takes a repo id: the server runs `scripts/hfget.sh` into `--download-dir`, serves the
+one `.rad` at the repo's top level, or the directory as a checkpoint when it holds none. A repo with
+several containers is refused, naming them. Every start checks the hub; a start that cannot reach it
+serves the copy a complete earlier fetch left, and a transfer that fails part-way stops startup.
+
+## Container
+
+The published image is `stilldeadcode/radiance`, kernels built for `gfx1201`. The host needs Docker
+and an `amdgpu` driver supporting ROCm 10.1 — **not** ROCm itself.
+
+```sh
+docker run -d --name radiance --network host --init --ulimit memlock=-1 \
+    --device /dev/kfd --device /dev/dri --security-opt seccomp=unconfined \
+    --group-add "$(getent group render | cut -d: -f3)" \
+    --group-add "$(getent group video | cut -d: -f3)" \
+    -v /srv/models/rad:/models:ro \
+    radiance --model /models/m.rad --tp 2 --host :: --port 8000
+```
+
+Each option is load-bearing:
+
+| option | why |
+|---|---|
+| `--device /dev/kfd --device /dev/dri`, `render`/`video` groups | the cards |
+| `--security-opt seccomp=unconfined` | Docker's default profile refuses io_uring (a disk-tier n-gram table's reads) and the NUMA calls the ROCm runtime makes for host memory |
+| `--init` | an init process as PID 1 forwards SIGTERM; the engine finishes its step and releases the cards |
+| `--ulimit memlock=-1` | a host-side pinned n-gram copy can be tens of GiB, far past Docker's default |
+| `--network host` | the server binds the host's addresses directly, IPv4 and IPv6, no NAT |
+
+The image is `FROM scratch` around the stripped install tree: busybox as the only shell, plus `curl`,
+`jq` and a CA bundle for `hfget.sh`; **no Python, no package manager**. Tools (`rad-convert`,
+`rad-info`, `rad-kbench`, `rad-tune`, `rad-schemas`, `rad-tokcheck`) are on `PATH`; recipes are in
+`/opt/radiance/share/radiance/recipes/`. Its entrypoint is `radiance`. It sets
+`GPU_MAX_ALLOC_PERCENT=100` and `GPU_MAX_HEAP_SIZE=100` because the HIP runtime's defaults cap one
+allocation and the heap below what the placement plan hands out.
+
+Build your own with `docker/build.sh` (`-t 'gfx1200;gfx1201'` for targets, `-j` compile jobs, `-E`
+to skip tests, `-s host` to build remotely). The context is `git archive` of the commit, so the image
+holds only committed content. The `rootfs` stage fails the build if any ELF cannot be resolved inside
+a chroot of the tree.
+
+Compose: one file per model plus `common.yaml`, copied to the serving host with an `.env` setting the
+image, models directory (mounted read-only at `/models`), API key, port, user and GPU groups. Every
+model listens on the same port, and most use both cards, so **run one at a time**. The models
+directory must be on a local filesystem that does direct I/O (ext4, XFS, btrfs); on btrfs with
+compression, a container copied into a `chattr +C` directory is stored uncompressed and direct reads
+work.
+
+A plugin of your own is mounted and put first on the search path; nothing in the image changes:
+
+```sh
+-v /srv/radiance-plugins:/plugins:ro -e RADIANCE_HOME=/plugins:/opt/radiance/share/radiance
+```
+
+## Host prerequisites (upstream's two-R9700 checklist)
+
+| item | setting |
+|---|---|
+| firmware | Above 4G Decoding on (Resizable BAR needs it); CSM off (pure UEFI) |
+| kernel command line | `iommu=pt amdgpu.ppfeaturemask=0xfff7ffff amdgpu.ras_enable=0` |
+| power/clock | OverDrive bit needed for `ppfeaturemask`; apply a power cap only while no process holds `/dev/kfd` |
+| memlock | `--ulimit memlock=-1` (container) / `LimitMEMLOCK=infinity` (systemd) when host memory is pinned |
+| two engines, one tmpfs table | `amdgpu.no_system_mem_limit=1`, or the second pin can be refused |
+| storage | a local fs with O_DIRECT for the container; a tmpfs sized for the table for `--ngram-placement ram=DIR` |
+
+Why each: `iommu=pt` keeps DMA identity so the cards can reach each other's memory and pinned host
+pages; `ppfeaturemask` enables the power controls; `ras_enable=0` turns VRAM ECC off (ECC reserves
+about 2 GiB a card). Verify peer access with a rate in `lspci`/the card tables — a missing cell means
+no peer access; check `iommu=pt` and the BAR size. The fleet's own boot-line law for Dankclaw
+(`iommu=pt`, tracked in the repo) is on [[dankclaw-iommu-translation-off]]; Dankserver's cards are a
+different box and its posture is on [[Dankserver System Specs]].
+
+## Readiness: the receipt, then `/health`
+
+Startup states every budget **before anything is allocated**, then loads. `/health` (and
+`/v1/health`, `/ping`) answers only once the load completes, so a bare health gate is honest. While it
+loads, expect 503 for minutes; do not restart over it. `--live` draws the live view (redirected, one
+`greppable key=value` line). `/diagnostics` returns the startup report as JSON — **same-machine-only
+without an API key** — with every option and its source and the hardware read again now.
+
+Read the receipt in this order:
+
+1. `elastic` — what the experts and KV cache share after non-expert weights.
+2. The split: `experts … / kv cache …`, and `unclaimed` if any.
+3. `kv: the pool backs N % of the worst case … M full-length sessions` — the referee for whether the
+   demand signal (`--max-model-len` × `--max-num-seqs`) is funded.
+4. `activation arena` and what each size lends; `prefill staging`; `hardware queues` and their VRAM.
+5. `host_pool` opened/used, `vram_weights`, the PLE or n-gram placement, the vision tower's placement.
+6. `device: the N cards reach each other's memory directly` or the failed pairs;
+   `expert prefetch on/off`; `chunk geometry`; `speculation:`; `chained launches`; `draft trees`.
+7. `card N settled at … free` — memory taken after the budget read comes out of headroom; a warning
+   names the `--gpu-headroom-mib` that clears it.
+
+A boot refusal prints the reason and exits nonzero; when it happens inside `podman run`, the
+container stops at once and the unit's restart policy re-runs the same refusal. A unit in
+`activating (auto-restart)` with a fast nonzero exit is therefore a **preflight/refusal or a boot
+refusal**, not an engine fault: read the journal line and the argv, not the GPU.
+
+## Lifecycle and restart
+
+- SIGTERM finishes the current step and releases the cards. In a container, `--init` forwards it;
+  `podman stop -t N`/`docker stop` is the graceful path.
+- A systemd unit should `exec podman wait` (or run in the foreground) so the script's lifetime is the
+  container's: a crash is a nonzero exit and the restart policy is real. Set `TimeoutStopSec` above
+  the stop grace, or systemd records `Failed with result timeout` on a healthy stop.
+- `Restart=always` rather than `on-failure` for a server whose front end can exit cleanly after a
+  worker fault.
+- Model load is slow (minutes cold). The deployer's LLM wait budget is on [[Deployment Flow]]; a
+  container restart loop during a legitimate load is worse than waiting.
+
+## Two engines on one set of cards
+
+- `ROCR_VISIBLE_DEVICES` picks each engine's cards explicitly.
+- `--weight-prefetch off`: a weight read beside a collective puts a queue switch into every decode
+  step when another server shares the cards.
+- `--ngram-placement ram=DIR`: two engines on separate card sets hold one table in host memory
+  (`DIR/ngram-<id>.tab` in a tmpfs, `MAP_SHARED`, pinned). The `<id>` hashes the container path, size,
+  mtime and the table's place, so a file never serves another model. The first engine fills it; a
+  second verifies samples and only pins it; the last to leave deletes it. The driver counts a shared
+  page against every process that pins it, so the pinned total is twice the table.
+- The engine prefers discrete cards (all of the first one's architecture), integrated only when
+  there is nothing else; on integrated graphics device memory is host memory and the budget leaves an
+  eighth of host RAM to the host.
+
+## Logs and diagnostics
+
+- `-v` / `-vv`; `RADIANCE_LOG` sets the level without a flag. `RADIANCE_LOG_STEPS=1` prints every
+  step's token count (how a chunk geometry is confirmed in practice).
+- The engine writes to stderr. Under a container, the journal holds only the container's own output;
+  a rootless `podman logs` covers only the current instance across recreations.
+- `/metrics` is Prometheus text: engine steps by kind and their seconds, decode/prefill rates,
+  prefill chunks held back, steps held by interleave, draft acceptance and depth, prompt-lookup
+  proposals held, KV bytes and the tokens they address, stream misses and bytes, prefix hits and
+  evictions, preemptions and queue depth. `/stats`,
+  `/graph`, `/sessions` back the dashboard. `scripts/statsdump.py` records `/stats` to JSONL with
+  time and gap, for a machine with no scraper (its key comes from `--api-key` or
+  `RADIANCE_API_KEY`).
+- `RADIANCE_PROFILE_EVERY`, `RADIANCE_DUMP_BUF`/`RADIANCE_DUMP_DIR`, `RADIANCE_OP_ORACLE`(+) and the
+  `RADIANCE_DEBUG_*` family are debug instruments; they are not deployment settings.
+
+## Verifying a deploy (fleet law)
+
+`deploy.sh` green is not proof of what is running. A leftover systemd drop-in or a stale container
+identity can serve a different model at a different shape while `/health` is 200. After deploying
+any recently-benched service:
+
+1. `systemctl show <svc> -p DropInPaths` — must be empty.
+2. `tr '\0' ' ' </proc/$(pidof radiance)/cmdline` (or the container's `Config.Cmd`) — the argv you
+   meant.
+3. The boot ledger — the shape you meant.
+4. A behavioural probe — `/v1/models` for the served id, one chat turn, one tool call, and a vision
+   probe with a solid colour and an exact expected word.
+
+`/v1/models` is an echo on some runtimes; on radiance it is authoritative for the alias, but the
+argv and the ledger are what prove the payload and shape. The three-pin revert discipline for a
+payload swap is on [[Model Swap and Rollback]]; the live deployment is [[frank-archon]].
+
+## Security posture
+
+- `--api-key` requires a bearer token or `X-Api-Key` on every route except `/health`, `/v1/health`,
+  `/ping` and the dashboard; the key is a command-line argument, so it is readable in
+  `/proc/<pid>/cmdline` by any local user of that host. On a single-tenant inference box the network
+  boundary is the real control; do not run one where untrusted local users share the host.
+- `/diagnostics` is same-machine-only without a key, because it prints the whole configuration.
+- `--security-opt seccomp=unconfined` is a genuine widening, required by io_uring and the ROCm
+  runtime's NUMA calls; it belongs on a dedicated model host, not a general-purpose one.
+- Mount model directories read-only; a container with `/models` writable adds nothing a serve needs.
+- A control vector can remove refusal behaviour. Loading one is a safety decision, not a
+  performance one; the flag is `--control-vector`, and `--control-vector disabled` is the default.
+
+## Failure signatures at startup
+
+| what you see | what it is |
+|---|---|
+| unknown/retired flag, or a flag refused by name | a boot refusal — the pin does not have it; change the trio, not the invocation |
+| "no plugin claims `<arch>` at `<quant>`; served at: …" | the (id, descriptor) pair has no plugin; fix the descriptor with `--set`/convert |
+| "reports ABI N, engine M" | rebuild the plugin against this engine |
+| budget refusal / `host_pool` short | the shape does not fit this card/host; lower demand or pin more host RAM |
+| `--ngram-placement ram` refuses | memlock short; raise it or use `auto`/`disk` |
+| `--kv-cache-dtype fp8_v4`/`k6_v4` refused, plugin named | no kernels for this geometry |
+| `--p2p on` refuses, pairs named | the driver has no peer access; use `auto`/`off` |
+| first launch of a never-run kernel faults | too little `--gpu-headroom-mib` for its code object |
+| unit loop, no container, no engine log | a preflight refusal (TTM/memlock/digest/model) re-run by `Restart=always` |
+
+## Related
+
+- [[Radiance Serve Options]] — every knob and its law.
+- [[Radiance HTTP API]] — readiness and the wire.
+- [[Radiance Tools]] — how to corroborate a receipt.
+- [[frank-archon]] — the fleet's radiance unit, guard chain and Eras.
+- [[Model Swap and Rollback]] — the swap discipline this page's verification serves.
+- [[Deployment Flow]] — how a committed change reaches the host.

serverAppConfigs-docs/LLM/Radiance Plugins.md

@@ -0,0 +1,371 @@
+---
+type: "howto"
+title: "Radiance Plugins — kernel libraries and quantisers"
+description: "Writing radiance kernel and quantiser plugins: the ABI, kernel rows, hooks, encodings, and the build path."
+tags:
+  - llm
+  - radiance
+  - plugins
+  - howto
+timestamp: 2026-10-30
+---
+
+# Radiance plugins — kernels and quantisers
+
+Three kinds of `.so` extend the engine, each exporting a few C symbols and linking **nothing of
+radiance's** (they compile against the installed `abi/` headers):
+
+| kind | value | contract |
+|---|---|---|
+| kernel library | `RAD_PLUGIN_KERNEL` (1) | ops for some hardware: a card family, the CPU, an oracle |
+| architecture | `RAD_PLUGIN_ARCH` (2) | one model family at one weight format — see [[Radiance Architecture Plugins]] |
+| quantiser | `RAD_PLUGIN_QUANT` (3) | weights → encoding planes, for `rad-convert` |
+
+In-tree plugins use the same interface a third party does: each builds with `rad_add_plugin` and is
+found and ranked by what it declares, never by its name. `ctest -L oot` rebuilds every bundled
+plugin out of tree against an install.
+
+## Identity and ABI
+
+```c
+#include "rad_abi.h"   /* required; pulls rad_types.h and rad_encoding.h */
+
+uint32_t rad_plugin_abi_version(void) { return RAD_ABI_VERSION; }
+
+const RadPluginInfo* rad_plugin_info(void) {
+    static const RadPluginInfo i = {
+        RAD_PLUGIN_KERNEL, "mykernels", "0.1", "a GEMM or two", "gfx1100",
+    };
+    return &i;
+}
+```
+
+- `RAD_ABI_VERSION` (31 at this commit) is bumped on any shape change in a header a plugin compiles
+  against. Return the macro; a mismatch is refused at load with both numbers printed.
+- `build_target` is printed, not trusted: the targets the library was built for (`"gfx1100"`,
+  `"gfx11-generic"`, several separated by spaces/commas), `"host"`, or `""` for portable. The engine
+  checks the library's actual code objects instead (§"A library for a new card").
+- Optional exports, resolved by exact name: `rad_plugin_open` (return `RAD_E_UNSUPPORTED` to decline
+  this machine and unload), `rad_plugin_close`, `rad_kernel_concurrent`, `rad_kernel_collective`,
+  `rad_plugin_reference`, `rad_kernel_reference`, `rad_kernel_chain`.
+
+## What a kernel library exports
+
+```c
+int                rad_kernel_schema_count(void);
+const RadOpSchema* rad_kernel_schema_at(int i);
+int                rad_kernel_count(void);
+const RadKernelInfo* rad_kernel_at(int i);
+```
+
+A row declares one op implementation and what it can serve:
+
+```c
+static const RadConstraint c_gemm[] = {
+    RAD_CLE("M", 64),              /* M <= 64            */
+    RAD_CDIV("K", 128),            /* K divisible by 128 */
+    RAD_CIN("dtype", "fp8a8 w4a8") /* dtype in that set  */
+};
+
+static const RadKernelInfo k[] = {{
+    .name = "mygemm_m64", .op = "gemm_nt_q", .family = "gemm",
+    .computes = "row-major A times column-major B, fp8 in, bf16 out",
+    .shape    = "M<=64, K%128==0",
+    .dtypes   = "fp8a8, w4a8",
+    .domain   = RAD_DOMAIN_DEVICE,
+    .priority = 50,
+    .constraints = c_gemm, .n_constraints = 3,
+    .launch = my_launch,
+}};
+```
+
+- Operators: `RAD_CEQ`, `RAD_CLE`, `RAD_CGE`, `RAD_CDIV`, `RAD_CIN`. The engine takes bucket
+  boundaries from `LE`/`GE`/`EQ` and resolves once per band; `DIV` and `IN` are not intervals.
+- **Declare only the band you serve.** The engine believes the claim; a kernel that says `M <= 2048`
+  but is correct to 64 is wrong at concurrency.
+- `device.*` keys are matched against what the engine measured about the ranks' cards, not against
+  op parameters: `RAD_CEQ(RAD_FACT_P2P, 1)` means the kernel touches another card's memory. libr4d
+  pairs every collective with a `_host` twin requiring `device.p2p == 0`, so exactly one matches any
+  machine.
+- **Priority breaks ties within one plugin**, highest first, then declaration order. **Between
+  plugins the hierarchy is absolute**: the first library in `--kernels` order with any matching row
+  supplies the op, however specialised a later library's row is. That makes a user's library an
+  override, not a competitor.
+- A derivative parameter (`RAD_DERIVED`) is supplied by the kernel; a constraint on a key the
+  geometry omits does not hold during matching, and the winner's `RAD_C_EQ` writes it into the
+  resolved geometry. `RAD_PROLE_SEQ_CHUNK` marks the parameter the scheduler must cut steps on;
+  `RAD_PROLE_CAPACITY` marks a count the kernel only bounds itself by.
+
+## Operands and extents
+
+`RadArgs.t` is positional in the schema's order; each `RadTensor` carries the extent the caller
+narrowed it to, usually smaller than the buffer. Size loops and grids from those extents, never from
+a declared maximum. Where an op takes `cu_seqlens`, derive the sequence count from its own length
+(`n = len - 1`) and each sequence's rows from `cu[n]` and `cu[n+1]`.
+
+An absent optional operand is a **null `RadTensor.data`**, not a shorter operand count.
+`rad_arg_in(a, i)` fetches one correctly; `rad_tensor_rows`, `rad_tensor_lastdim`,
+`rad_tensor_row_pitch`, `rad_args_geti_or`, `rad_args_getf_or`, `rad_args_gets_or` are the other
+everyday helpers in `rad_plugin.h`.
+
+## Optional hooks
+
+| hook | meaning |
+|---|---|
+| `init` / `fini` | per-resolved-instance state; `RadArgs.instance` hands it back. `init` runs once per (op, band, domain) after selection and before load |
+| `scratch` | bytes of scratch one launch needs; sized at declare from the band's upper bound |
+| `layout` / `relayout` | the kernel stores this weight in its own order, and how the loader produces it from the canonical planes |
+| `unrelayout` | the inverse, so a tool holding stored bytes can compare against a dense oracle |
+| `opd_shape` | size and fill every operand, so a tool can call the kernel cold |
+| `replaces` | the chain of conventional ops this kernel computes in one launch |
+| `tunables` / `tune_valid` | the instantiation axes, and which combinations are legal at a shape |
+
+`replaces` must be **exact** and is compared byte for byte against the chain — a kernel that is only
+close publishes nothing. `tunables` declares axes and legal values, not named variants; the core
+takes the cross product, and `deflt` must be launchable (the loader refuses the plugin otherwise).
+`rad-tune` measures survivors and writes the cache the selector reads; the chosen point arrives as
+ordinary parameters (`rad_args_geti("sk")`), with no variant index.
+
+`layout` answers: `RAD_OK` (own form, `relayout` will produce it), `RAD_E_UNSUPPORTED` (stored as
+is — the common case, and the only form that can be read in place from the file/mapped tier),
+`RAD_E_DTYPE` (this encoding is not mine — a declare-time error), `RAD_E_SHAPE`, or `RAD_LAYOUT_ANY`
+(reads the bytes without interpreting them; claims no storage). A weight only `LAYOUT_ANY` readers
+touch is stored as it is.
+
+## What a launch may do
+
+On a card the `RadStream` is **the engine's own AQL queue**, not a `hipStream_t`. The engine binary
+defines and serves these HIP entry points:
+
+- `hipLaunchKernel` (and `<<<>>>`, `hipLaunchKernelGGL`);
+- `hipMemcpyAsync`, `hipMemcpy2DAsync`, `hipMemsetAsync`;
+- `hipStreamSynchronize` (not from a launch — a launch must not synchronise);
+- `hipMemset` (runs on the engine's queue and waits);
+- `hipGetLastError` / `hipPeekAtLastError`, with HIP's semantics.
+
+**Nothing else in HIP takes an engine stream.** `hipModuleLaunchKernel`, `hipExtLaunchKernel`,
+cooperative launches, `hipLaunchHostFunc`, and libraries that launch through them (rocBLAS,
+hipBLASLt, MIOpen, hipRTC) would hand HIP a queue it does not know. Header-only device code
+(Composable Kernel templates, rocPRIM primitives) compiled into your `.hip` works. `hipEventRecord`
+and `hipStreamWaitEvent` on an engine stream are refused by name.
+
+The engine also reads the code object, which limits a kernel to: **no hostcall** (`printf`/`assert`
+are refused at the first launch), **no dynamic stack**, **≤ 768 bytes of kernel arguments**, **≤ 64
+KiB static+dynamic LDS per dispatch**, and an **uncompressed fat binary**. Device memory a kernel
+needs comes from the `scratch` hook, never `hipMalloc` — the engine's allocator knows what is left.
+
+`rad_plugin.h`'s conversions define the result: the reference contract is f32 accumulation, IEEE
+round-to-nearest-even, **no FMA and no reassociation**. A plugin whose bf16 narrowing truncates
+cannot be checked against the reference. `rad_load_f32`/`rad_store_f32`/`rad_load_bits`/
+`rad_store_bits` read and write any core dtype from a plane.
+
+## Colours of agreement
+
+- `rad_kernel_concurrent(i)` — kernel `i` touches only its operands and scratch, so independent
+  launches may overlap.
+- `rad_kernel_collective(i)` — it exchanges data with other ranks; the engine keeps bulk link
+  traffic away. `2` promises more: the launch ends the hold itself once the other ranks' bytes have
+  landed, through a `link_gate` word it reaches at system scope; returning `RAD_LINK_ENDED` says it
+  did.
+- `rad_kernel_chain(i)` — the kernel can wait in its own prologue instead of behind the packet
+  processor's barrier bit. Chained forms are recognised by `r4dchain` in the name and use a
+  `RAD_CHAIN_REC_BYTES` record at `RAD_CHAIN_KARG_OFF`.
+- `rad_plugin_reference` / `rad_kernel_reference` — an oracle library (or an oracle row beside a
+  device row). A reference library ranks below every unnamed library, is refused for serving unless
+  `--debug-accept-reference-kernels`, and is what `rad-kbench` checks against.
+
+## Defining a new op
+
+Only when `docs/OPS.md` has no name for it. The vocabulary document owns the rules: what a schema
+must declare, and that **the first plugin in hierarchy order to declare an op fixes its schema** — a
+later plugin declaring it differently is refused at load, both named. `ctest -R ops_doc` checks the
+document against the registries, so the rules live there only. An op in `docs/OPS.md` is describable
+by `libref`, and therefore checkable, even when your kernel cannot describe itself; an op you invent
+runs only on a library that implements it.
+
+A schema is positional data, not code:
+
+```c
+static const RadParamSpec params[] = {
+    {"M", RAD_P_INT, RAD_REQUIRED, RAD_PROLE_NONE},   /* the ranged extent */
+    {"n", RAD_P_INT, RAD_REQUIRED, RAD_PROLE_NONE},
+    {"eps", RAD_P_F64, RAD_REQUIRED, RAD_PROLE_NONE},
+    {"dtype", RAD_P_STR, RAD_REQUIRED, RAD_PROLE_NONE},
+};
+static const RadOperandSpec opds[] = {
+    {"x", RAD_OPD_IN, 0}, {"w", RAD_OPD_WEIGHT, 1}, {"y", RAD_OPD_OUT, 0},
+};
+static const RadOpSchema schemas[] = {
+    {"my_op", params, 4, opds, 3, "what it computes, for a human"},
+};
+```
+
+`rad_kernel_schema_count/at` return this table. `RAD_REQUIRED`/`RAD_OPTIONAL`/`RAD_DERIVED` and the
+operand roles (`IN`, `OUT`, `INOUT`, `WEIGHT`, `WTAB`) are the whole vocabulary; the kernel's
+constraints and launch do the rest.
+
+## Quantiser plugins
+
+`abi/rad_quant.h` is the whole contract:
+
+```c
+int                     rad_quant_count(void);
+const RadQuantizerInfo* rad_quant_at(int i);
+
+typedef struct RadQuantizerInfo {
+    const char*           name;       /* what a recipe names */
+    const char*           doc;
+    const RadQuantOption* options;    /* key, RAD_P_INT|F64|STR, default text, doc */
+    int                   n_options;
+    RadQuantEncodingFn    encoding;   /* the encoding this quantiser would write */
+    RadQuantRowBlockFn    row_block;  /* rows per quantize() call; 0 = whole weight */
+    RadQuantizeFn         quantize;   /* logical rows -> planes */
+    RadQuantDecodeFn      decode;     /* optional, only for a scheme of its own */
+} RadQuantizerInfo;
+```
+
+The flow for each weight a recipe rule picks: ask `encoding` (given name, shape, and — at declare —
+no movement group), let the architecture declare against that encoding, then hand the weight over in
+row blocks, so a 51 GB table streams. Other calibration data is the quantiser's own option
+(`calib=DIR`); the output is **canonical planes only**, so every kernel that accepts the encoding and
+the core's decoder can read it — that is how the converter measures error without knowing the method.
+
+A quantiser's own scheme must be decoded by its `decode` hook (the oracle, `rad-kbench` and the
+error report call it), and must return `RAD_E_UNSUPPORTED` for an encoding it did not write. Serving a
+new encoding also needs a kernel library whose `layout` accepts it and an architecture that declares
+the weight at it — a new format is a new kernel row plus a plugin, never a converter-only change.
+
+Two plugins naming the same quantiser are refused, both named.
+
+### The encoding model (`abi/rad_encoding.h`)
+
+An encoding is the code type, an optional codebook, the grouping of scales/zero points/minimums, and
+a transform. It is **data, not an enum**, so a plugin can invent one. The core can size, slice and
+place any encoding from the struct alone, and decodes the `plain` and `affine` schemes itself:
+
+```
+q  = code                                   codes TILED, block [1x1], or [1xV] with a grid
+q  = grid[code][k]                          grid TABLE [entries x V]
+q  = table[code]                            table TABLE [1 x entries]
+q  = -q if the sign bit is set              signs TILED u1
+q  = q - zero                               zero TILED
+v  = q * scale * scale.1 * ...              every scale level
+v  = v - min * min.1 * ...                  a minimum in the value's domain
+w  = v . T^-1                               the transform, per row over columns
+```
+
+So NVFP4 is fp4 codes, e4m3 scale per 16, f32 `scale.1` per tensor; Q4_K is u4 codes, u6 scale and
+u6 min per 32 each times an f16 per 256; IQ2_XXS is u8 codes per 8 into a 256×8 grid, a sign bit per
+value and two scale levels.
+
+- Planes are **TILED** (cover the weight, element `(i,j)` covering a `[br,bc]` block; `0` extent means
+  the whole extent) or **TABLE** (a codebook, grid, rotation parameter; every slice carries it).
+- Sub-byte elements are a bit stream along the row, lowest bits first; every row starts on a byte.
+- Transforms: `""`, `fwhtN` (Hadamard before quantisation; the `1/N` lives in the stored scale),
+  `perm` (GPTQ act-order, a TABLE `t.perm`), `givens` (ParoQuant rotations: `t.pairs`, `t.angle`,
+  optional `t.scale`, per-part `t.part`, composed `t.dense`).
+
+A kernel only *rearranges* planes (`relayout`); choosing numbers is the quantiser's job. That split
+is why a container never has to be reconverted when a kernel library changes its layout.
+
+## Build, install, search path
+
+```cmake
+cmake_minimum_required(VERSION 3.21)
+project(mykernels LANGUAGES CXX HIP)     # CXX alone for a host-only plugin
+find_package(radiance REQUIRED)
+rad_plugin_defaults()                    # C++20, optimising build unless asked otherwise
+rad_add_plugin(mykernels kernels SOURCES rows.cpp HIP_SOURCES kernels.hip)
+```
+
+The kind is `kernels`, `architectures` or `quantizers` — the directory the engine loads it from.
+Only `HIP_SOURCES` compile as HIP: for `RAD_GPU_TARGETS` or a `GPU_TARGETS` list, with `-DNDEBUG`
+outside Debug, uncompressed, against the HIP runtime the installed radiance found. `find_package`
+also sets `RADIANCE_ABI_VERSION` (to fail at configure time rather than at `dlopen`),
+`RADIANCE_HOME`, `RADIANCE_KERNELS_DIR`, `RADIANCE_ARCHITECTURES_DIR`,
+`RADIANCE_QUANTIZERS_DIR`. `cmake --install` puts the plugin under the home of the radiance you built
+against; `RAD_PLUGIN_INSTALL_HOME` overrides that.
+
+**`$RADIANCE_HOME` is a search path**: homes separated by `:`, the first winning; each may hold
+`kernels/`, `architectures/`, `quantizers/`, and an earlier home's same-name file shadows a later
+one. Your plugins go ahead of an installation without copying it:
+
+```sh
+RADIANCE_HOME=$HOME/radiance-plugins:/opt/radiance/share/radiance radiance --model m.rad
+# in the container:
+-v /my/plugins:/plugins -e RADIANCE_HOME=/plugins:/opt/radiance/share/radiance
+```
+
+The **kernel hierarchy** is `--kernels` (or `$RADIANCE_KERNELS`): `--kernels mykernels,libr4d` puts
+yours before libr4d. An entry matches a plugin's name or its file (`libr4d`, `r4d`, `libr4d.so`, a
+path). An entry matching nothing is warned about by name. With no list, every library found loads,
+reference last.
+
+## A library for a new card
+
+The engine's own device code is compiled for every AMD family's generic target
+(`gfx9-generic`, `gfx9-4-generic`, `gfx10-1-generic`, `gfx10-3-generic`, `gfx11-generic`,
+`gfx12-generic`) beside the build's targets, so one engine binary starts anywhere. What it computes
+there depends on the installed kernel library.
+
+- Configure with `-DRAD_GPU_TARGETS=gfx942` (or `gfx11-generic`). At load the engine asks the runtime
+  which targets the card accepts, in order (chip, then family: `gfx1036`, then `gfx10-3-generic`),
+  with the card's modes (`gfx942:sramecc+:xnack-`), and takes the best fit in the fat binary.
+- **A library with nothing for this card is left out, not failed**: its device rows leave selection,
+  its host rows stay, and the log says what it carries versus what the card runs. One installation
+  can carry a library per card. `rad_plugin_open` may decline explicitly with `RAD_E_UNSUPPORTED`.
+- The engine uses discrete cards if any exist (all of the first one's architecture), integrated only
+  when there is nothing else; `ROCR_VISIBLE_DEVICES` chooses explicitly. On integrated graphics,
+  device memory is host memory and the budget leaves an eighth of host RAM to the host.
+- **fp8 bytes are OCP E4M3** (`e4m3fn`, bias 7, max 448) wherever the engine reads them: container
+  weights, an fp8 KV cache, activations. A card whose fp8 is `e4m3fnuz` (gfx942) converts in its
+  `relayout` hook (the codes keep their bits except `0x80` and decode to half the value, so the scale
+  doubles) and keeps activations/KV in OCP bytes, or converts at its own boundaries.
+- **Tensor parallelism is ops you supply.** `all_reduce`, `all_gather` and the fused collectives are
+  rows like any other; the engine inserts none and assumes no link.
+
+## Checking your work
+
+```sh
+rad-kbench --kernels mylib,libr4d,libref --report mylib.md --bench
+```
+
+- It replays a recorded fixture of `libref`'s answers over real model geometries
+  (`$RADIANCE_HOME/kernels.rkb`, plus `kernels_moe.rkb`), running every library with a row for a
+  case. Correctness uses sampled values **and** the full-tensor norm **and** seeded projections;
+  completeness is the op-by-library matrix; speed is timed on the same bytes.
+- Red zones: every device allocation carries a poisoned 256-byte margin, verified after the launch.
+- Fusion: a row with `replaces` is run against its named chain, out of your library, compared with
+  `memcmp`.
+- A skip is named, not failed — usually a missing `opd_shape`. `--strict` also fails a stale fixture.
+- `rad-avx-check --ref libref.so --avx mylib.so` checks a **host** library without a card, op by op,
+  at every ISA level and with parallel regions forced onto several threads (for a library exporting
+  `avx_isa_force`/`avx_par_force`).
+- `rad-schemas mylib.so libref.so` prints every schema disagreement at once (it exits 1 on any), and
+  works on a plugin the engine cannot load.
+- `--debug-selection` prints which row took each band and which constraint refused each other
+  candidate; `/graph` on a running server shows the declared graph and which declarations the run
+  phase issued.
+- `rad-info --plugins` lists every plugin on the search path as the engine loads it: kind, version,
+  declarations, reference status, whether its device code runs on this card, and its file.
+
+## A worked example to copy
+
+`plugin/spechub/` is the smallest real kernel plugin in the tree: two ops (`mtp_couple_propose`,
+`mtp_couple_verify`), each with a device row and a host reference row. Read it in this order:
+
+- `exports.cpp` — the whole export block: info, ABI, schema count/at, kernel count/at, plus
+  `rad_kernel_reference` classifying device rows 0 and host rows 1 so the reference gate does not
+  refuse the device rows.
+- `spechub.cpp` — the rows, their constraints and their argument readers.
+- `spechub.hip` / `ref_spechub.cpp` — the device launch and the host oracle.
+- `CMakeLists.txt` — the `rad_add_plugin` call, the optional `enable_language(HIP)`, and
+  `-ffp-contract=off` because an FMA in the ordered FP64 mass sums would move a token off the host
+  reference.
+
+## Related
+
+- [[Radiance Architecture Plugins]] — the third plugin kind, one level up.
+- [[Radiance Conversion]] — how a quantiser is driven by a recipe.
+- [[Radiance Tools]] — `rad-kbench`, `rad-tune`, `rad-schemas`, `rad-avx-check` in full.
+- [[Radiance Serve Options]] — `--kernels`, `--radiance-home`, the tuning aids.

serverAppConfigs-docs/LLM/Radiance Serve Options.md

@@ -0,0 +1,240 @@
+---
+type: "reference"
+title: "Radiance Serve Options — every flag and the law behind it"
+description: "Every radiance serve flag grouped by task, with its default and the law that decides when to move it."
+tags:
+  - llm
+  - radiance
+  - reference
+  - serving
+timestamp: 2026-10-30
+---
+
+# Radiance serve options
+
+`radiance --help` prints this table. Every numeric option takes a whole number (not `12k`, not
+`auto`, unless the option names those); a malformed value exits 2. Upstream `main` at 2026-10-30;
+the fleet is pinned older — see the pin note on [[Radiance Engine]].
+
+The one thing to read before any table: **the engine derives the VRAM split; no flag sets either
+pool.** The startup receipt prints every term before anything is allocated.
+
+## Model, plugins, container
+
+| flag | default | meaning |
+|---|---|---|
+| `--model PATH` | required | a `.rad` container, a checkpoint served as it stands, or a Hugging Face repo id (`owner/name`) fetched before startup |
+| `--download-dir DIR` | `$XDG_CACHE_HOME/radiance/hf`, else `~/.cache/radiance/hf` | where a repo id is fetched |
+| `--radiance-home DIRS` | compiled prefix | plugin homes, `:`-separated, first wins |
+| `--kernels LIST` | every library found | kernel libraries, `,`/`:` separated, first wins; a library not named is not loaded |
+| `--served-model-name NAME` | container's name | the id `/v1/models` lists and responses carry; a request may name any model |
+| `--override-chat-template PATH` | container's | a Jinja template served instead; an unreadable or empty file stops startup |
+| `--generation-config PATH` | none | sampler defaults from a `generation_config.json` (else `<model>.generation.json`, then the container's `generation.*` metadata) |
+| `--control-vector off\|on\|disabled` | `disabled` | load the GGUF control vector found for the model and start requests off/on; `disabled` declares no projection and refuses a request that asks |
+| `--api-key KEY` | none | require `Authorization: Bearer KEY` or `X-Api-Key: KEY`, except `/health`, `/v1/health`, `/ping`, the dashboard |
+| `--mm-max-patches N` | `auto` (16384 with a vision tower) | patches one encoder pass carries: the largest image, 16×16 px a patch (14×14 on GLM-5.3). `0` serves text only and keeps the tower off the card |
+
+## Batch shape, context, scheduler
+
+| flag | default | meaning |
+|---|---|---|
+| `--max-num-batched-tokens N` | 2048 | the largest token count one step may carry; on MoE the knob that sets prefill chunk size, and the activation arena's size |
+| `--max-num-seqs N` | 256 | the most sequences one step may carry; the cache's **demand**, never its funding |
+| `--max-model-len N` | 0 = model's training context | context bound per request. Longer is a 400, never a truncation. Reserves nothing per request; it is the worst case the pool is sized against |
+| `--busy-prefill-chunk auto\|off\|N` | `auto` | a prefill chunk's share of a step that has other work: `auto` leaves decode rows and waiting requests room, `off` takes the whole step, `N` caps it |
+| `--prefill-interleave N` | 0 (off) | decode-only steps after each chunk taken beside decoders, at most 64; buys decoder cadence with the prompt's TTFT |
+| `--prefill-span N` | 1 | checkpoint intervals a prefill **alone in its step** may carry, from its second interval; needs `--max-num-batched-tokens` of N intervals. Every other step is planned at the shared budget; checkpoints inside a span are not written |
+| `--preempt-wait-ms N` | 5000 | how long a preempted prefill waits for the whole-recompute gate before reclaiming blocks and preempting lower-priority work; `0` falls back next step |
+| `--rope-scaling SPEC` | the checkpoint's | `yarn:FACTOR[:ORIG]` writes static YaRN over the container's own rotary keys with no reconvert; `none` takes a checkpoint's YaRN off |
+| `--max-queued-requests N` | 8 × `--max-num-seqs` | admitted, unfinished requests; past it the answer is 429 |
+| `--retry-after S` | 1 | the `Retry-After` a 429 carries |
+
+## Memory and placement
+
+| flag | default | meaning |
+|---|---|---|
+| `--placement MODE` | `auto` | `all_vram`, `layer_offload`, `expert_tiered`, `auto` |
+| `--gpu-headroom-mib N` | 96 | VRAM left unclaimed on every card; too little makes the first launch of a never-run kernel fail inside HIP |
+| `--host-pool-mib N` | 0 | pinned host tier for weights, **per rank**; also the KV lever under expert tiering (below) |
+| `--host-headroom-mib N` | 4096 | host RAM left free when startup pins memory; a snapshot store's floor takes the shortfall back only down to 1 GiB |
+| `--kv-cache-dtype T` | `bf16` | `bf16`, `fp8`, and on Qwen3.8-27B/-Flash-Next `fp8_v4` (V int4) or `k6_v4` (K int6 + V int4). A real quantisation; GLM-5.3 is bf16-only |
+| `--state-cache-dtype T` | `f32` | `f32` or `f16`: the recurrent delta-net state cache; f16 halves state pool, checkpoints and traffic and is lossy — `qwen35` plugins only |
+| `--attn-prefill-fp8 MODE` | `off` | `off`, `qk`, `pv`, `both`: prefill attention products in E4M3 over an `fp8`/`fp8_v4`/`k6_v4` cache; lossy, Qwen3.8-27B only where kernels exist |
+| `--kv-resident N` | 0 (off) | stream the sparse attention group's KV from pinned host RAM, keeping an N-token ring on the card. Raised to 32768 and to one decode step's selection; ignored with a warning where unsupported |
+| `--embedding-placement ram\|vram` | `ram` | token embedding pinned in host memory and read in place, or on the card. A tied head or a host that cannot pin keeps it on the card; an explicit `ram` refuses instead |
+| `--vision-tower-placement vram\|ram` | `vram` | encoder towers on the card, or pinned in host memory and read once per image, giving their VRAM to the KV cache |
+| `--ngram-placement auto\|vram\|ram\|disk` | `auto` | where an n-gram embedding table lives: all of it on every card, a pinned host copy the card reads, or rows from the container through a row cache. `auto` takes the first that fits. `ram=DIR` makes the pinned copy a tmpfs file shared by two engines |
+| `--weights-disk-tier` | off | read weights that fit neither VRAM nor the host pool from the container with O_DIRECT; every step is paced by the drive |
+| `--row-prefetch on\|off` | on | ask for a host gather's pages before its kernel; output-neutral |
+| `--mapped-rewarm on\|off` | on | at startup, re-ask the page cache for a mapped table's pages an earlier run left behind |
+| `--deterministic` | off | pin placement to the loader's plan so the resident set cannot move under the heat engine; it does **not** make output bit-identical across batch positions |
+
+## KV cache, host tier, checkpoints
+
+| flag | default | meaning |
+|---|---|---|
+| `--prefix-cache-mib N` | 0 (off) | copy finished conversations' KV to host memory so a return restores instead of re-prefilling; LRU when full; off while a KV group streams |
+| `--prefix-cache-ttl S` | 3600 | seconds an unused cached conversation is kept; `0` keeps it until memory is wanted |
+| `--no-prefix-cache` | cache on | disable prefix caching |
+| `--checkpoint-interval N` | 2048 | linear-attention state checkpoint interval, in tokens; also clamps a hybrid model's prefill chunk |
+| `--checkpoint-slots N` | `auto` | snapshots held **in total**, in pinned host memory; `auto` = per-policy need for each sequence within an eighth of host RAM; `0` keeps none |
+| `--checkpoint-policy NAME` | `geometric-backoff` | also `tip-only`, `keep-all`; retention only — writes happen at every interval unless a span skips them |
+| `--prefix-resume exact\|nearest` | `exact` | `exact` resumes only where a cold prefill starts a chunk, so a hit answers bit-identically; `nearest` takes the deepest snapshot, up to a chunk less re-prefilled |
+
+## Speculation
+
+| flag | default | meaning |
+|---|---|---|
+| `--num-speculative-tokens N` | `auto` (the drafter's own point; 0 with no head) | speculative window; 0 disables. The ceiling under `adaptive` |
+| `--spec-ngram K` | 0 (off) | prompt lookup alone, up to K drafts a step from the sequence's own n-grams; leaves a container's draft head out; sets the window to K |
+| `--spec-ngram-min N` / `--spec-ngram-max N` | 2 / 4 (max 15) | pattern lengths prompt lookup tries, longest first |
+| `--draft-window fixed\|adaptive` | `fixed` | `adaptive` follows each sequence's acceptance and the measured step cost |
+| `--draft-short-from N` | `auto` | a block drafter that declares a shorter block runs it from N decoding sequences; `0` never; `auto` = never for a chain, from 2 for a tree |
+| `--draft-coupling off\|on` | off | sample each proposal from the drafter's own shortlist and couple the target's draw to it; same output law, different tokens for a fixed seed, pays at high temperature |
+| `--draft-model PATH` | — | **not implemented** at serve: merge the drafter at convert time (`rad-convert --draft-model`) |
+
+## Tensor parallelism and collectives
+
+| flag | default | meaning |
+|---|---|---|
+| `--tp N` | 1 | tensor-parallel ranks; declare divides dimensions by world size and inserts no collective |
+| `--tp-wire exact\|wht6` | `exact` | `wht6` is lossy: rotated 6-bit at 2 ranks, tiered int8 over a two-shot at 3–4, unavailable at 8 |
+| `--tp-wire-min-kb N` | 128 | smallest all-reduce (KiB) the lossy wire compresses; keeps single-sequence decode exact |
+| `--tp-attn-split heads\|rows` | `heads` | a prefill step's attention split: `rows` gives each rank all heads of its rows plus an all-gather; faster where attention dominates, not bit-identical, plugin-gated |
+| `--weight-prefetch on\|off` | on | read a GEMM's weight beside the collective before it; `off` when another server shares the cards |
+| `--p2p auto\|on\|off` | `auto` | `auto` asks the driver; `on` refuses to start without peer access; `off` exchanges through pinned host memory |
+
+## LoRA adapters
+
+| flag | default | meaning |
+|---|---|---|
+| `--lora NAME=PATH` | none | serve an adapter: a PEFT directory as model NAME (text), or a `.safetensors` a request names in `loras` (image/video). Repeatable |
+| `--lora-dir DIR` | none | generators: any `NAME.safetensors` under DIR may be requested |
+| `--max-loras N` | count of `--lora`, 2 with `--lora-dir` alone | adapter slots resident at once |
+| `--max-lora-rank R` | 16 text / 64 generator | slot rank; a larger text adapter is refused, a larger generator combination truncated |
+| `--lora-target-modules LIST` | the adapters' own | HF module leaf names adapters may target; needed with `--max-loras` and no `--lora` |
+
+## Request defaults and bounds
+
+A request that names a field wins. These set defaults for requests that do not.
+
+| flag | default | meaning |
+|---|---|---|
+| `--temp F`, `--top-k N`, `--top-p F`, `--min-p F` | from the model | sampler defaults; `top_k` 0 is the 1024 most likely candidates |
+| `--typical-p F` | 1 | typical sampling |
+| `--presence-penalty F`, `--frequency-penalty F` | 0 | range [-2, 2] |
+| `--repetition-penalty F`, `--repeat-last-n N` | 1, 64 | `-1` looks back over the whole context |
+| `--dry-multiplier F`, `--dry-base F`, `--dry-allowed-length N`, `--dry-penalty-last-n N` | 0, 1.75, 2, -1 | DRY; multiplier 0 is off |
+| `--dry-sequence-breakers JSON` | `["\n", ":", "\"", "*"]` | at most 16 strings |
+| `--xtc-probability F`, `--xtc-threshold F` | 0, 0.1 | range [0, 1] |
+| `--default-max-tokens N` | `auto` | `max_tokens` for a request that sends none; `auto` is what the context leaves |
+| `--max-tokens-cap N` | 0 (none) | a larger request is **cut to it**, not refused; reply ends `finish_reason: "length"` |
+| `--max-n N` | max(8, `--max-num-seqs`) | the largest fan-out `n` |
+| `--max-stop-strings N`, `--max-stop-bytes N` | 64, 4096 | stop-string limits |
+| `--chat-template-kwargs JSON` | none | variables for the template on every chat request; a JSON object or `@FILE`; repeatable |
+| `--reasoning-effort S` | template's own | default `reasoning_effort`; passed to the template verbatim, so the legal set is the template's |
+| `--reasoning-effort-map MAP` | none | `FROM=TO,...`, checked against the template at startup; `none`/`minimal` turn thinking off |
+| `--thinking-token-budget N` | none | default budget: at N reasoning tokens the block is closed and the answer follows |
+| `--reasoning-format auto\|none` | `auto` | `auto` returns reasoning as `reasoning_content`; `none` leaves it inline |
+
+## Server and HTTP
+
+| flag | default | meaning |
+|---|---|---|
+| `--host ADDR` | 0.0.0.0 | listen address |
+| `--port N` | 8000 | listen port |
+| `--http-threads N` | one per hardware thread, at least 4 | workers kept while idle |
+| `--read-timeout S` | 30 | a client may go silent this long while sending |
+| `--write-timeout S` | 600 | one write to a client may block this long |
+| `--keep-alive-timeout S` | 5 | idle connection hold |
+| `--max-body-mib N` | 512 | largest body; images and video ride base64 inside it |
+| `--no-cors` | CORS on | send no CORS headers |
+
+## Media preprocessing
+
+Each replaces what the container's preprocessor config states, and does nothing for a model that
+takes no media.
+
+| flag | default | meaning |
+|---|---|---|
+| `--image-min-pixels N`, `--image-max-pixels N` | container's | the band an image resizes into; one encoder pass still bounds the top |
+| `--video-min-pixels N`, `--video-max-pixels N` | container's | same, for a video's sampled frames together |
+| `--video-fps F` | container's | frames sampled per second |
+| `--video-min-frames N`, `--video-max-frames N` | container's | frame-count bounds |
+| `--video-max-frame-tokens N` | container's | the most prompt tokens one frame becomes; `0` no cap; a token-budget processor takes 70/140/280/560/1120 |
+| `--image-tokens N` | container's | a token-budget processor's image budget (Gemma 4's fixed set; GLM-5.3's ceiling ≤ `--mm-max-patches`/4) |
+| `--max-source-pixels N` | 67108864 | largest input the decoder accepts before resize |
+
+## Quality measurement (instead of serving)
+
+| flag | meaning |
+|---|---|
+| `--kld-record DIR` | run `--kld-corpus` through prefill and write this model's per-position log-probabilities to DIR: the reference |
+| `--kld-ref DIR` | score this model against the reference: mean/median/percentile KL and top-1 agreement |
+| `--kld-corpus FILE` | JSONL, one `{"prompt", "score_from"?, "source"?}` a line; `--kld-record` only |
+| `--kld-out FILE` | the report as JSON; `FILE.rows` holds every position's numbers |
+| `--kld-lora NAME`, `--kld-lora-sources PREFIX` | score through a served adapter, all documents or a `source` prefix |
+
+The mode turns speculation and the prefix cache off and refuses `--num-speculative-tokens > 0`.
+See [[Radiance Conversion]] for the corpus and the scoring discipline.
+
+## Diagnostics
+
+| flag | meaning |
+|---|---|
+| `--debug-graph` | print the declared graph, every op's resolved kernel and why, then exit |
+| `--debug-selection` | the selection table only |
+| `--debug-placement` | the placement plan only |
+| `--debug-tape-dump` | the first recorded pass of each kind (`TAPEDUMP` on stderr, every rank) |
+| `--debug-draft-oracle FILE` | verify-path test: prompt lookup proposes from FILE with 30 % corrupted drafts; the text must not move |
+| `--debug-accept-reference-kernels` | serve even with `libref` rows — not a production configuration |
+| `--profile-ops` | per-op device timing |
+| `--expert-stats FILE` | routing counts per layer and step kind, JSON every 30 s and at shutdown |
+| `--expert-profile FILE` | rank the expert plane by a stats file's `decode` counts instead of the container profile |
+| `--expert-pin-top F` | pin the hottest fraction F of routed experts so heat never demotes them |
+| `--draft-expert-placement host\|auto` | pin a draft head's experts in the host pool before any strategy runs |
+| `--live` | the terminal live view; redirected, one `key=value` line |
+| `-v`, `-vv` | verbose, very verbose |
+
+## The laws behind the knobs
+
+**The split is derived, and the receipt is the referee.** Non-expert weights come off the top of
+the card; the remainder (`elastic`) is shared by the expert slab and the KV cache. `--max-model-len`
+× `--max-num-seqs` is the cache's **demand**; the funding is `elastic − expert_floor`. Under a
+lending cache (`auto`/`expert_tiered`, no `--deterministic`, VMM present) the cache takes what it
+can address and lends unused blocks back to experts. **Every MiB added to `--host-pool-mib` lowers
+the expert floor one-for-one and the cache takes it** — until the cache already holds its whole
+ceiling, where the extra host room shows up as unclaimed. Read the boot line
+`kv: the pool backs N % of the worst case …`, never arithmetic.
+
+**The activation arena comes off the budget before the split**, so both sides pay for
+`--max-num-batched-tokens`. While steps are small most of it is lent back to the expert slab; a
+larger step reclaims it synchronously. That is why a bigger chunk can cost decode residency. On a
+hybrid model the chunk is clamped to `--checkpoint-interval` before the token budget, and startup
+prints which bound applies.
+
+**The KV dtype law.** `bf16` and `fp8` are engine-wide; `fp8_v4`/`k6_v4` are narrower layouts whose
+kernels exist only for particular attention geometries, and the server refuses them elsewhere by
+plugin name. A drafter's cache stays fp8. Recurrent state and the QSA indexer keep their own widths.
+
+**The slot law.** A slot costs recurrent state, block tables at full context, snapshot-store RAM
+and all-reduce rows — never speed by itself. Raising `--max-num-seqs` raises demand; whether it is
+funded is the resolver's answer.
+
+**The spec law.** Measure speculation by step time, not tok/s. Prompt lookup pays where the reply
+copies its context and costs a verify row elsewhere. `adaptive` moves the depth only after eight
+steps and a 2 % gain, because a new window is a new pass shape.
+
+**Batch shape is part of the configuration.** A different chunk, span or spec depth changes the
+summation order, so greedy text can differ at a tie. Reproducibility holds per configuration, never
+across one.
+
+**Unknown request fields are ignored; known-but-unimplemented fields are guarded** (accepted at
+their off value, refused otherwise) — the list is on [[Radiance HTTP API]].
+
+## Related
+
+- [[Radiance HTTP API]] — what the flags expose on the wire.
+- [[Radiance Operations]] — the container contract and readiness.
+- [[Radiance Plugins]] — how a flag reaches a plugin (`RadBuildCtx`).
+- [[frank-archon]] — the measured deployment of this resolver on two R9700s.

serverAppConfigs-docs/LLM/Radiance Tools.md

@@ -0,0 +1,322 @@
+---
+type: "reference"
+title: "Radiance Tools — inspect, check, tune, tokenise"
+description: "The radiance inspection, measurement and tokeniser tools, with the flags and verdicts each one reports."
+tags:
+  - llm
+  - radiance
+  - tools
+  - reference
+timestamp: 2026-10-30
+---
+
+# Radiance tools
+
+Six binaries and `rad-export` are installed beside `radiance`; `rad-avx-check`, `r4d_selftest` and
+`rad-chatparse` live in the build tree. Every tool that loads plugins honours `RADIANCE_HOME`,
+`RADIANCE_KERNELS` and `--kernels` as the engine does. A numeric option takes a whole number: `12k`,
+`auto` (where the option does not name it) or an empty value exits 2.
+
+`rad-convert`, `rad-kbench` and `rad-tune` run the engine's startup with a different last step: the
+same plugins, in the same hierarchy, through the same declare phase, so they resolve kernels exactly
+as a serve would.
+
+## `rad-info` — what a container holds
+
+```
+rad-info [options] <model.rad>
+rad-info --plugins [--home DIRS]
+
+  -v, --verbose      list every tensor, not just the per-unit summary
+      --meta         metadata table, then stop
+      --vocab        vocab summary and normaliser chain, then stop
+      --profile N    top N expert-profile entries (default 20, 0 = all)
+      --recipe       the recipe the container was converted with, then stop
+      --plugins      every plugin the search path offers, as the engine loads it
+      --home DIRS    search path for --plugins (default $RADIANCE_HOME)
+```
+
+It opens the container read-only and loads no plugins (except for `--plugins`): a container names no
+kernel library. Output is aligned; byte counts are binary units and each count says what it counts.
+
+```sh
+rad-info m.rad                    # format, architecture, model, source quant, recipe, sections
+rad-info --recipe m.rad           # the recipe, expanded, one rule a line
+rad-info -v m.rad | head -40      # every tensor, encoding, shape and what made it
+rad-info --plugins                # kind, version, declarations, reference, device fit, file
+```
+
+The section table (strings, metadata, directory, planes, encodings, vocab, profile, data) is the
+container's layout; the metadata table, the weight directory grouped by layer and movement unit, the
+vocab, and the expert profile follow.
+
+## `rad-kbench` — correctness, completeness, speed
+
+```
+rad-kbench [options]                       replay the recorded fixture (no model)
+rad-kbench -m <model.rad> [options]        check against a real container
+rad-kbench -m <model.rad> --record FILE    record a new fixture
+rad-kbench -m <model.rad> --skeleton FILE  write a metadata-only copy to record from
+```
+
+**Replay** needs no model, weights or card. Geometries, seeds and `libref`'s reduced answers are
+checked in as `data/kernels.rkb` (dense model, block drafter, gqa 6) and `data/kernels_moe.rkb`
+(routed MoE, MTP head, gqa 8). `--fixture` is repeatable; `ctest` runs three replays — the other
+libraries against each recording, and `libref` against both at tolerance zero (its regression test,
+since a live comparison moves both sides together).
+
+Each library is asked for its row as the engine would ask it with that library alone: same
+constraints, derived parameters and weight-encoding filter, so a row the engine would pass over does
+not run here either (`--all-rows` runs every row whose constraints hold).
+
+**Against a container** (`-m`), weight operands are read from it: each weight's canonical planes,
+which the kernel gets through its own `layout`/`relayout` hooks and the oracle gets decoded, and
+every expert of a weight table. This is the only test of the artifact and of the layout hooks.
+
+**Recording** takes the reference's answer once at every point of every declared op, whichever
+library resolved it. Weights are drawn from a seed, not read from the container, so any machine can
+reproduce the reference and no card is needed. A skeleton (tables intact, data a hole) records the
+same fixture in a few megabytes.
+
+```sh
+rad-kbench -m model.rad --skeleton model.skel.rad
+rad-kbench -m model.skel.rad --max-cases 1 --record data/kernels.rkb
+rad-kbench -m moe.rad        --max-cases 1 --record data/kernels_moe.rkb
+```
+
+### Verdicts
+
+- An output is compared on sampled positions, the full-tensor norm, and 32 seeded random-sign
+  projections (which estimate rel_l2 to about 25 %). The largest of the three is the error.
+- Tolerances are per op and activation dtype (`dtype`, else `q_dtype`, else `kv_dtype`). Copies,
+  gathers, scatters and cache stores are exact. Every op in the fixtures has a row of its own.
+- Ids are compared exactly; where the contract leaves a tie open (`sample_argmax`, `router_topk`,
+  `sample_topk`), an id indexing a value equal to the reference id's is accepted.
+- A case whose reference and output are both identically zero checked nothing: **vacuous**, not
+  passed.
+- A failure, a red-zone overrun, a fusion that differs from its chain, or a host kernel that wrote
+  into an input exits 1. `--strict` also fails a stale fixture.
+- `--coverage DIR` keeps one file per fixture/library of the rows a replay reaches
+  (`data/kbench_rows/`); a listed row not reached fails, unless a memory skip kept its cases out.
+- Every pass ends with a line saying where its time went (draw, hashing, layout, upload, launch,
+  read-back, comparison, cache, timing).
+
+### Timing
+
+`--bench` times every case that passes, on the bytes it was checked on.
+
+- Device cases are timed **after** the checks, operands still on the device, after one sustained
+  warm-up. A time is device timestamps around a batch of back-to-back launches divided by the launch
+  count; the batch is queued behind a gate the host opens once submission is done, so neither the
+  host rate nor a sync is in the interval. Two untimed launches precede every batch; batches repeat
+  until the median's 95 % interval is within 1 % or `--bench-reps` is reached.
+- **Cold by rotation**: weights and inputs of 1/32 of the footprint or more are read from copies
+  spanning the footprint (`--bench-flush`, default 256 MiB, at least twice an R9700's caches), so
+  each launch reads them from memory as streamed weights do. Smaller inputs stay warm, as the
+  previous op leaves them.
+- In-place operands are restored from a pristine copy before every launch; the restores are timed
+  alone and subtracted.
+- Timing draws expert offsets/routing skewed (Zipf 0.5) and block tables as permutations; the
+  checked launch keeps the recorded draw.
+- The header prints the calibration: launch floor and streaming-read bandwidth warm and cold. If a
+  smaller cold probe runs >1.5× the largest's rate the rotation was cache-served and device times
+  are withheld. Throttler bits that came on during a batch repeat it, up to three times.
+- Host cases are timed in their own column, after an untimed launch, weights rotated over at least
+  256 MiB (twice LLC), for at most two seconds a case.
+
+### Flags
+
+| flag | meaning |
+|---|---|
+| `--fixture FILE` | replay recording (default `$RADIANCE_HOME/kernels.rkb`); repeatable |
+| `--record FILE` | with `-m`: write a fixture |
+| `--skeleton FILE` | with `-m`: write a sparse container copy |
+| `--rerecord FILE` | replay: write the fixtures' cases again with the reference's answers |
+| `--record-rows FILE` | replay: record a row-driven fixture from the cases |
+| `-m, --model FILE` | the `.rad` to read weight operands from |
+| `--home DIR`, `--kernels A:B` | plugin home; kernel libraries (default every one found) |
+| `--op NAME` | only this op (repeatable prefix match) |
+| `--ref NAME` | the oracle plugin's name (default: the library declaring itself a reference) |
+| `--no-ref` | replay without the reference among the tested libraries |
+| `--strict` | replay: a stale fixture fails |
+| `--all-rows` | run every row whose constraints hold |
+| `--coverage DIR`, `--coverage-write` | the row-coverage ratchet |
+| `--no-cache`, `--cache-dir DIR` | run the reference instead of reusing cached answers; cache location |
+| `--seed N` | base seed; every case derives its own |
+| `--tol X` | override every tolerance (bisecting only) |
+| `--max-ctx N` | deepest context considered (declare and the attention sweep) |
+| `--max-tok N` | `max_tok` declare runs at (default 512, small on purpose) |
+| `--mm-max-patches N` | encoder pass declare runs at (default 2048 when a tower is present; 0 leaves it out) |
+| `--kv-cache-dtype T` | `bf16` or `fp8`, as on the engine |
+| `--num-speculative-tokens N` | draft depth to declare at (default: the container's) |
+| `--max-cases N` | runs per distinct case (default 4; 0 = every one) |
+| `--max-bytes MIB` | skip a case needing more than this, by name (default: half available memory, 512–8192) |
+| `--no-fusion` | skip the fusion oracle |
+| `--bench`, `--bench-iters N`, `--bench-reps N`, `--bench-warm-ms N`, `--bench-flush MIB` | timing |
+| `--timing-only` | time every device kernel, check nothing (refuses `--record`) |
+| `--report FILE`, `--analysis FILE` | markdown report; splice in `tools/kbench-analysis.sh` output |
+| `-v` | every case, not only failures |
+
+Notes: `--max-bytes` matters on a loaded machine — one case holds each operand as drawn, as the
+kernel's copy, and on the device; without the cap a small machine is OOM-killed with no indication of
+the op. `--bench` times cold on purpose: an R9700's 52 MB bf16 weight read warm comes from cache at
+1.45 TB/s against 565 GB/s from memory, so warm times rank shapes but do not size a roofline. A
+kernel that cannot lay out an encoding is a **skip**, not a failure (`RAD_E_DTYPE`), and the report
+names the kernel, operand and encoding.
+
+## `rad-tune` — measure the instantiation space
+
+```
+rad-tune -m <model.rad> [options]
+
+  -m, --model FILE   the .rad whose declared graph supplies the shapes
+      --home DIR     $RADIANCE_HOME; cache at <home>/tune/<machine>.tune
+      --kernels A:B  kernel libraries to load, first wins
+      --op NAME      only ops whose name starts with NAME
+      --machine KEY  override the machine key (writing a cache for another machine)
+      --reps N       timing reps, median reported (default 5)
+      --iters N      launches per rep (default 32)
+      --max-tok N    max_tok declare runs at (default 8192)
+      --tp N         tensor-parallel width to declare at (default 1)
+      --num-speculative-tokens N|auto   draft depth to declare at
+      --dry-run      list what would be measured
+  -v, --verbose      every candidate, not just the winner
+```
+
+Kernels declare tunable axes, not named variants; `rad-tune` walks the cross product at this model's
+shapes and writes the winner per (op, geometry) to the cache the selector reads at startup.
+
+- **Declare at the shapes you serve**: `--tp N` divides every sharded N; a declare at world size 1
+  measures a program no `--tp 2` serve runs. `--num-speculative-tokens` puts the drafter's kernels in
+  the measured graph.
+- At `--tp > 1` every rank declares at once, each on its own thread and device: a collective's
+  `init` is a rendezvous, so declaring rank 0 alone hangs.
+- A collective is measured as the exchange: per-rank operands (own seeds, so identical halves cannot
+  pass), own scratch and stream, own instance; every rank fires at once and the clock stops when the
+  last stream drains. Non-collective rows run on rank 0 alone.
+- `RADIANCE_TUNE` pins a kernel's axes to compare a tuned cache against the defaults — a measurement
+  aid, not a deployment setting.
+
+## `rad-schemas` — the op vocabulary diff
+
+```
+rad-schemas libref.so                 print one plugin's vocabulary
+rad-schemas libr4d.so libref.so       diff them, in hierarchy order
+```
+
+The first plugin in hierarchy order to declare an op fixes its schema; a later plugin declaring the
+same op differently is refused at load, both named (arguments are positional, so disagreeing schemas
+would make one call mean different things). `rad-schemas` prints every disagreement at once, so a new
+plugin can be conformed before it is loaded beside another. It exits 1 on any disagreement. It links
+no core — it `dlopen`s plugins and reads static tables, so it works on a plugin the engine cannot
+load.
+
+## `rad-avx-check` — host libraries without a card
+
+```
+rad-avx-check --ref libref.so --avx LIB.so [options]
+
+  --ref PATH     the oracle plugin      (default libref.so)
+  --avx PATH     the plugin under test  (default libavx.so)
+  --op NAME      only this op
+  --level N      only this ISA level (0 scalar, 1 avx, 2 avx2, 3 avx512)
+  --threads N    rerun every case with each parallel region on N threads (default 4; 0 skips)
+  --tol X        override every tolerance
+  -v             every case
+```
+
+Checks a host library op by op against `libref`, on the CPU, with neither engine nor card: two
+`dlopen`s, a geometry table (`libavx/avx_cases.cpp`), libref's operand descriptions and hand-built
+cases. For every case: schemas as the loader compares them, outputs within `rad-kbench`'s tolerances,
+guard bytes, inputs unchanged, and an oracle output that is not just the draw. `libavx` is also
+checked at every ISA level the CPU executes and with parallel regions forced onto several threads.
+A new host library is checked with the same command; it is treated as a host library (not `libavx`),
+so libref ops it has no row for are listed rather than failed, and the ISA/thread sweeps run only if
+it exports `avx_isa_force`/`avx_par_force`. A row returning `RAD_E_UNSUPPORTED` for a geometry libref
+serves fails: the selector would resolve to it and the step would fail at issue.
+
+## `r4d_selftest` — libr4d's own check and timer
+
+```
+r4d_selftest <libr4d.so> [--case <row>]
+r4d_selftest <libr4d.so> --perf N K M [iters]      (and the other --perf* modes in its usage)
+```
+
+With no mode it runs the catalogue and static checks, then each row's smoke and numeric cases in a
+child process of its own, with fusion claims compared byte for byte (`ctest` labels `slow`, `gpu`).
+`--case` runs one row in-process as the reproducer. `--perf` covers the fp8 decode GEMMs and head;
+`--perfmxd` the dense OCP-MXFP4 rows with the weight bandwidth each reaches; `--perfmx` the MXFP4
+expert GEMM; `--cold MIB` times each launch alone behind a cache flush. `--perfar8` prints the
+eight-rank all-reduce shard weights and times `lockstep`/`ring`/`split` forms at every piece count;
+`R4D_SELFTEST_ONE_CARD=1` runs them on one card for their bits only.
+
+## `rad-tokcheck` — tokeniser against its references
+
+```
+rad-tokcheck [options] <tokenizer.json> <corpus-file>...
+
+  --hf-dir DIR    compare against HuggingFace ids in DIR/<corpus basename>.ids
+                  (little-endian int32, encode(text, add_special_tokens=False))
+  --no-ref        skip the reference encoder (slow)
+  --user          encode with parse_special off
+  --turns N       turns in the multi-turn measurement (default 40; 0 skips)
+  --repeat N      timing repeats, best taken (default 3)
+  --context N     ids shown either side of a difference (default 6)
+```
+
+Per corpus file: ids match `tools/tokref`'s reference encoder (llama.cpp's splitters and
+`std::regex`); ids match HuggingFace `tokenizers` read from files made elsewhere; and speed — MB/s
+for both encoders, plus the cost of re-tokenising a growing conversation with and without the
+segment cache. Any difference prints the ids and surrounding text and exits nonzero, so it gates a
+change.
+
+## `rad-chatparse` — where a tool call goes wrong
+
+```
+rad-chatparse --request REQ.json --rad MODEL.rad --reply R.txt
+rad-chatparse --render-raw T.jinja --context CTX.json
+```
+
+Renders one OpenAI chat request as the server would (container template or
+`--override-chat-template`, `--chat-template-kwargs`, the plugin's declared reply format, or the
+template's own with `--no-plugin`), then reads each reply file as the model's output: parsed whole,
+parsed through the SSE stream as a client reassembles it, and run through the request's tool-call
+grammar. It prints one JSON object; exit 1 when the two parses disagree or the grammar refuses a
+token. No weights are loaded.
+
+## The KL mode
+
+`radiance --kld-record/--kld-ref/--kld-corpus/--kld-out/--kld-lora` is a mode of the engine, not a
+separate tool: same placement and kernels, prefill instead of listening, then exit. The flags and the
+corpus format are on [[Radiance Conversion]].
+
+## Shell harnesses (`scripts/`)
+
+They drive a running engine: bring a model up, measure it, assert on what came back. Each header
+records why it measures the way it does.
+
+| script | what it does |
+|---|---|
+| `mcserve.sh`, `fnserve.sh` | bring MiniCPM5-2B+DSpark / Flash-Next up and leave it running (env knobs: `P TP MLEN SEQS M KVD HR MNBT EXTRA …`) |
+| `fnstop.sh` | stop whichever is running and wait for exit |
+| `convdspark.sh`, `convfn.sh` | rebuild the two reference containers (`convfn.sh` needs `CALIB=`) |
+| `hfget.sh` | resumable Hugging Face fetch with a content manifest |
+| `fnpf.sh`, `fnconc.sh`, `cstep2.sh`, `ident.sh`, `cardclock.py` | attach to a live server: prefill rate; prefill-vs-decode contention; steady-state decode ms/step per concurrency; run-to-run hashes; which clock limit binds and which rank the TP step waits for |
+| `fndrop.sh` | disconnect a client mid-request and check every other answer against a reference |
+| `tierident.sh`, `tiercross.sh` | whether a session that left VRAM returns byte for byte (attention and recurrent state), including two sessions in flight |
+| `mctools.sh`, `mcstream.sh`, `mcagent.sh`, `mcdecl.sh` | tool-call parsing shapes; streamed tool calls on raw SSE; speculation on an agentic workload; whether the drafter proposes |
+| `dprofd.sh`, `fntrace.sh`, `mctrace.sh`, `oracle.sh` | per-op device timing; rocprofv3 traces; every op against libref on the operands the model issued |
+| `kv-parity.sh`, `kv-stage-parity-64k.sh` | KV streaming: covering ring and staged prefill are byte-identical to streaming off |
+
+**Not all of them leave a server alone.** `fnserve.sh`, `mcserve.sh`, `fnstop.sh`, `fntrace.sh`,
+`dprofd.sh`, `mctrace.sh`, `mcagent.sh` and `mcdecl.sh` kill any running engine (`pkill` on the
+binary), whatever port it is on; `oracle.sh` stops whatever listens on its own port. `cstep2.sh`,
+`ident.sh`, `tierident.sh`, `tiercross.sh`, `mctools.sh`, `mcstream.sh`, `hfget.sh`, `convfn.sh`,
+`convdspark.sh`, `fnpf.sh`, `fnconc.sh`, `fndrop.sh` and `cardclock.py` leave it running
+(`fndrop.sh` deliberately disconnects clients). Never point the stopping ones at a production port.
+
+## Related
+
+- [[Radiance Conversion]] — `rad-convert`, `rad-export`, the KL mode.
+- [[Radiance Plugins]] — what `rad-kbench`/`rad-tune`/`rad-schemas` check.
+- [[Radiance Operations]] — the startup receipt these tools corroborate.

serverAppConfigs-docs/LLM/frank-archon.md

@@ -562,6 +562,52 @@ vllm-radiance era (this table opens on 09-26; the era page claims 10-10).
   fork bomb inside the container. Kill with `podman rm -f` (the cgroup owns every generation)
   never pattern-kill an interpreter name on this host; use `TORCH_LOGS` for dynamo questions.
 
+### The quantization pipeline
+
+- **A calibration tap is only a tap when the rig that opened it is running.** The runner writes
+  `.rig-boot` into the tap dir (container, image, pid) and retires it when its boot exits non-zero;
+  `calib-push.py` refuses without that marker, refuses when it names another container, and refuses
+  when the named container is not running; `qwen38-calib-window.sh` stops on a failed phase instead
+  of falling through its health wait. All three exist because on 2026-10-29 the unguarded chain's
+  rig boot died of an OOM, the health loop expired without failing, and the pusher fed four rounds
+  of 10M tokens into **restored production** — 86 minutes of live prefill that recorded nothing.
+- **A calibration step has an OPTIMUM, and it is not the widest step that boots.** The tap's cost is
+  expert re-reads per document (a 7,168-token document cut into four 2,048-token steps re-streams
+  each layer's routed expert union four times), which argues for the widest step — but the measured
+  curve turns over: at the 32,768 calibration window one 2-way prefill of the same document ran
+  **106 tok/s at an 8,192-token step and 76 tok/s at 16,384**, with the computed activation arena
+  growing 3.39 → 4.50 GiB, so the wider step pays more compute than it saves in reads. The
+  200,000-window `QSA score plane` refusal that made the wide step look impossible is an artifact of
+  serving the production window: the same 16,384-token step needs a 512 MiB plane at 32,768.
+  `Q38_CALIB_MNBT` is the only knob — the KL modes keep production's step, because their numbers are
+  only comparable at one shape.
+- **A container that exists, a port that answers, and a `/health` 200 are three different claims —
+  bind identity before believing any of them.** Measured 2026-10-30: `podman inspect` named the rig
+  while the port still belonged to the *dying* production process (`health ok waited=0s`, window dead
+  at second zero), and the identity check written to catch that compared two different number spaces:
+  `podman top <ctr> pid` prints **container-internal** pids (`1`, `7`), never host pids, and a rootless
+  cgroup carries the container ID rather than the service name — so both "matches" can only ever fail.
+  The binding is `podman top <ctr> hpid` against the listener from
+  `ss -H -ltnp 'sport = :7770'`, and `fuser /dev/kfd` likewise resolved through podman.
+  `scripts/qwen38-preflight.sh` owns those facts now (cwd, the digest a tag actually resolves to,
+  free space on the tap's **own** filesystem, stale grams, every GPU holder and the listener by host
+  pid), prints `PASS`/`FAIL` per fact, and `qwen38-calib-window.sh` calls it before a card is touched.
+  Corollary for every ad-hoc driver: a detached launcher that loses its cwd fails with
+  `No such file or directory` in a log nobody reads — assert cwd, and never bypass the orchestrator.
+- **The n-gram table's pin is what starves a rig boot, and the tier is a knob.** On the BF16 payload
+  the table is 95.37 GiB, `--ngram-placement ram` pins it, and the calibration boot died at
+  `startup stopped: out of memory` with 230.22 GiB pinned of 246.07 GiB and 8.06 GiB available —
+  while the reference recording and every score boot succeeded at the same pool and slot count.
+  Calibration only needs the routed weights prefilled, so it reads the table from the container's
+  file tier (`--ngram-placement disk`); kldref and score keep production's placement, because their
+  numbers are only comparable at an identical shape.
+- **Acceptance is three metrics and every slice, not a mean.** `scripts/kld-gate-qwen38.sh` grades
+  mean KL, the p99.9 tail, top-1 agreement and every `by_source` slice (code, prose, tools):
+  a mean can hide a few positions taking the whole hit, and an overall win can be bought by
+  regressing one distribution. The FP8-native anchor is an interval, not a requirement — without
+  the 172.8 GiB fetch the gate runs in two-report mode (candidate vs the served payload) and says
+  so on its first line.
+
 ## Contract (the wire)
 
 - **Live surface:** `/health` (== `/ping`), `/v1/models`, `/v1/chat/completions`
@@ -638,11 +684,19 @@ vllm-radiance era (this table opens on 09-26; the era page claims 10-10).
   rebuild), `scripts/download-frank-archon-swift15.sh` (the BF16 source a w4dn8 re-convert
   needs), `scripts/hfget.sh` / `scripts/hf-fetch.sh`; template
   `scripts/qwen38-froggeric-v22.5.jinja`.
-- Convert campaign (dormant, kept for a future quant): `scripts/convert-swift15-w4dn8.sh`
+- The campaign (qwen38-w5nl64, live): `scripts/run-qwen38-bf16.sh` (kldref / calib / score — the
-  `scripts/recipes/qwen4exp-swift15-w4dn8-i8-hc8m.recipe`, `scripts/run-swift15-bf16.sh`
+  ONE measuring shape), `scripts/qwen38-calib-window.sh` (the gated calibration window),
-  `scripts/kld-gate-swift15.sh`, `scripts/kld-then-calib.sh`, `scripts/calib-push.py`
+  `scripts/convert-qwen38-w5nl64.sh` with
-  `scripts/make-calib-corpus.py`, `scripts/derive-protected-experts.py`
+  `scripts/recipes/qwen4exp-qwen38-w5nl64-bf16trunk.recipe` (the arm driver and its template),
-  `scripts/apply-protected-cast.sh`, `scripts/verify-swift15-ple.py`.
+  `scripts/kld-gate-qwen38.sh` (acceptance), `scripts/calib-push.py`,
+  `scripts/make-calib-corpus.py`, `scripts/rechunk-kld-corpus.py`,
+  `scripts/qwen38_common.py` (the tokenizer, JSONL and refusal plumbing those three
+  corpus tools share — one `n_tokens`, so every tool counts the same way),
+  `scripts/derive-protected-experts.py`, `scripts/apply-protected-cast.sh`.
+- Retired swift15-era campaign scripts, superseded by the above and kept only as reference:
+  `scripts/convert-swift15-w4dn8.sh`, `scripts/recipes/qwen4exp-swift15-w4dn8-i8-hc8m.recipe`,
+  `scripts/run-swift15-bf16.sh`, `scripts/kld-gate-swift15.sh`, `scripts/kld-then-calib.sh`,
+  `scripts/verify-swift15-ple.py`.
 - Probes: `scripts/probe-frank-archon-kv.py` (reads slots/window/dtype from the deployed env so
   it cannot drift; leg D = decode), `scripts/frank-archon-expert-activity.py`
   `scripts/autoresearch-frank-archon.sh` + `scripts/autoresearch/frank-archon-measure.py`.
@@ -652,5 +706,6 @@ vllm-radiance era (this table opens on 09-26; the era page claims 10-10).
 - Retired-era leftovers in-tree: `scripts/patches/glm53-*.patch`
   `scripts/patches/gpu_worker-kv-cap.patch` (fork archaeology; none applies byte-exact today).
 
-Related: [[frank-dragoon]] (the abliterated family), [[frank-zealot]],
+Related: [[frank-dragoon]] (the abliterated family), [[frank-zealot]], and the engine reference —
----
+[[Radiance Engine]], [[Radiance Serve Options]], [[Radiance Conversion]], [[Radiance Plugins]],
+[[Radiance Operations]].

serverAppConfigs-docs/LLM/frank-dragoon.md

@@ -1,6 +1,6 @@
 ---
 type: reference
-title: "frank-dragoon on Dankclaw — halogen-flash-server 0.16.2, native v2 abliterated Qwen3.8-Flash-Next (canonical)"
+title: "frank-dragoon on Dankclaw — halogen-flash-server 0.18.20, native v2 abliterated Qwen3.8-Flash-Next (canonical)"
 description: "The iGPU endpoint: live shape, every era with its rollback path, the laws it paid for, and the files that own it."
 tags:
   - llm
@@ -10,10 +10,10 @@ tags:
   - gfx1151
   - halogen
   - endpoint
-timestamp: 2026-10-21
+timestamp: 2026-10-30
 ---
 
-# frank-dragoon — halogen-flash-server 0.16.2, native v2 abliterated (Dankclaw iGPU)
+# frank-dragoon — halogen-flash-server 0.18.20, native v2 abliterated (Dankclaw iGPU)
 
 Strix Halo (gfx1151) endpoint `http://192.168.0.143:7770`, served id `frank-dragoon`: a podman container
 under a root `User=dank` unit. Live shape since 2026-10-17, pool raised to 401,408 on 2026-10-18
@@ -33,8 +33,8 @@ flags and nothing else. Payload `/home/dank/models/halogen-v2-abliterated/`, ver
 | `qwen38-flash-next-vision.hgn` | 897,916,416 | `d62e0ae5…` | peonist `@e053f488`, hardlinked — vision ON |
 | `tokenizer/` (6 flat files) | ~22 MB | `tokenizer.json` LFS | peonist `@e053f488`, hardlinked |
 
-Image `ghcr.io/peonist-ai/halogen-flash-server@sha256:0c61bf84ac22308a53f5d1ca6b86806702d7039e5ebc51cae4c66621b92fe04a`
+Image `ghcr.io/peonist-ai/halogen-flash-server@sha256:e35133b206da3094cc4d70c78800b26235bd9124b9aeb0e029d4a7c153b9cff0`
-(0.16.2, closed source), container `frank-dragoon-halogen`, published `7770:8731`; the engine's
+(0.18.20, closed source), container `frank-dragoon-halogen`, published `7770:8731`; the engine's
 unauthenticated token port 8730 stays on container loopback. Engine stdout →
 `/home/dank/logs/frank_dragoon.log`, **not** the journal.
 
@@ -74,6 +74,7 @@ rollback plan — recoverable only until the trash is purged.
 
 | Window | Payload | Runtime | Shape | Replaced because | Rollback |
 |---|---|---|---|---|---|
+| 2026-10-30 → | same v2 abliterated `.hgn` trio (payload untouched) | flash-server **0.18.20** `e35133b2…` | unchanged (2 slots, 262,144 ceiling, pool 401,408, MTP always-on) | vendor moved two minor versions past the 0.16.2 quickstart pin; the payload lock **held** — the same boot served chat, a `tool_calls` turn, vision ("Red", 132 prompt tokens) and `/v1/embeddings`, with the NPU lane's 19-file set verified against the new catalog | env line back to `0c61bf84…` + redeploy |
 | 2026-10-06 → 10-17 | Swift1.5 abliterated v2 `.hgn` r2 66,687,678,464 B `9c880b8b…` + same trio | flash-server 0.16.2, same digest | 2 slots, ctx 262,144, pool 524,288 (14.6 GiB), MTP always-on | — | `/home/dank/models/halogen-flashnext-v2/` + `scripts/download-frank-dragoon-flashnext.sh` |
 | 2026-10-05 → 10-06 | peonist `halogen-qwen3.8-27b` `qwen3.8-27b-p1w4d-d2.hgn` 35,865,565,184 B `274c3fc7…` | halogen 0.1.4 `sha256:dc0a39a0…` | 1 slot × 200,700, DFlash2 drafter, `HALOGEN_CACHE_MB=16384`, text-only, censored base | flash-server v2 restored vision, abliteration and MTP | `/home/dank/models/halogen-qwen3.8-27b/` + `scripts/download-frank-dragoon-halogen.sh`; 0.1.4 digest still in podman storage |
 | 2026-10-03 → 10-05 | kingjones777 Uncensored `Q6_0_ROCMFPX_AGENT.gguf` 24,856,607,008 B `e966deb3…` + `mtp-Q4_0` `051a1764…` + BF16 mmproj | `kingjones30/ROCmFPX` @ `36e9acd` (`scripts/build-rocmfpx.sh`) | 1 × 262,144 f16 KV, `draft-mtp` n-max 4, vision on | — | `/home/dank/models/kingjones-Qwen3.8-27B-Uncensored-ROCmFP4-STRIX-MTP/` + `/home/dank/GitPackages/kingjones-ROCmFPX/build` + `download-frank-dragoon-kingjones-uncensored.sh` |
@@ -190,9 +191,11 @@ created it — earlier topologies in .
 
 ## Contract (wire)
 
-- Live: `/v1/models`, `/health`, `/metrics`, POST `/v1/chat/completions`; `404`: `/ping`, `/props`
+- Live: `/v1/models`, `/health`, `/metrics`, `/props` (model_alias, total_slots, chat_template),
-  `/completion`, `/tokenize`. **No `Server-Timing` header and no `/completion` timing object** — measure
+  POST `/v1/chat/completions`, POST `/v1/embeddings` when the NPU lane is armed; `404`: `/ping`,
-  wall-clock plus `usage.prompt_tokens`.
+  `/completion`; `/tokenize` answers **405 to a GET** (the route exists and wants POST). **No
+  `Server-Timing` header and no `/completion` timing object** — measure wall-clock plus
+  `usage.prompt_tokens`.
 - Thinking lands in `reasoning_content` and **counts against `max_tokens`**: a budget that dies
   mid-thought returns `finish_reason: "length"` with empty `content`, so clients read both fields; size
   vision probes at ≥512 tokens or the model looks blind. Budget for the base reasoning long: 1,476
@@ -232,11 +235,15 @@ created it — earlier topologies in .
 XDNA2 executes only vendor-precompiled AIE partition programs — that is the wall the TTS attempt died at
 ([[Frank TTS]]) — and flash-server ships those programs. `HALOGEN_NPU_MODELS=<names>` starts a second engine **in the
 same container behind the same port**, and a request picks one by its `model`; leave the variable out and the server
-is exactly what runs today. The pinned 0.16.2 image already carries the five-program catalog
+is exactly what runs today. The pinned 0.18.20 image carries an 18-model catalog
-(`/opt/halogen/npu/models.txt`: `decider-0.8b`, `qwen3-embedding-0.6b`, `qwen3-reranker-0.6b`, `qwen3guard-gen-0.6b`,
+(`/opt/halogen/npu/models.txt`: `decider-0.8b`, `decider-4b`, `qwen3-embedding-0.6b`, `qwen3-embedding-4b`,
-`qwen3.5-2b`) and `docs/NPU.md` at tag `v0.16.2` documents it, so **`/v1/embeddings` is one env var and three mounts
+`qwen3-reranker-0.6b`, `qwen3-reranker-4b`, `qwen3guard-gen-0.6b`, `qwen3guard-stream-0.6b`,
-away — no runtime bump needed.** Wider catalog: `qwen3-embedding-4b` and `decider-4b` from 0.17.4, `embeddinggemma-2`
+`embeddinggemma-2`, `wemm-embedding-4b`, `qwen3.5-2b`, `flux2-klein-4b`, `qwen3-asr-0.6b`, `qwen3-asr-1.7b`,
-(text ≤ 8,192 tokens, pictures and audio in one vector space) and `wemm-embedding-4b` from 0.18.3.
+`d3-lite`, `d3-nano`, `qwen3-tts-0.6b-customvoice`, `qwen3-tts-1.7b-customvoice`), so **`/v1/embeddings` is one
+env var and three mounts away — no runtime bump needed.** The catalog is **not** version-stable: 0.18.20
+re-cut `qwen3-embedding-0.6b`'s device programs (14 `u*.elf` + an 8,495,512 B `consts.hnpb`, where 0.16.2
+had 10 and 5,086,496 B) and the entrypoint verifies size+sha256 per file before it starts, so **an engine
+bump re-fetches the staged set** — `scripts/download-frank-dragoon-npu.sh` carries the catalog block to do it.
 
 Embedding contract on that lane: `POST /v1/embeddings`, `model: "qwen3-embedding-0.6b"`, `input` a list of **texts**
 (not token ids), 1024 unit-length floats back, the Qwen instruct prefix goes inside the string
@@ -282,7 +289,7 @@ lane is **armed and serving** since 2026-10-19 (`HALOGEN_NPU_MODELS=qwen3-embedd
    unset, so the container opens no outbound connection; `scripts/download-frank-dragoon-npu.sh` pins every file by
    size and sha256 against the catalog and the payload mount carries the tree in.
 
-Armed, the boot ledger reads (`/home/dank/logs/frank_dragoon.log`, 2026-10-19): the GPU fabric clock is held at its top level (2000 MHz) **before** the NPU starts; `/dev/accel/accel0` is `crw-rw-rw- nobody:nogroup` under driver `amdxdna` with the distro XRT and its NPU plugin; the program arrives as `15 files checked against this image's record`; then `npu_api: qwen3-embedding-0.6b (embeddings) on the NPU engine at 127.0.0.1:8740 (buckets u0…u9; inputs up to 4096 tokens)` and `up to 64 passes queued … up to 7200 s`. Callers never see `:8740` — the lane answers on the same `:8731` as the Flash model and a request picks the engine by its `model` field, so `/v1/models` lists `frank-dragoon` **and** `qwen3-embedding-0.6b`. The fleet's consumer is open-notebook's embedding default ([[Open Notebook]]); that page owns the app-side shapes (the URL guard, defaults-by-id, the `/api/embed` 500 that still writes the vector).
+Armed, the boot ledger reads (`/home/dank/logs/frank_dragoon.log`, 2026-10-19): the GPU fabric clock is held at its top level (2000 MHz) **before** the NPU starts; `/dev/accel/accel0` is `crw-rw-rw- nobody:nogroup` under driver `amdxdna` with the distro XRT and its NPU plugin; the program arrives as `19 files checked against this image's record`; then `npu_api: qwen3-embedding-0.6b (embeddings) on the NPU engine at 127.0.0.1:8740 (buckets u0, u1, u10, u11, u12, u13, u2, u3, u4, u5, u6, u7, u8, u9; inputs up to 4096 tokens)` and `up to 64 passes queued … up to 7200 s`. Callers never see `:8740` — the lane answers on the same `:8731` as the Flash model and a request picks the engine by its `model` field, so `/v1/models` lists `frank-dragoon` **and** `qwen3-embedding-0.6b`. The fleet's consumer is open-notebook's embedding default ([[Open Notebook]]); that page owns the app-side shapes (the URL guard, defaults-by-id, the `/api/embed` 500 that still writes the vector).
 
 Price: the programs and their weights sit in host RAM beside the locked weights ("a few GB in all" for the default
 set; ~1.1 GiB for `embeddinggemma-2`), and **the Flash model decodes slower while the NPU works** — same memory bus,

serverAppConfigs-docs/LLM/frank-zealot.md

@@ -1,6 +1,6 @@
 ---
 type: reference
-title: "frank-zealot on Dankclaw — llama.cpp (PrismML fork 10777), Ternary-Bonsai-2-9B PQ2_0-MTP + refusal adapter, 3 × 200704, f16 K + q8_0 V, MTP armed"
+title: "frank-zealot on Dankclaw — llama.cpp (PrismML fork 12118), Ternary-Bonsai-2-9B PQ2_0-MTP + refusal adapter, 3 × 200704, f16 K + q8_0 V, MTP armed"
 description: "The RTX 3090 endpoint: live shape exactly as the env ships it, every era with its rollback path, and its card laws."
 tags:
   - llm
@@ -31,7 +31,7 @@ is the bug.
 | Projector | `…/Bonsai2-9B-MTP/Ternary-Bonsai-2-9B-mmproj-Q8_0.gguf` — 624,229,664 B, sha256 `e0cb6f6988cea323…` — **Q8_0 is mandatory**, not a preference (see Contract) |
 | Adapter | `…/Bonsai2-9B-MTP/Ternary-Bonsai-2-9B-Uncensored-LoRA.gguf` — **3,155,424 B**, sha256 `861b9c099fb5cd81…` — a rank-1 refusal-direction LoRA (`adapter.type lora`, 128 tensors, every transformer layer, embedding untouched), applied at load by `--lora-scaled <file>:1`. Its metadata names `Ternary-Bonsai-2-9B-PTQ1_0.gguf` (`3561df7a…`) as the pack it was fitted against — **not this packing, and not on this host**; see Laws |
 | Fetch | `scripts/download-frank-zealot-bonsai9.sh [DEST]` (payload + projector) · `scripts/download-frank-zealot-uncensored-lora.sh [DEST]` (the adapter) — both `--verify-only`-able, `-f`, byte count and sha256 per file |
-| Runtime | `/home/dank/GitPackages/llama-prism-b10777/llama-server` — PrismML **fork**, `0.2.0-dev (build 10777, commit e3de71522)`; the only loader for this payload |
+| Runtime | `/home/dank/GitPackages/llama-prism-0244744/bin/llama-server` — PrismML **fork**, `0.7.0-dev (build 12118, commit 024474448)`; the only loader for this payload |
 | Trio | `scripts/frank-zealot.service` (root, relinked, `User=dank`, `Restart=on-failure`, `RestartSec=10`, `LimitMEMLOCK=infinity`, `TimeoutStopSec=180`) · `scripts/frank-zealot.env` · `scripts/serve-frank-zealot.sh` |
 | Address / alias | `BIND_ADDR=0.0.0.0`, `LISTEN_PORT=7771`, `ALIAS=frank-zealot` (a flag, not a fact) |
 | Flags | `--ctx-size 602112 --parallel 3` ⇒ 200,704 per slot · `--cache-type-k f16 --cache-type-v q8_0` (asymmetric, **14,406.00 MiB**) · `--flash-attn on` · `--batch-size 6144 --ubatch-size 1024` · `--n-gpu-layers 99` · `--jinja --reasoning auto --reasoning-format deepseek` · `--metrics` · `--mmproj` · **`--lora-scaled Ternary-Bonsai-2-9B-Uncensored-LoRA.gguf:1`** · `--log-verbosity 4` · `--presence-penalty 1.0` · **`--spec-type draft-mtp`** with `--spec-draft-n-max 2 --spec-draft-n-min 0 --spec-draft-p-min 0` |
@@ -136,6 +136,7 @@ with 448 completion tokens (at `max_tokens: 64` the same request returns EMPTY c
 
 | Window | Payload | Runtime | Shape | Replaced because | Rollback |
 |---|---|---|---|---|---|
+| 2026-10-30 → | same 9B MTP file, Q8_0 mmproj and refusal adapter (payload untouched) | PrismML fork **12118** (`0.7.0-dev`, `024474448`) | unchanged (3 × 200,704; f16 K + q8_0 V; MTP depth 2) | fork rebuilt from `origin/prism`; the payload gate held — 202 `pq2_0` tensors and 202 Hadamard-folded weights loaded, MTP drafting engaged, chat/tools/vision answered | `LLAMA_BIN` back to `llama-prism-b10777/llama-server` (kept on disk) + redeploy |
 | 2026-10-19 → | the same 9B MTP file and projector, **plus** `Ternary-Bonsai-2-9B-Uncensored-LoRA.gguf` (3,155,424 B `861b9c09…`) applied by `--lora-scaled …:1` | PrismML fork 10777 (its verified build) | — (shape unchanged; the adapter is behaviour) | Frank ordered the refusal direction removed on this endpoint | one env line — `LORA_FILE=` (or `LORA_SCALE=0`) — and redeploy; the plain pack is the same file it already loads |
 | 2026-10-18 21:06 → 2026-10-19 | the same 9B MTP file and projector | PrismML fork 10777 | — | Frank ordered the third conversation back AND the drafter armed, and exact V funds neither: it bills 18,816 MiB at the pool that pays for the third slot, leaving nothing for the draft's 2,352 | one line each — `CTX_SIZE=524288 PARALLEL=2 CACHE_TYPE_V=f16 BATCH_SIZE=4096 UBATCH_SIZE=2048 SPEC_TYPE=none` — and redeploy; no payload move |
 | 2026-10-18 13:19 → same evening (the boot `68afc6d` replaced) | the same 9B MTP file and projector | PrismML fork 10777 | — | Frank ordered unquantized KV, and that pool cannot survive it: 24,576 MiB of f16 cache > 24,123 addressable | one env line — `CTX_SIZE=786432 PARALLEL=3 CACHE_TYPE_K/V=q8_0` — and redeploy; no payload move |
@@ -201,7 +202,14 @@ with 448 completion tokens (at `max_tokens: 64` the same request returns EMPTY c
 - `serve-frank-zealot.sh` is a direct `ExecStart`, so a missing shebang is a `203/EXEC` boot loop at `Mem peak: 1.9M` with the GPU untouched: `head -1` is a deploy check.
 - `No CUDA GPUs are available` ⇒ `nvidia-modprobe -u -c=0`; KV spilled into unified memory halves decode silently, so re-read `nvidia-smi` after any shape change.
 - **An adapter's provenance is metadata, not a gate.** Its GGUF carries `prism.abliterate.base_gguf = Ternary-Bonsai-2-9B-PTQ1_0.gguf` and that pack's sha256; the endpoint applies the file to the PQ2_0+MTP packing anyway, because the loader matches **tensor names, not hashes**. So an adapter's effect is only ever proven behaviourally (a refusal probe against the live endpoint), never by the card, and an adapter whose rank landed on `token_embd` would be a crash rather than a degradation — the publisher leaves the embedding untouched for exactly that reason.
-- **The fork pin now has a third reason.** The payload's custom type 142, the b10776 floor that pairs MTP with a quantized V, and now this: older PrismML builds (e.g. `prism-b10709`) refuse the adapter outright with `unsupported prism.hadamard.version: 2`, so a runtime that can load the pack can still be unable to carry the adapter.
+- **A fork rebuild needs an install RPATH and a bin/lib-aware `LLAMA_BIN`.** `cmake --install` strips the
+  build-tree rpath, so the installed `llama-server` dies on `libllama-server-impl.so` unless the configure
+  carries `-DCMAKE_INSTALL_RPATH='$ORIGIN;$ORIGIN/../lib'`; the installed layout is `bin/` + `lib/`, not the
+  flat directory b10777 shipped, so the env's `LLAMA_BIN` has to name `…/bin/llama-server`. Build 12118 was
+  made from `origin/prism` with the 10777 configure (Release, `GGML_CUDA=ON`, `sm_86`,
+  `CMAKE_CUDA_HOST_COMPILER=/usr/bin/gcc-13`, Ninja) — gcc 15 is the box default and nvcc 12.4 rejects it.
+- **The fork pin now has a third reason.** The payload's custom type 142, the b10776 floor that pairs MTP with a quantized V, and now this: older PrismML builds (e.g. `prism-b10709`) refuse the adapter outright with `unsupported prism.hadamard.version: 2`, so a runtime that can load the pack can still be unable to carry the adapter. Build 12118 clears all
+three.
 
 ## Contract
 

serverAppConfigs-docs/LLM/index.md

@@ -13,6 +13,20 @@ the endpoint's **Eras** table, and the numbers a retired era measured are gone.
 | [[frank-dragoon]] | The iGPU endpoint: live shape, every era with its rollback path, the laws it paid for, and the files that own it. |
 | [[frank-zealot]] | The RTX 3090 endpoint: live shape exactly as the env ships it, every era with its rollback path, and its card laws. |
 
+## Radiance engine reference
+
+| Page | Owns |
+|---|---|
+| [[Radiance Engine]] | The radiance inference engine: its shape, plugin model, container format, and how the reference pages fit together. |
+| [[Radiance Serve Options]] | Every radiance serve flag grouped by task, with its default and the law that decides when to move it. |
+| [[Radiance HTTP API]] | The HTTP surface radiance serves: endpoints, request fields, streaming, tools, and refusal rules. |
+| [[Radiance Conversion]] | Turning a checkpoint into a .rad container: rad-convert, the recipe language, quantisers, reuse and export. |
+| [[Radiance Model Families]] | Which radiance plugin claims which model family, and the conversion command and recipe of record for each. |
+| [[Radiance Plugins]] | Writing radiance kernel and quantiser plugins: the ABI, kernel rows, hooks, encodings, and the build path. |
+| [[Radiance Architecture Plugins]] | Writing a radiance architecture plugin: declaring weights, buffers, KV groups and ops, then issuing the step. |
+| [[Radiance Tools]] | The radiance inspection, measurement and tokeniser tools, with the flags and verdicts each one reports. |
+| [[Radiance Operations]] | Running a radiance server: the container contract, host prerequisites, readiness, ledgers and safe restarts. |
+
 ## Retired endpoints
 
 | Page | Owns |

serverAppConfigs-docs/index.md

@@ -51,6 +51,7 @@ is a locator: one row, one clause, no numbers.
 | [[Entropy Volume]] | The Dankserver entropy volume is a RAID-backed LUKS filesystem mounted at |
 | [[Jellyfin]] | Jellyfin on Dankserver: quadlet unit, entropy-volume data layout, ports, client access, hardware-transcode status. |
 | [[mariadb]] | The MariaDB backend for RomM: quadlet unit, network alias, env-file secret handling, and the backup path. |
+| [[Media Relayout]] | Census, plan and apply the movie library into Jellyfin format: the gates to read, and the rollback. |
 | [[Plex]] | Plex Media Server is a **containerized media server on Dankserver**: a podman |
 | [[redis]] | The Redis cache and task queue for RomM: quadlet unit, network alias topology, and the backup path. |
 | [[RomM]] | RomM (ROM Manager) is a self-hosted ROM library manager and player. |
@@ -162,6 +163,15 @@ is a locator: one row, one clause, no numbers.
 | [[PCIe Link and Atomics]] | Link probing, the streaming-versus-compute ceiling arithmetic, and the atomics wall that decides multi-GPU collectives. |
 | [[Quantization by Hardware Architecture]] | This page maps LLM quantization formats to the GPU and CPU architectures that support them. |
 | [[Quantization Production]] | Making a quantized payload here: the conversion recipe, the scoring gate, and the campaign ladder behind a shipped file. |
+| [[Radiance Architecture Plugins]] | Writing a radiance architecture plugin: declaring weights, buffers, KV groups and ops, then issuing the step. |
+| [[Radiance Conversion]] | Turning a checkpoint into a .rad container: rad-convert, the recipe language, quantisers, reuse and export. |
+| [[Radiance Engine]] | The radiance inference engine: its shape, plugin model, container format, and how the reference pages fit together. |
+| [[Radiance HTTP API]] | The HTTP surface radiance serves: endpoints, request fields, streaming, tools, and refusal rules. |
+| [[Radiance Model Families]] | Which radiance plugin claims which model family, and the conversion command and recipe of record for each. |
+| [[Radiance Operations]] | Running a radiance server: the container contract, host prerequisites, readiness, ledgers and safe restarts. |
+| [[Radiance Plugins]] | Writing radiance kernel and quantiser plugins: the ABI, kernel rows, hooks, encodings, and the build path. |
+| [[Radiance Serve Options]] | Every radiance serve flag grouped by task, with its default and the law that decides when to move it. |
+| [[Radiance Tools]] | The radiance inspection, measurement and tokeniser tools, with the flags and verdicts each one reports. |
 | [[RDNA4 MoE Expert Offload VRAM Budget]] | Expert-offload arithmetic on the RDNA4 fork: movable terms, the utilisation law, graph charge, ceilings, context ladder. |
 | [[Shared Memory Architecture Dankclaw]] | Dankclaw uses AMD Strix Halo APU with unified memory architecture. |
 

tests/frank_dragoon_unit.bats

@@ -1,7 +1,7 @@
 #!/usr/bin/env bats
 # Seam: frank-dragoon serves Quat3rnion's NATIVE v2 conversion (2026-10-11)
 # of the abliterated Qwen3.8-Flash-Next (a .hgn flash-server payload) on the
-# Strix Halo iGPU through the halogen-flash-server CONTAINER (0.16.2,
+# Strix Halo iGPU through the halogen-flash-server CONTAINER (0.18.20,
 # digest-pinned), driven by the root unit's serve script — 2 slots over ONE
 # 401,408-position KV pool at the native 262,144 ceiling, MTP head always on (HALOGEN_SPEC_ADAPT=0)
 # and the vision tower mounted. 2026-10-17 cutover from the Swift1.5-finetune

tests/frank_zealot_unit.bats

@@ -126,11 +126,22 @@ run_stage() {
 @test "the shipped env points at the fork runtime, the MTP pair and the refusal adapter" {
     # The payload path IS the endpoint's identity, and the runtime is part of
     # it here: the file's custom tensor type 142 + prism.hadamard.* metadata
-    # make the PrismML build the only loader, and this build must be >= b10776
+    # make the PrismML build the only loader, and that build must be >= b10776
     # because MTP beside q8_0 V is illegal below it. A retarget that moves the
-    # model but leaves the old build behind ships an endpoint that cannot load
+    # model but leaves a stock llama.cpp behind ships an endpoint that cannot
-    # (type 142) or cannot run the documented MTP arm (the build gate).
+    # load (type 142) or cannot run the documented MTP arm (the build gate).
-    grep -qE '^LLAMA_BIN=/home/dank/GitPackages/llama-prism-b10777/llama-server$' "$ENV_FILE"
+    #
+    # Assert the fork FAMILY, not the directory one rebuild happens to install:
+    # a literal path turns every runtime move into a test edit, and the suite
+    # runs on a box that has no /home/dank at all, so existence cannot be
+    # checked here either.
+    local bin
+    bin="$(sed -n 's/^LLAMA_BIN=//p' "$ENV_FILE")"
+    if [[ ! "$bin" =~ ^/home/dank/GitPackages/llama-prism-[^/]+/llama-server$ ]] &&
+       [[ ! "$bin" =~ ^/home/dank/GitPackages/llama-prism-[^/]+/bin/llama-server$ ]]; then
+        echo "LLAMA_BIN does not name a PrismML fork build: '$bin'" >&2
+        return 1
+    fi
     grep -qE '^MODEL_FILE=/home/dank/models/Bonsai2-9B-MTP/Ternary-Bonsai-2-9B-PQ2_0-MTP-Q8_0\.gguf$' "$ENV_FILE"
     grep -qE '^MMPROJ_FILE=/home/dank/models/Bonsai2-9B-MTP/Ternary-Bonsai-2-9B-mmproj-Q8_0\.gguf$' "$ENV_FILE"
     grep -qE '^LORA_FILE=/home/dank/models/Bonsai2-9B-MTP/Ternary-Bonsai-2-9B-Uncensored-LoRA\.gguf$' "$ENV_FILE"

tests/media_relayout.bats

@@ -0,0 +1,333 @@
+#!/usr/bin/env bats
+# Seam: the movie-library relayout — scripts/media-census.py (the read-only
+# census), scripts/media-plan.py (the plan) and scripts/media-apply.py (the only
+# thing that moves files).
+#
+# The library is 8k files on a 100 %-full array, so the two things this seam
+# exists to catch are the ones that cost a re-run of a 20-minute census: a file
+# that ends up in the WRONG movie folder, and a move that destroys something.
+# Everything below runs on a synthetic library in the test's own tmpdir: no
+# ffprobe, no network, no server. The probe step is the one part a fixture
+# cannot run (it opens real containers), so the tests feed the planner the
+# probed TSV in the column order scripts/media-census.py exports — which pins
+# that contract as a side effect.
+#
+# The plan's own gate lines (unplanned, left-unmoved, double-claim, occupied)
+# are asserted as output, because they are how the operator learns the plan is
+# trustworthy before pointing the executor at 24 TB.
+
+load helpers
+
+PLAN="$REPO_ROOT/scripts/media-plan.py"
+CENSUS="$REPO_ROOT/scripts/media-census.py"
+APPLY="$REPO_ROOT/scripts/media-apply.py"
+
+# probe_row FILE DUR [RES] [AUDIO] [SUBS] — one probed-TSV row: the twelve
+# columns media-census.py probe writes (bytes, path, dur, vcodec, res, fps,
+# vbit, pix, audio, subs, spare, fmt).
+probe_row() {
+    local p="$1" dur="$2" res="${3:-1920x1080}" aud="${4:-aac+aac}" subs="${5:-0}"
+    printf '%s\t%s\t%s\thevc\t%s\t23.976\t5000000\t8bit\t%s\t%s\t0\tmatroska,webm\n' \
+        "$(stat -c %s "$p")" "$p" "$dur" "$res" "$aud" "$subs" >> "$PROBED"
+}
+
+# plex_row FILE TITLE YEAR — one Plex metadata row; the planner reads these
+# three columns and treats a row without a plausible year as no match at all.
+plex_row() {
+    printf '%s\t%s\t%s\n' "$1" "$2" "$3" >> "$PLEX"
+}
+
+# at PATH SIZE — a file with deterministic content, big enough to matter.
+at() { mkdir -p "$(dirname "$1")"; head -c "$2" /dev/zero > "$1"; }
+
+setup() {
+    LIB="$BATS_TEST_TMPDIR/Movies"
+    ENUM="$BATS_TEST_TMPDIR/enum.tsv"
+    PROBED="$BATS_TEST_TMPDIR/probed.tsv"
+    PLEX="$BATS_TEST_TMPDIR/plex.tsv"
+    OUT="$BATS_TEST_TMPDIR/plan.tsv"
+    EXC="$BATS_TEST_TMPDIR/exceptions.txt"
+    : > "$PROBED"
+    printf 'file\tleaf_title\tleaf_year\n' > "$PLEX"
+
+    # Two films that share a box set and a word, plus their bonus material:
+    # the shape that made one film's featurettes land in another's folder.
+    at "$LIB/Alien (1979)/Alien.1979.1080p.BluRay.x265.10bit.mkv" 4000
+    at "$LIB/Alien (1979)/Featurettes/A Bug And A Bite - Making Alien.mkv" 3300000
+    at "$LIB/Alien (1979)/Extras/Extra - Logo.mkv" 900000
+    at "$LIB/Alien (1979)/subs/3_Eng.srt" 40000
+    at "$LIB/Marnie (1964)/Marnie.1964.1080p.BluRay.x265.mkv" 5000000
+    at "$LIB/Marnie (1964)/Featurettes/The Trouble With Marnie.mkv" 2400000
+    # A long extra: over 45 minutes, so only the folder may call it bonus.
+    at "$LIB/Marnie (1964)/Featurettes/Commentary And Archives.mkv" 2600000
+    # A misspelled bonus folder: no bonus vocabulary, so only the folder that
+    # holds the feature can decide that this is not a film.
+    at "$LIB/Marnie (1964)/Exrtas/Marnie - A Closer Look.mkv" 2500000
+    # A Plex title that carries "/" and a Plex row that was never matched.
+    at "$LIB/VHS 2 (2013)/VHS.2.2013.1080p.BluRay.x265.mkv" 4200000
+    at "$LIB/AMERICAN POP/AMERICAN.POP.2023.1080p.WEB-DL.mkv" 3100000
+    # Junk the layout has to park somewhere other than a movie folder.
+    at "$LIB/.DS_Store" 6148
+    at "$LIB/Alien (1979)/._Alien.1979.1080p.BluRay.x265.10bit.mkv" 220
+    # A bonus file whose name is a number: never a movie folder named "1".
+    at "$LIB/Bruce Lee Collection 1971-1981/.Extra 1.mkv" 2700000
+
+    probe_row "$LIB/Alien (1979)/Alien.1979.1080p.BluRay.x265.10bit.mkv" 4700
+    probe_row "$LIB/Alien (1979)/Featurettes/A Bug And A Bite - Making Alien.mkv" 2700
+    probe_row "$LIB/Alien (1979)/Extras/Extra - Logo.mkv" 30
+    probe_row "$LIB/Marnie (1964)/Marnie.1964.1080p.BluRay.x265.mkv" 4800
+    probe_row "$LIB/Marnie (1964)/Featurettes/The Trouble With Marnie.mkv" 1200
+    probe_row "$LIB/Marnie (1964)/Featurettes/Commentary And Archives.mkv" 3000
+    probe_row "$LIB/Marnie (1964)/Exrtas/Marnie - A Closer Look.mkv" 2600
+    probe_row "$LIB/VHS 2 (2013)/VHS.2.2013.1080p.BluRay.x265.mkv" 5700
+    probe_row "$LIB/AMERICAN POP/AMERICAN.POP.2023.1080p.WEB-DL.mkv" 6300
+    probe_row "$LIB/Bruce Lee Collection 1971-1981/.Extra 1.mkv" 5220
+    plex_row "$LIB/Alien (1979)/Alien.1979.1080p.BluRay.x265.10bit.mkv" "Alien" "1979"
+    plex_row "$LIB/Marnie (1964)/Marnie.1964.1080p.BluRay.x265.mkv" "Marnie" "1964"
+    plex_row "$LIB/VHS 2 (2013)/VHS.2.2013.1080p.BluRay.x265.mkv" "V/H/S/2" "2013"
+    # An unmatched Plex row: the leaf title echoes the file and the year is 0.
+    plex_row "$LIB/AMERICAN POP/AMERICAN.POP.2023.1080p.WEB-DL.mkv" "AMERICAN POP" "0"
+
+    export MEDIA_ROOT="$LIB"
+    run python3 "$CENSUS" enumerate --root "$LIB" --out "$ENUM"
+    [ "$status" -eq 0 ]
+}
+
+# plan — run the planner over the fixture, keeping its plan TSV, its exception
+# report and the gate lines it prints (that stdout is the operator's evidence
+# that the plan accounts for every file, so the tests read it too).
+plan() {
+    PLANOUT="$BATS_TEST_TMPDIR/plan.out"
+    run python3 "$PLAN" --root "$LIB" --enum "$ENUM" --probed "$PROBED" \
+        --plex "$PLEX" --out "$OUT" --exceptions "$EXC"
+    [ "$status" -eq 0 ]
+    printf '%s\n' "$output" > "$PLANOUT"
+}
+
+# dst_of SRC — the destination the plan gives one file.
+dst_of() {
+    python3 -c '
+import sys
+for line in open(sys.argv[1]):
+    f = line.rstrip("\n").split("\t")
+    if len(f) > 2 and f[1] == sys.argv[2]:
+        print(f[2]); break
+' "$OUT" "$1"
+}
+
+@test "a film lands in Title (year)/Title (year).mkv, named by Plex" {
+    plan
+    [ "$(dst_of "$LIB/Alien (1979)/Alien.1979.1080p.BluRay.x265.10bit.mkv")" \
+        = "$LIB/Alien (1979)/Alien (1979).mkv" ]
+}
+
+@test "a Plex title that carries a slash becomes one folder, not a nested tree" {
+    plan
+    local d; d="$(dst_of "$LIB/VHS 2 (2013)/VHS.2.2013.1080p.BluRay.x265.mkv")"
+    [ "$(dirname "$d")" = "$LIB/V-H-S-2 (2013)" ]
+    [ ! -e "$LIB/V" ]
+    # the folder must be ONE mkdir: an un-sanitized name would make the
+    # executor build Movies/V/H/S/2 (2013) instead
+    run grep -F "mkdir" "$OUT"
+    [[ "$output" == *"$(printf '%s' "$LIB/V-H-S-2 (2013)")"* ]]
+    run awk -F'\t' '$1=="mkdir"{print $3}' "$OUT"
+    [[ "$output" != *"$LIB/V/"* ]]
+}
+
+@test "an unmatched Plex row never names a folder with a zero year" {
+    plan
+    run awk -F'\t' '{print $3}' "$OUT"
+    [[ "$output" != *"(0)"* ]]
+    # the filename's own year carries the folder, the folder and the file share
+    # the name (Jellyfin's rule), and the film sits at the library root rather
+    # than in an unsorted pile
+    local d; d="$(dst_of "$LIB/AMERICAN POP/AMERICAN.POP.2023.1080p.WEB-DL.mkv")"
+    [ "$(dirname "$(dirname "$d")")" = "$LIB" ]
+    [[ "$(basename "$(dirname "$d")")" == *" (2023)" ]]
+    [ "$(basename "$d")" = "$(basename "$(dirname "$d")").mkv" ]
+}
+
+@test "a bonus folder keeps its featurettes inside its own film, not a neighbour's" {
+    plan
+    local d
+    d="$(dst_of "$LIB/Alien (1979)/Featurettes/A Bug And A Bite - Making Alien.mkv")"
+    [ "$(dirname "$(dirname "$d")")" = "$LIB/Alien (1979)" ]
+    d="$(dst_of "$LIB/Marnie (1964)/Featurettes/The Trouble With Marnie.mkv")"
+    [ "$(dirname "$(dirname "$d")")" = "$LIB/Marnie (1964)" ]
+    d="$(dst_of "$LIB/Alien (1979)/Extras/Extra - Logo.mkv")"
+    [ "$(dirname "$(dirname "$d")")" = "$LIB/Alien (1979)" ]
+    # the feature-length extra is bonus too: it sits in a bonus folder and no
+    # metadata matched it, so it may not become a movie of its own
+    d="$(dst_of "$LIB/Marnie (1964)/Featurettes/Commentary And Archives.mkv")"
+    [ "$(dirname "$(dirname "$d")")" = "$LIB/Marnie (1964)" ]
+}
+
+@test "a folder inside a movie folder is bonus even when its name is misspelled" {
+    plan
+    local d; d="$(dst_of "$LIB/Marnie (1964)/Exrtas/Marnie - A Closer Look.mkv")"
+    [ "$(dirname "$(dirname "$d")")" = "$LIB/Marnie (1964)" ]
+}
+
+@test "a subtitle follows the film of the folder it sits in" {
+    plan
+    local d; d="$(dst_of "$LIB/Alien (1979)/subs/3_Eng.srt")"
+    [ "$(dirname "$(dirname "$d")")" = "$LIB/Alien (1979)" ]
+}
+
+@test "macOS sidecars are parked in one pile, never inside a movie folder" {
+    plan
+    local d; d="$(dst_of "$LIB/Alien (1979)/._Alien.1979.1080p.BluRay.x265.10bit.mkv")"
+    [[ "$(dirname "$d")" == "$LIB"/_* ]]
+    d="$(dst_of "$LIB/.DS_Store")"
+    [[ "$(dirname "$d")" == "$LIB"/_* ]]
+    grep -q "AppleDouble" "$EXC"
+}
+
+@test "a file whose name is a number becomes a review line, not a folder named 1" {
+    plan
+    run awk -F'\t' '$1=="mkdir"{print $3}' "$OUT"
+    [[ "$output" != "$LIB/1"$'\n'* ]] && [[ "$output" != "$LIB/1" ]]
+    [[ "$output" != *"$LIB/1/"* ]]
+    grep -q "UNNAMED" "$EXC"
+}
+
+@test "every file in the census is placed or kept — nothing is dropped" {
+    plan
+    python3 - "$ENUM" "$OUT" <<'PY'
+import sys
+enum = {l.rstrip("\n").split("\t")[2] for l in open(sys.argv[1])}
+placed = set()
+for line in open(sys.argv[2]):
+    f = line.rstrip("\n").split("\t")
+    if f[0] in ("move", "keep"):
+        placed.add(f[1])
+missing = enum - placed
+assert not missing, f"{len(missing)} census files with no plan row: {sorted(missing)[:3]}"
+PY
+    grep -q "left-unmoved=0" "$PLANOUT"
+    grep -q "double-claim=0" "$PLANOUT"
+    grep -q "occupied=0" "$PLANOUT"
+    grep -q "bad-names=0" "$PLANOUT"
+    grep -q "unplanned=0" "$EXC"
+}
+
+# ---------------------------------------------------------------- executor
+
+# apply_plan — plan, then apply for real; the journal goes in the tmpdir.
+apply_plan() {
+    JOURNAL="$BATS_TEST_TMPDIR/journal.jsonl"
+    run python3 "$APPLY" apply --plan "$OUT" --root "$LIB" --journal "$JOURNAL" --apply
+}
+
+@test "a run without --apply says what it would do and touches nothing" {
+    plan
+    JOURNAL="$BATS_TEST_TMPDIR/journal.jsonl"
+    run python3 "$APPLY" apply --plan "$OUT" --root "$LIB" --journal "$JOURNAL"
+    [ "$status" -eq 0 ]
+    [[ "$output" == *"--apply"* ]]
+    [ -e "$LIB/Alien (1979)/Alien.1979.1080p.BluRay.x265.10bit.mkv" ]
+    [ ! -e "$LIB/Alien (1979)/Alien (1979).mkv" ]
+    # a dry run may not even build the folders: an operator must be able to
+    # run it against the live library with no --apply and change nothing
+    [ ! -d "$LIB/V-H-S-2 (2013)" ]
+    [ ! -e "$JOURNAL" ]
+}
+
+@test "apply moves every planned file, keeps its bytes, and journals each move" {
+    plan
+    apply_plan
+    [ "$status" -eq 0 ]
+    [ -e "$LIB/Alien (1979)/Alien (1979).mkv" ]
+    [ ! -e "$LIB/Alien (1979)/Alien.1979.1080p.BluRay.x265.10bit.mkv" ]
+    local planned moved
+    planned=$(awk -F'\t' '$1=="move"' "$OUT" | wc -l)
+    moved=$(python3 -c 'import json,sys
+print(sum(1 for l in open(sys.argv[1]) if json.loads(l).get("op") == "move"))' "$JOURNAL")
+    [ "$moved" -eq "$planned" ]
+    run python3 "$APPLY" verify --plan "$OUT" --root "$LIB" --journal "$JOURNAL"
+    [ "$status" -eq 0 ]
+}
+
+@test "a target already holding a different file stops the whole run" {
+    plan
+    JOURNAL="$BATS_TEST_TMPDIR/journal.jsonl"
+    at "$LIB/Alien (1979)/Alien (1979).mkv" 7
+    run python3 "$APPLY" apply --plan "$OUT" --root "$LIB" --journal "$JOURNAL" --apply
+    [ "$status" -ne 0 ]
+    [[ "$output" == *"occupied"* ]]
+    # nothing moved at all: the other films are still where the census saw them
+    [ -e "$LIB/Marnie (1964)/Marnie.1964.1080p.BluRay.x265.mkv" ]
+    [ ! -e "$JOURNAL" ]
+}
+
+@test "a plan row whose source has gone stops the run instead of inventing a move" {
+    plan
+    JOURNAL="$BATS_TEST_TMPDIR/journal.jsonl"
+    rm "$LIB/Marnie (1964)/Marnie.1964.1080p.BluRay.x265.mkv"
+    run python3 "$APPLY" apply --plan "$OUT" --root "$LIB" --journal "$JOURNAL" --apply
+    [ "$status" -ne 0 ]
+    [[ "$output" == *"missing"* ]]
+    [ -e "$LIB/Alien (1979)/Alien.1979.1080p.BluRay.x265.10bit.mkv" ]
+}
+
+@test "a run interrupted between the journal write and the move finishes the job" {
+    plan
+    JOURNAL="$BATS_TEST_TMPDIR/journal.jsonl"
+    # The journal is written before each move, so an interruption leaves two
+    # states and only two: an intent whose move happened, and an intent whose
+    # move never ran. Both are simulated here by hand.
+    python3 - "$OUT" "$JOURNAL" <<'PY'
+import json, os, sys
+moves = [r for r in (l.rstrip("\n").split("\t") for l in open(sys.argv[1]))
+         if r[0] == "move"][:3]
+with open(sys.argv[2], "w") as fh:
+    for i, r in enumerate(moves):
+        fh.write(json.dumps({"op": "move", "src": r[1], "dst": r[2],
+                             "bytes": os.path.getsize(r[1])}) + "\n")
+        if i < 2:                     # the first two moves actually ran
+            os.makedirs(os.path.dirname(r[2]), exist_ok=True)
+            os.rename(r[1], r[2])
+PY
+    run python3 "$APPLY" apply --plan "$OUT" --root "$LIB" --journal "$JOURNAL" --apply
+    [ "$status" -eq 0 ]
+    run python3 "$APPLY" verify --plan "$OUT" --root "$LIB" --journal "$JOURNAL"
+    [ "$status" -eq 0 ]
+    python3 - "$OUT" "$JOURNAL" <<'PY'
+import json, os, sys
+moves = [r for r in (l.rstrip("\n").split("\t") for l in open(sys.argv[1]))
+         if r[0] == "move"]
+journal = [json.loads(l) for l in open(sys.argv[2])]
+seen = [j["dst"] for j in journal]
+assert len(seen) == len(set(seen)), "a destination was journalled twice"
+for r in moves:
+    assert os.path.exists(r[2]) and not os.path.exists(r[1]), r[2]
+PY
+}
+
+@test "rollback puts every file back where the census found it" {
+    plan
+    apply_plan
+    [ "$status" -eq 0 ]
+    run python3 "$APPLY" rollback --root "$LIB" --journal "$JOURNAL" --apply
+    [ "$status" -eq 0 ]
+    [ -e "$LIB/Alien (1979)/Alien.1979.1080p.BluRay.x265.10bit.mkv" ]
+    [ -e "$LIB/Marnie (1964)/Marnie.1964.1080p.BluRay.x265.mkv" ]
+    [ -e "$LIB/Alien (1979)/subs/3_Eng.srt" ]
+    [ ! -e "$LIB/Alien (1979)/Alien (1979).mkv" ]
+    # the folders the run created are empty afterwards and must not litter the
+    # library: this one never existed before the plan ran
+    [ -d "$LIB/VHS 2 (2013)" ]
+    [ ! -e "$LIB/V-H-S-2 (2013)" ]
+}
+
+@test "rollback refuses when a moved file has gone missing" {
+    plan
+    apply_plan
+    [ "$status" -eq 0 ]
+    rm "$LIB/Alien (1979)/Alien (1979).mkv"
+    run python3 "$APPLY" rollback --root "$LIB" --journal "$JOURNAL" --apply
+    [ "$status" -ne 0 ]
+    [[ "$output" == *"missing"* ]]
+    # the refusal is all-or-nothing: another film did not move back
+    [ -e "$LIB/Marnie (1964)/Marnie (1964).mkv" ]
+}

tests/qwen38_campaign.bats

@@ -200,6 +200,9 @@ DERIVE="$REPO_ROOT/scripts/derive-protected-experts.py"
 APPLY="$REPO_ROOT/scripts/apply-protected-cast.sh"
 CONVERT="$REPO_ROOT/scripts/convert-qwen38-w5nl64.sh"
 GATE="$REPO_ROOT/scripts/kld-gate-qwen38.sh"
+RUNNER="$REPO_ROOT/scripts/run-qwen38-bf16.sh"
+PUSHER="$REPO_ROOT/scripts/calib-push.py"
+WINDOW="$REPO_ROOT/scripts/qwen38-calib-window.sh"
 TEMPLATE="$REPO_ROOT/scripts/recipes/qwen4exp-qwen38-w5nl64-bf16trunk.recipe"
 
 # make_gram <dir> <layer> <expert> <energy> <rows> — a RADGRAM2 down-Gram whose
@@ -525,3 +528,411 @@ gate_case() { # gate_case <kl fp8> <kl iq4r> <kl cand> <top1 cand> [extra flags.
     [[ "$output" == *"has no slice 'code'"* ]]
     [[ "$output" == *"it has: ops"* ]]
 }
+
+# --- the n-gram tier: the pin that starved the calibration boot ---------------
+
+@test "the calibration boot reads the n-gram table from the file tier" {
+    # 2026-10-29: 95.37 GiB pinned beside a 65.59 GiB/rank pool left 8.06 GiB of
+    # RAM and the boot died "startup stopped: out of memory". calib only needs the
+    # routed weights prefilled, so it reads the table off the container's file tier.
+    run bash "$RUNNER" calib --dry-run
+    [ "$status" -eq 0 ]
+    grep -A1 -x -- '--ngram-placement' <<<"$output" | grep -qx 'disk'
+}
+
+@test "the recording and every score pass keep production's n-gram placement" {
+    # Those numbers are only comparable when the shape is identical; the tier is
+    # part of the shape.
+    run bash "$RUNNER" kldref --dry-run
+    [ "$status" -eq 0 ]
+    grep -A1 -x -- '--ngram-placement' <<<"$output" | grep -qx 'ram'
+}
+
+@test "an unknown n-gram placement is refused by name before the boot" {
+    run env FRANK_QWEN38_NGRAM_PLACEMENT=ssd bash "$RUNNER" calib --dry-run
+    [ "$status" -eq 2 ]
+    [[ "$output" == *"auto|vram|ram|disk"* ]]
+}
+
+# --- the pusher: a tap without a live rig is not a tap ------------------------
+
+# stub_podman <dir> <running|absent> — a podman that answers the one question the
+# pusher asks: is the rig container running?
+stub_podman() {
+    mkdir -p "$1/bin"
+    if [ "$2" = running ]; then
+        printf '#!/usr/bin/env bash\necho true\n' > "$1/bin/podman"
+    elif [ "$2" = stopped ]; then
+        printf '#!/usr/bin/env bash\necho false\n' > "$1/bin/podman"
+    else
+        printf '#!/usr/bin/env bash\nexit 125\n' > "$1/bin/podman"
+    fi
+    chmod +x "$1/bin/podman"
+}
+
+pusher_fixture() { # pusher_fixture <marker-container or empty>
+    D="$BATS_TEST_TMPDIR/push"
+    mkdir -p "$D/tap" "$D/bin"
+    printf '{"prompt":"hello"}\n' > "$D/corpus.jsonl"
+    if [ -n "$1" ]; then
+        printf 'container=%s\nimage=radiance@sha256:test\nmode=calib\npid=1\n' "$1" \
+            > "$D/tap/.rig-boot"
+    fi
+    echo "$D"
+}
+
+@test "the pusher refuses a tap that carries no rig marker: production is not a rig" {
+    # With no marker, the old driver fed four rounds of 10M tokens into restored
+    # production after the rig's boot had died, and recorded nothing.
+    D="$(pusher_fixture '')"
+    run python3 "$PUSHER" --corpus "$D/corpus.jsonl" --tap-dir "$D/tap" --port 1
+    [ "$status" -eq 2 ]
+    [[ "$output" == *"no rig marker"* ]]
+}
+
+@test "the pusher refuses a marker that names another container" {
+    D="$(pusher_fixture arbench)"
+    run python3 "$PUSHER" --corpus "$D/corpus.jsonl" --tap-dir "$D/tap" \
+        --container qwen38-bf16 --port 1
+    [ "$status" -eq 2 ]
+    [[ "$output" == *"names container 'arbench'"* ]]
+    [[ "$output" == *"targets 'qwen38-bf16'"* ]]
+}
+
+@test "the pusher refuses when the rig container is not running" {
+    D="$(pusher_fixture qwen38-bf16)"
+    stub_podman "$D" stopped
+    run env PATH="$D/bin:$PATH" python3 "$PUSHER" --corpus "$D/corpus.jsonl" \
+        --tap-dir "$D/tap" --container qwen38-bf16 --port 1
+    [ "$status" -eq 2 ]
+    [[ "$output" == *"is not running"* ]]
+}
+
+# --- the acceptance gate without an FP8 anchor -------------------------------
+
+# kld_report_slices <path> <model> <kl all> <kl code> <top1 all> <top1 code>
+kld_report_slices() {
+    python3 - "$@" <<'PY'
+import json, sys
+path, model, kl_all, kl_code, t_all, t_code = sys.argv[1:7]
+def g(kl, t, n=25000):
+    return {"kld": {"mean": float(kl), "p99_9": float(kl) * 4},
+            "top1_agreement": float(t), "positions": n}
+json.dump({"model": model, "all": g(kl_all, t_all),
+           "by_source": {"code": g(kl_code, t_code), "ops": g(kl_all, t_all)}},
+          open(path, "w"))
+PY
+}
+
+gate2() { # gate2 <kl iq4r> <kl cand> <top1 iq4r> <top1 cand> [extra]
+    D="$BATS_TEST_TMPDIR/kl2"
+    mkdir -p "$D"
+    kld_report_slices "$D/iq4r.json" iq4r-moe "$1" "$1" "$3" "$3"
+    kld_report_slices "$D/cand.json" w5nl64 "$2" "$2" "$4" "$4"
+    run bash "$GATE" "$D/iq4r.json" "$D/cand.json" "${@:5}"
+}
+
+@test "a candidate that beats the served payload on mean, tail and top-1 passes without an anchor" {
+    gate2 0.0190 0.0090 0.962 0.971
+    [ "$status" -eq 0 ]
+    [[ "$output" == *"candidate vs served (no FP8 anchor)"* ]]
+    [[ "$output" == *"KLD GATE PASS"* ]]
+}
+
+@test "a better mean with a worse tail fails: the tail is where a mean hides its damage" {
+    D="$BATS_TEST_TMPDIR/kl3"
+    mkdir -p "$D"
+    kld_report_slices "$D/iq4r.json" iq4r-moe 0.0190 0.0190 0.962 0.962
+    python3 - "$D/cand.json" <<'PY'
+import json, sys
+# mean better than the served payload, p99.9 worse: the shape a mean-only gate waves through
+g = {"kld": {"mean": 0.0090, "p99_9": 0.4000}, "top1_agreement": 0.971, "positions": 25000}
+json.dump({"model": "w5nl64", "all": g, "by_source": {"code": dict(g), "ops": dict(g)}},
+          open(sys.argv[1], "w"))
+PY
+    run bash "$GATE" "$D/iq4r.json" "$D/cand.json"
+    [ "$status" -eq 1 ]
+    [[ "$output" == *"[FAIL] 2."* ]]
+    [[ "$output" == *"KL p99.9"* ]]
+}
+
+@test "a candidate that wins overall but regresses one source slice fails" {
+    D="$BATS_TEST_TMPDIR/kl4"
+    mkdir -p "$D"
+    kld_report_slices "$D/iq4r.json" iq4r-moe 0.0190 0.0190 0.962 0.962
+    # 'all' is better, 'code' is worse: exactly the trade a mean-only gate cannot see
+    kld_report_slices "$D/cand.json" w5nl64 0.0090 0.0300 0.971 0.950
+    run bash "$GATE" "$D/iq4r.json" "$D/cand.json"
+    [ "$status" -eq 1 ]
+    [[ "$output" == *"[PASS] 1."* ]]
+    [[ "$output" == *"[FAIL] 4."* ]]
+    [[ "$output" == *"regressed: code"* ]]
+}
+
+# --- the calibration window: gated, and it says so ---------------------------
+
+window_fixture() {
+    D="$BATS_TEST_TMPDIR/win"
+    mkdir -p "$D/campaign/kld-ref" "$D/campaign/calib" "$D/rad" "$D/corpora" "$D/bin"
+    echo '{}' > "$D/campaign/kld-ref/kld.json"
+    : > "$D/rad/qwen38-bf16.rad"
+    echo "{\"p\":1}" > "$D/corpora/calib.jsonl"
+    # the window's preflight resolves the image digest through podman and asks about the
+    # port/device by host pid; these stubs answer for a quiet host
+    cat > "$D/bin/podman" <<'EOS'
+#!/usr/bin/env bash
+case "${1:-}" in
+    image) echo "sha256:69add58f7de1ed2bca2b1e20a4cc75717557df3f73569a178eca5aafe8c44093" ;;
+esac
+EOS
+    printf '#!/usr/bin/env bash\nexit 0\n' > "$D/bin/ss"
+    printf '#!/usr/bin/env bash\nexit 0\n' > "$D/bin/fuser"
+    chmod +x "$D/bin/podman" "$D/bin/ss" "$D/bin/fuser"
+    echo "$D"
+}
+
+@test "the calibration window refuses before touching a card without a reference recording" {
+    D="$(window_fixture)"
+    mv "$D/campaign/kld-ref/kld.json" "$D/campaign/kld-ref/kld.json.absent"
+    run env PATH="$D/bin:$PATH" \
+        FRANK_QWEN38_CAMPAIGN="$D/campaign" \
+        FRANK_QWEN38_RAD="$D/rad/qwen38-bf16.rad" \
+        Q38_WINDOW_CALIB_CORPUS="$D/corpora/calib.jsonl" \
+        Q38_CALIB_MIN_FREE_MIB=1 bash "$WINDOW"
+    [ "$status" -eq 2 ]
+    [[ "$output" == *"FAIL  reference"* ]]
+    [[ "$output" == *"must not take the cards"* ]]
+}
+
+@test "the calibration window's dry run prints the gated plan, including what it would bank" {
+    D="$(window_fixture)"
+    run env PATH="$D/bin:$PATH" \
+        FRANK_QWEN38_CAMPAIGN="$D/campaign" \
+        FRANK_QWEN38_RAD="$D/rad/qwen38-bf16.rad" \
+        Q38_WINDOW_CALIB_CORPUS="$D/corpora/calib.jsonl" \
+        Q38_CALIB_MIN_FREE_MIB=1 bash "$WINDOW" --dry-run
+    [ "$status" -eq 0 ]
+    [[ "$output" == *"run-qwen38-bf16.sh calib"* ]]
+    [[ "$output" == *"else kill the rig"* ]]
+    [[ "$output" == *"calib-push.py"* ]]
+    [[ "$output" == *"AGENTTRASH"* ]]
+}
+
+@test "a flat calibration round is not a failure: the tap's floor is pooled" {
+    # round 1 of a 10.03M-token corpus against a 12M floor seals nothing, and the
+    # first version of this gate aborted the run there — one round too early.
+    run python3 - "$PUSHER" <<'PY'
+import importlib.util, sys
+spec = importlib.util.spec_from_file_location("cp", sys.argv[1])
+m = importlib.util.module_from_spec(spec)
+spec.loader.exec_module(m)
+print(m.round_verdict(8192, 8192, False),   # flat: continue
+      m.round_verdict(8192, 4096, False),   # shrink: fail
+      m.round_verdict(8192, 20000, False),  # grew: continue
+      m.round_verdict(0, 0, True))          # done: stop
+PY
+    [ "$status" -eq 0 ]
+    [ "$output" = "flat shrink grew done" ]
+}
+
+@test "the calibration boot's step is its own knob; the KL modes keep production's" {
+    # Calib is the only phase whose cost is read volume per document: four 2,048-token steps
+    # re-stream each layer's routed experts four times. Measured 2026-10-29: 71 tok/s, 47 h tap.
+    run env Q38_CALIB_MNBT=8192 bash "$RUNNER" calib --dry-run
+    [ "$status" -eq 0 ]
+    grep -A1 -x -- '--max-num-batched-tokens' <<<"$output" | grep -qx '8192'
+    # and the same knob must NOT reach the measuring shape those numbers are compared in
+    run env Q38_CALIB_MNBT=8192 bash "$RUNNER" kldref --dry-run
+    [ "$status" -eq 0 ]
+    ! grep -qx '8192' <<<"$output"
+    run env Q38_CALIB_MNBT=wide bash "$RUNNER" calib --dry-run
+    [ "$status" -eq 2 ]
+    [[ "$output" == *"Q38_CALIB_MNBT must be a number"* ]]
+}
+
+# ---- the window must not mistake a dying production socket for the rig -------
+# 2026-10-30: the runner stopped frank-archon, the rig container was still loading,
+# and /health answered 200 from the production process draining its listener. The
+# window logged `health ok waited=0s`, the pusher refused on the container name, and
+# a 12M window died at second zero. The gate must prove the LISTENER is in the rig.
+
+tap_window_fixture() {
+    export W="$BATS_TEST_TMPDIR/window"
+    mkdir -p "$W/scripts" "$W/campaign/calib" "$W/campaign/kld-ref" "$W/models/rad" "$W/corpora" "$W/bin"
+    cp "$BATS_TEST_DIRNAME/../scripts/qwen38-calib-window.sh" "$W/scripts/"
+    cp "$BATS_TEST_DIRNAME/../scripts/qwen38-preflight.sh" "$W/scripts/"
+    echo "{}" > "$W/campaign/kld-ref/kld.json"
+    : > "$W/models/rad/qwen38-bf16.rad"
+    echo "{\"p\":1}" > "$W/corpora/calib.jsonl"
+    # a gram so the empty-tap check passes on the positive path
+    : > "$W/campaign/calib/gram.r0.blk.0.ffn_down_exps.0.bin"
+
+    cat > "$W/scripts/run-qwen38-bf16.sh" <<'EOS'
+#!/usr/bin/env bash
+[ -n "${STUB_RIG_MARKER:-}" ] && : > "$STUB_RIG_MARKER"
+sleep "${STUB_RIG_SLEEP:-3}"
+exit 0
+EOS
+    cat > "$W/bin/py" <<'EOS'
+#!/usr/bin/env bash
+[ -n "${STUB_PUSH_SENTINEL:-}" ] && : > "$STUB_PUSH_SENTINEL"
+exit 0
+EOS
+    cat > "$W/bin/podman" <<'EOS'
+#!/usr/bin/env bash
+case "${1:-}" in
+    inspect) echo true ;;
+    # the PREFLIGHT asks who owns :7770 (production, per LISTEN_PID); the WINDOW asks which
+    # pids belong to the rig (RIG_PID). Both stubs are the same binary, so keep them apart.
+    image)   echo "sha256:69add58f7de1ed2bca2b1e20a4cc75717557df3f73569a178eca5aafe8c44093" ;;
+    top)     echo PID
+             # the WINDOW asks "which pids are the rig's", the PREFLIGHT asks "which are
+             # production's". One stub, two answers: that separation IS the identity lesson.
+             case "${2:-}" in
+                 qwen38-bf16) echo "${RIG_PID:-424242}" ;;
+                 *)           echo "${LISTEN_PID:-424242}" ;;
+             esac ;;
+    *)       : ;;
+esac
+EOS
+    printf '#!/usr/bin/env bash\nexit 0\n' > "$W/bin/fuser"
+    cat > "$W/bin/ss" <<'EOS'
+#!/usr/bin/env bash
+echo "LISTEN 0 4096 *:7770 *:* users:((\"radiance\",pid=${LISTEN_PID:-424242},fd=9))"
+EOS
+    cat > "$W/bin/curl" <<'EOS'
+#!/usr/bin/env bash
+exit 0
+EOS
+    chmod +x "$W/scripts/run-qwen38-bf16.sh" "$W/bin/py" "$W/bin/podman" "$W/bin/ss" "$W/bin/curl" "$W/bin/fuser"
+}
+
+tap_window_env() {
+    env PATH="$W/bin:$PATH" \
+        FRANK_QWEN38_CAMPAIGN="$W/campaign" \
+        FRANK_QWEN38_RAD="$W/models/rad/qwen38-bf16.rad" \
+        Q38_WINDOW_CALIB_CORPUS="$W/corpora/calib.jsonl" \
+        Q38_CALIB_MIN_FREE_MIB=1 \
+        Q38_WINDOW_HEALTH_TIMEOUT=1 \
+        PY="$W/bin/py" \
+        STUB_RIG_MARKER="$W/rig.marker" \
+        STUB_RIG_SLEEP=0 \
+        STUB_PUSH_SENTINEL="$W/push.ran" \
+        bash "$W/scripts/qwen38-calib-window.sh"
+}
+
+@test "a health answer from a listener outside the rig is not the rig: no push, window fails" {
+    tap_window_fixture
+    # health 200, container "running", but the listener pid belongs to production
+    export LISTEN_PID=999999 RIG_PID=424242
+    run tap_window_env
+    [ "$status" -ne 0 ]
+    [ -z "$(ls "$W/push.ran" 2>/dev/null)" ]
+    [[ "$output" == *"before it ever answered /health"* ]]
+}
+
+@test "when the listener is inside the rig the window proceeds and banks the tap" {
+    tap_window_fixture
+    export LISTEN_PID=424242 RIG_PID=424242
+    run tap_window_env
+    [ "$status" -eq 0 ]
+    [ -e "$W/push.ran" ]
+    [[ "$output" == *"TAP BANKED"* ]]
+}
+
+# ---- the preflight: every fact by IDENTITY, and it must fail loudly -------------------------
+# Each case here is a lie that cost a real window on 2026-10-30.
+
+preflight_fixture() {
+    export PF="$BATS_TEST_TMPDIR/pf"
+    mkdir -p "$PF/campaign/calib" "$PF/campaign/kld-ref" "$PF/bin"
+    echo '{}' > "$PF/campaign/kld-ref/kld.json"
+    echo '{"p":1}' > "$PF/corpus.jsonl"
+    cat > "$PF/bin/podman" <<'EOS'
+#!/usr/bin/env bash
+case "$1" in
+  image) # an image carries every digest it was pulled under; the pin is one of them
+         printf '%s\n' "docker.io/stilldeadcode/radiance@sha256:4beb81eea874e3e308fbffbe1a9402a2f845c19e68ea612ba949b43bd5cb7cb0"
+         printf '%s\n' "docker.io/stilldeadcode/radiance@${STUB_IMAGE_DIGEST:-sha256:69add58f7de1ed2bca2b1e20a4cc75717557df3f73569a178eca5aafe8c44093}" ;;
+  top)   echo HPID; echo "${STUB_PROD_PIDS:-4242}" ;;
+esac
+EOS
+    cat > "$PF/bin/ss" <<'EOS'
+#!/usr/bin/env bash
+[ -n "${STUB_LISTENER_PID:-}" ] && echo "LISTEN 0 4096 *:7770 *:* users:((\"x\",pid=${STUB_LISTENER_PID},fd=9))"
+exit 0
+EOS
+    cat > "$PF/bin/fuser" <<'EOS'
+#!/usr/bin/env bash
+[ -n "${STUB_KFD_PID:-}" ] && echo "${STUB_KFD_PID}"
+exit 0
+EOS
+    chmod +x "$PF/bin/podman" "$PF/bin/ss" "$PF/bin/fuser"
+}
+
+run_preflight() {
+    ( cd "$BATS_TEST_DIRNAME/.." && env PATH="$PF/bin:$PATH" \
+        FRANK_QWEN38_CAMPAIGN="$PF/campaign" \
+        Q38_WINDOW_CALIB_CORPUS="$PF/corpus.jsonl" \
+        Q38_CALIB_MIN_FREE_MIB=1 \
+        bash scripts/qwen38-preflight.sh "$@" )
+}
+
+@test "preflight passes when every fact is true, and says so" {
+    preflight_fixture
+    run run_preflight
+    [ "$status" -eq 0 ]
+    [[ "$output" == *"PREFLIGHT PASSED"* ]]
+    [[ "$output" == *"carries sha256:69add58f"* ]]
+}
+
+@test "preflight accepts the service's own process on the port, not just any answer" {
+    preflight_fixture
+    export STUB_PROD_PIDS=4242 STUB_LISTENER_PID=4242 STUB_KFD_PID=4242
+    run run_preflight
+    [ "$status" -eq 0 ]
+    [[ "$output" == *"held by frank-archon (host pid 4242)"* ]]
+}
+
+@test "preflight fails when the tag resolves to a different digest" {
+    preflight_fixture
+    export STUB_IMAGE_DIGEST="sha256:deadbeefdeadbeef"
+    run run_preflight
+    [ "$status" -eq 2 ]
+    [[ "$output" == *"does not carry sha256:69add58f"* ]]
+}
+
+@test "preflight fails when :7770 belongs to a host pid the service does not own" {
+    preflight_fixture
+    export STUB_LISTENER_PID=999999
+    run run_preflight
+    [ "$status" -eq 2 ]
+    [[ "$output" == *":7770 is held by host pid 999999"* ]]
+}
+
+@test "preflight fails when another process holds the GPU" {
+    preflight_fixture
+    export STUB_KFD_PID=999998
+    run run_preflight
+    [ "$status" -eq 2 ]
+    [[ "$output" == *"foreign holders:"* ]]
+}
+
+@test "preflight fails when the tap dir already holds grams from another run" {
+    preflight_fixture
+    : > "$PF/campaign/calib/gram.r0.blk.0.ffn_down_exps.0.bin"
+    run run_preflight
+    [ "$status" -eq 2 ]
+    [[ "$output" == *"rotate them to .AGENTTRASH"* ]]
+    run run_preflight --allow-existing-tap
+    [ "$status" -eq 0 ]
+}
+
+@test "preflight fails when the launcher is not in the repo root" {
+    preflight_fixture
+    run env PATH="$PF/bin:$PATH" FRANK_QWEN38_CAMPAIGN="$PF/campaign" \
+        Q38_WINDOW_CALIB_CORPUS="$PF/corpus.jsonl" Q38_CALIB_MIN_FREE_MIB=1 \
+        bash -c "cd /tmp && bash '$BATS_TEST_DIRNAME/../scripts/qwen38-preflight.sh'"
+    [ "$status" -eq 2 ]
+    [[ "$output" == *"not the repo root"* ]]
+}

tests/qwen38_reference.bats

@@ -157,12 +157,15 @@ flag_value() {
     [[ "$output" != *"--vram-kv-mib"* ]]
 }
 
-@test "the calib dry run opens the tap at 12M tokens and serves on 7770" {
+@test "the calib dry run opens the tap at the 12M floor, on the latest pin, and serves on 7770" {
     run bash "$RUNNER" calib --dry-run
     [ "$status" -eq 0 ]
     flat="$(oneline)"
     [[ "$flat" == *"-e RADIANCE_CALIB_TOKENS=12000000"* ]]
     [[ "$flat" == *"-e RADIANCE_CALIB_DIR=/data/calib"* ]]
+    # the tap producer is the campaign's own pin (latest radiance), not production's
+    [[ "$flat" == *"radiance@sha256:69add58f"* ]]
+    [[ "$flat" != *"radiance@sha256:d590e300"* ]]
     [[ "$flat" == *"--port 7770"* ]]
     [[ "$flat" == *":/data/calib "* ]]
     # calib serves; it does not record