chore(media): retire the legacy Jellyfin block from dankserver/compose.yaml

The container has had one owner since the Quadlet cutover
(dankserver/jellyfin/quadlet/jellyfin.container). The block was a second owner,
and wrong on every path: jellyfin/jellyfin:latest instead of the pinned ghcr
image, /opt/jellyfin/* data roots that no longer exist, and a read-only
/mnt/entropy:/data bind that the metadata view has since replaced.

The planner no longer counts jellyfin as a compose service, so editing the
inventory cannot restart a media server; tests/deployment_scoping.bats pins that
contract for both jellyfin and plex.

AGENTS.md

@@ -26,8 +26,8 @@ merge absorbed the former `vllmScripts` repo, history preserved).
 | Target | Service | Runtime | Payload | Port |
 |--------|---------|---------|---------|------|
 | Dankserver | `frank-archon` | radiance, digest-pinned, root unit, TP=2 | publisher `qwen3.8-next-flash-fp8-iq4r-moe.rad` | 7770 |
-| Dankclaw | `frank-dragoon` | halogen-flash-server 0.16.2, digest-pinned, root unit | `qwen38-flash-next-v2-abliterated.hgn` + n-gram + vision tower | 7770 |
+| Dankclaw | `frank-dragoon` | halogen-flash-server 0.18.20, digest-pinned, root unit | `qwen38-flash-next-v2-abliterated.hgn` + n-gram + vision tower | 7770 |
-| Dankclaw | `frank-zealot` | llama.cpp PrismML fork build 10777, root unit — the only loader for this payload | Ternary-Bonsai-2-9B PQ2_0 + Q8_0 mmproj + rank-1 refusal adapter | 7771 |
+| Dankclaw | `frank-zealot` | llama.cpp PrismML fork build 12118, root unit — the only loader for this payload | Ternary-Bonsai-2-9B PQ2_0 + Q8_0 mmproj + rank-1 refusal adapter | 7771 |
 
 Payload identity, shape, eras, laws and rollback live in ONE page each:
 [[frank-archon]], [[frank-dragoon]], [[frank-zealot]]. `scripts/<service>.env`
@@ -176,11 +176,12 @@ alertmanager.container`, loopback :9093) → host-side Signal relay
 (`scripts/signal-relay.py`, user unit, 192.168.0.143:18093 — the host LAN
 IP, the one bind a container can reach as `host.containers.internal`;
 iptables admits loopback, the host and the podman bridge subnet only) → signal-cli account 318311.
-Prometheus targets read down for structural reasons (HTML-scraped "probes",
+The former structural target downs (HTML-scraped "probes", Gitea metrics off)
-Gitea metrics off); the remedy is a blackbox exporter. `frank-zealot` was down for
+are retired as of 2026-10-21: HTTP liveness rides a blackbox exporter
-a different reason until 2026-10-18: llama.cpp will not serve `/metrics` at all
+(`shared/blackbox/blackbox.yml`), Gitea serves `/metrics` from its quadlet, and
-without `--metrics`. See
+every Prometheus job reports up. `frank-zealot` was down for a different reason
-`serverAppConfigs-docs/Infrastructure/Observability.md`.
+until 2026-10-18: llama.cpp will not serve `/metrics` at all without
+`--metrics`. See `serverAppConfigs-docs/Infrastructure/Observability.md`.
 
 ## Testing
 
@@ -292,8 +293,8 @@ means off or NIC dead; a merely wedged kernel usually still answers ARP.
 2. Bench against the service itself. No scratch ports on an LLM host — one
    process owns the card. Use the runtime's own harness (`llama-bench`; a
    live llama.cpp server also reports honest `timings` per request, including
-   `draft_n`/`draft_n_accepted` — but on frank-zealot's PrismML fork build
+   `draft_n`/`draft_n_accepted` — but on frank-zealot's PrismML fork that object
-   10777 that object only reaches you on **`/v1/chat/completions`**: a greedy
+   only reaches you on **`/v1/chat/completions`** (measured on build 10777): a greedy
    `ignore_eos` run on the raw `/completion` path there answers **500 "The
    model produced output that does not match the expected Content-only
    format"** (measured 2026-10-18). halogen has no `/completion` path at all,
@@ -438,16 +439,16 @@ disagrees with it is the bug.
 ### frank-zealot — llama.cpp (PrismML fork), RTX 3090, :7771
 
 - Owns: `scripts/frank-zealot.env`, `scripts/serve-frank-zealot.sh`,
-  `scripts/frank-zealot.service`; fork build `llama-prism-b10777` is the ONLY
+  `scripts/frank-zealot.service`; fork build `llama-prism-0244744` (build 12118)
-  loader (the payload carries the fork's custom PQ2_0 tensor type plus
+  is the ONLY loader — `llama-prism-b10777` stays on disk as its rollback (the payload carries the fork's custom PQ2_0 tensor type plus
   `prism.hadamard.*` dequant metadata — stock llama.cpp refuses it, and a
   loader ignoring that metadata dequantizes wrong); payload
   `Bonsai2-9B-MTP/…PQ2_0.gguf` + the publisher's **Q8_0** mmproj (the BF16
   tower can crash the server with concurrent image requests) + the rank-1
   refusal-direction adapter applied at load by `--lora-scaled …:1`. The
   adapter's rollback is `LORA_FILE=` (empty) in the env — one line, no rebuild
-  — and build 10777 is also the gate that lets it load: older forks refuse it
+  — and the fork BUILD is also the gate that lets it load: anything older than
-  with `unsupported prism.hadamard.version: 2`.
+  b10777 refuses it with `unsupported prism.hadamard.version: 2`.
 - Shape: `--ctx-size 602112 --parallel 3` = **three conversations at
   200,704 each**, `CACHE_TYPE_K=f16` + `CACHE_TYPE_V=q8_0` (asymmetric: 25,088
   B/token, 14,406 MiB of pool), batch 6144 / ubatch 1024,
@@ -740,6 +741,13 @@ linked runbooks.
   `/etc/default/grub`. It replaced `amd_iommu=off`, and the local A/B on byte-identical traffic
   says identity mode costs nothing, so the XDNA2 NPU returned for free.
   [[dankclaw-iommu-translation-off]]
+- **A deploy handler that only `start`s a quadlet no-ops an image bump.**
+  `systemctl --user start` on an already-active unit does nothing, so installing
+  a new digest under a running service keeps the OLD container serving while the
+  deploy, its health gate and the Playwright suite all pass — the authentik
+  handler did exactly that on 2026-10-30 (2026.8.3 → 2026.8.5 moved only the
+  outpost, the one handler that restarted). A pin change is applied by
+  `restart`, never `start`.
 - **Rootless Quadlet containers run with `--replace --rm`**: `podman logs` covers only the CURRENT
   instance; history across recreations lives in `journalctl --user -u <name>.service`. Read
   rootless storage WITHOUT sudo — it is per-user storage, so `sudo podman logs` finds nothing.
@@ -799,6 +807,8 @@ per-attempt page and no per-swap ledger anywhere else.
 | `frank-tts` — cloned-voice speech, Dankclaw | `serverAppConfigs-docs/Dankclaw/Frank TTS.md` |
 | Dankclaw kernel posture (`iommu=pt` since 2026-10-18) | `serverAppConfigs-docs/LLM/dankclaw-iommu-translation-off.md` |
 | Deployment flow | `serverAppConfigs-docs/Architecture/Deployment Flow.md` |
+| Movie-library relayout (Jellyfin format, Dankserver) | `serverAppConfigs-docs/Dankserver/Media Relayout.md` |
+| SSD-backed metadata view (Jellyfin writes, Plex reads, Dankserver) | `serverAppConfigs-docs/Dankserver/Media Metadata View.md` |
 
 There is no page for a retired endpoint and no change diary. A payload swap
 appends one row to the live endpoint's **Eras** table; when the endpoint itself
@@ -840,5 +850,7 @@ history.
 | Writing a doc | **One owner per fact** — edit the endpoint/service page; everywhere else links. Never restate a number. |
 | Benchmarking (only when ordered) | **Stop and mask the service first**, bench, unmask, restart, verify health. |
 | Deleting a file | **Move it to trash** — `.AGENTTRASH/` (in-repo) or `~/.AGENTTRASH/YYYYMMDD-topic`. Never `rm`. |
+| Relayout the movie library | **Census → plan → read the exceptions → `scripts/media-apply.py`** — renames only, journal before every move; never hand-rename, never copy. |
+| Making media metadata visible to both media servers | **Bind `/var/lib/media-meta/view`** — the overlay `scripts/media-view.sh` installs (array below, NVMe above). Never write the array; never delete through the view. |
 | Tests exist | **Run them** — verify tests pass after every change. Do not commit failing tests. |
 | Real secrets | **Never commit** — host `.env` files; `.env.example` holds placeholders only. |

dankclaw/authentik/quadlet/authentik-outpost-ldap.container

@@ -23,7 +23,7 @@ Network=dankclaw_default
 # Image=ghcr.io/goauthentik/ldap:2026.8.3 (amd64 digest; podman 5.7.0 refuses
 # the tag's nested manifest list, so pin the single-platform digest).
 EnvironmentFile=/home/dank/serverAppConfigs/dankclaw/.env
-Image=ghcr.io/goauthentik/ldap@sha256:2e7b259eb8c37faac916946ddf48ec681ee9d3a9f724fe8521bf161973f13cf9
+Image=ghcr.io/goauthentik/ldap:2026.8.5@sha256:4c75bd27918418d2d6fbbfe4581bbd5a199fe38942d24559c4483edd64a1ea87
 Environment=AUTHENTIK_HOST=https://auth.franksdank.cloud
 Environment=AUTHENTIK_INSECURE=false
 Environment=AUTHENTIK_TOKEN=${AUTHENTIK_LDAP_OUTPOST_TOKEN}

dankclaw/authentik/quadlet/authentik-outpost.container

@@ -20,7 +20,7 @@ Wants=authentik-server.service
 
 [Container]
 ContainerName=authentik-outpost
-Image=ghcr.io/goauthentik/server:2026.8.3
+Image=ghcr.io/goauthentik/server:2026.8.5@sha256:602227a86455d6772e3d16b7d852713c4f864f40df5df894abd1285bb53979a5
 Exec=worker
 Network=dankclaw_default
 EnvironmentFile=/home/dank/serverAppConfigs/dankclaw/.env

dankclaw/authentik/quadlet/authentik-postgres.container

@@ -14,7 +14,7 @@ After=network.target
 
 [Container]
 ContainerName=authentik-postgres
-Image=docker.io/library/postgres:16-alpine
+Image=docker.io/library/postgres:16-alpine@sha256:0cefaabd882e42bcb2a2f3dc56d3ce4770560ba62fd7f197f1a2e008fd5731ab
 Network=dankclaw_default
 EnvironmentFile=/home/dank/serverAppConfigs/dankclaw/.env
 Environment=POSTGRES_DB=authentik

dankclaw/authentik/quadlet/authentik-server.container

@@ -19,7 +19,7 @@ Wants=authentik-postgres.service
 
 [Container]
 ContainerName=authentik-server
-Image=ghcr.io/goauthentik/server:2026.8.3
+Image=ghcr.io/goauthentik/server:2026.8.5@sha256:602227a86455d6772e3d16b7d852713c4f864f40df5df894abd1285bb53979a5
 Exec=server
 Network=dankclaw_default
 EnvironmentFile=/home/dank/serverAppConfigs/dankclaw/.env

dankclaw/authentik/quadlet/authentik-worker.container

@@ -16,7 +16,7 @@ Wants=authentik-server.service
 
 [Container]
 ContainerName=authentik-worker
-Image=ghcr.io/goauthentik/server:2026.8.3
+Image=ghcr.io/goauthentik/server:2026.8.5@sha256:602227a86455d6772e3d16b7d852713c4f864f40df5df894abd1285bb53979a5
 Exec=worker
 Network=dankclaw_default
 EnvironmentFile=/home/dank/serverAppConfigs/dankclaw/.env

dankclaw/gitea/quadlet/gitea-postgres.container

@@ -15,7 +15,7 @@ After=network.target
 [Container]
 ContainerName=gitea-postgres
 Network=gitea.network
-Image=docker.io/library/postgres:16-alpine
+Image=docker.io/library/postgres:16-alpine@sha256:0cefaabd882e42bcb2a2f3dc56d3ce4770560ba62fd7f197f1a2e008fd5731ab
 EnvironmentFile=/home/dank/Containers/gitea/env
 Environment=POSTGRES_USER=gitea
 Environment=POSTGRES_PASSWORD=${GITEA_DB_PASSWORD}

dankclaw/gitea/quadlet/gitea-server.container

@@ -19,7 +19,7 @@ After=gitea-postgres.service network.target
 [Container]
 ContainerName=gitea-server
 Network=gitea.network
-Image=docker.io/gitea/gitea:latest
+Image=docker.io/gitea/gitea:latest@sha256:30777c91280d973fa02be48b1ade735bd6859e8f6f7af3b4444d519437852504
 EnvironmentFile=/home/dank/Containers/gitea/env
 Environment=DATABASE_TYPE=postgres
 Environment=DATABASE_HOST=gitea-postgres
@@ -29,6 +29,13 @@ Environment=DATABASE_PASSWD=${GITEA_DB_PASSWORD}
 Environment=DATABASE_PORT=5432
 Environment=USER_UID=1000
 Environment=USER_GID=1000
+# Prometheus scrapes /metrics over the published LAN port (job "gitea" in
+# shared/prometheus/prometheus.yml). Off, /metrics is a 404 and the target is
+# permanently red (the audit's "disabled metrics" class). 0.0.0.0 binds inside
+# the container; the only route in is this host's published 3000, and the
+# exposed surface is counters only — same posture as the LLM /metrics.
+Environment=GITEA__metrics__ENABLED=true
+Environment=GITEA__metrics__ENABLED_HOST=0.0.0.0
 PublishPort=0.0.0.0:3000:3000
 PublishPort=2222:22
 Volume=/home/dank/Containers/gitea/data:/data

dankclaw/hermes/quadlet/chrome-devtools-protocol.container

@@ -25,7 +25,7 @@ Description=Chrome DevTools Protocol daemon (headless-shell) for hermes
 [Container]
 ContainerName=chrome-devtools-protocol
 Network=host
-Image=docker.io/chromedp/headless-shell@sha256:2d349b544a1ea6b5b5fd7c0fe99215ff662339c57407ee2e8c0a11af93516b04
+Image=docker.io/chromedp/headless-shell@sha256:7bee201478b2f6266aae01243740ee85c16281e32f42d3aed9fdf04f4eace7a1
 Entrypoint=/headless-shell/headless-shell
 Exec=--no-sandbox --disable-gpu --use-gl=angle --use-angle=swiftshader --remote-debugging-address=127.0.0.1 --remote-debugging-port=9222
 ShmSize=1073741824

dankclaw/marreta/quadlet/marreta-browser.container

@@ -23,7 +23,7 @@ After=gluetun.service network.target
 [Container]
 ContainerName=marreta_browser
 Network=gluetun.container
-Image=docker.io/lightpanda/browser:nightly
+Image=docker.io/lightpanda/browser:nightly@sha256:e72730e364d1fbf352c1149f29ffa4b278f3b1a55ff0dec3262298ebe624ec7a
 
 [Service]
 TimeoutStartSec=300

dankclaw/marreta/quadlet/marreta.container

@@ -32,7 +32,7 @@ After=gluetun.service marreta-browser.service network.target
 [Container]
 ContainerName=marreta
 Network=gluetun.container
-Image=ghcr.io/manualdousuario/marreta:latest
+Image=ghcr.io/manualdousuario/marreta:latest@sha256:30f2a34567b79b66da794b4b9423751bbbcd07ceff600d9f6a2d219e7e439362
 # Relocate off SearXNG's 8080 inside the shared netns.
 Environment=NGINX_HTTP_PORT=8085
 Environment=APP_NAME=Marreta

dankclaw/observability/quadlet/alertmanager.container

@@ -33,7 +33,7 @@ Wants=network-online.target
 
 [Container]
 ContainerName=alertmanager
-Image=docker.io/prom/alertmanager:latest
+Image=docker.io/prom/alertmanager:latest@sha256:39d8bb3966c694a124428a3a88fbcffd40bd069129ecd47c059f8a3f17c55f3b
 Network=dankclaw_default
 User=0:0
 Exec=--config.file=/etc/alertmanager/alertmanager.yml --storage.path=/alertmanager

dankclaw/observability/quadlet/blackbox-exporter.container

@@ -0,0 +1,56 @@
+# Quadlet unit: blackbox exporter (rootless, host network, boot-persistent).
+#
+# Install path on dankclaw:
+#   ~/.config/containers/systemd/blackbox-exporter.container
+#
+# WHY Network=host. The probes it runs are (a) the host's own LAN IP
+# (192.168.0.143:8081 SearXNG, :3005 OpenWebUI — host-process/portmap binds a
+# bridge container cannot reach; netavark DNATs published ports only, see
+# Container Host Reachability in the vault) and (b) ComfyUI's published port
+# via 127.0.0.1 (rootless --network=host shares the real host netns, so the
+# 0.0.0.0:8188 publish answers on loopback). A bridge-networked prober could
+# reach neither class reliably.
+#
+# WHY the listener binds 127.0.0.1 + LAN. A bridge container CANNOT dial
+# its own host's LAN IP (netavark hairpin gap, measured 2026-10-21), but a
+# bridge container dialing the podman gateway 169.254.1.2 lands on the host
+# LOOPBACK — that is the scrape path, and it needs this exporter to be
+# LISTENING ON LOOPBACK (proved: 200 from the prometheus container to
+# http://169.254.1.2:9115/-/healthy with a loopback bind). The gateway
+# address itself is NOT bindable from the host netns — it exists only
+# inside each rootless netns; binding it dies "bind: cannot assign
+# requested address" (live 2026-10-21). The LAN bind stays for diagnostics.
+# No public DNS name, no guard chain: internal net only, per the exposure
+# policy. The listen flag is --web.listen-address and it is REPEATABLE, one
+# flag per bind; plain --listen-address does not exist and the exporter
+# exits in 100 ms with "unknown long flag" (live 2026-10-21).
+
+[Unit]
+Description=Blackbox exporter (HTTP probe prober)
+After=network-online.target
+
+[Container]
+Image=docker.io/prom/blackbox-exporter@sha256:ea1c7aa3ef3f307cd03e85f34abb604478ccb90b567fc690da0cd9776da76b13
+ContainerName=blackbox-exporter
+Network=host
+# The image's ENTRYPOINT is /bin/blackbox_exporter itself (proved live
+# 2026-10-21: `podman run IMAGE sh` dies "unexpected sh"), so Exec= carries
+# FLAGS ONLY — repeating the binary makes it a positional arg and kingpin
+# exits "unexpected /bin/blackbox_exporter".
+Exec=--config.file=/etc/blackbox_exporter/config.yml --web.listen-address=127.0.0.1:9115 --web.listen-address=192.168.0.143:9115
+Volume=/opt/blackbox/blackbox.yml:/etc/blackbox_exporter/config.yml:ro
+# Plain exec form like every sibling unit. Shell form (`/bin/sh -c "…"`)
+# is passed as one quoted token and quadlet then folds the following
+# --health-interval/--health-retries (and the image name) into the command
+# string — unit died twice that way, live 2026-10-21. wget only; this
+# busybox variant has no curl, and host-network wget answers on loopback.
+HealthCmd=wget -q -O /dev/null http://127.0.0.1:9115/-/healthy
+HealthInterval=30s
+HealthRetries=3
+
+[Service]
+Restart=always
+RestartSec=5
+
+[Install]
+WantedBy=default.target

dankclaw/observability/quadlet/cadvisor.container

@@ -35,7 +35,7 @@ Wants=network-online.target
 
 [Container]
 ContainerName=cadvisor
-Image=gcr.io/cadvisor/cadvisor:v0.49.1
+Image=gcr.io/cadvisor/cadvisor:v0.49.1@sha256:3cde6faf0791ebf7b41d6f8ae7145466fed712ea6f252c935294d2608b1af388
 PodmanArgs=--pid=host --cgroupns=host
 PublishPort=127.0.0.1:8080:8080
 PublishPort=192.168.0.143:8080:8080

dankclaw/observability/quadlet/grafana.container

@@ -34,7 +34,7 @@ Wants=network-online.target prometheus.service
 
 [Container]
 ContainerName=grafana
-Image=docker.io/grafana/grafana:latest
+Image=docker.io/grafana/grafana:latest@sha256:488984b716c0862639f91a698adbe2862677a5a07bad42da5f03f3030883b205
 Network=dankclaw_default
 User=0:0
 PublishPort=127.0.0.1:3002:3000

dankclaw/observability/quadlet/node-exporter.container

@@ -33,7 +33,7 @@ Wants=network-online.target
 
 [Container]
 ContainerName=node-exporter
-Image=docker.io/prom/node-exporter:latest
+Image=docker.io/prom/node-exporter:latest@sha256:1b4e4438faca4dd7e001dd445d161a4a2091b0fededa84093b3a8dfeae1f1be0
 Network=host
 PodmanArgs=--pid=host
 Exec=--path.procfs=/host/proc --path.sysfs=/host/sys --path.rootfs=/rootfs --web.listen-address=127.0.0.1:9100 --web.listen-address=192.168.0.143:9100

dankclaw/observability/quadlet/prometheus.container

@@ -43,8 +43,16 @@ Wants=network-online.target
 
 [Container]
 ContainerName=prometheus
-Image=docker.io/prom/prometheus:latest
+Image=docker.io/prom/prometheus:latest@sha256:fa3302279710be2f21a8301ec17a8b4f2301fd829bd113839b3b6b02b6f191a2
 Network=dankclaw_default
+# Same-host scrape targets are dialed at host.containers.internal (the podman
+# gateway): a bridge container cannot dial its own host's LAN IP (netavark
+# hairpin gap, measured 2026-10-21), and traffic to the gateway lands on host
+# loopback / published ports (cadvisor/gpu jobs and the blackbox scrape all
+# ride it). The name resolves natively — NO HostAddr key: quadlet 5.7.0 calls
+# it unsupported and the WHOLE user quadlet batch then fails to generate,
+# silently orphaning every other unit (live 2026-10-21: prometheus.service
+# vanished while its container kept running as an orphan).
 User=0:0
 Exec=--config.file=/etc/prometheus/prometheus.yml --storage.tsdb.path=/prometheus --storage.tsdb.retention.time=30d --web.enable-lifecycle
 PublishPort=127.0.0.1:9090:9090

dankclaw/open-webui/quadlet/open-webui.container

@@ -27,7 +27,7 @@ After=network.target
 [Container]
 ContainerName=open-webui
 Network=open-webui.network
-Image=ghcr.io/open-webui/open-webui:main
+Image=ghcr.io/open-webui/open-webui:main@sha256:0b399a8ba34558e181d316750455f2a95376a38cd60a61b9c942e138898944bd
 PublishPort=3005:8080
 Environment=OPENAI_API_BASE_URL=http://192.168.0.143:7770
 Environment=ANONYMIZED_TELEMETRY=False

dankclaw/pihole/quadlet/pihole.container

@@ -24,7 +24,7 @@ Wants=network-online.target
 
 [Container]
 ContainerName=pihole
-Image=docker.io/pihole/pihole:latest
+Image=docker.io/pihole/pihole:latest@sha256:d34134716fc32038b052a8674c7c6879e0ff21f0e6d8af345166fffa2c27c113
 Network=slirp4netns
 PublishPort=0.0.0.0:53:53/udp
 PublishPort=0.0.0.0:53:53/tcp

dankclaw/searxng/quadlet/gluetun.container

@@ -16,7 +16,7 @@ After=searxng-network.service network.target
 [Container]
 ContainerName=gluetun
 Network=searxng.network
-Image=docker.io/qmcgaw/gluetun:latest
+Image=docker.io/qmcgaw/gluetun:latest@sha256:4af0c804003481e100180e1f8d4d5e9c48163fedc0f88d061881b615aef9d183
 AddCapability=NET_ADMIN
 AddDevice=/dev/net/tun:/dev/net/tun
 Environment=VPN_SERVICE_PROVIDER=${VPN_SERVICE_PROVIDER}

dankclaw/searxng/quadlet/searxng-valkey.container

@@ -16,7 +16,7 @@ After=searxng-network.service network.target
 ContainerName=searxng-valkey
 Network=searxng.network
 NetworkAlias=valkey
-Image=docker.io/valkey/valkey:9-alpine
+Image=docker.io/valkey/valkey:9-alpine@sha256:bf09d58e96d6b4524a15396b65b242bbe18beebbcc15a04d8ee75ddac62fae0b
 Volume=/opt/searxng/valkey:/data
 
 [Service]

dankclaw/searxng/quadlet/searxng.container

@@ -16,7 +16,7 @@ After=gluetun.service searxng-valkey.service network.target
 [Container]
 ContainerName=searxng
 Network=gluetun.container
-Image=ghcr.io/searxng/searxng:latest
+Image=ghcr.io/searxng/searxng:latest@sha256:f4aa660c54d062dedb268be84e821ea24e39f48c7a496af8dd02cc9eedb93847
 Environment=SEARXNG_SECRET=${SEARXNG_SECRET}
 Environment=SEARXNG_BASE_URL=${SEARXNG_BASE_URL}
 Volume=/opt/searxng/config:/etc/searxng

dankclaw/seerr/quadlet/seerr.container

@@ -29,7 +29,7 @@ Wants=network-online.target
 
 [Container]
 ContainerName=seerr
-Image=ghcr.io/seerr-team/seerr:v3.4.1
+Image=ghcr.io/seerr-team/seerr:v3.5.1@sha256:d8fab675b55ad293ded98fd71efcfd46054cbad9746534ae1cc4794a88c0b3be
 Network=dankclaw_default
 PublishPort=127.0.0.1:5055:5055
 Volume=/home/dank/Containers/seerr/data:/app/config

dankserver/calibre-server/quadlet/calibre-server.container

@@ -43,7 +43,7 @@ After=network.target
 
 [Container]
 ContainerName=calibre-server
-Image=lscr.io/linuxserver/calibre:9.15.0
+Image=lscr.io/linuxserver/calibre:9.17.0@sha256:b198fc03a6abc6be7e251a9958e50e74634b00e447c7b25e92c191c8077ba378
 Network=pasta
 EnvironmentFile=/home/dank/Containers/calibre-server/env
 Environment=PUID=1000

dankserver/calibre-web/quadlet/calibre-web.container

@@ -16,7 +16,7 @@ After=network.target
 
 [Container]
 ContainerName=calibre-web
-Image=lscr.io/linuxserver/calibre-web:0.6.27-ls402
+Image=lscr.io/linuxserver/calibre-web:0.6.27-ls407@sha256:470b4ed9a7d6ea1475fef17dbcf957cb60880910296d226bd0dca652c05f68ca
 Network=pasta
 Environment=DOCKER_MODS=linuxserver/mods:universal-calibre
 Environment=LANG=en_US.UTF-8

dankserver/compose.yaml

@@ -94,34 +94,12 @@ services:
       timeout: 10s
       retries: 3
 
-  # ═══════════════════════════════════════════════
+  # Jellyfin left this file when it became a Quadlet: the container is owned by
-  # Jellyfin — media server (movies, TV, music)
+  # dankserver/jellyfin/quadlet/jellyfin.container, its data roots are under
-  # ═══════════════════════════════════════════════
+  # /home/dank/Containers/jellyfin, and its library arrives through the SSD
-  jellyfin:
+  # overlay at /var/lib/media-meta/view (scripts/media-view.sh). The block that
-    image: jellyfin/jellyfin:latest
+  # was here named /opt/jellyfin/* and a read-only array bind — a second owner for
-    container_name: jellyfin
+  # the container name `jellyfin`, wrong on every path. One owner per container.
-    restart: unless-stopped
-    ports:
-      - "8096:8096/tcp"
-      - "8920:8920/tcp"
-      - "7359:7359/udp"
-      - "1900:1900/udp"
-    environment:
-      - JELLYFIN_PublishedServerUrl=http://192.168.0.235:8096
-      - DOTNET_SYSTEM_IO_DISABLEFILELOCKING=true
-    volumes:
-      - /opt/jellyfin/config:/config
-      - /opt/jellyfin/cache:/cache
-      - /mnt/entropy:/data:ro
-    devices:
-      - /dev/dri:/dev/dri
-    group_add:
-      - "992"
-    healthcheck:
-      test: ["CMD", "curl", "-f", "http://localhost:8096/health"]
-      interval: 30s
-      timeout: 10s
-      retries: 3
 
   # ═══════════════════════════════════════════════
   # Calibre-Web — e-book web frontend

dankserver/copyparty/quadlet/copyparty.container

@@ -21,7 +21,13 @@ After=network.target
 
 [Container]
 ContainerName=copyparty
-Image=ghcr.io/9001/copyparty-ac:latest
+# Digest-pinned: upstream stopped publishing version tags after 1.19.15, so
+# the pin rides :latest@digest (the tag in the reference is cosmetic; the
+# digest is the pull). Pinned 2026-10-21 at the build whose label is
+# 1.20.26 - closes GHSA-3c88-p299-7hvr (1.20.25) and GHSA-hg99-jqfp-rf76
+# (1.20.26); the previously running build was the rolling :latest at
+# 1.20.24 (issue #73 on command-center).
+Image=ghcr.io/9001/copyparty-ac:latest@sha256:7f77459160a4bf34fdca79f2965dff4f0c1d7069a3d9cdfb3632acb041dd2f6a
 Network=dankserver_appnet
 EnvironmentFile=/home/dank/Containers/copyparty/env
 Exec=-a ${COPAY_ACCOUNT}:${COPAY_PASSWORD} -v /w::r:ra,dank

dankserver/host/systemd/media-meta-view.mount

@@ -0,0 +1,30 @@
+# /etc/systemd/system/var-lib-media\x2dmeta-view.mount
+# (repo copy: dankserver/host/systemd/media-meta-view.mount — systemd names a
+# mount unit after its ESCAPED path, so the host file carries the \x2d spelling)
+#
+# Installed by `scripts/media-view.sh install` — the repo copy is the source of
+# truth (fleet law: never edit host state by hand).
+#
+# An OverlayFS merge that shows the media library as one tree while splitting
+# where its bytes live: lowerdir is the RAID6 array (the source of truth, never
+# written), upperdir is a directory on the NVMe that receives every write. The
+# two media servers bind the result at /var/lib/media-meta/view, so Jellyfin's
+# sidecars and artwork are SSD bytes — which is what makes them fast to browse
+# and what keeps Plex's NFO agent off the spinners — while the media itself stays
+# on the array untouched.
+#
+# RequiresMountsFor is load-bearing: without the array mounted, lowerdir is an
+# empty directory and the view would present an empty library as truth.
+
+[Unit]
+Description=SSD-backed overlay view of the media library (array + NVMe metadata layer)
+RequiresMountsFor=/mnt/entropy
+
+[Mount]
+What=overlay
+Where=/var/lib/media-meta/view
+Type=overlay
+Options=lowerdir=/mnt/entropy,upperdir=/var/lib/media-meta/upper,workdir=/var/lib/media-meta/work
+
+[Install]
+WantedBy=local-fs.target

dankserver/jellyfin/quadlet/jellyfin.container

@@ -4,8 +4,9 @@
 # Unit name: jellyfin.service (podman 5.x drops the container- prefix)
 #
 # Data roots migrated from /opt/jellyfin to /home/dank/Containers/jellyfin at
-# cutover. All of /mnt/entropy is mounted read-only at /data: nothing automated
+# cutover. The library is seen through /var/lib/media-meta/view, the SSD-backed
-# anywhere writes to the entropy volume. Every
+# overlay view (scripts/media-view.sh): the array stays read-only and untouched,
+# while the sidecars this server writes land on the NVMe. Every
 # creation-time environment variable of the compose-era container is kept as
 # an Environment= line (zero drift); none of them are secrets, so this unit
 # needs no EnvironmentFile.
@@ -16,7 +17,7 @@ After=network.target
 
 [Container]
 ContainerName=jellyfin
-Image=ghcr.io/jellyfin/jellyfin:latest
+Image=ghcr.io/jellyfin/jellyfin:latest@sha256:fea863d0cd9ec445985d9c75fc6ed64e38a1d2766cf0c250a33d553f4fc1b771
 Network=dankserver_appnet
 Environment=DEBIAN_FRONTEND=noninteractive
 Environment=DOTNET_SYSTEM_IO_DISABLEFILELOCKING=true
@@ -41,7 +42,12 @@ PublishPort=7359:7359/udp
 PublishPort=1900:1900/udp
 Volume=/home/dank/Containers/jellyfin/config:/config
 Volume=/home/dank/Containers/jellyfin/cache:/cache
-Volume=/mnt/entropy:/data:ro
+# The view, writable: Jellyfin is the metadata writer (movie.nfo / tvshow.nfo /
+# episode.nfo + artwork) and those bytes belong on the SSD. The condition is the
+# fail-closed guard — starting against a plain, unmounted view directory would let
+# a scan conclude the library is empty.
+Volume=/var/lib/media-meta/view:/data
+UnitConfigItem=ConditionPathIsMountPoint=/var/lib/media-meta/view
 AddDevice=/dev/dri
 GroupAdd=keep-groups
 HealthCmd=curl -f http://localhost:8096/health

dankserver/observability/quadlet/cadvisor.container

@@ -18,7 +18,7 @@ Wants=network-online.target
 
 [Container]
 ContainerName=cadvisor
-Image=gcr.io/cadvisor/cadvisor:v0.49.1
+Image=gcr.io/cadvisor/cadvisor:v0.49.1@sha256:3cde6faf0791ebf7b41d6f8ae7145466fed712ea6f252c935294d2608b1af388
 PodmanArgs=--pid=host --cgroupns=host
 PublishPort=127.0.0.1:8080:8080
 PublishPort=192.168.0.235:8080:8080

dankserver/observability/quadlet/node-exporter.container

@@ -37,7 +37,7 @@ Wants=network-online.target
 
 [Container]
 ContainerName=node-exporter
-Image=docker.io/prom/node-exporter:latest
+Image=docker.io/prom/node-exporter:latest@sha256:1b4e4438faca4dd7e001dd445d161a4a2091b0fededa84093b3a8dfeae1f1be0
 Network=host
 PodmanArgs=--pid=host
 Exec=--path.procfs=/host/proc --path.sysfs=/host/sys --path.rootfs=/rootfs --web.listen-address=127.0.0.1:9100 --web.listen-address=192.168.0.235:9100

dankserver/plex/quadlet/plex.container

@@ -4,14 +4,19 @@ After=network-online.target
 Wants=network-online.target
 
 [Container]
-Image=lscr.io/linuxserver/plex:1.43.5.11029-6bf8af3f0-ls329
+Image=lscr.io/linuxserver/plex:1.43.5.11079-b3b13e506-ls330@sha256:01a53d9c7adcbb3701fbb0a4230c0a37c85453759b6b05a67356c3c7a09de08d
 ContainerName=plex
 Network=dankserver_appnet:alias=plex
 Environment=PUID=0
 Environment=PGID=0
 Environment=TZ=America/Los_Angeles
 Volume=/var/lib/plexmediaserver:/config
-Volume=/mnt/entropy:/mnt/entropy:ro
+# The container path stays /mnt/entropy on purpose: every path Plex recorded in
+# its own database is written against it, so swapping the HOST source of this bind
+# to the overlay view re-points Plex at the sidecars without touching its DB.
+# Read-only: Plex consumes the NFO files Jellyfin writes and writes no metadata.
+Volume=/var/lib/media-meta/view:/mnt/entropy:ro
+UnitConfigItem=ConditionPathIsMountPoint=/var/lib/media-meta/view
 PublishPort=32400:32400
 
 [Service]

dankserver/romm/quadlet/romm-mariadb.container

@@ -14,7 +14,7 @@ After=network.target
 
 [Container]
 ContainerName=romm-mariadb
-Image=docker.io/library/mariadb:11
+Image=docker.io/library/mariadb:11@sha256:c4124d1afa875e47f2892edf26b482fa54f796e2f342d3717a04ca091a22a9bd
 Network=dankserver_appnet:alias=mariadb
 EnvironmentFile=/home/dank/Containers/romm/env
 Environment=MARIADB_DATABASE=romm

dankserver/romm/quadlet/romm-redis.container

@@ -12,7 +12,7 @@ After=network.target
 
 [Container]
 ContainerName=romm-redis
-Image=docker.io/library/redis:7-alpine
+Image=docker.io/library/redis:7-alpine@sha256:8da19c7399dca900c6822a1cf30f9d2285585e2c4b457ff4095b0f4500054d3f
 Network=dankserver_appnet:alias=redis
 Volume=/home/dank/Containers/romm/redis:/data
 HealthCmd=redis-cli ping

dankserver/romm/quadlet/romm.container

@@ -18,7 +18,7 @@ After=romm-mariadb.service romm-redis.service network.target
 
 [Container]
 ContainerName=romm
-Image=ghcr.io/rommapp/romm:latest
+Image=ghcr.io/rommapp/romm:latest@sha256:8677fb7b81b1d3db20b849e989a1dadc2a11159e51eb34a2acdcb01102274edf
 Network=dankserver_appnet
 EnvironmentFile=/home/dank/Containers/romm/env
 Environment=DB_HOST=mariadb

dankserver/tautulli/quadlet/tautulli.container

@@ -4,7 +4,7 @@ After=network-online.target
 Wants=network-online.target
 
 [Container]
-Image=lscr.io/linuxserver/tautulli:v2.18.1-ls245
+Image=lscr.io/linuxserver/tautulli:v2.18.2-ls250@sha256:529c7eabcc40554c1dc48f33f1f3cc3b1438462d73e0ec2dac8ac0b66aeeb8d4
 ContainerName=tautulli
 Network=dankserver_appnet:alias=tautulli
 Environment=PUID=0

frankbook-pro/gluetun/quadlet/gluetun.container

@@ -19,7 +19,7 @@ Description=Gluetun VPN tunnel (FrankBook-Pro podman machine)
 
 [Container]
 ContainerName=gluetun
-Image=docker.io/qmcgaw/gluetun:latest
+Image=docker.io/qmcgaw/gluetun:latest@sha256:c2ee4191762fca6b6a3c3359a731198a95d9c9af4db32d466e223bf3a84e779b
 AddCapability=NET_ADMIN
 AddDevice=/dev/net/tun:/dev/net/tun
 Environment=VPN_SERVICE_PROVIDER=${VPN_SERVICE_PROVIDER}

frankbook-pro/lidarr/quadlet/lidarr.container

@@ -15,7 +15,7 @@ After=gluetun.service
 
 [Container]
 ContainerName=lidarr
-Image=lscr.io/linuxserver/lidarr:latest
+Image=lscr.io/linuxserver/lidarr:latest@sha256:d8549ff9ea29b4015c32257360652c1000c31ea471a2b3f0835ae77292faf35b
 Environment=PUID=0
 Environment=PGID=0
 Environment=TZ=America/Los_Angeles

frankbook-pro/prowlarr/quadlet/prowlarr.container

@@ -24,7 +24,7 @@ After=gluetun.service
 
 [Container]
 ContainerName=prowlarr
-Image=lscr.io/linuxserver/prowlarr:latest
+Image=lscr.io/linuxserver/prowlarr:latest@sha256:b3b00047ed20f6ce2b16780145cb6a0acb51c9e28ea6aa6f5dcd33c7034f1797
 Environment=PUID=0
 Environment=PGID=0
 Environment=TZ=America/Los_Angeles

frankbook-pro/qbittorrent/quadlet/qbittorrent.container

@@ -29,7 +29,7 @@ Network=gluetun.container
 # Pinned to the last 4.x line: Readarr's retired qB client requires the
 # qB 4.x login response (HTTP 200, body "Ok."); qB 5.x answers 204 with an
 # empty body, which fails Readarr's auth check. All other clients accept 4.x.
-Image=lscr.io/linuxserver/qbittorrent:4.6.7
+Image=lscr.io/linuxserver/qbittorrent:4.6.7@sha256:55f15d44396315551f87294a176efae733b16e283f38980308e46073950257c6
 Environment=PUID=0
 Environment=PGID=0
 Environment=TZ=America/Los_Angeles

frankbook-pro/radarr/quadlet/radarr.container

@@ -15,7 +15,7 @@ After=gluetun.service
 
 [Container]
 ContainerName=radarr
-Image=lscr.io/linuxserver/radarr:latest
+Image=lscr.io/linuxserver/radarr:latest@sha256:fc0ae3c50f8128b97e84381903ec8d6845c3a569641cb7117c124376cccb57c1
 Environment=PUID=0
 Environment=PGID=0
 Environment=TZ=America/Los_Angeles

frankbook-pro/readarr/quadlet/readarr.container

@@ -19,7 +19,7 @@ After=gluetun.service
 
 [Container]
 ContainerName=readarr
-Image=lscr.io/linuxserver/readarr:amd64-1.0.980
+Image=lscr.io/linuxserver/readarr:amd64-1.0.980@sha256:a95e82dbb0eb2bb43de4c150f1dab9dd821306c67ea0b418983a957e731f2b61
 Environment=PUID=0
 Environment=PGID=0
 Environment=TZ=America/Los_Angeles

frankbook-pro/sonarr/quadlet/sonarr.container

@@ -16,7 +16,7 @@ After=gluetun.service
 
 [Container]
 ContainerName=sonarr
-Image=lscr.io/linuxserver/sonarr:latest
+Image=lscr.io/linuxserver/sonarr:latest@sha256:889dcc58c6aea859033985894b519b2e6b1e55d92527ed123f9f52e42ae7c97b
 Environment=PUID=0
 Environment=PGID=0
 Environment=TZ=America/Los_Angeles

scripts/apply-protected-cast.sh

@@ -1,27 +1,45 @@
 #!/usr/bin/env bash
-# apply-protected-cast.sh — RUNS ON DANKSERVER. Rewrite the recipe's
+# apply-protected-cast.sh — RUNS ON DANKSERVER. Rewrite a recipe's
-# PROTECTED-BEGIN/END section from Swift's OWN calibration grams: run
+# PROTECTED-BEGIN/END section from the campaign's OWN calibration grams: run
 # derive-protected-experts.py over the calib dir, splice its cast lines into
 # the marked section of the recipe in place, keep the pre-edit recipe in the
-# campaign trash. Refuses to run twice (marker text says derived).
+# campaign trash. Refuses to run twice (marker text says derived), and refuses
+# when a requested target count was not reached. Run it on a COPY of the
+# template so the committed recipe stays the clean, reviewable baseline.
 set -euo pipefail
-CALIB=${1:-/home/dank/corpora/calib}
+SELF_DIR="$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)"
-RECIPE=${2:-/home/dank/serverAppConfigs-swift15/scripts/recipes/qwen4exp-swift15-w4dn8-i8-hc8m.recipe}
+CALIB=${1:-/home/dank/campaign/qwen38-w5nl64/calib}
-PY=/home/dank/venvs/radcorpus/bin/python
+RECIPE=${2:-$SELF_DIR/recipes/qwen4exp-qwen38-w5nl64-bf16trunk.recipe}
-DERIVE=/home/dank/serverAppConfigs-swift15/scripts/derive-protected-experts.py
+TARGET_COUNT=${3:-0}
+PY=${PY:-/home/dank/venvs/radcorpus/bin/python}
+DERIVE=${DERIVE:-$SELF_DIR/derive-protected-experts.py}
 TS=$(date +%Y%m%d-%H%M%S)
+# OUT and TRASH_DIR are overridable so the seam can drive the whole script in a
+# scratch directory; on the box they are the campaign's usual places.
+OUT=${OUT:-/tmp/protected-cast-$TS.txt}
+TRASH_DIR=${TRASH_DIR:-/home/dank/.AGENTTRASH}
 
 grep -q "derive-protected-experts.py rewrote" "$RECIPE" && {
     echo "recipe already carries derived cast lines; refusing to re-derive" >&2; exit 1; }
 grep -q "PROTECTED-BEGIN" "$RECIPE" || { echo "no PROTECTED markers in $RECIPE" >&2; exit 1; }
 
-"$PY" "$DERIVE" "$CALIB" > "/tmp/protected-cast-$TS.txt"
+ARGS=("$DERIVE" "$CALIB")
-N=$(grep -c 'cast dtype=bf16' "/tmp/protected-cast-$TS.txt")
+if [ "$TARGET_COUNT" -gt 0 ]; then
-[ "$N" -ge 2 ] || { echo "derive produced $N cast lines; refusing" >&2; exit 1; }
+    ARGS+=(--target-count "$TARGET_COUNT")
-echo "derived $N protected experts -> /tmp/protected-cast-$TS.txt"
+fi
+"$PY" "${ARGS[@]}" > "$OUT"
+N=$(grep -c 'cast dtype=bf16' "$OUT")
+[ "$N" -ge 2 ] || { echo "derive produced $N cast lines; refusing (see $OUT)" >&2; exit 1; }
+if [ "$TARGET_COUNT" -gt 0 ] && [ "$N" -lt "$TARGET_COUNT" ]; then
+    echo "derive reached $N protected experts, target was $TARGET_COUNT; refusing to convert" >&2
+    echo "       (the bar sweep bottomed out: the recording is too thin or the target too high)" >&2
+    exit 1
+fi
+echo "derived $N protected experts -> $OUT"
 
-cp -f "$RECIPE" "/home/dank/.AGENTTRASH/$TS-qwen4exp-swift15-w4dn8-i8-hc8m.recipe"
+mkdir -p "$TRASH_DIR"
-"$PY" - "$RECIPE" "/tmp/protected-cast-$TS.txt" <<'PYEOF'
+cp -f "$RECIPE" "$TRASH_DIR/$TS-$(basename "$RECIPE")"
+"$PY" - "$RECIPE" "$OUT" <<'PYEOF'
 import sys
 recipe, castfile = sys.argv[1], sys.argv[2]
 lines = open(recipe).read().splitlines()
@@ -35,7 +53,7 @@ for l in lines:
     elif 'PROTECTED-END' in l:
         insec = False
         out.append(l)
-        out.append('# (derive-protected-experts.py rewrote this section from Swift\'s own grams)')
+        out.append('# (derive-protected-experts.py rewrote this section from the campaign grams)')
     elif not insec:
         out.append(l)
 open(recipe, 'w').write('\n'.join(out) + '\n')

scripts/build-qwen38-bf16-rad.sh

@@ -0,0 +1,220 @@
+#!/usr/bin/env bash
+# build-qwen38-bf16-rad.sh — RUNS ON DANKSERVER. rad-convert the vendor BF16
+# checkpoint /home/dank/models/Qwen3.8-Flash-Next into
+# /home/dank/models/rad/qwen38-bf16.rad with NO recipe and NO --reuse: a weight
+# no rule matches keeps the checkpoint's encoding, so every tensor stays the
+# bf16 it is in the safetensors and the container is the checkpoint's own
+# numbers behind a .rad header.
+#
+# This file is the campaign's measuring stick, twice over: scripts/
+# run-qwen38-bf16.sh kldref records the KLD reference logits from it (every
+# arm's --kld-ref is scored against that record) and the same container serves
+# with the calibration tap to write the Grams that derive the protected-expert
+# list and feed GPTQ. A reference that is not the vendor checkpoint, byte for
+# byte in bf16, silently moves every gate in the campaign — hence no recipe, no
+# --reuse, and the four refusal gates below.
+#
+# Why a container at all: a checkpoint dir has no mapped/file tiers, and a
+# weight only a host kernel reads (the 47.7 GiB PLE n-gram table) cannot be
+# served from one; rad-convert also bakes the tokenizer. The convert runs on
+# the HOST backend — no --device /dev/kfd, no /dev/dri — so declare resolves
+# kernels on CPU and production serving is untouched for the whole run (the
+# precedent that proved it: scripts/convert-swift15-w4dn8.sh).
+#
+# Gates, every one a refusal with exit 2 and a named reason, BEFORE anything is
+# written, cheapest first — the verifier runs last of them because its Hub gate
+# hashes 343 GiB and must not be paid for by a run that was going to refuse on
+# the output file anyway.
+#   checkpoint  the tree must be on disk; an absent dir names the command that
+#               makes one (scripts/download-qwen38-bf16.sh --fetch) and stops.
+#   free space  the convert writes ~343,323 MiB next to the ~343,323 MiB
+#               checkpoint; QWEN38_MIN_FREE_MIB (default 350000 = that figure
+#               plus 6.5 GiB of slack) is the floor. The volume that cannot
+#               hold both is discovered at minute 2, not hour 5.
+#   output      the campaign converts the reference ONCE. An existing
+#               qwen38-bf16.rad is refused; --force retires it to
+#               ~/.AGENTTRASH/<utc>-qwen38-bf16.rad (trash law: nothing is
+#               deleted, and the move is a rename on the same volume).
+#   verified    scripts/download-qwen38-bf16.sh's verify mode must pass. A
+#               failed verify exits 1, not 2: the tree being the wrong payload
+#               is a result, a missing environment is a refusal.
+#   plan        rad-convert --plan-only -v first — the oracle for "does this
+#               shape boot". A refused plan costs seconds and saves a convert.
+#
+# usage: scripts/build-qwen38-bf16-rad.sh [--dry-run] [--plan-only] [--force]
+#   --dry-run    print the argv it would exec (podman wrapping rad-convert),
+#               one shell-quoted argument a line, and exit 0. No gate runs, no
+#               container starts, nothing is written — the offline test seam.
+#               It does resolve the checkpoint and output paths through the
+#               /models mount, because an argv that carries host paths would
+#               not run either.
+#   --plan-only  run the plan and every gate except the disk and output ones —
+#               those two protect bytes the plan never writes, and this is the
+#               probe to run when the disk gate itself is what refuses
+#   --force      retire an existing output container to the trash dir first
+#
+# knobs: FRANK_QWEN38_MODELS   the host dir mounted at /models (default
+#        /home/dank/models); FRANK_QWEN38_CKPT (default $MODELS/
+#        Qwen3.8-Flash-Next, the same knob the verifier reads);
+#        FRANK_QWEN38_BF16_OUT (default $MODELS/rad/qwen38-bf16.rad);
+#        QWEN38_MIN_FREE_MIB; QWEN38_TRASH (default $HOME/.AGENTTRASH). The
+#        verifier's own census knobs (FRANK_QWEN38_SHARDS / _BYTES / _REPO)
+#        reach it through the environment untouched.
+# exit: 0 converted (or planned), 1 the checkpoint failed verification,
+#       2 refused before anything was written
+set -euo pipefail
+
+SCRIPT_DIR="$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)"
+# ONE owner of the engine pin: the digest frank-archon serves on. The reference
+# must be built by the SAME engine build that scores it, or a KLD delta is
+# partly a version delta.
+# shellcheck disable=SC1091
+source "$SCRIPT_DIR/frank-archon.env"
+IMAGE="${FRANK_ARCHON_IMAGE:?FRANK_ARCHON_IMAGE must name the radiance image}"
+
+MODELS="${FRANK_QWEN38_MODELS:-/home/dank/models}"
+CKPT="${FRANK_QWEN38_CKPT:-$MODELS/Qwen3.8-Flash-Next}"
+OUT="${FRANK_QWEN38_BF16_OUT:-$MODELS/rad/qwen38-bf16.rad}"
+TRASH="${QWEN38_TRASH:-$HOME/.AGENTTRASH}"
+MIN_FREE_MIB="${QWEN38_MIN_FREE_MIB:-350000}"
+LOG="/home/dank/logs/qwen38_bf16_convert.log"
+
+DRY=0
+PLAN_ONLY=0
+FORCE=0
+while [ $# -gt 0 ]; do
+    case "$1" in
+        --dry-run)   DRY=1 ;;
+        --plan-only) PLAN_ONLY=1 ;;
+        --force)     FORCE=1 ;;
+        *) echo "usage: ${0##*/} [--dry-run] [--plan-only] [--force]" >&2
+           exit 2 ;;
+    esac
+    shift
+done
+
+# The convert sees the tree only through the /models mount, so both paths must
+# be under it; the container-side spelling is what goes into the argv.
+case "$CKPT" in
+    "$MODELS"/*) CKPT_IN="/models/${CKPT#"$MODELS"/}" ;;
+    *) echo "REFUSED: checkpoint $CKPT is not under the mounted models dir $MODELS — the convert cannot see it" >&2
+       exit 2 ;;
+esac
+case "$OUT" in
+    "$MODELS"/*) OUT_IN="/models/${OUT#"$MODELS"/}" ;;
+    *) echo "REFUSED: output $OUT is not under the mounted models dir $MODELS — rad-convert could not write it" >&2
+       exit 2 ;;
+esac
+
+# Declare with NO recipe line — that is what keeps every tensor bf16 — and at the
+# bound this container will actually be served at. Both flags come off the pinned
+# image's own help: --kv-cache-dtype at convert "only picks the kernels it
+# checks" and the container is identical either way, so naming fp8 is free
+# insurance — if the fp8 KV kernels the fleet serves would not resolve, the
+# convert refuses here instead of the boot failing on a drained window.
+# --max-ctx is the bound declare is run at, whose default is the checkpoint's
+# training context (262,144 here, whose fp8 cache is 258.8 GiB against a 192 GiB
+# machine, so the default is not an option). 32,768 is set by the corpus, not by
+# taste: kld.jsonl's longest record is 38,263 characters — ~9,600 tokens at the
+# pessimistic chars/4, ~7,700 at the chars/4.97 this repo measured on the K3
+# corpus — so the window the rig boots at clears every record 2.6x over and the
+# reference boot cannot die on a sequence that will not fit. The rig's window is
+# one knob (FRANK_QWEN38_MAX_MODEL_LEN in scripts/run-qwen38-bf16.sh, default
+# 32,768) and that script refuses a corpus its window cannot hold, so raising one
+# without the other is caught before the drained window, not inside it. The
+# payload containers the gate scores declare at the training context because
+# production must hold the 200k shape; a declared bound is a declare-time check,
+# not a weight change, so the reference and the payloads carry the same numbers
+# whatever each declared at — which is why one can be measured against the other.
+CONVERT_ARGS=("$CKPT_IN" -o "$OUT_IN" --max-ctx 32768 --kv-cache-dtype fp8)
+PODMAN=(podman run --rm --entrypoint rad-convert -v "$MODELS:/models")
+PLAN_CMD=("${PODMAN[@]}" --name qwen38-bf16-plan "$IMAGE" "${CONVERT_ARGS[@]}" --plan-only -v)
+CONV_CMD=("${PODMAN[@]}" --name qwen38-bf16-convert "$IMAGE" "${CONVERT_ARGS[@]}")
+
+if [ "$DRY" = "1" ]; then
+    echo "dry run: no gate checked, no container started, nothing written"
+    echo "# $CKPT -> $OUT (the convert sees them as $CKPT_IN and $OUT_IN)"
+    echo "# plan argv:"
+    printf '%q\n' "${PLAN_CMD[@]}"
+    echo "# convert argv:"
+    printf '%q\n' "${CONV_CMD[@]}"
+    exit 0
+fi
+
+# ---- gate: the checkpoint is on disk ---------------------------------------
+if [ ! -d "$CKPT" ]; then
+    echo "REFUSED: no checkpoint at $CKPT — fetch and verify it first: scripts/download-qwen38-bf16.sh --fetch" >&2
+    exit 2
+fi
+
+# ---- gate: peak disk --------------------------------------------------------
+# The deepest existing ancestor is the right thing to ask: a fresh
+# /home/dank/models/rad has no free space of its own to report.
+OUT_DIR="$(dirname "$OUT")"
+PROBE="$OUT_DIR"
+while [ ! -d "$PROBE" ] && [ "$PROBE" != "/" ]; do PROBE="$(dirname "$PROBE")"; done
+FREE_MIB="$(df --output=avail -m "$PROBE" | tail -1 | tr -d ' ')"
+if ! [[ "$FREE_MIB" =~ ^[0-9]+$ ]]; then
+    echo "REFUSED: cannot read the free space under $PROBE (df said: $FREE_MIB)" >&2
+    exit 2
+fi
+if [ "$PLAN_ONLY" != "1" ] && [ "$FREE_MIB" -lt "$MIN_FREE_MIB" ]; then
+    echo "REFUSED peak-disk: $FREE_MIB MiB free under $PROBE, $MIN_FREE_MIB MiB wanted (QWEN38_MIN_FREE_MIB) — the bf16 container rewrites the checkpoint's own 343,323 MiB of tensors beside it" >&2
+    exit 2
+fi
+
+# ---- gate: one conversion ---------------------------------------------------
+if [ -e "$OUT" ]; then
+    if [ "$PLAN_ONLY" = "1" ]; then
+        echo "note: $OUT already exists; --plan-only writes nothing, so it stays"
+    elif [ "$FORCE" = "1" ]; then
+        mkdir -p "$TRASH"
+        DEST="$TRASH/$(date -u +%Y%m%dT%H%M%SZ)-$(basename "$OUT")"
+        mv "$OUT" "$DEST"
+        echo "TRASHED: retired the existing reference container $OUT -> $DEST"
+    else
+        echo "REFUSED: $OUT already exists — the campaign converts the reference once; re-run with --force to retire it to $TRASH first" >&2
+        exit 2
+    fi
+fi
+
+# ---- gate: the tree really is the checkpoint (the full verify) -------------
+echo "== verifying $CKPT (scripts/download-qwen38-bf16.sh, full gates)"
+rc=0
+bash "$SCRIPT_DIR/download-qwen38-bf16.sh" --dir "$CKPT" || rc=$?
+if [ "$rc" -ne 0 ]; then
+    if [ "$rc" -eq 2 ]; then
+        echo "REFUSED: the checkpoint verifier refused the environment (rc=2); nothing was converted" >&2
+        exit 2
+    fi
+    echo "REFUSED: $CKPT failed verification (rc=$rc) — a half tree converts to a half reference" >&2
+    exit 1
+fi
+
+# ---- plan, then convert ------------------------------------------------------
+mkdir -p "$(dirname "$LOG")"
+# rad-convert opens -o during declare as well, so the output directory exists
+# before the plan; that empty directory is all this creates ahead of it.
+mkdir -p "$OUT_DIR"
+echo "== rad-convert --plan-only -v (declare + plan; writes nothing)"
+"${PLAN_CMD[@]}" 2>&1 | tee -a "$LOG"
+rc=${PIPESTATUS[0]}
+if [ "$rc" -ne 0 ]; then
+    echo "REFUSED plan: rad-convert --plan-only exited rc=$rc — the engine refuses this encoding/shape, no container was written (log: $LOG)" >&2
+    exit 2
+fi
+if [ "$PLAN_ONLY" = "1" ]; then
+    echo "PLAN_OK: $CKPT_IN declares for $OUT_IN; no container written (log: $LOG)"
+    exit 0
+fi
+
+echo "== convert: $CKPT -> $OUT ($(date -Is))"
+"${CONV_CMD[@]}" 2>&1 | tee -a "$LOG"
+rc=${PIPESTATUS[0]}
+if [ "$rc" -ne 0 ]; then
+    echo "rad-convert rc=$rc; log: $LOG" >&2
+    exit "$rc"
+fi
+MIB=$(( $(stat -c %s "$OUT") / 1048576 ))
+echo "BUILD_DONE: $OUT is ${MIB} MiB (log: $LOG)"
+echo "NEXT: scripts/run-qwen38-bf16.sh kldref records the reference logits from it"

scripts/calib-push.py

@@ -22,6 +22,17 @@ Mechanics notes:
   prompts do not bias the Hessians (the corpus is drawn round-robin over
   source classes, so loss is class-neutral as long as failures are rare;
   a systematic 4xx is loud in the log and must be fixed, not ignored).
+- a round that seals no tap bytes is NORMAL and continues: RADIANCE_CALIB_TOKENS
+  is a pooled floor and this campaign's corpus is 10.03M tokens against a 12M
+  floor, so round 1 seals nothing by construction. A tap that LOSES bytes fails
+  the run, and a run that never sees the engine's completion line exits non-zero.
+  The first version failed on a flat round, which aborted exactly one round too
+  early (2026-10-29).
+- before any prompt, the tap must carry the rig's own marker
+  (`--tap-marker`, written by run-qwen38-bf16.sh calib) naming THIS
+  container, and that container must be running. Measured 2026-10-29:
+  with neither check, four rounds of 10M tokens went into restored
+  production after the rig's boot had died, and recorded nothing.
 
 Usage: calib-push.py [--port 7770] [--parallel 64] [--container arbench]
        [--corpus /home/dank/corpora/out/calib.jsonl] [--max-rounds 4]
@@ -29,11 +40,18 @@ Usage: calib-push.py [--port 7770] [--parallel 64] [--container arbench]
 import argparse
 import concurrent.futures
 import json
+import os
 import subprocess
 import sys
 import time
 import urllib.request
 
+_HERE = os.path.dirname(os.path.abspath(__file__)) if "__file__" in globals() else os.getcwd()
+if _HERE not in sys.path:
+    sys.path.insert(0, _HERE)
+
+import qwen38_common as qc  # noqa: E402 - sibling module, same directory
+
 DONE_MARK = "calibration wrote"
 
 
@@ -49,9 +67,9 @@ def tap_done(container):
     return DONE_MARK in (out.stdout + out.stderr)
 
 
-def push_one(url, prompt):
+def push_one(url, prompt, model):
     body = json.dumps({
-        "model": "frank-archon",
+        "model": model,
         "prompt": prompt,
         "max_tokens": 1,
         "temperature": 0,
@@ -68,38 +86,151 @@ def push_one(url, prompt):
         return False
 
 
-def main():
+def push_round(url, prompts, model, parallel):
+    """Push the whole corpus once; returns how many requests were accepted."""
+    ok = 0
+    with concurrent.futures.ThreadPoolExecutor(parallel) as ex:
+        for good in ex.map(lambda p: push_one(url, p, model), prompts):
+            ok += 1 if good else 0
+    return ok
+
+
+def tap_bytes(path):
+    """Bytes the tap has sealed so far; 0 when the directory is absent.
+
+    A tap whose directory is wrong, full or read-only stops writing while the
+    engine keeps answering 200 — the driver then re-pushes the whole corpus for
+    every round it is allowed and the loss is invisible until the hours are gone.
+    """
+    total = 0
+    try:
+        for root, _, files in os.walk(path):
+            for f in files:
+                try:
+                    total += os.path.getsize(os.path.join(root, f))
+                except OSError:
+                    pass
+    except OSError:
+        return 0
+    return total
+
+
+def round_verdict(before, after, done):
+    """What one round's tap bytes mean.
+
+    'done'   the engine logged that it sealed the tap — the run is over
+    'shrink' the tap LOST bytes: reset, pruned or replaced under us
+    'grew'   sealed bytes this round
+    'flat'   sealed nothing — which is legitimate: RADIANCE_CALIB_TOKENS is a
+             pooled floor, and a round that does not reach it has nothing to
+             write yet (the campaign's corpus is 10.03M tokens against a 12M
+             floor, so round 1 is always flat). Failing on it aborted the run
+             exactly one round too early.
+    """
+    if done:
+        return "done"
+    if after < before:
+        return "shrink"
+    if after == before:
+        return "flat"
+    return "grew"
+
+
+def require_rig(a):
+    """Refuse unless the rig that opened this tap is up and serving right now.
+
+    Three facts, in order: the runner wrote its marker into the tap dir, the
+    marker names the container this pusher targets, and that container is
+    running. A marker without a live container is a boot that died; a live
+    container without a marker is some other engine on the port — on 2026-10-29
+    it was restored production, which took four rounds of 10M tokens before the
+    old driver admitted it had recorded nothing.
+    """
+    marker = os.path.join(a.tap_dir, a.tap_marker)
+    if not os.path.isfile(marker):
+        qc.refuse(f"no rig marker at {marker}: this tap was not opened by "
+                  "run-qwen38-bf16.sh calib, so a push here would feed whatever "
+                  "answers the port and record nothing")
+    try:
+        lines = open(marker, encoding="utf-8", errors="replace").read().splitlines()
+    except OSError as exc:
+        qc.refuse(f"cannot read the rig marker {marker}: {exc}")
+    meta = dict(line.split("=", 1) for line in lines if "=" in line)
+    who = meta.get("container", "")
+    if who != a.container:
+        qc.refuse(f"the rig marker names container {who!r} and this pusher targets "
+                  f"{a.container!r} — wrong tap, or a stale marker")
+    try:
+        p = subprocess.run(["podman", "inspect", "-f", "{{.State.Running}}", a.container],
+                           capture_output=True, text=True, timeout=30)
+    except Exception as exc:
+        qc.refuse(f"cannot inspect the rig container {a.container}: {exc}")
+    if p.returncode != 0 or p.stdout.strip() != "true":
+        qc.refuse(f"the rig container {a.container} is not running — whatever is "
+                  "answering the port, it is not the calibration rig")
+
+
+def parse_args():
     ap = argparse.ArgumentParser()
     ap.add_argument("--port", type=int, default=7770)
     ap.add_argument("--parallel", type=int, default=64)
     ap.add_argument("--container", default="arbench")
+    ap.add_argument("--model", default="frank-archon")
+    ap.add_argument("--tap-dir", default="/home/dank/campaign/qwen38-w5nl64/calib")
+    ap.add_argument("--tap-marker", default=".rig-boot",
+                    help="the file run-qwen38-bf16.sh calib writes into the tap dir; without it the pusher refuses, because an unmarked tap is not a rig")
     ap.add_argument("--corpus", default="/home/dank/corpora/out/calib.jsonl")
     ap.add_argument("--max-rounds", type=int, default=4)
-    a = ap.parse_args()
+    return ap.parse_args()
-    url = f"http://127.0.0.1:{a.port}/v1/completions"
 
+
+def load_prompts(a) -> list[str]:
+    if not os.path.isfile(a.corpus):
+        qc.refuse(f"no corpus at {a.corpus}")
     prompts = []
-    with open(a.corpus) as f:
+    for lineno, rec in qc.iter_jsonl(a.corpus):
-        for line in f:
+        if not isinstance(rec.get("prompt"), str):
-            line = line.strip()
+            qc.refuse(f"{a.corpus}:{lineno} has no string 'prompt' field")
-            if line:
+        prompts.append(rec["prompt"])
-                prompts.append(json.loads(line)["prompt"])
+    if not prompts:
-    print(f"corpus: {len(prompts)} prompts", flush=True)
+        qc.refuse(f"the corpus at {a.corpus} holds no prompts")
+    return prompts
+
+
+def main():
+    a = parse_args()
+    url = f"http://127.0.0.1:{a.port}/v1/completions"
+    prompts = load_prompts(a)
+    require_rig(a)
+    chars = sum(len(p) for p in prompts)
+    print(f"corpus: {len(prompts)} prompts, ~{chars // 4} tokens (chars/4)",
+          flush=True)
 
     for rnd in range(1, a.max_rounds + 1):
         if tap_done(a.container):
             break
+        before = tap_bytes(a.tap_dir)
         t0 = time.time()
-        ok = 0
+        ok = push_round(url, prompts, a.model, a.parallel)
-        with concurrent.futures.ThreadPoolExecutor(a.parallel) as ex:
+        dt = max(time.time() - t0, 1e-9)
-            for good in ex.map(lambda p: push_one(url, p), prompts):
-                ok += 1 if good else 0
-        dt = time.time() - t0
         print(f"round {rnd}: {ok}/{len(prompts)} ok in {dt:.0f}s "
-              f"({sum(len(p) for p in prompts)/4/dt:.0f} chars/s est)",
+              f"({chars / 4 / dt:.0f} tokens/s est)", flush=True)
-              flush=True)
+        if ok == 0:
+            print("FAIL: not one prompt was accepted; fix the engine or the argv "
+                  "before spending another round", flush=True)
+            sys.exit(1)
         if tap_done(a.container):
             break
+        after = tap_bytes(a.tap_dir)
+        verdict = round_verdict(before, after, False)
+        if verdict == "shrink":
+            print(f"FAIL: the tap at {a.tap_dir} LOST bytes this round "
+                  f"({before} B before, {after} B after) — it was reset or replaced "
+                  "under us", flush=True)
+            sys.exit(1)
+        if verdict == "flat":
+            print(f"round {rnd}: the tap sealed nothing yet (byte floor not reached; "
+                  f"{before} B) — continuing", flush=True)
     else:
         print("max rounds exhausted, tap not done", flush=True)
         sys.exit(1)

scripts/census-frank-dragoon.sh

@@ -21,7 +21,7 @@
 # deployed knob has since dropped to 262144.
 set -euo pipefail
 
-IMG=ghcr.io/peonist-ai/halogen-flash-server@sha256:0c61bf84ac22308a53f5d1ca6b86806702d7039e5ebc51cae4c66621b92fe04a
+IMG=ghcr.io/peonist-ai/halogen-flash-server@sha256:e35133b206da3094cc4d70c78800b26235bd9124b9aeb0e029d4a7c153b9cff0
 CUR=/home/dank/models/halogen-flashnext-v2
 NEW=/home/dank/models/halogen-v2-abliterated
 CUR_CK=qwen38-flash-next-v2-swift15-abliterated.hgn

scripts/convert-qwen38-w5nl64.sh

@@ -0,0 +1,236 @@
+#!/usr/bin/env bash
+# convert-qwen38-w5nl64.sh — RUNS ON DANKSERVER. The CPU half of the Qwen3.8-Flash-Next five-bit
+# campaign: convert the vendor BF16 checkpoint into the campaign's arms, each one the whole
+# `w5nl64a8h` rung on BOTH expert projections with the protected experts kept bf16.
+#
+# Nothing here touches a GPU: the container is started without /dev/kfd or /dev/dri, so the
+# converter runs on the host backend and the fleet can keep serving through the whole script.
+# The engine pin is the digest that serves frank-archon, so a winning arm crosses no version
+# boundary between measurement and serving.
+#
+# Three arms, because the campaign's question is which search knob pays for its hours
+# (recipes/qwen4exp-qwen38-w5nl64-bf16trunk.recipe):
+#
+#   cd3     the recipe as written — coordinate descent three sweeps. The control.
+#   cd6     six sweeps: the same search, twice the passes.
+#   ao1     act_order=1 instead of cd: GPTQ sweeps columns in activation-importance order and
+#           stores them permuted. This is a SEPARATE arm, not a fourth sweep — libquant refuses
+#           cd= together with act_order=1 ("cd= sweeps the columns in their stored order, and
+#           act_order=1 stores them permuted; one or the other", libquant/lq_registry.cpp), so
+#           the two knobs cannot be combined into a best-of-both container.
+#
+# The arms are generated from ONE template with `--quant` overrides, not three near-duplicate
+# recipes: --quant rules are inserted BEFORE the recipe file and the first rule whose glob matches
+# a weight decides it (core/format/recipe.cpp: Recipe::match), so an override cleanly replaces the
+# expert line and the other 60 lines cannot drift between arms.
+#
+# Protected experts: apply-protected-cast.sh derives the bf16 set from THIS campaign's grams
+# (--target-count 48, two per layer so each rank's 256-expert half stays even) and splices it into
+# a per-run COPY of the template; the committed recipe keeps its markers and stays reviewable.
+#
+# Gates, in the order they bite — each one is a lesson the swift15 campaign paid for:
+#   inputs        checkpoint (dir + index), recipe, grams, imatrix, tokenizer. A missing grams
+#                 directory converts UNCALIBRATED with one stderr note, which is silently worse
+#                 than failing here.
+#   reference     the 335.3 GiB BF16 reference container must be GONE before the converts start.
+#                 It and the checkpoint together eat the volume; its job (the KLD recording and
+#                 the calibration grams) is finished by then, and both survive on disk without the
+#                 container beside it. The FP8-native anchor is FETCHED after the converts (disk
+#                 sequencing), so nothing here waits on it.
+#   plan          --plan-only runs FIRST, per arm, and prints "about X would be written". The
+#                 number is the bill: free space must cover it plus the 128k-prompt overshoot
+#                 margin, and the finished artifact is checked against it.
+#   outputs       no arm overwrites anything; move the old one to trash.
+#
+# Usage: convert-qwen38-w5nl64.sh [--arm cd3|cd6|ao1]... [--dry-run] [--plan-only]
+#                                 [--target-count N] [--keep-recipe]
+set -u -o pipefail
+
+SELF="$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)"
+# THE TOOLCHAIN PIN. `:latest` on 2026-10-30 is v1.14.0 (revision 4a61d7296bc5),
+# the same image the tap producer boots: a tap must be converted by the line that
+# is current, and rad-convert's `gptq:calib=<dir>` input contract is version-stable
+# (RADGRAM2 + gram.r<rank>.<stem>.bin), so nothing else about the recipe moves with
+# it. Production stays on its own pin — this is the campaign's toolchain.
+IMAGE=${IMAGE:-stilldeadcode/radiance@sha256:69add58f7de1ed2bca2b1e20a4cc75717557df3f73569a178eca5aafe8c44093}
+CKPT=${CKPT:-/home/dank/models/Qwen3.8-Flash-Next}
+CAMPAIGN=${CAMPAIGN:-/home/dank/campaign/qwen38-w5nl64}
+RECIPE=${RECIPE:-$SELF/recipes/qwen4exp-qwen38-w5nl64-bf16trunk.recipe}
+CALIB=${CALIB:-$CAMPAIGN/calib}
+IMATRIX=${IMATRIX:-$CAMPAIGN/imatrix.gguf}
+OUTDIR=${OUTDIR:-/home/dank/models/rad}
+BF16_RAD=${BF16_RAD:-$OUTDIR/qwen38-bf16.rad}   # the reference container, not an arm output
+STATE=${STATE:-$CAMPAIGN}                      # the dir the container mounts read-only: the
+                                               # spliced recipe and the imatrix live here
+TARGET_COUNT=${TARGET_COUNT:-48}
+MIN_GRAMS=${MIN_GRAMS:-24576}                 # per-expert DOWN grams: 48 recorded layers x 512
+                                               # experts. The checkpoint has 49 MoE layers, but the
+                                               # MTP layer is never recorded, so its 512 experts are
+                                               # absent from the tap's output by design
+MARGIN_MIB=${MARGIN_MIB:-40960}               # the long-prompt overshoot, off the plan number
+TS=$(date +%Y%m%d-%H%M%S)
+
+ARMS=()
+DRY=0
+PLAN_ONLY=0
+KEEP_RECIPE=0
+while [ "$#" -gt 0 ]; do
+    case "$1" in
+      --arm) [ "$#" -ge 2 ] || { echo "--arm needs a value (cd3|cd6|ao1)" >&2; exit 2; }
+             ARMS+=("$2"); shift 2 ;;
+      --dry-run) DRY=1; shift ;;
+      --plan-only) PLAN_ONLY=1; shift ;;
+      --target-count) [ "$#" -ge 2 ] || { echo "--target-count needs a value" >&2; exit 2; }
+             TARGET_COUNT="$2"; shift 2 ;;
+      --keep-recipe) KEEP_RECIPE=1; shift ;;
+      *) echo "unknown argument: $1" >&2; exit 2 ;;
+    esac
+done
+[ "${#ARMS[@]}" -gt 0 ] || ARMS=(cd3 cd6 ao1)
+for a in "${ARMS[@]}"; do
+    case "$a" in cd3|cd6|ao1) ;; *) echo "unknown arm: $a (cd3|cd6|ao1)" >&2; exit 2 ;; esac
+done
+
+# The expert rule's option set, spelled once. --quant replaces the template's line wholesale, so
+# every option has to be repeated here; `calib=$CALIB` is expanded by the recipe reader from the
+# container environment (core/format/recipe.cpp: expand refuses an unset name rather than taking
+# it for empty).
+expert_opts='table=w5nl,group=64,scale=fp8_e4m3,scale2=f32,block2=*x*,scale2_value=0.0001220703125,transform=fwht128,rule=search,calib=$CALIB'
+arm_quant() {
+    case "$1" in
+      # The control is the template itself: no override, so the recipe's own line decides.
+      cd3) return 0 ;;
+      cd6) echo "blk.*.ffn_*_exps.*.weight=gptq:${expert_opts},cd=6" ;;
+      # act_order=1 WITHOUT cd: libquant refuses the pair (lq_registry.cpp:105, "cd= sweeps the
+      # columns in their stored order, and act_order=1 stores them permuted; one or the other"), so
+      # this arm is the interleave knob alone, not cd=3 plus it.
+      ao1) echo "blk.*.ffn_*_exps.*.weight=gptq:${expert_opts},act_order=1" ;;
+    esac
+}
+
+# --- gates ------------------------------------------------------------------------------------
+[ -d "$CKPT" ] || { echo "no checkpoint at $CKPT; run scripts/download-qwen38-bf16.sh --fetch" >&2; exit 2; }
+[ -f "$CKPT/model.safetensors.index.json" ] || { echo "checkpoint at $CKPT has no safetensors index — a torn fetch" >&2; exit 2; }
+[ -f "$CKPT/tokenizer.json" ] || { echo "no tokenizer.json in $CKPT; the .rad cannot carry one without it" >&2; exit 2; }
+[ -f "$RECIPE" ] || { echo "no recipe at $RECIPE" >&2; exit 2; }
+GRAMS=$(find "$CALIB" -maxdepth 1 -name 'gram.r*.bin' 2>/dev/null | wc -l)
+[ "$GRAMS" -ge "$MIN_GRAMS" ] || {
+    echo "calib dir $CALIB holds $GRAMS gram files (<$MIN_GRAMS); run $0 after run-qwen38-bf16.sh calib" >&2; exit 2; }
+[ -f "$IMATRIX" ] || { echo "no imatrix at $IMATRIX; run scripts/grams-to-imatrix.py over $CALIB" >&2; exit 2; }
+# The container sees the campaign dir at one fixed mount point, so an imatrix kept elsewhere has
+# no path inside it: copy it in (or point IMATRIX at the campaign dir) rather than let the
+# converter fail halfway through a planned arm.
+[ "$(dirname "$IMATRIX")" = "$STATE" ] || {
+    echo "imatrix $IMATRIX is not in the campaign dir $STATE the container mounts; move it there" >&2; exit 2; }
+if [ -e "$BF16_RAD" ]; then
+    echo "the 335.3 GiB BF16 reference container $BF16_RAD is still on disk; its recording and" \
+         "the calibration grams are already taken, so move it to trash — the converts need its" \
+         "space (peak-disk law)" >&2
+    exit 2
+fi
+for a in "${ARMS[@]}"; do
+    [ -e "$OUTDIR/qwen38-w5nl64-$a.rad" ] && {
+        echo "$OUTDIR/qwen38-w5nl64-$a.rad already exists; move it to trash first" >&2; exit 2; }
+    # A killed run leaves its container behind (--rm only fires when podman itself
+    # exits), and the next arm then dies on "name already in use" AFTER the plan
+    # phase; refuse here, where the fix is one `podman rm -f`.
+    podman container exists "qwen38-convert-$a" && {
+        echo "container qwen38-convert-$a still exists (a killed run?);" \
+             "'podman rm -f qwen38-convert-$a' and re-run" >&2
+        exit 2; }
+done
+
+# --- the protected plane, spliced into a per-run copy of the template --------------------------
+mkdir -p "$STATE"
+WORK="$STATE/recipe-$TS.recipe"
+cp "$RECIPE" "$WORK"
+if [ "$DRY" = 1 ]; then
+    echo "would run: $SELF/apply-protected-cast.sh $CALIB $WORK $TARGET_COUNT"
+else
+    # Positional, because the tool reads its recipe and calib dir by position: passing them as
+    # environment variables would be silently ignored and it would rewrite the COMMITTED
+    # template in place instead of this run's copy.
+    "$SELF/apply-protected-cast.sh" "$CALIB" "$WORK" "$TARGET_COUNT" \
+        || { echo "the protected-expert derivation refused; the arms would convert unprotected" >&2; exit 1; }
+fi
+
+run_convert() { # run_convert ARM [extra flags...]
+    local arm="$1"; shift
+    local out="$OUTDIR/qwen38-w5nl64-$arm.rad" log="$STATE/convert-$arm-$TS.log"
+    local argv=(--rm --name "qwen38-convert-$arm"
+                -e "CALIB=/data/calib" -e "OMP_NUM_THREADS=$(nproc)"
+                -v "$(dirname "$CKPT"):/models:ro"
+                -v "$CALIB:/data/calib:ro"
+                -v "$STATE:/campaign:ro"
+                -v "$OUTDIR:/out")
+    local conv=(/models/"${CKPT##*/}" --recipe "/campaign/$(basename "$WORK")"
+                --tokenizer "/models/${CKPT##*/}/tokenizer.json"
+                --imatrix /campaign/"${IMATRIX##*/}"
+                # No --max-ctx: the context declare then runs at the checkpoint's training context
+                # (262144), which is what the shipped container did — rad-info --meta on it carries
+                # no override. A container declared at a smaller bound cannot be expected to serve
+                # the 200k window D2 holds, and the convert is not where to discover that.
+                # --kv-cache-dtype changes nothing in the artifact; it picks the kernels the declare
+                # checks, and the fleet serves fp8 KV, so an unresolvable fp8 kernel must be said no
+                # to by the convert, not by a boot on a drained window.
+                --kv-cache-dtype fp8 -o "/out/$(basename "$out")")
+    local q; q=$(arm_quant "$arm")
+    [ -n "$q" ] && conv+=(--quant "$q")
+    if [ "$DRY" = 1 ]; then
+        echo "podman run ${argv[*]} \$IMAGE --entrypoint /opt/radiance/bin/rad-convert ${conv[*]} $*"
+        return 0
+    fi
+    podman run "${argv[@]}" --entrypoint /opt/radiance/bin/rad-convert "$IMAGE" "${conv[@]}" "$@" \
+        2>&1 | tee "$log"
+    return "${PIPESTATUS[0]}"
+}
+
+# --- plan, then bill, then convert -------------------------------------------------------------
+overall=0
+for arm in "${ARMS[@]}"; do
+    echo "== arm $arm =="
+    if [ "$DRY" = 1 ]; then
+        # Both lines, in order: the plan runs before the convert, and the convert is what the
+        # operator is about to pay an afternoon for.
+        run_convert "$arm" --plan-only
+        run_convert "$arm"
+        continue
+    fi
+    rc=0; plan_log=$(run_convert "$arm" --plan-only) || rc=$?
+    [ "$rc" -eq 0 ] || { echo "arm $arm: --plan-only failed (rc=$rc); nothing was written" >&2; overall=1; continue; }
+    echo "$plan_log"
+    # "about 141.2 GiB would be written" is the converter's own bill; parse it rather than
+    # hardcoding a size, so the gate still works after any recipe edit.
+    want=$(printf '%s' "$plan_log" | sed -n 's/.*about \([0-9.]*\) \([KMGT]iB\) would be written.*/\1 \2/p' | tail -1)
+    [ -n "$want" ] || { echo "arm $arm: the plan printed no size; refusing to convert blind" >&2; overall=1; continue; }
+    need_mib=$(awk -v n="${want% *}" -v u="${want#* }" \
+        'BEGIN{ f = (u=="KiB") ? 2^-10 : (u=="MiB") ? 1 : (u=="GiB") ? 1024 : (u=="TiB") ? 1048576 : 0
+               if (f == 0) { exit 1 }; printf "%d", n*f + 0.5 }') || {
+        echo "arm $arm: the plan printed an unknown unit in '$want'" >&2; overall=1; continue; }
+    free_mib=$(df --output=avail -m "$OUTDIR" | tail -1 | tr -d ' ')
+    echo "plan: $want ($need_mib MiB); free ${free_mib} MiB on $OUTDIR (margin $MARGIN_MIB MiB)"
+    if [ "$free_mib" -lt $((need_mib + MARGIN_MIB)) ]; then
+        echo "arm $arm: needs ${need_mib} MiB + ${MARGIN_MIB} margin, ${free_mib} free; free space and re-run" >&2
+        overall=1; continue
+    fi
+    if [ "$PLAN_ONLY" = 1 ]; then echo "planned, not converted (--plan-only)"; continue; fi
+    rc=0; run_convert "$arm" || rc=$?
+    [ "$rc" -eq 0 ] || { echo "arm $arm: rad-convert rc=$rc; log: $STATE/convert-$arm-$TS.log" >&2; overall=1; continue; }
+    out="$OUTDIR/qwen38-w5nl64-$arm.rad"
+    [ -f "$out" ] || { echo "arm $arm: rad-convert exited 0 but wrote no $out; log: $STATE/convert-$arm-$TS.log" >&2; overall=1; continue; }
+    got_mib=$(( $(stat -c %s "$out") / 1048576 ))
+    # The receipt: the artifact is what the plan predicted, within the metadata the plan does not
+    # count. An arm that came out materially smaller lost a plane somewhere.
+    lo=$(( need_mib * 98 / 100 )); hi=$(( need_mib * 103 / 100 ))
+    if [ "$got_mib" -lt "$lo" ] || [ "$got_mib" -gt "$hi" ]; then
+        echo "arm $arm: $out is ${got_mib} MiB, the plan said ${need_mib} MiB (allowed ${lo}-${hi})" >&2
+        overall=1; continue
+    fi
+    echo "CONVERT_DONE $arm ${got_mib} MiB -> $out"
+done
+
+if [ "$KEEP_RECIPE" = 0 ] && [ "$DRY" = 0 ]; then
+    mkdir -p "$CAMPAIGN/.AGENTTRASH"
+    mv "$WORK" "$CAMPAIGN/.AGENTTRASH/$(basename "$WORK")" 2>/dev/null || true
+fi
+exit "$overall"

scripts/deploy-plan.py

@@ -34,7 +34,6 @@ SERVICE_ORDER = (
     "hermes",
     "hermes-dashboard",
     "hermes-webui",
-    "prometheus",
     "grafana",
     "loki",
     "alloy",
@@ -42,6 +41,9 @@ SERVICE_ORDER = (
     "cadvisor",
     "gpu-exporter",
     "alertmanager",
+    # The prober must answer before prometheus re-runs the services job.
+    "blackbox-exporter",
+    "prometheus",
     "authentik-postgres",
     "authentik-server",
     "authentik-worker",
@@ -97,6 +99,7 @@ DANKCLAW_SERVICES = {
     "cadvisor",
     "gpu-exporter",
     "alertmanager",
+    "blackbox-exporter",
     "authentik-postgres",
     "authentik-server",
     "authentik-worker",
@@ -152,13 +155,14 @@ DANKCLAW_COMPOSE_SERVICES = set()
 # Quadlet unit now — the last compose services were the hermes trio, whose units
 # have owned their containers since 2026-09-12 and whose compose blocks were
 # dead weight that could only create a duplicate container.
+# jellyfin left the Dankserver file on 2026-10-30 for the same reason: its Quadlet
+# has owned the container since the cutover, so a compose edit must not restart it.
 
 DANKSERVER_COMPOSE_SERVICES = {
     "mariadb",
     "redis",
     "romm",
     "copyparty",
-    "jellyfin",
     "calibre-web",
 }
 COMPOSE_NAME_MAP = {"mariadb": "romm-mariadb", "redis": "romm-redis"}
@@ -277,6 +281,8 @@ def services_for_path(path: str, target: str) -> set[str]:
             # Provisioned datasources and dashboards: git is the source of
             # truth, so a change here must restart the service that reads them.
             _add(services, "grafana")
+        elif group == "blackbox":
+            _add(services, "blackbox-exporter")
         elif group == "homepage":
             # Homepage is a Dankclaw-owned deployment. Shared homepage files
             # are intentionally not copied during a Dankserver-only run.
@@ -380,7 +386,7 @@ def services_for_path(path: str, target: str) -> set[str]:
             # consumer (prometheus before grafana, loki before alloy).
             if len(parts) >= 4 and parts[2] == "quadlet":
                 name = parts[3].removesuffix(".container")
-                if name in {"prometheus", "grafana", "loki", "alloy", "node-exporter", "cadvisor", "alertmanager"}:
+                if name in {"prometheus", "grafana", "loki", "alloy", "node-exporter", "cadvisor", "alertmanager", "blackbox-exporter"}:
                     _add(services, name)
             return services
         if len(parts) >= 2 and parts[1] == "gitea":
@@ -416,6 +422,11 @@ def services_for_path(path: str, target: str) -> set[str]:
                 "tautulli": {"tautulli"},
             }
             _add(services, *quadlet_services.get(parts[1], set()))
+        if len(parts) >= 2 and parts[1] == "host":
+            # dankserver/host/systemd/media-meta-view.mount is the overlay view the
+            # media servers bind (scripts/media-view.sh installs it on the host),
+            # so editing it restarts jellyfin and plex and nothing else.
+            _add(services, "jellyfin", "plex")
         if len(parts) >= 2 and parts[1] == "observability":
             # Dankserver's node-exporter became a Quadlet unit on 2026-09-16
             # (serverAppConfigs-docs/Infrastructure/Observability.md): its
@@ -461,6 +472,10 @@ def services_for_path(path: str, target: str) -> set[str]:
             # that service (target_services then keeps it only on the host being
             # planned). Editing the collector restarts the running exporter.
             _add(services, "gpu-exporter")
+        if name == "media-view.sh":
+            # The installer is what creates, mounts and reaps the view both media
+            # servers read, so a change to it is a restart of that pair.
+            _add(services, "jellyfin", "plex")
         if name in {"frank-zealot.env", "frank-zealot.service", "serve-frank-zealot.sh"}:
             # llama.cpp frank-zealot (2026-09-16) keeps its unit, env and serve
             # files here; each one selects frank-zealot only. The ik_llama.cpp

scripts/deploy-remote.sh

@@ -86,7 +86,7 @@ service_is_allowed() {
         dankclaw:open-webui|dankclaw:gitea-postgres|dankclaw:gitea-server|\
         dankclaw:comfyui|dankclaw:hermes|\
         dankclaw:hermes-dashboard|dankclaw:hermes-webui|dankclaw:prometheus|dankclaw:grafana|\
-        dankclaw:loki|dankclaw:alloy|dankclaw:node-exporter|dankclaw:cadvisor|dankclaw:gpu-exporter|dankclaw:alertmanager|\
+        dankclaw:loki|dankclaw:alloy|dankclaw:node-exporter|dankclaw:cadvisor|dankclaw:gpu-exporter|dankclaw:alertmanager|dankclaw:blackbox-exporter|\
         dankclaw:authentik-postgres|dankclaw:authentik-server|\
         dankclaw:authentik-worker|dankclaw:authentik-outpost|dankclaw:authentik-outpost-ldap|dankclaw:anubis|\
         dankclaw:caddy|dankclaw:homepage|dankclaw:seerr|dankclaw:marreta|\
@@ -252,16 +252,24 @@ install_signal_relay() {
     # The relay binds the host's LAN IP (the one address a bridge container
     # can reach as host.containers.internal), so keep the unauthenticated
     # endpoint off the rest of the LAN: admit loopback, the host itself
-    # (its own health probe) and the podman bridge subnet only. Idempotent:
+    # (its own health probe) and the podman bridge subnet only.
-    # -C before -I.
+    # ORDER IS THE GATE: `iptables -I` inserts at position 1, so the old
+    # shape (-C/-I x3 ACCEPT, then -C/-I the DROP) landed the blanket DROP
+    # ABOVE the ACCEPTs and every probe timed out beside a LISTENing socket
+    # (live 2026-10-21: 159 packets on the DROP, 0 on each ACCEPT; the whole
+    # dankclaw drain aborted at the alertmanager gate twice before the cause
+    # was read off `nft list ruleset`). Rebuild the tail instead of inserting
+    # it: delete any blanket DROP wherever it sits, keep the three ACCEPTs
+    # at the head (-C before -I, unchanged), then APPEND the DROP so it is
+    # last and the ACCEPTs decide first. Converges on poisoned hosts.
+    run_privileged iptables -D INPUT -p tcp --dport 18093 -j DROP 2>/dev/null || true
     run_privileged iptables -C INPUT -p tcp --dport 18093 -s 127.0.0.1 -j ACCEPT 2>/dev/null \
         || run_privileged iptables -I INPUT -p tcp --dport 18093 -s 127.0.0.1 -j ACCEPT
     run_privileged iptables -C INPUT -p tcp --dport 18093 -s 192.168.0.143 -j ACCEPT 2>/dev/null \
         || run_privileged iptables -I INPUT -p tcp --dport 18093 -s 192.168.0.143 -j ACCEPT
     run_privileged iptables -C INPUT -p tcp --dport 18093 -s 10.89.0.0/16 -j ACCEPT 2>/dev/null \
         || run_privileged iptables -I INPUT -p tcp --dport 18093 -s 10.89.0.0/16 -j ACCEPT
-    run_privileged iptables -C INPUT -p tcp --dport 18093 -j DROP 2>/dev/null \
+    run_privileged iptables -A INPUT -p tcp --dport 18093 -j DROP
-        || run_privileged iptables -I INPUT -p tcp --dport 18093 -j DROP
     # The interpreter needs a moment to bind after restart; a one-shot curl
     # raced the socket on 2026-09-17 and failed a healthy deploy.
     local i
@@ -311,8 +319,45 @@ copy_loki_config() {
 }
 
 copy_alloy_config() {
-    run_privileged mkdir -p /opt/alloy/config
+    # The repo config carries the token GRAFANA_HOST_LABEL; the installed copy
-    run_privileged cp shared/alloy/config.alloy /opt/alloy/config/config.alloy
+    # carries the host's name, so Loki streams say WHICH machine's journal a
+    # line came from without a second config file per host.
+    # *_CONF_ROOT are test seams: the bats cases execute these copiers for
+    # real against a sandbox with a stubbed run_privileged.
+    local root="${ALLOY_CONF_ROOT:-/opt/alloy}" tmp
+    tmp="$(mktemp)"
+    sed "s/GRAFANA_HOST_LABEL/$TARGET/g" shared/alloy/config.alloy > "$tmp"
+    if ! grep -q "host = \"$TARGET\"," "$tmp" || grep -q "GRAFANA_HOST_LABEL" "$tmp"; then
+        echo "alloy config: host-label substitution failed" >&2
+        rm -f "$tmp"
+        return 1
+    fi
+    # Syntax gate: alloy's config language has no schema — a misplaced
+    # attribute (e.g. `on_error` outside the json stage) parses only at RUN
+    # time, and the unit then crash-loops with a healthy-looking deploy.
+    # Validate against the very image that will run it, before installing.
+    # Skipped only in the bats sandbox (no image there).
+    if [ "$root" = "/opt/alloy" ] || podman image exists localhost/franksdank/alloy:latest 2>/dev/null; then
+        if ! podman run --rm -v "$tmp:/c.alloy:ro" \
+                --entrypoint /usr/bin/alloy localhost/franksdank/alloy:latest \
+                validate /c.alloy; then
+            echo "alloy config: validate FAILED against the pinned image" >&2
+            rm -f "$tmp"
+            return 1
+        fi
+    fi
+    run_privileged mkdir -p "$root/config"
+    run_privileged cp "$tmp" "$root/config/config.alloy"
+    rm -f "$tmp"
+}
+
+copy_blackbox_config() {
+    # Single-file mount target: the unit binds this exact path into
+    # /etc/blackbox_exporter/config.yml. cp (not cp -r) keeps it one file.
+    # BLACKBOX_CONF_ROOT is the same test seam.
+    local root="${BLACKBOX_CONF_ROOT:-/opt/blackbox}"
+    run_privileged mkdir -p "$root"
+    run_privileged cp shared/blackbox/blackbox.yml "$root/blackbox.yml"
 }
 
 copy_grafana_provisioning() {
@@ -321,14 +366,25 @@ copy_grafana_provisioning() {
     # datasource would be invisible to review, lost with the data volume, and
     # unknown to anyone reading the repository. The unit already mounts
     # /opt/grafana/provisioning read-only, so this only has to populate it.
-    run_privileged mkdir -p /opt/grafana/provisioning/datasources \
+    # GRAFANA_PROV_ROOT is the test seam: point it at a sandbox and the whole
-        /opt/grafana/provisioning/dashboards
+    # copy runs unprivileged (the bats case executes this function for real).
+    local root="${GRAFANA_PROV_ROOT:-/opt/grafana}"
+    run_privileged mkdir -p "$root/provisioning/datasources" \
+        "$root/provisioning/dashboards"
     run_privileged cp shared/grafana/provisioning/datasources/prometheus.yml \
-        /opt/grafana/provisioning/datasources/prometheus.yml
+        "$root/provisioning/datasources/prometheus.yml"
     run_privileged cp shared/grafana/provisioning/dashboards/provider.yml \
-        /opt/grafana/provisioning/dashboards/provider.yml
+        "$root/provisioning/dashboards/provider.yml"
-    run_privileged cp shared/grafana/provisioning/dashboards/llm-fleet.json \
+    # EVERY dashboard JSON rides, not a hand-listed subset: the fleet provider
-        /opt/grafana/provisioning/dashboards/llm-fleet.json
+    # turns each file in the directory into a dashboard, so a copier that names
+    # files one by one silently drops new boards (measured 2026-10-21: the
+    # containers board existed in the repo since its commit and was NEVER
+    # provisioned — the panel-census browser check could not find it while the
+    # hand-placed older boards happened to be on disk). Same rule as the quadlet
+    # tier: the directory is the contract, the copy is generated state.
+    for dash in shared/grafana/provisioning/dashboards/*.json; do
+        run_privileged cp "$dash" "$root/provisioning/dashboards/$(basename "$dash")"
+    done
 }
 
 render_caddyfile() {
@@ -552,6 +608,9 @@ deploy_observability() {
             ;;
         cadvisor)
             ;;
+        blackbox-exporter)
+            copy_blackbox_config
+            ;;
         *)
             echo "unknown observability service: $service" >&2
             return 2
@@ -566,6 +625,9 @@ deploy_observability() {
         loki) wait_for_health loki http://127.0.0.1:3100/ready 60 ;;
         node-exporter) wait_for_health node-exporter http://127.0.0.1:9100/metrics 60 ;;
         cadvisor) wait_for_health cadvisor http://127.0.0.1:8080/healthz 90 ;;
+        # Loopback: the exporter binds 127.0.0.1 + LAN (the gateway address
+        # is not bindable from the host netns — see the unit header).
+        blackbox-exporter) wait_for_health blackbox-exporter http://127.0.0.1:9115/-/healthy 60 ;;
     esac
 }
 
@@ -1140,7 +1202,22 @@ deploy_romm_stack() {
     systemctl --user restart "$service.service"
 }
 
+ensure_media_view() {
+    # Both media servers read their library through the SSD-backed overlay view
+    # (scripts/media-view.sh, dankserver/host/systemd/media-meta-view.mount): the
+    # array below stays read-only and every sidecar Jellyfin writes lands on the
+    # NVMe.
+    # Raise it BEFORE either container starts — a start against a bare view
+    # directory would let a scan clear the library database. The units also carry
+    # ConditionPathIsMountPoint, so a missing view skips the start rather than
+    # damaging anything. A relayout must be followed by `media-view.sh reap`
+    # (see the Media Relayout page), which this is not.
+    run_privileged bash scripts/media-view.sh install
+    bash scripts/media-view.sh verify
+}
+
 deploy_jellyfin() {
+    ensure_media_view
     cp "dankserver/jellyfin/quadlet/jellyfin.container" "$quadlet_unit_dir/"
     systemctl --user daemon-reload
     if ! systemctl --user is-active -q jellyfin.service; then
@@ -1228,6 +1305,7 @@ deploy_copyparty() {
 }
 
 deploy_plex() {
+    ensure_media_view
     cp "dankserver/plex/quadlet/plex.container" "$quadlet_unit_dir/"
     systemctl --user daemon-reload
     if ! systemctl --user is-active -q plex.service; then
@@ -1300,11 +1378,17 @@ deploy_authentik() {
     # generated". Boot persistence comes from [Install] WantedBy=default.target
     # inside each .container file, which the generator honors at user-manager
     # start (linger must stay on for the deploy user).
-    systemctl --user start authentik-postgres.service
+    # RESTART, not start: `start` on an already-active unit is a no-op, so an
+    # image bump would install the new digest and keep serving the old
+    # container while every gate stayed green (measured 2026-10-30: server and
+    # worker stayed on the previous release beside an outpost that moved,
+    # because only the outpost handler restarts). The generator has already
+    # re-rendered the unit from the .container, so a restart is what applies it.
+    systemctl --user restart authentik-postgres.service
     wait_container_healthy authentik-postgres 120 || return 1
-    systemctl --user start authentik-server.service
+    systemctl --user restart authentik-server.service
     wait_container_healthy authentik-server 300 || return 1
-    systemctl --user start authentik-worker.service
+    systemctl --user restart authentik-worker.service
 }
 
 deploy_authentik_outpost() {
@@ -1420,7 +1504,7 @@ deploy_anubis() {
 # artifact would leave a restart-looping unit and a dark endpoint. The
 # port gate is the same as every cutover since 2026-09-16: nothing
 # starts until 7770 is released.
-LLM_DRAGOON_IMAGE="${LLM_DRAGOON_IMAGE:-ghcr.io/peonist-ai/halogen-flash-server@sha256:0c61bf84ac22308a53f5d1ca6b86806702d7039e5ebc51cae4c66621b92fe04a}"
+LLM_DRAGOON_IMAGE="${LLM_DRAGOON_IMAGE:-ghcr.io/peonist-ai/halogen-flash-server@sha256:e35133b206da3094cc4d70c78800b26235bd9124b9aeb0e029d4a7c153b9cff0}"
 # The payload dir is its own twin because the NPU lane's containment gate below
 # reads it; every artifact path under it stays a full literal as usual.
 LLM_DRAGOON_MODEL_DIR="${LLM_DRAGOON_MODEL_DIR:-/home/dank/models/halogen-v2-abliterated}"
@@ -2034,7 +2118,7 @@ deploy_service() {
         plex) deploy_plex ;;
         tautulli) deploy_tautulli ;;
         alloy) deploy_alloy ;;
-        prometheus|grafana|loki|node-exporter|cadvisor|alertmanager) deploy_observability "$service" ;;
+        prometheus|grafana|loki|node-exporter|cadvisor|alertmanager|blackbox-exporter) deploy_observability "$service" ;;
         gpu-exporter) deploy_gpu_exporter ;;
         comfyui) deploy_comfyui ;;
         *) deploy_compose_service "$service" ;;

scripts/deploy.sh

@@ -140,7 +140,7 @@ valid_service() {
         dankclaw:gitea-server|dankclaw:homepage|\
         dankclaw:comfyui|dankclaw:hermes|dankclaw:hermes-dashboard|\
         dankclaw:hermes-webui|dankclaw:prometheus|dankclaw:grafana|dankclaw:loki|dankclaw:alloy|\
-        dankclaw:alertmanager|\
+        dankclaw:alertmanager|dankclaw:blackbox-exporter|\
         dankclaw:node-exporter|dankclaw:cadvisor|dankclaw:gpu-exporter|dankclaw:authentik-postgres|dankclaw:authentik-server|\
         dankclaw:authentik-worker|dankclaw:authentik-outpost|dankclaw:authentik-outpost-ldap|dankclaw:anubis|dankclaw:seerr|\
         dankclaw:open-notebook|\

scripts/derive-protected-experts.py

@@ -20,9 +20,16 @@ of the Gram is its down-input energy. Rank halves: 0-255, 256-511
 Output: the cast-rule lines (recipe format) on stdout, a report on stderr.
 
 Usage: derive-protected-experts.py <calib dir> [--min-share 0.03]
-       [--min-route-mult 3.0]
+       [--min-route-mult 3.0] [--target-count 0]
+       --target-count N fills the byte bill: the bars pick the base set exactly as
+       the bar-only pass does, then the energy ranking admits experts one at a time
+       (each with its parity partner) until the count lands. The energy bar NEVER
+       relaxes, so asking for 48 bf16 experts never quietly buys 80; the ratio bar
+       walks down only when the ranking cannot produce that many candidates.
+       0 takes the bars as given.
 """
 import argparse
+import collections
 import glob
 import os
 import re
@@ -32,6 +39,7 @@ import sys
 import numpy as np
 
 NAME_RE = re.compile(r"^gram\.r(\d)\.blk\.(\d+)\.ffn_down_exps\.(\d+)\.bin$")
+CAST_RE = re.compile(r"^blk\.(\d+)\.ffn_\*_exps\.(\d+)\.weight")
 
 
 def read_gram(path):
@@ -49,27 +57,8 @@ def read_gram(path):
     return k, rank, int(rows), tr
 
 
-def main():
+def select(data, min_share, min_route_mult):
-    ap = argparse.ArgumentParser()
+    """(cast lines, report lines, count) for one pair of bars."""
-    ap.add_argument("calib_dir")
-    ap.add_argument("--min-share", type=float, default=0.03)
-    ap.add_argument("--min-route-mult", type=float, default=3.0)
-    a = ap.parse_args()
-
-    # layer -> global expert -> (rank, rows, energy)
-    data = {}
-    for path in sorted(glob.glob(os.path.join(
-            a.calib_dir, "gram.r*.ffn_down_exps.*.bin"))):
-        m = NAME_RE.match(os.path.basename(path))
-        if not m:
-            continue
-        rank, layer, expert = int(m.group(1)), int(m.group(2)), int(m.group(3))
-        k, _, rows, tr = read_gram(path)
-        data.setdefault(layer, {})[expert] = (rank, rows, tr)
-
-    if not data:
-        raise SystemExit("no per-expert down grams found")
-
     lines, report = [], []
     total = 0
     for layer in sorted(data):
@@ -83,7 +72,7 @@ def main():
         for e, (rank, rows, tr) in experts.items():
             es, rs = tr / tot_e, rows / max(tot_r, 1)
             rec = (e, rank, es, rs)
-            if es >= a.min_share and es >= a.min_route_mult * rs:
+            if es >= min_share and es >= min_route_mult * rs:
                 qual.append(rec)
             elif es > 0:
                 pool.append(rec)
@@ -107,10 +96,148 @@ def main():
                           f"energy {es*100:5.2f}%, routing {rs*100:5.3f}%, "
                           f"ratio {es/max(rs,1e-12):6.1f}")
         total += len(chosen)
+    return lines, report, total
+
+
+def parity_repair(chosen, by_half, report, only=None):
+    """Add the next-best candidate to every half — or to `only` — holding an odd count.
+
+    The four-bit GEMM takes each half's remainder as two equal tables, so an odd count
+    of bf16 experts on a half is not a style problem: the two tables would not match.
+    `by_half` is energy-descending, so the first unchosen candidate is the next most
+    energetic, which is the expert worth keeping. Returns how many were added.
+    """
+    added = 0
+    for key in ([only] if only is not None else sorted(by_half)):
+        layer, half = key
+        n = sum(1 for (l, x) in chosen if l == layer and x // 256 == half)
+        if n % 2 == 0:
+            continue
+        for e, es, rs in by_half.get(key, []):
+            if (layer, e) not in chosen:
+                chosen[(layer, e)] = (es, rs)
+                added += 1
+                report.append(f"blk.{layer}: half {half} odd -> added e{e} "
+                              f"(energy {es*100:.2f}%)")
+                break
+        else:
+            report.append(f"blk.{layer}: half {half} holds {n} and no candidate is "
+                          f"left to make it even")
+    return added
+
+
+def select_to_count(data, min_share, min_route_mult, target):
+    """(lines, report, total, cutoff) for the set that LANDS the count.
+
+    The bars decide the base set; the count is then filled from the energy ranking,
+    one expert at a time, each addition taking its half's next-best candidate with it
+    so every rank's half stays even. Only the RATIO bar walks down to find candidates
+    — it is the law that says an expert is disproportionate and not merely busy, and
+    the caller's byte bill has to be filled — while the ENERGY bar holds, because that
+    bar is what the extra GiB of bf16 is supposed to buy. `cutoff` is the energy share
+    the set bottoms out at: the bar this campaign actually converted with.
+    """
+    report = []
+    shares, ranked = {}, []
+    for layer in sorted(data):
+        experts = data[layer]
+        tot_e = sum(v[2] for v in experts.values())
+        tot_r = sum(v[1] for v in experts.values())
+        if tot_e <= 0 or tot_r <= 0:
+            report.append(f"blk.{layer}: empty recording, skipped")
+            continue
+        for e, (_rank, rows, tr) in experts.items():
+            es, rs = tr / tot_e, rows / max(tot_r, 1)
+            if es <= 0:
+                continue
+            shares[(layer, e)] = (es, rs)
+            ranked.append((es, layer, e, rs))
+    ranked.sort(key=lambda r: (-r[0], r[1], r[2]))
+    by_half = {}
+    for es, layer, e, rs in ranked:
+        by_half.setdefault((layer, e // 256), []).append((e, es, rs))
+
+    chosen = {k: v for k, v in shares.items()
+              if v[0] >= min_share and v[0] >= min_route_mult * v[1]}
+    total = len(chosen)
+    total += parity_repair(chosen, by_half, report)
+
+    mult = min_route_mult
+    while True:
+        pool = [(es, l, e, rs) for es, l, e, rs in ranked
+                if es >= mult * rs and (l, e) not in chosen]
+        if len(pool) + total >= target or mult <= 0.0:
+            break
+        mult -= 0.5
+        report.append(f"ratio bar {mult + 0.5:g} leaves {len(pool)} fill candidates "
+                      f"for {target - total} slots; relaxing toward 0")
+    for es, layer, e, rs in pool:
+        if total >= target:
+            break
+        if (layer, e) in chosen:
+            continue
+        chosen[(layer, e)] = (es, rs)
+        total += 1
+        report.append(f"blk.{layer}: count fill -> added e{e} (energy {es*100:.2f}%)")
+        total += parity_repair(chosen, by_half, report, only=(layer, e // 256))
+    lines, cutoff = [], None
+    for (layer, e) in sorted(chosen):
+        es, rs = chosen[(layer, e)]
+        cutoff = es if cutoff is None else min(cutoff, es)
+        lines.append(f"blk.{layer}.ffn_*_exps.{e}.weight         cast dtype=bf16")
+    return lines, report, total, cutoff
+
+
+def main():
+    ap = argparse.ArgumentParser()
+    ap.add_argument("calib_dir")
+    ap.add_argument("--min-share", type=float, default=0.03)
+    ap.add_argument("--min-route-mult", type=float, default=3.0)
+    ap.add_argument("--target-count", type=int, default=0)
+    a = ap.parse_args()
+
+    # layer -> global expert -> (rank, rows, energy)
+    data = {}
+    for path in sorted(glob.glob(os.path.join(
+            a.calib_dir, "gram.r*.ffn_down_exps.*.bin"))):
+        m = NAME_RE.match(os.path.basename(path))
+        if not m:
+            continue
+        rank, layer, expert = int(m.group(1)), int(m.group(2)), int(m.group(3))
+        k, _, rows, tr = read_gram(path)
+        data.setdefault(layer, {})[expert] = (rank, rows, tr)
+
+    if not data:
+        raise SystemExit("no per-expert down grams found")
+
+    if a.target_count > 0:
+        lines, report, total, cutoff = select_to_count(
+            data, a.min_share, a.min_route_mult, a.target_count)
+        bar = f"energy cutoff {cutoff:.3g}" if cutoff is not None else "no candidates"
+    else:
+        lines, report, total = select(data, a.min_share, a.min_route_mult)
+        bar = f"min-share {a.min_share:.3g}"
+
+    # The emitted lines ARE the contract with the four-bit GEMM: every rank's half of
+    # every layer must hold an even number of bf16 experts. Judged on the output and
+    # not on the internal sets, so a slip in either pass is caught here rather than at
+    # engine load time.
+    halves = collections.Counter()
+    for rule in lines:
+        m = CAST_RE.match(rule)
+        if m:
+            halves[(int(m.group(1)), int(m.group(2)) // 256)] += 1
+    odd = {key: n for key, n in halves.items() if n % 2}
+    if odd:
+        print("PARITY LAW BROKEN: " + ", ".join(
+            f"blk.{lay} half {half} holds {n} protected experts"
+            for (lay, half), n in sorted(odd.items())), file=sys.stderr)
+        raise SystemExit(2)
 
     print("\n".join(lines))
     print("\n".join(report), file=sys.stderr)
-    print(f"total protected: {total}", file=sys.stderr)
+    print(f"total protected: {total} (target {a.target_count or 'none'}, {bar}, "
+          f"min-route-mult {a.min_route_mult:g})", file=sys.stderr)
 
 
 if __name__ == "__main__":

scripts/download-frank-dragoon-npu.sh

@@ -8,8 +8,10 @@
 # (/opt/halogen/npu/models.txt inside the image) and refuses to load a model
 # whose files do not match.
 #
-# The manifest below IS that catalog block (dumped from the pinned 0.16.2
+# The manifest below IS that catalog block (dumped from the pinned 0.18.20
-# image on 2026-10-19), so the files land in the layout the entrypoint
+# image on 2026-10-30; the 0.16.2 block this replaced named different device
+# programs — 10 `u*.elf` and a 5,086,496 B consts.hnpb — so a version bump
+# re-fetches the set or the entrypoint refuses the boot), so the files land in the layout the entrypoint
 # expects — <dir>/<model-id>/devices/*, <dir>/<model-id>/<model-id>.hnpw and
 # <dir>/<model-id>/tokenizer/tokenizer.json — and the endpoint can keep
 # /models read-only and open no outbound connection at boot (HALOGEN_DOWNLOAD
@@ -57,21 +59,25 @@ manifest() {
     case "$1" in
       qwen3-embedding-0.6b)
         NPU_REPO="peonist-ai/halogen-npu-qwen3-embedding-0.6b"
-        NPU_REV="4960fd9d4423dd15373ad21c83db48dbdb3b2798"
+        NPU_REV="8fd44c9f18861618a509bea75c51ff8a37e3bd5d"
         FILES=(
         "devices/NOTICE.md 3248 0f267b3d2523ddf14997ee58660a492a9eaf10ccf6b08bf19dfdc85b9a478b91"
-        "devices/consts.hnpb 5086496 ae430e41545a0b1012938cda373b0c450364d3be631111cca3a8c43679f93336"
+        "devices/consts.hnpb 8495512 5af7c1f2f24d45a9054872bba882461cf7f6ffda832e785b700643d6467ad88b"
-        "devices/devices.hnpm 24047 4e9bca9d95d084fb89ce5b99f579a00666c8ca1efae2613c53e3a4878cc9f174"
+        "devices/devices.hnpm 35465 85d870f6b900de1829659955f0dc701d57984580f5e5ec251add5d6969f03e3d"
-        "devices/u0.elf 4931968 b38f165886cb037b7d3951d9cb878473eaba58a432acd45416ff85fd88aa4e46"
+        "devices/u0.elf 2975008 f74df12eb8c79e4e90cbedcc1dd8f142219772d400e3a142fde8a95378b105eb"
-        "devices/u1.elf 5456000 b1811feee851b8af833b37721da92099d89772110815bbbe2c3fa1c276f0ff42"
+        "devices/u1.elf 3266848 ac5a3f89f934951b315b62ab53e1d0a74dbe6396134192641c405d08da332754"
-        "devices/u2.elf 6505552 406ea9a95b470628a7382e376f4a72cdf4d7aa4c59b4ddf0a4bdd139d63d6ff2"
+        "devices/u2.elf 4015328 5e4f0d3342f262816289866d7e8c966f56a9f938c2e12e7341718cc7312aa480"
-        "devices/u3.elf 6817216 bc205d51b523d770151a0c61968f7c70bf624412827580bb6fe0aaf4650dc2ee"
+        "devices/u3.elf 4326976 94cf5d269fa286d91c26bf4654cedf101c047d23c4495b74c9c129cdc33b210f"
-        "devices/u4.elf 7444032 a5a91338a2603acdcd5c11df532033fda2d3f5479ea44ae87f98fafbdff9a62d"
+        "devices/u4.elf 4953792 cbd2796050d4c675f5cb3da117111087d510173e26f262f1b36aab66dd49dc91"
-        "devices/u5.elf 7432416 6c2bf41835569e0eba25ded7c05c7264ae69c6700bdeb4791057d884f2a4c126"
+        "devices/u5.elf 4940112 7bf6a4470546bb888da02af395c07d92a24fb6d43aef27eb2151b3a9eda19e4d"
-        "devices/u6.elf 4941232 919d0038ca088eafddf195f3466ff38db27d3635be1cffa5d4d4b5140902a68e"
+        "devices/u6.elf 6174544 10b9d50b635993e1f9c23775c0b94901d922c1e433cd10910e9f97174235f5ac"
-        "devices/u7.elf 5488304 d904cf8afae9428dd167960549e18cce5441745f18c62f8c50975dffc077d3c2"
+        "devices/u7.elf 2983776 e0b06063b7379d05c85b3f435b819d96fda1223d8025d6d15958854b5711ace9"
-        "devices/u8.elf 4941232 db3aef69b17825854ee0dd2df475125a3f5c872f95c686793d6af1ee83596fa7"
+        "devices/u8.elf 3298656 3bad204f782a68b97e27c1172530b13d7fbb0b495fa95f9504530a1048560644"
-        "devices/u9.elf 5488304 d182727e551084db8fb9600441d8aa373c7f09a3d267ae0a5e921ef92326db1c"
+        "devices/u9.elf 2983776 875e313280358ce116e2a51a29a2c42794b5a9fadb3d48db702c90d18d838c2c"
+        "devices/u10.elf 3298656 ef2e0f399f149a5476955bcf233c6fa2e896fcef1f86ecb0d8113c952c41a8b2"
+        "devices/u11.elf 3298656 68a2519eee2d6a61ccc4ca2007e3186d44c9eb4e81162d742fe2f1756ec5a528"
+        "devices/u12.elf 3298656 2bdc6abfa58cac1d5795a3ef52fe8a9ab4e7c2a4792822d615499a913401f728"
+        "devices/u13.elf 3298656 4a3ce14f7ba15987b9bcbb4a010083e8aef68cd87c20cbe43403a98263179fe2"
         "qwen3-embedding-0.6b.hnpw 813032248 b9f9901429df7b46ec17153aa3899f738adbea8e4356080da0c6e6586fb750d1"
         "tokenizer/tokenizer.json 11423705 def76fb086971c7867b829c23a26261e38d9d74e02139253b38aeb9df8b4b50a"
         ) ;;
@@ -99,7 +105,18 @@ for MODEL in ${MODELS//,/ }; do
                 # 62 GiB fetch hit this on 2026-10-18).
                 url=$(curl -sfIL -o /dev/null -w '%{url_effective}' "$HF/$NPU_REPO/resolve/$NPU_REV/$f") \
                     || { echo "cannot resolve $NPU_REPO/$f" >&2; exit 1; }
-                curl -fL --retry 5 -C - -o "$path" "$url"
+                # Resume ONLY into a genuinely partial file. A staged file of a
+                # DIFFERENT size is an artifact of another revision (a version
+                # bump re-cuts the device programs), and `-C -` against it
+                # appends the tail to the old bytes: the result is the right
+                # BYTE COUNT with the wrong content, which reads as a corrupt
+                # program, not as a fetch fault (measured 2026-10-30 on
+                # consts.hnpb). Fetch those from zero.
+                resume=()
+                if [ -f "$path" ] && [ "$(stat -c %s "$path")" -lt "$want" ]; then
+                    resume=(-C -)
+                fi
+                curl -fL --retry 5 "${resume[@]+"${resume[@]}"}" -o "$path" "$url"
             fi
         fi
         [ -f "$path" ] || { echo "MISSING $path (run without --verify-only)" >&2; exit 1; }

scripts/download-qwen38-bf16.sh

@@ -0,0 +1,249 @@
+#!/usr/bin/env bash
+# download-qwen38-bf16.sh — RUNS ON DANKSERVER. Fetch and verify the vendor
+# BF16 checkpoint Qwen/Qwen3.8-Flash-Next: the source the five-bit campaign
+# converts, and the source the KLD reference is recorded from.
+#
+# Why the base and not a finetune: the campaign converts to compare against the
+# publisher's own iq4r container, and that container was built from this
+# checkpoint. Base-in, base-out; anything else confounds the gate. (The
+# predecessor campaign ran ukisai/Swift-1.5-Qwen3.8-Flash-Next-BF16, whose key
+# set is identical to this repo's but whose weights are not — none of its
+# calibration grams, KLD reference or protected list carries over.)
+#
+# Why re-fetch instead of reusing: every BF16 copy of this family was destroyed
+# in the 2026-10-15 storage purge — only the 113.6 GiB served .rad survived.
+# 335.3 GiB of safetensors, 131 shards, ungated (the predecessor campaign
+# verified the key set against this repo, so no gate letter is needed).
+#
+# Verification is four gates deep, because the failure this repo actually lived
+# through is a fetch that reported success: `curl -o FILE URL` exits 0 on an
+# HTTP 401/403 and writes the error body as the artifact — three shards of
+# identical tiny size and a cheerful DONE in 20 seconds for 104 GiB.
+#
+#   1. shard census: the file count must equal the expected shard count, so a
+#      partially fetched fragment tree (the .partNofM trap) cannot pass.
+#   2. per-shard completeness, from the shard's OWN safetensors header: the
+#      header declares where the tensor data ends, so every shard's true size is
+#      known offline. A torn tail is named here, by file, instead of surfacing
+#      as a load error five hours later.
+#   3. index coverage: every shard model.safetensors.index.json references
+#      exists and is named.
+#   4. total bytes against the number the Hub declared, then, unless --no-hub,
+#      every file's sha256 against the Hub's LFS record — including the shards
+#      already on disk, so a previous partial download is never trusted.
+#
+# The token never enters a command line (the 2026-09-16 leak law): it is read
+# into HF_TOKEN in the environment, where a local process listing is the only
+# reader. This repo is ungated, so the fetch simply runs without one when no
+# token file exists. scripts/hfget.sh is curl+jq only — no Python, no venv, no
+# hf_transfer — and it drops the pickle/ONNX weight duplicates by itself.
+#
+# usage: download-qwen38-bf16.sh [--fetch] [--no-hub] [--dir DIR]
+#                                [--shards N] [--bytes B] [--repo ID]
+#   --fetch    download first (resumable: re-run to continue), then verify
+#   --no-hub   skip the Hub-side sha256 sweep; the other gates always run
+# exit: 0 verified, 1 verification failed, 2 environment / usage error
+set -euo pipefail
+
+REPO="${FRANK_QWEN38_HF_REPO:-Qwen/Qwen3.8-Flash-Next}"
+LOCAL_DIR="${FRANK_QWEN38_CKPT:-/home/dank/models/Qwen3.8-Flash-Next}"
+# Hub census of 2026-10-27: 144 files, 131 of them .safetensors, together
+# 360,023,351,514 bytes; the shards alone are 360,000,192,888. This gate counts
+# shards only, so a config file edited on the Hub cannot fail it.
+EXPECTED_SHARDS="${FRANK_QWEN38_SHARDS:-131}"
+EXPECTED_BYTES="${FRANK_QWEN38_BYTES:-360000192888}"
+MIN_FREE_MIB="${FRANK_QWEN38_MIN_FREE_MIB:-350000}"
+
+FETCH=0
+HUB=1
+while [ $# -gt 0 ]; do
+    case "$1" in
+        --fetch) FETCH=1 ;;
+        --no-hub) HUB=0 ;;
+        --dir)    LOCAL_DIR="$2"; shift ;;
+        --shards) EXPECTED_SHARDS="$2"; shift ;;
+        --bytes)  EXPECTED_BYTES="$2"; shift ;;
+        --repo)   REPO="$2"; shift ;;
+        *) echo "usage: ${0##*/} [--fetch] [--no-hub] [--dir DIR] [--shards N] [--bytes B] [--repo ID]" >&2
+           exit 2 ;;
+    esac
+    shift
+done
+
+SELF_DIR="$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)"
+FAILS=0
+note() { echo "$@"; FAILS=$((FAILS + 1)); }
+
+if [ "$FETCH" = "1" ]; then
+    # Every fetch gate is a hard stop: skipping the download silently and then
+    # verifying a half tree is how a campaign ships a stale payload.
+    HFGET="$SELF_DIR/hfget.sh"
+    if [ ! -f "$HFGET" ]; then
+        echo "REFUSED: the fetch helper is not here: $HFGET" >&2
+        exit 2
+    fi
+    if ! command -v jq >/dev/null 2>&1; then
+        echo "REFUSED: hfget.sh needs jq on PATH" >&2
+        exit 2
+    fi
+    mkdir -p "$LOCAL_DIR"
+    # The campaign's peak-disk law starts at the fetch: the checkpoint and the
+    # reference container built from it are ~335 GiB each, and a tree that does
+    # not fit is discovered hours into a transfer, when it costs the most.
+    free_mib=$(df -m --output=avail "$LOCAL_DIR" | tail -1 | tr -dc '0-9')
+    if [ "${free_mib:-0}" -lt "$MIN_FREE_MIB" ]; then
+        echo "REFUSED: $free_mib MiB free at $LOCAL_DIR; the fetch wants $MIN_FREE_MIB MiB free" >&2
+        echo "         (retire the peak-disk predecessors first — nothing is downloaded, so" >&2
+        echo "         nothing is left half-written)" >&2
+        exit 2
+    fi
+    echo "fetch: $REPO -> $LOCAL_DIR ($free_mib MiB free)"
+    if [ -z "${HF_TOKEN:-}" ]; then
+        CREDSTORE="$HOME/.config/huggingface/credentials/https://huggingface.co"
+        if [ -f "$CREDSTORE" ]; then
+            HF_TOKEN="$(awk '/^ OauthToken: / {print $2; exit}' "$CREDSTORE")"
+        elif [ -f "$HOME/.cache/huggingface/token" ]; then
+            # The legacy single-line token file the hub library writes.
+            HF_TOKEN="$(tr -d '[:space:]' < "$HOME/.cache/huggingface/token")"
+        elif [ -f "$HOME/.hf_token" ]; then
+            HF_TOKEN="$(tr -d '[:space:]' < "$HOME/.hf_token")"
+        fi
+    fi
+    if [ -n "${HF_TOKEN:-}" ]; then
+        export HF_TOKEN
+    fi
+    bash "$HFGET" "$REPO" "$LOCAL_DIR" main
+    unset HF_TOKEN
+fi
+
+if [ ! -d "$LOCAL_DIR" ]; then
+    echo "REFUSED: $LOCAL_DIR does not exist (run with --fetch, or pass --dir)" >&2
+    exit 1
+fi
+INDEX="$LOCAL_DIR/model.safetensors.index.json"
+if [ ! -f "$INDEX" ]; then
+    echo "REFUSED: $INDEX is missing — the checkpoint tree is not addressable without it" >&2
+    exit 1
+fi
+
+echo "== gate 1: shard census ($LOCAL_DIR)"
+SHARDS=$(find "$LOCAL_DIR" -maxdepth 1 -type f -name '*.safetensors' | wc -l)
+if [ "$SHARDS" != "$EXPECTED_SHARDS" ]; then
+    note "FAIL SHARD_COUNT: $SHARDS safetensors files on disk, expected $EXPECTED_SHARDS"
+else
+    echo "  ok: $SHARDS shards"
+fi
+
+echo "== gate 2: every shard is whole, judged by its own header"
+# A safetensors file is [u64 header length][JSON header][tensor data]. The JSON
+# declares where the data ends, so the file's true size is computable offline:
+# 8 + header + the largest end offset. Truncation and error bodies both fall
+# here, and each is reported with the file that caused it.
+while IFS= read -r f; do
+    state=$(python3 - "$f" <<'PY'
+import json, os, struct, sys
+path = sys.argv[1]
+size = os.path.getsize(path)
+try:
+    with open(path, "rb") as fh:
+        (hlen,) = struct.unpack("<Q", fh.read(8))
+        if hlen == 0 or hlen > size:
+            print("NOT_SAFETENSORS")
+            raise SystemExit
+        header = json.loads(fh.read(hlen))
+except Exception:
+    print("NOT_SAFETENSORS")
+    raise SystemExit
+if not header:
+    print("NOT_SAFETENSORS")
+    raise SystemExit
+end = max((v["data_offsets"][1] for v in header.values()
+           if isinstance(v, dict) and "data_offsets" in v), default=0)
+want = 8 + hlen + end
+print("OK" if want == size else f"TRUNCATED want={want} have={size}")
+PY
+)
+    case "$state" in
+        OK) ;;
+        NOT_SAFETENSORS*) note "FAIL NOT_SAFETENSORS: $(basename "$f") is not a safetensors file (an error body written as an artifact?)" ;;
+        TRUNCATED*) note "FAIL TRUNCATED: $(basename "$f") $state" ;;
+    esac
+done < <(find "$LOCAL_DIR" -maxdepth 1 -type f -name '*.safetensors' | sort)
+if [ "$FAILS" -eq 0 ]; then
+    echo "  ok: every shard's data extent matches its size on disk"
+fi
+
+echo "== gate 3: the index's shards are all present"
+INDEX_SHARDS=$(python3 - "$INDEX" <<'PY'
+import json, sys
+map_ = json.load(open(sys.argv[1]))["weight_map"]
+print("\n".join(sorted(set(map_.values()))))
+PY
+)
+while IFS= read -r name; do
+    [ -n "$name" ] || continue
+    if [ ! -f "$LOCAL_DIR/$name" ]; then
+        note "FAIL MISSING: $name is named by the index and is not on disk"
+    fi
+done <<< "$INDEX_SHARDS"
+if [ "$FAILS" -eq 0 ]; then
+    echo "  ok: $(printf '%s\n' "$INDEX_SHARDS" | grep -c .) referenced shards on disk"
+fi
+
+echo "== gate 4: total bytes vs the Hub's declaration"
+TOTAL=$(find "$LOCAL_DIR" -maxdepth 1 -type f -name '*.safetensors' -printf '%s\n' | awk '{s+=$1} END{print s+0}')
+if [ "$TOTAL" != "$EXPECTED_BYTES" ]; then
+    note "FAIL TOTAL_BYTES: shards total $TOTAL bytes, expected $EXPECTED_BYTES"
+else
+    echo "  ok: $TOTAL bytes"
+fi
+
+if [ "$HUB" = "1" ]; then
+    echo "== gate 5: every file's sha256 vs the Hub's LFS record"
+    # size = file size; lfs.sha256 = sha256 of the file itself (LFS or not).
+    TMPD="$(mktemp -d)"
+    trap 'rm -rf "$TMPD"' EXIT
+    MANIFEST="$TMPD/tree.json"
+    PLAN="$TMPD/plan"
+    curl -sf "https://huggingface.co/api/models/$REPO/tree/main?recursive=true&expand=true" \
+        -o "$MANIFEST" || { echo "FAIL: the Hub API call failed — refusing to report success" >&2; exit 1; }
+    python3 - "$MANIFEST" > "$PLAN" <<'PY'
+import json, sys
+for e in json.load(open(sys.argv[1])):
+    if e.get("type") != "file":
+        continue
+    sha = (e.get("lfs") or {}).get("sha256") or e.get("sha") or ""
+    if sha:
+        print(e["path"], e["size"], sha)
+PY
+    BAD=0
+    while read -r rel size sha; do
+        [ -n "${rel:-}" ] || continue
+        if [ ! -f "$LOCAL_DIR/$rel" ]; then
+            note "FAIL MISSING: $rel is on the Hub and not on disk"
+            continue
+        fi
+        have=$(stat -c %s "$LOCAL_DIR/$rel")
+        if [ "$have" != "$size" ]; then
+            note "FAIL SIZE: $rel is $have bytes, Hub says $size"
+            continue
+        fi
+        got=$(sha256sum "$LOCAL_DIR/$rel" | awk '{print $1}')
+        if [ "$got" != "$sha" ]; then
+            note "FAIL SHA256: $rel got $got want $sha"
+            BAD=$((BAD + 1))
+        fi
+    done < "$PLAN"
+    if [ "$BAD" -eq 0 ] && [ "$FAILS" -eq 0 ]; then
+        echo "  ok: sha256 matches for every file"
+    fi
+    rm -rf "$TMPD"
+    trap - EXIT
+fi
+
+if [ "$FAILS" -gt 0 ]; then
+    echo "VERIFY_FAILED: $FAILS problem(s) in $LOCAL_DIR" >&2
+    exit 1
+fi
+echo "VERIFY_OK: $SHARDS shards, $TOTAL bytes of safetensors in $LOCAL_DIR"
+echo "NEXT: build the BF16 reference container with scripts/build-qwen38-bf16-rad.sh"

scripts/frank-archon.env

@@ -74,6 +74,13 @@ FRANK_ARCHON_PLACEMENT=expert_tiered
 # residency -- ~3.2 GiB a card of experts now lives in host RAM and is read over the
 # x8 link when the router picks it. The ledger's coverage line is the referee, not
 # this file: if vram_kv comes back under 700,000 tokens, raise this 1024 and redeploy.
+# 2026-10-29 A/B, measured on both payloads and folded back the same day: at 66,560 the
+# endpoint printed the identical `elastic 26.73 GiB / experts 20.72 / kv cache 5.64 /
+# vram_weights 24.30` it had printed at 49,152, opening the bigger pool with `0 B used`;
+# the campaign rig (bf16, same step) moved `vram_kv 269.99 -> 271.25 MiB`, a rounding
+# step — its window is fixed by its own n-gram table (95.37 GiB) and expert plane, not
+# by this pool, so the refused calibration documents were fixed by cutting them to the
+# window (scripts/rechunk-kld-corpus.py), not by widening the pool.
 FRANK_ARCHON_HOST_POOL_MIB=49152
 # VRAM left unclaimed per card (publisher default 96 MiB); the engine's settle
 # floor is built from it.
@@ -137,7 +144,8 @@ FRANK_ARCHON_NGRAM_PLACEMENT=ram
 # itself before refusing by name, so recovery is a boot, not a visit.
 # Floor = 2 x HOST_POOL + 48,824 (static weights + arena) + PREFIX_CACHE + MNBT + the
 # auto checkpoint store (MAX_SEQS x 10 slots of 111.98 MiB), per the bats ledger test;
-# at the 49,152 pool with 4 slots that computes to 161,848 MiB.
+# at the 49,152 pool with 4 slots that computes to 161,848 MiB (196,664 at the
+# 66,560 pool the 2026-10-29 A/B rejected).
 FRANK_ARCHON_TTM_MIN_MIB=162000
 FRANK_ARCHON_TTM_TARGET_MIB=204800
 # Escape hatch for experiments. Unset -> no extra argv; the serve script expands

scripts/frank-dragoon.env

@@ -10,8 +10,11 @@
 # The .hgn format is the lock: it loads ONLY in the flash-server line, so the
 # engine moves with the payload, never ahead of it.
 # Image digest-pinned (a tag pull silently re-points; the digest is what was
-# qualified — 0.16.2 is the version the model card names as its quickstart pin).
+# qualified). 0.18.20 since 2026-10-30 (was 0.16.2, the model card's quickstart
-FRANK_DRAGOON_IMAGE=ghcr.io/peonist-ai/halogen-flash-server@sha256:0c61bf84ac22308a53f5d1ca6b86806702d7039e5ebc51cae4c66621b92fe04a
+# pin) — the vendor has moved two minor versions and the payload lock is what
+# the deploy's health gate plus the boot ledger decide; rollback is the
+# previous digest, 0c61bf84…, on this same line.
+FRANK_DRAGOON_IMAGE=ghcr.io/peonist-ai/halogen-flash-server@sha256:e35133b206da3094cc4d70c78800b26235bd9124b9aeb0e029d4a7c153b9cff0
 # Container identity kept from the 27B era: the unit's ExecStop names it.
 FRANK_DRAGOON_CONTAINER=frank-dragoon-halogen
 # Predecessor identity the serve script clears if it is still holding the GPUs.

scripts/frank-zealot.env

@@ -1,12 +1,12 @@
-# frank-zealot — llama.cpp (PrismML fork build 10777, CUDA) on the RTX 3090,
+# frank-zealot — llama.cpp (PrismML fork build 12118 = 0.7.0-dev, commit
-# :7771. Payload: Ternary-Bonsai-2-9B PQ2_0 with the MTP head in the file, plus
+# 024474448, CUDA) on the RTX 3090, :7771. Payload: Ternary-Bonsai-2-9B PQ2_0 with the MTP head in the file, plus
 # the refusal-direction adapter applied at load.
 #
 # ONE OWNER: shape, payload identity, eras, laws and every measured number
 # live in serverAppConfigs-docs/LLM/frank-zealot.md. This file carries the
 # knobs and only the reason a reader editing THIS line cannot recover
 # elsewhere. Where the two disagree, this file wins on values.
-LLAMA_BIN=/home/dank/GitPackages/llama-prism-b10777/llama-server
+LLAMA_BIN=/home/dank/GitPackages/llama-prism-0244744/bin/llama-server
 # The fork is a LOCK, not a preference: custom tensor type 142 + prism.hadamard.*
 # metadata. Stock llama.cpp refuses the file; a loader ignoring the Hadamard
 # metadata dequantizes wrong, silently.
@@ -64,7 +64,8 @@ CACHE_TYPE_V=q8_0
 # weights, so there is no --spec-draft-model. Its bill is pool × 4,096 B of
 # draft KV plus ~0.45 GiB of buffers and a tripled recurrent state — paid for
 # by CACHE_TYPE_V=q8_0 and UBATCH_SIZE=1024 above. Build gate: quantized V
-# beside MTP needs fork build >= b10776; LLAMA_BIN is 10777.
+# beside MTP needs fork build >= b10776; LLAMA_BIN is 12118 (b10777 is kept on
+# disk as the rollback path).
 # SPEC_TYPE=none is the rollback and strips the WHOLE set (--spec-type and
 # every --spec-draft-*); half a set has segfaulted at load.
 SPEC_TYPE=draft-mtp

scripts/grams-to-imatrix.py

@@ -0,0 +1,418 @@
+#!/usr/bin/env python3
+"""grams-to-imatrix.py — the engine's own calibration recording, re-voiced as a
+llama.cpp importance matrix for `rad-convert --imatrix F`. RUNS ON DANKSERVER
+beside the recording. Python 3 stdlib only: numpy must NOT be required, because
+this runs in the plain container next to a 335 GiB checkpoint, not in a venv.
+
+WHY. `rad-convert` takes its per-column activation second moments and its
+expert-popularity profile from ONE input, an imatrix (core/format/imatrix.h).
+Only the GGUF form carries a per-expert dimension — with the legacy flat form,
+or with none, every expert in a layer is weighted by the layer's aggregate,
+which is exactly the wrong thing on a 512-expert MoE. The calibration tap has
+already measured everything; this tool only re-voices it, so the imatrix and the
+protected-expert census (derive-protected-experts.py) are the SAME recording:
+one measurement, two consumers, neither able to disagree about which experts are
+hot.
+
+INPUT. RADGRAM2 files in a RADIANCE_CALIB_DIR output directory
+(arch/common/rad_arch.h, calib_write). Header law, same parse as
+scripts/derive-protected-experts.py — 32 bytes, `<8sIIQI`: magic `RADGRAM2`,
+u32 k, u32 rank, u64 rows, u32 domain, a pad word, then k*k f32 row-major. Two
+kinds of name, both GGUF-style, both with the checkpoint's GLOBAL expert
+numbering (rad_block_moe_fp8.h: ge = expert_base + e):
+
+    gram.r<rank>.blk.L.ffn_down_exps.bin        the layer's pooled gram
+    gram.r<rank>.blk.L.ffn_down_exps.<e>.bin    expert e's own gram
+
+`rows` is that file's routing/token count and the matrix is Σ x xᵀ over those
+rows, so **the diagonal is exactly the per-column activation second moment an
+imatrix wants**, and `rows` is the count the reader hands back from `count()`.
+Every off-diagonal term is dropped on purpose: the GGUF format has nowhere to
+put a full matrix, and the quantiser asks for columns.
+
+OUTPUT, per layer L (core/format/imatrix.h: names WITHOUT `.weight`, a MoE
+layer's experts folded into ONE entry, so expert 9 is slice 9 of
+`blk.L.ffn_down_exps`):
+
+  blk.L.ffn_down_exps    in_sum2 [k, n_expert] expert-major, counts [1, n_expert]
+  blk.L.ffn_gate_up_exps the SAME matrices at the SAME offsets. A routed expert's
+                         gate_up input is the same hidden-state rows as its down
+                         input, so the second moment over them is identical and
+                         so is the routing count. Two tensor infos over one
+                         payload: the reader maps each half independently
+                         (imatrix.cpp joins `<base>.in_sum2` with `<base>.counts`
+                         by base name), and copying ~200 MB to state one fact
+                         twice buys no correctness.
+  blk.L.ssm_inz          [k] and [1], from the pooled gram: that row's input is
+                         the same hidden state, so its columns take the pooled
+                         diagonal and its count is the layer's pooled rows.
+
+  Then STOP. Nothing whose input width is not the gram's k — no ssm_out, no
+  attn_output, no mtp.*, no ple_ngram. importance_for() answers false when the
+  entry's cols do not equal the weight's width and the convert proceeds without
+  it: an entry the reader silently ignores is a lie in the artifact, worse than
+  an absent one. The corollary to check before trusting the two hidden-width
+  entries: the tap writes the gate_up gram at g.n_embd and the down gram at the
+  expert width, so the gram's k must equal the checkpoint's hidden width (its
+  config.json). Where they differ, gate_up/ssm_inz are inert, not wrong.
+
+INHERITANCE. An expert with no own gram carries the layer's POOLED diagonal, and
+its count stays its own routing rows — the rule the engine itself applies when it
+picks a Hessian (libquant/lq_hess.h, calib_resolve: an expert's own gram only
+once its rank files hold kOwnRowsPerK * K rows; below that a fit to so few rows
+is worse than round-to-nearest, and those experts carry a fraction of a percent
+of the routing). The count never inherits: popularity is `count`'s job. The count
+of inherited experts is reported on stderr.
+
+RANK FOLDING. TP=2 writes gram.r0.* and gram.r1.* with GLOBAL expert numbering,
+so the ranks describe DISJOINT experts and the fold is a UNION into the one
+[k, n_expert] entry. A pooled gram is the opposite case: every rank summed the
+same stem over its own rows, so the fold is a SUM, rows included (calib_sum,
+calib_rows). The same global expert in two per-expert files is legal only when
+the numbers agree exactly; otherwise the recording is corrupt and the entry would
+be whichever file sorted last, so refuse.
+
+GGUF v3 (the only version imatrix.cpp accepts — version != 3 is a refusal):
+`general.alignment` u32, `imatrix.chunk_size` u32, `imatrix.chunk_count` u32,
+`imatrix.datasets` array of strings; tensors f32 (ggml type 0) with offsets
+relative to the aligned data section. Every tensor offset is padded up to
+`general.alignment` and the data section itself starts aligned. That is the
+classic hand-rolled-writer bug: an unaligned offset makes the reader's extent
+arithmetic point into the middle of the previous tensor's floats, and it reads
+them without complaint.
+
+chunk_count DERIVATION. The corpus row count is the LARGEST single-layer pooled
+row total, not a sum over layers: every layer's tap accumulated over the whole
+corpus, so adding the layers counts each token once per layer and inflates the
+figure by the layer count. chunk_count = that row total // chunk_size, the
+"chunks of chunk_size" line imatrix.cpp prints for itself.
+
+REFUSALS, each exit 2 with a message naming what is missing:
+  NO_GRAMS         the directory holds no RADGRAM2 down-projection gram
+  NOT_RADGRAM2     a gram whose magic is not RADGRAM2
+  SHORT            a gram whose length is not 32 + k*k*4
+  NO_POOLED        a layer with expert grams and no pooled gram to inherit from
+  K_MISMATCH       a layer whose files disagree about k (rank pooled grams, or an
+                   expert gram against its layer's pooled gram)
+  EXPERT_CONFLICT  a global expert recorded twice with different numbers
+  EXISTS           the -o path already holds a file (the trash law: move it)
+  CHUNK_COUNT      the derived chunk count does not fit the u32 metadata field
+
+USAGE. grams-to-imatrix.py <calib-dir> -o <out.gguf>
+       [--datasets NAME,...] [--chunk-size N] [--min-rows N] [-v]
+"""
+import argparse
+import os
+import re
+import struct
+import sys
+
+MAGIC = b"RADGRAM2"
+HDR = 32                       # 8s I I Q I + pad, rad_arch.h
+KOWN_ROWS_PER_K = 4            # libquant/lq_hess.h kOwnRowsPerK
+ALIGNMENT = 32                 # general.alignment, and the pad unit
+
+POOLED_RE = re.compile(r"^gram\.r(\d+)\.blk\.(\d+)\.ffn_down_exps\.bin$")
+EXPERT_RE = re.compile(r"^gram\.r(\d+)\.blk\.(\d+)\.ffn_down_exps\.(\d+)\.bin$")
+
+GT_U32, GT_STR, GT_ARR = 4, 8, 9
+GGML_F32 = 0
+
+
+def refuse(msg):
+    sys.stderr.write("REFUSED %s\n" % msg)
+    sys.exit(2)
+
+
+def read_gram(path):
+    """(k, rank, rows, domain, diagonal) from one RADGRAM2 file."""
+    try:
+        with open(path, "rb") as f:
+            raw = f.read()
+    except OSError as e:
+        refuse("UNREADABLE %s: %s" % (path, e))
+    if len(raw) < HDR:
+        refuse("SHORT %s: %d bytes, under the %d-byte RADGRAM2 header"
+               % (path, len(raw), HDR))
+    magic, k, rank, rows, domain = struct.unpack_from("<8sIIQI", raw, 0)
+    if magic != MAGIC:
+        refuse("NOT_RADGRAM2 %s: magic %r, expected b'RADGRAM2'" % (path, magic))
+    if k <= 0:
+        refuse("SHORT %s: header claims k=%d, which describes no matrix" % (path, k))
+    need = HDR + k * k * 4
+    if len(raw) != need:
+        refuse("SHORT %s: %d bytes, expected %d (32 + k*k*4 for k=%d)"
+               % (path, len(raw), need, k))
+    # The diagonal only, read by stride out of the row-major matrix.
+    diag = [struct.unpack_from("<f", raw, HDR + (i * k + i) * 4)[0]
+            for i in range(k)]
+    return k, rank, rows, domain, diag
+
+
+def load_dir(d, verbose):
+    """{layer: {"pooled": gram|None, "experts": {global e: gram}}} + rotated count.
+
+    Files outside the ffn_down_exps stem are not ours to fold: the tap names a
+    pooled gram after every projection it taps, and those carry other widths."""
+    if not os.path.isdir(d):
+        refuse("NO_GRAMS %s: not a directory" % d)
+    layers, rotated = {}, 0
+    for name in sorted(os.listdir(d)):
+        m = POOLED_RE.match(name) or EXPERT_RE.match(name)
+        if not m:
+            continue
+        path = os.path.join(d, name)
+        if not os.path.isfile(path):
+            continue
+        pooled = bool(POOLED_RE.match(name))
+        layer = int(m.group(2))
+        k, rank, rows, domain, diag = read_gram(path)
+        if domain != 0:
+            rotated += 1
+        L = layers.setdefault(layer, {"pooled": None, "experts": {}})
+        if pooled:
+            P = L["pooled"]
+            if P is None:
+                L["pooled"] = {"k": k, "rows": rows, "diag": diag, "paths": [path]}
+            elif P["k"] != k:
+                refuse("K_MISMATCH layer %d: pooled grams %s (k=%d) and %s (k=%d)"
+                       " disagree about the width"
+                       % (layer, P["paths"][0], P["k"], path, k))
+            else:
+                # calib_sum: each rank summed the same stem over its own rows.
+                P["rows"] += rows
+                P["paths"].append(path)
+                for i in range(k):
+                    P["diag"][i] += diag[i]
+            if verbose:
+                sys.stderr.write("pooled layer %d: %s k=%d rows=%d\n"
+                                 % (layer, name, k, rows))
+            continue
+        e = int(m.group(3))
+        own = L["experts"].get(e)
+        if own is None:
+            L["experts"][e] = {"k": k, "rows": rows, "diag": diag,
+                               "path": path, "ranks": [rank]}
+            if verbose:
+                sys.stderr.write("expert layer %d r%d e%d: rows=%d\n"
+                                 % (layer, rank, e, rows))
+            continue
+        if own["rows"] != rows or own["diag"] != diag:
+            refuse("EXPERT_CONFLICT layer %d expert %d: %s (rows=%d) and %s"
+                   " (rows=%d) record the same global expert with different"
+                   " numbers" % (layer, e, own["path"], own["rows"], path, rows))
+        own["ranks"].append(rank)
+        sys.stderr.write("note: layer %d expert %d recorded identically by rank %s"
+                         " and %s; folded once\n"
+                         % (layer, e, own["path"], name))
+    if not layers:
+        refuse("NO_GRAMS %s: no RADGRAM2 gram.r<rank>.blk.<L>.ffn_down_exps"
+               "[.<expert>].bin files in the directory" % d)
+    return layers, rotated
+
+
+def build_entries(layers, min_rows, verbose):
+    """([ (name, dims, payload|None, key) ], census, layers emitted)."""
+    entries = []
+    census = {"inherited": 0, "experts": 0, "routing": 0, "rows": 0, "layers": 0}
+    for layer in sorted(layers):
+        P, E = layers[layer]["pooled"], layers[layer]["experts"]
+        if E and P is None:
+            refuse("NO_POOLED layer %d: %d expert gram(s) but no pooled gram (an"
+                   " expected gram.r<rank>.blk.%d.ffn_down_exps.bin), so an expert"
+                   " under the row threshold would have nothing to inherit"
+                   % (layer, len(E), layer))
+        if P is None:
+            continue
+        k = P["k"]
+        census["layers"] += 1
+        census["rows"] = max(census["rows"], P["rows"])
+        dn = "blk.%d.ffn_down_exps" % layer
+        # A staging key names ONE payload. A gate_up entry reuses the down
+        # layer's bytes by carrying the same key with no payload of its own;
+        # in_sum2 and counts are different bytes and must never share a key.
+        ksum, kcnt = (layer, "sum2"), (layer, "counts")
+        kpgm, kpcnt = (layer, "pooled_sum2"), (layer, "pooled_counts")
+        # One entry holds one width: an expert gram recorded at another k would
+        # inherit (or be inherited by) a diagonal of the wrong length, and the
+        # reader would then answer `cols` from whichever file set dims[0].
+        for e in sorted(E):
+            if E[e]["k"] != k:
+                refuse("K_MISMATCH layer %d expert %d: %s holds k=%d and this"
+                       " layer's pooled gram is k=%d" % (layer, e, E[e]["path"],
+                                                          E[e]["k"], k))
+
+        if not E:
+            # No per-expert recording for this layer: emit the dense form
+            # ([k], [1]) rather than 512 identical slices, which would claim a
+            # per-expert measurement that was never taken.
+            entries.append((dn + ".in_sum2", [k],
+                            struct.pack("<%df" % k, *P["diag"]), ksum))
+            entries.append((dn + ".counts", [1],
+                            struct.pack("<f", float(P["rows"])), kcnt))
+            sys.stderr.write("note: layer %d has no per-expert grams; %s is"
+                             " dense over the pooled gram\n" % (layer, dn))
+        else:
+            floor = min_rows if min_rows > 0 else KOWN_ROWS_PER_K * k
+            n_expert = max(E) + 1
+            sum2, counts = bytearray(), bytearray()
+            for e in range(n_expert):
+                own = E.get(e)
+                if own is not None and own["rows"] >= floor:
+                    src = own["diag"]
+                    rows = own["rows"]
+                else:
+                    src = P["diag"]
+                    rows = own["rows"] if own is not None else 0
+                    census["inherited"] += 1
+                sum2 += struct.pack("<%df" % k, *src)
+                counts += struct.pack("<f", float(rows))
+                census["experts"] += 1
+                census["routing"] += rows
+            if min(E) != 0:
+                sys.stderr.write("note: layer %d has no gram for experts 0..%d;"
+                                 " those slices carry the pooled gram at count 0\n"
+                                 % (layer, min(E) - 1))
+            entries.append((dn + ".in_sum2", [k, n_expert], bytes(sum2), ksum))
+            entries.append((dn + ".counts", [1, n_expert], bytes(counts), kcnt))
+            gu = "blk.%d.ffn_gate_up_exps" % layer
+            entries.append((gu + ".in_sum2", [k, n_expert], None, ksum))
+            entries.append((gu + ".counts", [1, n_expert], None, kcnt))
+            if verbose:
+                sys.stderr.write("layer %d: k=%d n_expert=%d pooled_rows=%d"
+                                 " own-gram floor=%d\n"
+                                 % (layer, k, n_expert, P["rows"], floor))
+
+        inz = "blk.%d.ssm_inz" % layer
+        entries.append((inz + ".in_sum2", [k],
+                        struct.pack("<%df" % k, *P["diag"]), kpgm))
+        entries.append((inz + ".counts", [1],
+                        struct.pack("<f", float(P["rows"])), kpcnt))
+    return entries, census
+
+
+def enc_str(v):
+    b = v.encode("utf-8")
+    return struct.pack("<Q", len(b)) + b
+
+
+def write_gguf(path, entries, datasets, chunk_size, chunk_count):
+    """Write GGUF v3 and return its size in bytes.
+
+    One payload per key: entries that share a key share the bytes and the
+    offset, so a second tensor info states the same fact without copying it."""
+    kv = bytearray()
+    kv += enc_str("general.alignment") + struct.pack("<I", GT_U32) \
+        + struct.pack("<I", ALIGNMENT)
+    kv += enc_str("imatrix.chunk_size") + struct.pack("<I", GT_U32) \
+        + struct.pack("<I", chunk_size)
+    kv += enc_str("imatrix.chunk_count") + struct.pack("<I", GT_U32) \
+        + struct.pack("<I", chunk_count)
+    kv += enc_str("imatrix.datasets") + struct.pack("<I", GT_ARR) \
+        + struct.pack("<I", GT_STR) + struct.pack("<Q", len(datasets))
+    for d in datasets:
+        kv += enc_str(d)
+
+    head = bytearray()
+    head += b"GGUF" + struct.pack("<I", 3)
+    head += struct.pack("<Q", len(entries)) + struct.pack("<Q", 4)
+    head += kv
+
+    payloads = {}
+    for name, _dims, payload, key in entries:
+        if payload is None:
+            continue
+        prev = payloads.get(key)
+        if prev is not None and prev != payload:
+            # Two different payloads on one key would ship whichever was
+            # appended last while BOTH tensor infos pointed at it: the counts
+            # payload landing over the in_sum2 payload is exactly how a
+            # hand-rolled GGUF writer loses half its data without an error.
+            refuse("INTERNAL %s: two different payloads share one staging key"
+                   % name)
+        payloads.setdefault(key, payload)
+    infos = bytearray()
+    offsets, order, size = {}, [], 0
+    for name, dims, payload, key in entries:
+        if key not in payloads:
+            refuse("INTERNAL %s: no payload staged for its key" % name)
+        if key not in offsets:
+            offsets[key] = (size + ALIGNMENT - 1) // ALIGNMENT * ALIGNMENT
+            size = offsets[key] + len(payloads[key])
+            order.append(key)
+        infos += enc_str(name) + struct.pack("<I", len(dims))
+        for d in dims:
+            infos += struct.pack("<Q", d)
+        infos += struct.pack("<I", GGML_F32) + struct.pack("<Q", offsets[key])
+
+    body = bytes(head) + bytes(infos)
+    data_start = (len(body) + ALIGNMENT - 1) // ALIGNMENT * ALIGNMENT
+    with open(path, "wb") as f:
+        f.write(body)
+        f.write(b"\0" * (data_start - len(body)))
+        # The offsets in the tensor infos are the file's law, not a hint: each
+        # payload must land at its own padded offset. Writing them back to back
+        # instead leaves every reader that trusts the info — imatrix.cpp does —
+        # reading its neighbour's floats: plausible values, no error, wrong
+        # quantisation.
+        here = data_start
+        for key in order:
+            f.write(b"\0" * (data_start + offsets[key] - here))
+            f.write(payloads[key])
+            here = data_start + offsets[key] + len(payloads[key])
+        return f.tell()
+
+
+def main():
+    ap = argparse.ArgumentParser(
+        description="Fold a radiance calibration recording into a GGUF v3 imatrix.")
+    ap.add_argument("calib_dir")
+    ap.add_argument("-o", "--out", required=True)
+    ap.add_argument("--datasets", default="",
+                    help="comma-separated names for imatrix.datasets")
+    ap.add_argument("--chunk-size", type=int, default=4096,
+                    help="rows per chunk for imatrix.chunk_size (default 4096)")
+    ap.add_argument("--min-rows", type=int, default=0,
+                    help="own-gram row threshold (default 4*k, the engine's"
+                         " kOwnRowsPerK law)")
+    ap.add_argument("-v", "--verbose", action="store_true")
+    a = ap.parse_args()
+
+    if os.path.exists(a.out):
+        # The repo's deletion law: an artifact that exists is moved to trash by
+        # a human, never silently overwritten by a re-run of the tool that made
+        # it, so a second convert cannot destroy the imatrix the first one fed.
+        refuse("EXISTS %s: an imatrix is already there; move it to"
+               " .AGENTTRASH/ or name another -o" % a.out)
+    if a.chunk_size <= 0:
+        refuse("BAD_CHUNK_SIZE --chunk-size %d is not positive" % a.chunk_size)
+    layers, rotated = load_dir(a.calib_dir, a.verbose)
+    entries, census = build_entries(layers, a.min_rows, a.verbose)
+    if not entries:
+        refuse("NO_GRAMS %s: no layer holds a pooled gram to fold" % a.calib_dir)
+    if rotated:
+        sys.stderr.write("note: %d gram(s) recorded in a rotated domain: their"
+                         " diagonal is the rotated activation's second moment,"
+                         " not the model's own\n" % rotated)
+
+    chunk_count = census["rows"] // a.chunk_size
+    if chunk_count > 0xFFFFFFFF:
+        # imatrix.chunk_count is a u32 in the reader's grammar, so there is no
+        # legal way to record a bigger one. Reaching this means either a chunk
+        # size picked for a far larger run or a row total read out of the wrong
+        # header offset — rows is a u64 and a u32 parse is off by 2**32 — and
+        # the reader would trust whatever number landed in the field.
+        refuse("CHUNK_COUNT %d = %d rows // %d does not fit the u32"
+               " imatrix.chunk_count field; raise --chunk-size"
+               % (chunk_count, census["rows"], a.chunk_size))
+    nbytes = write_gguf(a.out, entries, [d for d in a.datasets.split(",") if d],
+                        a.chunk_size, chunk_count)
+    sys.stderr.write("IMATRIX_OK entries=%d layers=%d experts=%d inherited=%d"
+                     " rows=%d routing=%d chunks=%d/%d bytes=%d out=%s\n"
+                     % (len(entries), census["layers"], census["experts"],
+                        census["inherited"], census["rows"], census["routing"],
+                        chunk_count, a.chunk_size, nbytes, a.out))
+
+
+if __name__ == "__main__":
+    main()

scripts/kld-gate-qwen38.sh

@@ -0,0 +1,186 @@
+#!/usr/bin/env bash
+# kld-gate-qwen38.sh — RUNS ON DANKSERVER (or anywhere the reports are). The campaign's
+# acceptance gate: one verdict from the KLD reports of the containers that ride one reference
+# recording.
+#
+# The reference is the BF16 container built from the vendor checkpoint (run-qwen38-bf16.sh
+# kldref, in the same drained window that records the calibration grams). Every scored
+# container is measured against that one recording, so the KL scale runs from 0 = the
+# checkpoint itself downward in faithfulness.
+#
+# TWO MODES, because the disk decides how many anchors can exist at once:
+#
+#   gate IQ4R.json CAND.json                    candidate vs the served payload
+#   gate FP8.json IQ4R.json CAND.json           the same, inside the absolute interval
+#
+# iq4r.json is the payload frank-archon serves — the bar to beat. When the FP8-native release
+# (172.8 GiB, fetched last because of the peak-disk law) is on disk, pass its report first:
+# "half the gap" is then half of the distance between the 4-bit payload and the near-lossless
+# release, not half of an invented distance to the checkpoint. Without it the gate still
+# answers the campaign's question — does the candidate beat what is served — and says so.
+#
+# The law, on the summary of one slice (--slice all | a corpus source):
+#
+#   candidate vs served (2 reports)
+#   1. mean KL(cand) < mean KL(iq4r)         the served payload is beaten on the corpus
+#   2. p99.9(cand) <= p99.9(iq4r)            and the tail is no worse — a mean can hide a few
+#                                            positions taking the whole hit
+#   3. top-1(cand) >= top-1(iq4r)            the agreement the harness keys on does not fall
+#   4. every source slice passes 1-3         a win bought by regressing one distribution
+#                                            (code, prose, tools) is not a win
+#
+#   inside the interval (3 reports), laws 1-4 above (1-3 renumbered to the interval and the
+#   recovery fraction, as agreed 2026-10-27) plus:
+#   5. p99.9(cand) <= p99.9(iq4r)            the tail law
+#   6. every source slice passes            the slice law
+#
+# Booting the containers is NOT this script's job: `run-qwen38-bf16.sh score RAD OUT.json`
+# writes the reports and owns the single engine argv (same pin, --tp 2, --kv-cache-dtype fp8,
+# same step size, --deterministic, no speculation) that makes the numbers comparable.
+# This script is arithmetic, so it re-runs against old reports and is testable without a GPU.
+#
+# Usage: kld-gate-qwen38.sh IQ4R.json CAND.json [--slice NAME]
+#        kld-gate-qwen38.sh FP8.json IQ4R.json CAND.json [--slice NAME] [--recover F]
+set -uo pipefail
+
+if [ "$#" -lt 2 ]; then
+    echo "usage: kld-gate-qwen38.sh IQ4R.json CAND.json [--slice NAME]" >&2
+    echo "       kld-gate-qwen38.sh FP8.json IQ4R.json CAND.json [--slice NAME] [--recover F]" >&2
+    exit 2
+fi
+python3 - "$@" <<'PY'
+import json, sys
+
+a = sys.argv[1:]
+opts, files = [], []
+i = 0
+while i < len(a):
+    if a[i].startswith("--"):
+        opts += [a[i], a[i + 1]]
+        i += 2
+    else:
+        files.append(a[i])
+        i += 1
+if len(files) not in (2, 3):
+    sys.exit("usage: kld-gate-qwen38.sh IQ4R.json CAND.json [--slice NAME]\n"
+             "       kld-gate-qwen38.sh FP8.json IQ4R.json CAND.json [--slice NAME] [--recover F]")
+o = dict(zip(opts[0::2], opts[1::2]))
+slice_ = o.get("--slice", "all")
+recover = float(o.get("--recover", 0.5))
+if not 0.0 < recover <= 1.0:
+    sys.exit(f"--recover must be in (0,1], got {recover}")
+interval = len(files) == 3
+
+
+def die(msg):
+    print(msg, file=sys.stderr)
+    raise SystemExit(2)
+
+
+def load(path):
+    try:
+        return json.load(open(path))
+    except Exception as e:
+        die(f"cannot read the KLD report {path}: {e}")
+
+
+def summary(path, d):
+    """The slice this gate grades: 'all', or one corpus source."""
+    if slice_ == "all":
+        g = d.get("all")
+    else:
+        g = (d.get("by_source") or {}).get(slice_)
+        if g is None:
+            die(f"{path} has no slice {slice_!r}; it has: "
+                f"{', '.join(sorted(d.get('by_source') or {})) or '(none)'}")
+    if g is None or "kld" not in g or "top1_agreement" not in g:
+        die(f"{path} has no {slice_!r} summary — wrong report version?")
+    return g
+
+
+docs = {label: load(path) for label, path in
+        zip(("fp8", "iq4r", "cand") if interval else ("iq4r", "cand"), files)}
+rows = []
+for label, path in zip(("fp8", "iq4r", "cand") if interval else ("iq4r", "cand"), files):
+    g = summary(path, docs[label])
+    rows.append((label, path, docs[label].get("model", "?"), g["kld"]["mean"],
+                 g["kld"]["p99_9"], g["top1_agreement"], g["positions"]))
+
+print(f"reference: BF16 container; slice: {slice_}; "
+      + ("inside the FP8 interval" if interval else "candidate vs served (no FP8 anchor)"))
+print(f"{'':6} {'model':38} {'KL mean':>10} {'KL p99.9':>10} {'top-1':>8}  positions")
+for label, path, model, kl, p999, top1, n in rows:
+    print(f"{label:6} {str(model)[:38]:38} {kl:10.6f} {p999:10.4f} {100*top1:7.2f}%  {n}")
+r = {label: (kl, p999, top1) for label, _p, _m, kl, p999, top1, _n in rows}
+kl_cand, p_cand, t_cand = r["cand"]
+kl_iq4r, p_iq4r, t_iq4r = r["iq4r"]
+ok = True
+
+
+def law(n, text, passed):
+    global ok
+    ok = ok and passed
+    print(f"  [{'PASS' if passed else 'FAIL'}] {n}. {text}")
+
+
+def tail_law(n):
+    law(n, f"candidate KL p99.9 {p_cand:.4f} <= served iq4r {p_iq4r:.4f}",
+        p_cand <= p_iq4r)
+
+
+# --- the slice law: every corpus source must pass the metric laws -------------
+def slice_law(n):
+    """A win that regresses one source class is not a win: check them all."""
+    mine = docs["cand"].get("by_source") or {}
+    theirs = docs["iq4r"].get("by_source") or {}
+    if not mine or not theirs:
+        law(n, "no per-source summaries to check", True)
+        return
+    missing = sorted(set(theirs) - set(mine))
+    extra = sorted(set(mine) - set(theirs))
+    if missing or extra:
+        law(n, "same source slices in both reports"
+            + (f" (missing from cand: {', '.join(missing)})" if missing else "")
+            + (f" (only in cand: {', '.join(extra)})" if extra else ""), False)
+        return
+    bad = []
+    for src in sorted(theirs):
+        c, q = mine[src], theirs[src]
+        worse_mean = c["kld"]["mean"] >= q["kld"]["mean"]
+        worse_tail = c["kld"]["p99_9"] > q["kld"]["p99_9"]
+        worse_top1 = c["top1_agreement"] < q["top1_agreement"]
+        if worse_mean or worse_tail or worse_top1:
+            why = ",".join(m for m, w in (("mean", worse_mean), ("p99.9", worse_tail),
+                                          ("top-1", worse_top1)) if w)
+            bad.append(f"{src} ({why})")
+    law(n, "every source slice passes (mean, p99.9, top-1)"
+        + (f" — regressed: {'; '.join(bad)}" if bad else ""), not bad)
+
+
+if interval:
+    kl_fp8 = r["fp8"][0]
+    gap = kl_iq4r - kl_fp8
+    law(1, f"anchors ordered: fp8 {kl_fp8:.6f} < iq4r {kl_iq4r:.6f} (gap {gap:.6f})",
+        gap > 0)
+    law(2, f"candidate inside the interval: {kl_fp8:.6f} < {kl_cand:.6f} < {kl_iq4r:.6f}",
+        kl_fp8 < kl_cand < kl_iq4r)
+    need = kl_iq4r - recover * gap
+    law(3, f"candidate closes {100*max((kl_iq4r - kl_cand)/gap, 0):.1f}% of the "
+           f"{gap:.6f} gap (needs {100*recover:.0f}%, i.e. KL <= {need:.6f})"
+           if gap > 0 else "gap is not positive; cannot score recovery",
+        gap > 0 and kl_cand <= need)
+    law(4, f"candidate top-1 {100*t_cand:.2f}% >= served iq4r top-1 {100*t_iq4r:.2f}%",
+        t_cand >= t_iq4r)
+    tail_law(5)
+    slice_law(6)
+else:
+    law(1, f"candidate beats the served payload: KL {kl_cand:.6f} < iq4r {kl_iq4r:.6f}",
+        kl_cand < kl_iq4r)
+    tail_law(2)
+    law(3, f"candidate top-1 {100*t_cand:.2f}% >= served iq4r top-1 {100*t_iq4r:.2f}%",
+        t_cand >= t_iq4r)
+    slice_law(4)
+
+print("KLD GATE " + ("PASS" if ok else "FAIL"))
+sys.exit(0 if ok else 1)
+PY

scripts/make-calib-corpus.py

@@ -34,14 +34,25 @@ import json
 import os
 import random
 import re
+import sys
 
 from jinja2 import Environment, FileSystemLoader
-from tokenizers import Tokenizer
+
+_HERE = os.path.dirname(os.path.abspath(__file__)) if "__file__" in globals() else os.getcwd()
+if _HERE not in sys.path:
+    sys.path.insert(0, _HERE)
+
+import qwen38_common as qc  # noqa: E402 - sibling module, same directory
 
 CODE_EXT = (".sh", ".py", ".jinja", ".recipe", ".env", ".example",
             ".yaml", ".yml", ".bats", ".service", ".container", ".toml")
 JSONL_EXTS = (".jsonl", ".json")
 DOC_DIR = "serverAppConfigs-docs"
+FRAGMENT_TOKENS = 2400   # what one fragment may contribute
+CLASS_MIX = {"code": .45, "prose": .30, "tools": .25}
+HF_CAP = {"code": 40000, "prose": 60000, "tools": 0}
+POOL_CYCLES = 40         # how many times a class pool may repeat before giving up
+KLD_GUARD = 5000         # rendered conversations tried before declaring the pool short
 
 FLEET_CMDS = [
     "systemctl status frank-archon --no-pager",
@@ -142,7 +153,7 @@ def load_parquet_texts(path: str, kind: str, rng: random.Random,
     return out[:cap]
 
 
-def main() -> None:
+def parse_args() -> argparse.Namespace:
     ap = argparse.ArgumentParser(description=__doc__)
     ap.add_argument("--repo", required=True)
     ap.add_argument("--tokenizer-json", required=True)
@@ -156,22 +167,26 @@ def main() -> None:
     ap.add_argument("--hf-code-parquet", default="")
     ap.add_argument("--hf-prose-parquet", default="")
     ap.add_argument("--seed", type=int, default=20261006)
-    a = ap.parse_args()
+    return ap.parse_args()
 
-    rng = random.Random(a.seed)
+
-    tokenizer = Tokenizer.from_file(a.tokenizer_json)
+def load_template(template: str) -> tuple[Environment, str]:
-    env = Environment(loader=FileSystemLoader(os.path.dirname(a.template)),
+    """The chat template as the engine will render it, plus its short name."""
+    env = Environment(loader=FileSystemLoader(os.path.dirname(template)),
                       trim_blocks=True, lstrip_blocks=True)
     env.filters["tojson"] = lambda v: json.dumps(v, ensure_ascii=False)
     env.filters["raise_exception"] = lambda m: (_ for _ in ()).throw(
         ValueError(m))
-    tname = os.path.basename(a.template)
+    return env, os.path.basename(template)
 
+
+def classify(repo: str) -> dict[str, list[str]]:
+    """Walk the checkout into the three source classes the mix draws from."""
     code, prose, jsonl = [], [], []
-    for root, dirs, files in os.walk(a.repo):
+    for root, dirs, files in os.walk(repo):
         dirs[:] = [d for d in dirs if not d.startswith(".")]
         for f in sorted(files):
-            p = os.path.relpath(os.path.join(root, f), a.repo)
+            p = os.path.relpath(os.path.join(root, f), repo)
             if DOC_DIR in p and f.endswith(".md"):
                 prose.append(p)
             elif p.startswith(DOC_DIR):
@@ -180,64 +195,92 @@ def main() -> None:
                 code.append(p)
             elif f.endswith(JSONL_EXTS) and f != "package-lock.json":
                 jsonl.append(p)
-    # file-level split: bucket 0 -> KLD pool, 1..4 -> calibration pool
+    return {"code": code, "prose": prose, "tools": jsonl}
-    pools: dict[str, tuple[list, list]] = {}
+
-    for cls, lst in (("code", code), ("prose", prose), ("tools", jsonl)):
+
-        pools[cls] = ([p for p in lst if fnv(p) % 5 == 0],
+def split_pools(classes: dict[str, list[str]]) -> dict[str, tuple[list, list]]:
+    """File-level split: bucket 0 -> KLD pool, everything else -> calibration."""
+    return {cls: ([p for p in lst if fnv(p) % 5 == 0],
                   [p for p in lst if fnv(p) % 5 != 0])
+            for cls, lst in classes.items()}
 
-    def text_of(p: str) -> str:
-        path = os.path.join(a.repo, p)
-        raw = open(path, encoding="utf-8", errors="replace").read()
-        if p.endswith(JSONL_EXTS):
-            lines = [json.loads(l)["content"] for l in raw.splitlines()
-                     if l.strip() and "content" in l]
-            raw = "\n".join(lines)
-        toks = tokenizer.encode(raw, add_special_tokens=False).ids
-        return tokenizer.decode(toks[:2400], skip_special_tokens=True)
 
+def jsonl_content(raw: str) -> str:
+    """The content fields of a content-keyed JSONL, or "" for anything else.
+
+    A .json in this repo is not necessarily a content-keyed JSONL: the substring
+    test passes on any line that merely mentions "content", and a list-shaped
+    document raises TypeError — both used to crash the whole build. Skip what
+    does not fit instead.
+    """
+    lines = []
+    for line in raw.splitlines():
+        if not line.strip() or "content" not in line:
+            continue
+        try:
+            rec = json.loads(line)
+        except json.JSONDecodeError:
+            continue
+        if isinstance(rec, dict) and isinstance(rec.get("content"), str):
+            lines.append(rec["content"])
+    return "\n".join(lines)
+
+
+def fragment_text(repo: str, rel: str, tok) -> str:
+    """One file as a fragment: its first FRAGMENT_TOKENS tokens, decoded."""
+    raw = open(os.path.join(repo, rel), encoding="utf-8",
+               errors="replace").read()
+    if rel.endswith(JSONL_EXTS):
+        raw = jsonl_content(raw)
+    ids = tok.encode(raw, add_special_tokens=False).ids
+    return tok.decode(ids[:FRAGMENT_TOKENS], skip_special_tokens=True)
+
+
+def build_frags(a: argparse.Namespace, tok,
+                rng: random.Random) -> dict[str, dict[str, list[str]]]:
+    """Both fragment pools per class, seeded so the same seed rebuilds the corpus."""
+    pools = split_pools(classify(a.repo))
     cmds = harvest_cmds(a.repo, rng)
-    # fragment pools per class, corpus-split first
     frags: dict[str, dict[str, list[str]]] = {}
-    for cls in pools:
+    for cls, (kld_files, cal_files) in pools.items():
-        kld_files, cal_files = pools[cls]
-        fr = {}
-        src = {"kld": kld_files, "cal": cal_files}
         if cls == "tools":
-            fr["kld"] = [tool_call(rng, cmds) for _ in range(600)]
+            frags[cls] = {"kld": [tool_call(rng, cmds) for _ in range(600)],
-            fr["cal"] = [tool_call(rng, cmds) for _ in range(40000)]
+                          "cal": [tool_call(rng, cmds) for _ in range(40000)]}
         else:
-            fr["kld"] = [text_of(p) for p in src["kld"]]
+            frags[cls] = {"kld": [fragment_text(a.repo, p, tok) for p in kld_files],
-            fr["cal"] = [text_of(p) for p in src["cal"]]
+                          "cal": [fragment_text(a.repo, p, tok) for p in cal_files]}
-        frags[cls] = fr
     # per-class fragment caps: prose rows are short (~66 tok) so need many;
     # codeparrot files are long (~2.6k tok) so a small cap suffices
-    hfcap = {"code": 40000, "prose": 60000, "tools": 0}
     for cls, spec in (("code", a.hf_code_parquet),
                       ("prose", a.hf_prose_parquet)):
         for path in [p for p in spec.split(",") if p]:
             frags[cls]["cal"].extend(
-                load_parquet_texts(path, cls, rng, cap=hfcap[cls]))
+                load_parquet_texts(path, cls, rng, cap=HF_CAP[cls]))
     for cls in frags:
         rng.shuffle(frags[cls]["cal"])
+    return frags
 
-    def toks(s: str) -> int:
-        return len(tokenizer.encode(s, add_special_tokens=False).ids)
 
-    def emit_calib() -> tuple[int, list[dict]]:
+def pack_class(cls: str, pool: list[str], want: int, tok,
-        targets = {"code": .45, "prose": .30, "tools": .25}
+               chunk_tokens: int) -> list[dict]:
-        rows: list[dict] = []
+    """Greedily pack `pool` into chunk_tokens-sized calibration prompts."""
-        for cls, frac in targets.items():
-            want = int(a.calib_tokens * frac)
-            pool = frags[cls]["cal"]
     if not pool:
-                continue
+        return []
-            got, i, buf, nbuf = 0, 0, [], 0
+    rows: list[dict] = []
+    # Count every fragment once: the loop cycles the pool up to POOL_CYCLES
+    # times, and re-tokenizing a 2.4k-token fragment on each pass dominated the
+    # build's wall time.
+    counts = [qc.n_tokens(tok, f) for f in pool]
+    got = 0
+    i = 0
+    buf: list[str] = []
+    nbuf = 0
     while got < want:
-                frag = pool[i % len(pool)]
+        j = i % len(pool)
+        frag = pool[j]
         i += 1
-                n = toks(frag)
+        n = counts[j]
-                if nbuf + n > a.chunk_tokens and buf:
+        if nbuf + n > chunk_tokens and buf:
             rows.append({"prompt": "\n".join(buf), "cls": cls})
             got += nbuf
             buf, nbuf = [], 0
@@ -245,22 +288,36 @@ def main() -> None:
                 break
         buf.append(frag)
         nbuf += n
-                if i > len(pool) * 40:
+        if i > len(pool) * POOL_CYCLES:
             break  # pool exhausted with repeats; stop
     if buf and got < want:
         rows.append({"prompt": "\n".join(buf), "cls": cls})
         got += nbuf
     print(f"calib {cls}: {got} tokens, {i} fragments")
-        rng.shuffle(rows)
+    return rows
-        return len(rows), [{k: v for k, v in r.items() if k == "prompt"}
-                           for r in rows]
 
-    def emit_kld() -> tuple[int, list[dict]]:
+
-        rows, total = [], 0
+def emit_calib(a: argparse.Namespace, frags, tok,
+               rng: random.Random) -> tuple[int, list[dict]]:
+    """calib.jsonl rows: the class mix, shuffled, prompts only."""
+    rows: list[dict] = []
+    for cls, frac in CLASS_MIX.items():
+        rows.extend(pack_class(cls, frags[cls]["cal"], int(a.calib_tokens * frac),
+                               tok, a.chunk_tokens))
+    rng.shuffle(rows)
+    return len(rows), [{"prompt": r["prompt"]} for r in rows]
+
+
+def emit_kld(a: argparse.Namespace, frags, tok, rng: random.Random,
+             env: Environment, tname: str) -> tuple[int, list[dict], int]:
+    """kld.jsonl rows: multi-turn conversations, each with a scored tail."""
+    template = env.get_template(tname)
+    rows: list[dict] = []
+    total = scored_total = 0
     classes = ["code", "prose", "tools"]
-        i = {c: 0 for c in classes}
+    cursor = {c: 0 for c in classes}
     guard = 0
-        while total < a.kld_tokens and guard < 5000:
+    while total < a.kld_tokens and guard < KLD_GUARD:
         guard += 1
         msgs, srcs = [], []
         for t in range(rng.choice([2, 3, 4])):
@@ -268,49 +325,55 @@ def main() -> None:
             pool = frags[cls]["kld"]
             if not pool:
                 continue
-                frag = pool[i[cls] % len(pool)]
+            frag = pool[cursor[cls] % len(pool)]
-                i[cls] += 2
+            cursor[cls] += 1
             msgs.append({"role": "user",
                          "content": "Summarize and explain this "
                                     f"engineering material:\n{frag}"})
             msgs.append({"role": "assistant",
                          "content": frag[:4000]})
             srcs.append(cls)
-            rendered = env.get_template(tname).render(
+        rendered = template.render(messages=msgs, add_generation_prompt=False)
-                messages=msgs, add_generation_prompt=False)
+        n = qc.n_tokens(tok, rendered)
-            n = toks(rendered)
         if n < a.kld_min_prompt:
             continue
         rows.append({"prompt": rendered,
                      "score_from": n - a.score_tail,
                      "source": "+".join(sorted(set(srcs)))})
         total += n
-        return len(rows), rows
+        scored_total += a.score_tail
+    return len(rows), rows, scored_total
 
-    os.makedirs(a.out_dir, exist_ok=True)
+
-    nc, rows = emit_calib()
+def manifest(a: argparse.Namespace, tname: str, nc: int, nk: int,
-    with open(os.path.join(a.out_dir, "calib.jsonl"), "w") as f:
+             scored: int) -> dict:
-        for r in rows:
+    return {
-            f.write(json.dumps(r) + "\n")
-    nk, krows = emit_kld()
-    with open(os.path.join(a.out_dir, "kld.jsonl"), "w") as f:
-        for r in krows:
-            f.write(json.dumps(r) + "\n")
-    scored = sum(len(tokenizer.encode(r["prompt"], add_special_tokens=False).ids)
-                 - r["score_from"] for r in krows)
-    manifest = {
         "seed": a.seed, "template": tname,
         "tokenizer": os.path.basename(a.tokenizer_json),
         "calib_prompts": nc, "kld_prompts": nk,
         "kld_scored_positions": scored,
         "kld_score_tail": a.score_tail,
         "disjoint_file_pools": "local KLD material only; HF code feeds calib",
-        "calib_mix": {"code": .45, "prose": .30, "tools": .25},
+        "calib_mix": dict(CLASS_MIX),
         "hf_code": a.hf_code_parquet, "hf_prose": a.hf_prose_parquet,
     }
-    with open(os.path.join(a.out_dir, "manifest.json"), "w") as f:
+
-        json.dump(manifest, f, indent=1)
+
-    print(json.dumps(manifest, indent=1))
+def main() -> None:
+    a = parse_args()
+    rng = random.Random(a.seed)
+    tok = qc.load_tokenizer(a.tokenizer_json)
+    env, tname = load_template(a.template)
+
+    frags = build_frags(a, tok, rng)
+    os.makedirs(a.out_dir, exist_ok=True)
+    nc, calib_rows = emit_calib(a, frags, tok, rng)
+    qc.write_jsonl(os.path.join(a.out_dir, "calib.jsonl"), calib_rows)
+    nk, kld_rows, scored = emit_kld(a, frags, tok, rng, env, tname)
+    qc.write_jsonl(os.path.join(a.out_dir, "kld.jsonl"), kld_rows)
+    man = manifest(a, tname, nc, nk, scored)
+    qc.write_json(os.path.join(a.out_dir, "manifest.json"), man)
+    print(json.dumps(man, indent=1))
 
 
 if __name__ == "__main__":

scripts/media-apply.py

@@ -0,0 +1,434 @@
+#!/usr/bin/env python3
+"""media-apply.py — the only thing that moves movie-library files.
+
+Reads the plan written by scripts/media-plan.py (rows: `action  src  dst  kind`,
+plus `mkdir` rows) and performs it with rename(2) — bytes are never copied, so
+an interrupted run has destroyed nothing. Every run is resumable and every run
+is undoable: the journal is a write-ahead log, one JSON line per intent, fsynced
+BEFORE the rename it authorises. That ordering is the whole design: a crash
+leaves either an intent whose move happened or an intent whose move never ran,
+and both are recognised on the next run. A file that moved with no journal line
+cannot have been moved by this tool, and saying so is better than guessing.
+
+Four subcommands:
+
+    apply    --plan PLAN [--root ROOT] [--journal J] [--apply]
+             no --apply => report only and write NOTHING, not even folders
+    rollback --journal J [--root ROOT] [--apply]
+             back to the paths the census saw; folders this run created are
+             removed only when empty, never anything else
+    verify   --plan PLAN --journal J [--root ROOT]
+             every planned file is at its target with the journalled byte count
+    sweep    --journal J [--root ROOT] [--apply]
+             removes the folders THIS RUN emptied — the journal has to show a
+             file that moved out of one, so a folder someone made by hand can
+             never be removed by this tool, and rollback puts them back
+
+Refusals (exit 2, before the first rename, with nothing touched): a target
+already occupied by a foreign file, a source that has gone, a duplicate target,
+a path outside --root, a journal that was already rolled back, a move that would
+cross a filesystem (rename cannot do it and a silent copy at 24 TB would not).
+
+--force relaxes only the guards that are opinion (journal already rolled back,
+paths outside --root); it never allows overwriting a file.
+
+The journal defaults to `media-relayout.jsonl` BESIDE the library root, never
+inside it: a log inside the library is a file Jellyfin will index.
+"""
+import argparse
+import datetime
+import json
+import os
+import sys
+
+DEFAULT_ROOT = os.environ.get("MEDIA_ROOT", "/mnt/entropy/Movies")
+
+
+class Refuse(Exception):
+    """A gate the operator has to see; every refusal leaves the library alone."""
+
+
+def journal_default(root):
+    return os.path.join(os.path.dirname(os.path.abspath(root)), "media-relayout.jsonl")
+
+
+class Journal:
+    """Append-only JSONL of intents. Writes are fsynced before the tool acts."""
+
+    def __init__(self, path, writing):
+        self.path, self.writing = path, writing
+        self.records = []
+        if os.path.exists(path):
+            for line in open(path):
+                line = line.strip()
+                if line:
+                    self.records.append(json.loads(line))
+        self._fh = None
+
+    def already(self, dst, src):
+        """An intent for this exact (src, dst) was written by an earlier run."""
+        return any(r.get("op") == "move" and r.get("dst") == dst
+                   and r.get("src") == src for r in self.records)
+
+    @property
+    def rolled_back(self):
+        return any(r.get("op") == "rollback" for r in self.records)
+
+    def record(self, **fields):
+        self.records.append(fields)
+        if not self.writing:
+            return
+        if self._fh is None:
+            self._fh = open(self.path, "a")
+        self._fh.write(json.dumps(fields) + "\n")
+        self._fh.flush()
+        os.fsync(self._fh.fileno())
+
+    def close(self):
+        if self._fh:
+            self._fh.close()
+
+
+def read_plan(path, root, force):
+    """Plan rows: (mkdirs, keeps, moves), header dropped, all paths under root."""
+    mkdirs, keeps, moves = [], [], []
+    with open(path) as fh:
+        header = fh.readline()
+        if not header.startswith("action\t"):
+            raise Refuse(f"{path} is not a plan (first column is "
+                         f"{header.split(chr(9))[0]!r}, expected 'action') —"
+                         " regenerate it with scripts/media-plan.py")
+        for line in fh:
+            f = line.rstrip("\n").split("\t")
+            if f[0] == "mkdir":
+                mkdirs.append(f[2])
+            elif f[0] == "keep":
+                keeps.append(f[1])
+            elif f[0] == "move":
+                moves.append((f[1], f[2], f[3] if len(f) > 3 else ""))
+    if not force:
+        for p in mkdirs + keeps + [p for m in moves for p in m[:2]]:
+            if p and not (p == root or p.startswith(root + os.sep)):
+                raise Refuse(f"{p} is outside --root {root}"
+                             " (a plan built for another library?)")
+    return mkdirs, keeps, moves
+
+
+# A relayout moves bytes on the raw array; the sidecars the media servers wrote
+# live in the SSD overlay and stay at the old coordinates. Naming the follow-up
+# here is the point of use for that fact — see the Media Metadata View page.
+METADATA_HINT = ("note: sidecars written through the metadata view did not follow"
+                 " these moves — next: scripts/media-view.sh reap")
+
+
+def do_apply(args, root, journal, mkdirs, keeps, moves):
+    """Preflight everything, then folders, then renames. Never a partial start."""
+    problems = []
+    for p in keeps:
+        if not os.path.exists(p):
+            problems.append(f"missing: keep row source is gone: {p}")
+    seen = {}
+    for src, dst, _kind in moves:
+        if src == dst:
+            problems.append(f"no-op: move row whose target is its source: {src}")
+        if dst in seen:
+            problems.append(f"double-claim: {dst} claimed by {seen[dst]} and {src}")
+        seen[dst] = src
+        if journal.already(dst, src) and os.path.exists(dst) and not os.path.exists(src):
+            continue                                    # this intent already ran
+        if not os.path.exists(src):
+            if os.path.exists(dst):
+                problems.append(f"missing: {src} is gone and {dst} carries no journal"
+                                " line — it moved outside this tool, refusing to guess")
+            else:
+                problems.append(f"missing: {src} is gone and {dst} is not there either")
+            continue
+        if os.path.exists(dst):
+            problems.append(f"occupied: target already holds another file: {dst}")
+    if problems:
+        print(f"refusing: {len(problems)} problem(s), nothing moved")
+        for p in sorted(problems)[:20]:
+            print("  " + p)
+        if len(problems) > 20:
+            print(f"  ... and {len(problems) - 20} more")
+        return 2
+
+    if not args.apply:
+        print(f"dry run: {len(moves)} moves, {len(mkdirs)} folders, {len(keeps)} files stay"
+              " — nothing written (pass --apply to run)")
+        for src, dst, _k in moves[:5]:
+            print(f"  would move {src} -> {dst}")
+        return 0
+
+    made = {r.get("dst") for r in journal.records if r.get("op") == "mkdir"}
+    dirs_new = 0
+    for d in sorted(mkdirs, key=lambda q: q.count(os.sep)):
+        if d in made or os.path.isdir(d):
+            continue
+        journal.record(op="mkdir", dst=d)
+        os.makedirs(d, exist_ok=True)
+        dirs_new += 1
+
+    # same-filesystem gate: rename cannot cross a device, and a silent copy of a
+    # library this size is not an option anyone would choose.
+    cross = []
+    for src, dst, _k in moves:
+        if journal.already(dst, src) and os.path.exists(dst) and not os.path.exists(src):
+            continue
+        parent = os.path.dirname(dst)
+        if os.path.exists(parent) and os.stat(parent).st_dev != os.stat(src).st_dev:
+            cross.append(dst)
+    if cross:
+        print(f"refusing: {len(cross)} moves would cross a filesystem"
+              f" (first: {cross[0]}) — this tool only renames")
+        return 2
+
+    moved = done = 0
+    for src, dst, _kind in moves:
+        if journal.already(dst, src):
+            if os.path.exists(dst) and not os.path.exists(src):
+                done += 1
+                continue
+            if os.path.exists(src) and not os.path.exists(dst):
+                os.rename(src, dst)             # the intent ran, the move did not
+                moved += 1
+                continue
+        st = os.stat(src)
+        journal.record(op="move", src=src, dst=dst, bytes=st.st_size, mode=st.st_mode,
+                       mtime=st.st_mtime, ino=st.st_ino, dev=st.st_dev)
+        os.rename(src, dst)
+        moved += 1
+    print(f"applied: {moved} moved, {done} already done in an earlier run,"
+          f" {dirs_new} folders created, {len(keeps)} files stayed put")
+    print(f"journal: {journal.path} — undo with: media-apply.py rollback"
+          f" --journal {journal.path} --root {root} --apply")
+    print(METADATA_HINT)
+    return 0
+
+
+def do_verify(root, journal, moves, keeps):
+    """Every planned file sits at its target carrying the journalled bytes."""
+    journalled = {(r["src"], r["dst"]): r for r in journal.records
+                  if r.get("op") == "move"}
+    todo = moves or [(r["src"], r["dst"], "") for r in journal.records
+                     if r.get("op") == "move"]
+    bad, missing, bytes_total = [], [], 0
+    for src, dst, _kind in todo:
+        rec = journalled.get((src, dst))
+        if not os.path.exists(dst):
+            missing.append(dst)
+            continue
+        size = os.path.getsize(dst)
+        bytes_total += size
+        if rec and rec.get("bytes") not in (None, size):
+            bad.append(f"{dst}: {size} bytes on disk, {rec['bytes']} at move time")
+        if os.path.exists(src) and src != dst:
+            bad.append(f"{src}: still in place as well as at {dst}")
+    missing += [p for p in keeps if not os.path.exists(p)]
+    if bad or missing:
+        print(f"verify FAILED: {len(missing)} absent, {len(bad)} wrong")
+        for line in (missing + bad)[:20]:
+            print("  " + line)
+        return 2
+    empty = [r["dst"] for r in journal.records
+             if r.get("op") == "mkdir" and os.path.isdir(r["dst"])
+             and not os.listdir(r["dst"])]
+    print(f"verified: {len(todo)} files at their targets"
+          f" ({bytes_total / 2**40:.2f} TiB), {len(keeps)} kept")
+    if empty:
+        print(f"note: {len(empty)} folders are empty (left in place — see the trash law)")
+    return 0
+
+
+def sweep_targets(root, journal):
+    """The folders this run emptied, deepest first — never one it did not touch.
+
+    Eligibility is evidence, not appearance. A folder is a target only when the
+    journal shows a file moved out of it or out of something beneath it, it holds
+    nothing but empty directories now, it is not a folder this run created, and no
+    journalled destination lives under it. A symlink anywhere inside disqualifies
+    the whole subtree: this tool never follows a link, and deleting the folder
+    that holds one would delete someone's mount point.
+
+    Proof carries downward: once a folder qualifies, every empty directory inside
+    it qualifies too. A release's junk (``Screens/``, ``Sample/``) is never itself
+    a move source, so without that rule it holds its emptied parent on disk
+    forever — the parent can never be removed while the child stands.
+    """
+    moves = [r for r in journal.records if r.get("op") == "move"]
+    made = {r["dst"] for r in journal.records if r.get("op") == "mkdir"}
+    # An rmdir INTENT that failed left its folder standing, and the next sweep
+    # must be able to finish it: only a folder that is really gone is out of
+    # scope. Counting every intent as done made 146 live survivors permanent.
+    gone = {r["path"] for r in journal.records if r.get("op") == "rmdir"
+            and not os.path.exists(r["path"])}
+    dsts = [r["dst"] for r in moves]
+    marked = set()
+    for r in moves:
+        d = os.path.dirname(r["src"])
+        while d.startswith(root + os.sep):
+            marked.add(d)
+            d = os.path.dirname(d)
+
+    def removable(d):
+        if d in made or d in gone or os.path.islink(d):
+            return False
+        if any(p == d or p.startswith(d + os.sep) for p in dsts):
+            return False
+        try:
+            entries = os.listdir(d)
+        except OSError:
+            return False
+        for e in entries:
+            c = os.path.join(d, e)
+            if os.path.islink(c) or not os.path.isdir(c) or not removable(c):
+                return False
+        return True
+
+    roots = [d for d in marked if removable(d)]
+    targets = set(roots)
+    for d in roots:
+        for dirpath, dirnames, _ in os.walk(d):
+            for c in dirnames:
+                p = os.path.join(dirpath, c)
+                if removable(p):
+                    targets.add(p)
+    return sorted(targets, key=lambda q: (-q.count(os.sep), q))
+
+
+def do_sweep(args, root, journal):
+    """Remove the empty husks an apply left standing at the library root."""
+    moves = [r for r in journal.records if r.get("op") == "move"]
+    if not moves:
+        raise Refuse("the journal holds no moves — a sweep follows an apply, it"
+                     " does not guess at emptiness")
+    outside = sorted({p for r in moves for p in (r["src"], r["dst"])
+                      if not (p == root or p.startswith(root + os.sep))})
+    if outside and not args.force:
+        raise Refuse(f"{len(outside)} journalled paths are outside --root {root}"
+                     f" (first: {outside[0]}) — this journal belongs to another"
+                     " library")
+    targets = sweep_targets(root, journal)
+    if not targets:
+        print("swept: 0 folders — nothing this run emptied is still standing empty")
+        return 0
+    if not args.apply:
+        print(f"dry run: {len(targets)} folders this run emptied would be removed"
+              " (pass --apply to run)")
+        for d in targets[:5]:
+            print(f"  would remove {d}")
+        return 0
+    removed = 0
+    for d in targets:
+        journal.record(op="rmdir", path=d)
+        try:
+            os.rmdir(d)
+        except OSError as e:
+            print(f"left in place: {d} ({e})")
+            continue
+        removed += 1
+    print(f"swept: {removed} folders removed, {len(targets) - removed} left in place")
+    print(f"journal: {journal.path} — rollback recreates them with their files")
+    return 0
+
+
+def do_rollback(args, root, journal):
+    """Undo every journalled move, newest first, after a full preflight."""
+    moves = [r for r in journal.records if r.get("op") == "move"]
+    made = [r["dst"] for r in journal.records if r.get("op") == "mkdir"]
+    swept = [r["path"] for r in journal.records if r.get("op") == "rmdir"]
+    problems = []
+    for r in reversed(moves):
+        src, dst = r["src"], r["dst"]
+        if not args.force and not (src.startswith(root + os.sep)
+                                   or dst.startswith(root + os.sep)):
+            problems.append(f"outside --root: {dst}")
+            continue
+        if os.path.exists(dst) and os.path.exists(src):
+            problems.append(f"occupied: both paths exist, refusing to choose:"
+                            f" {src} | {dst}")
+        elif os.path.exists(dst):
+            pass
+        elif os.path.exists(src):
+            continue                                    # already undone
+        else:
+            problems.append(f"missing: {dst} is gone and {src} never came back")
+    if problems:
+        print(f"refusing: {len(problems)} problem(s), nothing moved back")
+        for p in problems[:20]:
+            print("  " + p)
+        return 2
+    if not args.apply:
+        print(f"dry run: would move {len(moves)} files back to their census paths"
+              f" and recreate {len(set(swept))} swept folders (pass --apply)")
+        return 0
+    undone = 0
+    recreated = 0
+    for d in sorted(set(swept), key=lambda q: q.count(os.sep)):
+        if not os.path.isdir(d):
+            os.makedirs(d, exist_ok=True)
+            recreated += 1
+    for r in reversed(moves):
+        if os.path.exists(r["dst"]) and not os.path.exists(r["src"]):
+            os.makedirs(os.path.dirname(r["src"]), exist_ok=True)
+            os.rename(r["dst"], r["src"])
+            undone += 1
+    dirs_gone = 0
+    for d in sorted(set(made), key=lambda q: -q.count(os.sep)):
+        if d == root or not os.path.isdir(d) or os.listdir(d):
+            continue
+        try:
+            os.rmdir(d)
+            dirs_gone += 1
+        except OSError as e:
+            print(f"left in place: {d} ({e})")
+    journal.record(op="rollback", undone=undone, dirs_removed=dirs_gone,
+                   t=datetime.datetime.now(datetime.timezone.utc).isoformat())
+    print(f"rolled back: {undone} files back where the census found them,"
+          f" {dirs_gone} empty folders removed, {recreated} swept folders recreated")
+    print(METADATA_HINT)
+    return 0
+
+
+def main():
+    ap = argparse.ArgumentParser(description="apply or undo a media relayout plan")
+    ap.add_argument("cmd", choices=["apply", "rollback", "verify", "sweep"])
+    ap.add_argument("--plan")
+    ap.add_argument("--root", default=DEFAULT_ROOT)
+    ap.add_argument("--journal")
+    ap.add_argument("--apply", action="store_true", help="actually change the library")
+    ap.add_argument("--force", action="store_true", help="relax the opinion guards")
+    args = ap.parse_args()
+
+    root = os.path.abspath(args.root)
+    if not os.path.isdir(root):
+        raise Refuse(f"no library at {root}")
+    journal_path = args.journal or journal_default(root)
+    journal = Journal(journal_path, args.apply)
+    try:
+        if args.cmd in ("rollback", "verify", "sweep") and not os.path.exists(journal_path):
+            raise Refuse(f"no journal at {journal_path} — nothing to {args.cmd}")
+        if args.cmd in ("rollback", "sweep") and not args.force and journal.rolled_back:
+            raise Refuse("this journal has already been rolled back"
+                         " (--force to replay it)")
+        if args.cmd in ("apply", "verify") and not args.plan:
+            raise Refuse(f"{args.cmd} needs --plan")
+        mkdirs, keeps, moves = (read_plan(args.plan, root, args.force)
+                                if args.plan else ([], [], []))
+        if args.cmd == "apply":
+            return do_apply(args, root, journal, mkdirs, keeps, moves)
+        if args.cmd == "verify":
+            return do_verify(root, journal, moves, keeps)
+        if args.cmd == "sweep":
+            return do_sweep(args, root, journal)
+        return do_rollback(args, root, journal)
+    finally:
+        journal.close()
+
+
+if __name__ == "__main__":
+    try:
+        sys.exit(main())
+    except Refuse as r:
+        print(f"refusing: {r}")
+        sys.exit(2)

scripts/media-census.py

@@ -0,0 +1,203 @@
+#!/usr/bin/env python3
+r"""Read-only census of a movie library — the inputs a relayout plan is built from.
+
+  media-census.py enumerate [--root DIR] [--out TSV]
+  media-census.py probe     [--enum TSV] [--out TSV] [--jobs N]
+  media-census.py plex      [--root DIR] [--db PATH] [--out TSV]
+
+Run it on the machine that mounts the library. Nothing here writes inside the
+library and nothing is deleted; the only outputs are the three TSVs below, and
+scripts/media-plan.py reads them instead of touching media.
+
+  enum    bytes <TAB> ext <TAB> path                  every file under the root
+  probed  bytes <TAB> path <TAB> dur <TAB> vcodec <TAB> WxH <TAB> fps <TAB> vbit
+          <TAB> pix <TAB> audio <TAB> subs <TAB> spare <TAB> format
+          one row per video-container file; `audio` is "codec:lang+codec:lang",
+          `subs` the subtitle-stream count, `spare` the audio streams carrying
+          neither a default nor a forced tag
+  plex    file <TAB> leaf_title <TAB> … (see PLEX_COLS)
+          the titles Plex already matched — the authority over any filename guess
+
+`probe` is the slow step (every container is opened for its header, ~10 min for
+4,800 files on a spindle array at 6 workers) so it is `nice`d: a census beside a
+live service must not starve it. `enumerate` is a stat pass, so re-censusing
+before a plan is cheap — and a plan is only ever as good as these three files.
+"""
+import argparse
+import csv
+import json
+import os
+import re
+import sqlite3
+import subprocess
+import sys
+from concurrent.futures import ThreadPoolExecutor
+
+# Containers ffprobe can read. .iso/.img are deliberately NOT here: a disc image is
+# a mounted volume, not a stream, and the planner treats it as a movie file instead.
+PROBE_EXT = {".mkv", ".mp4", ".avi", ".m4v", ".mov", ".ts", ".m2ts", ".mpg",
+             ".mpeg", ".webm", ".wmv", ".divx", ".ogm", ".bdmv", ".vob"}
+# Trash and thumbnail caches are not library content, and this repo's trash policy
+# means .AGENTTRASH can sit inside a movie folder at any time.
+SKIP_DIR = re.compile(r"(^|/)(\.AGENTTRASH|\.thumbnails|\.Trash-\d+|@eaDir|lost\+found)(/|$)")
+PLEX_DB = ("/var/lib/plexmediaserver/Library/Application Support/Plex Media Server"
+           "/Plug-in Support/Databases/com.plexapp.plugins.library.db")
+PLEX_COLS = ["file", "leaf_title", "leaf_orig", "leaf_edition", "leaf_year",
+             "parent_title", "parent_orig", "parent_year", "parent_edition",
+             "container", "vcodec", "acodec", "width", "height"]
+PLEX_SQL = """
+select p.file, coalesce(mi.title,''), coalesce(mi.original_title,''),
+       coalesce(mi.edition_title,''), coalesce(mi.year,0),
+       coalesce(par.title,''), coalesce(par.original_title,''),
+       coalesce(par.year,0), coalesce(par.edition_title,''),
+       m.container, m.video_codec, m.audio_codec, m.width, m.height
+from media_parts p
+join media_items m on m.id = p.media_item_id
+join metadata_items mi on mi.id = m.metadata_item_id
+left join metadata_items par on par.id = mi.parent_id
+where p.file like ? and p.deleted_at is null and mi.deleted_at is null
+"""
+
+
+def enumerate_lib(root, out):
+    """One row per real file under root: size, extension, path."""
+    rows = []
+    for dirpath, dirnames, filenames in os.walk(root):
+        dirnames[:] = [d for d in dirnames
+                       if not SKIP_DIR.search(os.path.join(dirpath, d))]
+        for fn in filenames:
+            p = os.path.join(dirpath, fn)
+            if SKIP_DIR.search(p):
+                continue
+            try:
+                st = os.lstat(p)
+            except OSError:
+                continue
+            if not os.path.isfile(p) or os.path.islink(p):
+                continue                # links and sockets are not library content
+            rows.append((st.st_size, os.path.splitext(fn)[1].lower(), p))
+    with open(out, "w") as fh:
+        for size, ext, p in rows:
+            fh.write(f"{size}\t{ext}\t{p}\n")
+    vid = [r for r in rows if r[1] in PROBE_EXT]
+    print(f"files={len(rows)} bytes={sum(r[0] for r in rows)} "
+          f"probes={len(vid)} probe_bytes={sum(r[0] for r in vid)} -> {out}")
+
+
+def probe_one(path):
+    """ffprobe header read -> the probed TSV row (never the whole file)."""
+    try:
+        os.nice(19)                     # a census must never outrank the media server
+    except OSError:
+        pass
+    try:
+        size = os.stat(path).st_size
+    except OSError:
+        return None
+    cmd = ["ffprobe", "-v", "error", "-print_format", "json",
+           "-show_format", "-show_streams", path]
+    try:
+        done = subprocess.run(cmd, capture_output=True, text=True, timeout=180)
+        data = json.loads(done.stdout or "{}")
+    except (subprocess.TimeoutExpired, subprocess.SubprocessError,
+            json.JSONDecodeError, OSError):
+        data = {}
+    st = data.get("streams", [])
+    fmt = data.get("format", {})
+    v = next((s for s in st if s.get("codec_type") == "video"), None)
+    aud = [s for s in st if s.get("codec_type") == "audio"]
+    subs = [s for s in st if s.get("codec_type") == "subtitle"]
+    fps = "-"
+    if v and v.get("r_frame_rate"):
+        try:
+            n, d = (int(x) for x in v["r_frame_rate"].split("/", 1))
+            fps = f"{round(n / d * 100) / 100:g}" if d else "0"
+        except ValueError:
+            pass
+
+    def num(x):
+        try:
+            return round(float(x))
+        except (TypeError, ValueError):
+            return 0
+
+    return [size, path, num(fmt.get("duration")),
+            (v or {}).get("codec_name", "-"),
+            f"{v['width']}x{v['height']}" if v else "-",
+            fps, num(fmt.get("bit_rate")), (v or {}).get("pix_fmt", "-"),
+            "+".join(f"{a.get('codec_name', '?')}:{a.get('language', '?')}" for a in aud),
+            len(subs),
+            sum(1 for a in aud if not re.search(
+                r"default|forced", json.dumps(a.get("tags", {})), re.I)),
+            fmt.get("format_name", "-") or "-"]
+
+
+def probe(enum, out, jobs):
+    """Probe every video-container row of the enum TSV, in parallel."""
+    paths = []
+    with open(enum) as fh:
+        for line in fh:
+            f = line.rstrip("\n").split("\t", 2)
+            if len(f) == 3 and f[1] in PROBE_EXT:
+                paths.append(f[2])
+    print(f"probing {len(paths)} files with {jobs} workers", flush=True)
+    done = unreadable = 0
+    with open(out, "w") as fh, ThreadPoolExecutor(max_workers=jobs) as pool:
+        for row in pool.map(probe_one, paths):
+            if row is None:
+                continue
+            if row[3] == "-":
+                unreadable += 1         # header unreadable: the row still gets written
+            done += 1
+            fh.write("\t".join(str(x) for x in row) + "\n")
+            if done % 250 == 0:
+                print(f"  {done}/{len(paths)}", flush=True)
+    print(f"probed={done} unreadable={unreadable} -> {out}")
+
+
+def plex(root, db, out):
+    """Plex's matched titles for files under root, from its DB, read-only."""
+    uri = f"file:{db}?mode=ro&immutable=1"
+    try:
+        con = sqlite3.connect(uri, uri=True)
+    except sqlite3.Error as e:
+        sys.exit(f"cannot open the Plex database read-only ({db}): {e}\n"
+                 "Plex runs rootless here: run this as the container's owner, never "
+                 "with sudo — rootless storage is per-user, so root sees nothing.")
+    with open(out, "w", newline="") as fh:
+        w = csv.writer(fh, delimiter="\t")
+        w.writerow(PLEX_COLS)
+        n = 0
+        for r in con.execute(PLEX_SQL, (root + "/%",)):
+            w.writerow([str(x).replace("\t", " ").replace("\n", " ") for x in r])
+            n += 1
+    print(f"plex-matched={n} -> {out}")
+
+
+def main():
+    ap = argparse.ArgumentParser(description=__doc__.splitlines()[0])
+    sub = ap.add_subparsers(dest="cmd", required=True)
+    root = {"default": os.environ.get("MEDIA_ROOT", "/mnt/entropy/Movies"),
+            "help": "library root"}
+    e = sub.add_parser("enumerate", help="stat every file under the root")
+    e.add_argument("--root", **root)
+    e.add_argument("--out", default="/tmp/media-enum.tsv")
+    p = sub.add_parser("probe", help="ffprobe every video container")
+    p.add_argument("--enum", default="/tmp/media-enum.tsv")
+    p.add_argument("--out", default="/tmp/media-probed.tsv")
+    p.add_argument("--jobs", type=int, default=6)
+    x = sub.add_parser("plex", help="export Plex's matched titles for the root")
+    x.add_argument("--root", **root)
+    x.add_argument("--db", default=PLEX_DB)
+    x.add_argument("--out", default="/tmp/plex-movies.tsv")
+    a = ap.parse_args()
+    if a.cmd == "enumerate":
+        enumerate_lib(a.root, a.out)
+    elif a.cmd == "probe":
+        probe(a.enum, a.out, a.jobs)
+    else:
+        plex(a.root, a.db, a.out)
+
+
+if __name__ == "__main__":
+    main()

scripts/media-plan.py

@@ -0,0 +1,881 @@
+#!/usr/bin/env python3
+r"""Plan a Jellyfin-format relayout of a movie library.  DRY RUN ONLY: nothing moves.
+
+  media-plan.py [--root DIR] [--enum TSV] [--probed TSV] [--plex TSV]
+                [--out TSV] [--exceptions TXT]
+
+Inputs are the TSVs scripts/media-census.py exports, plus the optional Plex
+metadata-cache export whose clean titles outrank the filename parser; the title
+authority for everything Plex never matched is scripts/media-titles.py beside this
+file. No media file is opened, so planning the whole library costs seconds.
+
+PLAN is a TSV the executor (scripts/media-apply.py) replays verbatim —
+`action<TAB>src<TAB>dst<TAB>kind` with action in mkdir | move | keep, ordered so
+parents exist before children. EXCEPTIONS is the human review list: every file the
+plan could not name a film for, every collision it refused, the READMEs the
+executor must write into the piles it created.
+
+Target contract (jellyfin.org/docs/general/server/media/movies):
+  * one folder per movie, named  Title (year)
+  * the video inside is named exactly like the folder
+  * extra copies of one film are versions: "<base> - <label>.ext"; the base must
+    match the folder name character-for-character or Jellyfin splits them apart
+  * extras live in canonical subfolders (trailers, featurettes, deleted scenes,
+    interviews, samples, shorts, clips, other, extras, backdrops, theme-music)
+  * sidecars <base>.<lang>.<flag>.<ext>; images <base>-<poster|backdrop|...>
+  * <>:"/\|?* break the matcher, so they never survive here
+  * VIDEO_TS/BDMV folders support neither multiple versions nor parts
+"""
+import argparse
+import collections
+import csv
+import importlib.util
+import os
+import re
+
+# Inputs are the census TSVs scripts/media-census.py writes; the engine beside
+# this file is the title authority. Nothing here opens a media file.
+def parse_args():
+    ap = argparse.ArgumentParser(description=__doc__.splitlines()[0])
+    ap.add_argument("--root", default=os.environ.get("MEDIA_ROOT", "/mnt/entropy/Movies"),
+                    help="library root the plan moves files inside")
+    ap.add_argument("--enum", default="/tmp/media-enum.tsv", help="file census TSV")
+    ap.add_argument("--probed", default="/tmp/media-probed.tsv", help="ffprobe census TSV")
+    ap.add_argument("--plex", default="/tmp/plex-movies.tsv",
+                    help="Plex metadata export (clean titles); absent = parser only")
+    ap.add_argument("--out", default="/tmp/jf-plan.tsv")
+    ap.add_argument("--exceptions", default="/tmp/jf-exceptions.txt")
+    return ap.parse_args()
+
+
+ARGS = parse_args()
+ROOT = ARGS.root
+ENUM, PROBE, PLEX = ARGS.enum, ARGS.probed, ARGS.plex
+PLAN, EXCEPTS = ARGS.out, ARGS.exceptions
+ENGINE = os.path.join(os.path.dirname(os.path.abspath(__file__)), "media-titles.py")
+
+spec = importlib.util.spec_from_file_location("eng", ENGINE)
+eng = importlib.util.module_from_spec(spec)
+spec.loader.exec_module(eng)
+
+VIDEO_EXT = {".mkv", ".mp4", ".avi", ".m4v", ".mov", ".ts", ".m2ts", ".mpg",
+             ".mpeg", ".webm", ".wmv", ".divx", ".ogm", ".flv", ".vob", ".iso",
+             ".img", ".mpv", ".3gp", ".rm", ".rmvb", ".m2v", ".mpv"}
+DISC_DIRS = {"BDMV", "VIDEO_TS", "CERTIFICATE", "AACS", "MLPI", "CLIPINF"}
+SUB_EXT = {".srt", ".ass", ".ssa", ".sub", ".vtt"}
+IMG_EXT = {".jpg", ".jpeg", ".png", ".webp", ".bmp", ".gif", ".tbn"}
+AUDIO_DIR_RE = re.compile(r"soundtrack|score|\bost\b|music|song|album", re.I)
+SUB_KEEP_RE = re.compile(r"^(?!.*\d)(?!.*[._])(?!.*\[)(?!.*\()([A-Za-z][A-Za-z -]{1,24})$")
+MUSIC_TOP_RE = re.compile(r"(?i)soundtrack|score|\bost\b|music|album|song")
+SUB_DIR_OK = re.compile(r"(?i)^$|subs?|subtitles?|srt|custom ?subs?(\[.*\])?"
+                        r"|(additional|extra) languages|languages|resources")
+AUDIO_DIR_OK = re.compile(r"(?i)^$|audio(s)?|extra audio|additional audio"
+                          r"|(additional|extra) languages|commentar(y|ies)"
+                          r"|audio commentary|audio tracks?")
+
+
+AUDIO_EXT = {".mp3", ".flac", ".m4a", ".aac", ".dts", ".wav", ".ac3", ".mka",
+             ".opus", ".ogg", ".wma"}
+NFO_EXT = {".nfo"}
+IMG_PRIMARY = {"poster", "folder", "cover", "default", "movie", "show"}
+IMG_OTHER = {"backdrop": "backdrop", "fanart": "backdrop", "background": "backdrop",
+             "banner": "banner", "logo": "logo", "clearlogo": "logo",
+             "thumb": "thumb", "landscape": "thumb", "art": "backdrop",
+             "clearart": "backdrop", "jacket": "poster", "cdart": "poster",
+             "discart": "poster", "disc": "poster"}
+RES_RE = re.compile(r"\b(2160|4320|4k|1080|720|576|480|360|240)(i|p)\b", re.I)
+# The characters Jellyfin's matcher cannot carry in a file or folder name. "/" is
+# one of them and it is the dangerous one: Plex titles KEEP it ("V/H/S/2", and this
+# library holds six of those films), so leaving it in a name turns ONE movie folder
+# into a nested tree — Movies/V/H/S/2 (2013)/… — and Jellyfin then sees four
+# folders, none of them a movie.
+BAD_CHARS = re.compile(r'[<>:"/|?*]')
+PART_RE = re.compile(r"(?:^|[\s._-])(cd|dvd|part|pt|disc|disk|disco)"
+                     r"[\s._-]?([1-9abcd])(?:$|[\s._-])", re.I)
+LANGS = {"en", "eng", "it", "ita", "ja", "jpn", "de", "deu", "fr", "fra", "es",
+         "spa", "pt", "por", "ru", "rus", "zh", "chi", "ko", "kor", "sv", "no",
+         "da", "nl", "pl", "cs", "hu", "fi", "tr", "el", "he", "ar", "uk", "ro",
+         "bg", "hr", "sr", "sk", "et", "lv", "lt", "id", "th", "vi", "fa", "la"}
+FLAGS = {"default", "forced", "foreign", "sdh", "cc"}
+NOISE = {"sub", "subs", "subtitle", "subtitles", "srt", "ass", "ext", "external",
+         "the", "a", "an", "of", "and", "in", "on", "to", "for", "with", "ita",
+         "eng", "english", "italian", "dual", "audio", "forced", "default"}
+# Canonical title where the library only ever carries a localised name.
+OVERRIDES = {"28 giorni dopo": "28 Days Later",
+             "28 settimane dopo": "28 Weeks Later"}
+
+rows = []
+for line in open(ENUM):
+    size, ext, path = line.rstrip("\n").split("\t")
+    rows.append({"bytes": int(size), "ext": ext.lower(), "path": path})
+ext_by = {r["path"]: r["ext"] for r in rows}
+size_by = {r["path"]: r["bytes"] for r in rows}
+
+
+def rel(p):
+    return p[len(ROOT) + 1:] if p.startswith(ROOT + "/") else p
+
+
+def parts_of(p):
+    return [q for q in rel(p).split("/") if q]
+
+
+def is_qb(p):
+    return ".!q" in p.lower()
+
+
+def disc_root(p, fold=True):
+    """Absolute root of the disc tree containing p (the MOVIE folder that holds
+    BDMV/ or VIDEO_TS/), else None. `fold` also folds bookkeeping dirs that sit
+    beside the tree (CERTIFICATE/, AACS/) into the same movie folder."""
+    ps = parts_of(p)
+    kinds = DISC_DIRS if fold else {"BDMV", "VIDEO_TS"}
+    for i, q in enumerate(ps):
+        if q.upper() in kinds:
+            return ROOT + "/" + "/".join(ps[:i])
+    return None
+
+
+def row_name(r):
+    """The name a disc-tree entry should be identified by: the engine stores the
+    disc's own folder for it, because "BDMV/index.bdmv" is not a title."""
+    return r.get("name_src") or r["path"]
+
+
+def sanitize(name):
+    name = BAD_CHARS.sub("-", name).replace("\\", "-")
+    name = re.sub(r"\s+", " ", name).strip().strip(". ").strip()
+    return (name[:180] or "untitled")
+
+
+def ok_year(y):
+    """True for a year a real film could carry. Plex stores 0 for a file it indexed
+    but never MATCHED, and "0" is a digit string: trusting it named 67 movie
+    folders "Title (0)" — Jellyfin items with no identity, no year and no poster."""
+    return bool(re.fullmatch(r"(18|19|20)\d\d", y or "")) and 1870 <= int(y) <= 2035
+
+
+WORD_SPLIT = re.compile(r"[^0-9a-z]+")
+
+
+def words_of(text):
+    """The matching words of a name — or of a whole path, which is how callers
+    pass a file together with the folders it sits in.
+
+    Split on everything that is not a letter or a digit: eng.norm does not touch
+    "/", so feeding it "…/Psycho (1960)/Extras" left glued tokens like
+    "(1960)/extras" and "(1954)" that no other name could ever agree with, and a
+    film could not be found by the folder that holds it — which is how one box
+    set's featurettes ended up in another film's folder. Both sides of every
+    comparison below go through here. A bare number counts only when it could be
+    a release year, so release junk ("1080", "2160", "2", "7") never poses as
+    agreement between two names.
+    """
+    out = set()
+    for w in WORD_SPLIT.split(eng.norm(text)):
+        if not w or w in eng.STOP:
+            continue
+        if w.isdigit() and not 1870 <= int(w) <= 2100:
+            continue
+        out.add(w)
+    return out
+
+
+# ------------------------------------------------------------------ Plex titles
+# Plex has matched a large part of the library's media files to real titles
+# ("Alien", not "Alien.Directors.Cut.1979.1080p.BRrip.x265…"). Where the cache
+# holds a MATCH, its title wins; the filename parser carries everything else
+# (extras, disc trees, TV, and whatever Plex never matched).
+def plex_titles():
+    if not os.path.exists(PLEX):
+        return {}
+    out = {}
+    with open(PLEX, newline="") as fh:
+        for r in csv.DictReader(fh, delimiter="\t"):
+            t, y = r["leaf_title"].strip(), r["leaf_year"].strip()
+            # Only a match is an authority, and a match carries a year: an
+            # unmatched row echoes the filename, which the parser reads better
+            # than a copied release name. The junk filter stays for the matches
+            # whose title still carries the release.
+            if t and ok_year(y) and not re.search(
+                    r"2160p|1080p|x26[45]|hevc|bluray|web-?dl", t.lower()):
+                out[r["file"]] = (t, y)
+    return out
+
+
+PLEX_T = plex_titles()
+
+# ------------------------------------------------------------- classification
+erows = eng.load(PROBE)
+for r in erows:
+    for k, v in OVERRIDES.items():
+        if k in r["title"]:
+            r["title"] = v
+            r["key"] = eng.key_of(r["title"])
+            r["base"] = eng.base_words(r["title"], r["key"])
+marks = {r["path"] for r in erows if eng.is_watermark(r["path"], r["bytes"])}
+extras = {r["path"] for r in erows
+          if r["path"] not in marks
+          and eng.is_extra(r["path"], r["title"], r["dur"], ROOT,
+                           matched=r["path"] in PLEX_T)}
+eps = collections.defaultdict(list)
+for r in erows:
+    if r["path"] in marks | extras:
+        continue
+    ep = eng.ep_of(r["path"])
+    if ep:
+        eps[ep[0]].append(r)      # keyed by the SERIES, never the episode title
+tv_keys = {k for k, v in eps.items() if len(v) >= 5}
+tv_set = {r["path"] for k, v in eps.items() if k in tv_keys for r in v}
+# A folder INSIDE a movie's folder is bonus territory even when its own name is
+# not bonus vocabulary: "Saboteur (1942)/Exrtas/Saboteur - A Closer Look" is a
+# misspelled Extras folder, so the engine's folder rule never fired and the
+# featurette became a movie folder of its own — whose words then out-scored the
+# real Saboteur and drew that box set's other extras into it. The rule needs no
+# metadata: the folder holding the longest film owns the shorter video filed
+# under it, provided that shorter file's own name speaks the film's name — a
+# whole other film filed there stays a film, exactly as the engine's folder rule
+# exempts it. A different year in the file's own name IS that other film
+# speaking, so it vetoes the rule.
+def longest_of(rows):
+    """The longest video in each folder: that folder's feature."""
+    out = {}
+    for r in rows:
+        if r["dur"] > 0:
+            d = os.path.dirname(r["path"])
+            if d not in out or r["dur"] > out[d]["dur"]:
+                out[d] = r
+    return out
+
+
+def not_disc(r):
+    """A disc tree's innards belong to the tree, never to the folder's film."""
+    return not any(c in DISC_DIRS for c in r["path"].split("/"))
+
+
+FEATURE_OF = longest_of([r for r in erows if r["path"] not in marks and not_disc(r)])
+
+
+def under_a_feature(path, dur):
+    """True when a folder above this file holds a longer film that this file's
+    own name names."""
+    d = os.path.dirname(os.path.dirname(path))
+    while len(d) > len(ROOT):
+        f = FEATURE_OF.get(d)
+        if f:
+            if f["dur"] < dur:
+                return False
+            theirs = (words_of(os.path.basename(f["path"]))
+                      | words_of(os.path.basename(d)))
+            if not words_of(os.path.basename(path)) & theirs:
+                return False
+            y1 = re.search(r"(?:18|19|20)\d\d", os.path.basename(path))
+            y2 = re.search(r"(?:18|19|20)\d\d", os.path.basename(f["path"]))
+            return not (y1 and y2 and y1.group() != y2.group())
+        d = os.path.dirname(d)
+    return False
+
+
+extras |= {r["path"] for r in erows
+           if r["dur"] > 0 and r["path"] not in marks | extras | tv_set
+           and not_disc(r) and under_a_feature(r["path"], r["dur"])}
+film_rows = [r for r in erows if r["path"] not in marks | extras | tv_set]
+
+buckets = collections.defaultdict(list)
+for r in film_rows:
+    buckets[eng.key_of(r["title"])].append(r)
+
+
+def res_label(r):
+    m = RES_RE.search(r["res"] or "")
+    if m:
+        n, kind = m.group(1).lower(), m.group(2).lower()
+        return "2160p" if n in ("2160", "4k") else f"{n}{kind}"
+    try:
+        w = int((r["res"] or "0x0").split("x")[0])
+    except (ValueError, IndexError):
+        w = 0
+    return "2160p" if w >= 3000 else ("1080p" if w >= 1400 else "SD")
+
+
+def version_label(r):
+    bits = [res_label(r)]
+    vc = (r["vcodec"] or "").lower()
+    if vc in ("av1", "libaom", "libsvtav1"):
+        bits.append("AV1")
+    elif vc in ("vc1", "wmv3"):
+        bits.append("VC1")
+    if r["bytes"] > 40 * 2**30:
+        bits.append("REMUX")
+    if "imax" in r["path"].lower():
+        bits.append("IMAX")
+    return sanitize(" ".join(bits))
+
+
+def group_name(items):
+    titles = collections.Counter(r["title"] for r in items)
+    ascii_t = [t for t in titles if t.isascii()]
+    title = (collections.Counter(ascii_t) if ascii_t else titles).most_common(1)[0][0]
+    years = [r["year"] for r in items if r["year"] != "?"] or \
+            [r["soft"] for r in items if r["soft"]]
+    year = collections.Counter(years).most_common(1)[0][0] if years else None
+    return sanitize(f"{title} ({year})" if year else title)
+
+
+groups = []
+for key, items in buckets.items():
+    for c in eng.dur_clusters(items):
+        g = {"rows": c, "base": group_name(c)}
+        known = [PLEX_T[r["path"]] for r in c if r["path"] in PLEX_T]
+        if known:
+            # one film, one name: a Plex match for any copy names every copy
+            t, yr = collections.Counter(known).most_common(1)[0][0]
+            g["plex"] = t
+            g["base"] = sanitize(f"{t} ({yr})" if ok_year(yr) else t)
+        groups.append(g)
+clash = {b for b, n in collections.Counter(g["base"] for g in groups).items() if n > 1}
+for g in groups:
+    if g["base"] in clash:
+        # two genuinely different films share title+year: separate folders, and
+        # say so in the name, because Jellyfin can only match one item per folder
+        g["base"] = sanitize(g["base"] + " - " + version_label(g["rows"][0]))
+        g["flag"] = "same title and year as another film"
+
+moves, mkdirs, problems, readmes = [], set(), [], []
+# --------------------------------------------------------------- unnamed groups
+# A movie folder has to READ as a title. The engine's runtime veto says any file
+# over 45 minutes that is not named like an extra is a film whatever it is called
+# — true for Jellyfin, but a 118-minute documentary named "Extra 1" then produced
+# a movie folder named "1", and folders named "1", "2", "3" are Jellyfin items
+# with no identity whose numeric keys went on to capture every subtitle named
+# "3_Eng.srt" in the library. A year proves a dated feature ("300 (2006)" is a
+# real film), so the gate is: a year, or a word that is not release noise. With
+# neither, the rows are bonus material until a human names them.
+def named(g):
+    if re.search(r"\(\d{4}\)$", g["base"]):
+        return True
+    return any(w.isalpha() and len(w) >= 2 for w in eng.name_words(g["base"]))
+
+
+unnamed = [g for g in groups if not named(g)]
+groups = [g for g in groups if named(g)]
+for g in unnamed:
+    for r in g["rows"]:
+        extras.add(r["path"])
+        problems.append(
+            f"UNNAMED film {rel(r['path'])} parses to {g['base']!r} with no year: "
+            f"bonus pile, not a folder — give the file a real name and re-plan")
+
+
+group_of = {}
+for g in groups:
+    g["dir"] = f"{ROOT}/{g['base']}"
+    for r in g["rows"]:
+        group_of[r["path"]] = g
+        dr = disc_root(r["path"])
+        if dr:
+            group_of[dr] = g
+by_path = {r["path"]: r for r in erows}
+
+# -------------------------------------------------------------- bookkeeping
+claimed = {}
+existing = set(rows_map := {r["path"] for r in rows})
+
+
+at_target, booked_src = set(), set()
+
+
+def book(src, dst, kind):
+    if src == dst:
+        at_target.add(src)
+        booked_src.add(src)
+        return
+    if os.environ.get("JF_TRACE") and (src in booked_src or dst.lower() in claimed):
+        why = "rebook" if src in booked_src else "collision"
+        problems.append(f"TRACE {why} kind={kind} src={rel(src)} dst={rel(dst)}")
+    if src in booked_src:          # one source may only ever move once
+        problems.append(f"RE-BOOK {rel(src)} -> {rel(dst)} (already planned)")
+        return
+    if dst.lower() in claimed:
+        problems.append(f"COLLISION {rel(dst)}\n    from {rel(src)}\n    and  {rel(claimed[dst.lower()])}")
+        return
+    claimed[dst.lower()] = src
+    booked_src.add(src)
+    comp = dst.split("/")
+    for i in range(1, len(comp)):
+        mkdirs.add("/".join(comp[:i]))
+    moves.append((src, dst, kind))
+
+
+def uniq_dir(d, stem, ext, hint=""):
+    """A destination that no other source already took."""
+    cand = f"{d}/{stem}{ext}"
+    if cand.lower() not in claimed:
+        return cand
+    n = 2
+    while True:
+        cand = f"{d}/{stem} {n}{ext}"
+        if cand.lower() not in claimed:
+            return cand
+        n += 1
+
+
+def media_dir(p):
+    dr = disc_root(p)
+    return os.path.dirname(dr) if dr else os.path.dirname(p)
+
+
+# ------------------------------------------------------------------ films
+# film / disc moves: the largest copy carries the folder's own name, the rest
+# become Jellyfin versions; a disc tree moves as a whole into the movie folder
+
+
+def vext(p):
+    return ext_by.get(p) or os.path.splitext(p)[1].lower()
+disc_children = collections.defaultdict(list)
+for q in rows:
+    dr = disc_root(q["path"])
+    if dr:
+        disc_children[dr].append(q["path"])
+for g in groups:
+    primary_taken = False
+    disc_kinds = set()
+    for r in sorted(g["rows"], key=lambda r: -r["bytes"]):
+        dr = disc_root(r["path"])
+        if dr:
+            inner = os.path.basename(dr).upper()
+            if inner in disc_kinds:
+                problems.append(
+                    f"DISC-VERSION {g['base']}: extra disc tree {rel(dr)} cannot "
+                    f"share the folder — discs have no multiple versions")
+                continue
+            disc_kinds.add(inner)
+            for src in disc_children[dr]:
+                tail = os.path.relpath(src, dr)
+                fold = disc_root(src, fold=False)
+                if fold != dr and os.path.dirname(src) != dr:
+                    tail = os.path.basename(dr) + "/" + tail
+                book(src, f"{g['dir']}/{tail}", "disc-file")
+            continue
+        label = "" if not primary_taken else f" - {version_label(r)}"
+        dst = f"{g['dir']}/{g['base']}{label}{vext(r['path'])}"
+        n = 2
+        while dst.lower() in claimed:
+            dst = f"{g['dir']}/{g['base']}{label} {n}{vext(r['path'])}"
+            n += 1
+        r["dst"] = dst
+        primary_taken = True
+        book(r["path"], dst, "film")
+    g["primary"] = primary_taken
+
+# ------------------------------------------------------------------ extras
+EXTRA_KIND = [("trailers", r"trailer|teaser|preview|promo|tv spot|advert"),
+              ("deleted scenes", r"deleted|alt scene|gag reel|blooper|outtake"),
+              ("interviews", r"interview|cast and crew|session piece|commentary"),
+              ("featurettes", r"featurette|behind the scenes|making of|documentary"
+                              r"|heritage|archival|video archive|music video"),
+              ("samples", r"sample"),
+              ("shorts", r"\bshort\b|short film"),
+              ("clips", r"\bclip\b|scene selection|montage|epilogue|isolated score"),
+              ("other", r"menu|logo|gallery|still|trivia|storyboard|screen test"
+                        r"|audition|test footage|watermark|workprint|poster"
+                        r"|picture in picture|singalong|karaoke|original score")]
+THEME_RE = re.compile(r"theme|opening song|ending song|soundtrack|score"
+                      r"|\bost\b|music|song", re.I)
+BACKDROP_RE = re.compile(r"backdrop|intro", re.I)
+
+
+def extras_kind(path):
+    low = path.lower()
+    if THEME_RE.search(low):
+        return "theme-music"
+    if BACKDROP_RE.search(low):
+        return "backdrops"
+    for name, rx in EXTRA_KIND:
+        if re.search(rx, low):
+            return name
+    return "extras"
+
+
+def attachable_title(path):
+    t, _y = eng.title_of(path)
+    return eng.key_of(t) if t and not eng.is_watermark(path, size_by.get(path, 0)) else None
+
+
+for g in groups:
+    g["tops"] = {parts_of(r["path"])[0] for r in g["rows"]}
+top_groups = collections.defaultdict(list)
+for g in groups:
+    for t in g["tops"]:
+        top_groups[t].append(g)
+group_by_key = collections.defaultdict(list)
+for g in groups:
+    for r in g["rows"]:
+        group_by_key[eng.key_of(r["title"])].append(g)
+group_by_dir = {}
+for g in groups:
+    for r in g["rows"]:
+        group_by_dir.setdefault(media_dir(r["path"]), g)
+        dr = disc_root(r["path"])
+        if dr:
+            group_by_dir.setdefault(os.path.dirname(dr), g)
+
+
+def group_words(g):
+    """The words that identify a film group: its titles and the names of the
+    folders its files live in."""
+    if "words" not in g:
+        w = set()
+        if g.get("plex"):
+            w |= words_of(g["plex"])
+        for r in g["rows"]:
+            w |= words_of(r["title"])
+            w |= words_of(os.path.basename(media_dir(r["path"])))
+        g["words"] = w
+    return g["words"]
+
+
+def score_words(text, groups):
+    """Ranks film groups by how many of their words a path carries."""
+    words = words_of(text)
+    out = [(len(words & group_words(g)), g) for g in groups]
+    out.sort(key=lambda x: -x[0])
+    return out
+
+
+def group_for_orphan(path):
+    """Which movie folder an extra/sidecar belongs to. A collection folder holding
+    several films must be resolved by WORDS, never by position: taking the first
+    film of the folder attached every loose subtitle in a box set to whichever
+    movie happened to sort first."""
+    stem = os.path.splitext(os.path.basename(path))[0]
+    key = attachable_title(path)
+    if key and group_by_key.get(key):
+        cands = group_by_key[key]
+        top = parts_of(path)[0] if len(parts_of(path)) > 1 else ""
+        same_top = [g for g in cands if any(parts_of(r["path"])[0] == top
+                                            for r in g["rows"] if len(parts_of(r["path"])) > 1)]
+        pool = same_top or cands
+        return pool[0] if len(pool) == 1 else score_words(stem, pool)[0][1]
+    gtop = parts_of(path)[0] if len(parts_of(path)) > 1 else ""
+    pool = top_groups.get(gtop, [])
+    if len(pool) > 1:                    # a loose collection folder (box set, bonus
+        n, g = score_words(stem + " " + os.path.dirname(path), pool)[0]
+        return g if n >= 2 else None     # album) is resolved by words, not position
+    if len(pool) == 1:
+        return pool[0]
+    d = os.path.dirname(path)
+    while d and d != ROOT:
+        if d in group_by_dir:
+            return group_by_dir[d]
+        d = os.path.dirname(d)
+    return None
+
+
+tv_tops = {parts_of(p)[0] for p in tv_set}
+def film_folder_for(path):
+    """A movie folder for a lone feature that has no group: derived from its own
+    title when a year proves it, else None (goes to the review pile)."""
+    t, y = eng.title_of(path)
+    if not t or y == "?":
+        return None
+    d = uniq_dir(ROOT, sanitize(f"{t} ({y})"), "", "")
+    mkdirs.add(d)
+    return d
+
+
+for p in sorted(extras):
+    if disc_root(p):
+        continue
+    if len(parts_of(p)) > 1 and parts_of(p)[0] in tv_tops:
+        continue                        # a special inside a TV set: staged with it
+    g = group_for_orphan(p)
+    ext = ext_by.get(p, os.path.splitext(p)[1].lower())
+    stem = os.path.splitext(os.path.basename(p))[0]
+    if g is None:
+        solo = film_folder_for(p)
+        if solo:
+            dst = f"{solo}/{sanitize(stem)}{ext}"
+            problems.append(f"REVIEW lone feature {rel(p)}: gets its own folder "
+                            f"{os.path.basename(solo)}")
+        else:
+            dst = f"{ROOT}/_unsorted bonus/{extras_kind(p)}/{sanitize(stem)}{ext}"
+            problems.append(f"NO-FILM extra {rel(p)}")
+    else:
+        dst = f"{g['dir']}/{extras_kind(p)}/{sanitize(stem)}{ext}"
+    book(p, uniq_dir(os.path.dirname(dst), os.path.basename(dst)[:-(len(ext) or 1)],
+                     ext, os.path.dirname(p)), "extra")
+
+# Disc trees the engine never saw (a MakeMKV scratch folder, an AACS remnant):
+# they belong to the film of the folder they sit in, kept as a named subtree.
+for q in rows:
+    pth = q["path"]
+    dr = disc_root(pth)
+    if not dr or pth in by_path or pth in booked_src:
+        continue
+    g = group_by_dir.get(dr) or (top_groups[parts_of(pth)[0]][0]
+                                 if len(top_groups.get(parts_of(pth)[0], [])) == 1 else None)
+    if g is None:
+        continue
+    inner = os.path.basename(dr)
+    problems.append(f"REVIEW disc remnant {rel(dr)} moves inside {g['base']} "
+                    f"(a second master: compare before keeping both)")
+    for src in disc_children[dr]:
+        if src not in booked_src:
+            book(src, f"{g['dir']}/{inner}/{os.path.relpath(src, dr)}", "disc-file")
+
+
+# ----------------------------------------------------------------- sidecars
+media_by_dir = collections.defaultdict(list)
+for r in erows:
+    if r["path"] in marks:
+        continue
+    dst = r.get("dst")
+    if dst:
+        media_by_dir[media_dir(r["path"])].append(r)
+
+
+
+def sidecar_target(path, e):
+    d = os.path.dirname(path)
+    stem = os.path.splitext(os.path.basename(path))[0]
+    if d == ROOT:
+        # A file loose in the library ROOT names no film. media_by_dir[ROOT] is the
+        # set of root-level MOVIE files, and scoring a stray name against it
+        # attached every root-level leftover to whichever root-level film was
+        # largest — the plan sent ".DS_Store" into "Titanic (1997)/other/" that
+        # way. A root-level stray is nobody's sidecar: it goes to the review pile.
+        return None, None
+    cands = media_by_dir.get(d, [])
+    if not cands:
+        nd = os.path.dirname(d)          # a subs/ or Extras/ folder beside the media
+        while nd and nd != ROOT and not media_by_dir.get(nd):
+            nd = os.path.dirname(nd)
+        if media_by_dir.get(nd):
+            if nd != d:                  # a NAMED subfolder (Subs/, Extras/, the
+                g = group_for_orphan(path)   # box set's own root) carries evidence
+                dirs = {os.path.dirname(r.get("dst", "")) for r in media_by_dir[nd]}
+                if g is None or g["dir"] not in dirs:
+                    # ...but only a STRONG word match outranks the folder the file
+                    # sits in. A weak one is an accident: "3_Eng.srt" scored 1
+                    # against a junk group named "3" and escaped its own movie's
+                    # Subs/ folder, while "03 xXx 3 Return Of Xander Cage" scores 4
+                    # and genuinely belongs to the other film of the set.
+                    n = score_words(stem, [g])[0][0] if g else 0
+                    if n >= 2:
+                        return (g, None)
+                cands = media_by_dir[nd]
+    if not cands:
+        g = group_for_orphan(path)
+        return (g, None) if g else (None, None)
+    if len(cands) > 1:
+        w = [t for t in eng.norm(stem).split() if t not in NOISE]
+        cands = sorted(cands, key=lambda r: (
+            -len([t for t in eng.norm(os.path.basename(r["path"])).split() if t in w]),
+            -r["bytes"]))
+    return (None, cands[0])
+
+
+for q in rows:
+    p, e = q["path"], q["ext"]
+    if p in by_path or p in extras or p in marks or is_qb(p) or disc_root(p):
+        continue
+    if os.path.basename(p).startswith("._"):
+        # macOS AppleDouble resource forks: a few hundred bytes carrying a name
+        # COPIED FROM ANOTHER FILM. Word-matching them to "their" film scattered
+        # dead files through four movie folders (and they are video extensions in
+        # the AppleDouble case, so they also skipped every other filter). They
+        # claim nothing: one pile, and they stay in it — nothing is deleted here.
+        book(p, uniq_dir(f"{ROOT}/_macos sidecars", sanitize(os.path.basename(p))[:120],
+                         "", os.path.dirname(p)), "junk")
+        problems.append(f"JUNK AppleDouble sidecar {rel(p)} (macOS resource fork)")
+        continue
+    if e == ".iso":         # a DVD image: Jellyfin plays it as a movie file
+        g = group_for_orphan(p)
+        stem = os.path.splitext(os.path.basename(p))[0]
+        if g is not None:
+            book(p, uniq_dir(g["dir"], g["base"], e, stem) if g.get("primary") is not None
+                 else f"{g['dir']}/{g['base']}{e}", "iso")
+        else:
+            solo = film_folder_for(p)
+            if solo:
+                book(p, f"{solo}/{sanitize(stem)}.iso", "iso")
+                problems.append(f"REVIEW disc image {rel(p)}: DVD rip, alone in its "
+                                f"folder -> {os.path.basename(solo)}")
+            else:
+                book(p, f"{ROOT}/_unsorted files/{sanitize(os.path.basename(p))}", "orphan")
+                problems.append(f"NO-FILM disc image {rel(p)}")
+        continue
+    if e in VIDEO_EXT:
+        # A video the engine never probed (unreadable header) and not an AppleDouble
+        # remnant: the plan does not guess at it, so it stays where it is.
+        continue
+    stem = os.path.splitext(os.path.basename(p))[0]
+    g, media = sidecar_target(p, e)
+    if media is not None:
+        base = os.path.splitext(os.path.basename(media["dst"]))[0]
+        d = os.path.dirname(media["dst"])
+    elif g is not None:
+        base, d = g["base"], g["dir"]
+    else:
+        # Nothing names this file's film. NEVER scatter it: an archive set or an
+        # album tree must move as the unit it is, so the whole loose folder goes
+        # to the review pile and a film folder's own leftovers stay with it.
+        ps = parts_of(p)
+        if len(ps) < 2:
+            dst = f"{ROOT}/_unsorted files/{sanitize(stem)}{e}"
+        elif len(top_groups.get(ps[0], [])):
+            dst = f"{top_groups[ps[0]][0]['dir']}/other/{'/'.join(sanitize(c) for c in ps[1:])}"
+        else:
+            dst = f"{ROOT}/_unsorted files/{'/'.join(sanitize(c) for c in ps)}"
+        book(dst=dst, src=p, kind="orphan")
+        problems.append(f"NO-OWNER sidecar {rel(p)}")
+        continue
+    if e in IMG_EXT:
+        low = eng.norm(stem)
+        word = next((w for w in IMG_PRIMARY if re.search(rf"\b{w}\b", low)), None)
+        kind = "poster" if word else next(
+            (IMG_OTHER[w] for w in IMG_OTHER if re.search(rf"\b{w}\b", low)), None)
+        if re.search(r"\btheme\b", low):
+            dst = f"{d}/theme{e}"
+        else:
+            dst = f"{d}/{base}-{kind or 'art'}{e}" if kind else f"{d}/{sanitize(stem)}{e}"
+        book(p, uniq_dir(os.path.dirname(dst), os.path.basename(dst)[:-(len(e) or 1)],
+                         e, os.path.dirname(p)), "image")
+    elif e in NFO_EXT:
+        book(p, uniq_dir(d, base, ".nfo", os.path.dirname(p)), "nfo")
+    elif e in SUB_EXT | AUDIO_EXT:
+        toks = [t for t in re.split(r"[._\-\s]+", stem) if t]
+        langs = [t.lower() for t in toks if t.lower() in LANGS]
+        flags = [t.lower() for t in toks if t.lower() in FLAGS]
+        keep = [sanitize(t) for t in toks
+                if t.lower() not in langs + flags + list(NOISE)
+                and not re.fullmatch(r"\d{3,4}p?|cd\d+|\d{4}|\d+ch|\d+.\d", t.lower())]
+        bits = [base] + ([sanitize(" ".join(keep[:4]))] if keep else []) + langs + flags
+        if e in SUB_EXT:
+            # subtitles: <movie>/srt[/language]
+            d = f"{d}/srt"
+            pd = os.path.basename(os.path.dirname(p))
+            if SUB_KEEP_RE.match(pd) and not SUB_DIR_OK.match(pd):
+                d = f"{d}/{sanitize(pd)}"      # a real language folder survives
+        elif e in AUDIO_EXT and AUDIO_DIR_RE.search(os.path.dirname(p)):
+            d = f"{d}/theme-music"                 # bonus audio: one canonical folder
+            pd = os.path.basename(os.path.dirname(p))
+            if not MUSIC_TOP_RE.search(pd):        # an album folder keeps its name
+                d = f"{d}/{sanitize(pd)}"
+        else:
+                pd = os.path.basename(os.path.dirname(p))
+                if not AUDIO_DIR_OK.match(pd):
+                    d = f"{d}/{sanitize(pd)}"
+        stem2 = ".".join(bits) or sanitize(os.path.basename(p))
+        book(p, uniq_dir(d, stem2, e, os.path.basename(p)), "track")
+    else:
+        book(p, uniq_dir(f"{d}/other", sanitize(os.path.basename(p))[:120], e,
+                         os.path.dirname(p)), "misc")
+
+readmes.append(f"README in {ROOT}: this library was relaid out into Jellyfin's "
+               "one-folder-per-movie layout; `_release-group watermarks/`, "
+               "`_unsorted files/` and `_unsorted bonus/` hold files no film claims.")
+for key in tv_keys:
+    readmes.append(f"README in {ROOT}/{sanitize('_TV needs a TV library - ' + key)}: "
+                   "these are TV episodes found in the MOVIES library; move the folder "
+                   "to a TV library source, else Jellyfin groups them wrongly as one "
+                   "movie.")
+
+# --------------------------------------------------------------------- TV
+for key in tv_keys:
+    items = eps[key]
+    year = collections.Counter(r["year"] for r in items if r["year"] != "?")
+    yr = year.most_common(1)[0][0] if year else None
+    base = f"_TV needs a TV library - {sanitize(key)}" + (f" ({yr})" if yr else "")
+    tops = {parts_of(r["path"])[0] for r in items}
+    members = sorted([r["path"] for r in items]
+                     + [q for q in extras | marks if len(parts_of(q)) > 1
+                        and parts_of(q)[0] in tops],
+                     key=lambda q: q)
+    for src in members:
+        rel_parts = rel(src).split("/")
+        tail = "/".join(sanitize(c) for c in rel_parts[:-1] + [rel_parts[-1]])
+        book(src, f"{ROOT}/{base}/{tail}", "tv")
+for p in sorted(marks):
+    stem, ext = os.path.splitext(os.path.basename(p))
+    book(p, uniq_dir(f"{ROOT}/_release-group watermarks", sanitize(stem)[:120], ext,
+                     os.path.dirname(p)), "watermark")
+for q in rows:
+    if is_qb(q["path"]):
+        problems.append(f"SKIPPED qBittorrent partial {rel(q['path'])}")
+        book(q["path"], q["path"], "qb-partial")     # stays: resume data
+
+# --------------------------------------------------------------- debug hook
+if os.environ.get("JF_ISO"):
+    x = ROOT + "/High Anxiety (Mel Brooks) [1977] nordic subs  DVDrip/High Anxiety [1977].iso"
+    print("attachable", attachable_title(x), "| dir", os.path.dirname(x) in group_by_dir,
+          "| g", group_for_orphan(x) and group_for_orphan(x)["base"])
+    raise SystemExit
+if os.environ.get("JF_DEBUG"):
+    import json
+    unp = sorted({r["path"] for r in rows} - {m[0] for m in moves} - at_target)
+    print("at_target", len(at_target))
+    print("kinds-unplanned", collections.Counter(os.path.splitext(u)[1] for u in unp).most_common(8))
+    for up in unp[:20]:
+        d = os.path.dirname(up)
+        print("UNP", up[:90], "| mediadir?", d in media_by_dir, "| by_path", up in by_path,
+              "| extras", up in extras, "| discroot", disc_root(up), "| ext", os.path.splitext(up)[1])
+
+# -------------------------------------------------------------- validation
+srcs = [m[0] for m in moves]
+dup_src = {s for s, n in collections.Counter(srcs).items() if n > 1}
+for s in dup_src:
+    problems.append(f"DOUBLE-CLAIM source {rel(s)}")
+all_src = set(srcs)
+untouched = {r["path"] for r in rows} - all_src
+bad_names, into_alive, lost = [], [], []
+for src, dst, kind in moves:
+    for comp in dst[len(ROOT) + 1:].split("/"):
+        if (BAD_CHARS.search(comp) or comp != comp.strip(". ") or not comp
+                or len(comp) > 200):
+            bad_names.append(f"{kind}: {comp!r} in {rel(dst)}")
+    if dst in untouched:                       # something staying put is there
+        into_alive.append(f"{rel(src)} -> {rel(dst)} (occupied by a file that stays)")
+    if dst.lower() in {s.lower() for s in untouched}:
+        into_alive.append(f"{rel(src)} -> {rel(dst)} (case-collides with a survivor)")
+    if dst.startswith(src + "/") and src in all_src:
+        lost.append(f"{rel(src)} moves into its own subtree")
+# every accounted-for source must exist exactly once in the plan
+covered = {r["path"] for r in rows}
+not_planned = sorted(covered - all_src - at_target)
+
+with open(PLAN, "w") as out:
+    out.write("action\tsrc\tdst\tkind\n")
+    for d in sorted(mkdirs, key=lambda q: (q.count("/"), q)):
+        if d.startswith(ROOT + "/") and d != ROOT:
+            out.write(f"mkdir\t\t{d}\tdir\n")
+    for p in sorted(booked_src - {m[0] for m in moves}):
+        out.write(f"keep\t{p}\t{p}\n")
+    for src, dst, kind in sorted(moves, key=lambda m: m[0]):
+        out.write(f"move\t{src}\t{dst}\t{kind}\n")
+
+kinds = collections.Counter(k for _s, _d, k in moves)
+films_n = sum(1 for _s, _d, k in moves if k == "film")
+folders = len({os.path.dirname(d) for _s, d, k in moves if k in ("film", "disc-file")})
+with open(EXCEPTS, "w") as out:
+    out.write(f"groups={len(groups)} moves={len(moves)} folders={folders}\n")
+    out.write(f"kinds={dict(kinds)}\n")
+    out.write(f"at-target-already={len(at_target)} unplanned={len(not_planned)}\n")
+    out.write("\n=== WHAT THE EXECUTOR MUST ALSO WRITE ===\n")
+    out.write("\n".join(sorted(set(readmes))) + "\n")
+    out.write("\n=== PROBLEMS ===\n")
+    out.write("\n".join(sorted(set(problems))) + "\n")
+    out.write("\n=== BAD NAMES ===\n" + "\n".join(sorted(set(bad_names))) + "\n")
+    out.write("\n=== TARGETS OCCUPIED BY SURVIVORS ===\n"
+              + "\n".join(sorted(set(into_alive))) + "\n")
+    out.write("\n=== LEFTOVER SOURCES (first 60) ===\n"
+              + "\n".join(rel(p) for p in not_planned[:60]) + "\n")
+print(f"groups={len(groups)} folders={folders} moves={len(moves)} "
+      f"mkdirs={len(mkdirs)} film-files={films_n}")
+print(f"kinds={dict(kinds)}")
+print(f"problems={len(set(problems))} bad-names={len(set(bad_names))} "
+      f"occupied={len(set(into_alive))} double-claim={len(dup_src)}")
+print(f"left-unmoved={len(not_planned)}")
+print(f"tv-keys={len(tv_keys)} extras={len(extras)} marks={len(marks)} "
+      f"disc-trees={len(disc_children)}")

scripts/media-titles.py

@@ -0,0 +1,572 @@
+#!/usr/bin/env python3
+r"""The fleet's release-filename title parser: the grouping authority the relayout
+plan uses for every file the media server never matched.
+
+  report:  media-titles.py report [--probed TSV] [--out TSV]
+           the similar-copies census (DUP / REVIEW / EP-DUP groups) over the
+           ffprobe census scripts/media-census.py writes.
+  parse:   media-titles.py parse NAME [NAME…]
+           what title_of() makes of a name — the way to check a naming accident
+           before it merges two films into one folder.
+
+scripts/media-plan.py imports this file; over there the paths are arguments.
+
+Method — three independent signals, so a naming accident cannot merge two films:
+  * franchise KEY  : first two significant title tokens. Stable across parts of a
+    collection ("resident evil", "jump street") but NOT film-specific — it only
+    selects candidates.
+  * DURATION       : the identity check. Same master encodes differ by at most a
+    couple of seconds; different films / cuts do not collide.
+  * SIZE SPREAD + YEAR: decide the tier. Two files same size AND same duration
+    are byte-twins (the hash pass confirms); same duration but different sizes are
+    two different releases of one film (deletable redundancy).
+
+TV episodes (SxxEyy in the name) are grouped separately: their uniform runtime
+would otherwise collapse a whole series into one "duplicate" group.
+"""
+import argparse
+import collections, os, re, sys, unicodedata
+
+# Library root: the folder directly under this one names the film, and every
+# folder below it is structure (Extras/, Featurettes/, Disc 1/, BDMV/). Tests pass
+# their own; nothing here reads media, only the census TSVs about it.
+ROOT = os.environ.get("MEDIA_ROOT", "/mnt/entropy/Movies")
+
+# Release-tag vocabulary: never part of a title, so the title stops here.
+JUNK = {"5.1", "7.1", "2.0", "1.0", "6.1", "4.0", "extras"}
+JUNK_RE = re.compile(
+    r"^("
+    r"\d{3,5}p|\d{1,2}k|uhd|fhd|imax|x ?26[45]|hevc|h ?26[45]|avc|av1|x ?av1|vp[89]"
+    r"|divx|xvid|mpeg-?2?|\d{1,2}-?bit|hdr ?10\+?|hdr|dolby ?vision|dvh?|sdr|hybrid"
+    r"|re-?mux|bd-?re-?mux|blu-?ray|bdrip|brrip|bdmux|bd|web-?(rip|dl)?|hdtv|hdrip"
+    r"|dvd-?(rip|r|scr)?|hdcam|telesync|screener|ppv|re-?encode|extras?"
+    r"|ddp? ?\d(\.\d)?|dd\d(\.\d)?|dts-?hd(\.ma)?|dts|truehd|atmos|e-?ac-?3?|ac-?3"
+    r"|aac\d?(\.\d)?|lpcm|flac|opus|mp3|pcm|\d\.\d(\.\d)?|5 ?1|7 ?1|2 ?0|1 ?0|6 ?1|4 ?0"
+    r"|subs?-?(it|en|eng|ita)?|subbed|dual(-?audio)?|multi|dub(bed)?"
+    r"|ita|eng|en|it|de|fr|es|pl|jp|jap|lat|esp|ger|fre|rus|zho|hin|tam|tel"
+    r"|repack|proper|internal|limited|unrated|extended|director'?s|cut|remaster(ed)?|uncut"
+    r"|special|edition|version|crc32|sha1|sfv|par2|nfo|nzb|mux|complete|unpack"
+    r"|certified|scene|p2p|part\d*|cd\d+|disc\d+|disk\d+|vob|idx|srt|ass|ssa|sup"
+    r"|m3u|cue|log|txt|mkv|mp4|avi|imdb\d[\d.]*|yts(\.\w+)?|rarbg|collection"
+    r")$", re.I)
+RES_RE = re.compile(r"^\d{3,5}p$|^\d{1,2}k$|^\d+x\d+$", re.I)
+YEAR_RE = re.compile(r"^(19|20)\d{2}$")
+SEP_RE = re.compile(r"[._\s\-\[\](){}+!~,;:&@#/*'\"`´|]+")
+IDX_RE = re.compile(r"^\d{1,3}[a-z]?$")
+WORD_RE = re.compile(r"[A-Za-z0-9']+$")
+EP_RE = re.compile(r"(?i)(?<![a-z0-9])(s(?:eason[\s._-]*)?\d{1,2}[\s._-]*e(?:p(?:isode)?[\s._-]*)?"
+                   r"\d{1,3}|s\d{1,2}\s?e\d{1,3}|\d{1,2}x\d{2,3}|part[\s._-]*\d{1,2})")
+STOP = {"a", "an", "the", "il", "la", "le", "i", "un", "une", "el", "los", "las", "de",
+        "del", "du", "di", "e", "ed", "y", "and", "et", "for", "of", "to", "in", "on",
+        "vs", "avec", "con"}
+
+
+def norm(s):
+    s = unicodedata.normalize("NFKD", s)
+    return "".join(c for c in s if not unicodedata.combining(c)).lower().strip()
+
+
+def fold_sep(s):
+    """norm(), plus: every character that is left over is not a letter, digit or
+    apostrophe, so it is a SEPARATOR whatever codepoint it hides behind — not part
+    of the word. Release groups file titles with lookalikes for punctuation the
+    filesystem forbids: "V ∕ H ∕ S ∕ 2" carries U+2215 DIVISION SLASH, "S꞉ Viral"
+    U+A789, and en dashes hide in "Star Wars – A New Hope". NFKD does NOT fold
+    those (no decomposition mapping), and the scrubber inside title_of DELETES
+    what it does not recognise, so five different V/H/S films squashed into the one
+    title "vhs2" and would have shared one folder. Deleting is the wrong operation:
+    they separate. Accents are gone by then — norm() removed the combining marks —
+    so "Amélie" stays "amelie" and never splits into "ame lie"."""
+    return " ".join("".join(c if c.isalnum() or c == "'" else " "
+                            for c in norm(s)).split())
+
+def title_of(path):
+    """(title, year) from the basename.
+
+    Two passes. (1) Tokenize; every 4-digit year becomes a sentinel in place.
+    (2) Find the boundary: the first release-noise token (resolution / codec /
+    source / audio / language / group vocabulary) ends the title, so trailing
+    tags can never leak in. (3) Collect up to that boundary: leading index
+    numbers ("03."), leading year sentinels ("1940 - Fantasia") and leading noise
+    are skipped while nothing is collected; once a real word appears a sentinel
+    CLOSES the title. A title sitting BEFORE the year sentinel ("300.2006.…" or
+    "12. Monkeys. 1995") is a numeric title and is prepended to what follows.
+    """
+    # fold_sep BEFORE tokenizing, so a lookalike separator separates (see its
+    # docstring): the tokens below are then plain alnum/apostrophe words.
+    name = fold_sep(os.path.splitext(os.path.basename(path))[0])
+    raw = [x for x in SEP_RE.split(name) if x]
+    years = [t for t in raw if YEAR_RE.match(t)]
+    two_year = len(years) > 1                       # "2001…1968": one of them is the title
+    toks, year = [], None
+    for t in raw:
+        if YEAR_RE.match(t):
+            year = t                                    # last year wins
+            toks.append("\x00")
+        else:
+            toks.append(t)
+    stop, seen_word = len(toks), False
+    for i, t in enumerate(toks):
+        if t == "\x00":
+            continue
+        c = norm(re.sub(r"[^A-Za-z0-9']", "", t))
+        if (not c or RES_RE.match(c) or JUNK_RE.match(c) or c in JUNK
+                or not WORD_RE.match(c)) and not (IDX_RE.match(t) and not seen_word):
+            stop = i
+            break
+        if c.isalpha():
+            seen_word = True
+    # A leading number is a track index ("01. Bye Bye Baby", "05. Casablanca") unless
+    # it is three digits or more ("007", "000", "1408") or the name carries no year
+    # at all ("12 Monkeys", "831"). A 1-2 digit prefix beside a year is an index;
+    # the rare real title that looks like one ("0 Things I Hate About You (1999)")
+    # loses its zero to the parser — Plex's matched title overrides it anyway.
+    lead = re.split(r"[\s._-]", name, maxsplit=1)[0]
+    track_idx = bool(re.fullmatch(r"\d{1,2}[a-z]?", lead)) and bool(year)
+    keep, num_title = [], None
+    for t in toks[:stop]:
+        if t == "\x00":
+            if keep:
+                break                                   # year closes the title
+            if two_year and t is not None and years[0] != year:
+                # two DIFFERENT years ("2001…1968"): the leading one is the title;
+                # "1940 - Fantasia (1940)" repeats one year, so it is not a title
+                num_title = num_title or years[0]
+            continue                                    # leading year: not the title
+        if track_idx and IDX_RE.match(t) and not keep and t == lead:
+            continue                                    # "03." / "1c." ordering
+        c = norm(re.sub(r"[^A-Za-z0-9']", "", t))
+        if not keep and not c.isalpha():
+            num_title = num_title or (c or None)
+            continue                                    # numeric title candidate
+        keep.append(c)
+        if len(keep) >= 8:
+            break
+    keep = [k for k in keep if not re.fullmatch(r"(disc|disk|cd|dvd|part)\d*", k)]
+    title = " ".join(keep)
+    if not keep:
+        # nothing collected: a pure-numeric leading token is the title even when it
+        # sits where the release year sits ("1917.2019.2160p…"), and "300.2006…" too
+        for t in raw:
+            c = norm(re.sub(r"[^A-Za-z0-9']", "", t))
+            if re.fullmatch(r"(?:[12]\d{3}|\d{1,3})", c) and (c != year or two_year):
+                return c, year or "?"    # never "2160p" / "1080": they are not pure digits
+    if (num_title and title and not keep[0].isdigit()
+            and (two_year or not track_idx)):
+        # a leading number that only echoes the release year ("1917.2019.2160p…") is
+        # not a title; one that stands alone ("300.2006…", "1900") is
+        title = f"{num_title} {title}"
+    title = title or num_title or norm(name).replace("\x00", " ").strip()[:40]
+    return title, year or "?"
+
+
+# Bonus-material vocabulary: names that say nothing about the film, so an extras
+# file otherwise pairs with an extras file of another movie. "Short", "Spot" and
+# "Clip" are deliberately absent — they are film titles (Short Circuit, Spot,
+# Clips) — and the running-time guard in is_extra() carries the rest.
+SEP_CLS = r"[\s._-]+"
+
+
+def _bonus_rx(word):
+    return r"(?<![a-z0-9])" + re.sub(r"[ \-]+", lambda _: SEP_CLS, word.lower()) + r"(?![a-z0-9])"
+
+
+EXTRA_WORDS = [
+    "trailer", "teaser", "featurette", "featurettes", "promo", "promotional",
+    "tv spot", "advert", "commercial", "behind the scenes", "making of",
+    "deleted scene", "deleted scenes", "deleted footage", "gag reel", "blooper",
+    "blooper reel", "bloopers", "outtake", "outtakes", "scene selection", "menu",
+    "menus", "gallery", "still gallery", "stills", "poster", "posters",
+    "montage", "preview", "previews", "epilogue", "visual effects",
+    "music video", "singalong", "karaoke", "trivia", "storyboard", "workprint",
+    "test footage", "screen test", "audition", "cast and crew", "isolated score",
+    "picture in picture", "original score", "session piece", "sample",
+    "watermark",
+]
+# Words that describe bonus material which can run LONG (a feature-length
+# commentary-track cut, a documentary about the film). Trusted only below
+# LONG_BONUS_MAX, so a 40-minute "commentary" release stays a film.
+EXTRA_LONG_WORDS = ["interview", "interviews", "commentary", "video archive",
+                    "archival", "documentary"]
+EXTRA_NAME_RE = re.compile(
+    "|".join([_bonus_rx(w) for w in EXTRA_WORDS] + [r"scene[\s._-]?\d{1,3}"]), re.I)
+EXTRA_LONG_NAME_RE = re.compile(
+    "|".join(_bonus_rx(w) for w in EXTRA_LONG_WORDS), re.I)
+# Bonus-folder vocabulary. fullmatch = the folder IS an extras subtree; a bare
+# search = it only mentions extras ("... + Extras" box sets hold feature films).
+EXTRA_FOLDER_RE = re.compile(
+    r"(?:extras?|bonus(?:[\s._-]material)?|specials?|special[\s._-]featurettes?|"
+    r"featurettes?|deleted(?:[\s._-](?:scenes?|footage))?|trailers?|"
+    r"making[\s._-]of|behind[\s._-]the[\s._-]scenes|shorts?|interviews?|"
+    r"archiv(?:al|es)|video[\s._-]archive)", re.I)
+VIDEO_SUFFIX = (".mkv", ".mp4", ".avi", ".m4v", ".mov", ".m2ts", ".mpg", ".webm")
+
+# Anything longer than this is a feature film whatever it is called: a 118-minute
+# file named "The Menu" or "Trailer" is a movie, not bonus material.
+SHORT_MAX = 45 * 60
+
+# Long-shaped bonus vocabulary is only believed below this: a 40-minute
+# commentary cut is an alternate version of the film, not a featurette.
+LONG_BONUS_MAX = 25 * 60
+
+
+def name_words(x):
+    """Bare words of a file or folder name: no release metadata, no year."""
+    x = os.path.splitext(x)[0] if x.lower().endswith(VIDEO_SUFFIX) else x
+    return [w for w in SEP_RE.split(fold_sep(x))
+            if w and not YEAR_RE.match(w) and not RES_RE.match(w)
+            and not JUNK_RE.match(w) and w not in JUNK]
+
+
+def mentions(short, long_):
+    """True when one word list appears in the other, anywhere: a release names
+    the film at the front ("Alien (1979) - Trailer") or in the middle
+    ("Movies/Alien (1979)/Extra - Logo.mkv"), so a start-of-string rule fails."""
+    a, b = sorted([short, long_], key=len)
+    if not a:
+        return False
+    return any(b[i:i + len(a)] == a for i in range(len(b) - len(a) + 1))
+
+
+
+def folder_is_bonus(folder):
+    """True when a folder name carries bonus vocabulary as WHOLE words: the
+    vocabulary is matched against a padded word string, so "Extras" decides and
+    a word that merely CONTAINS one ("Extraordinary", "The Extras") does not."""
+    return bool(EXTRA_FOLDER_RE.search(" " + " ".join(SEP_RE.split(norm(folder))) + " "))
+
+
+def is_extra(path, title="", dur=0.0, root=ROOT, matched=False):
+    """True for bonus material, which must never be counted as a film: a file
+    called "Trailer", "Menu", "Sample" or "Gallery" carries no film identity, so
+    extras otherwise pair with each other across the whole library.
+
+    Evidence, strongest first:
+      * RUNTIME VETOES EVERYTHING. The film "The Menu", the film "Trailer", the
+        franchise "Trailer Park Boys" and "ET — The Extra-Terrestrial" all carry
+        bonus vocabulary and run 100+ minutes: anything over SHORT_MAX is a film —
+        unless it sits in an explicit bonus folder and metadata never matched it.
+        Bonus words that describe long-form material (commentary, documentary,
+        interview) are believed only
+        under LONG_BONUS_MAX, so a feature-length commentary cut stays a film.
+      * The file's OWN name carries bonus vocabulary -> bonus.
+      * ANY folder of the path below the film's own root folder carries bonus
+        vocabulary ("Alien (1979)/Extras/", ".../Featurettes/Deleted Scenes/",
+        "Angel's Egg (1985)/04b. Angel's Egg Extras/") -> bonus. The ROOT folder
+        is exempt because it names the film, and a film may be titled with a
+        bonus word; and a file whose own name IS that folder's film name is the
+        feature, not the bonus ("Ghost in the Shell Movies/Extras/
+        Ghost in the Shell - CM3.mp4" is a bonus, but a whole other film filed
+        there stays a film).
+    Bonus vocabulary is matched as whole words only, so the film
+    "Angel's Egg - Extra - Logo" inside "04b. Angel's Egg Extras" is reached by
+    the folder rule, and "…Extras" as a fragment of a word is never evidence.
+    """
+    parts = [q for q in path.split("/") if q]
+    if not parts:
+        return False
+    name = (os.path.splitext(parts[-1])[0]
+            if parts[-1].lower().endswith(VIDEO_SUFFIX) else "")
+    # A bonus folder anywhere BELOW the film's own folder decides for its
+    # contents; the film's own folder is the film's name, whatever words it
+    # contains, so it is exempt. It is the first component AFTER ROOT's own
+    # components — those are the mount path, not folders of the library.
+    head = len([q for q in root.split("/") if q]) + 1
+    bonus_folder = False
+    for folder in parts[head:-1]:
+        if folder_is_bonus(folder):
+            bonus_folder = not mentions(name_words(name), name_words(folder))
+            break
+    if dur and dur > SHORT_MAX:
+        # Runtime vetoes vocabulary, with ONE exception: an explicit bonus folder
+        # and no metadata match. A 90-minute "Feature Length Rough Cut" sitting in
+        # a movie's Featurettes/ is a disc extra, and nothing in the library claims
+        # it as a film. Long-form bonus words still describe a film, so a
+        # feature-length commentary cut or documentary keeps its runtime veto.
+        if bonus_folder and not matched and not EXTRA_LONG_NAME_RE.search(name):
+            return True
+        return False
+    if EXTRA_NAME_RE.search(name):
+        return True
+    if dur and dur <= LONG_BONUS_MAX and EXTRA_LONG_NAME_RE.search(name):
+        return True
+    return bonus_folder
+
+# Release-group watermark clips: a group drops a tiny ad ("ETRG.mp4",
+# "RARBG.com.mp4") into many different films, so its name carries no film
+# identity at all and it would otherwise pair with itself across the library.
+WATERMARK_MAX = 20 * 2**20
+WATERMARK_RE = re.compile(r"(?i)^(rarbg(\.com)?|etrg|tgx|fbt|cpbg|yify|yts|"
+                          r"rarbg\.com|playnomore|imprs|gauntlet|dfx|macmote|"
+                          r"psi-axiz|acm|tlf|sa8|ebp|gtv|w4g|don|aAF|enjoy|hd4u)$")
+
+
+def is_watermark(path, size):
+    """True for a release group's own watermark video, not a film or an extra."""
+    return (size < WATERMARK_MAX
+            and bool(WATERMARK_RE.match(os.path.splitext(os.path.basename(path))[0])))
+
+
+
+def key_of(title):
+    """Franchise key: first two alphabetic tokens (numeric-only title = itself).
+    Deliberately NOT film-specific — it only selects candidates for the duration
+    check, so a two-token collision (two films starting "star trek") cannot by
+    itself create a duplicate group."""
+    toks = [t for t in title.split() if t]
+    words = [t for t in toks if not t.isdigit()]
+    if not words:
+        return " ".join(toks)
+    sig = [t for t in words if t not in STOP] or words
+    return " ".join(sig[:2])
+
+
+def ep_of(path):
+    """(series, episode-token) when the basename carries SxxEyy / 1x04 / Part N.
+
+    The series is the name BEFORE the marker. The box-set convention writes
+    "Monk - S05E03 - Mr. Monk and the Big Game", and the episode title that
+    follows the marker is not the show's name: keying on the whole title gave
+    every episode of one series its own group, so a 124-episode box set never
+    reached the size that calls something a series and each episode was filed
+    as its own film. A file named only by its marker has nothing before it, so
+    its whole title stands in."""
+    name = os.path.splitext(os.path.basename(path))[0]
+    m = EP_RE.search(name)
+    if not m:
+        return None
+    head = name[:m.start()].strip(" ._\u2013-")
+    series = title_of(head)[0] if head else None
+    return series or title_of(path)[0], norm(m.group(1)).replace(" ", "")
+
+
+def base_words(title, key):
+    """Title tokens beyond the franchise key — the FILM identity part.
+    "evil dead rise" under key "evil dead" -> {rise}; "28 days later" -> {}."""
+    kt = set(key.split())
+    return frozenset(t for t in title.split() if t not in kt and t not in STOP
+                     and not t.isdigit() and not YEAR_RE.match(t))
+
+
+def folder_year(path):
+    """Soft year evidence: a 19xx/20xx token in any parent directory name
+    (collection folders are written both "(2002)" and "[2009]"; a range such as
+    "(1998-2000)" deliberately yields nothing)."""
+    for d in reversed(os.path.dirname(path).split("/")):
+        ys = [y for y in re.split(r"[^0-9]+", d) if YEAR_RE.match(y)]
+        if len(ys) == 1:
+            return ys[0]
+    return None
+
+
+def same_film(a, b):
+    """Two files are copies of ONE film only when the film-identity tokens agree
+    (one side may carry none) AND no year evidence contradicts them. The key says
+    the franchise matches; this says the actual film matches — it is what keeps
+    'Evil Dead' (2013) apart from 'Evil Dead Rise' (2023) and the four Austin
+    Powers apart when their runtimes happen to agree."""
+    ba, bb = a["base"], b["base"]
+    if ba and bb and ba != bb:
+        return False
+    ya = a["year"] if a["year"] != "?" else a["soft"]
+    yb = b["year"] if b["year"] != "?" else b["soft"]
+    if ya and yb and ya != yb:
+        return False
+    return True
+
+
+def dur_close(a, b):
+    if a <= 0 or b <= 0:
+        return True                                   # unknown duration: let titles decide
+    return abs(a - b) <= max(2.0, 0.003 * max(a, b))
+
+
+# Blu-ray folder trees: ffprobe only sees the clip it is pointed at, so each tree
+# is folded into ONE entry, described by its longest main-feature clip (measured
+# with ffprobe on the largest BDMV/STREAM file) and sized by the whole tree.
+BDMV_MAIN = {
+    "Avengers.Age.of.Ultron.2015.2160p.BluRay.HEVC.TrueHD.7.1.Atmos-TERMiNAL":
+        dict(dur=8478.2, vcodec="hevc", res="3840x2160", pix="yuv420p10le", fps=23.976),
+    "Full.Metal.Jacket.1987.2160p.BluRay.HEVC.DTS-HD.MA.5.1-EATDIK":
+        dict(dur=6993.3, vcodec="hevc", res="3840x2160", pix="yuv420p10le", fps=23.976),
+}
+
+
+def load(probed):
+    """Probe rows -> film entries. AppleDouble (._*) sidecars are dropped; each
+    BDMV tree collapses to one entry carrying the tree's total size."""
+    trees = {}
+    rows = []
+    for line in open(probed):
+        f = line.rstrip("\n").split("\t")
+        if len(f) < 12:
+            continue
+        b, p, dur, vc, res, fps, vbit, pix, aud, subs, _d, fmt = f[:12]
+        base = os.path.basename(p)
+        if base.startswith("._") or base == ".DS_Store":
+            continue                                    # macOS sidecar, not media
+        m = re.search(r"/(BDMV|VIDEO_TS)/", p)
+        root = p[:m.start()] if m else None
+        if root or re.search(r"/(CERTIFICATE|AACS|MLPI|CLIPINF|PLAYLIST|BACKUP|REGISTRL)/", p):
+            if not root:
+                continue      # disc bookkeeping: folds into its tree, never a film
+        if root:
+            t = trees.setdefault(root, dict(bytes=0, audio=set(), subs=0))
+            t["bytes"] += int(b)
+            t["audio"].update(a for a in aud.split("+") if a)
+            t["subs"] += int(subs or 0)
+            continue
+        try:
+            dur = float(dur)
+        except ValueError:
+            dur = 0.0
+        rows.append(dict(bytes=int(b), path=p, dur=dur, vcodec=vc, res=res, fps=fps,
+                         vbit=int(float(vbit or 0)), pix=pix, audio=aud, subs=subs, fmt=fmt))
+    for root, t in trees.items():
+        folder = os.path.basename(root)
+        m = BDMV_MAIN.get(folder, {})
+        rows.append(dict(bytes=t["bytes"], path=root + "/BDMV/index.bdmv",
+                         name_src=folder, dur=m.get("dur", 0.0),
+                         vcodec=m.get("vcodec", "hevc"), res=m.get("res", "-"),
+                         fps=m.get("fps", 0), vbit=0, pix=m.get("pix", "-"),
+                         audio="+".join(sorted(t["audio"])), subs=str(t["subs"]),
+                         fmt="BDMV tree"))
+    for r in rows:
+        r["title"], r["year"] = title_of(r.get("name_src") or r["path"])
+        r["key"] = key_of(r["title"])
+        r["base"] = base_words(r["title"], r["key"])
+        r["soft"] = folder_year(r["path"])
+        r["ep"] = ep_of(r["path"])
+    return rows
+
+
+def dur_clusters(items):
+    """Group files that are the SAME FILM and the SAME MASTER duration. Candidate
+    buckets come from the franchise key; same_film then rejects the different
+    installments whose runtimes happen to agree."""
+    clusters = []
+    for r in sorted(items, key=lambda r: -r["dur"]):
+        for c in clusters:
+            if dur_close(c[0]["dur"], r["dur"]) and same_film(c[0], r):
+                c.append(r)
+                break
+        else:
+            clusters.append([r])
+    return clusters
+
+
+def tier(groups):
+    """Split same-film/same-duration clusters: DUP = different releases of one
+    film (real, deletable redundancy); REVIEW = identical size AND duration
+    (byte-twins — the hash pass is the authority) or clashing year evidence."""
+    dup, review = [], []
+    for g in groups:
+        g.sort(key=lambda r: -r["bytes"])
+        years = {(r["year"] if r["year"] != "?" else r["soft"]) for r in g}
+        years.discard(None)
+        spread = (g[0]["bytes"] - g[-1]["bytes"]) / g[0]["bytes"] if g[0]["bytes"] else 0
+        if len(years) > 1 or spread < 0.005:
+            review.append(g)
+        else:
+            dup.append(g)
+    return dup, review
+
+
+def redun(g):
+    return sum(r["bytes"] for r in sorted(g, key=lambda r: -r["bytes"])[1:])
+
+
+def line(r):
+    return (f"{r['bytes']/2**30:8.2f} GiB {r['res']:>9} {r['vcodec']:<7} "
+            f"{r['pix']:<11} {r['dur']/60:5.0f}m {r['audio'][:34]:<34}")
+
+
+def dump(tag, groups, out, note=""):
+    for gi, g in enumerate(groups, 1):
+        g.sort(key=lambda r: -r["bytes"])
+        print(f"\n### {tag}-{gi}. {g[0]['title']} ({g[0]['year']}) — {len(g)} files, "
+              f"{redun(g)/2**30:.1f} GiB redundant {note}".rstrip())
+        for rank, r in enumerate(g):
+            print(f"  {line(r)} [{r['year']}] "
+                  f"{r['path'].replace(ROOT + '/', '')}")
+            out.write(f"{tag}\t{gi}\t{rank}\t{r['bytes']}\t{r['res']}\t{r['vcodec']}\t"
+                      f"{r['pix']}\t{int(r['dur'])}\t{r['audio']}\t{r['title']}|{r['year']}\t"
+                      f"{r['path']}\n")
+
+
+def report(probed, out_path):
+    rows = load(probed)
+    for r in rows:
+        r["mark"] = is_watermark(r["path"], r["bytes"])
+        r["extra"] = not r["mark"] and is_extra(r["path"], r["title"], r["dur"])
+    marks = [r for r in rows if r["mark"]]
+    movies = [r for r in rows if not r["ep"] and not r["extra"] and not r["mark"]]
+    episodes = [r for r in rows if r["ep"] and not r["extra"] and not r["mark"]]
+    extras = [r for r in rows if r["extra"]]
+    print(f"census: {len(rows)} video entries = {len(movies)} films + {len(episodes)} "
+          f"TV episodes + {len(extras)} bonus/trailers "
+          f"({sum(r['bytes'] for r in extras)/2**30:.1f} GiB) "
+          f"+ {len(marks)} release-group watermarks")
+
+    out = open(out_path, "w")
+
+    buckets = collections.defaultdict(list)
+    for r in movies:
+        buckets[r["key"]].append(r)
+    cand = {k: v for k, v in buckets.items() if len(v) > 1}
+    clusters = [c for v in cand.values() for c in dur_clusters(v) if len(c) > 1]
+    dup, review = tier(clusters)
+    dup.sort(key=lambda g: -redun(g))
+    review.sort(key=lambda g: -redun(g))
+    dump("DUP", dup, out)
+    dump("REVIEW", review, out, "— same key+duration, sizes near-identical OR years clash: RECHECK")
+
+
+    ebuf = collections.defaultdict(list)
+    for r in episodes:
+        ebuf[(r["ep"][0], r["ep"][1])].append(r)
+    edup, ereview = tier([c for v in ebuf.values() if len(v) > 1
+                          for c in dur_clusters(v) if len(c) > 1])
+    edup.sort(key=lambda g: -redun(g))
+    dump("EP-DUP", edup, out)
+
+    shows = collections.Counter()
+    for r in episodes:
+        shows[r["ep"][0]] += 1
+    multi = {s: n for s, n in shows.items() if n > 4}
+    print("\n=== TV-series content inside Movies/ (>4 episodes) ===")
+    for s, n in sorted(multi.items(), key=lambda kv: -kv[1]):
+        print(f"  {n:4d} episodes  {s}")
+    out.close()
+
+    print(f"\nDUP groups={len(dup)} files={sum(len(g) for g in dup)} "
+          f"redundant={sum(redun(g) for g in dup)/2**40:.2f} TiB", file=sys.stderr)
+    print(f"REVIEW groups={len(review)} "
+          f"redundant={sum(redun(g) for g in review)/2**40:.2f} TiB", file=sys.stderr)
+    print(f"EP-DUP groups={len(edup)} "
+          f"ep-redundant={sum(redun(g) for g in edup)/2**30:.1f} GiB", file=sys.stderr)
+
+
+def parse(names):
+    for n in names:
+        t, y = title_of(n)
+        print(f"{t!r} ({y})\tkey={key_of(t)!r}\t{n[:70]}")
+
+
+def main():
+    ap = argparse.ArgumentParser(description=__doc__.splitlines()[0])
+    sub = ap.add_subparsers(dest="cmd", required=True)
+    rep = sub.add_parser("report", help="similar-copy census over a probe TSV")
+    rep.add_argument("--probed", default="/tmp/media-probed.tsv")
+    rep.add_argument("--out", default="/tmp/media-similar.tsv")
+    par = sub.add_parser("parse", help="show the parsed title/year of a name")
+    par.add_argument("names", nargs="+")
+    a = ap.parse_args()
+    if a.cmd == "report":
+        report(a.probed, a.out)
+    else:
+        parse(a.names)
+
+
+if __name__ == "__main__":
+    main()

scripts/media-tv.py

@@ -0,0 +1,281 @@
+#!/usr/bin/env python3
+"""media-tv.py — move TV episodes out of the movie library into a TV library.
+
+`scripts/media-plan.py` refuses to file TV: it parks episode files in a
+`_TV needs a TV library - <series>` pile for a human. This is the other half of
+that refusal. It walks the movie library, groups the files whose own name
+carries an episode marker by SERIES, and writes a plan that places each series
+in the shape both Jellyfin and Plex read without help:
+
+    <library>/<Series> (<year>)/Season <NN>/<the file's own name>.<ext>
+
+It writes a plan and nothing else. Every rename belongs to
+`scripts/media-apply.py` — the write-ahead journal, the refusal to overwrite,
+the all-or-nothing preflight, the rollback. Apply it with `--root` set to the
+share that holds BOTH libraries (`/mnt/entropy`, not `/mnt/entropy/Movies`) so
+the executor's own path guard still has teeth, and give it its own journal —
+one journal is one undo, and a TV move must not be welded to a movie relayout:
+
+    python3 scripts/media-tv.py --root /mnt/entropy/Movies \
+        --library /mnt/entropy/TVShows --out /mnt/entropy/media-tv-plan.tsv
+    python3 scripts/media-apply.py apply --plan /mnt/entropy/media-tv-plan.tsv \
+        --root /mnt/entropy --journal /mnt/entropy/media-relayout-tv.jsonl
+    python3 scripts/media-apply.py verify --plan /mnt/entropy/media-tv-plan.tsv \
+        --root /mnt/entropy --journal /mnt/entropy/media-relayout-tv.jsonl
+    python3 scripts/media-apply.py sweep --root /mnt/entropy \
+        --journal /mnt/entropy/media-relayout-tv.jsonl --apply
+
+Guards, and why each one is here:
+
+* The same five-file threshold as the planner. An episode-shaped name on one
+  file is not a series — `The Godfather Part 2` is a film — so a smaller group
+  is reported and left exactly where it is.
+* An episode whose name carries no season number is reported and left, never
+  filed into an invented `Season 01`.
+* A sidecar that shares an episode's own name (`.srt`, `.nfo`, …) follows the
+  episode; an episode without its subtitles is a half-move.
+* A group that would claim one destination twice is not resolved here. The
+  executor refuses the whole plan before its first rename, which is the right
+  outcome: nobody but Frank decides which copy of an episode is the keeper.
+* An episode the destination library already holds at the same byte count is
+  left where it is. The TV library has held a complete Monk box set for years
+  while the movie library carried a second copy of all 124 episodes; a mover
+  that did not look where it was filing would have put both inside one library.
+* A symlink is never followed and never moved.
+"""
+import argparse
+import collections
+import importlib.util
+import os
+import re
+
+
+def parse_args():
+    ap = argparse.ArgumentParser(description=__doc__.splitlines()[0])
+    ap.add_argument("--root", default=os.environ.get("MEDIA_ROOT", "/mnt/entropy/Movies"),
+                    help="the movie library to empty of TV")
+    ap.add_argument("--library", default="/mnt/entropy/TVShows",
+                    help="the TV library to fill (a different share or folder)")
+    ap.add_argument("--out", default=None,
+                    help="plan TSV for scripts/media-apply.py"
+                         " (default: media-tv-plan.tsv beside --root)")
+    ap.add_argument("--min", type=int, default=5,
+                    help="episode files that make something a series")
+    a = ap.parse_args()
+    if not a.out:
+        a.out = os.path.join(os.path.dirname(os.path.abspath(a.root)),
+                             "media-tv-plan.tsv")
+    return a
+
+
+ARGS = parse_args()
+ENGINE = os.path.join(os.path.dirname(os.path.abspath(__file__)), "media-titles.py")
+spec = importlib.util.spec_from_file_location("eng", ENGINE)
+eng = importlib.util.module_from_spec(spec)
+spec.loader.exec_module(eng)
+
+BAD_CHARS = re.compile(r'[<>:"/|?*]')
+# The season and episode, from the token media-titles.ep_of() hands back: s05e03, 1x04.
+SEASON_RE = re.compile(r"^(?:s(\d{1,2})e(\d{1,4})|(\d{1,2})x(\d{1,4}))$")
+# Sidecars that follow an episode; macOS forks and resume data do not.
+SKIP_SIDECAR = re.compile(r"^(\._|\.DS_|dl_state)")
+# A box set that stamps the show's year into every file name; the folder's year
+# must not be added on top of it.
+YEAR_TAIL = re.compile(r"[(\[]\s*(18|19|20)\d\d\s*[)\]]\s*$")
+SMALL = {"a", "an", "the", "and", "of", "for", "in", "on", "to", "with", "at",
+         "by", "de", "la", "vs"}
+
+
+def clean(name):
+    """A name the filesystem and both matchers can read (same law as the plan)."""
+    name = BAD_CHARS.sub("-", name).replace("\\", "-")
+    name = re.sub(r"\s+", " ", name).strip().strip(". ").strip()
+    return name[:180] or "untitled"
+
+
+def episode_name(path):
+    """The file's own name, kept: the publisher's spelling beats a re-derivation.
+
+    A dot-separated release name (`monk.s05e03.mkv`) becomes words; a name that
+    already carries spaces is left exactly as it was written."""
+    stem, ext = os.path.splitext(os.path.basename(path))
+    if " " not in stem and stem.count(".") >= 2:
+        stem = stem.replace(".", " ")
+    return clean(stem) + ext.lower()
+
+
+def head_of(path):
+    """The publisher's series name: the text BEFORE the episode marker.
+
+    `Monk - S05E03 - Mr. Monk and the Big Game` is `Monk`; the episode title
+    after the marker is not the show's name, and case is kept because it is the
+    library's own spelling, not a normalised key."""
+    name = os.path.splitext(os.path.basename(path))[0]
+    m = eng.EP_RE.search(name)
+    head = name[:m.start()] if m else name
+    return re.sub(r"\s+", " ", re.sub(r"[._]+", " ", head)).strip(" -\u2013").strip()
+
+
+def title_series(head):
+    """The series folder in the case a TV library is written in.
+
+    Only a name with no capital at all is retitled: a box set that was
+    lower-cased on the way in would otherwise file `monk (2002)` among
+    `Dexter's Laboratory (1996)`. A name the publisher wrote with case is left
+    exactly as it was written."""
+    if any(c.isupper() for c in head):
+        return head            # the publisher wrote it with case; keep it
+    words = head.split()
+    return " ".join(w if i and w.lower() in SMALL else w.capitalize()
+                    for i, w in enumerate(words))
+
+
+def season_episode(token):
+    """(season, episode) an episode token states, or None when it states none."""
+    m = SEASON_RE.match(token)
+    if not m:
+        return None
+    return int(m.group(1) or m.group(3)), int(m.group(2) or m.group(4))
+
+
+def sidecars(path):
+    """The files beside PATH that carry its exact stem — its subtitles, its nfo."""
+    d, stem = os.path.split(path)
+    stem = os.path.splitext(stem)[0]
+    out = []
+    try:
+        entries = sorted(os.listdir(d))
+    except OSError:
+        return out
+    for e in entries:
+        p = os.path.join(d, e)
+        if p == path or os.path.islink(p) or not os.path.isfile(p):
+            continue
+        s, _ext = os.path.splitext(e)
+        if s == stem and not SKIP_SIDECAR.match(e):
+            out.append(p)
+    return out
+
+
+def episodes_under(root):
+    """(path, series, token) for every episode-shaped video file under root."""
+    for dirpath, dirnames, filenames in os.walk(root):
+        dirnames[:] = sorted(d for d in dirnames
+                             if not d.startswith(".") and d != "@eaDir")
+        for f in filenames:
+            p = os.path.join(dirpath, f)
+            if os.path.islink(p) or not f.lower().endswith(eng.VIDEO_SUFFIX):
+                continue
+            e = eng.ep_of(p)
+            if e:
+                yield p, e[0], e[1]
+
+
+def walk(root, library):
+    """Episode files under ROOT keyed by series, ignoring the destination."""
+    groups = collections.defaultdict(list)
+    lib = os.path.abspath(library)
+    for p, series, token in episodes_under(root):
+        if os.path.abspath(p).startswith(lib + os.sep):
+            continue
+        groups[series].append((p, token))
+    return groups
+
+
+def holdings(library):
+    """What the destination already carries: (series, season, episode) -> sizes.
+
+    The one expensive read here, and the one that matters. The movie library
+    holds a complete Monk box set that the TV library has held for years: 124
+    episodes, the same episode list, identical byte counts. Moving it would
+    have filed a second copy of a whole show into the library that already had
+    it. Byte count is the test on purpose — two encodes of one episode are two
+    files and both are wanted; the same bytes are the same file."""
+    out = collections.defaultdict(set)
+    for p, series, token in episodes_under(library):
+        se = season_episode(token)
+        if se is None:
+            continue
+        try:
+            out[(series, *se)].add(os.path.getsize(p))
+        except OSError:
+            continue
+    return out
+
+
+def main():
+    root, library = os.path.abspath(ARGS.root), os.path.abspath(ARGS.library)
+    groups = walk(root, library)
+    held_here = holdings(library)
+    mkdirs = {library}
+    rows, lines, skipped = set(), [], []
+    for series, items in sorted(groups.items()):
+        if len(items) < ARGS.min:
+            skipped.append(f"skipped: {head_of(items[0][0]) or series} — {len(items)} "
+                           f"episode-shaped file{'s' if len(items) != 1 else ''},"
+                           f" below {ARGS.min}: not a series")
+            continue
+        years = collections.Counter(y for y in (eng.folder_year(p) for p, _t in items)
+                                    if y)
+        year = years.most_common(1)[0][0] if years else None
+        heads = collections.Counter(head_of(p) for p, _t in items)
+        head = heads.most_common(1)[0][0]
+        if year and YEAR_TAIL.search(head):
+            year = None
+        name = clean(f"{title_series(head)}{f' ({year})' if year else ''}")
+        series_dir = os.path.join(library, name)
+        moved = held = 0
+        for path, token in items:
+            se = season_episode(token)
+            if se is None:
+                skipped.append(f"skipped: {path} — no season number in its name,"
+                               " nothing here invents one")
+                continue
+            try:
+                size = os.path.getsize(path)
+            except OSError:
+                continue
+            if size in held_here.get((series, *se), ()):
+                held += 1
+                continue
+            dst_dir = os.path.join(series_dir, f"Season {se[0]:02d}")
+            mkdirs.update([series_dir, dst_dir])
+            base = episode_name(path)
+            rows.add((path, os.path.join(dst_dir, base)))
+            moved += 1
+            for side in sidecars(path):
+                rows.add((side, os.path.join(
+                    dst_dir, clean(os.path.splitext(base)[0]
+                                   + os.path.splitext(side)[1].lower()))))
+        if moved:
+            lines.append(f"{name}: {moved} episodes"
+                         f"{'' if year else ' (no year in any folder name)'}"
+                         f" -> {series_dir}")
+        if held:
+            lines.append(f"{name}: {held}"
+                         f" {'episode' if held == 1 else 'episodes'} already in"
+                         f" {library} at the same byte count — left there")
+
+    # An empty plan creates nothing, not even the library folder: a run that
+    # found no series must not be able to leave a stray directory behind.
+    dirs = sorted(mkdirs, key=lambda p: (p.count(os.sep), p)) if rows else []
+    with open(ARGS.out, "w") as out:
+        out.write("action\tsrc\tdst\tkind\n")
+        for d in dirs:
+            out.write(f"mkdir\t\t{d}\tdir\n")
+        for src, dst in sorted(rows):
+            out.write(f"move\t{src}\t{dst}\ttv\n")
+
+    for line in lines + skipped:
+        print(line)
+    print(f"plan: {len(rows)} moves, {len(dirs)} folders -> {ARGS.out}")
+    if rows:
+        print(f"next: python3 scripts/media-apply.py apply --plan {ARGS.out}"
+              f" --root {os.path.dirname(root)}"
+              f" --journal {os.path.join(os.path.dirname(root),'media-relayout-tv.jsonl')}"
+              "   (add --apply to run it)")
+    return 0
+
+
+if __name__ == "__main__":
+    raise SystemExit(main())

scripts/media-view.sh

@@ -0,0 +1,241 @@
+#!/usr/bin/env bash
+# The SSD-backed metadata view that lets Jellyfin and Plex share one metadata set.
+#
+# Both servers read a Kodi-format sidecar (movie.nfo / tvshow.nfo / episode.nfo)
+# ONLY when it sits next to the media file. The media lives on the RAID6 array at
+# /mnt/entropy — five spinning disks, deliberately mounted read-only for every
+# container, and slow to seek for a thousand tiny XML reads. So neither media
+# server binds the array any more: they bind an OverlayFS merge of it with a
+# writable layer on the NVMe.
+#
+#   lowerdir  /mnt/entropy                RAID6 (ROTA=1) — source of truth, never written
+#   upperdir  /var/lib/media-meta/upper   NVMe — every sidecar and image byte
+#   workdir   /var/lib/media-meta/work    NVMe — kernel bookkeeping
+#   Where     /var/lib/media-meta/view    the merged tree jellyfin and plex bind
+#
+# Everything the view needs — layers, bookkeeping, trash, the merged door — lives
+# under the one directory /var/lib/media-meta, so nothing new appears under /mnt.
+#
+# Consequences worth knowing:
+#   * A sidecar write is a few KB onto the SSD; the media bytes never move and the
+#     array is still never written by anything automated.
+#   * Deleting through the view writes a whiteout into the upper layer instead of
+#     deleting from the array. Nothing should delete through the view: deletions
+#     go to .AGENTTRASH on the RAW path, and `reap` clears the resulting ghosts.
+#   * inotify events on the lower layer do not cross the merge, so a file that
+#     arrives on the raw array is invisible to a file-system watcher in the view:
+#     trigger a library refresh (see the Media Relayout page).
+#
+# Subcommands
+#   plan   [--upper-base D] [--lower D]  print the resolved shape, touch nothing
+#   verify                               prove the live mount IS that shape
+#   install [--upper-base D]             create the layers, install+enable+start the unit
+#   reap   [--apply] [--upper-base D] [--lower D]
+#                                       list upper-layer entries whose media is gone
+#   status                               one line per fact a human wants
+#
+# install must run as root (it writes /etc/systemd/system and mounts); the others
+# are read-only and safe unprivileged.
+set -euo pipefail
+
+LOWER=/mnt/entropy
+UPPER_BASE=/var/lib/media-meta
+VIEW=$UPPER_BASE/view
+# systemd names a mount unit after its ESCAPED path ('/' -> '-', a literal '-'
+# -> '\x2d'), so the unit on the host is var-lib-media\x2dmeta-view.mount. The
+# repo copy keeps a readable name; install writes it under the escaped one.
+UNIT='var-lib-media\x2dmeta-view.mount'
+UNIT_DIR=/etc/systemd/system
+# The rootless containers write as this host uid (the deploy user). The upper
+# layer has to accept those writes, so it is owned by them.
+MEDIA_UID="${MEDIA_VIEW_UID:-1000}"
+MEDIA_GID="${MEDIA_VIEW_GID:-1000}"
+
+REPO_ROOT="$(cd "$(dirname "$0")/.." && pwd)"
+UNIT_SRC="$REPO_ROOT/dankserver/host/systemd/media-meta-view.mount"
+
+die() {
+    printf 'media-view: %s\n' "$*" >&2
+    exit 1
+}
+
+usage() {
+    printf 'usage: %s plan|verify|install|reap|status [options]\n' "$0" >&2
+    exit 2
+}
+
+cmd="${1:-}"
+[ -n "$cmd" ] || usage
+shift || true
+
+upper_base="$UPPER_BASE"
+lower="$LOWER"
+apply=0
+while [ $# -gt 0 ]; do
+    case "$1" in
+        --upper-base) [ $# -ge 2 ] || usage; upper_base="$2"; shift 2 ;;
+        --lower)      [ $# -ge 2 ] || usage; lower="$2"; shift 2 ;;
+        --apply)      apply=1; shift ;;
+        *) usage ;;
+    esac
+done
+
+upper="$upper_base/upper"
+work="$upper_base/work"
+
+# The device backing a path, and whether the kernel calls it rotational.
+rotational_of() {
+    local src rota
+    src="$(findmnt -no SOURCE --target "$1" 2>/dev/null || true)"
+    [ -n "$src" ] || die "cannot find a block device behind $1"
+    rota="$(lsblk -no ROTA "$src" 2>/dev/null | head -1 || true)"
+    [ -n "$rota" ] || die "cannot read the rotational flag of $src"
+    printf '%s %s\n' "$src" "$rota"
+}
+
+require_upper_base() {
+    [ -d "$upper_base" ] \
+        || die "refused: upper base $upper_base does not exist (put it on the SSD first)"
+}
+
+do_plan() {
+    printf 'lowerdir=%s\nupperdir=%s\nworkdir=%s\nmountpoint=%s\nunit=%s/%s\n' \
+        "$lower" "$upper" "$work" "$VIEW" "$UNIT_DIR" "$UNIT"
+}
+
+do_verify() {
+    local fail=0 fst opts src rota
+    if ! mountpoint -q "$VIEW"; then
+        printf 'refused: %s is not a mountpoint — a plain directory here is a\n' "$VIEW" >&2
+        printf '  library that scans itself empty, not a metadata view.\n' >&2
+        return 1
+    fi
+    fst="$(findmnt -no FSTYPE --mountpoint "$VIEW" 2>/dev/null || true)"
+    if [ "$fst" != overlay ]; then
+        printf 'refused: %s is mounted %s, not overlay\n' "$VIEW" "${fst:-nothing}" >&2
+        fail=1
+    fi
+    opts="$(findmnt -no OPTIONS --mountpoint "$VIEW" 2>/dev/null || true)"
+    local want
+    for want in "lowerdir=$LOWER" "upperdir=$UPPER_BASE/upper" "workdir=$UPPER_BASE/work"; do
+        case ",$opts," in
+            *",${want},"*) ;;
+            *) printf 'refused: %s options lack %s (got: %s)\n' "$VIEW" "$want" "$opts" >&2
+               fail=1 ;;
+        esac
+    done
+    read -r src rota <<<"$(rotational_of "$UPPER_BASE")"
+    if [ "$rota" != 0 ]; then
+        printf 'refused: the upper layer sits on %s which is rotational — metadata\n' "$src" >&2
+        printf '  belongs on the SSD, not on the array.\n' >&2
+        fail=1
+    fi
+    if [ ! -w "$UPPER_BASE/upper" ]; then
+        printf 'refused: %s is not writable\n' "$upper" >&2
+        fail=1
+    fi
+    [ "$fail" -eq 0 ] || return 1
+    printf 'view ok: %s <- overlay(%s,%s)\n' "$VIEW" "$LOWER" "$UPPER_BASE/upper"
+}
+
+do_install() {
+    [ "$(id -u)" -eq 0 ] || die "install must run as root (writes $UNIT_DIR and mounts)"
+    require_upper_base
+    mountpoint -q "$LOWER" \
+        || die "refused: $LOWER is not mounted; overlaying an empty directory would present an empty library"
+    local src rota
+    read -r src rota <<<"$(rotational_of "$upper_base")"
+    [ "$rota" = 0 ] || die "refused: $upper_base is on $src, a rotational device — the metadata layer must be on the SSD"
+    [ -f "$UNIT_SRC" ] || die "refused: $UNIT_SRC is missing from $REPO_ROOT"
+    # systemd will not create a mount point; a missing directory fails the start.
+    install -d -m 0755 -o "$MEDIA_UID" -g "$MEDIA_GID" "$upper" "$work" "$VIEW"
+    install -d -m 0700 "$upper_base/.AGENTTRASH"
+    install -m 0644 "$UNIT_SRC" "$UNIT_DIR/$UNIT"
+    systemctl daemon-reload
+    systemctl enable "$UNIT" >/dev/null 2>&1 || true
+    systemctl start "$UNIT"
+    do_verify
+}
+
+# An upper-layer DIRECTORY with no counterpart below is metadata for a media
+# folder that has been moved or renamed on the RAW path — a ghost the media
+# servers would still browse. Files are never candidates on their own: a sidecar
+# (movie.nfo, an .srt) exists ONLY above, by design, and is exactly what the view
+# is for. A copy-up whose lower file was deleted by hand is the one case this
+# leaves in place; the raw path is where deletions belong, so it stays visible.
+do_reap() {
+    [ -d "$upper" ] || die "refused: $upper does not exist (no upper layer at $upper_base)"
+    stamp="$upper_base/.AGENTTRASH/$(date -u +%Y%m%d-%H%M%S)"
+    count=0
+    ghosts=0
+    reap_walk ""
+    if [ "$count" -eq 0 ] && [ "$ghosts" -eq 0 ]; then
+        printf 'reap: no ghosts under %s\n' "$upper"
+        return 0
+    fi
+    if [ "$ghosts" -gt 0 ]; then
+        printf 'reap: %d whiteout%s above — someone deleted THROUGH the view.\n' \
+            "$ghosts" "$([ "$ghosts" -eq 1 ] || printf s)" >&2
+        printf '  Leaving them: removing a whiteout un-hides the file it hides.\n' >&2
+        printf '  Deletions belong in .AGENTTRASH on the raw path.\n' >&2
+    fi
+    if [ "$count" -eq 0 ]; then return 0; fi
+    if [ "$apply" -eq 1 ]; then
+        printf 'reap: moved %d entr%s to %s\n' "$count" \
+            "$([ "$count" -eq 1 ] && printf y || printf ies)" "$stamp"
+    else
+        printf 'reap: %d ghost entr%s above; re-run with --apply to move them to %s\n' \
+            "$count" "$([ "$count" -eq 1 ] && printf y || printf ies)" "$stamp"
+    fi
+}
+
+# Descend only where the lower layer still has the directory: a directory that is
+# missing below is itself the ghost, and its whole subtree goes with it.
+reap_walk() {
+    local rel="$1" entries abs r name
+    entries="$(find "$upper${rel:+/$rel}" -mindepth 1 -maxdepth 1 -print 2>/dev/null || true)"
+    [ -n "$entries" ] || return 0
+    while IFS= read -r abs; do
+        r="${abs#"$upper"/}"
+        name="${abs##*/}"
+        if [ -c "$abs" ] || printf '%s' "$name" | grep -q '^\.wh\.'; then
+            ghosts=$((ghosts + 1))
+            continue
+        fi
+        if [ -e "$lower/$r" ]; then
+            [ -d "$abs" ] && reap_walk "$r"
+            continue
+        fi
+        [ -d "$abs" ] || continue
+        count=$((count + 1))
+        printf '%s\n' "$r"
+        if [ "$apply" -eq 1 ]; then
+            mkdir -p "$stamp/$(dirname "$r")"
+            mv "$abs" "$stamp/$r"
+        fi
+    done <<<"$entries"
+}
+
+do_status() {
+    if mountpoint -q "$VIEW"; then
+        printf 'view: %s (%s)\n' "$VIEW" "$(findmnt -no OPTIONS --mountpoint "$VIEW")"
+    else
+        printf 'view: %s NOT MOUNTED\n' "$VIEW"
+    fi
+    if [ -d "$upper" ]; then
+        printf 'upper: %s %s files %s\n' "$upper" \
+            "$(find "$upper" -type f 2>/dev/null | wc -l)" "$(du -sh "$upper" 2>/dev/null | cut -f1)"
+    else
+        printf 'upper: %s missing\n' "$upper"
+    fi
+    systemctl is-active "$UNIT" 2>/dev/null | sed 's/^/unit: /' || printf 'unit: %s unknown\n' "$UNIT"
+}
+
+case "$cmd" in
+    plan)    do_plan ;;
+    verify)  do_verify ;;
+    install) do_install ;;
+    reap)    do_reap ;;
+    status)  do_status ;;
+    *)       usage ;;
+esac

scripts/purge-agenttrash.sh

@@ -0,0 +1,335 @@
+#!/bin/sh
+# Deterministically empty every .AGENTTRASH directory under the given roots.
+#
+# PURPOSE
+#   The fleet's deletion policy parks deleted files in .AGENTTRASH/ — in-repo
+#   beside the item, elsewhere in $HOME as .AGENTTRASH/YYYYMMDD-topic — and
+#   waits for approval to purge. This script is that purge: given a machine,
+#   it clears .AGENTTRASH directories and nothing else.
+#
+# CONTRACT
+#   Default dry-run: the plan prints and nothing is deleted; --execute deletes.
+#   Every line the script can ever print for a candidate is decided at plan time
+#   and stored in the plan table, so a dry run prints exactly what --execute
+#   will print. A .AGENTTRASH that is a root's own trash (a repo's
+#   .AGENTTRASH, a machine's ~/.AGENTTRASH) prints `plan empty <path>`: the
+#   directory is the convention and survives, its staged contents are cleared.
+#   A .AGENTTRASH deeper than that is content and prints `plan remove <path>`
+#   plus, once executed, `removed <path>`; its own nested .AGENTTRASH entries
+#   belong to that tree and are never planned again. Candidates are found with
+#   `find -xdev` per root, so a mounted tree can never be walked by accident;
+#   ordering is lexicographic (LC_ALL=C). `/`, symlinked, and foreign-owned
+#   candidates are refused and reported (exit 1), never touched.
+#   A summary line `removed=<candidates> deleted=<removed> refused=<n>` closes
+#   every run and one dated line is appended to the log.
+#   Exit: 0 success, 1 operational failure, 2 usage error. A second run over
+#   the same tree plans nothing further and exits 0.
+#
+# PORTABILITY
+#   POSIX sh, clean under `shellcheck -S style`. Externals: find (must support
+#   -delete), stat, rm, date, mkdir. No bashisms, no GNU-only flags, no arrays,
+#   no `local`. Paths containing whitespace are rejected at parse time — the
+#   same rule applied to roots and candidates — so the plan can be walked with
+#   plain word splitting and nothing can be silently mis-split.
+#
+# USAGE
+#   purge-agenttrash [--execute] [--quiet] [--log PATH] [PATH...]
+#   --execute   delete; without it the run is a plan and deletes nothing
+#   --quiet     suppress per-line output; errors and the summary still print
+#   --log PATH  log destination (default: $HOME/.AGENTTRASH-cleanup.log)
+#   PATH...     roots to search (default: $HOME)
+#
+# Everything else is fixed by design: the root default, the ordering, the
+# guards. There is no configuration file and no environment override.
+
+set -eu
+
+usage() {
+    sed -n '3,44p' "$0" | sed 's/^# \{0,1\}//'
+}
+
+die() {
+    printf '%s: %s\n' 'purge-agenttrash' "$1" >&2
+    exit "${2:-1}"
+}
+
+# ------------------------------------------------------------------ arg parser
+# Roots accumulate as one space-separated string: a root containing whitespace is
+# rejected at parse time rather than silently split later. That leaves the plan
+# iteratable with plain `$@`-free word splitting, the only portable construct
+# that cannot corrupt a path it has already vetted.
+MODE=plan
+QUIET=0
+LOGPATH=
+ROOTS=
+while [ "$#" -gt 0 ]; do
+    case "$1" in
+        --help|-h)
+            usage
+            exit 0
+            ;;
+        --execute)
+            MODE=execute
+            ;;
+        --quiet)
+            QUIET=1
+            ;;
+        --log)
+            [ "$#" -ge 2 ] || die '--log needs a value' 2
+            LOGPATH=$2
+            shift
+            ;;
+        --log=*)
+            LOGPATH=${1#--log=}
+            ;;
+        -*)
+            die "unknown option: $1" 2
+            ;;
+        *)
+            case $1 in
+                *' '*) die "root contains whitespace: $1" 2 ;;
+            esac
+            if [ -z "$ROOTS" ]; then
+                ROOTS=$1
+            else
+                ROOTS="$ROOTS $1"
+            fi
+            ;;
+    esac
+    shift
+done
+
+[ -n "${HOME:-}" ] || die 'HOME is not set; pass roots explicitly or set HOME' 2
+[ -n "$LOGPATH" ] || LOGPATH=$HOME/.AGENTTRASH-cleanup.log
+
+# ------------------------------------------------------------ tool availability
+require() {
+    command -v "$1" >/dev/null 2>&1 || die "required tool not found: $1" 1
+}
+require find
+require stat
+require rm
+require date
+require mkdir
+
+# `find -delete` backs the in-place emptying. Probe it rather than trusting
+# `find --version`: a shell function named `find` must not be able to fake it.
+if ! find . -maxdepth 0 -name '.purge-probe-does-not-exist' -delete >/dev/null 2>&1; then
+    die 'find does not support -delete; refusing to guess at deletion' 1
+fi
+
+UID_SELF=$(id -u)
+
+# ------------------------------------------------------------------- locking
+# The lock dir also hosts this run's plan table, so both live and die together:
+# the exit trap removes the whole directory and only the log survives the run.
+LOCKDIR=$HOME/.agenttrash-purge.lock
+PLANFILE=$LOCKDIR/plan
+
+acquire_lock() {
+    if ! mkdir "$LOCKDIR" 2>/dev/null; then
+        held_pid=$(cat "$LOCKDIR/pid" 2>/dev/null || printf '%s\n' '')
+        if [ -n "$held_pid" ] && kill -0 "$held_pid" 2>/dev/null; then
+            die "another purge holds $LOCKDIR (pid $held_pid)" 1
+        fi
+        # Owner died mid-run: drop its pid, then retry exactly once. An
+        # unremovable/nonempty lock dir is not ours to reclaim.
+        rm -f "$LOCKDIR/pid" 2>/dev/null || true
+        rmdir "$LOCKDIR" 2>/dev/null || die "$LOCKDIR exists and is not ours to reclaim" 1
+        mkdir "$LOCKDIR" 2>/dev/null || die "cannot reacquire $LOCKDIR" 1
+    fi
+    printf '%s\n' "$$" >"$LOCKDIR/pid" || die "cannot stamp $LOCKDIR/pid" 1
+    : >"$PLANFILE" || die "cannot write $PLANFILE" 1
+    trap 'rm -rf "$LOCKDIR"' EXIT
+    trap 'exit 130' INT
+    trap 'exit 143' TERM
+}
+
+# ------------------------------------------------------------------ predicates
+is_dir() {  # true only for a real directory, never a symlink to one
+    [ -d "$1" ] && [ ! -L "$1" ]
+}
+
+is_ours() {  # true only when the invoking user owns the path
+    owner=$(stat -c '%u' "$1" 2>/dev/null) || return 1
+    [ "$owner" = "$UID_SELF" ]
+}
+
+refuse() {  # refuse <path> <reason>  — reported, never deleted
+    printf '%s: refused %s (%s)\n' 'purge-agenttrash' "$1" "$2" >&2
+    REFUSED=$((REFUSED + 1))
+}
+
+emit() {  # emit <line>  — silenced by --quiet
+    [ "$QUIET" -eq 1 ] || printf '%s\n' "$1"
+}
+
+# --------------------------------------------------------------- the plan table
+# One row per candidate: `<action> <path> <line>`. The first two fields tell
+# `act_on` what to do; the rest of the row is EXACTLY what the run prints for
+# that candidate — in plan mode verbatim, in execute mode as the first line of
+# its report. The table is a file, so no shell quoting rule (quoted-for,
+# heredoc, IFS games) can corrupt an approved plan, and the two modes can
+# never drift apart. A removal's second line (`removed <path>`) is derived from
+# the row's action, so it is also fixed at plan time.
+CANDIDATES=0
+REFUSED=0
+
+record() {  # record <action> <path> <line>  — append one row to the plan table
+    printf '%s %s %s\n' "$1" "$2" "$3" >>"$PLANFILE"
+    CANDIDATES=$((CANDIDATES + 1))
+}
+
+# ------------------------------------------------------------- candidate rule
+# The whole contract in one place:
+#   `/` is refused outright (never a candidate, never walked);
+#   a root that is a symlink or a plain file is refused, never followed;
+#   a root (or any path) that IS a real .AGENTTRASH prints `plan empty <path>`:
+#     the convention persists, its staged contents are what gets cleared;
+#   otherwise the root's DIRECT .AGENTTRASH child is also `empty` — it is that
+#     level's own trash ($HOME/.AGENTTRASH, x1/.AGENTTRASH, repo-a/.AGENTTRASH);
+#   every .AGENTTRASH deeper than that is content: `remove`, printed as
+#     `plan remove <path>` (+ `removed <path>` when executed), unless it lives
+#     inside a tree already claimed (then it dies with that tree, never planned
+#     twice) or is foreign-owned/symlinked (then refused).
+collect_root() {  # collect_root <path> -> 0 ok, 1 refused
+    target=$1
+    if [ "$target" = / ]; then
+        refuse / 'root refusal'
+        return 1
+    fi
+    if [ -L "$target" ] || ! [ -d "$target" ]; then
+        refuse "$target" 'not a real directory'
+        return 1
+    fi
+    if ! is_ours "$target"; then
+        refuse "$target" "not owned by uid $UID_SELF"
+        return 1
+    fi
+    if [ "${target##*/}" = .AGENTTRASH ]; then
+        record empty "$target" "plan empty $target"
+        return 0
+    fi
+    # `own` is the root's own .AGENTTRASH once handled; hits below it belong to
+    # that tree and must never be planned a second time.
+    own=
+    if [ -e "$target/.AGENTTRASH" ]; then
+        if ! is_ours "$target/.AGENTTRASH"; then
+            refuse "$target/.AGENTTRASH" "not owned by uid $UID_SELF"
+        elif ! is_dir "$target/.AGENTTRASH"; then
+            refuse "$target/.AGENTTRASH" 'not a real directory'
+        else
+            record empty "$target/.AGENTTRASH" "plan empty $target/.AGENTTRASH"
+            own=$target/.AGENTTRASH
+        fi
+    fi
+    # No `-type d`: implementations disagree on whether that matches a
+    # symlink-to-directory (GNU find says no, others yes), so every entry named
+    # .AGENTTRASH is collected and classified here instead of by `find`.
+    if ! found=$(find "$target" -xdev -name .AGENTTRASH 2>/dev/null); then
+        refuse "$target" 'find failed'
+        return 1
+    fi
+    previous=
+    for entry in $found; do
+        [ -n "$entry" ] || continue
+        if [ -n "$own" ]; then
+            case $entry in
+                "$own"|"$own"/*) continue ;;   # already emptied in place
+            esac
+        fi
+        if [ -n "$previous" ]; then
+            case $entry in
+                "$previous"/*) continue ;;   # nested: dies with its parent
+            esac
+        fi
+        previous=$entry
+        if ! is_ours "$entry"; then
+            refuse "$entry" "not owned by uid $UID_SELF"
+            continue
+        fi
+        if ! is_dir "$entry"; then
+            refuse "$entry" 'not a real directory'
+            continue
+        fi
+        record remove "$entry" "plan remove $entry"
+    done
+    return 0
+}
+
+# ------------------------------------------------------------------ execution
+remove_tree() {  # rm -rf a real directory; symlinks inside die as entries
+    is_dir "$1" || return 1
+    rm -rf -- "$1"
+}
+
+empty_dir() {  # empty a real directory in place, never following symlinks
+    is_dir "$1" || return 1
+    find "$1" -mindepth 1 -xdev -delete
+}
+
+# act_on <action> <path> <line> -> 0 when the action succeeded.
+# remove: prints the stored line (`plan remove <path>`) then `removed <path>`.
+# empty:  prints the stored line (`plan empty <path>`) and nothing else — the
+#         directory survives, so the plan line IS the outcome.
+act_on() {  # act_on <action> <path> <line> -> 0 when the action succeeded
+    if [ "$1" = remove ]; then
+        if remove_tree "$2" 2>/dev/null; then
+            emit "$3"
+            emit "removed $2"
+            return 0
+        fi
+        printf '%s: failed to remove %s\n' 'purge-agenttrash' "$2" >&2
+        return 1
+    fi
+    if empty_dir "$2" 2>/dev/null; then
+        emit "$3"
+        return 0
+    fi
+    printf '%s: failed to empty %s\n' 'purge-agenttrash' "$2" >&2
+    return 1
+}
+
+# Walk the plan table once: plan mode prints every row's line verbatim, execute
+# mode acts on each row and counts failures separately from plan-time refusals.
+walk_plan() {
+    while read -r action path line; do
+        [ -n "$action" ] || continue
+        if [ "$MODE" = plan ]; then
+            emit "$line"
+        elif act_on "$action" "$path" "$line"; then
+            REMOVED=$((REMOVED + 1))
+        else
+            EXEC_REFUSED=$((EXEC_REFUSED + 1))
+        fi
+    done <"$PLANFILE"
+}
+
+# ----------------------------------------------------------------------- main
+[ -n "$ROOTS" ] || ROOTS=$HOME
+
+acquire_lock
+
+for root in $ROOTS; do
+    collect_root "$root" || true
+done
+
+REMOVED=0
+EXEC_REFUSED=0
+walk_plan
+
+STATUS=0
+[ "$REFUSED" -eq 0 ] || STATUS=1
+[ "$EXEC_REFUSED" -eq 0 ] || STATUS=1
+
+printf 'removed=%s deleted=%s refused=%s\n' \
+    "$CANDIDATES" "$REMOVED" "$((REFUSED + EXEC_REFUSED))"
+
+if ! printf '%s\tplan\tcandidates=%s\tremoved=%s\trefused=%s\n' \
+    "$(date -u '+%Y-%m-%dT%H:%M:%SZ')" "$CANDIDATES" "$REMOVED" \
+    "$((REFUSED + EXEC_REFUSED))" >>"$LOGPATH" 2>/dev/null; then
+    printf '%s: cannot write log %s\n' 'purge-agenttrash' "$LOGPATH" >&2
+    STATUS=1
+fi
+
+exit "$STATUS"

scripts/qwen38-calib-window.sh

@@ -0,0 +1,143 @@
+#!/usr/bin/env bash
+# qwen38-calib-window.sh — RUNS ON DANKSERVER. The calibration window, and nothing else:
+# boot the BF16 rig, wait until it is actually serving, push the corpus until the engine says
+# the tap is full, stop the rig, restore frank-archon, and bank the reference container.
+#
+# The reference recording is already taken (kld-ref/kld.json exists), so this window is the
+# calibration tap alone — the one job that needs both cards after it. The kldref phase is NOT
+# re-run here: the chained launcher that used to do that moved the banked kld-ref into trash
+# first, which is how a 16-hour phase gets paid for twice.
+#
+# Every phase is gated on the previous one, because the last time this ran unguarded
+# (2026-10-29) the rig's boot died of an OOM, the health loop quietly fell through, and the
+# pusher fed four rounds of 10M tokens into RESTORED PRODUCTION with nothing recorded:
+#
+#   1. the runner must still be alive and its container must answer /health AND own the
+#      listening socket, or this script kills it, restores the service and exits non-zero —
+#      no push, ever. The socket clause is not decoration: the runner stops frank-archon
+#      first, the dying process keeps its listener for a moment, and a rig container that
+#      is still loading has not bound the port yet, so `podman inspect` + `/health` both
+#      report success for the WRONG process (measured 2026-10-30: `health ok waited=0s`,
+#      then the pusher refused on the container name and the window died at second zero);
+#   2. the pusher refuses on its own unless the rig wrote its marker into the tap dir, the
+#      marker names this container, and the container is running (scripts/calib-push.py
+#      --tap-marker), so a second lock survives a mistake in this file;
+#   3. the reference container is banked into trash only when BOTH the serve and the push
+#      exited 0 and the tap dir holds grams. The checkpoint that reproduces it stays.
+#
+# knobs: Q38_WINDOW_CALIB_CORPUS (calib-7168.jsonl — the re-chunked file the rig's pool can
+#        hold), Q38_WINDOW_HEALTH_TIMEOUT (seconds to wait for /health), Q38_WINDOW_ROUNDS
+#        (pusher rounds; one is 10M tokens), Q38_WINDOW_KEEP_RAD=1 to leave the .rad in place.
+# Usage: scripts/qwen38-calib-window.sh [--dry-run]
+set -uo pipefail
+cd "$(dirname "${BASH_SOURCE[0]}")/.." || exit 9
+
+DRY=0
+[ "${1:-}" = "--dry-run" ] && DRY=1
+say() { printf '%s %s\n' "$(date -Is)" "$*"; }
+
+MODELS="${FRANK_QWEN38_MODELS:-/home/dank/models}"
+CAMPAIGN="${FRANK_QWEN38_CAMPAIGN:-/home/dank/campaign/qwen38-w5nl64}"
+RAD="${FRANK_QWEN38_RAD:-$MODELS/rad/qwen38-bf16.rad}"
+CORPUS="${Q38_WINDOW_CALIB_CORPUS:-/home/dank/corpora/out/calib-7168.jsonl}"
+CALIB="${FRANK_QWEN38_CALIB_DIR:-$CAMPAIGN/calib}"
+HEALTH_TIMEOUT="${Q38_WINDOW_HEALTH_TIMEOUT:-3600}"
+ROUNDS="${Q38_WINDOW_ROUNDS:-6}"
+RIG=qwen38-bf16
+SERVICE=frank-archon
+PY="${PY:-python3}"
+
+# ---- preflight: every fact by identity, one owner (scripts/qwen38-preflight.sh) ----
+# The checks that used to live here were proxies, and each lied at least once: the space check
+# asked a directory that merely LOOKED related, `/health` answered for a dying production
+# process, `podman inspect` named a container that existed but had bound nothing. The shared
+# script binds each fact to its object — cwd, the digest the tag actually resolves to, free
+# space on the tap's own filesystem, every /dev/kfd holder's HOST pid against the service's,
+# the :7770 listener's host pid, the corpus and the reference recording. A resumed window
+# keeps an existing tap, so the tap-empty fact is waived.
+[ -f "$RAD" ]    || { echo "REFUSED: no BF16 reference container at $RAD — nothing to serve; scripts/build-qwen38-bf16-rad.sh" >&2; exit 2; }
+FRANK_QWEN38_CAMPAIGN="$CAMPAIGN" \
+Q38_WINDOW_CALIB_CORPUS="$CORPUS" \
+Q38_CALIB_MIN_FREE_MIB="${Q38_CALIB_MIN_FREE_MIB:-61440}" \
+    bash scripts/qwen38-preflight.sh --allow-existing-tap || exit 2
+! [[ "$HEALTH_TIMEOUT" =~ ^[0-9]+$ ]] && { echo "REFUSED: Q38_WINDOW_HEALTH_TIMEOUT must be seconds (got: $HEALTH_TIMEOUT)" >&2; exit 2; }
+! [[ "$ROUNDS" =~ ^[0-9]+$ ]] && { echo "REFUSED: Q38_WINDOW_ROUNDS must be a round count (got: $ROUNDS)" >&2; exit 2; }
+
+if [ "$DRY" = 1 ]; then
+    echo "dry run: calib window — the service is left running, nothing is mounted or written"
+    echo "would run: bash scripts/run-qwen38-bf16.sh calib          (rig boots, writes $CALIB/.rig-boot)"
+    echo "would wait: /health up within ${HEALTH_TIMEOUT}s, else kill the rig, restore $SERVICE, exit 1"
+    echo "would run: $PY scripts/calib-push.py --container $RIG --corpus $CORPUS --max-rounds $ROUNDS"
+    echo "would bank: $RAD -> ~/.AGENTTRASH/$(date +%Y%m%d)-qwen38-bf16-rad/  (only if both phases exit 0)"
+    exit 0
+fi
+
+say "calib window start pid=$$ corpus=$CORPUS rounds=$ROUNDS"
+[ -n "$(ls -A "$CALIB" 2>/dev/null)" ] && say "tap dir $CALIB is not empty; the engine's token floor is pooled, so an earlier partial tap still counts"
+
+# ---- phase 1: the rig -------------------------------------------------------
+say "phase=calib serve begin"
+bash scripts/run-qwen38-bf16.sh calib & spid=$!
+
+rig_up() {
+    podman inspect -f '{{.State.Running}}' "$RIG" 2>/dev/null | grep -qx true || return 1
+    curl -sf --max-time 10 -o /dev/null http://127.0.0.1:7770/health || return 1
+    # The answer must come from the CONTAINER. The runner stops frank-archon first, and the
+    # dying production process keeps its listener for a moment while the freshly created
+    # rig container is still loading — so `container running` + `/health` 200 both hold
+    # before the rig has bound anything. Measured 2026-10-30: `health ok waited=0s`, then the
+    # pusher refused on the container name and the window died at second zero. Ask the kernel
+    # who owns the listening socket and require it to be a rig process.
+    local lpid
+    lpid=$(ss -H -ltnp 'sport = :7770' 2>/dev/null | grep -o 'pid=[0-9]*' | head -1 | cut -d= -f2)
+    [ -n "${lpid:-}" ] || return 1
+    podman top "$RIG" pid 2>/dev/null | grep -qw "$lpid"
+}
+
+waited=0
+until rig_up; do
+    if ! kill -0 "$spid" 2>/dev/null; then
+        wait "$spid"; crc=$?
+        say "phase=calib serve end rc=$crc (the rig exited before it ever answered /health)"
+        echo "FAIL: the rig boot died (rc=$crc) — nothing was pushed; read /home/dank/logs/qwen38_bf16_calib.log" >&2
+        exit 1
+    fi
+    if [ "$waited" -ge "$HEALTH_TIMEOUT" ]; then
+        say "phase=calib health never came after ${HEALTH_TIMEOUT}s; killing the rig"
+        podman stop "$RIG" >/dev/null 2>&1 || true
+        wait "$spid"
+        echo "FAIL: the rig never answered /health within ${HEALTH_TIMEOUT}s — nothing was pushed" >&2
+        exit 1
+    fi
+    sleep 15
+    waited=$((waited + 15))
+done
+say "phase=calib health ok waited=${waited}s"
+
+# ---- phase 2: the tap -------------------------------------------------------
+say "phase=calib push begin"
+"$PY" scripts/calib-push.py --port 7770 --parallel 8 --container "$RIG" \
+    --tap-dir "$CALIB" --corpus "$CORPUS" --max-rounds "$ROUNDS"
+prc=$?
+say "phase=calib push end rc=$prc"
+
+podman stop "$RIG" >/dev/null 2>&1 || true
+wait "$spid"; crc=$?
+say "phase=calib serve end rc=$crc"
+
+if [ "$prc" -ne 0 ] || [ "$crc" -ne 0 ]; then
+    echo "FAIL: serve rc=$crc push rc=$prc — the tap is NOT banked and $RAD stays: $(basename "$RAD")" >&2
+    exit 1
+fi
+[ -n "$(ls -A "$CALIB" 2>/dev/null)" ] || { echo "FAIL: both phases exited 0 but the tap dir $CALIB is empty" >&2; exit 1; }
+say "TAP BANKED: $(ls -1 "$CALIB" | wc -l) file(s) in $CALIB"
+
+# ---- phase 3: bank the reference container ----------------------------------
+if [ "${Q38_WINDOW_KEEP_RAD:-0}" = "1" ]; then
+    say "reference .rad kept (Q38_WINDOW_KEEP_RAD=1)"
+else
+    T="${Q38_WINDOW_TRASH_ROOT:-$HOME/.AGENTTRASH}/$(date +%Y%m%d)-qwen38-bf16-rad"
+    mkdir -p "$T" || { echo "FAIL: cannot create trash dir $T" >&2; exit 1; }
+    mv "$RAD" "$T/" && say "reference .rad banked in $T (the checkpoint that rebuilds it stays)"
+fi
+say "calib window end"

scripts/qwen38-preflight.sh

@@ -0,0 +1,136 @@
+#!/usr/bin/env bash
+# qwen38-preflight.sh — the facts a calibration window must have BEFORE it takes the cards,
+# each checked by IDENTITY and not by proxy. Every line is PASS or FAIL with the reason.
+#
+# Why this file exists: on 2026-10-30 a window died at second zero because `/health` answered
+# 200 from the *production* process that was still draining its listener, while the freshly
+# created rig container had not bound anything. Other runs died from a port that answered for
+# the wrong process, a /dev/kfd held by an orphaned engine, a tap dir on a 35 GiB filesystem,
+# and a launcher whose cwd silently lacked the repo. None of those were math problems; all of
+# them were a check that measured something adjacent to the thing it meant.
+#
+# Usage:  bash scripts/qwen38-preflight.sh [--allow-existing-tap]
+# Exit:   0 all PASS · 2 any FAIL · 3 a fact could not be evaluated (missing tool)
+set -uo pipefail
+
+SELF_DIR="$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)"
+REPO_ROOT="$(cd "$SELF_DIR/.." && pwd)"
+
+CAMPAIGN="${FRANK_QWEN38_CAMPAIGN:-/home/dank/campaign/qwen38-w5nl64}"
+TAP="$CAMPAIGN/calib"
+CORPUS="${Q38_WINDOW_CALIB_CORPUS:-/home/dank/corpora/out/calib-7168.jsonl}"
+NEED_MIB="${Q38_CALIB_MIN_FREE_MIB:-61440}"
+ALLOW_EXISTING=0
+[ "${1:-}" = "--allow-existing-tap" ] && ALLOW_EXISTING=1
+
+# The producer pin is the calib image; the window fails closed if the tag on this host is not it.
+WANT_DIGEST="${Q38_PREFLIGHT_IMAGE_DIGEST:-sha256:69add58f7de1ed2bca2b1e20a4cc75717557df3f73569a178eca5aafe8c44093}"
+IMAGE_REF="${Q38_CALIB_IMAGE:-stilldeadcode/radiance@$WANT_DIGEST}"
+SERVICE="frank-archon"
+PORT=7770
+
+fails=0
+ok()   { printf 'PASS  %-14s %s\n' "$1" "$2"; }
+bad()  { printf 'FAIL  %-14s %s\n' "$1" "$2"; fails=$((fails + 1)); }
+skip() { printf 'WARN  %-14s %s\n' "$1" "$2"; }
+
+# 1. cwd — a detached launcher that lost its cwd fails here in a second, not in the log later.
+if [ "$(cd "$REPO_ROOT" && pwd)" = "$(pwd)" ] && [ -x "$REPO_ROOT/scripts/run-qwen38-bf16.sh" ]; then
+    ok cwd "running from the repo root ($REPO_ROOT)"
+else
+    bad cwd "cwd is $(pwd), not the repo root ($REPO_ROOT) — every relative script path would break"
+fi
+
+# 2. image — resolve the REF to a digest. A stale local `:latest` is a different model of the world.
+if command -v podman >/dev/null 2>&1; then
+    # An image carries EVERY digest it was pulled under (a tag plus the by-digest pull), so this
+    # asks whether the pin is one of them — reading RepoDigests[0] reports a healthy image as wrong.
+    got="$(podman image inspect -f '{{range .RepoDigests}}{{println .}}{{end}}' "$IMAGE_REF" 2>/dev/null \
+           | sed 's/.*@//' | grep -c "^${WANT_DIGEST}$")"
+    if [ "${got:-0}" -ge 1 ]; then
+        ok image "$IMAGE_REF carries ${WANT_DIGEST:0:19}…"
+    elif podman image exists "$IMAGE_REF" 2>/dev/null; then
+        bad image "$IMAGE_REF is present but does not carry $WANT_DIGEST (a retagged or older pull)"
+    else
+        bad image "$IMAGE_REF is not present on this host"
+    fi
+else
+    skip image "podman not on PATH — cannot resolve $IMAGE_REF"
+fi
+
+# 3. space — df the FILE PATH the seal writes to, never a directory that merely looks related.
+if [ -d "$TAP" ] || mkdir -p "$TAP" 2>/dev/null; then
+    avail="$(df --output=avail -B 1M "$TAP" 2>/dev/null | tail -1 | tr -d ' ')"
+    if [[ "${avail:-}" =~ ^[0-9]+$ ]] && [ "$avail" -ge "$NEED_MIB" ]; then
+        ok space "$avail MiB free on $(df --output=target "$TAP" | tail -1) (need $NEED_MIB)"
+    else
+        bad space "${avail:-unknown} MiB free on $TAP, need $NEED_MIB — one seal writes ~43 GB"
+    fi
+    # 4. tap cleanliness — stale grams from another run must never be mixed in.
+    n=$(ls -A "$TAP" 2>/dev/null | grep -c . || true)
+    if [ "$n" -eq 0 ] || [ "$ALLOW_EXISTING" = 1 ]; then
+        ok tap-empty "$n entries in $TAP"
+    else
+        bad tap-empty "$n entries in $TAP — rotate them to .AGENTTRASH first"
+    fi
+else
+    bad space "cannot create $TAP"
+fi
+
+# 5. device — every /dev/kfd holder must be a process of the service we are about to stop. An
+#    orphaned engine from a previous run makes the next boot die with `startup stopped: device
+#    error`. Identity comes from podman's own host-pid list: `podman top <ctr> pid` prints
+#    CONTAINER-internal pids (1, 7, …), which compare against nothing, and a rootless cgroup
+#    carries the container ID, never the service name — both are proxies that lie.
+PROD_CTR="${Q38_PREFLIGHT_PROD_CONTAINER:-$SERVICE}"
+service_pids() { podman top "$PROD_CTR" hpid 2>/dev/null | tail -n +2; }
+if [ -e /dev/kfd ]; then
+    mine="$(service_pids | tr '\n' ' ')"
+    foreign=""
+    for p in $(fuser /dev/kfd 2>/dev/null); do
+        case " $mine " in
+            *" $p "*) : ;;                                          # the service we will stop
+            *) foreign="$foreign $p:$(cat /proc/$p/comm 2>/dev/null)" ;;
+        esac
+    done
+    if [ -z "$foreign" ]; then
+        ok device "no foreign /dev/kfd holders"
+    else
+        bad device "foreign holders:$foreign — kill by pid, never by pattern"
+    fi
+else
+    skip device "/dev/kfd absent — not an AMD host?"
+fi
+
+# 6. port — a listener on :7770 must be a process of the service we stop, never an unknown one.
+lpid="$(ss -H -ltnp "sport = :$PORT" 2>/dev/null | grep -o 'pid=[0-9]*' | head -1 | cut -d= -f2)"
+if [ -z "${lpid:-}" ]; then
+    ok port ":$PORT is free"
+else
+    mine="$(service_pids | tr '\n' ' ')"
+    case " $mine " in
+        *" $lpid "*) ok port ":$PORT is held by $SERVICE (host pid $lpid) — the window stops it" ;;
+        *)           bad port ":$PORT is held by host pid $lpid ($(cat /proc/$lpid/comm 2>/dev/null)), not $SERVICE" ;;
+    esac
+fi
+
+# 7. corpus — the push source must exist and be the re-chunked file the window documents.
+if [ -s "$CORPUS" ]; then
+    ok corpus "$(wc -l < "$CORPUS") prompts in $CORPUS"
+else
+    bad corpus "$CORPUS is missing or empty"
+fi
+
+# 8. reference — the KL gate has nothing to compare against without it.
+if [ -s "$CAMPAIGN/kld-ref/kld.json" ]; then
+    ok reference "$(du -h "$CAMPAIGN/kld-ref/kld.json" | cut -f1) kld.json present"
+else
+    bad reference "$CAMPAIGN/kld-ref/kld.json is missing"
+fi
+
+echo
+if [ "$fails" -gt 0 ]; then
+    echo "PREFLIGHT FAILED: $fails fact(s). The window must not take the cards."
+    exit 2
+fi
+echo "PREFLIGHT PASSED: every fact checked by identity."

scripts/qwen38_common.py

@@ -0,0 +1,94 @@
+#!/usr/bin/env python3
+"""Shared plumbing for the qwen38 campaign corpus tools.
+
+The three tools move the same two shapes — a JSONL of prompt records and a
+one-artifact JSON manifest — and they all tokenize with the SAME tokenizer. That
+last part is not a detail: a KLD reference is only meaningful against the exact
+token sequence the corpus recorded, so `n_tokens` exists once, here, and every
+tool counts the same way. A window that decode/encode does not round-trip is a
+refusal, never a warning.
+
+Loaded as a sibling module. Each tool puts its own directory on `sys.path`
+first, so `python3 scripts/<tool>.py` and the driver's container mount
+(`runpy.run_path('/work/<tool>.py')`, `-v scripts:/work:ro`) both resolve it —
+run the tools by path, not piped through stdin.
+"""
+import contextlib
+import json
+import os
+import sys
+import tempfile
+
+
+def refuse(msg: str) -> None:
+    """Exit 2 with a REFUSED line — the code argparse uses for unusable input."""
+    print(f"REFUSED: {msg}", file=sys.stderr)
+    sys.exit(2)
+
+
+def load_tokenizer(path: str):
+    try:
+        from tokenizers import Tokenizer
+    except ImportError as exc:  # the corpus tools carry their own venv
+        refuse(f"tokenizers is not importable by {sys.executable} — run the "
+               f"corpus venv's python (on dankserver: "
+               f"/home/dank/venvs/radcorpus/bin/python) or set "
+               f"FRANK_QWEN38_TOKENIZER: {exc}")
+    if not os.path.isfile(path):
+        refuse(f"no tokenizer at {path}")
+    return Tokenizer.from_file(path)
+
+
+def n_tokens(tok, text: str) -> int:
+    """The corpus's own count: no special tokens added, prompt side only."""
+    return len(tok.encode(text, add_special_tokens=False).ids)
+
+
+def iter_jsonl(path: str):
+    """Yield `(lineno, record)` for every non-blank line; refuse on a bad line.
+
+    Field validation stays with the caller: the three corpora carry different
+    fields (`prompt` alone, `prompt`+`cls`, `prompt`+`score_from`+`source`), and
+    a record missing the one the tool needs is that tool's refusal to make.
+    """
+    with open(path, encoding="utf-8", errors="replace") as fh:
+        for lineno, line in enumerate(fh, 1):
+            if not line.strip():
+                continue
+            try:
+                rec = json.loads(line)
+            except json.JSONDecodeError as exc:
+                refuse(f"{path}:{lineno} is not JSON: {exc}")
+            yield lineno, rec
+
+
+def write_jsonl(path: str, rows) -> None:
+    with open(path, "w", encoding="utf-8") as fh:
+        for row in rows:
+            fh.write(json.dumps(row, ensure_ascii=False) + "\n")
+
+
+@contextlib.contextmanager
+def atomic_writer(path: str):
+    """Write `path` through a temporary sibling and rename on success.
+
+    An interrupted run then leaves no half corpus behind, and the temporary name
+    keeps the one-artifact-one-name law the driver applies to a score report.
+    """
+    d = os.path.dirname(os.path.abspath(path)) or "."
+    fd, tmp = tempfile.mkstemp(dir=d, prefix="." + os.path.basename(path) + "-")
+    try:
+        with os.fdopen(fd, "w", encoding="utf-8") as fh:
+            yield fh
+        os.replace(tmp, path)
+    except BaseException:
+        if os.path.exists(tmp):
+            os.unlink(tmp)
+        raise
+    os.chmod(path, 0o644)
+
+
+def write_json(path: str, obj) -> None:
+    with atomic_writer(path) as fh:
+        json.dump(obj, fh, indent=1)
+        fh.write("\n")

scripts/rechunk-kld-corpus.py

@@ -0,0 +1,192 @@
+#!/usr/bin/env python3
+"""Re-chunk a KLD corpus so every document fits the container that scores it.
+
+A KLD reference is recorded by booting one container and prefilling every corpus
+document through it. radiance refuses a document the KV pool cannot hold on its
+own (core/mem/kv.cpp `can_ever_fit`: a paged group holds `n_blocks *
+block_size` tokens), and the number that matters is printed by the boot ledger,
+not by a flag:
+
+    W kv group 'kv_attn' holds 7436 tokens but the context limit is 32768, so
+    a prompt longer than 7436 tokens is refused (400) ... or lower
+    --max-model-len to 7436
+
+The BF16 qwen4exp container at the campaign's shape gets 270.00 MiB of cache --
+the expert plane owns both cards -- so its real window is 7,436 tokens however
+`--max-model-len` is spelled. The corpus the campaign inherited from the swift15
+era is 197 documents of 8,193-11,970 tokens, because make-calib-corpus.py's
+`--kld-min-prompt 8192` sets a floor and no ceiling: every one of them is
+unscoreable there, and three drained windows have proved it.
+
+This script moves the document boundaries and nothing else. Each record's token
+ids -- counted with the same tokenizer the generator used -- are cut into
+windows of at most `--max-doc-tokens`, so a record's windows concatenate back to
+that record's own token sequence, unchanged: no token dropped, rewritten or
+reworded. Each window gets its own `score_from = len - --score-tail`, the same
+relation the generator wrote for the parent, so the reference scores 256
+positions per window instead of per document: 100,864 measured positions over
+the same text against the 50,432 the inherited corpus scored, at a shorter
+context. The calibration corpus is untouched, so the quantization itself does
+not move.
+
+Usage:
+  rechunk-kld-corpus.py --corpus kld.jsonl --out kld.jsonl.new \
+      [--tokenizer-json PATH] [--max-doc-tokens 7168] [--score-tail 256] [--force]
+
+--out must not exist and be non-empty without --force: the same
+one-artifact-one-name law run-qwen38-bf16.sh applies to a score report. The file
+is written through a temporary name and renamed, so an interrupted run leaves no
+half corpus behind. A JSON summary of what was written goes to stdout.
+"""
+import argparse
+import dataclasses
+import json
+import os
+import sys
+
+_HERE = os.path.dirname(os.path.abspath(__file__)) if "__file__" in globals() else os.getcwd()
+if _HERE not in sys.path:
+    sys.path.insert(0, _HERE)
+
+import qwen38_common as qc
+
+MISSING = object()  # "the parent record carried no source field"
+
+
+def parse_args() -> argparse.Namespace:
+    ap = argparse.ArgumentParser(description=__doc__.splitlines()[0])
+    ap.add_argument("--corpus", required=True,
+                    help="the kld.jsonl to re-chunk (records: prompt, score_from, source)")
+    ap.add_argument("--out", required=True,
+                    help="file to write; refuse if it exists and is non-empty without --force")
+    ap.add_argument("--tokenizer-json",
+                    default=os.environ.get("FRANK_QWEN38_TOKENIZER",
+                                           "/home/dank/models/Qwen3.8-Flash-Next/tokenizer.json"),
+                    help="the tokenizer the corpus was authored with (the same one the "
+                         "manifest in the corpus output dir names)")
+    ap.add_argument("--max-doc-tokens", type=int, default=7168,
+                    help="the widest document the scoring container can hold; 7,168 is "
+                         "7x1024, the largest whole kilopage under the 7,436 the bf16 "
+                         "container's ledger reports")
+    ap.add_argument("--score-tail", type=int, default=256,
+                    help="positions scored at the end of every document, as the generator "
+                         "recorded it")
+    ap.add_argument("--force", action="store_true",
+                    help="overwrite a non-empty --out (after moving the old file to trash)")
+    return ap.parse_args()
+
+
+@dataclasses.dataclass(frozen=True)
+class Window:
+    """One scoring document cut out of a parent record."""
+    prompt: str
+    score_from: int
+    n_tokens: int
+    clamped: bool
+    source: object = MISSING
+
+    def record(self) -> dict:
+        rec = {"prompt": self.prompt, "score_from": self.score_from}
+        if self.source is not MISSING:
+            rec["source"] = self.source
+        return rec
+
+
+def validate(a: argparse.Namespace) -> None:
+    """Everything that makes this run unusable, named before a token is cut."""
+    if a.max_doc_tokens <= 0:
+        qc.refuse(f"--max-doc-tokens must be positive (got {a.max_doc_tokens})")
+    if a.score_tail <= 0:
+        qc.refuse(f"--score-tail must be positive (got {a.score_tail})")
+    if a.max_doc_tokens <= a.score_tail:
+        qc.refuse(f"--max-doc-tokens {a.max_doc_tokens} cannot hold a "
+                  f"{a.score_tail}-position score tail: every window would be all "
+                  "tail and no context")
+    if not os.path.isfile(a.corpus):
+        qc.refuse(f"no corpus at {a.corpus}")
+    if os.path.realpath(a.out) == os.path.realpath(a.corpus):
+        qc.refuse(f"--out {a.out} is the corpus itself; move it to trash first — the "
+                  "trash law means the old segmentation stays recoverable")
+    if os.path.exists(a.out) and os.path.getsize(a.out) > 0 and not a.force:
+        qc.refuse(f"{a.out} exists and is not empty; pass --force after moving it to trash")
+
+
+def cut(rec: dict, tok, a: argparse.Namespace,
+        where: str) -> tuple[list[Window], int]:
+    """Cut one record's token ids into windows, proving each one round-trips.
+
+    The round-trip encode is the whole reason this tool may claim the reference
+    scores the corpus's own token sequence: decoding a window to text and
+    encoding it back must return the identical ids, or the score is of a
+    different document than the one on disk.
+    """
+    ids = tok.encode(rec["prompt"], add_special_tokens=False).ids
+    windows: list[Window] = []
+    for start in range(0, max(len(ids), 1), a.max_doc_tokens):
+        window = ids[start:start + a.max_doc_tokens]
+        if not window:
+            continue
+        text = tok.decode(window)
+        if tok.encode(text, add_special_tokens=False).ids != window:
+            qc.refuse(f"{where} window at token {start} does not round-trip through "
+                      "text — the reference would score a different token sequence "
+                      "than the corpus holds")
+        windows.append(Window(prompt=text,
+                              score_from=max(len(window) - a.score_tail, 0),
+                              n_tokens=len(window),
+                              clamped=len(window) < a.score_tail,
+                              source=rec["source"] if "source" in rec else MISSING))
+    return windows, len(ids)
+
+
+def rechunk(a: argparse.Namespace, tok) -> dict:
+    """Write the re-chunked corpus and return the summary it printed."""
+    docs_in = docs_out = tokens_in = tokens_out = 0
+    longest_child = clamped = scored = 0
+    sources: dict[str, int] = {}
+    with qc.atomic_writer(a.out) as out:
+        for lineno, rec in qc.iter_jsonl(a.corpus):
+            where = f"{a.corpus}:{lineno}"
+            if not isinstance(rec.get("prompt"), str):
+                qc.refuse(f"{where} has no string 'prompt' field")
+            windows, n_parent = cut(rec, tok, a, where)
+            docs_in += 1
+            tokens_in += n_parent
+            for w in windows:
+                out.write(json.dumps(w.record(), ensure_ascii=False) + "\n")
+                docs_out += 1
+                tokens_out += w.n_tokens
+                longest_child = max(longest_child, w.n_tokens)
+                clamped += 1 if w.clamped else 0
+                scored += w.n_tokens - w.score_from
+                if "source" in rec:
+                    for cls in str(rec["source"]).split("+"):
+                        sources[cls] = sources.get(cls, 0) + 1
+    if tokens_out != tokens_in:
+        raise SystemExit(f"REFUSED: the windows hold {tokens_out} tokens and the corpus "
+                         f"has {tokens_in} — re-chunking lost or invented tokens")
+    return {
+        "corpus": a.corpus,
+        "out": a.out,
+        "tokenizer": os.path.basename(a.tokenizer_json),
+        "documents_in": docs_in,
+        "documents_out": docs_out,
+        "tokens_in": tokens_in,
+        "tokens_out": tokens_out,
+        "longest_document": longest_child,
+        "max_doc_tokens": a.max_doc_tokens,
+        "score_tail": a.score_tail,
+        "scored_positions": scored,
+        "clamped_score_from": clamped,
+        "source_documents": sources,
+    }
+
+
+def main() -> None:
+    a = parse_args()
+    validate(a)
+    print(json.dumps(rechunk(a, qc.load_tokenizer(a.tokenizer_json)), indent=1))
+
+
+if __name__ == "__main__":
+    main()

scripts/recipes/qwen4exp-qwen38-w5nl64-bf16trunk.recipe

@@ -0,0 +1,80 @@
+# qwen4exp-qwen38-w5nl64-bf16trunk.recipe — every routed expert, BOTH projections, at five bits:
+# u5 codes into libquant's 32-level w5nl table, an E4M3 scale a group of 64 under the fixed 2^-13
+# second level, rotated by the 128-wide Hadamard, chosen by GPTQ against $CALIB and swept three
+# times by coordinate descent. This is `w5nl64a8h` in the engine's served-encoding table — the top of
+# the four-bit class — applied wider than any published radiance container of this checkpoint: the
+# author's own w5 rung (`qwen4exp-w5dn64-i8-hc8m.recipe`) moves only `down`, because that rung was
+# priced as a byte-for-error trade against prefill; this campaign buys the whole rung and pays for
+# it out of the pinned host pool (+15.3 GiB of expert plane, `--host-pool-mib 61440` per rank), not
+# out of the KV pool.
+#
+# Every other line is the served container's recipe (dump it with
+# `rad-info --recipe /home/dank/models/rad/qwen3.8-next-flash-fp8-iq4r-moe.rad`), minus the rows this
+# campaign leaves at the checkpoint:
+#
+#   * the trunk's block linears, the shared expert and the hyper-connection mixers are NOT listed —
+#     dropping their i8/fp8 lines leaves them bf16, which is what the checkpoint holds. Measured bill
+#     +1.978 GiB of always-resident weight = ~131k tokens of KV off the pool.
+#   * `ssm_inz`/`ssm_out` STAY int8: the delta-net gates are the largest trunk rows (+1.9 GiB, 126k
+#     tokens) and the least sensitive per byte.
+#   * `mtp.draft_head` STAYS 2-bit and the PLE n-gram table STAYS one fixed E4M3 scale: the draft
+#     head's rung is cheap and its acceptance cost is a decode question this campaign does not touch,
+#     and bf16 for the n-gram table would pin +47.69 GiB of host RAM for a lookup.
+#
+# Protected experts: none in this file. `scripts/apply-protected-cast.sh` appends the derived set
+# (48, two per layer so each rank's 256-expert half stays even — the four-bit GEMM takes the rest of
+# a half as two equal tables by parity) from the fresh 12M-token energy/routing census, and they must
+# sit ABOVE the expert row because the first rule that matches a weight decides it.
+#
+#   CALIB=/home/dank/campaign/qwen38-w5nl64/calib \
+#     rad-convert /home/dank/models/Qwen3.8-Flash-Next \
+#       --recipe this.file --imatrix .../imatrix.gguf -o out.rad
+#
+# The arm driver (`scripts/convert-qwen38-w5nl64.sh`) generates the per-arm variants of the expert
+# row from this template — cd=3 (control), cd=6, and `act_order=1` with NO cd= (libquant refuses the
+# pair, lq_registry.cpp:105) — rather than keeping three near-duplicate recipes that can drift apart.
+#
+# The whole campaign, in the order that fits on one disk (every step runs on dankserver):
+#
+#   1. scripts/download-qwen38-bf16.sh --fetch      the vendor BF16 checkpoint, 335.3 GiB
+#   2. scripts/build-qwen38-bf16-rad.sh             the BF16 reference container, another 335.3 GiB
+#   3. one drained window, two boots:
+#        scripts/run-qwen38-bf16.sh kldref          the KLD reference recording (the zero of scale)
+#        scripts/run-qwen38-bf16.sh calib           the 12M-position calibration tap -> grams
+#   4. trash qwen38-bf16.rad                        the peak-disk law: the converts need its room
+#   5. scripts/grams-to-imatrix.py                  grams -> imatrix.gguf in the campaign dir
+#   6. scripts/convert-qwen38-w5nl64.sh             the arms, planned and billed before each one runs
+#   7. fetch the published FP8-native release, then per container:
+#        scripts/run-qwen38-bf16.sh score RAD OUT   fp8 anchor, served iq4r, each arm
+#      then scripts/kld-gate-qwen38.sh fp8.json iq4r.json cand.json
+#   8. a winner is a trio edit plus `scripts/deploy.sh dankserver --service frank-archon`; the
+#      rollback is the commit before it. If the win needs the bigger pinned plane, raise
+#      FRANK_ARCHON_HOST_POOL_MIB and FRANK_ARCHON_TTM_TARGET_MIB in the SAME commit: the preflight
+#      floor is 2xHOST_POOL + 48,824 + prefix cache + MNBT + the store, and a pool raised without
+#      the TTM target is a boot refusal that Restart=always repeats forever, invisible to deploy.sh.
+
+# PROTECTED-BEGIN — scripts/apply-protected-cast.sh splices the derived cast lines in here.
+# They MUST sit above the expert rule: the first rule whose glob matches a weight decides it, so a
+# cast line below the five-bit rule would never win. The set is derived from the campaign's own
+# 12M-token recording (`derive-protected-experts.py --target-count 48` over the fresh grams), and the
+# driver splices it into a per-arm COPY so this template stays the clean, reviewable baseline.
+# PROTECTED-END
+
+# Every routed expert, trunk and MTP layer alike, both projections, at five bits. The experts' scales
+# sit between 2^-15 and 2^-10.6 under w4nl's top level of 22 and half a binade lower under w5nl's 30
+# — inside E4M3's normal range under 2^-13 either way. The MTP layer has no recording and is rounded
+# to nearest, which libquant says on stderr; it is one layer of 49 and only ever changes drafts.
+blk.*.ffn_*_exps.*.weight            gptq table=w5nl group=64 scale=fp8_e4m3 scale2=f32 block2=*x* scale2_value=0.0001220703125 transform=fwht128 rule=search cd=3 calib=$CALIB
+
+# The delta-net's in/out gates, int8 with a bf16 scale a group of 128 searched for least error — the
+# served recipe's lines, unchanged.
+blk.*.ssm_inz.weight                 rtn codes=i8 group=128 scale=bf16 clamp=sym rule=search
+blk.*.ssm_out.weight                 rtn codes=i8 group=128 scale=bf16 clamp=sym rule=search
+
+# The MTP head's two mixing rows, E4M3 as served. They only ever change the drafts.
+mtp.fc_hidden.weight                 rtn codes=fp8_e4m3 group=128 scale=f32
+mtp.fc_embedding.weight              rtn codes=fp8_e4m3 group=128 scale=f32
+
+# The n-gram table at one fixed E4M3 scale, and the 2-bit draft head — both the served lines.
+blk.*.ple_ngram.weight               rtn codes=fp8_e4m3 block=*x* scale=bf16 rule=fixed scale_value=1.99317932128906e-4
+mtp.draft_head.weight                rtn codes=u2 zero=u8 group=128 scale=f16

scripts/run-qwen38-bf16.sh

@@ -0,0 +1,485 @@
+#!/usr/bin/env bash
+# run-qwen38-bf16.sh — RUNS ON DANKSERVER. Boot a container over the campaign's
+# ONE measuring shape for the phases that need both R9700s. The engine owns the
+# cards, so frank-archon is stopped first and restored on every exit path — the
+# autoresearch pattern: stop, never mask (the unit file in /etc/systemd/system
+# is a symlink into the checkout and mask fails on a relinked unit; the deploy
+# restores it).
+#
+#   kldref            --kld-record the reference log-probabilities: the engine
+#                     runs /home/dank/corpora/out/kld.jsonl (197 prompts, 50,432
+#                     scored positions, score tail 256) through prefill, writes
+#                     /home/dank/campaign/qwen38-w5nl64/kld-ref, and EXITS. The
+#                     model behind it is the BF16 container from
+#                     scripts/build-qwen38-bf16-rad.sh — the vendor checkpoint's
+#                     own numbers, which is what makes it the zero point.
+#   score RAD OUT     --kld-ref score ONE container against that recording and
+#                     EXIT: the FP8-native anchor, the served iq4r, and every
+#                     arm. Deliberately the SAME engine pin, the same TP, the
+#                     same step size, the same fp8 KV, --deterministic and no
+#                     speculation as kldref — KL and top-1 agreement are only
+#                     comparable across boots when everything except the weights
+#                     is identical, so this mode is one argv with --model and
+#                     the two --kld-* flags changed, never a second shape.
+#                     RAD must sit under the models dir (the container reads it
+#                     through the ro /models mount); OUT lands on a writable
+#                     mount because --kld-out F also writes F.rows beside it.
+#   calib             serve on :7770 with the calibration tap open
+#                     (RADIANCE_CALIB_DIR + RADIANCE_CALIB_TOKENS=12000000). A
+#                     separate driver — scripts/calib-push.py --port 7770
+#                     --container qwen38-bf16 --corpus
+#                     /home/dank/corpora/out/calib-7168.jsonl — pushes the corpus;
+#                     at the floor the engine writes the Grams and the tap switches
+#                     itself off (calib-push prints TAP_DONE). Stop this script
+#                     after that; the trap brings frank-archon back. THE FLOOR IS
+#                     THE CAMPAIGN'S RULE (12 M tokens, Frank 2026-10-30): the
+#                     served iq4r this arm competes with was calibrated on a 10 M-
+#                     token set, so the arm is calibrated at least as deeply. One
+#                     round of the re-chunked corpus is ~9.31 M tokens, so sealing
+#                     takes round 1 plus part of round 2 — a round that seals
+#                     nothing is NORMAL (the floor is pooled), and the pusher's
+#                     round_verdict treats a flat round as "continue", never as a
+#                     failure. One seal writes ~40 GiB.
+#
+# Shape: the vendor's bf16 calibration lines — expert_tiered placement with the
+# weights disk tier, deterministic (the resident set pinned to the loader's plan,
+# so a recorded reference and a scored candidate see the same one), no
+# speculation, fp8 KV, a 32,768-token window, the PLE n-gram table pinned to RAM,
+# and the production chat template rather than the one baked into the container —
+# every scored container is asked the same tokenised questions the reference
+# answered. The window the flag spells is 32,768; the pool this payload's boot
+# actually gets is 7,436 tokens, so kld.jsonl is re-chunked by
+# scripts/rechunk-kld-corpus.py into documents of at most 7,168 tokens — the
+# widest one the reference can hold. The kldref gate below re-checks the file
+# against the declared window, and the re-chunker is what keeps that honest.
+#
+# NO --vram-kv-mib: it was REMOVED at 1.3.0 and naming it is a boot refusal, not
+# a no-op — verified on the pinned image 2026-10-27: "E --vram-kv-mib was
+# removed: the card is split between the weights and the cache from what the
+# model needs." The resolver derives the split from --host-pool-mib,
+# --gpu-headroom-mib and the activation arena; the rig inherits the first two from
+# frank-archon.env, the file that owns them. On THIS payload the host pool is not
+# the cache's supplier: the boot lands in vram_budget.cpp's `no lender` branch,
+# where the experts keep their plan-fixed 25.29 GiB and the cache gets the residue.
+# Measured twice — 12,288 and 49,152 MiB of host pool both printed the same
+# "kv cache 271.25 MiB ... 7,436"; only --gpu-headroom-mib moved anything
+# (768 -> 96 grew vram_weights 29.41 -> 30.06 GiB). The supplier the source names
+# is the arena: vram_budget.cpp:117, "lower --max-num-batched-tokens (the arena
+# scales with it)" — which is what the step size below now does.
+#
+# The pool, not the window, decides what the recorder accepts: it refuses any
+# document longer than the pool, so the 32,768-token window was decorative while the
+# cache held 7,436 — "E kld: document 196 is 8507 tokens and the KV pool holds
+# 7436, so it can never be scored" on all 197 records, exit 1. The longest record
+# is 10,554 tokens by the engine's own count (chars/3.6 for this file, not the
+# chars/4.97 this repo measured on the K3 corpus).
+#
+# The step size is ONE value for the recording and every scoring pass, and it is
+# the value frank-archon.env owns (FRANK_ARCHON_MAX_BATCHED_TOKENS, 2,048): the
+# quantised expert FILE tiers (the served iq4r, the arms) stream only at
+# MNBT<=2048 and refuse the widened form above it, and the arena a step costs the
+# card comes straight out of the cache — 1.08 GiB at a 2,048-token step against
+# 504.39 MiB at 1,024. The corpora are cut to the window the boot ledger reports,
+# which is what the pool raise is for (frank-archon.env: HOST_POOL_MIB); a narrow
+# step was never the fix, it was the symptom's disguise.
+#
+# usage: scripts/run-qwen38-bf16.sh kldref|calib|score RAD OUT [--dry-run] [--image REF]
+#   (RAD and OUT are score's two positionals: RAD is a container under the models
+#    dir, OUT the report path whose directory is created if missing)
+#   --dry-run   print the exact `podman run` argv, one shell-quoted argument a
+#               line, and exit 0: the service is NOT stopped, nothing is
+#               mounted, nothing is written — the offline test seam. It still
+#               validates the argv's own shape (paths through the /models mount,
+#               a step size that is a number), because an argv that could not
+#               boot is not worth printing.
+#   --image REF override the digest pin (a probe boot, or the test seam)
+#
+# knobs: FRANK_QWEN38_MODELS (the dir mounted at /models), FRANK_QWEN38_RAD (the
+#        bf16 container, kldref only), FRANK_QWEN38_CAMPAIGN (holds kld-ref/ and
+#        calib/), FRANK_QWEN38_KLD_CORPUS, FRANK_QWEN38_TPL (the template copy
+#        the container reads through /models), FRANK_QWEN38_LOG_DIR, and the step
+#        and slot counts inherited as FRANK_ARCHON_MAX_BATCHED_TOKENS /
+#        FRANK_ARCHON_MAX_SEQS — no per-mode literals (the audit law: a mode that
+#        carries its own step measures a shape nothing serves; calib is exempt, see
+#        Q38_CALIB_MNBT / Q38_CALIB_SEQS below — its output is never compared) —
+#        and FRANK_QWEN38_MAX_MODEL_LEN — the window every mode boots at (32,768;
+#        see the corpus census above, and raise it together with the corpus).
+# kldref refuses (exit 2) when the longest record of the corpus it is about to
+# read is longer than MAX_MODEL_LEN x 3 characters — chars/3 being the
+# pessimistic token:char ratio, so the script says no before the engine finds out
+# inside a drained window, after the checkpoint work is already done. score
+# refuses when the reference dir is absent, when RAD is not on disk or not under
+# the models mount, or when OUT already exists: the trash law means a second
+# score over the same name is a deliberate act, and two numbers in one file is
+# how a campaign loses its own history.
+# Engine stdout goes to $LOG_DIR/qwen38_bf16_<mode>.log — the boot ledger is the
+# record of what the resolver actually gave the cards; the journal never sees it.
+# exit: the engine's code; 2 = refused before the service was touched
+set -uo pipefail
+
+SCRIPT_DIR="$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)"
+# shellcheck disable=SC1091
+source "$SCRIPT_DIR/frank-archon.env"
+IMAGE="${FRANK_ARCHON_IMAGE:?FRANK_ARCHON_IMAGE must name the radiance image}"
+# THE CALIBRATION PIN: the LATEST radiance, pinned by digest because a tag is a
+# moving target and a tap must name the producer that wrote it. `:latest` on
+# 2026-10-30 resolves to v1.14.0 (revision 4a61d7296bc5), and on that image a
+# calib boot prints the arming line — "recorded passes: off, the program
+# accumulates calibration Grams (gram_accum)" — where the production pin
+# (1.8.1) prints nothing of the kind: a 17-hour run on 1.8.1 wrote ZERO bytes
+# into the tap dir while /health, /stats and the push all looked healthy
+# (measured 2026-10-30). The tap code is identical in both trees' sources, so
+# the shipped 1.8.1 build is what drops it; do not spend another window on it.
+# Any other image must be named deliberately with --image.
+CALIB_IMAGE="${Q38_CALIB_IMAGE:-stilldeadcode/radiance@sha256:69add58f7de1ed2bca2b1e20a4cc75717557df3f73569a178eca5aafe8c44093}"
+IMAGE_EXPLICIT=0
+
+MODE=""
+POS=()
+DRY=0
+while [ $# -gt 0 ]; do
+    case "$1" in
+        kldref|calib|score) MODE="$1" ;;
+        --dry-run)          DRY=1 ;;
+        --image)            [ "$#" -ge 2 ] || { echo "--image needs a reference" >&2; exit 2; }
+                            IMAGE="$2"; IMAGE_EXPLICIT=1; shift ;;
+        *)
+            # score's two positional arguments, and nothing else may be bare.
+            if [ "$MODE" = score ] && [ "${#POS[@]}" -lt 2 ]; then
+                POS+=("$1")
+            else
+                echo "usage: ${0##*/} kldref|calib|score RAD OUT [--dry-run] [--image REF]  (unknown argument: $1)" >&2
+                exit 2
+            fi ;;
+    esac
+    shift
+done
+if [ -z "$MODE" ]; then
+    echo "usage: ${0##*/} kldref|calib|score RAD OUT [--dry-run] [--image REF]  (no mode given)" >&2
+    exit 2
+fi
+if [ "$MODE" = score ] && [ "${#POS[@]}" -ne 2 ]; then
+    echo "usage: ${0##*/} score RAD OUT  — RAD is the container to score, OUT the report to write (got ${#POS[@]} argument(s))" >&2
+    exit 2
+fi
+
+MODELS="${FRANK_QWEN38_MODELS:-/home/dank/models}"
+CAMPAIGN="${FRANK_QWEN38_CAMPAIGN:-/home/dank/campaign/qwen38-w5nl64}"
+KLD_CORPUS="${FRANK_QWEN38_KLD_CORPUS:-/home/dank/corpora/out/kld.jsonl}"
+RAD="${FRANK_QWEN38_RAD:-$MODELS/rad/qwen38-bf16.rad}"
+TPL="${FRANK_QWEN38_TPL:-$MODELS/tpl.jinja}"
+TEMPLATE="$SCRIPT_DIR/qwen38-froggeric-v22.5.jinja"
+LOG_DIR="${FRANK_QWEN38_LOG_DIR:-/home/dank/logs}"
+LOG="$LOG_DIR/qwen38_bf16_${MODE}.log"
+KLD_REF_DIR="$CAMPAIGN/kld-ref"
+CALIB_DIR="${FRANK_QWEN38_CALIB_DIR:-$CAMPAIGN/calib}"
+C=qwen38-bf16
+SERVICE=frank-archon
+# The card split, taken from the file that owns it instead of a private pair:
+# --host-pool-mib also funds the KV cache (see the header), so hardcoding it
+# here once starved the recorder. The `:?` guards keep an empty env fatal.
+HOST_POOL_MIB="${FRANK_ARCHON_HOST_POOL_MIB:?FRANK_ARCHON_HOST_POOL_MIB must be set}"
+GPU_HEADROOM_MIB="${FRANK_ARCHON_GPU_HEADROOM_MIB:?FRANK_ARCHON_GPU_HEADROOM_MIB must be set}"
+# The slot count, from the same file that owns the card split: production runs
+# four slots and every mode measures at production's geometry. A per-mode
+# literal here is what let calib ask for eight slots and then starve its own
+# cache. The `:?` guard keeps an empty env fatal, as for the card split.
+MAX_SEQS="${FRANK_ARCHON_MAX_SEQS:?FRANK_ARCHON_MAX_SEQS must be set}"
+if ! [[ "$MAX_SEQS" =~ ^[0-9]+$ ]]; then
+    echo "REFUSED: FRANK_ARCHON_MAX_SEQS must be a slot count (got: $MAX_SEQS)" >&2
+    exit 2
+fi
+
+# ONE step size for every mode, quoted from the file that owns it — the audit
+# law for a radiance rig is to inherit the proven number, not to carry a
+# campaign-private value: production's FRANK_ARCHON_MAX_BATCHED_TOKENS. The
+# 1,024 default this script carried was PR #80's answer to a starved cache;
+# production moved to its widest legal step (2,048) on 2026-10-29 and the rig
+# follows. The pool raise that shipped with it (66,560 a rank) bought this
+# container 1.26 MiB of cache (`vram_kv 269.99 -> 271.25 MiB`, window still
+# 7,436 tokens — the 95.37 GiB n-gram table's claim decides the rig's cache,
+# not the pool) and was folded back the same day. The corpora stay cut to the
+# window the boot ledger reports, never the other way round.
+MNBT="${FRANK_ARCHON_MAX_BATCHED_TOKENS:?FRANK_ARCHON_MAX_BATCHED_TOKENS must be set}"
+if ! [[ "$MNBT" =~ ^[0-9]+$ ]]; then
+    echo "REFUSED: FRANK_ARCHON_MAX_BATCHED_TOKENS must be a token count (got: $MNBT)" >&2
+    exit 2
+fi
+
+# Calib's shape is NOT a measuring shape. kldref and score produce KL numbers that are only
+# comparable across boots at an identical shape, which is why they inherit production's step and
+# slots to the letter. The tap's output is layer Hessians, never a number any report compares, so
+# its step and slot count are its own — and they are the tap's dominant cost: a 7,168-token
+# document cut into four 2,048-token steps re-streams each layer's routed expert union four
+# times from the container, so one wide step is worth up to ~4x the read volume. Measured
+# 2026-10-29 at the inherited 2,048/4: prefill 71 tok/s and a 47-hour tap; the probe exists to
+# find out what the wide step is worth before the campaign pays for two days of drained serving.
+# Both default to the inherited value, so nothing moves unless a probe asks it to.
+CALIB_MNBT="${Q38_CALIB_MNBT:-$MNBT}"
+CALIB_SEQS="${Q38_CALIB_SEQS:-$MAX_SEQS}"
+for pair in "Q38_CALIB_MNBT=$CALIB_MNBT" "Q38_CALIB_SEQS=$CALIB_SEQS"; do
+    if ! [[ "${pair#*=}" =~ ^[0-9]+$ ]]; then
+        echo "REFUSED: ${pair%%=*} must be a number (got: ${pair#*=})" >&2
+        exit 2
+    fi
+done
+
+# The window every mode boots at. Over-length is a REFUSAL, not a truncation (the
+# env file says so for production), so this is the number the corpus must fit in.
+MAX_MODEL_LEN="${FRANK_QWEN38_MAX_MODEL_LEN:-32768}"
+if ! [[ "$MAX_MODEL_LEN" =~ ^[0-9]+$ ]]; then
+    echo "REFUSED: FRANK_QWEN38_MAX_MODEL_LEN must be a token count (got: $MAX_MODEL_LEN)" >&2
+    exit 2
+fi
+
+# Where the n-gram table lives. On the bf16 payload it is 95.37 GiB and `ram`
+# PINS it: the calibration boot died at "startup stopped: out of memory" with
+# 230.22 GiB pinned of 246.07 GiB and 8.06 GiB available (measured 2026-10-29),
+# while the reference recording and every score boot had already succeeded with
+# the same pool and slots. The difference that matters is the job: calib only
+# needs the routed weights prefilled, so it reads the table from the container's
+# own file tier and the pin never happens; kldref and score keep production's
+# placement, because their numbers are only comparable when every other term of
+# the shape is identical. FRANK_QWEN38_NGRAM_PLACEMENT overrides either default.
+case "$MODE" in
+    calib) NGRAM_PLACEMENT="${FRANK_QWEN38_NGRAM_PLACEMENT:-disk}" ;;
+    *)     NGRAM_PLACEMENT="${FRANK_QWEN38_NGRAM_PLACEMENT:-ram}" ;;
+esac
+case "$NGRAM_PLACEMENT" in
+    auto|vram|ram|disk) ;;
+    *) echo "REFUSED: FRANK_QWEN38_NGRAM_PLACEMENT must be auto|vram|ram|disk (got: $NGRAM_PLACEMENT)" >&2
+       exit 2 ;;
+esac
+
+# Every container this campaign boots is read through the ro /models mount, so
+# the host path must live under it; the container-side spelling is what goes
+# into the argv. Same rule for the container a score pass opens.
+under_models() {
+    case "$1" in
+        "$MODELS"/*) printf '/models/%s' "${1#"$MODELS"/}" ;;
+        *) return 1 ;;
+    esac
+}
+
+# same_dir A B — the two name one directory, judged only when BOTH exist: a dry
+# run has no fixtures on disk, and two absent paths are not the same place.
+same_dir() {
+    [ -d "$1" ] && [ -d "$2" ] && [ "$(cd "$1" && pwd)" = "$(cd "$2" && pwd)" ]
+}
+
+if [ "$MODE" = score ]; then
+    SCORE_RAD="${POS[0]}"
+    SCORE_OUT="${POS[1]}"
+    OUT_DIR="$(dirname "$SCORE_OUT")"
+    MODEL_IN="$(under_models "$SCORE_RAD")" || {
+        echo "REFUSED: container to score $SCORE_RAD is not under the mounted models dir $MODELS — the engine cannot read it" >&2
+        exit 2
+    }
+else
+    MODEL_IN="$(under_models "$RAD")" || {
+        echo "REFUSED: reference container $RAD is not under the mounted models dir $MODELS — the engine cannot read it" >&2
+        exit 2
+    }
+fi
+
+# The measuring shape. Identical for kldref and score by construction — this
+# array is the only place the campaign's KL numbers are defined.
+ARGS=(--tp 2 --placement expert_tiered --weights-disk-tier
+      --host-pool-mib "$HOST_POOL_MIB" --gpu-headroom-mib "$GPU_HEADROOM_MIB"
+      --deterministic --num-speculative-tokens 0
+      --kv-cache-dtype fp8 --max-model-len "$MAX_MODEL_LEN"
+      --ngram-placement "$NGRAM_PLACEMENT"
+      --override-chat-template /models/tpl.jinja)
+MOUNTS=(-v "$MODELS:/models:ro")
+TAPENV=()
+case "$MODE" in
+  kldref)
+    # Recording is prefill-only over four slots at the campaign's single step
+    # size, which is frank-archon.env's FRANK_ARCHON_MAX_BATCHED_TOKENS (2,048) —
+    # see the header: the file tier's ceiling and the slot count both come from
+    # the file that owns them. Corpus read-only, record dir rw.
+    ARGS+=(--model "$MODEL_IN"
+           --kld-record /data/kld-ref --kld-corpus /data/kld.jsonl
+           --max-num-seqs "$MAX_SEQS" --max-num-batched-tokens "$MNBT")
+    MOUNTS+=(-v "$KLD_CORPUS:/data/kld.jsonl:ro"
+             -v "$KLD_REF_DIR:/data/kld-ref")
+    ;;
+  score)
+    # The same argv with three words changed: which container, and that this
+    # time the engine reads a recording and writes a report instead of the
+    # reverse. The reference dir mounts READ-ONLY — a scoring pass must not be
+    # able to touch the zero point — and the report gets its own writable bind.
+    if [ "$OUT_DIR" = "$KLD_REF_DIR" ] || same_dir "$OUT_DIR" "$KLD_REF_DIR" \
+       || [ "$(realpath -m "$OUT_DIR")" = "$(realpath -m "$KLD_REF_DIR")" ]; then
+        echo "REFUSED: score output $SCORE_OUT is inside the reference dir $KLD_REF_DIR, which this mode mounts read-only" >&2
+        exit 2
+    fi
+    ARGS+=(--model "$MODEL_IN"
+           --kld-ref /data/kld-ref --kld-out "/data/out/$(basename "$SCORE_OUT")"
+           --max-num-seqs "$MAX_SEQS" --max-num-batched-tokens "$MNBT")
+    MOUNTS+=(-v "$KLD_REF_DIR:/data/kld-ref:ro"
+             -v "$OUT_DIR:/data/out")
+    ;;
+  calib)
+    # Calibration rides the shared step like every other mode. The old
+    # exemption — eight slots at MNBT 65,536 for "free throughput", because
+    # "a disk-tier step costs the same at 1 token as at 64k" — is false: the
+    # step's activation arena is billed on the card, not on the disk. The
+    # engine's own ledger for this container: at MNBT 1,024 the arena is
+    # 504 MiB and the card claims 30.87 GiB (weights 28.90 GiB of it); at
+    # 65,536 the arena is 10.65 GiB (fixed region 1.25 GiB), the claim falls
+    # to 16.85 GiB, the experts shrink to 10.98 GiB and the KV to 271 MiB —
+    # one slot's worth of cache for eight slots, and the boot dies at
+    # "startup stopped: out of memory" (measured 2026-10-29). A wide step
+    # buys throughput with the expert slab, so price it like any other
+    # VRAM knob. The slots follow frank-archon.env's FRANK_ARCHON_MAX_SEQS
+    # through MAX_SEQS, exactly as kldref and score do.
+    [ "$IMAGE_EXPLICIT" = 1 ] || IMAGE="$CALIB_IMAGE"
+    # ONE TAP SEAL WRITES ~40 GiB. Measured 2026-10-30: a 8,192-token floor wrote
+    # 39.4 GiB into the tap dir within seconds; pointed at a 37 GiB root it filled
+    # the filesystem to zero, after which every write came back short ("calibration
+    # cannot write", "write was short") and the system itself began refusing
+    # writes. The room is demanded up front, on the tap's own filesystem.
+    if [ "$DRY" != 1 ]; then
+        avail_mib=$(df --output=avail -B 1M "$CALIB_DIR" 2>/dev/null | tail -1 | tr -d ' ')
+        [[ "$avail_mib" =~ ^[0-9]+$ ]] || avail_mib=0
+        need_mib="${Q38_CALIB_MIN_FREE_MIB:-61440}"
+        if [ "$avail_mib" -lt "$need_mib" ]; then
+            echo "REFUSED: tap dir $CALIB_DIR has ${avail_mib} MiB free; one seal writes ~40 GiB and this run needs ${need_mib} MiB (Q38_CALIB_MIN_FREE_MIB overrides)" >&2
+            exit 2
+        fi
+    fi
+    ARGS+=(--model "$MODEL_IN" --port 7770
+           --max-num-seqs "$CALIB_SEQS" --max-num-batched-tokens "$CALIB_MNBT")
+    # The tap is only real if this directory exists and the engine can write it:
+    # without the check a typo'd CALIB_DIR auto-creates a root-owned empty dir in
+    # whatever path was given, and the run burns hours serving prompts into a tap
+    # nobody reads. RADIANCE_CALIB_TOKENS is the whole corpus' floor, not a round's.
+    [ "$DRY" = 1 ] || [ -w "$CALIB_DIR" ] || {
+        echo "REFUSED: calib tap dir $CALIB_DIR is not writable (mkdir it, chown it, then re-run)" >&2
+        exit 2
+    }
+    MOUNTS+=(-v "$CALIB_DIR:/data/calib")
+    TAPENV=(-e RADIANCE_CALIB_DIR=/data/calib
+            -e "RADIANCE_CALIB_TOKENS=${FRANK_QWEN38_CALIB_TOKENS:-12000000}")
+    # The tap's own authorization. A pusher must see this marker before it sends
+    # a prompt (calib-push.py --tap-marker), it names the container so a push
+    # aimed anywhere else refuses, and a boot that fails removes it again below.
+    # Measured 2026-10-29: with no marker, the chain pushed 4 rounds x 10M tokens
+    # into RESTORED PRODUCTION after the rig's boot died, and nothing recorded.
+    if [ "$DRY" = 1 ]; then
+        :
+    else
+        printf 'container=%s\nimage=%s\nmode=calib\nstarted=%s\npid=%s\n' \
+            "$C" "$IMAGE" "$(date -Is)" "$$" > "$CALIB_DIR/.rig-boot"
+    fi
+    ;;
+esac
+
+CMD=(podman run --rm --name "$C"
+     --network host --init
+     --security-opt seccomp=unconfined
+     --ulimit memlock=-1:-1
+     --device /dev/kfd --device /dev/dri
+     "${MOUNTS[@]}" "${TAPENV[@]}"
+     "$IMAGE" "${ARGS[@]}")
+
+# ---- gate: the window must hold the corpus (kldref alone can meet it) --------
+# score reads a recording instead of a corpus, and every arm declares at the
+# checkpoint's training context, so neither can meet a sequence too long to boot.
+# Measured on the raw JSON line — the pessimistic direction, and the same figure
+# the census in the header quotes. It runs on a dry run too, because a refusal
+# this cheap should never have to wait for the drained window to be discovered.
+if [ "$MODE" = kldref ] && [ -f "$KLD_CORPUS" ]; then
+    read -r REC_NO REC_CHARS < <(awk '{ if (length($0) > best) { best = length($0); where = NR } } END { print where + 0, best + 0 }' "$KLD_CORPUS")
+    LIMIT_CHARS=$(( MAX_MODEL_LEN * 3 ))
+    if [ "$REC_CHARS" -gt "$LIMIT_CHARS" ]; then
+        echo "REFUSED: record $REC_NO of $KLD_CORPUS is $REC_CHARS chars, past the $LIMIT_CHARS a $MAX_MODEL_LEN-token window promises at the pessimistic chars/3 — raise FRANK_QWEN38_MAX_MODEL_LEN, and the container's declared --max-ctx with it, before booting the reference" >&2
+        exit 2
+    fi
+    echo "corpus fits: longest record $REC_CHARS chars (number $REC_NO) inside the $MAX_MODEL_LEN-token window (limit $LIMIT_CHARS chars)"
+fi
+
+if [ "$DRY" = 1 ]; then
+    echo "dry run: $MODE — the production unit is left running, nothing is mounted, nothing is written"
+    printf '%q\n' "${CMD[@]}"
+    exit 0
+fi
+
+# ---- gates that only a real run needs ---------------------------------------
+if [ "$MODE" = score ]; then
+    if [ ! -d "$KLD_REF_DIR" ]; then
+        echo "REFUSED: no KLD reference at $KLD_REF_DIR — record it first in a drained window: scripts/run-qwen38-bf16.sh kldref" >&2
+        exit 2
+    fi
+    if [ ! -f "$SCORE_RAD" ]; then
+        echo "REFUSED: no container to score at $SCORE_RAD" >&2
+        exit 2
+    fi
+    if [ -e "$SCORE_OUT" ]; then
+        echo "REFUSED: $SCORE_OUT already exists — one report per name; move the old one to trash before re-scoring" >&2
+        exit 2
+    fi
+    mkdir -p "$OUT_DIR"
+else
+    if [ ! -f "$RAD" ]; then
+        echo "REFUSED: no reference container at $RAD — build it first: scripts/build-qwen38-bf16-rad.sh" >&2
+        exit 2
+    fi
+    [ "$MODE" = kldref ] && [ ! -f "$KLD_CORPUS" ] && {
+        echo "REFUSED: no KLD corpus at $KLD_CORPUS — the record would be empty" >&2
+        exit 2
+    }
+fi
+if [ ! -f "$TEMPLATE" ]; then
+    echo "REFUSED: no chat template at $TEMPLATE" >&2
+    exit 2
+fi
+mkdir -p "$LOG_DIR"
+# The prod template, staged where the ro /models mount reaches it: a reference
+# recorded with the container's baked template scores a different tokenisation
+# than the arms served with froggeric's, and the KL would be measuring that.
+cp -f "$TEMPLATE" "$TPL" || {
+    echo "REFUSED: cannot stage the chat template at $TPL (the container reads it through /models)" >&2
+    exit 2
+}
+[ "$MODE" = kldref ] && mkdir -p "$KLD_REF_DIR"
+[ "$MODE" = calib ]  && mkdir -p "$CALIB_DIR"
+
+# Stop the campaign container BEFORE handing the cards back: a killed run leaves
+# its conmon alive (--rm only fires when podman itself exits), and production
+# then boots beside a container still holding the GPUs — measured 2026-10-29,
+# where the service reported `active` while its engine starved on the rig's VRAM.
+restore() {
+    podman rm -f "$C" >/dev/null 2>&1 || true
+    # Removing the container does NOT kill its engine. Measured 2026-10-30: a rig
+    # radiance kept /dev/kfd after `podman rm -f`, production's
+    # ar_oneshot_2rank_wht6 init then failed `device error`, the engine exited 0,
+    # and Restart=always looped it 100 times with /health never answering. The
+    # service is stopped for the whole run, so any radiance still holding the
+    # device here is an orphan — kill it BY PID (pkill -f matches the caller's
+    # own command line, which is how a session gets killed instead of the orphan).
+    for p in $(fuser /dev/kfd 2>/dev/null); do
+        if [ "$(cat "/proc/$p/comm" 2>/dev/null)" = radiance ]; then
+            echo "restore: killing orphaned radiance pid $p still holding /dev/kfd" >&2
+            kill -9 "$p" 2>/dev/null || true
+        fi
+    done
+    sudo systemctl start "$SERVICE" >/dev/null 2>&1 || true
+}
+trap restore EXIT
+echo "run-qwen38-bf16 $MODE: stopping $SERVICE ($(date -Is)); log: $LOG"
+sudo systemctl stop "$SERVICE" >/dev/null 2>&1 || true
+podman rm -f "$C" >/dev/null 2>&1 || true
+
+"${CMD[@]}" 2>&1 | tee "$LOG"
+rc=${PIPESTATUS[0]}
+# A boot that failed must not leave the tap marked as rig-authorized: the marker
+# is what calib-push.py trusts, and the point of it is that a dead rig can never
+# be pushed into again.
+if [ "$MODE" = calib ] && [ "$rc" -ne 0 ] && [ -f "$CALIB_DIR/.rig-boot" ]; then
+    mv -f "$CALIB_DIR/.rig-boot" "$CALIB_DIR/.rig-boot.failed-$(date +%Y%m%d-%H%M%S)" 2>/dev/null || true
+    echo "the rig boot failed (rc=$rc): the tap marker is retired, so no pusher can take this tap for a live rig"
+fi
+echo "run-qwen38-bf16 $MODE exited rc=$rc; $SERVICE restored; log: $LOG"
+exit "$rc"

scripts/serve-frank-dragoon.sh

@@ -1,5 +1,5 @@
 #!/usr/bin/env bash
-# serve-frank-dragoon.sh — frank-dragoon endpoint: halogen-flash-server 0.16.2
+# serve-frank-dragoon.sh — frank-dragoon endpoint: halogen-flash-server 0.18.20
 # (closed-source engine, image ghcr.io/peonist-ai/halogen-flash-server,
 # digest-pinned in frank-dragoon.env) serving Quat3rnion's NATIVE v2
 # conversion (2026-10-11) of the abliterated Qwen3.8-Flash-Next on Dankclaw's

scripts/serve-frank-zealot.sh

@@ -1,7 +1,7 @@
 #!/usr/bin/env bash
 # Serve prism-ml/Ternary-Bonsai-2-9B-gguf (PQ2_0 ternary quant WITH the MTP head
 # in the file) as frank-zealot on Dankclaw's RTX 3090 via llama.cpp (PrismML
-# vendor build 10777, CUDA sm_86).
+# vendor build 12118 (0.7.0-dev, commit 024474448), CUDA sm_86).
 #
 # 2026-10-18 (Frank): Bonsai-2 27B PQ2_0 → the 9B MTP export, then the same
 # evening TWO 262,144-token conversations (--ctx-size 524288 --parallel 2) over
@@ -78,7 +78,7 @@ if [[ -f "$ENV_FILE" ]]; then
 fi
 
 # --- defaults (used when the env file does not set them) -------------------
-: "${LLAMA_BIN:=/home/dank/GitPackages/llama-prism-b10777/llama-server}"
+: "${LLAMA_BIN:=/home/dank/GitPackages/llama-prism-0244744/bin/llama-server}"
 : "${MODEL_FILE:=/home/dank/models/Bonsai2-9B-MTP/Ternary-Bonsai-2-9B-PQ2_0-MTP-Q8_0.gguf}"
 : "${ALIAS:=frank-zealot}"
 : "${BIND_ADDR:=0.0.0.0}"
@@ -119,7 +119,7 @@ fi
 # above are what pay for it. The draft cache dtype IS a further knob
 # (--spec-draft-type-k / --spec-draft-type-v, both default f16 — verified in this
 # build's --help) and stays at its default here. Card gate: quantized V alongside
-# MTP needs fork build >= prism-b10776, which is why LLAMA_BIN is the 10777
+# MTP needs fork build >= prism-b10776; LLAMA_BIN is 12118 (b10777 kept as rollback)
 # prebuilt; the type-142 loader lock keeps it pinned regardless.
 # SPEC_TYPE=none is the rollback and strips the whole set (flags move together).
 : "${SPEC_TYPE:=draft-mtp}"

scripts/stack-qwen38-official-fp8.py

@@ -337,8 +337,12 @@ def main():
     if any(meta[shard_ids[i]][4][1] != hd for i in range(n_shards)):
         log("ERROR: PLE shards disagree on HD")
         return 2
-    if hd % 128 != 0 and hd != 160:
+    if hd != 160 and hd % 128 != 0:
-        log(f"WARNING: PLE HD={hd} is neither 160 nor a multiple of 128")
+        # The pack below writes lanes of 128 codes; any other HD would produce a
+        # table that decodes to the wrong rows. The old warning let it continue.
+        log(f"ERROR: PLE HD={hd} is neither 160 nor a multiple of 128, and the "
+            "128-lane pack cannot express it")
+        return 2
 
     # ---- idempotence gate -------------------------------------------------
     stacked_names = [f"{p}.mlp.experts.gate_up_proj" for p in prefixes]
@@ -521,12 +525,16 @@ def verify(stage, prefixes, n_e, shapes, meta, shard_ids, rows, hd,
         }
         for name, (dt, shp) in want.items():
             e = hdr[name]
-            assert e["dtype"] == dt and list(e["shape"]) == shp, name
+            if e["dtype"] != dt or list(e["shape"]) != shp:
-            assert e["data_offsets"][1] - e["data_offsets"][0] == \
+                raise RuntimeError(f"{name}: header is {e['dtype']}{e['shape']}, expected {dt}{shp}")
-                int(np.prod(shp)) * (1 if dt == F8 else 2), name
+            if e["data_offsets"][1] - e["data_offsets"][0] != int(np.prod(shp)) * (1 if dt == F8 else 2):
+                raise RuntimeError(f"{name}: data span is {e['data_offsets']}, shape {shp} needs "
+                                   f"{int(np.prod(shp)) * (1 if dt == F8 else 2)} B")
     _, phdr = parse_header(out_path)
     e = phdr[next(iter(phdr))]
-    assert e["dtype"] == BF16 and list(e["shape"]) == [sum(rows), hd]
+    if e["dtype"] != BF16 or list(e["shape"]) != [sum(rows), hd]:
+        raise RuntimeError(f"PLE table header is {e['dtype']}{e['shape']}, "
+                           f"expected {BF16}[{sum(rows)}, {hd}]")
 
     # sampled PLE rows: recompute and compare bytes exactly
     lut = e4m3_lut()
@@ -546,7 +554,8 @@ def verify(stage, prefixes, n_e, shapes, meta, shard_ids, rows, hd,
                 with open(out_path, "rb") as ofh:
                     ofh.seek(out_off + got_row * hd * 2)
                     got = ofh.read(hd * 2)
-                assert got == want, f"PLE row {got_row} mismatch"
+                if got != want:
+                    raise RuntimeError(f"PLE row {got_row} does not match the decoded source")
             mm.close()
     log("verify: new-file headers, sizes and sampled PLE rows all exact; "
         "expert slabs were byte-checked in-stack")

scripts/test-changed.sh

@@ -94,6 +94,16 @@ for path in "${CHANGED[@]}"; do
         dankserver/.env.example|dankclaw/.env.example)
             add_test tests/env_consistency.bats
             ;;
+        # The SSD-backed metadata view: the installer, the host mount unit, and the
+        # two media units that bind the view rather than the raw array.
+        scripts/media-view.sh|dankserver/host/*)
+            add_test tests/media_overlay.bats
+            add_test tests/dankserver_quadlet.bats
+            add_test tests/deployment_scoping.bats
+            ;;
+        dankserver/jellyfin/quadlet/*|dankserver/plex/quadlet/*)
+            add_test tests/media_overlay.bats
+            ;;
         shared/homepage/*)
             add_test tests/homepage_caddy.bats
             add_test tests/homepage_config.bats
@@ -265,6 +275,13 @@ for path in "${CHANGED[@]}"; do
             add_test tests/voice_pack_unit.bats
             add_test tests/scripts_syntax.bats
             ;;
+        # The KLD corpus re-chunker: its seam is its own (the campaign's
+        # one-token-series invariant is asserted there), so a change to it does
+        # not land on the generic scripts/* fallback below.
+        scripts/rechunk-kld-corpus.py)
+            add_test tests/qwen38_rechunk.bats
+            add_test tests/scripts_syntax.bats
+            ;;
         scripts/*)
             add_test tests/scripts_syntax.bats
             ;;

serverAppConfigs-docs/Architecture/Quadlet Lifecycle.md

@@ -130,10 +130,19 @@ Quickest unblock: `podman compose up -d --force-recreate <svc>`. Durable fix
 
 ## 4. Podman 5.x quadlet v1 generator quirks (verified live 2026-08-28)
 
-- **Hard-fail keys** — the whole unit silently fails to generate
+- **Hard-fail keys abort the WHOLE BATCH, not one unit** (podman 5.7.0,
-  (`systemctl --user cat <unit>` shows "could not be found"): any
+  live 2026-10-21): one unsupported key — e.g. `HostAddr=` — makes the
-  dash-prefixed `[Container]` option (`--shm-size=…` → `unsupported key …
+  generator log `unsupported key … in group 'Container'` plus
-  in group 'Container'`); `Command=` is unsupported, use `Exec=`.
+  `processing encountered some errors` and generate NOTHING for that
+  manager. The running containers keep going as orphans, so the failure
+  first appears on the next deploy as `Unit <x>.service not found` for a
+  service that is visibly up. Other offenders: any dash-prefixed
+  `[Container]` option (`--shm-size=…`), `Command=` (use `Exec=`).
+  The seam `tests/quadlet_image_pins.bats` fails the build on any
+  generator-rejected key in any host's units.
+  Also: an image whose ENTRYPOINT is the binary takes FLAGS ONLY in
+  `Exec=` — repeating the binary makes it a positional arg
+  (`unexpected /bin/blackbox_exporter`, live 2026-10-21).
 - **HealthCmd space-splitting:** the v1 generator splits `HealthCmd` on
   spaces into the healthcheck argv; a quoted probe
   `/bin/sh -c "pg_isready …"` lands as argv token `"pg_isready` →
@@ -148,6 +157,16 @@ Quickest unblock: `podman compose up -d --force-recreate <svc>`. Durable fix
   "unsupported key"`, `systemctl --user cat probe.service` (shows the
   generated ExecStart verbatim), then remove + reload.
 
+## 4a. Image pins (fleet law, 2026-10-21)
+
+Every registry image in a quadlet is digest-pinned (`tag@sha256:…`);
+locally built images (`localhost/*`, `franksdank/*`) are exempt — their
+build is the pin. A rolling tag lets a rebuild move under a running
+service with nobody approving it, and it makes a CVE audit ungradeable.
+Gate: `tests/quadlet_image_pins.bats`. Where upstream publishes no version
+tags (copyparty-ac stops at 1.19.15), pin `:latest@sha256:` and name the
+version the config label carries in the comment beside it.
+
 ## 5. Rootless Quadlet log retrieval
 
 Step 0 — establish ownership: `systemctl --user status <name>.service`
@@ -176,8 +195,9 @@ deploy.sh, never a live env poke.
    check every namespace (`sudo podman ps -a`) and native units
    (`systemctl list-units`) before declaring a service absent. Compare
    compose mount sources against the real filesystem — compose files name
-   phantom paths (e.g. `/mnt/media/*` vs the real LUKS array at
+   phantom paths (the legacy `dankserver/compose.yaml` still names `/opt/jellyfin/*`
-   `/mnt/entropy`). Fix compose through git first, deploy second.
+   for a service whose roots moved to `/home/dank/Containers/jellyfin`). Fix
+   compose through git first, deploy second.
 2. **Pre-create bind-mount sources** (`sudo mkdir -p` + `chown 1000:1000`)
    or creation fails `statfs …: no such file or directory`.
 3. **Staged bring-up:** infra first (`up -d mariadb redis`), poll

serverAppConfigs-docs/Dankclaw/ComfyUI Qwen-Image Workflows.md

@@ -110,9 +110,9 @@ only `models`, `input`, `output`, `custom_nodes` and the healthcheck file
 
 ## Recommended additions
 
-Quantized preferred, sized for the 8060S iGPU (GTT-backed — `/system_stats`
+Quantized preferred. The iGPU's GTT aperture is not free RAM beside
-reports 122.4 GB of device VRAM on `cuda:0`, `curl http://127.0.0.1:8188/system_stats`
+frank-dragoon — size every pick against *What actually fits beside
-2026-10-18; the RTX 3090 is frank-zealot's). All repos verified 2026-10-05 and
+frank-dragoon* below (the RTX 3090 is frank-zealot's). All repos verified 2026-10-05 and
 all four still resolve today (`huggingface.co/lightx2v/Qwen-Image-Lightning`
 `huggingface.co/unsloth/Qwen-Image-Edit-2511-GGUF`
 `huggingface.co/Comfy-Org/Qwen-Image-DiffSynth-ControlNets`
@@ -133,8 +133,34 @@ sha256 checks) or a plain `hf download` with the token in the environment —
 never on the command line.
 
 Skip list: the original `city96/Qwen-Image-gguf` conversions (superseded by
-the 2.1 files already on disk), and any BF16 of #2/#4 (the iGPU does not need
+the 2.1 files already on disk), and any BF16 of #2/#4 — they double the disk
-them at GTT capacity, and they double the disk bill).
+bill, and see *What actually fits beside frank-dragoon* for why raw GTT size
+is not the real budget.
+
+## What actually fits beside frank-dragoon
+
+The 122 GB `/system_stats` figure is the GTT **aperture**, not free memory: on
+an APU the aperture and `MemAvailable` are the same physical RAM, and
+frank-dragoon pins tens of GiB of it. Measured 2026-10-21 with frank-dragoon
+live: `MemAvailable` 15.3 GiB, iGPU busy 1.75 GiB — ComfyUI itself idles at
+~10 MB, so the budget for a generation run is roughly what `free` says.
+
+Weights must land inside that budget with slack for compute buffers and
+activations (the mmproj-class lesson: price a model at ~1.5× its file set):
+
+| Pipeline (weights) | Fits beside frank-dragoon? |
+|---|---|
+| Qwen-Image 2.1 int8_convrot + w4a8 + bf16 VAE (~14.2 GB) — the two proven workflows | **No** — thrash-or-fail; do not queue it while the endpoint serves |
+| GGUF Q8_0 + w4a8 + small VAE (~13.3 GB) | No, same reason |
+| Any BF16 combination | No |
+| SDXL-Turbo / SD 1.5-class (~2–7 GB) | Yes, with real headroom |
+
+**The law: one process owns the memory — same rule as the 3090 bench law.**
+For a Qwen-Image render: stop and mask frank-dragoon
+(`sudo systemctl stop frank-dragoon` + `mask` — the mask stops
+`Restart=always` from firing mid-render), generate, unmask, verify
+`curl 192.168.0.143:7770/health`. The proven workflows above then run with
+the whole machine. Do not treat `/system_stats`'s 122 GB as an invitation.
 
 ## See Also
 

serverAppConfigs-docs/Dankclaw/ComfyUI.md

@@ -129,6 +129,12 @@ bind-mounted to `/usr/local/bin/comfyui-healthcheck.py` and run as `HealthCmd`
 - Do not let pip reinstall the torch family. The Containerfile filters it out on purpose.
 - The base image is about 32 GiB (`podman image inspect localhost/comfyui-rocm:gfx1151` → 33,765,227,471 B ≈ 31.4 GiB, 2026-10-18). A code update means a rebuild, because the code is baked in.
 - The empty-models note from the 2026-09-16 era is retired: `/home/dank/comfyui-data/models` now holds the Qwen-Image 2.1 payload — three `diffusion_models`, three `text_encoders`, two VAEs (~35 GB, `ls -l` 2026-10-18), listed in [[ComfyUI Qwen-Image Workflows]]. New model classes (LoRA, ControlNet) still need their directories created first: `models/lora` and `models/controlnet` do not exist yet.
+- **The 122 GB `/system_stats` VRAM figure is the GTT aperture, not free
+  memory.** On the APU it is system RAM, and frank-dragoon pins most of it
+  (`MemAvailable` 15.3 GiB with the endpoint live, measured 2026-10-21). A
+  Qwen-Image-class render beside the running endpoint is thrash-or-fail; the
+  stop-mask-render-verify procedure and the fit table are in
+  [[ComfyUI Qwen-Image Workflows]] → What actually fits beside frank-dragoon.
 
 ## See Also
 

serverAppConfigs-docs/Dankclaw/OpenWebUI.md

@@ -34,15 +34,16 @@ https://github.com/open-webui/open-webui
 The picker is built from the `model` table in `webui.db` (one row per model)
 plus the `model_ids` of each **enabled** OpenAI connection, NOT from live
 connection discovery — so the picker shows exactly what is seeded there.
-As of 2026-10-07 the picker exposes **exactly one model: `frank-zealot`** *(unverified
+Since 2026-10-21 the picker exposes **all three fleet models to every
-2026-10-18: `webui.db` is the authority; the fleet itself serves all three ids
+account**: `frank-dragoon`, `frank-archon`, `frank-zealot` (per Frank: open
-`frank-dragoon`/`frank-zealot`/`frank-archon`, per the cards in
+`frank-archon` to everyone; the other two were already public). Proven with
-`shared/homepage/services.json`)*.
+a `role=user` token: `/api/models` lists all three ids and a chat turn to
+`frank-archon` answers end-to-end.
 
-| Connection (`openai.api_configs` idx) | URL as recorded 2026-10-09 | Enabled | `model_ids` | Endpoint page |
+| Connection (`openai.api_configs` idx) | `openai.api_base_urls` (live 2026-10-21) | Enabled | `model_ids` | Endpoint page |
 |---|---|---|---|---|
-| 0 | `http://host.containers.internal:7770/v1` | no (2026-10-07) | `frank-dragoon` | [[frank-dragoon]] |
+| 0 | `http://host.containers.internal:7770/v1` | yes | `frank-dragoon` | [[frank-dragoon]] |
-| 1 | `http://192.168.0.235:7770/v1` | no (2026-10-07) | `frank-archon` | [[frank-archon]] |
+| 1 | `http://192.168.0.235:7770/v1` | yes | `frank-archon` | [[frank-archon]] |
 | 2 | `http://host.containers.internal:7771/v1` | yes | `frank-zealot` | [[frank-zealot]] |
 
 The URLs in the first column are the DB's recorded values and predate the 2026-08-22
@@ -50,9 +51,10 @@ bridge move; on this network they are wrong — see the reachability gotcha belo
 is the corrected law. The picker row for `frank-zealot` was re-keyed on 2026-10-18 after
 that day's payload swap (see [[frank-zealot]] for what was deployed and when).
 
-The `model` table carries exactly two decoration rows, `frank-zealot` and
+The `model` table carries three decoration rows — `frank-zealot`,
-`frank-dragoon`. Both the context length and the description live **inside
+`frank-dragoon`, `frank-archon` (200,000 context, added 2026-10-21).
-JSON columns** — `params` holds `{"context": {"length": …}}` and `meta` holds
+Both the context length and the description live **inside JSON columns** —
+`params` holds `{"context": {"length": …}}` and `meta` holds
 `{"description": …}`; there is no `context_length` or `description` column, so
 `sqlite3` one-liners against those names fail. `params.context.length` is a copy
 of the endpoint's per-slot window, so the authoritative number lives on the
@@ -380,6 +382,20 @@ neither.
   name: /selected model/i })`) and the welcome text "How can I help you
   today?"; that snapshot is also the cheapest proof the picker is populated.
 
+- **2026-10-21 — picker opened to all three models, for everyone.** Per
+  Frank: `frank-archon` accessible to everyone. Three-leg repair on
+  `webui.db` (snapshot `webui.db.bak-20261021` first): connection idx 1
+  `enable: true`, a `model` row `frank-archon` (params
+  `{"context":{"length":200000}}`, meta description from the endpoint card),
+  and an `access_grant` row `(resource_type='model', resource_id='frank-archon',
+  principal_id='*', 'read')` — `principal_id='*'` is the public grant, not a
+  role row. Verified as a plain `role=user` token: `/api/models` lists all
+  three ids, chat to `frank-archon` answers. Token-mint gotchas: the backend
+  package imports only with cwd `/app/backend`, and `WEBUI_SECRET_KEY` must
+  come from **PID 1's environ** (`podman exec` sessions do not inherit the
+  app process env; the `.webui_secret_key` file happens to equal it — use it
+  explicitly, `podman exec -e`).
+
 ## See Also
 
 - [[Dankclaw Stack]]

serverAppConfigs-docs/Dankserver/Jellyfin.md

@@ -1,9 +1,9 @@
 ---
 type: reference
 title: Jellyfin
-description: "Jellyfin on Dankserver: quadlet unit, entropy-volume data layout, ports, client access, hardware-transcode status."
+description: "Jellyfin on Dankserver: quadlet unit, overlay metadata view layout, ports, access, hardware-transcode status."
 tags: [jellyfin, dankserver, media-server]
-timestamp: 2026-09-03T00:00:00Z
+timestamp: 2026-10-30T00:00:00Z
 ---
 
 # Jellyfin
@@ -26,7 +26,7 @@ public vhost `jelly.franksdank.cloud` behind the [[Anubis]] guard chain →
 | Image | `ghcr.io/jellyfin/jellyfin:latest` |
 | Network | `dankserver_appnet`; publishes 8096/tcp, 8920/tcp, 7359/udp, 1900/udp |
 | Data roots | `/home/dank/Containers/jellyfin/config:/config`, `/home/dank/Containers/jellyfin/cache:/cache` |
-| Media | `Volume=/mnt/entropy:/data:ro` — the whole [[Entropy Volume]] mounted read-only; nothing automated writes to it |
+| Media | `Volume=/var/lib/media-meta/view:/data` — the [[Media Metadata View]] overlay, **writable**; sidecars land on the NVMe, the [[Entropy Volume]] array underneath stays read-only and untouched |
 | GPU | none — no `AddDevice`; CPU decode only (jellyfin-ffmpeg path) |
 | Env quirks | `DOTNET_SYSTEM_IO_DISABLEFILELOCKING=true`, `LD_PRELOAD=libjemalloc.so.2`, `MALLOC_TRIM_THRESHOLD_=131072` kept verbatim from the compose-era container (zero-drift cutover) |
 | Health | `curl -f http://localhost:8096/health` every 30 s |
@@ -36,11 +36,23 @@ public vhost `jelly.franksdank.cloud` behind the [[Anubis]] guard chain →
 ```
 /home/dank/Containers/jellyfin/config   -> /config   (JELLYFIN_DATA_DIR)
 /home/dank/Containers/jellyfin/cache    -> /cache    (JELLYFIN_CACHE_DIR, XDG_CACHE_HOME)
-/mnt/entropy                            -> /data     (read-only media volume)
+/var/lib/media-meta/view                -> /data     (the SSD-backed overlay view)
 ```
 
-Library folders point at `/data` paths; because the mount uses the same
+Library folders point at `/data` paths, which are the view's paths: stable
-absolute path as on the host, library paths survive container recreation.
+across container recreation, and identical to the raw array layout below it.
+
+`Local metadata` — save metadata to media folders — is enabled, and it works
+because `/data` is the [[Media Metadata View]] rather than the read-only array:
+`movie.nfo`, `tvshow.nfo`, `episode.nfo` and artwork are written once, on the
+NVMe upper layer, and [[Plex]] reads the same files. Jellyfin is the fleet's
+only metadata writer.
+
+**Failure signature:** `System.IO.IOException: Read-only file system :
+'/data/Movies/<film>/movie.nfo'` means the container is binding the raw array
+instead of the view — the view is unmounted or the quadlet drifted. That used to
+be the expected state (pre-view, every item update logged it); today it is a
+fault. Run `scripts/media-view.sh status`.
 
 ## Ports & access
 

serverAppConfigs-docs/Dankserver/Media Metadata View.md

@@ -0,0 +1,117 @@
+---
+type: reference
+title: Media Metadata View
+description: "The SSD-backed overlay the two media servers read and write: shape, guards, commands, laws, rollback."
+tags: [dankserver, media-server, jellyfin, plex, storage]
+timestamp: 2026-10-30T00:00:00Z
+---
+
+# Media Metadata View
+
+`/var/lib/media-meta/view` is an OverlayFS merge of the media library: the RAID6
+array below, a directory on the NVMe above. It exists so [[Jellyfin]] can write
+Kodi-format sidecars (`movie.nfo`, `tvshow.nfo`, `episode.nfo`, artwork) that
+[[Plex]] can read. Sidecar metadata is only useful sitting next to the media
+file, and the array is read-only and slow — so the media bytes stay there and the
+*writes* land on SSD. One tree to both servers, one copy of the media.
+
+The mount point is not a volume and holds nothing while unmounted. It sits inside
+`/var/lib/media-meta` so that layers, kernel bookkeeping, the reap trash and the
+merged door are one directory — and so the mount table grows no new `/mnt` entry
+next to the array.
+
+## Shape
+
+| Item | Value |
+|---|---|
+| Lower layer | `/mnt/entropy` — the RAID6 array, source of truth, never written |
+| Upper layer | `/var/lib/media-meta/upper` — NVMe; every write lands here |
+| Work dir | `/var/lib/media-meta/work` — overlayfs internals, same filesystem |
+| Mount point | `/var/lib/media-meta/view` — the merged tree both servers bind |
+| Unit (repo) | `dankserver/host/systemd/media-meta-view.mount` — the source of truth |
+| Unit (host) | `var-lib-media\x2dmeta-view.mount` — systemd names a mount unit after its escaped path (`-` becomes `\x2d`); `media-view.sh install` writes the repo file under that name |
+| Installer | `scripts/media-view.sh` — `plan` / `install` / `verify` / `status` / `reap` |
+| Written by | Jellyfin only (Plex binds read-only); nothing else in the fleet touches it |
+
+Both consumers guard the view with
+`UnitConfigItem=ConditionPathIsMountPoint=/var/lib/media-meta/view`, and
+`scripts/deploy.sh dankserver --service jellyfin|plex` calls `media-view.sh
+install` + `verify` before it restarts either container.
+
+| Consumer | Bind | Mode | Why that path |
+|---|---|---|---|
+| [[Jellyfin]] | `/var/lib/media-meta/view:/data` | rw | It is the metadata writer; its sidecars become SSD bytes |
+| [[Plex]] | `/var/lib/media-meta/view:/mnt/entropy` | ro | Every path in the PMS database was recorded as `/mnt/entropy/...`; only the HOST side of the bind moved, so its database still resolves |
+
+Putting the sidecars inside Plex's reach is not the same as making Plex use them:
+a library parses `.nfo` only with its *Local media information* agent enabled, and
+this change touches no library setting.
+
+## Commands
+
+```bash
+scripts/media-view.sh plan              # print the resolved shape, touch nothing
+sudo scripts/media-view.sh install      # create dirs, install the unit, mount, enable
+scripts/media-view.sh verify            # is the view really the overlay? (deploy gate)
+scripts/media-view.sh status            # mount options + upper-layer size/file count
+scripts/media-view.sh reap              # list ghost metadata (media moved, sidecars left)
+sudo scripts/media-view.sh reap --apply # move ghosts to the SSD trash, never delete
+```
+
+`--upper-base DIR` and `--lower DIR` relocate the layers for `plan` and `reap`;
+the defaults are the host values above.
+
+## Laws
+
+- **The upper layer must sit on non-rotational storage.** `install` and `verify`
+  refuse via `lsblk -dno ROTA` if the upper base lives on the array: an overlay
+  whose upper is the spinners is the original problem with extra steps, and it
+  silently turns every metadata write back into a seek.
+- **Never mount the view over an unmounted lower.** `RequiresMountsFor=/mnt/entropy`
+  plus the container `ConditionPathIsMountPoint` guards: an empty `lowerdir`
+  presents an *empty library* as truth, and a media scan would then clear the
+  database of everything. systemd does not create a mount point, so `install`
+  creates `view` itself; without it the mount simply fails.
+- **Never delete through the view.** A delete writes an overlayfs whiteout
+  (`.wh.<name>`) in the upper layer, which hides the array's copy without
+  removing it — the file reappears the moment the view is unmounted, and the
+  upper layer now carries a landmine. Deletions belong on the raw path
+  (`/mnt/entropy`), which the relayout tool already treats as the only writer.
+  `reap` refuses to remove whiteouts and only warns about them.
+- **A relayout must be followed by `reap`.** Renames happen on the raw path, so
+  sidecars the view already wrote stay at the old coordinates: a ghost folder
+  the servers browse and find nothing in. `reap` lists upper-layer directories
+  with no counterpart below and moves them (with their subtree) into a dated
+  `.AGENTTRASH/` under the SSD base. It is recursive and file-safe: a sidecar
+  that exists only above is normal, so only *directories* are candidates.
+- **File-watcher events do not cross the overlay.** Plex's NFO agent and
+  Jellyfin's real-time monitor see inotify events for writes that land *in the
+  upper layer*, not for changes on the array underneath. After any relayout or
+  externally-written metadata, trigger a library refresh — see
+  [[Media Relayout]] for the refresh procedure.
+
+## Failure signatures
+
+| Symptom | Meaning |
+|---|---|
+| `jellyfin.service` / `plex.service` show `ConditionPathIsMountPoint=... was not met` and stay inactive | The view is not mounted. Do not start them anyway: `scripts/media-view.sh install`, then `verify`. |
+| Jellyfin log: `Read-only file system : '/data/Movies/…/movie.nfo'` | The container is binding the raw array, not the view — the quadlet drifted back or the view is gone. `media-view.sh status`. |
+| A folder appears in a media server with artwork/`.nfo` but no video | Ghost metadata from a relayout. `media-view.sh reap`. |
+| `verify` prints `refused: …/view is not a mountpoint` | A plain directory is sitting where the overlay belongs. That is the fail-closed guard working; `install` on the box. |
+| `systemctl status` finds no such unit | Look under the escaped name, `var-lib-media\x2dmeta-view.mount`, not the repo file name. |
+
+## Rollback
+
+Revert the commit that pointed the two quadlets at the view and re-deploy both
+services — they bind `/mnt/entropy` read-only again, which is the pre-view shape.
+The NVMe upper layer can stay: nothing reads it once the binds are back, and it
+holds no state either server needs. Unmounting alone
+(`systemctl stop 'var-lib-media\x2dmeta-view.mount'`) is safe while no container
+binds it, and the array is never modified by any of this.
+
+## Links
+
+- [[Jellyfin]] — the metadata writer; its data layout section.
+- [[Plex]] — the read-only consumer; in-place bind-mount history.
+- [[Media Relayout]] — the tool that moves media on the raw path and the reason `reap` exists.
+- [[Entropy Volume]] — the array underneath.

serverAppConfigs-docs/Dankserver/Media Relayout.md

@@ -0,0 +1,281 @@
+---
+type: reference
+title: Media Relayout
+description: "Census, plan and apply the movie library into Jellyfin format, and move TV out of it: the gates, refusals, rollback."
+tags: [dankserver, media-server, jellyfin, plex, procedure]
+timestamp: 2026-10-30T00:00:00Z
+---
+
+# Media Relayout
+
+The movie library on [[Entropy Volume]] arrived as release folders —
+`Some.Movie.2019.1080p.WiKiTaLx.x265-FOO/` with the video inside named after
+the release. [[Jellyfin]] matches a movie by folder and file name, so the
+release noise cost posters, extras and versions. These scripts turn that
+library into Jellyfin's own shape — and move TV back out of it — and nothing
+else in the repo touches a media file.
+
+| Script | Touches media? | Output |
+|---|---|---|
+| `scripts/media-census.py` | reads headers only | `enum`, `probed`, `plex` TSVs |
+| `scripts/media-plan.py` | no — TSVs in, TSVs out | the plan + the exception report |
+| `scripts/media-apply.py` | **renames, never copies**; sweeps only what a run emptied | the moves, plus a journal |
+| `scripts/media-tv.py` | no — TSV out | a plan that moves TV into the TV library |
+
+The seam is `tests/media_relayout.bats`; the target contract is quoted from
+[jellyfin.org/docs/general/server/media/movies](https://jellyfin.org/docs/general/server/media/movies)
+in `media-plan.py`'s docstring.
+
+## Procedure
+
+Run the census on the machine that mounts the library. `probe` is the slow step
+(a header read per container) and is `nice`d — it is safe beside a live service
+because it never competes for the GPU and never writes inside the library.
+
+```bash
+C=scripts/media-census.py
+$C enumerate --root /mnt/entropy/Movies --out /tmp/media-enum.tsv
+$C probe     --enum /tmp/media-enum.tsv --out /tmp/media-probed.tsv --jobs 6
+$C plex      --root /mnt/entropy/Movies --out /tmp/plex-movies.tsv
+```
+
+`plex` reads the Plex metadata cache (`--db` defaults to the path the
+[[Plex]] container bind-mounts at `/var/lib/plexmediaserver`) with
+`mode=ro&immutable=1`: a live server is never locked or written.
+
+Then plan — this reads only the three TSVs, so it costs seconds and can run
+anywhere, including a laptop:
+
+```bash
+scripts/media-plan.py --root /mnt/entropy/Movies \
+    --enum /tmp/media-enum.tsv --probed /tmp/media-probed.tsv \
+    --plex /tmp/plex-movies.tsv --out /tmp/jf-plan.tsv --exceptions /tmp/jf-exceptions.txt
+```
+
+**Read `/tmp/jf-exceptions.txt` before applying.** It is not a log: every line
+is a file the plan could not name a film for, a pile that needs a README, or a
+collision it refused. The printed gate lines are the go/no-go:
+`problems`, `bad-names`, `occupied`, `double-claim`, `left-unmoved` must all be
+0 and `unplanned=0` must appear in the report. Any non-zero number means a file
+that would be filed wrong, and the plan is not run until it is understood.
+
+```bash
+scripts/media-apply.py apply    --plan /tmp/jf-plan.tsv --root /mnt/entropy/Movies   # dry, writes nothing
+scripts/media-apply.py apply    --plan /tmp/jf-plan.tsv --root /mnt/entropy/Movies --apply
+scripts/media-apply.py verify   --plan /tmp/jf-plan.tsv --journal /mnt/entropy/media-relayout.jsonl
+scripts/media-apply.py sweep    --journal /mnt/entropy/media-relayout.jsonl --root /mnt/entropy/Movies   # dry
+scripts/media-apply.py sweep    --journal /mnt/entropy/media-relayout.jsonl --root /mnt/entropy/Movies --apply
+scripts/media-apply.py rollback --journal /mnt/entropy/media-relayout.jsonl --root /mnt/entropy/Movies --apply
+```
+
+An apply also leaves the release folders it emptied standing where they were —
+hundreds of them at the library root — which is what `sweep` removes, dry-run
+first, from the same journal.
+
+After it lands, both servers need a scan or the change is invisible.
+
+```bash
+# Run this from the host shell through `podman exec`, never from a neighbour
+# container: Plex's HTTPD closes without a single byte on any request that
+# arrives from another container's netns (10.89.x.x reads as "empty reply", not
+# as a 401). The token rides stdin, never argv, where any process on the host
+# could read it out of /proc/<pid>/cmdline.
+umask 077
+podman exec plex sh -c "sed -n 's/.*PlexOnlineToken=\"\\([^\"]*\\)\".*/\\1/p' \
+  '/config/Library/Application Support/Plex Media Server/Preferences.xml'" >/tmp/px.tok
+podman exec -i plex sh -c 'read -r T
+  for k in $(curl -s -H "X-Plex-Token: $T" -H "Accept: application/json" \
+      http://127.0.0.1:32400/library/sections | grep -o "\"key\":\"[0-9]*\"" | cut -d\" -f4); do
+    curl -s -o /dev/null -w "plex section $k: %{http_code}\n" -X POST \
+      -H "X-Plex-Token: $T" "http://127.0.0.1:32400/library/sections/$k/refresh"
+  done' </tmp/px.tok
+```
+
+[[Jellyfin]]'s scan key lives in `~/.config/secrets/jellyfin` (mode 600; the
+same value is `JELLYFIN_API_KEY` in the ignored `dankserver/.env`). It rides a
+mode-600 curl config, never a header argument, and it must use the
+`MediaBrowser` form — Jellyfin 12.4 refuses a bare `X-Emby-Token` for any key:
+
+```bash
+umask 077
+{ printf 'header = "Authorization: MediaBrowser Token=\\"%s\\""\n' \
+    "$(cat ~/.config/secrets/jellyfin)"
+  echo 'url = "http://127.0.0.1:8096/Library/Refresh"'
+  echo 'request = "POST"'; } >/tmp/jf.hdr
+curl -s -K /tmp/jf.hdr -o /dev/null -w "jellyfin scan: %{http_code}\n"
+shred -u /tmp/jf.hdr
+```
+
+`204` is acceptance; the scan then runs on the server's own schedule.
+
+## TV in the movie library
+
+A box set that arrives in the movie share stays there until someone moves it:
+`media-plan.py` refuses to file TV and parks each series in
+`_TV needs a TV library - <series>`. `scripts/media-tv.py` is the mover. It
+walks the movie library itself (no census needed), groups episode-shaped names
+by series, and writes a plan in the executor's own format:
+
+```bash
+scripts/media-tv.py --root /mnt/entropy/Movies --library /mnt/entropy/TVShows \
+    --out /mnt/entropy/media-tv-plan.tsv
+scripts/media-apply.py apply --plan /mnt/entropy/media-tv-plan.tsv --root /mnt/entropy
+scripts/media-apply.py apply --plan /mnt/entropy/media-tv-plan.tsv --root /mnt/entropy \
+    --journal /mnt/entropy/media-relayout-tv.jsonl --apply
+scripts/media-apply.py sweep --journal /mnt/entropy/media-relayout-tv.jsonl \
+    --root /mnt/entropy --apply
+```
+
+The shape it writes is the one both matchers read — `Monk (2002)/Season 05/
+Monk - S05E03 - Mr. Monk and the Big Game.mkv` — keeping the file's own name,
+because the publisher's spelling beats a re-derivation, with the year taken
+from the box-set folder. What it refuses it prints and leaves alone:
+
+- fewer than `--min` (5) files under one series name: one file named `Part 2`
+  is a film, not a series;
+- an episode whose name states no season number — no invented `Season 01`;
+- a destination two files both claim: the executor refuses the whole plan
+  before its first rename, and only Frank decides which copy is the keeper.
+- an episode the destination library already holds at the same byte count. The
+  movie share carried a second copy of the whole Monk box set — 124 episodes
+  both sides, the same episode list, every byte count identical, 31.5 GiB each
+  way, and head+tail blake2b identical on a 6-episode sample. The mover leaves
+  such a series in place and says so; a duplicate is deleted or kept by Frank,
+  never copied into one library by a tool.
+
+A sidecar that shares an episode's own name (`.srt`, `.nfo`) moves with it.
+
+Below the threshold the mover prints and leaves, so a lone downloaded episode of
+a real show stays in the movie library on purpose — six did on 2026-10-30.
+Filing one by hand is the same executor on a hand-written plan and the same
+journal; the trap is the series year. Read it from the file's own name or the
+folder it owns, never from the title whose bonus folder it happens to be parked
+in — `The Cleveland Show S03E07` inside `Die Hard (1988)/extras/` is not a 1988
+show. A name whose marker is `Part 2` is a film and stays put.
+
+## Laws
+
+- **A moved file is a 404 in [[Plex]] until its section is rescanned.** The
+  database keeps the release-folder path, so playback fails `Couldn't find the
+  file to stream` while the scanner works through the section. Never read such a
+  404 as a lost file: the file is where the journal says it is.
+- **Plex's matched title outranks the filename**, and a Plex year of `0` means
+  *indexed but never matched* — trusting it named folders `Title (0)`, which is
+  a Jellyfin item with no year, no poster and no identity. A year that is not a
+  plausible year is no information.
+- **`<>:"/\|?*` never survive a name.** Plex's own title for *V/H/S/2* is
+  `V/H/S/2`; kept verbatim it became `Movies/V/H/S/2 (2013)/` — a tree five
+  levels deep that neither matcher reads as a movie.
+- **A folder inside a movie's folder is bonus territory even when its name is
+  misspelled.** `Saboteur (1942)/Exrtas/Saboteur - A Closer Look` is a typo for
+  `Extras`; the vocabulary rule never fired, the featurette became a movie
+  folder, and its words then out-scored the real film and drew that box set's
+  other extras into it. The rule that replaced it needs no metadata: the folder
+  holding the longest film owns the shorter video filed under it, unless that
+  file's own name carries a different year — which is another film speaking.
+- **Matching words are split on everything that is not a letter or a digit.**
+  Splitting on separators but not on `/` left tokens like `(1960)/extras` that
+  no other name could ever agree with, so a film could not be found by the
+  folder holding it, and extras drifted into a neighbour. A bare number counts
+  only when it could be a release year, so `1080` and `2160` never pose as
+  agreement between two names.
+- **Compare two media libraries with the series engine, never a path
+  substring.** Filtering paths for `monk` also returns *Monkey Dust*, and
+  letting whichever file a walk happens to reach first stand for an episode
+  paired two libraries into a fake duplicate — the same sloppiness pointed the
+  other way deletes a file that is unique. The comparison is `media-titles.ep_of`
+  on both sides: series, season, episode, bytes, and a hash before anything is
+  trashed.
+- **A deletion is staged on the item's own filesystem.** The trash rule names
+  `~/.AGENTTRASH`, but `/home` and `/mnt/entropy` are different devices: a 31 GiB
+  stage there is a copy rather than a rename, it has to fit in the 187 GiB of
+  free space, and `rename(2)` refuses it anyway. Trash goes beside the item —
+  `/mnt/entropy/.AGENTTRASH/<date>-<topic>/` — so it costs one inode change and
+  rolls back exactly like the move that made it.
+- **A Jellyfin 401 is the header, not the key.** 12.4 authenticates only
+  `Authorization: MediaBrowser Token="…"` and refuses a bare `X-Emby-Token` for
+  every key, valid or not. This page carried "no credential on this box can
+  trigger a scan, the two Seerr keys answer 401" for weeks because of that
+  header — the keys were innocent. Before believing a 401, prove the key against
+  the server's own table: `select count(*) from ApiKeys where AccessToken = ?`
+  on `jellyfin.db` opened `mode=ro&immutable=1`.
+- **A scan that must see a new folder has to be recursive.** `POST
+  /Library/Refresh` is the incremental pass, and inotify does not cross a
+  rootless podman bind, so folders created on the host stay invisible to it. The
+  move is `POST /Items/{id}/Refresh?Recursive=true` (204) per library, the id
+  from `GET /Library/VirtualFolders` — its field is `ItemId`, not `Id` — and
+  that same record's `RefreshStatus` is the progress read.
+- **A file whose name carries no words is bonus material, not a folder.**
+  `.Extra 1 ( 1H 27min multisub ).mkv` parses to `1`; a movie folder named `1`
+  is not a movie. It becomes a review line telling a human to name the file.
+- **The executor renames and nothing else.** `rename(2)` cannot cross a
+  filesystem, so a plan that would is refused rather than silently copying tens
+  of terabytes.
+- **The journal is a write-ahead log: the intent is fsynced before the rename.**
+  An interrupted run therefore leaves only two states — an intent whose move
+  happened, and an intent whose move never ran — and the next run finishes the
+  job without moving anything twice. A file that moved with no journal line did
+  not move through this tool, and the tool refuses to guess on its behalf.
+- **The journal lives BESIDE the library root, never inside it** (default:
+  `media-relayout.jsonl` next to the root): a log inside the library is a file
+  Jellyfin indexes.
+- **Refusals are all-or-nothing.** An occupied target, a vanished source or a
+  duplicate destination stops the run *before the first rename*, so the operator
+  fixes one thing and re-runs instead of untangling a half-applied plan.
+  `--force` relaxes only the opinion guards; it never permits overwriting.
+- **A sweep deletes only what the journal proves the run emptied.** A folder is
+  eligible when a journalled file moved out of it or out of something under it,
+  it now holds nothing but empty directories, no journalled destination lives
+  beneath it, and it is not a folder the run created. A folder nobody moved —
+  someone's placeholder — cannot be selected at all, and a symlink anywhere
+  inside disqualifies the subtree, because the thing behind a link is not this
+  tool's to delete. Every removal is journalled as `rmdir` and `rollback`
+  recreates the folder, so the sweep is undone by the same command as the moves.
+  Proof carries downward: inside a folder the journal proves emptied, every
+  empty directory goes with it. A release's `Screens/` or `Sample/` is never a
+  move source itself, so 146 of them held their emptied parents on disk after
+  the first sweep.
+  A failed `rmdir` is not a removal: only a folder that is really gone leaves
+  scope, so the next sweep finishes what a permission problem stopped.
+- **An episode-shaped name marks a SERIES, not a film — and the series is the
+  name BEFORE the marker.** `Monk - S05E03 - Mr. Monk and the Big Game` is series
+  `Monk`. Keyed on the whole title instead, every episode became a group of one
+  and never reached the five-episode size that calls something a series, so a
+  box set was filed as 124 movies in 124 folders. The threshold still earns its
+  keep: a lone film named `Part 2` stays a film. Episodes of a series are parked
+  in `_TV needs a TV library - <series>` for a human to move to a TV source —
+  the movie tool never files TV.
+- **One journal is one undo.** The TV move runs on its own journal, and with
+  `--root` set to the share that holds BOTH libraries: the executor refuses any
+  path outside `--root`, and a journal shared with the movie relayout would
+  undo both runs when only one of them is wanted.
+
+- **Sidecar metadata does not follow a rename, because it lives on another
+  filesystem.** The media servers write `.nfo` and artwork through the
+  [[Media Metadata View]] (NVMe upper layer); this tool moves bytes on the raw
+  array path. After any sweep or TV move, run `scripts/media-view.sh reap`, then
+  the same command with `--apply`: the sidecars left at the old coordinates become
+  ghost folders — artwork and an `.nfo` with no video — that both servers browse
+  and report as content.
+- **Nothing deletes through the metadata view.** A delete there writes an overlayfs
+  whiteout in the upper layer, which hides the array's copy without removing it;
+  the file returns when the view is unmounted. Deletions in this pipeline happen
+  on the raw path only, which is what the journal can undo.
+
+## Rollback
+
+One command, from the journal of the run:
+`media-apply.py rollback --journal …/media-relayout.jsonl --apply`. It recreates
+any folder a sweep removed, restores every source path, removes only the folders
+that run created *and that are now empty*, and refuses everything if any moved
+file cannot be accounted for. A rolled-back journal refuses to roll back twice.
+
+## See Also
+
+[[Jellyfin]], [[Plex]], [[Media Metadata View]], [[Entropy Volume]], [[Testing Policy]], [[Storage Policy]]
+
+## Citations
+
+- Jellyfin Movie documentation — folder/file naming, extras subfolders,
+  `Movie (2013) - 216p.mkv` version and `- 4K.mkv` handling:
+  <https://jellyfin.org/docs/general/server/media/movies>

serverAppConfigs-docs/Dankserver/Plex.md

@@ -4,7 +4,7 @@ description: "Plex Media Server is a **containerized media server on Dankserver*
 tags:
   - dankserver
   - plex
-timestamp: 2026-10-21
+timestamp: 2026-10-30
 ---
 
 # Plex
@@ -13,7 +13,8 @@ Plex Media Server is a **containerized media server on Dankserver**: a podman
 quadlet (`plex.service`) under the systemd **user** generator, generated from
 `dankserver/plex/quadlet/plex.container`. It shares the host with a second live
 media server, [[Jellyfin]] (`jellyfin.service`, listening on 8096); both serve
-from the same read-only Entropy Volume mount.
+from the same tree — the [[Media Metadata View]] overlay — so the Kodi-format
+sidecars Jellyfin writes are the files Plex reads, with no copy and no sync.
 
 **Host:** Dankserver — **Ports:** 32400/tcp (PMS web + API), 32400 is the
 only port PMS needs for client streaming; LAN clients connect directly,
@@ -26,7 +27,7 @@ public traffic can be fronted by Caddy like every other vhost.
 | Runtime | podman quadlet, `plex.service` (user manager) — migrated from the native `plexmediaserver.service` OS-package unit (2026-09) |
 | Image | `lscr.io/linuxserver/plex:1.43.5.11029-6bf8af3f0-ls329` (pinned; verified == the registry's `latest` manifest digest at pin time: `sha256:8f655bad…e9e2334`) |
 | Data root | `/var/lib/plexmediaserver` bind-mounted **in place** as `/config` (RW) — the migration moved zero bytes |
-| Media | `/mnt/entropy` mounted at the same absolute path **read-only** — every library path in the PMS database stayed valid, zero re-scan |
+| Media | `/var/lib/media-meta/view` (the [[Media Metadata View]] overlay) bound at the same absolute path `/mnt/entropy` **read-only** — every library path in the PMS database stayed valid, zero re-scan |
 | Identity | fleet rootless model — container root maps to the `dank` user (1000:1000); the deployer chowns the data root on first cutover |
 | Port | `32400/tcp` published on `0.0.0.0` — zero client reconfiguration |
 | Network | `dankserver_appnet`, alias `plex`; Tautulli reaches it at `http://plex:32400` |
@@ -38,7 +39,8 @@ public traffic can be fronted by Caddy like every other vhost.
 
 The migration moved zero bytes: the container bind-mounts the native data
 root where it already lived, and the media archive re-mounts at
-`/mnt/entropy` inside the container, so every absolute library path in the
+`/mnt/entropy` inside the container — only the HOST side of that bind has moved
+(array, then the overlay view), so every absolute library path in the
 PMS database keeps resolving. The one metadata change: rootless podman
 cannot map the real host uid 997, so the fleet model applies — container
 root runs as the `dank` user, and the deployer adopts the data root with

serverAppConfigs-docs/Infrastructure/Observability Provisioning.md

@@ -20,6 +20,8 @@ committed.
 | `shared/prometheus/prometheus.yml` | scrape jobs (global + per-host) |
 | `shared/prometheus/rules/llm-fleet.yml` | alert rules + recording rules |
 | `shared/grafana/provisioning/` | datasources + dashboards (provisioned tree) |
+| `shared/alloy/config.alloy` | the log pipeline: journal source → label promotion → Loki push |
+| `shared/blackbox/blackbox.yml` | prober modules (`http_2xx`); the listen bind lives on the unit's `Exec` |
 
 Targets and datasources reference **container names** (`grafana:3000`
 `loki:3100` — `shared/prometheus/prometheus.yml:52-61`), never
@@ -31,8 +33,8 @@ scraped as `host.containers.internal:<port>` because a bridge container cannot
 reach a host-netns bind at the host's LAN IP (`prometheus.yml:89-99`), and only
 the endpoint on another machine (`192.168.0.235:7770`) is addressed by LAN IP
 (`prometheus.yml:128-131`). Prometheus 3 refuses HTML responses as metrics —
-HTML-scraped "probes" are structurally inert and need a blackbox exporter (see
+HTTP liveness rides the **blackbox exporter** (`shared/blackbox/blackbox.yml`,
-[[Stale Signal Audit]]).
+deployed 2026-10-21; see [[Stale Signal Audit]]).
 
 ## The deploy path
 
@@ -43,8 +45,9 @@ from `deploy_observability`:
 - `copy_alertmanager_config` → `/opt/alertmanager/config/alertmanager.yml` (`:232-235`)
 - `install_signal_relay` → the user unit + iptables + the health wait on `192.168.0.143:18093/health` (`:243-274`)
 - `copy_loki_config` → `/opt/loki/config/loki-config.yaml` (`:276-279`)
-- `copy_alloy_config` → `/opt/alloy/config/config.alloy` (`:281-284`)
+- `copy_alloy_config` → `/opt/alloy/config/config.alloy` — substitutes the `GRAFANA_HOST_LABEL` token with the deploy target's name and **refuses to install** if the token survives; `ALLOY_CONF_ROOT` is the bats sandbox seam
-- `copy_grafana_provisioning` → `/opt/grafana/provisioning/` (`:286-300`)
+- `copy_blackbox_config` → `/opt/blackbox/blackbox.yml` (single-file mount of the prober); `BLACKBOX_CONF_ROOT` is its sandbox seam
+- `copy_grafana_provisioning` → `/opt/grafana/provisioning/` — datasources, provider, and **every** `dashboards/*.json` (a glob, not a hand-listed subset; `GRAFANA_PROV_ROOT` is the test seam the bats case uses to run the copier into a sandbox)
 
 `scripts/deploy-plan.py` maps `shared/prometheus/**` to `prometheus` (plus
 `alertmanager` when the changed file is `alertmanager.yml`, `:251-256`)
@@ -82,11 +85,14 @@ reports 4 alerting + 2 recording. Rules reload with the prometheus service.
 the backend (`http://prometheus:9090`), not loopback.
 
 **Add a dashboard** — drop the JSON into
-`shared/grafana/provisioning/dashboards/`. Provisioned dashboards live in
+`shared/grafana/provisioning/dashboards/`; the copier globs the directory, so
-unified storage, not (only) the SQLite table — grep the API, not the DB, when
+the next `--service grafana` deploy provisions it. This is a repaired law, not
-verifying (see the verify section of [[Fleet Audit and Posture]]). *(unverified
+a given: until 2026-10-21 the copier named `llm-fleet.json` alone, and the
-2026-10-18: the storage-backend claim needs an admin password to confirm
+`containers` board — merged, tested, documented as "working" — had **never
-against `grafana.db` and the API side by side)*
+been provisioned**; the other boards existed on disk only because someone had
+hand-copied them once. A repo JSON is not a live dashboard until the copier
+carries it, and `deploy.sh` cannot see the gap because it only reports the
+copy it performed.
 
 ## Verify through the APIs, not the filesystem
 
@@ -95,17 +101,22 @@ against `grafana.db` and the API side by side)*
 curl -s http://127.0.0.1:9090/api/v1/rules | python3 -m json.tool | head
 curl -s "http://127.0.0.1:9090/api/v1/targets?state=active" | \
   python3 -c "import json,sys; [print(t['labels'].get('job'), t['health'], (t['lastError'] or '')[:60]) for t in json.load(sys.stdin)['data']['activeTargets']]"
-# grafana: datasource + dashboard provisioned
+# grafana 13: /api/health, /api/org, /login all 404 UNAUTHENTICATED, and there
-curl -s -u admin:<pw> http://127.0.0.1:3002/api/datasources | python3 -m json.tool | grep -E '"name"|"url"'
+# is NO local admin password (root .env sets no GF_SECURITY_ADMIN_*; the only
+# account is the Authentik RP login). The proven surface is a real browser
+# session: warm the Anubis challenge on the vhost root, then hit
+# /login/generic_oauth with the e2e storage state (tests/e2e/.auth/qa-e2e-admin.json)
+# — exactly the path specs/20-grafana-sso.spec.ts drives — and read the boards.
 ```
 
 The two Prometheus queries were run on 2026-10-18 and are what every number on
 this page comes from: 4 alerting + 2 recording rules, and `up` for
 `prometheus`, `node-dankclaw`, `node-dankserver`, `grafana`, `loki`
-`frank-zealot`, `frank-dragoon`, `frank-archon` with the four structural downs
+`frank-zealot`, `frank-dragoon`, `frank-archon`. The four structural downs
-listed in [[Prometheus]]. The Grafana query needs the local admin password and
+that census recorded are retired (2026-10-21, [[Prometheus]]). The Grafana proof is the browser session above, run
-was not run here; the filesystem side (`ls /opt/grafana/provisioning/*`) is
+2026-10-21: it found a provisioned-file gap the filesystem listing hid. The
-proof of installation only — see the next paragraph.
+filesystem side (`ls /opt/grafana/provisioning/*`) is proof of installation
+only — see *Add a dashboard* for why a repo JSON is not a live board.
 
 Health answers from a stale process and provisioned resources can hide in
 unified storage — a file existing on disk is not proof the running server

serverAppConfigs-docs/Infrastructure/Observability.md

@@ -74,11 +74,19 @@ published ports — node `:9100`, cadvisor `:8080`, gpu `:9401` all answer there
 — while the host's own LAN IP is refused (`192.168.0.143:8080`, `:9401`), so
 every same-host Prometheus target uses the gateway and only cross-host targets
 (dankserver) use plain LAN. Whether LAN works at all is network-specific: the
-relay at `192.168.0.143:18093` IS reached from alertmanager's network, with
+relay at `192.168.0.143:18093` is reached from alertmanager's network only
-iptables INPUT admitting loopback, the host itself and the podman bridge subnet
+while `install_signal_relay`'s INPUT rules are in force: loopback, the host
-`10.89.0.0/16` before the DROP (`scripts/deploy-remote.sh:255-262`). Never
+itself and the podman bridge subnet `10.89.0.0/16` accepted **above** a
-point a scrape target at a host LAN IP from its own host on the strength of
+trailing blanket DROP. The law the poisoned host taught (2026-10-21):
-one working example; dial the gateway and probe.
+`iptables -I` prepends, so inserting the DROP last lands it first — the old
+install order dropped every packet (159 on the DROP, 0 on the ACCEPTs), the
+deployer's health gate failed beside a LISTENing socket, and the drain
+aborted twice before `nft list ruleset` named the cause. The install now
+rebuilds the tail (delete the DROP wherever it sits, ACCEPTs at the head,
+DROP appended); verify any change to it with the sandboxed bats case, not
+with `-C` greps — `-C` answers existence, never order. Never point a scrape
+target at a host LAN IP from its own host on the strength of one working
+example; dial the gateway and probe.
 
 ## Deployment
 
@@ -95,13 +103,14 @@ health endpoint (`/prometheus`: `/-/healthy`, `/alertmanager`: `/-/healthy`
 [[Observability Provisioning]] for what is provisioned as code and how to
 verify it through the APIs.
 
-## Current state (2026-10-20)
+## Current state (2026-10-21)
 
 * **Metrics: working.** `curl -s http://127.0.0.1:9090/api/v1/targets` reports
-  eight up targets — the three LLM endpoints (frank-archon on `vllm:*`,
+  every job up — the three LLM endpoints (frank-archon on `vllm:*`,
-  frank-dragoon and frank-zealot on `llamacpp:*`), grafana, loki, prometheus
+  frank-dragoon and frank-zealot on `llamacpp:*`), grafana, loki, prometheus,
-  and both node exporters. The four downs are structural (the `comfyui` and
+  both node exporters, cadvisor, gitea (metrics enabled in its quadlet) and
-  `services` HTML probes, and Gitea's disabled metrics endpoint), not outages.
+  the blackbox-probed `services` (SearXNG, OpenWebUI, ComfyUI). The structural
+  downs are retired — see [[Prometheus]] → Targets.
   Two rule groups in `shared/prometheus/rules/llm-fleet.yml`: `llm-fleet`
   (2 recording + 4 alert, `llamacpp:*`) and `vllm-fleet` (1 recording + 4
   alert, `vllm:*` — archon queueing, prefix collapse, preemption, p95
@@ -109,12 +118,15 @@ verify it through the APIs.
 * **Delivery: working.** Alertmanager's single `signal` receiver webhooks the
   relay, and `journalctl --user -u signal-relay` records successful
   `sent to +12062476336:` lines for every rule in the set.
-* **Dashboards: working.** Provisioned datasources — Prometheus (uid
+* **Dashboards: working (after the 2026-10-21 copier fix).** Provisioned
-  `prometheus`) and Loki (uid `loki`) — and the `llm-fleet`, `host-system`,
+  datasources — Prometheus (uid `prometheus`) and Loki (uid `loki`) — and the
-  `containers`, `observability` and `logs-overview` dashboards, all
+  `llm-fleet`, `host-system`, `containers`, `observability` and
-  installed from `shared/grafana/provisioning/`. The Loki datasource is how
+  `logs-overview` boards. Until 2026-10-21 the copier carried only
-  you read logs: Loki has no web UI of its own, so Grafana Explore is the
+  `llm-fleet.json` by name and the `containers` board had never been
-  human surface (the guarded `loki.franksdank.cloud` vhost serves only the API).
+  provisioned; the deployer now globs the directory. The Loki datasource is
+  how you read logs: Loki has no web UI of its own, so Grafana Explore is the
+  human surface (the guarded `loki.franksdank.cloud` vhost serves only the
+  API). Verify method and the copier law: [[Observability Provisioning]].
 * **Per-container metrics: the `containers` dashboard (added 2026-10-20; live
   after `deploy.sh <host> --service cadvisor`).** cadvisor is the only way to see
   per-container CPU/memory here, and on this rootless-Podman host it works on one
@@ -139,26 +151,26 @@ verify it through the APIs.
   `power1_average`, so they carry no `gpu_power_watts` line at all (only NVIDIA and
   the Dankclaw 8060S iGPU report power); and it binds loopback + the host LAN, like
   node-exporter, never the wildcard. The dashboard graphs util/VRAM/VRAM-%/temp/power.
-* **Logs: fixed in code, awaiting deploy.** The pipeline shipped nothing because
+* **Logs: flowing; container labels land with the 2026-10-21 deploy.** The
-  a rootless Alloy cannot read the `0640` journal files, and the container-log
+  pipeline once shipped nothing (rootless Alloy cannot read the `0640` journal;
-  globs point at a root-storage layout that does not exist here. The fix is built
+  the container-log globs pointed at a root-storage layout that does not exist
-  and tested in-repo (2026-10-20): Alloy is now a **root system service on
+  here) — fixed 2026-10-20 by making Alloy a **root system service on
-  `Network=host`**, reading `system.journal` + `user-*.journal` and pushing to
+  `Network=host`**. Journal lines now reach Loki (measured ~6.5k lines/5min),
-  Loki's `127.0.0.1:3100` loopback publish — a throwaway root probe read 31,825
+  but the stream carried only `job`/`service_name`, so every per-container
-  lines and pushed 26,966 with zero drops. The **live** instance is still the old
+  view was structurally empty: Loki metrics queries group only by INDEXED
-  rootless unit until `deploy.sh dankclaw --service alloy` runs. Detail in
+  stream labels. Alloy now promotes podman's `CONTAINER_NAME` to a `container`
-  [[Alloy]] → Why root; Loki's store state in [[Loki]] → Contents.
+  label (plus a `host` label) at ingestion — `loki.process` in
-* **Not configured:** no blackbox exporter (the structural downs have no real
+  `shared/alloy/config.alloy`; detail in [[Alloy]] → Why root, and the
-  uptime signal) and no Alertmanager vhost. The log pipeline's fix is in code but
+  purpose-test law in [[Observability Provisioning]].
-  not yet deployed (see *Logs* above).
 
 ## Citations
 
-- `dankclaw/observability/quadlet/{prometheus,alertmanager,grafana,loki,alloy,node-exporter,cadvisor}.container`
+- `dankclaw/observability/quadlet/{prometheus,alertmanager,grafana,loki,alloy,node-exporter,cadvisor,blackbox-exporter}.container`
 - `scripts/deploy-remote.sh:219-300` (copiers + relay), `:521-536` (health waits), `:477-486` (grafana env seed)
 - `scripts/signal-relay.py:26-49`, `scripts/signal-relay.service`
 - `scripts/gpu-exporter.py`, `scripts/gpu-exporter.service` (per-GPU host exporter)
 - `shared/prometheus/prometheus.yml`, `shared/prometheus/alertmanager.yml`, `shared/prometheus/rules/llm-fleet.yml`
 - `shared/grafana/provisioning/`, `shared/loki/loki-config.yaml`, `shared/alloy/config.alloy`
+- `shared/blackbox/blackbox.yml` (prober modules; the listen bind is on the unit's `Exec`)
 - `tests/observability_quadlet.bats` (boot persistence, no wildcard publish, probe addresses, delivery contract)
 - Host evidence 2026-10-18: `ss -ltnp`, `podman ps`, `systemctl --user is-enabled|is-active`, `curl` on `127.0.0.1:9090/api/v1/{targets,rules,alerts}`, `127.0.0.1:3100/loki/api/v1/labels`, `127.0.0.1:12345/metrics`, `127.0.0.1:3002/api/health`, `192.168.0.143:18093/health`, `journalctl --user -u signal-relay`

serverAppConfigs-docs/Infrastructure/Prometheus.md

@@ -42,29 +42,29 @@ Authentik; post-auth the Caddy vhost routes straight to `prometheus:9090`) —
 
 The `frank-medium` job is **retired**: no `frank-medium` job exists in the config or in `/api/v1/targets` (2026-10-18), and the endpoint itself left the fleet — its history is on [[frank-archon]], which holds the port today.
 
-**Three jobs read down today, and they are not outages** (re-audited against `curl -s http://127.0.0.1:9090/api/v1/targets`, 2026-10-18; first audited 2026-09-16):
+**No job reads down structurally any more** (the three permanent-reds were
+retired 2026-10-21; census re-run after each deploy):
 
-| Job | Target | `health` | `lastError`, verbatim |
+| Job | Target | how |
-|-----|--------|----------|------------------------|
+|---|---|---|
-| `gitea` | `http://192.168.0.143:3000/metrics` | down | `server returned HTTP status 404 Not Found` |
+| `gitea` | `192.168.0.143:3000/metrics` | metrics enabled in the quadlet (`GITEA__metrics__ENABLED=true`, `ENABLED_HOST=0.0.0.0`) — the page once said "stays off until `[metrics] ENABLED = true` in its own config"; the env form is what the deployer ships |
-| `comfyui` | `http://comfyui:8188/` | down | `received unsupported Content-Type "text/html" and no fallback_scrape_protocol specified for target` |
+| `services` | `http://192.168.0.143:8081` (SearXNG), `:3005` (OpenWebUI), `http://127.0.0.1:8188` (ComfyUI) | **blackbox exporter** probe, module `http_2xx` — the comfyui metrics job is retired, one owner per service. The prober runs in the host netns, so comfyui is probed over loopback (its published port); bridge container names do not resolve there |
-| `services` | `http://192.168.0.143:8081/` | down | `received unsupported Content-Type "text/html; charset=utf-8" and no fallback_scrape_protocol specified for target` |
-| `services` | `http://192.168.0.143:3005/` | down | same `unsupported Content-Type "text/html; charset=utf-8"` error |
 
-Gitea's exporter stays off until `[metrics] ENABLED = true` in its own config (no metrics env is set in the container, and `curl -s http://127.0.0.1:3000/metrics` → 404, 2026-10-18); ComfyUI and the two `services` probes are HTML pages, which Prometheus 3 refuses as metrics. Everything else is green: `prometheus`, `node-dankclaw`, `node-dankserver`, `grafana`, `loki`, `frank-zealot`, `frank-dragoon`, `frank-archon`.
+Scraping a web root is not a health check: an HTML page fails the scrape
+protocol (`unsupported Content-Type`), which is what kept those targets red
+beside healthy services. The classification rule and the honesty law live in
+[[Stale Signal Audit]]. The prober is `blackbox-exporter.container` (host
+netns, config `shared/blackbox/blackbox.yml`); it binds 9115 on loopback and
+the LAN, and Prometheus scrapes it at `host.containers.internal` — traffic to
+that gateway lands on the host's loopback listener. The gateway address is
+not itself bindable from the host netns.
+Plain targets stay `host:port`, never URLs — the probe job's URL targets are
+the documented exception, relabeled onto the prober.
 
 **`frank-zealot` reads up since 2026-10-18, and did not before.** llama.cpp only answers `/metrics` when started with `--metrics`, so `up{job="frank-zealot"}` was 0 across the whole retained history (the oldest sample, 2026-10-11, is already 0) while the service answered `/health` 200 the whole time; the flag and its opt-out are on [[frank-zealot]]. Read `lastError` before believing an outage: a target can read down beside a healthy service.
 
 Alert state is a live property, not a page fact: `curl -s http://127.0.0.1:9090/api/v1/alerts` returned no alerts at the 2026-10-18 morning census, one firing `LlmPrefixCacheCollapse` (info, `frank-zealot`, `activeAt` 11:10:48Z) minutes later, and none again by 11:56Z.
 
-The remedy for a probe-shaped question is the **blackbox exporter**; the
-classification rule and the honesty law for it live in [[Stale Signal Audit]].
-Until the exporter is deployed, these three stay red for a structural reason:
-deleting them would
-hide a real service, and fixing them by publishing databases or enabling metrics
-would trade a wrong signal for a real exposure. Targets are `host:port`, never
-URLs.
-
 ## Volumes
 
 | Container path | Host path |

serverAppConfigs-docs/LLM/Dankclaw System Specs.md

@@ -10,7 +10,7 @@ timestamp: 2026-09-03
 
 ## Server Overview
 
-Dankclaw is an AMD APU + RTX 3090 inference host. It ran CIRU ROCmFPX llama.cpp inference through 2026-08-22 ([[frank-dragoon]], retired). The host runs Ubuntu 26.04 with an AMD Ryzen AI MAX+ 395 Strix Halo APU. The APU uses unified memory — CPU and GPU share the same physical RAM pool. Today it serves [[frank-dragoon]] (halogen-flash-server 0.16.2 on the iGPU, port 7770) and [[frank-zealot]] (llama.cpp PrismML fork on the RTX 3090, port 7771).
+Dankclaw is an AMD APU + RTX 3090 inference host. It ran CIRU ROCmFPX llama.cpp inference through 2026-08-22 ([[frank-dragoon]], retired). The host runs Ubuntu 26.04 with an AMD Ryzen AI MAX+ 395 Strix Halo APU. The APU uses unified memory — CPU and GPU share the same physical RAM pool. Today it serves [[frank-dragoon]] (halogen-flash-server 0.18.20 on the iGPU, port 7770) and [[frank-zealot]] (llama.cpp PrismML fork on the RTX 3090, port 7771).
 ## Hardware Specifications
 
 ### CPU
@@ -105,7 +105,7 @@ cable between claw and server is required for 10G.
 
 - Unified memory eliminates separate VRAM limits. The GPU uses system RAM.
 - The GPU budget is the GTT aperture: ~114 GiB (`mem_info_gtt_total` in `/sys/class/drm/card1/device/` = 122,402,963,456 B; `card1` is the amdgpu iGPU, `PCI_ID=1002:1586`; `llama-server --list-devices` reports 116,732 MiB). rocm-smi's smaller SVM/dedicated carve-out is **not** the budget — the 2026-08 "64 GiB rocm-smi" lesson is stale. Do not use `free` or `dmidecode` for the GPU budget.
-- The active services are `frank-dragoon` on 7770 — **halogen-flash-server 0.16.2** serving the native v2 abliterated `.hgn` in a rootless podman container ([[frank-dragoon]]) — and `frank-zealot` on 7771 ([[frank-zealot]]). The gufo v0.1.1 / UD-Q8_K_XL runtime this page once described is retired (2026-09-28 → 10-01, [[frank-dragoon]]); the gufo build tree is still on disk at `/home/dank/GitPackages/gufo-v0.1.1/`.
+- The active services are `frank-dragoon` on 7770 — **halogen-flash-server 0.18.20** serving the native v2 abliterated `.hgn` in a rootless podman container ([[frank-dragoon]]) — and `frank-zealot` on 7771 ([[frank-zealot]]). The gufo v0.1.1 / UD-Q8_K_XL runtime this page once described is retired (2026-09-28 → 10-01, [[frank-dragoon]]); the gufo build tree is still on disk at `/home/dank/GitPackages/gufo-v0.1.1/`.
 - The gufo-era note that the active runtime links against the host ROCm (`/opt/rocm` → `/etc/alternatives/rocm` → 7.2.4) through its RPATH belongs to that retired source build. The serving runtimes do not depend on the host ROCm that way: halogen runs inside its digest-pinned image, and `frank-zealot` is a CUDA build on the 3090. The retired CIRU venv's TheRock 7.15 SDK (`LD_LIBRARY_PATH`) was never in the serving path either.
 - The retired vLLM venvs (`/home/dank/vllm-venv`, `/home/dank/vllm-vk-venv`) were moved to `/home/dank/.AGENTTRASH/` on 2026-08-18.
 - 424 GB free on `/` (`df -h`, 2026-10-18) is ample for model storage.

serverAppConfigs-docs/LLM/Radiance Architecture Plugins.md

@@ -0,0 +1,287 @@
+---
+type: "howto"
+title: "Radiance Architecture Plugins — declaring a model"
+description: "Writing a radiance architecture plugin: declaring weights, buffers, KV groups and ops, then issuing the step."
+tags:
+  - llm
+  - radiance
+  - plugins
+  - howto
+timestamp: 2026-10-30
+---
+
+# Radiance architecture plugins
+
+An architecture plugin declares a model in two phases: `rad_arch_declare` names every weight,
+buffer, KV group and op **once**, against the model's metadata; `rad_arch_step` issues the ops for
+one step. The engine picks the plugin by the exact `(rad_arch_id, rad_arch_quant)` pair — no
+fallback — so one family at two weight formats is two `.so` files, and branching on a quant
+descriptor inside one plugin is the anti-pattern the split exists to prevent.
+
+The in-tree plugins are the templates: `arch/llama_fp8/llama_fp8.cpp` (the smallest complete
+decoder), `arch/gemma3`, `arch/qwen35_bf16`, and the header-only blocks in `arch/common/`
+(`rad_block_attn*.h`, `rad_block_mlp*.h`, `rad_block_moe_fp8.h`, `rad_block_gdn*.h`,
+`rad_block_dflash2.h`, `rad_block_mtp*.h`, `rad_block_vit.h`, `rad_block_ple.h`,
+`rad_block_glm_*.h`, `rad_qsa.h`, …). Include those by `<arch/rad_arch.h>` — they are installed
+beside the ABI and the in-tree plugins include them by that spelling.
+
+## Exports
+
+```c
+const char* rad_arch_id(void);              /* "llama", "qwen35", ... */
+const char* rad_arch_quant(void);           /* "" unquantised, else the descriptor served */
+int         rad_arch_serves_imports(void);  /* optional: 0 declines an imported quantisation */
+int         rad_arch_probe(const RadModelMeta* meta, RadArchProbe* out);   /* optional */
+int         rad_arch_declare(RadBuilder* b, const RadModelMeta* meta, const RadBuildCtx* ctx);
+void        rad_arch_step(struct RadCtx* c, const struct RadBatch* batch);
+const RadChatFormat*  rad_arch_chat_format(const RadModelMeta* meta);      /* optional */
+const RadAudioFormat* rad_arch_audio_format(const RadModelMeta* meta);     /* optional */
+```
+
+Everything but declare/step is optional. `RAD_ARCH_PLUGIN(NS, ID, QUANT, VERSION, DESC)` in
+`arch/common/rad_arch.h` emits the whole export block, including `rad_plugin_info` (kind
+`RAD_PLUGIN_ARCH`). The plugin's **name must be unique**; the architecture id may repeat.
+
+`rad_arch_serves_imports` exists because a checkpoint whose quantisation was imported weight by
+weight (Quark, ParoQuant, GPTQ, NVFP4) names its producing tool, which no plugin claims: the family's
+one quantised plugin serves it, and two are refused by name. A second quantised plugin that reads
+only its own containers returns 0.
+
+## Declare: the metadata
+
+`RadModelMeta` is what the loader read from the container's header before any plugin ran: `arch_id`,
+`name`, `quant`, `n_layers`, `n_embd`, `n_head`, `n_head_kv`, **`head_dim`**, `n_ff`, `n_vocab`,
+`n_ctx_train`, expert counts, `rms_eps`, `rope_theta`, and a free-form `kv_key`/`kv_val` table for
+everything else (`rad_meta_geti/getf/gets`). Read dimensions from here — one plugin covers every size
+in a family.
+
+**`head_dim` is read, never derived.** On the Qwen hybrids it is 256 while `hidden / heads` is 213;
+a derived value silently breaks projection widths, KV page size and the attention kernel's band.
+
+`RadBuildCtx` is the deployment the plugin must declare against:
+
+| field | meaning |
+|---|---|
+| `rank`, `world_size` | declare runs once per rank; **dimensions are already divided** — the core inserts no collectives |
+| `max_tok`, `max_seqs`, `max_ctx`, `max_spec` | the step bounds: buffer sizing, kernel bands, the scheduler's window |
+| `max_mixed_tok` | the most a multi-sequence step may carry; a wider step is one sequence (prefill span) |
+| `tp_wire_lossy`, `tp_wire_min_bytes` | invert into `all_reduce`'s `exact`; below the floor a message must go exact |
+| `kv_dtype`, `kv_quant` | the attention cache width and, under fp8, its narrow layout (`RAD_KVQ_V4`, `RAD_KVQ_K6V4`) |
+| `attn_prefill_fp8` | `RAD_ATTN_FP8_*` bits to pass as `attn_paged`'s `prefill_fp8` |
+| `kv_resident_blocks`, `kv_stream_ok` | KV streaming: the ring depth, and permission to declare the streamed-only ops |
+| `state_dtype` | `RAD_F16` when `--state-cache-dtype f16` is served by this plugin |
+| `is_draft`, `scope` | a drafter/encoder declare; `scope` prefixes declared names (`draft`, `vision`, `audio`) |
+| `shape_probe` | a **sizing** declare at a smaller `max_tok`; must not touch state the real declare wrote |
+| `max_enc_patches`, `max_audio_frames` | the encoder pass sizes |
+| `max_out_rows` | logits rows the KL mode needs (0 = the plugin's own rule) |
+| `cvec` | the loaded control vector, or null |
+| `tp_attn_rows` | row-split prefill attention requested |
+| `draft_coupling`, `draft_tree` | declare the coupled/tree forms |
+| `weight_prefetch_off` | declare no weight prefetch beside collectives |
+
+## Declare: weights
+
+```c
+static const RadWeightDecl d = {
+    .dtype = RAD_BF16, .rank = 2, .shape = {n_embd, n_ff},
+    .access = RAD_ACCESS_PER_TOKEN, .shard = RAD_SHARD_ROW,
+    .group = grp_layer(l), .optional = 0,
+    .n_row_parts = 0, .source = nullptr, .planes = nullptr,
+};
+rad_weight w = rad_decl_weight(b, "blk.0.mlp.gate.weight", &d);
+```
+
+- **`access`** tells the placement planner how the weight is read: `PER_TOKEN` (dense),
+  `PER_REQUEST` (an encoder tower, read once per request), `CONDITIONAL` (routed experts),
+  `RARE`, `VOCAB` (lm_head/embedding: read every token but huge, and one is a gather).
+- **`shard`**: `NONE` (replicated), `ROW` (split dim 0), `COL` (split dim 1, needs a following
+  all-reduce). The plugin says; the core does not infer. Uneven splits use
+  `rad_weight_shard_span`/`_parts` (a width that does not divide into whole 128-column blocks; the
+  delta net's `[q|k|v|z]` of unequal part widths).
+- **`group`** is the unit of movement (`layer`, `expert`, `slot`); an expert's gate/up/down share one,
+  because the unit of transfer is an expert, not a tensor.
+- **`row_parts`** describes what a `SHARD_ROW` weight stacks along dim 0 (`[gate|up]`,
+  `[q|k|v|z]`). **A tensor-parallel rank owns a slice of each stacked projection**, never a
+  contiguous slice of the stack; the parts cannot be inferred from a fused container tensor, and a
+  contiguous slice produces fluent wrong text.
+- **`source`/`planes`** view an existing stored weight: a block-fp8 linear's `codes,scale` are one
+  logical weight taken by a GEMM as two operands. Null `source` is the weight of its own name; null
+  `planes` takes every plane.
+- A dtype other than the plane's is legal only as an exact widening (a bf16 norm read as f32).
+
+The **name map** (`rad_decl_name_map`) is the plugin's map from checkpoint tensor names to declared
+weight names, and it is the only place convert-time fusions live: `COPY`, `CONCAT`, `WNORM` (a
+weight-normalised pair folded to the plain weight), `ROWS`, `TRANSPOSE`, `EMA` (an EnCodec codebook),
+`TAP`, `COLS`. `rad-convert` reads this map from the plugin, so it needs no knowledge of the model.
+`RAD_MAP_COPY` with several sources means "first present wins" (a tied lm_head); `CONCAT` requires
+all. Ask for a weight's encoding with `rad_weight_encoding` **after** its map entry when the graph
+depends on the format (fp8 GEMM vs plain, rotated expert kernels).
+
+## Declare: buffers and KV groups
+
+`rad_decl_buffer` names every activation: `RAD_BUF_TRANSIENT` (liveness-analysed, may share bytes),
+`RAD_BUF_PERSIST` (across steps: a calibration accumulator, a routing histogram), or
+`RAD_BUF_DERIVED` (computed once per step by the core from the batch — `derive` names the
+derivation, e.g. `conv_state_index`). Buffers live in one arena sized once, not an allocator per
+step.
+
+`rad_decl_kv_group` declares each state kind: `RAD_KV_FULL` (paged per-token, growing),
+`RAD_KV_WINDOW` (bounded by a window), `RAD_KV_LINEAR` (per-sequence fixed state, GDN),
+`RAD_KV_CONV` (a rolling window of `width-1+num_spec`), `RAD_KV_RING` (window floor set at runtime).
+The **block size is not declared** — it comes from the resolved attention kernel's constraints; read
+it back with `rad_kv_block_size`. `rad_bind_layer_kv` binds each layer so the block manager knows how
+many state instances a sequence owns; `rad_kv_group_zero` (a rank with no heads),
+`rad_kv_group_replicated` (every rank holds the same bytes), and `rad_kv_group_dense_read` (read over
+the whole history every step, so it cannot stream) are the three refinements.
+
+## Declare: ops
+
+```c
+rad_op h = RAD_OP(b, "gemm_w4a8_nt",
+                  RAD_PARAMS(RAD_RANGE("M", 1, max_tok), RAD_INT("N", 8192), RAD_INT("K", 5120)),
+                  RAD_WEIGHTS(w));
+rad_op_reads (b, h, RAD_BUFS(b_x), n);
+rad_op_writes(b, h, RAD_BUFS(b_y), n);
+```
+
+- Parameter names and operand order come from `docs/OPS.md`; a new name is allowed only when the
+  vocabulary has none, and the first plugin to declare it fixes its schema.
+- `RAD_RANGE` on exactly one parameter per op gives the selector its bucket table. The core carries
+  every parameter opaquely except `RAD_PROLE_SEQ_CHUNK` (cut steps on it) and `RAD_PROLE_CAPACITY`.
+- `rad_op_reads`/`rad_op_writes` are the buffer planner's liveness. An undeclared use is assumed live
+  for the whole program: sound but wasteful.
+- **Fusion is selection.** Ask for the fused op; if `rad_op_resolved` is false, emit the chain. A
+  fusion's `replaces` hook is a checkable equivalence, not a rewrite.
+- `rad_op_declared(b, "op")` asks whether any library declares an optional op; use it before
+  emitting a fused form only an optional library defines.
+- Buffers touched from the **second lane** go through `rad_buf_concurrent`, and an architecture whose
+  lane-1 work lies in one stretch declares `rad_lane1_window_begin/end` around it. The runtime
+  refuses a lane-1 op outside the window; the window lets buffers used only outside it share bytes.
+
+Step output: `rad_declare_logits` names the buffer the sampler reads (its dim 0 bounds `n_out`; a
+[n_tok, vocab] logits buffer at a large vocabulary is gigabytes — declare the gather +
+`logits_gemm` shape instead). An embedding program declares `rad_declare_pooled` instead (one row a
+sequence, no sampler, no generation). `rad_declare_scores` opts into host-side scored steps.
+
+## Declare: drafters, encoders, generators
+
+- `rad_declare_drafter(RadDrafterDecl)` — depth, block, proposal shape. `rad_drafter_shared_kv` (a
+  drafter with no cache of its own), `rad_drafter_tree` (a tree over the proposals),
+  `rad_drafter_short_block` (a second, shorter pass) are refinements. Declare the drafter **last**.
+- `rad_declare_encoder(RadEncoderDecl)` — a vision/audio tower, media preprocessed by the core.
+- `rad_declare_generator` / `rad_declare_audio_generator` / `rad_declare_video_generator` — image,
+  audio and video generation programs; the engine sizes the step from the probe's `gen_*` fields
+  before declare.
+- Optional format exports let a model state what it knows and the server learn nothing new:
+  `rad_arch_chat_format` (a data description of reasoning/tool-call markup, replacing template
+  analysis), `rad_arch_audio_format` (how a speech model hears: clip tokens, prompt, language,
+  transcript start), `rad_arch_text_prepare` (rewrite request text into training form),
+  `rad_arch_video_positions`/`rad_arch_video_pack` (video pass geometry and packing),
+  `rad_arch_encode_geom` (encoder pass geometry). Each struct grows only at its end behind
+  `struct_size`, so adding a field does not move the ABI.
+
+**Media preprocessing is the core's**, not the plugin's: resizing, patching, placeholder expansion
+and positions in six processor families (Qwen-VL, Gemma 4, LFM2-VL, Unlimited-OCR, GLM-5.3, Pixtral
+for Nemotron), chosen by `vision_config.model_type` plus owning-model checks; audio through Gemma 4's
+extractor or Qwen3-ASR's Whisper one, chosen by `audio_config.model_type`. A vision model with another
+pipeline needs a core change.
+
+## Step: issuing the graph
+
+`RadBatch` is built once per step and shared by every block: `phase` (`PREFILL`/`DECODE`/`MIXED`),
+`n_tok`, `n_seq`, `token_ids`, `positions`, `cu_seqlens`, `seq_ids`, `lora_slot`, `n_out`/`out_ids`,
+`q_lens`/`ctx_lens`/`max_q_len`/`max_ctx_len`, the KV-group array, the speculative fields
+(`n_spec`, `draft_pass`, …).
+
+```c
+static void step(RadCtx* c, const RadBatch* batch) {
+    if (batch->draft_pass > 0) { m.ds.step(c, batch); return; }      /* query pass */
+    if (batch->draft_pass < 0) { m.ds.context(c, batch); return; }   /* context pass */
+    RAD_ISSUE(c, m.op_embed, praw(batch->token_ids, RAD_I32, batch->n_tok),
+              RAD_W(m.w_tok), brows(m.b_x, batch->n_tok));
+    for (auto& layer : m.layers) { layer.attn.step(c, batch); layer.mlp.step(c, batch); }
+    /* ...heads; the lm_head only over batch->n_out rows... */
+}
+```
+
+- `rad_issue(c, op, operands…, n)` / `RAD_ISSUE` (n = the batch's ranged extent) /
+  `RAD_ISSUE_N` (an explicit row count). `rad_step_fail(c, what)` sets the status helpers return.
+- **A mixed step is the trap.** Continuous batching produces steps with a prefill chunk beside decode
+  rows. A block that issues one path for the whole step sends its decode rows down the chunked path,
+  which commits recurrent state directly and takes no `num_accepted`; a rejected draft cannot be
+  rolled back, and the model stays fluent while drifting. Issue the recurrent path over
+  `RadBatch::n_seq_decode` sequences and the chunked path over the rest (the core orders steps
+  decode-first, so each half is contiguous; `cu_seqlens` holds absolute offsets, so pass
+  `cu + n_seq_decode` with the token operands at their base).
+- A step issued on lane 1 outside the declared window fails by name. An op the core appends behind a
+  declared one (runtime LoRA) counts at that op's position.
+- **Zero rows is not a call**: issuing an op over zero rows (a zero ranged extent, or all-empty
+  sequences) returns `RAD_E_SHAPE` and writes nothing. The caller skips the issue, as the
+  architectures' decode and prefill halves do.
+- `n_out` is 0 for a chunk that finishes no sequence — do not run the lm_head.
+
+## Probe: what the plugin measured
+
+`rad_arch_probe(meta, out)` runs before declare. Its fields are the engine's permissions:
+
+| field | effect |
+|---|---|
+| `draft_depth`, `draft_depth_fixed` | the container's spec depth; a block drafter's trained block is the only depth it serves |
+| `shape_probe_ok` | the plugin tolerates a sizing declare, so the activation arena can be lent |
+| `enc_merge`, `audio_frames` | encoder pass granularity; longest audio clip |
+| `gen_rows`, `gen_patches`, `gen_audio`, `gen_ctx`, `gen_video` | generator sizing before declare |
+| `kv_quant_ok` | bits of the narrow KV layouts this model's kernels serve |
+| `chain_pays`, `tree_pays` | the plugin measured chained launches / draft trees faster |
+| `kv_stream_ok` | the architecture can stream a KV group (sparse, block-addressed, with `kv_resolve`) |
+| `control_vector_ok` | the plugin applies `RadBuildCtx::cvec` |
+| `tp_attn_rows_ok` | the plugin serves row-split prefill attention |
+| `state_f16_ok` | the plugin declares recurrent state at f16 and its kernels read it |
+
+A flag whose probe bit is 0 is **refused at startup with the plugin named** (`--kv-cache-dtype
+fp8_v4`, `--control-vector`, `--tp-attn-split rows`, `--state-cache-dtype f16`, `--kv-resident`).
+That is the design: a deployment switch the plugin cannot honour is an error, never a silent
+substitution.
+
+## Worked pattern: the smallest complete plugin
+
+`arch/llama_fp8/llama_fp8.cpp` builds twice from one source — `llama_fp8` (quant `fp8_e4m3`) and
+`llama_dense_fp8` (quant `""`), differing only in the descriptor and the graph-dump text. It shows
+the whole shape:
+
+1. `struct Layer { AttnFP8 attn; MlpFP8 mlp; }`, `struct Model` holding `Names`, `Geom`, buffer and
+   op handles.
+2. `declare()`: name map entries first, then weights (with `rad_weight_encoding` where the head's
+   format decides the kernel), buffers, KV group, the embedding and its all-reduce, then each layer's
+   `declare()` from the shared blocks, the final norm, gather and lm_head, **the drafter last**, and
+   `rad_note` lines explaining the shape.
+3. `step()`: draft passes first (sign selects query/context), then embedding (+ exact all-reduce
+   when needed), each layer's `step()`, the final norm, and the head only over `n_out`.
+4. `probe()`: reads `draft.block_size` from the metadata and refuses any other depth for a block
+   drafter.
+5. `RAD_ARCH_PLUGIN(...)` at the bottom.
+
+Add a new family by copying a neighbour, replacing the name map and geometry, and composing the
+blocks you need; add a new *format* by cloning the plugin with another `(id, quant)` pair and the
+ops that format needs.
+
+## Verifying a new plugin
+
+```sh
+radiance --model m.rad --debug-graph          # every op, its resolved kernel, and why
+radiance --model m.rad --debug-placement      # the plan
+rad-convert $SRC -o m.rad --plan-only -v      # a convert that plans will boot
+rad-kbench -m m.rad --kernels mylib,libr4d,libref --report k.md --bench
+rad-tune -m m.rad --tp 2                      # only at the shapes you serve
+```
+
+`tests/arch_test.cpp` drives a plugin's builder the way the core does, including lane-window
+liveness, so the pinned graphs and steps are the ones the core plans. A plugin that declares a weight
+form no kernel `layout` accepts is a declare-time error naming the weight, encoding and kernel — the
+same check `rad-convert` runs before writing.
+
+## Related
+
+- [[Radiance Plugins]] — kernel libraries, encodings, the build path, ABI law.
+- [[Radiance Conversion]] — how the name map drives a convert, and `--set`/descriptor selection.
+- [[Radiance Model Families]] — the existing `(id, quant)` pairs and their recipes.
+- [[Radiance Tools]] — `rad-kbench`, `rad-tune`, `rad-info --plugins`.

serverAppConfigs-docs/LLM/Radiance Conversion.md

@@ -0,0 +1,354 @@
+---
+type: "howto"
+title: "Radiance Conversion — checkpoints into .rad containers"
+description: "Turning a checkpoint into a .rad container: rad-convert, the recipe language, quantisers, reuse and export."
+tags:
+  - llm
+  - radiance
+  - quantization
+  - howto
+timestamp: 2026-10-30
+---
+
+# Radiance conversion
+
+`rad-convert` turns a Hugging Face checkpoint into a `.rad` container, choosing each weight's
+encoding from a **recipe**. `rad-export` wraps it for finetunes (a merged checkpoint or a PEFT
+adapter) with the stock model's recipe. Neither opens a GPU: conversion is CPU-only and safe beside a
+running server.
+
+Fetch checkpoints with `scripts/hfget.sh <org/repo> <dir>` (resumable; reads `.safetensors`,
+skips `.bin`/ONNX/GGUF when safetensors exist; `HF_TOKEN` for gated repos; writes a
+`.hfget-manifest` of content keys). `radiance --model <owner/name>` runs the same fetch itself.
+
+## What a conversion does
+
+1. Read `config.json` (or GGUF metadata), and the shard index if present.
+2. Load the architecture, kernel and quantiser plugins and run **declare**, as the engine does.
+3. Resolve every op and ask each resolved kernel's `layout` hook about the planned encoding. A
+   recipe no kernel can serve is refused **here**, before any I/O.
+4. Write each logical weight once: the checkpoint's bytes where no rule matches, otherwise the
+   planes the matching quantiser writes, a block of rows at a time.
+5. Measure every quantised weight against its source through the core's decoder and print the worst.
+6. Bake the tokeniser, chat template, metadata, recipe and any draft head.
+
+```sh
+build/bin/rad-convert /srv/models/hf/Qwen/Qwen3.8-27B -o qwen3.8-27b-bf16.rad \
+    --recipe data/recipes/q38-27b-bf16.recipe
+build/bin/rad-convert /srv/models/hf/Qwen/Qwen3.8-Flash-Next -o flashnext-bf16.rad --max-ctx 4096
+```
+
+## Every `rad-convert` option
+
+| option | meaning |
+|---|---|
+| `<input>` | a `.gguf`, a `.safetensors`, an `index.json`, or a checkpoint directory |
+| `-o, --out FILE` | the container to write |
+| `--recipe FILE` | one rule a line: `PATTERN QUANTISER key=value …`; repeatable; first match decides |
+| `--quant RULE` | `PATTERN=QUANTISER[:key=value,…]`, placed **ahead of** the recipe's rules; repeatable |
+| `--list-quantizers` | every loaded quantiser and the options it takes, then exit |
+| `--arch ID` | override the architecture id read from the checkpoint |
+| `--home DIR` | `$RADIANCE_HOME` (kernels, architectures, quantizers) |
+| `--kernels A:B` | kernel libraries to load, first wins; declare checks the recipe against them |
+| `--tokenizer F` | `tokenizer.json`; its declared chain is baked instead of reconstructed from a pre-tokenizer name. Required for a GGUF, without it the chain is a guess |
+| `--tokenizer2 F` | a second `tokenizer.json` baked as the second vocab (a generator reading its prompt through two tokenisers); keep `tokenizer_config.json` beside it |
+| `--add FILE` | merge one more `.safetensors`, tensor names kept; repeatable; a name in two files is refused |
+| `--config DIR` | read `config.json`, `generation_config.json`, `tokenizer.json` from DIR for a single-file checkpoint |
+| `--chat-template F` | a Jinja template baked in, overriding the checkpoint's (or its absence); trailing newlines stripped; without one `/v1/chat/completions` answers 501 |
+| `--imatrix F` | llama.cpp imatrix: expert-popularity profile, per-column importances, error weighting; required by `ggml` types `iq2_xxs`, `iq2_xs`, `iq1_s` |
+| `--set K=V` | a metadata key, seen by declare **and** written into the container; repeatable |
+| `--max-tok N` | the `max_tok` declare runs at (default 8192, or a generator's largest pass) |
+| `--max-ctx N` | the context declare runs at (default: the checkpoint's training context) |
+| `--max-spec N` | the `max_spec` declare runs at (default 4, or the block a drafter states); `0` leaves a draft head out |
+| `--draft-model DIR` | a drafter shipped as its own repository, merged under `draft.` |
+| `--kv-cache-dtype T` | `bf16` or `fp8`: the cache width declare runs at; the container is identical at either |
+| `--reuse FILE` | copy every weight an earlier container holds with the same encoding, quantiser, options and shape |
+| `--changed FILE` | with `--reuse`: checkpoint tensors (one name a line) that differ; every weight reading one is converted |
+| `--in-place` | with `--reuse`: extend that container; new weights go past its end, the header last; keeps `<file>.pre-append` |
+| `--lora FILE\|DIR` | merge a PEFT adapter into the checkpoint tensors before converting (`W + strength · scale · B·A`); repeatable, merged in order; each merged tensor joins `--changed` |
+| `--lora-strength F` | strength for every `--lora` (default 1) |
+| `--lora-map FILE` | JSON map for adapters whose module names differ: `{"strip":[prefix…],"modules":{"<pat>":[target…]}}`; one `*` matches a block index; several targets mean a fused module (rows split in order) |
+| `--plan-only` | declare, plan every encoding, print the plan, stop before writing |
+| `--report N` | print the N worst quantisation errors (default 20) |
+| `-v, --verbose` | per-weight detail |
+
+`--reuse` matches a weight on its **encoding, quantiser, options and shape — not on its source
+bytes** — and carries whatever calibration made the old planes. Use it only when source and
+calibration are unchanged, or name the changed tensors with `--changed`. `--in-place` refuses to
+change a weight the container holds or to add an expert.
+
+## The recipe language
+
+```
+# PATTERN                      QUANTISER  OPTIONS
+blk.*.ffn_*_exps.*.weight      gptq       codes=i4 group=128 scale=bf16 transform=fwht128 calib=$CALIB
+blk.*.ple_ngram.weight         rtn        codes=fp8_e4m3 block=*x* scale=bf16 rule=fixed scale_value=1.99317932128906e-4
+output.weight                  rtn        codes=fp8_e4m3 block=128x128 scale=bf16
+blk.*.weight                   rtn        codes=fp8_e4m3 block=128x128 scale=bf16
+```
+
+- One rule a line: `PATTERN QUANTISER [key=value …]`. `#` starts a comment.
+- `PATTERN` is a glob over the **whole logical weight name**: `*` crosses dots, `?` is one character.
+- **The first matching rule decides.** No rule → the checkpoint's own encoding, which must be one
+  the engine serves as is: `bf16`, `f16`, `f32`, or block-scaled FP8 (`X.weight` with
+  `X.weight_scale_inv`). Anything else is named and refused.
+- `$NAME` / `${NAME}` in a value comes from the environment; an **unset variable is refused**, never
+  expanded empty (so an empty `$CALIB` cannot silently produce an uncalibrated convert).
+- A quantiser may **decline** a weight (`RAD_E_UNSUPPORTED`), which keeps the checkpoint's encoding
+  — `rtn` declines a rank-1 tensor, so a pattern that also matches a norm leaves the norm alone.
+- An option the quantiser does not declare is refused, by name, before anything is quantised.
+- Two plugins naming the same quantiser are refused, both named.
+
+## Quantisers
+
+| quantiser | what it does |
+|---|---|
+| `rtn` | round to nearest on a grid; each block's scale chosen by `rule` |
+| `gptq` | error feedback against the layer's input Hessian (`calib=DIR`, damped by `damp` = 0.01 of the mean diagonal by default); `act_order=1` quantises columns by descending input energy and stores the permutation; `cd=N` adds coordinate-descent sweeps |
+| `awq` | per-input-column scaling to a searched power, then a per-block clip search, both against the Hessian; column scales are the last scale level |
+| `autoround` | sign-gradient descent on a rounding offset per value and a range factor per block; `iters` steps; slow (a 27B dense model takes most of a working day at defaults; `iters=50` a quarter) |
+| `paroquant` | channel scaling plus `stages` Givens rotations learned against the Hessian, then AutoRound on the rotated weight; the `givens` transform |
+| `m2xfp4` | M2XFP's weight-side Sg-EM: MXFP4 plus one metadata byte per group of 32; its own `m2xfp4` scheme; `adaptive=0` pins the bias, `wide=1` widens it |
+| `ggml` | a llama.cpp format (`q4_0`, `q4_1`, `q5_0`, `q5_1`, `q8_0`, `q2_K`…`q6_K`, `iq4_nl`, `iq4_xs`, `iq2_xxs`, `iq2_xs`, `iq2_s`, `iq3_xxs`, `iq3_s`, `iq1_s`, `iq1_m`), by llama.cpp's own search, imatrix-weighted, codes byte for byte |
+| `cast` | a float weight in another float dtype (`dtype=` f32, bf16, f16, fp8_e4m3, fp8_e5m2), no scale |
+| `import` | rad-convert's own, not a plugin: write the numbers a quantised release ships, in the spelling the kernels read, instead of quantising again |
+
+Every quantiser except `m2xfp4`, `ggml`, `cast` and `import` takes the grid options:
+
+| option | values |
+|---|---|
+| `codes` | `i2`…`i8`, `i16`, `i32` symmetric; `u1`…`u8`, `u16` with `zero=`; `fp8_e4m3`, `fp8_e5m2`, `fp6_e2m3`, `fp6_e3m2`, `fp4_e2m1` |
+| `table` | a codebook the codes index: `nf4`, `iq4nl`, `w4nl`, `w5nl`, `binary`, `ternary`, or `;`-separated values |
+| `group=N` / `block=RxC` | where the scales sit; `*` is a whole extent |
+| `scale` | f32, bf16, f16, fp8_e4m3, e8m0; under `scale2` also an integer sub-scale `u4`…`u8`, `i6`, `i8` |
+| `scale2`, `group2` / `block2` | a second scale level the first is relative to (default the whole tensor) |
+| `scale2_value` | the second level fixed at this value, exact in `scale2`'s dtype |
+| `zero` | an integer zero point per block — an asymmetric grid |
+| `transform=fwhtN` | rotate each row by the Hadamard first |
+| `rule` | `absmax`, `mx` (OCP shared exponent), `fixed` (`scale_value=`), `search` (least squared error over candidates and refits, imatrix-weighted) |
+| `clamp` | a signed code's range: `full` or `sym` |
+
+Named grids: MXFP4 = `codes=fp4_e2m1 group=32 scale=e8m0`; NVFP4 =
+`codes=fp4_e2m1 group=16 scale=fp8_e4m3 scale2=f32`; NF4 = `table=nf4 group=64`; a K-quant-like grid
+= `codes=i4 group=32 scale=u6 scale2=f16 group2=256`; a ternary BitNet weight = `table=ternary`.
+
+**A container can hold only what some kernel reads.** rad-convert checks every planned encoding
+against the `layout` hooks of the kernels declare resolved and refuses one no kernel accepts. libr4d
+reads the plain floats; block-scaled fp8 and fp8 rows; int8 block rows plain or rotated; routed
+experts as fp8, int4 (`i4*bf16[1x128]/fwht128`) or the `w4nl`/`w5nl` codebooks under E4M3 scales;
+the dense w4, w8 and mxfp4 GEMM rows; and the 2-bit draft head. See [[Radiance Plugins]] for the
+encoding model.
+
+## Importing a release someone else quantised
+
+These forms are read weight by weight and written in the spelling the kernels read; a recipe rule
+re-quantises from the exact dequantisation where the family's kernels want another form.
+
+| `quantization_config` | read as | notes |
+|---|---|---|
+| `quark` MXFP4 (fp4, group 32, e8m0) | `fp4_e2m1*e8m0[1x32]` | byte for byte; also servable as it stands |
+| `paroquant_mxfp4` | `fp4_e2m1*e8m0[1x32]/givens` | codes/exponents byte for byte plus learned rotations as `givens` tables |
+| `paroquant`, `int5-bitplane`, plain `paroquant` | `u5*f16[1x128]-u8[1x128]/givens`, `u4*…` | AWQ-packed nibbles re-laid as a code bit stream; f16 scales, integer zeros, rotations |
+| `compressed-tensors` `pack-quantized`, 4-bit `int`, group, act-order none/static/weight | `i4*<scale>[1xG]` | `weight_packed` made signed beside `weight_scale`; group/dynamic reorder is refused |
+| same with `symmetric:false` (AWQ) | `u4*<scale>[1xG]-u4[1xG]` | nibbles plus `weight_zero_point`, bit for bit |
+| `auto-round` `auto_round:auto_gptq`, `gptq` v1 | `i4*bf16[1x128]` or `i8*bf16[1x128]` | scales rounded to bf16; convert-time; `import` keeps shipped f16 |
+| `nvfp4-pack-quantized` | `fp4_e2m1*f32[1x16]` | `weight_scale / weight_global_scale`; **no kernel serves NVFP4**, so it always takes a rule |
+| `modelopt` NVFP4 / FP8 / MIXED_PRECISION | NVFP4 `fp4_e2m1*f32[1x16]`; FP8 `fp8_e4m3*f32[1x*]` | ModelOpt's second scale **multiplies**; `input_scale` is not read |
+| compressed-tensors all-FP8 groups | `fp8_e4m3*<scale>[1x*]` or `[RxC]` | nothing imported: stock FP8 with its scale; a per-tensor scale is written once a row |
+| GGUF `Q8_0`, `Q4_K`, `Q6_K` | `i8*f16[1x32]`, affine u4/u6…, `i6*i8…` | blocks split bit for bit; **not servable as they stand** |
+
+The width is read **per tensor**, so a mixed release imports int4 experts and int8 projections
+together. What an import cannot read (an asymmetric GPTQ zero point, a group-reordering act-order
+index, a GPTQ group under 128, a missing/zero global scale, an unlisted format) is refused by name.
+The release's `quantization_config` names the tool, not a served form, so it is kept as
+`imported_quantization_config.*`.
+
+`import` options: `codes=uN` makes a symmetric grid unsigned under a zero point of `2^(N-1)`;
+`zero=` the zero plane's dtype (`u4`, `u8`); `scale=` f16/bf16/f32, each shipped scale rounded to
+nearest; `block=RxC` repeats an FP8 scale shipped per row/tensor over that grid. It reads an imported
+integer grid or FP8 with a per-row/per-tensor scale and keeps a plain float weight as it is.
+
+**NVFP4 to MXFP4 rounds twice** (e4m3 per 16 → power of two per 32), so it loses more than MXFP4 made
+from bf16. Prefer a calibrated MXFP4 container when one exists; the quality verdict is a KL gate, not
+a file size.
+
+## The descriptor law (`--set`)
+
+The architecture plugin is chosen by the exact `(arch id, quant descriptor)` pair. A convert-time
+choice that must also hold at serve time is stated once with `--set`, which hands it to declare and
+writes it into the container. Storage format is not such a choice (the encoding is in the container
+and the plugin reads it there), but the **serving plugin** is: a `quantization_config.*` key
+re-derives the descriptor.
+
+```sh
+rad-convert <bf16 Qwen3.8-27B> -o q38-27b-fp8.rad \
+    --recipe data/recipes/q38-27b-fp8block-df2.recipe \
+    --set quantization_config.quant_method=fp8 --set quantization_config.fmt=e4m3 \
+    --set "quantization_config.weight_block_size=128 128" \
+    --set quantization_config.activation_scheme=dynamic
+```
+
+A bf16 checkpoint whose recipe writes Qwen's block FP8 must therefore be declared *as that format*;
+otherwise the selection pair does not match any plugin and the convert (or the boot) is refused. The
+same law bites the other way when a release's own `quant_method` names a pair no plugin claims.
+
+## Inspect and validate the result
+
+```sh
+rad-info out.rad                  # architecture, source quant, recipe, sections
+rad-info --recipe out.rad         # the expanded rules — did the intended one win?
+rad-info -v out.rad | head -40    # every tensor, its encoding, and what made it
+rad-kbench -m out.rad --home $RADIANCE_HOME --report kbench.md
+radiance --model out.rad --debug-graph     # every op resolves
+```
+
+The converter's own error report (the worst `--report N` weights) measures each quantised weight
+against its source through the core decoder. `rad-kbench` adds the artifact test: weight operands
+read from the container and passed through each kernel's `layout`/`relayout` hooks. Both are on
+[[Radiance Tools]].
+
+## Calibration (GPTQ, AWQ, AutoRound, ParoQuant)
+
+`calib=DIR` is a directory an engine run wrote with `RADIANCE_CALIB_DIR` set: each layer's **Gram
+matrices of its input**, in the domain the kernel multiplies it in. A block that characterises an
+activation declares an f32 `[K,K]` PERSIST buffer and a `gram_accum` op; at `RADIANCE_CALIB_TOKENS`
+(default 65536) the run writes the files.
+
+Three-step bootstrap: convert with plain `rtn` → serve that container with `RADIANCE_CALIB_DIR` →
+push real prompts through prefill → convert again with the Hessians. The second moment is robust to
+the first container's small quantisation error. A weight with no file is quantised uncalibrated and
+the converter says so; `awq`, `autoround` and `paroquant` fall back to the imatrix diagonal when one
+is given.
+
+- The accumulator must be **PERSIST**: a transient buffer may share bytes and be overwritten between
+  the launches it sums.
+- Files are per rank; the consumer sums them. Under expert parallelism a rank's `down` input covers
+  only its experts' tokens; `gate_up` input is the full token set on every rank, so the sum scales
+  `H` by world size — GPTQ is invariant to a positive scale.
+- Under tensor parallelism a column-split linear sees only its rank's columns, so a tap may write
+  `gram.r<rank>.<stem>.part.bin`; libquant places the parts on the diagonal in rank order, leaving
+  cross-rank products zero. They must cover K and be in the model's own domain.
+- `--imatrix` supplies the expert-popularity profile the placement planner ranks units by, the
+  per-column importances a quantiser uses, and the error report's weighting.
+
+## Reuse, drafters, merging
+
+```sh
+# re-quantise only what changed
+rad-convert $SRC -o new.rad --recipe new.recipe --reuse old.rad --changed changed.txt
+
+# merge a drafter shipped as its own repo (DFlash2, DSpark, MTP assistant)
+rad-convert $SRC -o out.rad --draft-model /srv/models/z-lab/Qwen3.8-27B-DFlash2 \
+    --recipe data/recipes/q38-27b-mxfp4-df2.recipe
+
+# merge a PEFT adapter at convert time
+rad-convert $SRC -o tuned.rad --lora ~/run/adapter --lora-strength 1.0 \
+    --recipe data/recipes/q38-27b-mxfp4-df2.recipe
+```
+
+- A drafter's tensors go in under `draft.`, its config under `draft.` keys. It has no embedding and
+  no `lm_head` of its own; it uses the target's.
+- `--lora-map` exists for adapters trained on names the checkpoint does not use; a fused target
+  (`qkv`) takes several targets and splits the adapter's rows in order.
+- `--add` + `--config` + `--tokenizer2` assemble a ComfyUI-style release of separate single files
+  (Anima: diffusion model, text encoder, VAE). A name in two files is refused.
+
+## `rad-export` — finetunes without hand-writing the command
+
+```sh
+rad-export ~/outputs/my-run/checkpoint-400 --dry-run   # what it would run
+rad-export ~/outputs/my-run/checkpoint-400             # adapter -> <name>.rad
+rad-export ~/exports/merged -o ~/models/mine.rad --recipe q38-27b-fp8block-df2
+```
+
+| flag | default | meaning |
+|---|---|---|
+| `SRC` | required | a merged 16-bit checkpoint (`config.json` + `*.safetensors`) or a PEFT adapter (`adapter_config.json` + `adapter_model.safetensors`) |
+| `-o, --out FILE` | SRC's folder name + `.rad`, beside SRC or in `out_dir` | the container |
+| `--recipe NAME\|FILE\|auto` | `auto` | a recipe name, a file, or the one recipe whose header names the base model, takes a bf16 checkpoint and needs no calibration |
+| `--base DIR` | the adapter's `base_model_name_or_path`, from disk or the HF cache, else downloaded | the base a LoRA merges into |
+| `--draft DIR\|ORG/NAME` | the config, else the Hub repo the recipe's usage line names | the drafter when the recipe merges one; a usage line with a placeholder asks for `--draft` |
+| `--image IMAGE` | `$RADIANCE_IMAGE`, the config's, then `stilldeadcode/radiance` | run `rad-convert` in this image (docker, CPU-only, no network, **not pulled**) |
+| `--rad-convert PATH` | — | run this binary instead of docker; recipes come from its `share/radiance` |
+| `--memory M` | — | docker memory limit for the convert |
+| `--dry-run` | — | resolve recipe, drafter and paths; print the `rad-convert` line; write nothing |
+| `--keep-merged` | — | keep the merged checkpoint beside OUT |
+| `--config FILE` | `~/.config/rad-export/config.json` | the config below |
+| `--list-recipes [--json]` | — | list the recipes the image/installed tool carries |
+
+- Input: a merged 16-bit checkpoint (`config.json` + `*.safetensors`) or a PEFT LoRA adapter
+  (`adapter_config.json` + `adapter_model.safetensors`).
+- An adapter is merged **exactly as PEFT's `merge_and_unload`** computes it on CPU: `scale` =
+  alpha/r (alpha/√r with rsLoRA), `rank_pattern`/`alpha_pattern` honoured, delta in fp32 rounded to
+  the adapter's dtype. It streams tensor by tensor into 1 GiB shards, so a 27B merge needs a few GiB
+  of RAM, not the model twice. Plain linear LoRA only: embedding LoRA, trained biases, DoRA, saved
+  modules, convolution LoRA, a rank other than the config's, and a missing target are refused before
+  anything is written. The adapter's tokeniser/template replace the base's where shipped.
+- The recipe comes from the image (`/opt/radiance/share/radiance/recipes`) or the installed
+  `rad-convert`'s share; `auto` matches the base model's name in recipe headers and asks when several
+  fit. A recipe needing calibration is listed but never picked by `auto` — a finetune's Hessians are
+  not the stock model's.
+- It runs `rad-convert` on the CPU, under Docker as the caller with no network, and writes the log to
+  `OUT.log`.
+- `~/.config/rad-export/config.json` pins what auto cannot know: `image`, `out_dir`,
+  `recipes{model:recipe}`, `drafts{recipe:dir}`, `bases{repo:dir}`.
+- It is Python 3 with no dependencies; merging needs `torch`/`safetensors`, fetching a missing base or
+  drafter needs `huggingface_hub`. The image carries no Python; extract the script with
+  `docker run --rm --entrypoint busybox stilldeadcode/radiance cat /opt/radiance/bin/rad-export`.
+
+## Scoring a container (the KL mode)
+
+```sh
+# once, on the bf16 container
+radiance --model bf16.rad --kld-record /srv/kld/q38 --kld-corpus corpus.jsonl
+# each candidate, same flags otherwise
+radiance --model quant.rad --kld-ref /srv/kld/q38 --kld-out report.json
+```
+
+- The corpus is JSONL, one `{"prompt", "score_from"?, "source"?}` a line; only `prompt` is required.
+  Positions `[score_from, n-1)` are scored; `source` groups the report. Keep the scoring corpus
+  disjoint from the calibration corpus.
+- `--kld-record` writes the reference's log-probabilities (f16 over the vocabulary, the 16 most
+  likely tokens exactly, the token ids). DIR is never overwritten and is read only once `kld.json`
+  exists, which is written last.
+- `--kld-ref` scores this model's f32 logits against stored rows: mean, median, 99/99.9/99.99th
+  percentile KL, top-1 agreement, and both models' perplexity. `--kld-out` writes the report; `FILE.rows`
+  is `[positions, 4]` f32 (KL, both NLLs, argmax agreement) for position-by-position comparison.
+- The mode turns speculation and the prefix cache off, turns the step pipeline off, and refuses an
+  explicit spec depth above 0. The logits buffer holds a row per token of a step, so a large
+  `--max-num-batched-tokens` costs pinned host memory as well as VRAM.
+- `--kld-lora NAME` scores every document through a served adapter, against a reference recorded
+  from the merged checkpoint; `--kld-lora-sources PREFIX` runs only matching documents through it,
+  the rest as the base, in the same steps — each document's rows should be byte-identical to the
+  all-adapter and all-base runs.
+
+The gate the fleet uses (mean KL ≤ 0.03, top-1 ≥ 95 %) is on [[frank-archon]] and
+[[Quantization Production]]; the mode here is only the instrument.
+
+## Convert laws paid for in full
+
+- **`--plan-only` first.** A plan that resolves every op will boot; it costs seconds against an hour
+  of I/O on a large checkpoint.
+- **Declare runs at the step and context bounds, and some graphs depend on them.** Qwen4-Exp sizes
+  its QSA buffers by `--max-ctx` and refuses a `--max-tok`/context pair whose score plane would pass
+  2 GiB; the defaults (8192 tokens, the checkpoint's training context) fit most models, and a
+  generator's largest pass may override `--max-tok`.
+- **Scale names follow the tensor.** `scale_name()` appends `_scale_inv` to the *tensor* name, so a
+  stacked `…experts.gate_up_proj` keeps its scales at `…gate_up_proj_scale_inv`.
+- **File shadowing: first wins.** The reader opens index-named files sorted and emplaces every tensor
+  each file physically holds, first-wins. A merged PLE shard must sort first
+  (`00-ple.merged.safetensors`) or a raw shard wins the name.
+- **Never write through a staging symlink.** A checkpoint may symlink its raw index; `open(path,"w")`
+  writes through it and corrupts the checkpoint. Write a temp file and `os.replace`.
+- **One encoding per expert table.** A whole expert kept bf16 is the only thing legal beside a
+  table; a split tier (down=fp8, gate_up=w4) is unservable.
+- **The descriptor pair has no fallback** (above), and **convert is card-less**: GPTQ solves on CPU
+  BLAS, so the GPU is irrelevant to production and the card only decides what can be decoded.
+
+## Related
+
+- [[Radiance Model Families]] — the (arch id, quant descriptor) table and the command per model.
+- [[Radiance Plugins]] — quantiser and kernel-plugin contracts behind these options.
+- [[Radiance Tools]] — `rad-info`, `rad-kbench`, `rad-tune`.
+- [[Quantization Production]] — the fleet's calibration/KLD campaign and its gates.
+- [[frank-archon]] — the deployment whose recipes and convert laws these are.

serverAppConfigs-docs/LLM/Radiance Engine.md

@@ -0,0 +1,140 @@
+---
+type: "reference"
+title: "Radiance Engine — the inference server behind frank-archon"
+description: "The radiance inference engine: its shape, plugin model, container format, and how the reference pages fit together."
+tags:
+  - llm
+  - radiance
+  - reference
+timestamp: 2026-10-30
+---
+
+# Radiance Engine — shape, plugin model, container
+
+Radiance is StillDeadcode's inference server for AMD GPUs, written in C++20 and HIP: language
+models, plus image, video, speech and music generation and embeddings. One HTTP front end speaks
+OpenAI's chat, completions, Responses, images, audio and video APIs, Anthropic's Messages API and
+llama-server's native one. Nothing in the serving path is Python.
+
+The fleet's only radiance endpoint is [[frank-archon]] (2× R9700, TP=2, Dankserver). **This page
+and its siblings own the engine reference; the live deployment shape, its Eras and its measured
+laws stay on the endpoint page.** The numbers here describe upstream `main` as read 2026-10-30
+(release 1.14.0, plugin ABI 31). The fleet pin is older (1.8.1): a flag below that post-dates the pin
+is a boot refusal, never a silent no-op, and the endpoint page records the CLI families that pin
+itself retired relative to its own rollback target.
+
+Source: `https://codeberg.org/StillDeadcode/radiance` at `8da043c`. Engine docs live in `docs/`
+(`GUIDE.md` for serving, `TOOLS.md` for tools, `PLUGIN.md` for plugins, `ARCHITECTURES.md` per
+family, `OPS.md` for the op vocabulary, `spec.md` for the design).
+
+## The four moving parts
+
+```mermaid
+graph LR
+    A[Checkpoint<br/>safetensors / GGUF] -->|rad-convert + recipe| B[.rad container]
+    B --> C[radiance engine]
+    C --> D[OpenAI / Anthropic /<br/>llama-server wire APIs]
+    P1[kernel library .so<br/>libr4d, libavx, libref] --> C
+    P2[architecture .so<br/>one per arch id × quant descriptor] --> C
+    P3[quantiser .so<br/>libquant] -->|rad-convert| A
+```
+
+- **Engine** — `core/`: declare phase, placement/memory planner, scheduler, KV manager, sampler,
+  tokeniser, HTTP server, media preprocessing.
+- **Kernel libraries** — implement ops for hardware. `libr4d` targets RDNA4 (`gfx1200`,
+  `gfx1201`); `libavx` serves the host backend (AVX-512/AVX2) with no GPU at all; `libref` is the
+  slow reference implementation every other library is checked against.
+- **Architecture plugins** — declare one model family at one weight format: its weights, buffers,
+  KV groups and the ops it issues each step. The engine picks one by an **exact (architecture id,
+  quantisation descriptor) pair, with no fallback**.
+- **Quantiser plugins** — `libquant`'s `rtn`, `gptq`, `awq`, `autoround`, `paroquant`, `m2xfp4`,
+  `ggml`, `cast`, `import`; they write a weight's numbers into an encoding's planes.
+- **Tools** — `rad-info`, `rad-convert`, `rad-export`, `rad-kbench`, `rad-tune`, `rad-schemas`,
+  `rad-tokcheck`, plus `rad-chatparse`, `rad-avx-check`, `r4d_selftest` in the build tree.
+
+## Two phases, one static graph
+
+`rad_arch_declare` runs once per rank and names every weight, buffer, KV group and op, in order.
+The core resolves every op against the kernel hierarchy per band, per domain, and completes the
+whole plan before a byte of weights is read. `rad_arch_step` then only *issues* handles obtained at
+declare; it may not allocate and may not synchronise. A model whose declare succeeds is fully known
+statically, which is what makes `--debug-graph`, the buffer liveness plan and `rad-convert`'s
+plan-only check possible.
+
+Consequences a plugin author must respect:
+
+- An op that never resolves is `RAD_NULL_HANDLE` and only fails if issued. A fusion is a *selection*:
+  ask for the fused op, emit the chain when `rad_op_resolved` is false.
+- Declaration order is the buffer planner's liveness. A drafter must be declared after the trunk
+  it runs behind, or trunk and drafter buffers silently share bytes.
+- Every op in the graph must have a row in a loaded kernel library. `libref` implements the
+  conventional vocabulary in `docs/OPS.md`; device-only ops (most fusions) exist only in `libr4d`.
+
+## The `.rad` container (RAD2)
+
+A container is one file holding everything the server needs: weights as **canonical encoding
+planes**, the tokeniser (and optionally a second vocab), the chat template, metadata, the recipe it
+was made with, and any draft head. It names no kernel library.
+
+- Planes are kernel-independent: codes, scales, zero points, codebooks — row-major, bit-packed
+  sub-byte where needed. A kernel rearranges them into its own layout at load (`layout`/`relayout`
+  hooks), losslessly, so a kernel library can change layouts without reconverting the model.
+- Encodings are data, not an enum: `RadEncoding` names the scheme (`plain`, `affine`, or a
+  quantiser's own), the planes by role, the scale grouping, and a transform (`fwhtN`, `perm`,
+  `givens`). That is how one file can hold block-FP8, int4, MXFP4, NVFP4-imported, ParoQuant
+  rotations and llama.cpp grids.
+- `rad-info` is the reader: sections (strings, metadata, directory, planes, encodings, vocab,
+  profile, data), `--meta`, `--vocab`, `--recipe`, `--plugins`, and `-v` for every tensor with its
+  encoding and what made it.
+
+See [[Radiance Conversion]] for producing one and [[Radiance Tools]] for inspecting it.
+
+## Selection key and version law
+
+- A checkpoint's architecture id is its `config.json` `model_type` without underscores; `rad-convert
+  --arch ID` overrides it. `rad_arch_quant` supplies the quantisation descriptor (empty for an
+  unquantised checkpoint).
+- **No fallback.** A raw official FP8 checkpoint whose `quantization_config.quant_method` is `fp8`
+  is not the pair the `qwen4exp_fp8` plugin claims (`(qwen4exp, "")`), so both serve and convert
+  refuse it until the descriptor is changed with `--set` or a staged `config.json`. The deployment
+  lesson is on [[frank-archon]]; the mechanism is on [[Radiance Conversion]].
+- `RAD_ABI_VERSION` covers every header a plugin compiles against. A plugin reporting a different
+  number is refused at load, with both numbers printed. It moves on any shape change, not on a new
+  optional export; rebuild when it changes. Release version and ABI are independent: `/version`
+  reports the release, `rad-info --plugins` reports what loaded.
+
+## Environment
+
+| variable | meaning |
+|---|---|
+| `RADIANCE_HOME` | plugin search path: `:`-separated homes, each holding `kernels/`, `architectures/`, `quantizers/`; first wins, same-name files shadow later homes. Compiled-in default is `<prefix>/share/radiance` |
+| `RADIANCE_KERNELS` | default for `--kernels`: ordered kernel libraries, first wins; unnamed libraries are not loaded |
+| `RADIANCE_TUNE` | pin a kernel's tuned axes for a measurement (a measurement aid, not a deployment setting) |
+| `RADIANCE_CALIB_DIR` | a serve run records per-layer input Gram matrices here (the GPTQ calibration tap) |
+| `RADIANCE_CALIB_TOKENS` | tokens after which the tap stops and writes (default 65536) |
+| `RADIANCE_LOG`, `RADIANCE_LOG_STEPS` | log level; per-step token counts |
+| `RADIANCE_DEBUG_RESID` | residual-stream probe per (layer, site) |
+| `RADIANCE_DFLASH_Q_TEMP` | temperature of a DFlash2 coupled walk's candidate softmax |
+
+The full list of debug knobs (`RADIANCE_DEBUG_*`, `RADIANCE_DUMP_*`, `RADIANCE_OP_ORACLE`, …) is in
+the source's `getenv` calls; none is a deployment setting.
+
+## Where the pages are
+
+| Page | Owns |
+|---|---|
+| [[Radiance Serve Options]] | every serve flag, grouped, with defaults and the law behind each knob |
+| [[Radiance HTTP API]] | endpoints, request fields, streaming, tool calls, refusal rules |
+| [[Radiance Conversion]] | `rad-convert`, the recipe language, quantisers, imports, reuse, `rad-export`, KL scoring |
+| [[Radiance Model Families]] | which plugin claims which model, and the conversion command per family |
+| [[Radiance Plugins]] | kernel and quantiser plugins: ABI, rows, hooks, encodings, build and checks |
+| [[Radiance Architecture Plugins]] | declaring a model: weights, buffers, KV groups, ops, step, probe |
+| [[Radiance Tools]] | `rad-info`, `rad-kbench`, `rad-tune`, `rad-schemas`, `rad-tokcheck`, harnesses |
+| [[Radiance Operations]] | the container contract, host prerequisites, readiness, ledgers, restarts |
+| [[frank-archon]] | the fleet's live radiance endpoint: shape, Eras, rollback, measured laws |
+
+## Sources
+
+- `README.md`, `docs/GUIDE.md` §1–§4, `docs/TOOLS.md`, `docs/PLUGIN.md`, `docs/DOCKER.md`,
+  `docs/ARCHITECTURES.md`, `docs/OPS.md`, `spec.md` §2–§4, `abi/*.h`, `core/config.cpp`,
+  `core/rad_core.h` at commit `8da043c` (codeberg.org/StillDeadcode/radiance).

serverAppConfigs-docs/LLM/Radiance HTTP API.md

@@ -0,0 +1,198 @@
+---
+type: "reference"
+title: "Radiance HTTP API — endpoints, fields, refusals"
+description: "The HTTP surface radiance serves: endpoints, request fields, streaming, tools, and refusal rules."
+tags:
+  - llm
+  - radiance
+  - api
+  - reference
+timestamp: 2026-10-30
+---
+
+# Radiance HTTP API
+
+One server serves one model. It answers OpenAI's chat/completions/Responses/images/audio/video APIs,
+Anthropic's Messages API and llama-server's native one, so their clients and SDKs work unchanged.
+`model` in a request is accepted and **not enforced**.
+
+## Endpoints
+
+| route | what it serves |
+|---|---|
+| `POST /v1/chat/completions` | chat: tools, reasoning, streaming, grammars |
+| `POST /v1/completions` | raw text completion |
+| `POST /v1/embeddings` | an embedding head, else 501 |
+| `POST /v1/images/generations`, `/v1/images/edits` | OpenAI images API; needs an image model, else 501 |
+| `POST /v1/systemone` | LiquidAI d1 decisions: named yes/no, choice and score questions as probabilities; needs a d1 container, else 501 |
+| `POST /v1/audio/transcriptions` | OpenAI speech-to-text; needs a transcribing plugin, else 501 |
+| `GET /v1/realtime` (WebSocket) | streaming transcription session |
+| `POST /v1/audio/speech` | speech (Breeze-TTS-2, drama-3) or a song (MiniMax-Music3), whole or streamed |
+| `GET /v1/audio/speech/stream` (WebSocket) | text streamed in, each sentence's audio out |
+| `POST /v1/videos`; `GET /v1/videos[/:id[/content]]`; `DELETE /v1/videos/:id` | video jobs made in the background, downloaded as MP4 |
+| `POST /v1/messages`, `/v1/messages/count_tokens` | Anthropic Messages, for Claude Code and the Anthropic SDKs |
+| `POST /v1/responses`, `/v1/responses/input_tokens` | OpenAI Responses, for Codex and SDK clients |
+| `POST /completion`, `/completions`, `/infill` | llama-server native completion and fill-in-the-middle |
+| `GET /v1/models`, `/v1/models/:id` | what is loaded, with `max_model_len`, `root`, `capabilities` |
+| `GET /health`, `/v1/health`, `/ping`, `POST /ping` | liveness |
+| `GET /load`, `/version`, `/props`, `/server_info`, `/diagnostics` | load; release version; llama-server props; full configuration; the startup report as JSON (`/diagnostics` is same-machine-only without an API key) |
+| `POST /apply-template`, `/v1/chat/completions/input_tokens` | the rendered prompt; its token count |
+| `GET /metrics` | Prometheus |
+| `POST /tokenize`, `/detokenize`; `GET /get_tokenizer_info` | vLLM's shape and llama-server's at once |
+| `POST /reset_prefix_cache`; `DELETE /sessions/<label>`, `/sessions?id=<id>` | drop every cached prefix; forget a stored conversation |
+| `GET /`, `/dashboard`; `GET /stats`, `/graph`, `/sessions` | the dashboard and the JSON it is drawn from |
+
+Unversioned shorthands exist for chat, chat input tokens, embeddings, images, audio transcriptions,
+audio speech, models, responses and completions, for a client given the bare host as base URL.
+`/completions` **without** `/v1` is llama-server's native completion. A path nothing serves is a JSON
+404.
+
+A text prompt to `/v1/completions`, `/v1/embeddings` or `/tokenize` reads control tokens as tokens
+(as vLLM and llama.cpp do), so a client rendering its own template can send `<|im_start|>user\n…`.
+The model's declared BOS is prepended unless the prompt already opens with it.
+
+## Chat
+
+```sh
+curl -s localhost:8000/v1/chat/completions -H 'Content-Type: application/json' \
+  -d '{"model":"m","messages":[{"role":"user","content":"hi"}],"max_tokens":200,"temperature":0.7}'
+```
+
+- **No chat template → 501** on this route (the container carries one; `--override-chat-template`
+  replaces it). Use `/v1/completions` instead.
+- Thinking off has three spellings: `chat_template_kwargs.enable_thinking=false`,
+  top-level `enable_thinking=false`, or `reasoning_effort:"none"`.
+- Reasoning returns in `choices[].message.reasoning_content`, separate from `content`.
+  **Reasoning counts against `max_tokens`**; a reply that runs out inside it has empty `content` and
+  `finish_reason:"length"`. `usage.completion_tokens_details.reasoning_tokens` tells that case from a
+  model that answered nothing.
+- `thinking_token_budget` (aliases `thinking_budget`, `thinking_budget_tokens`,
+  `reasoning_budget_tokens`) closes the reasoning block at N tokens and forces the close, so it holds
+  under speculation and at any temperature. It needs a one-token end marker; otherwise the response
+  has no `completion_tokens_details` and a budget is a 400.
+- `continue_final_message` continues a conversation ending in an assistant message instead of
+  answering it; `add_generation_prompt` opens a new turn.
+- Tool calls: `tools`, `tool_choice` (`auto`, `none`, `required`, named function),
+  `parallel_tool_calls`. `required`/named hold the whole reply to calls after the reasoning block and
+  cannot combine with `response_format` or `grammar`.
+- Structured output: `response_format` (`json_object` or `json_schema`), llama-server's `json_schema`
+  shorthand, or a raw GBNF `grammar`. A chat template that drops the grammar makes the request a
+  refusal.
+
+## Streaming
+
+```sh
+curl -N localhost:8000/v1/chat/completions -H 'Content-Type: application/json' \
+  -d '{"model":"m","messages":[{"role":"user","content":"Count to ten."}],
+       "max_tokens":100,"stream":true,"stream_options":{"include_usage":true}}'
+```
+
+Server-sent events in OpenAI's delta format, terminated by `data: [DONE]`. **Nothing is sent while
+the prompt is read**: the first event (the `role: assistant` chunk) goes out with the first generated
+token, so a client timing its first event measures time-to-first-token. `return_progress` adds
+llama-server's `prompt_progress` chunks during prefill; `timings_per_token` adds `timings` to every
+chunk.
+
+## Completions, embeddings, images, audio, video, tokens
+
+- **Completions** — `/v1/completions` takes `prompt` (string or token ids), `echo`, `logprobs`,
+  `preserved_tokens`. `/completion` is llama-server's native route; `/infill` is fill-in-the-middle.
+- **Embeddings** — `input` is text, a batch, or token ids; `encoding_format` `float`/`base64`;
+  `dimensions` truncates to the first N and renormalises; `prompt_name` prepends the model's named
+  prompt. Pooling is the model's; the engine does not renormalise.
+- **Images** — `/v1/images/generations` returns `b64_json`; `size` defaults to 2048×2048. Edits take
+  one picture or up to ten (`image[]`), multipart as OpenAI's SDKs send them, with an optional mask
+  (repaint only where transparent). `partial_images` streams previews as SSE.
+- **Speech to text** — `/v1/audio/transcriptions`; `/v1/realtime` streams audio in and words back as
+  they are spoken. The model's prompt shape comes from its plugin's audio format, not from the server.
+- **Speech and music** — `/v1/audio/speech` (whole or streamed) with `instructions` describing a voice
+  (Breeze-TTS-2) or speakers written in the input (`<|speaker:1|>`, `[laughs]` for drama-3); a music
+  model takes the lyrics as `input` and the description as `instructions`. `/v1/audio/speech/stream`
+  streams text in and audio out as each sentence completes.
+- **Video** — `/v1/videos` is a job: a video with its stereo soundtrack from a prompt, 5–14 s, up to
+  1344×768, made in the background; `GET /v1/videos/:id/content` downloads the MP4.
+- **Tokenising** — `POST /tokenize` accepts vLLM's shape (`prompt` or `messages` in, `prompt` out) and
+  llama-server's (`content` in/out, `with_pieces`). `POST /detokenize` reverses it.
+  `GET /get_tokenizer_info` reports the tokeniser configuration.
+
+## Request fields
+
+Accepted fields, in OpenAI's, vLLM's and llama-server's spellings:
+
+| field | notes |
+|---|---|
+| `model` | accepted, not enforced |
+| `messages` / `prompt` / `input` | chat / completion / embedding |
+| `max_tokens`, `max_completion_tokens`, `n_predict` | default is what the context leaves; `-1` is llama-server's "no limit"; clamped to context and to `--max-tokens-cap` |
+| `temperature`, `top_p`, `top_k`, `min_p`, `typical_p` | sampler |
+| `presence_penalty`, `frequency_penalty`, `repetition_penalty`/`repeat_penalty`, `repeat_last_n` | penalties |
+| `xtc_probability`, `xtc_threshold`, `dry_*` | XTC and DRY |
+| `seed` | absent means "pick one", not 0 |
+| `n` | 1..`--max-n`; prefix cache makes the fan-out nearly free after the first |
+| `stop` | string or array, bounded by `--max-stop-strings`/`--max-stop-bytes` |
+| `priority` | scheduler hint |
+| `ignore_eos` | generate to `max_tokens` whatever the model emits |
+| `stream`, `stream_options.include_usage`, `timings_per_token`, `return_progress` | streaming controls |
+| `samplers` | llama-server's order; this engine's chain is penalties, DRY, top-k, top-p, min-p, typical, XTC, with temperature first or last; any other list is refused by name |
+| `session_id` | labels the conversation on `/sessions`, dashboard and `DELETE` |
+| `control_vector` | `true` applies the deployment's vector to this request; absent takes the deployment default; a deployment with no vector refuses `true` |
+| `logprobs`, `top_logprobs`, `n_probs` | up to 20 alternatives; not with `echo` |
+| `tools`, `tool_choice`, `parallel_tool_calls`, legacy `functions`/`function_call` | chat only |
+| `add_generation_prompt`, `chat_template_kwargs`, `continue_final_message` | chat only |
+| `reasoning_format`, `reasoning_effort`, `enable_thinking`, `thinking_token_budget`, `preserve_thinking` | chat only; levels are the template's (`capabilities.reasoning_efforts`); `none`/`minimal` turn thinking off |
+| `response_format`, `json_schema`, `grammar` | structured output; `grammar` and `response_format` are mutually exclusive |
+| `echo`, `preserved_tokens` | `/v1/completions` only |
+| `encoding_format`, `dimensions`, `prompt_name`, `image_tokens` | `/v1/embeddings` only |
+| `images_config` | image models only: a named processor mode or an object restating served sizes |
+
+## What is refused, ignored, and clamped
+
+The contract: **a 200 means every field that could change the answer was honoured.** Fields fall
+into four classes.
+
+- **Inert** — accepted and ignored because they cannot change the answer: OpenAI/OpenRouter
+  bookkeeping (`user`, `metadata`, `store`, `service_tier`, `prompt_cache_key`, …) and llama-server's
+  own speed/slot knobs (`cache_prompt`, `id_slot`, `n_cache_reuse`, `backend_sampling`,
+  `speculative.*`, …). The prefix cache and speculation here are the server's and never change output.
+- **Guarded** — known, unimplemented, and able to change the answer: accepted at their off value,
+  refused by name otherwise. `logit_bias` (empty), `mirostat` (0), `top_n_sigma` (≤0),
+  `dynatemp_range` (≤0), `min_keep` (0), `post_sampling_probs` (false), `best_of` (≤1), `suffix`
+  (empty), `lora` (empty — an adapter is chosen by `model`), `modalities`/`audio` (`["text"]`/null),
+  `n_indent` (0), `t_max_predict_ms` (≤0), `return_tokens` (false), `grammar_lazy`,
+  `grammar_triggers`, `preserved_tokens` (empty on chat).
+- **Ignored** — anything else unknown, logged at debug. An unknown field is far more often a client's
+  extension than a misunderstood request.
+- **`max_tokens` is the exception**: if prompt + `max_tokens` exceeds the context, it is **clamped**
+  and served, ending `finish_reason:"length"`, because refusing would produce no tokens instead of
+  fewer. A prompt that does not fit on its own is a 400 `context_length_exceeded` naming
+  `n_prompt_tokens`, `n_ctx`, and how many tokens must go.
+
+## Anthropic, Responses, llama-server
+
+- `/v1/messages` is translated into a chat request, so chat defaults, refusals and limits apply.
+  Streamed (`message_start`, content blocks, `message_delta`, `message_stop`) or not:
+  `ANTHROPIC_BASE_URL=http://host:8000 ANTHROPIC_API_KEY=$KEY claude` works. `/v1/messages/count_tokens`
+  returns the prompt length.
+- `/v1/responses` maps `instructions`/`input` (strings, messages, `function_call`,
+  `function_call_output`, `reasoning` items) into the chat request; `max_output_tokens`,
+  `reasoning.effort`, `text.format` map to chat fields; undefined fields pass through. The event stream
+  carries `output_index`, `content_index`, `sequence_number`, and opens with `response.created` and
+  `response.in_progress`. A reply cut at `max_output_tokens` is `incomplete`. Function tools a
+  `namespace` groups are named `<namespace>__<name>`; built-in tools and freeform custom tools are
+  left out.
+- `/completion`, `/completions`, `/infill` are llama-server's own shapes, for llama.vim/llama.vscode.
+
+## Auth, CORS, errors
+
+- `--api-key KEY` requires `Authorization: Bearer KEY` or `X-Api-Key: KEY` on every request except
+  `/health`, `/v1/health`, `/ping` and the dashboard page. An empty key is refused at startup.
+- CORS headers are sent unless `--no-cors`.
+- Errors are JSON. Over-context is 400 `context_length_exceeded`; a full queue is 429 with
+  `Retry-After`; a capability the model lacks is 501; a guarded field is 400 naming the field and its
+  accepted off value.
+
+## Related
+
+- [[Radiance Serve Options]] — the defaults every omitted field takes.
+- [[Radiance Operations]] — `/health` semantics, metrics, dashboard.
+- [[frank-archon]] — the harness contract built on this surface.

serverAppConfigs-docs/LLM/Radiance Model Families.md

@@ -0,0 +1,142 @@
+---
+type: "reference"
+title: "Radiance Model Families — arch ids, descriptors and recipes"
+description: "Which radiance plugin claims which model family, and the conversion command and recipe of record for each."
+tags:
+  - llm
+  - radiance
+  - models
+  - reference
+timestamp: 2026-10-30
+---
+
+# Radiance model families
+
+The selection key is the exact **(architecture id, quantisation descriptor)** pair, so a family at
+two weight formats is two `.so` files. A checkpoint's id is its `config.json` `model_type` without
+underscores (`qwen3_5` → `qwen35`); `rad-convert --arch ID` overrides it. `data/recipes/` is the
+authority: **read the recipe's header** — it names the exact `rad-convert` line, its `--set` keys
+and whether it needs calibration.
+
+## The plugin table
+
+| plugin | id | quant | what it serves |
+|---|---|---|---|
+| `llama_fp8` | `llama` | `fp8_e4m3` | plain LLaMA decoder (MiniCPM5, Llama 3, derivatives reporting `model_type: llama`) from a checkpoint holding E4M3 codes + `weight_scale_inv` |
+| `llama_dense_fp8` | `llama` | *(empty)* | the same graph from a bf16 checkpoint, linears made block fp8 by a recipe |
+| `qwen35_bf16` | `qwen35` | *(empty)* | Qwen3.5/3.6/3.8 dense hybrid (GDN + gated attention), bf16 |
+| `qwen35_fp8` | `qwen35` | `fp8_e4m3` | the same, block-scaled FP8 weights and activations |
+| `qwen35_w8a8` | `qwen35` | `w8a8_fwht128` | int8 trunk: i8 weights, bf16 scale per 128 of K, 128-wide Hadamard, int8 activations |
+| `qwen35moe_fp8` | `qwen35moe` | `fp8_e4m3` | `qwen35_fp8` with a routed feed-forward |
+| `qwen4exp_fp8` | `qwen4exp` | *(empty)* | Qwen4-Exp / Qwen3.8-Flash-Next: a bf16 checkpoint quantised at convert, or Qwen's FP8 release as it ships |
+| `glm5next` | `glm5next` | *(empty)* | GLM-5.3-Flash: four mHC residual streams, 34 KDA + 11 sparse MLA layers with a DSA indexer, 288 MXFP4 experts top-8, block-FP8 attention/dense at f32 scales, MTP drafter, vision tower |
+| `hyv3_bf16` | `hyv3` | *(empty)* | Tencent Hy-Memory-30B-A3B bf16; bf16/W4/W8A8/block-FP8/MXFP4 by recipe |
+| `hyv3_fp8` | `hyv3` | `fp8` | Hy-Memory-30B-A3B-FP8: imported per-tensor E4M3, one dense layer then sigmoid-routed MoE |
+| `hyv3_gguf` | `hyv3` | `gguf_ftype_15` | a llama.cpp Hy-Memory Q4_K_M GGUF, dequantised and recipe-converted |
+| `gemma4_bf16` | `gemma4` | `compressed-tensors` | Gemma 4 31B dense (and E2B/E4B at one rank): sliding-window + proportional-RoPE global attention, GELU, softcapped logits; int4/Q8 codes per 128; assistant (shared-KV), DFlash or DFlash2 drafter; vision |
+| `gemma4` | `gemma4` | *(empty)* | Gemma 4 text from Google's bf16 checkpoint (26B-A4B MoE, dense sizes) with z-lab's DFlash |
+| `gemma3` | `gemma3` | *(empty)* | Gemma 3 dense (TranslateGemma 27B): sliding-window + global attention, GemmaRMSNorm, GELU; bf16 or int8/MXFP4/asym-int4 linears; DFlash2 drafter |
+| `lfm2_bf16` | `lfm2` | *(empty)* | LiquidAI LFM2 / LFM2-VL, bf16, one rank |
+| `embgemma2` | `embeddinggemma2` | *(empty)* | EmbeddingGemma 2: bidirectional encoder with Gemma 4 vision tower and audio conformer, pooled |
+| `nomicbert` | `nomicbert` | *(empty)* | nomic-embed-text-v2-moe: bidirectional post-norm encoder, 8 experts top-2 on every second layer, mean-pooled |
+| `nemotron35_embed` | `mistral3` | *(empty)* | Nemotron 3.5 Embed 1B/8B and Rerank 8B: bf16 text/image retrieval with a Pixtral tower |
+| `qwenimage21` | `qwenimage21` | *(empty)* | Qwen-Image 2.1 / 2.1-Turbo image generator and editor (diffusers pipeline) |
+| `krea2` | `krea2` | *(empty)* | Krea 2 Raw/Turbo: Qwen3-VL-4B tapped layers, 12.9B single-stream DiT, Wan 2.1 VAE |
+| `sdxl` | `sdxl` | *(empty)* | Stable Diffusion XL and finetunes (Illustrious, NoobAI, Pony), single-file checkpoint |
+| `flux3` | `flux3` | *(empty)* | FLUX.3 [klein] and base: image generation and edits from up to ten pictures |
+| `flux3distilled` | `flux3distilled` | *(empty)* | the same plugin under the step-distilled release's own pipeline name |
+| `anima` | `anima` | *(empty)* | Anima Base/Aesthetic/Turbo from the release's three single files |
+| `qwen3asr` | `qwen3asr` | *(empty)* | Qwen3-ASR: Whisper-style audio encoder over a dense Qwen3 decoder, speech to text |
+| `minimaxmusic3` | `minimaxmusic3` | *(empty)* | MiniMax-Music3: Qwen3-8B frame model, depth decoder, flow-matching DiT, DAC vocoder |
+| `breeze` | `breeze` | *(empty)* | Breeze-TTS-2: T5Gemma 2 text encoder, Qwen3 backbone, 16-codebook depth decoder, Qwen3-TTS 12 Hz codec |
+| `fishomni` | `fishomni` | `bsgemm-moe` | Fish Audio drama-3: Qwen3.5-35B-A3B backbone, block-FP8 experts at f32 scales, 13-layer codebook tower, Fish S2 DAC/codec |
+| `minimaxh3` | `minimaxh3` | *(empty)* | MiniMax-H3 video with stereo sound: Qwen3-VL-32B text encoder, 33B DiT, tiled ViT decoder, BigVGAN audio |
+| `ltx2` | `ltx2` | *(empty)* | LTX-2.5 base/distilled: Gemma 4 LM + text connectors, duration head, audio-video DiT, causal video VAE, audio VAE |
+
+## Recipes of record, by family
+
+`$SRC` is a downloaded checkpoint; `$CALIB` is a calibration directory (see
+[[Radiance Conversion]]). Paths are shortened to the recipe name.
+
+| target | command |
+|---|---|
+| Qwen3.8-27B bf16 | `rad-convert $SRC -o q38-27b-bf16.rad --recipe data/recipes/q38-27b-bf16.recipe` |
+| Qwen3.8-27B block fp8 + DFlash2 | `rad-convert $SRC -o q38-27b-fp8.rad --recipe data/recipes/q38-27b-fp8block-df2.recipe --draft-model $DFLASH2 --set quantization_config.quant_method=fp8 --set quantization_config.fmt=e4m3 --set "quantization_config.weight_block_size=128 128" --set quantization_config.activation_scheme=dynamic` |
+| Qwen3.8-27B MXFP4 + DFlash2 | `rad-convert $SRC -o q38-27b-mxfp4.rad --recipe data/recipes/q38-27b-mxfp4-df2.recipe --draft-model $DFLASH2 --set quantization_config.quant_method=fp8 --set quantization_config.fmt=e4m3` |
+| Qwen3.8-27B w4a16 (an int4 release) | `rad-convert $SRC -o q38-27b-w4a16.rad --recipe data/recipes/q38-27b-w4a16.recipe` |
+| Qwen3.8-27B Quark AWQ-MXFP4 | `rad-convert $SRC -o q38-27b-quark.rad --recipe data/recipes/q38-27b-quark-mxfp4-df2.recipe --draft-model $DFLASH2` |
+| Qwen3.8-27B ParoQuant int4/int5/MXFP4 | `rad-convert $SRC -o q38-27b-paro.rad --recipe data/recipes/q38-27b-paro-mxfp4-df2.recipe --draft-model $DFLASH2` |
+| Qwen3.8-27B NVFP4 (compressed-tensors or ModelOpt) | `--recipe data/recipes/q38-27b-nvfp4-df2.recipe` or `q38-27b-modelopt-nvfp4.recipe`; every NVFP4 weight must take a rule (no kernel serves NVFP4) |
+| Qwen3.8-Flash-Next bf16 reference | `rad-convert $SRC -o flashnext-bf16.rad --max-ctx 4096` (no recipe; served with `--weights-disk-tier`) |
+| Qwen3.8-Flash-Next four-bit production | `CALIB=$CALIB rad-convert $SRC -o w4nl64-i8lm.rad --recipe data/recipes/qwen4exp-w4nl64-i8-hc8m.recipe` (GPTQ experts + int8 trunk) |
+| Qwen3.8-Flash-Next uncalibrated four-bit | `--recipe data/recipes/qwen4exp-w4.recipe` (rtn, no calibration) |
+| Flash-Next from an AutoRound int4 release | `--recipe data/recipes/qwen4exp-autoround-i8-hc8m.recipe` |
+| Gemma 4 31B w4a8 (no calibration) | `--recipe data/recipes/gemma4-31b-dense-bf16-w4a8.recipe --set quantization_config.quant_method=compressed-tensors --draft-model $ASSISTANT` |
+| Gemma 4 31B w4a8 + DFlash / DFlash2 | `gemma4-31b-dense-bf16-w4a8-df.recipe` / `-w4a8-df2.recipe`, same `--set` |
+| Gemma 4 31B Q8 grid | `gemma4-31b-dense-bf16-q8-df.recipe` |
+| Gemma 4 26B-A4B + DFlash | `gemma4-26b-a4b-df.recipe`; `-pad768` when the experts were padded to 768 (down may then be block fp8) |
+| Gemma 4 E4B (Nemotron Bolt) | `gemma4-e4b-bf16-df.recipe`, `-i8g-`, `-a4-`, `-w4a8-`, `-w8a8-` variants |
+| TranslateGemma 27B | `translategemma-27b-i8a8.recipe` (calibration container), then `CALIB=$CALIB translategemma-27b-mxfp4-gptq.recipe` (or `-df2`) |
+| GLM-5.3-Flash | `rad-convert $AMD_QUARK -o glm53.rad --recipe data/recipes/glm53-mxfp4.recipe` |
+| Hy-Memory-30B-A3B | `hy-memory-30b-a3b-bf16-fp8.recipe`, `-mxfp4`, `-w4`, `-w8`, `hy-memory-30b-a3b-fp8.recipe`, `hy-memory-30b-a3b-gguf-q4k-to-w4.recipe` |
+| MiniCPM5-2B | `minicpm5-2b-dspark.recipe` (or `-bf16-dspark`) with `--draft-model $DSPARK` |
+| Qwen3.6-35B-A3B | `q36-35b-a3b-fp8.recipe` |
+| Qwen-Image 2.1 | `qwen-image-2.1-fp8.recipe` (Turbo uses the same file) or `-bf16` |
+| Krea 2 | `krea-2-fp8.recipe`, `krea-2-bf16.recipe`, `krea-2-fp8-unscaled.recipe` for a plain-FP8 finetune file |
+| FLUX.3 [klein] | `flux-3-klein-fp8.recipe` |
+| SDXL / finetunes | `sdxl-bf16.recipe --tokenizer <openai/clip-vit-large-patch14>/tokenizer.json` |
+| Anima | `anima-fp8.recipe` (or `-bf16`) + `--add` ×2 + `--config` + `--tokenizer2` + `--arch anima --set anima.variant=aesthetic` |
+| LTX-2.5 | `ltx-2.5-fp8.recipe` (or `-bf16`); full DiT: symlink a folder whose `transformer/` is `transformer_full/` and `--set scheduler.use_dynamic_shifting=1 --set scheduler.shift_terminal=0.1` |
+| MiniMax-H3 / FastH3 | `minimax-h3-fp8.recipe`, `minimax-h3-w8a8.recipe`, `minimax-h3-curve-*` for curve-form checkpoints |
+| MiniMax-Music3 | `minimax-music3-bf16.recipe` |
+| Breeze-TTS-2 | `breeze-tts2-bf16.recipe` |
+| drama-3 | `drama3-fp8.recipe` |
+| nomic-embed-v2-moe | `nomic-embed-v2-moe-bf16.recipe` |
+| Unlimited-OCR | `unlimited-ocr-bf16.recipe` |
+| Qwen3-ASR | no container: serve the checkpoint directory as it ships |
+| EmbeddingGemma 2, Nemotron Embed/Rerank | serve the checkpoint, or convert bf16 with no recipe |
+
+A checkpoint served at its own precision needs no container at all:
+`radiance --model /srv/models/hf/Qwen/Qwen3.8-27B-FP8`. Convert when you want a tier only a container
+has (quantised encodings, a drafter, a second vocab) or a smaller file.
+
+## Family-specific traps
+
+- **`head_dim` is read, never derived.** On the Qwen hybrids it is 256 while `hidden_size /
+  num_attention_heads` is 213; a derived value silently breaks projection widths, KV page size and the
+  attention kernel's band.
+- **A fused projection shards by part, not by contiguous rows.** A tensor-parallel rank owns a slice
+  of *each* stacked projection in a fused `[gate|up]` or `[q|k|v|z]`; a contiguous slice would hand
+  rank 0 the whole gate, and the model would produce fluent wrong text. This is declared through
+  `RadWeightDecl::row_parts`, never inferred from the container.
+- **Gemma 4 26B-A4B:** the routed experts' `gate_up` may be block fp8, but `down` stays bf16 unless the
+  experts were padded to 768 — hence the `-pad768` recipe.
+- **GLM-5.3-Flash:** every weight is served as stored except the latent attention's small `kv_b`,
+  decoded to bf16. Its KV cache is bf16 only (the plugin refuses `fp8` at startup), and at 172 GiB it
+  needs `--tp 8`.
+- **TranslateGemma 27B:** the checkpoint's template takes structured turns naming two languages and
+  refuses the plain turn the server renders at startup. The container therefore serves a plain Gemma 3
+  template, and a client sends on `/v1/completions` the prompt the checkpoint's own template renders.
+- **ParoQuant containers made before 1.5.0 are refused at startup.** Convert again with the old
+  container as `--reuse`; every tensor that did not change is copied.
+- **SDXL carries no tokeniser.** Pass CLIP's `tokenizer.json` beside the single file. A Krea 2
+  finetune shipped as one DiT file in Krea's key names is rewritten into a directory whose
+  `transformer/` holds it (`scripts/krea2_native.py`), with the release's other parts copied in.
+- **Anima** ships as three single files: `--add` the text encoder and VAE, `--config` a Qwen3-0.6B-Base
+  directory (config + tokenizer), `--tokenizer2` ComfyUI's T5 tokenizer, `--arch anima`,
+  `--set anima.variant=…`.
+- **drama-3's codec is `codec.pth`**, a `torch.save` file read as data, never unpickled.
+- **Generators that load LoRA files declare their module aliases**; the checkpoint and a ComfyUI port
+  disagree in ways that load without error and generate garbage. The per-plugin alias table is in
+  `docs/ARCHITECTURES.md` ("Adapter key names of the generators").
+- **Licences travel with the checkpoint, not the engine.** Qwen-Image 2.1 is non-commercial;
+  MiniMax-H3's licence excludes the EU, UK, Republic of Korea and USA; drama-3's community licence
+  needs a commercial licence to serve speech to others, and cloning a real person needs consent;
+  Breeze-TTS-2 is research/non-commercial.
+
+## Related
+
+- [[Radiance Conversion]] — the tool, recipe language and quantisers used above.
+- [[Radiance Serve Options]] — the flags each family's compose file uses.
+- [[Radiance Architecture Plugins]] — writing a plugin for a family not in this table.
+- [[Quantization Production]] — the fleet's campaign and gates.

serverAppConfigs-docs/LLM/Radiance Operations.md

@@ -0,0 +1,240 @@
+---
+type: "howto"
+title: "Radiance Operations — running, reading, restarting"
+description: "Running a radiance server: the container contract, host prerequisites, readiness, ledgers and safe restarts."
+tags:
+  - llm
+  - radiance
+  - operations
+  - howto
+timestamp: 2026-10-30
+---
+
+# Radiance operations
+
+This is the engine-level runbook. The fleet's deployed shape, its unit and its deploy path live on
+[[frank-archon]]; the flags themselves on [[Radiance Serve Options]].
+
+## Install
+
+```sh
+git clone https://codeberg.org/StillDeadcode/radiance.git && cd radiance
+./build.sh                                   # configure, build for the cards present, run fast tests
+./build.sh --no-hip                          # host backend only
+cmake -S . -B build -G Ninja -DRAD_GPU_TARGETS='gfx1201'   # cards this machine lacks
+cmake --install build                        # after -DCMAKE_INSTALL_PREFIX=/opt/radiance at first configure
+```
+
+Needs Linux, git, a C++20 compiler, CMake 3.21+, Ninja; for device kernels ROCm 10.1 or 7.2
+(`hipcc`, `ROCM_PATH` if not `/opt/rocm`).
+
+- **Name the prefix at the first configure.** The default plugin path `<prefix>/share/radiance` is
+  compiled in from that configure; `cmake --install build --prefix DIR` alone gives a tree that looks
+  for its plugins elsewhere.
+- The prefix gets `bin/`, `include/radiance/`, `lib/cmake/radiance/`, `libexec/radiance/`,
+  `share/radiance/` (plugins and recipes) and `share/doc/radiance/`.
+- With ROCm 10.1 the build turns VOPD pairing off because its LLVM 24 pairs in heap-address order, so
+  two builds of one source can differ. The Docker image's compiler has the fix and keeps pairing
+  (faster fp8 prefill GEMMs); declare such a compiler with `-DRAD_HIP_VOPD_REPEATABLE=ON`. ROCm 7.2 is
+  unaffected.
+
+Run without Docker:
+
+```sh
+export RADIANCE_HOME=$PWD/build/radiance_home   # plugins; an install finds its own
+build/bin/radiance --model /srv/models/rad/minicpm5-2b-fp8.rad --port 8000 <flags>
+build/bin/radiance --model StillDeadcode/minicpm5-2b-fp8 --port 8000 <flags>
+```
+
+`--model` also takes a repo id: the server runs `scripts/hfget.sh` into `--download-dir`, serves the
+one `.rad` at the repo's top level, or the directory as a checkpoint when it holds none. A repo with
+several containers is refused, naming them. Every start checks the hub; a start that cannot reach it
+serves the copy a complete earlier fetch left, and a transfer that fails part-way stops startup.
+
+## Container
+
+The published image is `stilldeadcode/radiance`, kernels built for `gfx1201`. The host needs Docker
+and an `amdgpu` driver supporting ROCm 10.1 — **not** ROCm itself.
+
+```sh
+docker run -d --name radiance --network host --init --ulimit memlock=-1 \
+    --device /dev/kfd --device /dev/dri --security-opt seccomp=unconfined \
+    --group-add "$(getent group render | cut -d: -f3)" \
+    --group-add "$(getent group video | cut -d: -f3)" \
+    -v /srv/models/rad:/models:ro \
+    radiance --model /models/m.rad --tp 2 --host :: --port 8000
+```
+
+Each option is load-bearing:
+
+| option | why |
+|---|---|
+| `--device /dev/kfd --device /dev/dri`, `render`/`video` groups | the cards |
+| `--security-opt seccomp=unconfined` | Docker's default profile refuses io_uring (a disk-tier n-gram table's reads) and the NUMA calls the ROCm runtime makes for host memory |
+| `--init` | an init process as PID 1 forwards SIGTERM; the engine finishes its step and releases the cards |
+| `--ulimit memlock=-1` | a host-side pinned n-gram copy can be tens of GiB, far past Docker's default |
+| `--network host` | the server binds the host's addresses directly, IPv4 and IPv6, no NAT |
+
+The image is `FROM scratch` around the stripped install tree: busybox as the only shell, plus `curl`,
+`jq` and a CA bundle for `hfget.sh`; **no Python, no package manager**. Tools (`rad-convert`,
+`rad-info`, `rad-kbench`, `rad-tune`, `rad-schemas`, `rad-tokcheck`) are on `PATH`; recipes are in
+`/opt/radiance/share/radiance/recipes/`. Its entrypoint is `radiance`. It sets
+`GPU_MAX_ALLOC_PERCENT=100` and `GPU_MAX_HEAP_SIZE=100` because the HIP runtime's defaults cap one
+allocation and the heap below what the placement plan hands out.
+
+Build your own with `docker/build.sh` (`-t 'gfx1200;gfx1201'` for targets, `-j` compile jobs, `-E`
+to skip tests, `-s host` to build remotely). The context is `git archive` of the commit, so the image
+holds only committed content. The `rootfs` stage fails the build if any ELF cannot be resolved inside
+a chroot of the tree.
+
+Compose: one file per model plus `common.yaml`, copied to the serving host with an `.env` setting the
+image, models directory (mounted read-only at `/models`), API key, port, user and GPU groups. Every
+model listens on the same port, and most use both cards, so **run one at a time**. The models
+directory must be on a local filesystem that does direct I/O (ext4, XFS, btrfs); on btrfs with
+compression, a container copied into a `chattr +C` directory is stored uncompressed and direct reads
+work.
+
+A plugin of your own is mounted and put first on the search path; nothing in the image changes:
+
+```sh
+-v /srv/radiance-plugins:/plugins:ro -e RADIANCE_HOME=/plugins:/opt/radiance/share/radiance
+```
+
+## Host prerequisites (upstream's two-R9700 checklist)
+
+| item | setting |
+|---|---|
+| firmware | Above 4G Decoding on (Resizable BAR needs it); CSM off (pure UEFI) |
+| kernel command line | `iommu=pt amdgpu.ppfeaturemask=0xfff7ffff amdgpu.ras_enable=0` |
+| power/clock | OverDrive bit needed for `ppfeaturemask`; apply a power cap only while no process holds `/dev/kfd` |
+| memlock | `--ulimit memlock=-1` (container) / `LimitMEMLOCK=infinity` (systemd) when host memory is pinned |
+| two engines, one tmpfs table | `amdgpu.no_system_mem_limit=1`, or the second pin can be refused |
+| storage | a local fs with O_DIRECT for the container; a tmpfs sized for the table for `--ngram-placement ram=DIR` |
+
+Why each: `iommu=pt` keeps DMA identity so the cards can reach each other's memory and pinned host
+pages; `ppfeaturemask` enables the power controls; `ras_enable=0` turns VRAM ECC off (ECC reserves
+about 2 GiB a card). Verify peer access with a rate in `lspci`/the card tables — a missing cell means
+no peer access; check `iommu=pt` and the BAR size. The fleet's own boot-line law for Dankclaw
+(`iommu=pt`, tracked in the repo) is on [[dankclaw-iommu-translation-off]]; Dankserver's cards are a
+different box and its posture is on [[Dankserver System Specs]].
+
+## Readiness: the receipt, then `/health`
+
+Startup states every budget **before anything is allocated**, then loads. `/health` (and
+`/v1/health`, `/ping`) answers only once the load completes, so a bare health gate is honest. While it
+loads, expect 503 for minutes; do not restart over it. `--live` draws the live view (redirected, one
+`greppable key=value` line). `/diagnostics` returns the startup report as JSON — **same-machine-only
+without an API key** — with every option and its source and the hardware read again now.
+
+Read the receipt in this order:
+
+1. `elastic` — what the experts and KV cache share after non-expert weights.
+2. The split: `experts … / kv cache …`, and `unclaimed` if any.
+3. `kv: the pool backs N % of the worst case … M full-length sessions` — the referee for whether the
+   demand signal (`--max-model-len` × `--max-num-seqs`) is funded.
+4. `activation arena` and what each size lends; `prefill staging`; `hardware queues` and their VRAM.
+5. `host_pool` opened/used, `vram_weights`, the PLE or n-gram placement, the vision tower's placement.
+6. `device: the N cards reach each other's memory directly` or the failed pairs;
+   `expert prefetch on/off`; `chunk geometry`; `speculation:`; `chained launches`; `draft trees`.
+7. `card N settled at … free` — memory taken after the budget read comes out of headroom; a warning
+   names the `--gpu-headroom-mib` that clears it.
+
+A boot refusal prints the reason and exits nonzero; when it happens inside `podman run`, the
+container stops at once and the unit's restart policy re-runs the same refusal. A unit in
+`activating (auto-restart)` with a fast nonzero exit is therefore a **preflight/refusal or a boot
+refusal**, not an engine fault: read the journal line and the argv, not the GPU.
+
+## Lifecycle and restart
+
+- SIGTERM finishes the current step and releases the cards. In a container, `--init` forwards it;
+  `podman stop -t N`/`docker stop` is the graceful path.
+- A systemd unit should `exec podman wait` (or run in the foreground) so the script's lifetime is the
+  container's: a crash is a nonzero exit and the restart policy is real. Set `TimeoutStopSec` above
+  the stop grace, or systemd records `Failed with result timeout` on a healthy stop.
+- `Restart=always` rather than `on-failure` for a server whose front end can exit cleanly after a
+  worker fault.
+- Model load is slow (minutes cold). The deployer's LLM wait budget is on [[Deployment Flow]]; a
+  container restart loop during a legitimate load is worse than waiting.
+
+## Two engines on one set of cards
+
+- `ROCR_VISIBLE_DEVICES` picks each engine's cards explicitly.
+- `--weight-prefetch off`: a weight read beside a collective puts a queue switch into every decode
+  step when another server shares the cards.
+- `--ngram-placement ram=DIR`: two engines on separate card sets hold one table in host memory
+  (`DIR/ngram-<id>.tab` in a tmpfs, `MAP_SHARED`, pinned). The `<id>` hashes the container path, size,
+  mtime and the table's place, so a file never serves another model. The first engine fills it; a
+  second verifies samples and only pins it; the last to leave deletes it. The driver counts a shared
+  page against every process that pins it, so the pinned total is twice the table.
+- The engine prefers discrete cards (all of the first one's architecture), integrated only when
+  there is nothing else; on integrated graphics device memory is host memory and the budget leaves an
+  eighth of host RAM to the host.
+
+## Logs and diagnostics
+
+- `-v` / `-vv`; `RADIANCE_LOG` sets the level without a flag. `RADIANCE_LOG_STEPS=1` prints every
+  step's token count (how a chunk geometry is confirmed in practice).
+- The engine writes to stderr. Under a container, the journal holds only the container's own output;
+  a rootless `podman logs` covers only the current instance across recreations.
+- `/metrics` is Prometheus text: engine steps by kind and their seconds, decode/prefill rates,
+  prefill chunks held back, steps held by interleave, draft acceptance and depth, prompt-lookup
+  proposals held, KV bytes and the tokens they address, stream misses and bytes, prefix hits and
+  evictions, preemptions and queue depth. `/stats`,
+  `/graph`, `/sessions` back the dashboard. `scripts/statsdump.py` records `/stats` to JSONL with
+  time and gap, for a machine with no scraper (its key comes from `--api-key` or
+  `RADIANCE_API_KEY`).
+- `RADIANCE_PROFILE_EVERY`, `RADIANCE_DUMP_BUF`/`RADIANCE_DUMP_DIR`, `RADIANCE_OP_ORACLE`(+) and the
+  `RADIANCE_DEBUG_*` family are debug instruments; they are not deployment settings.
+
+## Verifying a deploy (fleet law)
+
+`deploy.sh` green is not proof of what is running. A leftover systemd drop-in or a stale container
+identity can serve a different model at a different shape while `/health` is 200. After deploying
+any recently-benched service:
+
+1. `systemctl show <svc> -p DropInPaths` — must be empty.
+2. `tr '\0' ' ' </proc/$(pidof radiance)/cmdline` (or the container's `Config.Cmd`) — the argv you
+   meant.
+3. The boot ledger — the shape you meant.
+4. A behavioural probe — `/v1/models` for the served id, one chat turn, one tool call, and a vision
+   probe with a solid colour and an exact expected word.
+
+`/v1/models` is an echo on some runtimes; on radiance it is authoritative for the alias, but the
+argv and the ledger are what prove the payload and shape. The three-pin revert discipline for a
+payload swap is on [[Model Swap and Rollback]]; the live deployment is [[frank-archon]].
+
+## Security posture
+
+- `--api-key` requires a bearer token or `X-Api-Key` on every route except `/health`, `/v1/health`,
+  `/ping` and the dashboard; the key is a command-line argument, so it is readable in
+  `/proc/<pid>/cmdline` by any local user of that host. On a single-tenant inference box the network
+  boundary is the real control; do not run one where untrusted local users share the host.
+- `/diagnostics` is same-machine-only without a key, because it prints the whole configuration.
+- `--security-opt seccomp=unconfined` is a genuine widening, required by io_uring and the ROCm
+  runtime's NUMA calls; it belongs on a dedicated model host, not a general-purpose one.
+- Mount model directories read-only; a container with `/models` writable adds nothing a serve needs.
+- A control vector can remove refusal behaviour. Loading one is a safety decision, not a
+  performance one; the flag is `--control-vector`, and `--control-vector disabled` is the default.
+
+## Failure signatures at startup
+
+| what you see | what it is |
+|---|---|
+| unknown/retired flag, or a flag refused by name | a boot refusal — the pin does not have it; change the trio, not the invocation |
+| "no plugin claims `<arch>` at `<quant>`; served at: …" | the (id, descriptor) pair has no plugin; fix the descriptor with `--set`/convert |
+| "reports ABI N, engine M" | rebuild the plugin against this engine |
+| budget refusal / `host_pool` short | the shape does not fit this card/host; lower demand or pin more host RAM |
+| `--ngram-placement ram` refuses | memlock short; raise it or use `auto`/`disk` |
+| `--kv-cache-dtype fp8_v4`/`k6_v4` refused, plugin named | no kernels for this geometry |
+| `--p2p on` refuses, pairs named | the driver has no peer access; use `auto`/`off` |
+| first launch of a never-run kernel faults | too little `--gpu-headroom-mib` for its code object |
+| unit loop, no container, no engine log | a preflight refusal (TTM/memlock/digest/model) re-run by `Restart=always` |
+
+## Related
+
+- [[Radiance Serve Options]] — every knob and its law.
+- [[Radiance HTTP API]] — readiness and the wire.
+- [[Radiance Tools]] — how to corroborate a receipt.
+- [[frank-archon]] — the fleet's radiance unit, guard chain and Eras.
+- [[Model Swap and Rollback]] — the swap discipline this page's verification serves.
+- [[Deployment Flow]] — how a committed change reaches the host.

serverAppConfigs-docs/LLM/Radiance Plugins.md

@@ -0,0 +1,371 @@
+---
+type: "howto"
+title: "Radiance Plugins — kernel libraries and quantisers"
+description: "Writing radiance kernel and quantiser plugins: the ABI, kernel rows, hooks, encodings, and the build path."
+tags:
+  - llm
+  - radiance
+  - plugins
+  - howto
+timestamp: 2026-10-30
+---
+
+# Radiance plugins — kernels and quantisers
+
+Three kinds of `.so` extend the engine, each exporting a few C symbols and linking **nothing of
+radiance's** (they compile against the installed `abi/` headers):
+
+| kind | value | contract |
+|---|---|---|
+| kernel library | `RAD_PLUGIN_KERNEL` (1) | ops for some hardware: a card family, the CPU, an oracle |
+| architecture | `RAD_PLUGIN_ARCH` (2) | one model family at one weight format — see [[Radiance Architecture Plugins]] |
+| quantiser | `RAD_PLUGIN_QUANT` (3) | weights → encoding planes, for `rad-convert` |
+
+In-tree plugins use the same interface a third party does: each builds with `rad_add_plugin` and is
+found and ranked by what it declares, never by its name. `ctest -L oot` rebuilds every bundled
+plugin out of tree against an install.
+
+## Identity and ABI
+
+```c
+#include "rad_abi.h"   /* required; pulls rad_types.h and rad_encoding.h */
+
+uint32_t rad_plugin_abi_version(void) { return RAD_ABI_VERSION; }
+
+const RadPluginInfo* rad_plugin_info(void) {
+    static const RadPluginInfo i = {
+        RAD_PLUGIN_KERNEL, "mykernels", "0.1", "a GEMM or two", "gfx1100",
+    };
+    return &i;
+}
+```
+
+- `RAD_ABI_VERSION` (31 at this commit) is bumped on any shape change in a header a plugin compiles
+  against. Return the macro; a mismatch is refused at load with both numbers printed.
+- `build_target` is printed, not trusted: the targets the library was built for (`"gfx1100"`,
+  `"gfx11-generic"`, several separated by spaces/commas), `"host"`, or `""` for portable. The engine
+  checks the library's actual code objects instead (§"A library for a new card").
+- Optional exports, resolved by exact name: `rad_plugin_open` (return `RAD_E_UNSUPPORTED` to decline
+  this machine and unload), `rad_plugin_close`, `rad_kernel_concurrent`, `rad_kernel_collective`,
+  `rad_plugin_reference`, `rad_kernel_reference`, `rad_kernel_chain`.
+
+## What a kernel library exports
+
+```c
+int                rad_kernel_schema_count(void);
+const RadOpSchema* rad_kernel_schema_at(int i);
+int                rad_kernel_count(void);
+const RadKernelInfo* rad_kernel_at(int i);
+```
+
+A row declares one op implementation and what it can serve:
+
+```c
+static const RadConstraint c_gemm[] = {
+    RAD_CLE("M", 64),              /* M <= 64            */
+    RAD_CDIV("K", 128),            /* K divisible by 128 */
+    RAD_CIN("dtype", "fp8a8 w4a8") /* dtype in that set  */
+};
+
+static const RadKernelInfo k[] = {{
+    .name = "mygemm_m64", .op = "gemm_nt_q", .family = "gemm",
+    .computes = "row-major A times column-major B, fp8 in, bf16 out",
+    .shape    = "M<=64, K%128==0",
+    .dtypes   = "fp8a8, w4a8",
+    .domain   = RAD_DOMAIN_DEVICE,
+    .priority = 50,
+    .constraints = c_gemm, .n_constraints = 3,
+    .launch = my_launch,
+}};
+```
+
+- Operators: `RAD_CEQ`, `RAD_CLE`, `RAD_CGE`, `RAD_CDIV`, `RAD_CIN`. The engine takes bucket
+  boundaries from `LE`/`GE`/`EQ` and resolves once per band; `DIV` and `IN` are not intervals.
+- **Declare only the band you serve.** The engine believes the claim; a kernel that says `M <= 2048`
+  but is correct to 64 is wrong at concurrency.
+- `device.*` keys are matched against what the engine measured about the ranks' cards, not against
+  op parameters: `RAD_CEQ(RAD_FACT_P2P, 1)` means the kernel touches another card's memory. libr4d
+  pairs every collective with a `_host` twin requiring `device.p2p == 0`, so exactly one matches any
+  machine.
+- **Priority breaks ties within one plugin**, highest first, then declaration order. **Between
+  plugins the hierarchy is absolute**: the first library in `--kernels` order with any matching row
+  supplies the op, however specialised a later library's row is. That makes a user's library an
+  override, not a competitor.
+- A derivative parameter (`RAD_DERIVED`) is supplied by the kernel; a constraint on a key the
+  geometry omits does not hold during matching, and the winner's `RAD_C_EQ` writes it into the
+  resolved geometry. `RAD_PROLE_SEQ_CHUNK` marks the parameter the scheduler must cut steps on;
+  `RAD_PROLE_CAPACITY` marks a count the kernel only bounds itself by.
+
+## Operands and extents
+
+`RadArgs.t` is positional in the schema's order; each `RadTensor` carries the extent the caller
+narrowed it to, usually smaller than the buffer. Size loops and grids from those extents, never from
+a declared maximum. Where an op takes `cu_seqlens`, derive the sequence count from its own length
+(`n = len - 1`) and each sequence's rows from `cu[n]` and `cu[n+1]`.
+
+An absent optional operand is a **null `RadTensor.data`**, not a shorter operand count.
+`rad_arg_in(a, i)` fetches one correctly; `rad_tensor_rows`, `rad_tensor_lastdim`,
+`rad_tensor_row_pitch`, `rad_args_geti_or`, `rad_args_getf_or`, `rad_args_gets_or` are the other
+everyday helpers in `rad_plugin.h`.
+
+## Optional hooks
+
+| hook | meaning |
+|---|---|
+| `init` / `fini` | per-resolved-instance state; `RadArgs.instance` hands it back. `init` runs once per (op, band, domain) after selection and before load |
+| `scratch` | bytes of scratch one launch needs; sized at declare from the band's upper bound |
+| `layout` / `relayout` | the kernel stores this weight in its own order, and how the loader produces it from the canonical planes |
+| `unrelayout` | the inverse, so a tool holding stored bytes can compare against a dense oracle |
+| `opd_shape` | size and fill every operand, so a tool can call the kernel cold |
+| `replaces` | the chain of conventional ops this kernel computes in one launch |
+| `tunables` / `tune_valid` | the instantiation axes, and which combinations are legal at a shape |
+
+`replaces` must be **exact** and is compared byte for byte against the chain — a kernel that is only
+close publishes nothing. `tunables` declares axes and legal values, not named variants; the core
+takes the cross product, and `deflt` must be launchable (the loader refuses the plugin otherwise).
+`rad-tune` measures survivors and writes the cache the selector reads; the chosen point arrives as
+ordinary parameters (`rad_args_geti("sk")`), with no variant index.
+
+`layout` answers: `RAD_OK` (own form, `relayout` will produce it), `RAD_E_UNSUPPORTED` (stored as
+is — the common case, and the only form that can be read in place from the file/mapped tier),
+`RAD_E_DTYPE` (this encoding is not mine — a declare-time error), `RAD_E_SHAPE`, or `RAD_LAYOUT_ANY`
+(reads the bytes without interpreting them; claims no storage). A weight only `LAYOUT_ANY` readers
+touch is stored as it is.
+
+## What a launch may do
+
+On a card the `RadStream` is **the engine's own AQL queue**, not a `hipStream_t`. The engine binary
+defines and serves these HIP entry points:
+
+- `hipLaunchKernel` (and `<<<>>>`, `hipLaunchKernelGGL`);
+- `hipMemcpyAsync`, `hipMemcpy2DAsync`, `hipMemsetAsync`;
+- `hipStreamSynchronize` (not from a launch — a launch must not synchronise);
+- `hipMemset` (runs on the engine's queue and waits);
+- `hipGetLastError` / `hipPeekAtLastError`, with HIP's semantics.
+
+**Nothing else in HIP takes an engine stream.** `hipModuleLaunchKernel`, `hipExtLaunchKernel`,
+cooperative launches, `hipLaunchHostFunc`, and libraries that launch through them (rocBLAS,
+hipBLASLt, MIOpen, hipRTC) would hand HIP a queue it does not know. Header-only device code
+(Composable Kernel templates, rocPRIM primitives) compiled into your `.hip` works. `hipEventRecord`
+and `hipStreamWaitEvent` on an engine stream are refused by name.
+
+The engine also reads the code object, which limits a kernel to: **no hostcall** (`printf`/`assert`
+are refused at the first launch), **no dynamic stack**, **≤ 768 bytes of kernel arguments**, **≤ 64
+KiB static+dynamic LDS per dispatch**, and an **uncompressed fat binary**. Device memory a kernel
+needs comes from the `scratch` hook, never `hipMalloc` — the engine's allocator knows what is left.
+
+`rad_plugin.h`'s conversions define the result: the reference contract is f32 accumulation, IEEE
+round-to-nearest-even, **no FMA and no reassociation**. A plugin whose bf16 narrowing truncates
+cannot be checked against the reference. `rad_load_f32`/`rad_store_f32`/`rad_load_bits`/
+`rad_store_bits` read and write any core dtype from a plane.
+
+## Colours of agreement
+
+- `rad_kernel_concurrent(i)` — kernel `i` touches only its operands and scratch, so independent
+  launches may overlap.
+- `rad_kernel_collective(i)` — it exchanges data with other ranks; the engine keeps bulk link
+  traffic away. `2` promises more: the launch ends the hold itself once the other ranks' bytes have
+  landed, through a `link_gate` word it reaches at system scope; returning `RAD_LINK_ENDED` says it
+  did.
+- `rad_kernel_chain(i)` — the kernel can wait in its own prologue instead of behind the packet
+  processor's barrier bit. Chained forms are recognised by `r4dchain` in the name and use a
+  `RAD_CHAIN_REC_BYTES` record at `RAD_CHAIN_KARG_OFF`.
+- `rad_plugin_reference` / `rad_kernel_reference` — an oracle library (or an oracle row beside a
+  device row). A reference library ranks below every unnamed library, is refused for serving unless
+  `--debug-accept-reference-kernels`, and is what `rad-kbench` checks against.
+
+## Defining a new op
+
+Only when `docs/OPS.md` has no name for it. The vocabulary document owns the rules: what a schema
+must declare, and that **the first plugin in hierarchy order to declare an op fixes its schema** — a
+later plugin declaring it differently is refused at load, both named. `ctest -R ops_doc` checks the
+document against the registries, so the rules live there only. An op in `docs/OPS.md` is describable
+by `libref`, and therefore checkable, even when your kernel cannot describe itself; an op you invent
+runs only on a library that implements it.
+
+A schema is positional data, not code:
+
+```c
+static const RadParamSpec params[] = {
+    {"M", RAD_P_INT, RAD_REQUIRED, RAD_PROLE_NONE},   /* the ranged extent */
+    {"n", RAD_P_INT, RAD_REQUIRED, RAD_PROLE_NONE},
+    {"eps", RAD_P_F64, RAD_REQUIRED, RAD_PROLE_NONE},
+    {"dtype", RAD_P_STR, RAD_REQUIRED, RAD_PROLE_NONE},
+};
+static const RadOperandSpec opds[] = {
+    {"x", RAD_OPD_IN, 0}, {"w", RAD_OPD_WEIGHT, 1}, {"y", RAD_OPD_OUT, 0},
+};
+static const RadOpSchema schemas[] = {
+    {"my_op", params, 4, opds, 3, "what it computes, for a human"},
+};
+```
+
+`rad_kernel_schema_count/at` return this table. `RAD_REQUIRED`/`RAD_OPTIONAL`/`RAD_DERIVED` and the
+operand roles (`IN`, `OUT`, `INOUT`, `WEIGHT`, `WTAB`) are the whole vocabulary; the kernel's
+constraints and launch do the rest.
+
+## Quantiser plugins
+
+`abi/rad_quant.h` is the whole contract:
+
+```c
+int                     rad_quant_count(void);
+const RadQuantizerInfo* rad_quant_at(int i);
+
+typedef struct RadQuantizerInfo {
+    const char*           name;       /* what a recipe names */
+    const char*           doc;
+    const RadQuantOption* options;    /* key, RAD_P_INT|F64|STR, default text, doc */
+    int                   n_options;
+    RadQuantEncodingFn    encoding;   /* the encoding this quantiser would write */
+    RadQuantRowBlockFn    row_block;  /* rows per quantize() call; 0 = whole weight */
+    RadQuantizeFn         quantize;   /* logical rows -> planes */
+    RadQuantDecodeFn      decode;     /* optional, only for a scheme of its own */
+} RadQuantizerInfo;
+```
+
+The flow for each weight a recipe rule picks: ask `encoding` (given name, shape, and — at declare —
+no movement group), let the architecture declare against that encoding, then hand the weight over in
+row blocks, so a 51 GB table streams. Other calibration data is the quantiser's own option
+(`calib=DIR`); the output is **canonical planes only**, so every kernel that accepts the encoding and
+the core's decoder can read it — that is how the converter measures error without knowing the method.
+
+A quantiser's own scheme must be decoded by its `decode` hook (the oracle, `rad-kbench` and the
+error report call it), and must return `RAD_E_UNSUPPORTED` for an encoding it did not write. Serving a
+new encoding also needs a kernel library whose `layout` accepts it and an architecture that declares
+the weight at it — a new format is a new kernel row plus a plugin, never a converter-only change.
+
+Two plugins naming the same quantiser are refused, both named.
+
+### The encoding model (`abi/rad_encoding.h`)
+
+An encoding is the code type, an optional codebook, the grouping of scales/zero points/minimums, and
+a transform. It is **data, not an enum**, so a plugin can invent one. The core can size, slice and
+place any encoding from the struct alone, and decodes the `plain` and `affine` schemes itself:
+
+```
+q  = code                                   codes TILED, block [1x1], or [1xV] with a grid
+q  = grid[code][k]                          grid TABLE [entries x V]
+q  = table[code]                            table TABLE [1 x entries]
+q  = -q if the sign bit is set              signs TILED u1
+q  = q - zero                               zero TILED
+v  = q * scale * scale.1 * ...              every scale level
+v  = v - min * min.1 * ...                  a minimum in the value's domain
+w  = v . T^-1                               the transform, per row over columns
+```
+
+So NVFP4 is fp4 codes, e4m3 scale per 16, f32 `scale.1` per tensor; Q4_K is u4 codes, u6 scale and
+u6 min per 32 each times an f16 per 256; IQ2_XXS is u8 codes per 8 into a 256×8 grid, a sign bit per
+value and two scale levels.
+
+- Planes are **TILED** (cover the weight, element `(i,j)` covering a `[br,bc]` block; `0` extent means
+  the whole extent) or **TABLE** (a codebook, grid, rotation parameter; every slice carries it).
+- Sub-byte elements are a bit stream along the row, lowest bits first; every row starts on a byte.
+- Transforms: `""`, `fwhtN` (Hadamard before quantisation; the `1/N` lives in the stored scale),
+  `perm` (GPTQ act-order, a TABLE `t.perm`), `givens` (ParoQuant rotations: `t.pairs`, `t.angle`,
+  optional `t.scale`, per-part `t.part`, composed `t.dense`).
+
+A kernel only *rearranges* planes (`relayout`); choosing numbers is the quantiser's job. That split
+is why a container never has to be reconverted when a kernel library changes its layout.
+
+## Build, install, search path
+
+```cmake
+cmake_minimum_required(VERSION 3.21)
+project(mykernels LANGUAGES CXX HIP)     # CXX alone for a host-only plugin
+find_package(radiance REQUIRED)
+rad_plugin_defaults()                    # C++20, optimising build unless asked otherwise
+rad_add_plugin(mykernels kernels SOURCES rows.cpp HIP_SOURCES kernels.hip)
+```
+
+The kind is `kernels`, `architectures` or `quantizers` — the directory the engine loads it from.
+Only `HIP_SOURCES` compile as HIP: for `RAD_GPU_TARGETS` or a `GPU_TARGETS` list, with `-DNDEBUG`
+outside Debug, uncompressed, against the HIP runtime the installed radiance found. `find_package`
+also sets `RADIANCE_ABI_VERSION` (to fail at configure time rather than at `dlopen`),
+`RADIANCE_HOME`, `RADIANCE_KERNELS_DIR`, `RADIANCE_ARCHITECTURES_DIR`,
+`RADIANCE_QUANTIZERS_DIR`. `cmake --install` puts the plugin under the home of the radiance you built
+against; `RAD_PLUGIN_INSTALL_HOME` overrides that.
+
+**`$RADIANCE_HOME` is a search path**: homes separated by `:`, the first winning; each may hold
+`kernels/`, `architectures/`, `quantizers/`, and an earlier home's same-name file shadows a later
+one. Your plugins go ahead of an installation without copying it:
+
+```sh
+RADIANCE_HOME=$HOME/radiance-plugins:/opt/radiance/share/radiance radiance --model m.rad
+# in the container:
+-v /my/plugins:/plugins -e RADIANCE_HOME=/plugins:/opt/radiance/share/radiance
+```
+
+The **kernel hierarchy** is `--kernels` (or `$RADIANCE_KERNELS`): `--kernels mykernels,libr4d` puts
+yours before libr4d. An entry matches a plugin's name or its file (`libr4d`, `r4d`, `libr4d.so`, a
+path). An entry matching nothing is warned about by name. With no list, every library found loads,
+reference last.
+
+## A library for a new card
+
+The engine's own device code is compiled for every AMD family's generic target
+(`gfx9-generic`, `gfx9-4-generic`, `gfx10-1-generic`, `gfx10-3-generic`, `gfx11-generic`,
+`gfx12-generic`) beside the build's targets, so one engine binary starts anywhere. What it computes
+there depends on the installed kernel library.
+
+- Configure with `-DRAD_GPU_TARGETS=gfx942` (or `gfx11-generic`). At load the engine asks the runtime
+  which targets the card accepts, in order (chip, then family: `gfx1036`, then `gfx10-3-generic`),
+  with the card's modes (`gfx942:sramecc+:xnack-`), and takes the best fit in the fat binary.
+- **A library with nothing for this card is left out, not failed**: its device rows leave selection,
+  its host rows stay, and the log says what it carries versus what the card runs. One installation
+  can carry a library per card. `rad_plugin_open` may decline explicitly with `RAD_E_UNSUPPORTED`.
+- The engine uses discrete cards if any exist (all of the first one's architecture), integrated only
+  when there is nothing else; `ROCR_VISIBLE_DEVICES` chooses explicitly. On integrated graphics,
+  device memory is host memory and the budget leaves an eighth of host RAM to the host.
+- **fp8 bytes are OCP E4M3** (`e4m3fn`, bias 7, max 448) wherever the engine reads them: container
+  weights, an fp8 KV cache, activations. A card whose fp8 is `e4m3fnuz` (gfx942) converts in its
+  `relayout` hook (the codes keep their bits except `0x80` and decode to half the value, so the scale
+  doubles) and keeps activations/KV in OCP bytes, or converts at its own boundaries.
+- **Tensor parallelism is ops you supply.** `all_reduce`, `all_gather` and the fused collectives are
+  rows like any other; the engine inserts none and assumes no link.
+
+## Checking your work
+
+```sh
+rad-kbench --kernels mylib,libr4d,libref --report mylib.md --bench
+```
+
+- It replays a recorded fixture of `libref`'s answers over real model geometries
+  (`$RADIANCE_HOME/kernels.rkb`, plus `kernels_moe.rkb`), running every library with a row for a
+  case. Correctness uses sampled values **and** the full-tensor norm **and** seeded projections;
+  completeness is the op-by-library matrix; speed is timed on the same bytes.
+- Red zones: every device allocation carries a poisoned 256-byte margin, verified after the launch.
+- Fusion: a row with `replaces` is run against its named chain, out of your library, compared with
+  `memcmp`.
+- A skip is named, not failed — usually a missing `opd_shape`. `--strict` also fails a stale fixture.
+- `rad-avx-check --ref libref.so --avx mylib.so` checks a **host** library without a card, op by op,
+  at every ISA level and with parallel regions forced onto several threads (for a library exporting
+  `avx_isa_force`/`avx_par_force`).
+- `rad-schemas mylib.so libref.so` prints every schema disagreement at once (it exits 1 on any), and
+  works on a plugin the engine cannot load.
+- `--debug-selection` prints which row took each band and which constraint refused each other
+  candidate; `/graph` on a running server shows the declared graph and which declarations the run
+  phase issued.
+- `rad-info --plugins` lists every plugin on the search path as the engine loads it: kind, version,
+  declarations, reference status, whether its device code runs on this card, and its file.
+
+## A worked example to copy
+
+`plugin/spechub/` is the smallest real kernel plugin in the tree: two ops (`mtp_couple_propose`,
+`mtp_couple_verify`), each with a device row and a host reference row. Read it in this order:
+
+- `exports.cpp` — the whole export block: info, ABI, schema count/at, kernel count/at, plus
+  `rad_kernel_reference` classifying device rows 0 and host rows 1 so the reference gate does not
+  refuse the device rows.
+- `spechub.cpp` — the rows, their constraints and their argument readers.
+- `spechub.hip` / `ref_spechub.cpp` — the device launch and the host oracle.
+- `CMakeLists.txt` — the `rad_add_plugin` call, the optional `enable_language(HIP)`, and
+  `-ffp-contract=off` because an FMA in the ordered FP64 mass sums would move a token off the host
+  reference.
+
+## Related
+
+- [[Radiance Architecture Plugins]] — the third plugin kind, one level up.
+- [[Radiance Conversion]] — how a quantiser is driven by a recipe.
+- [[Radiance Tools]] — `rad-kbench`, `rad-tune`, `rad-schemas`, `rad-avx-check` in full.
+- [[Radiance Serve Options]] — `--kernels`, `--radiance-home`, the tuning aids.

serverAppConfigs-docs/LLM/Radiance Serve Options.md

@@ -0,0 +1,240 @@
+---
+type: "reference"
+title: "Radiance Serve Options — every flag and the law behind it"
+description: "Every radiance serve flag grouped by task, with its default and the law that decides when to move it."
+tags:
+  - llm
+  - radiance
+  - reference
+  - serving
+timestamp: 2026-10-30
+---
+
+# Radiance serve options
+
+`radiance --help` prints this table. Every numeric option takes a whole number (not `12k`, not
+`auto`, unless the option names those); a malformed value exits 2. Upstream `main` at 2026-10-30;
+the fleet is pinned older — see the pin note on [[Radiance Engine]].
+
+The one thing to read before any table: **the engine derives the VRAM split; no flag sets either
+pool.** The startup receipt prints every term before anything is allocated.
+
+## Model, plugins, container
+
+| flag | default | meaning |
+|---|---|---|
+| `--model PATH` | required | a `.rad` container, a checkpoint served as it stands, or a Hugging Face repo id (`owner/name`) fetched before startup |
+| `--download-dir DIR` | `$XDG_CACHE_HOME/radiance/hf`, else `~/.cache/radiance/hf` | where a repo id is fetched |
+| `--radiance-home DIRS` | compiled prefix | plugin homes, `:`-separated, first wins |
+| `--kernels LIST` | every library found | kernel libraries, `,`/`:` separated, first wins; a library not named is not loaded |
+| `--served-model-name NAME` | container's name | the id `/v1/models` lists and responses carry; a request may name any model |
+| `--override-chat-template PATH` | container's | a Jinja template served instead; an unreadable or empty file stops startup |
+| `--generation-config PATH` | none | sampler defaults from a `generation_config.json` (else `<model>.generation.json`, then the container's `generation.*` metadata) |
+| `--control-vector off\|on\|disabled` | `disabled` | load the GGUF control vector found for the model and start requests off/on; `disabled` declares no projection and refuses a request that asks |
+| `--api-key KEY` | none | require `Authorization: Bearer KEY` or `X-Api-Key: KEY`, except `/health`, `/v1/health`, `/ping`, the dashboard |
+| `--mm-max-patches N` | `auto` (16384 with a vision tower) | patches one encoder pass carries: the largest image, 16×16 px a patch (14×14 on GLM-5.3). `0` serves text only and keeps the tower off the card |
+
+## Batch shape, context, scheduler
+
+| flag | default | meaning |
+|---|---|---|
+| `--max-num-batched-tokens N` | 2048 | the largest token count one step may carry; on MoE the knob that sets prefill chunk size, and the activation arena's size |
+| `--max-num-seqs N` | 256 | the most sequences one step may carry; the cache's **demand**, never its funding |
+| `--max-model-len N` | 0 = model's training context | context bound per request. Longer is a 400, never a truncation. Reserves nothing per request; it is the worst case the pool is sized against |
+| `--busy-prefill-chunk auto\|off\|N` | `auto` | a prefill chunk's share of a step that has other work: `auto` leaves decode rows and waiting requests room, `off` takes the whole step, `N` caps it |
+| `--prefill-interleave N` | 0 (off) | decode-only steps after each chunk taken beside decoders, at most 64; buys decoder cadence with the prompt's TTFT |
+| `--prefill-span N` | 1 | checkpoint intervals a prefill **alone in its step** may carry, from its second interval; needs `--max-num-batched-tokens` of N intervals. Every other step is planned at the shared budget; checkpoints inside a span are not written |
+| `--preempt-wait-ms N` | 5000 | how long a preempted prefill waits for the whole-recompute gate before reclaiming blocks and preempting lower-priority work; `0` falls back next step |
+| `--rope-scaling SPEC` | the checkpoint's | `yarn:FACTOR[:ORIG]` writes static YaRN over the container's own rotary keys with no reconvert; `none` takes a checkpoint's YaRN off |
+| `--max-queued-requests N` | 8 × `--max-num-seqs` | admitted, unfinished requests; past it the answer is 429 |
+| `--retry-after S` | 1 | the `Retry-After` a 429 carries |
+
+## Memory and placement
+
+| flag | default | meaning |
+|---|---|---|
+| `--placement MODE` | `auto` | `all_vram`, `layer_offload`, `expert_tiered`, `auto` |
+| `--gpu-headroom-mib N` | 96 | VRAM left unclaimed on every card; too little makes the first launch of a never-run kernel fail inside HIP |
+| `--host-pool-mib N` | 0 | pinned host tier for weights, **per rank**; also the KV lever under expert tiering (below) |
+| `--host-headroom-mib N` | 4096 | host RAM left free when startup pins memory; a snapshot store's floor takes the shortfall back only down to 1 GiB |
+| `--kv-cache-dtype T` | `bf16` | `bf16`, `fp8`, and on Qwen3.8-27B/-Flash-Next `fp8_v4` (V int4) or `k6_v4` (K int6 + V int4). A real quantisation; GLM-5.3 is bf16-only |
+| `--state-cache-dtype T` | `f32` | `f32` or `f16`: the recurrent delta-net state cache; f16 halves state pool, checkpoints and traffic and is lossy — `qwen35` plugins only |
+| `--attn-prefill-fp8 MODE` | `off` | `off`, `qk`, `pv`, `both`: prefill attention products in E4M3 over an `fp8`/`fp8_v4`/`k6_v4` cache; lossy, Qwen3.8-27B only where kernels exist |
+| `--kv-resident N` | 0 (off) | stream the sparse attention group's KV from pinned host RAM, keeping an N-token ring on the card. Raised to 32768 and to one decode step's selection; ignored with a warning where unsupported |
+| `--embedding-placement ram\|vram` | `ram` | token embedding pinned in host memory and read in place, or on the card. A tied head or a host that cannot pin keeps it on the card; an explicit `ram` refuses instead |
+| `--vision-tower-placement vram\|ram` | `vram` | encoder towers on the card, or pinned in host memory and read once per image, giving their VRAM to the KV cache |
+| `--ngram-placement auto\|vram\|ram\|disk` | `auto` | where an n-gram embedding table lives: all of it on every card, a pinned host copy the card reads, or rows from the container through a row cache. `auto` takes the first that fits. `ram=DIR` makes the pinned copy a tmpfs file shared by two engines |
+| `--weights-disk-tier` | off | read weights that fit neither VRAM nor the host pool from the container with O_DIRECT; every step is paced by the drive |
+| `--row-prefetch on\|off` | on | ask for a host gather's pages before its kernel; output-neutral |
+| `--mapped-rewarm on\|off` | on | at startup, re-ask the page cache for a mapped table's pages an earlier run left behind |
+| `--deterministic` | off | pin placement to the loader's plan so the resident set cannot move under the heat engine; it does **not** make output bit-identical across batch positions |
+
+## KV cache, host tier, checkpoints
+
+| flag | default | meaning |
+|---|---|---|
+| `--prefix-cache-mib N` | 0 (off) | copy finished conversations' KV to host memory so a return restores instead of re-prefilling; LRU when full; off while a KV group streams |
+| `--prefix-cache-ttl S` | 3600 | seconds an unused cached conversation is kept; `0` keeps it until memory is wanted |
+| `--no-prefix-cache` | cache on | disable prefix caching |
+| `--checkpoint-interval N` | 2048 | linear-attention state checkpoint interval, in tokens; also clamps a hybrid model's prefill chunk |
+| `--checkpoint-slots N` | `auto` | snapshots held **in total**, in pinned host memory; `auto` = per-policy need for each sequence within an eighth of host RAM; `0` keeps none |
+| `--checkpoint-policy NAME` | `geometric-backoff` | also `tip-only`, `keep-all`; retention only — writes happen at every interval unless a span skips them |
+| `--prefix-resume exact\|nearest` | `exact` | `exact` resumes only where a cold prefill starts a chunk, so a hit answers bit-identically; `nearest` takes the deepest snapshot, up to a chunk less re-prefilled |
+
+## Speculation
+
+| flag | default | meaning |
+|---|---|---|
+| `--num-speculative-tokens N` | `auto` (the drafter's own point; 0 with no head) | speculative window; 0 disables. The ceiling under `adaptive` |
+| `--spec-ngram K` | 0 (off) | prompt lookup alone, up to K drafts a step from the sequence's own n-grams; leaves a container's draft head out; sets the window to K |
+| `--spec-ngram-min N` / `--spec-ngram-max N` | 2 / 4 (max 15) | pattern lengths prompt lookup tries, longest first |
+| `--draft-window fixed\|adaptive` | `fixed` | `adaptive` follows each sequence's acceptance and the measured step cost |
+| `--draft-short-from N` | `auto` | a block drafter that declares a shorter block runs it from N decoding sequences; `0` never; `auto` = never for a chain, from 2 for a tree |
+| `--draft-coupling off\|on` | off | sample each proposal from the drafter's own shortlist and couple the target's draw to it; same output law, different tokens for a fixed seed, pays at high temperature |
+| `--draft-model PATH` | — | **not implemented** at serve: merge the drafter at convert time (`rad-convert --draft-model`) |
+
+## Tensor parallelism and collectives
+
+| flag | default | meaning |
+|---|---|---|
+| `--tp N` | 1 | tensor-parallel ranks; declare divides dimensions by world size and inserts no collective |
+| `--tp-wire exact\|wht6` | `exact` | `wht6` is lossy: rotated 6-bit at 2 ranks, tiered int8 over a two-shot at 3–4, unavailable at 8 |
+| `--tp-wire-min-kb N` | 128 | smallest all-reduce (KiB) the lossy wire compresses; keeps single-sequence decode exact |
+| `--tp-attn-split heads\|rows` | `heads` | a prefill step's attention split: `rows` gives each rank all heads of its rows plus an all-gather; faster where attention dominates, not bit-identical, plugin-gated |
+| `--weight-prefetch on\|off` | on | read a GEMM's weight beside the collective before it; `off` when another server shares the cards |
+| `--p2p auto\|on\|off` | `auto` | `auto` asks the driver; `on` refuses to start without peer access; `off` exchanges through pinned host memory |
+
+## LoRA adapters
+
+| flag | default | meaning |
+|---|---|---|
+| `--lora NAME=PATH` | none | serve an adapter: a PEFT directory as model NAME (text), or a `.safetensors` a request names in `loras` (image/video). Repeatable |
+| `--lora-dir DIR` | none | generators: any `NAME.safetensors` under DIR may be requested |
+| `--max-loras N` | count of `--lora`, 2 with `--lora-dir` alone | adapter slots resident at once |
+| `--max-lora-rank R` | 16 text / 64 generator | slot rank; a larger text adapter is refused, a larger generator combination truncated |
+| `--lora-target-modules LIST` | the adapters' own | HF module leaf names adapters may target; needed with `--max-loras` and no `--lora` |
+
+## Request defaults and bounds
+
+A request that names a field wins. These set defaults for requests that do not.
+
+| flag | default | meaning |
+|---|---|---|
+| `--temp F`, `--top-k N`, `--top-p F`, `--min-p F` | from the model | sampler defaults; `top_k` 0 is the 1024 most likely candidates |
+| `--typical-p F` | 1 | typical sampling |
+| `--presence-penalty F`, `--frequency-penalty F` | 0 | range [-2, 2] |
+| `--repetition-penalty F`, `--repeat-last-n N` | 1, 64 | `-1` looks back over the whole context |
+| `--dry-multiplier F`, `--dry-base F`, `--dry-allowed-length N`, `--dry-penalty-last-n N` | 0, 1.75, 2, -1 | DRY; multiplier 0 is off |
+| `--dry-sequence-breakers JSON` | `["\n", ":", "\"", "*"]` | at most 16 strings |
+| `--xtc-probability F`, `--xtc-threshold F` | 0, 0.1 | range [0, 1] |
+| `--default-max-tokens N` | `auto` | `max_tokens` for a request that sends none; `auto` is what the context leaves |
+| `--max-tokens-cap N` | 0 (none) | a larger request is **cut to it**, not refused; reply ends `finish_reason: "length"` |
+| `--max-n N` | max(8, `--max-num-seqs`) | the largest fan-out `n` |
+| `--max-stop-strings N`, `--max-stop-bytes N` | 64, 4096 | stop-string limits |
+| `--chat-template-kwargs JSON` | none | variables for the template on every chat request; a JSON object or `@FILE`; repeatable |
+| `--reasoning-effort S` | template's own | default `reasoning_effort`; passed to the template verbatim, so the legal set is the template's |
+| `--reasoning-effort-map MAP` | none | `FROM=TO,...`, checked against the template at startup; `none`/`minimal` turn thinking off |
+| `--thinking-token-budget N` | none | default budget: at N reasoning tokens the block is closed and the answer follows |
+| `--reasoning-format auto\|none` | `auto` | `auto` returns reasoning as `reasoning_content`; `none` leaves it inline |
+
+## Server and HTTP
+
+| flag | default | meaning |
+|---|---|---|
+| `--host ADDR` | 0.0.0.0 | listen address |
+| `--port N` | 8000 | listen port |
+| `--http-threads N` | one per hardware thread, at least 4 | workers kept while idle |
+| `--read-timeout S` | 30 | a client may go silent this long while sending |
+| `--write-timeout S` | 600 | one write to a client may block this long |
+| `--keep-alive-timeout S` | 5 | idle connection hold |
+| `--max-body-mib N` | 512 | largest body; images and video ride base64 inside it |
+| `--no-cors` | CORS on | send no CORS headers |
+
+## Media preprocessing
+
+Each replaces what the container's preprocessor config states, and does nothing for a model that
+takes no media.
+
+| flag | default | meaning |
+|---|---|---|
+| `--image-min-pixels N`, `--image-max-pixels N` | container's | the band an image resizes into; one encoder pass still bounds the top |
+| `--video-min-pixels N`, `--video-max-pixels N` | container's | same, for a video's sampled frames together |
+| `--video-fps F` | container's | frames sampled per second |
+| `--video-min-frames N`, `--video-max-frames N` | container's | frame-count bounds |
+| `--video-max-frame-tokens N` | container's | the most prompt tokens one frame becomes; `0` no cap; a token-budget processor takes 70/140/280/560/1120 |
+| `--image-tokens N` | container's | a token-budget processor's image budget (Gemma 4's fixed set; GLM-5.3's ceiling ≤ `--mm-max-patches`/4) |
+| `--max-source-pixels N` | 67108864 | largest input the decoder accepts before resize |
+
+## Quality measurement (instead of serving)
+
+| flag | meaning |
+|---|---|
+| `--kld-record DIR` | run `--kld-corpus` through prefill and write this model's per-position log-probabilities to DIR: the reference |
+| `--kld-ref DIR` | score this model against the reference: mean/median/percentile KL and top-1 agreement |
+| `--kld-corpus FILE` | JSONL, one `{"prompt", "score_from"?, "source"?}` a line; `--kld-record` only |
+| `--kld-out FILE` | the report as JSON; `FILE.rows` holds every position's numbers |
+| `--kld-lora NAME`, `--kld-lora-sources PREFIX` | score through a served adapter, all documents or a `source` prefix |
+
+The mode turns speculation and the prefix cache off and refuses `--num-speculative-tokens > 0`.
+See [[Radiance Conversion]] for the corpus and the scoring discipline.
+
+## Diagnostics
+
+| flag | meaning |
+|---|---|
+| `--debug-graph` | print the declared graph, every op's resolved kernel and why, then exit |
+| `--debug-selection` | the selection table only |
+| `--debug-placement` | the placement plan only |
+| `--debug-tape-dump` | the first recorded pass of each kind (`TAPEDUMP` on stderr, every rank) |
+| `--debug-draft-oracle FILE` | verify-path test: prompt lookup proposes from FILE with 30 % corrupted drafts; the text must not move |
+| `--debug-accept-reference-kernels` | serve even with `libref` rows — not a production configuration |
+| `--profile-ops` | per-op device timing |
+| `--expert-stats FILE` | routing counts per layer and step kind, JSON every 30 s and at shutdown |
+| `--expert-profile FILE` | rank the expert plane by a stats file's `decode` counts instead of the container profile |
+| `--expert-pin-top F` | pin the hottest fraction F of routed experts so heat never demotes them |
+| `--draft-expert-placement host\|auto` | pin a draft head's experts in the host pool before any strategy runs |
+| `--live` | the terminal live view; redirected, one `key=value` line |
+| `-v`, `-vv` | verbose, very verbose |
+
+## The laws behind the knobs
+
+**The split is derived, and the receipt is the referee.** Non-expert weights come off the top of
+the card; the remainder (`elastic`) is shared by the expert slab and the KV cache. `--max-model-len`
+× `--max-num-seqs` is the cache's **demand**; the funding is `elastic − expert_floor`. Under a
+lending cache (`auto`/`expert_tiered`, no `--deterministic`, VMM present) the cache takes what it
+can address and lends unused blocks back to experts. **Every MiB added to `--host-pool-mib` lowers
+the expert floor one-for-one and the cache takes it** — until the cache already holds its whole
+ceiling, where the extra host room shows up as unclaimed. Read the boot line
+`kv: the pool backs N % of the worst case …`, never arithmetic.
+
+**The activation arena comes off the budget before the split**, so both sides pay for
+`--max-num-batched-tokens`. While steps are small most of it is lent back to the expert slab; a
+larger step reclaims it synchronously. That is why a bigger chunk can cost decode residency. On a
+hybrid model the chunk is clamped to `--checkpoint-interval` before the token budget, and startup
+prints which bound applies.
+
+**The KV dtype law.** `bf16` and `fp8` are engine-wide; `fp8_v4`/`k6_v4` are narrower layouts whose
+kernels exist only for particular attention geometries, and the server refuses them elsewhere by
+plugin name. A drafter's cache stays fp8. Recurrent state and the QSA indexer keep their own widths.
+
+**The slot law.** A slot costs recurrent state, block tables at full context, snapshot-store RAM
+and all-reduce rows — never speed by itself. Raising `--max-num-seqs` raises demand; whether it is
+funded is the resolver's answer.
+
+**The spec law.** Measure speculation by step time, not tok/s. Prompt lookup pays where the reply
+copies its context and costs a verify row elsewhere. `adaptive` moves the depth only after eight
+steps and a 2 % gain, because a new window is a new pass shape.
+
+**Batch shape is part of the configuration.** A different chunk, span or spec depth changes the
+summation order, so greedy text can differ at a tie. Reproducibility holds per configuration, never
+across one.
+
+**Unknown request fields are ignored; known-but-unimplemented fields are guarded** (accepted at
+their off value, refused otherwise) — the list is on [[Radiance HTTP API]].
+
+## Related
+
+- [[Radiance HTTP API]] — what the flags expose on the wire.
+- [[Radiance Operations]] — the container contract and readiness.
+- [[Radiance Plugins]] — how a flag reaches a plugin (`RadBuildCtx`).
+- [[frank-archon]] — the measured deployment of this resolver on two R9700s.

serverAppConfigs-docs/LLM/Radiance Tools.md

@@ -0,0 +1,322 @@
+---
+type: "reference"
+title: "Radiance Tools — inspect, check, tune, tokenise"
+description: "The radiance inspection, measurement and tokeniser tools, with the flags and verdicts each one reports."
+tags:
+  - llm
+  - radiance
+  - tools
+  - reference
+timestamp: 2026-10-30
+---
+
+# Radiance tools
+
+Six binaries and `rad-export` are installed beside `radiance`; `rad-avx-check`, `r4d_selftest` and
+`rad-chatparse` live in the build tree. Every tool that loads plugins honours `RADIANCE_HOME`,
+`RADIANCE_KERNELS` and `--kernels` as the engine does. A numeric option takes a whole number: `12k`,
+`auto` (where the option does not name it) or an empty value exits 2.
+
+`rad-convert`, `rad-kbench` and `rad-tune` run the engine's startup with a different last step: the
+same plugins, in the same hierarchy, through the same declare phase, so they resolve kernels exactly
+as a serve would.
+
+## `rad-info` — what a container holds
+
+```
+rad-info [options] <model.rad>
+rad-info --plugins [--home DIRS]
+
+  -v, --verbose      list every tensor, not just the per-unit summary
+      --meta         metadata table, then stop
+      --vocab        vocab summary and normaliser chain, then stop
+      --profile N    top N expert-profile entries (default 20, 0 = all)
+      --recipe       the recipe the container was converted with, then stop
+      --plugins      every plugin the search path offers, as the engine loads it
+      --home DIRS    search path for --plugins (default $RADIANCE_HOME)
+```
+
+It opens the container read-only and loads no plugins (except for `--plugins`): a container names no
+kernel library. Output is aligned; byte counts are binary units and each count says what it counts.
+
+```sh
+rad-info m.rad                    # format, architecture, model, source quant, recipe, sections
+rad-info --recipe m.rad           # the recipe, expanded, one rule a line
+rad-info -v m.rad | head -40      # every tensor, encoding, shape and what made it
+rad-info --plugins                # kind, version, declarations, reference, device fit, file
+```
+
+The section table (strings, metadata, directory, planes, encodings, vocab, profile, data) is the
+container's layout; the metadata table, the weight directory grouped by layer and movement unit, the
+vocab, and the expert profile follow.
+
+## `rad-kbench` — correctness, completeness, speed
+
+```
+rad-kbench [options]                       replay the recorded fixture (no model)
+rad-kbench -m <model.rad> [options]        check against a real container
+rad-kbench -m <model.rad> --record FILE    record a new fixture
+rad-kbench -m <model.rad> --skeleton FILE  write a metadata-only copy to record from
+```
+
+**Replay** needs no model, weights or card. Geometries, seeds and `libref`'s reduced answers are
+checked in as `data/kernels.rkb` (dense model, block drafter, gqa 6) and `data/kernels_moe.rkb`
+(routed MoE, MTP head, gqa 8). `--fixture` is repeatable; `ctest` runs three replays — the other
+libraries against each recording, and `libref` against both at tolerance zero (its regression test,
+since a live comparison moves both sides together).
+
+Each library is asked for its row as the engine would ask it with that library alone: same
+constraints, derived parameters and weight-encoding filter, so a row the engine would pass over does
+not run here either (`--all-rows` runs every row whose constraints hold).
+
+**Against a container** (`-m`), weight operands are read from it: each weight's canonical planes,
+which the kernel gets through its own `layout`/`relayout` hooks and the oracle gets decoded, and
+every expert of a weight table. This is the only test of the artifact and of the layout hooks.
+
+**Recording** takes the reference's answer once at every point of every declared op, whichever
+library resolved it. Weights are drawn from a seed, not read from the container, so any machine can
+reproduce the reference and no card is needed. A skeleton (tables intact, data a hole) records the
+same fixture in a few megabytes.
+
+```sh
+rad-kbench -m model.rad --skeleton model.skel.rad
+rad-kbench -m model.skel.rad --max-cases 1 --record data/kernels.rkb
+rad-kbench -m moe.rad        --max-cases 1 --record data/kernels_moe.rkb
+```
+
+### Verdicts
+
+- An output is compared on sampled positions, the full-tensor norm, and 32 seeded random-sign
+  projections (which estimate rel_l2 to about 25 %). The largest of the three is the error.
+- Tolerances are per op and activation dtype (`dtype`, else `q_dtype`, else `kv_dtype`). Copies,
+  gathers, scatters and cache stores are exact. Every op in the fixtures has a row of its own.
+- Ids are compared exactly; where the contract leaves a tie open (`sample_argmax`, `router_topk`,
+  `sample_topk`), an id indexing a value equal to the reference id's is accepted.
+- A case whose reference and output are both identically zero checked nothing: **vacuous**, not
+  passed.
+- A failure, a red-zone overrun, a fusion that differs from its chain, or a host kernel that wrote
+  into an input exits 1. `--strict` also fails a stale fixture.
+- `--coverage DIR` keeps one file per fixture/library of the rows a replay reaches
+  (`data/kbench_rows/`); a listed row not reached fails, unless a memory skip kept its cases out.
+- Every pass ends with a line saying where its time went (draw, hashing, layout, upload, launch,
+  read-back, comparison, cache, timing).
+
+### Timing
+
+`--bench` times every case that passes, on the bytes it was checked on.
+
+- Device cases are timed **after** the checks, operands still on the device, after one sustained
+  warm-up. A time is device timestamps around a batch of back-to-back launches divided by the launch
+  count; the batch is queued behind a gate the host opens once submission is done, so neither the
+  host rate nor a sync is in the interval. Two untimed launches precede every batch; batches repeat
+  until the median's 95 % interval is within 1 % or `--bench-reps` is reached.
+- **Cold by rotation**: weights and inputs of 1/32 of the footprint or more are read from copies
+  spanning the footprint (`--bench-flush`, default 256 MiB, at least twice an R9700's caches), so
+  each launch reads them from memory as streamed weights do. Smaller inputs stay warm, as the
+  previous op leaves them.
+- In-place operands are restored from a pristine copy before every launch; the restores are timed
+  alone and subtracted.
+- Timing draws expert offsets/routing skewed (Zipf 0.5) and block tables as permutations; the
+  checked launch keeps the recorded draw.
+- The header prints the calibration: launch floor and streaming-read bandwidth warm and cold. If a
+  smaller cold probe runs >1.5× the largest's rate the rotation was cache-served and device times
+  are withheld. Throttler bits that came on during a batch repeat it, up to three times.
+- Host cases are timed in their own column, after an untimed launch, weights rotated over at least
+  256 MiB (twice LLC), for at most two seconds a case.
+
+### Flags
+
+| flag | meaning |
+|---|---|
+| `--fixture FILE` | replay recording (default `$RADIANCE_HOME/kernels.rkb`); repeatable |
+| `--record FILE` | with `-m`: write a fixture |
+| `--skeleton FILE` | with `-m`: write a sparse container copy |
+| `--rerecord FILE` | replay: write the fixtures' cases again with the reference's answers |
+| `--record-rows FILE` | replay: record a row-driven fixture from the cases |
+| `-m, --model FILE` | the `.rad` to read weight operands from |
+| `--home DIR`, `--kernels A:B` | plugin home; kernel libraries (default every one found) |
+| `--op NAME` | only this op (repeatable prefix match) |
+| `--ref NAME` | the oracle plugin's name (default: the library declaring itself a reference) |
+| `--no-ref` | replay without the reference among the tested libraries |
+| `--strict` | replay: a stale fixture fails |
+| `--all-rows` | run every row whose constraints hold |
+| `--coverage DIR`, `--coverage-write` | the row-coverage ratchet |
+| `--no-cache`, `--cache-dir DIR` | run the reference instead of reusing cached answers; cache location |
+| `--seed N` | base seed; every case derives its own |
+| `--tol X` | override every tolerance (bisecting only) |
+| `--max-ctx N` | deepest context considered (declare and the attention sweep) |
+| `--max-tok N` | `max_tok` declare runs at (default 512, small on purpose) |
+| `--mm-max-patches N` | encoder pass declare runs at (default 2048 when a tower is present; 0 leaves it out) |
+| `--kv-cache-dtype T` | `bf16` or `fp8`, as on the engine |
+| `--num-speculative-tokens N` | draft depth to declare at (default: the container's) |
+| `--max-cases N` | runs per distinct case (default 4; 0 = every one) |
+| `--max-bytes MIB` | skip a case needing more than this, by name (default: half available memory, 512–8192) |
+| `--no-fusion` | skip the fusion oracle |
+| `--bench`, `--bench-iters N`, `--bench-reps N`, `--bench-warm-ms N`, `--bench-flush MIB` | timing |
+| `--timing-only` | time every device kernel, check nothing (refuses `--record`) |
+| `--report FILE`, `--analysis FILE` | markdown report; splice in `tools/kbench-analysis.sh` output |
+| `-v` | every case, not only failures |
+
+Notes: `--max-bytes` matters on a loaded machine — one case holds each operand as drawn, as the
+kernel's copy, and on the device; without the cap a small machine is OOM-killed with no indication of
+the op. `--bench` times cold on purpose: an R9700's 52 MB bf16 weight read warm comes from cache at
+1.45 TB/s against 565 GB/s from memory, so warm times rank shapes but do not size a roofline. A
+kernel that cannot lay out an encoding is a **skip**, not a failure (`RAD_E_DTYPE`), and the report
+names the kernel, operand and encoding.
+
+## `rad-tune` — measure the instantiation space
+
+```
+rad-tune -m <model.rad> [options]
+
+  -m, --model FILE   the .rad whose declared graph supplies the shapes
+      --home DIR     $RADIANCE_HOME; cache at <home>/tune/<machine>.tune
+      --kernels A:B  kernel libraries to load, first wins
+      --op NAME      only ops whose name starts with NAME
+      --machine KEY  override the machine key (writing a cache for another machine)
+      --reps N       timing reps, median reported (default 5)
+      --iters N      launches per rep (default 32)
+      --max-tok N    max_tok declare runs at (default 8192)
+      --tp N         tensor-parallel width to declare at (default 1)
+      --num-speculative-tokens N|auto   draft depth to declare at
+      --dry-run      list what would be measured
+  -v, --verbose      every candidate, not just the winner
+```
+
+Kernels declare tunable axes, not named variants; `rad-tune` walks the cross product at this model's
+shapes and writes the winner per (op, geometry) to the cache the selector reads at startup.
+
+- **Declare at the shapes you serve**: `--tp N` divides every sharded N; a declare at world size 1
+  measures a program no `--tp 2` serve runs. `--num-speculative-tokens` puts the drafter's kernels in
+  the measured graph.
+- At `--tp > 1` every rank declares at once, each on its own thread and device: a collective's
+  `init` is a rendezvous, so declaring rank 0 alone hangs.
+- A collective is measured as the exchange: per-rank operands (own seeds, so identical halves cannot
+  pass), own scratch and stream, own instance; every rank fires at once and the clock stops when the
+  last stream drains. Non-collective rows run on rank 0 alone.
+- `RADIANCE_TUNE` pins a kernel's axes to compare a tuned cache against the defaults — a measurement
+  aid, not a deployment setting.
+
+## `rad-schemas` — the op vocabulary diff
+
+```
+rad-schemas libref.so                 print one plugin's vocabulary
+rad-schemas libr4d.so libref.so       diff them, in hierarchy order
+```
+
+The first plugin in hierarchy order to declare an op fixes its schema; a later plugin declaring the
+same op differently is refused at load, both named (arguments are positional, so disagreeing schemas
+would make one call mean different things). `rad-schemas` prints every disagreement at once, so a new
+plugin can be conformed before it is loaded beside another. It exits 1 on any disagreement. It links
+no core — it `dlopen`s plugins and reads static tables, so it works on a plugin the engine cannot
+load.
+
+## `rad-avx-check` — host libraries without a card
+
+```
+rad-avx-check --ref libref.so --avx LIB.so [options]
+
+  --ref PATH     the oracle plugin      (default libref.so)
+  --avx PATH     the plugin under test  (default libavx.so)
+  --op NAME      only this op
+  --level N      only this ISA level (0 scalar, 1 avx, 2 avx2, 3 avx512)
+  --threads N    rerun every case with each parallel region on N threads (default 4; 0 skips)
+  --tol X        override every tolerance
+  -v             every case
+```
+
+Checks a host library op by op against `libref`, on the CPU, with neither engine nor card: two
+`dlopen`s, a geometry table (`libavx/avx_cases.cpp`), libref's operand descriptions and hand-built
+cases. For every case: schemas as the loader compares them, outputs within `rad-kbench`'s tolerances,
+guard bytes, inputs unchanged, and an oracle output that is not just the draw. `libavx` is also
+checked at every ISA level the CPU executes and with parallel regions forced onto several threads.
+A new host library is checked with the same command; it is treated as a host library (not `libavx`),
+so libref ops it has no row for are listed rather than failed, and the ISA/thread sweeps run only if
+it exports `avx_isa_force`/`avx_par_force`. A row returning `RAD_E_UNSUPPORTED` for a geometry libref
+serves fails: the selector would resolve to it and the step would fail at issue.
+
+## `r4d_selftest` — libr4d's own check and timer
+
+```
+r4d_selftest <libr4d.so> [--case <row>]
+r4d_selftest <libr4d.so> --perf N K M [iters]      (and the other --perf* modes in its usage)
+```
+
+With no mode it runs the catalogue and static checks, then each row's smoke and numeric cases in a
+child process of its own, with fusion claims compared byte for byte (`ctest` labels `slow`, `gpu`).
+`--case` runs one row in-process as the reproducer. `--perf` covers the fp8 decode GEMMs and head;
+`--perfmxd` the dense OCP-MXFP4 rows with the weight bandwidth each reaches; `--perfmx` the MXFP4
+expert GEMM; `--cold MIB` times each launch alone behind a cache flush. `--perfar8` prints the
+eight-rank all-reduce shard weights and times `lockstep`/`ring`/`split` forms at every piece count;
+`R4D_SELFTEST_ONE_CARD=1` runs them on one card for their bits only.
+
+## `rad-tokcheck` — tokeniser against its references
+
+```
+rad-tokcheck [options] <tokenizer.json> <corpus-file>...
+
+  --hf-dir DIR    compare against HuggingFace ids in DIR/<corpus basename>.ids
+                  (little-endian int32, encode(text, add_special_tokens=False))
+  --no-ref        skip the reference encoder (slow)
+  --user          encode with parse_special off
+  --turns N       turns in the multi-turn measurement (default 40; 0 skips)
+  --repeat N      timing repeats, best taken (default 3)
+  --context N     ids shown either side of a difference (default 6)
+```
+
+Per corpus file: ids match `tools/tokref`'s reference encoder (llama.cpp's splitters and
+`std::regex`); ids match HuggingFace `tokenizers` read from files made elsewhere; and speed — MB/s
+for both encoders, plus the cost of re-tokenising a growing conversation with and without the
+segment cache. Any difference prints the ids and surrounding text and exits nonzero, so it gates a
+change.
+
+## `rad-chatparse` — where a tool call goes wrong
+
+```
+rad-chatparse --request REQ.json --rad MODEL.rad --reply R.txt
+rad-chatparse --render-raw T.jinja --context CTX.json
+```
+
+Renders one OpenAI chat request as the server would (container template or
+`--override-chat-template`, `--chat-template-kwargs`, the plugin's declared reply format, or the
+template's own with `--no-plugin`), then reads each reply file as the model's output: parsed whole,
+parsed through the SSE stream as a client reassembles it, and run through the request's tool-call
+grammar. It prints one JSON object; exit 1 when the two parses disagree or the grammar refuses a
+token. No weights are loaded.
+
+## The KL mode
+
+`radiance --kld-record/--kld-ref/--kld-corpus/--kld-out/--kld-lora` is a mode of the engine, not a
+separate tool: same placement and kernels, prefill instead of listening, then exit. The flags and the
+corpus format are on [[Radiance Conversion]].
+
+## Shell harnesses (`scripts/`)
+
+They drive a running engine: bring a model up, measure it, assert on what came back. Each header
+records why it measures the way it does.
+
+| script | what it does |
+|---|---|
+| `mcserve.sh`, `fnserve.sh` | bring MiniCPM5-2B+DSpark / Flash-Next up and leave it running (env knobs: `P TP MLEN SEQS M KVD HR MNBT EXTRA …`) |
+| `fnstop.sh` | stop whichever is running and wait for exit |
+| `convdspark.sh`, `convfn.sh` | rebuild the two reference containers (`convfn.sh` needs `CALIB=`) |
+| `hfget.sh` | resumable Hugging Face fetch with a content manifest |
+| `fnpf.sh`, `fnconc.sh`, `cstep2.sh`, `ident.sh`, `cardclock.py` | attach to a live server: prefill rate; prefill-vs-decode contention; steady-state decode ms/step per concurrency; run-to-run hashes; which clock limit binds and which rank the TP step waits for |
+| `fndrop.sh` | disconnect a client mid-request and check every other answer against a reference |
+| `tierident.sh`, `tiercross.sh` | whether a session that left VRAM returns byte for byte (attention and recurrent state), including two sessions in flight |
+| `mctools.sh`, `mcstream.sh`, `mcagent.sh`, `mcdecl.sh` | tool-call parsing shapes; streamed tool calls on raw SSE; speculation on an agentic workload; whether the drafter proposes |
+| `dprofd.sh`, `fntrace.sh`, `mctrace.sh`, `oracle.sh` | per-op device timing; rocprofv3 traces; every op against libref on the operands the model issued |
+| `kv-parity.sh`, `kv-stage-parity-64k.sh` | KV streaming: covering ring and staged prefill are byte-identical to streaming off |
+
+**Not all of them leave a server alone.** `fnserve.sh`, `mcserve.sh`, `fnstop.sh`, `fntrace.sh`,
+`dprofd.sh`, `mctrace.sh`, `mcagent.sh` and `mcdecl.sh` kill any running engine (`pkill` on the
+binary), whatever port it is on; `oracle.sh` stops whatever listens on its own port. `cstep2.sh`,
+`ident.sh`, `tierident.sh`, `tiercross.sh`, `mctools.sh`, `mcstream.sh`, `hfget.sh`, `convfn.sh`,
+`convdspark.sh`, `fnpf.sh`, `fnconc.sh`, `fndrop.sh` and `cardclock.py` leave it running
+(`fndrop.sh` deliberately disconnects clients). Never point the stopping ones at a production port.
+
+## Related
+
+- [[Radiance Conversion]] — `rad-convert`, `rad-export`, the KL mode.
+- [[Radiance Plugins]] — what `rad-kbench`/`rad-tune`/`rad-schemas` check.
+- [[Radiance Operations]] — the startup receipt these tools corroborate.

serverAppConfigs-docs/LLM/frank-archon.md

@@ -91,7 +91,7 @@ on stop. Boot budget 1800 s (the publisher's compose allows a 15-minute `start_p
 | Slots / window | `MAX_SEQS=4`, `MAX_MODEL_LEN=200000` (native 262,144; longer is refused, never truncated) — this pair is the **cache bound**: the resolver sizes the KV carve against `slots × window`, and no slot count funds itself (Laws) |
 | KV dtype | `KV_DTYPE=fp8` — the only dtype that funds the slot count (Laws) |
 | VRAM split | **derived at boot** from `GPU_HEADROOM_MIB=96` + the cache bound + `HOST_POOL_MIB`; naming `VRAM_WEIGHTS_MIB`/`VRAM_KV_MIB` is a boot refusal (removed at 1.3.0) |
-| Expert tier | `PLACEMENT=expert_tiered`, `HOST_POOL_MIB=49152` **per rank** — the pinned host pool is also the **KV lever**: the expert floor is `plane + 2 units − host room`, so every MiB pinned there lowers what the experts claim on the card one-for-one and the cache takes it (Laws) |
+| Expert tier | `PLACEMENT=expert_tiered`, `HOST_POOL_MIB=49152` **per rank** (raised to 66560 on 2026-10-29, folded back the same day — the A/B under *Laws* shows the raise bought no cache) — the pinned host pool is also the **KV lever**: the expert floor is `plane + 2 units − host room`, so every MiB pinned there lowers what the experts claim on the card one-for-one and the cache takes it (Laws) |
 | PLE | `NGRAM_PLACEMENT=ram` (47.7 GiB pinned; refuses boot on a short memlock) |
 | Spec / batch | `SPEC_TOKENS=2` — MTP depth 2, the `.rad` carries the model's own `mtp.*` head (`mtp.fc_hidden`/`fc_embedding`/`draft_head`); `MAX_BATCHED_TOKENS=2048` (MNBT — 2560 collapses decode, Laws) |
 | Prefix cache | `PREFIX_CACHE_MIB=8192`, pinned **host tier only** — the 64 GiB disk tier is gone, so an evicted prefix is re-prefilled; `STATE_DIR=/home/dank/radiance-state` now exists only to hold the rad-tune cache |
@@ -100,18 +100,27 @@ on stop. Boot budget 1800 s (the publisher's compose allows a 15-minute `start_p
 | TTM gate | `TTM_MIN_MIB=162000`, `TTM_TARGET_MIB=204800` |
 | Boot | `RAD_MIN_MIB=110000` deployer floor; the listener binds only after the full load (health ~100 s, budget 1800 s) |
 
-Boot ledger for the live shape (2026-10-20, pin 1.8.1, `MAX_SEQS=4`, `HOST_POOL_MIB=49152`):
+Boot ledger for the live payload. Last boot: 2026-10-29 at `MAX_SEQS=4`,
-the resolver prints `elastic 26.73 GiB` → **experts 20.72 GiB (78 %) / kv cache 5.64 GiB (21 %)**,
+`HOST_POOL_MIB=66560`, `MAX_BATCHED_TOKENS=2048`; the live pool is `49152` again, and the
-`prefill staging 373.65 MiB` out of the cache, `vram_weights 24.30 GiB`, `host_pool 48.59 GiB`.
+two boots are **identical above `host_pool`**, so one ledger serves both shapes:
-The referee line: `kv: the cache can grow to 5.64 GiB a card — 747,492 tokens of context, 3.7
+`elastic 26.73 GiB` → **experts 20.72 GiB (78 %) / kv cache 5.64 GiB (21 %)**,
-sessions at 200,000 — with every block it lends the experts taken back; 440.04 MiB of that is
+`activation arena 990.36 MiB` (the price of the 2,048-token step: `256 tokens use
-per-sequence state, and up to 5.20 GiB is lent to the experts while no request holds it`.
+155.60 MiB, 1024 tokens use 472.86 MiB of 896.20 MiB`), `prefill staging 373.65 MiB`,
-`--checkpoint-slots 40` is auto (`max_num_seqs × 10` slots of 111.98 MiB = 4,479 MiB of pinned
+`vram_weights 24.30 GiB`, `host_pool 65.59 GiB` (opened with `0 B used`; `48.59 GiB` at
-host), `/health` reports `queue_cap 32` (`max(8, seqs × 8)`), sampler **12 rows**
+the 49,152 boot), `unclaimed 96.89 MiB`; referee line `kv: the pool backs 93% of the worst
-(= `seqs × (1 + speculative depth)`), 4 hardware queues a card, template 28,234 B, vision live
+case (4 sequences all at 200000 tokens), 3.7 full-length sessions`; `device: the 2 cards
-(≤ 4,194,304 px, 16,384 patches/pass), P2P confirmed (`the 2 cards reach each other's memory
+reach each other's memory directly (peer to peer)`; sampler **12 rows**
-directly`). Experts living in the pinned host pool: **8.99 GiB a rank** (was 5.99) — the price of
+(= `seqs × (1 + speculative depth)`), 4 hardware queues a card (133.45 MiB), `already held
-the bigger cache, paid over PCIe when those experts are hit.
+256.67 MiB`, mover `17519 slab slots, 7624 pinned pool slots, 3.58 GiB static vram`,
+`load: rank 0, mover took 24.30 GiB`.
+
+The A/B that ended the raise: the 2026-10-20 boot at `HOST_POOL_MIB=49152` printed the
+**same** `elastic 26.73 / experts 20.72 / kv cache 5.64 / vram_weights 24.30` (Laws: the
+lever saturates at the expert floor), and the campaign rig — the other payload that
+inherits this env — moved `vram_kv 269.99 → 271.25 MiB`, a rounding step, because its
+window is set by its own n-gram table (95.37 GiB) and expert plane rather than by the
+pool. The refused calibration documents are fixed by cutting them to the window, not by
+widening the pool.
 
 The boot **ledger**, not this arithmetic, is the referee: radiance states every budget before
 anything is allocated (weights vs KV, the host pool's coverage of the worst case, the PLE pin,
@@ -150,6 +159,7 @@ marked "re-fetch" is gone from `/home/dank/models/`, which today holds only `Sti
 
 | Window | Payload | Runtime | Shape | Replaced because | Rollback |
 |---|---|---|---|---|---|
+| 2026-10-20 → 10-29 | same `qwen3.8-next-flash-fp8-iq4r-moe.rad` (payload unchanged) | radiance **1.8.1**, `sha256:d590e300…` (same digest) | 4 × 200,000 fp8 KV — resolver cache 5.64 GiB = 747,492 tokens, `prefill staging 373.65 MiB`, `vram_weights 24.30 GiB`, `host_pool 48.59 GiB`; host pool 49,152 MiB a rank, `TTM_MIN_MIB=162000` / `TTM_TARGET_MIB=204800`, `queue_cap 32`, sampler 12 rows, `--prefill-interleave 1` | Frank ordered the campaign's step (2,048) onto production; the pool half of that order (66,560) booted, came back byte-identical to this row's cache, and was folded back the same day — the calibration documents it was meant to save were cut to the window instead (`scripts/rechunk-kld-corpus.py`) | the live shape is this row's pool with `MAX_BATCHED_TOKENS=2048`: the pair is the rollback unit, not the merge commit (`scripts/frank-archon.env`) |
 | 2026-10-19 → 10-20 | same `qwen3.8-next-flash-fp8-iq4r-moe.rad` (payload unchanged) | radiance **1.8.1**, `sha256:d590e300…` (same digest) | 2 × 200,000 fp8 KV — resolver cache 2.76 GiB = 365,204 tokens (91 % backing); host pool 44,928 MiB a rank, `TTM_MIN_MIB=150000`, `queue_cap 16`, sampler 9 rows; `--prefill-interleave 1` armed on 10-20 | Frank ordered **four** concurrent conversations over a ≥ 700,000-token pool; the resolver funds that only by lowering the expert floor with more pinned host room (Laws) | revert `1bb6399` and redeploy `scripts/deploy.sh dankserver --service frank-archon` — trio back to `MAX_SEQS=2` / `HOST_POOL_MIB=44928` / `TTM_MIN_MIB=150000` |
 | 2026-10-15 → 10-19 | same `qwen3.8-next-flash-fp8-iq4r-moe.rad` (payload unchanged) | radiance **1.0.13**, `sha256:4c3f4845…` | 3 × 200,000 fp8 KV bought with the **explicit** pair `--vram-weights-mib 25600 --vram-kv-mib 5120`; host pool 44,928 → 45,568 MiB a rank; prefix cache 8 GiB host + 64 GiB disk | decode turned to sludge beside any long prompt — step contention, 18.7 ms decode-only steps against 317 ms steps carrying a prefill chunk (Laws) — and the only two knobs that fix it, `--busy-prefill-chunk` and `--prefill-interleave`, exist only after this pin | `FRANK_ARCHON_IMAGE` back to `sha256:4c3f4845…` **plus a trio rewrite**: restore the pair and the host+disk prefix knobs, delete `BUSY_PREFILL_CHUNK`/`PREFILL_INTERLEAVE`. A digest swap alone is a boot refusal (Runtime) |
 | 2026-10-08 → 10-15 | in-house `qwen4exp-swift15-w4dn8-i8-hc8m.rad`, 116,416 MiB (Swift1.5 abliterated BF16, GPTQ w4 experts + 24 bf16 protected, KLD 0.0131 / top-1 97.92 %) | radiance 1.0.13, same digest | 3 × 200,000 fp8 KV, slab 25,600 / pool 5,120 — the shape the live file still carries | the engine author's own container is the supported path and his quality ceiling; the finetune's refusal-bending left the endpoint | `.rad` purged 2026-10-17 (trash-staged then emptied, Frank's approval). Re-download `frank8465/Swift-1.5-Qwen3.8-Flash-Next-GPTQ-W4A8-Radiance-AMD-R9700` → `qwen4exp-w4nl64-i8-hc8m.rad`, 122,071,973,888 B, LFS sha256 `09dd9306…`, via the iq4r fetch script's hfget/verify pattern retargeted at that repo; point `MODEL_FILE` at it, redeploy. Re-**convert** is dead: the BF16 source was `rm`'d 2026-10-15 (`kld-ref`, grams, corpora and `/home/dank/logs/swift15_*.log` survive) |
@@ -243,7 +253,8 @@ vllm-radiance era (this table opens on 09-26; the era page claims 10-10).
   | 3 × 200,000, host pool 44,928 | 600,000 tok | 4.20 GiB | 93 % |
   | 3 × 200,000, host pool 45,568 | 600,000 tok | **4.20 GiB — byte-identical** | 93 % |
   | 2 × 200,000, host pool 44,928 | 400,000 tok | 2.76 GiB | 91 % |
-  | 4 × 200,000, host pool 49,152 (live) | 800,000 tok | **5.64 GiB = 747,492 tok** | 93 % |
+  | 4 × 200,000, host pool 49,152 (live) | 800,000 tok | 5.64 GiB = 747,492 tok | 93 % |
+  | 4 × 200,000, host pool 66,560 (2026-10-29 A/B, folded back) | 800,000 tok | **5.64 GiB — byte-identical to 49,152** | 93 % |
 
   The 45,568 row is the ceiling case, not a mystery: at 3 slots the cache already held its whole
   `ceiling`, so extra host room could not grow it and shows up as `unclaimed`. **The referee is the
@@ -551,6 +562,52 @@ vllm-radiance era (this table opens on 09-26; the era page claims 10-10).
   fork bomb inside the container. Kill with `podman rm -f` (the cgroup owns every generation)
   never pattern-kill an interpreter name on this host; use `TORCH_LOGS` for dynamo questions.
 
+### The quantization pipeline
+
+- **A calibration tap is only a tap when the rig that opened it is running.** The runner writes
+  `.rig-boot` into the tap dir (container, image, pid) and retires it when its boot exits non-zero;
+  `calib-push.py` refuses without that marker, refuses when it names another container, and refuses
+  when the named container is not running; `qwen38-calib-window.sh` stops on a failed phase instead
+  of falling through its health wait. All three exist because on 2026-10-29 the unguarded chain's
+  rig boot died of an OOM, the health loop expired without failing, and the pusher fed four rounds
+  of 10M tokens into **restored production** — 86 minutes of live prefill that recorded nothing.
+- **A calibration step has an OPTIMUM, and it is not the widest step that boots.** The tap's cost is
+  expert re-reads per document (a 7,168-token document cut into four 2,048-token steps re-streams
+  each layer's routed expert union four times), which argues for the widest step — but the measured
+  curve turns over: at the 32,768 calibration window one 2-way prefill of the same document ran
+  **106 tok/s at an 8,192-token step and 76 tok/s at 16,384**, with the computed activation arena
+  growing 3.39 → 4.50 GiB, so the wider step pays more compute than it saves in reads. The
+  200,000-window `QSA score plane` refusal that made the wide step look impossible is an artifact of
+  serving the production window: the same 16,384-token step needs a 512 MiB plane at 32,768.
+  `Q38_CALIB_MNBT` is the only knob — the KL modes keep production's step, because their numbers are
+  only comparable at one shape.
+- **A container that exists, a port that answers, and a `/health` 200 are three different claims —
+  bind identity before believing any of them.** Measured 2026-10-30: `podman inspect` named the rig
+  while the port still belonged to the *dying* production process (`health ok waited=0s`, window dead
+  at second zero), and the identity check written to catch that compared two different number spaces:
+  `podman top <ctr> pid` prints **container-internal** pids (`1`, `7`), never host pids, and a rootless
+  cgroup carries the container ID rather than the service name — so both "matches" can only ever fail.
+  The binding is `podman top <ctr> hpid` against the listener from
+  `ss -H -ltnp 'sport = :7770'`, and `fuser /dev/kfd` likewise resolved through podman.
+  `scripts/qwen38-preflight.sh` owns those facts now (cwd, the digest a tag actually resolves to,
+  free space on the tap's **own** filesystem, stale grams, every GPU holder and the listener by host
+  pid), prints `PASS`/`FAIL` per fact, and `qwen38-calib-window.sh` calls it before a card is touched.
+  Corollary for every ad-hoc driver: a detached launcher that loses its cwd fails with
+  `No such file or directory` in a log nobody reads — assert cwd, and never bypass the orchestrator.
+- **The n-gram table's pin is what starves a rig boot, and the tier is a knob.** On the BF16 payload
+  the table is 95.37 GiB, `--ngram-placement ram` pins it, and the calibration boot died at
+  `startup stopped: out of memory` with 230.22 GiB pinned of 246.07 GiB and 8.06 GiB available —
+  while the reference recording and every score boot succeeded at the same pool and slot count.
+  Calibration only needs the routed weights prefilled, so it reads the table from the container's
+  file tier (`--ngram-placement disk`); kldref and score keep production's placement, because their
+  numbers are only comparable at an identical shape.
+- **Acceptance is three metrics and every slice, not a mean.** `scripts/kld-gate-qwen38.sh` grades
+  mean KL, the p99.9 tail, top-1 agreement and every `by_source` slice (code, prose, tools):
+  a mean can hide a few positions taking the whole hit, and an overall win can be bought by
+  regressing one distribution. The FP8-native anchor is an interval, not a requirement — without
+  the 172.8 GiB fetch the gate runs in two-report mode (candidate vs the served payload) and says
+  so on its first line.
+
 ## Contract (the wire)
 
 - **Live surface:** `/health` (== `/ping`), `/v1/models`, `/v1/chat/completions`
@@ -627,11 +684,19 @@ vllm-radiance era (this table opens on 09-26; the era page claims 10-10).
   rebuild), `scripts/download-frank-archon-swift15.sh` (the BF16 source a w4dn8 re-convert
   needs), `scripts/hfget.sh` / `scripts/hf-fetch.sh`; template
   `scripts/qwen38-froggeric-v22.5.jinja`.
-- Convert campaign (dormant, kept for a future quant): `scripts/convert-swift15-w4dn8.sh`
+- The campaign (qwen38-w5nl64, live): `scripts/run-qwen38-bf16.sh` (kldref / calib / score — the
-  `scripts/recipes/qwen4exp-swift15-w4dn8-i8-hc8m.recipe`, `scripts/run-swift15-bf16.sh`
+  ONE measuring shape), `scripts/qwen38-calib-window.sh` (the gated calibration window),
-  `scripts/kld-gate-swift15.sh`, `scripts/kld-then-calib.sh`, `scripts/calib-push.py`
+  `scripts/convert-qwen38-w5nl64.sh` with
-  `scripts/make-calib-corpus.py`, `scripts/derive-protected-experts.py`
+  `scripts/recipes/qwen4exp-qwen38-w5nl64-bf16trunk.recipe` (the arm driver and its template),
-  `scripts/apply-protected-cast.sh`, `scripts/verify-swift15-ple.py`.
+  `scripts/kld-gate-qwen38.sh` (acceptance), `scripts/calib-push.py`,
+  `scripts/make-calib-corpus.py`, `scripts/rechunk-kld-corpus.py`,
+  `scripts/qwen38_common.py` (the tokenizer, JSONL and refusal plumbing those three
+  corpus tools share — one `n_tokens`, so every tool counts the same way),
+  `scripts/derive-protected-experts.py`, `scripts/apply-protected-cast.sh`.
+- Retired swift15-era campaign scripts, superseded by the above and kept only as reference:
+  `scripts/convert-swift15-w4dn8.sh`, `scripts/recipes/qwen4exp-swift15-w4dn8-i8-hc8m.recipe`,
+  `scripts/run-swift15-bf16.sh`, `scripts/kld-gate-swift15.sh`, `scripts/kld-then-calib.sh`,
+  `scripts/verify-swift15-ple.py`.
 - Probes: `scripts/probe-frank-archon-kv.py` (reads slots/window/dtype from the deployed env so
   it cannot drift; leg D = decode), `scripts/frank-archon-expert-activity.py`
   `scripts/autoresearch-frank-archon.sh` + `scripts/autoresearch/frank-archon-measure.py`.
@@ -641,5 +706,6 @@ vllm-radiance era (this table opens on 09-26; the era page claims 10-10).
 - Retired-era leftovers in-tree: `scripts/patches/glm53-*.patch`
   `scripts/patches/gpu_worker-kv-cap.patch` (fork archaeology; none applies byte-exact today).
 
-Related: [[frank-dragoon]] (the abliterated family), [[frank-zealot]],
+Related: [[frank-dragoon]] (the abliterated family), [[frank-zealot]], and the engine reference —
----
+[[Radiance Engine]], [[Radiance Serve Options]], [[Radiance Conversion]], [[Radiance Plugins]],
+[[Radiance Operations]].

serverAppConfigs-docs/LLM/frank-dragoon.md

@@ -1,6 +1,6 @@
 ---
 type: reference
-title: "frank-dragoon on Dankclaw — halogen-flash-server 0.16.2, native v2 abliterated Qwen3.8-Flash-Next (canonical)"
+title: "frank-dragoon on Dankclaw — halogen-flash-server 0.18.20, native v2 abliterated Qwen3.8-Flash-Next (canonical)"
 description: "The iGPU endpoint: live shape, every era with its rollback path, the laws it paid for, and the files that own it."
 tags:
   - llm
@@ -10,10 +10,10 @@ tags:
   - gfx1151
   - halogen
   - endpoint
-timestamp: 2026-10-21
+timestamp: 2026-10-30
 ---
 
-# frank-dragoon — halogen-flash-server 0.16.2, native v2 abliterated (Dankclaw iGPU)
+# frank-dragoon — halogen-flash-server 0.18.20, native v2 abliterated (Dankclaw iGPU)
 
 Strix Halo (gfx1151) endpoint `http://192.168.0.143:7770`, served id `frank-dragoon`: a podman container
 under a root `User=dank` unit. Live shape since 2026-10-17, pool raised to 401,408 on 2026-10-18
@@ -33,8 +33,8 @@ flags and nothing else. Payload `/home/dank/models/halogen-v2-abliterated/`, ver
 | `qwen38-flash-next-vision.hgn` | 897,916,416 | `d62e0ae5…` | peonist `@e053f488`, hardlinked — vision ON |
 | `tokenizer/` (6 flat files) | ~22 MB | `tokenizer.json` LFS | peonist `@e053f488`, hardlinked |
 
-Image `ghcr.io/peonist-ai/halogen-flash-server@sha256:0c61bf84ac22308a53f5d1ca6b86806702d7039e5ebc51cae4c66621b92fe04a`
+Image `ghcr.io/peonist-ai/halogen-flash-server@sha256:e35133b206da3094cc4d70c78800b26235bd9124b9aeb0e029d4a7c153b9cff0`
-(0.16.2, closed source), container `frank-dragoon-halogen`, published `7770:8731`; the engine's
+(0.18.20, closed source), container `frank-dragoon-halogen`, published `7770:8731`; the engine's
 unauthenticated token port 8730 stays on container loopback. Engine stdout →
 `/home/dank/logs/frank_dragoon.log`, **not** the journal.
 
@@ -74,6 +74,7 @@ rollback plan — recoverable only until the trash is purged.
 
 | Window | Payload | Runtime | Shape | Replaced because | Rollback |
 |---|---|---|---|---|---|
+| 2026-10-30 → | same v2 abliterated `.hgn` trio (payload untouched) | flash-server **0.18.20** `e35133b2…` | unchanged (2 slots, 262,144 ceiling, pool 401,408, MTP always-on) | vendor moved two minor versions past the 0.16.2 quickstart pin; the payload lock **held** — the same boot served chat, a `tool_calls` turn, vision ("Red", 132 prompt tokens) and `/v1/embeddings`, with the NPU lane's 19-file set verified against the new catalog | env line back to `0c61bf84…` + redeploy |
 | 2026-10-06 → 10-17 | Swift1.5 abliterated v2 `.hgn` r2 66,687,678,464 B `9c880b8b…` + same trio | flash-server 0.16.2, same digest | 2 slots, ctx 262,144, pool 524,288 (14.6 GiB), MTP always-on | — | `/home/dank/models/halogen-flashnext-v2/` + `scripts/download-frank-dragoon-flashnext.sh` |
 | 2026-10-05 → 10-06 | peonist `halogen-qwen3.8-27b` `qwen3.8-27b-p1w4d-d2.hgn` 35,865,565,184 B `274c3fc7…` | halogen 0.1.4 `sha256:dc0a39a0…` | 1 slot × 200,700, DFlash2 drafter, `HALOGEN_CACHE_MB=16384`, text-only, censored base | flash-server v2 restored vision, abliteration and MTP | `/home/dank/models/halogen-qwen3.8-27b/` + `scripts/download-frank-dragoon-halogen.sh`; 0.1.4 digest still in podman storage |
 | 2026-10-03 → 10-05 | kingjones777 Uncensored `Q6_0_ROCMFPX_AGENT.gguf` 24,856,607,008 B `e966deb3…` + `mtp-Q4_0` `051a1764…` + BF16 mmproj | `kingjones30/ROCmFPX` @ `36e9acd` (`scripts/build-rocmfpx.sh`) | 1 × 262,144 f16 KV, `draft-mtp` n-max 4, vision on | — | `/home/dank/models/kingjones-Qwen3.8-27B-Uncensored-ROCmFP4-STRIX-MTP/` + `/home/dank/GitPackages/kingjones-ROCmFPX/build` + `download-frank-dragoon-kingjones-uncensored.sh` |
@@ -190,9 +191,11 @@ created it — earlier topologies in .
 
 ## Contract (wire)
 
-- Live: `/v1/models`, `/health`, `/metrics`, POST `/v1/chat/completions`; `404`: `/ping`, `/props`
+- Live: `/v1/models`, `/health`, `/metrics`, `/props` (model_alias, total_slots, chat_template),
-  `/completion`, `/tokenize`. **No `Server-Timing` header and no `/completion` timing object** — measure
+  POST `/v1/chat/completions`, POST `/v1/embeddings` when the NPU lane is armed; `404`: `/ping`,
-  wall-clock plus `usage.prompt_tokens`.
+  `/completion`; `/tokenize` answers **405 to a GET** (the route exists and wants POST). **No
+  `Server-Timing` header and no `/completion` timing object** — measure wall-clock plus
+  `usage.prompt_tokens`.
 - Thinking lands in `reasoning_content` and **counts against `max_tokens`**: a budget that dies
   mid-thought returns `finish_reason: "length"` with empty `content`, so clients read both fields; size
   vision probes at ≥512 tokens or the model looks blind. Budget for the base reasoning long: 1,476
@@ -232,11 +235,15 @@ created it — earlier topologies in .
 XDNA2 executes only vendor-precompiled AIE partition programs — that is the wall the TTS attempt died at
 ([[Frank TTS]]) — and flash-server ships those programs. `HALOGEN_NPU_MODELS=<names>` starts a second engine **in the
 same container behind the same port**, and a request picks one by its `model`; leave the variable out and the server
-is exactly what runs today. The pinned 0.16.2 image already carries the five-program catalog
+is exactly what runs today. The pinned 0.18.20 image carries an 18-model catalog
-(`/opt/halogen/npu/models.txt`: `decider-0.8b`, `qwen3-embedding-0.6b`, `qwen3-reranker-0.6b`, `qwen3guard-gen-0.6b`,
+(`/opt/halogen/npu/models.txt`: `decider-0.8b`, `decider-4b`, `qwen3-embedding-0.6b`, `qwen3-embedding-4b`,
-`qwen3.5-2b`) and `docs/NPU.md` at tag `v0.16.2` documents it, so **`/v1/embeddings` is one env var and three mounts
+`qwen3-reranker-0.6b`, `qwen3-reranker-4b`, `qwen3guard-gen-0.6b`, `qwen3guard-stream-0.6b`,
-away — no runtime bump needed.** Wider catalog: `qwen3-embedding-4b` and `decider-4b` from 0.17.4, `embeddinggemma-2`
+`embeddinggemma-2`, `wemm-embedding-4b`, `qwen3.5-2b`, `flux2-klein-4b`, `qwen3-asr-0.6b`, `qwen3-asr-1.7b`,
-(text ≤ 8,192 tokens, pictures and audio in one vector space) and `wemm-embedding-4b` from 0.18.3.
+`d3-lite`, `d3-nano`, `qwen3-tts-0.6b-customvoice`, `qwen3-tts-1.7b-customvoice`), so **`/v1/embeddings` is one
+env var and three mounts away — no runtime bump needed.** The catalog is **not** version-stable: 0.18.20
+re-cut `qwen3-embedding-0.6b`'s device programs (14 `u*.elf` + an 8,495,512 B `consts.hnpb`, where 0.16.2
+had 10 and 5,086,496 B) and the entrypoint verifies size+sha256 per file before it starts, so **an engine
+bump re-fetches the staged set** — `scripts/download-frank-dragoon-npu.sh` carries the catalog block to do it.
 
 Embedding contract on that lane: `POST /v1/embeddings`, `model: "qwen3-embedding-0.6b"`, `input` a list of **texts**
 (not token ids), 1024 unit-length floats back, the Qwen instruct prefix goes inside the string
@@ -282,7 +289,7 @@ lane is **armed and serving** since 2026-10-19 (`HALOGEN_NPU_MODELS=qwen3-embedd
    unset, so the container opens no outbound connection; `scripts/download-frank-dragoon-npu.sh` pins every file by
    size and sha256 against the catalog and the payload mount carries the tree in.
 
-Armed, the boot ledger reads (`/home/dank/logs/frank_dragoon.log`, 2026-10-19): the GPU fabric clock is held at its top level (2000 MHz) **before** the NPU starts; `/dev/accel/accel0` is `crw-rw-rw- nobody:nogroup` under driver `amdxdna` with the distro XRT and its NPU plugin; the program arrives as `15 files checked against this image's record`; then `npu_api: qwen3-embedding-0.6b (embeddings) on the NPU engine at 127.0.0.1:8740 (buckets u0…u9; inputs up to 4096 tokens)` and `up to 64 passes queued … up to 7200 s`. Callers never see `:8740` — the lane answers on the same `:8731` as the Flash model and a request picks the engine by its `model` field, so `/v1/models` lists `frank-dragoon` **and** `qwen3-embedding-0.6b`. The fleet's consumer is open-notebook's embedding default ([[Open Notebook]]); that page owns the app-side shapes (the URL guard, defaults-by-id, the `/api/embed` 500 that still writes the vector).
+Armed, the boot ledger reads (`/home/dank/logs/frank_dragoon.log`, 2026-10-19): the GPU fabric clock is held at its top level (2000 MHz) **before** the NPU starts; `/dev/accel/accel0` is `crw-rw-rw- nobody:nogroup` under driver `amdxdna` with the distro XRT and its NPU plugin; the program arrives as `19 files checked against this image's record`; then `npu_api: qwen3-embedding-0.6b (embeddings) on the NPU engine at 127.0.0.1:8740 (buckets u0, u1, u10, u11, u12, u13, u2, u3, u4, u5, u6, u7, u8, u9; inputs up to 4096 tokens)` and `up to 64 passes queued … up to 7200 s`. Callers never see `:8740` — the lane answers on the same `:8731` as the Flash model and a request picks the engine by its `model` field, so `/v1/models` lists `frank-dragoon` **and** `qwen3-embedding-0.6b`. The fleet's consumer is open-notebook's embedding default ([[Open Notebook]]); that page owns the app-side shapes (the URL guard, defaults-by-id, the `/api/embed` 500 that still writes the vector).
 
 Price: the programs and their weights sit in host RAM beside the locked weights ("a few GB in all" for the default
 set; ~1.1 GiB for `embeddinggemma-2`), and **the Flash model decodes slower while the NPU works** — same memory bus,