docs(media): the TV library's own relayout, with the evidence chain it believes

One page owns relayout: the census commands, the order the planner believes things in, the two refusals a dry run on the real library proved necessary, the apply and rollback path, and the standing wart that a provider name can itself be scene junk ("Black Clover 001", 92 episodes) — renaming that is Jellyfin's job, not the mover's.
fix(media): a TV plan that follows the authorities it was given
2026-10-30 23:35:14 -07:00 · 2026-10-30 23:29:34 -07:00 · 2026-10-30 23:17:55 -07:00 · 2026-10-30 23:15:32 -07:00 · 2026-10-30 22:47:49 -07:00 · 2026-10-30 22:47:39 -07:00
140 changed files with 15904 additions and 776 deletions
@@ -26,8 +26,8 @@ merge absorbed the former `vllmScripts` repo, history preserved).
 | Target | Service | Runtime | Payload | Port |
 |--------|---------|---------|---------|------|
 | Dankserver | `frank-archon` | radiance, digest-pinned, root unit, TP=2 | publisher `qwen3.8-next-flash-fp8-iq4r-moe.rad` | 7770 |
-| Dankclaw | `frank-dragoon` | halogen-flash-server 0.16.2, digest-pinned, root unit | `qwen38-flash-next-v2-abliterated.hgn` + n-gram + vision tower | 7770 |
+| Dankclaw | `frank-dragoon` | halogen-flash-server 0.18.20, digest-pinned, root unit | `qwen38-flash-next-v2-abliterated.hgn` + n-gram + vision tower | 7770 |
-| Dankclaw | `frank-zealot` | llama.cpp PrismML fork build 10777, root unit — the only loader for this payload | Ternary-Bonsai-2-9B PQ2_0 + Q8_0 mmproj + rank-1 refusal adapter | 7771 |
+| Dankclaw | `frank-zealot` | llama.cpp PrismML fork build 12118, root unit — the only loader for this payload | Ternary-Bonsai-2-9B PQ2_0 + Q8_0 mmproj + rank-1 refusal adapter | 7771 |
 Payload identity, shape, eras, laws and rollback live in ONE page each:
 [[frank-archon]], [[frank-dragoon]], [[frank-zealot]]. `scripts/<service>.env`
@@ -176,11 +176,12 @@ alertmanager.container`, loopback :9093) → host-side Signal relay
 (`scripts/signal-relay.py`, user unit, 192.168.0.143:18093 — the host LAN
 IP, the one bind a container can reach as `host.containers.internal`;
 iptables admits loopback, the host and the podman bridge subnet only) → signal-cli account 318311.
-Prometheus targets read down for structural reasons (HTML-scraped "probes",
+The former structural target downs (HTML-scraped "probes", Gitea metrics off)
-Gitea metrics off); the remedy is a blackbox exporter. `frank-zealot` was down for
+are retired as of 2026-10-21: HTTP liveness rides a blackbox exporter
-a different reason until 2026-10-18: llama.cpp will not serve `/metrics` at all
+(`shared/blackbox/blackbox.yml`), Gitea serves `/metrics` from its quadlet, and
-without `--metrics`. See
+every Prometheus job reports up. `frank-zealot` was down for a different reason
-`serverAppConfigs-docs/Infrastructure/Observability.md`.
+until 2026-10-18: llama.cpp will not serve `/metrics` at all without
 `--metrics`. See `serverAppConfigs-docs/Infrastructure/Observability.md`.
 ## Testing
@@ -292,8 +293,8 @@ means off or NIC dead; a merely wedged kernel usually still answers ARP.
 2. Bench against the service itself. No scratch ports on an LLM host — one
   process owns the card. Use the runtime's own harness (`llama-bench`; a
   live llama.cpp server also reports honest `timings` per request, including
-   `draft_n`/`draft_n_accepted` — but on frank-zealot's PrismML fork build
+   `draft_n`/`draft_n_accepted` — but on frank-zealot's PrismML fork that object
-   10777 that object only reaches you on **`/v1/chat/completions`**: a greedy
+   only reaches you on **`/v1/chat/completions`** (measured on build 10777): a greedy
   `ignore_eos` run on the raw `/completion` path there answers **500 "The
   model produced output that does not match the expected Content-only
   format"** (measured 2026-10-18). halogen has no `/completion` path at all,
@@ -419,16 +420,17 @@ disagrees with it is the bug.
  digest-pinned `ghcr.io/peonist-ai/halogen-flash-server@sha256:0c61bf84…`.
  Engine stdout (the boot ledger: weights, KV pool, clamp) goes to
  `/home/dank/logs/frank_dragoon.log`, NOT the journal.
- Shape: `HALOGEN_CTX=200704` is the **per-request ceiling** (a longer prompt
+- Shape: `HALOGEN_CTX=262144` (the checkpoint's native window) is the
-  is a 400, never a truncation); `HALOGEN_KV_SLOTS=2` over ONE
+  **per-request ceiling** (a longer prompt is a 400, never a truncation);
-  `HALOGEN_KV_POOL_POSITIONS=401408` pool — pool positions buy concurrency,
+  `HALOGEN_KV_SLOTS=2` over ONE `HALOGEN_KV_POOL_POSITIONS=401408` pool —
-  the per-request window never moves; `HALOGEN_SPEC_ADAPT=0` keeps the MTP
+  pool positions buy concurrency, the per-request window never moves;
-  drafter always armed (the adaptive default disarms it and halves decode);
+  `HALOGEN_SPEC_ADAPT=0` keeps the MTP drafter always armed (the adaptive
-  server sampler 1.0/0.95/20.
+  default disarms it and halves decode); server sampler 1.0/0.95/20.
- Harness contract: `models.yml` `contextWindow: 155648` = ceiling − 45,056.
+- Harness contract: `models.yml` `contextWindow: 200000` — ~45k+ under the
-  The clamp is `min(45056, window − est − 4096)` and the chars/4 estimate
+  ceiling. The clamp is `min(45056, window − est − 4096)` and the chars/4
-  under-counts, so a harness window equal to the ceiling makes every long
+  estimate under-counts (trigger is 95 %), so a harness window at the ceiling
-  request a guaranteed 400. Reasoning arrives in `reasoning_content`; the
+  makes every long request a guaranteed 400.
  Reasoning arrives in `reasoning_content`; the
  template rejects `reasoning_effort` outside `xhigh|medium|low`; images are
  OpenAI `image_url` parts (probe ≥256 px); `response_format` unsupported;
  `/completion`, `/ping`, `/props`, `/tokenize` are 404.
@@ -437,16 +439,16 @@ disagrees with it is the bug.
 ### frank-zealot — llama.cpp (PrismML fork), RTX 3090, :7771
 - Owns: `scripts/frank-zealot.env`, `scripts/serve-frank-zealot.sh`,
-  `scripts/frank-zealot.service`; fork build `llama-prism-b10777` is the ONLY
+  `scripts/frank-zealot.service`; fork build `llama-prism-0244744` (build 12118)
-  loader (the payload carries the fork's custom PQ2_0 tensor type plus
+  is the ONLY loader — `llama-prism-b10777` stays on disk as its rollback (the payload carries the fork's custom PQ2_0 tensor type plus
  `prism.hadamard.*` dequant metadata — stock llama.cpp refuses it, and a
  loader ignoring that metadata dequantizes wrong); payload
  `Bonsai2-9B-MTP/…PQ2_0.gguf` + the publisher's **Q8_0** mmproj (the BF16
  tower can crash the server with concurrent image requests) + the rank-1
  refusal-direction adapter applied at load by `--lora-scaled …:1`. The
  adapter's rollback is `LORA_FILE=` (empty) in the env — one line, no rebuild
-  — and build 10777 is also the gate that lets it load: older forks refuse it
+  — and the fork BUILD is also the gate that lets it load: anything older than
-  with `unsupported prism.hadamard.version: 2`.
+  b10777 refuses it with `unsupported prism.hadamard.version: 2`.
 - Shape: `--ctx-size 602112 --parallel 3` = **three conversations at
  200,704 each**, `CACHE_TYPE_K=f16` + `CACHE_TYPE_V=q8_0` (asymmetric: 25,088
  B/token, 14,406 MiB of pool), batch 6144 / ubatch 1024,
@@ -739,6 +741,13 @@ linked runbooks.
  `/etc/default/grub`. It replaced `amd_iommu=off`, and the local A/B on byte-identical traffic
  says identity mode costs nothing, so the XDNA2 NPU returned for free.
  [[dankclaw-iommu-translation-off]]
 - **A deploy handler that only `start`s a quadlet no-ops an image bump.**
  `systemctl --user start` on an already-active unit does nothing, so installing
  a new digest under a running service keeps the OLD container serving while the
  deploy, its health gate and the Playwright suite all pass — the authentik
  handler did exactly that on 2026-10-30 (2026.8.3 → 2026.8.5 moved only the
  outpost, the one handler that restarted). A pin change is applied by
  `restart`, never `start`.
 - **Rootless Quadlet containers run with `--replace --rm`**: `podman logs` covers only the CURRENT
  instance; history across recreations lives in `journalctl --user -u <name>.service`. Read
  rootless storage WITHOUT sudo — it is per-user storage, so `sudo podman logs` finds nothing.
@@ -798,6 +807,8 @@ per-attempt page and no per-swap ledger anywhere else.
 | `frank-tts` — cloned-voice speech, Dankclaw | `serverAppConfigs-docs/Dankclaw/Frank TTS.md` |
 | Dankclaw kernel posture (`iommu=pt` since 2026-10-18) | `serverAppConfigs-docs/LLM/dankclaw-iommu-translation-off.md` |
 | Deployment flow | `serverAppConfigs-docs/Architecture/Deployment Flow.md` |
 | Movie-library relayout (Jellyfin format, Dankserver) | `serverAppConfigs-docs/Dankserver/Media Relayout.md` |
 | SSD-backed metadata view (Jellyfin writes, Plex reads, Dankserver) | `serverAppConfigs-docs/Dankserver/Media Metadata View.md` |
 There is no page for a retired endpoint and no change diary. A payload swap
 appends one row to the live endpoint's **Eras** table; when the endpoint itself
@@ -839,5 +850,7 @@ history.
 | Writing a doc | **One owner per fact** — edit the endpoint/service page; everywhere else links. Never restate a number. |
 | Benchmarking (only when ordered) | **Stop and mask the service first**, bench, unmask, restart, verify health. |
 | Deleting a file | **Move it to trash** — `.AGENTTRASH/` (in-repo) or `~/.AGENTTRASH/YYYYMMDD-topic`. Never `rm`. |
 | Relayout the movie library | **Census → plan → read the exceptions → `scripts/media-apply.py`** — renames only, journal before every move; never hand-rename, never copy. |
 | Making media metadata visible to both media servers | **Bind `/var/lib/media-meta/view`** — the overlay `scripts/media-view.sh` installs (array below, NVMe above). Never write the array; never delete through the view. |
 | Tests exist | **Run them** — verify tests pass after every change. Do not commit failing tests. |
 | Real secrets | **Never commit** — host `.env` files; `.env.example` holds placeholders only. |
@@ -23,7 +23,7 @@ Network=dankclaw_default
 # Image=ghcr.io/goauthentik/ldap:2026.8.3 (amd64 digest; podman 5.7.0 refuses
 # the tag's nested manifest list, so pin the single-platform digest).
 EnvironmentFile=/home/dank/serverAppConfigs/dankclaw/.env
-Image=ghcr.io/goauthentik/ldap@sha256:2e7b259eb8c37faac916946ddf48ec681ee9d3a9f724fe8521bf161973f13cf9
+Image=ghcr.io/goauthentik/ldap:2026.8.5@sha256:4c75bd27918418d2d6fbbfe4581bbd5a199fe38942d24559c4483edd64a1ea87
 Environment=AUTHENTIK_HOST=https://auth.franksdank.cloud
 Environment=AUTHENTIK_INSECURE=false
 Environment=AUTHENTIK_TOKEN=${AUTHENTIK_LDAP_OUTPOST_TOKEN}
@@ -20,7 +20,7 @@ Wants=authentik-server.service
 [Container]
 ContainerName=authentik-outpost
-Image=ghcr.io/goauthentik/server:2026.8.3
+Image=ghcr.io/goauthentik/server:2026.8.5@sha256:602227a86455d6772e3d16b7d852713c4f864f40df5df894abd1285bb53979a5
 Exec=worker
 Network=dankclaw_default
 EnvironmentFile=/home/dank/serverAppConfigs/dankclaw/.env
@@ -14,7 +14,7 @@ After=network.target
 [Container]
 ContainerName=authentik-postgres
-Image=docker.io/library/postgres:16-alpine
+Image=docker.io/library/postgres:16-alpine@sha256:0cefaabd882e42bcb2a2f3dc56d3ce4770560ba62fd7f197f1a2e008fd5731ab
 Network=dankclaw_default
 EnvironmentFile=/home/dank/serverAppConfigs/dankclaw/.env
 Environment=POSTGRES_DB=authentik
@@ -19,7 +19,7 @@ Wants=authentik-postgres.service
 [Container]
 ContainerName=authentik-server
-Image=ghcr.io/goauthentik/server:2026.8.3
+Image=ghcr.io/goauthentik/server:2026.8.5@sha256:602227a86455d6772e3d16b7d852713c4f864f40df5df894abd1285bb53979a5
 Exec=server
 Network=dankclaw_default
 EnvironmentFile=/home/dank/serverAppConfigs/dankclaw/.env
@@ -16,7 +16,7 @@ Wants=authentik-server.service
 [Container]
 ContainerName=authentik-worker
-Image=ghcr.io/goauthentik/server:2026.8.3
+Image=ghcr.io/goauthentik/server:2026.8.5@sha256:602227a86455d6772e3d16b7d852713c4f864f40df5df894abd1285bb53979a5
 Exec=worker
 Network=dankclaw_default
 EnvironmentFile=/home/dank/serverAppConfigs/dankclaw/.env
@@ -15,7 +15,7 @@ After=network.target
 [Container]
 ContainerName=gitea-postgres
 Network=gitea.network
-Image=docker.io/library/postgres:16-alpine
+Image=docker.io/library/postgres:16-alpine@sha256:0cefaabd882e42bcb2a2f3dc56d3ce4770560ba62fd7f197f1a2e008fd5731ab
 EnvironmentFile=/home/dank/Containers/gitea/env
 Environment=POSTGRES_USER=gitea
 Environment=POSTGRES_PASSWORD=${GITEA_DB_PASSWORD}
@@ -19,7 +19,7 @@ After=gitea-postgres.service network.target
 [Container]
 ContainerName=gitea-server
 Network=gitea.network
-Image=docker.io/gitea/gitea:latest
+Image=docker.io/gitea/gitea:latest@sha256:30777c91280d973fa02be48b1ade735bd6859e8f6f7af3b4444d519437852504
 EnvironmentFile=/home/dank/Containers/gitea/env
 Environment=DATABASE_TYPE=postgres
 Environment=DATABASE_HOST=gitea-postgres
@@ -29,6 +29,13 @@ Environment=DATABASE_PASSWD=${GITEA_DB_PASSWORD}
 Environment=DATABASE_PORT=5432
 Environment=USER_UID=1000
 Environment=USER_GID=1000
 # Prometheus scrapes /metrics over the published LAN port (job "gitea" in
 # shared/prometheus/prometheus.yml). Off, /metrics is a 404 and the target is
 # permanently red (the audit's "disabled metrics" class). 0.0.0.0 binds inside
 # the container; the only route in is this host's published 3000, and the
 # exposed surface is counters only — same posture as the LLM /metrics.
 Environment=GITEA__metrics__ENABLED=true
 Environment=GITEA__metrics__ENABLED_HOST=0.0.0.0
 PublishPort=0.0.0.0:3000:3000
 PublishPort=2222:22
 Volume=/home/dank/Containers/gitea/data:/data
@@ -25,7 +25,7 @@ Description=Chrome DevTools Protocol daemon (headless-shell) for hermes
 [Container]
 ContainerName=chrome-devtools-protocol
 Network=host
-Image=docker.io/chromedp/headless-shell@sha256:2d349b544a1ea6b5b5fd7c0fe99215ff662339c57407ee2e8c0a11af93516b04
+Image=docker.io/chromedp/headless-shell@sha256:7bee201478b2f6266aae01243740ee85c16281e32f42d3aed9fdf04f4eace7a1
 Entrypoint=/headless-shell/headless-shell
 Exec=--no-sandbox --disable-gpu --use-gl=angle --use-angle=swiftshader --remote-debugging-address=127.0.0.1 --remote-debugging-port=9222
 ShmSize=1073741824
@@ -23,7 +23,7 @@ After=gluetun.service network.target
 [Container]
 ContainerName=marreta_browser
 Network=gluetun.container
-Image=docker.io/lightpanda/browser:nightly
+Image=docker.io/lightpanda/browser:nightly@sha256:e72730e364d1fbf352c1149f29ffa4b278f3b1a55ff0dec3262298ebe624ec7a
 [Service]
 TimeoutStartSec=300
@@ -32,7 +32,7 @@ After=gluetun.service marreta-browser.service network.target
 [Container]
 ContainerName=marreta
 Network=gluetun.container
-Image=ghcr.io/manualdousuario/marreta:latest
+Image=ghcr.io/manualdousuario/marreta:latest@sha256:30f2a34567b79b66da794b4b9423751bbbcd07ceff600d9f6a2d219e7e439362
 # Relocate off SearXNG's 8080 inside the shared netns.
 Environment=NGINX_HTTP_PORT=8085
 Environment=APP_NAME=Marreta
@@ -33,7 +33,7 @@ Wants=network-online.target
 [Container]
 ContainerName=alertmanager
-Image=docker.io/prom/alertmanager:latest
+Image=docker.io/prom/alertmanager:latest@sha256:39d8bb3966c694a124428a3a88fbcffd40bd069129ecd47c059f8a3f17c55f3b
 Network=dankclaw_default
 User=0:0
 Exec=--config.file=/etc/alertmanager/alertmanager.yml --storage.path=/alertmanager
@@ -0,0 +1,56 @@
 # Quadlet unit: blackbox exporter (rootless, host network, boot-persistent).
 #
 # Install path on dankclaw:
 #   ~/.config/containers/systemd/blackbox-exporter.container
 #
 # WHY Network=host. The probes it runs are (a) the host's own LAN IP
 # (192.168.0.143:8081 SearXNG, :3005 OpenWebUI — host-process/portmap binds a
 # bridge container cannot reach; netavark DNATs published ports only, see
 # Container Host Reachability in the vault) and (b) ComfyUI's published port
 # via 127.0.0.1 (rootless --network=host shares the real host netns, so the
 # 0.0.0.0:8188 publish answers on loopback). A bridge-networked prober could
 # reach neither class reliably.
 #
 # WHY the listener binds 127.0.0.1 + LAN. A bridge container CANNOT dial
 # its own host's LAN IP (netavark hairpin gap, measured 2026-10-21), but a
 # bridge container dialing the podman gateway 169.254.1.2 lands on the host
 # LOOPBACK — that is the scrape path, and it needs this exporter to be
 # LISTENING ON LOOPBACK (proved: 200 from the prometheus container to
 # http://169.254.1.2:9115/-/healthy with a loopback bind). The gateway
 # address itself is NOT bindable from the host netns — it exists only
 # inside each rootless netns; binding it dies "bind: cannot assign
 # requested address" (live 2026-10-21). The LAN bind stays for diagnostics.
 # No public DNS name, no guard chain: internal net only, per the exposure
 # policy. The listen flag is --web.listen-address and it is REPEATABLE, one
 # flag per bind; plain --listen-address does not exist and the exporter
 # exits in 100 ms with "unknown long flag" (live 2026-10-21).
 [Unit]
 Description=Blackbox exporter (HTTP probe prober)
 After=network-online.target
 [Container]
 Image=docker.io/prom/blackbox-exporter@sha256:ea1c7aa3ef3f307cd03e85f34abb604478ccb90b567fc690da0cd9776da76b13
 ContainerName=blackbox-exporter
 Network=host
 # The image's ENTRYPOINT is /bin/blackbox_exporter itself (proved live
 # 2026-10-21: `podman run IMAGE sh` dies "unexpected sh"), so Exec= carries
 # FLAGS ONLY — repeating the binary makes it a positional arg and kingpin
 # exits "unexpected /bin/blackbox_exporter".
 Exec=--config.file=/etc/blackbox_exporter/config.yml --web.listen-address=127.0.0.1:9115 --web.listen-address=192.168.0.143:9115
 Volume=/opt/blackbox/blackbox.yml:/etc/blackbox_exporter/config.yml:ro
 # Plain exec form like every sibling unit. Shell form (`/bin/sh -c "…"`)
 # is passed as one quoted token and quadlet then folds the following
 # --health-interval/--health-retries (and the image name) into the command
 # string — unit died twice that way, live 2026-10-21. wget only; this
 # busybox variant has no curl, and host-network wget answers on loopback.
 HealthCmd=wget -q -O /dev/null http://127.0.0.1:9115/-/healthy
 HealthInterval=30s
 HealthRetries=3
 [Service]
 Restart=always
 RestartSec=5
 [Install]
 WantedBy=default.target
@@ -35,7 +35,7 @@ Wants=network-online.target
 [Container]
 ContainerName=cadvisor
-Image=gcr.io/cadvisor/cadvisor:v0.49.1
+Image=gcr.io/cadvisor/cadvisor:v0.49.1@sha256:3cde6faf0791ebf7b41d6f8ae7145466fed712ea6f252c935294d2608b1af388
 PodmanArgs=--pid=host --cgroupns=host
 PublishPort=127.0.0.1:8080:8080
 PublishPort=192.168.0.143:8080:8080
@@ -34,7 +34,7 @@ Wants=network-online.target prometheus.service
 [Container]
 ContainerName=grafana
-Image=docker.io/grafana/grafana:latest
+Image=docker.io/grafana/grafana:latest@sha256:488984b716c0862639f91a698adbe2862677a5a07bad42da5f03f3030883b205
 Network=dankclaw_default
 User=0:0
 PublishPort=127.0.0.1:3002:3000
@@ -33,7 +33,7 @@ Wants=network-online.target
 [Container]
 ContainerName=node-exporter
-Image=docker.io/prom/node-exporter:latest
+Image=docker.io/prom/node-exporter:latest@sha256:1b4e4438faca4dd7e001dd445d161a4a2091b0fededa84093b3a8dfeae1f1be0
 Network=host
 PodmanArgs=--pid=host
 Exec=--path.procfs=/host/proc --path.sysfs=/host/sys --path.rootfs=/rootfs --web.listen-address=127.0.0.1:9100 --web.listen-address=192.168.0.143:9100
@@ -43,8 +43,16 @@ Wants=network-online.target
 [Container]
 ContainerName=prometheus
-Image=docker.io/prom/prometheus:latest
+Image=docker.io/prom/prometheus:latest@sha256:fa3302279710be2f21a8301ec17a8b4f2301fd829bd113839b3b6b02b6f191a2
 Network=dankclaw_default
 # Same-host scrape targets are dialed at host.containers.internal (the podman
 # gateway): a bridge container cannot dial its own host's LAN IP (netavark
 # hairpin gap, measured 2026-10-21), and traffic to the gateway lands on host
 # loopback / published ports (cadvisor/gpu jobs and the blackbox scrape all
 # ride it). The name resolves natively — NO HostAddr key: quadlet 5.7.0 calls
 # it unsupported and the WHOLE user quadlet batch then fails to generate,
 # silently orphaning every other unit (live 2026-10-21: prometheus.service
 # vanished while its container kept running as an orphan).
 User=0:0
 Exec=--config.file=/etc/prometheus/prometheus.yml --storage.tsdb.path=/prometheus --storage.tsdb.retention.time=30d --web.enable-lifecycle
 PublishPort=127.0.0.1:9090:9090
@@ -27,7 +27,7 @@ After=network.target
 [Container]
 ContainerName=open-webui
 Network=open-webui.network
-Image=ghcr.io/open-webui/open-webui:main
+Image=ghcr.io/open-webui/open-webui:main@sha256:0b399a8ba34558e181d316750455f2a95376a38cd60a61b9c942e138898944bd
 PublishPort=3005:8080
 Environment=OPENAI_API_BASE_URL=http://192.168.0.143:7770
 Environment=ANONYMIZED_TELEMETRY=False
@@ -24,7 +24,7 @@ Wants=network-online.target
 [Container]
 ContainerName=pihole
-Image=docker.io/pihole/pihole:latest
+Image=docker.io/pihole/pihole:latest@sha256:d34134716fc32038b052a8674c7c6879e0ff21f0e6d8af345166fffa2c27c113
 Network=slirp4netns
 PublishPort=0.0.0.0:53:53/udp
 PublishPort=0.0.0.0:53:53/tcp
@@ -16,7 +16,7 @@ After=searxng-network.service network.target
 [Container]
 ContainerName=gluetun
 Network=searxng.network
-Image=docker.io/qmcgaw/gluetun:latest
+Image=docker.io/qmcgaw/gluetun:latest@sha256:4af0c804003481e100180e1f8d4d5e9c48163fedc0f88d061881b615aef9d183
 AddCapability=NET_ADMIN
 AddDevice=/dev/net/tun:/dev/net/tun
 Environment=VPN_SERVICE_PROVIDER=${VPN_SERVICE_PROVIDER}
@@ -16,7 +16,7 @@ After=searxng-network.service network.target
 ContainerName=searxng-valkey
 Network=searxng.network
 NetworkAlias=valkey
-Image=docker.io/valkey/valkey:9-alpine
+Image=docker.io/valkey/valkey:9-alpine@sha256:bf09d58e96d6b4524a15396b65b242bbe18beebbcc15a04d8ee75ddac62fae0b
 Volume=/opt/searxng/valkey:/data
 [Service]
@@ -16,7 +16,7 @@ After=gluetun.service searxng-valkey.service network.target
 [Container]
 ContainerName=searxng
 Network=gluetun.container
-Image=ghcr.io/searxng/searxng:latest
+Image=ghcr.io/searxng/searxng:latest@sha256:f4aa660c54d062dedb268be84e821ea24e39f48c7a496af8dd02cc9eedb93847
 Environment=SEARXNG_SECRET=${SEARXNG_SECRET}
 Environment=SEARXNG_BASE_URL=${SEARXNG_BASE_URL}
 Volume=/opt/searxng/config:/etc/searxng
@@ -29,7 +29,7 @@ Wants=network-online.target
 [Container]
 ContainerName=seerr
-Image=ghcr.io/seerr-team/seerr:v3.4.1
+Image=ghcr.io/seerr-team/seerr:v3.5.1@sha256:d8fab675b55ad293ded98fd71efcfd46054cbad9746534ae1cc4794a88c0b3be
 Network=dankclaw_default
 PublishPort=127.0.0.1:5055:5055
 Volume=/home/dank/Containers/seerr/data:/app/config
@@ -43,7 +43,7 @@ After=network.target
 [Container]
 ContainerName=calibre-server
-Image=lscr.io/linuxserver/calibre:9.15.0
+Image=lscr.io/linuxserver/calibre:9.17.0@sha256:b198fc03a6abc6be7e251a9958e50e74634b00e447c7b25e92c191c8077ba378
 Network=pasta
 EnvironmentFile=/home/dank/Containers/calibre-server/env
 Environment=PUID=1000
@@ -16,7 +16,7 @@ After=network.target
 [Container]
 ContainerName=calibre-web
-Image=lscr.io/linuxserver/calibre-web:0.6.27-ls402
+Image=lscr.io/linuxserver/calibre-web:0.6.27-ls407@sha256:470b4ed9a7d6ea1475fef17dbcf957cb60880910296d226bd0dca652c05f68ca
 Network=pasta
 Environment=DOCKER_MODS=linuxserver/mods:universal-calibre
 Environment=LANG=en_US.UTF-8
@@ -94,34 +94,12 @@ services:
      timeout: 10s
      retries: 3
-  # ═══════════════════════════════════════════════
+  # Jellyfin left this file when it became a Quadlet: the container is owned by
-  # Jellyfin — media server (movies, TV, music)
+  # dankserver/jellyfin/quadlet/jellyfin.container, its data roots are under
-  # ═══════════════════════════════════════════════
+  # /home/dank/Containers/jellyfin, and its library arrives through the SSD
-  jellyfin:
+  # overlay at /var/lib/media-meta/view (scripts/media-view.sh). The block that
-    image: jellyfin/jellyfin:latest
+  # was here named /opt/jellyfin/* and a read-only array bind — a second owner for
-    container_name: jellyfin
+  # the container name `jellyfin`, wrong on every path. One owner per container.
    restart: unless-stopped
    ports:
      - "8096:8096/tcp"
      - "8920:8920/tcp"
      - "7359:7359/udp"
      - "1900:1900/udp"
    environment:
      - JELLYFIN_PublishedServerUrl=http://192.168.0.235:8096
      - DOTNET_SYSTEM_IO_DISABLEFILELOCKING=true
    volumes:
      - /opt/jellyfin/config:/config
      - /opt/jellyfin/cache:/cache
      - /mnt/entropy:/data:ro
    devices:
      - /dev/dri:/dev/dri
    group_add:
      - "992"
    healthcheck:
      test: ["CMD", "curl", "-f", "http://localhost:8096/health"]
      interval: 30s
      timeout: 10s
      retries: 3
  # ═══════════════════════════════════════════════
  # Calibre-Web — e-book web frontend
@@ -21,7 +21,13 @@ After=network.target
 [Container]
 ContainerName=copyparty
-Image=ghcr.io/9001/copyparty-ac:latest
+# Digest-pinned: upstream stopped publishing version tags after 1.19.15, so
 # the pin rides :latest@digest (the tag in the reference is cosmetic; the
 # digest is the pull). Pinned 2026-10-21 at the build whose label is
 # 1.20.26 - closes GHSA-3c88-p299-7hvr (1.20.25) and GHSA-hg99-jqfp-rf76
 # (1.20.26); the previously running build was the rolling :latest at
 # 1.20.24 (issue #73 on command-center).
 Image=ghcr.io/9001/copyparty-ac:latest@sha256:7f77459160a4bf34fdca79f2965dff4f0c1d7069a3d9cdfb3632acb041dd2f6a
 Network=dankserver_appnet
 EnvironmentFile=/home/dank/Containers/copyparty/env
 Exec=-a ${COPAY_ACCOUNT}:${COPAY_PASSWORD} -v /w::r:ra,dank
@@ -0,0 +1,30 @@
 # /etc/systemd/system/var-lib-media\x2dmeta-view.mount
 # (repo copy: dankserver/host/systemd/media-meta-view.mount — systemd names a
 # mount unit after its ESCAPED path, so the host file carries the \x2d spelling)
 #
 # Installed by `scripts/media-view.sh install` — the repo copy is the source of
 # truth (fleet law: never edit host state by hand).
 #
 # An OverlayFS merge that shows the media library as one tree while splitting
 # where its bytes live: lowerdir is the RAID6 array (the source of truth, never
 # written), upperdir is a directory on the NVMe that receives every write. The
 # two media servers bind the result at /var/lib/media-meta/view, so Jellyfin's
 # sidecars and artwork are SSD bytes — which is what makes them fast to browse
 # and what keeps Plex's NFO agent off the spinners — while the media itself stays
 # on the array untouched.
 #
 # RequiresMountsFor is load-bearing: without the array mounted, lowerdir is an
 # empty directory and the view would present an empty library as truth.
 [Unit]
 Description=SSD-backed overlay view of the media library (array + NVMe metadata layer)
 RequiresMountsFor=/mnt/entropy
 [Mount]
 What=overlay
 Where=/var/lib/media-meta/view
 Type=overlay
 Options=lowerdir=/mnt/entropy,upperdir=/var/lib/media-meta/upper,workdir=/var/lib/media-meta/work
 [Install]
 WantedBy=local-fs.target
@@ -4,8 +4,9 @@
 # Unit name: jellyfin.service (podman 5.x drops the container- prefix)
 #
 # Data roots migrated from /opt/jellyfin to /home/dank/Containers/jellyfin at
-# cutover. All of /mnt/entropy is mounted read-only at /data: nothing automated
+# cutover. The library is seen through /var/lib/media-meta/view, the SSD-backed
-# anywhere writes to the entropy volume. Every
+# overlay view (scripts/media-view.sh): the array stays read-only and untouched,
 # while the sidecars this server writes land on the NVMe. Every
 # creation-time environment variable of the compose-era container is kept as
 # an Environment= line (zero drift); none of them are secrets, so this unit
 # needs no EnvironmentFile.
@@ -16,7 +17,7 @@ After=network.target
 [Container]
 ContainerName=jellyfin
-Image=ghcr.io/jellyfin/jellyfin:latest
+Image=ghcr.io/jellyfin/jellyfin:latest@sha256:fea863d0cd9ec445985d9c75fc6ed64e38a1d2766cf0c250a33d553f4fc1b771
 Network=dankserver_appnet
 Environment=DEBIAN_FRONTEND=noninteractive
 Environment=DOTNET_SYSTEM_IO_DISABLEFILELOCKING=true
@@ -41,7 +42,12 @@ PublishPort=7359:7359/udp
 PublishPort=1900:1900/udp
 Volume=/home/dank/Containers/jellyfin/config:/config
 Volume=/home/dank/Containers/jellyfin/cache:/cache
-Volume=/mnt/entropy:/data:ro
+# The view, writable: Jellyfin is the metadata writer (movie.nfo / tvshow.nfo /
 # episode.nfo + artwork) and those bytes belong on the SSD. The condition is the
 # fail-closed guard — starting against a plain, unmounted view directory would let
 # a scan conclude the library is empty.
 Volume=/var/lib/media-meta/view:/data
 UnitConfigItem=ConditionPathIsMountPoint=/var/lib/media-meta/view
 AddDevice=/dev/dri
 GroupAdd=keep-groups
 HealthCmd=curl -f http://localhost:8096/health
@@ -18,7 +18,7 @@ Wants=network-online.target
 [Container]
 ContainerName=cadvisor
-Image=gcr.io/cadvisor/cadvisor:v0.49.1
+Image=gcr.io/cadvisor/cadvisor:v0.49.1@sha256:3cde6faf0791ebf7b41d6f8ae7145466fed712ea6f252c935294d2608b1af388
 PodmanArgs=--pid=host --cgroupns=host
 PublishPort=127.0.0.1:8080:8080
 PublishPort=192.168.0.235:8080:8080
@@ -37,7 +37,7 @@ Wants=network-online.target
 [Container]
 ContainerName=node-exporter
-Image=docker.io/prom/node-exporter:latest
+Image=docker.io/prom/node-exporter:latest@sha256:1b4e4438faca4dd7e001dd445d161a4a2091b0fededa84093b3a8dfeae1f1be0
 Network=host
 PodmanArgs=--pid=host
 Exec=--path.procfs=/host/proc --path.sysfs=/host/sys --path.rootfs=/rootfs --web.listen-address=127.0.0.1:9100 --web.listen-address=192.168.0.235:9100
@@ -4,14 +4,19 @@ After=network-online.target
 Wants=network-online.target
 [Container]
-Image=lscr.io/linuxserver/plex:1.43.4.10903-e5521bd8c-ls325
+Image=lscr.io/linuxserver/plex:1.43.5.11079-b3b13e506-ls330@sha256:01a53d9c7adcbb3701fbb0a4230c0a37c85453759b6b05a67356c3c7a09de08d
 ContainerName=plex
 Network=dankserver_appnet:alias=plex
 Environment=PUID=0
 Environment=PGID=0
 Environment=TZ=America/Los_Angeles
 Volume=/var/lib/plexmediaserver:/config
-Volume=/mnt/entropy:/mnt/entropy:ro
+# The container path stays /mnt/entropy on purpose: every path Plex recorded in
 # its own database is written against it, so swapping the HOST source of this bind
 # to the overlay view re-points Plex at the sidecars without touching its DB.
 # Read-only: Plex consumes the NFO files Jellyfin writes and writes no metadata.
 Volume=/var/lib/media-meta/view:/mnt/entropy:ro
 UnitConfigItem=ConditionPathIsMountPoint=/var/lib/media-meta/view
 PublishPort=32400:32400
 [Service]
@@ -14,7 +14,7 @@ After=network.target
 [Container]
 ContainerName=romm-mariadb
-Image=docker.io/library/mariadb:11
+Image=docker.io/library/mariadb:11@sha256:c4124d1afa875e47f2892edf26b482fa54f796e2f342d3717a04ca091a22a9bd
 Network=dankserver_appnet:alias=mariadb
 EnvironmentFile=/home/dank/Containers/romm/env
 Environment=MARIADB_DATABASE=romm
@@ -12,7 +12,7 @@ After=network.target
 [Container]
 ContainerName=romm-redis
-Image=docker.io/library/redis:7-alpine
+Image=docker.io/library/redis:7-alpine@sha256:8da19c7399dca900c6822a1cf30f9d2285585e2c4b457ff4095b0f4500054d3f
 Network=dankserver_appnet:alias=redis
 Volume=/home/dank/Containers/romm/redis:/data
 HealthCmd=redis-cli ping
@@ -18,7 +18,7 @@ After=romm-mariadb.service romm-redis.service network.target
 [Container]
 ContainerName=romm
-Image=ghcr.io/rommapp/romm:latest
+Image=ghcr.io/rommapp/romm:latest@sha256:8677fb7b81b1d3db20b849e989a1dadc2a11159e51eb34a2acdcb01102274edf
 Network=dankserver_appnet
 EnvironmentFile=/home/dank/Containers/romm/env
 Environment=DB_HOST=mariadb
@@ -4,7 +4,7 @@ After=network-online.target
 Wants=network-online.target
 [Container]
-Image=lscr.io/linuxserver/tautulli:v2.18.1-ls245
+Image=lscr.io/linuxserver/tautulli:v2.18.2-ls250@sha256:529c7eabcc40554c1dc48f33f1f3cc3b1438462d73e0ec2dac8ac0b66aeeb8d4
 ContainerName=tautulli
 Network=dankserver_appnet:alias=tautulli
 Environment=PUID=0
@@ -19,7 +19,7 @@ Description=Gluetun VPN tunnel (FrankBook-Pro podman machine)
 [Container]
 ContainerName=gluetun
-Image=docker.io/qmcgaw/gluetun:latest
+Image=docker.io/qmcgaw/gluetun:latest@sha256:c2ee4191762fca6b6a3c3359a731198a95d9c9af4db32d466e223bf3a84e779b
 AddCapability=NET_ADMIN
 AddDevice=/dev/net/tun:/dev/net/tun
 Environment=VPN_SERVICE_PROVIDER=${VPN_SERVICE_PROVIDER}
@@ -15,7 +15,7 @@ After=gluetun.service
 [Container]
 ContainerName=lidarr
-Image=lscr.io/linuxserver/lidarr:latest
+Image=lscr.io/linuxserver/lidarr:latest@sha256:d8549ff9ea29b4015c32257360652c1000c31ea471a2b3f0835ae77292faf35b
 Environment=PUID=0
 Environment=PGID=0
 Environment=TZ=America/Los_Angeles
@@ -24,7 +24,7 @@ After=gluetun.service
 [Container]
 ContainerName=prowlarr
-Image=lscr.io/linuxserver/prowlarr:latest
+Image=lscr.io/linuxserver/prowlarr:latest@sha256:b3b00047ed20f6ce2b16780145cb6a0acb51c9e28ea6aa6f5dcd33c7034f1797
 Environment=PUID=0
 Environment=PGID=0
 Environment=TZ=America/Los_Angeles
@@ -29,7 +29,7 @@ Network=gluetun.container
 # Pinned to the last 4.x line: Readarr's retired qB client requires the
 # qB 4.x login response (HTTP 200, body "Ok."); qB 5.x answers 204 with an
 # empty body, which fails Readarr's auth check. All other clients accept 4.x.
-Image=lscr.io/linuxserver/qbittorrent:4.6.7
+Image=lscr.io/linuxserver/qbittorrent:4.6.7@sha256:55f15d44396315551f87294a176efae733b16e283f38980308e46073950257c6
 Environment=PUID=0
 Environment=PGID=0
 Environment=TZ=America/Los_Angeles
@@ -15,7 +15,7 @@ After=gluetun.service
 [Container]
 ContainerName=radarr
-Image=lscr.io/linuxserver/radarr:latest
+Image=lscr.io/linuxserver/radarr:latest@sha256:fc0ae3c50f8128b97e84381903ec8d6845c3a569641cb7117c124376cccb57c1
 Environment=PUID=0
 Environment=PGID=0
 Environment=TZ=America/Los_Angeles
@@ -19,7 +19,7 @@ After=gluetun.service
 [Container]
 ContainerName=readarr
-Image=lscr.io/linuxserver/readarr:amd64-1.0.980
+Image=lscr.io/linuxserver/readarr:amd64-1.0.980@sha256:a95e82dbb0eb2bb43de4c150f1dab9dd821306c67ea0b418983a957e731f2b61
 Environment=PUID=0
 Environment=PGID=0
 Environment=TZ=America/Los_Angeles
@@ -16,7 +16,7 @@ After=gluetun.service
 [Container]
 ContainerName=sonarr
-Image=lscr.io/linuxserver/sonarr:latest
+Image=lscr.io/linuxserver/sonarr:latest@sha256:889dcc58c6aea859033985894b519b2e6b1e55d92527ed123f9f52e42ae7c97b
 Environment=PUID=0
 Environment=PGID=0
 Environment=TZ=America/Los_Angeles
@@ -1,27 +1,45 @@
 #!/usr/bin/env bash
-# apply-protected-cast.sh — RUNS ON DANKSERVER. Rewrite the recipe's
+# apply-protected-cast.sh — RUNS ON DANKSERVER. Rewrite a recipe's
-# PROTECTED-BEGIN/END section from Swift's OWN calibration grams: run
+# PROTECTED-BEGIN/END section from the campaign's OWN calibration grams: run
 # derive-protected-experts.py over the calib dir, splice its cast lines into
 # the marked section of the recipe in place, keep the pre-edit recipe in the
-# campaign trash. Refuses to run twice (marker text says derived).
+# campaign trash. Refuses to run twice (marker text says derived), and refuses
 # when a requested target count was not reached. Run it on a COPY of the
 # template so the committed recipe stays the clean, reviewable baseline.
 set -euo pipefail
-CALIB=${1:-/home/dank/corpora/calib}
+SELF_DIR="$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)"
-RECIPE=${2:-/home/dank/serverAppConfigs-swift15/scripts/recipes/qwen4exp-swift15-w4dn8-i8-hc8m.recipe}
+CALIB=${1:-/home/dank/campaign/qwen38-w5nl64/calib}
-PY=/home/dank/venvs/radcorpus/bin/python
+RECIPE=${2:-$SELF_DIR/recipes/qwen4exp-qwen38-w5nl64-bf16trunk.recipe}
-DERIVE=/home/dank/serverAppConfigs-swift15/scripts/derive-protected-experts.py
+TARGET_COUNT=${3:-0}
 PY=${PY:-/home/dank/venvs/radcorpus/bin/python}
 DERIVE=${DERIVE:-$SELF_DIR/derive-protected-experts.py}
 TS=$(date +%Y%m%d-%H%M%S)
 # OUT and TRASH_DIR are overridable so the seam can drive the whole script in a
 # scratch directory; on the box they are the campaign's usual places.
 OUT=${OUT:-/tmp/protected-cast-$TS.txt}
 TRASH_DIR=${TRASH_DIR:-/home/dank/.AGENTTRASH}
 grep -q "derive-protected-experts.py rewrote" "$RECIPE" && {
    echo "recipe already carries derived cast lines; refusing to re-derive" >&2; exit 1; }
 grep -q "PROTECTED-BEGIN" "$RECIPE" || { echo "no PROTECTED markers in $RECIPE" >&2; exit 1; }
-"$PY" "$DERIVE" "$CALIB" > "/tmp/protected-cast-$TS.txt"
+ARGS=("$DERIVE" "$CALIB")
-N=$(grep -c 'cast dtype=bf16' "/tmp/protected-cast-$TS.txt")
+if [ "$TARGET_COUNT" -gt 0 ]; then
-[ "$N" -ge 2 ] || { echo "derive produced $N cast lines; refusing" >&2; exit 1; }
+    ARGS+=(--target-count "$TARGET_COUNT")
-echo "derived $N protected experts -> /tmp/protected-cast-$TS.txt"
+fi
 "$PY" "${ARGS[@]}" > "$OUT"
 N=$(grep -c 'cast dtype=bf16' "$OUT")
 [ "$N" -ge 2 ] || { echo "derive produced $N cast lines; refusing (see $OUT)" >&2; exit 1; }
 if [ "$TARGET_COUNT" -gt 0 ] && [ "$N" -lt "$TARGET_COUNT" ]; then
    echo "derive reached $N protected experts, target was $TARGET_COUNT; refusing to convert" >&2
    echo "       (the bar sweep bottomed out: the recording is too thin or the target too high)" >&2
    exit 1
 fi
 echo "derived $N protected experts -> $OUT"
-cp -f "$RECIPE" "/home/dank/.AGENTTRASH/$TS-qwen4exp-swift15-w4dn8-i8-hc8m.recipe"
+mkdir -p "$TRASH_DIR"
-"$PY" - "$RECIPE" "/tmp/protected-cast-$TS.txt" <<'PYEOF'
+cp -f "$RECIPE" "$TRASH_DIR/$TS-$(basename "$RECIPE")"
 "$PY" - "$RECIPE" "$OUT" <<'PYEOF'
 import sys
 recipe, castfile = sys.argv[1], sys.argv[2]
 lines = open(recipe).read().splitlines()
@@ -35,7 +53,7 @@ for l in lines:
    elif 'PROTECTED-END' in l:
        insec = False
        out.append(l)
-        out.append('# (derive-protected-experts.py rewrote this section from Swift\'s own grams)')
+        out.append('# (derive-protected-experts.py rewrote this section from the campaign grams)')
    elif not insec:
        out.append(l)
 open(recipe, 'w').write('\n'.join(out) + '\n')
@@ -0,0 +1,220 @@
 #!/usr/bin/env bash
 # build-qwen38-bf16-rad.sh — RUNS ON DANKSERVER. rad-convert the vendor BF16
 # checkpoint /home/dank/models/Qwen3.8-Flash-Next into
 # /home/dank/models/rad/qwen38-bf16.rad with NO recipe and NO --reuse: a weight
 # no rule matches keeps the checkpoint's encoding, so every tensor stays the
 # bf16 it is in the safetensors and the container is the checkpoint's own
 # numbers behind a .rad header.
 #
 # This file is the campaign's measuring stick, twice over: scripts/
 # run-qwen38-bf16.sh kldref records the KLD reference logits from it (every
 # arm's --kld-ref is scored against that record) and the same container serves
 # with the calibration tap to write the Grams that derive the protected-expert
 # list and feed GPTQ. A reference that is not the vendor checkpoint, byte for
 # byte in bf16, silently moves every gate in the campaign — hence no recipe, no
 # --reuse, and the four refusal gates below.
 #
 # Why a container at all: a checkpoint dir has no mapped/file tiers, and a
 # weight only a host kernel reads (the 47.7 GiB PLE n-gram table) cannot be
 # served from one; rad-convert also bakes the tokenizer. The convert runs on
 # the HOST backend — no --device /dev/kfd, no /dev/dri — so declare resolves
 # kernels on CPU and production serving is untouched for the whole run (the
 # precedent that proved it: scripts/convert-swift15-w4dn8.sh).
 #
 # Gates, every one a refusal with exit 2 and a named reason, BEFORE anything is
 # written, cheapest first — the verifier runs last of them because its Hub gate
 # hashes 343 GiB and must not be paid for by a run that was going to refuse on
 # the output file anyway.
 #   checkpoint  the tree must be on disk; an absent dir names the command that
 #               makes one (scripts/download-qwen38-bf16.sh --fetch) and stops.
 #   free space  the convert writes ~343,323 MiB next to the ~343,323 MiB
 #               checkpoint; QWEN38_MIN_FREE_MIB (default 350000 = that figure
 #               plus 6.5 GiB of slack) is the floor. The volume that cannot
 #               hold both is discovered at minute 2, not hour 5.
 #   output      the campaign converts the reference ONCE. An existing
 #               qwen38-bf16.rad is refused; --force retires it to
 #               ~/.AGENTTRASH/<utc>-qwen38-bf16.rad (trash law: nothing is
 #               deleted, and the move is a rename on the same volume).
 #   verified    scripts/download-qwen38-bf16.sh's verify mode must pass. A
 #               failed verify exits 1, not 2: the tree being the wrong payload
 #               is a result, a missing environment is a refusal.
 #   plan        rad-convert --plan-only -v first — the oracle for "does this
 #               shape boot". A refused plan costs seconds and saves a convert.
 #
 # usage: scripts/build-qwen38-bf16-rad.sh [--dry-run] [--plan-only] [--force]
 #   --dry-run    print the argv it would exec (podman wrapping rad-convert),
 #               one shell-quoted argument a line, and exit 0. No gate runs, no
 #               container starts, nothing is written — the offline test seam.
 #               It does resolve the checkpoint and output paths through the
 #               /models mount, because an argv that carries host paths would
 #               not run either.
 #   --plan-only  run the plan and every gate except the disk and output ones —
 #               those two protect bytes the plan never writes, and this is the
 #               probe to run when the disk gate itself is what refuses
 #   --force      retire an existing output container to the trash dir first
 #
 # knobs: FRANK_QWEN38_MODELS   the host dir mounted at /models (default
 #        /home/dank/models); FRANK_QWEN38_CKPT (default $MODELS/
 #        Qwen3.8-Flash-Next, the same knob the verifier reads);
 #        FRANK_QWEN38_BF16_OUT (default $MODELS/rad/qwen38-bf16.rad);
 #        QWEN38_MIN_FREE_MIB; QWEN38_TRASH (default $HOME/.AGENTTRASH). The
 #        verifier's own census knobs (FRANK_QWEN38_SHARDS / _BYTES / _REPO)
 #        reach it through the environment untouched.
 # exit: 0 converted (or planned), 1 the checkpoint failed verification,
 #       2 refused before anything was written
 set -euo pipefail
 SCRIPT_DIR="$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)"
 # ONE owner of the engine pin: the digest frank-archon serves on. The reference
 # must be built by the SAME engine build that scores it, or a KLD delta is
 # partly a version delta.
 # shellcheck disable=SC1091
 source "$SCRIPT_DIR/frank-archon.env"
 IMAGE="${FRANK_ARCHON_IMAGE:?FRANK_ARCHON_IMAGE must name the radiance image}"
 MODELS="${FRANK_QWEN38_MODELS:-/home/dank/models}"
 CKPT="${FRANK_QWEN38_CKPT:-$MODELS/Qwen3.8-Flash-Next}"
 OUT="${FRANK_QWEN38_BF16_OUT:-$MODELS/rad/qwen38-bf16.rad}"
 TRASH="${QWEN38_TRASH:-$HOME/.AGENTTRASH}"
 MIN_FREE_MIB="${QWEN38_MIN_FREE_MIB:-350000}"
 LOG="/home/dank/logs/qwen38_bf16_convert.log"
 DRY=0
 PLAN_ONLY=0
 FORCE=0
 while [ $# -gt 0 ]; do
    case "$1" in
        --dry-run)   DRY=1 ;;
        --plan-only) PLAN_ONLY=1 ;;
        --force)     FORCE=1 ;;
        *) echo "usage: ${0##*/} [--dry-run] [--plan-only] [--force]" >&2
           exit 2 ;;
    esac
    shift
 done
 # The convert sees the tree only through the /models mount, so both paths must
 # be under it; the container-side spelling is what goes into the argv.
 case "$CKPT" in
    "$MODELS"/*) CKPT_IN="/models/${CKPT#"$MODELS"/}" ;;
    *) echo "REFUSED: checkpoint $CKPT is not under the mounted models dir $MODELS — the convert cannot see it" >&2
       exit 2 ;;
 esac
 case "$OUT" in
    "$MODELS"/*) OUT_IN="/models/${OUT#"$MODELS"/}" ;;
    *) echo "REFUSED: output $OUT is not under the mounted models dir $MODELS — rad-convert could not write it" >&2
       exit 2 ;;
 esac
 # Declare with NO recipe line — that is what keeps every tensor bf16 — and at the
 # bound this container will actually be served at. Both flags come off the pinned
 # image's own help: --kv-cache-dtype at convert "only picks the kernels it
 # checks" and the container is identical either way, so naming fp8 is free
 # insurance — if the fp8 KV kernels the fleet serves would not resolve, the
 # convert refuses here instead of the boot failing on a drained window.
 # --max-ctx is the bound declare is run at, whose default is the checkpoint's
 # training context (262,144 here, whose fp8 cache is 258.8 GiB against a 192 GiB
 # machine, so the default is not an option). 32,768 is set by the corpus, not by
 # taste: kld.jsonl's longest record is 38,263 characters — ~9,600 tokens at the
 # pessimistic chars/4, ~7,700 at the chars/4.97 this repo measured on the K3
 # corpus — so the window the rig boots at clears every record 2.6x over and the
 # reference boot cannot die on a sequence that will not fit. The rig's window is
 # one knob (FRANK_QWEN38_MAX_MODEL_LEN in scripts/run-qwen38-bf16.sh, default
 # 32,768) and that script refuses a corpus its window cannot hold, so raising one
 # without the other is caught before the drained window, not inside it. The
 # payload containers the gate scores declare at the training context because
 # production must hold the 200k shape; a declared bound is a declare-time check,
 # not a weight change, so the reference and the payloads carry the same numbers
 # whatever each declared at — which is why one can be measured against the other.
 CONVERT_ARGS=("$CKPT_IN" -o "$OUT_IN" --max-ctx 32768 --kv-cache-dtype fp8)
 PODMAN=(podman run --rm --entrypoint rad-convert -v "$MODELS:/models")
 PLAN_CMD=("${PODMAN[@]}" --name qwen38-bf16-plan "$IMAGE" "${CONVERT_ARGS[@]}" --plan-only -v)
 CONV_CMD=("${PODMAN[@]}" --name qwen38-bf16-convert "$IMAGE" "${CONVERT_ARGS[@]}")
 if [ "$DRY" = "1" ]; then
    echo "dry run: no gate checked, no container started, nothing written"
    echo "# $CKPT -> $OUT (the convert sees them as $CKPT_IN and $OUT_IN)"
    echo "# plan argv:"
    printf '%q\n' "${PLAN_CMD[@]}"
    echo "# convert argv:"
    printf '%q\n' "${CONV_CMD[@]}"
    exit 0
 fi
 # ---- gate: the checkpoint is on disk ---------------------------------------
 if [ ! -d "$CKPT" ]; then
    echo "REFUSED: no checkpoint at $CKPT — fetch and verify it first: scripts/download-qwen38-bf16.sh --fetch" >&2
    exit 2
 fi
 # ---- gate: peak disk --------------------------------------------------------
 # The deepest existing ancestor is the right thing to ask: a fresh
 # /home/dank/models/rad has no free space of its own to report.
 OUT_DIR="$(dirname "$OUT")"
 PROBE="$OUT_DIR"
 while [ ! -d "$PROBE" ] && [ "$PROBE" != "/" ]; do PROBE="$(dirname "$PROBE")"; done
 FREE_MIB="$(df --output=avail -m "$PROBE" | tail -1 | tr -d ' ')"
 if ! [[ "$FREE_MIB" =~ ^[0-9]+$ ]]; then
    echo "REFUSED: cannot read the free space under $PROBE (df said: $FREE_MIB)" >&2
    exit 2
 fi
 if [ "$PLAN_ONLY" != "1" ] && [ "$FREE_MIB" -lt "$MIN_FREE_MIB" ]; then
    echo "REFUSED peak-disk: $FREE_MIB MiB free under $PROBE, $MIN_FREE_MIB MiB wanted (QWEN38_MIN_FREE_MIB) — the bf16 container rewrites the checkpoint's own 343,323 MiB of tensors beside it" >&2
    exit 2
 fi
 # ---- gate: one conversion ---------------------------------------------------
 if [ -e "$OUT" ]; then
    if [ "$PLAN_ONLY" = "1" ]; then
        echo "note: $OUT already exists; --plan-only writes nothing, so it stays"
    elif [ "$FORCE" = "1" ]; then
        mkdir -p "$TRASH"
        DEST="$TRASH/$(date -u +%Y%m%dT%H%M%SZ)-$(basename "$OUT")"
        mv "$OUT" "$DEST"
        echo "TRASHED: retired the existing reference container $OUT -> $DEST"
    else
        echo "REFUSED: $OUT already exists — the campaign converts the reference once; re-run with --force to retire it to $TRASH first" >&2
        exit 2
    fi
 fi
 # ---- gate: the tree really is the checkpoint (the full verify) -------------
 echo "== verifying $CKPT (scripts/download-qwen38-bf16.sh, full gates)"
 rc=0
 bash "$SCRIPT_DIR/download-qwen38-bf16.sh" --dir "$CKPT" || rc=$?
 if [ "$rc" -ne 0 ]; then
    if [ "$rc" -eq 2 ]; then
        echo "REFUSED: the checkpoint verifier refused the environment (rc=2); nothing was converted" >&2
        exit 2
    fi
    echo "REFUSED: $CKPT failed verification (rc=$rc) — a half tree converts to a half reference" >&2
    exit 1
 fi
 # ---- plan, then convert ------------------------------------------------------
 mkdir -p "$(dirname "$LOG")"
 # rad-convert opens -o during declare as well, so the output directory exists
 # before the plan; that empty directory is all this creates ahead of it.
 mkdir -p "$OUT_DIR"
 echo "== rad-convert --plan-only -v (declare + plan; writes nothing)"
 "${PLAN_CMD[@]}" 2>&1 | tee -a "$LOG"
 rc=${PIPESTATUS[0]}
 if [ "$rc" -ne 0 ]; then
    echo "REFUSED plan: rad-convert --plan-only exited rc=$rc — the engine refuses this encoding/shape, no container was written (log: $LOG)" >&2
    exit 2
 fi
 if [ "$PLAN_ONLY" = "1" ]; then
    echo "PLAN_OK: $CKPT_IN declares for $OUT_IN; no container written (log: $LOG)"
    exit 0
 fi
 echo "== convert: $CKPT -> $OUT ($(date -Is))"
 "${CONV_CMD[@]}" 2>&1 | tee -a "$LOG"
 rc=${PIPESTATUS[0]}
 if [ "$rc" -ne 0 ]; then
    echo "rad-convert rc=$rc; log: $LOG" >&2
    exit "$rc"
 fi
 MIB=$(( $(stat -c %s "$OUT") / 1048576 ))
 echo "BUILD_DONE: $OUT is ${MIB} MiB (log: $LOG)"
 echo "NEXT: scripts/run-qwen38-bf16.sh kldref records the reference logits from it"
@@ -22,6 +22,17 @@ Mechanics notes:
  prompts do not bias the Hessians (the corpus is drawn round-robin over
  source classes, so loss is class-neutral as long as failures are rare;
  a systematic 4xx is loud in the log and must be fixed, not ignored).
 - a round that seals no tap bytes is NORMAL and continues: RADIANCE_CALIB_TOKENS
  is a pooled floor and this campaign's corpus is 10.03M tokens against a 12M
  floor, so round 1 seals nothing by construction. A tap that LOSES bytes fails
  the run, and a run that never sees the engine's completion line exits non-zero.
  The first version failed on a flat round, which aborted exactly one round too
  early (2026-10-29).
 - before any prompt, the tap must carry the rig's own marker
  (`--tap-marker`, written by run-qwen38-bf16.sh calib) naming THIS
  container, and that container must be running. Measured 2026-10-29:
  with neither check, four rounds of 10M tokens went into restored
  production after the rig's boot had died, and recorded nothing.
 Usage: calib-push.py [--port 7770] [--parallel 64] [--container arbench]
       [--corpus /home/dank/corpora/out/calib.jsonl] [--max-rounds 4]
@@ -29,11 +40,18 @@ Usage: calib-push.py [--port 7770] [--parallel 64] [--container arbench]
 import argparse
 import concurrent.futures
 import json
 import os
 import subprocess
 import sys
 import time
 import urllib.request
 _HERE = os.path.dirname(os.path.abspath(__file__)) if "__file__" in globals() else os.getcwd()
 if _HERE not in sys.path:
    sys.path.insert(0, _HERE)
 import qwen38_common as qc  # noqa: E402 - sibling module, same directory
 DONE_MARK = "calibration wrote"
@@ -49,9 +67,9 @@ def tap_done(container):
    return DONE_MARK in (out.stdout + out.stderr)
-def push_one(url, prompt):
+def push_one(url, prompt, model):
    body = json.dumps({
-        "model": "frank-archon",
+        "model": model,
        "prompt": prompt,
        "max_tokens": 1,
        "temperature": 0,
@@ -68,38 +86,151 @@ def push_one(url, prompt):
        return False
-def main():
+def push_round(url, prompts, model, parallel):
    """Push the whole corpus once; returns how many requests were accepted."""
    ok = 0
    with concurrent.futures.ThreadPoolExecutor(parallel) as ex:
        for good in ex.map(lambda p: push_one(url, p, model), prompts):
            ok += 1 if good else 0
    return ok
 def tap_bytes(path):
    """Bytes the tap has sealed so far; 0 when the directory is absent.
    A tap whose directory is wrong, full or read-only stops writing while the
    engine keeps answering 200 — the driver then re-pushes the whole corpus for
    every round it is allowed and the loss is invisible until the hours are gone.
    """
    total = 0
    try:
        for root, _, files in os.walk(path):
            for f in files:
                try:
                    total += os.path.getsize(os.path.join(root, f))
                except OSError:
                    pass
    except OSError:
        return 0
    return total
 def round_verdict(before, after, done):
    """What one round's tap bytes mean.
    'done'   the engine logged that it sealed the tap — the run is over
    'shrink' the tap LOST bytes: reset, pruned or replaced under us
    'grew'   sealed bytes this round
    'flat'   sealed nothing — which is legitimate: RADIANCE_CALIB_TOKENS is a
             pooled floor, and a round that does not reach it has nothing to
             write yet (the campaign's corpus is 10.03M tokens against a 12M
             floor, so round 1 is always flat). Failing on it aborted the run
             exactly one round too early.
    """
    if done:
        return "done"
    if after < before:
        return "shrink"
    if after == before:
        return "flat"
    return "grew"
 def require_rig(a):
    """Refuse unless the rig that opened this tap is up and serving right now.
    Three facts, in order: the runner wrote its marker into the tap dir, the
    marker names the container this pusher targets, and that container is
    running. A marker without a live container is a boot that died; a live
    container without a marker is some other engine on the port — on 2026-10-29
    it was restored production, which took four rounds of 10M tokens before the
    old driver admitted it had recorded nothing.
    """
    marker = os.path.join(a.tap_dir, a.tap_marker)
    if not os.path.isfile(marker):
        qc.refuse(f"no rig marker at {marker}: this tap was not opened by "
                  "run-qwen38-bf16.sh calib, so a push here would feed whatever "
                  "answers the port and record nothing")
    try:
        lines = open(marker, encoding="utf-8", errors="replace").read().splitlines()
    except OSError as exc:
        qc.refuse(f"cannot read the rig marker {marker}: {exc}")
    meta = dict(line.split("=", 1) for line in lines if "=" in line)
    who = meta.get("container", "")
    if who != a.container:
        qc.refuse(f"the rig marker names container {who!r} and this pusher targets "
                  f"{a.container!r} — wrong tap, or a stale marker")
    try:
        p = subprocess.run(["podman", "inspect", "-f", "{{.State.Running}}", a.container],
                           capture_output=True, text=True, timeout=30)
    except Exception as exc:
        qc.refuse(f"cannot inspect the rig container {a.container}: {exc}")
    if p.returncode != 0 or p.stdout.strip() != "true":
        qc.refuse(f"the rig container {a.container} is not running — whatever is "
                  "answering the port, it is not the calibration rig")
 def parse_args():
    ap = argparse.ArgumentParser()
    ap.add_argument("--port", type=int, default=7770)
    ap.add_argument("--parallel", type=int, default=64)
    ap.add_argument("--container", default="arbench")
    ap.add_argument("--model", default="frank-archon")
    ap.add_argument("--tap-dir", default="/home/dank/campaign/qwen38-w5nl64/calib")
    ap.add_argument("--tap-marker", default=".rig-boot",
                    help="the file run-qwen38-bf16.sh calib writes into the tap dir; without it the pusher refuses, because an unmarked tap is not a rig")
    ap.add_argument("--corpus", default="/home/dank/corpora/out/calib.jsonl")
    ap.add_argument("--max-rounds", type=int, default=4)
-    a = ap.parse_args()
+    return ap.parse_args()
    url = f"http://127.0.0.1:{a.port}/v1/completions"
 def load_prompts(a) -> list[str]:
    if not os.path.isfile(a.corpus):
        qc.refuse(f"no corpus at {a.corpus}")
    prompts = []
-    with open(a.corpus) as f:
+    for lineno, rec in qc.iter_jsonl(a.corpus):
-        for line in f:
+        if not isinstance(rec.get("prompt"), str):
-            line = line.strip()
+            qc.refuse(f"{a.corpus}:{lineno} has no string 'prompt' field")
-            if line:
+        prompts.append(rec["prompt"])
-                prompts.append(json.loads(line)["prompt"])
+    if not prompts:
-    print(f"corpus: {len(prompts)} prompts", flush=True)
+        qc.refuse(f"the corpus at {a.corpus} holds no prompts")
    return prompts
 def main():
    a = parse_args()
    url = f"http://127.0.0.1:{a.port}/v1/completions"
    prompts = load_prompts(a)
    require_rig(a)
    chars = sum(len(p) for p in prompts)
    print(f"corpus: {len(prompts)} prompts, ~{chars // 4} tokens (chars/4)",
          flush=True)
    for rnd in range(1, a.max_rounds + 1):
        if tap_done(a.container):
            break
        before = tap_bytes(a.tap_dir)
        t0 = time.time()
-        ok = 0
+        ok = push_round(url, prompts, a.model, a.parallel)
-        with concurrent.futures.ThreadPoolExecutor(a.parallel) as ex:
+        dt = max(time.time() - t0, 1e-9)
            for good in ex.map(lambda p: push_one(url, p), prompts):
                ok += 1 if good else 0
        dt = time.time() - t0
        print(f"round {rnd}: {ok}/{len(prompts)} ok in {dt:.0f}s "
-              f"({sum(len(p) for p in prompts)/4/dt:.0f} chars/s est)",
+              f"({chars / 4 / dt:.0f} tokens/s est)", flush=True)
-              flush=True)
+        if ok == 0:
            print("FAIL: not one prompt was accepted; fix the engine or the argv "
                  "before spending another round", flush=True)
            sys.exit(1)
        if tap_done(a.container):
            break
        after = tap_bytes(a.tap_dir)
        verdict = round_verdict(before, after, False)
        if verdict == "shrink":
            print(f"FAIL: the tap at {a.tap_dir} LOST bytes this round "
                  f"({before} B before, {after} B after) — it was reset or replaced "
                  "under us", flush=True)
            sys.exit(1)
        if verdict == "flat":
            print(f"round {rnd}: the tap sealed nothing yet (byte floor not reached; "
                  f"{before} B) — continuing", flush=True)
    else:
        print("max rounds exhausted, tap not done", flush=True)
        sys.exit(1)
@@ -21,7 +21,7 @@
 # deployed knob has since dropped to 262144.
 set -euo pipefail
-IMG=ghcr.io/peonist-ai/halogen-flash-server@sha256:0c61bf84ac22308a53f5d1ca6b86806702d7039e5ebc51cae4c66621b92fe04a
+IMG=ghcr.io/peonist-ai/halogen-flash-server@sha256:e35133b206da3094cc4d70c78800b26235bd9124b9aeb0e029d4a7c153b9cff0
 CUR=/home/dank/models/halogen-flashnext-v2
 NEW=/home/dank/models/halogen-v2-abliterated
 CUR_CK=qwen38-flash-next-v2-swift15-abliterated.hgn
@@ -0,0 +1,236 @@
 #!/usr/bin/env bash
 # convert-qwen38-w5nl64.sh — RUNS ON DANKSERVER. The CPU half of the Qwen3.8-Flash-Next five-bit
 # campaign: convert the vendor BF16 checkpoint into the campaign's arms, each one the whole
 # `w5nl64a8h` rung on BOTH expert projections with the protected experts kept bf16.
 #
 # Nothing here touches a GPU: the container is started without /dev/kfd or /dev/dri, so the
 # converter runs on the host backend and the fleet can keep serving through the whole script.
 # The engine pin is the digest that serves frank-archon, so a winning arm crosses no version
 # boundary between measurement and serving.
 #
 # Three arms, because the campaign's question is which search knob pays for its hours
 # (recipes/qwen4exp-qwen38-w5nl64-bf16trunk.recipe):
 #
 #   cd3     the recipe as written — coordinate descent three sweeps. The control.
 #   cd6     six sweeps: the same search, twice the passes.
 #   ao1     act_order=1 instead of cd: GPTQ sweeps columns in activation-importance order and
 #           stores them permuted. This is a SEPARATE arm, not a fourth sweep — libquant refuses
 #           cd= together with act_order=1 ("cd= sweeps the columns in their stored order, and
 #           act_order=1 stores them permuted; one or the other", libquant/lq_registry.cpp), so
 #           the two knobs cannot be combined into a best-of-both container.
 #
 # The arms are generated from ONE template with `--quant` overrides, not three near-duplicate
 # recipes: --quant rules are inserted BEFORE the recipe file and the first rule whose glob matches
 # a weight decides it (core/format/recipe.cpp: Recipe::match), so an override cleanly replaces the
 # expert line and the other 60 lines cannot drift between arms.
 #
 # Protected experts: apply-protected-cast.sh derives the bf16 set from THIS campaign's grams
 # (--target-count 48, two per layer so each rank's 256-expert half stays even) and splices it into
 # a per-run COPY of the template; the committed recipe keeps its markers and stays reviewable.
 #
 # Gates, in the order they bite — each one is a lesson the swift15 campaign paid for:
 #   inputs        checkpoint (dir + index), recipe, grams, imatrix, tokenizer. A missing grams
 #                 directory converts UNCALIBRATED with one stderr note, which is silently worse
 #                 than failing here.
 #   reference     the 335.3 GiB BF16 reference container must be GONE before the converts start.
 #                 It and the checkpoint together eat the volume; its job (the KLD recording and
 #                 the calibration grams) is finished by then, and both survive on disk without the
 #                 container beside it. The FP8-native anchor is FETCHED after the converts (disk
 #                 sequencing), so nothing here waits on it.
 #   plan          --plan-only runs FIRST, per arm, and prints "about X would be written". The
 #                 number is the bill: free space must cover it plus the 128k-prompt overshoot
 #                 margin, and the finished artifact is checked against it.
 #   outputs       no arm overwrites anything; move the old one to trash.
 #
 # Usage: convert-qwen38-w5nl64.sh [--arm cd3|cd6|ao1]... [--dry-run] [--plan-only]
 #                                 [--target-count N] [--keep-recipe]
 set -u -o pipefail
 SELF="$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)"
 # THE TOOLCHAIN PIN. `:latest` on 2026-10-30 is v1.14.0 (revision 4a61d7296bc5),
 # the same image the tap producer boots: a tap must be converted by the line that
 # is current, and rad-convert's `gptq:calib=<dir>` input contract is version-stable
 # (RADGRAM2 + gram.r<rank>.<stem>.bin), so nothing else about the recipe moves with
 # it. Production stays on its own pin — this is the campaign's toolchain.
 IMAGE=${IMAGE:-stilldeadcode/radiance@sha256:69add58f7de1ed2bca2b1e20a4cc75717557df3f73569a178eca5aafe8c44093}
 CKPT=${CKPT:-/home/dank/models/Qwen3.8-Flash-Next}
 CAMPAIGN=${CAMPAIGN:-/home/dank/campaign/qwen38-w5nl64}
 RECIPE=${RECIPE:-$SELF/recipes/qwen4exp-qwen38-w5nl64-bf16trunk.recipe}
 CALIB=${CALIB:-$CAMPAIGN/calib}
 IMATRIX=${IMATRIX:-$CAMPAIGN/imatrix.gguf}
 OUTDIR=${OUTDIR:-/home/dank/models/rad}
 BF16_RAD=${BF16_RAD:-$OUTDIR/qwen38-bf16.rad}   # the reference container, not an arm output
 STATE=${STATE:-$CAMPAIGN}                      # the dir the container mounts read-only: the
                                               # spliced recipe and the imatrix live here
 TARGET_COUNT=${TARGET_COUNT:-48}
 MIN_GRAMS=${MIN_GRAMS:-24576}                 # per-expert DOWN grams: 48 recorded layers x 512
                                               # experts. The checkpoint has 49 MoE layers, but the
                                               # MTP layer is never recorded, so its 512 experts are
                                               # absent from the tap's output by design
 MARGIN_MIB=${MARGIN_MIB:-40960}               # the long-prompt overshoot, off the plan number
 TS=$(date +%Y%m%d-%H%M%S)
 ARMS=()
 DRY=0
 PLAN_ONLY=0
 KEEP_RECIPE=0
 while [ "$#" -gt 0 ]; do
    case "$1" in
      --arm) [ "$#" -ge 2 ] || { echo "--arm needs a value (cd3|cd6|ao1)" >&2; exit 2; }
             ARMS+=("$2"); shift 2 ;;
      --dry-run) DRY=1; shift ;;
      --plan-only) PLAN_ONLY=1; shift ;;
      --target-count) [ "$#" -ge 2 ] || { echo "--target-count needs a value" >&2; exit 2; }
             TARGET_COUNT="$2"; shift 2 ;;
      --keep-recipe) KEEP_RECIPE=1; shift ;;
      *) echo "unknown argument: $1" >&2; exit 2 ;;
    esac
 done
 [ "${#ARMS[@]}" -gt 0 ] || ARMS=(cd3 cd6 ao1)
 for a in "${ARMS[@]}"; do
    case "$a" in cd3|cd6|ao1) ;; *) echo "unknown arm: $a (cd3|cd6|ao1)" >&2; exit 2 ;; esac
 done
 # The expert rule's option set, spelled once. --quant replaces the template's line wholesale, so
 # every option has to be repeated here; `calib=$CALIB` is expanded by the recipe reader from the
 # container environment (core/format/recipe.cpp: expand refuses an unset name rather than taking
 # it for empty).
 expert_opts='table=w5nl,group=64,scale=fp8_e4m3,scale2=f32,block2=*x*,scale2_value=0.0001220703125,transform=fwht128,rule=search,calib=$CALIB'
 arm_quant() {
    case "$1" in
      # The control is the template itself: no override, so the recipe's own line decides.
      cd3) return 0 ;;
      cd6) echo "blk.*.ffn_*_exps.*.weight=gptq:${expert_opts},cd=6" ;;
      # act_order=1 WITHOUT cd: libquant refuses the pair (lq_registry.cpp:105, "cd= sweeps the
      # columns in their stored order, and act_order=1 stores them permuted; one or the other"), so
      # this arm is the interleave knob alone, not cd=3 plus it.
      ao1) echo "blk.*.ffn_*_exps.*.weight=gptq:${expert_opts},act_order=1" ;;
    esac
 }
 # --- gates ------------------------------------------------------------------------------------
 [ -d "$CKPT" ] || { echo "no checkpoint at $CKPT; run scripts/download-qwen38-bf16.sh --fetch" >&2; exit 2; }
 [ -f "$CKPT/model.safetensors.index.json" ] || { echo "checkpoint at $CKPT has no safetensors index — a torn fetch" >&2; exit 2; }
 [ -f "$CKPT/tokenizer.json" ] || { echo "no tokenizer.json in $CKPT; the .rad cannot carry one without it" >&2; exit 2; }
 [ -f "$RECIPE" ] || { echo "no recipe at $RECIPE" >&2; exit 2; }
 GRAMS=$(find "$CALIB" -maxdepth 1 -name 'gram.r*.bin' 2>/dev/null | wc -l)
 [ "$GRAMS" -ge "$MIN_GRAMS" ] || {
    echo "calib dir $CALIB holds $GRAMS gram files (<$MIN_GRAMS); run $0 after run-qwen38-bf16.sh calib" >&2; exit 2; }
 [ -f "$IMATRIX" ] || { echo "no imatrix at $IMATRIX; run scripts/grams-to-imatrix.py over $CALIB" >&2; exit 2; }
 # The container sees the campaign dir at one fixed mount point, so an imatrix kept elsewhere has
 # no path inside it: copy it in (or point IMATRIX at the campaign dir) rather than let the
 # converter fail halfway through a planned arm.
 [ "$(dirname "$IMATRIX")" = "$STATE" ] || {
    echo "imatrix $IMATRIX is not in the campaign dir $STATE the container mounts; move it there" >&2; exit 2; }
 if [ -e "$BF16_RAD" ]; then
    echo "the 335.3 GiB BF16 reference container $BF16_RAD is still on disk; its recording and" \
         "the calibration grams are already taken, so move it to trash — the converts need its" \
         "space (peak-disk law)" >&2
    exit 2
 fi
 for a in "${ARMS[@]}"; do
    [ -e "$OUTDIR/qwen38-w5nl64-$a.rad" ] && {
        echo "$OUTDIR/qwen38-w5nl64-$a.rad already exists; move it to trash first" >&2; exit 2; }
    # A killed run leaves its container behind (--rm only fires when podman itself
    # exits), and the next arm then dies on "name already in use" AFTER the plan
    # phase; refuse here, where the fix is one `podman rm -f`.
    podman container exists "qwen38-convert-$a" && {
        echo "container qwen38-convert-$a still exists (a killed run?);" \
             "'podman rm -f qwen38-convert-$a' and re-run" >&2
        exit 2; }
 done
 # --- the protected plane, spliced into a per-run copy of the template --------------------------
 mkdir -p "$STATE"
 WORK="$STATE/recipe-$TS.recipe"
 cp "$RECIPE" "$WORK"
 if [ "$DRY" = 1 ]; then
    echo "would run: $SELF/apply-protected-cast.sh $CALIB $WORK $TARGET_COUNT"
 else
    # Positional, because the tool reads its recipe and calib dir by position: passing them as
    # environment variables would be silently ignored and it would rewrite the COMMITTED
    # template in place instead of this run's copy.
    "$SELF/apply-protected-cast.sh" "$CALIB" "$WORK" "$TARGET_COUNT" \
        || { echo "the protected-expert derivation refused; the arms would convert unprotected" >&2; exit 1; }
 fi
 run_convert() { # run_convert ARM [extra flags...]
    local arm="$1"; shift
    local out="$OUTDIR/qwen38-w5nl64-$arm.rad" log="$STATE/convert-$arm-$TS.log"
    local argv=(--rm --name "qwen38-convert-$arm"
                -e "CALIB=/data/calib" -e "OMP_NUM_THREADS=$(nproc)"
                -v "$(dirname "$CKPT"):/models:ro"
                -v "$CALIB:/data/calib:ro"
                -v "$STATE:/campaign:ro"
                -v "$OUTDIR:/out")
    local conv=(/models/"${CKPT##*/}" --recipe "/campaign/$(basename "$WORK")"
                --tokenizer "/models/${CKPT##*/}/tokenizer.json"
                --imatrix /campaign/"${IMATRIX##*/}"
                # No --max-ctx: the context declare then runs at the checkpoint's training context
                # (262144), which is what the shipped container did — rad-info --meta on it carries
                # no override. A container declared at a smaller bound cannot be expected to serve
                # the 200k window D2 holds, and the convert is not where to discover that.
                # --kv-cache-dtype changes nothing in the artifact; it picks the kernels the declare
                # checks, and the fleet serves fp8 KV, so an unresolvable fp8 kernel must be said no
                # to by the convert, not by a boot on a drained window.
                --kv-cache-dtype fp8 -o "/out/$(basename "$out")")
    local q; q=$(arm_quant "$arm")
    [ -n "$q" ] && conv+=(--quant "$q")
    if [ "$DRY" = 1 ]; then
        echo "podman run ${argv[*]} \$IMAGE --entrypoint /opt/radiance/bin/rad-convert ${conv[*]} $*"
        return 0
    fi
    podman run "${argv[@]}" --entrypoint /opt/radiance/bin/rad-convert "$IMAGE" "${conv[@]}" "$@" \
        2>&1 | tee "$log"
    return "${PIPESTATUS[0]}"
 }
 # --- plan, then bill, then convert -------------------------------------------------------------
 overall=0
 for arm in "${ARMS[@]}"; do
    echo "== arm $arm =="
    if [ "$DRY" = 1 ]; then
        # Both lines, in order: the plan runs before the convert, and the convert is what the
        # operator is about to pay an afternoon for.
        run_convert "$arm" --plan-only
        run_convert "$arm"
        continue
    fi
    rc=0; plan_log=$(run_convert "$arm" --plan-only) || rc=$?
    [ "$rc" -eq 0 ] || { echo "arm $arm: --plan-only failed (rc=$rc); nothing was written" >&2; overall=1; continue; }
    echo "$plan_log"
    # "about 141.2 GiB would be written" is the converter's own bill; parse it rather than
    # hardcoding a size, so the gate still works after any recipe edit.
    want=$(printf '%s' "$plan_log" | sed -n 's/.*about \([0-9.]*\) \([KMGT]iB\) would be written.*/\1 \2/p' | tail -1)
    [ -n "$want" ] || { echo "arm $arm: the plan printed no size; refusing to convert blind" >&2; overall=1; continue; }
    need_mib=$(awk -v n="${want% *}" -v u="${want#* }" \
        'BEGIN{ f = (u=="KiB") ? 2^-10 : (u=="MiB") ? 1 : (u=="GiB") ? 1024 : (u=="TiB") ? 1048576 : 0
               if (f == 0) { exit 1 }; printf "%d", n*f + 0.5 }') || {
        echo "arm $arm: the plan printed an unknown unit in '$want'" >&2; overall=1; continue; }
    free_mib=$(df --output=avail -m "$OUTDIR" | tail -1 | tr -d ' ')
    echo "plan: $want ($need_mib MiB); free ${free_mib} MiB on $OUTDIR (margin $MARGIN_MIB MiB)"
    if [ "$free_mib" -lt $((need_mib + MARGIN_MIB)) ]; then
        echo "arm $arm: needs ${need_mib} MiB + ${MARGIN_MIB} margin, ${free_mib} free; free space and re-run" >&2
        overall=1; continue
    fi
    if [ "$PLAN_ONLY" = 1 ]; then echo "planned, not converted (--plan-only)"; continue; fi
    rc=0; run_convert "$arm" || rc=$?
    [ "$rc" -eq 0 ] || { echo "arm $arm: rad-convert rc=$rc; log: $STATE/convert-$arm-$TS.log" >&2; overall=1; continue; }
    out="$OUTDIR/qwen38-w5nl64-$arm.rad"
    [ -f "$out" ] || { echo "arm $arm: rad-convert exited 0 but wrote no $out; log: $STATE/convert-$arm-$TS.log" >&2; overall=1; continue; }
    got_mib=$(( $(stat -c %s "$out") / 1048576 ))
    # The receipt: the artifact is what the plan predicted, within the metadata the plan does not
    # count. An arm that came out materially smaller lost a plane somewhere.
    lo=$(( need_mib * 98 / 100 )); hi=$(( need_mib * 103 / 100 ))
    if [ "$got_mib" -lt "$lo" ] || [ "$got_mib" -gt "$hi" ]; then
        echo "arm $arm: $out is ${got_mib} MiB, the plan said ${need_mib} MiB (allowed ${lo}-${hi})" >&2
        overall=1; continue
    fi
    echo "CONVERT_DONE $arm ${got_mib} MiB -> $out"
 done
 if [ "$KEEP_RECIPE" = 0 ] && [ "$DRY" = 0 ]; then
    mkdir -p "$CAMPAIGN/.AGENTTRASH"
    mv "$WORK" "$CAMPAIGN/.AGENTTRASH/$(basename "$WORK")" 2>/dev/null || true
 fi
 exit "$overall"
@@ -34,13 +34,16 @@ SERVICE_ORDER = (
    "hermes",
    "hermes-dashboard",
    "hermes-webui",
    "prometheus",
    "grafana",
    "loki",
    "alloy",
    "node-exporter",
    "cadvisor",
    "gpu-exporter",
    "alertmanager",
    # The prober must answer before prometheus re-runs the services job.
    "blackbox-exporter",
    "prometheus",
    "authentik-postgres",
    "authentik-server",
    "authentik-worker",
@@ -94,7 +97,9 @@ DANKCLAW_SERVICES = {
    "alloy",
    "node-exporter",
    "cadvisor",
    "gpu-exporter",
    "alertmanager",
    "blackbox-exporter",
    "authentik-postgres",
    "authentik-server",
    "authentik-worker",
@@ -127,6 +132,7 @@ DANKSERVER_SERVICES = {
    "frank-archon",
    "node-exporter",
    "cadvisor",
    "gpu-exporter",
 }
 # FrankBook-Pro runs the *arr stack + qBittorrent inside a podman machine on
@@ -149,13 +155,14 @@ DANKCLAW_COMPOSE_SERVICES = set()
 # Quadlet unit now — the last compose services were the hermes trio, whose units
 # have owned their containers since 2026-09-12 and whose compose blocks were
 # dead weight that could only create a duplicate container.
 # jellyfin left the Dankserver file on 2026-10-30 for the same reason: its Quadlet
 # has owned the container since the cutover, so a compose edit must not restart it.
 DANKSERVER_COMPOSE_SERVICES = {
    "mariadb",
    "redis",
    "romm",
    "copyparty",
    "jellyfin",
    "calibre-web",
 }
 COMPOSE_NAME_MAP = {"mariadb": "romm-mariadb", "redis": "romm-redis"}
@@ -274,6 +281,8 @@ def services_for_path(path: str, target: str) -> set[str]:
            # Provisioned datasources and dashboards: git is the source of
            # truth, so a change here must restart the service that reads them.
            _add(services, "grafana")
        elif group == "blackbox":
            _add(services, "blackbox-exporter")
        elif group == "homepage":
            # Homepage is a Dankclaw-owned deployment. Shared homepage files
            # are intentionally not copied during a Dankserver-only run.
@@ -377,7 +386,7 @@ def services_for_path(path: str, target: str) -> set[str]:
            # consumer (prometheus before grafana, loki before alloy).
            if len(parts) >= 4 and parts[2] == "quadlet":
                name = parts[3].removesuffix(".container")
-                if name in {"prometheus", "grafana", "loki", "alloy", "node-exporter", "cadvisor", "alertmanager"}:
+                if name in {"prometheus", "grafana", "loki", "alloy", "node-exporter", "cadvisor", "alertmanager", "blackbox-exporter"}:
                    _add(services, name)
            return services
        if len(parts) >= 2 and parts[1] == "gitea":
@@ -413,6 +422,11 @@ def services_for_path(path: str, target: str) -> set[str]:
                "tautulli": {"tautulli"},
            }
            _add(services, *quadlet_services.get(parts[1], set()))
        if len(parts) >= 2 and parts[1] == "host":
            # dankserver/host/systemd/media-meta-view.mount is the overlay view the
            # media servers bind (scripts/media-view.sh installs it on the host),
            # so editing it restarts jellyfin and plex and nothing else.
            _add(services, "jellyfin", "plex")
        if len(parts) >= 2 and parts[1] == "observability":
            # Dankserver's node-exporter became a Quadlet unit on 2026-09-16
            # (serverAppConfigs-docs/Infrastructure/Observability.md): its
@@ -452,6 +466,21 @@ def services_for_path(path: str, target: str) -> set[str]:
            # change to it redeploying alertmanager is the only path that
            # reaches it on the host.
            _add(services, "alertmanager")
        if name in {"gpu-exporter.py", "gpu-exporter.service"}:
            # The host-process GPU exporter is installed directly by the
            # gpu-exporter deploy on BOTH hosts, so its script and unit select
            # that service (target_services then keeps it only on the host being
            # planned). Editing the collector restarts the running exporter.
            _add(services, "gpu-exporter")
        if name == "media-view.sh":
            # The installer is what creates, mounts and reaps the view both media
            # servers read, so a change to it is a restart of that pair.
            _add(services, "jellyfin", "plex")
        if name == "jellyfin-providers.sh":
            # The deployer runs this before jellyfin restarts, so a change to
            # the installer is a change to the provider state Jellyfin loads at
            # start-up: it selects jellyfin and nothing else.
            _add(services, "jellyfin")
        if name in {"frank-zealot.env", "frank-zealot.service", "serve-frank-zealot.sh"}:
            # llama.cpp frank-zealot (2026-09-16) keeps its unit, env and serve
            # files here; each one selects frank-zealot only. The ik_llama.cpp
@@ -86,7 +86,7 @@ service_is_allowed() {
        dankclaw:open-webui|dankclaw:gitea-postgres|dankclaw:gitea-server|\
        dankclaw:comfyui|dankclaw:hermes|\
        dankclaw:hermes-dashboard|dankclaw:hermes-webui|dankclaw:prometheus|dankclaw:grafana|\
-        dankclaw:loki|dankclaw:alloy|dankclaw:node-exporter|dankclaw:cadvisor|dankclaw:alertmanager|\
+        dankclaw:loki|dankclaw:alloy|dankclaw:node-exporter|dankclaw:cadvisor|dankclaw:gpu-exporter|dankclaw:alertmanager|dankclaw:blackbox-exporter|\
        dankclaw:authentik-postgres|dankclaw:authentik-server|\
        dankclaw:authentik-worker|dankclaw:authentik-outpost|dankclaw:authentik-outpost-ldap|dankclaw:anubis|\
        dankclaw:caddy|dankclaw:homepage|dankclaw:seerr|dankclaw:marreta|\
@@ -97,7 +97,7 @@ service_is_allowed() {
        dankserver:copyparty|dankserver:jellyfin|dankserver:calibre-web|dankserver:calibre-server|dankserver:plex|dankserver:tautulli|\
        dankserver:prowlarr|dankserver:qbittorrent|dankserver:radarr|\
        dankserver:sonarr|dankserver:lidarr|dankserver:readarr|\
-        dankserver:frank-archon|dankserver:node-exporter|dankserver:cadvisor|\
+        dankserver:frank-archon|dankserver:node-exporter|dankserver:cadvisor|dankserver:gpu-exporter|\
        frankbook-pro:gluetun|frankbook-pro:qbittorrent|frankbook-pro:prowlarr|\
        frankbook-pro:radarr|frankbook-pro:sonarr|frankbook-pro:lidarr|\
        frankbook-pro:readarr)
@@ -252,16 +252,24 @@ install_signal_relay() {
    # The relay binds the host's LAN IP (the one address a bridge container
    # can reach as host.containers.internal), so keep the unauthenticated
    # endpoint off the rest of the LAN: admit loopback, the host itself
-    # (its own health probe) and the podman bridge subnet only. Idempotent:
+    # (its own health probe) and the podman bridge subnet only.
-    # -C before -I.
+    # ORDER IS THE GATE: `iptables -I` inserts at position 1, so the old
    # shape (-C/-I x3 ACCEPT, then -C/-I the DROP) landed the blanket DROP
    # ABOVE the ACCEPTs and every probe timed out beside a LISTENing socket
    # (live 2026-10-21: 159 packets on the DROP, 0 on each ACCEPT; the whole
    # dankclaw drain aborted at the alertmanager gate twice before the cause
    # was read off `nft list ruleset`). Rebuild the tail instead of inserting
    # it: delete any blanket DROP wherever it sits, keep the three ACCEPTs
    # at the head (-C before -I, unchanged), then APPEND the DROP so it is
    # last and the ACCEPTs decide first. Converges on poisoned hosts.
    run_privileged iptables -D INPUT -p tcp --dport 18093 -j DROP 2>/dev/null || true
    run_privileged iptables -C INPUT -p tcp --dport 18093 -s 127.0.0.1 -j ACCEPT 2>/dev/null \
        || run_privileged iptables -I INPUT -p tcp --dport 18093 -s 127.0.0.1 -j ACCEPT
    run_privileged iptables -C INPUT -p tcp --dport 18093 -s 192.168.0.143 -j ACCEPT 2>/dev/null \
        || run_privileged iptables -I INPUT -p tcp --dport 18093 -s 192.168.0.143 -j ACCEPT
    run_privileged iptables -C INPUT -p tcp --dport 18093 -s 10.89.0.0/16 -j ACCEPT 2>/dev/null \
        || run_privileged iptables -I INPUT -p tcp --dport 18093 -s 10.89.0.0/16 -j ACCEPT
-    run_privileged iptables -C INPUT -p tcp --dport 18093 -j DROP 2>/dev/null \
+    run_privileged iptables -A INPUT -p tcp --dport 18093 -j DROP
        || run_privileged iptables -I INPUT -p tcp --dport 18093 -j DROP
    # The interpreter needs a moment to bind after restart; a one-shot curl
    # raced the socket on 2026-09-17 and failed a healthy deploy.
    local i
@@ -275,14 +283,81 @@ install_signal_relay() {
    return 1
 }
 # Host-process GPU metrics exporter (amdgpu sysfs + nvidia-smi). Not a quadlet:
 # it runs python3 from the checkout on the host so it can read the DRM/cgroup
 # sysfs and shell out to nvidia-smi. Same two-address bind posture as
 # node-exporter/cadvisor (loopback + this host's LAN), so it is never a wildcard.
 deploy_gpu_exporter() {
    local unit_dir="$HOME/.config/systemd/user" lan dropin i
    case "$TARGET" in
        dankclaw) lan=192.168.0.143 ;;
        dankserver) lan=192.168.0.235 ;;
        *) echo "gpu-exporter: unknown target '$TARGET'" >&2; return 2 ;;
    esac
    mkdir -p "$unit_dir"
    cp scripts/gpu-exporter.service "$unit_dir/gpu-exporter.service"
    dropin="$unit_dir/gpu-exporter.service.d"
    mkdir -p "$dropin"
    printf '[Service]\nEnvironment=GPU_EXPORTER_BINDS=127.0.0.1,%s\n' "$lan" \
        > "$dropin/bind.conf"
    systemctl --user daemon-reload
    systemctl --user enable gpu-exporter.service
    systemctl --user restart gpu-exporter.service
    # The interpreter needs a moment to bind after restart (the same race that
    # once failed a healthy signal-relay deploy).
    for i in $(seq 1 15); do
        curl -fsS -m 5 http://127.0.0.1:9401/metrics >/dev/null && return 0
        sleep 1
    done
    echo "gpu-exporter is not answering on 127.0.0.1:9401/metrics" >&2
    return 1
 }
 copy_loki_config() {
    run_privileged mkdir -p /opt/loki/config
    run_privileged cp shared/loki/loki-config.yaml /opt/loki/config/loki-config.yaml
 }
 copy_alloy_config() {
-    run_privileged mkdir -p /opt/alloy/config
+    # The repo config carries the token GRAFANA_HOST_LABEL; the installed copy
-    run_privileged cp shared/alloy/config.alloy /opt/alloy/config/config.alloy
+    # carries the host's name, so Loki streams say WHICH machine's journal a
    # line came from without a second config file per host.
    # *_CONF_ROOT are test seams: the bats cases execute these copiers for
    # real against a sandbox with a stubbed run_privileged.
    local root="${ALLOY_CONF_ROOT:-/opt/alloy}" tmp
    tmp="$(mktemp)"
    sed "s/GRAFANA_HOST_LABEL/$TARGET/g" shared/alloy/config.alloy > "$tmp"
    if ! grep -q "host = \"$TARGET\"," "$tmp" || grep -q "GRAFANA_HOST_LABEL" "$tmp"; then
        echo "alloy config: host-label substitution failed" >&2
        rm -f "$tmp"
        return 1
    fi
    # Syntax gate: alloy's config language has no schema — a misplaced
    # attribute (e.g. `on_error` outside the json stage) parses only at RUN
    # time, and the unit then crash-loops with a healthy-looking deploy.
    # Validate against the very image that will run it, before installing.
    # Skipped only in the bats sandbox (no image there).
    if [ "$root" = "/opt/alloy" ] || podman image exists localhost/franksdank/alloy:latest 2>/dev/null; then
        if ! podman run --rm -v "$tmp:/c.alloy:ro" \
                --entrypoint /usr/bin/alloy localhost/franksdank/alloy:latest \
                validate /c.alloy; then
            echo "alloy config: validate FAILED against the pinned image" >&2
            rm -f "$tmp"
            return 1
        fi
    fi
    run_privileged mkdir -p "$root/config"
    run_privileged cp "$tmp" "$root/config/config.alloy"
    rm -f "$tmp"
 }
 copy_blackbox_config() {
    # Single-file mount target: the unit binds this exact path into
    # /etc/blackbox_exporter/config.yml. cp (not cp -r) keeps it one file.
    # BLACKBOX_CONF_ROOT is the same test seam.
    local root="${BLACKBOX_CONF_ROOT:-/opt/blackbox}"
    run_privileged mkdir -p "$root"
    run_privileged cp shared/blackbox/blackbox.yml "$root/blackbox.yml"
 }
 copy_grafana_provisioning() {
@@ -291,14 +366,25 @@ copy_grafana_provisioning() {
    # datasource would be invisible to review, lost with the data volume, and
    # unknown to anyone reading the repository. The unit already mounts
    # /opt/grafana/provisioning read-only, so this only has to populate it.
-    run_privileged mkdir -p /opt/grafana/provisioning/datasources \
+    # GRAFANA_PROV_ROOT is the test seam: point it at a sandbox and the whole
-        /opt/grafana/provisioning/dashboards
+    # copy runs unprivileged (the bats case executes this function for real).
    local root="${GRAFANA_PROV_ROOT:-/opt/grafana}"
    run_privileged mkdir -p "$root/provisioning/datasources" \
        "$root/provisioning/dashboards"
    run_privileged cp shared/grafana/provisioning/datasources/prometheus.yml \
-        /opt/grafana/provisioning/datasources/prometheus.yml
+        "$root/provisioning/datasources/prometheus.yml"
    run_privileged cp shared/grafana/provisioning/dashboards/provider.yml \
-        /opt/grafana/provisioning/dashboards/provider.yml
+        "$root/provisioning/dashboards/provider.yml"
-    run_privileged cp shared/grafana/provisioning/dashboards/llm-fleet.json \
+    # EVERY dashboard JSON rides, not a hand-listed subset: the fleet provider
-        /opt/grafana/provisioning/dashboards/llm-fleet.json
+    # turns each file in the directory into a dashboard, so a copier that names
    # files one by one silently drops new boards (measured 2026-10-21: the
    # containers board existed in the repo since its commit and was NEVER
    # provisioned — the panel-census browser check could not find it while the
    # hand-placed older boards happened to be on disk). Same rule as the quadlet
    # tier: the directory is the contract, the copy is generated state.
    for dash in shared/grafana/provisioning/dashboards/*.json; do
        run_privileged cp "$dash" "$root/provisioning/dashboards/$(basename "$dash")"
    done
 }
 render_caddyfile() {
@@ -522,6 +608,9 @@ deploy_observability() {
            ;;
        cadvisor)
            ;;
        blackbox-exporter)
            copy_blackbox_config
            ;;
        *)
            echo "unknown observability service: $service" >&2
            return 2
@@ -536,6 +625,9 @@ deploy_observability() {
        loki) wait_for_health loki http://127.0.0.1:3100/ready 60 ;;
        node-exporter) wait_for_health node-exporter http://127.0.0.1:9100/metrics 60 ;;
        cadvisor) wait_for_health cadvisor http://127.0.0.1:8080/healthz 90 ;;
        # Loopback: the exporter binds 127.0.0.1 + LAN (the gateway address
        # is not bindable from the host netns — see the unit header).
        blackbox-exporter) wait_for_health blackbox-exporter http://127.0.0.1:9115/-/healthy 60 ;;
    esac
 }
@@ -1110,7 +1202,42 @@ deploy_romm_stack() {
    systemctl --user restart "$service.service"
 }
 ensure_media_view() {
    # Both media servers read their library through the SSD-backed overlay view
    # (scripts/media-view.sh, dankserver/host/systemd/media-meta-view.mount): the
    # array below stays read-only and every sidecar Jellyfin writes lands on the
    # NVMe.
    # Raise it BEFORE either container starts — a start against a bare view
    # directory would let a scan clear the library database. The units also carry
    # ConditionPathIsMountPoint, so a missing view skips the start rather than
    # damaging anything. A relayout must be followed by `media-view.sh reap`
    # (see the Media Relayout page), which this is not.
    run_privileged bash scripts/media-view.sh install
    bash scripts/media-view.sh verify
 }
 ensure_provider_keys() {
    # Jellyfin's metadata-provider keys live in the container-owned plugin
    # configuration under the data root, never in git; scripts/jellyfin-providers.sh
    # is what places them, reading the per-service host env file (the fleet
    # convention: grafana, caddy, pihole, copyparty, romm, seerr) that holds
    # TMDB_API_KEY= and TVDB_API_KEY=, mode 600.
    # Missing keys mean a keyless provider — a degraded library, not a broken
    # deploy — so this warns rather than failing the restart. But when the keys
    # ARE placed it must happen before the restart: the plugin and its keys are
    # read at start-up, so a restart first would boot the old state.
    local keys=/home/dank/Containers/jellyfin/provider-keys
    if [ -f "$keys" ]; then
        bash scripts/jellyfin-providers.sh install
    else
        echo "WARN: $keys is absent - provider keys NOT placed"
        echo "WARN: create it (TMDB_API_KEY=, TVDB_API_KEY=, chmod 600) and re-deploy jellyfin"
    fi
 }
 deploy_jellyfin() {
    ensure_media_view
    ensure_provider_keys
    cp "dankserver/jellyfin/quadlet/jellyfin.container" "$quadlet_unit_dir/"
    systemctl --user daemon-reload
    if ! systemctl --user is-active -q jellyfin.service; then
@@ -1198,6 +1325,7 @@ deploy_copyparty() {
 }
 deploy_plex() {
    ensure_media_view
    cp "dankserver/plex/quadlet/plex.container" "$quadlet_unit_dir/"
    systemctl --user daemon-reload
    if ! systemctl --user is-active -q plex.service; then
@@ -1270,11 +1398,17 @@ deploy_authentik() {
    # generated". Boot persistence comes from [Install] WantedBy=default.target
    # inside each .container file, which the generator honors at user-manager
    # start (linger must stay on for the deploy user).
-    systemctl --user start authentik-postgres.service
+    # RESTART, not start: `start` on an already-active unit is a no-op, so an
    # image bump would install the new digest and keep serving the old
    # container while every gate stayed green (measured 2026-10-30: server and
    # worker stayed on the previous release beside an outpost that moved,
    # because only the outpost handler restarts). The generator has already
    # re-rendered the unit from the .container, so a restart is what applies it.
    systemctl --user restart authentik-postgres.service
    wait_container_healthy authentik-postgres 120 || return 1
-    systemctl --user start authentik-server.service
+    systemctl --user restart authentik-server.service
    wait_container_healthy authentik-server 300 || return 1
-    systemctl --user start authentik-worker.service
+    systemctl --user restart authentik-worker.service
 }
 deploy_authentik_outpost() {
@@ -1371,9 +1505,10 @@ deploy_anubis() {
 # conversion (2026-10-11) of the abliterated Qwen3.8-Flash-Next —
 # qwen38-flash-next-v2-abliterated.hgn (66,687,678,432 B, ~62 GiB resident)
 # + the REQUIRED n-gram lookup table + the vision tower — MTP spec decode
-# (HALOGEN_SPEC_ADAPT=0), 2 slots over ONE 401,408-position KV pool (2 × ctx,
+# (HALOGEN_SPEC_ADAPT=0), 2 slots over ONE 401,408-position KV pool at the
-# raised 2026-10-18 so BOTH slots can hold a maximum-length request) at the
+# native ctx 262144 ceiling (raised 2026-10-21 from 200704 so the harness can
-# ctx 200704 ceiling, image input — on the Strix Halo iGPU. It stays a
+# run a 200,000 window with its ~45k fit gap; the pool still carries two
 # 200k conversations at once), image input — on the Strix Halo iGPU. It stays a
 # ROOT unit because the serve script drives `podman run` directly (the frank-archon pattern):
 # relink_unit points /etc/systemd/system/frank-dragoon.service at the repo
 # file, boot persistence is that unit's [Install], and LimitMEMLOCK=infinity
@@ -1389,7 +1524,7 @@ deploy_anubis() {
 # artifact would leave a restart-looping unit and a dark endpoint. The
 # port gate is the same as every cutover since 2026-09-16: nothing
 # starts until 7770 is released.
-LLM_DRAGOON_IMAGE="${LLM_DRAGOON_IMAGE:-ghcr.io/peonist-ai/halogen-flash-server@sha256:0c61bf84ac22308a53f5d1ca6b86806702d7039e5ebc51cae4c66621b92fe04a}"
+LLM_DRAGOON_IMAGE="${LLM_DRAGOON_IMAGE:-ghcr.io/peonist-ai/halogen-flash-server@sha256:e35133b206da3094cc4d70c78800b26235bd9124b9aeb0e029d4a7c153b9cff0}"
 # The payload dir is its own twin because the NPU lane's containment gate below
 # reads it; every artifact path under it stays a full literal as usual.
 LLM_DRAGOON_MODEL_DIR="${LLM_DRAGOON_MODEL_DIR:-/home/dank/models/halogen-v2-abliterated}"
@@ -2003,7 +2138,8 @@ deploy_service() {
        plex) deploy_plex ;;
        tautulli) deploy_tautulli ;;
        alloy) deploy_alloy ;;
-        prometheus|grafana|loki|node-exporter|cadvisor|alertmanager) deploy_observability "$service" ;;
+        prometheus|grafana|loki|node-exporter|cadvisor|alertmanager|blackbox-exporter) deploy_observability "$service" ;;
        gpu-exporter) deploy_gpu_exporter ;;
        comfyui) deploy_comfyui ;;
        *) deploy_compose_service "$service" ;;
    esac
@@ -133,13 +133,15 @@ fi
 valid_service() {
    local service="$1"
    case "$TARGET:$service" in
        # dankclaw:searxng-network is client-only: a network quadlet with no
        # remote dispatch handler, so the remote gate does not (must not) list it.
        dankclaw:caddy|dankclaw:gluetun|dankclaw:searxng-network|dankclaw:searxng-valkey|\
        dankclaw:searxng|dankclaw:marreta|dankclaw:pihole|dankclaw:open-webui|dankclaw:gitea-postgres|\
        dankclaw:gitea-server|dankclaw:homepage|\
        dankclaw:comfyui|dankclaw:hermes|dankclaw:hermes-dashboard|\
        dankclaw:hermes-webui|dankclaw:prometheus|dankclaw:grafana|dankclaw:loki|dankclaw:alloy|\
-        dankclaw:alertmanager|\
+        dankclaw:alertmanager|dankclaw:blackbox-exporter|\
-        dankclaw:node-exporter|dankclaw:authentik-postgres|dankclaw:authentik-server|\
+        dankclaw:node-exporter|dankclaw:cadvisor|dankclaw:gpu-exporter|dankclaw:authentik-postgres|dankclaw:authentik-server|\
        dankclaw:authentik-worker|dankclaw:authentik-outpost|dankclaw:authentik-outpost-ldap|dankclaw:anubis|dankclaw:seerr|\
        dankclaw:open-notebook|\
        dankclaw:frank-tts-engine|dankclaw:frank-tts|\
@@ -147,8 +149,8 @@ valid_service() {
        dankserver:romm-mariadb|dankserver:romm-redis|dankserver:romm|\
        dankserver:copyparty|dankserver:jellyfin|dankserver:calibre-web|dankserver:calibre-server|dankserver:plex|dankserver:tautulli|\
        dankserver:qbittorrent|dankserver:radarr|dankserver:sonarr|\
-        dankserver:lidarr|dankserver:readarr|dankserver:node-exporter|\
+        dankserver:lidarr|dankserver:readarr|dankserver:node-exporter|dankserver:cadvisor|dankserver:gpu-exporter|\
-        dankserver:frank-archon|\
+        dankserver:prowlarr|dankserver:frank-archon|\
        frankbook-pro:gluetun|frankbook-pro:qbittorrent|frankbook-pro:prowlarr|\
        frankbook-pro:radarr|frankbook-pro:sonarr|frankbook-pro:lidarr|\
        frankbook-pro:readarr)
@@ -20,9 +20,16 @@ of the Gram is its down-input energy. Rank halves: 0-255, 256-511
 Output: the cast-rule lines (recipe format) on stdout, a report on stderr.
 Usage: derive-protected-experts.py <calib dir> [--min-share 0.03]
-       [--min-route-mult 3.0]
+       [--min-route-mult 3.0] [--target-count 0]
       --target-count N fills the byte bill: the bars pick the base set exactly as
       the bar-only pass does, then the energy ranking admits experts one at a time
       (each with its parity partner) until the count lands. The energy bar NEVER
       relaxes, so asking for 48 bf16 experts never quietly buys 80; the ratio bar
       walks down only when the ranking cannot produce that many candidates.
       0 takes the bars as given.
 """
 import argparse
 import collections
 import glob
 import os
 import re
@@ -32,6 +39,7 @@ import sys
 import numpy as np
 NAME_RE = re.compile(r"^gram\.r(\d)\.blk\.(\d+)\.ffn_down_exps\.(\d+)\.bin$")
 CAST_RE = re.compile(r"^blk\.(\d+)\.ffn_\*_exps\.(\d+)\.weight")
 def read_gram(path):
@@ -49,27 +57,8 @@ def read_gram(path):
    return k, rank, int(rows), tr
-def main():
+def select(data, min_share, min_route_mult):
-    ap = argparse.ArgumentParser()
+    """(cast lines, report lines, count) for one pair of bars."""
    ap.add_argument("calib_dir")
    ap.add_argument("--min-share", type=float, default=0.03)
    ap.add_argument("--min-route-mult", type=float, default=3.0)
    a = ap.parse_args()
    # layer -> global expert -> (rank, rows, energy)
    data = {}
    for path in sorted(glob.glob(os.path.join(
            a.calib_dir, "gram.r*.ffn_down_exps.*.bin"))):
        m = NAME_RE.match(os.path.basename(path))
        if not m:
            continue
        rank, layer, expert = int(m.group(1)), int(m.group(2)), int(m.group(3))
        k, _, rows, tr = read_gram(path)
        data.setdefault(layer, {})[expert] = (rank, rows, tr)
    if not data:
        raise SystemExit("no per-expert down grams found")
    lines, report = [], []
    total = 0
    for layer in sorted(data):
@@ -83,7 +72,7 @@ def main():
        for e, (rank, rows, tr) in experts.items():
            es, rs = tr / tot_e, rows / max(tot_r, 1)
            rec = (e, rank, es, rs)
-            if es >= a.min_share and es >= a.min_route_mult * rs:
+            if es >= min_share and es >= min_route_mult * rs:
                qual.append(rec)
            elif es > 0:
                pool.append(rec)
@@ -107,10 +96,148 @@ def main():
                          f"energy {es*100:5.2f}%, routing {rs*100:5.3f}%, "
                          f"ratio {es/max(rs,1e-12):6.1f}")
        total += len(chosen)
    return lines, report, total
 def parity_repair(chosen, by_half, report, only=None):
    """Add the next-best candidate to every half — or to `only` — holding an odd count.
    The four-bit GEMM takes each half's remainder as two equal tables, so an odd count
    of bf16 experts on a half is not a style problem: the two tables would not match.
    `by_half` is energy-descending, so the first unchosen candidate is the next most
    energetic, which is the expert worth keeping. Returns how many were added.
    """
    added = 0
    for key in ([only] if only is not None else sorted(by_half)):
        layer, half = key
        n = sum(1 for (l, x) in chosen if l == layer and x // 256 == half)
        if n % 2 == 0:
            continue
        for e, es, rs in by_half.get(key, []):
            if (layer, e) not in chosen:
                chosen[(layer, e)] = (es, rs)
                added += 1
                report.append(f"blk.{layer}: half {half} odd -> added e{e} "
                              f"(energy {es*100:.2f}%)")
                break
        else:
            report.append(f"blk.{layer}: half {half} holds {n} and no candidate is "
                          f"left to make it even")
    return added
 def select_to_count(data, min_share, min_route_mult, target):
    """(lines, report, total, cutoff) for the set that LANDS the count.
    The bars decide the base set; the count is then filled from the energy ranking,
    one expert at a time, each addition taking its half's next-best candidate with it
    so every rank's half stays even. Only the RATIO bar walks down to find candidates
    — it is the law that says an expert is disproportionate and not merely busy, and
    the caller's byte bill has to be filled — while the ENERGY bar holds, because that
    bar is what the extra GiB of bf16 is supposed to buy. `cutoff` is the energy share
    the set bottoms out at: the bar this campaign actually converted with.
    """
    report = []
    shares, ranked = {}, []
    for layer in sorted(data):
        experts = data[layer]
        tot_e = sum(v[2] for v in experts.values())
        tot_r = sum(v[1] for v in experts.values())
        if tot_e <= 0 or tot_r <= 0:
            report.append(f"blk.{layer}: empty recording, skipped")
            continue
        for e, (_rank, rows, tr) in experts.items():
            es, rs = tr / tot_e, rows / max(tot_r, 1)
            if es <= 0:
                continue
            shares[(layer, e)] = (es, rs)
            ranked.append((es, layer, e, rs))
    ranked.sort(key=lambda r: (-r[0], r[1], r[2]))
    by_half = {}
    for es, layer, e, rs in ranked:
        by_half.setdefault((layer, e // 256), []).append((e, es, rs))
    chosen = {k: v for k, v in shares.items()
              if v[0] >= min_share and v[0] >= min_route_mult * v[1]}
    total = len(chosen)
    total += parity_repair(chosen, by_half, report)
    mult = min_route_mult
    while True:
        pool = [(es, l, e, rs) for es, l, e, rs in ranked
                if es >= mult * rs and (l, e) not in chosen]
        if len(pool) + total >= target or mult <= 0.0:
            break
        mult -= 0.5
        report.append(f"ratio bar {mult + 0.5:g} leaves {len(pool)} fill candidates "
                      f"for {target - total} slots; relaxing toward 0")
    for es, layer, e, rs in pool:
        if total >= target:
            break
        if (layer, e) in chosen:
            continue
        chosen[(layer, e)] = (es, rs)
        total += 1
        report.append(f"blk.{layer}: count fill -> added e{e} (energy {es*100:.2f}%)")
        total += parity_repair(chosen, by_half, report, only=(layer, e // 256))
    lines, cutoff = [], None
    for (layer, e) in sorted(chosen):
        es, rs = chosen[(layer, e)]
        cutoff = es if cutoff is None else min(cutoff, es)
        lines.append(f"blk.{layer}.ffn_*_exps.{e}.weight         cast dtype=bf16")
    return lines, report, total, cutoff
 def main():
    ap = argparse.ArgumentParser()
    ap.add_argument("calib_dir")
    ap.add_argument("--min-share", type=float, default=0.03)
    ap.add_argument("--min-route-mult", type=float, default=3.0)
    ap.add_argument("--target-count", type=int, default=0)
    a = ap.parse_args()
    # layer -> global expert -> (rank, rows, energy)
    data = {}
    for path in sorted(glob.glob(os.path.join(
            a.calib_dir, "gram.r*.ffn_down_exps.*.bin"))):
        m = NAME_RE.match(os.path.basename(path))
        if not m:
            continue
        rank, layer, expert = int(m.group(1)), int(m.group(2)), int(m.group(3))
        k, _, rows, tr = read_gram(path)
        data.setdefault(layer, {})[expert] = (rank, rows, tr)
    if not data:
        raise SystemExit("no per-expert down grams found")
    if a.target_count > 0:
        lines, report, total, cutoff = select_to_count(
            data, a.min_share, a.min_route_mult, a.target_count)
        bar = f"energy cutoff {cutoff:.3g}" if cutoff is not None else "no candidates"
    else:
        lines, report, total = select(data, a.min_share, a.min_route_mult)
        bar = f"min-share {a.min_share:.3g}"
    # The emitted lines ARE the contract with the four-bit GEMM: every rank's half of
    # every layer must hold an even number of bf16 experts. Judged on the output and
    # not on the internal sets, so a slip in either pass is caught here rather than at
    # engine load time.
    halves = collections.Counter()
    for rule in lines:
        m = CAST_RE.match(rule)
        if m:
            halves[(int(m.group(1)), int(m.group(2)) // 256)] += 1
    odd = {key: n for key, n in halves.items() if n % 2}
    if odd:
        print("PARITY LAW BROKEN: " + ", ".join(
            f"blk.{lay} half {half} holds {n} protected experts"
            for (lay, half), n in sorted(odd.items())), file=sys.stderr)
        raise SystemExit(2)
    print("\n".join(lines))
    print("\n".join(report), file=sys.stderr)
-    print(f"total protected: {total}", file=sys.stderr)
+    print(f"total protected: {total} (target {a.target_count or 'none'}, {bar}, "
          f"min-route-mult {a.min_route_mult:g})", file=sys.stderr)
 if __name__ == "__main__":
@@ -8,8 +8,10 @@
 # (/opt/halogen/npu/models.txt inside the image) and refuses to load a model
 # whose files do not match.
 #
-# The manifest below IS that catalog block (dumped from the pinned 0.16.2
+# The manifest below IS that catalog block (dumped from the pinned 0.18.20
-# image on 2026-10-19), so the files land in the layout the entrypoint
+# image on 2026-10-30; the 0.16.2 block this replaced named different device
 # programs — 10 `u*.elf` and a 5,086,496 B consts.hnpb — so a version bump
 # re-fetches the set or the entrypoint refuses the boot), so the files land in the layout the entrypoint
 # expects — <dir>/<model-id>/devices/*, <dir>/<model-id>/<model-id>.hnpw and
 # <dir>/<model-id>/tokenizer/tokenizer.json — and the endpoint can keep
 # /models read-only and open no outbound connection at boot (HALOGEN_DOWNLOAD
@@ -57,21 +59,25 @@ manifest() {
    case "$1" in
      qwen3-embedding-0.6b)
        NPU_REPO="peonist-ai/halogen-npu-qwen3-embedding-0.6b"
-        NPU_REV="4960fd9d4423dd15373ad21c83db48dbdb3b2798"
+        NPU_REV="8fd44c9f18861618a509bea75c51ff8a37e3bd5d"
        FILES=(
        "devices/NOTICE.md 3248 0f267b3d2523ddf14997ee58660a492a9eaf10ccf6b08bf19dfdc85b9a478b91"
-        "devices/consts.hnpb 5086496 ae430e41545a0b1012938cda373b0c450364d3be631111cca3a8c43679f93336"
+        "devices/consts.hnpb 8495512 5af7c1f2f24d45a9054872bba882461cf7f6ffda832e785b700643d6467ad88b"
-        "devices/devices.hnpm 24047 4e9bca9d95d084fb89ce5b99f579a00666c8ca1efae2613c53e3a4878cc9f174"
+        "devices/devices.hnpm 35465 85d870f6b900de1829659955f0dc701d57984580f5e5ec251add5d6969f03e3d"
-        "devices/u0.elf 4931968 b38f165886cb037b7d3951d9cb878473eaba58a432acd45416ff85fd88aa4e46"
+        "devices/u0.elf 2975008 f74df12eb8c79e4e90cbedcc1dd8f142219772d400e3a142fde8a95378b105eb"
-        "devices/u1.elf 5456000 b1811feee851b8af833b37721da92099d89772110815bbbe2c3fa1c276f0ff42"
+        "devices/u1.elf 3266848 ac5a3f89f934951b315b62ab53e1d0a74dbe6396134192641c405d08da332754"
-        "devices/u2.elf 6505552 406ea9a95b470628a7382e376f4a72cdf4d7aa4c59b4ddf0a4bdd139d63d6ff2"
+        "devices/u2.elf 4015328 5e4f0d3342f262816289866d7e8c966f56a9f938c2e12e7341718cc7312aa480"
-        "devices/u3.elf 6817216 bc205d51b523d770151a0c61968f7c70bf624412827580bb6fe0aaf4650dc2ee"
+        "devices/u3.elf 4326976 94cf5d269fa286d91c26bf4654cedf101c047d23c4495b74c9c129cdc33b210f"
-        "devices/u4.elf 7444032 a5a91338a2603acdcd5c11df532033fda2d3f5479ea44ae87f98fafbdff9a62d"
+        "devices/u4.elf 4953792 cbd2796050d4c675f5cb3da117111087d510173e26f262f1b36aab66dd49dc91"
-        "devices/u5.elf 7432416 6c2bf41835569e0eba25ded7c05c7264ae69c6700bdeb4791057d884f2a4c126"
+        "devices/u5.elf 4940112 7bf6a4470546bb888da02af395c07d92a24fb6d43aef27eb2151b3a9eda19e4d"
-        "devices/u6.elf 4941232 919d0038ca088eafddf195f3466ff38db27d3635be1cffa5d4d4b5140902a68e"
+        "devices/u6.elf 6174544 10b9d50b635993e1f9c23775c0b94901d922c1e433cd10910e9f97174235f5ac"
-        "devices/u7.elf 5488304 d904cf8afae9428dd167960549e18cce5441745f18c62f8c50975dffc077d3c2"
+        "devices/u7.elf 2983776 e0b06063b7379d05c85b3f435b819d96fda1223d8025d6d15958854b5711ace9"
-        "devices/u8.elf 4941232 db3aef69b17825854ee0dd2df475125a3f5c872f95c686793d6af1ee83596fa7"
+        "devices/u8.elf 3298656 3bad204f782a68b97e27c1172530b13d7fbb0b495fa95f9504530a1048560644"
-        "devices/u9.elf 5488304 d182727e551084db8fb9600441d8aa373c7f09a3d267ae0a5e921ef92326db1c"
+        "devices/u9.elf 2983776 875e313280358ce116e2a51a29a2c42794b5a9fadb3d48db702c90d18d838c2c"
        "devices/u10.elf 3298656 ef2e0f399f149a5476955bcf233c6fa2e896fcef1f86ecb0d8113c952c41a8b2"
        "devices/u11.elf 3298656 68a2519eee2d6a61ccc4ca2007e3186d44c9eb4e81162d742fe2f1756ec5a528"
        "devices/u12.elf 3298656 2bdc6abfa58cac1d5795a3ef52fe8a9ab4e7c2a4792822d615499a913401f728"
        "devices/u13.elf 3298656 4a3ce14f7ba15987b9bcbb4a010083e8aef68cd87c20cbe43403a98263179fe2"
        "qwen3-embedding-0.6b.hnpw 813032248 b9f9901429df7b46ec17153aa3899f738adbea8e4356080da0c6e6586fb750d1"
        "tokenizer/tokenizer.json 11423705 def76fb086971c7867b829c23a26261e38d9d74e02139253b38aeb9df8b4b50a"
        ) ;;
@@ -99,7 +105,18 @@ for MODEL in ${MODELS//,/ }; do
                # 62 GiB fetch hit this on 2026-10-18).
                url=$(curl -sfIL -o /dev/null -w '%{url_effective}' "$HF/$NPU_REPO/resolve/$NPU_REV/$f") \
                    || { echo "cannot resolve $NPU_REPO/$f" >&2; exit 1; }
-                curl -fL --retry 5 -C - -o "$path" "$url"
+                # Resume ONLY into a genuinely partial file. A staged file of a
                # DIFFERENT size is an artifact of another revision (a version
                # bump re-cuts the device programs), and `-C -` against it
                # appends the tail to the old bytes: the result is the right
                # BYTE COUNT with the wrong content, which reads as a corrupt
                # program, not as a fetch fault (measured 2026-10-30 on
                # consts.hnpb). Fetch those from zero.
                resume=()
                if [ -f "$path" ] && [ "$(stat -c %s "$path")" -lt "$want" ]; then
                    resume=(-C -)
                fi
                curl -fL --retry 5 "${resume[@]+"${resume[@]}"}" -o "$path" "$url"
            fi
        fi
        [ -f "$path" ] || { echo "MISSING $path (run without --verify-only)" >&2; exit 1; }
@@ -0,0 +1,249 @@
 #!/usr/bin/env bash
 # download-qwen38-bf16.sh — RUNS ON DANKSERVER. Fetch and verify the vendor
 # BF16 checkpoint Qwen/Qwen3.8-Flash-Next: the source the five-bit campaign
 # converts, and the source the KLD reference is recorded from.
 #
 # Why the base and not a finetune: the campaign converts to compare against the
 # publisher's own iq4r container, and that container was built from this
 # checkpoint. Base-in, base-out; anything else confounds the gate. (The
 # predecessor campaign ran ukisai/Swift-1.5-Qwen3.8-Flash-Next-BF16, whose key
 # set is identical to this repo's but whose weights are not — none of its
 # calibration grams, KLD reference or protected list carries over.)
 #
 # Why re-fetch instead of reusing: every BF16 copy of this family was destroyed
 # in the 2026-10-15 storage purge — only the 113.6 GiB served .rad survived.
 # 335.3 GiB of safetensors, 131 shards, ungated (the predecessor campaign
 # verified the key set against this repo, so no gate letter is needed).
 #
 # Verification is four gates deep, because the failure this repo actually lived
 # through is a fetch that reported success: `curl -o FILE URL` exits 0 on an
 # HTTP 401/403 and writes the error body as the artifact — three shards of
 # identical tiny size and a cheerful DONE in 20 seconds for 104 GiB.
 #
 #   1. shard census: the file count must equal the expected shard count, so a
 #      partially fetched fragment tree (the .partNofM trap) cannot pass.
 #   2. per-shard completeness, from the shard's OWN safetensors header: the
 #      header declares where the tensor data ends, so every shard's true size is
 #      known offline. A torn tail is named here, by file, instead of surfacing
 #      as a load error five hours later.
 #   3. index coverage: every shard model.safetensors.index.json references
 #      exists and is named.
 #   4. total bytes against the number the Hub declared, then, unless --no-hub,
 #      every file's sha256 against the Hub's LFS record — including the shards
 #      already on disk, so a previous partial download is never trusted.
 #
 # The token never enters a command line (the 2026-09-16 leak law): it is read
 # into HF_TOKEN in the environment, where a local process listing is the only
 # reader. This repo is ungated, so the fetch simply runs without one when no
 # token file exists. scripts/hfget.sh is curl+jq only — no Python, no venv, no
 # hf_transfer — and it drops the pickle/ONNX weight duplicates by itself.
 #
 # usage: download-qwen38-bf16.sh [--fetch] [--no-hub] [--dir DIR]
 #                                [--shards N] [--bytes B] [--repo ID]
 #   --fetch    download first (resumable: re-run to continue), then verify
 #   --no-hub   skip the Hub-side sha256 sweep; the other gates always run
 # exit: 0 verified, 1 verification failed, 2 environment / usage error
 set -euo pipefail
 REPO="${FRANK_QWEN38_HF_REPO:-Qwen/Qwen3.8-Flash-Next}"
 LOCAL_DIR="${FRANK_QWEN38_CKPT:-/home/dank/models/Qwen3.8-Flash-Next}"
 # Hub census of 2026-10-27: 144 files, 131 of them .safetensors, together
 # 360,023,351,514 bytes; the shards alone are 360,000,192,888. This gate counts
 # shards only, so a config file edited on the Hub cannot fail it.
 EXPECTED_SHARDS="${FRANK_QWEN38_SHARDS:-131}"
 EXPECTED_BYTES="${FRANK_QWEN38_BYTES:-360000192888}"
 MIN_FREE_MIB="${FRANK_QWEN38_MIN_FREE_MIB:-350000}"
 FETCH=0
 HUB=1
 while [ $# -gt 0 ]; do
    case "$1" in
        --fetch) FETCH=1 ;;
        --no-hub) HUB=0 ;;
        --dir)    LOCAL_DIR="$2"; shift ;;
        --shards) EXPECTED_SHARDS="$2"; shift ;;
        --bytes)  EXPECTED_BYTES="$2"; shift ;;
        --repo)   REPO="$2"; shift ;;
        *) echo "usage: ${0##*/} [--fetch] [--no-hub] [--dir DIR] [--shards N] [--bytes B] [--repo ID]" >&2
           exit 2 ;;
    esac
    shift
 done
 SELF_DIR="$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)"
 FAILS=0
 note() { echo "$@"; FAILS=$((FAILS + 1)); }
 if [ "$FETCH" = "1" ]; then
    # Every fetch gate is a hard stop: skipping the download silently and then
    # verifying a half tree is how a campaign ships a stale payload.
    HFGET="$SELF_DIR/hfget.sh"
    if [ ! -f "$HFGET" ]; then
        echo "REFUSED: the fetch helper is not here: $HFGET" >&2
        exit 2
    fi
    if ! command -v jq >/dev/null 2>&1; then
        echo "REFUSED: hfget.sh needs jq on PATH" >&2
        exit 2
    fi
    mkdir -p "$LOCAL_DIR"
    # The campaign's peak-disk law starts at the fetch: the checkpoint and the
    # reference container built from it are ~335 GiB each, and a tree that does
    # not fit is discovered hours into a transfer, when it costs the most.
    free_mib=$(df -m --output=avail "$LOCAL_DIR" | tail -1 | tr -dc '0-9')
    if [ "${free_mib:-0}" -lt "$MIN_FREE_MIB" ]; then
        echo "REFUSED: $free_mib MiB free at $LOCAL_DIR; the fetch wants $MIN_FREE_MIB MiB free" >&2
        echo "         (retire the peak-disk predecessors first — nothing is downloaded, so" >&2
        echo "         nothing is left half-written)" >&2
        exit 2
    fi
    echo "fetch: $REPO -> $LOCAL_DIR ($free_mib MiB free)"
    if [ -z "${HF_TOKEN:-}" ]; then
        CREDSTORE="$HOME/.config/huggingface/credentials/https://huggingface.co"
        if [ -f "$CREDSTORE" ]; then
            HF_TOKEN="$(awk '/^ OauthToken: / {print $2; exit}' "$CREDSTORE")"
        elif [ -f "$HOME/.cache/huggingface/token" ]; then
            # The legacy single-line token file the hub library writes.
            HF_TOKEN="$(tr -d '[:space:]' < "$HOME/.cache/huggingface/token")"
        elif [ -f "$HOME/.hf_token" ]; then
            HF_TOKEN="$(tr -d '[:space:]' < "$HOME/.hf_token")"
        fi
    fi
    if [ -n "${HF_TOKEN:-}" ]; then
        export HF_TOKEN
    fi
    bash "$HFGET" "$REPO" "$LOCAL_DIR" main
    unset HF_TOKEN
 fi
 if [ ! -d "$LOCAL_DIR" ]; then
    echo "REFUSED: $LOCAL_DIR does not exist (run with --fetch, or pass --dir)" >&2
    exit 1
 fi
 INDEX="$LOCAL_DIR/model.safetensors.index.json"
 if [ ! -f "$INDEX" ]; then
    echo "REFUSED: $INDEX is missing — the checkpoint tree is not addressable without it" >&2
    exit 1
 fi
 echo "== gate 1: shard census ($LOCAL_DIR)"
 SHARDS=$(find "$LOCAL_DIR" -maxdepth 1 -type f -name '*.safetensors' | wc -l)
 if [ "$SHARDS" != "$EXPECTED_SHARDS" ]; then
    note "FAIL SHARD_COUNT: $SHARDS safetensors files on disk, expected $EXPECTED_SHARDS"
 else
    echo "  ok: $SHARDS shards"
 fi
 echo "== gate 2: every shard is whole, judged by its own header"
 # A safetensors file is [u64 header length][JSON header][tensor data]. The JSON
 # declares where the data ends, so the file's true size is computable offline:
 # 8 + header + the largest end offset. Truncation and error bodies both fall
 # here, and each is reported with the file that caused it.
 while IFS= read -r f; do
    state=$(python3 - "$f" <<'PY'
 import json, os, struct, sys
 path = sys.argv[1]
 size = os.path.getsize(path)
 try:
    with open(path, "rb") as fh:
        (hlen,) = struct.unpack("<Q", fh.read(8))
        if hlen == 0 or hlen > size:
            print("NOT_SAFETENSORS")
            raise SystemExit
        header = json.loads(fh.read(hlen))
 except Exception:
    print("NOT_SAFETENSORS")
    raise SystemExit
 if not header:
    print("NOT_SAFETENSORS")
    raise SystemExit
 end = max((v["data_offsets"][1] for v in header.values()
           if isinstance(v, dict) and "data_offsets" in v), default=0)
 want = 8 + hlen + end
 print("OK" if want == size else f"TRUNCATED want={want} have={size}")
 PY
 )
    case "$state" in
        OK) ;;
        NOT_SAFETENSORS*) note "FAIL NOT_SAFETENSORS: $(basename "$f") is not a safetensors file (an error body written as an artifact?)" ;;
        TRUNCATED*) note "FAIL TRUNCATED: $(basename "$f") $state" ;;
    esac
 done < <(find "$LOCAL_DIR" -maxdepth 1 -type f -name '*.safetensors' | sort)
 if [ "$FAILS" -eq 0 ]; then
    echo "  ok: every shard's data extent matches its size on disk"
 fi
 echo "== gate 3: the index's shards are all present"
 INDEX_SHARDS=$(python3 - "$INDEX" <<'PY'
 import json, sys
 map_ = json.load(open(sys.argv[1]))["weight_map"]
 print("\n".join(sorted(set(map_.values()))))
 PY
 )
 while IFS= read -r name; do
    [ -n "$name" ] || continue
    if [ ! -f "$LOCAL_DIR/$name" ]; then
        note "FAIL MISSING: $name is named by the index and is not on disk"
    fi
 done <<< "$INDEX_SHARDS"
 if [ "$FAILS" -eq 0 ]; then
    echo "  ok: $(printf '%s\n' "$INDEX_SHARDS" | grep -c .) referenced shards on disk"
 fi
 echo "== gate 4: total bytes vs the Hub's declaration"
 TOTAL=$(find "$LOCAL_DIR" -maxdepth 1 -type f -name '*.safetensors' -printf '%s\n' | awk '{s+=$1} END{print s+0}')
 if [ "$TOTAL" != "$EXPECTED_BYTES" ]; then
    note "FAIL TOTAL_BYTES: shards total $TOTAL bytes, expected $EXPECTED_BYTES"
 else
    echo "  ok: $TOTAL bytes"
 fi
 if [ "$HUB" = "1" ]; then
    echo "== gate 5: every file's sha256 vs the Hub's LFS record"
    # size = file size; lfs.sha256 = sha256 of the file itself (LFS or not).
    TMPD="$(mktemp -d)"
    trap 'rm -rf "$TMPD"' EXIT
    MANIFEST="$TMPD/tree.json"
    PLAN="$TMPD/plan"
    curl -sf "https://huggingface.co/api/models/$REPO/tree/main?recursive=true&expand=true" \
        -o "$MANIFEST" || { echo "FAIL: the Hub API call failed — refusing to report success" >&2; exit 1; }
    python3 - "$MANIFEST" > "$PLAN" <<'PY'
 import json, sys
 for e in json.load(open(sys.argv[1])):
    if e.get("type") != "file":
        continue
    sha = (e.get("lfs") or {}).get("sha256") or e.get("sha") or ""
    if sha:
        print(e["path"], e["size"], sha)
 PY
    BAD=0
    while read -r rel size sha; do
        [ -n "${rel:-}" ] || continue
        if [ ! -f "$LOCAL_DIR/$rel" ]; then
            note "FAIL MISSING: $rel is on the Hub and not on disk"
            continue
        fi
        have=$(stat -c %s "$LOCAL_DIR/$rel")
        if [ "$have" != "$size" ]; then
            note "FAIL SIZE: $rel is $have bytes, Hub says $size"
            continue
        fi
        got=$(sha256sum "$LOCAL_DIR/$rel" | awk '{print $1}')
        if [ "$got" != "$sha" ]; then
            note "FAIL SHA256: $rel got $got want $sha"
            BAD=$((BAD + 1))
        fi
    done < "$PLAN"
    if [ "$BAD" -eq 0 ] && [ "$FAILS" -eq 0 ]; then
        echo "  ok: sha256 matches for every file"
    fi
    rm -rf "$TMPD"
    trap - EXIT
 fi
 if [ "$FAILS" -gt 0 ]; then
    echo "VERIFY_FAILED: $FAILS problem(s) in $LOCAL_DIR" >&2
    exit 1
 fi
 echo "VERIFY_OK: $SHARDS shards, $TOTAL bytes of safetensors in $LOCAL_DIR"
 echo "NEXT: build the BF16 reference container with scripts/build-qwen38-bf16-rad.sh"
@@ -74,6 +74,13 @@ FRANK_ARCHON_PLACEMENT=expert_tiered
 # residency -- ~3.2 GiB a card of experts now lives in host RAM and is read over the
 # x8 link when the router picks it. The ledger's coverage line is the referee, not
 # this file: if vram_kv comes back under 700,000 tokens, raise this 1024 and redeploy.
 # 2026-10-29 A/B, measured on both payloads and folded back the same day: at 66,560 the
 # endpoint printed the identical `elastic 26.73 GiB / experts 20.72 / kv cache 5.64 /
 # vram_weights 24.30` it had printed at 49,152, opening the bigger pool with `0 B used`;
 # the campaign rig (bf16, same step) moved `vram_kv 269.99 -> 271.25 MiB`, a rounding
 # step — its window is fixed by its own n-gram table (95.37 GiB) and expert plane, not
 # by this pool, so the refused calibration documents were fixed by cutting them to the
 # window (scripts/rechunk-kld-corpus.py), not by widening the pool.
 FRANK_ARCHON_HOST_POOL_MIB=49152
 # VRAM left unclaimed per card (publisher default 96 MiB); the engine's settle
 # floor is built from it.
@@ -137,7 +144,8 @@ FRANK_ARCHON_NGRAM_PLACEMENT=ram
 # itself before refusing by name, so recovery is a boot, not a visit.
 # Floor = 2 x HOST_POOL + 48,824 (static weights + arena) + PREFIX_CACHE + MNBT + the
 # auto checkpoint store (MAX_SEQS x 10 slots of 111.98 MiB), per the bats ledger test;
-# at the 49,152 pool with 4 slots that computes to 161,848 MiB.
+# at the 49,152 pool with 4 slots that computes to 161,848 MiB (196,664 at the
 # 66,560 pool the 2026-10-29 A/B rejected).
 FRANK_ARCHON_TTM_MIN_MIB=162000
 FRANK_ARCHON_TTM_TARGET_MIB=204800
 # Escape hatch for experiments. Unset -> no extra argv; the serve script expands
@@ -10,8 +10,11 @@
 # The .hgn format is the lock: it loads ONLY in the flash-server line, so the
 # engine moves with the payload, never ahead of it.
 # Image digest-pinned (a tag pull silently re-points; the digest is what was
-# qualified — 0.16.2 is the version the model card names as its quickstart pin).
+# qualified). 0.18.20 since 2026-10-30 (was 0.16.2, the model card's quickstart
-FRANK_DRAGOON_IMAGE=ghcr.io/peonist-ai/halogen-flash-server@sha256:0c61bf84ac22308a53f5d1ca6b86806702d7039e5ebc51cae4c66621b92fe04a
+# pin) — the vendor has moved two minor versions and the payload lock is what
 # the deploy's health gate plus the boot ledger decide; rollback is the
 # previous digest, 0c61bf84…, on this same line.
 FRANK_DRAGOON_IMAGE=ghcr.io/peonist-ai/halogen-flash-server@sha256:e35133b206da3094cc4d70c78800b26235bd9124b9aeb0e029d4a7c153b9cff0
 # Container identity kept from the 27B era: the unit's ExecStop names it.
 FRANK_DRAGOON_CONTAINER=frank-dragoon-halogen
 # Predecessor identity the serve script clears if it is still holding the GPUs.
@@ -49,14 +52,21 @@ FRANK_DRAGOON_TOKENIZER_FILES="tokenizer.json chat_template.jinja tokenizer_conf
 # stays at the image default, which is what was qualified) ------------------
 # Served id. OpenWebUI api_configs model_ids for the :7770 backend MUST name it.
 HALOGEN_MODEL_ID=frank-dragoon
-# PER-REQUEST ceiling: a longer prompt is a 400 naming the limit, never a
+# PER-REQUEST ceiling, set to the checkpoint's NATIVE window (262,144 — a
-# truncation. The harness window must sit ~45k UNDER this (the 400-zone law on
+# longer prompt is a 400 naming the limit, never a truncation; stretching
-# the page) — a window equal to the ceiling makes every long request a 400.
+# past native would need HALOGEN_ROPE_YARN and is not on plan). The harness
-HALOGEN_CTX=200704
+# `contextWindow` must sit ~45k under THIS number (the 400-zone law on the
-# ONE pool behind both slots, sized so BOTH can carry a maximum-length request
+# page): with the window at 200,000 the gap is 62,144, which absorbs the
-# at once. Pool positions buy concurrency; the per-request window never moves
+# chars/4 under-count. A window at or near the ceiling makes every long
-# with them. Legal band: ctx <= pool <= slots * ctx (the seam enforces it), and
+# request a guaranteed 400.
-# RAM the pool takes is RAM the n-gram table's page cache loses.
+HALOGEN_CTX=262144
 # ONE pool behind both slots. The harness window is 200,000, so 401,408
 # still carries TWO maximum harness-length conversations at once
 # (2 x 200,000 + one 45,056 budget <= pool + cap headroom); two literal
 # ceiling-length requests would queue instead — buying those means raising
 # the pool, and RAM the pool takes is RAM the n-gram table's page cache
 # loses. Pool positions buy concurrency; the per-request window never moves
 # with them. Legal band: ctx <= pool <= slots * ctx (the seam enforces it).
 HALOGEN_KV_POOL_POSITIONS=401408
 # Two concurrent conversations: the MTP drafter is not slot-locked on this
 # engine, so the second slot is capacity, not a speculation tax.
@@ -107,10 +117,13 @@ HALOGEN_NPU_MODELS=qwen3-embedding-0.6b
 FRANK_DRAGOON_NPU_DIR=/home/dank/models/halogen-v2-abliterated/npu
 # Node paths and the XRT directory are variables so the bats seam can drive
 # these gates on a box with no NPU. Ubuntu's distro XRT (libxrt2 + libxrt-npu2)
-# is mounted FILE BY FILE and TWICE EACH — once where the image looks for XRT,
+# is mounted FILE BY FILE and THREE TIMES EACH — where the image looks for XRT
-# once at its own path, because those libraries find each other at their install
+# (/opt/xilinx/xrt/lib), at its own path (those libraries find each other at
-# path; the vendor forbids mounting an /opt/xilinx/xrt of symlinks into the
+# their install path), and at /opt/xilinx/lib/<multiarch>/ (the NPU child
-# system dir, which dangle inside the container.
+# searches that third prefix; without it the lane dies naming it and the
 # supervisor takes the container down — 2026-10-21 boot loop). The basename of
 # this directory IS the multiarch name. The vendor forbids mounting an
 # /opt/xilinx/xrt of symlinks into the system dir, which dangle in the container.
 FRANK_DRAGOON_DEV_ACCEL=/dev/accel/accel0
 FRANK_DRAGOON_XRT_DIR=/usr/lib/x86_64-linux-gnu
 # sysfs root the fabric-clock preflight reads (same knob the vendor's
@@ -1,5 +1,5 @@
 [Unit]
-Description=halogen-flash-server container (Quat3rnion NATIVE v2 abliterated Qwen3.8-Flash-Next, MTP spec decode, 2 slots over ONE 401408-position pool at ctx 200704, vision ON), root systemd unit (dankclaw, frank-dragoon, Strix Halo iGPU)
+Description=halogen-flash-server container (Quat3rnion NATIVE v2 abliterated Qwen3.8-Flash-Next, MTP spec decode, 2 slots over ONE 401408-position pool at ctx 262144, vision ON), root systemd unit (dankclaw, frank-dragoon, Strix Halo iGPU)
 After=network-online.target
 Wants=network-online.target
@@ -1,12 +1,12 @@
-# frank-zealot — llama.cpp (PrismML fork build 10777, CUDA) on the RTX 3090,
+# frank-zealot — llama.cpp (PrismML fork build 12118 = 0.7.0-dev, commit
-# :7771. Payload: Ternary-Bonsai-2-9B PQ2_0 with the MTP head in the file, plus
+# 024474448, CUDA) on the RTX 3090, :7771. Payload: Ternary-Bonsai-2-9B PQ2_0 with the MTP head in the file, plus
 # the refusal-direction adapter applied at load.
 #
 # ONE OWNER: shape, payload identity, eras, laws and every measured number
 # live in serverAppConfigs-docs/LLM/frank-zealot.md. This file carries the
 # knobs and only the reason a reader editing THIS line cannot recover
 # elsewhere. Where the two disagree, this file wins on values.
-LLAMA_BIN=/home/dank/GitPackages/llama-prism-b10777/llama-server
+LLAMA_BIN=/home/dank/GitPackages/llama-prism-0244744/bin/llama-server
 # The fork is a LOCK, not a preference: custom tensor type 142 + prism.hadamard.*
 # metadata. Stock llama.cpp refuses the file; a loader ignoring the Hadamard
 # metadata dequantizes wrong, silently.
@@ -64,7 +64,8 @@ CACHE_TYPE_V=q8_0
 # weights, so there is no --spec-draft-model. Its bill is pool × 4,096 B of
 # draft KV plus ~0.45 GiB of buffers and a tripled recurrent state — paid for
 # by CACHE_TYPE_V=q8_0 and UBATCH_SIZE=1024 above. Build gate: quantized V
-# beside MTP needs fork build >= b10776; LLAMA_BIN is 10777.
+# beside MTP needs fork build >= b10776; LLAMA_BIN is 12118 (b10777 is kept on
 # disk as the rollback path).
 # SPEC_TYPE=none is the rollback and strips the WHOLE set (--spec-type and
 # every --spec-draft-*); half a set has segfaulted at load.
 SPEC_TYPE=draft-mtp
@@ -0,0 +1,212 @@
 #!/usr/bin/env python3
 """gpu-exporter: unified per-GPU metrics for a mixed AMD/NVIDIA host.
 Emits the `gpu_*` Prometheus families the host-system dashboard graphs:
    gpu_utilization_percent  gpu_memory_used_bytes  gpu_memory_total_bytes
    gpu_temperature_celsius  gpu_power_watts
 each labelled {gpu, vendor, model}. The scrape job adds `host`.
 Two sources, both host-native — no container, no GPU passthrough:
  * AMD (R9700s, the 8060S iGPU, the Raphael display GPU): the amdgpu sysfs tree.
    Utilization comes ONLY from gpu_busy_percent — amd-smi reports gfx_busy_inst
    as N/A on gfx1201 (verified 2026-10-20). VRAM is already bytes, temperature
    is milli-degrees, power is micro-watts. Crucially power1_average is EMPTY on
    the R9700s, so those cards legitimately have no gpu_power_watts line; a
    missing file becomes an ABSENT metric, never a fabricated 0 that a dashboard
    would plot as a real reading.
  * NVIDIA (the RTX 3090 on Dankclaw): one `nvidia-smi --query-gpu` CSV. On
    Dankserver there is an nvidia-smi binary but no NVIDIA GPU, so the command
    fails; that is tolerated and simply yields no NVIDIA series.
 Default mode prints one scrape to stdout and exits (the test harness, and any
 `service ... --oneshot` one-off). `--serve` binds the HTTP endpoint for
 Prometheus; it binds EXPLICITLY (loopback + the host LAN address), never a
 wildcard, matching node-exporter's posture on these boxes.
 """
 import argparse
 import glob
 import os
 import subprocess
 import sys
 import threading
 from http.server import BaseHTTPRequestHandler, ThreadingHTTPServer
 FAMILIES = [
    ("gpu_utilization_percent", "gauge", "GPU busy time as a percentage of wall time."),
    ("gpu_memory_used_bytes", "gauge", "Device memory in use, in bytes."),
    ("gpu_memory_total_bytes", "gauge", "Device memory capacity, in bytes."),
    ("gpu_temperature_celsius", "gauge", "Package temperature in degrees Celsius."),
    ("gpu_power_watts", "gauge", "Board power draw in watts."),
 ]
 DEFAULT_NVIDIA_CMD = (
    "nvidia-smi --query-gpu=index,name,utilization.gpu,memory.used,"
    "memory.total,temperature.gpu,power.draw --format=csv,noheader,nounits"
 )
 def _read(path):
    try:
        with open(path) as fh:
            return fh.read().strip()
    except OSError:
        return ""
 def _pci_slot(device_dir):
    uevent = _read(os.path.join(device_dir, "uevent"))
    for line in uevent.splitlines():
        if line.startswith("PCI_SLOT_NAME="):
            return line.split("=", 1)[1]
    return ""
 def _pci_id(device_dir):
    uevent = _read(os.path.join(device_dir, "uevent"))
    for line in uevent.splitlines():
        if line.startswith("PCI_ID="):
            return "amd:" + line.split("=", 1)[1].lower()
    return "amd"
 def collect_amd(sys_dir):
    for card in sorted(glob.glob(os.path.join(sys_dir, "card[0-9]*"))):
        device = os.path.join(card, "device")
        busy = _read(os.path.join(device, "gpu_busy_percent"))
        if busy == "":
            continue  # not an amdgpu compute node (e.g. a render node w/o metrics)
        labels = {"gpu": _pci_slot(device) or os.path.basename(card),
                  "vendor": "amd", "model": _pci_id(device)}
        yield "gpu_utilization_percent", labels, busy
        for name, leaf in (("gpu_memory_used_bytes", "mem_info_vram_used"),
                           ("gpu_memory_total_bytes", "mem_info_vram_total")):
            v = _read(os.path.join(device, leaf))
            if v != "":
                yield name, labels, v
        temp = ""
        for hwmon in sorted(glob.glob(os.path.join(device, "hwmon", "hwmon*", "temp1_input"))):
            temp = _read(hwmon)
            if temp != "":
                break
        if temp != "":
            yield "gpu_temperature_celsius", labels, str(int(temp) / 1000.0)
        power = _read(os.path.join(device, "power1_average"))
        if power != "":  # empty on the R9700s -> metric absent, not zero
            yield "gpu_power_watts", labels, str(int(power) / 1e6)
 def collect_nvidia(cmd):
    if not cmd:
        return
    try:
        out = subprocess.run(cmd, shell=True, capture_output=True, text=True,
                             timeout=10, check=True).stdout
    except (subprocess.SubprocessError, OSError):
        return  # no NVIDIA GPU / no driver: tolerate, emit nothing
    for line in out.splitlines():
        parts = [p.strip() for p in line.split(",")]
        if len(parts) < 7:
            continue
        idx, name, util, mused, mtotal, temp, power = parts[:7]
        labels = {"gpu": idx, "vendor": "nvidia", "model": name}
        yield "gpu_utilization_percent", labels, util
        yield "gpu_memory_used_bytes", labels, str(int(float(mused)) * 1048576)
        yield "gpu_memory_total_bytes", labels, str(int(float(mtotal)) * 1048576)
        if temp not in ("", "[N/A]"):
            yield "gpu_temperature_celsius", labels, temp
        if power not in ("", "[N/A]"):
            yield "gpu_power_watts", labels, power
 def collect(sys_dir, nvidia_cmd):
    samples = {}
    for source in (collect_amd(sys_dir), collect_nvidia(nvidia_cmd)):
        for name, labels, value in source:
            try:
                value = str(float(value))  # normalise; drop trailing junk
            except ValueError:
                continue
            samples.setdefault(name, []).append((labels, value))
    return samples
 def _fmt(value):
    f = float(value)
    if f == int(f) and abs(f) < 1e15:
        return str(int(f))
    return repr(f)
 def render(samples):
    out = []
    for name, kind, help_text in FAMILIES:
        out.append("# HELP %s %s" % (name, help_text))
        out.append("# TYPE %s %s" % (name, kind))
        for labels, value in sorted(samples.get(name, []), key=lambda x: sorted(x[0].items())):
            lset = ",".join('%s="%s"' % (k, v) for k, v in sorted(labels.items()))
            out.append("%s{%s} %s" % (name, lset, _fmt(value)))
    return "\n".join(out) + "\n"
 class Handler(BaseHTTPRequestHandler):
    sys_dir = "/sys/class/drm"
    nvidia_cmd = DEFAULT_NVIDIA_CMD
    def do_GET(self):  # noqa: N802 (http.server API)
        if self.path.split("?")[0] not in ("/metrics", "/"):
            self.send_error(404)
            return
        body = render(collect(self.sys_dir, self.nvidia_cmd)).encode()
        self.send_response(200)
        self.send_header("Content-Type", "text/plain; version=0.0.4")
        self.send_header("Content-Length", str(len(body)))
        self.end_headers()
        self.wfile.write(body)
    def log_message(self, *args):
        pass
 def main(argv):
    ap = argparse.ArgumentParser(description=__doc__.splitlines()[0])
    ap.add_argument("--amdgpu-sysfs", default="/sys/class/drm")
    ap.add_argument("--nvidia-cmd", default=DEFAULT_NVIDIA_CMD)
    ap.add_argument("--serve", action="store_true", help="serve /metrics instead of one-shot")
    ap.add_argument("--port", type=int, default=9401)
    ap.add_argument("--bind", action="append", default=None,
                    help="explicit bind address (repeatable); env GPU_EXPORTER_BINDS "
                         "comma-list if unset. NEVER a wildcard.")
    args = ap.parse_args(argv)
    if not args.serve:
        sys.stdout.write(render(collect(args.amdgpu_sysfs, args.nvidia_cmd)))
        return 0
    binds = args.bind
    if not binds:
        env = os.environ.get("GPU_EXPORTER_BINDS", "")
        binds = [b.strip() for b in env.split(",") if b.strip()] or ["127.0.0.1"]
    Handler.sys_dir = args.amdgpu_sysfs
    Handler.nvidia_cmd = args.nvidia_cmd
    servers = []
    for b in binds:
        if b in ("0.0.0.0", "::", "*"):
            sys.stderr.write("refusing wildcard bind %s\n" % b)
            return 2
        servers.append(ThreadingHTTPServer((b, args.port), Handler))
    sys.stderr.write("gpu-exporter serving :%d on %s\n" % (args.port, ",".join(binds)))
    # EVERY socket must be served: a listen()ed-but-never-accept()ed bind looks
    # healthy in `ss` yet times out every scrape once its SYN backlog fills —
    # and the loopback deploy gate cannot see it. One thread per extra bind.
    for s in servers[1:]:
        threading.Thread(target=s.serve_forever, daemon=True).start()
    servers[0].serve_forever()
    return 0
 if __name__ == "__main__":
    sys.exit(main(sys.argv[1:]))
@@ -0,0 +1,35 @@
 # Systemd USER unit: gpu-exporter (both Dankclaw and Dankserver).
 #
 # Install path: ~/.config/systemd/user/gpu-exporter.service
 # Installed and enabled by deploy_gpu_exporter in scripts/deploy-remote.sh.
 #
 # Per-GPU metrics (utilization / VRAM / temperature / power) for the mixed
 # AMD + NVIDIA fleet, scraped by Prometheus. Like signal-relay it is a plain
 # user unit, not a quadlet: it runs python3 straight from the checkout (a pull
 # updates it with no copy step), a hand-written user unit can be enabled where
 # the podman v1 generator's units cannot, and with Linger=yes for dank it starts
 # at boot. It reads the host's own cgroup/drm sysfs and shells to nvidia-smi, so
 # it must run on the host — a container would need GPU/DRM passthrough for
 # nothing gained.
 #
 # BINDING. The base unit serves loopback only. deploy_gpu_exporter writes a
 # drop-in adding THIS host's LAN address (the same two-address posture
 # node-exporter and cadvisor use), so the endpoint is never a wildcard. It is
 # unauthenticated metrics on the LAN, identical to node-exporter :9100 and
 # cadvisor :8080 — Prometheus reaches same-host via host.containers.internal and
 # the Dankserver instance over the LAN from Dankclaw.
 [Unit]
 Description=GPU metrics exporter (amdgpu sysfs + nvidia-smi)
 After=network-online.target
 Wants=network-online.target
 [Service]
 # Loopback by default; the deploy drop-in appends the host LAN address.
 Environment=GPU_EXPORTER_BINDS=127.0.0.1
 ExecStart=/usr/bin/python3 /home/dank/serverAppConfigs/scripts/gpu-exporter.py --serve
 Restart=always
 RestartSec=5
 [Install]
 WantedBy=default.target
@@ -0,0 +1,418 @@
 #!/usr/bin/env python3
 """grams-to-imatrix.py — the engine's own calibration recording, re-voiced as a
 llama.cpp importance matrix for `rad-convert --imatrix F`. RUNS ON DANKSERVER
 beside the recording. Python 3 stdlib only: numpy must NOT be required, because
 this runs in the plain container next to a 335 GiB checkpoint, not in a venv.
 WHY. `rad-convert` takes its per-column activation second moments and its
 expert-popularity profile from ONE input, an imatrix (core/format/imatrix.h).
 Only the GGUF form carries a per-expert dimension — with the legacy flat form,
 or with none, every expert in a layer is weighted by the layer's aggregate,
 which is exactly the wrong thing on a 512-expert MoE. The calibration tap has
 already measured everything; this tool only re-voices it, so the imatrix and the
 protected-expert census (derive-protected-experts.py) are the SAME recording:
 one measurement, two consumers, neither able to disagree about which experts are
 hot.
 INPUT. RADGRAM2 files in a RADIANCE_CALIB_DIR output directory
 (arch/common/rad_arch.h, calib_write). Header law, same parse as
 scripts/derive-protected-experts.py — 32 bytes, `<8sIIQI`: magic `RADGRAM2`,
 u32 k, u32 rank, u64 rows, u32 domain, a pad word, then k*k f32 row-major. Two
 kinds of name, both GGUF-style, both with the checkpoint's GLOBAL expert
 numbering (rad_block_moe_fp8.h: ge = expert_base + e):
    gram.r<rank>.blk.L.ffn_down_exps.bin        the layer's pooled gram
    gram.r<rank>.blk.L.ffn_down_exps.<e>.bin    expert e's own gram
 `rows` is that file's routing/token count and the matrix is Σ x xᵀ over those
 rows, so **the diagonal is exactly the per-column activation second moment an
 imatrix wants**, and `rows` is the count the reader hands back from `count()`.
 Every off-diagonal term is dropped on purpose: the GGUF format has nowhere to
 put a full matrix, and the quantiser asks for columns.
 OUTPUT, per layer L (core/format/imatrix.h: names WITHOUT `.weight`, a MoE
 layer's experts folded into ONE entry, so expert 9 is slice 9 of
 `blk.L.ffn_down_exps`):
  blk.L.ffn_down_exps    in_sum2 [k, n_expert] expert-major, counts [1, n_expert]
  blk.L.ffn_gate_up_exps the SAME matrices at the SAME offsets. A routed expert's
                         gate_up input is the same hidden-state rows as its down
                         input, so the second moment over them is identical and
                         so is the routing count. Two tensor infos over one
                         payload: the reader maps each half independently
                         (imatrix.cpp joins `<base>.in_sum2` with `<base>.counts`
                         by base name), and copying ~200 MB to state one fact
                         twice buys no correctness.
  blk.L.ssm_inz          [k] and [1], from the pooled gram: that row's input is
                         the same hidden state, so its columns take the pooled
                         diagonal and its count is the layer's pooled rows.
  Then STOP. Nothing whose input width is not the gram's k — no ssm_out, no
  attn_output, no mtp.*, no ple_ngram. importance_for() answers false when the
  entry's cols do not equal the weight's width and the convert proceeds without
  it: an entry the reader silently ignores is a lie in the artifact, worse than
  an absent one. The corollary to check before trusting the two hidden-width
  entries: the tap writes the gate_up gram at g.n_embd and the down gram at the
  expert width, so the gram's k must equal the checkpoint's hidden width (its
  config.json). Where they differ, gate_up/ssm_inz are inert, not wrong.
 INHERITANCE. An expert with no own gram carries the layer's POOLED diagonal, and
 its count stays its own routing rows — the rule the engine itself applies when it
 picks a Hessian (libquant/lq_hess.h, calib_resolve: an expert's own gram only
 once its rank files hold kOwnRowsPerK * K rows; below that a fit to so few rows
 is worse than round-to-nearest, and those experts carry a fraction of a percent
 of the routing). The count never inherits: popularity is `count`'s job. The count
 of inherited experts is reported on stderr.
 RANK FOLDING. TP=2 writes gram.r0.* and gram.r1.* with GLOBAL expert numbering,
 so the ranks describe DISJOINT experts and the fold is a UNION into the one
 [k, n_expert] entry. A pooled gram is the opposite case: every rank summed the
 same stem over its own rows, so the fold is a SUM, rows included (calib_sum,
 calib_rows). The same global expert in two per-expert files is legal only when
 the numbers agree exactly; otherwise the recording is corrupt and the entry would
 be whichever file sorted last, so refuse.
 GGUF v3 (the only version imatrix.cpp accepts — version != 3 is a refusal):
 `general.alignment` u32, `imatrix.chunk_size` u32, `imatrix.chunk_count` u32,
 `imatrix.datasets` array of strings; tensors f32 (ggml type 0) with offsets
 relative to the aligned data section. Every tensor offset is padded up to
 `general.alignment` and the data section itself starts aligned. That is the
 classic hand-rolled-writer bug: an unaligned offset makes the reader's extent
 arithmetic point into the middle of the previous tensor's floats, and it reads
 them without complaint.
 chunk_count DERIVATION. The corpus row count is the LARGEST single-layer pooled
 row total, not a sum over layers: every layer's tap accumulated over the whole
 corpus, so adding the layers counts each token once per layer and inflates the
 figure by the layer count. chunk_count = that row total // chunk_size, the
 "chunks of chunk_size" line imatrix.cpp prints for itself.
 REFUSALS, each exit 2 with a message naming what is missing:
  NO_GRAMS         the directory holds no RADGRAM2 down-projection gram
  NOT_RADGRAM2     a gram whose magic is not RADGRAM2
  SHORT            a gram whose length is not 32 + k*k*4
  NO_POOLED        a layer with expert grams and no pooled gram to inherit from
  K_MISMATCH       a layer whose files disagree about k (rank pooled grams, or an
                   expert gram against its layer's pooled gram)
  EXPERT_CONFLICT  a global expert recorded twice with different numbers
  EXISTS           the -o path already holds a file (the trash law: move it)
  CHUNK_COUNT      the derived chunk count does not fit the u32 metadata field
 USAGE. grams-to-imatrix.py <calib-dir> -o <out.gguf>
       [--datasets NAME,...] [--chunk-size N] [--min-rows N] [-v]
 """
 import argparse
 import os
 import re
 import struct
 import sys
 MAGIC = b"RADGRAM2"
 HDR = 32                       # 8s I I Q I + pad, rad_arch.h
 KOWN_ROWS_PER_K = 4            # libquant/lq_hess.h kOwnRowsPerK
 ALIGNMENT = 32                 # general.alignment, and the pad unit
 POOLED_RE = re.compile(r"^gram\.r(\d+)\.blk\.(\d+)\.ffn_down_exps\.bin$")
 EXPERT_RE = re.compile(r"^gram\.r(\d+)\.blk\.(\d+)\.ffn_down_exps\.(\d+)\.bin$")
 GT_U32, GT_STR, GT_ARR = 4, 8, 9
 GGML_F32 = 0
 def refuse(msg):
    sys.stderr.write("REFUSED %s\n" % msg)
    sys.exit(2)
 def read_gram(path):
    """(k, rank, rows, domain, diagonal) from one RADGRAM2 file."""
    try:
        with open(path, "rb") as f:
            raw = f.read()
    except OSError as e:
        refuse("UNREADABLE %s: %s" % (path, e))
    if len(raw) < HDR:
        refuse("SHORT %s: %d bytes, under the %d-byte RADGRAM2 header"
               % (path, len(raw), HDR))
    magic, k, rank, rows, domain = struct.unpack_from("<8sIIQI", raw, 0)
    if magic != MAGIC:
        refuse("NOT_RADGRAM2 %s: magic %r, expected b'RADGRAM2'" % (path, magic))
    if k <= 0:
        refuse("SHORT %s: header claims k=%d, which describes no matrix" % (path, k))
    need = HDR + k * k * 4
    if len(raw) != need:
        refuse("SHORT %s: %d bytes, expected %d (32 + k*k*4 for k=%d)"
               % (path, len(raw), need, k))
    # The diagonal only, read by stride out of the row-major matrix.
    diag = [struct.unpack_from("<f", raw, HDR + (i * k + i) * 4)[0]
            for i in range(k)]
    return k, rank, rows, domain, diag
 def load_dir(d, verbose):
    """{layer: {"pooled": gram|None, "experts": {global e: gram}}} + rotated count.
    Files outside the ffn_down_exps stem are not ours to fold: the tap names a
    pooled gram after every projection it taps, and those carry other widths."""
    if not os.path.isdir(d):
        refuse("NO_GRAMS %s: not a directory" % d)
    layers, rotated = {}, 0
    for name in sorted(os.listdir(d)):
        m = POOLED_RE.match(name) or EXPERT_RE.match(name)
        if not m:
            continue
        path = os.path.join(d, name)
        if not os.path.isfile(path):
            continue
        pooled = bool(POOLED_RE.match(name))
        layer = int(m.group(2))
        k, rank, rows, domain, diag = read_gram(path)
        if domain != 0:
            rotated += 1
        L = layers.setdefault(layer, {"pooled": None, "experts": {}})
        if pooled:
            P = L["pooled"]
            if P is None:
                L["pooled"] = {"k": k, "rows": rows, "diag": diag, "paths": [path]}
            elif P["k"] != k:
                refuse("K_MISMATCH layer %d: pooled grams %s (k=%d) and %s (k=%d)"
                       " disagree about the width"
                       % (layer, P["paths"][0], P["k"], path, k))
            else:
                # calib_sum: each rank summed the same stem over its own rows.
                P["rows"] += rows
                P["paths"].append(path)
                for i in range(k):
                    P["diag"][i] += diag[i]
            if verbose:
                sys.stderr.write("pooled layer %d: %s k=%d rows=%d\n"
                                 % (layer, name, k, rows))
            continue
        e = int(m.group(3))
        own = L["experts"].get(e)
        if own is None:
            L["experts"][e] = {"k": k, "rows": rows, "diag": diag,
                               "path": path, "ranks": [rank]}
            if verbose:
                sys.stderr.write("expert layer %d r%d e%d: rows=%d\n"
                                 % (layer, rank, e, rows))
            continue
        if own["rows"] != rows or own["diag"] != diag:
            refuse("EXPERT_CONFLICT layer %d expert %d: %s (rows=%d) and %s"
                   " (rows=%d) record the same global expert with different"
                   " numbers" % (layer, e, own["path"], own["rows"], path, rows))
        own["ranks"].append(rank)
        sys.stderr.write("note: layer %d expert %d recorded identically by rank %s"
                         " and %s; folded once\n"
                         % (layer, e, own["path"], name))
    if not layers:
        refuse("NO_GRAMS %s: no RADGRAM2 gram.r<rank>.blk.<L>.ffn_down_exps"
               "[.<expert>].bin files in the directory" % d)
    return layers, rotated
 def build_entries(layers, min_rows, verbose):
    """([ (name, dims, payload|None, key) ], census, layers emitted)."""
    entries = []
    census = {"inherited": 0, "experts": 0, "routing": 0, "rows": 0, "layers": 0}
    for layer in sorted(layers):
        P, E = layers[layer]["pooled"], layers[layer]["experts"]
        if E and P is None:
            refuse("NO_POOLED layer %d: %d expert gram(s) but no pooled gram (an"
                   " expected gram.r<rank>.blk.%d.ffn_down_exps.bin), so an expert"
                   " under the row threshold would have nothing to inherit"
                   % (layer, len(E), layer))
        if P is None:
            continue
        k = P["k"]
        census["layers"] += 1
        census["rows"] = max(census["rows"], P["rows"])
        dn = "blk.%d.ffn_down_exps" % layer
        # A staging key names ONE payload. A gate_up entry reuses the down
        # layer's bytes by carrying the same key with no payload of its own;
        # in_sum2 and counts are different bytes and must never share a key.
        ksum, kcnt = (layer, "sum2"), (layer, "counts")
        kpgm, kpcnt = (layer, "pooled_sum2"), (layer, "pooled_counts")
        # One entry holds one width: an expert gram recorded at another k would
        # inherit (or be inherited by) a diagonal of the wrong length, and the
        # reader would then answer `cols` from whichever file set dims[0].
        for e in sorted(E):
            if E[e]["k"] != k:
                refuse("K_MISMATCH layer %d expert %d: %s holds k=%d and this"
                       " layer's pooled gram is k=%d" % (layer, e, E[e]["path"],
                                                          E[e]["k"], k))
        if not E:
            # No per-expert recording for this layer: emit the dense form
            # ([k], [1]) rather than 512 identical slices, which would claim a
            # per-expert measurement that was never taken.
            entries.append((dn + ".in_sum2", [k],
                            struct.pack("<%df" % k, *P["diag"]), ksum))
            entries.append((dn + ".counts", [1],
                            struct.pack("<f", float(P["rows"])), kcnt))
            sys.stderr.write("note: layer %d has no per-expert grams; %s is"
                             " dense over the pooled gram\n" % (layer, dn))
        else:
            floor = min_rows if min_rows > 0 else KOWN_ROWS_PER_K * k
            n_expert = max(E) + 1
            sum2, counts = bytearray(), bytearray()
            for e in range(n_expert):
                own = E.get(e)
                if own is not None and own["rows"] >= floor:
                    src = own["diag"]
                    rows = own["rows"]
                else:
                    src = P["diag"]
                    rows = own["rows"] if own is not None else 0
                    census["inherited"] += 1
                sum2 += struct.pack("<%df" % k, *src)
                counts += struct.pack("<f", float(rows))
                census["experts"] += 1
                census["routing"] += rows
            if min(E) != 0:
                sys.stderr.write("note: layer %d has no gram for experts 0..%d;"
                                 " those slices carry the pooled gram at count 0\n"
                                 % (layer, min(E) - 1))
            entries.append((dn + ".in_sum2", [k, n_expert], bytes(sum2), ksum))
            entries.append((dn + ".counts", [1, n_expert], bytes(counts), kcnt))
            gu = "blk.%d.ffn_gate_up_exps" % layer
            entries.append((gu + ".in_sum2", [k, n_expert], None, ksum))
            entries.append((gu + ".counts", [1, n_expert], None, kcnt))
            if verbose:
                sys.stderr.write("layer %d: k=%d n_expert=%d pooled_rows=%d"
                                 " own-gram floor=%d\n"
                                 % (layer, k, n_expert, P["rows"], floor))
        inz = "blk.%d.ssm_inz" % layer
        entries.append((inz + ".in_sum2", [k],
                        struct.pack("<%df" % k, *P["diag"]), kpgm))
        entries.append((inz + ".counts", [1],
                        struct.pack("<f", float(P["rows"])), kpcnt))
    return entries, census
 def enc_str(v):
    b = v.encode("utf-8")
    return struct.pack("<Q", len(b)) + b
 def write_gguf(path, entries, datasets, chunk_size, chunk_count):
    """Write GGUF v3 and return its size in bytes.
    One payload per key: entries that share a key share the bytes and the
    offset, so a second tensor info states the same fact without copying it."""
    kv = bytearray()
    kv += enc_str("general.alignment") + struct.pack("<I", GT_U32) \
        + struct.pack("<I", ALIGNMENT)
    kv += enc_str("imatrix.chunk_size") + struct.pack("<I", GT_U32) \
        + struct.pack("<I", chunk_size)
    kv += enc_str("imatrix.chunk_count") + struct.pack("<I", GT_U32) \
        + struct.pack("<I", chunk_count)
    kv += enc_str("imatrix.datasets") + struct.pack("<I", GT_ARR) \
        + struct.pack("<I", GT_STR) + struct.pack("<Q", len(datasets))
    for d in datasets:
        kv += enc_str(d)
    head = bytearray()
    head += b"GGUF" + struct.pack("<I", 3)
    head += struct.pack("<Q", len(entries)) + struct.pack("<Q", 4)
    head += kv
    payloads = {}
    for name, _dims, payload, key in entries:
        if payload is None:
            continue
        prev = payloads.get(key)
        if prev is not None and prev != payload:
            # Two different payloads on one key would ship whichever was
            # appended last while BOTH tensor infos pointed at it: the counts
            # payload landing over the in_sum2 payload is exactly how a
            # hand-rolled GGUF writer loses half its data without an error.
            refuse("INTERNAL %s: two different payloads share one staging key"
                   % name)
        payloads.setdefault(key, payload)
    infos = bytearray()
    offsets, order, size = {}, [], 0
    for name, dims, payload, key in entries:
        if key not in payloads:
            refuse("INTERNAL %s: no payload staged for its key" % name)
        if key not in offsets:
            offsets[key] = (size + ALIGNMENT - 1) // ALIGNMENT * ALIGNMENT
            size = offsets[key] + len(payloads[key])
            order.append(key)
        infos += enc_str(name) + struct.pack("<I", len(dims))
        for d in dims:
            infos += struct.pack("<Q", d)
        infos += struct.pack("<I", GGML_F32) + struct.pack("<Q", offsets[key])
    body = bytes(head) + bytes(infos)
    data_start = (len(body) + ALIGNMENT - 1) // ALIGNMENT * ALIGNMENT
    with open(path, "wb") as f:
        f.write(body)
        f.write(b"\0" * (data_start - len(body)))
        # The offsets in the tensor infos are the file's law, not a hint: each
        # payload must land at its own padded offset. Writing them back to back
        # instead leaves every reader that trusts the info — imatrix.cpp does —
        # reading its neighbour's floats: plausible values, no error, wrong
        # quantisation.
        here = data_start
        for key in order:
            f.write(b"\0" * (data_start + offsets[key] - here))
            f.write(payloads[key])
            here = data_start + offsets[key] + len(payloads[key])
        return f.tell()
 def main():
    ap = argparse.ArgumentParser(
        description="Fold a radiance calibration recording into a GGUF v3 imatrix.")
    ap.add_argument("calib_dir")
    ap.add_argument("-o", "--out", required=True)
    ap.add_argument("--datasets", default="",
                    help="comma-separated names for imatrix.datasets")
    ap.add_argument("--chunk-size", type=int, default=4096,
                    help="rows per chunk for imatrix.chunk_size (default 4096)")
    ap.add_argument("--min-rows", type=int, default=0,
                    help="own-gram row threshold (default 4*k, the engine's"
                         " kOwnRowsPerK law)")
    ap.add_argument("-v", "--verbose", action="store_true")
    a = ap.parse_args()
    if os.path.exists(a.out):
        # The repo's deletion law: an artifact that exists is moved to trash by
        # a human, never silently overwritten by a re-run of the tool that made
        # it, so a second convert cannot destroy the imatrix the first one fed.
        refuse("EXISTS %s: an imatrix is already there; move it to"
               " .AGENTTRASH/ or name another -o" % a.out)
    if a.chunk_size <= 0:
        refuse("BAD_CHUNK_SIZE --chunk-size %d is not positive" % a.chunk_size)
    layers, rotated = load_dir(a.calib_dir, a.verbose)
    entries, census = build_entries(layers, a.min_rows, a.verbose)
    if not entries:
        refuse("NO_GRAMS %s: no layer holds a pooled gram to fold" % a.calib_dir)
    if rotated:
        sys.stderr.write("note: %d gram(s) recorded in a rotated domain: their"
                         " diagonal is the rotated activation's second moment,"
                         " not the model's own\n" % rotated)
    chunk_count = census["rows"] // a.chunk_size
    if chunk_count > 0xFFFFFFFF:
        # imatrix.chunk_count is a u32 in the reader's grammar, so there is no
        # legal way to record a bigger one. Reaching this means either a chunk
        # size picked for a far larger run or a row total read out of the wrong
        # header offset — rows is a u64 and a u32 parse is off by 2**32 — and
        # the reader would trust whatever number landed in the field.
        refuse("CHUNK_COUNT %d = %d rows // %d does not fit the u32"
               " imatrix.chunk_count field; raise --chunk-size"
               % (chunk_count, census["rows"], a.chunk_size))
    nbytes = write_gguf(a.out, entries, [d for d in a.datasets.split(",") if d],
                        a.chunk_size, chunk_count)
    sys.stderr.write("IMATRIX_OK entries=%d layers=%d experts=%d inherited=%d"
                     " rows=%d routing=%d chunks=%d/%d bytes=%d out=%s\n"
                     % (len(entries), census["layers"], census["experts"],
                        census["inherited"], census["rows"], census["routing"],
                        chunk_count, a.chunk_size, nbytes, a.out))
 if __name__ == "__main__":
    main()
@@ -0,0 +1,391 @@
 #!/usr/bin/env bash
 # Metadata-provider keys for Jellyfin on Dankserver, placed from the service's
 # host env file with the plugin artifact pinned by content hash.
 #
 # Jellyfin 12 ships TMDb, OMDb, StudioImages and MusicBrainz as CORE plugins:
 # they have no DLL, only a configuration file under the data root, and that
 # file is created by the running container. So the fleet's provider state lives
 # in two places the repo does not track —
 #
 #   plugins/configurations/Jellyfin.Plugin.Tmdb.xml   <TmdbApiKey /> (empty = keyless)
 #   plugins/configurations/Jellyfin.Plugin.Tvdb.xml   <ApiKey />     (needs the plugin)
 #
 # and both are owned by the container user. This tool is the only thing that
 # writes them: it is declarative, idempotent, and it never prints a key, so it
 # is safe to run while someone watches.
 #
 # Keys never enter git and never enter argv (a value in argv is world-readable
 # through /proc/<pid>/cmdline). They live in the per-service host env file that
 # is the fleet's convention for container credentials (grafana, caddy, pihole,
 # copyparty, romm, seerr), mode 600:
 #
 #   /home/dank/Containers/jellyfin/provider-keys
 #       TMDB_API_KEY=<key>
 #       TVDB_API_KEY=<key>
 #       TVDB_PIN=<pin>        # optional: only a subscriber-shaped TVDB key
 #                             # needs one; a licensed key does not
 #
 # TVDB itself is not core in 12 (the metadata provider was cut in 10.9), so it
 # is installed from the official catalog, pinned three ways: sha256 of the
 # artifact, the plugin's own guid, and its version. A matching sha proves only
 # that the bytes are the bytes fetched — the guid check is what says it is the
 # official plugin, the version check is what says it is the one this file was
 # reviewed against.
 #
 # Subcommands
 #   plan      print the resolved shape and every pending change, touch nothing
 #   install   fetch+verify the plugin, then write the keys (backing up first)
 #   verify    prove the plugin and both keys are actually in place
 #
 # Nothing here restarts Jellyfin: the deployer owns that
 # (`scripts/deploy.sh dankserver --service jellyfin`), and the plugin is read at
 # start-up, so a key written while the server runs is not live until then.
 set -euo pipefail
 # Jellyfin's data root on Dankserver (the rootless quadlet's config volume).
 JF_ROOT="${JF_ROOT:-/home/dank/Containers/jellyfin/config}"
 PROV_KEYS="${PROV_KEYS:-/home/dank/Containers/jellyfin/provider-keys}"
 TRASH_DIR="${TRASH_DIR:-$HOME/.AGENTTRASH/$(date +%Y%m%d)-jellyfin-providers}"
 PLUGINS="$JF_ROOT/plugins"
 CONF="$PLUGINS/configurations"
 TMDB_XML="$CONF/Jellyfin.Plugin.Tmdb.xml"
 TVDB_XML="$CONF/Jellyfin.Plugin.Tvdb.xml"
 # repo.jellyfin.org catalog entry for TheTVDB (owner: jellyfin), version 24.0.0.0,
 # targetAbi 12.0.0.0 — published 2026-09-09, verified against Jellyfin 12.4.0.
 TVDB_PLUGIN_URL="${TVDB_PLUGIN_URL:-https://repo.jellyfin.org/files/plugin/thetvdb/thetvdb_24.0.0.0.zip}"
 TVDB_PLUGIN_SHA256="${TVDB_PLUGIN_SHA256:-6302fba3857bdde8d186dd099fac09ca43247041f531a528a2c788add70a5319}"
 TVDB_PLUGIN_GUID="${TVDB_PLUGIN_GUID:-a677c0da-fac5-4cde-941a-7134223f14c8}"
 TVDB_PLUGIN_VERSION="${TVDB_PLUGIN_VERSION:-24.0.0.0}"
 PLUGIN_DIR="thetvdb_${TVDB_PLUGIN_VERSION}"
 TVDB_DLL="Jellyfin.Plugin.Tvdb.dll"
 die() {
    printf 'jellyfin-providers: %s\n' "$*" >&2
    exit 1
 }
 usage() {
    printf 'usage: %s plan|install|verify\n' "$0" >&2
    exit 2
 }
 # check_keys — refuse anything that is not a readable private file. An open
 # mode is an error, not a warning: a provider key readable by every user on the
 # box is the failure this check exists to catch.
 check_keys() {
    [ -f "$PROV_KEYS" ] ||
        die "$PROV_KEYS not found (create it with TMDB_API_KEY= and TVDB_API_KEY=, then chmod 600)"
    local mode
    mode="$(stat -c '%a' "$PROV_KEYS")"
    case "$mode" in
    600 | 400) ;;
    *) die "$PROV_KEYS is mode $mode; provider keys must be 600" ;;
    esac
 }
 # keys_value <VAR> — the value of one variable in the env file, or nothing.
 keys_value() {
    awk -v k="$1" -F= '$1 == k { sub(/^[^=]*=/, ""); print; exit }' "$PROV_KEYS"
 }
 # keys_len <VAR> — the length, and a refusal naming the variable when absent.
 # The value itself is never printed.
 keys_len() {
    local v
    v="$(keys_value "$1")"
    [ -n "$v" ] || die "$PROV_KEYS has no $1= line"
    printf '%s' "${#v}"
 }
 # xml_value <file> <element> — the element's text, empty when it is absent or
 # self-closing.
 xml_value() {
    python3 - "$1" "$2" <<'PYEOF'
 import re, sys
 src = open(sys.argv[1], encoding="utf-8").read()
 m = re.search(r"<%s(?:\s*/>|>(.*?)</%s>)" % (sys.argv[2], sys.argv[2]), src, re.S)
 print(m.group(1).strip() if (m and m.group(1)) else "")
 PYEOF
 }
 # xml_secret <file> <element> <var> — set <element> to that variable from the
 # env file. The python process reads the env file itself, so the value never
 # appears in argv.
 xml_secret() {
    python3 - "$1" "$2" "$3" "$PROV_KEYS" <<'PYEOF'
 import os, re, sys, tempfile
 from xml.sax.saxutils import escape
 xml_path, elem, var, keys_path = sys.argv[1:5]
 val = ""
 for line in open(keys_path, encoding="utf-8"):
    key, sep, rest = line.partition("=")
    if sep and key.strip() == var:
        val = rest.strip()
        break
 if not val:
    raise SystemExit(f"{keys_path} has no {var}= line")
 src = open(xml_path, encoding="utf-8").read()
 pat = re.compile(r"([ \t]*)<%s(?:\s*/>|>.*?</%s>)" % (elem, elem), re.S)
 text = escape(val)
 if pat.search(src):
    out = pat.sub(lambda m: f"{m.group(1)}<{elem}>{text}</{elem}>", src, count=1)
 else:
    out = src.replace("</Configuration>", f"  <{elem}>{text}</{elem}>\n</Configuration>", 1)
 if out == src:
    raise SystemExit(f"{xml_path}: no place to write <{elem}>")
 d, base = os.path.split(xml_path)
 fd, tmp = tempfile.mkstemp(prefix=base + ".", dir=d)
 with os.fdopen(fd, "w", encoding="utf-8") as fh:
    fh.write(out)
 os.chmod(tmp, os.stat(xml_path).st_mode & 0o777)
 os.replace(tmp, xml_path)
 PYEOF
 }
 # plugin_meta <zip> — "guid version" out of the artifact's own meta.json.
 plugin_meta() {
    python3 - "$1" <<'PYEOF'
 import json, sys, zipfile
 meta = json.loads(zipfile.ZipFile(sys.argv[1]).read("meta.json").decode("utf-8"))
 print(meta.get("guid", ""), meta.get("version", ""))
 PYEOF
 }
 check_root() {
    # Never fabricate a data root: creating plugins/ beside a typo'd path would
    # install a plugin Jellyfin never loads and hide the operator's mistake.
    [ -d "$CONF" ] || die "no Jellyfin configuration directory at $CONF (is JF_ROOT right?)"
 }
 check_keys_report() {
    check_keys
    # A refusal inside a command substitution used as a printf argument would
    # be swallowed, so the guard runs as its own statement.
    keys_len TMDB_API_KEY >/dev/null
    keys_len TVDB_API_KEY >/dev/null
    printf '= keys read from %s (mode %s)\n' "$PROV_KEYS" "$(stat -c '%a' "$PROV_KEYS")"
 }
 # The plugin's state, as one line in the shared vocabulary.
 plugin_state() {
    if [ -d "$PLUGINS/$PLUGIN_DIR" ]; then
        printf '= %s already installed\n' "$PLUGIN_DIR"
        return 0
    fi
    local other
    other="$(find "$PLUGINS" -maxdepth 1 -type d -name 'thetvdb_*' ! -name "$PLUGIN_DIR" -printf '%f\n' 2>/dev/null | sort | tr '\n' ' ')"
    if [ -n "$other" ]; then
        printf '+ install %s (a different version is present: %s)\n' "$PLUGIN_DIR" "$other"
        return 0
    fi
    printf '+ install %s from %s\n' "$PLUGIN_DIR" "${TVDB_PLUGIN_URL#*://}"
 }
 tmdb_state() {
    if [ ! -f "$TMDB_XML" ]; then
        printf '! %s is missing — save the TMDb plugin settings once in the UI, then re-run\n' "$TMDB_XML" >&2
        return 1
    fi
    local now want
    now="$(xml_value "$TMDB_XML" TmdbApiKey)"
    want="$(keys_value TMDB_API_KEY)"
    if [ -n "$now" ] && [ "$now" = "$want" ]; then
        printf '= TmdbApiKey already set (%s chars)\n' "${#now}"
    elif [ -n "$now" ]; then
        printf '+ replace the TmdbApiKey in Jellyfin.Plugin.Tmdb.xml\n'
    else
        printf '+ set TmdbApiKey in Jellyfin.Plugin.Tmdb.xml\n'
    fi
 }
 tvdb_state() {
    local now
    if [ -f "$TVDB_XML" ]; then
        now="$(xml_value "$TVDB_XML" ApiKey)"
        if [ -n "$now" ]; then
            printf '= Jellyfin.Plugin.Tvdb.xml already has an ApiKey (kept)\n'
        else
            printf '+ fill the empty ApiKey in Jellyfin.Plugin.Tvdb.xml\n'
        fi
    else
        printf '+ create Jellyfin.Plugin.Tvdb.xml with ApiKey\n'
    fi
 }
 write_plugin() {
    local tmp guid version other
    other="$(find "$PLUGINS" -maxdepth 1 -type d -name 'thetvdb_*' ! -name "$PLUGIN_DIR" -printf '%f\n' 2>/dev/null | sort | tr '\n' ' ')"
    [ -z "$other" ] ||
        die "another TheTVDB version is already installed ($other); move it to the trash first — two copies of one plugin means two loads"
    tmp="$(mktemp "${TMPDIR:-/tmp}/thetvdb.XXXXXX.zip")"
    # The pins are checked before anything is unpacked, so a bad artifact never
    # reaches the plugins directory.
    curl -fsSL --max-time 300 -o "$tmp" "$TVDB_PLUGIN_URL" ||
        { rm -f "$tmp"; die "could not fetch $TVDB_PLUGIN_URL"; }
    local got
    got="$(sha256sum "$tmp" | cut -d' ' -f1)"
    if [ "$got" != "$TVDB_PLUGIN_SHA256" ]; then
        rm -f "$tmp"
        die "artifact sha256 is $got, pinned $TVDB_PLUGIN_SHA256 — refusing to install"
    fi
    read -r guid version <<<"$(plugin_meta "$tmp")"
    if [ "$guid" != "$TVDB_PLUGIN_GUID" ]; then
        rm -f "$tmp"
        die "artifact guid is $guid, expected the official $TVDB_PLUGIN_GUID — refusing to install"
    fi
    if [ "$version" != "$TVDB_PLUGIN_VERSION" ]; then
        rm -f "$tmp"
        die "artifact version is $version, pinned $TVDB_PLUGIN_VERSION — refusing to install"
    fi
    mkdir -p "$PLUGINS/$PLUGIN_DIR"
    python3 -m zipfile -e "$tmp" "$PLUGINS/$PLUGIN_DIR/"
    rm -f "$tmp"
    [ -f "$PLUGINS/$PLUGIN_DIR/$TVDB_DLL" ] ||
        die "$PLUGIN_DIR has no $TVDB_DLL — the artifact does not match this plugin"
    printf '+ installed %s\n' "$PLUGIN_DIR"
 }
 backup_to_trash() {
    local f="$1"
    mkdir -p "$TRASH_DIR"
    local dest="$TRASH_DIR/$(basename "$f").$(date +%H%M%S)"
    cp -p "$f" "$dest"
    printf '+ previous copy kept at %s\n' "$dest"
 }
 cmd_write_keys() {
    local now want
    KEYS_CHANGED=0
    # TMDb: the running container owns this file and every setting in it. Only
    # the key element is touched, so country, language and adult settings
    # survive the edit.
    [ -f "$TMDB_XML" ] ||
        die "$TMDB_XML is missing — save the TMDb plugin settings once in the UI, then re-run"
    now="$(xml_value "$TMDB_XML" TmdbApiKey)"
    keys_len TMDB_API_KEY >/dev/null
    want="$(keys_value TMDB_API_KEY)"
    if [ "$now" = "$want" ]; then
        printf '= TmdbApiKey already set (%s chars)\n' "${#now}"
    else
        backup_to_trash "$TMDB_XML"
        xml_secret "$TMDB_XML" TmdbApiKey TMDB_API_KEY
        printf '+ set TmdbApiKey (%s chars)\n' "${#want}"
        KEYS_CHANGED=1
    fi
    # TVDB: never overwrite a config the operator saved through the UI.
    if [ -f "$TVDB_XML" ]; then
        now="$(xml_value "$TVDB_XML" ApiKey)"
        if [ -n "$now" ]; then
            printf '= Jellyfin.Plugin.Tvdb.xml already has an ApiKey (kept)\n'
        else
            keys_len TVDB_API_KEY >/dev/null
            backup_to_trash "$TVDB_XML"
            xml_secret "$TVDB_XML" ApiKey TVDB_API_KEY
            if [ -n "$(keys_value TVDB_PIN)" ] && [ -z "$(xml_value "$TVDB_XML" Pin)" ]; then
                xml_secret "$TVDB_XML" Pin TVDB_PIN
            fi
            printf '+ filled the ApiKey in Jellyfin.Plugin.Tvdb.xml\n'
            KEYS_CHANGED=1
        fi
    else
        keys_len TVDB_API_KEY >/dev/null
        cat >"$TVDB_XML" <<XML
 <?xml version="1.0" encoding="utf-8"?>
 <Configuration xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance" xmlns:xsd="http://www.w3.org/2001/XMLSchema">
  <ApiKey />
 </Configuration>
 XML
        xml_secret "$TVDB_XML" ApiKey TVDB_API_KEY
        if [ -n "$(keys_value TVDB_PIN)" ]; then
            xml_secret "$TVDB_XML" Pin TVDB_PIN
        fi
        printf '+ created Jellyfin.Plugin.Tvdb.xml with ApiKey\n'
        KEYS_CHANGED=1
    fi
 }
 cmd_plan() {
    check_root
    check_keys_report
    printf 'root  %s\n' "$JF_ROOT"
    plugin_state
    tmdb_state
    tvdb_state
 }
 cmd_install() {
    check_root
    check_keys_report
    KEYS_CHANGED=0
    if [ ! -d "$PLUGINS/$PLUGIN_DIR" ]; then
        write_plugin
        KEYS_CHANGED=1
    else
        printf '= %s already installed\n' "$PLUGIN_DIR"
    fi
    cmd_write_keys
    if [ "$KEYS_CHANGED" -eq 1 ]; then
        # Nothing is restarted here on purpose: this host may be playing media
        # right now, and both the plugin and its keys are read at start-up.
        printf 'next: scripts/deploy.sh dankserver --service jellyfin (the plugin and keys load at start-up)\n'
    else
        printf '= no change\n'
    fi
 }
 cmd_verify() {
    check_root
    local bad=0
    if [ ! -d "$PLUGINS/$PLUGIN_DIR" ]; then
        printf 'MISSING plugin directory %s (thetvdb not installed)\n' "$PLUGIN_DIR" >&2
        bad=1
    else
        local dlls abi
        dlls="$(find "$PLUGINS/$PLUGIN_DIR" -maxdepth 1 -name '*.dll' | wc -l)"
        [ "$dlls" -ge 1 ] || {
            printf 'MISSING %s inside %s\n' "$TVDB_DLL" "$PLUGIN_DIR" >&2
            bad=1
        }
        abi="$(python3 -c 'import json,sys; print(json.load(open(sys.argv[1])).get("targetAbi","?"))' \
            "$PLUGINS/$PLUGIN_DIR/meta.json" 2>/dev/null || echo '?')"
        printf 'plugin %s: %s dll(s), targetAbi %s\n' "$PLUGIN_DIR" "$dlls" "$abi"
    fi
    local pair elem n
    for pair in "$TMDB_XML|TmdbApiKey" "$TVDB_XML|ApiKey"; do
        elem="${pair##*|}"
        if [ ! -f "${pair%%|*}" ]; then
            printf 'MISSING %s\n' "${pair%%|*}" >&2
            bad=1
            continue
        fi
        if ! python3 -c 'import sys, xml.dom.minidom; xml.dom.minidom.parse(sys.argv[1])' "${pair%%|*}"; then
            printf 'UNPARSEABLE %s\n' "${pair%%|*}" >&2
            bad=1
            continue
        fi
        n="$(xml_value "${pair%%|*}" "$elem")"
        if [ -z "$n" ]; then
            printf 'MISSING key: %s has no %s value\n' "$(basename "${pair%%|*}")" "$elem" >&2
            bad=1
        else
            printf '%s %s chars\n' "$elem" "${#n}"
        fi
    done
    [ "$bad" -eq 0 ] || return 1
    printf 'providers configured\n'
 }
 cmd="${1:-}"
 case "$cmd" in
 plan) cmd_plan ;;
 install) cmd_install ;;
 verify) cmd_verify ;;
 *) usage ;;
 esac
@@ -0,0 +1,186 @@
 #!/usr/bin/env bash
 # kld-gate-qwen38.sh — RUNS ON DANKSERVER (or anywhere the reports are). The campaign's
 # acceptance gate: one verdict from the KLD reports of the containers that ride one reference
 # recording.
 #
 # The reference is the BF16 container built from the vendor checkpoint (run-qwen38-bf16.sh
 # kldref, in the same drained window that records the calibration grams). Every scored
 # container is measured against that one recording, so the KL scale runs from 0 = the
 # checkpoint itself downward in faithfulness.
 #
 # TWO MODES, because the disk decides how many anchors can exist at once:
 #
 #   gate IQ4R.json CAND.json                    candidate vs the served payload
 #   gate FP8.json IQ4R.json CAND.json           the same, inside the absolute interval
 #
 # iq4r.json is the payload frank-archon serves — the bar to beat. When the FP8-native release
 # (172.8 GiB, fetched last because of the peak-disk law) is on disk, pass its report first:
 # "half the gap" is then half of the distance between the 4-bit payload and the near-lossless
 # release, not half of an invented distance to the checkpoint. Without it the gate still
 # answers the campaign's question — does the candidate beat what is served — and says so.
 #
 # The law, on the summary of one slice (--slice all | a corpus source):
 #
 #   candidate vs served (2 reports)
 #   1. mean KL(cand) < mean KL(iq4r)         the served payload is beaten on the corpus
 #   2. p99.9(cand) <= p99.9(iq4r)            and the tail is no worse — a mean can hide a few
 #                                            positions taking the whole hit
 #   3. top-1(cand) >= top-1(iq4r)            the agreement the harness keys on does not fall
 #   4. every source slice passes 1-3         a win bought by regressing one distribution
 #                                            (code, prose, tools) is not a win
 #
 #   inside the interval (3 reports), laws 1-4 above (1-3 renumbered to the interval and the
 #   recovery fraction, as agreed 2026-10-27) plus:
 #   5. p99.9(cand) <= p99.9(iq4r)            the tail law
 #   6. every source slice passes            the slice law
 #
 # Booting the containers is NOT this script's job: `run-qwen38-bf16.sh score RAD OUT.json`
 # writes the reports and owns the single engine argv (same pin, --tp 2, --kv-cache-dtype fp8,
 # same step size, --deterministic, no speculation) that makes the numbers comparable.
 # This script is arithmetic, so it re-runs against old reports and is testable without a GPU.
 #
 # Usage: kld-gate-qwen38.sh IQ4R.json CAND.json [--slice NAME]
 #        kld-gate-qwen38.sh FP8.json IQ4R.json CAND.json [--slice NAME] [--recover F]
 set -uo pipefail
 if [ "$#" -lt 2 ]; then
    echo "usage: kld-gate-qwen38.sh IQ4R.json CAND.json [--slice NAME]" >&2
    echo "       kld-gate-qwen38.sh FP8.json IQ4R.json CAND.json [--slice NAME] [--recover F]" >&2
    exit 2
 fi
 python3 - "$@" <<'PY'
 import json, sys
 a = sys.argv[1:]
 opts, files = [], []
 i = 0
 while i < len(a):
    if a[i].startswith("--"):
        opts += [a[i], a[i + 1]]
        i += 2
    else:
        files.append(a[i])
        i += 1
 if len(files) not in (2, 3):
    sys.exit("usage: kld-gate-qwen38.sh IQ4R.json CAND.json [--slice NAME]\n"
             "       kld-gate-qwen38.sh FP8.json IQ4R.json CAND.json [--slice NAME] [--recover F]")
 o = dict(zip(opts[0::2], opts[1::2]))
 slice_ = o.get("--slice", "all")
 recover = float(o.get("--recover", 0.5))
 if not 0.0 < recover <= 1.0:
    sys.exit(f"--recover must be in (0,1], got {recover}")
 interval = len(files) == 3
 def die(msg):
    print(msg, file=sys.stderr)
    raise SystemExit(2)
 def load(path):
    try:
        return json.load(open(path))
    except Exception as e:
        die(f"cannot read the KLD report {path}: {e}")
 def summary(path, d):
    """The slice this gate grades: 'all', or one corpus source."""
    if slice_ == "all":
        g = d.get("all")
    else:
        g = (d.get("by_source") or {}).get(slice_)
        if g is None:
            die(f"{path} has no slice {slice_!r}; it has: "
                f"{', '.join(sorted(d.get('by_source') or {})) or '(none)'}")
    if g is None or "kld" not in g or "top1_agreement" not in g:
        die(f"{path} has no {slice_!r} summary — wrong report version?")
    return g
 docs = {label: load(path) for label, path in
        zip(("fp8", "iq4r", "cand") if interval else ("iq4r", "cand"), files)}
 rows = []
 for label, path in zip(("fp8", "iq4r", "cand") if interval else ("iq4r", "cand"), files):
    g = summary(path, docs[label])
    rows.append((label, path, docs[label].get("model", "?"), g["kld"]["mean"],
                 g["kld"]["p99_9"], g["top1_agreement"], g["positions"]))
 print(f"reference: BF16 container; slice: {slice_}; "
      + ("inside the FP8 interval" if interval else "candidate vs served (no FP8 anchor)"))
 print(f"{'':6} {'model':38} {'KL mean':>10} {'KL p99.9':>10} {'top-1':>8}  positions")
 for label, path, model, kl, p999, top1, n in rows:
    print(f"{label:6} {str(model)[:38]:38} {kl:10.6f} {p999:10.4f} {100*top1:7.2f}%  {n}")
 r = {label: (kl, p999, top1) for label, _p, _m, kl, p999, top1, _n in rows}
 kl_cand, p_cand, t_cand = r["cand"]
 kl_iq4r, p_iq4r, t_iq4r = r["iq4r"]
 ok = True
 def law(n, text, passed):
    global ok
    ok = ok and passed
    print(f"  [{'PASS' if passed else 'FAIL'}] {n}. {text}")
 def tail_law(n):
    law(n, f"candidate KL p99.9 {p_cand:.4f} <= served iq4r {p_iq4r:.4f}",
        p_cand <= p_iq4r)
 # --- the slice law: every corpus source must pass the metric laws -------------
 def slice_law(n):
    """A win that regresses one source class is not a win: check them all."""
    mine = docs["cand"].get("by_source") or {}
    theirs = docs["iq4r"].get("by_source") or {}
    if not mine or not theirs:
        law(n, "no per-source summaries to check", True)
        return
    missing = sorted(set(theirs) - set(mine))
    extra = sorted(set(mine) - set(theirs))
    if missing or extra:
        law(n, "same source slices in both reports"
            + (f" (missing from cand: {', '.join(missing)})" if missing else "")
            + (f" (only in cand: {', '.join(extra)})" if extra else ""), False)
        return
    bad = []
    for src in sorted(theirs):
        c, q = mine[src], theirs[src]
        worse_mean = c["kld"]["mean"] >= q["kld"]["mean"]
        worse_tail = c["kld"]["p99_9"] > q["kld"]["p99_9"]
        worse_top1 = c["top1_agreement"] < q["top1_agreement"]
        if worse_mean or worse_tail or worse_top1:
            why = ",".join(m for m, w in (("mean", worse_mean), ("p99.9", worse_tail),
                                          ("top-1", worse_top1)) if w)
            bad.append(f"{src} ({why})")
    law(n, "every source slice passes (mean, p99.9, top-1)"
        + (f" — regressed: {'; '.join(bad)}" if bad else ""), not bad)
 if interval:
    kl_fp8 = r["fp8"][0]
    gap = kl_iq4r - kl_fp8
    law(1, f"anchors ordered: fp8 {kl_fp8:.6f} < iq4r {kl_iq4r:.6f} (gap {gap:.6f})",
        gap > 0)
    law(2, f"candidate inside the interval: {kl_fp8:.6f} < {kl_cand:.6f} < {kl_iq4r:.6f}",
        kl_fp8 < kl_cand < kl_iq4r)
    need = kl_iq4r - recover * gap
    law(3, f"candidate closes {100*max((kl_iq4r - kl_cand)/gap, 0):.1f}% of the "
           f"{gap:.6f} gap (needs {100*recover:.0f}%, i.e. KL <= {need:.6f})"
           if gap > 0 else "gap is not positive; cannot score recovery",
        gap > 0 and kl_cand <= need)
    law(4, f"candidate top-1 {100*t_cand:.2f}% >= served iq4r top-1 {100*t_iq4r:.2f}%",
        t_cand >= t_iq4r)
    tail_law(5)
    slice_law(6)
 else:
    law(1, f"candidate beats the served payload: KL {kl_cand:.6f} < iq4r {kl_iq4r:.6f}",
        kl_cand < kl_iq4r)
    tail_law(2)
    law(3, f"candidate top-1 {100*t_cand:.2f}% >= served iq4r top-1 {100*t_iq4r:.2f}%",
        t_cand >= t_iq4r)
    slice_law(4)
 print("KLD GATE " + ("PASS" if ok else "FAIL"))
 sys.exit(0 if ok else 1)
 PY
@@ -34,14 +34,25 @@ import json
 import os
 import random
 import re
 import sys
 from jinja2 import Environment, FileSystemLoader
-from tokenizers import Tokenizer
+
 _HERE = os.path.dirname(os.path.abspath(__file__)) if "__file__" in globals() else os.getcwd()
 if _HERE not in sys.path:
    sys.path.insert(0, _HERE)
 import qwen38_common as qc  # noqa: E402 - sibling module, same directory
 CODE_EXT = (".sh", ".py", ".jinja", ".recipe", ".env", ".example",
            ".yaml", ".yml", ".bats", ".service", ".container", ".toml")
 JSONL_EXTS = (".jsonl", ".json")
 DOC_DIR = "serverAppConfigs-docs"
 FRAGMENT_TOKENS = 2400   # what one fragment may contribute
 CLASS_MIX = {"code": .45, "prose": .30, "tools": .25}
 HF_CAP = {"code": 40000, "prose": 60000, "tools": 0}
 POOL_CYCLES = 40         # how many times a class pool may repeat before giving up
 KLD_GUARD = 5000         # rendered conversations tried before declaring the pool short
 FLEET_CMDS = [
    "systemctl status frank-archon --no-pager",
@@ -142,7 +153,7 @@ def load_parquet_texts(path: str, kind: str, rng: random.Random,
    return out[:cap]
-def main() -> None:
+def parse_args() -> argparse.Namespace:
    ap = argparse.ArgumentParser(description=__doc__)
    ap.add_argument("--repo", required=True)
    ap.add_argument("--tokenizer-json", required=True)
@@ -156,22 +167,26 @@ def main() -> None:
    ap.add_argument("--hf-code-parquet", default="")
    ap.add_argument("--hf-prose-parquet", default="")
    ap.add_argument("--seed", type=int, default=20261006)
-    a = ap.parse_args()
+    return ap.parse_args()
-    rng = random.Random(a.seed)
+
-    tokenizer = Tokenizer.from_file(a.tokenizer_json)
+def load_template(template: str) -> tuple[Environment, str]:
-    env = Environment(loader=FileSystemLoader(os.path.dirname(a.template)),
+    """The chat template as the engine will render it, plus its short name."""
    env = Environment(loader=FileSystemLoader(os.path.dirname(template)),
                      trim_blocks=True, lstrip_blocks=True)
    env.filters["tojson"] = lambda v: json.dumps(v, ensure_ascii=False)
    env.filters["raise_exception"] = lambda m: (_ for _ in ()).throw(
        ValueError(m))
-    tname = os.path.basename(a.template)
+    return env, os.path.basename(template)
 def classify(repo: str) -> dict[str, list[str]]:
    """Walk the checkout into the three source classes the mix draws from."""
    code, prose, jsonl = [], [], []
-    for root, dirs, files in os.walk(a.repo):
+    for root, dirs, files in os.walk(repo):
        dirs[:] = [d for d in dirs if not d.startswith(".")]
        for f in sorted(files):
-            p = os.path.relpath(os.path.join(root, f), a.repo)
+            p = os.path.relpath(os.path.join(root, f), repo)
            if DOC_DIR in p and f.endswith(".md"):
                prose.append(p)
            elif p.startswith(DOC_DIR):
@@ -180,137 +195,185 @@ def main() -> None:
                code.append(p)
            elif f.endswith(JSONL_EXTS) and f != "package-lock.json":
                jsonl.append(p)
-    # file-level split: bucket 0 -> KLD pool, 1..4 -> calibration pool
+    return {"code": code, "prose": prose, "tools": jsonl}
    pools: dict[str, tuple[list, list]] = {}
    for cls, lst in (("code", code), ("prose", prose), ("tools", jsonl)):
        pools[cls] = ([p for p in lst if fnv(p) % 5 == 0],
                      [p for p in lst if fnv(p) % 5 != 0])
    def text_of(p: str) -> str:
        path = os.path.join(a.repo, p)
        raw = open(path, encoding="utf-8", errors="replace").read()
        if p.endswith(JSONL_EXTS):
            lines = [json.loads(l)["content"] for l in raw.splitlines()
                     if l.strip() and "content" in l]
            raw = "\n".join(lines)
        toks = tokenizer.encode(raw, add_special_tokens=False).ids
        return tokenizer.decode(toks[:2400], skip_special_tokens=True)
 def split_pools(classes: dict[str, list[str]]) -> dict[str, tuple[list, list]]:
    """File-level split: bucket 0 -> KLD pool, everything else -> calibration."""
    return {cls: ([p for p in lst if fnv(p) % 5 == 0],
                  [p for p in lst if fnv(p) % 5 != 0])
            for cls, lst in classes.items()}
 def jsonl_content(raw: str) -> str:
    """The content fields of a content-keyed JSONL, or "" for anything else.
    A .json in this repo is not necessarily a content-keyed JSONL: the substring
    test passes on any line that merely mentions "content", and a list-shaped
    document raises TypeError — both used to crash the whole build. Skip what
    does not fit instead.
    """
    lines = []
    for line in raw.splitlines():
        if not line.strip() or "content" not in line:
            continue
        try:
            rec = json.loads(line)
        except json.JSONDecodeError:
            continue
        if isinstance(rec, dict) and isinstance(rec.get("content"), str):
            lines.append(rec["content"])
    return "\n".join(lines)
 def fragment_text(repo: str, rel: str, tok) -> str:
    """One file as a fragment: its first FRAGMENT_TOKENS tokens, decoded."""
    raw = open(os.path.join(repo, rel), encoding="utf-8",
               errors="replace").read()
    if rel.endswith(JSONL_EXTS):
        raw = jsonl_content(raw)
    ids = tok.encode(raw, add_special_tokens=False).ids
    return tok.decode(ids[:FRAGMENT_TOKENS], skip_special_tokens=True)
 def build_frags(a: argparse.Namespace, tok,
                rng: random.Random) -> dict[str, dict[str, list[str]]]:
    """Both fragment pools per class, seeded so the same seed rebuilds the corpus."""
    pools = split_pools(classify(a.repo))
    cmds = harvest_cmds(a.repo, rng)
    # fragment pools per class, corpus-split first
    frags: dict[str, dict[str, list[str]]] = {}
-    for cls in pools:
+    for cls, (kld_files, cal_files) in pools.items():
        kld_files, cal_files = pools[cls]
        fr = {}
        src = {"kld": kld_files, "cal": cal_files}
        if cls == "tools":
-            fr["kld"] = [tool_call(rng, cmds) for _ in range(600)]
+            frags[cls] = {"kld": [tool_call(rng, cmds) for _ in range(600)],
-            fr["cal"] = [tool_call(rng, cmds) for _ in range(40000)]
+                          "cal": [tool_call(rng, cmds) for _ in range(40000)]}
        else:
-            fr["kld"] = [text_of(p) for p in src["kld"]]
+            frags[cls] = {"kld": [fragment_text(a.repo, p, tok) for p in kld_files],
-            fr["cal"] = [text_of(p) for p in src["cal"]]
+                          "cal": [fragment_text(a.repo, p, tok) for p in cal_files]}
        frags[cls] = fr
    # per-class fragment caps: prose rows are short (~66 tok) so need many;
    # codeparrot files are long (~2.6k tok) so a small cap suffices
    hfcap = {"code": 40000, "prose": 60000, "tools": 0}
    for cls, spec in (("code", a.hf_code_parquet),
                      ("prose", a.hf_prose_parquet)):
        for path in [p for p in spec.split(",") if p]:
            frags[cls]["cal"].extend(
-                load_parquet_texts(path, cls, rng, cap=hfcap[cls]))
+                load_parquet_texts(path, cls, rng, cap=HF_CAP[cls]))
    for cls in frags:
        rng.shuffle(frags[cls]["cal"])
    return frags
    def toks(s: str) -> int:
        return len(tokenizer.encode(s, add_special_tokens=False).ids)
-    def emit_calib() -> tuple[int, list[dict]]:
+def pack_class(cls: str, pool: list[str], want: int, tok,
-        targets = {"code": .45, "prose": .30, "tools": .25}
+               chunk_tokens: int) -> list[dict]:
-        rows: list[dict] = []
+    """Greedily pack `pool` into chunk_tokens-sized calibration prompts."""
-        for cls, frac in targets.items():
+    if not pool:
-            want = int(a.calib_tokens * frac)
+        return []
-            pool = frags[cls]["cal"]
+    rows: list[dict] = []
    # Count every fragment once: the loop cycles the pool up to POOL_CYCLES
    # times, and re-tokenizing a 2.4k-token fragment on each pass dominated the
    # build's wall time.
    counts = [qc.n_tokens(tok, f) for f in pool]
    got = 0
    i = 0
    buf: list[str] = []
    nbuf = 0
    while got < want:
        j = i % len(pool)
        frag = pool[j]
        i += 1
        n = counts[j]
        if nbuf + n > chunk_tokens and buf:
            rows.append({"prompt": "\n".join(buf), "cls": cls})
            got += nbuf
            buf, nbuf = [], 0
            if got >= want:
                break
        buf.append(frag)
        nbuf += n
        if i > len(pool) * POOL_CYCLES:
            break  # pool exhausted with repeats; stop
    if buf and got < want:
        rows.append({"prompt": "\n".join(buf), "cls": cls})
        got += nbuf
    print(f"calib {cls}: {got} tokens, {i} fragments")
    return rows
 def emit_calib(a: argparse.Namespace, frags, tok,
               rng: random.Random) -> tuple[int, list[dict]]:
    """calib.jsonl rows: the class mix, shuffled, prompts only."""
    rows: list[dict] = []
    for cls, frac in CLASS_MIX.items():
        rows.extend(pack_class(cls, frags[cls]["cal"], int(a.calib_tokens * frac),
                               tok, a.chunk_tokens))
    rng.shuffle(rows)
    return len(rows), [{"prompt": r["prompt"]} for r in rows]
 def emit_kld(a: argparse.Namespace, frags, tok, rng: random.Random,
             env: Environment, tname: str) -> tuple[int, list[dict], int]:
    """kld.jsonl rows: multi-turn conversations, each with a scored tail."""
    template = env.get_template(tname)
    rows: list[dict] = []
    total = scored_total = 0
    classes = ["code", "prose", "tools"]
    cursor = {c: 0 for c in classes}
    guard = 0
    while total < a.kld_tokens and guard < KLD_GUARD:
        guard += 1
        msgs, srcs = [], []
        for t in range(rng.choice([2, 3, 4])):
            cls = classes[(len(rows) + t) % 3]
            pool = frags[cls]["kld"]
            if not pool:
                continue
-            got, i, buf, nbuf = 0, 0, [], 0
+            frag = pool[cursor[cls] % len(pool)]
-            while got < want:
+            cursor[cls] += 1
-                frag = pool[i % len(pool)]
+            msgs.append({"role": "user",
-                i += 1
+                         "content": "Summarize and explain this "
-                n = toks(frag)
+                                    f"engineering material:\n{frag}"})
-                if nbuf + n > a.chunk_tokens and buf:
+            msgs.append({"role": "assistant",
-                    rows.append({"prompt": "\n".join(buf), "cls": cls})
+                         "content": frag[:4000]})
-                    got += nbuf
+            srcs.append(cls)
-                    buf, nbuf = [], 0
+        rendered = template.render(messages=msgs, add_generation_prompt=False)
-                    if got >= want:
+        n = qc.n_tokens(tok, rendered)
-                        break
+        if n < a.kld_min_prompt:
-                buf.append(frag)
+            continue
-                nbuf += n
+        rows.append({"prompt": rendered,
-                if i > len(pool) * 40:
+                     "score_from": n - a.score_tail,
-                    break  # pool exhausted with repeats; stop
+                     "source": "+".join(sorted(set(srcs)))})
-            if buf and got < want:
+        total += n
-                rows.append({"prompt": "\n".join(buf), "cls": cls})
+        scored_total += a.score_tail
-                got += nbuf
+    return len(rows), rows, scored_total
            print(f"calib {cls}: {got} tokens, {i} fragments")
        rng.shuffle(rows)
        return len(rows), [{k: v for k, v in r.items() if k == "prompt"}
                           for r in rows]
    def emit_kld() -> tuple[int, list[dict]]:
        rows, total = [], 0
        classes = ["code", "prose", "tools"]
        i = {c: 0 for c in classes}
        guard = 0
        while total < a.kld_tokens and guard < 5000:
            guard += 1
            msgs, srcs = [], []
            for t in range(rng.choice([2, 3, 4])):
                cls = classes[(len(rows) + t) % 3]
                pool = frags[cls]["kld"]
                if not pool:
                    continue
                frag = pool[i[cls] % len(pool)]
                i[cls] += 2
                msgs.append({"role": "user",
                             "content": "Summarize and explain this "
                                        f"engineering material:\n{frag}"})
                msgs.append({"role": "assistant",
                             "content": frag[:4000]})
                srcs.append(cls)
            rendered = env.get_template(tname).render(
                messages=msgs, add_generation_prompt=False)
            n = toks(rendered)
            if n < a.kld_min_prompt:
                continue
            rows.append({"prompt": rendered,
                         "score_from": n - a.score_tail,
                         "source": "+".join(sorted(set(srcs)))})
            total += n
        return len(rows), rows
-    os.makedirs(a.out_dir, exist_ok=True)
+def manifest(a: argparse.Namespace, tname: str, nc: int, nk: int,
-    nc, rows = emit_calib()
+             scored: int) -> dict:
-    with open(os.path.join(a.out_dir, "calib.jsonl"), "w") as f:
+    return {
        for r in rows:
            f.write(json.dumps(r) + "\n")
    nk, krows = emit_kld()
    with open(os.path.join(a.out_dir, "kld.jsonl"), "w") as f:
        for r in krows:
            f.write(json.dumps(r) + "\n")
    scored = sum(len(tokenizer.encode(r["prompt"], add_special_tokens=False).ids)
                 - r["score_from"] for r in krows)
    manifest = {
        "seed": a.seed, "template": tname,
        "tokenizer": os.path.basename(a.tokenizer_json),
        "calib_prompts": nc, "kld_prompts": nk,
        "kld_scored_positions": scored,
        "kld_score_tail": a.score_tail,
        "disjoint_file_pools": "local KLD material only; HF code feeds calib",
-        "calib_mix": {"code": .45, "prose": .30, "tools": .25},
+        "calib_mix": dict(CLASS_MIX),
        "hf_code": a.hf_code_parquet, "hf_prose": a.hf_prose_parquet,
    }
-    with open(os.path.join(a.out_dir, "manifest.json"), "w") as f:
+
-        json.dump(manifest, f, indent=1)
+
-    print(json.dumps(manifest, indent=1))
+def main() -> None:
    a = parse_args()
    rng = random.Random(a.seed)
    tok = qc.load_tokenizer(a.tokenizer_json)
    env, tname = load_template(a.template)
    frags = build_frags(a, tok, rng)
    os.makedirs(a.out_dir, exist_ok=True)
    nc, calib_rows = emit_calib(a, frags, tok, rng)
    qc.write_jsonl(os.path.join(a.out_dir, "calib.jsonl"), calib_rows)
    nk, kld_rows, scored = emit_kld(a, frags, tok, rng, env, tname)
    qc.write_jsonl(os.path.join(a.out_dir, "kld.jsonl"), kld_rows)
    man = manifest(a, tname, nc, nk, scored)
    qc.write_json(os.path.join(a.out_dir, "manifest.json"), man)
    print(json.dumps(man, indent=1))
 if __name__ == "__main__":
@@ -0,0 +1,434 @@
 #!/usr/bin/env python3
 """media-apply.py — the only thing that moves movie-library files.
 Reads the plan written by scripts/media-plan.py (rows: `action  src  dst  kind`,
 plus `mkdir` rows) and performs it with rename(2) — bytes are never copied, so
 an interrupted run has destroyed nothing. Every run is resumable and every run
 is undoable: the journal is a write-ahead log, one JSON line per intent, fsynced
 BEFORE the rename it authorises. That ordering is the whole design: a crash
 leaves either an intent whose move happened or an intent whose move never ran,
 and both are recognised on the next run. A file that moved with no journal line
 cannot have been moved by this tool, and saying so is better than guessing.
 Four subcommands:
    apply    --plan PLAN [--root ROOT] [--journal J] [--apply]
             no --apply => report only and write NOTHING, not even folders
    rollback --journal J [--root ROOT] [--apply]
             back to the paths the census saw; folders this run created are
             removed only when empty, never anything else
    verify   --plan PLAN --journal J [--root ROOT]
             every planned file is at its target with the journalled byte count
    sweep    --journal J [--root ROOT] [--apply]
             removes the folders THIS RUN emptied — the journal has to show a
             file that moved out of one, so a folder someone made by hand can
             never be removed by this tool, and rollback puts them back
 Refusals (exit 2, before the first rename, with nothing touched): a target
 already occupied by a foreign file, a source that has gone, a duplicate target,
 a path outside --root, a journal that was already rolled back, a move that would
 cross a filesystem (rename cannot do it and a silent copy at 24 TB would not).
 --force relaxes only the guards that are opinion (journal already rolled back,
 paths outside --root); it never allows overwriting a file.
 The journal defaults to `media-relayout.jsonl` BESIDE the library root, never
 inside it: a log inside the library is a file Jellyfin will index.
 """
 import argparse
 import datetime
 import json
 import os
 import sys
 DEFAULT_ROOT = os.environ.get("MEDIA_ROOT", "/mnt/entropy/Movies")
 class Refuse(Exception):
    """A gate the operator has to see; every refusal leaves the library alone."""
 def journal_default(root):
    return os.path.join(os.path.dirname(os.path.abspath(root)), "media-relayout.jsonl")
 class Journal:
    """Append-only JSONL of intents. Writes are fsynced before the tool acts."""
    def __init__(self, path, writing):
        self.path, self.writing = path, writing
        self.records = []
        if os.path.exists(path):
            for line in open(path):
                line = line.strip()
                if line:
                    self.records.append(json.loads(line))
        self._fh = None
    def already(self, dst, src):
        """An intent for this exact (src, dst) was written by an earlier run."""
        return any(r.get("op") == "move" and r.get("dst") == dst
                   and r.get("src") == src for r in self.records)
    @property
    def rolled_back(self):
        return any(r.get("op") == "rollback" for r in self.records)
    def record(self, **fields):
        self.records.append(fields)
        if not self.writing:
            return
        if self._fh is None:
            self._fh = open(self.path, "a")
        self._fh.write(json.dumps(fields) + "\n")
        self._fh.flush()
        os.fsync(self._fh.fileno())
    def close(self):
        if self._fh:
            self._fh.close()
 def read_plan(path, root, force):
    """Plan rows: (mkdirs, keeps, moves), header dropped, all paths under root."""
    mkdirs, keeps, moves = [], [], []
    with open(path) as fh:
        header = fh.readline()
        if not header.startswith("action\t"):
            raise Refuse(f"{path} is not a plan (first column is "
                         f"{header.split(chr(9))[0]!r}, expected 'action') —"
                         " regenerate it with scripts/media-plan.py")
        for line in fh:
            f = line.rstrip("\n").split("\t")
            if f[0] == "mkdir":
                mkdirs.append(f[2])
            elif f[0] == "keep":
                keeps.append(f[1])
            elif f[0] == "move":
                moves.append((f[1], f[2], f[3] if len(f) > 3 else ""))
    if not force:
        for p in mkdirs + keeps + [p for m in moves for p in m[:2]]:
            if p and not (p == root or p.startswith(root + os.sep)):
                raise Refuse(f"{p} is outside --root {root}"
                             " (a plan built for another library?)")
    return mkdirs, keeps, moves
 # A relayout moves bytes on the raw array; the sidecars the media servers wrote
 # live in the SSD overlay and stay at the old coordinates. Naming the follow-up
 # here is the point of use for that fact — see the Media Metadata View page.
 METADATA_HINT = ("note: sidecars written through the metadata view did not follow"
                 " these moves — next: scripts/media-view.sh reap")
 def do_apply(args, root, journal, mkdirs, keeps, moves):
    """Preflight everything, then folders, then renames. Never a partial start."""
    problems = []
    for p in keeps:
        if not os.path.exists(p):
            problems.append(f"missing: keep row source is gone: {p}")
    seen = {}
    for src, dst, _kind in moves:
        if src == dst:
            problems.append(f"no-op: move row whose target is its source: {src}")
        if dst in seen:
            problems.append(f"double-claim: {dst} claimed by {seen[dst]} and {src}")
        seen[dst] = src
        if journal.already(dst, src) and os.path.exists(dst) and not os.path.exists(src):
            continue                                    # this intent already ran
        if not os.path.exists(src):
            if os.path.exists(dst):
                problems.append(f"missing: {src} is gone and {dst} carries no journal"
                                " line — it moved outside this tool, refusing to guess")
            else:
                problems.append(f"missing: {src} is gone and {dst} is not there either")
            continue
        if os.path.exists(dst):
            problems.append(f"occupied: target already holds another file: {dst}")
    if problems:
        print(f"refusing: {len(problems)} problem(s), nothing moved")
        for p in sorted(problems)[:20]:
            print("  " + p)
        if len(problems) > 20:
            print(f"  ... and {len(problems) - 20} more")
        return 2
    if not args.apply:
        print(f"dry run: {len(moves)} moves, {len(mkdirs)} folders, {len(keeps)} files stay"
              " — nothing written (pass --apply to run)")
        for src, dst, _k in moves[:5]:
            print(f"  would move {src} -> {dst}")
        return 0
    made = {r.get("dst") for r in journal.records if r.get("op") == "mkdir"}
    dirs_new = 0
    for d in sorted(mkdirs, key=lambda q: q.count(os.sep)):
        if d in made or os.path.isdir(d):
            continue
        journal.record(op="mkdir", dst=d)
        os.makedirs(d, exist_ok=True)
        dirs_new += 1
    # same-filesystem gate: rename cannot cross a device, and a silent copy of a
    # library this size is not an option anyone would choose.
    cross = []
    for src, dst, _k in moves:
        if journal.already(dst, src) and os.path.exists(dst) and not os.path.exists(src):
            continue
        parent = os.path.dirname(dst)
        if os.path.exists(parent) and os.stat(parent).st_dev != os.stat(src).st_dev:
            cross.append(dst)
    if cross:
        print(f"refusing: {len(cross)} moves would cross a filesystem"
              f" (first: {cross[0]}) — this tool only renames")
        return 2
    moved = done = 0
    for src, dst, _kind in moves:
        if journal.already(dst, src):
            if os.path.exists(dst) and not os.path.exists(src):
                done += 1
                continue
            if os.path.exists(src) and not os.path.exists(dst):
                os.rename(src, dst)             # the intent ran, the move did not
                moved += 1
                continue
        st = os.stat(src)
        journal.record(op="move", src=src, dst=dst, bytes=st.st_size, mode=st.st_mode,
                       mtime=st.st_mtime, ino=st.st_ino, dev=st.st_dev)
        os.rename(src, dst)
        moved += 1
    print(f"applied: {moved} moved, {done} already done in an earlier run,"
          f" {dirs_new} folders created, {len(keeps)} files stayed put")
    print(f"journal: {journal.path} — undo with: media-apply.py rollback"
          f" --journal {journal.path} --root {root} --apply")
    print(METADATA_HINT)
    return 0
 def do_verify(root, journal, moves, keeps):
    """Every planned file sits at its target carrying the journalled bytes."""
    journalled = {(r["src"], r["dst"]): r for r in journal.records
                  if r.get("op") == "move"}
    todo = moves or [(r["src"], r["dst"], "") for r in journal.records
                     if r.get("op") == "move"]
    bad, missing, bytes_total = [], [], 0
    for src, dst, _kind in todo:
        rec = journalled.get((src, dst))
        if not os.path.exists(dst):
            missing.append(dst)
            continue
        size = os.path.getsize(dst)
        bytes_total += size
        if rec and rec.get("bytes") not in (None, size):
            bad.append(f"{dst}: {size} bytes on disk, {rec['bytes']} at move time")
        if os.path.exists(src) and src != dst:
            bad.append(f"{src}: still in place as well as at {dst}")
    missing += [p for p in keeps if not os.path.exists(p)]
    if bad or missing:
        print(f"verify FAILED: {len(missing)} absent, {len(bad)} wrong")
        for line in (missing + bad)[:20]:
            print("  " + line)
        return 2
    empty = [r["dst"] for r in journal.records
             if r.get("op") == "mkdir" and os.path.isdir(r["dst"])
             and not os.listdir(r["dst"])]
    print(f"verified: {len(todo)} files at their targets"
          f" ({bytes_total / 2**40:.2f} TiB), {len(keeps)} kept")
    if empty:
        print(f"note: {len(empty)} folders are empty (left in place — see the trash law)")
    return 0
 def sweep_targets(root, journal):
    """The folders this run emptied, deepest first — never one it did not touch.
    Eligibility is evidence, not appearance. A folder is a target only when the
    journal shows a file moved out of it or out of something beneath it, it holds
    nothing but empty directories now, it is not a folder this run created, and no
    journalled destination lives under it. A symlink anywhere inside disqualifies
    the whole subtree: this tool never follows a link, and deleting the folder
    that holds one would delete someone's mount point.
    Proof carries downward: once a folder qualifies, every empty directory inside
    it qualifies too. A release's junk (``Screens/``, ``Sample/``) is never itself
    a move source, so without that rule it holds its emptied parent on disk
    forever — the parent can never be removed while the child stands.
    """
    moves = [r for r in journal.records if r.get("op") == "move"]
    made = {r["dst"] for r in journal.records if r.get("op") == "mkdir"}
    # An rmdir INTENT that failed left its folder standing, and the next sweep
    # must be able to finish it: only a folder that is really gone is out of
    # scope. Counting every intent as done made 146 live survivors permanent.
    gone = {r["path"] for r in journal.records if r.get("op") == "rmdir"
            and not os.path.exists(r["path"])}
    dsts = [r["dst"] for r in moves]
    marked = set()
    for r in moves:
        d = os.path.dirname(r["src"])
        while d.startswith(root + os.sep):
            marked.add(d)
            d = os.path.dirname(d)
    def removable(d):
        if d in made or d in gone or os.path.islink(d):
            return False
        if any(p == d or p.startswith(d + os.sep) for p in dsts):
            return False
        try:
            entries = os.listdir(d)
        except OSError:
            return False
        for e in entries:
            c = os.path.join(d, e)
            if os.path.islink(c) or not os.path.isdir(c) or not removable(c):
                return False
        return True
    roots = [d for d in marked if removable(d)]
    targets = set(roots)
    for d in roots:
        for dirpath, dirnames, _ in os.walk(d):
            for c in dirnames:
                p = os.path.join(dirpath, c)
                if removable(p):
                    targets.add(p)
    return sorted(targets, key=lambda q: (-q.count(os.sep), q))
 def do_sweep(args, root, journal):
    """Remove the empty husks an apply left standing at the library root."""
    moves = [r for r in journal.records if r.get("op") == "move"]
    if not moves:
        raise Refuse("the journal holds no moves — a sweep follows an apply, it"
                     " does not guess at emptiness")
    outside = sorted({p for r in moves for p in (r["src"], r["dst"])
                      if not (p == root or p.startswith(root + os.sep))})
    if outside and not args.force:
        raise Refuse(f"{len(outside)} journalled paths are outside --root {root}"
                     f" (first: {outside[0]}) — this journal belongs to another"
                     " library")
    targets = sweep_targets(root, journal)
    if not targets:
        print("swept: 0 folders — nothing this run emptied is still standing empty")
        return 0
    if not args.apply:
        print(f"dry run: {len(targets)} folders this run emptied would be removed"
              " (pass --apply to run)")
        for d in targets[:5]:
            print(f"  would remove {d}")
        return 0
    removed = 0
    for d in targets:
        journal.record(op="rmdir", path=d)
        try:
            os.rmdir(d)
        except OSError as e:
            print(f"left in place: {d} ({e})")
            continue
        removed += 1
    print(f"swept: {removed} folders removed, {len(targets) - removed} left in place")
    print(f"journal: {journal.path} — rollback recreates them with their files")
    return 0
 def do_rollback(args, root, journal):
    """Undo every journalled move, newest first, after a full preflight."""
    moves = [r for r in journal.records if r.get("op") == "move"]
    made = [r["dst"] for r in journal.records if r.get("op") == "mkdir"]
    swept = [r["path"] for r in journal.records if r.get("op") == "rmdir"]
    problems = []
    for r in reversed(moves):
        src, dst = r["src"], r["dst"]
        if not args.force and not (src.startswith(root + os.sep)
                                   or dst.startswith(root + os.sep)):
            problems.append(f"outside --root: {dst}")
            continue
        if os.path.exists(dst) and os.path.exists(src):
            problems.append(f"occupied: both paths exist, refusing to choose:"
                            f" {src} | {dst}")
        elif os.path.exists(dst):
            pass
        elif os.path.exists(src):
            continue                                    # already undone
        else:
            problems.append(f"missing: {dst} is gone and {src} never came back")
    if problems:
        print(f"refusing: {len(problems)} problem(s), nothing moved back")
        for p in problems[:20]:
            print("  " + p)
        return 2
    if not args.apply:
        print(f"dry run: would move {len(moves)} files back to their census paths"
              f" and recreate {len(set(swept))} swept folders (pass --apply)")
        return 0
    undone = 0
    recreated = 0
    for d in sorted(set(swept), key=lambda q: q.count(os.sep)):
        if not os.path.isdir(d):
            os.makedirs(d, exist_ok=True)
            recreated += 1
    for r in reversed(moves):
        if os.path.exists(r["dst"]) and not os.path.exists(r["src"]):
            os.makedirs(os.path.dirname(r["src"]), exist_ok=True)
            os.rename(r["dst"], r["src"])
            undone += 1
    dirs_gone = 0
    for d in sorted(set(made), key=lambda q: -q.count(os.sep)):
        if d == root or not os.path.isdir(d) or os.listdir(d):
            continue
        try:
            os.rmdir(d)
            dirs_gone += 1
        except OSError as e:
            print(f"left in place: {d} ({e})")
    journal.record(op="rollback", undone=undone, dirs_removed=dirs_gone,
                   t=datetime.datetime.now(datetime.timezone.utc).isoformat())
    print(f"rolled back: {undone} files back where the census found them,"
          f" {dirs_gone} empty folders removed, {recreated} swept folders recreated")
    print(METADATA_HINT)
    return 0
 def main():
    ap = argparse.ArgumentParser(description="apply or undo a media relayout plan")
    ap.add_argument("cmd", choices=["apply", "rollback", "verify", "sweep"])
    ap.add_argument("--plan")
    ap.add_argument("--root", default=DEFAULT_ROOT)
    ap.add_argument("--journal")
    ap.add_argument("--apply", action="store_true", help="actually change the library")
    ap.add_argument("--force", action="store_true", help="relax the opinion guards")
    args = ap.parse_args()
    root = os.path.abspath(args.root)
    if not os.path.isdir(root):
        raise Refuse(f"no library at {root}")
    journal_path = args.journal or journal_default(root)
    journal = Journal(journal_path, args.apply)
    try:
        if args.cmd in ("rollback", "verify", "sweep") and not os.path.exists(journal_path):
            raise Refuse(f"no journal at {journal_path} — nothing to {args.cmd}")
        if args.cmd in ("rollback", "sweep") and not args.force and journal.rolled_back:
            raise Refuse("this journal has already been rolled back"
                         " (--force to replay it)")
        if args.cmd in ("apply", "verify") and not args.plan:
            raise Refuse(f"{args.cmd} needs --plan")
        mkdirs, keeps, moves = (read_plan(args.plan, root, args.force)
                                if args.plan else ([], [], []))
        if args.cmd == "apply":
            return do_apply(args, root, journal, mkdirs, keeps, moves)
        if args.cmd == "verify":
            return do_verify(root, journal, moves, keeps)
        if args.cmd == "sweep":
            return do_sweep(args, root, journal)
        return do_rollback(args, root, journal)
    finally:
        journal.close()
 if __name__ == "__main__":
    try:
        sys.exit(main())
    except Refuse as r:
        print(f"refusing: {r}")
        sys.exit(2)
@@ -0,0 +1,304 @@
 #!/usr/bin/env python3
 r"""Read-only census of a media library — the inputs a relayout plan is built from.
  media-census.py enumerate [--root DIR] [--out TSV]
  media-census.py probe     [--enum TSV] [--out TSV] [--jobs N]
  media-census.py plex      [--root DIR] [--db PATH] [--out TSV]
  media-census.py jellyfin  [--root DIR] [--db PATH] [--container PREFIX] [--out TSV]
 Run it on the machine that mounts the library. Nothing here writes inside the
 library and nothing is deleted; the only outputs are the TSVs below, and the
 scripts/media-plan.py and scripts/media-shows.py read them instead of touching media.
  enum    bytes <TAB> ext <TAB> path                  every file under the root
  probed  bytes <TAB> path <TAB> dur <TAB> vcodec <TAB> WxH <TAB> fps <TAB> vbit
          <TAB> pix <TAB> audio <TAB> subs <TAB> spare <TAB> format
          one row per video-container file; `audio` is "codec:lang+codec:lang",
          `subs` the subtitle-stream count, `spare` the audio streams carrying
          neither a default nor a forced tag
  plex    file <TAB> leaf_title <TAB> … (see PLEX_COLS)
          the titles Plex already matched — the authority over any filename guess
  jellyfin file <TAB> series <TAB> year <TAB> season <TAB> episode <TAB> extra
          <TAB> tmdb      (see JF_COLS)
          one row per episode Jellyfin has matched and numbered, its path mapped
          from the container's view back to the host; `year` and `tmdb` are the
          SERIES' values, `extra` is Jellyfin's own extra type when the row is
          bonus material rather than an episode
 `probe` is the slow step (every container is opened for its header, ~10 min for
 4,800 files on a spindle array at 6 workers) so it is `nice`d: a census beside a
 live service must not starve it. `enumerate` is a stat pass, so re-censusing
 before a plan is cheap — and a plan is only ever as good as these files.
 """
 import argparse
 import csv
 import json
 import os
 import re
 import sqlite3
 import subprocess
 import sys
 from concurrent.futures import ThreadPoolExecutor
 # Containers ffprobe can read. .iso/.img are deliberately NOT here: a disc image is
 # a mounted volume, not a stream, and the planner treats it as a movie file instead.
 PROBE_EXT = {".mkv", ".mp4", ".avi", ".m4v", ".mov", ".ts", ".m2ts", ".mpg",
             ".mpeg", ".webm", ".wmv", ".divx", ".ogm", ".bdmv", ".vob"}
 # Trash and thumbnail caches are not library content, and this repo's trash policy
 # means .AGENTTRASH can sit inside a movie folder at any time.
 SKIP_DIR = re.compile(r"(^|/)(\.AGENTTRASH|\.thumbnails|\.Trash-\d+|@eaDir|lost\+found)(/|$)")
 PLEX_DB = ("/var/lib/plexmediaserver/Library/Application Support/Plex Media Server"
           "/Plug-in Support/Databases/com.plexapp.plugins.library.db")
 PLEX_COLS = ["file", "leaf_title", "leaf_orig", "leaf_edition", "leaf_year",
             "parent_title", "parent_orig", "parent_year", "parent_edition",
             "container", "vcodec", "acodec", "width", "height"]
 PLEX_SQL = """
 select p.file, coalesce(mi.title,''), coalesce(mi.original_title,''),
       coalesce(mi.edition_title,''), coalesce(mi.year,0),
       coalesce(par.title,''), coalesce(par.original_title,''),
       coalesce(par.year,0), coalesce(par.edition_title,''),
       m.container, m.video_codec, m.audio_codec, m.width, m.height
 from media_parts p
 join media_items m on m.id = p.media_item_id
 join metadata_items mi on mi.id = m.metadata_item_id
 left join metadata_items par on par.id = mi.parent_id
 where p.file like ? and p.deleted_at is null and mi.deleted_at is null
 """
 JF_DB = "/home/dank/Containers/jellyfin/config/data/jellyfin.db"
 JF_COLS = ["file", "series", "year", "season", "episode", "extra", "tmdb"]
 # Jellyfin stores the entity's .NET class as its type; the TV tree is these three.
 JF_TV = ("MediaBrowser.Controller.Entities.TV.Episode",
         "MediaBrowser.Controller.Entities.TV.Season",
         "MediaBrowser.Controller.Entities.TV.Series")
 JF_SQL = ("""select Id, Path, IndexNumber, ParentIndexNumber, ExtraType, IsVirtualItem
              from BaseItems where Type = ? and Path like ?""")
 def enumerate_lib(root, out):
    """One row per real file under root: size, extension, path."""
    rows = []
    for dirpath, dirnames, filenames in os.walk(root):
        dirnames[:] = [d for d in dirnames
                       if not SKIP_DIR.search(os.path.join(dirpath, d))]
        for fn in filenames:
            p = os.path.join(dirpath, fn)
            if SKIP_DIR.search(p):
                continue
            try:
                st = os.lstat(p)
            except OSError:
                continue
            if not os.path.isfile(p) or os.path.islink(p):
                continue                # links and sockets are not library content
            rows.append((st.st_size, os.path.splitext(fn)[1].lower(), p))
    with open(out, "w") as fh:
        for size, ext, p in rows:
            fh.write(f"{size}\t{ext}\t{p}\n")
    vid = [r for r in rows if r[1] in PROBE_EXT]
    print(f"files={len(rows)} bytes={sum(r[0] for r in rows)} "
          f"probes={len(vid)} probe_bytes={sum(r[0] for r in vid)} -> {out}")
 def probe_one(path):
    """ffprobe header read -> the probed TSV row (never the whole file)."""
    try:
        os.nice(19)                     # a census must never outrank the media server
    except OSError:
        pass
    try:
        size = os.stat(path).st_size
    except OSError:
        return None
    cmd = ["ffprobe", "-v", "error", "-print_format", "json",
           "-show_format", "-show_streams", path]
    try:
        done = subprocess.run(cmd, capture_output=True, text=True, timeout=180)
        data = json.loads(done.stdout or "{}")
    except (subprocess.TimeoutExpired, subprocess.SubprocessError,
            json.JSONDecodeError, OSError):
        data = {}
    st = data.get("streams", [])
    fmt = data.get("format", {})
    v = next((s for s in st if s.get("codec_type") == "video"), None)
    aud = [s for s in st if s.get("codec_type") == "audio"]
    subs = [s for s in st if s.get("codec_type") == "subtitle"]
    fps = "-"
    if v and v.get("r_frame_rate"):
        try:
            n, d = (int(x) for x in v["r_frame_rate"].split("/", 1))
            fps = f"{round(n / d * 100) / 100:g}" if d else "0"
        except ValueError:
            pass
    def num(x):
        try:
            return round(float(x))
        except (TypeError, ValueError):
            return 0
    return [size, path, num(fmt.get("duration")),
            (v or {}).get("codec_name", "-"),
            f"{v['width']}x{v['height']}" if v else "-",
            fps, num(fmt.get("bit_rate")), (v or {}).get("pix_fmt", "-"),
            "+".join(f"{a.get('codec_name', '?')}:{a.get('language', '?')}" for a in aud),
            len(subs),
            sum(1 for a in aud if not re.search(
                r"default|forced", json.dumps(a.get("tags", {})), re.I)),
            fmt.get("format_name", "-") or "-"]
 def probe(enum, out, jobs):
    """Probe every video-container row of the enum TSV, in parallel."""
    paths = []
    with open(enum) as fh:
        for line in fh:
            f = line.rstrip("\n").split("\t", 2)
            if len(f) == 3 and f[1] in PROBE_EXT:
                paths.append(f[2])
    print(f"probing {len(paths)} files with {jobs} workers", flush=True)
    done = unreadable = 0
    with open(out, "w") as fh, ThreadPoolExecutor(max_workers=jobs) as pool:
        for row in pool.map(probe_one, paths):
            if row is None:
                continue
            if row[3] == "-":
                unreadable += 1         # header unreadable: the row still gets written
            done += 1
            fh.write("\t".join(str(x) for x in row) + "\n")
            if done % 250 == 0:
                print(f"  {done}/{len(paths)}", flush=True)
    print(f"probed={done} unreadable={unreadable} -> {out}")
 def plex(root, db, out):
    """Plex's matched titles for files under root, from its DB, read-only."""
    uri = f"file:{db}?mode=ro&immutable=1"
    try:
        con = sqlite3.connect(uri, uri=True)
    except sqlite3.Error as e:
        sys.exit(f"cannot open the Plex database read-only ({db}): {e}\n"
                 "Plex runs rootless here: run this as the container's owner, never "
                 "with sudo — rootless storage is per-user, so root sees nothing.")
    with open(out, "w", newline="") as fh:
        w = csv.writer(fh, delimiter="\t", lineterminator="\n")
        w.writerow(PLEX_COLS)
        n = 0
        for r in con.execute(PLEX_SQL, (root + "/%",)):
            w.writerow([str(x).replace("\t", " ").replace("\n", " ") for x in r])
            n += 1
    print(f"plex-matched={n} -> {out}")
 def jellyfin(root, db, container, out):
    """The episodes Jellyfin has already numbered, for the library at root.
    Jellyfin stores each path as the CONTAINER sees it (its quadlet binds the
    share at /data), so `container` is the in-container prefix that stands for
    root and every exported path is mapped back to the host before it is written
    — the planner reads host paths, because it plans moves on the array.
    The numbering is what matters, not the name: a release group's file name says
    nothing about which episode a file is, and Jellyfin has already answered that
    question against a provider. Season/episode come from the episode row; the
    series name, its year and its TMDb id come from the Series row the episode
    climbs to, because those are the only place Jellyfin keeps them.
    """
    try:
        con = sqlite3.connect(f"file:{db}?mode=ro", uri=True)
    except sqlite3.Error as e:
        sys.exit(f"cannot open the Jellyfin database read-only ({db}): {e}\n"
                 "It is a WAL database: copy it with its -wal and -shm beside it and"
                 " point --db at the copy. Never sudo it — the data dir belongs to"
                 " the container's owner, and rootless storage is per-user.")
    have = {r[1] for r in con.execute("pragma table_info(BaseItems)")}
    need = {"Id", "ParentId", "Type", "Path", "IndexNumber", "ParentIndexNumber",
            "ExtraType", "IsVirtualItem", "ProductionYear", "Data"}
    if not need <= have:
        sys.exit(f"{db} is not a Jellyfin library database (BaseItems has no"
                 f" {','.join(sorted(need - have))})")
    node = {}
    for i, pid, typ, name, year, data in con.execute(
            "select Id, ParentId, Type, Name, ProductionYear, Data from BaseItems"
            " where Type in (?,?,?)", JF_TV):
        node[i] = (pid, typ, name, year, data)
    def series_of(item_id):
        """Climb episode -> season -> series; empty when the chain is broken."""
        for _ in range(4):
            if item_id not in node:
                return "", "", ""
            pid, typ, name, year, data = node[item_id]
            if typ.endswith(".Series"):
                try:
                    tmdb = (json.loads(data or "{}")
                            .get("ProviderIds") or {}).get("Tmdb", "")
                except ValueError:
                    tmdb = ""
                return name or "", year or "", tmdb
            item_id = pid
        return "", "", ""
    # The fleet's quadlets bind the share at /data, so the container's view of
    # this library is /data/<the library's own folder name> unless told otherwise.
    if not container:
        container = "/data/" + os.path.basename(os.path.normpath(root))
    prefix = container.rstrip("/") + "/"
    written = skipped = 0
    with open(out, "w", newline="") as fh:
        w = csv.writer(fh, delimiter="\t", lineterminator="\n")
        w.writerow(JF_COLS)
        for iid, path, idx, pidx, extra, virt in con.execute(JF_SQL,
                                                             (JF_TV[0], prefix + "%")):
            if virt or not path or not path.startswith(prefix):
                skipped += 1          # a virtual item is a variant, not a file
                continue
            sname, syear, tmdb = series_of(iid)
            w.writerow([root + "/" + path[len(prefix):],
                        sname.replace("\t", " "), syear,
                        "" if pidx is None else pidx,
                        "" if idx is None else idx,
                        (extra or "").replace("\t", " "), tmdb])
            written += 1
    print(f"episodes={written} virtual-or-blank={skipped} -> {out}")
 def main():
    ap = argparse.ArgumentParser(description=__doc__.splitlines()[0])
    sub = ap.add_subparsers(dest="cmd", required=True)
    root = {"default": os.environ.get("MEDIA_ROOT", "/mnt/entropy/Movies"),
            "help": "library root"}
    e = sub.add_parser("enumerate", help="stat every file under the root")
    e.add_argument("--root", **root)
    e.add_argument("--out", default="/tmp/media-enum.tsv")
    p = sub.add_parser("probe", help="ffprobe every video container")
    p.add_argument("--enum", default="/tmp/media-enum.tsv")
    p.add_argument("--out", default="/tmp/media-probed.tsv")
    p.add_argument("--jobs", type=int, default=6)
    x = sub.add_parser("plex", help="export Plex's matched titles for the root")
    x.add_argument("--root", **root)
    x.add_argument("--db", default=PLEX_DB)
    x.add_argument("--out", default="/tmp/plex-movies.tsv")
    j = sub.add_parser("jellyfin",
                       help="export the episodes Jellyfin has numbered for the root")
    j.add_argument("--root", default=os.environ.get("MEDIA_ROOT",
                                                    "/mnt/entropy/TVShows"),
                   help="TV library root on the host")
    j.add_argument("--db", default=JF_DB)
    j.add_argument("--container", default=None,
                   help="the in-container prefix that stands for --root"
                        " (default /data/<root's folder name>)")
    j.add_argument("--out", default="/tmp/jellyfin-tv.tsv")
    a = ap.parse_args()
    if a.cmd == "enumerate":
        enumerate_lib(a.root, a.out)
    elif a.cmd == "probe":
        probe(a.enum, a.out, a.jobs)
    elif a.cmd == "plex":
        plex(a.root, a.db, a.out)
    else:
        jellyfin(a.root, a.db, a.container, a.out)
 if __name__ == "__main__":
    main()
@@ -0,0 +1,881 @@
 #!/usr/bin/env python3
 r"""Plan a Jellyfin-format relayout of a movie library.  DRY RUN ONLY: nothing moves.
  media-plan.py [--root DIR] [--enum TSV] [--probed TSV] [--plex TSV]
                [--out TSV] [--exceptions TXT]
 Inputs are the TSVs scripts/media-census.py exports, plus the optional Plex
 metadata-cache export whose clean titles outrank the filename parser; the title
 authority for everything Plex never matched is scripts/media-titles.py beside this
 file. No media file is opened, so planning the whole library costs seconds.
 PLAN is a TSV the executor (scripts/media-apply.py) replays verbatim —
 `action<TAB>src<TAB>dst<TAB>kind` with action in mkdir | move | keep, ordered so
 parents exist before children. EXCEPTIONS is the human review list: every file the
 plan could not name a film for, every collision it refused, the READMEs the
 executor must write into the piles it created.
 Target contract (jellyfin.org/docs/general/server/media/movies):
  * one folder per movie, named  Title (year)
  * the video inside is named exactly like the folder
  * extra copies of one film are versions: "<base> - <label>.ext"; the base must
    match the folder name character-for-character or Jellyfin splits them apart
  * extras live in canonical subfolders (trailers, featurettes, deleted scenes,
    interviews, samples, shorts, clips, other, extras, backdrops, theme-music)
  * sidecars <base>.<lang>.<flag>.<ext>; images <base>-<poster|backdrop|...>
  * <>:"/\|?* break the matcher, so they never survive here
  * VIDEO_TS/BDMV folders support neither multiple versions nor parts
 """
 import argparse
 import collections
 import csv
 import importlib.util
 import os
 import re
 # Inputs are the census TSVs scripts/media-census.py writes; the engine beside
 # this file is the title authority. Nothing here opens a media file.
 def parse_args():
    ap = argparse.ArgumentParser(description=__doc__.splitlines()[0])
    ap.add_argument("--root", default=os.environ.get("MEDIA_ROOT", "/mnt/entropy/Movies"),
                    help="library root the plan moves files inside")
    ap.add_argument("--enum", default="/tmp/media-enum.tsv", help="file census TSV")
    ap.add_argument("--probed", default="/tmp/media-probed.tsv", help="ffprobe census TSV")
    ap.add_argument("--plex", default="/tmp/plex-movies.tsv",
                    help="Plex metadata export (clean titles); absent = parser only")
    ap.add_argument("--out", default="/tmp/jf-plan.tsv")
    ap.add_argument("--exceptions", default="/tmp/jf-exceptions.txt")
    return ap.parse_args()
 ARGS = parse_args()
 ROOT = ARGS.root
 ENUM, PROBE, PLEX = ARGS.enum, ARGS.probed, ARGS.plex
 PLAN, EXCEPTS = ARGS.out, ARGS.exceptions
 ENGINE = os.path.join(os.path.dirname(os.path.abspath(__file__)), "media-titles.py")
 spec = importlib.util.spec_from_file_location("eng", ENGINE)
 eng = importlib.util.module_from_spec(spec)
 spec.loader.exec_module(eng)
 VIDEO_EXT = {".mkv", ".mp4", ".avi", ".m4v", ".mov", ".ts", ".m2ts", ".mpg",
             ".mpeg", ".webm", ".wmv", ".divx", ".ogm", ".flv", ".vob", ".iso",
             ".img", ".mpv", ".3gp", ".rm", ".rmvb", ".m2v", ".mpv"}
 DISC_DIRS = {"BDMV", "VIDEO_TS", "CERTIFICATE", "AACS", "MLPI", "CLIPINF"}
 SUB_EXT = {".srt", ".ass", ".ssa", ".sub", ".vtt"}
 IMG_EXT = {".jpg", ".jpeg", ".png", ".webp", ".bmp", ".gif", ".tbn"}
 AUDIO_DIR_RE = re.compile(r"soundtrack|score|\bost\b|music|song|album", re.I)
 SUB_KEEP_RE = re.compile(r"^(?!.*\d)(?!.*[._])(?!.*\[)(?!.*\()([A-Za-z][A-Za-z -]{1,24})$")
 MUSIC_TOP_RE = re.compile(r"(?i)soundtrack|score|\bost\b|music|album|song")
 SUB_DIR_OK = re.compile(r"(?i)^$|subs?|subtitles?|srt|custom ?subs?(\[.*\])?"
                        r"|(additional|extra) languages|languages|resources")
 AUDIO_DIR_OK = re.compile(r"(?i)^$|audio(s)?|extra audio|additional audio"
                          r"|(additional|extra) languages|commentar(y|ies)"
                          r"|audio commentary|audio tracks?")
 AUDIO_EXT = {".mp3", ".flac", ".m4a", ".aac", ".dts", ".wav", ".ac3", ".mka",
             ".opus", ".ogg", ".wma"}
 NFO_EXT = {".nfo"}
 IMG_PRIMARY = {"poster", "folder", "cover", "default", "movie", "show"}
 IMG_OTHER = {"backdrop": "backdrop", "fanart": "backdrop", "background": "backdrop",
             "banner": "banner", "logo": "logo", "clearlogo": "logo",
             "thumb": "thumb", "landscape": "thumb", "art": "backdrop",
             "clearart": "backdrop", "jacket": "poster", "cdart": "poster",
             "discart": "poster", "disc": "poster"}
 RES_RE = re.compile(r"\b(2160|4320|4k|1080|720|576|480|360|240)(i|p)\b", re.I)
 # The characters Jellyfin's matcher cannot carry in a file or folder name. "/" is
 # one of them and it is the dangerous one: Plex titles KEEP it ("V/H/S/2", and this
 # library holds six of those films), so leaving it in a name turns ONE movie folder
 # into a nested tree — Movies/V/H/S/2 (2013)/… — and Jellyfin then sees four
 # folders, none of them a movie.
 BAD_CHARS = re.compile(r'[<>:"/|?*]')
 PART_RE = re.compile(r"(?:^|[\s._-])(cd|dvd|part|pt|disc|disk|disco)"
                     r"[\s._-]?([1-9abcd])(?:$|[\s._-])", re.I)
 LANGS = {"en", "eng", "it", "ita", "ja", "jpn", "de", "deu", "fr", "fra", "es",
         "spa", "pt", "por", "ru", "rus", "zh", "chi", "ko", "kor", "sv", "no",
         "da", "nl", "pl", "cs", "hu", "fi", "tr", "el", "he", "ar", "uk", "ro",
         "bg", "hr", "sr", "sk", "et", "lv", "lt", "id", "th", "vi", "fa", "la"}
 FLAGS = {"default", "forced", "foreign", "sdh", "cc"}
 NOISE = {"sub", "subs", "subtitle", "subtitles", "srt", "ass", "ext", "external",
         "the", "a", "an", "of", "and", "in", "on", "to", "for", "with", "ita",
         "eng", "english", "italian", "dual", "audio", "forced", "default"}
 # Canonical title where the library only ever carries a localised name.
 OVERRIDES = {"28 giorni dopo": "28 Days Later",
             "28 settimane dopo": "28 Weeks Later"}
 rows = []
 for line in open(ENUM):
    size, ext, path = line.rstrip("\n").split("\t")
    rows.append({"bytes": int(size), "ext": ext.lower(), "path": path})
 ext_by = {r["path"]: r["ext"] for r in rows}
 size_by = {r["path"]: r["bytes"] for r in rows}
 def rel(p):
    return p[len(ROOT) + 1:] if p.startswith(ROOT + "/") else p
 def parts_of(p):
    return [q for q in rel(p).split("/") if q]
 def is_qb(p):
    return ".!q" in p.lower()
 def disc_root(p, fold=True):
    """Absolute root of the disc tree containing p (the MOVIE folder that holds
    BDMV/ or VIDEO_TS/), else None. `fold` also folds bookkeeping dirs that sit
    beside the tree (CERTIFICATE/, AACS/) into the same movie folder."""
    ps = parts_of(p)
    kinds = DISC_DIRS if fold else {"BDMV", "VIDEO_TS"}
    for i, q in enumerate(ps):
        if q.upper() in kinds:
            return ROOT + "/" + "/".join(ps[:i])
    return None
 def row_name(r):
    """The name a disc-tree entry should be identified by: the engine stores the
    disc's own folder for it, because "BDMV/index.bdmv" is not a title."""
    return r.get("name_src") or r["path"]
 def sanitize(name):
    name = BAD_CHARS.sub("-", name).replace("\\", "-")
    name = re.sub(r"\s+", " ", name).strip().strip(". ").strip()
    return (name[:180] or "untitled")
 def ok_year(y):
    """True for a year a real film could carry. Plex stores 0 for a file it indexed
    but never MATCHED, and "0" is a digit string: trusting it named 67 movie
    folders "Title (0)" — Jellyfin items with no identity, no year and no poster."""
    return bool(re.fullmatch(r"(18|19|20)\d\d", y or "")) and 1870 <= int(y) <= 2035
 WORD_SPLIT = re.compile(r"[^0-9a-z]+")
 def words_of(text):
    """The matching words of a name — or of a whole path, which is how callers
    pass a file together with the folders it sits in.
    Split on everything that is not a letter or a digit: eng.norm does not touch
    "/", so feeding it "…/Psycho (1960)/Extras" left glued tokens like
    "(1960)/extras" and "(1954)" that no other name could ever agree with, and a
    film could not be found by the folder that holds it — which is how one box
    set's featurettes ended up in another film's folder. Both sides of every
    comparison below go through here. A bare number counts only when it could be
    a release year, so release junk ("1080", "2160", "2", "7") never poses as
    agreement between two names.
    """
    out = set()
    for w in WORD_SPLIT.split(eng.norm(text)):
        if not w or w in eng.STOP:
            continue
        if w.isdigit() and not 1870 <= int(w) <= 2100:
            continue
        out.add(w)
    return out
 # ------------------------------------------------------------------ Plex titles
 # Plex has matched a large part of the library's media files to real titles
 # ("Alien", not "Alien.Directors.Cut.1979.1080p.BRrip.x265…"). Where the cache
 # holds a MATCH, its title wins; the filename parser carries everything else
 # (extras, disc trees, TV, and whatever Plex never matched).
 def plex_titles():
    if not os.path.exists(PLEX):
        return {}
    out = {}
    with open(PLEX, newline="") as fh:
        for r in csv.DictReader(fh, delimiter="\t"):
            t, y = r["leaf_title"].strip(), r["leaf_year"].strip()
            # Only a match is an authority, and a match carries a year: an
            # unmatched row echoes the filename, which the parser reads better
            # than a copied release name. The junk filter stays for the matches
            # whose title still carries the release.
            if t and ok_year(y) and not re.search(
                    r"2160p|1080p|x26[45]|hevc|bluray|web-?dl", t.lower()):
                out[r["file"]] = (t, y)
    return out
 PLEX_T = plex_titles()
 # ------------------------------------------------------------- classification
 erows = eng.load(PROBE)
 for r in erows:
    for k, v in OVERRIDES.items():
        if k in r["title"]:
            r["title"] = v
            r["key"] = eng.key_of(r["title"])
            r["base"] = eng.base_words(r["title"], r["key"])
 marks = {r["path"] for r in erows if eng.is_watermark(r["path"], r["bytes"])}
 extras = {r["path"] for r in erows
          if r["path"] not in marks
          and eng.is_extra(r["path"], r["title"], r["dur"], ROOT,
                           matched=r["path"] in PLEX_T)}
 eps = collections.defaultdict(list)
 for r in erows:
    if r["path"] in marks | extras:
        continue
    ep = eng.ep_of(r["path"])
    if ep:
        eps[ep[0]].append(r)      # keyed by the SERIES, never the episode title
 tv_keys = {k for k, v in eps.items() if len(v) >= 5}
 tv_set = {r["path"] for k, v in eps.items() if k in tv_keys for r in v}
 # A folder INSIDE a movie's folder is bonus territory even when its own name is
 # not bonus vocabulary: "Saboteur (1942)/Exrtas/Saboteur - A Closer Look" is a
 # misspelled Extras folder, so the engine's folder rule never fired and the
 # featurette became a movie folder of its own — whose words then out-scored the
 # real Saboteur and drew that box set's other extras into it. The rule needs no
 # metadata: the folder holding the longest film owns the shorter video filed
 # under it, provided that shorter file's own name speaks the film's name — a
 # whole other film filed there stays a film, exactly as the engine's folder rule
 # exempts it. A different year in the file's own name IS that other film
 # speaking, so it vetoes the rule.
 def longest_of(rows):
    """The longest video in each folder: that folder's feature."""
    out = {}
    for r in rows:
        if r["dur"] > 0:
            d = os.path.dirname(r["path"])
            if d not in out or r["dur"] > out[d]["dur"]:
                out[d] = r
    return out
 def not_disc(r):
    """A disc tree's innards belong to the tree, never to the folder's film."""
    return not any(c in DISC_DIRS for c in r["path"].split("/"))
 FEATURE_OF = longest_of([r for r in erows if r["path"] not in marks and not_disc(r)])
 def under_a_feature(path, dur):
    """True when a folder above this file holds a longer film that this file's
    own name names."""
    d = os.path.dirname(os.path.dirname(path))
    while len(d) > len(ROOT):
        f = FEATURE_OF.get(d)
        if f:
            if f["dur"] < dur:
                return False
            theirs = (words_of(os.path.basename(f["path"]))
                      | words_of(os.path.basename(d)))
            if not words_of(os.path.basename(path)) & theirs:
                return False
            y1 = re.search(r"(?:18|19|20)\d\d", os.path.basename(path))
            y2 = re.search(r"(?:18|19|20)\d\d", os.path.basename(f["path"]))
            return not (y1 and y2 and y1.group() != y2.group())
        d = os.path.dirname(d)
    return False
 extras |= {r["path"] for r in erows
           if r["dur"] > 0 and r["path"] not in marks | extras | tv_set
           and not_disc(r) and under_a_feature(r["path"], r["dur"])}
 film_rows = [r for r in erows if r["path"] not in marks | extras | tv_set]
 buckets = collections.defaultdict(list)
 for r in film_rows:
    buckets[eng.key_of(r["title"])].append(r)
 def res_label(r):
    m = RES_RE.search(r["res"] or "")
    if m:
        n, kind = m.group(1).lower(), m.group(2).lower()
        return "2160p" if n in ("2160", "4k") else f"{n}{kind}"
    try:
        w = int((r["res"] or "0x0").split("x")[0])
    except (ValueError, IndexError):
        w = 0
    return "2160p" if w >= 3000 else ("1080p" if w >= 1400 else "SD")
 def version_label(r):
    bits = [res_label(r)]
    vc = (r["vcodec"] or "").lower()
    if vc in ("av1", "libaom", "libsvtav1"):
        bits.append("AV1")
    elif vc in ("vc1", "wmv3"):
        bits.append("VC1")
    if r["bytes"] > 40 * 2**30:
        bits.append("REMUX")
    if "imax" in r["path"].lower():
        bits.append("IMAX")
    return sanitize(" ".join(bits))
 def group_name(items):
    titles = collections.Counter(r["title"] for r in items)
    ascii_t = [t for t in titles if t.isascii()]
    title = (collections.Counter(ascii_t) if ascii_t else titles).most_common(1)[0][0]
    years = [r["year"] for r in items if r["year"] != "?"] or \
            [r["soft"] for r in items if r["soft"]]
    year = collections.Counter(years).most_common(1)[0][0] if years else None
    return sanitize(f"{title} ({year})" if year else title)
 groups = []
 for key, items in buckets.items():
    for c in eng.dur_clusters(items):
        g = {"rows": c, "base": group_name(c)}
        known = [PLEX_T[r["path"]] for r in c if r["path"] in PLEX_T]
        if known:
            # one film, one name: a Plex match for any copy names every copy
            t, yr = collections.Counter(known).most_common(1)[0][0]
            g["plex"] = t
            g["base"] = sanitize(f"{t} ({yr})" if ok_year(yr) else t)
        groups.append(g)
 clash = {b for b, n in collections.Counter(g["base"] for g in groups).items() if n > 1}
 for g in groups:
    if g["base"] in clash:
        # two genuinely different films share title+year: separate folders, and
        # say so in the name, because Jellyfin can only match one item per folder
        g["base"] = sanitize(g["base"] + " - " + version_label(g["rows"][0]))
        g["flag"] = "same title and year as another film"
 moves, mkdirs, problems, readmes = [], set(), [], []
 # --------------------------------------------------------------- unnamed groups
 # A movie folder has to READ as a title. The engine's runtime veto says any file
 # over 45 minutes that is not named like an extra is a film whatever it is called
 # — true for Jellyfin, but a 118-minute documentary named "Extra 1" then produced
 # a movie folder named "1", and folders named "1", "2", "3" are Jellyfin items
 # with no identity whose numeric keys went on to capture every subtitle named
 # "3_Eng.srt" in the library. A year proves a dated feature ("300 (2006)" is a
 # real film), so the gate is: a year, or a word that is not release noise. With
 # neither, the rows are bonus material until a human names them.
 def named(g):
    if re.search(r"\(\d{4}\)$", g["base"]):
        return True
    return any(w.isalpha() and len(w) >= 2 for w in eng.name_words(g["base"]))
 unnamed = [g for g in groups if not named(g)]
 groups = [g for g in groups if named(g)]
 for g in unnamed:
    for r in g["rows"]:
        extras.add(r["path"])
        problems.append(
            f"UNNAMED film {rel(r['path'])} parses to {g['base']!r} with no year: "
            f"bonus pile, not a folder — give the file a real name and re-plan")
 group_of = {}
 for g in groups:
    g["dir"] = f"{ROOT}/{g['base']}"
    for r in g["rows"]:
        group_of[r["path"]] = g
        dr = disc_root(r["path"])
        if dr:
            group_of[dr] = g
 by_path = {r["path"]: r for r in erows}
 # -------------------------------------------------------------- bookkeeping
 claimed = {}
 existing = set(rows_map := {r["path"] for r in rows})
 at_target, booked_src = set(), set()
 def book(src, dst, kind):
    if src == dst:
        at_target.add(src)
        booked_src.add(src)
        return
    if os.environ.get("JF_TRACE") and (src in booked_src or dst.lower() in claimed):
        why = "rebook" if src in booked_src else "collision"
        problems.append(f"TRACE {why} kind={kind} src={rel(src)} dst={rel(dst)}")
    if src in booked_src:          # one source may only ever move once
        problems.append(f"RE-BOOK {rel(src)} -> {rel(dst)} (already planned)")
        return
    if dst.lower() in claimed:
        problems.append(f"COLLISION {rel(dst)}\n    from {rel(src)}\n    and  {rel(claimed[dst.lower()])}")
        return
    claimed[dst.lower()] = src
    booked_src.add(src)
    comp = dst.split("/")
    for i in range(1, len(comp)):
        mkdirs.add("/".join(comp[:i]))
    moves.append((src, dst, kind))
 def uniq_dir(d, stem, ext, hint=""):
    """A destination that no other source already took."""
    cand = f"{d}/{stem}{ext}"
    if cand.lower() not in claimed:
        return cand
    n = 2
    while True:
        cand = f"{d}/{stem} {n}{ext}"
        if cand.lower() not in claimed:
            return cand
        n += 1
 def media_dir(p):
    dr = disc_root(p)
    return os.path.dirname(dr) if dr else os.path.dirname(p)
 # ------------------------------------------------------------------ films
 # film / disc moves: the largest copy carries the folder's own name, the rest
 # become Jellyfin versions; a disc tree moves as a whole into the movie folder
 def vext(p):
    return ext_by.get(p) or os.path.splitext(p)[1].lower()
 disc_children = collections.defaultdict(list)
 for q in rows:
    dr = disc_root(q["path"])
    if dr:
        disc_children[dr].append(q["path"])
 for g in groups:
    primary_taken = False
    disc_kinds = set()
    for r in sorted(g["rows"], key=lambda r: -r["bytes"]):
        dr = disc_root(r["path"])
        if dr:
            inner = os.path.basename(dr).upper()
            if inner in disc_kinds:
                problems.append(
                    f"DISC-VERSION {g['base']}: extra disc tree {rel(dr)} cannot "
                    f"share the folder — discs have no multiple versions")
                continue
            disc_kinds.add(inner)
            for src in disc_children[dr]:
                tail = os.path.relpath(src, dr)
                fold = disc_root(src, fold=False)
                if fold != dr and os.path.dirname(src) != dr:
                    tail = os.path.basename(dr) + "/" + tail
                book(src, f"{g['dir']}/{tail}", "disc-file")
            continue
        label = "" if not primary_taken else f" - {version_label(r)}"
        dst = f"{g['dir']}/{g['base']}{label}{vext(r['path'])}"
        n = 2
        while dst.lower() in claimed:
            dst = f"{g['dir']}/{g['base']}{label} {n}{vext(r['path'])}"
            n += 1
        r["dst"] = dst
        primary_taken = True
        book(r["path"], dst, "film")
    g["primary"] = primary_taken
 # ------------------------------------------------------------------ extras
 EXTRA_KIND = [("trailers", r"trailer|teaser|preview|promo|tv spot|advert"),
              ("deleted scenes", r"deleted|alt scene|gag reel|blooper|outtake"),
              ("interviews", r"interview|cast and crew|session piece|commentary"),
              ("featurettes", r"featurette|behind the scenes|making of|documentary"
                              r"|heritage|archival|video archive|music video"),
              ("samples", r"sample"),
              ("shorts", r"\bshort\b|short film"),
              ("clips", r"\bclip\b|scene selection|montage|epilogue|isolated score"),
              ("other", r"menu|logo|gallery|still|trivia|storyboard|screen test"
                        r"|audition|test footage|watermark|workprint|poster"
                        r"|picture in picture|singalong|karaoke|original score")]
 THEME_RE = re.compile(r"theme|opening song|ending song|soundtrack|score"
                      r"|\bost\b|music|song", re.I)
 BACKDROP_RE = re.compile(r"backdrop|intro", re.I)
 def extras_kind(path):
    low = path.lower()
    if THEME_RE.search(low):
        return "theme-music"
    if BACKDROP_RE.search(low):
        return "backdrops"
    for name, rx in EXTRA_KIND:
        if re.search(rx, low):
            return name
    return "extras"
 def attachable_title(path):
    t, _y = eng.title_of(path)
    return eng.key_of(t) if t and not eng.is_watermark(path, size_by.get(path, 0)) else None
 for g in groups:
    g["tops"] = {parts_of(r["path"])[0] for r in g["rows"]}
 top_groups = collections.defaultdict(list)
 for g in groups:
    for t in g["tops"]:
        top_groups[t].append(g)
 group_by_key = collections.defaultdict(list)
 for g in groups:
    for r in g["rows"]:
        group_by_key[eng.key_of(r["title"])].append(g)
 group_by_dir = {}
 for g in groups:
    for r in g["rows"]:
        group_by_dir.setdefault(media_dir(r["path"]), g)
        dr = disc_root(r["path"])
        if dr:
            group_by_dir.setdefault(os.path.dirname(dr), g)
 def group_words(g):
    """The words that identify a film group: its titles and the names of the
    folders its files live in."""
    if "words" not in g:
        w = set()
        if g.get("plex"):
            w |= words_of(g["plex"])
        for r in g["rows"]:
            w |= words_of(r["title"])
            w |= words_of(os.path.basename(media_dir(r["path"])))
        g["words"] = w
    return g["words"]
 def score_words(text, groups):
    """Ranks film groups by how many of their words a path carries."""
    words = words_of(text)
    out = [(len(words & group_words(g)), g) for g in groups]
    out.sort(key=lambda x: -x[0])
    return out
 def group_for_orphan(path):
    """Which movie folder an extra/sidecar belongs to. A collection folder holding
    several films must be resolved by WORDS, never by position: taking the first
    film of the folder attached every loose subtitle in a box set to whichever
    movie happened to sort first."""
    stem = os.path.splitext(os.path.basename(path))[0]
    key = attachable_title(path)
    if key and group_by_key.get(key):
        cands = group_by_key[key]
        top = parts_of(path)[0] if len(parts_of(path)) > 1 else ""
        same_top = [g for g in cands if any(parts_of(r["path"])[0] == top
                                            for r in g["rows"] if len(parts_of(r["path"])) > 1)]
        pool = same_top or cands
        return pool[0] if len(pool) == 1 else score_words(stem, pool)[0][1]
    gtop = parts_of(path)[0] if len(parts_of(path)) > 1 else ""
    pool = top_groups.get(gtop, [])
    if len(pool) > 1:                    # a loose collection folder (box set, bonus
        n, g = score_words(stem + " " + os.path.dirname(path), pool)[0]
        return g if n >= 2 else None     # album) is resolved by words, not position
    if len(pool) == 1:
        return pool[0]
    d = os.path.dirname(path)
    while d and d != ROOT:
        if d in group_by_dir:
            return group_by_dir[d]
        d = os.path.dirname(d)
    return None
 tv_tops = {parts_of(p)[0] for p in tv_set}
 def film_folder_for(path):
    """A movie folder for a lone feature that has no group: derived from its own
    title when a year proves it, else None (goes to the review pile)."""
    t, y = eng.title_of(path)
    if not t or y == "?":
        return None
    d = uniq_dir(ROOT, sanitize(f"{t} ({y})"), "", "")
    mkdirs.add(d)
    return d
 for p in sorted(extras):
    if disc_root(p):
        continue
    if len(parts_of(p)) > 1 and parts_of(p)[0] in tv_tops:
        continue                        # a special inside a TV set: staged with it
    g = group_for_orphan(p)
    ext = ext_by.get(p, os.path.splitext(p)[1].lower())
    stem = os.path.splitext(os.path.basename(p))[0]
    if g is None:
        solo = film_folder_for(p)
        if solo:
            dst = f"{solo}/{sanitize(stem)}{ext}"
            problems.append(f"REVIEW lone feature {rel(p)}: gets its own folder "
                            f"{os.path.basename(solo)}")
        else:
            dst = f"{ROOT}/_unsorted bonus/{extras_kind(p)}/{sanitize(stem)}{ext}"
            problems.append(f"NO-FILM extra {rel(p)}")
    else:
        dst = f"{g['dir']}/{extras_kind(p)}/{sanitize(stem)}{ext}"
    book(p, uniq_dir(os.path.dirname(dst), os.path.basename(dst)[:-(len(ext) or 1)],
                     ext, os.path.dirname(p)), "extra")
 # Disc trees the engine never saw (a MakeMKV scratch folder, an AACS remnant):
 # they belong to the film of the folder they sit in, kept as a named subtree.
 for q in rows:
    pth = q["path"]
    dr = disc_root(pth)
    if not dr or pth in by_path or pth in booked_src:
        continue
    g = group_by_dir.get(dr) or (top_groups[parts_of(pth)[0]][0]
                                 if len(top_groups.get(parts_of(pth)[0], [])) == 1 else None)
    if g is None:
        continue
    inner = os.path.basename(dr)
    problems.append(f"REVIEW disc remnant {rel(dr)} moves inside {g['base']} "
                    f"(a second master: compare before keeping both)")
    for src in disc_children[dr]:
        if src not in booked_src:
            book(src, f"{g['dir']}/{inner}/{os.path.relpath(src, dr)}", "disc-file")
 # ----------------------------------------------------------------- sidecars
 media_by_dir = collections.defaultdict(list)
 for r in erows:
    if r["path"] in marks:
        continue
    dst = r.get("dst")
    if dst:
        media_by_dir[media_dir(r["path"])].append(r)
 def sidecar_target(path, e):
    d = os.path.dirname(path)
    stem = os.path.splitext(os.path.basename(path))[0]
    if d == ROOT:
        # A file loose in the library ROOT names no film. media_by_dir[ROOT] is the
        # set of root-level MOVIE files, and scoring a stray name against it
        # attached every root-level leftover to whichever root-level film was
        # largest — the plan sent ".DS_Store" into "Titanic (1997)/other/" that
        # way. A root-level stray is nobody's sidecar: it goes to the review pile.
        return None, None
    cands = media_by_dir.get(d, [])
    if not cands:
        nd = os.path.dirname(d)          # a subs/ or Extras/ folder beside the media
        while nd and nd != ROOT and not media_by_dir.get(nd):
            nd = os.path.dirname(nd)
        if media_by_dir.get(nd):
            if nd != d:                  # a NAMED subfolder (Subs/, Extras/, the
                g = group_for_orphan(path)   # box set's own root) carries evidence
                dirs = {os.path.dirname(r.get("dst", "")) for r in media_by_dir[nd]}
                if g is None or g["dir"] not in dirs:
                    # ...but only a STRONG word match outranks the folder the file
                    # sits in. A weak one is an accident: "3_Eng.srt" scored 1
                    # against a junk group named "3" and escaped its own movie's
                    # Subs/ folder, while "03 xXx 3 Return Of Xander Cage" scores 4
                    # and genuinely belongs to the other film of the set.
                    n = score_words(stem, [g])[0][0] if g else 0
                    if n >= 2:
                        return (g, None)
                cands = media_by_dir[nd]
    if not cands:
        g = group_for_orphan(path)
        return (g, None) if g else (None, None)
    if len(cands) > 1:
        w = [t for t in eng.norm(stem).split() if t not in NOISE]
        cands = sorted(cands, key=lambda r: (
            -len([t for t in eng.norm(os.path.basename(r["path"])).split() if t in w]),
            -r["bytes"]))
    return (None, cands[0])
 for q in rows:
    p, e = q["path"], q["ext"]
    if p in by_path or p in extras or p in marks or is_qb(p) or disc_root(p):
        continue
    if os.path.basename(p).startswith("._"):
        # macOS AppleDouble resource forks: a few hundred bytes carrying a name
        # COPIED FROM ANOTHER FILM. Word-matching them to "their" film scattered
        # dead files through four movie folders (and they are video extensions in
        # the AppleDouble case, so they also skipped every other filter). They
        # claim nothing: one pile, and they stay in it — nothing is deleted here.
        book(p, uniq_dir(f"{ROOT}/_macos sidecars", sanitize(os.path.basename(p))[:120],
                         "", os.path.dirname(p)), "junk")
        problems.append(f"JUNK AppleDouble sidecar {rel(p)} (macOS resource fork)")
        continue
    if e == ".iso":         # a DVD image: Jellyfin plays it as a movie file
        g = group_for_orphan(p)
        stem = os.path.splitext(os.path.basename(p))[0]
        if g is not None:
            book(p, uniq_dir(g["dir"], g["base"], e, stem) if g.get("primary") is not None
                 else f"{g['dir']}/{g['base']}{e}", "iso")
        else:
            solo = film_folder_for(p)
            if solo:
                book(p, f"{solo}/{sanitize(stem)}.iso", "iso")
                problems.append(f"REVIEW disc image {rel(p)}: DVD rip, alone in its "
                                f"folder -> {os.path.basename(solo)}")
            else:
                book(p, f"{ROOT}/_unsorted files/{sanitize(os.path.basename(p))}", "orphan")
                problems.append(f"NO-FILM disc image {rel(p)}")
        continue
    if e in VIDEO_EXT:
        # A video the engine never probed (unreadable header) and not an AppleDouble
        # remnant: the plan does not guess at it, so it stays where it is.
        continue
    stem = os.path.splitext(os.path.basename(p))[0]
    g, media = sidecar_target(p, e)
    if media is not None:
        base = os.path.splitext(os.path.basename(media["dst"]))[0]
        d = os.path.dirname(media["dst"])
    elif g is not None:
        base, d = g["base"], g["dir"]
    else:
        # Nothing names this file's film. NEVER scatter it: an archive set or an
        # album tree must move as the unit it is, so the whole loose folder goes
        # to the review pile and a film folder's own leftovers stay with it.
        ps = parts_of(p)
        if len(ps) < 2:
            dst = f"{ROOT}/_unsorted files/{sanitize(stem)}{e}"
        elif len(top_groups.get(ps[0], [])):
            dst = f"{top_groups[ps[0]][0]['dir']}/other/{'/'.join(sanitize(c) for c in ps[1:])}"
        else:
            dst = f"{ROOT}/_unsorted files/{'/'.join(sanitize(c) for c in ps)}"
        book(dst=dst, src=p, kind="orphan")
        problems.append(f"NO-OWNER sidecar {rel(p)}")
        continue
    if e in IMG_EXT:
        low = eng.norm(stem)
        word = next((w for w in IMG_PRIMARY if re.search(rf"\b{w}\b", low)), None)
        kind = "poster" if word else next(
            (IMG_OTHER[w] for w in IMG_OTHER if re.search(rf"\b{w}\b", low)), None)
        if re.search(r"\btheme\b", low):
            dst = f"{d}/theme{e}"
        else:
            dst = f"{d}/{base}-{kind or 'art'}{e}" if kind else f"{d}/{sanitize(stem)}{e}"
        book(p, uniq_dir(os.path.dirname(dst), os.path.basename(dst)[:-(len(e) or 1)],
                         e, os.path.dirname(p)), "image")
    elif e in NFO_EXT:
        book(p, uniq_dir(d, base, ".nfo", os.path.dirname(p)), "nfo")
    elif e in SUB_EXT | AUDIO_EXT:
        toks = [t for t in re.split(r"[._\-\s]+", stem) if t]
        langs = [t.lower() for t in toks if t.lower() in LANGS]
        flags = [t.lower() for t in toks if t.lower() in FLAGS]
        keep = [sanitize(t) for t in toks
                if t.lower() not in langs + flags + list(NOISE)
                and not re.fullmatch(r"\d{3,4}p?|cd\d+|\d{4}|\d+ch|\d+.\d", t.lower())]
        bits = [base] + ([sanitize(" ".join(keep[:4]))] if keep else []) + langs + flags
        if e in SUB_EXT:
            # subtitles: <movie>/srt[/language]
            d = f"{d}/srt"
            pd = os.path.basename(os.path.dirname(p))
            if SUB_KEEP_RE.match(pd) and not SUB_DIR_OK.match(pd):
                d = f"{d}/{sanitize(pd)}"      # a real language folder survives
        elif e in AUDIO_EXT and AUDIO_DIR_RE.search(os.path.dirname(p)):
            d = f"{d}/theme-music"                 # bonus audio: one canonical folder
            pd = os.path.basename(os.path.dirname(p))
            if not MUSIC_TOP_RE.search(pd):        # an album folder keeps its name
                d = f"{d}/{sanitize(pd)}"
        else:
                pd = os.path.basename(os.path.dirname(p))
                if not AUDIO_DIR_OK.match(pd):
                    d = f"{d}/{sanitize(pd)}"
        stem2 = ".".join(bits) or sanitize(os.path.basename(p))
        book(p, uniq_dir(d, stem2, e, os.path.basename(p)), "track")
    else:
        book(p, uniq_dir(f"{d}/other", sanitize(os.path.basename(p))[:120], e,
                         os.path.dirname(p)), "misc")
 readmes.append(f"README in {ROOT}: this library was relaid out into Jellyfin's "
               "one-folder-per-movie layout; `_release-group watermarks/`, "
               "`_unsorted files/` and `_unsorted bonus/` hold files no film claims.")
 for key in tv_keys:
    readmes.append(f"README in {ROOT}/{sanitize('_TV needs a TV library - ' + key)}: "
                   "these are TV episodes found in the MOVIES library; move the folder "
                   "to a TV library source, else Jellyfin groups them wrongly as one "
                   "movie.")
 # --------------------------------------------------------------------- TV
 for key in tv_keys:
    items = eps[key]
    year = collections.Counter(r["year"] for r in items if r["year"] != "?")
    yr = year.most_common(1)[0][0] if year else None
    base = f"_TV needs a TV library - {sanitize(key)}" + (f" ({yr})" if yr else "")
    tops = {parts_of(r["path"])[0] for r in items}
    members = sorted([r["path"] for r in items]
                     + [q for q in extras | marks if len(parts_of(q)) > 1
                        and parts_of(q)[0] in tops],
                     key=lambda q: q)
    for src in members:
        rel_parts = rel(src).split("/")
        tail = "/".join(sanitize(c) for c in rel_parts[:-1] + [rel_parts[-1]])
        book(src, f"{ROOT}/{base}/{tail}", "tv")
 for p in sorted(marks):
    stem, ext = os.path.splitext(os.path.basename(p))
    book(p, uniq_dir(f"{ROOT}/_release-group watermarks", sanitize(stem)[:120], ext,
                     os.path.dirname(p)), "watermark")
 for q in rows:
    if is_qb(q["path"]):
        problems.append(f"SKIPPED qBittorrent partial {rel(q['path'])}")
        book(q["path"], q["path"], "qb-partial")     # stays: resume data
 # --------------------------------------------------------------- debug hook
 if os.environ.get("JF_ISO"):
    x = ROOT + "/High Anxiety (Mel Brooks) [1977] nordic subs  DVDrip/High Anxiety [1977].iso"
    print("attachable", attachable_title(x), "| dir", os.path.dirname(x) in group_by_dir,
          "| g", group_for_orphan(x) and group_for_orphan(x)["base"])
    raise SystemExit
 if os.environ.get("JF_DEBUG"):
    import json
    unp = sorted({r["path"] for r in rows} - {m[0] for m in moves} - at_target)
    print("at_target", len(at_target))
    print("kinds-unplanned", collections.Counter(os.path.splitext(u)[1] for u in unp).most_common(8))
    for up in unp[:20]:
        d = os.path.dirname(up)
        print("UNP", up[:90], "| mediadir?", d in media_by_dir, "| by_path", up in by_path,
              "| extras", up in extras, "| discroot", disc_root(up), "| ext", os.path.splitext(up)[1])
 # -------------------------------------------------------------- validation
 srcs = [m[0] for m in moves]
 dup_src = {s for s, n in collections.Counter(srcs).items() if n > 1}
 for s in dup_src:
    problems.append(f"DOUBLE-CLAIM source {rel(s)}")
 all_src = set(srcs)
 untouched = {r["path"] for r in rows} - all_src
 bad_names, into_alive, lost = [], [], []
 for src, dst, kind in moves:
    for comp in dst[len(ROOT) + 1:].split("/"):
        if (BAD_CHARS.search(comp) or comp != comp.strip(". ") or not comp
                or len(comp) > 200):
            bad_names.append(f"{kind}: {comp!r} in {rel(dst)}")
    if dst in untouched:                       # something staying put is there
        into_alive.append(f"{rel(src)} -> {rel(dst)} (occupied by a file that stays)")
    if dst.lower() in {s.lower() for s in untouched}:
        into_alive.append(f"{rel(src)} -> {rel(dst)} (case-collides with a survivor)")
    if dst.startswith(src + "/") and src in all_src:
        lost.append(f"{rel(src)} moves into its own subtree")
 # every accounted-for source must exist exactly once in the plan
 covered = {r["path"] for r in rows}
 not_planned = sorted(covered - all_src - at_target)
 with open(PLAN, "w") as out:
    out.write("action\tsrc\tdst\tkind\n")
    for d in sorted(mkdirs, key=lambda q: (q.count("/"), q)):
        if d.startswith(ROOT + "/") and d != ROOT:
            out.write(f"mkdir\t\t{d}\tdir\n")
    for p in sorted(booked_src - {m[0] for m in moves}):
        out.write(f"keep\t{p}\t{p}\n")
    for src, dst, kind in sorted(moves, key=lambda m: m[0]):
        out.write(f"move\t{src}\t{dst}\t{kind}\n")
 kinds = collections.Counter(k for _s, _d, k in moves)
 films_n = sum(1 for _s, _d, k in moves if k == "film")
 folders = len({os.path.dirname(d) for _s, d, k in moves if k in ("film", "disc-file")})
 with open(EXCEPTS, "w") as out:
    out.write(f"groups={len(groups)} moves={len(moves)} folders={folders}\n")
    out.write(f"kinds={dict(kinds)}\n")
    out.write(f"at-target-already={len(at_target)} unplanned={len(not_planned)}\n")
    out.write("\n=== WHAT THE EXECUTOR MUST ALSO WRITE ===\n")
    out.write("\n".join(sorted(set(readmes))) + "\n")
    out.write("\n=== PROBLEMS ===\n")
    out.write("\n".join(sorted(set(problems))) + "\n")
    out.write("\n=== BAD NAMES ===\n" + "\n".join(sorted(set(bad_names))) + "\n")
    out.write("\n=== TARGETS OCCUPIED BY SURVIVORS ===\n"
              + "\n".join(sorted(set(into_alive))) + "\n")
    out.write("\n=== LEFTOVER SOURCES (first 60) ===\n"
              + "\n".join(rel(p) for p in not_planned[:60]) + "\n")
 print(f"groups={len(groups)} folders={folders} moves={len(moves)} "
      f"mkdirs={len(mkdirs)} film-files={films_n}")
 print(f"kinds={dict(kinds)}")
 print(f"problems={len(set(problems))} bad-names={len(set(bad_names))} "
      f"occupied={len(set(into_alive))} double-claim={len(dup_src)}")
 print(f"left-unmoved={len(not_planned)}")
 print(f"tv-keys={len(tv_keys)} extras={len(extras)} marks={len(marks)} "
      f"disc-trees={len(disc_children)}")
@@ -0,0 +1,705 @@
 #!/usr/bin/env python3
 r"""Plan a Jellyfin-format relayout of a TV library. DRY RUN ONLY: nothing moves.
 media-shows.py [--root DIR] [--enum TSV] [--titles TSV] [--probed TSV]
                [--tmdb-cache JSON] [--tmdb] [--movies DIR] [--park DIR]
                [--out TSV] [--exceptions TXT]
 Inputs are the TSVs scripts/media-census.py exports: `enumerate` lists what
 exists, `jellyfin` says which episode each file IS, and `probe` is optional
 duration evidence for the movie rule. The TMDb cache is a plain JSON file so a
 plan stays reproducible: answers are read from it, and a miss is looked up only
 with --tmdb and TMDB_API_KEY in the environment (the key never enters argv, and
 never the cache).
 Target contract (jellyfin.org/docs/general/server/media/shows):
    <root>/<Series> (<year>)/Season <NN>/<Series> SxxEyy.ext
    one file holding two episodes   SxxE01-E02      specials       Season 00
    a documented part of an episode SxxE03 Part 1   sidecars       <base>.<lang>.srt
    bonus material                  <series>/trailers|featurettes|… — never a season
 PLAN is the TSV scripts/media-apply.py replays: `action src dst kind`, every
 mkdir row before any move, and `keep` for a file already where the contract puts
 it. EXCEPTIONS is the review list: everything parked, and why.
 The authority chain, strongest first, for one video file:
  1. a provider row — Jellyfin matched this exact path and numbered it.
  2. the file's own SxxEyy / 1x04 marker, the series taken from a sibling's
     provider row, else from the nearest ancestor folder that has one, else from
     the release folder's own words.
  3. nothing. An unnumbered file is PARKED under --park with its relative path
     kept: an invented `Season 01` steals an episode number from the file that
     really is S01E01, and a wrong number is worse than no number.
 Guards, and why each is here:
  * One series, one folder, one year: the year ANY provider row of a series
    states is shared by all of them, so a box set split across five release
    folders lands inside ONE series folder instead of five.
  * Two files claiming one episode: the bigger keeps the standard name, the other
    parks with the reason. Nothing is overwritten and nothing is deleted — which
    encode survives is Frank's call, and a parked file is one rename from home.
  * A file with no episode evidence that runs long and that the MOVIE provider
    names leaves the TV library for --movies under the movie naming contract.
    Long and unnamed parks; short and unnumbered is bonus material.
  * Series art and the show nfo follow the SERIES, a season image follows its
    SEASON, a subtitle or episode nfo follows the episode's new base — an episode
    moved without its subtitles is a half-move.
  * Hidden files, macOS forks, qBittorrent resume parts and the park pile itself
    are never moved. Skipping the park is what makes re-planning stable.
 The plan crosses three trees (the library, its park, the movie library), so the
 executor runs with --root at the share that holds them, on its own journal —
 one journal is one undo:
    python3 scripts/media-shows.py --root /mnt/entropy/TVShows \
        --enum /tmp/tv-enum.tsv --titles /tmp/jellyfin-tv.tsv \
        --out /mnt/entropy/media-tv-plan.tsv
    python3 scripts/media-apply.py apply --plan /mnt/entropy/media-tv-plan.tsv \
        --root /mnt/entropy --journal /mnt/entropy/media-tv-relayout.jsonl
 """
 import argparse
 import collections
 import importlib.util
 import json
 import os
 import re
 import sys
 import urllib.parse
 import urllib.request
 ARGS = argparse.ArgumentParser(description=__doc__.splitlines()[0],
                               formatter_class=argparse.RawDescriptionHelpFormatter)
 ARGS.add_argument("--root", default=os.environ.get("TV_ROOT", "/mnt/entropy/TVShows"),
                  help="the TV library the plan reorganises")
 ARGS.add_argument("--enum", default="/tmp/tv-enum.tsv",
                  help="file census (media-census.py enumerate)")
 ARGS.add_argument("--titles", default="/tmp/jellyfin-tv.tsv",
                  help="the provider's numbering (media-census.py jellyfin)")
 ARGS.add_argument("--probed", default="",
                  help="ffprobe census for the movie rule; absent = no durations")
 ARGS.add_argument("--tmdb-cache", default="/tmp/tmdb-cache.json",
                  help="TMDb answers as an input file, so a plan is reproducible")
 ARGS.add_argument("--tmdb", action="store_true",
                  help="look up cache misses (needs TMDB_API_KEY)")
 ARGS.add_argument("--movies", default="/mnt/entropy/Movies",
                  help="movie library a mis-filed film moves into")
 ARGS.add_argument("--park", default="",
                  help="review pile (default: <root>/_unsorted)")
 ARGS.add_argument("--out", default="/tmp/tv-plan.tsv")
 ARGS.add_argument("--exceptions", default="/tmp/tv-exceptions.txt")
 ARGS = ARGS.parse_args()
 ROOT = os.path.abspath(ARGS.root)
 PARK = os.path.abspath(ARGS.park or os.path.join(ROOT, "_unsorted"))
 MOVIES = os.path.abspath(ARGS.movies)
 ENGINE = os.path.join(os.path.dirname(os.path.abspath(__file__)), "media-titles.py")
 _spec = importlib.util.spec_from_file_location("eng", ENGINE)
 eng = importlib.util.module_from_spec(_spec)
 _spec.loader.exec_module(eng)
 try:
    CACHE = json.load(open(ARGS.tmdb_cache))
 except (OSError, ValueError):
    CACHE = {}
 API_KEY = os.environ.get("TMDB_API_KEY", "")
 VIDEO_EXT = {".mkv", ".mp4", ".avi", ".m4v", ".mov", ".ts", ".m2ts", ".mpg",
             ".mpeg", ".webm", ".wmv", ".divx", ".ogm", ".flv", ".vob", ".rm",
             ".rmvb", ".iso", ".img"}
 SUB_EXT = {".srt", ".ass", ".ssa", ".sub", ".vtt"}
 IMG_EXT = {".jpg", ".jpeg", ".png", ".webp", ".bmp", ".gif", ".tbn"}
 NFO_EXT = {".nfo", ".xml"}
 # The container's own word, for a sidecar named "episode.mkv.srt": that component
 # is not a language flag, and both matchers read the plain form better.
 CONTAINER = {e[1:] for e in VIDEO_EXT}
 # Files that are never anybody's business: hidden, forks, resume state, thumbnails.
 SKIP_NAME = re.compile(r"^(\.|#|dl_state|@eaDir|Thumbs\.db|desktop\.ini)|\.!q$")
 # Show-level metadata keeps its own name at the series root; season-level at the
 # season root; folder.jpg stays with the folder it was written for.
 SERIES_META = re.compile(r"^(tvshow|series|show)(\.nfo|\.xml)?$"
                         r"|^(poster|tvposter|banner|clearlogo|clearart|logo|art"
                         r"|landscape|fanart|showbackdrop|backdrop)\d*\."
                         r"(jpg|jpeg|png|webp|bmp)$", re.I)
 SEASON_META = re.compile(r"^season\.nfo$", re.I)
 FOLDER_IMAGE = re.compile(r"^folder\.(jpg|jpeg|png|webp|bmp)$", re.I)
 SEASON_DIR_RE = re.compile(r"(?i)^\s*(?:season[\s._-]*(\d{1,3})|s(\d{1,2}))\s*$")
 BAD_CHARS = re.compile(r'[<>:"/|?*]')
 # SxxEyy with an optional second episode, plus the 1x04 form. The second number
 # is guarded so a resolution never reads as an episode: "S01E01-1080p" is ONE
 # episode, and the longest-match backtracking of \d{1,3} is what the (?![\dpP])
 # after it closes off.
 EP_RANGE = re.compile(r"(?i)(?<![a-z0-9])s(?:eason[\s._-]*)?(\d{1,2})[\s._-]*"
                      r"e(?:p(?:isode)[\s._-]*)?(\d{1,3})"
                      r"(?:[\s._-]*(?:-|to[\s._-]*|ep)[\s._-]*(?:s\d{1,2}[\s._-]*)?e?"
                      r"(\d{1,3})(?![\dpP]))?")
 NX_RE = re.compile(r"(?i)(?<![a-z0-9])(\d{1,2})x(\d{2,3})(?![a-z0-9])")
 PART_RE = re.compile(r"(?i)(?:^|[\s._-])(?:part|pt|cd|dvd|disc|disk)[\s._-]?"
                     r"([1-9]|i{1,3})(?:$|[\s._-])")
 # Jellyfin reads these subfolder names as bonus material. The planner emits only
 # these spellings, so 40k extras end up in nine folders instead of nine hundred.
 EXTRA_DIRS = [
    ("trailers", r"trailer|teaser|promo|advert|commercial|tv[\s._-]spot|preview"),
    ("deleted scenes", r"deleted"),
    ("behind the scenes", r"behind[\s._-]the[\s._-]scenes|making[\s._-]of|bts"
                          r"|b[\s._-]roll|gag[\s._-]reel|blooper|outtake|archival"
                          r"|video[\s._-]archive|scene[\s._-]selection|menu|gallery"
                          r"|stills?|visual[\s._-]effects|workprint"),
    ("interviews", r"interview|cast[\s._-]and[\s._-]crew|audition|commentary"),
    ("scene cuts", r"scene[\s._-]cuts?|scenecuts"),
    ("featurettes", r"featurettes?"),
    ("shorts", r"shorts?"),
    ("clips", r"clips?"),
 ]
 # Jellyfin's own ExtraType values, folded onto the same folder names.
 EXTRA_KIND = {"trailer": "trailers", "tmbd": "trailers", "intro": "other",
              "movie": "other", "scene": "scene cuts", "clip": "clips",
              "interview": "interviews", "behindthescenes": "behind the scenes",
              "deleted": "deleted scenes", "other": "other", "epilogue": "other",
              "opening": "other", "featurette": "featurettes", "short": "shorts"}
 SHORT_MAX = 45 * 60        # over this, a file with no episode evidence is film-shaped
 def rel(p):
    return p[len(ROOT) + 1:] if p.startswith(ROOT + "/") else p
 def clean(name):
    """A name the filesystem and both matchers can read."""
    name = BAD_CHARS.sub("-", name).replace("\\", "-").strip(" .")
    return name[:180] or "untitled"
 def title_of(text):
    """The engine's parser, applied to a name that has dots but no extension:
    splitext would read "The.Examples" as stem "The", so a sentinel extension is
    added and the parser sees an ordinary media name."""
    return eng.title_of(text + ".mkv")
 def markers(name):
    """(season, [episodes], position) the episode markers a name states; None when
    it states none. The position is where the series' own words stop."""
    stem = os.path.splitext(name)[0]
    m = EP_RANGE.search(stem)
    if m:
        eps = [int(m.group(2))] + ([int(m.group(3))] if m.group(3) else [])
        return int(m.group(1)), eps, m.start()
    m = NX_RE.search(stem)
    if m:
        return int(m.group(1)), [int(m.group(2))], m.start()
    return None
 LEAD_IDX_RE = re.compile(r"^\s*(\d{1,3})\s*[-\u2013.\s]\s*[A-Za-z]")
 PACK_IDX_RE = re.compile(r"^\s*\d+[a-z]?[.)]\s+")
 def num(x):
    try:
        return int(x)
    except (TypeError, ValueError):
        return 0
 def folder_season(path):
    """The season the FOLDER states (`Season 3`, `S3`). A human wrote it, and where
    a scraped row disagrees this is the one that survives a re-plan."""
    m = SEASON_DIR_RE.match(os.path.basename(os.path.dirname(path)))
    return int(next(g for g in m.groups() if g)) if m else None
 def leading_index(name):
    """The number a file leads with inside a season folder — `01 Kyle MacLachlan,
    Sinead OC.avi` is episode 1. Both matchers read it that way."""
    m = LEAD_IDX_RE.match(os.path.splitext(name)[0])
    return int(m.group(1)) if m else None
 def tmdb(kind, query):
    """A provider answer for a query: the cache first, TMDb once, then never again."""
    ck = f"{kind}:{query}"
    if ck in CACHE:
        hit = CACHE[ck]
        # A cache an operator edited by hand can carry the year as a number; the
        # comparison below is text, so the wire form is imposed on every hit.
        return None if hit is None else {**hit, "year": str(hit.get("year") or "")}
    if not (ARGS.tmdb and API_KEY):
        return None
    path = "movie" if kind == "movie" else "tv"
    url = ("https://api.themoviedb.org/3/search/" + path + "?query="
           + urllib.parse.quote(query) + "&include_adult=false")
    req = urllib.request.Request(url, headers={
        # The key rides a header, never the URL: a token in argv is world-readable.
        "Authorization": "Bearer " + API_KEY,
        "Accept": "application/json"})
    try:
        with urllib.request.urlopen(req, timeout=20) as r:
            found = json.load(r).get("results") or []
    except Exception as e:                                # noqa: BLE001
        print(f"tmdb {kind} lookup failed for {query!r}: {e}", file=sys.stderr)
        return None
    if not found:
        CACHE[ck] = None
        return None
    top = found[0]
    CACHE[ck] = {"name": (top.get("title") or top.get("name") or "").strip(),
                 "year": str(top.get("year") or "") or (top.get("first_air_date") or "")[:4],
                 "id": top.get("id")}
    return CACHE[ck]
 # ------------------------------------------------------------------- the inputs
 rows = []
 for line in open(ARGS.enum):
    size, ext, path = line.rstrip("\n").split("\t")
    p = os.path.abspath(path)
    base = os.path.basename(p)
    if not p.startswith(ROOT + "/") or SKIP_NAME.search(base):
        continue
    if p.startswith(PARK + "/"):
        continue                    # the park is already reviewed; re-parking is churn
    rows.append({"bytes": int(size), "ext": ext.lower(), "path": p, "base": base})
 held = {r["path"]: r for r in rows}
 titles = {}
 with open(ARGS.titles) as fh:
    cols = fh.readline().rstrip("\n").split("\t")
    for line in fh:
        f = line.rstrip("\n").split("\t")
        f += [""] * (len(cols) - len(f))
        row = dict(zip(cols, f))
        p = os.path.abspath(row.get("file", ""))
        if p in held:
            titles[p] = row
 dur = {}
 if ARGS.probed and os.path.exists(ARGS.probed):
    for line in open(ARGS.probed):
        f = line.rstrip("\n").split("\t")
        if len(f) > 2:
            try:
                dur[os.path.abspath(f[1])] = float(f[2] or 0)
            except ValueError:
                pass
 video = [r for r in rows if r["ext"] in VIDEO_EXT]
 other = [r for r in rows if r["ext"] not in VIDEO_EXT]
 # What a folder already knows: one provider row names the series for every sibling
 # that has no row of its own — that is how an anime pack gets its show.
 by_dir = collections.defaultdict(list)
 for r in video:
    by_dir[os.path.dirname(r["path"])].append(r)
 dir_series = {}
 for d, rs in by_dir.items():
    names = collections.Counter(titles[r["path"]]["series"] for r in rs
                                if r["path"] in titles and titles[r["path"]]["series"])
    if names:
        dir_series[d] = names.most_common(1)[0][0]
 # The registry: every series a provider names, judged before any folder is. Folder
 # words may resolve a file to one of these names; they may never invent one. That is
 # the whole difference between "NCOP 01.mkv" inside the Gurren Lagann pack, which the
 # pack names, and a new root folder called "01 NCOP".
 registry = {}
 for _p, _t in titles.items():
    if _t["series"]:
        registry.setdefault(eng.norm(_t["series"]), _t["series"])
 def known_in(text):
    """The registered series named inside this text: the longest registered name
    whose words appear in it as a run. "(CBB) Tengen Toppa Gurren Lagann - The
    Complete Series (BD 1080p)" names "Tengen Toppa Gurren Lagann"."""
    hay = " " + eng.norm(text) + " "
    best = ""
    for k, name in registry.items():
        if len(k) >= 5 and f" {k} " in hay and len(k) > len(best):
            best = name
    return best
 # Rows also name their own folders' ancestors, so an already-filed library stays
 # where it is when it is re-planned.
 for d, name in sorted(dir_series.items()):
    a = os.path.dirname(d)
    while a and a != ROOT and len(a) > len(ROOT):
        if a not in dir_series:
            got = known_in(os.path.basename(a))
            if got:
                dir_series[a] = got
        a = os.path.dirname(a)
 def series_of_dir(d):
    """The series an ancestor folder belongs to, nearest first: the folder holding
    provider rows, else a folder whose own words name a registered show."""
    while d and d != ROOT and len(d) > len(ROOT):
        if d in dir_series:
            return dir_series[d]
        got = known_in(os.path.basename(d))
        if got:
            return got
        d = os.path.dirname(d)
    return ""
 def extras_kind(path, extra_type):
    """The canonical bonus subfolder, from Jellyfin's own extra type first."""
    if extra_type:
        return EXTRA_KIND.get(eng.norm(extra_type).replace(" ", ""), "other")
    for d in rel(path).split("/")[:-1]:
        if eng.folder_is_bonus(d):
            for name, rx in EXTRA_DIRS:
                if re.search(rx, d, re.I):
                    return name
            return "extras"
    stem = os.path.splitext(os.path.basename(path))[0]
    for name, rx in EXTRA_DIRS:
        if re.search(rx, stem, re.I):
            return name
    return "extras"
 # ----------------------------------------------------------- classify every file
 series = {}                       # norm(name) -> {name, year, tmdb}
 claimants = []                    # (row, name, season, [episodes])
 extras = []                       # (row, name, bonus folder)
 films = []                        # film-shaped, no episode evidence
 unknown = []                      # (row, reason)
 def series_name(name):
    """The provider's name as a folder: the pack index a scene release leads with
    ("4a. Seed DESTINY") falls away, and the doubled spaces it leaves fold."""
    return clean(re.sub(r"\s{2,}", " ", PACK_IDX_RE.sub("", name)).strip(" ."))
 def note(name, year="", tmdb_id=""):
    """Register a series; the first name wins, the first year and id stick."""
    k = eng.norm(name)
    s = series.setdefault(k, {"name": series_name(name), "year": "", "tmdb": ""})
    if year and not s["year"]:
        s["year"] = str(year)
    if tmdb_id and not s["tmdb"]:
        s["tmdb"] = str(tmdb_id)
 def as_extra(r, owner, extra_type):
    """Bonus material needs a series to file under; with none it is parked."""
    if not owner:
        unknown.append((r, "bonus material with no series to own it"))
        return
    note(owner)
    extras.append((r, owner, extras_kind(r["path"], extra_type)))
 for r in video:
    p, t = r["path"], titles.get(r["path"])
    marked = markers(r["base"])
    fs = folder_season(p)
    if t and t["extra"]:
        as_extra(r, t["series"] or series_of_dir(os.path.dirname(p)), t["extra"])
        continue
    row = bool(t and t["series"] and (t["season"] or t["episode"]))
    if marked or row:
        # The provider row names the SHOW: its registered name, its year. The number
        # that orders the file comes from the file's own name first, then the folder
        # it sits in, then the row — scraped pack rows number episodes badly, where
        # every file under SNL 16-20/Season 16..19 carries season 16 episode 20.
        owner = (t["series"] if t and t["series"]
                 else series_of_dir(os.path.dirname(p))
                 or title_of(r["base"][:marked[2]])[0])
        if not eng.norm(owner).replace(" ", ""):
            unknown.append((r, "an episode marker with no series name to hang it on"))
            continue
        if marked:
            season, eps = marked[0], marked[1]
        else:
            season = fs if fs is not None else num(t["season"])
            ep = leading_index(r["base"]) or num(t["episode"])
            if not 0 < season <= 200 or ep <= 0:
                unknown.append((r, f"a provider row of season {season} episode {ep}"))
                continue
            eps = [ep]
        claimants.append((r, owner, season, eps))
        note(owner, t["year"] if t else "", t["tmdb"] if t else "")
        continue
    owner = series_of_dir(os.path.dirname(p))
    if fs is not None and owner and leading_index(r["base"]):
        claimants.append((r, owner, fs, [leading_index(r["base"])]))
        note(owner)
        continue
    if eng.is_extra(p, title_of(r["base"])[0], dur.get(p, 0.0), ROOT):
        as_extra(r, owner, "")
        continue
    if dur.get(p, 0.0) >= SHORT_MAX or (p not in dur and title_of(r["base"])[1] != "?"):
        films.append(r)
        continue
    unknown.append((r, "no episode marker, no provider row, nothing long enough to be a film"))
 # A provider that never states a year still has one: the search is exact, so a
 # miss parks nothing — the folder simply carries no year, which Jellyfin accepts.
 for k, s in series.items():
    if not s["year"]:
        got = tmdb("tv", s["name"])
        if got and eng.norm(got["name"]) == k:
            s["year"] = got["year"]
 if ARGS.tmdb:
    with open(ARGS.tmdb_cache, "w") as fh:
        json.dump(CACHE, fh, indent=1, sort_keys=True)
 def series_dir(name):
    s = series.get(eng.norm(name))
    if not s:
        s = {"name": clean(name), "year": ""}
    label = f"{s['name']} ({s['year']})" if s["year"] else s["name"]
    return f"{ROOT}/{clean(label)}"
 # ------------------------------------------------------------------- the plan
 moves, keeps, parked = [], [], []
 reasons = collections.defaultdict(list)
 used = {}                          # dst -> the file that claimed it
 def book(r, dst_dir, name, kind):
    """Claim one destination for one file, or park the file when it is taken."""
    dst = f"{dst_dir}/{name}"
    if dst == r["path"]:
        keeps.append((r["path"], dst, kind))
        return
    if dst in used:
        park(r, "something else in the plan already files there")
        return
    if dst in held:
        park(r, f"{rel(dst)} already occupies that name")
        return
    used[dst] = r["path"]
    moves.append((r["path"], dst, kind))
 def park(r, reason):
    parked.append((r, reason))
    reasons[reason].append(r)
    dst = f"{PARK}/{rel(r['path'])}"
    if dst in used or dst in held:
        dst = f"{PARK}/{rel(r['path'])}.{len(reasons)}"
    used[dst] = r["path"]
    moves.append((r["path"], dst, "park"))
 # Episodes first: two files claiming one number is a decision Frank makes, not one
 # the plan sneaks — the bigger copy takes the standard name, the other waits.
 by_ep = collections.defaultdict(list)
 for r, name, season, eps in claimants:
    for e in eps:
        by_ep[(eng.norm(name), season, e)].append(r)
 winners, losers = set(), {}
 for (key, season, e), rs in by_ep.items():
    rs.sort(key=lambda r: (-r["bytes"], r["path"]))
    winners.add(rs[0]["path"])
    for r in rs[1:]:
        losers.setdefault(r["path"],
                          f"duplicate of {series[key]['name']} S{season:02d}E{e:02d}"
                          " (the bigger copy keeps the standard name)")
 for r, name, season, eps in claimants:
    if r["path"] in losers:
        park(r, losers[r["path"]])
        continue
    s = series[eng.norm(name)]
    label = f" S{season:02d}E" + "-E".join(f"{e:02d}" for e in eps)
    part = PART_RE.search(os.path.splitext(r["base"])[0])
    tail = f" Part {part.group(1).upper()}" if part and len(eps) == 1 else ""
    book(r, f"{series_dir(name)}/Season {season:02d}", s["name"] + label + tail + r["ext"],
         "tv")
 for r, name, kind in extras:
    book(r, f"{series_dir(name)}/{kind}",
         clean(os.path.splitext(r["base"])[0]) + r["ext"], "extra")
 # A film inside the TV library leaves for the movie library only when the movie
 # provider answers with a title that matches the name and, when the name states a
 # year, that year too.
 for r in films:
    parsed, year = title_of(r["base"])
    got = tmdb("movie", parsed)
    words = eng.norm(got["name"]).split() if got else []
    if not got or not eng.mentions(parsed.split(), words) \
            or (year != "?" and got["year"] and got["year"] != year):
        park(r, "long file with no episode evidence, and no movie provider names it")
        continue
    label = f"{got['name']} ({got['year']})" if got["year"] else got["name"]
    book(r, f"{MOVIES}/{clean(label)}", clean(label) + r["ext"], "movie")
 # ------------------------------------------------- sidecars and folder metadata
 moved = {}                                          # src -> dst, for every booked file
 for src, dst, _kind in moves:
    moved[src] = dst
 for src, dst, _kind in keeps:
    moved[src] = dst
 dst_dirs = collections.defaultdict(collections.Counter)
 video_dst = collections.defaultdict(list)
 for src, dst in moved.items():
    if src in {r["path"] for r in video}:
        video_dst[os.path.dirname(src)].append((src, dst))
    dst_dirs[os.path.dirname(src)][os.path.dirname(dst)] += 1
 def series_name_at(d):
    """The series the files of a folder belong to — a registered name, or nothing.
    Folder words alone never name a show."""
    names = [titles[src]["series"] for src, _ in video_dst.get(d, [])
             if src in titles and titles[src]["series"]]
    if names:
        return collections.Counter(names).most_common(1)[0][0]
    return series_of_dir(d)
 def season_at(d):
    """The season number a folder carries: its own name, else its files' rows."""
    m = SEASON_DIR_RE.match(os.path.basename(d))
    if m:
        return int(m.group(1) or m.group(2))
    nums = [int(titles[src]["season"]) for src, _ in video_dst.get(d, [])
            if src in titles and titles[src]["season"]]
    return collections.Counter(nums).most_common(1)[0][0] if nums else None
 def folder_dst(d):
    """Where a folder's own images go: the one place its files landed, or the
    series root when they scattered across several seasons."""
    c = dst_dirs.get(d)
    if not c:
        return None
    if len(c) == 1:
        return c.most_common(1)[0][0]
    owner = series_name_at(d)
    return series_dir(owner) if owner else None
 def anchor_of(path, stem):
    """The video file this sidecar names: the longest file name it extends, in the
    NEAREST folder that holds one. A scene pack buries its subtitles two levels
    down (`Pack/Subs/<release-name>/<release>.jpn.srt`), so the search climbs to
    the library root instead of stopping at the parent."""
    d = os.path.dirname(path)
    while len(d) >= len(ROOT) and d.startswith(ROOT):
        best = None
        for src, dst in video_dst.get(d, []):
            base = os.path.basename(src)
            for prefix in (os.path.splitext(base)[0], base):
                if stem == prefix or stem.startswith(prefix + "."):
                    if best is None or len(prefix) > best[2]:
                        best = (src, dst, len(prefix))
        if best:
            return best[0], best[1]
        up = os.path.dirname(d)
        if up == d:
            break
        d = up
    return None
 def sidecar_name(r, src, dst):
    """The sidecar's own tail — language, flag, the media extension an nfo must
    carry — hung on the video file's new base."""
    stem = os.path.splitext(os.path.basename(src))[0]
    tail = os.path.splitext(r["base"])[0][len(stem):]
    # "Show.S01E01.mkv.srt" carries the container's name, not a flag: Jellyfin and
    # Plex both read "Show S01E01.srt", so the container word is dropped here.
    parts = tail.split(".")
    if len(parts) > 1 and parts[1].lower() in CONTAINER:
        tail = "".join("." + p for p in parts[2:])
    if r["ext"] in NFO_EXT:
        tail = os.path.splitext(os.path.basename(src))[1] + tail
    return clean(os.path.splitext(os.path.basename(dst))[0] + tail + r["ext"])
 for r in other:
    p, base = r["path"], r["base"]
    stem = os.path.splitext(base)[0]
    d = os.path.dirname(p)
    if r["ext"] in SUB_EXT or r["ext"] in NFO_EXT or r["ext"] in IMG_EXT:
        a = anchor_of(p, stem)
        if a:
            book(r, os.path.dirname(a[1]), sidecar_name(r, a[0], a[1]),
                 "subtitle" if r["ext"] in SUB_EXT else
                 ("nfo" if r["ext"] in NFO_EXT else "image"))
            continue
    if SERIES_META.match(base):
        owner = series_name_at(d)
        if owner:
            book(r, series_dir(owner), base, "meta")
            continue
    elif SEASON_META.match(base):
        owner, season = series_name_at(d), season_at(d)
        if owner and season is not None:
            book(r, f"{series_dir(owner)}/Season {season:02d}", base, "meta")
            continue
    elif r["ext"] in IMG_EXT or r["ext"] in NFO_EXT:
        dest = folder_dst(d)
        if dest:
            book(r, dest, base, "meta")
            continue
    unknown.append((r, "not media, or media with nothing beside it to belong to"))
 for r, reason in unknown:
    park(r, reason)
 # --------------------------------------------------------------------- output
 def write_plan():
    """mkdir rows for every destination folder and its whole chain, then the moves.
    Parents first: the executor creates directories in the order they are listed."""
    chain = set()
    for _s, dst, _k in moves:
        d = os.path.dirname(dst)
        while d and d not in (ROOT, MOVIES, PARK, os.path.dirname(d)):
            chain.add(d)
            d = os.path.dirname(d)
    dirs = sorted(chain, key=lambda p: (p.count("/"), p))
    with open(ARGS.out, "w") as fh:
        fh.write("action\tsrc\tdst\tkind\n")
        for d in dirs:
            fh.write(f"mkdir\t\t{d}\tdir\n")
        for src, dst, kind in sorted(moves, key=lambda m: (m[1], m[0])):
            fh.write(f"move\t{src}\t{dst}\t{kind}\n")
        for src, dst, kind in sorted(keeps, key=lambda m: (m[1], m[0])):
            fh.write(f"keep\t{src}\t{dst}\t{kind}\n")
    sized = sum(held[s]["bytes"] for s, _d, _k in moves if s in held)
    print(f"plan: {len(moves) - len(parked)} moves, {len(parked)} parked"
          f" ({sized / 2**40:.2f} TiB touched), {len(keeps)} already placed,"
          f" {len(dirs)} folders -> {ARGS.out}")
 def write_exceptions():
    no_year = sorted(s["name"] for s in series.values() if not s["year"])
    with open(ARGS.exceptions, "w") as fh:
        fh.write(f"TV relayout exceptions for {ROOT}\n\n")
        fh.write(f"Everything below is parked under {PARK}, keeping the relative path it\n")
        fh.write("has today. Nothing is deleted and nothing is overwritten: a parked file\n")
        fh.write("is one rename from wherever the plan is pointed next.\n\n")
        for reason in sorted(reasons, key=lambda r: -len(reasons[r])):
            rs = reasons[reason]
            fh.write(f"{len(rs):>6} files  {reason}\n")
            for r in sorted(rs, key=lambda r: rel(r["path"]))[:400]:
                fh.write(f"          {rel(r['path'])}\n")
            if len(rs) > 400:
                fh.write(f"          … and {len(rs) - 400} more\n")
            fh.write("\n")
        if no_year:
            fh.write(f"{len(no_year):>6} series carry no year from any provider, so their\n")
            fh.write("       folders are named without one; Jellyfin still matches them.\n")
            fh.write("       " + ", ".join(no_year[:300]) + "\n")
    print(f"exceptions -> {ARGS.exceptions}")
 write_plan()
 write_exceptions()
@@ -0,0 +1,579 @@
 #!/usr/bin/env python3
 r"""The fleet's release-filename title parser: the grouping authority the relayout
 plan uses for every file the media server never matched.
  report:  media-titles.py report [--probed TSV] [--out TSV]
           the similar-copies census (DUP / REVIEW / EP-DUP groups) over the
           ffprobe census scripts/media-census.py writes.
  parse:   media-titles.py parse NAME [NAME…]
           what title_of() makes of a name — the way to check a naming accident
           before it merges two films into one folder.
 scripts/media-plan.py imports this file; over there the paths are arguments.
 Method — three independent signals, so a naming accident cannot merge two films:
  * franchise KEY  : first two significant title tokens. Stable across parts of a
    collection ("resident evil", "jump street") but NOT film-specific — it only
    selects candidates.
  * DURATION       : the identity check. Same master encodes differ by at most a
    couple of seconds; different films / cuts do not collide.
  * SIZE SPREAD + YEAR: decide the tier. Two files same size AND same duration
    are byte-twins (the hash pass confirms); same duration but different sizes are
    two different releases of one film (deletable redundancy).
 TV episodes (SxxEyy in the name) are grouped separately: their uniform runtime
 would otherwise collapse a whole series into one "duplicate" group.
 """
 import argparse
 import collections, os, re, sys, unicodedata
 # Library root: the folder directly under this one names the film, and every
 # folder below it is structure (Extras/, Featurettes/, Disc 1/, BDMV/). Tests pass
 # their own; nothing here reads media, only the census TSVs about it.
 ROOT = os.environ.get("MEDIA_ROOT", "/mnt/entropy/Movies")
 # Release-tag vocabulary: never part of a title, so the title stops here.
 JUNK = {"5.1", "7.1", "2.0", "1.0", "6.1", "4.0", "extras"}
 JUNK_RE = re.compile(
    r"^("
    r"\d{3,5}p|\d{1,2}k|uhd|fhd|imax|x ?26[45]|hevc|h ?26[45]|avc|av1|x ?av1|vp[89]"
    r"|divx|xvid|mpeg-?2?|\d{1,2}-?bit|hdr ?10\+?|hdr|dolby ?vision|dvh?|sdr|hybrid"
    r"|re-?mux|bd-?re-?mux|blu-?ray|bdrip|brrip|bdmux|bd|web-?(rip|dl)?|hdtv|hdrip"
    r"|dvd-?(rip|r|scr)?|hdcam|telesync|screener|ppv|re-?encode|extras?"
    r"|ddp? ?\d(\.\d)?|dd\d(\.\d)?|dts-?hd(\.ma)?|dts|truehd|atmos|e-?ac-?3?|ac-?3"
    r"|aac\d?(\.\d)?|lpcm|flac|opus|mp3|pcm|\d\.\d(\.\d)?|5 ?1|7 ?1|2 ?0|1 ?0|6 ?1|4 ?0"
    r"|subs?-?(it|en|eng|ita)?|subbed|dual(-?audio)?|multi|dub(bed)?"
    r"|ita|eng|en|it|de|fr|es|pl|jp|jap|lat|esp|ger|fre|rus|zho|hin|tam|tel"
    r"|repack|proper|internal|limited|unrated|extended|director'?s|cut|remaster(ed)?|uncut"
    r"|special|edition|version|crc32|sha1|sfv|par2|nfo|nzb|mux|complete|unpack"
    r"|certified|scene|p2p|part\d*|cd\d+|disc\d+|disk\d+|vob|idx|srt|ass|ssa|sup"
    r"|m3u|cue|log|txt|mkv|mp4|avi|imdb\d[\d.]*|yts(\.\w+)?|rarbg|collection"
    r")$", re.I)
 RES_RE = re.compile(r"^\d{3,5}p$|^\d{1,2}k$|^\d+x\d+$", re.I)
 YEAR_RE = re.compile(r"^(19|20)\d{2}$")
 SEP_RE = re.compile(r"[._\s\-\[\](){}+!~,;:&@#/*'\"`´|]+")
 IDX_RE = re.compile(r"^\d{1,3}[a-z]?$")
 WORD_RE = re.compile(r"[A-Za-z0-9']+$")
 EP_RE = re.compile(r"(?i)(?<![a-z0-9])(s(?:eason[\s._-]*)?\d{1,2}[\s._-]*e(?:p(?:isode)?[\s._-]*)?"
                   r"\d{1,3}|s\d{1,2}\s?e\d{1,3}|\d{1,2}x\d{2,3}|part[\s._-]*\d{1,2})")
 STOP = {"a", "an", "the", "il", "la", "le", "i", "un", "une", "el", "los", "las", "de",
        "del", "du", "di", "e", "ed", "y", "and", "et", "for", "of", "to", "in", "on",
        "vs", "avec", "con"}
 def norm(s):
    s = unicodedata.normalize("NFKD", s)
    return "".join(c for c in s if not unicodedata.combining(c)).lower().strip()
 def fold_sep(s):
    """norm(), plus: every character that is left over is not a letter, digit or
    apostrophe, so it is a SEPARATOR whatever codepoint it hides behind — not part
    of the word. Release groups file titles with lookalikes for punctuation the
    filesystem forbids: "V ∕ H ∕ S ∕ 2" carries U+2215 DIVISION SLASH, "S꞉ Viral"
    U+A789, and en dashes hide in "Star Wars – A New Hope". NFKD does NOT fold
    those (no decomposition mapping), and the scrubber inside title_of DELETES
    what it does not recognise, so five different V/H/S films squashed into the one
    title "vhs2" and would have shared one folder. Deleting is the wrong operation:
    they separate. Accents are gone by then — norm() removed the combining marks —
    so "Amélie" stays "amelie" and never splits into "ame lie"."""
    return " ".join("".join(c if c.isalnum() or c == "'" else " "
                            for c in norm(s)).split())
 def title_of(path):
    """(title, year) from the basename.
    Two passes. (1) Tokenize; every 4-digit year becomes a sentinel in place.
    (2) Find the boundary: the first release-noise token (resolution / codec /
    source / audio / language / group vocabulary) ends the title, so trailing
    tags can never leak in. (3) Collect up to that boundary: leading index
    numbers ("03."), leading year sentinels ("1940 - Fantasia") and leading noise
    are skipped while nothing is collected; once a real word appears a sentinel
    CLOSES the title. A title sitting BEFORE the year sentinel ("300.2006.…" or
    "12. Monkeys. 1995") is a numeric title and is prepended to what follows.
    """
    # fold_sep BEFORE tokenizing, so a lookalike separator separates (see its
    # docstring): the tokens below are then plain alnum/apostrophe words.
    name = fold_sep(os.path.splitext(os.path.basename(path))[0])
    raw = [x for x in SEP_RE.split(name) if x]
    years = [t for t in raw if YEAR_RE.match(t)]
    two_year = len(years) > 1                       # "2001…1968": one of them is the title
    toks, year = [], None
    for t in raw:
        if YEAR_RE.match(t):
            year = t                                    # last year wins
            toks.append("\x00")
        else:
            toks.append(t)
    stop, seen_word = len(toks), False
    for i, t in enumerate(toks):
        if t == "\x00":
            continue
        c = norm(re.sub(r"[^A-Za-z0-9']", "", t))
        if (not c or RES_RE.match(c) or JUNK_RE.match(c) or c in JUNK
                or not WORD_RE.match(c)) and not (IDX_RE.match(t) and not seen_word):
            stop = i
            break
        if c.isalpha():
            seen_word = True
    # A leading number is a track index ("01. Bye Bye Baby", "05. Casablanca") unless
    # it is three digits or more ("007", "000", "1408") or the name carries no year
    # at all ("12 Monkeys", "831"). A 1-2 digit prefix beside a year is an index;
    # the rare real title that looks like one ("0 Things I Hate About You (1999)")
    # loses its zero to the parser — Plex's matched title overrides it anyway.
    lead = re.split(r"[\s._-]", name, maxsplit=1)[0]
    track_idx = bool(re.fullmatch(r"\d{1,2}[a-z]?", lead)) and bool(year)
    keep, num_title = [], None
    for t in toks[:stop]:
        if t == "\x00":
            if keep:
                break                                   # year closes the title
            if two_year and t is not None and years[0] != year:
                # two DIFFERENT years ("2001…1968"): the leading one is the title;
                # "1940 - Fantasia (1940)" repeats one year, so it is not a title
                num_title = num_title or years[0]
            continue                                    # leading year: not the title
        if track_idx and IDX_RE.match(t) and not keep and t == lead:
            continue                                    # "03." / "1c." ordering
        c = norm(re.sub(r"[^A-Za-z0-9']", "", t))
        if not keep and not c.isalpha():
            num_title = num_title or (c or None)
            continue                                    # numeric title candidate
        keep.append(c)
        if len(keep) >= 8:
            break
    keep = [k for k in keep if not re.fullmatch(r"(disc|disk|cd|dvd|part)\d*", k)]
    title = " ".join(keep)
    if not keep:
        # nothing collected: a pure-numeric leading token is the title even when it
        # sits where the release year sits ("1917.2019.2160p…"), and "300.2006…" too
        for t in raw:
            c = norm(re.sub(r"[^A-Za-z0-9']", "", t))
            if re.fullmatch(r"(?:[12]\d{3}|\d{1,3})", c) and (c != year or two_year):
                return c, year or "?"    # never "2160p" / "1080": they are not pure digits
    if (num_title and title and not keep[0].isdigit()
            and (two_year or not track_idx)):
        # a leading number that only echoes the release year ("1917.2019.2160p…") is
        # not a title; one that stands alone ("300.2006…", "1900") is
        title = f"{num_title} {title}"
    title = title or num_title or norm(name).replace("\x00", " ").strip()[:40]
    return title, year or "?"
 # Bonus-material vocabulary: names that say nothing about the film, so an extras
 # file otherwise pairs with an extras file of another movie. "Short", "Spot" and
 # "Clip" are deliberately absent — they are film titles (Short Circuit, Spot,
 # Clips) — and the running-time guard in is_extra() carries the rest.
 SEP_CLS = r"[\s._-]+"
 def _bonus_rx(word):
    return r"(?<![a-z0-9])" + re.sub(r"[ \-]+", lambda _: SEP_CLS, word.lower()) + r"(?![a-z0-9])"
 EXTRA_WORDS = [
    "trailer", "teaser", "featurette", "featurettes", "promo", "promotional",
    "tv spot", "advert", "commercial", "behind the scenes", "making of",
    "deleted scene", "deleted scenes", "deleted footage", "gag reel", "blooper",
    "blooper reel", "bloopers", "outtake", "outtakes", "scene selection", "menu",
    "menus", "gallery", "still gallery", "stills", "poster", "posters",
    "montage", "preview", "previews", "epilogue", "visual effects",
    "music video", "singalong", "karaoke", "trivia", "storyboard", "workprint",
    "test footage", "screen test", "audition", "cast and crew", "isolated score",
    "picture in picture", "original score", "session piece", "sample",
    "watermark",
 ]
 # Words that describe bonus material which can run LONG (a feature-length
 # commentary-track cut, a documentary about the film). Trusted only below
 # LONG_BONUS_MAX, so a 40-minute "commentary" release stays a film.
 EXTRA_LONG_WORDS = ["interview", "interviews", "commentary", "video archive",
                    "archival", "documentary"]
 EXTRA_NAME_RE = re.compile(
    "|".join([_bonus_rx(w) for w in EXTRA_WORDS] + [r"scene[\s._-]?\d{1,3}"]), re.I)
 EXTRA_LONG_NAME_RE = re.compile(
    "|".join(_bonus_rx(w) for w in EXTRA_LONG_WORDS), re.I)
 # Bonus-folder vocabulary. fullmatch = the folder IS an extras subtree; a bare
 # search = it only mentions extras ("... + Extras" box sets hold feature films).
 EXTRA_FOLDER_RE = re.compile(
    r"(?:extras?|bonus(?:[\s._-]material)?|specials?|special[\s._-]featurettes?|"
    r"featurettes?|deleted(?:[\s._-](?:scenes?|footage))?|trailers?|"
    r"making[\s._-]of|behind[\s._-]the[\s._-]scenes|shorts?|interviews?|"
    r"archiv(?:al|es)|video[\s._-]archive)", re.I)
 VIDEO_SUFFIX = (".mkv", ".mp4", ".avi", ".m4v", ".mov", ".m2ts", ".mpg", ".webm")
 # Anything longer than this is a feature film whatever it is called: a 118-minute
 # file named "The Menu" or "Trailer" is a movie, not bonus material.
 SHORT_MAX = 45 * 60
 # Long-shaped bonus vocabulary is only believed below this: a 40-minute
 # commentary cut is an alternate version of the film, not a featurette.
 LONG_BONUS_MAX = 25 * 60
 def name_words(x):
    """Bare words of a file or folder name: no release metadata, no year."""
    x = os.path.splitext(x)[0] if x.lower().endswith(VIDEO_SUFFIX) else x
    return [w for w in SEP_RE.split(fold_sep(x))
            if w and not YEAR_RE.match(w) and not RES_RE.match(w)
            and not JUNK_RE.match(w) and w not in JUNK]
 def mentions(short, long_):
    """True when one word list appears in the other, anywhere: a release names
    the film at the front ("Alien (1979) - Trailer") or in the middle
    ("Movies/Alien (1979)/Extra - Logo.mkv"), so a start-of-string rule fails."""
    a, b = sorted([short, long_], key=len)
    if not a:
        return False
    return any(b[i:i + len(a)] == a for i in range(len(b) - len(a) + 1))
 def folder_is_bonus(folder):
    """True when a folder name carries bonus vocabulary as WHOLE words: the
    vocabulary is matched against a padded word string, so "Extras" decides and
    a word that merely CONTAINS one ("Extraordinary", "The Extras") does not."""
    return bool(EXTRA_FOLDER_RE.search(" " + " ".join(SEP_RE.split(norm(folder))) + " "))
 def is_extra(path, title="", dur=0.0, root=ROOT, matched=False):
    """True for bonus material, which must never be counted as a film: a file
    called "Trailer", "Menu", "Sample" or "Gallery" carries no film identity, so
    extras otherwise pair with each other across the whole library.
    Evidence, strongest first:
      * RUNTIME VETOES EVERYTHING. The film "The Menu", the film "Trailer", the
        franchise "Trailer Park Boys" and "ET — The Extra-Terrestrial" all carry
        bonus vocabulary and run 100+ minutes: anything over SHORT_MAX is a film —
        unless it sits in an explicit bonus folder and metadata never matched it.
        Bonus words that describe long-form material (commentary, documentary,
        interview) are believed only
        under LONG_BONUS_MAX, so a feature-length commentary cut stays a film.
      * The file's OWN name carries bonus vocabulary -> bonus.
      * ANY folder of the path below the film's own root folder carries bonus
        vocabulary ("Alien (1979)/Extras/", ".../Featurettes/Deleted Scenes/",
        "Angel's Egg (1985)/04b. Angel's Egg Extras/") -> bonus. The ROOT folder
        is exempt because it names the film, and a film may be titled with a
        bonus word; and a file whose own name IS that folder's film name is the
        feature, not the bonus ("Ghost in the Shell Movies/Extras/
        Ghost in the Shell - CM3.mp4" is a bonus, but a whole other film filed
        there stays a film).
    Bonus vocabulary is matched as whole words only, so the film
    "Angel's Egg - Extra - Logo" inside "04b. Angel's Egg Extras" is reached by
    the folder rule, and "…Extras" as a fragment of a word is never evidence.
    """
    parts = [q for q in path.split("/") if q]
    if not parts:
        return False
    name = (os.path.splitext(parts[-1])[0]
            if parts[-1].lower().endswith(VIDEO_SUFFIX) else "")
    # A bonus folder anywhere BELOW the film's own folder decides for its
    # contents; the film's own folder is the film's name, whatever words it
    # contains, so it is exempt. It is the first component AFTER ROOT's own
    # components — those are the mount path, not folders of the library.
    head = len([q for q in root.split("/") if q]) + 1
    bonus_folder = False
    for folder in parts[head:-1]:
        if folder_is_bonus(folder):
            # The folder's words are often the film's own name ("Angel's Egg
            # Extras/"), and then they are not bonus evidence. Only WORDS can name
            # a film: "extras" is release junk that name_words drops, so a folder
            # like "01. Extras" reduces to its index — and an index matching an
            # index ("NCOP 01") is a coincidence, not a title.
            nw = [w for w in name_words(name) if not w.isdigit()]
            fw = [w for w in name_words(folder) if not w.isdigit()]
            bonus_folder = not fw or not mentions(nw, fw)
            break
    if dur and dur > SHORT_MAX:
        # Runtime vetoes vocabulary, with ONE exception: an explicit bonus folder
        # and no metadata match. A 90-minute "Feature Length Rough Cut" sitting in
        # a movie's Featurettes/ is a disc extra, and nothing in the library claims
        # it as a film. Long-form bonus words still describe a film, so a
        # feature-length commentary cut or documentary keeps its runtime veto.
        if bonus_folder and not matched and not EXTRA_LONG_NAME_RE.search(name):
            return True
        return False
    if EXTRA_NAME_RE.search(name):
        return True
    if dur and dur <= LONG_BONUS_MAX and EXTRA_LONG_NAME_RE.search(name):
        return True
    return bonus_folder
 # Release-group watermark clips: a group drops a tiny ad ("ETRG.mp4",
 # "RARBG.com.mp4") into many different films, so its name carries no film
 # identity at all and it would otherwise pair with itself across the library.
 WATERMARK_MAX = 20 * 2**20
 WATERMARK_RE = re.compile(r"(?i)^(rarbg(\.com)?|etrg|tgx|fbt|cpbg|yify|yts|"
                          r"rarbg\.com|playnomore|imprs|gauntlet|dfx|macmote|"
                          r"psi-axiz|acm|tlf|sa8|ebp|gtv|w4g|don|aAF|enjoy|hd4u)$")
 def is_watermark(path, size):
    """True for a release group's own watermark video, not a film or an extra."""
    return (size < WATERMARK_MAX
            and bool(WATERMARK_RE.match(os.path.splitext(os.path.basename(path))[0])))
 def key_of(title):
    """Franchise key: first two alphabetic tokens (numeric-only title = itself).
    Deliberately NOT film-specific — it only selects candidates for the duration
    check, so a two-token collision (two films starting "star trek") cannot by
    itself create a duplicate group."""
    toks = [t for t in title.split() if t]
    words = [t for t in toks if not t.isdigit()]
    if not words:
        return " ".join(toks)
    sig = [t for t in words if t not in STOP] or words
    return " ".join(sig[:2])
 def ep_of(path):
    """(series, episode-token) when the basename carries SxxEyy / 1x04 / Part N.
    The series is the name BEFORE the marker. The box-set convention writes
    "Monk - S05E03 - Mr. Monk and the Big Game", and the episode title that
    follows the marker is not the show's name: keying on the whole title gave
    every episode of one series its own group, so a 124-episode box set never
    reached the size that calls something a series and each episode was filed
    as its own film. A file named only by its marker has nothing before it, so
    its whole title stands in."""
    name = os.path.splitext(os.path.basename(path))[0]
    m = EP_RE.search(name)
    if not m:
        return None
    head = name[:m.start()].strip(" ._\u2013-")
    series = title_of(head)[0] if head else None
    return series or title_of(path)[0], norm(m.group(1)).replace(" ", "")
 def base_words(title, key):
    """Title tokens beyond the franchise key — the FILM identity part.
    "evil dead rise" under key "evil dead" -> {rise}; "28 days later" -> {}."""
    kt = set(key.split())
    return frozenset(t for t in title.split() if t not in kt and t not in STOP
                     and not t.isdigit() and not YEAR_RE.match(t))
 def folder_year(path):
    """Soft year evidence: a 19xx/20xx token in any parent directory name
    (collection folders are written both "(2002)" and "[2009]"; a range such as
    "(1998-2000)" deliberately yields nothing)."""
    for d in reversed(os.path.dirname(path).split("/")):
        ys = [y for y in re.split(r"[^0-9]+", d) if YEAR_RE.match(y)]
        if len(ys) == 1:
            return ys[0]
    return None
 def same_film(a, b):
    """Two files are copies of ONE film only when the film-identity tokens agree
    (one side may carry none) AND no year evidence contradicts them. The key says
    the franchise matches; this says the actual film matches — it is what keeps
    'Evil Dead' (2013) apart from 'Evil Dead Rise' (2023) and the four Austin
    Powers apart when their runtimes happen to agree."""
    ba, bb = a["base"], b["base"]
    if ba and bb and ba != bb:
        return False
    ya = a["year"] if a["year"] != "?" else a["soft"]
    yb = b["year"] if b["year"] != "?" else b["soft"]
    if ya and yb and ya != yb:
        return False
    return True
 def dur_close(a, b):
    if a <= 0 or b <= 0:
        return True                                   # unknown duration: let titles decide
    return abs(a - b) <= max(2.0, 0.003 * max(a, b))
 # Blu-ray folder trees: ffprobe only sees the clip it is pointed at, so each tree
 # is folded into ONE entry, described by its longest main-feature clip (measured
 # with ffprobe on the largest BDMV/STREAM file) and sized by the whole tree.
 BDMV_MAIN = {
    "Avengers.Age.of.Ultron.2015.2160p.BluRay.HEVC.TrueHD.7.1.Atmos-TERMiNAL":
        dict(dur=8478.2, vcodec="hevc", res="3840x2160", pix="yuv420p10le", fps=23.976),
    "Full.Metal.Jacket.1987.2160p.BluRay.HEVC.DTS-HD.MA.5.1-EATDIK":
        dict(dur=6993.3, vcodec="hevc", res="3840x2160", pix="yuv420p10le", fps=23.976),
 }
 def load(probed):
    """Probe rows -> film entries. AppleDouble (._*) sidecars are dropped; each
    BDMV tree collapses to one entry carrying the tree's total size."""
    trees = {}
    rows = []
    for line in open(probed):
        f = line.rstrip("\n").split("\t")
        if len(f) < 12:
            continue
        b, p, dur, vc, res, fps, vbit, pix, aud, subs, _d, fmt = f[:12]
        base = os.path.basename(p)
        if base.startswith("._") or base == ".DS_Store":
            continue                                    # macOS sidecar, not media
        m = re.search(r"/(BDMV|VIDEO_TS)/", p)
        root = p[:m.start()] if m else None
        if root or re.search(r"/(CERTIFICATE|AACS|MLPI|CLIPINF|PLAYLIST|BACKUP|REGISTRL)/", p):
            if not root:
                continue      # disc bookkeeping: folds into its tree, never a film
        if root:
            t = trees.setdefault(root, dict(bytes=0, audio=set(), subs=0))
            t["bytes"] += int(b)
            t["audio"].update(a for a in aud.split("+") if a)
            t["subs"] += int(subs or 0)
            continue
        try:
            dur = float(dur)
        except ValueError:
            dur = 0.0
        rows.append(dict(bytes=int(b), path=p, dur=dur, vcodec=vc, res=res, fps=fps,
                         vbit=int(float(vbit or 0)), pix=pix, audio=aud, subs=subs, fmt=fmt))
    for root, t in trees.items():
        folder = os.path.basename(root)
        m = BDMV_MAIN.get(folder, {})
        rows.append(dict(bytes=t["bytes"], path=root + "/BDMV/index.bdmv",
                         name_src=folder, dur=m.get("dur", 0.0),
                         vcodec=m.get("vcodec", "hevc"), res=m.get("res", "-"),
                         fps=m.get("fps", 0), vbit=0, pix=m.get("pix", "-"),
                         audio="+".join(sorted(t["audio"])), subs=str(t["subs"]),
                         fmt="BDMV tree"))
    for r in rows:
        r["title"], r["year"] = title_of(r.get("name_src") or r["path"])
        r["key"] = key_of(r["title"])
        r["base"] = base_words(r["title"], r["key"])
        r["soft"] = folder_year(r["path"])
        r["ep"] = ep_of(r["path"])
    return rows
 def dur_clusters(items):
    """Group files that are the SAME FILM and the SAME MASTER duration. Candidate
    buckets come from the franchise key; same_film then rejects the different
    installments whose runtimes happen to agree."""
    clusters = []
    for r in sorted(items, key=lambda r: -r["dur"]):
        for c in clusters:
            if dur_close(c[0]["dur"], r["dur"]) and same_film(c[0], r):
                c.append(r)
                break
        else:
            clusters.append([r])
    return clusters
 def tier(groups):
    """Split same-film/same-duration clusters: DUP = different releases of one
    film (real, deletable redundancy); REVIEW = identical size AND duration
    (byte-twins — the hash pass is the authority) or clashing year evidence."""
    dup, review = [], []
    for g in groups:
        g.sort(key=lambda r: -r["bytes"])
        years = {(r["year"] if r["year"] != "?" else r["soft"]) for r in g}
        years.discard(None)
        spread = (g[0]["bytes"] - g[-1]["bytes"]) / g[0]["bytes"] if g[0]["bytes"] else 0
        if len(years) > 1 or spread < 0.005:
            review.append(g)
        else:
            dup.append(g)
    return dup, review
 def redun(g):
    return sum(r["bytes"] for r in sorted(g, key=lambda r: -r["bytes"])[1:])
 def line(r):
    return (f"{r['bytes']/2**30:8.2f} GiB {r['res']:>9} {r['vcodec']:<7} "
            f"{r['pix']:<11} {r['dur']/60:5.0f}m {r['audio'][:34]:<34}")
 def dump(tag, groups, out, note=""):
    for gi, g in enumerate(groups, 1):
        g.sort(key=lambda r: -r["bytes"])
        print(f"\n### {tag}-{gi}. {g[0]['title']} ({g[0]['year']}) — {len(g)} files, "
              f"{redun(g)/2**30:.1f} GiB redundant {note}".rstrip())
        for rank, r in enumerate(g):
            print(f"  {line(r)} [{r['year']}] "
                  f"{r['path'].replace(ROOT + '/', '')}")
            out.write(f"{tag}\t{gi}\t{rank}\t{r['bytes']}\t{r['res']}\t{r['vcodec']}\t"
                      f"{r['pix']}\t{int(r['dur'])}\t{r['audio']}\t{r['title']}|{r['year']}\t"
                      f"{r['path']}\n")
 def report(probed, out_path):
    rows = load(probed)
    for r in rows:
        r["mark"] = is_watermark(r["path"], r["bytes"])
        r["extra"] = not r["mark"] and is_extra(r["path"], r["title"], r["dur"])
    marks = [r for r in rows if r["mark"]]
    movies = [r for r in rows if not r["ep"] and not r["extra"] and not r["mark"]]
    episodes = [r for r in rows if r["ep"] and not r["extra"] and not r["mark"]]
    extras = [r for r in rows if r["extra"]]
    print(f"census: {len(rows)} video entries = {len(movies)} films + {len(episodes)} "
          f"TV episodes + {len(extras)} bonus/trailers "
          f"({sum(r['bytes'] for r in extras)/2**30:.1f} GiB) "
          f"+ {len(marks)} release-group watermarks")
    out = open(out_path, "w")
    buckets = collections.defaultdict(list)
    for r in movies:
        buckets[r["key"]].append(r)
    cand = {k: v for k, v in buckets.items() if len(v) > 1}
    clusters = [c for v in cand.values() for c in dur_clusters(v) if len(c) > 1]
    dup, review = tier(clusters)
    dup.sort(key=lambda g: -redun(g))
    review.sort(key=lambda g: -redun(g))
    dump("DUP", dup, out)
    dump("REVIEW", review, out, "— same key+duration, sizes near-identical OR years clash: RECHECK")
    ebuf = collections.defaultdict(list)
    for r in episodes:
        ebuf[(r["ep"][0], r["ep"][1])].append(r)
    edup, ereview = tier([c for v in ebuf.values() if len(v) > 1
                          for c in dur_clusters(v) if len(c) > 1])
    edup.sort(key=lambda g: -redun(g))
    dump("EP-DUP", edup, out)
    shows = collections.Counter()
    for r in episodes:
        shows[r["ep"][0]] += 1
    multi = {s: n for s, n in shows.items() if n > 4}
    print("\n=== TV-series content inside Movies/ (>4 episodes) ===")
    for s, n in sorted(multi.items(), key=lambda kv: -kv[1]):
        print(f"  {n:4d} episodes  {s}")
    out.close()
    print(f"\nDUP groups={len(dup)} files={sum(len(g) for g in dup)} "
          f"redundant={sum(redun(g) for g in dup)/2**40:.2f} TiB", file=sys.stderr)
    print(f"REVIEW groups={len(review)} "
          f"redundant={sum(redun(g) for g in review)/2**40:.2f} TiB", file=sys.stderr)
    print(f"EP-DUP groups={len(edup)} "
          f"ep-redundant={sum(redun(g) for g in edup)/2**30:.1f} GiB", file=sys.stderr)
 def parse(names):
    for n in names:
        t, y = title_of(n)
        print(f"{t!r} ({y})\tkey={key_of(t)!r}\t{n[:70]}")
 def main():
    ap = argparse.ArgumentParser(description=__doc__.splitlines()[0])
    sub = ap.add_subparsers(dest="cmd", required=True)
    rep = sub.add_parser("report", help="similar-copy census over a probe TSV")
    rep.add_argument("--probed", default="/tmp/media-probed.tsv")
    rep.add_argument("--out", default="/tmp/media-similar.tsv")
    par = sub.add_parser("parse", help="show the parsed title/year of a name")
    par.add_argument("names", nargs="+")
    a = ap.parse_args()
    if a.cmd == "report":
        report(a.probed, a.out)
    else:
        parse(a.names)
 if __name__ == "__main__":
    main()
@@ -0,0 +1,281 @@
 #!/usr/bin/env python3
 """media-tv.py — move TV episodes out of the movie library into a TV library.
 `scripts/media-plan.py` refuses to file TV: it parks episode files in a
 `_TV needs a TV library - <series>` pile for a human. This is the other half of
 that refusal. It walks the movie library, groups the files whose own name
 carries an episode marker by SERIES, and writes a plan that places each series
 in the shape both Jellyfin and Plex read without help:
    <library>/<Series> (<year>)/Season <NN>/<the file's own name>.<ext>
 It writes a plan and nothing else. Every rename belongs to
 `scripts/media-apply.py` — the write-ahead journal, the refusal to overwrite,
 the all-or-nothing preflight, the rollback. Apply it with `--root` set to the
 share that holds BOTH libraries (`/mnt/entropy`, not `/mnt/entropy/Movies`) so
 the executor's own path guard still has teeth, and give it its own journal —
 one journal is one undo, and a TV move must not be welded to a movie relayout:
    python3 scripts/media-tv.py --root /mnt/entropy/Movies \
        --library /mnt/entropy/TVShows --out /mnt/entropy/media-tv-plan.tsv
    python3 scripts/media-apply.py apply --plan /mnt/entropy/media-tv-plan.tsv \
        --root /mnt/entropy --journal /mnt/entropy/media-relayout-tv.jsonl
    python3 scripts/media-apply.py verify --plan /mnt/entropy/media-tv-plan.tsv \
        --root /mnt/entropy --journal /mnt/entropy/media-relayout-tv.jsonl
    python3 scripts/media-apply.py sweep --root /mnt/entropy \
        --journal /mnt/entropy/media-relayout-tv.jsonl --apply
 Guards, and why each one is here:
 * The same five-file threshold as the planner. An episode-shaped name on one
  file is not a series — `The Godfather Part 2` is a film — so a smaller group
  is reported and left exactly where it is.
 * An episode whose name carries no season number is reported and left, never
  filed into an invented `Season 01`.
 * A sidecar that shares an episode's own name (`.srt`, `.nfo`, …) follows the
  episode; an episode without its subtitles is a half-move.
 * A group that would claim one destination twice is not resolved here. The
  executor refuses the whole plan before its first rename, which is the right
  outcome: nobody but Frank decides which copy of an episode is the keeper.
 * An episode the destination library already holds at the same byte count is
  left where it is. The TV library has held a complete Monk box set for years
  while the movie library carried a second copy of all 124 episodes; a mover
  that did not look where it was filing would have put both inside one library.
 * A symlink is never followed and never moved.
 """
 import argparse
 import collections
 import importlib.util
 import os
 import re
 def parse_args():
    ap = argparse.ArgumentParser(description=__doc__.splitlines()[0])
    ap.add_argument("--root", default=os.environ.get("MEDIA_ROOT", "/mnt/entropy/Movies"),
                    help="the movie library to empty of TV")
    ap.add_argument("--library", default="/mnt/entropy/TVShows",
                    help="the TV library to fill (a different share or folder)")
    ap.add_argument("--out", default=None,
                    help="plan TSV for scripts/media-apply.py"
                         " (default: media-tv-plan.tsv beside --root)")
    ap.add_argument("--min", type=int, default=5,
                    help="episode files that make something a series")
    a = ap.parse_args()
    if not a.out:
        a.out = os.path.join(os.path.dirname(os.path.abspath(a.root)),
                             "media-tv-plan.tsv")
    return a
 ARGS = parse_args()
 ENGINE = os.path.join(os.path.dirname(os.path.abspath(__file__)), "media-titles.py")
 spec = importlib.util.spec_from_file_location("eng", ENGINE)
 eng = importlib.util.module_from_spec(spec)
 spec.loader.exec_module(eng)
 BAD_CHARS = re.compile(r'[<>:"/|?*]')
 # The season and episode, from the token media-titles.ep_of() hands back: s05e03, 1x04.
 SEASON_RE = re.compile(r"^(?:s(\d{1,2})e(\d{1,4})|(\d{1,2})x(\d{1,4}))$")
 # Sidecars that follow an episode; macOS forks and resume data do not.
 SKIP_SIDECAR = re.compile(r"^(\._|\.DS_|dl_state)")
 # A box set that stamps the show's year into every file name; the folder's year
 # must not be added on top of it.
 YEAR_TAIL = re.compile(r"[(\[]\s*(18|19|20)\d\d\s*[)\]]\s*$")
 SMALL = {"a", "an", "the", "and", "of", "for", "in", "on", "to", "with", "at",
         "by", "de", "la", "vs"}
 def clean(name):
    """A name the filesystem and both matchers can read (same law as the plan)."""
    name = BAD_CHARS.sub("-", name).replace("\\", "-")
    name = re.sub(r"\s+", " ", name).strip().strip(". ").strip()
    return name[:180] or "untitled"
 def episode_name(path):
    """The file's own name, kept: the publisher's spelling beats a re-derivation.
    A dot-separated release name (`monk.s05e03.mkv`) becomes words; a name that
    already carries spaces is left exactly as it was written."""
    stem, ext = os.path.splitext(os.path.basename(path))
    if " " not in stem and stem.count(".") >= 2:
        stem = stem.replace(".", " ")
    return clean(stem) + ext.lower()
 def head_of(path):
    """The publisher's series name: the text BEFORE the episode marker.
    `Monk - S05E03 - Mr. Monk and the Big Game` is `Monk`; the episode title
    after the marker is not the show's name, and case is kept because it is the
    library's own spelling, not a normalised key."""
    name = os.path.splitext(os.path.basename(path))[0]
    m = eng.EP_RE.search(name)
    head = name[:m.start()] if m else name
    return re.sub(r"\s+", " ", re.sub(r"[._]+", " ", head)).strip(" -\u2013").strip()
 def title_series(head):
    """The series folder in the case a TV library is written in.
    Only a name with no capital at all is retitled: a box set that was
    lower-cased on the way in would otherwise file `monk (2002)` among
    `Dexter's Laboratory (1996)`. A name the publisher wrote with case is left
    exactly as it was written."""
    if any(c.isupper() for c in head):
        return head            # the publisher wrote it with case; keep it
    words = head.split()
    return " ".join(w if i and w.lower() in SMALL else w.capitalize()
                    for i, w in enumerate(words))
 def season_episode(token):
    """(season, episode) an episode token states, or None when it states none."""
    m = SEASON_RE.match(token)
    if not m:
        return None
    return int(m.group(1) or m.group(3)), int(m.group(2) or m.group(4))
 def sidecars(path):
    """The files beside PATH that carry its exact stem — its subtitles, its nfo."""
    d, stem = os.path.split(path)
    stem = os.path.splitext(stem)[0]
    out = []
    try:
        entries = sorted(os.listdir(d))
    except OSError:
        return out
    for e in entries:
        p = os.path.join(d, e)
        if p == path or os.path.islink(p) or not os.path.isfile(p):
            continue
        s, _ext = os.path.splitext(e)
        if s == stem and not SKIP_SIDECAR.match(e):
            out.append(p)
    return out
 def episodes_under(root):
    """(path, series, token) for every episode-shaped video file under root."""
    for dirpath, dirnames, filenames in os.walk(root):
        dirnames[:] = sorted(d for d in dirnames
                             if not d.startswith(".") and d != "@eaDir")
        for f in filenames:
            p = os.path.join(dirpath, f)
            if os.path.islink(p) or not f.lower().endswith(eng.VIDEO_SUFFIX):
                continue
            e = eng.ep_of(p)
            if e:
                yield p, e[0], e[1]
 def walk(root, library):
    """Episode files under ROOT keyed by series, ignoring the destination."""
    groups = collections.defaultdict(list)
    lib = os.path.abspath(library)
    for p, series, token in episodes_under(root):
        if os.path.abspath(p).startswith(lib + os.sep):
            continue
        groups[series].append((p, token))
    return groups
 def holdings(library):
    """What the destination already carries: (series, season, episode) -> sizes.
    The one expensive read here, and the one that matters. The movie library
    holds a complete Monk box set that the TV library has held for years: 124
    episodes, the same episode list, identical byte counts. Moving it would
    have filed a second copy of a whole show into the library that already had
    it. Byte count is the test on purpose — two encodes of one episode are two
    files and both are wanted; the same bytes are the same file."""
    out = collections.defaultdict(set)
    for p, series, token in episodes_under(library):
        se = season_episode(token)
        if se is None:
            continue
        try:
            out[(series, *se)].add(os.path.getsize(p))
        except OSError:
            continue
    return out
 def main():
    root, library = os.path.abspath(ARGS.root), os.path.abspath(ARGS.library)
    groups = walk(root, library)
    held_here = holdings(library)
    mkdirs = {library}
    rows, lines, skipped = set(), [], []
    for series, items in sorted(groups.items()):
        if len(items) < ARGS.min:
            skipped.append(f"skipped: {head_of(items[0][0]) or series} — {len(items)} "
                           f"episode-shaped file{'s' if len(items) != 1 else ''},"
                           f" below {ARGS.min}: not a series")
            continue
        years = collections.Counter(y for y in (eng.folder_year(p) for p, _t in items)
                                    if y)
        year = years.most_common(1)[0][0] if years else None
        heads = collections.Counter(head_of(p) for p, _t in items)
        head = heads.most_common(1)[0][0]
        if year and YEAR_TAIL.search(head):
            year = None
        name = clean(f"{title_series(head)}{f' ({year})' if year else ''}")
        series_dir = os.path.join(library, name)
        moved = held = 0
        for path, token in items:
            se = season_episode(token)
            if se is None:
                skipped.append(f"skipped: {path} — no season number in its name,"
                               " nothing here invents one")
                continue
            try:
                size = os.path.getsize(path)
            except OSError:
                continue
            if size in held_here.get((series, *se), ()):
                held += 1
                continue
            dst_dir = os.path.join(series_dir, f"Season {se[0]:02d}")
            mkdirs.update([series_dir, dst_dir])
            base = episode_name(path)
            rows.add((path, os.path.join(dst_dir, base)))
            moved += 1
            for side in sidecars(path):
                rows.add((side, os.path.join(
                    dst_dir, clean(os.path.splitext(base)[0]
                                   + os.path.splitext(side)[1].lower()))))
        if moved:
            lines.append(f"{name}: {moved} episodes"
                         f"{'' if year else ' (no year in any folder name)'}"
                         f" -> {series_dir}")
        if held:
            lines.append(f"{name}: {held}"
                         f" {'episode' if held == 1 else 'episodes'} already in"
                         f" {library} at the same byte count — left there")
    # An empty plan creates nothing, not even the library folder: a run that
    # found no series must not be able to leave a stray directory behind.
    dirs = sorted(mkdirs, key=lambda p: (p.count(os.sep), p)) if rows else []
    with open(ARGS.out, "w") as out:
        out.write("action\tsrc\tdst\tkind\n")
        for d in dirs:
            out.write(f"mkdir\t\t{d}\tdir\n")
        for src, dst in sorted(rows):
            out.write(f"move\t{src}\t{dst}\ttv\n")
    for line in lines + skipped:
        print(line)
    print(f"plan: {len(rows)} moves, {len(dirs)} folders -> {ARGS.out}")
    if rows:
        print(f"next: python3 scripts/media-apply.py apply --plan {ARGS.out}"
              f" --root {os.path.dirname(root)}"
              f" --journal {os.path.join(os.path.dirname(root),'media-relayout-tv.jsonl')}"
              "   (add --apply to run it)")
    return 0
 if __name__ == "__main__":
    raise SystemExit(main())
@@ -0,0 +1,241 @@
 #!/usr/bin/env bash
 # The SSD-backed metadata view that lets Jellyfin and Plex share one metadata set.
 #
 # Both servers read a Kodi-format sidecar (movie.nfo / tvshow.nfo / episode.nfo)
 # ONLY when it sits next to the media file. The media lives on the RAID6 array at
 # /mnt/entropy — five spinning disks, deliberately mounted read-only for every
 # container, and slow to seek for a thousand tiny XML reads. So neither media
 # server binds the array any more: they bind an OverlayFS merge of it with a
 # writable layer on the NVMe.
 #
 #   lowerdir  /mnt/entropy                RAID6 (ROTA=1) — source of truth, never written
 #   upperdir  /var/lib/media-meta/upper   NVMe — every sidecar and image byte
 #   workdir   /var/lib/media-meta/work    NVMe — kernel bookkeeping
 #   Where     /var/lib/media-meta/view    the merged tree jellyfin and plex bind
 #
 # Everything the view needs — layers, bookkeeping, trash, the merged door — lives
 # under the one directory /var/lib/media-meta, so nothing new appears under /mnt.
 #
 # Consequences worth knowing:
 #   * A sidecar write is a few KB onto the SSD; the media bytes never move and the
 #     array is still never written by anything automated.
 #   * Deleting through the view writes a whiteout into the upper layer instead of
 #     deleting from the array. Nothing should delete through the view: deletions
 #     go to .AGENTTRASH on the RAW path, and `reap` clears the resulting ghosts.
 #   * inotify events on the lower layer do not cross the merge, so a file that
 #     arrives on the raw array is invisible to a file-system watcher in the view:
 #     trigger a library refresh (see the Media Relayout page).
 #
 # Subcommands
 #   plan   [--upper-base D] [--lower D]  print the resolved shape, touch nothing
 #   verify                               prove the live mount IS that shape
 #   install [--upper-base D]             create the layers, install+enable+start the unit
 #   reap   [--apply] [--upper-base D] [--lower D]
 #                                       list upper-layer entries whose media is gone
 #   status                               one line per fact a human wants
 #
 # install must run as root (it writes /etc/systemd/system and mounts); the others
 # are read-only and safe unprivileged.
 set -euo pipefail
 LOWER=/mnt/entropy
 UPPER_BASE=/var/lib/media-meta
 VIEW=$UPPER_BASE/view
 # systemd names a mount unit after its ESCAPED path ('/' -> '-', a literal '-'
 # -> '\x2d'), so the unit on the host is var-lib-media\x2dmeta-view.mount. The
 # repo copy keeps a readable name; install writes it under the escaped one.
 UNIT='var-lib-media\x2dmeta-view.mount'
 UNIT_DIR=/etc/systemd/system
 # The rootless containers write as this host uid (the deploy user). The upper
 # layer has to accept those writes, so it is owned by them.
 MEDIA_UID="${MEDIA_VIEW_UID:-1000}"
 MEDIA_GID="${MEDIA_VIEW_GID:-1000}"
 REPO_ROOT="$(cd "$(dirname "$0")/.." && pwd)"
 UNIT_SRC="$REPO_ROOT/dankserver/host/systemd/media-meta-view.mount"
 die() {
    printf 'media-view: %s\n' "$*" >&2
    exit 1
 }
 usage() {
    printf 'usage: %s plan|verify|install|reap|status [options]\n' "$0" >&2
    exit 2
 }
 cmd="${1:-}"
 [ -n "$cmd" ] || usage
 shift || true
 upper_base="$UPPER_BASE"
 lower="$LOWER"
 apply=0
 while [ $# -gt 0 ]; do
    case "$1" in
        --upper-base) [ $# -ge 2 ] || usage; upper_base="$2"; shift 2 ;;
        --lower)      [ $# -ge 2 ] || usage; lower="$2"; shift 2 ;;
        --apply)      apply=1; shift ;;
        *) usage ;;
    esac
 done
 upper="$upper_base/upper"
 work="$upper_base/work"
 # The device backing a path, and whether the kernel calls it rotational.
 rotational_of() {
    local src rota
    src="$(findmnt -no SOURCE --target "$1" 2>/dev/null || true)"
    [ -n "$src" ] || die "cannot find a block device behind $1"
    rota="$(lsblk -no ROTA "$src" 2>/dev/null | head -1 || true)"
    [ -n "$rota" ] || die "cannot read the rotational flag of $src"
    printf '%s %s\n' "$src" "$rota"
 }
 require_upper_base() {
    [ -d "$upper_base" ] \
        || die "refused: upper base $upper_base does not exist (put it on the SSD first)"
 }
 do_plan() {
    printf 'lowerdir=%s\nupperdir=%s\nworkdir=%s\nmountpoint=%s\nunit=%s/%s\n' \
        "$lower" "$upper" "$work" "$VIEW" "$UNIT_DIR" "$UNIT"
 }
 do_verify() {
    local fail=0 fst opts src rota
    if ! mountpoint -q "$VIEW"; then
        printf 'refused: %s is not a mountpoint — a plain directory here is a\n' "$VIEW" >&2
        printf '  library that scans itself empty, not a metadata view.\n' >&2
        return 1
    fi
    fst="$(findmnt -no FSTYPE --mountpoint "$VIEW" 2>/dev/null || true)"
    if [ "$fst" != overlay ]; then
        printf 'refused: %s is mounted %s, not overlay\n' "$VIEW" "${fst:-nothing}" >&2
        fail=1
    fi
    opts="$(findmnt -no OPTIONS --mountpoint "$VIEW" 2>/dev/null || true)"
    local want
    for want in "lowerdir=$LOWER" "upperdir=$UPPER_BASE/upper" "workdir=$UPPER_BASE/work"; do
        case ",$opts," in
            *",${want},"*) ;;
            *) printf 'refused: %s options lack %s (got: %s)\n' "$VIEW" "$want" "$opts" >&2
               fail=1 ;;
        esac
    done
    read -r src rota <<<"$(rotational_of "$UPPER_BASE")"
    if [ "$rota" != 0 ]; then
        printf 'refused: the upper layer sits on %s which is rotational — metadata\n' "$src" >&2
        printf '  belongs on the SSD, not on the array.\n' >&2
        fail=1
    fi
    if [ ! -w "$UPPER_BASE/upper" ]; then
        printf 'refused: %s is not writable\n' "$upper" >&2
        fail=1
    fi
    [ "$fail" -eq 0 ] || return 1
    printf 'view ok: %s <- overlay(%s,%s)\n' "$VIEW" "$LOWER" "$UPPER_BASE/upper"
 }
 do_install() {
    [ "$(id -u)" -eq 0 ] || die "install must run as root (writes $UNIT_DIR and mounts)"
    require_upper_base
    mountpoint -q "$LOWER" \
        || die "refused: $LOWER is not mounted; overlaying an empty directory would present an empty library"
    local src rota
    read -r src rota <<<"$(rotational_of "$upper_base")"
    [ "$rota" = 0 ] || die "refused: $upper_base is on $src, a rotational device — the metadata layer must be on the SSD"
    [ -f "$UNIT_SRC" ] || die "refused: $UNIT_SRC is missing from $REPO_ROOT"
    # systemd will not create a mount point; a missing directory fails the start.
    install -d -m 0755 -o "$MEDIA_UID" -g "$MEDIA_GID" "$upper" "$work" "$VIEW"
    install -d -m 0700 "$upper_base/.AGENTTRASH"
    install -m 0644 "$UNIT_SRC" "$UNIT_DIR/$UNIT"
    systemctl daemon-reload
    systemctl enable "$UNIT" >/dev/null 2>&1 || true
    systemctl start "$UNIT"
    do_verify
 }
 # An upper-layer DIRECTORY with no counterpart below is metadata for a media
 # folder that has been moved or renamed on the RAW path — a ghost the media
 # servers would still browse. Files are never candidates on their own: a sidecar
 # (movie.nfo, an .srt) exists ONLY above, by design, and is exactly what the view
 # is for. A copy-up whose lower file was deleted by hand is the one case this
 # leaves in place; the raw path is where deletions belong, so it stays visible.
 do_reap() {
    [ -d "$upper" ] || die "refused: $upper does not exist (no upper layer at $upper_base)"
    stamp="$upper_base/.AGENTTRASH/$(date -u +%Y%m%d-%H%M%S)"
    count=0
    ghosts=0
    reap_walk ""
    if [ "$count" -eq 0 ] && [ "$ghosts" -eq 0 ]; then
        printf 'reap: no ghosts under %s\n' "$upper"
        return 0
    fi
    if [ "$ghosts" -gt 0 ]; then
        printf 'reap: %d whiteout%s above — someone deleted THROUGH the view.\n' \
            "$ghosts" "$([ "$ghosts" -eq 1 ] || printf s)" >&2
        printf '  Leaving them: removing a whiteout un-hides the file it hides.\n' >&2
        printf '  Deletions belong in .AGENTTRASH on the raw path.\n' >&2
    fi
    if [ "$count" -eq 0 ]; then return 0; fi
    if [ "$apply" -eq 1 ]; then
        printf 'reap: moved %d entr%s to %s\n' "$count" \
            "$([ "$count" -eq 1 ] && printf y || printf ies)" "$stamp"
    else
        printf 'reap: %d ghost entr%s above; re-run with --apply to move them to %s\n' \
            "$count" "$([ "$count" -eq 1 ] && printf y || printf ies)" "$stamp"
    fi
 }
 # Descend only where the lower layer still has the directory: a directory that is
 # missing below is itself the ghost, and its whole subtree goes with it.
 reap_walk() {
    local rel="$1" entries abs r name
    entries="$(find "$upper${rel:+/$rel}" -mindepth 1 -maxdepth 1 -print 2>/dev/null || true)"
    [ -n "$entries" ] || return 0
    while IFS= read -r abs; do
        r="${abs#"$upper"/}"
        name="${abs##*/}"
        if [ -c "$abs" ] || printf '%s' "$name" | grep -q '^\.wh\.'; then
            ghosts=$((ghosts + 1))
            continue
        fi
        if [ -e "$lower/$r" ]; then
            [ -d "$abs" ] && reap_walk "$r"
            continue
        fi
        [ -d "$abs" ] || continue
        count=$((count + 1))
        printf '%s\n' "$r"
        if [ "$apply" -eq 1 ]; then
            mkdir -p "$stamp/$(dirname "$r")"
            mv "$abs" "$stamp/$r"
        fi
    done <<<"$entries"
 }
 do_status() {
    if mountpoint -q "$VIEW"; then
        printf 'view: %s (%s)\n' "$VIEW" "$(findmnt -no OPTIONS --mountpoint "$VIEW")"
    else
        printf 'view: %s NOT MOUNTED\n' "$VIEW"
    fi
    if [ -d "$upper" ]; then
        printf 'upper: %s %s files %s\n' "$upper" \
            "$(find "$upper" -type f 2>/dev/null | wc -l)" "$(du -sh "$upper" 2>/dev/null | cut -f1)"
    else
        printf 'upper: %s missing\n' "$upper"
    fi
    systemctl is-active "$UNIT" 2>/dev/null | sed 's/^/unit: /' || printf 'unit: %s unknown\n' "$UNIT"
 }
 case "$cmd" in
    plan)    do_plan ;;
    verify)  do_verify ;;
    install) do_install ;;
    reap)    do_reap ;;
    status)  do_status ;;
    *)       usage ;;
 esac
@@ -0,0 +1,335 @@
 #!/bin/sh
 # Deterministically empty every .AGENTTRASH directory under the given roots.
 #
 # PURPOSE
 #   The fleet's deletion policy parks deleted files in .AGENTTRASH/ — in-repo
 #   beside the item, elsewhere in $HOME as .AGENTTRASH/YYYYMMDD-topic — and
 #   waits for approval to purge. This script is that purge: given a machine,
 #   it clears .AGENTTRASH directories and nothing else.
 #
 # CONTRACT
 #   Default dry-run: the plan prints and nothing is deleted; --execute deletes.
 #   Every line the script can ever print for a candidate is decided at plan time
 #   and stored in the plan table, so a dry run prints exactly what --execute
 #   will print. A .AGENTTRASH that is a root's own trash (a repo's
 #   .AGENTTRASH, a machine's ~/.AGENTTRASH) prints `plan empty <path>`: the
 #   directory is the convention and survives, its staged contents are cleared.
 #   A .AGENTTRASH deeper than that is content and prints `plan remove <path>`
 #   plus, once executed, `removed <path>`; its own nested .AGENTTRASH entries
 #   belong to that tree and are never planned again. Candidates are found with
 #   `find -xdev` per root, so a mounted tree can never be walked by accident;
 #   ordering is lexicographic (LC_ALL=C). `/`, symlinked, and foreign-owned
 #   candidates are refused and reported (exit 1), never touched.
 #   A summary line `removed=<candidates> deleted=<removed> refused=<n>` closes
 #   every run and one dated line is appended to the log.
 #   Exit: 0 success, 1 operational failure, 2 usage error. A second run over
 #   the same tree plans nothing further and exits 0.
 #
 # PORTABILITY
 #   POSIX sh, clean under `shellcheck -S style`. Externals: find (must support
 #   -delete), stat, rm, date, mkdir. No bashisms, no GNU-only flags, no arrays,
 #   no `local`. Paths containing whitespace are rejected at parse time — the
 #   same rule applied to roots and candidates — so the plan can be walked with
 #   plain word splitting and nothing can be silently mis-split.
 #
 # USAGE
 #   purge-agenttrash [--execute] [--quiet] [--log PATH] [PATH...]
 #   --execute   delete; without it the run is a plan and deletes nothing
 #   --quiet     suppress per-line output; errors and the summary still print
 #   --log PATH  log destination (default: $HOME/.AGENTTRASH-cleanup.log)
 #   PATH...     roots to search (default: $HOME)
 #
 # Everything else is fixed by design: the root default, the ordering, the
 # guards. There is no configuration file and no environment override.
 set -eu
 usage() {
    sed -n '3,44p' "$0" | sed 's/^# \{0,1\}//'
 }
 die() {
    printf '%s: %s\n' 'purge-agenttrash' "$1" >&2
    exit "${2:-1}"
 }
 # ------------------------------------------------------------------ arg parser
 # Roots accumulate as one space-separated string: a root containing whitespace is
 # rejected at parse time rather than silently split later. That leaves the plan
 # iteratable with plain `$@`-free word splitting, the only portable construct
 # that cannot corrupt a path it has already vetted.
 MODE=plan
 QUIET=0
 LOGPATH=
 ROOTS=
 while [ "$#" -gt 0 ]; do
    case "$1" in
        --help|-h)
            usage
            exit 0
            ;;
        --execute)
            MODE=execute
            ;;
        --quiet)
            QUIET=1
            ;;
        --log)
            [ "$#" -ge 2 ] || die '--log needs a value' 2
            LOGPATH=$2
            shift
            ;;
        --log=*)
            LOGPATH=${1#--log=}
            ;;
        -*)
            die "unknown option: $1" 2
            ;;
        *)
            case $1 in
                *' '*) die "root contains whitespace: $1" 2 ;;
            esac
            if [ -z "$ROOTS" ]; then
                ROOTS=$1
            else
                ROOTS="$ROOTS $1"
            fi
            ;;
    esac
    shift
 done
 [ -n "${HOME:-}" ] || die 'HOME is not set; pass roots explicitly or set HOME' 2
 [ -n "$LOGPATH" ] || LOGPATH=$HOME/.AGENTTRASH-cleanup.log
 # ------------------------------------------------------------ tool availability
 require() {
    command -v "$1" >/dev/null 2>&1 || die "required tool not found: $1" 1
 }
 require find
 require stat
 require rm
 require date
 require mkdir
 # `find -delete` backs the in-place emptying. Probe it rather than trusting
 # `find --version`: a shell function named `find` must not be able to fake it.
 if ! find . -maxdepth 0 -name '.purge-probe-does-not-exist' -delete >/dev/null 2>&1; then
    die 'find does not support -delete; refusing to guess at deletion' 1
 fi
 UID_SELF=$(id -u)
 # ------------------------------------------------------------------- locking
 # The lock dir also hosts this run's plan table, so both live and die together:
 # the exit trap removes the whole directory and only the log survives the run.
 LOCKDIR=$HOME/.agenttrash-purge.lock
 PLANFILE=$LOCKDIR/plan
 acquire_lock() {
    if ! mkdir "$LOCKDIR" 2>/dev/null; then
        held_pid=$(cat "$LOCKDIR/pid" 2>/dev/null || printf '%s\n' '')
        if [ -n "$held_pid" ] && kill -0 "$held_pid" 2>/dev/null; then
            die "another purge holds $LOCKDIR (pid $held_pid)" 1
        fi
        # Owner died mid-run: drop its pid, then retry exactly once. An
        # unremovable/nonempty lock dir is not ours to reclaim.
        rm -f "$LOCKDIR/pid" 2>/dev/null || true
        rmdir "$LOCKDIR" 2>/dev/null || die "$LOCKDIR exists and is not ours to reclaim" 1
        mkdir "$LOCKDIR" 2>/dev/null || die "cannot reacquire $LOCKDIR" 1
    fi
    printf '%s\n' "$$" >"$LOCKDIR/pid" || die "cannot stamp $LOCKDIR/pid" 1
    : >"$PLANFILE" || die "cannot write $PLANFILE" 1
    trap 'rm -rf "$LOCKDIR"' EXIT
    trap 'exit 130' INT
    trap 'exit 143' TERM
 }
 # ------------------------------------------------------------------ predicates
 is_dir() {  # true only for a real directory, never a symlink to one
    [ -d "$1" ] && [ ! -L "$1" ]
 }
 is_ours() {  # true only when the invoking user owns the path
    owner=$(stat -c '%u' "$1" 2>/dev/null) || return 1
    [ "$owner" = "$UID_SELF" ]
 }
 refuse() {  # refuse <path> <reason>  — reported, never deleted
    printf '%s: refused %s (%s)\n' 'purge-agenttrash' "$1" "$2" >&2
    REFUSED=$((REFUSED + 1))
 }
 emit() {  # emit <line>  — silenced by --quiet
    [ "$QUIET" -eq 1 ] || printf '%s\n' "$1"
 }
 # --------------------------------------------------------------- the plan table
 # One row per candidate: `<action> <path> <line>`. The first two fields tell
 # `act_on` what to do; the rest of the row is EXACTLY what the run prints for
 # that candidate — in plan mode verbatim, in execute mode as the first line of
 # its report. The table is a file, so no shell quoting rule (quoted-for,
 # heredoc, IFS games) can corrupt an approved plan, and the two modes can
 # never drift apart. A removal's second line (`removed <path>`) is derived from
 # the row's action, so it is also fixed at plan time.
 CANDIDATES=0
 REFUSED=0
 record() {  # record <action> <path> <line>  — append one row to the plan table
    printf '%s %s %s\n' "$1" "$2" "$3" >>"$PLANFILE"
    CANDIDATES=$((CANDIDATES + 1))
 }
 # ------------------------------------------------------------- candidate rule
 # The whole contract in one place:
 #   `/` is refused outright (never a candidate, never walked);
 #   a root that is a symlink or a plain file is refused, never followed;
 #   a root (or any path) that IS a real .AGENTTRASH prints `plan empty <path>`:
 #     the convention persists, its staged contents are what gets cleared;
 #   otherwise the root's DIRECT .AGENTTRASH child is also `empty` — it is that
 #     level's own trash ($HOME/.AGENTTRASH, x1/.AGENTTRASH, repo-a/.AGENTTRASH);
 #   every .AGENTTRASH deeper than that is content: `remove`, printed as
 #     `plan remove <path>` (+ `removed <path>` when executed), unless it lives
 #     inside a tree already claimed (then it dies with that tree, never planned
 #     twice) or is foreign-owned/symlinked (then refused).
 collect_root() {  # collect_root <path> -> 0 ok, 1 refused
    target=$1
    if [ "$target" = / ]; then
        refuse / 'root refusal'
        return 1
    fi
    if [ -L "$target" ] || ! [ -d "$target" ]; then
        refuse "$target" 'not a real directory'
        return 1
    fi
    if ! is_ours "$target"; then
        refuse "$target" "not owned by uid $UID_SELF"
        return 1
    fi
    if [ "${target##*/}" = .AGENTTRASH ]; then
        record empty "$target" "plan empty $target"
        return 0
    fi
    # `own` is the root's own .AGENTTRASH once handled; hits below it belong to
    # that tree and must never be planned a second time.
    own=
    if [ -e "$target/.AGENTTRASH" ]; then
        if ! is_ours "$target/.AGENTTRASH"; then
            refuse "$target/.AGENTTRASH" "not owned by uid $UID_SELF"
        elif ! is_dir "$target/.AGENTTRASH"; then
            refuse "$target/.AGENTTRASH" 'not a real directory'
        else
            record empty "$target/.AGENTTRASH" "plan empty $target/.AGENTTRASH"
            own=$target/.AGENTTRASH
        fi
    fi
    # No `-type d`: implementations disagree on whether that matches a
    # symlink-to-directory (GNU find says no, others yes), so every entry named
    # .AGENTTRASH is collected and classified here instead of by `find`.
    if ! found=$(find "$target" -xdev -name .AGENTTRASH 2>/dev/null); then
        refuse "$target" 'find failed'
        return 1
    fi
    previous=
    for entry in $found; do
        [ -n "$entry" ] || continue
        if [ -n "$own" ]; then
            case $entry in
                "$own"|"$own"/*) continue ;;   # already emptied in place
            esac
        fi
        if [ -n "$previous" ]; then
            case $entry in
                "$previous"/*) continue ;;   # nested: dies with its parent
            esac
        fi
        previous=$entry
        if ! is_ours "$entry"; then
            refuse "$entry" "not owned by uid $UID_SELF"
            continue
        fi
        if ! is_dir "$entry"; then
            refuse "$entry" 'not a real directory'
            continue
        fi
        record remove "$entry" "plan remove $entry"
    done
    return 0
 }
 # ------------------------------------------------------------------ execution
 remove_tree() {  # rm -rf a real directory; symlinks inside die as entries
    is_dir "$1" || return 1
    rm -rf -- "$1"
 }
 empty_dir() {  # empty a real directory in place, never following symlinks
    is_dir "$1" || return 1
    find "$1" -mindepth 1 -xdev -delete
 }
 # act_on <action> <path> <line> -> 0 when the action succeeded.
 # remove: prints the stored line (`plan remove <path>`) then `removed <path>`.
 # empty:  prints the stored line (`plan empty <path>`) and nothing else — the
 #         directory survives, so the plan line IS the outcome.
 act_on() {  # act_on <action> <path> <line> -> 0 when the action succeeded
    if [ "$1" = remove ]; then
        if remove_tree "$2" 2>/dev/null; then
            emit "$3"
            emit "removed $2"
            return 0
        fi
        printf '%s: failed to remove %s\n' 'purge-agenttrash' "$2" >&2
        return 1
    fi
    if empty_dir "$2" 2>/dev/null; then
        emit "$3"
        return 0
    fi
    printf '%s: failed to empty %s\n' 'purge-agenttrash' "$2" >&2
    return 1
 }
 # Walk the plan table once: plan mode prints every row's line verbatim, execute
 # mode acts on each row and counts failures separately from plan-time refusals.
 walk_plan() {
    while read -r action path line; do
        [ -n "$action" ] || continue
        if [ "$MODE" = plan ]; then
            emit "$line"
        elif act_on "$action" "$path" "$line"; then
            REMOVED=$((REMOVED + 1))
        else
            EXEC_REFUSED=$((EXEC_REFUSED + 1))
        fi
    done <"$PLANFILE"
 }
 # ----------------------------------------------------------------------- main
 [ -n "$ROOTS" ] || ROOTS=$HOME
 acquire_lock
 for root in $ROOTS; do
    collect_root "$root" || true
 done
 REMOVED=0
 EXEC_REFUSED=0
 walk_plan
 STATUS=0
 [ "$REFUSED" -eq 0 ] || STATUS=1
 [ "$EXEC_REFUSED" -eq 0 ] || STATUS=1
 printf 'removed=%s deleted=%s refused=%s\n' \
    "$CANDIDATES" "$REMOVED" "$((REFUSED + EXEC_REFUSED))"
 if ! printf '%s\tplan\tcandidates=%s\tremoved=%s\trefused=%s\n' \
    "$(date -u '+%Y-%m-%dT%H:%M:%SZ')" "$CANDIDATES" "$REMOVED" \
    "$((REFUSED + EXEC_REFUSED))" >>"$LOGPATH" 2>/dev/null; then
    printf '%s: cannot write log %s\n' 'purge-agenttrash' "$LOGPATH" >&2
    STATUS=1
 fi
 exit "$STATUS"
@@ -0,0 +1,143 @@
 #!/usr/bin/env bash
 # qwen38-calib-window.sh — RUNS ON DANKSERVER. The calibration window, and nothing else:
 # boot the BF16 rig, wait until it is actually serving, push the corpus until the engine says
 # the tap is full, stop the rig, restore frank-archon, and bank the reference container.
 #
 # The reference recording is already taken (kld-ref/kld.json exists), so this window is the
 # calibration tap alone — the one job that needs both cards after it. The kldref phase is NOT
 # re-run here: the chained launcher that used to do that moved the banked kld-ref into trash
 # first, which is how a 16-hour phase gets paid for twice.
 #
 # Every phase is gated on the previous one, because the last time this ran unguarded
 # (2026-10-29) the rig's boot died of an OOM, the health loop quietly fell through, and the
 # pusher fed four rounds of 10M tokens into RESTORED PRODUCTION with nothing recorded:
 #
 #   1. the runner must still be alive and its container must answer /health AND own the
 #      listening socket, or this script kills it, restores the service and exits non-zero —
 #      no push, ever. The socket clause is not decoration: the runner stops frank-archon
 #      first, the dying process keeps its listener for a moment, and a rig container that
 #      is still loading has not bound the port yet, so `podman inspect` + `/health` both
 #      report success for the WRONG process (measured 2026-10-30: `health ok waited=0s`,
 #      then the pusher refused on the container name and the window died at second zero);
 #   2. the pusher refuses on its own unless the rig wrote its marker into the tap dir, the
 #      marker names this container, and the container is running (scripts/calib-push.py
 #      --tap-marker), so a second lock survives a mistake in this file;
 #   3. the reference container is banked into trash only when BOTH the serve and the push
 #      exited 0 and the tap dir holds grams. The checkpoint that reproduces it stays.
 #
 # knobs: Q38_WINDOW_CALIB_CORPUS (calib-7168.jsonl — the re-chunked file the rig's pool can
 #        hold), Q38_WINDOW_HEALTH_TIMEOUT (seconds to wait for /health), Q38_WINDOW_ROUNDS
 #        (pusher rounds; one is 10M tokens), Q38_WINDOW_KEEP_RAD=1 to leave the .rad in place.
 # Usage: scripts/qwen38-calib-window.sh [--dry-run]
 set -uo pipefail
 cd "$(dirname "${BASH_SOURCE[0]}")/.." || exit 9
 DRY=0
 [ "${1:-}" = "--dry-run" ] && DRY=1
 say() { printf '%s %s\n' "$(date -Is)" "$*"; }
 MODELS="${FRANK_QWEN38_MODELS:-/home/dank/models}"
 CAMPAIGN="${FRANK_QWEN38_CAMPAIGN:-/home/dank/campaign/qwen38-w5nl64}"
 RAD="${FRANK_QWEN38_RAD:-$MODELS/rad/qwen38-bf16.rad}"
 CORPUS="${Q38_WINDOW_CALIB_CORPUS:-/home/dank/corpora/out/calib-7168.jsonl}"
 CALIB="${FRANK_QWEN38_CALIB_DIR:-$CAMPAIGN/calib}"
 HEALTH_TIMEOUT="${Q38_WINDOW_HEALTH_TIMEOUT:-3600}"
 ROUNDS="${Q38_WINDOW_ROUNDS:-6}"
 RIG=qwen38-bf16
 SERVICE=frank-archon
 PY="${PY:-python3}"
 # ---- preflight: every fact by identity, one owner (scripts/qwen38-preflight.sh) ----
 # The checks that used to live here were proxies, and each lied at least once: the space check
 # asked a directory that merely LOOKED related, `/health` answered for a dying production
 # process, `podman inspect` named a container that existed but had bound nothing. The shared
 # script binds each fact to its object — cwd, the digest the tag actually resolves to, free
 # space on the tap's own filesystem, every /dev/kfd holder's HOST pid against the service's,
 # the :7770 listener's host pid, the corpus and the reference recording. A resumed window
 # keeps an existing tap, so the tap-empty fact is waived.
 [ -f "$RAD" ]    || { echo "REFUSED: no BF16 reference container at $RAD — nothing to serve; scripts/build-qwen38-bf16-rad.sh" >&2; exit 2; }
 FRANK_QWEN38_CAMPAIGN="$CAMPAIGN" \
 Q38_WINDOW_CALIB_CORPUS="$CORPUS" \
 Q38_CALIB_MIN_FREE_MIB="${Q38_CALIB_MIN_FREE_MIB:-61440}" \
    bash scripts/qwen38-preflight.sh --allow-existing-tap || exit 2
 ! [[ "$HEALTH_TIMEOUT" =~ ^[0-9]+$ ]] && { echo "REFUSED: Q38_WINDOW_HEALTH_TIMEOUT must be seconds (got: $HEALTH_TIMEOUT)" >&2; exit 2; }
 ! [[ "$ROUNDS" =~ ^[0-9]+$ ]] && { echo "REFUSED: Q38_WINDOW_ROUNDS must be a round count (got: $ROUNDS)" >&2; exit 2; }
 if [ "$DRY" = 1 ]; then
    echo "dry run: calib window — the service is left running, nothing is mounted or written"
    echo "would run: bash scripts/run-qwen38-bf16.sh calib          (rig boots, writes $CALIB/.rig-boot)"
    echo "would wait: /health up within ${HEALTH_TIMEOUT}s, else kill the rig, restore $SERVICE, exit 1"
    echo "would run: $PY scripts/calib-push.py --container $RIG --corpus $CORPUS --max-rounds $ROUNDS"
    echo "would bank: $RAD -> ~/.AGENTTRASH/$(date +%Y%m%d)-qwen38-bf16-rad/  (only if both phases exit 0)"
    exit 0
 fi
 say "calib window start pid=$$ corpus=$CORPUS rounds=$ROUNDS"
 [ -n "$(ls -A "$CALIB" 2>/dev/null)" ] && say "tap dir $CALIB is not empty; the engine's token floor is pooled, so an earlier partial tap still counts"
 # ---- phase 1: the rig -------------------------------------------------------
 say "phase=calib serve begin"
 bash scripts/run-qwen38-bf16.sh calib & spid=$!
 rig_up() {
    podman inspect -f '{{.State.Running}}' "$RIG" 2>/dev/null | grep -qx true || return 1
    curl -sf --max-time 10 -o /dev/null http://127.0.0.1:7770/health || return 1
    # The answer must come from the CONTAINER. The runner stops frank-archon first, and the
    # dying production process keeps its listener for a moment while the freshly created
    # rig container is still loading — so `container running` + `/health` 200 both hold
    # before the rig has bound anything. Measured 2026-10-30: `health ok waited=0s`, then the
    # pusher refused on the container name and the window died at second zero. Ask the kernel
    # who owns the listening socket and require it to be a rig process.
    local lpid
    lpid=$(ss -H -ltnp 'sport = :7770' 2>/dev/null | grep -o 'pid=[0-9]*' | head -1 | cut -d= -f2)
    [ -n "${lpid:-}" ] || return 1
    podman top "$RIG" pid 2>/dev/null | grep -qw "$lpid"
 }
 waited=0
 until rig_up; do
    if ! kill -0 "$spid" 2>/dev/null; then
        wait "$spid"; crc=$?
        say "phase=calib serve end rc=$crc (the rig exited before it ever answered /health)"
        echo "FAIL: the rig boot died (rc=$crc) — nothing was pushed; read /home/dank/logs/qwen38_bf16_calib.log" >&2
        exit 1
    fi
    if [ "$waited" -ge "$HEALTH_TIMEOUT" ]; then
        say "phase=calib health never came after ${HEALTH_TIMEOUT}s; killing the rig"
        podman stop "$RIG" >/dev/null 2>&1 || true
        wait "$spid"
        echo "FAIL: the rig never answered /health within ${HEALTH_TIMEOUT}s — nothing was pushed" >&2
        exit 1
    fi
    sleep 15
    waited=$((waited + 15))
 done
 say "phase=calib health ok waited=${waited}s"
 # ---- phase 2: the tap -------------------------------------------------------
 say "phase=calib push begin"
 "$PY" scripts/calib-push.py --port 7770 --parallel 8 --container "$RIG" \
    --tap-dir "$CALIB" --corpus "$CORPUS" --max-rounds "$ROUNDS"
 prc=$?
 say "phase=calib push end rc=$prc"
 podman stop "$RIG" >/dev/null 2>&1 || true
 wait "$spid"; crc=$?
 say "phase=calib serve end rc=$crc"
 if [ "$prc" -ne 0 ] || [ "$crc" -ne 0 ]; then
    echo "FAIL: serve rc=$crc push rc=$prc — the tap is NOT banked and $RAD stays: $(basename "$RAD")" >&2
    exit 1
 fi
 [ -n "$(ls -A "$CALIB" 2>/dev/null)" ] || { echo "FAIL: both phases exited 0 but the tap dir $CALIB is empty" >&2; exit 1; }
 say "TAP BANKED: $(ls -1 "$CALIB" | wc -l) file(s) in $CALIB"
 # ---- phase 3: bank the reference container ----------------------------------
 if [ "${Q38_WINDOW_KEEP_RAD:-0}" = "1" ]; then
    say "reference .rad kept (Q38_WINDOW_KEEP_RAD=1)"
 else
    T="${Q38_WINDOW_TRASH_ROOT:-$HOME/.AGENTTRASH}/$(date +%Y%m%d)-qwen38-bf16-rad"
    mkdir -p "$T" || { echo "FAIL: cannot create trash dir $T" >&2; exit 1; }
    mv "$RAD" "$T/" && say "reference .rad banked in $T (the checkpoint that rebuilds it stays)"
 fi
 say "calib window end"
@@ -0,0 +1,136 @@
 #!/usr/bin/env bash
 # qwen38-preflight.sh — the facts a calibration window must have BEFORE it takes the cards,
 # each checked by IDENTITY and not by proxy. Every line is PASS or FAIL with the reason.
 #
 # Why this file exists: on 2026-10-30 a window died at second zero because `/health` answered
 # 200 from the *production* process that was still draining its listener, while the freshly
 # created rig container had not bound anything. Other runs died from a port that answered for
 # the wrong process, a /dev/kfd held by an orphaned engine, a tap dir on a 35 GiB filesystem,
 # and a launcher whose cwd silently lacked the repo. None of those were math problems; all of
 # them were a check that measured something adjacent to the thing it meant.
 #
 # Usage:  bash scripts/qwen38-preflight.sh [--allow-existing-tap]
 # Exit:   0 all PASS · 2 any FAIL · 3 a fact could not be evaluated (missing tool)
 set -uo pipefail
 SELF_DIR="$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)"
 REPO_ROOT="$(cd "$SELF_DIR/.." && pwd)"
 CAMPAIGN="${FRANK_QWEN38_CAMPAIGN:-/home/dank/campaign/qwen38-w5nl64}"
 TAP="$CAMPAIGN/calib"
 CORPUS="${Q38_WINDOW_CALIB_CORPUS:-/home/dank/corpora/out/calib-7168.jsonl}"
 NEED_MIB="${Q38_CALIB_MIN_FREE_MIB:-61440}"
 ALLOW_EXISTING=0
 [ "${1:-}" = "--allow-existing-tap" ] && ALLOW_EXISTING=1
 # The producer pin is the calib image; the window fails closed if the tag on this host is not it.
 WANT_DIGEST="${Q38_PREFLIGHT_IMAGE_DIGEST:-sha256:69add58f7de1ed2bca2b1e20a4cc75717557df3f73569a178eca5aafe8c44093}"
 IMAGE_REF="${Q38_CALIB_IMAGE:-stilldeadcode/radiance@$WANT_DIGEST}"
 SERVICE="frank-archon"
 PORT=7770
 fails=0
 ok()   { printf 'PASS  %-14s %s\n' "$1" "$2"; }
 bad()  { printf 'FAIL  %-14s %s\n' "$1" "$2"; fails=$((fails + 1)); }
 skip() { printf 'WARN  %-14s %s\n' "$1" "$2"; }
 # 1. cwd — a detached launcher that lost its cwd fails here in a second, not in the log later.
 if [ "$(cd "$REPO_ROOT" && pwd)" = "$(pwd)" ] && [ -x "$REPO_ROOT/scripts/run-qwen38-bf16.sh" ]; then
    ok cwd "running from the repo root ($REPO_ROOT)"
 else
    bad cwd "cwd is $(pwd), not the repo root ($REPO_ROOT) — every relative script path would break"
 fi
 # 2. image — resolve the REF to a digest. A stale local `:latest` is a different model of the world.
 if command -v podman >/dev/null 2>&1; then
    # An image carries EVERY digest it was pulled under (a tag plus the by-digest pull), so this
    # asks whether the pin is one of them — reading RepoDigests[0] reports a healthy image as wrong.
    got="$(podman image inspect -f '{{range .RepoDigests}}{{println .}}{{end}}' "$IMAGE_REF" 2>/dev/null \
           | sed 's/.*@//' | grep -c "^${WANT_DIGEST}$")"
    if [ "${got:-0}" -ge 1 ]; then
        ok image "$IMAGE_REF carries ${WANT_DIGEST:0:19}…"
    elif podman image exists "$IMAGE_REF" 2>/dev/null; then
        bad image "$IMAGE_REF is present but does not carry $WANT_DIGEST (a retagged or older pull)"
    else
        bad image "$IMAGE_REF is not present on this host"
    fi
 else
    skip image "podman not on PATH — cannot resolve $IMAGE_REF"
 fi
 # 3. space — df the FILE PATH the seal writes to, never a directory that merely looks related.
 if [ -d "$TAP" ] || mkdir -p "$TAP" 2>/dev/null; then
    avail="$(df --output=avail -B 1M "$TAP" 2>/dev/null | tail -1 | tr -d ' ')"
    if [[ "${avail:-}" =~ ^[0-9]+$ ]] && [ "$avail" -ge "$NEED_MIB" ]; then
        ok space "$avail MiB free on $(df --output=target "$TAP" | tail -1) (need $NEED_MIB)"
    else
        bad space "${avail:-unknown} MiB free on $TAP, need $NEED_MIB — one seal writes ~43 GB"
    fi
    # 4. tap cleanliness — stale grams from another run must never be mixed in.
    n=$(ls -A "$TAP" 2>/dev/null | grep -c . || true)
    if [ "$n" -eq 0 ] || [ "$ALLOW_EXISTING" = 1 ]; then
        ok tap-empty "$n entries in $TAP"
    else
        bad tap-empty "$n entries in $TAP — rotate them to .AGENTTRASH first"
    fi
 else
    bad space "cannot create $TAP"
 fi
 # 5. device — every /dev/kfd holder must be a process of the service we are about to stop. An
 #    orphaned engine from a previous run makes the next boot die with `startup stopped: device
 #    error`. Identity comes from podman's own host-pid list: `podman top <ctr> pid` prints
 #    CONTAINER-internal pids (1, 7, …), which compare against nothing, and a rootless cgroup
 #    carries the container ID, never the service name — both are proxies that lie.
 PROD_CTR="${Q38_PREFLIGHT_PROD_CONTAINER:-$SERVICE}"
 service_pids() { podman top "$PROD_CTR" hpid 2>/dev/null | tail -n +2; }
 if [ -e /dev/kfd ]; then
    mine="$(service_pids | tr '\n' ' ')"
    foreign=""
    for p in $(fuser /dev/kfd 2>/dev/null); do
        case " $mine " in
            *" $p "*) : ;;                                          # the service we will stop
            *) foreign="$foreign $p:$(cat /proc/$p/comm 2>/dev/null)" ;;
        esac
    done
    if [ -z "$foreign" ]; then
        ok device "no foreign /dev/kfd holders"
    else
        bad device "foreign holders:$foreign — kill by pid, never by pattern"
    fi
 else
    skip device "/dev/kfd absent — not an AMD host?"
 fi
 # 6. port — a listener on :7770 must be a process of the service we stop, never an unknown one.
 lpid="$(ss -H -ltnp "sport = :$PORT" 2>/dev/null | grep -o 'pid=[0-9]*' | head -1 | cut -d= -f2)"
 if [ -z "${lpid:-}" ]; then
    ok port ":$PORT is free"
 else
    mine="$(service_pids | tr '\n' ' ')"
    case " $mine " in
        *" $lpid "*) ok port ":$PORT is held by $SERVICE (host pid $lpid) — the window stops it" ;;
        *)           bad port ":$PORT is held by host pid $lpid ($(cat /proc/$lpid/comm 2>/dev/null)), not $SERVICE" ;;
    esac
 fi
 # 7. corpus — the push source must exist and be the re-chunked file the window documents.
 if [ -s "$CORPUS" ]; then
    ok corpus "$(wc -l < "$CORPUS") prompts in $CORPUS"
 else
    bad corpus "$CORPUS is missing or empty"
 fi
 # 8. reference — the KL gate has nothing to compare against without it.
 if [ -s "$CAMPAIGN/kld-ref/kld.json" ]; then
    ok reference "$(du -h "$CAMPAIGN/kld-ref/kld.json" | cut -f1) kld.json present"
 else
    bad reference "$CAMPAIGN/kld-ref/kld.json is missing"
 fi
 echo
 if [ "$fails" -gt 0 ]; then
    echo "PREFLIGHT FAILED: $fails fact(s). The window must not take the cards."
    exit 2
 fi
 echo "PREFLIGHT PASSED: every fact checked by identity."
@@ -0,0 +1,94 @@
 #!/usr/bin/env python3
 """Shared plumbing for the qwen38 campaign corpus tools.
 The three tools move the same two shapes — a JSONL of prompt records and a
 one-artifact JSON manifest — and they all tokenize with the SAME tokenizer. That
 last part is not a detail: a KLD reference is only meaningful against the exact
 token sequence the corpus recorded, so `n_tokens` exists once, here, and every
 tool counts the same way. A window that decode/encode does not round-trip is a
 refusal, never a warning.
 Loaded as a sibling module. Each tool puts its own directory on `sys.path`
 first, so `python3 scripts/<tool>.py` and the driver's container mount
 (`runpy.run_path('/work/<tool>.py')`, `-v scripts:/work:ro`) both resolve it —
 run the tools by path, not piped through stdin.
 """
 import contextlib
 import json
 import os
 import sys
 import tempfile
 def refuse(msg: str) -> None:
    """Exit 2 with a REFUSED line — the code argparse uses for unusable input."""
    print(f"REFUSED: {msg}", file=sys.stderr)
    sys.exit(2)
 def load_tokenizer(path: str):
    try:
        from tokenizers import Tokenizer
    except ImportError as exc:  # the corpus tools carry their own venv
        refuse(f"tokenizers is not importable by {sys.executable} — run the "
               f"corpus venv's python (on dankserver: "
               f"/home/dank/venvs/radcorpus/bin/python) or set "
               f"FRANK_QWEN38_TOKENIZER: {exc}")
    if not os.path.isfile(path):
        refuse(f"no tokenizer at {path}")
    return Tokenizer.from_file(path)
 def n_tokens(tok, text: str) -> int:
    """The corpus's own count: no special tokens added, prompt side only."""
    return len(tok.encode(text, add_special_tokens=False).ids)
 def iter_jsonl(path: str):
    """Yield `(lineno, record)` for every non-blank line; refuse on a bad line.
    Field validation stays with the caller: the three corpora carry different
    fields (`prompt` alone, `prompt`+`cls`, `prompt`+`score_from`+`source`), and
    a record missing the one the tool needs is that tool's refusal to make.
    """
    with open(path, encoding="utf-8", errors="replace") as fh:
        for lineno, line in enumerate(fh, 1):
            if not line.strip():
                continue
            try:
                rec = json.loads(line)
            except json.JSONDecodeError as exc:
                refuse(f"{path}:{lineno} is not JSON: {exc}")
            yield lineno, rec
 def write_jsonl(path: str, rows) -> None:
    with open(path, "w", encoding="utf-8") as fh:
        for row in rows:
            fh.write(json.dumps(row, ensure_ascii=False) + "\n")
@contextlib.contextmanager
 def atomic_writer(path: str):
    """Write `path` through a temporary sibling and rename on success.
    An interrupted run then leaves no half corpus behind, and the temporary name
    keeps the one-artifact-one-name law the driver applies to a score report.
    """
    d = os.path.dirname(os.path.abspath(path)) or "."
    fd, tmp = tempfile.mkstemp(dir=d, prefix="." + os.path.basename(path) + "-")
    try:
        with os.fdopen(fd, "w", encoding="utf-8") as fh:
            yield fh
        os.replace(tmp, path)
    except BaseException:
        if os.path.exists(tmp):
            os.unlink(tmp)
        raise
    os.chmod(path, 0o644)
 def write_json(path: str, obj) -> None:
    with atomic_writer(path) as fh:
        json.dump(obj, fh, indent=1)
        fh.write("\n")
@@ -0,0 +1,192 @@
 #!/usr/bin/env python3
 """Re-chunk a KLD corpus so every document fits the container that scores it.
 A KLD reference is recorded by booting one container and prefilling every corpus
 document through it. radiance refuses a document the KV pool cannot hold on its
 own (core/mem/kv.cpp `can_ever_fit`: a paged group holds `n_blocks *
 block_size` tokens), and the number that matters is printed by the boot ledger,
 not by a flag:
    W kv group 'kv_attn' holds 7436 tokens but the context limit is 32768, so
    a prompt longer than 7436 tokens is refused (400) ... or lower
    --max-model-len to 7436
 The BF16 qwen4exp container at the campaign's shape gets 270.00 MiB of cache --
 the expert plane owns both cards -- so its real window is 7,436 tokens however
 `--max-model-len` is spelled. The corpus the campaign inherited from the swift15
 era is 197 documents of 8,193-11,970 tokens, because make-calib-corpus.py's
 `--kld-min-prompt 8192` sets a floor and no ceiling: every one of them is
 unscoreable there, and three drained windows have proved it.
 This script moves the document boundaries and nothing else. Each record's token
 ids -- counted with the same tokenizer the generator used -- are cut into
 windows of at most `--max-doc-tokens`, so a record's windows concatenate back to
 that record's own token sequence, unchanged: no token dropped, rewritten or
 reworded. Each window gets its own `score_from = len - --score-tail`, the same
 relation the generator wrote for the parent, so the reference scores 256
 positions per window instead of per document: 100,864 measured positions over
 the same text against the 50,432 the inherited corpus scored, at a shorter
 context. The calibration corpus is untouched, so the quantization itself does
 not move.
 Usage:
  rechunk-kld-corpus.py --corpus kld.jsonl --out kld.jsonl.new \
      [--tokenizer-json PATH] [--max-doc-tokens 7168] [--score-tail 256] [--force]
 --out must not exist and be non-empty without --force: the same
 one-artifact-one-name law run-qwen38-bf16.sh applies to a score report. The file
 is written through a temporary name and renamed, so an interrupted run leaves no
 half corpus behind. A JSON summary of what was written goes to stdout.
 """
 import argparse
 import dataclasses
 import json
 import os
 import sys
 _HERE = os.path.dirname(os.path.abspath(__file__)) if "__file__" in globals() else os.getcwd()
 if _HERE not in sys.path:
    sys.path.insert(0, _HERE)
 import qwen38_common as qc
 MISSING = object()  # "the parent record carried no source field"
 def parse_args() -> argparse.Namespace:
    ap = argparse.ArgumentParser(description=__doc__.splitlines()[0])
    ap.add_argument("--corpus", required=True,
                    help="the kld.jsonl to re-chunk (records: prompt, score_from, source)")
    ap.add_argument("--out", required=True,
                    help="file to write; refuse if it exists and is non-empty without --force")
    ap.add_argument("--tokenizer-json",
                    default=os.environ.get("FRANK_QWEN38_TOKENIZER",
                                           "/home/dank/models/Qwen3.8-Flash-Next/tokenizer.json"),
                    help="the tokenizer the corpus was authored with (the same one the "
                         "manifest in the corpus output dir names)")
    ap.add_argument("--max-doc-tokens", type=int, default=7168,
                    help="the widest document the scoring container can hold; 7,168 is "
                         "7x1024, the largest whole kilopage under the 7,436 the bf16 "
                         "container's ledger reports")
    ap.add_argument("--score-tail", type=int, default=256,
                    help="positions scored at the end of every document, as the generator "
                         "recorded it")
    ap.add_argument("--force", action="store_true",
                    help="overwrite a non-empty --out (after moving the old file to trash)")
    return ap.parse_args()
@dataclasses.dataclass(frozen=True)
 class Window:
    """One scoring document cut out of a parent record."""
    prompt: str
    score_from: int
    n_tokens: int
    clamped: bool
    source: object = MISSING
    def record(self) -> dict:
        rec = {"prompt": self.prompt, "score_from": self.score_from}
        if self.source is not MISSING:
            rec["source"] = self.source
        return rec
 def validate(a: argparse.Namespace) -> None:
    """Everything that makes this run unusable, named before a token is cut."""
    if a.max_doc_tokens <= 0:
        qc.refuse(f"--max-doc-tokens must be positive (got {a.max_doc_tokens})")
    if a.score_tail <= 0:
        qc.refuse(f"--score-tail must be positive (got {a.score_tail})")
    if a.max_doc_tokens <= a.score_tail:
        qc.refuse(f"--max-doc-tokens {a.max_doc_tokens} cannot hold a "
                  f"{a.score_tail}-position score tail: every window would be all "
                  "tail and no context")
    if not os.path.isfile(a.corpus):
        qc.refuse(f"no corpus at {a.corpus}")
    if os.path.realpath(a.out) == os.path.realpath(a.corpus):
        qc.refuse(f"--out {a.out} is the corpus itself; move it to trash first — the "
                  "trash law means the old segmentation stays recoverable")
    if os.path.exists(a.out) and os.path.getsize(a.out) > 0 and not a.force:
        qc.refuse(f"{a.out} exists and is not empty; pass --force after moving it to trash")
 def cut(rec: dict, tok, a: argparse.Namespace,
        where: str) -> tuple[list[Window], int]:
    """Cut one record's token ids into windows, proving each one round-trips.
    The round-trip encode is the whole reason this tool may claim the reference
    scores the corpus's own token sequence: decoding a window to text and
    encoding it back must return the identical ids, or the score is of a
    different document than the one on disk.
    """
    ids = tok.encode(rec["prompt"], add_special_tokens=False).ids
    windows: list[Window] = []
    for start in range(0, max(len(ids), 1), a.max_doc_tokens):
        window = ids[start:start + a.max_doc_tokens]
        if not window:
            continue
        text = tok.decode(window)
        if tok.encode(text, add_special_tokens=False).ids != window:
            qc.refuse(f"{where} window at token {start} does not round-trip through "
                      "text — the reference would score a different token sequence "
                      "than the corpus holds")
        windows.append(Window(prompt=text,
                              score_from=max(len(window) - a.score_tail, 0),
                              n_tokens=len(window),
                              clamped=len(window) < a.score_tail,
                              source=rec["source"] if "source" in rec else MISSING))
    return windows, len(ids)
 def rechunk(a: argparse.Namespace, tok) -> dict:
    """Write the re-chunked corpus and return the summary it printed."""
    docs_in = docs_out = tokens_in = tokens_out = 0
    longest_child = clamped = scored = 0
    sources: dict[str, int] = {}
    with qc.atomic_writer(a.out) as out:
        for lineno, rec in qc.iter_jsonl(a.corpus):
            where = f"{a.corpus}:{lineno}"
            if not isinstance(rec.get("prompt"), str):
                qc.refuse(f"{where} has no string 'prompt' field")
            windows, n_parent = cut(rec, tok, a, where)
            docs_in += 1
            tokens_in += n_parent
            for w in windows:
                out.write(json.dumps(w.record(), ensure_ascii=False) + "\n")
                docs_out += 1
                tokens_out += w.n_tokens
                longest_child = max(longest_child, w.n_tokens)
                clamped += 1 if w.clamped else 0
                scored += w.n_tokens - w.score_from
                if "source" in rec:
                    for cls in str(rec["source"]).split("+"):
                        sources[cls] = sources.get(cls, 0) + 1
    if tokens_out != tokens_in:
        raise SystemExit(f"REFUSED: the windows hold {tokens_out} tokens and the corpus "
                         f"has {tokens_in} — re-chunking lost or invented tokens")
    return {
        "corpus": a.corpus,
        "out": a.out,
        "tokenizer": os.path.basename(a.tokenizer_json),
        "documents_in": docs_in,
        "documents_out": docs_out,
        "tokens_in": tokens_in,
        "tokens_out": tokens_out,
        "longest_document": longest_child,
        "max_doc_tokens": a.max_doc_tokens,
        "score_tail": a.score_tail,
        "scored_positions": scored,
        "clamped_score_from": clamped,
        "source_documents": sources,
    }
 def main() -> None:
    a = parse_args()
    validate(a)
    print(json.dumps(rechunk(a, qc.load_tokenizer(a.tokenizer_json)), indent=1))
 if __name__ == "__main__":
    main()
@@ -0,0 +1,80 @@
 # qwen4exp-qwen38-w5nl64-bf16trunk.recipe — every routed expert, BOTH projections, at five bits:
 # u5 codes into libquant's 32-level w5nl table, an E4M3 scale a group of 64 under the fixed 2^-13
 # second level, rotated by the 128-wide Hadamard, chosen by GPTQ against $CALIB and swept three
 # times by coordinate descent. This is `w5nl64a8h` in the engine's served-encoding table — the top of
 # the four-bit class — applied wider than any published radiance container of this checkpoint: the
 # author's own w5 rung (`qwen4exp-w5dn64-i8-hc8m.recipe`) moves only `down`, because that rung was
 # priced as a byte-for-error trade against prefill; this campaign buys the whole rung and pays for
 # it out of the pinned host pool (+15.3 GiB of expert plane, `--host-pool-mib 61440` per rank), not
 # out of the KV pool.
 #
 # Every other line is the served container's recipe (dump it with
 # `rad-info --recipe /home/dank/models/rad/qwen3.8-next-flash-fp8-iq4r-moe.rad`), minus the rows this
 # campaign leaves at the checkpoint:
 #
 #   * the trunk's block linears, the shared expert and the hyper-connection mixers are NOT listed —
 #     dropping their i8/fp8 lines leaves them bf16, which is what the checkpoint holds. Measured bill
 #     +1.978 GiB of always-resident weight = ~131k tokens of KV off the pool.
 #   * `ssm_inz`/`ssm_out` STAY int8: the delta-net gates are the largest trunk rows (+1.9 GiB, 126k
 #     tokens) and the least sensitive per byte.
 #   * `mtp.draft_head` STAYS 2-bit and the PLE n-gram table STAYS one fixed E4M3 scale: the draft
 #     head's rung is cheap and its acceptance cost is a decode question this campaign does not touch,
 #     and bf16 for the n-gram table would pin +47.69 GiB of host RAM for a lookup.
 #
 # Protected experts: none in this file. `scripts/apply-protected-cast.sh` appends the derived set
 # (48, two per layer so each rank's 256-expert half stays even — the four-bit GEMM takes the rest of
 # a half as two equal tables by parity) from the fresh 12M-token energy/routing census, and they must
 # sit ABOVE the expert row because the first rule that matches a weight decides it.
 #
 #   CALIB=/home/dank/campaign/qwen38-w5nl64/calib \
 #     rad-convert /home/dank/models/Qwen3.8-Flash-Next \
 #       --recipe this.file --imatrix .../imatrix.gguf -o out.rad
 #
 # The arm driver (`scripts/convert-qwen38-w5nl64.sh`) generates the per-arm variants of the expert
 # row from this template — cd=3 (control), cd=6, and `act_order=1` with NO cd= (libquant refuses the
 # pair, lq_registry.cpp:105) — rather than keeping three near-duplicate recipes that can drift apart.
 #
 # The whole campaign, in the order that fits on one disk (every step runs on dankserver):
 #
 #   1. scripts/download-qwen38-bf16.sh --fetch      the vendor BF16 checkpoint, 335.3 GiB
 #   2. scripts/build-qwen38-bf16-rad.sh             the BF16 reference container, another 335.3 GiB
 #   3. one drained window, two boots:
 #        scripts/run-qwen38-bf16.sh kldref          the KLD reference recording (the zero of scale)
 #        scripts/run-qwen38-bf16.sh calib           the 12M-position calibration tap -> grams
 #   4. trash qwen38-bf16.rad                        the peak-disk law: the converts need its room
 #   5. scripts/grams-to-imatrix.py                  grams -> imatrix.gguf in the campaign dir
 #   6. scripts/convert-qwen38-w5nl64.sh             the arms, planned and billed before each one runs
 #   7. fetch the published FP8-native release, then per container:
 #        scripts/run-qwen38-bf16.sh score RAD OUT   fp8 anchor, served iq4r, each arm
 #      then scripts/kld-gate-qwen38.sh fp8.json iq4r.json cand.json
 #   8. a winner is a trio edit plus `scripts/deploy.sh dankserver --service frank-archon`; the
 #      rollback is the commit before it. If the win needs the bigger pinned plane, raise
 #      FRANK_ARCHON_HOST_POOL_MIB and FRANK_ARCHON_TTM_TARGET_MIB in the SAME commit: the preflight
 #      floor is 2xHOST_POOL + 48,824 + prefix cache + MNBT + the store, and a pool raised without
 #      the TTM target is a boot refusal that Restart=always repeats forever, invisible to deploy.sh.
 # PROTECTED-BEGIN — scripts/apply-protected-cast.sh splices the derived cast lines in here.
 # They MUST sit above the expert rule: the first rule whose glob matches a weight decides it, so a
 # cast line below the five-bit rule would never win. The set is derived from the campaign's own
 # 12M-token recording (`derive-protected-experts.py --target-count 48` over the fresh grams), and the
 # driver splices it into a per-arm COPY so this template stays the clean, reviewable baseline.
 # PROTECTED-END
 # Every routed expert, trunk and MTP layer alike, both projections, at five bits. The experts' scales
 # sit between 2^-15 and 2^-10.6 under w4nl's top level of 22 and half a binade lower under w5nl's 30
 # — inside E4M3's normal range under 2^-13 either way. The MTP layer has no recording and is rounded
 # to nearest, which libquant says on stderr; it is one layer of 49 and only ever changes drafts.
 blk.*.ffn_*_exps.*.weight            gptq table=w5nl group=64 scale=fp8_e4m3 scale2=f32 block2=*x* scale2_value=0.0001220703125 transform=fwht128 rule=search cd=3 calib=$CALIB
 # The delta-net's in/out gates, int8 with a bf16 scale a group of 128 searched for least error — the
 # served recipe's lines, unchanged.
 blk.*.ssm_inz.weight                 rtn codes=i8 group=128 scale=bf16 clamp=sym rule=search
 blk.*.ssm_out.weight                 rtn codes=i8 group=128 scale=bf16 clamp=sym rule=search
 # The MTP head's two mixing rows, E4M3 as served. They only ever change the drafts.
 mtp.fc_hidden.weight                 rtn codes=fp8_e4m3 group=128 scale=f32
 mtp.fc_embedding.weight              rtn codes=fp8_e4m3 group=128 scale=f32
 # The n-gram table at one fixed E4M3 scale, and the 2-bit draft head — both the served lines.
 blk.*.ple_ngram.weight               rtn codes=fp8_e4m3 block=*x* scale=bf16 rule=fixed scale_value=1.99317932128906e-4
 mtp.draft_head.weight                rtn codes=u2 zero=u8 group=128 scale=f16
@@ -0,0 +1,485 @@
 #!/usr/bin/env bash
 # run-qwen38-bf16.sh — RUNS ON DANKSERVER. Boot a container over the campaign's
 # ONE measuring shape for the phases that need both R9700s. The engine owns the
 # cards, so frank-archon is stopped first and restored on every exit path — the
 # autoresearch pattern: stop, never mask (the unit file in /etc/systemd/system
 # is a symlink into the checkout and mask fails on a relinked unit; the deploy
 # restores it).
 #
 #   kldref            --kld-record the reference log-probabilities: the engine
 #                     runs /home/dank/corpora/out/kld.jsonl (197 prompts, 50,432
 #                     scored positions, score tail 256) through prefill, writes
 #                     /home/dank/campaign/qwen38-w5nl64/kld-ref, and EXITS. The
 #                     model behind it is the BF16 container from
 #                     scripts/build-qwen38-bf16-rad.sh — the vendor checkpoint's
 #                     own numbers, which is what makes it the zero point.
 #   score RAD OUT     --kld-ref score ONE container against that recording and
 #                     EXIT: the FP8-native anchor, the served iq4r, and every
 #                     arm. Deliberately the SAME engine pin, the same TP, the
 #                     same step size, the same fp8 KV, --deterministic and no
 #                     speculation as kldref — KL and top-1 agreement are only
 #                     comparable across boots when everything except the weights
 #                     is identical, so this mode is one argv with --model and
 #                     the two --kld-* flags changed, never a second shape.
 #                     RAD must sit under the models dir (the container reads it
 #                     through the ro /models mount); OUT lands on a writable
 #                     mount because --kld-out F also writes F.rows beside it.
 #   calib             serve on :7770 with the calibration tap open
 #                     (RADIANCE_CALIB_DIR + RADIANCE_CALIB_TOKENS=12000000). A
 #                     separate driver — scripts/calib-push.py --port 7770
 #                     --container qwen38-bf16 --corpus
 #                     /home/dank/corpora/out/calib-7168.jsonl — pushes the corpus;
 #                     at the floor the engine writes the Grams and the tap switches
 #                     itself off (calib-push prints TAP_DONE). Stop this script
 #                     after that; the trap brings frank-archon back. THE FLOOR IS
 #                     THE CAMPAIGN'S RULE (12 M tokens, Frank 2026-10-30): the
 #                     served iq4r this arm competes with was calibrated on a 10 M-
 #                     token set, so the arm is calibrated at least as deeply. One
 #                     round of the re-chunked corpus is ~9.31 M tokens, so sealing
 #                     takes round 1 plus part of round 2 — a round that seals
 #                     nothing is NORMAL (the floor is pooled), and the pusher's
 #                     round_verdict treats a flat round as "continue", never as a
 #                     failure. One seal writes ~40 GiB.
 #
 # Shape: the vendor's bf16 calibration lines — expert_tiered placement with the
 # weights disk tier, deterministic (the resident set pinned to the loader's plan,
 # so a recorded reference and a scored candidate see the same one), no
 # speculation, fp8 KV, a 32,768-token window, the PLE n-gram table pinned to RAM,
 # and the production chat template rather than the one baked into the container —
 # every scored container is asked the same tokenised questions the reference
 # answered. The window the flag spells is 32,768; the pool this payload's boot
 # actually gets is 7,436 tokens, so kld.jsonl is re-chunked by
 # scripts/rechunk-kld-corpus.py into documents of at most 7,168 tokens — the
 # widest one the reference can hold. The kldref gate below re-checks the file
 # against the declared window, and the re-chunker is what keeps that honest.
 #
 # NO --vram-kv-mib: it was REMOVED at 1.3.0 and naming it is a boot refusal, not
 # a no-op — verified on the pinned image 2026-10-27: "E --vram-kv-mib was
 # removed: the card is split between the weights and the cache from what the
 # model needs." The resolver derives the split from --host-pool-mib,
 # --gpu-headroom-mib and the activation arena; the rig inherits the first two from
 # frank-archon.env, the file that owns them. On THIS payload the host pool is not
 # the cache's supplier: the boot lands in vram_budget.cpp's `no lender` branch,
 # where the experts keep their plan-fixed 25.29 GiB and the cache gets the residue.
 # Measured twice — 12,288 and 49,152 MiB of host pool both printed the same
 # "kv cache 271.25 MiB ... 7,436"; only --gpu-headroom-mib moved anything
 # (768 -> 96 grew vram_weights 29.41 -> 30.06 GiB). The supplier the source names
 # is the arena: vram_budget.cpp:117, "lower --max-num-batched-tokens (the arena
 # scales with it)" — which is what the step size below now does.
 #
 # The pool, not the window, decides what the recorder accepts: it refuses any
 # document longer than the pool, so the 32,768-token window was decorative while the
 # cache held 7,436 — "E kld: document 196 is 8507 tokens and the KV pool holds
 # 7436, so it can never be scored" on all 197 records, exit 1. The longest record
 # is 10,554 tokens by the engine's own count (chars/3.6 for this file, not the
 # chars/4.97 this repo measured on the K3 corpus).
 #
 # The step size is ONE value for the recording and every scoring pass, and it is
 # the value frank-archon.env owns (FRANK_ARCHON_MAX_BATCHED_TOKENS, 2,048): the
 # quantised expert FILE tiers (the served iq4r, the arms) stream only at
 # MNBT<=2048 and refuse the widened form above it, and the arena a step costs the
 # card comes straight out of the cache — 1.08 GiB at a 2,048-token step against
 # 504.39 MiB at 1,024. The corpora are cut to the window the boot ledger reports,
 # which is what the pool raise is for (frank-archon.env: HOST_POOL_MIB); a narrow
 # step was never the fix, it was the symptom's disguise.
 #
 # usage: scripts/run-qwen38-bf16.sh kldref|calib|score RAD OUT [--dry-run] [--image REF]
 #   (RAD and OUT are score's two positionals: RAD is a container under the models
 #    dir, OUT the report path whose directory is created if missing)
 #   --dry-run   print the exact `podman run` argv, one shell-quoted argument a
 #               line, and exit 0: the service is NOT stopped, nothing is
 #               mounted, nothing is written — the offline test seam. It still
 #               validates the argv's own shape (paths through the /models mount,
 #               a step size that is a number), because an argv that could not
 #               boot is not worth printing.
 #   --image REF override the digest pin (a probe boot, or the test seam)
 #
 # knobs: FRANK_QWEN38_MODELS (the dir mounted at /models), FRANK_QWEN38_RAD (the
 #        bf16 container, kldref only), FRANK_QWEN38_CAMPAIGN (holds kld-ref/ and
 #        calib/), FRANK_QWEN38_KLD_CORPUS, FRANK_QWEN38_TPL (the template copy
 #        the container reads through /models), FRANK_QWEN38_LOG_DIR, and the step
 #        and slot counts inherited as FRANK_ARCHON_MAX_BATCHED_TOKENS /
 #        FRANK_ARCHON_MAX_SEQS — no per-mode literals (the audit law: a mode that
 #        carries its own step measures a shape nothing serves; calib is exempt, see
 #        Q38_CALIB_MNBT / Q38_CALIB_SEQS below — its output is never compared) —
 #        and FRANK_QWEN38_MAX_MODEL_LEN — the window every mode boots at (32,768;
 #        see the corpus census above, and raise it together with the corpus).
 # kldref refuses (exit 2) when the longest record of the corpus it is about to
 # read is longer than MAX_MODEL_LEN x 3 characters — chars/3 being the
 # pessimistic token:char ratio, so the script says no before the engine finds out
 # inside a drained window, after the checkpoint work is already done. score
 # refuses when the reference dir is absent, when RAD is not on disk or not under
 # the models mount, or when OUT already exists: the trash law means a second
 # score over the same name is a deliberate act, and two numbers in one file is
 # how a campaign loses its own history.
 # Engine stdout goes to $LOG_DIR/qwen38_bf16_<mode>.log — the boot ledger is the
 # record of what the resolver actually gave the cards; the journal never sees it.
 # exit: the engine's code; 2 = refused before the service was touched
 set -uo pipefail
 SCRIPT_DIR="$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)"
 # shellcheck disable=SC1091
 source "$SCRIPT_DIR/frank-archon.env"
 IMAGE="${FRANK_ARCHON_IMAGE:?FRANK_ARCHON_IMAGE must name the radiance image}"
 # THE CALIBRATION PIN: the LATEST radiance, pinned by digest because a tag is a
 # moving target and a tap must name the producer that wrote it. `:latest` on
 # 2026-10-30 resolves to v1.14.0 (revision 4a61d7296bc5), and on that image a
 # calib boot prints the arming line — "recorded passes: off, the program
 # accumulates calibration Grams (gram_accum)" — where the production pin
 # (1.8.1) prints nothing of the kind: a 17-hour run on 1.8.1 wrote ZERO bytes
 # into the tap dir while /health, /stats and the push all looked healthy
 # (measured 2026-10-30). The tap code is identical in both trees' sources, so
 # the shipped 1.8.1 build is what drops it; do not spend another window on it.
 # Any other image must be named deliberately with --image.
 CALIB_IMAGE="${Q38_CALIB_IMAGE:-stilldeadcode/radiance@sha256:69add58f7de1ed2bca2b1e20a4cc75717557df3f73569a178eca5aafe8c44093}"
 IMAGE_EXPLICIT=0
 MODE=""
 POS=()
 DRY=0
 while [ $# -gt 0 ]; do
    case "$1" in
        kldref|calib|score) MODE="$1" ;;
        --dry-run)          DRY=1 ;;
        --image)            [ "$#" -ge 2 ] || { echo "--image needs a reference" >&2; exit 2; }
                            IMAGE="$2"; IMAGE_EXPLICIT=1; shift ;;
        *)
            # score's two positional arguments, and nothing else may be bare.
            if [ "$MODE" = score ] && [ "${#POS[@]}" -lt 2 ]; then
                POS+=("$1")
            else
                echo "usage: ${0##*/} kldref|calib|score RAD OUT [--dry-run] [--image REF]  (unknown argument: $1)" >&2
                exit 2
            fi ;;
    esac
    shift
 done
 if [ -z "$MODE" ]; then
    echo "usage: ${0##*/} kldref|calib|score RAD OUT [--dry-run] [--image REF]  (no mode given)" >&2
    exit 2
 fi
 if [ "$MODE" = score ] && [ "${#POS[@]}" -ne 2 ]; then
    echo "usage: ${0##*/} score RAD OUT  — RAD is the container to score, OUT the report to write (got ${#POS[@]} argument(s))" >&2
    exit 2
 fi
 MODELS="${FRANK_QWEN38_MODELS:-/home/dank/models}"
 CAMPAIGN="${FRANK_QWEN38_CAMPAIGN:-/home/dank/campaign/qwen38-w5nl64}"
 KLD_CORPUS="${FRANK_QWEN38_KLD_CORPUS:-/home/dank/corpora/out/kld.jsonl}"
 RAD="${FRANK_QWEN38_RAD:-$MODELS/rad/qwen38-bf16.rad}"
 TPL="${FRANK_QWEN38_TPL:-$MODELS/tpl.jinja}"
 TEMPLATE="$SCRIPT_DIR/qwen38-froggeric-v22.5.jinja"
 LOG_DIR="${FRANK_QWEN38_LOG_DIR:-/home/dank/logs}"
 LOG="$LOG_DIR/qwen38_bf16_${MODE}.log"
 KLD_REF_DIR="$CAMPAIGN/kld-ref"
 CALIB_DIR="${FRANK_QWEN38_CALIB_DIR:-$CAMPAIGN/calib}"
 C=qwen38-bf16
 SERVICE=frank-archon
 # The card split, taken from the file that owns it instead of a private pair:
 # --host-pool-mib also funds the KV cache (see the header), so hardcoding it
 # here once starved the recorder. The `:?` guards keep an empty env fatal.
 HOST_POOL_MIB="${FRANK_ARCHON_HOST_POOL_MIB:?FRANK_ARCHON_HOST_POOL_MIB must be set}"
 GPU_HEADROOM_MIB="${FRANK_ARCHON_GPU_HEADROOM_MIB:?FRANK_ARCHON_GPU_HEADROOM_MIB must be set}"
 # The slot count, from the same file that owns the card split: production runs
 # four slots and every mode measures at production's geometry. A per-mode
 # literal here is what let calib ask for eight slots and then starve its own
 # cache. The `:?` guard keeps an empty env fatal, as for the card split.
 MAX_SEQS="${FRANK_ARCHON_MAX_SEQS:?FRANK_ARCHON_MAX_SEQS must be set}"
 if ! [[ "$MAX_SEQS" =~ ^[0-9]+$ ]]; then
    echo "REFUSED: FRANK_ARCHON_MAX_SEQS must be a slot count (got: $MAX_SEQS)" >&2
    exit 2
 fi
 # ONE step size for every mode, quoted from the file that owns it — the audit
 # law for a radiance rig is to inherit the proven number, not to carry a
 # campaign-private value: production's FRANK_ARCHON_MAX_BATCHED_TOKENS. The
 # 1,024 default this script carried was PR #80's answer to a starved cache;
 # production moved to its widest legal step (2,048) on 2026-10-29 and the rig
 # follows. The pool raise that shipped with it (66,560 a rank) bought this
 # container 1.26 MiB of cache (`vram_kv 269.99 -> 271.25 MiB`, window still
 # 7,436 tokens — the 95.37 GiB n-gram table's claim decides the rig's cache,
 # not the pool) and was folded back the same day. The corpora stay cut to the
 # window the boot ledger reports, never the other way round.
 MNBT="${FRANK_ARCHON_MAX_BATCHED_TOKENS:?FRANK_ARCHON_MAX_BATCHED_TOKENS must be set}"
 if ! [[ "$MNBT" =~ ^[0-9]+$ ]]; then
    echo "REFUSED: FRANK_ARCHON_MAX_BATCHED_TOKENS must be a token count (got: $MNBT)" >&2
    exit 2
 fi
 # Calib's shape is NOT a measuring shape. kldref and score produce KL numbers that are only
 # comparable across boots at an identical shape, which is why they inherit production's step and
 # slots to the letter. The tap's output is layer Hessians, never a number any report compares, so
 # its step and slot count are its own — and they are the tap's dominant cost: a 7,168-token
 # document cut into four 2,048-token steps re-streams each layer's routed expert union four
 # times from the container, so one wide step is worth up to ~4x the read volume. Measured
 # 2026-10-29 at the inherited 2,048/4: prefill 71 tok/s and a 47-hour tap; the probe exists to
 # find out what the wide step is worth before the campaign pays for two days of drained serving.
 # Both default to the inherited value, so nothing moves unless a probe asks it to.
 CALIB_MNBT="${Q38_CALIB_MNBT:-$MNBT}"
 CALIB_SEQS="${Q38_CALIB_SEQS:-$MAX_SEQS}"
 for pair in "Q38_CALIB_MNBT=$CALIB_MNBT" "Q38_CALIB_SEQS=$CALIB_SEQS"; do
    if ! [[ "${pair#*=}" =~ ^[0-9]+$ ]]; then
        echo "REFUSED: ${pair%%=*} must be a number (got: ${pair#*=})" >&2
        exit 2
    fi
 done
 # The window every mode boots at. Over-length is a REFUSAL, not a truncation (the
 # env file says so for production), so this is the number the corpus must fit in.
 MAX_MODEL_LEN="${FRANK_QWEN38_MAX_MODEL_LEN:-32768}"
 if ! [[ "$MAX_MODEL_LEN" =~ ^[0-9]+$ ]]; then
    echo "REFUSED: FRANK_QWEN38_MAX_MODEL_LEN must be a token count (got: $MAX_MODEL_LEN)" >&2
    exit 2
 fi
 # Where the n-gram table lives. On the bf16 payload it is 95.37 GiB and `ram`
 # PINS it: the calibration boot died at "startup stopped: out of memory" with
 # 230.22 GiB pinned of 246.07 GiB and 8.06 GiB available (measured 2026-10-29),
 # while the reference recording and every score boot had already succeeded with
 # the same pool and slots. The difference that matters is the job: calib only
 # needs the routed weights prefilled, so it reads the table from the container's
 # own file tier and the pin never happens; kldref and score keep production's
 # placement, because their numbers are only comparable when every other term of
 # the shape is identical. FRANK_QWEN38_NGRAM_PLACEMENT overrides either default.
 case "$MODE" in
    calib) NGRAM_PLACEMENT="${FRANK_QWEN38_NGRAM_PLACEMENT:-disk}" ;;
    *)     NGRAM_PLACEMENT="${FRANK_QWEN38_NGRAM_PLACEMENT:-ram}" ;;
 esac
 case "$NGRAM_PLACEMENT" in
    auto|vram|ram|disk) ;;
    *) echo "REFUSED: FRANK_QWEN38_NGRAM_PLACEMENT must be auto|vram|ram|disk (got: $NGRAM_PLACEMENT)" >&2
       exit 2 ;;
 esac
 # Every container this campaign boots is read through the ro /models mount, so
 # the host path must live under it; the container-side spelling is what goes
 # into the argv. Same rule for the container a score pass opens.
 under_models() {
    case "$1" in
        "$MODELS"/*) printf '/models/%s' "${1#"$MODELS"/}" ;;
        *) return 1 ;;
    esac
 }
 # same_dir A B — the two name one directory, judged only when BOTH exist: a dry
 # run has no fixtures on disk, and two absent paths are not the same place.
 same_dir() {
    [ -d "$1" ] && [ -d "$2" ] && [ "$(cd "$1" && pwd)" = "$(cd "$2" && pwd)" ]
 }
 if [ "$MODE" = score ]; then
    SCORE_RAD="${POS[0]}"
    SCORE_OUT="${POS[1]}"
    OUT_DIR="$(dirname "$SCORE_OUT")"
    MODEL_IN="$(under_models "$SCORE_RAD")" || {
        echo "REFUSED: container to score $SCORE_RAD is not under the mounted models dir $MODELS — the engine cannot read it" >&2
        exit 2
    }
 else
    MODEL_IN="$(under_models "$RAD")" || {
        echo "REFUSED: reference container $RAD is not under the mounted models dir $MODELS — the engine cannot read it" >&2
        exit 2
    }
 fi
 # The measuring shape. Identical for kldref and score by construction — this
 # array is the only place the campaign's KL numbers are defined.
 ARGS=(--tp 2 --placement expert_tiered --weights-disk-tier
      --host-pool-mib "$HOST_POOL_MIB" --gpu-headroom-mib "$GPU_HEADROOM_MIB"
      --deterministic --num-speculative-tokens 0
      --kv-cache-dtype fp8 --max-model-len "$MAX_MODEL_LEN"
      --ngram-placement "$NGRAM_PLACEMENT"
      --override-chat-template /models/tpl.jinja)
 MOUNTS=(-v "$MODELS:/models:ro")
 TAPENV=()
 case "$MODE" in
  kldref)
    # Recording is prefill-only over four slots at the campaign's single step
    # size, which is frank-archon.env's FRANK_ARCHON_MAX_BATCHED_TOKENS (2,048) —
    # see the header: the file tier's ceiling and the slot count both come from
    # the file that owns them. Corpus read-only, record dir rw.
    ARGS+=(--model "$MODEL_IN"
           --kld-record /data/kld-ref --kld-corpus /data/kld.jsonl
           --max-num-seqs "$MAX_SEQS" --max-num-batched-tokens "$MNBT")
    MOUNTS+=(-v "$KLD_CORPUS:/data/kld.jsonl:ro"
             -v "$KLD_REF_DIR:/data/kld-ref")
    ;;
  score)
    # The same argv with three words changed: which container, and that this
    # time the engine reads a recording and writes a report instead of the
    # reverse. The reference dir mounts READ-ONLY — a scoring pass must not be
    # able to touch the zero point — and the report gets its own writable bind.
    if [ "$OUT_DIR" = "$KLD_REF_DIR" ] || same_dir "$OUT_DIR" "$KLD_REF_DIR" \
       || [ "$(realpath -m "$OUT_DIR")" = "$(realpath -m "$KLD_REF_DIR")" ]; then
        echo "REFUSED: score output $SCORE_OUT is inside the reference dir $KLD_REF_DIR, which this mode mounts read-only" >&2
        exit 2
    fi
    ARGS+=(--model "$MODEL_IN"
           --kld-ref /data/kld-ref --kld-out "/data/out/$(basename "$SCORE_OUT")"
           --max-num-seqs "$MAX_SEQS" --max-num-batched-tokens "$MNBT")
    MOUNTS+=(-v "$KLD_REF_DIR:/data/kld-ref:ro"
             -v "$OUT_DIR:/data/out")
    ;;
  calib)
    # Calibration rides the shared step like every other mode. The old
    # exemption — eight slots at MNBT 65,536 for "free throughput", because
    # "a disk-tier step costs the same at 1 token as at 64k" — is false: the
    # step's activation arena is billed on the card, not on the disk. The
    # engine's own ledger for this container: at MNBT 1,024 the arena is
    # 504 MiB and the card claims 30.87 GiB (weights 28.90 GiB of it); at
    # 65,536 the arena is 10.65 GiB (fixed region 1.25 GiB), the claim falls
    # to 16.85 GiB, the experts shrink to 10.98 GiB and the KV to 271 MiB —
    # one slot's worth of cache for eight slots, and the boot dies at
    # "startup stopped: out of memory" (measured 2026-10-29). A wide step
    # buys throughput with the expert slab, so price it like any other
    # VRAM knob. The slots follow frank-archon.env's FRANK_ARCHON_MAX_SEQS
    # through MAX_SEQS, exactly as kldref and score do.
    [ "$IMAGE_EXPLICIT" = 1 ] || IMAGE="$CALIB_IMAGE"
    # ONE TAP SEAL WRITES ~40 GiB. Measured 2026-10-30: a 8,192-token floor wrote
    # 39.4 GiB into the tap dir within seconds; pointed at a 37 GiB root it filled
    # the filesystem to zero, after which every write came back short ("calibration
    # cannot write", "write was short") and the system itself began refusing
    # writes. The room is demanded up front, on the tap's own filesystem.
    if [ "$DRY" != 1 ]; then
        avail_mib=$(df --output=avail -B 1M "$CALIB_DIR" 2>/dev/null | tail -1 | tr -d ' ')
        [[ "$avail_mib" =~ ^[0-9]+$ ]] || avail_mib=0
        need_mib="${Q38_CALIB_MIN_FREE_MIB:-61440}"
        if [ "$avail_mib" -lt "$need_mib" ]; then
            echo "REFUSED: tap dir $CALIB_DIR has ${avail_mib} MiB free; one seal writes ~40 GiB and this run needs ${need_mib} MiB (Q38_CALIB_MIN_FREE_MIB overrides)" >&2
            exit 2
        fi
    fi
    ARGS+=(--model "$MODEL_IN" --port 7770
           --max-num-seqs "$CALIB_SEQS" --max-num-batched-tokens "$CALIB_MNBT")
    # The tap is only real if this directory exists and the engine can write it:
    # without the check a typo'd CALIB_DIR auto-creates a root-owned empty dir in
    # whatever path was given, and the run burns hours serving prompts into a tap
    # nobody reads. RADIANCE_CALIB_TOKENS is the whole corpus' floor, not a round's.
    [ "$DRY" = 1 ] || [ -w "$CALIB_DIR" ] || {
        echo "REFUSED: calib tap dir $CALIB_DIR is not writable (mkdir it, chown it, then re-run)" >&2
        exit 2
    }
    MOUNTS+=(-v "$CALIB_DIR:/data/calib")
    TAPENV=(-e RADIANCE_CALIB_DIR=/data/calib
            -e "RADIANCE_CALIB_TOKENS=${FRANK_QWEN38_CALIB_TOKENS:-12000000}")
    # The tap's own authorization. A pusher must see this marker before it sends
    # a prompt (calib-push.py --tap-marker), it names the container so a push
    # aimed anywhere else refuses, and a boot that fails removes it again below.
    # Measured 2026-10-29: with no marker, the chain pushed 4 rounds x 10M tokens
    # into RESTORED PRODUCTION after the rig's boot died, and nothing recorded.
    if [ "$DRY" = 1 ]; then
        :
    else
        printf 'container=%s\nimage=%s\nmode=calib\nstarted=%s\npid=%s\n' \
            "$C" "$IMAGE" "$(date -Is)" "$$" > "$CALIB_DIR/.rig-boot"
    fi
    ;;
 esac
 CMD=(podman run --rm --name "$C"
     --network host --init
     --security-opt seccomp=unconfined
     --ulimit memlock=-1:-1
     --device /dev/kfd --device /dev/dri
     "${MOUNTS[@]}" "${TAPENV[@]}"
     "$IMAGE" "${ARGS[@]}")
 # ---- gate: the window must hold the corpus (kldref alone can meet it) --------
 # score reads a recording instead of a corpus, and every arm declares at the
 # checkpoint's training context, so neither can meet a sequence too long to boot.
 # Measured on the raw JSON line — the pessimistic direction, and the same figure
 # the census in the header quotes. It runs on a dry run too, because a refusal
 # this cheap should never have to wait for the drained window to be discovered.
 if [ "$MODE" = kldref ] && [ -f "$KLD_CORPUS" ]; then
    read -r REC_NO REC_CHARS < <(awk '{ if (length($0) > best) { best = length($0); where = NR } } END { print where + 0, best + 0 }' "$KLD_CORPUS")
    LIMIT_CHARS=$(( MAX_MODEL_LEN * 3 ))
    if [ "$REC_CHARS" -gt "$LIMIT_CHARS" ]; then
        echo "REFUSED: record $REC_NO of $KLD_CORPUS is $REC_CHARS chars, past the $LIMIT_CHARS a $MAX_MODEL_LEN-token window promises at the pessimistic chars/3 — raise FRANK_QWEN38_MAX_MODEL_LEN, and the container's declared --max-ctx with it, before booting the reference" >&2
        exit 2
    fi
    echo "corpus fits: longest record $REC_CHARS chars (number $REC_NO) inside the $MAX_MODEL_LEN-token window (limit $LIMIT_CHARS chars)"
 fi
 if [ "$DRY" = 1 ]; then
    echo "dry run: $MODE — the production unit is left running, nothing is mounted, nothing is written"
    printf '%q\n' "${CMD[@]}"
    exit 0
 fi
 # ---- gates that only a real run needs ---------------------------------------
 if [ "$MODE" = score ]; then
    if [ ! -d "$KLD_REF_DIR" ]; then
        echo "REFUSED: no KLD reference at $KLD_REF_DIR — record it first in a drained window: scripts/run-qwen38-bf16.sh kldref" >&2
        exit 2
    fi
    if [ ! -f "$SCORE_RAD" ]; then
        echo "REFUSED: no container to score at $SCORE_RAD" >&2
        exit 2
    fi
    if [ -e "$SCORE_OUT" ]; then
        echo "REFUSED: $SCORE_OUT already exists — one report per name; move the old one to trash before re-scoring" >&2
        exit 2
    fi
    mkdir -p "$OUT_DIR"
 else
    if [ ! -f "$RAD" ]; then
        echo "REFUSED: no reference container at $RAD — build it first: scripts/build-qwen38-bf16-rad.sh" >&2
        exit 2
    fi
    [ "$MODE" = kldref ] && [ ! -f "$KLD_CORPUS" ] && {
        echo "REFUSED: no KLD corpus at $KLD_CORPUS — the record would be empty" >&2
        exit 2
    }
 fi
 if [ ! -f "$TEMPLATE" ]; then
    echo "REFUSED: no chat template at $TEMPLATE" >&2
    exit 2
 fi
 mkdir -p "$LOG_DIR"
 # The prod template, staged where the ro /models mount reaches it: a reference
 # recorded with the container's baked template scores a different tokenisation
 # than the arms served with froggeric's, and the KL would be measuring that.
 cp -f "$TEMPLATE" "$TPL" || {
    echo "REFUSED: cannot stage the chat template at $TPL (the container reads it through /models)" >&2
    exit 2
 }
 [ "$MODE" = kldref ] && mkdir -p "$KLD_REF_DIR"
 [ "$MODE" = calib ]  && mkdir -p "$CALIB_DIR"
 # Stop the campaign container BEFORE handing the cards back: a killed run leaves
 # its conmon alive (--rm only fires when podman itself exits), and production
 # then boots beside a container still holding the GPUs — measured 2026-10-29,
 # where the service reported `active` while its engine starved on the rig's VRAM.
 restore() {
    podman rm -f "$C" >/dev/null 2>&1 || true
    # Removing the container does NOT kill its engine. Measured 2026-10-30: a rig
    # radiance kept /dev/kfd after `podman rm -f`, production's
    # ar_oneshot_2rank_wht6 init then failed `device error`, the engine exited 0,
    # and Restart=always looped it 100 times with /health never answering. The
    # service is stopped for the whole run, so any radiance still holding the
    # device here is an orphan — kill it BY PID (pkill -f matches the caller's
    # own command line, which is how a session gets killed instead of the orphan).
    for p in $(fuser /dev/kfd 2>/dev/null); do
        if [ "$(cat "/proc/$p/comm" 2>/dev/null)" = radiance ]; then
            echo "restore: killing orphaned radiance pid $p still holding /dev/kfd" >&2
            kill -9 "$p" 2>/dev/null || true
        fi
    done
    sudo systemctl start "$SERVICE" >/dev/null 2>&1 || true
 }
 trap restore EXIT
 echo "run-qwen38-bf16 $MODE: stopping $SERVICE ($(date -Is)); log: $LOG"
 sudo systemctl stop "$SERVICE" >/dev/null 2>&1 || true
 podman rm -f "$C" >/dev/null 2>&1 || true
 "${CMD[@]}" 2>&1 | tee "$LOG"
 rc=${PIPESTATUS[0]}
 # A boot that failed must not leave the tap marked as rig-authorized: the marker
 # is what calib-push.py trusts, and the point of it is that a dead rig can never
 # be pushed into again.
 if [ "$MODE" = calib ] && [ "$rc" -ne 0 ] && [ -f "$CALIB_DIR/.rig-boot" ]; then
    mv -f "$CALIB_DIR/.rig-boot" "$CALIB_DIR/.rig-boot.failed-$(date +%Y%m%d-%H%M%S)" 2>/dev/null || true
    echo "the rig boot failed (rc=$rc): the tap marker is retired, so no pusher can take this tap for a live rig"
 fi
 echo "run-qwen38-bf16 $MODE exited rc=$rc; $SERVICE restored; log: $LOG"
 exit "$rc"
@@ -1,5 +1,5 @@
 #!/usr/bin/env bash
-# serve-frank-dragoon.sh — frank-dragoon endpoint: halogen-flash-server 0.16.2
+# serve-frank-dragoon.sh — frank-dragoon endpoint: halogen-flash-server 0.18.20
 # (closed-source engine, image ghcr.io/peonist-ai/halogen-flash-server,
 # digest-pinned in frank-dragoon.env) serving Quat3rnion's NATIVE v2
 # conversion (2026-10-11) of the abliterated Qwen3.8-Flash-Next on Dankclaw's
@@ -136,10 +136,17 @@ fi
 #      The engine re-checks size+sha256 against that catalog at boot; it cannot
 #      fetch anything here because /models is read-only and HALOGEN_DOWNLOAD
 #      stays unset — no outbound connection at boot),
-#   2. the distro XRT's three libraries, each mounted twice (where the image
+#   2. the distro XRT's three libraries, each mounted THREE times: where the
-#      looks for XRT, and at its own path — the libraries find each other there;
+#      image looks for XRT (/opt/xilinx/xrt/lib), at their own path (the
-#      an /opt/xilinx/xrt of symlinks into the system dir dangles inside the
+#      libraries find each other there), and at /opt/xilinx/lib/<multiarch>
-#      container, which the vendor calls out explicitly),
+#      — the NPU child searches that third prefix too, and since the host's
 #      libxrt2 was reconfigured 2026-10-20 it is the ONLY prefix the child
 #      names: omitting it crashes the lane with `No such library
 #      '/opt/xilinx/lib/x86_64-linux-gnu/libxrt_core.so.2'` and the engine
 #      supervisor takes the dead lane down on the whole container (measured
 #      2026-10-21: 88-restart boot loop). Real files only — an /opt/xilinx/xrt
 #      of symlinks into the system dir dangles inside the container, which the
 #      vendor calls out explicitly),
 #   3. the GPU fabric clock HELD: on Strix Halo the iGPU and the NPU co-running
 #      hang the machine while FCLK changes speed, and the engine refuses to
 #      start when it moves — it cannot read host sysfs from inside the
@@ -190,7 +197,9 @@ if [ -n "$NPU_MODELS" ]; then
            exit 1
        fi
        REAL="$(readlink -f "$XRT_DIR/$L")"
-        NPU_ARGS+=(-v "$REAL:/opt/xilinx/xrt/lib/$L:ro" -v "$REAL:$XRT_DIR/$L:ro")
+        MULTIARCH="${XRT_DIR##*/}"
        NPU_ARGS+=(-v "$REAL:/opt/xilinx/xrt/lib/$L:ro" -v "$REAL:$XRT_DIR/$L:ro"
                   -v "$REAL:/opt/xilinx/lib/$MULTIARCH/$L:ro")
    done
    HELD=0
    for D in "$SYSFS"/class/drm/card*/device; do
@@ -1,7 +1,7 @@
 #!/usr/bin/env bash
 # Serve prism-ml/Ternary-Bonsai-2-9B-gguf (PQ2_0 ternary quant WITH the MTP head
 # in the file) as frank-zealot on Dankclaw's RTX 3090 via llama.cpp (PrismML
-# vendor build 10777, CUDA sm_86).
+# vendor build 12118 (0.7.0-dev, commit 024474448), CUDA sm_86).
 #
 # 2026-10-18 (Frank): Bonsai-2 27B PQ2_0 → the 9B MTP export, then the same
 # evening TWO 262,144-token conversations (--ctx-size 524288 --parallel 2) over
@@ -78,7 +78,7 @@ if [[ -f "$ENV_FILE" ]]; then
 fi
 # --- defaults (used when the env file does not set them) -------------------
-: "${LLAMA_BIN:=/home/dank/GitPackages/llama-prism-b10777/llama-server}"
+: "${LLAMA_BIN:=/home/dank/GitPackages/llama-prism-0244744/bin/llama-server}"
 : "${MODEL_FILE:=/home/dank/models/Bonsai2-9B-MTP/Ternary-Bonsai-2-9B-PQ2_0-MTP-Q8_0.gguf}"
 : "${ALIAS:=frank-zealot}"
 : "${BIND_ADDR:=0.0.0.0}"
@@ -119,7 +119,7 @@ fi
 # above are what pay for it. The draft cache dtype IS a further knob
 # (--spec-draft-type-k / --spec-draft-type-v, both default f16 — verified in this
 # build's --help) and stays at its default here. Card gate: quantized V alongside
-# MTP needs fork build >= prism-b10776, which is why LLAMA_BIN is the 10777
+# MTP needs fork build >= prism-b10776; LLAMA_BIN is 12118 (b10777 kept as rollback)
 # prebuilt; the type-142 loader lock keeps it pinned regardless.
 # SPEC_TYPE=none is the rollback and strips the whole set (flags move together).
 : "${SPEC_TYPE:=draft-mtp}"
@@ -337,8 +337,12 @@ def main():
    if any(meta[shard_ids[i]][4][1] != hd for i in range(n_shards)):
        log("ERROR: PLE shards disagree on HD")
        return 2
-    if hd % 128 != 0 and hd != 160:
+    if hd != 160 and hd % 128 != 0:
-        log(f"WARNING: PLE HD={hd} is neither 160 nor a multiple of 128")
+        # The pack below writes lanes of 128 codes; any other HD would produce a
        # table that decodes to the wrong rows. The old warning let it continue.
        log(f"ERROR: PLE HD={hd} is neither 160 nor a multiple of 128, and the "
            "128-lane pack cannot express it")
        return 2
    # ---- idempotence gate -------------------------------------------------
    stacked_names = [f"{p}.mlp.experts.gate_up_proj" for p in prefixes]
@@ -521,12 +525,16 @@ def verify(stage, prefixes, n_e, shapes, meta, shard_ids, rows, hd,
        }
        for name, (dt, shp) in want.items():
            e = hdr[name]
-            assert e["dtype"] == dt and list(e["shape"]) == shp, name
+            if e["dtype"] != dt or list(e["shape"]) != shp:
-            assert e["data_offsets"][1] - e["data_offsets"][0] == \
+                raise RuntimeError(f"{name}: header is {e['dtype']}{e['shape']}, expected {dt}{shp}")
-                int(np.prod(shp)) * (1 if dt == F8 else 2), name
+            if e["data_offsets"][1] - e["data_offsets"][0] != int(np.prod(shp)) * (1 if dt == F8 else 2):
                raise RuntimeError(f"{name}: data span is {e['data_offsets']}, shape {shp} needs "
                                   f"{int(np.prod(shp)) * (1 if dt == F8 else 2)} B")
    _, phdr = parse_header(out_path)
    e = phdr[next(iter(phdr))]
-    assert e["dtype"] == BF16 and list(e["shape"]) == [sum(rows), hd]
+    if e["dtype"] != BF16 or list(e["shape"]) != [sum(rows), hd]:
        raise RuntimeError(f"PLE table header is {e['dtype']}{e['shape']}, "
                           f"expected {BF16}[{sum(rows)}, {hd}]")
    # sampled PLE rows: recompute and compare bytes exactly
    lut = e4m3_lut()
@@ -546,7 +554,8 @@ def verify(stage, prefixes, n_e, shapes, meta, shard_ids, rows, hd,
                with open(out_path, "rb") as ofh:
                    ofh.seek(out_off + got_row * hd * 2)
                    got = ofh.read(hd * 2)
-                assert got == want, f"PLE row {got_row} mismatch"
+                if got != want:
                    raise RuntimeError(f"PLE row {got_row} does not match the decoded source")
            mm.close()
    log("verify: new-file headers, sizes and sampled PLE rows all exact; "
        "expert slabs were byte-checked in-stack")
@@ -94,6 +94,16 @@ for path in "${CHANGED[@]}"; do
        dankserver/.env.example|dankclaw/.env.example)
            add_test tests/env_consistency.bats
            ;;
        # The SSD-backed metadata view: the installer, the host mount unit, and the
        # two media units that bind the view rather than the raw array.
        scripts/media-view.sh|dankserver/host/*)
            add_test tests/media_overlay.bats
            add_test tests/dankserver_quadlet.bats
            add_test tests/deployment_scoping.bats
            ;;
        dankserver/jellyfin/quadlet/*|dankserver/plex/quadlet/*)
            add_test tests/media_overlay.bats
            ;;
        shared/homepage/*)
            add_test tests/homepage_caddy.bats
            add_test tests/homepage_config.bats
@@ -265,6 +275,13 @@ for path in "${CHANGED[@]}"; do
            add_test tests/voice_pack_unit.bats
            add_test tests/scripts_syntax.bats
            ;;
        # The KLD corpus re-chunker: its seam is its own (the campaign's
        # one-token-series invariant is asserted there), so a change to it does
        # not land on the generic scripts/* fallback below.
        scripts/rechunk-kld-corpus.py)
            add_test tests/qwen38_rechunk.bats
            add_test tests/scripts_syntax.bats
            ;;
        scripts/*)
            add_test tests/scripts_syntax.bats
            ;;
@@ -293,6 +310,14 @@ for path in "${CHANGED[@]}"; do
            add_test tests/caddyfile.bats
            add_test tests/deployment_scoping.bats
            ;;
        # The GPU exporter is a host process; its seam pins the collector, the
        # user unit, the deployer function, the planner route, the Prometheus
        # scrape jobs and the host-system GPU panels — a cross-file contract, so
        # the whole seam rides on any of its own files.
        scripts/gpu-exporter.py|scripts/gpu-exporter.service)
            add_test tests/gpu_exporter.bats
            add_test tests/deployment_scoping.bats
            ;;
        dankclaw/udev/*)
            # Host-level udev rules are applied by manual sudo install + udevadm
            # trigger, not dispatched as a service; nothing here is bats-testable.
@@ -52,28 +52,31 @@ Where the contract lives:
 The arithmetic (the coupled contract every endpoint's maxTokens comment
 encodes). The authoritative `contextWindow`/`maxTokens` pair per model is on the
 endpoint page ([[frank-archon]], [[frank-dragoon]], [[frank-zealot]]); what
-follows is a **worked example** recomputed from the live files on 2026-10-18 —
+follows is a **worked example** recomputed from the live files on 2026-10-21 —
-a snapshot, not fleet state — with windows 200,000 / 155,648 / 200,704 and
+a snapshot, not fleet state — with windows 200,000 / 200,000 / 200,704 and
 maxTokens 16,384 / 45,056 / 16,384 (archon / dragoon / zealot):
 1. **Source trigger** = `thresholdPercent × source contextWindow`; at the live
-   95 % that is 190,000 (archon), 147,866 (dragoon), 190,669 (zealot). The
+   95 % that is 190,000 (archon and dragoon), 190,669 (zealot). The
   bundle clamps the percentage to 1–99 and converts it to a token threshold.
 2. **Output budget** = capped at the target model's `maxTokens` — every role
   here compacts to frank-zealot, so the budget is its 16,384.
-3. **REQUIREMENT**: prompt (≤ trigger) + budget ≤ the target's per-slot
+3. **REQUIREMENT**: prompt (≤ trigger) + budget ≤ the fitting window. For a
-   window. **At 95 % this no longer holds**: 190,000 + 16,384 = 206,384 >
+   dragoon session the fitting window is the ENGINE ceiling, not a slot: the
-   200,704. The registry's own comments design around an 85 % trigger (archon:
+   190,000 trigger is 62,144 under the 262,144 ceiling, so even ×1.03 est
-   170,000 est × 1.03 + 16,384 ≈ 191,500 ≤ 200,704, `models.yml:49-56`), which
+   error plus the 45,056 request budget lands inside — that headroom is the
-   does hold. While `config.yml` and those comments disagree, treat 85 % as the
+   reason the ceiling moved to native on 2026-10-21. For a session compacting
-   design point and re-run this arithmetic after ANY window change — a 400 from
+   INTO frank-zealot the window is its 200,704-token slot, and **at 95 % this
-   the compaction target mid-conversation is the symptom.
+   does not hold for archon/zealot sources**: 190,000 + 16,384 = 206,384 >
   200,704. Treat 85 % as the design point until config.yml and the registry
   comments agree, and re-run this arithmetic after ANY window change — a 400
   from the compaction target mid-conversation is the symptom.
   (Historic 2026-10-05 pairs, at `thresholdPercent: 70`: 140,492 + 36,864 =
   177,356 ≤ 200,704 archon; 140,490 + 57,344 ≤ 200,700 dragoon; 140,492 +
   32,768 ≤ 200,704 zealot.)
 Procedure (recompute from the LIVE files every time — windows rotate with
-every swap; the block above is the 2026-10-18 state):
+every swap; the block above is the 2026-10-21 state):
 1. Read the `config.yml` compaction block + `models.yml` entries for source
   and target. Every endpoint's `contextWindow` comment carries its own
@@ -94,9 +97,10 @@ Known reconciliation trap: comments in models.yml have described compaction
 as running "on the dankserver endpoint" while `remoteEndpoint` pointed at
 dankclaw — the comment/config disagreed. The comments now name the real
 target (`compaction.remoteEndpoint` = dankclaw :7771). **The config is
-truth; fix comments in the same change.** A fresh instance of the same trap is
+truth; fix comments in the same change.** The 85 %-comments vs 95 %-config
-open 2026-10-18: every `maxTokens` comment still says "compaction fires at
+disagreement is not academic: it is why long dragoon sessions kept drawing
-85 % of the window" while `config.yml:84` sets `thresholdPercent: 95`.
+400s at the old 200,704 ceiling — the trigger fires late AND the est
 under-counts, and a window parked near the ceiling has no room for either.
 `maxTokens` never changes for a reslice — it caps output; `contextWindow` caps
 input. Boot-shape OOM laddering is a different problem
 ([[Llama Cpp Serving Sizing]]).
@@ -130,10 +130,19 @@ Quickest unblock: `podman compose up -d --force-recreate <svc>`. Durable fix
 ## 4. Podman 5.x quadlet v1 generator quirks (verified live 2026-08-28)
- **Hard-fail keys** — the whole unit silently fails to generate
+- **Hard-fail keys abort the WHOLE BATCH, not one unit** (podman 5.7.0,
-  (`systemctl --user cat <unit>` shows "could not be found"): any
+  live 2026-10-21): one unsupported key — e.g. `HostAddr=` — makes the
-  dash-prefixed `[Container]` option (`--shm-size=…` → `unsupported key …
+  generator log `unsupported key … in group 'Container'` plus
-  in group 'Container'`); `Command=` is unsupported, use `Exec=`.
+  `processing encountered some errors` and generate NOTHING for that
  manager. The running containers keep going as orphans, so the failure
  first appears on the next deploy as `Unit <x>.service not found` for a
  service that is visibly up. Other offenders: any dash-prefixed
  `[Container]` option (`--shm-size=…`), `Command=` (use `Exec=`).
  The seam `tests/quadlet_image_pins.bats` fails the build on any
  generator-rejected key in any host's units.
  Also: an image whose ENTRYPOINT is the binary takes FLAGS ONLY in
  `Exec=` — repeating the binary makes it a positional arg
  (`unexpected /bin/blackbox_exporter`, live 2026-10-21).
 - **HealthCmd space-splitting:** the v1 generator splits `HealthCmd` on
  spaces into the healthcheck argv; a quoted probe
  `/bin/sh -c "pg_isready …"` lands as argv token `"pg_isready` →
@@ -148,6 +157,16 @@ Quickest unblock: `podman compose up -d --force-recreate <svc>`. Durable fix
  "unsupported key"`, `systemctl --user cat probe.service` (shows the
  generated ExecStart verbatim), then remove + reload.
 ## 4a. Image pins (fleet law, 2026-10-21)
 Every registry image in a quadlet is digest-pinned (`tag@sha256:…`);
 locally built images (`localhost/*`, `franksdank/*`) are exempt — their
 build is the pin. A rolling tag lets a rebuild move under a running
 service with nobody approving it, and it makes a CVE audit ungradeable.
 Gate: `tests/quadlet_image_pins.bats`. Where upstream publishes no version
 tags (copyparty-ac stops at 1.19.15), pin `:latest@sha256:` and name the
 version the config label carries in the comment beside it.
 ## 5. Rootless Quadlet log retrieval
 Step 0 — establish ownership: `systemctl --user status <name>.service`
@@ -176,8 +195,9 @@ deploy.sh, never a live env poke.
   check every namespace (`sudo podman ps -a`) and native units
   (`systemctl list-units`) before declaring a service absent. Compare
   compose mount sources against the real filesystem — compose files name
-   phantom paths (e.g. `/mnt/media/*` vs the real LUKS array at
+   phantom paths (the legacy `dankserver/compose.yaml` still names `/opt/jellyfin/*`
-   `/mnt/entropy`). Fix compose through git first, deploy second.
+   for a service whose roots moved to `/home/dank/Containers/jellyfin`). Fix
   compose through git first, deploy second.
 2. **Pre-create bind-mount sources** (`sudo mkdir -p` + `chown 1000:1000`)
   or creation fails `statfs …: no such file or directory`.
 3. **Staged bring-up:** infra first (`up -d mariadb redis`), poll
@@ -116,6 +116,20 @@ as working when they could never report up. **Clean cutover includes prose** —
 docs, unit docstrings, and the repo's rule file (the most expensive place to
 be stale).
 ## Three rules that outlive the incident that taught them
 * **A target can read down beside a healthy service.** Read `lastError` from
  `/api/v1/targets` before believing an outage — llama.cpp refuses `/metrics` at the
  protocol layer without `--metrics` (501) while `/health` answers 200 for weeks.
 * **Alert state lives where the tool says it does.** Prometheus keeps rule state in
  process memory — a restart re-stamps `activeAt`, so a "fresh" alert after a reboot is
  not evidence of a fresh fault. Alertmanager **does** persist (`--storage.path` on a
  host volume: silences + notification log, [[Alertmanager]]).
 * **Never quote a candidate arm's flags or VRAM as fleet state.** A stopped production
  unit (`failed` after `stop-sigterm timed out. Killing` — the known shutdown shape,
  [[frank-zealot]]) beside an answering port means the metric measured a scratch boot;
  the process shape can change between two censuses ten minutes apart.
 ## Order and verification
 1. Enumerate the surfaces (status board, metrics targets, alert rules, test
@@ -110,9 +110,9 @@ only `models`, `input`, `output`, `custom_nodes` and the healthcheck file
 ## Recommended additions
-Quantized preferred, sized for the 8060S iGPU (GTT-backed — `/system_stats`
+Quantized preferred. The iGPU's GTT aperture is not free RAM beside
-reports 122.4 GB of device VRAM on `cuda:0`, `curl http://127.0.0.1:8188/system_stats`
+frank-dragoon — size every pick against *What actually fits beside
-2026-10-18; the RTX 3090 is frank-zealot's). All repos verified 2026-10-05 and
+frank-dragoon* below (the RTX 3090 is frank-zealot's). All repos verified 2026-10-05 and
 all four still resolve today (`huggingface.co/lightx2v/Qwen-Image-Lightning`
 `huggingface.co/unsloth/Qwen-Image-Edit-2511-GGUF`
 `huggingface.co/Comfy-Org/Qwen-Image-DiffSynth-ControlNets`
@@ -133,8 +133,34 @@ sha256 checks) or a plain `hf download` with the token in the environment —
 never on the command line.
 Skip list: the original `city96/Qwen-Image-gguf` conversions (superseded by
-the 2.1 files already on disk), and any BF16 of #2/#4 (the iGPU does not need
+the 2.1 files already on disk), and any BF16 of #2/#4 — they double the disk
-them at GTT capacity, and they double the disk bill).
+bill, and see *What actually fits beside frank-dragoon* for why raw GTT size
 is not the real budget.
 ## What actually fits beside frank-dragoon
 The 122 GB `/system_stats` figure is the GTT **aperture**, not free memory: on
 an APU the aperture and `MemAvailable` are the same physical RAM, and
 frank-dragoon pins tens of GiB of it. Measured 2026-10-21 with frank-dragoon
 live: `MemAvailable` 15.3 GiB, iGPU busy 1.75 GiB — ComfyUI itself idles at
 ~10 MB, so the budget for a generation run is roughly what `free` says.
 Weights must land inside that budget with slack for compute buffers and
 activations (the mmproj-class lesson: price a model at ~1.5× its file set):
 | Pipeline (weights) | Fits beside frank-dragoon? |
 |---|---|
 | Qwen-Image 2.1 int8_convrot + w4a8 + bf16 VAE (~14.2 GB) — the two proven workflows | **No** — thrash-or-fail; do not queue it while the endpoint serves |
 | GGUF Q8_0 + w4a8 + small VAE (~13.3 GB) | No, same reason |
 | Any BF16 combination | No |
 | SDXL-Turbo / SD 1.5-class (~2–7 GB) | Yes, with real headroom |
 **The law: one process owns the memory — same rule as the 3090 bench law.**
 For a Qwen-Image render: stop and mask frank-dragoon
 (`sudo systemctl stop frank-dragoon` + `mask` — the mask stops
 `Restart=always` from firing mid-render), generate, unmask, verify
 `curl 192.168.0.143:7770/health`. The proven workflows above then run with
 the whole machine. Do not treat `/system_stats`'s 122 GB as an invitation.
 ## See Also
@@ -129,6 +129,12 @@ bind-mounted to `/usr/local/bin/comfyui-healthcheck.py` and run as `HealthCmd`
 - Do not let pip reinstall the torch family. The Containerfile filters it out on purpose.
 - The base image is about 32 GiB (`podman image inspect localhost/comfyui-rocm:gfx1151` → 33,765,227,471 B ≈ 31.4 GiB, 2026-10-18). A code update means a rebuild, because the code is baked in.
 - The empty-models note from the 2026-09-16 era is retired: `/home/dank/comfyui-data/models` now holds the Qwen-Image 2.1 payload — three `diffusion_models`, three `text_encoders`, two VAEs (~35 GB, `ls -l` 2026-10-18), listed in [[ComfyUI Qwen-Image Workflows]]. New model classes (LoRA, ControlNet) still need their directories created first: `models/lora` and `models/controlnet` do not exist yet.
 - **The 122 GB `/system_stats` VRAM figure is the GTT aperture, not free
  memory.** On the APU it is system RAM, and frank-dragoon pins most of it
  (`MemAvailable` 15.3 GiB with the endpoint live, measured 2026-10-21). A
  Qwen-Image-class render beside the running endpoint is thrash-or-fail; the
  stop-mask-render-verify procedure and the fit table are in
  [[ComfyUI Qwen-Image Workflows]] → What actually fits beside frank-dragoon.
 ## See Also
@@ -34,15 +34,16 @@ https://github.com/open-webui/open-webui
 The picker is built from the `model` table in `webui.db` (one row per model)
 plus the `model_ids` of each **enabled** OpenAI connection, NOT from live
 connection discovery — so the picker shows exactly what is seeded there.
-As of 2026-10-07 the picker exposes **exactly one model: `frank-zealot`** *(unverified
+Since 2026-10-21 the picker exposes **all three fleet models to every
-2026-10-18: `webui.db` is the authority; the fleet itself serves all three ids
+account**: `frank-dragoon`, `frank-archon`, `frank-zealot` (per Frank: open
-`frank-dragoon`/`frank-zealot`/`frank-archon`, per the cards in
+`frank-archon` to everyone; the other two were already public). Proven with
-`shared/homepage/services.json`)*.
+a `role=user` token: `/api/models` lists all three ids and a chat turn to
 `frank-archon` answers end-to-end.
-| Connection (`openai.api_configs` idx) | URL as recorded 2026-10-09 | Enabled | `model_ids` | Endpoint page |
+| Connection (`openai.api_configs` idx) | `openai.api_base_urls` (live 2026-10-21) | Enabled | `model_ids` | Endpoint page |
 |---|---|---|---|---|
-| 0 | `http://host.containers.internal:7770/v1` | no (2026-10-07) | `frank-dragoon` | [[frank-dragoon]] |
+| 0 | `http://host.containers.internal:7770/v1` | yes | `frank-dragoon` | [[frank-dragoon]] |
-| 1 | `http://192.168.0.235:7770/v1` | no (2026-10-07) | `frank-archon` | [[frank-archon]] |
+| 1 | `http://192.168.0.235:7770/v1` | yes | `frank-archon` | [[frank-archon]] |
 | 2 | `http://host.containers.internal:7771/v1` | yes | `frank-zealot` | [[frank-zealot]] |
 The URLs in the first column are the DB's recorded values and predate the 2026-08-22
@@ -50,9 +51,10 @@ bridge move; on this network they are wrong — see the reachability gotcha belo
 is the corrected law. The picker row for `frank-zealot` was re-keyed on 2026-10-18 after
 that day's payload swap (see [[frank-zealot]] for what was deployed and when).
-The `model` table carries exactly two decoration rows, `frank-zealot` and
+The `model` table carries three decoration rows — `frank-zealot`,
-`frank-dragoon`. Both the context length and the description live **inside
+`frank-dragoon`, `frank-archon` (200,000 context, added 2026-10-21).
-JSON columns** — `params` holds `{"context": {"length": …}}` and `meta` holds
+Both the context length and the description live **inside JSON columns** —
 `params` holds `{"context": {"length": …}}` and `meta` holds
 `{"description": …}`; there is no `context_length` or `description` column, so
 `sqlite3` one-liners against those names fail. `params.context.length` is a copy
 of the endpoint's per-slot window, so the authoritative number lives on the
@@ -380,6 +382,20 @@ neither.
  name: /selected model/i })`) and the welcome text "How can I help you
  today?"; that snapshot is also the cheapest proof the picker is populated.
 - **2026-10-21 — picker opened to all three models, for everyone.** Per
  Frank: `frank-archon` accessible to everyone. Three-leg repair on
  `webui.db` (snapshot `webui.db.bak-20261021` first): connection idx 1
  `enable: true`, a `model` row `frank-archon` (params
  `{"context":{"length":200000}}`, meta description from the endpoint card),
  and an `access_grant` row `(resource_type='model', resource_id='frank-archon',
  principal_id='*', 'read')` — `principal_id='*'` is the public grant, not a
  role row. Verified as a plain `role=user` token: `/api/models` lists all
  three ids, chat to `frank-archon` answers. Token-mint gotchas: the backend
  package imports only with cwd `/app/backend`, and `WEBUI_SECRET_KEY` must
  come from **PID 1's environ** (`podman exec` sessions do not inherit the
  app process env; the `.webui_secret_key` file happens to equal it — use it
  explicitly, `podman exec -e`).
 ## See Also
 - [[Dankclaw Stack]]
@@ -1,9 +1,9 @@
 ---
 type: reference
 title: Jellyfin
-description: "Jellyfin on Dankserver: quadlet unit, entropy-volume data layout, ports, client access, hardware-transcode status."
+description: "Jellyfin on Dankserver: quadlet unit, overlay metadata view, provider keys, ports, access, transcode status."
 tags: [jellyfin, dankserver, media-server]
-timestamp: 2026-09-03T00:00:00Z
+timestamp: 2026-10-30T00:00:00Z
 ---
 # Jellyfin
@@ -26,7 +26,7 @@ public vhost `jelly.franksdank.cloud` behind the [[Anubis]] guard chain →
 | Image | `ghcr.io/jellyfin/jellyfin:latest` |
 | Network | `dankserver_appnet`; publishes 8096/tcp, 8920/tcp, 7359/udp, 1900/udp |
 | Data roots | `/home/dank/Containers/jellyfin/config:/config`, `/home/dank/Containers/jellyfin/cache:/cache` |
-| Media | `Volume=/mnt/entropy:/data:ro` — the whole [[Entropy Volume]] mounted read-only; nothing automated writes to it |
+| Media | `Volume=/var/lib/media-meta/view:/data` — the [[Media Metadata View]] overlay, **writable**; sidecars land on the NVMe, the [[Entropy Volume]] array underneath stays read-only and untouched |
 | GPU | none — no `AddDevice`; CPU decode only (jellyfin-ffmpeg path) |
 | Env quirks | `DOTNET_SYSTEM_IO_DISABLEFILELOCKING=true`, `LD_PRELOAD=libjemalloc.so.2`, `MALLOC_TRIM_THRESHOLD_=131072` kept verbatim from the compose-era container (zero-drift cutover) |
 | Health | `curl -f http://localhost:8096/health` every 30 s |
@@ -36,11 +36,23 @@ public vhost `jelly.franksdank.cloud` behind the [[Anubis]] guard chain →
 ```
 /home/dank/Containers/jellyfin/config   -> /config   (JELLYFIN_DATA_DIR)
 /home/dank/Containers/jellyfin/cache    -> /cache    (JELLYFIN_CACHE_DIR, XDG_CACHE_HOME)
-/mnt/entropy                            -> /data     (read-only media volume)
+/var/lib/media-meta/view                -> /data     (the SSD-backed overlay view)
 ```
-Library folders point at `/data` paths; because the mount uses the same
+Library folders point at `/data` paths, which are the view's paths: stable
-absolute path as on the host, library paths survive container recreation.
+across container recreation, and identical to the raw array layout below it.
 `Local metadata` — save metadata to media folders — is enabled, and it works
 because `/data` is the [[Media Metadata View]] rather than the read-only array:
 `movie.nfo`, `tvshow.nfo`, `episode.nfo` and artwork are written once, on the
 NVMe upper layer, and [[Plex]] reads the same files. Jellyfin is the fleet's
 only metadata writer.
 **Failure signature:** `System.IO.IOException: Read-only file system :
 '/data/Movies/<film>/movie.nfo'` means the container is binding the raw array
 instead of the view — the view is unmounted or the quadlet drifted. That used to
 be the expected state (pre-view, every item update logged it); today it is a
 fault. Run `scripts/media-view.sh status`.
 ## Ports & access
@@ -105,6 +117,61 @@ where the LDAP plugin runs. Verified 2026-09-25: `alex` authenticates over
 `AuthenticateByName` (HTTP 200; full plugin flow logged — service bind,
 search, `ldapUid`, user bind, token).
 ## Metadata providers and their keys (2026-10-30)
 TMDb, OMDb, StudioImages and MusicBrainz are **core plugins** in Jellyfin 12 —
 no DLL, just a configuration file the running container owns under the data
 root. TVDB is not core (its metadata provider was removed in 10.9), so it is an
 official-catalog plugin.
 | Provider | Shape on disk | Key lives in |
 |---|---|---|
 | TMDb | `config/plugins/configurations/Jellyfin.Plugin.Tmdb.xml` → `<TmdbApiKey>` | `/home/dank/Containers/jellyfin/provider-keys` → `TMDB_API_KEY=` (mode 600) |
 | TVDB | `config/plugins/thetvdb_24.0.0.0/` + `…Jellyfin.Plugin.Tvdb.xml` → `<ApiKey>` | the same file → `TVDB_API_KEY=` |
 | OMDb | `…Jellyfin.Plugin.Omdb.xml` | **not configured** — its key is issued by email, 1 000 req/day, and its ToS forbids storing the returned data; ratings stay Jellyfin's own lookup |
 **None of it is tracked in git.** Those XML files belong to the container and
 hold credentials, and `tests/jellyfin_providers.bats` fails the suite if one is
 ever committed. The keys themselves sit in the per-service host env file that is
 the fleet's convention for container credentials (grafana, caddy, pihole,
 copyparty, romm, seerr): `provider-keys`, mode 600, beside the data root.
 `scripts/jellyfin-providers.sh` is the only thing that writes the XML:
 ```
 scripts/jellyfin-providers.sh plan      # what would change; prints no key
 scripts/jellyfin-providers.sh install   # plugin + keys, previous copy to trash
 scripts/jellyfin-providers.sh verify    # plugin + key LENGTHS, never key bytes
 ```
 - **A key never enters git, argv, or the tool's output.** The installer reads
  that 600-perm file itself (a value in argv is world-readable through
  `/proc/<pid>/cmdline`) and refuses it when it is group- or world-readable —
  an exposed key is an error, not a warning. A key that shows up in a transcript
  or `ps` output has leaked: rotate it with the provider, rewrite
  `provider-keys`, re-run `install`.
 - **The TVDB artifact is pinned three ways**: sha256
  `6302fba3857bdde8d186dd099fac09ca43247041f531a528a2c788add70a5319`, the
  official plugin guid `a677c0da-fac5-4cde-941a-7134223f14c8`, and version
  `24.0.0.0` (`targetAbi 12.0.0.0`, which fits the deployed 12.4.0). A sha
  match alone only proves the bytes are the bytes fetched — the guid check is
  what says it is the official plugin. Rollback: move `thetvdb_24.0.0.0/` into
  the trash and revert the pin; a second `thetvdb_*` directory is a refusal,
  never a quiet second copy.
 - **The homelab TVDB key is a licensed key** (its JWT claims say
  `community_supported: false`), so it needs **no PIN** — the optional
  `TVDB_PIN=` line exists only for a subscriber-shaped key.
 - **TVDB is installed and keyed but is not a metadata agent**: TMDb still
  identifies everything. Enabling TVDB as a provider would re-fetch show
  metadata across the library, so it is a deliberate, separate change.
 - **The deployer places the keys before it restarts jellyfin**
  (`ensure_provider_keys` in `scripts/deploy-remote.sh`) and warns instead of
  failing when `provider-keys` is absent, so a host without it still deploys.
 - **Keys and plugins are read at start-up**: after a manual `install` the
  change is inert until `./scripts/deploy.sh dankserver --service jellyfin`.
 - TMDb's terms are the reason this stays narrow: attribution required, no
  caching beyond 6 months, no derivative works. Fine for a personal homelab,
  and the reason no bulk enrichment pass runs against it.
 ## Configuration workflow
 1. Create libraries pointing at `/data` folders (movies, shows, audio).
@@ -0,0 +1,117 @@
 ---
 type: reference
 title: Media Metadata View
 description: "The SSD-backed overlay the two media servers read and write: shape, guards, commands, laws, rollback."
 tags: [dankserver, media-server, jellyfin, plex, storage]
 timestamp: 2026-10-30T00:00:00Z
 ---
 # Media Metadata View
 `/var/lib/media-meta/view` is an OverlayFS merge of the media library: the RAID6
 array below, a directory on the NVMe above. It exists so [[Jellyfin]] can write
 Kodi-format sidecars (`movie.nfo`, `tvshow.nfo`, `episode.nfo`, artwork) that
 [[Plex]] can read. Sidecar metadata is only useful sitting next to the media
 file, and the array is read-only and slow — so the media bytes stay there and the
 *writes* land on SSD. One tree to both servers, one copy of the media.
 The mount point is not a volume and holds nothing while unmounted. It sits inside
 `/var/lib/media-meta` so that layers, kernel bookkeeping, the reap trash and the
 merged door are one directory — and so the mount table grows no new `/mnt` entry
 next to the array.
 ## Shape
 | Item | Value |
 |---|---|
 | Lower layer | `/mnt/entropy` — the RAID6 array, source of truth, never written |
 | Upper layer | `/var/lib/media-meta/upper` — NVMe; every write lands here |
 | Work dir | `/var/lib/media-meta/work` — overlayfs internals, same filesystem |
 | Mount point | `/var/lib/media-meta/view` — the merged tree both servers bind |
 | Unit (repo) | `dankserver/host/systemd/media-meta-view.mount` — the source of truth |
 | Unit (host) | `var-lib-media\x2dmeta-view.mount` — systemd names a mount unit after its escaped path (`-` becomes `\x2d`); `media-view.sh install` writes the repo file under that name |
 | Installer | `scripts/media-view.sh` — `plan` / `install` / `verify` / `status` / `reap` |
 | Written by | Jellyfin only (Plex binds read-only); nothing else in the fleet touches it |
 Both consumers guard the view with
 `UnitConfigItem=ConditionPathIsMountPoint=/var/lib/media-meta/view`, and
 `scripts/deploy.sh dankserver --service jellyfin|plex` calls `media-view.sh
 install` + `verify` before it restarts either container.
 | Consumer | Bind | Mode | Why that path |
 |---|---|---|---|
 | [[Jellyfin]] | `/var/lib/media-meta/view:/data` | rw | It is the metadata writer; its sidecars become SSD bytes |
 | [[Plex]] | `/var/lib/media-meta/view:/mnt/entropy` | ro | Every path in the PMS database was recorded as `/mnt/entropy/...`; only the HOST side of the bind moved, so its database still resolves |
 Putting the sidecars inside Plex's reach is not the same as making Plex use them:
 a library parses `.nfo` only with its *Local media information* agent enabled, and
 this change touches no library setting.
 ## Commands
 ```bash
 scripts/media-view.sh plan              # print the resolved shape, touch nothing
 sudo scripts/media-view.sh install      # create dirs, install the unit, mount, enable
 scripts/media-view.sh verify            # is the view really the overlay? (deploy gate)
 scripts/media-view.sh status            # mount options + upper-layer size/file count
 scripts/media-view.sh reap              # list ghost metadata (media moved, sidecars left)
 sudo scripts/media-view.sh reap --apply # move ghosts to the SSD trash, never delete
 ```
 `--upper-base DIR` and `--lower DIR` relocate the layers for `plan` and `reap`;
 the defaults are the host values above.
 ## Laws
 - **The upper layer must sit on non-rotational storage.** `install` and `verify`
  refuse via `lsblk -dno ROTA` if the upper base lives on the array: an overlay
  whose upper is the spinners is the original problem with extra steps, and it
  silently turns every metadata write back into a seek.
 - **Never mount the view over an unmounted lower.** `RequiresMountsFor=/mnt/entropy`
  plus the container `ConditionPathIsMountPoint` guards: an empty `lowerdir`
  presents an *empty library* as truth, and a media scan would then clear the
  database of everything. systemd does not create a mount point, so `install`
  creates `view` itself; without it the mount simply fails.
 - **Never delete through the view.** A delete writes an overlayfs whiteout
  (`.wh.<name>`) in the upper layer, which hides the array's copy without
  removing it — the file reappears the moment the view is unmounted, and the
  upper layer now carries a landmine. Deletions belong on the raw path
  (`/mnt/entropy`), which the relayout tool already treats as the only writer.
  `reap` refuses to remove whiteouts and only warns about them.
 - **A relayout must be followed by `reap`.** Renames happen on the raw path, so
  sidecars the view already wrote stay at the old coordinates: a ghost folder
  the servers browse and find nothing in. `reap` lists upper-layer directories
  with no counterpart below and moves them (with their subtree) into a dated
  `.AGENTTRASH/` under the SSD base. It is recursive and file-safe: a sidecar
  that exists only above is normal, so only *directories* are candidates.
 - **File-watcher events do not cross the overlay.** Plex's NFO agent and
  Jellyfin's real-time monitor see inotify events for writes that land *in the
  upper layer*, not for changes on the array underneath. After any relayout or
  externally-written metadata, trigger a library refresh — see
  [[Media Relayout]] for the refresh procedure.
 ## Failure signatures
 | Symptom | Meaning |
 |---|---|
 | `jellyfin.service` / `plex.service` show `ConditionPathIsMountPoint=... was not met` and stay inactive | The view is not mounted. Do not start them anyway: `scripts/media-view.sh install`, then `verify`. |
 | Jellyfin log: `Read-only file system : '/data/Movies/…/movie.nfo'` | The container is binding the raw array, not the view — the quadlet drifted back or the view is gone. `media-view.sh status`. |
 | A folder appears in a media server with artwork/`.nfo` but no video | Ghost metadata from a relayout. `media-view.sh reap`. |
 | `verify` prints `refused: …/view is not a mountpoint` | A plain directory is sitting where the overlay belongs. That is the fail-closed guard working; `install` on the box. |
 | `systemctl status` finds no such unit | Look under the escaped name, `var-lib-media\x2dmeta-view.mount`, not the repo file name. |
 ## Rollback
 Revert the commit that pointed the two quadlets at the view and re-deploy both
 services — they bind `/mnt/entropy` read-only again, which is the pre-view shape.
 The NVMe upper layer can stay: nothing reads it once the binds are back, and it
 holds no state either server needs. Unmounting alone
 (`systemctl stop 'var-lib-media\x2dmeta-view.mount'`) is safe while no container
 binds it, and the array is never modified by any of this.
 ## Links
 - [[Jellyfin]] — the metadata writer; its data layout section.
 - [[Plex]] — the read-only consumer; in-place bind-mount history.
 - [[Media Relayout]] — the tool that moves media on the raw path and the reason `reap` exists.
 - [[Entropy Volume]] — the array underneath.
@@ -0,0 +1,342 @@
 ---
 type: reference
 title: Media Relayout
 description: "Census, plan and apply the movie and TV libraries into Jellyfin format: gates, evidence chain, refusals, rollback."
 tags: [dankserver, media-server, jellyfin, plex, procedure]
 timestamp: 2026-10-30T00:00:00Z
 ---
 # Media Relayout
 The movie library on [[Entropy Volume]] arrived as release folders —
 `Some.Movie.2019.1080p.WiKiTaLx.x265-FOO/` with the video inside named after
 the release. [[Jellyfin]] matches a movie by folder and file name, so the
 release noise cost posters, extras and versions. These scripts turn that
 library into Jellyfin's own shape — the movies, the TV packs, and the TV that
 turns up in the movie share — and nothing else in the repo touches a media file.
 | Script | Touches media? | Output |
 |---|---|---|
 | `scripts/media-census.py` | reads headers only | `enum`, `probed`, `plex` TSVs |
 | `scripts/media-plan.py` | no — TSVs in, TSVs out | the plan + the exception report |
 | `scripts/media-apply.py` | **renames, never copies**; sweeps only what a run emptied | the moves, plus a journal |
 | `scripts/media-tv.py` | no — TSV out | a plan that moves TV into the TV library |
 | `scripts/media-shows.py` | no — TSV out | a plan that re-files the TV library itself |
 The seams are `tests/media_relayout.bats` and `tests/media_shows.bats`; the target contract is quoted from
 [jellyfin.org/docs/general/server/media/movies](https://jellyfin.org/docs/general/server/media/movies)
 in `media-plan.py`'s docstring.
 ## Procedure
 Run the census on the machine that mounts the library. `probe` is the slow step
 (a header read per container) and is `nice`d — it is safe beside a live service
 because it never competes for the GPU and never writes inside the library.
 ```bash
 C=scripts/media-census.py
 $C enumerate --root /mnt/entropy/Movies --out /tmp/media-enum.tsv
 $C probe     --enum /tmp/media-enum.tsv --out /tmp/media-probed.tsv --jobs 6
 $C plex      --root /mnt/entropy/Movies --out /tmp/plex-movies.tsv
 ```
 `plex` reads the Plex metadata cache (`--db` defaults to the path the
 [[Plex]] container bind-mounts at `/var/lib/plexmediaserver`) with
 `mode=ro&immutable=1`: a live server is never locked or written.
 Then plan — this reads only the three TSVs, so it costs seconds and can run
 anywhere, including a laptop:
 ```bash
 scripts/media-plan.py --root /mnt/entropy/Movies \
    --enum /tmp/media-enum.tsv --probed /tmp/media-probed.tsv \
    --plex /tmp/plex-movies.tsv --out /tmp/jf-plan.tsv --exceptions /tmp/jf-exceptions.txt
 ```
 **Read `/tmp/jf-exceptions.txt` before applying.** It is not a log: every line
 is a file the plan could not name a film for, a pile that needs a README, or a
 collision it refused. The printed gate lines are the go/no-go:
 `problems`, `bad-names`, `occupied`, `double-claim`, `left-unmoved` must all be
 0 and `unplanned=0` must appear in the report. Any non-zero number means a file
 that would be filed wrong, and the plan is not run until it is understood.
 ```bash
 scripts/media-apply.py apply    --plan /tmp/jf-plan.tsv --root /mnt/entropy/Movies   # dry, writes nothing
 scripts/media-apply.py apply    --plan /tmp/jf-plan.tsv --root /mnt/entropy/Movies --apply
 scripts/media-apply.py verify   --plan /tmp/jf-plan.tsv --journal /mnt/entropy/media-relayout.jsonl
 scripts/media-apply.py sweep    --journal /mnt/entropy/media-relayout.jsonl --root /mnt/entropy/Movies   # dry
 scripts/media-apply.py sweep    --journal /mnt/entropy/media-relayout.jsonl --root /mnt/entropy/Movies --apply
 scripts/media-apply.py rollback --journal /mnt/entropy/media-relayout.jsonl --root /mnt/entropy/Movies --apply
 ```
 An apply also leaves the release folders it emptied standing where they were —
 hundreds of them at the library root — which is what `sweep` removes, dry-run
 first, from the same journal.
 After it lands, both servers need a scan or the change is invisible.
 ```bash
 # Run this from the host shell through `podman exec`, never from a neighbour
 # container: Plex's HTTPD closes without a single byte on any request that
 # arrives from another container's netns (10.89.x.x reads as "empty reply", not
 # as a 401). The token rides stdin, never argv, where any process on the host
 # could read it out of /proc/<pid>/cmdline.
 umask 077
 podman exec plex sh -c "sed -n 's/.*PlexOnlineToken=\"\\([^\"]*\\)\".*/\\1/p' \
  '/config/Library/Application Support/Plex Media Server/Preferences.xml'" >/tmp/px.tok
 podman exec -i plex sh -c 'read -r T
  for k in $(curl -s -H "X-Plex-Token: $T" -H "Accept: application/json" \
      http://127.0.0.1:32400/library/sections | grep -o "\"key\":\"[0-9]*\"" | cut -d\" -f4); do
    curl -s -o /dev/null -w "plex section $k: %{http_code}\n" -X POST \
      -H "X-Plex-Token: $T" "http://127.0.0.1:32400/library/sections/$k/refresh"
  done' </tmp/px.tok
 ```
 [[Jellyfin]]'s scan key lives in `~/.config/secrets/jellyfin` (mode 600; the
 same value is `JELLYFIN_API_KEY` in the ignored `dankserver/.env`). It rides a
 mode-600 curl config, never a header argument, and it must use the
 `MediaBrowser` form — Jellyfin 12.4 refuses a bare `X-Emby-Token` for any key:
 ```bash
 umask 077
 { printf 'header = "Authorization: MediaBrowser Token=\\"%s\\""\n' \
    "$(cat ~/.config/secrets/jellyfin)"
  echo 'url = "http://127.0.0.1:8096/Library/Refresh"'
  echo 'request = "POST"'; } >/tmp/jf.hdr
 curl -s -K /tmp/jf.hdr -o /dev/null -w "jellyfin scan: %{http_code}\n"
 shred -u /tmp/jf.hdr
 ```
 `204` is acceptance; the scan then runs on the server's own schedule.
 ## TV in the movie library
 A box set that arrives in the movie share stays there until someone moves it:
 `media-plan.py` refuses to file TV and parks each series in
 `_TV needs a TV library - <series>`. `scripts/media-tv.py` is the mover. It
 walks the movie library itself (no census needed), groups episode-shaped names
 by series, and writes a plan in the executor's own format:
 ```bash
 scripts/media-tv.py --root /mnt/entropy/Movies --library /mnt/entropy/TVShows \
    --out /mnt/entropy/media-tv-plan.tsv
 scripts/media-apply.py apply --plan /mnt/entropy/media-tv-plan.tsv --root /mnt/entropy
 scripts/media-apply.py apply --plan /mnt/entropy/media-tv-plan.tsv --root /mnt/entropy \
    --journal /mnt/entropy/media-relayout-tv.jsonl --apply
 scripts/media-apply.py sweep --journal /mnt/entropy/media-relayout-tv.jsonl \
    --root /mnt/entropy --apply
 ```
 The shape it writes is the one both matchers read — `Monk (2002)/Season 05/
 Monk - S05E03 - Mr. Monk and the Big Game.mkv` — keeping the file's own name,
 because the publisher's spelling beats a re-derivation, with the year taken
 from the box-set folder. What it refuses it prints and leaves alone:
 - fewer than `--min` (5) files under one series name: one file named `Part 2`
  is a film, not a series;
 - an episode whose name states no season number — no invented `Season 01`;
 - a destination two files both claim: the executor refuses the whole plan
  before its first rename, and only Frank decides which copy is the keeper.
 - an episode the destination library already holds at the same byte count. The
  movie share carried a second copy of the whole Monk box set — 124 episodes
  both sides, the same episode list, every byte count identical, 31.5 GiB each
  way, and head+tail blake2b identical on a 6-episode sample. The mover leaves
  such a series in place and says so; a duplicate is deleted or kept by Frank,
  never copied into one library by a tool.
 A sidecar that shares an episode's own name (`.srt`, `.nfo`) moves with it.
 Below the threshold the mover prints and leaves, so a lone downloaded episode of
 a real show stays in the movie library on purpose — six did on 2026-10-30.
 Filing one by hand is the same executor on a hand-written plan and the same
 journal; the trap is the series year. Read it from the file's own name or the
 folder it owns, never from the title whose bonus folder it happens to be parked
 in — `The Cleveland Show S03E07` inside `Die Hard (1988)/extras/` is not a 1988
 show. A name whose marker is `Part 2` is a film and stays put.
 ## The TV library's own relayout
 `/mnt/entropy/TVShows` is scene packs rather than a library, and Jellyfin has
 already matched and numbered most of it. `scripts/media-shows.py` re-files that
 library into the same shape, and it never guesses: every series name and every
 episode number it writes traces to an authority, and what has none is parked for
 a human to read.
 ```bash
 # 1. The authority: Jellyfin's own matched rows (the DB opens read-only).
 python3 scripts/media-census.py jellyfin --root /mnt/entropy/TVShows \
    --out /tmp/tv_titles.tsv
 # 2. What is on disk (stat only), and the runtimes the film arm needs.
 python3 scripts/media-census.py enumerate --root /mnt/entropy/TVShows \
    --out /tmp/tv_enum.tsv
 python3 scripts/media-census.py probe --root /mnt/entropy/TVShows \
    --enum /tmp/tv_enum.tsv --jobs 6 --out /tmp/tv_probed.tsv
 #    The probe is the slow step: ffprobe reads both ends of every file.
 # 3. The plan. --tmdb needs TMDB_API_KEY; without it a series no provider dated
 #    is simply named without a year, which Jellyfin accepts.
 python3 scripts/media-shows.py --root /mnt/entropy/TVShows --enum /tmp/tv_enum.tsv \
    --titles /tmp/tv_titles.tsv --probed /tmp/tv_probed.tsv \
    --out /tmp/tv_plan.tsv --exceptions /tmp/tv_exceptions.txt
 # 4. Read the exceptions file FIRST — it is the review, not a log. Then the same
 #    executor, journal and rollback as the movie relayout above.
 scripts/media-apply.py apply --plan /tmp/tv_plan.tsv --root /mnt/entropy \
    --journal /mnt/entropy/media-relayout-tv.jsonl --apply
 ```
 Evidence, strongest first — the order the planner believes things:
 - the **episode marker the file's own name states** (`S01E02-E03`, `1x04`);
 - the **`Season NN` folder it sits in and the number its name leads with inside
  it**: `Season 19/01 Charles Barkley, Nirvana.avi` is S19E01;
 - the **provider row**: identity (the series name and its year), and the
  numbering of a file whose own name says nothing;
 - a **bonus folder** anywhere above the file, or Jellyfin's `ExtraType` → the
  series' canonical extra folder;
 - a runtime over `SHORT_MAX` makes it film-shaped, and it leaves for `--movies`
  only when a movie provider names it — otherwise it parks.
 Two refusals carry the design, and a dry run on the real library found both:
 - **Folders resolve a series; they never invent one.** A folder's words may
  resolve a file to a series a provider has already named — `(CBB) Tengen Toppa
  Gurren Lagann - The Complete Series (BD 1080p)` names the registered *Tengen
  Toppa Gurren Lagann* — but a bonus subfolder is not a show. An early run made
  `…/03. Extra/01. NCOP/NCOP 01.mkv` into a root series called `01 NCOP`.
 - **A scraped row does not outrank the filesystem.** Jellyfin numbers
  multi-season packs badly: every file under `SNL 16-20/Season 16..19` carries
  season 16 episode 20, and one Gundam row says season 2004. The folder, then
  the file's own leading number, win; a row that still insists on season 1989
  parks its file instead of creating `Season 1989`.
 The provider's name is copied verbatim, warts included: Jellyfin carries a Series
 named `Black Clover 001` (92 episodes, no year) because it matched a pack off a
 filename. Renaming a show is Jellyfin's work — a folder rename never touches its
 provider ids — so the mover keeps the name it is handed and strips only the pack
 index a release leads with (`4a. Seed DESTINY` → `Seed DESTINY`).
 ## Laws
 - **A moved file is a 404 in [[Plex]] until its section is rescanned.** The
  database keeps the release-folder path, so playback fails `Couldn't find the
  file to stream` while the scanner works through the section. Never read such a
  404 as a lost file: the file is where the journal says it is.
 - **Plex's matched title outranks the filename**, and a Plex year of `0` means
  *indexed but never matched* — trusting it named folders `Title (0)`, which is
  a Jellyfin item with no year, no poster and no identity. A year that is not a
  plausible year is no information.
 - **`<>:"/\|?*` never survive a name.** Plex's own title for *V/H/S/2* is
  `V/H/S/2`; kept verbatim it became `Movies/V/H/S/2 (2013)/` — a tree five
  levels deep that neither matcher reads as a movie.
 - **A folder inside a movie's folder is bonus territory even when its name is
  misspelled.** `Saboteur (1942)/Exrtas/Saboteur - A Closer Look` is a typo for
  `Extras`; the vocabulary rule never fired, the featurette became a movie
  folder, and its words then out-scored the real film and drew that box set's
  other extras into it. The rule that replaced it needs no metadata: the folder
  holding the longest film owns the shorter video filed under it, unless that
  file's own name carries a different year — which is another film speaking.
 - **Matching words are split on everything that is not a letter or a digit.**
  Splitting on separators but not on `/` left tokens like `(1960)/extras` that
  no other name could ever agree with, so a film could not be found by the
  folder holding it, and extras drifted into a neighbour. A bare number counts
  only when it could be a release year, so `1080` and `2160` never pose as
  agreement between two names.
 - **Compare two media libraries with the series engine, never a path
  substring.** Filtering paths for `monk` also returns *Monkey Dust*, and
  letting whichever file a walk happens to reach first stand for an episode
  paired two libraries into a fake duplicate — the same sloppiness pointed the
  other way deletes a file that is unique. The comparison is `media-titles.ep_of`
  on both sides: series, season, episode, bytes, and a hash before anything is
  trashed.
 - **A deletion is staged on the item's own filesystem.** The trash rule names
  `~/.AGENTTRASH`, but `/home` and `/mnt/entropy` are different devices: a 31 GiB
  stage there is a copy rather than a rename, it has to fit in the 187 GiB of
  free space, and `rename(2)` refuses it anyway. Trash goes beside the item —
  `/mnt/entropy/.AGENTTRASH/<date>-<topic>/` — so it costs one inode change and
  rolls back exactly like the move that made it.
 - **A Jellyfin 401 is the header, not the key.** 12.4 authenticates only
  `Authorization: MediaBrowser Token="…"` and refuses a bare `X-Emby-Token` for
  every key, valid or not. This page carried "no credential on this box can
  trigger a scan, the two Seerr keys answer 401" for weeks because of that
  header — the keys were innocent. Before believing a 401, prove the key against
  the server's own table: `select count(*) from ApiKeys where AccessToken = ?`
  on `jellyfin.db` opened `mode=ro&immutable=1`.
 - **A scan that must see a new folder has to be recursive.** `POST
  /Library/Refresh` is the incremental pass, and inotify does not cross a
  rootless podman bind, so folders created on the host stay invisible to it. The
  move is `POST /Items/{id}/Refresh?Recursive=true` (204) per library, the id
  from `GET /Library/VirtualFolders` — its field is `ItemId`, not `Id` — and
  that same record's `RefreshStatus` is the progress read.
 - **A file whose name carries no words is bonus material, not a folder.**
  `.Extra 1 ( 1H 27min multisub ).mkv` parses to `1`; a movie folder named `1`
  is not a movie. It becomes a review line telling a human to name the file.
 - **The executor renames and nothing else.** `rename(2)` cannot cross a
  filesystem, so a plan that would is refused rather than silently copying tens
  of terabytes.
 - **The journal is a write-ahead log: the intent is fsynced before the rename.**
  An interrupted run therefore leaves only two states — an intent whose move
  happened, and an intent whose move never ran — and the next run finishes the
  job without moving anything twice. A file that moved with no journal line did
  not move through this tool, and the tool refuses to guess on its behalf.
 - **The journal lives BESIDE the library root, never inside it** (default:
  `media-relayout.jsonl` next to the root): a log inside the library is a file
  Jellyfin indexes.
 - **Refusals are all-or-nothing.** An occupied target, a vanished source or a
  duplicate destination stops the run *before the first rename*, so the operator
  fixes one thing and re-runs instead of untangling a half-applied plan.
  `--force` relaxes only the opinion guards; it never permits overwriting.
 - **A sweep deletes only what the journal proves the run emptied.** A folder is
  eligible when a journalled file moved out of it or out of something under it,
  it now holds nothing but empty directories, no journalled destination lives
  beneath it, and it is not a folder the run created. A folder nobody moved —
  someone's placeholder — cannot be selected at all, and a symlink anywhere
  inside disqualifies the subtree, because the thing behind a link is not this
  tool's to delete. Every removal is journalled as `rmdir` and `rollback`
  recreates the folder, so the sweep is undone by the same command as the moves.
  Proof carries downward: inside a folder the journal proves emptied, every
  empty directory goes with it. A release's `Screens/` or `Sample/` is never a
  move source itself, so 146 of them held their emptied parents on disk after
  the first sweep.
  A failed `rmdir` is not a removal: only a folder that is really gone leaves
  scope, so the next sweep finishes what a permission problem stopped.
 - **An episode-shaped name marks a SERIES, not a film — and the series is the
  name BEFORE the marker.** `Monk - S05E03 - Mr. Monk and the Big Game` is series
  `Monk`. Keyed on the whole title instead, every episode became a group of one
  and never reached the five-episode size that calls something a series, so a
  box set was filed as 124 movies in 124 folders. The threshold still earns its
  keep: a lone film named `Part 2` stays a film. Episodes of a series are parked
  in `_TV needs a TV library - <series>` for a human to move to a TV source —
  the movie tool never files TV.
 - **One journal is one undo.** The TV move runs on its own journal, and with
  `--root` set to the share that holds BOTH libraries: the executor refuses any
  path outside `--root`, and a journal shared with the movie relayout would
  undo both runs when only one of them is wanted.
 - **Sidecar metadata does not follow a rename, because it lives on another
  filesystem.** The media servers write `.nfo` and artwork through the
  [[Media Metadata View]] (NVMe upper layer); this tool moves bytes on the raw
  array path. After any sweep or TV move, run `scripts/media-view.sh reap`, then
  the same command with `--apply`: the sidecars left at the old coordinates become
  ghost folders — artwork and an `.nfo` with no video — that both servers browse
  and report as content.
 - **Nothing deletes through the metadata view.** A delete there writes an overlayfs
  whiteout in the upper layer, which hides the array's copy without removing it;
  the file returns when the view is unmounted. Deletions in this pipeline happen
  on the raw path only, which is what the journal can undo.
 ## Rollback
 One command, from the journal of the run:
 `media-apply.py rollback --journal …/media-relayout.jsonl --apply`. It recreates
 any folder a sweep removed, restores every source path, removes only the folders
 that run created *and that are now empty*, and refuses everything if any moved
 file cannot be accounted for. A rolled-back journal refuses to roll back twice.
 ## See Also
 [[Jellyfin]], [[Plex]], [[Media Metadata View]], [[Entropy Volume]], [[Testing Policy]], [[Storage Policy]]
 ## Citations
 - Jellyfin Movie documentation — folder/file naming, extras subfolders,
  `Movie (2013) - 216p.mkv` version and `- 4K.mkv` handling:
  <https://jellyfin.org/docs/general/server/media/movies>
@@ -4,7 +4,7 @@ description: "Plex Media Server is a **containerized media server on Dankserver*
 tags:
  - dankserver
  - plex
-timestamp: 2026-09-15
+timestamp: 2026-10-30
 ---
 # Plex
@@ -13,7 +13,8 @@ Plex Media Server is a **containerized media server on Dankserver**: a podman
 quadlet (`plex.service`) under the systemd **user** generator, generated from
 `dankserver/plex/quadlet/plex.container`. It shares the host with a second live
 media server, [[Jellyfin]] (`jellyfin.service`, listening on 8096); both serve
-from the same read-only Entropy Volume mount.
+from the same tree — the [[Media Metadata View]] overlay — so the Kodi-format
 sidecars Jellyfin writes are the files Plex reads, with no copy and no sync.
 **Host:** Dankserver — **Ports:** 32400/tcp (PMS web + API), 32400 is the
 only port PMS needs for client streaming; LAN clients connect directly,
@@ -24,9 +25,9 @@ public traffic can be fronted by Caddy like every other vhost.
 | Item | Value |
 |------|-------|
 | Runtime | podman quadlet, `plex.service` (user manager) — migrated from the native `plexmediaserver.service` OS-package unit (2026-09) |
-| Image | `lscr.io/linuxserver/plex:1.43.4.10903-e5521bd8c-ls325` (pinned; verified == the ghcr `latest` manifest digest at pin time) |
+| Image | `lscr.io/linuxserver/plex:1.43.5.11029-6bf8af3f0-ls329` (pinned; verified == the registry's `latest` manifest digest at pin time: `sha256:8f655bad…e9e2334`) |
 | Data root | `/var/lib/plexmediaserver` bind-mounted **in place** as `/config` (RW) — the migration moved zero bytes |
-| Media | `/mnt/entropy` mounted at the same absolute path **read-only** — every library path in the PMS database stayed valid, zero re-scan |
+| Media | `/var/lib/media-meta/view` (the [[Media Metadata View]] overlay) bound at the same absolute path `/mnt/entropy` **read-only** — every library path in the PMS database stayed valid, zero re-scan |
 | Identity | fleet rootless model — container root maps to the `dank` user (1000:1000); the deployer chowns the data root on first cutover |
 | Port | `32400/tcp` published on `0.0.0.0` — zero client reconfiguration |
 | Network | `dankserver_appnet`, alias `plex`; Tautulli reaches it at `http://plex:32400` |
@@ -38,7 +39,8 @@ public traffic can be fronted by Caddy like every other vhost.
 The migration moved zero bytes: the container bind-mounts the native data
 root where it already lived, and the media archive re-mounts at
-`/mnt/entropy` inside the container, so every absolute library path in the
+`/mnt/entropy` inside the container — only the HOST side of that bind has moved
 (array, then the overlay view), so every absolute library path in the
 PMS database keeps resolving. The one metadata change: rootless podman
 cannot map the real host uid 997, so the fleet model applies — container
 root runs as the `dank` user, and the deployer adopts the data root with
@@ -3,7 +3,7 @@ type: reference
 title: Observability Provisioning
 description: "Monitoring config as code: the config triad, deploy copiers, planner mappings, and where each change type lands."
 tags: [observability, prometheus, grafana, provisioning, deployment]
-timestamp: 2026-10-05T00:00:00-07:00
+timestamp: 2026-10-21T00:00:00-07:00
 ---
 # Observability Provisioning
@@ -20,6 +20,8 @@ committed.
 | `shared/prometheus/prometheus.yml` | scrape jobs (global + per-host) |
 | `shared/prometheus/rules/llm-fleet.yml` | alert rules + recording rules |
 | `shared/grafana/provisioning/` | datasources + dashboards (provisioned tree) |
 | `shared/alloy/config.alloy` | the log pipeline: journal source → label promotion → Loki push |
 | `shared/blackbox/blackbox.yml` | prober modules (`http_2xx`); the listen bind lives on the unit's `Exec` |
 Targets and datasources reference **container names** (`grafana:3000`
 `loki:3100` — `shared/prometheus/prometheus.yml:52-61`), never
@@ -31,8 +33,8 @@ scraped as `host.containers.internal:<port>` because a bridge container cannot
 reach a host-netns bind at the host's LAN IP (`prometheus.yml:89-99`), and only
 the endpoint on another machine (`192.168.0.235:7770`) is addressed by LAN IP
 (`prometheus.yml:128-131`). Prometheus 3 refuses HTML responses as metrics —
-HTML-scraped "probes" are structurally inert and need a blackbox exporter (see
+HTTP liveness rides the **blackbox exporter** (`shared/blackbox/blackbox.yml`,
-[[Stale Signal Audit]]).
+deployed 2026-10-21; see [[Stale Signal Audit]]).
 ## The deploy path
@@ -43,8 +45,9 @@ from `deploy_observability`:
 - `copy_alertmanager_config` → `/opt/alertmanager/config/alertmanager.yml` (`:232-235`)
 - `install_signal_relay` → the user unit + iptables + the health wait on `192.168.0.143:18093/health` (`:243-274`)
 - `copy_loki_config` → `/opt/loki/config/loki-config.yaml` (`:276-279`)
- `copy_alloy_config` → `/opt/alloy/config/config.alloy` (`:281-284`)
+- `copy_alloy_config` → `/opt/alloy/config/config.alloy` — substitutes the `GRAFANA_HOST_LABEL` token with the deploy target's name and **refuses to install** if the token survives; `ALLOY_CONF_ROOT` is the bats sandbox seam
- `copy_grafana_provisioning` → `/opt/grafana/provisioning/` (`:286-300`)
+- `copy_blackbox_config` → `/opt/blackbox/blackbox.yml` (single-file mount of the prober); `BLACKBOX_CONF_ROOT` is its sandbox seam
 - `copy_grafana_provisioning` → `/opt/grafana/provisioning/` — datasources, provider, and **every** `dashboards/*.json` (a glob, not a hand-listed subset; `GRAFANA_PROV_ROOT` is the test seam the bats case uses to run the copier into a sandbox)
 `scripts/deploy-plan.py` maps `shared/prometheus/**` to `prometheus` (plus
 `alertmanager` when the changed file is `alertmanager.yml`, `:251-256`)
@@ -82,11 +85,14 @@ reports 4 alerting + 2 recording. Rules reload with the prometheus service.
 the backend (`http://prometheus:9090`), not loopback.
 **Add a dashboard** — drop the JSON into
-`shared/grafana/provisioning/dashboards/`. Provisioned dashboards live in
+`shared/grafana/provisioning/dashboards/`; the copier globs the directory, so
-unified storage, not (only) the SQLite table — grep the API, not the DB, when
+the next `--service grafana` deploy provisions it. This is a repaired law, not
-verifying (see the verify section of [[Fleet Audit and Posture]]). *(unverified
+a given: until 2026-10-21 the copier named `llm-fleet.json` alone, and the
-2026-10-18: the storage-backend claim needs an admin password to confirm
+`containers` board — merged, tested, documented as "working" — had **never
-against `grafana.db` and the API side by side)*
+been provisioned**; the other boards existed on disk only because someone had
 hand-copied them once. A repo JSON is not a live dashboard until the copier
 carries it, and `deploy.sh` cannot see the gap because it only reports the
 copy it performed.
 ## Verify through the APIs, not the filesystem
@@ -95,90 +101,32 @@ against `grafana.db` and the API side by side)*
 curl -s http://127.0.0.1:9090/api/v1/rules | python3 -m json.tool | head
 curl -s "http://127.0.0.1:9090/api/v1/targets?state=active" | \
  python3 -c "import json,sys; [print(t['labels'].get('job'), t['health'], (t['lastError'] or '')[:60]) for t in json.load(sys.stdin)['data']['activeTargets']]"
-# grafana: datasource + dashboard provisioned
+# grafana 13: /api/health, /api/org, /login all 404 UNAUTHENTICATED, and there
-curl -s -u admin:<pw> http://127.0.0.1:3002/api/datasources | python3 -m json.tool | grep -E '"name"|"url"'
+# is NO local admin password (root .env sets no GF_SECURITY_ADMIN_*; the only
 # account is the Authentik RP login). The proven surface is a real browser
 # session: warm the Anubis challenge on the vhost root, then hit
 # /login/generic_oauth with the e2e storage state (tests/e2e/.auth/qa-e2e-admin.json)
 # — exactly the path specs/20-grafana-sso.spec.ts drives — and read the boards.
 ```
 The two Prometheus queries were run on 2026-10-18 and are what every number on
 this page comes from: 4 alerting + 2 recording rules, and `up` for
 `prometheus`, `node-dankclaw`, `node-dankserver`, `grafana`, `loki`
-`frank-zealot`, `frank-dragoon`, `frank-archon` with the four structural downs
+`frank-zealot`, `frank-dragoon`, `frank-archon`. The four structural downs
-listed in [[Prometheus]]. The Grafana query needs the local admin password and
+that census recorded are retired (2026-10-21, [[Prometheus]]). The Grafana proof is the browser session above, run
-was not run here; the filesystem side (`ls /opt/grafana/provisioning/*`) is
+2026-10-21: it found a provisioned-file gap the filesystem listing hid. The
-proof of installation only — see the next paragraph.
+filesystem side (`ls /opt/grafana/provisioning/*`) is proof of installation
 only — see *Add a dashboard* for why a repo JSON is not a live board.
 Health answers from a stale process and provisioned resources can hide in
 unified storage — a file existing on disk is not proof the running server
 loaded it. See [[Stale Signal Audit]] for the never-green-target decision
 table (refused = outage, 404/HTML = config defect).
 ## Current delivered state (delivery chain live 2026-09-17, re-verified 2026-10-18)
 Prometheus → Alertmanager (`dankclaw/observability/quadlet/alertmanager.container`
 loopback `:9093`) → host-side Signal relay (`scripts/signal-relay.py`, user
 unit, `192.168.0.143:18093` — the LAN IP, the one bind a container can reach
 via `host.containers.internal`) → signal-cli, addressed by number
 `+12062476336` (`scripts/signal-relay.py:36-43`). Every hop re-checked
 2026-10-18: `ss -ltnp` shows `python3` on `192.168.0.143:18093`
 `curl -s http://192.168.0.143:18093/health` → `ok`, and
 `journalctl --user -u signal-relay` ends with successful `sent to +12062476336:`
 lines for `LlmEndpointDown`, `LlmRequestsQueueing`, `LlmSpecDecodeOff` and
 `LlmPrefixCacheCollapse` (firing and resolved). Some targets read down for
 structural reasons (HTML-scraped "probes", Gitea metrics off); the remedy is
 the blackbox exporter. See [[Prometheus]] → Targets.
 **A llama.cpp endpoint is invisible unless it opts in.** Until 2026-10-18,
 `up{job="frank-zealot"}` was 0 for the entire retained history of the metric —
 the oldest sample, 2026-10-11, is already 0 — while the service itself answered
 `/health` 200 the whole time. Cause: llama.cpp only exports `/metrics` when it
 is started with the flag, so the scrape was refused at the protocol layer (the
 flag and its opt-out are on [[frank-zealot]]) and `LlmEndpointDown` (critical)
 stayed firing:
 the relay re-sent it on the 4 h `repeat_interval` for days (`journalctl --user
 -u signal-relay` — firing sends at 01:37, 05:42, 09:47, 13:52, 17:57 and 22:02
 on 2026-10-12). It finally resolved at `2026-10-18T01:53:21-07:00`, after the
 metrics opt-in landed that day; the target
 reports up since (`curl -s http://127.0.0.1:9090/api/v1/targets` 2026-10-18).
 How many payload generations it spanned is not readable from any host
 artifact. *(unverified 2026-10-18)*
 Two laws fall out of it. A target can read down beside a healthy service, so
 read `lastError` from `/api/v1/targets` before believing an outage. And rule
 state lives in Prometheus' process, not on disk — a restart re-stamps
 `activeAt` on a long-standing alert, so a "fresh" alert after a reboot is not
 evidence of a fresh fault. (Alertmanager by contrast **does** persist:
 `--storage.path=/alertmanager` on a host volume keeps silences and the
 notification log — `dankclaw/observability/quadlet/alertmanager.container:39-42`
 `/opt/alertmanager/data`.)
 Remaining known downs are the structural ones above (HTML-scraped probes, Gitea
 metrics, ComfyUI), and `frank-dragoon` exports `/metrics` natively
 ([[frank-dragoon]]), which is why
 `frank-dragoon` has always been green. The active-alert count is a
 moment-in-time value and should never be documented as a constant:
 `/api/v1/alerts` was empty at the 2026-10-18 morning census, held one firing
 `LlmPrefixCacheCollapse` (`activeAt` 11:10:48Z, value 2.6e-4) when sampled
 minutes later, and was empty again by 11:56Z; the relay logged the matching
 firing and resolved sends at `04:26:21-07:00` and `04:41:21-07:00`.
 **What that zealot scrape measured on 2026-10-18.** As of ~04:30 PDT the
 production `frank-zealot` root unit was stopped **on purpose** for a benchmark
 window on the 3090 — the repo Benchmarking Policy says stop the production
 service before benching — and `:7771` answered from a manually booted candidate
 arm (`systemctl is-active frank-zealot` → `failed` after
 `stop-sigterm timed out. Killing`, which is llama.cpp ignoring SIGTERM past
 `TimeoutStopSec`, the known shutdown shape for these engines). So
 `up{job="frank-zealot"} == 1` that day measured the trial, not the production
 shape in `scripts/frank-zealot.env` — that shape is [[frank-zealot]]'s to state,
 and it is the repo contract.
 The unit stayed `enabled`; `./scripts/deploy.sh dankclaw --service frank-zealot`
 reinstalls and starts it, restoring boot persistence and self-healing. Never
 quote a candidate arm's flags or VRAM as fleet state — the process and its shape
 changed between two censuses ten minutes apart.
 ## Related
 - [[Observability]] — stack topology and the alert delivery chain
- [[Stale Signal Audit]] — classifying down targets before "fixing" them
+- [[Stale Signal Audit]] — classifying down targets before "fixing" them, and the
  down-beside-healthy / alert-state-persistence / candidate-arm rules
 - [[Quadlet Lifecycle]] — the units that host these services
 - [[Deploy State Triage]] — how the planner selects these services
@@ -2,7 +2,7 @@
 type: reference
 title: "Observability on Dankclaw"
 description: "The observability stack: what runs, how it is deployed, how it is exposed, and what is still unconfigured."
-tags: [observability, prometheus, grafana, loki, alloy, node-exporter, quadlet, dankclaw]
+tags: [observability, prometheus, grafana, loki, alloy, node-exporter, gpu-exporter, quadlet, dankclaw]
 timestamp: 2026-10-20
 ---
@@ -10,10 +10,10 @@ timestamp: 2026-10-20
 ## Summary
-Seven containers and one host-side process: Prometheus (metrics), Grafana
+Seven containers and two host-side processes: Prometheus (metrics), Grafana
 (dashboards), Loki (log store), Alloy (log shipper), node-exporter (host
-metrics), cadvisor (per-container metrics) and Alertmanager (alert delivery), with `signal-relay` on the host
+metrics), cadvisor (per-container metrics) and Alertmanager (alert delivery),
-carrying every firing group to Signal. All of it is Quadlet-managed,
+plus two host `python3` processes — `gpu-exporter` (per-GPU metrics) and `signal-relay` (Signal delivery). The containers are all Quadlet-managed,
 boot-persistent and loopback-only, deployed by
 `scripts/deploy-remote.sh` → `deploy_observability`. The metrics half works
 end to end; the log half is configured, healthy, and carries no data.
@@ -35,6 +35,7 @@ that proves it today.
 | Alloy | `localhost/franksdank/alloy:latest` | `127.0.0.1:12345` | `alloy.container` | [[Alloy]] |
 | node-exporter | `docker.io/prom/node-exporter:latest` | host netns: `127.0.0.1:9100` + `192.168.0.143:9100` | `node-exporter.container` | [[Node-Exporter]] |
 | cadvisor | `gcr.io/cadvisor/cadvisor:v0.49.1` | bridge publish `127.0.0.1:8080` + `192.168.0.143:8080` | `cadvisor.container` | — |
 | GPU exporter | host `python3`, not a container | `127.0.0.1:9401` + host LAN `:9401` | `scripts/gpu-exporter.service` (user unit) | — |
 | Signal relay | host `python3`, not a container | `192.168.0.143:18093` | `scripts/signal-relay.service` (user unit) | [[Alertmanager]] |
 Every unit lives in `dankclaw/observability/quadlet/`, carries
@@ -45,10 +46,11 @@ linger is on for `dank`). `signal-relay.service` reports `enabled` + `active`
 the same day, and `homepage-status.timer` — the root unit that probes all of
 these on loopback — is `enabled` + `active` (`deploy-remote.sh:319-321` copies
 the probe registry `shared/homepage/services.json` to `/srv/homepage`).
-Dankserver runs two members of this family, node-exporter and cadvisor, as
+Dankserver runs the same family: node-exporter and cadvisor as its own quadlets
-its own units (`dankserver/observability/quadlet/{node-exporter,cadvisor}.container`,
+(`dankserver/observability/quadlet/{node-exporter,cadvisor}.container`,
 `generated` + `active`; node-exporter binds `127.0.0.1:9100` + `192.168.0.235:9100`,
-cadvisor publishes `127.0.0.1:8080` + `192.168.0.235:8080`).
+cadvisor publishes `127.0.0.1:8080` + `192.168.0.235:8080`), plus the host-process
 `gpu-exporter` (a user unit installed the same way as `signal-relay`).
 ## Exposure
@@ -65,10 +67,26 @@ Alertmanager has no vhost at all. node-exporter is the exception that must live
 in the host network namespace, and it binds exactly two addresses rather than
 the wildcard — see [[Node-Exporter]].
-The relay's bind is the one address a bridge container can reach on the host:
+**Same-host reachability from a bridge container is measured, not inferred**
-`192.168.0.143:18093`, with iptables INPUT admitting loopback, the host itself
+(2026-10-21, from inside the Prometheus container): `host.containers.internal`
-and the podman bridge subnet `10.89.0.0/16` before the DROP
+(the podman gateway) reaches host services whether they bind host-net or are
-(`scripts/deploy-remote.sh:255-262`).
+published ports — node `:9100`, cadvisor `:8080`, gpu `:9401` all answer there
 — while the host's own LAN IP is refused (`192.168.0.143:8080`, `:9401`), so
 every same-host Prometheus target uses the gateway and only cross-host targets
 (dankserver) use plain LAN. Whether LAN works at all is network-specific: the
 relay at `192.168.0.143:18093` is reached from alertmanager's network only
 while `install_signal_relay`'s INPUT rules are in force: loopback, the host
 itself and the podman bridge subnet `10.89.0.0/16` accepted **above** a
 trailing blanket DROP. The law the poisoned host taught (2026-10-21):
 `iptables -I` prepends, so inserting the DROP last lands it first — the old
 install order dropped every packet (159 on the DROP, 0 on the ACCEPTs), the
 deployer's health gate failed beside a LISTENing socket, and the drain
 aborted twice before `nft list ruleset` named the cause. The install now
 rebuilds the tail (delete the DROP wherever it sits, ACCEPTs at the head,
 DROP appended); verify any change to it with the sandboxed bats case, not
 with `-C` greps — `-C` answers existence, never order. Never point a scrape
 target at a host LAN IP from its own host on the strength of one working
 example; dial the gateway and probe.
 ## Deployment
@@ -85,13 +103,14 @@ health endpoint (`/prometheus`: `/-/healthy`, `/alertmanager`: `/-/healthy`
 [[Observability Provisioning]] for what is provisioned as code and how to
 verify it through the APIs.
-## Current state (2026-10-20)
+## Current state (2026-10-21)
 * **Metrics: working.** `curl -s http://127.0.0.1:9090/api/v1/targets` reports
-  eight up targets — the three LLM endpoints (frank-archon on `vllm:*`,
+  every job up — the three LLM endpoints (frank-archon on `vllm:*`,
-  frank-dragoon and frank-zealot on `llamacpp:*`), grafana, loki, prometheus
+  frank-dragoon and frank-zealot on `llamacpp:*`), grafana, loki, prometheus,
-  and both node exporters. The four downs are structural (the `comfyui` and
+  both node exporters, cadvisor, gitea (metrics enabled in its quadlet) and
-  `services` HTML probes, and Gitea's disabled metrics endpoint), not outages.
+  the blackbox-probed `services` (SearXNG, OpenWebUI, ComfyUI). The structural
  downs are retired — see [[Prometheus]] → Targets.
  Two rule groups in `shared/prometheus/rules/llm-fleet.yml`: `llm-fleet`
  (2 recording + 4 alert, `llamacpp:*`) and `vllm-fleet` (1 recording + 4
  alert, `vllm:*` — archon queueing, prefix collapse, preemption, p95
@@ -99,12 +118,15 @@ verify it through the APIs.
 * **Delivery: working.** Alertmanager's single `signal` receiver webhooks the
  relay, and `journalctl --user -u signal-relay` records successful
  `sent to +12062476336:` lines for every rule in the set.
-* **Dashboards: working.** Provisioned datasources — Prometheus (uid
+* **Dashboards: working (after the 2026-10-21 copier fix).** Provisioned
-  `prometheus`) and Loki (uid `loki`) — and the `llm-fleet`, `host-system`,
+  datasources — Prometheus (uid `prometheus`) and Loki (uid `loki`) — and the
-  `containers`, `observability` and `logs-overview` dashboards, all
+  `llm-fleet`, `host-system`, `containers`, `observability` and
-  installed from `shared/grafana/provisioning/`. The Loki datasource is how
+  `logs-overview` boards. Until 2026-10-21 the copier carried only
-  you read logs: Loki has no web UI of its own, so Grafana Explore is the
+  `llm-fleet.json` by name and the `containers` board had never been
-  human surface (the guarded `loki.franksdank.cloud` vhost serves only the API).
+  provisioned; the deployer now globs the directory. The Loki datasource is
  how you read logs: Loki has no web UI of its own, so Grafana Explore is the
  human surface (the guarded `loki.franksdank.cloud` vhost serves only the
  API). Verify method and the copier law: [[Observability Provisioning]].
 * **Per-container metrics: the `containers` dashboard (added 2026-10-20; live
  after `deploy.sh <host> --service cadvisor`).** cadvisor is the only way to see
  per-container CPU/memory here, and on this rootless-Podman host it works on one
@@ -117,25 +139,38 @@ verify it through the APIs.
  `container` label. Network and filesystem are NOT exported for these containers
  (no netns/classid attribution under the host-PID layout), so the dashboard
  graphs only CPU, memory and OOM and does not fake the rest.
-* **Logs: fixed in code, awaiting deploy.** The pipeline shipped nothing because
+* **Per-GPU metrics: the `host-system` GPU panels (added 2026-10-20; live after
-  a rootless Alloy cannot read the `0640` journal files, and the container-log
+  `deploy.sh <host> --service gpu-exporter`).** `scripts/gpu-exporter.py` is a
-  globs point at a root-storage layout that does not exist here. The fix is built
+  host-process user unit on BOTH hosts — not a quadlet, because it reads the host's
-  and tested in-repo (2026-10-20): Alloy is now a **root system service on
+  own DRM sysfs and shells to `nvidia-smi`, neither of which a container can do
-  `Network=host`**, reading `system.journal` + `user-*.journal` and pushing to
+  without GPU passthrough. It reads each AMD card's `gpu_busy_percent` /
-  Loki's `127.0.0.1:3100` loopback publish — a throwaway root probe read 31,825
+  `mem_info_vram_*` / `hwmon temp1_input` and one `nvidia-smi --query-gpu` CSV, and
-  lines and pushed 26,966 with zero drops. The **live** instance is still the old
+  emits unified `gpu_*` families (utilization %, VRAM used/total bytes, temperature
-  rootless unit until `deploy.sh dankclaw --service alloy` runs. Detail in
+  °C, power W) labelled `gpu` / `vendor` / `model`, with `host` added by the scrape
-  [[Alloy]] → Why root; Loki's store state in [[Loki]] → Contents.
+  job. Two laws: **absent means absent, never a zero** — the R9700s expose no sysfs
-* **Not configured:** no blackbox exporter (the structural downs have no real
+  `power1_average`, so they carry no `gpu_power_watts` line at all (only NVIDIA and
-  uptime signal) and no Alertmanager vhost. The log pipeline's fix is in code but
+  the Dankclaw 8060S iGPU report power); and it binds loopback + the host LAN, like
-  not yet deployed (see *Logs* above).
+  node-exporter, never the wildcard. The dashboard graphs util/VRAM/VRAM-%/temp/power.
 * **Logs: flowing; container labels land with the 2026-10-21 deploy.** The
  pipeline once shipped nothing (rootless Alloy cannot read the `0640` journal;
  the container-log globs pointed at a root-storage layout that does not exist
  here) — fixed 2026-10-20 by making Alloy a **root system service on
  `Network=host`**. Journal lines now reach Loki (measured ~6.5k lines/5min),
  but the stream carried only `job`/`service_name`, so every per-container
  view was structurally empty: Loki metrics queries group only by INDEXED
  stream labels. Alloy now promotes podman's `CONTAINER_NAME` to a `container`
  label (plus a `host` label) at ingestion — `loki.process` in
  `shared/alloy/config.alloy`; detail in [[Alloy]] → Why root, and the
  purpose-test law in [[Observability Provisioning]].
 ## Citations
- `dankclaw/observability/quadlet/{prometheus,alertmanager,grafana,loki,alloy,node-exporter,cadvisor}.container`
+- `dankclaw/observability/quadlet/{prometheus,alertmanager,grafana,loki,alloy,node-exporter,cadvisor,blackbox-exporter}.container`
 - `scripts/deploy-remote.sh:219-300` (copiers + relay), `:521-536` (health waits), `:477-486` (grafana env seed)
 - `scripts/signal-relay.py:26-49`, `scripts/signal-relay.service`
 - `scripts/gpu-exporter.py`, `scripts/gpu-exporter.service` (per-GPU host exporter)
 - `shared/prometheus/prometheus.yml`, `shared/prometheus/alertmanager.yml`, `shared/prometheus/rules/llm-fleet.yml`
 - `shared/grafana/provisioning/`, `shared/loki/loki-config.yaml`, `shared/alloy/config.alloy`
 - `shared/blackbox/blackbox.yml` (prober modules; the listen bind is on the unit's `Exec`)
 - `tests/observability_quadlet.bats` (boot persistence, no wildcard publish, probe addresses, delivery contract)
 - Host evidence 2026-10-18: `ss -ltnp`, `podman ps`, `systemctl --user is-enabled|is-active`, `curl` on `127.0.0.1:9090/api/v1/{targets,rules,alerts}`, `127.0.0.1:3100/loki/api/v1/labels`, `127.0.0.1:12345/metrics`, `127.0.0.1:3002/api/health`, `192.168.0.143:18093/health`, `journalctl --user -u signal-relay`
@@ -42,29 +42,29 @@ Authentik; post-auth the Caddy vhost routes straight to `prometheus:9090`) —
 The `frank-medium` job is **retired**: no `frank-medium` job exists in the config or in `/api/v1/targets` (2026-10-18), and the endpoint itself left the fleet — its history is on [[frank-archon]], which holds the port today.
-**Three jobs read down today, and they are not outages** (re-audited against `curl -s http://127.0.0.1:9090/api/v1/targets`, 2026-10-18; first audited 2026-09-16):
+**No job reads down structurally any more** (the three permanent-reds were
 retired 2026-10-21; census re-run after each deploy):
-| Job | Target | `health` | `lastError`, verbatim |
+| Job | Target | how |
-|-----|--------|----------|------------------------|
+|---|---|---|
-| `gitea` | `http://192.168.0.143:3000/metrics` | down | `server returned HTTP status 404 Not Found` |
+| `gitea` | `192.168.0.143:3000/metrics` | metrics enabled in the quadlet (`GITEA__metrics__ENABLED=true`, `ENABLED_HOST=0.0.0.0`) — the page once said "stays off until `[metrics] ENABLED = true` in its own config"; the env form is what the deployer ships |
-| `comfyui` | `http://comfyui:8188/` | down | `received unsupported Content-Type "text/html" and no fallback_scrape_protocol specified for target` |
+| `services` | `http://192.168.0.143:8081` (SearXNG), `:3005` (OpenWebUI), `http://127.0.0.1:8188` (ComfyUI) | **blackbox exporter** probe, module `http_2xx` — the comfyui metrics job is retired, one owner per service. The prober runs in the host netns, so comfyui is probed over loopback (its published port); bridge container names do not resolve there |
 | `services` | `http://192.168.0.143:8081/` | down | `received unsupported Content-Type "text/html; charset=utf-8" and no fallback_scrape_protocol specified for target` |
 | `services` | `http://192.168.0.143:3005/` | down | same `unsupported Content-Type "text/html; charset=utf-8"` error |
-Gitea's exporter stays off until `[metrics] ENABLED = true` in its own config (no metrics env is set in the container, and `curl -s http://127.0.0.1:3000/metrics` → 404, 2026-10-18); ComfyUI and the two `services` probes are HTML pages, which Prometheus 3 refuses as metrics. Everything else is green: `prometheus`, `node-dankclaw`, `node-dankserver`, `grafana`, `loki`, `frank-zealot`, `frank-dragoon`, `frank-archon`.
+Scraping a web root is not a health check: an HTML page fails the scrape
 protocol (`unsupported Content-Type`), which is what kept those targets red
 beside healthy services. The classification rule and the honesty law live in
 [[Stale Signal Audit]]. The prober is `blackbox-exporter.container` (host
 netns, config `shared/blackbox/blackbox.yml`); it binds 9115 on loopback and
 the LAN, and Prometheus scrapes it at `host.containers.internal` — traffic to
 that gateway lands on the host's loopback listener. The gateway address is
 not itself bindable from the host netns.
 Plain targets stay `host:port`, never URLs — the probe job's URL targets are
 the documented exception, relabeled onto the prober.
 **`frank-zealot` reads up since 2026-10-18, and did not before.** llama.cpp only answers `/metrics` when started with `--metrics`, so `up{job="frank-zealot"}` was 0 across the whole retained history (the oldest sample, 2026-10-11, is already 0) while the service answered `/health` 200 the whole time; the flag and its opt-out are on [[frank-zealot]]. Read `lastError` before believing an outage: a target can read down beside a healthy service.
 Alert state is a live property, not a page fact: `curl -s http://127.0.0.1:9090/api/v1/alerts` returned no alerts at the 2026-10-18 morning census, one firing `LlmPrefixCacheCollapse` (info, `frank-zealot`, `activeAt` 11:10:48Z) minutes later, and none again by 11:56Z.
 The remedy for a probe-shaped question is the **blackbox exporter**; the
 classification rule and the honesty law for it live in [[Stale Signal Audit]].
 Until the exporter is deployed, these three stay red for a structural reason:
 deleting them would
 hide a real service, and fixing them by publishing databases or enabling metrics
 would trade a wrong signal for a real exposure. Targets are `host:port`, never
 URLs.
 ## Volumes
 | Container path | Host path |
@@ -10,7 +10,7 @@ timestamp: 2026-09-03
 ## Server Overview
-Dankclaw is an AMD APU + RTX 3090 inference host. It ran CIRU ROCmFPX llama.cpp inference through 2026-08-22 ([[frank-dragoon]], retired). The host runs Ubuntu 26.04 with an AMD Ryzen AI MAX+ 395 Strix Halo APU. The APU uses unified memory — CPU and GPU share the same physical RAM pool. Today it serves [[frank-dragoon]] (halogen-flash-server 0.16.2 on the iGPU, port 7770) and [[frank-zealot]] (llama.cpp PrismML fork on the RTX 3090, port 7771).
+Dankclaw is an AMD APU + RTX 3090 inference host. It ran CIRU ROCmFPX llama.cpp inference through 2026-08-22 ([[frank-dragoon]], retired). The host runs Ubuntu 26.04 with an AMD Ryzen AI MAX+ 395 Strix Halo APU. The APU uses unified memory — CPU and GPU share the same physical RAM pool. Today it serves [[frank-dragoon]] (halogen-flash-server 0.18.20 on the iGPU, port 7770) and [[frank-zealot]] (llama.cpp PrismML fork on the RTX 3090, port 7771).
 ## Hardware Specifications
 ### CPU
@@ -105,7 +105,7 @@ cable between claw and server is required for 10G.
 - Unified memory eliminates separate VRAM limits. The GPU uses system RAM.
 - The GPU budget is the GTT aperture: ~114 GiB (`mem_info_gtt_total` in `/sys/class/drm/card1/device/` = 122,402,963,456 B; `card1` is the amdgpu iGPU, `PCI_ID=1002:1586`; `llama-server --list-devices` reports 116,732 MiB). rocm-smi's smaller SVM/dedicated carve-out is **not** the budget — the 2026-08 "64 GiB rocm-smi" lesson is stale. Do not use `free` or `dmidecode` for the GPU budget.
- The active services are `frank-dragoon` on 7770 — **halogen-flash-server 0.16.2** serving the native v2 abliterated `.hgn` in a rootless podman container ([[frank-dragoon]]) — and `frank-zealot` on 7771 ([[frank-zealot]]). The gufo v0.1.1 / UD-Q8_K_XL runtime this page once described is retired (2026-09-28 → 10-01, [[frank-dragoon]]); the gufo build tree is still on disk at `/home/dank/GitPackages/gufo-v0.1.1/`.
+- The active services are `frank-dragoon` on 7770 — **halogen-flash-server 0.18.20** serving the native v2 abliterated `.hgn` in a rootless podman container ([[frank-dragoon]]) — and `frank-zealot` on 7771 ([[frank-zealot]]). The gufo v0.1.1 / UD-Q8_K_XL runtime this page once described is retired (2026-09-28 → 10-01, [[frank-dragoon]]); the gufo build tree is still on disk at `/home/dank/GitPackages/gufo-v0.1.1/`.
 - The gufo-era note that the active runtime links against the host ROCm (`/opt/rocm` → `/etc/alternatives/rocm` → 7.2.4) through its RPATH belongs to that retired source build. The serving runtimes do not depend on the host ROCm that way: halogen runs inside its digest-pinned image, and `frank-zealot` is a CUDA build on the 3090. The retired CIRU venv's TheRock 7.15 SDK (`LD_LIBRARY_PATH`) was never in the serving path either.
 - The retired vLLM venvs (`/home/dank/vllm-venv`, `/home/dank/vllm-vk-venv`) were moved to `/home/dank/.AGENTTRASH/` on 2026-08-18.
 - 424 GB free on `/` (`df -h`, 2026-10-18) is ample for model storage.
@@ -70,12 +70,12 @@ back toward 52. Check `VLLM_CUDAGRAPH_*` actually present in
 `--kv-cache-memory-bytes` does NOT appear in the arena arithmetic — the plan
 reserves the registry price, not the actual pool. Raising the pool consumes
-real free VRAM without growing the arena; the registry over-prices the pool
+real free VRAM without growing the arena; the registry OVER-PRICES the pool
-(9190 B/token planned vs 7,891.8 measured at 200704 fp8 — 7,701 at 262144;
+(era G measured ~14 % under the planned B/token — the gap is the lesson, not
-**never reuse a B/token ratio across geometries**, and subtract ~3.87 MB
+the number; **never reuse a B/token ratio across geometries**), and there is a
-block-alignment loss when converting a token target to bytes). The ask lands
+~4 MB block-alignment loss to subtract when converting a token target to
-block-aligned: 3,166,500,000 B → 401,408 tokens; verify with
+bytes. The ask lands block-aligned: verify against the engine's own
-`GPU KV cache size: N tokens, Maximum concurrency … X.Xx`.
+`GPU KV cache size: N tokens, Maximum concurrency … X.Xx` line at boot.
 **Driver-reported free VRAM after hours of traffic is NOT allocatable
 arena.** It is the estimator gap: the registry over-prices KV and charges
@@ -117,11 +117,13 @@ done, and read post-boot free VRAM per rank (`rocm-smi --showmeminfo vram`)
 **KV accounting — pin the pool BYTES, floor-sized:** set
 `--kv-cache-memory-bytes` PER RANK to the contract floor
 (`seqs × MAXLEN × measured B/token`), not the native window — extra pool bytes
-come out of the arena/headroom and buy nothing. Measured B/token on this
+come out of the arena/headroom and buy nothing. The B/token rate is a property
-hybrid: **7,701 fp8 / 13,839 bf16** (decomposed: attention 6,138/12,276 +
+of the GEOMETRY: era G (3 × 200,704 fp8, vLLM fork) measured **7,891.8**;
-dtype-independent mamba/side 1,563). Measured rates are shape-dependent —
+the radiance-era hybrid measured ~7,700 fp8 / ~13,800 bf16 at its own shape.
-7,891.8 at the 200704×3 shape. Capacity does NOT scale with `SEQS`: the
+Read your live rate from the boot ledger every time; the decomposition (a
-planner's KV reserve is for ONE max-model-len request.
+dtype-independent mamba/side stream rides on top of the attention KV) is why
 the ratio moves. Capacity does NOT scale with `SEQS`: the planner's KV reserve
 is for ONE max-model-len request.
 **Parallelism ceilings you cannot flag past:**
@@ -179,18 +181,6 @@ Cutover order: ladder → winner passes and serves → repo env + seam + harness
 (`git stash -u`) if the tree is dirty → deploy (one restart) → verify health
 + engine flags in the log.
 **Measured ladder on this box (200704×3, fp8 KV, precise memory):**
 | shape | util | graphs | arena | result |
 |---|---|---|---|---|
 | E | 0.98 | 0.55 | 17.22 GiB / 7,522 slots | boot ok |
 | F | 0.982 | 0.32 | 17.52 GiB / 7,653 slots | boot ok |
 | 1.0 | 1.00 | 0.07 | — | UNBOOTABLE (worker-init ValueError, both ranks) |
 | G (deployed) | 0.982 | 0.07 | 17.77 GiB / 7,762 slots | KV 401,408 tokens / 2.00x; 0.89 GiB real free/card held through a live 4,096-token generation at ~38–39 tok/s |
 bf16 ceilings at util 0.98: 3 seqs → 241,664 passes / 245,760 and 262,144 OOM
 at graph capture; 6 seqs fp8 → 10.70 GiB ask with 9.18 GiB free (fails) —
 ~207k per context max.
 ## 7. Arena audit procedure (when asked for "more experts")
@@ -0,0 +1,287 @@
 ---
 type: "howto"
 title: "Radiance Architecture Plugins — declaring a model"
 description: "Writing a radiance architecture plugin: declaring weights, buffers, KV groups and ops, then issuing the step."
 tags:
  - llm
  - radiance
  - plugins
  - howto
 timestamp: 2026-10-30
 ---
 # Radiance architecture plugins
 An architecture plugin declares a model in two phases: `rad_arch_declare` names every weight,
 buffer, KV group and op **once**, against the model's metadata; `rad_arch_step` issues the ops for
 one step. The engine picks the plugin by the exact `(rad_arch_id, rad_arch_quant)` pair — no
 fallback — so one family at two weight formats is two `.so` files, and branching on a quant
 descriptor inside one plugin is the anti-pattern the split exists to prevent.
 The in-tree plugins are the templates: `arch/llama_fp8/llama_fp8.cpp` (the smallest complete
 decoder), `arch/gemma3`, `arch/qwen35_bf16`, and the header-only blocks in `arch/common/`
 (`rad_block_attn*.h`, `rad_block_mlp*.h`, `rad_block_moe_fp8.h`, `rad_block_gdn*.h`,
 `rad_block_dflash2.h`, `rad_block_mtp*.h`, `rad_block_vit.h`, `rad_block_ple.h`,
 `rad_block_glm_*.h`, `rad_qsa.h`, …). Include those by `<arch/rad_arch.h>` — they are installed
 beside the ABI and the in-tree plugins include them by that spelling.
 ## Exports
 ```c
 const char* rad_arch_id(void);              /* "llama", "qwen35", ... */
 const char* rad_arch_quant(void);           /* "" unquantised, else the descriptor served */
 int         rad_arch_serves_imports(void);  /* optional: 0 declines an imported quantisation */
 int         rad_arch_probe(const RadModelMeta* meta, RadArchProbe* out);   /* optional */
 int         rad_arch_declare(RadBuilder* b, const RadModelMeta* meta, const RadBuildCtx* ctx);
 void        rad_arch_step(struct RadCtx* c, const struct RadBatch* batch);
 const RadChatFormat*  rad_arch_chat_format(const RadModelMeta* meta);      /* optional */
 const RadAudioFormat* rad_arch_audio_format(const RadModelMeta* meta);     /* optional */
 ```
 Everything but declare/step is optional. `RAD_ARCH_PLUGIN(NS, ID, QUANT, VERSION, DESC)` in
 `arch/common/rad_arch.h` emits the whole export block, including `rad_plugin_info` (kind
 `RAD_PLUGIN_ARCH`). The plugin's **name must be unique**; the architecture id may repeat.
 `rad_arch_serves_imports` exists because a checkpoint whose quantisation was imported weight by
 weight (Quark, ParoQuant, GPTQ, NVFP4) names its producing tool, which no plugin claims: the family's
 one quantised plugin serves it, and two are refused by name. A second quantised plugin that reads
 only its own containers returns 0.
 ## Declare: the metadata
 `RadModelMeta` is what the loader read from the container's header before any plugin ran: `arch_id`,
 `name`, `quant`, `n_layers`, `n_embd`, `n_head`, `n_head_kv`, **`head_dim`**, `n_ff`, `n_vocab`,
 `n_ctx_train`, expert counts, `rms_eps`, `rope_theta`, and a free-form `kv_key`/`kv_val` table for
 everything else (`rad_meta_geti/getf/gets`). Read dimensions from here — one plugin covers every size
 in a family.
 **`head_dim` is read, never derived.** On the Qwen hybrids it is 256 while `hidden / heads` is 213;
 a derived value silently breaks projection widths, KV page size and the attention kernel's band.
 `RadBuildCtx` is the deployment the plugin must declare against:
 | field | meaning |
 |---|---|
 | `rank`, `world_size` | declare runs once per rank; **dimensions are already divided** — the core inserts no collectives |
 | `max_tok`, `max_seqs`, `max_ctx`, `max_spec` | the step bounds: buffer sizing, kernel bands, the scheduler's window |
 | `max_mixed_tok` | the most a multi-sequence step may carry; a wider step is one sequence (prefill span) |
 | `tp_wire_lossy`, `tp_wire_min_bytes` | invert into `all_reduce`'s `exact`; below the floor a message must go exact |
 | `kv_dtype`, `kv_quant` | the attention cache width and, under fp8, its narrow layout (`RAD_KVQ_V4`, `RAD_KVQ_K6V4`) |
 | `attn_prefill_fp8` | `RAD_ATTN_FP8_*` bits to pass as `attn_paged`'s `prefill_fp8` |
 | `kv_resident_blocks`, `kv_stream_ok` | KV streaming: the ring depth, and permission to declare the streamed-only ops |
 | `state_dtype` | `RAD_F16` when `--state-cache-dtype f16` is served by this plugin |
 | `is_draft`, `scope` | a drafter/encoder declare; `scope` prefixes declared names (`draft`, `vision`, `audio`) |
 | `shape_probe` | a **sizing** declare at a smaller `max_tok`; must not touch state the real declare wrote |
 | `max_enc_patches`, `max_audio_frames` | the encoder pass sizes |
 | `max_out_rows` | logits rows the KL mode needs (0 = the plugin's own rule) |
 | `cvec` | the loaded control vector, or null |
 | `tp_attn_rows` | row-split prefill attention requested |
 | `draft_coupling`, `draft_tree` | declare the coupled/tree forms |
 | `weight_prefetch_off` | declare no weight prefetch beside collectives |
 ## Declare: weights
 ```c
 static const RadWeightDecl d = {
    .dtype = RAD_BF16, .rank = 2, .shape = {n_embd, n_ff},
    .access = RAD_ACCESS_PER_TOKEN, .shard = RAD_SHARD_ROW,
    .group = grp_layer(l), .optional = 0,
    .n_row_parts = 0, .source = nullptr, .planes = nullptr,
 };
 rad_weight w = rad_decl_weight(b, "blk.0.mlp.gate.weight", &d);
 ```
 - **`access`** tells the placement planner how the weight is read: `PER_TOKEN` (dense),
  `PER_REQUEST` (an encoder tower, read once per request), `CONDITIONAL` (routed experts),
  `RARE`, `VOCAB` (lm_head/embedding: read every token but huge, and one is a gather).
 - **`shard`**: `NONE` (replicated), `ROW` (split dim 0), `COL` (split dim 1, needs a following
  all-reduce). The plugin says; the core does not infer. Uneven splits use
  `rad_weight_shard_span`/`_parts` (a width that does not divide into whole 128-column blocks; the
  delta net's `[q|k|v|z]` of unequal part widths).
 - **`group`** is the unit of movement (`layer`, `expert`, `slot`); an expert's gate/up/down share one,
  because the unit of transfer is an expert, not a tensor.
 - **`row_parts`** describes what a `SHARD_ROW` weight stacks along dim 0 (`[gate|up]`,
  `[q|k|v|z]`). **A tensor-parallel rank owns a slice of each stacked projection**, never a
  contiguous slice of the stack; the parts cannot be inferred from a fused container tensor, and a
  contiguous slice produces fluent wrong text.
 - **`source`/`planes`** view an existing stored weight: a block-fp8 linear's `codes,scale` are one
  logical weight taken by a GEMM as two operands. Null `source` is the weight of its own name; null
  `planes` takes every plane.
 - A dtype other than the plane's is legal only as an exact widening (a bf16 norm read as f32).
 The **name map** (`rad_decl_name_map`) is the plugin's map from checkpoint tensor names to declared
 weight names, and it is the only place convert-time fusions live: `COPY`, `CONCAT`, `WNORM` (a
 weight-normalised pair folded to the plain weight), `ROWS`, `TRANSPOSE`, `EMA` (an EnCodec codebook),
 `TAP`, `COLS`. `rad-convert` reads this map from the plugin, so it needs no knowledge of the model.
 `RAD_MAP_COPY` with several sources means "first present wins" (a tied lm_head); `CONCAT` requires
 all. Ask for a weight's encoding with `rad_weight_encoding` **after** its map entry when the graph
 depends on the format (fp8 GEMM vs plain, rotated expert kernels).
 ## Declare: buffers and KV groups
 `rad_decl_buffer` names every activation: `RAD_BUF_TRANSIENT` (liveness-analysed, may share bytes),
 `RAD_BUF_PERSIST` (across steps: a calibration accumulator, a routing histogram), or
 `RAD_BUF_DERIVED` (computed once per step by the core from the batch — `derive` names the
 derivation, e.g. `conv_state_index`). Buffers live in one arena sized once, not an allocator per
 step.
 `rad_decl_kv_group` declares each state kind: `RAD_KV_FULL` (paged per-token, growing),
 `RAD_KV_WINDOW` (bounded by a window), `RAD_KV_LINEAR` (per-sequence fixed state, GDN),
 `RAD_KV_CONV` (a rolling window of `width-1+num_spec`), `RAD_KV_RING` (window floor set at runtime).
 The **block size is not declared** — it comes from the resolved attention kernel's constraints; read
 it back with `rad_kv_block_size`. `rad_bind_layer_kv` binds each layer so the block manager knows how
 many state instances a sequence owns; `rad_kv_group_zero` (a rank with no heads),
 `rad_kv_group_replicated` (every rank holds the same bytes), and `rad_kv_group_dense_read` (read over
 the whole history every step, so it cannot stream) are the three refinements.
 ## Declare: ops
 ```c
 rad_op h = RAD_OP(b, "gemm_w4a8_nt",
                  RAD_PARAMS(RAD_RANGE("M", 1, max_tok), RAD_INT("N", 8192), RAD_INT("K", 5120)),
                  RAD_WEIGHTS(w));
 rad_op_reads (b, h, RAD_BUFS(b_x), n);
 rad_op_writes(b, h, RAD_BUFS(b_y), n);
 ```
 - Parameter names and operand order come from `docs/OPS.md`; a new name is allowed only when the
  vocabulary has none, and the first plugin to declare it fixes its schema.
 - `RAD_RANGE` on exactly one parameter per op gives the selector its bucket table. The core carries
  every parameter opaquely except `RAD_PROLE_SEQ_CHUNK` (cut steps on it) and `RAD_PROLE_CAPACITY`.
 - `rad_op_reads`/`rad_op_writes` are the buffer planner's liveness. An undeclared use is assumed live
  for the whole program: sound but wasteful.
 - **Fusion is selection.** Ask for the fused op; if `rad_op_resolved` is false, emit the chain. A
  fusion's `replaces` hook is a checkable equivalence, not a rewrite.
 - `rad_op_declared(b, "op")` asks whether any library declares an optional op; use it before
  emitting a fused form only an optional library defines.
 - Buffers touched from the **second lane** go through `rad_buf_concurrent`, and an architecture whose
  lane-1 work lies in one stretch declares `rad_lane1_window_begin/end` around it. The runtime
  refuses a lane-1 op outside the window; the window lets buffers used only outside it share bytes.
 Step output: `rad_declare_logits` names the buffer the sampler reads (its dim 0 bounds `n_out`; a
 [n_tok, vocab] logits buffer at a large vocabulary is gigabytes — declare the gather +
 `logits_gemm` shape instead). An embedding program declares `rad_declare_pooled` instead (one row a
 sequence, no sampler, no generation). `rad_declare_scores` opts into host-side scored steps.
 ## Declare: drafters, encoders, generators
 - `rad_declare_drafter(RadDrafterDecl)` — depth, block, proposal shape. `rad_drafter_shared_kv` (a
  drafter with no cache of its own), `rad_drafter_tree` (a tree over the proposals),
  `rad_drafter_short_block` (a second, shorter pass) are refinements. Declare the drafter **last**.
 - `rad_declare_encoder(RadEncoderDecl)` — a vision/audio tower, media preprocessed by the core.
 - `rad_declare_generator` / `rad_declare_audio_generator` / `rad_declare_video_generator` — image,
  audio and video generation programs; the engine sizes the step from the probe's `gen_*` fields
  before declare.
 - Optional format exports let a model state what it knows and the server learn nothing new:
  `rad_arch_chat_format` (a data description of reasoning/tool-call markup, replacing template
  analysis), `rad_arch_audio_format` (how a speech model hears: clip tokens, prompt, language,
  transcript start), `rad_arch_text_prepare` (rewrite request text into training form),
  `rad_arch_video_positions`/`rad_arch_video_pack` (video pass geometry and packing),
  `rad_arch_encode_geom` (encoder pass geometry). Each struct grows only at its end behind
  `struct_size`, so adding a field does not move the ABI.
 **Media preprocessing is the core's**, not the plugin's: resizing, patching, placeholder expansion
 and positions in six processor families (Qwen-VL, Gemma 4, LFM2-VL, Unlimited-OCR, GLM-5.3, Pixtral
 for Nemotron), chosen by `vision_config.model_type` plus owning-model checks; audio through Gemma 4's
 extractor or Qwen3-ASR's Whisper one, chosen by `audio_config.model_type`. A vision model with another
 pipeline needs a core change.
 ## Step: issuing the graph
 `RadBatch` is built once per step and shared by every block: `phase` (`PREFILL`/`DECODE`/`MIXED`),
 `n_tok`, `n_seq`, `token_ids`, `positions`, `cu_seqlens`, `seq_ids`, `lora_slot`, `n_out`/`out_ids`,
 `q_lens`/`ctx_lens`/`max_q_len`/`max_ctx_len`, the KV-group array, the speculative fields
 (`n_spec`, `draft_pass`, …).
 ```c
 static void step(RadCtx* c, const RadBatch* batch) {
    if (batch->draft_pass > 0) { m.ds.step(c, batch); return; }      /* query pass */
    if (batch->draft_pass < 0) { m.ds.context(c, batch); return; }   /* context pass */
    RAD_ISSUE(c, m.op_embed, praw(batch->token_ids, RAD_I32, batch->n_tok),
              RAD_W(m.w_tok), brows(m.b_x, batch->n_tok));
    for (auto& layer : m.layers) { layer.attn.step(c, batch); layer.mlp.step(c, batch); }
    /* ...heads; the lm_head only over batch->n_out rows... */
 }
 ```
 - `rad_issue(c, op, operands…, n)` / `RAD_ISSUE` (n = the batch's ranged extent) /
  `RAD_ISSUE_N` (an explicit row count). `rad_step_fail(c, what)` sets the status helpers return.
 - **A mixed step is the trap.** Continuous batching produces steps with a prefill chunk beside decode
  rows. A block that issues one path for the whole step sends its decode rows down the chunked path,
  which commits recurrent state directly and takes no `num_accepted`; a rejected draft cannot be
  rolled back, and the model stays fluent while drifting. Issue the recurrent path over
  `RadBatch::n_seq_decode` sequences and the chunked path over the rest (the core orders steps
  decode-first, so each half is contiguous; `cu_seqlens` holds absolute offsets, so pass
  `cu + n_seq_decode` with the token operands at their base).
 - A step issued on lane 1 outside the declared window fails by name. An op the core appends behind a
  declared one (runtime LoRA) counts at that op's position.
 - **Zero rows is not a call**: issuing an op over zero rows (a zero ranged extent, or all-empty
  sequences) returns `RAD_E_SHAPE` and writes nothing. The caller skips the issue, as the
  architectures' decode and prefill halves do.
 - `n_out` is 0 for a chunk that finishes no sequence — do not run the lm_head.
 ## Probe: what the plugin measured
 `rad_arch_probe(meta, out)` runs before declare. Its fields are the engine's permissions:
 | field | effect |
 |---|---|
 | `draft_depth`, `draft_depth_fixed` | the container's spec depth; a block drafter's trained block is the only depth it serves |
 | `shape_probe_ok` | the plugin tolerates a sizing declare, so the activation arena can be lent |
 | `enc_merge`, `audio_frames` | encoder pass granularity; longest audio clip |
 | `gen_rows`, `gen_patches`, `gen_audio`, `gen_ctx`, `gen_video` | generator sizing before declare |
 | `kv_quant_ok` | bits of the narrow KV layouts this model's kernels serve |
 | `chain_pays`, `tree_pays` | the plugin measured chained launches / draft trees faster |
 | `kv_stream_ok` | the architecture can stream a KV group (sparse, block-addressed, with `kv_resolve`) |
 | `control_vector_ok` | the plugin applies `RadBuildCtx::cvec` |
 | `tp_attn_rows_ok` | the plugin serves row-split prefill attention |
 | `state_f16_ok` | the plugin declares recurrent state at f16 and its kernels read it |
 A flag whose probe bit is 0 is **refused at startup with the plugin named** (`--kv-cache-dtype
 fp8_v4`, `--control-vector`, `--tp-attn-split rows`, `--state-cache-dtype f16`, `--kv-resident`).
 That is the design: a deployment switch the plugin cannot honour is an error, never a silent
 substitution.
 ## Worked pattern: the smallest complete plugin
 `arch/llama_fp8/llama_fp8.cpp` builds twice from one source — `llama_fp8` (quant `fp8_e4m3`) and
 `llama_dense_fp8` (quant `""`), differing only in the descriptor and the graph-dump text. It shows
 the whole shape:
 1. `struct Layer { AttnFP8 attn; MlpFP8 mlp; }`, `struct Model` holding `Names`, `Geom`, buffer and
   op handles.
 2. `declare()`: name map entries first, then weights (with `rad_weight_encoding` where the head's
   format decides the kernel), buffers, KV group, the embedding and its all-reduce, then each layer's
   `declare()` from the shared blocks, the final norm, gather and lm_head, **the drafter last**, and
   `rad_note` lines explaining the shape.
 3. `step()`: draft passes first (sign selects query/context), then embedding (+ exact all-reduce
   when needed), each layer's `step()`, the final norm, and the head only over `n_out`.
 4. `probe()`: reads `draft.block_size` from the metadata and refuses any other depth for a block
   drafter.
 5. `RAD_ARCH_PLUGIN(...)` at the bottom.
 Add a new family by copying a neighbour, replacing the name map and geometry, and composing the
 blocks you need; add a new *format* by cloning the plugin with another `(id, quant)` pair and the
 ops that format needs.
 ## Verifying a new plugin
 ```sh
 radiance --model m.rad --debug-graph          # every op, its resolved kernel, and why
 radiance --model m.rad --debug-placement      # the plan
 rad-convert $SRC -o m.rad --plan-only -v      # a convert that plans will boot
 rad-kbench -m m.rad --kernels mylib,libr4d,libref --report k.md --bench
 rad-tune -m m.rad --tp 2                      # only at the shapes you serve
 ```
 `tests/arch_test.cpp` drives a plugin's builder the way the core does, including lane-window
 liveness, so the pinned graphs and steps are the ones the core plans. A plugin that declares a weight
 form no kernel `layout` accepts is a declare-time error naming the weight, encoding and kernel — the
 same check `rad-convert` runs before writing.
 ## Related
 - [[Radiance Plugins]] — kernel libraries, encodings, the build path, ABI law.
 - [[Radiance Conversion]] — how the name map drives a convert, and `--set`/descriptor selection.
 - [[Radiance Model Families]] — the existing `(id, quant)` pairs and their recipes.
 - [[Radiance Tools]] — `rad-kbench`, `rad-tune`, `rad-info --plugins`.
@@ -0,0 +1,354 @@
 ---
 type: "howto"
 title: "Radiance Conversion — checkpoints into .rad containers"
 description: "Turning a checkpoint into a .rad container: rad-convert, the recipe language, quantisers, reuse and export."
 tags:
  - llm
  - radiance
  - quantization
  - howto
 timestamp: 2026-10-30
 ---
 # Radiance conversion
 `rad-convert` turns a Hugging Face checkpoint into a `.rad` container, choosing each weight's
 encoding from a **recipe**. `rad-export` wraps it for finetunes (a merged checkpoint or a PEFT
 adapter) with the stock model's recipe. Neither opens a GPU: conversion is CPU-only and safe beside a
 running server.
 Fetch checkpoints with `scripts/hfget.sh <org/repo> <dir>` (resumable; reads `.safetensors`,
 skips `.bin`/ONNX/GGUF when safetensors exist; `HF_TOKEN` for gated repos; writes a
 `.hfget-manifest` of content keys). `radiance --model <owner/name>` runs the same fetch itself.
 ## What a conversion does
 1. Read `config.json` (or GGUF metadata), and the shard index if present.
 2. Load the architecture, kernel and quantiser plugins and run **declare**, as the engine does.
 3. Resolve every op and ask each resolved kernel's `layout` hook about the planned encoding. A
   recipe no kernel can serve is refused **here**, before any I/O.
 4. Write each logical weight once: the checkpoint's bytes where no rule matches, otherwise the
   planes the matching quantiser writes, a block of rows at a time.
 5. Measure every quantised weight against its source through the core's decoder and print the worst.
 6. Bake the tokeniser, chat template, metadata, recipe and any draft head.
 ```sh
 build/bin/rad-convert /srv/models/hf/Qwen/Qwen3.8-27B -o qwen3.8-27b-bf16.rad \
    --recipe data/recipes/q38-27b-bf16.recipe
 build/bin/rad-convert /srv/models/hf/Qwen/Qwen3.8-Flash-Next -o flashnext-bf16.rad --max-ctx 4096
 ```
 ## Every `rad-convert` option
 | option | meaning |
 |---|---|
 | `<input>` | a `.gguf`, a `.safetensors`, an `index.json`, or a checkpoint directory |
 | `-o, --out FILE` | the container to write |
 | `--recipe FILE` | one rule a line: `PATTERN QUANTISER key=value …`; repeatable; first match decides |
 | `--quant RULE` | `PATTERN=QUANTISER[:key=value,…]`, placed **ahead of** the recipe's rules; repeatable |
 | `--list-quantizers` | every loaded quantiser and the options it takes, then exit |
 | `--arch ID` | override the architecture id read from the checkpoint |
 | `--home DIR` | `$RADIANCE_HOME` (kernels, architectures, quantizers) |
 | `--kernels A:B` | kernel libraries to load, first wins; declare checks the recipe against them |
 | `--tokenizer F` | `tokenizer.json`; its declared chain is baked instead of reconstructed from a pre-tokenizer name. Required for a GGUF, without it the chain is a guess |
 | `--tokenizer2 F` | a second `tokenizer.json` baked as the second vocab (a generator reading its prompt through two tokenisers); keep `tokenizer_config.json` beside it |
 | `--add FILE` | merge one more `.safetensors`, tensor names kept; repeatable; a name in two files is refused |
 | `--config DIR` | read `config.json`, `generation_config.json`, `tokenizer.json` from DIR for a single-file checkpoint |
 | `--chat-template F` | a Jinja template baked in, overriding the checkpoint's (or its absence); trailing newlines stripped; without one `/v1/chat/completions` answers 501 |
 | `--imatrix F` | llama.cpp imatrix: expert-popularity profile, per-column importances, error weighting; required by `ggml` types `iq2_xxs`, `iq2_xs`, `iq1_s` |
 | `--set K=V` | a metadata key, seen by declare **and** written into the container; repeatable |
 | `--max-tok N` | the `max_tok` declare runs at (default 8192, or a generator's largest pass) |
 | `--max-ctx N` | the context declare runs at (default: the checkpoint's training context) |
 | `--max-spec N` | the `max_spec` declare runs at (default 4, or the block a drafter states); `0` leaves a draft head out |
 | `--draft-model DIR` | a drafter shipped as its own repository, merged under `draft.` |
 | `--kv-cache-dtype T` | `bf16` or `fp8`: the cache width declare runs at; the container is identical at either |
 | `--reuse FILE` | copy every weight an earlier container holds with the same encoding, quantiser, options and shape |
 | `--changed FILE` | with `--reuse`: checkpoint tensors (one name a line) that differ; every weight reading one is converted |
 | `--in-place` | with `--reuse`: extend that container; new weights go past its end, the header last; keeps `<file>.pre-append` |
 | `--lora FILE\|DIR` | merge a PEFT adapter into the checkpoint tensors before converting (`W + strength · scale · B·A`); repeatable, merged in order; each merged tensor joins `--changed` |
 | `--lora-strength F` | strength for every `--lora` (default 1) |
 | `--lora-map FILE` | JSON map for adapters whose module names differ: `{"strip":[prefix…],"modules":{"<pat>":[target…]}}`; one `*` matches a block index; several targets mean a fused module (rows split in order) |
 | `--plan-only` | declare, plan every encoding, print the plan, stop before writing |
 | `--report N` | print the N worst quantisation errors (default 20) |
 | `-v, --verbose` | per-weight detail |
 `--reuse` matches a weight on its **encoding, quantiser, options and shape — not on its source
 bytes** — and carries whatever calibration made the old planes. Use it only when source and
 calibration are unchanged, or name the changed tensors with `--changed`. `--in-place` refuses to
 change a weight the container holds or to add an expert.
 ## The recipe language
 ```
 # PATTERN                      QUANTISER  OPTIONS
 blk.*.ffn_*_exps.*.weight      gptq       codes=i4 group=128 scale=bf16 transform=fwht128 calib=$CALIB
 blk.*.ple_ngram.weight         rtn        codes=fp8_e4m3 block=*x* scale=bf16 rule=fixed scale_value=1.99317932128906e-4
 output.weight                  rtn        codes=fp8_e4m3 block=128x128 scale=bf16
 blk.*.weight                   rtn        codes=fp8_e4m3 block=128x128 scale=bf16
 ```
 - One rule a line: `PATTERN QUANTISER [key=value …]`. `#` starts a comment.
 - `PATTERN` is a glob over the **whole logical weight name**: `*` crosses dots, `?` is one character.
 - **The first matching rule decides.** No rule → the checkpoint's own encoding, which must be one
  the engine serves as is: `bf16`, `f16`, `f32`, or block-scaled FP8 (`X.weight` with
  `X.weight_scale_inv`). Anything else is named and refused.
 - `$NAME` / `${NAME}` in a value comes from the environment; an **unset variable is refused**, never
  expanded empty (so an empty `$CALIB` cannot silently produce an uncalibrated convert).
 - A quantiser may **decline** a weight (`RAD_E_UNSUPPORTED`), which keeps the checkpoint's encoding
  — `rtn` declines a rank-1 tensor, so a pattern that also matches a norm leaves the norm alone.
 - An option the quantiser does not declare is refused, by name, before anything is quantised.
 - Two plugins naming the same quantiser are refused, both named.
 ## Quantisers
 | quantiser | what it does |
 |---|---|
 | `rtn` | round to nearest on a grid; each block's scale chosen by `rule` |
 | `gptq` | error feedback against the layer's input Hessian (`calib=DIR`, damped by `damp` = 0.01 of the mean diagonal by default); `act_order=1` quantises columns by descending input energy and stores the permutation; `cd=N` adds coordinate-descent sweeps |
 | `awq` | per-input-column scaling to a searched power, then a per-block clip search, both against the Hessian; column scales are the last scale level |
 | `autoround` | sign-gradient descent on a rounding offset per value and a range factor per block; `iters` steps; slow (a 27B dense model takes most of a working day at defaults; `iters=50` a quarter) |
 | `paroquant` | channel scaling plus `stages` Givens rotations learned against the Hessian, then AutoRound on the rotated weight; the `givens` transform |
 | `m2xfp4` | M2XFP's weight-side Sg-EM: MXFP4 plus one metadata byte per group of 32; its own `m2xfp4` scheme; `adaptive=0` pins the bias, `wide=1` widens it |
 | `ggml` | a llama.cpp format (`q4_0`, `q4_1`, `q5_0`, `q5_1`, `q8_0`, `q2_K`…`q6_K`, `iq4_nl`, `iq4_xs`, `iq2_xxs`, `iq2_xs`, `iq2_s`, `iq3_xxs`, `iq3_s`, `iq1_s`, `iq1_m`), by llama.cpp's own search, imatrix-weighted, codes byte for byte |
 | `cast` | a float weight in another float dtype (`dtype=` f32, bf16, f16, fp8_e4m3, fp8_e5m2), no scale |
 | `import` | rad-convert's own, not a plugin: write the numbers a quantised release ships, in the spelling the kernels read, instead of quantising again |
 Every quantiser except `m2xfp4`, `ggml`, `cast` and `import` takes the grid options:
 | option | values |
 |---|---|
 | `codes` | `i2`…`i8`, `i16`, `i32` symmetric; `u1`…`u8`, `u16` with `zero=`; `fp8_e4m3`, `fp8_e5m2`, `fp6_e2m3`, `fp6_e3m2`, `fp4_e2m1` |
 | `table` | a codebook the codes index: `nf4`, `iq4nl`, `w4nl`, `w5nl`, `binary`, `ternary`, or `;`-separated values |
 | `group=N` / `block=RxC` | where the scales sit; `*` is a whole extent |
 | `scale` | f32, bf16, f16, fp8_e4m3, e8m0; under `scale2` also an integer sub-scale `u4`…`u8`, `i6`, `i8` |
 | `scale2`, `group2` / `block2` | a second scale level the first is relative to (default the whole tensor) |
 | `scale2_value` | the second level fixed at this value, exact in `scale2`'s dtype |
 | `zero` | an integer zero point per block — an asymmetric grid |
 | `transform=fwhtN` | rotate each row by the Hadamard first |
 | `rule` | `absmax`, `mx` (OCP shared exponent), `fixed` (`scale_value=`), `search` (least squared error over candidates and refits, imatrix-weighted) |
 | `clamp` | a signed code's range: `full` or `sym` |
 Named grids: MXFP4 = `codes=fp4_e2m1 group=32 scale=e8m0`; NVFP4 =
 `codes=fp4_e2m1 group=16 scale=fp8_e4m3 scale2=f32`; NF4 = `table=nf4 group=64`; a K-quant-like grid
 = `codes=i4 group=32 scale=u6 scale2=f16 group2=256`; a ternary BitNet weight = `table=ternary`.
 **A container can hold only what some kernel reads.** rad-convert checks every planned encoding
 against the `layout` hooks of the kernels declare resolved and refuses one no kernel accepts. libr4d
 reads the plain floats; block-scaled fp8 and fp8 rows; int8 block rows plain or rotated; routed
 experts as fp8, int4 (`i4*bf16[1x128]/fwht128`) or the `w4nl`/`w5nl` codebooks under E4M3 scales;
 the dense w4, w8 and mxfp4 GEMM rows; and the 2-bit draft head. See [[Radiance Plugins]] for the
 encoding model.
 ## Importing a release someone else quantised
 These forms are read weight by weight and written in the spelling the kernels read; a recipe rule
 re-quantises from the exact dequantisation where the family's kernels want another form.
 | `quantization_config` | read as | notes |
 |---|---|---|
 | `quark` MXFP4 (fp4, group 32, e8m0) | `fp4_e2m1*e8m0[1x32]` | byte for byte; also servable as it stands |
 | `paroquant_mxfp4` | `fp4_e2m1*e8m0[1x32]/givens` | codes/exponents byte for byte plus learned rotations as `givens` tables |
 | `paroquant`, `int5-bitplane`, plain `paroquant` | `u5*f16[1x128]-u8[1x128]/givens`, `u4*…` | AWQ-packed nibbles re-laid as a code bit stream; f16 scales, integer zeros, rotations |
 | `compressed-tensors` `pack-quantized`, 4-bit `int`, group, act-order none/static/weight | `i4*<scale>[1xG]` | `weight_packed` made signed beside `weight_scale`; group/dynamic reorder is refused |
 | same with `symmetric:false` (AWQ) | `u4*<scale>[1xG]-u4[1xG]` | nibbles plus `weight_zero_point`, bit for bit |
 | `auto-round` `auto_round:auto_gptq`, `gptq` v1 | `i4*bf16[1x128]` or `i8*bf16[1x128]` | scales rounded to bf16; convert-time; `import` keeps shipped f16 |
 | `nvfp4-pack-quantized` | `fp4_e2m1*f32[1x16]` | `weight_scale / weight_global_scale`; **no kernel serves NVFP4**, so it always takes a rule |
 | `modelopt` NVFP4 / FP8 / MIXED_PRECISION | NVFP4 `fp4_e2m1*f32[1x16]`; FP8 `fp8_e4m3*f32[1x*]` | ModelOpt's second scale **multiplies**; `input_scale` is not read |
 | compressed-tensors all-FP8 groups | `fp8_e4m3*<scale>[1x*]` or `[RxC]` | nothing imported: stock FP8 with its scale; a per-tensor scale is written once a row |
 | GGUF `Q8_0`, `Q4_K`, `Q6_K` | `i8*f16[1x32]`, affine u4/u6…, `i6*i8…` | blocks split bit for bit; **not servable as they stand** |
 The width is read **per tensor**, so a mixed release imports int4 experts and int8 projections
 together. What an import cannot read (an asymmetric GPTQ zero point, a group-reordering act-order
 index, a GPTQ group under 128, a missing/zero global scale, an unlisted format) is refused by name.
 The release's `quantization_config` names the tool, not a served form, so it is kept as
 `imported_quantization_config.*`.
 `import` options: `codes=uN` makes a symmetric grid unsigned under a zero point of `2^(N-1)`;
 `zero=` the zero plane's dtype (`u4`, `u8`); `scale=` f16/bf16/f32, each shipped scale rounded to
 nearest; `block=RxC` repeats an FP8 scale shipped per row/tensor over that grid. It reads an imported
 integer grid or FP8 with a per-row/per-tensor scale and keeps a plain float weight as it is.
 **NVFP4 to MXFP4 rounds twice** (e4m3 per 16 → power of two per 32), so it loses more than MXFP4 made
 from bf16. Prefer a calibrated MXFP4 container when one exists; the quality verdict is a KL gate, not
 a file size.
 ## The descriptor law (`--set`)
 The architecture plugin is chosen by the exact `(arch id, quant descriptor)` pair. A convert-time
 choice that must also hold at serve time is stated once with `--set`, which hands it to declare and
 writes it into the container. Storage format is not such a choice (the encoding is in the container
 and the plugin reads it there), but the **serving plugin** is: a `quantization_config.*` key
 re-derives the descriptor.
 ```sh
 rad-convert <bf16 Qwen3.8-27B> -o q38-27b-fp8.rad \
    --recipe data/recipes/q38-27b-fp8block-df2.recipe \
    --set quantization_config.quant_method=fp8 --set quantization_config.fmt=e4m3 \
    --set "quantization_config.weight_block_size=128 128" \
    --set quantization_config.activation_scheme=dynamic
 ```
 A bf16 checkpoint whose recipe writes Qwen's block FP8 must therefore be declared *as that format*;
 otherwise the selection pair does not match any plugin and the convert (or the boot) is refused. The
 same law bites the other way when a release's own `quant_method` names a pair no plugin claims.
 ## Inspect and validate the result
 ```sh
 rad-info out.rad                  # architecture, source quant, recipe, sections
 rad-info --recipe out.rad         # the expanded rules — did the intended one win?
 rad-info -v out.rad | head -40    # every tensor, its encoding, and what made it
 rad-kbench -m out.rad --home $RADIANCE_HOME --report kbench.md
 radiance --model out.rad --debug-graph     # every op resolves
 ```
 The converter's own error report (the worst `--report N` weights) measures each quantised weight
 against its source through the core decoder. `rad-kbench` adds the artifact test: weight operands
 read from the container and passed through each kernel's `layout`/`relayout` hooks. Both are on
 [[Radiance Tools]].
 ## Calibration (GPTQ, AWQ, AutoRound, ParoQuant)
 `calib=DIR` is a directory an engine run wrote with `RADIANCE_CALIB_DIR` set: each layer's **Gram
 matrices of its input**, in the domain the kernel multiplies it in. A block that characterises an
 activation declares an f32 `[K,K]` PERSIST buffer and a `gram_accum` op; at `RADIANCE_CALIB_TOKENS`
 (default 65536) the run writes the files.
 Three-step bootstrap: convert with plain `rtn` → serve that container with `RADIANCE_CALIB_DIR` →
 push real prompts through prefill → convert again with the Hessians. The second moment is robust to
 the first container's small quantisation error. A weight with no file is quantised uncalibrated and
 the converter says so; `awq`, `autoround` and `paroquant` fall back to the imatrix diagonal when one
 is given.
 - The accumulator must be **PERSIST**: a transient buffer may share bytes and be overwritten between
  the launches it sums.
 - Files are per rank; the consumer sums them. Under expert parallelism a rank's `down` input covers
  only its experts' tokens; `gate_up` input is the full token set on every rank, so the sum scales
  `H` by world size — GPTQ is invariant to a positive scale.
 - Under tensor parallelism a column-split linear sees only its rank's columns, so a tap may write
  `gram.r<rank>.<stem>.part.bin`; libquant places the parts on the diagonal in rank order, leaving
  cross-rank products zero. They must cover K and be in the model's own domain.
 - `--imatrix` supplies the expert-popularity profile the placement planner ranks units by, the
  per-column importances a quantiser uses, and the error report's weighting.
 ## Reuse, drafters, merging
 ```sh
 # re-quantise only what changed
 rad-convert $SRC -o new.rad --recipe new.recipe --reuse old.rad --changed changed.txt
 # merge a drafter shipped as its own repo (DFlash2, DSpark, MTP assistant)
 rad-convert $SRC -o out.rad --draft-model /srv/models/z-lab/Qwen3.8-27B-DFlash2 \
    --recipe data/recipes/q38-27b-mxfp4-df2.recipe
 # merge a PEFT adapter at convert time
 rad-convert $SRC -o tuned.rad --lora ~/run/adapter --lora-strength 1.0 \
    --recipe data/recipes/q38-27b-mxfp4-df2.recipe
 ```
 - A drafter's tensors go in under `draft.`, its config under `draft.` keys. It has no embedding and
  no `lm_head` of its own; it uses the target's.
 - `--lora-map` exists for adapters trained on names the checkpoint does not use; a fused target
  (`qkv`) takes several targets and splits the adapter's rows in order.
 - `--add` + `--config` + `--tokenizer2` assemble a ComfyUI-style release of separate single files
  (Anima: diffusion model, text encoder, VAE). A name in two files is refused.
 ## `rad-export` — finetunes without hand-writing the command
 ```sh
 rad-export ~/outputs/my-run/checkpoint-400 --dry-run   # what it would run
 rad-export ~/outputs/my-run/checkpoint-400             # adapter -> <name>.rad
 rad-export ~/exports/merged -o ~/models/mine.rad --recipe q38-27b-fp8block-df2
 ```
 | flag | default | meaning |
 |---|---|---|
 | `SRC` | required | a merged 16-bit checkpoint (`config.json` + `*.safetensors`) or a PEFT adapter (`adapter_config.json` + `adapter_model.safetensors`) |
 | `-o, --out FILE` | SRC's folder name + `.rad`, beside SRC or in `out_dir` | the container |
 | `--recipe NAME\|FILE\|auto` | `auto` | a recipe name, a file, or the one recipe whose header names the base model, takes a bf16 checkpoint and needs no calibration |
 | `--base DIR` | the adapter's `base_model_name_or_path`, from disk or the HF cache, else downloaded | the base a LoRA merges into |
 | `--draft DIR\|ORG/NAME` | the config, else the Hub repo the recipe's usage line names | the drafter when the recipe merges one; a usage line with a placeholder asks for `--draft` |
 | `--image IMAGE` | `$RADIANCE_IMAGE`, the config's, then `stilldeadcode/radiance` | run `rad-convert` in this image (docker, CPU-only, no network, **not pulled**) |
 | `--rad-convert PATH` | — | run this binary instead of docker; recipes come from its `share/radiance` |
 | `--memory M` | — | docker memory limit for the convert |
 | `--dry-run` | — | resolve recipe, drafter and paths; print the `rad-convert` line; write nothing |
 | `--keep-merged` | — | keep the merged checkpoint beside OUT |
 | `--config FILE` | `~/.config/rad-export/config.json` | the config below |
 | `--list-recipes [--json]` | — | list the recipes the image/installed tool carries |
 - Input: a merged 16-bit checkpoint (`config.json` + `*.safetensors`) or a PEFT LoRA adapter
  (`adapter_config.json` + `adapter_model.safetensors`).
 - An adapter is merged **exactly as PEFT's `merge_and_unload`** computes it on CPU: `scale` =
  alpha/r (alpha/√r with rsLoRA), `rank_pattern`/`alpha_pattern` honoured, delta in fp32 rounded to
  the adapter's dtype. It streams tensor by tensor into 1 GiB shards, so a 27B merge needs a few GiB
  of RAM, not the model twice. Plain linear LoRA only: embedding LoRA, trained biases, DoRA, saved
  modules, convolution LoRA, a rank other than the config's, and a missing target are refused before
  anything is written. The adapter's tokeniser/template replace the base's where shipped.
 - The recipe comes from the image (`/opt/radiance/share/radiance/recipes`) or the installed
  `rad-convert`'s share; `auto` matches the base model's name in recipe headers and asks when several
  fit. A recipe needing calibration is listed but never picked by `auto` — a finetune's Hessians are
  not the stock model's.
 - It runs `rad-convert` on the CPU, under Docker as the caller with no network, and writes the log to
  `OUT.log`.
 - `~/.config/rad-export/config.json` pins what auto cannot know: `image`, `out_dir`,
  `recipes{model:recipe}`, `drafts{recipe:dir}`, `bases{repo:dir}`.
 - It is Python 3 with no dependencies; merging needs `torch`/`safetensors`, fetching a missing base or
  drafter needs `huggingface_hub`. The image carries no Python; extract the script with
  `docker run --rm --entrypoint busybox stilldeadcode/radiance cat /opt/radiance/bin/rad-export`.
 ## Scoring a container (the KL mode)
 ```sh
 # once, on the bf16 container
 radiance --model bf16.rad --kld-record /srv/kld/q38 --kld-corpus corpus.jsonl
 # each candidate, same flags otherwise
 radiance --model quant.rad --kld-ref /srv/kld/q38 --kld-out report.json
 ```
 - The corpus is JSONL, one `{"prompt", "score_from"?, "source"?}` a line; only `prompt` is required.
  Positions `[score_from, n-1)` are scored; `source` groups the report. Keep the scoring corpus
  disjoint from the calibration corpus.
 - `--kld-record` writes the reference's log-probabilities (f16 over the vocabulary, the 16 most
  likely tokens exactly, the token ids). DIR is never overwritten and is read only once `kld.json`
  exists, which is written last.
 - `--kld-ref` scores this model's f32 logits against stored rows: mean, median, 99/99.9/99.99th
  percentile KL, top-1 agreement, and both models' perplexity. `--kld-out` writes the report; `FILE.rows`
  is `[positions, 4]` f32 (KL, both NLLs, argmax agreement) for position-by-position comparison.
 - The mode turns speculation and the prefix cache off, turns the step pipeline off, and refuses an
  explicit spec depth above 0. The logits buffer holds a row per token of a step, so a large
  `--max-num-batched-tokens` costs pinned host memory as well as VRAM.
 - `--kld-lora NAME` scores every document through a served adapter, against a reference recorded
  from the merged checkpoint; `--kld-lora-sources PREFIX` runs only matching documents through it,
  the rest as the base, in the same steps — each document's rows should be byte-identical to the
  all-adapter and all-base runs.
 The gate the fleet uses (mean KL ≤ 0.03, top-1 ≥ 95 %) is on [[frank-archon]] and
 [[Quantization Production]]; the mode here is only the instrument.
 ## Convert laws paid for in full
 - **`--plan-only` first.** A plan that resolves every op will boot; it costs seconds against an hour
  of I/O on a large checkpoint.
 - **Declare runs at the step and context bounds, and some graphs depend on them.** Qwen4-Exp sizes
  its QSA buffers by `--max-ctx` and refuses a `--max-tok`/context pair whose score plane would pass
  2 GiB; the defaults (8192 tokens, the checkpoint's training context) fit most models, and a
  generator's largest pass may override `--max-tok`.
 - **Scale names follow the tensor.** `scale_name()` appends `_scale_inv` to the *tensor* name, so a
  stacked `…experts.gate_up_proj` keeps its scales at `…gate_up_proj_scale_inv`.
 - **File shadowing: first wins.** The reader opens index-named files sorted and emplaces every tensor
  each file physically holds, first-wins. A merged PLE shard must sort first
  (`00-ple.merged.safetensors`) or a raw shard wins the name.
 - **Never write through a staging symlink.** A checkpoint may symlink its raw index; `open(path,"w")`
  writes through it and corrupts the checkpoint. Write a temp file and `os.replace`.
 - **One encoding per expert table.** A whole expert kept bf16 is the only thing legal beside a
  table; a split tier (down=fp8, gate_up=w4) is unservable.
 - **The descriptor pair has no fallback** (above), and **convert is card-less**: GPTQ solves on CPU
  BLAS, so the GPU is irrelevant to production and the card only decides what can be decoded.
 ## Related
 - [[Radiance Model Families]] — the (arch id, quant descriptor) table and the command per model.
 - [[Radiance Plugins]] — quantiser and kernel-plugin contracts behind these options.
 - [[Radiance Tools]] — `rad-info`, `rad-kbench`, `rad-tune`.
 - [[Quantization Production]] — the fleet's calibration/KLD campaign and its gates.
 - [[frank-archon]] — the deployment whose recipes and convert laws these are.
@@ -0,0 +1,140 @@
 ---
 type: "reference"
 title: "Radiance Engine — the inference server behind frank-archon"
 description: "The radiance inference engine: its shape, plugin model, container format, and how the reference pages fit together."
 tags:
  - llm
  - radiance
  - reference
 timestamp: 2026-10-30
 ---
 # Radiance Engine — shape, plugin model, container
 Radiance is StillDeadcode's inference server for AMD GPUs, written in C++20 and HIP: language
 models, plus image, video, speech and music generation and embeddings. One HTTP front end speaks
 OpenAI's chat, completions, Responses, images, audio and video APIs, Anthropic's Messages API and
 llama-server's native one. Nothing in the serving path is Python.
 The fleet's only radiance endpoint is [[frank-archon]] (2× R9700, TP=2, Dankserver). **This page
 and its siblings own the engine reference; the live deployment shape, its Eras and its measured
 laws stay on the endpoint page.** The numbers here describe upstream `main` as read 2026-10-30
 (release 1.14.0, plugin ABI 31). The fleet pin is older (1.8.1): a flag below that post-dates the pin
 is a boot refusal, never a silent no-op, and the endpoint page records the CLI families that pin
 itself retired relative to its own rollback target.
 Source: `https://codeberg.org/StillDeadcode/radiance` at `8da043c`. Engine docs live in `docs/`
 (`GUIDE.md` for serving, `TOOLS.md` for tools, `PLUGIN.md` for plugins, `ARCHITECTURES.md` per
 family, `OPS.md` for the op vocabulary, `spec.md` for the design).
 ## The four moving parts
 ```mermaid
 graph LR
    A[Checkpoint<br/>safetensors / GGUF] -->|rad-convert + recipe| B[.rad container]
    B --> C[radiance engine]
    C --> D[OpenAI / Anthropic /<br/>llama-server wire APIs]
    P1[kernel library .so<br/>libr4d, libavx, libref] --> C
    P2[architecture .so<br/>one per arch id × quant descriptor] --> C
    P3[quantiser .so<br/>libquant] -->|rad-convert| A
 ```
 - **Engine** — `core/`: declare phase, placement/memory planner, scheduler, KV manager, sampler,
  tokeniser, HTTP server, media preprocessing.
 - **Kernel libraries** — implement ops for hardware. `libr4d` targets RDNA4 (`gfx1200`,
  `gfx1201`); `libavx` serves the host backend (AVX-512/AVX2) with no GPU at all; `libref` is the
  slow reference implementation every other library is checked against.
 - **Architecture plugins** — declare one model family at one weight format: its weights, buffers,
  KV groups and the ops it issues each step. The engine picks one by an **exact (architecture id,
  quantisation descriptor) pair, with no fallback**.
 - **Quantiser plugins** — `libquant`'s `rtn`, `gptq`, `awq`, `autoround`, `paroquant`, `m2xfp4`,
  `ggml`, `cast`, `import`; they write a weight's numbers into an encoding's planes.
 - **Tools** — `rad-info`, `rad-convert`, `rad-export`, `rad-kbench`, `rad-tune`, `rad-schemas`,
  `rad-tokcheck`, plus `rad-chatparse`, `rad-avx-check`, `r4d_selftest` in the build tree.
 ## Two phases, one static graph
 `rad_arch_declare` runs once per rank and names every weight, buffer, KV group and op, in order.
 The core resolves every op against the kernel hierarchy per band, per domain, and completes the
 whole plan before a byte of weights is read. `rad_arch_step` then only *issues* handles obtained at
 declare; it may not allocate and may not synchronise. A model whose declare succeeds is fully known
 statically, which is what makes `--debug-graph`, the buffer liveness plan and `rad-convert`'s
 plan-only check possible.
 Consequences a plugin author must respect:
 - An op that never resolves is `RAD_NULL_HANDLE` and only fails if issued. A fusion is a *selection*:
  ask for the fused op, emit the chain when `rad_op_resolved` is false.
 - Declaration order is the buffer planner's liveness. A drafter must be declared after the trunk
  it runs behind, or trunk and drafter buffers silently share bytes.
 - Every op in the graph must have a row in a loaded kernel library. `libref` implements the
  conventional vocabulary in `docs/OPS.md`; device-only ops (most fusions) exist only in `libr4d`.
 ## The `.rad` container (RAD2)
 A container is one file holding everything the server needs: weights as **canonical encoding
 planes**, the tokeniser (and optionally a second vocab), the chat template, metadata, the recipe it
 was made with, and any draft head. It names no kernel library.
 - Planes are kernel-independent: codes, scales, zero points, codebooks — row-major, bit-packed
  sub-byte where needed. A kernel rearranges them into its own layout at load (`layout`/`relayout`
  hooks), losslessly, so a kernel library can change layouts without reconverting the model.
 - Encodings are data, not an enum: `RadEncoding` names the scheme (`plain`, `affine`, or a
  quantiser's own), the planes by role, the scale grouping, and a transform (`fwhtN`, `perm`,
  `givens`). That is how one file can hold block-FP8, int4, MXFP4, NVFP4-imported, ParoQuant
  rotations and llama.cpp grids.
 - `rad-info` is the reader: sections (strings, metadata, directory, planes, encodings, vocab,
  profile, data), `--meta`, `--vocab`, `--recipe`, `--plugins`, and `-v` for every tensor with its
  encoding and what made it.
 See [[Radiance Conversion]] for producing one and [[Radiance Tools]] for inspecting it.
 ## Selection key and version law
 - A checkpoint's architecture id is its `config.json` `model_type` without underscores; `rad-convert
  --arch ID` overrides it. `rad_arch_quant` supplies the quantisation descriptor (empty for an
  unquantised checkpoint).
 - **No fallback.** A raw official FP8 checkpoint whose `quantization_config.quant_method` is `fp8`
  is not the pair the `qwen4exp_fp8` plugin claims (`(qwen4exp, "")`), so both serve and convert
  refuse it until the descriptor is changed with `--set` or a staged `config.json`. The deployment
  lesson is on [[frank-archon]]; the mechanism is on [[Radiance Conversion]].
 - `RAD_ABI_VERSION` covers every header a plugin compiles against. A plugin reporting a different
  number is refused at load, with both numbers printed. It moves on any shape change, not on a new
  optional export; rebuild when it changes. Release version and ABI are independent: `/version`
  reports the release, `rad-info --plugins` reports what loaded.
 ## Environment
 | variable | meaning |
 |---|---|
 | `RADIANCE_HOME` | plugin search path: `:`-separated homes, each holding `kernels/`, `architectures/`, `quantizers/`; first wins, same-name files shadow later homes. Compiled-in default is `<prefix>/share/radiance` |
 | `RADIANCE_KERNELS` | default for `--kernels`: ordered kernel libraries, first wins; unnamed libraries are not loaded |
 | `RADIANCE_TUNE` | pin a kernel's tuned axes for a measurement (a measurement aid, not a deployment setting) |
 | `RADIANCE_CALIB_DIR` | a serve run records per-layer input Gram matrices here (the GPTQ calibration tap) |
 | `RADIANCE_CALIB_TOKENS` | tokens after which the tap stops and writes (default 65536) |
 | `RADIANCE_LOG`, `RADIANCE_LOG_STEPS` | log level; per-step token counts |
 | `RADIANCE_DEBUG_RESID` | residual-stream probe per (layer, site) |
 | `RADIANCE_DFLASH_Q_TEMP` | temperature of a DFlash2 coupled walk's candidate softmax |
 The full list of debug knobs (`RADIANCE_DEBUG_*`, `RADIANCE_DUMP_*`, `RADIANCE_OP_ORACLE`, …) is in
 the source's `getenv` calls; none is a deployment setting.
 ## Where the pages are
 | Page | Owns |
 |---|---|
 | [[Radiance Serve Options]] | every serve flag, grouped, with defaults and the law behind each knob |
 | [[Radiance HTTP API]] | endpoints, request fields, streaming, tool calls, refusal rules |
 | [[Radiance Conversion]] | `rad-convert`, the recipe language, quantisers, imports, reuse, `rad-export`, KL scoring |
 | [[Radiance Model Families]] | which plugin claims which model, and the conversion command per family |
 | [[Radiance Plugins]] | kernel and quantiser plugins: ABI, rows, hooks, encodings, build and checks |
 | [[Radiance Architecture Plugins]] | declaring a model: weights, buffers, KV groups, ops, step, probe |
 | [[Radiance Tools]] | `rad-info`, `rad-kbench`, `rad-tune`, `rad-schemas`, `rad-tokcheck`, harnesses |
 | [[Radiance Operations]] | the container contract, host prerequisites, readiness, ledgers, restarts |
 | [[frank-archon]] | the fleet's live radiance endpoint: shape, Eras, rollback, measured laws |
 ## Sources
 - `README.md`, `docs/GUIDE.md` §1–§4, `docs/TOOLS.md`, `docs/PLUGIN.md`, `docs/DOCKER.md`,
  `docs/ARCHITECTURES.md`, `docs/OPS.md`, `spec.md` §2–§4, `abi/*.h`, `core/config.cpp`,
  `core/rad_core.h` at commit `8da043c` (codeberg.org/StillDeadcode/radiance).