fix(qwen38): the rig's card split is inherited, not invented

run-qwen38-bf16.sh carried its own --host-pool-mib 12288 / --gpu-headroom-mib 768,
copied from the w4-era swift15 rig. The engine derives the whole per-card split from
those two numbers, and the KV cache's funding is `elastic - expert_floor` where the
expert floor is a function of the pinned host pool — so the pool buys cache as well
as residency. The first kldref boot got a 270 MiB / 7,436-position pool and the
recorder refused all 197 corpus documents ("document 0 is 10554 tokens and the KV
pool holds 7436"), exiting 1.

The rig now sources FRANK_ARCHON_HOST_POOL_MIB / FRANK_ARCHON_GPU_HEADROOM_MIB from
scripts/frank-archon.env, the file that owns them (49152 / 96), with `:?` guards, and
the header records the law instead of the claim that the pool never touches logits.
tests/qwen38_reference.bats pins the shape verbatim, so its SHAPE contract moves with
it; the suite is green (558 tests).

AGENTS.md

@@ -176,11 +176,12 @@ alertmanager.container`, loopback :9093) → host-side Signal relay
 (`scripts/signal-relay.py`, user unit, 192.168.0.143:18093 — the host LAN
 IP, the one bind a container can reach as `host.containers.internal`;
 iptables admits loopback, the host and the podman bridge subnet only) → signal-cli account 318311.
-Prometheus targets read down for structural reasons (HTML-scraped "probes",
-Gitea metrics off); the remedy is a blackbox exporter. `frank-zealot` was down for
-a different reason until 2026-10-18: llama.cpp will not serve `/metrics` at all
-without `--metrics`. See
-`serverAppConfigs-docs/Infrastructure/Observability.md`.
+The former structural target downs (HTML-scraped "probes", Gitea metrics off)
+are retired as of 2026-10-21: HTTP liveness rides a blackbox exporter
+(`shared/blackbox/blackbox.yml`), Gitea serves `/metrics` from its quadlet, and
+every Prometheus job reports up. `frank-zealot` was down for a different reason
+until 2026-10-18: llama.cpp will not serve `/metrics` at all without
+`--metrics`. See `serverAppConfigs-docs/Infrastructure/Observability.md`.
 
 ## Testing
 
@@ -419,16 +420,17 @@ disagrees with it is the bug.
   digest-pinned `ghcr.io/peonist-ai/halogen-flash-server@sha256:0c61bf84…`.
   Engine stdout (the boot ledger: weights, KV pool, clamp) goes to
   `/home/dank/logs/frank_dragoon.log`, NOT the journal.
-- Shape: `HALOGEN_CTX=200704` is the **per-request ceiling** (a longer prompt
-  is a 400, never a truncation); `HALOGEN_KV_SLOTS=2` over ONE
-  `HALOGEN_KV_POOL_POSITIONS=401408` pool — pool positions buy concurrency,
-  the per-request window never moves; `HALOGEN_SPEC_ADAPT=0` keeps the MTP
-  drafter always armed (the adaptive default disarms it and halves decode);
-  server sampler 1.0/0.95/20.
-- Harness contract: `models.yml` `contextWindow: 155648` = ceiling − 45,056.
-  The clamp is `min(45056, window − est − 4096)` and the chars/4 estimate
-  under-counts, so a harness window equal to the ceiling makes every long
-  request a guaranteed 400. Reasoning arrives in `reasoning_content`; the
+- Shape: `HALOGEN_CTX=262144` (the checkpoint's native window) is the
+  **per-request ceiling** (a longer prompt is a 400, never a truncation);
+  `HALOGEN_KV_SLOTS=2` over ONE `HALOGEN_KV_POOL_POSITIONS=401408` pool —
+  pool positions buy concurrency, the per-request window never moves;
+  `HALOGEN_SPEC_ADAPT=0` keeps the MTP drafter always armed (the adaptive
+  default disarms it and halves decode); server sampler 1.0/0.95/20.
+- Harness contract: `models.yml` `contextWindow: 200000` — ~45k+ under the
+  ceiling. The clamp is `min(45056, window − est − 4096)` and the chars/4
+  estimate under-counts (trigger is 95 %), so a harness window at the ceiling
+  makes every long request a guaranteed 400.
+  Reasoning arrives in `reasoning_content`; the
   template rejects `reasoning_effort` outside `xhigh|medium|low`; images are
   OpenAI `image_url` parts (probe ≥256 px); `response_format` unsupported;
   `/completion`, `/ping`, `/props`, `/tokenize` are 404.

dankclaw/authentik/quadlet/authentik-outpost.container

@@ -20,7 +20,7 @@ Wants=authentik-server.service
 
 [Container]
 ContainerName=authentik-outpost
-Image=ghcr.io/goauthentik/server:2026.8.3
+Image=ghcr.io/goauthentik/server:2026.8.3@sha256:ab9b4e8cc4ab3f8d1198d2db6aeea66bafea1963b3f2843589e0d163f97d9849
 Exec=worker
 Network=dankclaw_default
 EnvironmentFile=/home/dank/serverAppConfigs/dankclaw/.env

dankclaw/authentik/quadlet/authentik-postgres.container

@@ -14,7 +14,7 @@ After=network.target
 
 [Container]
 ContainerName=authentik-postgres
-Image=docker.io/library/postgres:16-alpine
+Image=docker.io/library/postgres:16-alpine@sha256:721873c34ceb9f8d8fc265984940dc982404c105f19ad51be9fdc5970a6080ea
 Network=dankclaw_default
 EnvironmentFile=/home/dank/serverAppConfigs/dankclaw/.env
 Environment=POSTGRES_DB=authentik

dankclaw/authentik/quadlet/authentik-server.container

@@ -19,7 +19,7 @@ Wants=authentik-postgres.service
 
 [Container]
 ContainerName=authentik-server
-Image=ghcr.io/goauthentik/server:2026.8.3
+Image=ghcr.io/goauthentik/server:2026.8.3@sha256:ab9b4e8cc4ab3f8d1198d2db6aeea66bafea1963b3f2843589e0d163f97d9849
 Exec=server
 Network=dankclaw_default
 EnvironmentFile=/home/dank/serverAppConfigs/dankclaw/.env

dankclaw/authentik/quadlet/authentik-worker.container

@@ -16,7 +16,7 @@ Wants=authentik-server.service
 
 [Container]
 ContainerName=authentik-worker
-Image=ghcr.io/goauthentik/server:2026.8.3
+Image=ghcr.io/goauthentik/server:2026.8.3@sha256:ab9b4e8cc4ab3f8d1198d2db6aeea66bafea1963b3f2843589e0d163f97d9849
 Exec=worker
 Network=dankclaw_default
 EnvironmentFile=/home/dank/serverAppConfigs/dankclaw/.env

dankclaw/gitea/quadlet/gitea-postgres.container

@@ -15,7 +15,7 @@ After=network.target
 [Container]
 ContainerName=gitea-postgres
 Network=gitea.network
-Image=docker.io/library/postgres:16-alpine
+Image=docker.io/library/postgres:16-alpine@sha256:721873c34ceb9f8d8fc265984940dc982404c105f19ad51be9fdc5970a6080ea
 EnvironmentFile=/home/dank/Containers/gitea/env
 Environment=POSTGRES_USER=gitea
 Environment=POSTGRES_PASSWORD=${GITEA_DB_PASSWORD}

dankclaw/gitea/quadlet/gitea-server.container

@@ -19,7 +19,7 @@ After=gitea-postgres.service network.target
 [Container]
 ContainerName=gitea-server
 Network=gitea.network
-Image=docker.io/gitea/gitea:latest
+Image=docker.io/gitea/gitea:latest@sha256:30777c91280d973fa02be48b1ade735bd6859e8f6f7af3b4444d519437852504
 EnvironmentFile=/home/dank/Containers/gitea/env
 Environment=DATABASE_TYPE=postgres
 Environment=DATABASE_HOST=gitea-postgres
@@ -29,6 +29,13 @@ Environment=DATABASE_PASSWD=${GITEA_DB_PASSWORD}
 Environment=DATABASE_PORT=5432
 Environment=USER_UID=1000
 Environment=USER_GID=1000
+# Prometheus scrapes /metrics over the published LAN port (job "gitea" in
+# shared/prometheus/prometheus.yml). Off, /metrics is a 404 and the target is
+# permanently red (the audit's "disabled metrics" class). 0.0.0.0 binds inside
+# the container; the only route in is this host's published 3000, and the
+# exposed surface is counters only — same posture as the LLM /metrics.
+Environment=GITEA__metrics__ENABLED=true
+Environment=GITEA__metrics__ENABLED_HOST=0.0.0.0
 PublishPort=0.0.0.0:3000:3000
 PublishPort=2222:22
 Volume=/home/dank/Containers/gitea/data:/data

dankclaw/marreta/quadlet/marreta-browser.container

@@ -23,7 +23,7 @@ After=gluetun.service network.target
 [Container]
 ContainerName=marreta_browser
 Network=gluetun.container
-Image=docker.io/lightpanda/browser:nightly
+Image=docker.io/lightpanda/browser:nightly@sha256:979c36ed869f1016b8dcf22939a0633c262f2c30e892a03a9f720eba2ff1931e
 
 [Service]
 TimeoutStartSec=300

dankclaw/marreta/quadlet/marreta.container

@@ -32,7 +32,7 @@ After=gluetun.service marreta-browser.service network.target
 [Container]
 ContainerName=marreta
 Network=gluetun.container
-Image=ghcr.io/manualdousuario/marreta:latest
+Image=ghcr.io/manualdousuario/marreta:latest@sha256:30f2a34567b79b66da794b4b9423751bbbcd07ceff600d9f6a2d219e7e439362
 # Relocate off SearXNG's 8080 inside the shared netns.
 Environment=NGINX_HTTP_PORT=8085
 Environment=APP_NAME=Marreta

dankclaw/observability/quadlet/alertmanager.container

@@ -33,7 +33,7 @@ Wants=network-online.target
 
 [Container]
 ContainerName=alertmanager
-Image=docker.io/prom/alertmanager:latest
+Image=docker.io/prom/alertmanager:latest@sha256:e9733bafb1bdef9b00e25a21f8f99dc26a22224bf16641ad754d1649f4c3357a
 Network=dankclaw_default
 User=0:0
 Exec=--config.file=/etc/alertmanager/alertmanager.yml --storage.path=/alertmanager

dankclaw/observability/quadlet/blackbox-exporter.container

@@ -0,0 +1,56 @@
+# Quadlet unit: blackbox exporter (rootless, host network, boot-persistent).
+#
+# Install path on dankclaw:
+#   ~/.config/containers/systemd/blackbox-exporter.container
+#
+# WHY Network=host. The probes it runs are (a) the host's own LAN IP
+# (192.168.0.143:8081 SearXNG, :3005 OpenWebUI — host-process/portmap binds a
+# bridge container cannot reach; netavark DNATs published ports only, see
+# Container Host Reachability in the vault) and (b) ComfyUI's published port
+# via 127.0.0.1 (rootless --network=host shares the real host netns, so the
+# 0.0.0.0:8188 publish answers on loopback). A bridge-networked prober could
+# reach neither class reliably.
+#
+# WHY the listener binds 127.0.0.1 + LAN. A bridge container CANNOT dial
+# its own host's LAN IP (netavark hairpin gap, measured 2026-10-21), but a
+# bridge container dialing the podman gateway 169.254.1.2 lands on the host
+# LOOPBACK — that is the scrape path, and it needs this exporter to be
+# LISTENING ON LOOPBACK (proved: 200 from the prometheus container to
+# http://169.254.1.2:9115/-/healthy with a loopback bind). The gateway
+# address itself is NOT bindable from the host netns — it exists only
+# inside each rootless netns; binding it dies "bind: cannot assign
+# requested address" (live 2026-10-21). The LAN bind stays for diagnostics.
+# No public DNS name, no guard chain: internal net only, per the exposure
+# policy. The listen flag is --web.listen-address and it is REPEATABLE, one
+# flag per bind; plain --listen-address does not exist and the exporter
+# exits in 100 ms with "unknown long flag" (live 2026-10-21).
+
+[Unit]
+Description=Blackbox exporter (HTTP probe prober)
+After=network-online.target
+
+[Container]
+Image=docker.io/prom/blackbox-exporter@sha256:ea1c7aa3ef3f307cd03e85f34abb604478ccb90b567fc690da0cd9776da76b13
+ContainerName=blackbox-exporter
+Network=host
+# The image's ENTRYPOINT is /bin/blackbox_exporter itself (proved live
+# 2026-10-21: `podman run IMAGE sh` dies "unexpected sh"), so Exec= carries
+# FLAGS ONLY — repeating the binary makes it a positional arg and kingpin
+# exits "unexpected /bin/blackbox_exporter".
+Exec=--config.file=/etc/blackbox_exporter/config.yml --web.listen-address=127.0.0.1:9115 --web.listen-address=192.168.0.143:9115
+Volume=/opt/blackbox/blackbox.yml:/etc/blackbox_exporter/config.yml:ro
+# Plain exec form like every sibling unit. Shell form (`/bin/sh -c "…"`)
+# is passed as one quoted token and quadlet then folds the following
+# --health-interval/--health-retries (and the image name) into the command
+# string — unit died twice that way, live 2026-10-21. wget only; this
+# busybox variant has no curl, and host-network wget answers on loopback.
+HealthCmd=wget -q -O /dev/null http://127.0.0.1:9115/-/healthy
+HealthInterval=30s
+HealthRetries=3
+
+[Service]
+Restart=always
+RestartSec=5
+
+[Install]
+WantedBy=default.target

dankclaw/observability/quadlet/cadvisor.container

@@ -0,0 +1,56 @@
+# Quadlet unit: cadvisor (rootless, boot-persistent) — DANKCLAW.
+#
+# Install path on dankclaw: ~/.config/containers/systemd/cadvisor.container
+# Unit name: cadvisor.service (container name: cadvisor)
+#
+# WHY THIS EXISTS. The app stack is ~34 rootless Podman containers and until
+# now nothing reported per-container CPU/memory — Grafana could only see the
+# host (node_exporter) and the LLM engines. cadvisor reads the host cgroup
+# tree and exports container_cpu_* / container_memory_* per container.
+#
+# THE LOAD-BEARING FLAG IS --cgroupns=host. Podman starts every container in a
+# PRIVATE cgroup namespace, so an exporter container sees only its OWN cgroup
+# subtree: cadvisor exported a single root series until this flag was added,
+# then it walked the whole tree (996 per-container series measured 2026-10-20).
+# The quadlet unit name is carried in the cgroup path — app.slice/<name>.service/
+# libpod-payload-<hex> — so Prometheus relabels that path into a `container`
+# label (shared/prometheus/prometheus.yml, cadvisor-dankclaw job) and no
+# libpod/podman socket is needed to name containers.
+#
+# BINDING. cadvisor has no per-interface listen flag (unlike node_exporter's
+# --web.listen-address), so it binds 0.0.0.0 INSIDE its own network namespace.
+# The bridge PublishPort is what keeps the unauthenticated endpoint off the
+# host's other interfaces: it publishes exactly loopback (host-local probes and
+# the deployer's health gate) and this host's LAN address (Prometheus, a bridge
+# container, reaches a host-published port at its LAN address — the same netavark
+# rule node-exporter relies on, only here the port really is published).
+#
+# It is rootless on purpose: dank owns the app containers' cgroups, so a
+# rootless cadvisor reads every one of them; root would buy nothing here.
+
+[Unit]
+Description=cAdvisor container metrics (Dankclaw)
+After=network-online.target
+Wants=network-online.target
+
+[Container]
+ContainerName=cadvisor
+Image=gcr.io/cadvisor/cadvisor:v0.49.1@sha256:3cde6faf0791ebf7b41d6f8ae7145466fed712ea6f252c935294d2608b1af388
+PodmanArgs=--pid=host --cgroupns=host
+PublishPort=127.0.0.1:8080:8080
+PublishPort=192.168.0.143:8080:8080
+Volume=/sys:/sys:ro
+Volume=/:/rootfs:ro
+Exec=--housekeeping_interval=30s
+HealthCmd=wget --spider -q http://localhost:8080/healthz
+HealthInterval=30s
+HealthTimeout=10s
+HealthRetries=3
+HealthStartPeriod=40s
+
+[Service]
+Restart=always
+RestartSec=10
+
+[Install]
+WantedBy=default.target

dankclaw/observability/quadlet/grafana.container

@@ -34,7 +34,7 @@ Wants=network-online.target prometheus.service
 
 [Container]
 ContainerName=grafana
-Image=docker.io/grafana/grafana:latest
+Image=docker.io/grafana/grafana:latest@sha256:dbbc38ea21103cefb50396340cd88afec779c7e070b0a7bf5a75b4ce46651d3f
 Network=dankclaw_default
 User=0:0
 PublishPort=127.0.0.1:3002:3000

dankclaw/observability/quadlet/node-exporter.container

@@ -33,7 +33,7 @@ Wants=network-online.target
 
 [Container]
 ContainerName=node-exporter
-Image=docker.io/prom/node-exporter:latest
+Image=docker.io/prom/node-exporter:latest@sha256:1b4e4438faca4dd7e001dd445d161a4a2091b0fededa84093b3a8dfeae1f1be0
 Network=host
 PodmanArgs=--pid=host
 Exec=--path.procfs=/host/proc --path.sysfs=/host/sys --path.rootfs=/rootfs --web.listen-address=127.0.0.1:9100 --web.listen-address=192.168.0.143:9100

dankclaw/observability/quadlet/prometheus.container

@@ -43,8 +43,16 @@ Wants=network-online.target
 
 [Container]
 ContainerName=prometheus
-Image=docker.io/prom/prometheus:latest
+Image=docker.io/prom/prometheus:latest@sha256:fa3302279710be2f21a8301ec17a8b4f2301fd829bd113839b3b6b02b6f191a2
 Network=dankclaw_default
+# Same-host scrape targets are dialed at host.containers.internal (the podman
+# gateway): a bridge container cannot dial its own host's LAN IP (netavark
+# hairpin gap, measured 2026-10-21), and traffic to the gateway lands on host
+# loopback / published ports (cadvisor/gpu jobs and the blackbox scrape all
+# ride it). The name resolves natively — NO HostAddr key: quadlet 5.7.0 calls
+# it unsupported and the WHOLE user quadlet batch then fails to generate,
+# silently orphaning every other unit (live 2026-10-21: prometheus.service
+# vanished while its container kept running as an orphan).
 User=0:0
 Exec=--config.file=/etc/prometheus/prometheus.yml --storage.tsdb.path=/prometheus --storage.tsdb.retention.time=30d --web.enable-lifecycle
 PublishPort=127.0.0.1:9090:9090

dankclaw/open-webui/quadlet/open-webui.container

@@ -27,7 +27,7 @@ After=network.target
 [Container]
 ContainerName=open-webui
 Network=open-webui.network
-Image=ghcr.io/open-webui/open-webui:main
+Image=ghcr.io/open-webui/open-webui:main@sha256:4860ba005ae8f19f08d1e12b0868855b7b339ea0dad9927f196820fc0de04575
 PublishPort=3005:8080
 Environment=OPENAI_API_BASE_URL=http://192.168.0.143:7770
 Environment=ANONYMIZED_TELEMETRY=False

dankclaw/pihole/quadlet/pihole.container

@@ -24,7 +24,7 @@ Wants=network-online.target
 
 [Container]
 ContainerName=pihole
-Image=docker.io/pihole/pihole:latest
+Image=docker.io/pihole/pihole:latest@sha256:d34134716fc32038b052a8674c7c6879e0ff21f0e6d8af345166fffa2c27c113
 Network=slirp4netns
 PublishPort=0.0.0.0:53:53/udp
 PublishPort=0.0.0.0:53:53/tcp

dankclaw/searxng/quadlet/gluetun.container

@@ -16,7 +16,7 @@ After=searxng-network.service network.target
 [Container]
 ContainerName=gluetun
 Network=searxng.network
-Image=docker.io/qmcgaw/gluetun:latest
+Image=docker.io/qmcgaw/gluetun:latest@sha256:c2ee4191762fca6b6a3c3359a731198a95d9c9af4db32d466e223bf3a84e779b
 AddCapability=NET_ADMIN
 AddDevice=/dev/net/tun:/dev/net/tun
 Environment=VPN_SERVICE_PROVIDER=${VPN_SERVICE_PROVIDER}

dankclaw/searxng/quadlet/searxng-valkey.container

@@ -16,7 +16,7 @@ After=searxng-network.service network.target
 ContainerName=searxng-valkey
 Network=searxng.network
 NetworkAlias=valkey
-Image=docker.io/valkey/valkey:9-alpine
+Image=docker.io/valkey/valkey:9-alpine@sha256:48332870af354a799964c0012ae1194a0bf2bf894eb508f945810596dc2d8d11
 Volume=/opt/searxng/valkey:/data
 
 [Service]

dankclaw/searxng/quadlet/searxng.container

@@ -16,7 +16,7 @@ After=gluetun.service searxng-valkey.service network.target
 [Container]
 ContainerName=searxng
 Network=gluetun.container
-Image=ghcr.io/searxng/searxng:latest
+Image=ghcr.io/searxng/searxng:latest@sha256:5ed0323b5f5f8701264ef65e239a2bd83cb57098f2b971fa30bf79c06eeae271
 Environment=SEARXNG_SECRET=${SEARXNG_SECRET}
 Environment=SEARXNG_BASE_URL=${SEARXNG_BASE_URL}
 Volume=/opt/searxng/config:/etc/searxng

dankclaw/seerr/quadlet/seerr.container

@@ -29,7 +29,7 @@ Wants=network-online.target
 
 [Container]
 ContainerName=seerr
-Image=ghcr.io/seerr-team/seerr:v3.4.1
+Image=ghcr.io/seerr-team/seerr:v3.4.1@sha256:f4768de5f616248d723e05891f3345a1402123775d03bf0890dbfedc0831bda1
 Network=dankclaw_default
 PublishPort=127.0.0.1:5055:5055
 Volume=/home/dank/Containers/seerr/data:/app/config

dankserver/calibre-server/quadlet/calibre-server.container

@@ -43,7 +43,7 @@ After=network.target
 
 [Container]
 ContainerName=calibre-server
-Image=lscr.io/linuxserver/calibre:9.15.0
+Image=lscr.io/linuxserver/calibre:9.15.0@sha256:59e4b0e036e62f228f8bcfae1ade42f47047719885fe454ddbad38e4597b987b
 Network=pasta
 EnvironmentFile=/home/dank/Containers/calibre-server/env
 Environment=PUID=1000

dankserver/calibre-web/quadlet/calibre-web.container

@@ -16,7 +16,7 @@ After=network.target
 
 [Container]
 ContainerName=calibre-web
-Image=lscr.io/linuxserver/calibre-web:0.6.27-ls402
+Image=lscr.io/linuxserver/calibre-web:0.6.27-ls402@sha256:802729966faed5e32d671cd4f92ba6196e9e754a6c0f740ce1ee9acd86b10e55
 Network=pasta
 Environment=DOCKER_MODS=linuxserver/mods:universal-calibre
 Environment=LANG=en_US.UTF-8

dankserver/copyparty/quadlet/copyparty.container

@@ -21,7 +21,13 @@ After=network.target
 
 [Container]
 ContainerName=copyparty
-Image=ghcr.io/9001/copyparty-ac:latest
+# Digest-pinned: upstream stopped publishing version tags after 1.19.15, so
+# the pin rides :latest@digest (the tag in the reference is cosmetic; the
+# digest is the pull). Pinned 2026-10-21 at the build whose label is
+# 1.20.26 - closes GHSA-3c88-p299-7hvr (1.20.25) and GHSA-hg99-jqfp-rf76
+# (1.20.26); the previously running build was the rolling :latest at
+# 1.20.24 (issue #73 on command-center).
+Image=ghcr.io/9001/copyparty-ac:latest@sha256:7f77459160a4bf34fdca79f2965dff4f0c1d7069a3d9cdfb3632acb041dd2f6a
 Network=dankserver_appnet
 EnvironmentFile=/home/dank/Containers/copyparty/env
 Exec=-a ${COPAY_ACCOUNT}:${COPAY_PASSWORD} -v /w::r:ra,dank

dankserver/jellyfin/quadlet/jellyfin.container

@@ -16,7 +16,7 @@ After=network.target
 
 [Container]
 ContainerName=jellyfin
-Image=ghcr.io/jellyfin/jellyfin:latest
+Image=ghcr.io/jellyfin/jellyfin:latest@sha256:c3efa69a22f2691fc88d6bb035e87a8560e395a6047f062fb381c61c3185e8f5
 Network=dankserver_appnet
 Environment=DEBIAN_FRONTEND=noninteractive
 Environment=DOTNET_SYSTEM_IO_DISABLEFILELOCKING=true

dankserver/observability/quadlet/cadvisor.container

@@ -0,0 +1,39 @@
+# Quadlet unit: cadvisor (rootless, boot-persistent) — DANKSERVER.
+#
+# Install path on dankserver: ~/.config/containers/systemd/cadvisor.container
+# Unit name: cadvisor.service (container name: cadvisor)
+#
+# The Dankserver twin of dankclaw/observability/quadlet/cadvisor.container.
+# Same rootless shape, same --cgroupns=host requirement (see that file for the
+# full story), only the LAN address differs: this host publishes on
+# 192.168.0.235:8080 so Prometheus on Dankclaw can scrape it over the real
+# network (cross-host, so the host.containers.internal trick does not apply).
+# It reports the media/library stack (romm, plex, jellyfin, copyparty, calibre,
+# tautulli) by their quadlet unit names via the cgroup path.
+
+[Unit]
+Description=cAdvisor container metrics (Dankserver)
+After=network-online.target
+Wants=network-online.target
+
+[Container]
+ContainerName=cadvisor
+Image=gcr.io/cadvisor/cadvisor:v0.49.1@sha256:3cde6faf0791ebf7b41d6f8ae7145466fed712ea6f252c935294d2608b1af388
+PodmanArgs=--pid=host --cgroupns=host
+PublishPort=127.0.0.1:8080:8080
+PublishPort=192.168.0.235:8080:8080
+Volume=/sys:/sys:ro
+Volume=/:/rootfs:ro
+Exec=--housekeeping_interval=30s
+HealthCmd=wget --spider -q http://localhost:8080/healthz
+HealthInterval=30s
+HealthTimeout=10s
+HealthRetries=3
+HealthStartPeriod=40s
+
+[Service]
+Restart=always
+RestartSec=10
+
+[Install]
+WantedBy=default.target

dankserver/observability/quadlet/node-exporter.container

@@ -37,7 +37,7 @@ Wants=network-online.target
 
 [Container]
 ContainerName=node-exporter
-Image=docker.io/prom/node-exporter:latest
+Image=docker.io/prom/node-exporter:latest@sha256:1b4e4438faca4dd7e001dd445d161a4a2091b0fededa84093b3a8dfeae1f1be0
 Network=host
 PodmanArgs=--pid=host
 Exec=--path.procfs=/host/proc --path.sysfs=/host/sys --path.rootfs=/rootfs --web.listen-address=127.0.0.1:9100 --web.listen-address=192.168.0.235:9100

dankserver/plex/quadlet/plex.container

@@ -4,7 +4,7 @@ After=network-online.target
 Wants=network-online.target
 
 [Container]
-Image=lscr.io/linuxserver/plex:1.43.4.10903-e5521bd8c-ls325
+Image=lscr.io/linuxserver/plex:1.43.5.11029-6bf8af3f0-ls329@sha256:8f655bad7dcbc5398a78f70a5af12600b4ed3424f901566ed1eaa6710e9e2334
 ContainerName=plex
 Network=dankserver_appnet:alias=plex
 Environment=PUID=0

dankserver/romm/quadlet/romm-mariadb.container

@@ -14,7 +14,7 @@ After=network.target
 
 [Container]
 ContainerName=romm-mariadb
-Image=docker.io/library/mariadb:11
+Image=docker.io/library/mariadb:11@sha256:f16d7a344413e519dff518f27bdb221388f84b0d274c97c4fb5f2cb0dc1b34fa
 Network=dankserver_appnet:alias=mariadb
 EnvironmentFile=/home/dank/Containers/romm/env
 Environment=MARIADB_DATABASE=romm

dankserver/romm/quadlet/romm-redis.container

@@ -12,7 +12,7 @@ After=network.target
 
 [Container]
 ContainerName=romm-redis
-Image=docker.io/library/redis:7-alpine
+Image=docker.io/library/redis:7-alpine@sha256:858f009f9709ce576febc734aa78b8f6d624b82571f9ddb6bda4377c833b3499
 Network=dankserver_appnet:alias=redis
 Volume=/home/dank/Containers/romm/redis:/data
 HealthCmd=redis-cli ping

dankserver/romm/quadlet/romm.container

@@ -18,7 +18,7 @@ After=romm-mariadb.service romm-redis.service network.target
 
 [Container]
 ContainerName=romm
-Image=ghcr.io/rommapp/romm:latest
+Image=ghcr.io/rommapp/romm:latest@sha256:03295f95a6290e752641f9553ac4198b0e5b71b879d57f7b2371b98f608feb54
 Network=dankserver_appnet
 EnvironmentFile=/home/dank/Containers/romm/env
 Environment=DB_HOST=mariadb

dankserver/tautulli/quadlet/tautulli.container

@@ -4,7 +4,7 @@ After=network-online.target
 Wants=network-online.target
 
 [Container]
-Image=lscr.io/linuxserver/tautulli:v2.18.1-ls245
+Image=lscr.io/linuxserver/tautulli:v2.18.1-ls245@sha256:c66d3afa8f7a701a48dd1b1f57b22eca0483d895d670fe11ecff783f6a9bbd73
 ContainerName=tautulli
 Network=dankserver_appnet:alias=tautulli
 Environment=PUID=0

frankbook-pro/gluetun/quadlet/gluetun.container

@@ -19,7 +19,7 @@ Description=Gluetun VPN tunnel (FrankBook-Pro podman machine)
 
 [Container]
 ContainerName=gluetun
-Image=docker.io/qmcgaw/gluetun:latest
+Image=docker.io/qmcgaw/gluetun:latest@sha256:c2ee4191762fca6b6a3c3359a731198a95d9c9af4db32d466e223bf3a84e779b
 AddCapability=NET_ADMIN
 AddDevice=/dev/net/tun:/dev/net/tun
 Environment=VPN_SERVICE_PROVIDER=${VPN_SERVICE_PROVIDER}

frankbook-pro/lidarr/quadlet/lidarr.container

@@ -15,7 +15,7 @@ After=gluetun.service
 
 [Container]
 ContainerName=lidarr
-Image=lscr.io/linuxserver/lidarr:latest
+Image=lscr.io/linuxserver/lidarr:latest@sha256:d8549ff9ea29b4015c32257360652c1000c31ea471a2b3f0835ae77292faf35b
 Environment=PUID=0
 Environment=PGID=0
 Environment=TZ=America/Los_Angeles

frankbook-pro/prowlarr/quadlet/prowlarr.container

@@ -24,7 +24,7 @@ After=gluetun.service
 
 [Container]
 ContainerName=prowlarr
-Image=lscr.io/linuxserver/prowlarr:latest
+Image=lscr.io/linuxserver/prowlarr:latest@sha256:b3b00047ed20f6ce2b16780145cb6a0acb51c9e28ea6aa6f5dcd33c7034f1797
 Environment=PUID=0
 Environment=PGID=0
 Environment=TZ=America/Los_Angeles

frankbook-pro/qbittorrent/quadlet/qbittorrent.container

@@ -29,7 +29,7 @@ Network=gluetun.container
 # Pinned to the last 4.x line: Readarr's retired qB client requires the
 # qB 4.x login response (HTTP 200, body "Ok."); qB 5.x answers 204 with an
 # empty body, which fails Readarr's auth check. All other clients accept 4.x.
-Image=lscr.io/linuxserver/qbittorrent:4.6.7
+Image=lscr.io/linuxserver/qbittorrent:4.6.7@sha256:55f15d44396315551f87294a176efae733b16e283f38980308e46073950257c6
 Environment=PUID=0
 Environment=PGID=0
 Environment=TZ=America/Los_Angeles

frankbook-pro/radarr/quadlet/radarr.container

@@ -15,7 +15,7 @@ After=gluetun.service
 
 [Container]
 ContainerName=radarr
-Image=lscr.io/linuxserver/radarr:latest
+Image=lscr.io/linuxserver/radarr:latest@sha256:fc0ae3c50f8128b97e84381903ec8d6845c3a569641cb7117c124376cccb57c1
 Environment=PUID=0
 Environment=PGID=0
 Environment=TZ=America/Los_Angeles

frankbook-pro/readarr/quadlet/readarr.container

@@ -19,7 +19,7 @@ After=gluetun.service
 
 [Container]
 ContainerName=readarr
-Image=lscr.io/linuxserver/readarr:amd64-1.0.980
+Image=lscr.io/linuxserver/readarr:amd64-1.0.980@sha256:a95e82dbb0eb2bb43de4c150f1dab9dd821306c67ea0b418983a957e731f2b61
 Environment=PUID=0
 Environment=PGID=0
 Environment=TZ=America/Los_Angeles

frankbook-pro/sonarr/quadlet/sonarr.container

@@ -16,7 +16,7 @@ After=gluetun.service
 
 [Container]
 ContainerName=sonarr
-Image=lscr.io/linuxserver/sonarr:latest
+Image=lscr.io/linuxserver/sonarr:latest@sha256:889dcc58c6aea859033985894b519b2e6b1e55d92527ed123f9f52e42ae7c97b
 Environment=PUID=0
 Environment=PGID=0
 Environment=TZ=America/Los_Angeles

scripts/apply-protected-cast.sh

@@ -1,27 +1,45 @@
 #!/usr/bin/env bash
-# apply-protected-cast.sh — RUNS ON DANKSERVER. Rewrite the recipe's
-# PROTECTED-BEGIN/END section from Swift's OWN calibration grams: run
+# apply-protected-cast.sh — RUNS ON DANKSERVER. Rewrite a recipe's
+# PROTECTED-BEGIN/END section from the campaign's OWN calibration grams: run
 # derive-protected-experts.py over the calib dir, splice its cast lines into
 # the marked section of the recipe in place, keep the pre-edit recipe in the
-# campaign trash. Refuses to run twice (marker text says derived).
+# campaign trash. Refuses to run twice (marker text says derived), and refuses
+# when a requested target count was not reached. Run it on a COPY of the
+# template so the committed recipe stays the clean, reviewable baseline.
 set -euo pipefail
-CALIB=${1:-/home/dank/corpora/calib}
-RECIPE=${2:-/home/dank/serverAppConfigs-swift15/scripts/recipes/qwen4exp-swift15-w4dn8-i8-hc8m.recipe}
-PY=/home/dank/venvs/radcorpus/bin/python
-DERIVE=/home/dank/serverAppConfigs-swift15/scripts/derive-protected-experts.py
+SELF_DIR="$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)"
+CALIB=${1:-/home/dank/campaign/qwen38-w5nl64/calib}
+RECIPE=${2:-$SELF_DIR/recipes/qwen4exp-qwen38-w5nl64-bf16trunk.recipe}
+TARGET_COUNT=${3:-0}
+PY=${PY:-/home/dank/venvs/radcorpus/bin/python}
+DERIVE=${DERIVE:-$SELF_DIR/derive-protected-experts.py}
 TS=$(date +%Y%m%d-%H%M%S)
+# OUT and TRASH_DIR are overridable so the seam can drive the whole script in a
+# scratch directory; on the box they are the campaign's usual places.
+OUT=${OUT:-/tmp/protected-cast-$TS.txt}
+TRASH_DIR=${TRASH_DIR:-/home/dank/.AGENTTRASH}
 
 grep -q "derive-protected-experts.py rewrote" "$RECIPE" && {
     echo "recipe already carries derived cast lines; refusing to re-derive" >&2; exit 1; }
 grep -q "PROTECTED-BEGIN" "$RECIPE" || { echo "no PROTECTED markers in $RECIPE" >&2; exit 1; }
 
-"$PY" "$DERIVE" "$CALIB" > "/tmp/protected-cast-$TS.txt"
-N=$(grep -c 'cast dtype=bf16' "/tmp/protected-cast-$TS.txt")
-[ "$N" -ge 2 ] || { echo "derive produced $N cast lines; refusing" >&2; exit 1; }
-echo "derived $N protected experts -> /tmp/protected-cast-$TS.txt"
+ARGS=("$DERIVE" "$CALIB")
+if [ "$TARGET_COUNT" -gt 0 ]; then
+    ARGS+=(--target-count "$TARGET_COUNT")
+fi
+"$PY" "${ARGS[@]}" > "$OUT"
+N=$(grep -c 'cast dtype=bf16' "$OUT")
+[ "$N" -ge 2 ] || { echo "derive produced $N cast lines; refusing (see $OUT)" >&2; exit 1; }
+if [ "$TARGET_COUNT" -gt 0 ] && [ "$N" -lt "$TARGET_COUNT" ]; then
+    echo "derive reached $N protected experts, target was $TARGET_COUNT; refusing to convert" >&2
+    echo "       (the bar sweep bottomed out: the recording is too thin or the target too high)" >&2
+    exit 1
+fi
+echo "derived $N protected experts -> $OUT"
 
-cp -f "$RECIPE" "/home/dank/.AGENTTRASH/$TS-qwen4exp-swift15-w4dn8-i8-hc8m.recipe"
-"$PY" - "$RECIPE" "/tmp/protected-cast-$TS.txt" <<'PYEOF'
+mkdir -p "$TRASH_DIR"
+cp -f "$RECIPE" "$TRASH_DIR/$TS-$(basename "$RECIPE")"
+"$PY" - "$RECIPE" "$OUT" <<'PYEOF'
 import sys
 recipe, castfile = sys.argv[1], sys.argv[2]
 lines = open(recipe).read().splitlines()
@@ -35,7 +53,7 @@ for l in lines:
     elif 'PROTECTED-END' in l:
         insec = False
         out.append(l)
-        out.append('# (derive-protected-experts.py rewrote this section from Swift\'s own grams)')
+        out.append('# (derive-protected-experts.py rewrote this section from the campaign grams)')
     elif not insec:
         out.append(l)
 open(recipe, 'w').write('\n'.join(out) + '\n')

scripts/build-qwen38-bf16-rad.sh

@@ -0,0 +1,220 @@
+#!/usr/bin/env bash
+# build-qwen38-bf16-rad.sh — RUNS ON DANKSERVER. rad-convert the vendor BF16
+# checkpoint /home/dank/models/Qwen3.8-Flash-Next into
+# /home/dank/models/rad/qwen38-bf16.rad with NO recipe and NO --reuse: a weight
+# no rule matches keeps the checkpoint's encoding, so every tensor stays the
+# bf16 it is in the safetensors and the container is the checkpoint's own
+# numbers behind a .rad header.
+#
+# This file is the campaign's measuring stick, twice over: scripts/
+# run-qwen38-bf16.sh kldref records the KLD reference logits from it (every
+# arm's --kld-ref is scored against that record) and the same container serves
+# with the calibration tap to write the Grams that derive the protected-expert
+# list and feed GPTQ. A reference that is not the vendor checkpoint, byte for
+# byte in bf16, silently moves every gate in the campaign — hence no recipe, no
+# --reuse, and the four refusal gates below.
+#
+# Why a container at all: a checkpoint dir has no mapped/file tiers, and a
+# weight only a host kernel reads (the 47.7 GiB PLE n-gram table) cannot be
+# served from one; rad-convert also bakes the tokenizer. The convert runs on
+# the HOST backend — no --device /dev/kfd, no /dev/dri — so declare resolves
+# kernels on CPU and production serving is untouched for the whole run (the
+# precedent that proved it: scripts/convert-swift15-w4dn8.sh).
+#
+# Gates, every one a refusal with exit 2 and a named reason, BEFORE anything is
+# written, cheapest first — the verifier runs last of them because its Hub gate
+# hashes 343 GiB and must not be paid for by a run that was going to refuse on
+# the output file anyway.
+#   checkpoint  the tree must be on disk; an absent dir names the command that
+#               makes one (scripts/download-qwen38-bf16.sh --fetch) and stops.
+#   free space  the convert writes ~343,323 MiB next to the ~343,323 MiB
+#               checkpoint; QWEN38_MIN_FREE_MIB (default 350000 = that figure
+#               plus 6.5 GiB of slack) is the floor. The volume that cannot
+#               hold both is discovered at minute 2, not hour 5.
+#   output      the campaign converts the reference ONCE. An existing
+#               qwen38-bf16.rad is refused; --force retires it to
+#               ~/.AGENTTRASH/<utc>-qwen38-bf16.rad (trash law: nothing is
+#               deleted, and the move is a rename on the same volume).
+#   verified    scripts/download-qwen38-bf16.sh's verify mode must pass. A
+#               failed verify exits 1, not 2: the tree being the wrong payload
+#               is a result, a missing environment is a refusal.
+#   plan        rad-convert --plan-only -v first — the oracle for "does this
+#               shape boot". A refused plan costs seconds and saves a convert.
+#
+# usage: scripts/build-qwen38-bf16-rad.sh [--dry-run] [--plan-only] [--force]
+#   --dry-run    print the argv it would exec (podman wrapping rad-convert),
+#               one shell-quoted argument a line, and exit 0. No gate runs, no
+#               container starts, nothing is written — the offline test seam.
+#               It does resolve the checkpoint and output paths through the
+#               /models mount, because an argv that carries host paths would
+#               not run either.
+#   --plan-only  run the plan and every gate except the disk and output ones —
+#               those two protect bytes the plan never writes, and this is the
+#               probe to run when the disk gate itself is what refuses
+#   --force      retire an existing output container to the trash dir first
+#
+# knobs: FRANK_QWEN38_MODELS   the host dir mounted at /models (default
+#        /home/dank/models); FRANK_QWEN38_CKPT (default $MODELS/
+#        Qwen3.8-Flash-Next, the same knob the verifier reads);
+#        FRANK_QWEN38_BF16_OUT (default $MODELS/rad/qwen38-bf16.rad);
+#        QWEN38_MIN_FREE_MIB; QWEN38_TRASH (default $HOME/.AGENTTRASH). The
+#        verifier's own census knobs (FRANK_QWEN38_SHARDS / _BYTES / _REPO)
+#        reach it through the environment untouched.
+# exit: 0 converted (or planned), 1 the checkpoint failed verification,
+#       2 refused before anything was written
+set -euo pipefail
+
+SCRIPT_DIR="$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)"
+# ONE owner of the engine pin: the digest frank-archon serves on. The reference
+# must be built by the SAME engine build that scores it, or a KLD delta is
+# partly a version delta.
+# shellcheck disable=SC1091
+source "$SCRIPT_DIR/frank-archon.env"
+IMAGE="${FRANK_ARCHON_IMAGE:?FRANK_ARCHON_IMAGE must name the radiance image}"
+
+MODELS="${FRANK_QWEN38_MODELS:-/home/dank/models}"
+CKPT="${FRANK_QWEN38_CKPT:-$MODELS/Qwen3.8-Flash-Next}"
+OUT="${FRANK_QWEN38_BF16_OUT:-$MODELS/rad/qwen38-bf16.rad}"
+TRASH="${QWEN38_TRASH:-$HOME/.AGENTTRASH}"
+MIN_FREE_MIB="${QWEN38_MIN_FREE_MIB:-350000}"
+LOG="/home/dank/logs/qwen38_bf16_convert.log"
+
+DRY=0
+PLAN_ONLY=0
+FORCE=0
+while [ $# -gt 0 ]; do
+    case "$1" in
+        --dry-run)   DRY=1 ;;
+        --plan-only) PLAN_ONLY=1 ;;
+        --force)     FORCE=1 ;;
+        *) echo "usage: ${0##*/} [--dry-run] [--plan-only] [--force]" >&2
+           exit 2 ;;
+    esac
+    shift
+done
+
+# The convert sees the tree only through the /models mount, so both paths must
+# be under it; the container-side spelling is what goes into the argv.
+case "$CKPT" in
+    "$MODELS"/*) CKPT_IN="/models/${CKPT#"$MODELS"/}" ;;
+    *) echo "REFUSED: checkpoint $CKPT is not under the mounted models dir $MODELS — the convert cannot see it" >&2
+       exit 2 ;;
+esac
+case "$OUT" in
+    "$MODELS"/*) OUT_IN="/models/${OUT#"$MODELS"/}" ;;
+    *) echo "REFUSED: output $OUT is not under the mounted models dir $MODELS — rad-convert could not write it" >&2
+       exit 2 ;;
+esac
+
+# Declare with NO recipe line — that is what keeps every tensor bf16 — and at the
+# bound this container will actually be served at. Both flags come off the pinned
+# image's own help: --kv-cache-dtype at convert "only picks the kernels it
+# checks" and the container is identical either way, so naming fp8 is free
+# insurance — if the fp8 KV kernels the fleet serves would not resolve, the
+# convert refuses here instead of the boot failing on a drained window.
+# --max-ctx is the bound declare is run at, whose default is the checkpoint's
+# training context (262,144 here, whose fp8 cache is 258.8 GiB against a 192 GiB
+# machine, so the default is not an option). 32,768 is set by the corpus, not by
+# taste: kld.jsonl's longest record is 38,263 characters — ~9,600 tokens at the
+# pessimistic chars/4, ~7,700 at the chars/4.97 this repo measured on the K3
+# corpus — so the window the rig boots at clears every record 2.6x over and the
+# reference boot cannot die on a sequence that will not fit. The rig's window is
+# one knob (FRANK_QWEN38_MAX_MODEL_LEN in scripts/run-qwen38-bf16.sh, default
+# 32,768) and that script refuses a corpus its window cannot hold, so raising one
+# without the other is caught before the drained window, not inside it. The
+# payload containers the gate scores declare at the training context because
+# production must hold the 200k shape; a declared bound is a declare-time check,
+# not a weight change, so the reference and the payloads carry the same numbers
+# whatever each declared at — which is why one can be measured against the other.
+CONVERT_ARGS=("$CKPT_IN" -o "$OUT_IN" --max-ctx 32768 --kv-cache-dtype fp8)
+PODMAN=(podman run --rm --entrypoint rad-convert -v "$MODELS:/models")
+PLAN_CMD=("${PODMAN[@]}" --name qwen38-bf16-plan "$IMAGE" "${CONVERT_ARGS[@]}" --plan-only -v)
+CONV_CMD=("${PODMAN[@]}" --name qwen38-bf16-convert "$IMAGE" "${CONVERT_ARGS[@]}")
+
+if [ "$DRY" = "1" ]; then
+    echo "dry run: no gate checked, no container started, nothing written"
+    echo "# $CKPT -> $OUT (the convert sees them as $CKPT_IN and $OUT_IN)"
+    echo "# plan argv:"
+    printf '%q\n' "${PLAN_CMD[@]}"
+    echo "# convert argv:"
+    printf '%q\n' "${CONV_CMD[@]}"
+    exit 0
+fi
+
+# ---- gate: the checkpoint is on disk ---------------------------------------
+if [ ! -d "$CKPT" ]; then
+    echo "REFUSED: no checkpoint at $CKPT — fetch and verify it first: scripts/download-qwen38-bf16.sh --fetch" >&2
+    exit 2
+fi
+
+# ---- gate: peak disk --------------------------------------------------------
+# The deepest existing ancestor is the right thing to ask: a fresh
+# /home/dank/models/rad has no free space of its own to report.
+OUT_DIR="$(dirname "$OUT")"
+PROBE="$OUT_DIR"
+while [ ! -d "$PROBE" ] && [ "$PROBE" != "/" ]; do PROBE="$(dirname "$PROBE")"; done
+FREE_MIB="$(df --output=avail -m "$PROBE" | tail -1 | tr -d ' ')"
+if ! [[ "$FREE_MIB" =~ ^[0-9]+$ ]]; then
+    echo "REFUSED: cannot read the free space under $PROBE (df said: $FREE_MIB)" >&2
+    exit 2
+fi
+if [ "$PLAN_ONLY" != "1" ] && [ "$FREE_MIB" -lt "$MIN_FREE_MIB" ]; then
+    echo "REFUSED peak-disk: $FREE_MIB MiB free under $PROBE, $MIN_FREE_MIB MiB wanted (QWEN38_MIN_FREE_MIB) — the bf16 container rewrites the checkpoint's own 343,323 MiB of tensors beside it" >&2
+    exit 2
+fi
+
+# ---- gate: one conversion ---------------------------------------------------
+if [ -e "$OUT" ]; then
+    if [ "$PLAN_ONLY" = "1" ]; then
+        echo "note: $OUT already exists; --plan-only writes nothing, so it stays"
+    elif [ "$FORCE" = "1" ]; then
+        mkdir -p "$TRASH"
+        DEST="$TRASH/$(date -u +%Y%m%dT%H%M%SZ)-$(basename "$OUT")"
+        mv "$OUT" "$DEST"
+        echo "TRASHED: retired the existing reference container $OUT -> $DEST"
+    else
+        echo "REFUSED: $OUT already exists — the campaign converts the reference once; re-run with --force to retire it to $TRASH first" >&2
+        exit 2
+    fi
+fi
+
+# ---- gate: the tree really is the checkpoint (the full verify) -------------
+echo "== verifying $CKPT (scripts/download-qwen38-bf16.sh, full gates)"
+rc=0
+bash "$SCRIPT_DIR/download-qwen38-bf16.sh" --dir "$CKPT" || rc=$?
+if [ "$rc" -ne 0 ]; then
+    if [ "$rc" -eq 2 ]; then
+        echo "REFUSED: the checkpoint verifier refused the environment (rc=2); nothing was converted" >&2
+        exit 2
+    fi
+    echo "REFUSED: $CKPT failed verification (rc=$rc) — a half tree converts to a half reference" >&2
+    exit 1
+fi
+
+# ---- plan, then convert ------------------------------------------------------
+mkdir -p "$(dirname "$LOG")"
+# rad-convert opens -o during declare as well, so the output directory exists
+# before the plan; that empty directory is all this creates ahead of it.
+mkdir -p "$OUT_DIR"
+echo "== rad-convert --plan-only -v (declare + plan; writes nothing)"
+"${PLAN_CMD[@]}" 2>&1 | tee -a "$LOG"
+rc=${PIPESTATUS[0]}
+if [ "$rc" -ne 0 ]; then
+    echo "REFUSED plan: rad-convert --plan-only exited rc=$rc — the engine refuses this encoding/shape, no container was written (log: $LOG)" >&2
+    exit 2
+fi
+if [ "$PLAN_ONLY" = "1" ]; then
+    echo "PLAN_OK: $CKPT_IN declares for $OUT_IN; no container written (log: $LOG)"
+    exit 0
+fi
+
+echo "== convert: $CKPT -> $OUT ($(date -Is))"
+"${CONV_CMD[@]}" 2>&1 | tee -a "$LOG"
+rc=${PIPESTATUS[0]}
+if [ "$rc" -ne 0 ]; then
+    echo "rad-convert rc=$rc; log: $LOG" >&2
+    exit "$rc"
+fi
+MIB=$(( $(stat -c %s "$OUT") / 1048576 ))
+echo "BUILD_DONE: $OUT is ${MIB} MiB (log: $LOG)"
+echo "NEXT: scripts/run-qwen38-bf16.sh kldref records the reference logits from it"

scripts/convert-qwen38-w5nl64.sh

@@ -0,0 +1,221 @@
+#!/usr/bin/env bash
+# convert-qwen38-w5nl64.sh — RUNS ON DANKSERVER. The CPU half of the Qwen3.8-Flash-Next five-bit
+# campaign: convert the vendor BF16 checkpoint into the campaign's arms, each one the whole
+# `w5nl64a8h` rung on BOTH expert projections with the protected experts kept bf16.
+#
+# Nothing here touches a GPU: the container is started without /dev/kfd or /dev/dri, so the
+# converter runs on the host backend and the fleet can keep serving through the whole script.
+# The engine pin is the digest that serves frank-archon, so a winning arm crosses no version
+# boundary between measurement and serving.
+#
+# Three arms, because the campaign's question is which search knob pays for its hours
+# (recipes/qwen4exp-qwen38-w5nl64-bf16trunk.recipe):
+#
+#   cd3     the recipe as written — coordinate descent three sweeps. The control.
+#   cd6     six sweeps: the same search, twice the passes.
+#   ao1     act_order=1 instead of cd: GPTQ sweeps columns in activation-importance order and
+#           stores them permuted. This is a SEPARATE arm, not a fourth sweep — libquant refuses
+#           cd= together with act_order=1 ("cd= sweeps the columns in their stored order, and
+#           act_order=1 stores them permuted; one or the other", libquant/lq_registry.cpp), so
+#           the two knobs cannot be combined into a best-of-both container.
+#
+# The arms are generated from ONE template with `--quant` overrides, not three near-duplicate
+# recipes: --quant rules are inserted BEFORE the recipe file and the first rule whose glob matches
+# a weight decides it (core/format/recipe.cpp: Recipe::match), so an override cleanly replaces the
+# expert line and the other 60 lines cannot drift between arms.
+#
+# Protected experts: apply-protected-cast.sh derives the bf16 set from THIS campaign's grams
+# (--target-count 48, two per layer so each rank's 256-expert half stays even) and splices it into
+# a per-run COPY of the template; the committed recipe keeps its markers and stays reviewable.
+#
+# Gates, in the order they bite — each one is a lesson the swift15 campaign paid for:
+#   inputs        checkpoint (dir + index), recipe, grams, imatrix, tokenizer. A missing grams
+#                 directory converts UNCALIBRATED with one stderr note, which is silently worse
+#                 than failing here.
+#   reference     the 335.3 GiB BF16 reference container must be GONE before the converts start.
+#                 It and the checkpoint together eat the volume; its job (the KLD recording and
+#                 the calibration grams) is finished by then, and both survive on disk without the
+#                 container beside it. The FP8-native anchor is FETCHED after the converts (disk
+#                 sequencing), so nothing here waits on it.
+#   plan          --plan-only runs FIRST, per arm, and prints "about X would be written". The
+#                 number is the bill: free space must cover it plus the 128k-prompt overshoot
+#                 margin, and the finished artifact is checked against it.
+#   outputs       no arm overwrites anything; move the old one to trash.
+#
+# Usage: convert-qwen38-w5nl64.sh [--arm cd3|cd6|ao1]... [--dry-run] [--plan-only]
+#                                 [--target-count N] [--keep-recipe]
+set -u -o pipefail
+
+SELF="$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)"
+IMAGE=${IMAGE:-stilldeadcode/radiance@sha256:d590e300234bf6d2f942a0b8d456d183153fd808f28820e6c2fa69c9389178ed}
+CKPT=${CKPT:-/home/dank/models/Qwen3.8-Flash-Next}
+CAMPAIGN=${CAMPAIGN:-/home/dank/campaign/qwen38-w5nl64}
+RECIPE=${RECIPE:-$SELF/recipes/qwen4exp-qwen38-w5nl64-bf16trunk.recipe}
+CALIB=${CALIB:-$CAMPAIGN/calib}
+IMATRIX=${IMATRIX:-$CAMPAIGN/imatrix.gguf}
+OUTDIR=${OUTDIR:-/home/dank/models/rad}
+BF16_RAD=${BF16_RAD:-$OUTDIR/qwen38-bf16.rad}   # the reference container, not an arm output
+STATE=${STATE:-$CAMPAIGN}                      # the dir the container mounts read-only: the
+                                               # spliced recipe and the imatrix live here
+TARGET_COUNT=${TARGET_COUNT:-48}
+MIN_GRAMS=${MIN_GRAMS:-24576}                 # per-expert DOWN grams: 48 recorded layers x 512
+                                               # experts. The checkpoint has 49 MoE layers, but the
+                                               # MTP layer is never recorded, so its 512 experts are
+                                               # absent from the tap's output by design
+MARGIN_MIB=${MARGIN_MIB:-40960}               # the long-prompt overshoot, off the plan number
+TS=$(date +%Y%m%d-%H%M%S)
+
+ARMS=()
+DRY=0
+PLAN_ONLY=0
+KEEP_RECIPE=0
+while [ "$#" -gt 0 ]; do
+    case "$1" in
+      --arm) ARMS+=("$2"); shift 2 ;;
+      --dry-run) DRY=1; shift ;;
+      --plan-only) PLAN_ONLY=1; shift ;;
+      --target-count) TARGET_COUNT="$2"; shift 2 ;;
+      --keep-recipe) KEEP_RECIPE=1; shift ;;
+      *) echo "unknown argument: $1" >&2; exit 2 ;;
+    esac
+done
+[ "${#ARMS[@]}" -gt 0 ] || ARMS=(cd3 cd6 ao1)
+for a in "${ARMS[@]}"; do
+    case "$a" in cd3|cd6|ao1) ;; *) echo "unknown arm: $a (cd3|cd6|ao1)" >&2; exit 2 ;; esac
+done
+
+# The expert rule's option set, spelled once. --quant replaces the template's line wholesale, so
+# every option has to be repeated here; `calib=$CALIB` is expanded by the recipe reader from the
+# container environment (core/format/recipe.cpp: expand refuses an unset name rather than taking
+# it for empty).
+expert_opts='table=w5nl,group=64,scale=fp8_e4m3,scale2=f32,block2=*x*,scale2_value=0.0001220703125,transform=fwht128,rule=search,calib=$CALIB'
+arm_quant() {
+    case "$1" in
+      # The control is the template itself: no override, so the recipe's own line decides.
+      cd3) return 0 ;;
+      cd6) echo "blk.*.ffn_*_exps.*.weight=gptq:${expert_opts},cd=6" ;;
+      # act_order=1 WITHOUT cd: libquant refuses the pair (lq_registry.cpp:105, "cd= sweeps the
+      # columns in their stored order, and act_order=1 stores them permuted; one or the other"), so
+      # this arm is the interleave knob alone, not cd=3 plus it.
+      ao1) echo "blk.*.ffn_*_exps.*.weight=gptq:${expert_opts},act_order=1" ;;
+    esac
+}
+
+# --- gates ------------------------------------------------------------------------------------
+[ -d "$CKPT" ] || { echo "no checkpoint at $CKPT; run scripts/download-qwen38-bf16.sh --fetch" >&2; exit 2; }
+[ -f "$CKPT/model.safetensors.index.json" ] || { echo "checkpoint at $CKPT has no safetensors index — a torn fetch" >&2; exit 2; }
+[ -f "$CKPT/tokenizer.json" ] || { echo "no tokenizer.json in $CKPT; the .rad cannot carry one without it" >&2; exit 2; }
+[ -f "$RECIPE" ] || { echo "no recipe at $RECIPE" >&2; exit 2; }
+GRAMS=$(find "$CALIB" -maxdepth 1 -name 'gram.r*.bin' 2>/dev/null | wc -l)
+[ "$GRAMS" -ge "$MIN_GRAMS" ] || {
+    echo "calib dir $CALIB holds $GRAMS gram files (<$MIN_GRAMS); run $0 after run-qwen38-bf16.sh calib" >&2; exit 2; }
+[ -f "$IMATRIX" ] || { echo "no imatrix at $IMATRIX; run scripts/grams-to-imatrix.py over $CALIB" >&2; exit 2; }
+# The container sees the campaign dir at one fixed mount point, so an imatrix kept elsewhere has
+# no path inside it: copy it in (or point IMATRIX at the campaign dir) rather than let the
+# converter fail halfway through a planned arm.
+[ "$(dirname "$IMATRIX")" = "$STATE" ] || {
+    echo "imatrix $IMATRIX is not in the campaign dir $STATE the container mounts; move it there" >&2; exit 2; }
+if [ -e "$BF16_RAD" ]; then
+    echo "the 335.3 GiB BF16 reference container $BF16_RAD is still on disk; its recording and" \
+         "the calibration grams are already taken, so move it to trash — the converts need its" \
+         "space (peak-disk law)" >&2
+    exit 2
+fi
+for a in "${ARMS[@]}"; do
+    [ -e "$OUTDIR/qwen38-w5nl64-$a.rad" ] && {
+        echo "$OUTDIR/qwen38-w5nl64-$a.rad already exists; move it to trash first" >&2; exit 2; }
+done
+
+# --- the protected plane, spliced into a per-run copy of the template --------------------------
+mkdir -p "$STATE"
+WORK="$STATE/recipe-$TS.recipe"
+cp "$RECIPE" "$WORK"
+if [ "$DRY" = 1 ]; then
+    echo "would run: $SELF/apply-protected-cast.sh $CALIB $WORK $TARGET_COUNT"
+else
+    # Positional, because the tool reads its recipe and calib dir by position: passing them as
+    # environment variables would be silently ignored and it would rewrite the COMMITTED
+    # template in place instead of this run's copy.
+    "$SELF/apply-protected-cast.sh" "$CALIB" "$WORK" "$TARGET_COUNT" \
+        || { echo "the protected-expert derivation refused; the arms would convert unprotected" >&2; exit 1; }
+fi
+
+run_convert() { # run_convert ARM [extra flags...]
+    local arm="$1"; shift
+    local out="$OUTDIR/qwen38-w5nl64-$arm.rad" log="$STATE/convert-$arm-$TS.log"
+    local argv=(--rm --name "qwen38-convert-$arm"
+                -e "CALIB=/data/calib" -e "OMP_NUM_THREADS=$(nproc)"
+                -v "$(dirname "$CKPT"):/models:ro"
+                -v "$CALIB:/data/calib:ro"
+                -v "$STATE:/campaign:ro"
+                -v "$OUTDIR:/out")
+    local conv=(/models/"${CKPT##*/}" --recipe "/campaign/$(basename "$WORK")"
+                --tokenizer "/models/${CKPT##*/}/tokenizer.json"
+                --imatrix /campaign/"${IMATRIX##*/}"
+                # No --max-ctx: the context declare then runs at the checkpoint's training context
+                # (262144), which is what the shipped container did — rad-info --meta on it carries
+                # no override. A container declared at a smaller bound cannot be expected to serve
+                # the 200k window D2 holds, and the convert is not where to discover that.
+                # --kv-cache-dtype changes nothing in the artifact; it picks the kernels the declare
+                # checks, and the fleet serves fp8 KV, so an unresolvable fp8 kernel must be said no
+                # to by the convert, not by a boot on a drained window.
+                --kv-cache-dtype fp8 -o "/out/$(basename "$out")")
+    local q; q=$(arm_quant "$arm")
+    [ -n "$q" ] && conv+=(--quant "$q")
+    if [ "$DRY" = 1 ]; then
+        echo "podman run ${argv[*]} \$IMAGE --entrypoint /opt/radiance/bin/rad-convert ${conv[*]} $*"
+        return 0
+    fi
+    podman run "${argv[@]}" --entrypoint /opt/radiance/bin/rad-convert "$IMAGE" "${conv[@]}" "$@" \
+        2>&1 | tee "$log"
+    return "${PIPESTATUS[0]}"
+}
+
+# --- plan, then bill, then convert -------------------------------------------------------------
+overall=0
+for arm in "${ARMS[@]}"; do
+    echo "== arm $arm =="
+    if [ "$DRY" = 1 ]; then
+        # Both lines, in order: the plan runs before the convert, and the convert is what the
+        # operator is about to pay an afternoon for.
+        run_convert "$arm" --plan-only
+        run_convert "$arm"
+        continue
+    fi
+    rc=0; plan_log=$(run_convert "$arm" --plan-only) || rc=$?
+    [ "$rc" -eq 0 ] || { echo "arm $arm: --plan-only failed (rc=$rc); nothing was written" >&2; overall=1; continue; }
+    echo "$plan_log"
+    # "about 141.2 GiB would be written" is the converter's own bill; parse it rather than
+    # hardcoding a size, so the gate still works after any recipe edit.
+    want=$(printf '%s' "$plan_log" | sed -n 's/.*about \([0-9.]*\) \([KMGT]iB\) would be written.*/\1 \2/p' | tail -1)
+    [ -n "$want" ] || { echo "arm $arm: the plan printed no size; refusing to convert blind" >&2; overall=1; continue; }
+    need_mib=$(awk -v n="${want% *}" -v u="${want#* }" \
+        'BEGIN{ f = (u=="KiB") ? 2^-10 : (u=="MiB") ? 1 : (u=="GiB") ? 1024 : (u=="TiB") ? 1048576 : 0
+               if (f == 0) { exit 1 }; printf "%d", n*f + 0.5 }') || {
+        echo "arm $arm: the plan printed an unknown unit in '$want'" >&2; overall=1; continue; }
+    free_mib=$(df --output=avail -m "$OUTDIR" | tail -1 | tr -d ' ')
+    echo "plan: $want ($need_mib MiB); free ${free_mib} MiB on $OUTDIR (margin $MARGIN_MIB MiB)"
+    if [ "$free_mib" -lt $((need_mib + MARGIN_MIB)) ]; then
+        echo "arm $arm: needs ${need_mib} MiB + ${MARGIN_MIB} margin, ${free_mib} free; free space and re-run" >&2
+        overall=1; continue
+    fi
+    if [ "$PLAN_ONLY" = 1 ]; then echo "planned, not converted (--plan-only)"; continue; fi
+    rc=0; run_convert "$arm" || rc=$?
+    [ "$rc" -eq 0 ] || { echo "arm $arm: rad-convert rc=$rc; log: $STATE/convert-$arm-$TS.log" >&2; overall=1; continue; }
+    out="$OUTDIR/qwen38-w5nl64-$arm.rad"
+    got_mib=$(( $(stat -c %s "$out") / 1048576 ))
+    # The receipt: the artifact is what the plan predicted, within the metadata the plan does not
+    # count. An arm that came out materially smaller lost a plane somewhere.
+    lo=$(( need_mib * 98 / 100 )); hi=$(( need_mib * 103 / 100 ))
+    if [ "$got_mib" -lt "$lo" ] || [ "$got_mib" -gt "$hi" ]; then
+        echo "arm $arm: $out is ${got_mib} MiB, the plan said ${need_mib} MiB (allowed ${lo}-${hi})" >&2
+        overall=1; continue
+    fi
+    echo "CONVERT_DONE $arm ${got_mib} MiB -> $out"
+done
+
+if [ "$KEEP_RECIPE" = 0 ] && [ "$DRY" = 0 ]; then
+    mkdir -p "$CAMPAIGN/.AGENTTRASH"
+    mv "$WORK" "$CAMPAIGN/.AGENTTRASH/$(basename "$WORK")" 2>/dev/null || true
+fi
+exit "$overall"

scripts/deploy-plan.py

@@ -34,12 +34,16 @@ SERVICE_ORDER = (
     "hermes",
     "hermes-dashboard",
     "hermes-webui",
-    "prometheus",
     "grafana",
     "loki",
     "alloy",
     "node-exporter",
+    "cadvisor",
+    "gpu-exporter",
     "alertmanager",
+    # The prober must answer before prometheus re-runs the services job.
+    "blackbox-exporter",
+    "prometheus",
     "authentik-postgres",
     "authentik-server",
     "authentik-worker",
@@ -92,7 +96,10 @@ DANKCLAW_SERVICES = {
     "loki",
     "alloy",
     "node-exporter",
+    "cadvisor",
+    "gpu-exporter",
     "alertmanager",
+    "blackbox-exporter",
     "authentik-postgres",
     "authentik-server",
     "authentik-worker",
@@ -124,6 +131,8 @@ DANKSERVER_SERVICES = {
     "readarr",
     "frank-archon",
     "node-exporter",
+    "cadvisor",
+    "gpu-exporter",
 }
 
 # FrankBook-Pro runs the *arr stack + qBittorrent inside a podman machine on
@@ -271,6 +280,8 @@ def services_for_path(path: str, target: str) -> set[str]:
             # Provisioned datasources and dashboards: git is the source of
             # truth, so a change here must restart the service that reads them.
             _add(services, "grafana")
+        elif group == "blackbox":
+            _add(services, "blackbox-exporter")
         elif group == "homepage":
             # Homepage is a Dankclaw-owned deployment. Shared homepage files
             # are intentionally not copied during a Dankserver-only run.
@@ -374,7 +385,7 @@ def services_for_path(path: str, target: str) -> set[str]:
             # consumer (prometheus before grafana, loki before alloy).
             if len(parts) >= 4 and parts[2] == "quadlet":
                 name = parts[3].removesuffix(".container")
-                if name in {"prometheus", "grafana", "loki", "alloy", "node-exporter", "alertmanager"}:
+                if name in {"prometheus", "grafana", "loki", "alloy", "node-exporter", "cadvisor", "alertmanager", "blackbox-exporter"}:
                     _add(services, name)
             return services
         if len(parts) >= 2 and parts[1] == "gitea":
@@ -417,8 +428,8 @@ def services_for_path(path: str, target: str) -> set[str]:
             # node-dankserver job had nothing to scrape.
             if len(parts) >= 4 and parts[2] == "quadlet":
                 name = parts[3].removesuffix(".container")
-                if name == "node-exporter":
-                    _add(services, "node-exporter")
+                if name in {"node-exporter", "cadvisor"}:
+                    _add(services, name)
             return services
         # frank-archon's quadlet-era directory left the tree on 2026-09-26
         # (the Lucebox DS4.1 unit was retired for the llama.cpp root unit —
@@ -449,6 +460,12 @@ def services_for_path(path: str, target: str) -> set[str]:
             # change to it redeploying alertmanager is the only path that
             # reaches it on the host.
             _add(services, "alertmanager")
+        if name in {"gpu-exporter.py", "gpu-exporter.service"}:
+            # The host-process GPU exporter is installed directly by the
+            # gpu-exporter deploy on BOTH hosts, so its script and unit select
+            # that service (target_services then keeps it only on the host being
+            # planned). Editing the collector restarts the running exporter.
+            _add(services, "gpu-exporter")
         if name in {"frank-zealot.env", "frank-zealot.service", "serve-frank-zealot.sh"}:
             # llama.cpp frank-zealot (2026-09-16) keeps its unit, env and serve
             # files here; each one selects frank-zealot only. The ik_llama.cpp

scripts/deploy-remote.sh

@@ -86,7 +86,7 @@ service_is_allowed() {
         dankclaw:open-webui|dankclaw:gitea-postgres|dankclaw:gitea-server|\
         dankclaw:comfyui|dankclaw:hermes|\
         dankclaw:hermes-dashboard|dankclaw:hermes-webui|dankclaw:prometheus|dankclaw:grafana|\
-        dankclaw:loki|dankclaw:alloy|dankclaw:node-exporter|dankclaw:alertmanager|\
+        dankclaw:loki|dankclaw:alloy|dankclaw:node-exporter|dankclaw:cadvisor|dankclaw:gpu-exporter|dankclaw:alertmanager|dankclaw:blackbox-exporter|\
         dankclaw:authentik-postgres|dankclaw:authentik-server|\
         dankclaw:authentik-worker|dankclaw:authentik-outpost|dankclaw:authentik-outpost-ldap|dankclaw:anubis|\
         dankclaw:caddy|dankclaw:homepage|dankclaw:seerr|dankclaw:marreta|\
@@ -97,7 +97,7 @@ service_is_allowed() {
         dankserver:copyparty|dankserver:jellyfin|dankserver:calibre-web|dankserver:calibre-server|dankserver:plex|dankserver:tautulli|\
         dankserver:prowlarr|dankserver:qbittorrent|dankserver:radarr|\
         dankserver:sonarr|dankserver:lidarr|dankserver:readarr|\
-        dankserver:frank-archon|dankserver:node-exporter|\
+        dankserver:frank-archon|dankserver:node-exporter|dankserver:cadvisor|dankserver:gpu-exporter|\
         frankbook-pro:gluetun|frankbook-pro:qbittorrent|frankbook-pro:prowlarr|\
         frankbook-pro:radarr|frankbook-pro:sonarr|frankbook-pro:lidarr|\
         frankbook-pro:readarr)
@@ -252,16 +252,24 @@ install_signal_relay() {
     # The relay binds the host's LAN IP (the one address a bridge container
     # can reach as host.containers.internal), so keep the unauthenticated
     # endpoint off the rest of the LAN: admit loopback, the host itself
-    # (its own health probe) and the podman bridge subnet only. Idempotent:
-    # -C before -I.
+    # (its own health probe) and the podman bridge subnet only.
+    # ORDER IS THE GATE: `iptables -I` inserts at position 1, so the old
+    # shape (-C/-I x3 ACCEPT, then -C/-I the DROP) landed the blanket DROP
+    # ABOVE the ACCEPTs and every probe timed out beside a LISTENing socket
+    # (live 2026-10-21: 159 packets on the DROP, 0 on each ACCEPT; the whole
+    # dankclaw drain aborted at the alertmanager gate twice before the cause
+    # was read off `nft list ruleset`). Rebuild the tail instead of inserting
+    # it: delete any blanket DROP wherever it sits, keep the three ACCEPTs
+    # at the head (-C before -I, unchanged), then APPEND the DROP so it is
+    # last and the ACCEPTs decide first. Converges on poisoned hosts.
+    run_privileged iptables -D INPUT -p tcp --dport 18093 -j DROP 2>/dev/null || true
     run_privileged iptables -C INPUT -p tcp --dport 18093 -s 127.0.0.1 -j ACCEPT 2>/dev/null \
         || run_privileged iptables -I INPUT -p tcp --dport 18093 -s 127.0.0.1 -j ACCEPT
     run_privileged iptables -C INPUT -p tcp --dport 18093 -s 192.168.0.143 -j ACCEPT 2>/dev/null \
         || run_privileged iptables -I INPUT -p tcp --dport 18093 -s 192.168.0.143 -j ACCEPT
     run_privileged iptables -C INPUT -p tcp --dport 18093 -s 10.89.0.0/16 -j ACCEPT 2>/dev/null \
         || run_privileged iptables -I INPUT -p tcp --dport 18093 -s 10.89.0.0/16 -j ACCEPT
-    run_privileged iptables -C INPUT -p tcp --dport 18093 -j DROP 2>/dev/null \
-        || run_privileged iptables -I INPUT -p tcp --dport 18093 -j DROP
+    run_privileged iptables -A INPUT -p tcp --dport 18093 -j DROP
     # The interpreter needs a moment to bind after restart; a one-shot curl
     # raced the socket on 2026-09-17 and failed a healthy deploy.
     local i
@@ -275,14 +283,81 @@ install_signal_relay() {
     return 1
 }
 
+# Host-process GPU metrics exporter (amdgpu sysfs + nvidia-smi). Not a quadlet:
+# it runs python3 from the checkout on the host so it can read the DRM/cgroup
+# sysfs and shell out to nvidia-smi. Same two-address bind posture as
+# node-exporter/cadvisor (loopback + this host's LAN), so it is never a wildcard.
+deploy_gpu_exporter() {
+    local unit_dir="$HOME/.config/systemd/user" lan dropin i
+    case "$TARGET" in
+        dankclaw) lan=192.168.0.143 ;;
+        dankserver) lan=192.168.0.235 ;;
+        *) echo "gpu-exporter: unknown target '$TARGET'" >&2; return 2 ;;
+    esac
+    mkdir -p "$unit_dir"
+    cp scripts/gpu-exporter.service "$unit_dir/gpu-exporter.service"
+    dropin="$unit_dir/gpu-exporter.service.d"
+    mkdir -p "$dropin"
+    printf '[Service]\nEnvironment=GPU_EXPORTER_BINDS=127.0.0.1,%s\n' "$lan" \
+        > "$dropin/bind.conf"
+    systemctl --user daemon-reload
+    systemctl --user enable gpu-exporter.service
+    systemctl --user restart gpu-exporter.service
+    # The interpreter needs a moment to bind after restart (the same race that
+    # once failed a healthy signal-relay deploy).
+    for i in $(seq 1 15); do
+        curl -fsS -m 5 http://127.0.0.1:9401/metrics >/dev/null && return 0
+        sleep 1
+    done
+    echo "gpu-exporter is not answering on 127.0.0.1:9401/metrics" >&2
+    return 1
+}
+
 copy_loki_config() {
     run_privileged mkdir -p /opt/loki/config
     run_privileged cp shared/loki/loki-config.yaml /opt/loki/config/loki-config.yaml
 }
 
 copy_alloy_config() {
-    run_privileged mkdir -p /opt/alloy/config
-    run_privileged cp shared/alloy/config.alloy /opt/alloy/config/config.alloy
+    # The repo config carries the token GRAFANA_HOST_LABEL; the installed copy
+    # carries the host's name, so Loki streams say WHICH machine's journal a
+    # line came from without a second config file per host.
+    # *_CONF_ROOT are test seams: the bats cases execute these copiers for
+    # real against a sandbox with a stubbed run_privileged.
+    local root="${ALLOY_CONF_ROOT:-/opt/alloy}" tmp
+    tmp="$(mktemp)"
+    sed "s/GRAFANA_HOST_LABEL/$TARGET/g" shared/alloy/config.alloy > "$tmp"
+    if ! grep -q "host = \"$TARGET\"," "$tmp" || grep -q "GRAFANA_HOST_LABEL" "$tmp"; then
+        echo "alloy config: host-label substitution failed" >&2
+        rm -f "$tmp"
+        return 1
+    fi
+    # Syntax gate: alloy's config language has no schema — a misplaced
+    # attribute (e.g. `on_error` outside the json stage) parses only at RUN
+    # time, and the unit then crash-loops with a healthy-looking deploy.
+    # Validate against the very image that will run it, before installing.
+    # Skipped only in the bats sandbox (no image there).
+    if [ "$root" = "/opt/alloy" ] || podman image exists localhost/franksdank/alloy:latest 2>/dev/null; then
+        if ! podman run --rm -v "$tmp:/c.alloy:ro" \
+                --entrypoint /usr/bin/alloy localhost/franksdank/alloy:latest \
+                validate /c.alloy; then
+            echo "alloy config: validate FAILED against the pinned image" >&2
+            rm -f "$tmp"
+            return 1
+        fi
+    fi
+    run_privileged mkdir -p "$root/config"
+    run_privileged cp "$tmp" "$root/config/config.alloy"
+    rm -f "$tmp"
+}
+
+copy_blackbox_config() {
+    # Single-file mount target: the unit binds this exact path into
+    # /etc/blackbox_exporter/config.yml. cp (not cp -r) keeps it one file.
+    # BLACKBOX_CONF_ROOT is the same test seam.
+    local root="${BLACKBOX_CONF_ROOT:-/opt/blackbox}"
+    run_privileged mkdir -p "$root"
+    run_privileged cp shared/blackbox/blackbox.yml "$root/blackbox.yml"
 }
 
 copy_grafana_provisioning() {
@@ -291,14 +366,25 @@ copy_grafana_provisioning() {
     # datasource would be invisible to review, lost with the data volume, and
     # unknown to anyone reading the repository. The unit already mounts
     # /opt/grafana/provisioning read-only, so this only has to populate it.
-    run_privileged mkdir -p /opt/grafana/provisioning/datasources \
-        /opt/grafana/provisioning/dashboards
+    # GRAFANA_PROV_ROOT is the test seam: point it at a sandbox and the whole
+    # copy runs unprivileged (the bats case executes this function for real).
+    local root="${GRAFANA_PROV_ROOT:-/opt/grafana}"
+    run_privileged mkdir -p "$root/provisioning/datasources" \
+        "$root/provisioning/dashboards"
     run_privileged cp shared/grafana/provisioning/datasources/prometheus.yml \
-        /opt/grafana/provisioning/datasources/prometheus.yml
+        "$root/provisioning/datasources/prometheus.yml"
     run_privileged cp shared/grafana/provisioning/dashboards/provider.yml \
-        /opt/grafana/provisioning/dashboards/provider.yml
-    run_privileged cp shared/grafana/provisioning/dashboards/llm-fleet.json \
-        /opt/grafana/provisioning/dashboards/llm-fleet.json
+        "$root/provisioning/dashboards/provider.yml"
+    # EVERY dashboard JSON rides, not a hand-listed subset: the fleet provider
+    # turns each file in the directory into a dashboard, so a copier that names
+    # files one by one silently drops new boards (measured 2026-10-21: the
+    # containers board existed in the repo since its commit and was NEVER
+    # provisioned — the panel-census browser check could not find it while the
+    # hand-placed older boards happened to be on disk). Same rule as the quadlet
+    # tier: the directory is the contract, the copy is generated state.
+    for dash in shared/grafana/provisioning/dashboards/*.json; do
+        run_privileged cp "$dash" "$root/provisioning/dashboards/$(basename "$dash")"
+    done
 }
 
 render_caddyfile() {
@@ -520,6 +606,11 @@ deploy_observability() {
             ;;
         node-exporter)
             ;;
+        cadvisor)
+            ;;
+        blackbox-exporter)
+            copy_blackbox_config
+            ;;
         *)
             echo "unknown observability service: $service" >&2
             return 2
@@ -533,6 +624,10 @@ deploy_observability() {
         grafana) wait_for_health grafana http://127.0.0.1:3002/api/health 90 ;;
         loki) wait_for_health loki http://127.0.0.1:3100/ready 60 ;;
         node-exporter) wait_for_health node-exporter http://127.0.0.1:9100/metrics 60 ;;
+        cadvisor) wait_for_health cadvisor http://127.0.0.1:8080/healthz 90 ;;
+        # Loopback: the exporter binds 127.0.0.1 + LAN (the gateway address
+        # is not bindable from the host netns — see the unit header).
+        blackbox-exporter) wait_for_health blackbox-exporter http://127.0.0.1:9115/-/healthy 60 ;;
     esac
 }
 
@@ -1368,9 +1463,10 @@ deploy_anubis() {
 # conversion (2026-10-11) of the abliterated Qwen3.8-Flash-Next —
 # qwen38-flash-next-v2-abliterated.hgn (66,687,678,432 B, ~62 GiB resident)
 # + the REQUIRED n-gram lookup table + the vision tower — MTP spec decode
-# (HALOGEN_SPEC_ADAPT=0), 2 slots over ONE 401,408-position KV pool (2 × ctx,
-# raised 2026-10-18 so BOTH slots can hold a maximum-length request) at the
-# ctx 200704 ceiling, image input — on the Strix Halo iGPU. It stays a
+# (HALOGEN_SPEC_ADAPT=0), 2 slots over ONE 401,408-position KV pool at the
+# native ctx 262144 ceiling (raised 2026-10-21 from 200704 so the harness can
+# run a 200,000 window with its ~45k fit gap; the pool still carries two
+# 200k conversations at once), image input — on the Strix Halo iGPU. It stays a
 # ROOT unit because the serve script drives `podman run` directly (the frank-archon pattern):
 # relink_unit points /etc/systemd/system/frank-dragoon.service at the repo
 # file, boot persistence is that unit's [Install], and LimitMEMLOCK=infinity
@@ -2000,7 +2096,8 @@ deploy_service() {
         plex) deploy_plex ;;
         tautulli) deploy_tautulli ;;
         alloy) deploy_alloy ;;
-        prometheus|grafana|loki|node-exporter|alertmanager) deploy_observability "$service" ;;
+        prometheus|grafana|loki|node-exporter|cadvisor|alertmanager|blackbox-exporter) deploy_observability "$service" ;;
+        gpu-exporter) deploy_gpu_exporter ;;
         comfyui) deploy_comfyui ;;
         *) deploy_compose_service "$service" ;;
     esac

scripts/deploy.sh

@@ -133,13 +133,15 @@ fi
 valid_service() {
     local service="$1"
     case "$TARGET:$service" in
+        # dankclaw:searxng-network is client-only: a network quadlet with no
+        # remote dispatch handler, so the remote gate does not (must not) list it.
         dankclaw:caddy|dankclaw:gluetun|dankclaw:searxng-network|dankclaw:searxng-valkey|\
         dankclaw:searxng|dankclaw:marreta|dankclaw:pihole|dankclaw:open-webui|dankclaw:gitea-postgres|\
         dankclaw:gitea-server|dankclaw:homepage|\
         dankclaw:comfyui|dankclaw:hermes|dankclaw:hermes-dashboard|\
         dankclaw:hermes-webui|dankclaw:prometheus|dankclaw:grafana|dankclaw:loki|dankclaw:alloy|\
-        dankclaw:alertmanager|\
-        dankclaw:node-exporter|dankclaw:authentik-postgres|dankclaw:authentik-server|\
+        dankclaw:alertmanager|dankclaw:blackbox-exporter|\
+        dankclaw:node-exporter|dankclaw:cadvisor|dankclaw:gpu-exporter|dankclaw:authentik-postgres|dankclaw:authentik-server|\
         dankclaw:authentik-worker|dankclaw:authentik-outpost|dankclaw:authentik-outpost-ldap|dankclaw:anubis|dankclaw:seerr|\
         dankclaw:open-notebook|\
         dankclaw:frank-tts-engine|dankclaw:frank-tts|\
@@ -147,8 +149,8 @@ valid_service() {
         dankserver:romm-mariadb|dankserver:romm-redis|dankserver:romm|\
         dankserver:copyparty|dankserver:jellyfin|dankserver:calibre-web|dankserver:calibre-server|dankserver:plex|dankserver:tautulli|\
         dankserver:qbittorrent|dankserver:radarr|dankserver:sonarr|\
-        dankserver:lidarr|dankserver:readarr|dankserver:node-exporter|\
-        dankserver:frank-archon|\
+        dankserver:lidarr|dankserver:readarr|dankserver:node-exporter|dankserver:cadvisor|dankserver:gpu-exporter|\
+        dankserver:prowlarr|dankserver:frank-archon|\
         frankbook-pro:gluetun|frankbook-pro:qbittorrent|frankbook-pro:prowlarr|\
         frankbook-pro:radarr|frankbook-pro:sonarr|frankbook-pro:lidarr|\
         frankbook-pro:readarr)

scripts/derive-protected-experts.py

@@ -20,9 +20,16 @@ of the Gram is its down-input energy. Rank halves: 0-255, 256-511
 Output: the cast-rule lines (recipe format) on stdout, a report on stderr.
 
 Usage: derive-protected-experts.py <calib dir> [--min-share 0.03]
-       [--min-route-mult 3.0]
+       [--min-route-mult 3.0] [--target-count 0]
+       --target-count N fills the byte bill: the bars pick the base set exactly as
+       the bar-only pass does, then the energy ranking admits experts one at a time
+       (each with its parity partner) until the count lands. The energy bar NEVER
+       relaxes, so asking for 48 bf16 experts never quietly buys 80; the ratio bar
+       walks down only when the ranking cannot produce that many candidates.
+       0 takes the bars as given.
 """
 import argparse
+import collections
 import glob
 import os
 import re
@@ -32,6 +39,7 @@ import sys
 import numpy as np
 
 NAME_RE = re.compile(r"^gram\.r(\d)\.blk\.(\d+)\.ffn_down_exps\.(\d+)\.bin$")
+CAST_RE = re.compile(r"^blk\.(\d+)\.ffn_\*_exps\.(\d+)\.weight")
 
 
 def read_gram(path):
@@ -49,27 +57,8 @@ def read_gram(path):
     return k, rank, int(rows), tr
 
 
-def main():
-    ap = argparse.ArgumentParser()
-    ap.add_argument("calib_dir")
-    ap.add_argument("--min-share", type=float, default=0.03)
-    ap.add_argument("--min-route-mult", type=float, default=3.0)
-    a = ap.parse_args()
-
-    # layer -> global expert -> (rank, rows, energy)
-    data = {}
-    for path in sorted(glob.glob(os.path.join(
-            a.calib_dir, "gram.r*.ffn_down_exps.*.bin"))):
-        m = NAME_RE.match(os.path.basename(path))
-        if not m:
-            continue
-        rank, layer, expert = int(m.group(1)), int(m.group(2)), int(m.group(3))
-        k, _, rows, tr = read_gram(path)
-        data.setdefault(layer, {})[expert] = (rank, rows, tr)
-
-    if not data:
-        raise SystemExit("no per-expert down grams found")
-
+def select(data, min_share, min_route_mult):
+    """(cast lines, report lines, count) for one pair of bars."""
     lines, report = [], []
     total = 0
     for layer in sorted(data):
@@ -83,7 +72,7 @@ def main():
         for e, (rank, rows, tr) in experts.items():
             es, rs = tr / tot_e, rows / max(tot_r, 1)
             rec = (e, rank, es, rs)
-            if es >= a.min_share and es >= a.min_route_mult * rs:
+            if es >= min_share and es >= min_route_mult * rs:
                 qual.append(rec)
             elif es > 0:
                 pool.append(rec)
@@ -107,10 +96,148 @@ def main():
                           f"energy {es*100:5.2f}%, routing {rs*100:5.3f}%, "
                           f"ratio {es/max(rs,1e-12):6.1f}")
         total += len(chosen)
+    return lines, report, total
+
+
+def parity_repair(chosen, by_half, report, only=None):
+    """Add the next-best candidate to every half — or to `only` — holding an odd count.
+
+    The four-bit GEMM takes each half's remainder as two equal tables, so an odd count
+    of bf16 experts on a half is not a style problem: the two tables would not match.
+    `by_half` is energy-descending, so the first unchosen candidate is the next most
+    energetic, which is the expert worth keeping. Returns how many were added.
+    """
+    added = 0
+    for key in ([only] if only is not None else sorted(by_half)):
+        layer, half = key
+        n = sum(1 for (l, x) in chosen if l == layer and x // 256 == half)
+        if n % 2 == 0:
+            continue
+        for e, es, rs in by_half.get(key, []):
+            if (layer, e) not in chosen:
+                chosen[(layer, e)] = (es, rs)
+                added += 1
+                report.append(f"blk.{layer}: half {half} odd -> added e{e} "
+                              f"(energy {es*100:.2f}%)")
+                break
+        else:
+            report.append(f"blk.{layer}: half {half} holds {n} and no candidate is "
+                          f"left to make it even")
+    return added
+
+
+def select_to_count(data, min_share, min_route_mult, target):
+    """(lines, report, total, cutoff) for the set that LANDS the count.
+
+    The bars decide the base set; the count is then filled from the energy ranking,
+    one expert at a time, each addition taking its half's next-best candidate with it
+    so every rank's half stays even. Only the RATIO bar walks down to find candidates
+    — it is the law that says an expert is disproportionate and not merely busy, and
+    the caller's byte bill has to be filled — while the ENERGY bar holds, because that
+    bar is what the extra GiB of bf16 is supposed to buy. `cutoff` is the energy share
+    the set bottoms out at: the bar this campaign actually converted with.
+    """
+    report = []
+    shares, ranked = {}, []
+    for layer in sorted(data):
+        experts = data[layer]
+        tot_e = sum(v[2] for v in experts.values())
+        tot_r = sum(v[1] for v in experts.values())
+        if tot_e <= 0 or tot_r <= 0:
+            report.append(f"blk.{layer}: empty recording, skipped")
+            continue
+        for e, (_rank, rows, tr) in experts.items():
+            es, rs = tr / tot_e, rows / max(tot_r, 1)
+            if es <= 0:
+                continue
+            shares[(layer, e)] = (es, rs)
+            ranked.append((es, layer, e, rs))
+    ranked.sort(key=lambda r: (-r[0], r[1], r[2]))
+    by_half = {}
+    for es, layer, e, rs in ranked:
+        by_half.setdefault((layer, e // 256), []).append((e, es, rs))
+
+    chosen = {k: v for k, v in shares.items()
+              if v[0] >= min_share and v[0] >= min_route_mult * v[1]}
+    total = len(chosen)
+    total += parity_repair(chosen, by_half, report)
+
+    mult = min_route_mult
+    while True:
+        pool = [(es, l, e, rs) for es, l, e, rs in ranked
+                if es >= mult * rs and (l, e) not in chosen]
+        if len(pool) + total >= target or mult <= 0.0:
+            break
+        mult -= 0.5
+        report.append(f"ratio bar {mult + 0.5:g} leaves {len(pool)} fill candidates "
+                      f"for {target - total} slots; relaxing toward 0")
+    for es, layer, e, rs in pool:
+        if total >= target:
+            break
+        if (layer, e) in chosen:
+            continue
+        chosen[(layer, e)] = (es, rs)
+        total += 1
+        report.append(f"blk.{layer}: count fill -> added e{e} (energy {es*100:.2f}%)")
+        total += parity_repair(chosen, by_half, report, only=(layer, e // 256))
+    lines, cutoff = [], None
+    for (layer, e) in sorted(chosen):
+        es, rs = chosen[(layer, e)]
+        cutoff = es if cutoff is None else min(cutoff, es)
+        lines.append(f"blk.{layer}.ffn_*_exps.{e}.weight         cast dtype=bf16")
+    return lines, report, total, cutoff
+
+
+def main():
+    ap = argparse.ArgumentParser()
+    ap.add_argument("calib_dir")
+    ap.add_argument("--min-share", type=float, default=0.03)
+    ap.add_argument("--min-route-mult", type=float, default=3.0)
+    ap.add_argument("--target-count", type=int, default=0)
+    a = ap.parse_args()
+
+    # layer -> global expert -> (rank, rows, energy)
+    data = {}
+    for path in sorted(glob.glob(os.path.join(
+            a.calib_dir, "gram.r*.ffn_down_exps.*.bin"))):
+        m = NAME_RE.match(os.path.basename(path))
+        if not m:
+            continue
+        rank, layer, expert = int(m.group(1)), int(m.group(2)), int(m.group(3))
+        k, _, rows, tr = read_gram(path)
+        data.setdefault(layer, {})[expert] = (rank, rows, tr)
+
+    if not data:
+        raise SystemExit("no per-expert down grams found")
+
+    if a.target_count > 0:
+        lines, report, total, cutoff = select_to_count(
+            data, a.min_share, a.min_route_mult, a.target_count)
+        bar = f"energy cutoff {cutoff:.3g}" if cutoff is not None else "no candidates"
+    else:
+        lines, report, total = select(data, a.min_share, a.min_route_mult)
+        bar = f"min-share {a.min_share:.3g}"
+
+    # The emitted lines ARE the contract with the four-bit GEMM: every rank's half of
+    # every layer must hold an even number of bf16 experts. Judged on the output and
+    # not on the internal sets, so a slip in either pass is caught here rather than at
+    # engine load time.
+    halves = collections.Counter()
+    for rule in lines:
+        m = CAST_RE.match(rule)
+        if m:
+            halves[(int(m.group(1)), int(m.group(2)) // 256)] += 1
+    odd = {key: n for key, n in halves.items() if n % 2}
+    if odd:
+        print("PARITY LAW BROKEN: " + ", ".join(
+            f"blk.{lay} half {half} holds {n} protected experts"
+            for (lay, half), n in sorted(odd.items())), file=sys.stderr)
+        raise SystemExit(2)
 
     print("\n".join(lines))
     print("\n".join(report), file=sys.stderr)
-    print(f"total protected: {total}", file=sys.stderr)
+    print(f"total protected: {total} (target {a.target_count or 'none'}, {bar}, "
+          f"min-route-mult {a.min_route_mult:g})", file=sys.stderr)
 
 
 if __name__ == "__main__":

scripts/download-qwen38-bf16.sh

@@ -0,0 +1,249 @@
+#!/usr/bin/env bash
+# download-qwen38-bf16.sh — RUNS ON DANKSERVER. Fetch and verify the vendor
+# BF16 checkpoint Qwen/Qwen3.8-Flash-Next: the source the five-bit campaign
+# converts, and the source the KLD reference is recorded from.
+#
+# Why the base and not a finetune: the campaign converts to compare against the
+# publisher's own iq4r container, and that container was built from this
+# checkpoint. Base-in, base-out; anything else confounds the gate. (The
+# predecessor campaign ran ukisai/Swift-1.5-Qwen3.8-Flash-Next-BF16, whose key
+# set is identical to this repo's but whose weights are not — none of its
+# calibration grams, KLD reference or protected list carries over.)
+#
+# Why re-fetch instead of reusing: every BF16 copy of this family was destroyed
+# in the 2026-10-15 storage purge — only the 113.6 GiB served .rad survived.
+# 335.3 GiB of safetensors, 131 shards, ungated (the predecessor campaign
+# verified the key set against this repo, so no gate letter is needed).
+#
+# Verification is four gates deep, because the failure this repo actually lived
+# through is a fetch that reported success: `curl -o FILE URL` exits 0 on an
+# HTTP 401/403 and writes the error body as the artifact — three shards of
+# identical tiny size and a cheerful DONE in 20 seconds for 104 GiB.
+#
+#   1. shard census: the file count must equal the expected shard count, so a
+#      partially fetched fragment tree (the .partNofM trap) cannot pass.
+#   2. per-shard completeness, from the shard's OWN safetensors header: the
+#      header declares where the tensor data ends, so every shard's true size is
+#      known offline. A torn tail is named here, by file, instead of surfacing
+#      as a load error five hours later.
+#   3. index coverage: every shard model.safetensors.index.json references
+#      exists and is named.
+#   4. total bytes against the number the Hub declared, then, unless --no-hub,
+#      every file's sha256 against the Hub's LFS record — including the shards
+#      already on disk, so a previous partial download is never trusted.
+#
+# The token never enters a command line (the 2026-09-16 leak law): it is read
+# into HF_TOKEN in the environment, where a local process listing is the only
+# reader. This repo is ungated, so the fetch simply runs without one when no
+# token file exists. scripts/hfget.sh is curl+jq only — no Python, no venv, no
+# hf_transfer — and it drops the pickle/ONNX weight duplicates by itself.
+#
+# usage: download-qwen38-bf16.sh [--fetch] [--no-hub] [--dir DIR]
+#                                [--shards N] [--bytes B] [--repo ID]
+#   --fetch    download first (resumable: re-run to continue), then verify
+#   --no-hub   skip the Hub-side sha256 sweep; the other gates always run
+# exit: 0 verified, 1 verification failed, 2 environment / usage error
+set -euo pipefail
+
+REPO="${FRANK_QWEN38_HF_REPO:-Qwen/Qwen3.8-Flash-Next}"
+LOCAL_DIR="${FRANK_QWEN38_CKPT:-/home/dank/models/Qwen3.8-Flash-Next}"
+# Hub census of 2026-10-27: 144 files, 131 of them .safetensors, together
+# 360,023,351,514 bytes; the shards alone are 360,000,192,888. This gate counts
+# shards only, so a config file edited on the Hub cannot fail it.
+EXPECTED_SHARDS="${FRANK_QWEN38_SHARDS:-131}"
+EXPECTED_BYTES="${FRANK_QWEN38_BYTES:-360000192888}"
+MIN_FREE_MIB="${FRANK_QWEN38_MIN_FREE_MIB:-350000}"
+
+FETCH=0
+HUB=1
+while [ $# -gt 0 ]; do
+    case "$1" in
+        --fetch) FETCH=1 ;;
+        --no-hub) HUB=0 ;;
+        --dir)    LOCAL_DIR="$2"; shift ;;
+        --shards) EXPECTED_SHARDS="$2"; shift ;;
+        --bytes)  EXPECTED_BYTES="$2"; shift ;;
+        --repo)   REPO="$2"; shift ;;
+        *) echo "usage: ${0##*/} [--fetch] [--no-hub] [--dir DIR] [--shards N] [--bytes B] [--repo ID]" >&2
+           exit 2 ;;
+    esac
+    shift
+done
+
+SELF_DIR="$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)"
+FAILS=0
+note() { echo "$@"; FAILS=$((FAILS + 1)); }
+
+if [ "$FETCH" = "1" ]; then
+    # Every fetch gate is a hard stop: skipping the download silently and then
+    # verifying a half tree is how a campaign ships a stale payload.
+    HFGET="$SELF_DIR/hfget.sh"
+    if [ ! -f "$HFGET" ]; then
+        echo "REFUSED: the fetch helper is not here: $HFGET" >&2
+        exit 2
+    fi
+    if ! command -v jq >/dev/null 2>&1; then
+        echo "REFUSED: hfget.sh needs jq on PATH" >&2
+        exit 2
+    fi
+    mkdir -p "$LOCAL_DIR"
+    # The campaign's peak-disk law starts at the fetch: the checkpoint and the
+    # reference container built from it are ~335 GiB each, and a tree that does
+    # not fit is discovered hours into a transfer, when it costs the most.
+    free_mib=$(df -m --output=avail "$LOCAL_DIR" | tail -1 | tr -dc '0-9')
+    if [ "${free_mib:-0}" -lt "$MIN_FREE_MIB" ]; then
+        echo "REFUSED: $free_mib MiB free at $LOCAL_DIR; the fetch wants $MIN_FREE_MIB MiB free" >&2
+        echo "         (retire the peak-disk predecessors first — nothing is downloaded, so" >&2
+        echo "         nothing is left half-written)" >&2
+        exit 2
+    fi
+    echo "fetch: $REPO -> $LOCAL_DIR ($free_mib MiB free)"
+    if [ -z "${HF_TOKEN:-}" ]; then
+        CREDSTORE="$HOME/.config/huggingface/credentials/https://huggingface.co"
+        if [ -f "$CREDSTORE" ]; then
+            HF_TOKEN="$(awk '/^ OauthToken: / {print $2; exit}' "$CREDSTORE")"
+        elif [ -f "$HOME/.cache/huggingface/token" ]; then
+            # The legacy single-line token file the hub library writes.
+            HF_TOKEN="$(tr -d '[:space:]' < "$HOME/.cache/huggingface/token")"
+        elif [ -f "$HOME/.hf_token" ]; then
+            HF_TOKEN="$(tr -d '[:space:]' < "$HOME/.hf_token")"
+        fi
+    fi
+    if [ -n "${HF_TOKEN:-}" ]; then
+        export HF_TOKEN
+    fi
+    bash "$HFGET" "$REPO" "$LOCAL_DIR" main
+    unset HF_TOKEN
+fi
+
+if [ ! -d "$LOCAL_DIR" ]; then
+    echo "REFUSED: $LOCAL_DIR does not exist (run with --fetch, or pass --dir)" >&2
+    exit 1
+fi
+INDEX="$LOCAL_DIR/model.safetensors.index.json"
+if [ ! -f "$INDEX" ]; then
+    echo "REFUSED: $INDEX is missing — the checkpoint tree is not addressable without it" >&2
+    exit 1
+fi
+
+echo "== gate 1: shard census ($LOCAL_DIR)"
+SHARDS=$(find "$LOCAL_DIR" -maxdepth 1 -type f -name '*.safetensors' | wc -l)
+if [ "$SHARDS" != "$EXPECTED_SHARDS" ]; then
+    note "FAIL SHARD_COUNT: $SHARDS safetensors files on disk, expected $EXPECTED_SHARDS"
+else
+    echo "  ok: $SHARDS shards"
+fi
+
+echo "== gate 2: every shard is whole, judged by its own header"
+# A safetensors file is [u64 header length][JSON header][tensor data]. The JSON
+# declares where the data ends, so the file's true size is computable offline:
+# 8 + header + the largest end offset. Truncation and error bodies both fall
+# here, and each is reported with the file that caused it.
+while IFS= read -r f; do
+    state=$(python3 - "$f" <<'PY'
+import json, os, struct, sys
+path = sys.argv[1]
+size = os.path.getsize(path)
+try:
+    with open(path, "rb") as fh:
+        (hlen,) = struct.unpack("<Q", fh.read(8))
+        if hlen == 0 or hlen > size:
+            print("NOT_SAFETENSORS")
+            raise SystemExit
+        header = json.loads(fh.read(hlen))
+except Exception:
+    print("NOT_SAFETENSORS")
+    raise SystemExit
+if not header:
+    print("NOT_SAFETENSORS")
+    raise SystemExit
+end = max((v["data_offsets"][1] for v in header.values()
+           if isinstance(v, dict) and "data_offsets" in v), default=0)
+want = 8 + hlen + end
+print("OK" if want == size else f"TRUNCATED want={want} have={size}")
+PY
+)
+    case "$state" in
+        OK) ;;
+        NOT_SAFETENSORS*) note "FAIL NOT_SAFETENSORS: $(basename "$f") is not a safetensors file (an error body written as an artifact?)" ;;
+        TRUNCATED*) note "FAIL TRUNCATED: $(basename "$f") $state" ;;
+    esac
+done < <(find "$LOCAL_DIR" -maxdepth 1 -type f -name '*.safetensors' | sort)
+if [ "$FAILS" -eq 0 ]; then
+    echo "  ok: every shard's data extent matches its size on disk"
+fi
+
+echo "== gate 3: the index's shards are all present"
+INDEX_SHARDS=$(python3 - "$INDEX" <<'PY'
+import json, sys
+map_ = json.load(open(sys.argv[1]))["weight_map"]
+print("\n".join(sorted(set(map_.values()))))
+PY
+)
+while IFS= read -r name; do
+    [ -n "$name" ] || continue
+    if [ ! -f "$LOCAL_DIR/$name" ]; then
+        note "FAIL MISSING: $name is named by the index and is not on disk"
+    fi
+done <<< "$INDEX_SHARDS"
+if [ "$FAILS" -eq 0 ]; then
+    echo "  ok: $(printf '%s\n' "$INDEX_SHARDS" | grep -c .) referenced shards on disk"
+fi
+
+echo "== gate 4: total bytes vs the Hub's declaration"
+TOTAL=$(find "$LOCAL_DIR" -maxdepth 1 -type f -name '*.safetensors' -printf '%s\n' | awk '{s+=$1} END{print s+0}')
+if [ "$TOTAL" != "$EXPECTED_BYTES" ]; then
+    note "FAIL TOTAL_BYTES: shards total $TOTAL bytes, expected $EXPECTED_BYTES"
+else
+    echo "  ok: $TOTAL bytes"
+fi
+
+if [ "$HUB" = "1" ]; then
+    echo "== gate 5: every file's sha256 vs the Hub's LFS record"
+    # size = file size; lfs.sha256 = sha256 of the file itself (LFS or not).
+    TMPD="$(mktemp -d)"
+    trap 'rm -rf "$TMPD"' EXIT
+    MANIFEST="$TMPD/tree.json"
+    PLAN="$TMPD/plan"
+    curl -sf "https://huggingface.co/api/models/$REPO/tree/main?recursive=true&expand=true" \
+        -o "$MANIFEST" || { echo "FAIL: the Hub API call failed — refusing to report success" >&2; exit 1; }
+    python3 - "$MANIFEST" > "$PLAN" <<'PY'
+import json, sys
+for e in json.load(open(sys.argv[1])):
+    if e.get("type") != "file":
+        continue
+    sha = (e.get("lfs") or {}).get("sha256") or e.get("sha") or ""
+    if sha:
+        print(e["path"], e["size"], sha)
+PY
+    BAD=0
+    while read -r rel size sha; do
+        [ -n "${rel:-}" ] || continue
+        if [ ! -f "$LOCAL_DIR/$rel" ]; then
+            note "FAIL MISSING: $rel is on the Hub and not on disk"
+            continue
+        fi
+        have=$(stat -c %s "$LOCAL_DIR/$rel")
+        if [ "$have" != "$size" ]; then
+            note "FAIL SIZE: $rel is $have bytes, Hub says $size"
+            continue
+        fi
+        got=$(sha256sum "$LOCAL_DIR/$rel" | awk '{print $1}')
+        if [ "$got" != "$sha" ]; then
+            note "FAIL SHA256: $rel got $got want $sha"
+            BAD=$((BAD + 1))
+        fi
+    done < "$PLAN"
+    if [ "$BAD" -eq 0 ] && [ "$FAILS" -eq 0 ]; then
+        echo "  ok: sha256 matches for every file"
+    fi
+    rm -rf "$TMPD"
+    trap - EXIT
+fi
+
+if [ "$FAILS" -gt 0 ]; then
+    echo "VERIFY_FAILED: $FAILS problem(s) in $LOCAL_DIR" >&2
+    exit 1
+fi
+echo "VERIFY_OK: $SHARDS shards, $TOTAL bytes of safetensors in $LOCAL_DIR"
+echo "NEXT: build the BF16 reference container with scripts/build-qwen38-bf16-rad.sh"

scripts/frank-archon.env

@@ -46,31 +46,35 @@ FRANK_ARCHON_TP=2
 # stays EXACT for single-sequence decode and any step of <= 25 rows — see
 # MAX_SEQS x (1 + SPEC_TOKENS) below.
 FRANK_ARCHON_TP_WIRE=wht6
-# Decode slots. A slot costs KV, never decode; the pool is sized to this
-# contract and nothing more. At 1.8.1 the budget resolver caps the cache at
-# 556,348 fp8 tokens — 2.8 sessions at 200,000 — so a THIRD full-height
-# conversation answers 400 while /health and every shorter chat work fine
-# (3 x 50k streams run at 77.6 t/s aggregate). Frank's trade: declare the two
-# slots the engine can hold and let the freed ~1.1 GiB go to the expert plane,
-# instead of advertising a third slot that only exists at shorter prompts.
-FRANK_ARCHON_MAX_SEQS=2
+# Decode slots. A slot costs KV and pinned RAM, never decode: the cache target is
+# exactly MAX_SEQS x MAX_MODEL_LEN and the checkpoint store asks (1 + what the
+# retention policy keeps) per slot. Ordered 2026-10-20: FOUR slots over a pool of at
+# least 700,000 fp8 tokens, rounding UP if the resolver cannot land it exactly. The
+# target is therefore 4 x 200,000 = 800,000 tokens and HOST_POOL_MIB decides whether
+# the cache gets it (expert_vram_floor, core/place/planner.cpp:100-104: the card must
+# keep expert-plane-minus-host-room). The 2-slot shape was the answer to a
+# 365,204-token pool that backed 1.8 full-height sessions; at this pool the third and
+# fourth conversations are backed by the cache instead of over-committing it.
+FRANK_ARCHON_MAX_SEQS=4
 # Served window. The engine treats `kv group tokens >= max-model-len` as a HARD
 # limit, and models.yml contextWindow MUST quote this number (over-length is a
 # refusal, not a truncation). The pool is sized for seqs x maxlen, not one
 # sequence.
 FRANK_ARCHON_MAX_MODEL_LEN=200000
 FRANK_ARCHON_PLACEMENT=expert_tiered
-# Pinned host pool for expert rows that do not fit VRAM. Under the budget
-# resolver this is ALSO the lever that decides which experts stay on the card:
-# the card's leftover goes to the expert plane first and the KV cache takes the
-# rest, so growing the pool moves experts off the card and shrinking it spills
-# them onto VRAM the cache wanted. Under-sizing it is a boot failure (36,864
-# refused a boot). The ledger's coverage line is the referee, not this file.
-# Measured, not guessed: 45,568 (PR #30) moved host_pool to 45.09 GiB and left
-# vram_kv byte-identical at 4.20 GiB / 556,348 tokens on both ranks — extra
-# host RAM above what the expert plane's footprint needs buys no cache. Back at
-# 44,928, the size that boots clean.
-FRANK_ARCHON_HOST_POOL_MIB=44928
+# Pinned host pool for expert rows that do not fit VRAM, PINNED ONCE PER RANK: the
+# host must pin tp x this number, and the refusal that names it is
+# core/engine_bringup.cpp:250-265. Under the budget resolver this is ALSO the KV
+# lever -- the card keeps expert-plane-minus-this-room, so every MiB added moves one
+# MiB of experts off the card and into the cache. Derived from the boot ledger (a
+# 23.73 GiB floor at 44,928 room puts plane + 2 units at 67.60 GiB a rank), 49,152
+# drops the floor to 19.6 GiB and leaves the cache ~6.9 GiB, which is enough to back
+# the whole 4 x 200,000 target; the leftover returns to the experts. Cost is paid
+# twice over: pinned RAM (tp x 48 GiB, against the TTM target below) and expert
+# residency -- ~3.2 GiB a card of experts now lives in host RAM and is read over the
+# x8 link when the router picks it. The ledger's coverage line is the referee, not
+# this file: if vram_kv comes back under 700,000 tokens, raise this 1024 and redeploy.
+FRANK_ARCHON_HOST_POOL_MIB=49152
 # VRAM left unclaimed per card (publisher default 96 MiB); the engine's settle
 # floor is built from it.
 FRANK_ARCHON_GPU_HEADROOM_MIB=96
@@ -93,16 +97,29 @@ FRANK_ARCHON_SPEC_TOKENS=2
 FRANK_ARCHON_MAX_BATCHED_TOKENS=2048
 # Step sharing. A prefill chunk that shares a step with decoding sequences holds
 # the device for the chunk's whole time, and every decode row in that step waits
-# that long for its next token (the census: [[LLM Performance Diagnosis]] §9).
-# `auto` leaves each decode row behind the chunk a row plus its draft window and
-# keeps a share for whatever is waiting to be admitted; a number pins the chunk's
-# cap; `off` is the pre-1.4 behaviour this endpoint ran for two eras, where the
-# chunk took the step and admission never ran. Interleave inserts N decode-only
-# steps after a chunk step taken beside decoders: it protects their cadence and
-# holds a waiting request out of those steps, so it stays 0 until the post-migration
-# decode-vs-prefill buckets say it is wanted.
+# that long for its next token: measured at this shape, decode-only steps 15.5 ms
+# against chunk steps 166 ms, which is the harness's ~146 t/s falling to ~11 t/s
+# while another session prefills (law and census: [[frank-archon]]).
+# `auto` is a PREFILL-SPEED dial, not a decoder dial: it reserves one row plus the
+# draft window per decoding sequence (3 tokens) and an eighth of the budget per
+# waiting request or waiting prefill, and the source states it will not halve for
+# decode rows alone. Live proof: prefill_chunks_capped 5,018 across 62,386 chunk
+# steps. A number is the only value that applies whenever the chunk shares a step,
+# and it must not go under the prefill stager's smallest pass -- 1025 rows here,
+# named on the boot line "prefill staging: ... a step of more than 1024 tokens" --
+# because below it every routed layer reads its pooled experts synchronously, so
+# 512 is a trap and the vendor measures the whole job slower there. Interleave
+# inserts N decode-only steps after a chunk step taken beside decoders: it protects
+# their cadence and holds a waiting request out of those steps. That is the decoder
+# dial. ARMED at 1 on 2026-10-20: at N=1 a stream takes its token from the chunk step
+# AND one from the held step, ~2 tokens per 182 ms instead of one per 166 ms; the
+# price falls on the prefilling prompt (its chunks lose every held step) and on any
+# request still waiting to enter. `--prefill-span` re-funds a prefill only when it is
+# ALONE, so it cannot buy this back while decoders run. Meter:
+# radiance:engine_steps_held_total must leave zero, and the §9 buckets must move
+# before N rises.
 FRANK_ARCHON_BUSY_PREFILL_CHUNK=auto
-FRANK_ARCHON_PREFILL_INTERLEAVE=0
+FRANK_ARCHON_PREFILL_INTERLEAVE=1
 # Prefix cache: pinned host memory only. The disk tier was removed outright
 # (--prefix-cache-dir and --prefix-cache-disk-mib are refusals), so what used to
 # be restored from NVMe is now re-prefilled. The tier is sized in ENTRIES and one
@@ -118,9 +135,10 @@ FRANK_ARCHON_NGRAM_PLACEMENT=ram
 # RESETS pages_limit to half of RAM on every boot; below the plan the load dies
 # mid-pin with an anonymous allocator error. ttm-preflight.sh raises the knob
 # itself before refusing by name, so recovery is a boot, not a visit.
-# Floor = 2 x HOST_POOL + 48,824 (static weights + arena) + PREFIX_CACHE + MNBT
-# per the bats ledger test; at the 44,928 pool that computes to 148,920 MiB.
-FRANK_ARCHON_TTM_MIN_MIB=150000
+# Floor = 2 x HOST_POOL + 48,824 (static weights + arena) + PREFIX_CACHE + MNBT + the
+# auto checkpoint store (MAX_SEQS x 10 slots of 111.98 MiB), per the bats ledger test;
+# at the 49,152 pool with 4 slots that computes to 161,848 MiB.
+FRANK_ARCHON_TTM_MIN_MIB=162000
 FRANK_ARCHON_TTM_TARGET_MIB=204800
 # Escape hatch for experiments. Unset -> no extra argv; the serve script expands
 # this unquoted on purpose, so keep multi-flag values as one string.

scripts/frank-dragoon.env

@@ -49,14 +49,21 @@ FRANK_DRAGOON_TOKENIZER_FILES="tokenizer.json chat_template.jinja tokenizer_conf
 # stays at the image default, which is what was qualified) ------------------
 # Served id. OpenWebUI api_configs model_ids for the :7770 backend MUST name it.
 HALOGEN_MODEL_ID=frank-dragoon
-# PER-REQUEST ceiling: a longer prompt is a 400 naming the limit, never a
-# truncation. The harness window must sit ~45k UNDER this (the 400-zone law on
-# the page) — a window equal to the ceiling makes every long request a 400.
-HALOGEN_CTX=200704
-# ONE pool behind both slots, sized so BOTH can carry a maximum-length request
-# at once. Pool positions buy concurrency; the per-request window never moves
-# with them. Legal band: ctx <= pool <= slots * ctx (the seam enforces it), and
-# RAM the pool takes is RAM the n-gram table's page cache loses.
+# PER-REQUEST ceiling, set to the checkpoint's NATIVE window (262,144 — a
+# longer prompt is a 400 naming the limit, never a truncation; stretching
+# past native would need HALOGEN_ROPE_YARN and is not on plan). The harness
+# `contextWindow` must sit ~45k under THIS number (the 400-zone law on the
+# page): with the window at 200,000 the gap is 62,144, which absorbs the
+# chars/4 under-count. A window at or near the ceiling makes every long
+# request a guaranteed 400.
+HALOGEN_CTX=262144
+# ONE pool behind both slots. The harness window is 200,000, so 401,408
+# still carries TWO maximum harness-length conversations at once
+# (2 x 200,000 + one 45,056 budget <= pool + cap headroom); two literal
+# ceiling-length requests would queue instead — buying those means raising
+# the pool, and RAM the pool takes is RAM the n-gram table's page cache
+# loses. Pool positions buy concurrency; the per-request window never moves
+# with them. Legal band: ctx <= pool <= slots * ctx (the seam enforces it).
 HALOGEN_KV_POOL_POSITIONS=401408
 # Two concurrent conversations: the MTP drafter is not slot-locked on this
 # engine, so the second slot is capacity, not a speculation tax.
@@ -107,10 +114,13 @@ HALOGEN_NPU_MODELS=qwen3-embedding-0.6b
 FRANK_DRAGOON_NPU_DIR=/home/dank/models/halogen-v2-abliterated/npu
 # Node paths and the XRT directory are variables so the bats seam can drive
 # these gates on a box with no NPU. Ubuntu's distro XRT (libxrt2 + libxrt-npu2)
-# is mounted FILE BY FILE and TWICE EACH — once where the image looks for XRT,
-# once at its own path, because those libraries find each other at their install
-# path; the vendor forbids mounting an /opt/xilinx/xrt of symlinks into the
-# system dir, which dangle inside the container.
+# is mounted FILE BY FILE and THREE TIMES EACH — where the image looks for XRT
+# (/opt/xilinx/xrt/lib), at its own path (those libraries find each other at
+# their install path), and at /opt/xilinx/lib/<multiarch>/ (the NPU child
+# searches that third prefix; without it the lane dies naming it and the
+# supervisor takes the container down — 2026-10-21 boot loop). The basename of
+# this directory IS the multiarch name. The vendor forbids mounting an
+# /opt/xilinx/xrt of symlinks into the system dir, which dangle in the container.
 FRANK_DRAGOON_DEV_ACCEL=/dev/accel/accel0
 FRANK_DRAGOON_XRT_DIR=/usr/lib/x86_64-linux-gnu
 # sysfs root the fabric-clock preflight reads (same knob the vendor's

scripts/frank-dragoon.service

@@ -1,5 +1,5 @@
 [Unit]
-Description=halogen-flash-server container (Quat3rnion NATIVE v2 abliterated Qwen3.8-Flash-Next, MTP spec decode, 2 slots over ONE 401408-position pool at ctx 200704, vision ON), root systemd unit (dankclaw, frank-dragoon, Strix Halo iGPU)
+Description=halogen-flash-server container (Quat3rnion NATIVE v2 abliterated Qwen3.8-Flash-Next, MTP spec decode, 2 slots over ONE 401408-position pool at ctx 262144, vision ON), root systemd unit (dankclaw, frank-dragoon, Strix Halo iGPU)
 After=network-online.target
 Wants=network-online.target
 

scripts/gpu-exporter.py

@@ -0,0 +1,212 @@
+#!/usr/bin/env python3
+"""gpu-exporter: unified per-GPU metrics for a mixed AMD/NVIDIA host.
+
+Emits the `gpu_*` Prometheus families the host-system dashboard graphs:
+
+    gpu_utilization_percent  gpu_memory_used_bytes  gpu_memory_total_bytes
+    gpu_temperature_celsius  gpu_power_watts
+
+each labelled {gpu, vendor, model}. The scrape job adds `host`.
+
+Two sources, both host-native — no container, no GPU passthrough:
+
+  * AMD (R9700s, the 8060S iGPU, the Raphael display GPU): the amdgpu sysfs tree.
+    Utilization comes ONLY from gpu_busy_percent — amd-smi reports gfx_busy_inst
+    as N/A on gfx1201 (verified 2026-10-20). VRAM is already bytes, temperature
+    is milli-degrees, power is micro-watts. Crucially power1_average is EMPTY on
+    the R9700s, so those cards legitimately have no gpu_power_watts line; a
+    missing file becomes an ABSENT metric, never a fabricated 0 that a dashboard
+    would plot as a real reading.
+
+  * NVIDIA (the RTX 3090 on Dankclaw): one `nvidia-smi --query-gpu` CSV. On
+    Dankserver there is an nvidia-smi binary but no NVIDIA GPU, so the command
+    fails; that is tolerated and simply yields no NVIDIA series.
+
+Default mode prints one scrape to stdout and exits (the test harness, and any
+`service ... --oneshot` one-off). `--serve` binds the HTTP endpoint for
+Prometheus; it binds EXPLICITLY (loopback + the host LAN address), never a
+wildcard, matching node-exporter's posture on these boxes.
+"""
+import argparse
+import glob
+import os
+import subprocess
+import sys
+import threading
+from http.server import BaseHTTPRequestHandler, ThreadingHTTPServer
+
+FAMILIES = [
+    ("gpu_utilization_percent", "gauge", "GPU busy time as a percentage of wall time."),
+    ("gpu_memory_used_bytes", "gauge", "Device memory in use, in bytes."),
+    ("gpu_memory_total_bytes", "gauge", "Device memory capacity, in bytes."),
+    ("gpu_temperature_celsius", "gauge", "Package temperature in degrees Celsius."),
+    ("gpu_power_watts", "gauge", "Board power draw in watts."),
+]
+
+DEFAULT_NVIDIA_CMD = (
+    "nvidia-smi --query-gpu=index,name,utilization.gpu,memory.used,"
+    "memory.total,temperature.gpu,power.draw --format=csv,noheader,nounits"
+)
+
+
+def _read(path):
+    try:
+        with open(path) as fh:
+            return fh.read().strip()
+    except OSError:
+        return ""
+
+
+def _pci_slot(device_dir):
+    uevent = _read(os.path.join(device_dir, "uevent"))
+    for line in uevent.splitlines():
+        if line.startswith("PCI_SLOT_NAME="):
+            return line.split("=", 1)[1]
+    return ""
+
+
+def _pci_id(device_dir):
+    uevent = _read(os.path.join(device_dir, "uevent"))
+    for line in uevent.splitlines():
+        if line.startswith("PCI_ID="):
+            return "amd:" + line.split("=", 1)[1].lower()
+    return "amd"
+
+
+def collect_amd(sys_dir):
+    for card in sorted(glob.glob(os.path.join(sys_dir, "card[0-9]*"))):
+        device = os.path.join(card, "device")
+        busy = _read(os.path.join(device, "gpu_busy_percent"))
+        if busy == "":
+            continue  # not an amdgpu compute node (e.g. a render node w/o metrics)
+        labels = {"gpu": _pci_slot(device) or os.path.basename(card),
+                  "vendor": "amd", "model": _pci_id(device)}
+        yield "gpu_utilization_percent", labels, busy
+        for name, leaf in (("gpu_memory_used_bytes", "mem_info_vram_used"),
+                           ("gpu_memory_total_bytes", "mem_info_vram_total")):
+            v = _read(os.path.join(device, leaf))
+            if v != "":
+                yield name, labels, v
+        temp = ""
+        for hwmon in sorted(glob.glob(os.path.join(device, "hwmon", "hwmon*", "temp1_input"))):
+            temp = _read(hwmon)
+            if temp != "":
+                break
+        if temp != "":
+            yield "gpu_temperature_celsius", labels, str(int(temp) / 1000.0)
+        power = _read(os.path.join(device, "power1_average"))
+        if power != "":  # empty on the R9700s -> metric absent, not zero
+            yield "gpu_power_watts", labels, str(int(power) / 1e6)
+
+
+def collect_nvidia(cmd):
+    if not cmd:
+        return
+    try:
+        out = subprocess.run(cmd, shell=True, capture_output=True, text=True,
+                             timeout=10, check=True).stdout
+    except (subprocess.SubprocessError, OSError):
+        return  # no NVIDIA GPU / no driver: tolerate, emit nothing
+    for line in out.splitlines():
+        parts = [p.strip() for p in line.split(",")]
+        if len(parts) < 7:
+            continue
+        idx, name, util, mused, mtotal, temp, power = parts[:7]
+        labels = {"gpu": idx, "vendor": "nvidia", "model": name}
+        yield "gpu_utilization_percent", labels, util
+        yield "gpu_memory_used_bytes", labels, str(int(float(mused)) * 1048576)
+        yield "gpu_memory_total_bytes", labels, str(int(float(mtotal)) * 1048576)
+        if temp not in ("", "[N/A]"):
+            yield "gpu_temperature_celsius", labels, temp
+        if power not in ("", "[N/A]"):
+            yield "gpu_power_watts", labels, power
+
+
+def collect(sys_dir, nvidia_cmd):
+    samples = {}
+    for source in (collect_amd(sys_dir), collect_nvidia(nvidia_cmd)):
+        for name, labels, value in source:
+            try:
+                value = str(float(value))  # normalise; drop trailing junk
+            except ValueError:
+                continue
+            samples.setdefault(name, []).append((labels, value))
+    return samples
+
+
+def _fmt(value):
+    f = float(value)
+    if f == int(f) and abs(f) < 1e15:
+        return str(int(f))
+    return repr(f)
+
+
+def render(samples):
+    out = []
+    for name, kind, help_text in FAMILIES:
+        out.append("# HELP %s %s" % (name, help_text))
+        out.append("# TYPE %s %s" % (name, kind))
+        for labels, value in sorted(samples.get(name, []), key=lambda x: sorted(x[0].items())):
+            lset = ",".join('%s="%s"' % (k, v) for k, v in sorted(labels.items()))
+            out.append("%s{%s} %s" % (name, lset, _fmt(value)))
+    return "\n".join(out) + "\n"
+
+
+class Handler(BaseHTTPRequestHandler):
+    sys_dir = "/sys/class/drm"
+    nvidia_cmd = DEFAULT_NVIDIA_CMD
+
+    def do_GET(self):  # noqa: N802 (http.server API)
+        if self.path.split("?")[0] not in ("/metrics", "/"):
+            self.send_error(404)
+            return
+        body = render(collect(self.sys_dir, self.nvidia_cmd)).encode()
+        self.send_response(200)
+        self.send_header("Content-Type", "text/plain; version=0.0.4")
+        self.send_header("Content-Length", str(len(body)))
+        self.end_headers()
+        self.wfile.write(body)
+
+    def log_message(self, *args):
+        pass
+
+
+def main(argv):
+    ap = argparse.ArgumentParser(description=__doc__.splitlines()[0])
+    ap.add_argument("--amdgpu-sysfs", default="/sys/class/drm")
+    ap.add_argument("--nvidia-cmd", default=DEFAULT_NVIDIA_CMD)
+    ap.add_argument("--serve", action="store_true", help="serve /metrics instead of one-shot")
+    ap.add_argument("--port", type=int, default=9401)
+    ap.add_argument("--bind", action="append", default=None,
+                    help="explicit bind address (repeatable); env GPU_EXPORTER_BINDS "
+                         "comma-list if unset. NEVER a wildcard.")
+    args = ap.parse_args(argv)
+
+    if not args.serve:
+        sys.stdout.write(render(collect(args.amdgpu_sysfs, args.nvidia_cmd)))
+        return 0
+
+    binds = args.bind
+    if not binds:
+        env = os.environ.get("GPU_EXPORTER_BINDS", "")
+        binds = [b.strip() for b in env.split(",") if b.strip()] or ["127.0.0.1"]
+    Handler.sys_dir = args.amdgpu_sysfs
+    Handler.nvidia_cmd = args.nvidia_cmd
+    servers = []
+    for b in binds:
+        if b in ("0.0.0.0", "::", "*"):
+            sys.stderr.write("refusing wildcard bind %s\n" % b)
+            return 2
+        servers.append(ThreadingHTTPServer((b, args.port), Handler))
+    sys.stderr.write("gpu-exporter serving :%d on %s\n" % (args.port, ",".join(binds)))
+    # EVERY socket must be served: a listen()ed-but-never-accept()ed bind looks
+    # healthy in `ss` yet times out every scrape once its SYN backlog fills —
+    # and the loopback deploy gate cannot see it. One thread per extra bind.
+    for s in servers[1:]:
+        threading.Thread(target=s.serve_forever, daemon=True).start()
+    servers[0].serve_forever()
+    return 0
+
+
+if __name__ == "__main__":
+    sys.exit(main(sys.argv[1:]))

scripts/gpu-exporter.service

@@ -0,0 +1,35 @@
+# Systemd USER unit: gpu-exporter (both Dankclaw and Dankserver).
+#
+# Install path: ~/.config/systemd/user/gpu-exporter.service
+# Installed and enabled by deploy_gpu_exporter in scripts/deploy-remote.sh.
+#
+# Per-GPU metrics (utilization / VRAM / temperature / power) for the mixed
+# AMD + NVIDIA fleet, scraped by Prometheus. Like signal-relay it is a plain
+# user unit, not a quadlet: it runs python3 straight from the checkout (a pull
+# updates it with no copy step), a hand-written user unit can be enabled where
+# the podman v1 generator's units cannot, and with Linger=yes for dank it starts
+# at boot. It reads the host's own cgroup/drm sysfs and shells to nvidia-smi, so
+# it must run on the host — a container would need GPU/DRM passthrough for
+# nothing gained.
+#
+# BINDING. The base unit serves loopback only. deploy_gpu_exporter writes a
+# drop-in adding THIS host's LAN address (the same two-address posture
+# node-exporter and cadvisor use), so the endpoint is never a wildcard. It is
+# unauthenticated metrics on the LAN, identical to node-exporter :9100 and
+# cadvisor :8080 — Prometheus reaches same-host via host.containers.internal and
+# the Dankserver instance over the LAN from Dankclaw.
+
+[Unit]
+Description=GPU metrics exporter (amdgpu sysfs + nvidia-smi)
+After=network-online.target
+Wants=network-online.target
+
+[Service]
+# Loopback by default; the deploy drop-in appends the host LAN address.
+Environment=GPU_EXPORTER_BINDS=127.0.0.1
+ExecStart=/usr/bin/python3 /home/dank/serverAppConfigs/scripts/gpu-exporter.py --serve
+Restart=always
+RestartSec=5
+
+[Install]
+WantedBy=default.target

scripts/grams-to-imatrix.py

@@ -0,0 +1,418 @@
+#!/usr/bin/env python3
+"""grams-to-imatrix.py — the engine's own calibration recording, re-voiced as a
+llama.cpp importance matrix for `rad-convert --imatrix F`. RUNS ON DANKSERVER
+beside the recording. Python 3 stdlib only: numpy must NOT be required, because
+this runs in the plain container next to a 335 GiB checkpoint, not in a venv.
+
+WHY. `rad-convert` takes its per-column activation second moments and its
+expert-popularity profile from ONE input, an imatrix (core/format/imatrix.h).
+Only the GGUF form carries a per-expert dimension — with the legacy flat form,
+or with none, every expert in a layer is weighted by the layer's aggregate,
+which is exactly the wrong thing on a 512-expert MoE. The calibration tap has
+already measured everything; this tool only re-voices it, so the imatrix and the
+protected-expert census (derive-protected-experts.py) are the SAME recording:
+one measurement, two consumers, neither able to disagree about which experts are
+hot.
+
+INPUT. RADGRAM2 files in a RADIANCE_CALIB_DIR output directory
+(arch/common/rad_arch.h, calib_write). Header law, same parse as
+scripts/derive-protected-experts.py — 32 bytes, `<8sIIQI`: magic `RADGRAM2`,
+u32 k, u32 rank, u64 rows, u32 domain, a pad word, then k*k f32 row-major. Two
+kinds of name, both GGUF-style, both with the checkpoint's GLOBAL expert
+numbering (rad_block_moe_fp8.h: ge = expert_base + e):
+
+    gram.r<rank>.blk.L.ffn_down_exps.bin        the layer's pooled gram
+    gram.r<rank>.blk.L.ffn_down_exps.<e>.bin    expert e's own gram
+
+`rows` is that file's routing/token count and the matrix is Σ x xᵀ over those
+rows, so **the diagonal is exactly the per-column activation second moment an
+imatrix wants**, and `rows` is the count the reader hands back from `count()`.
+Every off-diagonal term is dropped on purpose: the GGUF format has nowhere to
+put a full matrix, and the quantiser asks for columns.
+
+OUTPUT, per layer L (core/format/imatrix.h: names WITHOUT `.weight`, a MoE
+layer's experts folded into ONE entry, so expert 9 is slice 9 of
+`blk.L.ffn_down_exps`):
+
+  blk.L.ffn_down_exps    in_sum2 [k, n_expert] expert-major, counts [1, n_expert]
+  blk.L.ffn_gate_up_exps the SAME matrices at the SAME offsets. A routed expert's
+                         gate_up input is the same hidden-state rows as its down
+                         input, so the second moment over them is identical and
+                         so is the routing count. Two tensor infos over one
+                         payload: the reader maps each half independently
+                         (imatrix.cpp joins `<base>.in_sum2` with `<base>.counts`
+                         by base name), and copying ~200 MB to state one fact
+                         twice buys no correctness.
+  blk.L.ssm_inz          [k] and [1], from the pooled gram: that row's input is
+                         the same hidden state, so its columns take the pooled
+                         diagonal and its count is the layer's pooled rows.
+
+  Then STOP. Nothing whose input width is not the gram's k — no ssm_out, no
+  attn_output, no mtp.*, no ple_ngram. importance_for() answers false when the
+  entry's cols do not equal the weight's width and the convert proceeds without
+  it: an entry the reader silently ignores is a lie in the artifact, worse than
+  an absent one. The corollary to check before trusting the two hidden-width
+  entries: the tap writes the gate_up gram at g.n_embd and the down gram at the
+  expert width, so the gram's k must equal the checkpoint's hidden width (its
+  config.json). Where they differ, gate_up/ssm_inz are inert, not wrong.
+
+INHERITANCE. An expert with no own gram carries the layer's POOLED diagonal, and
+its count stays its own routing rows — the rule the engine itself applies when it
+picks a Hessian (libquant/lq_hess.h, calib_resolve: an expert's own gram only
+once its rank files hold kOwnRowsPerK * K rows; below that a fit to so few rows
+is worse than round-to-nearest, and those experts carry a fraction of a percent
+of the routing). The count never inherits: popularity is `count`'s job. The count
+of inherited experts is reported on stderr.
+
+RANK FOLDING. TP=2 writes gram.r0.* and gram.r1.* with GLOBAL expert numbering,
+so the ranks describe DISJOINT experts and the fold is a UNION into the one
+[k, n_expert] entry. A pooled gram is the opposite case: every rank summed the
+same stem over its own rows, so the fold is a SUM, rows included (calib_sum,
+calib_rows). The same global expert in two per-expert files is legal only when
+the numbers agree exactly; otherwise the recording is corrupt and the entry would
+be whichever file sorted last, so refuse.
+
+GGUF v3 (the only version imatrix.cpp accepts — version != 3 is a refusal):
+`general.alignment` u32, `imatrix.chunk_size` u32, `imatrix.chunk_count` u32,
+`imatrix.datasets` array of strings; tensors f32 (ggml type 0) with offsets
+relative to the aligned data section. Every tensor offset is padded up to
+`general.alignment` and the data section itself starts aligned. That is the
+classic hand-rolled-writer bug: an unaligned offset makes the reader's extent
+arithmetic point into the middle of the previous tensor's floats, and it reads
+them without complaint.
+
+chunk_count DERIVATION. The corpus row count is the LARGEST single-layer pooled
+row total, not a sum over layers: every layer's tap accumulated over the whole
+corpus, so adding the layers counts each token once per layer and inflates the
+figure by the layer count. chunk_count = that row total // chunk_size, the
+"chunks of chunk_size" line imatrix.cpp prints for itself.
+
+REFUSALS, each exit 2 with a message naming what is missing:
+  NO_GRAMS         the directory holds no RADGRAM2 down-projection gram
+  NOT_RADGRAM2     a gram whose magic is not RADGRAM2
+  SHORT            a gram whose length is not 32 + k*k*4
+  NO_POOLED        a layer with expert grams and no pooled gram to inherit from
+  K_MISMATCH       a layer whose files disagree about k (rank pooled grams, or an
+                   expert gram against its layer's pooled gram)
+  EXPERT_CONFLICT  a global expert recorded twice with different numbers
+  EXISTS           the -o path already holds a file (the trash law: move it)
+  CHUNK_COUNT      the derived chunk count does not fit the u32 metadata field
+
+USAGE. grams-to-imatrix.py <calib-dir> -o <out.gguf>
+       [--datasets NAME,...] [--chunk-size N] [--min-rows N] [-v]
+"""
+import argparse
+import os
+import re
+import struct
+import sys
+
+MAGIC = b"RADGRAM2"
+HDR = 32                       # 8s I I Q I + pad, rad_arch.h
+KOWN_ROWS_PER_K = 4            # libquant/lq_hess.h kOwnRowsPerK
+ALIGNMENT = 32                 # general.alignment, and the pad unit
+
+POOLED_RE = re.compile(r"^gram\.r(\d+)\.blk\.(\d+)\.ffn_down_exps\.bin$")
+EXPERT_RE = re.compile(r"^gram\.r(\d+)\.blk\.(\d+)\.ffn_down_exps\.(\d+)\.bin$")
+
+GT_U32, GT_STR, GT_ARR = 4, 8, 9
+GGML_F32 = 0
+
+
+def refuse(msg):
+    sys.stderr.write("REFUSED %s\n" % msg)
+    sys.exit(2)
+
+
+def read_gram(path):
+    """(k, rank, rows, domain, diagonal) from one RADGRAM2 file."""
+    try:
+        with open(path, "rb") as f:
+            raw = f.read()
+    except OSError as e:
+        refuse("UNREADABLE %s: %s" % (path, e))
+    if len(raw) < HDR:
+        refuse("SHORT %s: %d bytes, under the %d-byte RADGRAM2 header"
+               % (path, len(raw), HDR))
+    magic, k, rank, rows, domain = struct.unpack_from("<8sIIQI", raw, 0)
+    if magic != MAGIC:
+        refuse("NOT_RADGRAM2 %s: magic %r, expected b'RADGRAM2'" % (path, magic))
+    if k <= 0:
+        refuse("SHORT %s: header claims k=%d, which describes no matrix" % (path, k))
+    need = HDR + k * k * 4
+    if len(raw) != need:
+        refuse("SHORT %s: %d bytes, expected %d (32 + k*k*4 for k=%d)"
+               % (path, len(raw), need, k))
+    # The diagonal only, read by stride out of the row-major matrix.
+    diag = [struct.unpack_from("<f", raw, HDR + (i * k + i) * 4)[0]
+            for i in range(k)]
+    return k, rank, rows, domain, diag
+
+
+def load_dir(d, verbose):
+    """{layer: {"pooled": gram|None, "experts": {global e: gram}}} + rotated count.
+
+    Files outside the ffn_down_exps stem are not ours to fold: the tap names a
+    pooled gram after every projection it taps, and those carry other widths."""
+    if not os.path.isdir(d):
+        refuse("NO_GRAMS %s: not a directory" % d)
+    layers, rotated = {}, 0
+    for name in sorted(os.listdir(d)):
+        m = POOLED_RE.match(name) or EXPERT_RE.match(name)
+        if not m:
+            continue
+        path = os.path.join(d, name)
+        if not os.path.isfile(path):
+            continue
+        pooled = bool(POOLED_RE.match(name))
+        layer = int(m.group(2))
+        k, rank, rows, domain, diag = read_gram(path)
+        if domain != 0:
+            rotated += 1
+        L = layers.setdefault(layer, {"pooled": None, "experts": {}})
+        if pooled:
+            P = L["pooled"]
+            if P is None:
+                L["pooled"] = {"k": k, "rows": rows, "diag": diag, "paths": [path]}
+            elif P["k"] != k:
+                refuse("K_MISMATCH layer %d: pooled grams %s (k=%d) and %s (k=%d)"
+                       " disagree about the width"
+                       % (layer, P["paths"][0], P["k"], path, k))
+            else:
+                # calib_sum: each rank summed the same stem over its own rows.
+                P["rows"] += rows
+                P["paths"].append(path)
+                for i in range(k):
+                    P["diag"][i] += diag[i]
+            if verbose:
+                sys.stderr.write("pooled layer %d: %s k=%d rows=%d\n"
+                                 % (layer, name, k, rows))
+            continue
+        e = int(m.group(3))
+        own = L["experts"].get(e)
+        if own is None:
+            L["experts"][e] = {"k": k, "rows": rows, "diag": diag,
+                               "path": path, "ranks": [rank]}
+            if verbose:
+                sys.stderr.write("expert layer %d r%d e%d: rows=%d\n"
+                                 % (layer, rank, e, rows))
+            continue
+        if own["rows"] != rows or own["diag"] != diag:
+            refuse("EXPERT_CONFLICT layer %d expert %d: %s (rows=%d) and %s"
+                   " (rows=%d) record the same global expert with different"
+                   " numbers" % (layer, e, own["path"], own["rows"], path, rows))
+        own["ranks"].append(rank)
+        sys.stderr.write("note: layer %d expert %d recorded identically by rank %s"
+                         " and %s; folded once\n"
+                         % (layer, e, own["path"], name))
+    if not layers:
+        refuse("NO_GRAMS %s: no RADGRAM2 gram.r<rank>.blk.<L>.ffn_down_exps"
+               "[.<expert>].bin files in the directory" % d)
+    return layers, rotated
+
+
+def build_entries(layers, min_rows, verbose):
+    """([ (name, dims, payload|None, key) ], census, layers emitted)."""
+    entries = []
+    census = {"inherited": 0, "experts": 0, "routing": 0, "rows": 0, "layers": 0}
+    for layer in sorted(layers):
+        P, E = layers[layer]["pooled"], layers[layer]["experts"]
+        if E and P is None:
+            refuse("NO_POOLED layer %d: %d expert gram(s) but no pooled gram (an"
+                   " expected gram.r<rank>.blk.%d.ffn_down_exps.bin), so an expert"
+                   " under the row threshold would have nothing to inherit"
+                   % (layer, len(E), layer))
+        if P is None:
+            continue
+        k = P["k"]
+        census["layers"] += 1
+        census["rows"] = max(census["rows"], P["rows"])
+        dn = "blk.%d.ffn_down_exps" % layer
+        # A staging key names ONE payload. A gate_up entry reuses the down
+        # layer's bytes by carrying the same key with no payload of its own;
+        # in_sum2 and counts are different bytes and must never share a key.
+        ksum, kcnt = (layer, "sum2"), (layer, "counts")
+        kpgm, kpcnt = (layer, "pooled_sum2"), (layer, "pooled_counts")
+        # One entry holds one width: an expert gram recorded at another k would
+        # inherit (or be inherited by) a diagonal of the wrong length, and the
+        # reader would then answer `cols` from whichever file set dims[0].
+        for e in sorted(E):
+            if E[e]["k"] != k:
+                refuse("K_MISMATCH layer %d expert %d: %s holds k=%d and this"
+                       " layer's pooled gram is k=%d" % (layer, e, E[e]["path"],
+                                                          E[e]["k"], k))
+
+        if not E:
+            # No per-expert recording for this layer: emit the dense form
+            # ([k], [1]) rather than 512 identical slices, which would claim a
+            # per-expert measurement that was never taken.
+            entries.append((dn + ".in_sum2", [k],
+                            struct.pack("<%df" % k, *P["diag"]), ksum))
+            entries.append((dn + ".counts", [1],
+                            struct.pack("<f", float(P["rows"])), kcnt))
+            sys.stderr.write("note: layer %d has no per-expert grams; %s is"
+                             " dense over the pooled gram\n" % (layer, dn))
+        else:
+            floor = min_rows if min_rows > 0 else KOWN_ROWS_PER_K * k
+            n_expert = max(E) + 1
+            sum2, counts = bytearray(), bytearray()
+            for e in range(n_expert):
+                own = E.get(e)
+                if own is not None and own["rows"] >= floor:
+                    src = own["diag"]
+                    rows = own["rows"]
+                else:
+                    src = P["diag"]
+                    rows = own["rows"] if own is not None else 0
+                    census["inherited"] += 1
+                sum2 += struct.pack("<%df" % k, *src)
+                counts += struct.pack("<f", float(rows))
+                census["experts"] += 1
+                census["routing"] += rows
+            if min(E) != 0:
+                sys.stderr.write("note: layer %d has no gram for experts 0..%d;"
+                                 " those slices carry the pooled gram at count 0\n"
+                                 % (layer, min(E) - 1))
+            entries.append((dn + ".in_sum2", [k, n_expert], bytes(sum2), ksum))
+            entries.append((dn + ".counts", [1, n_expert], bytes(counts), kcnt))
+            gu = "blk.%d.ffn_gate_up_exps" % layer
+            entries.append((gu + ".in_sum2", [k, n_expert], None, ksum))
+            entries.append((gu + ".counts", [1, n_expert], None, kcnt))
+            if verbose:
+                sys.stderr.write("layer %d: k=%d n_expert=%d pooled_rows=%d"
+                                 " own-gram floor=%d\n"
+                                 % (layer, k, n_expert, P["rows"], floor))
+
+        inz = "blk.%d.ssm_inz" % layer
+        entries.append((inz + ".in_sum2", [k],
+                        struct.pack("<%df" % k, *P["diag"]), kpgm))
+        entries.append((inz + ".counts", [1],
+                        struct.pack("<f", float(P["rows"])), kpcnt))
+    return entries, census
+
+
+def enc_str(v):
+    b = v.encode("utf-8")
+    return struct.pack("<Q", len(b)) + b
+
+
+def write_gguf(path, entries, datasets, chunk_size, chunk_count):
+    """Write GGUF v3 and return its size in bytes.
+
+    One payload per key: entries that share a key share the bytes and the
+    offset, so a second tensor info states the same fact without copying it."""
+    kv = bytearray()
+    kv += enc_str("general.alignment") + struct.pack("<I", GT_U32) \
+        + struct.pack("<I", ALIGNMENT)
+    kv += enc_str("imatrix.chunk_size") + struct.pack("<I", GT_U32) \
+        + struct.pack("<I", chunk_size)
+    kv += enc_str("imatrix.chunk_count") + struct.pack("<I", GT_U32) \
+        + struct.pack("<I", chunk_count)
+    kv += enc_str("imatrix.datasets") + struct.pack("<I", GT_ARR) \
+        + struct.pack("<I", GT_STR) + struct.pack("<Q", len(datasets))
+    for d in datasets:
+        kv += enc_str(d)
+
+    head = bytearray()
+    head += b"GGUF" + struct.pack("<I", 3)
+    head += struct.pack("<Q", len(entries)) + struct.pack("<Q", 4)
+    head += kv
+
+    payloads = {}
+    for name, _dims, payload, key in entries:
+        if payload is None:
+            continue
+        prev = payloads.get(key)
+        if prev is not None and prev != payload:
+            # Two different payloads on one key would ship whichever was
+            # appended last while BOTH tensor infos pointed at it: the counts
+            # payload landing over the in_sum2 payload is exactly how a
+            # hand-rolled GGUF writer loses half its data without an error.
+            refuse("INTERNAL %s: two different payloads share one staging key"
+                   % name)
+        payloads.setdefault(key, payload)
+    infos = bytearray()
+    offsets, order, size = {}, [], 0
+    for name, dims, payload, key in entries:
+        if key not in payloads:
+            refuse("INTERNAL %s: no payload staged for its key" % name)
+        if key not in offsets:
+            offsets[key] = (size + ALIGNMENT - 1) // ALIGNMENT * ALIGNMENT
+            size = offsets[key] + len(payloads[key])
+            order.append(key)
+        infos += enc_str(name) + struct.pack("<I", len(dims))
+        for d in dims:
+            infos += struct.pack("<Q", d)
+        infos += struct.pack("<I", GGML_F32) + struct.pack("<Q", offsets[key])
+
+    body = bytes(head) + bytes(infos)
+    data_start = (len(body) + ALIGNMENT - 1) // ALIGNMENT * ALIGNMENT
+    with open(path, "wb") as f:
+        f.write(body)
+        f.write(b"\0" * (data_start - len(body)))
+        # The offsets in the tensor infos are the file's law, not a hint: each
+        # payload must land at its own padded offset. Writing them back to back
+        # instead leaves every reader that trusts the info — imatrix.cpp does —
+        # reading its neighbour's floats: plausible values, no error, wrong
+        # quantisation.
+        here = data_start
+        for key in order:
+            f.write(b"\0" * (data_start + offsets[key] - here))
+            f.write(payloads[key])
+            here = data_start + offsets[key] + len(payloads[key])
+        return f.tell()
+
+
+def main():
+    ap = argparse.ArgumentParser(
+        description="Fold a radiance calibration recording into a GGUF v3 imatrix.")
+    ap.add_argument("calib_dir")
+    ap.add_argument("-o", "--out", required=True)
+    ap.add_argument("--datasets", default="",
+                    help="comma-separated names for imatrix.datasets")
+    ap.add_argument("--chunk-size", type=int, default=4096,
+                    help="rows per chunk for imatrix.chunk_size (default 4096)")
+    ap.add_argument("--min-rows", type=int, default=0,
+                    help="own-gram row threshold (default 4*k, the engine's"
+                         " kOwnRowsPerK law)")
+    ap.add_argument("-v", "--verbose", action="store_true")
+    a = ap.parse_args()
+
+    if os.path.exists(a.out):
+        # The repo's deletion law: an artifact that exists is moved to trash by
+        # a human, never silently overwritten by a re-run of the tool that made
+        # it, so a second convert cannot destroy the imatrix the first one fed.
+        refuse("EXISTS %s: an imatrix is already there; move it to"
+               " .AGENTTRASH/ or name another -o" % a.out)
+    if a.chunk_size <= 0:
+        refuse("BAD_CHUNK_SIZE --chunk-size %d is not positive" % a.chunk_size)
+    layers, rotated = load_dir(a.calib_dir, a.verbose)
+    entries, census = build_entries(layers, a.min_rows, a.verbose)
+    if not entries:
+        refuse("NO_GRAMS %s: no layer holds a pooled gram to fold" % a.calib_dir)
+    if rotated:
+        sys.stderr.write("note: %d gram(s) recorded in a rotated domain: their"
+                         " diagonal is the rotated activation's second moment,"
+                         " not the model's own\n" % rotated)
+
+    chunk_count = census["rows"] // a.chunk_size
+    if chunk_count > 0xFFFFFFFF:
+        # imatrix.chunk_count is a u32 in the reader's grammar, so there is no
+        # legal way to record a bigger one. Reaching this means either a chunk
+        # size picked for a far larger run or a row total read out of the wrong
+        # header offset — rows is a u64 and a u32 parse is off by 2**32 — and
+        # the reader would trust whatever number landed in the field.
+        refuse("CHUNK_COUNT %d = %d rows // %d does not fit the u32"
+               " imatrix.chunk_count field; raise --chunk-size"
+               % (chunk_count, census["rows"], a.chunk_size))
+    nbytes = write_gguf(a.out, entries, [d for d in a.datasets.split(",") if d],
+                        a.chunk_size, chunk_count)
+    sys.stderr.write("IMATRIX_OK entries=%d layers=%d experts=%d inherited=%d"
+                     " rows=%d routing=%d chunks=%d/%d bytes=%d out=%s\n"
+                     % (len(entries), census["layers"], census["experts"],
+                        census["inherited"], census["rows"], census["routing"],
+                        chunk_count, a.chunk_size, nbytes, a.out))
+
+
+if __name__ == "__main__":
+    main()

scripts/kld-gate-qwen38.sh

@@ -0,0 +1,122 @@
+#!/usr/bin/env bash
+# kld-gate-qwen38.sh — RUNS ON DANKSERVER (or anywhere the reports are). The campaign's
+# acceptance gate: three KLD reports against ONE reference recording, one verdict.
+#
+# The reference is the BF16 container built from the vendor checkpoint (run-qwen38-bf16.sh
+# kldref, in the same drained window that records the calibration grams). Every scored
+# container is measured against that one recording, so the KL scale runs from 0 = the
+# checkpoint itself downward in faithfulness. Three containers ride it:
+#
+#   fp8.json    the published FP8-native release   — the achievable end of the interval
+#   iq4r.json   the payload frank-archon serves    — the bar to beat
+#   cand.json   the arm under test                 — the subject
+#
+# The FP8 number is what makes the interval absolute instead of relative: "half the gap"
+# is half of the distance between the 4-bit payload and the near-lossless release, not half
+# of an invented distance to the checkpoint. That is also why the FP8 payload is FETCHED and
+# scored, never converted: it is a second opinion about what this checkpoint tolerates, not a
+# step of the campaign. It lands LAST (disk sequencing) — by then the 335 GiB bf16 container
+# is gone, so its 172.8 GiB fits.
+#
+# The law, in order of severity, on the mean KL of one slice (--slice all | a corpus source):
+#   1. KL(fp8) < KL(iq4r)                 the anchors are ordered. If the near-lossless
+#                                         release does NOT sit closer to the checkpoint than
+#                                         the 4-bit payload, the corpus, the tokenizer or the
+#                                         boot shape is wrong and no arm number means anything.
+#   2. KL(fp8) < KL(cand) < KL(iq4r)      the candidate lies strictly inside the interval —
+#                                         outside it, something is mis-measured, not better
+#   3. KL(cand) <= KL(iq4r) - R x gap     it closes at least R (--recover, default 0.5) of the
+#                                         gap = KL(iq4r) - KL(fp8): at least half the way from
+#                                         what is served toward the release (agreed 2026-10-27)
+#   4. top-1(cand) >= top-1(iq4r)         the agreement the harness keys on does not fall
+#
+# Booting the containers is NOT this script's job: `run-qwen38-bf16.sh score RAD OUT.json`
+# writes the reports and owns the single engine argv (same pin, --tp 2, --kv-cache-dtype fp8,
+# same step size, --deterministic, no speculation) that makes the three numbers comparable.
+# This script is arithmetic, so it re-runs against old reports and is testable without a GPU.
+#
+# Usage: kld-gate-qwen38.sh FP8.json IQ4R.json CAND.json [--slice NAME] [--recover F]
+set -uo pipefail
+
+[ "$#" -ge 3 ] || { echo "usage: kld-gate-qwen38.sh FP8.json IQ4R.json CAND.json [--slice NAME] [--recover F]" >&2; exit 2; }
+python3 - "$@" <<'PY'
+import json, sys
+
+a = sys.argv[1:]
+opts, files = [], []
+i = 0
+while i < len(a):
+    if a[i].startswith("--"):
+        opts += [a[i], a[i + 1]]
+        i += 2
+    else:
+        files.append(a[i])
+        i += 1
+if len(files) != 3:
+    sys.exit("usage: kld-gate-qwen38.sh FP8.json IQ4R.json CAND.json [--slice NAME] [--recover F]")
+o = dict(zip(opts[0::2], opts[1::2]))
+slice_ = o.get("--slice", "all")
+recover = float(o.get("--recover", 0.5))
+if not 0.0 < recover <= 1.0:
+    sys.exit(f"--recover must be in (0,1], got {recover}")
+
+
+def die(msg):
+    print(msg, file=sys.stderr)
+    raise SystemExit(2)
+
+
+def read(path):
+    try:
+        d = json.load(open(path))
+    except Exception as e:
+        die(f"cannot read the KLD report {path}: {e}")
+    if slice_ == "all":
+        g = d.get("all")
+    else:
+        g = (d.get("by_source") or {}).get(slice_)
+        if g is None:
+            die(f"{path} has no slice {slice_!r}; it has: "
+                f"{', '.join(sorted(d.get('by_source') or {})) or '(none)'}")
+    if g is None or "kld" not in g or "top1_agreement" not in g:
+        die(f"{path} has no {slice_!r} summary — wrong report version?")
+    return d, g
+
+
+rows = []
+for label, path in zip(("fp8", "iq4r", "cand"), files):
+    d, g = read(path)
+    rows.append((label, path, d.get("model", "?"), g["kld"]["mean"], g["kld"]["p99_9"],
+                 g["top1_agreement"], g["positions"]))
+
+print(f"reference: BF16 container; slice: {slice_}")
+print(f"{'':6} {'model':38} {'KL mean':>10} {'KL p99.9':>10} {'top-1':>8}  positions")
+for label, path, model, kl, p999, top1, n in rows:
+    print(f"{label:6} {str(model)[:38]:38} {kl:10.6f} {p999:10.4f} {100*top1:7.2f}%  {n}")
+r = {label: (kl, top1) for label, _p, _m, kl, _q, top1, _n in rows}
+kl_fp8, kl_iq4r, kl_cand = r["fp8"][0], r["iq4r"][0], r["cand"][0]
+t_iq4r, t_cand = r["iq4r"][1], r["cand"][1]
+gap = kl_iq4r - kl_fp8
+ok = True
+
+
+def law(n, text, passed):
+    global ok
+    ok = ok and passed
+    print(f"  [{'PASS' if passed else 'FAIL'}] {n}. {text}")
+
+
+law(1, f"anchors ordered: fp8 {kl_fp8:.6f} < iq4r {kl_iq4r:.6f} (gap {gap:.6f})",
+    gap > 0)
+law(2, f"candidate inside the interval: {kl_fp8:.6f} < {kl_cand:.6f} < {kl_iq4r:.6f}",
+    kl_fp8 < kl_cand < kl_iq4r)
+need = kl_iq4r - recover * gap
+law(3, f"candidate closes {100*max((kl_iq4r - kl_cand)/gap, 0):.1f}% of the "
+       f"{gap:.6f} gap (needs {100*recover:.0f}%, i.e. KL <= {need:.6f})"
+       if gap > 0 else "gap is not positive; cannot score recovery",
+    gap > 0 and kl_cand <= need)
+law(4, f"candidate top-1 {100*t_cand:.2f}% >= served iq4r top-1 {100*t_iq4r:.2f}%",
+    t_cand >= t_iq4r)
+print("KLD GATE " + ("PASS" if ok else "FAIL"))
+sys.exit(0 if ok else 1)
+PY

scripts/probe-frank-archon-kv.py

@@ -147,10 +147,21 @@ def main():
     deep = corpus(160_000)
     wide = corpus(1_100_000)
 
-    # 3.28 chars/token is the measured ratio of this corpus through the
-    # froggeric template (646,000 chars -> 196,919 tokens at the 2026-10-11
-    # deploy); 98.5 % of the window leaves the response room under the ceiling.
-    chars = int(0.985 * maxlen * 3.28)
+    # The corpus's chars/token ratio is MEASURED here, not stored. A stored 3.28
+    # (646,000 chars -> 196,919 tokens, 2026-10-11) had drifted to 3.20 by
+    # 2026-10-20 (646,299 chars -> 201,876 tokens) as the repo's docs grew, and a
+    # stale constant sends leg B past --max-model-len, where the engine answers a
+    # perfectly legitimate 400 ("the request (201900 tokens) exceeds the available
+    # context size (200000)") and the leg reads as a capacity failure. One small
+    # request prices the ratio; 97 % of the window leaves the response room.
+    sample = prompt(wide, 4000, "CAL", "cal-marker")
+    cal, _, _ = post({"model": MODEL,
+                      "messages": [{"role": "user", "content": sample}],
+                      "max_tokens": 1, "temperature": 0.0,
+                      "chat_template_kwargs": {"enable_thinking": False}})
+    ratio = len(sample) / cal["usage"]["prompt_tokens"]
+    print(f"[cal] corpus ratio {ratio:.2f} chars/token")
+    chars = int(0.97 * maxlen * ratio)
     # Each marker's FIRST hyphen-separated word is unique, because show()
     # scores the hit on that word — a shared prefix would score every slot HIT
     # on any answer that says the word.
@@ -176,7 +187,7 @@ def main():
                 print(f"[B-{s}] FAILED after {time.time() - t0:.1f}s: {e!r}")
     print(f"[B] total {time.time() - t0:.1f}s")
     # D — concurrency decode: seqs streams, 512 tokens each, ~25 % depth.
-    dchars = int(0.25 * maxlen * 3.28)
+    dchars = int(0.25 * maxlen * ratio)
     dbodies = [chat(deep, dchars, f"D{i + 1}-9137", f"flint-marker-{i}", 512)
                for i in range(seqs)]
     print(f"[D] {seqs} concurrent streams x 512 tokens, "

scripts/recipes/qwen4exp-qwen38-w5nl64-bf16trunk.recipe

@@ -0,0 +1,80 @@
+# qwen4exp-qwen38-w5nl64-bf16trunk.recipe — every routed expert, BOTH projections, at five bits:
+# u5 codes into libquant's 32-level w5nl table, an E4M3 scale a group of 64 under the fixed 2^-13
+# second level, rotated by the 128-wide Hadamard, chosen by GPTQ against $CALIB and swept three
+# times by coordinate descent. This is `w5nl64a8h` in the engine's served-encoding table — the top of
+# the four-bit class — applied wider than any published radiance container of this checkpoint: the
+# author's own w5 rung (`qwen4exp-w5dn64-i8-hc8m.recipe`) moves only `down`, because that rung was
+# priced as a byte-for-error trade against prefill; this campaign buys the whole rung and pays for
+# it out of the pinned host pool (+15.3 GiB of expert plane, `--host-pool-mib 61440` per rank), not
+# out of the KV pool.
+#
+# Every other line is the served container's recipe (dump it with
+# `rad-info --recipe /home/dank/models/rad/qwen3.8-next-flash-fp8-iq4r-moe.rad`), minus the rows this
+# campaign leaves at the checkpoint:
+#
+#   * the trunk's block linears, the shared expert and the hyper-connection mixers are NOT listed —
+#     dropping their i8/fp8 lines leaves them bf16, which is what the checkpoint holds. Measured bill
+#     +1.978 GiB of always-resident weight = ~131k tokens of KV off the pool.
+#   * `ssm_inz`/`ssm_out` STAY int8: the delta-net gates are the largest trunk rows (+1.9 GiB, 126k
+#     tokens) and the least sensitive per byte.
+#   * `mtp.draft_head` STAYS 2-bit and the PLE n-gram table STAYS one fixed E4M3 scale: the draft
+#     head's rung is cheap and its acceptance cost is a decode question this campaign does not touch,
+#     and bf16 for the n-gram table would pin +47.69 GiB of host RAM for a lookup.
+#
+# Protected experts: none in this file. `scripts/apply-protected-cast.sh` appends the derived set
+# (48, two per layer so each rank's 256-expert half stays even — the four-bit GEMM takes the rest of
+# a half as two equal tables by parity) from the fresh 12M-token energy/routing census, and they must
+# sit ABOVE the expert row because the first rule that matches a weight decides it.
+#
+#   CALIB=/home/dank/campaign/qwen38-w5nl64/calib \
+#     rad-convert /home/dank/models/Qwen3.8-Flash-Next \
+#       --recipe this.file --imatrix .../imatrix.gguf -o out.rad
+#
+# The arm driver (`scripts/convert-qwen38-w5nl64.sh`) generates the per-arm variants of the expert
+# row from this template — cd=3 (control), cd=6, and `act_order=1` with NO cd= (libquant refuses the
+# pair, lq_registry.cpp:105) — rather than keeping three near-duplicate recipes that can drift apart.
+#
+# The whole campaign, in the order that fits on one disk (every step runs on dankserver):
+#
+#   1. scripts/download-qwen38-bf16.sh --fetch      the vendor BF16 checkpoint, 335.3 GiB
+#   2. scripts/build-qwen38-bf16-rad.sh             the BF16 reference container, another 335.3 GiB
+#   3. one drained window, two boots:
+#        scripts/run-qwen38-bf16.sh kldref          the KLD reference recording (the zero of scale)
+#        scripts/run-qwen38-bf16.sh calib           the 12M-position calibration tap -> grams
+#   4. trash qwen38-bf16.rad                        the peak-disk law: the converts need its room
+#   5. scripts/grams-to-imatrix.py                  grams -> imatrix.gguf in the campaign dir
+#   6. scripts/convert-qwen38-w5nl64.sh             the arms, planned and billed before each one runs
+#   7. fetch the published FP8-native release, then per container:
+#        scripts/run-qwen38-bf16.sh score RAD OUT   fp8 anchor, served iq4r, each arm
+#      then scripts/kld-gate-qwen38.sh fp8.json iq4r.json cand.json
+#   8. a winner is a trio edit plus `scripts/deploy.sh dankserver --service frank-archon`; the
+#      rollback is the commit before it. If the win needs the bigger pinned plane, raise
+#      FRANK_ARCHON_HOST_POOL_MIB and FRANK_ARCHON_TTM_TARGET_MIB in the SAME commit: the preflight
+#      floor is 2xHOST_POOL + 48,824 + prefix cache + MNBT + the store, and a pool raised without
+#      the TTM target is a boot refusal that Restart=always repeats forever, invisible to deploy.sh.
+
+# PROTECTED-BEGIN — scripts/apply-protected-cast.sh splices the derived cast lines in here.
+# They MUST sit above the expert rule: the first rule whose glob matches a weight decides it, so a
+# cast line below the five-bit rule would never win. The set is derived from the campaign's own
+# 12M-token recording (`derive-protected-experts.py --target-count 48` over the fresh grams), and the
+# driver splices it into a per-arm COPY so this template stays the clean, reviewable baseline.
+# PROTECTED-END
+
+# Every routed expert, trunk and MTP layer alike, both projections, at five bits. The experts' scales
+# sit between 2^-15 and 2^-10.6 under w4nl's top level of 22 and half a binade lower under w5nl's 30
+# — inside E4M3's normal range under 2^-13 either way. The MTP layer has no recording and is rounded
+# to nearest, which libquant says on stderr; it is one layer of 49 and only ever changes drafts.
+blk.*.ffn_*_exps.*.weight            gptq table=w5nl group=64 scale=fp8_e4m3 scale2=f32 block2=*x* scale2_value=0.0001220703125 transform=fwht128 rule=search cd=3 calib=$CALIB
+
+# The delta-net's in/out gates, int8 with a bf16 scale a group of 128 searched for least error — the
+# served recipe's lines, unchanged.
+blk.*.ssm_inz.weight                 rtn codes=i8 group=128 scale=bf16 clamp=sym rule=search
+blk.*.ssm_out.weight                 rtn codes=i8 group=128 scale=bf16 clamp=sym rule=search
+
+# The MTP head's two mixing rows, E4M3 as served. They only ever change the drafts.
+mtp.fc_hidden.weight                 rtn codes=fp8_e4m3 group=128 scale=f32
+mtp.fc_embedding.weight              rtn codes=fp8_e4m3 group=128 scale=f32
+
+# The n-gram table at one fixed E4M3 scale, and the 2-bit draft head — both the served lines.
+blk.*.ple_ngram.weight               rtn codes=fp8_e4m3 block=*x* scale=bf16 rule=fixed scale_value=1.99317932128906e-4
+mtp.draft_head.weight                rtn codes=u2 zero=u8 group=128 scale=f16

scripts/run-qwen38-bf16.sh

@@ -0,0 +1,342 @@
+#!/usr/bin/env bash
+# run-qwen38-bf16.sh — RUNS ON DANKSERVER. Boot a container over the campaign's
+# ONE measuring shape for the phases that need both R9700s. The engine owns the
+# cards, so frank-archon is stopped first and restored on every exit path — the
+# autoresearch pattern: stop, never mask (the unit file in /etc/systemd/system
+# is a symlink into the checkout and mask fails on a relinked unit; the deploy
+# restores it).
+#
+#   kldref            --kld-record the reference log-probabilities: the engine
+#                     runs /home/dank/corpora/out/kld.jsonl (197 prompts, 50,432
+#                     scored positions, score tail 256) through prefill, writes
+#                     /home/dank/campaign/qwen38-w5nl64/kld-ref, and EXITS. The
+#                     model behind it is the BF16 container from
+#                     scripts/build-qwen38-bf16-rad.sh — the vendor checkpoint's
+#                     own numbers, which is what makes it the zero point.
+#   score RAD OUT     --kld-ref score ONE container against that recording and
+#                     EXIT: the FP8-native anchor, the served iq4r, and every
+#                     arm. Deliberately the SAME engine pin, the same TP, the
+#                     same step size, the same fp8 KV, --deterministic and no
+#                     speculation as kldref — KL and top-1 agreement are only
+#                     comparable across boots when everything except the weights
+#                     is identical, so this mode is one argv with --model and
+#                     the two --kld-* flags changed, never a second shape.
+#                     RAD must sit under the models dir (the container reads it
+#                     through the ro /models mount); OUT lands on a writable
+#                     mount because --kld-out F also writes F.rows beside it.
+#   calib             serve on :7770 with the calibration tap open
+#                     (RADIANCE_CALIB_DIR + RADIANCE_CALIB_TOKENS=12000000). A
+#                     separate driver — scripts/calib-push.py --port 7770
+#                     --container qwen38-bf16 --corpus
+#                     /home/dank/corpora/out/calib.jsonl — pushes the corpus; at
+#                     12M pooled positions the engine writes the Grams and the
+#                     tap switches itself off (calib-push prints TAP_DONE). Stop
+#                     this script after that; the trap brings frank-archon back.
+#
+# Shape: the vendor's bf16 calibration lines — expert_tiered placement with the
+# weights disk tier, deterministic (the resident set pinned to the loader's plan,
+# so a recorded reference and a scored candidate see the same one), no
+# speculation, fp8 KV, a 32,768-token window, the PLE n-gram table pinned to RAM,
+# and the production chat template rather than the one baked into the container —
+# every scored container is asked the same tokenised questions the reference
+# answered. The window is 32,768: kld.jsonl is 197 records with a longest of
+# 38,263 characters (longest prompt FIELD 36,855, median record 27,228). Its
+# token length — 10,554 for that longest record, counted by the engine itself —
+# is in the funding paragraph below; the kldref gate re-checks the file against
+# the window before it boots anything.
+#
+# NO --vram-kv-mib: it was REMOVED at 1.3.0 and naming it is a boot refusal, not
+# a no-op — verified on the pinned image 2026-10-27: "E --vram-kv-mib was
+# removed: the card is split between the weights and the cache from what the
+# model needs." The budget resolver derives that split from --host-pool-mib and
+# --gpu-headroom-mib, and this rig inherits BOTH from frank-archon.env, the file
+# that owns them: the cache's funding is `elastic - expert_floor` and the expert
+# floor falls one MiB for every MiB of host pool (frank-archon.md, Laws), so the
+# pool buys KV as well as residency — a private pair is not a free knob. What
+# buying one cost, measured on the first kldref boot the same day with the w4
+# era's 12288 / 768: "I kv: the pool backs 6% of the worst case (4 sequences all
+# at 32768 tokens), 0.2 full-length sessions; a larger --host-pool-mib gives the
+# cache more" — a 270 MiB, 7,436-position cache, after which the recorder refused
+# every one of the 197 corpus documents ("E kld: document 0 is 10554 tokens and
+# the KV pool holds 7436, so it can never be scored") and exited 1. Production's
+# 49,152 per rank is affordable exactly when the rig runs: the rig stops
+# frank-archon first, and that stop frees the same 96 GiB the service was pinning.
+#
+# The 32,768-token window it pairs with is not up for grabs: the corpus's longest
+# prompt is 10,554 tokens by the engine's own count (the header's chars/4.97
+# estimate called that record ~7,700; the true ratio for this file is chars/3.6),
+# so the window clears it by 3x and the cache now funds it.
+#
+# The step size is ONE value for the recording and every scoring pass:
+# FRANK_QWEN38_MNBT, default 2,048. The gate's whole claim is that the reference,
+# the served iq4r and every arm rode one shape, so the ceiling is set by the
+# TIGHTEST payload the gate boots rather than by the bf16 reference: the quantised
+# expert FILE tiers (the served iq4r, the arms) stream only at MNBT<=2048 and
+# refuse the widened form above it, and 2,048 is what production holds anyway.
+# Recording the reference at 32,768 would therefore measure a shape the payloads
+# cannot reproduce; at 2,048 the 50,432 positions cost more steps, which is still
+# one drained window. Calibration keeps the wide 65,536 — grams are not logits,
+# nothing downstream is compared against a step size, and that is where a big
+# step pays (a disk-tier step costs the same at 1 token as at 64k).
+#
+# usage: scripts/run-qwen38-bf16.sh kldref|calib|score RAD OUT [--dry-run] [--image REF]
+#   (RAD and OUT are score's two positionals: RAD is a container under the models
+#    dir, OUT the report path whose directory is created if missing)
+#   --dry-run   print the exact `podman run` argv, one shell-quoted argument a
+#               line, and exit 0: the service is NOT stopped, nothing is
+#               mounted, nothing is written — the offline test seam. It still
+#               validates the argv's own shape (paths through the /models mount,
+#               a step size that is a number), because an argv that could not
+#               boot is not worth printing.
+#   --image REF override the digest pin (a probe boot, or the test seam)
+#
+# knobs: FRANK_QWEN38_MODELS (the dir mounted at /models), FRANK_QWEN38_RAD (the
+#        bf16 container, kldref only), FRANK_QWEN38_CAMPAIGN (holds kld-ref/ and
+#        calib/), FRANK_QWEN38_KLD_CORPUS, FRANK_QWEN38_TPL (the template copy
+#        the container reads through /models), FRANK_QWEN38_LOG_DIR, and
+#        FRANK_QWEN38_MNBT — the step size shared by kldref and score (2,048 is
+#        the file tier's ceiling and production's own value, four slots and all),
+#        and FRANK_QWEN38_MAX_MODEL_LEN — the window every mode boots at (32,768;
+#        see the corpus census above, and raise it together with the corpus).
+# kldref refuses (exit 2) when the longest record of the corpus it is about to
+# read is longer than MAX_MODEL_LEN x 3 characters — chars/3 being the
+# pessimistic token:char ratio, so the script says no before the engine finds out
+# inside a drained window, after the checkpoint work is already done. score
+# refuses when the reference dir is absent, when RAD is not on disk or not under
+# the models mount, or when OUT already exists: the trash law means a second
+# score over the same name is a deliberate act, and two numbers in one file is
+# how a campaign loses its own history.
+# Engine stdout goes to $LOG_DIR/qwen38_bf16_<mode>.log — the boot ledger is the
+# record of what the resolver actually gave the cards; the journal never sees it.
+# exit: the engine's code; 2 = refused before the service was touched
+set -uo pipefail
+
+SCRIPT_DIR="$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)"
+# shellcheck disable=SC1091
+source "$SCRIPT_DIR/frank-archon.env"
+IMAGE="${FRANK_ARCHON_IMAGE:?FRANK_ARCHON_IMAGE must name the radiance image}"
+
+MODE=""
+POS=()
+DRY=0
+while [ $# -gt 0 ]; do
+    case "$1" in
+        kldref|calib|score) MODE="$1" ;;
+        --dry-run)          DRY=1 ;;
+        --image)            IMAGE="$2"; shift ;;
+        *)
+            # score's two positional arguments, and nothing else may be bare.
+            if [ "$MODE" = score ] && [ "${#POS[@]}" -lt 2 ]; then
+                POS+=("$1")
+            else
+                echo "usage: ${0##*/} kldref|calib|score RAD OUT [--dry-run] [--image REF]  (unknown argument: $1)" >&2
+                exit 2
+            fi ;;
+    esac
+    shift
+done
+if [ -z "$MODE" ]; then
+    echo "usage: ${0##*/} kldref|calib|score RAD OUT [--dry-run] [--image REF]  (no mode given)" >&2
+    exit 2
+fi
+if [ "$MODE" = score ] && [ "${#POS[@]}" -ne 2 ]; then
+    echo "usage: ${0##*/} score RAD OUT  — RAD is the container to score, OUT the report to write (got ${#POS[@]} argument(s))" >&2
+    exit 2
+fi
+
+MODELS="${FRANK_QWEN38_MODELS:-/home/dank/models}"
+CAMPAIGN="${FRANK_QWEN38_CAMPAIGN:-/home/dank/campaign/qwen38-w5nl64}"
+KLD_CORPUS="${FRANK_QWEN38_KLD_CORPUS:-/home/dank/corpora/out/kld.jsonl}"
+RAD="${FRANK_QWEN38_RAD:-$MODELS/rad/qwen38-bf16.rad}"
+TPL="${FRANK_QWEN38_TPL:-$MODELS/tpl.jinja}"
+TEMPLATE="$SCRIPT_DIR/qwen38-froggeric-v22.5.jinja"
+LOG_DIR="${FRANK_QWEN38_LOG_DIR:-/home/dank/logs}"
+LOG="$LOG_DIR/qwen38_bf16_${MODE}.log"
+KLD_REF_DIR="$CAMPAIGN/kld-ref"
+CALIB_DIR="$CAMPAIGN/calib"
+C=qwen38-bf16
+SERVICE=frank-archon
+# The card split, taken from the file that owns it instead of a private pair:
+# --host-pool-mib also funds the KV cache (see the header), so hardcoding it
+# here once starved the recorder. The `:?` guards keep an empty env fatal.
+HOST_POOL_MIB="${FRANK_ARCHON_HOST_POOL_MIB:?FRANK_ARCHON_HOST_POOL_MIB must be set}"
+GPU_HEADROOM_MIB="${FRANK_ARCHON_GPU_HEADROOM_MIB:?FRANK_ARCHON_GPU_HEADROOM_MIB must be set}"
+
+# ONE step size for both logits modes. The file tier streams only at <=2,048 and
+# the gate scores file-tier payloads (the served iq4r, the arms) against the
+# reference, so the reference is recorded at the ceiling the payloads can meet —
+# never two numbers that a KL between them would have to forgive.
+MNBT="${FRANK_QWEN38_MNBT:-2048}"
+if ! [[ "$MNBT" =~ ^[0-9]+$ ]]; then
+    echo "REFUSED: FRANK_QWEN38_MNBT must be a token count (got: $MNBT)" >&2
+    exit 2
+fi
+
+# The window every mode boots at. Over-length is a REFUSAL, not a truncation (the
+# env file says so for production), so this is the number the corpus must fit in.
+MAX_MODEL_LEN="${FRANK_QWEN38_MAX_MODEL_LEN:-32768}"
+if ! [[ "$MAX_MODEL_LEN" =~ ^[0-9]+$ ]]; then
+    echo "REFUSED: FRANK_QWEN38_MAX_MODEL_LEN must be a token count (got: $MAX_MODEL_LEN)" >&2
+    exit 2
+fi
+
+# Every container this campaign boots is read through the ro /models mount, so
+# the host path must live under it; the container-side spelling is what goes
+# into the argv. Same rule for the container a score pass opens.
+under_models() {
+    case "$1" in
+        "$MODELS"/*) printf '/models/%s' "${1#"$MODELS"/}" ;;
+        *) return 1 ;;
+    esac
+}
+
+# same_dir A B — the two name one directory, judged only when BOTH exist: a dry
+# run has no fixtures on disk, and two absent paths are not the same place.
+same_dir() {
+    [ -d "$1" ] && [ -d "$2" ] && [ "$(cd "$1" && pwd)" = "$(cd "$2" && pwd)" ]
+}
+
+if [ "$MODE" = score ]; then
+    SCORE_RAD="${POS[0]}"
+    SCORE_OUT="${POS[1]}"
+    OUT_DIR="$(dirname "$SCORE_OUT")"
+    MODEL_IN="$(under_models "$SCORE_RAD")" || {
+        echo "REFUSED: container to score $SCORE_RAD is not under the mounted models dir $MODELS — the engine cannot read it" >&2
+        exit 2
+    }
+else
+    MODEL_IN="$(under_models "$RAD")" || {
+        echo "REFUSED: reference container $RAD is not under the mounted models dir $MODELS — the engine cannot read it" >&2
+        exit 2
+    }
+fi
+
+# The measuring shape. Identical for kldref and score by construction — this
+# array is the only place the campaign's KL numbers are defined.
+ARGS=(--tp 2 --placement expert_tiered --weights-disk-tier
+      --host-pool-mib "$HOST_POOL_MIB" --gpu-headroom-mib "$GPU_HEADROOM_MIB"
+      --deterministic --num-speculative-tokens 0
+      --kv-cache-dtype fp8 --max-model-len "$MAX_MODEL_LEN"
+      --ngram-placement ram
+      --override-chat-template /models/tpl.jinja)
+MOUNTS=(-v "$MODELS:/models:ro")
+TAPENV=()
+case "$MODE" in
+  kldref)
+    # Recording is prefill-only over four slots and the shared step, which is
+    # production's own operating point (frank-archon.env: FRANK_ARCHON_MAX_SEQS=4
+    # with FRANK_ARCHON_MAX_BATCHED_TOKENS=2048). Corpus read-only, record dir rw.
+    ARGS+=(--model "$MODEL_IN"
+           --kld-record /data/kld-ref --kld-corpus /data/kld.jsonl
+           --max-num-seqs 4 --max-num-batched-tokens "$MNBT")
+    MOUNTS+=(-v "$KLD_CORPUS:/data/kld.jsonl:ro"
+             -v "$KLD_REF_DIR:/data/kld-ref")
+    ;;
+  score)
+    # The same argv with three words changed: which container, and that this
+    # time the engine reads a recording and writes a report instead of the
+    # reverse. The reference dir mounts READ-ONLY — a scoring pass must not be
+    # able to touch the zero point — and the report gets its own writable bind.
+    if [ "$OUT_DIR" = "$KLD_REF_DIR" ] || same_dir "$OUT_DIR" "$KLD_REF_DIR"; then
+        echo "REFUSED: score output $SCORE_OUT is inside the reference dir $KLD_REF_DIR, which this mode mounts read-only" >&2
+        exit 2
+    fi
+    ARGS+=(--model "$MODEL_IN"
+           --kld-ref /data/kld-ref --kld-out "/data/out/$(basename "$SCORE_OUT")"
+           --max-num-seqs 4 --max-num-batched-tokens "$MNBT")
+    MOUNTS+=(-v "$KLD_REF_DIR:/data/kld-ref:ro"
+             -v "$OUT_DIR:/data/out")
+    ;;
+  calib)
+    # Calibration is exempt from the shared step: the tap counts pooled
+    # positions and nothing downstream is compared against a step size, so the
+    # wide step is free throughput (a disk-tier step costs the same at 1 token
+    # as at 64k).
+    ARGS+=(--model "$MODEL_IN" --port 7770
+           --max-num-seqs 8 --max-num-batched-tokens 65536)
+    MOUNTS+=(-v "$CALIB_DIR:/data/calib")
+    TAPENV=(-e RADIANCE_CALIB_DIR=/data/calib
+            -e RADIANCE_CALIB_TOKENS=12000000)
+    ;;
+esac
+
+CMD=(podman run --rm --name "$C"
+     --network host --init
+     --security-opt seccomp=unconfined
+     --ulimit memlock=-1:-1
+     --device /dev/kfd --device /dev/dri
+     "${MOUNTS[@]}" "${TAPENV[@]}"
+     "$IMAGE" "${ARGS[@]}")
+
+# ---- gate: the window must hold the corpus (kldref alone can meet it) --------
+# score reads a recording instead of a corpus, and every arm declares at the
+# checkpoint's training context, so neither can meet a sequence too long to boot.
+# Measured on the raw JSON line — the pessimistic direction, and the same figure
+# the census in the header quotes. It runs on a dry run too, because a refusal
+# this cheap should never have to wait for the drained window to be discovered.
+if [ "$MODE" = kldref ] && [ -f "$KLD_CORPUS" ]; then
+    read -r REC_NO REC_CHARS < <(awk '{ if (length($0) > best) { best = length($0); where = NR } } END { print where + 0, best + 0 }' "$KLD_CORPUS")
+    LIMIT_CHARS=$(( MAX_MODEL_LEN * 3 ))
+    if [ "$REC_CHARS" -gt "$LIMIT_CHARS" ]; then
+        echo "REFUSED: record $REC_NO of $KLD_CORPUS is $REC_CHARS chars, past the $LIMIT_CHARS a $MAX_MODEL_LEN-token window promises at the pessimistic chars/3 — raise FRANK_QWEN38_MAX_MODEL_LEN, and the container's declared --max-ctx with it, before booting the reference" >&2
+        exit 2
+    fi
+    echo "corpus fits: longest record $REC_CHARS chars (number $REC_NO) inside the $MAX_MODEL_LEN-token window (limit $LIMIT_CHARS chars)"
+fi
+
+if [ "$DRY" = 1 ]; then
+    echo "dry run: $MODE — the production unit is left running, nothing is mounted, nothing is written"
+    printf '%q\n' "${CMD[@]}"
+    exit 0
+fi
+
+# ---- gates that only a real run needs ---------------------------------------
+if [ "$MODE" = score ]; then
+    if [ ! -d "$KLD_REF_DIR" ]; then
+        echo "REFUSED: no KLD reference at $KLD_REF_DIR — record it first in a drained window: scripts/run-qwen38-bf16.sh kldref" >&2
+        exit 2
+    fi
+    if [ ! -f "$SCORE_RAD" ]; then
+        echo "REFUSED: no container to score at $SCORE_RAD" >&2
+        exit 2
+    fi
+    if [ -e "$SCORE_OUT" ]; then
+        echo "REFUSED: $SCORE_OUT already exists — one report per name; move the old one to trash before re-scoring" >&2
+        exit 2
+    fi
+    mkdir -p "$OUT_DIR"
+else
+    if [ ! -f "$RAD" ]; then
+        echo "REFUSED: no reference container at $RAD — build it first: scripts/build-qwen38-bf16-rad.sh" >&2
+        exit 2
+    fi
+    [ "$MODE" = kldref ] && [ ! -f "$KLD_CORPUS" ] && {
+        echo "REFUSED: no KLD corpus at $KLD_CORPUS — the record would be empty" >&2
+        exit 2
+    }
+fi
+if [ ! -f "$TEMPLATE" ]; then
+    echo "REFUSED: no chat template at $TEMPLATE" >&2
+    exit 2
+fi
+mkdir -p "$LOG_DIR"
+# The prod template, staged where the ro /models mount reaches it: a reference
+# recorded with the container's baked template scores a different tokenisation
+# than the arms served with froggeric's, and the KL would be measuring that.
+cp -f "$TEMPLATE" "$TPL" || {
+    echo "REFUSED: cannot stage the chat template at $TPL (the container reads it through /models)" >&2
+    exit 2
+}
+[ "$MODE" = kldref ] && mkdir -p "$KLD_REF_DIR"
+[ "$MODE" = calib ]  && mkdir -p "$CALIB_DIR"
+
+restore() { sudo systemctl start "$SERVICE" >/dev/null 2>&1 || true; }
+trap restore EXIT
+echo "run-qwen38-bf16 $MODE: stopping $SERVICE ($(date -Is)); log: $LOG"
+sudo systemctl stop "$SERVICE" >/dev/null 2>&1 || true
+podman rm -f "$C" >/dev/null 2>&1 || true
+
+"${CMD[@]}" 2>&1 | tee "$LOG"
+rc=${PIPESTATUS[0]}
+echo "run-qwen38-bf16 $MODE exited rc=$rc; $SERVICE restored; log: $LOG"
+exit "$rc"

scripts/serve-frank-archon.sh

@@ -15,18 +15,21 @@
 # log states every budget before anything is allocated (weights vs KV
 # split, the expert host pool's coverage of the worst case, the PLE pin,
 # the P2P answer per pair) — read it after every resize.
-# Deployed shape (2026-10-19, radiance 1.8.1): THREE decode slots x 200,000
-# tokens, fp8 KV, 44 GiB/rank pinned expert pool, MTP depth 2, prefix cache in
-# pinned host RAM, PLE pinned in RAM. The per-card split between the expert slab
-# and the KV pool is NO LONGER an operator number — 1.3.0 removed the VRAM pair
-# and the budget resolver derives it from headroom, the cache bound and the host
-# pool, so the slab size and the pool's token count are LEDGER lines: after every
-# boot read them out of the log and check the pool still backs 3 x 200,000.
+# Deployed shape (2026-10-20, radiance 1.8.1): FOUR decode slots x 200,000 tokens over a
+# >= 700,000-token fp8 KV pool, 48 GiB/rank pinned expert pool, MTP depth 2, prefix cache
+# in pinned host RAM, PLE pinned in RAM. The per-card split between the expert slab and the KV
+# pool is NO LONGER an operator number — 1.3.0 removed the VRAM pair and the budget
+# resolver derives it from headroom, the cache bound and the host pool, so the slab
+# size and the pool's token count are LEDGER lines. After every boot read them, plus
+# the two lines that size a prefill step — "activation arena: ... lends ... while
+# steps are at most N tokens" (the arena levels the expert slab borrows) and
+# "prefill staging: ... a step of more than N tokens ..." (the chunk size below which
+# pooled experts are read synchronously) — and the cache's session backing.
 #
 # The root unit (scripts/frank-archon.service, User=dank,
-# LimitMEMLOCK=infinity) grants the pin budget: 2 x 44 GiB expert pools +
-# the 47.7 GiB PLE table + 8 GiB prefix host tier — ~147 GiB of pinned
-# pages, which is ALSO why this script refuses to start when ttm's
+# LimitMEMLOCK=infinity) grants the pin budget: 2 x 48 GiB expert pools + the 47.7 GiB PLE
+# table + 8 GiB prefix host tier + the checkpoint store (slots x 112 MiB) — ~159 GiB of
+# pinned pages, which is ALSO why this script refuses to start when ttm's
 # pages_limit/pages_max cap is under FRANK_ARCHON_TTM_MIN_MIB (see below).
 set -euo pipefail
 

scripts/serve-frank-dragoon.sh

@@ -136,10 +136,17 @@ fi
 #      The engine re-checks size+sha256 against that catalog at boot; it cannot
 #      fetch anything here because /models is read-only and HALOGEN_DOWNLOAD
 #      stays unset — no outbound connection at boot),
-#   2. the distro XRT's three libraries, each mounted twice (where the image
-#      looks for XRT, and at its own path — the libraries find each other there;
-#      an /opt/xilinx/xrt of symlinks into the system dir dangles inside the
-#      container, which the vendor calls out explicitly),
+#   2. the distro XRT's three libraries, each mounted THREE times: where the
+#      image looks for XRT (/opt/xilinx/xrt/lib), at their own path (the
+#      libraries find each other there), and at /opt/xilinx/lib/<multiarch>
+#      — the NPU child searches that third prefix too, and since the host's
+#      libxrt2 was reconfigured 2026-10-20 it is the ONLY prefix the child
+#      names: omitting it crashes the lane with `No such library
+#      '/opt/xilinx/lib/x86_64-linux-gnu/libxrt_core.so.2'` and the engine
+#      supervisor takes the dead lane down on the whole container (measured
+#      2026-10-21: 88-restart boot loop). Real files only — an /opt/xilinx/xrt
+#      of symlinks into the system dir dangles inside the container, which the
+#      vendor calls out explicitly),
 #   3. the GPU fabric clock HELD: on Strix Halo the iGPU and the NPU co-running
 #      hang the machine while FCLK changes speed, and the engine refuses to
 #      start when it moves — it cannot read host sysfs from inside the
@@ -190,7 +197,9 @@ if [ -n "$NPU_MODELS" ]; then
             exit 1
         fi
         REAL="$(readlink -f "$XRT_DIR/$L")"
-        NPU_ARGS+=(-v "$REAL:/opt/xilinx/xrt/lib/$L:ro" -v "$REAL:$XRT_DIR/$L:ro")
+        MULTIARCH="${XRT_DIR##*/}"
+        NPU_ARGS+=(-v "$REAL:/opt/xilinx/xrt/lib/$L:ro" -v "$REAL:$XRT_DIR/$L:ro"
+                   -v "$REAL:/opt/xilinx/lib/$MULTIARCH/$L:ro")
     done
     HELD=0
     for D in "$SYSFS"/class/drm/card*/device; do

scripts/test-changed.sh

@@ -293,6 +293,14 @@ for path in "${CHANGED[@]}"; do
             add_test tests/caddyfile.bats
             add_test tests/deployment_scoping.bats
             ;;
+        # The GPU exporter is a host process; its seam pins the collector, the
+        # user unit, the deployer function, the planner route, the Prometheus
+        # scrape jobs and the host-system GPU panels — a cross-file contract, so
+        # the whole seam rides on any of its own files.
+        scripts/gpu-exporter.py|scripts/gpu-exporter.service)
+            add_test tests/gpu_exporter.bats
+            add_test tests/deployment_scoping.bats
+            ;;
         dankclaw/udev/*)
             # Host-level udev rules are applied by manual sudo install + udevadm
             # trigger, not dispatched as a service; nothing here is bats-testable.

serverAppConfigs-docs/Agent/OMP Harness.md

@@ -52,28 +52,31 @@ Where the contract lives:
 The arithmetic (the coupled contract every endpoint's maxTokens comment
 encodes). The authoritative `contextWindow`/`maxTokens` pair per model is on the
 endpoint page ([[frank-archon]], [[frank-dragoon]], [[frank-zealot]]); what
-follows is a **worked example** recomputed from the live files on 2026-10-18 —
-a snapshot, not fleet state — with windows 200,000 / 155,648 / 200,704 and
+follows is a **worked example** recomputed from the live files on 2026-10-21 —
+a snapshot, not fleet state — with windows 200,000 / 200,000 / 200,704 and
 maxTokens 16,384 / 45,056 / 16,384 (archon / dragoon / zealot):
 
 1. **Source trigger** = `thresholdPercent × source contextWindow`; at the live
-   95 % that is 190,000 (archon), 147,866 (dragoon), 190,669 (zealot). The
+   95 % that is 190,000 (archon and dragoon), 190,669 (zealot). The
    bundle clamps the percentage to 1–99 and converts it to a token threshold.
 2. **Output budget** = capped at the target model's `maxTokens` — every role
    here compacts to frank-zealot, so the budget is its 16,384.
-3. **REQUIREMENT**: prompt (≤ trigger) + budget ≤ the target's per-slot
-   window. **At 95 % this no longer holds**: 190,000 + 16,384 = 206,384 >
-   200,704. The registry's own comments design around an 85 % trigger (archon:
-   170,000 est × 1.03 + 16,384 ≈ 191,500 ≤ 200,704, `models.yml:49-56`), which
-   does hold. While `config.yml` and those comments disagree, treat 85 % as the
-   design point and re-run this arithmetic after ANY window change — a 400 from
-   the compaction target mid-conversation is the symptom.
+3. **REQUIREMENT**: prompt (≤ trigger) + budget ≤ the fitting window. For a
+   dragoon session the fitting window is the ENGINE ceiling, not a slot: the
+   190,000 trigger is 62,144 under the 262,144 ceiling, so even ×1.03 est
+   error plus the 45,056 request budget lands inside — that headroom is the
+   reason the ceiling moved to native on 2026-10-21. For a session compacting
+   INTO frank-zealot the window is its 200,704-token slot, and **at 95 % this
+   does not hold for archon/zealot sources**: 190,000 + 16,384 = 206,384 >
+   200,704. Treat 85 % as the design point until config.yml and the registry
+   comments agree, and re-run this arithmetic after ANY window change — a 400
+   from the compaction target mid-conversation is the symptom.
    (Historic 2026-10-05 pairs, at `thresholdPercent: 70`: 140,492 + 36,864 =
    177,356 ≤ 200,704 archon; 140,490 + 57,344 ≤ 200,700 dragoon; 140,492 +
    32,768 ≤ 200,704 zealot.)
 
 Procedure (recompute from the LIVE files every time — windows rotate with
-every swap; the block above is the 2026-10-18 state):
+every swap; the block above is the 2026-10-21 state):
 
 1. Read the `config.yml` compaction block + `models.yml` entries for source
    and target. Every endpoint's `contextWindow` comment carries its own
@@ -94,9 +97,10 @@ Known reconciliation trap: comments in models.yml have described compaction
 as running "on the dankserver endpoint" while `remoteEndpoint` pointed at
 dankclaw — the comment/config disagreed. The comments now name the real
 target (`compaction.remoteEndpoint` = dankclaw :7771). **The config is
-truth; fix comments in the same change.** A fresh instance of the same trap is
-open 2026-10-18: every `maxTokens` comment still says "compaction fires at
-85 % of the window" while `config.yml:84` sets `thresholdPercent: 95`.
+truth; fix comments in the same change.** The 85 %-comments vs 95 %-config
+disagreement is not academic: it is why long dragoon sessions kept drawing
+400s at the old 200,704 ceiling — the trigger fires late AND the est
+under-counts, and a window parked near the ceiling has no room for either.
 `maxTokens` never changes for a reslice — it caps output; `contextWindow` caps
 input. Boot-shape OOM laddering is a different problem
 ([[Llama Cpp Serving Sizing]]).

serverAppConfigs-docs/Architecture/Quadlet Lifecycle.md

@@ -130,10 +130,19 @@ Quickest unblock: `podman compose up -d --force-recreate <svc>`. Durable fix
 
 ## 4. Podman 5.x quadlet v1 generator quirks (verified live 2026-08-28)
 
-- **Hard-fail keys** — the whole unit silently fails to generate
-  (`systemctl --user cat <unit>` shows "could not be found"): any
-  dash-prefixed `[Container]` option (`--shm-size=…` → `unsupported key …
-  in group 'Container'`); `Command=` is unsupported, use `Exec=`.
+- **Hard-fail keys abort the WHOLE BATCH, not one unit** (podman 5.7.0,
+  live 2026-10-21): one unsupported key — e.g. `HostAddr=` — makes the
+  generator log `unsupported key … in group 'Container'` plus
+  `processing encountered some errors` and generate NOTHING for that
+  manager. The running containers keep going as orphans, so the failure
+  first appears on the next deploy as `Unit <x>.service not found` for a
+  service that is visibly up. Other offenders: any dash-prefixed
+  `[Container]` option (`--shm-size=…`), `Command=` (use `Exec=`).
+  The seam `tests/quadlet_image_pins.bats` fails the build on any
+  generator-rejected key in any host's units.
+  Also: an image whose ENTRYPOINT is the binary takes FLAGS ONLY in
+  `Exec=` — repeating the binary makes it a positional arg
+  (`unexpected /bin/blackbox_exporter`, live 2026-10-21).
 - **HealthCmd space-splitting:** the v1 generator splits `HealthCmd` on
   spaces into the healthcheck argv; a quoted probe
   `/bin/sh -c "pg_isready …"` lands as argv token `"pg_isready` →
@@ -148,6 +157,16 @@ Quickest unblock: `podman compose up -d --force-recreate <svc>`. Durable fix
   "unsupported key"`, `systemctl --user cat probe.service` (shows the
   generated ExecStart verbatim), then remove + reload.
 
+## 4a. Image pins (fleet law, 2026-10-21)
+
+Every registry image in a quadlet is digest-pinned (`tag@sha256:…`);
+locally built images (`localhost/*`, `franksdank/*`) are exempt — their
+build is the pin. A rolling tag lets a rebuild move under a running
+service with nobody approving it, and it makes a CVE audit ungradeable.
+Gate: `tests/quadlet_image_pins.bats`. Where upstream publishes no version
+tags (copyparty-ac stops at 1.19.15), pin `:latest@sha256:` and name the
+version the config label carries in the comment beside it.
+
 ## 5. Rootless Quadlet log retrieval
 
 Step 0 — establish ownership: `systemctl --user status <name>.service`

serverAppConfigs-docs/Architecture/Stale Signal Audit.md

@@ -116,6 +116,20 @@ as working when they could never report up. **Clean cutover includes prose** —
 docs, unit docstrings, and the repo's rule file (the most expensive place to
 be stale).
 
+## Three rules that outlive the incident that taught them
+
+* **A target can read down beside a healthy service.** Read `lastError` from
+  `/api/v1/targets` before believing an outage — llama.cpp refuses `/metrics` at the
+  protocol layer without `--metrics` (501) while `/health` answers 200 for weeks.
+* **Alert state lives where the tool says it does.** Prometheus keeps rule state in
+  process memory — a restart re-stamps `activeAt`, so a "fresh" alert after a reboot is
+  not evidence of a fresh fault. Alertmanager **does** persist (`--storage.path` on a
+  host volume: silences + notification log, [[Alertmanager]]).
+* **Never quote a candidate arm's flags or VRAM as fleet state.** A stopped production
+  unit (`failed` after `stop-sigterm timed out. Killing` — the known shutdown shape,
+  [[frank-zealot]]) beside an answering port means the metric measured a scratch boot;
+  the process shape can change between two censuses ten minutes apart.
+
 ## Order and verification
 
 1. Enumerate the surfaces (status board, metrics targets, alert rules, test

serverAppConfigs-docs/Dankclaw/ComfyUI Qwen-Image Workflows.md

@@ -110,9 +110,9 @@ only `models`, `input`, `output`, `custom_nodes` and the healthcheck file
 
 ## Recommended additions
 
-Quantized preferred, sized for the 8060S iGPU (GTT-backed — `/system_stats`
-reports 122.4 GB of device VRAM on `cuda:0`, `curl http://127.0.0.1:8188/system_stats`
-2026-10-18; the RTX 3090 is frank-zealot's). All repos verified 2026-10-05 and
+Quantized preferred. The iGPU's GTT aperture is not free RAM beside
+frank-dragoon — size every pick against *What actually fits beside
+frank-dragoon* below (the RTX 3090 is frank-zealot's). All repos verified 2026-10-05 and
 all four still resolve today (`huggingface.co/lightx2v/Qwen-Image-Lightning`
 `huggingface.co/unsloth/Qwen-Image-Edit-2511-GGUF`
 `huggingface.co/Comfy-Org/Qwen-Image-DiffSynth-ControlNets`
@@ -133,8 +133,34 @@ sha256 checks) or a plain `hf download` with the token in the environment —
 never on the command line.
 
 Skip list: the original `city96/Qwen-Image-gguf` conversions (superseded by
-the 2.1 files already on disk), and any BF16 of #2/#4 (the iGPU does not need
-them at GTT capacity, and they double the disk bill).
+the 2.1 files already on disk), and any BF16 of #2/#4 — they double the disk
+bill, and see *What actually fits beside frank-dragoon* for why raw GTT size
+is not the real budget.
+
+## What actually fits beside frank-dragoon
+
+The 122 GB `/system_stats` figure is the GTT **aperture**, not free memory: on
+an APU the aperture and `MemAvailable` are the same physical RAM, and
+frank-dragoon pins tens of GiB of it. Measured 2026-10-21 with frank-dragoon
+live: `MemAvailable` 15.3 GiB, iGPU busy 1.75 GiB — ComfyUI itself idles at
+~10 MB, so the budget for a generation run is roughly what `free` says.
+
+Weights must land inside that budget with slack for compute buffers and
+activations (the mmproj-class lesson: price a model at ~1.5× its file set):
+
+| Pipeline (weights) | Fits beside frank-dragoon? |
+|---|---|
+| Qwen-Image 2.1 int8_convrot + w4a8 + bf16 VAE (~14.2 GB) — the two proven workflows | **No** — thrash-or-fail; do not queue it while the endpoint serves |
+| GGUF Q8_0 + w4a8 + small VAE (~13.3 GB) | No, same reason |
+| Any BF16 combination | No |
+| SDXL-Turbo / SD 1.5-class (~2–7 GB) | Yes, with real headroom |
+
+**The law: one process owns the memory — same rule as the 3090 bench law.**
+For a Qwen-Image render: stop and mask frank-dragoon
+(`sudo systemctl stop frank-dragoon` + `mask` — the mask stops
+`Restart=always` from firing mid-render), generate, unmask, verify
+`curl 192.168.0.143:7770/health`. The proven workflows above then run with
+the whole machine. Do not treat `/system_stats`'s 122 GB as an invitation.
 
 ## See Also
 

serverAppConfigs-docs/Dankclaw/ComfyUI.md

@@ -129,6 +129,12 @@ bind-mounted to `/usr/local/bin/comfyui-healthcheck.py` and run as `HealthCmd`
 - Do not let pip reinstall the torch family. The Containerfile filters it out on purpose.
 - The base image is about 32 GiB (`podman image inspect localhost/comfyui-rocm:gfx1151` → 33,765,227,471 B ≈ 31.4 GiB, 2026-10-18). A code update means a rebuild, because the code is baked in.
 - The empty-models note from the 2026-09-16 era is retired: `/home/dank/comfyui-data/models` now holds the Qwen-Image 2.1 payload — three `diffusion_models`, three `text_encoders`, two VAEs (~35 GB, `ls -l` 2026-10-18), listed in [[ComfyUI Qwen-Image Workflows]]. New model classes (LoRA, ControlNet) still need their directories created first: `models/lora` and `models/controlnet` do not exist yet.
+- **The 122 GB `/system_stats` VRAM figure is the GTT aperture, not free
+  memory.** On the APU it is system RAM, and frank-dragoon pins most of it
+  (`MemAvailable` 15.3 GiB with the endpoint live, measured 2026-10-21). A
+  Qwen-Image-class render beside the running endpoint is thrash-or-fail; the
+  stop-mask-render-verify procedure and the fit table are in
+  [[ComfyUI Qwen-Image Workflows]] → What actually fits beside frank-dragoon.
 
 ## See Also
 

serverAppConfigs-docs/Dankclaw/OpenWebUI.md

@@ -34,15 +34,16 @@ https://github.com/open-webui/open-webui
 The picker is built from the `model` table in `webui.db` (one row per model)
 plus the `model_ids` of each **enabled** OpenAI connection, NOT from live
 connection discovery — so the picker shows exactly what is seeded there.
-As of 2026-10-07 the picker exposes **exactly one model: `frank-zealot`** *(unverified
-2026-10-18: `webui.db` is the authority; the fleet itself serves all three ids
-`frank-dragoon`/`frank-zealot`/`frank-archon`, per the cards in
-`shared/homepage/services.json`)*.
+Since 2026-10-21 the picker exposes **all three fleet models to every
+account**: `frank-dragoon`, `frank-archon`, `frank-zealot` (per Frank: open
+`frank-archon` to everyone; the other two were already public). Proven with
+a `role=user` token: `/api/models` lists all three ids and a chat turn to
+`frank-archon` answers end-to-end.
 
-| Connection (`openai.api_configs` idx) | URL as recorded 2026-10-09 | Enabled | `model_ids` | Endpoint page |
+| Connection (`openai.api_configs` idx) | `openai.api_base_urls` (live 2026-10-21) | Enabled | `model_ids` | Endpoint page |
 |---|---|---|---|---|
-| 0 | `http://host.containers.internal:7770/v1` | no (2026-10-07) | `frank-dragoon` | [[frank-dragoon]] |
-| 1 | `http://192.168.0.235:7770/v1` | no (2026-10-07) | `frank-archon` | [[frank-archon]] |
+| 0 | `http://host.containers.internal:7770/v1` | yes | `frank-dragoon` | [[frank-dragoon]] |
+| 1 | `http://192.168.0.235:7770/v1` | yes | `frank-archon` | [[frank-archon]] |
 | 2 | `http://host.containers.internal:7771/v1` | yes | `frank-zealot` | [[frank-zealot]] |
 
 The URLs in the first column are the DB's recorded values and predate the 2026-08-22
@@ -50,9 +51,10 @@ bridge move; on this network they are wrong — see the reachability gotcha belo
 is the corrected law. The picker row for `frank-zealot` was re-keyed on 2026-10-18 after
 that day's payload swap (see [[frank-zealot]] for what was deployed and when).
 
-The `model` table carries exactly two decoration rows, `frank-zealot` and
-`frank-dragoon`. Both the context length and the description live **inside
-JSON columns** — `params` holds `{"context": {"length": …}}` and `meta` holds
+The `model` table carries three decoration rows — `frank-zealot`,
+`frank-dragoon`, `frank-archon` (200,000 context, added 2026-10-21).
+Both the context length and the description live **inside JSON columns** —
+`params` holds `{"context": {"length": …}}` and `meta` holds
 `{"description": …}`; there is no `context_length` or `description` column, so
 `sqlite3` one-liners against those names fail. `params.context.length` is a copy
 of the endpoint's per-slot window, so the authoritative number lives on the
@@ -380,6 +382,20 @@ neither.
   name: /selected model/i })`) and the welcome text "How can I help you
   today?"; that snapshot is also the cheapest proof the picker is populated.
 
+- **2026-10-21 — picker opened to all three models, for everyone.** Per
+  Frank: `frank-archon` accessible to everyone. Three-leg repair on
+  `webui.db` (snapshot `webui.db.bak-20261021` first): connection idx 1
+  `enable: true`, a `model` row `frank-archon` (params
+  `{"context":{"length":200000}}`, meta description from the endpoint card),
+  and an `access_grant` row `(resource_type='model', resource_id='frank-archon',
+  principal_id='*', 'read')` — `principal_id='*'` is the public grant, not a
+  role row. Verified as a plain `role=user` token: `/api/models` lists all
+  three ids, chat to `frank-archon` answers. Token-mint gotchas: the backend
+  package imports only with cwd `/app/backend`, and `WEBUI_SECRET_KEY` must
+  come from **PID 1's environ** (`podman exec` sessions do not inherit the
+  app process env; the `.webui_secret_key` file happens to equal it — use it
+  explicitly, `podman exec -e`).
+
 ## See Also
 
 - [[Dankclaw Stack]]

serverAppConfigs-docs/Dankserver/Plex.md

@@ -4,7 +4,7 @@ description: "Plex Media Server is a **containerized media server on Dankserver*
 tags:
   - dankserver
   - plex
-timestamp: 2026-09-15
+timestamp: 2026-10-21
 ---
 
 # Plex
@@ -24,7 +24,7 @@ public traffic can be fronted by Caddy like every other vhost.
 | Item | Value |
 |------|-------|
 | Runtime | podman quadlet, `plex.service` (user manager) — migrated from the native `plexmediaserver.service` OS-package unit (2026-09) |
-| Image | `lscr.io/linuxserver/plex:1.43.4.10903-e5521bd8c-ls325` (pinned; verified == the ghcr `latest` manifest digest at pin time) |
+| Image | `lscr.io/linuxserver/plex:1.43.5.11029-6bf8af3f0-ls329` (pinned; verified == the registry's `latest` manifest digest at pin time: `sha256:8f655bad…e9e2334`) |
 | Data root | `/var/lib/plexmediaserver` bind-mounted **in place** as `/config` (RW) — the migration moved zero bytes |
 | Media | `/mnt/entropy` mounted at the same absolute path **read-only** — every library path in the PMS database stayed valid, zero re-scan |
 | Identity | fleet rootless model — container root maps to the `dank` user (1000:1000); the deployer chowns the data root on first cutover |

serverAppConfigs-docs/Infrastructure/Observability Provisioning.md

@@ -3,7 +3,7 @@ type: reference
 title: Observability Provisioning
 description: "Monitoring config as code: the config triad, deploy copiers, planner mappings, and where each change type lands."
 tags: [observability, prometheus, grafana, provisioning, deployment]
-timestamp: 2026-10-05T00:00:00-07:00
+timestamp: 2026-10-21T00:00:00-07:00
 ---
 
 # Observability Provisioning
@@ -20,6 +20,8 @@ committed.
 | `shared/prometheus/prometheus.yml` | scrape jobs (global + per-host) |
 | `shared/prometheus/rules/llm-fleet.yml` | alert rules + recording rules |
 | `shared/grafana/provisioning/` | datasources + dashboards (provisioned tree) |
+| `shared/alloy/config.alloy` | the log pipeline: journal source → label promotion → Loki push |
+| `shared/blackbox/blackbox.yml` | prober modules (`http_2xx`); the listen bind lives on the unit's `Exec` |
 
 Targets and datasources reference **container names** (`grafana:3000`
 `loki:3100` — `shared/prometheus/prometheus.yml:52-61`), never
@@ -31,8 +33,8 @@ scraped as `host.containers.internal:<port>` because a bridge container cannot
 reach a host-netns bind at the host's LAN IP (`prometheus.yml:89-99`), and only
 the endpoint on another machine (`192.168.0.235:7770`) is addressed by LAN IP
 (`prometheus.yml:128-131`). Prometheus 3 refuses HTML responses as metrics —
-HTML-scraped "probes" are structurally inert and need a blackbox exporter (see
-[[Stale Signal Audit]]).
+HTTP liveness rides the **blackbox exporter** (`shared/blackbox/blackbox.yml`,
+deployed 2026-10-21; see [[Stale Signal Audit]]).
 
 ## The deploy path
 
@@ -43,8 +45,9 @@ from `deploy_observability`:
 - `copy_alertmanager_config` → `/opt/alertmanager/config/alertmanager.yml` (`:232-235`)
 - `install_signal_relay` → the user unit + iptables + the health wait on `192.168.0.143:18093/health` (`:243-274`)
 - `copy_loki_config` → `/opt/loki/config/loki-config.yaml` (`:276-279`)
-- `copy_alloy_config` → `/opt/alloy/config/config.alloy` (`:281-284`)
-- `copy_grafana_provisioning` → `/opt/grafana/provisioning/` (`:286-300`)
+- `copy_alloy_config` → `/opt/alloy/config/config.alloy` — substitutes the `GRAFANA_HOST_LABEL` token with the deploy target's name and **refuses to install** if the token survives; `ALLOY_CONF_ROOT` is the bats sandbox seam
+- `copy_blackbox_config` → `/opt/blackbox/blackbox.yml` (single-file mount of the prober); `BLACKBOX_CONF_ROOT` is its sandbox seam
+- `copy_grafana_provisioning` → `/opt/grafana/provisioning/` — datasources, provider, and **every** `dashboards/*.json` (a glob, not a hand-listed subset; `GRAFANA_PROV_ROOT` is the test seam the bats case uses to run the copier into a sandbox)
 
 `scripts/deploy-plan.py` maps `shared/prometheus/**` to `prometheus` (plus
 `alertmanager` when the changed file is `alertmanager.yml`, `:251-256`)
@@ -82,11 +85,14 @@ reports 4 alerting + 2 recording. Rules reload with the prometheus service.
 the backend (`http://prometheus:9090`), not loopback.
 
 **Add a dashboard** — drop the JSON into
-`shared/grafana/provisioning/dashboards/`. Provisioned dashboards live in
-unified storage, not (only) the SQLite table — grep the API, not the DB, when
-verifying (see the verify section of [[Fleet Audit and Posture]]). *(unverified
-2026-10-18: the storage-backend claim needs an admin password to confirm
-against `grafana.db` and the API side by side)*
+`shared/grafana/provisioning/dashboards/`; the copier globs the directory, so
+the next `--service grafana` deploy provisions it. This is a repaired law, not
+a given: until 2026-10-21 the copier named `llm-fleet.json` alone, and the
+`containers` board — merged, tested, documented as "working" — had **never
+been provisioned**; the other boards existed on disk only because someone had
+hand-copied them once. A repo JSON is not a live dashboard until the copier
+carries it, and `deploy.sh` cannot see the gap because it only reports the
+copy it performed.
 
 ## Verify through the APIs, not the filesystem
 
@@ -95,90 +101,32 @@ against `grafana.db` and the API side by side)*
 curl -s http://127.0.0.1:9090/api/v1/rules | python3 -m json.tool | head
 curl -s "http://127.0.0.1:9090/api/v1/targets?state=active" | \
   python3 -c "import json,sys; [print(t['labels'].get('job'), t['health'], (t['lastError'] or '')[:60]) for t in json.load(sys.stdin)['data']['activeTargets']]"
-# grafana: datasource + dashboard provisioned
-curl -s -u admin:<pw> http://127.0.0.1:3002/api/datasources | python3 -m json.tool | grep -E '"name"|"url"'
+# grafana 13: /api/health, /api/org, /login all 404 UNAUTHENTICATED, and there
+# is NO local admin password (root .env sets no GF_SECURITY_ADMIN_*; the only
+# account is the Authentik RP login). The proven surface is a real browser
+# session: warm the Anubis challenge on the vhost root, then hit
+# /login/generic_oauth with the e2e storage state (tests/e2e/.auth/qa-e2e-admin.json)
+# — exactly the path specs/20-grafana-sso.spec.ts drives — and read the boards.
 ```
 
 The two Prometheus queries were run on 2026-10-18 and are what every number on
 this page comes from: 4 alerting + 2 recording rules, and `up` for
 `prometheus`, `node-dankclaw`, `node-dankserver`, `grafana`, `loki`
-`frank-zealot`, `frank-dragoon`, `frank-archon` with the four structural downs
-listed in [[Prometheus]]. The Grafana query needs the local admin password and
-was not run here; the filesystem side (`ls /opt/grafana/provisioning/*`) is
-proof of installation only — see the next paragraph.
+`frank-zealot`, `frank-dragoon`, `frank-archon`. The four structural downs
+that census recorded are retired (2026-10-21, [[Prometheus]]). The Grafana proof is the browser session above, run
+2026-10-21: it found a provisioned-file gap the filesystem listing hid. The
+filesystem side (`ls /opt/grafana/provisioning/*`) is proof of installation
+only — see *Add a dashboard* for why a repo JSON is not a live board.
 
 Health answers from a stale process and provisioned resources can hide in
 unified storage — a file existing on disk is not proof the running server
 loaded it. See [[Stale Signal Audit]] for the never-green-target decision
 table (refused = outage, 404/HTML = config defect).
 
-## Current delivered state (delivery chain live 2026-09-17, re-verified 2026-10-18)
-
-Prometheus → Alertmanager (`dankclaw/observability/quadlet/alertmanager.container`
-loopback `:9093`) → host-side Signal relay (`scripts/signal-relay.py`, user
-unit, `192.168.0.143:18093` — the LAN IP, the one bind a container can reach
-via `host.containers.internal`) → signal-cli, addressed by number
-`+12062476336` (`scripts/signal-relay.py:36-43`). Every hop re-checked
-2026-10-18: `ss -ltnp` shows `python3` on `192.168.0.143:18093`
-`curl -s http://192.168.0.143:18093/health` → `ok`, and
-`journalctl --user -u signal-relay` ends with successful `sent to +12062476336:`
-lines for `LlmEndpointDown`, `LlmRequestsQueueing`, `LlmSpecDecodeOff` and
-`LlmPrefixCacheCollapse` (firing and resolved). Some targets read down for
-structural reasons (HTML-scraped "probes", Gitea metrics off); the remedy is
-the blackbox exporter. See [[Prometheus]] → Targets.
-
-**A llama.cpp endpoint is invisible unless it opts in.** Until 2026-10-18,
-`up{job="frank-zealot"}` was 0 for the entire retained history of the metric —
-the oldest sample, 2026-10-11, is already 0 — while the service itself answered
-`/health` 200 the whole time. Cause: llama.cpp only exports `/metrics` when it
-is started with the flag, so the scrape was refused at the protocol layer (the
-flag and its opt-out are on [[frank-zealot]]) and `LlmEndpointDown` (critical)
-stayed firing:
-the relay re-sent it on the 4 h `repeat_interval` for days (`journalctl --user
--u signal-relay` — firing sends at 01:37, 05:42, 09:47, 13:52, 17:57 and 22:02
-on 2026-10-12). It finally resolved at `2026-10-18T01:53:21-07:00`, after the
-metrics opt-in landed that day; the target
-reports up since (`curl -s http://127.0.0.1:9090/api/v1/targets` 2026-10-18).
-How many payload generations it spanned is not readable from any host
-artifact. *(unverified 2026-10-18)*
-
-Two laws fall out of it. A target can read down beside a healthy service, so
-read `lastError` from `/api/v1/targets` before believing an outage. And rule
-state lives in Prometheus' process, not on disk — a restart re-stamps
-`activeAt` on a long-standing alert, so a "fresh" alert after a reboot is not
-evidence of a fresh fault. (Alertmanager by contrast **does** persist:
-`--storage.path=/alertmanager` on a host volume keeps silences and the
-notification log — `dankclaw/observability/quadlet/alertmanager.container:39-42`
-`/opt/alertmanager/data`.)
-
-Remaining known downs are the structural ones above (HTML-scraped probes, Gitea
-metrics, ComfyUI), and `frank-dragoon` exports `/metrics` natively
-([[frank-dragoon]]), which is why
-`frank-dragoon` has always been green. The active-alert count is a
-moment-in-time value and should never be documented as a constant:
-`/api/v1/alerts` was empty at the 2026-10-18 morning census, held one firing
-`LlmPrefixCacheCollapse` (`activeAt` 11:10:48Z, value 2.6e-4) when sampled
-minutes later, and was empty again by 11:56Z; the relay logged the matching
-firing and resolved sends at `04:26:21-07:00` and `04:41:21-07:00`.
-
-**What that zealot scrape measured on 2026-10-18.** As of ~04:30 PDT the
-production `frank-zealot` root unit was stopped **on purpose** for a benchmark
-window on the 3090 — the repo Benchmarking Policy says stop the production
-service before benching — and `:7771` answered from a manually booted candidate
-arm (`systemctl is-active frank-zealot` → `failed` after
-`stop-sigterm timed out. Killing`, which is llama.cpp ignoring SIGTERM past
-`TimeoutStopSec`, the known shutdown shape for these engines). So
-`up{job="frank-zealot"} == 1` that day measured the trial, not the production
-shape in `scripts/frank-zealot.env` — that shape is [[frank-zealot]]'s to state,
-and it is the repo contract.
-The unit stayed `enabled`; `./scripts/deploy.sh dankclaw --service frank-zealot`
-reinstalls and starts it, restoring boot persistence and self-healing. Never
-quote a candidate arm's flags or VRAM as fleet state — the process and its shape
-changed between two censuses ten minutes apart.
-
 ## Related
 
 - [[Observability]] — stack topology and the alert delivery chain
-- [[Stale Signal Audit]] — classifying down targets before "fixing" them
+- [[Stale Signal Audit]] — classifying down targets before "fixing" them, and the
+  down-beside-healthy / alert-state-persistence / candidate-arm rules
 - [[Quadlet Lifecycle]] — the units that host these services
 - [[Deploy State Triage]] — how the planner selects these services

serverAppConfigs-docs/Infrastructure/Observability.md

@@ -2,7 +2,7 @@
 type: reference
 title: "Observability on Dankclaw"
 description: "The observability stack: what runs, how it is deployed, how it is exposed, and what is still unconfigured."
-tags: [observability, prometheus, grafana, loki, alloy, node-exporter, quadlet, dankclaw]
+tags: [observability, prometheus, grafana, loki, alloy, node-exporter, gpu-exporter, quadlet, dankclaw]
 timestamp: 2026-10-20
 ---
 
@@ -10,10 +10,10 @@ timestamp: 2026-10-20
 
 ## Summary
 
-Six containers and one host-side process: Prometheus (metrics), Grafana
+Seven containers and two host-side processes: Prometheus (metrics), Grafana
 (dashboards), Loki (log store), Alloy (log shipper), node-exporter (host
-metrics) and Alertmanager (alert delivery), with `signal-relay` on the host
-carrying every firing group to Signal. All of it is Quadlet-managed,
+metrics), cadvisor (per-container metrics) and Alertmanager (alert delivery),
+plus two host `python3` processes — `gpu-exporter` (per-GPU metrics) and `signal-relay` (Signal delivery). The containers are all Quadlet-managed,
 boot-persistent and loopback-only, deployed by
 `scripts/deploy-remote.sh` → `deploy_observability`. The metrics half works
 end to end; the log half is configured, healthy, and carries no data.
@@ -34,6 +34,8 @@ that proves it today.
 | Loki | `localhost/franksdank/loki:latest` | `127.0.0.1:3100` | `loki.container` | [[Loki]] |
 | Alloy | `localhost/franksdank/alloy:latest` | `127.0.0.1:12345` | `alloy.container` | [[Alloy]] |
 | node-exporter | `docker.io/prom/node-exporter:latest` | host netns: `127.0.0.1:9100` + `192.168.0.143:9100` | `node-exporter.container` | [[Node-Exporter]] |
+| cadvisor | `gcr.io/cadvisor/cadvisor:v0.49.1` | bridge publish `127.0.0.1:8080` + `192.168.0.143:8080` | `cadvisor.container` | — |
+| GPU exporter | host `python3`, not a container | `127.0.0.1:9401` + host LAN `:9401` | `scripts/gpu-exporter.service` (user unit) | — |
 | Signal relay | host `python3`, not a container | `192.168.0.143:18093` | `scripts/signal-relay.service` (user unit) | [[Alertmanager]] |
 
 Every unit lives in `dankclaw/observability/quadlet/`, carries
@@ -44,9 +46,11 @@ linger is on for `dank`). `signal-relay.service` reports `enabled` + `active`
 the same day, and `homepage-status.timer` — the root unit that probes all of
 these on loopback — is `enabled` + `active` (`deploy-remote.sh:319-321` copies
 the probe registry `shared/homepage/services.json` to `/srv/homepage`).
-Dankserver runs one member of this family, node-exporter, as its own unit
-(`dankserver/observability/quadlet/node-exporter.container`, `generated` +
-`active`, binding `127.0.0.1:9100` + `192.168.0.235:9100`).
+Dankserver runs the same family: node-exporter and cadvisor as its own quadlets
+(`dankserver/observability/quadlet/{node-exporter,cadvisor}.container`,
+`generated` + `active`; node-exporter binds `127.0.0.1:9100` + `192.168.0.235:9100`,
+cadvisor publishes `127.0.0.1:8080` + `192.168.0.235:8080`), plus the host-process
+`gpu-exporter` (a user unit installed the same way as `signal-relay`).
 
 ## Exposure
 
@@ -63,10 +67,26 @@ Alertmanager has no vhost at all. node-exporter is the exception that must live
 in the host network namespace, and it binds exactly two addresses rather than
 the wildcard — see [[Node-Exporter]].
 
-The relay's bind is the one address a bridge container can reach on the host:
-`192.168.0.143:18093`, with iptables INPUT admitting loopback, the host itself
-and the podman bridge subnet `10.89.0.0/16` before the DROP
-(`scripts/deploy-remote.sh:255-262`).
+**Same-host reachability from a bridge container is measured, not inferred**
+(2026-10-21, from inside the Prometheus container): `host.containers.internal`
+(the podman gateway) reaches host services whether they bind host-net or are
+published ports — node `:9100`, cadvisor `:8080`, gpu `:9401` all answer there
+— while the host's own LAN IP is refused (`192.168.0.143:8080`, `:9401`), so
+every same-host Prometheus target uses the gateway and only cross-host targets
+(dankserver) use plain LAN. Whether LAN works at all is network-specific: the
+relay at `192.168.0.143:18093` is reached from alertmanager's network only
+while `install_signal_relay`'s INPUT rules are in force: loopback, the host
+itself and the podman bridge subnet `10.89.0.0/16` accepted **above** a
+trailing blanket DROP. The law the poisoned host taught (2026-10-21):
+`iptables -I` prepends, so inserting the DROP last lands it first — the old
+install order dropped every packet (159 on the DROP, 0 on the ACCEPTs), the
+deployer's health gate failed beside a LISTENing socket, and the drain
+aborted twice before `nft list ruleset` named the cause. The install now
+rebuilds the tail (delete the DROP wherever it sits, ACCEPTs at the head,
+DROP appended); verify any change to it with the sandboxed bats case, not
+with `-C` greps — `-C` answers existence, never order. Never point a scrape
+target at a host LAN IP from its own host on the strength of one working
+example; dial the gateway and probe.
 
 ## Deployment
 
@@ -83,13 +103,14 @@ health endpoint (`/prometheus`: `/-/healthy`, `/alertmanager`: `/-/healthy`
 [[Observability Provisioning]] for what is provisioned as code and how to
 verify it through the APIs.
 
-## Current state (2026-10-20)
+## Current state (2026-10-21)
 
 * **Metrics: working.** `curl -s http://127.0.0.1:9090/api/v1/targets` reports
-  eight up targets — the three LLM endpoints (frank-archon on `vllm:*`,
-  frank-dragoon and frank-zealot on `llamacpp:*`), grafana, loki, prometheus
-  and both node exporters. The four downs are structural (the `comfyui` and
-  `services` HTML probes, and Gitea's disabled metrics endpoint), not outages.
+  every job up — the three LLM endpoints (frank-archon on `vllm:*`,
+  frank-dragoon and frank-zealot on `llamacpp:*`), grafana, loki, prometheus,
+  both node exporters, cadvisor, gitea (metrics enabled in its quadlet) and
+  the blackbox-probed `services` (SearXNG, OpenWebUI, ComfyUI). The structural
+  downs are retired — see [[Prometheus]] → Targets.
   Two rule groups in `shared/prometheus/rules/llm-fleet.yml`: `llm-fleet`
   (2 recording + 4 alert, `llamacpp:*`) and `vllm-fleet` (1 recording + 4
   alert, `vllm:*` — archon queueing, prefix collapse, preemption, p95
@@ -97,30 +118,59 @@ verify it through the APIs.
 * **Delivery: working.** Alertmanager's single `signal` receiver webhooks the
   relay, and `journalctl --user -u signal-relay` records successful
   `sent to +12062476336:` lines for every rule in the set.
-* **Dashboards: working.** Provisioned datasources — Prometheus (uid
-  `prometheus`) and Loki (uid `loki`) — and the `llm-fleet` dashboard, all
-  installed from `shared/grafana/provisioning/`. The Loki datasource is how
-  you read logs: Loki has no web UI of its own, so Grafana Explore is the
-  human surface (the guarded `loki.franksdank.cloud` vhost serves only the API).
-* **Logs: fixed in code, awaiting deploy.** The pipeline shipped nothing because
-  a rootless Alloy cannot read the `0640` journal files, and the container-log
-  globs point at a root-storage layout that does not exist here. The fix is built
-  and tested in-repo (2026-10-20): Alloy is now a **root system service on
-  `Network=host`**, reading `system.journal` + `user-*.journal` and pushing to
-  Loki's `127.0.0.1:3100` loopback publish — a throwaway root probe read 31,825
-  lines and pushed 26,966 with zero drops. The **live** instance is still the old
-  rootless unit until `deploy.sh dankclaw --service alloy` runs. Detail in
-  [[Alloy]] → Why root; Loki's store state in [[Loki]] → Contents.
-* **Not configured:** no blackbox exporter (the structural downs have no real
-  uptime signal) and no Alertmanager vhost. The log pipeline's fix is in code but
-  not yet deployed (see *Logs* above).
+* **Dashboards: working (after the 2026-10-21 copier fix).** Provisioned
+  datasources — Prometheus (uid `prometheus`) and Loki (uid `loki`) — and the
+  `llm-fleet`, `host-system`, `containers`, `observability` and
+  `logs-overview` boards. Until 2026-10-21 the copier carried only
+  `llm-fleet.json` by name and the `containers` board had never been
+  provisioned; the deployer now globs the directory. The Loki datasource is
+  how you read logs: Loki has no web UI of its own, so Grafana Explore is the
+  human surface (the guarded `loki.franksdank.cloud` vhost serves only the
+  API). Verify method and the copier law: [[Observability Provisioning]].
+* **Per-container metrics: the `containers` dashboard (added 2026-10-20; live
+  after `deploy.sh <host> --service cadvisor`).** cadvisor is the only way to see
+  per-container CPU/memory here, and on this rootless-Podman host it works on one
+  condition: `--cgroupns=host`. Podman starts every container in a PRIVATE cgroup
+  namespace, so without that flag cadvisor sees only its own cgroup root and
+  exports a single series (measured 2026-10-20: 1 series off, 996 on). cadvisor
+  names nothing itself — the libpod socket resolved only 2 of 34 rootless
+  containers — but the quadlet unit name is carried in the cgroup path
+  (`app.slice/<name>.service/…`), so `prometheus.yml` relabels that path into a
+  `container` label. Network and filesystem are NOT exported for these containers
+  (no netns/classid attribution under the host-PID layout), so the dashboard
+  graphs only CPU, memory and OOM and does not fake the rest.
+* **Per-GPU metrics: the `host-system` GPU panels (added 2026-10-20; live after
+  `deploy.sh <host> --service gpu-exporter`).** `scripts/gpu-exporter.py` is a
+  host-process user unit on BOTH hosts — not a quadlet, because it reads the host's
+  own DRM sysfs and shells to `nvidia-smi`, neither of which a container can do
+  without GPU passthrough. It reads each AMD card's `gpu_busy_percent` /
+  `mem_info_vram_*` / `hwmon temp1_input` and one `nvidia-smi --query-gpu` CSV, and
+  emits unified `gpu_*` families (utilization %, VRAM used/total bytes, temperature
+  °C, power W) labelled `gpu` / `vendor` / `model`, with `host` added by the scrape
+  job. Two laws: **absent means absent, never a zero** — the R9700s expose no sysfs
+  `power1_average`, so they carry no `gpu_power_watts` line at all (only NVIDIA and
+  the Dankclaw 8060S iGPU report power); and it binds loopback + the host LAN, like
+  node-exporter, never the wildcard. The dashboard graphs util/VRAM/VRAM-%/temp/power.
+* **Logs: flowing; container labels land with the 2026-10-21 deploy.** The
+  pipeline once shipped nothing (rootless Alloy cannot read the `0640` journal;
+  the container-log globs pointed at a root-storage layout that does not exist
+  here) — fixed 2026-10-20 by making Alloy a **root system service on
+  `Network=host`**. Journal lines now reach Loki (measured ~6.5k lines/5min),
+  but the stream carried only `job`/`service_name`, so every per-container
+  view was structurally empty: Loki metrics queries group only by INDEXED
+  stream labels. Alloy now promotes podman's `CONTAINER_NAME` to a `container`
+  label (plus a `host` label) at ingestion — `loki.process` in
+  `shared/alloy/config.alloy`; detail in [[Alloy]] → Why root, and the
+  purpose-test law in [[Observability Provisioning]].
 
 ## Citations
 
-- `dankclaw/observability/quadlet/{prometheus,alertmanager,grafana,loki,alloy,node-exporter}.container`
+- `dankclaw/observability/quadlet/{prometheus,alertmanager,grafana,loki,alloy,node-exporter,cadvisor,blackbox-exporter}.container`
 - `scripts/deploy-remote.sh:219-300` (copiers + relay), `:521-536` (health waits), `:477-486` (grafana env seed)
 - `scripts/signal-relay.py:26-49`, `scripts/signal-relay.service`
+- `scripts/gpu-exporter.py`, `scripts/gpu-exporter.service` (per-GPU host exporter)
 - `shared/prometheus/prometheus.yml`, `shared/prometheus/alertmanager.yml`, `shared/prometheus/rules/llm-fleet.yml`
 - `shared/grafana/provisioning/`, `shared/loki/loki-config.yaml`, `shared/alloy/config.alloy`
+- `shared/blackbox/blackbox.yml` (prober modules; the listen bind is on the unit's `Exec`)
 - `tests/observability_quadlet.bats` (boot persistence, no wildcard publish, probe addresses, delivery contract)
 - Host evidence 2026-10-18: `ss -ltnp`, `podman ps`, `systemctl --user is-enabled|is-active`, `curl` on `127.0.0.1:9090/api/v1/{targets,rules,alerts}`, `127.0.0.1:3100/loki/api/v1/labels`, `127.0.0.1:12345/metrics`, `127.0.0.1:3002/api/health`, `192.168.0.143:18093/health`, `journalctl --user -u signal-relay`

serverAppConfigs-docs/Infrastructure/Prometheus.md

@@ -42,29 +42,29 @@ Authentik; post-auth the Caddy vhost routes straight to `prometheus:9090`) —
 
 The `frank-medium` job is **retired**: no `frank-medium` job exists in the config or in `/api/v1/targets` (2026-10-18), and the endpoint itself left the fleet — its history is on [[frank-archon]], which holds the port today.
 
-**Three jobs read down today, and they are not outages** (re-audited against `curl -s http://127.0.0.1:9090/api/v1/targets`, 2026-10-18; first audited 2026-09-16):
+**No job reads down structurally any more** (the three permanent-reds were
+retired 2026-10-21; census re-run after each deploy):
 
-| Job | Target | `health` | `lastError`, verbatim |
-|-----|--------|----------|------------------------|
-| `gitea` | `http://192.168.0.143:3000/metrics` | down | `server returned HTTP status 404 Not Found` |
-| `comfyui` | `http://comfyui:8188/` | down | `received unsupported Content-Type "text/html" and no fallback_scrape_protocol specified for target` |
-| `services` | `http://192.168.0.143:8081/` | down | `received unsupported Content-Type "text/html; charset=utf-8" and no fallback_scrape_protocol specified for target` |
-| `services` | `http://192.168.0.143:3005/` | down | same `unsupported Content-Type "text/html; charset=utf-8"` error |
+| Job | Target | how |
+|---|---|---|
+| `gitea` | `192.168.0.143:3000/metrics` | metrics enabled in the quadlet (`GITEA__metrics__ENABLED=true`, `ENABLED_HOST=0.0.0.0`) — the page once said "stays off until `[metrics] ENABLED = true` in its own config"; the env form is what the deployer ships |
+| `services` | `http://192.168.0.143:8081` (SearXNG), `:3005` (OpenWebUI), `http://127.0.0.1:8188` (ComfyUI) | **blackbox exporter** probe, module `http_2xx` — the comfyui metrics job is retired, one owner per service. The prober runs in the host netns, so comfyui is probed over loopback (its published port); bridge container names do not resolve there |
 
-Gitea's exporter stays off until `[metrics] ENABLED = true` in its own config (no metrics env is set in the container, and `curl -s http://127.0.0.1:3000/metrics` → 404, 2026-10-18); ComfyUI and the two `services` probes are HTML pages, which Prometheus 3 refuses as metrics. Everything else is green: `prometheus`, `node-dankclaw`, `node-dankserver`, `grafana`, `loki`, `frank-zealot`, `frank-dragoon`, `frank-archon`.
+Scraping a web root is not a health check: an HTML page fails the scrape
+protocol (`unsupported Content-Type`), which is what kept those targets red
+beside healthy services. The classification rule and the honesty law live in
+[[Stale Signal Audit]]. The prober is `blackbox-exporter.container` (host
+netns, config `shared/blackbox/blackbox.yml`); it binds 9115 on loopback and
+the LAN, and Prometheus scrapes it at `host.containers.internal` — traffic to
+that gateway lands on the host's loopback listener. The gateway address is
+not itself bindable from the host netns.
+Plain targets stay `host:port`, never URLs — the probe job's URL targets are
+the documented exception, relabeled onto the prober.
 
 **`frank-zealot` reads up since 2026-10-18, and did not before.** llama.cpp only answers `/metrics` when started with `--metrics`, so `up{job="frank-zealot"}` was 0 across the whole retained history (the oldest sample, 2026-10-11, is already 0) while the service answered `/health` 200 the whole time; the flag and its opt-out are on [[frank-zealot]]. Read `lastError` before believing an outage: a target can read down beside a healthy service.
 
 Alert state is a live property, not a page fact: `curl -s http://127.0.0.1:9090/api/v1/alerts` returned no alerts at the 2026-10-18 morning census, one firing `LlmPrefixCacheCollapse` (info, `frank-zealot`, `activeAt` 11:10:48Z) minutes later, and none again by 11:56Z.
 
-The remedy for a probe-shaped question is the **blackbox exporter**; the
-classification rule and the honesty law for it live in [[Stale Signal Audit]].
-Until the exporter is deployed, these three stay red for a structural reason:
-deleting them would
-hide a real service, and fixing them by publishing databases or enabling metrics
-would trade a wrong signal for a real exposure. Targets are `host:port`, never
-URLs.
-
 ## Volumes
 
 | Container path | Host path |

serverAppConfigs-docs/LLM/LLM Performance Diagnosis.md

@@ -239,45 +239,92 @@ prefill is over. `preemptions` separates the third shape: a preempted-and-recomp
 sequence stops producing entirely and then re-prefills from token 0, which is a
 stall, not sludge.
 
-Knobs, in the order worth trying them (radiance GUIDE §4.5):
+**Split the step ledger before choosing a knob.** Three counters decide which mechanism you are
+looking at, and none is a rate: `engine_step_seconds_decode_total / engine_steps_decode_total` is
+the mean **decode-only** step, and `(step_seconds_total − step_seconds_decode_total) /
+(steps_total − steps_decode_total)` is the mean **chunk step** (a chunk alone, or a chunk with
+decode rows beside it). On frank-archon's live 1.8.1 shape that pair is **15.5 ms against 166 ms**,
+with 4.7 % of the steps holding 34 % of the step time. Decode-only steps themselves move only
+14.1 → 16.4 ms between a prefill-free minute and a heavy one, and the lagged buckets stay flat: so
+a decode-rate collapse on that box is **step time sharing**, and the eviction/residency story is
+worth ~10 % of one step — not a lever.
 
-- `--prefill-interleave N` — after a chunk step taken beside decoders, the next N
-  steps carry decode rows only and admit nothing. Decoders keep their cadence; the
-  prompt pays that in TTFT. Counts as `engine_steps_held_total`.
-- `--busy-prefill-chunk MODE` — the chunk's share of a step that has other work:
-  `auto` (default) reserves a row plus the draft window per decode row behind it and
-  an eighth of the budget per waiting prefill/request, `off` lets the chunk take the
-  whole step (admission never runs while it does), `N` pins the share. Never under
-  the prefill stager's smallest pass — the startup line names it, and below it every
-  routed layer reads its pooled experts synchronously, which costs the prefill far
-  more than sharing saves. Counts as `engine_prefill_chunks_capped_total`.
-- `--max-num-batched-tokens` — the blunt rung, and the only one older pins have. The vendor's own
-  curve prices **halving the shared chunk at roughly a fifth of prefill throughput** (a 48-layer MoE
-  at 200k on two cards — knob pricing, not this endpoint's number, so treat it as the order of
-  magnitude a chunk decision costs). Raising it past the checkpoint interval buys nothing and loses
-  expert residency to the arena.
-- `--prefill-span N` — the inverse trade: a *lone* prefill carries N intervals a step while every
-  shared step is planned at a smaller shared budget, which on an expert-tiered box lends that arena
-  back to the expert slab. The vendor's own measurement says the win is **long-prompt-only**: nothing
-  at short prompts, a large TTFT gain at the far end of the window.
+**Four traps in the PromQL above.** Learn them before trusting a bucket:
+- **Latency is `vllm:`, cadence is `radiance:`.** The TTFT / inter-token / per-output-token / e2e
+  histograms and the running/waiting gauges exist only under the vLLM compatibility prefix; preemption
+  is `vllm:num_preemptions_total` (the counter) while `radiance:num_requests_preempted` is the
+  *instant* depth. A `radiance:`-only query finds no latency and no preemption, and reads as "the
+  endpoint never stalls anyone".
+- **The tps gauges are 1 s decayed windows, batch-pooled.** Per-stream rate is
+  `decode_tokens_per_second / vllm:num_requests_running`. With a 15 s scrape, a 120 s step samples one
+  scrape in eight — use step ≤ 30 s for a bucket table.
+- **Every counter resets at a restart and the metric set changes with the build.** Prove the window is
+  one shape (`up`, or the first sample of a counter that only the current build exports) before
+  ranking buckets; frank-archon's 48 h window straddled two containers, and one bucket's "collapse"
+  turned out to belong entirely to the older one.
+- **A 2-minute mean step time dilutes the chunk.** The sharp variable is the phase ratio in the query
+  above, never `step_seconds/steps` — 4.7 % of steps cannot move a mean that is 95 % decode.
 
-Two traps. **The arena/slab coupling makes a shared step doubly expensive on an
-expert-tiered shape**: the full-plan arena takes VRAM the resident expert slab would have had, so the
-decode rows inside that step also read experts over the link — a smaller shared budget helps twice.
-And **re-check
-`prefix_hit_rate` / `prefix_evictions` after any chunk change**: the arena and the KV
-pool are the same memory, and the failure looks like turn two taking as long as turn
-one.
+Three instruments the PromQL above cannot reach:
 
-Availability is a version question, not a config question: `--busy-prefill-chunk` and
-`--prefill-interleave` are absent from older pins — prove it on the running container
-(`podman exec <c> <binary> --help | grep -iE "prefill|chunk"`), never from the vendored
-tree, and check the source clone's depth before dating a flag from `git log -S`
-(a `--depth 1` clone reports its own tip for every search).
+- **The engine's own step-detail histograms (`prefill_steps_duration`, `prefill_step_tokens`,
+  `token_queue_wait`) are exported but reach no target**, and the arena / expert slab / heat engine /
+  prefill stager have **no metric at all**. Per-request forensics mean `curl localhost:7770/stats` on
+  the endpoint's own host — same phase counters plus `preemptions` (the count), the chunk geometry,
+  and the expert/link byte totals. `/diagnostics` answers "is the *box* the limit" in one call:
+  configuration drift, the VRAM split, each card's link chain and clocks, findings with fixes.
+- **`prefix_hit_cut_tokens_total` measures prefill load you created for yourself.** On a hybrid
+  (linear + paged) model a prefix hit reuses only as far as the last **linear-state** checkpoint, so
+  cached tokens are replayed as prefill and stall decoders for it; frank-archon ran 38 % of all its
+  prefill tokens that way, every `prefix_hit_cuts_total` labelled `reason="no_snapshot"`. A high
+  prefill rate is not always a client that sends new tokens. Retention is `--checkpoint-policy` -
+  exactly three names, `geometric-backoff` (default), `tip-only`, `keep-all`, one owner:
+  [[frank-archon]].
+- **Bucket the arriving request too** — `histogram_quantile` over
+  `vllm:time_to_first_token_seconds_bucket` by the same prefill series. The decode gauge only sees
+  sequences that are already running; the one waiting for its first token pays the same steps and
+  shows in no decode number.
 
-Run on: [[frank-archon]] — the census was taken on its 3-slot radiance 1.0.13 era (budget 2048 =
-interval 2048); the live 1.8.1 shape carries the two chunk knobs. The measured census and the verdict
-for this fleet's shape are that page's.
+Knobs, in the order worth trying them (radiance GUIDE §4.5; the pinned source is the referee):
+
+- `--prefill-interleave N` — after a chunk step taken beside decoders, the next N steps carry decode
+  rows only and admit nothing. **The knob that acts on the measured mechanism.** Decoders keep their
+  cadence; the prefilling prompt pays in TTFT and an arriving request waits out the held steps.
+  Counts as `engine_steps_held_total` — if that counter is zero, nothing is protecting anyone.
+- `--busy-prefill-chunk N` — pins a shared chunk's share of the step. **`auto` is a prefill-speed
+  setting, not decoder protection:** it reserves `1 + spec` rows per decoding sequence plus an eighth
+  of the budget per *waiting* request or prefill, and the source refuses to halve for "decode rows
+  alone", so a shape can run `auto` and still lose every chunk step. Prove engagement from
+  `engine_prefill_chunks_capped_total` against the chunk-step count. The floor is the prefill
+  stager's smallest pass — the largest arena level that lends, plus one, named on the startup line
+  `prefill staging: … a step of more than N tokens copies the pooled experts of …` — and under it
+  every routed layer reads its pooled experts synchronously: the vendor prices the whole job (four
+  decoders plus a long prompt) at **29.5 s with a 2048 chunk and 31.9 s at 512**, so shrinking the
+  chunk blindly buys latency in both directions.
+- `--prefill-span N` — needs `--max-num-batched-tokens` of N intervals. A *lone* prefill then
+  carries N intervals a step (on an expert-tiered box only from 8 intervals, where the vendor's own
+  measurement says a span starts to win), while **every other step is planned at the shared
+  budget** — a lending arena level that also keeps the stager's ring, the one combination the chunk
+  knob cannot express. Priced by the vendor at 320 MiB a card of expert residency, ~3 % off the KV
+  pool, and TTFT-neutral on one of his two 2×R9700 boxes: measure your own prompt lengths first.
+- `--max-num-batched-tokens` alone — the blunt rung, and the only one older pins have. It raises the
+  arena the expert slab loses, and on this payload its resonance is violent (MNBT 2560 collapses
+  decode — [[frank-archon]]).
+
+Two traps. **The arena/slab coupling makes a shared step expensive twice:** the full-plan arena is
+reclaimed from the expert slab *synchronously before the step*, and the loan regrows only after a
+quiet interval, so every step of a prefill run — the decode-only ones included — runs on the smaller
+slab. And **re-check `prefix_hit_rate` / `prefix_evictions` after any chunk or span change**: the
+arena, the staging ring and the KV pool are the same memory, and the failure looks like turn two
+taking exactly as long as turn one.
+
+Availability is a version question, not a config question: `--busy-prefill-chunk`,
+`--prefill-interleave` and `--prefill-span` are absent from older pins — prove it on the running
+container (`podman exec <c> <binary> --help | grep -iE "prefill|chunk"`) or from `/diagnostics`'s
+`configuration` block, never from a vendored tree, and check a source clone's depth before dating a
+flag from `git log -S` (a `--depth 1` clone reports its own tip for every search).
+
+Run on: [[frank-archon]] — owner of the live census, the measured step pair, and the arms.
 
 ## Related
 

serverAppConfigs-docs/LLM/RDNA4 MoE Expert Offload VRAM Budget.md

@@ -70,12 +70,12 @@ back toward 52. Check `VLLM_CUDAGRAPH_*` actually present in
 
 `--kv-cache-memory-bytes` does NOT appear in the arena arithmetic — the plan
 reserves the registry price, not the actual pool. Raising the pool consumes
-real free VRAM without growing the arena; the registry over-prices the pool
-(9190 B/token planned vs 7,891.8 measured at 200704 fp8 — 7,701 at 262144;
-**never reuse a B/token ratio across geometries**, and subtract ~3.87 MB
-block-alignment loss when converting a token target to bytes). The ask lands
-block-aligned: 3,166,500,000 B → 401,408 tokens; verify with
-`GPU KV cache size: N tokens, Maximum concurrency … X.Xx`.
+real free VRAM without growing the arena; the registry OVER-PRICES the pool
+(era G measured ~14 % under the planned B/token — the gap is the lesson, not
+the number; **never reuse a B/token ratio across geometries**), and there is a
+~4 MB block-alignment loss to subtract when converting a token target to
+bytes. The ask lands block-aligned: verify against the engine's own
+`GPU KV cache size: N tokens, Maximum concurrency … X.Xx` line at boot.
 
 **Driver-reported free VRAM after hours of traffic is NOT allocatable
 arena.** It is the estimator gap: the registry over-prices KV and charges
@@ -117,11 +117,13 @@ done, and read post-boot free VRAM per rank (`rocm-smi --showmeminfo vram`)
 **KV accounting — pin the pool BYTES, floor-sized:** set
 `--kv-cache-memory-bytes` PER RANK to the contract floor
 (`seqs × MAXLEN × measured B/token`), not the native window — extra pool bytes
-come out of the arena/headroom and buy nothing. Measured B/token on this
-hybrid: **7,701 fp8 / 13,839 bf16** (decomposed: attention 6,138/12,276 +
-dtype-independent mamba/side 1,563). Measured rates are shape-dependent —
-7,891.8 at the 200704×3 shape. Capacity does NOT scale with `SEQS`: the
-planner's KV reserve is for ONE max-model-len request.
+come out of the arena/headroom and buy nothing. The B/token rate is a property
+of the GEOMETRY: era G (3 × 200,704 fp8, vLLM fork) measured **7,891.8**;
+the radiance-era hybrid measured ~7,700 fp8 / ~13,800 bf16 at its own shape.
+Read your live rate from the boot ledger every time; the decomposition (a
+dtype-independent mamba/side stream rides on top of the attention KV) is why
+the ratio moves. Capacity does NOT scale with `SEQS`: the planner's KV reserve
+is for ONE max-model-len request.
 
 **Parallelism ceilings you cannot flag past:**
 
@@ -179,18 +181,6 @@ Cutover order: ladder → winner passes and serves → repo env + seam + harness
 (`git stash -u`) if the tree is dirty → deploy (one restart) → verify health
 + engine flags in the log.
 
-**Measured ladder on this box (200704×3, fp8 KV, precise memory):**
-
-| shape | util | graphs | arena | result |
-|---|---|---|---|---|
-| E | 0.98 | 0.55 | 17.22 GiB / 7,522 slots | boot ok |
-| F | 0.982 | 0.32 | 17.52 GiB / 7,653 slots | boot ok |
-| 1.0 | 1.00 | 0.07 | — | UNBOOTABLE (worker-init ValueError, both ranks) |
-| G (deployed) | 0.982 | 0.07 | 17.77 GiB / 7,762 slots | KV 401,408 tokens / 2.00x; 0.89 GiB real free/card held through a live 4,096-token generation at ~38–39 tok/s |
-
-bf16 ceilings at util 0.98: 3 seqs → 241,664 passes / 245,760 and 262,144 OOM
-at graph capture; 6 seqs fp8 → 10.70 GiB ask with 9.18 GiB free (fails) —
-~207k per context max.
 
 ## 7. Arena audit procedure (when asked for "more experts")
 

serverAppConfigs-docs/LLM/frank-archon.md

@@ -88,25 +88,30 @@ on stop. Boot budget 1800 s (the publisher's compose allows a 15-minute `start_p
 | Knob (env name, `FRANK_ARCHON_` dropped) | Live value |
 |---|---|
 | TP / wire | `TP=2`, `TP_WIRE=wht6` — libr4d one-shot all-reduce, not RCCL; `min-kb` stays the engine default |
-| Slots / window | `MAX_SEQS=2`, `MAX_MODEL_LEN=200000` (native 262,144; longer is refused, never truncated) — this pair is the **cache bound**, not a KV allocation |
+| Slots / window | `MAX_SEQS=4`, `MAX_MODEL_LEN=200000` (native 262,144; longer is refused, never truncated) — this pair is the **cache bound**: the resolver sizes the KV carve against `slots × window`, and no slot count funds itself (Laws) |
 | KV dtype | `KV_DTYPE=fp8` — the only dtype that funds the slot count (Laws) |
 | VRAM split | **derived at boot** from `GPU_HEADROOM_MIB=96` + the cache bound + `HOST_POOL_MIB`; naming `VRAM_WEIGHTS_MIB`/`VRAM_KV_MIB` is a boot refusal (removed at 1.3.0) |
-| Expert tier | `PLACEMENT=expert_tiered`, `HOST_POOL_MIB=44928` **per rank** |
+| Expert tier | `PLACEMENT=expert_tiered`, `HOST_POOL_MIB=49152` **per rank** — the pinned host pool is also the **KV lever**: the expert floor is `plane + 2 units − host room`, so every MiB pinned there lowers what the experts claim on the card one-for-one and the cache takes it (Laws) |
 | PLE | `NGRAM_PLACEMENT=ram` (47.7 GiB pinned; refuses boot on a short memlock) |
 | Spec / batch | `SPEC_TOKENS=2` — MTP depth 2, the `.rad` carries the model's own `mtp.*` head (`mtp.fc_hidden`/`fc_embedding`/`draft_head`); `MAX_BATCHED_TOKENS=2048` (MNBT — 2560 collapses decode, Laws) |
 | Prefix cache | `PREFIX_CACHE_MIB=8192`, pinned **host tier only** — the 64 GiB disk tier is gone, so an evicted prefix is re-prefilled; `STATE_DIR=/home/dank/radiance-state` now exists only to hold the rad-tune cache |
 | Template | `TEMPLATE=scripts/qwen38-froggeric-v22.5.jinja` (28,234 B) via `--override-chat-template` — froggeric's fixed Qwen3.8 template (the repo root's `chat_template.jinja`, vendored); the checkpoint's own template is baked into the `.rad` by rad-convert and this overrides it, and radiance derives the reasoning and tool-call reply format from it |
-| Step sharing | `BUSY_PREFILL_CHUNK=auto` (a shared chunk reserves a row plus the draft window per waiting decoder) · `PREFILL_INTERLEAVE=0` — held at zero until the decode-vs-prefill buckets ask for it |
-| TTM gate | `TTM_MIN_MIB=150000`, `TTM_TARGET_MIB=204800` |
+| Step sharing | `BUSY_PREFILL_CHUNK=auto` — prefill-speed first: it reserves 3 tokens per waiting decoder and ⅛ of the budget per waiting request, and does **not** shrink the chunk for decoders alone · `PREFILL_INTERLEAVE=1` — the decoder dial, **armed 2026-10-20**: one decode-only step after each chunk step taken beside decoders; meter `engine_steps_held_total`, price is the prefilling prompt's TTFT |
+| TTM gate | `TTM_MIN_MIB=162000`, `TTM_TARGET_MIB=204800` |
 | Boot | `RAD_MIN_MIB=110000` deployer floor; the listener binds only after the full load (health ~100 s, budget 1800 s) |
 
-Boot ledger for the live shape (2026-10-19 11:10, pin 1.8.1, `MAX_SEQS=2`, `HOST_POOL_MIB=44928`):
-the resolver prints `elastic 26.73 GiB` → **experts 23.73 GiB (89 %) / kv cache 2.76 GiB (10 %)**,
-`vram_weights 27.31 GiB`, `host_pool 44.47 GiB`. `kv_attn` (13 layers, 4-token blocks) pages
-**26.00 KiB** → 91,301 blocks = **365,204 tokens = 1.8 full-length sessions = 91 % of the
-2 × 200,000 worst case** (the resolver never funds 100 %, see Laws); `mover` lends 2,146 slots
-(2.54 GiB) out of the cache to the expert slab; sampler **9 rows**, template 28,234 B, vision live
-(≤ 4,194,304 px, 16,384 patches/pass).
+Boot ledger for the live shape (2026-10-20, pin 1.8.1, `MAX_SEQS=4`, `HOST_POOL_MIB=49152`):
+the resolver prints `elastic 26.73 GiB` → **experts 20.72 GiB (78 %) / kv cache 5.64 GiB (21 %)**,
+`prefill staging 373.65 MiB` out of the cache, `vram_weights 24.30 GiB`, `host_pool 48.59 GiB`.
+The referee line: `kv: the cache can grow to 5.64 GiB a card — 747,492 tokens of context, 3.7
+sessions at 200,000 — with every block it lends the experts taken back; 440.04 MiB of that is
+per-sequence state, and up to 5.20 GiB is lent to the experts while no request holds it`.
+`--checkpoint-slots 40` is auto (`max_num_seqs × 10` slots of 111.98 MiB = 4,479 MiB of pinned
+host), `/health` reports `queue_cap 32` (`max(8, seqs × 8)`), sampler **12 rows**
+(= `seqs × (1 + speculative depth)`), 4 hardware queues a card, template 28,234 B, vision live
+(≤ 4,194,304 px, 16,384 patches/pass), P2P confirmed (`the 2 cards reach each other's memory
+directly`). Experts living in the pinned host pool: **8.99 GiB a rank** (was 5.99) — the price of
+the bigger cache, paid over PCIe when those experts are hit.
 
 The boot **ledger**, not this arithmetic, is the referee: radiance states every budget before
 anything is allocated (weights vs KV, the host pool's coverage of the worst case, the PLE pin,
@@ -145,6 +150,7 @@ marked "re-fetch" is gone from `/home/dank/models/`, which today holds only `Sti
 
 | Window | Payload | Runtime | Shape | Replaced because | Rollback |
 |---|---|---|---|---|---|
+| 2026-10-19 → 10-20 | same `qwen3.8-next-flash-fp8-iq4r-moe.rad` (payload unchanged) | radiance **1.8.1**, `sha256:d590e300…` (same digest) | 2 × 200,000 fp8 KV — resolver cache 2.76 GiB = 365,204 tokens (91 % backing); host pool 44,928 MiB a rank, `TTM_MIN_MIB=150000`, `queue_cap 16`, sampler 9 rows; `--prefill-interleave 1` armed on 10-20 | Frank ordered **four** concurrent conversations over a ≥ 700,000-token pool; the resolver funds that only by lowering the expert floor with more pinned host room (Laws) | revert `1bb6399` and redeploy `scripts/deploy.sh dankserver --service frank-archon` — trio back to `MAX_SEQS=2` / `HOST_POOL_MIB=44928` / `TTM_MIN_MIB=150000` |
 | 2026-10-15 → 10-19 | same `qwen3.8-next-flash-fp8-iq4r-moe.rad` (payload unchanged) | radiance **1.0.13**, `sha256:4c3f4845…` | 3 × 200,000 fp8 KV bought with the **explicit** pair `--vram-weights-mib 25600 --vram-kv-mib 5120`; host pool 44,928 → 45,568 MiB a rank; prefix cache 8 GiB host + 64 GiB disk | decode turned to sludge beside any long prompt — step contention, 18.7 ms decode-only steps against 317 ms steps carrying a prefill chunk (Laws) — and the only two knobs that fix it, `--busy-prefill-chunk` and `--prefill-interleave`, exist only after this pin | `FRANK_ARCHON_IMAGE` back to `sha256:4c3f4845…` **plus a trio rewrite**: restore the pair and the host+disk prefix knobs, delete `BUSY_PREFILL_CHUNK`/`PREFILL_INTERLEAVE`. A digest swap alone is a boot refusal (Runtime) |
 | 2026-10-08 → 10-15 | in-house `qwen4exp-swift15-w4dn8-i8-hc8m.rad`, 116,416 MiB (Swift1.5 abliterated BF16, GPTQ w4 experts + 24 bf16 protected, KLD 0.0131 / top-1 97.92 %) | radiance 1.0.13, same digest | 3 × 200,000 fp8 KV, slab 25,600 / pool 5,120 — the shape the live file still carries | the engine author's own container is the supported path and his quality ceiling; the finetune's refusal-bending left the endpoint | `.rad` purged 2026-10-17 (trash-staged then emptied, Frank's approval). Re-download `frank8465/Swift-1.5-Qwen3.8-Flash-Next-GPTQ-W4A8-Radiance-AMD-R9700` → `qwen4exp-w4nl64-i8-hc8m.rad`, 122,071,973,888 B, LFS sha256 `09dd9306…`, via the iq4r fetch script's hfget/verify pattern retargeted at that repo; point `MODEL_FILE` at it, redeploy. Re-**convert** is dead: the BF16 source was `rm`'d 2026-10-15 (`kld-ref`, grams, corpora and `/home/dank/logs/swift15_*.log` survive) |
 | 2026-10-05 → 10-08 | `qwen4exp-fp8-native.rad`, 177,109 MiB (173.0 GiB) — native modelopt block-FP8, no requantisation | radiance 1.0.7 line, same digest | — | native FP8 experts (56.4 GiB/rank) ate a 40 GiB pool and left KV at 3,543 MiB = 386,904 tokens, **1.93 sessions** — context economics, not quality; and the host pool could not be trimmed to pay for it (36,864 refused the boot) | rebuild-only: `scripts/download-frank-archon.sh` (official `Qwen/Qwen3.8-Flash-Next-FP8`, 131 shards, 172.8 GB, ungated) + `scripts/build-frank-archon-rad.sh` (no-recipe convert under the descriptor law); raw **and** `.rad` were `rm`'d 2026-10-15 (Frank's call, to free the volume for the iq4r download) |
@@ -211,27 +217,44 @@ vllm-radiance era (this table opens on 09-26; the era page claims 10-10).
   never `--kv-cache-size-mib`; with the disk tier engaged the engine also forced ~9.6 GiB of weight
   buffers + arena onto the weights side (~29.6 GiB/card effective). **Neither flag exists at 1.8.1:**
   the same ceiling is enforced by the resolver, and passing the pair is a boot refusal.
-- **At 1.8.1 the split belongs to the resolver, and it funds ~90 % of the worst case, never 100 %.**
-  The engine takes `--gpu-headroom-mib`, the cache bound (`--max-model-len` × `--max-num-seqs`) and
-  `--host-pool-mib`, prints `claimable → elastic → experts / kv cache` before allocating anything, and
-  sizes the expert plane from the model's own weight footprint plus slack — not from a number typed by
-  hand. Three boots of evidence:
+- **At 1.8.1 the split belongs to the resolver, and the source fixes its order.**
+  `core/mem/vram_budget.cpp` claims the card bottom-up (`capacity → already held → drift → arena →
+  staging → queues → prefetch → headroom = claimable`), takes the **static weights off the top as a
+  floor, not a share**, and divides the `elastic` remainder in three lines (146-156):
 
-  | shape | demand | cache given | backing |
+  ```cpp
+  exp = std::min(b.expert_floor, b.elastic);
+  kv  = std::min(kv_cap, std::max(b.elastic - exp, kv_one));
+  exp = std::min(std::max(b.expert_ceiling, b.expert_floor), b.elastic - kv);
+  ```
+
+  `kv_cap` is `KVManager::ceiling()` = `state + Σ_paged max_seqs × ceil(max_ctx / block) × bytes`
+  (`core/mem/kv.cpp:266`), so `--max-num-seqs` and `--max-model-len` set the cache's **demand**, never
+  its funding. The funding is `elastic − expert_floor`, and the floor is
+  `min(plane + spare, plane + spare + 2·unit_max − host_room)` (`core/place/planner.cpp:100-104`, with
+  `host_room = host_pool − file_stage_reserve` at `core/engine_bringup.cpp:1418-1422`). **That is the
+  cache's lever: every MiB added to `--host-pool-mib` drops the expert floor one-for-one and the cache
+  takes it** — minus `stage`, the prefill stager's ring (two layers of the experts left off the card),
+  which is charged back out of the cache whenever the cache lends (`vram_budget.cpp:193-203`). Boots of
+  evidence, all on 1.8.1:
+
+  | shape | demand | cache given | engine's `backs %` |
   |---|---|---|---|
-  | 3 × 200,000, host pool 44,928 | 600,000 tok | 4.20 GiB | 93 % (2.8 sessions) |
+  | 3 × 200,000, host pool 44,928 | 600,000 tok | 4.20 GiB | 93 % |
   | 3 × 200,000, host pool 45,568 | 600,000 tok | **4.20 GiB — byte-identical** | 93 % |
-  | 2 × 200,000, host pool 44,928 (live) | 400,000 tok | 2.76 GiB = 365,204 tok | 91 % (1.8 sessions) |
+  | 2 × 200,000, host pool 44,928 | 400,000 tok | 2.76 GiB | 91 % |
+  | 4 × 200,000, host pool 49,152 (live) | 800,000 tok | **5.64 GiB = 747,492 tok** | 93 % |
 
-  So `--max-num-seqs` is a **demand signal** for the resolver and an admission cap for the scheduler —
-  never a KV allocation — and no slot count fully funds the worst case. Growing host RAM by less than a
-  slab step (640 MiB) moved nothing: the expert plane's claim decides what is left for KV, so cache is
-  grown through slots × window or the dtype, and experts are grown by not leaving elastic pool idle.
-  `--host-pool-mib` is still load-bearing the other way — it is where the expert rows the cards cannot
-  hold get pinned, and under-sizing it has always been a boot failure (36,864 refused a boot in the fp8
-  era). The two-slot boot's own behavioural check: three concurrent conversations all answer `200` —
-  the third **queues** (`queue_cap: 16`), it is not refused; the 400 class is a single request over
-  `--max-model-len`, or three concurrent full-height conversations against a 1.8-session cache.
+  The 45,568 row is the ceiling case, not a mystery: at 3 slots the cache already held its whole
+  `ceiling`, so extra host room could not grow it and shows up as `unclaimed`. **The referee is the
+  ledger line `kv: the pool backs N % … the cache can grow to … N tokens`, never arithmetic.** To buy
+  window: raise `--host-pool-mib` (cost: pinned host RAM, and experts evicted to host cross PCIe —
+  `experts-from-host` was 5.99 GiB a rank at 44,928 and is 8.99 at 49,152) or cut the per-token cost
+  (`fp8` KV). Raising slots raises demand **and** costs 10 × 111.98 MiB of pinned checkpoint-store RAM
+  per slot. Under-sizing the pool the other way has always been a boot failure (36,864 refused a boot in
+  the fp8 era: the host cannot hold what the cards will not). Behavioural check on the live shape:
+  `/health` reports `queue_cap 32`; a request beyond the slot count queues rather than being refused,
+  and the 400 class is a single request over `--max-model-len`.
 - **Idle KV VRAM is the expert cache's spillover — the price is paid twice.** `mover: 2275
   slots can be lent out of the KV cache (2.69 GiB)`, `arena: lends 740 MiB while steps are <=
   256 tokens`; the lend grows with the pool (2.69 GiB from 3.46, 6.08 from 6.912, 4.22 in the pair era)
@@ -280,41 +303,104 @@ vllm-radiance era (this table opens on 09-26; the era page claims 10-10).
 - **MNBT 2048.** 2560 collapses decode to 2.6-7.7 t/s (MTP resonance); the w4 file tier streams
   in place only at MNBT ≤ 2048 and the activation arena scales with the chunk (32768 → 33.3
   GiB, refused).
-- **Decode pays for the chunk it shares a step with.** Census of the live engine taken at the
-  3-slot shape on 1.0.13 (`/stats` + `/metrics`, cumulative over the 2026-10-15 boot):
-  1,435,585 decode-only steps at **18.7 ms** each against 127,991 prefill+mixed steps
-  at **317 ms** — 60 % of engine wall-clock spent inside 8 % of the steps, because the
-  chunk is `min(remaining, checkpoint interval 2048, MNBT 2048)` = 2048 rows and every
-  decode row in that step waits the whole chunk. Day of Prometheus (120 s step, 721
-  samples): `decode_tokens_per_second` median **13.3 t/s** in the 56 % of samples where
-  `prefill_tokens_per_second` ≥ 1000, against 106-240 t/s in the light-prefill buckets;
-  bucketed by mean step time it is 128.7 t/s at 0.02-0.05 s steps and 12.0 t/s at
-  0.15-0.4 s. The stall is **during**, not after — decode in windows 2-14 min after a
-  busy-prefill window sits at the idle baseline (58-64 vs 77-85 t/s, the residual
-  confounded by co-scheduling), so no lingering expert-residency penalty at 2-minute
-  resolution, and `preemptions: 0` rules out preempt-and-re-prefill. Method:
-  [[LLM Performance Diagnosis]] §9. The law is shape-independent; the fix lives at 1.8.1 (next law),
-  and nobody has re-run the census since the migration — so the fix is ordered, **not verified**.
-- **The knobs for it arrived with 1.8.1, and only one of them is on.** `--busy-prefill-chunk` caps the
-  shared chunk so a step with waiting decoders takes a smaller piece (live: `auto`, the engine picks;
-  the explicit value is 512), and `--prefill-interleave N` inserts N decode-only steps after a chunk
-  step taken beside decoders. `interleave` is deliberately **0** — the env comment holds it there until
-  the decode-vs-prefill buckets ask for it, and those buckets have not been read since the migration.
-  The vendor prices a 1024 chunk at about a fifth off prefill throughput to halve the stall; that is a
-  knob for choosing tomorrow's chunk, not a measurement of this shape. MNBT remains an arm and not an
-  edit: 2560 collapses decode through MTP resonance. Any chunk change re-scores draft acceptance before
-  it is trusted.
-- **Slots cost KV, rows, and window — never speed.** `--max-num-seqs` carries no radiance constraint: it
-  adds ≈110 MiB of recurrent state per slot (the live `kv_full_backing` arithmetic prices them exactly),
-  declares the fused TP all-reduce over `seqs × (1 + spec)` rows, and `wht6` is bit-exact up to 25 rows
-  (single-sequence decode and any step of ≤ 25 rows is exact; only busier steps are lossy) — 2 × 3 = 6
-  today. Measured 2026-10-06: seqs 2 and seqs 3 are statistically tied (30.9-33.1 t/s, inside the ±3 %
-  floor). Measured 2026-10-19 at 1.8.1: the third slot's real price is **KV, not rows** — dropping it
-  moved the resolver's cache from 4.20 GiB to 2.76 GiB and the backing from 2.8 to 1.8 sessions, while
-  the ≈4 GiB of host RAM it freed came back unbudgeted, landing in elastic pool that turned into a bigger
-  mover lend instead of cache (resolver law above). The old "seqs 2 cannot boot" law belonged to the
-  retired tcclaviger fork's `torch.compile` shape specialization, and radiance booted seqs 2 on its first
-  fp8-native start.
+- **Decode pays for the chunk it shares a step with, and `--busy-prefill-chunk auto` does not
+  protect it.** Re-measured on the live 1.8.1 shape (34.8 h boot, Prometheus at 60 s,
+  `engine_step_seconds{,_decode}_total` / `engine_steps{,_decode,_mixed,_prefill}_total` /
+  `prefill_tokens_total`): decode-only steps run at **15.5 ms**, chunk steps (a chunk alone, or a
+  chunk with decode rows beside it) at **166 ms** — 4.7 % of the steps hold **34 % of the engine's
+  step time**, so a session decoding while another prompts advances one token per 166 ms. Bucket the
+  harness's `radiance:decode_tokens_per_second` by `radiance:prefill_tokens_per_second` (15 s
+  scrapes; per-stream rate = the gauge ÷ `vllm:num_requests_running`): median **166 t/s** under light
+  prefill, 81 t/s at 1-5k, **13 t/s above 5k** — a **12.7×** collapse, and at 13 t/s with two
+  sessions running each stream sits at 6.7 t/s = 149 ms a token, one token per shared step. Nothing
+  else moves: decode-*only* steps stay 14-18 ms in every bucket (14.1 ms in a prefill-free minute,
+  17.9 ms in the deep one) and the relationship is lag-0 only (r = −0.74 at lag 0, |r| ≤ 0.19 from
+  30 s on, decode back at baseline by 5 min) — residency is not the mechanism either. The second
+  victim is the **arriving** request: through a high-prefill window TTFT p90 goes 2.4 → **18.0 s**
+  and 15.5 % of requests wait over 10 s for their first token, with `http_queue_depth` at 2-3, while
+  median TTFT (~1.9 s) and per-token latency barely move. Preemption is real but burst-shaped: 264
+  `vllm:num_preemptions_total` in the boot, 108 of them inside 105 s of a 6.5k-tok/s prefill. The
+  chunk is `min(remaining, checkpoint interval, MNBT, busy_cap)`; the first three read 2048 here,
+  and `busy_cap` is the fourth law.
+- **`auto` is a prefill-speed dial, not a decoder-protection dial.** `Scheduler::busy_cap` reserves
+  `1 + spec` rows per decoding sequence behind the chunk (3 tokens at MTP 2) plus one eighth of the
+  budget per *waiting request or waiting prefill*, and the code refuses to do more for decoders:
+  "not for decode rows alone: their reserve is a few tokens, so the chunk ends a quantum short and
+  the next step finishes the interval… Halving would hold a long prompt to half-interval chunks while
+  anything decodes behind it, which costs more" (`core/sched/scheduler.cpp:874-878`, 1.8.1). Live
+  proof: `prefill_chunks_capped` 5,018 against 62,386 chunk steps — 8 %, and those are the minutes
+  with a waiter, not the minutes with a decoder. An explicit `N` is the only value that applies
+  "whenever the chunk shares the step".
+- **At MNBT 2048 no chunk value keeps both the arena loan and the staged copies.** The arena is a
+  ladder of levels (256 and 1024 rows here) and a step's rows are its **whole** token count —
+  `max(n_tok, n_seq × (1 + spec))`, `core/engine.cpp:1142` — so a 1024-token chunk with two decode
+  rows is a 1030-row step: it falls back to the full plan, which "lends nothing", and it arms the
+  prefill stager. The stager's floor is the largest lending level plus one, 1025 rows here — named on
+  the boot line `prefill staging: … a step of more than 1024 tokens copies the pooled experts of 48
+  routed layers onto the card up to 3 layers ahead` — and below it "every routed layer reads its
+  pooled experts synchronously and the prefill slows far more than sharing saves" (GUIDE §4.5). The
+  vendor's own number for naive shrinking, on this architecture: four decoders with a long prompt
+  beside them finished in **29.5 s at a 2048 chunk and 31.9 s at 512**.
+- **A big step takes the arena from the expert slab and keeps it for the whole prefill.** The arena
+  lends 987.77 MiB (steps ≤ 256 rows) or 670.26 MiB (≤ 1024) to the slab; a full-plan step reclaims
+  it **synchronously before the step** — units revert to their host copies (an address change, not a
+  copy) and both read streams drain — and `arena_tick` regrows the loan only **2 s after the last big
+  step** (`core/engine.cpp:1129-1202`), so every step of the prefill run, decode-only included, runs
+  with a slab ~780 expert units smaller over a link also carrying the staged copies. `/diagnostics`
+  says the binding constraint in one line: "a prefill step streams every routed layer's pooled
+  experts onto the card, so its speed is bounded by the link; decode reads the ones it routes to".
+  Measured size of the whole effect: the ~10 % above. Real, small, not worth chasing.
+- **`--prefill-interleave` is the dial that buys decode its cadence; armed at 1 on 2026-10-20.**
+  N decode-only steps follow any chunk step taken beside decoders; held steps admit nothing, so it
+  costs the prefilling prompt its TTFT and an arriving request its first step. Rate arithmetic at the
+  two measured step times and 0.657 acceptance (≈2.3 tokens a step): N=0 → ~14 t/s, **N=1 (armed) →
+  ~23**, N=2 → ~33, N=4 → ~48, N=8 → ~69, with the prefill slowed by (166 + 15.5N)/166. **Arithmetic, not
+  measurement** — the arm is chosen by the §9 bucket pair plus `steps_held`, and `auto` is not a
+  substitute for it.
+- **`--prefill-span` is the shape that keeps the loan, and it is priced.** It needs MNBT = N
+  intervals, so `--prefill-span 2 --max-num-batched-tokens 4096`: a *lone* prefill carries 2
+  intervals a step (and on an expert-tiered box only from 8 intervals / 16k tokens, where the span
+  starts to win), while **every other step is planned at the shared budget of 2048** — which is a
+  lending level *and* keeps the stager's ring, the one combination `--busy-prefill-chunk` cannot
+  express. Costs per GUIDE §4.5: 320 MiB a card of expert residency, ~3 % off the KV pool (§4.3's
+  cliff is sharp), a decode step uploads twice the per-step arrays at 4096, and checkpoints inside a
+  span are not written, so a partially-shared prompt resumes one interval earlier. On the vendor's
+  second box (2×R9700, w4nl64, TP2) it was TTFT-neutral — 2.61 vs 2.56 s at 18k, 10.72 vs 10.55 at
+  75k — hence their own rule: measure your own prompts at two ranks before turning it on. MNBT 4096
+  does not boot at 3 slots; this shape is 2.
+- **Most of this endpoint's prefill work is replay of tokens the prefix cache already had.** Over
+  the same window `prefill_tokens_total` 65.2 M against `prefix_hit_cut_tokens_total` 24.9 M = **38 %
+  replayed**, and it is not an eviction: a hit on a hybrid model reuses only as far as the last
+  **linear-state** checkpoint (`reuse = min(h.n_attn_tokens, h.n_linear_tokens)`,
+  `core/sched/scheduler.cpp:953-978`), and the retention default keeps a **geometric ladder** - about
+  `log2(depth/interval) + 2` snapshots - so a turn that diverges `d` behind the tip can resume as
+  deep as `tip − 3d`. The live ladder for a 200k session at interval 2048 is 2048 · 131072 · 163840
+  · … · 198656: diverge at 120k and 21.7k tokens replay, diverge at 50k and 48k. Every
+  `prefix_hit_cuts_total` carries `reason="no_snapshot"` (the other two reasons stay dead), so the
+  cause object agrees and `hit_cut_tokens` is the cost. `--checkpoint-policy` admits exactly
+  **`geometric-backoff` | `tip-only` | `keep-all`** and refuses anything else at boot. It governs
+  retention only - the writes happen at every interval whatever the policy says (only `--prefill-span`
+  skips them) - so `keep-all` buys a replay bounded by one interval at any depth for **pinned host RAM
+  alone**: 97 snapshots a 200k session, `--checkpoint-slots auto` asking 196 × 111.98 MiB = 21.4 GiB
+  against a store budget of an eighth of host RAM (31.5 GiB here, so `share` admits it) with `room` -
+  the pinning room left beside the expert pool at boot - as the hard bound the ledger line names.
+  Retention also spares a snapshot a later turn resumed from (`core/mem/prefix.cpp:709-723`), so a
+  chain that keeps resuming at one deep point stops paying for it. Shortening
+  `--checkpoint-interval` does **not** buy the deep case: the coarsest ladder scale is the largest
+  `interval × 2^k` under the depth, so the deepest gap stays ≈ half the prompt (129,024 tokens at
+  interval 2048 against 130,816 at 256) while every cut multiplies the snapshot writes.
+- **Slots cost KV, host RAM, rows, and pin — never speed.** `--max-num-seqs` carries no radiance
+  constraint: it adds ≈110 MiB of recurrent state per slot, 10 × 111.98 MiB of pinned checkpoint-store
+  RAM per slot (`--checkpoint-slots auto`), and declares the fused TP all-reduce over
+  `seqs × (1 + spec)` rows — `wht6` is bit-exact up to 25 rows, and the live shape is 4 × 3 = **12**, so
+  single-sequence decode stays exact and so does any step up to that many rows. Raising it also raises
+  the cache's *demand* (`ceiling = slots × window`), which the resolver funds from `elastic −
+  expert_floor` — so a slot count without matching host room is paid for by expert residency. Measured
+  2026-10-06: seqs 2 and 3 statistically tied (30.9-33.1 t/s, inside the ±3 % floor). Measured
+  2026-10-19 → 10-20 at 1.8.1: 2 → 4 slots cost **+4,224 MiB of pinned host pool** and **−3.01 GiB of
+  expert residency a card** and bought 2.76 → 5.64 GiB of cache (365,204 → 747,492 tokens). The old
+  "seqs 2 cannot boot" law belonged to the retired tcclaviger fork's `torch.compile` shape
+  specialization, and radiance booted seqs 2 on its first fp8-native start.
 - **A dequantize-to-compute path bills host RAM twice.** On the EMULATION backend an offload request
   of **118 GiB** landed as **~238 GiB** of real host RAM — the dequantized copy rides on top of the
   pinned weights, and the doubling is invisible in the weight count. Read the engine log for
@@ -396,12 +482,35 @@ vllm-radiance era (this table opens on 09-26; the era page claims 10-10).
   4600 MT/s ended it, with zero `gdn_hip` faults either side; the prefix-caching / GDN re-seed
   theory was confounded on both arms. Consumer non-ECC raises no MCE for a silent flip and a
   60 s / 8 GiB pattern check covers 0.5 % of 246 GiB — silent RAM corruption outranks every
-  software theory.
+  software theory. **That mitigation is a BIOS setting and it is not in effect now:** `sudo
+  dmidecode -t memory | grep -i configured` answers 4800 MT/s on 2026-10-20, and the only event
+  between the fix and today is the 2026-10-12 power failure that also reset TTM. Check the speed
+  before crediting any software revert with a fix — and before believing a clean run of the arm
+  you just tested, because the confounder is still in the slot.
 - **TP=2 here** is legal since the 2026-09-20 ProArt B850 swap (x8/x8 on CPU root ports,
   `Routing+`/`ReqEn+` end to end) only because radiance's wire is `libr4d`'s own one-shot
   all-reduce; RCCL needs atomics on the whole chain and fails `hipErrorIllegalState` without
   them (). `wht6` buys +5.6-6.3 % prefill for ~1.3 % of 8-stream
   decode, `exact` is the bit-exact alternative.
+- **The link is already at the board's ceiling — do not chase the `link-below-max` warning.**
+  Both root ports (`00:01.1`, `00:01.3`) are **x8 max** at 32 GT/s, so PCIe 5.0 x8 ≈ 31.5 GB/s is
+  the whole budget per card and the cards' own `pcie_dpm` tables expose only x8 states.
+  `/diagnostics` still warns that the on-card switch hop (`01:00.0`, `05:00.0`) runs "x8 of its
+  x16 while the card is 100 % busy" and suggests a reboot or a reseat — pointless when the host
+  side of the chain is x8 max, which the same endpoint reports two findings later as
+  `link-capped`. The width that matters is the minimum of the chain, and it is x8 either way, so
+  the only lever on a link-bound prefill is **less traffic over the link** (more expert
+  residency, fewer pooled experts), never a wider link. P2P resolves **on** (`auto` asks the
+  driver, the two cards reach each other's memory), so the collectives do not hairpin through host
+  RAM, and the engine's copies ride one DMA queue that "give[s] way to the collectives" — the
+  all-reduce wins the link, the staged experts wait.
+- **Eight cores is the whole host, media stack included.** Dankserver is a Ryzen 7 7700X (8
+  cores / 8 threads, one NUMA node, `nodes: -1` for both cards) running Jellyfin, Plex, Calibre
+  and RomM beside the engine, and `--http-threads` defaults to one thread per core. Measured
+  during a heavy prefill: load average ≈ 1.0, `jellyfin` 11 % CPU, `radiance` 7 % — **CPU
+  contention is not the current decode culprit**, so do not spend an experiment window on it;
+  re-read `uptime` and `ps -eo pcpu,comm --sort=-pcpu | head` first if a future collapse looks
+  different from the step-sharing one.
 - `--prefix-cache-dir` creation is **not recursive** — the serve script `mkdir -p`s the path or
   the disk tier silently never opens. The tier does survive a payload swap when arch and
   tokenizer match (it recovered 900,176 blocks on the 2026-10-15 swap; `cache_n` proves a hit).
@@ -445,11 +554,19 @@ vllm-radiance era (this table opens on 09-26; the era page claims 10-10).
 ## Contract (the wire)
 
 - **Live surface:** `/health` (== `/ping`), `/v1/models`, `/v1/chat/completions`
-  `/v1/completions`, `/metrics`, `/stats`, `/sessions`, `/dashboard`. `/health` answers 200
-  JSON (model id, `in_flight`, `queue_depth`) while serving and **503 only while stopping**
-  (`core/server/admin.cpp:385`), and the listener binds *after* the full load (`main.cpp`:
-  `engine.start()` then `engine.serve()`) — during the load the port refuses connections, so a
-  bare `curl -fsS …/health` gate is honest and needs no body grep.
+  `/v1/completions`, `/metrics`, `/stats`, `/sessions`, `/dashboard`, **`/diagnostics`**. `/health`
+  answers 200 JSON (model id, `in_flight`, `queue_depth`) while serving and **503 only while
+  stopping** (`core/server/admin.cpp:385`), and the listener binds *after* the full load
+  (`main.cpp`: `engine.start()` then `engine.serve()`) — during the load the port refuses
+  connections, so a bare `curl -fsS …/health` gate is honest and needs no body grep.
+- **`/diagnostics` is the instrument for "is the box, not the config, the limit"** — one call
+  returns `configuration` (every flag, so drift from the env file is checkable without the host),
+  `environment`, `derived` (the VRAM split as the ledger prints it), `hardware` (host, per-card
+  `link_chain` + `pcie_dpm` + clocks + power cap + DMA engine choice + P2P answer) and `findings`
+  (level / what / why / fix). On this box it reports `experts-from-host` (5.99 GiB a rank crosses
+  PCIe when used), `link-capped` (root ports are x8 max — the board), `swap-in-use`,
+  `tp-wire-lossy`, and **`link-below-max` as a `warn` at 100 % card busy on both cards** — which is
+  structural on this board; see the x8-ceiling law under *This host, and the deploy path*.
 - **Served id.** `FRANK_ARCHON_SERVED_NAME=frank-archon` (the wire rename of 2026-10-09 moved
   only the alias; container, unit, scripts and env keep the service name). The env's contract
   line is that **radiance validates the requested model name, so every client — harness, bench
@@ -476,12 +593,28 @@ vllm-radiance era (this table opens on 09-26; the era page claims 10-10).
   and 16,384 patches per pass. Probe with a ≥ 256 px solid-colour PNG and a ≥ 256 token budget
   — at 64 tokens you get empty `content` and the colour only inside `reasoning_content`.
 - **Capacity behaviour:** a prompt over `--max-model-len` is refused, never truncated; a request
-  longer than the KV pool's token count is accepted then refused as too large to fit; under the
-  3-slot contract a **fourth** full-height conversation is refused, not paged or squeezed.
+  longer than the KV pool's token count is accepted then refused as too large to fit; a request
+  beyond `--max-num-seqs` is held by the HTTP port (`queue_cap` = `max(8, 8 × slots)` — **32** here)
+  and answered 503 only past that, never paged or squeezed. The cache funds **3.7** full-height
+  sessions, so four conversations can be resident but a fourth near-full-height one can still
+  **preempt a running sequence from the back** to free its blocks
+  (`core/sched/scheduler.cpp:1291-1322`); the victim is recomputed, so its session halts and then
+  re-prefills. Meter: `vllm:num_preemptions_total` (cumulative for the boot, `/stats preemptions`);
+  `radiance:num_requests_preempted` is only the instant depth.
 - **Observability:** `/metrics` carries vLLM-compatible names plus `radiance:*`, scraped by the
-  Dankclaw Prometheus job at `192.168.0.235:7770` over plain LAN routing (the netavark
-  published-port quirk is same-host only). The boot ledger — the referee for every shape number
-  above — is in the container log (`podman logs frank-archon`)
+  Dankclaw Prometheus job at `192.168.0.235:7770` (15 s interval) over plain LAN routing (the
+  netavark published-port quirk is same-host only). **Latency lives under `vllm:`, cadence under
+  `radiance:`** — TTFT / inter-token / per-output-token / e2e / decode-time histograms and the
+  running/waiting gauges are `vllm:*` with no radiance-native equivalent, so a `radiance:`-only
+  query misses every latency answer. The step-phase counters are all scraped:
+  `engine_steps{,_decode,_mixed,_prefill,_held}_total`, `engine_step_seconds{,_decode}_total`,
+  `engine_prefill_chunks_capped_total`, `prefill_tokens_total`, `decode_tokens_per_second`,
+  `prefill_tokens_per_second`, `prefix_hit_cut_tokens_total`, `http_queue_depth`. Two holes: the
+  engine's own step-detail histograms (`prefill_steps_duration`, `prefill_step_tokens`,
+  `token_queue_wait`) are exported but reach no target, and the **arena / expert slab / heat engine /
+  prefill stager have no metric at all** — that half of every law above is read from the boot ledger,
+  `/stats` and `/diagnostics`, never from a graph. The boot ledger — the referee for every shape
+  number here — is in the container log (`podman logs frank-archon`);
   `/home/dank/logs/frank_archon.log` is where the serve script redirects `podman run`.
 
 ## Files

serverAppConfigs-docs/LLM/frank-dragoon.md

@@ -10,14 +10,15 @@ tags:
   - gfx1151
   - halogen
   - endpoint
-timestamp: 2026-10-19
+timestamp: 2026-10-21
 ---
 
 # frank-dragoon — halogen-flash-server 0.16.2, native v2 abliterated (Dankclaw iGPU)
 
 Strix Halo (gfx1151) endpoint `http://192.168.0.143:7770`, served id `frank-dragoon`: a podman container
 under a root `User=dank` unit. Live shape since 2026-10-17, pool raised to 401,408 on 2026-10-18
-(`2b12716`). This page replaces every per-era dragoon runbook; the Eras table is the rollback record.
+(`2b12716`), ceiling raised to the native 262,144 on 2026-10-21 so the harness window is 200,000.
+This page replaces every per-era dragoon runbook; the Eras table is the rollback record.
 
 ## Current shape
 
@@ -37,10 +38,12 @@ Image `ghcr.io/peonist-ai/halogen-flash-server@sha256:0c61bf84ac22308a53f5d1ca6b
 unauthenticated token port 8730 stays on container loopback. Engine stdout →
 `/home/dank/logs/frank_dragoon.log`, **not** the journal.
 
-- Shape: `HALOGEN_CTX=200704` hard per-request ceiling · `HALOGEN_KV_SLOTS=2` ·
-  `HALOGEN_KV_POOL_POSITIONS=401408` = slots × ctx, the top of the band `tests/frank_dragoon_unit.bats`
-  enforces (`ctx ≤ pool ≤ slots × ctx`) · KV ~28 KiB/position → **11.0 GiB reserved** · no KV-dtype knob
-  exists here, the pool is the only KV lever · native window 262,144, `HALOGEN_ROPE_YARN` beyond it.
+- Shape: `HALOGEN_CTX=262144` hard per-request ceiling = the checkpoint's native window (raised
+  2026-10-21 from 200704; `HALOGEN_ROPE_YARN` beyond native is not on plan) · `HALOGEN_KV_SLOTS=2` ·
+  `HALOGEN_KV_POOL_POSITIONS=401408` inside the band `tests/frank_dragoon_unit.bats` enforces
+  (`ctx ≤ pool ≤ slots × ctx`) · KV ~28 KiB/position → **11.0 GiB reserved** · no KV-dtype knob
+  exists here, the pool is the only KV lever · the pool carries TWO maximum harness-length (200k)
+  conversations at once; two literal ceiling-length requests would queue — that is a pool raise.
 - Knobs: `HALOGEN_MAX_TOK=8192` sizes the prefill arena (5.7 GiB) and is only the default for requests
   omitting `max_tokens` · `HALOGEN_MAX_TOKENS_CAP=65536` + `HALOGEN_QUEUE_TIMEOUT=7200` are a coupled
   pair · MTP depth 2 with the head INSIDE the checkpoint (**`HALOGEN_MTP_HEAD` stays unset**) and
@@ -53,10 +56,11 @@ unauthenticated token port 8730 stays on container loopback. Engine stdout →
   (Pin::None)`; no clamp line. Host `MemAvailable` read 22 GiB right after the 2026-10-18 raise —
   ~20 GiB above the 20 GiB reserve, against 10–15 GiB of baseline co-tenancy (frank-zealot, ComfyUI,
   Caddy/observability).
-- Harness (`~/.omp/agent/models.yml`, provider `llamacpp (dankclaw)`): `contextWindow: 155648`
-  (= ceiling − 45k per the 400-zone law, never higher), `maxTokens: 45056`, `input: [text, image]`
-  `reasoningContentField: "reasoning_content"`, `high→xhigh` fold, `extraBody` 1.0/0.95. Compaction
-  rides [[frank-zealot]].
+- Harness (`~/.omp/agent/models.yml`, provider `llamacpp (dankclaw)`): `contextWindow: 200000`
+  — sits 62,144 under the ceiling, clearing the ~45k fit-law gap (400-zone law below);
+  `tests/frank_dragoon_unit.bats` fails if the page's declared window and `HALOGEN_CTX` drift back
+  together. `maxTokens: 45056`, `input: [text, image]`, `reasoningContentField: "reasoning_content"`,
+  `high→xhigh` fold, `extraBody` 1.0/0.95. Compaction rides [[frank-zealot]].
 - Deploy `./scripts/deploy.sh dankclaw --service frank-dragoon`; gate = boot-ledger greps (weights,
   pool, MTP, prompt cache, **no clamp**) + `/health` 200 + a tool call + the 512 px red-PNG probe.
   Kernel: `CONFIG_HSA_AMD_SVM=y` (CachyOS ships it) and `amdgpu.noretry=0` — the env says check it at
@@ -124,14 +128,18 @@ created it — earlier topologies in .
   prompt tokens. halogen 0.1.4's auto-sizing refused here and re-prefilled every 79–81k turn in
   85–180 s until `HALOGEN_CACHE_MB=16384` was pinned. What the pool takes is what the table loses, so
   re-measure the warm ratio after any pool change.
-- **400-zone law:** the harness window MUST sit ~45k under the engine ceiling (`contextWindow:
-  155648` = 200,704 − 45,056) because the harness clamps `max_tokens` to `contextWindow − est − 4096`
-  its chars/4 `est` under-counts code/agent traffic by ~3 %, and compaction only fires at 85 %. The
-  3 % is measured, not theory: 2026-10-07 at the old 200,192 ceiling, est 152,938 vs server 157,388 —
-  over by 354, a 400. At the live shape compaction fires at 132,301 est ≈ 136k measured (the 180k
-  target of the 262,144 era is unreachable — the window itself is 155,648), and the worst-case request
-  is 155,648 − 4,096 + 3 % ≈ 156.2k ≤ 200,704: ~9.5× the error of margin, robust to a ~25 %
-  under-count. Compaction rides [[frank-zealot]] at ≈ threshold − keepRecent(20k) ≈ 112k input.
+- **400-zone law:** the harness window MUST sit ~45k under the engine ceiling, because the harness
+  clamps `max_tokens` to `contextWindow − est − 4096`, its chars/4 `est` UNDER-counts code/agent
+  traffic by ~3 %, and the compaction trigger fires at 95 % of the window (`config.yml
+  thresholdPercent`, while the comments designed around 85 % — check the trigger BEFORE touching a
+  window; that disagreement is why long sessions kept drawing 400s). The 3 % is measured, not
+  theory: 2026-10-07 at the old 200,192 ceiling, est 152,938 vs server 157,388 — over by 354, a 400.
+  At the live pair (ceiling 262,144 / window 200,000) the worst-case ordinary request is
+  200,000 − 4,096 + 3 % ≈ 201.8k, and the trigger point is 190,000 est × 1.03 + 45,056 ≈ 240.7k —
+  both inside the ceiling with ~60k of margin. Compaction rides [[frank-zealot]] at
+  ≈ threshold − keepRecent(20k) ≈ 175k measured against its 200,704-token slot. The seam in
+  `tests/frank_dragoon_unit.bats` re-runs this arithmetic at the 95 % trigger and reads the window
+  from THIS page, so ceiling/window pairs that drift back together fail the suite, not a session.
 - **Admission reserves prompt + `max_tokens`:** an un-fittable request waits in arrival order, then
   503s `engine_busy` for the whole queue after `QUEUE_TIMEOUT` — and the cap/timeout pair moves
   together: 65536 tokens at ~10 t/s serial needs >6550 s, which is why 7200 is pinned (env, 2026-10-17).
@@ -248,9 +256,14 @@ lane is **armed and serving** since 2026-10-19 (`HALOGEN_NPU_MODELS=qwen3-embedd
    (`dankclaw/udev/98-halogen-npu-access.rules`, installed and re-triggered by the deployer, `0666` like the GPU
    compute nodes beside it); the serve script refuses any node that is not other-accessible, so the fault cannot
    return as a crash-loop.
-2. The host's XRT reachable inside: there is **no `/opt/xilinx/xrt`** here — distro packages put the three files the
-   container needs (`libxrt_coreutil.so.2`, `libxrt_core.so.2`, `libxrt_driver_xdna.so.2`) in
-   `/usr/lib/x86_64-linux-gnu`, where all three exist.
+2. The host's XRT reachable inside, at **THREE prefixes per library**: there is **no `/opt/xilinx/xrt`** here —
+   distro packages put the three files the container needs (`libxrt_coreutil.so.2`, `libxrt_core.so.2`,
+   `libxrt_driver_xdna.so.2`) in `/usr/lib/x86_64-linux-gnu`. The serve script mounts each REAL file (never a
+   symlink) at `/opt/xilinx/xrt/lib/`, at its own path, AND at `/opt/xilinx/lib/<multiarch>/` — the NPU child
+   searches that third prefix and a missing one is fatal: since the host's `libxrt2` was reconfigured
+   2026-10-20 the child died with `No such library '/opt/xilinx/lib/x86_64-linux-gnu/libxrt_core.so.2'` and the
+   engine supervisor took the WHOLE container down on it (88-restart loop, chat dead beside a healthy GPU line).
+   A dead lane is not survivable here — arm the lane only with all three prefixes mounted.
 3. **The GPU fabric clock held at its top level, or the engine refuses to start.** GPU and NPU work share the die:
    while `pp_dpm_fclk` is changing speed, concurrent GPU work can hang the machine and corrupt NPU results. The
    engine holds the clock itself only when it runs as root with `/sys` mounted; `frank-dragoon` is a **rootless**

shared/alloy/config.alloy

@@ -17,6 +17,9 @@ logging {
 // Loki is published on the host's loopback (127.0.0.1:3100). Alloy shares the
 // host network, so it pushes by loopback, never by container name — a root
 // container is not on the rootless dankclaw_default network.
+// GRAFANA_HOST_LABEL is replaced with this host's name by
+// copy_alloy_config at deploy time; it must not appear verbatim in the
+// installed config.
 loki.write "default" {
   endpoint {
     url = "http://127.0.0.1:3100/loki/api/v1/push"
@@ -32,7 +35,34 @@ loki.write "default" {
 loki.source.journal "journal" {
   format_as_json = true
   labels = {
-    job = "systemd-journal",
+    job  = "systemd-journal",
+    host = "GRAFANA_HOST_LABEL",
+  }
+  forward_to = [loki.process.relabel.receiver]
+}
+
+// Promote the fields that ARE the point of the log stream to real Loki labels.
+// Without this the journal arrives as one opaque JSON line per entry and every
+// per-container view is structurally impossible: Loki metrics queries can only
+// group by INDEXED stream labels, so `sum by (container_name)` returns a single
+// anonymous series and the logs-overview board read empty beside 6.5k
+// lines/5min of live traffic (measured 2026-10-21). CONTAINER_NAME is what
+// podman's journald logging writes (uppercase); host units lack it and stay on
+// the host-only stream. `format_as_json = true` above guarantees every line
+// reaching this stage is a JSON object, so there is no non-JSON branch to
+// protect — alloy 1.21's json stage has no `on_error` attribute (validated
+// against the pinned image); a malformed line would drop, which is the
+// correct fate for a line Loki could not index anyway.
+loki.process "relabel" {
+  stage.json {
+    expressions = {
+      container = "CONTAINER_NAME",
+    }
+  }
+  stage.labels {
+    values = {
+      container = "container",
+    }
   }
   forward_to = [loki.write.default.receiver]
 }

shared/blackbox/blackbox.yml

@@ -0,0 +1,17 @@
+# Blackbox exporter modules — the honest answer to "is this HTTP service
+# answering?". A Prometheus scrape cannot express it (an HTML front page is a
+# 200 that fails the scrape protocol), which is why the old `services` and
+# `comfyui` jobs sat permanently red on `unsupported Content-Type`.
+# See serverAppConfigs-docs/Architecture/Stale Signal Audit.md §1.
+# The listen address is NOT set here: the quadlet unit's Exec passes
+# --listen-address (LAN bind — see the unit header for why loopback is
+# unreachable from the bridge-networked Prometheus). Keep this file to
+# modules only, so the two sources cannot disagree.
+
+modules:
+  http_2xx:
+    prober: http
+    timeout: 10s
+    http:
+      preferred_ip_protocol: ip4
+      valid_status_codes: [] # default: 2xx + 3xx

shared/grafana/provisioning/dashboards/containers.json

@@ -0,0 +1,137 @@
+{
+  "uid": "containers",
+  "title": "Containers — Dankclaw & Dankserver",
+  "description": "Per-container CPU and memory for the quadlet app stacks, from cadvisor (a rootless quadlet on each host). cadvisor keys each series by the cgroup path, and Prometheus relabels the quadlet unit name (app.slice/<name>.service) into a `container` label. Panels use only the metric families cadvisor actually exports for these rootless containers: CPU usage and memory working-set/limit/OOM. Network and filesystem are NOT exported for them (no netns/classid attribution under the rootless+host-PID layout), so no panels pretend otherwise. See serverAppConfigs-docs/Infrastructure/Observability.md.",
+  "tags": ["fleet", "cadvisor", "containers"],
+  "timezone": "browser",
+  "schemaVersion": 39,
+  "refresh": "30s",
+  "time": { "from": "now-6h", "to": "now" },
+  "templating": {
+    "list": [
+      {
+        "name": "host",
+        "label": "Host",
+        "type": "query",
+        "datasource": { "type": "prometheus", "uid": "prometheus" },
+        "query": { "query": "label_values(container_cpu_usage_seconds_total{job=~\"cadvisor-.*\"}, host)", "refId": "host_var" },
+        "refresh": 2,
+        "includeAll": true,
+        "multi": true,
+        "current": { "text": "All", "value": "$__all" }
+      },
+      {
+        "name": "container",
+        "label": "Container",
+        "type": "query",
+        "datasource": { "type": "prometheus", "uid": "prometheus" },
+        "query": { "query": "label_values(container_cpu_usage_seconds_total{job=~\"cadvisor-.*\", host=~\"$host\"}, container)", "refId": "container_var" },
+        "refresh": 2,
+        "includeAll": true,
+        "multi": true,
+        "current": { "text": "All", "value": "$__all" }
+      }
+    ]
+  },
+  "panels": [
+    {
+      "id": 1,
+      "type": "row",
+      "title": "CPU",
+      "collapsed": false,
+      "gridPos": { "h": 1, "w": 24, "x": 0, "y": 0 },
+      "panels": []
+    },
+    {
+      "id": 2,
+      "type": "timeseries",
+      "title": "CPU usage (cores)",
+      "description": "CPU-seconds/s per container = cores busy. A steady high line is a busy service; a spike is a burst (a render, an index, a backup). Only containers are shown, not the host or its daemons.",
+      "gridPos": { "h": 9, "w": 12, "x": 0, "y": 1 },
+      "datasource": { "type": "prometheus", "uid": "prometheus" },
+      "fieldConfig": { "defaults": { "unit": "short", "min": 0, "custom": { "fillOpacity": 8, "lineWidth": 1, "stacking": { "mode": "none" } } }, "overrides": [] },
+      "targets": [
+        {
+          "expr": "sum by (host, container) (rate(container_cpu_usage_seconds_total{job=~\"cadvisor-.*\", host=~\"$host\", container=~\"$container\"}[5m]))",
+          "legendFormat": "{{host}} / {{container}}",
+          "refId": "A"
+        }
+      ]
+    },
+    {
+      "id": 3,
+      "type": "bargauge",
+      "title": "Top containers by CPU (cores)",
+      "description": "Instant ranking over the selected window. Tells you which container to look at when the host is loaded.",
+      "gridPos": { "h": 9, "w": 12, "x": 12, "y": 1 },
+      "datasource": { "type": "prometheus", "uid": "prometheus" },
+      "fieldConfig": { "defaults": { "unit": "short", "min": 0, "max": null, "color": { "mode": "thresholds" }, "thresholds": { "mode": "absolute", "steps": [ { "color": "green", "value": null }, { "color": "yellow", "value": 1 }, { "color": "red", "value": 4 } ] } }, "overrides": [] },
+      "options": { "reduceOptions": { "calcs": [ "lastNotNull" ], "fields": "", "values": false }, "displayMode": "gradient", "orientation": "horizontal", "showUnfilled": true },
+      "targets": [
+        {
+          "expr": "topk(12, sum by (host, container) (rate(container_cpu_usage_seconds_total{job=~\"cadvisor-.*\", host=~\"$host\", container=~\"$container\"}[5m])))",
+          "legendFormat": "{{host}} / {{container}}",
+          "format": "table",
+          "instant": true,
+          "refId": "A"
+        }
+      ]
+    },
+    {
+      "id": 4,
+      "type": "row",
+      "title": "Memory",
+      "collapsed": false,
+      "gridPos": { "h": 1, "w": 24, "x": 0, "y": 10 },
+      "panels": []
+    },
+    {
+      "id": 5,
+      "type": "timeseries",
+      "title": "Memory working set",
+      "description": "container_memory_working_set_bytes per container — the number the OOM killer acts on. Cache-inclusive and reclaimable; a flat high line is resident data, a sawtooth is churn.",
+      "gridPos": { "h": 9, "w": 12, "x": 0, "y": 11 },
+      "datasource": { "type": "prometheus", "uid": "prometheus" },
+      "fieldConfig": { "defaults": { "unit": "bytes", "min": 0, "custom": { "fillOpacity": 8, "lineWidth": 1, "stacking": { "mode": "none" } } }, "overrides": [] },
+      "targets": [
+        {
+          "expr": "container_memory_working_set_bytes{job=~\"cadvisor-.*\", host=~\"$host\", container=~\"$container\"}",
+          "legendFormat": "{{host}} / {{container}}",
+          "refId": "A"
+        }
+      ]
+    },
+    {
+      "id": 6,
+      "type": "timeseries",
+      "title": "Memory vs host total (%)",
+      "description": "Working set over the host's total RAM. The fleet sets NO cgroup memory limits (every container_spec_memory_limit_bytes is 0 — a limit-share panel is structurally empty here), so the honest pressure view is share of the host. The join labels the node exporter's MemTotal onto the cadvisor series by host.",
+      "gridPos": { "h": 9, "w": 12, "x": 12, "y": 11 },
+      "datasource": { "type": "prometheus", "uid": "prometheus" },
+      "fieldConfig": { "defaults": { "unit": "percentunit", "min": 0, "max": 1, "custom": { "fillOpacity": 8, "lineWidth": 1, "stacking": { "mode": "none" } } }, "overrides": [] },
+      "targets": [
+        {
+          "expr": "container_memory_working_set_bytes{job=~\"cadvisor-.*\", host=~\"$host\", container=~\"$container\", container!=\"\"} / on(host) group_left label_replace(node_memory_MemTotal_bytes{job=~\"node-.*\"}, \"host\", \"$1\", \"job\", \"node-(.*)\")",
+          "legendFormat": "{{host}} / {{container}}",
+          "refId": "A"
+        }
+      ]
+    },
+    {
+      "id": 7,
+      "type": "timeseries",
+      "title": "OOM kills",
+      "description": "container_oom_events_total rate. Any non-zero line means the kernel killed a process inside that container — the honest signal that its memory limit is too tight for its real working set.",
+      "gridPos": { "h": 8, "w": 24, "x": 0, "y": 20 },
+      "datasource": { "type": "prometheus", "uid": "prometheus" },
+      "fieldConfig": { "defaults": { "unit": "short", "min": 0, "custom": { "fillOpacity": 20, "lineWidth": 1, "stacking": { "mode": "normal" } } }, "overrides": [] },
+      "targets": [
+        {
+          "expr": "sum by (host, container) (rate(container_oom_events_total{job=~\"cadvisor-.*\", host=~\"$host\", container=~\"$container\"}[5m]))",
+          "legendFormat": "{{host}} / {{container}}",
+          "refId": "A"
+        }
+      ]
+    }
+  ]
+}

shared/grafana/provisioning/dashboards/host-system.json

@@ -0,0 +1,204 @@
+{
+  "uid": "host-system",
+  "title": "Hosts \u2014 Dankclaw & Dankserver",
+  "description": "node_exporter for both servers. Starter set \u2014 throw-at-the-wall breadth; refine the windows and drop the noise. The picker selects the host; CPU/load/memory/filesystem/network are per-node.",
+  "tags": ["host", "system"],
+  "timezone": "browser",
+  "schemaVersion": 39,
+  "refresh": "30s",
+  "time": { "from": "now-6h", "to": "now" },
+  "templating": {
+    "list": [
+      {
+        "name": "node",
+        "label": "host",
+        "type": "query",
+        "datasource": { "type": "prometheus", "uid": "prometheus" },
+        "query": { "query": "label_values(up{job=~\"node-.*\"}, job)", "refId": "A" },
+        "includeAll": true,
+        "multi": true,
+        "current": { "text": "All", "value": "$__all" }
+      },
+      {
+        "name": "gpuhost",
+        "label": "gpu host",
+        "type": "query",
+        "datasource": { "type": "prometheus", "uid": "prometheus" },
+        "query": { "query": "label_values(gpu_utilization_percent, host)", "refId": "A" },
+        "includeAll": true,
+        "multi": true,
+        "current": { "text": "All", "value": "$__all" }
+      }
+    ]
+  },
+  "panels": [
+    {
+      "id": 1, "type": "stat", "title": "Uptime",
+      "description": "Time since boot. A drop means a reboot.",
+      "gridPos": { "h": 4, "w": 6, "x": 0, "y": 0 },
+      "datasource": { "type": "prometheus", "uid": "prometheus" },
+      "fieldConfig": { "defaults": { "unit": "s" }, "overrides": [] },
+      "targets": [{ "expr": "time() - node_boot_time_seconds{job=~\"$node\"}", "legendFormat": "{{job}}", "refId": "A" }]
+    },
+    {
+      "id": 2, "type": "stat", "title": "Memory used",
+      "description": "1 - MemAvailable/MemTotal. The honest \"in use\" figure (excludes reclaimable cache).",
+      "gridPos": { "h": 4, "w": 6, "x": 6, "y": 0 },
+      "datasource": { "type": "prometheus", "uid": "prometheus" },
+      "fieldConfig": { "defaults": { "unit": "percentunit", "min": 0, "max": 1, "thresholds": { "mode": "absolute", "steps": [{ "color": "green", "value": null }, { "color": "yellow", "value": 0.8 }, { "color": "red", "value": 0.92 }] } }, "overrides": [] },
+      "targets": [{ "expr": "1 - node_memory_MemAvailable_bytes{job=~\"$node\"} / node_memory_MemTotal_bytes{job=~\"$node\"}", "legendFormat": "{{job}}", "refId": "A" }]
+    },
+    {
+      "id": 3, "type": "stat", "title": "Load 5m / core",
+      "description": "Per-core load5. >1 sustained means runnable tasks are queuing behind cores.",
+      "gridPos": { "h": 4, "w": 6, "x": 12, "y": 0 },
+      "datasource": { "type": "prometheus", "uid": "prometheus" },
+      "fieldConfig": { "defaults": { "unit": "short", "thresholds": { "mode": "absolute", "steps": [{ "color": "green", "value": null }, { "color": "yellow", "value": 0.7 }, { "color": "red", "value": 1 }] } }, "overrides": [] },
+      "targets": [{ "expr": "node_load5{job=~\"$node\"} / on(job) count by (job) (node_cpu_seconds_total{job=~\"$node\",mode=\"idle\"})", "legendFormat": "{{job}}", "refId": "A" }]
+    },
+    {
+      "id": 4, "type": "stat", "title": "Open file handles",
+      "description": "FDs allocated vs the kernel maximum. Rising toward the max precedes EMFILE across the host.",
+      "gridPos": { "h": 4, "w": 6, "x": 18, "y": 0 },
+      "datasource": { "type": "prometheus", "uid": "prometheus" },
+      "fieldConfig": { "defaults": { "unit": "percentunit", "max": 1, "thresholds": { "mode": "absolute", "steps": [{ "color": "green", "value": null }, { "color": "yellow", "value": 0.7 }, { "color": "red", "value": 0.9 }] } }, "overrides": [] },
+      "targets": [{ "expr": "node_filefd_allocated{job=~\"$node\"} / node_filefd_maximum{job=~\"$node\"}", "legendFormat": "{{job}}", "refId": "A" }]
+    },
+    {
+      "id": 5, "type": "timeseries", "title": "CPU busy %",
+      "description": "CPU seconds/s in each mode = cores busy per mode; the stack sums to total cores in use (16/host here). A user or system spike points at what is loading the box; steal > 0 means the hypervisor is throttling.",
+      "gridPos": { "h": 8, "w": 12, "x": 0, "y": 4 },
+      "datasource": { "type": "prometheus", "uid": "prometheus" },
+      "fieldConfig": { "defaults": { "unit": "short", "min": 0, "custom": { "fillOpacity": 10, "lineWidth": 1, "stacking": { "mode": "normal" } } }, "overrides": [] },
+      "targets": [
+        { "expr": "sum by (job,mode) (rate(node_cpu_seconds_total{job=~\"$node\",mode!=\"idle\",mode!=\"iowait\"}[5m]))", "legendFormat": "{{job}} {{mode}}", "refId": "A" }
+      ]
+    },
+    {
+      "id": 6, "type": "timeseries", "title": "Load average",
+      "description": "load1/5/15 per host.",
+      "gridPos": { "h": 8, "w": 12, "x": 12, "y": 4 },
+      "datasource": { "type": "prometheus", "uid": "prometheus" },
+      "fieldConfig": { "defaults": { "unit": "short" }, "overrides": [] },
+      "targets": [
+        { "expr": "node_load1{job=~\"$node\"}", "legendFormat": "{{job}} load1", "refId": "A" },
+        { "expr": "node_load5{job=~\"$node\"}", "legendFormat": "{{job}} load5", "refId": "B" },
+        { "expr": "node_load15{job=~\"$node\"}", "legendFormat": "{{job}} load15", "refId": "C" }
+      ]
+    },
+    {
+      "id": 7, "type": "timeseries", "title": "Memory breakdown",
+      "description": "Total, free, buffers+cache. Reclaimable cache is not pressure; MemAvailable is the real headroom.",
+      "gridPos": { "h": 8, "w": 12, "x": 0, "y": 12 },
+      "datasource": { "type": "prometheus", "uid": "prometheus" },
+      "fieldConfig": { "defaults": { "unit": "bytes" }, "overrides": [] },
+      "targets": [
+        { "expr": "node_memory_MemTotal_bytes{job=~\"$node\"} - node_memory_MemAvailable_bytes{job=~\"$node\"}", "legendFormat": "{{job}} used", "refId": "A" },
+        { "expr": "node_memory_MemAvailable_bytes{job=~\"$node\"}", "legendFormat": "{{job}} available", "refId": "B" }
+      ]
+    },
+    {
+      "id": 8, "type": "timeseries", "title": "Filesystem free %",
+      "description": "Per mountpoint. The media/library stacks fill disks.",
+      "gridPos": { "h": 8, "w": 12, "x": 12, "y": 12 },
+      "datasource": { "type": "prometheus", "uid": "prometheus" },
+      "fieldConfig": { "defaults": { "unit": "percentunit", "min": 0, "max": 1, "thresholds": { "mode": "absolute", "steps": [{ "color": "red", "value": null }, { "color": "green", "value": 0.15 }] } }, "overrides": [] },
+      "targets": [
+        { "expr": "node_filesystem_avail_bytes{job=~\"$node\",fstype!~\"tmpfs|overlay|squashfs\"} / node_filesystem_size_bytes{job=~\"$node\",fstype!~\"tmpfs|overlay|squashfs\"}", "legendFormat": "{{job}} {{mountpoint}}", "refId": "A" }
+      ]
+    },
+    {
+      "id": 9, "type": "timeseries", "title": "Disk throughput",
+      "description": "Read/write bytes/s per device.",
+      "gridPos": { "h": 8, "w": 12, "x": 0, "y": 20 },
+      "datasource": { "type": "prometheus", "uid": "prometheus" },
+      "fieldConfig": { "defaults": { "unit": "Bps" }, "overrides": [] },
+      "targets": [
+        { "expr": "sum by (job,device) (rate(node_disk_read_bytes_total{job=~\"$node\"}[5m]))", "legendFormat": "{{job}} {{device}} R", "refId": "A" },
+        { "expr": "-sum by (job,device) (rate(node_disk_written_bytes_total{job=~\"$node\"}[5m]))", "legendFormat": "{{job}} {{device}} W", "refId": "B" }
+      ]
+    },
+    {
+      "id": 10, "type": "timeseries", "title": "Disk busy %",
+      "description": "Fraction of time the device had I/O in flight. Near 1 = the device is the bottleneck.",
+      "gridPos": { "h": 8, "w": 12, "x": 12, "y": 20 },
+      "datasource": { "type": "prometheus", "uid": "prometheus" },
+      "fieldConfig": { "defaults": { "unit": "percentunit", "min": 0, "max": 1 }, "overrides": [] },
+      "targets": [
+        { "expr": "rate(node_disk_io_time_seconds_total{job=~\"$node\"}[5m])", "legendFormat": "{{job}} {{device}}", "refId": "A" }
+      ]
+    },
+    {
+      "id": 11, "type": "timeseries", "title": "Network throughput",
+      "description": "Receive (up) / transmit (down) bytes/s per interface.",
+      "gridPos": { "h": 8, "w": 12, "x": 0, "y": 28 },
+      "datasource": { "type": "prometheus", "uid": "prometheus" },
+      "fieldConfig": { "defaults": { "unit": "Bps" }, "overrides": [] },
+      "targets": [
+        { "expr": "sum by (job,device) (rate(node_network_receive_bytes_total{job=~\"$node\",device!~\"lo|veth.*|br.*|docker.*\"}[5m]))", "legendFormat": "{{job}} {{device}} RX", "refId": "A" },
+        { "expr": "-sum by (job,device) (rate(node_network_transmit_bytes_total{job=~\"$node\",device!~\"lo|veth.*|br.*|docker.*\"}[5m]))", "legendFormat": "{{job}} {{device}} TX", "refId": "B" }
+      ]
+    },
+    {
+      "id": 12, "type": "timeseries", "title": "Network errors / drops",
+      "description": "Interface errors and drops \u2014 the signature of a saturated link or a bad NIC/cable.",
+      "gridPos": { "h": 8, "w": 12, "x": 12, "y": 28 },
+      "datasource": { "type": "prometheus", "uid": "prometheus" },
+      "fieldConfig": { "defaults": { "unit": "ops" }, "overrides": [] },
+      "targets": [
+        { "expr": "sum by (job,device) (rate(node_network_receive_errs_total{job=~\"$node\"}[5m]))", "legendFormat": "{{job}} {{device}} RX err", "refId": "A" },
+        { "expr": "sum by (job,device) (rate(node_network_transmit_errs_total{job=~\"$node\"}[5m]))", "legendFormat": "{{job}} {{device}} TX err", "refId": "B" }
+      ]
+    },
+    {
+      "id": 13, "type": "timeseries", "title": "GPU busy %",
+      "description": "Per-GPU utilization from gpu-exporter (amdgpu gpu_busy_percent / nvidia-smi). The picker above selects the host; each series is one card.",
+      "gridPos": { "h": 8, "w": 12, "x": 0, "y": 36 },
+      "datasource": { "type": "prometheus", "uid": "prometheus" },
+      "fieldConfig": { "defaults": { "unit": "percent", "min": 0, "max": 100 }, "overrides": [] },
+      "targets": [
+        { "expr": "gpu_utilization_percent{host=~\"$gpuhost\"}", "legendFormat": "{{host}} {{gpu}} ({{model}})", "refId": "A" }
+      ]
+    },
+    {
+      "id": 14, "type": "timeseries", "title": "GPU VRAM used",
+      "description": "Device memory in use. On the R9700s (frank-archon) and the 3090 (frank-zealot) this is the loaded model's footprint.",
+      "gridPos": { "h": 8, "w": 12, "x": 12, "y": 36 },
+      "datasource": { "type": "prometheus", "uid": "prometheus" },
+      "fieldConfig": { "defaults": { "unit": "bytes", "min": 0 }, "overrides": [] },
+      "targets": [
+        { "expr": "gpu_memory_used_bytes{host=~\"$gpuhost\"}", "legendFormat": "{{host}} {{gpu}} ({{model}})", "refId": "A" }
+      ]
+    },
+    {
+      "id": 15, "type": "timeseries", "title": "GPU VRAM used %",
+      "description": "Used / total device memory. Near 100% is the pre-OOM wall.",
+      "gridPos": { "h": 8, "w": 12, "x": 0, "y": 44 },
+      "datasource": { "type": "prometheus", "uid": "prometheus" },
+      "fieldConfig": { "defaults": { "unit": "percent", "min": 0, "max": 100 }, "overrides": [] },
+      "targets": [
+        { "expr": "100 * gpu_memory_used_bytes{host=~\"$gpuhost\"} / gpu_memory_total_bytes{host=~\"$gpuhost\"}", "legendFormat": "{{host}} {{gpu}} ({{model}})", "refId": "A" }
+      ]
+    },
+    {
+      "id": 16, "type": "timeseries", "title": "GPU temperature",
+      "description": "Package temperature. AMD edge temp (hwmon temp1_input); NVIDIA gpu temp.",
+      "gridPos": { "h": 8, "w": 12, "x": 12, "y": 44 },
+      "datasource": { "type": "prometheus", "uid": "prometheus" },
+      "fieldConfig": { "defaults": { "unit": "celsius", "min": 0 }, "overrides": [] },
+      "targets": [
+        { "expr": "gpu_temperature_celsius{host=~\"$gpuhost\"}", "legendFormat": "{{host}} {{gpu}} ({{model}})", "refId": "A" }
+      ]
+    },
+    {
+      "id": 17, "type": "timeseries", "title": "GPU power draw",
+      "description": "Board power draw. Emitted only where readable: NVIDIA and the Dankclaw iGPU. The R9700s have no sysfs power sensor, so they are simply absent here — never a fabricated 0.",
+      "gridPos": { "h": 8, "w": 12, "x": 0, "y": 52 },
+      "datasource": { "type": "prometheus", "uid": "prometheus" },
+      "fieldConfig": { "defaults": { "unit": "watt", "min": 0 }, "overrides": [] },
+      "targets": [
+        { "expr": "gpu_power_watts{host=~\"$gpuhost\"}", "legendFormat": "{{host}} {{gpu}} ({{model}})", "refId": "A" }
+      ]
+    }
+  ]
+}

shared/grafana/provisioning/dashboards/llm-fleet.json

@@ -194,7 +194,7 @@
       "id": 6,
       "type": "timeseries",
       "title": "Firing fleet alerts",
-      "description": "The LLM-fleet rules from shared/prometheus/rules/llm-fleet.yml. A firing alert is delivered to Signal: Prometheus -> Alertmanager (dankclaw quadlet) -> the host-side signal-relay user unit -> signal-cli, since 2026-09-17.",
+      "description": "The LLM-fleet rules from shared/prometheus/rules/llm-fleet.yml. EMPTY IS THE HEALTHY STATE: zero series means nothing is firing (ALERTS only carries active/pending alert series — do not read this panel as broken). A firing alert is delivered to Signal: Prometheus -> Alertmanager (dankclaw quadlet) -> the host-side signal-relay user unit -> signal-cli, since 2026-09-17.",
       "gridPos": {
         "h": 8,
         "w": 12,

shared/grafana/provisioning/dashboards/logs-overview.json

@@ -0,0 +1,243 @@
+{
+  "uid": "logs-overview",
+  "title": "Logs \u2014 Loki (systemd journal)",
+  "description": "Everything in Loki arrives as the journal (job=systemd-journal); the container is a JSON field parsed at query time, not a stream label. Starter breadth \u2014 the volume/error panels group by the parsed container_name. Loki keeps ~7 days; wide ranges are cheap counts, not the raw log panel.",
+  "tags": [
+    "logs",
+    "loki"
+  ],
+  "timezone": "browser",
+  "schemaVersion": 39,
+  "refresh": "30s",
+  "time": {
+    "from": "now-1h",
+    "to": "now"
+  },
+  "templating": {
+    "list": [
+      {
+        "name": "container",
+        "label": "container",
+        "type": "query",
+        "datasource": {
+          "type": "loki",
+          "uid": "loki"
+        },
+        "query": {
+          "query": "label_values({job=\"systemd-journal\"}, container)",
+          "refId": "LokiVariableQueryEditor-VariableQuery"
+        },
+        "includeAll": true,
+        "multi": true,
+        "refresh": 2,
+        "current": {
+          "text": "All",
+          "value": "$__all"
+        }
+      }
+    ]
+  },
+  "panels": [
+    {
+      "id": 1,
+      "type": "timeseries",
+      "title": "Log volume by container (lines/s)",
+      "description": "Per-container journal line rate. The `container` label is promoted at ingestion by Alloy (loki.process) from podman's CONTAINER_NAME journald field - Loki metrics can only group by indexed stream labels.",
+      "gridPos": {
+        "h": 9,
+        "w": 24,
+        "x": 0,
+        "y": 0
+      },
+      "datasource": {
+        "type": "loki",
+        "uid": "loki"
+      },
+      "fieldConfig": {
+        "defaults": {
+          "unit": "reqps",
+          "custom": {
+            "fillOpacity": 20,
+            "stacking": {
+              "mode": "normal"
+            }
+          }
+        },
+        "overrides": []
+      },
+      "targets": [
+        {
+          "expr": "sum by (container) (count_over_time({job=\"systemd-journal\", container!=\"\"} [$__auto]))",
+          "legendFormat": "{{container_name}}",
+          "refId": "A"
+        }
+      ]
+    },
+    {
+      "id": 2,
+      "type": "timeseries",
+      "title": "Total log rate",
+      "description": "All journal lines per second.",
+      "gridPos": {
+        "h": 8,
+        "w": 12,
+        "x": 0,
+        "y": 9
+      },
+      "datasource": {
+        "type": "loki",
+        "uid": "loki"
+      },
+      "fieldConfig": {
+        "defaults": {
+          "unit": "reqps"
+        },
+        "overrides": []
+      },
+      "targets": [
+        {
+          "expr": "sum(count_over_time({job=\"systemd-journal\"} [$__auto]))",
+          "legendFormat": "all lines/s",
+          "refId": "A"
+        }
+      ]
+    },
+    {
+      "id": 3,
+      "type": "timeseries",
+      "title": "Error-ish line rate (text match)",
+      "description": "Lines matching error/panic/fatal/fail. A blunt text match \u2014 the journal carries no structured level for all units.",
+      "gridPos": {
+        "h": 8,
+        "w": 12,
+        "x": 12,
+        "y": 9
+      },
+      "datasource": {
+        "type": "loki",
+        "uid": "loki"
+      },
+      "fieldConfig": {
+        "defaults": {
+          "unit": "reqps",
+          "color": {
+            "mode": "fixed",
+            "fixedColor": "red"
+          }
+        },
+        "overrides": []
+      },
+      "targets": [
+        {
+          "expr": "sum(count_over_time({job=\"systemd-journal\"} |~ \"(?i)(error|panic|fatal|fail)\" [$__auto]))",
+          "legendFormat": "error lines/s",
+          "refId": "A"
+        }
+      ]
+    },
+    {
+      "id": 4,
+      "type": "timeseries",
+      "title": "Errors by container",
+      "description": "Which unit is throwing. Rank by the top offenders.",
+      "gridPos": {
+        "h": 8,
+        "w": 12,
+        "x": 0,
+        "y": 17
+      },
+      "datasource": {
+        "type": "loki",
+        "uid": "loki"
+      },
+      "fieldConfig": {
+        "defaults": {
+          "unit": "reqps"
+        },
+        "overrides": []
+      },
+      "targets": [
+        {
+          "expr": "sum by (container) (count_over_time({job=\"systemd-journal\", container!=\"\"} |~ \"(?i)(error|panic|fatal)\" [$__auto]))",
+          "legendFormat": "{{container_name}}",
+          "refId": "A"
+        }
+      ]
+    },
+    {
+      "id": 5,
+      "type": "stat",
+      "title": "Error lines (last 15m)",
+      "description": "Absolute count over the window, not a rate.",
+      "gridPos": {
+        "h": 8,
+        "w": 12,
+        "x": 12,
+        "y": 17
+      },
+      "datasource": {
+        "type": "loki",
+        "uid": "loki"
+      },
+      "fieldConfig": {
+        "defaults": {
+          "unit": "short",
+          "thresholds": {
+            "mode": "absolute",
+            "steps": [
+              {
+                "color": "green",
+                "value": null
+              },
+              {
+                "color": "yellow",
+                "value": 50
+              },
+              {
+                "color": "red",
+                "value": 500
+              }
+            ]
+          }
+        },
+        "overrides": []
+      },
+      "targets": [
+        {
+          "expr": "sum(count_over_time({job=\"systemd-journal\"} |~ \"(?i)(error|panic|fatal)\" [15m]))",
+          "instant": true,
+          "legendFormat": "errors/15m",
+          "refId": "A"
+        }
+      ]
+    },
+    {
+      "id": 6,
+      "type": "logs",
+      "title": "Journal (filtered)",
+      "description": "Raw lines for the selected containers. Set the container picker; keep the window tight.",
+      "gridPos": {
+        "h": 12,
+        "w": 24,
+        "x": 0,
+        "y": 25
+      },
+      "datasource": {
+        "type": "loki",
+        "uid": "loki"
+      },
+      "options": {
+        "showTime": true,
+        "wrapLogMessage": true,
+        "sortOrder": "Descending",
+        "enableLogDetails": true
+      },
+      "targets": [
+        {
+          "expr": "{job=\"systemd-journal\", container=~\"$container\"}",
+          "refId": "A"
+        }
+      ]
+    }
+  ]
+}

shared/grafana/provisioning/dashboards/observability.json

@@ -0,0 +1,385 @@
+{
+  "uid": "observability",
+  "title": "Observability \u2014 self-monitoring",
+  "description": "The health of the stack that watches the fleet: Prometheus scrape/TSDB, Loki ingest, Grafana. If this is red the dashboards lie. Note: Alertmanager is running but NOT scraped by Prometheus (no job) \u2014 its metrics are absent by design gap, not fault.",
+  "tags": [
+    "observability",
+    "meta"
+  ],
+  "timezone": "browser",
+  "schemaVersion": 39,
+  "refresh": "30s",
+  "time": {
+    "from": "now-6h",
+    "to": "now"
+  },
+  "panels": [
+    {
+      "id": 1,
+      "type": "stat",
+      "title": "Targets up",
+      "gridPos": {
+        "h": 4,
+        "w": 4,
+        "x": 0,
+        "y": 0
+      },
+      "datasource": {
+        "type": "prometheus",
+        "uid": "prometheus"
+      },
+      "fieldConfig": {
+        "defaults": {
+          "unit": "short",
+          "thresholds": {
+            "mode": "absolute",
+            "steps": [
+              {
+                "color": "green",
+                "value": null
+              }
+            ]
+          }
+        },
+        "overrides": []
+      },
+      "targets": [
+        {
+          "expr": "count(up == 1)",
+          "instant": true,
+          "refId": "A"
+        }
+      ]
+    },
+    {
+      "id": 2,
+      "type": "stat",
+      "title": "Targets down",
+      "description": "Every scrape target that is down. ZERO IS THE HEALTHY STATE: the HTML-probe jobs and the disabled Gitea metrics were retired 2026-10-21 (HTTP liveness now rides the blackbox exporter, job \"services\"). A non-zero count here is a real regression - read /api/v1/targets lastError before triaging.",
+      "gridPos": {
+        "h": 4,
+        "w": 4,
+        "x": 4,
+        "y": 0
+      },
+      "datasource": {
+        "type": "prometheus",
+        "uid": "prometheus"
+      },
+      "fieldConfig": {
+        "defaults": {
+          "unit": "short",
+          "thresholds": {
+            "mode": "absolute",
+            "steps": [
+              {
+                "color": "green",
+                "value": null
+              },
+              {
+                "color": "red",
+                "value": 1
+              }
+            ]
+          }
+        },
+        "overrides": []
+      },
+      "targets": [
+        {
+          "expr": "count(up == 0)",
+          "instant": true,
+          "refId": "A"
+        }
+      ]
+    },
+    {
+      "id": 3,
+      "type": "stat",
+      "title": "TSDB head series",
+      "description": "Active series in memory. A jump = a new exporter or a label-cardinality leak.",
+      "gridPos": {
+        "h": 4,
+        "w": 5,
+        "x": 8,
+        "y": 0
+      },
+      "datasource": {
+        "type": "prometheus",
+        "uid": "prometheus"
+      },
+      "fieldConfig": {
+        "defaults": {
+          "unit": "short"
+        },
+        "overrides": []
+      },
+      "targets": [
+        {
+          "expr": "prometheus_tsdb_head_series",
+          "instant": true,
+          "refId": "A"
+        }
+      ]
+    },
+    {
+      "id": 4,
+      "type": "stat",
+      "title": "Discovered targets",
+      "gridPos": {
+        "h": 4,
+        "w": 5,
+        "x": 13,
+        "y": 0
+      },
+      "datasource": {
+        "type": "prometheus",
+        "uid": "prometheus"
+      },
+      "fieldConfig": {
+        "defaults": {
+          "unit": "short"
+        },
+        "overrides": []
+      },
+      "targets": [
+        {
+          "expr": "sum(prometheus_sd_discovered_targets)",
+          "instant": true,
+          "refId": "A"
+        }
+      ]
+    },
+    {
+      "id": 5,
+      "type": "stat",
+      "title": "Prometheus mem",
+      "gridPos": {
+        "h": 4,
+        "w": 3,
+        "x": 18,
+        "y": 0
+      },
+      "datasource": {
+        "type": "prometheus",
+        "uid": "prometheus"
+      },
+      "fieldConfig": {
+        "defaults": {
+          "unit": "bytes"
+        },
+        "overrides": []
+      },
+      "targets": [
+        {
+          "expr": "process_resident_memory_bytes{job=\"prometheus\"}",
+          "instant": true,
+          "refId": "A"
+        }
+      ]
+    },
+    {
+      "id": 6,
+      "type": "stat",
+      "title": "Grafana up",
+      "gridPos": {
+        "h": 4,
+        "w": 3,
+        "x": 21,
+        "y": 0
+      },
+      "datasource": {
+        "type": "prometheus",
+        "uid": "prometheus"
+      },
+      "fieldConfig": {
+        "defaults": {
+          "mappings": [
+            {
+              "type": "value",
+              "options": {
+                "1": {
+                  "text": "UP",
+                  "color": "green"
+                },
+                "0": {
+                  "text": "DOWN",
+                  "color": "red"
+                }
+              }
+            }
+          ],
+          "thresholds": {
+            "mode": "absolute",
+            "steps": [
+              {
+                "color": "red",
+                "value": null
+              },
+              {
+                "color": "green",
+                "value": 1
+              }
+            ]
+          }
+        },
+        "overrides": []
+      },
+      "targets": [
+        {
+          "expr": "up{job=\"grafana\"}",
+          "instant": true,
+          "refId": "A"
+        }
+      ]
+    },
+    {
+      "id": 7,
+      "type": "table",
+      "title": "Target health",
+      "description": "Every scrape target and its current up. Sort by Value to surface regressions.",
+      "gridPos": {
+        "h": 9,
+        "w": 12,
+        "x": 0,
+        "y": 4
+      },
+      "datasource": {
+        "type": "prometheus",
+        "uid": "prometheus"
+      },
+      "options": {
+        "sortBy": [
+          {
+            "displayName": "Value",
+            "sortOrder": "Asc"
+          }
+        ]
+      },
+      "targets": [
+        {
+          "expr": "up",
+          "format": "table",
+          "instant": true,
+          "refId": "A"
+        }
+      ]
+    },
+    {
+      "id": 8,
+      "type": "timeseries",
+      "title": "Scrape interval drift (p99 actual)",
+      "description": "Actual seconds between scrapes (summary quantile). Above the 15s scrape_interval = Prometheus is falling behind its own cadence (slow endpoints or overload).",
+      "gridPos": {
+        "h": 9,
+        "w": 12,
+        "x": 12,
+        "y": 4
+      },
+      "datasource": {
+        "type": "prometheus",
+        "uid": "prometheus"
+      },
+      "fieldConfig": {
+        "defaults": {
+          "unit": "s"
+        },
+        "overrides": []
+      },
+      "targets": [
+        {
+          "expr": "prometheus_target_interval_length_seconds{quantile=\"0.99\"}",
+          "legendFormat": "{{instance}}",
+          "refId": "A"
+        }
+      ]
+    },
+    {
+      "id": 9,
+      "type": "timeseries",
+      "title": "Scrape samples/s",
+      "description": "Total samples ingested across all jobs.",
+      "gridPos": {
+        "h": 8,
+        "w": 12,
+        "x": 0,
+        "y": 13
+      },
+      "datasource": {
+        "type": "prometheus",
+        "uid": "prometheus"
+      },
+      "fieldConfig": {
+        "defaults": {
+          "unit": "short"
+        },
+        "overrides": []
+      },
+      "targets": [
+        {
+          "expr": "sum(rate(scrape_samples_scraped[5m]))",
+          "legendFormat": "samples/s",
+          "refId": "A"
+        }
+      ]
+    },
+    {
+      "id": 10,
+      "type": "timeseries",
+      "title": "Loki request rate",
+      "description": "Loki served requests/s (push, query). A drop to zero means ingest or query has stopped.",
+      "gridPos": {
+        "h": 8,
+        "w": 12,
+        "x": 12,
+        "y": 13
+      },
+      "datasource": {
+        "type": "prometheus",
+        "uid": "prometheus"
+      },
+      "fieldConfig": {
+        "defaults": {
+          "unit": "reqps"
+        },
+        "overrides": []
+      },
+      "targets": [
+        {
+          "expr": "sum by (route) (rate(loki_request_duration_seconds_count[5m]))",
+          "legendFormat": "{{route}}",
+          "refId": "A"
+        }
+      ]
+    },
+    {
+      "id": 11,
+      "type": "timeseries",
+      "title": "Exporter process CPU",
+      "description": "CPU by monitored process \u2014 spot a runaway exporter or the Prometheus process itself.",
+      "gridPos": {
+        "h": 8,
+        "w": 24,
+        "x": 0,
+        "y": 21
+      },
+      "datasource": {
+        "type": "prometheus",
+        "uid": "prometheus"
+      },
+      "fieldConfig": {
+        "defaults": {
+          "unit": "percentunit"
+        },
+        "overrides": []
+      },
+      "targets": [
+        {
+          "expr": "sum by (job) (rate(process_cpu_seconds_total[5m]))",
+          "legendFormat": "{{job}}",
+          "refId": "A"
+        }
+      ]
+    }
+  ]
+}

shared/prometheus/prometheus.yml

@@ -60,20 +60,33 @@ scrape_configs:
       - targets: ["loki:3100"]
     metrics_path: "/metrics"
 
-  # ComfyUI (no native /metrics; probe the UI)
-  - job_name: "comfyui"
-    metrics_path: "/"
-    static_configs:
-      - targets: ["comfyui:8188"]
-
-  # Blackbox-style probes for service up/down (published LAN ports;
-  # gitea and comfyui are covered by their own jobs above)
+  # HTTP liveness, probed honestly through the blackbox exporter. Scraping a
+  # web root is NOT a health check — an HTML front page fails the scrape
+  # protocol (`unsupported Content-Type`), which is exactly why the old
+  # `services` and `comfyui` jobs were red forever beside healthy services
+  # (Stale Signal Audit §1). The prober is a host-netns container listening on
+  # the podman gateway (the scrape address below - the netavark hairpin gap
+  # bars this container from its own host's LAN IP, cadvisor job note) and on
+  # the LAN; from that netns it reaches the host LAN IP, and published ports
+  # via 127.0.0.1 - bridge container names do NOT resolve in the host netns,
+  # which is why comfyui's probe is a loopback URL. comfyui's own
+  # metrics-scrape job is retired, one owner per service.
   - job_name: "services"
-    metrics_path: "/"
+    metrics_path: /probe
+    params:
+      module: [http_2xx]
     static_configs:
       - targets:
-          - "192.168.0.143:8081"   # SearXNG (via the Gluetun tunnel)
-          - "192.168.0.143:3005"   # OpenWebUI
+          - "http://192.168.0.143:8081"   # SearXNG (via the Gluetun tunnel)
+          - "http://192.168.0.143:3005"   # OpenWebUI
+          - "http://127.0.0.1:8188"        # ComfyUI UI (published 0.0.0.0:8188; probed from the prober's host netns)
+    relabel_configs:
+      - source_labels: [__address__]
+        target_label: __param_target
+      - source_labels: [__param_target]
+        target_label: instance
+      - target_label: __address__
+        replacement: "host.containers.internal:9115"
 
   # LLM fleet. Scraped on the LAN addresses where the endpoint is reachable from
   # this container, not the guarded vhosts: these are machine clients and
@@ -131,3 +144,88 @@ scrape_configs:
       - targets: ["192.168.0.235:7770"]
         labels:
           service: frank-archon
+
+  # Per-container metrics from cadvisor (a rootless quadlet on each host).
+  # cadvisor keys every series by the cgroup path; for a quadlet container that
+  # path is …/app.slice/<name>.service[/libpod-payload-<hex>], so the unit NAME
+  # lives in it. The relabel below pulls that name out into a `container` label
+  # and keeps only the per-unit aggregate — cadvisor's raw host root,
+  # system.slice daemons and machine.slice scopes carry no such name and are
+  # dropped, as is the per-CPU-core split (only cpu="total" survives), so the
+  # scrape stays small. cadvisor cannot bind per-interface, so it is reached at
+  # the podman gateway like every same-host target: a BRIDGE CONTAINER CANNOT
+  # DIAL ITS OWN HOST'S LAN IP AT ALL — the netavark hairpin gap refuses the
+  # connection whether the port is a host-net bind or a published port (the
+  # original note here claimed published ports were exempt; measured 2026-10-21
+  # from inside this container: LAN refused, gateway answers). Cross-host
+  # targets (dankserver) have no hairpin problem and stay plain LAN.
+  - job_name: "cadvisor-dankclaw"
+    static_configs:
+      - targets: ["host.containers.internal:8080"]
+        labels:
+          host: dankclaw
+    metric_relabel_configs:
+      - source_labels: [id]
+        regex: '.*/app\.slice/([^.]+)\.service(?:/.*)?$'
+        target_label: container
+        replacement: '$1'
+      # Keep only the unit-level cgroup; drop its payload/control children so a
+      # container is not counted twice.
+      - source_labels: [id]
+        regex: '.*/\.service/.*'
+        action: drop
+      # Drop everything that is not a named unit (no `container` was extracted).
+      - source_labels: [container]
+        regex: '.+'
+        action: keep
+      # cpu="total" for cpu series; metrics with no cpu label (memory, network,
+      # filesystem) are kept.
+      - source_labels: [cpu]
+        regex: '^$|total'
+        action: keep
+      - regex: 'id|name|image|container_label.*|machine_id|boot_id'
+        action: labeldrop
+
+  - job_name: "cadvisor-dankserver"
+    static_configs:
+      - targets: ["192.168.0.235:8080"]
+        labels:
+          host: dankserver
+    metric_relabel_configs:
+      - source_labels: [id]
+        regex: '.*/app\.slice/([^.]+)\.service(?:/.*)?$'
+        target_label: container
+        replacement: '$1'
+      - source_labels: [id]
+        regex: '.*/\.service/.*'
+        action: drop
+      - source_labels: [container]
+        regex: '.+'
+        action: keep
+      - source_labels: [cpu]
+        regex: '^$|total'
+        action: keep
+      - regex: 'id|name|image|container_label.*|machine_id|boot_id'
+        action: labeldrop
+
+  # Per-GPU metrics (gpu_utilization_percent / gpu_memory_used_bytes /
+  # gpu_memory_total_bytes / gpu_temperature_celsius / gpu_power_watts) from the
+  # host-process gpu-exporter (scripts/gpu-exporter.py). One job per host, same
+  # reach rules as node/cadvisor: Dankclaw is scraped at the podman gateway
+  # because the netavark hairpin quirk makes Prometheus fail to dial its own
+  # host LAN IP; Dankserver is plain LAN. The gpu label (PCI BDF on AMD, index on
+  # NVIDIA) + vendor/model come from the exporter, so the GPU row on the
+  # host-system dashboard filters by host. Power is emitted only where readable
+  # (NVIDIA + the Dankclaw iGPU); the R9700s have no sysfs power, so they simply
+  # have no gpu_power_watts series — no fabricated zero.
+  - job_name: "gpu-dankclaw"
+    static_configs:
+      - targets: ["host.containers.internal:9401"]
+        labels:
+          host: dankclaw
+
+  - job_name: "gpu-dankserver"
+    static_configs:
+      - targets: ["192.168.0.235:9401"]
+        labels:
+          host: dankserver

tests/caddyfile.bats

@@ -193,7 +193,18 @@ CADDYFILE="$REPO_ROOT/dankclaw/Caddyfile"
 }
 
 @test "prometheus targets are host:port (no URL schemes)" {
-    ! grep -nE '^\s*-\s*"?http' "$REPO_ROOT/shared/prometheus/prometheus.yml"
+    # Plain scrape targets are host:port; a URL target is only meaningful for
+    # the blackbox prober job, whose relabel_configs copy it to __param_target
+    # and point __address__ at the prober. Anywhere else a URL target is the
+    # old HTML-probe defect wearing a new hat.
+    awk '
+        /job_name:/ { job=$0; sub(/.*"/, "", job); sub(/".*/, "", job); probe = 0 }
+        /metrics_path: \/probe/ { probe = 1 }
+        /^[[:space:]]*-[[:space:]]*"http/ && !probe {
+            print FILENAME ":" FNR ": URL target outside a prober job: " $0; bad = 1
+        }
+        END { exit bad }
+    ' "$REPO_ROOT/shared/prometheus/prometheus.yml"
 }
 
 @test "observability containers run as container root (rootless = host user)" {

tests/deployment_flow.bats

@@ -125,3 +125,48 @@ load helpers
     [ ! -e "$sandbox/frank-default-sightsound" ]
     [ -e "$BATS_TEST_TMPDIR/.AGENTTRASH/$(date +%Y%m%d)-retired-deploy-state/frank-default-sightsound" ]
 }
+
+@test "the client and remote allowlists agree per target" {
+    # Two gates guard `deploy.sh --service X`: the client's valid_service and
+    # the remote runner's service_is_allowed. When only one side gained a
+    # service the other answered "is not deployed on <target>" and the only
+    # repair path was a fleet-wide drain — cadvisor and gpu-exporter both fell
+    # through this crack within a day of each other. The lists MUST agree;
+    # searxng-network is the one sanctioned exception (client-only network
+    # unit, no remote dispatch handler).
+    run env OUT="$BATS_TEST_TMPDIR" bash -c '
+        grep -oE "dank(claw|server):[a-z0-9-]+" \
+            <(sed -n "/^valid_service()/,/^}/p" "$1/scripts/deploy.sh") \
+            | grep -v "dankclaw:searxng-network" | sort -u > "$OUT/cli"
+        grep -oE "dank(claw|server):[a-z0-9-]+" \
+            <(sed -n "/^service_is_allowed()/,/^}/p" "$1/scripts/deploy-remote.sh") \
+            | sort -u > "$OUT/rem"
+        diff "$OUT/cli" "$OUT/rem"
+    ' _ "$REPO_ROOT"
+    [ "$status" -eq 0 ] || { echo "deploy.sh vs deploy-remote.sh allowlist drift:"; echo "$output"; return 1; }
+    # Drift aside, the gate must actually know the host-process exporters —
+    # their first-ever deploy has to pass the client gate.
+    for pair in dankclaw:cadvisor dankclaw:gpu-exporter dankserver:cadvisor dankserver:gpu-exporter; do
+        grep -q "$pair" "$BATS_TEST_TMPDIR/cli" || { echo "client gate missing $pair"; return 1; }
+    done
+}
+
+@test "every gated dankclaw service is dispatched" {
+    # A service can clear both allowlists and still die inside the deployer:
+    # dankclaw has no compose file, and deploy_compose_service (the case
+    # catch-all) hard-fails with "no quadlet handler" since 2026-09-16. So
+    # every name the remote gate allows for dankclaw MUST be named by a
+    # deploy_service pattern — blackbox-exporter (2026-10-21) shipped with
+    # both gates and the health gate wired but no dispatch case, and only
+    # the deploy attempt found it.
+    run env OUT="$BATS_TEST_TMPDIR" bash -c '
+        grep -oE "dankclaw:[a-z0-9-]+" \
+            <(sed -n "/^service_is_allowed()/,/^}/p" "$1/scripts/deploy-remote.sh") \
+            | cut -d: -f2 | sort -u > "$OUT/gated"
+        sed -n "/^deploy_service()/,/^}/p" "$1/scripts/deploy-remote.sh" \
+            | grep -oE "^ +[a-z][a-z0-9|-]*\)" | tr -d " )" | tr "|" "\n" \
+            | sort -u > "$OUT/dispatched"
+        comm -23 "$OUT/gated" "$OUT/dispatched"
+    ' _ "$REPO_ROOT"
+    [ -z "$output" ] || { echo "gated dankclaw services with no deploy_service case:"; echo "$output"; return 1; }
+}

tests/frank_archon_unit.bats

@@ -145,7 +145,7 @@ nosudo_bin() {
     # (torch.compile query_start_loc constraint, 2026-10-02); radiance booted
     # seqs 2 in the fp8-native era and the 2026-10-06 sweep had seqs 2 and 3
     # tied inside the +-3% noise floor — slots buy concurrency, not speed.
-    grep -q '^FRANK_ARCHON_MAX_SEQS=2$' "$ENV_FILE"
+    grep -q '^FRANK_ARCHON_MAX_SEQS=4$' "$ENV_FILE"
     grep -q '^FRANK_ARCHON_TP=2$' "$ENV_FILE"
     grep -q '^FRANK_ARCHON_TP_WIRE=wht6$' "$ENV_FILE"
     grep -q '^FRANK_ARCHON_SPEC_TOKENS=2$' "$ENV_FILE"
@@ -158,7 +158,7 @@ nosudo_bin() {
     # so the pool's token count is a boot-ledger line — verified live by the
     # capacity leg of scripts/probe-frank-archon-kv.py at every cutover.
     grep -q '^FRANK_ARCHON_PLACEMENT=expert_tiered$' "$ENV_FILE"
-    grep -q '^FRANK_ARCHON_HOST_POOL_MIB=44928$' "$ENV_FILE"
+    grep -q '^FRANK_ARCHON_HOST_POOL_MIB=49152$' "$ENV_FILE"
     grep -q '^FRANK_ARCHON_KV_DTYPE=fp8$' "$ENV_FILE"
     grep -q '^FRANK_ARCHON_NGRAM_PLACEMENT=ram$' "$ENV_FILE"
     # Prefix cache: pinned host RAM only (the disk tier was removed), and the
@@ -177,8 +177,11 @@ nosudo_bin() {
     [ "$ttmt" -gt "$ttmm" ]
     pool=$(awk -F= '/^FRANK_ARCHON_HOST_POOL_MIB=/{print $2}' "$ENV_FILE")
     ph=$(awk -F= '/^FRANK_ARCHON_PREFIX_CACHE_MIB=/{print $2}' "$ENV_FILE")
-    # 2 pools + PLE 47.7 GiB (48,824 MiB) + prefix host, rounded up.
-    ledger=$(( 2 * pool + 48824 + ph + 2048 ))
+    # 2 pools + PLE 47.7 GiB (48,824 MiB) + prefix host + MNBT + the auto
+    # checkpoint store (MAX_SEQS x 10 slots of 112 MiB -- the store is pinned host
+    # memory too, and it grows with the slot count).
+    seqs=$(awk -F= '/^FRANK_ARCHON_MAX_SEQS=/{print $2}' "$ENV_FILE")
+    ledger=$(( 2 * pool + 48824 + ph + 2048 + seqs * 10 * 112 ))
     [ "$ttmm" -ge "$ledger" ]
     # The .rad size floor is positive (a truncated convert must fail fast).
     grep -q '^FRANK_ARCHON_RAD_MIN_MIB=' "$ENV_FILE"

tests/frank_dragoon_unit.bats

@@ -3,7 +3,7 @@
 # of the abliterated Qwen3.8-Flash-Next (a .hgn flash-server payload) on the
 # Strix Halo iGPU through the halogen-flash-server CONTAINER (0.16.2,
 # digest-pinned), driven by the root unit's serve script — 2 slots over ONE
-# 200,704-position KV pool with the MTP head always on (HALOGEN_SPEC_ADAPT=0)
+# 401,408-position KV pool at the native 262,144 ceiling, MTP head always on (HALOGEN_SPEC_ADAPT=0)
 # and the vision tower mounted. 2026-10-17 cutover from the Swift1.5-finetune
 # shape, which itself cut over the halogen 27B-dense shape on 2026-10-06.
 #
@@ -85,7 +85,7 @@ good_env() {
         "FRANK_DRAGOON_VISION_BYTES=0"
         "FRANK_DRAGOON_TOKENIZER_FILES=tok.json"
         "HALOGEN_MODEL_ID=frank-dragoon"
-        "HALOGEN_CTX=200704"
+        "HALOGEN_CTX=262144"
         "HALOGEN_KV_POOL_POSITIONS=401408"
         "HALOGEN_KV_SLOTS=2"
         "HALOGEN_SPEC_ADAPT=0"
@@ -159,9 +159,11 @@ good_env() {
     reserve="$(sed -nE 's/^HALOGEN_HOST_RESERVE_GIB=([0-9]+)$/\1/p' "$ENV_FILE")"
     # The pool is ONE budget behind the slots (boot ledger form: "2 slots over
     # ONE <pool>-position KV pool, each request up to HALOGEN_CTX"), NOT
-    # a per-slot private pool. Since 2026-10-18 it is sized pool = slots * ctx
-    # (401,408 = 2 x 200,704), so two MAXIMUM-length conversations can be
-    # resident at once — frank-zealot parity on total in-flight context. What
+    # a per-slot private pool. Since 2026-10-21 the ceiling is the checkpoint's
+    # native 262,144 while the pool stays 401,408 (raised 2026-10-18): two
+    # MAXIMUM harness-length (200,000-token) conversations are resident at once;
+    # two literal ceiling-length requests would queue — widening that is a pool
+    # raise, not a window move.
     # still binds: the pool must back ONE full-length request (below that the
     # advertised per-request ceiling would be unreachable, hence pool >= ctx),
     # it must not pretend to be more than slots x ctx (pool <= slots * ctx), and
@@ -287,7 +289,7 @@ $(cat "$TEST_TMP/serve.log")"
     [[ "$output" == *"--name frank-dragoon-halogen"* ]]
     [[ "$output" == *"--publish 7770:8731"* ]]
     [[ "$output" == *"-e HALOGEN_MODEL_ID=frank-dragoon"* ]]
-    [[ "$output" == *"-e HALOGEN_CTX=200704"* ]]
+    [[ "$output" == *"-e HALOGEN_CTX=262144"* ]]
     [[ "$output" == *"-e HALOGEN_KV_POOL_POSITIONS=401408"* ]]
     [[ "$output" == *"-e HALOGEN_KV_SLOTS=2"* ]]
     [[ "$output" == *"-e HALOGEN_SPEC_ADAPT=0"* ]]
@@ -306,27 +308,32 @@ $(cat "$TEST_TMP/serve.log")"
 
 @test "the shipped env keeps the harness fit law inside the ceiling" {
     # HALOGEN_CTX is the hard per-request ceiling (a longer prompt is a 400
-    # naming the limit, never a silent truncation). The law is INVARIANT to
-    # the value: the harness window must sit ~45k UNDER it, because the
-    # harness clamps max_tokens = min(45056, window − est − 4096), compacts
-    # at 85 % of its declared window, and its chars/4 est UNDER-counts the
-    # rendered prompt by ~3 % on code/agent traffic (measured 4,450 tokens at
-    # est 152,938 on 2026-10-07 — with window == ceiling, every request over
-    # ~65 % of the window was a guaranteed 400 before compaction could fire).
-    # The shipped pair is ctx 200704 / window 155648 (lowered from
-    # 262144/217088 on 2026-10-17); the gap must absorb the error at BOTH
-    # monotone maxima of est + budget (est ≤ window − 4096):
-    #   full budget at the compaction threshold: 1.03*132300 + 45056
-    #   clamped budget at the clamp's zero point: 1.03*151552 + 0
-    # and the compaction request that rides on frank-zealot (its
-    # 200,704-token slot) must fit: input ≈ threshold − keepRecent(20k).
-    grep -qE '^HALOGEN_CTX=200704$' "$ENV_FILE"
-    ctx=200704
-    hw=155648; mt=45056
-    thr=$(( hw * 85 / 100 ))
+    # naming the limit, never a silent truncation). The law: the harness
+    # window must sit ~45k UNDER the ceiling, because the harness clamps
+    # max_tokens = min(45056, window − est − 4096), compacts at the trigger
+    # percent of its declared window, and its chars/4 est UNDER-counts the
+    # rendered prompt by ~3 % on code/agent traffic (measured 4,450 tokens
+    # at est 152,938 on 2026-10-07 — with window ≈ ceiling, every long
+    # request is a guaranteed 400 before compaction can fire). The 2026-10-21
+    # pair: ceiling 262144 (native) under a harness window of 200,000.
+    # The declared window is read from the endpoint page — the one owner —
+    # so env, models.yml and page that disagree FAIL HERE, not as a 400
+    # mid-session (the recurring failure this seam exists for).
+    grep -qE '^HALOGEN_CTX=262144$' "$ENV_FILE"
+    ctx="$(sed -nE 's/^HALOGEN_CTX=([0-9]+)$/\1/p' "$ENV_FILE")"
+    page="$ROOT/serverAppConfigs-docs/LLM/frank-dragoon.md"
+    grep -qE 'contextWindow: 200000' "$page"
+    hw="$(grep -oE 'contextWindow: [0-9]+' "$page" | head -1 | grep -oE '[0-9]+')"
+    mt=45056
+    # Invariant checked at the live 95 % trigger (config.yml
+    # thresholdPercent) — strictly stronger than the 85 % design point.
+    # Both monotone maxima of est + budget (est ≤ window − 4096):
+    thr=$(( hw * 95 / 100 ))
     [ $(( thr * 103 / 100 + mt )) -le "$ctx" ]
     zero=$(( hw - 4096 ))
     [ $(( zero * 103 / 100 )) -le "$ctx" ]
+    # The compaction request rides on frank-zealot's 200,704-token slot:
+    # input ≈ threshold − keepRecent(20k).
     [ $(( (thr - 20000) * 103 / 100 )) -le 200704 ]
 }
 
@@ -423,9 +430,12 @@ npu_fixture() {
     # the stock root:render 0660 mode, and the serve gate checks exactly that.
     touch "$TEST_TMP/accel0"
     chmod 666 "$TEST_TMP/accel0"
-    mkdir -p "$TEST_TMP/xrt"
+    # basename carries a multiarch shape: the third mount prefix is derived
+    # from the XRT dir's basename (the NPU child searches
+    # /opt/xilinx/lib/<multiarch>/, measured 2026-10-21).
+    mkdir -p "$TEST_TMP/x86_64-linux-gnu"
     for l in libxrt_coreutil.so.2 libxrt_core.so.2 libxrt_driver_xdna.so.2; do
-        touch "$TEST_TMP/xrt/$l"
+        touch "$TEST_TMP/x86_64-linux-gnu/$l"
     done
     mkdir -p "$TEST_TMP/npu/qwen3-embedding-0.6b/devices"
     mkdir -p "$TEST_TMP/sys/class/drm/card1/device"
@@ -435,7 +445,7 @@ npu_fixture() {
         "HALOGEN_NPU_MODELS=qwen3-embedding-0.6b"
         "FRANK_DRAGOON_NPU_DIR=$TEST_TMP/npu"
         "FRANK_DRAGOON_DEV_ACCEL=$TEST_TMP/accel0"
-        "FRANK_DRAGOON_XRT_DIR=$TEST_TMP/xrt"
+        "FRANK_DRAGOON_XRT_DIR=$TEST_TMP/x86_64-linux-gnu"
         "FRANK_DRAGOON_SYSFS=$TEST_TMP/sys"
     )
 }
@@ -465,11 +475,15 @@ $(cat "$TEST_TMP/serve.log")"
     ! grep -q 'accel0' "$TEST_TMP/serve.log"
 }
 
-@test "the lane arms the accel node, the staged model dir and each XRT library twice" {
-    # Each library mounts at the image's XRT path AND at its own path: the
-    # distro libraries find each other where they were installed, and an
-    # /opt/xilinx/xrt of symlinks into the system dir dangles inside the
-    # container (the vendor calls that out by name).
+@test "the lane arms the accel node, the staged model dir and each XRT library three times" {
+    # Each library mounts at the image's XRT path, at its own path (the distro
+    # libraries find each other where they were installed), and at
+    # /opt/xilinx/lib/<multiarch>/ — the prefix the NPU child searches and
+    # names when it is missing (`No such library
+    # '/opt/xilinx/lib/x86_64-linux-gnu/libxrt_core.so.2'`, measured
+    # 2026-10-21: the lane died and the supervisor took the whole container
+    # with it, 88-restart loop). An /opt/xilinx/xrt of symlinks into the
+    # system dir dangles inside the container (the vendor calls that out).
     podman_stub
     good_env
     npu_fixture
@@ -484,8 +498,9 @@ $(cat "$TEST_TMP/serve.log")"
     [[ "$output" != *":/models/npu"* ]]
     [[ "$output" == *"-e HALOGEN_NPU_MODELS=qwen3-embedding-0.6b"* ]]
     for l in libxrt_coreutil.so.2 libxrt_core.so.2 libxrt_driver_xdna.so.2; do
-        [[ "$output" == *"-v $TEST_TMP/xrt/$l:/opt/xilinx/xrt/lib/$l:ro"* ]]
-        [[ "$output" == *"-v $TEST_TMP/xrt/$l:$TEST_TMP/xrt/$l:ro"* ]]
+        [[ "$output" == *"-v $TEST_TMP/x86_64-linux-gnu/$l:/opt/xilinx/xrt/lib/$l:ro"* ]]
+        [[ "$output" == *"-v $TEST_TMP/x86_64-linux-gnu/$l:$TEST_TMP/x86_64-linux-gnu/$l:ro"* ]]
+        [[ "$output" == *"-v $TEST_TMP/x86_64-linux-gnu/$l:/opt/xilinx/lib/x86_64-linux-gnu/$l:ro"* ]]
     done
     # The Flash payload still rides the same container: the lane is additive.
     [[ "$output" == *"-e HALOGEN_CHECKPOINT=/models/qwen.hgn"* ]]
@@ -560,7 +575,7 @@ $(cat "$TEST_TMP/serve.log")"
     podman_stub
     good_env
     npu_fixture
-    rm "$TEST_TMP/xrt/libxrt_driver_xdna.so.2"
+    rm "$TEST_TMP/x86_64-linux-gnu/libxrt_driver_xdna.so.2"
     run_with_log
     [ "$status" -ne 0 ]
     [[ "$output" == *"libxrt_driver_xdna.so.2 missing"* ]]

tests/frankbook_quadlet.bats

@@ -77,6 +77,7 @@ FRANKBOOK_UNITS=(
 }
 
 @test "readarr keeps its pinned image" {
-    grep -q '^Image=lscr.io/linuxserver/readarr:amd64-1.0.980$' \
+    # 2026-10-21 fleet law: version tag AND digest (tests/quadlet_image_pins.bats)
+    grep -qE '^Image=lscr\.io/linuxserver/readarr:amd64-1\.0\.980@sha256:[0-9a-f]{64}$' \
         "$REPO_ROOT/frankbook-pro/readarr/quadlet/readarr.container" || return 1
 }

tests/gpu_exporter.bats

@@ -0,0 +1,227 @@
+#!/usr/bin/env bats
+# Seam: the GPU metrics exporter's black-box contract.
+#
+# The exporter reads amdgpu sysfs (one card per /sys/class/drm/cardN that has a
+# device/gpu_busy_percent file) and, on the box that has a working NVIDIA stack,
+# one `nvidia-smi --query-gpu` CSV, and emits the unified `gpu_*` Prometheus
+# families the host-system dashboard graphs. Both sources are injectable so the
+# whole thing is testable with fixtures on a GPU-less dev box.
+#
+# Only the inputs a consumer sees are pinned: the metric families, their label
+# set, and that absent hardware yields ABSENT series (never a fabricated 0 — a
+# wrong number on a dashboard gets believed). The R9700s expose no sysfs
+# power1_average, so those cards legitimately carry no gpu_power_watts line.
+
+load helpers
+
+EXPORTER="$REPO_ROOT/scripts/gpu-exporter.py"
+
+# Build a fixture amdgpu sysfs tree. $BATS_TEST_TMPDIR/sys
+mk_amd() { # gpu busy vram_used vram_total power_uw_or_blank temp_mc
+    local card=$1 busy=$2 used=$3 total=$4 power=$5 temp=$6
+    local d="$SYS/card$card/device"
+    mkdir -p "$d/hwmon/hwmon9"
+    printf '%s\n' "$busy" > "$d/gpu_busy_percent"
+    printf '%s\n' "$used" > "$d/mem_info_vram_used"
+    printf '%s\n' "$total" > "$d/mem_info_vram_total"
+    printf '%s\n' "$power" > "$d/power1_average"          # blank => absent on R9700
+    [ -n "$temp" ] && printf '%s\n' "$temp" > "$d/hwmon/hwmon9/temp1_input"
+    printf 'DRIVER=amdgpu\nPCI_ID=1002:7551\nPCI_SLOT_NAME=0000:03:00.0\n' > "$d/uevent"
+}
+
+setup() {
+    SYS="$BATS_TEST_TMPDIR/sys"
+    mkdir -p "$SYS"
+}
+
+@test "exporter exists and compiles" {
+    [ -f "$EXPORTER" ]
+    python3 -m py_compile "$EXPORTER"
+}
+
+@test "amd card with full sysfs yields util/vram/temp/power with the right labels and units" {
+    mk_amd 1 42 1073741824 34359738368 250000000 71000   # 250 W, 71 C
+    run python3 "$EXPORTER" --amdgpu-sysfs "$SYS" --nvidia-cmd false
+    [ "$status" -eq 0 ] || { echo "$output"; return 1; }
+    # util is a raw percent, VRAM verbatim bytes, temp milli->C, power microW->W.
+    echo "$output" | grep -Eq 'gpu_utilization_percent\{[^}]*vendor="amd"[^}]*\} 42$' || { echo "$output"; return 1; }
+    echo "$output" | grep -Eq 'gpu_memory_used_bytes\{[^}]*\} 1073741824$' || { echo "$output"; return 1; }
+    echo "$output" | grep -Eq 'gpu_memory_total_bytes\{[^}]*\} 34359738368$' || { echo "$output"; return 1; }
+    echo "$output" | grep -Eq 'gpu_temperature_celsius\{[^}]*\} 71$' || { echo "$output"; return 1; }
+    echo "$output" | grep -Eq 'gpu_power_watts\{[^}]*\} 250$' || { echo "$output"; return 1; }
+    # identity labels: the PCI slot is a stable unique id, vendor marks the family.
+    echo "$output" | grep -Eq 'gpu_utilization_percent\{.*gpu="0000:03:00\.0"' || { echo "$output"; return 1; }
+}
+
+@test "card with an empty power file and no hwmon yields util/vram but omits power and temp" {
+    mk_amd 1 99 34108338176 34208743424 "" ""           # the R9700 shape
+    run python3 "$EXPORTER" --amdgpu-sysfs "$SYS" --nvidia-cmd false
+    [ "$status" -eq 0 ] || { echo "$output"; return 1; }
+    echo "$output" | grep -Eq 'gpu_utilization_percent\{[^}]*\} 99$' || { echo "$output"; return 1; }
+    echo "$output" | grep -Eq 'gpu_memory_used_bytes\{[^}]*\} 34108338176$'
+    # Absent means ABSENT, not a zero the dashboard would plot as a real reading.
+    ! echo "$output" | grep -Eq '^gpu_power_watts\{' || { echo "fabricated power line: $output"; return 1; }
+    ! echo "$output" | grep -Eq '^gpu_temperature_celsius\{' || { echo "fabricated temp line: $output"; return 1; }
+}
+
+@test "nvidia csv yields util/vram/temp/power tagged vendor=nvidia" {
+    nvidia="$BATS_TEST_TMPDIR/nvidia-smi"
+    cat > "$nvidia" <<'STUB'
+echo '0, NVIDIA GeForce RTX 3090, 93, 23004, 24576, 76, 397.40'
+STUB
+    chmod +x "$nvidia"
+    run python3 "$EXPORTER" --amdgpu-sysfs "$SYS" --nvidia-cmd "$nvidia"
+    [ "$status" -eq 0 ] || { echo "$output"; return 1; }
+    echo "$output" | grep -Eq 'gpu_utilization_percent\{[^}]*vendor="nvidia"[^}]*\} 93$' || { echo "$output"; return 1; }
+    # memory.used MiB -> bytes (23004 * 1048576)
+    echo "$output" | grep -Eq 'gpu_memory_used_bytes\{[^}]*vendor="nvidia"[^}]*\} 24121442304$' || { echo "$output"; return 1; }
+    echo "$output" | grep -Eq 'gpu_temperature_celsius\{[^}]*vendor="nvidia"[^}]*\} 76$'
+    echo "$output" | grep -Eq 'gpu_power_watts\{[^}]*vendor="nvidia"[^}]*\} 397\.4$'
+    echo "$output" | grep -Eq 'model="NVIDIA GeForce RTX 3090"'
+}
+
+@test "a failing nvidia command is tolerated: amd still emits and exit is 0" {
+    mk_amd 1 50 1000000 2000000 "" 40000
+    failing="$BATS_TEST_TMPDIR/nvidia-fail"
+    printf '#!/bin/sh\nexit 1\n' > "$failing"; chmod +x "$failing"
+    run python3 "$EXPORTER" --amdgpu-sysfs "$SYS" --nvidia-cmd "$failing"
+    [ "$status" -eq 0 ] || { echo "$output"; return 1; }
+    echo "$output" | grep -Eq 'gpu_utilization_percent\{[^}]*vendor="amd"[^}]*\} 50$'
+    ! echo "$output" | grep -q 'vendor="nvidia"'
+}
+
+@test "no GPUs at all yields only HELP/TYPE headers and exits 0" {
+    run python3 "$EXPORTER" --amdgpu-sysfs "$SYS" --nvidia-cmd false
+    [ "$status" -eq 0 ] || { echo "$output"; return 1; }
+    # families are declared even when empty, so the dashboard/queries stay valid.
+    echo "$output" | grep -q '# TYPE gpu_utilization_percent gauge'
+    echo "$output" | grep -q '# TYPE gpu_memory_used_bytes gauge'
+    ! echo "$output" | grep -Eq '^gpu_utilization_percent\{'
+}
+
+@test "output is valid Prometheus text" {
+    mk_amd 1 42 1000 2000 "" 50000
+    run python3 "$EXPORTER" --amdgpu-sysfs "$SYS" --nvidia-cmd false
+    [ "$status" -eq 0 ]
+    # every non-comment line is "<name>{labels} value" or "<name> value"
+    printf '%s\n' "$output" | python3 -c '
+import sys, re
+pat = re.compile(r"^[a-z_]+(\{[^}]*\})? [-+0-9.eE+]+$")
+for i, l in enumerate(sys.stdin):
+    l = l.rstrip("\n")
+    if not l or l.startswith("#"):
+        continue
+    assert pat.match(l), "malformed line %d: %r" % (i, l)
+print("prometheus text ok")
+'
+}
+
+# ---- deployment + wiring contracts -------------------------------------
+# The exporter is a HOST-PROCESS user unit (not a quadlet): it must read the
+# host's own DRM sysfs and shell to nvidia-smi, which a container could not do
+# without GPU passthrough. These tests pin the cross-file wiring a reader
+# cannot eyeball — a lone change on one side leaves the metric silently absent.
+
+@test "the base unit runs the exporter from the checkout and never binds a wildcard" {
+    local u="$REPO_ROOT/scripts/gpu-exporter.service"
+    [ -f "$u" ]
+    grep -q '^ExecStart=/usr/bin/python3 /home/dank/serverAppConfigs/scripts/gpu-exporter.py --serve$' "$u" \
+        || { echo "unit does not run the exporter --serve from the checkout"; return 1; }
+    grep -q '^Restart=always$' "$u"
+    grep -q '^WantedBy=default.target$' "$u"
+    # Base unit binds loopback only; the per-host LAN address is added by a
+    # deploy drop-in, so a checked-in unit is never a wildcard.
+    grep -q '^Environment=GPU_EXPORTER_BINDS=127.0.0.1$' "$u"
+    ! grep -qE '0\.0\.0\.0' "$u"
+}
+
+@test "the deployer installs, drops in the LAN bind, and health-gates gpu-exporter" {
+    local handler="$REPO_ROOT/scripts/deploy-remote.sh"
+    grep -q '^deploy_gpu_exporter() {' "$handler"
+    # Per-host LAN from $TARGET (never a hardcoded host, never a wildcard).
+    grep -q 'dankclaw) lan=192.168.0.143' "$handler"
+    grep -q 'dankserver) lan=192.168.0.235' "$handler"
+    # Drop-in binds loopback + this host's LAN as one comma list.
+    grep -q "GPU_EXPORTER_BINDS=127.0.0.1,%s" "$handler"
+    grep -q 'systemctl --user enable gpu-exporter.service' "$handler"
+    grep -q 'systemctl --user restart gpu-exporter.service' "$handler"
+    # Health gate waits on the loopback metrics endpoint.
+    grep -q 'http://127.0.0.1:9401/metrics' "$handler"
+    # Dispatched from the central case.
+    grep -q 'gpu-exporter) deploy_gpu_exporter ;;' "$handler"
+    # Allowlisted on BOTH hosts.
+    grep -q 'dankclaw:gpu-exporter' "$handler"
+    grep -q 'dankserver:gpu-exporter' "$handler"
+}
+
+@test "the planner selects gpu-exporter for its files on each target" {
+    local t f
+    for t in dankclaw dankserver; do
+        for f in scripts/gpu-exporter.py scripts/gpu-exporter.service; do
+            run python3 "$REPO_ROOT/scripts/deploy-plan.py" --target "$t" --paths "$f"
+            [ "$status" -eq 0 ]
+            [ "$output" = "gpu-exporter" ] || { echo "$t/$f -> '$output'"; return 1; }
+        done
+    done
+}
+
+@test "Prometheus scrapes gpu-exporter where node/cadvisor are reachable on each host" {
+    # Cross-file contract, same posture as node-exporter: the local instance is
+    # reached via the podman gateway (this Prometheus is a bridge container and
+    # a host-process bind is refused at the host LAN IP); the Dankserver
+    # instance is a different machine, so plain LAN routing reaches it. If either
+    # address drifts from the exporter's bind, GPU panels go blank with no error.
+    local cfg="$REPO_ROOT/shared/prometheus/prometheus.yml"
+    grep -q 'job_name: "gpu-dankclaw"' "$cfg" || { echo "no gpu-dankclaw job"; return 1; }
+    grep -q 'job_name: "gpu-dankserver"' "$cfg" || { echo "no gpu-dankserver job"; return 1; }
+    grep -q '"host.containers.internal:9401"' "$cfg" \
+        || { echo "gpu-dankclaw does not scrape the podman gateway"; return 1; }
+    grep -q '"192.168.0.235:9401"' "$cfg" \
+        || { echo "gpu-dankserver does not scrape the LAN address"; return 1; }
+    # A vhost target would be red forever (Prometheus cannot solve Anubis).
+    ! grep -qE '"[^"]*franksdank[^"]*:9401"' "$cfg"
+}
+
+@test "the host-system dashboard graphs the gpu_* families the exporter emits" {
+    local dash="$REPO_ROOT/shared/grafana/provisioning/dashboards/host-system.json"
+    [ -f "$dash" ]
+    python3 -c "
+import json
+d = json.load(open('$dash'))
+exprs = [t['expr'] for p in d['panels'] for t in p.get('targets', [])]
+for want in ('gpu_utilization_percent', 'gpu_memory_used_bytes',
+             'gpu_temperature_celsius', 'gpu_power_watts'):
+    assert any(want in e for e in exprs), ('missing', want, exprs)
+print('gpu dashboard panels ok')
+"
+}
+
+@test "--serve answers on EVERY declared bind, not just the first" {
+    # The deployed unit serves loopback + the host LAN address, and Prometheus
+    # reaches the host through the gateway — which dials the LAN-class bind.
+    # A server that only runs serve_forever() on the first socket leaves the
+    # second one listen()ing-but-never-accepting: the deploy health gate on
+    # loopback passes, every scrape on the other bind dies on the SYN backlog,
+    # and no gate but this one sees it. (This exact bug shipped 2026-10-21 and
+    # left both gpu-* targets down beside a green deploy.)
+    mk_amd 0 42 1073741824 34359738368 "" 71000
+    port=$((19400 + RANDOM % 500))
+    python3 "$EXPORTER" --serve --amdgpu-sysfs "$SYS" --nvidia-cmd false \
+        --bind 127.0.0.1 --bind 127.0.0.2 --port "$port" \
+        > "$BATS_TEST_TMPDIR/serve.log" 2>&1 &
+    local pid=$!
+    trap 'kill "$pid" 2>/dev/null' EXIT
+    local ok=0
+    for _ in $(seq 1 20); do
+        curl -fsS -m2 "http://127.0.0.1:$port/metrics" >/dev/null 2>&1 && { ok=1; break; }
+        sleep 0.25
+    done
+    [ "$ok" -eq 1 ] || { kill "$pid" 2>/dev/null; return 1; }
+    # Both binds must ANSWER within 2 s each — the second one is the bug.
+    for b in 127.0.0.1 127.0.0.2; do
+        curl -fsS -m2 "http://$b:$port/metrics" | grep -q '^gpu_utilization_percent{' \
+            || { echo "bind $b did not answer /metrics"; kill "$pid" 2>/dev/null; return 1; }
+    done
+    kill "$pid" 2>/dev/null
+    trap - EXIT
+}

tests/observability_quadlet.bats

@@ -21,7 +21,7 @@ load helpers
 
 UNIT_DIR="$REPO_ROOT/dankclaw/observability/quadlet"
 REMOTE_DEPLOY="$REPO_ROOT/scripts/deploy-remote.sh"
-UNITS=(prometheus grafana loki alloy node-exporter alertmanager)
+UNITS=(prometheus grafana loki alloy node-exporter cadvisor alertmanager blackbox-exporter)
 
 # container_section <unit> — the [Container] key/value block only
 container_section() {
@@ -33,7 +33,7 @@ service_section() {
     awk '/^\[Service\]/{f=1;next} /^\[/{f=0} f' "$1"
 }
 
-@test "all six observability units exist under dankclaw/observability/quadlet/" {
+@test "all eight observability units exist under dankclaw/observability/quadlet/" {
     local svc
     for svc in "${UNITS[@]}"; do
         [ -f "$UNIT_DIR/$svc.container" ] || { echo "missing $svc.container"; return 1; }
@@ -277,6 +277,7 @@ loki http://127.0.0.1:3100/ready
 alloy http://127.0.0.1:12345/-/healthy
 node-exporter http://127.0.0.1:9100/metrics
 alertmanager http://127.0.0.1:9093/-/healthy
+cadvisor http://127.0.0.1:8080/healthz
 PAIRS
 }
 
@@ -558,6 +559,64 @@ assert any('ALERTS' in e for e in exprs), exprs
     grep -q -- '--dport 18093 -j DROP' "$handler"
 }
 
+@test "the deployer installs the relay firewall with the ACCEPTs deciding before the DROP" {
+    # Behavioral, run for real against a sandbox that plays the kernel: the
+    # rule set is a LIST and iptables -I prepends, so the shape this function
+    # had until 2026-10-21 (-C/-I ACCEPT x3, then -C/-I the blanket DROP)
+    # landed the DROP at position 1 and every packet died there — the drain
+    # failed its health gate beside a LISTENing socket twice. Seeding the
+    # sandbox with that poisoned order also pins the convergent repair: a
+    # re-deploy must rewrite the tail, not skip because -C finds every rule.
+    local sandbox="$BATS_TEST_TMPDIR/fw"
+    mkdir -p "$sandbox/bin"
+    cat > "$sandbox/bin/iptables" <<'EOF'
+#!/usr/bin/env bash
+# ordered rule list in $FW_STATE: -C lookup, -I at head, -A at tail, -D delete.
+state="${FW_STATE:?}"
+touch "$state"
+op="$1"; shift
+case "$op" in
+    -C) grep -Fxq -- "$*" "$state" ;;
+    -I) grep -Fxq -- "$*" "$state" || { { echo "$*"; cat "$state"; } > "$state.new"; mv "$state.new" "$state"; } ;;
+    -A) grep -Fxq -- "$*" "$state" || echo "$*" >> "$state" ;;
+    -D) if grep -Fxq -- "$*" "$state"; then
+            grep -Fxv -- "$*" "$state" > "$state.new"; mv "$state.new" "$state"
+        else exit 1; fi ;;
+    *) exit 0 ;;
+esac
+EOF
+    chmod +x "$sandbox/bin/iptables"
+    # Poisoned state exactly as the pre-fix deployer left Dankclaw.
+    printf '%s\n' \
+        'INPUT -p tcp --dport 18093 -j DROP' \
+        'INPUT -p tcp --dport 18093 -s 10.89.0.0/16 -j ACCEPT' \
+        'INPUT -p tcp --dport 18093 -s 192.168.0.143 -j ACCEPT' \
+        'INPUT -p tcp --dport 18093 -s 127.0.0.1 -j ACCEPT' \
+        > "$sandbox/rules"
+    (
+        cd "$REPO_ROOT"
+        export PATH="$sandbox/bin:$PATH" FW_STATE="$sandbox/rules" HOME="$sandbox/home"
+        mkdir -p "$HOME"
+        eval "$(sed -n '/^install_signal_relay()/,/^}/p' scripts/deploy-remote.sh)"
+        run_privileged() { "$@"; }
+        systemctl() { :; }
+        curl() { return 0; }
+        sleep() { :; }
+        install_signal_relay
+    )
+    # Exactly one blanket DROP, and it sits after every ACCEPT.
+    [ "$(grep -cF -- 'INPUT -p tcp --dport 18093 -j DROP' "$sandbox/rules")" -eq 1 ]
+    local drop_at total
+    drop_at="$(grep -nF -- 'INPUT -p tcp --dport 18093 -j DROP' "$sandbox/rules" | cut -d: -f1)"
+    total="$(wc -l < "$sandbox/rules")"
+    [ "$drop_at" -eq "$total" ]
+    local s seen
+    for s in 127.0.0.1 192.168.0.143 10.89.0.0/16; do
+        seen="$(grep -nF -- "-s $s -j ACCEPT" "$sandbox/rules" | cut -d: -f1)"
+        [ -n "$seen" ] && [ "$seen" -lt "$drop_at" ]
+    done
+}
+
 @test "the planner selects alertmanager for its config and relay files" {
     run python3 "$REPO_ROOT/scripts/deploy-plan.py" --target dankclaw \
         --paths shared/prometheus/alertmanager.yml
@@ -568,3 +627,200 @@ assert any('ALERTS' in e for e in exprs), exprs
     [ "$status" -eq 0 ]
     [ "$output" = "alertmanager" ]
 }
+
+@test "cadvisor exports container metrics only because of --cgroupns=host, and binds exactly its two consumers" {
+    # Podman starts every container in a private cgroup namespace, so a cadvisor
+    # without --cgroupns=host sees only its own cgroup root and exports a single
+    # series (measured 2026-10-20: 1 series before the flag, 996 after). That
+    # flag and the host /sys mount are the entire mechanism — pin them, plus the
+    # publish set: loopback (host probe + health gate) and this host's LAN
+    # address (Prometheus reaches a host-published port there). Never a wildcard.
+    local unit lan
+    for pair in "dankclaw/observability/quadlet:192.168.0.143" "dankserver/observability/quadlet:192.168.0.235"; do
+        unit="$REPO_ROOT/${pair%%:*}/cadvisor.container"
+        lan="${pair##*:}"
+        [ -f "$unit" ] || { echo "missing $unit"; return 1; }
+        grep -q -- '--cgroupns=host' "$unit" || { echo "$unit lacks --cgroupns=host"; return 1; }
+        grep -q -- '--pid=host' "$unit" || { echo "$unit lacks --pid=host"; return 1; }
+        grep -q '^Volume=/sys:/sys:ro$' "$unit" || { echo "$unit lacks the host /sys mount"; return 1; }
+        grep -q '^WantedBy=default.target$' "$unit" || { echo "$unit is not boot-persistent"; return 1; }
+        grep -q '^Restart=always$' "$unit" || { echo "$unit is not self-healing"; return 1; }
+        grep -q '^HealthCmd=' "$unit" || { echo "$unit has no health check"; return 1; }
+        grep -q "^PublishPort=127.0.0.1:8080:8080\$" "$unit" || { echo "$unit must publish loopback :8080"; return 1; }
+        grep -q "^PublishPort=$lan:8080:8080\$" "$unit" || { echo "$unit must publish $lan:8080"; return 1; }
+        ! grep -qE '^PublishPort=(0\.0\.0\.0|:)' "$unit" || { echo "$unit publishes on a wildcard"; return 1; }
+    done
+}
+
+@test "Prometheus scrapes cadvisor where it is published and relabels the cgroup path to a container name" {
+    # Cross-file contract, measured 2026-10-21 from inside the Prometheus
+    # container: a bridge container cannot dial its OWN host's LAN IP — the
+    # hairpin refuses published ports exactly like host-net binds, so
+    # cadvisor-dankclaw must scrape via the podman gateway (the original pin
+    # here demanded the LAN address and shipped a permanently-down target).
+    # dankserver is a different machine: plain LAN there. And the whole
+    # dashboard depends on the id -> container relabel: cadvisor names nothing
+    # itself, the quadlet unit name is only in the cgroup path. If either half
+    # changes alone, the Containers dashboard is blank.
+    local cfg="$REPO_ROOT/shared/prometheus/prometheus.yml"
+    grep -q 'job_name: "cadvisor-dankclaw"' "$cfg" || { echo "no cadvisor-dankclaw job"; return 1; }
+    grep -q 'job_name: "cadvisor-dankserver"' "$cfg" || { echo "no cadvisor-dankserver job"; return 1; }
+    grep -q '"host.containers.internal:8080"' "$cfg" \
+        || { echo "cadvisor-dankclaw does not scrape via the podman gateway"; return 1; }
+    ! grep -q '"192.168.0.143:8080"' "$cfg" \
+        || { echo "cadvisor-dankclaw dials its own host LAN IP — hairpin refuses it"; return 1; }
+    grep -q '"192.168.0.235:8080"' "$cfg" || { echo "cadvisor-dankserver does not scrape the published LAN port"; return 1; }
+    # The relabel that turns …/app.slice/<name>.service… into a `container`.
+    grep -q 'app\\.slice/(\[\^\.\]+)\\.service' "$cfg" || { echo "no cgroup-path -> container relabel"; return 1; }
+    grep -q 'target_label: container' "$cfg" || { echo "relabel does not set the container label"; return 1; }
+}
+
+@test "the Containers dashboard is provisioned from git" {
+    local dash="$REPO_ROOT/shared/grafana/provisioning/dashboards/containers.json"
+    [ -f "$dash" ]
+    python3 -c "
+import json
+d = json.load(open('$dash'))
+assert d['uid'] == 'containers', d['uid']
+names = [t['expr'] for p in d['panels'] for t in p.get('targets', [])]
+# Only families cadvisor actually exports for rootless quadlet containers are
+# graphed; network/fs are absent for them, so a network/fs panel would be a
+# permanent empty lie.
+assert any('container_cpu_usage_seconds_total' in e for e in names), names
+assert any('container_memory_working_set_bytes' in e for e in names), names
+assert not any('container_network' in e or 'container_fs_' in e for e in names), names
+"
+}
+
+@test "the planner selects cadvisor for its unit on each target" {
+    run python3 "$REPO_ROOT/scripts/deploy-plan.py" --target dankclaw \
+        --paths dankclaw/observability/quadlet/cadvisor.container
+    [ "$status" -eq 0 ]
+    [ "$output" = "cadvisor" ]
+    run python3 "$REPO_ROOT/scripts/deploy-plan.py" --target dankserver \
+        --paths dankserver/observability/quadlet/cadvisor.container
+    [ "$status" -eq 0 ]
+    [ "$output" = "cadvisor" ]
+}
+
+@test "the grafana copier provisions EVERY dashboard json, not a hand-listed subset" {
+    # Behavioral, not a text grep: the copier is extracted and executed with a
+    # stubbed run_privileged against a sandbox. The fleet provider turns each
+    # file in its directory into a dashboard, so a file the copier skips is a
+    # dashboard that silently never exists (the containers board lived in the
+    # repo, never reached /opt, and its fixes were undeployable).
+    # BATS_TEST_TMPDIR: per-test, removed by bats itself; a mktemp dir with a
+    # RETURN trap races bats' own trap machinery and vanishes mid-function.
+    local sandbox="$BATS_TEST_TMPDIR/sandbox"
+    (
+        cd "$REPO_ROOT"
+        eval "$(sed -n '/^copy_grafana_provisioning()/,/^}/p' scripts/deploy-remote.sh)"
+        run_privileged() { "$@"; }
+        export GRAFANA_PROV_ROOT="$sandbox/opt/grafana"
+        copy_grafana_provisioning
+    )
+    local f
+    for f in "$REPO_ROOT"/shared/grafana/provisioning/dashboards/*.json; do
+        [ -f "$sandbox/opt/grafana/provisioning/dashboards/$(basename "$f")" ]
+    done
+    [ -f "$sandbox/opt/grafana/provisioning/dashboards/provider.yml" ]
+    [ -f "$sandbox/opt/grafana/provisioning/datasources/prometheus.yml" ]
+}
+
+@test "every observability unit declares an Image" {
+    # Quadlet refuses to GENERATE a service for a .container without Image=
+    # - the deployer then dies at `systemctl restart` with "Unit not found"
+    # and no diagnosis (blackbox-exporter, live 2026-10-21: merged with the
+    # copier, gates and health check wired, but no Image line).
+    local svc
+    for svc in "${UNITS[@]}"; do
+        grep -q '^Image=' "$UNIT_DIR/$svc.container" \
+            || { echo "$svc.container has no Image= - quadlet will not generate it"; return 1; }
+    done
+}
+
+@test "changes under shared/blackbox/ plan the prober" {
+    run python3 "$REPO_ROOT/scripts/deploy-plan.py" --target dankclaw \
+        --paths shared/blackbox/blackbox.yml
+    [ "$status" -eq 0 ]
+    [ "$output" = "blackbox-exporter" ]
+}
+
+@test "the copier runs the same code path the deployer runs" {
+    # Textually pinned so a refactor cannot quietly route the deployer around
+    # the copy_* functions the seams execute.
+    grep -q '        prometheus)' "$REMOTE_DEPLOY"
+    grep -q 'copy_prometheus_config' "$REMOTE_DEPLOY"
+    grep -q 'copy_alertmanager_config' "$REMOTE_DEPLOY"
+    grep -q 'copy_loki_config' "$REMOTE_DEPLOY"
+    grep -q '        blackbox-exporter)' "$REMOTE_DEPLOY"
+    grep -q 'copy_blackbox_config' "$REMOTE_DEPLOY"
+}
+
+@test "the alloy copier installs the host-labeled config, token-free" {
+    # Executes copy_alloy_config for real against a sandbox (stubbed
+    # run_privileged). A surviving GRAFANA_HOST_LABEL token would ship a
+    # config whose host label is a literal placeholder — the exact class of
+    # silent-label defect this tranche retired the board for.
+    local sandbox="$BATS_TEST_TMPDIR/opt"
+    (
+        cd "$REPO_ROOT"
+        eval "$(sed -n '/^copy_alloy_config()/,/^}/p' scripts/deploy-remote.sh)"
+        run_privileged() { "$@"; }
+        # NOTE: VAR=x func does not bind for shell functions — export.
+        export TARGET=dankclaw ALLOY_CONF_ROOT="$sandbox"
+        copy_alloy_config
+    )
+    [ -f "$sandbox/config/config.alloy" ]
+    grep -q 'host = "dankclaw",' "$sandbox/config/config.alloy"
+    ! grep -q "GRAFANA_HOST_LABEL" "$sandbox/config/config.alloy"
+}
+
+@test "the blackbox copier installs the prober config" {
+    local sandbox="$BATS_TEST_TMPDIR/blackbox"
+    (
+        cd "$REPO_ROOT"
+        eval "$(sed -n '/^copy_blackbox_config()/,/^}/p' scripts/deploy-remote.sh)"
+        run_privileged() { "$@"; }
+        export BLACKBOX_CONF_ROOT="$sandbox"
+        copy_blackbox_config
+    )
+    [ -f "$sandbox/blackbox.yml" ]
+    grep -q '^  http_2xx:' "$sandbox/blackbox.yml"
+}
+
+@test "the alloy copier refuses to install a config the engine rejects" {
+    # The syntax gate: alloy attributes are position-sensitive and a
+    # misplaced one only explodes at run time (crash-loop behind a green
+    # deploy). Prove the copier refuses a config the pinned image won't
+    # validate, and installs nothing. Needs the alloy image; skipped
+    # where it is absent (the copier's gate is likewise image-gated).
+    if ! podman image exists localhost/franksdank/alloy:latest; then
+        skip "franksdank/alloy image not present on this box"
+    fi
+    local sandbox="$BATS_TEST_TMPDIR/opt" cfg="$BATS_TEST_TMPDIR/broken.alloy"
+    sed 's/GRAFANA_HOST_LABEL/dankclaw/g' shared/alloy/config.alloy > "$cfg"
+    printf 'bogus_attr = "x"\n' >> "$cfg"
+    ! (
+        cd "$REPO_ROOT"
+        eval "$(sed -n '/^copy_alloy_config()/,/^}/p' scripts/deploy-remote.sh)"
+        run_privileged() { "$@"; }
+        cp() { if [ "$1" = "$cfg" ]; then return 1; fi; command cp "$@"; }
+        export TARGET=dankclaw ALLOY_CONF_ROOT="$sandbox"
+        copy_alloy_config
+    )
+    [ ! -f "$sandbox/config/config.alloy" ]
+}
+
+@test "the installed alloy config is engine-valid" {
+    # Runs the pinned image's own validator over the shipped config. The
+    # 2026-10-21 defect (`on_error` on the labels stage, which alloy 1.21
+    # does not define) passed every textual seam and only failed at boot.
+    if ! podman image exists localhost/franksdank/alloy:latest; then
+        skip "franksdank/alloy image not present on this box"
+    fi
+    local cfg="$BATS_TEST_TMPDIR/rendered.alloy"
+    sed 's/GRAFANA_HOST_LABEL/dankclaw/g' shared/alloy/config.alloy > "$cfg"
+    podman run --rm -v "$cfg:/c.alloy:ro" --entrypoint /usr/bin/alloy \
+        localhost/franksdank/alloy:latest validate /c.alloy
+}

tests/open_webui_quadlet.bats

@@ -15,8 +15,9 @@ QDIR="$REPO_ROOT/dankclaw/open-webui/quadlet"
 }
 
 @test "open-webui uses the CPU image, not :cuda (dankclaw is CPU-only)" {
-    grep -qE '^Image=ghcr\.io/open-webui/open-webui:main$' "$QDIR/open-webui.container" \
-        || { echo "open-webui.container must use the :main (CPU) image"; return 1; }
+    # 2026-10-21 fleet law: tag AND digest (tests/quadlet_image_pins.bats)
+    grep -qE '^Image=ghcr\.io/open-webui/open-webui:main@sha256:[0-9a-f]{64}$' "$QDIR/open-webui.container" \
+        || { echo "open-webui.container must use the :main (CPU) image, digest-pinned"; return 1; }
     if grep -qE '^Image=.*:cuda' "$QDIR/open-webui.container"; then
         echo "open-webui.container must not use the :cuda image on a CPU-only host"
         return 1

tests/quadlet_image_pins.bats

@@ -0,0 +1,47 @@
+#!/usr/bin/env bats
+# Seam: fleet-wide quadlet laws learned the hard way.
+#   1. Every registry image is digest-pinned. A rolling tag (":latest",
+#      ":9-alpine", ":nightly") lets a rebuild move under a running
+#      service at the next restart/reboot with nobody approving it, and it
+#      makes a CVE audit ungradeable (you cannot score a tag). Locally
+#      built images (localhost/*, franksdank/*) are exempt: their build
+#      IS the pin, and they never pull.
+#   2. No unit carries a key the podman 5.7.0 quadlet generator rejects.
+#      The generator aborts the WHOLE batch on an unsupported key, so one
+#      bad file silently un-generates every other unit in that manager -
+#      live 2026-10-21: HostAddr= in prometheus.container orphaned
+#      prometheus.service while its container kept running, and the next
+#      deploy died on "Unit prometheus.service not found".
+
+load helpers
+
+@test "every registry Image= is digest-pinned across all three hosts" {
+    bad=""
+    while IFS= read -r unit; do
+        while IFS= read -r img; do
+            case "$img" in
+                localhost/*|franksdank/*) continue ;;
+            esac
+            case "$img" in
+                *@sha256:*) ;;
+                *) bad="$bad$unit: $img"$'\n' ;;
+            esac
+        done < <(sed -n 's/^Image=//p' "$unit")
+    done < <(find "$REPO_ROOT"/dankserver "$REPO_ROOT"/dankclaw "$REPO_ROOT"/frankbook-pro \
+                 -path '*/quadlet/*.container' -not -path '*.AGENTTRASH*')
+    [ -z "$bad" ] || { printf 'unpinned images:\n%s' "$bad" >&2; return 1; }
+}
+
+@test "no unit uses a key the 5.7.0 quadlet generator rejects (batch killer)" {
+    # HostAddr rejected by podman 5.7.0 (live 2026-10-21); Device= and
+    # Command= are the older v1 findings, guarded host-locally too.
+    # Keep this list current with every generator finding.
+    bad=""
+    while IFS= read -r unit; do
+        for key in HostAddr Device Command; do
+            grep -q "^${key}=" "$unit" && bad="$bad$unit: ${key}="$'\n'
+        done
+    done < <(find "$REPO_ROOT"/dankserver "$REPO_ROOT"/dankclaw "$REPO_ROOT"/frankbook-pro \
+                 -path '*/quadlet/*.container' -not -path '*.AGENTTRASH*')
+    [ -z "$bad" ] || { printf 'generator-rejected keys:\n%s' "$bad" >&2; return 1; }
+}

tests/qwen38_campaign.bats

@@ -0,0 +1,527 @@
+#!/usr/bin/env bats
+# Seam: the Qwen3.8-Flash-Next five-bit campaign (feat/qwen38-w5nl64).
+#
+# The campaign converts the vendor BF16 checkpoint into a w5nl64a8h .rad on
+# dankserver while frank-archon keeps serving. Every gate that protects the box
+# or the result lives in a script under scripts/, and each gate is tested here
+# through that script's own exit status and printed output — no engine, no GPU,
+# no network. Fixtures are synthetic; the expected numbers are worked examples,
+# not values recomputed the way the script computes them.
+#
+# Seams under test (the campaign's public boundaries):
+#   scripts/download-qwen38-bf16.sh      fetch/verify of the BF16 checkpoint
+#   scripts/grams-to-imatrix.py          calibration recording -> imatrix GGUF
+#   scripts/derive-protected-experts.py  --target-count sweep + the parity law
+#   scripts/build-qwen38-bf16-rad.sh     the reference container's convert argv
+#   scripts/run-qwen38-bf16.sh           the drained-serve argv, mode selection
+#   scripts/convert-qwen38-w5nl64.sh     the arm driver's refusal gates
+#   scripts/kld-gate-qwen38.sh           the three-anchor interval law
+#   scripts/deploy-plan.py               campaign files must deploy nothing
+
+load helpers
+
+DOWNLOAD="$REPO_ROOT/scripts/download-qwen38-bf16.sh"
+PLANNER="$REPO_ROOT/scripts/deploy-plan.py"
+
+# make_shard <dir> <name> <total-bytes> — a REAL safetensors file whose size on
+# disk is exactly <total-bytes>: 8-byte header length, JSON header, tensor data.
+# Real, because the verifier reads each shard's own header to learn how many
+# bytes it claims to be, which is the only per-file truth available offline.
+make_shard() {
+    mkdir -p "$1"
+    python3 - "$1/$2" "$3" <<'PY'
+import json, struct, sys
+path, want = sys.argv[1], int(sys.argv[2])
+payload = max(0, want - 96)
+header = b""
+for _ in range(16):
+    # The header is part of the file, so its own length moves the target: solve
+    # payload = want - 8 - len(header(payload)) by fixed-point iteration.
+    header = json.dumps({"w": {"dtype": "F32", "shape": [payload // 4],
+                               "data_offsets": [0, payload]}},
+                        separators=(",", ":")).encode()
+    want_payload = want - 8 - len(header)
+    if want_payload == payload:
+        break
+    if want_payload < 0:
+        raise SystemExit("fixture too small to carry a header")
+    payload = want_payload
+else:
+    raise SystemExit("fixture header never converged")
+with open(path, "wb") as f:
+    f.write(struct.pack("<Q", len(header)) + header + b"\x00" * payload)
+PY
+}
+
+# make_index <dir> <shard>... — a safetensors index that references them
+make_index() {
+    local dir="$1"; shift
+    python3 - "$dir/model.safetensors.index.json" "$@" <<'PY'
+import json, sys
+out, names = sys.argv[1], sys.argv[2:]
+json.dump({"metadata": {"total_size": 0},
+           "weight_map": {f"model.layers.{i}.attn.q.weight": n
+                          for i, n in enumerate(names)}},
+          open(out, "w"))
+PY
+}
+
+# three_fixture <dir> — three valid 1000-byte shards plus the index naming them
+three_fixture() {
+    make_shard "$1" model-00001-of-00003.safetensors 1000
+    make_shard "$1" model-00002-of-00003.safetensors 1000
+    make_shard "$1" model-00003-of-00003.safetensors 1000
+    make_index "$1" model-00001-of-00003.safetensors \
+                  model-00002-of-00003.safetensors \
+                  model-00003-of-00003.safetensors
+}
+
+# --- download / verify -------------------------------------------------------
+
+@test "the checkpoint verifier accepts a complete checkpoint and reports its footprint" {
+    C="$BATS_TEST_TMPDIR/complete"
+    three_fixture "$C"
+    # 3 x 1000 bytes of shards. The index file itself is deliberately not
+    # counted, so a Hub-side edit to a config file cannot fail this gate.
+    run bash "$DOWNLOAD" --no-hub --dir "$C" --shards 3 --bytes 3000
+    [ "$status" -eq 0 ]
+    [[ "$output" == *"VERIFY_OK"*"3 shards"*"3000 bytes"* ]]
+}
+
+@test "the verifier names the shard whose own header disagrees with its size on disk" {
+    C="$BATS_TEST_TMPDIR/truncated"
+    three_fixture "$C"
+    # Cut the tail, keep the header: that is a torn transfer, which is what the
+    # box actually gets from an interrupted fetch. Zero-filling instead would
+    # destroy the header and be caught as "not a safetensors file" — a different
+    # gate, and it would prove nothing about the size check.
+    python3 -c 'import sys
+p = sys.argv[1]
+keep = open(p, "rb").read(400)
+open(p, "wb").write(keep)' "$C/model-00002-of-00003.safetensors"
+    run bash "$DOWNLOAD" --no-hub --dir "$C" --shards 3 --bytes 3000
+    [ "$status" -eq 1 ]
+    [[ "$output" == *"TRUNCATED"*"model-00002-of-00003.safetensors"* ]]
+}
+
+@test "the verifier refuses a missing shard and names it" {
+    C="$BATS_TEST_TMPDIR/missing"
+    three_fixture "$C"
+    mv "$C/model-00003-of-00003.safetensors" "$C/hidden.bin"
+    run bash "$DOWNLOAD" --no-hub --dir "$C" --shards 3 --bytes 3000
+    [ "$status" -eq 1 ]
+    [[ "$output" == *"MISSING"*"model-00003-of-00003.safetensors"* ]]
+}
+
+@test "the verifier refuses when the shard census disagrees with the expected count" {
+    # The fragment law this repo learned the hard way: large quants ship as
+    # .partNofM fragments and a naive glob reports success on a partial tree.
+    C="$BATS_TEST_TMPDIR/count"
+    three_fixture "$C"
+    make_shard "$C" model-00004-of-00004.safetensors 1000
+    run bash "$DOWNLOAD" --no-hub --dir "$C" --shards 3 --bytes 3000
+    [ "$status" -eq 1 ]
+    [[ "$output" == *"SHARD_COUNT"*"4"*"3"* ]]
+}
+
+@test "the verifier refuses a shard the index names that never landed" {
+    C="$BATS_TEST_TMPDIR/orphan-index"
+    three_fixture "$C"
+    rm "$C/model-00003-of-00003.safetensors"
+    make_index "$C" model-00001-of-00003.safetensors \
+                  model-00002-of-00003.safetensors \
+                  model-00003-of-00003.safetensors
+    run bash "$DOWNLOAD" --no-hub --dir "$C" --shards 2 --bytes 2000
+    [ "$status" -eq 1 ]
+    [[ "$output" == *"MISSING"*"model-00003-of-00003.safetensors"* ]]
+}
+
+@test "the verifier treats an HTTP error body written as a shard as a failure, not a download" {
+    # curl -o FILE exits 0 on 401/403 and writes the error body; three shards
+    # of identical tiny size must never look like a checkpoint.
+    C="$BATS_TEST_TMPDIR/errorbody"
+    mkdir -p "$C"
+    for n in 1 2 3; do
+        printf '{"error":"You are not logged in"}' \
+            > "$C/model-0000${n}-of-00003.safetensors"
+    done
+    make_index "$C" model-00001-of-00003.safetensors \
+                  model-00002-of-00003.safetensors \
+                  model-00003-of-00003.safetensors
+    run bash "$DOWNLOAD" --no-hub --dir "$C" --shards 3 --bytes 3000
+    [ "$status" -eq 1 ]
+    [[ "$output" == *"NOT_SAFETENSORS"* ]]
+}
+
+@test "the verifier refuses a shard census whose bytes do not total what the Hub declared" {
+    # Every shard header can be self-consistent and the tree still be the wrong
+    # payload: the total is the cross-check against the Hub's declared size.
+    C="$BATS_TEST_TMPDIR/wrongtotal"
+    three_fixture "$C"
+    run bash "$DOWNLOAD" --no-hub --dir "$C" --shards 3 --bytes 2500
+    [ "$status" -eq 1 ]
+    [[ "$output" == *"TOTAL_BYTES"*"3000"*"2500"* ]]
+}
+
+@test "the fetch path refuses when the destination cannot hold the checkpoint" {
+    # The campaign's peak-disk law starts at the fetch: the BF16 checkpoint and
+    # the reference container built from it are both ~335 GiB, and the tree that
+    # does not fit is discovered hours into a transfer.
+    C="$BATS_TEST_TMPDIR/tiny"
+    three_fixture "$C"
+    run env FRANK_QWEN38_MIN_FREE_MIB=999999999 \
+        bash "$DOWNLOAD" --fetch --no-hub --dir "$C" --shards 3 --bytes 3000
+    [ "$status" -eq 2 ]
+    [[ "$output" == *"REFUSED"*"free"* ]]
+}
+
+# --- deployment scoping ------------------------------------------------------
+
+@test "the campaign tooling deploys nothing: merging it cannot drain frank-archon" {
+    # Existence is asserted first: a planner that plans nothing because the
+    # files vanished would otherwise pass.
+    [ -f "$REPO_ROOT/scripts/download-qwen38-bf16.sh" ]
+    run python3 "$PLANNER" --target dankserver --paths \
+        scripts/download-qwen38-bf16.sh \
+        scripts/build-qwen38-bf16-rad.sh \
+        scripts/run-qwen38-bf16.sh \
+        scripts/convert-qwen38-w5nl64.sh \
+        scripts/kld-gate-qwen38.sh \
+        scripts/grams-to-imatrix.py \
+        scripts/recipes/qwen4exp-qwen38-w5nl64-bf16trunk.recipe \
+        tests/qwen38_campaign.bats
+    [ "$status" -eq 0 ]
+    [ -z "$output" ]
+}
+
+# --- protected experts -------------------------------------------------------
+
+DERIVE="$REPO_ROOT/scripts/derive-protected-experts.py"
+APPLY="$REPO_ROOT/scripts/apply-protected-cast.sh"
+CONVERT="$REPO_ROOT/scripts/convert-qwen38-w5nl64.sh"
+GATE="$REPO_ROOT/scripts/kld-gate-qwen38.sh"
+TEMPLATE="$REPO_ROOT/scripts/recipes/qwen4exp-qwen38-w5nl64-bf16trunk.recipe"
+
+# make_gram <dir> <layer> <expert> <energy> <rows> — a RADGRAM2 down-Gram whose
+# trace (its down-input energy) is <energy> and whose routing count is <rows>.
+make_gram() {
+    python3 - "$@" <<'PY'
+import os, struct, sys
+d, layer, e, energy, rows = sys.argv[1:6]
+os.makedirs(d, exist_ok=True)
+hdr = struct.pack("<8sIIQI4x", b"RADGRAM2", 2, 0, int(rows), 0)
+mat = struct.pack("<4f", float(energy), 0.0, 0.0, 0.0)
+with open(os.path.join(d, f"gram.r0.blk.{layer}.ffn_down_exps.{e}.bin"), "wb") as f:
+    f.write(hdr + mat)
+PY
+}
+
+# eight_grams <dir> — one layer, eight experts inside rank half 0, energies
+# 60/15/10/8/4/2/1/0.5 against equal routing. Under the default bars (3% of the
+# layer's energy AND 3x its routing share) exactly ONE expert is disproportionate,
+# so the parity law has to hand it a partner.
+eight_grams() {
+    local d="$1"
+    make_gram "$d" 0 0 60 100
+    make_gram "$d" 0 1 15 100
+    make_gram "$d" 0 2 10 100
+    make_gram "$d" 0 3 8 100
+    make_gram "$d" 0 4 4 100
+    make_gram "$d" 0 5 2 100
+    make_gram "$d" 0 6 1 100
+    make_gram "$d" 0 7 0.5 100
+}
+
+cast_count() { grep -c 'cast dtype=bf16' <<< "$output"; }
+
+@test "the bar-only pass keeps the disproportionate expert and repairs the odd rank half" {
+    C="$BATS_TEST_TMPDIR/calib"
+    eight_grams "$C"
+    run python3 "$DERIVE" "$C"
+    [ "$status" -eq 0 ]
+    # e0 holds 59.7% of the layer's energy on 12.5% of the routing: the only expert
+    # that clears three times its routing share. e1 joins it because half 0 was odd.
+    [ "$(cast_count)" -eq 2 ]
+    [[ "$output" == *"blk.0.ffn_*_exps.0.weight"* ]]
+    [[ "$output" == *"blk.0.ffn_*_exps.1.weight"* ]]
+    [[ "$output" != *"ffn_*_exps.2.weight"* ]]
+    [[ "$output" == *"half 0 odd -> added e1"* ]]
+}
+
+@test "--target-count lands the byte bill that was asked for, and never buys more" {
+    # The count is a disk and VRAM decision (each protected expert costs bf16 twice,
+    # once per rank half's copy), so landing it matters as much as not missing it.
+    C="$BATS_TEST_TMPDIR/calib"
+    eight_grams "$C"
+    run python3 "$DERIVE" "$C" --target-count 6
+    [ "$status" -eq 0 ]
+    [ "$(cast_count)" -eq 6 ]
+    [[ "$output" == *"count fill -> added e2"* ]]
+    [[ "$output" != *"ffn_*_exps.6.weight"* ]]
+    # the energy share the set bottoms out at is the bar the campaign paid for
+    [[ "$output" == *"total protected: 6 (target 6, energy cutoff"* ]]
+}
+
+@test "--target-count cannot invent experts: the pool's ceiling is reported, not padded" {
+    C="$BATS_TEST_TMPDIR/calib"
+    eight_grams "$C"
+    run python3 "$DERIVE" "$C" --target-count 20
+    [ "$status" -eq 0 ]
+    [ "$(cast_count)" -eq 8 ]
+    [[ "$output" == *"total protected: 8 (target 20"* ]]
+}
+
+@test "a protected set that cannot be made even is refused instead of emitted" {
+    # One disproportionate expert in half 0, its only companion in half 1: no
+    # candidate could make half 0 even, and an odd half hands the four-bit GEMM two
+    # tables that are not equal. Refusing is the only honest answer.
+    C="$BATS_TEST_TMPDIR/calib"
+    make_gram "$C" 0 0 100 100
+    make_gram "$C" 0 300 1 1000
+    run python3 "$DERIVE" "$C"
+    [ "$status" -eq 2 ]
+    [[ "$output" == *"PARITY LAW BROKEN"* ]]
+    [[ "$output" == *"blk.0 half 0 holds 1"* ]]
+    [ "$(cast_count)" -eq 0 ]
+}
+
+@test "the splice rewrites the run's copy above the expert rule and refuses a second run" {
+    C="$BATS_TEST_TMPDIR/calib"
+    eight_grams "$C"
+    W="$BATS_TEST_TMPDIR/recipe-work.recipe"
+    cp "$TEMPLATE" "$W"
+    run env PY=python3 TRASH_DIR="$BATS_TEST_TMPDIR/trash" bash "$APPLY" "$C" "$W" 0
+    [ "$status" -eq 0 ]
+    grep -q "derive-protected-experts.py rewrote" "$W"
+    # The derived lines must precede the expert rule: the first rule whose glob
+    # matches a weight decides it, so a cast line below the expert line never fires.
+    cast_line=$(grep -n 'cast dtype=bf16' "$W" | head -1 | cut -d: -f1)
+    expert_line=$(grep -n 'ffn_\*_exps\.\*\.weight' "$W" | head -1 | cut -d: -f1)
+    [ "$cast_line" -lt "$expert_line" ]
+    # The committed template stays the clean, reviewable baseline.
+    ! grep -q "derive-protected-experts.py rewrote" "$TEMPLATE"
+    # A second run on the same copy would derive from a recording it already spent.
+    run env PY=python3 TRASH_DIR="$BATS_TEST_TMPDIR/trash" bash "$APPLY" "$C" "$W" 0
+    [ "$status" -ne 0 ]
+    [[ "$output" == *"refusing to re-derive"* ]]
+}
+
+@test "the splice refuses before touching the recipe when the count is not reached" {
+    C="$BATS_TEST_TMPDIR/calib"
+    eight_grams "$C"
+    W="$BATS_TEST_TMPDIR/recipe-work.recipe"
+    cp "$TEMPLATE" "$W"
+    run env PY=python3 TRASH_DIR="$BATS_TEST_TMPDIR/trash" bash "$APPLY" "$C" "$W" 20
+    [ "$status" -eq 1 ]
+    [[ "$output" == *"target was 20"* ]]
+    ! grep -q "derive-protected-experts.py rewrote" "$W"
+}
+
+# --- the arm driver ----------------------------------------------------------
+
+# drv_fixture <dir> — the smallest tree the driver's gates accept: a checkpoint
+# with its index and tokenizer, three grams, an imatrix inside the campaign dir
+# the container mounts, and an empty output dir.
+drv_fixture() {
+    local d="$1"
+    mkdir -p "$d/ck/Qwen3.8-Flash-Next" "$d/campaign" "$d/calib" "$d/out"
+    : > "$d/ck/Qwen3.8-Flash-Next/model.safetensors.index.json"
+    : > "$d/ck/Qwen3.8-Flash-Next/tokenizer.json"
+    make_gram "$d/calib" 0 0 60 100
+    make_gram "$d/calib" 0 1 15 100
+    make_gram "$d/calib" 0 2 10 100
+    # four experts, not three: a thin pool that cannot be made even trips the parity
+    # audit, and this fixture is meant to reach the later gates
+    make_gram "$d/calib" 0 3 8 100
+    : > "$d/campaign/imatrix.gguf"
+}
+
+# run_drv <driver args and K=V overrides, any order> — the driver with every path
+# pointed at the fixture. Override words are separated out because `env` only
+# accepts assignments ahead of the command; later assignments win, so a test can
+# break exactly one input.
+run_drv() {
+    local d="$BATS_TEST_TMPDIR/drv" assigns=() flags=() w
+    for w in "$@"; do
+        if [[ "$w" =~ ^[A-Za-z_][A-Za-z0-9_]*= ]]; then assigns+=("$w"); else flags+=("$w"); fi
+    done
+    run env CKPT="$d/ck/Qwen3.8-Flash-Next" CAMPAIGN="$d/campaign" STATE="$d/campaign" \
+        CALIB="$d/calib" OUTDIR="$d/out" IMATRIX="$d/campaign/imatrix.gguf" \
+        MIN_GRAMS=2 TARGET_COUNT=0 ${assigns[@]+"${assigns[@]}"} \
+        bash "$CONVERT" ${flags[@]+"${flags[@]}"}
+}
+
+@test "the dry run prints the plan and then the convert for every arm" {
+    drv_fixture "$BATS_TEST_TMPDIR/drv"
+    run_drv --dry-run
+    [ "$status" -eq 0 ]
+    [[ "$output" == *"== arm cd3 =="* ]]
+    [[ "$output" == *"== arm cd6 =="* ]]
+    [[ "$output" == *"== arm ao1 =="* ]]
+    # one --plan-only bill per arm, and one convert line per arm beside it
+    [ "$(grep -c -- '--plan-only' <<< "$output")" -eq 3 ]
+    [ "$(grep -c 'rad-convert' <<< "$output")" -eq 6 ]
+    awk '/arm cd3/ {next}
+         /qwen38-w5nl64-cd3\.rad/ { if ($0 ~ /--plan-only/) p = NR; else c = NR }
+         END { exit !(p && c && p < c) }' <<< "$output"
+}
+
+@test "the cd6 arm re-sends the whole expert option set with cd=6; the control sends none" {
+    # --quant replaces the recipe line wholesale, so an override that forgot an
+    # option would quietly convert with the default for it.
+    drv_fixture "$BATS_TEST_TMPDIR/drv"
+    run_drv --arm cd3 --arm cd6 --dry-run
+    [ "$status" -eq 0 ]
+    cd6line=$(grep -- '-cd6\.rad --quant' <<< "$output" | head -1)
+    [[ "$cd6line" == *"table=w5nl"* ]]
+    [[ "$cd6line" == *"group=64"* ]]
+    [[ "$cd6line" == *"scale=fp8_e4m3"* ]]
+    [[ "$cd6line" == *"transform=fwht128"* ]]
+    [[ "$cd6line" == *"calib=\$CALIB"* ]]
+    [[ "$cd6line" == *",cd=6"* ]]
+    cd3line=$(grep -- '-cd3\.rad' <<< "$output" | grep -v -- '--plan-only' | head -1)
+    [[ "$cd3line" != *"--quant"* ]]
+}
+
+@test "the ao1 arm carries act_order=1 and no cd=, because libquant refuses the pair" {
+    drv_fixture "$BATS_TEST_TMPDIR/drv"
+    run_drv --arm ao1 --dry-run
+    [ "$status" -eq 0 ]
+    line=$(grep -- '-ao1\.rad --quant' <<< "$output" | head -1)
+    [[ "$line" == *"act_order=1"* ]]
+    [[ "$line" != *",cd="* ]]
+}
+
+@test "the driver splices the protected plane into its own copy, never the template" {
+    drv_fixture "$BATS_TEST_TMPDIR/drv"
+    run_drv --dry-run
+    [[ "$output" == *"apply-protected-cast.sh $BATS_TEST_TMPDIR/drv/calib "* ]]
+    [[ "$output" == *"/campaign/recipe-"* ]]
+    [[ "$output" != *"$TEMPLATE"* ]]
+}
+
+@test "the driver refuses while the BF16 reference container still holds the volume" {
+    drv_fixture "$BATS_TEST_TMPDIR/drv"
+    : > "$BATS_TEST_TMPDIR/drv/out/qwen38-bf16.rad"
+    run_drv --dry-run
+    [ "$status" -eq 2 ]
+    [[ "$output" == *"peak-disk"* ]]
+}
+
+@test "a thin recording is refused: the driver never converts uncalibrated" {
+    # rad-convert converts WITHOUT a calibration recording with one stderr note,
+    # which is silently worse than a GPTQ run that failed.
+    drv_fixture "$BATS_TEST_TMPDIR/drv"
+    run_drv MIN_GRAMS=9 --dry-run
+    [ "$status" -eq 2 ]
+    [[ "$output" == *"run-qwen38-bf16.sh calib"* ]]
+}
+
+@test "an imatrix outside the mounted campaign dir is refused before any convert" {
+    drv_fixture "$BATS_TEST_TMPDIR/drv"
+    mv "$BATS_TEST_TMPDIR/drv/campaign/imatrix.gguf" "$BATS_TEST_TMPDIR/imatrix.gguf"
+    run_drv IMATRIX="$BATS_TEST_TMPDIR/imatrix.gguf" --dry-run
+    [ "$status" -eq 2 ]
+    [[ "$output" == *"not in the campaign dir"* ]]
+}
+
+@test "the driver never overwrites an arm that already exists" {
+    drv_fixture "$BATS_TEST_TMPDIR/drv"
+    : > "$BATS_TEST_TMPDIR/drv/out/qwen38-w5nl64-cd6.rad"
+    run_drv --arm cd6 --dry-run
+    [ "$status" -eq 2 ]
+    [[ "$output" == *"move it to trash first"* ]]
+}
+
+@test "an unknown arm name is refused before the gates run" {
+    # No fixture built: the name check must not need a checkpoint to exist.
+    run_drv --arm cd9 --dry-run
+    [ "$status" -eq 2 ]
+    [[ "$output" == *"unknown arm: cd9"* ]]
+}
+
+@test "the driver stops when the protected derivation cannot land the target" {
+    drv_fixture "$BATS_TEST_TMPDIR/drv"
+    run_drv PY=python3 TARGET_COUNT=20 --arm cd3
+    [ "$status" -eq 1 ]
+    [[ "$output" == *"target was 20"* ]]
+    [[ "$output" == *"would convert unprotected"* ]]
+    [[ "$output" != *"--entrypoint"* ]]
+}
+
+# --- the acceptance gate -----------------------------------------------------
+
+# kld_report <path> <model> <kl mean> <top1 fraction> — one score report, in the
+# shape `run-qwen38-bf16.sh score` writes: an `all` summary plus a per-source one.
+kld_report() {
+    python3 - "$@" <<'PY'
+import json, sys
+path, model, kl, top1 = sys.argv[1:5]
+kl, top1 = float(kl), float(top1)
+group = {"kld": {"mean": kl, "p99_9": kl * 4}, "top1_agreement": top1, "positions": 50432}
+json.dump({"model": model, "all": group,
+           "by_source": {"ops": dict(group, positions=20000)}}, open(path, "w"))
+PY
+}
+
+gate_case() { # gate_case <kl fp8> <kl iq4r> <kl cand> <top1 cand> [extra flags...]
+    D="$BATS_TEST_TMPDIR/kl"
+    mkdir -p "$D"
+    kld_report "$D/fp8.json" fp8-native "$1" 0.981
+    kld_report "$D/iq4r.json" iq4r-moe "$2" 0.962
+    kld_report "$D/cand.json" w5nl64 "$3" "$4"
+    run bash "$GATE" "$D/fp8.json" "$D/iq4r.json" "$D/cand.json" "${@:5}"
+}
+
+@test "an arm strictly inside the interval that closes half the gap passes" {
+    gate_case 0.0021 0.0190 0.0090 0.971
+    [ "$status" -eq 0 ]
+    [[ "$output" == *"closes 59.2%"* ]]
+    [[ "$output" == *"KLD GATE PASS"* ]]
+}
+
+@test "an arm worse than the served payload fails the interval and the recovery" {
+    gate_case 0.0021 0.0190 0.0210 0.971
+    [ "$status" -eq 1 ]
+    [[ "$output" == *"[FAIL] 2."* ]]
+    [[ "$output" == *"[FAIL] 3."* ]]
+    [[ "$output" == *"KLD GATE FAIL"* ]]
+}
+
+@test "anchors that are not ordered stop the gate: the instrument is wrong, not the arm" {
+    # If the near-lossless FP8 release does not sit CLOSER to the bf16 reference
+    # than the 4-bit payload does, the corpus, tokenizer or boot shape is broken and
+    # no arm number can mean anything.
+    gate_case 0.0300 0.0190 0.0120 0.971
+    [ "$status" -eq 1 ]
+    [[ "$output" == *"[FAIL] 1."* ]]
+    [[ "$output" == *"gap is not positive"* ]]
+}
+
+@test "an arm measured outside the interval fails even when it looks better" {
+    # Closer to the reference than the achievable anchor is a measurement fault.
+    gate_case 0.0021 0.0190 0.0010 0.971
+    [ "$status" -eq 1 ]
+    [[ "$output" == *"[FAIL] 2."* ]]
+}
+
+@test "an arm that buys KL with top-1 agreement fails" {
+    gate_case 0.0021 0.0190 0.0090 0.940
+    [ "$status" -eq 1 ]
+    [[ "$output" == *"[PASS] 3."* ]]
+    [[ "$output" == *"[FAIL] 4."* ]]
+}
+
+@test "--recover tightens the same numbers until they fail" {
+    gate_case 0.0021 0.0190 0.0090 0.971 --recover 0.8
+    [ "$status" -eq 1 ]
+    [[ "$output" == *"needs 80%"* ]]
+    [[ "$output" == *"[FAIL] 3."* ]]
+}
+
+@test "a slice the report does not carry is named, not guessed at" {
+    gate_case 0.0021 0.0190 0.0090 0.971 --slice code
+    [ "$status" -eq 2 ]
+    [[ "$output" == *"has no slice 'code'"* ]]
+    [[ "$output" == *"it has: ops"* ]]
+}

tests/qwen38_grams.bats

@@ -0,0 +1,445 @@
+#!/usr/bin/env bats
+# Seam: scripts/grams-to-imatrix.py — the calibration recording (RADGRAM2 grams)
+# folded into the GGUF v3 imatrix `rad-convert --imatrix F` reads.
+#
+# The only contract that matters is the reader's: core/format/imatrix.cpp joins
+# `<base>.in_sum2` with `<base>.counts`, takes cols from in_sum2 dims[0] and
+# n_expert from dims[1], demands ggml f32 tensors and GGUF version 3, and
+# silently IGNORES an entry whose width does not match the weight it is asked
+# about. So every test here parses the artifact back with its own tiny GGUF
+# reader and asserts bytes — a file that merely exists and exits 0 proves
+# nothing about a quantiser that will read it hours later on another machine.
+#
+# Fixtures are synthetic: k=4 grams with real RADGRAM2 headers (rows is a u64
+# at offset 16 — pack it as u32 and 100 rows read back as 4294967296000) and
+# real f32 payloads whose OFF-DIAGONAL is filled with 99.0, so an entry built
+# from anything but the diagonal is instantly visible. Expected numbers are
+# worked examples from the fixture tables below, never recomputed the way the
+# tool computes them.
+
+load helpers
+
+TOOL="$REPO_ROOT/scripts/grams-to-imatrix.py"
+
+# make_gram <path> <k> <rank> <rows> <domain> <diag,csv>
+# A RADGRAM2 file: 32-byte header <8sIIQI (magic, k, rank, rows, domain, pad)
+# then k*k f32 row-major. Off-diagonals are 99.0 — a sentinel, not data.
+make_gram() {
+    python3 - "$@" <<'PY'
+import struct, sys
+path, k, rank, rows, domain = sys.argv[1], int(sys.argv[2]), int(sys.argv[3]), \
+    int(sys.argv[4]), int(sys.argv[5])
+diag = [float(x) for x in sys.argv[6].split(",")]
+assert len(diag) == k, "diagonal must hold k values"
+vals = [99.0] * (k * k)
+for i, d in enumerate(diag):
+    vals[i * k + i] = d
+hdr = struct.pack("<8sIIQI", b"RADGRAM2", k, rank, rows, domain) + struct.pack("<I", 0)
+with open(path, "wb") as f:
+    f.write(hdr + struct.pack("<%df" % (k * k), *vals))
+PY
+}
+
+# full_recording <dir> — the fixture every happy-path test reads.
+#
+# layer 0 (k=4, own-gram floor 4*k = 16 rows):
+#   pooled   r0 rows 40 diag 1,2,3,4   + r1 rows 8 diag .5,.5,.5,.5
+#            -> summed gram (the engine's calib_sum law): diag 1.5,2.5,3.5,4.5,
+#               pooled rows 48
+#   expert 0 r0 rows 100 diag 10,20,30,40   (>= 16: own gram kept)
+#   expert 1 r0 rows   3 diag 7,7,7,7       (<  16: inherits pooled)
+#   expert 2 r1 rows  64 diag 1,1,1,2       (>= 16: own gram kept)
+#   expert 3           no file at all       (inherits pooled, count 0)
+#   expert 4 r1 rows 100 diag 5,6,7,8       (>= 16: own gram kept)
+#   -> n_expert 5, counts 100 3 64 0 100, inherited 2, routing 267
+# layer 1: pooled r0 rows 48 diag 2,2,2,2 ; expert 0 on r1 rows 48 diag 1,1,1,1
+#   -> n_expert 1, counts 48, inherited 0, routing 48
+# Corpus rows = the largest single-layer pooled rows = 48.
+full_recording() {
+    local d="$1"
+    mkdir -p "$d"
+    make_gram "$d/gram.r0.blk.0.ffn_down_exps.bin" 4 0 40 0 "1,2,3,4"
+    make_gram "$d/gram.r1.blk.0.ffn_down_exps.bin" 4 1 8 0 "0.5,0.5,0.5,0.5"
+    make_gram "$d/gram.r0.blk.0.ffn_down_exps.0.bin" 4 0 100 0 "10,20,30,40"
+    make_gram "$d/gram.r0.blk.0.ffn_down_exps.1.bin" 4 0 3 0 "7,7,7,7"
+    make_gram "$d/gram.r1.blk.0.ffn_down_exps.2.bin" 4 1 64 0 "1,1,1,2"
+    make_gram "$d/gram.r1.blk.0.ffn_down_exps.4.bin" 4 1 100 0 "5,6,7,8"
+    make_gram "$d/gram.r0.blk.1.ffn_down_exps.bin" 4 0 48 0 "2,2,2,2"
+    make_gram "$d/gram.r1.blk.1.ffn_down_exps.0.bin" 4 1 48 0 "1,1,1,1"
+}
+
+# gguf_read <file> [vals <tensor> <float-index> <count>] — an independent GGUF
+# v3 reader: KV block, tensor infos, and (on request) the f32 values behind a
+# tensor's own declared offset. Prints KV/HDR/TENSOR lines. Nothing here shares
+# code with the writer, which is the only reason the byte assertions mean
+# anything: imatrix.cpp walks the same three structures.
+gguf_read() {
+    python3 - "$@" <<'PY'
+import struct, sys
+d = open(sys.argv[1], "rb").read()
+def u32(o): return struct.unpack_from("<I", d, o)[0]
+def u64(o): return struct.unpack_from("<Q", d, o)[0]
+def st(o):
+    n = u64(o)
+    return d[o + 8:o + 8 + n].decode(), o + 8 + n
+assert d[:4] == b"GGUF", "magic is %r" % d[:4]
+ver, n_t, n_kv = u32(4), u64(8), u64(16)
+p, align, out = 24, 32, []
+for _ in range(n_kv):
+    key, p = st(p)
+    ty = u32(p); p += 4
+    if ty == 4:                                   # uint32
+        v = u32(p); p += 4
+        if key == "general.alignment":
+            align = v
+    elif ty == 8:                                 # string
+        v, p = st(p)
+    elif ty == 9:                                 # array
+        et = u32(p); n = u64(p + 4); p += 12
+        got = []
+        for _i in range(n):
+            if et != 8:
+                raise SystemExit("kv array element type %d" % et)
+            s, p = st(p)
+            got.append(s)
+        v = ",".join(got)
+    else:
+        raise SystemExit("kv type %d is not steppable" % ty)
+    out.append("KV %s %s" % (key, v))
+tens = []
+for _ in range(n_t):
+    name, p = st(p)
+    nd = u32(p); p += 4
+    dims = [u64(p + 8 * i) for i in range(nd)]; p += 8 * nd
+    ty = u32(p); off = u64(p + 4); p += 12
+    tens.append((name, dims, ty, off))
+ds = (p + align - 1) // align * align
+out.append("HDR version=%d n_tensors=%d n_kv=%d alignment=%d data_start=%d"
+           % (ver, n_t, n_kv, align, ds))
+for name, dims, ty, off in tens:
+    out.append("TENSOR %s %s %d %d" % (name, "x".join(str(x) for x in dims), ty, off))
+if len(sys.argv) > 2 and sys.argv[2] == "vals":
+    name, i0, n = sys.argv[3], int(sys.argv[4]), int(sys.argv[5])
+    for nm, dims, ty, off in tens:
+        if nm == name:
+            base = ds + off
+            vals = [("%d" % v) if float(v).is_integer() else repr(v)
+                    for v in (struct.unpack_from("<f", d, base + 4 * (i0 + i))[0]
+                              for i in range(n))]
+            out.append("VALS %s %s" % (name, " ".join(vals)))
+print("\n".join(out))
+PY
+}
+
+# aligned_offsets <file> — exit 0 iff the data section starts aligned AND every
+# tensor's absolute data offset (data_start + t.off) is a multiple of
+# general.alignment. This is the writer bug that no value assertion can see: an
+# unaligned offset makes the reader's extent arithmetic land mid-float in the
+# previous tensor, and it reads the result without complaining.
+aligned_offsets() {
+    python3 - "$1" <<'PY'
+import struct, sys
+d = open(sys.argv[1], "rb").read()
+def u32(o): return struct.unpack_from("<I", d, o)[0]
+def u64(o): return struct.unpack_from("<Q", d, o)[0]
+def st(o):
+    n = u64(o)
+    return d[o + 8:o + 8 + n].decode(), o + 8 + n
+ver, n_t, n_kv = u32(4), u64(8), u64(16)
+p, align = 24, 32
+for _ in range(n_kv):
+    key, p = st(p)
+    ty = u32(p); p += 4
+    if ty == 4:
+        v = u32(p); p += 4
+        if key == "general.alignment":
+            align = v
+    if ty == 8:
+        _v, p = st(p)
+    elif ty == 9:
+        et = u32(p); n = u64(p + 4); p += 12
+        for _i in range(n):
+            if et == 8:
+                _v, p = st(p)
+            elif et == 4:
+                p += 4
+            else:
+                raise SystemExit("kv array element type %d" % et)
+    elif ty in (0, 1):
+        p += 1
+    elif ty in (2, 3):
+        p += 2
+    elif ty in (5, 6):
+        p += 4
+    elif ty in (10, 11, 12):
+        p += 8
+tens = []
+for _ in range(n_t):
+    name, p = st(p)
+    nd = u32(p); p += 4
+    p += 8 * nd
+    ty = u32(p); off = u64(p + 4); p += 12
+    tens.append((name, ty, off))
+ds = (p + align - 1) // align * align
+bad = [] if ds % align == 0 else ["data_start=%d" % ds]
+bad += ["%s=%d" % (nm, ds + off) for nm, _ty, off in tens if (ds + off) % align]
+if bad:
+    print("UNALIGNED", ";".join(bad))
+    sys.exit(1)
+print("ALIGNED %d tensor(s), data_start=%d, alignment=%d" % (len(tens), ds, align))
+PY
+}
+
+# --- the artifact -----------------------------------------------------------
+
+@test "a complete recording becomes a GGUF v3 file and the run reports itself" {
+    C="$BATS_TEST_TMPDIR/full"
+    full_recording "$C"
+    run python3 "$TOOL" "$C" -o "$BATS_TEST_TMPDIR/im.gguf" \
+        --datasets calib,chat --chunk-size 16
+    [ "$status" -eq 0 ]
+    [[ "$output" == *"IMATRIX_OK"* ]]
+    head -c 4 "$BATS_TEST_TMPDIR/im.gguf" | grep -q "GGUF"
+    run gguf_read "$BATS_TEST_TMPDIR/im.gguf"
+    [[ "$output" == *"HDR version=3"* ]]
+    # The reader refuses anything but version 3, so this is the whole gate.
+    [[ "$output" != *"version=2"* ]]
+    # Everything the reader looks for by key, and the honesty of the chunk math:
+    # 48 corpus rows over 16-row chunks is 3 chunks.
+    [[ "$output" == *"KV general.alignment 32"* ]]
+    [[ "$output" == *"KV imatrix.chunk_size 16"* ]]
+    [[ "$output" == *"KV imatrix.chunk_count 3"* ]]
+    [[ "$output" == *"KV imatrix.datasets calib,chat"* ]]
+    [[ "$output" == *"n_tensors=12"* ]]
+}
+
+@test "the reader finds the folded MoE entry: one per layer, expert-major, f32" {
+    C="$BATS_TEST_TMPDIR/full"
+    full_recording "$C"
+    run python3 "$TOOL" "$C" -o "$BATS_TEST_TMPDIR/im.gguf"
+    [ "$status" -eq 0 ]
+    run gguf_read "$BATS_TEST_TMPDIR/im.gguf"
+    # dims (k, n_expert) with k fastest, ggml type 0 (f32) or the entry is
+    # warned away, and 5 experts because the fixture's largest global id is 4.
+    [[ "$output" == *"TENSOR blk.0.ffn_down_exps.in_sum2 4x5 0 "* ]]
+    [[ "$output" == *"TENSOR blk.0.ffn_down_exps.counts 1x5 0 "* ]]
+    # counts are the routing rows, not the energy trace: 100 3 64 0 100.
+    run gguf_read "$BATS_TEST_TMPDIR/im.gguf" vals blk.0.ffn_down_exps.counts 0 5
+    [[ "$output" == *"VALS blk.0.ffn_down_exps.counts 100 3 64 0 100"* ]]
+    # Expert 0 (floats 0..3) and expert 4 (floats 16..19) carry their own
+    # matrices' diagonals; the 99.0 off-diagonal fill appears nowhere.
+    run gguf_read "$BATS_TEST_TMPDIR/im.gguf" vals blk.0.ffn_down_exps.in_sum2 0 4
+    [[ "$output" == *"VALS blk.0.ffn_down_exps.in_sum2 10 20 30 40"* ]]
+    run gguf_read "$BATS_TEST_TMPDIR/im.gguf" vals blk.0.ffn_down_exps.in_sum2 16 4
+    [[ "$output" == *"VALS blk.0.ffn_down_exps.in_sum2 5 6 7 8"* ]]
+    [[ "$output" != *"99"* ]]
+}
+
+@test "an expert with no own gram carries the pooled diagonal, and the count of them is reported" {
+    C="$BATS_TEST_TMPDIR/full"
+    full_recording "$C"
+    run python3 "$TOOL" "$C" -o "$BATS_TEST_TMPDIR/im.gguf"
+    [ "$status" -eq 0 ]
+    [[ "$output" == *"inherited=2"* ]]
+    # Expert 1 has a gram (3 rows, under the 4*k floor) and expert 3 has none at
+    # all; both slices must be the POOLED diag 1+0.5, 2+0.5, 3+0.5, 4+0.5.
+    run gguf_read "$BATS_TEST_TMPDIR/im.gguf" vals blk.0.ffn_down_exps.in_sum2 4 4
+    [[ "$output" == *"VALS blk.0.ffn_down_exps.in_sum2 1.5 2.5 3.5 4.5"* ]]
+    run gguf_read "$BATS_TEST_TMPDIR/im.gguf" vals blk.0.ffn_down_exps.in_sum2 12 4
+    [[ "$output" == *"VALS blk.0.ffn_down_exps.in_sum2 1.5 2.5 3.5 4.5"* ]]
+    # Their counts stay their own routing: 3 for the under-threshold expert, 0
+    # for the unrecorded one — popularity is count's job, never the gram's.
+    run gguf_read "$BATS_TEST_TMPDIR/im.gguf" vals blk.0.ffn_down_exps.counts 1 3
+    [[ "$output" == *"VALS blk.0.ffn_down_exps.counts 3 64 0"* ]]
+    # The census is a report for the operator, so it belongs on stderr. `run`
+    # folds both streams into $output, so a second run into a different -o (an
+    # existing output is refused by design) captures stderr alone: without the
+    # stderr write there is no file to grep and this assertion goes red.
+    python3 "$TOOL" "$C" -o "$BATS_TEST_TMPDIR/im2.gguf" 2>"$BATS_TEST_TMPDIR/err"
+    [ -s "$BATS_TEST_TMPDIR/err" ]
+    grep -q "inherited=2" "$BATS_TEST_TMPDIR/err"
+}
+
+@test "the hidden-width entries are the same bytes, and nothing else is emitted" {
+    C="$BATS_TEST_TMPDIR/full"
+    full_recording "$C"
+    run python3 "$TOOL" "$C" -o "$BATS_TEST_TMPDIR/im.gguf"
+    [ "$status" -eq 0 ]
+    run gguf_read "$BATS_TEST_TMPDIR/im.gguf"
+    # gate_up shares the down matrices (same hidden-state rows), so its slice 4
+    # must read 5 6 7 8 at its own tensor info's offset.
+    [[ "$output" == *"TENSOR blk.0.ffn_gate_up_exps.in_sum2 4x5 0 "* ]]
+    [[ "$output" == *"TENSOR blk.0.ssm_inz.in_sum2 4 0 "* ]]
+    [[ "$output" == *"TENSOR blk.0.ssm_inz.counts 1 0 "* ]]
+    run gguf_read "$BATS_TEST_TMPDIR/im.gguf" vals blk.0.ffn_gate_up_exps.in_sum2 16 4
+    [[ "$output" == *"VALS blk.0.ffn_gate_up_exps.in_sum2 5 6 7 8"* ]]
+    run gguf_read "$BATS_TEST_TMPDIR/im.gguf" vals blk.0.ssm_inz.in_sum2 0 4
+    [[ "$output" == *"VALS blk.0.ssm_inz.in_sum2 1.5 2.5 3.5 4.5"* ]]
+    run gguf_read "$BATS_TEST_TMPDIR/im.gguf" vals blk.0.ssm_inz.counts 0 1
+    [[ "$output" == *"VALS blk.0.ssm_inz.counts 48"* ]]
+    # The stop rule: importance_for() false-on-a-width-mismatch means an entry
+    # for a projection whose input is NOT the gram's width is a silent lie.
+    [[ "$output" != *"ssm_out"* ]]
+    [[ "$output" != *"attn"* ]]
+    [[ "$output" != *"mtp."* ]]
+    [[ "$output" != *"ple_ngram"* ]]
+    [[ "$output" != *"output.weight"* ]]
+}
+
+@test "rows is a u64: a corpus too big for u32 survives the header" {
+    # The trap this fixture law exists for: pack rows as u32 at offset 16 and a
+    # 100-row file reads back as 4294967296000. 2**32 is exactly representable
+    # in f32, so the count round-trips only if the tool read 8 bytes.
+    C="$BATS_TEST_TMPDIR/wide"
+    mkdir -p "$C"
+    make_gram "$C/gram.r0.blk.0.ffn_down_exps.bin" 4 0 4294967296 0 "1,1,1,1"
+    make_gram "$C/gram.r0.blk.0.ffn_down_exps.0.bin" 4 0 4294967296 0 "3,3,3,3"
+    run python3 "$TOOL" "$C" -o "$BATS_TEST_TMPDIR/wide.gguf" --chunk-size 4096
+    [ "$status" -eq 0 ]
+    run gguf_read "$BATS_TEST_TMPDIR/wide.gguf" vals blk.0.ffn_down_exps.counts 0 1
+    [[ "$output" == *"VALS blk.0.ffn_down_exps.counts 4294967296"* ]]
+    [[ "$output" != *"4294967296000"* ]]
+    run gguf_read "$BATS_TEST_TMPDIR/wide.gguf"
+    [[ "$output" == *"KV imatrix.chunk_count 1048576"* ]]
+    # imatrix.chunk_count is a u32 field, so 2**32 rows in 1-row chunks has no
+    # legal encoding: wrapping it would put a number downstream believes.
+    run python3 "$TOOL" "$C" -o "$BATS_TEST_TMPDIR/wide2.gguf" --chunk-size 1
+    [ "$status" -eq 2 ]
+    [[ "$output" == *"REFUSED"*"CHUNK_COUNT"* ]]
+    [ ! -e "$BATS_TEST_TMPDIR/wide2.gguf" ]
+}
+
+@test "every tensor's data offset is a multiple of general.alignment" {
+    C="$BATS_TEST_TMPDIR/full"
+    full_recording "$C"
+    run python3 "$TOOL" "$C" -o "$BATS_TEST_TMPDIR/im.gguf"
+    [ "$status" -eq 0 ]
+    run aligned_offsets "$BATS_TEST_TMPDIR/im.gguf"
+    [ "$status" -eq 0 ]
+    [[ "$output" == "ALIGNED 12 tensor(s),"*"alignment=32" ]]
+
+    # Red proof, in the same test so the assertion cannot rot: the same file
+    # with one tensor's declared offset nudged by 4. A counts tensor is 4 bytes,
+    # so a writer that forgets to pad between tensors produces exactly this
+    # shape of file — and every value assertion above still passes on it.
+    python3 - "$BATS_TEST_TMPDIR/im.gguf" "$BATS_TEST_TMPDIR/unaligned.gguf" <<'PY'
+import struct, sys
+src, dst = sys.argv[1], sys.argv[2]
+d = bytearray(open(src, "rb").read())
+def u32(o): return struct.unpack_from("<I", d, o)[0]
+def u64(o): return struct.unpack_from("<Q", d, o)[0]
+def st(o):
+    n = u64(o)
+    return d[o + 8:o + 8 + n].decode(), o + 8 + n
+p = 24
+for _ in range(u64(16)):
+    key, p = st(p)
+    ty = u32(p); p += 4
+    if ty == 4:
+        p += 4
+    elif ty == 8:
+        _v, p = st(p)
+    elif ty == 9:
+        et = u32(p); n = u64(p + 4); p += 12
+        for _i in range(n):
+            _v, p = st(p)
+for _ in range(u64(8)):
+    name, p = st(p)
+    nd = u32(p); p += 4
+    p += 8 * nd
+    off_at = p + 4                      # the u64 offset field of this tensor
+    if name == "blk.0.ssm_inz.counts":
+        struct.pack_into("<Q", d, off_at, u64(off_at) + 4)
+    p += 12
+open(dst, "wb").write(bytes(d))
+PY
+    run aligned_offsets "$BATS_TEST_TMPDIR/unaligned.gguf"
+    [ "$status" -eq 1 ]
+    [[ "$output" == *"UNALIGNED"*"blk.0.ssm_inz.counts"* ]]
+}
+
+# --- the refusals -----------------------------------------------------------
+
+@test "a gram short by one float is refused and the file is named" {
+    C="$BATS_TEST_TMPDIR/short"
+    full_recording "$C"
+    # Cut the last 4 bytes of one expert gram: a torn write, which is what a
+    # killed tap actually leaves behind.
+    p="$C/gram.r0.blk.0.ffn_down_exps.0.bin"
+    python3 -c 'import sys
+p = sys.argv[1]
+open(p, "wb").write(open(p, "rb").read()[:-4])' "$p"
+    run python3 "$TOOL" "$C" -o "$BATS_TEST_TMPDIR/no.gguf"
+    [ "$status" -eq 2 ]
+    [[ "$output" == *"REFUSED"*"SHORT"* ]]
+    [[ "$output" == *"gram.r0.blk.0.ffn_down_exps.0.bin"* ]]
+    [ ! -e "$BATS_TEST_TMPDIR/no.gguf" ]
+}
+
+@test "an empty recording directory is refused" {
+    mkdir -p "$BATS_TEST_TMPDIR/empty"
+    run python3 "$TOOL" "$BATS_TEST_TMPDIR/empty" -o "$BATS_TEST_TMPDIR/no.gguf"
+    [ "$status" -eq 2 ]
+    [[ "$output" == *"REFUSED"*"NO_GRAMS"* ]]
+    [[ "$output" == *"$BATS_TEST_TMPDIR/empty"* ]]
+    [ ! -e "$BATS_TEST_TMPDIR/no.gguf" ]
+}
+
+@test "expert grams with no pooled gram for the layer are refused" {
+    # Without the pooled gram there is nothing for an under-threshold expert to
+    # inherit, and the tool would quietly invent a zero diagonal for it.
+    C="$BATS_TEST_TMPDIR/nopooled"
+    mkdir -p "$C"
+    make_gram "$C/gram.r0.blk.3.ffn_down_exps.7.bin" 4 0 500 0 "1,2,3,4"
+    run python3 "$TOOL" "$C" -o "$BATS_TEST_TMPDIR/no.gguf"
+    [ "$status" -eq 2 ]
+    [[ "$output" == *"REFUSED"*"NO_POOLED"*"layer 3"* ]]
+    [[ "$output" == *"gram.r<rank>.blk.3.ffn_down_exps.bin"* ]]
+}
+
+@test "the same global expert recorded twice with different numbers is refused" {
+    # Rank files carry GLOBAL expert numbering and so describe disjoint experts;
+    # a collision means the recording is corrupt, and whichever file sorted last
+    # would win.
+    C="$BATS_TEST_TMPDIR/conflict"
+    full_recording "$C"
+    make_gram "$C/gram.r1.blk.0.ffn_down_exps.0.bin" 4 1 77 0 "10,20,30,40"
+    run python3 "$TOOL" "$C" -o "$BATS_TEST_TMPDIR/no.gguf"
+    [ "$status" -eq 2 ]
+    [[ "$output" == *"REFUSED"*"EXPERT_CONFLICT"*"expert 0"* ]]
+    [[ "$output" == *"gram.r0.blk.0.ffn_down_exps.0.bin"* ]]
+    [[ "$output" == *"gram.r1.blk.0.ffn_down_exps.0.bin"* ]]
+}
+
+@test "an existing output is refused by name rather than overwritten" {
+    # The trash law: a re-run must not destroy the imatrix a convert is reading.
+    C="$BATS_TEST_TMPDIR/full"
+    full_recording "$C"
+    : > "$BATS_TEST_TMPDIR/keep.gguf"
+    printf 'do not clobber me\n' > "$BATS_TEST_TMPDIR/keep.gguf"
+    run python3 "$TOOL" "$C" -o "$BATS_TEST_TMPDIR/keep.gguf"
+    [ "$status" -eq 2 ]
+    [[ "$output" == *"REFUSED"*"EXISTS"*"keep.gguf"* ]]
+    run cat "$BATS_TEST_TMPDIR/keep.gguf"
+    [[ "$output" == *"do not clobber me"* ]]
+}
+
+@test "--min-rows moves the threshold and the bytes follow it" {
+    # The default threshold is the engine's own law (4*k rows, lq_hess.h), but
+    # the operator has to be able to raise it when a corpus is thin. Asserted on
+    # the slices, not on a report line: a flag that only changes prose is a
+    # flag that changes nothing.
+    C="$BATS_TEST_TMPDIR/floor"
+    full_recording "$C"
+    run python3 "$TOOL" "$C" -o "$BATS_TEST_TMPDIR/far.gguf" --min-rows 200
+    [ "$status" -eq 0 ]
+    # 200 is above every expert's routing (100, 3, 64, 0, 100 and layer 1's 48),
+    # so all six slices now inherit.
+    [[ "$output" == *"inherited=6"* ]]
+    run gguf_read "$BATS_TEST_TMPDIR/far.gguf" vals blk.0.ffn_down_exps.in_sum2 0 4
+    [[ "$output" == *"VALS blk.0.ffn_down_exps.in_sum2 1.5 2.5 3.5 4.5"* ]]
+    # The counts do not move with the threshold: routing is a measurement, not a
+    # judgement about how much data is enough.
+    run gguf_read "$BATS_TEST_TMPDIR/far.gguf" vals blk.0.ffn_down_exps.counts 0 5
+    [[ "$output" == *"VALS blk.0.ffn_down_exps.counts 100 3 64 0 100"* ]]
+}

tests/qwen38_reference.bats

@@ -0,0 +1,337 @@
+#!/usr/bin/env bats
+# Seam: the BF16 REFERENCE container of the Qwen3.8-Flash-Next five-bit
+# campaign (feat/qwen38-w5nl64) — scripts/build-qwen38-bf16-rad.sh and
+# scripts/run-qwen38-bf16.sh.
+#
+# The reference is what every later stage is judged or derived from (the KLD
+# record scores the arms, the Grams derive the protected experts), so the
+# deliverable here is the set of refusals: a half checkpoint, a second convert,
+# a volume that cannot hold the convert, an engine-refused plan, and any run —
+# including a dry one — that would take the GPUs from frank-archon. Each test
+# drives the script's own exit status and printed argv. Offline: no engine, no
+# GPU, no container, no systemctl, no network. Path knobs aim at
+# $BATS_TEST_TMPDIR.
+#
+# The third mode, score, carries the campaign's comparability law: its engine
+# argv must equal kldref's except for --model and the --kld-* pair, or the KL
+# numbers of the FP8-native anchor, the served payload and the arms are not one
+# measurement series. The seam proves that by diffing the two dry runs.
+
+# The 1.8.1 law asserted in two places below: --vram-kv-mib was REMOVED at
+# 1.3.0, so naming it is a boot refusal — the reference serve must carry the
+# budget-resolver knobs (--host-pool-mib / --gpu-headroom-mib) instead.
+
+load helpers
+
+BUILD="$REPO_ROOT/scripts/build-qwen38-bf16-rad.sh"
+RUNNER="$REPO_ROOT/scripts/run-qwen38-bf16.sh"
+
+# The reference shape, spelled exactly as run-qwen38-bf16.sh must pass it: this
+# is the contract the recorded reference is measured at, so a silent edit here
+# is a silent edit to every KLD number in the campaign.
+SHAPE="--tp 2 --placement expert_tiered --weights-disk-tier --host-pool-mib 49152 --gpu-headroom-mib 96 --deterministic --num-speculative-tokens 0 --kv-cache-dtype fp8 --max-model-len 32768 --ngram-placement ram --override-chat-template /models/tpl.jinja"
+
+# oneline — both dry runs print one shell-quoted argument a line; joining them
+# lets an option and its value be asserted as the pair they are.
+oneline() { local s="${output//$'\n'/ }"; printf '%s' "$s"; }
+
+# engine_argv — the arguments AFTER the image line of a dry run, with --model
+# and every --kld-* flag and its value removed: what is left is the measuring
+# shape, and it must be identical between kldref and score.
+engine_argv() {
+    local seen=0 skip=0 a
+    while IFS= read -r a; do
+        if [ "$seen" = 0 ]; then
+            [[ "$a" == *"radiance@sha256:"* ]] && seen=1
+            continue
+        fi
+        case "$a" in
+            --model|--kld-record|--kld-corpus|--kld-ref|--kld-out) skip=1; continue ;;
+        esac
+        if [ "$skip" = 1 ]; then skip=0; continue; fi
+        printf '%s\n' "$a"
+    done <<< "$1"
+}
+
+# flag_value FLAG — the value a dry run printed for FLAG (the argv is one
+# argument a line, so the value is the line after it). Lets a test ask what
+# number the script chose instead of re-declaring the literal it guards.
+flag_value() {
+    local want="$1" prev="" a
+    while IFS= read -r a; do
+        if [ "$prev" = "$want" ]; then printf '%s' "$a"; return 0; fi
+        prev="$a"
+    done
+    return 1
+}
+
+@test "the build dry run prints the rad-convert argv for the checkpoint and the bf16 output" {
+    M="$BATS_TEST_TMPDIR/models"
+    run env FRANK_QWEN38_MODELS="$M" bash "$BUILD" --dry-run
+    [ "$status" -eq 0 ]
+    [[ "$output" == *"rad-convert"* ]]
+    # the argv is the container's view: the models dir rides the /models mount
+    [[ "$output" == *"$M:/models"* ]]
+    flat="$(oneline)"
+    [[ "$flat" == *"/models/Qwen3.8-Flash-Next -o /models/rad/qwen38-bf16.rad"* ]]
+    # a reference is the checkpoint re-headed, not re-encoded, and it declares a
+    # window the long KLD corpus can be read inside of
+    [[ "$output" != *"--recipe"* ]]
+    [[ "$output" != *"--reuse"* ]]
+    [[ "$flat" == *"--max-ctx 32768"* ]]
+    # and a dry run looked at nothing: the fixture dirs were never created
+    [ ! -e "$M" ]
+}
+
+@test "the build refuses a checkpoint that is not on disk and names the fetch command" {
+    M="$BATS_TEST_TMPDIR/models"
+    run env FRANK_QWEN38_MODELS="$M" bash "$BUILD"
+    [ "$status" -eq 2 ]
+    [[ "$output" == *"$M/Qwen3.8-Flash-Next"* ]]
+    [[ "$output" == *"scripts/download-qwen38-bf16.sh --fetch"* ]]
+    [ ! -e "$M/rad/qwen38-bf16.rad" ]
+}
+
+@test "the build refuses an existing reference container and names --force as the way past it" {
+    M="$BATS_TEST_TMPDIR/models"
+    mkdir -p "$M/Qwen3.8-Flash-Next" "$M/rad"
+    echo "not a container" > "$M/rad/qwen38-bf16.rad"
+    run env FRANK_QWEN38_MODELS="$M" QWEN38_MIN_FREE_MIB=1 \
+        QWEN38_TRASH="$BATS_TEST_TMPDIR/trash" bash "$BUILD"
+    [ "$status" -eq 2 ]
+    [[ "$output" == *"$M/rad/qwen38-bf16.rad"* ]]
+    [[ "$output" == *"--force"* ]]
+    # refusing retires nothing: the trash law means the file survives a refusal
+    [ -f "$M/rad/qwen38-bf16.rad" ]
+    [ ! -e "$BATS_TEST_TMPDIR/trash" ]
+}
+
+@test "the build refuses when the output filesystem cannot hold a second 335 GiB" {
+    # The campaign's peak-disk law: 343,323 MiB of container written next to
+    # 343,323 MiB of checkpoint. The wanted figure is set above whatever this
+    # box really has, which is what the gate reads.
+    M="$BATS_TEST_TMPDIR/models"
+    mkdir -p "$M/Qwen3.8-Flash-Next" "$M/rad"
+    run env FRANK_QWEN38_MODELS="$M" QWEN38_MIN_FREE_MIB=999999999 bash "$BUILD"
+    [ "$status" -eq 2 ]
+    [[ "$output" == *"REFUSED"* ]]
+    [[ "$output" =~ [0-9]+" MiB free" ]]
+    [[ "$output" == *"999999999 MiB wanted"* ]]
+    # the disk gate is the cheap one: it refuses long before the Hub sweep runs
+    [[ "$output" != *"VERIFY_OK"* ]]
+    [ ! -e "$M/rad/qwen38-bf16.rad" ]
+}
+
+@test "the reference serve rejects an unknown mode and names every valid mode" {
+    # `probe` is the swift15 era's third mode; it is retired and must not boot
+    # a container here either.
+    run bash "$RUNNER" probe
+    [ "$status" -eq 2 ]
+    [[ "$output" == *"probe"* ]]
+    [[ "$output" == *"kldref"* ]]
+    [[ "$output" == *"calib"* ]]
+    [[ "$output" == *"score"* ]]
+    [[ "$output" != *"podman"* ]]
+    run bash "$RUNNER"
+    [ "$status" -eq 2 ]
+    [[ "$output" == *"kldref"* ]]
+    [[ "$output" == *"calib"* ]]
+    [[ "$output" == *"score"* ]]
+    # score without its two positionals is the same usage refusal
+    run bash "$RUNNER" score
+    [ "$status" -eq 2 ]
+    [[ "$output" == *"score RAD OUT"* ]]
+}
+
+@test "the kldref dry run carries the record flags at the reference shape and no --vram-kv-mib" {
+    run bash "$RUNNER" kldref --dry-run
+    [ "$status" -eq 0 ]
+    flat="$(oneline)"
+    [[ "$flat" == *"--kld-record /data/kld-ref"* ]]
+    [[ "$flat" == *"--kld-corpus /data/kld.jsonl"* ]]
+    # the corpus arrives read-only, the record dir writable
+    [[ "$flat" == *":/data/kld.jsonl:ro"* ]]
+    [[ "$flat" == *":/data/kld-ref "* ]]
+    [[ "$flat" == *"$SHAPE"* ]]
+    # 1.8.1: the flag does not exist — naming it is a boot refusal
+    [[ "$output" != *"--vram-kv-mib"* ]]
+}
+
+@test "the calib dry run opens the tap at 12M tokens and serves on 7770" {
+    run bash "$RUNNER" calib --dry-run
+    [ "$status" -eq 0 ]
+    flat="$(oneline)"
+    [[ "$flat" == *"-e RADIANCE_CALIB_TOKENS=12000000"* ]]
+    [[ "$flat" == *"-e RADIANCE_CALIB_DIR=/data/calib"* ]]
+    [[ "$flat" == *"--port 7770"* ]]
+    [[ "$flat" == *":/data/calib "* ]]
+    # calib serves; it does not record
+    [[ "$output" != *"--kld-record"* ]]
+}
+
+@test "a dry run prints argv only: no drain, no container, nothing written" {
+    L="$BATS_TEST_TMPDIR/logs"
+    run env FRANK_QWEN38_LOG_DIR="$L" \
+        bash "$RUNNER" kldref --dry-run --image example.invalid/radiance@sha256:0000000000000000000000000000000000000000000000000000000000000000
+    [ "$status" -eq 0 ]
+    [[ "$output" != *"systemctl"* ]]
+    [[ "$output" != *"frank-archon stopped"* ]]
+    [[ "$output" != *"podman rm"* ]]
+    # not vacuous: the podman argv WAS printed, and --image replaced the pin
+    [[ "$output" == *"podman"* ]]
+    [[ "$output" == *"example.invalid/radiance@sha256:000"* ]]
+    [[ "$output" != *"stilldeadcode/radiance"* ]]
+    [ ! -d "$L" ]
+}
+
+@test "the score dry run is the kldref argv with only --model and the --kld-* pair changed" {
+    M="$BATS_TEST_TMPDIR/models"
+    C="$BATS_TEST_TMPDIR/campaign"
+    run env FRANK_QWEN38_MODELS="$M" FRANK_QWEN38_CAMPAIGN="$C" \
+        bash "$RUNNER" kldref --dry-run
+    [ "$status" -eq 0 ]
+    ref_argv="$(engine_argv "$output")"
+    run env FRANK_QWEN38_MODELS="$M" FRANK_QWEN38_CAMPAIGN="$C" \
+        bash "$RUNNER" score "$M/rad/arm-w5nl64.rad" "$C/scores/arm1.json" --dry-run
+    [ "$status" -eq 0 ]
+    score_argv="$(engine_argv "$output")"
+    # The whole value of the campaign's three KL numbers is that they rode this
+    # one shape: same pin, --tp 2, fp8 KV, step size, --deterministic, no spec.
+    if [[ "$score_argv" != "$ref_argv" ]]; then
+        echo "--- kldref engine argv ---"; echo "$ref_argv"
+        echo "--- score engine argv ---"; echo "$score_argv"
+        return 1
+    fi
+    flat="$(oneline)"
+    [[ "$flat" == *"--model /models/rad/arm-w5nl64.rad"* ]]
+    [[ "$flat" == *"--kld-ref /data/kld-ref --kld-out /data/out/arm1.json"* ]]
+    # the zero point mounts read-only; the report needs a writable bind because
+    # --kld-out F also writes F.rows beside it
+    [[ "$flat" == *":/data/kld-ref:ro "* ]]
+    [[ "$flat" == *":/data/out "* ]]
+    [[ "$output" != *"--kld-record"* ]]
+    [[ "$output" != *"--kld-corpus"* ]]
+    [[ "$output" != *"--vram-kv-mib"* ]]
+}
+
+@test "score refuses a missing reference, a missing container and an existing report" {
+    M="$BATS_TEST_TMPDIR/models"
+    C="$BATS_TEST_TMPDIR/campaign"
+    L="$BATS_TEST_TMPDIR/logs"
+    mkdir -p "$M/rad"
+    : > "$M/rad/arm-w5nl64.rad"
+    run env FRANK_QWEN38_MODELS="$M" FRANK_QWEN38_CAMPAIGN="$C" FRANK_QWEN38_LOG_DIR="$L" \
+        bash "$RUNNER" score "$M/rad/arm-w5nl64.rad" "$C/scores/arm1.json"
+    [ "$status" -eq 2 ]
+    [[ "$output" == *"$C/kld-ref"* ]]
+    [[ "$output" == *"run-qwen38-bf16.sh kldref"* ]]
+    # refused before the log dir, so before anything that could reach systemctl
+    [ ! -d "$L" ]
+
+    mkdir -p "$C/kld-ref"
+    run env FRANK_QWEN38_MODELS="$M" FRANK_QWEN38_CAMPAIGN="$C" FRANK_QWEN38_LOG_DIR="$L" \
+        bash "$RUNNER" score "$M/rad/gone.rad" "$C/scores/arm1.json"
+    [ "$status" -eq 2 ]
+    [[ "$output" == *"$M/rad/gone.rad"* ]]
+    [ ! -d "$L" ]
+
+    mkdir -p "$C/scores"
+    : > "$C/scores/arm1.json"
+    run env FRANK_QWEN38_MODELS="$M" FRANK_QWEN38_CAMPAIGN="$C" FRANK_QWEN38_LOG_DIR="$L" \
+        bash "$RUNNER" score "$M/rad/arm-w5nl64.rad" "$C/scores/arm1.json"
+    [ "$status" -eq 2 ]
+    [[ "$output" == *"$C/scores/arm1.json"* ]]
+    [[ "$output" == *"trash"* ]]
+    # the trash law: a refusal keeps the old report, it does not overwrite it
+    [ -f "$C/scores/arm1.json" ]
+    [ ! -d "$L" ]
+}
+
+@test "the reference and a score pass ride ONE step size, legal on the file tier" {
+    # The gate's claim is one shape; that shape's ceiling is set by the tightest
+    # payload it boots, and the served iq4r and every arm are file-tier
+    # containers, which stream only at <=2048. A reference recorded above that is
+    # a KL between two shapes, not two weights.
+    M="$BATS_TEST_TMPDIR/models"
+    C="$BATS_TEST_TMPDIR/campaign"
+    run env FRANK_QWEN38_MODELS="$M" FRANK_QWEN38_CAMPAIGN="$C" bash "$RUNNER" kldref --dry-run
+    [ "$status" -eq 0 ]
+    ref_step="$(flag_value --max-num-batched-tokens <<< "$output")"
+    [ -n "$ref_step" ]
+    [ "$ref_step" -le 2048 ]
+    run env FRANK_QWEN38_MODELS="$M" FRANK_QWEN38_CAMPAIGN="$C" \
+        bash "$RUNNER" score "$M/rad/arm-w5nl64.rad" "$C/scores/arm1.json" --dry-run
+    [ "$status" -eq 0 ]
+    score_step="$(flag_value --max-num-batched-tokens <<< "$output")"
+    [ "$ref_step" = "$score_step" ]
+    # one knob moves both, so the operator can lower it at 2 a.m. without an edit
+    run env FRANK_QWEN38_MODELS="$M" FRANK_QWEN38_CAMPAIGN="$C" FRANK_QWEN38_MNBT=1024 \
+        bash "$RUNNER" kldref --dry-run
+    [ "$(flag_value --max-num-batched-tokens <<< "$output")" = "1024" ]
+    run env FRANK_QWEN38_MODELS="$M" FRANK_QWEN38_CAMPAIGN="$C" FRANK_QWEN38_MNBT=1024 \
+        bash "$RUNNER" score "$M/rad/arm-w5nl64.rad" "$C/scores/arm1.json" --dry-run
+    [ "$(flag_value --max-num-batched-tokens <<< "$output")" = "1024" ]
+    # calibration is exempt: grams are not logits, so the tap keeps the wide step
+    run env FRANK_QWEN38_MODELS="$M" FRANK_QWEN38_CAMPAIGN="$C" FRANK_QWEN38_MNBT=1024 \
+        bash "$RUNNER" calib --dry-run
+    [ "$(flag_value --max-num-batched-tokens <<< "$output")" = "65536" ]
+    # a step size that is not a token count is refused, never handed to the engine
+    run env FRANK_QWEN38_MNBT=broad bash "$RUNNER" kldref --dry-run
+    [ "$status" -eq 2 ]
+    [[ "$output" == *"FRANK_QWEN38_MNBT"* ]]
+    [[ "$output" == *"must be a token count"* ]]
+}
+
+@test "kldref refuses a corpus its window cannot hold, before the service is touched" {
+    M="$BATS_TEST_TMPDIR/models" R="$M/rad"
+    mkdir -p "$M/tpl" "$R" "$BATS_TEST_TMPDIR/out"
+    : > "$M/tpl/qwen38-froggeric-v22.5.jinja"
+    : > "$R/qwen38-bf16.rad"
+    CORPUS="$BATS_TEST_TMPDIR/long.jsonl"
+    {
+        printf '{"prompt":"short one"}\n'
+        printf '{"prompt":"%s"}\n' "$(head -c 120000 /dev/zero | tr '\0' 'x')"
+        printf '{"prompt":"short three"}\n'
+    } > "$CORPUS"
+    # 32,768 tokens x 3 chars is the pessimistic limit; the 120,002-char line
+    # that opens record 2 is well past it, so the campaign says no up front.
+    run env FRANK_QWEN38_MODELS="$M" FRANK_QWEN38_KLD_CORPUS="$CORPUS" \
+        bash "$RUNNER" kldref --dry-run
+    [ "$status" -eq 2 ]
+    [[ "$output" == *"REFUSED"* ]]
+    [[ "$output" == *"record 2"* ]]
+    [[ "$output" == *"120013 chars"* ]]
+    [[ "$output" == *"past the 98304 a 32768-token window"* ]]
+    [[ "$output" == *"FRANK_QWEN38_MAX_MODEL_LEN"* ]]
+    # the same corpus is fine once the window is raised past it, and the raised
+    # window is what the boot is asked for
+    run env FRANK_QWEN38_MODELS="$M" FRANK_QWEN38_KLD_CORPUS="$CORPUS" \
+        FRANK_QWEN38_MAX_MODEL_LEN=65536 FRANK_QWEN38_LOG_DIR="$BATS_TEST_TMPDIR/logs" \
+        bash "$RUNNER" kldref --dry-run
+    [ "$status" -eq 0 ]
+    [[ "$output" == *"corpus fits"* ]]
+    flat="$(oneline)"
+    [[ "$flat" == *"--max-model-len 65536"* ]]
+    # the shipped corpus of three short records fits the shipped window
+    SHORT="$BATS_TEST_TMPDIR/short.jsonl"
+    printf '{"prompt":"tiny"}\n{"prompt":"also tiny"}\n' > "$SHORT"
+    run env FRANK_QWEN38_MODELS="$M" FRANK_QWEN38_KLD_CORPUS="$SHORT" \
+        FRANK_QWEN38_LOG_DIR="$BATS_TEST_TMPDIR/logs" bash "$RUNNER" kldref --dry-run
+    [ "$status" -eq 0 ]
+    [[ "$output" == *"corpus fits: longest record 22 chars (number 2)"* ]]
+    # a real kldref run refuses at the same place: no log dir, no drain, and the
+    # service would still be up when the script gave up
+    L="$BATS_TEST_TMPDIR/logs-real"
+    run env FRANK_QWEN38_MODELS="$M" FRANK_QWEN38_KLD_CORPUS="$CORPUS" \
+        FRANK_QWEN38_LOG_DIR="$L" bash "$RUNNER" kldref
+    [ "$status" -eq 2 ]
+    [ ! -d "$L" ]
+    # and it is kldref that reads the corpus: score and calib never consult it
+    run env FRANK_QWEN38_MODELS="$M" FRANK_QWEN38_KLD_CORPUS="$CORPUS" \
+        FRANK_QWEN38_KLD_REF_DIR="$BATS_TEST_TMPDIR/ref" \
+        FRANK_QWEN38_LOG_DIR="$BATS_TEST_TMPDIR/logs" \
+        bash "$RUNNER" score "$R/qwen38-bf16.rad" "$BATS_TEST_TMPDIR/out/arm.json" --dry-run
+    [ "$status" -eq 0 ]
+    [[ "$output" != *"corpus fits"* ]]
+    [[ "$output" != *"REFUSED"* ]]
+}

tests/seerr_quadlet.bats

@@ -20,8 +20,9 @@ SEERR_UNIT="$REPO_ROOT/dankclaw/seerr/quadlet/seerr.container"
     grep -q '^Restart=always$' "$SEERR_UNIT"
 }
 
-@test "seerr image is pinned to an exact version tag" {
-    grep -Eq '^Image=ghcr\.io/seerr-team/seerr:v[0-9]+\.[0-9]+\.[0-9]+$' "$SEERR_UNIT"
+@test "seerr image is pinned to an exact version tag and digest" {
+    # 2026-10-21 fleet law: tag AND digest (tests/quadlet_image_pins.bats)
+    grep -Eq '^Image=ghcr\.io/seerr-team/seerr:v[0-9]+\.[0-9]+\.[0-9]+@sha256:[0-9a-f]{64}$' "$SEERR_UNIT"
 }
 
 @test "seerr publishes 5055 on loopback only (guard chain is the user path)" {