docs(radiance): correct the tokeniser corpus name and statsdump auth note

Nine pages under LLM/ document the upstream radiance engine at 8da043c:
- engine shape, plugin model, RAD2 container, ABI and environment
- every serve flag grouped with defaults and the law behind each knob
- the HTTP surface, request fields and refusal classes
- rad-convert, the recipe language, quantisers, imports, reuse, rad-export, KL scoring
- the (arch id, quant) family table and the recipe of record per model
- kernel/quantiser plugin authoring, encodings, build and checks
- architecture plugin authoring: declare, step, probe, lane and media rules
- rad-info/radbench/radtune and the shell harnesses
- the container contract, host prerequisites, readiness and restart triage

Also completes the dangling Related line on the frank-archon endpoint page.

AGENTS.md

@@ -176,11 +176,12 @@ alertmanager.container`, loopback :9093) → host-side Signal relay
 (`scripts/signal-relay.py`, user unit, 192.168.0.143:18093 — the host LAN
 IP, the one bind a container can reach as `host.containers.internal`;
 iptables admits loopback, the host and the podman bridge subnet only) → signal-cli account 318311.
-Prometheus targets read down for structural reasons (HTML-scraped "probes",
-Gitea metrics off); the remedy is a blackbox exporter. `frank-zealot` was down for
-a different reason until 2026-10-18: llama.cpp will not serve `/metrics` at all
-without `--metrics`. See
-`serverAppConfigs-docs/Infrastructure/Observability.md`.
+The former structural target downs (HTML-scraped "probes", Gitea metrics off)
+are retired as of 2026-10-21: HTTP liveness rides a blackbox exporter
+(`shared/blackbox/blackbox.yml`), Gitea serves `/metrics` from its quadlet, and
+every Prometheus job reports up. `frank-zealot` was down for a different reason
+until 2026-10-18: llama.cpp will not serve `/metrics` at all without
+`--metrics`. See `serverAppConfigs-docs/Infrastructure/Observability.md`.
 
 ## Testing
 
@@ -419,16 +420,17 @@ disagrees with it is the bug.
   digest-pinned `ghcr.io/peonist-ai/halogen-flash-server@sha256:0c61bf84…`.
   Engine stdout (the boot ledger: weights, KV pool, clamp) goes to
   `/home/dank/logs/frank_dragoon.log`, NOT the journal.
-- Shape: `HALOGEN_CTX=200704` is the **per-request ceiling** (a longer prompt
-  is a 400, never a truncation); `HALOGEN_KV_SLOTS=2` over ONE
-  `HALOGEN_KV_POOL_POSITIONS=401408` pool — pool positions buy concurrency,
-  the per-request window never moves; `HALOGEN_SPEC_ADAPT=0` keeps the MTP
-  drafter always armed (the adaptive default disarms it and halves decode);
-  server sampler 1.0/0.95/20.
-- Harness contract: `models.yml` `contextWindow: 155648` = ceiling − 45,056.
-  The clamp is `min(45056, window − est − 4096)` and the chars/4 estimate
-  under-counts, so a harness window equal to the ceiling makes every long
-  request a guaranteed 400. Reasoning arrives in `reasoning_content`; the
+- Shape: `HALOGEN_CTX=262144` (the checkpoint's native window) is the
+  **per-request ceiling** (a longer prompt is a 400, never a truncation);
+  `HALOGEN_KV_SLOTS=2` over ONE `HALOGEN_KV_POOL_POSITIONS=401408` pool —
+  pool positions buy concurrency, the per-request window never moves;
+  `HALOGEN_SPEC_ADAPT=0` keeps the MTP drafter always armed (the adaptive
+  default disarms it and halves decode); server sampler 1.0/0.95/20.
+- Harness contract: `models.yml` `contextWindow: 200000` — ~45k+ under the
+  ceiling. The clamp is `min(45056, window − est − 4096)` and the chars/4
+  estimate under-counts (trigger is 95 %), so a harness window at the ceiling
+  makes every long request a guaranteed 400.
+  Reasoning arrives in `reasoning_content`; the
   template rejects `reasoning_effort` outside `xhigh|medium|low`; images are
   OpenAI `image_url` parts (probe ≥256 px); `response_format` unsupported;
   `/completion`, `/ping`, `/props`, `/tokenize` are 404.

dankclaw/authentik/quadlet/authentik-outpost-ldap.container

@@ -23,7 +23,7 @@ Network=dankclaw_default
 # Image=ghcr.io/goauthentik/ldap:2026.8.3 (amd64 digest; podman 5.7.0 refuses
 # the tag's nested manifest list, so pin the single-platform digest).
 EnvironmentFile=/home/dank/serverAppConfigs/dankclaw/.env
-Image=ghcr.io/goauthentik/ldap@sha256:2e7b259eb8c37faac916946ddf48ec681ee9d3a9f724fe8521bf161973f13cf9
+Image=ghcr.io/goauthentik/ldap:2026.8.5@sha256:4c75bd27918418d2d6fbbfe4581bbd5a199fe38942d24559c4483edd64a1ea87
 Environment=AUTHENTIK_HOST=https://auth.franksdank.cloud
 Environment=AUTHENTIK_INSECURE=false
 Environment=AUTHENTIK_TOKEN=${AUTHENTIK_LDAP_OUTPOST_TOKEN}

dankclaw/authentik/quadlet/authentik-outpost.container

@@ -20,7 +20,7 @@ Wants=authentik-server.service
 
 [Container]
 ContainerName=authentik-outpost
-Image=ghcr.io/goauthentik/server:2026.8.3
+Image=ghcr.io/goauthentik/server:2026.8.5@sha256:602227a86455d6772e3d16b7d852713c4f864f40df5df894abd1285bb53979a5
 Exec=worker
 Network=dankclaw_default
 EnvironmentFile=/home/dank/serverAppConfigs/dankclaw/.env

dankclaw/authentik/quadlet/authentik-postgres.container

@@ -14,7 +14,7 @@ After=network.target
 
 [Container]
 ContainerName=authentik-postgres
-Image=docker.io/library/postgres:16-alpine
+Image=docker.io/library/postgres:16-alpine@sha256:0cefaabd882e42bcb2a2f3dc56d3ce4770560ba62fd7f197f1a2e008fd5731ab
 Network=dankclaw_default
 EnvironmentFile=/home/dank/serverAppConfigs/dankclaw/.env
 Environment=POSTGRES_DB=authentik

dankclaw/authentik/quadlet/authentik-server.container

@@ -19,7 +19,7 @@ Wants=authentik-postgres.service
 
 [Container]
 ContainerName=authentik-server
-Image=ghcr.io/goauthentik/server:2026.8.3
+Image=ghcr.io/goauthentik/server:2026.8.5@sha256:602227a86455d6772e3d16b7d852713c4f864f40df5df894abd1285bb53979a5
 Exec=server
 Network=dankclaw_default
 EnvironmentFile=/home/dank/serverAppConfigs/dankclaw/.env

dankclaw/authentik/quadlet/authentik-worker.container

@@ -16,7 +16,7 @@ Wants=authentik-server.service
 
 [Container]
 ContainerName=authentik-worker
-Image=ghcr.io/goauthentik/server:2026.8.3
+Image=ghcr.io/goauthentik/server:2026.8.5@sha256:602227a86455d6772e3d16b7d852713c4f864f40df5df894abd1285bb53979a5
 Exec=worker
 Network=dankclaw_default
 EnvironmentFile=/home/dank/serverAppConfigs/dankclaw/.env

dankclaw/gitea/quadlet/gitea-postgres.container

@@ -15,7 +15,7 @@ After=network.target
 [Container]
 ContainerName=gitea-postgres
 Network=gitea.network
-Image=docker.io/library/postgres:16-alpine
+Image=docker.io/library/postgres:16-alpine@sha256:0cefaabd882e42bcb2a2f3dc56d3ce4770560ba62fd7f197f1a2e008fd5731ab
 EnvironmentFile=/home/dank/Containers/gitea/env
 Environment=POSTGRES_USER=gitea
 Environment=POSTGRES_PASSWORD=${GITEA_DB_PASSWORD}

dankclaw/gitea/quadlet/gitea-server.container

@@ -19,7 +19,7 @@ After=gitea-postgres.service network.target
 [Container]
 ContainerName=gitea-server
 Network=gitea.network
-Image=docker.io/gitea/gitea:latest
+Image=docker.io/gitea/gitea:latest@sha256:30777c91280d973fa02be48b1ade735bd6859e8f6f7af3b4444d519437852504
 EnvironmentFile=/home/dank/Containers/gitea/env
 Environment=DATABASE_TYPE=postgres
 Environment=DATABASE_HOST=gitea-postgres
@@ -29,6 +29,13 @@ Environment=DATABASE_PASSWD=${GITEA_DB_PASSWORD}
 Environment=DATABASE_PORT=5432
 Environment=USER_UID=1000
 Environment=USER_GID=1000
+# Prometheus scrapes /metrics over the published LAN port (job "gitea" in
+# shared/prometheus/prometheus.yml). Off, /metrics is a 404 and the target is
+# permanently red (the audit's "disabled metrics" class). 0.0.0.0 binds inside
+# the container; the only route in is this host's published 3000, and the
+# exposed surface is counters only — same posture as the LLM /metrics.
+Environment=GITEA__metrics__ENABLED=true
+Environment=GITEA__metrics__ENABLED_HOST=0.0.0.0
 PublishPort=0.0.0.0:3000:3000
 PublishPort=2222:22
 Volume=/home/dank/Containers/gitea/data:/data

dankclaw/hermes/quadlet/chrome-devtools-protocol.container

@@ -25,7 +25,7 @@ Description=Chrome DevTools Protocol daemon (headless-shell) for hermes
 [Container]
 ContainerName=chrome-devtools-protocol
 Network=host
-Image=docker.io/chromedp/headless-shell@sha256:2d349b544a1ea6b5b5fd7c0fe99215ff662339c57407ee2e8c0a11af93516b04
+Image=docker.io/chromedp/headless-shell@sha256:7bee201478b2f6266aae01243740ee85c16281e32f42d3aed9fdf04f4eace7a1
 Entrypoint=/headless-shell/headless-shell
 Exec=--no-sandbox --disable-gpu --use-gl=angle --use-angle=swiftshader --remote-debugging-address=127.0.0.1 --remote-debugging-port=9222
 ShmSize=1073741824

dankclaw/marreta/quadlet/marreta-browser.container

@@ -23,7 +23,7 @@ After=gluetun.service network.target
 [Container]
 ContainerName=marreta_browser
 Network=gluetun.container
-Image=docker.io/lightpanda/browser:nightly
+Image=docker.io/lightpanda/browser:nightly@sha256:e72730e364d1fbf352c1149f29ffa4b278f3b1a55ff0dec3262298ebe624ec7a
 
 [Service]
 TimeoutStartSec=300

dankclaw/marreta/quadlet/marreta.container

@@ -32,7 +32,7 @@ After=gluetun.service marreta-browser.service network.target
 [Container]
 ContainerName=marreta
 Network=gluetun.container
-Image=ghcr.io/manualdousuario/marreta:latest
+Image=ghcr.io/manualdousuario/marreta:latest@sha256:30f2a34567b79b66da794b4b9423751bbbcd07ceff600d9f6a2d219e7e439362
 # Relocate off SearXNG's 8080 inside the shared netns.
 Environment=NGINX_HTTP_PORT=8085
 Environment=APP_NAME=Marreta

dankclaw/observability/quadlet/alertmanager.container

@@ -33,7 +33,7 @@ Wants=network-online.target
 
 [Container]
 ContainerName=alertmanager
-Image=docker.io/prom/alertmanager:latest
+Image=docker.io/prom/alertmanager:latest@sha256:39d8bb3966c694a124428a3a88fbcffd40bd069129ecd47c059f8a3f17c55f3b
 Network=dankclaw_default
 User=0:0
 Exec=--config.file=/etc/alertmanager/alertmanager.yml --storage.path=/alertmanager

dankclaw/observability/quadlet/blackbox-exporter.container

@@ -0,0 +1,56 @@
+# Quadlet unit: blackbox exporter (rootless, host network, boot-persistent).
+#
+# Install path on dankclaw:
+#   ~/.config/containers/systemd/blackbox-exporter.container
+#
+# WHY Network=host. The probes it runs are (a) the host's own LAN IP
+# (192.168.0.143:8081 SearXNG, :3005 OpenWebUI — host-process/portmap binds a
+# bridge container cannot reach; netavark DNATs published ports only, see
+# Container Host Reachability in the vault) and (b) ComfyUI's published port
+# via 127.0.0.1 (rootless --network=host shares the real host netns, so the
+# 0.0.0.0:8188 publish answers on loopback). A bridge-networked prober could
+# reach neither class reliably.
+#
+# WHY the listener binds 127.0.0.1 + LAN. A bridge container CANNOT dial
+# its own host's LAN IP (netavark hairpin gap, measured 2026-10-21), but a
+# bridge container dialing the podman gateway 169.254.1.2 lands on the host
+# LOOPBACK — that is the scrape path, and it needs this exporter to be
+# LISTENING ON LOOPBACK (proved: 200 from the prometheus container to
+# http://169.254.1.2:9115/-/healthy with a loopback bind). The gateway
+# address itself is NOT bindable from the host netns — it exists only
+# inside each rootless netns; binding it dies "bind: cannot assign
+# requested address" (live 2026-10-21). The LAN bind stays for diagnostics.
+# No public DNS name, no guard chain: internal net only, per the exposure
+# policy. The listen flag is --web.listen-address and it is REPEATABLE, one
+# flag per bind; plain --listen-address does not exist and the exporter
+# exits in 100 ms with "unknown long flag" (live 2026-10-21).
+
+[Unit]
+Description=Blackbox exporter (HTTP probe prober)
+After=network-online.target
+
+[Container]
+Image=docker.io/prom/blackbox-exporter@sha256:ea1c7aa3ef3f307cd03e85f34abb604478ccb90b567fc690da0cd9776da76b13
+ContainerName=blackbox-exporter
+Network=host
+# The image's ENTRYPOINT is /bin/blackbox_exporter itself (proved live
+# 2026-10-21: `podman run IMAGE sh` dies "unexpected sh"), so Exec= carries
+# FLAGS ONLY — repeating the binary makes it a positional arg and kingpin
+# exits "unexpected /bin/blackbox_exporter".
+Exec=--config.file=/etc/blackbox_exporter/config.yml --web.listen-address=127.0.0.1:9115 --web.listen-address=192.168.0.143:9115
+Volume=/opt/blackbox/blackbox.yml:/etc/blackbox_exporter/config.yml:ro
+# Plain exec form like every sibling unit. Shell form (`/bin/sh -c "…"`)
+# is passed as one quoted token and quadlet then folds the following
+# --health-interval/--health-retries (and the image name) into the command
+# string — unit died twice that way, live 2026-10-21. wget only; this
+# busybox variant has no curl, and host-network wget answers on loopback.
+HealthCmd=wget -q -O /dev/null http://127.0.0.1:9115/-/healthy
+HealthInterval=30s
+HealthRetries=3
+
+[Service]
+Restart=always
+RestartSec=5
+
+[Install]
+WantedBy=default.target

dankclaw/observability/quadlet/cadvisor.container

@@ -35,7 +35,7 @@ Wants=network-online.target
 
 [Container]
 ContainerName=cadvisor
-Image=gcr.io/cadvisor/cadvisor:v0.49.1
+Image=gcr.io/cadvisor/cadvisor:v0.49.1@sha256:3cde6faf0791ebf7b41d6f8ae7145466fed712ea6f252c935294d2608b1af388
 PodmanArgs=--pid=host --cgroupns=host
 PublishPort=127.0.0.1:8080:8080
 PublishPort=192.168.0.143:8080:8080

dankclaw/observability/quadlet/grafana.container

@@ -34,7 +34,7 @@ Wants=network-online.target prometheus.service
 
 [Container]
 ContainerName=grafana
-Image=docker.io/grafana/grafana:latest
+Image=docker.io/grafana/grafana:latest@sha256:488984b716c0862639f91a698adbe2862677a5a07bad42da5f03f3030883b205
 Network=dankclaw_default
 User=0:0
 PublishPort=127.0.0.1:3002:3000

dankclaw/observability/quadlet/node-exporter.container

@@ -33,7 +33,7 @@ Wants=network-online.target
 
 [Container]
 ContainerName=node-exporter
-Image=docker.io/prom/node-exporter:latest
+Image=docker.io/prom/node-exporter:latest@sha256:1b4e4438faca4dd7e001dd445d161a4a2091b0fededa84093b3a8dfeae1f1be0
 Network=host
 PodmanArgs=--pid=host
 Exec=--path.procfs=/host/proc --path.sysfs=/host/sys --path.rootfs=/rootfs --web.listen-address=127.0.0.1:9100 --web.listen-address=192.168.0.143:9100

dankclaw/observability/quadlet/prometheus.container

@@ -43,8 +43,16 @@ Wants=network-online.target
 
 [Container]
 ContainerName=prometheus
-Image=docker.io/prom/prometheus:latest
+Image=docker.io/prom/prometheus:latest@sha256:fa3302279710be2f21a8301ec17a8b4f2301fd829bd113839b3b6b02b6f191a2
 Network=dankclaw_default
+# Same-host scrape targets are dialed at host.containers.internal (the podman
+# gateway): a bridge container cannot dial its own host's LAN IP (netavark
+# hairpin gap, measured 2026-10-21), and traffic to the gateway lands on host
+# loopback / published ports (cadvisor/gpu jobs and the blackbox scrape all
+# ride it). The name resolves natively — NO HostAddr key: quadlet 5.7.0 calls
+# it unsupported and the WHOLE user quadlet batch then fails to generate,
+# silently orphaning every other unit (live 2026-10-21: prometheus.service
+# vanished while its container kept running as an orphan).
 User=0:0
 Exec=--config.file=/etc/prometheus/prometheus.yml --storage.tsdb.path=/prometheus --storage.tsdb.retention.time=30d --web.enable-lifecycle
 PublishPort=127.0.0.1:9090:9090

dankclaw/open-webui/quadlet/open-webui.container

@@ -27,7 +27,7 @@ After=network.target
 [Container]
 ContainerName=open-webui
 Network=open-webui.network
-Image=ghcr.io/open-webui/open-webui:main
+Image=ghcr.io/open-webui/open-webui:main@sha256:0b399a8ba34558e181d316750455f2a95376a38cd60a61b9c942e138898944bd
 PublishPort=3005:8080
 Environment=OPENAI_API_BASE_URL=http://192.168.0.143:7770
 Environment=ANONYMIZED_TELEMETRY=False

dankclaw/pihole/quadlet/pihole.container

@@ -24,7 +24,7 @@ Wants=network-online.target
 
 [Container]
 ContainerName=pihole
-Image=docker.io/pihole/pihole:latest
+Image=docker.io/pihole/pihole:latest@sha256:d34134716fc32038b052a8674c7c6879e0ff21f0e6d8af345166fffa2c27c113
 Network=slirp4netns
 PublishPort=0.0.0.0:53:53/udp
 PublishPort=0.0.0.0:53:53/tcp

dankclaw/searxng/quadlet/gluetun.container

@@ -16,7 +16,7 @@ After=searxng-network.service network.target
 [Container]
 ContainerName=gluetun
 Network=searxng.network
-Image=docker.io/qmcgaw/gluetun:latest
+Image=docker.io/qmcgaw/gluetun:latest@sha256:4af0c804003481e100180e1f8d4d5e9c48163fedc0f88d061881b615aef9d183
 AddCapability=NET_ADMIN
 AddDevice=/dev/net/tun:/dev/net/tun
 Environment=VPN_SERVICE_PROVIDER=${VPN_SERVICE_PROVIDER}

dankclaw/searxng/quadlet/searxng-valkey.container

@@ -16,7 +16,7 @@ After=searxng-network.service network.target
 ContainerName=searxng-valkey
 Network=searxng.network
 NetworkAlias=valkey
-Image=docker.io/valkey/valkey:9-alpine
+Image=docker.io/valkey/valkey:9-alpine@sha256:bf09d58e96d6b4524a15396b65b242bbe18beebbcc15a04d8ee75ddac62fae0b
 Volume=/opt/searxng/valkey:/data
 
 [Service]

dankclaw/searxng/quadlet/searxng.container

@@ -16,7 +16,7 @@ After=gluetun.service searxng-valkey.service network.target
 [Container]
 ContainerName=searxng
 Network=gluetun.container
-Image=ghcr.io/searxng/searxng:latest
+Image=ghcr.io/searxng/searxng:latest@sha256:f4aa660c54d062dedb268be84e821ea24e39f48c7a496af8dd02cc9eedb93847
 Environment=SEARXNG_SECRET=${SEARXNG_SECRET}
 Environment=SEARXNG_BASE_URL=${SEARXNG_BASE_URL}
 Volume=/opt/searxng/config:/etc/searxng

dankclaw/seerr/quadlet/seerr.container

@@ -29,7 +29,7 @@ Wants=network-online.target
 
 [Container]
 ContainerName=seerr
-Image=ghcr.io/seerr-team/seerr:v3.4.1
+Image=ghcr.io/seerr-team/seerr:v3.5.1@sha256:d8fab675b55ad293ded98fd71efcfd46054cbad9746534ae1cc4794a88c0b3be
 Network=dankclaw_default
 PublishPort=127.0.0.1:5055:5055
 Volume=/home/dank/Containers/seerr/data:/app/config

dankserver/calibre-server/quadlet/calibre-server.container

@@ -43,7 +43,7 @@ After=network.target
 
 [Container]
 ContainerName=calibre-server
-Image=lscr.io/linuxserver/calibre:9.15.0
+Image=lscr.io/linuxserver/calibre:9.17.0@sha256:b198fc03a6abc6be7e251a9958e50e74634b00e447c7b25e92c191c8077ba378
 Network=pasta
 EnvironmentFile=/home/dank/Containers/calibre-server/env
 Environment=PUID=1000

dankserver/calibre-web/quadlet/calibre-web.container

@@ -16,7 +16,7 @@ After=network.target
 
 [Container]
 ContainerName=calibre-web
-Image=lscr.io/linuxserver/calibre-web:0.6.27-ls402
+Image=lscr.io/linuxserver/calibre-web:0.6.27-ls407@sha256:470b4ed9a7d6ea1475fef17dbcf957cb60880910296d226bd0dca652c05f68ca
 Network=pasta
 Environment=DOCKER_MODS=linuxserver/mods:universal-calibre
 Environment=LANG=en_US.UTF-8

dankserver/copyparty/quadlet/copyparty.container

@@ -21,7 +21,13 @@ After=network.target
 
 [Container]
 ContainerName=copyparty
-Image=ghcr.io/9001/copyparty-ac:latest
+# Digest-pinned: upstream stopped publishing version tags after 1.19.15, so
+# the pin rides :latest@digest (the tag in the reference is cosmetic; the
+# digest is the pull). Pinned 2026-10-21 at the build whose label is
+# 1.20.26 - closes GHSA-3c88-p299-7hvr (1.20.25) and GHSA-hg99-jqfp-rf76
+# (1.20.26); the previously running build was the rolling :latest at
+# 1.20.24 (issue #73 on command-center).
+Image=ghcr.io/9001/copyparty-ac:latest@sha256:7f77459160a4bf34fdca79f2965dff4f0c1d7069a3d9cdfb3632acb041dd2f6a
 Network=dankserver_appnet
 EnvironmentFile=/home/dank/Containers/copyparty/env
 Exec=-a ${COPAY_ACCOUNT}:${COPAY_PASSWORD} -v /w::r:ra,dank

dankserver/jellyfin/quadlet/jellyfin.container

@@ -16,7 +16,7 @@ After=network.target
 
 [Container]
 ContainerName=jellyfin
-Image=ghcr.io/jellyfin/jellyfin:latest
+Image=ghcr.io/jellyfin/jellyfin:latest@sha256:fea863d0cd9ec445985d9c75fc6ed64e38a1d2766cf0c250a33d553f4fc1b771
 Network=dankserver_appnet
 Environment=DEBIAN_FRONTEND=noninteractive
 Environment=DOTNET_SYSTEM_IO_DISABLEFILELOCKING=true

dankserver/observability/quadlet/cadvisor.container

@@ -18,7 +18,7 @@ Wants=network-online.target
 
 [Container]
 ContainerName=cadvisor
-Image=gcr.io/cadvisor/cadvisor:v0.49.1
+Image=gcr.io/cadvisor/cadvisor:v0.49.1@sha256:3cde6faf0791ebf7b41d6f8ae7145466fed712ea6f252c935294d2608b1af388
 PodmanArgs=--pid=host --cgroupns=host
 PublishPort=127.0.0.1:8080:8080
 PublishPort=192.168.0.235:8080:8080

dankserver/observability/quadlet/node-exporter.container

@@ -37,7 +37,7 @@ Wants=network-online.target
 
 [Container]
 ContainerName=node-exporter
-Image=docker.io/prom/node-exporter:latest
+Image=docker.io/prom/node-exporter:latest@sha256:1b4e4438faca4dd7e001dd445d161a4a2091b0fededa84093b3a8dfeae1f1be0
 Network=host
 PodmanArgs=--pid=host
 Exec=--path.procfs=/host/proc --path.sysfs=/host/sys --path.rootfs=/rootfs --web.listen-address=127.0.0.1:9100 --web.listen-address=192.168.0.235:9100

dankserver/plex/quadlet/plex.container

@@ -4,7 +4,7 @@ After=network-online.target
 Wants=network-online.target
 
 [Container]
-Image=lscr.io/linuxserver/plex:1.43.4.10903-e5521bd8c-ls325
+Image=lscr.io/linuxserver/plex:1.43.5.11079-b3b13e506-ls330@sha256:01a53d9c7adcbb3701fbb0a4230c0a37c85453759b6b05a67356c3c7a09de08d
 ContainerName=plex
 Network=dankserver_appnet:alias=plex
 Environment=PUID=0

dankserver/romm/quadlet/romm-mariadb.container

@@ -14,7 +14,7 @@ After=network.target
 
 [Container]
 ContainerName=romm-mariadb
-Image=docker.io/library/mariadb:11
+Image=docker.io/library/mariadb:11@sha256:c4124d1afa875e47f2892edf26b482fa54f796e2f342d3717a04ca091a22a9bd
 Network=dankserver_appnet:alias=mariadb
 EnvironmentFile=/home/dank/Containers/romm/env
 Environment=MARIADB_DATABASE=romm

dankserver/romm/quadlet/romm-redis.container

@@ -12,7 +12,7 @@ After=network.target
 
 [Container]
 ContainerName=romm-redis
-Image=docker.io/library/redis:7-alpine
+Image=docker.io/library/redis:7-alpine@sha256:8da19c7399dca900c6822a1cf30f9d2285585e2c4b457ff4095b0f4500054d3f
 Network=dankserver_appnet:alias=redis
 Volume=/home/dank/Containers/romm/redis:/data
 HealthCmd=redis-cli ping

dankserver/romm/quadlet/romm.container

@@ -18,7 +18,7 @@ After=romm-mariadb.service romm-redis.service network.target
 
 [Container]
 ContainerName=romm
-Image=ghcr.io/rommapp/romm:latest
+Image=ghcr.io/rommapp/romm:latest@sha256:8677fb7b81b1d3db20b849e989a1dadc2a11159e51eb34a2acdcb01102274edf
 Network=dankserver_appnet
 EnvironmentFile=/home/dank/Containers/romm/env
 Environment=DB_HOST=mariadb

dankserver/tautulli/quadlet/tautulli.container

@@ -4,7 +4,7 @@ After=network-online.target
 Wants=network-online.target
 
 [Container]
-Image=lscr.io/linuxserver/tautulli:v2.18.1-ls245
+Image=lscr.io/linuxserver/tautulli:v2.18.2-ls250@sha256:529c7eabcc40554c1dc48f33f1f3cc3b1438462d73e0ec2dac8ac0b66aeeb8d4
 ContainerName=tautulli
 Network=dankserver_appnet:alias=tautulli
 Environment=PUID=0

frankbook-pro/gluetun/quadlet/gluetun.container

@@ -19,7 +19,7 @@ Description=Gluetun VPN tunnel (FrankBook-Pro podman machine)
 
 [Container]
 ContainerName=gluetun
-Image=docker.io/qmcgaw/gluetun:latest
+Image=docker.io/qmcgaw/gluetun:latest@sha256:c2ee4191762fca6b6a3c3359a731198a95d9c9af4db32d466e223bf3a84e779b
 AddCapability=NET_ADMIN
 AddDevice=/dev/net/tun:/dev/net/tun
 Environment=VPN_SERVICE_PROVIDER=${VPN_SERVICE_PROVIDER}

frankbook-pro/lidarr/quadlet/lidarr.container

@@ -15,7 +15,7 @@ After=gluetun.service
 
 [Container]
 ContainerName=lidarr
-Image=lscr.io/linuxserver/lidarr:latest
+Image=lscr.io/linuxserver/lidarr:latest@sha256:d8549ff9ea29b4015c32257360652c1000c31ea471a2b3f0835ae77292faf35b
 Environment=PUID=0
 Environment=PGID=0
 Environment=TZ=America/Los_Angeles

frankbook-pro/prowlarr/quadlet/prowlarr.container

@@ -24,7 +24,7 @@ After=gluetun.service
 
 [Container]
 ContainerName=prowlarr
-Image=lscr.io/linuxserver/prowlarr:latest
+Image=lscr.io/linuxserver/prowlarr:latest@sha256:b3b00047ed20f6ce2b16780145cb6a0acb51c9e28ea6aa6f5dcd33c7034f1797
 Environment=PUID=0
 Environment=PGID=0
 Environment=TZ=America/Los_Angeles

frankbook-pro/qbittorrent/quadlet/qbittorrent.container

@@ -29,7 +29,7 @@ Network=gluetun.container
 # Pinned to the last 4.x line: Readarr's retired qB client requires the
 # qB 4.x login response (HTTP 200, body "Ok."); qB 5.x answers 204 with an
 # empty body, which fails Readarr's auth check. All other clients accept 4.x.
-Image=lscr.io/linuxserver/qbittorrent:4.6.7
+Image=lscr.io/linuxserver/qbittorrent:4.6.7@sha256:55f15d44396315551f87294a176efae733b16e283f38980308e46073950257c6
 Environment=PUID=0
 Environment=PGID=0
 Environment=TZ=America/Los_Angeles

frankbook-pro/radarr/quadlet/radarr.container

@@ -15,7 +15,7 @@ After=gluetun.service
 
 [Container]
 ContainerName=radarr
-Image=lscr.io/linuxserver/radarr:latest
+Image=lscr.io/linuxserver/radarr:latest@sha256:fc0ae3c50f8128b97e84381903ec8d6845c3a569641cb7117c124376cccb57c1
 Environment=PUID=0
 Environment=PGID=0
 Environment=TZ=America/Los_Angeles

frankbook-pro/readarr/quadlet/readarr.container

@@ -19,7 +19,7 @@ After=gluetun.service
 
 [Container]
 ContainerName=readarr
-Image=lscr.io/linuxserver/readarr:amd64-1.0.980
+Image=lscr.io/linuxserver/readarr:amd64-1.0.980@sha256:a95e82dbb0eb2bb43de4c150f1dab9dd821306c67ea0b418983a957e731f2b61
 Environment=PUID=0
 Environment=PGID=0
 Environment=TZ=America/Los_Angeles

frankbook-pro/sonarr/quadlet/sonarr.container

@@ -16,7 +16,7 @@ After=gluetun.service
 
 [Container]
 ContainerName=sonarr
-Image=lscr.io/linuxserver/sonarr:latest
+Image=lscr.io/linuxserver/sonarr:latest@sha256:889dcc58c6aea859033985894b519b2e6b1e55d92527ed123f9f52e42ae7c97b
 Environment=PUID=0
 Environment=PGID=0
 Environment=TZ=America/Los_Angeles

scripts/apply-protected-cast.sh

@@ -1,27 +1,45 @@
 #!/usr/bin/env bash
-# apply-protected-cast.sh — RUNS ON DANKSERVER. Rewrite the recipe's
-# PROTECTED-BEGIN/END section from Swift's OWN calibration grams: run
+# apply-protected-cast.sh — RUNS ON DANKSERVER. Rewrite a recipe's
+# PROTECTED-BEGIN/END section from the campaign's OWN calibration grams: run
 # derive-protected-experts.py over the calib dir, splice its cast lines into
 # the marked section of the recipe in place, keep the pre-edit recipe in the
-# campaign trash. Refuses to run twice (marker text says derived).
+# campaign trash. Refuses to run twice (marker text says derived), and refuses
+# when a requested target count was not reached. Run it on a COPY of the
+# template so the committed recipe stays the clean, reviewable baseline.
 set -euo pipefail
-CALIB=${1:-/home/dank/corpora/calib}
-RECIPE=${2:-/home/dank/serverAppConfigs-swift15/scripts/recipes/qwen4exp-swift15-w4dn8-i8-hc8m.recipe}
-PY=/home/dank/venvs/radcorpus/bin/python
-DERIVE=/home/dank/serverAppConfigs-swift15/scripts/derive-protected-experts.py
+SELF_DIR="$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)"
+CALIB=${1:-/home/dank/campaign/qwen38-w5nl64/calib}
+RECIPE=${2:-$SELF_DIR/recipes/qwen4exp-qwen38-w5nl64-bf16trunk.recipe}
+TARGET_COUNT=${3:-0}
+PY=${PY:-/home/dank/venvs/radcorpus/bin/python}
+DERIVE=${DERIVE:-$SELF_DIR/derive-protected-experts.py}
 TS=$(date +%Y%m%d-%H%M%S)
+# OUT and TRASH_DIR are overridable so the seam can drive the whole script in a
+# scratch directory; on the box they are the campaign's usual places.
+OUT=${OUT:-/tmp/protected-cast-$TS.txt}
+TRASH_DIR=${TRASH_DIR:-/home/dank/.AGENTTRASH}
 
 grep -q "derive-protected-experts.py rewrote" "$RECIPE" && {
     echo "recipe already carries derived cast lines; refusing to re-derive" >&2; exit 1; }
 grep -q "PROTECTED-BEGIN" "$RECIPE" || { echo "no PROTECTED markers in $RECIPE" >&2; exit 1; }
 
-"$PY" "$DERIVE" "$CALIB" > "/tmp/protected-cast-$TS.txt"
-N=$(grep -c 'cast dtype=bf16' "/tmp/protected-cast-$TS.txt")
-[ "$N" -ge 2 ] || { echo "derive produced $N cast lines; refusing" >&2; exit 1; }
-echo "derived $N protected experts -> /tmp/protected-cast-$TS.txt"
+ARGS=("$DERIVE" "$CALIB")
+if [ "$TARGET_COUNT" -gt 0 ]; then
+    ARGS+=(--target-count "$TARGET_COUNT")
+fi
+"$PY" "${ARGS[@]}" > "$OUT"
+N=$(grep -c 'cast dtype=bf16' "$OUT")
+[ "$N" -ge 2 ] || { echo "derive produced $N cast lines; refusing (see $OUT)" >&2; exit 1; }
+if [ "$TARGET_COUNT" -gt 0 ] && [ "$N" -lt "$TARGET_COUNT" ]; then
+    echo "derive reached $N protected experts, target was $TARGET_COUNT; refusing to convert" >&2
+    echo "       (the bar sweep bottomed out: the recording is too thin or the target too high)" >&2
+    exit 1
+fi
+echo "derived $N protected experts -> $OUT"
 
-cp -f "$RECIPE" "/home/dank/.AGENTTRASH/$TS-qwen4exp-swift15-w4dn8-i8-hc8m.recipe"
-"$PY" - "$RECIPE" "/tmp/protected-cast-$TS.txt" <<'PYEOF'
+mkdir -p "$TRASH_DIR"
+cp -f "$RECIPE" "$TRASH_DIR/$TS-$(basename "$RECIPE")"
+"$PY" - "$RECIPE" "$OUT" <<'PYEOF'
 import sys
 recipe, castfile = sys.argv[1], sys.argv[2]
 lines = open(recipe).read().splitlines()
@@ -35,7 +53,7 @@ for l in lines:
     elif 'PROTECTED-END' in l:
         insec = False
         out.append(l)
-        out.append('# (derive-protected-experts.py rewrote this section from Swift\'s own grams)')
+        out.append('# (derive-protected-experts.py rewrote this section from the campaign grams)')
     elif not insec:
         out.append(l)
 open(recipe, 'w').write('\n'.join(out) + '\n')

scripts/build-qwen38-bf16-rad.sh

@@ -0,0 +1,220 @@
+#!/usr/bin/env bash
+# build-qwen38-bf16-rad.sh — RUNS ON DANKSERVER. rad-convert the vendor BF16
+# checkpoint /home/dank/models/Qwen3.8-Flash-Next into
+# /home/dank/models/rad/qwen38-bf16.rad with NO recipe and NO --reuse: a weight
+# no rule matches keeps the checkpoint's encoding, so every tensor stays the
+# bf16 it is in the safetensors and the container is the checkpoint's own
+# numbers behind a .rad header.
+#
+# This file is the campaign's measuring stick, twice over: scripts/
+# run-qwen38-bf16.sh kldref records the KLD reference logits from it (every
+# arm's --kld-ref is scored against that record) and the same container serves
+# with the calibration tap to write the Grams that derive the protected-expert
+# list and feed GPTQ. A reference that is not the vendor checkpoint, byte for
+# byte in bf16, silently moves every gate in the campaign — hence no recipe, no
+# --reuse, and the four refusal gates below.
+#
+# Why a container at all: a checkpoint dir has no mapped/file tiers, and a
+# weight only a host kernel reads (the 47.7 GiB PLE n-gram table) cannot be
+# served from one; rad-convert also bakes the tokenizer. The convert runs on
+# the HOST backend — no --device /dev/kfd, no /dev/dri — so declare resolves
+# kernels on CPU and production serving is untouched for the whole run (the
+# precedent that proved it: scripts/convert-swift15-w4dn8.sh).
+#
+# Gates, every one a refusal with exit 2 and a named reason, BEFORE anything is
+# written, cheapest first — the verifier runs last of them because its Hub gate
+# hashes 343 GiB and must not be paid for by a run that was going to refuse on
+# the output file anyway.
+#   checkpoint  the tree must be on disk; an absent dir names the command that
+#               makes one (scripts/download-qwen38-bf16.sh --fetch) and stops.
+#   free space  the convert writes ~343,323 MiB next to the ~343,323 MiB
+#               checkpoint; QWEN38_MIN_FREE_MIB (default 350000 = that figure
+#               plus 6.5 GiB of slack) is the floor. The volume that cannot
+#               hold both is discovered at minute 2, not hour 5.
+#   output      the campaign converts the reference ONCE. An existing
+#               qwen38-bf16.rad is refused; --force retires it to
+#               ~/.AGENTTRASH/<utc>-qwen38-bf16.rad (trash law: nothing is
+#               deleted, and the move is a rename on the same volume).
+#   verified    scripts/download-qwen38-bf16.sh's verify mode must pass. A
+#               failed verify exits 1, not 2: the tree being the wrong payload
+#               is a result, a missing environment is a refusal.
+#   plan        rad-convert --plan-only -v first — the oracle for "does this
+#               shape boot". A refused plan costs seconds and saves a convert.
+#
+# usage: scripts/build-qwen38-bf16-rad.sh [--dry-run] [--plan-only] [--force]
+#   --dry-run    print the argv it would exec (podman wrapping rad-convert),
+#               one shell-quoted argument a line, and exit 0. No gate runs, no
+#               container starts, nothing is written — the offline test seam.
+#               It does resolve the checkpoint and output paths through the
+#               /models mount, because an argv that carries host paths would
+#               not run either.
+#   --plan-only  run the plan and every gate except the disk and output ones —
+#               those two protect bytes the plan never writes, and this is the
+#               probe to run when the disk gate itself is what refuses
+#   --force      retire an existing output container to the trash dir first
+#
+# knobs: FRANK_QWEN38_MODELS   the host dir mounted at /models (default
+#        /home/dank/models); FRANK_QWEN38_CKPT (default $MODELS/
+#        Qwen3.8-Flash-Next, the same knob the verifier reads);
+#        FRANK_QWEN38_BF16_OUT (default $MODELS/rad/qwen38-bf16.rad);
+#        QWEN38_MIN_FREE_MIB; QWEN38_TRASH (default $HOME/.AGENTTRASH). The
+#        verifier's own census knobs (FRANK_QWEN38_SHARDS / _BYTES / _REPO)
+#        reach it through the environment untouched.
+# exit: 0 converted (or planned), 1 the checkpoint failed verification,
+#       2 refused before anything was written
+set -euo pipefail
+
+SCRIPT_DIR="$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)"
+# ONE owner of the engine pin: the digest frank-archon serves on. The reference
+# must be built by the SAME engine build that scores it, or a KLD delta is
+# partly a version delta.
+# shellcheck disable=SC1091
+source "$SCRIPT_DIR/frank-archon.env"
+IMAGE="${FRANK_ARCHON_IMAGE:?FRANK_ARCHON_IMAGE must name the radiance image}"
+
+MODELS="${FRANK_QWEN38_MODELS:-/home/dank/models}"
+CKPT="${FRANK_QWEN38_CKPT:-$MODELS/Qwen3.8-Flash-Next}"
+OUT="${FRANK_QWEN38_BF16_OUT:-$MODELS/rad/qwen38-bf16.rad}"
+TRASH="${QWEN38_TRASH:-$HOME/.AGENTTRASH}"
+MIN_FREE_MIB="${QWEN38_MIN_FREE_MIB:-350000}"
+LOG="/home/dank/logs/qwen38_bf16_convert.log"
+
+DRY=0
+PLAN_ONLY=0
+FORCE=0
+while [ $# -gt 0 ]; do
+    case "$1" in
+        --dry-run)   DRY=1 ;;
+        --plan-only) PLAN_ONLY=1 ;;
+        --force)     FORCE=1 ;;
+        *) echo "usage: ${0##*/} [--dry-run] [--plan-only] [--force]" >&2
+           exit 2 ;;
+    esac
+    shift
+done
+
+# The convert sees the tree only through the /models mount, so both paths must
+# be under it; the container-side spelling is what goes into the argv.
+case "$CKPT" in
+    "$MODELS"/*) CKPT_IN="/models/${CKPT#"$MODELS"/}" ;;
+    *) echo "REFUSED: checkpoint $CKPT is not under the mounted models dir $MODELS — the convert cannot see it" >&2
+       exit 2 ;;
+esac
+case "$OUT" in
+    "$MODELS"/*) OUT_IN="/models/${OUT#"$MODELS"/}" ;;
+    *) echo "REFUSED: output $OUT is not under the mounted models dir $MODELS — rad-convert could not write it" >&2
+       exit 2 ;;
+esac
+
+# Declare with NO recipe line — that is what keeps every tensor bf16 — and at the
+# bound this container will actually be served at. Both flags come off the pinned
+# image's own help: --kv-cache-dtype at convert "only picks the kernels it
+# checks" and the container is identical either way, so naming fp8 is free
+# insurance — if the fp8 KV kernels the fleet serves would not resolve, the
+# convert refuses here instead of the boot failing on a drained window.
+# --max-ctx is the bound declare is run at, whose default is the checkpoint's
+# training context (262,144 here, whose fp8 cache is 258.8 GiB against a 192 GiB
+# machine, so the default is not an option). 32,768 is set by the corpus, not by
+# taste: kld.jsonl's longest record is 38,263 characters — ~9,600 tokens at the
+# pessimistic chars/4, ~7,700 at the chars/4.97 this repo measured on the K3
+# corpus — so the window the rig boots at clears every record 2.6x over and the
+# reference boot cannot die on a sequence that will not fit. The rig's window is
+# one knob (FRANK_QWEN38_MAX_MODEL_LEN in scripts/run-qwen38-bf16.sh, default
+# 32,768) and that script refuses a corpus its window cannot hold, so raising one
+# without the other is caught before the drained window, not inside it. The
+# payload containers the gate scores declare at the training context because
+# production must hold the 200k shape; a declared bound is a declare-time check,
+# not a weight change, so the reference and the payloads carry the same numbers
+# whatever each declared at — which is why one can be measured against the other.
+CONVERT_ARGS=("$CKPT_IN" -o "$OUT_IN" --max-ctx 32768 --kv-cache-dtype fp8)
+PODMAN=(podman run --rm --entrypoint rad-convert -v "$MODELS:/models")
+PLAN_CMD=("${PODMAN[@]}" --name qwen38-bf16-plan "$IMAGE" "${CONVERT_ARGS[@]}" --plan-only -v)
+CONV_CMD=("${PODMAN[@]}" --name qwen38-bf16-convert "$IMAGE" "${CONVERT_ARGS[@]}")
+
+if [ "$DRY" = "1" ]; then
+    echo "dry run: no gate checked, no container started, nothing written"
+    echo "# $CKPT -> $OUT (the convert sees them as $CKPT_IN and $OUT_IN)"
+    echo "# plan argv:"
+    printf '%q\n' "${PLAN_CMD[@]}"
+    echo "# convert argv:"
+    printf '%q\n' "${CONV_CMD[@]}"
+    exit 0
+fi
+
+# ---- gate: the checkpoint is on disk ---------------------------------------
+if [ ! -d "$CKPT" ]; then
+    echo "REFUSED: no checkpoint at $CKPT — fetch and verify it first: scripts/download-qwen38-bf16.sh --fetch" >&2
+    exit 2
+fi
+
+# ---- gate: peak disk --------------------------------------------------------
+# The deepest existing ancestor is the right thing to ask: a fresh
+# /home/dank/models/rad has no free space of its own to report.
+OUT_DIR="$(dirname "$OUT")"
+PROBE="$OUT_DIR"
+while [ ! -d "$PROBE" ] && [ "$PROBE" != "/" ]; do PROBE="$(dirname "$PROBE")"; done
+FREE_MIB="$(df --output=avail -m "$PROBE" | tail -1 | tr -d ' ')"
+if ! [[ "$FREE_MIB" =~ ^[0-9]+$ ]]; then
+    echo "REFUSED: cannot read the free space under $PROBE (df said: $FREE_MIB)" >&2
+    exit 2
+fi
+if [ "$PLAN_ONLY" != "1" ] && [ "$FREE_MIB" -lt "$MIN_FREE_MIB" ]; then
+    echo "REFUSED peak-disk: $FREE_MIB MiB free under $PROBE, $MIN_FREE_MIB MiB wanted (QWEN38_MIN_FREE_MIB) — the bf16 container rewrites the checkpoint's own 343,323 MiB of tensors beside it" >&2
+    exit 2
+fi
+
+# ---- gate: one conversion ---------------------------------------------------
+if [ -e "$OUT" ]; then
+    if [ "$PLAN_ONLY" = "1" ]; then
+        echo "note: $OUT already exists; --plan-only writes nothing, so it stays"
+    elif [ "$FORCE" = "1" ]; then
+        mkdir -p "$TRASH"
+        DEST="$TRASH/$(date -u +%Y%m%dT%H%M%SZ)-$(basename "$OUT")"
+        mv "$OUT" "$DEST"
+        echo "TRASHED: retired the existing reference container $OUT -> $DEST"
+    else
+        echo "REFUSED: $OUT already exists — the campaign converts the reference once; re-run with --force to retire it to $TRASH first" >&2
+        exit 2
+    fi
+fi
+
+# ---- gate: the tree really is the checkpoint (the full verify) -------------
+echo "== verifying $CKPT (scripts/download-qwen38-bf16.sh, full gates)"
+rc=0
+bash "$SCRIPT_DIR/download-qwen38-bf16.sh" --dir "$CKPT" || rc=$?
+if [ "$rc" -ne 0 ]; then
+    if [ "$rc" -eq 2 ]; then
+        echo "REFUSED: the checkpoint verifier refused the environment (rc=2); nothing was converted" >&2
+        exit 2
+    fi
+    echo "REFUSED: $CKPT failed verification (rc=$rc) — a half tree converts to a half reference" >&2
+    exit 1
+fi
+
+# ---- plan, then convert ------------------------------------------------------
+mkdir -p "$(dirname "$LOG")"
+# rad-convert opens -o during declare as well, so the output directory exists
+# before the plan; that empty directory is all this creates ahead of it.
+mkdir -p "$OUT_DIR"
+echo "== rad-convert --plan-only -v (declare + plan; writes nothing)"
+"${PLAN_CMD[@]}" 2>&1 | tee -a "$LOG"
+rc=${PIPESTATUS[0]}
+if [ "$rc" -ne 0 ]; then
+    echo "REFUSED plan: rad-convert --plan-only exited rc=$rc — the engine refuses this encoding/shape, no container was written (log: $LOG)" >&2
+    exit 2
+fi
+if [ "$PLAN_ONLY" = "1" ]; then
+    echo "PLAN_OK: $CKPT_IN declares for $OUT_IN; no container written (log: $LOG)"
+    exit 0
+fi
+
+echo "== convert: $CKPT -> $OUT ($(date -Is))"
+"${CONV_CMD[@]}" 2>&1 | tee -a "$LOG"
+rc=${PIPESTATUS[0]}
+if [ "$rc" -ne 0 ]; then
+    echo "rad-convert rc=$rc; log: $LOG" >&2
+    exit "$rc"
+fi
+MIB=$(( $(stat -c %s "$OUT") / 1048576 ))
+echo "BUILD_DONE: $OUT is ${MIB} MiB (log: $LOG)"
+echo "NEXT: scripts/run-qwen38-bf16.sh kldref records the reference logits from it"

scripts/calib-push.py

@@ -22,6 +22,17 @@ Mechanics notes:
   prompts do not bias the Hessians (the corpus is drawn round-robin over
   source classes, so loss is class-neutral as long as failures are rare;
   a systematic 4xx is loud in the log and must be fixed, not ignored).
+- a round that seals no tap bytes is NORMAL and continues: RADIANCE_CALIB_TOKENS
+  is a pooled floor and this campaign's corpus is 10.03M tokens against a 12M
+  floor, so round 1 seals nothing by construction. A tap that LOSES bytes fails
+  the run, and a run that never sees the engine's completion line exits non-zero.
+  The first version failed on a flat round, which aborted exactly one round too
+  early (2026-10-29).
+- before any prompt, the tap must carry the rig's own marker
+  (`--tap-marker`, written by run-qwen38-bf16.sh calib) naming THIS
+  container, and that container must be running. Measured 2026-10-29:
+  with neither check, four rounds of 10M tokens went into restored
+  production after the rig's boot had died, and recorded nothing.
 
 Usage: calib-push.py [--port 7770] [--parallel 64] [--container arbench]
        [--corpus /home/dank/corpora/out/calib.jsonl] [--max-rounds 4]
@@ -29,11 +40,18 @@ Usage: calib-push.py [--port 7770] [--parallel 64] [--container arbench]
 import argparse
 import concurrent.futures
 import json
+import os
 import subprocess
 import sys
 import time
 import urllib.request
 
+_HERE = os.path.dirname(os.path.abspath(__file__)) if "__file__" in globals() else os.getcwd()
+if _HERE not in sys.path:
+    sys.path.insert(0, _HERE)
+
+import qwen38_common as qc  # noqa: E402 - sibling module, same directory
+
 DONE_MARK = "calibration wrote"
 
 
@@ -49,9 +67,9 @@ def tap_done(container):
     return DONE_MARK in (out.stdout + out.stderr)
 
 
-def push_one(url, prompt):
+def push_one(url, prompt, model):
     body = json.dumps({
-        "model": "frank-archon",
+        "model": model,
         "prompt": prompt,
         "max_tokens": 1,
         "temperature": 0,
@@ -68,38 +86,151 @@ def push_one(url, prompt):
         return False
 
 
-def main():
+def push_round(url, prompts, model, parallel):
+    """Push the whole corpus once; returns how many requests were accepted."""
+    ok = 0
+    with concurrent.futures.ThreadPoolExecutor(parallel) as ex:
+        for good in ex.map(lambda p: push_one(url, p, model), prompts):
+            ok += 1 if good else 0
+    return ok
+
+
+def tap_bytes(path):
+    """Bytes the tap has sealed so far; 0 when the directory is absent.
+
+    A tap whose directory is wrong, full or read-only stops writing while the
+    engine keeps answering 200 — the driver then re-pushes the whole corpus for
+    every round it is allowed and the loss is invisible until the hours are gone.
+    """
+    total = 0
+    try:
+        for root, _, files in os.walk(path):
+            for f in files:
+                try:
+                    total += os.path.getsize(os.path.join(root, f))
+                except OSError:
+                    pass
+    except OSError:
+        return 0
+    return total
+
+
+def round_verdict(before, after, done):
+    """What one round's tap bytes mean.
+
+    'done'   the engine logged that it sealed the tap — the run is over
+    'shrink' the tap LOST bytes: reset, pruned or replaced under us
+    'grew'   sealed bytes this round
+    'flat'   sealed nothing — which is legitimate: RADIANCE_CALIB_TOKENS is a
+             pooled floor, and a round that does not reach it has nothing to
+             write yet (the campaign's corpus is 10.03M tokens against a 12M
+             floor, so round 1 is always flat). Failing on it aborted the run
+             exactly one round too early.
+    """
+    if done:
+        return "done"
+    if after < before:
+        return "shrink"
+    if after == before:
+        return "flat"
+    return "grew"
+
+
+def require_rig(a):
+    """Refuse unless the rig that opened this tap is up and serving right now.
+
+    Three facts, in order: the runner wrote its marker into the tap dir, the
+    marker names the container this pusher targets, and that container is
+    running. A marker without a live container is a boot that died; a live
+    container without a marker is some other engine on the port — on 2026-10-29
+    it was restored production, which took four rounds of 10M tokens before the
+    old driver admitted it had recorded nothing.
+    """
+    marker = os.path.join(a.tap_dir, a.tap_marker)
+    if not os.path.isfile(marker):
+        qc.refuse(f"no rig marker at {marker}: this tap was not opened by "
+                  "run-qwen38-bf16.sh calib, so a push here would feed whatever "
+                  "answers the port and record nothing")
+    try:
+        lines = open(marker, encoding="utf-8", errors="replace").read().splitlines()
+    except OSError as exc:
+        qc.refuse(f"cannot read the rig marker {marker}: {exc}")
+    meta = dict(line.split("=", 1) for line in lines if "=" in line)
+    who = meta.get("container", "")
+    if who != a.container:
+        qc.refuse(f"the rig marker names container {who!r} and this pusher targets "
+                  f"{a.container!r} — wrong tap, or a stale marker")
+    try:
+        p = subprocess.run(["podman", "inspect", "-f", "{{.State.Running}}", a.container],
+                           capture_output=True, text=True, timeout=30)
+    except Exception as exc:
+        qc.refuse(f"cannot inspect the rig container {a.container}: {exc}")
+    if p.returncode != 0 or p.stdout.strip() != "true":
+        qc.refuse(f"the rig container {a.container} is not running — whatever is "
+                  "answering the port, it is not the calibration rig")
+
+
+def parse_args():
     ap = argparse.ArgumentParser()
     ap.add_argument("--port", type=int, default=7770)
     ap.add_argument("--parallel", type=int, default=64)
     ap.add_argument("--container", default="arbench")
+    ap.add_argument("--model", default="frank-archon")
+    ap.add_argument("--tap-dir", default="/home/dank/campaign/qwen38-w5nl64/calib")
+    ap.add_argument("--tap-marker", default=".rig-boot",
+                    help="the file run-qwen38-bf16.sh calib writes into the tap dir; without it the pusher refuses, because an unmarked tap is not a rig")
     ap.add_argument("--corpus", default="/home/dank/corpora/out/calib.jsonl")
     ap.add_argument("--max-rounds", type=int, default=4)
-    a = ap.parse_args()
-    url = f"http://127.0.0.1:{a.port}/v1/completions"
+    return ap.parse_args()
 
+
+def load_prompts(a) -> list[str]:
+    if not os.path.isfile(a.corpus):
+        qc.refuse(f"no corpus at {a.corpus}")
     prompts = []
-    with open(a.corpus) as f:
-        for line in f:
-            line = line.strip()
-            if line:
-                prompts.append(json.loads(line)["prompt"])
-    print(f"corpus: {len(prompts)} prompts", flush=True)
+    for lineno, rec in qc.iter_jsonl(a.corpus):
+        if not isinstance(rec.get("prompt"), str):
+            qc.refuse(f"{a.corpus}:{lineno} has no string 'prompt' field")
+        prompts.append(rec["prompt"])
+    if not prompts:
+        qc.refuse(f"the corpus at {a.corpus} holds no prompts")
+    return prompts
+
+
+def main():
+    a = parse_args()
+    url = f"http://127.0.0.1:{a.port}/v1/completions"
+    prompts = load_prompts(a)
+    require_rig(a)
+    chars = sum(len(p) for p in prompts)
+    print(f"corpus: {len(prompts)} prompts, ~{chars // 4} tokens (chars/4)",
+          flush=True)
 
     for rnd in range(1, a.max_rounds + 1):
         if tap_done(a.container):
             break
+        before = tap_bytes(a.tap_dir)
         t0 = time.time()
-        ok = 0
-        with concurrent.futures.ThreadPoolExecutor(a.parallel) as ex:
-            for good in ex.map(lambda p: push_one(url, p), prompts):
-                ok += 1 if good else 0
-        dt = time.time() - t0
+        ok = push_round(url, prompts, a.model, a.parallel)
+        dt = max(time.time() - t0, 1e-9)
         print(f"round {rnd}: {ok}/{len(prompts)} ok in {dt:.0f}s "
-              f"({sum(len(p) for p in prompts)/4/dt:.0f} chars/s est)",
-              flush=True)
+              f"({chars / 4 / dt:.0f} tokens/s est)", flush=True)
+        if ok == 0:
+            print("FAIL: not one prompt was accepted; fix the engine or the argv "
+                  "before spending another round", flush=True)
+            sys.exit(1)
         if tap_done(a.container):
             break
+        after = tap_bytes(a.tap_dir)
+        verdict = round_verdict(before, after, False)
+        if verdict == "shrink":
+            print(f"FAIL: the tap at {a.tap_dir} LOST bytes this round "
+                  f"({before} B before, {after} B after) — it was reset or replaced "
+                  "under us", flush=True)
+            sys.exit(1)
+        if verdict == "flat":
+            print(f"round {rnd}: the tap sealed nothing yet (byte floor not reached; "
+                  f"{before} B) — continuing", flush=True)
     else:
         print("max rounds exhausted, tap not done", flush=True)
         sys.exit(1)

scripts/convert-qwen38-w5nl64.sh

@@ -0,0 +1,231 @@
+#!/usr/bin/env bash
+# convert-qwen38-w5nl64.sh — RUNS ON DANKSERVER. The CPU half of the Qwen3.8-Flash-Next five-bit
+# campaign: convert the vendor BF16 checkpoint into the campaign's arms, each one the whole
+# `w5nl64a8h` rung on BOTH expert projections with the protected experts kept bf16.
+#
+# Nothing here touches a GPU: the container is started without /dev/kfd or /dev/dri, so the
+# converter runs on the host backend and the fleet can keep serving through the whole script.
+# The engine pin is the digest that serves frank-archon, so a winning arm crosses no version
+# boundary between measurement and serving.
+#
+# Three arms, because the campaign's question is which search knob pays for its hours
+# (recipes/qwen4exp-qwen38-w5nl64-bf16trunk.recipe):
+#
+#   cd3     the recipe as written — coordinate descent three sweeps. The control.
+#   cd6     six sweeps: the same search, twice the passes.
+#   ao1     act_order=1 instead of cd: GPTQ sweeps columns in activation-importance order and
+#           stores them permuted. This is a SEPARATE arm, not a fourth sweep — libquant refuses
+#           cd= together with act_order=1 ("cd= sweeps the columns in their stored order, and
+#           act_order=1 stores them permuted; one or the other", libquant/lq_registry.cpp), so
+#           the two knobs cannot be combined into a best-of-both container.
+#
+# The arms are generated from ONE template with `--quant` overrides, not three near-duplicate
+# recipes: --quant rules are inserted BEFORE the recipe file and the first rule whose glob matches
+# a weight decides it (core/format/recipe.cpp: Recipe::match), so an override cleanly replaces the
+# expert line and the other 60 lines cannot drift between arms.
+#
+# Protected experts: apply-protected-cast.sh derives the bf16 set from THIS campaign's grams
+# (--target-count 48, two per layer so each rank's 256-expert half stays even) and splices it into
+# a per-run COPY of the template; the committed recipe keeps its markers and stays reviewable.
+#
+# Gates, in the order they bite — each one is a lesson the swift15 campaign paid for:
+#   inputs        checkpoint (dir + index), recipe, grams, imatrix, tokenizer. A missing grams
+#                 directory converts UNCALIBRATED with one stderr note, which is silently worse
+#                 than failing here.
+#   reference     the 335.3 GiB BF16 reference container must be GONE before the converts start.
+#                 It and the checkpoint together eat the volume; its job (the KLD recording and
+#                 the calibration grams) is finished by then, and both survive on disk without the
+#                 container beside it. The FP8-native anchor is FETCHED after the converts (disk
+#                 sequencing), so nothing here waits on it.
+#   plan          --plan-only runs FIRST, per arm, and prints "about X would be written". The
+#                 number is the bill: free space must cover it plus the 128k-prompt overshoot
+#                 margin, and the finished artifact is checked against it.
+#   outputs       no arm overwrites anything; move the old one to trash.
+#
+# Usage: convert-qwen38-w5nl64.sh [--arm cd3|cd6|ao1]... [--dry-run] [--plan-only]
+#                                 [--target-count N] [--keep-recipe]
+set -u -o pipefail
+
+SELF="$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)"
+IMAGE=${IMAGE:-stilldeadcode/radiance@sha256:d590e300234bf6d2f942a0b8d456d183153fd808f28820e6c2fa69c9389178ed}
+CKPT=${CKPT:-/home/dank/models/Qwen3.8-Flash-Next}
+CAMPAIGN=${CAMPAIGN:-/home/dank/campaign/qwen38-w5nl64}
+RECIPE=${RECIPE:-$SELF/recipes/qwen4exp-qwen38-w5nl64-bf16trunk.recipe}
+CALIB=${CALIB:-$CAMPAIGN/calib}
+IMATRIX=${IMATRIX:-$CAMPAIGN/imatrix.gguf}
+OUTDIR=${OUTDIR:-/home/dank/models/rad}
+BF16_RAD=${BF16_RAD:-$OUTDIR/qwen38-bf16.rad}   # the reference container, not an arm output
+STATE=${STATE:-$CAMPAIGN}                      # the dir the container mounts read-only: the
+                                               # spliced recipe and the imatrix live here
+TARGET_COUNT=${TARGET_COUNT:-48}
+MIN_GRAMS=${MIN_GRAMS:-24576}                 # per-expert DOWN grams: 48 recorded layers x 512
+                                               # experts. The checkpoint has 49 MoE layers, but the
+                                               # MTP layer is never recorded, so its 512 experts are
+                                               # absent from the tap's output by design
+MARGIN_MIB=${MARGIN_MIB:-40960}               # the long-prompt overshoot, off the plan number
+TS=$(date +%Y%m%d-%H%M%S)
+
+ARMS=()
+DRY=0
+PLAN_ONLY=0
+KEEP_RECIPE=0
+while [ "$#" -gt 0 ]; do
+    case "$1" in
+      --arm) [ "$#" -ge 2 ] || { echo "--arm needs a value (cd3|cd6|ao1)" >&2; exit 2; }
+             ARMS+=("$2"); shift 2 ;;
+      --dry-run) DRY=1; shift ;;
+      --plan-only) PLAN_ONLY=1; shift ;;
+      --target-count) [ "$#" -ge 2 ] || { echo "--target-count needs a value" >&2; exit 2; }
+             TARGET_COUNT="$2"; shift 2 ;;
+      --keep-recipe) KEEP_RECIPE=1; shift ;;
+      *) echo "unknown argument: $1" >&2; exit 2 ;;
+    esac
+done
+[ "${#ARMS[@]}" -gt 0 ] || ARMS=(cd3 cd6 ao1)
+for a in "${ARMS[@]}"; do
+    case "$a" in cd3|cd6|ao1) ;; *) echo "unknown arm: $a (cd3|cd6|ao1)" >&2; exit 2 ;; esac
+done
+
+# The expert rule's option set, spelled once. --quant replaces the template's line wholesale, so
+# every option has to be repeated here; `calib=$CALIB` is expanded by the recipe reader from the
+# container environment (core/format/recipe.cpp: expand refuses an unset name rather than taking
+# it for empty).
+expert_opts='table=w5nl,group=64,scale=fp8_e4m3,scale2=f32,block2=*x*,scale2_value=0.0001220703125,transform=fwht128,rule=search,calib=$CALIB'
+arm_quant() {
+    case "$1" in
+      # The control is the template itself: no override, so the recipe's own line decides.
+      cd3) return 0 ;;
+      cd6) echo "blk.*.ffn_*_exps.*.weight=gptq:${expert_opts},cd=6" ;;
+      # act_order=1 WITHOUT cd: libquant refuses the pair (lq_registry.cpp:105, "cd= sweeps the
+      # columns in their stored order, and act_order=1 stores them permuted; one or the other"), so
+      # this arm is the interleave knob alone, not cd=3 plus it.
+      ao1) echo "blk.*.ffn_*_exps.*.weight=gptq:${expert_opts},act_order=1" ;;
+    esac
+}
+
+# --- gates ------------------------------------------------------------------------------------
+[ -d "$CKPT" ] || { echo "no checkpoint at $CKPT; run scripts/download-qwen38-bf16.sh --fetch" >&2; exit 2; }
+[ -f "$CKPT/model.safetensors.index.json" ] || { echo "checkpoint at $CKPT has no safetensors index — a torn fetch" >&2; exit 2; }
+[ -f "$CKPT/tokenizer.json" ] || { echo "no tokenizer.json in $CKPT; the .rad cannot carry one without it" >&2; exit 2; }
+[ -f "$RECIPE" ] || { echo "no recipe at $RECIPE" >&2; exit 2; }
+GRAMS=$(find "$CALIB" -maxdepth 1 -name 'gram.r*.bin' 2>/dev/null | wc -l)
+[ "$GRAMS" -ge "$MIN_GRAMS" ] || {
+    echo "calib dir $CALIB holds $GRAMS gram files (<$MIN_GRAMS); run $0 after run-qwen38-bf16.sh calib" >&2; exit 2; }
+[ -f "$IMATRIX" ] || { echo "no imatrix at $IMATRIX; run scripts/grams-to-imatrix.py over $CALIB" >&2; exit 2; }
+# The container sees the campaign dir at one fixed mount point, so an imatrix kept elsewhere has
+# no path inside it: copy it in (or point IMATRIX at the campaign dir) rather than let the
+# converter fail halfway through a planned arm.
+[ "$(dirname "$IMATRIX")" = "$STATE" ] || {
+    echo "imatrix $IMATRIX is not in the campaign dir $STATE the container mounts; move it there" >&2; exit 2; }
+if [ -e "$BF16_RAD" ]; then
+    echo "the 335.3 GiB BF16 reference container $BF16_RAD is still on disk; its recording and" \
+         "the calibration grams are already taken, so move it to trash — the converts need its" \
+         "space (peak-disk law)" >&2
+    exit 2
+fi
+for a in "${ARMS[@]}"; do
+    [ -e "$OUTDIR/qwen38-w5nl64-$a.rad" ] && {
+        echo "$OUTDIR/qwen38-w5nl64-$a.rad already exists; move it to trash first" >&2; exit 2; }
+    # A killed run leaves its container behind (--rm only fires when podman itself
+    # exits), and the next arm then dies on "name already in use" AFTER the plan
+    # phase; refuse here, where the fix is one `podman rm -f`.
+    podman container exists "qwen38-convert-$a" && {
+        echo "container qwen38-convert-$a still exists (a killed run?);" \
+             "'podman rm -f qwen38-convert-$a' and re-run" >&2
+        exit 2; }
+done
+
+# --- the protected plane, spliced into a per-run copy of the template --------------------------
+mkdir -p "$STATE"
+WORK="$STATE/recipe-$TS.recipe"
+cp "$RECIPE" "$WORK"
+if [ "$DRY" = 1 ]; then
+    echo "would run: $SELF/apply-protected-cast.sh $CALIB $WORK $TARGET_COUNT"
+else
+    # Positional, because the tool reads its recipe and calib dir by position: passing them as
+    # environment variables would be silently ignored and it would rewrite the COMMITTED
+    # template in place instead of this run's copy.
+    "$SELF/apply-protected-cast.sh" "$CALIB" "$WORK" "$TARGET_COUNT" \
+        || { echo "the protected-expert derivation refused; the arms would convert unprotected" >&2; exit 1; }
+fi
+
+run_convert() { # run_convert ARM [extra flags...]
+    local arm="$1"; shift
+    local out="$OUTDIR/qwen38-w5nl64-$arm.rad" log="$STATE/convert-$arm-$TS.log"
+    local argv=(--rm --name "qwen38-convert-$arm"
+                -e "CALIB=/data/calib" -e "OMP_NUM_THREADS=$(nproc)"
+                -v "$(dirname "$CKPT"):/models:ro"
+                -v "$CALIB:/data/calib:ro"
+                -v "$STATE:/campaign:ro"
+                -v "$OUTDIR:/out")
+    local conv=(/models/"${CKPT##*/}" --recipe "/campaign/$(basename "$WORK")"
+                --tokenizer "/models/${CKPT##*/}/tokenizer.json"
+                --imatrix /campaign/"${IMATRIX##*/}"
+                # No --max-ctx: the context declare then runs at the checkpoint's training context
+                # (262144), which is what the shipped container did — rad-info --meta on it carries
+                # no override. A container declared at a smaller bound cannot be expected to serve
+                # the 200k window D2 holds, and the convert is not where to discover that.
+                # --kv-cache-dtype changes nothing in the artifact; it picks the kernels the declare
+                # checks, and the fleet serves fp8 KV, so an unresolvable fp8 kernel must be said no
+                # to by the convert, not by a boot on a drained window.
+                --kv-cache-dtype fp8 -o "/out/$(basename "$out")")
+    local q; q=$(arm_quant "$arm")
+    [ -n "$q" ] && conv+=(--quant "$q")
+    if [ "$DRY" = 1 ]; then
+        echo "podman run ${argv[*]} \$IMAGE --entrypoint /opt/radiance/bin/rad-convert ${conv[*]} $*"
+        return 0
+    fi
+    podman run "${argv[@]}" --entrypoint /opt/radiance/bin/rad-convert "$IMAGE" "${conv[@]}" "$@" \
+        2>&1 | tee "$log"
+    return "${PIPESTATUS[0]}"
+}
+
+# --- plan, then bill, then convert -------------------------------------------------------------
+overall=0
+for arm in "${ARMS[@]}"; do
+    echo "== arm $arm =="
+    if [ "$DRY" = 1 ]; then
+        # Both lines, in order: the plan runs before the convert, and the convert is what the
+        # operator is about to pay an afternoon for.
+        run_convert "$arm" --plan-only
+        run_convert "$arm"
+        continue
+    fi
+    rc=0; plan_log=$(run_convert "$arm" --plan-only) || rc=$?
+    [ "$rc" -eq 0 ] || { echo "arm $arm: --plan-only failed (rc=$rc); nothing was written" >&2; overall=1; continue; }
+    echo "$plan_log"
+    # "about 141.2 GiB would be written" is the converter's own bill; parse it rather than
+    # hardcoding a size, so the gate still works after any recipe edit.
+    want=$(printf '%s' "$plan_log" | sed -n 's/.*about \([0-9.]*\) \([KMGT]iB\) would be written.*/\1 \2/p' | tail -1)
+    [ -n "$want" ] || { echo "arm $arm: the plan printed no size; refusing to convert blind" >&2; overall=1; continue; }
+    need_mib=$(awk -v n="${want% *}" -v u="${want#* }" \
+        'BEGIN{ f = (u=="KiB") ? 2^-10 : (u=="MiB") ? 1 : (u=="GiB") ? 1024 : (u=="TiB") ? 1048576 : 0
+               if (f == 0) { exit 1 }; printf "%d", n*f + 0.5 }') || {
+        echo "arm $arm: the plan printed an unknown unit in '$want'" >&2; overall=1; continue; }
+    free_mib=$(df --output=avail -m "$OUTDIR" | tail -1 | tr -d ' ')
+    echo "plan: $want ($need_mib MiB); free ${free_mib} MiB on $OUTDIR (margin $MARGIN_MIB MiB)"
+    if [ "$free_mib" -lt $((need_mib + MARGIN_MIB)) ]; then
+        echo "arm $arm: needs ${need_mib} MiB + ${MARGIN_MIB} margin, ${free_mib} free; free space and re-run" >&2
+        overall=1; continue
+    fi
+    if [ "$PLAN_ONLY" = 1 ]; then echo "planned, not converted (--plan-only)"; continue; fi
+    rc=0; run_convert "$arm" || rc=$?
+    [ "$rc" -eq 0 ] || { echo "arm $arm: rad-convert rc=$rc; log: $STATE/convert-$arm-$TS.log" >&2; overall=1; continue; }
+    out="$OUTDIR/qwen38-w5nl64-$arm.rad"
+    [ -f "$out" ] || { echo "arm $arm: rad-convert exited 0 but wrote no $out; log: $STATE/convert-$arm-$TS.log" >&2; overall=1; continue; }
+    got_mib=$(( $(stat -c %s "$out") / 1048576 ))
+    # The receipt: the artifact is what the plan predicted, within the metadata the plan does not
+    # count. An arm that came out materially smaller lost a plane somewhere.
+    lo=$(( need_mib * 98 / 100 )); hi=$(( need_mib * 103 / 100 ))
+    if [ "$got_mib" -lt "$lo" ] || [ "$got_mib" -gt "$hi" ]; then
+        echo "arm $arm: $out is ${got_mib} MiB, the plan said ${need_mib} MiB (allowed ${lo}-${hi})" >&2
+        overall=1; continue
+    fi
+    echo "CONVERT_DONE $arm ${got_mib} MiB -> $out"
+done
+
+if [ "$KEEP_RECIPE" = 0 ] && [ "$DRY" = 0 ]; then
+    mkdir -p "$CAMPAIGN/.AGENTTRASH"
+    mv "$WORK" "$CAMPAIGN/.AGENTTRASH/$(basename "$WORK")" 2>/dev/null || true
+fi
+exit "$overall"

scripts/deploy-plan.py

@@ -34,7 +34,6 @@ SERVICE_ORDER = (
     "hermes",
     "hermes-dashboard",
     "hermes-webui",
-    "prometheus",
     "grafana",
     "loki",
     "alloy",
@@ -42,6 +41,9 @@ SERVICE_ORDER = (
     "cadvisor",
     "gpu-exporter",
     "alertmanager",
+    # The prober must answer before prometheus re-runs the services job.
+    "blackbox-exporter",
+    "prometheus",
     "authentik-postgres",
     "authentik-server",
     "authentik-worker",
@@ -97,6 +99,7 @@ DANKCLAW_SERVICES = {
     "cadvisor",
     "gpu-exporter",
     "alertmanager",
+    "blackbox-exporter",
     "authentik-postgres",
     "authentik-server",
     "authentik-worker",
@@ -277,6 +280,8 @@ def services_for_path(path: str, target: str) -> set[str]:
             # Provisioned datasources and dashboards: git is the source of
             # truth, so a change here must restart the service that reads them.
             _add(services, "grafana")
+        elif group == "blackbox":
+            _add(services, "blackbox-exporter")
         elif group == "homepage":
             # Homepage is a Dankclaw-owned deployment. Shared homepage files
             # are intentionally not copied during a Dankserver-only run.
@@ -380,7 +385,7 @@ def services_for_path(path: str, target: str) -> set[str]:
             # consumer (prometheus before grafana, loki before alloy).
             if len(parts) >= 4 and parts[2] == "quadlet":
                 name = parts[3].removesuffix(".container")
-                if name in {"prometheus", "grafana", "loki", "alloy", "node-exporter", "cadvisor", "alertmanager"}:
+                if name in {"prometheus", "grafana", "loki", "alloy", "node-exporter", "cadvisor", "alertmanager", "blackbox-exporter"}:
                     _add(services, name)
             return services
         if len(parts) >= 2 and parts[1] == "gitea":

scripts/deploy-remote.sh

@@ -86,7 +86,7 @@ service_is_allowed() {
         dankclaw:open-webui|dankclaw:gitea-postgres|dankclaw:gitea-server|\
         dankclaw:comfyui|dankclaw:hermes|\
         dankclaw:hermes-dashboard|dankclaw:hermes-webui|dankclaw:prometheus|dankclaw:grafana|\
-        dankclaw:loki|dankclaw:alloy|dankclaw:node-exporter|dankclaw:cadvisor|dankclaw:gpu-exporter|dankclaw:alertmanager|\
+        dankclaw:loki|dankclaw:alloy|dankclaw:node-exporter|dankclaw:cadvisor|dankclaw:gpu-exporter|dankclaw:alertmanager|dankclaw:blackbox-exporter|\
         dankclaw:authentik-postgres|dankclaw:authentik-server|\
         dankclaw:authentik-worker|dankclaw:authentik-outpost|dankclaw:authentik-outpost-ldap|dankclaw:anubis|\
         dankclaw:caddy|dankclaw:homepage|dankclaw:seerr|dankclaw:marreta|\
@@ -252,16 +252,24 @@ install_signal_relay() {
     # The relay binds the host's LAN IP (the one address a bridge container
     # can reach as host.containers.internal), so keep the unauthenticated
     # endpoint off the rest of the LAN: admit loopback, the host itself
-    # (its own health probe) and the podman bridge subnet only. Idempotent:
-    # -C before -I.
+    # (its own health probe) and the podman bridge subnet only.
+    # ORDER IS THE GATE: `iptables -I` inserts at position 1, so the old
+    # shape (-C/-I x3 ACCEPT, then -C/-I the DROP) landed the blanket DROP
+    # ABOVE the ACCEPTs and every probe timed out beside a LISTENing socket
+    # (live 2026-10-21: 159 packets on the DROP, 0 on each ACCEPT; the whole
+    # dankclaw drain aborted at the alertmanager gate twice before the cause
+    # was read off `nft list ruleset`). Rebuild the tail instead of inserting
+    # it: delete any blanket DROP wherever it sits, keep the three ACCEPTs
+    # at the head (-C before -I, unchanged), then APPEND the DROP so it is
+    # last and the ACCEPTs decide first. Converges on poisoned hosts.
+    run_privileged iptables -D INPUT -p tcp --dport 18093 -j DROP 2>/dev/null || true
     run_privileged iptables -C INPUT -p tcp --dport 18093 -s 127.0.0.1 -j ACCEPT 2>/dev/null \
         || run_privileged iptables -I INPUT -p tcp --dport 18093 -s 127.0.0.1 -j ACCEPT
     run_privileged iptables -C INPUT -p tcp --dport 18093 -s 192.168.0.143 -j ACCEPT 2>/dev/null \
         || run_privileged iptables -I INPUT -p tcp --dport 18093 -s 192.168.0.143 -j ACCEPT
     run_privileged iptables -C INPUT -p tcp --dport 18093 -s 10.89.0.0/16 -j ACCEPT 2>/dev/null \
         || run_privileged iptables -I INPUT -p tcp --dport 18093 -s 10.89.0.0/16 -j ACCEPT
-    run_privileged iptables -C INPUT -p tcp --dport 18093 -j DROP 2>/dev/null \
-        || run_privileged iptables -I INPUT -p tcp --dport 18093 -j DROP
+    run_privileged iptables -A INPUT -p tcp --dport 18093 -j DROP
     # The interpreter needs a moment to bind after restart; a one-shot curl
     # raced the socket on 2026-09-17 and failed a healthy deploy.
     local i
@@ -311,8 +319,45 @@ copy_loki_config() {
 }
 
 copy_alloy_config() {
-    run_privileged mkdir -p /opt/alloy/config
-    run_privileged cp shared/alloy/config.alloy /opt/alloy/config/config.alloy
+    # The repo config carries the token GRAFANA_HOST_LABEL; the installed copy
+    # carries the host's name, so Loki streams say WHICH machine's journal a
+    # line came from without a second config file per host.
+    # *_CONF_ROOT are test seams: the bats cases execute these copiers for
+    # real against a sandbox with a stubbed run_privileged.
+    local root="${ALLOY_CONF_ROOT:-/opt/alloy}" tmp
+    tmp="$(mktemp)"
+    sed "s/GRAFANA_HOST_LABEL/$TARGET/g" shared/alloy/config.alloy > "$tmp"
+    if ! grep -q "host = \"$TARGET\"," "$tmp" || grep -q "GRAFANA_HOST_LABEL" "$tmp"; then
+        echo "alloy config: host-label substitution failed" >&2
+        rm -f "$tmp"
+        return 1
+    fi
+    # Syntax gate: alloy's config language has no schema — a misplaced
+    # attribute (e.g. `on_error` outside the json stage) parses only at RUN
+    # time, and the unit then crash-loops with a healthy-looking deploy.
+    # Validate against the very image that will run it, before installing.
+    # Skipped only in the bats sandbox (no image there).
+    if [ "$root" = "/opt/alloy" ] || podman image exists localhost/franksdank/alloy:latest 2>/dev/null; then
+        if ! podman run --rm -v "$tmp:/c.alloy:ro" \
+                --entrypoint /usr/bin/alloy localhost/franksdank/alloy:latest \
+                validate /c.alloy; then
+            echo "alloy config: validate FAILED against the pinned image" >&2
+            rm -f "$tmp"
+            return 1
+        fi
+    fi
+    run_privileged mkdir -p "$root/config"
+    run_privileged cp "$tmp" "$root/config/config.alloy"
+    rm -f "$tmp"
+}
+
+copy_blackbox_config() {
+    # Single-file mount target: the unit binds this exact path into
+    # /etc/blackbox_exporter/config.yml. cp (not cp -r) keeps it one file.
+    # BLACKBOX_CONF_ROOT is the same test seam.
+    local root="${BLACKBOX_CONF_ROOT:-/opt/blackbox}"
+    run_privileged mkdir -p "$root"
+    run_privileged cp shared/blackbox/blackbox.yml "$root/blackbox.yml"
 }
 
 copy_grafana_provisioning() {
@@ -321,14 +366,25 @@ copy_grafana_provisioning() {
     # datasource would be invisible to review, lost with the data volume, and
     # unknown to anyone reading the repository. The unit already mounts
     # /opt/grafana/provisioning read-only, so this only has to populate it.
-    run_privileged mkdir -p /opt/grafana/provisioning/datasources \
-        /opt/grafana/provisioning/dashboards
+    # GRAFANA_PROV_ROOT is the test seam: point it at a sandbox and the whole
+    # copy runs unprivileged (the bats case executes this function for real).
+    local root="${GRAFANA_PROV_ROOT:-/opt/grafana}"
+    run_privileged mkdir -p "$root/provisioning/datasources" \
+        "$root/provisioning/dashboards"
     run_privileged cp shared/grafana/provisioning/datasources/prometheus.yml \
-        /opt/grafana/provisioning/datasources/prometheus.yml
+        "$root/provisioning/datasources/prometheus.yml"
     run_privileged cp shared/grafana/provisioning/dashboards/provider.yml \
-        /opt/grafana/provisioning/dashboards/provider.yml
-    run_privileged cp shared/grafana/provisioning/dashboards/llm-fleet.json \
-        /opt/grafana/provisioning/dashboards/llm-fleet.json
+        "$root/provisioning/dashboards/provider.yml"
+    # EVERY dashboard JSON rides, not a hand-listed subset: the fleet provider
+    # turns each file in the directory into a dashboard, so a copier that names
+    # files one by one silently drops new boards (measured 2026-10-21: the
+    # containers board existed in the repo since its commit and was NEVER
+    # provisioned — the panel-census browser check could not find it while the
+    # hand-placed older boards happened to be on disk). Same rule as the quadlet
+    # tier: the directory is the contract, the copy is generated state.
+    for dash in shared/grafana/provisioning/dashboards/*.json; do
+        run_privileged cp "$dash" "$root/provisioning/dashboards/$(basename "$dash")"
+    done
 }
 
 render_caddyfile() {
@@ -552,6 +608,9 @@ deploy_observability() {
             ;;
         cadvisor)
             ;;
+        blackbox-exporter)
+            copy_blackbox_config
+            ;;
         *)
             echo "unknown observability service: $service" >&2
             return 2
@@ -566,6 +625,9 @@ deploy_observability() {
         loki) wait_for_health loki http://127.0.0.1:3100/ready 60 ;;
         node-exporter) wait_for_health node-exporter http://127.0.0.1:9100/metrics 60 ;;
         cadvisor) wait_for_health cadvisor http://127.0.0.1:8080/healthz 90 ;;
+        # Loopback: the exporter binds 127.0.0.1 + LAN (the gateway address
+        # is not bindable from the host netns — see the unit header).
+        blackbox-exporter) wait_for_health blackbox-exporter http://127.0.0.1:9115/-/healthy 60 ;;
     esac
 }
 
@@ -1401,9 +1463,10 @@ deploy_anubis() {
 # conversion (2026-10-11) of the abliterated Qwen3.8-Flash-Next —
 # qwen38-flash-next-v2-abliterated.hgn (66,687,678,432 B, ~62 GiB resident)
 # + the REQUIRED n-gram lookup table + the vision tower — MTP spec decode
-# (HALOGEN_SPEC_ADAPT=0), 2 slots over ONE 401,408-position KV pool (2 × ctx,
-# raised 2026-10-18 so BOTH slots can hold a maximum-length request) at the
-# ctx 200704 ceiling, image input — on the Strix Halo iGPU. It stays a
+# (HALOGEN_SPEC_ADAPT=0), 2 slots over ONE 401,408-position KV pool at the
+# native ctx 262144 ceiling (raised 2026-10-21 from 200704 so the harness can
+# run a 200,000 window with its ~45k fit gap; the pool still carries two
+# 200k conversations at once), image input — on the Strix Halo iGPU. It stays a
 # ROOT unit because the serve script drives `podman run` directly (the frank-archon pattern):
 # relink_unit points /etc/systemd/system/frank-dragoon.service at the repo
 # file, boot persistence is that unit's [Install], and LimitMEMLOCK=infinity
@@ -2033,7 +2096,7 @@ deploy_service() {
         plex) deploy_plex ;;
         tautulli) deploy_tautulli ;;
         alloy) deploy_alloy ;;
-        prometheus|grafana|loki|node-exporter|cadvisor|alertmanager) deploy_observability "$service" ;;
+        prometheus|grafana|loki|node-exporter|cadvisor|alertmanager|blackbox-exporter) deploy_observability "$service" ;;
         gpu-exporter) deploy_gpu_exporter ;;
         comfyui) deploy_comfyui ;;
         *) deploy_compose_service "$service" ;;

scripts/deploy.sh

@@ -133,13 +133,15 @@ fi
 valid_service() {
     local service="$1"
     case "$TARGET:$service" in
+        # dankclaw:searxng-network is client-only: a network quadlet with no
+        # remote dispatch handler, so the remote gate does not (must not) list it.
         dankclaw:caddy|dankclaw:gluetun|dankclaw:searxng-network|dankclaw:searxng-valkey|\
         dankclaw:searxng|dankclaw:marreta|dankclaw:pihole|dankclaw:open-webui|dankclaw:gitea-postgres|\
         dankclaw:gitea-server|dankclaw:homepage|\
         dankclaw:comfyui|dankclaw:hermes|dankclaw:hermes-dashboard|\
         dankclaw:hermes-webui|dankclaw:prometheus|dankclaw:grafana|dankclaw:loki|dankclaw:alloy|\
-        dankclaw:alertmanager|\
-        dankclaw:node-exporter|dankclaw:authentik-postgres|dankclaw:authentik-server|\
+        dankclaw:alertmanager|dankclaw:blackbox-exporter|\
+        dankclaw:node-exporter|dankclaw:cadvisor|dankclaw:gpu-exporter|dankclaw:authentik-postgres|dankclaw:authentik-server|\
         dankclaw:authentik-worker|dankclaw:authentik-outpost|dankclaw:authentik-outpost-ldap|dankclaw:anubis|dankclaw:seerr|\
         dankclaw:open-notebook|\
         dankclaw:frank-tts-engine|dankclaw:frank-tts|\
@@ -147,8 +149,8 @@ valid_service() {
         dankserver:romm-mariadb|dankserver:romm-redis|dankserver:romm|\
         dankserver:copyparty|dankserver:jellyfin|dankserver:calibre-web|dankserver:calibre-server|dankserver:plex|dankserver:tautulli|\
         dankserver:qbittorrent|dankserver:radarr|dankserver:sonarr|\
-        dankserver:lidarr|dankserver:readarr|dankserver:node-exporter|\
-        dankserver:frank-archon|\
+        dankserver:lidarr|dankserver:readarr|dankserver:node-exporter|dankserver:cadvisor|dankserver:gpu-exporter|\
+        dankserver:prowlarr|dankserver:frank-archon|\
         frankbook-pro:gluetun|frankbook-pro:qbittorrent|frankbook-pro:prowlarr|\
         frankbook-pro:radarr|frankbook-pro:sonarr|frankbook-pro:lidarr|\
         frankbook-pro:readarr)

scripts/derive-protected-experts.py

@@ -20,9 +20,16 @@ of the Gram is its down-input energy. Rank halves: 0-255, 256-511
 Output: the cast-rule lines (recipe format) on stdout, a report on stderr.
 
 Usage: derive-protected-experts.py <calib dir> [--min-share 0.03]
-       [--min-route-mult 3.0]
+       [--min-route-mult 3.0] [--target-count 0]
+       --target-count N fills the byte bill: the bars pick the base set exactly as
+       the bar-only pass does, then the energy ranking admits experts one at a time
+       (each with its parity partner) until the count lands. The energy bar NEVER
+       relaxes, so asking for 48 bf16 experts never quietly buys 80; the ratio bar
+       walks down only when the ranking cannot produce that many candidates.
+       0 takes the bars as given.
 """
 import argparse
+import collections
 import glob
 import os
 import re
@@ -32,6 +39,7 @@ import sys
 import numpy as np
 
 NAME_RE = re.compile(r"^gram\.r(\d)\.blk\.(\d+)\.ffn_down_exps\.(\d+)\.bin$")
+CAST_RE = re.compile(r"^blk\.(\d+)\.ffn_\*_exps\.(\d+)\.weight")
 
 
 def read_gram(path):
@@ -49,27 +57,8 @@ def read_gram(path):
     return k, rank, int(rows), tr
 
 
-def main():
-    ap = argparse.ArgumentParser()
-    ap.add_argument("calib_dir")
-    ap.add_argument("--min-share", type=float, default=0.03)
-    ap.add_argument("--min-route-mult", type=float, default=3.0)
-    a = ap.parse_args()
-
-    # layer -> global expert -> (rank, rows, energy)
-    data = {}
-    for path in sorted(glob.glob(os.path.join(
-            a.calib_dir, "gram.r*.ffn_down_exps.*.bin"))):
-        m = NAME_RE.match(os.path.basename(path))
-        if not m:
-            continue
-        rank, layer, expert = int(m.group(1)), int(m.group(2)), int(m.group(3))
-        k, _, rows, tr = read_gram(path)
-        data.setdefault(layer, {})[expert] = (rank, rows, tr)
-
-    if not data:
-        raise SystemExit("no per-expert down grams found")
-
+def select(data, min_share, min_route_mult):
+    """(cast lines, report lines, count) for one pair of bars."""
     lines, report = [], []
     total = 0
     for layer in sorted(data):
@@ -83,7 +72,7 @@ def main():
         for e, (rank, rows, tr) in experts.items():
             es, rs = tr / tot_e, rows / max(tot_r, 1)
             rec = (e, rank, es, rs)
-            if es >= a.min_share and es >= a.min_route_mult * rs:
+            if es >= min_share and es >= min_route_mult * rs:
                 qual.append(rec)
             elif es > 0:
                 pool.append(rec)
@@ -107,10 +96,148 @@ def main():
                           f"energy {es*100:5.2f}%, routing {rs*100:5.3f}%, "
                           f"ratio {es/max(rs,1e-12):6.1f}")
         total += len(chosen)
+    return lines, report, total
+
+
+def parity_repair(chosen, by_half, report, only=None):
+    """Add the next-best candidate to every half — or to `only` — holding an odd count.
+
+    The four-bit GEMM takes each half's remainder as two equal tables, so an odd count
+    of bf16 experts on a half is not a style problem: the two tables would not match.
+    `by_half` is energy-descending, so the first unchosen candidate is the next most
+    energetic, which is the expert worth keeping. Returns how many were added.
+    """
+    added = 0
+    for key in ([only] if only is not None else sorted(by_half)):
+        layer, half = key
+        n = sum(1 for (l, x) in chosen if l == layer and x // 256 == half)
+        if n % 2 == 0:
+            continue
+        for e, es, rs in by_half.get(key, []):
+            if (layer, e) not in chosen:
+                chosen[(layer, e)] = (es, rs)
+                added += 1
+                report.append(f"blk.{layer}: half {half} odd -> added e{e} "
+                              f"(energy {es*100:.2f}%)")
+                break
+        else:
+            report.append(f"blk.{layer}: half {half} holds {n} and no candidate is "
+                          f"left to make it even")
+    return added
+
+
+def select_to_count(data, min_share, min_route_mult, target):
+    """(lines, report, total, cutoff) for the set that LANDS the count.
+
+    The bars decide the base set; the count is then filled from the energy ranking,
+    one expert at a time, each addition taking its half's next-best candidate with it
+    so every rank's half stays even. Only the RATIO bar walks down to find candidates
+    — it is the law that says an expert is disproportionate and not merely busy, and
+    the caller's byte bill has to be filled — while the ENERGY bar holds, because that
+    bar is what the extra GiB of bf16 is supposed to buy. `cutoff` is the energy share
+    the set bottoms out at: the bar this campaign actually converted with.
+    """
+    report = []
+    shares, ranked = {}, []
+    for layer in sorted(data):
+        experts = data[layer]
+        tot_e = sum(v[2] for v in experts.values())
+        tot_r = sum(v[1] for v in experts.values())
+        if tot_e <= 0 or tot_r <= 0:
+            report.append(f"blk.{layer}: empty recording, skipped")
+            continue
+        for e, (_rank, rows, tr) in experts.items():
+            es, rs = tr / tot_e, rows / max(tot_r, 1)
+            if es <= 0:
+                continue
+            shares[(layer, e)] = (es, rs)
+            ranked.append((es, layer, e, rs))
+    ranked.sort(key=lambda r: (-r[0], r[1], r[2]))
+    by_half = {}
+    for es, layer, e, rs in ranked:
+        by_half.setdefault((layer, e // 256), []).append((e, es, rs))
+
+    chosen = {k: v for k, v in shares.items()
+              if v[0] >= min_share and v[0] >= min_route_mult * v[1]}
+    total = len(chosen)
+    total += parity_repair(chosen, by_half, report)
+
+    mult = min_route_mult
+    while True:
+        pool = [(es, l, e, rs) for es, l, e, rs in ranked
+                if es >= mult * rs and (l, e) not in chosen]
+        if len(pool) + total >= target or mult <= 0.0:
+            break
+        mult -= 0.5
+        report.append(f"ratio bar {mult + 0.5:g} leaves {len(pool)} fill candidates "
+                      f"for {target - total} slots; relaxing toward 0")
+    for es, layer, e, rs in pool:
+        if total >= target:
+            break
+        if (layer, e) in chosen:
+            continue
+        chosen[(layer, e)] = (es, rs)
+        total += 1
+        report.append(f"blk.{layer}: count fill -> added e{e} (energy {es*100:.2f}%)")
+        total += parity_repair(chosen, by_half, report, only=(layer, e // 256))
+    lines, cutoff = [], None
+    for (layer, e) in sorted(chosen):
+        es, rs = chosen[(layer, e)]
+        cutoff = es if cutoff is None else min(cutoff, es)
+        lines.append(f"blk.{layer}.ffn_*_exps.{e}.weight         cast dtype=bf16")
+    return lines, report, total, cutoff
+
+
+def main():
+    ap = argparse.ArgumentParser()
+    ap.add_argument("calib_dir")
+    ap.add_argument("--min-share", type=float, default=0.03)
+    ap.add_argument("--min-route-mult", type=float, default=3.0)
+    ap.add_argument("--target-count", type=int, default=0)
+    a = ap.parse_args()
+
+    # layer -> global expert -> (rank, rows, energy)
+    data = {}
+    for path in sorted(glob.glob(os.path.join(
+            a.calib_dir, "gram.r*.ffn_down_exps.*.bin"))):
+        m = NAME_RE.match(os.path.basename(path))
+        if not m:
+            continue
+        rank, layer, expert = int(m.group(1)), int(m.group(2)), int(m.group(3))
+        k, _, rows, tr = read_gram(path)
+        data.setdefault(layer, {})[expert] = (rank, rows, tr)
+
+    if not data:
+        raise SystemExit("no per-expert down grams found")
+
+    if a.target_count > 0:
+        lines, report, total, cutoff = select_to_count(
+            data, a.min_share, a.min_route_mult, a.target_count)
+        bar = f"energy cutoff {cutoff:.3g}" if cutoff is not None else "no candidates"
+    else:
+        lines, report, total = select(data, a.min_share, a.min_route_mult)
+        bar = f"min-share {a.min_share:.3g}"
+
+    # The emitted lines ARE the contract with the four-bit GEMM: every rank's half of
+    # every layer must hold an even number of bf16 experts. Judged on the output and
+    # not on the internal sets, so a slip in either pass is caught here rather than at
+    # engine load time.
+    halves = collections.Counter()
+    for rule in lines:
+        m = CAST_RE.match(rule)
+        if m:
+            halves[(int(m.group(1)), int(m.group(2)) // 256)] += 1
+    odd = {key: n for key, n in halves.items() if n % 2}
+    if odd:
+        print("PARITY LAW BROKEN: " + ", ".join(
+            f"blk.{lay} half {half} holds {n} protected experts"
+            for (lay, half), n in sorted(odd.items())), file=sys.stderr)
+        raise SystemExit(2)
 
     print("\n".join(lines))
     print("\n".join(report), file=sys.stderr)
-    print(f"total protected: {total}", file=sys.stderr)
+    print(f"total protected: {total} (target {a.target_count or 'none'}, {bar}, "
+          f"min-route-mult {a.min_route_mult:g})", file=sys.stderr)
 
 
 if __name__ == "__main__":

scripts/download-qwen38-bf16.sh

@@ -0,0 +1,249 @@
+#!/usr/bin/env bash
+# download-qwen38-bf16.sh — RUNS ON DANKSERVER. Fetch and verify the vendor
+# BF16 checkpoint Qwen/Qwen3.8-Flash-Next: the source the five-bit campaign
+# converts, and the source the KLD reference is recorded from.
+#
+# Why the base and not a finetune: the campaign converts to compare against the
+# publisher's own iq4r container, and that container was built from this
+# checkpoint. Base-in, base-out; anything else confounds the gate. (The
+# predecessor campaign ran ukisai/Swift-1.5-Qwen3.8-Flash-Next-BF16, whose key
+# set is identical to this repo's but whose weights are not — none of its
+# calibration grams, KLD reference or protected list carries over.)
+#
+# Why re-fetch instead of reusing: every BF16 copy of this family was destroyed
+# in the 2026-10-15 storage purge — only the 113.6 GiB served .rad survived.
+# 335.3 GiB of safetensors, 131 shards, ungated (the predecessor campaign
+# verified the key set against this repo, so no gate letter is needed).
+#
+# Verification is four gates deep, because the failure this repo actually lived
+# through is a fetch that reported success: `curl -o FILE URL` exits 0 on an
+# HTTP 401/403 and writes the error body as the artifact — three shards of
+# identical tiny size and a cheerful DONE in 20 seconds for 104 GiB.
+#
+#   1. shard census: the file count must equal the expected shard count, so a
+#      partially fetched fragment tree (the .partNofM trap) cannot pass.
+#   2. per-shard completeness, from the shard's OWN safetensors header: the
+#      header declares where the tensor data ends, so every shard's true size is
+#      known offline. A torn tail is named here, by file, instead of surfacing
+#      as a load error five hours later.
+#   3. index coverage: every shard model.safetensors.index.json references
+#      exists and is named.
+#   4. total bytes against the number the Hub declared, then, unless --no-hub,
+#      every file's sha256 against the Hub's LFS record — including the shards
+#      already on disk, so a previous partial download is never trusted.
+#
+# The token never enters a command line (the 2026-09-16 leak law): it is read
+# into HF_TOKEN in the environment, where a local process listing is the only
+# reader. This repo is ungated, so the fetch simply runs without one when no
+# token file exists. scripts/hfget.sh is curl+jq only — no Python, no venv, no
+# hf_transfer — and it drops the pickle/ONNX weight duplicates by itself.
+#
+# usage: download-qwen38-bf16.sh [--fetch] [--no-hub] [--dir DIR]
+#                                [--shards N] [--bytes B] [--repo ID]
+#   --fetch    download first (resumable: re-run to continue), then verify
+#   --no-hub   skip the Hub-side sha256 sweep; the other gates always run
+# exit: 0 verified, 1 verification failed, 2 environment / usage error
+set -euo pipefail
+
+REPO="${FRANK_QWEN38_HF_REPO:-Qwen/Qwen3.8-Flash-Next}"
+LOCAL_DIR="${FRANK_QWEN38_CKPT:-/home/dank/models/Qwen3.8-Flash-Next}"
+# Hub census of 2026-10-27: 144 files, 131 of them .safetensors, together
+# 360,023,351,514 bytes; the shards alone are 360,000,192,888. This gate counts
+# shards only, so a config file edited on the Hub cannot fail it.
+EXPECTED_SHARDS="${FRANK_QWEN38_SHARDS:-131}"
+EXPECTED_BYTES="${FRANK_QWEN38_BYTES:-360000192888}"
+MIN_FREE_MIB="${FRANK_QWEN38_MIN_FREE_MIB:-350000}"
+
+FETCH=0
+HUB=1
+while [ $# -gt 0 ]; do
+    case "$1" in
+        --fetch) FETCH=1 ;;
+        --no-hub) HUB=0 ;;
+        --dir)    LOCAL_DIR="$2"; shift ;;
+        --shards) EXPECTED_SHARDS="$2"; shift ;;
+        --bytes)  EXPECTED_BYTES="$2"; shift ;;
+        --repo)   REPO="$2"; shift ;;
+        *) echo "usage: ${0##*/} [--fetch] [--no-hub] [--dir DIR] [--shards N] [--bytes B] [--repo ID]" >&2
+           exit 2 ;;
+    esac
+    shift
+done
+
+SELF_DIR="$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)"
+FAILS=0
+note() { echo "$@"; FAILS=$((FAILS + 1)); }
+
+if [ "$FETCH" = "1" ]; then
+    # Every fetch gate is a hard stop: skipping the download silently and then
+    # verifying a half tree is how a campaign ships a stale payload.
+    HFGET="$SELF_DIR/hfget.sh"
+    if [ ! -f "$HFGET" ]; then
+        echo "REFUSED: the fetch helper is not here: $HFGET" >&2
+        exit 2
+    fi
+    if ! command -v jq >/dev/null 2>&1; then
+        echo "REFUSED: hfget.sh needs jq on PATH" >&2
+        exit 2
+    fi
+    mkdir -p "$LOCAL_DIR"
+    # The campaign's peak-disk law starts at the fetch: the checkpoint and the
+    # reference container built from it are ~335 GiB each, and a tree that does
+    # not fit is discovered hours into a transfer, when it costs the most.
+    free_mib=$(df -m --output=avail "$LOCAL_DIR" | tail -1 | tr -dc '0-9')
+    if [ "${free_mib:-0}" -lt "$MIN_FREE_MIB" ]; then
+        echo "REFUSED: $free_mib MiB free at $LOCAL_DIR; the fetch wants $MIN_FREE_MIB MiB free" >&2
+        echo "         (retire the peak-disk predecessors first — nothing is downloaded, so" >&2
+        echo "         nothing is left half-written)" >&2
+        exit 2
+    fi
+    echo "fetch: $REPO -> $LOCAL_DIR ($free_mib MiB free)"
+    if [ -z "${HF_TOKEN:-}" ]; then
+        CREDSTORE="$HOME/.config/huggingface/credentials/https://huggingface.co"
+        if [ -f "$CREDSTORE" ]; then
+            HF_TOKEN="$(awk '/^ OauthToken: / {print $2; exit}' "$CREDSTORE")"
+        elif [ -f "$HOME/.cache/huggingface/token" ]; then
+            # The legacy single-line token file the hub library writes.
+            HF_TOKEN="$(tr -d '[:space:]' < "$HOME/.cache/huggingface/token")"
+        elif [ -f "$HOME/.hf_token" ]; then
+            HF_TOKEN="$(tr -d '[:space:]' < "$HOME/.hf_token")"
+        fi
+    fi
+    if [ -n "${HF_TOKEN:-}" ]; then
+        export HF_TOKEN
+    fi
+    bash "$HFGET" "$REPO" "$LOCAL_DIR" main
+    unset HF_TOKEN
+fi
+
+if [ ! -d "$LOCAL_DIR" ]; then
+    echo "REFUSED: $LOCAL_DIR does not exist (run with --fetch, or pass --dir)" >&2
+    exit 1
+fi
+INDEX="$LOCAL_DIR/model.safetensors.index.json"
+if [ ! -f "$INDEX" ]; then
+    echo "REFUSED: $INDEX is missing — the checkpoint tree is not addressable without it" >&2
+    exit 1
+fi
+
+echo "== gate 1: shard census ($LOCAL_DIR)"
+SHARDS=$(find "$LOCAL_DIR" -maxdepth 1 -type f -name '*.safetensors' | wc -l)
+if [ "$SHARDS" != "$EXPECTED_SHARDS" ]; then
+    note "FAIL SHARD_COUNT: $SHARDS safetensors files on disk, expected $EXPECTED_SHARDS"
+else
+    echo "  ok: $SHARDS shards"
+fi
+
+echo "== gate 2: every shard is whole, judged by its own header"
+# A safetensors file is [u64 header length][JSON header][tensor data]. The JSON
+# declares where the data ends, so the file's true size is computable offline:
+# 8 + header + the largest end offset. Truncation and error bodies both fall
+# here, and each is reported with the file that caused it.
+while IFS= read -r f; do
+    state=$(python3 - "$f" <<'PY'
+import json, os, struct, sys
+path = sys.argv[1]
+size = os.path.getsize(path)
+try:
+    with open(path, "rb") as fh:
+        (hlen,) = struct.unpack("<Q", fh.read(8))
+        if hlen == 0 or hlen > size:
+            print("NOT_SAFETENSORS")
+            raise SystemExit
+        header = json.loads(fh.read(hlen))
+except Exception:
+    print("NOT_SAFETENSORS")
+    raise SystemExit
+if not header:
+    print("NOT_SAFETENSORS")
+    raise SystemExit
+end = max((v["data_offsets"][1] for v in header.values()
+           if isinstance(v, dict) and "data_offsets" in v), default=0)
+want = 8 + hlen + end
+print("OK" if want == size else f"TRUNCATED want={want} have={size}")
+PY
+)
+    case "$state" in
+        OK) ;;
+        NOT_SAFETENSORS*) note "FAIL NOT_SAFETENSORS: $(basename "$f") is not a safetensors file (an error body written as an artifact?)" ;;
+        TRUNCATED*) note "FAIL TRUNCATED: $(basename "$f") $state" ;;
+    esac
+done < <(find "$LOCAL_DIR" -maxdepth 1 -type f -name '*.safetensors' | sort)
+if [ "$FAILS" -eq 0 ]; then
+    echo "  ok: every shard's data extent matches its size on disk"
+fi
+
+echo "== gate 3: the index's shards are all present"
+INDEX_SHARDS=$(python3 - "$INDEX" <<'PY'
+import json, sys
+map_ = json.load(open(sys.argv[1]))["weight_map"]
+print("\n".join(sorted(set(map_.values()))))
+PY
+)
+while IFS= read -r name; do
+    [ -n "$name" ] || continue
+    if [ ! -f "$LOCAL_DIR/$name" ]; then
+        note "FAIL MISSING: $name is named by the index and is not on disk"
+    fi
+done <<< "$INDEX_SHARDS"
+if [ "$FAILS" -eq 0 ]; then
+    echo "  ok: $(printf '%s\n' "$INDEX_SHARDS" | grep -c .) referenced shards on disk"
+fi
+
+echo "== gate 4: total bytes vs the Hub's declaration"
+TOTAL=$(find "$LOCAL_DIR" -maxdepth 1 -type f -name '*.safetensors' -printf '%s\n' | awk '{s+=$1} END{print s+0}')
+if [ "$TOTAL" != "$EXPECTED_BYTES" ]; then
+    note "FAIL TOTAL_BYTES: shards total $TOTAL bytes, expected $EXPECTED_BYTES"
+else
+    echo "  ok: $TOTAL bytes"
+fi
+
+if [ "$HUB" = "1" ]; then
+    echo "== gate 5: every file's sha256 vs the Hub's LFS record"
+    # size = file size; lfs.sha256 = sha256 of the file itself (LFS or not).
+    TMPD="$(mktemp -d)"
+    trap 'rm -rf "$TMPD"' EXIT
+    MANIFEST="$TMPD/tree.json"
+    PLAN="$TMPD/plan"
+    curl -sf "https://huggingface.co/api/models/$REPO/tree/main?recursive=true&expand=true" \
+        -o "$MANIFEST" || { echo "FAIL: the Hub API call failed — refusing to report success" >&2; exit 1; }
+    python3 - "$MANIFEST" > "$PLAN" <<'PY'
+import json, sys
+for e in json.load(open(sys.argv[1])):
+    if e.get("type") != "file":
+        continue
+    sha = (e.get("lfs") or {}).get("sha256") or e.get("sha") or ""
+    if sha:
+        print(e["path"], e["size"], sha)
+PY
+    BAD=0
+    while read -r rel size sha; do
+        [ -n "${rel:-}" ] || continue
+        if [ ! -f "$LOCAL_DIR/$rel" ]; then
+            note "FAIL MISSING: $rel is on the Hub and not on disk"
+            continue
+        fi
+        have=$(stat -c %s "$LOCAL_DIR/$rel")
+        if [ "$have" != "$size" ]; then
+            note "FAIL SIZE: $rel is $have bytes, Hub says $size"
+            continue
+        fi
+        got=$(sha256sum "$LOCAL_DIR/$rel" | awk '{print $1}')
+        if [ "$got" != "$sha" ]; then
+            note "FAIL SHA256: $rel got $got want $sha"
+            BAD=$((BAD + 1))
+        fi
+    done < "$PLAN"
+    if [ "$BAD" -eq 0 ] && [ "$FAILS" -eq 0 ]; then
+        echo "  ok: sha256 matches for every file"
+    fi
+    rm -rf "$TMPD"
+    trap - EXIT
+fi
+
+if [ "$FAILS" -gt 0 ]; then
+    echo "VERIFY_FAILED: $FAILS problem(s) in $LOCAL_DIR" >&2
+    exit 1
+fi
+echo "VERIFY_OK: $SHARDS shards, $TOTAL bytes of safetensors in $LOCAL_DIR"
+echo "NEXT: build the BF16 reference container with scripts/build-qwen38-bf16-rad.sh"

scripts/frank-archon.env

@@ -74,6 +74,13 @@ FRANK_ARCHON_PLACEMENT=expert_tiered
 # residency -- ~3.2 GiB a card of experts now lives in host RAM and is read over the
 # x8 link when the router picks it. The ledger's coverage line is the referee, not
 # this file: if vram_kv comes back under 700,000 tokens, raise this 1024 and redeploy.
+# 2026-10-29 A/B, measured on both payloads and folded back the same day: at 66,560 the
+# endpoint printed the identical `elastic 26.73 GiB / experts 20.72 / kv cache 5.64 /
+# vram_weights 24.30` it had printed at 49,152, opening the bigger pool with `0 B used`;
+# the campaign rig (bf16, same step) moved `vram_kv 269.99 -> 271.25 MiB`, a rounding
+# step — its window is fixed by its own n-gram table (95.37 GiB) and expert plane, not
+# by this pool, so the refused calibration documents were fixed by cutting them to the
+# window (scripts/rechunk-kld-corpus.py), not by widening the pool.
 FRANK_ARCHON_HOST_POOL_MIB=49152
 # VRAM left unclaimed per card (publisher default 96 MiB); the engine's settle
 # floor is built from it.
@@ -137,7 +144,8 @@ FRANK_ARCHON_NGRAM_PLACEMENT=ram
 # itself before refusing by name, so recovery is a boot, not a visit.
 # Floor = 2 x HOST_POOL + 48,824 (static weights + arena) + PREFIX_CACHE + MNBT + the
 # auto checkpoint store (MAX_SEQS x 10 slots of 111.98 MiB), per the bats ledger test;
-# at the 49,152 pool with 4 slots that computes to 161,848 MiB.
+# at the 49,152 pool with 4 slots that computes to 161,848 MiB (196,664 at the
+# 66,560 pool the 2026-10-29 A/B rejected).
 FRANK_ARCHON_TTM_MIN_MIB=162000
 FRANK_ARCHON_TTM_TARGET_MIB=204800
 # Escape hatch for experiments. Unset -> no extra argv; the serve script expands

scripts/frank-dragoon.env

@@ -49,14 +49,21 @@ FRANK_DRAGOON_TOKENIZER_FILES="tokenizer.json chat_template.jinja tokenizer_conf
 # stays at the image default, which is what was qualified) ------------------
 # Served id. OpenWebUI api_configs model_ids for the :7770 backend MUST name it.
 HALOGEN_MODEL_ID=frank-dragoon
-# PER-REQUEST ceiling: a longer prompt is a 400 naming the limit, never a
-# truncation. The harness window must sit ~45k UNDER this (the 400-zone law on
-# the page) — a window equal to the ceiling makes every long request a 400.
-HALOGEN_CTX=200704
-# ONE pool behind both slots, sized so BOTH can carry a maximum-length request
-# at once. Pool positions buy concurrency; the per-request window never moves
-# with them. Legal band: ctx <= pool <= slots * ctx (the seam enforces it), and
-# RAM the pool takes is RAM the n-gram table's page cache loses.
+# PER-REQUEST ceiling, set to the checkpoint's NATIVE window (262,144 — a
+# longer prompt is a 400 naming the limit, never a truncation; stretching
+# past native would need HALOGEN_ROPE_YARN and is not on plan). The harness
+# `contextWindow` must sit ~45k under THIS number (the 400-zone law on the
+# page): with the window at 200,000 the gap is 62,144, which absorbs the
+# chars/4 under-count. A window at or near the ceiling makes every long
+# request a guaranteed 400.
+HALOGEN_CTX=262144
+# ONE pool behind both slots. The harness window is 200,000, so 401,408
+# still carries TWO maximum harness-length conversations at once
+# (2 x 200,000 + one 45,056 budget <= pool + cap headroom); two literal
+# ceiling-length requests would queue instead — buying those means raising
+# the pool, and RAM the pool takes is RAM the n-gram table's page cache
+# loses. Pool positions buy concurrency; the per-request window never moves
+# with them. Legal band: ctx <= pool <= slots * ctx (the seam enforces it).
 HALOGEN_KV_POOL_POSITIONS=401408
 # Two concurrent conversations: the MTP drafter is not slot-locked on this
 # engine, so the second slot is capacity, not a speculation tax.
@@ -107,10 +114,13 @@ HALOGEN_NPU_MODELS=qwen3-embedding-0.6b
 FRANK_DRAGOON_NPU_DIR=/home/dank/models/halogen-v2-abliterated/npu
 # Node paths and the XRT directory are variables so the bats seam can drive
 # these gates on a box with no NPU. Ubuntu's distro XRT (libxrt2 + libxrt-npu2)
-# is mounted FILE BY FILE and TWICE EACH — once where the image looks for XRT,
-# once at its own path, because those libraries find each other at their install
-# path; the vendor forbids mounting an /opt/xilinx/xrt of symlinks into the
-# system dir, which dangle inside the container.
+# is mounted FILE BY FILE and THREE TIMES EACH — where the image looks for XRT
+# (/opt/xilinx/xrt/lib), at its own path (those libraries find each other at
+# their install path), and at /opt/xilinx/lib/<multiarch>/ (the NPU child
+# searches that third prefix; without it the lane dies naming it and the
+# supervisor takes the container down — 2026-10-21 boot loop). The basename of
+# this directory IS the multiarch name. The vendor forbids mounting an
+# /opt/xilinx/xrt of symlinks into the system dir, which dangle in the container.
 FRANK_DRAGOON_DEV_ACCEL=/dev/accel/accel0
 FRANK_DRAGOON_XRT_DIR=/usr/lib/x86_64-linux-gnu
 # sysfs root the fabric-clock preflight reads (same knob the vendor's

scripts/frank-dragoon.service

@@ -1,5 +1,5 @@
 [Unit]
-Description=halogen-flash-server container (Quat3rnion NATIVE v2 abliterated Qwen3.8-Flash-Next, MTP spec decode, 2 slots over ONE 401408-position pool at ctx 200704, vision ON), root systemd unit (dankclaw, frank-dragoon, Strix Halo iGPU)
+Description=halogen-flash-server container (Quat3rnion NATIVE v2 abliterated Qwen3.8-Flash-Next, MTP spec decode, 2 slots over ONE 401408-position pool at ctx 262144, vision ON), root systemd unit (dankclaw, frank-dragoon, Strix Halo iGPU)
 After=network-online.target
 Wants=network-online.target
 

scripts/gpu-exporter.py

@@ -32,6 +32,7 @@ import glob
 import os
 import subprocess
 import sys
+import threading
 from http.server import BaseHTTPRequestHandler, ThreadingHTTPServer
 
 FAMILIES = [
@@ -198,6 +199,11 @@ def main(argv):
             return 2
         servers.append(ThreadingHTTPServer((b, args.port), Handler))
     sys.stderr.write("gpu-exporter serving :%d on %s\n" % (args.port, ",".join(binds)))
+    # EVERY socket must be served: a listen()ed-but-never-accept()ed bind looks
+    # healthy in `ss` yet times out every scrape once its SYN backlog fills —
+    # and the loopback deploy gate cannot see it. One thread per extra bind.
+    for s in servers[1:]:
+        threading.Thread(target=s.serve_forever, daemon=True).start()
     servers[0].serve_forever()
     return 0
 

scripts/grams-to-imatrix.py

@@ -0,0 +1,418 @@
+#!/usr/bin/env python3
+"""grams-to-imatrix.py — the engine's own calibration recording, re-voiced as a
+llama.cpp importance matrix for `rad-convert --imatrix F`. RUNS ON DANKSERVER
+beside the recording. Python 3 stdlib only: numpy must NOT be required, because
+this runs in the plain container next to a 335 GiB checkpoint, not in a venv.
+
+WHY. `rad-convert` takes its per-column activation second moments and its
+expert-popularity profile from ONE input, an imatrix (core/format/imatrix.h).
+Only the GGUF form carries a per-expert dimension — with the legacy flat form,
+or with none, every expert in a layer is weighted by the layer's aggregate,
+which is exactly the wrong thing on a 512-expert MoE. The calibration tap has
+already measured everything; this tool only re-voices it, so the imatrix and the
+protected-expert census (derive-protected-experts.py) are the SAME recording:
+one measurement, two consumers, neither able to disagree about which experts are
+hot.
+
+INPUT. RADGRAM2 files in a RADIANCE_CALIB_DIR output directory
+(arch/common/rad_arch.h, calib_write). Header law, same parse as
+scripts/derive-protected-experts.py — 32 bytes, `<8sIIQI`: magic `RADGRAM2`,
+u32 k, u32 rank, u64 rows, u32 domain, a pad word, then k*k f32 row-major. Two
+kinds of name, both GGUF-style, both with the checkpoint's GLOBAL expert
+numbering (rad_block_moe_fp8.h: ge = expert_base + e):
+
+    gram.r<rank>.blk.L.ffn_down_exps.bin        the layer's pooled gram
+    gram.r<rank>.blk.L.ffn_down_exps.<e>.bin    expert e's own gram
+
+`rows` is that file's routing/token count and the matrix is Σ x xᵀ over those
+rows, so **the diagonal is exactly the per-column activation second moment an
+imatrix wants**, and `rows` is the count the reader hands back from `count()`.
+Every off-diagonal term is dropped on purpose: the GGUF format has nowhere to
+put a full matrix, and the quantiser asks for columns.
+
+OUTPUT, per layer L (core/format/imatrix.h: names WITHOUT `.weight`, a MoE
+layer's experts folded into ONE entry, so expert 9 is slice 9 of
+`blk.L.ffn_down_exps`):
+
+  blk.L.ffn_down_exps    in_sum2 [k, n_expert] expert-major, counts [1, n_expert]
+  blk.L.ffn_gate_up_exps the SAME matrices at the SAME offsets. A routed expert's
+                         gate_up input is the same hidden-state rows as its down
+                         input, so the second moment over them is identical and
+                         so is the routing count. Two tensor infos over one
+                         payload: the reader maps each half independently
+                         (imatrix.cpp joins `<base>.in_sum2` with `<base>.counts`
+                         by base name), and copying ~200 MB to state one fact
+                         twice buys no correctness.
+  blk.L.ssm_inz          [k] and [1], from the pooled gram: that row's input is
+                         the same hidden state, so its columns take the pooled
+                         diagonal and its count is the layer's pooled rows.
+
+  Then STOP. Nothing whose input width is not the gram's k — no ssm_out, no
+  attn_output, no mtp.*, no ple_ngram. importance_for() answers false when the
+  entry's cols do not equal the weight's width and the convert proceeds without
+  it: an entry the reader silently ignores is a lie in the artifact, worse than
+  an absent one. The corollary to check before trusting the two hidden-width
+  entries: the tap writes the gate_up gram at g.n_embd and the down gram at the
+  expert width, so the gram's k must equal the checkpoint's hidden width (its
+  config.json). Where they differ, gate_up/ssm_inz are inert, not wrong.
+
+INHERITANCE. An expert with no own gram carries the layer's POOLED diagonal, and
+its count stays its own routing rows — the rule the engine itself applies when it
+picks a Hessian (libquant/lq_hess.h, calib_resolve: an expert's own gram only
+once its rank files hold kOwnRowsPerK * K rows; below that a fit to so few rows
+is worse than round-to-nearest, and those experts carry a fraction of a percent
+of the routing). The count never inherits: popularity is `count`'s job. The count
+of inherited experts is reported on stderr.
+
+RANK FOLDING. TP=2 writes gram.r0.* and gram.r1.* with GLOBAL expert numbering,
+so the ranks describe DISJOINT experts and the fold is a UNION into the one
+[k, n_expert] entry. A pooled gram is the opposite case: every rank summed the
+same stem over its own rows, so the fold is a SUM, rows included (calib_sum,
+calib_rows). The same global expert in two per-expert files is legal only when
+the numbers agree exactly; otherwise the recording is corrupt and the entry would
+be whichever file sorted last, so refuse.
+
+GGUF v3 (the only version imatrix.cpp accepts — version != 3 is a refusal):
+`general.alignment` u32, `imatrix.chunk_size` u32, `imatrix.chunk_count` u32,
+`imatrix.datasets` array of strings; tensors f32 (ggml type 0) with offsets
+relative to the aligned data section. Every tensor offset is padded up to
+`general.alignment` and the data section itself starts aligned. That is the
+classic hand-rolled-writer bug: an unaligned offset makes the reader's extent
+arithmetic point into the middle of the previous tensor's floats, and it reads
+them without complaint.
+
+chunk_count DERIVATION. The corpus row count is the LARGEST single-layer pooled
+row total, not a sum over layers: every layer's tap accumulated over the whole
+corpus, so adding the layers counts each token once per layer and inflates the
+figure by the layer count. chunk_count = that row total // chunk_size, the
+"chunks of chunk_size" line imatrix.cpp prints for itself.
+
+REFUSALS, each exit 2 with a message naming what is missing:
+  NO_GRAMS         the directory holds no RADGRAM2 down-projection gram
+  NOT_RADGRAM2     a gram whose magic is not RADGRAM2
+  SHORT            a gram whose length is not 32 + k*k*4
+  NO_POOLED        a layer with expert grams and no pooled gram to inherit from
+  K_MISMATCH       a layer whose files disagree about k (rank pooled grams, or an
+                   expert gram against its layer's pooled gram)
+  EXPERT_CONFLICT  a global expert recorded twice with different numbers
+  EXISTS           the -o path already holds a file (the trash law: move it)
+  CHUNK_COUNT      the derived chunk count does not fit the u32 metadata field
+
+USAGE. grams-to-imatrix.py <calib-dir> -o <out.gguf>
+       [--datasets NAME,...] [--chunk-size N] [--min-rows N] [-v]
+"""
+import argparse
+import os
+import re
+import struct
+import sys
+
+MAGIC = b"RADGRAM2"
+HDR = 32                       # 8s I I Q I + pad, rad_arch.h
+KOWN_ROWS_PER_K = 4            # libquant/lq_hess.h kOwnRowsPerK
+ALIGNMENT = 32                 # general.alignment, and the pad unit
+
+POOLED_RE = re.compile(r"^gram\.r(\d+)\.blk\.(\d+)\.ffn_down_exps\.bin$")
+EXPERT_RE = re.compile(r"^gram\.r(\d+)\.blk\.(\d+)\.ffn_down_exps\.(\d+)\.bin$")
+
+GT_U32, GT_STR, GT_ARR = 4, 8, 9
+GGML_F32 = 0
+
+
+def refuse(msg):
+    sys.stderr.write("REFUSED %s\n" % msg)
+    sys.exit(2)
+
+
+def read_gram(path):
+    """(k, rank, rows, domain, diagonal) from one RADGRAM2 file."""
+    try:
+        with open(path, "rb") as f:
+            raw = f.read()
+    except OSError as e:
+        refuse("UNREADABLE %s: %s" % (path, e))
+    if len(raw) < HDR:
+        refuse("SHORT %s: %d bytes, under the %d-byte RADGRAM2 header"
+               % (path, len(raw), HDR))
+    magic, k, rank, rows, domain = struct.unpack_from("<8sIIQI", raw, 0)
+    if magic != MAGIC:
+        refuse("NOT_RADGRAM2 %s: magic %r, expected b'RADGRAM2'" % (path, magic))
+    if k <= 0:
+        refuse("SHORT %s: header claims k=%d, which describes no matrix" % (path, k))
+    need = HDR + k * k * 4
+    if len(raw) != need:
+        refuse("SHORT %s: %d bytes, expected %d (32 + k*k*4 for k=%d)"
+               % (path, len(raw), need, k))
+    # The diagonal only, read by stride out of the row-major matrix.
+    diag = [struct.unpack_from("<f", raw, HDR + (i * k + i) * 4)[0]
+            for i in range(k)]
+    return k, rank, rows, domain, diag
+
+
+def load_dir(d, verbose):
+    """{layer: {"pooled": gram|None, "experts": {global e: gram}}} + rotated count.
+
+    Files outside the ffn_down_exps stem are not ours to fold: the tap names a
+    pooled gram after every projection it taps, and those carry other widths."""
+    if not os.path.isdir(d):
+        refuse("NO_GRAMS %s: not a directory" % d)
+    layers, rotated = {}, 0
+    for name in sorted(os.listdir(d)):
+        m = POOLED_RE.match(name) or EXPERT_RE.match(name)
+        if not m:
+            continue
+        path = os.path.join(d, name)
+        if not os.path.isfile(path):
+            continue
+        pooled = bool(POOLED_RE.match(name))
+        layer = int(m.group(2))
+        k, rank, rows, domain, diag = read_gram(path)
+        if domain != 0:
+            rotated += 1
+        L = layers.setdefault(layer, {"pooled": None, "experts": {}})
+        if pooled:
+            P = L["pooled"]
+            if P is None:
+                L["pooled"] = {"k": k, "rows": rows, "diag": diag, "paths": [path]}
+            elif P["k"] != k:
+                refuse("K_MISMATCH layer %d: pooled grams %s (k=%d) and %s (k=%d)"
+                       " disagree about the width"
+                       % (layer, P["paths"][0], P["k"], path, k))
+            else:
+                # calib_sum: each rank summed the same stem over its own rows.
+                P["rows"] += rows
+                P["paths"].append(path)
+                for i in range(k):
+                    P["diag"][i] += diag[i]
+            if verbose:
+                sys.stderr.write("pooled layer %d: %s k=%d rows=%d\n"
+                                 % (layer, name, k, rows))
+            continue
+        e = int(m.group(3))
+        own = L["experts"].get(e)
+        if own is None:
+            L["experts"][e] = {"k": k, "rows": rows, "diag": diag,
+                               "path": path, "ranks": [rank]}
+            if verbose:
+                sys.stderr.write("expert layer %d r%d e%d: rows=%d\n"
+                                 % (layer, rank, e, rows))
+            continue
+        if own["rows"] != rows or own["diag"] != diag:
+            refuse("EXPERT_CONFLICT layer %d expert %d: %s (rows=%d) and %s"
+                   " (rows=%d) record the same global expert with different"
+                   " numbers" % (layer, e, own["path"], own["rows"], path, rows))
+        own["ranks"].append(rank)
+        sys.stderr.write("note: layer %d expert %d recorded identically by rank %s"
+                         " and %s; folded once\n"
+                         % (layer, e, own["path"], name))
+    if not layers:
+        refuse("NO_GRAMS %s: no RADGRAM2 gram.r<rank>.blk.<L>.ffn_down_exps"
+               "[.<expert>].bin files in the directory" % d)
+    return layers, rotated
+
+
+def build_entries(layers, min_rows, verbose):
+    """([ (name, dims, payload|None, key) ], census, layers emitted)."""
+    entries = []
+    census = {"inherited": 0, "experts": 0, "routing": 0, "rows": 0, "layers": 0}
+    for layer in sorted(layers):
+        P, E = layers[layer]["pooled"], layers[layer]["experts"]
+        if E and P is None:
+            refuse("NO_POOLED layer %d: %d expert gram(s) but no pooled gram (an"
+                   " expected gram.r<rank>.blk.%d.ffn_down_exps.bin), so an expert"
+                   " under the row threshold would have nothing to inherit"
+                   % (layer, len(E), layer))
+        if P is None:
+            continue
+        k = P["k"]
+        census["layers"] += 1
+        census["rows"] = max(census["rows"], P["rows"])
+        dn = "blk.%d.ffn_down_exps" % layer
+        # A staging key names ONE payload. A gate_up entry reuses the down
+        # layer's bytes by carrying the same key with no payload of its own;
+        # in_sum2 and counts are different bytes and must never share a key.
+        ksum, kcnt = (layer, "sum2"), (layer, "counts")
+        kpgm, kpcnt = (layer, "pooled_sum2"), (layer, "pooled_counts")
+        # One entry holds one width: an expert gram recorded at another k would
+        # inherit (or be inherited by) a diagonal of the wrong length, and the
+        # reader would then answer `cols` from whichever file set dims[0].
+        for e in sorted(E):
+            if E[e]["k"] != k:
+                refuse("K_MISMATCH layer %d expert %d: %s holds k=%d and this"
+                       " layer's pooled gram is k=%d" % (layer, e, E[e]["path"],
+                                                          E[e]["k"], k))
+
+        if not E:
+            # No per-expert recording for this layer: emit the dense form
+            # ([k], [1]) rather than 512 identical slices, which would claim a
+            # per-expert measurement that was never taken.
+            entries.append((dn + ".in_sum2", [k],
+                            struct.pack("<%df" % k, *P["diag"]), ksum))
+            entries.append((dn + ".counts", [1],
+                            struct.pack("<f", float(P["rows"])), kcnt))
+            sys.stderr.write("note: layer %d has no per-expert grams; %s is"
+                             " dense over the pooled gram\n" % (layer, dn))
+        else:
+            floor = min_rows if min_rows > 0 else KOWN_ROWS_PER_K * k
+            n_expert = max(E) + 1
+            sum2, counts = bytearray(), bytearray()
+            for e in range(n_expert):
+                own = E.get(e)
+                if own is not None and own["rows"] >= floor:
+                    src = own["diag"]
+                    rows = own["rows"]
+                else:
+                    src = P["diag"]
+                    rows = own["rows"] if own is not None else 0
+                    census["inherited"] += 1
+                sum2 += struct.pack("<%df" % k, *src)
+                counts += struct.pack("<f", float(rows))
+                census["experts"] += 1
+                census["routing"] += rows
+            if min(E) != 0:
+                sys.stderr.write("note: layer %d has no gram for experts 0..%d;"
+                                 " those slices carry the pooled gram at count 0\n"
+                                 % (layer, min(E) - 1))
+            entries.append((dn + ".in_sum2", [k, n_expert], bytes(sum2), ksum))
+            entries.append((dn + ".counts", [1, n_expert], bytes(counts), kcnt))
+            gu = "blk.%d.ffn_gate_up_exps" % layer
+            entries.append((gu + ".in_sum2", [k, n_expert], None, ksum))
+            entries.append((gu + ".counts", [1, n_expert], None, kcnt))
+            if verbose:
+                sys.stderr.write("layer %d: k=%d n_expert=%d pooled_rows=%d"
+                                 " own-gram floor=%d\n"
+                                 % (layer, k, n_expert, P["rows"], floor))
+
+        inz = "blk.%d.ssm_inz" % layer
+        entries.append((inz + ".in_sum2", [k],
+                        struct.pack("<%df" % k, *P["diag"]), kpgm))
+        entries.append((inz + ".counts", [1],
+                        struct.pack("<f", float(P["rows"])), kpcnt))
+    return entries, census
+
+
+def enc_str(v):
+    b = v.encode("utf-8")
+    return struct.pack("<Q", len(b)) + b
+
+
+def write_gguf(path, entries, datasets, chunk_size, chunk_count):
+    """Write GGUF v3 and return its size in bytes.
+
+    One payload per key: entries that share a key share the bytes and the
+    offset, so a second tensor info states the same fact without copying it."""
+    kv = bytearray()
+    kv += enc_str("general.alignment") + struct.pack("<I", GT_U32) \
+        + struct.pack("<I", ALIGNMENT)
+    kv += enc_str("imatrix.chunk_size") + struct.pack("<I", GT_U32) \
+        + struct.pack("<I", chunk_size)
+    kv += enc_str("imatrix.chunk_count") + struct.pack("<I", GT_U32) \
+        + struct.pack("<I", chunk_count)
+    kv += enc_str("imatrix.datasets") + struct.pack("<I", GT_ARR) \
+        + struct.pack("<I", GT_STR) + struct.pack("<Q", len(datasets))
+    for d in datasets:
+        kv += enc_str(d)
+
+    head = bytearray()
+    head += b"GGUF" + struct.pack("<I", 3)
+    head += struct.pack("<Q", len(entries)) + struct.pack("<Q", 4)
+    head += kv
+
+    payloads = {}
+    for name, _dims, payload, key in entries:
+        if payload is None:
+            continue
+        prev = payloads.get(key)
+        if prev is not None and prev != payload:
+            # Two different payloads on one key would ship whichever was
+            # appended last while BOTH tensor infos pointed at it: the counts
+            # payload landing over the in_sum2 payload is exactly how a
+            # hand-rolled GGUF writer loses half its data without an error.
+            refuse("INTERNAL %s: two different payloads share one staging key"
+                   % name)
+        payloads.setdefault(key, payload)
+    infos = bytearray()
+    offsets, order, size = {}, [], 0
+    for name, dims, payload, key in entries:
+        if key not in payloads:
+            refuse("INTERNAL %s: no payload staged for its key" % name)
+        if key not in offsets:
+            offsets[key] = (size + ALIGNMENT - 1) // ALIGNMENT * ALIGNMENT
+            size = offsets[key] + len(payloads[key])
+            order.append(key)
+        infos += enc_str(name) + struct.pack("<I", len(dims))
+        for d in dims:
+            infos += struct.pack("<Q", d)
+        infos += struct.pack("<I", GGML_F32) + struct.pack("<Q", offsets[key])
+
+    body = bytes(head) + bytes(infos)
+    data_start = (len(body) + ALIGNMENT - 1) // ALIGNMENT * ALIGNMENT
+    with open(path, "wb") as f:
+        f.write(body)
+        f.write(b"\0" * (data_start - len(body)))
+        # The offsets in the tensor infos are the file's law, not a hint: each
+        # payload must land at its own padded offset. Writing them back to back
+        # instead leaves every reader that trusts the info — imatrix.cpp does —
+        # reading its neighbour's floats: plausible values, no error, wrong
+        # quantisation.
+        here = data_start
+        for key in order:
+            f.write(b"\0" * (data_start + offsets[key] - here))
+            f.write(payloads[key])
+            here = data_start + offsets[key] + len(payloads[key])
+        return f.tell()
+
+
+def main():
+    ap = argparse.ArgumentParser(
+        description="Fold a radiance calibration recording into a GGUF v3 imatrix.")
+    ap.add_argument("calib_dir")
+    ap.add_argument("-o", "--out", required=True)
+    ap.add_argument("--datasets", default="",
+                    help="comma-separated names for imatrix.datasets")
+    ap.add_argument("--chunk-size", type=int, default=4096,
+                    help="rows per chunk for imatrix.chunk_size (default 4096)")
+    ap.add_argument("--min-rows", type=int, default=0,
+                    help="own-gram row threshold (default 4*k, the engine's"
+                         " kOwnRowsPerK law)")
+    ap.add_argument("-v", "--verbose", action="store_true")
+    a = ap.parse_args()
+
+    if os.path.exists(a.out):
+        # The repo's deletion law: an artifact that exists is moved to trash by
+        # a human, never silently overwritten by a re-run of the tool that made
+        # it, so a second convert cannot destroy the imatrix the first one fed.
+        refuse("EXISTS %s: an imatrix is already there; move it to"
+               " .AGENTTRASH/ or name another -o" % a.out)
+    if a.chunk_size <= 0:
+        refuse("BAD_CHUNK_SIZE --chunk-size %d is not positive" % a.chunk_size)
+    layers, rotated = load_dir(a.calib_dir, a.verbose)
+    entries, census = build_entries(layers, a.min_rows, a.verbose)
+    if not entries:
+        refuse("NO_GRAMS %s: no layer holds a pooled gram to fold" % a.calib_dir)
+    if rotated:
+        sys.stderr.write("note: %d gram(s) recorded in a rotated domain: their"
+                         " diagonal is the rotated activation's second moment,"
+                         " not the model's own\n" % rotated)
+
+    chunk_count = census["rows"] // a.chunk_size
+    if chunk_count > 0xFFFFFFFF:
+        # imatrix.chunk_count is a u32 in the reader's grammar, so there is no
+        # legal way to record a bigger one. Reaching this means either a chunk
+        # size picked for a far larger run or a row total read out of the wrong
+        # header offset — rows is a u64 and a u32 parse is off by 2**32 — and
+        # the reader would trust whatever number landed in the field.
+        refuse("CHUNK_COUNT %d = %d rows // %d does not fit the u32"
+               " imatrix.chunk_count field; raise --chunk-size"
+               % (chunk_count, census["rows"], a.chunk_size))
+    nbytes = write_gguf(a.out, entries, [d for d in a.datasets.split(",") if d],
+                        a.chunk_size, chunk_count)
+    sys.stderr.write("IMATRIX_OK entries=%d layers=%d experts=%d inherited=%d"
+                     " rows=%d routing=%d chunks=%d/%d bytes=%d out=%s\n"
+                     % (len(entries), census["layers"], census["experts"],
+                        census["inherited"], census["rows"], census["routing"],
+                        chunk_count, a.chunk_size, nbytes, a.out))
+
+
+if __name__ == "__main__":
+    main()

scripts/kld-gate-qwen38.sh

@@ -0,0 +1,186 @@
+#!/usr/bin/env bash
+# kld-gate-qwen38.sh — RUNS ON DANKSERVER (or anywhere the reports are). The campaign's
+# acceptance gate: one verdict from the KLD reports of the containers that ride one reference
+# recording.
+#
+# The reference is the BF16 container built from the vendor checkpoint (run-qwen38-bf16.sh
+# kldref, in the same drained window that records the calibration grams). Every scored
+# container is measured against that one recording, so the KL scale runs from 0 = the
+# checkpoint itself downward in faithfulness.
+#
+# TWO MODES, because the disk decides how many anchors can exist at once:
+#
+#   gate IQ4R.json CAND.json                    candidate vs the served payload
+#   gate FP8.json IQ4R.json CAND.json           the same, inside the absolute interval
+#
+# iq4r.json is the payload frank-archon serves — the bar to beat. When the FP8-native release
+# (172.8 GiB, fetched last because of the peak-disk law) is on disk, pass its report first:
+# "half the gap" is then half of the distance between the 4-bit payload and the near-lossless
+# release, not half of an invented distance to the checkpoint. Without it the gate still
+# answers the campaign's question — does the candidate beat what is served — and says so.
+#
+# The law, on the summary of one slice (--slice all | a corpus source):
+#
+#   candidate vs served (2 reports)
+#   1. mean KL(cand) < mean KL(iq4r)         the served payload is beaten on the corpus
+#   2. p99.9(cand) <= p99.9(iq4r)            and the tail is no worse — a mean can hide a few
+#                                            positions taking the whole hit
+#   3. top-1(cand) >= top-1(iq4r)            the agreement the harness keys on does not fall
+#   4. every source slice passes 1-3         a win bought by regressing one distribution
+#                                            (code, prose, tools) is not a win
+#
+#   inside the interval (3 reports), laws 1-4 above (1-3 renumbered to the interval and the
+#   recovery fraction, as agreed 2026-10-27) plus:
+#   5. p99.9(cand) <= p99.9(iq4r)            the tail law
+#   6. every source slice passes            the slice law
+#
+# Booting the containers is NOT this script's job: `run-qwen38-bf16.sh score RAD OUT.json`
+# writes the reports and owns the single engine argv (same pin, --tp 2, --kv-cache-dtype fp8,
+# same step size, --deterministic, no speculation) that makes the numbers comparable.
+# This script is arithmetic, so it re-runs against old reports and is testable without a GPU.
+#
+# Usage: kld-gate-qwen38.sh IQ4R.json CAND.json [--slice NAME]
+#        kld-gate-qwen38.sh FP8.json IQ4R.json CAND.json [--slice NAME] [--recover F]
+set -uo pipefail
+
+if [ "$#" -lt 2 ]; then
+    echo "usage: kld-gate-qwen38.sh IQ4R.json CAND.json [--slice NAME]" >&2
+    echo "       kld-gate-qwen38.sh FP8.json IQ4R.json CAND.json [--slice NAME] [--recover F]" >&2
+    exit 2
+fi
+python3 - "$@" <<'PY'
+import json, sys
+
+a = sys.argv[1:]
+opts, files = [], []
+i = 0
+while i < len(a):
+    if a[i].startswith("--"):
+        opts += [a[i], a[i + 1]]
+        i += 2
+    else:
+        files.append(a[i])
+        i += 1
+if len(files) not in (2, 3):
+    sys.exit("usage: kld-gate-qwen38.sh IQ4R.json CAND.json [--slice NAME]\n"
+             "       kld-gate-qwen38.sh FP8.json IQ4R.json CAND.json [--slice NAME] [--recover F]")
+o = dict(zip(opts[0::2], opts[1::2]))
+slice_ = o.get("--slice", "all")
+recover = float(o.get("--recover", 0.5))
+if not 0.0 < recover <= 1.0:
+    sys.exit(f"--recover must be in (0,1], got {recover}")
+interval = len(files) == 3
+
+
+def die(msg):
+    print(msg, file=sys.stderr)
+    raise SystemExit(2)
+
+
+def load(path):
+    try:
+        return json.load(open(path))
+    except Exception as e:
+        die(f"cannot read the KLD report {path}: {e}")
+
+
+def summary(path, d):
+    """The slice this gate grades: 'all', or one corpus source."""
+    if slice_ == "all":
+        g = d.get("all")
+    else:
+        g = (d.get("by_source") or {}).get(slice_)
+        if g is None:
+            die(f"{path} has no slice {slice_!r}; it has: "
+                f"{', '.join(sorted(d.get('by_source') or {})) or '(none)'}")
+    if g is None or "kld" not in g or "top1_agreement" not in g:
+        die(f"{path} has no {slice_!r} summary — wrong report version?")
+    return g
+
+
+docs = {label: load(path) for label, path in
+        zip(("fp8", "iq4r", "cand") if interval else ("iq4r", "cand"), files)}
+rows = []
+for label, path in zip(("fp8", "iq4r", "cand") if interval else ("iq4r", "cand"), files):
+    g = summary(path, docs[label])
+    rows.append((label, path, docs[label].get("model", "?"), g["kld"]["mean"],
+                 g["kld"]["p99_9"], g["top1_agreement"], g["positions"]))
+
+print(f"reference: BF16 container; slice: {slice_}; "
+      + ("inside the FP8 interval" if interval else "candidate vs served (no FP8 anchor)"))
+print(f"{'':6} {'model':38} {'KL mean':>10} {'KL p99.9':>10} {'top-1':>8}  positions")
+for label, path, model, kl, p999, top1, n in rows:
+    print(f"{label:6} {str(model)[:38]:38} {kl:10.6f} {p999:10.4f} {100*top1:7.2f}%  {n}")
+r = {label: (kl, p999, top1) for label, _p, _m, kl, p999, top1, _n in rows}
+kl_cand, p_cand, t_cand = r["cand"]
+kl_iq4r, p_iq4r, t_iq4r = r["iq4r"]
+ok = True
+
+
+def law(n, text, passed):
+    global ok
+    ok = ok and passed
+    print(f"  [{'PASS' if passed else 'FAIL'}] {n}. {text}")
+
+
+def tail_law(n):
+    law(n, f"candidate KL p99.9 {p_cand:.4f} <= served iq4r {p_iq4r:.4f}",
+        p_cand <= p_iq4r)
+
+
+# --- the slice law: every corpus source must pass the metric laws -------------
+def slice_law(n):
+    """A win that regresses one source class is not a win: check them all."""
+    mine = docs["cand"].get("by_source") or {}
+    theirs = docs["iq4r"].get("by_source") or {}
+    if not mine or not theirs:
+        law(n, "no per-source summaries to check", True)
+        return
+    missing = sorted(set(theirs) - set(mine))
+    extra = sorted(set(mine) - set(theirs))
+    if missing or extra:
+        law(n, "same source slices in both reports"
+            + (f" (missing from cand: {', '.join(missing)})" if missing else "")
+            + (f" (only in cand: {', '.join(extra)})" if extra else ""), False)
+        return
+    bad = []
+    for src in sorted(theirs):
+        c, q = mine[src], theirs[src]
+        worse_mean = c["kld"]["mean"] >= q["kld"]["mean"]
+        worse_tail = c["kld"]["p99_9"] > q["kld"]["p99_9"]
+        worse_top1 = c["top1_agreement"] < q["top1_agreement"]
+        if worse_mean or worse_tail or worse_top1:
+            why = ",".join(m for m, w in (("mean", worse_mean), ("p99.9", worse_tail),
+                                          ("top-1", worse_top1)) if w)
+            bad.append(f"{src} ({why})")
+    law(n, "every source slice passes (mean, p99.9, top-1)"
+        + (f" — regressed: {'; '.join(bad)}" if bad else ""), not bad)
+
+
+if interval:
+    kl_fp8 = r["fp8"][0]
+    gap = kl_iq4r - kl_fp8
+    law(1, f"anchors ordered: fp8 {kl_fp8:.6f} < iq4r {kl_iq4r:.6f} (gap {gap:.6f})",
+        gap > 0)
+    law(2, f"candidate inside the interval: {kl_fp8:.6f} < {kl_cand:.6f} < {kl_iq4r:.6f}",
+        kl_fp8 < kl_cand < kl_iq4r)
+    need = kl_iq4r - recover * gap
+    law(3, f"candidate closes {100*max((kl_iq4r - kl_cand)/gap, 0):.1f}% of the "
+           f"{gap:.6f} gap (needs {100*recover:.0f}%, i.e. KL <= {need:.6f})"
+           if gap > 0 else "gap is not positive; cannot score recovery",
+        gap > 0 and kl_cand <= need)
+    law(4, f"candidate top-1 {100*t_cand:.2f}% >= served iq4r top-1 {100*t_iq4r:.2f}%",
+        t_cand >= t_iq4r)
+    tail_law(5)
+    slice_law(6)
+else:
+    law(1, f"candidate beats the served payload: KL {kl_cand:.6f} < iq4r {kl_iq4r:.6f}",
+        kl_cand < kl_iq4r)
+    tail_law(2)
+    law(3, f"candidate top-1 {100*t_cand:.2f}% >= served iq4r top-1 {100*t_iq4r:.2f}%",
+        t_cand >= t_iq4r)
+    slice_law(4)
+
+print("KLD GATE " + ("PASS" if ok else "FAIL"))
+sys.exit(0 if ok else 1)
+PY

scripts/make-calib-corpus.py

@@ -34,14 +34,25 @@ import json
 import os
 import random
 import re
+import sys
 
 from jinja2 import Environment, FileSystemLoader
-from tokenizers import Tokenizer
+
+_HERE = os.path.dirname(os.path.abspath(__file__)) if "__file__" in globals() else os.getcwd()
+if _HERE not in sys.path:
+    sys.path.insert(0, _HERE)
+
+import qwen38_common as qc  # noqa: E402 - sibling module, same directory
 
 CODE_EXT = (".sh", ".py", ".jinja", ".recipe", ".env", ".example",
             ".yaml", ".yml", ".bats", ".service", ".container", ".toml")
 JSONL_EXTS = (".jsonl", ".json")
 DOC_DIR = "serverAppConfigs-docs"
+FRAGMENT_TOKENS = 2400   # what one fragment may contribute
+CLASS_MIX = {"code": .45, "prose": .30, "tools": .25}
+HF_CAP = {"code": 40000, "prose": 60000, "tools": 0}
+POOL_CYCLES = 40         # how many times a class pool may repeat before giving up
+KLD_GUARD = 5000         # rendered conversations tried before declaring the pool short
 
 FLEET_CMDS = [
     "systemctl status frank-archon --no-pager",
@@ -142,7 +153,7 @@ def load_parquet_texts(path: str, kind: str, rng: random.Random,
     return out[:cap]
 
 
-def main() -> None:
+def parse_args() -> argparse.Namespace:
     ap = argparse.ArgumentParser(description=__doc__)
     ap.add_argument("--repo", required=True)
     ap.add_argument("--tokenizer-json", required=True)
@@ -156,22 +167,26 @@ def main() -> None:
     ap.add_argument("--hf-code-parquet", default="")
     ap.add_argument("--hf-prose-parquet", default="")
     ap.add_argument("--seed", type=int, default=20261006)
-    a = ap.parse_args()
+    return ap.parse_args()
 
-    rng = random.Random(a.seed)
-    tokenizer = Tokenizer.from_file(a.tokenizer_json)
-    env = Environment(loader=FileSystemLoader(os.path.dirname(a.template)),
+
+def load_template(template: str) -> tuple[Environment, str]:
+    """The chat template as the engine will render it, plus its short name."""
+    env = Environment(loader=FileSystemLoader(os.path.dirname(template)),
                       trim_blocks=True, lstrip_blocks=True)
     env.filters["tojson"] = lambda v: json.dumps(v, ensure_ascii=False)
     env.filters["raise_exception"] = lambda m: (_ for _ in ()).throw(
         ValueError(m))
-    tname = os.path.basename(a.template)
+    return env, os.path.basename(template)
 
+
+def classify(repo: str) -> dict[str, list[str]]:
+    """Walk the checkout into the three source classes the mix draws from."""
     code, prose, jsonl = [], [], []
-    for root, dirs, files in os.walk(a.repo):
+    for root, dirs, files in os.walk(repo):
         dirs[:] = [d for d in dirs if not d.startswith(".")]
         for f in sorted(files):
-            p = os.path.relpath(os.path.join(root, f), a.repo)
+            p = os.path.relpath(os.path.join(root, f), repo)
             if DOC_DIR in p and f.endswith(".md"):
                 prose.append(p)
             elif p.startswith(DOC_DIR):
@@ -180,137 +195,185 @@ def main() -> None:
                 code.append(p)
             elif f.endswith(JSONL_EXTS) and f != "package-lock.json":
                 jsonl.append(p)
-    # file-level split: bucket 0 -> KLD pool, 1..4 -> calibration pool
-    pools: dict[str, tuple[list, list]] = {}
-    for cls, lst in (("code", code), ("prose", prose), ("tools", jsonl)):
-        pools[cls] = ([p for p in lst if fnv(p) % 5 == 0],
-                      [p for p in lst if fnv(p) % 5 != 0])
+    return {"code": code, "prose": prose, "tools": jsonl}
 
-    def text_of(p: str) -> str:
-        path = os.path.join(a.repo, p)
-        raw = open(path, encoding="utf-8", errors="replace").read()
-        if p.endswith(JSONL_EXTS):
-            lines = [json.loads(l)["content"] for l in raw.splitlines()
-                     if l.strip() and "content" in l]
-            raw = "\n".join(lines)
-        toks = tokenizer.encode(raw, add_special_tokens=False).ids
-        return tokenizer.decode(toks[:2400], skip_special_tokens=True)
 
+def split_pools(classes: dict[str, list[str]]) -> dict[str, tuple[list, list]]:
+    """File-level split: bucket 0 -> KLD pool, everything else -> calibration."""
+    return {cls: ([p for p in lst if fnv(p) % 5 == 0],
+                  [p for p in lst if fnv(p) % 5 != 0])
+            for cls, lst in classes.items()}
+
+
+def jsonl_content(raw: str) -> str:
+    """The content fields of a content-keyed JSONL, or "" for anything else.
+
+    A .json in this repo is not necessarily a content-keyed JSONL: the substring
+    test passes on any line that merely mentions "content", and a list-shaped
+    document raises TypeError — both used to crash the whole build. Skip what
+    does not fit instead.
+    """
+    lines = []
+    for line in raw.splitlines():
+        if not line.strip() or "content" not in line:
+            continue
+        try:
+            rec = json.loads(line)
+        except json.JSONDecodeError:
+            continue
+        if isinstance(rec, dict) and isinstance(rec.get("content"), str):
+            lines.append(rec["content"])
+    return "\n".join(lines)
+
+
+def fragment_text(repo: str, rel: str, tok) -> str:
+    """One file as a fragment: its first FRAGMENT_TOKENS tokens, decoded."""
+    raw = open(os.path.join(repo, rel), encoding="utf-8",
+               errors="replace").read()
+    if rel.endswith(JSONL_EXTS):
+        raw = jsonl_content(raw)
+    ids = tok.encode(raw, add_special_tokens=False).ids
+    return tok.decode(ids[:FRAGMENT_TOKENS], skip_special_tokens=True)
+
+
+def build_frags(a: argparse.Namespace, tok,
+                rng: random.Random) -> dict[str, dict[str, list[str]]]:
+    """Both fragment pools per class, seeded so the same seed rebuilds the corpus."""
+    pools = split_pools(classify(a.repo))
     cmds = harvest_cmds(a.repo, rng)
-    # fragment pools per class, corpus-split first
     frags: dict[str, dict[str, list[str]]] = {}
-    for cls in pools:
-        kld_files, cal_files = pools[cls]
-        fr = {}
-        src = {"kld": kld_files, "cal": cal_files}
+    for cls, (kld_files, cal_files) in pools.items():
         if cls == "tools":
-            fr["kld"] = [tool_call(rng, cmds) for _ in range(600)]
-            fr["cal"] = [tool_call(rng, cmds) for _ in range(40000)]
+            frags[cls] = {"kld": [tool_call(rng, cmds) for _ in range(600)],
+                          "cal": [tool_call(rng, cmds) for _ in range(40000)]}
         else:
-            fr["kld"] = [text_of(p) for p in src["kld"]]
-            fr["cal"] = [text_of(p) for p in src["cal"]]
-        frags[cls] = fr
+            frags[cls] = {"kld": [fragment_text(a.repo, p, tok) for p in kld_files],
+                          "cal": [fragment_text(a.repo, p, tok) for p in cal_files]}
     # per-class fragment caps: prose rows are short (~66 tok) so need many;
     # codeparrot files are long (~2.6k tok) so a small cap suffices
-    hfcap = {"code": 40000, "prose": 60000, "tools": 0}
     for cls, spec in (("code", a.hf_code_parquet),
                       ("prose", a.hf_prose_parquet)):
         for path in [p for p in spec.split(",") if p]:
             frags[cls]["cal"].extend(
-                load_parquet_texts(path, cls, rng, cap=hfcap[cls]))
+                load_parquet_texts(path, cls, rng, cap=HF_CAP[cls]))
     for cls in frags:
         rng.shuffle(frags[cls]["cal"])
+    return frags
 
-    def toks(s: str) -> int:
-        return len(tokenizer.encode(s, add_special_tokens=False).ids)
 
-    def emit_calib() -> tuple[int, list[dict]]:
-        targets = {"code": .45, "prose": .30, "tools": .25}
-        rows: list[dict] = []
-        for cls, frac in targets.items():
-            want = int(a.calib_tokens * frac)
-            pool = frags[cls]["cal"]
+def pack_class(cls: str, pool: list[str], want: int, tok,
+               chunk_tokens: int) -> list[dict]:
+    """Greedily pack `pool` into chunk_tokens-sized calibration prompts."""
+    if not pool:
+        return []
+    rows: list[dict] = []
+    # Count every fragment once: the loop cycles the pool up to POOL_CYCLES
+    # times, and re-tokenizing a 2.4k-token fragment on each pass dominated the
+    # build's wall time.
+    counts = [qc.n_tokens(tok, f) for f in pool]
+    got = 0
+    i = 0
+    buf: list[str] = []
+    nbuf = 0
+    while got < want:
+        j = i % len(pool)
+        frag = pool[j]
+        i += 1
+        n = counts[j]
+        if nbuf + n > chunk_tokens and buf:
+            rows.append({"prompt": "\n".join(buf), "cls": cls})
+            got += nbuf
+            buf, nbuf = [], 0
+            if got >= want:
+                break
+        buf.append(frag)
+        nbuf += n
+        if i > len(pool) * POOL_CYCLES:
+            break  # pool exhausted with repeats; stop
+    if buf and got < want:
+        rows.append({"prompt": "\n".join(buf), "cls": cls})
+        got += nbuf
+    print(f"calib {cls}: {got} tokens, {i} fragments")
+    return rows
+
+
+def emit_calib(a: argparse.Namespace, frags, tok,
+               rng: random.Random) -> tuple[int, list[dict]]:
+    """calib.jsonl rows: the class mix, shuffled, prompts only."""
+    rows: list[dict] = []
+    for cls, frac in CLASS_MIX.items():
+        rows.extend(pack_class(cls, frags[cls]["cal"], int(a.calib_tokens * frac),
+                               tok, a.chunk_tokens))
+    rng.shuffle(rows)
+    return len(rows), [{"prompt": r["prompt"]} for r in rows]
+
+
+def emit_kld(a: argparse.Namespace, frags, tok, rng: random.Random,
+             env: Environment, tname: str) -> tuple[int, list[dict], int]:
+    """kld.jsonl rows: multi-turn conversations, each with a scored tail."""
+    template = env.get_template(tname)
+    rows: list[dict] = []
+    total = scored_total = 0
+    classes = ["code", "prose", "tools"]
+    cursor = {c: 0 for c in classes}
+    guard = 0
+    while total < a.kld_tokens and guard < KLD_GUARD:
+        guard += 1
+        msgs, srcs = [], []
+        for t in range(rng.choice([2, 3, 4])):
+            cls = classes[(len(rows) + t) % 3]
+            pool = frags[cls]["kld"]
             if not pool:
                 continue
-            got, i, buf, nbuf = 0, 0, [], 0
-            while got < want:
-                frag = pool[i % len(pool)]
-                i += 1
-                n = toks(frag)
-                if nbuf + n > a.chunk_tokens and buf:
-                    rows.append({"prompt": "\n".join(buf), "cls": cls})
-                    got += nbuf
-                    buf, nbuf = [], 0
-                    if got >= want:
-                        break
-                buf.append(frag)
-                nbuf += n
-                if i > len(pool) * 40:
-                    break  # pool exhausted with repeats; stop
-            if buf and got < want:
-                rows.append({"prompt": "\n".join(buf), "cls": cls})
-                got += nbuf
-            print(f"calib {cls}: {got} tokens, {i} fragments")
-        rng.shuffle(rows)
-        return len(rows), [{k: v for k, v in r.items() if k == "prompt"}
-                           for r in rows]
+            frag = pool[cursor[cls] % len(pool)]
+            cursor[cls] += 1
+            msgs.append({"role": "user",
+                         "content": "Summarize and explain this "
+                                    f"engineering material:\n{frag}"})
+            msgs.append({"role": "assistant",
+                         "content": frag[:4000]})
+            srcs.append(cls)
+        rendered = template.render(messages=msgs, add_generation_prompt=False)
+        n = qc.n_tokens(tok, rendered)
+        if n < a.kld_min_prompt:
+            continue
+        rows.append({"prompt": rendered,
+                     "score_from": n - a.score_tail,
+                     "source": "+".join(sorted(set(srcs)))})
+        total += n
+        scored_total += a.score_tail
+    return len(rows), rows, scored_total
 
-    def emit_kld() -> tuple[int, list[dict]]:
-        rows, total = [], 0
-        classes = ["code", "prose", "tools"]
-        i = {c: 0 for c in classes}
-        guard = 0
-        while total < a.kld_tokens and guard < 5000:
-            guard += 1
-            msgs, srcs = [], []
-            for t in range(rng.choice([2, 3, 4])):
-                cls = classes[(len(rows) + t) % 3]
-                pool = frags[cls]["kld"]
-                if not pool:
-                    continue
-                frag = pool[i[cls] % len(pool)]
-                i[cls] += 2
-                msgs.append({"role": "user",
-                             "content": "Summarize and explain this "
-                                        f"engineering material:\n{frag}"})
-                msgs.append({"role": "assistant",
-                             "content": frag[:4000]})
-                srcs.append(cls)
-            rendered = env.get_template(tname).render(
-                messages=msgs, add_generation_prompt=False)
-            n = toks(rendered)
-            if n < a.kld_min_prompt:
-                continue
-            rows.append({"prompt": rendered,
-                         "score_from": n - a.score_tail,
-                         "source": "+".join(sorted(set(srcs)))})
-            total += n
-        return len(rows), rows
 
-    os.makedirs(a.out_dir, exist_ok=True)
-    nc, rows = emit_calib()
-    with open(os.path.join(a.out_dir, "calib.jsonl"), "w") as f:
-        for r in rows:
-            f.write(json.dumps(r) + "\n")
-    nk, krows = emit_kld()
-    with open(os.path.join(a.out_dir, "kld.jsonl"), "w") as f:
-        for r in krows:
-            f.write(json.dumps(r) + "\n")
-    scored = sum(len(tokenizer.encode(r["prompt"], add_special_tokens=False).ids)
-                 - r["score_from"] for r in krows)
-    manifest = {
+def manifest(a: argparse.Namespace, tname: str, nc: int, nk: int,
+             scored: int) -> dict:
+    return {
         "seed": a.seed, "template": tname,
         "tokenizer": os.path.basename(a.tokenizer_json),
         "calib_prompts": nc, "kld_prompts": nk,
         "kld_scored_positions": scored,
         "kld_score_tail": a.score_tail,
         "disjoint_file_pools": "local KLD material only; HF code feeds calib",
-        "calib_mix": {"code": .45, "prose": .30, "tools": .25},
+        "calib_mix": dict(CLASS_MIX),
         "hf_code": a.hf_code_parquet, "hf_prose": a.hf_prose_parquet,
     }
-    with open(os.path.join(a.out_dir, "manifest.json"), "w") as f:
-        json.dump(manifest, f, indent=1)
-    print(json.dumps(manifest, indent=1))
+
+
+def main() -> None:
+    a = parse_args()
+    rng = random.Random(a.seed)
+    tok = qc.load_tokenizer(a.tokenizer_json)
+    env, tname = load_template(a.template)
+
+    frags = build_frags(a, tok, rng)
+    os.makedirs(a.out_dir, exist_ok=True)
+    nc, calib_rows = emit_calib(a, frags, tok, rng)
+    qc.write_jsonl(os.path.join(a.out_dir, "calib.jsonl"), calib_rows)
+    nk, kld_rows, scored = emit_kld(a, frags, tok, rng, env, tname)
+    qc.write_jsonl(os.path.join(a.out_dir, "kld.jsonl"), kld_rows)
+    man = manifest(a, tname, nc, nk, scored)
+    qc.write_json(os.path.join(a.out_dir, "manifest.json"), man)
+    print(json.dumps(man, indent=1))
 
 
 if __name__ == "__main__":

scripts/purge-agenttrash.sh

@@ -0,0 +1,335 @@
+#!/bin/sh
+# Deterministically empty every .AGENTTRASH directory under the given roots.
+#
+# PURPOSE
+#   The fleet's deletion policy parks deleted files in .AGENTTRASH/ — in-repo
+#   beside the item, elsewhere in $HOME as .AGENTTRASH/YYYYMMDD-topic — and
+#   waits for approval to purge. This script is that purge: given a machine,
+#   it clears .AGENTTRASH directories and nothing else.
+#
+# CONTRACT
+#   Default dry-run: the plan prints and nothing is deleted; --execute deletes.
+#   Every line the script can ever print for a candidate is decided at plan time
+#   and stored in the plan table, so a dry run prints exactly what --execute
+#   will print. A .AGENTTRASH that is a root's own trash (a repo's
+#   .AGENTTRASH, a machine's ~/.AGENTTRASH) prints `plan empty <path>`: the
+#   directory is the convention and survives, its staged contents are cleared.
+#   A .AGENTTRASH deeper than that is content and prints `plan remove <path>`
+#   plus, once executed, `removed <path>`; its own nested .AGENTTRASH entries
+#   belong to that tree and are never planned again. Candidates are found with
+#   `find -xdev` per root, so a mounted tree can never be walked by accident;
+#   ordering is lexicographic (LC_ALL=C). `/`, symlinked, and foreign-owned
+#   candidates are refused and reported (exit 1), never touched.
+#   A summary line `removed=<candidates> deleted=<removed> refused=<n>` closes
+#   every run and one dated line is appended to the log.
+#   Exit: 0 success, 1 operational failure, 2 usage error. A second run over
+#   the same tree plans nothing further and exits 0.
+#
+# PORTABILITY
+#   POSIX sh, clean under `shellcheck -S style`. Externals: find (must support
+#   -delete), stat, rm, date, mkdir. No bashisms, no GNU-only flags, no arrays,
+#   no `local`. Paths containing whitespace are rejected at parse time — the
+#   same rule applied to roots and candidates — so the plan can be walked with
+#   plain word splitting and nothing can be silently mis-split.
+#
+# USAGE
+#   purge-agenttrash [--execute] [--quiet] [--log PATH] [PATH...]
+#   --execute   delete; without it the run is a plan and deletes nothing
+#   --quiet     suppress per-line output; errors and the summary still print
+#   --log PATH  log destination (default: $HOME/.AGENTTRASH-cleanup.log)
+#   PATH...     roots to search (default: $HOME)
+#
+# Everything else is fixed by design: the root default, the ordering, the
+# guards. There is no configuration file and no environment override.
+
+set -eu
+
+usage() {
+    sed -n '3,44p' "$0" | sed 's/^# \{0,1\}//'
+}
+
+die() {
+    printf '%s: %s\n' 'purge-agenttrash' "$1" >&2
+    exit "${2:-1}"
+}
+
+# ------------------------------------------------------------------ arg parser
+# Roots accumulate as one space-separated string: a root containing whitespace is
+# rejected at parse time rather than silently split later. That leaves the plan
+# iteratable with plain `$@`-free word splitting, the only portable construct
+# that cannot corrupt a path it has already vetted.
+MODE=plan
+QUIET=0
+LOGPATH=
+ROOTS=
+while [ "$#" -gt 0 ]; do
+    case "$1" in
+        --help|-h)
+            usage
+            exit 0
+            ;;
+        --execute)
+            MODE=execute
+            ;;
+        --quiet)
+            QUIET=1
+            ;;
+        --log)
+            [ "$#" -ge 2 ] || die '--log needs a value' 2
+            LOGPATH=$2
+            shift
+            ;;
+        --log=*)
+            LOGPATH=${1#--log=}
+            ;;
+        -*)
+            die "unknown option: $1" 2
+            ;;
+        *)
+            case $1 in
+                *' '*) die "root contains whitespace: $1" 2 ;;
+            esac
+            if [ -z "$ROOTS" ]; then
+                ROOTS=$1
+            else
+                ROOTS="$ROOTS $1"
+            fi
+            ;;
+    esac
+    shift
+done
+
+[ -n "${HOME:-}" ] || die 'HOME is not set; pass roots explicitly or set HOME' 2
+[ -n "$LOGPATH" ] || LOGPATH=$HOME/.AGENTTRASH-cleanup.log
+
+# ------------------------------------------------------------ tool availability
+require() {
+    command -v "$1" >/dev/null 2>&1 || die "required tool not found: $1" 1
+}
+require find
+require stat
+require rm
+require date
+require mkdir
+
+# `find -delete` backs the in-place emptying. Probe it rather than trusting
+# `find --version`: a shell function named `find` must not be able to fake it.
+if ! find . -maxdepth 0 -name '.purge-probe-does-not-exist' -delete >/dev/null 2>&1; then
+    die 'find does not support -delete; refusing to guess at deletion' 1
+fi
+
+UID_SELF=$(id -u)
+
+# ------------------------------------------------------------------- locking
+# The lock dir also hosts this run's plan table, so both live and die together:
+# the exit trap removes the whole directory and only the log survives the run.
+LOCKDIR=$HOME/.agenttrash-purge.lock
+PLANFILE=$LOCKDIR/plan
+
+acquire_lock() {
+    if ! mkdir "$LOCKDIR" 2>/dev/null; then
+        held_pid=$(cat "$LOCKDIR/pid" 2>/dev/null || printf '%s\n' '')
+        if [ -n "$held_pid" ] && kill -0 "$held_pid" 2>/dev/null; then
+            die "another purge holds $LOCKDIR (pid $held_pid)" 1
+        fi
+        # Owner died mid-run: drop its pid, then retry exactly once. An
+        # unremovable/nonempty lock dir is not ours to reclaim.
+        rm -f "$LOCKDIR/pid" 2>/dev/null || true
+        rmdir "$LOCKDIR" 2>/dev/null || die "$LOCKDIR exists and is not ours to reclaim" 1
+        mkdir "$LOCKDIR" 2>/dev/null || die "cannot reacquire $LOCKDIR" 1
+    fi
+    printf '%s\n' "$$" >"$LOCKDIR/pid" || die "cannot stamp $LOCKDIR/pid" 1
+    : >"$PLANFILE" || die "cannot write $PLANFILE" 1
+    trap 'rm -rf "$LOCKDIR"' EXIT
+    trap 'exit 130' INT
+    trap 'exit 143' TERM
+}
+
+# ------------------------------------------------------------------ predicates
+is_dir() {  # true only for a real directory, never a symlink to one
+    [ -d "$1" ] && [ ! -L "$1" ]
+}
+
+is_ours() {  # true only when the invoking user owns the path
+    owner=$(stat -c '%u' "$1" 2>/dev/null) || return 1
+    [ "$owner" = "$UID_SELF" ]
+}
+
+refuse() {  # refuse <path> <reason>  — reported, never deleted
+    printf '%s: refused %s (%s)\n' 'purge-agenttrash' "$1" "$2" >&2
+    REFUSED=$((REFUSED + 1))
+}
+
+emit() {  # emit <line>  — silenced by --quiet
+    [ "$QUIET" -eq 1 ] || printf '%s\n' "$1"
+}
+
+# --------------------------------------------------------------- the plan table
+# One row per candidate: `<action> <path> <line>`. The first two fields tell
+# `act_on` what to do; the rest of the row is EXACTLY what the run prints for
+# that candidate — in plan mode verbatim, in execute mode as the first line of
+# its report. The table is a file, so no shell quoting rule (quoted-for,
+# heredoc, IFS games) can corrupt an approved plan, and the two modes can
+# never drift apart. A removal's second line (`removed <path>`) is derived from
+# the row's action, so it is also fixed at plan time.
+CANDIDATES=0
+REFUSED=0
+
+record() {  # record <action> <path> <line>  — append one row to the plan table
+    printf '%s %s %s\n' "$1" "$2" "$3" >>"$PLANFILE"
+    CANDIDATES=$((CANDIDATES + 1))
+}
+
+# ------------------------------------------------------------- candidate rule
+# The whole contract in one place:
+#   `/` is refused outright (never a candidate, never walked);
+#   a root that is a symlink or a plain file is refused, never followed;
+#   a root (or any path) that IS a real .AGENTTRASH prints `plan empty <path>`:
+#     the convention persists, its staged contents are what gets cleared;
+#   otherwise the root's DIRECT .AGENTTRASH child is also `empty` — it is that
+#     level's own trash ($HOME/.AGENTTRASH, x1/.AGENTTRASH, repo-a/.AGENTTRASH);
+#   every .AGENTTRASH deeper than that is content: `remove`, printed as
+#     `plan remove <path>` (+ `removed <path>` when executed), unless it lives
+#     inside a tree already claimed (then it dies with that tree, never planned
+#     twice) or is foreign-owned/symlinked (then refused).
+collect_root() {  # collect_root <path> -> 0 ok, 1 refused
+    target=$1
+    if [ "$target" = / ]; then
+        refuse / 'root refusal'
+        return 1
+    fi
+    if [ -L "$target" ] || ! [ -d "$target" ]; then
+        refuse "$target" 'not a real directory'
+        return 1
+    fi
+    if ! is_ours "$target"; then
+        refuse "$target" "not owned by uid $UID_SELF"
+        return 1
+    fi
+    if [ "${target##*/}" = .AGENTTRASH ]; then
+        record empty "$target" "plan empty $target"
+        return 0
+    fi
+    # `own` is the root's own .AGENTTRASH once handled; hits below it belong to
+    # that tree and must never be planned a second time.
+    own=
+    if [ -e "$target/.AGENTTRASH" ]; then
+        if ! is_ours "$target/.AGENTTRASH"; then
+            refuse "$target/.AGENTTRASH" "not owned by uid $UID_SELF"
+        elif ! is_dir "$target/.AGENTTRASH"; then
+            refuse "$target/.AGENTTRASH" 'not a real directory'
+        else
+            record empty "$target/.AGENTTRASH" "plan empty $target/.AGENTTRASH"
+            own=$target/.AGENTTRASH
+        fi
+    fi
+    # No `-type d`: implementations disagree on whether that matches a
+    # symlink-to-directory (GNU find says no, others yes), so every entry named
+    # .AGENTTRASH is collected and classified here instead of by `find`.
+    if ! found=$(find "$target" -xdev -name .AGENTTRASH 2>/dev/null); then
+        refuse "$target" 'find failed'
+        return 1
+    fi
+    previous=
+    for entry in $found; do
+        [ -n "$entry" ] || continue
+        if [ -n "$own" ]; then
+            case $entry in
+                "$own"|"$own"/*) continue ;;   # already emptied in place
+            esac
+        fi
+        if [ -n "$previous" ]; then
+            case $entry in
+                "$previous"/*) continue ;;   # nested: dies with its parent
+            esac
+        fi
+        previous=$entry
+        if ! is_ours "$entry"; then
+            refuse "$entry" "not owned by uid $UID_SELF"
+            continue
+        fi
+        if ! is_dir "$entry"; then
+            refuse "$entry" 'not a real directory'
+            continue
+        fi
+        record remove "$entry" "plan remove $entry"
+    done
+    return 0
+}
+
+# ------------------------------------------------------------------ execution
+remove_tree() {  # rm -rf a real directory; symlinks inside die as entries
+    is_dir "$1" || return 1
+    rm -rf -- "$1"
+}
+
+empty_dir() {  # empty a real directory in place, never following symlinks
+    is_dir "$1" || return 1
+    find "$1" -mindepth 1 -xdev -delete
+}
+
+# act_on <action> <path> <line> -> 0 when the action succeeded.
+# remove: prints the stored line (`plan remove <path>`) then `removed <path>`.
+# empty:  prints the stored line (`plan empty <path>`) and nothing else — the
+#         directory survives, so the plan line IS the outcome.
+act_on() {  # act_on <action> <path> <line> -> 0 when the action succeeded
+    if [ "$1" = remove ]; then
+        if remove_tree "$2" 2>/dev/null; then
+            emit "$3"
+            emit "removed $2"
+            return 0
+        fi
+        printf '%s: failed to remove %s\n' 'purge-agenttrash' "$2" >&2
+        return 1
+    fi
+    if empty_dir "$2" 2>/dev/null; then
+        emit "$3"
+        return 0
+    fi
+    printf '%s: failed to empty %s\n' 'purge-agenttrash' "$2" >&2
+    return 1
+}
+
+# Walk the plan table once: plan mode prints every row's line verbatim, execute
+# mode acts on each row and counts failures separately from plan-time refusals.
+walk_plan() {
+    while read -r action path line; do
+        [ -n "$action" ] || continue
+        if [ "$MODE" = plan ]; then
+            emit "$line"
+        elif act_on "$action" "$path" "$line"; then
+            REMOVED=$((REMOVED + 1))
+        else
+            EXEC_REFUSED=$((EXEC_REFUSED + 1))
+        fi
+    done <"$PLANFILE"
+}
+
+# ----------------------------------------------------------------------- main
+[ -n "$ROOTS" ] || ROOTS=$HOME
+
+acquire_lock
+
+for root in $ROOTS; do
+    collect_root "$root" || true
+done
+
+REMOVED=0
+EXEC_REFUSED=0
+walk_plan
+
+STATUS=0
+[ "$REFUSED" -eq 0 ] || STATUS=1
+[ "$EXEC_REFUSED" -eq 0 ] || STATUS=1
+
+printf 'removed=%s deleted=%s refused=%s\n' \
+    "$CANDIDATES" "$REMOVED" "$((REFUSED + EXEC_REFUSED))"
+
+if ! printf '%s\tplan\tcandidates=%s\tremoved=%s\trefused=%s\n' \
+    "$(date -u '+%Y-%m-%dT%H:%M:%SZ')" "$CANDIDATES" "$REMOVED" \
+    "$((REFUSED + EXEC_REFUSED))" >>"$LOGPATH" 2>/dev/null; then
+    printf '%s: cannot write log %s\n' 'purge-agenttrash' "$LOGPATH" >&2
+    STATUS=1
+fi
+
+exit "$STATUS"

scripts/qwen38-calib-window.sh

@@ -0,0 +1,122 @@
+#!/usr/bin/env bash
+# qwen38-calib-window.sh — RUNS ON DANKSERVER. The calibration window, and nothing else:
+# boot the BF16 rig, wait until it is actually serving, push the corpus until the engine says
+# the tap is full, stop the rig, restore frank-archon, and bank the reference container.
+#
+# The reference recording is already taken (kld-ref/kld.json exists), so this window is the
+# calibration tap alone — the one job that needs both cards after it. The kldref phase is NOT
+# re-run here: the chained launcher that used to do that moved the banked kld-ref into trash
+# first, which is how a 16-hour phase gets paid for twice.
+#
+# Every phase is gated on the previous one, because the last time this ran unguarded
+# (2026-10-29) the rig's boot died of an OOM, the health loop quietly fell through, and the
+# pusher fed four rounds of 10M tokens into RESTORED PRODUCTION with nothing recorded:
+#
+#   1. the runner must still be alive and its container must answer /health, or this script
+#      kills it, restores the service and exits non-zero — no push, ever;
+#   2. the pusher refuses on its own unless the rig wrote its marker into the tap dir, the
+#      marker names this container, and the container is running (scripts/calib-push.py
+#      --tap-marker), so a second lock survives a mistake in this file;
+#   3. the reference container is banked into trash only when BOTH the serve and the push
+#      exited 0 and the tap dir holds grams. The checkpoint that reproduces it stays.
+#
+# knobs: Q38_WINDOW_CALIB_CORPUS (calib-7168.jsonl — the re-chunked file the rig's pool can
+#        hold), Q38_WINDOW_HEALTH_TIMEOUT (seconds to wait for /health), Q38_WINDOW_ROUNDS
+#        (pusher rounds; one is 10M tokens), Q38_WINDOW_KEEP_RAD=1 to leave the .rad in place.
+# Usage: scripts/qwen38-calib-window.sh [--dry-run]
+set -uo pipefail
+cd "$(dirname "${BASH_SOURCE[0]}")/.." || exit 9
+
+DRY=0
+[ "${1:-}" = "--dry-run" ] && DRY=1
+say() { printf '%s %s\n' "$(date -Is)" "$*"; }
+
+MODELS="${FRANK_QWEN38_MODELS:-/home/dank/models}"
+CAMPAIGN="${FRANK_QWEN38_CAMPAIGN:-/home/dank/campaign/qwen38-w5nl64}"
+RAD="${FRANK_QWEN38_RAD:-$MODELS/rad/qwen38-bf16.rad}"
+CORPUS="${Q38_WINDOW_CALIB_CORPUS:-/home/dank/corpora/out/calib-7168.jsonl}"
+KLDREF="$CAMPAIGN/kld-ref/kld.json"
+CALIB="$CAMPAIGN/calib"
+HEALTH_TIMEOUT="${Q38_WINDOW_HEALTH_TIMEOUT:-3600}"
+ROUNDS="${Q38_WINDOW_ROUNDS:-6}"
+RIG=qwen38-bf16
+SERVICE=frank-archon
+PY="${PY:-python3}"
+
+# ---- preflight: everything this window needs, before a card is touched -------
+[ -f "$KLDREF" ] || { echo "REFUSED: no reference recording at $KLDREF — record it first (scripts/run-qwen38-bf16.sh kldref); this window only fills the calibration tap" >&2; exit 2; }
+[ -f "$RAD" ]    || { echo "REFUSED: no BF16 reference container at $RAD — nothing to serve; scripts/build-qwen38-bf16-rad.sh" >&2; exit 2; }
+[ -f "$CORPUS" ] || { echo "REFUSED: no calibration corpus at $CORPUS" >&2; exit 2; }
+[ -d "$CALIB" ]  || { echo "REFUSED: no tap dir at $CALIB" >&2; exit 2; }
+[ -w "$CALIB" ]  || { echo "REFUSED: tap dir $CALIB is not writable by $(id -un)" >&2; exit 2; }
+! [[ "$HEALTH_TIMEOUT" =~ ^[0-9]+$ ]] && { echo "REFUSED: Q38_WINDOW_HEALTH_TIMEOUT must be seconds (got: $HEALTH_TIMEOUT)" >&2; exit 2; }
+! [[ "$ROUNDS" =~ ^[0-9]+$ ]] && { echo "REFUSED: Q38_WINDOW_ROUNDS must be a round count (got: $ROUNDS)" >&2; exit 2; }
+
+if [ "$DRY" = 1 ]; then
+    echo "dry run: calib window — the service is left running, nothing is mounted or written"
+    echo "would run: bash scripts/run-qwen38-bf16.sh calib          (rig boots, writes $CALIB/.rig-boot)"
+    echo "would wait: /health up within ${HEALTH_TIMEOUT}s, else kill the rig, restore $SERVICE, exit 1"
+    echo "would run: $PY scripts/calib-push.py --container $RIG --corpus $CORPUS --max-rounds $ROUNDS"
+    echo "would bank: $RAD -> ~/.AGENTTRASH/$(date +%Y%m%d)-qwen38-bf16-rad/  (only if both phases exit 0)"
+    exit 0
+fi
+
+say "calib window start pid=$$ corpus=$CORPUS rounds=$ROUNDS"
+[ -n "$(ls -A "$CALIB" 2>/dev/null)" ] && say "tap dir $CALIB is not empty; the engine's token floor is pooled, so an earlier partial tap still counts"
+
+# ---- phase 1: the rig -------------------------------------------------------
+say "phase=calib serve begin"
+bash scripts/run-qwen38-bf16.sh calib & spid=$!
+
+rig_up() {
+    podman inspect -f '{{.State.Running}}' "$RIG" 2>/dev/null | grep -qx true \
+        && curl -sf --max-time 10 -o /dev/null http://127.0.0.1:7770/health
+}
+
+waited=0
+until rig_up; do
+    if ! kill -0 "$spid" 2>/dev/null; then
+        wait "$spid"; crc=$?
+        say "phase=calib serve end rc=$crc (the rig exited before it ever answered /health)"
+        echo "FAIL: the rig boot died (rc=$crc) — nothing was pushed; read /home/dank/logs/qwen38_bf16_calib.log" >&2
+        exit 1
+    fi
+    if [ "$waited" -ge "$HEALTH_TIMEOUT" ]; then
+        say "phase=calib health never came after ${HEALTH_TIMEOUT}s; killing the rig"
+        podman stop "$RIG" >/dev/null 2>&1 || true
+        wait "$spid"
+        echo "FAIL: the rig never answered /health within ${HEALTH_TIMEOUT}s — nothing was pushed" >&2
+        exit 1
+    fi
+    sleep 15
+    waited=$((waited + 15))
+done
+say "phase=calib health ok waited=${waited}s"
+
+# ---- phase 2: the tap -------------------------------------------------------
+say "phase=calib push begin"
+"$PY" scripts/calib-push.py --port 7770 --parallel 8 --container "$RIG" \
+    --tap-dir "$CALIB" --corpus "$CORPUS" --max-rounds "$ROUNDS"
+prc=$?
+say "phase=calib push end rc=$prc"
+
+podman stop "$RIG" >/dev/null 2>&1 || true
+wait "$spid"; crc=$?
+say "phase=calib serve end rc=$crc"
+
+if [ "$prc" -ne 0 ] || [ "$crc" -ne 0 ]; then
+    echo "FAIL: serve rc=$crc push rc=$prc — the tap is NOT banked and $RAD stays: $(basename "$RAD")" >&2
+    exit 1
+fi
+[ -n "$(ls -A "$CALIB" 2>/dev/null)" ] || { echo "FAIL: both phases exited 0 but the tap dir $CALIB is empty" >&2; exit 1; }
+say "TAP BANKED: $(ls -1 "$CALIB" | wc -l) file(s) in $CALIB"
+
+# ---- phase 3: bank the reference container ----------------------------------
+if [ "${Q38_WINDOW_KEEP_RAD:-0}" = "1" ]; then
+    say "reference .rad kept (Q38_WINDOW_KEEP_RAD=1)"
+else
+    T="$HOME/.AGENTTRASH/$(date +%Y%m%d)-qwen38-bf16-rad"
+    mkdir -p "$T" || { echo "FAIL: cannot create trash dir $T" >&2; exit 1; }
+    mv "$RAD" "$T/" && say "reference .rad banked in $T (the checkpoint that rebuilds it stays)"
+fi
+say "calib window end"

scripts/qwen38_common.py

@@ -0,0 +1,94 @@
+#!/usr/bin/env python3
+"""Shared plumbing for the qwen38 campaign corpus tools.
+
+The three tools move the same two shapes — a JSONL of prompt records and a
+one-artifact JSON manifest — and they all tokenize with the SAME tokenizer. That
+last part is not a detail: a KLD reference is only meaningful against the exact
+token sequence the corpus recorded, so `n_tokens` exists once, here, and every
+tool counts the same way. A window that decode/encode does not round-trip is a
+refusal, never a warning.
+
+Loaded as a sibling module. Each tool puts its own directory on `sys.path`
+first, so `python3 scripts/<tool>.py` and the driver's container mount
+(`runpy.run_path('/work/<tool>.py')`, `-v scripts:/work:ro`) both resolve it —
+run the tools by path, not piped through stdin.
+"""
+import contextlib
+import json
+import os
+import sys
+import tempfile
+
+
+def refuse(msg: str) -> None:
+    """Exit 2 with a REFUSED line — the code argparse uses for unusable input."""
+    print(f"REFUSED: {msg}", file=sys.stderr)
+    sys.exit(2)
+
+
+def load_tokenizer(path: str):
+    try:
+        from tokenizers import Tokenizer
+    except ImportError as exc:  # the corpus tools carry their own venv
+        refuse(f"tokenizers is not importable by {sys.executable} — run the "
+               f"corpus venv's python (on dankserver: "
+               f"/home/dank/venvs/radcorpus/bin/python) or set "
+               f"FRANK_QWEN38_TOKENIZER: {exc}")
+    if not os.path.isfile(path):
+        refuse(f"no tokenizer at {path}")
+    return Tokenizer.from_file(path)
+
+
+def n_tokens(tok, text: str) -> int:
+    """The corpus's own count: no special tokens added, prompt side only."""
+    return len(tok.encode(text, add_special_tokens=False).ids)
+
+
+def iter_jsonl(path: str):
+    """Yield `(lineno, record)` for every non-blank line; refuse on a bad line.
+
+    Field validation stays with the caller: the three corpora carry different
+    fields (`prompt` alone, `prompt`+`cls`, `prompt`+`score_from`+`source`), and
+    a record missing the one the tool needs is that tool's refusal to make.
+    """
+    with open(path, encoding="utf-8", errors="replace") as fh:
+        for lineno, line in enumerate(fh, 1):
+            if not line.strip():
+                continue
+            try:
+                rec = json.loads(line)
+            except json.JSONDecodeError as exc:
+                refuse(f"{path}:{lineno} is not JSON: {exc}")
+            yield lineno, rec
+
+
+def write_jsonl(path: str, rows) -> None:
+    with open(path, "w", encoding="utf-8") as fh:
+        for row in rows:
+            fh.write(json.dumps(row, ensure_ascii=False) + "\n")
+
+
+@contextlib.contextmanager
+def atomic_writer(path: str):
+    """Write `path` through a temporary sibling and rename on success.
+
+    An interrupted run then leaves no half corpus behind, and the temporary name
+    keeps the one-artifact-one-name law the driver applies to a score report.
+    """
+    d = os.path.dirname(os.path.abspath(path)) or "."
+    fd, tmp = tempfile.mkstemp(dir=d, prefix="." + os.path.basename(path) + "-")
+    try:
+        with os.fdopen(fd, "w", encoding="utf-8") as fh:
+            yield fh
+        os.replace(tmp, path)
+    except BaseException:
+        if os.path.exists(tmp):
+            os.unlink(tmp)
+        raise
+    os.chmod(path, 0o644)
+
+
+def write_json(path: str, obj) -> None:
+    with atomic_writer(path) as fh:
+        json.dump(obj, fh, indent=1)
+        fh.write("\n")

scripts/rechunk-kld-corpus.py

@@ -0,0 +1,192 @@
+#!/usr/bin/env python3
+"""Re-chunk a KLD corpus so every document fits the container that scores it.
+
+A KLD reference is recorded by booting one container and prefilling every corpus
+document through it. radiance refuses a document the KV pool cannot hold on its
+own (core/mem/kv.cpp `can_ever_fit`: a paged group holds `n_blocks *
+block_size` tokens), and the number that matters is printed by the boot ledger,
+not by a flag:
+
+    W kv group 'kv_attn' holds 7436 tokens but the context limit is 32768, so
+    a prompt longer than 7436 tokens is refused (400) ... or lower
+    --max-model-len to 7436
+
+The BF16 qwen4exp container at the campaign's shape gets 270.00 MiB of cache --
+the expert plane owns both cards -- so its real window is 7,436 tokens however
+`--max-model-len` is spelled. The corpus the campaign inherited from the swift15
+era is 197 documents of 8,193-11,970 tokens, because make-calib-corpus.py's
+`--kld-min-prompt 8192` sets a floor and no ceiling: every one of them is
+unscoreable there, and three drained windows have proved it.
+
+This script moves the document boundaries and nothing else. Each record's token
+ids -- counted with the same tokenizer the generator used -- are cut into
+windows of at most `--max-doc-tokens`, so a record's windows concatenate back to
+that record's own token sequence, unchanged: no token dropped, rewritten or
+reworded. Each window gets its own `score_from = len - --score-tail`, the same
+relation the generator wrote for the parent, so the reference scores 256
+positions per window instead of per document: 100,864 measured positions over
+the same text against the 50,432 the inherited corpus scored, at a shorter
+context. The calibration corpus is untouched, so the quantization itself does
+not move.
+
+Usage:
+  rechunk-kld-corpus.py --corpus kld.jsonl --out kld.jsonl.new \
+      [--tokenizer-json PATH] [--max-doc-tokens 7168] [--score-tail 256] [--force]
+
+--out must not exist and be non-empty without --force: the same
+one-artifact-one-name law run-qwen38-bf16.sh applies to a score report. The file
+is written through a temporary name and renamed, so an interrupted run leaves no
+half corpus behind. A JSON summary of what was written goes to stdout.
+"""
+import argparse
+import dataclasses
+import json
+import os
+import sys
+
+_HERE = os.path.dirname(os.path.abspath(__file__)) if "__file__" in globals() else os.getcwd()
+if _HERE not in sys.path:
+    sys.path.insert(0, _HERE)
+
+import qwen38_common as qc
+
+MISSING = object()  # "the parent record carried no source field"
+
+
+def parse_args() -> argparse.Namespace:
+    ap = argparse.ArgumentParser(description=__doc__.splitlines()[0])
+    ap.add_argument("--corpus", required=True,
+                    help="the kld.jsonl to re-chunk (records: prompt, score_from, source)")
+    ap.add_argument("--out", required=True,
+                    help="file to write; refuse if it exists and is non-empty without --force")
+    ap.add_argument("--tokenizer-json",
+                    default=os.environ.get("FRANK_QWEN38_TOKENIZER",
+                                           "/home/dank/models/Qwen3.8-Flash-Next/tokenizer.json"),
+                    help="the tokenizer the corpus was authored with (the same one the "
+                         "manifest in the corpus output dir names)")
+    ap.add_argument("--max-doc-tokens", type=int, default=7168,
+                    help="the widest document the scoring container can hold; 7,168 is "
+                         "7x1024, the largest whole kilopage under the 7,436 the bf16 "
+                         "container's ledger reports")
+    ap.add_argument("--score-tail", type=int, default=256,
+                    help="positions scored at the end of every document, as the generator "
+                         "recorded it")
+    ap.add_argument("--force", action="store_true",
+                    help="overwrite a non-empty --out (after moving the old file to trash)")
+    return ap.parse_args()
+
+
+@dataclasses.dataclass(frozen=True)
+class Window:
+    """One scoring document cut out of a parent record."""
+    prompt: str
+    score_from: int
+    n_tokens: int
+    clamped: bool
+    source: object = MISSING
+
+    def record(self) -> dict:
+        rec = {"prompt": self.prompt, "score_from": self.score_from}
+        if self.source is not MISSING:
+            rec["source"] = self.source
+        return rec
+
+
+def validate(a: argparse.Namespace) -> None:
+    """Everything that makes this run unusable, named before a token is cut."""
+    if a.max_doc_tokens <= 0:
+        qc.refuse(f"--max-doc-tokens must be positive (got {a.max_doc_tokens})")
+    if a.score_tail <= 0:
+        qc.refuse(f"--score-tail must be positive (got {a.score_tail})")
+    if a.max_doc_tokens <= a.score_tail:
+        qc.refuse(f"--max-doc-tokens {a.max_doc_tokens} cannot hold a "
+                  f"{a.score_tail}-position score tail: every window would be all "
+                  "tail and no context")
+    if not os.path.isfile(a.corpus):
+        qc.refuse(f"no corpus at {a.corpus}")
+    if os.path.realpath(a.out) == os.path.realpath(a.corpus):
+        qc.refuse(f"--out {a.out} is the corpus itself; move it to trash first — the "
+                  "trash law means the old segmentation stays recoverable")
+    if os.path.exists(a.out) and os.path.getsize(a.out) > 0 and not a.force:
+        qc.refuse(f"{a.out} exists and is not empty; pass --force after moving it to trash")
+
+
+def cut(rec: dict, tok, a: argparse.Namespace,
+        where: str) -> tuple[list[Window], int]:
+    """Cut one record's token ids into windows, proving each one round-trips.
+
+    The round-trip encode is the whole reason this tool may claim the reference
+    scores the corpus's own token sequence: decoding a window to text and
+    encoding it back must return the identical ids, or the score is of a
+    different document than the one on disk.
+    """
+    ids = tok.encode(rec["prompt"], add_special_tokens=False).ids
+    windows: list[Window] = []
+    for start in range(0, max(len(ids), 1), a.max_doc_tokens):
+        window = ids[start:start + a.max_doc_tokens]
+        if not window:
+            continue
+        text = tok.decode(window)
+        if tok.encode(text, add_special_tokens=False).ids != window:
+            qc.refuse(f"{where} window at token {start} does not round-trip through "
+                      "text — the reference would score a different token sequence "
+                      "than the corpus holds")
+        windows.append(Window(prompt=text,
+                              score_from=max(len(window) - a.score_tail, 0),
+                              n_tokens=len(window),
+                              clamped=len(window) < a.score_tail,
+                              source=rec["source"] if "source" in rec else MISSING))
+    return windows, len(ids)
+
+
+def rechunk(a: argparse.Namespace, tok) -> dict:
+    """Write the re-chunked corpus and return the summary it printed."""
+    docs_in = docs_out = tokens_in = tokens_out = 0
+    longest_child = clamped = scored = 0
+    sources: dict[str, int] = {}
+    with qc.atomic_writer(a.out) as out:
+        for lineno, rec in qc.iter_jsonl(a.corpus):
+            where = f"{a.corpus}:{lineno}"
+            if not isinstance(rec.get("prompt"), str):
+                qc.refuse(f"{where} has no string 'prompt' field")
+            windows, n_parent = cut(rec, tok, a, where)
+            docs_in += 1
+            tokens_in += n_parent
+            for w in windows:
+                out.write(json.dumps(w.record(), ensure_ascii=False) + "\n")
+                docs_out += 1
+                tokens_out += w.n_tokens
+                longest_child = max(longest_child, w.n_tokens)
+                clamped += 1 if w.clamped else 0
+                scored += w.n_tokens - w.score_from
+                if "source" in rec:
+                    for cls in str(rec["source"]).split("+"):
+                        sources[cls] = sources.get(cls, 0) + 1
+    if tokens_out != tokens_in:
+        raise SystemExit(f"REFUSED: the windows hold {tokens_out} tokens and the corpus "
+                         f"has {tokens_in} — re-chunking lost or invented tokens")
+    return {
+        "corpus": a.corpus,
+        "out": a.out,
+        "tokenizer": os.path.basename(a.tokenizer_json),
+        "documents_in": docs_in,
+        "documents_out": docs_out,
+        "tokens_in": tokens_in,
+        "tokens_out": tokens_out,
+        "longest_document": longest_child,
+        "max_doc_tokens": a.max_doc_tokens,
+        "score_tail": a.score_tail,
+        "scored_positions": scored,
+        "clamped_score_from": clamped,
+        "source_documents": sources,
+    }
+
+
+def main() -> None:
+    a = parse_args()
+    validate(a)
+    print(json.dumps(rechunk(a, qc.load_tokenizer(a.tokenizer_json)), indent=1))
+
+
+if __name__ == "__main__":
+    main()

scripts/recipes/qwen4exp-qwen38-w5nl64-bf16trunk.recipe

@@ -0,0 +1,80 @@
+# qwen4exp-qwen38-w5nl64-bf16trunk.recipe — every routed expert, BOTH projections, at five bits:
+# u5 codes into libquant's 32-level w5nl table, an E4M3 scale a group of 64 under the fixed 2^-13
+# second level, rotated by the 128-wide Hadamard, chosen by GPTQ against $CALIB and swept three
+# times by coordinate descent. This is `w5nl64a8h` in the engine's served-encoding table — the top of
+# the four-bit class — applied wider than any published radiance container of this checkpoint: the
+# author's own w5 rung (`qwen4exp-w5dn64-i8-hc8m.recipe`) moves only `down`, because that rung was
+# priced as a byte-for-error trade against prefill; this campaign buys the whole rung and pays for
+# it out of the pinned host pool (+15.3 GiB of expert plane, `--host-pool-mib 61440` per rank), not
+# out of the KV pool.
+#
+# Every other line is the served container's recipe (dump it with
+# `rad-info --recipe /home/dank/models/rad/qwen3.8-next-flash-fp8-iq4r-moe.rad`), minus the rows this
+# campaign leaves at the checkpoint:
+#
+#   * the trunk's block linears, the shared expert and the hyper-connection mixers are NOT listed —
+#     dropping their i8/fp8 lines leaves them bf16, which is what the checkpoint holds. Measured bill
+#     +1.978 GiB of always-resident weight = ~131k tokens of KV off the pool.
+#   * `ssm_inz`/`ssm_out` STAY int8: the delta-net gates are the largest trunk rows (+1.9 GiB, 126k
+#     tokens) and the least sensitive per byte.
+#   * `mtp.draft_head` STAYS 2-bit and the PLE n-gram table STAYS one fixed E4M3 scale: the draft
+#     head's rung is cheap and its acceptance cost is a decode question this campaign does not touch,
+#     and bf16 for the n-gram table would pin +47.69 GiB of host RAM for a lookup.
+#
+# Protected experts: none in this file. `scripts/apply-protected-cast.sh` appends the derived set
+# (48, two per layer so each rank's 256-expert half stays even — the four-bit GEMM takes the rest of
+# a half as two equal tables by parity) from the fresh 12M-token energy/routing census, and they must
+# sit ABOVE the expert row because the first rule that matches a weight decides it.
+#
+#   CALIB=/home/dank/campaign/qwen38-w5nl64/calib \
+#     rad-convert /home/dank/models/Qwen3.8-Flash-Next \
+#       --recipe this.file --imatrix .../imatrix.gguf -o out.rad
+#
+# The arm driver (`scripts/convert-qwen38-w5nl64.sh`) generates the per-arm variants of the expert
+# row from this template — cd=3 (control), cd=6, and `act_order=1` with NO cd= (libquant refuses the
+# pair, lq_registry.cpp:105) — rather than keeping three near-duplicate recipes that can drift apart.
+#
+# The whole campaign, in the order that fits on one disk (every step runs on dankserver):
+#
+#   1. scripts/download-qwen38-bf16.sh --fetch      the vendor BF16 checkpoint, 335.3 GiB
+#   2. scripts/build-qwen38-bf16-rad.sh             the BF16 reference container, another 335.3 GiB
+#   3. one drained window, two boots:
+#        scripts/run-qwen38-bf16.sh kldref          the KLD reference recording (the zero of scale)
+#        scripts/run-qwen38-bf16.sh calib           the 12M-position calibration tap -> grams
+#   4. trash qwen38-bf16.rad                        the peak-disk law: the converts need its room
+#   5. scripts/grams-to-imatrix.py                  grams -> imatrix.gguf in the campaign dir
+#   6. scripts/convert-qwen38-w5nl64.sh             the arms, planned and billed before each one runs
+#   7. fetch the published FP8-native release, then per container:
+#        scripts/run-qwen38-bf16.sh score RAD OUT   fp8 anchor, served iq4r, each arm
+#      then scripts/kld-gate-qwen38.sh fp8.json iq4r.json cand.json
+#   8. a winner is a trio edit plus `scripts/deploy.sh dankserver --service frank-archon`; the
+#      rollback is the commit before it. If the win needs the bigger pinned plane, raise
+#      FRANK_ARCHON_HOST_POOL_MIB and FRANK_ARCHON_TTM_TARGET_MIB in the SAME commit: the preflight
+#      floor is 2xHOST_POOL + 48,824 + prefix cache + MNBT + the store, and a pool raised without
+#      the TTM target is a boot refusal that Restart=always repeats forever, invisible to deploy.sh.
+
+# PROTECTED-BEGIN — scripts/apply-protected-cast.sh splices the derived cast lines in here.
+# They MUST sit above the expert rule: the first rule whose glob matches a weight decides it, so a
+# cast line below the five-bit rule would never win. The set is derived from the campaign's own
+# 12M-token recording (`derive-protected-experts.py --target-count 48` over the fresh grams), and the
+# driver splices it into a per-arm COPY so this template stays the clean, reviewable baseline.
+# PROTECTED-END
+
+# Every routed expert, trunk and MTP layer alike, both projections, at five bits. The experts' scales
+# sit between 2^-15 and 2^-10.6 under w4nl's top level of 22 and half a binade lower under w5nl's 30
+# — inside E4M3's normal range under 2^-13 either way. The MTP layer has no recording and is rounded
+# to nearest, which libquant says on stderr; it is one layer of 49 and only ever changes drafts.
+blk.*.ffn_*_exps.*.weight            gptq table=w5nl group=64 scale=fp8_e4m3 scale2=f32 block2=*x* scale2_value=0.0001220703125 transform=fwht128 rule=search cd=3 calib=$CALIB
+
+# The delta-net's in/out gates, int8 with a bf16 scale a group of 128 searched for least error — the
+# served recipe's lines, unchanged.
+blk.*.ssm_inz.weight                 rtn codes=i8 group=128 scale=bf16 clamp=sym rule=search
+blk.*.ssm_out.weight                 rtn codes=i8 group=128 scale=bf16 clamp=sym rule=search
+
+# The MTP head's two mixing rows, E4M3 as served. They only ever change the drafts.
+mtp.fc_hidden.weight                 rtn codes=fp8_e4m3 group=128 scale=f32
+mtp.fc_embedding.weight              rtn codes=fp8_e4m3 group=128 scale=f32
+
+# The n-gram table at one fixed E4M3 scale, and the 2-bit draft head — both the served lines.
+blk.*.ple_ngram.weight               rtn codes=fp8_e4m3 block=*x* scale=bf16 rule=fixed scale_value=1.99317932128906e-4
+mtp.draft_head.weight                rtn codes=u2 zero=u8 group=128 scale=f16

scripts/run-qwen38-bf16.sh

@@ -0,0 +1,437 @@
+#!/usr/bin/env bash
+# run-qwen38-bf16.sh — RUNS ON DANKSERVER. Boot a container over the campaign's
+# ONE measuring shape for the phases that need both R9700s. The engine owns the
+# cards, so frank-archon is stopped first and restored on every exit path — the
+# autoresearch pattern: stop, never mask (the unit file in /etc/systemd/system
+# is a symlink into the checkout and mask fails on a relinked unit; the deploy
+# restores it).
+#
+#   kldref            --kld-record the reference log-probabilities: the engine
+#                     runs /home/dank/corpora/out/kld.jsonl (197 prompts, 50,432
+#                     scored positions, score tail 256) through prefill, writes
+#                     /home/dank/campaign/qwen38-w5nl64/kld-ref, and EXITS. The
+#                     model behind it is the BF16 container from
+#                     scripts/build-qwen38-bf16-rad.sh — the vendor checkpoint's
+#                     own numbers, which is what makes it the zero point.
+#   score RAD OUT     --kld-ref score ONE container against that recording and
+#                     EXIT: the FP8-native anchor, the served iq4r, and every
+#                     arm. Deliberately the SAME engine pin, the same TP, the
+#                     same step size, the same fp8 KV, --deterministic and no
+#                     speculation as kldref — KL and top-1 agreement are only
+#                     comparable across boots when everything except the weights
+#                     is identical, so this mode is one argv with --model and
+#                     the two --kld-* flags changed, never a second shape.
+#                     RAD must sit under the models dir (the container reads it
+#                     through the ro /models mount); OUT lands on a writable
+#                     mount because --kld-out F also writes F.rows beside it.
+#   calib             serve on :7770 with the calibration tap open
+#                     (RADIANCE_CALIB_DIR + RADIANCE_CALIB_TOKENS=12000000). A
+#                     separate driver — scripts/calib-push.py --port 7770
+#                     --container qwen38-bf16 --corpus
+#                     /home/dank/corpora/out/calib.jsonl — pushes the corpus; at
+#                     12M pooled positions the engine writes the Grams and the
+#                     tap switches itself off (calib-push prints TAP_DONE). Stop
+#                     this script after that; the trap brings frank-archon back.
+#
+# Shape: the vendor's bf16 calibration lines — expert_tiered placement with the
+# weights disk tier, deterministic (the resident set pinned to the loader's plan,
+# so a recorded reference and a scored candidate see the same one), no
+# speculation, fp8 KV, a 32,768-token window, the PLE n-gram table pinned to RAM,
+# and the production chat template rather than the one baked into the container —
+# every scored container is asked the same tokenised questions the reference
+# answered. The window the flag spells is 32,768; the pool this payload's boot
+# actually gets is 7,436 tokens, so kld.jsonl is re-chunked by
+# scripts/rechunk-kld-corpus.py into documents of at most 7,168 tokens — the
+# widest one the reference can hold. The kldref gate below re-checks the file
+# against the declared window, and the re-chunker is what keeps that honest.
+#
+# NO --vram-kv-mib: it was REMOVED at 1.3.0 and naming it is a boot refusal, not
+# a no-op — verified on the pinned image 2026-10-27: "E --vram-kv-mib was
+# removed: the card is split between the weights and the cache from what the
+# model needs." The resolver derives the split from --host-pool-mib,
+# --gpu-headroom-mib and the activation arena; the rig inherits the first two from
+# frank-archon.env, the file that owns them. On THIS payload the host pool is not
+# the cache's supplier: the boot lands in vram_budget.cpp's `no lender` branch,
+# where the experts keep their plan-fixed 25.29 GiB and the cache gets the residue.
+# Measured twice — 12,288 and 49,152 MiB of host pool both printed the same
+# "kv cache 271.25 MiB ... 7,436"; only --gpu-headroom-mib moved anything
+# (768 -> 96 grew vram_weights 29.41 -> 30.06 GiB). The supplier the source names
+# is the arena: vram_budget.cpp:117, "lower --max-num-batched-tokens (the arena
+# scales with it)" — which is what the step size below now does.
+#
+# The pool, not the window, decides what the recorder accepts: it refuses any
+# document longer than the pool, so the 32,768-token window was decorative while the
+# cache held 7,436 — "E kld: document 196 is 8507 tokens and the KV pool holds
+# 7436, so it can never be scored" on all 197 records, exit 1. The longest record
+# is 10,554 tokens by the engine's own count (chars/3.6 for this file, not the
+# chars/4.97 this repo measured on the K3 corpus).
+#
+# The step size is ONE value for the recording and every scoring pass, and it is
+# the value frank-archon.env owns (FRANK_ARCHON_MAX_BATCHED_TOKENS, 2,048): the
+# quantised expert FILE tiers (the served iq4r, the arms) stream only at
+# MNBT<=2048 and refuse the widened form above it, and the arena a step costs the
+# card comes straight out of the cache — 1.08 GiB at a 2,048-token step against
+# 504.39 MiB at 1,024. The corpora are cut to the window the boot ledger reports,
+# which is what the pool raise is for (frank-archon.env: HOST_POOL_MIB); a narrow
+# step was never the fix, it was the symptom's disguise.
+#
+# usage: scripts/run-qwen38-bf16.sh kldref|calib|score RAD OUT [--dry-run] [--image REF]
+#   (RAD and OUT are score's two positionals: RAD is a container under the models
+#    dir, OUT the report path whose directory is created if missing)
+#   --dry-run   print the exact `podman run` argv, one shell-quoted argument a
+#               line, and exit 0: the service is NOT stopped, nothing is
+#               mounted, nothing is written — the offline test seam. It still
+#               validates the argv's own shape (paths through the /models mount,
+#               a step size that is a number), because an argv that could not
+#               boot is not worth printing.
+#   --image REF override the digest pin (a probe boot, or the test seam)
+#
+# knobs: FRANK_QWEN38_MODELS (the dir mounted at /models), FRANK_QWEN38_RAD (the
+#        bf16 container, kldref only), FRANK_QWEN38_CAMPAIGN (holds kld-ref/ and
+#        calib/), FRANK_QWEN38_KLD_CORPUS, FRANK_QWEN38_TPL (the template copy
+#        the container reads through /models), FRANK_QWEN38_LOG_DIR, and the step
+#        and slot counts inherited as FRANK_ARCHON_MAX_BATCHED_TOKENS /
+#        FRANK_ARCHON_MAX_SEQS — no per-mode literals (the audit law: a mode that
+#        carries its own step measures a shape nothing serves; calib is exempt, see
+#        Q38_CALIB_MNBT / Q38_CALIB_SEQS below — its output is never compared) —
+#        and FRANK_QWEN38_MAX_MODEL_LEN — the window every mode boots at (32,768;
+#        see the corpus census above, and raise it together with the corpus).
+# kldref refuses (exit 2) when the longest record of the corpus it is about to
+# read is longer than MAX_MODEL_LEN x 3 characters — chars/3 being the
+# pessimistic token:char ratio, so the script says no before the engine finds out
+# inside a drained window, after the checkpoint work is already done. score
+# refuses when the reference dir is absent, when RAD is not on disk or not under
+# the models mount, or when OUT already exists: the trash law means a second
+# score over the same name is a deliberate act, and two numbers in one file is
+# how a campaign loses its own history.
+# Engine stdout goes to $LOG_DIR/qwen38_bf16_<mode>.log — the boot ledger is the
+# record of what the resolver actually gave the cards; the journal never sees it.
+# exit: the engine's code; 2 = refused before the service was touched
+set -uo pipefail
+
+SCRIPT_DIR="$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)"
+# shellcheck disable=SC1091
+source "$SCRIPT_DIR/frank-archon.env"
+IMAGE="${FRANK_ARCHON_IMAGE:?FRANK_ARCHON_IMAGE must name the radiance image}"
+
+MODE=""
+POS=()
+DRY=0
+while [ $# -gt 0 ]; do
+    case "$1" in
+        kldref|calib|score) MODE="$1" ;;
+        --dry-run)          DRY=1 ;;
+        --image)            [ "$#" -ge 2 ] || { echo "--image needs a reference" >&2; exit 2; }
+                            IMAGE="$2"; shift ;;
+        *)
+            # score's two positional arguments, and nothing else may be bare.
+            if [ "$MODE" = score ] && [ "${#POS[@]}" -lt 2 ]; then
+                POS+=("$1")
+            else
+                echo "usage: ${0##*/} kldref|calib|score RAD OUT [--dry-run] [--image REF]  (unknown argument: $1)" >&2
+                exit 2
+            fi ;;
+    esac
+    shift
+done
+if [ -z "$MODE" ]; then
+    echo "usage: ${0##*/} kldref|calib|score RAD OUT [--dry-run] [--image REF]  (no mode given)" >&2
+    exit 2
+fi
+if [ "$MODE" = score ] && [ "${#POS[@]}" -ne 2 ]; then
+    echo "usage: ${0##*/} score RAD OUT  — RAD is the container to score, OUT the report to write (got ${#POS[@]} argument(s))" >&2
+    exit 2
+fi
+
+MODELS="${FRANK_QWEN38_MODELS:-/home/dank/models}"
+CAMPAIGN="${FRANK_QWEN38_CAMPAIGN:-/home/dank/campaign/qwen38-w5nl64}"
+KLD_CORPUS="${FRANK_QWEN38_KLD_CORPUS:-/home/dank/corpora/out/kld.jsonl}"
+RAD="${FRANK_QWEN38_RAD:-$MODELS/rad/qwen38-bf16.rad}"
+TPL="${FRANK_QWEN38_TPL:-$MODELS/tpl.jinja}"
+TEMPLATE="$SCRIPT_DIR/qwen38-froggeric-v22.5.jinja"
+LOG_DIR="${FRANK_QWEN38_LOG_DIR:-/home/dank/logs}"
+LOG="$LOG_DIR/qwen38_bf16_${MODE}.log"
+KLD_REF_DIR="$CAMPAIGN/kld-ref"
+CALIB_DIR="$CAMPAIGN/calib"
+C=qwen38-bf16
+SERVICE=frank-archon
+# The card split, taken from the file that owns it instead of a private pair:
+# --host-pool-mib also funds the KV cache (see the header), so hardcoding it
+# here once starved the recorder. The `:?` guards keep an empty env fatal.
+HOST_POOL_MIB="${FRANK_ARCHON_HOST_POOL_MIB:?FRANK_ARCHON_HOST_POOL_MIB must be set}"
+GPU_HEADROOM_MIB="${FRANK_ARCHON_GPU_HEADROOM_MIB:?FRANK_ARCHON_GPU_HEADROOM_MIB must be set}"
+# The slot count, from the same file that owns the card split: production runs
+# four slots and every mode measures at production's geometry. A per-mode
+# literal here is what let calib ask for eight slots and then starve its own
+# cache. The `:?` guard keeps an empty env fatal, as for the card split.
+MAX_SEQS="${FRANK_ARCHON_MAX_SEQS:?FRANK_ARCHON_MAX_SEQS must be set}"
+if ! [[ "$MAX_SEQS" =~ ^[0-9]+$ ]]; then
+    echo "REFUSED: FRANK_ARCHON_MAX_SEQS must be a slot count (got: $MAX_SEQS)" >&2
+    exit 2
+fi
+
+# ONE step size for every mode, quoted from the file that owns it — the audit
+# law for a radiance rig is to inherit the proven number, not to carry a
+# campaign-private value: production's FRANK_ARCHON_MAX_BATCHED_TOKENS. The
+# 1,024 default this script carried was PR #80's answer to a starved cache;
+# production moved to its widest legal step (2,048) on 2026-10-29 and the rig
+# follows. The pool raise that shipped with it (66,560 a rank) bought this
+# container 1.26 MiB of cache (`vram_kv 269.99 -> 271.25 MiB`, window still
+# 7,436 tokens — the 95.37 GiB n-gram table's claim decides the rig's cache,
+# not the pool) and was folded back the same day. The corpora stay cut to the
+# window the boot ledger reports, never the other way round.
+MNBT="${FRANK_ARCHON_MAX_BATCHED_TOKENS:?FRANK_ARCHON_MAX_BATCHED_TOKENS must be set}"
+if ! [[ "$MNBT" =~ ^[0-9]+$ ]]; then
+    echo "REFUSED: FRANK_ARCHON_MAX_BATCHED_TOKENS must be a token count (got: $MNBT)" >&2
+    exit 2
+fi
+
+# Calib's shape is NOT a measuring shape. kldref and score produce KL numbers that are only
+# comparable across boots at an identical shape, which is why they inherit production's step and
+# slots to the letter. The tap's output is layer Hessians, never a number any report compares, so
+# its step and slot count are its own — and they are the tap's dominant cost: a 7,168-token
+# document cut into four 2,048-token steps re-streams each layer's routed expert union four
+# times from the container, so one wide step is worth up to ~4x the read volume. Measured
+# 2026-10-29 at the inherited 2,048/4: prefill 71 tok/s and a 47-hour tap; the probe exists to
+# find out what the wide step is worth before the campaign pays for two days of drained serving.
+# Both default to the inherited value, so nothing moves unless a probe asks it to.
+CALIB_MNBT="${Q38_CALIB_MNBT:-$MNBT}"
+CALIB_SEQS="${Q38_CALIB_SEQS:-$MAX_SEQS}"
+for pair in "Q38_CALIB_MNBT=$CALIB_MNBT" "Q38_CALIB_SEQS=$CALIB_SEQS"; do
+    if ! [[ "${pair#*=}" =~ ^[0-9]+$ ]]; then
+        echo "REFUSED: ${pair%%=*} must be a number (got: ${pair#*=})" >&2
+        exit 2
+    fi
+done
+
+# The window every mode boots at. Over-length is a REFUSAL, not a truncation (the
+# env file says so for production), so this is the number the corpus must fit in.
+MAX_MODEL_LEN="${FRANK_QWEN38_MAX_MODEL_LEN:-32768}"
+if ! [[ "$MAX_MODEL_LEN" =~ ^[0-9]+$ ]]; then
+    echo "REFUSED: FRANK_QWEN38_MAX_MODEL_LEN must be a token count (got: $MAX_MODEL_LEN)" >&2
+    exit 2
+fi
+
+# Where the n-gram table lives. On the bf16 payload it is 95.37 GiB and `ram`
+# PINS it: the calibration boot died at "startup stopped: out of memory" with
+# 230.22 GiB pinned of 246.07 GiB and 8.06 GiB available (measured 2026-10-29),
+# while the reference recording and every score boot had already succeeded with
+# the same pool and slots. The difference that matters is the job: calib only
+# needs the routed weights prefilled, so it reads the table from the container's
+# own file tier and the pin never happens; kldref and score keep production's
+# placement, because their numbers are only comparable when every other term of
+# the shape is identical. FRANK_QWEN38_NGRAM_PLACEMENT overrides either default.
+case "$MODE" in
+    calib) NGRAM_PLACEMENT="${FRANK_QWEN38_NGRAM_PLACEMENT:-disk}" ;;
+    *)     NGRAM_PLACEMENT="${FRANK_QWEN38_NGRAM_PLACEMENT:-ram}" ;;
+esac
+case "$NGRAM_PLACEMENT" in
+    auto|vram|ram|disk) ;;
+    *) echo "REFUSED: FRANK_QWEN38_NGRAM_PLACEMENT must be auto|vram|ram|disk (got: $NGRAM_PLACEMENT)" >&2
+       exit 2 ;;
+esac
+
+# Every container this campaign boots is read through the ro /models mount, so
+# the host path must live under it; the container-side spelling is what goes
+# into the argv. Same rule for the container a score pass opens.
+under_models() {
+    case "$1" in
+        "$MODELS"/*) printf '/models/%s' "${1#"$MODELS"/}" ;;
+        *) return 1 ;;
+    esac
+}
+
+# same_dir A B — the two name one directory, judged only when BOTH exist: a dry
+# run has no fixtures on disk, and two absent paths are not the same place.
+same_dir() {
+    [ -d "$1" ] && [ -d "$2" ] && [ "$(cd "$1" && pwd)" = "$(cd "$2" && pwd)" ]
+}
+
+if [ "$MODE" = score ]; then
+    SCORE_RAD="${POS[0]}"
+    SCORE_OUT="${POS[1]}"
+    OUT_DIR="$(dirname "$SCORE_OUT")"
+    MODEL_IN="$(under_models "$SCORE_RAD")" || {
+        echo "REFUSED: container to score $SCORE_RAD is not under the mounted models dir $MODELS — the engine cannot read it" >&2
+        exit 2
+    }
+else
+    MODEL_IN="$(under_models "$RAD")" || {
+        echo "REFUSED: reference container $RAD is not under the mounted models dir $MODELS — the engine cannot read it" >&2
+        exit 2
+    }
+fi
+
+# The measuring shape. Identical for kldref and score by construction — this
+# array is the only place the campaign's KL numbers are defined.
+ARGS=(--tp 2 --placement expert_tiered --weights-disk-tier
+      --host-pool-mib "$HOST_POOL_MIB" --gpu-headroom-mib "$GPU_HEADROOM_MIB"
+      --deterministic --num-speculative-tokens 0
+      --kv-cache-dtype fp8 --max-model-len "$MAX_MODEL_LEN"
+      --ngram-placement "$NGRAM_PLACEMENT"
+      --override-chat-template /models/tpl.jinja)
+MOUNTS=(-v "$MODELS:/models:ro")
+TAPENV=()
+case "$MODE" in
+  kldref)
+    # Recording is prefill-only over four slots at the campaign's single step
+    # size, which is frank-archon.env's FRANK_ARCHON_MAX_BATCHED_TOKENS (2,048) —
+    # see the header: the file tier's ceiling and the slot count both come from
+    # the file that owns them. Corpus read-only, record dir rw.
+    ARGS+=(--model "$MODEL_IN"
+           --kld-record /data/kld-ref --kld-corpus /data/kld.jsonl
+           --max-num-seqs "$MAX_SEQS" --max-num-batched-tokens "$MNBT")
+    MOUNTS+=(-v "$KLD_CORPUS:/data/kld.jsonl:ro"
+             -v "$KLD_REF_DIR:/data/kld-ref")
+    ;;
+  score)
+    # The same argv with three words changed: which container, and that this
+    # time the engine reads a recording and writes a report instead of the
+    # reverse. The reference dir mounts READ-ONLY — a scoring pass must not be
+    # able to touch the zero point — and the report gets its own writable bind.
+    if [ "$OUT_DIR" = "$KLD_REF_DIR" ] || same_dir "$OUT_DIR" "$KLD_REF_DIR" \
+       || [ "$(realpath -m "$OUT_DIR")" = "$(realpath -m "$KLD_REF_DIR")" ]; then
+        echo "REFUSED: score output $SCORE_OUT is inside the reference dir $KLD_REF_DIR, which this mode mounts read-only" >&2
+        exit 2
+    fi
+    ARGS+=(--model "$MODEL_IN"
+           --kld-ref /data/kld-ref --kld-out "/data/out/$(basename "$SCORE_OUT")"
+           --max-num-seqs "$MAX_SEQS" --max-num-batched-tokens "$MNBT")
+    MOUNTS+=(-v "$KLD_REF_DIR:/data/kld-ref:ro"
+             -v "$OUT_DIR:/data/out")
+    ;;
+  calib)
+    # Calibration rides the shared step like every other mode. The old
+    # exemption — eight slots at MNBT 65,536 for "free throughput", because
+    # "a disk-tier step costs the same at 1 token as at 64k" — is false: the
+    # step's activation arena is billed on the card, not on the disk. The
+    # engine's own ledger for this container: at MNBT 1,024 the arena is
+    # 504 MiB and the card claims 30.87 GiB (weights 28.90 GiB of it); at
+    # 65,536 the arena is 10.65 GiB (fixed region 1.25 GiB), the claim falls
+    # to 16.85 GiB, the experts shrink to 10.98 GiB and the KV to 271 MiB —
+    # one slot's worth of cache for eight slots, and the boot dies at
+    # "startup stopped: out of memory" (measured 2026-10-29). A wide step
+    # buys throughput with the expert slab, so price it like any other
+    # VRAM knob. The slots follow frank-archon.env's FRANK_ARCHON_MAX_SEQS
+    # through MAX_SEQS, exactly as kldref and score do.
+    ARGS+=(--model "$MODEL_IN" --port 7770
+           --max-num-seqs "$CALIB_SEQS" --max-num-batched-tokens "$CALIB_MNBT")
+    # The tap is only real if this directory exists and the engine can write it:
+    # without the check a typo'd CALIB_DIR auto-creates a root-owned empty dir in
+    # whatever path was given, and the run burns hours serving prompts into a tap
+    # nobody reads. RADIANCE_CALIB_TOKENS is the whole corpus' floor, not a round's.
+    [ "$DRY" = 1 ] || [ -w "$CALIB_DIR" ] || {
+        echo "REFUSED: calib tap dir $CALIB_DIR is not writable (mkdir it, chown it, then re-run)" >&2
+        exit 2
+    }
+    MOUNTS+=(-v "$CALIB_DIR:/data/calib")
+    TAPENV=(-e RADIANCE_CALIB_DIR=/data/calib
+            -e "RADIANCE_CALIB_TOKENS=${FRANK_QWEN38_CALIB_TOKENS:-12000000}")
+    # The tap's own authorization. A pusher must see this marker before it sends
+    # a prompt (calib-push.py --tap-marker), it names the container so a push
+    # aimed anywhere else refuses, and a boot that fails removes it again below.
+    # Measured 2026-10-29: with no marker, the chain pushed 4 rounds x 10M tokens
+    # into RESTORED PRODUCTION after the rig's boot died, and nothing recorded.
+    if [ "$DRY" = 1 ]; then
+        :
+    else
+        printf 'container=%s\nimage=%s\nmode=calib\nstarted=%s\npid=%s\n' \
+            "$C" "$IMAGE" "$(date -Is)" "$$" > "$CALIB_DIR/.rig-boot"
+    fi
+    ;;
+esac
+
+CMD=(podman run --rm --name "$C"
+     --network host --init
+     --security-opt seccomp=unconfined
+     --ulimit memlock=-1:-1
+     --device /dev/kfd --device /dev/dri
+     "${MOUNTS[@]}" "${TAPENV[@]}"
+     "$IMAGE" "${ARGS[@]}")
+
+# ---- gate: the window must hold the corpus (kldref alone can meet it) --------
+# score reads a recording instead of a corpus, and every arm declares at the
+# checkpoint's training context, so neither can meet a sequence too long to boot.
+# Measured on the raw JSON line — the pessimistic direction, and the same figure
+# the census in the header quotes. It runs on a dry run too, because a refusal
+# this cheap should never have to wait for the drained window to be discovered.
+if [ "$MODE" = kldref ] && [ -f "$KLD_CORPUS" ]; then
+    read -r REC_NO REC_CHARS < <(awk '{ if (length($0) > best) { best = length($0); where = NR } } END { print where + 0, best + 0 }' "$KLD_CORPUS")
+    LIMIT_CHARS=$(( MAX_MODEL_LEN * 3 ))
+    if [ "$REC_CHARS" -gt "$LIMIT_CHARS" ]; then
+        echo "REFUSED: record $REC_NO of $KLD_CORPUS is $REC_CHARS chars, past the $LIMIT_CHARS a $MAX_MODEL_LEN-token window promises at the pessimistic chars/3 — raise FRANK_QWEN38_MAX_MODEL_LEN, and the container's declared --max-ctx with it, before booting the reference" >&2
+        exit 2
+    fi
+    echo "corpus fits: longest record $REC_CHARS chars (number $REC_NO) inside the $MAX_MODEL_LEN-token window (limit $LIMIT_CHARS chars)"
+fi
+
+if [ "$DRY" = 1 ]; then
+    echo "dry run: $MODE — the production unit is left running, nothing is mounted, nothing is written"
+    printf '%q\n' "${CMD[@]}"
+    exit 0
+fi
+
+# ---- gates that only a real run needs ---------------------------------------
+if [ "$MODE" = score ]; then
+    if [ ! -d "$KLD_REF_DIR" ]; then
+        echo "REFUSED: no KLD reference at $KLD_REF_DIR — record it first in a drained window: scripts/run-qwen38-bf16.sh kldref" >&2
+        exit 2
+    fi
+    if [ ! -f "$SCORE_RAD" ]; then
+        echo "REFUSED: no container to score at $SCORE_RAD" >&2
+        exit 2
+    fi
+    if [ -e "$SCORE_OUT" ]; then
+        echo "REFUSED: $SCORE_OUT already exists — one report per name; move the old one to trash before re-scoring" >&2
+        exit 2
+    fi
+    mkdir -p "$OUT_DIR"
+else
+    if [ ! -f "$RAD" ]; then
+        echo "REFUSED: no reference container at $RAD — build it first: scripts/build-qwen38-bf16-rad.sh" >&2
+        exit 2
+    fi
+    [ "$MODE" = kldref ] && [ ! -f "$KLD_CORPUS" ] && {
+        echo "REFUSED: no KLD corpus at $KLD_CORPUS — the record would be empty" >&2
+        exit 2
+    }
+fi
+if [ ! -f "$TEMPLATE" ]; then
+    echo "REFUSED: no chat template at $TEMPLATE" >&2
+    exit 2
+fi
+mkdir -p "$LOG_DIR"
+# The prod template, staged where the ro /models mount reaches it: a reference
+# recorded with the container's baked template scores a different tokenisation
+# than the arms served with froggeric's, and the KL would be measuring that.
+cp -f "$TEMPLATE" "$TPL" || {
+    echo "REFUSED: cannot stage the chat template at $TPL (the container reads it through /models)" >&2
+    exit 2
+}
+[ "$MODE" = kldref ] && mkdir -p "$KLD_REF_DIR"
+[ "$MODE" = calib ]  && mkdir -p "$CALIB_DIR"
+
+# Stop the campaign container BEFORE handing the cards back: a killed run leaves
+# its conmon alive (--rm only fires when podman itself exits), and production
+# then boots beside a container still holding the GPUs — measured 2026-10-29,
+# where the service reported `active` while its engine starved on the rig's VRAM.
+restore() {
+    podman rm -f "$C" >/dev/null 2>&1 || true
+    sudo systemctl start "$SERVICE" >/dev/null 2>&1 || true
+}
+trap restore EXIT
+echo "run-qwen38-bf16 $MODE: stopping $SERVICE ($(date -Is)); log: $LOG"
+sudo systemctl stop "$SERVICE" >/dev/null 2>&1 || true
+podman rm -f "$C" >/dev/null 2>&1 || true
+
+"${CMD[@]}" 2>&1 | tee "$LOG"
+rc=${PIPESTATUS[0]}
+# A boot that failed must not leave the tap marked as rig-authorized: the marker
+# is what calib-push.py trusts, and the point of it is that a dead rig can never
+# be pushed into again.
+if [ "$MODE" = calib ] && [ "$rc" -ne 0 ] && [ -f "$CALIB_DIR/.rig-boot" ]; then
+    mv -f "$CALIB_DIR/.rig-boot" "$CALIB_DIR/.rig-boot.failed-$(date +%Y%m%d-%H%M%S)" 2>/dev/null || true
+    echo "the rig boot failed (rc=$rc): the tap marker is retired, so no pusher can take this tap for a live rig"
+fi
+echo "run-qwen38-bf16 $MODE exited rc=$rc; $SERVICE restored; log: $LOG"
+exit "$rc"

scripts/serve-frank-dragoon.sh

@@ -136,10 +136,17 @@ fi
 #      The engine re-checks size+sha256 against that catalog at boot; it cannot
 #      fetch anything here because /models is read-only and HALOGEN_DOWNLOAD
 #      stays unset — no outbound connection at boot),
-#   2. the distro XRT's three libraries, each mounted twice (where the image
-#      looks for XRT, and at its own path — the libraries find each other there;
-#      an /opt/xilinx/xrt of symlinks into the system dir dangles inside the
-#      container, which the vendor calls out explicitly),
+#   2. the distro XRT's three libraries, each mounted THREE times: where the
+#      image looks for XRT (/opt/xilinx/xrt/lib), at their own path (the
+#      libraries find each other there), and at /opt/xilinx/lib/<multiarch>
+#      — the NPU child searches that third prefix too, and since the host's
+#      libxrt2 was reconfigured 2026-10-20 it is the ONLY prefix the child
+#      names: omitting it crashes the lane with `No such library
+#      '/opt/xilinx/lib/x86_64-linux-gnu/libxrt_core.so.2'` and the engine
+#      supervisor takes the dead lane down on the whole container (measured
+#      2026-10-21: 88-restart boot loop). Real files only — an /opt/xilinx/xrt
+#      of symlinks into the system dir dangles inside the container, which the
+#      vendor calls out explicitly),
 #   3. the GPU fabric clock HELD: on Strix Halo the iGPU and the NPU co-running
 #      hang the machine while FCLK changes speed, and the engine refuses to
 #      start when it moves — it cannot read host sysfs from inside the
@@ -190,7 +197,9 @@ if [ -n "$NPU_MODELS" ]; then
             exit 1
         fi
         REAL="$(readlink -f "$XRT_DIR/$L")"
-        NPU_ARGS+=(-v "$REAL:/opt/xilinx/xrt/lib/$L:ro" -v "$REAL:$XRT_DIR/$L:ro")
+        MULTIARCH="${XRT_DIR##*/}"
+        NPU_ARGS+=(-v "$REAL:/opt/xilinx/xrt/lib/$L:ro" -v "$REAL:$XRT_DIR/$L:ro"
+                   -v "$REAL:/opt/xilinx/lib/$MULTIARCH/$L:ro")
     done
     HELD=0
     for D in "$SYSFS"/class/drm/card*/device; do

scripts/stack-qwen38-official-fp8.py

@@ -337,8 +337,12 @@ def main():
     if any(meta[shard_ids[i]][4][1] != hd for i in range(n_shards)):
         log("ERROR: PLE shards disagree on HD")
         return 2
-    if hd % 128 != 0 and hd != 160:
-        log(f"WARNING: PLE HD={hd} is neither 160 nor a multiple of 128")
+    if hd != 160 and hd % 128 != 0:
+        # The pack below writes lanes of 128 codes; any other HD would produce a
+        # table that decodes to the wrong rows. The old warning let it continue.
+        log(f"ERROR: PLE HD={hd} is neither 160 nor a multiple of 128, and the "
+            "128-lane pack cannot express it")
+        return 2
 
     # ---- idempotence gate -------------------------------------------------
     stacked_names = [f"{p}.mlp.experts.gate_up_proj" for p in prefixes]
@@ -521,12 +525,16 @@ def verify(stage, prefixes, n_e, shapes, meta, shard_ids, rows, hd,
         }
         for name, (dt, shp) in want.items():
             e = hdr[name]
-            assert e["dtype"] == dt and list(e["shape"]) == shp, name
-            assert e["data_offsets"][1] - e["data_offsets"][0] == \
-                int(np.prod(shp)) * (1 if dt == F8 else 2), name
+            if e["dtype"] != dt or list(e["shape"]) != shp:
+                raise RuntimeError(f"{name}: header is {e['dtype']}{e['shape']}, expected {dt}{shp}")
+            if e["data_offsets"][1] - e["data_offsets"][0] != int(np.prod(shp)) * (1 if dt == F8 else 2):
+                raise RuntimeError(f"{name}: data span is {e['data_offsets']}, shape {shp} needs "
+                                   f"{int(np.prod(shp)) * (1 if dt == F8 else 2)} B")
     _, phdr = parse_header(out_path)
     e = phdr[next(iter(phdr))]
-    assert e["dtype"] == BF16 and list(e["shape"]) == [sum(rows), hd]
+    if e["dtype"] != BF16 or list(e["shape"]) != [sum(rows), hd]:
+        raise RuntimeError(f"PLE table header is {e['dtype']}{e['shape']}, "
+                           f"expected {BF16}[{sum(rows)}, {hd}]")
 
     # sampled PLE rows: recompute and compare bytes exactly
     lut = e4m3_lut()
@@ -546,7 +554,8 @@ def verify(stage, prefixes, n_e, shapes, meta, shard_ids, rows, hd,
                 with open(out_path, "rb") as ofh:
                     ofh.seek(out_off + got_row * hd * 2)
                     got = ofh.read(hd * 2)
-                assert got == want, f"PLE row {got_row} mismatch"
+                if got != want:
+                    raise RuntimeError(f"PLE row {got_row} does not match the decoded source")
             mm.close()
     log("verify: new-file headers, sizes and sampled PLE rows all exact; "
         "expert slabs were byte-checked in-stack")

scripts/test-changed.sh

@@ -265,6 +265,13 @@ for path in "${CHANGED[@]}"; do
             add_test tests/voice_pack_unit.bats
             add_test tests/scripts_syntax.bats
             ;;
+        # The KLD corpus re-chunker: its seam is its own (the campaign's
+        # one-token-series invariant is asserted there), so a change to it does
+        # not land on the generic scripts/* fallback below.
+        scripts/rechunk-kld-corpus.py)
+            add_test tests/qwen38_rechunk.bats
+            add_test tests/scripts_syntax.bats
+            ;;
         scripts/*)
             add_test tests/scripts_syntax.bats
             ;;

serverAppConfigs-docs/Agent/OMP Harness.md

@@ -52,28 +52,31 @@ Where the contract lives:
 The arithmetic (the coupled contract every endpoint's maxTokens comment
 encodes). The authoritative `contextWindow`/`maxTokens` pair per model is on the
 endpoint page ([[frank-archon]], [[frank-dragoon]], [[frank-zealot]]); what
-follows is a **worked example** recomputed from the live files on 2026-10-18 —
-a snapshot, not fleet state — with windows 200,000 / 155,648 / 200,704 and
+follows is a **worked example** recomputed from the live files on 2026-10-21 —
+a snapshot, not fleet state — with windows 200,000 / 200,000 / 200,704 and
 maxTokens 16,384 / 45,056 / 16,384 (archon / dragoon / zealot):
 
 1. **Source trigger** = `thresholdPercent × source contextWindow`; at the live
-   95 % that is 190,000 (archon), 147,866 (dragoon), 190,669 (zealot). The
+   95 % that is 190,000 (archon and dragoon), 190,669 (zealot). The
    bundle clamps the percentage to 1–99 and converts it to a token threshold.
 2. **Output budget** = capped at the target model's `maxTokens` — every role
    here compacts to frank-zealot, so the budget is its 16,384.
-3. **REQUIREMENT**: prompt (≤ trigger) + budget ≤ the target's per-slot
-   window. **At 95 % this no longer holds**: 190,000 + 16,384 = 206,384 >
-   200,704. The registry's own comments design around an 85 % trigger (archon:
-   170,000 est × 1.03 + 16,384 ≈ 191,500 ≤ 200,704, `models.yml:49-56`), which
-   does hold. While `config.yml` and those comments disagree, treat 85 % as the
-   design point and re-run this arithmetic after ANY window change — a 400 from
-   the compaction target mid-conversation is the symptom.
+3. **REQUIREMENT**: prompt (≤ trigger) + budget ≤ the fitting window. For a
+   dragoon session the fitting window is the ENGINE ceiling, not a slot: the
+   190,000 trigger is 62,144 under the 262,144 ceiling, so even ×1.03 est
+   error plus the 45,056 request budget lands inside — that headroom is the
+   reason the ceiling moved to native on 2026-10-21. For a session compacting
+   INTO frank-zealot the window is its 200,704-token slot, and **at 95 % this
+   does not hold for archon/zealot sources**: 190,000 + 16,384 = 206,384 >
+   200,704. Treat 85 % as the design point until config.yml and the registry
+   comments agree, and re-run this arithmetic after ANY window change — a 400
+   from the compaction target mid-conversation is the symptom.
    (Historic 2026-10-05 pairs, at `thresholdPercent: 70`: 140,492 + 36,864 =
    177,356 ≤ 200,704 archon; 140,490 + 57,344 ≤ 200,700 dragoon; 140,492 +
    32,768 ≤ 200,704 zealot.)
 
 Procedure (recompute from the LIVE files every time — windows rotate with
-every swap; the block above is the 2026-10-18 state):
+every swap; the block above is the 2026-10-21 state):
 
 1. Read the `config.yml` compaction block + `models.yml` entries for source
    and target. Every endpoint's `contextWindow` comment carries its own
@@ -94,9 +97,10 @@ Known reconciliation trap: comments in models.yml have described compaction
 as running "on the dankserver endpoint" while `remoteEndpoint` pointed at
 dankclaw — the comment/config disagreed. The comments now name the real
 target (`compaction.remoteEndpoint` = dankclaw :7771). **The config is
-truth; fix comments in the same change.** A fresh instance of the same trap is
-open 2026-10-18: every `maxTokens` comment still says "compaction fires at
-85 % of the window" while `config.yml:84` sets `thresholdPercent: 95`.
+truth; fix comments in the same change.** The 85 %-comments vs 95 %-config
+disagreement is not academic: it is why long dragoon sessions kept drawing
+400s at the old 200,704 ceiling — the trigger fires late AND the est
+under-counts, and a window parked near the ceiling has no room for either.
 `maxTokens` never changes for a reslice — it caps output; `contextWindow` caps
 input. Boot-shape OOM laddering is a different problem
 ([[Llama Cpp Serving Sizing]]).

serverAppConfigs-docs/Architecture/Quadlet Lifecycle.md

@@ -130,10 +130,19 @@ Quickest unblock: `podman compose up -d --force-recreate <svc>`. Durable fix
 
 ## 4. Podman 5.x quadlet v1 generator quirks (verified live 2026-08-28)
 
-- **Hard-fail keys** — the whole unit silently fails to generate
-  (`systemctl --user cat <unit>` shows "could not be found"): any
-  dash-prefixed `[Container]` option (`--shm-size=…` → `unsupported key …
-  in group 'Container'`); `Command=` is unsupported, use `Exec=`.
+- **Hard-fail keys abort the WHOLE BATCH, not one unit** (podman 5.7.0,
+  live 2026-10-21): one unsupported key — e.g. `HostAddr=` — makes the
+  generator log `unsupported key … in group 'Container'` plus
+  `processing encountered some errors` and generate NOTHING for that
+  manager. The running containers keep going as orphans, so the failure
+  first appears on the next deploy as `Unit <x>.service not found` for a
+  service that is visibly up. Other offenders: any dash-prefixed
+  `[Container]` option (`--shm-size=…`), `Command=` (use `Exec=`).
+  The seam `tests/quadlet_image_pins.bats` fails the build on any
+  generator-rejected key in any host's units.
+  Also: an image whose ENTRYPOINT is the binary takes FLAGS ONLY in
+  `Exec=` — repeating the binary makes it a positional arg
+  (`unexpected /bin/blackbox_exporter`, live 2026-10-21).
 - **HealthCmd space-splitting:** the v1 generator splits `HealthCmd` on
   spaces into the healthcheck argv; a quoted probe
   `/bin/sh -c "pg_isready …"` lands as argv token `"pg_isready` →
@@ -148,6 +157,16 @@ Quickest unblock: `podman compose up -d --force-recreate <svc>`. Durable fix
   "unsupported key"`, `systemctl --user cat probe.service` (shows the
   generated ExecStart verbatim), then remove + reload.
 
+## 4a. Image pins (fleet law, 2026-10-21)
+
+Every registry image in a quadlet is digest-pinned (`tag@sha256:…`);
+locally built images (`localhost/*`, `franksdank/*`) are exempt — their
+build is the pin. A rolling tag lets a rebuild move under a running
+service with nobody approving it, and it makes a CVE audit ungradeable.
+Gate: `tests/quadlet_image_pins.bats`. Where upstream publishes no version
+tags (copyparty-ac stops at 1.19.15), pin `:latest@sha256:` and name the
+version the config label carries in the comment beside it.
+
 ## 5. Rootless Quadlet log retrieval
 
 Step 0 — establish ownership: `systemctl --user status <name>.service`

serverAppConfigs-docs/Architecture/Stale Signal Audit.md

@@ -116,6 +116,20 @@ as working when they could never report up. **Clean cutover includes prose** —
 docs, unit docstrings, and the repo's rule file (the most expensive place to
 be stale).
 
+## Three rules that outlive the incident that taught them
+
+* **A target can read down beside a healthy service.** Read `lastError` from
+  `/api/v1/targets` before believing an outage — llama.cpp refuses `/metrics` at the
+  protocol layer without `--metrics` (501) while `/health` answers 200 for weeks.
+* **Alert state lives where the tool says it does.** Prometheus keeps rule state in
+  process memory — a restart re-stamps `activeAt`, so a "fresh" alert after a reboot is
+  not evidence of a fresh fault. Alertmanager **does** persist (`--storage.path` on a
+  host volume: silences + notification log, [[Alertmanager]]).
+* **Never quote a candidate arm's flags or VRAM as fleet state.** A stopped production
+  unit (`failed` after `stop-sigterm timed out. Killing` — the known shutdown shape,
+  [[frank-zealot]]) beside an answering port means the metric measured a scratch boot;
+  the process shape can change between two censuses ten minutes apart.
+
 ## Order and verification
 
 1. Enumerate the surfaces (status board, metrics targets, alert rules, test

serverAppConfigs-docs/Dankclaw/ComfyUI Qwen-Image Workflows.md

@@ -110,9 +110,9 @@ only `models`, `input`, `output`, `custom_nodes` and the healthcheck file
 
 ## Recommended additions
 
-Quantized preferred, sized for the 8060S iGPU (GTT-backed — `/system_stats`
-reports 122.4 GB of device VRAM on `cuda:0`, `curl http://127.0.0.1:8188/system_stats`
-2026-10-18; the RTX 3090 is frank-zealot's). All repos verified 2026-10-05 and
+Quantized preferred. The iGPU's GTT aperture is not free RAM beside
+frank-dragoon — size every pick against *What actually fits beside
+frank-dragoon* below (the RTX 3090 is frank-zealot's). All repos verified 2026-10-05 and
 all four still resolve today (`huggingface.co/lightx2v/Qwen-Image-Lightning`
 `huggingface.co/unsloth/Qwen-Image-Edit-2511-GGUF`
 `huggingface.co/Comfy-Org/Qwen-Image-DiffSynth-ControlNets`
@@ -133,8 +133,34 @@ sha256 checks) or a plain `hf download` with the token in the environment —
 never on the command line.
 
 Skip list: the original `city96/Qwen-Image-gguf` conversions (superseded by
-the 2.1 files already on disk), and any BF16 of #2/#4 (the iGPU does not need
-them at GTT capacity, and they double the disk bill).
+the 2.1 files already on disk), and any BF16 of #2/#4 — they double the disk
+bill, and see *What actually fits beside frank-dragoon* for why raw GTT size
+is not the real budget.
+
+## What actually fits beside frank-dragoon
+
+The 122 GB `/system_stats` figure is the GTT **aperture**, not free memory: on
+an APU the aperture and `MemAvailable` are the same physical RAM, and
+frank-dragoon pins tens of GiB of it. Measured 2026-10-21 with frank-dragoon
+live: `MemAvailable` 15.3 GiB, iGPU busy 1.75 GiB — ComfyUI itself idles at
+~10 MB, so the budget for a generation run is roughly what `free` says.
+
+Weights must land inside that budget with slack for compute buffers and
+activations (the mmproj-class lesson: price a model at ~1.5× its file set):
+
+| Pipeline (weights) | Fits beside frank-dragoon? |
+|---|---|
+| Qwen-Image 2.1 int8_convrot + w4a8 + bf16 VAE (~14.2 GB) — the two proven workflows | **No** — thrash-or-fail; do not queue it while the endpoint serves |
+| GGUF Q8_0 + w4a8 + small VAE (~13.3 GB) | No, same reason |
+| Any BF16 combination | No |
+| SDXL-Turbo / SD 1.5-class (~2–7 GB) | Yes, with real headroom |
+
+**The law: one process owns the memory — same rule as the 3090 bench law.**
+For a Qwen-Image render: stop and mask frank-dragoon
+(`sudo systemctl stop frank-dragoon` + `mask` — the mask stops
+`Restart=always` from firing mid-render), generate, unmask, verify
+`curl 192.168.0.143:7770/health`. The proven workflows above then run with
+the whole machine. Do not treat `/system_stats`'s 122 GB as an invitation.
 
 ## See Also
 

serverAppConfigs-docs/Dankclaw/ComfyUI.md

@@ -129,6 +129,12 @@ bind-mounted to `/usr/local/bin/comfyui-healthcheck.py` and run as `HealthCmd`
 - Do not let pip reinstall the torch family. The Containerfile filters it out on purpose.
 - The base image is about 32 GiB (`podman image inspect localhost/comfyui-rocm:gfx1151` → 33,765,227,471 B ≈ 31.4 GiB, 2026-10-18). A code update means a rebuild, because the code is baked in.
 - The empty-models note from the 2026-09-16 era is retired: `/home/dank/comfyui-data/models` now holds the Qwen-Image 2.1 payload — three `diffusion_models`, three `text_encoders`, two VAEs (~35 GB, `ls -l` 2026-10-18), listed in [[ComfyUI Qwen-Image Workflows]]. New model classes (LoRA, ControlNet) still need their directories created first: `models/lora` and `models/controlnet` do not exist yet.
+- **The 122 GB `/system_stats` VRAM figure is the GTT aperture, not free
+  memory.** On the APU it is system RAM, and frank-dragoon pins most of it
+  (`MemAvailable` 15.3 GiB with the endpoint live, measured 2026-10-21). A
+  Qwen-Image-class render beside the running endpoint is thrash-or-fail; the
+  stop-mask-render-verify procedure and the fit table are in
+  [[ComfyUI Qwen-Image Workflows]] → What actually fits beside frank-dragoon.
 
 ## See Also
 

serverAppConfigs-docs/Dankclaw/OpenWebUI.md

@@ -34,15 +34,16 @@ https://github.com/open-webui/open-webui
 The picker is built from the `model` table in `webui.db` (one row per model)
 plus the `model_ids` of each **enabled** OpenAI connection, NOT from live
 connection discovery — so the picker shows exactly what is seeded there.
-As of 2026-10-07 the picker exposes **exactly one model: `frank-zealot`** *(unverified
-2026-10-18: `webui.db` is the authority; the fleet itself serves all three ids
-`frank-dragoon`/`frank-zealot`/`frank-archon`, per the cards in
-`shared/homepage/services.json`)*.
+Since 2026-10-21 the picker exposes **all three fleet models to every
+account**: `frank-dragoon`, `frank-archon`, `frank-zealot` (per Frank: open
+`frank-archon` to everyone; the other two were already public). Proven with
+a `role=user` token: `/api/models` lists all three ids and a chat turn to
+`frank-archon` answers end-to-end.
 
-| Connection (`openai.api_configs` idx) | URL as recorded 2026-10-09 | Enabled | `model_ids` | Endpoint page |
+| Connection (`openai.api_configs` idx) | `openai.api_base_urls` (live 2026-10-21) | Enabled | `model_ids` | Endpoint page |
 |---|---|---|---|---|
-| 0 | `http://host.containers.internal:7770/v1` | no (2026-10-07) | `frank-dragoon` | [[frank-dragoon]] |
-| 1 | `http://192.168.0.235:7770/v1` | no (2026-10-07) | `frank-archon` | [[frank-archon]] |
+| 0 | `http://host.containers.internal:7770/v1` | yes | `frank-dragoon` | [[frank-dragoon]] |
+| 1 | `http://192.168.0.235:7770/v1` | yes | `frank-archon` | [[frank-archon]] |
 | 2 | `http://host.containers.internal:7771/v1` | yes | `frank-zealot` | [[frank-zealot]] |
 
 The URLs in the first column are the DB's recorded values and predate the 2026-08-22
@@ -50,9 +51,10 @@ bridge move; on this network they are wrong — see the reachability gotcha belo
 is the corrected law. The picker row for `frank-zealot` was re-keyed on 2026-10-18 after
 that day's payload swap (see [[frank-zealot]] for what was deployed and when).
 
-The `model` table carries exactly two decoration rows, `frank-zealot` and
-`frank-dragoon`. Both the context length and the description live **inside
-JSON columns** — `params` holds `{"context": {"length": …}}` and `meta` holds
+The `model` table carries three decoration rows — `frank-zealot`,
+`frank-dragoon`, `frank-archon` (200,000 context, added 2026-10-21).
+Both the context length and the description live **inside JSON columns** —
+`params` holds `{"context": {"length": …}}` and `meta` holds
 `{"description": …}`; there is no `context_length` or `description` column, so
 `sqlite3` one-liners against those names fail. `params.context.length` is a copy
 of the endpoint's per-slot window, so the authoritative number lives on the
@@ -380,6 +382,20 @@ neither.
   name: /selected model/i })`) and the welcome text "How can I help you
   today?"; that snapshot is also the cheapest proof the picker is populated.
 
+- **2026-10-21 — picker opened to all three models, for everyone.** Per
+  Frank: `frank-archon` accessible to everyone. Three-leg repair on
+  `webui.db` (snapshot `webui.db.bak-20261021` first): connection idx 1
+  `enable: true`, a `model` row `frank-archon` (params
+  `{"context":{"length":200000}}`, meta description from the endpoint card),
+  and an `access_grant` row `(resource_type='model', resource_id='frank-archon',
+  principal_id='*', 'read')` — `principal_id='*'` is the public grant, not a
+  role row. Verified as a plain `role=user` token: `/api/models` lists all
+  three ids, chat to `frank-archon` answers. Token-mint gotchas: the backend
+  package imports only with cwd `/app/backend`, and `WEBUI_SECRET_KEY` must
+  come from **PID 1's environ** (`podman exec` sessions do not inherit the
+  app process env; the `.webui_secret_key` file happens to equal it — use it
+  explicitly, `podman exec -e`).
+
 ## See Also
 
 - [[Dankclaw Stack]]

serverAppConfigs-docs/Dankserver/Plex.md

@@ -4,7 +4,7 @@ description: "Plex Media Server is a **containerized media server on Dankserver*
 tags:
   - dankserver
   - plex
-timestamp: 2026-09-15
+timestamp: 2026-10-21
 ---
 
 # Plex
@@ -24,7 +24,7 @@ public traffic can be fronted by Caddy like every other vhost.
 | Item | Value |
 |------|-------|
 | Runtime | podman quadlet, `plex.service` (user manager) — migrated from the native `plexmediaserver.service` OS-package unit (2026-09) |
-| Image | `lscr.io/linuxserver/plex:1.43.4.10903-e5521bd8c-ls325` (pinned; verified == the ghcr `latest` manifest digest at pin time) |
+| Image | `lscr.io/linuxserver/plex:1.43.5.11029-6bf8af3f0-ls329` (pinned; verified == the registry's `latest` manifest digest at pin time: `sha256:8f655bad…e9e2334`) |
 | Data root | `/var/lib/plexmediaserver` bind-mounted **in place** as `/config` (RW) — the migration moved zero bytes |
 | Media | `/mnt/entropy` mounted at the same absolute path **read-only** — every library path in the PMS database stayed valid, zero re-scan |
 | Identity | fleet rootless model — container root maps to the `dank` user (1000:1000); the deployer chowns the data root on first cutover |

serverAppConfigs-docs/Infrastructure/Observability Provisioning.md

@@ -3,7 +3,7 @@ type: reference
 title: Observability Provisioning
 description: "Monitoring config as code: the config triad, deploy copiers, planner mappings, and where each change type lands."
 tags: [observability, prometheus, grafana, provisioning, deployment]
-timestamp: 2026-10-05T00:00:00-07:00
+timestamp: 2026-10-21T00:00:00-07:00
 ---
 
 # Observability Provisioning
@@ -20,6 +20,8 @@ committed.
 | `shared/prometheus/prometheus.yml` | scrape jobs (global + per-host) |
 | `shared/prometheus/rules/llm-fleet.yml` | alert rules + recording rules |
 | `shared/grafana/provisioning/` | datasources + dashboards (provisioned tree) |
+| `shared/alloy/config.alloy` | the log pipeline: journal source → label promotion → Loki push |
+| `shared/blackbox/blackbox.yml` | prober modules (`http_2xx`); the listen bind lives on the unit's `Exec` |
 
 Targets and datasources reference **container names** (`grafana:3000`
 `loki:3100` — `shared/prometheus/prometheus.yml:52-61`), never
@@ -31,8 +33,8 @@ scraped as `host.containers.internal:<port>` because a bridge container cannot
 reach a host-netns bind at the host's LAN IP (`prometheus.yml:89-99`), and only
 the endpoint on another machine (`192.168.0.235:7770`) is addressed by LAN IP
 (`prometheus.yml:128-131`). Prometheus 3 refuses HTML responses as metrics —
-HTML-scraped "probes" are structurally inert and need a blackbox exporter (see
-[[Stale Signal Audit]]).
+HTTP liveness rides the **blackbox exporter** (`shared/blackbox/blackbox.yml`,
+deployed 2026-10-21; see [[Stale Signal Audit]]).
 
 ## The deploy path
 
@@ -43,8 +45,9 @@ from `deploy_observability`:
 - `copy_alertmanager_config` → `/opt/alertmanager/config/alertmanager.yml` (`:232-235`)
 - `install_signal_relay` → the user unit + iptables + the health wait on `192.168.0.143:18093/health` (`:243-274`)
 - `copy_loki_config` → `/opt/loki/config/loki-config.yaml` (`:276-279`)
-- `copy_alloy_config` → `/opt/alloy/config/config.alloy` (`:281-284`)
-- `copy_grafana_provisioning` → `/opt/grafana/provisioning/` (`:286-300`)
+- `copy_alloy_config` → `/opt/alloy/config/config.alloy` — substitutes the `GRAFANA_HOST_LABEL` token with the deploy target's name and **refuses to install** if the token survives; `ALLOY_CONF_ROOT` is the bats sandbox seam
+- `copy_blackbox_config` → `/opt/blackbox/blackbox.yml` (single-file mount of the prober); `BLACKBOX_CONF_ROOT` is its sandbox seam
+- `copy_grafana_provisioning` → `/opt/grafana/provisioning/` — datasources, provider, and **every** `dashboards/*.json` (a glob, not a hand-listed subset; `GRAFANA_PROV_ROOT` is the test seam the bats case uses to run the copier into a sandbox)
 
 `scripts/deploy-plan.py` maps `shared/prometheus/**` to `prometheus` (plus
 `alertmanager` when the changed file is `alertmanager.yml`, `:251-256`)
@@ -82,11 +85,14 @@ reports 4 alerting + 2 recording. Rules reload with the prometheus service.
 the backend (`http://prometheus:9090`), not loopback.
 
 **Add a dashboard** — drop the JSON into
-`shared/grafana/provisioning/dashboards/`. Provisioned dashboards live in
-unified storage, not (only) the SQLite table — grep the API, not the DB, when
-verifying (see the verify section of [[Fleet Audit and Posture]]). *(unverified
-2026-10-18: the storage-backend claim needs an admin password to confirm
-against `grafana.db` and the API side by side)*
+`shared/grafana/provisioning/dashboards/`; the copier globs the directory, so
+the next `--service grafana` deploy provisions it. This is a repaired law, not
+a given: until 2026-10-21 the copier named `llm-fleet.json` alone, and the
+`containers` board — merged, tested, documented as "working" — had **never
+been provisioned**; the other boards existed on disk only because someone had
+hand-copied them once. A repo JSON is not a live dashboard until the copier
+carries it, and `deploy.sh` cannot see the gap because it only reports the
+copy it performed.
 
 ## Verify through the APIs, not the filesystem
 
@@ -95,90 +101,32 @@ against `grafana.db` and the API side by side)*
 curl -s http://127.0.0.1:9090/api/v1/rules | python3 -m json.tool | head
 curl -s "http://127.0.0.1:9090/api/v1/targets?state=active" | \
   python3 -c "import json,sys; [print(t['labels'].get('job'), t['health'], (t['lastError'] or '')[:60]) for t in json.load(sys.stdin)['data']['activeTargets']]"
-# grafana: datasource + dashboard provisioned
-curl -s -u admin:<pw> http://127.0.0.1:3002/api/datasources | python3 -m json.tool | grep -E '"name"|"url"'
+# grafana 13: /api/health, /api/org, /login all 404 UNAUTHENTICATED, and there
+# is NO local admin password (root .env sets no GF_SECURITY_ADMIN_*; the only
+# account is the Authentik RP login). The proven surface is a real browser
+# session: warm the Anubis challenge on the vhost root, then hit
+# /login/generic_oauth with the e2e storage state (tests/e2e/.auth/qa-e2e-admin.json)
+# — exactly the path specs/20-grafana-sso.spec.ts drives — and read the boards.
 ```
 
 The two Prometheus queries were run on 2026-10-18 and are what every number on
 this page comes from: 4 alerting + 2 recording rules, and `up` for
 `prometheus`, `node-dankclaw`, `node-dankserver`, `grafana`, `loki`
-`frank-zealot`, `frank-dragoon`, `frank-archon` with the four structural downs
-listed in [[Prometheus]]. The Grafana query needs the local admin password and
-was not run here; the filesystem side (`ls /opt/grafana/provisioning/*`) is
-proof of installation only — see the next paragraph.
+`frank-zealot`, `frank-dragoon`, `frank-archon`. The four structural downs
+that census recorded are retired (2026-10-21, [[Prometheus]]). The Grafana proof is the browser session above, run
+2026-10-21: it found a provisioned-file gap the filesystem listing hid. The
+filesystem side (`ls /opt/grafana/provisioning/*`) is proof of installation
+only — see *Add a dashboard* for why a repo JSON is not a live board.
 
 Health answers from a stale process and provisioned resources can hide in
 unified storage — a file existing on disk is not proof the running server
 loaded it. See [[Stale Signal Audit]] for the never-green-target decision
 table (refused = outage, 404/HTML = config defect).
 
-## Current delivered state (delivery chain live 2026-09-17, re-verified 2026-10-18)
-
-Prometheus → Alertmanager (`dankclaw/observability/quadlet/alertmanager.container`
-loopback `:9093`) → host-side Signal relay (`scripts/signal-relay.py`, user
-unit, `192.168.0.143:18093` — the LAN IP, the one bind a container can reach
-via `host.containers.internal`) → signal-cli, addressed by number
-`+12062476336` (`scripts/signal-relay.py:36-43`). Every hop re-checked
-2026-10-18: `ss -ltnp` shows `python3` on `192.168.0.143:18093`
-`curl -s http://192.168.0.143:18093/health` → `ok`, and
-`journalctl --user -u signal-relay` ends with successful `sent to +12062476336:`
-lines for `LlmEndpointDown`, `LlmRequestsQueueing`, `LlmSpecDecodeOff` and
-`LlmPrefixCacheCollapse` (firing and resolved). Some targets read down for
-structural reasons (HTML-scraped "probes", Gitea metrics off); the remedy is
-the blackbox exporter. See [[Prometheus]] → Targets.
-
-**A llama.cpp endpoint is invisible unless it opts in.** Until 2026-10-18,
-`up{job="frank-zealot"}` was 0 for the entire retained history of the metric —
-the oldest sample, 2026-10-11, is already 0 — while the service itself answered
-`/health` 200 the whole time. Cause: llama.cpp only exports `/metrics` when it
-is started with the flag, so the scrape was refused at the protocol layer (the
-flag and its opt-out are on [[frank-zealot]]) and `LlmEndpointDown` (critical)
-stayed firing:
-the relay re-sent it on the 4 h `repeat_interval` for days (`journalctl --user
--u signal-relay` — firing sends at 01:37, 05:42, 09:47, 13:52, 17:57 and 22:02
-on 2026-10-12). It finally resolved at `2026-10-18T01:53:21-07:00`, after the
-metrics opt-in landed that day; the target
-reports up since (`curl -s http://127.0.0.1:9090/api/v1/targets` 2026-10-18).
-How many payload generations it spanned is not readable from any host
-artifact. *(unverified 2026-10-18)*
-
-Two laws fall out of it. A target can read down beside a healthy service, so
-read `lastError` from `/api/v1/targets` before believing an outage. And rule
-state lives in Prometheus' process, not on disk — a restart re-stamps
-`activeAt` on a long-standing alert, so a "fresh" alert after a reboot is not
-evidence of a fresh fault. (Alertmanager by contrast **does** persist:
-`--storage.path=/alertmanager` on a host volume keeps silences and the
-notification log — `dankclaw/observability/quadlet/alertmanager.container:39-42`
-`/opt/alertmanager/data`.)
-
-Remaining known downs are the structural ones above (HTML-scraped probes, Gitea
-metrics, ComfyUI), and `frank-dragoon` exports `/metrics` natively
-([[frank-dragoon]]), which is why
-`frank-dragoon` has always been green. The active-alert count is a
-moment-in-time value and should never be documented as a constant:
-`/api/v1/alerts` was empty at the 2026-10-18 morning census, held one firing
-`LlmPrefixCacheCollapse` (`activeAt` 11:10:48Z, value 2.6e-4) when sampled
-minutes later, and was empty again by 11:56Z; the relay logged the matching
-firing and resolved sends at `04:26:21-07:00` and `04:41:21-07:00`.
-
-**What that zealot scrape measured on 2026-10-18.** As of ~04:30 PDT the
-production `frank-zealot` root unit was stopped **on purpose** for a benchmark
-window on the 3090 — the repo Benchmarking Policy says stop the production
-service before benching — and `:7771` answered from a manually booted candidate
-arm (`systemctl is-active frank-zealot` → `failed` after
-`stop-sigterm timed out. Killing`, which is llama.cpp ignoring SIGTERM past
-`TimeoutStopSec`, the known shutdown shape for these engines). So
-`up{job="frank-zealot"} == 1` that day measured the trial, not the production
-shape in `scripts/frank-zealot.env` — that shape is [[frank-zealot]]'s to state,
-and it is the repo contract.
-The unit stayed `enabled`; `./scripts/deploy.sh dankclaw --service frank-zealot`
-reinstalls and starts it, restoring boot persistence and self-healing. Never
-quote a candidate arm's flags or VRAM as fleet state — the process and its shape
-changed between two censuses ten minutes apart.
-
 ## Related
 
 - [[Observability]] — stack topology and the alert delivery chain
-- [[Stale Signal Audit]] — classifying down targets before "fixing" them
+- [[Stale Signal Audit]] — classifying down targets before "fixing" them, and the
+  down-beside-healthy / alert-state-persistence / candidate-arm rules
 - [[Quadlet Lifecycle]] — the units that host these services
 - [[Deploy State Triage]] — how the planner selects these services

serverAppConfigs-docs/Infrastructure/Observability.md

@@ -67,10 +67,26 @@ Alertmanager has no vhost at all. node-exporter is the exception that must live
 in the host network namespace, and it binds exactly two addresses rather than
 the wildcard — see [[Node-Exporter]].
 
-The relay's bind is the one address a bridge container can reach on the host:
-`192.168.0.143:18093`, with iptables INPUT admitting loopback, the host itself
-and the podman bridge subnet `10.89.0.0/16` before the DROP
-(`scripts/deploy-remote.sh:255-262`).
+**Same-host reachability from a bridge container is measured, not inferred**
+(2026-10-21, from inside the Prometheus container): `host.containers.internal`
+(the podman gateway) reaches host services whether they bind host-net or are
+published ports — node `:9100`, cadvisor `:8080`, gpu `:9401` all answer there
+— while the host's own LAN IP is refused (`192.168.0.143:8080`, `:9401`), so
+every same-host Prometheus target uses the gateway and only cross-host targets
+(dankserver) use plain LAN. Whether LAN works at all is network-specific: the
+relay at `192.168.0.143:18093` is reached from alertmanager's network only
+while `install_signal_relay`'s INPUT rules are in force: loopback, the host
+itself and the podman bridge subnet `10.89.0.0/16` accepted **above** a
+trailing blanket DROP. The law the poisoned host taught (2026-10-21):
+`iptables -I` prepends, so inserting the DROP last lands it first — the old
+install order dropped every packet (159 on the DROP, 0 on the ACCEPTs), the
+deployer's health gate failed beside a LISTENing socket, and the drain
+aborted twice before `nft list ruleset` named the cause. The install now
+rebuilds the tail (delete the DROP wherever it sits, ACCEPTs at the head,
+DROP appended); verify any change to it with the sandboxed bats case, not
+with `-C` greps — `-C` answers existence, never order. Never point a scrape
+target at a host LAN IP from its own host on the strength of one working
+example; dial the gateway and probe.
 
 ## Deployment
 
@@ -87,13 +103,14 @@ health endpoint (`/prometheus`: `/-/healthy`, `/alertmanager`: `/-/healthy`
 [[Observability Provisioning]] for what is provisioned as code and how to
 verify it through the APIs.
 
-## Current state (2026-10-20)
+## Current state (2026-10-21)
 
 * **Metrics: working.** `curl -s http://127.0.0.1:9090/api/v1/targets` reports
-  eight up targets — the three LLM endpoints (frank-archon on `vllm:*`,
-  frank-dragoon and frank-zealot on `llamacpp:*`), grafana, loki, prometheus
-  and both node exporters. The four downs are structural (the `comfyui` and
-  `services` HTML probes, and Gitea's disabled metrics endpoint), not outages.
+  every job up — the three LLM endpoints (frank-archon on `vllm:*`,
+  frank-dragoon and frank-zealot on `llamacpp:*`), grafana, loki, prometheus,
+  both node exporters, cadvisor, gitea (metrics enabled in its quadlet) and
+  the blackbox-probed `services` (SearXNG, OpenWebUI, ComfyUI). The structural
+  downs are retired — see [[Prometheus]] → Targets.
   Two rule groups in `shared/prometheus/rules/llm-fleet.yml`: `llm-fleet`
   (2 recording + 4 alert, `llamacpp:*`) and `vllm-fleet` (1 recording + 4
   alert, `vllm:*` — archon queueing, prefix collapse, preemption, p95
@@ -101,12 +118,15 @@ verify it through the APIs.
 * **Delivery: working.** Alertmanager's single `signal` receiver webhooks the
   relay, and `journalctl --user -u signal-relay` records successful
   `sent to +12062476336:` lines for every rule in the set.
-* **Dashboards: working.** Provisioned datasources — Prometheus (uid
-  `prometheus`) and Loki (uid `loki`) — and the `llm-fleet`, `host-system`,
-  `containers`, `observability` and `logs-overview` dashboards, all
-  installed from `shared/grafana/provisioning/`. The Loki datasource is how
-  you read logs: Loki has no web UI of its own, so Grafana Explore is the
-  human surface (the guarded `loki.franksdank.cloud` vhost serves only the API).
+* **Dashboards: working (after the 2026-10-21 copier fix).** Provisioned
+  datasources — Prometheus (uid `prometheus`) and Loki (uid `loki`) — and the
+  `llm-fleet`, `host-system`, `containers`, `observability` and
+  `logs-overview` boards. Until 2026-10-21 the copier carried only
+  `llm-fleet.json` by name and the `containers` board had never been
+  provisioned; the deployer now globs the directory. The Loki datasource is
+  how you read logs: Loki has no web UI of its own, so Grafana Explore is the
+  human surface (the guarded `loki.franksdank.cloud` vhost serves only the
+  API). Verify method and the copier law: [[Observability Provisioning]].
 * **Per-container metrics: the `containers` dashboard (added 2026-10-20; live
   after `deploy.sh <host> --service cadvisor`).** cadvisor is the only way to see
   per-container CPU/memory here, and on this rootless-Podman host it works on one
@@ -131,26 +151,26 @@ verify it through the APIs.
   `power1_average`, so they carry no `gpu_power_watts` line at all (only NVIDIA and
   the Dankclaw 8060S iGPU report power); and it binds loopback + the host LAN, like
   node-exporter, never the wildcard. The dashboard graphs util/VRAM/VRAM-%/temp/power.
-* **Logs: fixed in code, awaiting deploy.** The pipeline shipped nothing because
-  a rootless Alloy cannot read the `0640` journal files, and the container-log
-  globs point at a root-storage layout that does not exist here. The fix is built
-  and tested in-repo (2026-10-20): Alloy is now a **root system service on
-  `Network=host`**, reading `system.journal` + `user-*.journal` and pushing to
-  Loki's `127.0.0.1:3100` loopback publish — a throwaway root probe read 31,825
-  lines and pushed 26,966 with zero drops. The **live** instance is still the old
-  rootless unit until `deploy.sh dankclaw --service alloy` runs. Detail in
-  [[Alloy]] → Why root; Loki's store state in [[Loki]] → Contents.
-* **Not configured:** no blackbox exporter (the structural downs have no real
-  uptime signal) and no Alertmanager vhost. The log pipeline's fix is in code but
-  not yet deployed (see *Logs* above).
+* **Logs: flowing; container labels land with the 2026-10-21 deploy.** The
+  pipeline once shipped nothing (rootless Alloy cannot read the `0640` journal;
+  the container-log globs pointed at a root-storage layout that does not exist
+  here) — fixed 2026-10-20 by making Alloy a **root system service on
+  `Network=host`**. Journal lines now reach Loki (measured ~6.5k lines/5min),
+  but the stream carried only `job`/`service_name`, so every per-container
+  view was structurally empty: Loki metrics queries group only by INDEXED
+  stream labels. Alloy now promotes podman's `CONTAINER_NAME` to a `container`
+  label (plus a `host` label) at ingestion — `loki.process` in
+  `shared/alloy/config.alloy`; detail in [[Alloy]] → Why root, and the
+  purpose-test law in [[Observability Provisioning]].
 
 ## Citations
 
-- `dankclaw/observability/quadlet/{prometheus,alertmanager,grafana,loki,alloy,node-exporter,cadvisor}.container`
+- `dankclaw/observability/quadlet/{prometheus,alertmanager,grafana,loki,alloy,node-exporter,cadvisor,blackbox-exporter}.container`
 - `scripts/deploy-remote.sh:219-300` (copiers + relay), `:521-536` (health waits), `:477-486` (grafana env seed)
 - `scripts/signal-relay.py:26-49`, `scripts/signal-relay.service`
 - `scripts/gpu-exporter.py`, `scripts/gpu-exporter.service` (per-GPU host exporter)
 - `shared/prometheus/prometheus.yml`, `shared/prometheus/alertmanager.yml`, `shared/prometheus/rules/llm-fleet.yml`
 - `shared/grafana/provisioning/`, `shared/loki/loki-config.yaml`, `shared/alloy/config.alloy`
+- `shared/blackbox/blackbox.yml` (prober modules; the listen bind is on the unit's `Exec`)
 - `tests/observability_quadlet.bats` (boot persistence, no wildcard publish, probe addresses, delivery contract)
 - Host evidence 2026-10-18: `ss -ltnp`, `podman ps`, `systemctl --user is-enabled|is-active`, `curl` on `127.0.0.1:9090/api/v1/{targets,rules,alerts}`, `127.0.0.1:3100/loki/api/v1/labels`, `127.0.0.1:12345/metrics`, `127.0.0.1:3002/api/health`, `192.168.0.143:18093/health`, `journalctl --user -u signal-relay`

serverAppConfigs-docs/Infrastructure/Prometheus.md

@@ -42,29 +42,29 @@ Authentik; post-auth the Caddy vhost routes straight to `prometheus:9090`) —
 
 The `frank-medium` job is **retired**: no `frank-medium` job exists in the config or in `/api/v1/targets` (2026-10-18), and the endpoint itself left the fleet — its history is on [[frank-archon]], which holds the port today.
 
-**Three jobs read down today, and they are not outages** (re-audited against `curl -s http://127.0.0.1:9090/api/v1/targets`, 2026-10-18; first audited 2026-09-16):
+**No job reads down structurally any more** (the three permanent-reds were
+retired 2026-10-21; census re-run after each deploy):
 
-| Job | Target | `health` | `lastError`, verbatim |
-|-----|--------|----------|------------------------|
-| `gitea` | `http://192.168.0.143:3000/metrics` | down | `server returned HTTP status 404 Not Found` |
-| `comfyui` | `http://comfyui:8188/` | down | `received unsupported Content-Type "text/html" and no fallback_scrape_protocol specified for target` |
-| `services` | `http://192.168.0.143:8081/` | down | `received unsupported Content-Type "text/html; charset=utf-8" and no fallback_scrape_protocol specified for target` |
-| `services` | `http://192.168.0.143:3005/` | down | same `unsupported Content-Type "text/html; charset=utf-8"` error |
+| Job | Target | how |
+|---|---|---|
+| `gitea` | `192.168.0.143:3000/metrics` | metrics enabled in the quadlet (`GITEA__metrics__ENABLED=true`, `ENABLED_HOST=0.0.0.0`) — the page once said "stays off until `[metrics] ENABLED = true` in its own config"; the env form is what the deployer ships |
+| `services` | `http://192.168.0.143:8081` (SearXNG), `:3005` (OpenWebUI), `http://127.0.0.1:8188` (ComfyUI) | **blackbox exporter** probe, module `http_2xx` — the comfyui metrics job is retired, one owner per service. The prober runs in the host netns, so comfyui is probed over loopback (its published port); bridge container names do not resolve there |
 
-Gitea's exporter stays off until `[metrics] ENABLED = true` in its own config (no metrics env is set in the container, and `curl -s http://127.0.0.1:3000/metrics` → 404, 2026-10-18); ComfyUI and the two `services` probes are HTML pages, which Prometheus 3 refuses as metrics. Everything else is green: `prometheus`, `node-dankclaw`, `node-dankserver`, `grafana`, `loki`, `frank-zealot`, `frank-dragoon`, `frank-archon`.
+Scraping a web root is not a health check: an HTML page fails the scrape
+protocol (`unsupported Content-Type`), which is what kept those targets red
+beside healthy services. The classification rule and the honesty law live in
+[[Stale Signal Audit]]. The prober is `blackbox-exporter.container` (host
+netns, config `shared/blackbox/blackbox.yml`); it binds 9115 on loopback and
+the LAN, and Prometheus scrapes it at `host.containers.internal` — traffic to
+that gateway lands on the host's loopback listener. The gateway address is
+not itself bindable from the host netns.
+Plain targets stay `host:port`, never URLs — the probe job's URL targets are
+the documented exception, relabeled onto the prober.
 
 **`frank-zealot` reads up since 2026-10-18, and did not before.** llama.cpp only answers `/metrics` when started with `--metrics`, so `up{job="frank-zealot"}` was 0 across the whole retained history (the oldest sample, 2026-10-11, is already 0) while the service answered `/health` 200 the whole time; the flag and its opt-out are on [[frank-zealot]]. Read `lastError` before believing an outage: a target can read down beside a healthy service.
 
 Alert state is a live property, not a page fact: `curl -s http://127.0.0.1:9090/api/v1/alerts` returned no alerts at the 2026-10-18 morning census, one firing `LlmPrefixCacheCollapse` (info, `frank-zealot`, `activeAt` 11:10:48Z) minutes later, and none again by 11:56Z.
 
-The remedy for a probe-shaped question is the **blackbox exporter**; the
-classification rule and the honesty law for it live in [[Stale Signal Audit]].
-Until the exporter is deployed, these three stay red for a structural reason:
-deleting them would
-hide a real service, and fixing them by publishing databases or enabling metrics
-would trade a wrong signal for a real exposure. Targets are `host:port`, never
-URLs.
-
 ## Volumes
 
 | Container path | Host path |

serverAppConfigs-docs/LLM/RDNA4 MoE Expert Offload VRAM Budget.md

@@ -70,12 +70,12 @@ back toward 52. Check `VLLM_CUDAGRAPH_*` actually present in
 
 `--kv-cache-memory-bytes` does NOT appear in the arena arithmetic — the plan
 reserves the registry price, not the actual pool. Raising the pool consumes
-real free VRAM without growing the arena; the registry over-prices the pool
-(9190 B/token planned vs 7,891.8 measured at 200704 fp8 — 7,701 at 262144;
-**never reuse a B/token ratio across geometries**, and subtract ~3.87 MB
-block-alignment loss when converting a token target to bytes). The ask lands
-block-aligned: 3,166,500,000 B → 401,408 tokens; verify with
-`GPU KV cache size: N tokens, Maximum concurrency … X.Xx`.
+real free VRAM without growing the arena; the registry OVER-PRICES the pool
+(era G measured ~14 % under the planned B/token — the gap is the lesson, not
+the number; **never reuse a B/token ratio across geometries**), and there is a
+~4 MB block-alignment loss to subtract when converting a token target to
+bytes. The ask lands block-aligned: verify against the engine's own
+`GPU KV cache size: N tokens, Maximum concurrency … X.Xx` line at boot.
 
 **Driver-reported free VRAM after hours of traffic is NOT allocatable
 arena.** It is the estimator gap: the registry over-prices KV and charges
@@ -117,11 +117,13 @@ done, and read post-boot free VRAM per rank (`rocm-smi --showmeminfo vram`)
 **KV accounting — pin the pool BYTES, floor-sized:** set
 `--kv-cache-memory-bytes` PER RANK to the contract floor
 (`seqs × MAXLEN × measured B/token`), not the native window — extra pool bytes
-come out of the arena/headroom and buy nothing. Measured B/token on this
-hybrid: **7,701 fp8 / 13,839 bf16** (decomposed: attention 6,138/12,276 +
-dtype-independent mamba/side 1,563). Measured rates are shape-dependent —
-7,891.8 at the 200704×3 shape. Capacity does NOT scale with `SEQS`: the
-planner's KV reserve is for ONE max-model-len request.
+come out of the arena/headroom and buy nothing. The B/token rate is a property
+of the GEOMETRY: era G (3 × 200,704 fp8, vLLM fork) measured **7,891.8**;
+the radiance-era hybrid measured ~7,700 fp8 / ~13,800 bf16 at its own shape.
+Read your live rate from the boot ledger every time; the decomposition (a
+dtype-independent mamba/side stream rides on top of the attention KV) is why
+the ratio moves. Capacity does NOT scale with `SEQS`: the planner's KV reserve
+is for ONE max-model-len request.
 
 **Parallelism ceilings you cannot flag past:**
 
@@ -179,18 +181,6 @@ Cutover order: ladder → winner passes and serves → repo env + seam + harness
 (`git stash -u`) if the tree is dirty → deploy (one restart) → verify health
 + engine flags in the log.
 
-**Measured ladder on this box (200704×3, fp8 KV, precise memory):**
-
-| shape | util | graphs | arena | result |
-|---|---|---|---|---|
-| E | 0.98 | 0.55 | 17.22 GiB / 7,522 slots | boot ok |
-| F | 0.982 | 0.32 | 17.52 GiB / 7,653 slots | boot ok |
-| 1.0 | 1.00 | 0.07 | — | UNBOOTABLE (worker-init ValueError, both ranks) |
-| G (deployed) | 0.982 | 0.07 | 17.77 GiB / 7,762 slots | KV 401,408 tokens / 2.00x; 0.89 GiB real free/card held through a live 4,096-token generation at ~38–39 tok/s |
-
-bf16 ceilings at util 0.98: 3 seqs → 241,664 passes / 245,760 and 262,144 OOM
-at graph capture; 6 seqs fp8 → 10.70 GiB ask with 9.18 GiB free (fails) —
-~207k per context max.
 
 ## 7. Arena audit procedure (when asked for "more experts")
 

serverAppConfigs-docs/LLM/Radiance Architecture Plugins.md

@@ -0,0 +1,287 @@
+---
+type: "howto"
+title: "Radiance Architecture Plugins — declaring a model"
+description: "Writing a radiance architecture plugin: declaring weights, buffers, KV groups and ops, then issuing the step."
+tags:
+  - llm
+  - radiance
+  - plugins
+  - howto
+timestamp: 2026-10-30
+---
+
+# Radiance architecture plugins
+
+An architecture plugin declares a model in two phases: `rad_arch_declare` names every weight,
+buffer, KV group and op **once**, against the model's metadata; `rad_arch_step` issues the ops for
+one step. The engine picks the plugin by the exact `(rad_arch_id, rad_arch_quant)` pair — no
+fallback — so one family at two weight formats is two `.so` files, and branching on a quant
+descriptor inside one plugin is the anti-pattern the split exists to prevent.
+
+The in-tree plugins are the templates: `arch/llama_fp8/llama_fp8.cpp` (the smallest complete
+decoder), `arch/gemma3`, `arch/qwen35_bf16`, and the header-only blocks in `arch/common/`
+(`rad_block_attn*.h`, `rad_block_mlp*.h`, `rad_block_moe_fp8.h`, `rad_block_gdn*.h`,
+`rad_block_dflash2.h`, `rad_block_mtp*.h`, `rad_block_vit.h`, `rad_block_ple.h`,
+`rad_block_glm_*.h`, `rad_qsa.h`, …). Include those by `<arch/rad_arch.h>` — they are installed
+beside the ABI and the in-tree plugins include them by that spelling.
+
+## Exports
+
+```c
+const char* rad_arch_id(void);              /* "llama", "qwen35", ... */
+const char* rad_arch_quant(void);           /* "" unquantised, else the descriptor served */
+int         rad_arch_serves_imports(void);  /* optional: 0 declines an imported quantisation */
+int         rad_arch_probe(const RadModelMeta* meta, RadArchProbe* out);   /* optional */
+int         rad_arch_declare(RadBuilder* b, const RadModelMeta* meta, const RadBuildCtx* ctx);
+void        rad_arch_step(struct RadCtx* c, const struct RadBatch* batch);
+const RadChatFormat*  rad_arch_chat_format(const RadModelMeta* meta);      /* optional */
+const RadAudioFormat* rad_arch_audio_format(const RadModelMeta* meta);     /* optional */
+```
+
+Everything but declare/step is optional. `RAD_ARCH_PLUGIN(NS, ID, QUANT, VERSION, DESC)` in
+`arch/common/rad_arch.h` emits the whole export block, including `rad_plugin_info` (kind
+`RAD_PLUGIN_ARCH`). The plugin's **name must be unique**; the architecture id may repeat.
+
+`rad_arch_serves_imports` exists because a checkpoint whose quantisation was imported weight by
+weight (Quark, ParoQuant, GPTQ, NVFP4) names its producing tool, which no plugin claims: the family's
+one quantised plugin serves it, and two are refused by name. A second quantised plugin that reads
+only its own containers returns 0.
+
+## Declare: the metadata
+
+`RadModelMeta` is what the loader read from the container's header before any plugin ran: `arch_id`,
+`name`, `quant`, `n_layers`, `n_embd`, `n_head`, `n_head_kv`, **`head_dim`**, `n_ff`, `n_vocab`,
+`n_ctx_train`, expert counts, `rms_eps`, `rope_theta`, and a free-form `kv_key`/`kv_val` table for
+everything else (`rad_meta_geti/getf/gets`). Read dimensions from here — one plugin covers every size
+in a family.
+
+**`head_dim` is read, never derived.** On the Qwen hybrids it is 256 while `hidden / heads` is 213;
+a derived value silently breaks projection widths, KV page size and the attention kernel's band.
+
+`RadBuildCtx` is the deployment the plugin must declare against:
+
+| field | meaning |
+|---|---|
+| `rank`, `world_size` | declare runs once per rank; **dimensions are already divided** — the core inserts no collectives |
+| `max_tok`, `max_seqs`, `max_ctx`, `max_spec` | the step bounds: buffer sizing, kernel bands, the scheduler's window |
+| `max_mixed_tok` | the most a multi-sequence step may carry; a wider step is one sequence (prefill span) |
+| `tp_wire_lossy`, `tp_wire_min_bytes` | invert into `all_reduce`'s `exact`; below the floor a message must go exact |
+| `kv_dtype`, `kv_quant` | the attention cache width and, under fp8, its narrow layout (`RAD_KVQ_V4`, `RAD_KVQ_K6V4`) |
+| `attn_prefill_fp8` | `RAD_ATTN_FP8_*` bits to pass as `attn_paged`'s `prefill_fp8` |
+| `kv_resident_blocks`, `kv_stream_ok` | KV streaming: the ring depth, and permission to declare the streamed-only ops |
+| `state_dtype` | `RAD_F16` when `--state-cache-dtype f16` is served by this plugin |
+| `is_draft`, `scope` | a drafter/encoder declare; `scope` prefixes declared names (`draft`, `vision`, `audio`) |
+| `shape_probe` | a **sizing** declare at a smaller `max_tok`; must not touch state the real declare wrote |
+| `max_enc_patches`, `max_audio_frames` | the encoder pass sizes |
+| `max_out_rows` | logits rows the KL mode needs (0 = the plugin's own rule) |
+| `cvec` | the loaded control vector, or null |
+| `tp_attn_rows` | row-split prefill attention requested |
+| `draft_coupling`, `draft_tree` | declare the coupled/tree forms |
+| `weight_prefetch_off` | declare no weight prefetch beside collectives |
+
+## Declare: weights
+
+```c
+static const RadWeightDecl d = {
+    .dtype = RAD_BF16, .rank = 2, .shape = {n_embd, n_ff},
+    .access = RAD_ACCESS_PER_TOKEN, .shard = RAD_SHARD_ROW,
+    .group = grp_layer(l), .optional = 0,
+    .n_row_parts = 0, .source = nullptr, .planes = nullptr,
+};
+rad_weight w = rad_decl_weight(b, "blk.0.mlp.gate.weight", &d);
+```
+
+- **`access`** tells the placement planner how the weight is read: `PER_TOKEN` (dense),
+  `PER_REQUEST` (an encoder tower, read once per request), `CONDITIONAL` (routed experts),
+  `RARE`, `VOCAB` (lm_head/embedding: read every token but huge, and one is a gather).
+- **`shard`**: `NONE` (replicated), `ROW` (split dim 0), `COL` (split dim 1, needs a following
+  all-reduce). The plugin says; the core does not infer. Uneven splits use
+  `rad_weight_shard_span`/`_parts` (a width that does not divide into whole 128-column blocks; the
+  delta net's `[q|k|v|z]` of unequal part widths).
+- **`group`** is the unit of movement (`layer`, `expert`, `slot`); an expert's gate/up/down share one,
+  because the unit of transfer is an expert, not a tensor.
+- **`row_parts`** describes what a `SHARD_ROW` weight stacks along dim 0 (`[gate|up]`,
+  `[q|k|v|z]`). **A tensor-parallel rank owns a slice of each stacked projection**, never a
+  contiguous slice of the stack; the parts cannot be inferred from a fused container tensor, and a
+  contiguous slice produces fluent wrong text.
+- **`source`/`planes`** view an existing stored weight: a block-fp8 linear's `codes,scale` are one
+  logical weight taken by a GEMM as two operands. Null `source` is the weight of its own name; null
+  `planes` takes every plane.
+- A dtype other than the plane's is legal only as an exact widening (a bf16 norm read as f32).
+
+The **name map** (`rad_decl_name_map`) is the plugin's map from checkpoint tensor names to declared
+weight names, and it is the only place convert-time fusions live: `COPY`, `CONCAT`, `WNORM` (a
+weight-normalised pair folded to the plain weight), `ROWS`, `TRANSPOSE`, `EMA` (an EnCodec codebook),
+`TAP`, `COLS`. `rad-convert` reads this map from the plugin, so it needs no knowledge of the model.
+`RAD_MAP_COPY` with several sources means "first present wins" (a tied lm_head); `CONCAT` requires
+all. Ask for a weight's encoding with `rad_weight_encoding` **after** its map entry when the graph
+depends on the format (fp8 GEMM vs plain, rotated expert kernels).
+
+## Declare: buffers and KV groups
+
+`rad_decl_buffer` names every activation: `RAD_BUF_TRANSIENT` (liveness-analysed, may share bytes),
+`RAD_BUF_PERSIST` (across steps: a calibration accumulator, a routing histogram), or
+`RAD_BUF_DERIVED` (computed once per step by the core from the batch — `derive` names the
+derivation, e.g. `conv_state_index`). Buffers live in one arena sized once, not an allocator per
+step.
+
+`rad_decl_kv_group` declares each state kind: `RAD_KV_FULL` (paged per-token, growing),
+`RAD_KV_WINDOW` (bounded by a window), `RAD_KV_LINEAR` (per-sequence fixed state, GDN),
+`RAD_KV_CONV` (a rolling window of `width-1+num_spec`), `RAD_KV_RING` (window floor set at runtime).
+The **block size is not declared** — it comes from the resolved attention kernel's constraints; read
+it back with `rad_kv_block_size`. `rad_bind_layer_kv` binds each layer so the block manager knows how
+many state instances a sequence owns; `rad_kv_group_zero` (a rank with no heads),
+`rad_kv_group_replicated` (every rank holds the same bytes), and `rad_kv_group_dense_read` (read over
+the whole history every step, so it cannot stream) are the three refinements.
+
+## Declare: ops
+
+```c
+rad_op h = RAD_OP(b, "gemm_w4a8_nt",
+                  RAD_PARAMS(RAD_RANGE("M", 1, max_tok), RAD_INT("N", 8192), RAD_INT("K", 5120)),
+                  RAD_WEIGHTS(w));
+rad_op_reads (b, h, RAD_BUFS(b_x), n);
+rad_op_writes(b, h, RAD_BUFS(b_y), n);
+```
+
+- Parameter names and operand order come from `docs/OPS.md`; a new name is allowed only when the
+  vocabulary has none, and the first plugin to declare it fixes its schema.
+- `RAD_RANGE` on exactly one parameter per op gives the selector its bucket table. The core carries
+  every parameter opaquely except `RAD_PROLE_SEQ_CHUNK` (cut steps on it) and `RAD_PROLE_CAPACITY`.
+- `rad_op_reads`/`rad_op_writes` are the buffer planner's liveness. An undeclared use is assumed live
+  for the whole program: sound but wasteful.
+- **Fusion is selection.** Ask for the fused op; if `rad_op_resolved` is false, emit the chain. A
+  fusion's `replaces` hook is a checkable equivalence, not a rewrite.
+- `rad_op_declared(b, "op")` asks whether any library declares an optional op; use it before
+  emitting a fused form only an optional library defines.
+- Buffers touched from the **second lane** go through `rad_buf_concurrent`, and an architecture whose
+  lane-1 work lies in one stretch declares `rad_lane1_window_begin/end` around it. The runtime
+  refuses a lane-1 op outside the window; the window lets buffers used only outside it share bytes.
+
+Step output: `rad_declare_logits` names the buffer the sampler reads (its dim 0 bounds `n_out`; a
+[n_tok, vocab] logits buffer at a large vocabulary is gigabytes — declare the gather +
+`logits_gemm` shape instead). An embedding program declares `rad_declare_pooled` instead (one row a
+sequence, no sampler, no generation). `rad_declare_scores` opts into host-side scored steps.
+
+## Declare: drafters, encoders, generators
+
+- `rad_declare_drafter(RadDrafterDecl)` — depth, block, proposal shape. `rad_drafter_shared_kv` (a
+  drafter with no cache of its own), `rad_drafter_tree` (a tree over the proposals),
+  `rad_drafter_short_block` (a second, shorter pass) are refinements. Declare the drafter **last**.
+- `rad_declare_encoder(RadEncoderDecl)` — a vision/audio tower, media preprocessed by the core.
+- `rad_declare_generator` / `rad_declare_audio_generator` / `rad_declare_video_generator` — image,
+  audio and video generation programs; the engine sizes the step from the probe's `gen_*` fields
+  before declare.
+- Optional format exports let a model state what it knows and the server learn nothing new:
+  `rad_arch_chat_format` (a data description of reasoning/tool-call markup, replacing template
+  analysis), `rad_arch_audio_format` (how a speech model hears: clip tokens, prompt, language,
+  transcript start), `rad_arch_text_prepare` (rewrite request text into training form),
+  `rad_arch_video_positions`/`rad_arch_video_pack` (video pass geometry and packing),
+  `rad_arch_encode_geom` (encoder pass geometry). Each struct grows only at its end behind
+  `struct_size`, so adding a field does not move the ABI.
+
+**Media preprocessing is the core's**, not the plugin's: resizing, patching, placeholder expansion
+and positions in six processor families (Qwen-VL, Gemma 4, LFM2-VL, Unlimited-OCR, GLM-5.3, Pixtral
+for Nemotron), chosen by `vision_config.model_type` plus owning-model checks; audio through Gemma 4's
+extractor or Qwen3-ASR's Whisper one, chosen by `audio_config.model_type`. A vision model with another
+pipeline needs a core change.
+
+## Step: issuing the graph
+
+`RadBatch` is built once per step and shared by every block: `phase` (`PREFILL`/`DECODE`/`MIXED`),
+`n_tok`, `n_seq`, `token_ids`, `positions`, `cu_seqlens`, `seq_ids`, `lora_slot`, `n_out`/`out_ids`,
+`q_lens`/`ctx_lens`/`max_q_len`/`max_ctx_len`, the KV-group array, the speculative fields
+(`n_spec`, `draft_pass`, …).
+
+```c
+static void step(RadCtx* c, const RadBatch* batch) {
+    if (batch->draft_pass > 0) { m.ds.step(c, batch); return; }      /* query pass */
+    if (batch->draft_pass < 0) { m.ds.context(c, batch); return; }   /* context pass */
+    RAD_ISSUE(c, m.op_embed, praw(batch->token_ids, RAD_I32, batch->n_tok),
+              RAD_W(m.w_tok), brows(m.b_x, batch->n_tok));
+    for (auto& layer : m.layers) { layer.attn.step(c, batch); layer.mlp.step(c, batch); }
+    /* ...heads; the lm_head only over batch->n_out rows... */
+}
+```
+
+- `rad_issue(c, op, operands…, n)` / `RAD_ISSUE` (n = the batch's ranged extent) /
+  `RAD_ISSUE_N` (an explicit row count). `rad_step_fail(c, what)` sets the status helpers return.
+- **A mixed step is the trap.** Continuous batching produces steps with a prefill chunk beside decode
+  rows. A block that issues one path for the whole step sends its decode rows down the chunked path,
+  which commits recurrent state directly and takes no `num_accepted`; a rejected draft cannot be
+  rolled back, and the model stays fluent while drifting. Issue the recurrent path over
+  `RadBatch::n_seq_decode` sequences and the chunked path over the rest (the core orders steps
+  decode-first, so each half is contiguous; `cu_seqlens` holds absolute offsets, so pass
+  `cu + n_seq_decode` with the token operands at their base).
+- A step issued on lane 1 outside the declared window fails by name. An op the core appends behind a
+  declared one (runtime LoRA) counts at that op's position.
+- **Zero rows is not a call**: issuing an op over zero rows (a zero ranged extent, or all-empty
+  sequences) returns `RAD_E_SHAPE` and writes nothing. The caller skips the issue, as the
+  architectures' decode and prefill halves do.
+- `n_out` is 0 for a chunk that finishes no sequence — do not run the lm_head.
+
+## Probe: what the plugin measured
+
+`rad_arch_probe(meta, out)` runs before declare. Its fields are the engine's permissions:
+
+| field | effect |
+|---|---|
+| `draft_depth`, `draft_depth_fixed` | the container's spec depth; a block drafter's trained block is the only depth it serves |
+| `shape_probe_ok` | the plugin tolerates a sizing declare, so the activation arena can be lent |
+| `enc_merge`, `audio_frames` | encoder pass granularity; longest audio clip |
+| `gen_rows`, `gen_patches`, `gen_audio`, `gen_ctx`, `gen_video` | generator sizing before declare |
+| `kv_quant_ok` | bits of the narrow KV layouts this model's kernels serve |
+| `chain_pays`, `tree_pays` | the plugin measured chained launches / draft trees faster |
+| `kv_stream_ok` | the architecture can stream a KV group (sparse, block-addressed, with `kv_resolve`) |
+| `control_vector_ok` | the plugin applies `RadBuildCtx::cvec` |
+| `tp_attn_rows_ok` | the plugin serves row-split prefill attention |
+| `state_f16_ok` | the plugin declares recurrent state at f16 and its kernels read it |
+
+A flag whose probe bit is 0 is **refused at startup with the plugin named** (`--kv-cache-dtype
+fp8_v4`, `--control-vector`, `--tp-attn-split rows`, `--state-cache-dtype f16`, `--kv-resident`).
+That is the design: a deployment switch the plugin cannot honour is an error, never a silent
+substitution.
+
+## Worked pattern: the smallest complete plugin
+
+`arch/llama_fp8/llama_fp8.cpp` builds twice from one source — `llama_fp8` (quant `fp8_e4m3`) and
+`llama_dense_fp8` (quant `""`), differing only in the descriptor and the graph-dump text. It shows
+the whole shape:
+
+1. `struct Layer { AttnFP8 attn; MlpFP8 mlp; }`, `struct Model` holding `Names`, `Geom`, buffer and
+   op handles.
+2. `declare()`: name map entries first, then weights (with `rad_weight_encoding` where the head's
+   format decides the kernel), buffers, KV group, the embedding and its all-reduce, then each layer's
+   `declare()` from the shared blocks, the final norm, gather and lm_head, **the drafter last**, and
+   `rad_note` lines explaining the shape.
+3. `step()`: draft passes first (sign selects query/context), then embedding (+ exact all-reduce
+   when needed), each layer's `step()`, the final norm, and the head only over `n_out`.
+4. `probe()`: reads `draft.block_size` from the metadata and refuses any other depth for a block
+   drafter.
+5. `RAD_ARCH_PLUGIN(...)` at the bottom.
+
+Add a new family by copying a neighbour, replacing the name map and geometry, and composing the
+blocks you need; add a new *format* by cloning the plugin with another `(id, quant)` pair and the
+ops that format needs.
+
+## Verifying a new plugin
+
+```sh
+radiance --model m.rad --debug-graph          # every op, its resolved kernel, and why
+radiance --model m.rad --debug-placement      # the plan
+rad-convert $SRC -o m.rad --plan-only -v      # a convert that plans will boot
+rad-kbench -m m.rad --kernels mylib,libr4d,libref --report k.md --bench
+rad-tune -m m.rad --tp 2                      # only at the shapes you serve
+```
+
+`tests/arch_test.cpp` drives a plugin's builder the way the core does, including lane-window
+liveness, so the pinned graphs and steps are the ones the core plans. A plugin that declares a weight
+form no kernel `layout` accepts is a declare-time error naming the weight, encoding and kernel — the
+same check `rad-convert` runs before writing.
+
+## Related
+
+- [[Radiance Plugins]] — kernel libraries, encodings, the build path, ABI law.
+- [[Radiance Conversion]] — how the name map drives a convert, and `--set`/descriptor selection.
+- [[Radiance Model Families]] — the existing `(id, quant)` pairs and their recipes.
+- [[Radiance Tools]] — `rad-kbench`, `rad-tune`, `rad-info --plugins`.

serverAppConfigs-docs/LLM/Radiance Conversion.md

@@ -0,0 +1,354 @@
+---
+type: "howto"
+title: "Radiance Conversion — checkpoints into .rad containers"
+description: "Turning a checkpoint into a .rad container: rad-convert, the recipe language, quantisers, reuse and export."
+tags:
+  - llm
+  - radiance
+  - quantization
+  - howto
+timestamp: 2026-10-30
+---
+
+# Radiance conversion
+
+`rad-convert` turns a Hugging Face checkpoint into a `.rad` container, choosing each weight's
+encoding from a **recipe**. `rad-export` wraps it for finetunes (a merged checkpoint or a PEFT
+adapter) with the stock model's recipe. Neither opens a GPU: conversion is CPU-only and safe beside a
+running server.
+
+Fetch checkpoints with `scripts/hfget.sh <org/repo> <dir>` (resumable; reads `.safetensors`,
+skips `.bin`/ONNX/GGUF when safetensors exist; `HF_TOKEN` for gated repos; writes a
+`.hfget-manifest` of content keys). `radiance --model <owner/name>` runs the same fetch itself.
+
+## What a conversion does
+
+1. Read `config.json` (or GGUF metadata), and the shard index if present.
+2. Load the architecture, kernel and quantiser plugins and run **declare**, as the engine does.
+3. Resolve every op and ask each resolved kernel's `layout` hook about the planned encoding. A
+   recipe no kernel can serve is refused **here**, before any I/O.
+4. Write each logical weight once: the checkpoint's bytes where no rule matches, otherwise the
+   planes the matching quantiser writes, a block of rows at a time.
+5. Measure every quantised weight against its source through the core's decoder and print the worst.
+6. Bake the tokeniser, chat template, metadata, recipe and any draft head.
+
+```sh
+build/bin/rad-convert /srv/models/hf/Qwen/Qwen3.8-27B -o qwen3.8-27b-bf16.rad \
+    --recipe data/recipes/q38-27b-bf16.recipe
+build/bin/rad-convert /srv/models/hf/Qwen/Qwen3.8-Flash-Next -o flashnext-bf16.rad --max-ctx 4096
+```
+
+## Every `rad-convert` option
+
+| option | meaning |
+|---|---|
+| `<input>` | a `.gguf`, a `.safetensors`, an `index.json`, or a checkpoint directory |
+| `-o, --out FILE` | the container to write |
+| `--recipe FILE` | one rule a line: `PATTERN QUANTISER key=value …`; repeatable; first match decides |
+| `--quant RULE` | `PATTERN=QUANTISER[:key=value,…]`, placed **ahead of** the recipe's rules; repeatable |
+| `--list-quantizers` | every loaded quantiser and the options it takes, then exit |
+| `--arch ID` | override the architecture id read from the checkpoint |
+| `--home DIR` | `$RADIANCE_HOME` (kernels, architectures, quantizers) |
+| `--kernels A:B` | kernel libraries to load, first wins; declare checks the recipe against them |
+| `--tokenizer F` | `tokenizer.json`; its declared chain is baked instead of reconstructed from a pre-tokenizer name. Required for a GGUF, without it the chain is a guess |
+| `--tokenizer2 F` | a second `tokenizer.json` baked as the second vocab (a generator reading its prompt through two tokenisers); keep `tokenizer_config.json` beside it |
+| `--add FILE` | merge one more `.safetensors`, tensor names kept; repeatable; a name in two files is refused |
+| `--config DIR` | read `config.json`, `generation_config.json`, `tokenizer.json` from DIR for a single-file checkpoint |
+| `--chat-template F` | a Jinja template baked in, overriding the checkpoint's (or its absence); trailing newlines stripped; without one `/v1/chat/completions` answers 501 |
+| `--imatrix F` | llama.cpp imatrix: expert-popularity profile, per-column importances, error weighting; required by `ggml` types `iq2_xxs`, `iq2_xs`, `iq1_s` |
+| `--set K=V` | a metadata key, seen by declare **and** written into the container; repeatable |
+| `--max-tok N` | the `max_tok` declare runs at (default 8192, or a generator's largest pass) |
+| `--max-ctx N` | the context declare runs at (default: the checkpoint's training context) |
+| `--max-spec N` | the `max_spec` declare runs at (default 4, or the block a drafter states); `0` leaves a draft head out |
+| `--draft-model DIR` | a drafter shipped as its own repository, merged under `draft.` |
+| `--kv-cache-dtype T` | `bf16` or `fp8`: the cache width declare runs at; the container is identical at either |
+| `--reuse FILE` | copy every weight an earlier container holds with the same encoding, quantiser, options and shape |
+| `--changed FILE` | with `--reuse`: checkpoint tensors (one name a line) that differ; every weight reading one is converted |
+| `--in-place` | with `--reuse`: extend that container; new weights go past its end, the header last; keeps `<file>.pre-append` |
+| `--lora FILE\|DIR` | merge a PEFT adapter into the checkpoint tensors before converting (`W + strength · scale · B·A`); repeatable, merged in order; each merged tensor joins `--changed` |
+| `--lora-strength F` | strength for every `--lora` (default 1) |
+| `--lora-map FILE` | JSON map for adapters whose module names differ: `{"strip":[prefix…],"modules":{"<pat>":[target…]}}`; one `*` matches a block index; several targets mean a fused module (rows split in order) |
+| `--plan-only` | declare, plan every encoding, print the plan, stop before writing |
+| `--report N` | print the N worst quantisation errors (default 20) |
+| `-v, --verbose` | per-weight detail |
+
+`--reuse` matches a weight on its **encoding, quantiser, options and shape — not on its source
+bytes** — and carries whatever calibration made the old planes. Use it only when source and
+calibration are unchanged, or name the changed tensors with `--changed`. `--in-place` refuses to
+change a weight the container holds or to add an expert.
+
+## The recipe language
+
+```
+# PATTERN                      QUANTISER  OPTIONS
+blk.*.ffn_*_exps.*.weight      gptq       codes=i4 group=128 scale=bf16 transform=fwht128 calib=$CALIB
+blk.*.ple_ngram.weight         rtn        codes=fp8_e4m3 block=*x* scale=bf16 rule=fixed scale_value=1.99317932128906e-4
+output.weight                  rtn        codes=fp8_e4m3 block=128x128 scale=bf16
+blk.*.weight                   rtn        codes=fp8_e4m3 block=128x128 scale=bf16
+```
+
+- One rule a line: `PATTERN QUANTISER [key=value …]`. `#` starts a comment.
+- `PATTERN` is a glob over the **whole logical weight name**: `*` crosses dots, `?` is one character.
+- **The first matching rule decides.** No rule → the checkpoint's own encoding, which must be one
+  the engine serves as is: `bf16`, `f16`, `f32`, or block-scaled FP8 (`X.weight` with
+  `X.weight_scale_inv`). Anything else is named and refused.
+- `$NAME` / `${NAME}` in a value comes from the environment; an **unset variable is refused**, never
+  expanded empty (so an empty `$CALIB` cannot silently produce an uncalibrated convert).
+- A quantiser may **decline** a weight (`RAD_E_UNSUPPORTED`), which keeps the checkpoint's encoding
+  — `rtn` declines a rank-1 tensor, so a pattern that also matches a norm leaves the norm alone.
+- An option the quantiser does not declare is refused, by name, before anything is quantised.
+- Two plugins naming the same quantiser are refused, both named.
+
+## Quantisers
+
+| quantiser | what it does |
+|---|---|
+| `rtn` | round to nearest on a grid; each block's scale chosen by `rule` |
+| `gptq` | error feedback against the layer's input Hessian (`calib=DIR`, damped by `damp` = 0.01 of the mean diagonal by default); `act_order=1` quantises columns by descending input energy and stores the permutation; `cd=N` adds coordinate-descent sweeps |
+| `awq` | per-input-column scaling to a searched power, then a per-block clip search, both against the Hessian; column scales are the last scale level |
+| `autoround` | sign-gradient descent on a rounding offset per value and a range factor per block; `iters` steps; slow (a 27B dense model takes most of a working day at defaults; `iters=50` a quarter) |
+| `paroquant` | channel scaling plus `stages` Givens rotations learned against the Hessian, then AutoRound on the rotated weight; the `givens` transform |
+| `m2xfp4` | M2XFP's weight-side Sg-EM: MXFP4 plus one metadata byte per group of 32; its own `m2xfp4` scheme; `adaptive=0` pins the bias, `wide=1` widens it |
+| `ggml` | a llama.cpp format (`q4_0`, `q4_1`, `q5_0`, `q5_1`, `q8_0`, `q2_K`…`q6_K`, `iq4_nl`, `iq4_xs`, `iq2_xxs`, `iq2_xs`, `iq2_s`, `iq3_xxs`, `iq3_s`, `iq1_s`, `iq1_m`), by llama.cpp's own search, imatrix-weighted, codes byte for byte |
+| `cast` | a float weight in another float dtype (`dtype=` f32, bf16, f16, fp8_e4m3, fp8_e5m2), no scale |
+| `import` | rad-convert's own, not a plugin: write the numbers a quantised release ships, in the spelling the kernels read, instead of quantising again |
+
+Every quantiser except `m2xfp4`, `ggml`, `cast` and `import` takes the grid options:
+
+| option | values |
+|---|---|
+| `codes` | `i2`…`i8`, `i16`, `i32` symmetric; `u1`…`u8`, `u16` with `zero=`; `fp8_e4m3`, `fp8_e5m2`, `fp6_e2m3`, `fp6_e3m2`, `fp4_e2m1` |
+| `table` | a codebook the codes index: `nf4`, `iq4nl`, `w4nl`, `w5nl`, `binary`, `ternary`, or `;`-separated values |
+| `group=N` / `block=RxC` | where the scales sit; `*` is a whole extent |
+| `scale` | f32, bf16, f16, fp8_e4m3, e8m0; under `scale2` also an integer sub-scale `u4`…`u8`, `i6`, `i8` |
+| `scale2`, `group2` / `block2` | a second scale level the first is relative to (default the whole tensor) |
+| `scale2_value` | the second level fixed at this value, exact in `scale2`'s dtype |
+| `zero` | an integer zero point per block — an asymmetric grid |
+| `transform=fwhtN` | rotate each row by the Hadamard first |
+| `rule` | `absmax`, `mx` (OCP shared exponent), `fixed` (`scale_value=`), `search` (least squared error over candidates and refits, imatrix-weighted) |
+| `clamp` | a signed code's range: `full` or `sym` |
+
+Named grids: MXFP4 = `codes=fp4_e2m1 group=32 scale=e8m0`; NVFP4 =
+`codes=fp4_e2m1 group=16 scale=fp8_e4m3 scale2=f32`; NF4 = `table=nf4 group=64`; a K-quant-like grid
+= `codes=i4 group=32 scale=u6 scale2=f16 group2=256`; a ternary BitNet weight = `table=ternary`.
+
+**A container can hold only what some kernel reads.** rad-convert checks every planned encoding
+against the `layout` hooks of the kernels declare resolved and refuses one no kernel accepts. libr4d
+reads the plain floats; block-scaled fp8 and fp8 rows; int8 block rows plain or rotated; routed
+experts as fp8, int4 (`i4*bf16[1x128]/fwht128`) or the `w4nl`/`w5nl` codebooks under E4M3 scales;
+the dense w4, w8 and mxfp4 GEMM rows; and the 2-bit draft head. See [[Radiance Plugins]] for the
+encoding model.
+
+## Importing a release someone else quantised
+
+These forms are read weight by weight and written in the spelling the kernels read; a recipe rule
+re-quantises from the exact dequantisation where the family's kernels want another form.
+
+| `quantization_config` | read as | notes |
+|---|---|---|
+| `quark` MXFP4 (fp4, group 32, e8m0) | `fp4_e2m1*e8m0[1x32]` | byte for byte; also servable as it stands |
+| `paroquant_mxfp4` | `fp4_e2m1*e8m0[1x32]/givens` | codes/exponents byte for byte plus learned rotations as `givens` tables |
+| `paroquant`, `int5-bitplane`, plain `paroquant` | `u5*f16[1x128]-u8[1x128]/givens`, `u4*…` | AWQ-packed nibbles re-laid as a code bit stream; f16 scales, integer zeros, rotations |
+| `compressed-tensors` `pack-quantized`, 4-bit `int`, group, act-order none/static/weight | `i4*<scale>[1xG]` | `weight_packed` made signed beside `weight_scale`; group/dynamic reorder is refused |
+| same with `symmetric:false` (AWQ) | `u4*<scale>[1xG]-u4[1xG]` | nibbles plus `weight_zero_point`, bit for bit |
+| `auto-round` `auto_round:auto_gptq`, `gptq` v1 | `i4*bf16[1x128]` or `i8*bf16[1x128]` | scales rounded to bf16; convert-time; `import` keeps shipped f16 |
+| `nvfp4-pack-quantized` | `fp4_e2m1*f32[1x16]` | `weight_scale / weight_global_scale`; **no kernel serves NVFP4**, so it always takes a rule |
+| `modelopt` NVFP4 / FP8 / MIXED_PRECISION | NVFP4 `fp4_e2m1*f32[1x16]`; FP8 `fp8_e4m3*f32[1x*]` | ModelOpt's second scale **multiplies**; `input_scale` is not read |
+| compressed-tensors all-FP8 groups | `fp8_e4m3*<scale>[1x*]` or `[RxC]` | nothing imported: stock FP8 with its scale; a per-tensor scale is written once a row |
+| GGUF `Q8_0`, `Q4_K`, `Q6_K` | `i8*f16[1x32]`, affine u4/u6…, `i6*i8…` | blocks split bit for bit; **not servable as they stand** |
+
+The width is read **per tensor**, so a mixed release imports int4 experts and int8 projections
+together. What an import cannot read (an asymmetric GPTQ zero point, a group-reordering act-order
+index, a GPTQ group under 128, a missing/zero global scale, an unlisted format) is refused by name.
+The release's `quantization_config` names the tool, not a served form, so it is kept as
+`imported_quantization_config.*`.
+
+`import` options: `codes=uN` makes a symmetric grid unsigned under a zero point of `2^(N-1)`;
+`zero=` the zero plane's dtype (`u4`, `u8`); `scale=` f16/bf16/f32, each shipped scale rounded to
+nearest; `block=RxC` repeats an FP8 scale shipped per row/tensor over that grid. It reads an imported
+integer grid or FP8 with a per-row/per-tensor scale and keeps a plain float weight as it is.
+
+**NVFP4 to MXFP4 rounds twice** (e4m3 per 16 → power of two per 32), so it loses more than MXFP4 made
+from bf16. Prefer a calibrated MXFP4 container when one exists; the quality verdict is a KL gate, not
+a file size.
+
+## The descriptor law (`--set`)
+
+The architecture plugin is chosen by the exact `(arch id, quant descriptor)` pair. A convert-time
+choice that must also hold at serve time is stated once with `--set`, which hands it to declare and
+writes it into the container. Storage format is not such a choice (the encoding is in the container
+and the plugin reads it there), but the **serving plugin** is: a `quantization_config.*` key
+re-derives the descriptor.
+
+```sh
+rad-convert <bf16 Qwen3.8-27B> -o q38-27b-fp8.rad \
+    --recipe data/recipes/q38-27b-fp8block-df2.recipe \
+    --set quantization_config.quant_method=fp8 --set quantization_config.fmt=e4m3 \
+    --set "quantization_config.weight_block_size=128 128" \
+    --set quantization_config.activation_scheme=dynamic
+```
+
+A bf16 checkpoint whose recipe writes Qwen's block FP8 must therefore be declared *as that format*;
+otherwise the selection pair does not match any plugin and the convert (or the boot) is refused. The
+same law bites the other way when a release's own `quant_method` names a pair no plugin claims.
+
+## Inspect and validate the result
+
+```sh
+rad-info out.rad                  # architecture, source quant, recipe, sections
+rad-info --recipe out.rad         # the expanded rules — did the intended one win?
+rad-info -v out.rad | head -40    # every tensor, its encoding, and what made it
+rad-kbench -m out.rad --home $RADIANCE_HOME --report kbench.md
+radiance --model out.rad --debug-graph     # every op resolves
+```
+
+The converter's own error report (the worst `--report N` weights) measures each quantised weight
+against its source through the core decoder. `rad-kbench` adds the artifact test: weight operands
+read from the container and passed through each kernel's `layout`/`relayout` hooks. Both are on
+[[Radiance Tools]].
+
+## Calibration (GPTQ, AWQ, AutoRound, ParoQuant)
+
+`calib=DIR` is a directory an engine run wrote with `RADIANCE_CALIB_DIR` set: each layer's **Gram
+matrices of its input**, in the domain the kernel multiplies it in. A block that characterises an
+activation declares an f32 `[K,K]` PERSIST buffer and a `gram_accum` op; at `RADIANCE_CALIB_TOKENS`
+(default 65536) the run writes the files.
+
+Three-step bootstrap: convert with plain `rtn` → serve that container with `RADIANCE_CALIB_DIR` →
+push real prompts through prefill → convert again with the Hessians. The second moment is robust to
+the first container's small quantisation error. A weight with no file is quantised uncalibrated and
+the converter says so; `awq`, `autoround` and `paroquant` fall back to the imatrix diagonal when one
+is given.
+
+- The accumulator must be **PERSIST**: a transient buffer may share bytes and be overwritten between
+  the launches it sums.
+- Files are per rank; the consumer sums them. Under expert parallelism a rank's `down` input covers
+  only its experts' tokens; `gate_up` input is the full token set on every rank, so the sum scales
+  `H` by world size — GPTQ is invariant to a positive scale.
+- Under tensor parallelism a column-split linear sees only its rank's columns, so a tap may write
+  `gram.r<rank>.<stem>.part.bin`; libquant places the parts on the diagonal in rank order, leaving
+  cross-rank products zero. They must cover K and be in the model's own domain.
+- `--imatrix` supplies the expert-popularity profile the placement planner ranks units by, the
+  per-column importances a quantiser uses, and the error report's weighting.
+
+## Reuse, drafters, merging
+
+```sh
+# re-quantise only what changed
+rad-convert $SRC -o new.rad --recipe new.recipe --reuse old.rad --changed changed.txt
+
+# merge a drafter shipped as its own repo (DFlash2, DSpark, MTP assistant)
+rad-convert $SRC -o out.rad --draft-model /srv/models/z-lab/Qwen3.8-27B-DFlash2 \
+    --recipe data/recipes/q38-27b-mxfp4-df2.recipe
+
+# merge a PEFT adapter at convert time
+rad-convert $SRC -o tuned.rad --lora ~/run/adapter --lora-strength 1.0 \
+    --recipe data/recipes/q38-27b-mxfp4-df2.recipe
+```
+
+- A drafter's tensors go in under `draft.`, its config under `draft.` keys. It has no embedding and
+  no `lm_head` of its own; it uses the target's.
+- `--lora-map` exists for adapters trained on names the checkpoint does not use; a fused target
+  (`qkv`) takes several targets and splits the adapter's rows in order.
+- `--add` + `--config` + `--tokenizer2` assemble a ComfyUI-style release of separate single files
+  (Anima: diffusion model, text encoder, VAE). A name in two files is refused.
+
+## `rad-export` — finetunes without hand-writing the command
+
+```sh
+rad-export ~/outputs/my-run/checkpoint-400 --dry-run   # what it would run
+rad-export ~/outputs/my-run/checkpoint-400             # adapter -> <name>.rad
+rad-export ~/exports/merged -o ~/models/mine.rad --recipe q38-27b-fp8block-df2
+```
+
+| flag | default | meaning |
+|---|---|---|
+| `SRC` | required | a merged 16-bit checkpoint (`config.json` + `*.safetensors`) or a PEFT adapter (`adapter_config.json` + `adapter_model.safetensors`) |
+| `-o, --out FILE` | SRC's folder name + `.rad`, beside SRC or in `out_dir` | the container |
+| `--recipe NAME\|FILE\|auto` | `auto` | a recipe name, a file, or the one recipe whose header names the base model, takes a bf16 checkpoint and needs no calibration |
+| `--base DIR` | the adapter's `base_model_name_or_path`, from disk or the HF cache, else downloaded | the base a LoRA merges into |
+| `--draft DIR\|ORG/NAME` | the config, else the Hub repo the recipe's usage line names | the drafter when the recipe merges one; a usage line with a placeholder asks for `--draft` |
+| `--image IMAGE` | `$RADIANCE_IMAGE`, the config's, then `stilldeadcode/radiance` | run `rad-convert` in this image (docker, CPU-only, no network, **not pulled**) |
+| `--rad-convert PATH` | — | run this binary instead of docker; recipes come from its `share/radiance` |
+| `--memory M` | — | docker memory limit for the convert |
+| `--dry-run` | — | resolve recipe, drafter and paths; print the `rad-convert` line; write nothing |
+| `--keep-merged` | — | keep the merged checkpoint beside OUT |
+| `--config FILE` | `~/.config/rad-export/config.json` | the config below |
+| `--list-recipes [--json]` | — | list the recipes the image/installed tool carries |
+
+- Input: a merged 16-bit checkpoint (`config.json` + `*.safetensors`) or a PEFT LoRA adapter
+  (`adapter_config.json` + `adapter_model.safetensors`).
+- An adapter is merged **exactly as PEFT's `merge_and_unload`** computes it on CPU: `scale` =
+  alpha/r (alpha/√r with rsLoRA), `rank_pattern`/`alpha_pattern` honoured, delta in fp32 rounded to
+  the adapter's dtype. It streams tensor by tensor into 1 GiB shards, so a 27B merge needs a few GiB
+  of RAM, not the model twice. Plain linear LoRA only: embedding LoRA, trained biases, DoRA, saved
+  modules, convolution LoRA, a rank other than the config's, and a missing target are refused before
+  anything is written. The adapter's tokeniser/template replace the base's where shipped.
+- The recipe comes from the image (`/opt/radiance/share/radiance/recipes`) or the installed
+  `rad-convert`'s share; `auto` matches the base model's name in recipe headers and asks when several
+  fit. A recipe needing calibration is listed but never picked by `auto` — a finetune's Hessians are
+  not the stock model's.
+- It runs `rad-convert` on the CPU, under Docker as the caller with no network, and writes the log to
+  `OUT.log`.
+- `~/.config/rad-export/config.json` pins what auto cannot know: `image`, `out_dir`,
+  `recipes{model:recipe}`, `drafts{recipe:dir}`, `bases{repo:dir}`.
+- It is Python 3 with no dependencies; merging needs `torch`/`safetensors`, fetching a missing base or
+  drafter needs `huggingface_hub`. The image carries no Python; extract the script with
+  `docker run --rm --entrypoint busybox stilldeadcode/radiance cat /opt/radiance/bin/rad-export`.
+
+## Scoring a container (the KL mode)
+
+```sh
+# once, on the bf16 container
+radiance --model bf16.rad --kld-record /srv/kld/q38 --kld-corpus corpus.jsonl
+# each candidate, same flags otherwise
+radiance --model quant.rad --kld-ref /srv/kld/q38 --kld-out report.json
+```
+
+- The corpus is JSONL, one `{"prompt", "score_from"?, "source"?}` a line; only `prompt` is required.
+  Positions `[score_from, n-1)` are scored; `source` groups the report. Keep the scoring corpus
+  disjoint from the calibration corpus.
+- `--kld-record` writes the reference's log-probabilities (f16 over the vocabulary, the 16 most
+  likely tokens exactly, the token ids). DIR is never overwritten and is read only once `kld.json`
+  exists, which is written last.
+- `--kld-ref` scores this model's f32 logits against stored rows: mean, median, 99/99.9/99.99th
+  percentile KL, top-1 agreement, and both models' perplexity. `--kld-out` writes the report; `FILE.rows`
+  is `[positions, 4]` f32 (KL, both NLLs, argmax agreement) for position-by-position comparison.
+- The mode turns speculation and the prefix cache off, turns the step pipeline off, and refuses an
+  explicit spec depth above 0. The logits buffer holds a row per token of a step, so a large
+  `--max-num-batched-tokens` costs pinned host memory as well as VRAM.
+- `--kld-lora NAME` scores every document through a served adapter, against a reference recorded
+  from the merged checkpoint; `--kld-lora-sources PREFIX` runs only matching documents through it,
+  the rest as the base, in the same steps — each document's rows should be byte-identical to the
+  all-adapter and all-base runs.
+
+The gate the fleet uses (mean KL ≤ 0.03, top-1 ≥ 95 %) is on [[frank-archon]] and
+[[Quantization Production]]; the mode here is only the instrument.
+
+## Convert laws paid for in full
+
+- **`--plan-only` first.** A plan that resolves every op will boot; it costs seconds against an hour
+  of I/O on a large checkpoint.
+- **Declare runs at the step and context bounds, and some graphs depend on them.** Qwen4-Exp sizes
+  its QSA buffers by `--max-ctx` and refuses a `--max-tok`/context pair whose score plane would pass
+  2 GiB; the defaults (8192 tokens, the checkpoint's training context) fit most models, and a
+  generator's largest pass may override `--max-tok`.
+- **Scale names follow the tensor.** `scale_name()` appends `_scale_inv` to the *tensor* name, so a
+  stacked `…experts.gate_up_proj` keeps its scales at `…gate_up_proj_scale_inv`.
+- **File shadowing: first wins.** The reader opens index-named files sorted and emplaces every tensor
+  each file physically holds, first-wins. A merged PLE shard must sort first
+  (`00-ple.merged.safetensors`) or a raw shard wins the name.
+- **Never write through a staging symlink.** A checkpoint may symlink its raw index; `open(path,"w")`
+  writes through it and corrupts the checkpoint. Write a temp file and `os.replace`.
+- **One encoding per expert table.** A whole expert kept bf16 is the only thing legal beside a
+  table; a split tier (down=fp8, gate_up=w4) is unservable.
+- **The descriptor pair has no fallback** (above), and **convert is card-less**: GPTQ solves on CPU
+  BLAS, so the GPU is irrelevant to production and the card only decides what can be decoded.
+
+## Related
+
+- [[Radiance Model Families]] — the (arch id, quant descriptor) table and the command per model.
+- [[Radiance Plugins]] — quantiser and kernel-plugin contracts behind these options.
+- [[Radiance Tools]] — `rad-info`, `rad-kbench`, `rad-tune`.
+- [[Quantization Production]] — the fleet's calibration/KLD campaign and its gates.
+- [[frank-archon]] — the deployment whose recipes and convert laws these are.

serverAppConfigs-docs/LLM/Radiance Engine.md

@@ -0,0 +1,140 @@
+---
+type: "reference"
+title: "Radiance Engine — the inference server behind frank-archon"
+description: "The radiance inference engine: its shape, plugin model, container format, and how the reference pages fit together."
+tags:
+  - llm
+  - radiance
+  - reference
+timestamp: 2026-10-30
+---
+
+# Radiance Engine — shape, plugin model, container
+
+Radiance is StillDeadcode's inference server for AMD GPUs, written in C++20 and HIP: language
+models, plus image, video, speech and music generation and embeddings. One HTTP front end speaks
+OpenAI's chat, completions, Responses, images, audio and video APIs, Anthropic's Messages API and
+llama-server's native one. Nothing in the serving path is Python.
+
+The fleet's only radiance endpoint is [[frank-archon]] (2× R9700, TP=2, Dankserver). **This page
+and its siblings own the engine reference; the live deployment shape, its Eras and its measured
+laws stay on the endpoint page.** The numbers here describe upstream `main` as read 2026-10-30
+(release 1.14.0, plugin ABI 31). The fleet pin is older (1.8.1): a flag below that post-dates the pin
+is a boot refusal, never a silent no-op, and the endpoint page records the CLI families that pin
+itself retired relative to its own rollback target.
+
+Source: `https://codeberg.org/StillDeadcode/radiance` at `8da043c`. Engine docs live in `docs/`
+(`GUIDE.md` for serving, `TOOLS.md` for tools, `PLUGIN.md` for plugins, `ARCHITECTURES.md` per
+family, `OPS.md` for the op vocabulary, `spec.md` for the design).
+
+## The four moving parts
+
+```mermaid
+graph LR
+    A[Checkpoint<br/>safetensors / GGUF] -->|rad-convert + recipe| B[.rad container]
+    B --> C[radiance engine]
+    C --> D[OpenAI / Anthropic /<br/>llama-server wire APIs]
+    P1[kernel library .so<br/>libr4d, libavx, libref] --> C
+    P2[architecture .so<br/>one per arch id × quant descriptor] --> C
+    P3[quantiser .so<br/>libquant] -->|rad-convert| A
+```
+
+- **Engine** — `core/`: declare phase, placement/memory planner, scheduler, KV manager, sampler,
+  tokeniser, HTTP server, media preprocessing.
+- **Kernel libraries** — implement ops for hardware. `libr4d` targets RDNA4 (`gfx1200`,
+  `gfx1201`); `libavx` serves the host backend (AVX-512/AVX2) with no GPU at all; `libref` is the
+  slow reference implementation every other library is checked against.
+- **Architecture plugins** — declare one model family at one weight format: its weights, buffers,
+  KV groups and the ops it issues each step. The engine picks one by an **exact (architecture id,
+  quantisation descriptor) pair, with no fallback**.
+- **Quantiser plugins** — `libquant`'s `rtn`, `gptq`, `awq`, `autoround`, `paroquant`, `m2xfp4`,
+  `ggml`, `cast`, `import`; they write a weight's numbers into an encoding's planes.
+- **Tools** — `rad-info`, `rad-convert`, `rad-export`, `rad-kbench`, `rad-tune`, `rad-schemas`,
+  `rad-tokcheck`, plus `rad-chatparse`, `rad-avx-check`, `r4d_selftest` in the build tree.
+
+## Two phases, one static graph
+
+`rad_arch_declare` runs once per rank and names every weight, buffer, KV group and op, in order.
+The core resolves every op against the kernel hierarchy per band, per domain, and completes the
+whole plan before a byte of weights is read. `rad_arch_step` then only *issues* handles obtained at
+declare; it may not allocate and may not synchronise. A model whose declare succeeds is fully known
+statically, which is what makes `--debug-graph`, the buffer liveness plan and `rad-convert`'s
+plan-only check possible.
+
+Consequences a plugin author must respect:
+
+- An op that never resolves is `RAD_NULL_HANDLE` and only fails if issued. A fusion is a *selection*:
+  ask for the fused op, emit the chain when `rad_op_resolved` is false.
+- Declaration order is the buffer planner's liveness. A drafter must be declared after the trunk
+  it runs behind, or trunk and drafter buffers silently share bytes.
+- Every op in the graph must have a row in a loaded kernel library. `libref` implements the
+  conventional vocabulary in `docs/OPS.md`; device-only ops (most fusions) exist only in `libr4d`.
+
+## The `.rad` container (RAD2)
+
+A container is one file holding everything the server needs: weights as **canonical encoding
+planes**, the tokeniser (and optionally a second vocab), the chat template, metadata, the recipe it
+was made with, and any draft head. It names no kernel library.
+
+- Planes are kernel-independent: codes, scales, zero points, codebooks — row-major, bit-packed
+  sub-byte where needed. A kernel rearranges them into its own layout at load (`layout`/`relayout`
+  hooks), losslessly, so a kernel library can change layouts without reconverting the model.
+- Encodings are data, not an enum: `RadEncoding` names the scheme (`plain`, `affine`, or a
+  quantiser's own), the planes by role, the scale grouping, and a transform (`fwhtN`, `perm`,
+  `givens`). That is how one file can hold block-FP8, int4, MXFP4, NVFP4-imported, ParoQuant
+  rotations and llama.cpp grids.
+- `rad-info` is the reader: sections (strings, metadata, directory, planes, encodings, vocab,
+  profile, data), `--meta`, `--vocab`, `--recipe`, `--plugins`, and `-v` for every tensor with its
+  encoding and what made it.
+
+See [[Radiance Conversion]] for producing one and [[Radiance Tools]] for inspecting it.
+
+## Selection key and version law
+
+- A checkpoint's architecture id is its `config.json` `model_type` without underscores; `rad-convert
+  --arch ID` overrides it. `rad_arch_quant` supplies the quantisation descriptor (empty for an
+  unquantised checkpoint).
+- **No fallback.** A raw official FP8 checkpoint whose `quantization_config.quant_method` is `fp8`
+  is not the pair the `qwen4exp_fp8` plugin claims (`(qwen4exp, "")`), so both serve and convert
+  refuse it until the descriptor is changed with `--set` or a staged `config.json`. The deployment
+  lesson is on [[frank-archon]]; the mechanism is on [[Radiance Conversion]].
+- `RAD_ABI_VERSION` covers every header a plugin compiles against. A plugin reporting a different
+  number is refused at load, with both numbers printed. It moves on any shape change, not on a new
+  optional export; rebuild when it changes. Release version and ABI are independent: `/version`
+  reports the release, `rad-info --plugins` reports what loaded.
+
+## Environment
+
+| variable | meaning |
+|---|---|
+| `RADIANCE_HOME` | plugin search path: `:`-separated homes, each holding `kernels/`, `architectures/`, `quantizers/`; first wins, same-name files shadow later homes. Compiled-in default is `<prefix>/share/radiance` |
+| `RADIANCE_KERNELS` | default for `--kernels`: ordered kernel libraries, first wins; unnamed libraries are not loaded |
+| `RADIANCE_TUNE` | pin a kernel's tuned axes for a measurement (a measurement aid, not a deployment setting) |
+| `RADIANCE_CALIB_DIR` | a serve run records per-layer input Gram matrices here (the GPTQ calibration tap) |
+| `RADIANCE_CALIB_TOKENS` | tokens after which the tap stops and writes (default 65536) |
+| `RADIANCE_LOG`, `RADIANCE_LOG_STEPS` | log level; per-step token counts |
+| `RADIANCE_DEBUG_RESID` | residual-stream probe per (layer, site) |
+| `RADIANCE_DFLASH_Q_TEMP` | temperature of a DFlash2 coupled walk's candidate softmax |
+
+The full list of debug knobs (`RADIANCE_DEBUG_*`, `RADIANCE_DUMP_*`, `RADIANCE_OP_ORACLE`, …) is in
+the source's `getenv` calls; none is a deployment setting.
+
+## Where the pages are
+
+| Page | Owns |
+|---|---|
+| [[Radiance Serve Options]] | every serve flag, grouped, with defaults and the law behind each knob |
+| [[Radiance HTTP API]] | endpoints, request fields, streaming, tool calls, refusal rules |
+| [[Radiance Conversion]] | `rad-convert`, the recipe language, quantisers, imports, reuse, `rad-export`, KL scoring |
+| [[Radiance Model Families]] | which plugin claims which model, and the conversion command per family |
+| [[Radiance Plugins]] | kernel and quantiser plugins: ABI, rows, hooks, encodings, build and checks |
+| [[Radiance Architecture Plugins]] | declaring a model: weights, buffers, KV groups, ops, step, probe |
+| [[Radiance Tools]] | `rad-info`, `rad-kbench`, `rad-tune`, `rad-schemas`, `rad-tokcheck`, harnesses |
+| [[Radiance Operations]] | the container contract, host prerequisites, readiness, ledgers, restarts |
+| [[frank-archon]] | the fleet's live radiance endpoint: shape, Eras, rollback, measured laws |
+
+## Sources
+
+- `README.md`, `docs/GUIDE.md` §1–§4, `docs/TOOLS.md`, `docs/PLUGIN.md`, `docs/DOCKER.md`,
+  `docs/ARCHITECTURES.md`, `docs/OPS.md`, `spec.md` §2–§4, `abi/*.h`, `core/config.cpp`,
+  `core/rad_core.h` at commit `8da043c` (codeberg.org/StillDeadcode/radiance).

serverAppConfigs-docs/LLM/Radiance HTTP API.md

@@ -0,0 +1,198 @@
+---
+type: "reference"
+title: "Radiance HTTP API — endpoints, fields, refusals"
+description: "The HTTP surface radiance serves: endpoints, request fields, streaming, tools, and refusal rules."
+tags:
+  - llm
+  - radiance
+  - api
+  - reference
+timestamp: 2026-10-30
+---
+
+# Radiance HTTP API
+
+One server serves one model. It answers OpenAI's chat/completions/Responses/images/audio/video APIs,
+Anthropic's Messages API and llama-server's native one, so their clients and SDKs work unchanged.
+`model` in a request is accepted and **not enforced**.
+
+## Endpoints
+
+| route | what it serves |
+|---|---|
+| `POST /v1/chat/completions` | chat: tools, reasoning, streaming, grammars |
+| `POST /v1/completions` | raw text completion |
+| `POST /v1/embeddings` | an embedding head, else 501 |
+| `POST /v1/images/generations`, `/v1/images/edits` | OpenAI images API; needs an image model, else 501 |
+| `POST /v1/systemone` | LiquidAI d1 decisions: named yes/no, choice and score questions as probabilities; needs a d1 container, else 501 |
+| `POST /v1/audio/transcriptions` | OpenAI speech-to-text; needs a transcribing plugin, else 501 |
+| `GET /v1/realtime` (WebSocket) | streaming transcription session |
+| `POST /v1/audio/speech` | speech (Breeze-TTS-2, drama-3) or a song (MiniMax-Music3), whole or streamed |
+| `GET /v1/audio/speech/stream` (WebSocket) | text streamed in, each sentence's audio out |
+| `POST /v1/videos`; `GET /v1/videos[/:id[/content]]`; `DELETE /v1/videos/:id` | video jobs made in the background, downloaded as MP4 |
+| `POST /v1/messages`, `/v1/messages/count_tokens` | Anthropic Messages, for Claude Code and the Anthropic SDKs |
+| `POST /v1/responses`, `/v1/responses/input_tokens` | OpenAI Responses, for Codex and SDK clients |
+| `POST /completion`, `/completions`, `/infill` | llama-server native completion and fill-in-the-middle |
+| `GET /v1/models`, `/v1/models/:id` | what is loaded, with `max_model_len`, `root`, `capabilities` |
+| `GET /health`, `/v1/health`, `/ping`, `POST /ping` | liveness |
+| `GET /load`, `/version`, `/props`, `/server_info`, `/diagnostics` | load; release version; llama-server props; full configuration; the startup report as JSON (`/diagnostics` is same-machine-only without an API key) |
+| `POST /apply-template`, `/v1/chat/completions/input_tokens` | the rendered prompt; its token count |
+| `GET /metrics` | Prometheus |
+| `POST /tokenize`, `/detokenize`; `GET /get_tokenizer_info` | vLLM's shape and llama-server's at once |
+| `POST /reset_prefix_cache`; `DELETE /sessions/<label>`, `/sessions?id=<id>` | drop every cached prefix; forget a stored conversation |
+| `GET /`, `/dashboard`; `GET /stats`, `/graph`, `/sessions` | the dashboard and the JSON it is drawn from |
+
+Unversioned shorthands exist for chat, chat input tokens, embeddings, images, audio transcriptions,
+audio speech, models, responses and completions, for a client given the bare host as base URL.
+`/completions` **without** `/v1` is llama-server's native completion. A path nothing serves is a JSON
+404.
+
+A text prompt to `/v1/completions`, `/v1/embeddings` or `/tokenize` reads control tokens as tokens
+(as vLLM and llama.cpp do), so a client rendering its own template can send `<|im_start|>user\n…`.
+The model's declared BOS is prepended unless the prompt already opens with it.
+
+## Chat
+
+```sh
+curl -s localhost:8000/v1/chat/completions -H 'Content-Type: application/json' \
+  -d '{"model":"m","messages":[{"role":"user","content":"hi"}],"max_tokens":200,"temperature":0.7}'
+```
+
+- **No chat template → 501** on this route (the container carries one; `--override-chat-template`
+  replaces it). Use `/v1/completions` instead.
+- Thinking off has three spellings: `chat_template_kwargs.enable_thinking=false`,
+  top-level `enable_thinking=false`, or `reasoning_effort:"none"`.
+- Reasoning returns in `choices[].message.reasoning_content`, separate from `content`.
+  **Reasoning counts against `max_tokens`**; a reply that runs out inside it has empty `content` and
+  `finish_reason:"length"`. `usage.completion_tokens_details.reasoning_tokens` tells that case from a
+  model that answered nothing.
+- `thinking_token_budget` (aliases `thinking_budget`, `thinking_budget_tokens`,
+  `reasoning_budget_tokens`) closes the reasoning block at N tokens and forces the close, so it holds
+  under speculation and at any temperature. It needs a one-token end marker; otherwise the response
+  has no `completion_tokens_details` and a budget is a 400.
+- `continue_final_message` continues a conversation ending in an assistant message instead of
+  answering it; `add_generation_prompt` opens a new turn.
+- Tool calls: `tools`, `tool_choice` (`auto`, `none`, `required`, named function),
+  `parallel_tool_calls`. `required`/named hold the whole reply to calls after the reasoning block and
+  cannot combine with `response_format` or `grammar`.
+- Structured output: `response_format` (`json_object` or `json_schema`), llama-server's `json_schema`
+  shorthand, or a raw GBNF `grammar`. A chat template that drops the grammar makes the request a
+  refusal.
+
+## Streaming
+
+```sh
+curl -N localhost:8000/v1/chat/completions -H 'Content-Type: application/json' \
+  -d '{"model":"m","messages":[{"role":"user","content":"Count to ten."}],
+       "max_tokens":100,"stream":true,"stream_options":{"include_usage":true}}'
+```
+
+Server-sent events in OpenAI's delta format, terminated by `data: [DONE]`. **Nothing is sent while
+the prompt is read**: the first event (the `role: assistant` chunk) goes out with the first generated
+token, so a client timing its first event measures time-to-first-token. `return_progress` adds
+llama-server's `prompt_progress` chunks during prefill; `timings_per_token` adds `timings` to every
+chunk.
+
+## Completions, embeddings, images, audio, video, tokens
+
+- **Completions** — `/v1/completions` takes `prompt` (string or token ids), `echo`, `logprobs`,
+  `preserved_tokens`. `/completion` is llama-server's native route; `/infill` is fill-in-the-middle.
+- **Embeddings** — `input` is text, a batch, or token ids; `encoding_format` `float`/`base64`;
+  `dimensions` truncates to the first N and renormalises; `prompt_name` prepends the model's named
+  prompt. Pooling is the model's; the engine does not renormalise.
+- **Images** — `/v1/images/generations` returns `b64_json`; `size` defaults to 2048×2048. Edits take
+  one picture or up to ten (`image[]`), multipart as OpenAI's SDKs send them, with an optional mask
+  (repaint only where transparent). `partial_images` streams previews as SSE.
+- **Speech to text** — `/v1/audio/transcriptions`; `/v1/realtime` streams audio in and words back as
+  they are spoken. The model's prompt shape comes from its plugin's audio format, not from the server.
+- **Speech and music** — `/v1/audio/speech` (whole or streamed) with `instructions` describing a voice
+  (Breeze-TTS-2) or speakers written in the input (`<|speaker:1|>`, `[laughs]` for drama-3); a music
+  model takes the lyrics as `input` and the description as `instructions`. `/v1/audio/speech/stream`
+  streams text in and audio out as each sentence completes.
+- **Video** — `/v1/videos` is a job: a video with its stereo soundtrack from a prompt, 5–14 s, up to
+  1344×768, made in the background; `GET /v1/videos/:id/content` downloads the MP4.
+- **Tokenising** — `POST /tokenize` accepts vLLM's shape (`prompt` or `messages` in, `prompt` out) and
+  llama-server's (`content` in/out, `with_pieces`). `POST /detokenize` reverses it.
+  `GET /get_tokenizer_info` reports the tokeniser configuration.
+
+## Request fields
+
+Accepted fields, in OpenAI's, vLLM's and llama-server's spellings:
+
+| field | notes |
+|---|---|
+| `model` | accepted, not enforced |
+| `messages` / `prompt` / `input` | chat / completion / embedding |
+| `max_tokens`, `max_completion_tokens`, `n_predict` | default is what the context leaves; `-1` is llama-server's "no limit"; clamped to context and to `--max-tokens-cap` |
+| `temperature`, `top_p`, `top_k`, `min_p`, `typical_p` | sampler |
+| `presence_penalty`, `frequency_penalty`, `repetition_penalty`/`repeat_penalty`, `repeat_last_n` | penalties |
+| `xtc_probability`, `xtc_threshold`, `dry_*` | XTC and DRY |
+| `seed` | absent means "pick one", not 0 |
+| `n` | 1..`--max-n`; prefix cache makes the fan-out nearly free after the first |
+| `stop` | string or array, bounded by `--max-stop-strings`/`--max-stop-bytes` |
+| `priority` | scheduler hint |
+| `ignore_eos` | generate to `max_tokens` whatever the model emits |
+| `stream`, `stream_options.include_usage`, `timings_per_token`, `return_progress` | streaming controls |
+| `samplers` | llama-server's order; this engine's chain is penalties, DRY, top-k, top-p, min-p, typical, XTC, with temperature first or last; any other list is refused by name |
+| `session_id` | labels the conversation on `/sessions`, dashboard and `DELETE` |
+| `control_vector` | `true` applies the deployment's vector to this request; absent takes the deployment default; a deployment with no vector refuses `true` |
+| `logprobs`, `top_logprobs`, `n_probs` | up to 20 alternatives; not with `echo` |
+| `tools`, `tool_choice`, `parallel_tool_calls`, legacy `functions`/`function_call` | chat only |
+| `add_generation_prompt`, `chat_template_kwargs`, `continue_final_message` | chat only |
+| `reasoning_format`, `reasoning_effort`, `enable_thinking`, `thinking_token_budget`, `preserve_thinking` | chat only; levels are the template's (`capabilities.reasoning_efforts`); `none`/`minimal` turn thinking off |
+| `response_format`, `json_schema`, `grammar` | structured output; `grammar` and `response_format` are mutually exclusive |
+| `echo`, `preserved_tokens` | `/v1/completions` only |
+| `encoding_format`, `dimensions`, `prompt_name`, `image_tokens` | `/v1/embeddings` only |
+| `images_config` | image models only: a named processor mode or an object restating served sizes |
+
+## What is refused, ignored, and clamped
+
+The contract: **a 200 means every field that could change the answer was honoured.** Fields fall
+into four classes.
+
+- **Inert** — accepted and ignored because they cannot change the answer: OpenAI/OpenRouter
+  bookkeeping (`user`, `metadata`, `store`, `service_tier`, `prompt_cache_key`, …) and llama-server's
+  own speed/slot knobs (`cache_prompt`, `id_slot`, `n_cache_reuse`, `backend_sampling`,
+  `speculative.*`, …). The prefix cache and speculation here are the server's and never change output.
+- **Guarded** — known, unimplemented, and able to change the answer: accepted at their off value,
+  refused by name otherwise. `logit_bias` (empty), `mirostat` (0), `top_n_sigma` (≤0),
+  `dynatemp_range` (≤0), `min_keep` (0), `post_sampling_probs` (false), `best_of` (≤1), `suffix`
+  (empty), `lora` (empty — an adapter is chosen by `model`), `modalities`/`audio` (`["text"]`/null),
+  `n_indent` (0), `t_max_predict_ms` (≤0), `return_tokens` (false), `grammar_lazy`,
+  `grammar_triggers`, `preserved_tokens` (empty on chat).
+- **Ignored** — anything else unknown, logged at debug. An unknown field is far more often a client's
+  extension than a misunderstood request.
+- **`max_tokens` is the exception**: if prompt + `max_tokens` exceeds the context, it is **clamped**
+  and served, ending `finish_reason:"length"`, because refusing would produce no tokens instead of
+  fewer. A prompt that does not fit on its own is a 400 `context_length_exceeded` naming
+  `n_prompt_tokens`, `n_ctx`, and how many tokens must go.
+
+## Anthropic, Responses, llama-server
+
+- `/v1/messages` is translated into a chat request, so chat defaults, refusals and limits apply.
+  Streamed (`message_start`, content blocks, `message_delta`, `message_stop`) or not:
+  `ANTHROPIC_BASE_URL=http://host:8000 ANTHROPIC_API_KEY=$KEY claude` works. `/v1/messages/count_tokens`
+  returns the prompt length.
+- `/v1/responses` maps `instructions`/`input` (strings, messages, `function_call`,
+  `function_call_output`, `reasoning` items) into the chat request; `max_output_tokens`,
+  `reasoning.effort`, `text.format` map to chat fields; undefined fields pass through. The event stream
+  carries `output_index`, `content_index`, `sequence_number`, and opens with `response.created` and
+  `response.in_progress`. A reply cut at `max_output_tokens` is `incomplete`. Function tools a
+  `namespace` groups are named `<namespace>__<name>`; built-in tools and freeform custom tools are
+  left out.
+- `/completion`, `/completions`, `/infill` are llama-server's own shapes, for llama.vim/llama.vscode.
+
+## Auth, CORS, errors
+
+- `--api-key KEY` requires `Authorization: Bearer KEY` or `X-Api-Key: KEY` on every request except
+  `/health`, `/v1/health`, `/ping` and the dashboard page. An empty key is refused at startup.
+- CORS headers are sent unless `--no-cors`.
+- Errors are JSON. Over-context is 400 `context_length_exceeded`; a full queue is 429 with
+  `Retry-After`; a capability the model lacks is 501; a guarded field is 400 naming the field and its
+  accepted off value.
+
+## Related
+
+- [[Radiance Serve Options]] — the defaults every omitted field takes.
+- [[Radiance Operations]] — `/health` semantics, metrics, dashboard.
+- [[frank-archon]] — the harness contract built on this surface.

serverAppConfigs-docs/LLM/Radiance Model Families.md

@@ -0,0 +1,142 @@
+---
+type: "reference"
+title: "Radiance Model Families — arch ids, descriptors and recipes"
+description: "Which radiance plugin claims which model family, and the conversion command and recipe of record for each."
+tags:
+  - llm
+  - radiance
+  - models
+  - reference
+timestamp: 2026-10-30
+---
+
+# Radiance model families
+
+The selection key is the exact **(architecture id, quantisation descriptor)** pair, so a family at
+two weight formats is two `.so` files. A checkpoint's id is its `config.json` `model_type` without
+underscores (`qwen3_5` → `qwen35`); `rad-convert --arch ID` overrides it. `data/recipes/` is the
+authority: **read the recipe's header** — it names the exact `rad-convert` line, its `--set` keys
+and whether it needs calibration.
+
+## The plugin table
+
+| plugin | id | quant | what it serves |
+|---|---|---|---|
+| `llama_fp8` | `llama` | `fp8_e4m3` | plain LLaMA decoder (MiniCPM5, Llama 3, derivatives reporting `model_type: llama`) from a checkpoint holding E4M3 codes + `weight_scale_inv` |
+| `llama_dense_fp8` | `llama` | *(empty)* | the same graph from a bf16 checkpoint, linears made block fp8 by a recipe |
+| `qwen35_bf16` | `qwen35` | *(empty)* | Qwen3.5/3.6/3.8 dense hybrid (GDN + gated attention), bf16 |
+| `qwen35_fp8` | `qwen35` | `fp8_e4m3` | the same, block-scaled FP8 weights and activations |
+| `qwen35_w8a8` | `qwen35` | `w8a8_fwht128` | int8 trunk: i8 weights, bf16 scale per 128 of K, 128-wide Hadamard, int8 activations |
+| `qwen35moe_fp8` | `qwen35moe` | `fp8_e4m3` | `qwen35_fp8` with a routed feed-forward |
+| `qwen4exp_fp8` | `qwen4exp` | *(empty)* | Qwen4-Exp / Qwen3.8-Flash-Next: a bf16 checkpoint quantised at convert, or Qwen's FP8 release as it ships |
+| `glm5next` | `glm5next` | *(empty)* | GLM-5.3-Flash: four mHC residual streams, 34 KDA + 11 sparse MLA layers with a DSA indexer, 288 MXFP4 experts top-8, block-FP8 attention/dense at f32 scales, MTP drafter, vision tower |
+| `hyv3_bf16` | `hyv3` | *(empty)* | Tencent Hy-Memory-30B-A3B bf16; bf16/W4/W8A8/block-FP8/MXFP4 by recipe |
+| `hyv3_fp8` | `hyv3` | `fp8` | Hy-Memory-30B-A3B-FP8: imported per-tensor E4M3, one dense layer then sigmoid-routed MoE |
+| `hyv3_gguf` | `hyv3` | `gguf_ftype_15` | a llama.cpp Hy-Memory Q4_K_M GGUF, dequantised and recipe-converted |
+| `gemma4_bf16` | `gemma4` | `compressed-tensors` | Gemma 4 31B dense (and E2B/E4B at one rank): sliding-window + proportional-RoPE global attention, GELU, softcapped logits; int4/Q8 codes per 128; assistant (shared-KV), DFlash or DFlash2 drafter; vision |
+| `gemma4` | `gemma4` | *(empty)* | Gemma 4 text from Google's bf16 checkpoint (26B-A4B MoE, dense sizes) with z-lab's DFlash |
+| `gemma3` | `gemma3` | *(empty)* | Gemma 3 dense (TranslateGemma 27B): sliding-window + global attention, GemmaRMSNorm, GELU; bf16 or int8/MXFP4/asym-int4 linears; DFlash2 drafter |
+| `lfm2_bf16` | `lfm2` | *(empty)* | LiquidAI LFM2 / LFM2-VL, bf16, one rank |
+| `embgemma2` | `embeddinggemma2` | *(empty)* | EmbeddingGemma 2: bidirectional encoder with Gemma 4 vision tower and audio conformer, pooled |
+| `nomicbert` | `nomicbert` | *(empty)* | nomic-embed-text-v2-moe: bidirectional post-norm encoder, 8 experts top-2 on every second layer, mean-pooled |
+| `nemotron35_embed` | `mistral3` | *(empty)* | Nemotron 3.5 Embed 1B/8B and Rerank 8B: bf16 text/image retrieval with a Pixtral tower |
+| `qwenimage21` | `qwenimage21` | *(empty)* | Qwen-Image 2.1 / 2.1-Turbo image generator and editor (diffusers pipeline) |
+| `krea2` | `krea2` | *(empty)* | Krea 2 Raw/Turbo: Qwen3-VL-4B tapped layers, 12.9B single-stream DiT, Wan 2.1 VAE |
+| `sdxl` | `sdxl` | *(empty)* | Stable Diffusion XL and finetunes (Illustrious, NoobAI, Pony), single-file checkpoint |
+| `flux3` | `flux3` | *(empty)* | FLUX.3 [klein] and base: image generation and edits from up to ten pictures |
+| `flux3distilled` | `flux3distilled` | *(empty)* | the same plugin under the step-distilled release's own pipeline name |
+| `anima` | `anima` | *(empty)* | Anima Base/Aesthetic/Turbo from the release's three single files |
+| `qwen3asr` | `qwen3asr` | *(empty)* | Qwen3-ASR: Whisper-style audio encoder over a dense Qwen3 decoder, speech to text |
+| `minimaxmusic3` | `minimaxmusic3` | *(empty)* | MiniMax-Music3: Qwen3-8B frame model, depth decoder, flow-matching DiT, DAC vocoder |
+| `breeze` | `breeze` | *(empty)* | Breeze-TTS-2: T5Gemma 2 text encoder, Qwen3 backbone, 16-codebook depth decoder, Qwen3-TTS 12 Hz codec |
+| `fishomni` | `fishomni` | `bsgemm-moe` | Fish Audio drama-3: Qwen3.5-35B-A3B backbone, block-FP8 experts at f32 scales, 13-layer codebook tower, Fish S2 DAC/codec |
+| `minimaxh3` | `minimaxh3` | *(empty)* | MiniMax-H3 video with stereo sound: Qwen3-VL-32B text encoder, 33B DiT, tiled ViT decoder, BigVGAN audio |
+| `ltx2` | `ltx2` | *(empty)* | LTX-2.5 base/distilled: Gemma 4 LM + text connectors, duration head, audio-video DiT, causal video VAE, audio VAE |
+
+## Recipes of record, by family
+
+`$SRC` is a downloaded checkpoint; `$CALIB` is a calibration directory (see
+[[Radiance Conversion]]). Paths are shortened to the recipe name.
+
+| target | command |
+|---|---|
+| Qwen3.8-27B bf16 | `rad-convert $SRC -o q38-27b-bf16.rad --recipe data/recipes/q38-27b-bf16.recipe` |
+| Qwen3.8-27B block fp8 + DFlash2 | `rad-convert $SRC -o q38-27b-fp8.rad --recipe data/recipes/q38-27b-fp8block-df2.recipe --draft-model $DFLASH2 --set quantization_config.quant_method=fp8 --set quantization_config.fmt=e4m3 --set "quantization_config.weight_block_size=128 128" --set quantization_config.activation_scheme=dynamic` |
+| Qwen3.8-27B MXFP4 + DFlash2 | `rad-convert $SRC -o q38-27b-mxfp4.rad --recipe data/recipes/q38-27b-mxfp4-df2.recipe --draft-model $DFLASH2 --set quantization_config.quant_method=fp8 --set quantization_config.fmt=e4m3` |
+| Qwen3.8-27B w4a16 (an int4 release) | `rad-convert $SRC -o q38-27b-w4a16.rad --recipe data/recipes/q38-27b-w4a16.recipe` |
+| Qwen3.8-27B Quark AWQ-MXFP4 | `rad-convert $SRC -o q38-27b-quark.rad --recipe data/recipes/q38-27b-quark-mxfp4-df2.recipe --draft-model $DFLASH2` |
+| Qwen3.8-27B ParoQuant int4/int5/MXFP4 | `rad-convert $SRC -o q38-27b-paro.rad --recipe data/recipes/q38-27b-paro-mxfp4-df2.recipe --draft-model $DFLASH2` |
+| Qwen3.8-27B NVFP4 (compressed-tensors or ModelOpt) | `--recipe data/recipes/q38-27b-nvfp4-df2.recipe` or `q38-27b-modelopt-nvfp4.recipe`; every NVFP4 weight must take a rule (no kernel serves NVFP4) |
+| Qwen3.8-Flash-Next bf16 reference | `rad-convert $SRC -o flashnext-bf16.rad --max-ctx 4096` (no recipe; served with `--weights-disk-tier`) |
+| Qwen3.8-Flash-Next four-bit production | `CALIB=$CALIB rad-convert $SRC -o w4nl64-i8lm.rad --recipe data/recipes/qwen4exp-w4nl64-i8-hc8m.recipe` (GPTQ experts + int8 trunk) |
+| Qwen3.8-Flash-Next uncalibrated four-bit | `--recipe data/recipes/qwen4exp-w4.recipe` (rtn, no calibration) |
+| Flash-Next from an AutoRound int4 release | `--recipe data/recipes/qwen4exp-autoround-i8-hc8m.recipe` |
+| Gemma 4 31B w4a8 (no calibration) | `--recipe data/recipes/gemma4-31b-dense-bf16-w4a8.recipe --set quantization_config.quant_method=compressed-tensors --draft-model $ASSISTANT` |
+| Gemma 4 31B w4a8 + DFlash / DFlash2 | `gemma4-31b-dense-bf16-w4a8-df.recipe` / `-w4a8-df2.recipe`, same `--set` |
+| Gemma 4 31B Q8 grid | `gemma4-31b-dense-bf16-q8-df.recipe` |
+| Gemma 4 26B-A4B + DFlash | `gemma4-26b-a4b-df.recipe`; `-pad768` when the experts were padded to 768 (down may then be block fp8) |
+| Gemma 4 E4B (Nemotron Bolt) | `gemma4-e4b-bf16-df.recipe`, `-i8g-`, `-a4-`, `-w4a8-`, `-w8a8-` variants |
+| TranslateGemma 27B | `translategemma-27b-i8a8.recipe` (calibration container), then `CALIB=$CALIB translategemma-27b-mxfp4-gptq.recipe` (or `-df2`) |
+| GLM-5.3-Flash | `rad-convert $AMD_QUARK -o glm53.rad --recipe data/recipes/glm53-mxfp4.recipe` |
+| Hy-Memory-30B-A3B | `hy-memory-30b-a3b-bf16-fp8.recipe`, `-mxfp4`, `-w4`, `-w8`, `hy-memory-30b-a3b-fp8.recipe`, `hy-memory-30b-a3b-gguf-q4k-to-w4.recipe` |
+| MiniCPM5-2B | `minicpm5-2b-dspark.recipe` (or `-bf16-dspark`) with `--draft-model $DSPARK` |
+| Qwen3.6-35B-A3B | `q36-35b-a3b-fp8.recipe` |
+| Qwen-Image 2.1 | `qwen-image-2.1-fp8.recipe` (Turbo uses the same file) or `-bf16` |
+| Krea 2 | `krea-2-fp8.recipe`, `krea-2-bf16.recipe`, `krea-2-fp8-unscaled.recipe` for a plain-FP8 finetune file |
+| FLUX.3 [klein] | `flux-3-klein-fp8.recipe` |
+| SDXL / finetunes | `sdxl-bf16.recipe --tokenizer <openai/clip-vit-large-patch14>/tokenizer.json` |
+| Anima | `anima-fp8.recipe` (or `-bf16`) + `--add` ×2 + `--config` + `--tokenizer2` + `--arch anima --set anima.variant=aesthetic` |
+| LTX-2.5 | `ltx-2.5-fp8.recipe` (or `-bf16`); full DiT: symlink a folder whose `transformer/` is `transformer_full/` and `--set scheduler.use_dynamic_shifting=1 --set scheduler.shift_terminal=0.1` |
+| MiniMax-H3 / FastH3 | `minimax-h3-fp8.recipe`, `minimax-h3-w8a8.recipe`, `minimax-h3-curve-*` for curve-form checkpoints |
+| MiniMax-Music3 | `minimax-music3-bf16.recipe` |
+| Breeze-TTS-2 | `breeze-tts2-bf16.recipe` |
+| drama-3 | `drama3-fp8.recipe` |
+| nomic-embed-v2-moe | `nomic-embed-v2-moe-bf16.recipe` |
+| Unlimited-OCR | `unlimited-ocr-bf16.recipe` |
+| Qwen3-ASR | no container: serve the checkpoint directory as it ships |
+| EmbeddingGemma 2, Nemotron Embed/Rerank | serve the checkpoint, or convert bf16 with no recipe |
+
+A checkpoint served at its own precision needs no container at all:
+`radiance --model /srv/models/hf/Qwen/Qwen3.8-27B-FP8`. Convert when you want a tier only a container
+has (quantised encodings, a drafter, a second vocab) or a smaller file.
+
+## Family-specific traps
+
+- **`head_dim` is read, never derived.** On the Qwen hybrids it is 256 while `hidden_size /
+  num_attention_heads` is 213; a derived value silently breaks projection widths, KV page size and the
+  attention kernel's band.
+- **A fused projection shards by part, not by contiguous rows.** A tensor-parallel rank owns a slice
+  of *each* stacked projection in a fused `[gate|up]` or `[q|k|v|z]`; a contiguous slice would hand
+  rank 0 the whole gate, and the model would produce fluent wrong text. This is declared through
+  `RadWeightDecl::row_parts`, never inferred from the container.
+- **Gemma 4 26B-A4B:** the routed experts' `gate_up` may be block fp8, but `down` stays bf16 unless the
+  experts were padded to 768 — hence the `-pad768` recipe.
+- **GLM-5.3-Flash:** every weight is served as stored except the latent attention's small `kv_b`,
+  decoded to bf16. Its KV cache is bf16 only (the plugin refuses `fp8` at startup), and at 172 GiB it
+  needs `--tp 8`.
+- **TranslateGemma 27B:** the checkpoint's template takes structured turns naming two languages and
+  refuses the plain turn the server renders at startup. The container therefore serves a plain Gemma 3
+  template, and a client sends on `/v1/completions` the prompt the checkpoint's own template renders.
+- **ParoQuant containers made before 1.5.0 are refused at startup.** Convert again with the old
+  container as `--reuse`; every tensor that did not change is copied.
+- **SDXL carries no tokeniser.** Pass CLIP's `tokenizer.json` beside the single file. A Krea 2
+  finetune shipped as one DiT file in Krea's key names is rewritten into a directory whose
+  `transformer/` holds it (`scripts/krea2_native.py`), with the release's other parts copied in.
+- **Anima** ships as three single files: `--add` the text encoder and VAE, `--config` a Qwen3-0.6B-Base
+  directory (config + tokenizer), `--tokenizer2` ComfyUI's T5 tokenizer, `--arch anima`,
+  `--set anima.variant=…`.
+- **drama-3's codec is `codec.pth`**, a `torch.save` file read as data, never unpickled.
+- **Generators that load LoRA files declare their module aliases**; the checkpoint and a ComfyUI port
+  disagree in ways that load without error and generate garbage. The per-plugin alias table is in
+  `docs/ARCHITECTURES.md` ("Adapter key names of the generators").
+- **Licences travel with the checkpoint, not the engine.** Qwen-Image 2.1 is non-commercial;
+  MiniMax-H3's licence excludes the EU, UK, Republic of Korea and USA; drama-3's community licence
+  needs a commercial licence to serve speech to others, and cloning a real person needs consent;
+  Breeze-TTS-2 is research/non-commercial.
+
+## Related
+
+- [[Radiance Conversion]] — the tool, recipe language and quantisers used above.
+- [[Radiance Serve Options]] — the flags each family's compose file uses.
+- [[Radiance Architecture Plugins]] — writing a plugin for a family not in this table.
+- [[Quantization Production]] — the fleet's campaign and gates.

serverAppConfigs-docs/LLM/Radiance Operations.md

@@ -0,0 +1,240 @@
+---
+type: "howto"
+title: "Radiance Operations — running, reading, restarting"
+description: "Running a radiance server: the container contract, host prerequisites, readiness, ledgers and safe restarts."
+tags:
+  - llm
+  - radiance
+  - operations
+  - howto
+timestamp: 2026-10-30
+---
+
+# Radiance operations
+
+This is the engine-level runbook. The fleet's deployed shape, its unit and its deploy path live on
+[[frank-archon]]; the flags themselves on [[Radiance Serve Options]].
+
+## Install
+
+```sh
+git clone https://codeberg.org/StillDeadcode/radiance.git && cd radiance
+./build.sh                                   # configure, build for the cards present, run fast tests
+./build.sh --no-hip                          # host backend only
+cmake -S . -B build -G Ninja -DRAD_GPU_TARGETS='gfx1201'   # cards this machine lacks
+cmake --install build                        # after -DCMAKE_INSTALL_PREFIX=/opt/radiance at first configure
+```
+
+Needs Linux, git, a C++20 compiler, CMake 3.21+, Ninja; for device kernels ROCm 10.1 or 7.2
+(`hipcc`, `ROCM_PATH` if not `/opt/rocm`).
+
+- **Name the prefix at the first configure.** The default plugin path `<prefix>/share/radiance` is
+  compiled in from that configure; `cmake --install build --prefix DIR` alone gives a tree that looks
+  for its plugins elsewhere.
+- The prefix gets `bin/`, `include/radiance/`, `lib/cmake/radiance/`, `libexec/radiance/`,
+  `share/radiance/` (plugins and recipes) and `share/doc/radiance/`.
+- With ROCm 10.1 the build turns VOPD pairing off because its LLVM 24 pairs in heap-address order, so
+  two builds of one source can differ. The Docker image's compiler has the fix and keeps pairing
+  (faster fp8 prefill GEMMs); declare such a compiler with `-DRAD_HIP_VOPD_REPEATABLE=ON`. ROCm 7.2 is
+  unaffected.
+
+Run without Docker:
+
+```sh
+export RADIANCE_HOME=$PWD/build/radiance_home   # plugins; an install finds its own
+build/bin/radiance --model /srv/models/rad/minicpm5-2b-fp8.rad --port 8000 <flags>
+build/bin/radiance --model StillDeadcode/minicpm5-2b-fp8 --port 8000 <flags>
+```
+
+`--model` also takes a repo id: the server runs `scripts/hfget.sh` into `--download-dir`, serves the
+one `.rad` at the repo's top level, or the directory as a checkpoint when it holds none. A repo with
+several containers is refused, naming them. Every start checks the hub; a start that cannot reach it
+serves the copy a complete earlier fetch left, and a transfer that fails part-way stops startup.
+
+## Container
+
+The published image is `stilldeadcode/radiance`, kernels built for `gfx1201`. The host needs Docker
+and an `amdgpu` driver supporting ROCm 10.1 — **not** ROCm itself.
+
+```sh
+docker run -d --name radiance --network host --init --ulimit memlock=-1 \
+    --device /dev/kfd --device /dev/dri --security-opt seccomp=unconfined \
+    --group-add "$(getent group render | cut -d: -f3)" \
+    --group-add "$(getent group video | cut -d: -f3)" \
+    -v /srv/models/rad:/models:ro \
+    radiance --model /models/m.rad --tp 2 --host :: --port 8000
+```
+
+Each option is load-bearing:
+
+| option | why |
+|---|---|
+| `--device /dev/kfd --device /dev/dri`, `render`/`video` groups | the cards |
+| `--security-opt seccomp=unconfined` | Docker's default profile refuses io_uring (a disk-tier n-gram table's reads) and the NUMA calls the ROCm runtime makes for host memory |
+| `--init` | an init process as PID 1 forwards SIGTERM; the engine finishes its step and releases the cards |
+| `--ulimit memlock=-1` | a host-side pinned n-gram copy can be tens of GiB, far past Docker's default |
+| `--network host` | the server binds the host's addresses directly, IPv4 and IPv6, no NAT |
+
+The image is `FROM scratch` around the stripped install tree: busybox as the only shell, plus `curl`,
+`jq` and a CA bundle for `hfget.sh`; **no Python, no package manager**. Tools (`rad-convert`,
+`rad-info`, `rad-kbench`, `rad-tune`, `rad-schemas`, `rad-tokcheck`) are on `PATH`; recipes are in
+`/opt/radiance/share/radiance/recipes/`. Its entrypoint is `radiance`. It sets
+`GPU_MAX_ALLOC_PERCENT=100` and `GPU_MAX_HEAP_SIZE=100` because the HIP runtime's defaults cap one
+allocation and the heap below what the placement plan hands out.
+
+Build your own with `docker/build.sh` (`-t 'gfx1200;gfx1201'` for targets, `-j` compile jobs, `-E`
+to skip tests, `-s host` to build remotely). The context is `git archive` of the commit, so the image
+holds only committed content. The `rootfs` stage fails the build if any ELF cannot be resolved inside
+a chroot of the tree.
+
+Compose: one file per model plus `common.yaml`, copied to the serving host with an `.env` setting the
+image, models directory (mounted read-only at `/models`), API key, port, user and GPU groups. Every
+model listens on the same port, and most use both cards, so **run one at a time**. The models
+directory must be on a local filesystem that does direct I/O (ext4, XFS, btrfs); on btrfs with
+compression, a container copied into a `chattr +C` directory is stored uncompressed and direct reads
+work.
+
+A plugin of your own is mounted and put first on the search path; nothing in the image changes:
+
+```sh
+-v /srv/radiance-plugins:/plugins:ro -e RADIANCE_HOME=/plugins:/opt/radiance/share/radiance
+```
+
+## Host prerequisites (upstream's two-R9700 checklist)
+
+| item | setting |
+|---|---|
+| firmware | Above 4G Decoding on (Resizable BAR needs it); CSM off (pure UEFI) |
+| kernel command line | `iommu=pt amdgpu.ppfeaturemask=0xfff7ffff amdgpu.ras_enable=0` |
+| power/clock | OverDrive bit needed for `ppfeaturemask`; apply a power cap only while no process holds `/dev/kfd` |
+| memlock | `--ulimit memlock=-1` (container) / `LimitMEMLOCK=infinity` (systemd) when host memory is pinned |
+| two engines, one tmpfs table | `amdgpu.no_system_mem_limit=1`, or the second pin can be refused |
+| storage | a local fs with O_DIRECT for the container; a tmpfs sized for the table for `--ngram-placement ram=DIR` |
+
+Why each: `iommu=pt` keeps DMA identity so the cards can reach each other's memory and pinned host
+pages; `ppfeaturemask` enables the power controls; `ras_enable=0` turns VRAM ECC off (ECC reserves
+about 2 GiB a card). Verify peer access with a rate in `lspci`/the card tables — a missing cell means
+no peer access; check `iommu=pt` and the BAR size. The fleet's own boot-line law for Dankclaw
+(`iommu=pt`, tracked in the repo) is on [[dankclaw-iommu-translation-off]]; Dankserver's cards are a
+different box and its posture is on [[Dankserver System Specs]].
+
+## Readiness: the receipt, then `/health`
+
+Startup states every budget **before anything is allocated**, then loads. `/health` (and
+`/v1/health`, `/ping`) answers only once the load completes, so a bare health gate is honest. While it
+loads, expect 503 for minutes; do not restart over it. `--live` draws the live view (redirected, one
+`greppable key=value` line). `/diagnostics` returns the startup report as JSON — **same-machine-only
+without an API key** — with every option and its source and the hardware read again now.
+
+Read the receipt in this order:
+
+1. `elastic` — what the experts and KV cache share after non-expert weights.
+2. The split: `experts … / kv cache …`, and `unclaimed` if any.
+3. `kv: the pool backs N % of the worst case … M full-length sessions` — the referee for whether the
+   demand signal (`--max-model-len` × `--max-num-seqs`) is funded.
+4. `activation arena` and what each size lends; `prefill staging`; `hardware queues` and their VRAM.
+5. `host_pool` opened/used, `vram_weights`, the PLE or n-gram placement, the vision tower's placement.
+6. `device: the N cards reach each other's memory directly` or the failed pairs;
+   `expert prefetch on/off`; `chunk geometry`; `speculation:`; `chained launches`; `draft trees`.
+7. `card N settled at … free` — memory taken after the budget read comes out of headroom; a warning
+   names the `--gpu-headroom-mib` that clears it.
+
+A boot refusal prints the reason and exits nonzero; when it happens inside `podman run`, the
+container stops at once and the unit's restart policy re-runs the same refusal. A unit in
+`activating (auto-restart)` with a fast nonzero exit is therefore a **preflight/refusal or a boot
+refusal**, not an engine fault: read the journal line and the argv, not the GPU.
+
+## Lifecycle and restart
+
+- SIGTERM finishes the current step and releases the cards. In a container, `--init` forwards it;
+  `podman stop -t N`/`docker stop` is the graceful path.
+- A systemd unit should `exec podman wait` (or run in the foreground) so the script's lifetime is the
+  container's: a crash is a nonzero exit and the restart policy is real. Set `TimeoutStopSec` above
+  the stop grace, or systemd records `Failed with result timeout` on a healthy stop.
+- `Restart=always` rather than `on-failure` for a server whose front end can exit cleanly after a
+  worker fault.
+- Model load is slow (minutes cold). The deployer's LLM wait budget is on [[Deployment Flow]]; a
+  container restart loop during a legitimate load is worse than waiting.
+
+## Two engines on one set of cards
+
+- `ROCR_VISIBLE_DEVICES` picks each engine's cards explicitly.
+- `--weight-prefetch off`: a weight read beside a collective puts a queue switch into every decode
+  step when another server shares the cards.
+- `--ngram-placement ram=DIR`: two engines on separate card sets hold one table in host memory
+  (`DIR/ngram-<id>.tab` in a tmpfs, `MAP_SHARED`, pinned). The `<id>` hashes the container path, size,
+  mtime and the table's place, so a file never serves another model. The first engine fills it; a
+  second verifies samples and only pins it; the last to leave deletes it. The driver counts a shared
+  page against every process that pins it, so the pinned total is twice the table.
+- The engine prefers discrete cards (all of the first one's architecture), integrated only when
+  there is nothing else; on integrated graphics device memory is host memory and the budget leaves an
+  eighth of host RAM to the host.
+
+## Logs and diagnostics
+
+- `-v` / `-vv`; `RADIANCE_LOG` sets the level without a flag. `RADIANCE_LOG_STEPS=1` prints every
+  step's token count (how a chunk geometry is confirmed in practice).
+- The engine writes to stderr. Under a container, the journal holds only the container's own output;
+  a rootless `podman logs` covers only the current instance across recreations.
+- `/metrics` is Prometheus text: engine steps by kind and their seconds, decode/prefill rates,
+  prefill chunks held back, steps held by interleave, draft acceptance and depth, prompt-lookup
+  proposals held, KV bytes and the tokens they address, stream misses and bytes, prefix hits and
+  evictions, preemptions and queue depth. `/stats`,
+  `/graph`, `/sessions` back the dashboard. `scripts/statsdump.py` records `/stats` to JSONL with
+  time and gap, for a machine with no scraper (its key comes from `--api-key` or
+  `RADIANCE_API_KEY`).
+- `RADIANCE_PROFILE_EVERY`, `RADIANCE_DUMP_BUF`/`RADIANCE_DUMP_DIR`, `RADIANCE_OP_ORACLE`(+) and the
+  `RADIANCE_DEBUG_*` family are debug instruments; they are not deployment settings.
+
+## Verifying a deploy (fleet law)
+
+`deploy.sh` green is not proof of what is running. A leftover systemd drop-in or a stale container
+identity can serve a different model at a different shape while `/health` is 200. After deploying
+any recently-benched service:
+
+1. `systemctl show <svc> -p DropInPaths` — must be empty.
+2. `tr '\0' ' ' </proc/$(pidof radiance)/cmdline` (or the container's `Config.Cmd`) — the argv you
+   meant.
+3. The boot ledger — the shape you meant.
+4. A behavioural probe — `/v1/models` for the served id, one chat turn, one tool call, and a vision
+   probe with a solid colour and an exact expected word.
+
+`/v1/models` is an echo on some runtimes; on radiance it is authoritative for the alias, but the
+argv and the ledger are what prove the payload and shape. The three-pin revert discipline for a
+payload swap is on [[Model Swap and Rollback]]; the live deployment is [[frank-archon]].
+
+## Security posture
+
+- `--api-key` requires a bearer token or `X-Api-Key` on every route except `/health`, `/v1/health`,
+  `/ping` and the dashboard; the key is a command-line argument, so it is readable in
+  `/proc/<pid>/cmdline` by any local user of that host. On a single-tenant inference box the network
+  boundary is the real control; do not run one where untrusted local users share the host.
+- `/diagnostics` is same-machine-only without a key, because it prints the whole configuration.
+- `--security-opt seccomp=unconfined` is a genuine widening, required by io_uring and the ROCm
+  runtime's NUMA calls; it belongs on a dedicated model host, not a general-purpose one.
+- Mount model directories read-only; a container with `/models` writable adds nothing a serve needs.
+- A control vector can remove refusal behaviour. Loading one is a safety decision, not a
+  performance one; the flag is `--control-vector`, and `--control-vector disabled` is the default.
+
+## Failure signatures at startup
+
+| what you see | what it is |
+|---|---|
+| unknown/retired flag, or a flag refused by name | a boot refusal — the pin does not have it; change the trio, not the invocation |
+| "no plugin claims `<arch>` at `<quant>`; served at: …" | the (id, descriptor) pair has no plugin; fix the descriptor with `--set`/convert |
+| "reports ABI N, engine M" | rebuild the plugin against this engine |
+| budget refusal / `host_pool` short | the shape does not fit this card/host; lower demand or pin more host RAM |
+| `--ngram-placement ram` refuses | memlock short; raise it or use `auto`/`disk` |
+| `--kv-cache-dtype fp8_v4`/`k6_v4` refused, plugin named | no kernels for this geometry |
+| `--p2p on` refuses, pairs named | the driver has no peer access; use `auto`/`off` |
+| first launch of a never-run kernel faults | too little `--gpu-headroom-mib` for its code object |
+| unit loop, no container, no engine log | a preflight refusal (TTM/memlock/digest/model) re-run by `Restart=always` |
+
+## Related
+
+- [[Radiance Serve Options]] — every knob and its law.
+- [[Radiance HTTP API]] — readiness and the wire.
+- [[Radiance Tools]] — how to corroborate a receipt.
+- [[frank-archon]] — the fleet's radiance unit, guard chain and Eras.
+- [[Model Swap and Rollback]] — the swap discipline this page's verification serves.
+- [[Deployment Flow]] — how a committed change reaches the host.

serverAppConfigs-docs/LLM/Radiance Plugins.md

@@ -0,0 +1,371 @@
+---
+type: "howto"
+title: "Radiance Plugins — kernel libraries and quantisers"
+description: "Writing radiance kernel and quantiser plugins: the ABI, kernel rows, hooks, encodings, and the build path."
+tags:
+  - llm
+  - radiance
+  - plugins
+  - howto
+timestamp: 2026-10-30
+---
+
+# Radiance plugins — kernels and quantisers
+
+Three kinds of `.so` extend the engine, each exporting a few C symbols and linking **nothing of
+radiance's** (they compile against the installed `abi/` headers):
+
+| kind | value | contract |
+|---|---|---|
+| kernel library | `RAD_PLUGIN_KERNEL` (1) | ops for some hardware: a card family, the CPU, an oracle |
+| architecture | `RAD_PLUGIN_ARCH` (2) | one model family at one weight format — see [[Radiance Architecture Plugins]] |
+| quantiser | `RAD_PLUGIN_QUANT` (3) | weights → encoding planes, for `rad-convert` |
+
+In-tree plugins use the same interface a third party does: each builds with `rad_add_plugin` and is
+found and ranked by what it declares, never by its name. `ctest -L oot` rebuilds every bundled
+plugin out of tree against an install.
+
+## Identity and ABI
+
+```c
+#include "rad_abi.h"   /* required; pulls rad_types.h and rad_encoding.h */
+
+uint32_t rad_plugin_abi_version(void) { return RAD_ABI_VERSION; }
+
+const RadPluginInfo* rad_plugin_info(void) {
+    static const RadPluginInfo i = {
+        RAD_PLUGIN_KERNEL, "mykernels", "0.1", "a GEMM or two", "gfx1100",
+    };
+    return &i;
+}
+```
+
+- `RAD_ABI_VERSION` (31 at this commit) is bumped on any shape change in a header a plugin compiles
+  against. Return the macro; a mismatch is refused at load with both numbers printed.
+- `build_target` is printed, not trusted: the targets the library was built for (`"gfx1100"`,
+  `"gfx11-generic"`, several separated by spaces/commas), `"host"`, or `""` for portable. The engine
+  checks the library's actual code objects instead (§"A library for a new card").
+- Optional exports, resolved by exact name: `rad_plugin_open` (return `RAD_E_UNSUPPORTED` to decline
+  this machine and unload), `rad_plugin_close`, `rad_kernel_concurrent`, `rad_kernel_collective`,
+  `rad_plugin_reference`, `rad_kernel_reference`, `rad_kernel_chain`.
+
+## What a kernel library exports
+
+```c
+int                rad_kernel_schema_count(void);
+const RadOpSchema* rad_kernel_schema_at(int i);
+int                rad_kernel_count(void);
+const RadKernelInfo* rad_kernel_at(int i);
+```
+
+A row declares one op implementation and what it can serve:
+
+```c
+static const RadConstraint c_gemm[] = {
+    RAD_CLE("M", 64),              /* M <= 64            */
+    RAD_CDIV("K", 128),            /* K divisible by 128 */
+    RAD_CIN("dtype", "fp8a8 w4a8") /* dtype in that set  */
+};
+
+static const RadKernelInfo k[] = {{
+    .name = "mygemm_m64", .op = "gemm_nt_q", .family = "gemm",
+    .computes = "row-major A times column-major B, fp8 in, bf16 out",
+    .shape    = "M<=64, K%128==0",
+    .dtypes   = "fp8a8, w4a8",
+    .domain   = RAD_DOMAIN_DEVICE,
+    .priority = 50,
+    .constraints = c_gemm, .n_constraints = 3,
+    .launch = my_launch,
+}};
+```
+
+- Operators: `RAD_CEQ`, `RAD_CLE`, `RAD_CGE`, `RAD_CDIV`, `RAD_CIN`. The engine takes bucket
+  boundaries from `LE`/`GE`/`EQ` and resolves once per band; `DIV` and `IN` are not intervals.
+- **Declare only the band you serve.** The engine believes the claim; a kernel that says `M <= 2048`
+  but is correct to 64 is wrong at concurrency.
+- `device.*` keys are matched against what the engine measured about the ranks' cards, not against
+  op parameters: `RAD_CEQ(RAD_FACT_P2P, 1)` means the kernel touches another card's memory. libr4d
+  pairs every collective with a `_host` twin requiring `device.p2p == 0`, so exactly one matches any
+  machine.
+- **Priority breaks ties within one plugin**, highest first, then declaration order. **Between
+  plugins the hierarchy is absolute**: the first library in `--kernels` order with any matching row
+  supplies the op, however specialised a later library's row is. That makes a user's library an
+  override, not a competitor.
+- A derivative parameter (`RAD_DERIVED`) is supplied by the kernel; a constraint on a key the
+  geometry omits does not hold during matching, and the winner's `RAD_C_EQ` writes it into the
+  resolved geometry. `RAD_PROLE_SEQ_CHUNK` marks the parameter the scheduler must cut steps on;
+  `RAD_PROLE_CAPACITY` marks a count the kernel only bounds itself by.
+
+## Operands and extents
+
+`RadArgs.t` is positional in the schema's order; each `RadTensor` carries the extent the caller
+narrowed it to, usually smaller than the buffer. Size loops and grids from those extents, never from
+a declared maximum. Where an op takes `cu_seqlens`, derive the sequence count from its own length
+(`n = len - 1`) and each sequence's rows from `cu[n]` and `cu[n+1]`.
+
+An absent optional operand is a **null `RadTensor.data`**, not a shorter operand count.
+`rad_arg_in(a, i)` fetches one correctly; `rad_tensor_rows`, `rad_tensor_lastdim`,
+`rad_tensor_row_pitch`, `rad_args_geti_or`, `rad_args_getf_or`, `rad_args_gets_or` are the other
+everyday helpers in `rad_plugin.h`.
+
+## Optional hooks
+
+| hook | meaning |
+|---|---|
+| `init` / `fini` | per-resolved-instance state; `RadArgs.instance` hands it back. `init` runs once per (op, band, domain) after selection and before load |
+| `scratch` | bytes of scratch one launch needs; sized at declare from the band's upper bound |
+| `layout` / `relayout` | the kernel stores this weight in its own order, and how the loader produces it from the canonical planes |
+| `unrelayout` | the inverse, so a tool holding stored bytes can compare against a dense oracle |
+| `opd_shape` | size and fill every operand, so a tool can call the kernel cold |
+| `replaces` | the chain of conventional ops this kernel computes in one launch |
+| `tunables` / `tune_valid` | the instantiation axes, and which combinations are legal at a shape |
+
+`replaces` must be **exact** and is compared byte for byte against the chain — a kernel that is only
+close publishes nothing. `tunables` declares axes and legal values, not named variants; the core
+takes the cross product, and `deflt` must be launchable (the loader refuses the plugin otherwise).
+`rad-tune` measures survivors and writes the cache the selector reads; the chosen point arrives as
+ordinary parameters (`rad_args_geti("sk")`), with no variant index.
+
+`layout` answers: `RAD_OK` (own form, `relayout` will produce it), `RAD_E_UNSUPPORTED` (stored as
+is — the common case, and the only form that can be read in place from the file/mapped tier),
+`RAD_E_DTYPE` (this encoding is not mine — a declare-time error), `RAD_E_SHAPE`, or `RAD_LAYOUT_ANY`
+(reads the bytes without interpreting them; claims no storage). A weight only `LAYOUT_ANY` readers
+touch is stored as it is.
+
+## What a launch may do
+
+On a card the `RadStream` is **the engine's own AQL queue**, not a `hipStream_t`. The engine binary
+defines and serves these HIP entry points:
+
+- `hipLaunchKernel` (and `<<<>>>`, `hipLaunchKernelGGL`);
+- `hipMemcpyAsync`, `hipMemcpy2DAsync`, `hipMemsetAsync`;
+- `hipStreamSynchronize` (not from a launch — a launch must not synchronise);
+- `hipMemset` (runs on the engine's queue and waits);
+- `hipGetLastError` / `hipPeekAtLastError`, with HIP's semantics.
+
+**Nothing else in HIP takes an engine stream.** `hipModuleLaunchKernel`, `hipExtLaunchKernel`,
+cooperative launches, `hipLaunchHostFunc`, and libraries that launch through them (rocBLAS,
+hipBLASLt, MIOpen, hipRTC) would hand HIP a queue it does not know. Header-only device code
+(Composable Kernel templates, rocPRIM primitives) compiled into your `.hip` works. `hipEventRecord`
+and `hipStreamWaitEvent` on an engine stream are refused by name.
+
+The engine also reads the code object, which limits a kernel to: **no hostcall** (`printf`/`assert`
+are refused at the first launch), **no dynamic stack**, **≤ 768 bytes of kernel arguments**, **≤ 64
+KiB static+dynamic LDS per dispatch**, and an **uncompressed fat binary**. Device memory a kernel
+needs comes from the `scratch` hook, never `hipMalloc` — the engine's allocator knows what is left.
+
+`rad_plugin.h`'s conversions define the result: the reference contract is f32 accumulation, IEEE
+round-to-nearest-even, **no FMA and no reassociation**. A plugin whose bf16 narrowing truncates
+cannot be checked against the reference. `rad_load_f32`/`rad_store_f32`/`rad_load_bits`/
+`rad_store_bits` read and write any core dtype from a plane.
+
+## Colours of agreement
+
+- `rad_kernel_concurrent(i)` — kernel `i` touches only its operands and scratch, so independent
+  launches may overlap.
+- `rad_kernel_collective(i)` — it exchanges data with other ranks; the engine keeps bulk link
+  traffic away. `2` promises more: the launch ends the hold itself once the other ranks' bytes have
+  landed, through a `link_gate` word it reaches at system scope; returning `RAD_LINK_ENDED` says it
+  did.
+- `rad_kernel_chain(i)` — the kernel can wait in its own prologue instead of behind the packet
+  processor's barrier bit. Chained forms are recognised by `r4dchain` in the name and use a
+  `RAD_CHAIN_REC_BYTES` record at `RAD_CHAIN_KARG_OFF`.
+- `rad_plugin_reference` / `rad_kernel_reference` — an oracle library (or an oracle row beside a
+  device row). A reference library ranks below every unnamed library, is refused for serving unless
+  `--debug-accept-reference-kernels`, and is what `rad-kbench` checks against.
+
+## Defining a new op
+
+Only when `docs/OPS.md` has no name for it. The vocabulary document owns the rules: what a schema
+must declare, and that **the first plugin in hierarchy order to declare an op fixes its schema** — a
+later plugin declaring it differently is refused at load, both named. `ctest -R ops_doc` checks the
+document against the registries, so the rules live there only. An op in `docs/OPS.md` is describable
+by `libref`, and therefore checkable, even when your kernel cannot describe itself; an op you invent
+runs only on a library that implements it.
+
+A schema is positional data, not code:
+
+```c
+static const RadParamSpec params[] = {
+    {"M", RAD_P_INT, RAD_REQUIRED, RAD_PROLE_NONE},   /* the ranged extent */
+    {"n", RAD_P_INT, RAD_REQUIRED, RAD_PROLE_NONE},
+    {"eps", RAD_P_F64, RAD_REQUIRED, RAD_PROLE_NONE},
+    {"dtype", RAD_P_STR, RAD_REQUIRED, RAD_PROLE_NONE},
+};
+static const RadOperandSpec opds[] = {
+    {"x", RAD_OPD_IN, 0}, {"w", RAD_OPD_WEIGHT, 1}, {"y", RAD_OPD_OUT, 0},
+};
+static const RadOpSchema schemas[] = {
+    {"my_op", params, 4, opds, 3, "what it computes, for a human"},
+};
+```
+
+`rad_kernel_schema_count/at` return this table. `RAD_REQUIRED`/`RAD_OPTIONAL`/`RAD_DERIVED` and the
+operand roles (`IN`, `OUT`, `INOUT`, `WEIGHT`, `WTAB`) are the whole vocabulary; the kernel's
+constraints and launch do the rest.
+
+## Quantiser plugins
+
+`abi/rad_quant.h` is the whole contract:
+
+```c
+int                     rad_quant_count(void);
+const RadQuantizerInfo* rad_quant_at(int i);
+
+typedef struct RadQuantizerInfo {
+    const char*           name;       /* what a recipe names */
+    const char*           doc;
+    const RadQuantOption* options;    /* key, RAD_P_INT|F64|STR, default text, doc */
+    int                   n_options;
+    RadQuantEncodingFn    encoding;   /* the encoding this quantiser would write */
+    RadQuantRowBlockFn    row_block;  /* rows per quantize() call; 0 = whole weight */
+    RadQuantizeFn         quantize;   /* logical rows -> planes */
+    RadQuantDecodeFn      decode;     /* optional, only for a scheme of its own */
+} RadQuantizerInfo;
+```
+
+The flow for each weight a recipe rule picks: ask `encoding` (given name, shape, and — at declare —
+no movement group), let the architecture declare against that encoding, then hand the weight over in
+row blocks, so a 51 GB table streams. Other calibration data is the quantiser's own option
+(`calib=DIR`); the output is **canonical planes only**, so every kernel that accepts the encoding and
+the core's decoder can read it — that is how the converter measures error without knowing the method.
+
+A quantiser's own scheme must be decoded by its `decode` hook (the oracle, `rad-kbench` and the
+error report call it), and must return `RAD_E_UNSUPPORTED` for an encoding it did not write. Serving a
+new encoding also needs a kernel library whose `layout` accepts it and an architecture that declares
+the weight at it — a new format is a new kernel row plus a plugin, never a converter-only change.
+
+Two plugins naming the same quantiser are refused, both named.
+
+### The encoding model (`abi/rad_encoding.h`)
+
+An encoding is the code type, an optional codebook, the grouping of scales/zero points/minimums, and
+a transform. It is **data, not an enum**, so a plugin can invent one. The core can size, slice and
+place any encoding from the struct alone, and decodes the `plain` and `affine` schemes itself:
+
+```
+q  = code                                   codes TILED, block [1x1], or [1xV] with a grid
+q  = grid[code][k]                          grid TABLE [entries x V]
+q  = table[code]                            table TABLE [1 x entries]
+q  = -q if the sign bit is set              signs TILED u1
+q  = q - zero                               zero TILED
+v  = q * scale * scale.1 * ...              every scale level
+v  = v - min * min.1 * ...                  a minimum in the value's domain
+w  = v . T^-1                               the transform, per row over columns
+```
+
+So NVFP4 is fp4 codes, e4m3 scale per 16, f32 `scale.1` per tensor; Q4_K is u4 codes, u6 scale and
+u6 min per 32 each times an f16 per 256; IQ2_XXS is u8 codes per 8 into a 256×8 grid, a sign bit per
+value and two scale levels.
+
+- Planes are **TILED** (cover the weight, element `(i,j)` covering a `[br,bc]` block; `0` extent means
+  the whole extent) or **TABLE** (a codebook, grid, rotation parameter; every slice carries it).
+- Sub-byte elements are a bit stream along the row, lowest bits first; every row starts on a byte.
+- Transforms: `""`, `fwhtN` (Hadamard before quantisation; the `1/N` lives in the stored scale),
+  `perm` (GPTQ act-order, a TABLE `t.perm`), `givens` (ParoQuant rotations: `t.pairs`, `t.angle`,
+  optional `t.scale`, per-part `t.part`, composed `t.dense`).
+
+A kernel only *rearranges* planes (`relayout`); choosing numbers is the quantiser's job. That split
+is why a container never has to be reconverted when a kernel library changes its layout.
+
+## Build, install, search path
+
+```cmake
+cmake_minimum_required(VERSION 3.21)
+project(mykernels LANGUAGES CXX HIP)     # CXX alone for a host-only plugin
+find_package(radiance REQUIRED)
+rad_plugin_defaults()                    # C++20, optimising build unless asked otherwise
+rad_add_plugin(mykernels kernels SOURCES rows.cpp HIP_SOURCES kernels.hip)
+```
+
+The kind is `kernels`, `architectures` or `quantizers` — the directory the engine loads it from.
+Only `HIP_SOURCES` compile as HIP: for `RAD_GPU_TARGETS` or a `GPU_TARGETS` list, with `-DNDEBUG`
+outside Debug, uncompressed, against the HIP runtime the installed radiance found. `find_package`
+also sets `RADIANCE_ABI_VERSION` (to fail at configure time rather than at `dlopen`),
+`RADIANCE_HOME`, `RADIANCE_KERNELS_DIR`, `RADIANCE_ARCHITECTURES_DIR`,
+`RADIANCE_QUANTIZERS_DIR`. `cmake --install` puts the plugin under the home of the radiance you built
+against; `RAD_PLUGIN_INSTALL_HOME` overrides that.
+
+**`$RADIANCE_HOME` is a search path**: homes separated by `:`, the first winning; each may hold
+`kernels/`, `architectures/`, `quantizers/`, and an earlier home's same-name file shadows a later
+one. Your plugins go ahead of an installation without copying it:
+
+```sh
+RADIANCE_HOME=$HOME/radiance-plugins:/opt/radiance/share/radiance radiance --model m.rad
+# in the container:
+-v /my/plugins:/plugins -e RADIANCE_HOME=/plugins:/opt/radiance/share/radiance
+```
+
+The **kernel hierarchy** is `--kernels` (or `$RADIANCE_KERNELS`): `--kernels mykernels,libr4d` puts
+yours before libr4d. An entry matches a plugin's name or its file (`libr4d`, `r4d`, `libr4d.so`, a
+path). An entry matching nothing is warned about by name. With no list, every library found loads,
+reference last.
+
+## A library for a new card
+
+The engine's own device code is compiled for every AMD family's generic target
+(`gfx9-generic`, `gfx9-4-generic`, `gfx10-1-generic`, `gfx10-3-generic`, `gfx11-generic`,
+`gfx12-generic`) beside the build's targets, so one engine binary starts anywhere. What it computes
+there depends on the installed kernel library.
+
+- Configure with `-DRAD_GPU_TARGETS=gfx942` (or `gfx11-generic`). At load the engine asks the runtime
+  which targets the card accepts, in order (chip, then family: `gfx1036`, then `gfx10-3-generic`),
+  with the card's modes (`gfx942:sramecc+:xnack-`), and takes the best fit in the fat binary.
+- **A library with nothing for this card is left out, not failed**: its device rows leave selection,
+  its host rows stay, and the log says what it carries versus what the card runs. One installation
+  can carry a library per card. `rad_plugin_open` may decline explicitly with `RAD_E_UNSUPPORTED`.
+- The engine uses discrete cards if any exist (all of the first one's architecture), integrated only
+  when there is nothing else; `ROCR_VISIBLE_DEVICES` chooses explicitly. On integrated graphics,
+  device memory is host memory and the budget leaves an eighth of host RAM to the host.
+- **fp8 bytes are OCP E4M3** (`e4m3fn`, bias 7, max 448) wherever the engine reads them: container
+  weights, an fp8 KV cache, activations. A card whose fp8 is `e4m3fnuz` (gfx942) converts in its
+  `relayout` hook (the codes keep their bits except `0x80` and decode to half the value, so the scale
+  doubles) and keeps activations/KV in OCP bytes, or converts at its own boundaries.
+- **Tensor parallelism is ops you supply.** `all_reduce`, `all_gather` and the fused collectives are
+  rows like any other; the engine inserts none and assumes no link.
+
+## Checking your work
+
+```sh
+rad-kbench --kernels mylib,libr4d,libref --report mylib.md --bench
+```
+
+- It replays a recorded fixture of `libref`'s answers over real model geometries
+  (`$RADIANCE_HOME/kernels.rkb`, plus `kernels_moe.rkb`), running every library with a row for a
+  case. Correctness uses sampled values **and** the full-tensor norm **and** seeded projections;
+  completeness is the op-by-library matrix; speed is timed on the same bytes.
+- Red zones: every device allocation carries a poisoned 256-byte margin, verified after the launch.
+- Fusion: a row with `replaces` is run against its named chain, out of your library, compared with
+  `memcmp`.
+- A skip is named, not failed — usually a missing `opd_shape`. `--strict` also fails a stale fixture.
+- `rad-avx-check --ref libref.so --avx mylib.so` checks a **host** library without a card, op by op,
+  at every ISA level and with parallel regions forced onto several threads (for a library exporting
+  `avx_isa_force`/`avx_par_force`).
+- `rad-schemas mylib.so libref.so` prints every schema disagreement at once (it exits 1 on any), and
+  works on a plugin the engine cannot load.
+- `--debug-selection` prints which row took each band and which constraint refused each other
+  candidate; `/graph` on a running server shows the declared graph and which declarations the run
+  phase issued.
+- `rad-info --plugins` lists every plugin on the search path as the engine loads it: kind, version,
+  declarations, reference status, whether its device code runs on this card, and its file.
+
+## A worked example to copy
+
+`plugin/spechub/` is the smallest real kernel plugin in the tree: two ops (`mtp_couple_propose`,
+`mtp_couple_verify`), each with a device row and a host reference row. Read it in this order:
+
+- `exports.cpp` — the whole export block: info, ABI, schema count/at, kernel count/at, plus
+  `rad_kernel_reference` classifying device rows 0 and host rows 1 so the reference gate does not
+  refuse the device rows.
+- `spechub.cpp` — the rows, their constraints and their argument readers.
+- `spechub.hip` / `ref_spechub.cpp` — the device launch and the host oracle.
+- `CMakeLists.txt` — the `rad_add_plugin` call, the optional `enable_language(HIP)`, and
+  `-ffp-contract=off` because an FMA in the ordered FP64 mass sums would move a token off the host
+  reference.
+
+## Related
+
+- [[Radiance Architecture Plugins]] — the third plugin kind, one level up.
+- [[Radiance Conversion]] — how a quantiser is driven by a recipe.
+- [[Radiance Tools]] — `rad-kbench`, `rad-tune`, `rad-schemas`, `rad-avx-check` in full.
+- [[Radiance Serve Options]] — `--kernels`, `--radiance-home`, the tuning aids.

serverAppConfigs-docs/LLM/Radiance Serve Options.md

@@ -0,0 +1,240 @@
+---
+type: "reference"
+title: "Radiance Serve Options — every flag and the law behind it"
+description: "Every radiance serve flag grouped by task, with its default and the law that decides when to move it."
+tags:
+  - llm
+  - radiance
+  - reference
+  - serving
+timestamp: 2026-10-30
+---
+
+# Radiance serve options
+
+`radiance --help` prints this table. Every numeric option takes a whole number (not `12k`, not
+`auto`, unless the option names those); a malformed value exits 2. Upstream `main` at 2026-10-30;
+the fleet is pinned older — see the pin note on [[Radiance Engine]].
+
+The one thing to read before any table: **the engine derives the VRAM split; no flag sets either
+pool.** The startup receipt prints every term before anything is allocated.
+
+## Model, plugins, container
+
+| flag | default | meaning |
+|---|---|---|
+| `--model PATH` | required | a `.rad` container, a checkpoint served as it stands, or a Hugging Face repo id (`owner/name`) fetched before startup |
+| `--download-dir DIR` | `$XDG_CACHE_HOME/radiance/hf`, else `~/.cache/radiance/hf` | where a repo id is fetched |
+| `--radiance-home DIRS` | compiled prefix | plugin homes, `:`-separated, first wins |
+| `--kernels LIST` | every library found | kernel libraries, `,`/`:` separated, first wins; a library not named is not loaded |
+| `--served-model-name NAME` | container's name | the id `/v1/models` lists and responses carry; a request may name any model |
+| `--override-chat-template PATH` | container's | a Jinja template served instead; an unreadable or empty file stops startup |
+| `--generation-config PATH` | none | sampler defaults from a `generation_config.json` (else `<model>.generation.json`, then the container's `generation.*` metadata) |
+| `--control-vector off\|on\|disabled` | `disabled` | load the GGUF control vector found for the model and start requests off/on; `disabled` declares no projection and refuses a request that asks |
+| `--api-key KEY` | none | require `Authorization: Bearer KEY` or `X-Api-Key: KEY`, except `/health`, `/v1/health`, `/ping`, the dashboard |
+| `--mm-max-patches N` | `auto` (16384 with a vision tower) | patches one encoder pass carries: the largest image, 16×16 px a patch (14×14 on GLM-5.3). `0` serves text only and keeps the tower off the card |
+
+## Batch shape, context, scheduler
+
+| flag | default | meaning |
+|---|---|---|
+| `--max-num-batched-tokens N` | 2048 | the largest token count one step may carry; on MoE the knob that sets prefill chunk size, and the activation arena's size |
+| `--max-num-seqs N` | 256 | the most sequences one step may carry; the cache's **demand**, never its funding |
+| `--max-model-len N` | 0 = model's training context | context bound per request. Longer is a 400, never a truncation. Reserves nothing per request; it is the worst case the pool is sized against |
+| `--busy-prefill-chunk auto\|off\|N` | `auto` | a prefill chunk's share of a step that has other work: `auto` leaves decode rows and waiting requests room, `off` takes the whole step, `N` caps it |
+| `--prefill-interleave N` | 0 (off) | decode-only steps after each chunk taken beside decoders, at most 64; buys decoder cadence with the prompt's TTFT |
+| `--prefill-span N` | 1 | checkpoint intervals a prefill **alone in its step** may carry, from its second interval; needs `--max-num-batched-tokens` of N intervals. Every other step is planned at the shared budget; checkpoints inside a span are not written |
+| `--preempt-wait-ms N` | 5000 | how long a preempted prefill waits for the whole-recompute gate before reclaiming blocks and preempting lower-priority work; `0` falls back next step |
+| `--rope-scaling SPEC` | the checkpoint's | `yarn:FACTOR[:ORIG]` writes static YaRN over the container's own rotary keys with no reconvert; `none` takes a checkpoint's YaRN off |
+| `--max-queued-requests N` | 8 × `--max-num-seqs` | admitted, unfinished requests; past it the answer is 429 |
+| `--retry-after S` | 1 | the `Retry-After` a 429 carries |
+
+## Memory and placement
+
+| flag | default | meaning |
+|---|---|---|
+| `--placement MODE` | `auto` | `all_vram`, `layer_offload`, `expert_tiered`, `auto` |
+| `--gpu-headroom-mib N` | 96 | VRAM left unclaimed on every card; too little makes the first launch of a never-run kernel fail inside HIP |
+| `--host-pool-mib N` | 0 | pinned host tier for weights, **per rank**; also the KV lever under expert tiering (below) |
+| `--host-headroom-mib N` | 4096 | host RAM left free when startup pins memory; a snapshot store's floor takes the shortfall back only down to 1 GiB |
+| `--kv-cache-dtype T` | `bf16` | `bf16`, `fp8`, and on Qwen3.8-27B/-Flash-Next `fp8_v4` (V int4) or `k6_v4` (K int6 + V int4). A real quantisation; GLM-5.3 is bf16-only |
+| `--state-cache-dtype T` | `f32` | `f32` or `f16`: the recurrent delta-net state cache; f16 halves state pool, checkpoints and traffic and is lossy — `qwen35` plugins only |
+| `--attn-prefill-fp8 MODE` | `off` | `off`, `qk`, `pv`, `both`: prefill attention products in E4M3 over an `fp8`/`fp8_v4`/`k6_v4` cache; lossy, Qwen3.8-27B only where kernels exist |
+| `--kv-resident N` | 0 (off) | stream the sparse attention group's KV from pinned host RAM, keeping an N-token ring on the card. Raised to 32768 and to one decode step's selection; ignored with a warning where unsupported |
+| `--embedding-placement ram\|vram` | `ram` | token embedding pinned in host memory and read in place, or on the card. A tied head or a host that cannot pin keeps it on the card; an explicit `ram` refuses instead |
+| `--vision-tower-placement vram\|ram` | `vram` | encoder towers on the card, or pinned in host memory and read once per image, giving their VRAM to the KV cache |
+| `--ngram-placement auto\|vram\|ram\|disk` | `auto` | where an n-gram embedding table lives: all of it on every card, a pinned host copy the card reads, or rows from the container through a row cache. `auto` takes the first that fits. `ram=DIR` makes the pinned copy a tmpfs file shared by two engines |
+| `--weights-disk-tier` | off | read weights that fit neither VRAM nor the host pool from the container with O_DIRECT; every step is paced by the drive |
+| `--row-prefetch on\|off` | on | ask for a host gather's pages before its kernel; output-neutral |
+| `--mapped-rewarm on\|off` | on | at startup, re-ask the page cache for a mapped table's pages an earlier run left behind |
+| `--deterministic` | off | pin placement to the loader's plan so the resident set cannot move under the heat engine; it does **not** make output bit-identical across batch positions |
+
+## KV cache, host tier, checkpoints
+
+| flag | default | meaning |
+|---|---|---|
+| `--prefix-cache-mib N` | 0 (off) | copy finished conversations' KV to host memory so a return restores instead of re-prefilling; LRU when full; off while a KV group streams |
+| `--prefix-cache-ttl S` | 3600 | seconds an unused cached conversation is kept; `0` keeps it until memory is wanted |
+| `--no-prefix-cache` | cache on | disable prefix caching |
+| `--checkpoint-interval N` | 2048 | linear-attention state checkpoint interval, in tokens; also clamps a hybrid model's prefill chunk |
+| `--checkpoint-slots N` | `auto` | snapshots held **in total**, in pinned host memory; `auto` = per-policy need for each sequence within an eighth of host RAM; `0` keeps none |
+| `--checkpoint-policy NAME` | `geometric-backoff` | also `tip-only`, `keep-all`; retention only — writes happen at every interval unless a span skips them |
+| `--prefix-resume exact\|nearest` | `exact` | `exact` resumes only where a cold prefill starts a chunk, so a hit answers bit-identically; `nearest` takes the deepest snapshot, up to a chunk less re-prefilled |
+
+## Speculation
+
+| flag | default | meaning |
+|---|---|---|
+| `--num-speculative-tokens N` | `auto` (the drafter's own point; 0 with no head) | speculative window; 0 disables. The ceiling under `adaptive` |
+| `--spec-ngram K` | 0 (off) | prompt lookup alone, up to K drafts a step from the sequence's own n-grams; leaves a container's draft head out; sets the window to K |
+| `--spec-ngram-min N` / `--spec-ngram-max N` | 2 / 4 (max 15) | pattern lengths prompt lookup tries, longest first |
+| `--draft-window fixed\|adaptive` | `fixed` | `adaptive` follows each sequence's acceptance and the measured step cost |
+| `--draft-short-from N` | `auto` | a block drafter that declares a shorter block runs it from N decoding sequences; `0` never; `auto` = never for a chain, from 2 for a tree |
+| `--draft-coupling off\|on` | off | sample each proposal from the drafter's own shortlist and couple the target's draw to it; same output law, different tokens for a fixed seed, pays at high temperature |
+| `--draft-model PATH` | — | **not implemented** at serve: merge the drafter at convert time (`rad-convert --draft-model`) |
+
+## Tensor parallelism and collectives
+
+| flag | default | meaning |
+|---|---|---|
+| `--tp N` | 1 | tensor-parallel ranks; declare divides dimensions by world size and inserts no collective |
+| `--tp-wire exact\|wht6` | `exact` | `wht6` is lossy: rotated 6-bit at 2 ranks, tiered int8 over a two-shot at 3–4, unavailable at 8 |
+| `--tp-wire-min-kb N` | 128 | smallest all-reduce (KiB) the lossy wire compresses; keeps single-sequence decode exact |
+| `--tp-attn-split heads\|rows` | `heads` | a prefill step's attention split: `rows` gives each rank all heads of its rows plus an all-gather; faster where attention dominates, not bit-identical, plugin-gated |
+| `--weight-prefetch on\|off` | on | read a GEMM's weight beside the collective before it; `off` when another server shares the cards |
+| `--p2p auto\|on\|off` | `auto` | `auto` asks the driver; `on` refuses to start without peer access; `off` exchanges through pinned host memory |
+
+## LoRA adapters
+
+| flag | default | meaning |
+|---|---|---|
+| `--lora NAME=PATH` | none | serve an adapter: a PEFT directory as model NAME (text), or a `.safetensors` a request names in `loras` (image/video). Repeatable |
+| `--lora-dir DIR` | none | generators: any `NAME.safetensors` under DIR may be requested |
+| `--max-loras N` | count of `--lora`, 2 with `--lora-dir` alone | adapter slots resident at once |
+| `--max-lora-rank R` | 16 text / 64 generator | slot rank; a larger text adapter is refused, a larger generator combination truncated |
+| `--lora-target-modules LIST` | the adapters' own | HF module leaf names adapters may target; needed with `--max-loras` and no `--lora` |
+
+## Request defaults and bounds
+
+A request that names a field wins. These set defaults for requests that do not.
+
+| flag | default | meaning |
+|---|---|---|
+| `--temp F`, `--top-k N`, `--top-p F`, `--min-p F` | from the model | sampler defaults; `top_k` 0 is the 1024 most likely candidates |
+| `--typical-p F` | 1 | typical sampling |
+| `--presence-penalty F`, `--frequency-penalty F` | 0 | range [-2, 2] |
+| `--repetition-penalty F`, `--repeat-last-n N` | 1, 64 | `-1` looks back over the whole context |
+| `--dry-multiplier F`, `--dry-base F`, `--dry-allowed-length N`, `--dry-penalty-last-n N` | 0, 1.75, 2, -1 | DRY; multiplier 0 is off |
+| `--dry-sequence-breakers JSON` | `["\n", ":", "\"", "*"]` | at most 16 strings |
+| `--xtc-probability F`, `--xtc-threshold F` | 0, 0.1 | range [0, 1] |
+| `--default-max-tokens N` | `auto` | `max_tokens` for a request that sends none; `auto` is what the context leaves |
+| `--max-tokens-cap N` | 0 (none) | a larger request is **cut to it**, not refused; reply ends `finish_reason: "length"` |
+| `--max-n N` | max(8, `--max-num-seqs`) | the largest fan-out `n` |
+| `--max-stop-strings N`, `--max-stop-bytes N` | 64, 4096 | stop-string limits |
+| `--chat-template-kwargs JSON` | none | variables for the template on every chat request; a JSON object or `@FILE`; repeatable |
+| `--reasoning-effort S` | template's own | default `reasoning_effort`; passed to the template verbatim, so the legal set is the template's |
+| `--reasoning-effort-map MAP` | none | `FROM=TO,...`, checked against the template at startup; `none`/`minimal` turn thinking off |
+| `--thinking-token-budget N` | none | default budget: at N reasoning tokens the block is closed and the answer follows |
+| `--reasoning-format auto\|none` | `auto` | `auto` returns reasoning as `reasoning_content`; `none` leaves it inline |
+
+## Server and HTTP
+
+| flag | default | meaning |
+|---|---|---|
+| `--host ADDR` | 0.0.0.0 | listen address |
+| `--port N` | 8000 | listen port |
+| `--http-threads N` | one per hardware thread, at least 4 | workers kept while idle |
+| `--read-timeout S` | 30 | a client may go silent this long while sending |
+| `--write-timeout S` | 600 | one write to a client may block this long |
+| `--keep-alive-timeout S` | 5 | idle connection hold |
+| `--max-body-mib N` | 512 | largest body; images and video ride base64 inside it |
+| `--no-cors` | CORS on | send no CORS headers |
+
+## Media preprocessing
+
+Each replaces what the container's preprocessor config states, and does nothing for a model that
+takes no media.
+
+| flag | default | meaning |
+|---|---|---|
+| `--image-min-pixels N`, `--image-max-pixels N` | container's | the band an image resizes into; one encoder pass still bounds the top |
+| `--video-min-pixels N`, `--video-max-pixels N` | container's | same, for a video's sampled frames together |
+| `--video-fps F` | container's | frames sampled per second |
+| `--video-min-frames N`, `--video-max-frames N` | container's | frame-count bounds |
+| `--video-max-frame-tokens N` | container's | the most prompt tokens one frame becomes; `0` no cap; a token-budget processor takes 70/140/280/560/1120 |
+| `--image-tokens N` | container's | a token-budget processor's image budget (Gemma 4's fixed set; GLM-5.3's ceiling ≤ `--mm-max-patches`/4) |
+| `--max-source-pixels N` | 67108864 | largest input the decoder accepts before resize |
+
+## Quality measurement (instead of serving)
+
+| flag | meaning |
+|---|---|
+| `--kld-record DIR` | run `--kld-corpus` through prefill and write this model's per-position log-probabilities to DIR: the reference |
+| `--kld-ref DIR` | score this model against the reference: mean/median/percentile KL and top-1 agreement |
+| `--kld-corpus FILE` | JSONL, one `{"prompt", "score_from"?, "source"?}` a line; `--kld-record` only |
+| `--kld-out FILE` | the report as JSON; `FILE.rows` holds every position's numbers |
+| `--kld-lora NAME`, `--kld-lora-sources PREFIX` | score through a served adapter, all documents or a `source` prefix |
+
+The mode turns speculation and the prefix cache off and refuses `--num-speculative-tokens > 0`.
+See [[Radiance Conversion]] for the corpus and the scoring discipline.
+
+## Diagnostics
+
+| flag | meaning |
+|---|---|
+| `--debug-graph` | print the declared graph, every op's resolved kernel and why, then exit |
+| `--debug-selection` | the selection table only |
+| `--debug-placement` | the placement plan only |
+| `--debug-tape-dump` | the first recorded pass of each kind (`TAPEDUMP` on stderr, every rank) |
+| `--debug-draft-oracle FILE` | verify-path test: prompt lookup proposes from FILE with 30 % corrupted drafts; the text must not move |
+| `--debug-accept-reference-kernels` | serve even with `libref` rows — not a production configuration |
+| `--profile-ops` | per-op device timing |
+| `--expert-stats FILE` | routing counts per layer and step kind, JSON every 30 s and at shutdown |
+| `--expert-profile FILE` | rank the expert plane by a stats file's `decode` counts instead of the container profile |
+| `--expert-pin-top F` | pin the hottest fraction F of routed experts so heat never demotes them |
+| `--draft-expert-placement host\|auto` | pin a draft head's experts in the host pool before any strategy runs |
+| `--live` | the terminal live view; redirected, one `key=value` line |
+| `-v`, `-vv` | verbose, very verbose |
+
+## The laws behind the knobs
+
+**The split is derived, and the receipt is the referee.** Non-expert weights come off the top of
+the card; the remainder (`elastic`) is shared by the expert slab and the KV cache. `--max-model-len`
+× `--max-num-seqs` is the cache's **demand**; the funding is `elastic − expert_floor`. Under a
+lending cache (`auto`/`expert_tiered`, no `--deterministic`, VMM present) the cache takes what it
+can address and lends unused blocks back to experts. **Every MiB added to `--host-pool-mib` lowers
+the expert floor one-for-one and the cache takes it** — until the cache already holds its whole
+ceiling, where the extra host room shows up as unclaimed. Read the boot line
+`kv: the pool backs N % of the worst case …`, never arithmetic.
+
+**The activation arena comes off the budget before the split**, so both sides pay for
+`--max-num-batched-tokens`. While steps are small most of it is lent back to the expert slab; a
+larger step reclaims it synchronously. That is why a bigger chunk can cost decode residency. On a
+hybrid model the chunk is clamped to `--checkpoint-interval` before the token budget, and startup
+prints which bound applies.
+
+**The KV dtype law.** `bf16` and `fp8` are engine-wide; `fp8_v4`/`k6_v4` are narrower layouts whose
+kernels exist only for particular attention geometries, and the server refuses them elsewhere by
+plugin name. A drafter's cache stays fp8. Recurrent state and the QSA indexer keep their own widths.
+
+**The slot law.** A slot costs recurrent state, block tables at full context, snapshot-store RAM
+and all-reduce rows — never speed by itself. Raising `--max-num-seqs` raises demand; whether it is
+funded is the resolver's answer.
+
+**The spec law.** Measure speculation by step time, not tok/s. Prompt lookup pays where the reply
+copies its context and costs a verify row elsewhere. `adaptive` moves the depth only after eight
+steps and a 2 % gain, because a new window is a new pass shape.
+
+**Batch shape is part of the configuration.** A different chunk, span or spec depth changes the
+summation order, so greedy text can differ at a tie. Reproducibility holds per configuration, never
+across one.
+
+**Unknown request fields are ignored; known-but-unimplemented fields are guarded** (accepted at
+their off value, refused otherwise) — the list is on [[Radiance HTTP API]].
+
+## Related
+
+- [[Radiance HTTP API]] — what the flags expose on the wire.
+- [[Radiance Operations]] — the container contract and readiness.
+- [[Radiance Plugins]] — how a flag reaches a plugin (`RadBuildCtx`).
+- [[frank-archon]] — the measured deployment of this resolver on two R9700s.

serverAppConfigs-docs/LLM/Radiance Tools.md

@@ -0,0 +1,322 @@
+---
+type: "reference"
+title: "Radiance Tools — inspect, check, tune, tokenise"
+description: "The radiance inspection, measurement and tokeniser tools, with the flags and verdicts each one reports."
+tags:
+  - llm
+  - radiance
+  - tools
+  - reference
+timestamp: 2026-10-30
+---
+
+# Radiance tools
+
+Six binaries and `rad-export` are installed beside `radiance`; `rad-avx-check`, `r4d_selftest` and
+`rad-chatparse` live in the build tree. Every tool that loads plugins honours `RADIANCE_HOME`,
+`RADIANCE_KERNELS` and `--kernels` as the engine does. A numeric option takes a whole number: `12k`,
+`auto` (where the option does not name it) or an empty value exits 2.
+
+`rad-convert`, `rad-kbench` and `rad-tune` run the engine's startup with a different last step: the
+same plugins, in the same hierarchy, through the same declare phase, so they resolve kernels exactly
+as a serve would.
+
+## `rad-info` — what a container holds
+
+```
+rad-info [options] <model.rad>
+rad-info --plugins [--home DIRS]
+
+  -v, --verbose      list every tensor, not just the per-unit summary
+      --meta         metadata table, then stop
+      --vocab        vocab summary and normaliser chain, then stop
+      --profile N    top N expert-profile entries (default 20, 0 = all)
+      --recipe       the recipe the container was converted with, then stop
+      --plugins      every plugin the search path offers, as the engine loads it
+      --home DIRS    search path for --plugins (default $RADIANCE_HOME)
+```
+
+It opens the container read-only and loads no plugins (except for `--plugins`): a container names no
+kernel library. Output is aligned; byte counts are binary units and each count says what it counts.
+
+```sh
+rad-info m.rad                    # format, architecture, model, source quant, recipe, sections
+rad-info --recipe m.rad           # the recipe, expanded, one rule a line
+rad-info -v m.rad | head -40      # every tensor, encoding, shape and what made it
+rad-info --plugins                # kind, version, declarations, reference, device fit, file
+```
+
+The section table (strings, metadata, directory, planes, encodings, vocab, profile, data) is the
+container's layout; the metadata table, the weight directory grouped by layer and movement unit, the
+vocab, and the expert profile follow.
+
+## `rad-kbench` — correctness, completeness, speed
+
+```
+rad-kbench [options]                       replay the recorded fixture (no model)
+rad-kbench -m <model.rad> [options]        check against a real container
+rad-kbench -m <model.rad> --record FILE    record a new fixture
+rad-kbench -m <model.rad> --skeleton FILE  write a metadata-only copy to record from
+```
+
+**Replay** needs no model, weights or card. Geometries, seeds and `libref`'s reduced answers are
+checked in as `data/kernels.rkb` (dense model, block drafter, gqa 6) and `data/kernels_moe.rkb`
+(routed MoE, MTP head, gqa 8). `--fixture` is repeatable; `ctest` runs three replays — the other
+libraries against each recording, and `libref` against both at tolerance zero (its regression test,
+since a live comparison moves both sides together).
+
+Each library is asked for its row as the engine would ask it with that library alone: same
+constraints, derived parameters and weight-encoding filter, so a row the engine would pass over does
+not run here either (`--all-rows` runs every row whose constraints hold).
+
+**Against a container** (`-m`), weight operands are read from it: each weight's canonical planes,
+which the kernel gets through its own `layout`/`relayout` hooks and the oracle gets decoded, and
+every expert of a weight table. This is the only test of the artifact and of the layout hooks.
+
+**Recording** takes the reference's answer once at every point of every declared op, whichever
+library resolved it. Weights are drawn from a seed, not read from the container, so any machine can
+reproduce the reference and no card is needed. A skeleton (tables intact, data a hole) records the
+same fixture in a few megabytes.
+
+```sh
+rad-kbench -m model.rad --skeleton model.skel.rad
+rad-kbench -m model.skel.rad --max-cases 1 --record data/kernels.rkb
+rad-kbench -m moe.rad        --max-cases 1 --record data/kernels_moe.rkb
+```
+
+### Verdicts
+
+- An output is compared on sampled positions, the full-tensor norm, and 32 seeded random-sign
+  projections (which estimate rel_l2 to about 25 %). The largest of the three is the error.
+- Tolerances are per op and activation dtype (`dtype`, else `q_dtype`, else `kv_dtype`). Copies,
+  gathers, scatters and cache stores are exact. Every op in the fixtures has a row of its own.
+- Ids are compared exactly; where the contract leaves a tie open (`sample_argmax`, `router_topk`,
+  `sample_topk`), an id indexing a value equal to the reference id's is accepted.
+- A case whose reference and output are both identically zero checked nothing: **vacuous**, not
+  passed.
+- A failure, a red-zone overrun, a fusion that differs from its chain, or a host kernel that wrote
+  into an input exits 1. `--strict` also fails a stale fixture.
+- `--coverage DIR` keeps one file per fixture/library of the rows a replay reaches
+  (`data/kbench_rows/`); a listed row not reached fails, unless a memory skip kept its cases out.
+- Every pass ends with a line saying where its time went (draw, hashing, layout, upload, launch,
+  read-back, comparison, cache, timing).
+
+### Timing
+
+`--bench` times every case that passes, on the bytes it was checked on.
+
+- Device cases are timed **after** the checks, operands still on the device, after one sustained
+  warm-up. A time is device timestamps around a batch of back-to-back launches divided by the launch
+  count; the batch is queued behind a gate the host opens once submission is done, so neither the
+  host rate nor a sync is in the interval. Two untimed launches precede every batch; batches repeat
+  until the median's 95 % interval is within 1 % or `--bench-reps` is reached.
+- **Cold by rotation**: weights and inputs of 1/32 of the footprint or more are read from copies
+  spanning the footprint (`--bench-flush`, default 256 MiB, at least twice an R9700's caches), so
+  each launch reads them from memory as streamed weights do. Smaller inputs stay warm, as the
+  previous op leaves them.
+- In-place operands are restored from a pristine copy before every launch; the restores are timed
+  alone and subtracted.
+- Timing draws expert offsets/routing skewed (Zipf 0.5) and block tables as permutations; the
+  checked launch keeps the recorded draw.
+- The header prints the calibration: launch floor and streaming-read bandwidth warm and cold. If a
+  smaller cold probe runs >1.5× the largest's rate the rotation was cache-served and device times
+  are withheld. Throttler bits that came on during a batch repeat it, up to three times.
+- Host cases are timed in their own column, after an untimed launch, weights rotated over at least
+  256 MiB (twice LLC), for at most two seconds a case.
+
+### Flags
+
+| flag | meaning |
+|---|---|
+| `--fixture FILE` | replay recording (default `$RADIANCE_HOME/kernels.rkb`); repeatable |
+| `--record FILE` | with `-m`: write a fixture |
+| `--skeleton FILE` | with `-m`: write a sparse container copy |
+| `--rerecord FILE` | replay: write the fixtures' cases again with the reference's answers |
+| `--record-rows FILE` | replay: record a row-driven fixture from the cases |
+| `-m, --model FILE` | the `.rad` to read weight operands from |
+| `--home DIR`, `--kernels A:B` | plugin home; kernel libraries (default every one found) |
+| `--op NAME` | only this op (repeatable prefix match) |
+| `--ref NAME` | the oracle plugin's name (default: the library declaring itself a reference) |
+| `--no-ref` | replay without the reference among the tested libraries |
+| `--strict` | replay: a stale fixture fails |
+| `--all-rows` | run every row whose constraints hold |
+| `--coverage DIR`, `--coverage-write` | the row-coverage ratchet |
+| `--no-cache`, `--cache-dir DIR` | run the reference instead of reusing cached answers; cache location |
+| `--seed N` | base seed; every case derives its own |
+| `--tol X` | override every tolerance (bisecting only) |
+| `--max-ctx N` | deepest context considered (declare and the attention sweep) |
+| `--max-tok N` | `max_tok` declare runs at (default 512, small on purpose) |
+| `--mm-max-patches N` | encoder pass declare runs at (default 2048 when a tower is present; 0 leaves it out) |
+| `--kv-cache-dtype T` | `bf16` or `fp8`, as on the engine |
+| `--num-speculative-tokens N` | draft depth to declare at (default: the container's) |
+| `--max-cases N` | runs per distinct case (default 4; 0 = every one) |
+| `--max-bytes MIB` | skip a case needing more than this, by name (default: half available memory, 512–8192) |
+| `--no-fusion` | skip the fusion oracle |
+| `--bench`, `--bench-iters N`, `--bench-reps N`, `--bench-warm-ms N`, `--bench-flush MIB` | timing |
+| `--timing-only` | time every device kernel, check nothing (refuses `--record`) |
+| `--report FILE`, `--analysis FILE` | markdown report; splice in `tools/kbench-analysis.sh` output |
+| `-v` | every case, not only failures |
+
+Notes: `--max-bytes` matters on a loaded machine — one case holds each operand as drawn, as the
+kernel's copy, and on the device; without the cap a small machine is OOM-killed with no indication of
+the op. `--bench` times cold on purpose: an R9700's 52 MB bf16 weight read warm comes from cache at
+1.45 TB/s against 565 GB/s from memory, so warm times rank shapes but do not size a roofline. A
+kernel that cannot lay out an encoding is a **skip**, not a failure (`RAD_E_DTYPE`), and the report
+names the kernel, operand and encoding.
+
+## `rad-tune` — measure the instantiation space
+
+```
+rad-tune -m <model.rad> [options]
+
+  -m, --model FILE   the .rad whose declared graph supplies the shapes
+      --home DIR     $RADIANCE_HOME; cache at <home>/tune/<machine>.tune
+      --kernels A:B  kernel libraries to load, first wins
+      --op NAME      only ops whose name starts with NAME
+      --machine KEY  override the machine key (writing a cache for another machine)
+      --reps N       timing reps, median reported (default 5)
+      --iters N      launches per rep (default 32)
+      --max-tok N    max_tok declare runs at (default 8192)
+      --tp N         tensor-parallel width to declare at (default 1)
+      --num-speculative-tokens N|auto   draft depth to declare at
+      --dry-run      list what would be measured
+  -v, --verbose      every candidate, not just the winner
+```
+
+Kernels declare tunable axes, not named variants; `rad-tune` walks the cross product at this model's
+shapes and writes the winner per (op, geometry) to the cache the selector reads at startup.
+
+- **Declare at the shapes you serve**: `--tp N` divides every sharded N; a declare at world size 1
+  measures a program no `--tp 2` serve runs. `--num-speculative-tokens` puts the drafter's kernels in
+  the measured graph.
+- At `--tp > 1` every rank declares at once, each on its own thread and device: a collective's
+  `init` is a rendezvous, so declaring rank 0 alone hangs.
+- A collective is measured as the exchange: per-rank operands (own seeds, so identical halves cannot
+  pass), own scratch and stream, own instance; every rank fires at once and the clock stops when the
+  last stream drains. Non-collective rows run on rank 0 alone.
+- `RADIANCE_TUNE` pins a kernel's axes to compare a tuned cache against the defaults — a measurement
+  aid, not a deployment setting.
+
+## `rad-schemas` — the op vocabulary diff
+
+```
+rad-schemas libref.so                 print one plugin's vocabulary
+rad-schemas libr4d.so libref.so       diff them, in hierarchy order
+```
+
+The first plugin in hierarchy order to declare an op fixes its schema; a later plugin declaring the
+same op differently is refused at load, both named (arguments are positional, so disagreeing schemas
+would make one call mean different things). `rad-schemas` prints every disagreement at once, so a new
+plugin can be conformed before it is loaded beside another. It exits 1 on any disagreement. It links
+no core — it `dlopen`s plugins and reads static tables, so it works on a plugin the engine cannot
+load.
+
+## `rad-avx-check` — host libraries without a card
+
+```
+rad-avx-check --ref libref.so --avx LIB.so [options]
+
+  --ref PATH     the oracle plugin      (default libref.so)
+  --avx PATH     the plugin under test  (default libavx.so)
+  --op NAME      only this op
+  --level N      only this ISA level (0 scalar, 1 avx, 2 avx2, 3 avx512)
+  --threads N    rerun every case with each parallel region on N threads (default 4; 0 skips)
+  --tol X        override every tolerance
+  -v             every case
+```
+
+Checks a host library op by op against `libref`, on the CPU, with neither engine nor card: two
+`dlopen`s, a geometry table (`libavx/avx_cases.cpp`), libref's operand descriptions and hand-built
+cases. For every case: schemas as the loader compares them, outputs within `rad-kbench`'s tolerances,
+guard bytes, inputs unchanged, and an oracle output that is not just the draw. `libavx` is also
+checked at every ISA level the CPU executes and with parallel regions forced onto several threads.
+A new host library is checked with the same command; it is treated as a host library (not `libavx`),
+so libref ops it has no row for are listed rather than failed, and the ISA/thread sweeps run only if
+it exports `avx_isa_force`/`avx_par_force`. A row returning `RAD_E_UNSUPPORTED` for a geometry libref
+serves fails: the selector would resolve to it and the step would fail at issue.
+
+## `r4d_selftest` — libr4d's own check and timer
+
+```
+r4d_selftest <libr4d.so> [--case <row>]
+r4d_selftest <libr4d.so> --perf N K M [iters]      (and the other --perf* modes in its usage)
+```
+
+With no mode it runs the catalogue and static checks, then each row's smoke and numeric cases in a
+child process of its own, with fusion claims compared byte for byte (`ctest` labels `slow`, `gpu`).
+`--case` runs one row in-process as the reproducer. `--perf` covers the fp8 decode GEMMs and head;
+`--perfmxd` the dense OCP-MXFP4 rows with the weight bandwidth each reaches; `--perfmx` the MXFP4
+expert GEMM; `--cold MIB` times each launch alone behind a cache flush. `--perfar8` prints the
+eight-rank all-reduce shard weights and times `lockstep`/`ring`/`split` forms at every piece count;
+`R4D_SELFTEST_ONE_CARD=1` runs them on one card for their bits only.
+
+## `rad-tokcheck` — tokeniser against its references
+
+```
+rad-tokcheck [options] <tokenizer.json> <corpus-file>...
+
+  --hf-dir DIR    compare against HuggingFace ids in DIR/<corpus basename>.ids
+                  (little-endian int32, encode(text, add_special_tokens=False))
+  --no-ref        skip the reference encoder (slow)
+  --user          encode with parse_special off
+  --turns N       turns in the multi-turn measurement (default 40; 0 skips)
+  --repeat N      timing repeats, best taken (default 3)
+  --context N     ids shown either side of a difference (default 6)
+```
+
+Per corpus file: ids match `tools/tokref`'s reference encoder (llama.cpp's splitters and
+`std::regex`); ids match HuggingFace `tokenizers` read from files made elsewhere; and speed — MB/s
+for both encoders, plus the cost of re-tokenising a growing conversation with and without the
+segment cache. Any difference prints the ids and surrounding text and exits nonzero, so it gates a
+change.
+
+## `rad-chatparse` — where a tool call goes wrong
+
+```
+rad-chatparse --request REQ.json --rad MODEL.rad --reply R.txt
+rad-chatparse --render-raw T.jinja --context CTX.json
+```
+
+Renders one OpenAI chat request as the server would (container template or
+`--override-chat-template`, `--chat-template-kwargs`, the plugin's declared reply format, or the
+template's own with `--no-plugin`), then reads each reply file as the model's output: parsed whole,
+parsed through the SSE stream as a client reassembles it, and run through the request's tool-call
+grammar. It prints one JSON object; exit 1 when the two parses disagree or the grammar refuses a
+token. No weights are loaded.
+
+## The KL mode
+
+`radiance --kld-record/--kld-ref/--kld-corpus/--kld-out/--kld-lora` is a mode of the engine, not a
+separate tool: same placement and kernels, prefill instead of listening, then exit. The flags and the
+corpus format are on [[Radiance Conversion]].
+
+## Shell harnesses (`scripts/`)
+
+They drive a running engine: bring a model up, measure it, assert on what came back. Each header
+records why it measures the way it does.
+
+| script | what it does |
+|---|---|
+| `mcserve.sh`, `fnserve.sh` | bring MiniCPM5-2B+DSpark / Flash-Next up and leave it running (env knobs: `P TP MLEN SEQS M KVD HR MNBT EXTRA …`) |
+| `fnstop.sh` | stop whichever is running and wait for exit |
+| `convdspark.sh`, `convfn.sh` | rebuild the two reference containers (`convfn.sh` needs `CALIB=`) |
+| `hfget.sh` | resumable Hugging Face fetch with a content manifest |
+| `fnpf.sh`, `fnconc.sh`, `cstep2.sh`, `ident.sh`, `cardclock.py` | attach to a live server: prefill rate; prefill-vs-decode contention; steady-state decode ms/step per concurrency; run-to-run hashes; which clock limit binds and which rank the TP step waits for |
+| `fndrop.sh` | disconnect a client mid-request and check every other answer against a reference |
+| `tierident.sh`, `tiercross.sh` | whether a session that left VRAM returns byte for byte (attention and recurrent state), including two sessions in flight |
+| `mctools.sh`, `mcstream.sh`, `mcagent.sh`, `mcdecl.sh` | tool-call parsing shapes; streamed tool calls on raw SSE; speculation on an agentic workload; whether the drafter proposes |
+| `dprofd.sh`, `fntrace.sh`, `mctrace.sh`, `oracle.sh` | per-op device timing; rocprofv3 traces; every op against libref on the operands the model issued |
+| `kv-parity.sh`, `kv-stage-parity-64k.sh` | KV streaming: covering ring and staged prefill are byte-identical to streaming off |
+
+**Not all of them leave a server alone.** `fnserve.sh`, `mcserve.sh`, `fnstop.sh`, `fntrace.sh`,
+`dprofd.sh`, `mctrace.sh`, `mcagent.sh` and `mcdecl.sh` kill any running engine (`pkill` on the
+binary), whatever port it is on; `oracle.sh` stops whatever listens on its own port. `cstep2.sh`,
+`ident.sh`, `tierident.sh`, `tiercross.sh`, `mctools.sh`, `mcstream.sh`, `hfget.sh`, `convfn.sh`,
+`convdspark.sh`, `fnpf.sh`, `fnconc.sh`, `fndrop.sh` and `cardclock.py` leave it running
+(`fndrop.sh` deliberately disconnects clients). Never point the stopping ones at a production port.
+
+## Related
+
+- [[Radiance Conversion]] — `rad-convert`, `rad-export`, the KL mode.
+- [[Radiance Plugins]] — what `rad-kbench`/`rad-tune`/`rad-schemas` check.
+- [[Radiance Operations]] — the startup receipt these tools corroborate.

serverAppConfigs-docs/LLM/frank-archon.md

@@ -91,7 +91,7 @@ on stop. Boot budget 1800 s (the publisher's compose allows a 15-minute `start_p
 | Slots / window | `MAX_SEQS=4`, `MAX_MODEL_LEN=200000` (native 262,144; longer is refused, never truncated) — this pair is the **cache bound**: the resolver sizes the KV carve against `slots × window`, and no slot count funds itself (Laws) |
 | KV dtype | `KV_DTYPE=fp8` — the only dtype that funds the slot count (Laws) |
 | VRAM split | **derived at boot** from `GPU_HEADROOM_MIB=96` + the cache bound + `HOST_POOL_MIB`; naming `VRAM_WEIGHTS_MIB`/`VRAM_KV_MIB` is a boot refusal (removed at 1.3.0) |
-| Expert tier | `PLACEMENT=expert_tiered`, `HOST_POOL_MIB=49152` **per rank** — the pinned host pool is also the **KV lever**: the expert floor is `plane + 2 units − host room`, so every MiB pinned there lowers what the experts claim on the card one-for-one and the cache takes it (Laws) |
+| Expert tier | `PLACEMENT=expert_tiered`, `HOST_POOL_MIB=49152` **per rank** (raised to 66560 on 2026-10-29, folded back the same day — the A/B under *Laws* shows the raise bought no cache) — the pinned host pool is also the **KV lever**: the expert floor is `plane + 2 units − host room`, so every MiB pinned there lowers what the experts claim on the card one-for-one and the cache takes it (Laws) |
 | PLE | `NGRAM_PLACEMENT=ram` (47.7 GiB pinned; refuses boot on a short memlock) |
 | Spec / batch | `SPEC_TOKENS=2` — MTP depth 2, the `.rad` carries the model's own `mtp.*` head (`mtp.fc_hidden`/`fc_embedding`/`draft_head`); `MAX_BATCHED_TOKENS=2048` (MNBT — 2560 collapses decode, Laws) |
 | Prefix cache | `PREFIX_CACHE_MIB=8192`, pinned **host tier only** — the 64 GiB disk tier is gone, so an evicted prefix is re-prefilled; `STATE_DIR=/home/dank/radiance-state` now exists only to hold the rad-tune cache |
@@ -100,18 +100,27 @@ on stop. Boot budget 1800 s (the publisher's compose allows a 15-minute `start_p
 | TTM gate | `TTM_MIN_MIB=162000`, `TTM_TARGET_MIB=204800` |
 | Boot | `RAD_MIN_MIB=110000` deployer floor; the listener binds only after the full load (health ~100 s, budget 1800 s) |
 
-Boot ledger for the live shape (2026-10-20, pin 1.8.1, `MAX_SEQS=4`, `HOST_POOL_MIB=49152`):
-the resolver prints `elastic 26.73 GiB` → **experts 20.72 GiB (78 %) / kv cache 5.64 GiB (21 %)**,
-`prefill staging 373.65 MiB` out of the cache, `vram_weights 24.30 GiB`, `host_pool 48.59 GiB`.
-The referee line: `kv: the cache can grow to 5.64 GiB a card — 747,492 tokens of context, 3.7
-sessions at 200,000 — with every block it lends the experts taken back; 440.04 MiB of that is
-per-sequence state, and up to 5.20 GiB is lent to the experts while no request holds it`.
-`--checkpoint-slots 40` is auto (`max_num_seqs × 10` slots of 111.98 MiB = 4,479 MiB of pinned
-host), `/health` reports `queue_cap 32` (`max(8, seqs × 8)`), sampler **12 rows**
-(= `seqs × (1 + speculative depth)`), 4 hardware queues a card, template 28,234 B, vision live
-(≤ 4,194,304 px, 16,384 patches/pass), P2P confirmed (`the 2 cards reach each other's memory
-directly`). Experts living in the pinned host pool: **8.99 GiB a rank** (was 5.99) — the price of
-the bigger cache, paid over PCIe when those experts are hit.
+Boot ledger for the live payload. Last boot: 2026-10-29 at `MAX_SEQS=4`,
+`HOST_POOL_MIB=66560`, `MAX_BATCHED_TOKENS=2048`; the live pool is `49152` again, and the
+two boots are **identical above `host_pool`**, so one ledger serves both shapes:
+`elastic 26.73 GiB` → **experts 20.72 GiB (78 %) / kv cache 5.64 GiB (21 %)**,
+`activation arena 990.36 MiB` (the price of the 2,048-token step: `256 tokens use
+155.60 MiB, 1024 tokens use 472.86 MiB of 896.20 MiB`), `prefill staging 373.65 MiB`,
+`vram_weights 24.30 GiB`, `host_pool 65.59 GiB` (opened with `0 B used`; `48.59 GiB` at
+the 49,152 boot), `unclaimed 96.89 MiB`; referee line `kv: the pool backs 93% of the worst
+case (4 sequences all at 200000 tokens), 3.7 full-length sessions`; `device: the 2 cards
+reach each other's memory directly (peer to peer)`; sampler **12 rows**
+(= `seqs × (1 + speculative depth)`), 4 hardware queues a card (133.45 MiB), `already held
+256.67 MiB`, mover `17519 slab slots, 7624 pinned pool slots, 3.58 GiB static vram`,
+`load: rank 0, mover took 24.30 GiB`.
+
+The A/B that ended the raise: the 2026-10-20 boot at `HOST_POOL_MIB=49152` printed the
+**same** `elastic 26.73 / experts 20.72 / kv cache 5.64 / vram_weights 24.30` (Laws: the
+lever saturates at the expert floor), and the campaign rig — the other payload that
+inherits this env — moved `vram_kv 269.99 → 271.25 MiB`, a rounding step, because its
+window is set by its own n-gram table (95.37 GiB) and expert plane rather than by the
+pool. The refused calibration documents are fixed by cutting them to the window, not by
+widening the pool.
 
 The boot **ledger**, not this arithmetic, is the referee: radiance states every budget before
 anything is allocated (weights vs KV, the host pool's coverage of the worst case, the PLE pin,
@@ -150,6 +159,7 @@ marked "re-fetch" is gone from `/home/dank/models/`, which today holds only `Sti
 
 | Window | Payload | Runtime | Shape | Replaced because | Rollback |
 |---|---|---|---|---|---|
+| 2026-10-20 → 10-29 | same `qwen3.8-next-flash-fp8-iq4r-moe.rad` (payload unchanged) | radiance **1.8.1**, `sha256:d590e300…` (same digest) | 4 × 200,000 fp8 KV — resolver cache 5.64 GiB = 747,492 tokens, `prefill staging 373.65 MiB`, `vram_weights 24.30 GiB`, `host_pool 48.59 GiB`; host pool 49,152 MiB a rank, `TTM_MIN_MIB=162000` / `TTM_TARGET_MIB=204800`, `queue_cap 32`, sampler 12 rows, `--prefill-interleave 1` | Frank ordered the campaign's step (2,048) onto production; the pool half of that order (66,560) booted, came back byte-identical to this row's cache, and was folded back the same day — the calibration documents it was meant to save were cut to the window instead (`scripts/rechunk-kld-corpus.py`) | the live shape is this row's pool with `MAX_BATCHED_TOKENS=2048`: the pair is the rollback unit, not the merge commit (`scripts/frank-archon.env`) |
 | 2026-10-19 → 10-20 | same `qwen3.8-next-flash-fp8-iq4r-moe.rad` (payload unchanged) | radiance **1.8.1**, `sha256:d590e300…` (same digest) | 2 × 200,000 fp8 KV — resolver cache 2.76 GiB = 365,204 tokens (91 % backing); host pool 44,928 MiB a rank, `TTM_MIN_MIB=150000`, `queue_cap 16`, sampler 9 rows; `--prefill-interleave 1` armed on 10-20 | Frank ordered **four** concurrent conversations over a ≥ 700,000-token pool; the resolver funds that only by lowering the expert floor with more pinned host room (Laws) | revert `1bb6399` and redeploy `scripts/deploy.sh dankserver --service frank-archon` — trio back to `MAX_SEQS=2` / `HOST_POOL_MIB=44928` / `TTM_MIN_MIB=150000` |
 | 2026-10-15 → 10-19 | same `qwen3.8-next-flash-fp8-iq4r-moe.rad` (payload unchanged) | radiance **1.0.13**, `sha256:4c3f4845…` | 3 × 200,000 fp8 KV bought with the **explicit** pair `--vram-weights-mib 25600 --vram-kv-mib 5120`; host pool 44,928 → 45,568 MiB a rank; prefix cache 8 GiB host + 64 GiB disk | decode turned to sludge beside any long prompt — step contention, 18.7 ms decode-only steps against 317 ms steps carrying a prefill chunk (Laws) — and the only two knobs that fix it, `--busy-prefill-chunk` and `--prefill-interleave`, exist only after this pin | `FRANK_ARCHON_IMAGE` back to `sha256:4c3f4845…` **plus a trio rewrite**: restore the pair and the host+disk prefix knobs, delete `BUSY_PREFILL_CHUNK`/`PREFILL_INTERLEAVE`. A digest swap alone is a boot refusal (Runtime) |
 | 2026-10-08 → 10-15 | in-house `qwen4exp-swift15-w4dn8-i8-hc8m.rad`, 116,416 MiB (Swift1.5 abliterated BF16, GPTQ w4 experts + 24 bf16 protected, KLD 0.0131 / top-1 97.92 %) | radiance 1.0.13, same digest | 3 × 200,000 fp8 KV, slab 25,600 / pool 5,120 — the shape the live file still carries | the engine author's own container is the supported path and his quality ceiling; the finetune's refusal-bending left the endpoint | `.rad` purged 2026-10-17 (trash-staged then emptied, Frank's approval). Re-download `frank8465/Swift-1.5-Qwen3.8-Flash-Next-GPTQ-W4A8-Radiance-AMD-R9700` → `qwen4exp-w4nl64-i8-hc8m.rad`, 122,071,973,888 B, LFS sha256 `09dd9306…`, via the iq4r fetch script's hfget/verify pattern retargeted at that repo; point `MODEL_FILE` at it, redeploy. Re-**convert** is dead: the BF16 source was `rm`'d 2026-10-15 (`kld-ref`, grams, corpora and `/home/dank/logs/swift15_*.log` survive) |
@@ -243,7 +253,8 @@ vllm-radiance era (this table opens on 09-26; the era page claims 10-10).
   | 3 × 200,000, host pool 44,928 | 600,000 tok | 4.20 GiB | 93 % |
   | 3 × 200,000, host pool 45,568 | 600,000 tok | **4.20 GiB — byte-identical** | 93 % |
   | 2 × 200,000, host pool 44,928 | 400,000 tok | 2.76 GiB | 91 % |
-  | 4 × 200,000, host pool 49,152 (live) | 800,000 tok | **5.64 GiB = 747,492 tok** | 93 % |
+  | 4 × 200,000, host pool 49,152 (live) | 800,000 tok | 5.64 GiB = 747,492 tok | 93 % |
+  | 4 × 200,000, host pool 66,560 (2026-10-29 A/B, folded back) | 800,000 tok | **5.64 GiB — byte-identical to 49,152** | 93 % |
 
   The 45,568 row is the ceiling case, not a mystery: at 3 slots the cache already held its whole
   `ceiling`, so extra host room could not grow it and shows up as `unclaimed`. **The referee is the
@@ -551,6 +562,29 @@ vllm-radiance era (this table opens on 09-26; the era page claims 10-10).
   fork bomb inside the container. Kill with `podman rm -f` (the cgroup owns every generation)
   never pattern-kill an interpreter name on this host; use `TORCH_LOGS` for dynamo questions.
 
+### The quantization pipeline
+
+- **A calibration tap is only a tap when the rig that opened it is running.** The runner writes
+  `.rig-boot` into the tap dir (container, image, pid) and retires it when its boot exits non-zero;
+  `calib-push.py` refuses without that marker, refuses when it names another container, and refuses
+  when the named container is not running; `qwen38-calib-window.sh` stops on a failed phase instead
+  of falling through its health wait. All three exist because on 2026-10-29 the unguarded chain's
+  rig boot died of an OOM, the health loop expired without failing, and the pusher fed four rounds
+  of 10M tokens into **restored production** — 86 minutes of live prefill that recorded nothing.
+- **The n-gram table's pin is what starves a rig boot, and the tier is a knob.** On the BF16 payload
+  the table is 95.37 GiB, `--ngram-placement ram` pins it, and the calibration boot died at
+  `startup stopped: out of memory` with 230.22 GiB pinned of 246.07 GiB and 8.06 GiB available —
+  while the reference recording and every score boot succeeded at the same pool and slot count.
+  Calibration only needs the routed weights prefilled, so it reads the table from the container's
+  file tier (`--ngram-placement disk`); kldref and score keep production's placement, because their
+  numbers are only comparable at an identical shape.
+- **Acceptance is three metrics and every slice, not a mean.** `scripts/kld-gate-qwen38.sh` grades
+  mean KL, the p99.9 tail, top-1 agreement and every `by_source` slice (code, prose, tools):
+  a mean can hide a few positions taking the whole hit, and an overall win can be bought by
+  regressing one distribution. The FP8-native anchor is an interval, not a requirement — without
+  the 172.8 GiB fetch the gate runs in two-report mode (candidate vs the served payload) and says
+  so on its first line.
+
 ## Contract (the wire)
 
 - **Live surface:** `/health` (== `/ping`), `/v1/models`, `/v1/chat/completions`
@@ -627,11 +661,19 @@ vllm-radiance era (this table opens on 09-26; the era page claims 10-10).
   rebuild), `scripts/download-frank-archon-swift15.sh` (the BF16 source a w4dn8 re-convert
   needs), `scripts/hfget.sh` / `scripts/hf-fetch.sh`; template
   `scripts/qwen38-froggeric-v22.5.jinja`.
-- Convert campaign (dormant, kept for a future quant): `scripts/convert-swift15-w4dn8.sh`
-  `scripts/recipes/qwen4exp-swift15-w4dn8-i8-hc8m.recipe`, `scripts/run-swift15-bf16.sh`
-  `scripts/kld-gate-swift15.sh`, `scripts/kld-then-calib.sh`, `scripts/calib-push.py`
-  `scripts/make-calib-corpus.py`, `scripts/derive-protected-experts.py`
-  `scripts/apply-protected-cast.sh`, `scripts/verify-swift15-ple.py`.
+- The campaign (qwen38-w5nl64, live): `scripts/run-qwen38-bf16.sh` (kldref / calib / score — the
+  ONE measuring shape), `scripts/qwen38-calib-window.sh` (the gated calibration window),
+  `scripts/convert-qwen38-w5nl64.sh` with
+  `scripts/recipes/qwen4exp-qwen38-w5nl64-bf16trunk.recipe` (the arm driver and its template),
+  `scripts/kld-gate-qwen38.sh` (acceptance), `scripts/calib-push.py`,
+  `scripts/make-calib-corpus.py`, `scripts/rechunk-kld-corpus.py`,
+  `scripts/qwen38_common.py` (the tokenizer, JSONL and refusal plumbing those three
+  corpus tools share — one `n_tokens`, so every tool counts the same way),
+  `scripts/derive-protected-experts.py`, `scripts/apply-protected-cast.sh`.
+- Retired swift15-era campaign scripts, superseded by the above and kept only as reference:
+  `scripts/convert-swift15-w4dn8.sh`, `scripts/recipes/qwen4exp-swift15-w4dn8-i8-hc8m.recipe`,
+  `scripts/run-swift15-bf16.sh`, `scripts/kld-gate-swift15.sh`, `scripts/kld-then-calib.sh`,
+  `scripts/verify-swift15-ple.py`.
 - Probes: `scripts/probe-frank-archon-kv.py` (reads slots/window/dtype from the deployed env so
   it cannot drift; leg D = decode), `scripts/frank-archon-expert-activity.py`
   `scripts/autoresearch-frank-archon.sh` + `scripts/autoresearch/frank-archon-measure.py`.
@@ -641,5 +683,6 @@ vllm-radiance era (this table opens on 09-26; the era page claims 10-10).
 - Retired-era leftovers in-tree: `scripts/patches/glm53-*.patch`
   `scripts/patches/gpu_worker-kv-cap.patch` (fork archaeology; none applies byte-exact today).
 
-Related: [[frank-dragoon]] (the abliterated family), [[frank-zealot]],
----
+Related: [[frank-dragoon]] (the abliterated family), [[frank-zealot]], and the engine reference —
+[[Radiance Engine]], [[Radiance Serve Options]], [[Radiance Conversion]], [[Radiance Plugins]],
+[[Radiance Operations]].

serverAppConfigs-docs/LLM/frank-dragoon.md

@@ -10,14 +10,15 @@ tags:
   - gfx1151
   - halogen
   - endpoint
-timestamp: 2026-10-19
+timestamp: 2026-10-21
 ---
 
 # frank-dragoon — halogen-flash-server 0.16.2, native v2 abliterated (Dankclaw iGPU)
 
 Strix Halo (gfx1151) endpoint `http://192.168.0.143:7770`, served id `frank-dragoon`: a podman container
 under a root `User=dank` unit. Live shape since 2026-10-17, pool raised to 401,408 on 2026-10-18
-(`2b12716`). This page replaces every per-era dragoon runbook; the Eras table is the rollback record.
+(`2b12716`), ceiling raised to the native 262,144 on 2026-10-21 so the harness window is 200,000.
+This page replaces every per-era dragoon runbook; the Eras table is the rollback record.
 
 ## Current shape
 
@@ -37,10 +38,12 @@ Image `ghcr.io/peonist-ai/halogen-flash-server@sha256:0c61bf84ac22308a53f5d1ca6b
 unauthenticated token port 8730 stays on container loopback. Engine stdout →
 `/home/dank/logs/frank_dragoon.log`, **not** the journal.
 
-- Shape: `HALOGEN_CTX=200704` hard per-request ceiling · `HALOGEN_KV_SLOTS=2` ·
-  `HALOGEN_KV_POOL_POSITIONS=401408` = slots × ctx, the top of the band `tests/frank_dragoon_unit.bats`
-  enforces (`ctx ≤ pool ≤ slots × ctx`) · KV ~28 KiB/position → **11.0 GiB reserved** · no KV-dtype knob
-  exists here, the pool is the only KV lever · native window 262,144, `HALOGEN_ROPE_YARN` beyond it.
+- Shape: `HALOGEN_CTX=262144` hard per-request ceiling = the checkpoint's native window (raised
+  2026-10-21 from 200704; `HALOGEN_ROPE_YARN` beyond native is not on plan) · `HALOGEN_KV_SLOTS=2` ·
+  `HALOGEN_KV_POOL_POSITIONS=401408` inside the band `tests/frank_dragoon_unit.bats` enforces
+  (`ctx ≤ pool ≤ slots × ctx`) · KV ~28 KiB/position → **11.0 GiB reserved** · no KV-dtype knob
+  exists here, the pool is the only KV lever · the pool carries TWO maximum harness-length (200k)
+  conversations at once; two literal ceiling-length requests would queue — that is a pool raise.
 - Knobs: `HALOGEN_MAX_TOK=8192` sizes the prefill arena (5.7 GiB) and is only the default for requests
   omitting `max_tokens` · `HALOGEN_MAX_TOKENS_CAP=65536` + `HALOGEN_QUEUE_TIMEOUT=7200` are a coupled
   pair · MTP depth 2 with the head INSIDE the checkpoint (**`HALOGEN_MTP_HEAD` stays unset**) and
@@ -53,10 +56,11 @@ unauthenticated token port 8730 stays on container loopback. Engine stdout →
   (Pin::None)`; no clamp line. Host `MemAvailable` read 22 GiB right after the 2026-10-18 raise —
   ~20 GiB above the 20 GiB reserve, against 10–15 GiB of baseline co-tenancy (frank-zealot, ComfyUI,
   Caddy/observability).
-- Harness (`~/.omp/agent/models.yml`, provider `llamacpp (dankclaw)`): `contextWindow: 155648`
-  (= ceiling − 45k per the 400-zone law, never higher), `maxTokens: 45056`, `input: [text, image]`
-  `reasoningContentField: "reasoning_content"`, `high→xhigh` fold, `extraBody` 1.0/0.95. Compaction
-  rides [[frank-zealot]].
+- Harness (`~/.omp/agent/models.yml`, provider `llamacpp (dankclaw)`): `contextWindow: 200000`
+  — sits 62,144 under the ceiling, clearing the ~45k fit-law gap (400-zone law below);
+  `tests/frank_dragoon_unit.bats` fails if the page's declared window and `HALOGEN_CTX` drift back
+  together. `maxTokens: 45056`, `input: [text, image]`, `reasoningContentField: "reasoning_content"`,
+  `high→xhigh` fold, `extraBody` 1.0/0.95. Compaction rides [[frank-zealot]].
 - Deploy `./scripts/deploy.sh dankclaw --service frank-dragoon`; gate = boot-ledger greps (weights,
   pool, MTP, prompt cache, **no clamp**) + `/health` 200 + a tool call + the 512 px red-PNG probe.
   Kernel: `CONFIG_HSA_AMD_SVM=y` (CachyOS ships it) and `amdgpu.noretry=0` — the env says check it at
@@ -124,14 +128,18 @@ created it — earlier topologies in .
   prompt tokens. halogen 0.1.4's auto-sizing refused here and re-prefilled every 79–81k turn in
   85–180 s until `HALOGEN_CACHE_MB=16384` was pinned. What the pool takes is what the table loses, so
   re-measure the warm ratio after any pool change.
-- **400-zone law:** the harness window MUST sit ~45k under the engine ceiling (`contextWindow:
-  155648` = 200,704 − 45,056) because the harness clamps `max_tokens` to `contextWindow − est − 4096`
-  its chars/4 `est` under-counts code/agent traffic by ~3 %, and compaction only fires at 85 %. The
-  3 % is measured, not theory: 2026-10-07 at the old 200,192 ceiling, est 152,938 vs server 157,388 —
-  over by 354, a 400. At the live shape compaction fires at 132,301 est ≈ 136k measured (the 180k
-  target of the 262,144 era is unreachable — the window itself is 155,648), and the worst-case request
-  is 155,648 − 4,096 + 3 % ≈ 156.2k ≤ 200,704: ~9.5× the error of margin, robust to a ~25 %
-  under-count. Compaction rides [[frank-zealot]] at ≈ threshold − keepRecent(20k) ≈ 112k input.
+- **400-zone law:** the harness window MUST sit ~45k under the engine ceiling, because the harness
+  clamps `max_tokens` to `contextWindow − est − 4096`, its chars/4 `est` UNDER-counts code/agent
+  traffic by ~3 %, and the compaction trigger fires at 95 % of the window (`config.yml
+  thresholdPercent`, while the comments designed around 85 % — check the trigger BEFORE touching a
+  window; that disagreement is why long sessions kept drawing 400s). The 3 % is measured, not
+  theory: 2026-10-07 at the old 200,192 ceiling, est 152,938 vs server 157,388 — over by 354, a 400.
+  At the live pair (ceiling 262,144 / window 200,000) the worst-case ordinary request is
+  200,000 − 4,096 + 3 % ≈ 201.8k, and the trigger point is 190,000 est × 1.03 + 45,056 ≈ 240.7k —
+  both inside the ceiling with ~60k of margin. Compaction rides [[frank-zealot]] at
+  ≈ threshold − keepRecent(20k) ≈ 175k measured against its 200,704-token slot. The seam in
+  `tests/frank_dragoon_unit.bats` re-runs this arithmetic at the 95 % trigger and reads the window
+  from THIS page, so ceiling/window pairs that drift back together fail the suite, not a session.
 - **Admission reserves prompt + `max_tokens`:** an un-fittable request waits in arrival order, then
   503s `engine_busy` for the whole queue after `QUEUE_TIMEOUT` — and the cap/timeout pair moves
   together: 65536 tokens at ~10 t/s serial needs >6550 s, which is why 7200 is pinned (env, 2026-10-17).
@@ -248,9 +256,14 @@ lane is **armed and serving** since 2026-10-19 (`HALOGEN_NPU_MODELS=qwen3-embedd
    (`dankclaw/udev/98-halogen-npu-access.rules`, installed and re-triggered by the deployer, `0666` like the GPU
    compute nodes beside it); the serve script refuses any node that is not other-accessible, so the fault cannot
    return as a crash-loop.
-2. The host's XRT reachable inside: there is **no `/opt/xilinx/xrt`** here — distro packages put the three files the
-   container needs (`libxrt_coreutil.so.2`, `libxrt_core.so.2`, `libxrt_driver_xdna.so.2`) in
-   `/usr/lib/x86_64-linux-gnu`, where all three exist.
+2. The host's XRT reachable inside, at **THREE prefixes per library**: there is **no `/opt/xilinx/xrt`** here —
+   distro packages put the three files the container needs (`libxrt_coreutil.so.2`, `libxrt_core.so.2`,
+   `libxrt_driver_xdna.so.2`) in `/usr/lib/x86_64-linux-gnu`. The serve script mounts each REAL file (never a
+   symlink) at `/opt/xilinx/xrt/lib/`, at its own path, AND at `/opt/xilinx/lib/<multiarch>/` — the NPU child
+   searches that third prefix and a missing one is fatal: since the host's `libxrt2` was reconfigured
+   2026-10-20 the child died with `No such library '/opt/xilinx/lib/x86_64-linux-gnu/libxrt_core.so.2'` and the
+   engine supervisor took the WHOLE container down on it (88-restart loop, chat dead beside a healthy GPU line).
+   A dead lane is not survivable here — arm the lane only with all three prefixes mounted.
 3. **The GPU fabric clock held at its top level, or the engine refuses to start.** GPU and NPU work share the die:
    while `pp_dpm_fclk` is changing speed, concurrent GPU work can hang the machine and corrupt NPU results. The
    engine holds the clock itself only when it runs as root with `/sys` mounted; `frank-dragoon` is a **rootless**

serverAppConfigs-docs/LLM/index.md

@@ -13,6 +13,20 @@ the endpoint's **Eras** table, and the numbers a retired era measured are gone.
 | [[frank-dragoon]] | The iGPU endpoint: live shape, every era with its rollback path, the laws it paid for, and the files that own it. |
 | [[frank-zealot]] | The RTX 3090 endpoint: live shape exactly as the env ships it, every era with its rollback path, and its card laws. |
 
+## Radiance engine reference
+
+| Page | Owns |
+|---|---|
+| [[Radiance Engine]] | The radiance inference engine: its shape, plugin model, container format, and how the reference pages fit together. |
+| [[Radiance Serve Options]] | Every radiance serve flag grouped by task, with its default and the law that decides when to move it. |
+| [[Radiance HTTP API]] | The HTTP surface radiance serves: endpoints, request fields, streaming, tools, and refusal rules. |
+| [[Radiance Conversion]] | Turning a checkpoint into a .rad container: rad-convert, the recipe language, quantisers, reuse and export. |
+| [[Radiance Model Families]] | Which radiance plugin claims which model family, and the conversion command and recipe of record for each. |
+| [[Radiance Plugins]] | Writing radiance kernel and quantiser plugins: the ABI, kernel rows, hooks, encodings, and the build path. |
+| [[Radiance Architecture Plugins]] | Writing a radiance architecture plugin: declaring weights, buffers, KV groups and ops, then issuing the step. |
+| [[Radiance Tools]] | The radiance inspection, measurement and tokeniser tools, with the flags and verdicts each one reports. |
+| [[Radiance Operations]] | Running a radiance server: the container contract, host prerequisites, readiness, ledgers and safe restarts. |
+
 ## Retired endpoints
 
 | Page | Owns |

serverAppConfigs-docs/Operations/Payload and Disk Reclaim.md

@@ -107,6 +107,23 @@ curl -s -o /dev/null -w '%{http_code}\n' http://127.0.0.1:7770/health
 Report: staged items and sizes, what was kept and why, what is **still not
 freed**, remaining candidates untouched. Purge only on explicit approval.
 
+## 6. The approved purge — `scripts/purge-agenttrash.sh`
+
+Approval turns into one command. The script is POSIX sh, dry-run by default,
+and touches nothing but `.AGENTTRASH` directories: pass it the machine's roots
+(default `$HOME`) and it prints the plan; `--execute` performs it and appends
+one dated line to `$HOME/.AGENTTRASH-cleanup.log`. A repo's (or the home's own)
+`.AGENTTRASH` is **emptied in place** — the directory is the convention and
+survives — while `.AGENTTRASH` trees that are themselves inside staged content
+are removed with their parent. Foreign-owned and symlinked candidates are
+refused, never followed; `rm -rf` deletes symlink entries but never traverses
+them, so a planted world-writable trash cannot reach outside itself.
+
+```sh
+scripts/purge-agenttrash.sh /home/dank          # plan
+scripts/purge-agenttrash.sh --execute /home/dank  # purge
+```
+
 ## Repo-flow gotcha: do NOT push before you deploy
 
 `scripts/deploy.sh` sets `BASE_REVISION=$(git rev-parse origin/main)`. If you

serverAppConfigs-docs/index.md

@@ -162,6 +162,15 @@ is a locator: one row, one clause, no numbers.
 | [[PCIe Link and Atomics]] | Link probing, the streaming-versus-compute ceiling arithmetic, and the atomics wall that decides multi-GPU collectives. |
 | [[Quantization by Hardware Architecture]] | This page maps LLM quantization formats to the GPU and CPU architectures that support them. |
 | [[Quantization Production]] | Making a quantized payload here: the conversion recipe, the scoring gate, and the campaign ladder behind a shipped file. |
+| [[Radiance Architecture Plugins]] | Writing a radiance architecture plugin: declaring weights, buffers, KV groups and ops, then issuing the step. |
+| [[Radiance Conversion]] | Turning a checkpoint into a .rad container: rad-convert, the recipe language, quantisers, reuse and export. |
+| [[Radiance Engine]] | The radiance inference engine: its shape, plugin model, container format, and how the reference pages fit together. |
+| [[Radiance HTTP API]] | The HTTP surface radiance serves: endpoints, request fields, streaming, tools, and refusal rules. |
+| [[Radiance Model Families]] | Which radiance plugin claims which model family, and the conversion command and recipe of record for each. |
+| [[Radiance Operations]] | Running a radiance server: the container contract, host prerequisites, readiness, ledgers and safe restarts. |
+| [[Radiance Plugins]] | Writing radiance kernel and quantiser plugins: the ABI, kernel rows, hooks, encodings, and the build path. |
+| [[Radiance Serve Options]] | Every radiance serve flag grouped by task, with its default and the law that decides when to move it. |
+| [[Radiance Tools]] | The radiance inspection, measurement and tokeniser tools, with the flags and verdicts each one reports. |
 | [[RDNA4 MoE Expert Offload VRAM Budget]] | Expert-offload arithmetic on the RDNA4 fork: movable terms, the utilisation law, graph charge, ceilings, context ladder. |
 | [[Shared Memory Architecture Dankclaw]] | Dankclaw uses AMD Strix Halo APU with unified memory architecture. |
 

shared/alloy/config.alloy

@@ -17,6 +17,9 @@ logging {
 // Loki is published on the host's loopback (127.0.0.1:3100). Alloy shares the
 // host network, so it pushes by loopback, never by container name — a root
 // container is not on the rootless dankclaw_default network.
+// GRAFANA_HOST_LABEL is replaced with this host's name by
+// copy_alloy_config at deploy time; it must not appear verbatim in the
+// installed config.
 loki.write "default" {
   endpoint {
     url = "http://127.0.0.1:3100/loki/api/v1/push"
@@ -32,7 +35,34 @@ loki.write "default" {
 loki.source.journal "journal" {
   format_as_json = true
   labels = {
-    job = "systemd-journal",
+    job  = "systemd-journal",
+    host = "GRAFANA_HOST_LABEL",
+  }
+  forward_to = [loki.process.relabel.receiver]
+}
+
+// Promote the fields that ARE the point of the log stream to real Loki labels.
+// Without this the journal arrives as one opaque JSON line per entry and every
+// per-container view is structurally impossible: Loki metrics queries can only
+// group by INDEXED stream labels, so `sum by (container_name)` returns a single
+// anonymous series and the logs-overview board read empty beside 6.5k
+// lines/5min of live traffic (measured 2026-10-21). CONTAINER_NAME is what
+// podman's journald logging writes (uppercase); host units lack it and stay on
+// the host-only stream. `format_as_json = true` above guarantees every line
+// reaching this stage is a JSON object, so there is no non-JSON branch to
+// protect — alloy 1.21's json stage has no `on_error` attribute (validated
+// against the pinned image); a malformed line would drop, which is the
+// correct fate for a line Loki could not index anyway.
+loki.process "relabel" {
+  stage.json {
+    expressions = {
+      container = "CONTAINER_NAME",
+    }
+  }
+  stage.labels {
+    values = {
+      container = "container",
+    }
   }
   forward_to = [loki.write.default.receiver]
 }

shared/blackbox/blackbox.yml

@@ -0,0 +1,17 @@
+# Blackbox exporter modules — the honest answer to "is this HTTP service
+# answering?". A Prometheus scrape cannot express it (an HTML front page is a
+# 200 that fails the scrape protocol), which is why the old `services` and
+# `comfyui` jobs sat permanently red on `unsupported Content-Type`.
+# See serverAppConfigs-docs/Architecture/Stale Signal Audit.md §1.
+# The listen address is NOT set here: the quadlet unit's Exec passes
+# --listen-address (LAN bind — see the unit header for why loopback is
+# unreachable from the bridge-networked Prometheus). Keep this file to
+# modules only, so the two sources cannot disagree.
+
+modules:
+  http_2xx:
+    prober: http
+    timeout: 10s
+    http:
+      preferred_ip_protocol: ip4
+      valid_status_codes: [] # default: 2xx + 3xx

shared/grafana/provisioning/dashboards/containers.json

@@ -104,14 +104,14 @@
     {
       "id": 6,
       "type": "timeseries",
-      "title": "Memory vs limit (%)",
-      "description": "Working set over the cgroup memory limit (container_spec_memory_limit_bytes). Only containers that carry a limit are shown; approaching 1.0 means an OOM is near. Containers with no limit are omitted.",
+      "title": "Memory vs host total (%)",
+      "description": "Working set over the host's total RAM. The fleet sets NO cgroup memory limits (every container_spec_memory_limit_bytes is 0 — a limit-share panel is structurally empty here), so the honest pressure view is share of the host. The join labels the node exporter's MemTotal onto the cadvisor series by host.",
       "gridPos": { "h": 9, "w": 12, "x": 12, "y": 11 },
       "datasource": { "type": "prometheus", "uid": "prometheus" },
       "fieldConfig": { "defaults": { "unit": "percentunit", "min": 0, "max": 1, "custom": { "fillOpacity": 8, "lineWidth": 1, "stacking": { "mode": "none" } } }, "overrides": [] },
       "targets": [
         {
-          "expr": "(container_memory_working_set_bytes{job=~\"cadvisor-.*\", host=~\"$host\", container=~\"$container\"} / on (job, host, container) group_left container_spec_memory_limit_bytes{job=~\"cadvisor-.*\", host=~\"$host\", container=~\"$container\"}) and (container_spec_memory_limit_bytes{job=~\"cadvisor-.*\", host=~\"$host\", container=~\"$container\"} > 0)",
+          "expr": "container_memory_working_set_bytes{job=~\"cadvisor-.*\", host=~\"$host\", container=~\"$container\", container!=\"\"} / on(host) group_left label_replace(node_memory_MemTotal_bytes{job=~\"node-.*\"}, \"host\", \"$1\", \"job\", \"node-(.*)\")",
           "legendFormat": "{{host}} / {{container}}",
           "refId": "A"
         }

shared/grafana/provisioning/dashboards/host-system.json

@@ -54,7 +54,7 @@
       "gridPos": { "h": 4, "w": 6, "x": 12, "y": 0 },
       "datasource": { "type": "prometheus", "uid": "prometheus" },
       "fieldConfig": { "defaults": { "unit": "short", "thresholds": { "mode": "absolute", "steps": [{ "color": "green", "value": null }, { "color": "yellow", "value": 0.7 }, { "color": "red", "value": 1 }] } }, "overrides": [] },
-      "targets": [{ "expr": "node_load5{job=~\"$node\"} / count by (job) (node_cpu_seconds_total{job=~\"$node\",mode=\"idle\"})", "legendFormat": "{{job}}", "refId": "A" }]
+      "targets": [{ "expr": "node_load5{job=~\"$node\"} / on(job) count by (job) (node_cpu_seconds_total{job=~\"$node\",mode=\"idle\"})", "legendFormat": "{{job}}", "refId": "A" }]
     },
     {
       "id": 4, "type": "stat", "title": "Open file handles",

shared/grafana/provisioning/dashboards/llm-fleet.json

@@ -194,7 +194,7 @@
       "id": 6,
       "type": "timeseries",
       "title": "Firing fleet alerts",
-      "description": "The LLM-fleet rules from shared/prometheus/rules/llm-fleet.yml. A firing alert is delivered to Signal: Prometheus -> Alertmanager (dankclaw quadlet) -> the host-side signal-relay user unit -> signal-cli, since 2026-09-17.",
+      "description": "The LLM-fleet rules from shared/prometheus/rules/llm-fleet.yml. EMPTY IS THE HEALTHY STATE: zero series means nothing is firing (ALERTS only carries active/pending alert series — do not read this panel as broken). A firing alert is delivered to Signal: Prometheus -> Alertmanager (dankclaw quadlet) -> the host-side signal-relay user unit -> signal-cli, since 2026-09-17.",
       "gridPos": {
         "h": 8,
         "w": 12,

shared/grafana/provisioning/dashboards/logs-overview.json

@@ -2,74 +2,242 @@
   "uid": "logs-overview",
   "title": "Logs \u2014 Loki (systemd journal)",
   "description": "Everything in Loki arrives as the journal (job=systemd-journal); the container is a JSON field parsed at query time, not a stream label. Starter breadth \u2014 the volume/error panels group by the parsed container_name. Loki keeps ~7 days; wide ranges are cheap counts, not the raw log panel.",
-  "tags": ["logs", "loki"],
+  "tags": [
+    "logs",
+    "loki"
+  ],
   "timezone": "browser",
   "schemaVersion": 39,
   "refresh": "30s",
-  "time": { "from": "now-1h", "to": "now" },
+  "time": {
+    "from": "now-1h",
+    "to": "now"
+  },
   "templating": {
     "list": [
       {
         "name": "container",
         "label": "container",
         "type": "query",
-        "datasource": { "type": "loki", "uid": "loki" },
-        "query": { "query": "{job=\"systemd-journal\"} | json", "label": "container_name", "refId": "A" },
+        "datasource": {
+          "type": "loki",
+          "uid": "loki"
+        },
+        "query": {
+          "query": "label_values({job=\"systemd-journal\"}, container)",
+          "refId": "LokiVariableQueryEditor-VariableQuery"
+        },
         "includeAll": true,
         "multi": true,
         "refresh": 2,
-        "current": { "text": "All", "value": "$__all" }
+        "current": {
+          "text": "All",
+          "value": "$__all"
+        }
       }
     ]
   },
   "panels": [
     {
-      "id": 1, "type": "timeseries", "title": "Log volume by container (lines/s)",
-      "description": "Parse the journal JSON, count lines per container. Spikes = a chatty or crashing unit.",
-      "gridPos": { "h": 9, "w": 24, "x": 0, "y": 0 },
-      "datasource": { "type": "loki", "uid": "loki" },
-      "fieldConfig": { "defaults": { "unit": "reqps", "custom": { "fillOpacity": 20, "stacking": { "mode": "normal" } } }, "overrides": [] },
-      "targets": [{ "expr": "sum by (container_name) (count_over_time({job=\"systemd-journal\"} | json [$__auto]))", "legendFormat": "{{container_name}}", "refId": "A" }]
+      "id": 1,
+      "type": "timeseries",
+      "title": "Log volume by container (lines/s)",
+      "description": "Per-container journal line rate. The `container` label is promoted at ingestion by Alloy (loki.process) from podman's CONTAINER_NAME journald field - Loki metrics can only group by indexed stream labels.",
+      "gridPos": {
+        "h": 9,
+        "w": 24,
+        "x": 0,
+        "y": 0
+      },
+      "datasource": {
+        "type": "loki",
+        "uid": "loki"
+      },
+      "fieldConfig": {
+        "defaults": {
+          "unit": "reqps",
+          "custom": {
+            "fillOpacity": 20,
+            "stacking": {
+              "mode": "normal"
+            }
+          }
+        },
+        "overrides": []
+      },
+      "targets": [
+        {
+          "expr": "sum by (container) (count_over_time({job=\"systemd-journal\", container!=\"\"} [$__auto]))",
+          "legendFormat": "{{container_name}}",
+          "refId": "A"
+        }
+      ]
     },
     {
-      "id": 2, "type": "timeseries", "title": "Total log rate",
+      "id": 2,
+      "type": "timeseries",
+      "title": "Total log rate",
       "description": "All journal lines per second.",
-      "gridPos": { "h": 8, "w": 12, "x": 0, "y": 9 },
-      "datasource": { "type": "loki", "uid": "loki" },
-      "fieldConfig": { "defaults": { "unit": "reqps" }, "overrides": [] },
-      "targets": [{ "expr": "sum(count_over_time({job=\"systemd-journal\"} [$__auto]))", "legendFormat": "all lines/s", "refId": "A" }]
+      "gridPos": {
+        "h": 8,
+        "w": 12,
+        "x": 0,
+        "y": 9
+      },
+      "datasource": {
+        "type": "loki",
+        "uid": "loki"
+      },
+      "fieldConfig": {
+        "defaults": {
+          "unit": "reqps"
+        },
+        "overrides": []
+      },
+      "targets": [
+        {
+          "expr": "sum(count_over_time({job=\"systemd-journal\"} [$__auto]))",
+          "legendFormat": "all lines/s",
+          "refId": "A"
+        }
+      ]
     },
     {
-      "id": 3, "type": "timeseries", "title": "Error-level line rate",
+      "id": 3,
+      "type": "timeseries",
+      "title": "Error-ish line rate (text match)",
       "description": "Lines matching error/panic/fatal/fail. A blunt text match \u2014 the journal carries no structured level for all units.",
-      "gridPos": { "h": 8, "w": 12, "x": 12, "y": 9 },
-      "datasource": { "type": "loki", "uid": "loki" },
-      "fieldConfig": { "defaults": { "unit": "reqps", "color": { "mode": "fixed", "fixedColor": "red" } }, "overrides": [] },
-      "targets": [{ "expr": "sum(count_over_time({job=\"systemd-journal\"} |~ \"(?i)(error|panic|fatal|fail)\" [$__auto]))", "legendFormat": "error lines/s", "refId": "A" }]
+      "gridPos": {
+        "h": 8,
+        "w": 12,
+        "x": 12,
+        "y": 9
+      },
+      "datasource": {
+        "type": "loki",
+        "uid": "loki"
+      },
+      "fieldConfig": {
+        "defaults": {
+          "unit": "reqps",
+          "color": {
+            "mode": "fixed",
+            "fixedColor": "red"
+          }
+        },
+        "overrides": []
+      },
+      "targets": [
+        {
+          "expr": "sum(count_over_time({job=\"systemd-journal\"} |~ \"(?i)(error|panic|fatal|fail)\" [$__auto]))",
+          "legendFormat": "error lines/s",
+          "refId": "A"
+        }
+      ]
     },
     {
-      "id": 4, "type": "timeseries", "title": "Errors by container",
+      "id": 4,
+      "type": "timeseries",
+      "title": "Errors by container",
       "description": "Which unit is throwing. Rank by the top offenders.",
-      "gridPos": { "h": 8, "w": 12, "x": 0, "y": 17 },
-      "datasource": { "type": "loki", "uid": "loki" },
-      "fieldConfig": { "defaults": { "unit": "reqps" }, "overrides": [] },
-      "targets": [{ "expr": "sum by (container_name) (count_over_time({job=\"systemd-journal\"} |~ \"(?i)(error|panic|fatal)\" | json [$__auto]))", "legendFormat": "{{container_name}}", "refId": "A" }]
+      "gridPos": {
+        "h": 8,
+        "w": 12,
+        "x": 0,
+        "y": 17
+      },
+      "datasource": {
+        "type": "loki",
+        "uid": "loki"
+      },
+      "fieldConfig": {
+        "defaults": {
+          "unit": "reqps"
+        },
+        "overrides": []
+      },
+      "targets": [
+        {
+          "expr": "sum by (container) (count_over_time({job=\"systemd-journal\", container!=\"\"} |~ \"(?i)(error|panic|fatal)\" [$__auto]))",
+          "legendFormat": "{{container_name}}",
+          "refId": "A"
+        }
+      ]
     },
     {
-      "id": 5, "type": "stat", "title": "Error lines (last 15m)",
+      "id": 5,
+      "type": "stat",
+      "title": "Error lines (last 15m)",
       "description": "Absolute count over the window, not a rate.",
-      "gridPos": { "h": 8, "w": 12, "x": 12, "y": 17 },
-      "datasource": { "type": "loki", "uid": "loki" },
-      "fieldConfig": { "defaults": { "unit": "short", "thresholds": { "mode": "absolute", "steps": [{ "color": "green", "value": null }, { "color": "yellow", "value": 50 }, { "color": "red", "value": 500 }] } }, "overrides": [] },
-      "targets": [{ "expr": "sum(count_over_time({job=\"systemd-journal\"} |~ \"(?i)(error|panic|fatal)\" [15m]))", "instant": true, "legendFormat": "errors/15m", "refId": "A" }]
+      "gridPos": {
+        "h": 8,
+        "w": 12,
+        "x": 12,
+        "y": 17
+      },
+      "datasource": {
+        "type": "loki",
+        "uid": "loki"
+      },
+      "fieldConfig": {
+        "defaults": {
+          "unit": "short",
+          "thresholds": {
+            "mode": "absolute",
+            "steps": [
+              {
+                "color": "green",
+                "value": null
+              },
+              {
+                "color": "yellow",
+                "value": 50
+              },
+              {
+                "color": "red",
+                "value": 500
+              }
+            ]
+          }
+        },
+        "overrides": []
+      },
+      "targets": [
+        {
+          "expr": "sum(count_over_time({job=\"systemd-journal\"} |~ \"(?i)(error|panic|fatal)\" [15m]))",
+          "instant": true,
+          "legendFormat": "errors/15m",
+          "refId": "A"
+        }
+      ]
     },
     {
-      "id": 6, "type": "logs", "title": "Journal (filtered)",
+      "id": 6,
+      "type": "logs",
+      "title": "Journal (filtered)",
       "description": "Raw lines for the selected containers. Set the container picker; keep the window tight.",
-      "gridPos": { "h": 12, "w": 24, "x": 0, "y": 25 },
-      "datasource": { "type": "loki", "uid": "loki" },
-      "options": { "showTime": true, "wrapLogMessage": true, "sortOrder": "Descending", "enableLogDetails": true },
-      "targets": [{ "expr": "{job=\"systemd-journal\"} | json | container_name=~\"$container\"", "refId": "A" }]
+      "gridPos": {
+        "h": 12,
+        "w": 24,
+        "x": 0,
+        "y": 25
+      },
+      "datasource": {
+        "type": "loki",
+        "uid": "loki"
+      },
+      "options": {
+        "showTime": true,
+        "wrapLogMessage": true,
+        "sortOrder": "Descending",
+        "enableLogDetails": true
+      },
+      "targets": [
+        {
+          "expr": "{job=\"systemd-journal\", container=~\"$container\"}",
+          "refId": "A"
+        }
+      ]
     }
   ]
-}
+}

shared/grafana/provisioning/dashboards/observability.json

@@ -2,95 +2,384 @@
   "uid": "observability",
   "title": "Observability \u2014 self-monitoring",
   "description": "The health of the stack that watches the fleet: Prometheus scrape/TSDB, Loki ingest, Grafana. If this is red the dashboards lie. Note: Alertmanager is running but NOT scraped by Prometheus (no job) \u2014 its metrics are absent by design gap, not fault.",
-  "tags": ["observability", "meta"],
+  "tags": [
+    "observability",
+    "meta"
+  ],
   "timezone": "browser",
   "schemaVersion": 39,
   "refresh": "30s",
-  "time": { "from": "now-6h", "to": "now" },
+  "time": {
+    "from": "now-6h",
+    "to": "now"
+  },
   "panels": [
     {
-      "id": 1, "type": "stat", "title": "Targets up",
-      "gridPos": { "h": 4, "w": 4, "x": 0, "y": 0 },
-      "datasource": { "type": "prometheus", "uid": "prometheus" },
-      "fieldConfig": { "defaults": { "unit": "short", "thresholds": { "mode": "absolute", "steps": [{ "color": "green", "value": null }] } }, "overrides": [] },
-      "targets": [{ "expr": "count(up == 1)", "instant": true, "refId": "A" }]
+      "id": 1,
+      "type": "stat",
+      "title": "Targets up",
+      "gridPos": {
+        "h": 4,
+        "w": 4,
+        "x": 0,
+        "y": 0
+      },
+      "datasource": {
+        "type": "prometheus",
+        "uid": "prometheus"
+      },
+      "fieldConfig": {
+        "defaults": {
+          "unit": "short",
+          "thresholds": {
+            "mode": "absolute",
+            "steps": [
+              {
+                "color": "green",
+                "value": null
+              }
+            ]
+          }
+        },
+        "overrides": []
+      },
+      "targets": [
+        {
+          "expr": "count(up == 1)",
+          "instant": true,
+          "refId": "A"
+        }
+      ]
     },
     {
-      "id": 2, "type": "stat", "title": "Targets down",
-      "description": "Known structural downs: comfyui (no metrics endpoint), gitea (metrics off), and the homepage 'services' probes. Real regressions ride up from here too \u2014 check the table before dismissing.",
-      "gridPos": { "h": 4, "w": 4, "x": 4, "y": 0 },
-      "datasource": { "type": "prometheus", "uid": "prometheus" },
-      "fieldConfig": { "defaults": { "unit": "short", "thresholds": { "mode": "absolute", "steps": [{ "color": "green", "value": null }, { "color": "red", "value": 1 }] } }, "overrides": [] },
-      "targets": [{ "expr": "count(up == 0)", "instant": true, "refId": "A" }]
+      "id": 2,
+      "type": "stat",
+      "title": "Targets down",
+      "description": "Every scrape target that is down. ZERO IS THE HEALTHY STATE: the HTML-probe jobs and the disabled Gitea metrics were retired 2026-10-21 (HTTP liveness now rides the blackbox exporter, job \"services\"). A non-zero count here is a real regression - read /api/v1/targets lastError before triaging.",
+      "gridPos": {
+        "h": 4,
+        "w": 4,
+        "x": 4,
+        "y": 0
+      },
+      "datasource": {
+        "type": "prometheus",
+        "uid": "prometheus"
+      },
+      "fieldConfig": {
+        "defaults": {
+          "unit": "short",
+          "thresholds": {
+            "mode": "absolute",
+            "steps": [
+              {
+                "color": "green",
+                "value": null
+              },
+              {
+                "color": "red",
+                "value": 1
+              }
+            ]
+          }
+        },
+        "overrides": []
+      },
+      "targets": [
+        {
+          "expr": "count(up == 0)",
+          "instant": true,
+          "refId": "A"
+        }
+      ]
     },
     {
-      "id": 3, "type": "stat", "title": "TSDB head series",
+      "id": 3,
+      "type": "stat",
+      "title": "TSDB head series",
       "description": "Active series in memory. A jump = a new exporter or a label-cardinality leak.",
-      "gridPos": { "h": 4, "w": 5, "x": 8, "y": 0 },
-      "datasource": { "type": "prometheus", "uid": "prometheus" },
-      "fieldConfig": { "defaults": { "unit": "short" }, "overrides": [] },
-      "targets": [{ "expr": "prometheus_tsdb_head_series", "instant": true, "refId": "A" }]
+      "gridPos": {
+        "h": 4,
+        "w": 5,
+        "x": 8,
+        "y": 0
+      },
+      "datasource": {
+        "type": "prometheus",
+        "uid": "prometheus"
+      },
+      "fieldConfig": {
+        "defaults": {
+          "unit": "short"
+        },
+        "overrides": []
+      },
+      "targets": [
+        {
+          "expr": "prometheus_tsdb_head_series",
+          "instant": true,
+          "refId": "A"
+        }
+      ]
     },
     {
-      "id": 4, "type": "stat", "title": "Discovered targets",
-      "gridPos": { "h": 4, "w": 5, "x": 13, "y": 0 },
-      "datasource": { "type": "prometheus", "uid": "prometheus" },
-      "fieldConfig": { "defaults": { "unit": "short" }, "overrides": [] },
-      "targets": [{ "expr": "sum(prometheus_sd_discovered_targets)", "instant": true, "refId": "A" }]
+      "id": 4,
+      "type": "stat",
+      "title": "Discovered targets",
+      "gridPos": {
+        "h": 4,
+        "w": 5,
+        "x": 13,
+        "y": 0
+      },
+      "datasource": {
+        "type": "prometheus",
+        "uid": "prometheus"
+      },
+      "fieldConfig": {
+        "defaults": {
+          "unit": "short"
+        },
+        "overrides": []
+      },
+      "targets": [
+        {
+          "expr": "sum(prometheus_sd_discovered_targets)",
+          "instant": true,
+          "refId": "A"
+        }
+      ]
     },
     {
-      "id": 5, "type": "stat", "title": "Prometheus mem",
-      "gridPos": { "h": 4, "w": 3, "x": 18, "y": 0 },
-      "datasource": { "type": "prometheus", "uid": "prometheus" },
-      "fieldConfig": { "defaults": { "unit": "bytes" }, "overrides": [] },
-      "targets": [{ "expr": "process_resident_memory_bytes{job=\"prometheus\"}", "instant": true, "refId": "A" }]
+      "id": 5,
+      "type": "stat",
+      "title": "Prometheus mem",
+      "gridPos": {
+        "h": 4,
+        "w": 3,
+        "x": 18,
+        "y": 0
+      },
+      "datasource": {
+        "type": "prometheus",
+        "uid": "prometheus"
+      },
+      "fieldConfig": {
+        "defaults": {
+          "unit": "bytes"
+        },
+        "overrides": []
+      },
+      "targets": [
+        {
+          "expr": "process_resident_memory_bytes{job=\"prometheus\"}",
+          "instant": true,
+          "refId": "A"
+        }
+      ]
     },
     {
-      "id": 6, "type": "stat", "title": "Grafana up",
-      "gridPos": { "h": 4, "w": 3, "x": 21, "y": 0 },
-      "datasource": { "type": "prometheus", "uid": "prometheus" },
-      "fieldConfig": { "defaults": { "mappings": [{ "type": "value", "options": { "1": { "text": "UP", "color": "green" }, "0": { "text": "DOWN", "color": "red" } } }], "thresholds": { "mode": "absolute", "steps": [{ "color": "red", "value": null }, { "color": "green", "value": 1 }] } }, "overrides": [] },
-      "targets": [{ "expr": "up{job=\"grafana\"}", "instant": true, "refId": "A" }]
+      "id": 6,
+      "type": "stat",
+      "title": "Grafana up",
+      "gridPos": {
+        "h": 4,
+        "w": 3,
+        "x": 21,
+        "y": 0
+      },
+      "datasource": {
+        "type": "prometheus",
+        "uid": "prometheus"
+      },
+      "fieldConfig": {
+        "defaults": {
+          "mappings": [
+            {
+              "type": "value",
+              "options": {
+                "1": {
+                  "text": "UP",
+                  "color": "green"
+                },
+                "0": {
+                  "text": "DOWN",
+                  "color": "red"
+                }
+              }
+            }
+          ],
+          "thresholds": {
+            "mode": "absolute",
+            "steps": [
+              {
+                "color": "red",
+                "value": null
+              },
+              {
+                "color": "green",
+                "value": 1
+              }
+            ]
+          }
+        },
+        "overrides": []
+      },
+      "targets": [
+        {
+          "expr": "up{job=\"grafana\"}",
+          "instant": true,
+          "refId": "A"
+        }
+      ]
     },
     {
-      "id": 7, "type": "table", "title": "Target health",
+      "id": 7,
+      "type": "table",
+      "title": "Target health",
       "description": "Every scrape target and its current up. Sort by Value to surface regressions.",
-      "gridPos": { "h": 9, "w": 12, "x": 0, "y": 4 },
-      "datasource": { "type": "prometheus", "uid": "prometheus" },
-      "options": { "sortBy": [{ "displayName": "Value", "sortOrder": "Asc" }] },
-      "targets": [{ "expr": "up", "format": "table", "instant": true, "refId": "A" }]
+      "gridPos": {
+        "h": 9,
+        "w": 12,
+        "x": 0,
+        "y": 4
+      },
+      "datasource": {
+        "type": "prometheus",
+        "uid": "prometheus"
+      },
+      "options": {
+        "sortBy": [
+          {
+            "displayName": "Value",
+            "sortOrder": "Asc"
+          }
+        ]
+      },
+      "targets": [
+        {
+          "expr": "up",
+          "format": "table",
+          "instant": true,
+          "refId": "A"
+        }
+      ]
     },
     {
-      "id": 8, "type": "timeseries", "title": "Scrape interval drift (p99 actual)",
+      "id": 8,
+      "type": "timeseries",
+      "title": "Scrape interval drift (p99 actual)",
       "description": "Actual seconds between scrapes (summary quantile). Above the 15s scrape_interval = Prometheus is falling behind its own cadence (slow endpoints or overload).",
-      "gridPos": { "h": 9, "w": 12, "x": 12, "y": 4 },
-      "datasource": { "type": "prometheus", "uid": "prometheus" },
-      "fieldConfig": { "defaults": { "unit": "s" }, "overrides": [] },
-      "targets": [{ "expr": "prometheus_target_interval_length_seconds{quantile=\"0.99\"}", "legendFormat": "{{instance}}", "refId": "A" }]
+      "gridPos": {
+        "h": 9,
+        "w": 12,
+        "x": 12,
+        "y": 4
+      },
+      "datasource": {
+        "type": "prometheus",
+        "uid": "prometheus"
+      },
+      "fieldConfig": {
+        "defaults": {
+          "unit": "s"
+        },
+        "overrides": []
+      },
+      "targets": [
+        {
+          "expr": "prometheus_target_interval_length_seconds{quantile=\"0.99\"}",
+          "legendFormat": "{{instance}}",
+          "refId": "A"
+        }
+      ]
     },
     {
-      "id": 9, "type": "timeseries", "title": "Scrape samples/s",
+      "id": 9,
+      "type": "timeseries",
+      "title": "Scrape samples/s",
       "description": "Total samples ingested across all jobs.",
-      "gridPos": { "h": 8, "w": 12, "x": 0, "y": 13 },
-      "datasource": { "type": "prometheus", "uid": "prometheus" },
-      "fieldConfig": { "defaults": { "unit": "short" }, "overrides": [] },
-      "targets": [{ "expr": "sum(rate(scrape_samples_scraped[5m]))", "legendFormat": "samples/s", "refId": "A" }]
+      "gridPos": {
+        "h": 8,
+        "w": 12,
+        "x": 0,
+        "y": 13
+      },
+      "datasource": {
+        "type": "prometheus",
+        "uid": "prometheus"
+      },
+      "fieldConfig": {
+        "defaults": {
+          "unit": "short"
+        },
+        "overrides": []
+      },
+      "targets": [
+        {
+          "expr": "sum(rate(scrape_samples_scraped[5m]))",
+          "legendFormat": "samples/s",
+          "refId": "A"
+        }
+      ]
     },
     {
-      "id": 10, "type": "timeseries", "title": "Loki request rate",
+      "id": 10,
+      "type": "timeseries",
+      "title": "Loki request rate",
       "description": "Loki served requests/s (push, query). A drop to zero means ingest or query has stopped.",
-      "gridPos": { "h": 8, "w": 12, "x": 12, "y": 13 },
-      "datasource": { "type": "prometheus", "uid": "prometheus" },
-      "fieldConfig": { "defaults": { "unit": "reqps" }, "overrides": [] },
-      "targets": [{ "expr": "sum by (route) (rate(loki_request_duration_seconds_count[5m]))", "legendFormat": "{{route}}", "refId": "A" }]
+      "gridPos": {
+        "h": 8,
+        "w": 12,
+        "x": 12,
+        "y": 13
+      },
+      "datasource": {
+        "type": "prometheus",
+        "uid": "prometheus"
+      },
+      "fieldConfig": {
+        "defaults": {
+          "unit": "reqps"
+        },
+        "overrides": []
+      },
+      "targets": [
+        {
+          "expr": "sum by (route) (rate(loki_request_duration_seconds_count[5m]))",
+          "legendFormat": "{{route}}",
+          "refId": "A"
+        }
+      ]
     },
     {
-      "id": 11, "type": "timeseries", "title": "Exporter process CPU",
+      "id": 11,
+      "type": "timeseries",
+      "title": "Exporter process CPU",
       "description": "CPU by monitored process \u2014 spot a runaway exporter or the Prometheus process itself.",
-      "gridPos": { "h": 8, "w": 24, "x": 0, "y": 21 },
-      "datasource": { "type": "prometheus", "uid": "prometheus" },
-      "fieldConfig": { "defaults": { "unit": "percentunit" }, "overrides": [] },
-      "targets": [{ "expr": "sum by (job) (rate(process_cpu_seconds_total[5m]))", "legendFormat": "{{job}}", "refId": "A" }]
+      "gridPos": {
+        "h": 8,
+        "w": 24,
+        "x": 0,
+        "y": 21
+      },
+      "datasource": {
+        "type": "prometheus",
+        "uid": "prometheus"
+      },
+      "fieldConfig": {
+        "defaults": {
+          "unit": "percentunit"
+        },
+        "overrides": []
+      },
+      "targets": [
+        {
+          "expr": "sum by (job) (rate(process_cpu_seconds_total[5m]))",
+          "legendFormat": "{{job}}",
+          "refId": "A"
+        }
+      ]
     }
   ]
-}
+}

shared/prometheus/prometheus.yml

@@ -60,20 +60,33 @@ scrape_configs:
       - targets: ["loki:3100"]
     metrics_path: "/metrics"
 
-  # ComfyUI (no native /metrics; probe the UI)
-  - job_name: "comfyui"
-    metrics_path: "/"
-    static_configs:
-      - targets: ["comfyui:8188"]
-
-  # Blackbox-style probes for service up/down (published LAN ports;
-  # gitea and comfyui are covered by their own jobs above)
+  # HTTP liveness, probed honestly through the blackbox exporter. Scraping a
+  # web root is NOT a health check — an HTML front page fails the scrape
+  # protocol (`unsupported Content-Type`), which is exactly why the old
+  # `services` and `comfyui` jobs were red forever beside healthy services
+  # (Stale Signal Audit §1). The prober is a host-netns container listening on
+  # the podman gateway (the scrape address below - the netavark hairpin gap
+  # bars this container from its own host's LAN IP, cadvisor job note) and on
+  # the LAN; from that netns it reaches the host LAN IP, and published ports
+  # via 127.0.0.1 - bridge container names do NOT resolve in the host netns,
+  # which is why comfyui's probe is a loopback URL. comfyui's own
+  # metrics-scrape job is retired, one owner per service.
   - job_name: "services"
-    metrics_path: "/"
+    metrics_path: /probe
+    params:
+      module: [http_2xx]
     static_configs:
       - targets:
-          - "192.168.0.143:8081"   # SearXNG (via the Gluetun tunnel)
-          - "192.168.0.143:3005"   # OpenWebUI
+          - "http://192.168.0.143:8081"   # SearXNG (via the Gluetun tunnel)
+          - "http://192.168.0.143:3005"   # OpenWebUI
+          - "http://127.0.0.1:8188"        # ComfyUI UI (published 0.0.0.0:8188; probed from the prober's host netns)
+    relabel_configs:
+      - source_labels: [__address__]
+        target_label: __param_target
+      - source_labels: [__param_target]
+        target_label: instance
+      - target_label: __address__
+        replacement: "host.containers.internal:9115"
 
   # LLM fleet. Scraped on the LAN addresses where the endpoint is reachable from
   # this container, not the guarded vhosts: these are machine clients and
@@ -140,12 +153,15 @@ scrape_configs:
   # system.slice daemons and machine.slice scopes carry no such name and are
   # dropped, as is the per-CPU-core split (only cpu="total" survives), so the
   # scrape stays small. cadvisor cannot bind per-interface, so it is reached at
-  # the host's PUBLISHED LAN port: unlike node-exporter's host-net bind (which
-  # is why node-dankclaw scrapes via host.containers.internal), a published port
-  # IS reachable from this bridge container at the host LAN address.
+  # the podman gateway like every same-host target: a BRIDGE CONTAINER CANNOT
+  # DIAL ITS OWN HOST'S LAN IP AT ALL — the netavark hairpin gap refuses the
+  # connection whether the port is a host-net bind or a published port (the
+  # original note here claimed published ports were exempt; measured 2026-10-21
+  # from inside this container: LAN refused, gateway answers). Cross-host
+  # targets (dankserver) have no hairpin problem and stay plain LAN.
   - job_name: "cadvisor-dankclaw"
     static_configs:
-      - targets: ["192.168.0.143:8080"]
+      - targets: ["host.containers.internal:8080"]
         labels:
           host: dankclaw
     metric_relabel_configs:

tests/caddyfile.bats

@@ -193,7 +193,18 @@ CADDYFILE="$REPO_ROOT/dankclaw/Caddyfile"
 }
 
 @test "prometheus targets are host:port (no URL schemes)" {
-    ! grep -nE '^\s*-\s*"?http' "$REPO_ROOT/shared/prometheus/prometheus.yml"
+    # Plain scrape targets are host:port; a URL target is only meaningful for
+    # the blackbox prober job, whose relabel_configs copy it to __param_target
+    # and point __address__ at the prober. Anywhere else a URL target is the
+    # old HTML-probe defect wearing a new hat.
+    awk '
+        /job_name:/ { job=$0; sub(/.*"/, "", job); sub(/".*/, "", job); probe = 0 }
+        /metrics_path: \/probe/ { probe = 1 }
+        /^[[:space:]]*-[[:space:]]*"http/ && !probe {
+            print FILENAME ":" FNR ": URL target outside a prober job: " $0; bad = 1
+        }
+        END { exit bad }
+    ' "$REPO_ROOT/shared/prometheus/prometheus.yml"
 }
 
 @test "observability containers run as container root (rootless = host user)" {